2024-04-16T15:59:38.326 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-16T15:59:38.326 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211 --name leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/metrics} --verbose -- /home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211/orig.config.yaml 2024-04-16T15:59:38.375 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T15:59:38.455 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211 branch: wip-lusov-shared-log-segment description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/metrics} email: null first_in_suite: false job_id: '7658211' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-shared-log-segment ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds debug zombie log segments: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f kclient: syntax: v2 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-shared-log-segment sha1: c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f owner: scheduled_leonidus@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 2402 sha1: c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f sleep_before_teardown: 0 subset: 111/111111 suite: fs suite_branch: wip-lusov-shared-log-segment suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDUcFo+1C2BSV3ek3BKzwod8oCPB6aZrPtasnCnVl6dygq8XCIbHNDNU52dSpibNF24UahNi9guSMLh9OOEjgJ8= smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEPWcbwSh0lFEkxkF0bbAAqBwyITW4H1XFt//MpKTkkqQPLdN+m3u/lwLQsQJTlaCBI9Jd0PvGwcT00Zkht+jM8= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_mds_metrics teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-16_12:36:42 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-16T15:59:38.455 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/qa; will attempt to use it 2024-04-16T15:59:38.456 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/qa/tasks 2024-04-16T15:59:38.456 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-16T15:59:38.458 INFO:teuthology.task.internal:Checking packages... 2024-04-16T15:59:38.481 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f' 2024-04-16T15:59:38.482 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T15:59:38.482 INFO:teuthology.packaging:ref: None 2024-04-16T15:59:38.482 INFO:teuthology.packaging:tag: None 2024-04-16T15:59:38.482 INFO:teuthology.packaging:branch: wip-lusov-shared-log-segment 2024-04-16T15:59:38.482 INFO:teuthology.packaging:sha1: c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f 2024-04-16T15:59:38.482 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-shared-log-segment 2024-04-16T15:59:38.630 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2868.gc3c87ed0 2024-04-16T15:59:38.632 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-16T15:59:38.638 INFO:teuthology.task.internal:no buildpackages task found 2024-04-16T15:59:38.638 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-16T15:59:38.669 INFO:teuthology.task.internal:Saving configuration 2024-04-16T15:59:38.686 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-16T15:59:38.692 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T15:59:38.718 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 15:54:00.936964', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDUcFo+1C2BSV3ek3BKzwod8oCPB6aZrPtasnCnVl6dygq8XCIbHNDNU52dSpibNF24UahNi9guSMLh9OOEjgJ8='} 2024-04-16T15:59:38.737 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 15:54:00.938850', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEPWcbwSh0lFEkxkF0bbAAqBwyITW4H1XFt//MpKTkkqQPLdN+m3u/lwLQsQJTlaCBI9Jd0PvGwcT00Zkht+jM8='} 2024-04-16T15:59:38.737 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-16T15:59:38.743 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-16T15:59:38.743 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-16T15:59:38.743 INFO:teuthology.run_tasks:Running task console_log... 2024-04-16T15:59:38.815 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f85fc9fa160>, signals=[15]) 2024-04-16T15:59:38.815 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-16T15:59:38.821 INFO:teuthology.task.internal:Opening connections... 2024-04-16T15:59:38.821 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T15:59:38.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T15:59:38.929 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T15:59:38.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T15:59:39.018 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-16T15:59:39.025 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-04-16T15:59:39.042 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-04-16T15:59:39.042 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-04-16T15:59:39.099 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream" 2024-04-16T15:59:39.099 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="9" 2024-04-16T15:59:39.099 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2024-04-16T15:59:39.099 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2024-04-16T15:59:39.099 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="9" 2024-04-16T15:59:39.099 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el9" 2024-04-16T15:59:39.099 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T15:59:39.099 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2024-04-16T15:59:39.099 INFO:teuthology.orchestra.run.smithi045.stdout:LOGO="fedora-logo-icon" 2024-04-16T15:59:39.099 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T15:59:39.100 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/" 2024-04-16T15:59:39.100 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T15:59:39.100 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T15:59:39.100 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T15:59:39.100 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-04-16T15:59:39.120 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-04-16T15:59:39.137 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-04-16T15:59:39.137 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-04-16T15:59:39.194 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-04-16T15:59:39.194 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-04-16T15:59:39.194 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-04-16T15:59:39.194 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-04-16T15:59:39.194 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-04-16T15:59:39.194 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-04-16T15:59:39.194 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T15:59:39.194 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-04-16T15:59:39.194 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-04-16T15:59:39.194 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T15:59:39.195 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-04-16T15:59:39.195 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T15:59:39.195 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T15:59:39.195 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T15:59:39.195 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-04-16T15:59:39.211 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-16T15:59:39.221 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-16T15:59:39.226 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-16T15:59:39.226 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-04-16T15:59:39.228 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-04-16T15:59:39.250 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-16T15:59:39.257 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-16T15:59:39.257 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-04-16T15:59:39.285 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-04-16T15:59:39.337 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-16T15:59:39.392 INFO:teuthology.run_tasks:Running task kernel... 2024-04-16T15:59:39.412 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-04-16T15:59:39.412 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-04-16T15:59:39.412 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-04-16T15:59:39.413 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T15:59:39.413 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T15:59:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T15:59:39.430 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-16T15:59:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T15:59:39.434 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-16T15:59:39.607 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T15:59:39.607 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-16T15:59:39.607 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-04-16T15:59:39.625 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-361.el9.x86_64 2024-04-16T15:59:39.625 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T15:59:39.625 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-16T15:59:39.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum upgrade -y linux-firmware 2024-04-16T15:59:39.755 DEBUG:teuthology.task.kernel:sha1 for client.1 is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T15:59:39.755 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-16T15:59:39.755 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-04-16T15:59:39.772 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-361.el9.x86_64 2024-04-16T15:59:39.773 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T15:59:39.773 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-16T15:59:39.773 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum upgrade -y linux-firmware 2024-04-16T15:59:41.743 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-04-16T15:59:42.254 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - AppStream 9.2 MB/s | 6.7 MB 00:00 2024-04-16T15:59:43.177 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-04-16T15:59:43.662 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-04-16T15:59:44.025 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-04-16T15:59:45.753 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-04-16T15:59:46.001 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-04-16T15:59:47.580 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-04-16T15:59:50.160 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-16T15:59:52.869 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 16 kB 00:01 2024-04-16T15:59:53.088 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.2 MB 00:01 2024-04-16T15:59:54.723 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-16T15:59:54.898 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-16T15:59:56.483 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-16T16:00:00.062 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-16T16:00:01.491 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-16T16:00:02.656 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T16:00:02.666 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T16:00:02.667 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-04-16T16:00:02.667 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T16:00:02.667 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2024-04-16T16:00:02.667 INFO:teuthology.orchestra.run.smithi045.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-04-16T16:00:02.667 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:00:02.667 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T16:00:02.667 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T16:00:02.667 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 1 Package 2024-04-16T16:00:02.667 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:00:02.667 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 375 M 2024-04-16T16:00:02.667 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T16:00:04.003 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-16T16:00:04.011 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-16T16:00:04.011 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-16T16:00:04.011 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-16T16:00:04.011 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-04-16T16:00:04.011 INFO:teuthology.orchestra.run.smithi103.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-04-16T16:00:04.011 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:00:04.011 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-16T16:00:04.011 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-16T16:00:04.011 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 1 Package 2024-04-16T16:00:04.011 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:00:04.012 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 375 M 2024-04-16T16:00:04.012 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-16T16:00:12.750 INFO:teuthology.orchestra.run.smithi103.stdout:linux-firmware-20240219-143.el9.noarch.rpm 43 MB/s | 375 MB 00:08 2024-04-16T16:00:12.751 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-16T16:00:12.752 INFO:teuthology.orchestra.run.smithi103.stdout:Total 43 MB/s | 375 MB 00:08 2024-04-16T16:00:12.752 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-16T16:00:12.847 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-16T16:00:12.847 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-16T16:00:14.779 INFO:teuthology.orchestra.run.smithi045.stdout:linux-firmware-20240219-143.el9.noarch.rpm 31 MB/s | 375 MB 00:12 2024-04-16T16:00:14.780 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T16:00:14.780 INFO:teuthology.orchestra.run.smithi045.stdout:Total 31 MB/s | 375 MB 00:12 2024-04-16T16:00:14.780 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T16:00:14.888 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T16:00:14.888 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T16:00:15.245 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-16T16:00:15.245 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-16T16:00:17.395 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T16:00:17.395 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T16:00:21.753 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-16T16:00:22.071 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T16:00:22.161 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T16:00:23.479 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T16:00:23.479 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T16:00:23.906 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T16:00:23.906 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:00:23.907 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-04-16T16:00:23.907 INFO:teuthology.orchestra.run.smithi103.stdout: linux-firmware-20240219-143.el9.noarch 2024-04-16T16:00:23.907 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:00:23.907 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-16T16:00:24.108 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T16:00:24.217 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.1... 2024-04-16T16:00:24.217 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T16:00:24.342 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T16:00:24.441 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T16:00:24.478 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ 2024-04-16T16:00:24.478 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-16T16:00:24.481 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi103.front.sepia.ceph.com to complete... 2024-04-16T16:00:25.705 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T16:00:25.705 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T16:00:26.157 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T16:00:26.158 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:00:26.158 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2024-04-16T16:00:26.158 INFO:teuthology.orchestra.run.smithi045.stdout: linux-firmware-20240219-143.el9.noarch 2024-04-16T16:00:26.158 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:00:26.158 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T16:00:26.543 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.0... 2024-04-16T16:00:26.544 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T16:00:26.696 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ 2024-04-16T16:00:26.696 DEBUG:teuthology.orchestra.run.smithi045:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-16T16:00:26.699 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi045.front.sepia.ceph.com to complete... 2024-04-16T16:00:48.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16 16:00:48 URL:https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm [357344123/357344123] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-16T16:00:48.190 INFO:teuthology.orchestra.run.smithi045.stderr:FINISHED --2024-04-16 16:00:48-- 2024-04-16T16:00:48.190 INFO:teuthology.orchestra.run.smithi045.stderr:Total wall clock time: 21s 2024-04-16T16:00:48.190 INFO:teuthology.orchestra.run.smithi045.stderr:Downloaded: 1 files, 341M in 21s (16.2 MB/s) 2024-04-16T16:00:48.194 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.0... 2024-04-16T16:00:48.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-16T16:00:48.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16 16:00:48 URL:https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm [357344123/357344123] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-16T16:00:48.318 INFO:teuthology.orchestra.run.smithi103.stderr:FINISHED --2024-04-16 16:00:48-- 2024-04-16T16:00:48.318 INFO:teuthology.orchestra.run.smithi103.stderr:Total wall clock time: 24s 2024-04-16T16:00:48.318 INFO:teuthology.orchestra.run.smithi103.stderr:Downloaded: 1 files, 341M in 24s (14.5 MB/s) 2024-04-16T16:00:48.320 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.1... 2024-04-16T16:00:48.321 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-16T16:00:50.353 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying... ######################################## 2024-04-16T16:00:50.371 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying... ######################################## 2024-04-16T16:00:50.416 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing... ######################################## 2024-04-16T16:00:50.425 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing... ######################################## 2024-04-16T16:00:50.500 INFO:teuthology.orchestra.run.smithi045.stdout:Updating / installing... 2024-04-16T16:00:50.504 INFO:teuthology.orchestra.run.smithi103.stdout:Updating / installing... 2024-04-16T16:00:58.476 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-6.9.0_rc3_g3f60ed14f7c1-1 ######################################## 2024-04-16T16:00:58.815 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-6.9.0_rc3_g3f60ed14f7c1-1 ######################################## 2024-04-16T16:01:15.404 INFO:teuthology.orchestra.run.smithi045.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T16:01:16.159 INFO:teuthology.orchestra.run.smithi103.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T16:01:39.092 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-16T16:01:39.092 DEBUG:teuthology.orchestra.run.smithi045:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-16T16:01:39.161 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-16T16:01:39.161 DEBUG:teuthology.orchestra.run.smithi103:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-16T16:01:39.205 INFO:teuthology.orchestra.run.smithi045.stdout:/boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:39.205 INFO:teuthology.orchestra.run.smithi045.stdout:/boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:39.205 INFO:teuthology.orchestra.run.smithi045.stdout:/boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:39.205 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/System.map 2024-04-16T16:01:39.205 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/config 2024-04-16T16:01:39.205 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel 2024-04-16T16:01:39.206 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch 2024-04-16T16:01:39.206 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86 2024-04-16T16:01:39.206 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto 2024-04-16T16:01:39.206 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-16T16:01:39.206 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-16T16:01:39.206 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-16T16:01:39.206 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-16T16:01:39.206 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-16T16:01:39.206 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-16T16:01:39.206 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-16T16:01:39.206 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-16T16:01:39.206 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-16T16:01:39.207 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-16T16:01:39.207 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-16T16:01:39.207 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-16T16:01:39.207 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-16T16:01:39.207 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-16T16:01:39.207 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-16T16:01:39.207 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-16T16:01:39.207 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-16T16:01:39.207 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-16T16:01:39.207 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-16T16:01:39.207 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-16T16:01:39.207 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel 2024-04-16T16:01:39.208 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu 2024-04-16T16:01:39.208 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce 2024-04-16T16:01:39.208 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-16T16:01:39.208 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm 2024-04-16T16:01:39.208 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-16T16:01:39.208 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-16T16:01:39.208 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-16T16:01:39.208 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto 2024-04-16T16:01:39.208 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-16T16:01:39.208 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-16T16:01:39.208 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-16T16:01:39.208 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx 2024-04-16T16:01:39.208 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-16T16:01:39.209 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-16T16:01:39.209 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-16T16:01:39.209 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-16T16:01:39.209 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-16T16:01:39.209 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-16T16:01:39.209 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-16T16:01:39.209 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-16T16:01:39.209 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-16T16:01:39.209 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-16T16:01:39.209 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-16T16:01:39.209 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-16T16:01:39.209 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-16T16:01:39.209 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-16T16:01:39.210 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-16T16:01:39.210 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-16T16:01:39.210 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-16T16:01:39.210 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-16T16:01:39.210 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-16T16:01:39.210 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-16T16:01:39.210 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-16T16:01:39.210 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-16T16:01:39.210 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-16T16:01:39.210 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-16T16:01:39.210 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-16T16:01:39.210 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-16T16:01:39.211 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-16T16:01:39.211 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-16T16:01:39.211 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-16T16:01:39.211 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-16T16:01:39.211 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-16T16:01:39.211 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-16T16:01:39.211 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-16T16:01:39.211 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-16T16:01:39.211 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-16T16:01:39.211 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-16T16:01:39.211 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-16T16:01:39.211 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-16T16:01:39.211 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-16T16:01:39.211 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-16T16:01:39.212 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-16T16:01:39.212 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-16T16:01:39.212 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-16T16:01:39.212 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-16T16:01:39.212 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-16T16:01:39.212 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers 2024-04-16T16:01:39.212 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi 2024-04-16T16:01:39.212 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-16T16:01:39.212 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-16T16:01:39.212 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-16T16:01:39.212 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-16T16:01:39.212 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-16T16:01:39.212 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-16T16:01:39.213 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-16T16:01:39.213 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata 2024-04-16T16:01:39.213 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-16T16:01:39.213 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-16T16:01:39.213 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ata_generic.ko 2024-04-16T16:01:39.213 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-16T16:01:39.213 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-16T16:01:39.213 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-16T16:01:39.213 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-16T16:01:39.213 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-16T16:01:39.213 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-16T16:01:39.213 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-16T16:01:39.213 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-16T16:01:39.214 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-16T16:01:39.214 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-16T16:01:39.214 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-16T16:01:39.214 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-16T16:01:39.214 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-16T16:01:39.214 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-16T16:01:39.214 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-16T16:01:39.214 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-16T16:01:39.214 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-16T16:01:39.214 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-16T16:01:39.214 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-16T16:01:39.214 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-16T16:01:39.214 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-16T16:01:39.214 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-16T16:01:39.215 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-16T16:01:39.215 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-16T16:01:39.215 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-16T16:01:39.215 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-16T16:01:39.215 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-16T16:01:39.215 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-16T16:01:39.215 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-16T16:01:39.215 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-16T16:01:39.215 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-16T16:01:39.215 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-16T16:01:39.215 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-16T16:01:39.215 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-16T16:01:39.215 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-16T16:01:39.216 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sis.ko 2024-04-16T16:01:39.216 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-16T16:01:39.216 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-16T16:01:39.216 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-16T16:01:39.216 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-16T16:01:39.216 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-16T16:01:39.216 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-16T16:01:39.216 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-16T16:01:39.216 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-16T16:01:39.216 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-16T16:01:39.216 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-16T16:01:39.216 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-16T16:01:39.216 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-16T16:01:39.217 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-16T16:01:39.217 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-16T16:01:39.217 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-16T16:01:39.217 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-16T16:01:39.217 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-16T16:01:39.217 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay 2024-04-16T16:01:39.217 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-16T16:01:39.217 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base 2024-04-16T16:01:39.217 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap 2024-04-16T16:01:39.217 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-16T16:01:39.217 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-16T16:01:39.217 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma 2024-04-16T16:01:39.217 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-16T16:01:39.218 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block 2024-04-16T16:01:39.218 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe 2024-04-16T16:01:39.218 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-16T16:01:39.218 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/brd.ko 2024-04-16T16:01:39.218 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd 2024-04-16T16:01:39.218 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-16T16:01:39.218 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-16T16:01:39.218 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/loop.ko 2024-04-16T16:01:39.218 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx 2024-04-16T16:01:39.218 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-16T16:01:39.218 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-16T16:01:39.218 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk 2024-04-16T16:01:39.218 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-16T16:01:39.219 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-16T16:01:39.219 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-16T16:01:39.219 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/virtio_blk.ko 2024-04-16T16:01:39.219 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback 2024-04-16T16:01:39.219 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-16T16:01:39.219 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkfront.ko 2024-04-16T16:01:39.219 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram 2024-04-16T16:01:39.219 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-16T16:01:39.219 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char 2024-04-16T16:01:39.219 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-16T16:01:39.219 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random 2024-04-16T16:01:39.219 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-16T16:01:39.219 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-16T16:01:39.220 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-16T16:01:39.220 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-16T16:01:39.220 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-16T16:01:39.220 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi 2024-04-16T16:01:39.220 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-16T16:01:39.220 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-16T16:01:39.220 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-16T16:01:39.220 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-16T16:01:39.220 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-16T16:01:39.220 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-16T16:01:39.220 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-16T16:01:39.220 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave 2024-04-16T16:01:39.221 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-16T16:01:39.221 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-16T16:01:39.221 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-16T16:01:39.221 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm 2024-04-16T16:01:39.221 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm.ko 2024-04-16T16:01:39.221 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-16T16:01:39.221 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-16T16:01:39.221 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-16T16:01:39.221 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-16T16:01:39.221 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-16T16:01:39.221 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/uv_mmtimer.ko 2024-04-16T16:01:39.221 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/virtio_console.ko 2024-04-16T16:01:39.221 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq 2024-04-16T16:01:39.222 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-16T16:01:39.222 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-16T16:01:39.222 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-16T16:01:39.222 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-16T16:01:39.222 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-16T16:01:39.222 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-16T16:01:39.222 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto 2024-04-16T16:01:39.222 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp 2024-04-16T16:01:39.222 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-16T16:01:39.222 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-16T16:01:39.222 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel 2024-04-16T16:01:39.222 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat 2024-04-16T16:01:39.222 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common 2024-04-16T16:01:39.223 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-16T16:01:39.223 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-16T16:01:39.223 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-16T16:01:39.223 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-16T16:01:39.223 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-16T16:01:39.223 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio 2024-04-16T16:01:39.223 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-16T16:01:39.223 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca 2024-04-16T16:01:39.223 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-16T16:01:39.223 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq 2024-04-16T16:01:39.223 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-16T16:01:39.223 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma 2024-04-16T16:01:39.223 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw 2024-04-16T16:01:39.223 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-16T16:01:39.223 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat 2024-04-16T16:01:39.224 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-16T16:01:39.224 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac 2024-04-16T16:01:39.224 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-16T16:01:39.224 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-16T16:01:39.224 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-16T16:01:39.224 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-16T16:01:39.224 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-16T16:01:39.224 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-16T16:01:39.224 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-16T16:01:39.224 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-16T16:01:39.224 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-16T16:01:39.224 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-16T16:01:39.224 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-16T16:01:39.224 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-16T16:01:39.225 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-16T16:01:39.225 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware 2024-04-16T16:01:39.225 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/edd.ko 2024-04-16T16:01:39.225 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-16T16:01:39.225 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio 2024-04-16T16:01:39.225 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-16T16:01:39.225 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-16T16:01:39.225 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid 2024-04-16T16:01:39.225 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-16T16:01:39.225 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-16T16:01:39.225 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-16T16:01:39.225 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-16T16:01:39.225 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-16T16:01:39.225 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-16T16:01:39.226 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-16T16:01:39.226 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-16T16:01:39.226 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-16T16:01:39.226 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-16T16:01:39.226 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-16T16:01:39.226 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-16T16:01:39.226 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-16T16:01:39.226 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-16T16:01:39.226 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-16T16:01:39.226 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-16T16:01:39.226 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-16T16:01:39.226 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-16T16:01:39.227 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-16T16:01:39.227 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-16T16:01:39.227 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-16T16:01:39.227 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-16T16:01:39.227 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-16T16:01:39.227 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-16T16:01:39.227 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-16T16:01:39.227 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-16T16:01:39.227 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-16T16:01:39.227 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-16T16:01:39.227 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-16T16:01:39.227 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-16T16:01:39.227 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-16T16:01:39.227 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-16T16:01:39.228 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-16T16:01:39.228 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-16T16:01:39.228 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-16T16:01:39.228 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-16T16:01:39.228 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-16T16:01:39.228 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-16T16:01:39.228 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-16T16:01:39.228 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-16T16:01:39.228 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-16T16:01:39.228 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-16T16:01:39.228 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-16T16:01:39.228 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-16T16:01:39.228 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-16T16:01:39.229 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-16T16:01:39.229 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-16T16:01:39.229 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-16T16:01:39.229 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-16T16:01:39.229 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-16T16:01:39.229 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-16T16:01:39.229 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-16T16:01:39.229 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-16T16:01:39.229 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-16T16:01:39.229 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-16T16:01:39.229 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-16T16:01:39.229 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-16T16:01:39.229 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-16T16:01:39.230 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-16T16:01:39.230 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-16T16:01:39.230 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-16T16:01:39.230 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-16T16:01:39.230 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-16T16:01:39.230 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-16T16:01:39.230 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv 2024-04-16T16:01:39.230 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-16T16:01:39.230 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-16T16:01:39.230 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-16T16:01:39.230 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon 2024-04-16T16:01:39.230 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-16T16:01:39.230 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-16T16:01:39.231 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-16T16:01:39.231 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-16T16:01:39.231 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-16T16:01:39.231 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-16T16:01:39.231 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-16T16:01:39.231 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-16T16:01:39.231 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-16T16:01:39.231 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-16T16:01:39.231 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-16T16:01:39.231 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-16T16:01:39.231 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-16T16:01:39.231 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-16T16:01:39.231 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-16T16:01:39.232 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-16T16:01:39.232 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-16T16:01:39.232 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-16T16:01:39.232 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-16T16:01:39.232 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-16T16:01:39.232 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-16T16:01:39.232 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-16T16:01:39.232 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-16T16:01:39.232 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-16T16:01:39.232 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-16T16:01:39.232 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-16T16:01:39.232 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-16T16:01:39.232 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-16T16:01:39.233 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-16T16:01:39.233 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-16T16:01:39.233 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-16T16:01:39.233 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-16T16:01:39.233 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-16T16:01:39.233 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-16T16:01:39.233 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-16T16:01:39.233 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-16T16:01:39.233 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-16T16:01:39.233 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-16T16:01:39.233 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-16T16:01:39.233 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-16T16:01:39.233 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-16T16:01:39.233 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-16T16:01:39.234 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-16T16:01:39.234 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-16T16:01:39.234 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-16T16:01:39.234 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-16T16:01:39.234 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-16T16:01:39.234 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-16T16:01:39.234 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-16T16:01:39.234 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-16T16:01:39.234 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-16T16:01:39.234 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-16T16:01:39.234 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-16T16:01:39.234 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-16T16:01:39.235 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-16T16:01:39.235 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-16T16:01:39.235 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-16T16:01:39.235 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-16T16:01:39.235 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-16T16:01:39.235 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-16T16:01:39.235 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-16T16:01:39.235 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-16T16:01:39.235 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-16T16:01:39.235 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-16T16:01:39.235 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-16T16:01:39.235 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-16T16:01:39.235 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-16T16:01:39.235 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-16T16:01:39.236 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-16T16:01:39.236 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-16T16:01:39.236 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-16T16:01:39.236 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-16T16:01:39.236 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-16T16:01:39.236 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-16T16:01:39.236 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-16T16:01:39.236 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-16T16:01:39.236 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-16T16:01:39.236 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-16T16:01:39.236 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-16T16:01:39.236 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-16T16:01:39.237 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-16T16:01:39.237 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-16T16:01:39.237 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-16T16:01:39.237 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-16T16:01:39.237 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-16T16:01:39.237 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-16T16:01:39.237 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-16T16:01:39.237 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-16T16:01:39.237 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus 2024-04-16T16:01:39.237 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-16T16:01:39.237 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-16T16:01:39.237 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-16T16:01:39.237 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-16T16:01:39.238 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-16T16:01:39.238 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-16T16:01:39.238 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-16T16:01:39.238 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-16T16:01:39.238 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-16T16:01:39.238 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-16T16:01:39.238 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-16T16:01:39.238 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-16T16:01:39.238 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-16T16:01:39.238 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-16T16:01:39.238 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-16T16:01:39.238 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-16T16:01:39.238 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-16T16:01:39.239 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-16T16:01:39.239 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-16T16:01:39.239 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-16T16:01:39.239 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-16T16:01:39.239 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-16T16:01:39.239 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-16T16:01:39.239 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-16T16:01:39.239 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-16T16:01:39.239 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-16T16:01:39.239 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-16T16:01:39.239 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-16T16:01:39.239 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-16T16:01:39.239 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-16T16:01:39.239 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-16T16:01:39.240 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-16T16:01:39.240 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-16T16:01:39.240 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-16T16:01:39.240 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-16T16:01:39.240 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-16T16:01:39.240 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-16T16:01:39.240 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-16T16:01:39.240 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-16T16:01:39.240 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-16T16:01:39.240 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-16T16:01:39.240 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c 2024-04-16T16:01:39.240 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos 2024-04-16T16:01:39.240 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-16T16:01:39.240 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-16T16:01:39.241 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses 2024-04-16T16:01:39.241 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-16T16:01:39.241 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-16T16:01:39.241 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-16T16:01:39.241 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-16T16:01:39.241 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-16T16:01:39.241 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-16T16:01:39.241 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-16T16:01:39.241 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-16T16:01:39.241 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-16T16:01:39.241 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-16T16:01:39.241 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-16T16:01:39.241 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-16T16:01:39.242 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-16T16:01:39.242 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-16T16:01:39.242 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-16T16:01:39.242 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-16T16:01:39.242 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-16T16:01:39.242 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-16T16:01:39.242 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-16T16:01:39.242 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-16T16:01:39.242 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-16T16:01:39.242 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-16T16:01:39.242 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-16T16:01:39.242 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-dev.ko 2024-04-16T16:01:39.243 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-16T16:01:39.243 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-16T16:01:39.243 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-16T16:01:39.243 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-16T16:01:39.243 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband 2024-04-16T16:01:39.243 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core 2024-04-16T16:01:39.243 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-16T16:01:39.243 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-16T16:01:39.243 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-16T16:01:39.243 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-16T16:01:39.243 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-16T16:01:39.243 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-16T16:01:39.243 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-16T16:01:39.244 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw 2024-04-16T16:01:39.244 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4 2024-04-16T16:01:39.244 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-16T16:01:39.244 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4 2024-04-16T16:01:39.244 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-16T16:01:39.244 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5 2024-04-16T16:01:39.244 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-16T16:01:39.244 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca 2024-04-16T16:01:39.244 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-16T16:01:39.244 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp 2024-04-16T16:01:39.244 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib 2024-04-16T16:01:39.244 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-16T16:01:39.244 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser 2024-04-16T16:01:39.244 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-16T16:01:39.245 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert 2024-04-16T16:01:39.245 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-16T16:01:39.245 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp 2024-04-16T16:01:39.245 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-16T16:01:39.245 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt 2024-04-16T16:01:39.245 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-16T16:01:39.245 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input 2024-04-16T16:01:39.245 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport 2024-04-16T16:01:39.245 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-16T16:01:39.245 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-16T16:01:39.245 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-16T16:01:39.245 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-16T16:01:39.245 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard 2024-04-16T16:01:39.246 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-16T16:01:39.246 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse 2024-04-16T16:01:39.246 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-16T16:01:39.246 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-16T16:01:39.246 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-16T16:01:39.246 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-16T16:01:39.246 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-16T16:01:39.246 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-16T16:01:39.246 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-16T16:01:39.246 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-16T16:01:39.246 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4 2024-04-16T16:01:39.246 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-16T16:01:39.246 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio 2024-04-16T16:01:39.247 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-16T16:01:39.247 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-16T16:01:39.247 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-16T16:01:39.247 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-16T16:01:39.247 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-16T16:01:39.247 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds 2024-04-16T16:01:39.247 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-16T16:01:39.247 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-16T16:01:39.247 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-16T16:01:39.247 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-16T16:01:39.247 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger 2024-04-16T16:01:39.247 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-16T16:01:39.247 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-16T16:01:39.248 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-16T16:01:39.248 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-16T16:01:39.248 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-16T16:01:39.248 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-16T16:01:39.248 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-16T16:01:39.248 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-16T16:01:39.248 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-16T16:01:39.248 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md 2024-04-16T16:01:39.248 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache 2024-04-16T16:01:39.248 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-16T16:01:39.248 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-16T16:01:39.248 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-16T16:01:39.248 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-16T16:01:39.248 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-16T16:01:39.249 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-16T16:01:39.249 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-16T16:01:39.249 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-16T16:01:39.249 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-16T16:01:39.249 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-16T16:01:39.249 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-16T16:01:39.249 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-16T16:01:39.249 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-16T16:01:39.249 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-16T16:01:39.249 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-16T16:01:39.249 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-16T16:01:39.249 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data 2024-04-16T16:01:39.249 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-16T16:01:39.249 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-16T16:01:39.250 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-16T16:01:39.250 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-16T16:01:39.250 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-16T16:01:39.250 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message 2024-04-16T16:01:39.250 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion 2024-04-16T16:01:39.250 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-16T16:01:39.250 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-16T16:01:39.250 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-16T16:01:39.250 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-16T16:01:39.250 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-16T16:01:39.250 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-16T16:01:39.250 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-16T16:01:39.250 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd 2024-04-16T16:01:39.250 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-16T16:01:39.251 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-16T16:01:39.251 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mfd-core.ko 2024-04-16T16:01:39.251 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-16T16:01:39.251 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-16T16:01:39.251 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-16T16:01:39.251 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-16T16:01:39.251 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc 2024-04-16T16:01:39.251 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl 2024-04-16T16:01:39.251 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-16T16:01:39.251 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-16T16:01:39.251 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-16T16:01:39.251 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-16T16:01:39.251 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710 2024-04-16T16:01:39.251 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-16T16:01:39.252 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom 2024-04-16T16:01:39.252 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-16T16:01:39.252 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-16T16:01:39.252 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-16T16:01:39.252 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-16T16:01:39.252 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-16T16:01:39.252 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-16T16:01:39.252 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-16T16:01:39.252 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d 2024-04-16T16:01:39.252 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-16T16:01:39.252 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-16T16:01:39.252 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei 2024-04-16T16:01:39.252 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-16T16:01:39.252 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-16T16:01:39.253 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-16T16:01:39.253 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru 2024-04-16T16:01:39.253 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-16T16:01:39.253 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp 2024-04-16T16:01:39.253 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-16T16:01:39.253 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-16T16:01:39.253 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-16T16:01:39.253 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-16T16:01:39.253 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-16T16:01:39.253 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-16T16:01:39.253 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-16T16:01:39.253 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci 2024-04-16T16:01:39.253 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-16T16:01:39.253 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net 2024-04-16T16:01:39.253 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding 2024-04-16T16:01:39.254 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-16T16:01:39.254 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa 2024-04-16T16:01:39.254 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53 2024-04-16T16:01:39.254 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-16T16:01:39.254 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-16T16:01:39.254 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-16T16:01:39.254 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx 2024-04-16T16:01:39.254 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-16T16:01:39.254 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-16T16:01:39.254 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-16T16:01:39.254 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet 2024-04-16T16:01:39.254 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com 2024-04-16T16:01:39.255 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-16T16:01:39.255 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-16T16:01:39.255 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-16T16:01:39.255 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-16T16:01:39.255 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390 2024-04-16T16:01:39.255 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-16T16:01:39.255 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-16T16:01:39.255 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-16T16:01:39.255 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-16T16:01:39.255 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec 2024-04-16T16:01:39.255 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-16T16:01:39.255 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere 2024-04-16T16:01:39.255 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-16T16:01:39.255 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon 2024-04-16T16:01:39.256 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-16T16:01:39.256 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera 2024-04-16T16:01:39.256 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-16T16:01:39.256 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd 2024-04-16T16:01:39.256 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-16T16:01:39.256 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-16T16:01:39.256 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-16T16:01:39.256 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros 2024-04-16T16:01:39.256 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx 2024-04-16T16:01:39.256 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-16T16:01:39.256 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-16T16:01:39.256 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-16T16:01:39.256 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-16T16:01:39.256 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-16T16:01:39.256 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx 2024-04-16T16:01:39.257 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-16T16:01:39.257 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-16T16:01:39.257 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom 2024-04-16T16:01:39.257 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-16T16:01:39.257 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-16T16:01:39.257 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-16T16:01:39.257 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-16T16:01:39.257 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-16T16:01:39.257 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet 2024-04-16T16:01:39.257 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-16T16:01:39.257 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-16T16:01:39.257 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade 2024-04-16T16:01:39.257 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna 2024-04-16T16:01:39.258 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-16T16:01:39.258 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio 2024-04-16T16:01:39.258 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-16T16:01:39.258 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-16T16:01:39.258 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-16T16:01:39.258 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-16T16:01:39.258 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-16T16:01:39.258 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-16T16:01:39.258 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-16T16:01:39.258 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-16T16:01:39.258 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-16T16:01:39.258 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-16T16:01:39.258 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco 2024-04-16T16:01:39.258 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic 2024-04-16T16:01:39.259 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-16T16:01:39.259 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec 2024-04-16T16:01:39.259 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip 2024-04-16T16:01:39.259 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-16T16:01:39.259 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-16T16:01:39.259 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-16T16:01:39.259 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-16T16:01:39.259 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-16T16:01:39.259 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-16T16:01:39.259 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink 2024-04-16T16:01:39.259 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-16T16:01:39.259 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-16T16:01:39.259 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-16T16:01:39.260 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex 2024-04-16T16:01:39.260 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet 2024-04-16T16:01:39.260 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-16T16:01:39.260 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-16T16:01:39.260 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-16T16:01:39.260 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel 2024-04-16T16:01:39.260 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-16T16:01:39.260 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000 2024-04-16T16:01:39.260 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-16T16:01:39.260 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e 2024-04-16T16:01:39.260 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-16T16:01:39.260 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k 2024-04-16T16:01:39.260 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-16T16:01:39.260 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e 2024-04-16T16:01:39.261 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-16T16:01:39.261 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf 2024-04-16T16:01:39.261 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-16T16:01:39.261 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb 2024-04-16T16:01:39.261 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-16T16:01:39.261 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf 2024-04-16T16:01:39.261 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-16T16:01:39.261 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-16T16:01:39.261 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-16T16:01:39.261 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-16T16:01:39.261 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-16T16:01:39.261 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-16T16:01:39.261 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell 2024-04-16T16:01:39.261 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-16T16:01:39.262 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-16T16:01:39.262 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-16T16:01:39.262 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox 2024-04-16T16:01:39.262 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-16T16:01:39.262 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-16T16:01:39.262 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-16T16:01:39.262 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-16T16:01:39.262 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-16T16:01:39.262 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-16T16:01:39.262 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel 2024-04-16T16:01:39.262 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-16T16:01:39.262 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom 2024-04-16T16:01:39.263 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-16T16:01:39.263 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-16T16:01:39.263 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi 2024-04-16T16:01:39.263 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-16T16:01:39.263 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-16T16:01:39.263 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion 2024-04-16T16:01:39.263 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-16T16:01:39.263 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia 2024-04-16T16:01:39.263 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-16T16:01:39.263 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines 2024-04-16T16:01:39.263 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-16T16:01:39.263 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-16T16:01:39.263 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic 2024-04-16T16:01:39.263 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-16T16:01:39.264 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-16T16:01:39.264 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-16T16:01:39.264 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-16T16:01:39.264 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-16T16:01:39.264 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc 2024-04-16T16:01:39.264 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-16T16:01:39.264 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek 2024-04-16T16:01:39.264 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-16T16:01:39.264 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-16T16:01:39.264 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-16T16:01:39.264 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-16T16:01:39.264 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker 2024-04-16T16:01:39.264 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-16T16:01:39.264 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc 2024-04-16T16:01:39.265 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-16T16:01:39.265 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan 2024-04-16T16:01:39.265 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-16T16:01:39.265 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis 2024-04-16T16:01:39.265 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-16T16:01:39.265 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-16T16:01:39.265 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc 2024-04-16T16:01:39.265 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-16T16:01:39.265 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-16T16:01:39.265 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-16T16:01:39.265 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-16T16:01:39.265 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro 2024-04-16T16:01:39.265 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-16T16:01:39.265 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-16T16:01:39.266 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-16T16:01:39.266 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun 2024-04-16T16:01:39.266 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-16T16:01:39.266 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-16T16:01:39.266 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-16T16:01:39.266 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-16T16:01:39.266 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti 2024-04-16T16:01:39.266 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-16T16:01:39.266 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti 2024-04-16T16:01:39.266 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-16T16:01:39.266 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via 2024-04-16T16:01:39.266 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-16T16:01:39.267 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-16T16:01:39.267 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet 2024-04-16T16:01:39.267 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-16T16:01:39.267 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-16T16:01:39.267 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom 2024-04-16T16:01:39.267 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-16T16:01:39.267 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-16T16:01:39.267 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv 2024-04-16T16:01:39.267 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-16T16:01:39.267 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154 2024-04-16T16:01:39.267 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-16T16:01:39.267 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-16T16:01:39.267 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan 2024-04-16T16:01:39.267 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-16T16:01:39.268 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-16T16:01:39.268 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-16T16:01:39.268 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio 2024-04-16T16:01:39.268 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-16T16:01:39.268 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-16T16:01:39.268 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-16T16:01:39.268 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-16T16:01:39.268 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-16T16:01:39.268 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/net_failover.ko 2024-04-16T16:01:39.268 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-16T16:01:39.268 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-16T16:01:39.268 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs 2024-04-16T16:01:39.268 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-16T16:01:39.268 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-16T16:01:39.269 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy 2024-04-16T16:01:39.269 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-16T16:01:39.269 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-16T16:01:39.269 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-16T16:01:39.269 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-16T16:01:39.269 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-16T16:01:39.269 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-16T16:01:39.269 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-16T16:01:39.269 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-16T16:01:39.269 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/dp83640.ko 2024-04-16T16:01:39.269 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-16T16:01:39.269 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-16T16:01:39.269 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-16T16:01:39.269 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-16T16:01:39.270 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-16T16:01:39.270 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-16T16:01:39.270 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-16T16:01:39.270 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-16T16:01:39.270 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-16T16:01:39.270 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-16T16:01:39.270 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-16T16:01:39.270 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-16T16:01:39.270 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp 2024-04-16T16:01:39.270 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-16T16:01:39.270 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-16T16:01:39.270 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-16T16:01:39.270 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-16T16:01:39.271 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-16T16:01:39.271 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-16T16:01:39.271 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-16T16:01:39.271 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-16T16:01:39.271 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-16T16:01:39.271 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip 2024-04-16T16:01:39.271 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slhc.ko 2024-04-16T16:01:39.271 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-16T16:01:39.271 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-16T16:01:39.271 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-16T16:01:39.271 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team 2024-04-16T16:01:39.271 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-16T16:01:39.271 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-16T16:01:39.272 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-16T16:01:39.272 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-16T16:01:39.272 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-16T16:01:39.272 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-16T16:01:39.272 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tun.ko 2024-04-16T16:01:39.272 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-16T16:01:39.272 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/virtio_net.ko 2024-04-16T16:01:39.272 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3 2024-04-16T16:01:39.272 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-16T16:01:39.272 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan 2024-04-16T16:01:39.272 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-16T16:01:39.272 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback 2024-04-16T16:01:39.272 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-16T16:01:39.273 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netfront.ko 2024-04-16T16:01:39.273 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb 2024-04-16T16:01:39.273 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-16T16:01:39.273 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme 2024-04-16T16:01:39.273 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host 2024-04-16T16:01:39.273 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-16T16:01:39.273 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-16T16:01:39.273 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport 2024-04-16T16:01:39.273 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-16T16:01:39.273 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-16T16:01:39.273 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-16T16:01:39.273 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-16T16:01:39.273 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci 2024-04-16T16:01:39.273 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug 2024-04-16T16:01:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-16T16:01:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie 2024-04-16T16:01:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-16T16:01:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-16T16:01:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia 2024-04-16T16:01:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-16T16:01:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-16T16:01:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-16T16:01:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl 2024-04-16T16:01:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel 2024-04-16T16:01:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-16T16:01:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform 2024-04-16T16:01:39.275 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86 2024-04-16T16:01:39.275 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-16T16:01:39.275 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-16T16:01:39.275 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-16T16:01:39.275 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-16T16:01:39.275 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-16T16:01:39.275 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-16T16:01:39.275 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-16T16:01:39.275 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-16T16:01:39.275 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-16T16:01:39.275 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-16T16:01:39.275 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-16T16:01:39.275 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-16T16:01:39.275 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel 2024-04-16T16:01:39.276 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-16T16:01:39.276 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-16T16:01:39.276 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-16T16:01:39.276 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-16T16:01:39.276 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-16T16:01:39.276 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-16T16:01:39.276 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-16T16:01:39.276 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-16T16:01:39.276 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-16T16:01:39.276 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-16T16:01:39.276 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-16T16:01:39.276 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-16T16:01:39.276 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap 2024-04-16T16:01:39.277 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-16T16:01:39.277 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-16T16:01:39.277 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps 2024-04-16T16:01:39.277 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients 2024-04-16T16:01:39.277 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-16T16:01:39.277 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-16T16:01:39.277 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-16T16:01:39.277 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-16T16:01:39.277 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp 2024-04-16T16:01:39.277 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-16T16:01:39.277 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-16T16:01:39.277 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras 2024-04-16T16:01:39.277 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd 2024-04-16T16:01:39.277 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl 2024-04-16T16:01:39.278 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-16T16:01:39.278 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-16T16:01:39.278 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc 2024-04-16T16:01:39.278 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-16T16:01:39.278 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-16T16:01:39.278 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-16T16:01:39.278 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-16T16:01:39.278 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-16T16:01:39.278 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-16T16:01:39.278 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-16T16:01:39.278 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-16T16:01:39.278 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-16T16:01:39.278 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-16T16:01:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-16T16:01:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-16T16:01:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-16T16:01:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-16T16:01:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-16T16:01:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-16T16:01:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-16T16:01:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-16T16:01:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-16T16:01:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-16T16:01:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-16T16:01:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-16T16:01:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-16T16:01:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-16T16:01:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-16T16:01:39.280 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx 2024-04-16T16:01:39.281 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-16T16:01:39.281 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-16T16:01:39.281 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-16T16:01:39.281 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr 2024-04-16T16:01:39.281 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-16T16:01:39.281 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-16T16:01:39.281 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi 2024-04-16T16:01:39.281 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-16T16:01:39.281 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa 2024-04-16T16:01:39.281 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-16T16:01:39.281 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc 2024-04-16T16:01:39.281 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-16T16:01:39.281 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i 2024-04-16T16:01:39.281 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-16T16:01:39.281 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-16T16:01:39.282 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor 2024-04-16T16:01:39.282 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-16T16:01:39.282 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi 2024-04-16T16:01:39.282 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-16T16:01:39.282 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-16T16:01:39.282 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-16T16:01:39.282 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-16T16:01:39.282 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-16T16:01:39.282 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-16T16:01:39.282 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler 2024-04-16T16:01:39.282 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-16T16:01:39.282 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-16T16:01:39.282 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-16T16:01:39.282 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-16T16:01:39.283 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-16T16:01:39.283 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r 2024-04-16T16:01:39.283 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-16T16:01:39.283 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-16T16:01:39.283 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe 2024-04-16T16:01:39.283 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-16T16:01:39.283 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-16T16:01:39.283 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic 2024-04-16T16:01:39.283 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-16T16:01:39.283 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-16T16:01:39.283 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-16T16:01:39.283 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-16T16:01:39.283 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-16T16:01:39.283 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-16T16:01:39.284 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-16T16:01:39.284 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci 2024-04-16T16:01:39.284 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-16T16:01:39.284 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-16T16:01:39.284 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-16T16:01:39.284 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc 2024-04-16T16:01:39.284 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-16T16:01:39.284 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-16T16:01:39.284 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-16T16:01:39.284 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas 2024-04-16T16:01:39.284 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-16T16:01:39.284 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc 2024-04-16T16:01:39.284 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-16T16:01:39.284 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid 2024-04-16T16:01:39.284 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-16T16:01:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-16T16:01:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-16T16:01:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-16T16:01:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas 2024-04-16T16:01:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-16T16:01:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas 2024-04-16T16:01:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-16T16:01:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-16T16:01:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001 2024-04-16T16:01:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-16T16:01:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-16T16:01:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-16T16:01:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx 2024-04-16T16:01:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-16T16:01:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-16T16:01:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx 2024-04-16T16:01:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-16T16:01:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-16T16:01:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-16T16:01:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-16T16:01:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-16T16:01:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-16T16:01:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-16T16:01:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-16T16:01:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-16T16:01:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-16T16:01:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-16T16:01:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2 2024-04-16T16:01:39.287 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-16T16:01:39.287 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-16T16:01:39.287 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-16T16:01:39.287 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-16T16:01:39.287 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb 2024-04-16T16:01:39.287 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-16T16:01:39.287 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target 2024-04-16T16:01:39.287 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi 2024-04-16T16:01:39.287 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-16T16:01:39.287 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback 2024-04-16T16:01:39.287 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-16T16:01:39.287 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-16T16:01:39.287 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-16T16:01:39.288 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-16T16:01:39.288 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-16T16:01:39.288 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-16T16:01:39.288 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc 2024-04-16T16:01:39.288 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-16T16:01:39.288 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal 2024-04-16T16:01:39.288 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel 2024-04-16T16:01:39.288 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal 2024-04-16T16:01:39.288 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-16T16:01:39.288 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-16T16:01:39.288 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-16T16:01:39.288 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-16T16:01:39.288 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-16T16:01:39.288 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-16T16:01:39.289 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-16T16:01:39.289 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-16T16:01:39.289 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-16T16:01:39.289 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-16T16:01:39.289 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-16T16:01:39.289 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-16T16:01:39.289 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-16T16:01:39.289 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-16T16:01:39.289 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-16T16:01:39.289 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-16T16:01:39.289 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-16T16:01:39.289 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-16T16:01:39.289 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty 2024-04-16T16:01:39.289 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless 2024-04-16T16:01:39.290 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-16T16:01:39.290 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-16T16:01:39.290 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-16T16:01:39.290 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-16T16:01:39.290 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial 2024-04-16T16:01:39.290 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250 2024-04-16T16:01:39.290 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-16T16:01:39.290 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-16T16:01:39.290 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm 2024-04-16T16:01:39.290 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-16T16:01:39.290 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs 2024-04-16T16:01:39.290 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core 2024-04-16T16:01:39.290 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-16T16:01:39.291 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host 2024-04-16T16:01:39.291 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-16T16:01:39.291 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio 2024-04-16T16:01:39.291 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-16T16:01:39.291 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-16T16:01:39.291 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-16T16:01:39.291 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-16T16:01:39.291 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-16T16:01:39.291 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb 2024-04-16T16:01:39.291 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class 2024-04-16T16:01:39.291 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-16T16:01:39.291 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-16T16:01:39.291 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-16T16:01:39.291 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-16T16:01:39.292 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host 2024-04-16T16:01:39.292 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-16T16:01:39.292 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image 2024-04-16T16:01:39.292 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-16T16:01:39.292 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-16T16:01:39.292 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc 2024-04-16T16:01:39.292 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-16T16:01:39.292 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-16T16:01:39.292 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-16T16:01:39.292 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-16T16:01:39.292 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-16T16:01:39.292 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-16T16:01:39.292 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-16T16:01:39.292 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-16T16:01:39.293 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-16T16:01:39.293 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-16T16:01:39.293 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga 2024-04-16T16:01:39.293 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-16T16:01:39.293 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-16T16:01:39.293 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-16T16:01:39.293 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-16T16:01:39.293 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-16T16:01:39.293 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-16T16:01:39.293 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-16T16:01:39.293 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy 2024-04-16T16:01:39.293 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-16T16:01:39.293 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial 2024-04-16T16:01:39.293 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-16T16:01:39.294 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-16T16:01:39.294 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-16T16:01:39.294 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-16T16:01:39.294 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-16T16:01:39.294 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-16T16:01:39.294 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-16T16:01:39.294 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-16T16:01:39.294 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-16T16:01:39.294 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-16T16:01:39.294 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-16T16:01:39.294 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-16T16:01:39.294 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-16T16:01:39.294 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-16T16:01:39.294 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-16T16:01:39.295 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-16T16:01:39.295 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-16T16:01:39.295 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-16T16:01:39.295 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-16T16:01:39.295 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-16T16:01:39.295 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-16T16:01:39.295 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-16T16:01:39.295 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-16T16:01:39.295 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-16T16:01:39.295 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-16T16:01:39.295 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-16T16:01:39.295 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-16T16:01:39.295 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-16T16:01:39.295 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-16T16:01:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-16T16:01:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-16T16:01:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-16T16:01:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-16T16:01:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-16T16:01:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-16T16:01:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-16T16:01:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-16T16:01:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-16T16:01:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-16T16:01:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-16T16:01:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-16T16:01:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-16T16:01:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-16T16:01:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-16T16:01:39.297 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-16T16:01:39.297 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage 2024-04-16T16:01:39.297 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-16T16:01:39.297 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-16T16:01:39.297 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-16T16:01:39.297 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-16T16:01:39.297 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-16T16:01:39.297 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-16T16:01:39.297 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-16T16:01:39.297 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-16T16:01:39.297 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-16T16:01:39.297 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-16T16:01:39.297 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-16T16:01:39.297 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-16T16:01:39.298 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-16T16:01:39.298 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-16T16:01:39.298 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-16T16:01:39.298 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip 2024-04-16T16:01:39.298 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-16T16:01:39.298 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-16T16:01:39.298 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-16T16:01:39.298 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio 2024-04-16T16:01:39.298 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci 2024-04-16T16:01:39.298 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-16T16:01:39.298 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-16T16:01:39.298 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-16T16:01:39.298 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-16T16:01:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost 2024-04-16T16:01:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-16T16:01:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-16T16:01:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-16T16:01:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-16T16:01:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video 2024-04-16T16:01:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight 2024-04-16T16:01:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-16T16:01:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-16T16:01:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-16T16:01:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio 2024-04-16T16:01:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio.ko 2024-04-16T16:01:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_balloon.ko 2024-04-16T16:01:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_mmio.ko 2024-04-16T16:01:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci.ko 2024-04-16T16:01:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-16T16:01:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-16T16:01:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_ring.ko 2024-04-16T16:01:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1 2024-04-16T16:01:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters 2024-04-16T16:01:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-16T16:01:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-16T16:01:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves 2024-04-16T16:01:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-16T16:01:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-16T16:01:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-16T16:01:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-16T16:01:39.301 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-16T16:01:39.301 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-16T16:01:39.301 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-16T16:01:39.301 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-16T16:01:39.301 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-16T16:01:39.301 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-16T16:01:39.301 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-16T16:01:39.301 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-16T16:01:39.301 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog 2024-04-16T16:01:39.301 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-16T16:01:39.301 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-16T16:01:39.301 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-16T16:01:39.301 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-16T16:01:39.302 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-16T16:01:39.302 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-16T16:01:39.302 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-16T16:01:39.302 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-16T16:01:39.302 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-16T16:01:39.302 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-16T16:01:39.302 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-16T16:01:39.302 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-16T16:01:39.302 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-16T16:01:39.302 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-16T16:01:39.302 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-16T16:01:39.302 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-16T16:01:39.302 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-16T16:01:39.302 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-16T16:01:39.303 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-16T16:01:39.303 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-16T16:01:39.303 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-16T16:01:39.303 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-16T16:01:39.303 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-16T16:01:39.303 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-16T16:01:39.303 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-16T16:01:39.303 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-16T16:01:39.303 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen 2024-04-16T16:01:39.303 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-16T16:01:39.303 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-16T16:01:39.303 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-16T16:01:39.303 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-16T16:01:39.303 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback 2024-04-16T16:01:39.304 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-16T16:01:39.304 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-16T16:01:39.304 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs 2024-04-16T16:01:39.304 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-16T16:01:39.304 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs 2024-04-16T16:01:39.304 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p 2024-04-16T16:01:39.304 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-16T16:01:39.304 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs 2024-04-16T16:01:39.304 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-16T16:01:39.304 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs 2024-04-16T16:01:39.304 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-16T16:01:39.304 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-16T16:01:39.304 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs 2024-04-16T16:01:39.304 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-16T16:01:39.304 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles 2024-04-16T16:01:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-16T16:01:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph 2024-04-16T16:01:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-16T16:01:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda 2024-04-16T16:01:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-16T16:01:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs 2024-04-16T16:01:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-16T16:01:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm 2024-04-16T16:01:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-16T16:01:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs 2024-04-16T16:01:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-16T16:01:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs 2024-04-16T16:01:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-16T16:01:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat 2024-04-16T16:01:39.306 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/fat.ko 2024-04-16T16:01:39.306 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-16T16:01:39.306 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/vfat.ko 2024-04-16T16:01:39.306 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse 2024-04-16T16:01:39.306 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-16T16:01:39.306 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/fuse.ko 2024-04-16T16:01:39.306 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2 2024-04-16T16:01:39.306 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-16T16:01:39.306 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs 2024-04-16T16:01:39.306 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-16T16:01:39.306 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus 2024-04-16T16:01:39.306 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-16T16:01:39.306 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs 2024-04-16T16:01:39.306 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-16T16:01:39.307 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs 2024-04-16T16:01:39.307 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-16T16:01:39.307 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd 2024-04-16T16:01:39.307 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-16T16:01:39.307 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix 2024-04-16T16:01:39.307 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-16T16:01:39.307 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs 2024-04-16T16:01:39.307 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-16T16:01:39.307 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs 2024-04-16T16:01:39.307 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout 2024-04-16T16:01:39.307 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-16T16:01:39.307 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout 2024-04-16T16:01:39.307 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-16T16:01:39.308 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout 2024-04-16T16:01:39.308 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-16T16:01:39.308 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-16T16:01:39.308 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-16T16:01:39.308 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-16T16:01:39.308 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common 2024-04-16T16:01:39.308 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-16T16:01:39.308 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-16T16:01:39.308 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd 2024-04-16T16:01:39.308 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-16T16:01:39.308 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2 2024-04-16T16:01:39.308 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-16T16:01:39.308 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls 2024-04-16T16:01:39.309 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-16T16:01:39.309 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-16T16:01:39.309 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-16T16:01:39.309 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-16T16:01:39.309 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-16T16:01:39.309 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-16T16:01:39.309 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-16T16:01:39.309 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-16T16:01:39.309 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-16T16:01:39.309 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-16T16:01:39.309 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-16T16:01:39.309 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-16T16:01:39.309 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-16T16:01:39.310 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-16T16:01:39.310 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-16T16:01:39.310 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-16T16:01:39.310 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-16T16:01:39.310 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-16T16:01:39.310 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-16T16:01:39.310 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-16T16:01:39.310 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-16T16:01:39.310 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-16T16:01:39.310 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-16T16:01:39.310 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-16T16:01:39.310 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-16T16:01:39.310 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-16T16:01:39.310 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-16T16:01:39.311 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-16T16:01:39.311 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-16T16:01:39.311 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-16T16:01:39.311 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-16T16:01:39.311 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-16T16:01:39.311 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-16T16:01:39.311 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-16T16:01:39.311 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-16T16:01:39.311 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-16T16:01:39.311 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-16T16:01:39.311 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-16T16:01:39.311 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-16T16:01:39.311 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-16T16:01:39.311 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-16T16:01:39.312 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-16T16:01:39.312 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-16T16:01:39.312 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-16T16:01:39.312 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-16T16:01:39.312 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-16T16:01:39.312 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-16T16:01:39.312 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-16T16:01:39.312 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-16T16:01:39.312 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-16T16:01:39.312 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2 2024-04-16T16:01:39.312 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster 2024-04-16T16:01:39.312 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-16T16:01:39.312 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm 2024-04-16T16:01:39.312 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-16T16:01:39.313 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs 2024-04-16T16:01:39.313 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-16T16:01:39.313 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-16T16:01:39.313 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-16T16:01:39.313 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-16T16:01:39.313 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-16T16:01:39.313 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs 2024-04-16T16:01:39.313 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-16T16:01:39.313 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore 2024-04-16T16:01:39.313 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-16T16:01:39.313 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs 2024-04-16T16:01:39.313 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-16T16:01:39.313 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs 2024-04-16T16:01:39.314 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-16T16:01:39.314 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb 2024-04-16T16:01:39.314 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client 2024-04-16T16:01:39.314 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-16T16:01:39.314 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common 2024-04-16T16:01:39.314 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-16T16:01:39.314 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-16T16:01:39.314 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs 2024-04-16T16:01:39.314 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-16T16:01:39.314 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv 2024-04-16T16:01:39.314 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-16T16:01:39.314 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf 2024-04-16T16:01:39.314 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-16T16:01:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs 2024-04-16T16:01:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-16T16:01:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs 2024-04-16T16:01:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-16T16:01:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel 2024-04-16T16:01:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu 2024-04-16T16:01:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu/rcutorture.ko 2024-04-16T16:01:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-16T16:01:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace 2024-04-16T16:01:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-16T16:01:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib 2024-04-16T16:01:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/asn1_encoder.ko 2024-04-16T16:01:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-16T16:01:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-16T16:01:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto 2024-04-16T16:01:39.316 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-16T16:01:39.316 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-16T16:01:39.316 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-16T16:01:39.316 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-16T16:01:39.316 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4 2024-04-16T16:01:39.316 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-16T16:01:39.316 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-16T16:01:39.316 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math 2024-04-16T16:01:39.316 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-16T16:01:39.316 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6 2024-04-16T16:01:39.316 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-16T16:01:39.316 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon 2024-04-16T16:01:39.316 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-16T16:01:39.316 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-16T16:01:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-16T16:01:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-16T16:01:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd 2024-04-16T16:01:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd/zstd_compress.ko 2024-04-16T16:01:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm 2024-04-16T16:01:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-16T16:01:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net 2024-04-16T16:01:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan 2024-04-16T16:01:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-16T16:01:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-16T16:01:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-16T16:01:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-16T16:01:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-16T16:01:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-16T16:01:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-16T16:01:39.318 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-16T16:01:39.318 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802 2024-04-16T16:01:39.318 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-16T16:01:39.318 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-16T16:01:39.318 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-16T16:01:39.318 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-16T16:01:39.318 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-16T16:01:39.318 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q 2024-04-16T16:01:39.318 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-16T16:01:39.318 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p 2024-04-16T16:01:39.318 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-16T16:01:39.318 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-16T16:01:39.318 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-16T16:01:39.319 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-16T16:01:39.319 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv 2024-04-16T16:01:39.319 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-16T16:01:39.319 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge 2024-04-16T16:01:39.319 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-16T16:01:39.319 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-16T16:01:39.319 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter 2024-04-16T16:01:39.319 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-16T16:01:39.319 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-16T16:01:39.319 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-16T16:01:39.319 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-16T16:01:39.319 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-16T16:01:39.319 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-16T16:01:39.320 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-16T16:01:39.320 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-16T16:01:39.320 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-16T16:01:39.320 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-16T16:01:39.320 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-16T16:01:39.320 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-16T16:01:39.320 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-16T16:01:39.320 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-16T16:01:39.320 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-16T16:01:39.320 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-16T16:01:39.320 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-16T16:01:39.320 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-16T16:01:39.320 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-16T16:01:39.320 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-16T16:01:39.321 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-16T16:01:39.321 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-16T16:01:39.321 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-16T16:01:39.321 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-16T16:01:39.321 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph 2024-04-16T16:01:39.321 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-16T16:01:39.321 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core 2024-04-16T16:01:39.321 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/failover.ko 2024-04-16T16:01:39.321 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-16T16:01:39.321 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp 2024-04-16T16:01:39.321 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-16T16:01:39.321 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-16T16:01:39.321 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-16T16:01:39.321 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-16T16:01:39.322 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver 2024-04-16T16:01:39.322 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver/dns_resolver.ko 2024-04-16T16:01:39.322 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa 2024-04-16T16:01:39.322 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-16T16:01:39.322 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-16T16:01:39.322 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-16T16:01:39.322 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-16T16:01:39.322 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-16T16:01:39.322 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154 2024-04-16T16:01:39.322 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan 2024-04-16T16:01:39.322 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-16T16:01:39.322 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-16T16:01:39.322 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-16T16:01:39.322 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4 2024-04-16T16:01:39.323 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-16T16:01:39.323 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-16T16:01:39.323 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-16T16:01:39.323 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-16T16:01:39.323 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-16T16:01:39.323 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-16T16:01:39.323 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-16T16:01:39.323 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-16T16:01:39.323 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-16T16:01:39.323 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-16T16:01:39.323 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter 2024-04-16T16:01:39.323 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-16T16:01:39.323 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-16T16:01:39.324 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-16T16:01:39.325 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-16T16:01:39.325 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-16T16:01:39.325 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-16T16:01:39.325 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-16T16:01:39.325 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-16T16:01:39.325 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-16T16:01:39.325 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-16T16:01:39.325 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-16T16:01:39.325 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-16T16:01:39.325 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-16T16:01:39.325 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-16T16:01:39.325 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-16T16:01:39.325 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-16T16:01:39.325 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-16T16:01:39.326 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-16T16:01:39.326 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-16T16:01:39.326 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-16T16:01:39.326 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-16T16:01:39.326 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-16T16:01:39.326 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-16T16:01:39.326 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6 2024-04-16T16:01:39.326 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-16T16:01:39.326 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-16T16:01:39.326 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-16T16:01:39.326 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-16T16:01:39.326 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-16T16:01:39.326 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-16T16:01:39.326 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-16T16:01:39.327 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter 2024-04-16T16:01:39.327 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-16T16:01:39.327 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-16T16:01:39.327 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-16T16:01:39.327 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-16T16:01:39.327 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-16T16:01:39.327 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-16T16:01:39.327 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-16T16:01:39.327 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-16T16:01:39.327 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-16T16:01:39.327 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-16T16:01:39.327 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-16T16:01:39.327 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-16T16:01:39.328 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-16T16:01:39.328 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-16T16:01:39.328 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-16T16:01:39.328 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-16T16:01:39.328 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-16T16:01:39.328 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-16T16:01:39.328 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-16T16:01:39.328 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-16T16:01:39.328 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-16T16:01:39.328 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-16T16:01:39.328 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-16T16:01:39.328 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-16T16:01:39.328 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-16T16:01:39.329 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-16T16:01:39.329 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-16T16:01:39.329 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key 2024-04-16T16:01:39.329 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-16T16:01:39.329 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp 2024-04-16T16:01:39.329 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-16T16:01:39.329 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-16T16:01:39.329 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-16T16:01:39.329 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-16T16:01:39.329 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-16T16:01:39.329 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-16T16:01:39.329 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-16T16:01:39.329 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc 2024-04-16T16:01:39.330 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-16T16:01:39.330 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154 2024-04-16T16:01:39.330 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-16T16:01:39.330 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls 2024-04-16T16:01:39.330 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-16T16:01:39.330 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter 2024-04-16T16:01:39.330 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset 2024-04-16T16:01:39.330 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-16T16:01:39.330 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-16T16:01:39.330 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-16T16:01:39.330 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-16T16:01:39.330 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-16T16:01:39.330 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-16T16:01:39.331 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-16T16:01:39.331 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-16T16:01:39.331 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-16T16:01:39.331 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-16T16:01:39.331 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-16T16:01:39.331 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-16T16:01:39.331 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-16T16:01:39.331 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-16T16:01:39.331 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-16T16:01:39.331 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-16T16:01:39.331 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs 2024-04-16T16:01:39.331 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-16T16:01:39.331 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-16T16:01:39.331 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-16T16:01:39.332 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-16T16:01:39.332 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-16T16:01:39.332 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-16T16:01:39.332 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-16T16:01:39.332 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-16T16:01:39.332 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-16T16:01:39.332 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-16T16:01:39.332 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-16T16:01:39.332 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-16T16:01:39.332 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-16T16:01:39.332 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-16T16:01:39.332 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-16T16:01:39.332 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-16T16:01:39.332 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-16T16:01:39.333 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-16T16:01:39.333 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-16T16:01:39.333 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-16T16:01:39.333 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-16T16:01:39.333 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-16T16:01:39.333 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-16T16:01:39.333 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-16T16:01:39.333 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-16T16:01:39.333 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-16T16:01:39.333 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-16T16:01:39.333 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-16T16:01:39.333 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-16T16:01:39.333 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-16T16:01:39.333 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-16T16:01:39.334 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-16T16:01:39.334 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-16T16:01:39.334 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-16T16:01:39.334 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-16T16:01:39.334 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-16T16:01:39.334 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-16T16:01:39.334 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-16T16:01:39.334 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-16T16:01:39.334 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-16T16:01:39.334 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-16T16:01:39.334 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-16T16:01:39.334 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-16T16:01:39.335 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-16T16:01:39.335 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-16T16:01:39.335 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-16T16:01:39.335 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-16T16:01:39.335 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-16T16:01:39.335 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-16T16:01:39.335 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-16T16:01:39.335 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-16T16:01:39.335 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-16T16:01:39.335 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-16T16:01:39.335 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-16T16:01:39.335 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-16T16:01:39.335 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-16T16:01:39.335 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-16T16:01:39.336 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-16T16:01:39.336 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_quota.ko 2024-04-16T16:01:39.336 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-16T16:01:39.336 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-16T16:01:39.336 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-16T16:01:39.336 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-16T16:01:39.336 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tunnel.ko 2024-04-16T16:01:39.336 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-16T16:01:39.336 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-16T16:01:39.336 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-16T16:01:39.336 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-16T16:01:39.336 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-16T16:01:39.336 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-16T16:01:39.336 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-16T16:01:39.336 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-16T16:01:39.337 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-16T16:01:39.337 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-16T16:01:39.337 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-16T16:01:39.337 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-16T16:01:39.337 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-16T16:01:39.337 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-16T16:01:39.337 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-16T16:01:39.337 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-16T16:01:39.337 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-16T16:01:39.337 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-16T16:01:39.337 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-16T16:01:39.337 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-16T16:01:39.337 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-16T16:01:39.337 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-16T16:01:39.338 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-16T16:01:39.338 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-16T16:01:39.338 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-16T16:01:39.338 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-16T16:01:39.338 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-16T16:01:39.338 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-16T16:01:39.338 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-16T16:01:39.338 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-16T16:01:39.338 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-16T16:01:39.338 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-16T16:01:39.338 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-16T16:01:39.338 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-16T16:01:39.338 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-16T16:01:39.339 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-16T16:01:39.339 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-16T16:01:39.339 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-16T16:01:39.339 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-16T16:01:39.339 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-16T16:01:39.339 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-16T16:01:39.339 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-16T16:01:39.339 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-16T16:01:39.339 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-16T16:01:39.339 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-16T16:01:39.339 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-16T16:01:39.339 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-16T16:01:39.339 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-16T16:01:39.339 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-16T16:01:39.340 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-16T16:01:39.340 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-16T16:01:39.340 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-16T16:01:39.340 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-16T16:01:39.340 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-16T16:01:39.340 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-16T16:01:39.340 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-16T16:01:39.340 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-16T16:01:39.340 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-16T16:01:39.340 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-16T16:01:39.340 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-16T16:01:39.340 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-16T16:01:39.340 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-16T16:01:39.340 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-16T16:01:39.341 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-16T16:01:39.341 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-16T16:01:39.341 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-16T16:01:39.341 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-16T16:01:39.341 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-16T16:01:39.341 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-16T16:01:39.341 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-16T16:01:39.341 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-16T16:01:39.341 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink 2024-04-16T16:01:39.341 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-16T16:01:39.341 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh 2024-04-16T16:01:39.341 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-16T16:01:39.341 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch 2024-04-16T16:01:39.341 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-16T16:01:39.342 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-16T16:01:39.342 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-16T16:01:39.342 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-16T16:01:39.342 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet 2024-04-16T16:01:39.342 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-16T16:01:39.342 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds 2024-04-16T16:01:39.342 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-16T16:01:39.342 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-16T16:01:39.342 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-16T16:01:39.342 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched 2024-04-16T16:01:39.342 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-16T16:01:39.342 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-16T16:01:39.342 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-16T16:01:39.343 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-16T16:01:39.343 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-16T16:01:39.343 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-16T16:01:39.343 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-16T16:01:39.343 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-16T16:01:39.343 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-16T16:01:39.343 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-16T16:01:39.343 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-16T16:01:39.343 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-16T16:01:39.343 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-16T16:01:39.343 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-16T16:01:39.343 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-16T16:01:39.343 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-16T16:01:39.343 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-16T16:01:39.344 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-16T16:01:39.344 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-16T16:01:39.344 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-16T16:01:39.344 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-16T16:01:39.344 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-16T16:01:39.344 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-16T16:01:39.344 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-16T16:01:39.344 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-16T16:01:39.344 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-16T16:01:39.344 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-16T16:01:39.344 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-16T16:01:39.344 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-16T16:01:39.344 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-16T16:01:39.344 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-16T16:01:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-16T16:01:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-16T16:01:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-16T16:01:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-16T16:01:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-16T16:01:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-16T16:01:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-16T16:01:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-16T16:01:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-16T16:01:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-16T16:01:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-16T16:01:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-16T16:01:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp 2024-04-16T16:01:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-16T16:01:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-16T16:01:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc 2024-04-16T16:01:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss 2024-04-16T16:01:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-16T16:01:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-16T16:01:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-16T16:01:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma 2024-04-16T16:01:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-16T16:01:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc 2024-04-16T16:01:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-16T16:01:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-16T16:01:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix 2024-04-16T16:01:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-16T16:01:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock 2024-04-16T16:01:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-16T16:01:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-16T16:01:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-16T16:01:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-16T16:01:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-16T16:01:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm 2024-04-16T16:01:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-16T16:01:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security 2024-04-16T16:01:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys 2024-04-16T16:01:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys 2024-04-16T16:01:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-16T16:01:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys 2024-04-16T16:01:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys/trusted.ko 2024-04-16T16:01:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-16T16:01:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-16T16:01:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-16T16:01:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-16T16:01:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-16T16:01:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-16T16:01:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-16T16:01:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-16T16:01:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-16T16:01:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-16T16:01:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-16T16:01:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-16T16:01:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-16T16:01:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T16:01:39.353 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:39.353 DEBUG:teuthology.orchestra.run.smithi045:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-16T16:01:39.376 INFO:teuthology.orchestra.run.smithi103.stdout:/boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:39.377 INFO:teuthology.orchestra.run.smithi103.stdout:/boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:39.377 INFO:teuthology.orchestra.run.smithi103.stdout:/boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:39.377 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/System.map 2024-04-16T16:01:39.377 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/config 2024-04-16T16:01:39.377 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel 2024-04-16T16:01:39.377 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch 2024-04-16T16:01:39.377 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86 2024-04-16T16:01:39.377 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto 2024-04-16T16:01:39.377 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-16T16:01:39.377 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-16T16:01:39.377 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-16T16:01:39.377 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-16T16:01:39.378 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-16T16:01:39.378 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-16T16:01:39.378 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-16T16:01:39.378 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-16T16:01:39.378 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-16T16:01:39.378 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-16T16:01:39.378 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-16T16:01:39.378 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-16T16:01:39.378 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-16T16:01:39.378 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-16T16:01:39.378 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-16T16:01:39.378 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-16T16:01:39.378 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-16T16:01:39.379 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-16T16:01:39.379 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-16T16:01:39.379 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-16T16:01:39.379 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel 2024-04-16T16:01:39.379 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu 2024-04-16T16:01:39.379 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce 2024-04-16T16:01:39.379 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-16T16:01:39.379 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm 2024-04-16T16:01:39.379 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-16T16:01:39.379 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-16T16:01:39.379 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-16T16:01:39.379 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto 2024-04-16T16:01:39.379 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-16T16:01:39.379 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-16T16:01:39.380 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-16T16:01:39.380 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx 2024-04-16T16:01:39.380 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-16T16:01:39.380 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-16T16:01:39.380 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-16T16:01:39.380 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-16T16:01:39.380 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-16T16:01:39.380 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-16T16:01:39.380 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-16T16:01:39.380 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-16T16:01:39.380 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-16T16:01:39.380 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-16T16:01:39.380 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-16T16:01:39.381 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-16T16:01:39.381 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-16T16:01:39.381 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-16T16:01:39.381 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-16T16:01:39.381 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-16T16:01:39.381 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-16T16:01:39.381 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-16T16:01:39.381 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-16T16:01:39.381 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-16T16:01:39.381 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-16T16:01:39.381 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-16T16:01:39.381 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-16T16:01:39.381 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-16T16:01:39.382 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-16T16:01:39.382 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-16T16:01:39.382 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-16T16:01:39.382 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-16T16:01:39.382 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-16T16:01:39.382 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-16T16:01:39.382 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-16T16:01:39.382 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-16T16:01:39.382 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-16T16:01:39.382 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-16T16:01:39.382 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-16T16:01:39.382 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-16T16:01:39.382 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-16T16:01:39.382 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-16T16:01:39.383 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-16T16:01:39.383 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-16T16:01:39.383 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-16T16:01:39.383 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-16T16:01:39.383 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-16T16:01:39.383 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-16T16:01:39.383 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-16T16:01:39.383 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers 2024-04-16T16:01:39.383 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi 2024-04-16T16:01:39.383 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-16T16:01:39.383 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-16T16:01:39.383 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-16T16:01:39.383 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-16T16:01:39.383 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-16T16:01:39.383 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ata_generic.ko 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-16T16:01:39.384 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-16T16:01:39.385 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-16T16:01:39.385 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-16T16:01:39.385 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-16T16:01:39.385 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-16T16:01:39.385 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-16T16:01:39.385 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-16T16:01:39.385 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-16T16:01:39.385 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-16T16:01:39.385 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-16T16:01:39.385 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-16T16:01:39.385 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-16T16:01:39.385 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-16T16:01:39.385 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-16T16:01:39.385 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-16T16:01:39.385 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-16T16:01:39.386 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-16T16:01:39.386 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-16T16:01:39.386 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-16T16:01:39.386 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-16T16:01:39.386 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-16T16:01:39.386 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-16T16:01:39.386 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-16T16:01:39.386 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-16T16:01:39.386 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-16T16:01:39.386 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sis.ko 2024-04-16T16:01:39.386 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-16T16:01:39.386 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-16T16:01:39.386 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-16T16:01:39.387 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-16T16:01:39.387 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-16T16:01:39.387 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-16T16:01:39.387 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-16T16:01:39.387 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-16T16:01:39.387 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-16T16:01:39.387 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-16T16:01:39.387 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-16T16:01:39.387 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-16T16:01:39.387 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-16T16:01:39.387 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-16T16:01:39.387 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-16T16:01:39.387 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-16T16:01:39.388 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-16T16:01:39.388 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay 2024-04-16T16:01:39.388 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-16T16:01:39.388 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base 2024-04-16T16:01:39.388 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap 2024-04-16T16:01:39.388 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-16T16:01:39.388 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-16T16:01:39.388 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma 2024-04-16T16:01:39.388 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-16T16:01:39.388 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block 2024-04-16T16:01:39.388 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe 2024-04-16T16:01:39.388 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-16T16:01:39.388 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/brd.ko 2024-04-16T16:01:39.388 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd 2024-04-16T16:01:39.389 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-16T16:01:39.389 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-16T16:01:39.389 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/loop.ko 2024-04-16T16:01:39.389 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx 2024-04-16T16:01:39.389 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-16T16:01:39.389 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-16T16:01:39.389 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk 2024-04-16T16:01:39.389 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-16T16:01:39.389 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-16T16:01:39.389 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-16T16:01:39.389 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/virtio_blk.ko 2024-04-16T16:01:39.389 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback 2024-04-16T16:01:39.389 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-16T16:01:39.389 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkfront.ko 2024-04-16T16:01:39.390 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram 2024-04-16T16:01:39.390 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-16T16:01:39.390 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char 2024-04-16T16:01:39.390 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-16T16:01:39.390 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random 2024-04-16T16:01:39.390 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-16T16:01:39.390 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-16T16:01:39.390 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-16T16:01:39.390 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-16T16:01:39.390 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-16T16:01:39.390 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi 2024-04-16T16:01:39.390 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-16T16:01:39.390 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-16T16:01:39.391 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-16T16:01:39.391 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-16T16:01:39.391 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-16T16:01:39.391 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-16T16:01:39.391 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-16T16:01:39.391 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave 2024-04-16T16:01:39.391 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-16T16:01:39.391 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-16T16:01:39.391 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-16T16:01:39.391 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm 2024-04-16T16:01:39.391 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm.ko 2024-04-16T16:01:39.391 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-16T16:01:39.391 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-16T16:01:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-16T16:01:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-16T16:01:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-16T16:01:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/uv_mmtimer.ko 2024-04-16T16:01:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/virtio_console.ko 2024-04-16T16:01:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq 2024-04-16T16:01:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-16T16:01:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-16T16:01:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-16T16:01:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-16T16:01:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-16T16:01:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-16T16:01:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto 2024-04-16T16:01:39.393 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp 2024-04-16T16:01:39.393 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-16T16:01:39.393 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-16T16:01:39.393 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel 2024-04-16T16:01:39.393 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat 2024-04-16T16:01:39.393 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common 2024-04-16T16:01:39.393 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-16T16:01:39.393 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-16T16:01:39.393 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-16T16:01:39.393 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-16T16:01:39.393 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-16T16:01:39.393 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio 2024-04-16T16:01:39.393 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-16T16:01:39.393 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca 2024-04-16T16:01:39.394 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-16T16:01:39.394 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq 2024-04-16T16:01:39.394 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-16T16:01:39.394 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma 2024-04-16T16:01:39.394 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw 2024-04-16T16:01:39.394 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-16T16:01:39.394 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat 2024-04-16T16:01:39.394 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-16T16:01:39.394 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac 2024-04-16T16:01:39.394 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-16T16:01:39.394 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-16T16:01:39.394 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-16T16:01:39.394 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-16T16:01:39.395 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-16T16:01:39.395 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-16T16:01:39.395 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-16T16:01:39.395 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-16T16:01:39.395 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-16T16:01:39.395 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-16T16:01:39.395 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-16T16:01:39.395 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-16T16:01:39.395 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-16T16:01:39.395 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware 2024-04-16T16:01:39.395 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/edd.ko 2024-04-16T16:01:39.395 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-16T16:01:39.395 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio 2024-04-16T16:01:39.395 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-16T16:01:39.395 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-16T16:01:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid 2024-04-16T16:01:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-16T16:01:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-16T16:01:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-16T16:01:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-16T16:01:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-16T16:01:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-16T16:01:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-16T16:01:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-16T16:01:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-16T16:01:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-16T16:01:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-16T16:01:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-16T16:01:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-16T16:01:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-16T16:01:39.397 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-16T16:01:39.397 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-16T16:01:39.397 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-16T16:01:39.397 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-16T16:01:39.397 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-16T16:01:39.397 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-16T16:01:39.397 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-16T16:01:39.397 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-16T16:01:39.397 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-16T16:01:39.397 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-16T16:01:39.397 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-16T16:01:39.397 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-16T16:01:39.398 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-16T16:01:39.398 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-16T16:01:39.398 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-16T16:01:39.398 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-16T16:01:39.398 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-16T16:01:39.398 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-16T16:01:39.398 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-16T16:01:39.398 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-16T16:01:39.398 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-16T16:01:39.398 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-16T16:01:39.398 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-16T16:01:39.398 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-16T16:01:39.398 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-16T16:01:39.398 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-16T16:01:39.399 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-16T16:01:39.399 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-16T16:01:39.399 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-16T16:01:39.399 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-16T16:01:39.399 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-16T16:01:39.399 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-16T16:01:39.399 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-16T16:01:39.399 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-16T16:01:39.399 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-16T16:01:39.399 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-16T16:01:39.399 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-16T16:01:39.399 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-16T16:01:39.399 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-16T16:01:39.399 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-16T16:01:39.400 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-16T16:01:39.400 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-16T16:01:39.400 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-16T16:01:39.400 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-16T16:01:39.400 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-16T16:01:39.400 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-16T16:01:39.400 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-16T16:01:39.400 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-16T16:01:39.400 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-16T16:01:39.400 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-16T16:01:39.400 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv 2024-04-16T16:01:39.400 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-16T16:01:39.400 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-16T16:01:39.400 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-16T16:01:39.401 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon 2024-04-16T16:01:39.401 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-16T16:01:39.401 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-16T16:01:39.401 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-16T16:01:39.401 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-16T16:01:39.401 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-16T16:01:39.401 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-16T16:01:39.401 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-16T16:01:39.401 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-16T16:01:39.401 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-16T16:01:39.401 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-16T16:01:39.401 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-16T16:01:39.401 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-16T16:01:39.401 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-16T16:01:39.402 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-16T16:01:39.402 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-16T16:01:39.402 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-16T16:01:39.402 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-16T16:01:39.402 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-16T16:01:39.402 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-16T16:01:39.402 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-16T16:01:39.402 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-16T16:01:39.402 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-16T16:01:39.402 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-16T16:01:39.402 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-16T16:01:39.402 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-16T16:01:39.402 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-16T16:01:39.403 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-16T16:01:39.403 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-16T16:01:39.403 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-16T16:01:39.403 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-16T16:01:39.403 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-16T16:01:39.403 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-16T16:01:39.403 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-16T16:01:39.403 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-16T16:01:39.403 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-16T16:01:39.403 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-16T16:01:39.403 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-16T16:01:39.403 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-16T16:01:39.403 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-16T16:01:39.403 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-16T16:01:39.405 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-16T16:01:39.405 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-16T16:01:39.405 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-16T16:01:39.405 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-16T16:01:39.405 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-16T16:01:39.406 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-16T16:01:39.406 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-16T16:01:39.406 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-16T16:01:39.406 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-16T16:01:39.406 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-16T16:01:39.406 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-16T16:01:39.406 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-16T16:01:39.406 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-16T16:01:39.406 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-16T16:01:39.406 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-16T16:01:39.406 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-16T16:01:39.406 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-16T16:01:39.406 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-16T16:01:39.406 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-16T16:01:39.407 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-16T16:01:39.407 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-16T16:01:39.407 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-16T16:01:39.407 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-16T16:01:39.407 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-16T16:01:39.407 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-16T16:01:39.407 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-16T16:01:39.407 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-16T16:01:39.407 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-16T16:01:39.407 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-16T16:01:39.407 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-16T16:01:39.407 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-16T16:01:39.407 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-16T16:01:39.407 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-16T16:01:39.407 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-16T16:01:39.408 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-16T16:01:39.408 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-16T16:01:39.408 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-16T16:01:39.408 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-16T16:01:39.408 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-16T16:01:39.408 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-16T16:01:39.408 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-16T16:01:39.408 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-16T16:01:39.408 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-16T16:01:39.408 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-16T16:01:39.408 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-16T16:01:39.408 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-16T16:01:39.408 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-16T16:01:39.408 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-16T16:01:39.409 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus 2024-04-16T16:01:39.409 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-16T16:01:39.409 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-16T16:01:39.409 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-16T16:01:39.409 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-16T16:01:39.409 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-16T16:01:39.409 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-16T16:01:39.409 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-16T16:01:39.409 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-16T16:01:39.409 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-16T16:01:39.409 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-16T16:01:39.409 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-16T16:01:39.409 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-16T16:01:39.409 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-16T16:01:39.409 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-16T16:01:39.410 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-16T16:01:39.410 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-16T16:01:39.410 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-16T16:01:39.410 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-16T16:01:39.410 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-16T16:01:39.410 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-16T16:01:39.410 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-16T16:01:39.410 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-16T16:01:39.410 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-16T16:01:39.410 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-16T16:01:39.410 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-16T16:01:39.410 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-16T16:01:39.410 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-16T16:01:39.411 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-16T16:01:39.411 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-16T16:01:39.411 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-16T16:01:39.411 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-16T16:01:39.411 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-16T16:01:39.411 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-16T16:01:39.411 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-16T16:01:39.411 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-16T16:01:39.411 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-16T16:01:39.411 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-16T16:01:39.411 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-16T16:01:39.411 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-16T16:01:39.411 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-16T16:01:39.412 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-16T16:01:39.412 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c 2024-04-16T16:01:39.412 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos 2024-04-16T16:01:39.412 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-16T16:01:39.412 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-16T16:01:39.412 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses 2024-04-16T16:01:39.412 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-16T16:01:39.412 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-16T16:01:39.412 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-16T16:01:39.412 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-16T16:01:39.412 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-16T16:01:39.412 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-16T16:01:39.412 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-16T16:01:39.413 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-16T16:01:39.413 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-16T16:01:39.413 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-16T16:01:39.413 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-16T16:01:39.413 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-16T16:01:39.413 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-16T16:01:39.413 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-16T16:01:39.413 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-16T16:01:39.413 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-16T16:01:39.413 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-16T16:01:39.413 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-16T16:01:39.413 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-16T16:01:39.413 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-16T16:01:39.414 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-16T16:01:39.414 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-16T16:01:39.414 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-16T16:01:39.414 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-dev.ko 2024-04-16T16:01:39.414 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-16T16:01:39.414 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-16T16:01:39.414 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-16T16:01:39.414 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-16T16:01:39.414 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband 2024-04-16T16:01:39.414 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core 2024-04-16T16:01:39.414 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-16T16:01:39.414 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-16T16:01:39.414 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-16T16:01:39.414 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-16T16:01:39.415 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-16T16:01:39.415 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-16T16:01:39.415 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-16T16:01:39.415 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw 2024-04-16T16:01:39.415 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4 2024-04-16T16:01:39.415 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-16T16:01:39.415 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4 2024-04-16T16:01:39.415 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-16T16:01:39.415 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5 2024-04-16T16:01:39.415 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-16T16:01:39.415 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca 2024-04-16T16:01:39.415 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-16T16:01:39.415 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp 2024-04-16T16:01:39.416 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib 2024-04-16T16:01:39.416 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-16T16:01:39.416 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser 2024-04-16T16:01:39.416 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-16T16:01:39.416 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert 2024-04-16T16:01:39.416 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-16T16:01:39.416 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp 2024-04-16T16:01:39.416 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-16T16:01:39.416 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt 2024-04-16T16:01:39.416 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-16T16:01:39.416 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input 2024-04-16T16:01:39.416 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport 2024-04-16T16:01:39.416 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-16T16:01:39.416 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-16T16:01:39.417 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-16T16:01:39.417 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-16T16:01:39.417 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard 2024-04-16T16:01:39.417 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-16T16:01:39.417 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse 2024-04-16T16:01:39.417 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-16T16:01:39.417 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-16T16:01:39.417 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-16T16:01:39.417 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-16T16:01:39.417 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-16T16:01:39.417 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-16T16:01:39.417 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-16T16:01:39.417 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-16T16:01:39.418 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4 2024-04-16T16:01:39.418 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-16T16:01:39.418 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio 2024-04-16T16:01:39.418 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-16T16:01:39.418 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-16T16:01:39.418 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-16T16:01:39.418 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-16T16:01:39.418 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-16T16:01:39.418 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds 2024-04-16T16:01:39.418 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-16T16:01:39.418 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-16T16:01:39.418 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-16T16:01:39.418 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-16T16:01:39.418 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger 2024-04-16T16:01:39.419 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-16T16:01:39.419 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-16T16:01:39.419 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-16T16:01:39.419 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-16T16:01:39.419 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-16T16:01:39.419 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-16T16:01:39.419 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-16T16:01:39.419 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-16T16:01:39.419 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-16T16:01:39.419 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md 2024-04-16T16:01:39.419 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache 2024-04-16T16:01:39.419 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-16T16:01:39.419 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-16T16:01:39.419 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-16T16:01:39.420 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-16T16:01:39.420 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-16T16:01:39.420 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-16T16:01:39.420 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-16T16:01:39.420 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-16T16:01:39.420 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-16T16:01:39.420 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-16T16:01:39.420 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-16T16:01:39.420 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-16T16:01:39.420 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-16T16:01:39.420 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-16T16:01:39.420 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-16T16:01:39.420 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-16T16:01:39.420 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data 2024-04-16T16:01:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-16T16:01:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-16T16:01:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-16T16:01:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-16T16:01:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-16T16:01:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message 2024-04-16T16:01:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion 2024-04-16T16:01:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-16T16:01:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-16T16:01:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-16T16:01:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-16T16:01:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-16T16:01:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-16T16:01:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-16T16:01:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd 2024-04-16T16:01:39.422 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-16T16:01:39.422 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-16T16:01:39.422 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mfd-core.ko 2024-04-16T16:01:39.422 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-16T16:01:39.422 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-16T16:01:39.422 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-16T16:01:39.422 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-16T16:01:39.422 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc 2024-04-16T16:01:39.422 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl 2024-04-16T16:01:39.422 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-16T16:01:39.422 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-16T16:01:39.422 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-16T16:01:39.422 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-16T16:01:39.422 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710 2024-04-16T16:01:39.423 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-16T16:01:39.423 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom 2024-04-16T16:01:39.423 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-16T16:01:39.423 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-16T16:01:39.423 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-16T16:01:39.423 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-16T16:01:39.423 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-16T16:01:39.423 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-16T16:01:39.423 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-16T16:01:39.423 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d 2024-04-16T16:01:39.423 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-16T16:01:39.423 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-16T16:01:39.423 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-16T16:01:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net 2024-04-16T16:01:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding 2024-04-16T16:01:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-16T16:01:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa 2024-04-16T16:01:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53 2024-04-16T16:01:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-16T16:01:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-16T16:01:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-16T16:01:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx 2024-04-16T16:01:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-16T16:01:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-16T16:01:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-16T16:01:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet 2024-04-16T16:01:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com 2024-04-16T16:01:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-16T16:01:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-16T16:01:39.426 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-16T16:01:39.426 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-16T16:01:39.426 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390 2024-04-16T16:01:39.426 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-16T16:01:39.426 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-16T16:01:39.426 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-16T16:01:39.426 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-16T16:01:39.426 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec 2024-04-16T16:01:39.426 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-16T16:01:39.426 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere 2024-04-16T16:01:39.426 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-16T16:01:39.426 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon 2024-04-16T16:01:39.426 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-16T16:01:39.426 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera 2024-04-16T16:01:39.426 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-16T16:01:39.427 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd 2024-04-16T16:01:39.427 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-16T16:01:39.427 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-16T16:01:39.427 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-16T16:01:39.427 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros 2024-04-16T16:01:39.427 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx 2024-04-16T16:01:39.427 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-16T16:01:39.427 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-16T16:01:39.427 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-16T16:01:39.427 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-16T16:01:39.427 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-16T16:01:39.427 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx 2024-04-16T16:01:39.427 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-16T16:01:39.428 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-16T16:01:39.428 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom 2024-04-16T16:01:39.428 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-16T16:01:39.428 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-16T16:01:39.428 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-16T16:01:39.428 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-16T16:01:39.428 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-16T16:01:39.428 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet 2024-04-16T16:01:39.428 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-16T16:01:39.428 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-16T16:01:39.428 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade 2024-04-16T16:01:39.428 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna 2024-04-16T16:01:39.428 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-16T16:01:39.428 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio 2024-04-16T16:01:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-16T16:01:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-16T16:01:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-16T16:01:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-16T16:01:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-16T16:01:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-16T16:01:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-16T16:01:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-16T16:01:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-16T16:01:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-16T16:01:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco 2024-04-16T16:01:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic 2024-04-16T16:01:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-16T16:01:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec 2024-04-16T16:01:39.430 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip 2024-04-16T16:01:39.430 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-16T16:01:39.430 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-16T16:01:39.430 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-16T16:01:39.430 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-16T16:01:39.430 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-16T16:01:39.430 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-16T16:01:39.430 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink 2024-04-16T16:01:39.430 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-16T16:01:39.430 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-16T16:01:39.430 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-16T16:01:39.430 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex 2024-04-16T16:01:39.430 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet 2024-04-16T16:01:39.431 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-16T16:01:39.431 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-16T16:01:39.431 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-16T16:01:39.431 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel 2024-04-16T16:01:39.431 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-16T16:01:39.431 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000 2024-04-16T16:01:39.431 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-16T16:01:39.431 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e 2024-04-16T16:01:39.431 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-16T16:01:39.431 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k 2024-04-16T16:01:39.431 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-16T16:01:39.431 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e 2024-04-16T16:01:39.431 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-16T16:01:39.431 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf 2024-04-16T16:01:39.432 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-16T16:01:39.432 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb 2024-04-16T16:01:39.432 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-16T16:01:39.432 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf 2024-04-16T16:01:39.432 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-16T16:01:39.432 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-16T16:01:39.432 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-16T16:01:39.432 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-16T16:01:39.432 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-16T16:01:39.432 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-16T16:01:39.432 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell 2024-04-16T16:01:39.432 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-16T16:01:39.432 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-16T16:01:39.433 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-16T16:01:39.433 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox 2024-04-16T16:01:39.433 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-16T16:01:39.433 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-16T16:01:39.433 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-16T16:01:39.433 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-16T16:01:39.433 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-16T16:01:39.433 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-16T16:01:39.433 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel 2024-04-16T16:01:39.433 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-16T16:01:39.433 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom 2024-04-16T16:01:39.433 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-16T16:01:39.433 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-16T16:01:39.433 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi 2024-04-16T16:01:39.433 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-16T16:01:39.434 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-16T16:01:39.434 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion 2024-04-16T16:01:39.434 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-16T16:01:39.434 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia 2024-04-16T16:01:39.434 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-16T16:01:39.434 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines 2024-04-16T16:01:39.434 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-16T16:01:39.434 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-16T16:01:39.434 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic 2024-04-16T16:01:39.434 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-16T16:01:39.434 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-16T16:01:39.434 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-16T16:01:39.434 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-16T16:01:39.435 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-16T16:01:39.435 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc 2024-04-16T16:01:39.435 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-16T16:01:39.435 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek 2024-04-16T16:01:39.435 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-16T16:01:39.435 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-16T16:01:39.435 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-16T16:01:39.435 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-16T16:01:39.435 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker 2024-04-16T16:01:39.435 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-16T16:01:39.435 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc 2024-04-16T16:01:39.435 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-16T16:01:39.435 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan 2024-04-16T16:01:39.435 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-16T16:01:39.436 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis 2024-04-16T16:01:39.436 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-16T16:01:39.436 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-16T16:01:39.436 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc 2024-04-16T16:01:39.436 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-16T16:01:39.436 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-16T16:01:39.436 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-16T16:01:39.436 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-16T16:01:39.436 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro 2024-04-16T16:01:39.436 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-16T16:01:39.436 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-16T16:01:39.436 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-16T16:01:39.436 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun 2024-04-16T16:01:39.436 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-16T16:01:39.437 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-16T16:01:39.437 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-16T16:01:39.437 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-16T16:01:39.437 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti 2024-04-16T16:01:39.437 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-16T16:01:39.437 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti 2024-04-16T16:01:39.437 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-16T16:01:39.437 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via 2024-04-16T16:01:39.437 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-16T16:01:39.437 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-16T16:01:39.437 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet 2024-04-16T16:01:39.437 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-16T16:01:39.437 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-16T16:01:39.437 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom 2024-04-16T16:01:39.438 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-16T16:01:39.438 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-16T16:01:39.438 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv 2024-04-16T16:01:39.438 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-16T16:01:39.438 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154 2024-04-16T16:01:39.438 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-16T16:01:39.438 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-16T16:01:39.438 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan 2024-04-16T16:01:39.438 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-16T16:01:39.438 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-16T16:01:39.438 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-16T16:01:39.438 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio 2024-04-16T16:01:39.438 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-16T16:01:39.439 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-16T16:01:39.439 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-16T16:01:39.439 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-16T16:01:39.439 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-16T16:01:39.439 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/net_failover.ko 2024-04-16T16:01:39.439 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-16T16:01:39.439 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-16T16:01:39.439 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs 2024-04-16T16:01:39.439 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-16T16:01:39.439 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-16T16:01:39.439 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy 2024-04-16T16:01:39.439 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-16T16:01:39.439 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-16T16:01:39.440 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-16T16:01:39.440 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-16T16:01:39.440 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-16T16:01:39.440 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-16T16:01:39.440 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-16T16:01:39.440 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-16T16:01:39.440 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/dp83640.ko 2024-04-16T16:01:39.440 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-16T16:01:39.440 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-16T16:01:39.440 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-16T16:01:39.440 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-16T16:01:39.440 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-16T16:01:39.440 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-16T16:01:39.440 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-16T16:01:39.441 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-16T16:01:39.441 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-16T16:01:39.441 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-16T16:01:39.441 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-16T16:01:39.441 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-16T16:01:39.441 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp 2024-04-16T16:01:39.441 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-16T16:01:39.441 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-16T16:01:39.441 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-16T16:01:39.441 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-16T16:01:39.441 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-16T16:01:39.441 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-16T16:01:39.441 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-16T16:01:39.441 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-16T16:01:39.442 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-16T16:01:39.442 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip 2024-04-16T16:01:39.442 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slhc.ko 2024-04-16T16:01:39.442 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-16T16:01:39.442 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-16T16:01:39.442 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-16T16:01:39.442 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team 2024-04-16T16:01:39.442 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-16T16:01:39.442 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-16T16:01:39.442 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-16T16:01:39.442 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-16T16:01:39.442 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-16T16:01:39.442 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-16T16:01:39.443 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tun.ko 2024-04-16T16:01:39.443 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-16T16:01:39.443 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/virtio_net.ko 2024-04-16T16:01:39.443 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3 2024-04-16T16:01:39.443 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-16T16:01:39.443 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan 2024-04-16T16:01:39.443 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-16T16:01:39.443 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback 2024-04-16T16:01:39.443 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-16T16:01:39.443 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netfront.ko 2024-04-16T16:01:39.443 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb 2024-04-16T16:01:39.443 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-16T16:01:39.443 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme 2024-04-16T16:01:39.443 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host 2024-04-16T16:01:39.444 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-16T16:01:39.444 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-16T16:01:39.444 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport 2024-04-16T16:01:39.444 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-16T16:01:39.444 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-16T16:01:39.444 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-16T16:01:39.444 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-16T16:01:39.444 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci 2024-04-16T16:01:39.444 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug 2024-04-16T16:01:39.444 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-16T16:01:39.444 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie 2024-04-16T16:01:39.444 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-16T16:01:39.444 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-16T16:01:39.444 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia 2024-04-16T16:01:39.445 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-16T16:01:39.445 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-16T16:01:39.445 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-16T16:01:39.445 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl 2024-04-16T16:01:39.445 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel 2024-04-16T16:01:39.445 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-16T16:01:39.445 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform 2024-04-16T16:01:39.445 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86 2024-04-16T16:01:39.445 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-16T16:01:39.445 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-16T16:01:39.445 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-16T16:01:39.445 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-16T16:01:39.445 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-16T16:01:39.446 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-16T16:01:39.446 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-16T16:01:39.446 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-16T16:01:39.446 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-16T16:01:39.446 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-16T16:01:39.446 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-16T16:01:39.446 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-16T16:01:39.446 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel 2024-04-16T16:01:39.446 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-16T16:01:39.446 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-16T16:01:39.446 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-16T16:01:39.446 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-16T16:01:39.446 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-16T16:01:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-16T16:01:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-16T16:01:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-16T16:01:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-16T16:01:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-16T16:01:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-16T16:01:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-16T16:01:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap 2024-04-16T16:01:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-16T16:01:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-16T16:01:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps 2024-04-16T16:01:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients 2024-04-16T16:01:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-16T16:01:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-16T16:01:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-16T16:01:39.448 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-16T16:01:39.448 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp 2024-04-16T16:01:39.448 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-16T16:01:39.448 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-16T16:01:39.448 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras 2024-04-16T16:01:39.448 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd 2024-04-16T16:01:39.448 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl 2024-04-16T16:01:39.448 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-16T16:01:39.448 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-16T16:01:39.448 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc 2024-04-16T16:01:39.448 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-16T16:01:39.448 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-16T16:01:39.448 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-16T16:01:39.448 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-16T16:01:39.449 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-16T16:01:39.449 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-16T16:01:39.449 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-16T16:01:39.449 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-16T16:01:39.449 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-16T16:01:39.449 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-16T16:01:39.449 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-16T16:01:39.449 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-16T16:01:39.449 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-16T16:01:39.449 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-16T16:01:39.449 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-16T16:01:39.449 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-16T16:01:39.449 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-16T16:01:39.449 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-16T16:01:39.449 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-16T16:01:39.450 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-16T16:01:39.450 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-16T16:01:39.450 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-16T16:01:39.450 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-16T16:01:39.450 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-16T16:01:39.450 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-16T16:01:39.450 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-16T16:01:39.450 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-16T16:01:39.450 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-16T16:01:39.450 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-16T16:01:39.450 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-16T16:01:39.450 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-16T16:01:39.450 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi 2024-04-16T16:01:39.450 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-16T16:01:39.451 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-16T16:01:39.451 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-16T16:01:39.451 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-16T16:01:39.451 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-16T16:01:39.451 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid 2024-04-16T16:01:39.451 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-16T16:01:39.451 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-16T16:01:39.451 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx 2024-04-16T16:01:39.451 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-16T16:01:39.451 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-16T16:01:39.451 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-16T16:01:39.451 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr 2024-04-16T16:01:39.451 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-16T16:01:39.451 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-16T16:01:39.451 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi 2024-04-16T16:01:39.452 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-16T16:01:39.452 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa 2024-04-16T16:01:39.452 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-16T16:01:39.452 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc 2024-04-16T16:01:39.452 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-16T16:01:39.452 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i 2024-04-16T16:01:39.452 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-16T16:01:39.452 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-16T16:01:39.452 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor 2024-04-16T16:01:39.452 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-16T16:01:39.452 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi 2024-04-16T16:01:39.452 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-16T16:01:39.452 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-16T16:01:39.452 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-16T16:01:39.453 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-16T16:01:39.453 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-16T16:01:39.453 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-16T16:01:39.453 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler 2024-04-16T16:01:39.453 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-16T16:01:39.453 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-16T16:01:39.453 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-16T16:01:39.453 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-16T16:01:39.453 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-16T16:01:39.453 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r 2024-04-16T16:01:39.453 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-16T16:01:39.453 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-16T16:01:39.453 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe 2024-04-16T16:01:39.453 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-16T16:01:39.453 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-16T16:01:39.454 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic 2024-04-16T16:01:39.454 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-16T16:01:39.454 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-16T16:01:39.454 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-16T16:01:39.454 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-16T16:01:39.454 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-16T16:01:39.454 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-16T16:01:39.454 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-16T16:01:39.454 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci 2024-04-16T16:01:39.454 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-16T16:01:39.454 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-16T16:01:39.454 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-16T16:01:39.454 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc 2024-04-16T16:01:39.455 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-16T16:01:39.455 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-16T16:01:39.455 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-16T16:01:39.455 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas 2024-04-16T16:01:39.455 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-16T16:01:39.455 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc 2024-04-16T16:01:39.455 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-16T16:01:39.455 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid 2024-04-16T16:01:39.455 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-16T16:01:39.455 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-16T16:01:39.455 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-16T16:01:39.455 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-16T16:01:39.455 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas 2024-04-16T16:01:39.456 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-16T16:01:39.456 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas 2024-04-16T16:01:39.456 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-16T16:01:39.456 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-16T16:01:39.456 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001 2024-04-16T16:01:39.456 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-16T16:01:39.456 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-16T16:01:39.456 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-16T16:01:39.456 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx 2024-04-16T16:01:39.456 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-16T16:01:39.456 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-16T16:01:39.456 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx 2024-04-16T16:01:39.456 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-16T16:01:39.456 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-16T16:01:39.456 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-16T16:01:39.457 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-16T16:01:39.457 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-16T16:01:39.457 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-16T16:01:39.457 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-16T16:01:39.457 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-16T16:01:39.457 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-16T16:01:39.457 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-16T16:01:39.457 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-16T16:01:39.457 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2 2024-04-16T16:01:39.457 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-16T16:01:39.457 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-16T16:01:39.457 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-16T16:01:39.457 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-16T16:01:39.457 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb 2024-04-16T16:01:39.458 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-16T16:01:39.458 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target 2024-04-16T16:01:39.458 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi 2024-04-16T16:01:39.458 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-16T16:01:39.458 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback 2024-04-16T16:01:39.458 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-16T16:01:39.458 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-16T16:01:39.458 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-16T16:01:39.458 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-16T16:01:39.458 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-16T16:01:39.458 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-16T16:01:39.458 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc 2024-04-16T16:01:39.458 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-16T16:01:39.458 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal 2024-04-16T16:01:39.459 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel 2024-04-16T16:01:39.459 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal 2024-04-16T16:01:39.459 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-16T16:01:39.459 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-16T16:01:39.459 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-16T16:01:39.459 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-16T16:01:39.459 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-16T16:01:39.459 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-16T16:01:39.459 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-16T16:01:39.459 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-16T16:01:39.459 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-16T16:01:39.459 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-16T16:01:39.459 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-16T16:01:39.459 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-16T16:01:39.460 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-16T16:01:39.461 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm 2024-04-16T16:01:39.461 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-16T16:01:39.461 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs 2024-04-16T16:01:39.461 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core 2024-04-16T16:01:39.461 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-16T16:01:39.461 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host 2024-04-16T16:01:39.461 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-16T16:01:39.461 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio 2024-04-16T16:01:39.461 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-16T16:01:39.461 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-16T16:01:39.461 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-16T16:01:39.461 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-16T16:01:39.461 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-16T16:01:39.461 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb 2024-04-16T16:01:39.462 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class 2024-04-16T16:01:39.462 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-16T16:01:39.462 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-16T16:01:39.462 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-16T16:01:39.462 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-16T16:01:39.462 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host 2024-04-16T16:01:39.462 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-16T16:01:39.462 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image 2024-04-16T16:01:39.462 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-16T16:01:39.462 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-16T16:01:39.462 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc 2024-04-16T16:01:39.462 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-16T16:01:39.462 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-16T16:01:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-16T16:01:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-16T16:01:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-16T16:01:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-16T16:01:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-16T16:01:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-16T16:01:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-16T16:01:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-16T16:01:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga 2024-04-16T16:01:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-16T16:01:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-16T16:01:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-16T16:01:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-16T16:01:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-16T16:01:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-16T16:01:39.464 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-16T16:01:39.464 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy 2024-04-16T16:01:39.464 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-16T16:01:39.464 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial 2024-04-16T16:01:39.464 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-16T16:01:39.464 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-16T16:01:39.464 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-16T16:01:39.464 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-16T16:01:39.464 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-16T16:01:39.464 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-16T16:01:39.464 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-16T16:01:39.464 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-16T16:01:39.464 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-16T16:01:39.465 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-16T16:01:39.465 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-16T16:01:39.465 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-16T16:01:39.465 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-16T16:01:39.465 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-16T16:01:39.465 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-16T16:01:39.465 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-16T16:01:39.465 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-16T16:01:39.465 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-16T16:01:39.465 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-16T16:01:39.465 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-16T16:01:39.465 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-16T16:01:39.465 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-16T16:01:39.465 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-16T16:01:39.466 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-16T16:01:39.466 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-16T16:01:39.466 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-16T16:01:39.466 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-16T16:01:39.466 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-16T16:01:39.466 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-16T16:01:39.466 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-16T16:01:39.466 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-16T16:01:39.466 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-16T16:01:39.466 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-16T16:01:39.466 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-16T16:01:39.466 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-16T16:01:39.466 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-16T16:01:39.466 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-16T16:01:39.467 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-16T16:01:39.467 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-16T16:01:39.467 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-16T16:01:39.467 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-16T16:01:39.467 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-16T16:01:39.467 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-16T16:01:39.467 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-16T16:01:39.467 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-16T16:01:39.467 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage 2024-04-16T16:01:39.467 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-16T16:01:39.467 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-16T16:01:39.467 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-16T16:01:39.467 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-16T16:01:39.468 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-16T16:01:39.468 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-16T16:01:39.468 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-16T16:01:39.468 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-16T16:01:39.468 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-16T16:01:39.468 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-16T16:01:39.468 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-16T16:01:39.468 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-16T16:01:39.468 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-16T16:01:39.468 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-16T16:01:39.468 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-16T16:01:39.468 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip 2024-04-16T16:01:39.469 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-16T16:01:39.469 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-16T16:01:39.469 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-16T16:01:39.469 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio 2024-04-16T16:01:39.469 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci 2024-04-16T16:01:39.469 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-16T16:01:39.469 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-16T16:01:39.469 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-16T16:01:39.469 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-16T16:01:39.469 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost 2024-04-16T16:01:39.469 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-16T16:01:39.469 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-16T16:01:39.469 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-16T16:01:39.469 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-16T16:01:39.469 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video 2024-04-16T16:01:39.470 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight 2024-04-16T16:01:39.470 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-16T16:01:39.470 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-16T16:01:39.470 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-16T16:01:39.470 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio 2024-04-16T16:01:39.470 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio.ko 2024-04-16T16:01:39.470 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_balloon.ko 2024-04-16T16:01:39.470 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_mmio.ko 2024-04-16T16:01:39.470 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci.ko 2024-04-16T16:01:39.470 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-16T16:01:39.470 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-16T16:01:39.470 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_ring.ko 2024-04-16T16:01:39.470 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1 2024-04-16T16:01:39.470 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters 2024-04-16T16:01:39.471 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-16T16:01:39.471 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-16T16:01:39.471 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves 2024-04-16T16:01:39.471 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-16T16:01:39.471 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-16T16:01:39.471 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-16T16:01:39.471 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-16T16:01:39.471 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-16T16:01:39.471 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-16T16:01:39.471 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-16T16:01:39.471 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-16T16:01:39.471 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-16T16:01:39.471 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-16T16:01:39.471 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-16T16:01:39.472 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-16T16:01:39.472 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog 2024-04-16T16:01:39.472 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-16T16:01:39.472 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-16T16:01:39.472 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-16T16:01:39.472 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-16T16:01:39.472 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-16T16:01:39.472 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-16T16:01:39.472 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-16T16:01:39.472 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-16T16:01:39.472 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-16T16:01:39.472 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-16T16:01:39.472 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-16T16:01:39.472 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen 2024-04-16T16:01:39.473 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-16T16:01:39.474 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-16T16:01:39.474 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-16T16:01:39.474 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-16T16:01:39.474 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback 2024-04-16T16:01:39.474 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-16T16:01:39.474 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-16T16:01:39.474 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs 2024-04-16T16:01:39.474 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-16T16:01:39.474 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs 2024-04-16T16:01:39.474 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p 2024-04-16T16:01:39.474 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-16T16:01:39.474 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs 2024-04-16T16:01:39.474 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-16T16:01:39.475 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs 2024-04-16T16:01:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-16T16:01:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs 2024-04-16T16:01:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-16T16:01:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat 2024-04-16T16:01:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/fat.ko 2024-04-16T16:01:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-16T16:01:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/vfat.ko 2024-04-16T16:01:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse 2024-04-16T16:01:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-16T16:01:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/fuse.ko 2024-04-16T16:01:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2 2024-04-16T16:01:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-16T16:01:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs 2024-04-16T16:01:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-16T16:01:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus 2024-04-16T16:01:39.477 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-16T16:01:39.477 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs 2024-04-16T16:01:39.477 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-16T16:01:39.477 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs 2024-04-16T16:01:39.477 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-16T16:01:39.477 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd 2024-04-16T16:01:39.477 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-16T16:01:39.477 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix 2024-04-16T16:01:39.477 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-16T16:01:39.477 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs 2024-04-16T16:01:39.477 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-16T16:01:39.477 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs 2024-04-16T16:01:39.477 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout 2024-04-16T16:01:39.477 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-16T16:01:39.477 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout 2024-04-16T16:01:39.478 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-16T16:01:39.478 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout 2024-04-16T16:01:39.478 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-16T16:01:39.478 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-16T16:01:39.478 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-16T16:01:39.478 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-16T16:01:39.478 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common 2024-04-16T16:01:39.478 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-16T16:01:39.478 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-16T16:01:39.478 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd 2024-04-16T16:01:39.478 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-16T16:01:39.478 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2 2024-04-16T16:01:39.478 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-16T16:01:39.478 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-16T16:01:39.479 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-16T16:01:39.480 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-16T16:01:39.481 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-16T16:01:39.481 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-16T16:01:39.481 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-16T16:01:39.481 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-16T16:01:39.481 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-16T16:01:39.481 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-16T16:01:39.481 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-16T16:01:39.481 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-16T16:01:39.481 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-16T16:01:39.481 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-16T16:01:39.481 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-16T16:01:39.481 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-16T16:01:39.481 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-16T16:01:39.481 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-16T16:01:39.481 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-16T16:01:39.482 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-16T16:01:39.482 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-16T16:01:39.482 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-16T16:01:39.482 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2 2024-04-16T16:01:39.482 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster 2024-04-16T16:01:39.482 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-16T16:01:39.482 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm 2024-04-16T16:01:39.482 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-16T16:01:39.482 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs 2024-04-16T16:01:39.482 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-16T16:01:39.482 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-16T16:01:39.482 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-16T16:01:39.482 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-16T16:01:39.482 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-16T16:01:39.482 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs 2024-04-16T16:01:39.483 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-16T16:01:39.483 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore 2024-04-16T16:01:39.483 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-16T16:01:39.483 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs 2024-04-16T16:01:39.483 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-16T16:01:39.483 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs 2024-04-16T16:01:39.483 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-16T16:01:39.483 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb 2024-04-16T16:01:39.483 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client 2024-04-16T16:01:39.483 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-16T16:01:39.483 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common 2024-04-16T16:01:39.483 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-16T16:01:39.483 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-16T16:01:39.483 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs 2024-04-16T16:01:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-16T16:01:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv 2024-04-16T16:01:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-16T16:01:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf 2024-04-16T16:01:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-16T16:01:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs 2024-04-16T16:01:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-16T16:01:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs 2024-04-16T16:01:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-16T16:01:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel 2024-04-16T16:01:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu 2024-04-16T16:01:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu/rcutorture.ko 2024-04-16T16:01:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-16T16:01:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace 2024-04-16T16:01:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-16T16:01:39.485 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib 2024-04-16T16:01:39.485 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/asn1_encoder.ko 2024-04-16T16:01:39.485 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-16T16:01:39.485 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-16T16:01:39.485 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto 2024-04-16T16:01:39.485 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-16T16:01:39.485 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-16T16:01:39.485 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-16T16:01:39.485 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-16T16:01:39.485 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4 2024-04-16T16:01:39.485 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-16T16:01:39.485 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-16T16:01:39.485 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math 2024-04-16T16:01:39.485 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-16T16:01:39.486 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6 2024-04-16T16:01:39.486 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-16T16:01:39.486 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon 2024-04-16T16:01:39.486 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-16T16:01:39.486 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-16T16:01:39.486 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-16T16:01:39.486 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-16T16:01:39.486 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd 2024-04-16T16:01:39.486 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd/zstd_compress.ko 2024-04-16T16:01:39.486 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm 2024-04-16T16:01:39.486 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-16T16:01:39.486 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net 2024-04-16T16:01:39.487 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan 2024-04-16T16:01:39.487 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-16T16:01:39.487 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-16T16:01:39.487 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-16T16:01:39.487 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-16T16:01:39.487 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-16T16:01:39.487 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-16T16:01:39.487 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-16T16:01:39.487 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-16T16:01:39.487 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802 2024-04-16T16:01:39.487 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-16T16:01:39.487 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-16T16:01:39.487 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-16T16:01:39.487 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-16T16:01:39.488 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-16T16:01:39.488 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q 2024-04-16T16:01:39.488 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-16T16:01:39.488 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p 2024-04-16T16:01:39.488 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-16T16:01:39.488 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-16T16:01:39.488 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-16T16:01:39.488 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-16T16:01:39.488 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv 2024-04-16T16:01:39.488 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-16T16:01:39.488 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge 2024-04-16T16:01:39.488 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-16T16:01:39.488 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-16T16:01:39.488 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter 2024-04-16T16:01:39.489 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-16T16:01:39.489 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-16T16:01:39.489 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-16T16:01:39.489 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-16T16:01:39.489 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-16T16:01:39.489 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-16T16:01:39.489 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-16T16:01:39.489 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-16T16:01:39.489 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-16T16:01:39.489 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-16T16:01:39.489 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-16T16:01:39.489 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-16T16:01:39.489 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-16T16:01:39.489 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-16T16:01:39.490 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-16T16:01:39.490 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-16T16:01:39.490 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-16T16:01:39.490 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-16T16:01:39.490 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-16T16:01:39.490 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-16T16:01:39.490 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-16T16:01:39.490 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-16T16:01:39.490 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-16T16:01:39.490 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-16T16:01:39.490 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph 2024-04-16T16:01:39.490 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-16T16:01:39.490 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core 2024-04-16T16:01:39.490 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/failover.ko 2024-04-16T16:01:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-16T16:01:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp 2024-04-16T16:01:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-16T16:01:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-16T16:01:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-16T16:01:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-16T16:01:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver 2024-04-16T16:01:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver/dns_resolver.ko 2024-04-16T16:01:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa 2024-04-16T16:01:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-16T16:01:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-16T16:01:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-16T16:01:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-16T16:01:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-16T16:01:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154 2024-04-16T16:01:39.492 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan 2024-04-16T16:01:39.492 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-16T16:01:39.492 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-16T16:01:39.492 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-16T16:01:39.492 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4 2024-04-16T16:01:39.492 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-16T16:01:39.492 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-16T16:01:39.492 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-16T16:01:39.492 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-16T16:01:39.492 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-16T16:01:39.492 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-16T16:01:39.492 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-16T16:01:39.492 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-16T16:01:39.492 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-16T16:01:39.492 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-16T16:01:39.493 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter 2024-04-16T16:01:39.493 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-16T16:01:39.493 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-16T16:01:39.493 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-16T16:01:39.493 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-16T16:01:39.493 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-16T16:01:39.493 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-16T16:01:39.493 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-16T16:01:39.493 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-16T16:01:39.493 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-16T16:01:39.493 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-16T16:01:39.493 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-16T16:01:39.493 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-16T16:01:39.493 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-16T16:01:39.494 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-16T16:01:39.494 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-16T16:01:39.494 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-16T16:01:39.494 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-16T16:01:39.494 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-16T16:01:39.494 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-16T16:01:39.494 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-16T16:01:39.494 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-16T16:01:39.494 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-16T16:01:39.494 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-16T16:01:39.494 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-16T16:01:39.494 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-16T16:01:39.494 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-16T16:01:39.494 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-16T16:01:39.495 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-16T16:01:39.495 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-16T16:01:39.495 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-16T16:01:39.495 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-16T16:01:39.495 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-16T16:01:39.495 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-16T16:01:39.495 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-16T16:01:39.495 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-16T16:01:39.495 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-16T16:01:39.495 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-16T16:01:39.495 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-16T16:01:39.495 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6 2024-04-16T16:01:39.495 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-16T16:01:39.496 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-16T16:01:39.496 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-16T16:01:39.496 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-16T16:01:39.496 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-16T16:01:39.496 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-16T16:01:39.496 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-16T16:01:39.496 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter 2024-04-16T16:01:39.496 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-16T16:01:39.496 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-16T16:01:39.496 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-16T16:01:39.496 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-16T16:01:39.496 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-16T16:01:39.496 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-16T16:01:39.496 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-16T16:01:39.496 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-16T16:01:39.497 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-16T16:01:39.497 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-16T16:01:39.497 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-16T16:01:39.497 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-16T16:01:39.497 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-16T16:01:39.497 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-16T16:01:39.497 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-16T16:01:39.497 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-16T16:01:39.497 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-16T16:01:39.497 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-16T16:01:39.497 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-16T16:01:39.497 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-16T16:01:39.497 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-16T16:01:39.498 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-16T16:01:39.498 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-16T16:01:39.498 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-16T16:01:39.498 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-16T16:01:39.498 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-16T16:01:39.498 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-16T16:01:39.498 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key 2024-04-16T16:01:39.498 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-16T16:01:39.498 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp 2024-04-16T16:01:39.498 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-16T16:01:39.498 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-16T16:01:39.498 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-16T16:01:39.498 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-16T16:01:39.499 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-16T16:01:39.499 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-16T16:01:39.499 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-16T16:01:39.499 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc 2024-04-16T16:01:39.499 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-16T16:01:39.499 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154 2024-04-16T16:01:39.499 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-16T16:01:39.499 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls 2024-04-16T16:01:39.499 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-16T16:01:39.499 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter 2024-04-16T16:01:39.499 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset 2024-04-16T16:01:39.499 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-16T16:01:39.499 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-16T16:01:39.500 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-16T16:01:39.500 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-16T16:01:39.500 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-16T16:01:39.500 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-16T16:01:39.500 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-16T16:01:39.500 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-16T16:01:39.500 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-16T16:01:39.500 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-16T16:01:39.500 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-16T16:01:39.500 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-16T16:01:39.500 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-16T16:01:39.500 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-16T16:01:39.500 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-16T16:01:39.501 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-16T16:01:39.501 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs 2024-04-16T16:01:39.501 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-16T16:01:39.501 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-16T16:01:39.501 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-16T16:01:39.501 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-16T16:01:39.501 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-16T16:01:39.501 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-16T16:01:39.501 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-16T16:01:39.501 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-16T16:01:39.501 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-16T16:01:39.501 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-16T16:01:39.501 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-16T16:01:39.501 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-16T16:01:39.502 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-16T16:01:39.502 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-16T16:01:39.502 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-16T16:01:39.502 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-16T16:01:39.502 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-16T16:01:39.502 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-16T16:01:39.502 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-16T16:01:39.502 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-16T16:01:39.502 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-16T16:01:39.502 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-16T16:01:39.502 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-16T16:01:39.502 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-16T16:01:39.502 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-16T16:01:39.503 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-16T16:01:39.503 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-16T16:01:39.503 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-16T16:01:39.503 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-16T16:01:39.503 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-16T16:01:39.503 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-16T16:01:39.503 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-16T16:01:39.503 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-16T16:01:39.503 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-16T16:01:39.503 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-16T16:01:39.503 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-16T16:01:39.503 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-16T16:01:39.503 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-16T16:01:39.504 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-16T16:01:39.504 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-16T16:01:39.504 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-16T16:01:39.504 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-16T16:01:39.504 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-16T16:01:39.504 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-16T16:01:39.504 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-16T16:01:39.504 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-16T16:01:39.504 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-16T16:01:39.504 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-16T16:01:39.504 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-16T16:01:39.504 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-16T16:01:39.504 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-16T16:01:39.504 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-16T16:01:39.505 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-16T16:01:39.505 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-16T16:01:39.505 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-16T16:01:39.505 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-16T16:01:39.505 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-16T16:01:39.505 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-16T16:01:39.505 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_quota.ko 2024-04-16T16:01:39.505 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-16T16:01:39.505 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-16T16:01:39.505 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-16T16:01:39.505 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-16T16:01:39.505 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tunnel.ko 2024-04-16T16:01:39.505 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-16T16:01:39.505 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-16T16:01:39.506 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-16T16:01:39.506 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-16T16:01:39.506 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-16T16:01:39.506 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-16T16:01:39.506 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-16T16:01:39.506 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-16T16:01:39.506 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-16T16:01:39.506 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-16T16:01:39.506 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-16T16:01:39.506 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-16T16:01:39.506 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-16T16:01:39.506 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-16T16:01:39.506 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-16T16:01:39.507 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-16T16:01:39.507 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-16T16:01:39.507 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-16T16:01:39.507 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-16T16:01:39.507 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-16T16:01:39.507 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-16T16:01:39.507 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-16T16:01:39.507 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-16T16:01:39.507 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-16T16:01:39.507 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-16T16:01:39.507 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-16T16:01:39.507 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-16T16:01:39.507 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-16T16:01:39.508 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-16T16:01:39.508 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-16T16:01:39.508 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-16T16:01:39.508 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-16T16:01:39.508 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-16T16:01:39.508 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-16T16:01:39.508 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-16T16:01:39.508 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-16T16:01:39.508 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-16T16:01:39.508 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-16T16:01:39.508 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-16T16:01:39.508 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-16T16:01:39.508 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-16T16:01:39.508 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-16T16:01:39.509 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-16T16:01:39.509 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-16T16:01:39.509 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-16T16:01:39.509 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-16T16:01:39.509 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-16T16:01:39.509 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-16T16:01:39.509 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-16T16:01:39.509 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-16T16:01:39.509 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-16T16:01:39.509 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-16T16:01:39.509 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-16T16:01:39.509 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-16T16:01:39.509 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-16T16:01:39.509 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-16T16:01:39.510 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-16T16:01:39.510 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-16T16:01:39.510 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-16T16:01:39.510 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-16T16:01:39.510 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-16T16:01:39.510 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-16T16:01:39.510 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-16T16:01:39.510 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-16T16:01:39.510 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-16T16:01:39.510 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-16T16:01:39.510 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-16T16:01:39.510 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-16T16:01:39.510 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-16T16:01:39.510 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-16T16:01:39.511 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-16T16:01:39.511 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink 2024-04-16T16:01:39.511 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-16T16:01:39.511 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh 2024-04-16T16:01:39.511 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-16T16:01:39.511 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch 2024-04-16T16:01:39.511 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-16T16:01:39.511 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-16T16:01:39.511 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-16T16:01:39.511 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-16T16:01:39.511 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet 2024-04-16T16:01:39.511 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-16T16:01:39.511 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds 2024-04-16T16:01:39.511 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-16T16:01:39.512 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-16T16:01:39.512 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-16T16:01:39.512 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched 2024-04-16T16:01:39.512 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-16T16:01:39.512 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-16T16:01:39.512 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-16T16:01:39.512 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-16T16:01:39.512 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-16T16:01:39.512 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-16T16:01:39.512 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-16T16:01:39.512 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-16T16:01:39.512 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-16T16:01:39.512 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-16T16:01:39.513 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-16T16:01:39.513 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-16T16:01:39.513 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-16T16:01:39.513 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-16T16:01:39.513 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-16T16:01:39.513 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-16T16:01:39.513 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-16T16:01:39.513 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-16T16:01:39.513 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-16T16:01:39.513 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-16T16:01:39.513 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-16T16:01:39.513 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-16T16:01:39.513 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-16T16:01:39.513 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-16T16:01:39.514 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-16T16:01:39.514 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-16T16:01:39.514 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-16T16:01:39.514 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-16T16:01:39.514 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-16T16:01:39.514 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-16T16:01:39.514 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-16T16:01:39.514 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-16T16:01:39.514 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-16T16:01:39.514 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-16T16:01:39.514 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-16T16:01:39.514 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-16T16:01:39.514 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-16T16:01:39.514 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-16T16:01:39.515 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-16T16:01:39.515 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-16T16:01:39.515 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-16T16:01:39.515 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-16T16:01:39.515 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-16T16:01:39.515 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp 2024-04-16T16:01:39.515 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-16T16:01:39.515 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-16T16:01:39.515 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc 2024-04-16T16:01:39.515 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss 2024-04-16T16:01:39.515 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-16T16:01:39.515 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-16T16:01:39.515 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-16T16:01:39.515 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma 2024-04-16T16:01:39.516 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-16T16:01:39.516 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc 2024-04-16T16:01:39.516 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-16T16:01:39.516 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-16T16:01:39.516 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix 2024-04-16T16:01:39.516 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-16T16:01:39.516 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock 2024-04-16T16:01:39.516 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-16T16:01:39.516 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-16T16:01:39.516 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-16T16:01:39.516 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-16T16:01:39.516 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-16T16:01:39.516 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm 2024-04-16T16:01:39.516 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-16T16:01:39.516 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security 2024-04-16T16:01:39.517 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys 2024-04-16T16:01:39.517 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys 2024-04-16T16:01:39.517 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-16T16:01:39.517 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys 2024-04-16T16:01:39.517 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys/trusted.ko 2024-04-16T16:01:39.517 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-16T16:01:39.517 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-16T16:01:39.517 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-16T16:01:39.517 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-16T16:01:39.517 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-16T16:01:39.517 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-16T16:01:39.517 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-16T16:01:39.517 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-16T16:01:39.517 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-16T16:01:39.517 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-16T16:01:39.518 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-16T16:01:39.518 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-16T16:01:39.518 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-16T16:01:39.518 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T16:01:39.521 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:39.521 DEBUG:teuthology.orchestra.run.smithi103:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-16T16:01:39.523 INFO:teuthology.orchestra.run.smithi045.stdout:postinstall scriptlet (using /bin/sh): 2024-04-16T16:01:39.523 INFO:teuthology.orchestra.run.smithi045.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T16:01:39.523 INFO:teuthology.orchestra.run.smithi045.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g3f60ed14f7c1 /lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T16:01:39.524 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2024-04-16T16:01:39.524 INFO:teuthology.orchestra.run.smithi045.stdout:for file in vmlinuz System.map config; do 2024-04-16T16:01:39.524 INFO:teuthology.orchestra.run.smithi045.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1"; then 2024-04-16T16:01:39.524 INFO:teuthology.orchestra.run.smithi045.stdout: cp "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T16:01:39.524 INFO:teuthology.orchestra.run.smithi045.stdout: fi 2024-04-16T16:01:39.524 INFO:teuthology.orchestra.run.smithi045.stdout:done 2024-04-16T16:01:39.524 INFO:teuthology.orchestra.run.smithi045.stdout:if [ -d "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" ] && \ 2024-04-16T16:01:39.524 INFO:teuthology.orchestra.run.smithi045.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" >/dev/null 2>&1; then 2024-04-16T16:01:39.524 INFO:teuthology.orchestra.run.smithi045.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T16:01:39.524 INFO:teuthology.orchestra.run.smithi045.stdout: cp -r "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T16:01:39.524 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2024-04-16T16:01:39.524 INFO:teuthology.orchestra.run.smithi045.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep" ]; then 2024-04-16T16:01:39.524 INFO:teuthology.orchestra.run.smithi045.stdout: /usr/sbin/depmod 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:39.524 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2024-04-16T16:01:39.524 INFO:teuthology.orchestra.run.smithi045.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-16T16:01:39.525 INFO:teuthology.orchestra.run.smithi045.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T16:01:39.525 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-install remove 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:39.525 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2024-04-16T16:01:39.525 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -qi grub2-tools 2024-04-16T16:01:39.544 INFO:teuthology.orchestra.run.smithi103.stdout:postinstall scriptlet (using /bin/sh): 2024-04-16T16:01:39.544 INFO:teuthology.orchestra.run.smithi103.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T16:01:39.544 INFO:teuthology.orchestra.run.smithi103.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g3f60ed14f7c1 /lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T16:01:39.544 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-04-16T16:01:39.544 INFO:teuthology.orchestra.run.smithi103.stdout:for file in vmlinuz System.map config; do 2024-04-16T16:01:39.544 INFO:teuthology.orchestra.run.smithi103.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1"; then 2024-04-16T16:01:39.545 INFO:teuthology.orchestra.run.smithi103.stdout: cp "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T16:01:39.545 INFO:teuthology.orchestra.run.smithi103.stdout: fi 2024-04-16T16:01:39.545 INFO:teuthology.orchestra.run.smithi103.stdout:done 2024-04-16T16:01:39.545 INFO:teuthology.orchestra.run.smithi103.stdout:if [ -d "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" ] && \ 2024-04-16T16:01:39.545 INFO:teuthology.orchestra.run.smithi103.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" >/dev/null 2>&1; then 2024-04-16T16:01:39.545 INFO:teuthology.orchestra.run.smithi103.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T16:01:39.545 INFO:teuthology.orchestra.run.smithi103.stdout: cp -r "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T16:01:39.545 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-04-16T16:01:39.545 INFO:teuthology.orchestra.run.smithi103.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep" ]; then 2024-04-16T16:01:39.545 INFO:teuthology.orchestra.run.smithi103.stdout: /usr/sbin/depmod 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:39.545 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-04-16T16:01:39.545 INFO:teuthology.orchestra.run.smithi103.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-16T16:01:39.545 INFO:teuthology.orchestra.run.smithi103.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T16:01:39.545 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-install remove 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:39.546 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-04-16T16:01:39.546 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -qi grub2-tools 2024-04-16T16:01:39.605 INFO:teuthology.orchestra.run.smithi045.stdout:Name : grub2-tools 2024-04-16T16:01:39.605 INFO:teuthology.orchestra.run.smithi045.stdout:Epoch : 1 2024-04-16T16:01:39.606 INFO:teuthology.orchestra.run.smithi045.stdout:Version : 2.06 2024-04-16T16:01:39.606 INFO:teuthology.orchestra.run.smithi045.stdout:Release : 68.el9 2024-04-16T16:01:39.606 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: x86_64 2024-04-16T16:01:39.606 INFO:teuthology.orchestra.run.smithi045.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T16:01:39.606 INFO:teuthology.orchestra.run.smithi045.stdout:Group : Unspecified 2024-04-16T16:01:39.606 INFO:teuthology.orchestra.run.smithi045.stdout:Size : 8264254 2024-04-16T16:01:39.606 INFO:teuthology.orchestra.run.smithi045.stdout:License : GPLv3+ 2024-04-16T16:01:39.606 INFO:teuthology.orchestra.run.smithi045.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T16:01:39.606 INFO:teuthology.orchestra.run.smithi045.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T16:01:39.606 INFO:teuthology.orchestra.run.smithi045.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T16:01:39.606 INFO:teuthology.orchestra.run.smithi045.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T16:01:39.606 INFO:teuthology.orchestra.run.smithi045.stdout:Packager : builder@centos.org 2024-04-16T16:01:39.607 INFO:teuthology.orchestra.run.smithi045.stdout:Vendor : CentOS 2024-04-16T16:01:39.607 INFO:teuthology.orchestra.run.smithi045.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T16:01:39.607 INFO:teuthology.orchestra.run.smithi045.stdout:Summary : Support tools for GRUB. 2024-04-16T16:01:39.607 INFO:teuthology.orchestra.run.smithi045.stdout:Description : 2024-04-16T16:01:39.607 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:01:39.607 INFO:teuthology.orchestra.run.smithi045.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T16:01:39.607 INFO:teuthology.orchestra.run.smithi045.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T16:01:39.607 INFO:teuthology.orchestra.run.smithi045.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T16:01:39.607 INFO:teuthology.orchestra.run.smithi045.stdout:hardware devices. 2024-04-16T16:01:39.607 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:01:39.607 INFO:teuthology.orchestra.run.smithi045.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T16:01:39.610 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T16:01:39.610 INFO:teuthology.task.kernel:Updating grub on smithi045 to boot 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:39.610 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T16:01:39.622 INFO:teuthology.orchestra.run.smithi103.stdout:Name : grub2-tools 2024-04-16T16:01:39.622 INFO:teuthology.orchestra.run.smithi103.stdout:Epoch : 1 2024-04-16T16:01:39.622 INFO:teuthology.orchestra.run.smithi103.stdout:Version : 2.06 2024-04-16T16:01:39.622 INFO:teuthology.orchestra.run.smithi103.stdout:Release : 68.el9 2024-04-16T16:01:39.622 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: x86_64 2024-04-16T16:01:39.622 INFO:teuthology.orchestra.run.smithi103.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T16:01:39.622 INFO:teuthology.orchestra.run.smithi103.stdout:Group : Unspecified 2024-04-16T16:01:39.622 INFO:teuthology.orchestra.run.smithi103.stdout:Size : 8264254 2024-04-16T16:01:39.622 INFO:teuthology.orchestra.run.smithi103.stdout:License : GPLv3+ 2024-04-16T16:01:39.622 INFO:teuthology.orchestra.run.smithi103.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T16:01:39.623 INFO:teuthology.orchestra.run.smithi103.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T16:01:39.623 INFO:teuthology.orchestra.run.smithi103.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T16:01:39.623 INFO:teuthology.orchestra.run.smithi103.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T16:01:39.623 INFO:teuthology.orchestra.run.smithi103.stdout:Packager : builder@centos.org 2024-04-16T16:01:39.623 INFO:teuthology.orchestra.run.smithi103.stdout:Vendor : CentOS 2024-04-16T16:01:39.623 INFO:teuthology.orchestra.run.smithi103.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T16:01:39.623 INFO:teuthology.orchestra.run.smithi103.stdout:Summary : Support tools for GRUB. 2024-04-16T16:01:39.623 INFO:teuthology.orchestra.run.smithi103.stdout:Description : 2024-04-16T16:01:39.623 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:01:39.623 INFO:teuthology.orchestra.run.smithi103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T16:01:39.623 INFO:teuthology.orchestra.run.smithi103.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T16:01:39.623 INFO:teuthology.orchestra.run.smithi103.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T16:01:39.623 INFO:teuthology.orchestra.run.smithi103.stdout:hardware devices. 2024-04-16T16:01:39.623 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:01:39.623 INFO:teuthology.orchestra.run.smithi103.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T16:01:39.624 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T16:01:39.624 INFO:teuthology.task.kernel:Updating grub on smithi103 to boot 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:39.625 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T16:01:40.200 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2024-04-16T16:01:40.269 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2024-04-16T16:01:41.590 INFO:teuthology.orchestra.run.smithi103.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T16:01:41.623 INFO:teuthology.orchestra.run.smithi103.stderr:done 2024-04-16T16:01:41.625 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-04-16T16:01:41.640 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.P540WlawAI 2024-04-16T16:01:41.640 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.P540WlawAI 2024-04-16T16:01:41.704 INFO:teuthology.orchestra.run.smithi045.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T16:01:41.717 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.P540WlawAI 2024-04-16T16:01:41.740 INFO:teuthology.orchestra.run.smithi045.stderr:done 2024-04-16T16:01:41.751 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-04-16T16:01:41.806 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.EowoXo3DRc 2024-04-16T16:01:41.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EowoXo3DRc 2024-04-16T16:01:41.863 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.P540WlawAI is 6KB 2024-04-16T16:01:41.886 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.EowoXo3DRc 2024-04-16T16:01:41.912 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.P540WlawAI 2024-04-16T16:01:41.926 DEBUG:teuthology.orchestra.run.smithi103:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T16:01:41.999 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T16:01:42.000 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T16:01:42.000 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T16:01:42.000 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1.conf 2024-04-16T16:01:42.001 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:42.025 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.EowoXo3DRc is 6KB 2024-04-16T16:01:42.075 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.EowoXo3DRc 2024-04-16T16:01:42.090 DEBUG:teuthology.orchestra.run.smithi045:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T16:01:42.159 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T16:01:42.159 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T16:01:42.159 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T16:01:42.159 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1.conf 2024-04-16T16:01:42.160 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:01:42.163 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2024-04-16T16:01:42.272 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-04-16T16:01:42.315 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2024-04-16T16:02:12.195 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-04-16T16:02:12.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-16T16:02:12.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T16:02:12.318 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-16T16:02:12.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-16T16:02:12.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T16:02:30.563 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-04-16T16:02:30.814 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2024-04-16T16:02:39.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-16T16:02:39.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T16:02:39.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-16T16:02:39.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T16:02:42.654 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-04-16T16:02:54.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-16T16:02:54.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T16:02:57.726 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-04-16T16:03:12.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-16T16:03:12.740 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T16:03:31.230 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-04-16T16:03:39.824 DEBUG:teuthology.orchestra.remote:timed out 2024-04-16T16:03:49.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-16T16:03:49.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T16:03:49.702 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-04-16T16:03:50.345 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-16T16:03:50.345 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-16T16:03:50.346 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-04-16T16:03:50.387 INFO:teuthology.orchestra.run.smithi103.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:03:50.387 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:03:50.387 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T16:03:50.387 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T16:03:51.390 DEBUG:teuthology.parallel:result is None 2024-04-16T16:03:51.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-16T16:03:51.825 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T16:03:52.235 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-04-16T16:03:52.970 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-16T16:03:52.970 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-16T16:03:52.970 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-04-16T16:03:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:03:53.030 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T16:03:53.030 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T16:03:53.030 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T16:03:54.031 DEBUG:teuthology.parallel:result is None 2024-04-16T16:03:54.031 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-16T16:03:54.043 INFO:teuthology.task.internal:Creating test directory... 2024-04-16T16:03:54.043 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T16:03:54.051 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T16:03:54.103 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-16T16:03:54.110 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-16T16:03:54.122 INFO:teuthology.task.internal:Creating archive directory... 2024-04-16T16:03:54.122 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T16:03:54.129 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T16:03:54.196 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-16T16:03:54.203 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-16T16:03:54.204 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T16:03:54.210 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T16:03:54.336 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T16:03:54.347 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T16:03:54.411 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T16:03:54.420 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T16:03:54.424 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-16T16:03:54.432 INFO:teuthology.task.internal:Configuring sudo... 2024-04-16T16:03:54.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T16:03:54.437 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T16:03:54.533 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-16T16:03:54.543 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-16T16:03:54.543 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T16:03:54.550 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T16:03:54.599 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T16:03:54.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T16:03:54.794 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T16:03:54.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T16:03:54.981 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:03:54.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T16:03:55.074 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T16:03:55.152 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T16:03:55.253 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T16:03:55.331 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T16:03:55.419 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:03:55.419 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T16:03:55.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-04-16T16:03:55.511 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-04-16T16:03:55.634 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T16:03:55.651 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T16:03:56.002 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-16T16:03:56.010 INFO:teuthology.task.internal:Starting timer... 2024-04-16T16:03:56.010 INFO:teuthology.run_tasks:Running task pcp... 2024-04-16T16:03:56.030 INFO:teuthology.run_tasks:Running task selinux... 2024-04-16T16:03:56.039 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-16T16:03:56.040 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2024-04-16T16:03:56.161 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: 2024-04-16T16:03:56.166 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-04-16T16:03:56.275 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-04-16T16:03:56.278 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-16T16:03:56.278 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2024-04-16T16:03:56.338 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2024-04-16T16:03:56.338 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-04-16T16:03:56.379 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-04-16T16:03:56.379 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi045.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi103.front.sepia.ceph.com': 'permissive'} 2024-04-16T16:03:56.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T16:03:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T16:03:56.476 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T16:03:56.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T16:03:56.547 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-16T16:03:56.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2024-04-16T16:03:56.651 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-04-16T16:03:56.732 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-16T16:03:56.743 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-16T16:03:56.749 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-16T16:03:56.878 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-16T16:03:56.893 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-16T16:03:56.894 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi045.front.sepia.ceph.com,smithi103.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-16T16:10:10.378 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi045.front.sepia.ceph.com'), Remote(name='ubuntu@smithi103.front.sepia.ceph.com')] 2024-04-16T16:10:10.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-16T16:10:10.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T16:10:10.555 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-04-16T16:10:10.715 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-16T16:10:10.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-16T16:10:10.716 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T16:10:10.857 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-04-16T16:10:11.025 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-16T16:10:11.025 INFO:teuthology.run_tasks:Running task clock... 2024-04-16T16:10:11.157 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-16T16:10:11.158 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T16:10:11.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T16:10:11.166 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T16:10:11.166 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T16:10:11.270 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T16:10:11.276 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T16:10:11.329 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T16:10:11.330 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T16:10:11.422 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-04-16T16:10:11.427 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2024-04-16T16:10:11.464 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-04-16T16:10:11.466 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2024-04-16T16:10:11.517 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T16:10:11.519 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T16:10:11.570 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T16:10:11.575 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T16:10:11.708 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-04-16T16:10:11.714 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-04-16T16:10:11.716 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T16:10:11.717 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-16T16:10:11.717 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T16:10:11.717 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T16:10:11.717 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T16:10:11.717 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T16:10:11.722 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T16:10:11.723 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-04-16T16:10:11.723 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T16:10:11.723 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T16:10:11.723 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T16:10:11.723 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T16:10:11.724 INFO:teuthology.run_tasks:Running task install... 2024-04-16T16:10:11.734 DEBUG:teuthology.task.install:project ceph 2024-04-16T16:10:11.735 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f'}} 2024-04-16T16:10:11.735 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f'} 2024-04-16T16:10:11.735 INFO:teuthology.task.install:Using flavor: default 2024-04-16T16:10:11.741 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-16T16:10:11.741 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-16T16:10:11.741 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f', 'tag': None, 'wait_for_package': False} 2024-04-16T16:10:11.742 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f 2024-04-16T16:10:11.744 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f', 'tag': None, 'wait_for_package': False} 2024-04-16T16:10:11.744 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f 2024-04-16T16:10:11.889 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-lusov-shared-log-segment/c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/centos/9/flavors/default/ 2024-04-16T16:10:11.889 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2868.gc3c87ed0 2024-04-16T16:10:11.912 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-lusov-shared-log-segment/c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/centos/9/flavors/default/ 2024-04-16T16:10:11.913 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2868.gc3c87ed0 2024-04-16T16:10:12.024 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-shared-log-segment/c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-shared-log-segment/c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-shared-log-segment/c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T16:10:12.024 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:10:12.024 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T16:10:12.125 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-16T16:10:12.125 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T16:10:12.164 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-shared-log-segment/c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-shared-log-segment/c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-shared-log-segment/c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T16:10:12.164 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:10:12.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T16:10:12.264 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T16:10:12.272 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-16T16:10:12.273 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T16:10:12.397 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T16:10:12.419 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T16:10:12.488 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-04-16T16:10:12.492 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-04-16T16:10:12.560 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T16:10:12.649 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2024-04-16T16:10:12.654 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2024-04-16T16:10:12.843 INFO:teuthology.orchestra.run.smithi103.stdout:74 files removed 2024-04-16T16:10:12.882 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-16T16:10:13.030 INFO:teuthology.orchestra.run.smithi045.stdout:74 files removed 2024-04-16T16:10:13.070 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-16T16:10:13.925 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 159 kB/s | 83 kB 00:00 2024-04-16T16:10:14.152 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 153 kB/s | 83 kB 00:00 2024-04-16T16:10:14.476 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 25 kB/s | 11 kB 00:00 2024-04-16T16:10:14.662 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 28 kB/s | 11 kB 00:00 2024-04-16T16:10:15.035 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-04-16T16:10:15.220 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-04-16T16:10:15.856 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-04-16T16:10:15.888 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - AppStream 8.6 MB/s | 6.7 MB 00:00 2024-04-16T16:10:17.549 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-04-16T16:10:17.564 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-04-16T16:10:18.602 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-16T16:10:18.920 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 8.3 MB/s | 8.0 MB 00:00 2024-04-16T16:10:20.695 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-16T16:10:21.187 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-04-16T16:10:26.421 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.2 MB 00:01 2024-04-16T16:10:26.880 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.2 MB 00:01 2024-04-16T16:10:28.507 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-04-16T16:10:28.622 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-04-16T16:10:30.174 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-16T16:10:31.989 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 6.4 MB/s | 21 MB 00:03 2024-04-16T16:10:36.224 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-16T16:10:37.939 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-16T16:10:39.263 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-16T16:10:39.265 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-16T16:10:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-16T16:10:39.287 INFO:teuthology.orchestra.run.smithi045.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-16T16:10:39.289 INFO:teuthology.orchestra.run.smithi045.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-16T16:10:39.290 INFO:teuthology.orchestra.run.smithi045.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-16T16:10:39.292 INFO:teuthology.orchestra.run.smithi045.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-16T16:10:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-16T16:10:39.297 INFO:teuthology.orchestra.run.smithi045.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-16T16:10:39.392 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T16:10:39.409 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-04-16T16:10:39.409 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-04-16T16:10:39.409 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-04-16T16:10:39.409 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T16:10:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-16T16:10:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 6.6 k 2024-04-16T16:10:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 5.4 M 2024-04-16T16:10:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 890 k 2024-04-16T16:10:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 145 k 2024-04-16T16:10:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 1.5 M 2024-04-16T16:10:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 142 k 2024-04-16T16:10:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 3.5 M 2024-04-16T16:10:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 7.4 M 2024-04-16T16:10:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 49 k 2024-04-16T16:10:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 10 M 2024-04-16T16:10:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 49 M 2024-04-16T16:10:39.411 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 264 k 2024-04-16T16:10:39.411 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 762 k 2024-04-16T16:10:39.411 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-mirror x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 225 k 2024-04-16T16:10:39.411 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-shell x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 27 k 2024-04-16T16:10:39.411 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-top noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 18 k 2024-04-16T16:10:39.411 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-16T16:10:39.411 INFO:teuthology.orchestra.run.smithi045.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-16T16:10:39.411 INFO:teuthology.orchestra.run.smithi045.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-16T16:10:39.411 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 32 k 2024-04-16T16:10:39.411 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 746 k 2024-04-16T16:10:39.411 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 127 k 2024-04-16T16:10:39.411 INFO:teuthology.orchestra.run.smithi045.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-16T16:10:39.411 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-16T16:10:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-16T16:10:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-16T16:10:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-16T16:10:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-16T16:10:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 161 k 2024-04-16T16:10:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 320 k 2024-04-16T16:10:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 298 k 2024-04-16T16:10:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 99 k 2024-04-16T16:10:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 85 k 2024-04-16T16:10:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 3.1 M 2024-04-16T16:10:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 170 k 2024-04-16T16:10:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-16T16:10:39.412 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2024-04-16T16:10:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager x86_64 1:1.47.4-1.el9 CentOS-BaseOS 2.3 M 2024-04-16T16:10:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 CentOS-BaseOS 1.8 M 2024-04-16T16:10:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 CentOS-BaseOS 41 k 2024-04-16T16:10:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 CentOS-BaseOS 248 k 2024-04-16T16:10:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-16T16:10:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-16T16:10:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-16T16:10:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-16T16:10:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-16T16:10:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-16T16:10:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-16T16:10:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-16T16:10:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-16T16:10:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 3.4 M 2024-04-16T16:10:39.414 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 3.1 M 2024-04-16T16:10:39.414 INFO:teuthology.orchestra.run.smithi045.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-16T16:10:39.414 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-16T16:10:39.414 INFO:teuthology.orchestra.run.smithi045.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-16T16:10:39.414 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-16T16:10:39.414 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-16T16:10:39.414 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-16T16:10:39.414 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-16T16:10:39.414 INFO:teuthology.orchestra.run.smithi045.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-16T16:10:39.414 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-16T16:10:39.414 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-04-16T16:10:39.414 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T16:10:39.414 INFO:teuthology.orchestra.run.smithi045.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-16T16:10:39.415 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T16:10:39.415 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 21 M 2024-04-16T16:10:39.415 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 34 k 2024-04-16T16:10:39.415 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 2.4 M 2024-04-16T16:10:39.415 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 249 k 2024-04-16T16:10:39.415 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 4.7 M 2024-04-16T16:10:39.415 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 17 M 2024-04-16T16:10:39.415 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 16 k 2024-04-16T16:10:39.415 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 25 k 2024-04-16T16:10:39.415 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T16:10:39.415 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T16:10:39.415 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T16:10:39.415 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T16:10:39.415 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T16:10:39.416 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T16:10:39.416 INFO:teuthology.orchestra.run.smithi045.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T16:10:39.416 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T16:10:39.416 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T16:10:39.416 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 163 k 2024-04-16T16:10:39.416 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T16:10:39.416 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T16:10:39.416 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T16:10:39.416 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T16:10:39.416 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T16:10:39.416 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 500 k 2024-04-16T16:10:39.416 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T16:10:39.416 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 5.2 M 2024-04-16T16:10:39.417 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T16:10:39.417 INFO:teuthology.orchestra.run.smithi045.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-16T16:10:39.417 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T16:10:39.417 INFO:teuthology.orchestra.run.smithi045.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T16:10:39.417 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T16:10:39.417 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T16:10:39.417 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T16:10:39.417 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T16:10:39.417 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T16:10:39.417 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T16:10:39.417 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-16T16:10:39.417 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T16:10:39.417 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T16:10:39.417 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T16:10:39.418 INFO:teuthology.orchestra.run.smithi045.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-16T16:10:39.418 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T16:10:39.418 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T16:10:39.418 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T16:10:39.418 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 45 k 2024-04-16T16:10:39.418 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 135 k 2024-04-16T16:10:39.418 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T16:10:39.418 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T16:10:39.418 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T16:10:39.418 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T16:10:39.418 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-16T16:10:39.418 INFO:teuthology.orchestra.run.smithi045.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-16T16:10:39.418 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T16:10:39.418 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T16:10:39.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T16:10:39.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T16:10:39.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T16:10:39.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T16:10:39.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T16:10:39.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T16:10:39.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T16:10:39.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T16:10:39.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T16:10:39.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T16:10:39.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T16:10:39.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T16:10:39.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T16:10:39.420 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T16:10:39.420 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T16:10:39.420 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T16:10:39.420 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T16:10:39.420 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T16:10:39.420 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T16:10:39.420 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T16:10:39.420 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T16:10:39.420 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T16:10:39.420 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T16:10:39.420 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T16:10:39.420 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T16:10:39.420 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T16:10:39.420 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T16:10:39.421 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T16:10:39.421 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T16:10:39.421 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T16:10:39.421 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-16T16:10:39.421 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T16:10:39.421 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T16:10:39.421 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T16:10:39.421 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T16:10:39.421 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T16:10:39.421 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T16:10:39.421 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T16:10:39.421 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T16:10:39.421 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T16:10:39.421 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T16:10:39.422 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T16:10:39.422 INFO:teuthology.orchestra.run.smithi045.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-16T16:10:39.422 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T16:10:39.422 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T16:10:39.422 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T16:10:39.422 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T16:10:39.422 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T16:10:39.422 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T16:10:39.422 INFO:teuthology.orchestra.run.smithi045.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T16:10:39.422 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T16:10:39.422 INFO:teuthology.orchestra.run.smithi045.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T16:10:39.422 INFO:teuthology.orchestra.run.smithi045.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-16T16:10:39.422 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T16:10:39.423 INFO:teuthology.orchestra.run.smithi045.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-16T16:10:39.423 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2024-04-16T16:10:39.423 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T16:10:39.423 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:10:39.423 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T16:10:39.423 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-04-16T16:10:39.423 INFO:teuthology.orchestra.run.smithi045.stdout:Install 150 Packages 2024-04-16T16:10:39.423 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 24 Packages 2024-04-16T16:10:39.423 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:10:39.424 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 228 M 2024-04-16T16:10:39.424 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T16:10:40.977 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-16T16:10:42.008 INFO:teuthology.orchestra.run.smithi045.stdout:(1/174): ceph-19.0.0-2868.gc3c87ed0.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-04-16T16:10:42.008 INFO:teuthology.orchestra.run.smithi045.stdout:(2/174): ceph-fuse-19.0.0-2868.gc3c87ed0.el9.x8 1.4 MB/s | 890 kB 00:00 2024-04-16T16:10:42.008 INFO:teuthology.orchestra.run.smithi045.stdout:(3/174): ceph-immutable-object-cache-19.0.0-286 1.7 MB/s | 145 kB 00:00 2024-04-16T16:10:42.010 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-16T16:10:42.010 INFO:teuthology.orchestra.run.smithi103.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-16T16:10:42.010 INFO:teuthology.orchestra.run.smithi103.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-16T16:10:42.010 INFO:teuthology.orchestra.run.smithi103.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-16T16:10:42.010 INFO:teuthology.orchestra.run.smithi103.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-16T16:10:42.010 INFO:teuthology.orchestra.run.smithi103.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-16T16:10:42.010 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-16T16:10:42.010 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-16T16:10:42.011 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-16T16:10:42.011 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================================ 2024-04-16T16:10:42.011 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-16T16:10:42.011 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================================ 2024-04-16T16:10:42.011 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-16T16:10:42.011 INFO:teuthology.orchestra.run.smithi103.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-16T16:10:42.011 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 6.6 k 2024-04-16T16:10:42.011 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 5.4 M 2024-04-16T16:10:42.011 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 890 k 2024-04-16T16:10:42.011 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 145 k 2024-04-16T16:10:42.011 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 1.5 M 2024-04-16T16:10:42.011 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 142 k 2024-04-16T16:10:42.011 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 3.5 M 2024-04-16T16:10:42.012 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 7.4 M 2024-04-16T16:10:42.012 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 49 k 2024-04-16T16:10:42.012 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 10 M 2024-04-16T16:10:42.012 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 49 M 2024-04-16T16:10:42.012 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 264 k 2024-04-16T16:10:42.012 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 762 k 2024-04-16T16:10:42.012 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 225 k 2024-04-16T16:10:42.012 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 27 k 2024-04-16T16:10:42.012 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 18 k 2024-04-16T16:10:42.012 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-16T16:10:42.012 INFO:teuthology.orchestra.run.smithi103.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-16T16:10:42.013 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-16T16:10:42.013 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 32 k 2024-04-16T16:10:42.013 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 746 k 2024-04-16T16:10:42.013 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 127 k 2024-04-16T16:10:42.013 INFO:teuthology.orchestra.run.smithi103.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-16T16:10:42.013 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-16T16:10:42.013 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-16T16:10:42.013 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-16T16:10:42.013 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-16T16:10:42.013 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-16T16:10:42.013 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 161 k 2024-04-16T16:10:42.014 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 320 k 2024-04-16T16:10:42.014 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 298 k 2024-04-16T16:10:42.014 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 99 k 2024-04-16T16:10:42.014 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 85 k 2024-04-16T16:10:42.014 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 3.1 M 2024-04-16T16:10:42.014 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 170 k 2024-04-16T16:10:42.014 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-16T16:10:42.014 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-04-16T16:10:42.015 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager x86_64 1:1.47.4-1.el9 CentOS-BaseOS 2.3 M 2024-04-16T16:10:42.015 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 CentOS-BaseOS 1.8 M 2024-04-16T16:10:42.015 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 CentOS-BaseOS 41 k 2024-04-16T16:10:42.015 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 CentOS-BaseOS 248 k 2024-04-16T16:10:42.016 INFO:teuthology.orchestra.run.smithi103.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-16T16:10:42.016 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-16T16:10:42.016 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-16T16:10:42.016 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-16T16:10:42.016 INFO:teuthology.orchestra.run.smithi103.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-16T16:10:42.016 INFO:teuthology.orchestra.run.smithi103.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-16T16:10:42.016 INFO:teuthology.orchestra.run.smithi103.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-16T16:10:42.016 INFO:teuthology.orchestra.run.smithi103.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-16T16:10:42.017 INFO:teuthology.orchestra.run.smithi103.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-16T16:10:42.017 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 3.4 M 2024-04-16T16:10:42.017 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 3.1 M 2024-04-16T16:10:42.017 INFO:teuthology.orchestra.run.smithi103.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-16T16:10:42.017 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-16T16:10:42.017 INFO:teuthology.orchestra.run.smithi103.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-16T16:10:42.017 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-16T16:10:42.018 INFO:teuthology.orchestra.run.smithi103.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-16T16:10:42.018 INFO:teuthology.orchestra.run.smithi103.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-16T16:10:42.018 INFO:teuthology.orchestra.run.smithi103.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-16T16:10:42.018 INFO:teuthology.orchestra.run.smithi103.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-16T16:10:42.018 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-16T16:10:42.018 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-16T16:10:42.018 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T16:10:42.018 INFO:teuthology.orchestra.run.smithi103.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-16T16:10:42.019 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T16:10:42.021 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 21 M 2024-04-16T16:10:42.021 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 34 k 2024-04-16T16:10:42.021 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 2.4 M 2024-04-16T16:10:42.021 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 249 k 2024-04-16T16:10:42.021 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 4.7 M 2024-04-16T16:10:42.021 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 17 M 2024-04-16T16:10:42.021 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2868.gc3c87ed0.el9 ceph-noarch 16 k 2024-04-16T16:10:42.022 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 25 k 2024-04-16T16:10:42.022 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T16:10:42.022 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T16:10:42.022 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T16:10:42.022 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T16:10:42.022 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T16:10:42.022 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T16:10:42.022 INFO:teuthology.orchestra.run.smithi103.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T16:10:42.022 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T16:10:42.022 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T16:10:42.023 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 163 k 2024-04-16T16:10:42.023 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T16:10:42.023 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T16:10:42.023 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T16:10:42.023 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T16:10:42.023 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T16:10:42.023 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 500 k 2024-04-16T16:10:42.023 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T16:10:42.023 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 5.2 M 2024-04-16T16:10:42.023 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T16:10:42.023 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-16T16:10:42.024 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T16:10:42.024 INFO:teuthology.orchestra.run.smithi103.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T16:10:42.024 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T16:10:42.024 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T16:10:42.024 INFO:teuthology.orchestra.run.smithi103.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T16:10:42.024 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T16:10:42.024 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T16:10:42.024 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T16:10:42.024 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-16T16:10:42.024 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T16:10:42.024 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T16:10:42.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T16:10:42.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-16T16:10:42.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T16:10:42.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T16:10:42.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T16:10:42.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 45 k 2024-04-16T16:10:42.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:19.0.0-2868.gc3c87ed0.el9 ceph 135 k 2024-04-16T16:10:42.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T16:10:42.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T16:10:42.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T16:10:42.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T16:10:42.026 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-16T16:10:42.026 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-16T16:10:42.026 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T16:10:42.026 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T16:10:42.026 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T16:10:42.026 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T16:10:42.026 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T16:10:42.026 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T16:10:42.026 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T16:10:42.026 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T16:10:42.026 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T16:10:42.026 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T16:10:42.026 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T16:10:42.027 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T16:10:42.027 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T16:10:42.027 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T16:10:42.027 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T16:10:42.027 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T16:10:42.027 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T16:10:42.027 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T16:10:42.027 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T16:10:42.027 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T16:10:42.027 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T16:10:42.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T16:10:42.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T16:10:42.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T16:10:42.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T16:10:42.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T16:10:42.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T16:10:42.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T16:10:42.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T16:10:42.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T16:10:42.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T16:10:42.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T16:10:42.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-16T16:10:42.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T16:10:42.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T16:10:42.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T16:10:42.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T16:10:42.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T16:10:42.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T16:10:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T16:10:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T16:10:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T16:10:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T16:10:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T16:10:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-16T16:10:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T16:10:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T16:10:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T16:10:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T16:10:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T16:10:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T16:10:42.031 INFO:teuthology.orchestra.run.smithi103.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T16:10:42.031 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T16:10:42.031 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T16:10:42.031 INFO:teuthology.orchestra.run.smithi103.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-16T16:10:42.031 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T16:10:42.031 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-16T16:10:42.031 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-04-16T16:10:42.031 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T16:10:42.031 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:10:42.031 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-16T16:10:42.031 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================================ 2024-04-16T16:10:42.031 INFO:teuthology.orchestra.run.smithi103.stdout:Install 150 Packages 2024-04-16T16:10:42.031 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 24 Packages 2024-04-16T16:10:42.032 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:10:42.032 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 228 M 2024-04-16T16:10:42.032 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-16T16:10:42.559 INFO:teuthology.orchestra.run.smithi045.stdout:(4/174): ceph-base-19.0.0-2868.gc3c87ed0.el9.x8 2.8 MB/s | 5.4 MB 00:01 2024-04-16T16:10:42.773 INFO:teuthology.orchestra.run.smithi103.stdout:(1/174): ceph-19.0.0-2868.gc3c87ed0.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-04-16T16:10:42.792 INFO:teuthology.orchestra.run.smithi045.stdout:(5/174): ceph-mds-19.0.0-2868.gc3c87ed0.el9.x86 2.3 MB/s | 2.4 MB 00:01 2024-04-16T16:10:42.843 INFO:teuthology.orchestra.run.smithi045.stdout:(6/174): ceph-mgr-19.0.0-2868.gc3c87ed0.el9.x86 5.3 MB/s | 1.5 MB 00:00 2024-04-16T16:10:43.033 INFO:teuthology.orchestra.run.smithi103.stdout:(2/174): ceph-fuse-19.0.0-2868.gc3c87ed0.el9.x8 3.4 MB/s | 890 kB 00:00 2024-04-16T16:10:43.116 INFO:teuthology.orchestra.run.smithi103.stdout:(3/174): ceph-immutable-object-cache-19.0.0-286 1.7 MB/s | 145 kB 00:00 2024-04-16T16:10:43.375 INFO:teuthology.orchestra.run.smithi103.stdout:(4/174): ceph-base-19.0.0-2868.gc3c87ed0.el9.x8 6.1 MB/s | 5.4 MB 00:00 2024-04-16T16:10:43.558 INFO:teuthology.orchestra.run.smithi103.stdout:(5/174): ceph-mgr-19.0.0-2868.gc3c87ed0.el9.x86 8.2 MB/s | 1.5 MB 00:00 2024-04-16T16:10:43.634 INFO:teuthology.orchestra.run.smithi103.stdout:(6/174): ceph-mds-19.0.0-2868.gc3c87ed0.el9.x86 4.6 MB/s | 2.4 MB 00:00 2024-04-16T16:10:44.017 INFO:teuthology.orchestra.run.smithi103.stdout:(7/174): ceph-mon-19.0.0-2868.gc3c87ed0.el9.x86 10 MB/s | 4.7 MB 00:00 2024-04-16T16:10:45.101 INFO:teuthology.orchestra.run.smithi103.stdout:(8/174): ceph-radosgw-19.0.0-2868.gc3c87ed0.el9 9.2 MB/s | 10 MB 00:01 2024-04-16T16:10:45.159 INFO:teuthology.orchestra.run.smithi103.stdout:(9/174): ceph-selinux-19.0.0-2868.gc3c87ed0.el9 438 kB/s | 25 kB 00:00 2024-04-16T16:10:45.426 INFO:teuthology.orchestra.run.smithi045.stdout:(7/174): ceph-osd-19.0.0-2868.gc3c87ed0.el9.x86 6.5 MB/s | 17 MB 00:02 2024-04-16T16:10:45.710 INFO:teuthology.orchestra.run.smithi045.stdout:(8/174): ceph-mon-19.0.0-2868.gc3c87ed0.el9.x86 1.6 MB/s | 4.7 MB 00:02 2024-04-16T16:10:45.818 INFO:teuthology.orchestra.run.smithi045.stdout:(9/174): ceph-selinux-19.0.0-2868.gc3c87ed0.el9 234 kB/s | 25 kB 00:00 2024-04-16T16:10:47.043 INFO:teuthology.orchestra.run.smithi103.stdout:(10/174): ceph-osd-19.0.0-2868.gc3c87ed0.el9.x8 4.9 MB/s | 17 MB 00:03 2024-04-16T16:10:47.185 INFO:teuthology.orchestra.run.smithi103.stdout:(11/174): cephfs-mirror-19.0.0-2868.gc3c87ed0.e 1.6 MB/s | 225 kB 00:00 2024-04-16T16:10:47.268 INFO:teuthology.orchestra.run.smithi103.stdout:(12/174): cephfs-shell-19.0.0-2868.gc3c87ed0.el 328 kB/s | 27 kB 00:00 2024-04-16T16:10:47.373 INFO:teuthology.orchestra.run.smithi103.stdout:(13/174): libcephfs-devel-19.0.0-2868.gc3c87ed0 307 kB/s | 32 kB 00:00 2024-04-16T16:10:47.552 INFO:teuthology.orchestra.run.smithi103.stdout:(14/174): libcephfs2-19.0.0-2868.gc3c87ed0.el9. 4.1 MB/s | 746 kB 00:00 2024-04-16T16:10:47.568 INFO:teuthology.orchestra.run.smithi045.stdout:(10/174): ceph-common-19.0.0-2868.gc3c87ed0.el9 3.0 MB/s | 21 MB 00:06 2024-04-16T16:10:47.628 INFO:teuthology.orchestra.run.smithi103.stdout:(15/174): libcephsqlite-19.0.0-2868.gc3c87ed0.e 2.2 MB/s | 163 kB 00:00 2024-04-16T16:10:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:(16/174): librados-devel-19.0.0-2868.gc3c87ed0. 1.7 MB/s | 127 kB 00:00 2024-04-16T16:10:47.744 INFO:teuthology.orchestra.run.smithi045.stdout:(11/174): ceph-radosgw-19.0.0-2868.gc3c87ed0.el 4.3 MB/s | 10 MB 00:02 2024-04-16T16:10:47.768 INFO:teuthology.orchestra.run.smithi045.stdout:(12/174): cephfs-mirror-19.0.0-2868.gc3c87ed0.e 1.1 MB/s | 225 kB 00:00 2024-04-16T16:10:47.828 INFO:teuthology.orchestra.run.smithi103.stdout:(17/174): libradosstriper1-19.0.0-2868.gc3c87ed 3.9 MB/s | 500 kB 00:00 2024-04-16T16:10:47.844 INFO:teuthology.orchestra.run.smithi045.stdout:(13/174): cephfs-shell-19.0.0-2868.gc3c87ed0.el 273 kB/s | 27 kB 00:00 2024-04-16T16:10:47.878 INFO:teuthology.orchestra.run.smithi045.stdout:(14/174): libcephfs-devel-19.0.0-2868.gc3c87ed0 292 kB/s | 32 kB 00:00 2024-04-16T16:10:47.986 INFO:teuthology.orchestra.run.smithi045.stdout:(15/174): libcephsqlite-19.0.0-2868.gc3c87ed0.e 1.5 MB/s | 163 kB 00:00 2024-04-16T16:10:48.154 INFO:teuthology.orchestra.run.smithi045.stdout:(16/174): librados-devel-19.0.0-2868.gc3c87ed0. 764 kB/s | 127 kB 00:00 2024-04-16T16:10:48.187 INFO:teuthology.orchestra.run.smithi103.stdout:(18/174): ceph-common-19.0.0-2868.gc3c87ed0.el9 3.7 MB/s | 21 MB 00:05 2024-04-16T16:10:48.212 INFO:teuthology.orchestra.run.smithi045.stdout:(17/174): libcephfs2-19.0.0-2868.gc3c87ed0.el9. 2.0 MB/s | 746 kB 00:00 2024-04-16T16:10:48.312 INFO:teuthology.orchestra.run.smithi045.stdout:(18/174): libradosstriper1-19.0.0-2868.gc3c87ed 3.1 MB/s | 500 kB 00:00 2024-04-16T16:10:48.371 INFO:teuthology.orchestra.run.smithi103.stdout:(19/174): python3-ceph-argparse-19.0.0-2868.gc3 247 kB/s | 45 kB 00:00 2024-04-16T16:10:48.396 INFO:teuthology.orchestra.run.smithi045.stdout:(19/174): python3-ceph-argparse-19.0.0-2868.gc3 546 kB/s | 45 kB 00:00 2024-04-16T16:10:48.463 INFO:teuthology.orchestra.run.smithi103.stdout:(20/174): python3-ceph-common-19.0.0-2868.gc3c8 1.4 MB/s | 135 kB 00:00 2024-04-16T16:10:48.496 INFO:teuthology.orchestra.run.smithi045.stdout:(20/174): python3-ceph-common-19.0.0-2868.gc3c8 1.3 MB/s | 135 kB 00:00 2024-04-16T16:10:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:(21/174): python3-cephfs-19.0.0-2868.gc3c87ed0. 1.5 MB/s | 161 kB 00:00 2024-04-16T16:10:48.605 INFO:teuthology.orchestra.run.smithi045.stdout:(21/174): python3-cephfs-19.0.0-2868.gc3c87ed0. 1.5 MB/s | 161 kB 00:00 2024-04-16T16:10:48.697 INFO:teuthology.orchestra.run.smithi103.stdout:(22/174): python3-rados-19.0.0-2868.gc3c87ed0.e 2.5 MB/s | 320 kB 00:00 2024-04-16T16:10:48.797 INFO:teuthology.orchestra.run.smithi045.stdout:(22/174): python3-rados-19.0.0-2868.gc3c87ed0.e 1.6 MB/s | 320 kB 00:00 2024-04-16T16:10:48.805 INFO:teuthology.orchestra.run.smithi103.stdout:(23/174): librgw2-19.0.0-2868.gc3c87ed0.el9.x86 5.4 MB/s | 5.2 MB 00:00 2024-04-16T16:10:48.847 INFO:teuthology.orchestra.run.smithi103.stdout:(24/174): python3-rbd-19.0.0-2868.gc3c87ed0.el9 2.0 MB/s | 298 kB 00:00 2024-04-16T16:10:48.881 INFO:teuthology.orchestra.run.smithi103.stdout:(25/174): python3-rgw-19.0.0-2868.gc3c87ed0.el9 1.3 MB/s | 99 kB 00:00 2024-04-16T16:10:48.948 INFO:teuthology.orchestra.run.smithi103.stdout:(26/174): rbd-fuse-19.0.0-2868.gc3c87ed0.el9.x8 859 kB/s | 85 kB 00:00 2024-04-16T16:10:48.955 INFO:teuthology.orchestra.run.smithi045.stdout:(23/174): python3-rbd-19.0.0-2868.gc3c87ed0.el9 1.8 MB/s | 298 kB 00:00 2024-04-16T16:10:49.056 INFO:teuthology.orchestra.run.smithi103.stdout:(27/174): rbd-nbd-19.0.0-2868.gc3c87ed0.el9.x86 1.5 MB/s | 170 kB 00:00 2024-04-16T16:10:49.072 INFO:teuthology.orchestra.run.smithi045.stdout:(24/174): python3-rgw-19.0.0-2868.gc3c87ed0.el9 858 kB/s | 99 kB 00:00 2024-04-16T16:10:49.264 INFO:teuthology.orchestra.run.smithi045.stdout:(25/174): librgw2-19.0.0-2868.gc3c87ed0.el9.x86 5.0 MB/s | 5.2 MB 00:01 2024-04-16T16:10:49.289 INFO:teuthology.orchestra.run.smithi045.stdout:(26/174): rbd-fuse-19.0.0-2868.gc3c87ed0.el9.x8 394 kB/s | 85 kB 00:00 2024-04-16T16:10:49.381 INFO:teuthology.orchestra.run.smithi045.stdout:(27/174): rbd-nbd-19.0.0-2868.gc3c87ed0.el9.x86 1.8 MB/s | 170 kB 00:00 2024-04-16T16:10:49.590 INFO:teuthology.orchestra.run.smithi103.stdout:(28/174): ceph-test-19.0.0-2868.gc3c87ed0.el9.x 11 MB/s | 49 MB 00:04 2024-04-16T16:10:49.615 INFO:teuthology.orchestra.run.smithi103.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2868.g 61 kB/s | 34 kB 00:00 2024-04-16T16:10:49.665 INFO:teuthology.orchestra.run.smithi045.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2868.g 121 kB/s | 34 kB 00:00 2024-04-16T16:10:49.669 INFO:teuthology.orchestra.run.smithi103.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2868.gc3c87ed 1.8 MB/s | 142 kB 00:00 2024-04-16T16:10:49.749 INFO:teuthology.orchestra.run.smithi045.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2868.gc3c87ed 1.7 MB/s | 142 kB 00:00 2024-04-16T16:10:49.791 INFO:teuthology.orchestra.run.smithi103.stdout:(31/174): rbd-mirror-19.0.0-2868.gc3c87ed0.el9. 3.4 MB/s | 3.1 MB 00:00 2024-04-16T16:10:49.832 INFO:teuthology.orchestra.run.smithi045.stdout:(30/174): rbd-mirror-19.0.0-2868.gc3c87ed0.el9. 5.5 MB/s | 3.1 MB 00:00 2024-04-16T16:10:49.941 INFO:teuthology.orchestra.run.smithi103.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2868.gc3 1.6 MB/s | 249 kB 00:00 2024-04-16T16:10:50.025 INFO:teuthology.orchestra.run.smithi103.stdout:(33/174): ceph-mgr-rook-19.0.0-2868.gc3c87ed0.e 600 kB/s | 49 kB 00:00 2024-04-16T16:10:50.100 INFO:teuthology.orchestra.run.smithi103.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2868.gc 222 kB/s | 16 kB 00:00 2024-04-16T16:10:50.192 INFO:teuthology.orchestra.run.smithi103.stdout:(35/174): ceph-volume-19.0.0-2868.gc3c87ed0.el9 2.8 MB/s | 264 kB 00:00 2024-04-16T16:10:50.326 INFO:teuthology.orchestra.run.smithi103.stdout:(36/174): ceph-mgr-dashboard-19.0.0-2868.gc3c87 5.0 MB/s | 3.5 MB 00:00 2024-04-16T16:10:50.468 INFO:teuthology.orchestra.run.smithi103.stdout:(37/174): ceph-mgr-diskprediction-local-19.0.0- 9.3 MB/s | 7.4 MB 00:00 2024-04-16T16:10:50.493 INFO:teuthology.orchestra.run.smithi103.stdout:(38/174): cephfs-top-19.0.0-2868.gc3c87ed0.el9. 110 kB/s | 18 kB 00:00 2024-04-16T16:10:50.535 INFO:teuthology.orchestra.run.smithi103.stdout:(39/174): cephadm-19.0.0-2868.gc3c87ed0.el9.noa 2.2 MB/s | 762 kB 00:00 2024-04-16T16:10:50.766 INFO:teuthology.orchestra.run.smithi045.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2868.gc3c87 3.5 MB/s | 3.5 MB 00:01 2024-04-16T16:10:50.852 INFO:teuthology.orchestra.run.smithi103.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 526 kB/s | 106 kB 00:00 2024-04-16T16:10:50.899 INFO:teuthology.orchestra.run.smithi045.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2868.gc3 1.8 MB/s | 249 kB 00:00 2024-04-16T16:10:50.910 INFO:teuthology.orchestra.run.smithi103.stdout:(41/174): bison-3.7.4-5.el9.x86_64.rpm 2.2 MB/s | 920 kB 00:00 2024-04-16T16:10:50.975 INFO:teuthology.orchestra.run.smithi045.stdout:(33/174): ceph-mgr-rook-19.0.0-2868.gc3c87ed0.e 664 kB/s | 49 kB 00:00 2024-04-16T16:10:50.985 INFO:teuthology.orchestra.run.smithi103.stdout:(42/174): automake-1.16.2-8.el9.noarch.rpm 1.3 MB/s | 662 kB 00:00 2024-04-16T16:10:51.058 INFO:teuthology.orchestra.run.smithi045.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2868.gc 200 kB/s | 16 kB 00:00 2024-04-16T16:10:51.077 INFO:teuthology.orchestra.run.smithi103.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 1.4 MB/s | 307 kB 00:00 2024-04-16T16:10:51.103 INFO:teuthology.orchestra.run.smithi103.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 166 kB/s | 32 kB 00:00 2024-04-16T16:10:51.192 INFO:teuthology.orchestra.run.smithi045.stdout:(35/174): ceph-mgr-diskprediction-local-19.0.0- 5.4 MB/s | 7.4 MB 00:01 2024-04-16T16:10:51.225 INFO:teuthology.orchestra.run.smithi045.stdout:(36/174): ceph-volume-19.0.0-2868.gc3c87ed0.el9 1.6 MB/s | 264 kB 00:00 2024-04-16T16:10:51.301 INFO:teuthology.orchestra.run.smithi045.stdout:(37/174): cephfs-top-19.0.0-2868.gc3c87ed0.el9. 245 kB/s | 18 kB 00:00 2024-04-16T16:10:51.420 INFO:teuthology.orchestra.run.smithi103.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 50 kB/s | 17 kB 00:00 2024-04-16T16:10:51.426 INFO:teuthology.orchestra.run.smithi045.stdout:(38/174): cephadm-19.0.0-2868.gc3c87ed0.el9.noa 3.2 MB/s | 762 kB 00:00 2024-04-16T16:10:51.487 INFO:teuthology.orchestra.run.smithi103.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 26 kB/s | 10 kB 00:00 2024-04-16T16:10:51.668 INFO:teuthology.orchestra.run.smithi045.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 3.8 MB/s | 920 kB 00:00 2024-04-16T16:10:51.687 INFO:teuthology.orchestra.run.smithi103.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 599 kB/s | 159 kB 00:00 2024-04-16T16:10:51.710 INFO:teuthology.orchestra.run.smithi045.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 1.6 MB/s | 662 kB 00:00 2024-04-16T16:10:51.735 INFO:teuthology.orchestra.run.smithi045.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-04-16T16:10:51.777 INFO:teuthology.orchestra.run.smithi045.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 4.6 MB/s | 307 kB 00:00 2024-04-16T16:10:51.779 INFO:teuthology.orchestra.run.smithi103.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 152 kB/s | 44 kB 00:00 2024-04-16T16:10:51.794 INFO:teuthology.orchestra.run.smithi045.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 542 kB/s | 32 kB 00:00 2024-04-16T16:10:51.853 INFO:teuthology.orchestra.run.smithi045.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 298 kB/s | 17 kB 00:00 2024-04-16T16:10:51.854 INFO:teuthology.orchestra.run.smithi103.stdout:(49/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.4 MB/s | 3.0 MB 00:00 2024-04-16T16:10:51.938 INFO:teuthology.orchestra.run.smithi103.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 2.6 MB/s | 661 kB 00:00 2024-04-16T16:10:51.953 INFO:teuthology.orchestra.run.smithi045.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2024-04-16T16:10:51.970 INFO:teuthology.orchestra.run.smithi045.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 97 kB/s | 10 kB 00:00 2024-04-16T16:10:51.995 INFO:teuthology.orchestra.run.smithi045.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2024-04-16T16:10:52.012 INFO:teuthology.orchestra.run.smithi045.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-04-16T16:10:52.013 INFO:teuthology.orchestra.run.smithi103.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 2.4 MB/s | 578 kB 00:00 2024-04-16T16:10:52.039 INFO:teuthology.orchestra.run.smithi103.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 257 kB/s | 46 kB 00:00 2024-04-16T16:10:52.063 INFO:teuthology.orchestra.run.smithi045.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.9 MB/s | 661 kB 00:00 2024-04-16T16:10:52.081 INFO:teuthology.orchestra.run.smithi103.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 282 kB 00:00 2024-04-16T16:10:52.104 INFO:teuthology.orchestra.run.smithi045.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 578 kB 00:00 2024-04-16T16:10:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 888 kB/s | 46 kB 00:00 2024-04-16T16:10:52.122 INFO:teuthology.orchestra.run.smithi103.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-04-16T16:10:52.147 INFO:teuthology.orchestra.run.smithi103.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.0 MB/s | 217 kB 00:00 2024-04-16T16:10:52.173 INFO:teuthology.orchestra.run.smithi103.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 389 kB/s | 34 kB 00:00 2024-04-16T16:10:52.302 INFO:teuthology.orchestra.run.smithi045.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 282 kB 00:00 2024-04-16T16:10:52.347 INFO:teuthology.orchestra.run.smithi045.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 869 kB/s | 187 kB 00:00 2024-04-16T16:10:52.381 INFO:teuthology.orchestra.run.smithi045.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-04-16T16:10:52.398 INFO:teuthology.orchestra.run.smithi045.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 721 kB/s | 34 kB 00:00 2024-04-16T16:10:52.548 INFO:teuthology.orchestra.run.smithi045.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 29 MB/s | 4.7 MB 00:00 2024-04-16T16:10:52.590 INFO:teuthology.orchestra.run.smithi103.stdout:(57/174): postgresql-contrib-13.14-1.el9.x86_64 1.9 MB/s | 817 kB 00:00 2024-04-16T16:10:52.607 INFO:teuthology.orchestra.run.smithi045.stdout:(57/174): postgresql-13.14-1.el9.x86_64.rpm 8.3 MB/s | 1.5 MB 00:00 2024-04-16T16:10:52.648 INFO:teuthology.orchestra.run.smithi045.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 9.0 MB/s | 817 kB 00:00 2024-04-16T16:10:52.665 INFO:teuthology.orchestra.run.smithi103.stdout:(58/174): postgresql-13.14-1.el9.x86_64.rpm 3.1 MB/s | 1.5 MB 00:00 2024-04-16T16:10:52.674 INFO:teuthology.orchestra.run.smithi045.stdout:(59/174): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 132 kB 00:00 2024-04-16T16:10:52.766 INFO:teuthology.orchestra.run.smithi103.stdout:(59/174): openblas-openmp-0.3.21-2.el9.x86_64.r 7.4 MB/s | 4.7 MB 00:00 2024-04-16T16:10:52.791 INFO:teuthology.orchestra.run.smithi103.stdout:(60/174): postgresql-private-libs-13.14-1.el9.x 669 kB/s | 132 kB 00:00 2024-04-16T16:10:52.833 INFO:teuthology.orchestra.run.smithi045.stdout:(60/174): postgresql-server-13.14-1.el9.x86_64. 31 MB/s | 5.7 MB 00:00 2024-04-16T16:10:52.891 INFO:teuthology.orchestra.run.smithi045.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-04-16T16:10:52.907 INFO:teuthology.orchestra.run.smithi045.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 78 kB 00:00 2024-04-16T16:10:53.117 INFO:teuthology.orchestra.run.smithi045.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-04-16T16:10:53.142 INFO:teuthology.orchestra.run.smithi045.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 105 kB/s | 24 kB 00:00 2024-04-16T16:10:53.149 INFO:teuthology.orchestra.run.smithi103.stdout:(61/174): postgresql-server-13.14-1.el9.x86_64. 12 MB/s | 5.7 MB 00:00 2024-04-16T16:10:53.175 INFO:teuthology.orchestra.run.smithi103.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 211 kB/s | 78 kB 00:00 2024-04-16T16:10:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 2.3 MB/s | 1.0 MB 00:00 2024-04-16T16:10:53.267 INFO:teuthology.orchestra.run.smithi045.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 146 kB/s | 18 kB 00:00 2024-04-16T16:10:53.313 INFO:teuthology.orchestra.run.smithi045.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 849 kB/s | 144 kB 00:00 2024-04-16T16:10:53.359 INFO:teuthology.orchestra.run.smithi045.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 714 kB/s | 32 kB 00:00 2024-04-16T16:10:53.501 INFO:teuthology.orchestra.run.smithi045.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2024-04-16T16:10:53.526 INFO:teuthology.orchestra.run.smithi045.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-04-16T16:10:53.542 INFO:teuthology.orchestra.run.smithi103.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-04-16T16:10:53.559 INFO:teuthology.orchestra.run.smithi103.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 67 kB/s | 24 kB 00:00 2024-04-16T16:10:53.568 INFO:teuthology.orchestra.run.smithi045.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 1.7 MB/s | 69 kB 00:00 2024-04-16T16:10:53.584 INFO:teuthology.orchestra.run.smithi103.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 56 kB/s | 18 kB 00:00 2024-04-16T16:10:53.593 INFO:teuthology.orchestra.run.smithi045.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 174 kB 00:00 2024-04-16T16:10:53.618 INFO:teuthology.orchestra.run.smithi103.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 144 kB 00:00 2024-04-16T16:10:53.619 INFO:teuthology.orchestra.run.smithi045.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 852 kB/s | 41 kB 00:00 2024-04-16T16:10:53.643 INFO:teuthology.orchestra.run.smithi103.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 384 kB/s | 32 kB 00:00 2024-04-16T16:10:53.653 INFO:teuthology.orchestra.run.smithi045.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-04-16T16:10:53.678 INFO:teuthology.orchestra.run.smithi045.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-04-16T16:10:53.711 INFO:teuthology.orchestra.run.smithi045.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.7 MB/s | 211 kB 00:00 2024-04-16T16:10:53.728 INFO:teuthology.orchestra.run.smithi045.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 47 kB 00:00 2024-04-16T16:10:53.754 INFO:teuthology.orchestra.run.smithi045.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 43 kB 00:00 2024-04-16T16:10:53.835 INFO:teuthology.orchestra.run.smithi103.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-04-16T16:10:53.860 INFO:teuthology.orchestra.run.smithi103.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 217 kB 00:00 2024-04-16T16:10:53.893 INFO:teuthology.orchestra.run.smithi103.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 753 kB/s | 174 kB 00:00 2024-04-16T16:10:53.913 INFO:teuthology.orchestra.run.smithi045.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 253 kB/s | 39 kB 00:00 2024-04-16T16:10:53.919 INFO:teuthology.orchestra.run.smithi103.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.2 MB/s | 69 kB 00:00 2024-04-16T16:10:53.936 INFO:teuthology.orchestra.run.smithi103.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 566 kB/s | 41 kB 00:00 2024-04-16T16:10:53.969 INFO:teuthology.orchestra.run.smithi103.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-04-16T16:10:54.003 INFO:teuthology.orchestra.run.smithi103.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-04-16T16:10:54.028 INFO:teuthology.orchestra.run.smithi103.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 211 kB 00:00 2024-04-16T16:10:54.053 INFO:teuthology.orchestra.run.smithi103.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 570 kB/s | 47 kB 00:00 2024-04-16T16:10:54.179 INFO:teuthology.orchestra.run.smithi103.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 287 kB/s | 43 kB 00:00 2024-04-16T16:10:54.196 INFO:teuthology.orchestra.run.smithi045.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-04-16T16:10:54.213 INFO:teuthology.orchestra.run.smithi045.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 143 kB/s | 41 kB 00:00 2024-04-16T16:10:54.246 INFO:teuthology.orchestra.run.smithi045.stdout:(81/174): qatlib-service-24.02.0-1.el9.x86_64.r 1.1 MB/s | 35 kB 00:00 2024-04-16T16:10:54.280 INFO:teuthology.orchestra.run.smithi045.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 3.3 MB/s | 220 kB 00:00 2024-04-16T16:10:54.296 INFO:teuthology.orchestra.run.smithi103.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 175 kB/s | 39 kB 00:00 2024-04-16T16:10:54.297 INFO:teuthology.orchestra.run.smithi045.stdout:(83/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 972 kB/s | 46 kB 00:00 2024-04-16T16:10:54.347 INFO:teuthology.orchestra.run.smithi045.stdout:(84/174): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-04-16T16:10:54.364 INFO:teuthology.orchestra.run.smithi045.stdout:(85/174): uuid-1.6.2-55.el9.x86_64.rpm 852 kB/s | 56 kB 00:00 2024-04-16T16:10:54.381 INFO:teuthology.orchestra.run.smithi045.stdout:(86/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-04-16T16:10:54.398 INFO:teuthology.orchestra.run.smithi045.stdout:(87/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-04-16T16:10:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:(88/174): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-04-16T16:10:54.457 INFO:teuthology.orchestra.run.smithi045.stdout:(89/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 241 kB 00:00 2024-04-16T16:10:54.516 INFO:teuthology.orchestra.run.smithi045.stdout:(90/174): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.1 MB 00:00 2024-04-16T16:10:54.532 INFO:teuthology.orchestra.run.smithi045.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2024-04-16T16:10:54.563 INFO:teuthology.orchestra.run.smithi103.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-04-16T16:10:54.566 INFO:teuthology.orchestra.run.smithi045.stdout:(92/174): python3-pycparser-2.20-6.el9.noarch.r 2.5 MB/s | 124 kB 00:00 2024-04-16T16:10:54.588 INFO:teuthology.orchestra.run.smithi103.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 104 kB/s | 41 kB 00:00 2024-04-16T16:10:54.600 INFO:teuthology.orchestra.run.smithi045.stdout:(93/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 187 kB 00:00 2024-04-16T16:10:54.613 INFO:teuthology.orchestra.run.smithi103.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 735 kB/s | 220 kB 00:00 2024-04-16T16:10:54.655 INFO:teuthology.orchestra.run.smithi103.stdout:(83/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-04-16T16:10:54.688 INFO:teuthology.orchestra.run.smithi103.stdout:(84/174): socat-1.7.4.1-5.el9.x86_64.rpm 4.0 MB/s | 300 kB 00:00 2024-04-16T16:10:54.714 INFO:teuthology.orchestra.run.smithi103.stdout:(85/174): qatlib-service-24.02.0-1.el9.x86_64.r 351 kB/s | 35 kB 00:00 2024-04-16T16:10:54.725 INFO:teuthology.orchestra.run.smithi045.stdout:(94/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-04-16T16:10:54.739 INFO:teuthology.orchestra.run.smithi103.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 700 kB/s | 56 kB 00:00 2024-04-16T16:10:54.773 INFO:teuthology.orchestra.run.smithi103.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 760 kB/s | 63 kB 00:00 2024-04-16T16:10:54.775 INFO:teuthology.orchestra.run.smithi045.stdout:(95/174): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-04-16T16:10:54.798 INFO:teuthology.orchestra.run.smithi103.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 774 kB/s | 44 kB 00:00 2024-04-16T16:10:54.817 INFO:teuthology.orchestra.run.smithi045.stdout:(96/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-04-16T16:10:54.823 INFO:teuthology.orchestra.run.smithi103.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 960 kB/s | 79 kB 00:00 2024-04-16T16:10:54.851 INFO:teuthology.orchestra.run.smithi045.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-04-16T16:10:54.873 INFO:teuthology.orchestra.run.smithi103.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.4 MB/s | 241 kB 00:00 2024-04-16T16:10:54.907 INFO:teuthology.orchestra.run.smithi103.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-04-16T16:10:54.910 INFO:teuthology.orchestra.run.smithi045.stdout:(98/174): elfutils-libelf-devel-0.190-2.el9.x86 237 kB/s | 21 kB 00:00 2024-04-16T16:10:54.935 INFO:teuthology.orchestra.run.smithi045.stdout:(99/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 307 kB/s | 25 kB 00:00 2024-04-16T16:10:54.966 INFO:teuthology.orchestra.run.smithi103.stdout:(92/174): python3-cryptography-36.0.1-4.el9.x86 8.2 MB/s | 1.1 MB 00:00 2024-04-16T16:10:54.991 INFO:teuthology.orchestra.run.smithi103.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 124 kB 00:00 2024-04-16T16:10:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 1.8 MB/s | 187 kB 00:00 2024-04-16T16:10:55.085 INFO:teuthology.orchestra.run.smithi045.stdout:(100/174): python3-devel-3.9.17-2.el9.x86_64.rp 1.6 MB/s | 245 kB 00:00 2024-04-16T16:10:55.167 INFO:teuthology.orchestra.run.smithi103.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-04-16T16:10:55.177 INFO:teuthology.orchestra.run.smithi045.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 2.7 MB/s | 248 kB 00:00 2024-04-16T16:10:55.191 INFO:teuthology.orchestra.run.smithi103.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-16T16:10:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-16T16:10:55.253 INFO:teuthology.orchestra.run.smithi045.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-16T16:10:55.284 INFO:teuthology.orchestra.run.smithi103.stdout:(98/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.7 MB/s | 803 kB 00:00 2024-04-16T16:10:55.309 INFO:teuthology.orchestra.run.smithi103.stdout:(99/174): elfutils-libelf-devel-0.190-2.el9.x86 185 kB/s | 21 kB 00:00 2024-04-16T16:10:55.334 INFO:teuthology.orchestra.run.smithi103.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 235 kB/s | 25 kB 00:00 2024-04-16T16:10:55.353 INFO:teuthology.orchestra.run.smithi045.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 226 kB/s | 22 kB 00:00 2024-04-16T16:10:55.443 INFO:teuthology.orchestra.run.smithi103.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 1.8 MB/s | 245 kB 00:00 2024-04-16T16:10:55.485 INFO:teuthology.orchestra.run.smithi103.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.2 MB/s | 94 kB 00:00 2024-04-16T16:10:55.495 INFO:teuthology.orchestra.run.smithi045.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 6.0 MB/s | 863 kB 00:00 2024-04-16T16:10:55.519 INFO:teuthology.orchestra.run.smithi103.stdout:(103/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.3 MB/s | 248 kB 00:00 2024-04-16T16:10:55.544 INFO:teuthology.orchestra.run.smithi103.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 386 kB/s | 22 kB 00:00 2024-04-16T16:10:55.728 INFO:teuthology.orchestra.run.smithi103.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 9.3 MB/s | 4.1 MB 00:00 2024-04-16T16:10:55.778 INFO:teuthology.orchestra.run.smithi103.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.3 MB/s | 863 kB 00:00 2024-04-16T16:10:55.886 INFO:teuthology.orchestra.run.smithi103.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 181 kB/s | 19 kB 00:00 2024-04-16T16:10:55.920 INFO:teuthology.orchestra.run.smithi103.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.6 MB/s | 309 kB 00:00 2024-04-16T16:10:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-04-16T16:10:56.029 INFO:teuthology.orchestra.run.smithi103.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 243 kB/s | 26 kB 00:00 2024-04-16T16:10:56.088 INFO:teuthology.orchestra.run.smithi103.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 415 kB/s | 49 kB 00:00 2024-04-16T16:10:56.196 INFO:teuthology.orchestra.run.smithi103.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-16T16:10:56.220 INFO:teuthology.orchestra.run.smithi045.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.2 MB/s | 4.1 MB 00:01 2024-04-16T16:10:56.221 INFO:teuthology.orchestra.run.smithi103.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 375 kB/s | 67 kB 00:00 2024-04-16T16:10:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 995 kB/s | 151 kB 00:00 2024-04-16T16:10:56.254 INFO:teuthology.orchestra.run.smithi045.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 724 kB/s | 548 kB 00:00 2024-04-16T16:10:56.330 INFO:teuthology.orchestra.run.smithi103.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.3 MB/s | 838 kB 00:00 2024-04-16T16:10:56.354 INFO:teuthology.orchestra.run.smithi045.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 196 kB/s | 19 kB 00:00 2024-04-16T16:10:56.381 INFO:teuthology.orchestra.run.smithi103.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-04-16T16:10:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 292 kB/s | 43 kB 00:00 2024-04-16T16:10:56.423 INFO:teuthology.orchestra.run.smithi103.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 376 kB/s | 32 kB 00:00 2024-04-16T16:10:56.440 INFO:teuthology.orchestra.run.smithi103.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 371 kB/s | 15 kB 00:00 2024-04-16T16:10:56.465 INFO:teuthology.orchestra.run.smithi103.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.6 MB/s | 172 kB 00:00 2024-04-16T16:10:56.498 INFO:teuthology.orchestra.run.smithi103.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-16T16:10:56.524 INFO:teuthology.orchestra.run.smithi103.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-04-16T16:10:56.549 INFO:teuthology.orchestra.run.smithi103.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 467 kB/s | 35 kB 00:00 2024-04-16T16:10:56.674 INFO:teuthology.orchestra.run.smithi103.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.3 MB/s | 203 kB 00:00 2024-04-16T16:10:56.712 INFO:teuthology.orchestra.run.smithi045.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 629 kB/s | 309 kB 00:00 2024-04-16T16:10:56.779 INFO:teuthology.orchestra.run.smithi103.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 8.0 MB/s | 2.0 MB 00:00 2024-04-16T16:10:56.804 INFO:teuthology.orchestra.run.smithi103.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 567 kB/s | 144 kB 00:00 2024-04-16T16:10:56.821 INFO:teuthology.orchestra.run.smithi045.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 241 kB/s | 26 kB 00:00 2024-04-16T16:10:56.830 INFO:teuthology.orchestra.run.smithi103.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 69 kB/s | 11 kB 00:00 2024-04-16T16:10:56.855 INFO:teuthology.orchestra.run.smithi103.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 242 kB/s | 18 kB 00:00 2024-04-16T16:10:56.880 INFO:teuthology.orchestra.run.smithi103.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 473 kB/s | 23 kB 00:00 2024-04-16T16:10:56.906 INFO:teuthology.orchestra.run.smithi103.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 262 kB/s | 19 kB 00:00 2024-04-16T16:10:56.931 INFO:teuthology.orchestra.run.smithi103.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-04-16T16:10:56.938 INFO:teuthology.orchestra.run.smithi045.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 421 kB/s | 49 kB 00:00 2024-04-16T16:10:57.006 INFO:teuthology.orchestra.run.smithi103.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 8.2 MB/s | 1.0 MB 00:00 2024-04-16T16:10:57.023 INFO:teuthology.orchestra.run.smithi103.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 512 kB/s | 46 kB 00:00 2024-04-16T16:10:57.039 INFO:teuthology.orchestra.run.smithi045.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 679 kB/s | 67 kB 00:00 2024-04-16T16:10:57.048 INFO:teuthology.orchestra.run.smithi103.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 684 kB/s | 79 kB 00:00 2024-04-16T16:10:57.066 INFO:teuthology.orchestra.run.smithi103.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 1.0 MB/s | 58 kB 00:00 2024-04-16T16:10:57.164 INFO:teuthology.orchestra.run.smithi045.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-16T16:10:57.167 INFO:teuthology.orchestra.run.smithi103.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-16T16:10:57.191 INFO:teuthology.orchestra.run.smithi103.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 117 kB/s | 16 kB 00:00 2024-04-16T16:10:57.216 INFO:teuthology.orchestra.run.smithi103.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 600 kB/s | 90 kB 00:00 2024-04-16T16:10:57.242 INFO:teuthology.orchestra.run.smithi103.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 388 kB/s | 28 kB 00:00 2024-04-16T16:10:57.267 INFO:teuthology.orchestra.run.smithi103.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 629 kB/s | 31 kB 00:00 2024-04-16T16:10:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-04-16T16:10:57.326 INFO:teuthology.orchestra.run.smithi103.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 736 kB/s | 59 kB 00:00 2024-04-16T16:10:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 626 kB/s | 36 kB 00:00 2024-04-16T16:10:57.385 INFO:teuthology.orchestra.run.smithi103.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 620 kB/s | 51 kB 00:00 2024-04-16T16:10:57.418 INFO:teuthology.orchestra.run.smithi103.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.5 MB/s | 230 kB 00:00 2024-04-16T16:10:57.456 INFO:teuthology.orchestra.run.smithi045.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.8 MB/s | 838 kB 00:00 2024-04-16T16:10:57.460 INFO:teuthology.orchestra.run.smithi103.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 925 kB/s | 90 kB 00:00 2024-04-16T16:10:57.502 INFO:teuthology.orchestra.run.smithi103.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-04-16T16:10:57.527 INFO:teuthology.orchestra.run.smithi103.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 185 kB/s | 20 kB 00:00 2024-04-16T16:10:57.569 INFO:teuthology.orchestra.run.smithi103.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.8 MB/s | 192 kB 00:00 2024-04-16T16:10:57.614 INFO:teuthology.orchestra.run.smithi045.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-04-16T16:10:57.636 INFO:teuthology.orchestra.run.smithi103.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-16T16:10:57.723 INFO:teuthology.orchestra.run.smithi045.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 405 kB/s | 43 kB 00:00 2024-04-16T16:10:57.778 INFO:teuthology.orchestra.run.smithi103.stdout:(151/174): NetworkManager-1.47.4-1.el9.x86_64.r 16 MB/s | 2.3 MB 00:00 2024-04-16T16:10:57.870 INFO:teuthology.orchestra.run.smithi103.stdout:(152/174): NetworkManager-libnm-1.47.4-1.el9.x8 20 MB/s | 1.8 MB 00:00 2024-04-16T16:10:57.873 INFO:teuthology.orchestra.run.smithi045.stdout:(116/174): libarrow-9.0.0-10.el9.x86_64.rpm 2.9 MB/s | 4.4 MB 00:01 2024-04-16T16:10:57.898 INFO:teuthology.orchestra.run.smithi045.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 184 kB/s | 32 kB 00:00 2024-04-16T16:10:57.920 INFO:teuthology.orchestra.run.smithi103.stdout:(153/174): NetworkManager-team-1.47.4-1.el9.x86 831 kB/s | 41 kB 00:00 2024-04-16T16:10:57.962 INFO:teuthology.orchestra.run.smithi103.stdout:(154/174): NetworkManager-tui-1.47.4-1.el9.x86_ 6.0 MB/s | 248 kB 00:00 2024-04-16T16:10:57.974 INFO:teuthology.orchestra.run.smithi045.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-04-16T16:10:58.016 INFO:teuthology.orchestra.run.smithi045.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-04-16T16:10:58.062 INFO:teuthology.orchestra.run.smithi103.stdout:(155/174): librados2-19.0.0-2868.gc3c87ed0.el9. 6.4 MB/s | 3.4 MB 00:00 2024-04-16T16:10:58.105 INFO:teuthology.orchestra.run.smithi103.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.2 MB/s | 321 kB 00:00 2024-04-16T16:10:58.149 INFO:teuthology.orchestra.run.smithi045.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 2.0 MB/s | 358 kB 00:00 2024-04-16T16:10:58.155 INFO:teuthology.orchestra.run.smithi103.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 809 kB/s | 71 kB 00:00 2024-04-16T16:10:58.180 INFO:teuthology.orchestra.run.smithi103.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 490 kB/s | 36 kB 00:00 2024-04-16T16:10:58.199 INFO:teuthology.orchestra.run.smithi045.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.6 MB/s | 303 kB 00:00 2024-04-16T16:10:58.266 INFO:teuthology.orchestra.run.smithi045.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 304 kB/s | 35 kB 00:00 2024-04-16T16:10:58.272 INFO:teuthology.orchestra.run.smithi103.stdout:(159/174): librbd1-19.0.0-2868.gc3c87ed0.el9.x8 4.5 MB/s | 3.1 MB 00:00 2024-04-16T16:10:58.322 INFO:teuthology.orchestra.run.smithi103.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.1 MB/s | 192 kB 00:00 2024-04-16T16:10:58.347 INFO:teuthology.orchestra.run.smithi103.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 309 kB/s | 23 kB 00:00 2024-04-16T16:10:58.381 INFO:teuthology.orchestra.run.smithi103.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.2 MB/s | 256 kB 00:00 2024-04-16T16:10:58.415 INFO:teuthology.orchestra.run.smithi103.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 108 kB 00:00 2024-04-16T16:10:58.449 INFO:teuthology.orchestra.run.smithi103.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 136 kB 00:00 2024-04-16T16:10:58.474 INFO:teuthology.orchestra.run.smithi103.stdout:(165/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 155 kB 00:00 2024-04-16T16:10:58.529 INFO:teuthology.orchestra.run.smithi045.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 617 kB/s | 203 kB 00:00 2024-04-16T16:10:58.659 INFO:teuthology.orchestra.run.smithi045.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-16T16:10:58.691 INFO:teuthology.orchestra.run.smithi103.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 232 kB/s | 64 kB 00:00 2024-04-16T16:10:58.725 INFO:teuthology.orchestra.run.smithi103.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 104 kB/s | 29 kB 00:00 2024-04-16T16:10:58.759 INFO:teuthology.orchestra.run.smithi045.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 108 kB/s | 11 kB 00:00 2024-04-16T16:10:58.818 INFO:teuthology.orchestra.run.smithi045.stdout:(126/174): python3-grpcio-1.46.7-10.el9.x86_64. 3.7 MB/s | 2.0 MB 00:00 2024-04-16T16:10:58.860 INFO:teuthology.orchestra.run.smithi045.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 178 kB/s | 18 kB 00:00 2024-04-16T16:10:58.900 INFO:teuthology.orchestra.run.smithi103.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 10 MB/s | 2.2 MB 00:00 2024-04-16T16:10:58.927 INFO:teuthology.orchestra.run.smithi045.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-04-16T16:10:58.942 INFO:teuthology.orchestra.run.smithi103.stdout:(169/174): openssl-3.0.7-27.el9.x86_64.rpm 4.9 MB/s | 1.2 MB 00:00 2024-04-16T16:10:58.969 INFO:teuthology.orchestra.run.smithi045.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 180 kB/s | 19 kB 00:00 2024-04-16T16:10:59.017 INFO:teuthology.orchestra.run.smithi103.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 7.8 MB/s | 2.3 MB 00:00 2024-04-16T16:10:59.028 INFO:teuthology.orchestra.run.smithi045.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 198 kB/s | 20 kB 00:00 2024-04-16T16:10:59.042 INFO:teuthology.orchestra.run.smithi103.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 311 kB/s | 31 kB 00:00 2024-04-16T16:10:59.076 INFO:teuthology.orchestra.run.smithi103.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 2.6 MB/s | 465 kB 00:00 2024-04-16T16:10:59.134 INFO:teuthology.orchestra.run.smithi103.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.2 MB/s | 1.1 MB 00:00 2024-04-16T16:10:59.159 INFO:teuthology.orchestra.run.smithi103.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 642 kB/s | 74 kB 00:00 2024-04-16T16:10:59.170 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-16T16:10:59.171 INFO:teuthology.orchestra.run.smithi103.stdout:Total 13 MB/s | 228 MB 00:18 2024-04-16T16:10:59.174 INFO:teuthology.orchestra.run.smithi045.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 5.0 MB/s | 1.0 MB 00:00 2024-04-16T16:10:59.200 INFO:teuthology.orchestra.run.smithi045.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 270 kB/s | 46 kB 00:00 2024-04-16T16:10:59.275 INFO:teuthology.orchestra.run.smithi045.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 792 kB/s | 79 kB 00:00 2024-04-16T16:10:59.301 INFO:teuthology.orchestra.run.smithi045.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 579 kB/s | 58 kB 00:00 2024-04-16T16:10:59.476 INFO:teuthology.orchestra.run.smithi045.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-04-16T16:10:59.527 INFO:teuthology.orchestra.run.smithi045.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 73 kB/s | 16 kB 00:00 2024-04-16T16:10:59.577 INFO:teuthology.orchestra.run.smithi045.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 904 kB/s | 90 kB 00:00 2024-04-16T16:10:59.636 INFO:teuthology.orchestra.run.smithi045.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 262 kB/s | 28 kB 00:00 2024-04-16T16:10:59.686 INFO:teuthology.orchestra.run.smithi045.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 285 kB/s | 31 kB 00:00 2024-04-16T16:10:59.744 INFO:teuthology.orchestra.run.smithi045.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-16T16:10:59.787 INFO:teuthology.orchestra.run.smithi045.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 594 kB/s | 59 kB 00:00 2024-04-16T16:10:59.845 INFO:teuthology.orchestra.run.smithi045.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 358 kB/s | 36 kB 00:00 2024-04-16T16:10:59.888 INFO:teuthology.orchestra.run.smithi045.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 511 kB/s | 51 kB 00:00 2024-04-16T16:11:00.004 INFO:teuthology.orchestra.run.smithi045.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-04-16T16:11:00.029 INFO:teuthology.orchestra.run.smithi045.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 638 kB/s | 90 kB 00:00 2024-04-16T16:11:00.138 INFO:teuthology.orchestra.run.smithi045.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 186 kB/s | 20 kB 00:00 2024-04-16T16:11:00.197 INFO:teuthology.orchestra.run.smithi045.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.2 MB/s | 427 kB 00:00 2024-04-16T16:11:00.272 INFO:teuthology.orchestra.run.smithi045.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-04-16T16:11:00.336 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-16T16:11:00.631 INFO:teuthology.orchestra.run.smithi045.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 3.7 MB/s | 1.6 MB 00:00 2024-04-16T16:11:00.718 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-16T16:11:00.718 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-16T16:11:01.039 INFO:teuthology.orchestra.run.smithi045.stdout:(150/174): librados2-19.0.0-2868.gc3c87ed0.el9. 4.4 MB/s | 3.4 MB 00:00 2024-04-16T16:11:01.181 INFO:teuthology.orchestra.run.smithi045.stdout:(151/174): NetworkManager-1.47.4-1.el9.x86_64.r 16 MB/s | 2.3 MB 00:00 2024-04-16T16:11:01.307 INFO:teuthology.orchestra.run.smithi045.stdout:(152/174): NetworkManager-libnm-1.47.4-1.el9.x8 15 MB/s | 1.8 MB 00:00 2024-04-16T16:11:01.348 INFO:teuthology.orchestra.run.smithi045.stdout:(153/174): NetworkManager-team-1.47.4-1.el9.x86 1.0 MB/s | 41 kB 00:00 2024-04-16T16:11:01.407 INFO:teuthology.orchestra.run.smithi045.stdout:(154/174): NetworkManager-tui-1.47.4-1.el9.x86_ 4.2 MB/s | 248 kB 00:00 2024-04-16T16:11:01.457 INFO:teuthology.orchestra.run.smithi045.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.4 MB/s | 321 kB 00:00 2024-04-16T16:11:01.608 INFO:teuthology.orchestra.run.smithi045.stdout:(156/174): librbd1-19.0.0-2868.gc3c87ed0.el9.x8 3.2 MB/s | 3.1 MB 00:00 2024-04-16T16:11:01.633 INFO:teuthology.orchestra.run.smithi045.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 409 kB/s | 71 kB 00:00 2024-04-16T16:11:01.675 INFO:teuthology.orchestra.run.smithi045.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 552 kB/s | 36 kB 00:00 2024-04-16T16:11:01.708 INFO:teuthology.orchestra.run.smithi045.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.5 MB/s | 192 kB 00:00 2024-04-16T16:11:01.759 INFO:teuthology.orchestra.run.smithi045.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 467 kB/s | 23 kB 00:00 2024-04-16T16:11:01.793 INFO:teuthology.orchestra.run.smithi045.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.1 MB/s | 256 kB 00:00 2024-04-16T16:11:01.826 INFO:teuthology.orchestra.run.smithi045.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-04-16T16:11:01.860 INFO:teuthology.orchestra.run.smithi045.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 155 kB 00:00 2024-04-16T16:11:01.893 INFO:teuthology.orchestra.run.smithi045.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 136 kB 00:00 2024-04-16T16:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-04-16T16:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 582 kB/s | 29 kB 00:00 2024-04-16T16:11:02.036 INFO:teuthology.orchestra.run.smithi045.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 10 MB/s | 1.2 MB 00:00 2024-04-16T16:11:02.120 INFO:teuthology.orchestra.run.smithi045.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 12 MB/s | 2.2 MB 00:00 2024-04-16T16:11:02.203 INFO:teuthology.orchestra.run.smithi045.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 14 MB/s | 2.3 MB 00:00 2024-04-16T16:11:02.237 INFO:teuthology.orchestra.run.smithi045.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 3.9 MB/s | 465 kB 00:00 2024-04-16T16:11:02.262 INFO:teuthology.orchestra.run.smithi045.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 541 kB/s | 31 kB 00:00 2024-04-16T16:11:02.320 INFO:teuthology.orchestra.run.smithi045.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-04-16T16:11:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 370 kB/s | 74 kB 00:00 2024-04-16T16:11:03.079 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-16T16:11:03.080 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-16T16:11:03.379 INFO:teuthology.orchestra.run.smithi045.stdout:(174/174): ceph-test-19.0.0-2868.gc3c87ed0.el9. 2.8 MB/s | 49 MB 00:17 2024-04-16T16:11:03.390 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T16:11:03.390 INFO:teuthology.orchestra.run.smithi045.stdout:Total 9.5 MB/s | 228 MB 00:23 2024-04-16T16:11:04.432 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T16:11:04.838 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T16:11:04.838 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T16:11:07.237 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T16:11:07.238 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T16:11:10.629 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-16T16:11:10.755 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-16T16:11:10.795 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-16T16:11:10.841 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-16T16:11:10.885 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-16T16:11:10.981 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-16T16:11:11.088 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-16T16:11:11.150 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-16T16:11:11.182 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-16T16:11:11.343 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-16T16:11:11.801 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-16T16:11:12.392 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 10/198 2024-04-16T16:11:12.393 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 10/198 2024-04-16T16:11:12.393 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 11/198 2024-04-16T16:11:12.393 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 11/198 2024-04-16T16:11:12.393 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:19.0.0-2868.gc3c87ed0.el9.x86_64 12/198 2024-04-16T16:11:12.393 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-16T16:11:12.582 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-16T16:11:12.646 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-16T16:11:12.671 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:19.0.0-2868.gc3c87ed0.el9.x86_64 16/198 2024-04-16T16:11:12.866 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2868.gc3c87ed0.el9.x86_64 16/198 2024-04-16T16:11:12.897 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:19.0.0-2868.gc3c87ed0.el9.x86 17/198 2024-04-16T16:11:12.995 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2868.gc3c87ed0.el9.x86 17/198 2024-04-16T16:11:13.041 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-16T16:11:13.065 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-16T16:11:13.137 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-16T16:11:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-16T16:11:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-16T16:11:13.575 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-16T16:11:13.897 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-16T16:11:14.107 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-16T16:11:14.195 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-16T16:11:14.398 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-16T16:11:14.501 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-16T16:11:14.890 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-16T16:11:14.956 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T16:11:15.081 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-16T16:11:15.091 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-16T16:11:15.132 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-16T16:11:15.169 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-16T16:11:15.216 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-16T16:11:15.224 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-16T16:11:15.266 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-16T16:11:15.303 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-16T16:11:15.312 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-16T16:11:15.364 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-16T16:11:15.417 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-16T16:11:15.478 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-16T16:11:15.491 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-16T16:11:15.510 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-16T16:11:15.558 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-16T16:11:15.589 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-16T16:11:15.624 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-16T16:11:15.710 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-16T16:11:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-16T16:11:15.826 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-16T16:11:15.868 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-16T16:11:15.901 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-16T16:11:16.014 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-16T16:11:16.022 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-16T16:11:16.055 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:19.0.0-2868.gc3c87ed0.el9. 39/198 2024-04-16T16:11:16.075 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 10/198 2024-04-16T16:11:16.121 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:19.0.0-2868.gc3c87ed0.el 40/198 2024-04-16T16:11:16.186 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:19.0.0-2868.gc3c87ed0.el9.x86_6 41/198 2024-04-16T16:11:16.246 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 10/198 2024-04-16T16:11:16.284 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 11/198 2024-04-16T16:11:16.313 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-16T16:11:16.384 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-16T16:11:16.411 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 11/198 2024-04-16T16:11:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:19.0.0-2868.gc3c87ed0.el9.x86_64 12/198 2024-04-16T16:11:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-16T16:11:16.507 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-16T16:11:16.689 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-16T16:11:16.791 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-16T16:11:16.805 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-16T16:11:16.864 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-16T16:11:16.889 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:19.0.0-2868.gc3c87ed0.el9.x86_64 16/198 2024-04-16T16:11:17.036 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2868.gc3c87ed0.el9.x86_64 16/198 2024-04-16T16:11:17.066 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:19.0.0-2868.gc3c87ed0.el9.x86 17/198 2024-04-16T16:11:17.148 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2868.gc3c87ed0.el9.x86 17/198 2024-04-16T16:11:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-16T16:11:17.226 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-16T16:11:17.290 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-16T16:11:17.508 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-16T16:11:17.580 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-16T16:11:17.598 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-16T16:11:17.636 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-16T16:11:17.677 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-16T16:11:17.728 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-16T16:11:17.799 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-16T16:11:17.830 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-16T16:11:17.872 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-16T16:11:17.986 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-16T16:11:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-16T16:11:18.168 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-16T16:11:18.256 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-16T16:11:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:19.0.0-2868.gc3c87ed0.el9.x86_6 53/198 2024-04-16T16:11:18.449 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-16T16:11:18.462 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-16T16:11:18.512 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-16T16:11:18.858 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-16T16:11:18.915 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-16T16:11:19.053 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-16T16:11:19.173 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-16T16:11:19.186 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-16T16:11:19.218 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-16T16:11:19.255 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-16T16:11:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-16T16:11:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-16T16:11:19.355 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-16T16:11:19.443 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-16T16:11:19.513 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-16T16:11:19.557 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-16T16:11:19.562 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-16T16:11:19.610 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-16T16:11:19.665 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-16T16:11:19.683 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-16T16:11:19.707 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-16T16:11:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-16T16:11:19.778 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-16T16:11:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-16T16:11:19.826 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-16T16:11:19.844 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-16T16:11:19.856 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-16T16:11:19.924 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-16T16:11:19.960 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-16T16:11:19.965 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-16T16:11:20.016 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:19.0.0-2868.gc3c87ed0.el9. 39/198 2024-04-16T16:11:20.027 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-16T16:11:20.069 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:19.0.0-2868.gc3c87ed0.el 40/198 2024-04-16T16:11:20.088 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-16T16:11:20.106 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:19.0.0-2868.gc3c87ed0.el9.x86_6 41/198 2024-04-16T16:11:20.232 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-16T16:11:20.280 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-16T16:11:20.295 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-16T16:11:20.340 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-16T16:11:20.368 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-16T16:11:20.411 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-16T16:11:20.425 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-16T16:11:20.557 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-16T16:11:20.594 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-16T16:11:20.600 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-16T16:11:20.646 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-16T16:11:20.671 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-16T16:11:20.683 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-16T16:11:20.717 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-16T16:11:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-16T16:11:21.062 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-16T16:11:21.208 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-16T16:11:21.261 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-16T16:11:21.323 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-16T16:11:21.397 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-16T16:11:21.406 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-16T16:11:21.505 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-16T16:11:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-16T16:11:21.656 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-16T16:11:21.668 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-16T16:11:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-16T16:11:21.731 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-16T16:11:21.748 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-16T16:11:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-16T16:11:21.860 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:19.0.0-2868.gc3c87ed0.el9.x86_6 53/198 2024-04-16T16:11:21.902 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-16T16:11:21.910 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-16T16:11:21.953 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-16T16:11:22.027 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-16T16:11:22.066 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-16T16:11:22.108 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-16T16:11:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-16T16:11:22.221 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-16T16:11:22.278 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-16T16:11:22.316 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-16T16:11:22.327 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-16T16:11:22.455 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-16T16:11:22.497 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-16T16:11:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-16T16:11:22.534 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-16T16:11:22.612 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-16T16:11:22.641 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-16T16:11:22.654 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-16T16:11:22.709 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-16T16:11:22.800 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-16T16:11:22.922 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-16T16:11:22.929 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-16T16:11:22.982 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-16T16:11:22.987 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-16T16:11:23.018 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-16T16:11:23.068 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-16T16:11:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-16T16:11:23.170 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-16T16:11:23.213 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-16T16:11:23.221 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-16T16:11:23.263 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-16T16:11:23.285 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-16T16:11:23.334 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-16T16:11:23.379 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-16T16:11:23.443 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-16T16:11:23.508 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-16T16:11:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-16T16:11:23.775 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-16T16:11:23.813 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-16T16:11:23.873 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-16T16:11:24.017 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-16T16:11:24.062 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-16T16:11:24.136 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-16T16:11:24.170 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-16T16:11:24.202 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-16T16:11:24.223 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-16T16:11:24.276 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-16T16:11:24.326 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-16T16:11:24.375 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-16T16:11:24.398 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-16T16:11:24.487 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-16T16:11:24.535 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-16T16:11:24.583 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-16T16:11:24.726 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-16T16:11:24.774 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-16T16:11:24.780 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-16T16:11:24.815 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-16T16:11:24.845 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-16T16:11:24.911 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-16T16:11:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-16T16:11:25.190 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-16T16:11:25.252 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-16T16:11:25.443 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-16T16:11:25.501 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-16T16:11:25.546 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-16T16:11:25.585 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-16T16:11:25.627 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-16T16:11:25.673 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-16T16:11:25.723 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-16T16:11:25.742 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-16T16:11:25.801 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-16T16:11:25.844 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-16T16:11:25.847 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-16T16:11:25.980 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-16T16:11:26.026 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-16T16:11:26.069 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-16T16:11:26.137 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-16T16:11:26.181 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-16T16:11:26.332 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-16T16:11:26.455 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-16T16:11:26.511 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-16T16:11:26.545 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-16T16:11:26.603 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-16T16:11:26.746 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-16T16:11:26.821 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-16T16:11:27.393 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-16T16:11:27.480 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-16T16:11:27.517 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-16T16:11:27.548 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-16T16:11:27.587 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-16T16:11:27.851 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-16T16:11:27.903 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-16T16:11:27.936 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-16T16:11:27.969 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-16T16:11:27.994 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-16T16:11:28.004 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:19.0.0-2868.gc3c87ed0.el9.x86_64 124/198 2024-04-16T16:11:28.110 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-16T16:11:28.125 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:19.0.0-2868.gc3c87ed0.el9.x86_64 124/198 2024-04-16T16:11:28.151 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 125/198 2024-04-16T16:11:28.154 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-16T16:11:28.206 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-16T16:11:28.428 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-16T16:11:28.459 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-16T16:11:28.556 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-16T16:11:28.742 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-16T16:11:28.774 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 128/198 2024-04-16T16:11:28.865 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 128/198 2024-04-16T16:11:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:19.0.0-2868.gc3c87ed0.el9.x86_64 129/198 2024-04-16T16:11:29.402 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-16T16:11:29.531 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-16T16:11:31.578 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-2868.gc3c87ed0.el9.x86_64 130/198 2024-04-16T16:11:31.590 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-16T16:11:31.623 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:19.0.0-2868.gc3c87ed0.el9.x86_64 130/198 2024-04-16T16:11:31.691 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-16T16:11:31.728 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-16T16:11:31.759 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-16T16:11:31.799 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-16T16:11:32.204 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-16T16:11:32.230 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:19.0.0-2868.gc3c87ed0.el9.x86_64 124/198 2024-04-16T16:11:32.272 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-2868.gc3c87ed0.el9.x86_64 130/198 2024-04-16T16:11:32.310 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:19.0.0-2868.gc3c87ed0.el9.x86_64 131/198 2024-04-16T16:11:32.361 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:19.0.0-2868.gc3c87ed0.el9.x86_64 124/198 2024-04-16T16:11:32.395 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 125/198 2024-04-16T16:11:32.452 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-16T16:11:32.484 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:19.0.0-2868.gc3c87ed0.el9.x86_64 131/198 2024-04-16T16:11:32.485 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T16:11:32.485 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T16:11:32.485 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:11:32.527 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:19.0.0-2868.gc3c87ed0.el9.x86_64 132/198 2024-04-16T16:11:33.014 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-16T16:11:33.042 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 128/198 2024-04-16T16:11:33.125 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 128/198 2024-04-16T16:11:33.161 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:19.0.0-2868.gc3c87ed0.el9.x86_64 129/198 2024-04-16T16:11:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.0.0-2868.gc3c87ed0.el9.x86_64 130/198 2024-04-16T16:11:35.958 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:19.0.0-2868.gc3c87ed0.el9.x86_64 130/198 2024-04-16T16:11:36.630 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.0.0-2868.gc3c87ed0.el9.x86_64 130/198 2024-04-16T16:11:36.670 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:19.0.0-2868.gc3c87ed0.el9.x86_64 131/198 2024-04-16T16:11:36.829 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:19.0.0-2868.gc3c87ed0.el9.x86_64 131/198 2024-04-16T16:11:36.829 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T16:11:36.829 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T16:11:36.829 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:11:36.871 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:19.0.0-2868.gc3c87ed0.el9.x86_64 132/198 2024-04-16T16:11:45.583 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2868.gc3c87ed0.el9.x86_64 132/198 2024-04-16T16:11:45.583 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-04-16T16:11:45.584 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-04-16T16:11:45.584 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-04-16T16:11:45.584 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-04-16T16:11:45.584 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-04-16T16:11:45.584 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-04-16T16:11:45.584 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-04-16T16:11:45.585 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:11:45.813 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-volume-2:19.0.0-2868.gc3c87ed0.el9.noarch 133/198 2024-04-16T16:11:45.860 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-volume-2:19.0.0-2868.gc3c87ed0.el9.noarch 133/198 2024-04-16T16:11:45.860 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:45.860 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T16:11:45.860 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:11:47.543 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 134/198 2024-04-16T16:11:47.651 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 134/198 2024-04-16T16:11:47.651 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:47.651 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T16:11:47.651 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T16:11:47.651 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T16:11:47.651 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:11:47.918 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:19.0.0-2868.gc3c87ed0.el9.x86_64 135/198 2024-04-16T16:11:47.986 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:19.0.0-2868.gc3c87ed0.el9.x86_64 135/198 2024-04-16T16:11:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T16:11:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T16:11:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T16:11:47.986 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:11:48.476 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:19.0.0-2868.gc3c87ed0.el9.x86_64 136/198 2024-04-16T16:11:48.547 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:19.0.0-2868.gc3c87ed0.el9.x86_64 136/198 2024-04-16T16:11:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T16:11:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T16:11:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T16:11:48.548 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:11:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-16T16:11:49.336 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.0.0-2868.gc3c87ed0.el9.noarch 138/198 2024-04-16T16:11:49.373 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:19.0.0-2868.gc3c87ed0.el9.noarch 138/198 2024-04-16T16:11:49.424 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2868.gc3c87ed0.e 139/198 2024-04-16T16:11:49.524 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2868.gc3c87ed0. 140/198 2024-04-16T16:11:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2868.gc3c87ed0.el9.noa 141/198 2024-04-16T16:11:50.713 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2868.gc3c87ed0.el9.x86_64 132/198 2024-04-16T16:11:50.713 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2024-04-16T16:11:50.713 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2024-04-16T16:11:50.713 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2024-04-16T16:11:50.713 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2024-04-16T16:11:50.714 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2024-04-16T16:11:50.714 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /root 2024-04-16T16:11:50.714 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2024-04-16T16:11:50.714 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:11:50.882 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2868.gc3c87ed0.el9.noa 141/198 2024-04-16T16:11:50.940 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-volume-2:19.0.0-2868.gc3c87ed0.el9.noarch 133/198 2024-04-16T16:11:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-volume-2:19.0.0-2868.gc3c87ed0.el9.noarch 133/198 2024-04-16T16:11:50.992 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:50.992 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T16:11:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:11:51.120 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2868.gc3c87ed0.el9.n 142/198 2024-04-16T16:11:52.252 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2868.gc3c87ed0.el9.n 142/198 2024-04-16T16:11:52.278 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2868.gc3c 143/198 2024-04-16T16:11:52.423 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2868.gc3c 143/198 2024-04-16T16:11:52.606 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2868.gc3c87ed0.el 144/198 2024-04-16T16:11:52.639 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:19.0.0-2868.gc3c87ed0.el9.x86_64 145/198 2024-04-16T16:11:52.684 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 134/198 2024-04-16T16:11:52.708 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2868.gc3c87ed0.el9.x86_64 145/198 2024-04-16T16:11:52.708 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:52.708 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T16:11:52.708 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T16:11:52.708 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T16:11:52.708 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:11:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 134/198 2024-04-16T16:11:52.769 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:52.769 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T16:11:52.769 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T16:11:52.770 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T16:11:52.770 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:11:52.773 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:19.0.0-2868.gc3c87ed0.el9.noarch 146/198 2024-04-16T16:11:52.812 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2868.gc3c87ed0.el9.noarch 146/198 2024-04-16T16:11:52.903 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:19.0.0-2868.gc3c87ed0.el9.x86_64 147/198 2024-04-16T16:11:53.028 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:19.0.0-2868.gc3c87ed0.el9.x86_64 135/198 2024-04-16T16:11:53.117 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:19.0.0-2868.gc3c87ed0.el9.x86_64 135/198 2024-04-16T16:11:53.117 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:53.117 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T16:11:53.117 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T16:11:53.117 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T16:11:53.117 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:11:53.629 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:19.0.0-2868.gc3c87ed0.el9.x86_64 136/198 2024-04-16T16:11:53.700 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:19.0.0-2868.gc3c87ed0.el9.x86_64 136/198 2024-04-16T16:11:53.700 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:53.701 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T16:11:53.701 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T16:11:53.701 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T16:11:53.701 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:11:53.814 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-16T16:11:53.902 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-16T16:11:53.934 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:19.0.0-2868.gc3c87ed0.el9.x86_64 149/198 2024-04-16T16:11:54.028 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2868.gc3c87ed0.el9.x86_64 149/198 2024-04-16T16:11:54.029 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:54.029 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T16:11:54.029 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T16:11:54.029 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T16:11:54.029 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:11:54.084 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2868.gc3c87 150/198 2024-04-16T16:11:54.179 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2868.gc3c87 150/198 2024-04-16T16:11:54.179 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:54.179 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T16:11:54.179 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:11:54.269 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 151/198 2024-04-16T16:11:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 151/198 2024-04-16T16:11:54.374 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:54.374 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-16T16:11:54.374 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-16T16:11:54.374 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-16T16:11:54.374 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:11:54.397 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:19.0.0-2868.gc3c87ed0.el9.noarch 138/198 2024-04-16T16:11:54.458 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:19.0.0-2868.gc3c87ed0.el9.noarch 138/198 2024-04-16T16:11:54.511 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2868.gc3c87ed0.e 139/198 2024-04-16T16:11:54.606 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2868.gc3c87ed0. 140/198 2024-04-16T16:11:54.645 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2868.gc3c87ed0.el9.noa 141/198 2024-04-16T16:11:54.752 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 152/198 2024-04-16T16:11:54.820 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 152/198 2024-04-16T16:11:54.820 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:54.820 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T16:11:54.820 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T16:11:54.821 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T16:11:54.821 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:11:55.998 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2868.gc3c87ed0.el9.noa 141/198 2024-04-16T16:11:56.038 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2868.gc3c87ed0.el9.n 142/198 2024-04-16T16:11:57.198 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2868.gc3c87ed0.el9.n 142/198 2024-04-16T16:11:57.258 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2868.gc3c 143/198 2024-04-16T16:11:57.420 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2868.gc3c 143/198 2024-04-16T16:11:57.595 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2868.gc3c87ed0.el 144/198 2024-04-16T16:11:57.624 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:19.0.0-2868.gc3c87ed0.el9.x86_64 145/198 2024-04-16T16:11:57.697 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2868.gc3c87ed0.el9.x86_64 145/198 2024-04-16T16:11:57.697 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:57.697 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T16:11:57.697 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T16:11:57.697 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T16:11:57.698 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:11:57.750 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:19.0.0-2868.gc3c87ed0.el9.noarch 146/198 2024-04-16T16:11:57.789 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2868.gc3c87ed0.el9.noarch 146/198 2024-04-16T16:11:57.853 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:19.0.0-2868.gc3c87ed0.el9.x86_64 147/198 2024-04-16T16:11:58.898 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-16T16:11:58.935 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:19.0.0-2868.gc3c87ed0.el9.x86_64 149/198 2024-04-16T16:11:59.008 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2868.gc3c87ed0.el9.x86_64 149/198 2024-04-16T16:11:59.008 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:59.008 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T16:11:59.008 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T16:11:59.008 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T16:11:59.008 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:11:59.061 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2868.gc3c87 150/198 2024-04-16T16:11:59.151 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2868.gc3c87 150/198 2024-04-16T16:11:59.151 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:59.152 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T16:11:59.152 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:11:59.254 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 151/198 2024-04-16T16:11:59.335 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 151/198 2024-04-16T16:11:59.335 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:59.335 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-16T16:11:59.335 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-16T16:11:59.335 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-16T16:11:59.335 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:11:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 152/198 2024-04-16T16:11:59.788 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 152/198 2024-04-16T16:11:59.788 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T16:11:59.788 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T16:11:59.788 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T16:11:59.789 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T16:11:59.789 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:11:59.852 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:19.0.0-2868.gc3c87ed0.el9.x86_64 153/198 2024-04-16T16:11:59.908 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:19.0.0-2868.gc3c87ed0.el9.x86_64 154/198 2024-04-16T16:11:59.950 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 155/198 2024-04-16T16:11:59.989 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-16T16:12:00.252 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-shell-2:19.0.0-2868.gc3c87ed0.el9.x86_64 157/198 2024-04-16T16:12:00.381 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-16T16:12:00.435 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:19.0.0-2868.gc3c87ed0.el9.x86_64 159/198 2024-04-16T16:12:00.488 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-16T16:12:00.551 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-16T16:12:00.596 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-16T16:12:00.630 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:19.0.0-2868.gc3c87ed0.el9.x86_ 163/198 2024-04-16T16:12:00.693 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-16T16:12:00.734 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-16T16:12:00.791 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-16T16:12:00.864 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-16T16:12:00.901 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-16T16:12:01.811 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-16T16:12:01.858 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-16T16:12:01.927 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-16T16:12:03.291 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-top-2:19.0.0-2868.gc3c87ed0.el9.noarch 170/198 2024-04-16T16:12:03.390 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-16T16:12:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-16T16:12:03.574 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-16T16:12:03.609 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-16T16:12:03.643 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-16T16:12:03.677 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-16T16:12:03.715 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-16T16:12:03.715 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-16T16:12:03.848 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-16T16:12:03.889 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-16T16:12:04.486 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-16T16:12:04.488 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-16T16:12:04.488 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-16T16:12:04.488 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-16T16:12:04.488 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-16T16:12:04.488 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-16T16:12:04.488 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-16T16:12:04.488 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-16T16:12:04.488 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-16T16:12:04.488 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-16T16:12:04.497 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-16T16:12:04.531 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-16T16:12:04.564 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-16T16:12:04.600 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-16T16:12:04.600 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-16T16:12:04.923 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-16T16:12:04.923 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-16T16:12:05.023 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:19.0.0-2868.gc3c87ed0.el9.x86_64 153/198 2024-04-16T16:12:05.062 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-16T16:12:05.078 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:19.0.0-2868.gc3c87ed0.el9.x86_64 154/198 2024-04-16T16:12:05.115 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-16T16:12:05.119 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 155/198 2024-04-16T16:12:05.151 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-16T16:12:05.158 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-16T16:12:05.196 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-16T16:12:05.261 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-16T16:12:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-16T16:12:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-shell-2:19.0.0-2868.gc3c87ed0.el9.x86_64 157/198 2024-04-16T16:12:05.569 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-16T16:12:05.652 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-16T16:12:05.653 INFO:teuthology.orchestra.run.smithi103.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-16T16:12:05.653 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:05.696 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:19.0.0-2868.gc3c87ed0.el9.x86_64 159/198 2024-04-16T16:12:05.763 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-16T16:12:05.831 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-16T16:12:05.877 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-16T16:12:05.920 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:19.0.0-2868.gc3c87ed0.el9.x86_ 163/198 2024-04-16T16:12:05.962 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-16T16:12:06.004 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-16T16:12:06.096 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-16T16:12:06.211 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-16T16:12:06.253 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-16T16:12:07.217 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-16T16:12:07.268 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-16T16:12:07.329 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-16T16:12:08.800 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-top-2:19.0.0-2868.gc3c87ed0.el9.noarch 170/198 2024-04-16T16:12:08.891 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-16T16:12:08.941 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-16T16:12:09.087 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-16T16:12:09.127 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-16T16:12:09.161 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-16T16:12:09.196 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-16T16:12:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-16T16:12:09.239 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-16T16:12:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-16T16:12:09.443 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-16T16:12:09.444 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-16T16:12:09.610 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-16T16:12:09.665 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-16T16:12:09.704 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-16T16:12:09.744 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-16T16:12:09.769 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-16T16:12:09.842 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-16T16:12:09.918 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-16T16:12:09.966 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-16T16:12:09.999 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-16T16:12:10.033 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-16T16:12:10.066 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-16T16:12:10.108 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-16T16:12:10.147 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-16T16:12:10.147 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-16T16:12:10.500 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-16T16:12:10.500 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-16T16:12:10.647 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-16T16:12:10.702 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-16T16:12:10.727 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-16T16:12:10.765 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-16T16:12:10.804 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-16T16:12:10.927 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-16T16:12:11.169 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:19.0.0-2868.gc3c87ed0.el9.x86_64 1/198 2024-04-16T16:12:11.169 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2/198 2024-04-16T16:12:11.169 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:19.0.0-2868.gc3c87ed0.el9.x86_64 3/198 2024-04-16T16:12:11.169 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:19.0.0-2868.gc3c87ed0.el9.x86_64 4/198 2024-04-16T16:12:11.169 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2868.gc3c87 5/198 2024-04-16T16:12:11.169 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:19.0.0-2868.gc3c87ed0.el9.x86_64 6/198 2024-04-16T16:12:11.169 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:19.0.0-2868.gc3c87ed0.el9.x86_64 7/198 2024-04-16T16:12:11.170 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:19.0.0-2868.gc3c87ed0.el9.x86_64 8/198 2024-04-16T16:12:11.170 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 9/198 2024-04-16T16:12:11.170 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:19.0.0-2868.gc3c87ed0.el9.x86_64 10/198 2024-04-16T16:12:11.170 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:19.0.0-2868.gc3c87ed0.el9.x86_64 11/198 2024-04-16T16:12:11.170 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:19.0.0-2868.gc3c87ed0.el9.x86_64 12/198 2024-04-16T16:12:11.170 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 13/198 2024-04-16T16:12:11.170 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-shell-2:19.0.0-2868.gc3c87ed0.el9.x86_64 14/198 2024-04-16T16:12:11.170 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:19.0.0-2868.gc3c87ed0.el9.x86_ 15/198 2024-04-16T16:12:11.170 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 16/198 2024-04-16T16:12:11.170 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:19.0.0-2868.gc3c87ed0.el9.x86_64 17/198 2024-04-16T16:12:11.170 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:19.0.0-2868.gc3c87ed0.el9.x86_6 18/198 2024-04-16T16:12:11.170 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:19.0.0-2868.gc3c87ed0.el9.x86 19/198 2024-04-16T16:12:11.171 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 20/198 2024-04-16T16:12:11.171 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2868.gc3c87ed0.el 21/198 2024-04-16T16:12:11.171 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:19.0.0-2868.gc3c87ed0.el9. 22/198 2024-04-16T16:12:11.171 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:19.0.0-2868.gc3c87ed0.el9.x86_6 23/198 2024-04-16T16:12:11.171 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:19.0.0-2868.gc3c87ed0.el9.x86_64 24/198 2024-04-16T16:12:11.171 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 25/198 2024-04-16T16:12:11.171 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:19.0.0-2868.gc3c87ed0.el9.x86_64 26/198 2024-04-16T16:12:11.171 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:19.0.0-2868.gc3c87ed0.el9.x86_64 27/198 2024-04-16T16:12:11.171 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 28/198 2024-04-16T16:12:11.171 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 29/198 2024-04-16T16:12:11.171 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2868.gc3c87ed0. 30/198 2024-04-16T16:12:11.172 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2868.gc3c87ed0.el9.noa 31/198 2024-04-16T16:12:11.172 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2868.gc3c87ed0.el9.n 32/198 2024-04-16T16:12:11.172 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2868.gc3c 33/198 2024-04-16T16:12:11.172 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2868.gc3c87ed0.el 34/198 2024-04-16T16:12:11.172 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2868.gc3c87ed0.el9.noarch 35/198 2024-04-16T16:12:11.172 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2868.gc3c87ed0.e 36/198 2024-04-16T16:12:11.172 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-volume-2:19.0.0-2868.gc3c87ed0.el9.noarch 37/198 2024-04-16T16:12:11.172 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:19.0.0-2868.gc3c87ed0.el9.noarch 38/198 2024-04-16T16:12:11.172 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-top-2:19.0.0-2868.gc3c87ed0.el9.noarch 39/198 2024-04-16T16:12:11.172 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-16T16:12:11.172 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-16T16:12:11.173 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-16T16:12:11.173 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-16T16:12:11.173 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-16T16:12:11.173 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-16T16:12:11.173 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-16T16:12:11.173 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-16T16:12:11.173 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-16T16:12:11.173 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-16T16:12:11.173 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-16T16:12:11.173 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-16T16:12:11.173 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-16T16:12:11.173 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-16T16:12:11.174 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-16T16:12:11.174 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-16T16:12:11.174 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-16T16:12:11.174 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-16T16:12:11.174 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-16T16:12:11.174 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-16T16:12:11.174 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-16T16:12:11.174 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-16T16:12:11.174 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-16T16:12:11.174 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-16T16:12:11.174 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-16T16:12:11.174 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-16T16:12:11.175 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-16T16:12:11.175 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-16T16:12:11.175 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-16T16:12:11.175 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-16T16:12:11.175 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-16T16:12:11.175 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-16T16:12:11.175 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-16T16:12:11.175 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-16T16:12:11.175 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-16T16:12:11.175 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-16T16:12:11.175 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-16T16:12:11.175 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-16T16:12:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-16T16:12:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-16T16:12:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-16T16:12:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-16T16:12:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-16T16:12:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-16T16:12:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-16T16:12:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-16T16:12:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-16T16:12:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-16T16:12:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-16T16:12:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-16T16:12:11.177 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-16T16:12:11.177 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-16T16:12:11.177 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-16T16:12:11.177 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-16T16:12:11.177 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-16T16:12:11.177 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-16T16:12:11.177 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-16T16:12:11.177 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-16T16:12:11.177 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-16T16:12:11.177 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-16T16:12:11.177 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-16T16:12:11.177 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-16T16:12:11.178 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-16T16:12:11.178 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-16T16:12:11.178 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-16T16:12:11.178 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-16T16:12:11.178 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-16T16:12:11.178 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-16T16:12:11.178 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-16T16:12:11.178 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-16T16:12:11.178 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-16T16:12:11.178 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-16T16:12:11.178 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-16T16:12:11.179 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-16T16:12:11.179 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-16T16:12:11.179 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-16T16:12:11.179 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-16T16:12:11.179 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-16T16:12:11.179 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-16T16:12:11.179 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-16T16:12:11.179 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-16T16:12:11.179 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-16T16:12:11.179 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-16T16:12:11.179 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-16T16:12:11.179 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-16T16:12:11.180 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-16T16:12:11.180 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-16T16:12:11.180 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-16T16:12:11.180 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-16T16:12:11.180 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-16T16:12:11.180 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-16T16:12:11.180 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-16T16:12:11.180 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-16T16:12:11.180 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-16T16:12:11.180 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-16T16:12:11.180 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-16T16:12:11.180 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-16T16:12:11.181 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-16T16:12:11.181 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-16T16:12:11.181 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-16T16:12:11.181 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-16T16:12:11.181 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-16T16:12:11.181 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-16T16:12:11.181 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-16T16:12:11.181 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-16T16:12:11.181 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-16T16:12:11.181 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-16T16:12:11.181 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-16T16:12:11.181 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-16T16:12:11.182 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-16T16:12:11.182 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-16T16:12:11.182 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 151/198 2024-04-16T16:12:11.182 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-16T16:12:11.182 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:19.0.0-2868.gc3c87ed0.el9.x86_64 153/198 2024-04-16T16:12:11.182 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-16T16:12:11.182 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-16T16:12:11.182 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-16T16:12:11.182 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-16T16:12:11.182 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-16T16:12:11.182 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-16T16:12:11.182 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-16T16:12:11.182 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-16T16:12:11.183 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-16T16:12:11.183 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-16T16:12:11.183 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-16T16:12:11.183 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-16T16:12:11.183 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-16T16:12:11.183 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-16T16:12:11.183 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-16T16:12:11.183 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-16T16:12:11.183 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-16T16:12:11.183 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-16T16:12:11.183 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-16T16:12:11.183 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-16T16:12:11.184 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-16T16:12:11.184 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-16T16:12:11.184 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-16T16:12:11.184 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-16T16:12:11.184 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-16T16:12:11.184 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-16T16:12:11.184 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-16T16:12:11.184 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-16T16:12:11.184 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-16T16:12:11.184 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-16T16:12:11.184 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-16T16:12:11.184 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-16T16:12:11.184 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-16T16:12:11.185 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-16T16:12:11.185 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-16T16:12:11.185 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-16T16:12:11.185 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-16T16:12:11.185 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-16T16:12:11.185 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-16T16:12:11.185 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-16T16:12:11.185 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-16T16:12:11.185 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-16T16:12:11.185 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-16T16:12:11.185 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-16T16:12:11.194 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-16T16:12:11.194 INFO:teuthology.orchestra.run.smithi045.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-16T16:12:11.195 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:13.697 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-16T16:12:13.697 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:13.697 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-04-16T16:12:13.697 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-16T16:12:13.697 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-16T16:12:13.697 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-16T16:12:13.698 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-16T16:12:13.698 INFO:teuthology.orchestra.run.smithi103.stdout: acl-2.3.1-4.el9.x86_64 2024-04-16T16:12:13.698 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-16T16:12:13.698 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-16T16:12:13.698 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-16T16:12:13.698 INFO:teuthology.orchestra.run.smithi103.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-16T16:12:13.698 INFO:teuthology.orchestra.run.smithi103.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-16T16:12:13.698 INFO:teuthology.orchestra.run.smithi103.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-16T16:12:13.698 INFO:teuthology.orchestra.run.smithi103.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-16T16:12:13.698 INFO:teuthology.orchestra.run.smithi103.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-16T16:12:13.698 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.698 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.698 INFO:teuthology.orchestra.run.smithi103.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-16T16:12:13.699 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-16T16:12:13.699 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-16T16:12:13.699 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-16T16:12:13.699 INFO:teuthology.orchestra.run.smithi103.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-16T16:12:13.699 INFO:teuthology.orchestra.run.smithi103.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-16T16:12:13.699 INFO:teuthology.orchestra.run.smithi103.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-16T16:12:13.699 INFO:teuthology.orchestra.run.smithi103.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-16T16:12:13.699 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-16T16:12:13.699 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-16T16:12:13.699 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T16:12:13.699 INFO:teuthology.orchestra.run.smithi103.stdout: automake-1.16.2-8.el9.noarch 2024-04-16T16:12:13.699 INFO:teuthology.orchestra.run.smithi103.stdout: bison-3.7.4-5.el9.x86_64 2024-04-16T16:12:13.699 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T16:12:13.699 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.700 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.700 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.700 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.700 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:13.700 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.700 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.700 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.700 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:13.700 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:13.700 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:13.700 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:13.700 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:13.700 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.700 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.701 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:13.701 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.701 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.701 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.701 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:13.701 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:13.701 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.701 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.701 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:13.701 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-16T16:12:13.701 INFO:teuthology.orchestra.run.smithi103.stdout: flex-2.6.4-9.el9.x86_64 2024-04-16T16:12:13.701 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T16:12:13.702 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T16:12:13.702 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T16:12:13.702 INFO:teuthology.orchestra.run.smithi103.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-16T16:12:13.702 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T16:12:13.702 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T16:12:13.702 INFO:teuthology.orchestra.run.smithi103.stdout: jq-1.6-16.el9.x86_64 2024-04-16T16:12:13.702 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-16T16:12:13.702 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T16:12:13.702 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T16:12:13.702 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.702 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.702 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.702 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T16:12:13.703 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T16:12:13.703 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T16:12:13.703 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T16:12:13.703 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T16:12:13.703 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.703 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.703 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T16:12:13.703 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.703 INFO:teuthology.orchestra.run.smithi103.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-16T16:12:13.703 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T16:12:13.703 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-16T16:12:13.703 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-16T16:12:13.703 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T16:12:13.703 INFO:teuthology.orchestra.run.smithi103.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T16:12:13.704 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T16:12:13.704 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T16:12:13.704 INFO:teuthology.orchestra.run.smithi103.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T16:12:13.704 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T16:12:13.704 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T16:12:13.704 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-16T16:12:13.704 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T16:12:13.704 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-13.14-1.el9.x86_64 2024-04-16T16:12:13.704 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-16T16:12:13.704 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-16T16:12:13.704 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-16T16:12:13.704 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T16:12:13.704 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T16:12:13.704 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T16:12:13.705 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-16T16:12:13.705 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T16:12:13.705 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T16:12:13.705 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T16:12:13.705 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.705 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.705 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.705 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T16:12:13.705 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T16:12:13.705 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T16:12:13.705 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T16:12:13.705 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-16T16:12:13.705 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-16T16:12:13.706 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T16:12:13.706 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T16:12:13.706 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T16:12:13.706 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T16:12:13.706 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T16:12:13.706 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T16:12:13.706 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T16:12:13.706 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T16:12:13.706 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T16:12:13.706 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T16:12:13.706 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T16:12:13.707 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T16:12:13.707 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T16:12:13.707 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T16:12:13.707 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T16:12:13.707 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T16:12:13.707 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T16:12:13.707 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T16:12:13.707 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T16:12:13.707 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T16:12:13.707 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T16:12:13.707 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T16:12:13.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T16:12:13.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T16:12:13.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T16:12:13.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T16:12:13.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T16:12:13.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T16:12:13.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T16:12:13.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T16:12:13.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T16:12:13.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T16:12:13.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-16T16:12:13.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T16:12:13.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T16:12:13.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T16:12:13.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T16:12:13.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T16:12:13.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T16:12:13.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T16:12:13.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T16:12:13.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T16:12:13.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T16:12:13.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T16:12:13.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-16T16:12:13.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T16:12:13.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T16:12:13.710 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T16:12:13.710 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T16:12:13.710 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T16:12:13.710 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T16:12:13.710 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T16:12:13.710 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.710 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.710 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:13.710 INFO:teuthology.orchestra.run.smithi103.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T16:12:13.710 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T16:12:13.710 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T16:12:13.710 INFO:teuthology.orchestra.run.smithi103.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-16T16:12:13.710 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-16T16:12:13.711 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T16:12:13.711 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-16T16:12:13.711 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:13.711 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-16T16:12:14.393 DEBUG:teuthology.parallel:result is None 2024-04-16T16:12:16.470 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:19.0.0-2868.gc3c87ed0.el9.x86_64 1/198 2024-04-16T16:12:16.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2/198 2024-04-16T16:12:16.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:19.0.0-2868.gc3c87ed0.el9.x86_64 3/198 2024-04-16T16:12:16.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:19.0.0-2868.gc3c87ed0.el9.x86_64 4/198 2024-04-16T16:12:16.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2868.gc3c87 5/198 2024-04-16T16:12:16.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:19.0.0-2868.gc3c87ed0.el9.x86_64 6/198 2024-04-16T16:12:16.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:19.0.0-2868.gc3c87ed0.el9.x86_64 7/198 2024-04-16T16:12:16.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:19.0.0-2868.gc3c87ed0.el9.x86_64 8/198 2024-04-16T16:12:16.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 9/198 2024-04-16T16:12:16.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:19.0.0-2868.gc3c87ed0.el9.x86_64 10/198 2024-04-16T16:12:16.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:19.0.0-2868.gc3c87ed0.el9.x86_64 11/198 2024-04-16T16:12:16.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:19.0.0-2868.gc3c87ed0.el9.x86_64 12/198 2024-04-16T16:12:16.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 13/198 2024-04-16T16:12:16.472 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-shell-2:19.0.0-2868.gc3c87ed0.el9.x86_64 14/198 2024-04-16T16:12:16.472 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:19.0.0-2868.gc3c87ed0.el9.x86_ 15/198 2024-04-16T16:12:16.472 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 16/198 2024-04-16T16:12:16.472 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:19.0.0-2868.gc3c87ed0.el9.x86_64 17/198 2024-04-16T16:12:16.472 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:19.0.0-2868.gc3c87ed0.el9.x86_6 18/198 2024-04-16T16:12:16.472 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:19.0.0-2868.gc3c87ed0.el9.x86 19/198 2024-04-16T16:12:16.472 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 20/198 2024-04-16T16:12:16.472 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2868.gc3c87ed0.el 21/198 2024-04-16T16:12:16.472 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:19.0.0-2868.gc3c87ed0.el9. 22/198 2024-04-16T16:12:16.472 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:19.0.0-2868.gc3c87ed0.el9.x86_6 23/198 2024-04-16T16:12:16.472 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:19.0.0-2868.gc3c87ed0.el9.x86_64 24/198 2024-04-16T16:12:16.472 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 25/198 2024-04-16T16:12:16.472 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:19.0.0-2868.gc3c87ed0.el9.x86_64 26/198 2024-04-16T16:12:16.473 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:19.0.0-2868.gc3c87ed0.el9.x86_64 27/198 2024-04-16T16:12:16.473 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 28/198 2024-04-16T16:12:16.473 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 29/198 2024-04-16T16:12:16.473 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2868.gc3c87ed0. 30/198 2024-04-16T16:12:16.473 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2868.gc3c87ed0.el9.noa 31/198 2024-04-16T16:12:16.473 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2868.gc3c87ed0.el9.n 32/198 2024-04-16T16:12:16.473 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2868.gc3c 33/198 2024-04-16T16:12:16.473 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2868.gc3c87ed0.el 34/198 2024-04-16T16:12:16.473 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2868.gc3c87ed0.el9.noarch 35/198 2024-04-16T16:12:16.473 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2868.gc3c87ed0.e 36/198 2024-04-16T16:12:16.473 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-volume-2:19.0.0-2868.gc3c87ed0.el9.noarch 37/198 2024-04-16T16:12:16.473 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:19.0.0-2868.gc3c87ed0.el9.noarch 38/198 2024-04-16T16:12:16.473 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-top-2:19.0.0-2868.gc3c87ed0.el9.noarch 39/198 2024-04-16T16:12:16.473 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-16T16:12:16.474 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-16T16:12:16.474 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-16T16:12:16.474 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-16T16:12:16.474 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-16T16:12:16.474 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-16T16:12:16.474 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-16T16:12:16.474 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-16T16:12:16.474 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-16T16:12:16.474 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-16T16:12:16.474 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-16T16:12:16.474 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-16T16:12:16.474 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-16T16:12:16.474 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-16T16:12:16.474 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-16T16:12:16.475 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-16T16:12:16.475 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-16T16:12:16.475 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-16T16:12:16.475 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-16T16:12:16.475 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-16T16:12:16.475 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-16T16:12:16.475 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-16T16:12:16.475 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-16T16:12:16.475 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-16T16:12:16.475 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-16T16:12:16.475 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-16T16:12:16.475 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-16T16:12:16.475 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-16T16:12:16.476 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-16T16:12:16.476 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-16T16:12:16.476 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-16T16:12:16.476 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-16T16:12:16.476 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-16T16:12:16.476 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-16T16:12:16.476 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-16T16:12:16.476 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-16T16:12:16.476 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-16T16:12:16.476 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-16T16:12:16.476 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-16T16:12:16.476 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-16T16:12:16.476 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-16T16:12:16.476 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-16T16:12:16.477 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-16T16:12:16.477 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-16T16:12:16.477 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-16T16:12:16.477 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-16T16:12:16.477 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-16T16:12:16.477 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-16T16:12:16.477 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-16T16:12:16.477 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-16T16:12:16.477 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-16T16:12:16.477 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-16T16:12:16.477 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-16T16:12:16.477 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-16T16:12:16.477 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-16T16:12:16.478 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-16T16:12:16.478 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-16T16:12:16.478 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-16T16:12:16.478 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-16T16:12:16.478 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-16T16:12:16.478 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-16T16:12:16.478 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-16T16:12:16.478 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-16T16:12:16.478 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-16T16:12:16.478 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-16T16:12:16.479 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-16T16:12:16.479 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-16T16:12:16.479 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-16T16:12:16.479 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-16T16:12:16.479 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-16T16:12:16.479 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-16T16:12:16.479 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-16T16:12:16.479 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-16T16:12:16.479 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-16T16:12:16.479 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-16T16:12:16.479 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-16T16:12:16.479 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-16T16:12:16.479 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-16T16:12:16.479 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-16T16:12:16.480 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-16T16:12:16.480 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-16T16:12:16.480 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-16T16:12:16.480 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-16T16:12:16.480 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-16T16:12:16.480 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-16T16:12:16.480 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-16T16:12:16.480 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-16T16:12:16.480 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-16T16:12:16.480 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-16T16:12:16.481 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-16T16:12:16.481 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-16T16:12:16.481 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-16T16:12:16.481 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-16T16:12:16.481 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-16T16:12:16.481 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-16T16:12:16.481 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-16T16:12:16.481 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-16T16:12:16.481 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-16T16:12:16.481 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-16T16:12:16.481 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-16T16:12:16.481 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-16T16:12:16.481 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-16T16:12:16.481 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-16T16:12:16.482 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-16T16:12:16.482 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-16T16:12:16.482 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-16T16:12:16.482 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-16T16:12:16.482 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-16T16:12:16.482 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-16T16:12:16.482 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-16T16:12:16.482 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-16T16:12:16.482 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 151/198 2024-04-16T16:12:16.482 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-16T16:12:16.482 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:19.0.0-2868.gc3c87ed0.el9.x86_64 153/198 2024-04-16T16:12:16.482 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-16T16:12:16.482 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-16T16:12:16.482 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-16T16:12:16.483 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-16T16:12:16.483 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-16T16:12:16.483 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-16T16:12:16.483 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-16T16:12:16.483 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-16T16:12:16.483 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-16T16:12:16.483 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-16T16:12:16.483 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-16T16:12:16.483 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-16T16:12:16.483 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-16T16:12:16.483 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-16T16:12:16.483 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-16T16:12:16.483 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-16T16:12:16.484 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-16T16:12:16.484 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-16T16:12:16.484 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-16T16:12:16.484 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-16T16:12:16.484 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-16T16:12:16.484 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-16T16:12:16.484 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-16T16:12:16.484 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-16T16:12:16.484 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-16T16:12:16.484 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-16T16:12:16.484 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-16T16:12:16.484 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-16T16:12:16.484 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-16T16:12:16.484 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-16T16:12:16.485 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-16T16:12:16.485 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-16T16:12:16.485 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-16T16:12:16.485 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-16T16:12:16.485 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-16T16:12:16.485 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-16T16:12:16.485 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-16T16:12:16.485 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-16T16:12:16.485 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-16T16:12:16.485 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-16T16:12:16.485 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-16T16:12:16.485 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-16T16:12:16.485 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-16T16:12:16.485 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-16T16:12:18.937 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-16T16:12:18.938 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:18.938 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2024-04-16T16:12:18.938 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-16T16:12:18.938 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-16T16:12:18.938 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-16T16:12:18.938 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-16T16:12:18.938 INFO:teuthology.orchestra.run.smithi045.stdout: acl-2.3.1-4.el9.x86_64 2024-04-16T16:12:18.938 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-16T16:12:18.938 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-16T16:12:18.938 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-16T16:12:18.938 INFO:teuthology.orchestra.run.smithi045.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-16T16:12:18.939 INFO:teuthology.orchestra.run.smithi045.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-16T16:12:18.939 INFO:teuthology.orchestra.run.smithi045.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-16T16:12:18.939 INFO:teuthology.orchestra.run.smithi045.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-16T16:12:18.939 INFO:teuthology.orchestra.run.smithi045.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-16T16:12:18.939 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.939 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.939 INFO:teuthology.orchestra.run.smithi045.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-16T16:12:18.939 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-16T16:12:18.939 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-16T16:12:18.939 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-16T16:12:18.939 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-16T16:12:18.939 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-16T16:12:18.939 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-16T16:12:18.939 INFO:teuthology.orchestra.run.smithi045.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-16T16:12:18.940 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-16T16:12:18.940 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T16:12:18.940 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T16:12:18.940 INFO:teuthology.orchestra.run.smithi045.stdout: automake-1.16.2-8.el9.noarch 2024-04-16T16:12:18.940 INFO:teuthology.orchestra.run.smithi045.stdout: bison-3.7.4-5.el9.x86_64 2024-04-16T16:12:18.940 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T16:12:18.940 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.940 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.940 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.940 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.940 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:18.940 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:18.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:18.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:18.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:18.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:18.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:18.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:18.942 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:18.942 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.942 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-shell-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.942 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-top-2:19.0.0-2868.gc3c87ed0.el9.noarch 2024-04-16T16:12:18.942 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-16T16:12:18.942 INFO:teuthology.orchestra.run.smithi045.stdout: flex-2.6.4-9.el9.x86_64 2024-04-16T16:12:18.942 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T16:12:18.942 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T16:12:18.942 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T16:12:18.942 INFO:teuthology.orchestra.run.smithi045.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-16T16:12:18.942 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T16:12:18.942 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T16:12:18.943 INFO:teuthology.orchestra.run.smithi045.stdout: jq-1.6-16.el9.x86_64 2024-04-16T16:12:18.943 INFO:teuthology.orchestra.run.smithi045.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-16T16:12:18.943 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T16:12:18.943 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T16:12:18.943 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.943 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.943 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.943 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T16:12:18.943 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T16:12:18.943 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T16:12:18.943 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T16:12:18.943 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T16:12:18.943 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.944 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.944 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T16:12:18.944 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.944 INFO:teuthology.orchestra.run.smithi045.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-16T16:12:18.944 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T16:12:18.944 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-16T16:12:18.944 INFO:teuthology.orchestra.run.smithi045.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-16T16:12:18.944 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T16:12:18.944 INFO:teuthology.orchestra.run.smithi045.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T16:12:18.944 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T16:12:18.944 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T16:12:18.944 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T16:12:18.944 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T16:12:18.944 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T16:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-16T16:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T16:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-13.14-1.el9.x86_64 2024-04-16T16:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-16T16:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-16T16:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-16T16:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T16:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T16:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T16:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-16T16:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T16:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T16:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T16:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T16:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T16:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T16:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T16:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-16T16:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-16T16:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T16:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T16:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T16:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T16:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T16:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T16:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T16:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T16:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T16:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T16:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T16:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T16:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T16:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T16:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T16:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T16:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T16:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T16:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T16:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T16:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T16:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T16:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T16:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T16:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T16:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T16:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T16:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T16:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T16:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T16:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T16:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T16:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-16T16:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T16:12:18.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T16:12:18.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T16:12:18.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T16:12:18.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T16:12:18.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T16:12:18.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T16:12:18.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T16:12:18.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T16:12:18.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T16:12:18.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T16:12:18.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-16T16:12:18.950 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T16:12:18.950 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T16:12:18.950 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T16:12:18.950 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T16:12:18.950 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T16:12:18.950 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T16:12:18.950 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T16:12:18.950 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.950 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.950 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:19.0.0-2868.gc3c87ed0.el9.x86_64 2024-04-16T16:12:18.950 INFO:teuthology.orchestra.run.smithi045.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T16:12:18.950 INFO:teuthology.orchestra.run.smithi045.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T16:12:18.950 INFO:teuthology.orchestra.run.smithi045.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T16:12:18.951 INFO:teuthology.orchestra.run.smithi045.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-16T16:12:18.951 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-16T16:12:18.951 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T16:12:18.951 INFO:teuthology.orchestra.run.smithi045.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-16T16:12:18.951 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:18.951 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T16:12:19.700 DEBUG:teuthology.parallel:result is None 2024-04-16T16:12:19.701 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f 2024-04-16T16:12:19.848 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T16:12:19.907 INFO:teuthology.orchestra.run.smithi045.stdout:19.0.0-2868.gc3c87ed0.el9 2024-04-16T16:12:19.907 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2868.gc3c87ed0.el9 2024-04-16T16:12:19.907 INFO:teuthology.task.install:The correct ceph version 19.0.0-2868.gc3c87ed0 is installed. 2024-04-16T16:12:19.909 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f 2024-04-16T16:12:20.177 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T16:12:20.248 INFO:teuthology.orchestra.run.smithi103.stdout:19.0.0-2868.gc3c87ed0.el9 2024-04-16T16:12:20.248 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2868.gc3c87ed0.el9 2024-04-16T16:12:20.248 INFO:teuthology.task.install:The correct ceph version 19.0.0-2868.gc3c87ed0 is installed. 2024-04-16T16:12:20.250 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-16T16:12:20.250 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:12:20.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T16:12:20.330 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:12:20.330 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T16:12:20.427 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-16T16:12:20.427 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:12:20.427 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T16:12:20.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T16:12:20.612 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:12:20.612 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T16:12:20.711 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T16:12:20.801 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-16T16:12:20.801 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:12:20.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T16:12:20.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T16:12:20.985 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:12:20.985 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T16:12:21.083 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T16:12:21.223 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-16T16:12:21.223 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:12:21.223 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T16:12:21.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T16:12:21.407 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:12:21.407 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T16:12:21.503 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T16:12:21.595 INFO:teuthology.run_tasks:Running task ceph... 2024-04-16T16:12:21.688 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-16T16:12:21.688 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /var/log/ceph 2024-04-16T16:12:21.692 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /var/log/ceph 2024-04-16T16:12:21.791 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-16T16:12:21.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T16:12:21.795 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T16:12:21.891 INFO:tasks.ceph:Creating extra log directories... 2024-04-16T16:12:21.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T16:12:21.895 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T16:12:21.995 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-16T16:12:21.995 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-16T16:12:21.995 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-16T16:12:21.995 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:12:21.995 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-16T16:12:22.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-16T16:12:22.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-16T16:12:22.323 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:12:22.323 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-16T16:12:22.378 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-16T16:12:22.627 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-16T16:12:22.733 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-16T16:12:22.733 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds debug zombie log segments': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-16T16:12:22.733 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211', 'branch': 'wip-lusov-shared-log-segment', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/metrics}', 'email': None, 'first_in_suite': False, 'job_id': '7658211', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-shared-log-segment'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds debug zombie log segments': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-shared-log-segment', 'sha1': 'c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 2402, 'sha1': 'c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f', 'sleep_before_teardown': 0, 'subset': '111/111111', 'suite': 'fs', 'suite_branch': 'wip-lusov-shared-log-segment', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi045.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDUcFo+1C2BSV3ek3BKzwod8oCPB6aZrPtasnCnVl6dygq8XCIbHNDNU52dSpibNF24UahNi9guSMLh9OOEjgJ8=', 'smithi103.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEPWcbwSh0lFEkxkF0bbAAqBwyITW4H1XFt//MpKTkkqQPLdN+m3u/lwLQsQJTlaCBI9Jd0PvGwcT00Zkht+jM8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds debug zombie log segments': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_mds_metrics']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-16_12:36:42', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-16T16:12:22.734 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-16T16:12:22.739 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-16T16:12:22.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T16:12:22.825 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T16:12:22.920 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:12:22.920 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T16:12:22.961 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T16:12:22.961 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2024-04-16T16:12:23.048 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T16:12:23.048 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T16:12:23.048 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-04-16T16:12:23.048 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T16:12:23.048 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T16:12:23.048 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-16 16:12:11.649143924 +0000 2024-04-16T16:12:23.049 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-16 16:09:38.079507104 +0000 2024-04-16T16:12:23.049 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-16 16:09:38.079507104 +0000 2024-04-16T16:12:23.049 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-04-16 16:09:38.079507104 +0000 2024-04-16T16:12:23.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T16:12:23.157 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:12:23.157 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:12:23.158 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000300095 s, 1.7 MB/s 2024-04-16T16:12:23.160 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T16:12:23.203 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2024-04-16T16:12:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T16:12:23.286 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T16:12:23.286 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-04-16T16:12:23.286 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T16:12:23.286 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T16:12:23.286 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-16 16:12:11.650143909 +0000 2024-04-16T16:12:23.286 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-16 16:09:38.691497719 +0000 2024-04-16T16:12:23.286 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-16 16:09:38.691497719 +0000 2024-04-16T16:12:23.286 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-04-16 16:09:38.691497719 +0000 2024-04-16T16:12:23.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T16:12:23.394 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:12:23.394 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:12:23.394 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000196719 s, 2.6 MB/s 2024-04-16T16:12:23.396 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T16:12:23.443 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2024-04-16T16:12:23.488 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T16:12:23.488 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T16:12:23.488 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 670 Links: 1 2024-04-16T16:12:23.488 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T16:12:23.488 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T16:12:23.488 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-16 16:12:11.651143893 +0000 2024-04-16T16:12:23.488 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-16 16:09:39.337487813 +0000 2024-04-16T16:12:23.488 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-16 16:09:39.337487813 +0000 2024-04-16T16:12:23.488 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-04-16 16:09:39.337487813 +0000 2024-04-16T16:12:23.489 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T16:12:23.623 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:12:23.623 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:12:23.623 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000237993 s, 2.2 MB/s 2024-04-16T16:12:23.628 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T16:12:23.689 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2024-04-16T16:12:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T16:12:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T16:12:23.742 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-04-16T16:12:23.742 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T16:12:23.742 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T16:12:23.742 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-16 16:12:11.652143878 +0000 2024-04-16T16:12:23.742 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-16 16:09:39.944478505 +0000 2024-04-16T16:12:23.742 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-16 16:09:39.944478505 +0000 2024-04-16T16:12:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-04-16 16:09:39.944478505 +0000 2024-04-16T16:12:23.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T16:12:23.827 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:12:23.827 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:12:23.827 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000216514 s, 2.4 MB/s 2024-04-16T16:12:23.831 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T16:12:23.882 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-16T16:12:23.883 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:12:23.883 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T16:12:23.941 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T16:12:23.942 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-04-16T16:12:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T16:12:23.994 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T16:12:23.994 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 634 Links: 1 2024-04-16T16:12:23.994 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T16:12:23.994 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T16:12:23.994 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-16 16:12:06.085961653 +0000 2024-04-16T16:12:23.994 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-16 16:09:29.635600649 +0000 2024-04-16T16:12:23.994 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-16 16:09:29.635600649 +0000 2024-04-16T16:12:23.994 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-16 16:09:29.635600649 +0000 2024-04-16T16:12:23.994 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T16:12:24.081 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-16T16:12:24.081 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-16T16:12:24.081 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000383815 s, 1.3 MB/s 2024-04-16T16:12:24.085 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T16:12:24.140 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-04-16T16:12:24.190 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T16:12:24.191 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T16:12:24.191 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 653 Links: 1 2024-04-16T16:12:24.191 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T16:12:24.191 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T16:12:24.191 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-16 16:12:06.086961634 +0000 2024-04-16T16:12:24.191 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-16 16:09:30.230589146 +0000 2024-04-16T16:12:24.191 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-16 16:09:30.230589146 +0000 2024-04-16T16:12:24.191 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-16 16:09:30.230589146 +0000 2024-04-16T16:12:24.192 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T16:12:24.274 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-16T16:12:24.274 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-16T16:12:24.274 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000349258 s, 1.5 MB/s 2024-04-16T16:12:24.277 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T16:12:24.331 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-04-16T16:12:24.382 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T16:12:24.382 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T16:12:24.382 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 670 Links: 1 2024-04-16T16:12:24.383 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T16:12:24.383 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T16:12:24.383 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-16 16:12:06.087961615 +0000 2024-04-16T16:12:24.383 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-16 16:09:30.859576986 +0000 2024-04-16T16:12:24.383 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-16 16:09:30.859576986 +0000 2024-04-16T16:12:24.383 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-16 16:09:30.859576986 +0000 2024-04-16T16:12:24.384 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T16:12:24.472 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-16T16:12:24.472 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-16T16:12:24.472 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000325871 s, 1.6 MB/s 2024-04-16T16:12:24.476 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T16:12:24.532 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-04-16T16:12:24.579 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T16:12:24.579 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T16:12:24.579 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-04-16T16:12:24.579 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T16:12:24.579 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T16:12:24.579 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-16 16:12:06.087961615 +0000 2024-04-16T16:12:24.579 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-16 16:09:31.462565328 +0000 2024-04-16T16:12:24.580 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-16 16:09:31.462565328 +0000 2024-04-16T16:12:24.580 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-16 16:09:31.462565328 +0000 2024-04-16T16:12:24.580 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T16:12:24.664 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-16T16:12:24.664 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-16T16:12:24.664 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000366056 s, 1.4 MB/s 2024-04-16T16:12:24.668 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T16:12:24.722 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-16T16:12:24.723 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-16T16:12:24.723 INFO:tasks.ceph:Generating config... 2024-04-16T16:12:24.725 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-16T16:12:24.725 INFO:tasks.ceph:[client] debug client = 20 2024-04-16T16:12:24.726 INFO:tasks.ceph:[client] debug ms = 1 2024-04-16T16:12:24.726 INFO:tasks.ceph:[client] pid file = 2024-04-16T16:12:24.726 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-16T16:12:24.726 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-16T16:12:24.726 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-16T16:12:24.726 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-16T16:12:24.726 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-16T16:12:24.726 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-16T16:12:24.726 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-16T16:12:24.727 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-16T16:12:24.727 INFO:tasks.ceph:[mds] mds debug zombie log segments = True 2024-04-16T16:12:24.727 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-16T16:12:24.727 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-16T16:12:24.727 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-16T16:12:24.727 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-16T16:12:24.727 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-16T16:12:24.727 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-16T16:12:24.727 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-16T16:12:24.727 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-16T16:12:24.727 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-16T16:12:24.728 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-16T16:12:24.728 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-16T16:12:24.728 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-16T16:12:24.728 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-16T16:12:24.728 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-16T16:12:24.728 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-16T16:12:24.728 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-16T16:12:24.728 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-16T16:12:24.728 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-16T16:12:24.728 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-16T16:12:24.728 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-16T16:12:24.728 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-16T16:12:24.728 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-16T16:12:24.729 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-16T16:12:24.729 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-16T16:12:24.729 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-16T16:12:24.729 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-16T16:12:24.729 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-16T16:12:24.729 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-16T16:12:24.729 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-16T16:12:24.729 INFO:tasks.ceph:Setting up mon.a... 2024-04-16T16:12:24.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-16T16:12:24.874 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/ceph.keyring 2024-04-16T16:12:24.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-16T16:12:25.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T16:12:25.116 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.45'), ('mon.b', '172.21.15.103'), ('mon.c', '[v2:172.21.15.103:3301,v1:172.21.15.103:6790]')] 2024-04-16T16:12:25.117 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.45,172.21.15.103,[v2:172.21.15.103:3301,v1:172.21.15.103:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds debug zombie log segments': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-16T16:12:25.118 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:12:25.118 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T16:12:25.173 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.45 --add b 172.21.15.103 --addv c '[v2:172.21.15.103:3301,v1:172.21.15.103:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-16T16:12:25.330 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-16T16:12:25.330 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: generated fsid 748c4b85-0190-4c59-80ad-098a6a5007c0 2024-04-16T16:12:25.330 INFO:teuthology.orchestra.run.smithi045.stdout:setting min_mon_release = quincy 2024-04-16T16:12:25.330 INFO:teuthology.orchestra.run.smithi045.stdout:epoch 0 2024-04-16T16:12:25.330 INFO:teuthology.orchestra.run.smithi045.stdout:fsid 748c4b85-0190-4c59-80ad-098a6a5007c0 2024-04-16T16:12:25.330 INFO:teuthology.orchestra.run.smithi045.stdout:last_changed 2024-04-16T16:12:25.331516+0000 2024-04-16T16:12:25.330 INFO:teuthology.orchestra.run.smithi045.stdout:created 2024-04-16T16:12:25.331516+0000 2024-04-16T16:12:25.330 INFO:teuthology.orchestra.run.smithi045.stdout:min_mon_release 17 (quincy) 2024-04-16T16:12:25.330 INFO:teuthology.orchestra.run.smithi045.stdout:election_strategy: 1 2024-04-16T16:12:25.330 INFO:teuthology.orchestra.run.smithi045.stdout:0: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.a 2024-04-16T16:12:25.331 INFO:teuthology.orchestra.run.smithi045.stdout:1: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-04-16T16:12:25.331 INFO:teuthology.orchestra.run.smithi045.stdout:2: [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] mon.c 2024-04-16T16:12:25.331 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-16T16:12:25.333 DEBUG:teuthology.orchestra.run.smithi045:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T16:12:25.385 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 748c4b85-0190-4c59-80ad-098a6a5007c0... 2024-04-16T16:12:25.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-16T16:12:25.393 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-16T16:12:25.562 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2024-04-16T16:12:25.563 INFO:teuthology.orchestra.run.smithi045.stdout: chdir = "" 2024-04-16T16:12:25.563 INFO:teuthology.orchestra.run.smithi045.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T16:12:25.563 INFO:teuthology.orchestra.run.smithi045.stdout: auth supported = cephx 2024-04-16T16:12:25.563 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.563 INFO:teuthology.orchestra.run.smithi045.stdout: filestore xattr use omap = true 2024-04-16T16:12:25.563 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.563 INFO:teuthology.orchestra.run.smithi045.stdout: mon clock drift allowed = 1.000 2024-04-16T16:12:25.563 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.563 INFO:teuthology.orchestra.run.smithi045.stdout: osd crush chooseleaf type = 0 2024-04-16T16:12:25.563 INFO:teuthology.orchestra.run.smithi045.stdout: auth debug = true 2024-04-16T16:12:25.563 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.563 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on old message = true 2024-04-16T16:12:25.564 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on bug = true 2024-04-16T16:12:25.564 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.564 INFO:teuthology.orchestra.run.smithi045.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T16:12:25.564 INFO:teuthology.orchestra.run.smithi045.stdout: mon pg warn max object skew = 0 2024-04-16T16:12:25.564 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.564 INFO:teuthology.orchestra.run.smithi045.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T16:12:25.564 INFO:teuthology.orchestra.run.smithi045.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T16:12:25.564 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.564 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default size = 2 2024-04-16T16:12:25.564 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.564 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow primary affinity = true 2024-04-16T16:12:25.564 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow pg remap = true 2024-04-16T16:12:25.564 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on legacy crush tunables = false 2024-04-16T16:12:25.564 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on crush straw calc version zero = false 2024-04-16T16:12:25.565 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on no sortbitwise = false 2024-04-16T16:12:25.565 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on osd down out interval zero = false 2024-04-16T16:12:25.565 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on too few osds = false 2024-04-16T16:12:25.565 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T16:12:25.565 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T16:12:25.565 INFO:teuthology.orchestra.run.smithi045.stdout: mon_allow_pool_size_one = true 2024-04-16T16:12:25.565 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.565 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T16:12:25.565 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.565 INFO:teuthology.orchestra.run.smithi045.stdout: osd default data pool replay window = 5 2024-04-16T16:12:25.565 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.565 INFO:teuthology.orchestra.run.smithi045.stdout: mon allow pool delete = true 2024-04-16T16:12:25.566 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.566 INFO:teuthology.orchestra.run.smithi045.stdout: mon cluster log file level = debug 2024-04-16T16:12:25.566 INFO:teuthology.orchestra.run.smithi045.stdout: debug asserts on shutdown = true 2024-04-16T16:12:25.566 INFO:teuthology.orchestra.run.smithi045.stdout: mon health detail to clog = false 2024-04-16T16:12:25.566 INFO:teuthology.orchestra.run.smithi045.stdout: mon host = "172.21.15.45,172.21.15.103,[v2:172.21.15.103:3301,v1:172.21.15.103:6790]" 2024-04-16T16:12:25.566 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on skipped message = False 2024-04-16T16:12:25.566 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = 748c4b85-0190-4c59-80ad-098a6a5007c0 2024-04-16T16:12:25.566 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.566 INFO:teuthology.orchestra.run.smithi045.stdout:[osd] 2024-04-16T16:12:25.566 INFO:teuthology.orchestra.run.smithi045.stdout: osd journal size = 100 2024-04-16T16:12:25.566 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.567 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub load threshold = 5.0 2024-04-16T16:12:25.567 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub max interval = 600 2024-04-16T16:12:25.567 INFO:teuthology.orchestra.run.smithi045.stdout: osd mclock profile = high_recovery_ops 2024-04-16T16:12:25.567 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.567 INFO:teuthology.orchestra.run.smithi045.stdout: osd recover clone overlap = true 2024-04-16T16:12:25.567 INFO:teuthology.orchestra.run.smithi045.stdout: osd recovery max chunk = 1048576 2024-04-16T16:12:25.567 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.567 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug shutdown = true 2024-04-16T16:12:25.567 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug op order = true 2024-04-16T16:12:25.567 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify stray on activate = true 2024-04-16T16:12:25.567 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug trim objects = true 2024-04-16T16:12:25.567 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.567 INFO:teuthology.orchestra.run.smithi045.stdout: osd open classes on start = true 2024-04-16T16:12:25.567 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug pg log writeout = true 2024-04-16T16:12:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: osd deep scrub update digest min age = 30 2024-04-16T16:12:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: osd map max advance = 10 2024-04-16T16:12:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: journal zero on create = true 2024-04-16T16:12:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: filestore ondisk finisher threads = 3 2024-04-16T16:12:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: filestore apply finisher threads = 3 2024-04-16T16:12:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: bdev debug aio = true 2024-04-16T16:12:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug misdirected ops = true 2024-04-16T16:12:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: bdev async discard = True 2024-04-16T16:12:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: bdev enable discard = True 2024-04-16T16:12:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore allocator = bitmap 2024-04-16T16:12:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore block size = 96636764160 2024-04-16T16:12:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore fsck on mount = True 2024-04-16T16:12:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluefs = 1/20 2024-04-16T16:12:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluestore = 1/20 2024-04-16T16:12:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-04-16T16:12:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: debug osd = 20 2024-04-16T16:12:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: debug rocksdb = 4/10 2024-04-16T16:12:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd backfillfull_ratio = 0.85 2024-04-16T16:12:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd full ratio = 0.9 2024-04-16T16:12:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd nearfull ratio = 0.8 2024-04-16T16:12:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: osd failsafe full ratio = 0.95 2024-04-16T16:12:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: osd objectstore = bluestore 2024-04-16T16:12:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: osd op complaint time = 180 2024-04-16T16:12:25.570 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.570 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr] 2024-04-16T16:12:25.570 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-04-16T16:12:25.570 INFO:teuthology.orchestra.run.smithi045.stdout: debug mgr = 20 2024-04-16T16:12:25.570 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2024-04-16T16:12:25.570 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2024-04-16T16:12:25.570 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2024-04-16T16:12:25.570 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2024-04-16T16:12:25.570 INFO:teuthology.orchestra.run.smithi045.stdout: mgr/telemetry/nag = false 2024-04-16T16:12:25.570 INFO:teuthology.orchestra.run.smithi045.stdout: debug client = 20 2024-04-16T16:12:25.570 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.570 INFO:teuthology.orchestra.run.smithi045.stdout:[mon] 2024-04-16T16:12:25.570 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-04-16T16:12:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2024-04-16T16:12:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: debug paxos = 20 2024-04-16T16:12:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2024-04-16T16:12:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: mon data avail warn = 5 2024-04-16T16:12:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: mon mgr mkfs grace = 240 2024-04-16T16:12:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2024-04-16T16:12:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd reporter subtree level = osd 2024-04-16T16:12:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd prime pg temp = true 2024-04-16T16:12:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2024-04-16T16:12:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T16:12:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T16:12:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T16:12:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T16:12:25.572 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T16:12:25.572 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T16:12:25.572 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.572 INFO:teuthology.orchestra.run.smithi045.stdout: # 1m isn't quite enough 2024-04-16T16:12:25.572 INFO:teuthology.orchestra.run.smithi045.stdout: mon_down_mkfs_grace = 2m 2024-04-16T16:12:25.572 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.572 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_filestore_osds = false 2024-04-16T16:12:25.572 INFO:teuthology.orchestra.run.smithi045.stdout: mon down mkfs grace = 300 2024-04-16T16:12:25.572 INFO:teuthology.orchestra.run.smithi045.stdout: mon op complaint time = 120 2024-04-16T16:12:25.572 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:12:25.572 INFO:teuthology.orchestra.run.smithi045.stdout:[client] 2024-04-16T16:12:25.572 INFO:teuthology.orchestra.run.smithi045.stdout: rgw cache enabled = true 2024-04-16T16:12:25.572 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable ops log = true 2024-04-16T16:12:25.573 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable usage log = true 2024-04-16T16:12:25.573 INFO:teuthology.orchestra.run.smithi045.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T16:12:25.573 INFO:teuthology.orchestra.run.smithi045.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T16:12:25.573 INFO:teuthology.orchestra.run.smithi045.stdout: client mount timeout = 600 2024-04-16T16:12:25.573 INFO:teuthology.orchestra.run.smithi045.stdout: debug client = 20 2024-04-16T16:12:25.573 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-04-16T16:12:25.573 INFO:teuthology.orchestra.run.smithi045.stdout: pid file = "" 2024-04-16T16:12:25.573 INFO:teuthology.orchestra.run.smithi045.stdout: rados mon op timeout = 900 2024-04-16T16:12:25.573 INFO:teuthology.orchestra.run.smithi045.stdout: rados osd op timeout = 900 2024-04-16T16:12:25.573 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.a] 2024-04-16T16:12:25.573 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.b] 2024-04-16T16:12:25.573 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.c] 2024-04-16T16:12:25.573 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.a] 2024-04-16T16:12:25.573 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.c] 2024-04-16T16:12:25.574 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.b] 2024-04-16T16:12:25.574 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.d] 2024-04-16T16:12:25.574 INFO:teuthology.orchestra.run.smithi045.stdout:[mds] 2024-04-16T16:12:25.574 INFO:teuthology.orchestra.run.smithi045.stdout: debug mds = 20 2024-04-16T16:12:25.574 INFO:teuthology.orchestra.run.smithi045.stdout: debug mds balancer = 20 2024-04-16T16:12:25.574 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-04-16T16:12:25.574 INFO:teuthology.orchestra.run.smithi045.stdout: mds debug frag = True 2024-04-16T16:12:25.574 INFO:teuthology.orchestra.run.smithi045.stdout: mds debug scatterstat = True 2024-04-16T16:12:25.574 INFO:teuthology.orchestra.run.smithi045.stdout: mds debug zombie log segments = True 2024-04-16T16:12:25.574 INFO:teuthology.orchestra.run.smithi045.stdout: mds op complaint time = 180 2024-04-16T16:12:25.574 INFO:teuthology.orchestra.run.smithi045.stdout: mds verify scatter = True 2024-04-16T16:12:25.574 INFO:teuthology.orchestra.run.smithi045.stdout: osd op complaint time = 180 2024-04-16T16:12:25.574 INFO:teuthology.orchestra.run.smithi045.stdout: rados mon op timeout = 900 2024-04-16T16:12:25.575 INFO:teuthology.orchestra.run.smithi045.stdout: rados osd op timeout = 900 2024-04-16T16:12:25.575 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-04-16T16:12:25.575 INFO:teuthology.orchestra.run.smithi103.stdout: chdir = "" 2024-04-16T16:12:25.575 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T16:12:25.575 INFO:teuthology.orchestra.run.smithi103.stdout: auth supported = cephx 2024-04-16T16:12:25.575 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.575 INFO:teuthology.orchestra.run.smithi103.stdout: filestore xattr use omap = true 2024-04-16T16:12:25.575 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.575 INFO:teuthology.orchestra.run.smithi103.stdout: mon clock drift allowed = 1.000 2024-04-16T16:12:25.576 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.576 INFO:teuthology.orchestra.run.smithi103.stdout: osd crush chooseleaf type = 0 2024-04-16T16:12:25.576 INFO:teuthology.orchestra.run.smithi103.stdout: auth debug = true 2024-04-16T16:12:25.576 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.576 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on old message = true 2024-04-16T16:12:25.576 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on bug = true 2024-04-16T16:12:25.576 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.576 INFO:teuthology.orchestra.run.smithi103.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T16:12:25.576 INFO:teuthology.orchestra.run.smithi103.stdout: mon pg warn max object skew = 0 2024-04-16T16:12:25.576 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.576 INFO:teuthology.orchestra.run.smithi103.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T16:12:25.576 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T16:12:25.576 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.576 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default size = 2 2024-04-16T16:12:25.577 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.577 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow primary affinity = true 2024-04-16T16:12:25.577 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow pg remap = true 2024-04-16T16:12:25.577 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on legacy crush tunables = false 2024-04-16T16:12:25.577 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on crush straw calc version zero = false 2024-04-16T16:12:25.577 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on no sortbitwise = false 2024-04-16T16:12:25.577 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on osd down out interval zero = false 2024-04-16T16:12:25.577 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on too few osds = false 2024-04-16T16:12:25.577 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T16:12:25.577 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T16:12:25.577 INFO:teuthology.orchestra.run.smithi103.stdout: mon_allow_pool_size_one = true 2024-04-16T16:12:25.577 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.577 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T16:12:25.577 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.578 INFO:teuthology.orchestra.run.smithi103.stdout: osd default data pool replay window = 5 2024-04-16T16:12:25.578 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.578 INFO:teuthology.orchestra.run.smithi103.stdout: mon allow pool delete = true 2024-04-16T16:12:25.578 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.578 INFO:teuthology.orchestra.run.smithi103.stdout: mon cluster log file level = debug 2024-04-16T16:12:25.578 INFO:teuthology.orchestra.run.smithi103.stdout: debug asserts on shutdown = true 2024-04-16T16:12:25.578 INFO:teuthology.orchestra.run.smithi103.stdout: mon health detail to clog = false 2024-04-16T16:12:25.578 INFO:teuthology.orchestra.run.smithi103.stdout: mon host = "172.21.15.45,172.21.15.103,[v2:172.21.15.103:3301,v1:172.21.15.103:6790]" 2024-04-16T16:12:25.578 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on skipped message = False 2024-04-16T16:12:25.578 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = 748c4b85-0190-4c59-80ad-098a6a5007c0 2024-04-16T16:12:25.578 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.578 INFO:teuthology.orchestra.run.smithi103.stdout:[osd] 2024-04-16T16:12:25.578 INFO:teuthology.orchestra.run.smithi103.stdout: osd journal size = 100 2024-04-16T16:12:25.579 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.579 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub load threshold = 5.0 2024-04-16T16:12:25.579 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub max interval = 600 2024-04-16T16:12:25.579 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock profile = high_recovery_ops 2024-04-16T16:12:25.579 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.579 INFO:teuthology.orchestra.run.smithi103.stdout: osd recover clone overlap = true 2024-04-16T16:12:25.579 INFO:teuthology.orchestra.run.smithi103.stdout: osd recovery max chunk = 1048576 2024-04-16T16:12:25.579 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.579 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug shutdown = true 2024-04-16T16:12:25.579 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug op order = true 2024-04-16T16:12:25.579 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify stray on activate = true 2024-04-16T16:12:25.579 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug trim objects = true 2024-04-16T16:12:25.579 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.579 INFO:teuthology.orchestra.run.smithi103.stdout: osd open classes on start = true 2024-04-16T16:12:25.580 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug pg log writeout = true 2024-04-16T16:12:25.580 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.580 INFO:teuthology.orchestra.run.smithi103.stdout: osd deep scrub update digest min age = 30 2024-04-16T16:12:25.580 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.580 INFO:teuthology.orchestra.run.smithi103.stdout: osd map max advance = 10 2024-04-16T16:12:25.580 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.580 INFO:teuthology.orchestra.run.smithi103.stdout: journal zero on create = true 2024-04-16T16:12:25.580 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.580 INFO:teuthology.orchestra.run.smithi103.stdout: filestore ondisk finisher threads = 3 2024-04-16T16:12:25.580 INFO:teuthology.orchestra.run.smithi103.stdout: filestore apply finisher threads = 3 2024-04-16T16:12:25.580 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.580 INFO:teuthology.orchestra.run.smithi103.stdout: bdev debug aio = true 2024-04-16T16:12:25.580 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug misdirected ops = true 2024-04-16T16:12:25.581 INFO:teuthology.orchestra.run.smithi103.stdout: bdev async discard = True 2024-04-16T16:12:25.581 INFO:teuthology.orchestra.run.smithi103.stdout: bdev enable discard = True 2024-04-16T16:12:25.581 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore allocator = bitmap 2024-04-16T16:12:25.581 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore block size = 96636764160 2024-04-16T16:12:25.581 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore fsck on mount = True 2024-04-16T16:12:25.581 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluefs = 1/20 2024-04-16T16:12:25.581 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluestore = 1/20 2024-04-16T16:12:25.581 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-16T16:12:25.581 INFO:teuthology.orchestra.run.smithi103.stdout: debug osd = 20 2024-04-16T16:12:25.581 INFO:teuthology.orchestra.run.smithi103.stdout: debug rocksdb = 4/10 2024-04-16T16:12:25.581 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd backfillfull_ratio = 0.85 2024-04-16T16:12:25.582 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd full ratio = 0.9 2024-04-16T16:12:25.582 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd nearfull ratio = 0.8 2024-04-16T16:12:25.582 INFO:teuthology.orchestra.run.smithi103.stdout: osd failsafe full ratio = 0.95 2024-04-16T16:12:25.582 INFO:teuthology.orchestra.run.smithi103.stdout: osd objectstore = bluestore 2024-04-16T16:12:25.582 INFO:teuthology.orchestra.run.smithi103.stdout: osd op complaint time = 180 2024-04-16T16:12:25.582 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.582 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr] 2024-04-16T16:12:25.582 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-16T16:12:25.582 INFO:teuthology.orchestra.run.smithi103.stdout: debug mgr = 20 2024-04-16T16:12:25.582 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-16T16:12:25.582 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-16T16:12:25.583 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-16T16:12:25.583 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-16T16:12:25.583 INFO:teuthology.orchestra.run.smithi103.stdout: mgr/telemetry/nag = false 2024-04-16T16:12:25.583 INFO:teuthology.orchestra.run.smithi103.stdout: debug client = 20 2024-04-16T16:12:25.583 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.583 INFO:teuthology.orchestra.run.smithi103.stdout:[mon] 2024-04-16T16:12:25.583 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-16T16:12:25.583 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-16T16:12:25.583 INFO:teuthology.orchestra.run.smithi103.stdout: debug paxos = 20 2024-04-16T16:12:25.583 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-16T16:12:25.583 INFO:teuthology.orchestra.run.smithi103.stdout: mon data avail warn = 5 2024-04-16T16:12:25.583 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr mkfs grace = 240 2024-04-16T16:12:25.583 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-16T16:12:25.583 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd reporter subtree level = osd 2024-04-16T16:12:25.584 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd prime pg temp = true 2024-04-16T16:12:25.584 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-16T16:12:25.584 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.584 INFO:teuthology.orchestra.run.smithi103.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T16:12:25.584 INFO:teuthology.orchestra.run.smithi103.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T16:12:25.584 INFO:teuthology.orchestra.run.smithi103.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T16:12:25.584 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.584 INFO:teuthology.orchestra.run.smithi103.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T16:12:25.584 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T16:12:25.584 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T16:12:25.584 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.584 INFO:teuthology.orchestra.run.smithi103.stdout: # 1m isn't quite enough 2024-04-16T16:12:25.584 INFO:teuthology.orchestra.run.smithi103.stdout: mon_down_mkfs_grace = 2m 2024-04-16T16:12:25.584 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.584 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_filestore_osds = false 2024-04-16T16:12:25.585 INFO:teuthology.orchestra.run.smithi103.stdout: mon down mkfs grace = 300 2024-04-16T16:12:25.585 INFO:teuthology.orchestra.run.smithi103.stdout: mon op complaint time = 120 2024-04-16T16:12:25.585 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-16T16:12:25.585 INFO:teuthology.orchestra.run.smithi103.stdout:[client] 2024-04-16T16:12:25.585 INFO:teuthology.orchestra.run.smithi103.stdout: rgw cache enabled = true 2024-04-16T16:12:25.585 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable ops log = true 2024-04-16T16:12:25.585 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable usage log = true 2024-04-16T16:12:25.585 INFO:teuthology.orchestra.run.smithi103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T16:12:25.585 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T16:12:25.585 INFO:teuthology.orchestra.run.smithi103.stdout: client mount timeout = 600 2024-04-16T16:12:25.585 INFO:teuthology.orchestra.run.smithi103.stdout: debug client = 20 2024-04-16T16:12:25.585 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-16T16:12:25.585 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = "" 2024-04-16T16:12:25.585 INFO:teuthology.orchestra.run.smithi103.stdout: rados mon op timeout = 900 2024-04-16T16:12:25.586 INFO:teuthology.orchestra.run.smithi103.stdout: rados osd op timeout = 900 2024-04-16T16:12:25.586 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.a] 2024-04-16T16:12:25.586 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.b] 2024-04-16T16:12:25.586 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.c] 2024-04-16T16:12:25.586 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.a] 2024-04-16T16:12:25.586 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.c] 2024-04-16T16:12:25.586 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.b] 2024-04-16T16:12:25.586 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.d] 2024-04-16T16:12:25.586 INFO:teuthology.orchestra.run.smithi103.stdout:[mds] 2024-04-16T16:12:25.586 INFO:teuthology.orchestra.run.smithi103.stdout: debug mds = 20 2024-04-16T16:12:25.586 INFO:teuthology.orchestra.run.smithi103.stdout: debug mds balancer = 20 2024-04-16T16:12:25.586 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-16T16:12:25.586 INFO:teuthology.orchestra.run.smithi103.stdout: mds debug frag = True 2024-04-16T16:12:25.587 INFO:teuthology.orchestra.run.smithi103.stdout: mds debug scatterstat = True 2024-04-16T16:12:25.587 INFO:teuthology.orchestra.run.smithi103.stdout: mds debug zombie log segments = True 2024-04-16T16:12:25.587 INFO:teuthology.orchestra.run.smithi103.stdout: mds op complaint time = 180 2024-04-16T16:12:25.587 INFO:teuthology.orchestra.run.smithi103.stdout: mds verify scatter = True 2024-04-16T16:12:25.587 INFO:teuthology.orchestra.run.smithi103.stdout: osd op complaint time = 180 2024-04-16T16:12:25.587 INFO:teuthology.orchestra.run.smithi103.stdout: rados mon op timeout = 900 2024-04-16T16:12:25.587 INFO:teuthology.orchestra.run.smithi103.stdout: rados osd op timeout = 900 2024-04-16T16:12:25.609 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-16T16:12:25.609 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-16T16:12:25.758 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-16T16:12:25.758 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:12:25.758 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-16T16:12:25.811 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:12:25.812 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-16T16:12:25.866 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:12:25.867 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:12:25.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-16T16:12:25.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T16:12:26.026 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:12:26.026 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-16T16:12:26.127 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:12:26.127 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:12:26.127 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-16T16:12:26.127 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T16:12:26.264 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:12:26.265 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-16T16:12:26.359 INFO:tasks.ceph:Setting up mon nodes... 2024-04-16T16:12:26.360 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-16T16:12:26.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-16T16:12:26.557 INFO:teuthology.orchestra.run.smithi045.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-16T16:12:26.568 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T16:12:26.750 INFO:teuthology.orchestra.run.smithi103.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T16:12:26.759 INFO:tasks.ceph:Setting up mds nodes... 2024-04-16T16:12:26.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-16T16:12:26.960 INFO:teuthology.orchestra.run.smithi045.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-16T16:12:26.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-16T16:12:27.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-16T16:12:27.259 INFO:teuthology.orchestra.run.smithi045.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-16T16:12:27.268 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-16T16:12:27.371 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-16T16:12:27.543 INFO:teuthology.orchestra.run.smithi103.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-16T16:12:27.550 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-16T16:12:27.644 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-16T16:12:27.821 INFO:teuthology.orchestra.run.smithi103.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-16T16:12:27.830 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-16T16:12:27.924 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-16T16:12:27.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-16T16:12:28.063 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-16T16:12:28.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-16T16:12:28.255 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-16T16:12:28.312 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-16T16:12:28.440 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-16T16:12:28.489 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-16T16:12:28.607 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-16T16:12:28.658 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-16T16:12:28.658 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-16T16:12:28.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-16T16:12:28.765 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-16T16:12:28.765 INFO:tasks.ceph:role: osd.0 2024-04-16T16:12:28.765 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:12:28.765 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T16:12:28.863 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T16:12:28.863 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T16:12:28.863 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T16:12:28.863 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T16:12:28.863 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T16:12:28.863 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-04-16T16:12:28.864 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T16:12:28.864 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T16:12:28.864 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T16:12:28.864 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T16:12:28.899 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-04-16T16:12:28.903 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-04-16T16:12:28.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-16T16:12:29.020 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-16T16:12:29.106 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-16T16:12:29.198 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-16T16:12:29.198 INFO:tasks.ceph:role: osd.1 2024-04-16T16:12:29.198 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:12:29.198 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T16:12:29.295 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T16:12:29.296 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T16:12:29.296 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T16:12:29.296 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T16:12:29.296 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T16:12:29.296 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-04-16T16:12:29.296 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T16:12:29.296 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T16:12:29.296 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T16:12:29.296 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T16:12:29.330 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-04-16T16:12:29.335 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-04-16T16:12:29.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-16T16:12:29.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-16T16:12:29.539 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-16T16:12:29.622 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-16T16:12:29.623 INFO:tasks.ceph:role: osd.2 2024-04-16T16:12:29.623 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:12:29.623 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-16T16:12:29.708 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T16:12:29.708 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T16:12:29.708 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T16:12:29.708 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T16:12:29.708 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T16:12:29.708 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-04-16T16:12:29.708 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T16:12:29.708 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T16:12:29.708 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T16:12:29.708 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T16:12:29.742 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-04-16T16:12:29.748 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-04-16T16:12:29.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-16T16:12:29.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-16T16:12:29.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-16T16:12:30.049 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-16T16:12:30.050 INFO:tasks.ceph:role: osd.3 2024-04-16T16:12:30.050 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:12:30.050 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-16T16:12:30.146 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T16:12:30.146 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T16:12:30.146 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T16:12:30.146 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T16:12:30.146 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T16:12:30.146 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-04-16T16:12:30.146 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T16:12:30.146 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T16:12:30.146 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T16:12:30.146 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T16:12:30.179 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-04-16T16:12:30.184 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-04-16T16:12:30.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-16T16:12:30.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-16T16:12:30.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T16:12:30.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:30.531+0000 7f6e1f068640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-16T16:12:30.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:30.532+0000 7f6e1f068640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-16T16:12:30.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:30.532+0000 7f6e1f068640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-16T16:12:30.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:30.532+0000 7f6e1f068640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-16T16:12:30.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:30.532+0000 7f6e1f068640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-16T16:12:30.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:30.536+0000 7f6e1f068640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-16T16:12:33.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T16:12:33.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T16:12:33.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:33.777+0000 7f3d3da23640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-16T16:12:33.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:33.778+0000 7f3d3da23640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-16T16:12:33.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:33.778+0000 7f3d3da23640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-16T16:12:33.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:33.778+0000 7f3d3da23640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-16T16:12:33.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:33.778+0000 7f3d3da23640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-16T16:12:33.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:33.780+0000 7f3d3da23640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-16T16:12:36.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T16:12:36.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T16:12:37.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:37.013+0000 7f726a4f7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-16T16:12:37.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:37.013+0000 7f726a4f7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-16T16:12:37.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:37.013+0000 7f726a4f7640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-16T16:12:37.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:37.013+0000 7f726a4f7640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-16T16:12:37.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:37.013+0000 7f726a4f7640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-16T16:12:37.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:37.016+0000 7f726a4f7640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-16T16:12:39.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T16:12:40.073 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T16:12:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:40.222+0000 7f274b606640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-16T16:12:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:40.222+0000 7f274b606640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-16T16:12:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:40.223+0000 7f274b606640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-16T16:12:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:40.223+0000 7f274b606640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-16T16:12:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:40.223+0000 7f274b606640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-16T16:12:40.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:12:40.225+0000 7f274b606640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-16T16:12:43.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T16:12:43.536 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-16T16:12:43.637 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-16T16:12:43.637 INFO:tasks.ceph:role: osd.4 2024-04-16T16:12:43.637 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:12:43.638 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T16:12:43.739 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T16:12:43.740 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T16:12:43.740 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T16:12:43.740 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T16:12:43.740 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T16:12:43.740 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-16T16:12:43.740 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T16:12:43.740 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T16:12:43.740 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T16:12:43.740 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T16:12:44.157 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-16T16:12:44.165 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-16T16:12:44.165 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-16T16:12:44.337 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-16T16:12:44.433 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-16T16:12:44.518 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-16T16:12:44.518 INFO:tasks.ceph:role: osd.5 2024-04-16T16:12:44.519 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:12:44.519 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T16:12:44.606 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T16:12:44.606 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T16:12:44.606 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T16:12:44.606 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T16:12:44.606 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T16:12:44.607 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-16T16:12:44.607 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T16:12:44.607 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T16:12:44.607 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T16:12:44.607 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T16:12:44.970 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-16T16:12:44.977 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-16T16:12:44.977 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-16T16:12:45.115 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-16T16:12:45.204 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-16T16:12:45.283 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-16T16:12:45.283 INFO:tasks.ceph:role: osd.6 2024-04-16T16:12:45.284 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:12:45.284 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-16T16:12:45.363 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T16:12:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T16:12:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T16:12:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T16:12:45.363 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T16:12:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-16T16:12:45.364 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T16:12:45.364 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T16:12:45.364 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T16:12:45.364 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T16:12:45.747 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-16T16:12:45.753 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-16T16:12:45.753 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-16T16:12:45.914 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-16T16:12:46.017 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-16T16:12:46.109 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-16T16:12:46.109 INFO:tasks.ceph:role: osd.7 2024-04-16T16:12:46.109 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:12:46.109 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-16T16:12:46.208 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T16:12:46.208 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T16:12:46.209 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T16:12:46.209 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T16:12:46.209 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T16:12:46.209 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-16T16:12:46.209 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T16:12:46.209 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T16:12:46.209 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T16:12:46.209 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T16:12:46.620 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-16T16:12:46.626 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-16T16:12:46.626 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-16T16:12:46.750 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-16T16:12:46.836 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T16:12:46.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:46.971+0000 7f088be11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-16T16:12:46.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:46.972+0000 7f088be11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-16T16:12:46.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:46.972+0000 7f088be11640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-16T16:12:46.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:46.972+0000 7f088be11640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-16T16:12:46.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:46.972+0000 7f088be11640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-16T16:12:46.974 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:46.976+0000 7f088be11640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-16T16:12:49.940 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T16:12:50.039 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T16:12:50.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:50.178+0000 7f265e011640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-16T16:12:50.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:50.179+0000 7f265e011640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-16T16:12:50.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:50.179+0000 7f265e011640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-16T16:12:50.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:50.179+0000 7f265e011640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-16T16:12:50.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:50.179+0000 7f265e011640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-16T16:12:50.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:50.181+0000 7f265e011640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-16T16:12:52.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:12:52.744 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:12:53.149 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T16:12:53.243 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T16:12:53.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:53.377+0000 7fb7604c8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-16T16:12:53.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:53.378+0000 7fb7604c8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-16T16:12:53.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:53.378+0000 7fb7604c8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-16T16:12:53.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:53.378+0000 7fb7604c8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-16T16:12:53.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:53.378+0000 7fb7604c8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-16T16:12:53.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:53.380+0000 7fb7604c8640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-16T16:12:56.347 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-16T16:12:56.447 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T16:12:56.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:56.587+0000 7f1fb4e48640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-16T16:12:56.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:56.587+0000 7f1fb4e48640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-16T16:12:56.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:56.587+0000 7f1fb4e48640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-16T16:12:56.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:56.587+0000 7f1fb4e48640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-16T16:12:56.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:56.587+0000 7f1fb4e48640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-16T16:12:56.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:12:56.590+0000 7f1fb4e48640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-16T16:12:59.566 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-16T16:12:59.660 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-16T16:12:59.661 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:12:59.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-16T16:12:59.769 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:12:59.769 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-16T16:12:59.869 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:12:59.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-16T16:12:59.967 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:12:59.967 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-16T16:13:00.058 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:13:00.059 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-16T16:13:00.156 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:13:00.156 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-16T16:13:00.256 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:13:00.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-16T16:13:00.350 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:13:00.350 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-16T16:13:00.452 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:13:00.452 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-16T16:13:00.545 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:13:00.545 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-16T16:13:00.634 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:13:00.635 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-16T16:13:00.730 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:13:00.730 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-16T16:13:00.824 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:13:00.825 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-16T16:13:00.921 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:13:00.921 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-16T16:13:01.008 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:13:01.008 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-16T16:13:01.068 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:13:01.068 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-16T16:13:01.124 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:13:01.124 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-16T16:13:01.181 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:13:01.181 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-16T16:13:01.233 INFO:tasks.ceph:Adding keys to all mons... 2024-04-16T16:13:01.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-16T16:13:01.240 DEBUG:teuthology.orchestra.run.smithi103:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-16T16:13:01.332 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr.y] 2024-04-16T16:13:01.332 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBqox5moiNRIRAAz5JzOrjKNLkcphT+iMo+zw== 2024-04-16T16:13:01.332 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.a] 2024-04-16T16:13:01.332 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBqox5mpjReORAAJMoNf4qvfuxPSiV9+VlKfg== 2024-04-16T16:13:01.332 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.c] 2024-04-16T16:13:01.332 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBrox5m+AaODxAA08XSjc44/i2KYExYQSukaQ== 2024-04-16T16:13:01.332 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.0] 2024-04-16T16:13:01.333 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBuox5m3l3OHxAA+UjwZx8Mn6tCu95TTYIG3w== 2024-04-16T16:13:01.333 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.1] 2024-04-16T16:13:01.333 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxox5mozt1LhAAJL5NtMQdPVBtKOaVlMQBtw== 2024-04-16T16:13:01.333 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.2] 2024-04-16T16:13:01.333 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB1ox5mfE7kABAAIvgZ5Wx+eIJ6QNKj2GVdpw== 2024-04-16T16:13:01.333 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.3] 2024-04-16T16:13:01.333 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB4ox5mEvxcDRAAO0yXBKvsC7hZ6bpOAJyuag== 2024-04-16T16:13:01.333 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr.x] 2024-04-16T16:13:01.333 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBqox5mGevaLBAAPpA6g+Fdi/XHqAqG0QZNag== 2024-04-16T16:13:01.333 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.b] 2024-04-16T16:13:01.333 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBrox5mxHeBIBAA3G8maaK65zaOaqXpAOA6xA== 2024-04-16T16:13:01.333 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.d] 2024-04-16T16:13:01.334 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBrox5mEQ4UMRAAOslSsxQ17bOxPrI6QMlRRw== 2024-04-16T16:13:01.334 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.4] 2024-04-16T16:13:01.334 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB+ox5mZubrORAAcGJLyRN5Ob9aBnopumnEUg== 2024-04-16T16:13:01.334 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.5] 2024-04-16T16:13:01.334 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCCox5mcRqsChAACktmuUHvqWkp/mHT3UcjRw== 2024-04-16T16:13:01.334 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.6] 2024-04-16T16:13:01.334 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCFox5mT52FFhAAChhDDfNZmIMgxE1j90LZpQ== 2024-04-16T16:13:01.334 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.7] 2024-04-16T16:13:01.334 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCIox5mC+EDIxAAxtvM0on00lxTLaoRjN0Gfw== 2024-04-16T16:13:01.334 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-04-16T16:13:01.334 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ== 2024-04-16T16:13:01.334 INFO:teuthology.orchestra.run.smithi045.stdout:[client.2] 2024-04-16T16:13:01.335 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mWdlUDxAAuO2FsGH1/E+qP1xcyUEe3A== 2024-04-16T16:13:01.335 INFO:teuthology.orchestra.run.smithi045.stdout:[client.1] 2024-04-16T16:13:01.335 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mYpVUGhAAYdR+40O6zbWqgbI55F0FYA== 2024-04-16T16:13:01.335 INFO:teuthology.orchestra.run.smithi045.stdout:[client.3] 2024-04-16T16:13:01.335 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5miPdMJBAAqjKpctNYu9ot1lff3rouxg== 2024-04-16T16:13:01.335 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr.y] 2024-04-16T16:13:01.336 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBqox5moiNRIRAAz5JzOrjKNLkcphT+iMo+zw== 2024-04-16T16:13:01.336 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.a] 2024-04-16T16:13:01.336 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBqox5mpjReORAAJMoNf4qvfuxPSiV9+VlKfg== 2024-04-16T16:13:01.336 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.c] 2024-04-16T16:13:01.336 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBrox5m+AaODxAA08XSjc44/i2KYExYQSukaQ== 2024-04-16T16:13:01.336 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.0] 2024-04-16T16:13:01.336 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBuox5m3l3OHxAA+UjwZx8Mn6tCu95TTYIG3w== 2024-04-16T16:13:01.337 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.1] 2024-04-16T16:13:01.337 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBxox5mozt1LhAAJL5NtMQdPVBtKOaVlMQBtw== 2024-04-16T16:13:01.337 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.2] 2024-04-16T16:13:01.337 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQB1ox5mfE7kABAAIvgZ5Wx+eIJ6QNKj2GVdpw== 2024-04-16T16:13:01.337 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.3] 2024-04-16T16:13:01.337 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQB4ox5mEvxcDRAAO0yXBKvsC7hZ6bpOAJyuag== 2024-04-16T16:13:01.337 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr.x] 2024-04-16T16:13:01.337 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBqox5mGevaLBAAPpA6g+Fdi/XHqAqG0QZNag== 2024-04-16T16:13:01.337 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.b] 2024-04-16T16:13:01.337 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBrox5mxHeBIBAA3G8maaK65zaOaqXpAOA6xA== 2024-04-16T16:13:01.337 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.d] 2024-04-16T16:13:01.337 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBrox5mEQ4UMRAAOslSsxQ17bOxPrI6QMlRRw== 2024-04-16T16:13:01.338 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.4] 2024-04-16T16:13:01.338 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQB+ox5mZubrORAAcGJLyRN5Ob9aBnopumnEUg== 2024-04-16T16:13:01.338 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.5] 2024-04-16T16:13:01.338 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCCox5mcRqsChAACktmuUHvqWkp/mHT3UcjRw== 2024-04-16T16:13:01.338 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.6] 2024-04-16T16:13:01.338 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCFox5mT52FFhAAChhDDfNZmIMgxE1j90LZpQ== 2024-04-16T16:13:01.338 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.7] 2024-04-16T16:13:01.338 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCIox5mC+EDIxAAxtvM0on00lxTLaoRjN0Gfw== 2024-04-16T16:13:01.339 INFO:teuthology.orchestra.run.smithi103.stdout:[client.0] 2024-04-16T16:13:01.339 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ== 2024-04-16T16:13:01.339 INFO:teuthology.orchestra.run.smithi103.stdout:[client.2] 2024-04-16T16:13:01.339 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBsox5mWdlUDxAAuO2FsGH1/E+qP1xcyUEe3A== 2024-04-16T16:13:01.339 INFO:teuthology.orchestra.run.smithi103.stdout:[client.1] 2024-04-16T16:13:01.339 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBsox5mYpVUGhAAYdR+40O6zbWqgbI55F0FYA== 2024-04-16T16:13:01.339 INFO:teuthology.orchestra.run.smithi103.stdout:[client.3] 2024-04-16T16:13:01.339 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBsox5miPdMJBAAqjKpctNYu9ot1lff3rouxg== 2024-04-16T16:13:01.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-16T16:13:01.350 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-16T16:13:01.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-16T16:13:01.504 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-16T16:13:01.656 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-16T16:13:01.662 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-16T16:13:01.806 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:01.813 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:01.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:02.228 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:02.377 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:02.383 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:02.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:02.535 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:02.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-16T16:13:02.687 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-16T16:13:02.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-16T16:13:02.838 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-16T16:13:02.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-16T16:13:02.993 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-16T16:13:03.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:03.144 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:03.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:03.291 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:03.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:03.437 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:03.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:03.591 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T16:13:03.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T16:13:03.743 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T16:13:03.894 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T16:13:03.900 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T16:13:04.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T16:13:04.058 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T16:13:04.201 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T16:13:04.206 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T16:13:04.360 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-16T16:13:04.361 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-16T16:13:04.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-16T16:13:04.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-16T16:13:04.699 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-16T16:13:04.797 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-16T16:13:04.955 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-16T16:13:05.042 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-16T16:13:05.129 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-16T16:13:05.276 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-16T16:13:05.349 DEBUG:teuthology.orchestra.run.smithi045:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-16T16:13:05.356 DEBUG:teuthology.orchestra.run.smithi103:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-16T16:13:05.408 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-16T16:13:05.409 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-16T16:13:05.409 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-16T16:13:05.414 INFO:tasks.ceph.mon.a:Started 2024-04-16T16:13:05.414 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-16T16:13:05.414 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-16T16:13:05.418 INFO:tasks.ceph.mon.b:Started 2024-04-16T16:13:05.418 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-16T16:13:05.418 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-16T16:13:05.423 INFO:tasks.ceph.mon.c:Started 2024-04-16T16:13:05.423 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-16T16:13:05.423 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-16T16:13:05.423 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-16T16:13:05.428 INFO:tasks.ceph.mgr.y:Started 2024-04-16T16:13:05.428 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-16T16:13:05.428 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-16T16:13:05.432 INFO:tasks.ceph.mgr.x:Started 2024-04-16T16:13:05.432 INFO:tasks.ceph:Setting crush tunables to default 2024-04-16T16:13:05.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd crush tunables default 2024-04-16T16:13:05.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.618+0000 7f8183200640 1 Processor -- start 2024-04-16T16:13:05.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.618+0000 7f8183200640 1 -- start start 2024-04-16T16:13:05.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.619+0000 7f8183200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f817c15eef0 0x7f817c0d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:05.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.619+0000 7f8183200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f817c0d42d0 0x7f817c0b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:05.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.620+0000 7f8183200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f817c0d35a0 con 0x7f817c0cf5c0 2024-04-16T16:13:05.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.620+0000 7f8183200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f817c0d2740 con 0x7f817c0d42d0 2024-04-16T16:13:05.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.620+0000 7f8183200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f817c0d2a20 con 0x7f817c15eef0 2024-04-16T16:13:05.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.620+0000 7f8182800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f817c0d42d0 0x7f817c0b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:05.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.620+0000 7f8182800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f817c0d42d0 0x7f817c0b79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49250/0 (socket says 172.21.15.45:49250) 2024-04-16T16:13:05.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.620+0000 7f8182800640 1 -- 172.21.15.45:0/2126011791 learned_addr learned my addr 172.21.15.45:0/2126011791 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:05.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.620+0000 7f8182800640 1 -- 172.21.15.45:0/2126011791 >> v2:172.21.15.45:3300/0 conn(0x7f817c0d42d0 msgr2=0x7f817c0b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-16T16:13:05.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.620+0000 7f8182800640 1 -- 172.21.15.45:0/2126011791 >> v2:172.21.15.45:3300/0 conn(0x7f817c0d42d0 msgr2=0x7f817c0b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-16T16:13:05.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.620+0000 7f8182800640 1 --2- 172.21.15.45:0/2126011791 >> v2:172.21.15.45:3300/0 conn(0x7f817c0d42d0 0x7f817c0b79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-16T16:13:05.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.620+0000 7f8182800640 1 --2- 172.21.15.45:0/2126011791 >> v2:172.21.15.45:3300/0 conn(0x7f817c0d42d0 0x7f817c0b79d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:13:05.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.620+0000 7f8181400640 1 --2- 172.21.15.45:0/2126011791 >> v2:172.21.15.103:3300/0 conn(0x7f817c15eef0 0x7f817c0d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:05.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.621+0000 7f8181400640 1 -- 172.21.15.45:0/2126011791 >> v2:172.21.15.103:3300/0 conn(0x7f817c15eef0 msgr2=0x7f817c0d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-16T16:13:05.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.621+0000 7f8181400640 1 -- 172.21.15.45:0/2126011791 >> v2:172.21.15.103:3300/0 conn(0x7f817c15eef0 msgr2=0x7f817c0d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-16T16:13:05.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.621+0000 7f8181400640 1 --2- 172.21.15.45:0/2126011791 >> v2:172.21.15.103:3300/0 conn(0x7f817c15eef0 0x7f817c0d3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-16T16:13:05.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.621+0000 7f8181400640 1 --2- 172.21.15.45:0/2126011791 >> v2:172.21.15.103:3300/0 conn(0x7f817c15eef0 0x7f817c0d3980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:13:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.821+0000 7f8182800640 1 --2- 172.21.15.45:0/2126011791 >> v2:172.21.15.45:3300/0 conn(0x7f817c0d42d0 0x7f817c0b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.822+0000 7f8182800640 1 -- 172.21.15.45:0/2126011791 >> v2:172.21.15.45:3300/0 conn(0x7f817c0d42d0 msgr2=0x7f817c0b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-16T16:13:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.822+0000 7f8182800640 1 -- 172.21.15.45:0/2126011791 >> v2:172.21.15.45:3300/0 conn(0x7f817c0d42d0 msgr2=0x7f817c0b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-16T16:13:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.822+0000 7f8182800640 1 --2- 172.21.15.45:0/2126011791 >> v2:172.21.15.45:3300/0 conn(0x7f817c0d42d0 0x7f817c0b79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-16T16:13:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.822+0000 7f8182800640 1 --2- 172.21.15.45:0/2126011791 >> v2:172.21.15.45:3300/0 conn(0x7f817c0d42d0 0x7f817c0b79d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:13:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.823+0000 7f8181400640 1 --2- 172.21.15.45:0/2126011791 >> v2:172.21.15.103:3300/0 conn(0x7f817c15eef0 0x7f817c0d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.823+0000 7f8181400640 1 -- 172.21.15.45:0/2126011791 >> v2:172.21.15.103:3300/0 conn(0x7f817c15eef0 msgr2=0x7f817c0d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-16T16:13:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.823+0000 7f8181400640 1 -- 172.21.15.45:0/2126011791 >> v2:172.21.15.103:3300/0 conn(0x7f817c15eef0 msgr2=0x7f817c0d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-16T16:13:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.823+0000 7f8181400640 1 --2- 172.21.15.45:0/2126011791 >> v2:172.21.15.103:3300/0 conn(0x7f817c15eef0 0x7f817c0d3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-16T16:13:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.823+0000 7f8181400640 1 --2- 172.21.15.45:0/2126011791 >> v2:172.21.15.103:3300/0 conn(0x7f817c15eef0 0x7f817c0d3980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:13:05.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.877+0000 7f8180a00640 1 -- 172.21.15.45:0/2126011791 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3511416503 0 0) 0x7f817c0d35a0 con 0x7f817c0cf5c0 2024-04-16T16:13:05.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.877+0000 7f8180a00640 1 -- 172.21.15.45:0/2126011791 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8164003520 con 0x7f817c0cf5c0 2024-04-16T16:13:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.878+0000 7f8180a00640 1 -- 172.21.15.45:0/2126011791 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1223086593 0 0) 0x7f8164003520 con 0x7f817c0cf5c0 2024-04-16T16:13:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.878+0000 7f8180a00640 1 -- 172.21.15.45:0/2126011791 >> v2:172.21.15.103:3300/0 conn(0x7f817c15eef0 msgr2=0x7f817c0d3980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-16T16:13:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.878+0000 7f8180a00640 1 --2- 172.21.15.45:0/2126011791 >> v2:172.21.15.103:3300/0 conn(0x7f817c15eef0 0x7f817c0d3980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.878+0000 7f8180a00640 1 -- 172.21.15.45:0/2126011791 >> v2:172.21.15.45:3300/0 conn(0x7f817c0d42d0 msgr2=0x7f817c0b79d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-16T16:13:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.878+0000 7f8180a00640 1 --2- 172.21.15.45:0/2126011791 >> v2:172.21.15.45:3300/0 conn(0x7f817c0d42d0 0x7f817c0b79d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.878+0000 7f8180a00640 1 -- 172.21.15.45:0/2126011791 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c0d2d00 con 0x7f817c0cf5c0 2024-04-16T16:13:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.878+0000 7f8180a00640 1 -- 172.21.15.45:0/2126011791 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f8170001670 con 0x7f817c0cf5c0 2024-04-16T16:13:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.878+0000 7f8180a00640 1 -- 172.21.15.45:0/2126011791 >> v1:172.21.15.45:6789/0 conn(0x7f817c0cf5c0 legacy=0x7f817c0cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.878+0000 7f8180a00640 1 --2- 172.21.15.45:0/2126011791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8164003a00 0x7f8164003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.878+0000 7f8180a00640 1 --2- 172.21.15.45:0/2126011791 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8164004800 0x7f8164004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.878+0000 7f8180a00640 1 --2- 172.21.15.45:0/2126011791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81640051c0 0x7f816400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.878+0000 7f8180a00640 1 -- 172.21.15.45:0/2126011791 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f817c0d2740 con 0x7f81640051c0 2024-04-16T16:13:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.878+0000 7f8180a00640 1 -- 172.21.15.45:0/2126011791 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f817c0d2a20 con 0x7f8164003a00 2024-04-16T16:13:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.878+0000 7f8180a00640 1 -- 172.21.15.45:0/2126011791 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f816400c110 con 0x7f8164004800 2024-04-16T16:13:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.879+0000 7f8181400640 1 --2- 172.21.15.45:0/2126011791 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8164004800 0x7f8164004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.879+0000 7f8182800640 1 --2- 172.21.15.45:0/2126011791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81640051c0 0x7f816400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.880+0000 7f8181400640 1 -- 172.21.15.45:0/2126011791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8164003a00 msgr2=0x7f8164003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:05.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.880+0000 7f8181400640 1 --2- 172.21.15.45:0/2126011791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8164003a00 0x7f8164003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:05.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.880+0000 7f8181400640 1 -- 172.21.15.45:0/2126011791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81640051c0 msgr2=0x7f816400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:05.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.880+0000 7f8181400640 1 --2- 172.21.15.45:0/2126011791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81640051c0 0x7f816400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.880+0000 7f8181400640 1 -- 172.21.15.45:0/2126011791 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8164003520 con 0x7f8164004800 2024-04-16T16:13:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.880+0000 7f8182800640 1 -- 172.21.15.45:0/2126011791 reap_dead start 2024-04-16T16:13:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.882+0000 7f8181400640 1 --2- 172.21.15.45:0/2126011791 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8164004800 0x7f8164004c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f816c00da90 tx=0x7f8164005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6138b0da1d1e8f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.883+0000 7f8180a00640 1 -- 172.21.15.45:0/2126011791 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c032070 con 0x7f8164004800 2024-04-16T16:13:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.883+0000 7f8180a00640 1 -- 172.21.15.45:0/2126011791 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f816c02c7d0 con 0x7f8164004800 2024-04-16T16:13:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.883+0000 7f8180a00640 1 -- 172.21.15.45:0/2126011791 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c00d690 con 0x7f8164004800 2024-04-16T16:13:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.884+0000 7f8183200640 1 -- 172.21.15.45:0/2126011791 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8164004800 msgr2=0x7f8164004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.884+0000 7f8183200640 1 --2- 172.21.15.45:0/2126011791 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8164004800 0x7f8164004c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f816c00da90 tx=0x7f8164005830 comp rx=0 tx=0).stop 2024-04-16T16:13:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.884+0000 7f8183200640 1 -- 172.21.15.45:0/2126011791 shutdown_connections 2024-04-16T16:13:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.884+0000 7f8183200640 1 --2- 172.21.15.45:0/2126011791 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8164004800 0x7f8164004c80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.884+0000 7f8183200640 1 -- 172.21.15.45:0/2126011791 >> 172.21.15.45:0/2126011791 conn(0x7f817c06d300 msgr2=0x7f817c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.884+0000 7f8183200640 1 -- 172.21.15.45:0/2126011791 shutdown_connections 2024-04-16T16:13:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.885+0000 7f8183200640 1 -- 172.21.15.45:0/2126011791 wait complete. 2024-04-16T16:13:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.887+0000 7f8183200640 1 Processor -- start 2024-04-16T16:13:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.888+0000 7f8183200640 1 -- start start 2024-04-16T16:13:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.889+0000 7f8183200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8164004800 0x7f817c154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.889+0000 7f8183200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f817c0cae00 0x7f817c14acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.889+0000 7f8183200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f817c1568b0 0x7f817c156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.889+0000 7f8183200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f817c0d3980 con 0x7f817c1568b0 2024-04-16T16:13:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.889+0000 7f8183200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f817c0d32c0 con 0x7f817c0cae00 2024-04-16T16:13:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.889+0000 7f8183200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f817c0d3b00 con 0x7f8164004800 2024-04-16T16:13:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.890+0000 7f8182800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f817c1568b0 0x7f817c156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.890+0000 7f8182800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f817c1568b0 0x7f817c156d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49290/0 (socket says 172.21.15.45:49290) 2024-04-16T16:13:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.890+0000 7f8182800640 1 -- 172.21.15.45:0/2184712646 learned_addr learned my addr 172.21.15.45:0/2184712646 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.890+0000 7f8181400640 1 --2- 172.21.15.45:0/2184712646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f817c0cae00 0x7f817c14acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.890+0000 7f8181e00640 1 --2- 172.21.15.45:0/2184712646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8164004800 0x7f817c154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.890+0000 7f8182800640 1 -- 172.21.15.45:0/2184712646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8164004800 msgr2=0x7f817c154cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.890+0000 7f8182800640 1 --2- 172.21.15.45:0/2184712646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8164004800 0x7f817c154cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.890+0000 7f8182800640 1 -- 172.21.15.45:0/2184712646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f817c0cae00 msgr2=0x7f817c14acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.890+0000 7f8182800640 1 --2- 172.21.15.45:0/2184712646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f817c0cae00 0x7f817c14acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.890+0000 7f8182800640 1 -- 172.21.15.45:0/2184712646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c0cb260 con 0x7f817c1568b0 2024-04-16T16:13:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.890+0000 7f8181400640 1 --2- 172.21.15.45:0/2184712646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f817c0cae00 0x7f817c14acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.891+0000 7f8182800640 1 --2- 172.21.15.45:0/2184712646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f817c1568b0 0x7f817c156d60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f81780096d0 tx=0x7f8178007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.891+0000 7f8176000640 1 -- 172.21.15.45:0/2184712646 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8178011af0 con 0x7f817c1568b0 2024-04-16T16:13:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.891+0000 7f8176000640 1 -- 172.21.15.45:0/2184712646 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f817801c070 con 0x7f817c1568b0 2024-04-16T16:13:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.891+0000 7f8183200640 1 -- 172.21.15.45:0/2184712646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f817c14b230 con 0x7f817c1568b0 2024-04-16T16:13:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.891+0000 7f8176000640 1 -- 172.21.15.45:0/2184712646 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8178016450 con 0x7f817c1568b0 2024-04-16T16:13:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.891+0000 7f8183200640 1 -- 172.21.15.45:0/2184712646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f817c1578f0 con 0x7f817c1568b0 2024-04-16T16:13:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.892+0000 7f8176000640 1 -- 172.21.15.45:0/2184712646 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f81780165f0 con 0x7f817c1568b0 2024-04-16T16:13:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.892+0000 7f8176000640 1 -- 172.21.15.45:0/2184712646 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f81780219a0 con 0x7f817c1568b0 2024-04-16T16:13:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.893+0000 7f8183200640 1 -- 172.21.15.45:0/2184712646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f817c075780 con 0x7f817c1568b0 2024-04-16T16:13:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.897+0000 7f8176000640 1 -- 172.21.15.45:0/2184712646 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8178021460 con 0x7f817c1568b0 2024-04-16T16:13:05.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.972+0000 7f8183200640 1 -- 172.21.15.45:0/2184712646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f817c0d3fd0 con 0x7f817c1568b0 2024-04-16T16:13:05.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.976+0000 7f8176000640 1 -- 172.21.15.45:0/2184712646 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f8178016a70 con 0x7f817c1568b0 2024-04-16T16:13:05.975 INFO:teuthology.orchestra.run.smithi045.stderr:adjusted tunables profile to default 2024-04-16T16:13:05.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.978+0000 7f8183200640 1 -- 172.21.15.45:0/2184712646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f817c1568b0 msgr2=0x7f817c156d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:05.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.978+0000 7f8183200640 1 --2- 172.21.15.45:0/2184712646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f817c1568b0 0x7f817c156d60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f81780096d0 tx=0x7f8178007f60 comp rx=0 tx=0).stop 2024-04-16T16:13:05.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.978+0000 7f8183200640 1 -- 172.21.15.45:0/2184712646 shutdown_connections 2024-04-16T16:13:05.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.978+0000 7f8183200640 1 --2- 172.21.15.45:0/2184712646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f817c1568b0 0x7f817c156d60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:05.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.978+0000 7f8183200640 1 --2- 172.21.15.45:0/2184712646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f817c0cae00 0x7f817c14acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:05.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.978+0000 7f8183200640 1 --2- 172.21.15.45:0/2184712646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8164004800 0x7f817c154cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:05.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.978+0000 7f8183200640 1 -- 172.21.15.45:0/2184712646 >> 172.21.15.45:0/2184712646 conn(0x7f817c06d300 msgr2=0x7f817c0d4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:05.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.978+0000 7f8183200640 1 -- 172.21.15.45:0/2184712646 shutdown_connections 2024-04-16T16:13:05.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:05.978+0000 7f8183200640 1 -- 172.21.15.45:0/2184712646 wait complete. 2024-04-16T16:13:05.996 INFO:tasks.ceph:check_enable_crimson: False 2024-04-16T16:13:05.996 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-16T16:13:05.997 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:13:05.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-16T16:13:06.091 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:13:06.091 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-16T16:13:06.189 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:13:06.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-16T16:13:06.271 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:13:06.271 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-16T16:13:06.409 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:13:06.409 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-16T16:13:06.474 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:06.474+0000 7f76089db040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:13:06.499 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:13:06.499 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-16T16:13:06.538 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:06.539+0000 7f76089db040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:13:06.637 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:13:06.637 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-16T16:13:06.762 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:13:06.763 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-16T16:13:06.807 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:06.808+0000 7fc088893040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:13:06.874 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:06.875+0000 7fc088893040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:13:06.886 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 3887da33-8db2-468b-a30b-3835ff05c375 0 2024-04-16T16:13:07.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.124+0000 7f36bf200640 1 Processor -- start 2024-04-16T16:13:07.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.124+0000 7f36bf200640 1 -- start start 2024-04-16T16:13:07.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.125+0000 7f36bf200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f36c015ea60 0x7f36c00ba100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:07.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.125+0000 7f36bf200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f36c00d3d60 0x7f36c00d1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:07.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.125+0000 7f36bf200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36c00d2cc0 con 0x7f36c00d17a0 2024-04-16T16:13:07.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.125+0000 7f36bf200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f36c00d1bf0 con 0x7f36c015ea60 2024-04-16T16:13:07.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.125+0000 7f36bf200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f36c00b8920 con 0x7f36c00d3d60 2024-04-16T16:13:07.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.126+0000 7f36bd400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f36c00d3d60 0x7f36c00d1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:07.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.126+0000 7f36be800640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f36c00d17a0 0x7f36c00b62d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:59752/0 (socket says 172.21.15.103:59752) 2024-04-16T16:13:07.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.126+0000 7f36be800640 1 -- 172.21.15.103:0/3196067204 learned_addr learned my addr 172.21.15.103:0/3196067204 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-04-16T16:13:07.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.126+0000 7f36bd400640 1 -- 172.21.15.103:0/3196067204 >> v2:172.21.15.45:3300/0 conn(0x7f36c015ea60 msgr2=0x7f36c00ba100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:07.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.126+0000 7f36bde00640 1 --2- 172.21.15.103:0/3196067204 >> v2:172.21.15.45:3300/0 conn(0x7f36c015ea60 0x7f36c00ba100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:07.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.126+0000 7f36bd400640 1 --2- 172.21.15.103:0/3196067204 >> v2:172.21.15.45:3300/0 conn(0x7f36c015ea60 0x7f36c00ba100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.126+0000 7f36bca00640 1 -- 172.21.15.103:0/3196067204 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4066153101 0 0) 0x7f36c00d2cc0 con 0x7f36c00d17a0 2024-04-16T16:13:07.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.126+0000 7f36bd400640 1 -- 172.21.15.103:0/3196067204 >> v1:172.21.15.103:6789/0 conn(0x7f36c00d17a0 legacy=0x7f36c00b62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:07.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.126+0000 7f36bd400640 1 -- 172.21.15.103:0/3196067204 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36c00d2fa0 con 0x7f36c00d3d60 2024-04-16T16:13:07.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.126+0000 7f36bde00640 1 --2- 172.21.15.103:0/3196067204 >> v2:172.21.15.45:3300/0 conn(0x7f36c015ea60 0x7f36c00ba100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:07.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.128+0000 7f36bd400640 1 --2- 172.21.15.103:0/3196067204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36c00d3d60 0x7f36c00d1260 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f36c00cf0a0 tx=0x7f36ac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddbc4f5e750fc0c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:07.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.128+0000 7f36bca00640 1 -- 172.21.15.103:0/3196067204 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac035020 con 0x7f36c00d3d60 2024-04-16T16:13:07.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.128+0000 7f36bca00640 1 -- 172.21.15.103:0/3196067204 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36ac031830 con 0x7f36c00d3d60 2024-04-16T16:13:07.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.128+0000 7f36bca00640 1 -- 172.21.15.103:0/3196067204 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac031c00 con 0x7f36c00d3d60 2024-04-16T16:13:07.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.129+0000 7f36bf200640 1 -- 172.21.15.103:0/3196067204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36c00d3d60 msgr2=0x7f36c00d1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:07.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.129+0000 7f36bf200640 1 --2- 172.21.15.103:0/3196067204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36c00d3d60 0x7f36c00d1260 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f36c00cf0a0 tx=0x7f36ac02b8d0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.129+0000 7f36bf200640 1 -- 172.21.15.103:0/3196067204 shutdown_connections 2024-04-16T16:13:07.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.129+0000 7f36bf200640 1 --2- 172.21.15.103:0/3196067204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36c00d3d60 0x7f36c00d1260 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.129+0000 7f36bf200640 1 --2- 172.21.15.103:0/3196067204 >> v2:172.21.15.45:3300/0 conn(0x7f36c015ea60 0x7f36c00ba100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.129+0000 7f36bf200640 1 -- 172.21.15.103:0/3196067204 >> 172.21.15.103:0/3196067204 conn(0x7f36c0073690 msgr2=0x7f36c006cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:07.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.129+0000 7f36bf200640 1 -- 172.21.15.103:0/3196067204 shutdown_connections 2024-04-16T16:13:07.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.129+0000 7f36bf200640 1 -- 172.21.15.103:0/3196067204 wait complete. 2024-04-16T16:13:07.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.131+0000 7f36bf200640 1 Processor -- start 2024-04-16T16:13:07.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.131+0000 7f36bf200640 1 -- start start 2024-04-16T16:13:07.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.132+0000 7f36bf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f36c00d17a0 0x7f36c015ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:07.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.132+0000 7f36bf200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36c00d3d60 0x7f36c00b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:07.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.132+0000 7f36bf200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36c015ea60 0x7f36c00b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:07.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.132+0000 7f36bf200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f36c00d3560 con 0x7f36c00d3d60 2024-04-16T16:13:07.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.132+0000 7f36bf200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f36c00d38d0 con 0x7f36c015ea60 2024-04-16T16:13:07.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.132+0000 7f36bf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f36c015ee40 con 0x7f36c00d17a0 2024-04-16T16:13:07.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.132+0000 7f36be800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36c015ea60 0x7f36c00b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:07.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.132+0000 7f36bde00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f36c00d17a0 0x7f36c015ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:07.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.132+0000 7f36be800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36c015ea60 0x7f36c00b5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48976/0 (socket says 172.21.15.103:48976) 2024-04-16T16:13:07.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.132+0000 7f36be800640 1 -- 172.21.15.103:0/281057391 learned_addr learned my addr 172.21.15.103:0/281057391 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-16T16:13:07.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.132+0000 7f36bd400640 1 --2- 172.21.15.103:0/281057391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36c00d3d60 0x7f36c00b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:07.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.133+0000 7f36be800640 1 -- 172.21.15.103:0/281057391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f36c00d17a0 msgr2=0x7f36c015ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:07.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.133+0000 7f36be800640 1 --2- 172.21.15.103:0/281057391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f36c00d17a0 0x7f36c015ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.133+0000 7f36be800640 1 -- 172.21.15.103:0/281057391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36c00d3d60 msgr2=0x7f36c00b58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:07.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.133+0000 7f36be800640 1 --2- 172.21.15.103:0/281057391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36c00d3d60 0x7f36c00b58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.133+0000 7f36be800640 1 -- 172.21.15.103:0/281057391 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36c01f4180 con 0x7f36c015ea60 2024-04-16T16:13:07.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.133+0000 7f36bde00640 1 --2- 172.21.15.103:0/281057391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f36c00d17a0 0x7f36c015ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:07.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.133+0000 7f36be800640 1 --2- 172.21.15.103:0/281057391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36c015ea60 0x7f36c00b5de0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f36b800c7a0 tx=0x7f36b800cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:07.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.133+0000 7f36b6000640 1 -- 172.21.15.103:0/281057391 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b8018070 con 0x7f36c015ea60 2024-04-16T16:13:07.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.133+0000 7f36b6000640 1 -- 172.21.15.103:0/281057391 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36b800a640 con 0x7f36c015ea60 2024-04-16T16:13:07.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.133+0000 7f36b6000640 1 -- 172.21.15.103:0/281057391 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b800aa70 con 0x7f36c015ea60 2024-04-16T16:13:07.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.134+0000 7f36bf200640 1 -- 172.21.15.103:0/281057391 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36c01f4460 con 0x7f36c015ea60 2024-04-16T16:13:07.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.134+0000 7f36bf200640 1 -- 172.21.15.103:0/281057391 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36c01f4a20 con 0x7f36c015ea60 2024-04-16T16:13:07.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.134+0000 7f36b6000640 1 -- 172.21.15.103:0/281057391 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f36b800ac10 con 0x7f36c015ea60 2024-04-16T16:13:07.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.134+0000 7f36b6000640 1 -- 172.21.15.103:0/281057391 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f36b8012490 con 0x7f36c015ea60 2024-04-16T16:13:07.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.135+0000 7f36bf200640 1 -- 172.21.15.103:0/281057391 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36c0075780 con 0x7f36c015ea60 2024-04-16T16:13:07.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.138+0000 7f36b6000640 1 -- 172.21.15.103:0/281057391 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f36b8021070 con 0x7f36c015ea60 2024-04-16T16:13:07.217 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:07.218+0000 7f76089db040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:13:07.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.249+0000 7f36bf200640 1 -- 172.21.15.103:0/281057391 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3887da33-8db2-468b-a30b-3835ff05c375", "id": 0} v 0) -- 0x7f36c01f3e70 con 0x7f36c015ea60 2024-04-16T16:13:07.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.255+0000 7f36b6000640 1 -- 172.21.15.103:0/281057391 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3887da33-8db2-468b-a30b-3835ff05c375", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f36b80070d0 con 0x7f36c015ea60 2024-04-16T16:13:07.255 INFO:teuthology.orchestra.run.smithi103.stdout:0 2024-04-16T16:13:07.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.257+0000 7f36bf200640 1 -- 172.21.15.103:0/281057391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36c015ea60 msgr2=0x7f36c00b5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:07.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.257+0000 7f36bf200640 1 --2- 172.21.15.103:0/281057391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36c015ea60 0x7f36c00b5de0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f36b800c7a0 tx=0x7f36b800cc60 comp rx=0 tx=0).stop 2024-04-16T16:13:07.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.257+0000 7f36bf200640 1 -- 172.21.15.103:0/281057391 shutdown_connections 2024-04-16T16:13:07.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.257+0000 7f36bf200640 1 --2- 172.21.15.103:0/281057391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36c015ea60 0x7f36c00b5de0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.257+0000 7f36bf200640 1 --2- 172.21.15.103:0/281057391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36c00d3d60 0x7f36c00b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.257+0000 7f36bf200640 1 --2- 172.21.15.103:0/281057391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f36c00d17a0 0x7f36c015ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.257+0000 7f36bf200640 1 -- 172.21.15.103:0/281057391 >> 172.21.15.103:0/281057391 conn(0x7f36c0073690 msgr2=0x7f36c006cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:07.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.257+0000 7f36bf200640 1 -- 172.21.15.103:0/281057391 shutdown_connections 2024-04-16T16:13:07.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.257+0000 7f36bf200640 1 -- 172.21.15.103:0/281057391 wait complete. 2024-04-16T16:13:07.274 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 361acb09-4d66-455b-a5b9-c1d9a8794735 1 2024-04-16T16:13:07.286 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:07.287+0000 7f76089db040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:13:07.341 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:07.341+0000 7f76089db040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:13:07.408 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:07.408+0000 7f76089db040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:13:07.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.459+0000 7f7a5f200640 1 Processor -- start 2024-04-16T16:13:07.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.459+0000 7f7a5f200640 1 -- start start 2024-04-16T16:13:07.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.460+0000 7f7a5f200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7a6015eef0 0x7f7a600d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:07.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.460+0000 7f7a5f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7a600d42d0 0x7f7a600b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:07.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.460+0000 7f7a5f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a600d35a0 con 0x7f7a600cf5c0 2024-04-16T16:13:07.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.460+0000 7f7a5f200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f7a600d2740 con 0x7f7a6015eef0 2024-04-16T16:13:07.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.460+0000 7f7a5f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7a600d2a20 con 0x7f7a600d42d0 2024-04-16T16:13:07.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.460+0000 7f7a5e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7a600d42d0 0x7f7a600b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:07.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.460+0000 7f7a5de00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7a600cf5c0 0x7f7a600cf9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:59766/0 (socket says 172.21.15.103:59766) 2024-04-16T16:13:07.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.460+0000 7f7a5de00640 1 -- 172.21.15.103:0/945582085 learned_addr learned my addr 172.21.15.103:0/945582085 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-04-16T16:13:07.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.460+0000 7f7a57e00640 1 --2- 172.21.15.103:0/945582085 >> v2:172.21.15.45:3300/0 conn(0x7f7a6015eef0 0x7f7a600d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:07.461 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:07.461+0000 7f76089db040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:13:07.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.461+0000 7f7a5d400640 1 -- 172.21.15.103:0/945582085 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3364655801 0 0) 0x7f7a600d35a0 con 0x7f7a600cf5c0 2024-04-16T16:13:07.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.461+0000 7f7a5e800640 1 -- 172.21.15.103:0/945582085 >> v2:172.21.15.45:3300/0 conn(0x7f7a6015eef0 msgr2=0x7f7a600d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:07.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.461+0000 7f7a5e800640 1 --2- 172.21.15.103:0/945582085 >> v2:172.21.15.45:3300/0 conn(0x7f7a6015eef0 0x7f7a600d3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.461+0000 7f7a5e800640 1 -- 172.21.15.103:0/945582085 >> v1:172.21.15.103:6789/0 conn(0x7f7a600cf5c0 legacy=0x7f7a600cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:07.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.461+0000 7f7a5e800640 1 -- 172.21.15.103:0/945582085 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a600d2d00 con 0x7f7a600d42d0 2024-04-16T16:13:07.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.462+0000 7f7a5e800640 1 --2- 172.21.15.103:0/945582085 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7a600d42d0 0x7f7a600b79d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7a58009d90 tx=0x7f7a5802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d0912dfe94a70f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:07.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.463+0000 7f7a5d400640 1 -- 172.21.15.103:0/945582085 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a58039070 con 0x7f7a600d42d0 2024-04-16T16:13:07.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.463+0000 7f7a5d400640 1 -- 172.21.15.103:0/945582085 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a5802cc50 con 0x7f7a600d42d0 2024-04-16T16:13:07.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.463+0000 7f7a5d400640 1 -- 172.21.15.103:0/945582085 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a58034440 con 0x7f7a600d42d0 2024-04-16T16:13:07.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.463+0000 7f7a5f200640 1 -- 172.21.15.103:0/945582085 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7a600d42d0 msgr2=0x7f7a600b79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:07.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.463+0000 7f7a5f200640 1 --2- 172.21.15.103:0/945582085 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7a600d42d0 0x7f7a600b79d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7a58009d90 tx=0x7f7a5802bea0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.463+0000 7f7a5f200640 1 -- 172.21.15.103:0/945582085 shutdown_connections 2024-04-16T16:13:07.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.463+0000 7f7a5f200640 1 --2- 172.21.15.103:0/945582085 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7a600d42d0 0x7f7a600b79d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.463+0000 7f7a5f200640 1 --2- 172.21.15.103:0/945582085 >> v2:172.21.15.45:3300/0 conn(0x7f7a6015eef0 0x7f7a600d3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.463+0000 7f7a5f200640 1 -- 172.21.15.103:0/945582085 >> 172.21.15.103:0/945582085 conn(0x7f7a6006d300 msgr2=0x7f7a600d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:07.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.463+0000 7f7a5f200640 1 -- 172.21.15.103:0/945582085 shutdown_connections 2024-04-16T16:13:07.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.463+0000 7f7a5f200640 1 -- 172.21.15.103:0/945582085 wait complete. 2024-04-16T16:13:07.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.465+0000 7f7a5f200640 1 Processor -- start 2024-04-16T16:13:07.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.465+0000 7f7a5f200640 1 -- start start 2024-04-16T16:13:07.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.466+0000 7f7a5f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a600cf5c0 0x7f7a600c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:07.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.466+0000 7f7a5f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7a600d42d0 0x7f7a6015b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:07.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.466+0000 7f7a5f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a6015eef0 0x7f7a60148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:07.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.466+0000 7f7a5f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7a600b7f10 con 0x7f7a6015eef0 2024-04-16T16:13:07.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.466+0000 7f7a5f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7a600d32c0 con 0x7f7a600cf5c0 2024-04-16T16:13:07.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.466+0000 7f7a5f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7a600b8090 con 0x7f7a600d42d0 2024-04-16T16:13:07.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.466+0000 7f7a5de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a600cf5c0 0x7f7a600c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:07.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.466+0000 7f7a57e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7a600d42d0 0x7f7a6015b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:07.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.466+0000 7f7a5de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a600cf5c0 0x7f7a600c6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48988/0 (socket says 172.21.15.103:48988) 2024-04-16T16:13:07.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.466+0000 7f7a5de00640 1 -- 172.21.15.103:0/3193449175 learned_addr learned my addr 172.21.15.103:0/3193449175 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-16T16:13:07.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.467+0000 7f7a5e800640 1 --2- 172.21.15.103:0/3193449175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a6015eef0 0x7f7a60148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:07.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.467+0000 7f7a5de00640 1 -- 172.21.15.103:0/3193449175 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7a600d42d0 msgr2=0x7f7a6015b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:07.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.467+0000 7f7a5de00640 1 --2- 172.21.15.103:0/3193449175 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7a600d42d0 0x7f7a6015b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.467+0000 7f7a5de00640 1 -- 172.21.15.103:0/3193449175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a6015eef0 msgr2=0x7f7a60148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:07.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.467+0000 7f7a5de00640 1 --2- 172.21.15.103:0/3193449175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a6015eef0 0x7f7a60148eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.467+0000 7f7a5de00640 1 -- 172.21.15.103:0/3193449175 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a600c7220 con 0x7f7a600cf5c0 2024-04-16T16:13:07.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.467+0000 7f7a57e00640 1 --2- 172.21.15.103:0/3193449175 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7a600d42d0 0x7f7a6015b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:07.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.467+0000 7f7a5de00640 1 --2- 172.21.15.103:0/3193449175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a600cf5c0 0x7f7a600c6ce0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7a5000c4d0 tx=0x7f7a5000c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:07.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.467+0000 7f7a56000640 1 -- 172.21.15.103:0/3193449175 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a50018070 con 0x7f7a600cf5c0 2024-04-16T16:13:07.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.468+0000 7f7a56000640 1 -- 172.21.15.103:0/3193449175 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a50007500 con 0x7f7a600cf5c0 2024-04-16T16:13:07.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.468+0000 7f7a5f200640 1 -- 172.21.15.103:0/3193449175 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a6015a5c0 con 0x7f7a600cf5c0 2024-04-16T16:13:07.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.468+0000 7f7a56000640 1 -- 172.21.15.103:0/3193449175 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a50004a20 con 0x7f7a600cf5c0 2024-04-16T16:13:07.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.468+0000 7f7a5f200640 1 -- 172.21.15.103:0/3193449175 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a6015ab80 con 0x7f7a600cf5c0 2024-04-16T16:13:07.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.468+0000 7f7a56000640 1 -- 172.21.15.103:0/3193449175 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7a5001b070 con 0x7f7a600cf5c0 2024-04-16T16:13:07.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.468+0000 7f7a56000640 1 -- 172.21.15.103:0/3193449175 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f7a500127b0 con 0x7f7a600cf5c0 2024-04-16T16:13:07.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.469+0000 7f7a5f200640 1 -- 172.21.15.103:0/3193449175 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a60075780 con 0x7f7a600cf5c0 2024-04-16T16:13:07.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.471+0000 7f7a56000640 1 -- 172.21.15.103:0/3193449175 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7a600c7220 con 0x7f7a600cf5c0 2024-04-16T16:13:07.503 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:07.505+0000 7fc088893040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:13:07.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.550+0000 7f7a5f200640 1 -- 172.21.15.103:0/3193449175 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "361acb09-4d66-455b-a5b9-c1d9a8794735", "id": 1} v 0) -- 0x7f7a600d3fd0 con 0x7f7a600cf5c0 2024-04-16T16:13:07.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.556+0000 7f7a56000640 1 -- 172.21.15.103:0/3193449175 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "361acb09-4d66-455b-a5b9-c1d9a8794735", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f7a50004db0 con 0x7f7a600cf5c0 2024-04-16T16:13:07.556 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-16T16:13:07.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.557+0000 7f7a5f200640 1 -- 172.21.15.103:0/3193449175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a600cf5c0 msgr2=0x7f7a600c6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:07.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.557+0000 7f7a5f200640 1 --2- 172.21.15.103:0/3193449175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a600cf5c0 0x7f7a600c6ce0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7a5000c4d0 tx=0x7f7a5000c990 comp rx=0 tx=0).stop 2024-04-16T16:13:07.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.557+0000 7f7a5f200640 1 -- 172.21.15.103:0/3193449175 shutdown_connections 2024-04-16T16:13:07.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.557+0000 7f7a5f200640 1 --2- 172.21.15.103:0/3193449175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a6015eef0 0x7f7a60148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.557+0000 7f7a5f200640 1 --2- 172.21.15.103:0/3193449175 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7a600d42d0 0x7f7a6015b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.557+0000 7f7a5f200640 1 --2- 172.21.15.103:0/3193449175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a600cf5c0 0x7f7a600c6ce0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.558+0000 7f7a5f200640 1 -- 172.21.15.103:0/3193449175 >> 172.21.15.103:0/3193449175 conn(0x7f7a6006d300 msgr2=0x7f7a600c7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:07.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.558+0000 7f7a5f200640 1 -- 172.21.15.103:0/3193449175 shutdown_connections 2024-04-16T16:13:07.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.558+0000 7f7a5f200640 1 -- 172.21.15.103:0/3193449175 wait complete. 2024-04-16T16:13:07.573 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:07.573+0000 7fc088893040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:13:07.579 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 13d87862-41f8-4b4f-b19f-10984092f273 2 2024-04-16T16:13:07.606 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:07.607+0000 7f76089db040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:13:07.627 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:07.627+0000 7fc088893040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:13:07.692 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:07.692+0000 7f76089db040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:13:07.694 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:07.694+0000 7fc088893040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:13:07.747 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:07.748+0000 7fc088893040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:13:07.748 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:07.749+0000 7f76089db040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:13:07.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.800+0000 7f3dcfe00640 1 Processor -- start 2024-04-16T16:13:07.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.801+0000 7f3dcfe00640 1 -- start start 2024-04-16T16:13:07.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.802+0000 7f3dcfe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd015eef0 0x7f3dd00d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:07.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.802+0000 7f3dcfe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3dd00d42d0 0x7f3dd00b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:07.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.802+0000 7f3dcfe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dd00d35a0 con 0x7f3dd00cf5c0 2024-04-16T16:13:07.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.802+0000 7f3dcfe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3dd00d2740 con 0x7f3dd00d42d0 2024-04-16T16:13:07.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.802+0000 7f3dcfe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3dd00d2a20 con 0x7f3dd015eef0 2024-04-16T16:13:07.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.802+0000 7f3dcea00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f3dd00cf5c0 0x7f3dd00cf9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:59774/0 (socket says 172.21.15.103:59774) 2024-04-16T16:13:07.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.802+0000 7f3dc7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd015eef0 0x7f3dd00d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:07.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.802+0000 7f3dcea00640 1 -- 172.21.15.103:0/2803082242 learned_addr learned my addr 172.21.15.103:0/2803082242 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-04-16T16:13:07.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.803+0000 7f3dcf400640 1 --2- 172.21.15.103:0/2803082242 >> v2:172.21.15.45:3300/0 conn(0x7f3dd00d42d0 0x7f3dd00b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:07.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.803+0000 7f3dc7e00640 1 -- 172.21.15.103:0/2803082242 >> v2:172.21.15.45:3300/0 conn(0x7f3dd00d42d0 msgr2=0x7f3dd00b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:07.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.803+0000 7f3dc7e00640 1 --2- 172.21.15.103:0/2803082242 >> v2:172.21.15.45:3300/0 conn(0x7f3dd00d42d0 0x7f3dd00b79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.803+0000 7f3dc7e00640 1 -- 172.21.15.103:0/2803082242 >> v1:172.21.15.103:6789/0 conn(0x7f3dd00cf5c0 legacy=0x7f3dd00cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:07.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.803+0000 7f3dc7e00640 1 -- 172.21.15.103:0/2803082242 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd00d2d00 con 0x7f3dd015eef0 2024-04-16T16:13:07.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.803+0000 7f3dce000640 1 -- 172.21.15.103:0/2803082242 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3541929455 0 0) 0x7f3dd00d35a0 con 0x7f3dd00cf5c0 2024-04-16T16:13:07.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.804+0000 7f3dc7e00640 1 --2- 172.21.15.103:0/2803082242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd015eef0 0x7f3dd00d3980 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3db80097a0 tx=0x7f3db802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43e55f7e002c60aa server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:07.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.804+0000 7f3dce000640 1 -- 172.21.15.103:0/2803082242 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8035020 con 0x7f3dd015eef0 2024-04-16T16:13:07.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.804+0000 7f3dce000640 1 -- 172.21.15.103:0/2803082242 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db8031830 con 0x7f3dd015eef0 2024-04-16T16:13:07.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.804+0000 7f3dce000640 1 -- 172.21.15.103:0/2803082242 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8031c00 con 0x7f3dd015eef0 2024-04-16T16:13:07.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.805+0000 7f3dcfe00640 1 -- 172.21.15.103:0/2803082242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd015eef0 msgr2=0x7f3dd00d3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:07.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.805+0000 7f3dcfe00640 1 --2- 172.21.15.103:0/2803082242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd015eef0 0x7f3dd00d3980 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3db80097a0 tx=0x7f3db802b8d0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.805+0000 7f3dcfe00640 1 -- 172.21.15.103:0/2803082242 shutdown_connections 2024-04-16T16:13:07.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.805+0000 7f3dcfe00640 1 --2- 172.21.15.103:0/2803082242 >> v2:172.21.15.45:3300/0 conn(0x7f3dd00d42d0 0x7f3dd00b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.805+0000 7f3dcfe00640 1 --2- 172.21.15.103:0/2803082242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd015eef0 0x7f3dd00d3980 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.805+0000 7f3dcfe00640 1 -- 172.21.15.103:0/2803082242 >> 172.21.15.103:0/2803082242 conn(0x7f3dd006d300 msgr2=0x7f3dd00d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:07.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.805+0000 7f3dcfe00640 1 -- 172.21.15.103:0/2803082242 shutdown_connections 2024-04-16T16:13:07.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.805+0000 7f3dcfe00640 1 -- 172.21.15.103:0/2803082242 wait complete. 2024-04-16T16:13:07.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.807+0000 7f3dcfe00640 1 Processor -- start 2024-04-16T16:13:07.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.807+0000 7f3dcfe00640 1 -- start start 2024-04-16T16:13:07.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.808+0000 7f3dcfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd00cf5c0 0x7f3dd014ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:07.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.808+0000 7f3dcfe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd00d42d0 0x7f3dd01f4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:07.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.808+0000 7f3dcfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dd015eef0 0x7f3dd0161840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:07.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.808+0000 7f3dcfe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd00b7f10 con 0x7f3dd00cf5c0 2024-04-16T16:13:07.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.808+0000 7f3dcfe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd00d32c0 con 0x7f3dd015eef0 2024-04-16T16:13:07.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.808+0000 7f3dcfe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3dd00b8090 con 0x7f3dd00d42d0 2024-04-16T16:13:07.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.808+0000 7f3dcf400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dd015eef0 0x7f3dd0161840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:07.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.808+0000 7f3dc7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd00d42d0 0x7f3dd01f4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:07.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.808+0000 7f3dcf400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dd015eef0 0x7f3dd0161840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48994/0 (socket says 172.21.15.103:48994) 2024-04-16T16:13:07.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.808+0000 7f3dcf400640 1 -- 172.21.15.103:0/2826837632 learned_addr learned my addr 172.21.15.103:0/2826837632 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-16T16:13:07.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.808+0000 7f3dcea00640 1 --2- 172.21.15.103:0/2826837632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd00cf5c0 0x7f3dd014ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:07.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.809+0000 7f3dcf400640 1 -- 172.21.15.103:0/2826837632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd00d42d0 msgr2=0x7f3dd01f4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:07.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.809+0000 7f3dcf400640 1 --2- 172.21.15.103:0/2826837632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd00d42d0 0x7f3dd01f4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.809+0000 7f3dcf400640 1 -- 172.21.15.103:0/2826837632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd00cf5c0 msgr2=0x7f3dd014ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:07.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.809+0000 7f3dcf400640 1 --2- 172.21.15.103:0/2826837632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd00cf5c0 0x7f3dd014ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.809+0000 7f3dcf400640 1 -- 172.21.15.103:0/2826837632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd0160a80 con 0x7f3dd015eef0 2024-04-16T16:13:07.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.809+0000 7f3dcea00640 1 --2- 172.21.15.103:0/2826837632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd00cf5c0 0x7f3dd014ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:13:07.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.809+0000 7f3dc7e00640 1 --2- 172.21.15.103:0/2826837632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd00d42d0 0x7f3dd01f4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:07.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.809+0000 7f3dcf400640 1 --2- 172.21.15.103:0/2826837632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dd015eef0 0x7f3dd0161840 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8007de0 tx=0x7f3dc800e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:07.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.809+0000 7f3dc6a00640 1 -- 172.21.15.103:0/2826837632 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8014620 con 0x7f3dd015eef0 2024-04-16T16:13:07.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.809+0000 7f3dc6a00640 1 -- 172.21.15.103:0/2826837632 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc8019070 con 0x7f3dd015eef0 2024-04-16T16:13:07.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.809+0000 7f3dc6a00640 1 -- 172.21.15.103:0/2826837632 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8014a00 con 0x7f3dd015eef0 2024-04-16T16:13:07.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.809+0000 7f3dcfe00640 1 -- 172.21.15.103:0/2826837632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dd0160d60 con 0x7f3dd015eef0 2024-04-16T16:13:07.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.809+0000 7f3dcfe00640 1 -- 172.21.15.103:0/2826837632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dd0161320 con 0x7f3dd015eef0 2024-04-16T16:13:07.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.810+0000 7f3dc6a00640 1 -- 172.21.15.103:0/2826837632 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3dc8014ba0 con 0x7f3dd015eef0 2024-04-16T16:13:07.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.810+0000 7f3dc6a00640 1 -- 172.21.15.103:0/2826837632 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f3dc801e880 con 0x7f3dd015eef0 2024-04-16T16:13:07.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.810+0000 7f3dcfe00640 1 -- 172.21.15.103:0/2826837632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dd0075780 con 0x7f3dd015eef0 2024-04-16T16:13:07.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.813+0000 7f3dc6a00640 1 -- 172.21.15.103:0/2826837632 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f3dc801e460 con 0x7f3dd015eef0 2024-04-16T16:13:07.813 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:07.814+0000 7f76089db040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:13:07.873 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:07.873+0000 7f76089db040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:13:07.888 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:07.889+0000 7fc088893040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:13:07.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.894+0000 7f3dcfe00640 1 -- 172.21.15.103:0/2826837632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "13d87862-41f8-4b4f-b19f-10984092f273", "id": 2} v 0) -- 0x7f3dd00cf090 con 0x7f3dd015eef0 2024-04-16T16:13:07.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.899+0000 7f3dc6a00640 1 -- 172.21.15.103:0/2826837632 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "13d87862-41f8-4b4f-b19f-10984092f273", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f3dc8028430 con 0x7f3dd015eef0 2024-04-16T16:13:07.899 INFO:teuthology.orchestra.run.smithi103.stdout:2 2024-04-16T16:13:07.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.900+0000 7f3dcfe00640 1 -- 172.21.15.103:0/2826837632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dd015eef0 msgr2=0x7f3dd0161840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:07.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.900+0000 7f3dcfe00640 1 --2- 172.21.15.103:0/2826837632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dd015eef0 0x7f3dd0161840 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8007de0 tx=0x7f3dc800e4b0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.900+0000 7f3dcfe00640 1 -- 172.21.15.103:0/2826837632 shutdown_connections 2024-04-16T16:13:07.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.900+0000 7f3dcfe00640 1 --2- 172.21.15.103:0/2826837632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dd015eef0 0x7f3dd0161840 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.900+0000 7f3dcfe00640 1 --2- 172.21.15.103:0/2826837632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd00d42d0 0x7f3dd01f4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.900+0000 7f3dcfe00640 1 --2- 172.21.15.103:0/2826837632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd00cf5c0 0x7f3dd014ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:07.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.900+0000 7f3dcfe00640 1 -- 172.21.15.103:0/2826837632 >> 172.21.15.103:0/2826837632 conn(0x7f3dd006d300 msgr2=0x7f3dd00b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:07.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.901+0000 7f3dcfe00640 1 -- 172.21.15.103:0/2826837632 shutdown_connections 2024-04-16T16:13:07.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:07.901+0000 7f3dcfe00640 1 -- 172.21.15.103:0/2826837632 wait complete. 2024-04-16T16:13:07.917 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new d1387f39-070b-4cd8-8abe-8a700504e189 3 2024-04-16T16:13:07.974 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:07.975+0000 7fc088893040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:13:08.032 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:08.032+0000 7fc088893040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:13:08.096 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:08.097+0000 7fc088893040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:13:08.102 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.102+0000 7f510e800640 1 Processor -- start 2024-04-16T16:13:08.102 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.102+0000 7f510e800640 1 -- start start 2024-04-16T16:13:08.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.103+0000 7f510e800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f51080cf5c0 0x7f51080d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:08.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.103+0000 7f510e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f510815eef0 0x7f51080cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:08.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.104+0000 7f510e800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51080d35a0 con 0x7f51080d42d0 2024-04-16T16:13:08.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.104+0000 7f510e800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f51080cfee0 con 0x7f51080cf5c0 2024-04-16T16:13:08.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.104+0000 7f510e800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f51080d2740 con 0x7f510815eef0 2024-04-16T16:13:08.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.104+0000 7f510ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f510815eef0 0x7f51080cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:08.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.104+0000 7f510ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f510815eef0 0x7f51080cf9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:60246/0 (socket says 172.21.15.103:60246) 2024-04-16T16:13:08.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.104+0000 7f510ca00640 1 -- 172.21.15.103:0/3055606091 learned_addr learned my addr 172.21.15.103:0/3055606091 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-16T16:13:08.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.104+0000 7f510d400640 1 --2- 172.21.15.103:0/3055606091 >> v2:172.21.15.45:3300/0 conn(0x7f51080cf5c0 0x7f51080d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:08.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.104+0000 7f510ca00640 1 -- 172.21.15.103:0/3055606091 >> v2:172.21.15.45:3300/0 conn(0x7f51080cf5c0 msgr2=0x7f51080d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:08.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.104+0000 7f510ca00640 1 --2- 172.21.15.103:0/3055606091 >> v2:172.21.15.45:3300/0 conn(0x7f51080cf5c0 0x7f51080d3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.104+0000 7f510ca00640 1 -- 172.21.15.103:0/3055606091 >> v1:172.21.15.45:6789/0 conn(0x7f51080d42d0 legacy=0x7f51080b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:08.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.104+0000 7f510ca00640 1 -- 172.21.15.103:0/3055606091 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51080d2a20 con 0x7f510815eef0 2024-04-16T16:13:08.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.106+0000 7f510ca00640 1 --2- 172.21.15.103:0/3055606091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f510815eef0 0x7f51080cf9a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f50f40098a0 tx=0x7f50f402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=800bab7dfc2319a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:08.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.106+0000 7f5107e00640 1 -- 172.21.15.103:0/3055606091 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f4035020 con 0x7f510815eef0 2024-04-16T16:13:08.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.106+0000 7f5107e00640 1 -- 172.21.15.103:0/3055606091 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50f4031640 con 0x7f510815eef0 2024-04-16T16:13:08.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.106+0000 7f5107e00640 1 -- 172.21.15.103:0/3055606091 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f402d040 con 0x7f510815eef0 2024-04-16T16:13:08.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.107+0000 7f510e800640 1 -- 172.21.15.103:0/3055606091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f510815eef0 msgr2=0x7f51080cf9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:08.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.107+0000 7f510e800640 1 --2- 172.21.15.103:0/3055606091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f510815eef0 0x7f51080cf9a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f50f40098a0 tx=0x7f50f402b8f0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.107+0000 7f510e800640 1 -- 172.21.15.103:0/3055606091 shutdown_connections 2024-04-16T16:13:08.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.107+0000 7f510e800640 1 --2- 172.21.15.103:0/3055606091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f510815eef0 0x7f51080cf9a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.107+0000 7f510e800640 1 --2- 172.21.15.103:0/3055606091 >> v2:172.21.15.45:3300/0 conn(0x7f51080cf5c0 0x7f51080d3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.107+0000 7f510e800640 1 -- 172.21.15.103:0/3055606091 >> 172.21.15.103:0/3055606091 conn(0x7f510806d300 msgr2=0x7f51080d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:08.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.107+0000 7f510e800640 1 -- 172.21.15.103:0/3055606091 shutdown_connections 2024-04-16T16:13:08.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.107+0000 7f510e800640 1 -- 172.21.15.103:0/3055606091 wait complete. 2024-04-16T16:13:08.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.109+0000 7f510e800640 1 Processor -- start 2024-04-16T16:13:08.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.109+0000 7f510e800640 1 -- start start 2024-04-16T16:13:08.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.110+0000 7f510e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51080cf5c0 0x7f5108152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:08.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.110+0000 7f510e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51080d42d0 0x7f51080cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:08.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.110+0000 7f510e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f510815eef0 0x7f51080d4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:08.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.110+0000 7f510e800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f51080d32c0 con 0x7f51080cf5c0 2024-04-16T16:13:08.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.110+0000 7f510e800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f51080d2fe0 con 0x7f510815eef0 2024-04-16T16:13:08.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.110+0000 7f510e800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5108162b50 con 0x7f51080d42d0 2024-04-16T16:13:08.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.110+0000 7f510ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51080d42d0 0x7f51080cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:08.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.110+0000 7f510ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51080d42d0 0x7f51080cb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:60262/0 (socket says 172.21.15.103:60262) 2024-04-16T16:13:08.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.110+0000 7f510ca00640 1 -- 172.21.15.103:0/2915271516 learned_addr learned my addr 172.21.15.103:0/2915271516 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-16T16:13:08.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.110+0000 7f510de00640 1 --2- 172.21.15.103:0/2915271516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f510815eef0 0x7f51080d4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:08.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.110+0000 7f510d400640 1 --2- 172.21.15.103:0/2915271516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51080cf5c0 0x7f5108152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:08.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.111+0000 7f510ca00640 1 -- 172.21.15.103:0/2915271516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f510815eef0 msgr2=0x7f51080d4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:08.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.111+0000 7f510ca00640 1 --2- 172.21.15.103:0/2915271516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f510815eef0 0x7f51080d4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.111+0000 7f510ca00640 1 -- 172.21.15.103:0/2915271516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51080cf5c0 msgr2=0x7f5108152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:08.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.111+0000 7f510ca00640 1 --2- 172.21.15.103:0/2915271516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51080cf5c0 0x7f5108152350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.111+0000 7f510ca00640 1 -- 172.21.15.103:0/2915271516 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51080cbe20 con 0x7f51080d42d0 2024-04-16T16:13:08.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.111+0000 7f510ca00640 1 --2- 172.21.15.103:0/2915271516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51080d42d0 0x7f51080cb8e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f50f402b930 tx=0x7f50f40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:08.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.111+0000 7f5105600640 1 -- 172.21.15.103:0/2915271516 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f403aa60 con 0x7f51080d42d0 2024-04-16T16:13:08.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.112+0000 7f5105600640 1 -- 172.21.15.103:0/2915271516 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50f403ac20 con 0x7f51080d42d0 2024-04-16T16:13:08.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.112+0000 7f5105600640 1 -- 172.21.15.103:0/2915271516 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f40443f0 con 0x7f51080d42d0 2024-04-16T16:13:08.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.112+0000 7f510e800640 1 -- 172.21.15.103:0/2915271516 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51080cc0c0 con 0x7f51080d42d0 2024-04-16T16:13:08.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.112+0000 7f510e800640 1 -- 172.21.15.103:0/2915271516 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f51080cc680 con 0x7f51080d42d0 2024-04-16T16:13:08.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.112+0000 7f5105600640 1 -- 172.21.15.103:0/2915271516 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f50f404b060 con 0x7f51080d42d0 2024-04-16T16:13:08.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.112+0000 7f5105600640 1 -- 172.21.15.103:0/2915271516 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f50f404dc00 con 0x7f51080d42d0 2024-04-16T16:13:08.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.113+0000 7f510e800640 1 -- 172.21.15.103:0/2915271516 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5108075780 con 0x7f51080d42d0 2024-04-16T16:13:08.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.115+0000 7f5105600640 1 -- 172.21.15.103:0/2915271516 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f50f4052020 con 0x7f51080d42d0 2024-04-16T16:13:08.155 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:08.156+0000 7fc088893040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:13:08.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.196+0000 7f510e800640 1 -- 172.21.15.103:0/2915271516 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d1387f39-070b-4cd8-8abe-8a700504e189", "id": 3} v 0) -- 0x7f51080b3560 con 0x7f51080d42d0 2024-04-16T16:13:08.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.205+0000 7f5105600640 1 -- 172.21.15.103:0/2915271516 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d1387f39-070b-4cd8-8abe-8a700504e189", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f50f403f070 con 0x7f51080d42d0 2024-04-16T16:13:08.205 INFO:teuthology.orchestra.run.smithi103.stdout:3 2024-04-16T16:13:08.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.206+0000 7f510e800640 1 -- 172.21.15.103:0/2915271516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51080d42d0 msgr2=0x7f51080cb8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:08.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.206+0000 7f510e800640 1 --2- 172.21.15.103:0/2915271516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51080d42d0 0x7f51080cb8e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f50f402b930 tx=0x7f50f40046c0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.206+0000 7f510e800640 1 -- 172.21.15.103:0/2915271516 shutdown_connections 2024-04-16T16:13:08.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.206+0000 7f510e800640 1 --2- 172.21.15.103:0/2915271516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f510815eef0 0x7f51080d4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.206+0000 7f510e800640 1 --2- 172.21.15.103:0/2915271516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51080d42d0 0x7f51080cb8e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.206+0000 7f510e800640 1 --2- 172.21.15.103:0/2915271516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51080cf5c0 0x7f5108152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.206+0000 7f510e800640 1 -- 172.21.15.103:0/2915271516 >> 172.21.15.103:0/2915271516 conn(0x7f510806d300 msgr2=0x7f510815cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:08.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.206+0000 7f510e800640 1 -- 172.21.15.103:0/2915271516 shutdown_connections 2024-04-16T16:13:08.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.206+0000 7f510e800640 1 -- 172.21.15.103:0/2915271516 wait complete. 2024-04-16T16:13:08.223 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new b53b6063-c9a0-487a-a7f8-401c8032ea6c 4 2024-04-16T16:13:08.353 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:08.353+0000 7f76089db040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:13:08.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.415+0000 7f488aa00640 1 Processor -- start 2024-04-16T16:13:08.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.416+0000 7f488aa00640 1 -- start start 2024-04-16T16:13:08.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.417+0000 7f488aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f488415ec30 0x7f48840b7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:08.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.417+0000 7f488aa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f48840b8060 0x7f48840d2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:08.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.417+0000 7f488aa00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48840b8500 con 0x7f48840d3030 2024-04-16T16:13:08.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.417+0000 7f488aa00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f48840b87e0 con 0x7f48840b8060 2024-04-16T16:13:08.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.417+0000 7f488aa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f48840b8ac0 con 0x7f488415ec30 2024-04-16T16:13:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.417+0000 7f4883e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f488415ec30 0x7f48840b7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.417+0000 7f4883e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f488415ec30 0x7f48840b7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:60264/0 (socket says 172.21.15.103:60264) 2024-04-16T16:13:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.417+0000 7f4883e00640 1 -- 172.21.15.103:0/3721433668 learned_addr learned my addr 172.21.15.103:0/3721433668 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-16T16:13:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.417+0000 7f4888a00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f48840d3030 0x7f48840b95a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:59784/0 (socket says 172.21.15.103:59784) 2024-04-16T16:13:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.418+0000 7f4883400640 1 --2- 172.21.15.103:0/3721433668 >> v2:172.21.15.45:3300/0 conn(0x7f48840b8060 0x7f48840d2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.418+0000 7f4883e00640 1 -- 172.21.15.103:0/3721433668 >> v2:172.21.15.45:3300/0 conn(0x7f48840b8060 msgr2=0x7f48840d2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.418+0000 7f4883e00640 1 --2- 172.21.15.103:0/3721433668 >> v2:172.21.15.45:3300/0 conn(0x7f48840b8060 0x7f48840d2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.418+0000 7f4883e00640 1 -- 172.21.15.103:0/3721433668 >> v1:172.21.15.103:6789/0 conn(0x7f48840d3030 legacy=0x7f48840b95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.418+0000 7f4883e00640 1 -- 172.21.15.103:0/3721433668 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48840b8da0 con 0x7f488415ec30 2024-04-16T16:13:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.418+0000 7f4883400640 1 --2- 172.21.15.103:0/3721433668 >> v2:172.21.15.45:3300/0 conn(0x7f48840b8060 0x7f48840d2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:08.419 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.419+0000 7f4883e00640 1 --2- 172.21.15.103:0/3721433668 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f488415ec30 0x7f48840b7b20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4874009870 tx=0x7f487402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaa00592b51f56a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:08.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.419+0000 7f4882a00640 1 -- 172.21.15.103:0/3721433668 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487402f830 con 0x7f488415ec30 2024-04-16T16:13:08.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.419+0000 7f4882a00640 1 -- 172.21.15.103:0/3721433668 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f487402f9d0 con 0x7f488415ec30 2024-04-16T16:13:08.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.419+0000 7f4882a00640 1 -- 172.21.15.103:0/3721433668 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487402fda0 con 0x7f488415ec30 2024-04-16T16:13:08.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.420+0000 7f488aa00640 1 -- 172.21.15.103:0/3721433668 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f488415ec30 msgr2=0x7f48840b7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:08.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.420+0000 7f488aa00640 1 --2- 172.21.15.103:0/3721433668 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f488415ec30 0x7f48840b7b20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4874009870 tx=0x7f487402ba10 comp rx=0 tx=0).stop 2024-04-16T16:13:08.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.420+0000 7f488aa00640 1 -- 172.21.15.103:0/3721433668 shutdown_connections 2024-04-16T16:13:08.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.420+0000 7f488aa00640 1 --2- 172.21.15.103:0/3721433668 >> v2:172.21.15.45:3300/0 conn(0x7f48840b8060 0x7f48840d2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.420+0000 7f488aa00640 1 --2- 172.21.15.103:0/3721433668 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f488415ec30 0x7f48840b7b20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.420+0000 7f488aa00640 1 -- 172.21.15.103:0/3721433668 >> 172.21.15.103:0/3721433668 conn(0x7f4884073480 msgr2=0x7f4884073890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:08.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.420+0000 7f488aa00640 1 -- 172.21.15.103:0/3721433668 shutdown_connections 2024-04-16T16:13:08.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.420+0000 7f488aa00640 1 -- 172.21.15.103:0/3721433668 wait complete. 2024-04-16T16:13:08.421 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.421+0000 7f488aa00640 1 Processor -- start 2024-04-16T16:13:08.421 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.422+0000 7f488aa00640 1 -- start start 2024-04-16T16:13:08.423 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.422+0000 7f488aa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48840b8060 0x7f488415b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:08.423 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.422+0000 7f488aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48840d3030 0x7f48840c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:08.423 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.422+0000 7f488aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f488415ec30 0x7f48840c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:08.423 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.422+0000 7f488aa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f48840b9cb0 con 0x7f48840d3030 2024-04-16T16:13:08.423 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.422+0000 7f488aa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f488415f010 con 0x7f48840b8060 2024-04-16T16:13:08.423 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.423+0000 7f488aa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f48840b9e30 con 0x7f488415ec30 2024-04-16T16:13:08.423 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.423+0000 7f4883e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48840b8060 0x7f488415b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:08.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.423+0000 7f4883e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48840b8060 0x7f488415b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49014/0 (socket says 172.21.15.103:49014) 2024-04-16T16:13:08.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.423+0000 7f4883e00640 1 -- 172.21.15.103:0/1401353597 learned_addr learned my addr 172.21.15.103:0/1401353597 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-16T16:13:08.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.423+0000 7f4888a00640 1 --2- 172.21.15.103:0/1401353597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f488415ec30 0x7f48840c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:08.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.423+0000 7f4883400640 1 --2- 172.21.15.103:0/1401353597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48840d3030 0x7f48840c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:08.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.423+0000 7f4883e00640 1 -- 172.21.15.103:0/1401353597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f488415ec30 msgr2=0x7f48840c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:08.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.423+0000 7f4883e00640 1 --2- 172.21.15.103:0/1401353597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f488415ec30 0x7f48840c8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.423+0000 7f4883e00640 1 -- 172.21.15.103:0/1401353597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48840d3030 msgr2=0x7f48840c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:08.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.423+0000 7f4883e00640 1 --2- 172.21.15.103:0/1401353597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48840d3030 0x7f48840c85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.423+0000 7f4883e00640 1 -- 172.21.15.103:0/1401353597 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48840cda90 con 0x7f48840b8060 2024-04-16T16:13:08.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.423+0000 7f4883400640 1 --2- 172.21.15.103:0/1401353597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48840d3030 0x7f48840c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:08.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.424+0000 7f4883e00640 1 --2- 172.21.15.103:0/1401353597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48840b8060 0x7f488415b050 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4874000c00 tx=0x7f4874004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:08.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.424+0000 7f487be00640 1 -- 172.21.15.103:0/1401353597 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487402f800 con 0x7f48840b8060 2024-04-16T16:13:08.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.424+0000 7f487be00640 1 -- 172.21.15.103:0/1401353597 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4874046070 con 0x7f48840b8060 2024-04-16T16:13:08.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.424+0000 7f487be00640 1 -- 172.21.15.103:0/1401353597 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4874041660 con 0x7f48840b8060 2024-04-16T16:13:08.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.424+0000 7f488aa00640 1 -- 172.21.15.103:0/1401353597 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48840c9430 con 0x7f48840b8060 2024-04-16T16:13:08.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.424+0000 7f488aa00640 1 -- 172.21.15.103:0/1401353597 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48840c99f0 con 0x7f48840b8060 2024-04-16T16:13:08.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.424+0000 7f487be00640 1 -- 172.21.15.103:0/1401353597 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4874040070 con 0x7f48840b8060 2024-04-16T16:13:08.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.424+0000 7f487be00640 1 -- 172.21.15.103:0/1401353597 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f487404a3c0 con 0x7f48840b8060 2024-04-16T16:13:08.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.425+0000 7f488aa00640 1 -- 172.21.15.103:0/1401353597 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4884075780 con 0x7f48840b8060 2024-04-16T16:13:08.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.427+0000 7f487be00640 1 -- 172.21.15.103:0/1401353597 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4874033030 con 0x7f48840b8060 2024-04-16T16:13:08.506 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.506+0000 7f488aa00640 1 -- 172.21.15.103:0/1401353597 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b53b6063-c9a0-487a-a7f8-401c8032ea6c", "id": 4} v 0) -- 0x7f48840c8290 con 0x7f48840b8060 2024-04-16T16:13:08.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.511+0000 7f487be00640 1 -- 172.21.15.103:0/1401353597 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b53b6063-c9a0-487a-a7f8-401c8032ea6c", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f487402f9a0 con 0x7f48840b8060 2024-04-16T16:13:08.519 INFO:teuthology.orchestra.run.smithi103.stdout:4 2024-04-16T16:13:08.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.513+0000 7f488aa00640 1 -- 172.21.15.103:0/1401353597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48840b8060 msgr2=0x7f488415b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:08.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.513+0000 7f488aa00640 1 --2- 172.21.15.103:0/1401353597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48840b8060 0x7f488415b050 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4874000c00 tx=0x7f4874004730 comp rx=0 tx=0).stop 2024-04-16T16:13:08.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.513+0000 7f488aa00640 1 -- 172.21.15.103:0/1401353597 shutdown_connections 2024-04-16T16:13:08.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.513+0000 7f488aa00640 1 --2- 172.21.15.103:0/1401353597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f488415ec30 0x7f48840c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.513+0000 7f488aa00640 1 --2- 172.21.15.103:0/1401353597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48840d3030 0x7f48840c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.513+0000 7f488aa00640 1 --2- 172.21.15.103:0/1401353597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48840b8060 0x7f488415b050 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.513+0000 7f488aa00640 1 -- 172.21.15.103:0/1401353597 >> 172.21.15.103:0/1401353597 conn(0x7f4884073480 msgr2=0x7f48840b3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:08.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.514+0000 7f488aa00640 1 -- 172.21.15.103:0/1401353597 shutdown_connections 2024-04-16T16:13:08.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.514+0000 7f488aa00640 1 -- 172.21.15.103:0/1401353597 wait complete. 2024-04-16T16:13:08.526 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:08.524+0000 7f76089db040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:13:08.530 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new b9ca6760-7092-4896-977b-585d9887e13c 5 2024-04-16T16:13:08.637 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:08.638+0000 7fc088893040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:13:08.670 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:08.670+0000 7f76089db040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:13:08.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.727+0000 7ff447e00640 1 Processor -- start 2024-04-16T16:13:08.727 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:08.727+0000 7f76089db040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:13:08.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.727+0000 7ff447e00640 1 -- start start 2024-04-16T16:13:08.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.728+0000 7ff447e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff4480cf8e0 0x7ff44815f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:08.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.729+0000 7ff447e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff4480b0980 0x7ff4480b0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:08.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.729+0000 7ff447e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4480ae8d0 con 0x7ff4480ae3d0 2024-04-16T16:13:08.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.729+0000 7ff447e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff4480aebb0 con 0x7ff4480cf8e0 2024-04-16T16:13:08.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.729+0000 7ff447e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff4480aee90 con 0x7ff4480b0980 2024-04-16T16:13:08.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.729+0000 7ff447400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff4480b0980 0x7ff4480b0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:08.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.729+0000 7ff447400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff4480b0980 0x7ff4480b0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49020/0 (socket says 172.21.15.103:49020) 2024-04-16T16:13:08.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.729+0000 7ff447400640 1 -- 172.21.15.103:0/2937705241 learned_addr learned my addr 172.21.15.103:0/2937705241 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-16T16:13:08.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.729+0000 7ff446a00640 1 --2- 172.21.15.103:0/2937705241 >> v2:172.21.15.45:3300/0 conn(0x7ff4480cf8e0 0x7ff44815f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:08.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.729+0000 7ff447400640 1 -- 172.21.15.103:0/2937705241 >> v2:172.21.15.45:3300/0 conn(0x7ff4480cf8e0 msgr2=0x7ff44815f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:08.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.729+0000 7ff447400640 1 --2- 172.21.15.103:0/2937705241 >> v2:172.21.15.45:3300/0 conn(0x7ff4480cf8e0 0x7ff44815f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.729+0000 7ff447400640 1 -- 172.21.15.103:0/2937705241 >> v1:172.21.15.45:6789/0 conn(0x7ff4480ae3d0 legacy=0x7ff44815fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:08.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.729+0000 7ff447400640 1 -- 172.21.15.103:0/2937705241 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4480af170 con 0x7ff4480b0980 2024-04-16T16:13:08.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.730+0000 7ff447400640 1 --2- 172.21.15.103:0/2937705241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4480b0980 0x7ff4480b0e20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff4480b3e50 tx=0x7ff44002b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1516c0ddbd45b175 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:08.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.731+0000 7ff445600640 1 -- 172.21.15.103:0/2937705241 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff440038070 con 0x7ff4480b0980 2024-04-16T16:13:08.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.731+0000 7ff445600640 1 -- 172.21.15.103:0/2937705241 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff44002f050 con 0x7ff4480b0980 2024-04-16T16:13:08.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.731+0000 7ff445600640 1 -- 172.21.15.103:0/2937705241 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff440031970 con 0x7ff4480b0980 2024-04-16T16:13:08.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.732+0000 7ff447e00640 1 -- 172.21.15.103:0/2937705241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4480b0980 msgr2=0x7ff4480b0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:08.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.732+0000 7ff447e00640 1 --2- 172.21.15.103:0/2937705241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4480b0980 0x7ff4480b0e20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff4480b3e50 tx=0x7ff44002b7f0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.732+0000 7ff447e00640 1 -- 172.21.15.103:0/2937705241 shutdown_connections 2024-04-16T16:13:08.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.732+0000 7ff447e00640 1 --2- 172.21.15.103:0/2937705241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4480b0980 0x7ff4480b0e20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.732+0000 7ff447e00640 1 --2- 172.21.15.103:0/2937705241 >> v2:172.21.15.45:3300/0 conn(0x7ff4480cf8e0 0x7ff44815f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.732+0000 7ff447e00640 1 -- 172.21.15.103:0/2937705241 >> 172.21.15.103:0/2937705241 conn(0x7ff44809aa70 msgr2=0x7ff4480b3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:08.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.732+0000 7ff447e00640 1 -- 172.21.15.103:0/2937705241 shutdown_connections 2024-04-16T16:13:08.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.732+0000 7ff447e00640 1 -- 172.21.15.103:0/2937705241 wait complete. 2024-04-16T16:13:08.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.734+0000 7ff447e00640 1 Processor -- start 2024-04-16T16:13:08.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.734+0000 7ff447e00640 1 -- start start 2024-04-16T16:13:08.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.735+0000 7ff447e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4480ae3d0 0x7ff4481523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:08.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.735+0000 7ff447e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4480b0980 0x7ff4481f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:08.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.735+0000 7ff447e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4480cf8e0 0x7ff4480b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:08.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.735+0000 7ff447e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff448160390 con 0x7ff4480b0980 2024-04-16T16:13:08.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.735+0000 7ff447e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff4480b3bd0 con 0x7ff4480ae3d0 2024-04-16T16:13:08.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.735+0000 7ff447e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff448160510 con 0x7ff4480cf8e0 2024-04-16T16:13:08.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.735+0000 7ff446a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4480ae3d0 0x7ff4481523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:08.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.735+0000 7ff446a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4480ae3d0 0x7ff4481523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49036/0 (socket says 172.21.15.103:49036) 2024-04-16T16:13:08.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.735+0000 7ff446a00640 1 -- 172.21.15.103:0/2105648713 learned_addr learned my addr 172.21.15.103:0/2105648713 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-16T16:13:08.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.735+0000 7ff447400640 1 --2- 172.21.15.103:0/2105648713 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4480cf8e0 0x7ff4480b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:08.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.735+0000 7ff446000640 1 --2- 172.21.15.103:0/2105648713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4480b0980 0x7ff4481f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:08.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.736+0000 7ff446a00640 1 -- 172.21.15.103:0/2105648713 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4480cf8e0 msgr2=0x7ff4480b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:08.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.736+0000 7ff446a00640 1 --2- 172.21.15.103:0/2105648713 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4480cf8e0 0x7ff4480b48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.736+0000 7ff446a00640 1 -- 172.21.15.103:0/2105648713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4480b0980 msgr2=0x7ff4481f5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:08.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.736+0000 7ff446a00640 1 --2- 172.21.15.103:0/2105648713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4480b0980 0x7ff4481f5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.736+0000 7ff446a00640 1 -- 172.21.15.103:0/2105648713 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4481f2310 con 0x7ff4480ae3d0 2024-04-16T16:13:08.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.736+0000 7ff446a00640 1 --2- 172.21.15.103:0/2105648713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4480ae3d0 0x7ff4481523a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff438007f20 tx=0x7ff43800e570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:08.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.736+0000 7ff43ea00640 1 -- 172.21.15.103:0/2105648713 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff438014620 con 0x7ff4480ae3d0 2024-04-16T16:13:08.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.736+0000 7ff43ea00640 1 -- 172.21.15.103:0/2105648713 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff438019070 con 0x7ff4480ae3d0 2024-04-16T16:13:08.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.736+0000 7ff43ea00640 1 -- 172.21.15.103:0/2105648713 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff438014a00 con 0x7ff4480ae3d0 2024-04-16T16:13:08.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.736+0000 7ff447e00640 1 -- 172.21.15.103:0/2105648713 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4481f62e0 con 0x7ff4480ae3d0 2024-04-16T16:13:08.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.736+0000 7ff447e00640 1 -- 172.21.15.103:0/2105648713 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4481f68a0 con 0x7ff4480ae3d0 2024-04-16T16:13:08.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.736+0000 7ff43ea00640 1 -- 172.21.15.103:0/2105648713 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff438014ba0 con 0x7ff4480ae3d0 2024-04-16T16:13:08.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.737+0000 7ff43ea00640 1 -- 172.21.15.103:0/2105648713 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7ff43801e9e0 con 0x7ff4480ae3d0 2024-04-16T16:13:08.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.738+0000 7ff447e00640 1 -- 172.21.15.103:0/2105648713 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff448075780 con 0x7ff4480ae3d0 2024-04-16T16:13:08.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.740+0000 7ff43ea00640 1 -- 172.21.15.103:0/2105648713 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ff43801e460 con 0x7ff4480ae3d0 2024-04-16T16:13:08.808 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:08.809+0000 7fc088893040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:13:08.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.826+0000 7ff447e00640 1 -- 172.21.15.103:0/2105648713 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b9ca6760-7092-4896-977b-585d9887e13c", "id": 5} v 0) -- 0x7ff4480703f0 con 0x7ff4480ae3d0 2024-04-16T16:13:08.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.834+0000 7ff43ea00640 1 -- 172.21.15.103:0/2105648713 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b9ca6760-7092-4896-977b-585d9887e13c", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7ff43801e640 con 0x7ff4480ae3d0 2024-04-16T16:13:08.834 INFO:teuthology.orchestra.run.smithi103.stdout:5 2024-04-16T16:13:08.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.836+0000 7ff447e00640 1 -- 172.21.15.103:0/2105648713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4480ae3d0 msgr2=0x7ff4481523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:08.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.836+0000 7ff447e00640 1 --2- 172.21.15.103:0/2105648713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4480ae3d0 0x7ff4481523a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff438007f20 tx=0x7ff43800e570 comp rx=0 tx=0).stop 2024-04-16T16:13:08.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.836+0000 7ff447e00640 1 -- 172.21.15.103:0/2105648713 shutdown_connections 2024-04-16T16:13:08.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.836+0000 7ff447e00640 1 --2- 172.21.15.103:0/2105648713 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4480cf8e0 0x7ff4480b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.836+0000 7ff447e00640 1 --2- 172.21.15.103:0/2105648713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4480b0980 0x7ff4481f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.836+0000 7ff447e00640 1 --2- 172.21.15.103:0/2105648713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4480ae3d0 0x7ff4481523a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:08.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.836+0000 7ff447e00640 1 -- 172.21.15.103:0/2105648713 >> 172.21.15.103:0/2105648713 conn(0x7ff44809aa70 msgr2=0x7ff44815dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:08.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.836+0000 7ff447e00640 1 -- 172.21.15.103:0/2105648713 shutdown_connections 2024-04-16T16:13:08.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:08.836+0000 7ff447e00640 1 -- 172.21.15.103:0/2105648713 wait complete. 2024-04-16T16:13:08.852 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new f3e011e1-0ece-4d60-aa69-44310d9c6a8f 6 2024-04-16T16:13:08.879 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:08.879+0000 7f76089db040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:13:08.959 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:08.959+0000 7fc088893040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:13:09.002 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:09.002+0000 7f76089db040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:13:09.015 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:09.016+0000 7fc088893040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:13:09.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.034+0000 7f6c4aa00640 1 Processor -- start 2024-04-16T16:13:09.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.034+0000 7f6c4aa00640 1 -- start start 2024-04-16T16:13:09.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.035+0000 7f6c4aa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6c440cf5c0 0x7f6c440d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:09.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.036+0000 7f6c4aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c440d42d0 0x7f6c440b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:09.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.036+0000 7f6c4aa00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c440d35a0 con 0x7f6c4415eef0 2024-04-16T16:13:09.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.036+0000 7f6c4aa00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f6c440d2740 con 0x7f6c440cf5c0 2024-04-16T16:13:09.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.036+0000 7f6c4aa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6c440d2a20 con 0x7f6c440d42d0 2024-04-16T16:13:09.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.036+0000 7f6c4a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c440d42d0 0x7f6c440b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:09.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.036+0000 7f6c4a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c440d42d0 0x7f6c440b79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:60290/0 (socket says 172.21.15.103:60290) 2024-04-16T16:13:09.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.036+0000 7f6c4a000640 1 -- 172.21.15.103:0/1571035852 learned_addr learned my addr 172.21.15.103:0/1571035852 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-16T16:13:09.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.036+0000 7f6c49600640 1 --2- 172.21.15.103:0/1571035852 >> v2:172.21.15.45:3300/0 conn(0x7f6c440cf5c0 0x7f6c440d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:09.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.037+0000 7f6c4a000640 1 -- 172.21.15.103:0/1571035852 >> v2:172.21.15.45:3300/0 conn(0x7f6c440cf5c0 msgr2=0x7f6c440d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:09.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.037+0000 7f6c4a000640 1 --2- 172.21.15.103:0/1571035852 >> v2:172.21.15.45:3300/0 conn(0x7f6c440cf5c0 0x7f6c440d3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.037+0000 7f6c4a000640 1 -- 172.21.15.103:0/1571035852 >> v1:172.21.15.45:6789/0 conn(0x7f6c4415eef0 legacy=0x7f6c440cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.037+0000 7f6c4a000640 1 -- 172.21.15.103:0/1571035852 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c440d2d00 con 0x7f6c440d42d0 2024-04-16T16:13:09.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.038+0000 7f6c4a000640 1 --2- 172.21.15.103:0/1571035852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c440d42d0 0x7f6c440b79d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c009870 tx=0x7f6c3c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cfef66019c75b43 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:09.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.039+0000 7f6c43e00640 1 -- 172.21.15.103:0/1571035852 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c034070 con 0x7f6c440d42d0 2024-04-16T16:13:09.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.039+0000 7f6c43e00640 1 -- 172.21.15.103:0/1571035852 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c3c030840 con 0x7f6c440d42d0 2024-04-16T16:13:09.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.039+0000 7f6c43e00640 1 -- 172.21.15.103:0/1571035852 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c030c10 con 0x7f6c440d42d0 2024-04-16T16:13:09.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.039+0000 7f6c4aa00640 1 -- 172.21.15.103:0/1571035852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c440d42d0 msgr2=0x7f6c440b79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.039+0000 7f6c4aa00640 1 --2- 172.21.15.103:0/1571035852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c440d42d0 0x7f6c440b79d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c009870 tx=0x7f6c3c02b980 comp rx=0 tx=0).stop 2024-04-16T16:13:09.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.040+0000 7f6c4aa00640 1 -- 172.21.15.103:0/1571035852 shutdown_connections 2024-04-16T16:13:09.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.040+0000 7f6c4aa00640 1 --2- 172.21.15.103:0/1571035852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c440d42d0 0x7f6c440b79d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.040+0000 7f6c4aa00640 1 --2- 172.21.15.103:0/1571035852 >> v2:172.21.15.45:3300/0 conn(0x7f6c440cf5c0 0x7f6c440d3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.040+0000 7f6c4aa00640 1 -- 172.21.15.103:0/1571035852 >> 172.21.15.103:0/1571035852 conn(0x7f6c4406d300 msgr2=0x7f6c440d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:09.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.040+0000 7f6c4aa00640 1 -- 172.21.15.103:0/1571035852 shutdown_connections 2024-04-16T16:13:09.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.040+0000 7f6c4aa00640 1 -- 172.21.15.103:0/1571035852 wait complete. 2024-04-16T16:13:09.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.042+0000 7f6c4aa00640 1 Processor -- start 2024-04-16T16:13:09.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.043+0000 7f6c4aa00640 1 -- start start 2024-04-16T16:13:09.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.044+0000 7f6c4aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c440cf5c0 0x7f6c441613b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:09.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.044+0000 7f6c4aa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c440d42d0 0x7f6c44154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:09.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.044+0000 7f6c4aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c4415eef0 0x7f6c4415ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:09.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.044+0000 7f6c4aa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6c440b7f10 con 0x7f6c440cf5c0 2024-04-16T16:13:09.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.044+0000 7f6c4aa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6c440d32c0 con 0x7f6c440d42d0 2024-04-16T16:13:09.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.044+0000 7f6c4aa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6c440b8090 con 0x7f6c4415eef0 2024-04-16T16:13:09.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.044+0000 7f6c4a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c4415eef0 0x7f6c4415ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:09.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.044+0000 7f6c48c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c440d42d0 0x7f6c44154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:09.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.045+0000 7f6c48c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c440d42d0 0x7f6c44154f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49048/0 (socket says 172.21.15.103:49048) 2024-04-16T16:13:09.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.045+0000 7f6c48c00640 1 -- 172.21.15.103:0/1806401580 learned_addr learned my addr 172.21.15.103:0/1806401580 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-16T16:13:09.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.045+0000 7f6c49600640 1 --2- 172.21.15.103:0/1806401580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c440cf5c0 0x7f6c441613b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:09.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.045+0000 7f6c48c00640 1 -- 172.21.15.103:0/1806401580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c4415eef0 msgr2=0x7f6c4415ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.045+0000 7f6c48c00640 1 --2- 172.21.15.103:0/1806401580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c4415eef0 0x7f6c4415ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.045+0000 7f6c48c00640 1 -- 172.21.15.103:0/1806401580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c440cf5c0 msgr2=0x7f6c441613b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.045+0000 7f6c48c00640 1 --2- 172.21.15.103:0/1806401580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c440cf5c0 0x7f6c441613b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.045+0000 7f6c48c00640 1 -- 172.21.15.103:0/1806401580 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c44160310 con 0x7f6c440d42d0 2024-04-16T16:13:09.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.045+0000 7f6c48c00640 1 --2- 172.21.15.103:0/1806401580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c440d42d0 0x7f6c44154f30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6c3000c470 tx=0x7f6c3000c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:09.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.046+0000 7f6c41600640 1 -- 172.21.15.103:0/1806401580 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c300052d0 con 0x7f6c440d42d0 2024-04-16T16:13:09.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.046+0000 7f6c41600640 1 -- 172.21.15.103:0/1806401580 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c30005490 con 0x7f6c440d42d0 2024-04-16T16:13:09.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.046+0000 7f6c4aa00640 1 -- 172.21.15.103:0/1806401580 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c441605f0 con 0x7f6c440d42d0 2024-04-16T16:13:09.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.046+0000 7f6c41600640 1 -- 172.21.15.103:0/1806401580 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c30007260 con 0x7f6c440d42d0 2024-04-16T16:13:09.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.046+0000 7f6c4aa00640 1 -- 172.21.15.103:0/1806401580 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c44160bb0 con 0x7f6c440d42d0 2024-04-16T16:13:09.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.046+0000 7f6c41600640 1 -- 172.21.15.103:0/1806401580 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6c30007400 con 0x7f6c440d42d0 2024-04-16T16:13:09.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.046+0000 7f6c41600640 1 -- 172.21.15.103:0/1806401580 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f6c300067d0 con 0x7f6c440d42d0 2024-04-16T16:13:09.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.047+0000 7f6c4aa00640 1 -- 172.21.15.103:0/1806401580 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c440d01b0 con 0x7f6c440d42d0 2024-04-16T16:13:09.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.049+0000 7f6c41600640 1 -- 172.21.15.103:0/1806401580 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f6c440d01b0 con 0x7f6c440d42d0 2024-04-16T16:13:09.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.124+0000 7f6c4aa00640 1 -- 172.21.15.103:0/1806401580 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f3e011e1-0ece-4d60-aa69-44310d9c6a8f", "id": 6} v 0) -- 0x7f6c44075780 con 0x7f6c440d42d0 2024-04-16T16:13:09.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.134+0000 7f6c41600640 1 -- 172.21.15.103:0/1806401580 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f3e011e1-0ece-4d60-aa69-44310d9c6a8f", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f6c300035f0 con 0x7f6c440d42d0 2024-04-16T16:13:09.134 INFO:teuthology.orchestra.run.smithi103.stdout:6 2024-04-16T16:13:09.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.135+0000 7f6c4aa00640 1 -- 172.21.15.103:0/1806401580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c440d42d0 msgr2=0x7f6c44154f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.135+0000 7f6c4aa00640 1 --2- 172.21.15.103:0/1806401580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c440d42d0 0x7f6c44154f30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6c3000c470 tx=0x7f6c3000c930 comp rx=0 tx=0).stop 2024-04-16T16:13:09.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.135+0000 7f6c4aa00640 1 -- 172.21.15.103:0/1806401580 shutdown_connections 2024-04-16T16:13:09.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.135+0000 7f6c4aa00640 1 --2- 172.21.15.103:0/1806401580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c4415eef0 0x7f6c4415ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.135+0000 7f6c4aa00640 1 --2- 172.21.15.103:0/1806401580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c440d42d0 0x7f6c44154f30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.135+0000 7f6c4aa00640 1 --2- 172.21.15.103:0/1806401580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c440cf5c0 0x7f6c441613b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.135+0000 7f6c4aa00640 1 -- 172.21.15.103:0/1806401580 >> 172.21.15.103:0/1806401580 conn(0x7f6c4406d300 msgr2=0x7f6c440b8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:09.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.135+0000 7f6c4aa00640 1 -- 172.21.15.103:0/1806401580 shutdown_connections 2024-04-16T16:13:09.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.135+0000 7f6c4aa00640 1 -- 172.21.15.103:0/1806401580 wait complete. 2024-04-16T16:13:09.149 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 122cbe72-024a-41a0-aa44-fc95cb04511c 7 2024-04-16T16:13:09.152 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:09.153+0000 7fc088893040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:13:09.201 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:09.201+0000 7f76089db040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:13:09.255 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:09.256+0000 7f76089db040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:13:09.270 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:09.271+0000 7fc088893040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:13:09.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.385+0000 7f841b200640 1 Processor -- start 2024-04-16T16:13:09.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.385+0000 7f841b200640 1 -- start start 2024-04-16T16:13:09.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.387+0000 7f841b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f84140d1090 0x7f84140ce000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:09.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.387+0000 7f841b200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f841409bcb0 con 0x7f841415c9f0 2024-04-16T16:13:09.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.387+0000 7f841b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84140cd260 con 0x7f84140d01f0 2024-04-16T16:13:09.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.387+0000 7f841b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f841415cdd0 con 0x7f84140d1090 2024-04-16T16:13:09.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.387+0000 7f841a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f84140d1090 0x7f84140ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:09.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.387+0000 7f841a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f84140d1090 0x7f84140ce000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49050/0 (socket says 172.21.15.103:49050) 2024-04-16T16:13:09.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.387+0000 7f841a800640 1 -- 172.21.15.103:0/2619417243 learned_addr learned my addr 172.21.15.103:0/2619417243 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-16T16:13:09.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.387+0000 7f8419400640 1 --1- 172.21.15.103:0/2619417243 >> v1:172.21.15.103:6789/0 conn(0x7f84140d01f0 0x7f84140d0980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:59792/0 (socket says 172.21.15.103:59792) 2024-04-16T16:13:09.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.388+0000 7f841a800640 1 -- 172.21.15.103:0/2619417243 >> v1:172.21.15.103:6789/0 conn(0x7f84140d01f0 legacy=0x7f84140d0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.388+0000 7f841a800640 1 -- 172.21.15.103:0/2619417243 >> v1:172.21.15.45:6789/0 conn(0x7f841415c9f0 legacy=0x7f841409b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.388+0000 7f841a800640 1 -- 172.21.15.103:0/2619417243 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84140cd540 con 0x7f84140d1090 2024-04-16T16:13:09.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.389+0000 7f841a800640 1 --2- 172.21.15.103:0/2619417243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84140d1090 0x7f84140ce000 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8410009e50 tx=0x7f841002bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcc5b33dbbe25f57 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:09.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.389+0000 7f8418a00640 1 -- 172.21.15.103:0/2619417243 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841002cb20 con 0x7f84140d1090 2024-04-16T16:13:09.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.389+0000 7f8418a00640 1 -- 172.21.15.103:0/2619417243 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f841002ccc0 con 0x7f84140d1090 2024-04-16T16:13:09.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.389+0000 7f8418a00640 1 -- 172.21.15.103:0/2619417243 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8410034610 con 0x7f84140d1090 2024-04-16T16:13:09.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.390+0000 7f841b200640 1 -- 172.21.15.103:0/2619417243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84140d1090 msgr2=0x7f84140ce000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.390+0000 7f841b200640 1 --2- 172.21.15.103:0/2619417243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84140d1090 0x7f84140ce000 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8410009e50 tx=0x7f841002bf80 comp rx=0 tx=0).stop 2024-04-16T16:13:09.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.390+0000 7f841b200640 1 -- 172.21.15.103:0/2619417243 shutdown_connections 2024-04-16T16:13:09.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.390+0000 7f841b200640 1 --2- 172.21.15.103:0/2619417243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84140d1090 0x7f84140ce000 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.390+0000 7f841b200640 1 -- 172.21.15.103:0/2619417243 >> 172.21.15.103:0/2619417243 conn(0x7f84140b4110 msgr2=0x7f84140b4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:09.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.390+0000 7f841b200640 1 -- 172.21.15.103:0/2619417243 shutdown_connections 2024-04-16T16:13:09.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.390+0000 7f841b200640 1 -- 172.21.15.103:0/2619417243 wait complete. 2024-04-16T16:13:09.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.391+0000 7f841b200640 1 Processor -- start 2024-04-16T16:13:09.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.392+0000 7f841b200640 1 -- start start 2024-04-16T16:13:09.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.392+0000 7f841b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84140d01f0 0x7f84141519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:09.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.392+0000 7f841b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84140d1090 0x7f84140b5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:09.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f841b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f841415c9f0 0x7f84140c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:09.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f841b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f84140c6170 con 0x7f841415c9f0 2024-04-16T16:13:09.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f841b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f84140cdb00 con 0x7f84140d1090 2024-04-16T16:13:09.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f841b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f84140c62f0 con 0x7f84140d01f0 2024-04-16T16:13:09.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f8419e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84140d01f0 0x7f84141519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:09.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f8419400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84140d1090 0x7f84140b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:09.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f8419e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84140d01f0 0x7f84141519b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:60316/0 (socket says 172.21.15.103:60316) 2024-04-16T16:13:09.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f8419400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84140d1090 0x7f84140b5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49064/0 (socket says 172.21.15.103:49064) 2024-04-16T16:13:09.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f8419e00640 1 -- 172.21.15.103:0/2897019871 learned_addr learned my addr 172.21.15.103:0/2897019871 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-16T16:13:09.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f841a800640 1 --2- 172.21.15.103:0/2897019871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f841415c9f0 0x7f84140c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:09.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f8419400640 1 -- 172.21.15.103:0/2897019871 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84140d01f0 msgr2=0x7f84141519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f8419400640 1 --2- 172.21.15.103:0/2897019871 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84140d01f0 0x7f84141519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f8419400640 1 -- 172.21.15.103:0/2897019871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f841415c9f0 msgr2=0x7f84140c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f8419400640 1 --2- 172.21.15.103:0/2897019871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f841415c9f0 0x7f84140c96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f8419400640 1 -- 172.21.15.103:0/2897019871 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84140b5bb0 con 0x7f84140d1090 2024-04-16T16:13:09.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.393+0000 7f8419e00640 1 --2- 172.21.15.103:0/2897019871 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84140d01f0 0x7f84141519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:09.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.394+0000 7f8419400640 1 --2- 172.21.15.103:0/2897019871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84140d1090 0x7f84140b5240 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f840400c430 tx=0x7f840400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:09.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.394+0000 7f840e000640 1 -- 172.21.15.103:0/2897019871 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84040074e0 con 0x7f84140d1090 2024-04-16T16:13:09.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.394+0000 7f840e000640 1 -- 172.21.15.103:0/2897019871 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8404007680 con 0x7f84140d1090 2024-04-16T16:13:09.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.394+0000 7f840e000640 1 -- 172.21.15.103:0/2897019871 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84040049e0 con 0x7f84140d1090 2024-04-16T16:13:09.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.394+0000 7f841b200640 1 -- 172.21.15.103:0/2897019871 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84140b5e90 con 0x7f84140d1090 2024-04-16T16:13:09.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.394+0000 7f841b200640 1 -- 172.21.15.103:0/2897019871 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84140b6450 con 0x7f84140d1090 2024-04-16T16:13:09.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.394+0000 7f840e000640 1 -- 172.21.15.103:0/2897019871 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f840401a020 con 0x7f84140d1090 2024-04-16T16:13:09.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.395+0000 7f840e000640 1 -- 172.21.15.103:0/2897019871 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f8404013a60 con 0x7f84140d1090 2024-04-16T16:13:09.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.395+0000 7f841b200640 1 -- 172.21.15.103:0/2897019871 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84140b6b90 con 0x7f84140d1090 2024-04-16T16:13:09.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.397+0000 7f840e000640 1 -- 172.21.15.103:0/2897019871 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8404021070 con 0x7f84140d1090 2024-04-16T16:13:09.441 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:09.441+0000 7f76089db040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:13:09.459 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:09.460+0000 7fc088893040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:13:09.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.470+0000 7f841b200640 1 -- 172.21.15.103:0/2897019871 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "122cbe72-024a-41a0-aa44-fc95cb04511c", "id": 7} v 0) -- 0x7f841415c500 con 0x7f84140d1090 2024-04-16T16:13:09.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.481+0000 7f840e000640 1 -- 172.21.15.103:0/2897019871 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "122cbe72-024a-41a0-aa44-fc95cb04511c", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f8404013400 con 0x7f84140d1090 2024-04-16T16:13:09.481 INFO:teuthology.orchestra.run.smithi103.stdout:7 2024-04-16T16:13:09.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.482+0000 7f841b200640 1 -- 172.21.15.103:0/2897019871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84140d1090 msgr2=0x7f84140b5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.482+0000 7f841b200640 1 --2- 172.21.15.103:0/2897019871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84140d1090 0x7f84140b5240 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f840400c430 tx=0x7f840400c8f0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.482+0000 7f841b200640 1 -- 172.21.15.103:0/2897019871 shutdown_connections 2024-04-16T16:13:09.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.482+0000 7f841b200640 1 --2- 172.21.15.103:0/2897019871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f841415c9f0 0x7f84140c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.482+0000 7f841b200640 1 --2- 172.21.15.103:0/2897019871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84140d1090 0x7f84140b5240 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.482+0000 7f841b200640 1 --2- 172.21.15.103:0/2897019871 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84140d01f0 0x7f84141519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.482+0000 7f841b200640 1 -- 172.21.15.103:0/2897019871 >> 172.21.15.103:0/2897019871 conn(0x7f84140b4110 msgr2=0x7f84140ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:09.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.482+0000 7f841b200640 1 -- 172.21.15.103:0/2897019871 shutdown_connections 2024-04-16T16:13:09.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:09.482+0000 7f841b200640 1 -- 172.21.15.103:0/2897019871 wait complete. 2024-04-16T16:13:09.497 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-16T16:13:09.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-16T16:13:09.505 INFO:tasks.ceph.osd.0:Started 2024-04-16T16:13:09.505 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-16T16:13:09.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-16T16:13:09.512 INFO:tasks.ceph.osd.1:Started 2024-04-16T16:13:09.512 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-16T16:13:09.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-16T16:13:09.513 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:09.514+0000 7fc088893040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:13:09.516 INFO:tasks.ceph.osd.2:Started 2024-04-16T16:13:09.517 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-16T16:13:09.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-16T16:13:09.521 INFO:tasks.ceph.osd.3:Started 2024-04-16T16:13:09.521 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-16T16:13:09.521 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-16T16:13:09.545 INFO:tasks.ceph.osd.4:Started 2024-04-16T16:13:09.545 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-16T16:13:09.545 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-16T16:13:09.552 INFO:tasks.ceph.osd.5:Started 2024-04-16T16:13:09.552 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-16T16:13:09.552 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-16T16:13:09.559 INFO:tasks.ceph.osd.6:Started 2024-04-16T16:13:09.559 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-16T16:13:09.559 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-16T16:13:09.565 INFO:tasks.ceph.osd.7:Started 2024-04-16T16:13:09.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:13:09.759 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:09.760+0000 7fc088893040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:13:09.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.774+0000 7fb897400640 1 Processor -- start 2024-04-16T16:13:09.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.775+0000 7fb897400640 1 -- start start 2024-04-16T16:13:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.776+0000 7fb897400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb890072a90 0x7fb8900ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.776+0000 7fb897400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89015ea70 0x7fb8900cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.776+0000 7fb897400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8900cf140 con 0x7fb8901462e0 2024-04-16T16:13:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.776+0000 7fb897400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb8900cf6d0 con 0x7fb890072a90 2024-04-16T16:13:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.776+0000 7fb897400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb8901466c0 con 0x7fb89015ea70 2024-04-16T16:13:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.776+0000 7fb88fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb890072a90 0x7fb8900ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.776+0000 7fb88fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb890072a90 0x7fb8900ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49324/0 (socket says 172.21.15.45:49324) 2024-04-16T16:13:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.776+0000 7fb88fe00640 1 -- 172.21.15.45:0/3965556339 learned_addr learned my addr 172.21.15.45:0/3965556339 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.776+0000 7fb894a00640 1 --1- 172.21.15.45:0/3965556339 >> v1:172.21.15.45:6789/0 conn(0x7fb8901462e0 0x7fb890072380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42212/0 (socket says 172.21.15.45:42212) 2024-04-16T16:13:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.776+0000 7fb895400640 1 --2- 172.21.15.45:0/3965556339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89015ea70 0x7fb8900cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.777+0000 7fb88fe00640 1 -- 172.21.15.45:0/3965556339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89015ea70 msgr2=0x7fb8900cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.777+0000 7fb88fe00640 1 --2- 172.21.15.45:0/3965556339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89015ea70 0x7fb8900cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.777+0000 7fb88fe00640 1 -- 172.21.15.45:0/3965556339 >> v1:172.21.15.45:6789/0 conn(0x7fb8901462e0 legacy=0x7fb890072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.777+0000 7fb88fe00640 1 -- 172.21.15.45:0/3965556339 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8900cf3f0 con 0x7fb890072a90 2024-04-16T16:13:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.777+0000 7fb88f400640 1 -- 172.21.15.45:0/3965556339 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1327496865 0 0) 0x7fb8900cf140 con 0x7fb8901462e0 2024-04-16T16:13:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.778+0000 7fb88fe00640 1 --2- 172.21.15.45:0/3965556339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb890072a90 0x7fb8900ce6c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009870 tx=0x7fb87c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1888bd21108dbe server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.778+0000 7fb88f400640 1 -- 172.21.15.45:0/3965556339 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c035020 con 0x7fb890072a90 2024-04-16T16:13:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.778+0000 7fb88f400640 1 -- 172.21.15.45:0/3965556339 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb87c031640 con 0x7fb890072a90 2024-04-16T16:13:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.778+0000 7fb88f400640 1 -- 172.21.15.45:0/3965556339 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c02d040 con 0x7fb890072a90 2024-04-16T16:13:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.779+0000 7fb897400640 1 -- 172.21.15.45:0/3965556339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb890072a90 msgr2=0x7fb8900ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.779+0000 7fb897400640 1 --2- 172.21.15.45:0/3965556339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb890072a90 0x7fb8900ce6c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009870 tx=0x7fb87c02b8f0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.779+0000 7fb897400640 1 -- 172.21.15.45:0/3965556339 shutdown_connections 2024-04-16T16:13:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.779+0000 7fb897400640 1 --2- 172.21.15.45:0/3965556339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89015ea70 0x7fb8900cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.779+0000 7fb897400640 1 --2- 172.21.15.45:0/3965556339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb890072a90 0x7fb8900ce6c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.779+0000 7fb897400640 1 -- 172.21.15.45:0/3965556339 >> 172.21.15.45:0/3965556339 conn(0x7fb8900b2750 msgr2=0x7fb8900b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.779+0000 7fb897400640 1 -- 172.21.15.45:0/3965556339 shutdown_connections 2024-04-16T16:13:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.779+0000 7fb897400640 1 -- 172.21.15.45:0/3965556339 wait complete. 2024-04-16T16:13:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.781+0000 7fb897400640 1 Processor -- start 2024-04-16T16:13:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.781+0000 7fb897400640 1 -- start start 2024-04-16T16:13:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.782+0000 7fb897400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb890072a90 0x7fb89015b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.782+0000 7fb897400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8901462e0 0x7fb890165cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.782+0000 7fb897400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89015ea70 0x7fb89013d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.782+0000 7fb897400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb89015d540 con 0x7fb890072a90 2024-04-16T16:13:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.782+0000 7fb897400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb89015d260 con 0x7fb8901462e0 2024-04-16T16:13:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.782+0000 7fb897400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb890166490 con 0x7fb89015ea70 2024-04-16T16:13:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.782+0000 7fb894a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb890072a90 0x7fb89015b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.782+0000 7fb894a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb890072a90 0x7fb89015b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49336/0 (socket says 172.21.15.45:49336) 2024-04-16T16:13:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.782+0000 7fb894a00640 1 -- 172.21.15.45:0/2507850373 learned_addr learned my addr 172.21.15.45:0/2507850373 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.782+0000 7fb895400640 1 --2- 172.21.15.45:0/2507850373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89015ea70 0x7fb89013d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.782+0000 7fb88fe00640 1 --2- 172.21.15.45:0/2507850373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8901462e0 0x7fb890165cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.783+0000 7fb894a00640 1 -- 172.21.15.45:0/2507850373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89015ea70 msgr2=0x7fb89013d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.783+0000 7fb894a00640 1 --2- 172.21.15.45:0/2507850373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89015ea70 0x7fb89013d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.783+0000 7fb894a00640 1 -- 172.21.15.45:0/2507850373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8901462e0 msgr2=0x7fb890165cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.783+0000 7fb894a00640 1 --2- 172.21.15.45:0/2507850373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8901462e0 0x7fb890165cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.783+0000 7fb894a00640 1 -- 172.21.15.45:0/2507850373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb89015bec0 con 0x7fb890072a90 2024-04-16T16:13:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.783+0000 7fb88fe00640 1 --2- 172.21.15.45:0/2507850373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8901462e0 0x7fb890165cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.783+0000 7fb895400640 1 --2- 172.21.15.45:0/2507850373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89015ea70 0x7fb89013d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.783+0000 7fb894a00640 1 --2- 172.21.15.45:0/2507850373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb890072a90 0x7fb89015b980 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb88400c430 tx=0x7fb88400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.783+0000 7fb88cc00640 1 -- 172.21.15.45:0/2507850373 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8840074e0 con 0x7fb890072a90 2024-04-16T16:13:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.783+0000 7fb88cc00640 1 -- 172.21.15.45:0/2507850373 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8840047e0 con 0x7fb890072a90 2024-04-16T16:13:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.783+0000 7fb897400640 1 -- 172.21.15.45:0/2507850373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8900b4560 con 0x7fb890072a90 2024-04-16T16:13:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.783+0000 7fb88cc00640 1 -- 172.21.15.45:0/2507850373 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb884004bd0 con 0x7fb890072a90 2024-04-16T16:13:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.783+0000 7fb897400640 1 -- 172.21.15.45:0/2507850373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8900b4b20 con 0x7fb890072a90 2024-04-16T16:13:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.784+0000 7fb88cc00640 1 -- 172.21.15.45:0/2507850373 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb884011070 con 0x7fb890072a90 2024-04-16T16:13:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.784+0000 7fb88cc00640 1 -- 172.21.15.45:0/2507850373 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fb884004d70 con 0x7fb890072a90 2024-04-16T16:13:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.784+0000 7fb897400640 1 -- 172.21.15.45:0/2507850373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb890075780 con 0x7fb890072a90 2024-04-16T16:13:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.789+0000 7fb88cc00640 1 -- 172.21.15.45:0/2507850373 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fb89015bec0 con 0x7fb890072a90 2024-04-16T16:13:09.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.855+0000 7fb897400640 1 -- 172.21.15.45:0/2507850373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb890072590 con 0x7fb890072a90 2024-04-16T16:13:09.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.856+0000 7fb88cc00640 1 -- 172.21.15.45:0/2507850373 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fb88401d3f0 con 0x7fb890072a90 2024-04-16T16:13:09.855 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:09.855 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":10,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:09.473563+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:13:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.857+0000 7fb897400640 1 -- 172.21.15.45:0/2507850373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb890072a90 msgr2=0x7fb89015b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.857+0000 7fb897400640 1 --2- 172.21.15.45:0/2507850373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb890072a90 0x7fb89015b980 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb88400c430 tx=0x7fb88400c8f0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.857+0000 7fb897400640 1 -- 172.21.15.45:0/2507850373 shutdown_connections 2024-04-16T16:13:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.857+0000 7fb897400640 1 --2- 172.21.15.45:0/2507850373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89015ea70 0x7fb89013d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.857+0000 7fb897400640 1 --2- 172.21.15.45:0/2507850373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8901462e0 0x7fb890165cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.857+0000 7fb897400640 1 --2- 172.21.15.45:0/2507850373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb890072a90 0x7fb89015b980 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.857+0000 7fb897400640 1 -- 172.21.15.45:0/2507850373 >> 172.21.15.45:0/2507850373 conn(0x7fb8900b2750 msgr2=0x7fb8900b18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.857+0000 7fb897400640 1 -- 172.21.15.45:0/2507850373 shutdown_connections 2024-04-16T16:13:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:09.857+0000 7fb897400640 1 -- 172.21.15.45:0/2507850373 wait complete. 2024-04-16T16:13:09.871 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-16T16:13:09.871 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-16T16:13:10.224 INFO:tasks.ceph.osd.0.smithi045.stderr:2024-04-16T16:13:10.225+0000 7f5f7b813640 -1 Falling back to public interface 2024-04-16T16:13:10.230 INFO:tasks.ceph.osd.2.smithi045.stderr:2024-04-16T16:13:10.231+0000 7fb31aad9640 -1 Falling back to public interface 2024-04-16T16:13:10.234 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-16T16:13:10.235+0000 7fe49ed9a640 -1 Falling back to public interface 2024-04-16T16:13:10.237 INFO:tasks.ceph.osd.3.smithi045.stderr:2024-04-16T16:13:10.239+0000 7f8251613640 -1 Falling back to public interface 2024-04-16T16:13:10.245 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-16T16:13:10.246+0000 7f5ab709b640 -1 Falling back to public interface 2024-04-16T16:13:10.253 INFO:tasks.ceph.osd.6.smithi103.stderr:2024-04-16T16:13:10.254+0000 7fa2daac8640 -1 Falling back to public interface 2024-04-16T16:13:10.276 INFO:tasks.ceph.osd.7.smithi103.stderr:2024-04-16T16:13:10.277+0000 7f5b00613640 -1 Falling back to public interface 2024-04-16T16:13:10.281 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-16T16:13:10.282+0000 7fba58017640 -1 Falling back to public interface 2024-04-16T16:13:10.557 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:10.557+0000 7f76089db040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:13:10.677 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T16:13:10.845 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.846+0000 7f6060c00640 1 Processor -- start 2024-04-16T16:13:10.845 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.846+0000 7f6060c00640 1 -- start start 2024-04-16T16:13:10.847 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.847+0000 7f6060c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f605c0cd210 0x7f605c1457c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:10.847 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.847+0000 7f6060c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f605c161060 con 0x7f605c0cce30 2024-04-16T16:13:10.847 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.847+0000 7f6060c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f605c15cfc0 con 0x7f605c072800 2024-04-16T16:13:10.847 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.847+0000 7f6060c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f605c15d2a0 con 0x7f605c0cd210 2024-04-16T16:13:10.847 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.847+0000 7f605b400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f605c0cce30 0x7f605c0720f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42228/0 (socket says 172.21.15.45:42228) 2024-04-16T16:13:10.847 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.847+0000 7f605b400640 1 -- 172.21.15.45:0/1818302145 learned_addr learned my addr 172.21.15.45:0/1818302145 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:10.847 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.847+0000 7f605be00640 1 --2- 172.21.15.45:0/1818302145 >> v2:172.21.15.45:3300/0 conn(0x7f605c0cd210 0x7f605c1457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:10.847 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.848+0000 7f605a000640 1 -- 172.21.15.45:0/1818302145 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1913025604 0 0) 0x7f605c161060 con 0x7f605c0cce30 2024-04-16T16:13:10.847 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.848+0000 7f605be00640 1 -- 172.21.15.45:0/1818302145 >> v1:172.21.15.103:6789/0 conn(0x7f605c072800 legacy=0x7f605c1603a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:10.847 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.848+0000 7f605be00640 1 -- 172.21.15.45:0/1818302145 >> v1:172.21.15.45:6789/0 conn(0x7f605c0cce30 legacy=0x7f605c0720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:10.848 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.848+0000 7f605be00640 1 -- 172.21.15.45:0/1818302145 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c15daa0 con 0x7f605c0cd210 2024-04-16T16:13:10.848 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.849+0000 7f605be00640 1 --2- 172.21.15.45:0/1818302145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605c0cd210 0x7f605c1457c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6054009d90 tx=0x7f605402bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7666bf454d959c49 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:10.848 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.849+0000 7f605a000640 1 -- 172.21.15.45:0/1818302145 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6054039070 con 0x7f605c0cd210 2024-04-16T16:13:10.848 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.849+0000 7f605a000640 1 -- 172.21.15.45:0/1818302145 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f605402ca50 con 0x7f605c0cd210 2024-04-16T16:13:10.848 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.849+0000 7f605a000640 1 -- 172.21.15.45:0/1818302145 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605402ce20 con 0x7f605c0cd210 2024-04-16T16:13:10.849 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.850+0000 7f6060c00640 1 -- 172.21.15.45:0/1818302145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605c0cd210 msgr2=0x7f605c1457c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:10.849 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.850+0000 7f6060c00640 1 --2- 172.21.15.45:0/1818302145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605c0cd210 0x7f605c1457c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6054009d90 tx=0x7f605402bd20 comp rx=0 tx=0).stop 2024-04-16T16:13:10.849 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.850+0000 7f6060c00640 1 -- 172.21.15.45:0/1818302145 shutdown_connections 2024-04-16T16:13:10.849 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.850+0000 7f6060c00640 1 --2- 172.21.15.45:0/1818302145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605c0cd210 0x7f605c1457c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:10.850 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.850+0000 7f6060c00640 1 -- 172.21.15.45:0/1818302145 >> 172.21.15.45:0/1818302145 conn(0x7f605c06b9b0 msgr2=0x7f605c06bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:10.850 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.850+0000 7f6060c00640 1 -- 172.21.15.45:0/1818302145 shutdown_connections 2024-04-16T16:13:10.850 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.850+0000 7f6060c00640 1 -- 172.21.15.45:0/1818302145 wait complete. 2024-04-16T16:13:10.850 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.852+0000 7f6060c00640 1 Processor -- start 2024-04-16T16:13:10.851 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.852+0000 7f6060c00640 1 -- start start 2024-04-16T16:13:10.852 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.853+0000 7f6060c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f605c072800 0x7f605c1f45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:10.853 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.853+0000 7f6060c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605c0cce30 0x7f605c13a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:10.853 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.853+0000 7f6060c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f605c0cd210 0x7f605c137fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:10.853 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.853+0000 7f6060c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f605c160c30 con 0x7f605c0cce30 2024-04-16T16:13:10.853 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.853+0000 7f6060c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f605c160ab0 con 0x7f605c0cd210 2024-04-16T16:13:10.853 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.853+0000 7f6060c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f605c160db0 con 0x7f605c072800 2024-04-16T16:13:10.853 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.853+0000 7f605aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605c0cce30 0x7f605c13a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:10.853 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.853+0000 7f605aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605c0cce30 0x7f605c13a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49352/0 (socket says 172.21.15.45:49352) 2024-04-16T16:13:10.853 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.853+0000 7f605aa00640 1 -- 172.21.15.45:0/2786952398 learned_addr learned my addr 172.21.15.45:0/2786952398 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:10.853 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.853+0000 7f605b400640 1 --2- 172.21.15.45:0/2786952398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f605c072800 0x7f605c1f45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:10.853 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.853+0000 7f605be00640 1 --2- 172.21.15.45:0/2786952398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f605c0cd210 0x7f605c137fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:10.853 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.854+0000 7f605aa00640 1 -- 172.21.15.45:0/2786952398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f605c072800 msgr2=0x7f605c1f45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:10.853 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.854+0000 7f605aa00640 1 --2- 172.21.15.45:0/2786952398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f605c072800 0x7f605c1f45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:10.853 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.854+0000 7f605aa00640 1 -- 172.21.15.45:0/2786952398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f605c0cd210 msgr2=0x7f605c137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:10.854 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.854+0000 7f605aa00640 1 --2- 172.21.15.45:0/2786952398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f605c0cd210 0x7f605c137fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:10.854 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.854+0000 7f605aa00640 1 -- 172.21.15.45:0/2786952398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c138510 con 0x7f605c0cce30 2024-04-16T16:13:10.854 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.854+0000 7f605b400640 1 --2- 172.21.15.45:0/2786952398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f605c072800 0x7f605c1f45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:13:10.854 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.854+0000 7f605aa00640 1 --2- 172.21.15.45:0/2786952398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605c0cce30 0x7f605c13a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f604800c7b0 tx=0x7f604800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:10.855 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.854+0000 7f6053400640 1 -- 172.21.15.45:0/2786952398 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6048017070 con 0x7f605c0cce30 2024-04-16T16:13:10.855 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.854+0000 7f6053400640 1 -- 172.21.15.45:0/2786952398 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6048005290 con 0x7f605c0cce30 2024-04-16T16:13:10.855 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.854+0000 7f6053400640 1 -- 172.21.15.45:0/2786952398 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604800a880 con 0x7f605c0cce30 2024-04-16T16:13:10.855 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.854+0000 7f6060c00640 1 -- 172.21.15.45:0/2786952398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605c13b020 con 0x7f605c0cce30 2024-04-16T16:13:10.855 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.854+0000 7f6060c00640 1 -- 172.21.15.45:0/2786952398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605c13b5e0 con 0x7f605c0cce30 2024-04-16T16:13:10.855 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.854+0000 7f6053400640 1 -- 172.21.15.45:0/2786952398 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f604801a070 con 0x7f605c0cce30 2024-04-16T16:13:10.855 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.855+0000 7f6053400640 1 -- 172.21.15.45:0/2786952398 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f60480035f0 con 0x7f605c0cce30 2024-04-16T16:13:10.855 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.855+0000 7f6060c00640 1 -- 172.21.15.45:0/2786952398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f605c04fe30 con 0x7f605c0cce30 2024-04-16T16:13:10.858 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.859+0000 7f6053400640 1 -- 172.21.15.45:0/2786952398 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f6048021020 con 0x7f605c0cce30 2024-04-16T16:13:10.872 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:10.872+0000 7f76089db040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:13:10.874 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:10.875+0000 7fc088893040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:13:10.923 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.925+0000 7f6060c00640 1 -- 172.21.15.45:0/2786952398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f605c15c6f0 con 0x7f605c0cce30 2024-04-16T16:13:10.924 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.925+0000 7f6053400640 1 -- 172.21.15.45:0/2786952398 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f605c138510 con 0x7f605c0cce30 2024-04-16T16:13:10.925 INFO:teuthology.misc.health.smithi045.stdout: 2024-04-16T16:13:10.925 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:09.473563+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:13:10.926 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.927+0000 7f6060c00640 1 -- 172.21.15.45:0/2786952398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605c0cce30 msgr2=0x7f605c13a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:10.926 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.927+0000 7f6060c00640 1 --2- 172.21.15.45:0/2786952398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605c0cce30 0x7f605c13a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f604800c7b0 tx=0x7f604800cc70 comp rx=0 tx=0).stop 2024-04-16T16:13:10.926 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.927+0000 7f6060c00640 1 -- 172.21.15.45:0/2786952398 shutdown_connections 2024-04-16T16:13:10.926 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.927+0000 7f6060c00640 1 --2- 172.21.15.45:0/2786952398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f605c0cd210 0x7f605c137fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:10.926 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.927+0000 7f6060c00640 1 --2- 172.21.15.45:0/2786952398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605c0cce30 0x7f605c13a880 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:10.926 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.927+0000 7f6060c00640 1 --2- 172.21.15.45:0/2786952398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f605c072800 0x7f605c1f45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:10.926 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.927+0000 7f6060c00640 1 -- 172.21.15.45:0/2786952398 >> 172.21.15.45:0/2786952398 conn(0x7f605c06b9b0 msgr2=0x7f605c0720f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:10.926 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.927+0000 7f6060c00640 1 -- 172.21.15.45:0/2786952398 shutdown_connections 2024-04-16T16:13:10.926 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:10.927+0000 7f6060c00640 1 -- 172.21.15.45:0/2786952398 wait complete. 2024-04-16T16:13:10.932 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:10.932+0000 7f76089db040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:13:10.938 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-16T16:13:11.063 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:11.063+0000 7f76089db040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:13:11.212 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:11.213+0000 7fc088893040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:13:11.276 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:11.277+0000 7fc088893040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:13:11.415 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:13:11.416+0000 7fc088893040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:13:12.120 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:12.121+0000 7f75a9a00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-16T16:13:12.121 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:13:12.122+0000 7f75a9a00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-16T16:13:13.630 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-16T16:13:13.631+0000 7f5ab709b640 -1 osd.1 0 log_to_monitors true 2024-04-16T16:13:13.655 INFO:tasks.ceph.osd.6.smithi103.stderr:2024-04-16T16:13:13.655+0000 7fa2daac8640 -1 osd.6 0 log_to_monitors true 2024-04-16T16:13:13.655 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-16T16:13:13.656+0000 7fba58017640 -1 osd.5 0 log_to_monitors true 2024-04-16T16:13:13.796 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-16T16:13:13.797+0000 7fe49ed9a640 -1 osd.4 0 log_to_monitors true 2024-04-16T16:13:13.852 INFO:tasks.ceph.osd.7.smithi103.stderr:2024-04-16T16:13:13.852+0000 7f5b00613640 -1 osd.7 0 log_to_monitors true 2024-04-16T16:13:13.870 INFO:tasks.ceph.osd.2.smithi045.stderr:2024-04-16T16:13:13.871+0000 7fb31aad9640 -1 osd.2 0 log_to_monitors true 2024-04-16T16:13:13.891 INFO:tasks.ceph.osd.0.smithi045.stderr:2024-04-16T16:13:13.892+0000 7f5f7b813640 -1 osd.0 0 log_to_monitors true 2024-04-16T16:13:13.902 INFO:tasks.ceph.osd.3.smithi045.stderr:2024-04-16T16:13:13.904+0000 7f8251613640 -1 osd.3 0 log_to_monitors true 2024-04-16T16:13:16.805 INFO:tasks.ceph.osd.3.smithi045.stderr:2024-04-16T16:13:16.806+0000 7f824c000640 -1 osd.3 0 waiting for initial osdmap 2024-04-16T16:13:16.812 INFO:tasks.ceph.osd.3.smithi045.stderr:2024-04-16T16:13:16.814+0000 7f8242000640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T16:13:16.824 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-16T16:13:16.825+0000 7f5ab2a00640 -1 osd.1 0 waiting for initial osdmap 2024-04-16T16:13:16.834 INFO:tasks.ceph.osd.0.smithi045.stderr:2024-04-16T16:13:16.835+0000 7f5f76000640 -1 osd.0 0 waiting for initial osdmap 2024-04-16T16:13:16.838 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-16T16:13:16.839+0000 7f5aa8000640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T16:13:16.842 INFO:tasks.ceph.osd.0.smithi045.stderr:2024-04-16T16:13:16.843+0000 7f5f6c000640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T16:13:16.872 INFO:tasks.ceph.osd.2.smithi045.stderr:2024-04-16T16:13:16.873+0000 7fb315a00640 -1 osd.2 0 waiting for initial osdmap 2024-04-16T16:13:16.879 INFO:tasks.ceph.osd.2.smithi045.stderr:2024-04-16T16:13:16.880+0000 7fb30ba00640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T16:13:17.403 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-16T16:13:17.404+0000 7fba52800640 -1 osd.5 0 waiting for initial osdmap 2024-04-16T16:13:17.405 INFO:tasks.ceph.osd.7.smithi103.stderr:2024-04-16T16:13:17.406+0000 7f5afba00640 -1 osd.7 0 waiting for initial osdmap 2024-04-16T16:13:17.414 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-16T16:13:17.414+0000 7fba48800640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T16:13:17.414 INFO:tasks.ceph.osd.7.smithi103.stderr:2024-04-16T16:13:17.415+0000 7f5af1000640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T16:13:17.486 INFO:tasks.ceph.osd.6.smithi103.stderr:2024-04-16T16:13:17.486+0000 7fa2d5a00640 -1 osd.6 0 waiting for initial osdmap 2024-04-16T16:13:17.491 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-16T16:13:17.492+0000 7fe499c00640 -1 osd.4 0 waiting for initial osdmap 2024-04-16T16:13:17.492 INFO:tasks.ceph.osd.6.smithi103.stderr:2024-04-16T16:13:17.492+0000 7fa2cba00640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T16:13:17.502 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-16T16:13:17.503+0000 7fe48fc00640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T16:13:17.746 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T16:13:17.936 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.937+0000 7f420e400640 1 Processor -- start 2024-04-16T16:13:17.936 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.937+0000 7f420e400640 1 -- start start 2024-04-16T16:13:17.938 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.938+0000 7f420e400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f420815ed10 0x7f42080d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:17.938 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.939+0000 7f420e400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42080d3260 con 0x7f42080cf160 2024-04-16T16:13:17.938 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.939+0000 7f420e400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42080d3540 con 0x7f42080b6b20 2024-04-16T16:13:17.938 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.939+0000 7f420e400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f420815f0f0 con 0x7f420815ed10 2024-04-16T16:13:17.939 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.939+0000 7f420d000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f420815ed10 0x7f42080d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:17.939 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.939+0000 7f4207e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f42080cf160 0x7f42080b6410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54886/0 (socket says 172.21.15.45:54886) 2024-04-16T16:13:17.939 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.939+0000 7f4207e00640 1 -- 172.21.15.45:0/3571962232 learned_addr learned my addr 172.21.15.45:0/3571962232 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:17.939 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.939+0000 7f420d000640 1 -- 172.21.15.45:0/3571962232 >> v1:172.21.15.103:6789/0 conn(0x7f42080b6b20 legacy=0x7f42080b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:17.939 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.939+0000 7f420d000640 1 -- 172.21.15.45:0/3571962232 >> v1:172.21.15.45:6789/0 conn(0x7f42080cf160 legacy=0x7f42080b6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:17.939 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.939+0000 7f420d000640 1 -- 172.21.15.45:0/3571962232 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42080d3820 con 0x7f420815ed10 2024-04-16T16:13:17.939 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.939+0000 7f4207400640 1 -- 172.21.15.45:0/3571962232 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3826067055 0 0) 0x7f42080d3260 con 0x7f42080cf160 2024-04-16T16:13:17.940 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.940+0000 7f420d000640 1 --2- 172.21.15.45:0/3571962232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f420815ed10 0x7f42080d2b60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0097a0 tx=0x7f41fc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59304a077e0e375 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:17.940 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.941+0000 7f4207400640 1 -- 172.21.15.45:0/3571962232 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc034070 con 0x7f420815ed10 2024-04-16T16:13:17.940 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.941+0000 7f4207400640 1 -- 172.21.15.45:0/3571962232 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41fc030610 con 0x7f420815ed10 2024-04-16T16:13:17.940 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.941+0000 7f4207400640 1 -- 172.21.15.45:0/3571962232 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0309e0 con 0x7f420815ed10 2024-04-16T16:13:17.941 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.941+0000 7f420e400640 1 -- 172.21.15.45:0/3571962232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f420815ed10 msgr2=0x7f42080d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:17.941 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.941+0000 7f420e400640 1 --2- 172.21.15.45:0/3571962232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f420815ed10 0x7f42080d2b60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0097a0 tx=0x7f41fc02b960 comp rx=0 tx=0).stop 2024-04-16T16:13:17.941 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.941+0000 7f420e400640 1 -- 172.21.15.45:0/3571962232 shutdown_connections 2024-04-16T16:13:17.941 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.941+0000 7f420e400640 1 --2- 172.21.15.45:0/3571962232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f420815ed10 0x7f42080d2b60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:17.941 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.941+0000 7f420e400640 1 -- 172.21.15.45:0/3571962232 >> 172.21.15.45:0/3571962232 conn(0x7f4208073710 msgr2=0x7f42080d4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:17.941 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.942+0000 7f420e400640 1 -- 172.21.15.45:0/3571962232 shutdown_connections 2024-04-16T16:13:17.941 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.942+0000 7f420e400640 1 -- 172.21.15.45:0/3571962232 wait complete. 2024-04-16T16:13:17.942 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.943+0000 7f420e400640 1 Processor -- start 2024-04-16T16:13:17.942 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.943+0000 7f420e400640 1 -- start start 2024-04-16T16:13:17.945 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.944+0000 7f420e400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42080b6b20 0x7f42080ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:17.945 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.944+0000 7f420e400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42080cf160 0x7f42080ba2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:17.946 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.944+0000 7f420e400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f420815ed10 0x7f4208160880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:17.946 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.944+0000 7f420e400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f42080d3de0 con 0x7f42080cf160 2024-04-16T16:13:17.946 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.944+0000 7f420e400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f42080b8220 con 0x7f420815ed10 2024-04-16T16:13:17.946 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.944+0000 7f420e400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f42080b83a0 con 0x7f42080b6b20 2024-04-16T16:13:17.946 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.944+0000 7f4207e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42080cf160 0x7f42080ba2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:17.946 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.945+0000 7f4207e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42080cf160 0x7f42080ba2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38064/0 (socket says 172.21.15.45:38064) 2024-04-16T16:13:17.947 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.945+0000 7f4207e00640 1 -- 172.21.15.45:0/1958578229 learned_addr learned my addr 172.21.15.45:0/1958578229 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:17.947 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.945+0000 7f420d000640 1 --2- 172.21.15.45:0/1958578229 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42080b6b20 0x7f42080ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:17.947 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.945+0000 7f4207e00640 1 -- 172.21.15.45:0/1958578229 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42080b6b20 msgr2=0x7f42080ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:17.947 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.945+0000 7f4207e00640 1 --2- 172.21.15.45:0/1958578229 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42080b6b20 0x7f42080ca910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:17.947 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.945+0000 7f420da00640 1 --2- 172.21.15.45:0/1958578229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f420815ed10 0x7f4208160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:17.947 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.945+0000 7f4207e00640 1 -- 172.21.15.45:0/1958578229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f420815ed10 msgr2=0x7f4208160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:17.948 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.945+0000 7f4207e00640 1 --2- 172.21.15.45:0/1958578229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f420815ed10 0x7f4208160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:17.948 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.945+0000 7f4207e00640 1 -- 172.21.15.45:0/1958578229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420815f7e0 con 0x7f42080cf160 2024-04-16T16:13:17.948 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.945+0000 7f420da00640 1 --2- 172.21.15.45:0/1958578229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f420815ed10 0x7f4208160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:17.948 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.945+0000 7f4207e00640 1 --2- 172.21.15.45:0/1958578229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42080cf160 0x7f42080ba2e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f41f400c4d0 tx=0x7f41f400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:17.948 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.946+0000 7f4204c00640 1 -- 172.21.15.45:0/1958578229 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4018070 con 0x7f42080cf160 2024-04-16T16:13:17.949 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.946+0000 7f4204c00640 1 -- 172.21.15.45:0/1958578229 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41f4007500 con 0x7f42080cf160 2024-04-16T16:13:17.949 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.946+0000 7f4204c00640 1 -- 172.21.15.45:0/1958578229 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4004a20 con 0x7f42080cf160 2024-04-16T16:13:17.949 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.946+0000 7f420e400640 1 -- 172.21.15.45:0/1958578229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f420815fac0 con 0x7f42080cf160 2024-04-16T16:13:17.949 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.946+0000 7f420e400640 1 -- 172.21.15.45:0/1958578229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4208160080 con 0x7f42080cf160 2024-04-16T16:13:17.951 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.947+0000 7f420e400640 1 -- 172.21.15.45:0/1958578229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f420804fe30 con 0x7f42080cf160 2024-04-16T16:13:17.951 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.948+0000 7f4204c00640 1 -- 172.21.15.45:0/1958578229 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f41f4004bc0 con 0x7f42080cf160 2024-04-16T16:13:17.951 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.949+0000 7f4204c00640 1 --2- 172.21.15.45:0/1958578229 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f41e806c860 0x7f41e806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:17.951 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.949+0000 7f4204c00640 1 -- 172.21.15.45:0/1958578229 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7f41f408a840 con 0x7f42080cf160 2024-04-16T16:13:17.951 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.949+0000 7f420d000640 1 --2- 172.21.15.45:0/1958578229 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f41e806c860 0x7f41e806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:17.951 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.950+0000 7f420d000640 1 --2- 172.21.15.45:0/1958578229 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f41e806c860 0x7f41e806ed20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f41fc036000 tx=0x7f41fc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:17.953 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:17.954+0000 7f4204c00640 1 -- 172.21.15.45:0/1958578229 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f41f4059210 con 0x7f42080cf160 2024-04-16T16:13:18.152 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:18.153+0000 7f420e400640 1 -- 172.21.15.45:0/1958578229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42080b5840 con 0x7f42080cf160 2024-04-16T16:13:18.154 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:18.155+0000 7f4204c00640 1 -- 172.21.15.45:0/1958578229 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+10909 (secure 0 0 0) 0x7f41f4058b10 con 0x7f42080cf160 2024-04-16T16:13:18.155 INFO:teuthology.misc.health.smithi045.stdout: 2024-04-16T16:13:18.155 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":13,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:17.804403+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:13:18.157 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:18.158+0000 7f420e400640 1 -- 172.21.15.45:0/1958578229 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f41e806c860 msgr2=0x7f41e806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:18.158 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:18.158+0000 7f420e400640 1 --2- 172.21.15.45:0/1958578229 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f41e806c860 0x7f41e806ed20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f41fc036000 tx=0x7f41fc008030 comp rx=0 tx=0).stop 2024-04-16T16:13:18.158 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:18.158+0000 7f420e400640 1 -- 172.21.15.45:0/1958578229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42080cf160 msgr2=0x7f42080ba2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:18.158 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:18.158+0000 7f420e400640 1 --2- 172.21.15.45:0/1958578229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42080cf160 0x7f42080ba2e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f41f400c4d0 tx=0x7f41f400c990 comp rx=0 tx=0).stop 2024-04-16T16:13:18.158 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:18.158+0000 7f420e400640 1 -- 172.21.15.45:0/1958578229 shutdown_connections 2024-04-16T16:13:18.158 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:18.158+0000 7f420e400640 1 --2- 172.21.15.45:0/1958578229 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f41e806c860 0x7f41e806ed20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.158 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:18.158+0000 7f420e400640 1 --2- 172.21.15.45:0/1958578229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f420815ed10 0x7f4208160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.158 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:18.158+0000 7f420e400640 1 --2- 172.21.15.45:0/1958578229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42080cf160 0x7f42080ba2e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.159 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:18.158+0000 7f420e400640 1 --2- 172.21.15.45:0/1958578229 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42080b6b20 0x7f42080ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.159 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:18.158+0000 7f420e400640 1 -- 172.21.15.45:0/1958578229 >> 172.21.15.45:0/1958578229 conn(0x7f4208073710 msgr2=0x7f42080b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:18.159 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:18.158+0000 7f420e400640 1 -- 172.21.15.45:0/1958578229 shutdown_connections 2024-04-16T16:13:18.159 INFO:teuthology.misc.health.smithi045.stderr:2024-04-16T16:13:18.158+0000 7f420e400640 1 -- 172.21.15.45:0/1958578229 wait complete. 2024-04-16T16:13:18.171 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-16T16:13:18.171 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-16T16:13:18.172 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-16T16:13:18.172 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-16T16:13:18.179 INFO:tasks.ceph.mds.a:Started 2024-04-16T16:13:18.179 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-16T16:13:18.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-16T16:13:18.186 INFO:tasks.ceph.mds.c:Started 2024-04-16T16:13:18.186 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-16T16:13:18.187 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-16T16:13:18.194 INFO:tasks.ceph.mds.b:Started 2024-04-16T16:13:18.194 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-16T16:13:18.194 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-16T16:13:18.201 INFO:tasks.ceph.mds.d:Started 2024-04-16T16:13:18.202 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-16T16:13:18.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:13:18.326 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2024-04-16T16:13:18.336 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2024-04-16T16:13:18.360 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-16T16:13:18.360 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-16T16:13:18.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.382+0000 7fe1cea00640 1 Processor -- start 2024-04-16T16:13:18.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.382+0000 7fe1cea00640 1 -- start start 2024-04-16T16:13:18.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.383+0000 7fe1cea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1c80d1ac0 0x7fe1c8169680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:18.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.383+0000 7fe1cea00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1c80b89b0 con 0x7fe1c80d4560 2024-04-16T16:13:18.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.384+0000 7fe1cea00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1c80d34c0 con 0x7fe1c815ec60 2024-04-16T16:13:18.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.384+0000 7fe1cea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe1c80d37a0 con 0x7fe1c80d1ac0 2024-04-16T16:13:18.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.384+0000 7fe1c7400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe1c80d4560 0x7fe1c80d13b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54908/0 (socket says 172.21.15.45:54908) 2024-04-16T16:13:18.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.384+0000 7fe1c7400640 1 -- 172.21.15.45:0/3380487591 learned_addr learned my addr 172.21.15.45:0/3380487591 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:18.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.384+0000 7fe1cca00640 1 --2- 172.21.15.45:0/3380487591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1c80d1ac0 0x7fe1c8169680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:18.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.384+0000 7fe1c6a00640 1 -- 172.21.15.45:0/3380487591 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 146610035 0 0) 0x7fe1c80b89b0 con 0x7fe1c80d4560 2024-04-16T16:13:18.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.384+0000 7fe1c6a00640 1 -- 172.21.15.45:0/3380487591 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1ac003520 con 0x7fe1c80d4560 2024-04-16T16:13:18.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.384+0000 7fe1c6a00640 1 -- 172.21.15.45:0/3380487591 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 971977799 0 0) 0x7fe1c80d34c0 con 0x7fe1c815ec60 2024-04-16T16:13:18.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.384+0000 7fe1c6a00640 1 -- 172.21.15.45:0/3380487591 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1c80b89b0 con 0x7fe1c815ec60 2024-04-16T16:13:18.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.384+0000 7fe1cca00640 1 -- 172.21.15.45:0/3380487591 >> v1:172.21.15.103:6789/0 conn(0x7fe1c815ec60 legacy=0x7fe1c80d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:18.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.384+0000 7fe1cca00640 1 -- 172.21.15.45:0/3380487591 >> v1:172.21.15.45:6789/0 conn(0x7fe1c80d4560 legacy=0x7fe1c80d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:18.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.384+0000 7fe1cca00640 1 -- 172.21.15.45:0/3380487591 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c80d3a80 con 0x7fe1c80d1ac0 2024-04-16T16:13:18.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.385+0000 7fe1cca00640 1 --2- 172.21.15.45:0/3380487591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1c80d1ac0 0x7fe1c8169680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00097d0 tx=0x7fe1c002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32b5b531243f2a7a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:18.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.386+0000 7fe1c6a00640 1 -- 172.21.15.45:0/3380487591 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0035020 con 0x7fe1c80d1ac0 2024-04-16T16:13:18.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.386+0000 7fe1c6a00640 1 -- 172.21.15.45:0/3380487591 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c0031890 con 0x7fe1c80d1ac0 2024-04-16T16:13:18.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.386+0000 7fe1c6a00640 1 -- 172.21.15.45:0/3380487591 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0031c60 con 0x7fe1c80d1ac0 2024-04-16T16:13:18.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.386+0000 7fe1cea00640 1 -- 172.21.15.45:0/3380487591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1c80d1ac0 msgr2=0x7fe1c8169680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:18.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.386+0000 7fe1cea00640 1 --2- 172.21.15.45:0/3380487591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1c80d1ac0 0x7fe1c8169680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00097d0 tx=0x7fe1c002b930 comp rx=0 tx=0).stop 2024-04-16T16:13:18.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.386+0000 7fe1cea00640 1 -- 172.21.15.45:0/3380487591 shutdown_connections 2024-04-16T16:13:18.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.386+0000 7fe1cea00640 1 --2- 172.21.15.45:0/3380487591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1c80d1ac0 0x7fe1c8169680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.386+0000 7fe1cea00640 1 -- 172.21.15.45:0/3380487591 >> 172.21.15.45:0/3380487591 conn(0x7fe1c8073690 msgr2=0x7fe1c806cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:18.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.386+0000 7fe1cea00640 1 -- 172.21.15.45:0/3380487591 shutdown_connections 2024-04-16T16:13:18.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.386+0000 7fe1cea00640 1 -- 172.21.15.45:0/3380487591 wait complete. 2024-04-16T16:13:18.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.388+0000 7fe1cea00640 1 Processor -- start 2024-04-16T16:13:18.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.388+0000 7fe1cea00640 1 -- start start 2024-04-16T16:13:18.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.389+0000 7fe1cea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1c80d1ac0 0x7fe1c80b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:18.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.389+0000 7fe1cea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1c80d4560 0x7fe1c80adf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:18.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.389+0000 7fe1cea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1c815ec60 0x7fe1c815ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:18.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.389+0000 7fe1cea00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c80ba460 con 0x7fe1c815ec60 2024-04-16T16:13:18.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.389+0000 7fe1cea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c80d4040 con 0x7fe1c80d1ac0 2024-04-16T16:13:18.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.389+0000 7fe1cea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe1c80d3110 con 0x7fe1c80d4560 2024-04-16T16:13:18.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.389+0000 7fe1cca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1c815ec60 0x7fe1c815ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:18.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.389+0000 7fe1cca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1c815ec60 0x7fe1c815ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38088/0 (socket says 172.21.15.45:38088) 2024-04-16T16:13:18.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.389+0000 7fe1cca00640 1 -- 172.21.15.45:0/229889719 learned_addr learned my addr 172.21.15.45:0/229889719 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:18.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.389+0000 7fe1c7e00640 1 --2- 172.21.15.45:0/229889719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1c80d1ac0 0x7fe1c80b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:18.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.389+0000 7fe1c7400640 1 --2- 172.21.15.45:0/229889719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1c80d4560 0x7fe1c80adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:18.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.390+0000 7fe1cca00640 1 -- 172.21.15.45:0/229889719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1c80d4560 msgr2=0x7fe1c80adf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:18.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.390+0000 7fe1cca00640 1 --2- 172.21.15.45:0/229889719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1c80d4560 0x7fe1c80adf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.390+0000 7fe1cca00640 1 -- 172.21.15.45:0/229889719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1c80d1ac0 msgr2=0x7fe1c80b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:18.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.390+0000 7fe1cca00640 1 --2- 172.21.15.45:0/229889719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1c80d1ac0 0x7fe1c80b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.390+0000 7fe1cca00640 1 -- 172.21.15.45:0/229889719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c80ae460 con 0x7fe1c815ec60 2024-04-16T16:13:18.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.390+0000 7fe1c7400640 1 --2- 172.21.15.45:0/229889719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1c80d4560 0x7fe1c80adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:18.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.390+0000 7fe1cca00640 1 --2- 172.21.15.45:0/229889719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1c815ec60 0x7fe1c815ce70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00098f0 tx=0x7fe1c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:18.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.390+0000 7fe1bfe00640 1 -- 172.21.15.45:0/229889719 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0035040 con 0x7fe1c815ec60 2024-04-16T16:13:18.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.390+0000 7fe1bfe00640 1 -- 172.21.15.45:0/229889719 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c0030a90 con 0x7fe1c815ec60 2024-04-16T16:13:18.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.390+0000 7fe1bfe00640 1 -- 172.21.15.45:0/229889719 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0044430 con 0x7fe1c815ec60 2024-04-16T16:13:18.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.390+0000 7fe1cea00640 1 -- 172.21.15.45:0/229889719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1c80ae700 con 0x7fe1c815ec60 2024-04-16T16:13:18.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.390+0000 7fe1cea00640 1 -- 172.21.15.45:0/229889719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1c80ac6a0 con 0x7fe1c815ec60 2024-04-16T16:13:18.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.391+0000 7fe1cea00640 1 -- 172.21.15.45:0/229889719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1c8075780 con 0x7fe1c815ec60 2024-04-16T16:13:18.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.392+0000 7fe1bfe00640 1 -- 172.21.15.45:0/229889719 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7fe1c0038020 con 0x7fe1c815ec60 2024-04-16T16:13:18.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.393+0000 7fe1bfe00640 1 --2- 172.21.15.45:0/229889719 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe1a406c860 0x7fe1a406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:18.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.393+0000 7fe1bfe00640 1 -- 172.21.15.45:0/229889719 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7fe1c00adab0 con 0x7fe1c815ec60 2024-04-16T16:13:18.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.394+0000 7fe1c7e00640 1 --2- 172.21.15.45:0/229889719 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe1a406c860 0x7fe1a406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:18.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.394+0000 7fe1c7e00640 1 --2- 172.21.15.45:0/229889719 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe1a406c860 0x7fe1a406ed20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8001560 tx=0x7fe1b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:18.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.398+0000 7fe1bfe00640 1 -- 172.21.15.45:0/229889719 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe1c002d040 con 0x7fe1c815ec60 2024-04-16T16:13:18.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.593+0000 7fe1cea00640 1 -- 172.21.15.45:0/229889719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe1c80ba960 con 0x7fe1c815ec60 2024-04-16T16:13:18.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.594+0000 7fe1bfe00640 1 -- 172.21.15.45:0/229889719 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+10909 (secure 0 0 0) 0x7fe1c002f340 con 0x7fe1c815ec60 2024-04-16T16:13:18.593 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:18.593 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":13,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:17.804403+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:13:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.596+0000 7fe1cea00640 1 -- 172.21.15.45:0/229889719 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe1a406c860 msgr2=0x7fe1a406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.596+0000 7fe1cea00640 1 --2- 172.21.15.45:0/229889719 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe1a406c860 0x7fe1a406ed20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8001560 tx=0x7fe1b8009040 comp rx=0 tx=0).stop 2024-04-16T16:13:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.596+0000 7fe1cea00640 1 -- 172.21.15.45:0/229889719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1c815ec60 msgr2=0x7fe1c815ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.596+0000 7fe1cea00640 1 --2- 172.21.15.45:0/229889719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1c815ec60 0x7fe1c815ce70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00098f0 tx=0x7fe1c00046c0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.597+0000 7fe1cea00640 1 -- 172.21.15.45:0/229889719 shutdown_connections 2024-04-16T16:13:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.597+0000 7fe1cea00640 1 --2- 172.21.15.45:0/229889719 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe1a406c860 0x7fe1a406ed20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.597+0000 7fe1cea00640 1 --2- 172.21.15.45:0/229889719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1c815ec60 0x7fe1c815ce70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.597+0000 7fe1cea00640 1 --2- 172.21.15.45:0/229889719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1c80d4560 0x7fe1c80adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.597+0000 7fe1cea00640 1 --2- 172.21.15.45:0/229889719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1c80d1ac0 0x7fe1c80b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.597+0000 7fe1cea00640 1 -- 172.21.15.45:0/229889719 >> 172.21.15.45:0/229889719 conn(0x7fe1c8073690 msgr2=0x7fe1c80cf360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.597+0000 7fe1cea00640 1 -- 172.21.15.45:0/229889719 shutdown_connections 2024-04-16T16:13:18.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.597+0000 7fe1cea00640 1 -- 172.21.15.45:0/229889719 wait complete. 2024-04-16T16:13:18.612 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-04-16T16:13:18.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:13:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.824+0000 7f33e0200640 1 Processor -- start 2024-04-16T16:13:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.824+0000 7f33e0200640 1 -- start start 2024-04-16T16:13:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.826+0000 7f33e0200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f33d80d0980 0x7f33d80d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.826+0000 7f33e0200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33d8164590 0x7f33d8166980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.826+0000 7f33e0200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33d806c040 con 0x7f33d8146350 2024-04-16T16:13:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.826+0000 7f33e0200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f33d80cf2a0 con 0x7f33d80d0980 2024-04-16T16:13:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.826+0000 7f33e0200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f33d815d920 con 0x7f33d8164590 2024-04-16T16:13:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.827+0000 7f33dce00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f33d80d0980 0x7f33d80d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.827+0000 7f33dd800640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f33d8146350 0x7f33d806b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:43002/0 (socket says 172.21.15.45:43002) 2024-04-16T16:13:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.827+0000 7f33de200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33d8164590 0x7f33d8166980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.827+0000 7f33dd800640 1 -- 172.21.15.45:0/142738875 learned_addr learned my addr 172.21.15.45:0/142738875 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.827+0000 7f33d7e00640 1 -- 172.21.15.45:0/142738875 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 257081616 0 0) 0x7f33d806c040 con 0x7f33d8146350 2024-04-16T16:13:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.827+0000 7f33d7e00640 1 -- 172.21.15.45:0/142738875 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33c0003520 con 0x7f33d8146350 2024-04-16T16:13:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.828+0000 7f33dce00640 1 -- 172.21.15.45:0/142738875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33d8164590 msgr2=0x7f33d8166980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.828+0000 7f33dce00640 1 --2- 172.21.15.45:0/142738875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33d8164590 0x7f33d8166980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.828+0000 7f33dce00640 1 -- 172.21.15.45:0/142738875 >> v1:172.21.15.103:6789/0 conn(0x7f33d8146350 legacy=0x7f33d806b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.828+0000 7f33dce00640 1 -- 172.21.15.45:0/142738875 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d806c310 con 0x7f33d80d0980 2024-04-16T16:13:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.828+0000 7f33de200640 1 --2- 172.21.15.45:0/142738875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33d8164590 0x7f33d8166980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.829+0000 7f33dce00640 1 --2- 172.21.15.45:0/142738875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33d80d0980 0x7f33d80d0e10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0097d0 tx=0x7f33cc02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41a27baa1e410710 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.829+0000 7f33d7e00640 1 -- 172.21.15.45:0/142738875 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc035020 con 0x7f33d80d0980 2024-04-16T16:13:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.829+0000 7f33d7e00640 1 -- 172.21.15.45:0/142738875 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33cc031810 con 0x7f33d80d0980 2024-04-16T16:13:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.829+0000 7f33d7e00640 1 -- 172.21.15.45:0/142738875 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc031bc0 con 0x7f33d80d0980 2024-04-16T16:13:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.830+0000 7f33e0200640 1 -- 172.21.15.45:0/142738875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33d80d0980 msgr2=0x7f33d80d0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.830+0000 7f33e0200640 1 --2- 172.21.15.45:0/142738875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33d80d0980 0x7f33d80d0e10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0097d0 tx=0x7f33cc02b8b0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.830+0000 7f33e0200640 1 -- 172.21.15.45:0/142738875 shutdown_connections 2024-04-16T16:13:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.830+0000 7f33e0200640 1 --2- 172.21.15.45:0/142738875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33d8164590 0x7f33d8166980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.830+0000 7f33e0200640 1 --2- 172.21.15.45:0/142738875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33d80d0980 0x7f33d80d0e10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.830+0000 7f33e0200640 1 -- 172.21.15.45:0/142738875 >> 172.21.15.45:0/142738875 conn(0x7f33d809aa70 msgr2=0x7f33d80b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.830+0000 7f33e0200640 1 -- 172.21.15.45:0/142738875 shutdown_connections 2024-04-16T16:13:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.830+0000 7f33e0200640 1 -- 172.21.15.45:0/142738875 wait complete. 2024-04-16T16:13:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.833+0000 7f33e0200640 1 Processor -- start 2024-04-16T16:13:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.833+0000 7f33e0200640 1 -- start start 2024-04-16T16:13:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.834+0000 7f33e0200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33d80d0980 0x7f33d815f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.834+0000 7f33e0200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33d8146350 0x7f33d813fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.834+0000 7f33e0200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33d8164590 0x7f33d8140270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.834+0000 7f33e0200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f33d815cf80 con 0x7f33d80d0980 2024-04-16T16:13:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.834+0000 7f33e0200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33d815d5b0 con 0x7f33d8164590 2024-04-16T16:13:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.834+0000 7f33e0200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f33d815ddb0 con 0x7f33d8146350 2024-04-16T16:13:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.834+0000 7f33dd800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33d80d0980 0x7f33d815f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.834+0000 7f33dd800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33d80d0980 0x7f33d815f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38098/0 (socket says 172.21.15.45:38098) 2024-04-16T16:13:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.834+0000 7f33dd800640 1 -- 172.21.15.45:0/2315155636 learned_addr learned my addr 172.21.15.45:0/2315155636 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.835+0000 7f33dce00640 1 --2- 172.21.15.45:0/2315155636 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33d8146350 0x7f33d813fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.835+0000 7f33de200640 1 --2- 172.21.15.45:0/2315155636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33d8164590 0x7f33d8140270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.835+0000 7f33dd800640 1 -- 172.21.15.45:0/2315155636 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33d8146350 msgr2=0x7f33d813fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.835+0000 7f33dd800640 1 --2- 172.21.15.45:0/2315155636 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33d8146350 0x7f33d813fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.835+0000 7f33dd800640 1 -- 172.21.15.45:0/2315155636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33d8164590 msgr2=0x7f33d8140270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.835+0000 7f33dd800640 1 --2- 172.21.15.45:0/2315155636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33d8164590 0x7f33d8140270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.835+0000 7f33dd800640 1 -- 172.21.15.45:0/2315155636 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d80caef0 con 0x7f33d80d0980 2024-04-16T16:13:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.835+0000 7f33dd800640 1 --2- 172.21.15.45:0/2315155636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33d80d0980 0x7f33d815f370 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f33c800c430 tx=0x7f33c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.836+0000 7f33d5600640 1 -- 172.21.15.45:0/2315155636 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c8017070 con 0x7f33d80d0980 2024-04-16T16:13:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.836+0000 7f33d5600640 1 -- 172.21.15.45:0/2315155636 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33c8005ce0 con 0x7f33d80d0980 2024-04-16T16:13:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.836+0000 7f33d5600640 1 -- 172.21.15.45:0/2315155636 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c80072f0 con 0x7f33d80d0980 2024-04-16T16:13:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.836+0000 7f33e0200640 1 -- 172.21.15.45:0/2315155636 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33d80cb1d0 con 0x7f33d80d0980 2024-04-16T16:13:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.836+0000 7f33e0200640 1 -- 172.21.15.45:0/2315155636 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33d80cb790 con 0x7f33d80d0980 2024-04-16T16:13:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.837+0000 7f33e0200640 1 -- 172.21.15.45:0/2315155636 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33d8075780 con 0x7f33d80d0980 2024-04-16T16:13:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.838+0000 7f33d5600640 1 -- 172.21.15.45:0/2315155636 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f33c801a070 con 0x7f33d80d0980 2024-04-16T16:13:18.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.839+0000 7f33d5600640 1 --2- 172.21.15.45:0/2315155636 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f33bc06c860 0x7f33bc06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:18.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.839+0000 7f33d5600640 1 -- 172.21.15.45:0/2315155636 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 4471+0+0 (secure 0 0 0) 0x7f33c808aad0 con 0x7f33d80d0980 2024-04-16T16:13:18.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.840+0000 7f33dce00640 1 --2- 172.21.15.45:0/2315155636 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f33bc06c860 0x7f33bc06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:18.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.841+0000 7f33dce00640 1 --2- 172.21.15.45:0/2315155636 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f33bc06c860 0x7f33bc06ed20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f33cc036000 tx=0x7f33cc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:18.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:18.844+0000 7f33d5600640 1 -- 172.21.15.45:0/2315155636 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f33c808ce20 con 0x7f33d80d0980 2024-04-16T16:13:19.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:19.096+0000 7f33e0200640 1 -- 172.21.15.45:0/2315155636 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33d806f5d0 con 0x7f33d80d0980 2024-04-16T16:13:19.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:19.097+0000 7f33d5600640 1 -- 172.21.15.45:0/2315155636 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1389 (secure 0 0 0) 0x7f33c8059410 con 0x7f33d80d0980 2024-04-16T16:13:19.096 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:19.097 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/315848518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":315848518},{"type":"v1","addr":"172.21.15.45:6833","nonce":315848518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-16T16:13:19.097 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 2 2024-04-16T16:13:19.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:19.101+0000 7f33e0200640 1 -- 172.21.15.45:0/2315155636 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f33bc06c860 msgr2=0x7f33bc06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:19.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:19.101+0000 7f33e0200640 1 --2- 172.21.15.45:0/2315155636 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f33bc06c860 0x7f33bc06ed20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f33cc036000 tx=0x7f33cc008030 comp rx=0 tx=0).stop 2024-04-16T16:13:19.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:19.101+0000 7f33e0200640 1 -- 172.21.15.45:0/2315155636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33d80d0980 msgr2=0x7f33d815f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:19.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:19.101+0000 7f33e0200640 1 --2- 172.21.15.45:0/2315155636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33d80d0980 0x7f33d815f370 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f33c800c430 tx=0x7f33c800c8f0 comp rx=0 tx=0).stop 2024-04-16T16:13:19.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:19.101+0000 7f33e0200640 1 -- 172.21.15.45:0/2315155636 shutdown_connections 2024-04-16T16:13:19.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:19.101+0000 7f33e0200640 1 --2- 172.21.15.45:0/2315155636 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f33bc06c860 0x7f33bc06ed20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:19.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:19.101+0000 7f33e0200640 1 --2- 172.21.15.45:0/2315155636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33d8164590 0x7f33d8140270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:19.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:19.101+0000 7f33e0200640 1 --2- 172.21.15.45:0/2315155636 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33d8146350 0x7f33d813fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:19.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:19.101+0000 7f33e0200640 1 --2- 172.21.15.45:0/2315155636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33d80d0980 0x7f33d815f370 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:19.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:19.101+0000 7f33e0200640 1 -- 172.21.15.45:0/2315155636 >> 172.21.15.45:0/2315155636 conn(0x7f33d809aa70 msgr2=0x7f33d80ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:19.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:19.101+0000 7f33e0200640 1 -- 172.21.15.45:0/2315155636 shutdown_connections 2024-04-16T16:13:19.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:19.101+0000 7f33e0200640 1 -- 172.21.15.45:0/2315155636 wait complete. 2024-04-16T16:13:21.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:13:21.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.343+0000 7f8ce0a00640 1 Processor -- start 2024-04-16T16:13:21.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.343+0000 7f8ce0a00640 1 -- start start 2024-04-16T16:13:21.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.345+0000 7f8ce0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8cdc15ec60 0x7f8cdc0d2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:21.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.345+0000 7f8ce0a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8cdc0d4560 0x7f8cdc0d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:21.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.345+0000 7f8ce0a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cdc0b89b0 con 0x7f8cdc0d2f40 2024-04-16T16:13:21.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.345+0000 7f8ce0a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f8cdc0d34c0 con 0x7f8cdc0d4560 2024-04-16T16:13:21.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.345+0000 7f8ce0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8cdc0d37a0 con 0x7f8cdc15ec60 2024-04-16T16:13:21.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.345+0000 7f8cda800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8cdc0d2f40 0x7f8cdc0b6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54918/0 (socket says 172.21.15.45:54918) 2024-04-16T16:13:21.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.345+0000 7f8cda800640 1 -- 172.21.15.45:0/1325528578 learned_addr learned my addr 172.21.15.45:0/1325528578 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:21.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd9e00640 1 --2- 172.21.15.45:0/1325528578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8cdc15ec60 0x7f8cdc0d2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:21.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 -- 172.21.15.45:0/1325528578 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1696082579 0 0) 0x7f8cdc0b89b0 con 0x7f8cdc0d2f40 2024-04-16T16:13:21.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd9400640 1 --2- 172.21.15.45:0/1325528578 >> v2:172.21.15.103:3300/0 conn(0x7f8cdc0d4560 0x7f8cdc0d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:21.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 -- 172.21.15.45:0/1325528578 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cc0003520 con 0x7f8cdc0d2f40 2024-04-16T16:13:21.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 -- 172.21.15.45:0/1325528578 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3132730504 0 0) 0x7f8cc0003520 con 0x7f8cdc0d2f40 2024-04-16T16:13:21.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 -- 172.21.15.45:0/1325528578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8cdc15ec60 msgr2=0x7f8cdc0d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:21.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 --2- 172.21.15.45:0/1325528578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8cdc15ec60 0x7f8cdc0d2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 -- 172.21.15.45:0/1325528578 >> v2:172.21.15.103:3300/0 conn(0x7f8cdc0d4560 msgr2=0x7f8cdc0d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:21.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 --2- 172.21.15.45:0/1325528578 >> v2:172.21.15.103:3300/0 conn(0x7f8cdc0d4560 0x7f8cdc0d13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 -- 172.21.15.45:0/1325528578 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cdc0d3a80 con 0x7f8cdc0d2f40 2024-04-16T16:13:21.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 -- 172.21.15.45:0/1325528578 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f8cd40019f0 con 0x7f8cdc0d2f40 2024-04-16T16:13:21.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 -- 172.21.15.45:0/1325528578 >> v1:172.21.15.45:6789/0 conn(0x7f8cdc0d2f40 legacy=0x7f8cdc0b6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:21.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 --2- 172.21.15.45:0/1325528578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cc00039b0 0x7f8cc0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:21.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 --2- 172.21.15.45:0/1325528578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8cdc15ec60 0x7f8cc0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:21.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 --2- 172.21.15.45:0/1325528578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8cc0004d40 0x7f8cc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:21.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 -- 172.21.15.45:0/1325528578 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8cdc0d34c0 con 0x7f8cc00039b0 2024-04-16T16:13:21.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 -- 172.21.15.45:0/1325528578 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8cdc0d37a0 con 0x7f8cc0004d40 2024-04-16T16:13:21.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd8a00640 1 -- 172.21.15.45:0/1325528578 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8cc000b9c0 con 0x7f8cdc15ec60 2024-04-16T16:13:21.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.346+0000 7f8cd9e00640 1 --2- 172.21.15.45:0/1325528578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cc00039b0 0x7f8cc0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:21.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.347+0000 7f8cd9400640 1 --2- 172.21.15.45:0/1325528578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8cdc15ec60 0x7f8cc0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:21.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.347+0000 7f8cd9e00640 1 -- 172.21.15.45:0/1325528578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8cdc15ec60 msgr2=0x7f8cc0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:21.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.347+0000 7f8cda800640 1 --2- 172.21.15.45:0/1325528578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8cc0004d40 0x7f8cc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:21.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.347+0000 7f8cd9e00640 1 --2- 172.21.15.45:0/1325528578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8cdc15ec60 0x7f8cc0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.347+0000 7f8cd9e00640 1 -- 172.21.15.45:0/1325528578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8cc0004d40 msgr2=0x7f8cc00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:21.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.347+0000 7f8cd9e00640 1 --2- 172.21.15.45:0/1325528578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8cc0004d40 0x7f8cc00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.347+0000 7f8cd9e00640 1 -- 172.21.15.45:0/1325528578 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc0003520 con 0x7f8cc00039b0 2024-04-16T16:13:21.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.347+0000 7f8cda800640 1 --2- 172.21.15.45:0/1325528578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8cc0004d40 0x7f8cc00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:21.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.348+0000 7f8cd9e00640 1 --2- 172.21.15.45:0/1325528578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cc00039b0 0x7f8cc0003dd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc009ac0 tx=0x7f8cc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657dc9edb4b29780 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:21.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.348+0000 7f8cd8a00640 1 -- 172.21.15.45:0/1325528578 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc039070 con 0x7f8cc00039b0 2024-04-16T16:13:21.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.348+0000 7f8cd8a00640 1 -- 172.21.15.45:0/1325528578 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ccc02c6b0 con 0x7f8cc00039b0 2024-04-16T16:13:21.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.348+0000 7f8cd8a00640 1 -- 172.21.15.45:0/1325528578 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc009690 con 0x7f8cc00039b0 2024-04-16T16:13:21.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.349+0000 7f8ce0a00640 1 -- 172.21.15.45:0/1325528578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cc00039b0 msgr2=0x7f8cc0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:21.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.349+0000 7f8ce0a00640 1 --2- 172.21.15.45:0/1325528578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cc00039b0 0x7f8cc0003dd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc009ac0 tx=0x7f8cc0005830 comp rx=0 tx=0).stop 2024-04-16T16:13:21.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.349+0000 7f8cda800640 1 -- 172.21.15.45:0/1325528578 reap_dead start 2024-04-16T16:13:21.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.349+0000 7f8ce0a00640 1 -- 172.21.15.45:0/1325528578 shutdown_connections 2024-04-16T16:13:21.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.349+0000 7f8ce0a00640 1 -- 172.21.15.45:0/1325528578 >> 172.21.15.45:0/1325528578 conn(0x7f8cdc073690 msgr2=0x7f8cdc06cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:21.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.349+0000 7f8ce0a00640 1 -- 172.21.15.45:0/1325528578 shutdown_connections 2024-04-16T16:13:21.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.349+0000 7f8ce0a00640 1 -- 172.21.15.45:0/1325528578 wait complete. 2024-04-16T16:13:21.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.351+0000 7f8ce0a00640 1 Processor -- start 2024-04-16T16:13:21.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.351+0000 7f8ce0a00640 1 -- start start 2024-04-16T16:13:21.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.352+0000 7f8ce0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8cc00039b0 0x7f8cdc0b8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:21.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.352+0000 7f8ce0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8cdc1f5a00 0x7f8cdc0cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:21.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.352+0000 7f8ce0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cdc14d8e0 0x7f8cdc14dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:21.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.352+0000 7f8ce0a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8cdc0ba460 con 0x7f8cdc14d8e0 2024-04-16T16:13:21.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.352+0000 7f8ce0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8cdc0d4040 con 0x7f8cc00039b0 2024-04-16T16:13:21.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.352+0000 7f8ce0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8cdc0d2a00 con 0x7f8cdc1f5a00 2024-04-16T16:13:21.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.352+0000 7f8cda800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cdc14d8e0 0x7f8cdc14dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:21.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.353+0000 7f8cda800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cdc14d8e0 0x7f8cdc14dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38126/0 (socket says 172.21.15.45:38126) 2024-04-16T16:13:21.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.353+0000 7f8cda800640 1 -- 172.21.15.45:0/2129441062 learned_addr learned my addr 172.21.15.45:0/2129441062 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:21.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.353+0000 7f8cd9400640 1 --2- 172.21.15.45:0/2129441062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8cdc1f5a00 0x7f8cdc0cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:21.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.353+0000 7f8cd9e00640 1 --2- 172.21.15.45:0/2129441062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8cc00039b0 0x7f8cdc0b8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:21.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.353+0000 7f8cda800640 1 -- 172.21.15.45:0/2129441062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8cdc1f5a00 msgr2=0x7f8cdc0cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:21.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.353+0000 7f8cda800640 1 --2- 172.21.15.45:0/2129441062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8cdc1f5a00 0x7f8cdc0cf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.353+0000 7f8cda800640 1 -- 172.21.15.45:0/2129441062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8cc00039b0 msgr2=0x7f8cdc0b8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:21.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.353+0000 7f8cda800640 1 --2- 172.21.15.45:0/2129441062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8cc00039b0 0x7f8cdc0b8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.353+0000 7f8cda800640 1 -- 172.21.15.45:0/2129441062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cdc14e780 con 0x7f8cdc14d8e0 2024-04-16T16:13:21.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.353+0000 7f8cda800640 1 --2- 172.21.15.45:0/2129441062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cdc14d8e0 0x7f8cdc14dd60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc0b7530 tx=0x7f8cd400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:21.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.354+0000 7f8cd2000640 1 -- 172.21.15.45:0/2129441062 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd4004d60 con 0x7f8cdc14d8e0 2024-04-16T16:13:21.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.354+0000 7f8cd2000640 1 -- 172.21.15.45:0/2129441062 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cd400bb80 con 0x7f8cdc14d8e0 2024-04-16T16:13:21.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.354+0000 7f8ce0a00640 1 -- 172.21.15.45:0/2129441062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cdc14ea60 con 0x7f8cdc14d8e0 2024-04-16T16:13:21.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.354+0000 7f8cd2000640 1 -- 172.21.15.45:0/2129441062 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd4007070 con 0x7f8cdc14d8e0 2024-04-16T16:13:21.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.354+0000 7f8ce0a00640 1 -- 172.21.15.45:0/2129441062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cdc14f020 con 0x7f8cdc14d8e0 2024-04-16T16:13:21.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.356+0000 7f8ce0a00640 1 -- 172.21.15.45:0/2129441062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cdc075780 con 0x7f8cdc14d8e0 2024-04-16T16:13:21.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.356+0000 7f8cd2000640 1 -- 172.21.15.45:0/2129441062 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f8cd401b020 con 0x7f8cdc14d8e0 2024-04-16T16:13:21.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.358+0000 7f8cd2000640 1 --2- 172.21.15.45:0/2129441062 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8cbc06c890 0x7f8cbc06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:21.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.358+0000 7f8cd2000640 1 -- 172.21.15.45:0/2129441062 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f8cd408b8d0 con 0x7f8cdc14d8e0 2024-04-16T16:13:21.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.359+0000 7f8cd9e00640 1 --2- 172.21.15.45:0/2129441062 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8cbc06c890 0x7f8cbc06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:21.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.360+0000 7f8cd9e00640 1 --2- 172.21.15.45:0/2129441062 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8cbc06c890 0x7f8cbc06ed50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc036000 tx=0x7f8ccc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:21.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.363+0000 7f8cd2000640 1 -- 172.21.15.45:0/2129441062 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8cd405a1f0 con 0x7f8cdc14d8e0 2024-04-16T16:13:21.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.621+0000 7f8ce0a00640 1 -- 172.21.15.45:0/2129441062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8cdc0ba960 con 0x7f8cdc14d8e0 2024-04-16T16:13:21.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.622+0000 7f8cd2000640 1 -- 172.21.15.45:0/2129441062 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3867 (secure 0 0 0) 0x7f8cd4059af0 con 0x7f8cdc14d8e0 2024-04-16T16:13:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:21.621 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2486891866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2486891866},{"type":"v1","addr":"172.21.15.103:6836","nonce":2486891866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1877281473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1877281473},{"type":"v1","addr":"172.21.15.103:6837","nonce":1877281473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/315848518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":315848518},{"type":"v1","addr":"172.21.15.45:6833","nonce":315848518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2461453826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2461453826},{"type":"v1","addr":"172.21.15.45:6835","nonce":2461453826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-16T16:13:21.621 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 3 2024-04-16T16:13:21.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.625+0000 7f8ce0a00640 1 -- 172.21.15.45:0/2129441062 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8cbc06c890 msgr2=0x7f8cbc06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:21.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.625+0000 7f8ce0a00640 1 --2- 172.21.15.45:0/2129441062 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8cbc06c890 0x7f8cbc06ed50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc036000 tx=0x7f8ccc030000 comp rx=0 tx=0).stop 2024-04-16T16:13:21.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.625+0000 7f8ce0a00640 1 -- 172.21.15.45:0/2129441062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cdc14d8e0 msgr2=0x7f8cdc14dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:21.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.625+0000 7f8ce0a00640 1 --2- 172.21.15.45:0/2129441062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cdc14d8e0 0x7f8cdc14dd60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc0b7530 tx=0x7f8cd400b4a0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.625+0000 7f8ce0a00640 1 -- 172.21.15.45:0/2129441062 shutdown_connections 2024-04-16T16:13:21.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.625+0000 7f8ce0a00640 1 --2- 172.21.15.45:0/2129441062 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8cbc06c890 0x7f8cbc06ed50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.625+0000 7f8ce0a00640 1 --2- 172.21.15.45:0/2129441062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cdc14d8e0 0x7f8cdc14dd60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.625+0000 7f8ce0a00640 1 --2- 172.21.15.45:0/2129441062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8cdc1f5a00 0x7f8cdc0cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.625+0000 7f8ce0a00640 1 --2- 172.21.15.45:0/2129441062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8cc00039b0 0x7f8cdc0b8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.625+0000 7f8ce0a00640 1 -- 172.21.15.45:0/2129441062 >> 172.21.15.45:0/2129441062 conn(0x7f8cdc073690 msgr2=0x7f8cdc0ca0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:21.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.625+0000 7f8ce0a00640 1 -- 172.21.15.45:0/2129441062 shutdown_connections 2024-04-16T16:13:21.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.625+0000 7f8ce0a00640 1 -- 172.21.15.45:0/2129441062 wait complete. 2024-04-16T16:13:21.642 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:13:21.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.837+0000 7fa763200640 1 Processor -- start 2024-04-16T16:13:21.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.837+0000 7fa763200640 1 -- start start 2024-04-16T16:13:21.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.839+0000 7fa763200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa764161e90 0x7fa7640cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:21.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.839+0000 7fa763200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7640d52d0 0x7fa76406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:21.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.839+0000 7fa763200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7640d3080 con 0x7fa764162270 2024-04-16T16:13:21.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.839+0000 7fa763200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa7640d2e30 con 0x7fa764161e90 2024-04-16T16:13:21.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.839+0000 7fa763200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa7640d3360 con 0x7fa7640d52d0 2024-04-16T16:13:21.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.839+0000 7fa761e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa764161e90 0x7fa7640cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:21.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.839+0000 7fa761e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa764161e90 0x7fa7640cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38138/0 (socket says 172.21.15.45:38138) 2024-04-16T16:13:21.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.839+0000 7fa761e00640 1 -- 172.21.15.45:0/3238766568 learned_addr learned my addr 172.21.15.45:0/3238766568 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:21.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.840+0000 7fa761e00640 1 -- 172.21.15.45:0/3238766568 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7640d52d0 msgr2=0x7fa76406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:21.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.840+0000 7fa761e00640 1 --2- 172.21.15.45:0/3238766568 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7640d52d0 0x7fa76406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.840+0000 7fa761e00640 1 -- 172.21.15.45:0/3238766568 >> v1:172.21.15.103:6789/0 conn(0x7fa764162270 legacy=0x7fa7640d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:21.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.840+0000 7fa761e00640 1 -- 172.21.15.45:0/3238766568 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7640d3640 con 0x7fa764161e90 2024-04-16T16:13:21.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.841+0000 7fa761e00640 1 --2- 172.21.15.45:0/3238766568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa764161e90 0x7fa7640cf7d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa7540098a0 tx=0x7fa75402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc84a20cdd60b72 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:21.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.841+0000 7fa760a00640 1 -- 172.21.15.45:0/3238766568 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754034070 con 0x7fa764161e90 2024-04-16T16:13:21.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.841+0000 7fa760a00640 1 -- 172.21.15.45:0/3238766568 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa754030840 con 0x7fa764161e90 2024-04-16T16:13:21.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.842+0000 7fa760a00640 1 -- 172.21.15.45:0/3238766568 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754030c10 con 0x7fa764161e90 2024-04-16T16:13:21.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.842+0000 7fa763200640 1 -- 172.21.15.45:0/3238766568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa764161e90 msgr2=0x7fa7640cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:21.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.842+0000 7fa763200640 1 --2- 172.21.15.45:0/3238766568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa764161e90 0x7fa7640cf7d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa7540098a0 tx=0x7fa75402b980 comp rx=0 tx=0).stop 2024-04-16T16:13:21.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.842+0000 7fa763200640 1 -- 172.21.15.45:0/3238766568 shutdown_connections 2024-04-16T16:13:21.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.842+0000 7fa763200640 1 --2- 172.21.15.45:0/3238766568 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7640d52d0 0x7fa76406ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.842+0000 7fa763200640 1 --2- 172.21.15.45:0/3238766568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa764161e90 0x7fa7640cf7d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.842+0000 7fa763200640 1 -- 172.21.15.45:0/3238766568 >> 172.21.15.45:0/3238766568 conn(0x7fa7640d1940 msgr2=0x7fa7640d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:21.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.843+0000 7fa763200640 1 -- 172.21.15.45:0/3238766568 shutdown_connections 2024-04-16T16:13:21.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.843+0000 7fa763200640 1 -- 172.21.15.45:0/3238766568 wait complete. 2024-04-16T16:13:21.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.844+0000 7fa763200640 1 Processor -- start 2024-04-16T16:13:21.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.845+0000 7fa763200640 1 -- start start 2024-04-16T16:13:21.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.846+0000 7fa763200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7640d52d0 0x7fa7641e7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:21.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.846+0000 7fa763200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa764161e90 0x7fa76415efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:21.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.846+0000 7fa763200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa764162270 0x7fa7641e85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:21.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.846+0000 7fa763200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa7640d4550 con 0x7fa7640d52d0 2024-04-16T16:13:21.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.846+0000 7fa763200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa7640cfd10 con 0x7fa764161e90 2024-04-16T16:13:21.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.846+0000 7fa763200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa7640d46d0 con 0x7fa764162270 2024-04-16T16:13:21.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.846+0000 7fa761e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7640d52d0 0x7fa7641e7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:21.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.846+0000 7fa761e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7640d52d0 0x7fa7641e7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38150/0 (socket says 172.21.15.45:38150) 2024-04-16T16:13:21.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.846+0000 7fa761e00640 1 -- 172.21.15.45:0/227033494 learned_addr learned my addr 172.21.15.45:0/227033494 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:21.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.846+0000 7fa761400640 1 --2- 172.21.15.45:0/227033494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa764161e90 0x7fa76415efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:21.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.847+0000 7fa762800640 1 --2- 172.21.15.45:0/227033494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa764162270 0x7fa7641e85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:21.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.847+0000 7fa761e00640 1 -- 172.21.15.45:0/227033494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa764162270 msgr2=0x7fa7641e85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:21.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.847+0000 7fa761e00640 1 --2- 172.21.15.45:0/227033494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa764162270 0x7fa7641e85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.847+0000 7fa761e00640 1 -- 172.21.15.45:0/227033494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa764161e90 msgr2=0x7fa76415efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:21.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.847+0000 7fa761e00640 1 --2- 172.21.15.45:0/227033494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa764161e90 0x7fa76415efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:21.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.847+0000 7fa761e00640 1 -- 172.21.15.45:0/227033494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7640b5310 con 0x7fa7640d52d0 2024-04-16T16:13:21.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.847+0000 7fa761400640 1 --2- 172.21.15.45:0/227033494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa764161e90 0x7fa76415efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:13:21.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.847+0000 7fa761e00640 1 --2- 172.21.15.45:0/227033494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7640d52d0 0x7fa7641e7db0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa7540098a0 tx=0x7fa754004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:21.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.848+0000 7fa75a000640 1 -- 172.21.15.45:0/227033494 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754048070 con 0x7fa7640d52d0 2024-04-16T16:13:21.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.848+0000 7fa75a000640 1 -- 172.21.15.45:0/227033494 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa754030640 con 0x7fa7640d52d0 2024-04-16T16:13:21.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.848+0000 7fa75a000640 1 -- 172.21.15.45:0/227033494 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754034040 con 0x7fa7640d52d0 2024-04-16T16:13:21.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.848+0000 7fa763200640 1 -- 172.21.15.45:0/227033494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7640b55b0 con 0x7fa7640d52d0 2024-04-16T16:13:21.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.848+0000 7fa763200640 1 -- 172.21.15.45:0/227033494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7641e82f0 con 0x7fa7640d52d0 2024-04-16T16:13:21.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.849+0000 7fa763200640 1 -- 172.21.15.45:0/227033494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa764075780 con 0x7fa7640d52d0 2024-04-16T16:13:21.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.850+0000 7fa75a000640 1 -- 172.21.15.45:0/227033494 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7fa754037070 con 0x7fa7640d52d0 2024-04-16T16:13:21.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.851+0000 7fa75a000640 1 --2- 172.21.15.45:0/227033494 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa74406c860 0x7fa74406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:21.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.852+0000 7fa75a000640 1 -- 172.21.15.45:0/227033494 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fa7540ad600 con 0x7fa7640d52d0 2024-04-16T16:13:21.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.852+0000 7fa761400640 1 --2- 172.21.15.45:0/227033494 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa74406c860 0x7fa74406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:21.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.854+0000 7fa761400640 1 --2- 172.21.15.45:0/227033494 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa74406c860 0x7fa74406ed20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa750006fd0 tx=0x7fa750008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:21.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:21.856+0000 7fa75a000640 1 -- 172.21.15.45:0/227033494 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa75407c030 con 0x7fa7640d52d0 2024-04-16T16:13:22.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.068+0000 7fa763200640 1 -- 172.21.15.45:0/227033494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa76406dae0 con 0x7fa7640d52d0 2024-04-16T16:13:22.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.070+0000 7fa75a000640 1 -- 172.21.15.45:0/227033494 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12884 (secure 0 0 0) 0x7fa75407b930 con 0x7fa7640d52d0 2024-04-16T16:13:22.070 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:22.070 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":17,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:21.836099+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:13:22.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.073+0000 7fa763200640 1 -- 172.21.15.45:0/227033494 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa74406c860 msgr2=0x7fa74406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:22.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.073+0000 7fa763200640 1 --2- 172.21.15.45:0/227033494 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa74406c860 0x7fa74406ed20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa750006fd0 tx=0x7fa750008040 comp rx=0 tx=0).stop 2024-04-16T16:13:22.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.073+0000 7fa763200640 1 -- 172.21.15.45:0/227033494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7640d52d0 msgr2=0x7fa7641e7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:22.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.073+0000 7fa763200640 1 --2- 172.21.15.45:0/227033494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7640d52d0 0x7fa7641e7db0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa7540098a0 tx=0x7fa754004730 comp rx=0 tx=0).stop 2024-04-16T16:13:22.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.074+0000 7fa763200640 1 -- 172.21.15.45:0/227033494 shutdown_connections 2024-04-16T16:13:22.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.074+0000 7fa763200640 1 --2- 172.21.15.45:0/227033494 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa74406c860 0x7fa74406ed20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.074+0000 7fa763200640 1 --2- 172.21.15.45:0/227033494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa764162270 0x7fa7641e85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.074+0000 7fa763200640 1 --2- 172.21.15.45:0/227033494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa764161e90 0x7fa76415efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.074+0000 7fa763200640 1 --2- 172.21.15.45:0/227033494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7640d52d0 0x7fa7641e7db0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.074+0000 7fa763200640 1 -- 172.21.15.45:0/227033494 >> 172.21.15.45:0/227033494 conn(0x7fa7640d1940 msgr2=0x7fa7640ca390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:22.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.074+0000 7fa763200640 1 -- 172.21.15.45:0/227033494 shutdown_connections 2024-04-16T16:13:22.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.074+0000 7fa763200640 1 -- 172.21.15.45:0/227033494 wait complete. 2024-04-16T16:13:22.095 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:13:22.095 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:13:22.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.327+0000 7f1e69800640 1 Processor -- start 2024-04-16T16:13:22.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.328+0000 7f1e69800640 1 -- start start 2024-04-16T16:13:22.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.329+0000 7f1e69800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1e640d10e0 0x7f1e64072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:22.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.329+0000 7f1e69800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1e640b4930 0x7f1e6406f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:22.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.329+0000 7f1e69800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e640b5530 con 0x7f1e6415c7c0 2024-04-16T16:13:22.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.329+0000 7f1e69800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1e640b52e0 con 0x7f1e640b4930 2024-04-16T16:13:22.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.329+0000 7f1e69800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1e640d0260 con 0x7f1e640d10e0 2024-04-16T16:13:22.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.329+0000 7f1e63400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1e640b4930 0x7f1e6406f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:22.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.330+0000 7f1e63400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1e640b4930 0x7f1e6406f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38162/0 (socket says 172.21.15.45:38162) 2024-04-16T16:13:22.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.330+0000 7f1e63400640 1 -- 172.21.15.45:0/6216568 learned_addr learned my addr 172.21.15.45:0/6216568 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:22.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.330+0000 7f1e5be00640 1 --2- 172.21.15.45:0/6216568 >> v2:172.21.15.103:3300/0 conn(0x7f1e640d10e0 0x7f1e64072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:22.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.330+0000 7f1e63400640 1 -- 172.21.15.45:0/6216568 >> v2:172.21.15.103:3300/0 conn(0x7f1e640d10e0 msgr2=0x7f1e64072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:22.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.330+0000 7f1e63400640 1 --2- 172.21.15.45:0/6216568 >> v2:172.21.15.103:3300/0 conn(0x7f1e640d10e0 0x7f1e64072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.330+0000 7f1e63400640 1 -- 172.21.15.45:0/6216568 >> v1:172.21.15.103:6789/0 conn(0x7f1e6415c7c0 legacy=0x7f1e640d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:22.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.330+0000 7f1e63400640 1 -- 172.21.15.45:0/6216568 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e640b5810 con 0x7f1e640b4930 2024-04-16T16:13:22.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.331+0000 7f1e63400640 1 --2- 172.21.15.45:0/6216568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e640b4930 0x7f1e6406f160 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1e640b3900 tx=0x7f1e5c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc21face452cf95c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:22.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.331+0000 7f1e62000640 1 -- 172.21.15.45:0/6216568 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c02c960 con 0x7f1e640b4930 2024-04-16T16:13:22.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.332+0000 7f1e62000640 1 -- 172.21.15.45:0/6216568 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e5c02cb00 con 0x7f1e640b4930 2024-04-16T16:13:22.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.332+0000 7f1e62000640 1 -- 172.21.15.45:0/6216568 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c0343f0 con 0x7f1e640b4930 2024-04-16T16:13:22.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.332+0000 7f1e69800640 1 -- 172.21.15.45:0/6216568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e640b4930 msgr2=0x7f1e6406f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:22.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.332+0000 7f1e69800640 1 --2- 172.21.15.45:0/6216568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e640b4930 0x7f1e6406f160 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1e640b3900 tx=0x7f1e5c02bdc0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.332+0000 7f1e69800640 1 -- 172.21.15.45:0/6216568 shutdown_connections 2024-04-16T16:13:22.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.332+0000 7f1e69800640 1 --2- 172.21.15.45:0/6216568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e640b4930 0x7f1e6406f160 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.332+0000 7f1e69800640 1 --2- 172.21.15.45:0/6216568 >> v2:172.21.15.103:3300/0 conn(0x7f1e640d10e0 0x7f1e64072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.332+0000 7f1e69800640 1 -- 172.21.15.45:0/6216568 >> 172.21.15.45:0/6216568 conn(0x7f1e640b3970 msgr2=0x7f1e640b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:22.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.332+0000 7f1e69800640 1 -- 172.21.15.45:0/6216568 shutdown_connections 2024-04-16T16:13:22.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.332+0000 7f1e69800640 1 -- 172.21.15.45:0/6216568 wait complete. 2024-04-16T16:13:22.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.334+0000 7f1e69800640 1 Processor -- start 2024-04-16T16:13:22.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.334+0000 7f1e69800640 1 -- start start 2024-04-16T16:13:22.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.335+0000 7f1e69800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e640b4930 0x7f1e64145de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:22.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.335+0000 7f1e69800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e640d10e0 0x7f1e640ce950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:22.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.335+0000 7f1e69800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e6415c7c0 0x7f1e640cee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:22.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.335+0000 7f1e69800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6406f6a0 con 0x7f1e6415c7c0 2024-04-16T16:13:22.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.335+0000 7f1e69800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e64073350 con 0x7f1e640d10e0 2024-04-16T16:13:22.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.335+0000 7f1e69800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1e6406f820 con 0x7f1e640b4930 2024-04-16T16:13:22.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.336+0000 7f1e63400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e6415c7c0 0x7f1e640cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:22.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.336+0000 7f1e63400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e6415c7c0 0x7f1e640cee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38168/0 (socket says 172.21.15.45:38168) 2024-04-16T16:13:22.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.336+0000 7f1e63400640 1 -- 172.21.15.45:0/618717588 learned_addr learned my addr 172.21.15.45:0/618717588 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:22.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.336+0000 7f1e62a00640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e640b4930 0x7f1e64145de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:22.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.336+0000 7f1e5be00640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e640d10e0 0x7f1e640ce950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:22.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.336+0000 7f1e63400640 1 -- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e640b4930 msgr2=0x7f1e64145de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:22.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.336+0000 7f1e63400640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e640b4930 0x7f1e64145de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.336+0000 7f1e63400640 1 -- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e640d10e0 msgr2=0x7f1e640ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:22.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.336+0000 7f1e63400640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e640d10e0 0x7f1e640ce950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.336+0000 7f1e63400640 1 -- 172.21.15.45:0/618717588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e641e8410 con 0x7f1e6415c7c0 2024-04-16T16:13:22.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.336+0000 7f1e62a00640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e640b4930 0x7f1e64145de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:13:22.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.336+0000 7f1e5be00640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e640d10e0 0x7f1e640ce950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:13:22.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.337+0000 7f1e63400640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e6415c7c0 0x7f1e640cee90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c009f40 tx=0x7f1e5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:22.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.337+0000 7f1e5aa00640 1 -- 172.21.15.45:0/618717588 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c043020 con 0x7f1e6415c7c0 2024-04-16T16:13:22.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.337+0000 7f1e5aa00640 1 -- 172.21.15.45:0/618717588 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e5c03e450 con 0x7f1e6415c7c0 2024-04-16T16:13:22.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.337+0000 7f1e5aa00640 1 -- 172.21.15.45:0/618717588 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c03e820 con 0x7f1e6415c7c0 2024-04-16T16:13:22.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.337+0000 7f1e69800640 1 -- 172.21.15.45:0/618717588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e64149200 con 0x7f1e6415c7c0 2024-04-16T16:13:22.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.337+0000 7f1e69800640 1 -- 172.21.15.45:0/618717588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e641468c0 con 0x7f1e6415c7c0 2024-04-16T16:13:22.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.338+0000 7f1e69800640 1 -- 172.21.15.45:0/618717588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e64075780 con 0x7f1e6415c7c0 2024-04-16T16:13:22.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.339+0000 7f1e5aa00640 1 -- 172.21.15.45:0/618717588 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f1e5c046020 con 0x7f1e6415c7c0 2024-04-16T16:13:22.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.340+0000 7f1e5aa00640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1e4006c860 0x7f1e4006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:22.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.340+0000 7f1e5aa00640 1 -- 172.21.15.45:0/618717588 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f1e5c0ae170 con 0x7f1e6415c7c0 2024-04-16T16:13:22.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.341+0000 7f1e62a00640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1e4006c860 0x7f1e4006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:22.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.342+0000 7f1e62a00640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1e4006c860 0x7f1e4006ed20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1e54006fd0 tx=0x7f1e54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:22.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.344+0000 7f1e5aa00640 1 -- 172.21.15.45:0/618717588 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1e5c04d030 con 0x7f1e6415c7c0 2024-04-16T16:13:22.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.551+0000 7f1e69800640 1 -- 172.21.15.45:0/618717588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1e6415aa80 con 0x7f1e6415c7c0 2024-04-16T16:13:22.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.552+0000 7f1e5aa00640 1 -- 172.21.15.45:0/618717588 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f1e5c07c8a0 con 0x7f1e6415c7c0 2024-04-16T16:13:22.551 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:13:22.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.555+0000 7f1e69800640 1 -- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1e4006c860 msgr2=0x7f1e4006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:22.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.555+0000 7f1e69800640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1e4006c860 0x7f1e4006ed20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1e54006fd0 tx=0x7f1e54008040 comp rx=0 tx=0).stop 2024-04-16T16:13:22.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.555+0000 7f1e69800640 1 -- 172.21.15.45:0/618717588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e6415c7c0 msgr2=0x7f1e640cee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:22.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.555+0000 7f1e69800640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e6415c7c0 0x7f1e640cee90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c009f40 tx=0x7f1e5c004a40 comp rx=0 tx=0).stop 2024-04-16T16:13:22.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.555+0000 7f1e69800640 1 -- 172.21.15.45:0/618717588 shutdown_connections 2024-04-16T16:13:22.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.555+0000 7f1e69800640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1e4006c860 0x7f1e4006ed20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.555+0000 7f1e69800640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e6415c7c0 0x7f1e640cee90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.555+0000 7f1e69800640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e640d10e0 0x7f1e640ce950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.555+0000 7f1e69800640 1 --2- 172.21.15.45:0/618717588 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e640b4930 0x7f1e64145de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.555+0000 7f1e69800640 1 -- 172.21.15.45:0/618717588 >> 172.21.15.45:0/618717588 conn(0x7f1e640b3970 msgr2=0x7f1e640b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:22.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.555+0000 7f1e69800640 1 -- 172.21.15.45:0/618717588 shutdown_connections 2024-04-16T16:13:22.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.556+0000 7f1e69800640 1 -- 172.21.15.45:0/618717588 wait complete. 2024-04-16T16:13:22.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-16T16:13:22.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.803+0000 7eff4b200640 1 Processor -- start 2024-04-16T16:13:22.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.803+0000 7eff4b200640 1 -- start start 2024-04-16T16:13:22.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.805+0000 7eff4b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff4406e550 0x7eff4409abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:22.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.805+0000 7eff4b200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff440ccf50 con 0x7eff440d1990 2024-04-16T16:13:22.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.805+0000 7eff4b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff440cd230 con 0x7eff44070420 2024-04-16T16:13:22.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.805+0000 7eff4b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7eff440cd880 con 0x7eff4406e550 2024-04-16T16:13:22.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.805+0000 7eff49200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7eff440d1990 0x7eff440ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54932/0 (socket says 172.21.15.45:54932) 2024-04-16T16:13:22.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.805+0000 7eff49200640 1 -- 172.21.15.45:0/2204777598 learned_addr learned my addr 172.21.15.45:0/2204777598 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:22.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.806+0000 7eff43e00640 1 --2- 172.21.15.45:0/2204777598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff4406e550 0x7eff4409abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:22.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.806+0000 7eff43400640 1 -- 172.21.15.45:0/2204777598 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3777728263 0 0) 0x7eff440ccf50 con 0x7eff440d1990 2024-04-16T16:13:22.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.806+0000 7eff43400640 1 -- 172.21.15.45:0/2204777598 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff28003520 con 0x7eff440d1990 2024-04-16T16:13:22.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.806+0000 7eff43400640 1 -- 172.21.15.45:0/2204777598 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1735318933 0 0) 0x7eff440cd230 con 0x7eff44070420 2024-04-16T16:13:22.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.806+0000 7eff43400640 1 -- 172.21.15.45:0/2204777598 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff440ccf50 con 0x7eff44070420 2024-04-16T16:13:22.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.806+0000 7eff43e00640 1 -- 172.21.15.45:0/2204777598 >> v1:172.21.15.103:6789/0 conn(0x7eff44070420 legacy=0x7eff440cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:22.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.806+0000 7eff43e00640 1 -- 172.21.15.45:0/2204777598 >> v1:172.21.15.45:6789/0 conn(0x7eff440d1990 legacy=0x7eff440ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:22.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.806+0000 7eff43e00640 1 -- 172.21.15.45:0/2204777598 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff440cd510 con 0x7eff4406e550 2024-04-16T16:13:22.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.808+0000 7eff43e00640 1 --2- 172.21.15.45:0/2204777598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff4406e550 0x7eff4409abc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7eff440cda00 tx=0x7eff3402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d07239e6b4cc385 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.808+0000 7eff43400640 1 -- 172.21.15.45:0/2204777598 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff34035020 con 0x7eff4406e550 2024-04-16T16:13:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.808+0000 7eff43400640 1 -- 172.21.15.45:0/2204777598 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff34031890 con 0x7eff4406e550 2024-04-16T16:13:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.808+0000 7eff43400640 1 -- 172.21.15.45:0/2204777598 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff34031c60 con 0x7eff4406e550 2024-04-16T16:13:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.809+0000 7eff4b200640 1 -- 172.21.15.45:0/2204777598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff4406e550 msgr2=0x7eff4409abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.809+0000 7eff4b200640 1 --2- 172.21.15.45:0/2204777598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff4406e550 0x7eff4409abc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7eff440cda00 tx=0x7eff3402b930 comp rx=0 tx=0).stop 2024-04-16T16:13:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.809+0000 7eff4b200640 1 -- 172.21.15.45:0/2204777598 shutdown_connections 2024-04-16T16:13:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.809+0000 7eff4b200640 1 --2- 172.21.15.45:0/2204777598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff4406e550 0x7eff4409abc0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.809+0000 7eff4b200640 1 -- 172.21.15.45:0/2204777598 >> 172.21.15.45:0/2204777598 conn(0x7eff440b4350 msgr2=0x7eff440b4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.809+0000 7eff4b200640 1 -- 172.21.15.45:0/2204777598 shutdown_connections 2024-04-16T16:13:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.809+0000 7eff4b200640 1 -- 172.21.15.45:0/2204777598 wait complete. 2024-04-16T16:13:22.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.811+0000 7eff4b200640 1 Processor -- start 2024-04-16T16:13:22.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.811+0000 7eff4b200640 1 -- start start 2024-04-16T16:13:22.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.812+0000 7eff4b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff4406e550 0x7eff441e31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:22.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.812+0000 7eff4b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff44070420 0x7eff440cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:22.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.812+0000 7eff4b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff440d1990 0x7eff440cb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:22.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.812+0000 7eff4b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7eff4406ca40 con 0x7eff440d1990 2024-04-16T16:13:22.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.812+0000 7eff4b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7eff4406e930 con 0x7eff4406e550 2024-04-16T16:13:22.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.812+0000 7eff4b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7eff441f8d10 con 0x7eff44070420 2024-04-16T16:13:22.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.812+0000 7eff49200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff440d1990 0x7eff440cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:22.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.812+0000 7eff49200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff440d1990 0x7eff440cb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38184/0 (socket says 172.21.15.45:38184) 2024-04-16T16:13:22.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.812+0000 7eff49200640 1 -- 172.21.15.45:0/34239890 learned_addr learned my addr 172.21.15.45:0/34239890 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:22.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.812+0000 7eff43e00640 1 --2- 172.21.15.45:0/34239890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff4406e550 0x7eff441e31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:22.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.812+0000 7eff3be00640 1 --2- 172.21.15.45:0/34239890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff44070420 0x7eff440cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:22.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.812+0000 7eff49200640 1 -- 172.21.15.45:0/34239890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff44070420 msgr2=0x7eff440cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:22.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.813+0000 7eff49200640 1 --2- 172.21.15.45:0/34239890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff44070420 0x7eff440cafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.813+0000 7eff49200640 1 -- 172.21.15.45:0/34239890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff4406e550 msgr2=0x7eff441e31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:22.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.813+0000 7eff49200640 1 --2- 172.21.15.45:0/34239890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff4406e550 0x7eff441e31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:22.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.813+0000 7eff49200640 1 -- 172.21.15.45:0/34239890 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff441f7f50 con 0x7eff440d1990 2024-04-16T16:13:22.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.813+0000 7eff3be00640 1 --2- 172.21.15.45:0/34239890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff44070420 0x7eff440cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:22.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.813+0000 7eff49200640 1 --2- 172.21.15.45:0/34239890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff440d1990 0x7eff440cb510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7eff3c00cbc0 tx=0x7eff3c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:22.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.813+0000 7eff40c00640 1 -- 172.21.15.45:0/34239890 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3c017070 con 0x7eff440d1990 2024-04-16T16:13:22.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.813+0000 7eff40c00640 1 -- 172.21.15.45:0/34239890 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff3c00ace0 con 0x7eff440d1990 2024-04-16T16:13:22.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.813+0000 7eff40c00640 1 -- 172.21.15.45:0/34239890 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3c012600 con 0x7eff440d1990 2024-04-16T16:13:22.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.813+0000 7eff4b200640 1 -- 172.21.15.45:0/34239890 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff441f8230 con 0x7eff440d1990 2024-04-16T16:13:22.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.813+0000 7eff4b200640 1 -- 172.21.15.45:0/34239890 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff441f87f0 con 0x7eff440d1990 2024-04-16T16:13:22.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.814+0000 7eff4b200640 1 -- 172.21.15.45:0/34239890 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff441f89f0 con 0x7eff440d1990 2024-04-16T16:13:22.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.815+0000 7eff40c00640 1 -- 172.21.15.45:0/34239890 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7eff3c01a070 con 0x7eff440d1990 2024-04-16T16:13:22.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.816+0000 7eff40c00640 1 --2- 172.21.15.45:0/34239890 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7eff2006c860 0x7eff2006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:22.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.817+0000 7eff40c00640 1 -- 172.21.15.45:0/34239890 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7eff3c08b510 con 0x7eff440d1990 2024-04-16T16:13:22.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.817+0000 7eff43e00640 1 --2- 172.21.15.45:0/34239890 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7eff2006c860 0x7eff2006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:22.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.818+0000 7eff43e00640 1 --2- 172.21.15.45:0/34239890 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7eff2006c860 0x7eff2006ed20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7eff34036000 tx=0x7eff3403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:22.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:22.821+0000 7eff40c00640 1 -- 172.21.15.45:0/34239890 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7eff441f89f0 con 0x7eff440d1990 2024-04-16T16:13:22.889 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:13:22.897 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:13:23.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.033+0000 7eff4b200640 1 -- 172.21.15.45:0/34239890 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7eff44075780 con 0x7eff440d1990 2024-04-16T16:13:23.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.065+0000 7eff40c00640 1 -- 172.21.15.45:0/34239890 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7eff3c059680 con 0x7eff440d1990 2024-04-16T16:13:23.064 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:23.064 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2024-04-16T16:13:23.064 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:13:23.064 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2024-04-16T16:13:23.064 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2024-04-16T16:13:23.064 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:13:23.064 INFO:teuthology.orchestra.run.smithi045.stdout:] 2024-04-16T16:13:23.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.067+0000 7eff4b200640 1 -- 172.21.15.45:0/34239890 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7eff2006c860 msgr2=0x7eff2006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:23.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.067+0000 7eff4b200640 1 --2- 172.21.15.45:0/34239890 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7eff2006c860 0x7eff2006ed20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7eff34036000 tx=0x7eff3403c040 comp rx=0 tx=0).stop 2024-04-16T16:13:23.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.067+0000 7eff4b200640 1 -- 172.21.15.45:0/34239890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff440d1990 msgr2=0x7eff440cb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:23.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.067+0000 7eff4b200640 1 --2- 172.21.15.45:0/34239890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff440d1990 0x7eff440cb510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7eff3c00cbc0 tx=0x7eff3c00a430 comp rx=0 tx=0).stop 2024-04-16T16:13:23.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.068+0000 7eff4b200640 1 -- 172.21.15.45:0/34239890 shutdown_connections 2024-04-16T16:13:23.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.068+0000 7eff4b200640 1 --2- 172.21.15.45:0/34239890 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7eff2006c860 0x7eff2006ed20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:23.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.068+0000 7eff4b200640 1 --2- 172.21.15.45:0/34239890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff440d1990 0x7eff440cb510 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:23.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.068+0000 7eff4b200640 1 --2- 172.21.15.45:0/34239890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff44070420 0x7eff440cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:23.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.068+0000 7eff4b200640 1 --2- 172.21.15.45:0/34239890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff4406e550 0x7eff441e31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:23.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.068+0000 7eff4b200640 1 -- 172.21.15.45:0/34239890 >> 172.21.15.45:0/34239890 conn(0x7eff440b4350 msgr2=0x7eff441e6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:23.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.068+0000 7eff4b200640 1 -- 172.21.15.45:0/34239890 shutdown_connections 2024-04-16T16:13:23.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.068+0000 7eff4b200640 1 -- 172.21.15.45:0/34239890 wait complete. 2024-04-16T16:13:23.087 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-16T16:13:23.087 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-16T16:13:23.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.359+0000 7f9af3e00640 1 Processor -- start 2024-04-16T16:13:23.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.360+0000 7f9af3e00640 1 -- start start 2024-04-16T16:13:23.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.361+0000 7f9af3e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9af4155ab0 0x7f9af4162ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:23.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.361+0000 7f9af3e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9af40ba610 0x7f9af415efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:23.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.361+0000 7f9af3e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9af4155e90 con 0x7f9af40d18c0 2024-04-16T16:13:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.361+0000 7f9af3e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9af415ff00 con 0x7f9af4155ab0 2024-04-16T16:13:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.361+0000 7f9af3e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9af415f940 con 0x7f9af40ba610 2024-04-16T16:13:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.361+0000 7f9af2a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9af4155ab0 0x7f9af4162ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.361+0000 7f9af2a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9af4155ab0 0x7f9af4162ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49538/0 (socket says 172.21.15.45:49538) 2024-04-16T16:13:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.361+0000 7f9af3400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9af40d18c0 0x7f9af40d2a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40248/0 (socket says 172.21.15.45:40248) 2024-04-16T16:13:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.361+0000 7f9af2a00640 1 -- 172.21.15.45:0/3479861669 learned_addr learned my addr 172.21.15.45:0/3479861669 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.362+0000 7f9af2000640 1 --2- 172.21.15.45:0/3479861669 >> v2:172.21.15.103:3300/0 conn(0x7f9af40ba610 0x7f9af415efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:23.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.362+0000 7f9af2a00640 1 -- 172.21.15.45:0/3479861669 >> v2:172.21.15.103:3300/0 conn(0x7f9af40ba610 msgr2=0x7f9af415efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:23.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.362+0000 7f9af2a00640 1 --2- 172.21.15.45:0/3479861669 >> v2:172.21.15.103:3300/0 conn(0x7f9af40ba610 0x7f9af415efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:23.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.362+0000 7f9af2a00640 1 -- 172.21.15.45:0/3479861669 >> v1:172.21.15.45:6789/0 conn(0x7f9af40d18c0 legacy=0x7f9af40d2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:23.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.362+0000 7f9af2a00640 1 -- 172.21.15.45:0/3479861669 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af415fc20 con 0x7f9af4155ab0 2024-04-16T16:13:23.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.362+0000 7f9af1600640 1 -- 172.21.15.45:0/3479861669 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4047881594 0 0) 0x7f9af4155e90 con 0x7f9af40d18c0 2024-04-16T16:13:23.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.363+0000 7f9af2a00640 1 --2- 172.21.15.45:0/3479861669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af4155ab0 0x7f9af4162ef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9af40b66c0 tx=0x7f9ae402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ee8cc69500040a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:23.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.364+0000 7f9af1600640 1 -- 172.21.15.45:0/3479861669 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4035020 con 0x7f9af4155ab0 2024-04-16T16:13:23.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.364+0000 7f9af1600640 1 -- 172.21.15.45:0/3479861669 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ae4031870 con 0x7f9af4155ab0 2024-04-16T16:13:23.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.364+0000 7f9af1600640 1 -- 172.21.15.45:0/3479861669 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4031c40 con 0x7f9af4155ab0 2024-04-16T16:13:23.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.364+0000 7f9af3e00640 1 -- 172.21.15.45:0/3479861669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af4155ab0 msgr2=0x7f9af4162ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:23.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.364+0000 7f9af3e00640 1 --2- 172.21.15.45:0/3479861669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af4155ab0 0x7f9af4162ef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9af40b66c0 tx=0x7f9ae402b910 comp rx=0 tx=0).stop 2024-04-16T16:13:23.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.364+0000 7f9af3e00640 1 -- 172.21.15.45:0/3479861669 shutdown_connections 2024-04-16T16:13:23.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.364+0000 7f9af3e00640 1 --2- 172.21.15.45:0/3479861669 >> v2:172.21.15.103:3300/0 conn(0x7f9af40ba610 0x7f9af415efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:23.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.364+0000 7f9af3e00640 1 --2- 172.21.15.45:0/3479861669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af4155ab0 0x7f9af4162ef0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:23.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.364+0000 7f9af3e00640 1 -- 172.21.15.45:0/3479861669 >> 172.21.15.45:0/3479861669 conn(0x7f9af406d1a0 msgr2=0x7f9af406d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:23.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.364+0000 7f9af3e00640 1 -- 172.21.15.45:0/3479861669 shutdown_connections 2024-04-16T16:13:23.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.364+0000 7f9af3e00640 1 -- 172.21.15.45:0/3479861669 wait complete. 2024-04-16T16:13:23.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.366+0000 7f9af3e00640 1 Processor -- start 2024-04-16T16:13:23.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.366+0000 7f9af3e00640 1 -- start start 2024-04-16T16:13:23.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.367+0000 7f9af3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af40ba610 0x7f9af4155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:23.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.367+0000 7f9af3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9af40d18c0 0x7f9af4150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:23.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.367+0000 7f9af3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af4155ab0 0x7f9af41576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:23.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.367+0000 7f9af3e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9af4163430 con 0x7f9af4155ab0 2024-04-16T16:13:23.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.367+0000 7f9af3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9af41629d0 con 0x7f9af40ba610 2024-04-16T16:13:23.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.367+0000 7f9af3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9af415f510 con 0x7f9af40d18c0 2024-04-16T16:13:23.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.368+0000 7f9af2a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af40ba610 0x7f9af4155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:23.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.368+0000 7f9af2a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af40ba610 0x7f9af4155e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:37302/0 (socket says 172.21.15.45:37302) 2024-04-16T16:13:23.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.368+0000 7f9af2a00640 1 -- 172.21.15.45:0/2860609514 learned_addr learned my addr 172.21.15.45:0/2860609514 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:23.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.368+0000 7f9af3400640 1 --2- 172.21.15.45:0/2860609514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af4155ab0 0x7f9af41576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:23.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.368+0000 7f9af2a00640 1 -- 172.21.15.45:0/2860609514 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9af40d18c0 msgr2=0x7f9af4150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:23.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.368+0000 7f9af2a00640 1 --2- 172.21.15.45:0/2860609514 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9af40d18c0 0x7f9af4150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.368+0000 7f9af2a00640 1 -- 172.21.15.45:0/2860609514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af4155ab0 msgr2=0x7f9af41576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.368+0000 7f9af2a00640 1 --2- 172.21.15.45:0/2860609514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af4155ab0 0x7f9af41576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.368+0000 7f9af2a00640 1 -- 172.21.15.45:0/2860609514 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af4156600 con 0x7f9af40ba610 2024-04-16T16:13:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.369+0000 7f9af2a00640 1 --2- 172.21.15.45:0/2860609514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af40ba610 0x7f9af4155e90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9ae402b950 tx=0x7f9ae40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:23.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.369+0000 7f9aeaa00640 1 -- 172.21.15.45:0/2860609514 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4040070 con 0x7f9af40ba610 2024-04-16T16:13:23.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.369+0000 7f9aeaa00640 1 -- 172.21.15.45:0/2860609514 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ae4030d00 con 0x7f9af40ba610 2024-04-16T16:13:23.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.369+0000 7f9aeaa00640 1 -- 172.21.15.45:0/2860609514 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4035040 con 0x7f9af40ba610 2024-04-16T16:13:23.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.369+0000 7f9af3e00640 1 -- 172.21.15.45:0/2860609514 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9af41568e0 con 0x7f9af40ba610 2024-04-16T16:13:23.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.369+0000 7f9af3e00640 1 -- 172.21.15.45:0/2860609514 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9af4156ea0 con 0x7f9af40ba610 2024-04-16T16:13:23.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.370+0000 7f9af3e00640 1 -- 172.21.15.45:0/2860609514 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9af4075780 con 0x7f9af40ba610 2024-04-16T16:13:23.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.372+0000 7f9aeaa00640 1 -- 172.21.15.45:0/2860609514 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f9ae404b060 con 0x7f9af40ba610 2024-04-16T16:13:23.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.373+0000 7f9aeaa00640 1 --2- 172.21.15.45:0/2860609514 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9ad006c860 0x7f9ad006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:23.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.373+0000 7f9aeaa00640 1 -- 172.21.15.45:0/2860609514 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f9ae40ad800 con 0x7f9af40ba610 2024-04-16T16:13:23.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.373+0000 7f9af2000640 1 --2- 172.21.15.45:0/2860609514 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9ad006c860 0x7f9ad006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:23.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.374+0000 7f9af2000640 1 --2- 172.21.15.45:0/2860609514 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9ad006c860 0x7f9ad006ed20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9af40cb0b0 tx=0x7f9adc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:23.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.378+0000 7f9aeaa00640 1 -- 172.21.15.45:0/2860609514 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9ae407c040 con 0x7f9af40ba610 2024-04-16T16:13:23.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.582+0000 7f9af3e00640 1 -- 172.21.15.45:0/2860609514 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f9af4162df0 con 0x7f9af40ba610 2024-04-16T16:13:23.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.891+0000 7f9aeaa00640 1 -- 172.21.15.45:0/2860609514 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9ae4038020 con 0x7f9af40ba610 2024-04-16T16:13:23.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.892+0000 7f9aeaa00640 1 -- 172.21.15.45:0/2860609514 <== mon.1 v2:172.21.15.103:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f9ae40af500 con 0x7f9af40ba610 2024-04-16T16:13:23.892 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2024-04-16T16:13:23.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.899+0000 7f9af3e00640 1 -- 172.21.15.45:0/2860609514 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9ad006c860 msgr2=0x7f9ad006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:23.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.899+0000 7f9af3e00640 1 --2- 172.21.15.45:0/2860609514 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9ad006c860 0x7f9ad006ed20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9af40cb0b0 tx=0x7f9adc00b040 comp rx=0 tx=0).stop 2024-04-16T16:13:23.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.899+0000 7f9af3e00640 1 -- 172.21.15.45:0/2860609514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af40ba610 msgr2=0x7f9af4155e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:23.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.899+0000 7f9af3e00640 1 --2- 172.21.15.45:0/2860609514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af40ba610 0x7f9af4155e90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9ae402b950 tx=0x7f9ae40046c0 comp rx=0 tx=0).stop 2024-04-16T16:13:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.900+0000 7f9af3e00640 1 -- 172.21.15.45:0/2860609514 shutdown_connections 2024-04-16T16:13:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.900+0000 7f9af3e00640 1 --2- 172.21.15.45:0/2860609514 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9ad006c860 0x7f9ad006ed20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.900+0000 7f9af3e00640 1 --2- 172.21.15.45:0/2860609514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af4155ab0 0x7f9af41576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.900+0000 7f9af3e00640 1 --2- 172.21.15.45:0/2860609514 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9af40d18c0 0x7f9af4150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.900+0000 7f9af3e00640 1 --2- 172.21.15.45:0/2860609514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af40ba610 0x7f9af4155e90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.900+0000 7f9af3e00640 1 -- 172.21.15.45:0/2860609514 >> 172.21.15.45:0/2860609514 conn(0x7f9af406d1a0 msgr2=0x7f9af40d48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.900+0000 7f9af3e00640 1 -- 172.21.15.45:0/2860609514 shutdown_connections 2024-04-16T16:13:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:23.900+0000 7f9af3e00640 1 -- 172.21.15.45:0/2860609514 wait complete. 2024-04-16T16:13:23.920 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-16T16:13:24.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.128+0000 7fb113200640 1 Processor -- start 2024-04-16T16:13:24.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.128+0000 7fb113200640 1 -- start start 2024-04-16T16:13:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.130+0000 7fb113200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb10c15c7c0 0x7fb10c072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.130+0000 7fb113200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb10c0b4930 0x7fb10c0d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.131+0000 7fb113200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb10c0d1310 con 0x7fb10c0d0f10 2024-04-16T16:13:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.131+0000 7fb113200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb10c0b52e0 con 0x7fb10c15c7c0 2024-04-16T16:13:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.131+0000 7fb113200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb10c0d0260 con 0x7fb10c0b4930 2024-04-16T16:13:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.131+0000 7fb10be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb10c15c7c0 0x7fb10c072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.131+0000 7fb10be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb10c15c7c0 0x7fb10c072e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49558/0 (socket says 172.21.15.45:49558) 2024-04-16T16:13:24.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.131+0000 7fb10be00640 1 -- 172.21.15.45:0/1603441322 learned_addr learned my addr 172.21.15.45:0/1603441322 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:24.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.131+0000 7fb10be00640 1 -- 172.21.15.45:0/1603441322 >> v2:172.21.15.103:3300/0 conn(0x7fb10c0b4930 msgr2=0x7fb10c0d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:24.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.131+0000 7fb103e00640 1 --2- 172.21.15.45:0/1603441322 >> v2:172.21.15.103:3300/0 conn(0x7fb10c0b4930 0x7fb10c0d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:24.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.132+0000 7fb10b400640 1 -- 172.21.15.45:0/1603441322 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1264888888 0 0) 0x7fb10c0d1310 con 0x7fb10c0d0f10 2024-04-16T16:13:24.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.132+0000 7fb10be00640 1 --2- 172.21.15.45:0/1603441322 >> v2:172.21.15.103:3300/0 conn(0x7fb10c0b4930 0x7fb10c0d09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:24.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.132+0000 7fb10be00640 1 -- 172.21.15.45:0/1603441322 >> v1:172.21.15.45:6789/0 conn(0x7fb10c0d0f10 legacy=0x7fb10c06f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:24.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.132+0000 7fb10be00640 1 -- 172.21.15.45:0/1603441322 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10c0b5530 con 0x7fb10c15c7c0 2024-04-16T16:13:24.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.132+0000 7fb103e00640 1 --2- 172.21.15.45:0/1603441322 >> v2:172.21.15.103:3300/0 conn(0x7fb10c0b4930 0x7fb10c0d09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:24.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.133+0000 7fb10be00640 1 --2- 172.21.15.45:0/1603441322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb10c15c7c0 0x7fb10c072e10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0b3900 tx=0x7fb0fc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2a375c288bda00 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:24.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.133+0000 7fb10b400640 1 -- 172.21.15.45:0/1603441322 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc02e070 con 0x7fb10c15c7c0 2024-04-16T16:13:24.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.133+0000 7fb10b400640 1 -- 172.21.15.45:0/1603441322 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0fc037070 con 0x7fb10c15c7c0 2024-04-16T16:13:24.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.133+0000 7fb10b400640 1 -- 172.21.15.45:0/1603441322 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc031a00 con 0x7fb10c15c7c0 2024-04-16T16:13:24.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.134+0000 7fb113200640 1 -- 172.21.15.45:0/1603441322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb10c15c7c0 msgr2=0x7fb10c072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:24.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.134+0000 7fb113200640 1 --2- 172.21.15.45:0/1603441322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb10c15c7c0 0x7fb10c072e10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0b3900 tx=0x7fb0fc02b880 comp rx=0 tx=0).stop 2024-04-16T16:13:24.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.134+0000 7fb113200640 1 -- 172.21.15.45:0/1603441322 shutdown_connections 2024-04-16T16:13:24.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.134+0000 7fb113200640 1 --2- 172.21.15.45:0/1603441322 >> v2:172.21.15.103:3300/0 conn(0x7fb10c0b4930 0x7fb10c0d09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:24.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.134+0000 7fb113200640 1 --2- 172.21.15.45:0/1603441322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb10c15c7c0 0x7fb10c072e10 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:24.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.134+0000 7fb113200640 1 -- 172.21.15.45:0/1603441322 >> 172.21.15.45:0/1603441322 conn(0x7fb10c0b3970 msgr2=0x7fb10c0b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:24.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.134+0000 7fb113200640 1 -- 172.21.15.45:0/1603441322 shutdown_connections 2024-04-16T16:13:24.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.134+0000 7fb113200640 1 -- 172.21.15.45:0/1603441322 wait complete. 2024-04-16T16:13:24.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.136+0000 7fb113200640 1 Processor -- start 2024-04-16T16:13:24.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.136+0000 7fb113200640 1 -- start start 2024-04-16T16:13:24.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.137+0000 7fb113200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10c0b4930 0x7fb10c1f8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:24.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.137+0000 7fb113200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb10c0d0f10 0x7fb10c0adb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:24.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.137+0000 7fb113200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c15c7c0 0x7fb10c1e6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:24.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.137+0000 7fb113200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c0d04b0 con 0x7fb10c0d0f10 2024-04-16T16:13:24.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.137+0000 7fb113200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c073350 con 0x7fb10c15c7c0 2024-04-16T16:13:24.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.137+0000 7fb113200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb10c06f870 con 0x7fb10c0b4930 2024-04-16T16:13:24.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.137+0000 7fb103e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb10c0d0f10 0x7fb10c0adb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:24.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.137+0000 7fb103e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb10c0d0f10 0x7fb10c0adb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49574/0 (socket says 172.21.15.45:49574) 2024-04-16T16:13:24.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.137+0000 7fb103e00640 1 -- 172.21.15.45:0/423071653 learned_addr learned my addr 172.21.15.45:0/423071653 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:24.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.137+0000 7fb111200640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c15c7c0 0x7fb10c1e6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:24.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.137+0000 7fb103e00640 1 -- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10c0b4930 msgr2=0x7fb10c1f8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:24.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.137+0000 7fb10be00640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10c0b4930 0x7fb10c1f8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:24.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.138+0000 7fb103e00640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10c0b4930 0x7fb10c1f8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:24.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.138+0000 7fb103e00640 1 -- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c15c7c0 msgr2=0x7fb10c1e6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:24.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.138+0000 7fb103e00640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c15c7c0 0x7fb10c1e6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:24.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.138+0000 7fb103e00640 1 -- 172.21.15.45:0/423071653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10c0aef00 con 0x7fb10c0d0f10 2024-04-16T16:13:24.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.138+0000 7fb111200640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c15c7c0 0x7fb10c1e6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:24.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.138+0000 7fb10be00640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10c0b4930 0x7fb10c1f8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:24.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.138+0000 7fb103e00640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb10c0d0f10 0x7fb10c0adb70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400c7f0 tx=0x7fb0f400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:24.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.138+0000 7fb108c00640 1 -- 172.21.15.45:0/423071653 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4015070 con 0x7fb10c0d0f10 2024-04-16T16:13:24.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.138+0000 7fb108c00640 1 -- 172.21.15.45:0/423071653 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f4007a30 con 0x7fb10c0d0f10 2024-04-16T16:13:24.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.138+0000 7fb108c00640 1 -- 172.21.15.45:0/423071653 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4007e00 con 0x7fb10c0d0f10 2024-04-16T16:13:24.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.138+0000 7fb113200640 1 -- 172.21.15.45:0/423071653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb10c0ae0b0 con 0x7fb10c0d0f10 2024-04-16T16:13:24.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.138+0000 7fb113200640 1 -- 172.21.15.45:0/423071653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb10c0ae660 con 0x7fb10c0d0f10 2024-04-16T16:13:24.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.140+0000 7fb113200640 1 -- 172.21.15.45:0/423071653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb10c075780 con 0x7fb10c0d0f10 2024-04-16T16:13:24.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.141+0000 7fb108c00640 1 -- 172.21.15.45:0/423071653 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb0f4018070 con 0x7fb10c0d0f10 2024-04-16T16:13:24.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.141+0000 7fb108c00640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0e806c8f0 0x7fb0e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:24.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.142+0000 7fb108c00640 1 -- 172.21.15.45:0/423071653 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7fb0f408b8e0 con 0x7fb10c0d0f10 2024-04-16T16:13:24.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.142+0000 7fb10be00640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0e806c8f0 0x7fb0e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:24.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.143+0000 7fb10be00640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0e806c8f0 0x7fb0e806edb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc036000 tx=0x7fb0fc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:24.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.147+0000 7fb108c00640 1 -- 172.21.15.45:0/423071653 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb0f4059ef0 con 0x7fb10c0d0f10 2024-04-16T16:13:24.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.355+0000 7fb113200640 1 -- 172.21.15.45:0/423071653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb10c075420 con 0x7fb10c0d0f10 2024-04-16T16:13:24.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.874+0000 7fb108c00640 1 -- 172.21.15.45:0/423071653 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7fb0f40600c0 con 0x7fb10c0d0f10 2024-04-16T16:13:24.873 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2024-04-16T16:13:24.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.878+0000 7fb113200640 1 -- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0e806c8f0 msgr2=0x7fb0e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:24.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.878+0000 7fb113200640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0e806c8f0 0x7fb0e806edb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc036000 tx=0x7fb0fc008030 comp rx=0 tx=0).stop 2024-04-16T16:13:24.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.878+0000 7fb113200640 1 -- 172.21.15.45:0/423071653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb10c0d0f10 msgr2=0x7fb10c0adb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:24.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.878+0000 7fb113200640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb10c0d0f10 0x7fb10c0adb70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400c7f0 tx=0x7fb0f400cbf0 comp rx=0 tx=0).stop 2024-04-16T16:13:24.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.878+0000 7fb113200640 1 -- 172.21.15.45:0/423071653 shutdown_connections 2024-04-16T16:13:24.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.878+0000 7fb113200640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0e806c8f0 0x7fb0e806edb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:24.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.878+0000 7fb113200640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c15c7c0 0x7fb10c1e6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:24.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.878+0000 7fb113200640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb10c0d0f10 0x7fb10c0adb70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:24.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.878+0000 7fb113200640 1 --2- 172.21.15.45:0/423071653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10c0b4930 0x7fb10c1f8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:24.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.878+0000 7fb113200640 1 -- 172.21.15.45:0/423071653 >> 172.21.15.45:0/423071653 conn(0x7fb10c0b3970 msgr2=0x7fb10c0b2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:24.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.880+0000 7fb113200640 1 -- 172.21.15.45:0/423071653 shutdown_connections 2024-04-16T16:13:24.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:24.881+0000 7fb113200640 1 -- 172.21.15.45:0/423071653 wait complete. 2024-04-16T16:13:24.915 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-16T16:13:25.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.105+0000 7f3c22600640 1 Processor -- start 2024-04-16T16:13:25.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.105+0000 7f3c22600640 1 -- start start 2024-04-16T16:13:25.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.106+0000 7f3c22600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3c1c15ea70 0x7f3c1c0d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:25.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.106+0000 7f3c22600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c1c0d4370 0x7f3c1c0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:25.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.106+0000 7f3c22600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c1c15ee50 con 0x7f3c1c0cfa70 2024-04-16T16:13:25.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.106+0000 7f3c22600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3c1c0d1a10 con 0x7f3c1c15ea70 2024-04-16T16:13:25.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.106+0000 7f3c22600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c1c0b6480 con 0x7f3c1c0d4370 2024-04-16T16:13:25.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.106+0000 7f3c21c00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3c1c0cfa70 0x7f3c1c0d2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40272/0 (socket says 172.21.15.45:40272) 2024-04-16T16:13:25.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.106+0000 7f3c21c00640 1 -- 172.21.15.45:0/3932352233 learned_addr learned my addr 172.21.15.45:0/3932352233 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:25.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c21200640 1 --2- 172.21.15.45:0/3932352233 >> v2:172.21.15.103:3300/0 conn(0x7f3c1c15ea70 0x7f3c1c0d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:25.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1be00640 1 --2- 172.21.15.45:0/3932352233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c1c0d4370 0x7f3c1c0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:25.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 -- 172.21.15.45:0/3932352233 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4152725423 0 0) 0x7f3c1c15ee50 con 0x7f3c1c0cfa70 2024-04-16T16:13:25.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 -- 172.21.15.45:0/3932352233 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c00003520 con 0x7f3c1c0cfa70 2024-04-16T16:13:25.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 -- 172.21.15.45:0/3932352233 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4180832167 0 0) 0x7f3c00003520 con 0x7f3c1c0cfa70 2024-04-16T16:13:25.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 -- 172.21.15.45:0/3932352233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c1c0d4370 msgr2=0x7f3c1c0cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:25.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 --2- 172.21.15.45:0/3932352233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c1c0d4370 0x7f3c1c0cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:25.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 -- 172.21.15.45:0/3932352233 >> v2:172.21.15.103:3300/0 conn(0x7f3c1c15ea70 msgr2=0x7f3c1c0d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:25.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 --2- 172.21.15.45:0/3932352233 >> v2:172.21.15.103:3300/0 conn(0x7f3c1c15ea70 0x7f3c1c0d11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:25.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 -- 172.21.15.45:0/3932352233 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1c0b6760 con 0x7f3c1c0cfa70 2024-04-16T16:13:25.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 -- 172.21.15.45:0/3932352233 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f3c14001670 con 0x7f3c1c0cfa70 2024-04-16T16:13:25.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 -- 172.21.15.45:0/3932352233 >> v1:172.21.15.45:6789/0 conn(0x7f3c1c0cfa70 legacy=0x7f3c1c0d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:25.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 --2- 172.21.15.45:0/3932352233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c000039b0 0x7f3c00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:25.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 --2- 172.21.15.45:0/3932352233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c1c0d4370 0x7f3c00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:25.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 --2- 172.21.15.45:0/3932352233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c00004d40 0x7f3c000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:25.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 -- 172.21.15.45:0/3932352233 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1c0d1a10 con 0x7f3c000039b0 2024-04-16T16:13:25.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 -- 172.21.15.45:0/3932352233 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1c0b6480 con 0x7f3c00004d40 2024-04-16T16:13:25.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c1b400640 1 -- 172.21.15.45:0/3932352233 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c0000b9c0 con 0x7f3c1c0d4370 2024-04-16T16:13:25.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.107+0000 7f3c21200640 1 --2- 172.21.15.45:0/3932352233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c000039b0 0x7f3c00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:25.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.108+0000 7f3c1be00640 1 --2- 172.21.15.45:0/3932352233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c1c0d4370 0x7f3c00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:25.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.108+0000 7f3c21200640 1 -- 172.21.15.45:0/3932352233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c1c0d4370 msgr2=0x7f3c00004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:25.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.108+0000 7f3c21200640 1 --2- 172.21.15.45:0/3932352233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c1c0d4370 0x7f3c00004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:25.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.108+0000 7f3c21200640 1 -- 172.21.15.45:0/3932352233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c00004d40 msgr2=0x7f3c000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:25.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.108+0000 7f3c21c00640 1 --2- 172.21.15.45:0/3932352233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c00004d40 0x7f3c000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:25.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.108+0000 7f3c21200640 1 --2- 172.21.15.45:0/3932352233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c00004d40 0x7f3c000051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:25.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.108+0000 7f3c21200640 1 -- 172.21.15.45:0/3932352233 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c00003520 con 0x7f3c000039b0 2024-04-16T16:13:25.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.109+0000 7f3c21200640 1 --2- 172.21.15.45:0/3932352233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c000039b0 0x7f3c00003dd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c00e170 tx=0x7f3c00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eda3a0737049b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:25.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.109+0000 7f3c1b400640 1 -- 172.21.15.45:0/3932352233 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c039070 con 0x7f3c000039b0 2024-04-16T16:13:25.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.109+0000 7f3c1b400640 1 -- 172.21.15.45:0/3932352233 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c0c02cc20 con 0x7f3c000039b0 2024-04-16T16:13:25.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.109+0000 7f3c1b400640 1 -- 172.21.15.45:0/3932352233 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c009c10 con 0x7f3c000039b0 2024-04-16T16:13:25.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.110+0000 7f3c22600640 1 -- 172.21.15.45:0/3932352233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c000039b0 msgr2=0x7f3c00003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:25.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.110+0000 7f3c22600640 1 --2- 172.21.15.45:0/3932352233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c000039b0 0x7f3c00003dd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c00e170 tx=0x7f3c00005830 comp rx=0 tx=0).stop 2024-04-16T16:13:25.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.110+0000 7f3c21c00640 1 -- 172.21.15.45:0/3932352233 reap_dead start 2024-04-16T16:13:25.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.110+0000 7f3c22600640 1 -- 172.21.15.45:0/3932352233 shutdown_connections 2024-04-16T16:13:25.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.110+0000 7f3c22600640 1 -- 172.21.15.45:0/3932352233 >> 172.21.15.45:0/3932352233 conn(0x7f3c1c06d3b0 msgr2=0x7f3c1c073060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:25.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.110+0000 7f3c22600640 1 -- 172.21.15.45:0/3932352233 shutdown_connections 2024-04-16T16:13:25.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.110+0000 7f3c22600640 1 -- 172.21.15.45:0/3932352233 wait complete. 2024-04-16T16:13:25.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.112+0000 7f3c22600640 1 Processor -- start 2024-04-16T16:13:25.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.112+0000 7f3c22600640 1 -- start start 2024-04-16T16:13:25.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.113+0000 7f3c22600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c000039b0 0x7f3c1c0cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:25.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.113+0000 7f3c22600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c1c0c8b60 0x7f3c1c0c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:25.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.113+0000 7f3c22600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c1c153390 0x7f3c1c153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.113+0000 7f3c22600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1c0b6f60 con 0x7f3c000039b0 2024-04-16T16:13:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.113+0000 7f3c22600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1c0d16e0 con 0x7f3c1c0c8b60 2024-04-16T16:13:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.113+0000 7f3c22600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c1c15fd50 con 0x7f3c1c153390 2024-04-16T16:13:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.114+0000 7f3c21200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c000039b0 0x7f3c1c0cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.114+0000 7f3c21200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c000039b0 0x7f3c1c0cacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49586/0 (socket says 172.21.15.45:49586) 2024-04-16T16:13:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.114+0000 7f3c21200640 1 -- 172.21.15.45:0/3674529467 learned_addr learned my addr 172.21.15.45:0/3674529467 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.114+0000 7f3c21c00640 1 --2- 172.21.15.45:0/3674529467 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c1c153390 0x7f3c1c153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.114+0000 7f3c1be00640 1 --2- 172.21.15.45:0/3674529467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c1c0c8b60 0x7f3c1c0c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.114+0000 7f3c21200640 1 -- 172.21.15.45:0/3674529467 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c1c153390 msgr2=0x7f3c1c153810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.114+0000 7f3c21200640 1 --2- 172.21.15.45:0/3674529467 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c1c153390 0x7f3c1c153810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.114+0000 7f3c21200640 1 -- 172.21.15.45:0/3674529467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c1c0c8b60 msgr2=0x7f3c1c0c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.114+0000 7f3c21200640 1 --2- 172.21.15.45:0/3674529467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c1c0c8b60 0x7f3c1c0c8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.114+0000 7f3c21200640 1 -- 172.21.15.45:0/3674529467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1c153d50 con 0x7f3c000039b0 2024-04-16T16:13:25.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.114+0000 7f3c1be00640 1 --2- 172.21.15.45:0/3674529467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c1c0c8b60 0x7f3c1c0c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:25.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.115+0000 7f3c21200640 1 --2- 172.21.15.45:0/3674529467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c000039b0 0x7f3c1c0cacf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009f90 tx=0x7f3c0c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:25.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.115+0000 7f3c18c00640 1 -- 172.21.15.45:0/3674529467 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c044020 con 0x7f3c000039b0 2024-04-16T16:13:25.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.115+0000 7f3c18c00640 1 -- 172.21.15.45:0/3674529467 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c0c03f470 con 0x7f3c000039b0 2024-04-16T16:13:25.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.115+0000 7f3c18c00640 1 -- 172.21.15.45:0/3674529467 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c039040 con 0x7f3c000039b0 2024-04-16T16:13:25.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.115+0000 7f3c22600640 1 -- 172.21.15.45:0/3674529467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c1c0cdac0 con 0x7f3c000039b0 2024-04-16T16:13:25.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.115+0000 7f3c22600640 1 -- 172.21.15.45:0/3674529467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c1c0ce080 con 0x7f3c000039b0 2024-04-16T16:13:25.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.116+0000 7f3c22600640 1 -- 172.21.15.45:0/3674529467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c1c075780 con 0x7f3c000039b0 2024-04-16T16:13:25.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.117+0000 7f3c18c00640 1 -- 172.21.15.45:0/3674529467 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3c0c005d10 con 0x7f3c000039b0 2024-04-16T16:13:25.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.118+0000 7f3c18c00640 1 --2- 172.21.15.45:0/3674529467 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3bf806c8f0 0x7f3bf806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:25.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.118+0000 7f3c18c00640 1 -- 172.21.15.45:0/3674529467 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7f3c0c0ae410 con 0x7f3c000039b0 2024-04-16T16:13:25.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.119+0000 7f3c1be00640 1 --2- 172.21.15.45:0/3674529467 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3bf806c8f0 0x7f3bf806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:25.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.120+0000 7f3c1be00640 1 --2- 172.21.15.45:0/3674529467 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3bf806c8f0 0x7f3bf806edb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c157e20 tx=0x7f3c04008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:25.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.123+0000 7f3c18c00640 1 -- 172.21.15.45:0/3674529467 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3c0c04e070 con 0x7f3c000039b0 2024-04-16T16:13:25.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.368+0000 7f3c22600640 1 -- 172.21.15.45:0/3674529467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f3c1c0d2500 con 0x7f3c000039b0 2024-04-16T16:13:25.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.920+0000 7f3c18c00640 1 -- 172.21.15.45:0/3674529467 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f3c1c153d50 con 0x7f3c000039b0 2024-04-16T16:13:25.920 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 2 and data pool 3 2024-04-16T16:13:25.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.933+0000 7f3c22600640 1 -- 172.21.15.45:0/3674529467 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3bf806c8f0 msgr2=0x7f3bf806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:25.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.933+0000 7f3c22600640 1 --2- 172.21.15.45:0/3674529467 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3bf806c8f0 0x7f3bf806edb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c157e20 tx=0x7f3c04008040 comp rx=0 tx=0).stop 2024-04-16T16:13:25.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.933+0000 7f3c22600640 1 -- 172.21.15.45:0/3674529467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c000039b0 msgr2=0x7f3c1c0cacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:25.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.933+0000 7f3c22600640 1 --2- 172.21.15.45:0/3674529467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c000039b0 0x7f3c1c0cacf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009f90 tx=0x7f3c0c009750 comp rx=0 tx=0).stop 2024-04-16T16:13:25.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.934+0000 7f3c22600640 1 -- 172.21.15.45:0/3674529467 shutdown_connections 2024-04-16T16:13:25.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.934+0000 7f3c22600640 1 --2- 172.21.15.45:0/3674529467 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3bf806c8f0 0x7f3bf806edb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:25.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.934+0000 7f3c22600640 1 --2- 172.21.15.45:0/3674529467 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c1c153390 0x7f3c1c153810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:25.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.934+0000 7f3c22600640 1 --2- 172.21.15.45:0/3674529467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c1c0c8b60 0x7f3c1c0c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:25.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.934+0000 7f3c22600640 1 --2- 172.21.15.45:0/3674529467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c000039b0 0x7f3c1c0cacf0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:25.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.934+0000 7f3c22600640 1 -- 172.21.15.45:0/3674529467 >> 172.21.15.45:0/3674529467 conn(0x7f3c1c06d3b0 msgr2=0x7f3c1c0b8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:25.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.935+0000 7f3c22600640 1 -- 172.21.15.45:0/3674529467 shutdown_connections 2024-04-16T16:13:25.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:25.935+0000 7f3c22600640 1 -- 172.21.15.45:0/3674529467 wait complete. 2024-04-16T16:13:25.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:13:26.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.141+0000 7f6fa2c00640 1 Processor -- start 2024-04-16T16:13:26.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.141+0000 7f6fa2c00640 1 -- start start 2024-04-16T16:13:26.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.143+0000 7f6fa2c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6f9c15e8f0 0x7f6f9c0cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.143+0000 7f6fa2c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f9c1632a0 con 0x7f6f9c0cf080 2024-04-16T16:13:26.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.143+0000 7f6fa2c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f9c15ecd0 con 0x7f6f9c0ce9a0 2024-04-16T16:13:26.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.143+0000 7f6fa2c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6f9c0ce2e0 con 0x7f6f9c15e8f0 2024-04-16T16:13:26.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.143+0000 7f6fa2200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6f9c0cf080 0x7f6f9c162b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40276/0 (socket says 172.21.15.45:40276) 2024-04-16T16:13:26.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.143+0000 7f6fa2200640 1 -- 172.21.15.45:0/3047890780 learned_addr learned my addr 172.21.15.45:0/3047890780 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:26.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6fa1800640 1 --2- 172.21.15.45:0/3047890780 >> v2:172.21.15.103:3300/0 conn(0x7f6f9c15e8f0 0x7f6f9c0cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:26.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6f9be00640 1 -- 172.21.15.45:0/3047890780 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 516767636 0 0) 0x7f6f9c1632a0 con 0x7f6f9c0cf080 2024-04-16T16:13:26.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6f9be00640 1 -- 172.21.15.45:0/3047890780 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f84003520 con 0x7f6f9c0cf080 2024-04-16T16:13:26.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6f9be00640 1 -- 172.21.15.45:0/3047890780 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2775986521 0 0) 0x7f6f84003520 con 0x7f6f9c0cf080 2024-04-16T16:13:26.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6f9be00640 1 -- 172.21.15.45:0/3047890780 >> v2:172.21.15.103:3300/0 conn(0x7f6f9c15e8f0 msgr2=0x7f6f9c0cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:26.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6f9be00640 1 --2- 172.21.15.45:0/3047890780 >> v2:172.21.15.103:3300/0 conn(0x7f6f9c15e8f0 0x7f6f9c0cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6f9be00640 1 -- 172.21.15.45:0/3047890780 >> v1:172.21.15.103:6789/0 conn(0x7f6f9c0ce9a0 legacy=0x7f6f9c162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:26.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6f9be00640 1 -- 172.21.15.45:0/3047890780 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9c0cd6d0 con 0x7f6f9c0cf080 2024-04-16T16:13:26.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6f9be00640 1 -- 172.21.15.45:0/3047890780 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f6f940019f0 con 0x7f6f9c0cf080 2024-04-16T16:13:26.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6f9be00640 1 -- 172.21.15.45:0/3047890780 >> v1:172.21.15.45:6789/0 conn(0x7f6f9c0cf080 legacy=0x7f6f9c162b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:26.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6f9be00640 1 --2- 172.21.15.45:0/3047890780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f84003a00 0x7f6f84003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6f9be00640 1 --2- 172.21.15.45:0/3047890780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f84004800 0x7f6f84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6f9be00640 1 --2- 172.21.15.45:0/3047890780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f840051c0 0x7f6f8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6f9be00640 1 -- 172.21.15.45:0/3047890780 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c0ce2e0 con 0x7f6f84004800 2024-04-16T16:13:26.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6f9be00640 1 -- 172.21.15.45:0/3047890780 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8400c0c0 con 0x7f6f84003a00 2024-04-16T16:13:26.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.144+0000 7f6f9be00640 1 -- 172.21.15.45:0/3047890780 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6f8400c2c0 con 0x7f6f840051c0 2024-04-16T16:13:26.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.145+0000 7f6fa0e00640 1 --2- 172.21.15.45:0/3047890780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f84004800 0x7f6f84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:26.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.145+0000 7f6fa1800640 1 --2- 172.21.15.45:0/3047890780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f84003a00 0x7f6f84003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:26.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.145+0000 7f6fa2200640 1 --2- 172.21.15.45:0/3047890780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f840051c0 0x7f6f8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:26.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.145+0000 7f6fa0e00640 1 -- 172.21.15.45:0/3047890780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f840051c0 msgr2=0x7f6f8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:26.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.145+0000 7f6fa0e00640 1 --2- 172.21.15.45:0/3047890780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f840051c0 0x7f6f8400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.145+0000 7f6fa0e00640 1 -- 172.21.15.45:0/3047890780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f84003a00 msgr2=0x7f6f84003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:26.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.145+0000 7f6fa0e00640 1 --2- 172.21.15.45:0/3047890780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f84003a00 0x7f6f84003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.145+0000 7f6fa0e00640 1 -- 172.21.15.45:0/3047890780 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f84003520 con 0x7f6f84004800 2024-04-16T16:13:26.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.145+0000 7f6fa2200640 1 -- 172.21.15.45:0/3047890780 reap_dead start 2024-04-16T16:13:26.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.146+0000 7f6fa0e00640 1 --2- 172.21.15.45:0/3047890780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f84004800 0x7f6f84004c80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6f88001720 tx=0x7f6f84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a7529f21a57a194 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:26.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.146+0000 7f6f9be00640 1 -- 172.21.15.45:0/3047890780 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f88001370 con 0x7f6f84004800 2024-04-16T16:13:26.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.146+0000 7f6f9be00640 1 -- 172.21.15.45:0/3047890780 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f880052d0 con 0x7f6f84004800 2024-04-16T16:13:26.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.146+0000 7f6f9be00640 1 -- 172.21.15.45:0/3047890780 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f88003830 con 0x7f6f84004800 2024-04-16T16:13:26.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.147+0000 7f6fa2c00640 1 -- 172.21.15.45:0/3047890780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f84004800 msgr2=0x7f6f84004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:26.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.147+0000 7f6fa2c00640 1 --2- 172.21.15.45:0/3047890780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f84004800 0x7f6f84004c80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6f88001720 tx=0x7f6f84005830 comp rx=0 tx=0).stop 2024-04-16T16:13:26.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.147+0000 7f6fa2c00640 1 -- 172.21.15.45:0/3047890780 shutdown_connections 2024-04-16T16:13:26.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.147+0000 7f6fa2c00640 1 --2- 172.21.15.45:0/3047890780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f84004800 0x7f6f84004c80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.147+0000 7f6fa2c00640 1 -- 172.21.15.45:0/3047890780 >> 172.21.15.45:0/3047890780 conn(0x7f6f9c06b930 msgr2=0x7f6f9c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:26.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.147+0000 7f6fa2c00640 1 -- 172.21.15.45:0/3047890780 shutdown_connections 2024-04-16T16:13:26.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.147+0000 7f6fa2c00640 1 -- 172.21.15.45:0/3047890780 wait complete. 2024-04-16T16:13:26.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.149+0000 7f6fa2c00640 1 Processor -- start 2024-04-16T16:13:26.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.149+0000 7f6fa2c00640 1 -- start start 2024-04-16T16:13:26.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.150+0000 7f6fa2c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f84004800 0x7f6f9c144090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.150+0000 7f6fa2c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f9c1f4ed0 0x7f6f9c142000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.150+0000 7f6fa2c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f9c141660 0x7f6f9c0b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.150+0000 7f6fa2c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c0cdf70 con 0x7f6f9c141660 2024-04-16T16:13:26.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.150+0000 7f6fa2c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c0cdc90 con 0x7f6f9c1f4ed0 2024-04-16T16:13:26.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.150+0000 7f6fa2c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6f9c0cd360 con 0x7f6f84004800 2024-04-16T16:13:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.150+0000 7f6fa2200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f9c141660 0x7f6f9c0b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.150+0000 7f6fa2200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f9c141660 0x7f6f9c0b3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49592/0 (socket says 172.21.15.45:49592) 2024-04-16T16:13:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.150+0000 7f6fa2200640 1 -- 172.21.15.45:0/2868781118 learned_addr learned my addr 172.21.15.45:0/2868781118 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.150+0000 7f6fa1800640 1 --2- 172.21.15.45:0/2868781118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f84004800 0x7f6f9c144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.150+0000 7f6fa0e00640 1 --2- 172.21.15.45:0/2868781118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f9c1f4ed0 0x7f6f9c142000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.150+0000 7f6fa2200640 1 -- 172.21.15.45:0/2868781118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f84004800 msgr2=0x7f6f9c144090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.150+0000 7f6fa2200640 1 --2- 172.21.15.45:0/2868781118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f84004800 0x7f6f9c144090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.151+0000 7f6fa2200640 1 -- 172.21.15.45:0/2868781118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f9c1f4ed0 msgr2=0x7f6f9c142000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.151+0000 7f6fa2200640 1 --2- 172.21.15.45:0/2868781118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f9c1f4ed0 0x7f6f9c142000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.151+0000 7f6fa2200640 1 -- 172.21.15.45:0/2868781118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9c142540 con 0x7f6f9c141660 2024-04-16T16:13:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.151+0000 7f6fa2200640 1 --2- 172.21.15.45:0/2868781118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f9c141660 0x7f6f9c0b3b80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c070030 tx=0x7f6f9400b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.151+0000 7f6f99600640 1 -- 172.21.15.45:0/2868781118 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9400b8d0 con 0x7f6f9c141660 2024-04-16T16:13:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.151+0000 7f6f99600640 1 -- 172.21.15.45:0/2868781118 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f9400ba70 con 0x7f6f9c141660 2024-04-16T16:13:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.151+0000 7f6f99600640 1 -- 172.21.15.45:0/2868781118 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9400be60 con 0x7f6f9c141660 2024-04-16T16:13:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.151+0000 7f6fa2c00640 1 -- 172.21.15.45:0/2868781118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f9c1f8900 con 0x7f6f9c141660 2024-04-16T16:13:26.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.151+0000 7f6fa2c00640 1 -- 172.21.15.45:0/2868781118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f9c144950 con 0x7f6f9c141660 2024-04-16T16:13:26.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.152+0000 7f6fa2c00640 1 -- 172.21.15.45:0/2868781118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f9c075780 con 0x7f6f9c141660 2024-04-16T16:13:26.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.153+0000 7f6f99600640 1 -- 172.21.15.45:0/2868781118 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6f9401b020 con 0x7f6f9c141660 2024-04-16T16:13:26.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.154+0000 7f6f99600640 1 --2- 172.21.15.45:0/2868781118 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6f7c06c920 0x7f6f7c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.154+0000 7f6f99600640 1 -- 172.21.15.45:0/2868781118 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6f9408bd60 con 0x7f6f9c141660 2024-04-16T16:13:26.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.155+0000 7f6fa1800640 1 --2- 172.21.15.45:0/2868781118 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6f7c06c920 0x7f6f7c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:26.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.156+0000 7f6fa1800640 1 --2- 172.21.15.45:0/2868781118 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6f7c06c920 0x7f6f7c06ede0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0b4bd0 tx=0x7f6f90008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:26.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.159+0000 7f6f99600640 1 -- 172.21.15.45:0/2868781118 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6f9405a0f0 con 0x7f6f9c141660 2024-04-16T16:13:26.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.358+0000 7f6fa2c00640 1 -- 172.21.15.45:0/2868781118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f9c0b5b90 con 0x7f6f9c141660 2024-04-16T16:13:26.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.361+0000 7f6f99600640 1 -- 172.21.15.45:0/2868781118 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16618 (secure 0 0 0) 0x7f6f940599f0 con 0x7f6f9c141660 2024-04-16T16:13:26.362 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:26.362 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":20,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:25.369922+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:13:26.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.364+0000 7f6fa2c00640 1 -- 172.21.15.45:0/2868781118 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6f7c06c920 msgr2=0x7f6f7c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:26.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.364+0000 7f6fa2c00640 1 --2- 172.21.15.45:0/2868781118 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6f7c06c920 0x7f6f7c06ede0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0b4bd0 tx=0x7f6f90008040 comp rx=0 tx=0).stop 2024-04-16T16:13:26.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.364+0000 7f6fa2c00640 1 -- 172.21.15.45:0/2868781118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f9c141660 msgr2=0x7f6f9c0b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.364+0000 7f6fa2c00640 1 --2- 172.21.15.45:0/2868781118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f9c141660 0x7f6f9c0b3b80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c070030 tx=0x7f6f9400b0d0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.364+0000 7f6fa2c00640 1 -- 172.21.15.45:0/2868781118 shutdown_connections 2024-04-16T16:13:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.364+0000 7f6fa2c00640 1 --2- 172.21.15.45:0/2868781118 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6f7c06c920 0x7f6f7c06ede0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.364+0000 7f6fa2c00640 1 --2- 172.21.15.45:0/2868781118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f9c141660 0x7f6f9c0b3b80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.364+0000 7f6fa2c00640 1 --2- 172.21.15.45:0/2868781118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f9c1f4ed0 0x7f6f9c142000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.364+0000 7f6fa2c00640 1 --2- 172.21.15.45:0/2868781118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f84004800 0x7f6f9c144090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.364+0000 7f6fa2c00640 1 -- 172.21.15.45:0/2868781118 >> 172.21.15.45:0/2868781118 conn(0x7f6f9c06b930 msgr2=0x7f6f9c06eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.364+0000 7f6fa2c00640 1 -- 172.21.15.45:0/2868781118 shutdown_connections 2024-04-16T16:13:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.364+0000 7f6fa2c00640 1 -- 172.21.15.45:0/2868781118 wait complete. 2024-04-16T16:13:26.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:13:26.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.594+0000 7f4947200640 1 Processor -- start 2024-04-16T16:13:26.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.595+0000 7f4947200640 1 -- start start 2024-04-16T16:13:26.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.596+0000 7f4947200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4948071a40 0x7f49480b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.596+0000 7f4947200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f494815cd40 0x7f49480d1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.596+0000 7f4947200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49481604d0 con 0x7f49480b7750 2024-04-16T16:13:26.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.596+0000 7f4947200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f49481607b0 con 0x7f494815cd40 2024-04-16T16:13:26.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.596+0000 7f4947200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4948160a90 con 0x7f4948071a40 2024-04-16T16:13:26.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.597+0000 7f4945400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f49480b7750 0x7f4948071e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40278/0 (socket says 172.21.15.45:40278) 2024-04-16T16:13:26.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.597+0000 7f4945400640 1 -- 172.21.15.45:0/3063305009 learned_addr learned my addr 172.21.15.45:0/3063305009 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:26.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.597+0000 7f4945e00640 1 --2- 172.21.15.45:0/3063305009 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4948071a40 0x7f49480b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:26.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.597+0000 7f4946800640 1 --2- 172.21.15.45:0/3063305009 >> v2:172.21.15.103:3300/0 conn(0x7f494815cd40 0x7f49480d1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:26.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.597+0000 7f4944a00640 1 -- 172.21.15.45:0/3063305009 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 648940063 0 0) 0x7f49481604d0 con 0x7f49480b7750 2024-04-16T16:13:26.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.597+0000 7f4944a00640 1 -- 172.21.15.45:0/3063305009 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f492c003520 con 0x7f49480b7750 2024-04-16T16:13:26.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.597+0000 7f4944a00640 1 -- 172.21.15.45:0/3063305009 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3821698706 0 0) 0x7f492c003520 con 0x7f49480b7750 2024-04-16T16:13:26.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.597+0000 7f4944a00640 1 -- 172.21.15.45:0/3063305009 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4948071a40 msgr2=0x7f49480b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:26.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.597+0000 7f4944a00640 1 --2- 172.21.15.45:0/3063305009 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4948071a40 0x7f49480b56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4944a00640 1 -- 172.21.15.45:0/3063305009 >> v2:172.21.15.103:3300/0 conn(0x7f494815cd40 msgr2=0x7f49480d1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:26.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4944a00640 1 --2- 172.21.15.45:0/3063305009 >> v2:172.21.15.103:3300/0 conn(0x7f494815cd40 0x7f49480d1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4944a00640 1 -- 172.21.15.45:0/3063305009 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4948160d70 con 0x7f49480b7750 2024-04-16T16:13:26.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4944a00640 1 -- 172.21.15.45:0/3063305009 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f4934001670 con 0x7f49480b7750 2024-04-16T16:13:26.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4944a00640 1 -- 172.21.15.45:0/3063305009 >> v1:172.21.15.45:6789/0 conn(0x7f49480b7750 legacy=0x7f4948071e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:26.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4944a00640 1 --2- 172.21.15.45:0/3063305009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f492c0039b0 0x7f492c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4944a00640 1 --2- 172.21.15.45:0/3063305009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f492c004800 0x7f492c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4944a00640 1 --2- 172.21.15.45:0/3063305009 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4948071a40 0x7f492c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4944a00640 1 -- 172.21.15.45:0/3063305009 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f49481607b0 con 0x7f492c004800 2024-04-16T16:13:26.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4944a00640 1 -- 172.21.15.45:0/3063305009 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4948160a90 con 0x7f492c0039b0 2024-04-16T16:13:26.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4944a00640 1 -- 172.21.15.45:0/3063305009 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f492c00b9c0 con 0x7f4948071a40 2024-04-16T16:13:26.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4945400640 1 --2- 172.21.15.45:0/3063305009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f492c004800 0x7f492c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:26.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4945400640 1 -- 172.21.15.45:0/3063305009 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4948071a40 msgr2=0x7f492c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:26.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4945400640 1 --2- 172.21.15.45:0/3063305009 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4948071a40 0x7f492c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4945400640 1 -- 172.21.15.45:0/3063305009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f492c0039b0 msgr2=0x7f492c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:26.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4945400640 1 --2- 172.21.15.45:0/3063305009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f492c0039b0 0x7f492c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.598+0000 7f4945400640 1 -- 172.21.15.45:0/3063305009 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492c003520 con 0x7f492c004800 2024-04-16T16:13:26.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.600+0000 7f4945400640 1 --2- 172.21.15.45:0/3063305009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f492c004800 0x7f492c004c80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f49480b7b70 tx=0x7f492c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6270a7321a4a646 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:26.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.600+0000 7f4944a00640 1 -- 172.21.15.45:0/3063305009 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4934007d50 con 0x7f492c004800 2024-04-16T16:13:26.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.600+0000 7f4944a00640 1 -- 172.21.15.45:0/3063305009 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4934030070 con 0x7f492c004800 2024-04-16T16:13:26.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.600+0000 7f4944a00640 1 -- 172.21.15.45:0/3063305009 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4934005d10 con 0x7f492c004800 2024-04-16T16:13:26.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.600+0000 7f4947200640 1 -- 172.21.15.45:0/3063305009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f492c004800 msgr2=0x7f492c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:26.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.600+0000 7f4947200640 1 --2- 172.21.15.45:0/3063305009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f492c004800 0x7f492c004c80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f49480b7b70 tx=0x7f492c005830 comp rx=0 tx=0).stop 2024-04-16T16:13:26.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.600+0000 7f4946800640 1 -- 172.21.15.45:0/3063305009 reap_dead start 2024-04-16T16:13:26.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.600+0000 7f4947200640 1 -- 172.21.15.45:0/3063305009 shutdown_connections 2024-04-16T16:13:26.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.600+0000 7f4947200640 1 -- 172.21.15.45:0/3063305009 >> 172.21.15.45:0/3063305009 conn(0x7f49480b43a0 msgr2=0x7f49480b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:26.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.601+0000 7f4947200640 1 -- 172.21.15.45:0/3063305009 shutdown_connections 2024-04-16T16:13:26.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.601+0000 7f4947200640 1 -- 172.21.15.45:0/3063305009 wait complete. 2024-04-16T16:13:26.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.602+0000 7f4947200640 1 Processor -- start 2024-04-16T16:13:26.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.603+0000 7f4947200640 1 -- start start 2024-04-16T16:13:26.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.603+0000 7f4947200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f492c004800 0x7f4948159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.603+0000 7f4947200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4948156b50 0x7f494814a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.604+0000 7f4947200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49481f8a20 0x7f49481f8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.604+0000 7f4947200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4948072b20 con 0x7f49481f8a20 2024-04-16T16:13:26.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.604+0000 7f4947200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f494815a670 con 0x7f4948156b50 2024-04-16T16:13:26.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.604+0000 7f4947200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f49480b56a0 con 0x7f492c004800 2024-04-16T16:13:26.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.604+0000 7f4946800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49481f8a20 0x7f49481f8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:26.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.604+0000 7f4946800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49481f8a20 0x7f49481f8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49604/0 (socket says 172.21.15.45:49604) 2024-04-16T16:13:26.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.604+0000 7f4946800640 1 -- 172.21.15.45:0/2710686109 learned_addr learned my addr 172.21.15.45:0/2710686109 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.604+0000 7f4945e00640 1 --2- 172.21.15.45:0/2710686109 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f492c004800 0x7f4948159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.604+0000 7f4945400640 1 --2- 172.21.15.45:0/2710686109 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4948156b50 0x7f494814a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.604+0000 7f4946800640 1 -- 172.21.15.45:0/2710686109 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f492c004800 msgr2=0x7f4948159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.604+0000 7f4946800640 1 --2- 172.21.15.45:0/2710686109 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f492c004800 0x7f4948159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.604+0000 7f4946800640 1 -- 172.21.15.45:0/2710686109 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4948156b50 msgr2=0x7f494814a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.604+0000 7f4946800640 1 --2- 172.21.15.45:0/2710686109 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4948156b50 0x7f494814a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.604+0000 7f4946800640 1 -- 172.21.15.45:0/2710686109 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4948159a00 con 0x7f49481f8a20 2024-04-16T16:13:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.604+0000 7f4946800640 1 --2- 172.21.15.45:0/2710686109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49481f8a20 0x7f49481f8ea0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f494000ede0 tx=0x7f494000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.605+0000 7f493e000640 1 -- 172.21.15.45:0/2710686109 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4940018020 con 0x7f49481f8a20 2024-04-16T16:13:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.605+0000 7f493e000640 1 -- 172.21.15.45:0/2710686109 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f494000bd10 con 0x7f49481f8a20 2024-04-16T16:13:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.605+0000 7f493e000640 1 -- 172.21.15.45:0/2710686109 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4940014640 con 0x7f49481f8a20 2024-04-16T16:13:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.605+0000 7f4947200640 1 -- 172.21.15.45:0/2710686109 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4948159ce0 con 0x7f49481f8a20 2024-04-16T16:13:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.605+0000 7f4947200640 1 -- 172.21.15.45:0/2710686109 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49481f8500 con 0x7f49481f8a20 2024-04-16T16:13:26.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.606+0000 7f4947200640 1 -- 172.21.15.45:0/2710686109 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f494815c5f0 con 0x7f49481f8a20 2024-04-16T16:13:26.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.607+0000 7f493e000640 1 -- 172.21.15.45:0/2710686109 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4940005d10 con 0x7f49481f8a20 2024-04-16T16:13:26.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.608+0000 7f493e000640 1 --2- 172.21.15.45:0/2710686109 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f492806c8f0 0x7f492806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:26.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.608+0000 7f493e000640 1 -- 172.21.15.45:0/2710686109 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f494008c200 con 0x7f49481f8a20 2024-04-16T16:13:26.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.608+0000 7f4945e00640 1 --2- 172.21.15.45:0/2710686109 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f492806c8f0 0x7f492806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:26.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.609+0000 7f4945e00640 1 --2- 172.21.15.45:0/2710686109 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f492806c8f0 0x7f492806edb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4938004770 tx=0x7f4938008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:26.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.612+0000 7f493e000640 1 -- 172.21.15.45:0/2710686109 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4940022030 con 0x7f49481f8a20 2024-04-16T16:13:26.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.821+0000 7f4947200640 1 -- 172.21.15.45:0/2710686109 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49480b4f50 con 0x7f49481f8a20 2024-04-16T16:13:26.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.823+0000 7f493e000640 1 -- 172.21.15.45:0/2710686109 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16618 (secure 0 0 0) 0x7f494005a620 con 0x7f49481f8a20 2024-04-16T16:13:26.823 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:26.824 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":20,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:25.369922+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:13:26.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.825+0000 7f4947200640 1 -- 172.21.15.45:0/2710686109 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f492806c8f0 msgr2=0x7f492806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.825+0000 7f4947200640 1 --2- 172.21.15.45:0/2710686109 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f492806c8f0 0x7f492806edb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4938004770 tx=0x7f4938008040 comp rx=0 tx=0).stop 2024-04-16T16:13:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.825+0000 7f4947200640 1 -- 172.21.15.45:0/2710686109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49481f8a20 msgr2=0x7f49481f8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.825+0000 7f4947200640 1 --2- 172.21.15.45:0/2710686109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49481f8a20 0x7f49481f8ea0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f494000ede0 tx=0x7f494000b480 comp rx=0 tx=0).stop 2024-04-16T16:13:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.825+0000 7f4947200640 1 -- 172.21.15.45:0/2710686109 shutdown_connections 2024-04-16T16:13:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.825+0000 7f4947200640 1 --2- 172.21.15.45:0/2710686109 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f492806c8f0 0x7f492806edb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.825+0000 7f4947200640 1 --2- 172.21.15.45:0/2710686109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49481f8a20 0x7f49481f8ea0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.825+0000 7f4947200640 1 --2- 172.21.15.45:0/2710686109 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4948156b50 0x7f494814a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.825+0000 7f4947200640 1 --2- 172.21.15.45:0/2710686109 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f492c004800 0x7f4948159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.825+0000 7f4947200640 1 -- 172.21.15.45:0/2710686109 >> 172.21.15.45:0/2710686109 conn(0x7f49480b43a0 msgr2=0x7f49480b6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.825+0000 7f4947200640 1 -- 172.21.15.45:0/2710686109 shutdown_connections 2024-04-16T16:13:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:26.826+0000 7f4947200640 1 -- 172.21.15.45:0/2710686109 wait complete. 2024-04-16T16:13:26.845 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-16T16:13:27.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.069+0000 7f8691200640 1 Processor -- start 2024-04-16T16:13:27.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.070+0000 7f8691200640 1 -- start start 2024-04-16T16:13:27.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.071+0000 7f8691200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f868c0d18f0 0x7f868c0ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:27.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.071+0000 7f8691200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f868c0b6310 0x7f868c0d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:27.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.071+0000 7f8691200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f868c0b6810 con 0x7f868c0d1cd0 2024-04-16T16:13:27.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.071+0000 7f8691200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f868c0d29f0 con 0x7f868c0d18f0 2024-04-16T16:13:27.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.071+0000 7f8691200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f868c0b6af0 con 0x7f868c0b6310 2024-04-16T16:13:27.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.072+0000 7f868b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f868c0d18f0 0x7f868c0ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:27.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.072+0000 7f868b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f868c0d18f0 0x7f868c0ba220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49616/0 (socket says 172.21.15.45:49616) 2024-04-16T16:13:27.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.072+0000 7f868b400640 1 -- 172.21.15.45:0/1011484857 learned_addr learned my addr 172.21.15.45:0/1011484857 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:27.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.072+0000 7f868aa00640 1 --2- 172.21.15.45:0/1011484857 >> v2:172.21.15.103:3300/0 conn(0x7f868c0b6310 0x7f868c0d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:27.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.072+0000 7f868b400640 1 -- 172.21.15.45:0/1011484857 >> v2:172.21.15.103:3300/0 conn(0x7f868c0b6310 msgr2=0x7f868c0d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:27.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.072+0000 7f868b400640 1 --2- 172.21.15.45:0/1011484857 >> v2:172.21.15.103:3300/0 conn(0x7f868c0b6310 0x7f868c0d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:27.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.072+0000 7f868b400640 1 -- 172.21.15.45:0/1011484857 >> v1:172.21.15.45:6789/0 conn(0x7f868c0d1cd0 legacy=0x7f868c0d33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:27.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.072+0000 7f868b400640 1 -- 172.21.15.45:0/1011484857 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f868c0b6dd0 con 0x7f868c0d18f0 2024-04-16T16:13:27.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.072+0000 7f868a000640 1 -- 172.21.15.45:0/1011484857 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1974973275 0 0) 0x7f868c0b6810 con 0x7f868c0d1cd0 2024-04-16T16:13:27.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.074+0000 7f868b400640 1 --2- 172.21.15.45:0/1011484857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f868c0d18f0 0x7f868c0ba220 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f868c0d58a0 tx=0x7f867c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6544d3417e01458e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:27.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.074+0000 7f868a000640 1 -- 172.21.15.45:0/1011484857 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c02c9f0 con 0x7f868c0d18f0 2024-04-16T16:13:27.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.074+0000 7f868a000640 1 -- 172.21.15.45:0/1011484857 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f867c02cb90 con 0x7f868c0d18f0 2024-04-16T16:13:27.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.074+0000 7f868a000640 1 -- 172.21.15.45:0/1011484857 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c0343f0 con 0x7f868c0d18f0 2024-04-16T16:13:27.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.075+0000 7f8691200640 1 -- 172.21.15.45:0/1011484857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f868c0d18f0 msgr2=0x7f868c0ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:27.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.075+0000 7f8691200640 1 --2- 172.21.15.45:0/1011484857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f868c0d18f0 0x7f868c0ba220 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f868c0d58a0 tx=0x7f867c02be50 comp rx=0 tx=0).stop 2024-04-16T16:13:27.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.075+0000 7f8691200640 1 -- 172.21.15.45:0/1011484857 shutdown_connections 2024-04-16T16:13:27.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.075+0000 7f8691200640 1 --2- 172.21.15.45:0/1011484857 >> v2:172.21.15.103:3300/0 conn(0x7f868c0b6310 0x7f868c0d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:27.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.075+0000 7f8691200640 1 --2- 172.21.15.45:0/1011484857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f868c0d18f0 0x7f868c0ba220 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:27.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.075+0000 7f8691200640 1 -- 172.21.15.45:0/1011484857 >> 172.21.15.45:0/1011484857 conn(0x7f868c06cf90 msgr2=0x7f868c06d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:27.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.075+0000 7f8691200640 1 -- 172.21.15.45:0/1011484857 shutdown_connections 2024-04-16T16:13:27.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.075+0000 7f8691200640 1 -- 172.21.15.45:0/1011484857 wait complete. 2024-04-16T16:13:27.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.077+0000 7f8691200640 1 Processor -- start 2024-04-16T16:13:27.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.077+0000 7f8691200640 1 -- start start 2024-04-16T16:13:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.078+0000 7f8691200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f868c0b6310 0x7f868c0c9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.078+0000 7f8691200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f868c0d18f0 0x7f868c0cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.078+0000 7f8691200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f868c0d1cd0 0x7f868c155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.078+0000 7f8691200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f868c162dc0 con 0x7f868c0d18f0 2024-04-16T16:13:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.078+0000 7f8691200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f868c0ba760 con 0x7f868c0b6310 2024-04-16T16:13:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.078+0000 7f8691200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f868c162f40 con 0x7f868c0d1cd0 2024-04-16T16:13:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.078+0000 7f868aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f868c0d18f0 0x7f868c0cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.078+0000 7f868aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f868c0d18f0 0x7f868c0cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49620/0 (socket says 172.21.15.45:49620) 2024-04-16T16:13:27.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.078+0000 7f868aa00640 1 -- 172.21.15.45:0/1005954816 learned_addr learned my addr 172.21.15.45:0/1005954816 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:27.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.078+0000 7f868aa00640 1 -- 172.21.15.45:0/1005954816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f868c0d1cd0 msgr2=0x7f868c155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:27.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.078+0000 7f868aa00640 1 --2- 172.21.15.45:0/1005954816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f868c0d1cd0 0x7f868c155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:27.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.078+0000 7f868aa00640 1 -- 172.21.15.45:0/1005954816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f868c0b6310 msgr2=0x7f868c0c9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:27.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.078+0000 7f868aa00640 1 --2- 172.21.15.45:0/1005954816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f868c0b6310 0x7f868c0c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:27.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.078+0000 7f868aa00640 1 -- 172.21.15.45:0/1005954816 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f868c0cb220 con 0x7f868c0d18f0 2024-04-16T16:13:27.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.079+0000 7f868aa00640 1 --2- 172.21.15.45:0/1005954816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f868c0d18f0 0x7f868c0cace0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f867400d830 tx=0x7f867400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:27.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.079+0000 7f8683400640 1 -- 172.21.15.45:0/1005954816 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867400b6c0 con 0x7f868c0d18f0 2024-04-16T16:13:27.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.079+0000 7f8683400640 1 -- 172.21.15.45:0/1005954816 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8674019070 con 0x7f868c0d18f0 2024-04-16T16:13:27.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.079+0000 7f8683400640 1 -- 172.21.15.45:0/1005954816 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867400baa0 con 0x7f868c0d18f0 2024-04-16T16:13:27.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.079+0000 7f8691200640 1 -- 172.21.15.45:0/1005954816 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f868c155990 con 0x7f868c0d18f0 2024-04-16T16:13:27.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.079+0000 7f8691200640 1 -- 172.21.15.45:0/1005954816 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f868c0c8700 con 0x7f868c0d18f0 2024-04-16T16:13:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.081+0000 7f8691200640 1 -- 172.21.15.45:0/1005954816 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f868c075780 con 0x7f868c0d18f0 2024-04-16T16:13:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.082+0000 7f8683400640 1 -- 172.21.15.45:0/1005954816 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f867401e460 con 0x7f868c0d18f0 2024-04-16T16:13:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.082+0000 7f8683400640 1 --2- 172.21.15.45:0/1005954816 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f866806c8f0 0x7f866806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.082+0000 7f8683400640 1 -- 172.21.15.45:0/1005954816 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f867408c280 con 0x7f868c0d18f0 2024-04-16T16:13:27.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.083+0000 7f868b400640 1 --2- 172.21.15.45:0/1005954816 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f866806c8f0 0x7f866806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.084+0000 7f868b400640 1 --2- 172.21.15.45:0/1005954816 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f866806c8f0 0x7f866806edb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f867c035000 tx=0x7f867c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.087+0000 7f8683400640 1 -- 172.21.15.45:0/1005954816 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f867405a6a0 con 0x7f868c0d18f0 2024-04-16T16:13:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.329+0000 7f8691200640 1 -- 172.21.15.45:0/1005954816 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f868c0d25a0 con 0x7f868c0d18f0 2024-04-16T16:13:27.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.941+0000 7f8683400640 1 -- 172.21.15.45:0/1005954816 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f8674059fa0 con 0x7f868c0d18f0 2024-04-16T16:13:27.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.944+0000 7f8691200640 1 -- 172.21.15.45:0/1005954816 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f866806c8f0 msgr2=0x7f866806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:27.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.944+0000 7f8691200640 1 --2- 172.21.15.45:0/1005954816 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f866806c8f0 0x7f866806edb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f867c035000 tx=0x7f867c02e000 comp rx=0 tx=0).stop 2024-04-16T16:13:27.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.944+0000 7f8691200640 1 -- 172.21.15.45:0/1005954816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f868c0d18f0 msgr2=0x7f868c0cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:27.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.944+0000 7f8691200640 1 --2- 172.21.15.45:0/1005954816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f868c0d18f0 0x7f868c0cace0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f867400d830 tx=0x7f867400dcf0 comp rx=0 tx=0).stop 2024-04-16T16:13:27.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.945+0000 7f8691200640 1 -- 172.21.15.45:0/1005954816 shutdown_connections 2024-04-16T16:13:27.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.945+0000 7f8691200640 1 --2- 172.21.15.45:0/1005954816 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f866806c8f0 0x7f866806edb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:27.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.945+0000 7f8691200640 1 --2- 172.21.15.45:0/1005954816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f868c0d1cd0 0x7f868c155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:27.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.945+0000 7f8691200640 1 --2- 172.21.15.45:0/1005954816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f868c0d18f0 0x7f868c0cace0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:27.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.945+0000 7f8691200640 1 --2- 172.21.15.45:0/1005954816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f868c0b6310 0x7f868c0c9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:27.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.945+0000 7f8691200640 1 -- 172.21.15.45:0/1005954816 >> 172.21.15.45:0/1005954816 conn(0x7f868c06cf90 msgr2=0x7f868c0b9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:27.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.945+0000 7f8691200640 1 -- 172.21.15.45:0/1005954816 shutdown_connections 2024-04-16T16:13:27.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:27.945+0000 7f8691200640 1 -- 172.21.15.45:0/1005954816 wait complete. 2024-04-16T16:13:27.971 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-16T16:13:27.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-16T16:13:28.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.195+0000 7fc6c6a00640 1 Processor -- start 2024-04-16T16:13:28.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.196+0000 7fc6c6a00640 1 -- start start 2024-04-16T16:13:28.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.197+0000 7fc6c6a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6c00b6310 0x7fc6c0161db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:28.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.197+0000 7fc6c6a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6c00b6810 con 0x7fc6c00d3e90 2024-04-16T16:13:28.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.197+0000 7fc6c6a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6c00b6af0 con 0x7fc6c00d18f0 2024-04-16T16:13:28.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.197+0000 7fc6c6a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc6c00d29f0 con 0x7fc6c00b6310 2024-04-16T16:13:28.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.197+0000 7fc6c6000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc6c00d3e90 0x7fc6c01622f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40298/0 (socket says 172.21.15.45:40298) 2024-04-16T16:13:28.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.197+0000 7fc6c6000640 1 -- 172.21.15.45:0/1584811032 learned_addr learned my addr 172.21.15.45:0/1584811032 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.198+0000 7fc6c4c00640 1 --2- 172.21.15.45:0/1584811032 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6c00b6310 0x7fc6c0161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.198+0000 7fc6bfe00640 1 -- 172.21.15.45:0/1584811032 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4029323733 0 0) 0x7fc6c00b6810 con 0x7fc6c00d3e90 2024-04-16T16:13:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.198+0000 7fc6bfe00640 1 -- 172.21.15.45:0/1584811032 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6a8003520 con 0x7fc6c00d3e90 2024-04-16T16:13:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.198+0000 7fc6bfe00640 1 -- 172.21.15.45:0/1584811032 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 192269941 0 0) 0x7fc6a8003520 con 0x7fc6c00d3e90 2024-04-16T16:13:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.198+0000 7fc6bfe00640 1 -- 172.21.15.45:0/1584811032 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6c00b6310 msgr2=0x7fc6c0161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.198+0000 7fc6bfe00640 1 --2- 172.21.15.45:0/1584811032 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6c00b6310 0x7fc6c0161db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.198+0000 7fc6bfe00640 1 -- 172.21.15.45:0/1584811032 >> v1:172.21.15.103:6789/0 conn(0x7fc6c00d18f0 legacy=0x7fc6c00ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.198+0000 7fc6bfe00640 1 -- 172.21.15.45:0/1584811032 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c00b6dd0 con 0x7fc6c00d3e90 2024-04-16T16:13:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.198+0000 7fc6bfe00640 1 -- 172.21.15.45:0/1584811032 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fc6b80019f0 con 0x7fc6c00d3e90 2024-04-16T16:13:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.198+0000 7fc6bfe00640 1 -- 172.21.15.45:0/1584811032 >> v1:172.21.15.45:6789/0 conn(0x7fc6c00d3e90 legacy=0x7fc6c01622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.198+0000 7fc6bfe00640 1 --2- 172.21.15.45:0/1584811032 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6c00b6310 0x7fc6a80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.198+0000 7fc6bfe00640 1 --2- 172.21.15.45:0/1584811032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6a8003fb0 0x7fc6a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.198+0000 7fc6bfe00640 1 --2- 172.21.15.45:0/1584811032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6a8004d40 0x7fc6a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.198+0000 7fc6bfe00640 1 -- 172.21.15.45:0/1584811032 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c00d29f0 con 0x7fc6a8004d40 2024-04-16T16:13:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.198+0000 7fc6bfe00640 1 -- 172.21.15.45:0/1584811032 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a800bbb0 con 0x7fc6a8003fb0 2024-04-16T16:13:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.199+0000 7fc6bfe00640 1 -- 172.21.15.45:0/1584811032 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc6a800bdb0 con 0x7fc6c00b6310 2024-04-16T16:13:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.199+0000 7fc6c6000640 1 --2- 172.21.15.45:0/1584811032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6a8004d40 0x7fc6a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.199+0000 7fc6c5600640 1 --2- 172.21.15.45:0/1584811032 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6c00b6310 0x7fc6a80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.199+0000 7fc6c6000640 1 -- 172.21.15.45:0/1584811032 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6c00b6310 msgr2=0x7fc6a80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.199+0000 7fc6c6000640 1 --2- 172.21.15.45:0/1584811032 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6c00b6310 0x7fc6a80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.199+0000 7fc6c6000640 1 -- 172.21.15.45:0/1584811032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6a8003fb0 msgr2=0x7fc6a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.199+0000 7fc6c6000640 1 --2- 172.21.15.45:0/1584811032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6a8003fb0 0x7fc6a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.199+0000 7fc6c6000640 1 -- 172.21.15.45:0/1584811032 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6a8003520 con 0x7fc6a8004d40 2024-04-16T16:13:28.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.200+0000 7fc6c6000640 1 --2- 172.21.15.45:0/1584811032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6a8004d40 0x7fc6a80051c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800a760 tx=0x7fc6a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230b274cd65628db server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:28.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.201+0000 7fc6bfe00640 1 -- 172.21.15.45:0/1584811032 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b800a3b0 con 0x7fc6a8004d40 2024-04-16T16:13:28.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.201+0000 7fc6bfe00640 1 -- 172.21.15.45:0/1584811032 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6b8034650 con 0x7fc6a8004d40 2024-04-16T16:13:28.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.201+0000 7fc6bfe00640 1 -- 172.21.15.45:0/1584811032 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8034a80 con 0x7fc6a8004d40 2024-04-16T16:13:28.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.201+0000 7fc6c6a00640 1 -- 172.21.15.45:0/1584811032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6a8004d40 msgr2=0x7fc6a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:28.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.201+0000 7fc6c6a00640 1 --2- 172.21.15.45:0/1584811032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6a8004d40 0x7fc6a80051c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800a760 tx=0x7fc6a8005830 comp rx=0 tx=0).stop 2024-04-16T16:13:28.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.201+0000 7fc6c6000640 1 -- 172.21.15.45:0/1584811032 reap_dead start 2024-04-16T16:13:28.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.201+0000 7fc6c6a00640 1 -- 172.21.15.45:0/1584811032 shutdown_connections 2024-04-16T16:13:28.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.201+0000 7fc6c6a00640 1 -- 172.21.15.45:0/1584811032 >> 172.21.15.45:0/1584811032 conn(0x7fc6c006cf90 msgr2=0x7fc6c006d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:28.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.201+0000 7fc6c6a00640 1 -- 172.21.15.45:0/1584811032 shutdown_connections 2024-04-16T16:13:28.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.201+0000 7fc6c6a00640 1 -- 172.21.15.45:0/1584811032 wait complete. 2024-04-16T16:13:28.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.203+0000 7fc6c6a00640 1 Processor -- start 2024-04-16T16:13:28.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.203+0000 7fc6c6a00640 1 -- start start 2024-04-16T16:13:28.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.204+0000 7fc6c6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6a8004d40 0x7fc6c00cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:28.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.204+0000 7fc6c6a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6c00b2d10 0x7fc6c00ab260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:28.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.204+0000 7fc6c6a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6c00ab7a0 0x7fc6c00ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:28.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.204+0000 7fc6c6a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c00d1e50 con 0x7fc6a8004d40 2024-04-16T16:13:28.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.204+0000 7fc6c6a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c00d1cd0 con 0x7fc6c00ab7a0 2024-04-16T16:13:28.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.204+0000 7fc6c6a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc6c00d1fd0 con 0x7fc6c00b2d10 2024-04-16T16:13:28.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.204+0000 7fc6c5600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6a8004d40 0x7fc6c00cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:28.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.204+0000 7fc6c5600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6a8004d40 0x7fc6c00cc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49640/0 (socket says 172.21.15.45:49640) 2024-04-16T16:13:28.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.205+0000 7fc6c5600640 1 -- 172.21.15.45:0/302676169 learned_addr learned my addr 172.21.15.45:0/302676169 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:28.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.205+0000 7fc6c4c00640 1 --2- 172.21.15.45:0/302676169 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6c00b2d10 0x7fc6c00ab260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:28.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.205+0000 7fc6c6000640 1 --2- 172.21.15.45:0/302676169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6c00ab7a0 0x7fc6c00ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:28.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.205+0000 7fc6c5600640 1 -- 172.21.15.45:0/302676169 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6c00b2d10 msgr2=0x7fc6c00ab260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:28.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.205+0000 7fc6c5600640 1 --2- 172.21.15.45:0/302676169 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6c00b2d10 0x7fc6c00ab260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:28.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.205+0000 7fc6c5600640 1 -- 172.21.15.45:0/302676169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6c00ab7a0 msgr2=0x7fc6c00ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:28.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.205+0000 7fc6c5600640 1 --2- 172.21.15.45:0/302676169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6c00ab7a0 0x7fc6c00ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:28.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.205+0000 7fc6c5600640 1 -- 172.21.15.45:0/302676169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c014ebf0 con 0x7fc6a8004d40 2024-04-16T16:13:28.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.205+0000 7fc6c6000640 1 --2- 172.21.15.45:0/302676169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6c00ab7a0 0x7fc6c00ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:28.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.205+0000 7fc6c5600640 1 --2- 172.21.15.45:0/302676169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6a8004d40 0x7fc6c00cc130 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400e580 tx=0x7fc6b400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:28.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.205+0000 7fc6bd600640 1 -- 172.21.15.45:0/302676169 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b40052e0 con 0x7fc6a8004d40 2024-04-16T16:13:28.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.206+0000 7fc6bd600640 1 -- 172.21.15.45:0/302676169 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6b4018070 con 0x7fc6a8004d40 2024-04-16T16:13:28.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.206+0000 7fc6c6a00640 1 -- 172.21.15.45:0/302676169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c014eed0 con 0x7fc6a8004d40 2024-04-16T16:13:28.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.206+0000 7fc6bd600640 1 -- 172.21.15.45:0/302676169 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4003810 con 0x7fc6a8004d40 2024-04-16T16:13:28.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.206+0000 7fc6c6a00640 1 -- 172.21.15.45:0/302676169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c014f490 con 0x7fc6a8004d40 2024-04-16T16:13:28.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.207+0000 7fc6c6a00640 1 -- 172.21.15.45:0/302676169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6c00d10d0 con 0x7fc6a8004d40 2024-04-16T16:13:28.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.208+0000 7fc6bd600640 1 -- 172.21.15.45:0/302676169 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc6b40039b0 con 0x7fc6a8004d40 2024-04-16T16:13:28.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.209+0000 7fc6bd600640 1 --2- 172.21.15.45:0/302676169 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc6a006c8f0 0x7fc6a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:28.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.209+0000 7fc6bd600640 1 -- 172.21.15.45:0/302676169 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc6b408b830 con 0x7fc6a8004d40 2024-04-16T16:13:28.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.210+0000 7fc6c4c00640 1 --2- 172.21.15.45:0/302676169 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc6a006c8f0 0x7fc6a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:28.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.211+0000 7fc6c4c00640 1 --2- 172.21.15.45:0/302676169 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc6a006c8f0 0x7fc6a006edb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac004770 tx=0x7fc6ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:28.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.214+0000 7fc6bd600640 1 -- 172.21.15.45:0/302676169 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc6c00d10d0 con 0x7fc6a8004d40 2024-04-16T16:13:28.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.460+0000 7fc6c6a00640 1 -- 172.21.15.45:0/302676169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fc6c0075780 con 0x7fc6a8004d40 2024-04-16T16:13:28.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.944+0000 7fc6bd600640 1 -- 172.21.15.45:0/302676169 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7fc6b4059c70 con 0x7fc6a8004d40 2024-04-16T16:13:28.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.948+0000 7fc6c6a00640 1 -- 172.21.15.45:0/302676169 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc6a006c8f0 msgr2=0x7fc6a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:28.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.948+0000 7fc6c6a00640 1 --2- 172.21.15.45:0/302676169 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc6a006c8f0 0x7fc6a006edb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac004770 tx=0x7fc6ac008040 comp rx=0 tx=0).stop 2024-04-16T16:13:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.948+0000 7fc6c6a00640 1 -- 172.21.15.45:0/302676169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6a8004d40 msgr2=0x7fc6c00cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.948+0000 7fc6c6a00640 1 --2- 172.21.15.45:0/302676169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6a8004d40 0x7fc6c00cc130 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400e580 tx=0x7fc6b400ea40 comp rx=0 tx=0).stop 2024-04-16T16:13:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.948+0000 7fc6c6a00640 1 -- 172.21.15.45:0/302676169 shutdown_connections 2024-04-16T16:13:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.948+0000 7fc6c6a00640 1 --2- 172.21.15.45:0/302676169 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc6a006c8f0 0x7fc6a006edb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.948+0000 7fc6c6a00640 1 --2- 172.21.15.45:0/302676169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6c00ab7a0 0x7fc6c00ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.948+0000 7fc6c6a00640 1 --2- 172.21.15.45:0/302676169 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6c00b2d10 0x7fc6c00ab260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.948+0000 7fc6c6a00640 1 --2- 172.21.15.45:0/302676169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6a8004d40 0x7fc6c00cc130 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.948+0000 7fc6c6a00640 1 -- 172.21.15.45:0/302676169 >> 172.21.15.45:0/302676169 conn(0x7fc6c006cf90 msgr2=0x7fc6c00b5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.948+0000 7fc6c6a00640 1 -- 172.21.15.45:0/302676169 shutdown_connections 2024-04-16T16:13:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:28.948+0000 7fc6c6a00640 1 -- 172.21.15.45:0/302676169 wait complete. 2024-04-16T16:13:28.973 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-16T16:13:28.974 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:13:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.196+0000 7f3e01200640 1 Processor -- start 2024-04-16T16:13:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.196+0000 7f3e01200640 1 -- start start 2024-04-16T16:13:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.197+0000 7f3e01200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3dfc15ea60 0x7f3dfc06e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.197+0000 7f3e01200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3dfc067be0 0x7f3dfc06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.197+0000 7f3e01200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dfc0cf4b0 con 0x7f3dfc1625a0 2024-04-16T16:13:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.197+0000 7f3e01200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3dfc0d0030 con 0x7f3dfc067be0 2024-04-16T16:13:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.197+0000 7f3e01200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3dfc0cf790 con 0x7f3dfc15ea60 2024-04-16T16:13:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.197+0000 7f3dfaa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3dfc067be0 0x7f3dfc06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.197+0000 7f3dfaa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3dfc067be0 0x7f3dfc06ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49650/0 (socket says 172.21.15.45:49650) 2024-04-16T16:13:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.197+0000 7f3dfaa00640 1 -- 172.21.15.45:0/2033976516 learned_addr learned my addr 172.21.15.45:0/2033976516 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.197+0000 7f3dfbe00640 1 --1- 172.21.15.45:0/2033976516 >> v1:172.21.15.45:6789/0 conn(0x7f3dfc1625a0 0x7f3dfc15ee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40312/0 (socket says 172.21.15.45:40312) 2024-04-16T16:13:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.198+0000 7f3dfb400640 1 --2- 172.21.15.45:0/2033976516 >> v2:172.21.15.103:3300/0 conn(0x7f3dfc15ea60 0x7f3dfc06e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.198+0000 7f3dfaa00640 1 -- 172.21.15.45:0/2033976516 >> v2:172.21.15.103:3300/0 conn(0x7f3dfc15ea60 msgr2=0x7f3dfc06e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.198+0000 7f3dfaa00640 1 --2- 172.21.15.45:0/2033976516 >> v2:172.21.15.103:3300/0 conn(0x7f3dfc15ea60 0x7f3dfc06e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.198+0000 7f3dfaa00640 1 -- 172.21.15.45:0/2033976516 >> v1:172.21.15.45:6789/0 conn(0x7f3dfc1625a0 legacy=0x7f3dfc15ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.198+0000 7f3dfaa00640 1 -- 172.21.15.45:0/2033976516 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dfc0cfa70 con 0x7f3dfc067be0 2024-04-16T16:13:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.199+0000 7f3dfaa00640 1 --2- 172.21.15.45:0/2033976516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc067be0 0x7f3dfc06ecd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc06e040 tx=0x7f3de402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7f7516ebc56b2a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.199+0000 7f3dfa000640 1 -- 172.21.15.45:0/2033976516 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de402cb20 con 0x7f3dfc067be0 2024-04-16T16:13:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.199+0000 7f3dfa000640 1 -- 172.21.15.45:0/2033976516 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3de402ccc0 con 0x7f3dfc067be0 2024-04-16T16:13:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.199+0000 7f3dfa000640 1 -- 172.21.15.45:0/2033976516 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de4034610 con 0x7f3dfc067be0 2024-04-16T16:13:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.200+0000 7f3e01200640 1 -- 172.21.15.45:0/2033976516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc067be0 msgr2=0x7f3dfc06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.200+0000 7f3e01200640 1 --2- 172.21.15.45:0/2033976516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc067be0 0x7f3dfc06ecd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc06e040 tx=0x7f3de402bf80 comp rx=0 tx=0).stop 2024-04-16T16:13:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.200+0000 7f3e01200640 1 -- 172.21.15.45:0/2033976516 shutdown_connections 2024-04-16T16:13:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.200+0000 7f3e01200640 1 --2- 172.21.15.45:0/2033976516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc067be0 0x7f3dfc06ecd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.200+0000 7f3e01200640 1 --2- 172.21.15.45:0/2033976516 >> v2:172.21.15.103:3300/0 conn(0x7f3dfc15ea60 0x7f3dfc06e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.200+0000 7f3e01200640 1 -- 172.21.15.45:0/2033976516 >> 172.21.15.45:0/2033976516 conn(0x7f3dfc0cd820 msgr2=0x7f3dfc0cdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.200+0000 7f3e01200640 1 -- 172.21.15.45:0/2033976516 shutdown_connections 2024-04-16T16:13:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.200+0000 7f3e01200640 1 -- 172.21.15.45:0/2033976516 wait complete. 2024-04-16T16:13:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.202+0000 7f3e01200640 1 Processor -- start 2024-04-16T16:13:29.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.202+0000 7f3e01200640 1 -- start start 2024-04-16T16:13:29.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.203+0000 7f3e01200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dfc067be0 0x7f3dfc160a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:29.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.203+0000 7f3e01200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc15ea60 0x7f3dfc154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:29.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.203+0000 7f3e01200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc1625a0 0x7f3dfc14f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.203+0000 7f3e01200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc0cf270 con 0x7f3dfc15ea60 2024-04-16T16:13:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.203+0000 7f3e01200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc0cf0f0 con 0x7f3dfc1625a0 2024-04-16T16:13:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.203+0000 7f3e01200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3dfc06cd20 con 0x7f3dfc067be0 2024-04-16T16:13:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.203+0000 7f3dfaa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc15ea60 0x7f3dfc154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.203+0000 7f3dfaa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc15ea60 0x7f3dfc154040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49658/0 (socket says 172.21.15.45:49658) 2024-04-16T16:13:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.203+0000 7f3dfaa00640 1 -- 172.21.15.45:0/2428453218 learned_addr learned my addr 172.21.15.45:0/2428453218 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.204+0000 7f3dfb400640 1 --2- 172.21.15.45:0/2428453218 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dfc067be0 0x7f3dfc160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.204+0000 7f3dfaa00640 1 -- 172.21.15.45:0/2428453218 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dfc067be0 msgr2=0x7f3dfc160a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.204+0000 7f3dfbe00640 1 --2- 172.21.15.45:0/2428453218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc1625a0 0x7f3dfc14f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.204+0000 7f3dfaa00640 1 --2- 172.21.15.45:0/2428453218 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dfc067be0 0x7f3dfc160a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.204+0000 7f3dfaa00640 1 -- 172.21.15.45:0/2428453218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc1625a0 msgr2=0x7f3dfc14f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.204+0000 7f3dfaa00640 1 --2- 172.21.15.45:0/2428453218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc1625a0 0x7f3dfc14f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.204+0000 7f3dfaa00640 1 -- 172.21.15.45:0/2428453218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dfc1547e0 con 0x7f3dfc15ea60 2024-04-16T16:13:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.204+0000 7f3dfbe00640 1 --2- 172.21.15.45:0/2428453218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc1625a0 0x7f3dfc14f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.204+0000 7f3dfaa00640 1 --2- 172.21.15.45:0/2428453218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc15ea60 0x7f3dfc154040 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3de4009f40 tx=0x7f3de4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.204+0000 7f3df3400640 1 -- 172.21.15.45:0/2428453218 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de4043020 con 0x7f3dfc15ea60 2024-04-16T16:13:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.204+0000 7f3df3400640 1 -- 172.21.15.45:0/2428453218 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3de403e450 con 0x7f3dfc15ea60 2024-04-16T16:13:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.204+0000 7f3df3400640 1 -- 172.21.15.45:0/2428453218 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de4009510 con 0x7f3dfc15ea60 2024-04-16T16:13:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.204+0000 7f3e01200640 1 -- 172.21.15.45:0/2428453218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dfc154ac0 con 0x7f3dfc15ea60 2024-04-16T16:13:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.205+0000 7f3e01200640 1 -- 172.21.15.45:0/2428453218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dfc155080 con 0x7f3dfc15ea60 2024-04-16T16:13:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.206+0000 7f3e01200640 1 -- 172.21.15.45:0/2428453218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dfc075780 con 0x7f3dfc15ea60 2024-04-16T16:13:29.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.207+0000 7f3df3400640 1 -- 172.21.15.45:0/2428453218 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3de4046020 con 0x7f3dfc15ea60 2024-04-16T16:13:29.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.208+0000 7f3df3400640 1 --2- 172.21.15.45:0/2428453218 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3dd806c940 0x7f3dd806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:29.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.208+0000 7f3df3400640 1 -- 172.21.15.45:0/2428453218 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3de40ae2f0 con 0x7f3dfc15ea60 2024-04-16T16:13:29.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.208+0000 7f3dfb400640 1 --2- 172.21.15.45:0/2428453218 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3dd806c940 0x7f3dd806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.210+0000 7f3dfb400640 1 --2- 172.21.15.45:0/2428453218 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3dd806c940 0x7f3dd806ee00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3dec006fd0 tx=0x7f3dec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:29.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.212+0000 7f3df3400640 1 -- 172.21.15.45:0/2428453218 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3de404d030 con 0x7f3dfc15ea60 2024-04-16T16:13:29.358 INFO:tasks.ceph.mds.c.smithi045.stderr:2024-04-16T16:13:29.358+0000 7f992ec00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:13:29.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.462+0000 7f3e01200640 1 -- 172.21.15.45:0/2428453218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3dfc151160 con 0x7f3dfc15ea60 2024-04-16T16:13:29.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.463+0000 7f3df3400640 1 -- 172.21.15.45:0/2428453218 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5178 (secure 0 0 0) 0x7f3de407c800 con 0x7f3dfc15ea60 2024-04-16T16:13:29.463 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:29.463 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2486891866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2486891866},{"type":"v1","addr":"172.21.15.103:6836","nonce":2486891866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1877281473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1877281473},{"type":"v1","addr":"172.21.15.103:6837","nonce":1877281473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/315848518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":315848518},{"type":"v1","addr":"172.21.15.45:6833","nonce":315848518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:13:25.369839+0000","modified":"2024-04-16T16:13:28.463144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2461453826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2461453826},{"type":"v1","addr":"172.21.15.45:6835","nonce":2461453826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-04-16T16:13:29.463 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 8 2024-04-16T16:13:29.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.466+0000 7f3e01200640 1 -- 172.21.15.45:0/2428453218 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3dd806c940 msgr2=0x7f3dd806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:29.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.466+0000 7f3e01200640 1 --2- 172.21.15.45:0/2428453218 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3dd806c940 0x7f3dd806ee00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3dec006fd0 tx=0x7f3dec008040 comp rx=0 tx=0).stop 2024-04-16T16:13:29.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.466+0000 7f3e01200640 1 -- 172.21.15.45:0/2428453218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc15ea60 msgr2=0x7f3dfc154040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:29.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.466+0000 7f3e01200640 1 --2- 172.21.15.45:0/2428453218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc15ea60 0x7f3dfc154040 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3de4009f40 tx=0x7f3de4004a40 comp rx=0 tx=0).stop 2024-04-16T16:13:29.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.466+0000 7f3e01200640 1 -- 172.21.15.45:0/2428453218 shutdown_connections 2024-04-16T16:13:29.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.466+0000 7f3e01200640 1 --2- 172.21.15.45:0/2428453218 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3dd806c940 0x7f3dd806ee00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.466+0000 7f3e01200640 1 --2- 172.21.15.45:0/2428453218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc1625a0 0x7f3dfc14f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.466+0000 7f3e01200640 1 --2- 172.21.15.45:0/2428453218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc15ea60 0x7f3dfc154040 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.466+0000 7f3e01200640 1 --2- 172.21.15.45:0/2428453218 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dfc067be0 0x7f3dfc160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.466+0000 7f3e01200640 1 -- 172.21.15.45:0/2428453218 >> 172.21.15.45:0/2428453218 conn(0x7f3dfc0cd820 msgr2=0x7f3dfc159000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:29.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.466+0000 7f3e01200640 1 -- 172.21.15.45:0/2428453218 shutdown_connections 2024-04-16T16:13:29.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.466+0000 7f3e01200640 1 -- 172.21.15.45:0/2428453218 wait complete. 2024-04-16T16:13:29.486 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:13:29.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.713+0000 7f8557e00640 1 Processor -- start 2024-04-16T16:13:29.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.714+0000 7f8557e00640 1 -- start start 2024-04-16T16:13:29.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.715+0000 7f8557e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8558146220 0x7f855815ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:29.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.715+0000 7f8557e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f85581608a0 0x7f8558160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:29.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.715+0000 7f8557e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f855815d360 con 0x7f855806f7a0 2024-04-16T16:13:29.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.715+0000 7f8557e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8558161230 con 0x7f85581608a0 2024-04-16T16:13:29.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.715+0000 7f8557e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f8558160410 con 0x7f8558146220 2024-04-16T16:13:29.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.715+0000 7f8557400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f85581608a0 0x7f8558160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:29.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.715+0000 7f8557400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f85581608a0 0x7f8558160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49660/0 (socket says 172.21.15.45:49660) 2024-04-16T16:13:29.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.715+0000 7f8557400640 1 -- 172.21.15.45:0/3990672959 learned_addr learned my addr 172.21.15.45:0/3990672959 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:29.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.716+0000 7f8556a00640 1 --2- 172.21.15.45:0/3990672959 >> v2:172.21.15.103:3300/0 conn(0x7f8558146220 0x7f855815ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:29.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.716+0000 7f8557400640 1 -- 172.21.15.45:0/3990672959 >> v2:172.21.15.103:3300/0 conn(0x7f8558146220 msgr2=0x7f855815ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:29.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.716+0000 7f8557400640 1 --2- 172.21.15.45:0/3990672959 >> v2:172.21.15.103:3300/0 conn(0x7f8558146220 0x7f855815ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.716+0000 7f8557400640 1 -- 172.21.15.45:0/3990672959 >> v1:172.21.15.103:6789/0 conn(0x7f855806f7a0 legacy=0x7f855806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:29.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.716+0000 7f8557400640 1 -- 172.21.15.45:0/3990672959 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855815d640 con 0x7f85581608a0 2024-04-16T16:13:29.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.717+0000 7f8557400640 1 --2- 172.21.15.45:0/3990672959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85581608a0 0x7f8558160cf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8558070210 tx=0x7f855002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc95b1c65d4169a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:29.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.717+0000 7f8555600640 1 -- 172.21.15.45:0/3990672959 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8550035020 con 0x7f85581608a0 2024-04-16T16:13:29.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.717+0000 7f8555600640 1 -- 172.21.15.45:0/3990672959 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8550031870 con 0x7f85581608a0 2024-04-16T16:13:29.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.717+0000 7f8555600640 1 -- 172.21.15.45:0/3990672959 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8550031c40 con 0x7f85581608a0 2024-04-16T16:13:29.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.718+0000 7f8557e00640 1 -- 172.21.15.45:0/3990672959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85581608a0 msgr2=0x7f8558160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:29.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.718+0000 7f8557e00640 1 --2- 172.21.15.45:0/3990672959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85581608a0 0x7f8558160cf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8558070210 tx=0x7f855002b910 comp rx=0 tx=0).stop 2024-04-16T16:13:29.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.718+0000 7f8557e00640 1 -- 172.21.15.45:0/3990672959 shutdown_connections 2024-04-16T16:13:29.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.718+0000 7f8557e00640 1 --2- 172.21.15.45:0/3990672959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85581608a0 0x7f8558160cf0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.718+0000 7f8557e00640 1 --2- 172.21.15.45:0/3990672959 >> v2:172.21.15.103:3300/0 conn(0x7f8558146220 0x7f855815ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.718+0000 7f8557e00640 1 -- 172.21.15.45:0/3990672959 >> 172.21.15.45:0/3990672959 conn(0x7f855806b930 msgr2=0x7f855806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:29.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.718+0000 7f8557e00640 1 -- 172.21.15.45:0/3990672959 shutdown_connections 2024-04-16T16:13:29.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.718+0000 7f8557e00640 1 -- 172.21.15.45:0/3990672959 wait complete. 2024-04-16T16:13:29.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.720+0000 7f8557e00640 1 Processor -- start 2024-04-16T16:13:29.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.720+0000 7f8557e00640 1 -- start start 2024-04-16T16:13:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.721+0000 7f8557e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f855806f7a0 0x7f85581f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.721+0000 7f8557e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8558146220 0x7f855813d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.721+0000 7f8557e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85581608a0 0x7f855813d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.721+0000 7f8557e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f855815dc00 con 0x7f855806f7a0 2024-04-16T16:13:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.721+0000 7f8557e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f855806e650 con 0x7f8558146220 2024-04-16T16:13:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.721+0000 7f8557e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f855806fc20 con 0x7f85581608a0 2024-04-16T16:13:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.721+0000 7f8556a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f855806f7a0 0x7f85581f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.721+0000 7f8556a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f855806f7a0 0x7f85581f23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49666/0 (socket says 172.21.15.45:49666) 2024-04-16T16:13:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.721+0000 7f8556a00640 1 -- 172.21.15.45:0/1440885019 learned_addr learned my addr 172.21.15.45:0/1440885019 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.722+0000 7f8556a00640 1 -- 172.21.15.45:0/1440885019 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85581608a0 msgr2=0x7f855813d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.722+0000 7f8556a00640 1 --2- 172.21.15.45:0/1440885019 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85581608a0 0x7f855813d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.722+0000 7f8556a00640 1 -- 172.21.15.45:0/1440885019 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8558146220 msgr2=0x7f855813d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.722+0000 7f8556a00640 1 --2- 172.21.15.45:0/1440885019 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8558146220 0x7f855813d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.722+0000 7f8556a00640 1 -- 172.21.15.45:0/1440885019 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85580b4770 con 0x7f855806f7a0 2024-04-16T16:13:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.722+0000 7f8556a00640 1 --2- 172.21.15.45:0/1440885019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f855806f7a0 0x7f85581f23f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f854000d800 tx=0x7f854000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.722+0000 7f854ea00640 1 -- 172.21.15.45:0/1440885019 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854000b6c0 con 0x7f855806f7a0 2024-04-16T16:13:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.722+0000 7f854ea00640 1 -- 172.21.15.45:0/1440885019 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8540019070 con 0x7f855806f7a0 2024-04-16T16:13:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.722+0000 7f854ea00640 1 -- 172.21.15.45:0/1440885019 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854000ba80 con 0x7f855806f7a0 2024-04-16T16:13:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.722+0000 7f8557e00640 1 -- 172.21.15.45:0/1440885019 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85581f2ce0 con 0x7f855806f7a0 2024-04-16T16:13:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.723+0000 7f8557e00640 1 -- 172.21.15.45:0/1440885019 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85581f32a0 con 0x7f855806f7a0 2024-04-16T16:13:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.724+0000 7f8557e00640 1 -- 172.21.15.45:0/1440885019 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8558075780 con 0x7f855806f7a0 2024-04-16T16:13:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.725+0000 7f854ea00640 1 -- 172.21.15.45:0/1440885019 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f854000bc20 con 0x7f855806f7a0 2024-04-16T16:13:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.725+0000 7f854ea00640 1 --2- 172.21.15.45:0/1440885019 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f853406c8f0 0x7f853406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.726+0000 7f854ea00640 1 -- 172.21.15.45:0/1440885019 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f854008c040 con 0x7f855806f7a0 2024-04-16T16:13:29.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.726+0000 7f8556000640 1 --2- 172.21.15.45:0/1440885019 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f853406c8f0 0x7f853406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.727+0000 7f8556000640 1 --2- 172.21.15.45:0/1440885019 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f853406c8f0 0x7f853406edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f855813dfe0 tx=0x7f8548007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.730+0000 7f854ea00640 1 -- 172.21.15.45:0/1440885019 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f854001c360 con 0x7f855806f7a0 2024-04-16T16:13:29.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.941+0000 7f8557e00640 1 -- 172.21.15.45:0/1440885019 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85580cff60 con 0x7f855806f7a0 2024-04-16T16:13:29.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.944+0000 7f854ea00640 1 -- 172.21.15.45:0/1440885019 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f854005a470 con 0x7f855806f7a0 2024-04-16T16:13:29.945 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:29.945 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":21,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:26.918617+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:13:29.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.946+0000 7f8557e00640 1 -- 172.21.15.45:0/1440885019 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f853406c8f0 msgr2=0x7f853406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:29.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.946+0000 7f8557e00640 1 --2- 172.21.15.45:0/1440885019 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f853406c8f0 0x7f853406edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f855813dfe0 tx=0x7f8548007d30 comp rx=0 tx=0).stop 2024-04-16T16:13:29.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.946+0000 7f8557e00640 1 -- 172.21.15.45:0/1440885019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f855806f7a0 msgr2=0x7f85581f23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:29.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.946+0000 7f8557e00640 1 --2- 172.21.15.45:0/1440885019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f855806f7a0 0x7f85581f23f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f854000d800 tx=0x7f854000dcc0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.947+0000 7f8557e00640 1 -- 172.21.15.45:0/1440885019 shutdown_connections 2024-04-16T16:13:29.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.947+0000 7f8557e00640 1 --2- 172.21.15.45:0/1440885019 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f853406c8f0 0x7f853406edb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.947+0000 7f8557e00640 1 --2- 172.21.15.45:0/1440885019 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85581608a0 0x7f855813d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.947+0000 7f8557e00640 1 --2- 172.21.15.45:0/1440885019 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8558146220 0x7f855813d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.947+0000 7f8557e00640 1 --2- 172.21.15.45:0/1440885019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f855806f7a0 0x7f85581f23f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:29.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.947+0000 7f8557e00640 1 -- 172.21.15.45:0/1440885019 >> 172.21.15.45:0/1440885019 conn(0x7f855806b930 msgr2=0x7f85580c6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:29.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.947+0000 7f8557e00640 1 -- 172.21.15.45:0/1440885019 shutdown_connections 2024-04-16T16:13:29.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:29.947+0000 7f8557e00640 1 -- 172.21.15.45:0/1440885019 wait complete. 2024-04-16T16:13:29.970 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-16T16:13:29.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:13:30.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.189+0000 7fdd06000640 1 Processor -- start 2024-04-16T16:13:30.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.189+0000 7fdd06000640 1 -- start start 2024-04-16T16:13:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.192+0000 7fdd06000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd000ce9a0 0x7fdd000cb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.192+0000 7fdd06000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd0015ecd0 con 0x7fdd000ccb20 2024-04-16T16:13:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.192+0000 7fdd06000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd000cd6d0 con 0x7fdd0015e8f0 2024-04-16T16:13:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.192+0000 7fdd06000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdd000ce2e0 con 0x7fdd000ce9a0 2024-04-16T16:13:30.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.192+0000 7fdcffe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdd000ccb20 0x7fdd000cac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40322/0 (socket says 172.21.15.45:40322) 2024-04-16T16:13:30.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.192+0000 7fdcffe00640 1 -- 172.21.15.45:0/3141653473 learned_addr learned my addr 172.21.15.45:0/3141653473 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:30.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.192+0000 7fdd05600640 1 --2- 172.21.15.45:0/3141653473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd000ce9a0 0x7fdd000cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:30.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.193+0000 7fdcff400640 1 -- 172.21.15.45:0/3141653473 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2168967694 0 0) 0x7fdd0015ecd0 con 0x7fdd000ccb20 2024-04-16T16:13:30.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.193+0000 7fdcff400640 1 -- 172.21.15.45:0/3141653473 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdce4003520 con 0x7fdd000ccb20 2024-04-16T16:13:30.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.193+0000 7fdcff400640 1 -- 172.21.15.45:0/3141653473 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1056824908 0 0) 0x7fdce4003520 con 0x7fdd000ccb20 2024-04-16T16:13:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.193+0000 7fdcff400640 1 -- 172.21.15.45:0/3141653473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd000ce9a0 msgr2=0x7fdd000cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.193+0000 7fdcff400640 1 --2- 172.21.15.45:0/3141653473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd000ce9a0 0x7fdd000cb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.193+0000 7fdcff400640 1 -- 172.21.15.45:0/3141653473 >> v1:172.21.15.103:6789/0 conn(0x7fdd0015e8f0 legacy=0x7fdd000cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.193+0000 7fdcff400640 1 -- 172.21.15.45:0/3141653473 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd000cd9b0 con 0x7fdd000ccb20 2024-04-16T16:13:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.193+0000 7fdcff400640 1 -- 172.21.15.45:0/3141653473 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fdcec001670 con 0x7fdd000ccb20 2024-04-16T16:13:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.193+0000 7fdcff400640 1 -- 172.21.15.45:0/3141653473 >> v1:172.21.15.45:6789/0 conn(0x7fdd000ccb20 legacy=0x7fdd000cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.193+0000 7fdcff400640 1 --2- 172.21.15.45:0/3141653473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdce40039e0 0x7fdce4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.193+0000 7fdcff400640 1 --2- 172.21.15.45:0/3141653473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd000ce9a0 0x7fdce4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.193+0000 7fdcff400640 1 --2- 172.21.15.45:0/3141653473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdce4004d40 0x7fdce40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.193+0000 7fdcff400640 1 -- 172.21.15.45:0/3141653473 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdd000ce2e0 con 0x7fdce40039e0 2024-04-16T16:13:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.193+0000 7fdcff400640 1 -- 172.21.15.45:0/3141653473 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdce400baa0 con 0x7fdce4004d40 2024-04-16T16:13:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.193+0000 7fdcff400640 1 -- 172.21.15.45:0/3141653473 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdce400bca0 con 0x7fdd000ce9a0 2024-04-16T16:13:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.194+0000 7fdd04c00640 1 --2- 172.21.15.45:0/3141653473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdce40039e0 0x7fdce4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:30.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.194+0000 7fdcffe00640 1 --2- 172.21.15.45:0/3141653473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd000ce9a0 0x7fdce4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:30.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.194+0000 7fdd05600640 1 --2- 172.21.15.45:0/3141653473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdce4004d40 0x7fdce40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:30.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.194+0000 7fdd04c00640 1 -- 172.21.15.45:0/3141653473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd000ce9a0 msgr2=0x7fdce4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:30.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.194+0000 7fdd04c00640 1 --2- 172.21.15.45:0/3141653473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd000ce9a0 0x7fdce4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.194+0000 7fdd04c00640 1 -- 172.21.15.45:0/3141653473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdce4004d40 msgr2=0x7fdce40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:30.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.194+0000 7fdd04c00640 1 --2- 172.21.15.45:0/3141653473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdce4004d40 0x7fdce40051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.194+0000 7fdd04c00640 1 -- 172.21.15.45:0/3141653473 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce4003520 con 0x7fdce40039e0 2024-04-16T16:13:30.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.195+0000 7fdd04c00640 1 --2- 172.21.15.45:0/3141653473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdce40039e0 0x7fdce4003e00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0001720 tx=0x7fdce4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641e1abbd8635ba3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:30.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.195+0000 7fdcff400640 1 -- 172.21.15.45:0/3141653473 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0001370 con 0x7fdce40039e0 2024-04-16T16:13:30.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.196+0000 7fdcff400640 1 -- 172.21.15.45:0/3141653473 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcf00052d0 con 0x7fdce40039e0 2024-04-16T16:13:30.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.196+0000 7fdcff400640 1 -- 172.21.15.45:0/3141653473 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0003830 con 0x7fdce40039e0 2024-04-16T16:13:30.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.196+0000 7fdd06000640 1 -- 172.21.15.45:0/3141653473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdce40039e0 msgr2=0x7fdce4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:30.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.196+0000 7fdd06000640 1 --2- 172.21.15.45:0/3141653473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdce40039e0 0x7fdce4003e00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0001720 tx=0x7fdce4005830 comp rx=0 tx=0).stop 2024-04-16T16:13:30.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.196+0000 7fdd05600640 1 -- 172.21.15.45:0/3141653473 reap_dead start 2024-04-16T16:13:30.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.196+0000 7fdd06000640 1 -- 172.21.15.45:0/3141653473 shutdown_connections 2024-04-16T16:13:30.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.196+0000 7fdd06000640 1 -- 172.21.15.45:0/3141653473 >> 172.21.15.45:0/3141653473 conn(0x7fdd0006b930 msgr2=0x7fdd0006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:30.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.196+0000 7fdd06000640 1 -- 172.21.15.45:0/3141653473 shutdown_connections 2024-04-16T16:13:30.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.196+0000 7fdd06000640 1 -- 172.21.15.45:0/3141653473 wait complete. 2024-04-16T16:13:30.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.198+0000 7fdd06000640 1 Processor -- start 2024-04-16T16:13:30.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.198+0000 7fdd06000640 1 -- start start 2024-04-16T16:13:30.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.199+0000 7fdd06000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdce40039e0 0x7fdd00150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.199+0000 7fdd06000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd0014f100 0x7fdd00153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.199+0000 7fdd06000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd000b3e60 0x7fdd000b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.199+0000 7fdd06000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdd000cd360 con 0x7fdce40039e0 2024-04-16T16:13:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.199+0000 7fdd06000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdd000cdf70 con 0x7fdd000b3e60 2024-04-16T16:13:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.199+0000 7fdd06000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdd000cf080 con 0x7fdd0014f100 2024-04-16T16:13:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.199+0000 7fdd04c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdce40039e0 0x7fdd00150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.200+0000 7fdd04c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdce40039e0 0x7fdd00150270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49676/0 (socket says 172.21.15.45:49676) 2024-04-16T16:13:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.200+0000 7fdd04c00640 1 -- 172.21.15.45:0/3583911480 learned_addr learned my addr 172.21.15.45:0/3583911480 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.200+0000 7fdd04c00640 1 -- 172.21.15.45:0/3583911480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd0014f100 msgr2=0x7fdd00153010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.200+0000 7fdcffe00640 1 --2- 172.21.15.45:0/3583911480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd0014f100 0x7fdd00153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.200+0000 7fdd04c00640 1 --2- 172.21.15.45:0/3583911480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd0014f100 0x7fdd00153010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.200+0000 7fdd04c00640 1 -- 172.21.15.45:0/3583911480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd000b3e60 msgr2=0x7fdd000b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.200+0000 7fdd05600640 1 --2- 172.21.15.45:0/3583911480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd000b3e60 0x7fdd000b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.200+0000 7fdd04c00640 1 --2- 172.21.15.45:0/3583911480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd000b3e60 0x7fdd000b4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.200+0000 7fdd04c00640 1 -- 172.21.15.45:0/3583911480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd00153550 con 0x7fdce40039e0 2024-04-16T16:13:30.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.200+0000 7fdd04c00640 1 --2- 172.21.15.45:0/3583911480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdce40039e0 0x7fdd00150270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0008fd0 tx=0x7fdcf0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:30.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.201+0000 7fdcfcc00640 1 -- 172.21.15.45:0/3583911480 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0041070 con 0x7fdce40039e0 2024-04-16T16:13:30.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.201+0000 7fdcfcc00640 1 -- 172.21.15.45:0/3583911480 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcf0000ed0 con 0x7fdce40039e0 2024-04-16T16:13:30.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.201+0000 7fdcfcc00640 1 -- 172.21.15.45:0/3583911480 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0007a30 con 0x7fdce40039e0 2024-04-16T16:13:30.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.201+0000 7fdd06000640 1 -- 172.21.15.45:0/3583911480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd00150850 con 0x7fdce40039e0 2024-04-16T16:13:30.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.201+0000 7fdd06000640 1 -- 172.21.15.45:0/3583911480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd00150e10 con 0x7fdce40039e0 2024-04-16T16:13:30.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.202+0000 7fdd06000640 1 -- 172.21.15.45:0/3583911480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd000cc410 con 0x7fdce40039e0 2024-04-16T16:13:30.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.203+0000 7fdcfcc00640 1 -- 172.21.15.45:0/3583911480 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdcf0044070 con 0x7fdce40039e0 2024-04-16T16:13:30.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.204+0000 7fdcfcc00640 1 --2- 172.21.15.45:0/3583911480 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdce006c920 0x7fdce006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:30.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.204+0000 7fdcfcc00640 1 -- 172.21.15.45:0/3583911480 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdcf00ad640 con 0x7fdce40039e0 2024-04-16T16:13:30.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.205+0000 7fdcffe00640 1 --2- 172.21.15.45:0/3583911480 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdce006c920 0x7fdce006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:30.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.206+0000 7fdcffe00640 1 --2- 172.21.15.45:0/3583911480 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdce006c920 0x7fdce006ede0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdcec001c00 tx=0x7fdcec000ef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:30.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.209+0000 7fdcfcc00640 1 -- 172.21.15.45:0/3583911480 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd000cc410 con 0x7fdce40039e0 2024-04-16T16:13:30.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.409+0000 7fdd06000640 1 -- 172.21.15.45:0/3583911480 --> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdd00075780 con 0x7fdce006c920 2024-04-16T16:13:30.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.432+0000 7fdcfcc00640 1 -- 172.21.15.45:0/3583911480 <== mgr.4105 v2:172.21.15.103:6832/2711977587 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251928 (secure 0 0 0) 0x7fdd00075780 con 0x7fdce006c920 2024-04-16T16:13:30.436 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:13:30.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.438+0000 7fdd06000640 1 -- 172.21.15.45:0/3583911480 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdce006c920 msgr2=0x7fdce006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:30.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.438+0000 7fdd06000640 1 --2- 172.21.15.45:0/3583911480 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdce006c920 0x7fdce006ede0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdcec001c00 tx=0x7fdcec000ef0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.438+0000 7fdd06000640 1 -- 172.21.15.45:0/3583911480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdce40039e0 msgr2=0x7fdd00150270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:30.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.438+0000 7fdd06000640 1 --2- 172.21.15.45:0/3583911480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdce40039e0 0x7fdd00150270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0008fd0 tx=0x7fdcf0001120 comp rx=0 tx=0).stop 2024-04-16T16:13:30.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.439+0000 7fdd06000640 1 -- 172.21.15.45:0/3583911480 shutdown_connections 2024-04-16T16:13:30.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.439+0000 7fdd06000640 1 --2- 172.21.15.45:0/3583911480 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdce006c920 0x7fdce006ede0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.439+0000 7fdd06000640 1 --2- 172.21.15.45:0/3583911480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd000b3e60 0x7fdd000b4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.439+0000 7fdd06000640 1 --2- 172.21.15.45:0/3583911480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd0014f100 0x7fdd00153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.439+0000 7fdd06000640 1 --2- 172.21.15.45:0/3583911480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdce40039e0 0x7fdd00150270 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.439+0000 7fdd06000640 1 -- 172.21.15.45:0/3583911480 >> 172.21.15.45:0/3583911480 conn(0x7fdd0006b930 msgr2=0x7fdd0006eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:30.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.439+0000 7fdd06000640 1 -- 172.21.15.45:0/3583911480 shutdown_connections 2024-04-16T16:13:30.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.439+0000 7fdd06000640 1 -- 172.21.15.45:0/3583911480 wait complete. 2024-04-16T16:13:30.459 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-04-16T16:13:29.092606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":135,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235740,"kb_used_data":2896,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738980,"statfs":{"total":773094113280,"available":772852715520,"internally_reserved":0,"allocated":2965504,"data_stored":1777796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.220988"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943618+0000","last_change":"2024-04-16T16:13:25.934978+0000","last_active":"2024-04-16T16:13:26.943618+0000","last_peered":"2024-04-16T16:13:26.943618+0000","last_clean":"2024-04-16T16:13:26.943618+0000","last_became_active":"2024-04-16T16:13:25.934469+0000","last_became_peered":"2024-04-16T16:13:25.934469+0000","last_unstale":"2024-04-16T16:13:26.943618+0000","last_undegraded":"2024-04-16T16:13:26.943618+0000","last_fullsized":"2024-04-16T16:13:26.943618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:40:10.575471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955303+0000","last_change":"2024-04-16T16:13:25.936302+0000","last_active":"2024-04-16T16:13:26.955303+0000","last_peered":"2024-04-16T16:13:26.955303+0000","last_clean":"2024-04-16T16:13:26.955303+0000","last_became_active":"2024-04-16T16:13:25.935907+0000","last_became_peered":"2024-04-16T16:13:25.935907+0000","last_unstale":"2024-04-16T16:13:26.955303+0000","last_undegraded":"2024-04-16T16:13:26.955303+0000","last_fullsized":"2024-04-16T16:13:26.955303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.951569+0000","last_change":"2024-04-16T16:13:25.935281+0000","last_active":"2024-04-16T16:13:25.951569+0000","last_peered":"2024-04-16T16:13:25.951569+0000","last_clean":"2024-04-16T16:13:25.951569+0000","last_became_active":"2024-04-16T16:13:25.934986+0000","last_became_peered":"2024-04-16T16:13:25.934986+0000","last_unstale":"2024-04-16T16:13:25.951569+0000","last_undegraded":"2024-04-16T16:13:25.951569+0000","last_fullsized":"2024-04-16T16:13:25.951569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:21.924263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970937+0000","last_change":"2024-04-16T16:13:25.952485+0000","last_active":"2024-04-16T16:13:26.970937+0000","last_peered":"2024-04-16T16:13:26.970937+0000","last_clean":"2024-04-16T16:13:26.970937+0000","last_became_active":"2024-04-16T16:13:25.952189+0000","last_became_peered":"2024-04-16T16:13:25.952189+0000","last_unstale":"2024-04-16T16:13:26.970937+0000","last_undegraded":"2024-04-16T16:13:26.970937+0000","last_fullsized":"2024-04-16T16:13:26.970937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:47.987635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942605+0000","last_change":"2024-04-16T16:13:25.930237+0000","last_active":"2024-04-16T16:13:26.942605+0000","last_peered":"2024-04-16T16:13:26.942605+0000","last_clean":"2024-04-16T16:13:26.942605+0000","last_became_active":"2024-04-16T16:13:25.929635+0000","last_became_peered":"2024-04-16T16:13:25.929635+0000","last_unstale":"2024-04-16T16:13:26.942605+0000","last_undegraded":"2024-04-16T16:13:26.942605+0000","last_fullsized":"2024-04-16T16:13:26.942605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:30:58.937717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970737+0000","last_change":"2024-04-16T16:13:25.947705+0000","last_active":"2024-04-16T16:13:26.970737+0000","last_peered":"2024-04-16T16:13:26.970737+0000","last_clean":"2024-04-16T16:13:26.970737+0000","last_became_active":"2024-04-16T16:13:25.947433+0000","last_became_peered":"2024-04-16T16:13:25.947433+0000","last_unstale":"2024-04-16T16:13:26.970737+0000","last_undegraded":"2024-04-16T16:13:26.970737+0000","last_fullsized":"2024-04-16T16:13:26.970737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941969+0000","last_change":"2024-04-16T16:13:25.930905+0000","last_active":"2024-04-16T16:13:26.941969+0000","last_peered":"2024-04-16T16:13:26.941969+0000","last_clean":"2024-04-16T16:13:26.941969+0000","last_became_active":"2024-04-16T16:13:25.929782+0000","last_became_peered":"2024-04-16T16:13:25.929782+0000","last_unstale":"2024-04-16T16:13:26.941969+0000","last_undegraded":"2024-04-16T16:13:26.941969+0000","last_fullsized":"2024-04-16T16:13:26.941969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.951126+0000","last_change":"2024-04-16T16:13:25.947751+0000","last_active":"2024-04-16T16:13:25.951126+0000","last_peered":"2024-04-16T16:13:25.951126+0000","last_clean":"2024-04-16T16:13:25.951126+0000","last_became_active":"2024-04-16T16:13:25.947110+0000","last_became_peered":"2024-04-16T16:13:25.947110+0000","last_unstale":"2024-04-16T16:13:25.951126+0000","last_undegraded":"2024-04-16T16:13:25.951126+0000","last_fullsized":"2024-04-16T16:13:25.951126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:32:38.024739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943813+0000","last_change":"2024-04-16T16:13:25.950917+0000","last_active":"2024-04-16T16:13:26.943813+0000","last_peered":"2024-04-16T16:13:26.943813+0000","last_clean":"2024-04-16T16:13:26.943813+0000","last_became_active":"2024-04-16T16:13:25.946716+0000","last_became_peered":"2024-04-16T16:13:25.946716+0000","last_unstale":"2024-04-16T16:13:26.943813+0000","last_undegraded":"2024-04-16T16:13:26.943813+0000","last_fullsized":"2024-04-16T16:13:26.943813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:52:47.172342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961550+0000","last_change":"2024-04-16T16:13:25.941148+0000","last_active":"2024-04-16T16:13:26.961550+0000","last_peered":"2024-04-16T16:13:26.961550+0000","last_clean":"2024-04-16T16:13:26.961550+0000","last_became_active":"2024-04-16T16:13:25.940818+0000","last_became_peered":"2024-04-16T16:13:25.940818+0000","last_unstale":"2024-04-16T16:13:26.961550+0000","last_undegraded":"2024-04-16T16:13:26.961550+0000","last_fullsized":"2024-04-16T16:13:26.961550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960975+0000","last_change":"2024-04-16T16:13:25.943177+0000","last_active":"2024-04-16T16:13:26.960975+0000","last_peered":"2024-04-16T16:13:26.960975+0000","last_clean":"2024-04-16T16:13:26.960975+0000","last_became_active":"2024-04-16T16:13:25.942902+0000","last_became_peered":"2024-04-16T16:13:25.942902+0000","last_unstale":"2024-04-16T16:13:26.960975+0000","last_undegraded":"2024-04-16T16:13:26.960975+0000","last_fullsized":"2024-04-16T16:13:26.960975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:40:58.670474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941834+0000","last_change":"2024-04-16T16:13:25.938811+0000","last_active":"2024-04-16T16:13:26.941834+0000","last_peered":"2024-04-16T16:13:26.941834+0000","last_clean":"2024-04-16T16:13:26.941834+0000","last_became_active":"2024-04-16T16:13:25.938479+0000","last_became_peered":"2024-04-16T16:13:25.938479+0000","last_unstale":"2024-04-16T16:13:26.941834+0000","last_undegraded":"2024-04-16T16:13:26.941834+0000","last_fullsized":"2024-04-16T16:13:26.941834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:41.335228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.953463+0000","last_change":"2024-04-16T16:13:25.950387+0000","last_active":"2024-04-16T16:13:25.953463+0000","last_peered":"2024-04-16T16:13:25.953463+0000","last_clean":"2024-04-16T16:13:25.953463+0000","last_became_active":"2024-04-16T16:13:25.949164+0000","last_became_peered":"2024-04-16T16:13:25.949164+0000","last_unstale":"2024-04-16T16:13:25.953463+0000","last_undegraded":"2024-04-16T16:13:25.953463+0000","last_fullsized":"2024-04-16T16:13:25.953463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:38:46.469621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.951916+0000","last_change":"2024-04-16T16:13:25.939438+0000","last_active":"2024-04-16T16:13:25.951916+0000","last_peered":"2024-04-16T16:13:25.951916+0000","last_clean":"2024-04-16T16:13:25.951916+0000","last_became_active":"2024-04-16T16:13:25.939066+0000","last_became_peered":"2024-04-16T16:13:25.939066+0000","last_unstale":"2024-04-16T16:13:25.951916+0000","last_undegraded":"2024-04-16T16:13:25.951916+0000","last_fullsized":"2024-04-16T16:13:25.951916+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:26.052535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954570+0000","last_change":"2024-04-16T16:13:25.937157+0000","last_active":"2024-04-16T16:13:26.954570+0000","last_peered":"2024-04-16T16:13:26.954570+0000","last_clean":"2024-04-16T16:13:26.954570+0000","last_became_active":"2024-04-16T16:13:25.936851+0000","last_became_peered":"2024-04-16T16:13:25.936851+0000","last_unstale":"2024-04-16T16:13:26.954570+0000","last_undegraded":"2024-04-16T16:13:26.954570+0000","last_fullsized":"2024-04-16T16:13:26.954570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:55.948136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970498+0000","last_change":"2024-04-16T16:13:25.949240+0000","last_active":"2024-04-16T16:13:26.970498+0000","last_peered":"2024-04-16T16:13:26.970498+0000","last_clean":"2024-04-16T16:13:26.970498+0000","last_became_active":"2024-04-16T16:13:25.948913+0000","last_became_peered":"2024-04-16T16:13:25.948913+0000","last_unstale":"2024-04-16T16:13:26.970498+0000","last_undegraded":"2024-04-16T16:13:26.970498+0000","last_fullsized":"2024-04-16T16:13:26.970498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.972475+0000","last_change":"2024-04-16T16:13:25.947113+0000","last_active":"2024-04-16T16:13:26.972475+0000","last_peered":"2024-04-16T16:13:26.972475+0000","last_clean":"2024-04-16T16:13:26.972475+0000","last_became_active":"2024-04-16T16:13:25.943174+0000","last_became_peered":"2024-04-16T16:13:25.943174+0000","last_unstale":"2024-04-16T16:13:26.972475+0000","last_undegraded":"2024-04-16T16:13:26.972475+0000","last_fullsized":"2024-04-16T16:13:26.972475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:08.801668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.937163+0000","last_change":"2024-04-16T16:13:25.942311+0000","last_active":"2024-04-16T16:13:26.937163+0000","last_peered":"2024-04-16T16:13:26.937163+0000","last_clean":"2024-04-16T16:13:26.937163+0000","last_became_active":"2024-04-16T16:13:25.942015+0000","last_became_peered":"2024-04-16T16:13:25.942015+0000","last_unstale":"2024-04-16T16:13:26.937163+0000","last_undegraded":"2024-04-16T16:13:26.937163+0000","last_fullsized":"2024-04-16T16:13:26.937163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:25.740193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943340+0000","last_change":"2024-04-16T16:13:25.929976+0000","last_active":"2024-04-16T16:13:26.943340+0000","last_peered":"2024-04-16T16:13:26.943340+0000","last_clean":"2024-04-16T16:13:26.943340+0000","last_became_active":"2024-04-16T16:13:25.929347+0000","last_became_peered":"2024-04-16T16:13:25.929347+0000","last_unstale":"2024-04-16T16:13:26.943340+0000","last_undegraded":"2024-04-16T16:13:26.943340+0000","last_fullsized":"2024-04-16T16:13:26.943340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:19:05.123994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960773+0000","last_change":"2024-04-16T16:13:25.945575+0000","last_active":"2024-04-16T16:13:26.960773+0000","last_peered":"2024-04-16T16:13:26.960773+0000","last_clean":"2024-04-16T16:13:26.960773+0000","last_became_active":"2024-04-16T16:13:25.945217+0000","last_became_peered":"2024-04-16T16:13:25.945217+0000","last_unstale":"2024-04-16T16:13:26.960773+0000","last_undegraded":"2024-04-16T16:13:26.960773+0000","last_fullsized":"2024-04-16T16:13:26.960773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:55.948136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.952491+0000","last_change":"2024-04-16T16:13:25.945082+0000","last_active":"2024-04-16T16:13:25.952491+0000","last_peered":"2024-04-16T16:13:25.952491+0000","last_clean":"2024-04-16T16:13:25.952491+0000","last_became_active":"2024-04-16T16:13:25.944781+0000","last_became_peered":"2024-04-16T16:13:25.944781+0000","last_unstale":"2024-04-16T16:13:25.952491+0000","last_undegraded":"2024-04-16T16:13:25.952491+0000","last_fullsized":"2024-04-16T16:13:25.952491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:41.335228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970886+0000","last_change":"2024-04-16T16:13:25.943741+0000","last_active":"2024-04-16T16:13:26.970886+0000","last_peered":"2024-04-16T16:13:26.970886+0000","last_clean":"2024-04-16T16:13:26.970886+0000","last_became_active":"2024-04-16T16:13:25.942843+0000","last_became_peered":"2024-04-16T16:13:25.942843+0000","last_unstale":"2024-04-16T16:13:26.970886+0000","last_undegraded":"2024-04-16T16:13:26.970886+0000","last_fullsized":"2024-04-16T16:13:26.970886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:30:58.937717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.946699+0000","last_change":"2024-04-16T16:13:25.943232+0000","last_active":"2024-04-16T16:13:25.946699+0000","last_peered":"2024-04-16T16:13:25.946699+0000","last_clean":"2024-04-16T16:13:25.946699+0000","last_became_active":"2024-04-16T16:13:25.940853+0000","last_became_peered":"2024-04-16T16:13:25.940853+0000","last_unstale":"2024-04-16T16:13:25.946699+0000","last_undegraded":"2024-04-16T16:13:25.946699+0000","last_fullsized":"2024-04-16T16:13:25.946699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:12.895579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970797+0000","last_change":"2024-04-16T16:13:25.943328+0000","last_active":"2024-04-16T16:13:26.970797+0000","last_peered":"2024-04-16T16:13:26.970797+0000","last_clean":"2024-04-16T16:13:26.970797+0000","last_became_active":"2024-04-16T16:13:25.941855+0000","last_became_peered":"2024-04-16T16:13:25.941855+0000","last_unstale":"2024-04-16T16:13:26.970797+0000","last_undegraded":"2024-04-16T16:13:26.970797+0000","last_fullsized":"2024-04-16T16:13:26.970797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:41.243756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.945451+0000","last_change":"2024-04-16T16:13:25.942627+0000","last_active":"2024-04-16T16:13:25.945451+0000","last_peered":"2024-04-16T16:13:25.945451+0000","last_clean":"2024-04-16T16:13:25.945451+0000","last_became_active":"2024-04-16T16:13:25.942285+0000","last_became_peered":"2024-04-16T16:13:25.942285+0000","last_unstale":"2024-04-16T16:13:25.945451+0000","last_undegraded":"2024-04-16T16:13:25.945451+0000","last_fullsized":"2024-04-16T16:13:25.945451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:41.335228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961635+0000","last_change":"2024-04-16T16:13:25.950401+0000","last_active":"2024-04-16T16:13:26.961635+0000","last_peered":"2024-04-16T16:13:26.961635+0000","last_clean":"2024-04-16T16:13:26.961635+0000","last_became_active":"2024-04-16T16:13:25.948651+0000","last_became_peered":"2024-04-16T16:13:25.948651+0000","last_unstale":"2024-04-16T16:13:26.961635+0000","last_undegraded":"2024-04-16T16:13:26.961635+0000","last_fullsized":"2024-04-16T16:13:26.961635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955096+0000","last_change":"2024-04-16T16:13:25.945071+0000","last_active":"2024-04-16T16:13:26.955096+0000","last_peered":"2024-04-16T16:13:26.955096+0000","last_clean":"2024-04-16T16:13:26.955096+0000","last_became_active":"2024-04-16T16:13:25.944708+0000","last_became_peered":"2024-04-16T16:13:25.944708+0000","last_unstale":"2024-04-16T16:13:26.955096+0000","last_undegraded":"2024-04-16T16:13:26.955096+0000","last_fullsized":"2024-04-16T16:13:26.955096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.946937+0000","last_change":"2024-04-16T16:13:25.944654+0000","last_active":"2024-04-16T16:13:25.946937+0000","last_peered":"2024-04-16T16:13:25.946937+0000","last_clean":"2024-04-16T16:13:25.946937+0000","last_became_active":"2024-04-16T16:13:25.941914+0000","last_became_peered":"2024-04-16T16:13:25.941914+0000","last_unstale":"2024-04-16T16:13:25.946937+0000","last_undegraded":"2024-04-16T16:13:25.946937+0000","last_fullsized":"2024-04-16T16:13:25.946937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:25.740193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.952512+0000","last_change":"2024-04-16T16:13:25.947081+0000","last_active":"2024-04-16T16:13:25.952512+0000","last_peered":"2024-04-16T16:13:25.952512+0000","last_clean":"2024-04-16T16:13:25.952512+0000","last_became_active":"2024-04-16T16:13:25.946791+0000","last_became_peered":"2024-04-16T16:13:25.946791+0000","last_unstale":"2024-04-16T16:13:25.952512+0000","last_undegraded":"2024-04-16T16:13:25.952512+0000","last_fullsized":"2024-04-16T16:13:25.952512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:25.740193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971641+0000","last_change":"2024-04-16T16:13:25.952506+0000","last_active":"2024-04-16T16:13:26.971641+0000","last_peered":"2024-04-16T16:13:26.971641+0000","last_clean":"2024-04-16T16:13:26.971641+0000","last_became_active":"2024-04-16T16:13:25.952184+0000","last_became_peered":"2024-04-16T16:13:25.952184+0000","last_unstale":"2024-04-16T16:13:26.971641+0000","last_undegraded":"2024-04-16T16:13:26.971641+0000","last_fullsized":"2024-04-16T16:13:26.971641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:19:05.123994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936714+0000","last_change":"2024-04-16T16:13:25.936689+0000","last_active":"2024-04-16T16:13:26.936714+0000","last_peered":"2024-04-16T16:13:26.936714+0000","last_clean":"2024-04-16T16:13:26.936714+0000","last_became_active":"2024-04-16T16:13:25.936385+0000","last_became_peered":"2024-04-16T16:13:25.936385+0000","last_unstale":"2024-04-16T16:13:26.936714+0000","last_undegraded":"2024-04-16T16:13:26.936714+0000","last_fullsized":"2024-04-16T16:13:26.936714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:12.895579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960388+0000","last_change":"2024-04-16T16:13:25.953671+0000","last_active":"2024-04-16T16:13:26.960388+0000","last_peered":"2024-04-16T16:13:26.960388+0000","last_clean":"2024-04-16T16:13:26.960388+0000","last_became_active":"2024-04-16T16:13:25.953325+0000","last_became_peered":"2024-04-16T16:13:25.953325+0000","last_unstale":"2024-04-16T16:13:26.960388+0000","last_undegraded":"2024-04-16T16:13:26.960388+0000","last_fullsized":"2024-04-16T16:13:26.960388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:51:20.451559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941086+0000","last_change":"2024-04-16T16:13:25.936138+0000","last_active":"2024-04-16T16:13:26.941086+0000","last_peered":"2024-04-16T16:13:26.941086+0000","last_clean":"2024-04-16T16:13:26.941086+0000","last_became_active":"2024-04-16T16:13:25.935759+0000","last_became_peered":"2024-04-16T16:13:25.935759+0000","last_unstale":"2024-04-16T16:13:26.941086+0000","last_undegraded":"2024-04-16T16:13:26.941086+0000","last_fullsized":"2024-04-16T16:13:26.941086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:26.052535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.965680+0000","last_change":"2024-04-16T16:13:24.912677+0000","last_active":"2024-04-16T16:13:25.965680+0000","last_peered":"2024-04-16T16:13:25.965680+0000","last_clean":"2024-04-16T16:13:25.965680+0000","last_became_active":"2024-04-16T16:13:24.912364+0000","last_became_peered":"2024-04-16T16:13:24.912364+0000","last_unstale":"2024-04-16T16:13:25.965680+0000","last_undegraded":"2024-04-16T16:13:25.965680+0000","last_fullsized":"2024-04-16T16:13:25.965680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.952533+0000","last_change":"2024-04-16T16:13:25.947853+0000","last_active":"2024-04-16T16:13:25.952533+0000","last_peered":"2024-04-16T16:13:25.952533+0000","last_clean":"2024-04-16T16:13:25.952533+0000","last_became_active":"2024-04-16T16:13:25.947545+0000","last_became_peered":"2024-04-16T16:13:25.947545+0000","last_unstale":"2024-04-16T16:13:25.952533+0000","last_undegraded":"2024-04-16T16:13:25.952533+0000","last_fullsized":"2024-04-16T16:13:25.952533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:42.567277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960762+0000","last_change":"2024-04-16T16:13:24.917069+0000","last_active":"2024-04-16T16:13:26.960762+0000","last_peered":"2024-04-16T16:13:26.960762+0000","last_clean":"2024-04-16T16:13:26.960762+0000","last_became_active":"2024-04-16T16:13:24.916846+0000","last_became_peered":"2024-04-16T16:13:24.916846+0000","last_unstale":"2024-04-16T16:13:26.960762+0000","last_undegraded":"2024-04-16T16:13:26.960762+0000","last_fullsized":"2024-04-16T16:13:26.960762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.935567+0000","last_change":"2024-04-16T16:13:24.919388+0000","last_active":"2024-04-16T16:13:25.935567+0000","last_peered":"2024-04-16T16:13:25.935567+0000","last_clean":"2024-04-16T16:13:25.935567+0000","last_became_active":"2024-04-16T16:13:24.918937+0000","last_became_peered":"2024-04-16T16:13:24.918937+0000","last_unstale":"2024-04-16T16:13:25.935567+0000","last_undegraded":"2024-04-16T16:13:25.935567+0000","last_fullsized":"2024-04-16T16:13:25.935567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:46:08.208026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.955331+0000","last_change":"2024-04-16T16:13:25.954029+0000","last_active":"2024-04-16T16:13:25.955331+0000","last_peered":"2024-04-16T16:13:25.955331+0000","last_clean":"2024-04-16T16:13:25.955331+0000","last_became_active":"2024-04-16T16:13:25.953740+0000","last_became_peered":"2024-04-16T16:13:25.953740+0000","last_unstale":"2024-04-16T16:13:25.955331+0000","last_undegraded":"2024-04-16T16:13:25.955331+0000","last_fullsized":"2024-04-16T16:13:25.955331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:23.603931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971052+0000","last_change":"2024-04-16T16:13:24.923031+0000","last_active":"2024-04-16T16:13:26.971052+0000","last_peered":"2024-04-16T16:13:26.971052+0000","last_clean":"2024-04-16T16:13:26.971052+0000","last_became_active":"2024-04-16T16:13:24.922708+0000","last_became_peered":"2024-04-16T16:13:24.922708+0000","last_unstale":"2024-04-16T16:13:26.971052+0000","last_undegraded":"2024-04-16T16:13:26.971052+0000","last_fullsized":"2024-04-16T16:13:26.971052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954570+0000","last_change":"2024-04-16T16:13:25.949124+0000","last_active":"2024-04-16T16:13:26.954570+0000","last_peered":"2024-04-16T16:13:26.954570+0000","last_clean":"2024-04-16T16:13:26.954570+0000","last_became_active":"2024-04-16T16:13:25.948436+0000","last_became_peered":"2024-04-16T16:13:25.948436+0000","last_unstale":"2024-04-16T16:13:26.954570+0000","last_undegraded":"2024-04-16T16:13:26.954570+0000","last_fullsized":"2024-04-16T16:13:26.954570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:10.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961547+0000","last_change":"2024-04-16T16:13:24.925547+0000","last_active":"2024-04-16T16:13:26.961547+0000","last_peered":"2024-04-16T16:13:26.961547+0000","last_clean":"2024-04-16T16:13:26.961547+0000","last_became_active":"2024-04-16T16:13:24.925182+0000","last_became_peered":"2024-04-16T16:13:24.925182+0000","last_unstale":"2024-04-16T16:13:26.961547+0000","last_undegraded":"2024-04-16T16:13:26.961547+0000","last_fullsized":"2024-04-16T16:13:26.961547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.972351+0000","last_change":"2024-04-16T16:13:25.945204+0000","last_active":"2024-04-16T16:13:26.972351+0000","last_peered":"2024-04-16T16:13:26.972351+0000","last_clean":"2024-04-16T16:13:26.972351+0000","last_became_active":"2024-04-16T16:13:25.944902+0000","last_became_peered":"2024-04-16T16:13:25.944902+0000","last_unstale":"2024-04-16T16:13:26.972351+0000","last_undegraded":"2024-04-16T16:13:26.972351+0000","last_fullsized":"2024-04-16T16:13:26.972351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:52:47.172342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971774+0000","last_change":"2024-04-16T16:13:24.928041+0000","last_active":"2024-04-16T16:13:26.971774+0000","last_peered":"2024-04-16T16:13:26.971774+0000","last_clean":"2024-04-16T16:13:26.971774+0000","last_became_active":"2024-04-16T16:13:24.927555+0000","last_became_peered":"2024-04-16T16:13:24.927555+0000","last_unstale":"2024-04-16T16:13:26.971774+0000","last_undegraded":"2024-04-16T16:13:26.971774+0000","last_fullsized":"2024-04-16T16:13:26.971774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.952119+0000","last_change":"2024-04-16T16:13:25.943385+0000","last_active":"2024-04-16T16:13:25.952119+0000","last_peered":"2024-04-16T16:13:25.952119+0000","last_clean":"2024-04-16T16:13:25.952119+0000","last_became_active":"2024-04-16T16:13:25.943095+0000","last_became_peered":"2024-04-16T16:13:25.943095+0000","last_unstale":"2024-04-16T16:13:25.952119+0000","last_undegraded":"2024-04-16T16:13:25.952119+0000","last_fullsized":"2024-04-16T16:13:25.952119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:00.025740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955094+0000","last_change":"2024-04-16T16:13:24.922351+0000","last_active":"2024-04-16T16:13:26.955094+0000","last_peered":"2024-04-16T16:13:26.955094+0000","last_clean":"2024-04-16T16:13:26.955094+0000","last_became_active":"2024-04-16T16:13:24.922017+0000","last_became_peered":"2024-04-16T16:13:24.922017+0000","last_unstale":"2024-04-16T16:13:26.955094+0000","last_undegraded":"2024-04-16T16:13:26.955094+0000","last_fullsized":"2024-04-16T16:13:26.955094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:24.722256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942286+0000","last_change":"2024-04-16T16:13:25.944764+0000","last_active":"2024-04-16T16:13:26.942286+0000","last_peered":"2024-04-16T16:13:26.942286+0000","last_clean":"2024-04-16T16:13:26.942286+0000","last_became_active":"2024-04-16T16:13:25.944418+0000","last_became_peered":"2024-04-16T16:13:25.944418+0000","last_unstale":"2024-04-16T16:13:26.942286+0000","last_undegraded":"2024-04-16T16:13:26.942286+0000","last_fullsized":"2024-04-16T16:13:26.942286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:25.740193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960752+0000","last_change":"2024-04-16T16:13:24.916122+0000","last_active":"2024-04-16T16:13:26.960752+0000","last_peered":"2024-04-16T16:13:26.960752+0000","last_clean":"2024-04-16T16:13:26.960752+0000","last_became_active":"2024-04-16T16:13:24.915132+0000","last_became_peered":"2024-04-16T16:13:24.915132+0000","last_unstale":"2024-04-16T16:13:26.960752+0000","last_undegraded":"2024-04-16T16:13:26.960752+0000","last_fullsized":"2024-04-16T16:13:26.960752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942858+0000","last_change":"2024-04-16T16:13:25.954813+0000","last_active":"2024-04-16T16:13:26.942858+0000","last_peered":"2024-04-16T16:13:26.942858+0000","last_clean":"2024-04-16T16:13:26.942858+0000","last_became_active":"2024-04-16T16:13:25.952310+0000","last_became_peered":"2024-04-16T16:13:25.952310+0000","last_unstale":"2024-04-16T16:13:26.942858+0000","last_undegraded":"2024-04-16T16:13:26.942858+0000","last_fullsized":"2024-04-16T16:13:26.942858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:13:01.042309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.967305+0000","last_change":"2024-04-16T16:13:24.907364+0000","last_active":"2024-04-16T16:13:25.967305+0000","last_peered":"2024-04-16T16:13:25.967305+0000","last_clean":"2024-04-16T16:13:25.967305+0000","last_became_active":"2024-04-16T16:13:24.906972+0000","last_became_peered":"2024-04-16T16:13:24.906972+0000","last_unstale":"2024-04-16T16:13:25.967305+0000","last_undegraded":"2024-04-16T16:13:25.967305+0000","last_fullsized":"2024-04-16T16:13:25.967305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:22.075493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.951939+0000","last_change":"2024-04-16T16:13:25.946280+0000","last_active":"2024-04-16T16:13:25.951939+0000","last_peered":"2024-04-16T16:13:25.951939+0000","last_clean":"2024-04-16T16:13:25.951939+0000","last_became_active":"2024-04-16T16:13:25.945919+0000","last_became_peered":"2024-04-16T16:13:25.945919+0000","last_unstale":"2024-04-16T16:13:25.951939+0000","last_undegraded":"2024-04-16T16:13:25.951939+0000","last_fullsized":"2024-04-16T16:13:25.951939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:12.895579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971179+0000","last_change":"2024-04-16T16:13:24.921202+0000","last_active":"2024-04-16T16:13:26.971179+0000","last_peered":"2024-04-16T16:13:26.971179+0000","last_clean":"2024-04-16T16:13:26.971179+0000","last_became_active":"2024-04-16T16:13:24.920850+0000","last_became_peered":"2024-04-16T16:13:24.920850+0000","last_unstale":"2024-04-16T16:13:26.971179+0000","last_undegraded":"2024-04-16T16:13:26.971179+0000","last_fullsized":"2024-04-16T16:13:26.971179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942649+0000","last_change":"2024-04-16T16:13:25.947511+0000","last_active":"2024-04-16T16:13:26.942649+0000","last_peered":"2024-04-16T16:13:26.942649+0000","last_clean":"2024-04-16T16:13:26.942649+0000","last_became_active":"2024-04-16T16:13:25.947213+0000","last_became_peered":"2024-04-16T16:13:25.947213+0000","last_unstale":"2024-04-16T16:13:26.942649+0000","last_undegraded":"2024-04-16T16:13:26.942649+0000","last_fullsized":"2024-04-16T16:13:26.942649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:47.987635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960500+0000","last_change":"2024-04-16T16:13:24.923552+0000","last_active":"2024-04-16T16:13:26.960500+0000","last_peered":"2024-04-16T16:13:26.960500+0000","last_clean":"2024-04-16T16:13:26.960500+0000","last_became_active":"2024-04-16T16:13:24.922940+0000","last_became_peered":"2024-04-16T16:13:24.922940+0000","last_unstale":"2024-04-16T16:13:26.960500+0000","last_undegraded":"2024-04-16T16:13:26.960500+0000","last_fullsized":"2024-04-16T16:13:26.960500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960474+0000","last_change":"2024-04-16T16:13:25.954523+0000","last_active":"2024-04-16T16:13:26.960474+0000","last_peered":"2024-04-16T16:13:26.960474+0000","last_clean":"2024-04-16T16:13:26.960474+0000","last_became_active":"2024-04-16T16:13:25.954058+0000","last_became_peered":"2024-04-16T16:13:25.954058+0000","last_unstale":"2024-04-16T16:13:26.960474+0000","last_undegraded":"2024-04-16T16:13:26.960474+0000","last_fullsized":"2024-04-16T16:13:26.960474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:11:32.536402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936502+0000","last_change":"2024-04-16T16:13:24.904943+0000","last_active":"2024-04-16T16:13:26.936502+0000","last_peered":"2024-04-16T16:13:26.936502+0000","last_clean":"2024-04-16T16:13:26.936502+0000","last_became_active":"2024-04-16T16:13:24.904527+0000","last_became_peered":"2024-04-16T16:13:24.904527+0000","last_unstale":"2024-04-16T16:13:26.936502+0000","last_undegraded":"2024-04-16T16:13:26.936502+0000","last_fullsized":"2024-04-16T16:13:26.936502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:22.075493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942098+0000","last_change":"2024-04-16T16:13:25.937269+0000","last_active":"2024-04-16T16:13:26.942098+0000","last_peered":"2024-04-16T16:13:26.942098+0000","last_clean":"2024-04-16T16:13:26.942098+0000","last_became_active":"2024-04-16T16:13:25.936953+0000","last_became_peered":"2024-04-16T16:13:25.936953+0000","last_unstale":"2024-04-16T16:13:26.942098+0000","last_undegraded":"2024-04-16T16:13:26.942098+0000","last_fullsized":"2024-04-16T16:13:26.942098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:41.243756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941662+0000","last_change":"2024-04-16T16:13:24.909436+0000","last_active":"2024-04-16T16:13:26.941662+0000","last_peered":"2024-04-16T16:13:26.941662+0000","last_clean":"2024-04-16T16:13:26.941662+0000","last_became_active":"2024-04-16T16:13:24.908963+0000","last_became_peered":"2024-04-16T16:13:24.908963+0000","last_unstale":"2024-04-16T16:13:26.941662+0000","last_undegraded":"2024-04-16T16:13:26.941662+0000","last_fullsized":"2024-04-16T16:13:26.941662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:21.269919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943634+0000","last_change":"2024-04-16T16:13:25.939892+0000","last_active":"2024-04-16T16:13:26.943634+0000","last_peered":"2024-04-16T16:13:26.943634+0000","last_clean":"2024-04-16T16:13:26.943634+0000","last_became_active":"2024-04-16T16:13:25.939610+0000","last_became_peered":"2024-04-16T16:13:25.939610+0000","last_unstale":"2024-04-16T16:13:26.943634+0000","last_undegraded":"2024-04-16T16:13:26.943634+0000","last_fullsized":"2024-04-16T16:13:26.943634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:58.462911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.937081+0000","last_change":"2024-04-16T16:13:24.898787+0000","last_active":"2024-04-16T16:13:26.937081+0000","last_peered":"2024-04-16T16:13:26.937081+0000","last_clean":"2024-04-16T16:13:26.937081+0000","last_became_active":"2024-04-16T16:13:24.898427+0000","last_became_peered":"2024-04-16T16:13:24.898427+0000","last_unstale":"2024-04-16T16:13:26.937081+0000","last_undegraded":"2024-04-16T16:13:26.937081+0000","last_fullsized":"2024-04-16T16:13:26.937081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943828+0000","last_change":"2024-04-16T16:13:25.953661+0000","last_active":"2024-04-16T16:13:26.943828+0000","last_peered":"2024-04-16T16:13:26.943828+0000","last_clean":"2024-04-16T16:13:26.943828+0000","last_became_active":"2024-04-16T16:13:25.950243+0000","last_became_peered":"2024-04-16T16:13:25.950243+0000","last_unstale":"2024-04-16T16:13:26.943828+0000","last_undegraded":"2024-04-16T16:13:26.943828+0000","last_fullsized":"2024-04-16T16:13:26.943828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:08.801668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955224+0000","last_change":"2024-04-16T16:13:24.906785+0000","last_active":"2024-04-16T16:13:26.955224+0000","last_peered":"2024-04-16T16:13:26.955224+0000","last_clean":"2024-04-16T16:13:26.955224+0000","last_became_active":"2024-04-16T16:13:24.906478+0000","last_became_peered":"2024-04-16T16:13:24.906478+0000","last_unstale":"2024-04-16T16:13:26.955224+0000","last_undegraded":"2024-04-16T16:13:26.955224+0000","last_fullsized":"2024-04-16T16:13:26.955224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:41.549340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971400+0000","last_change":"2024-04-16T16:13:25.951170+0000","last_active":"2024-04-16T16:13:26.971400+0000","last_peered":"2024-04-16T16:13:26.971400+0000","last_clean":"2024-04-16T16:13:26.971400+0000","last_became_active":"2024-04-16T16:13:25.950423+0000","last_became_peered":"2024-04-16T16:13:25.950423+0000","last_unstale":"2024-04-16T16:13:26.971400+0000","last_undegraded":"2024-04-16T16:13:26.971400+0000","last_fullsized":"2024-04-16T16:13:26.971400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:13:01.042309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936858+0000","last_change":"2024-04-16T16:13:24.920097+0000","last_active":"2024-04-16T16:13:26.936858+0000","last_peered":"2024-04-16T16:13:26.936858+0000","last_clean":"2024-04-16T16:13:26.936858+0000","last_became_active":"2024-04-16T16:13:24.919848+0000","last_became_peered":"2024-04-16T16:13:24.919848+0000","last_unstale":"2024-04-16T16:13:26.936858+0000","last_undegraded":"2024-04-16T16:13:26.936858+0000","last_fullsized":"2024-04-16T16:13:26.936858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:46:08.208026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942819+0000","last_change":"2024-04-16T16:13:25.952599+0000","last_active":"2024-04-16T16:13:26.942819+0000","last_peered":"2024-04-16T16:13:26.942819+0000","last_clean":"2024-04-16T16:13:26.942819+0000","last_became_active":"2024-04-16T16:13:25.949217+0000","last_became_peered":"2024-04-16T16:13:25.949217+0000","last_unstale":"2024-04-16T16:13:26.942819+0000","last_undegraded":"2024-04-16T16:13:26.942819+0000","last_fullsized":"2024-04-16T16:13:26.942819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:39.592952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.940554+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:13:26.940554+0000","last_peered":"2024-04-16T16:13:26.940554+0000","last_clean":"2024-04-16T16:13:26.940554+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:13:26.940554+0000","last_undegraded":"2024-04-16T16:13:26.940554+0000","last_fullsized":"2024-04-16T16:13:26.940554+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970651+0000","last_change":"2024-04-16T16:13:24.919303+0000","last_active":"2024-04-16T16:13:26.970651+0000","last_peered":"2024-04-16T16:13:26.970651+0000","last_clean":"2024-04-16T16:13:26.970651+0000","last_became_active":"2024-04-16T16:13:24.918056+0000","last_became_peered":"2024-04-16T16:13:24.918056+0000","last_unstale":"2024-04-16T16:13:26.970651+0000","last_undegraded":"2024-04-16T16:13:26.970651+0000","last_fullsized":"2024-04-16T16:13:26.970651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.951015+0000","last_change":"2024-04-16T16:13:25.947227+0000","last_active":"2024-04-16T16:13:25.951015+0000","last_peered":"2024-04-16T16:13:25.951015+0000","last_clean":"2024-04-16T16:13:25.951015+0000","last_became_active":"2024-04-16T16:13:25.946951+0000","last_became_peered":"2024-04-16T16:13:25.946951+0000","last_unstale":"2024-04-16T16:13:25.951015+0000","last_undegraded":"2024-04-16T16:13:25.951015+0000","last_fullsized":"2024-04-16T16:13:25.951015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:02:29.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.940404+0000","last_change":"2024-04-16T16:13:24.912738+0000","last_active":"2024-04-16T16:13:26.940404+0000","last_peered":"2024-04-16T16:13:26.940404+0000","last_clean":"2024-04-16T16:13:26.940404+0000","last_became_active":"2024-04-16T16:13:24.912432+0000","last_became_peered":"2024-04-16T16:13:24.912432+0000","last_unstale":"2024-04-16T16:13:26.940404+0000","last_undegraded":"2024-04-16T16:13:26.940404+0000","last_fullsized":"2024-04-16T16:13:26.940404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.962470+0000","last_change":"2024-04-16T16:13:25.952202+0000","last_active":"2024-04-16T16:13:25.962470+0000","last_peered":"2024-04-16T16:13:25.962470+0000","last_clean":"2024-04-16T16:13:25.962470+0000","last_became_active":"2024-04-16T16:13:25.951932+0000","last_became_peered":"2024-04-16T16:13:25.951932+0000","last_unstale":"2024-04-16T16:13:25.962470+0000","last_undegraded":"2024-04-16T16:13:25.962470+0000","last_fullsized":"2024-04-16T16:13:25.962470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:02:29.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936531+0000","last_change":"2024-04-16T16:13:24.901093+0000","last_active":"2024-04-16T16:13:26.936531+0000","last_peered":"2024-04-16T16:13:26.936531+0000","last_clean":"2024-04-16T16:13:26.936531+0000","last_became_active":"2024-04-16T16:13:24.900772+0000","last_became_peered":"2024-04-16T16:13:24.900772+0000","last_unstale":"2024-04-16T16:13:26.936531+0000","last_undegraded":"2024-04-16T16:13:26.936531+0000","last_fullsized":"2024-04-16T16:13:26.936531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954397+0000","last_change":"2024-04-16T16:13:25.929184+0000","last_active":"2024-04-16T16:13:26.954397+0000","last_peered":"2024-04-16T16:13:26.954397+0000","last_clean":"2024-04-16T16:13:26.954397+0000","last_became_active":"2024-04-16T16:13:25.927325+0000","last_became_peered":"2024-04-16T16:13:25.927325+0000","last_unstale":"2024-04-16T16:13:26.954397+0000","last_undegraded":"2024-04-16T16:13:26.954397+0000","last_fullsized":"2024-04-16T16:13:26.954397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:11:32.536402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971426+0000","last_change":"2024-04-16T16:13:24.923465+0000","last_active":"2024-04-16T16:13:26.971426+0000","last_peered":"2024-04-16T16:13:26.971426+0000","last_clean":"2024-04-16T16:13:26.971426+0000","last_became_active":"2024-04-16T16:13:24.923145+0000","last_became_peered":"2024-04-16T16:13:24.923145+0000","last_unstale":"2024-04-16T16:13:26.971426+0000","last_undegraded":"2024-04-16T16:13:26.971426+0000","last_fullsized":"2024-04-16T16:13:26.971426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941524+0000","last_change":"2024-04-16T16:13:25.948469+0000","last_active":"2024-04-16T16:13:26.941524+0000","last_peered":"2024-04-16T16:13:26.941524+0000","last_clean":"2024-04-16T16:13:26.941524+0000","last_became_active":"2024-04-16T16:13:25.947992+0000","last_became_peered":"2024-04-16T16:13:25.947992+0000","last_unstale":"2024-04-16T16:13:26.941524+0000","last_undegraded":"2024-04-16T16:13:26.941524+0000","last_fullsized":"2024-04-16T16:13:26.941524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:00.025740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954768+0000","last_change":"2024-04-16T16:13:24.905663+0000","last_active":"2024-04-16T16:13:26.954768+0000","last_peered":"2024-04-16T16:13:26.954768+0000","last_clean":"2024-04-16T16:13:26.954768+0000","last_became_active":"2024-04-16T16:13:24.905306+0000","last_became_peered":"2024-04-16T16:13:24.905306+0000","last_unstale":"2024-04-16T16:13:26.954768+0000","last_undegraded":"2024-04-16T16:13:26.954768+0000","last_fullsized":"2024-04-16T16:13:26.954768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971908+0000","last_change":"2024-04-16T16:13:25.955137+0000","last_active":"2024-04-16T16:13:26.971908+0000","last_peered":"2024-04-16T16:13:26.971908+0000","last_clean":"2024-04-16T16:13:26.971908+0000","last_became_active":"2024-04-16T16:13:25.954852+0000","last_became_peered":"2024-04-16T16:13:25.954852+0000","last_unstale":"2024-04-16T16:13:26.971908+0000","last_undegraded":"2024-04-16T16:13:26.971908+0000","last_fullsized":"2024-04-16T16:13:26.971908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:58.462911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943455+0000","last_change":"2024-04-16T16:13:24.915400+0000","last_active":"2024-04-16T16:13:26.943455+0000","last_peered":"2024-04-16T16:13:26.943455+0000","last_clean":"2024-04-16T16:13:26.943455+0000","last_became_active":"2024-04-16T16:13:24.915096+0000","last_became_peered":"2024-04-16T16:13:24.915096+0000","last_unstale":"2024-04-16T16:13:26.943455+0000","last_undegraded":"2024-04-16T16:13:26.943455+0000","last_fullsized":"2024-04-16T16:13:26.943455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:41.549340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954886+0000","last_change":"2024-04-16T16:13:25.924316+0000","last_active":"2024-04-16T16:13:26.954886+0000","last_peered":"2024-04-16T16:13:26.954886+0000","last_clean":"2024-04-16T16:13:26.954886+0000","last_became_active":"2024-04-16T16:13:25.923986+0000","last_became_peered":"2024-04-16T16:13:25.923986+0000","last_unstale":"2024-04-16T16:13:26.954886+0000","last_undegraded":"2024-04-16T16:13:26.954886+0000","last_fullsized":"2024-04-16T16:13:26.954886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:40:58.670474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.958843+0000","last_change":"2024-04-16T16:13:24.920876+0000","last_active":"2024-04-16T16:13:26.958843+0000","last_peered":"2024-04-16T16:13:26.958843+0000","last_clean":"2024-04-16T16:13:26.958843+0000","last_became_active":"2024-04-16T16:13:24.920409+0000","last_became_peered":"2024-04-16T16:13:24.920409+0000","last_unstale":"2024-04-16T16:13:26.958843+0000","last_undegraded":"2024-04-16T16:13:26.958843+0000","last_fullsized":"2024-04-16T16:13:26.958843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.943879+0000","last_change":"2024-04-16T16:13:25.941909+0000","last_active":"2024-04-16T16:13:25.943879+0000","last_peered":"2024-04-16T16:13:25.943879+0000","last_clean":"2024-04-16T16:13:25.943879+0000","last_became_active":"2024-04-16T16:13:25.941614+0000","last_became_peered":"2024-04-16T16:13:25.941614+0000","last_unstale":"2024-04-16T16:13:25.943879+0000","last_undegraded":"2024-04-16T16:13:25.943879+0000","last_fullsized":"2024-04-16T16:13:25.943879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:00.025740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961615+0000","last_change":"2024-04-16T16:13:24.918600+0000","last_active":"2024-04-16T16:13:26.961615+0000","last_peered":"2024-04-16T16:13:26.961615+0000","last_clean":"2024-04-16T16:13:26.961615+0000","last_became_active":"2024-04-16T16:13:24.918273+0000","last_became_peered":"2024-04-16T16:13:24.918273+0000","last_unstale":"2024-04-16T16:13:26.961615+0000","last_undegraded":"2024-04-16T16:13:26.961615+0000","last_fullsized":"2024-04-16T16:13:26.961615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.954467+0000","last_change":"2024-04-16T16:13:25.937881+0000","last_active":"2024-04-16T16:13:25.954467+0000","last_peered":"2024-04-16T16:13:25.954467+0000","last_clean":"2024-04-16T16:13:25.954467+0000","last_became_active":"2024-04-16T16:13:25.936555+0000","last_became_peered":"2024-04-16T16:13:25.936555+0000","last_unstale":"2024-04-16T16:13:25.954467+0000","last_undegraded":"2024-04-16T16:13:25.954467+0000","last_fullsized":"2024-04-16T16:13:25.954467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:45.411606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.940400+0000","last_change":"2024-04-16T16:13:24.915314+0000","last_active":"2024-04-16T16:13:26.940400+0000","last_peered":"2024-04-16T16:13:26.940400+0000","last_clean":"2024-04-16T16:13:26.940400+0000","last_became_active":"2024-04-16T16:13:24.913831+0000","last_became_peered":"2024-04-16T16:13:24.913831+0000","last_unstale":"2024-04-16T16:13:26.940400+0000","last_undegraded":"2024-04-16T16:13:26.940400+0000","last_fullsized":"2024-04-16T16:13:26.940400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941863+0000","last_change":"2024-04-16T16:13:25.939017+0000","last_active":"2024-04-16T16:13:26.941863+0000","last_peered":"2024-04-16T16:13:26.941863+0000","last_clean":"2024-04-16T16:13:26.941863+0000","last_became_active":"2024-04-16T16:13:25.936578+0000","last_became_peered":"2024-04-16T16:13:25.936578+0000","last_unstale":"2024-04-16T16:13:26.941863+0000","last_undegraded":"2024-04-16T16:13:26.941863+0000","last_fullsized":"2024-04-16T16:13:26.941863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:12.895579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954750+0000","last_change":"2024-04-16T16:13:24.909354+0000","last_active":"2024-04-16T16:13:26.954750+0000","last_peered":"2024-04-16T16:13:26.954750+0000","last_clean":"2024-04-16T16:13:26.954750+0000","last_became_active":"2024-04-16T16:13:24.909073+0000","last_became_peered":"2024-04-16T16:13:24.909073+0000","last_unstale":"2024-04-16T16:13:26.954750+0000","last_undegraded":"2024-04-16T16:13:26.954750+0000","last_fullsized":"2024-04-16T16:13:26.954750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.943947+0000","last_change":"2024-04-16T16:13:25.940225+0000","last_active":"2024-04-16T16:13:25.943947+0000","last_peered":"2024-04-16T16:13:25.943947+0000","last_clean":"2024-04-16T16:13:25.943947+0000","last_became_active":"2024-04-16T16:13:25.939871+0000","last_became_peered":"2024-04-16T16:13:25.939871+0000","last_unstale":"2024-04-16T16:13:25.943947+0000","last_undegraded":"2024-04-16T16:13:25.943947+0000","last_fullsized":"2024-04-16T16:13:25.943947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:26.052535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954700+0000","last_change":"2024-04-16T16:13:24.907229+0000","last_active":"2024-04-16T16:13:26.954700+0000","last_peered":"2024-04-16T16:13:26.954700+0000","last_clean":"2024-04-16T16:13:26.954700+0000","last_became_active":"2024-04-16T16:13:24.906948+0000","last_became_peered":"2024-04-16T16:13:24.906948+0000","last_unstale":"2024-04-16T16:13:26.954700+0000","last_undegraded":"2024-04-16T16:13:26.954700+0000","last_fullsized":"2024-04-16T16:13:26.954700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:11.877642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.965771+0000","last_change":"2024-04-16T16:13:25.963968+0000","last_active":"2024-04-16T16:13:25.965771+0000","last_peered":"2024-04-16T16:13:25.965771+0000","last_clean":"2024-04-16T16:13:25.965771+0000","last_became_active":"2024-04-16T16:13:25.963699+0000","last_became_peered":"2024-04-16T16:13:25.963699+0000","last_unstale":"2024-04-16T16:13:25.965771+0000","last_undegraded":"2024-04-16T16:13:25.965771+0000","last_fullsized":"2024-04-16T16:13:25.965771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:07.860908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942898+0000","last_change":"2024-04-16T16:13:24.924622+0000","last_active":"2024-04-16T16:13:26.942898+0000","last_peered":"2024-04-16T16:13:26.942898+0000","last_clean":"2024-04-16T16:13:26.942898+0000","last_became_active":"2024-04-16T16:13:24.923313+0000","last_became_peered":"2024-04-16T16:13:24.923313+0000","last_unstale":"2024-04-16T16:13:26.942898+0000","last_undegraded":"2024-04-16T16:13:26.942898+0000","last_fullsized":"2024-04-16T16:13:26.942898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960479+0000","last_change":"2024-04-16T16:13:25.938982+0000","last_active":"2024-04-16T16:13:26.960479+0000","last_peered":"2024-04-16T16:13:26.960479+0000","last_clean":"2024-04-16T16:13:26.960479+0000","last_became_active":"2024-04-16T16:13:25.938694+0000","last_became_peered":"2024-04-16T16:13:25.938694+0000","last_unstale":"2024-04-16T16:13:26.960479+0000","last_undegraded":"2024-04-16T16:13:26.960479+0000","last_fullsized":"2024-04-16T16:13:26.960479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:10.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971847+0000","last_change":"2024-04-16T16:13:25.944450+0000","last_active":"2024-04-16T16:13:26.971847+0000","last_peered":"2024-04-16T16:13:26.971847+0000","last_clean":"2024-04-16T16:13:26.971847+0000","last_became_active":"2024-04-16T16:13:25.941384+0000","last_became_peered":"2024-04-16T16:13:25.941384+0000","last_unstale":"2024-04-16T16:13:26.971847+0000","last_undegraded":"2024-04-16T16:13:26.971847+0000","last_fullsized":"2024-04-16T16:13:26.971847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:40:10.575471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942812+0000","last_change":"2024-04-16T16:13:24.925176+0000","last_active":"2024-04-16T16:13:26.942812+0000","last_peered":"2024-04-16T16:13:26.942812+0000","last_clean":"2024-04-16T16:13:26.942812+0000","last_became_active":"2024-04-16T16:13:24.924863+0000","last_became_peered":"2024-04-16T16:13:24.924863+0000","last_unstale":"2024-04-16T16:13:26.942812+0000","last_undegraded":"2024-04-16T16:13:26.942812+0000","last_fullsized":"2024-04-16T16:13:26.942812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:11.877642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971280+0000","last_change":"2024-04-16T16:13:25.944098+0000","last_active":"2024-04-16T16:13:26.971280+0000","last_peered":"2024-04-16T16:13:26.971280+0000","last_clean":"2024-04-16T16:13:26.971280+0000","last_became_active":"2024-04-16T16:13:25.943843+0000","last_became_peered":"2024-04-16T16:13:25.943843+0000","last_unstale":"2024-04-16T16:13:26.971280+0000","last_undegraded":"2024-04-16T16:13:26.971280+0000","last_fullsized":"2024-04-16T16:13:26.971280+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:39.592952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943140+0000","last_change":"2024-04-16T16:13:24.923704+0000","last_active":"2024-04-16T16:13:26.943140+0000","last_peered":"2024-04-16T16:13:26.943140+0000","last_clean":"2024-04-16T16:13:26.943140+0000","last_became_active":"2024-04-16T16:13:24.923214+0000","last_became_peered":"2024-04-16T16:13:24.923214+0000","last_unstale":"2024-04-16T16:13:26.943140+0000","last_undegraded":"2024-04-16T16:13:26.943140+0000","last_fullsized":"2024-04-16T16:13:26.943140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:24.722256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.966716+0000","last_change":"2024-04-16T16:13:25.955067+0000","last_active":"2024-04-16T16:13:25.966716+0000","last_peered":"2024-04-16T16:13:25.966716+0000","last_clean":"2024-04-16T16:13:25.966716+0000","last_became_active":"2024-04-16T16:13:25.954780+0000","last_became_peered":"2024-04-16T16:13:25.954780+0000","last_unstale":"2024-04-16T16:13:25.966716+0000","last_undegraded":"2024-04-16T16:13:25.966716+0000","last_fullsized":"2024-04-16T16:13:25.966716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:32:38.024739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.967682+0000","last_change":"2024-04-16T16:13:24.913036+0000","last_active":"2024-04-16T16:13:25.967682+0000","last_peered":"2024-04-16T16:13:25.967682+0000","last_clean":"2024-04-16T16:13:25.967682+0000","last_became_active":"2024-04-16T16:13:24.911928+0000","last_became_peered":"2024-04-16T16:13:24.911928+0000","last_unstale":"2024-04-16T16:13:25.967682+0000","last_undegraded":"2024-04-16T16:13:25.967682+0000","last_fullsized":"2024-04-16T16:13:25.967682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.950077+0000","last_change":"2024-04-16T16:13:25.946331+0000","last_active":"2024-04-16T16:13:25.950077+0000","last_peered":"2024-04-16T16:13:25.950077+0000","last_clean":"2024-04-16T16:13:25.950077+0000","last_became_active":"2024-04-16T16:13:25.946053+0000","last_became_peered":"2024-04-16T16:13:25.946053+0000","last_unstale":"2024-04-16T16:13:25.950077+0000","last_undegraded":"2024-04-16T16:13:25.950077+0000","last_fullsized":"2024-04-16T16:13:25.950077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:42.567277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.967407+0000","last_change":"2024-04-16T16:13:24.909349+0000","last_active":"2024-04-16T16:13:25.967407+0000","last_peered":"2024-04-16T16:13:25.967407+0000","last_clean":"2024-04-16T16:13:25.967407+0000","last_became_active":"2024-04-16T16:13:24.909069+0000","last_became_peered":"2024-04-16T16:13:24.909069+0000","last_unstale":"2024-04-16T16:13:25.967407+0000","last_undegraded":"2024-04-16T16:13:25.967407+0000","last_fullsized":"2024-04-16T16:13:25.967407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:46:08.208026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":811008,"data_stored":660311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:13:30.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:13:30.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.721+0000 7fabb9e00640 1 Processor -- start 2024-04-16T16:13:30.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.721+0000 7fabb9e00640 1 -- start start 2024-04-16T16:13:30.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.723+0000 7fabb9e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fabb415ccb0 0x7fabb406df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:30.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.723+0000 7fabb9e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabb406e480 0x7fabb41608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:30.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.723+0000 7fabb9e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb415d360 con 0x7fabb4146220 2024-04-16T16:13:30.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.723+0000 7fabb9e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fabb415d130 con 0x7fabb415ccb0 2024-04-16T16:13:30.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.723+0000 7fabb9e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fabb4160410 con 0x7fabb406e480 2024-04-16T16:13:30.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.723+0000 7fabb3e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fabb415ccb0 0x7fabb406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:30.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.723+0000 7fabb3e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fabb415ccb0 0x7fabb406df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49684/0 (socket says 172.21.15.45:49684) 2024-04-16T16:13:30.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.723+0000 7fabb3e00640 1 -- 172.21.15.45:0/3024819042 learned_addr learned my addr 172.21.15.45:0/3024819042 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:30.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.724+0000 7fabb9400640 1 --2- 172.21.15.45:0/3024819042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabb406e480 0x7fabb41608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:30.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.724+0000 7fabb3e00640 1 -- 172.21.15.45:0/3024819042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabb406e480 msgr2=0x7fabb41608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:30.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.724+0000 7fabb3e00640 1 --2- 172.21.15.45:0/3024819042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabb406e480 0x7fabb41608a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.724+0000 7fabb3e00640 1 -- 172.21.15.45:0/3024819042 >> v1:172.21.15.103:6789/0 conn(0x7fabb4146220 legacy=0x7fabb406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:30.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.724+0000 7fabb3e00640 1 -- 172.21.15.45:0/3024819042 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb415d640 con 0x7fabb415ccb0 2024-04-16T16:13:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.725+0000 7fabb3e00640 1 --2- 172.21.15.45:0/3024819042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb415ccb0 0x7fabb406df40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faba0009870 tx=0x7faba002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11425cc472a7b1a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.725+0000 7fabb3400640 1 -- 172.21.15.45:0/3024819042 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0035020 con 0x7fabb415ccb0 2024-04-16T16:13:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.725+0000 7fabb3400640 1 -- 172.21.15.45:0/3024819042 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba0031640 con 0x7fabb415ccb0 2024-04-16T16:13:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.725+0000 7fabb3400640 1 -- 172.21.15.45:0/3024819042 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba002d040 con 0x7fabb415ccb0 2024-04-16T16:13:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.726+0000 7fabb9e00640 1 -- 172.21.15.45:0/3024819042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb415ccb0 msgr2=0x7fabb406df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.726+0000 7fabb9e00640 1 --2- 172.21.15.45:0/3024819042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb415ccb0 0x7fabb406df40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faba0009870 tx=0x7faba002b8f0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.726+0000 7fabb9e00640 1 -- 172.21.15.45:0/3024819042 shutdown_connections 2024-04-16T16:13:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.726+0000 7fabb9e00640 1 --2- 172.21.15.45:0/3024819042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabb406e480 0x7fabb41608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.726+0000 7fabb9e00640 1 --2- 172.21.15.45:0/3024819042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb415ccb0 0x7fabb406df40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.726+0000 7fabb9e00640 1 -- 172.21.15.45:0/3024819042 >> 172.21.15.45:0/3024819042 conn(0x7fabb406b930 msgr2=0x7fabb406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.726+0000 7fabb9e00640 1 -- 172.21.15.45:0/3024819042 shutdown_connections 2024-04-16T16:13:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.726+0000 7fabb9e00640 1 -- 172.21.15.45:0/3024819042 wait complete. 2024-04-16T16:13:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.728+0000 7fabb9e00640 1 Processor -- start 2024-04-16T16:13:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.728+0000 7fabb9e00640 1 -- start start 2024-04-16T16:13:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.729+0000 7fabb9e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabb406e480 0x7fabb413d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.730+0000 7fabb9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabb4146220 0x7fabb41416f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.730+0000 7fabb9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb415ccb0 0x7fabb415aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.730+0000 7fabb9e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fabb4160de0 con 0x7fabb415ccb0 2024-04-16T16:13:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.730+0000 7fabb9e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fabb415dc00 con 0x7fabb4146220 2024-04-16T16:13:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.730+0000 7fabb9e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fabb4160f60 con 0x7fabb406e480 2024-04-16T16:13:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.730+0000 7fabb9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb415ccb0 0x7fabb415aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.730+0000 7fabb9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb415ccb0 0x7fabb415aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49692/0 (socket says 172.21.15.45:49692) 2024-04-16T16:13:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.730+0000 7fabb9400640 1 -- 172.21.15.45:0/1122799993 learned_addr learned my addr 172.21.15.45:0/1122799993 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:30.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.730+0000 7fabb3e00640 1 --2- 172.21.15.45:0/1122799993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabb4146220 0x7fabb41416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:30.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.730+0000 7fabb8a00640 1 --2- 172.21.15.45:0/1122799993 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabb406e480 0x7fabb413d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:30.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.731+0000 7fabb9400640 1 -- 172.21.15.45:0/1122799993 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabb406e480 msgr2=0x7fabb413d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:30.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.731+0000 7fabb9400640 1 --2- 172.21.15.45:0/1122799993 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabb406e480 0x7fabb413d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.731+0000 7fabb9400640 1 -- 172.21.15.45:0/1122799993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabb4146220 msgr2=0x7fabb41416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:30.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.731+0000 7fabb9400640 1 --2- 172.21.15.45:0/1122799993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabb4146220 0x7fabb41416f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.731+0000 7fabb9400640 1 -- 172.21.15.45:0/1122799993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb413d870 con 0x7fabb415ccb0 2024-04-16T16:13:30.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.731+0000 7fabb3e00640 1 --2- 172.21.15.45:0/1122799993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabb4146220 0x7fabb41416f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:13:30.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.731+0000 7fabb9400640 1 --2- 172.21.15.45:0/1122799993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb415ccb0 0x7fabb415aa60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fabac00cdc0 tx=0x7fabac00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:30.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.731+0000 7fabb0c00640 1 -- 172.21.15.45:0/1122799993 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabac003b90 con 0x7fabb415ccb0 2024-04-16T16:13:30.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.731+0000 7fabb0c00640 1 -- 172.21.15.45:0/1122799993 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabac003d50 con 0x7fabb415ccb0 2024-04-16T16:13:30.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.731+0000 7fabb0c00640 1 -- 172.21.15.45:0/1122799993 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabac004260 con 0x7fabb415ccb0 2024-04-16T16:13:30.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.731+0000 7fabb9e00640 1 -- 172.21.15.45:0/1122799993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabb413db50 con 0x7fabb415ccb0 2024-04-16T16:13:30.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.731+0000 7fabb9e00640 1 -- 172.21.15.45:0/1122799993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabb413e110 con 0x7fabb415ccb0 2024-04-16T16:13:30.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.733+0000 7fabb9e00640 1 -- 172.21.15.45:0/1122799993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabb4075780 con 0x7fabb415ccb0 2024-04-16T16:13:30.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.733+0000 7fabb0c00640 1 -- 172.21.15.45:0/1122799993 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fabac004400 con 0x7fabb415ccb0 2024-04-16T16:13:30.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.735+0000 7fabb0c00640 1 --2- 172.21.15.45:0/1122799993 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fab9406c8f0 0x7fab9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:30.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.735+0000 7fabb0c00640 1 -- 172.21.15.45:0/1122799993 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fabac08c600 con 0x7fabb415ccb0 2024-04-16T16:13:30.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.735+0000 7fabb8a00640 1 --2- 172.21.15.45:0/1122799993 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fab9406c8f0 0x7fab9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:30.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.737+0000 7fabb8a00640 1 --2- 172.21.15.45:0/1122799993 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fab9406c8f0 0x7fab9406edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faba8007fd0 tx=0x7faba8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:30.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.740+0000 7fabb0c00640 1 -- 172.21.15.45:0/1122799993 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fabac01d340 con 0x7fabb415ccb0 2024-04-16T16:13:30.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.941+0000 7fabb9e00640 1 -- 172.21.15.45:0/1122799993 --> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fabb40cff60 con 0x7fab9406c8f0 2024-04-16T16:13:30.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.961+0000 7fabb0c00640 1 -- 172.21.15.45:0/1122799993 <== mgr.4105 v2:172.21.15.103:6832/2711977587 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251928 (secure 0 0 0) 0x7fabb40cff60 con 0x7fab9406c8f0 2024-04-16T16:13:30.965 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:30.966 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:13:30.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.967+0000 7fabb9e00640 1 -- 172.21.15.45:0/1122799993 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fab9406c8f0 msgr2=0x7fab9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:30.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.967+0000 7fabb9e00640 1 --2- 172.21.15.45:0/1122799993 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fab9406c8f0 0x7fab9406edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faba8007fd0 tx=0x7faba8009040 comp rx=0 tx=0).stop 2024-04-16T16:13:30.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.967+0000 7fabb9e00640 1 -- 172.21.15.45:0/1122799993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb415ccb0 msgr2=0x7fabb415aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:30.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.967+0000 7fabb9e00640 1 --2- 172.21.15.45:0/1122799993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb415ccb0 0x7fabb415aa60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fabac00cdc0 tx=0x7fabac00a660 comp rx=0 tx=0).stop 2024-04-16T16:13:30.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.967+0000 7fabb9e00640 1 -- 172.21.15.45:0/1122799993 shutdown_connections 2024-04-16T16:13:30.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.967+0000 7fabb9e00640 1 --2- 172.21.15.45:0/1122799993 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fab9406c8f0 0x7fab9406edb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.967+0000 7fabb9e00640 1 --2- 172.21.15.45:0/1122799993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb415ccb0 0x7fabb415aa60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.967+0000 7fabb9e00640 1 --2- 172.21.15.45:0/1122799993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabb4146220 0x7fabb41416f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.967+0000 7fabb9e00640 1 --2- 172.21.15.45:0/1122799993 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabb406e480 0x7fabb413d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:30.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.967+0000 7fabb9e00640 1 -- 172.21.15.45:0/1122799993 >> 172.21.15.45:0/1122799993 conn(0x7fabb406b930 msgr2=0x7fabb413f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:30.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.967+0000 7fabb9e00640 1 -- 172.21.15.45:0/1122799993 shutdown_connections 2024-04-16T16:13:30.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:30.967+0000 7fabb9e00640 1 -- 172.21.15.45:0/1122799993 wait complete. 2024-04-16T16:13:30.986 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-04-16T16:13:29.092606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":135,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235740,"kb_used_data":2896,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738980,"statfs":{"total":773094113280,"available":772852715520,"internally_reserved":0,"allocated":2965504,"data_stored":1777796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.220988"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943618+0000","last_change":"2024-04-16T16:13:25.934978+0000","last_active":"2024-04-16T16:13:26.943618+0000","last_peered":"2024-04-16T16:13:26.943618+0000","last_clean":"2024-04-16T16:13:26.943618+0000","last_became_active":"2024-04-16T16:13:25.934469+0000","last_became_peered":"2024-04-16T16:13:25.934469+0000","last_unstale":"2024-04-16T16:13:26.943618+0000","last_undegraded":"2024-04-16T16:13:26.943618+0000","last_fullsized":"2024-04-16T16:13:26.943618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:40:10.575471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955303+0000","last_change":"2024-04-16T16:13:25.936302+0000","last_active":"2024-04-16T16:13:26.955303+0000","last_peered":"2024-04-16T16:13:26.955303+0000","last_clean":"2024-04-16T16:13:26.955303+0000","last_became_active":"2024-04-16T16:13:25.935907+0000","last_became_peered":"2024-04-16T16:13:25.935907+0000","last_unstale":"2024-04-16T16:13:26.955303+0000","last_undegraded":"2024-04-16T16:13:26.955303+0000","last_fullsized":"2024-04-16T16:13:26.955303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.951569+0000","last_change":"2024-04-16T16:13:25.935281+0000","last_active":"2024-04-16T16:13:25.951569+0000","last_peered":"2024-04-16T16:13:25.951569+0000","last_clean":"2024-04-16T16:13:25.951569+0000","last_became_active":"2024-04-16T16:13:25.934986+0000","last_became_peered":"2024-04-16T16:13:25.934986+0000","last_unstale":"2024-04-16T16:13:25.951569+0000","last_undegraded":"2024-04-16T16:13:25.951569+0000","last_fullsized":"2024-04-16T16:13:25.951569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:21.924263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970937+0000","last_change":"2024-04-16T16:13:25.952485+0000","last_active":"2024-04-16T16:13:26.970937+0000","last_peered":"2024-04-16T16:13:26.970937+0000","last_clean":"2024-04-16T16:13:26.970937+0000","last_became_active":"2024-04-16T16:13:25.952189+0000","last_became_peered":"2024-04-16T16:13:25.952189+0000","last_unstale":"2024-04-16T16:13:26.970937+0000","last_undegraded":"2024-04-16T16:13:26.970937+0000","last_fullsized":"2024-04-16T16:13:26.970937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:47.987635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942605+0000","last_change":"2024-04-16T16:13:25.930237+0000","last_active":"2024-04-16T16:13:26.942605+0000","last_peered":"2024-04-16T16:13:26.942605+0000","last_clean":"2024-04-16T16:13:26.942605+0000","last_became_active":"2024-04-16T16:13:25.929635+0000","last_became_peered":"2024-04-16T16:13:25.929635+0000","last_unstale":"2024-04-16T16:13:26.942605+0000","last_undegraded":"2024-04-16T16:13:26.942605+0000","last_fullsized":"2024-04-16T16:13:26.942605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:30:58.937717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970737+0000","last_change":"2024-04-16T16:13:25.947705+0000","last_active":"2024-04-16T16:13:26.970737+0000","last_peered":"2024-04-16T16:13:26.970737+0000","last_clean":"2024-04-16T16:13:26.970737+0000","last_became_active":"2024-04-16T16:13:25.947433+0000","last_became_peered":"2024-04-16T16:13:25.947433+0000","last_unstale":"2024-04-16T16:13:26.970737+0000","last_undegraded":"2024-04-16T16:13:26.970737+0000","last_fullsized":"2024-04-16T16:13:26.970737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941969+0000","last_change":"2024-04-16T16:13:25.930905+0000","last_active":"2024-04-16T16:13:26.941969+0000","last_peered":"2024-04-16T16:13:26.941969+0000","last_clean":"2024-04-16T16:13:26.941969+0000","last_became_active":"2024-04-16T16:13:25.929782+0000","last_became_peered":"2024-04-16T16:13:25.929782+0000","last_unstale":"2024-04-16T16:13:26.941969+0000","last_undegraded":"2024-04-16T16:13:26.941969+0000","last_fullsized":"2024-04-16T16:13:26.941969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.951126+0000","last_change":"2024-04-16T16:13:25.947751+0000","last_active":"2024-04-16T16:13:25.951126+0000","last_peered":"2024-04-16T16:13:25.951126+0000","last_clean":"2024-04-16T16:13:25.951126+0000","last_became_active":"2024-04-16T16:13:25.947110+0000","last_became_peered":"2024-04-16T16:13:25.947110+0000","last_unstale":"2024-04-16T16:13:25.951126+0000","last_undegraded":"2024-04-16T16:13:25.951126+0000","last_fullsized":"2024-04-16T16:13:25.951126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:32:38.024739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943813+0000","last_change":"2024-04-16T16:13:25.950917+0000","last_active":"2024-04-16T16:13:26.943813+0000","last_peered":"2024-04-16T16:13:26.943813+0000","last_clean":"2024-04-16T16:13:26.943813+0000","last_became_active":"2024-04-16T16:13:25.946716+0000","last_became_peered":"2024-04-16T16:13:25.946716+0000","last_unstale":"2024-04-16T16:13:26.943813+0000","last_undegraded":"2024-04-16T16:13:26.943813+0000","last_fullsized":"2024-04-16T16:13:26.943813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:52:47.172342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961550+0000","last_change":"2024-04-16T16:13:25.941148+0000","last_active":"2024-04-16T16:13:26.961550+0000","last_peered":"2024-04-16T16:13:26.961550+0000","last_clean":"2024-04-16T16:13:26.961550+0000","last_became_active":"2024-04-16T16:13:25.940818+0000","last_became_peered":"2024-04-16T16:13:25.940818+0000","last_unstale":"2024-04-16T16:13:26.961550+0000","last_undegraded":"2024-04-16T16:13:26.961550+0000","last_fullsized":"2024-04-16T16:13:26.961550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960975+0000","last_change":"2024-04-16T16:13:25.943177+0000","last_active":"2024-04-16T16:13:26.960975+0000","last_peered":"2024-04-16T16:13:26.960975+0000","last_clean":"2024-04-16T16:13:26.960975+0000","last_became_active":"2024-04-16T16:13:25.942902+0000","last_became_peered":"2024-04-16T16:13:25.942902+0000","last_unstale":"2024-04-16T16:13:26.960975+0000","last_undegraded":"2024-04-16T16:13:26.960975+0000","last_fullsized":"2024-04-16T16:13:26.960975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:40:58.670474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941834+0000","last_change":"2024-04-16T16:13:25.938811+0000","last_active":"2024-04-16T16:13:26.941834+0000","last_peered":"2024-04-16T16:13:26.941834+0000","last_clean":"2024-04-16T16:13:26.941834+0000","last_became_active":"2024-04-16T16:13:25.938479+0000","last_became_peered":"2024-04-16T16:13:25.938479+0000","last_unstale":"2024-04-16T16:13:26.941834+0000","last_undegraded":"2024-04-16T16:13:26.941834+0000","last_fullsized":"2024-04-16T16:13:26.941834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:41.335228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.953463+0000","last_change":"2024-04-16T16:13:25.950387+0000","last_active":"2024-04-16T16:13:25.953463+0000","last_peered":"2024-04-16T16:13:25.953463+0000","last_clean":"2024-04-16T16:13:25.953463+0000","last_became_active":"2024-04-16T16:13:25.949164+0000","last_became_peered":"2024-04-16T16:13:25.949164+0000","last_unstale":"2024-04-16T16:13:25.953463+0000","last_undegraded":"2024-04-16T16:13:25.953463+0000","last_fullsized":"2024-04-16T16:13:25.953463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:38:46.469621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.951916+0000","last_change":"2024-04-16T16:13:25.939438+0000","last_active":"2024-04-16T16:13:25.951916+0000","last_peered":"2024-04-16T16:13:25.951916+0000","last_clean":"2024-04-16T16:13:25.951916+0000","last_became_active":"2024-04-16T16:13:25.939066+0000","last_became_peered":"2024-04-16T16:13:25.939066+0000","last_unstale":"2024-04-16T16:13:25.951916+0000","last_undegraded":"2024-04-16T16:13:25.951916+0000","last_fullsized":"2024-04-16T16:13:25.951916+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:26.052535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954570+0000","last_change":"2024-04-16T16:13:25.937157+0000","last_active":"2024-04-16T16:13:26.954570+0000","last_peered":"2024-04-16T16:13:26.954570+0000","last_clean":"2024-04-16T16:13:26.954570+0000","last_became_active":"2024-04-16T16:13:25.936851+0000","last_became_peered":"2024-04-16T16:13:25.936851+0000","last_unstale":"2024-04-16T16:13:26.954570+0000","last_undegraded":"2024-04-16T16:13:26.954570+0000","last_fullsized":"2024-04-16T16:13:26.954570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:55.948136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970498+0000","last_change":"2024-04-16T16:13:25.949240+0000","last_active":"2024-04-16T16:13:26.970498+0000","last_peered":"2024-04-16T16:13:26.970498+0000","last_clean":"2024-04-16T16:13:26.970498+0000","last_became_active":"2024-04-16T16:13:25.948913+0000","last_became_peered":"2024-04-16T16:13:25.948913+0000","last_unstale":"2024-04-16T16:13:26.970498+0000","last_undegraded":"2024-04-16T16:13:26.970498+0000","last_fullsized":"2024-04-16T16:13:26.970498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.972475+0000","last_change":"2024-04-16T16:13:25.947113+0000","last_active":"2024-04-16T16:13:26.972475+0000","last_peered":"2024-04-16T16:13:26.972475+0000","last_clean":"2024-04-16T16:13:26.972475+0000","last_became_active":"2024-04-16T16:13:25.943174+0000","last_became_peered":"2024-04-16T16:13:25.943174+0000","last_unstale":"2024-04-16T16:13:26.972475+0000","last_undegraded":"2024-04-16T16:13:26.972475+0000","last_fullsized":"2024-04-16T16:13:26.972475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:08.801668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.937163+0000","last_change":"2024-04-16T16:13:25.942311+0000","last_active":"2024-04-16T16:13:26.937163+0000","last_peered":"2024-04-16T16:13:26.937163+0000","last_clean":"2024-04-16T16:13:26.937163+0000","last_became_active":"2024-04-16T16:13:25.942015+0000","last_became_peered":"2024-04-16T16:13:25.942015+0000","last_unstale":"2024-04-16T16:13:26.937163+0000","last_undegraded":"2024-04-16T16:13:26.937163+0000","last_fullsized":"2024-04-16T16:13:26.937163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:25.740193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943340+0000","last_change":"2024-04-16T16:13:25.929976+0000","last_active":"2024-04-16T16:13:26.943340+0000","last_peered":"2024-04-16T16:13:26.943340+0000","last_clean":"2024-04-16T16:13:26.943340+0000","last_became_active":"2024-04-16T16:13:25.929347+0000","last_became_peered":"2024-04-16T16:13:25.929347+0000","last_unstale":"2024-04-16T16:13:26.943340+0000","last_undegraded":"2024-04-16T16:13:26.943340+0000","last_fullsized":"2024-04-16T16:13:26.943340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:19:05.123994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960773+0000","last_change":"2024-04-16T16:13:25.945575+0000","last_active":"2024-04-16T16:13:26.960773+0000","last_peered":"2024-04-16T16:13:26.960773+0000","last_clean":"2024-04-16T16:13:26.960773+0000","last_became_active":"2024-04-16T16:13:25.945217+0000","last_became_peered":"2024-04-16T16:13:25.945217+0000","last_unstale":"2024-04-16T16:13:26.960773+0000","last_undegraded":"2024-04-16T16:13:26.960773+0000","last_fullsized":"2024-04-16T16:13:26.960773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:55.948136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.952491+0000","last_change":"2024-04-16T16:13:25.945082+0000","last_active":"2024-04-16T16:13:25.952491+0000","last_peered":"2024-04-16T16:13:25.952491+0000","last_clean":"2024-04-16T16:13:25.952491+0000","last_became_active":"2024-04-16T16:13:25.944781+0000","last_became_peered":"2024-04-16T16:13:25.944781+0000","last_unstale":"2024-04-16T16:13:25.952491+0000","last_undegraded":"2024-04-16T16:13:25.952491+0000","last_fullsized":"2024-04-16T16:13:25.952491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:41.335228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970886+0000","last_change":"2024-04-16T16:13:25.943741+0000","last_active":"2024-04-16T16:13:26.970886+0000","last_peered":"2024-04-16T16:13:26.970886+0000","last_clean":"2024-04-16T16:13:26.970886+0000","last_became_active":"2024-04-16T16:13:25.942843+0000","last_became_peered":"2024-04-16T16:13:25.942843+0000","last_unstale":"2024-04-16T16:13:26.970886+0000","last_undegraded":"2024-04-16T16:13:26.970886+0000","last_fullsized":"2024-04-16T16:13:26.970886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:30:58.937717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.946699+0000","last_change":"2024-04-16T16:13:25.943232+0000","last_active":"2024-04-16T16:13:25.946699+0000","last_peered":"2024-04-16T16:13:25.946699+0000","last_clean":"2024-04-16T16:13:25.946699+0000","last_became_active":"2024-04-16T16:13:25.940853+0000","last_became_peered":"2024-04-16T16:13:25.940853+0000","last_unstale":"2024-04-16T16:13:25.946699+0000","last_undegraded":"2024-04-16T16:13:25.946699+0000","last_fullsized":"2024-04-16T16:13:25.946699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:12.895579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970797+0000","last_change":"2024-04-16T16:13:25.943328+0000","last_active":"2024-04-16T16:13:26.970797+0000","last_peered":"2024-04-16T16:13:26.970797+0000","last_clean":"2024-04-16T16:13:26.970797+0000","last_became_active":"2024-04-16T16:13:25.941855+0000","last_became_peered":"2024-04-16T16:13:25.941855+0000","last_unstale":"2024-04-16T16:13:26.970797+0000","last_undegraded":"2024-04-16T16:13:26.970797+0000","last_fullsized":"2024-04-16T16:13:26.970797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:41.243756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.945451+0000","last_change":"2024-04-16T16:13:25.942627+0000","last_active":"2024-04-16T16:13:25.945451+0000","last_peered":"2024-04-16T16:13:25.945451+0000","last_clean":"2024-04-16T16:13:25.945451+0000","last_became_active":"2024-04-16T16:13:25.942285+0000","last_became_peered":"2024-04-16T16:13:25.942285+0000","last_unstale":"2024-04-16T16:13:25.945451+0000","last_undegraded":"2024-04-16T16:13:25.945451+0000","last_fullsized":"2024-04-16T16:13:25.945451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:41.335228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961635+0000","last_change":"2024-04-16T16:13:25.950401+0000","last_active":"2024-04-16T16:13:26.961635+0000","last_peered":"2024-04-16T16:13:26.961635+0000","last_clean":"2024-04-16T16:13:26.961635+0000","last_became_active":"2024-04-16T16:13:25.948651+0000","last_became_peered":"2024-04-16T16:13:25.948651+0000","last_unstale":"2024-04-16T16:13:26.961635+0000","last_undegraded":"2024-04-16T16:13:26.961635+0000","last_fullsized":"2024-04-16T16:13:26.961635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955096+0000","last_change":"2024-04-16T16:13:25.945071+0000","last_active":"2024-04-16T16:13:26.955096+0000","last_peered":"2024-04-16T16:13:26.955096+0000","last_clean":"2024-04-16T16:13:26.955096+0000","last_became_active":"2024-04-16T16:13:25.944708+0000","last_became_peered":"2024-04-16T16:13:25.944708+0000","last_unstale":"2024-04-16T16:13:26.955096+0000","last_undegraded":"2024-04-16T16:13:26.955096+0000","last_fullsized":"2024-04-16T16:13:26.955096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.946937+0000","last_change":"2024-04-16T16:13:25.944654+0000","last_active":"2024-04-16T16:13:25.946937+0000","last_peered":"2024-04-16T16:13:25.946937+0000","last_clean":"2024-04-16T16:13:25.946937+0000","last_became_active":"2024-04-16T16:13:25.941914+0000","last_became_peered":"2024-04-16T16:13:25.941914+0000","last_unstale":"2024-04-16T16:13:25.946937+0000","last_undegraded":"2024-04-16T16:13:25.946937+0000","last_fullsized":"2024-04-16T16:13:25.946937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:25.740193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.952512+0000","last_change":"2024-04-16T16:13:25.947081+0000","last_active":"2024-04-16T16:13:25.952512+0000","last_peered":"2024-04-16T16:13:25.952512+0000","last_clean":"2024-04-16T16:13:25.952512+0000","last_became_active":"2024-04-16T16:13:25.946791+0000","last_became_peered":"2024-04-16T16:13:25.946791+0000","last_unstale":"2024-04-16T16:13:25.952512+0000","last_undegraded":"2024-04-16T16:13:25.952512+0000","last_fullsized":"2024-04-16T16:13:25.952512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:25.740193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971641+0000","last_change":"2024-04-16T16:13:25.952506+0000","last_active":"2024-04-16T16:13:26.971641+0000","last_peered":"2024-04-16T16:13:26.971641+0000","last_clean":"2024-04-16T16:13:26.971641+0000","last_became_active":"2024-04-16T16:13:25.952184+0000","last_became_peered":"2024-04-16T16:13:25.952184+0000","last_unstale":"2024-04-16T16:13:26.971641+0000","last_undegraded":"2024-04-16T16:13:26.971641+0000","last_fullsized":"2024-04-16T16:13:26.971641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:19:05.123994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936714+0000","last_change":"2024-04-16T16:13:25.936689+0000","last_active":"2024-04-16T16:13:26.936714+0000","last_peered":"2024-04-16T16:13:26.936714+0000","last_clean":"2024-04-16T16:13:26.936714+0000","last_became_active":"2024-04-16T16:13:25.936385+0000","last_became_peered":"2024-04-16T16:13:25.936385+0000","last_unstale":"2024-04-16T16:13:26.936714+0000","last_undegraded":"2024-04-16T16:13:26.936714+0000","last_fullsized":"2024-04-16T16:13:26.936714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:12.895579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960388+0000","last_change":"2024-04-16T16:13:25.953671+0000","last_active":"2024-04-16T16:13:26.960388+0000","last_peered":"2024-04-16T16:13:26.960388+0000","last_clean":"2024-04-16T16:13:26.960388+0000","last_became_active":"2024-04-16T16:13:25.953325+0000","last_became_peered":"2024-04-16T16:13:25.953325+0000","last_unstale":"2024-04-16T16:13:26.960388+0000","last_undegraded":"2024-04-16T16:13:26.960388+0000","last_fullsized":"2024-04-16T16:13:26.960388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:51:20.451559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941086+0000","last_change":"2024-04-16T16:13:25.936138+0000","last_active":"2024-04-16T16:13:26.941086+0000","last_peered":"2024-04-16T16:13:26.941086+0000","last_clean":"2024-04-16T16:13:26.941086+0000","last_became_active":"2024-04-16T16:13:25.935759+0000","last_became_peered":"2024-04-16T16:13:25.935759+0000","last_unstale":"2024-04-16T16:13:26.941086+0000","last_undegraded":"2024-04-16T16:13:26.941086+0000","last_fullsized":"2024-04-16T16:13:26.941086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:26.052535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.965680+0000","last_change":"2024-04-16T16:13:24.912677+0000","last_active":"2024-04-16T16:13:25.965680+0000","last_peered":"2024-04-16T16:13:25.965680+0000","last_clean":"2024-04-16T16:13:25.965680+0000","last_became_active":"2024-04-16T16:13:24.912364+0000","last_became_peered":"2024-04-16T16:13:24.912364+0000","last_unstale":"2024-04-16T16:13:25.965680+0000","last_undegraded":"2024-04-16T16:13:25.965680+0000","last_fullsized":"2024-04-16T16:13:25.965680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.952533+0000","last_change":"2024-04-16T16:13:25.947853+0000","last_active":"2024-04-16T16:13:25.952533+0000","last_peered":"2024-04-16T16:13:25.952533+0000","last_clean":"2024-04-16T16:13:25.952533+0000","last_became_active":"2024-04-16T16:13:25.947545+0000","last_became_peered":"2024-04-16T16:13:25.947545+0000","last_unstale":"2024-04-16T16:13:25.952533+0000","last_undegraded":"2024-04-16T16:13:25.952533+0000","last_fullsized":"2024-04-16T16:13:25.952533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:42.567277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960762+0000","last_change":"2024-04-16T16:13:24.917069+0000","last_active":"2024-04-16T16:13:26.960762+0000","last_peered":"2024-04-16T16:13:26.960762+0000","last_clean":"2024-04-16T16:13:26.960762+0000","last_became_active":"2024-04-16T16:13:24.916846+0000","last_became_peered":"2024-04-16T16:13:24.916846+0000","last_unstale":"2024-04-16T16:13:26.960762+0000","last_undegraded":"2024-04-16T16:13:26.960762+0000","last_fullsized":"2024-04-16T16:13:26.960762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.935567+0000","last_change":"2024-04-16T16:13:24.919388+0000","last_active":"2024-04-16T16:13:25.935567+0000","last_peered":"2024-04-16T16:13:25.935567+0000","last_clean":"2024-04-16T16:13:25.935567+0000","last_became_active":"2024-04-16T16:13:24.918937+0000","last_became_peered":"2024-04-16T16:13:24.918937+0000","last_unstale":"2024-04-16T16:13:25.935567+0000","last_undegraded":"2024-04-16T16:13:25.935567+0000","last_fullsized":"2024-04-16T16:13:25.935567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:46:08.208026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.955331+0000","last_change":"2024-04-16T16:13:25.954029+0000","last_active":"2024-04-16T16:13:25.955331+0000","last_peered":"2024-04-16T16:13:25.955331+0000","last_clean":"2024-04-16T16:13:25.955331+0000","last_became_active":"2024-04-16T16:13:25.953740+0000","last_became_peered":"2024-04-16T16:13:25.953740+0000","last_unstale":"2024-04-16T16:13:25.955331+0000","last_undegraded":"2024-04-16T16:13:25.955331+0000","last_fullsized":"2024-04-16T16:13:25.955331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:23.603931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971052+0000","last_change":"2024-04-16T16:13:24.923031+0000","last_active":"2024-04-16T16:13:26.971052+0000","last_peered":"2024-04-16T16:13:26.971052+0000","last_clean":"2024-04-16T16:13:26.971052+0000","last_became_active":"2024-04-16T16:13:24.922708+0000","last_became_peered":"2024-04-16T16:13:24.922708+0000","last_unstale":"2024-04-16T16:13:26.971052+0000","last_undegraded":"2024-04-16T16:13:26.971052+0000","last_fullsized":"2024-04-16T16:13:26.971052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954570+0000","last_change":"2024-04-16T16:13:25.949124+0000","last_active":"2024-04-16T16:13:26.954570+0000","last_peered":"2024-04-16T16:13:26.954570+0000","last_clean":"2024-04-16T16:13:26.954570+0000","last_became_active":"2024-04-16T16:13:25.948436+0000","last_became_peered":"2024-04-16T16:13:25.948436+0000","last_unstale":"2024-04-16T16:13:26.954570+0000","last_undegraded":"2024-04-16T16:13:26.954570+0000","last_fullsized":"2024-04-16T16:13:26.954570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:10.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961547+0000","last_change":"2024-04-16T16:13:24.925547+0000","last_active":"2024-04-16T16:13:26.961547+0000","last_peered":"2024-04-16T16:13:26.961547+0000","last_clean":"2024-04-16T16:13:26.961547+0000","last_became_active":"2024-04-16T16:13:24.925182+0000","last_became_peered":"2024-04-16T16:13:24.925182+0000","last_unstale":"2024-04-16T16:13:26.961547+0000","last_undegraded":"2024-04-16T16:13:26.961547+0000","last_fullsized":"2024-04-16T16:13:26.961547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.972351+0000","last_change":"2024-04-16T16:13:25.945204+0000","last_active":"2024-04-16T16:13:26.972351+0000","last_peered":"2024-04-16T16:13:26.972351+0000","last_clean":"2024-04-16T16:13:26.972351+0000","last_became_active":"2024-04-16T16:13:25.944902+0000","last_became_peered":"2024-04-16T16:13:25.944902+0000","last_unstale":"2024-04-16T16:13:26.972351+0000","last_undegraded":"2024-04-16T16:13:26.972351+0000","last_fullsized":"2024-04-16T16:13:26.972351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:52:47.172342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971774+0000","last_change":"2024-04-16T16:13:24.928041+0000","last_active":"2024-04-16T16:13:26.971774+0000","last_peered":"2024-04-16T16:13:26.971774+0000","last_clean":"2024-04-16T16:13:26.971774+0000","last_became_active":"2024-04-16T16:13:24.927555+0000","last_became_peered":"2024-04-16T16:13:24.927555+0000","last_unstale":"2024-04-16T16:13:26.971774+0000","last_undegraded":"2024-04-16T16:13:26.971774+0000","last_fullsized":"2024-04-16T16:13:26.971774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.952119+0000","last_change":"2024-04-16T16:13:25.943385+0000","last_active":"2024-04-16T16:13:25.952119+0000","last_peered":"2024-04-16T16:13:25.952119+0000","last_clean":"2024-04-16T16:13:25.952119+0000","last_became_active":"2024-04-16T16:13:25.943095+0000","last_became_peered":"2024-04-16T16:13:25.943095+0000","last_unstale":"2024-04-16T16:13:25.952119+0000","last_undegraded":"2024-04-16T16:13:25.952119+0000","last_fullsized":"2024-04-16T16:13:25.952119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:00.025740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955094+0000","last_change":"2024-04-16T16:13:24.922351+0000","last_active":"2024-04-16T16:13:26.955094+0000","last_peered":"2024-04-16T16:13:26.955094+0000","last_clean":"2024-04-16T16:13:26.955094+0000","last_became_active":"2024-04-16T16:13:24.922017+0000","last_became_peered":"2024-04-16T16:13:24.922017+0000","last_unstale":"2024-04-16T16:13:26.955094+0000","last_undegraded":"2024-04-16T16:13:26.955094+0000","last_fullsized":"2024-04-16T16:13:26.955094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:24.722256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942286+0000","last_change":"2024-04-16T16:13:25.944764+0000","last_active":"2024-04-16T16:13:26.942286+0000","last_peered":"2024-04-16T16:13:26.942286+0000","last_clean":"2024-04-16T16:13:26.942286+0000","last_became_active":"2024-04-16T16:13:25.944418+0000","last_became_peered":"2024-04-16T16:13:25.944418+0000","last_unstale":"2024-04-16T16:13:26.942286+0000","last_undegraded":"2024-04-16T16:13:26.942286+0000","last_fullsized":"2024-04-16T16:13:26.942286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:25.740193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960752+0000","last_change":"2024-04-16T16:13:24.916122+0000","last_active":"2024-04-16T16:13:26.960752+0000","last_peered":"2024-04-16T16:13:26.960752+0000","last_clean":"2024-04-16T16:13:26.960752+0000","last_became_active":"2024-04-16T16:13:24.915132+0000","last_became_peered":"2024-04-16T16:13:24.915132+0000","last_unstale":"2024-04-16T16:13:26.960752+0000","last_undegraded":"2024-04-16T16:13:26.960752+0000","last_fullsized":"2024-04-16T16:13:26.960752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942858+0000","last_change":"2024-04-16T16:13:25.954813+0000","last_active":"2024-04-16T16:13:26.942858+0000","last_peered":"2024-04-16T16:13:26.942858+0000","last_clean":"2024-04-16T16:13:26.942858+0000","last_became_active":"2024-04-16T16:13:25.952310+0000","last_became_peered":"2024-04-16T16:13:25.952310+0000","last_unstale":"2024-04-16T16:13:26.942858+0000","last_undegraded":"2024-04-16T16:13:26.942858+0000","last_fullsized":"2024-04-16T16:13:26.942858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:13:01.042309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.967305+0000","last_change":"2024-04-16T16:13:24.907364+0000","last_active":"2024-04-16T16:13:25.967305+0000","last_peered":"2024-04-16T16:13:25.967305+0000","last_clean":"2024-04-16T16:13:25.967305+0000","last_became_active":"2024-04-16T16:13:24.906972+0000","last_became_peered":"2024-04-16T16:13:24.906972+0000","last_unstale":"2024-04-16T16:13:25.967305+0000","last_undegraded":"2024-04-16T16:13:25.967305+0000","last_fullsized":"2024-04-16T16:13:25.967305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:22.075493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.951939+0000","last_change":"2024-04-16T16:13:25.946280+0000","last_active":"2024-04-16T16:13:25.951939+0000","last_peered":"2024-04-16T16:13:25.951939+0000","last_clean":"2024-04-16T16:13:25.951939+0000","last_became_active":"2024-04-16T16:13:25.945919+0000","last_became_peered":"2024-04-16T16:13:25.945919+0000","last_unstale":"2024-04-16T16:13:25.951939+0000","last_undegraded":"2024-04-16T16:13:25.951939+0000","last_fullsized":"2024-04-16T16:13:25.951939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:12.895579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971179+0000","last_change":"2024-04-16T16:13:24.921202+0000","last_active":"2024-04-16T16:13:26.971179+0000","last_peered":"2024-04-16T16:13:26.971179+0000","last_clean":"2024-04-16T16:13:26.971179+0000","last_became_active":"2024-04-16T16:13:24.920850+0000","last_became_peered":"2024-04-16T16:13:24.920850+0000","last_unstale":"2024-04-16T16:13:26.971179+0000","last_undegraded":"2024-04-16T16:13:26.971179+0000","last_fullsized":"2024-04-16T16:13:26.971179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942649+0000","last_change":"2024-04-16T16:13:25.947511+0000","last_active":"2024-04-16T16:13:26.942649+0000","last_peered":"2024-04-16T16:13:26.942649+0000","last_clean":"2024-04-16T16:13:26.942649+0000","last_became_active":"2024-04-16T16:13:25.947213+0000","last_became_peered":"2024-04-16T16:13:25.947213+0000","last_unstale":"2024-04-16T16:13:26.942649+0000","last_undegraded":"2024-04-16T16:13:26.942649+0000","last_fullsized":"2024-04-16T16:13:26.942649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:47.987635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960500+0000","last_change":"2024-04-16T16:13:24.923552+0000","last_active":"2024-04-16T16:13:26.960500+0000","last_peered":"2024-04-16T16:13:26.960500+0000","last_clean":"2024-04-16T16:13:26.960500+0000","last_became_active":"2024-04-16T16:13:24.922940+0000","last_became_peered":"2024-04-16T16:13:24.922940+0000","last_unstale":"2024-04-16T16:13:26.960500+0000","last_undegraded":"2024-04-16T16:13:26.960500+0000","last_fullsized":"2024-04-16T16:13:26.960500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960474+0000","last_change":"2024-04-16T16:13:25.954523+0000","last_active":"2024-04-16T16:13:26.960474+0000","last_peered":"2024-04-16T16:13:26.960474+0000","last_clean":"2024-04-16T16:13:26.960474+0000","last_became_active":"2024-04-16T16:13:25.954058+0000","last_became_peered":"2024-04-16T16:13:25.954058+0000","last_unstale":"2024-04-16T16:13:26.960474+0000","last_undegraded":"2024-04-16T16:13:26.960474+0000","last_fullsized":"2024-04-16T16:13:26.960474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:11:32.536402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936502+0000","last_change":"2024-04-16T16:13:24.904943+0000","last_active":"2024-04-16T16:13:26.936502+0000","last_peered":"2024-04-16T16:13:26.936502+0000","last_clean":"2024-04-16T16:13:26.936502+0000","last_became_active":"2024-04-16T16:13:24.904527+0000","last_became_peered":"2024-04-16T16:13:24.904527+0000","last_unstale":"2024-04-16T16:13:26.936502+0000","last_undegraded":"2024-04-16T16:13:26.936502+0000","last_fullsized":"2024-04-16T16:13:26.936502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:22.075493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942098+0000","last_change":"2024-04-16T16:13:25.937269+0000","last_active":"2024-04-16T16:13:26.942098+0000","last_peered":"2024-04-16T16:13:26.942098+0000","last_clean":"2024-04-16T16:13:26.942098+0000","last_became_active":"2024-04-16T16:13:25.936953+0000","last_became_peered":"2024-04-16T16:13:25.936953+0000","last_unstale":"2024-04-16T16:13:26.942098+0000","last_undegraded":"2024-04-16T16:13:26.942098+0000","last_fullsized":"2024-04-16T16:13:26.942098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:41.243756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941662+0000","last_change":"2024-04-16T16:13:24.909436+0000","last_active":"2024-04-16T16:13:26.941662+0000","last_peered":"2024-04-16T16:13:26.941662+0000","last_clean":"2024-04-16T16:13:26.941662+0000","last_became_active":"2024-04-16T16:13:24.908963+0000","last_became_peered":"2024-04-16T16:13:24.908963+0000","last_unstale":"2024-04-16T16:13:26.941662+0000","last_undegraded":"2024-04-16T16:13:26.941662+0000","last_fullsized":"2024-04-16T16:13:26.941662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:21.269919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943634+0000","last_change":"2024-04-16T16:13:25.939892+0000","last_active":"2024-04-16T16:13:26.943634+0000","last_peered":"2024-04-16T16:13:26.943634+0000","last_clean":"2024-04-16T16:13:26.943634+0000","last_became_active":"2024-04-16T16:13:25.939610+0000","last_became_peered":"2024-04-16T16:13:25.939610+0000","last_unstale":"2024-04-16T16:13:26.943634+0000","last_undegraded":"2024-04-16T16:13:26.943634+0000","last_fullsized":"2024-04-16T16:13:26.943634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:58.462911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.937081+0000","last_change":"2024-04-16T16:13:24.898787+0000","last_active":"2024-04-16T16:13:26.937081+0000","last_peered":"2024-04-16T16:13:26.937081+0000","last_clean":"2024-04-16T16:13:26.937081+0000","last_became_active":"2024-04-16T16:13:24.898427+0000","last_became_peered":"2024-04-16T16:13:24.898427+0000","last_unstale":"2024-04-16T16:13:26.937081+0000","last_undegraded":"2024-04-16T16:13:26.937081+0000","last_fullsized":"2024-04-16T16:13:26.937081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943828+0000","last_change":"2024-04-16T16:13:25.953661+0000","last_active":"2024-04-16T16:13:26.943828+0000","last_peered":"2024-04-16T16:13:26.943828+0000","last_clean":"2024-04-16T16:13:26.943828+0000","last_became_active":"2024-04-16T16:13:25.950243+0000","last_became_peered":"2024-04-16T16:13:25.950243+0000","last_unstale":"2024-04-16T16:13:26.943828+0000","last_undegraded":"2024-04-16T16:13:26.943828+0000","last_fullsized":"2024-04-16T16:13:26.943828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:08.801668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955224+0000","last_change":"2024-04-16T16:13:24.906785+0000","last_active":"2024-04-16T16:13:26.955224+0000","last_peered":"2024-04-16T16:13:26.955224+0000","last_clean":"2024-04-16T16:13:26.955224+0000","last_became_active":"2024-04-16T16:13:24.906478+0000","last_became_peered":"2024-04-16T16:13:24.906478+0000","last_unstale":"2024-04-16T16:13:26.955224+0000","last_undegraded":"2024-04-16T16:13:26.955224+0000","last_fullsized":"2024-04-16T16:13:26.955224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:41.549340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971400+0000","last_change":"2024-04-16T16:13:25.951170+0000","last_active":"2024-04-16T16:13:26.971400+0000","last_peered":"2024-04-16T16:13:26.971400+0000","last_clean":"2024-04-16T16:13:26.971400+0000","last_became_active":"2024-04-16T16:13:25.950423+0000","last_became_peered":"2024-04-16T16:13:25.950423+0000","last_unstale":"2024-04-16T16:13:26.971400+0000","last_undegraded":"2024-04-16T16:13:26.971400+0000","last_fullsized":"2024-04-16T16:13:26.971400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:13:01.042309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936858+0000","last_change":"2024-04-16T16:13:24.920097+0000","last_active":"2024-04-16T16:13:26.936858+0000","last_peered":"2024-04-16T16:13:26.936858+0000","last_clean":"2024-04-16T16:13:26.936858+0000","last_became_active":"2024-04-16T16:13:24.919848+0000","last_became_peered":"2024-04-16T16:13:24.919848+0000","last_unstale":"2024-04-16T16:13:26.936858+0000","last_undegraded":"2024-04-16T16:13:26.936858+0000","last_fullsized":"2024-04-16T16:13:26.936858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:46:08.208026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942819+0000","last_change":"2024-04-16T16:13:25.952599+0000","last_active":"2024-04-16T16:13:26.942819+0000","last_peered":"2024-04-16T16:13:26.942819+0000","last_clean":"2024-04-16T16:13:26.942819+0000","last_became_active":"2024-04-16T16:13:25.949217+0000","last_became_peered":"2024-04-16T16:13:25.949217+0000","last_unstale":"2024-04-16T16:13:26.942819+0000","last_undegraded":"2024-04-16T16:13:26.942819+0000","last_fullsized":"2024-04-16T16:13:26.942819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:39.592952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.940554+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:13:26.940554+0000","last_peered":"2024-04-16T16:13:26.940554+0000","last_clean":"2024-04-16T16:13:26.940554+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:13:26.940554+0000","last_undegraded":"2024-04-16T16:13:26.940554+0000","last_fullsized":"2024-04-16T16:13:26.940554+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970651+0000","last_change":"2024-04-16T16:13:24.919303+0000","last_active":"2024-04-16T16:13:26.970651+0000","last_peered":"2024-04-16T16:13:26.970651+0000","last_clean":"2024-04-16T16:13:26.970651+0000","last_became_active":"2024-04-16T16:13:24.918056+0000","last_became_peered":"2024-04-16T16:13:24.918056+0000","last_unstale":"2024-04-16T16:13:26.970651+0000","last_undegraded":"2024-04-16T16:13:26.970651+0000","last_fullsized":"2024-04-16T16:13:26.970651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.951015+0000","last_change":"2024-04-16T16:13:25.947227+0000","last_active":"2024-04-16T16:13:25.951015+0000","last_peered":"2024-04-16T16:13:25.951015+0000","last_clean":"2024-04-16T16:13:25.951015+0000","last_became_active":"2024-04-16T16:13:25.946951+0000","last_became_peered":"2024-04-16T16:13:25.946951+0000","last_unstale":"2024-04-16T16:13:25.951015+0000","last_undegraded":"2024-04-16T16:13:25.951015+0000","last_fullsized":"2024-04-16T16:13:25.951015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:02:29.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.940404+0000","last_change":"2024-04-16T16:13:24.912738+0000","last_active":"2024-04-16T16:13:26.940404+0000","last_peered":"2024-04-16T16:13:26.940404+0000","last_clean":"2024-04-16T16:13:26.940404+0000","last_became_active":"2024-04-16T16:13:24.912432+0000","last_became_peered":"2024-04-16T16:13:24.912432+0000","last_unstale":"2024-04-16T16:13:26.940404+0000","last_undegraded":"2024-04-16T16:13:26.940404+0000","last_fullsized":"2024-04-16T16:13:26.940404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.962470+0000","last_change":"2024-04-16T16:13:25.952202+0000","last_active":"2024-04-16T16:13:25.962470+0000","last_peered":"2024-04-16T16:13:25.962470+0000","last_clean":"2024-04-16T16:13:25.962470+0000","last_became_active":"2024-04-16T16:13:25.951932+0000","last_became_peered":"2024-04-16T16:13:25.951932+0000","last_unstale":"2024-04-16T16:13:25.962470+0000","last_undegraded":"2024-04-16T16:13:25.962470+0000","last_fullsized":"2024-04-16T16:13:25.962470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:02:29.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936531+0000","last_change":"2024-04-16T16:13:24.901093+0000","last_active":"2024-04-16T16:13:26.936531+0000","last_peered":"2024-04-16T16:13:26.936531+0000","last_clean":"2024-04-16T16:13:26.936531+0000","last_became_active":"2024-04-16T16:13:24.900772+0000","last_became_peered":"2024-04-16T16:13:24.900772+0000","last_unstale":"2024-04-16T16:13:26.936531+0000","last_undegraded":"2024-04-16T16:13:26.936531+0000","last_fullsized":"2024-04-16T16:13:26.936531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954397+0000","last_change":"2024-04-16T16:13:25.929184+0000","last_active":"2024-04-16T16:13:26.954397+0000","last_peered":"2024-04-16T16:13:26.954397+0000","last_clean":"2024-04-16T16:13:26.954397+0000","last_became_active":"2024-04-16T16:13:25.927325+0000","last_became_peered":"2024-04-16T16:13:25.927325+0000","last_unstale":"2024-04-16T16:13:26.954397+0000","last_undegraded":"2024-04-16T16:13:26.954397+0000","last_fullsized":"2024-04-16T16:13:26.954397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:11:32.536402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971426+0000","last_change":"2024-04-16T16:13:24.923465+0000","last_active":"2024-04-16T16:13:26.971426+0000","last_peered":"2024-04-16T16:13:26.971426+0000","last_clean":"2024-04-16T16:13:26.971426+0000","last_became_active":"2024-04-16T16:13:24.923145+0000","last_became_peered":"2024-04-16T16:13:24.923145+0000","last_unstale":"2024-04-16T16:13:26.971426+0000","last_undegraded":"2024-04-16T16:13:26.971426+0000","last_fullsized":"2024-04-16T16:13:26.971426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941524+0000","last_change":"2024-04-16T16:13:25.948469+0000","last_active":"2024-04-16T16:13:26.941524+0000","last_peered":"2024-04-16T16:13:26.941524+0000","last_clean":"2024-04-16T16:13:26.941524+0000","last_became_active":"2024-04-16T16:13:25.947992+0000","last_became_peered":"2024-04-16T16:13:25.947992+0000","last_unstale":"2024-04-16T16:13:26.941524+0000","last_undegraded":"2024-04-16T16:13:26.941524+0000","last_fullsized":"2024-04-16T16:13:26.941524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:00.025740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954768+0000","last_change":"2024-04-16T16:13:24.905663+0000","last_active":"2024-04-16T16:13:26.954768+0000","last_peered":"2024-04-16T16:13:26.954768+0000","last_clean":"2024-04-16T16:13:26.954768+0000","last_became_active":"2024-04-16T16:13:24.905306+0000","last_became_peered":"2024-04-16T16:13:24.905306+0000","last_unstale":"2024-04-16T16:13:26.954768+0000","last_undegraded":"2024-04-16T16:13:26.954768+0000","last_fullsized":"2024-04-16T16:13:26.954768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971908+0000","last_change":"2024-04-16T16:13:25.955137+0000","last_active":"2024-04-16T16:13:26.971908+0000","last_peered":"2024-04-16T16:13:26.971908+0000","last_clean":"2024-04-16T16:13:26.971908+0000","last_became_active":"2024-04-16T16:13:25.954852+0000","last_became_peered":"2024-04-16T16:13:25.954852+0000","last_unstale":"2024-04-16T16:13:26.971908+0000","last_undegraded":"2024-04-16T16:13:26.971908+0000","last_fullsized":"2024-04-16T16:13:26.971908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:58.462911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943455+0000","last_change":"2024-04-16T16:13:24.915400+0000","last_active":"2024-04-16T16:13:26.943455+0000","last_peered":"2024-04-16T16:13:26.943455+0000","last_clean":"2024-04-16T16:13:26.943455+0000","last_became_active":"2024-04-16T16:13:24.915096+0000","last_became_peered":"2024-04-16T16:13:24.915096+0000","last_unstale":"2024-04-16T16:13:26.943455+0000","last_undegraded":"2024-04-16T16:13:26.943455+0000","last_fullsized":"2024-04-16T16:13:26.943455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:41.549340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954886+0000","last_change":"2024-04-16T16:13:25.924316+0000","last_active":"2024-04-16T16:13:26.954886+0000","last_peered":"2024-04-16T16:13:26.954886+0000","last_clean":"2024-04-16T16:13:26.954886+0000","last_became_active":"2024-04-16T16:13:25.923986+0000","last_became_peered":"2024-04-16T16:13:25.923986+0000","last_unstale":"2024-04-16T16:13:26.954886+0000","last_undegraded":"2024-04-16T16:13:26.954886+0000","last_fullsized":"2024-04-16T16:13:26.954886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:40:58.670474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.958843+0000","last_change":"2024-04-16T16:13:24.920876+0000","last_active":"2024-04-16T16:13:26.958843+0000","last_peered":"2024-04-16T16:13:26.958843+0000","last_clean":"2024-04-16T16:13:26.958843+0000","last_became_active":"2024-04-16T16:13:24.920409+0000","last_became_peered":"2024-04-16T16:13:24.920409+0000","last_unstale":"2024-04-16T16:13:26.958843+0000","last_undegraded":"2024-04-16T16:13:26.958843+0000","last_fullsized":"2024-04-16T16:13:26.958843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.943879+0000","last_change":"2024-04-16T16:13:25.941909+0000","last_active":"2024-04-16T16:13:25.943879+0000","last_peered":"2024-04-16T16:13:25.943879+0000","last_clean":"2024-04-16T16:13:25.943879+0000","last_became_active":"2024-04-16T16:13:25.941614+0000","last_became_peered":"2024-04-16T16:13:25.941614+0000","last_unstale":"2024-04-16T16:13:25.943879+0000","last_undegraded":"2024-04-16T16:13:25.943879+0000","last_fullsized":"2024-04-16T16:13:25.943879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:00.025740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961615+0000","last_change":"2024-04-16T16:13:24.918600+0000","last_active":"2024-04-16T16:13:26.961615+0000","last_peered":"2024-04-16T16:13:26.961615+0000","last_clean":"2024-04-16T16:13:26.961615+0000","last_became_active":"2024-04-16T16:13:24.918273+0000","last_became_peered":"2024-04-16T16:13:24.918273+0000","last_unstale":"2024-04-16T16:13:26.961615+0000","last_undegraded":"2024-04-16T16:13:26.961615+0000","last_fullsized":"2024-04-16T16:13:26.961615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.954467+0000","last_change":"2024-04-16T16:13:25.937881+0000","last_active":"2024-04-16T16:13:25.954467+0000","last_peered":"2024-04-16T16:13:25.954467+0000","last_clean":"2024-04-16T16:13:25.954467+0000","last_became_active":"2024-04-16T16:13:25.936555+0000","last_became_peered":"2024-04-16T16:13:25.936555+0000","last_unstale":"2024-04-16T16:13:25.954467+0000","last_undegraded":"2024-04-16T16:13:25.954467+0000","last_fullsized":"2024-04-16T16:13:25.954467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:45.411606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.940400+0000","last_change":"2024-04-16T16:13:24.915314+0000","last_active":"2024-04-16T16:13:26.940400+0000","last_peered":"2024-04-16T16:13:26.940400+0000","last_clean":"2024-04-16T16:13:26.940400+0000","last_became_active":"2024-04-16T16:13:24.913831+0000","last_became_peered":"2024-04-16T16:13:24.913831+0000","last_unstale":"2024-04-16T16:13:26.940400+0000","last_undegraded":"2024-04-16T16:13:26.940400+0000","last_fullsized":"2024-04-16T16:13:26.940400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941863+0000","last_change":"2024-04-16T16:13:25.939017+0000","last_active":"2024-04-16T16:13:26.941863+0000","last_peered":"2024-04-16T16:13:26.941863+0000","last_clean":"2024-04-16T16:13:26.941863+0000","last_became_active":"2024-04-16T16:13:25.936578+0000","last_became_peered":"2024-04-16T16:13:25.936578+0000","last_unstale":"2024-04-16T16:13:26.941863+0000","last_undegraded":"2024-04-16T16:13:26.941863+0000","last_fullsized":"2024-04-16T16:13:26.941863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:12.895579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954750+0000","last_change":"2024-04-16T16:13:24.909354+0000","last_active":"2024-04-16T16:13:26.954750+0000","last_peered":"2024-04-16T16:13:26.954750+0000","last_clean":"2024-04-16T16:13:26.954750+0000","last_became_active":"2024-04-16T16:13:24.909073+0000","last_became_peered":"2024-04-16T16:13:24.909073+0000","last_unstale":"2024-04-16T16:13:26.954750+0000","last_undegraded":"2024-04-16T16:13:26.954750+0000","last_fullsized":"2024-04-16T16:13:26.954750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.943947+0000","last_change":"2024-04-16T16:13:25.940225+0000","last_active":"2024-04-16T16:13:25.943947+0000","last_peered":"2024-04-16T16:13:25.943947+0000","last_clean":"2024-04-16T16:13:25.943947+0000","last_became_active":"2024-04-16T16:13:25.939871+0000","last_became_peered":"2024-04-16T16:13:25.939871+0000","last_unstale":"2024-04-16T16:13:25.943947+0000","last_undegraded":"2024-04-16T16:13:25.943947+0000","last_fullsized":"2024-04-16T16:13:25.943947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:26.052535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954700+0000","last_change":"2024-04-16T16:13:24.907229+0000","last_active":"2024-04-16T16:13:26.954700+0000","last_peered":"2024-04-16T16:13:26.954700+0000","last_clean":"2024-04-16T16:13:26.954700+0000","last_became_active":"2024-04-16T16:13:24.906948+0000","last_became_peered":"2024-04-16T16:13:24.906948+0000","last_unstale":"2024-04-16T16:13:26.954700+0000","last_undegraded":"2024-04-16T16:13:26.954700+0000","last_fullsized":"2024-04-16T16:13:26.954700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:11.877642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.965771+0000","last_change":"2024-04-16T16:13:25.963968+0000","last_active":"2024-04-16T16:13:25.965771+0000","last_peered":"2024-04-16T16:13:25.965771+0000","last_clean":"2024-04-16T16:13:25.965771+0000","last_became_active":"2024-04-16T16:13:25.963699+0000","last_became_peered":"2024-04-16T16:13:25.963699+0000","last_unstale":"2024-04-16T16:13:25.965771+0000","last_undegraded":"2024-04-16T16:13:25.965771+0000","last_fullsized":"2024-04-16T16:13:25.965771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:07.860908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942898+0000","last_change":"2024-04-16T16:13:24.924622+0000","last_active":"2024-04-16T16:13:26.942898+0000","last_peered":"2024-04-16T16:13:26.942898+0000","last_clean":"2024-04-16T16:13:26.942898+0000","last_became_active":"2024-04-16T16:13:24.923313+0000","last_became_peered":"2024-04-16T16:13:24.923313+0000","last_unstale":"2024-04-16T16:13:26.942898+0000","last_undegraded":"2024-04-16T16:13:26.942898+0000","last_fullsized":"2024-04-16T16:13:26.942898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960479+0000","last_change":"2024-04-16T16:13:25.938982+0000","last_active":"2024-04-16T16:13:26.960479+0000","last_peered":"2024-04-16T16:13:26.960479+0000","last_clean":"2024-04-16T16:13:26.960479+0000","last_became_active":"2024-04-16T16:13:25.938694+0000","last_became_peered":"2024-04-16T16:13:25.938694+0000","last_unstale":"2024-04-16T16:13:26.960479+0000","last_undegraded":"2024-04-16T16:13:26.960479+0000","last_fullsized":"2024-04-16T16:13:26.960479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:10.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971847+0000","last_change":"2024-04-16T16:13:25.944450+0000","last_active":"2024-04-16T16:13:26.971847+0000","last_peered":"2024-04-16T16:13:26.971847+0000","last_clean":"2024-04-16T16:13:26.971847+0000","last_became_active":"2024-04-16T16:13:25.941384+0000","last_became_peered":"2024-04-16T16:13:25.941384+0000","last_unstale":"2024-04-16T16:13:26.971847+0000","last_undegraded":"2024-04-16T16:13:26.971847+0000","last_fullsized":"2024-04-16T16:13:26.971847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:40:10.575471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942812+0000","last_change":"2024-04-16T16:13:24.925176+0000","last_active":"2024-04-16T16:13:26.942812+0000","last_peered":"2024-04-16T16:13:26.942812+0000","last_clean":"2024-04-16T16:13:26.942812+0000","last_became_active":"2024-04-16T16:13:24.924863+0000","last_became_peered":"2024-04-16T16:13:24.924863+0000","last_unstale":"2024-04-16T16:13:26.942812+0000","last_undegraded":"2024-04-16T16:13:26.942812+0000","last_fullsized":"2024-04-16T16:13:26.942812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:11.877642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971280+0000","last_change":"2024-04-16T16:13:25.944098+0000","last_active":"2024-04-16T16:13:26.971280+0000","last_peered":"2024-04-16T16:13:26.971280+0000","last_clean":"2024-04-16T16:13:26.971280+0000","last_became_active":"2024-04-16T16:13:25.943843+0000","last_became_peered":"2024-04-16T16:13:25.943843+0000","last_unstale":"2024-04-16T16:13:26.971280+0000","last_undegraded":"2024-04-16T16:13:26.971280+0000","last_fullsized":"2024-04-16T16:13:26.971280+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:39.592952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943140+0000","last_change":"2024-04-16T16:13:24.923704+0000","last_active":"2024-04-16T16:13:26.943140+0000","last_peered":"2024-04-16T16:13:26.943140+0000","last_clean":"2024-04-16T16:13:26.943140+0000","last_became_active":"2024-04-16T16:13:24.923214+0000","last_became_peered":"2024-04-16T16:13:24.923214+0000","last_unstale":"2024-04-16T16:13:26.943140+0000","last_undegraded":"2024-04-16T16:13:26.943140+0000","last_fullsized":"2024-04-16T16:13:26.943140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:24.722256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.966716+0000","last_change":"2024-04-16T16:13:25.955067+0000","last_active":"2024-04-16T16:13:25.966716+0000","last_peered":"2024-04-16T16:13:25.966716+0000","last_clean":"2024-04-16T16:13:25.966716+0000","last_became_active":"2024-04-16T16:13:25.954780+0000","last_became_peered":"2024-04-16T16:13:25.954780+0000","last_unstale":"2024-04-16T16:13:25.966716+0000","last_undegraded":"2024-04-16T16:13:25.966716+0000","last_fullsized":"2024-04-16T16:13:25.966716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:32:38.024739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.967682+0000","last_change":"2024-04-16T16:13:24.913036+0000","last_active":"2024-04-16T16:13:25.967682+0000","last_peered":"2024-04-16T16:13:25.967682+0000","last_clean":"2024-04-16T16:13:25.967682+0000","last_became_active":"2024-04-16T16:13:24.911928+0000","last_became_peered":"2024-04-16T16:13:24.911928+0000","last_unstale":"2024-04-16T16:13:25.967682+0000","last_undegraded":"2024-04-16T16:13:25.967682+0000","last_fullsized":"2024-04-16T16:13:25.967682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.950077+0000","last_change":"2024-04-16T16:13:25.946331+0000","last_active":"2024-04-16T16:13:25.950077+0000","last_peered":"2024-04-16T16:13:25.950077+0000","last_clean":"2024-04-16T16:13:25.950077+0000","last_became_active":"2024-04-16T16:13:25.946053+0000","last_became_peered":"2024-04-16T16:13:25.946053+0000","last_unstale":"2024-04-16T16:13:25.950077+0000","last_undegraded":"2024-04-16T16:13:25.950077+0000","last_fullsized":"2024-04-16T16:13:25.950077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:42.567277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-16T16:13:25.967407+0000","last_change":"2024-04-16T16:13:24.909349+0000","last_active":"2024-04-16T16:13:25.967407+0000","last_peered":"2024-04-16T16:13:25.967407+0000","last_clean":"2024-04-16T16:13:25.967407+0000","last_became_active":"2024-04-16T16:13:24.909069+0000","last_became_peered":"2024-04-16T16:13:24.909069+0000","last_unstale":"2024-04-16T16:13:25.967407+0000","last_undegraded":"2024-04-16T16:13:25.967407+0000","last_fullsized":"2024-04-16T16:13:25.967407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:46:08.208026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":811008,"data_stored":660311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:13:30.993 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-16T16:13:30.994 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-16T16:13:30.994 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-16T16:13:30.994 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-16T16:13:30.994 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-16T16:13:31.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.248+0000 7f80e5e00640 1 Processor -- start 2024-04-16T16:13:31.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.248+0000 7f80e5e00640 1 -- start start 2024-04-16T16:13:31.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.250+0000 7f80e5e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f80e015e8f0 0x7f80e00cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.250+0000 7f80e5e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80e01632a0 con 0x7f80e00ce9a0 2024-04-16T16:13:31.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.250+0000 7f80e5e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80e015ecd0 con 0x7f80e00cf080 2024-04-16T16:13:31.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.250+0000 7f80e5e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f80e00ce2e0 con 0x7f80e015e8f0 2024-04-16T16:13:31.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.250+0000 7f80dfe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f80e00ce9a0 0x7f80e0162480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40324/0 (socket says 172.21.15.45:40324) 2024-04-16T16:13:31.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.250+0000 7f80dfe00640 1 -- 172.21.15.45:0/1555642501 learned_addr learned my addr 172.21.15.45:0/1555642501 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:31.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80e4a00640 1 --2- 172.21.15.45:0/1555642501 >> v2:172.21.15.103:3300/0 conn(0x7f80e015e8f0 0x7f80e00cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:31.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80df400640 1 -- 172.21.15.45:0/1555642501 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2304853967 0 0) 0x7f80e01632a0 con 0x7f80e00ce9a0 2024-04-16T16:13:31.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80df400640 1 -- 172.21.15.45:0/1555642501 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80c8003520 con 0x7f80e00ce9a0 2024-04-16T16:13:31.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80df400640 1 -- 172.21.15.45:0/1555642501 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3250005384 0 0) 0x7f80c8003520 con 0x7f80e00ce9a0 2024-04-16T16:13:31.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80df400640 1 -- 172.21.15.45:0/1555642501 >> v2:172.21.15.103:3300/0 conn(0x7f80e015e8f0 msgr2=0x7f80e00cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:31.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80df400640 1 --2- 172.21.15.45:0/1555642501 >> v2:172.21.15.103:3300/0 conn(0x7f80e015e8f0 0x7f80e00cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80df400640 1 -- 172.21.15.45:0/1555642501 >> v1:172.21.15.103:6789/0 conn(0x7f80e00cf080 legacy=0x7f80e0162b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:31.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80df400640 1 -- 172.21.15.45:0/1555642501 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e00cd6d0 con 0x7f80e00ce9a0 2024-04-16T16:13:31.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80df400640 1 -- 172.21.15.45:0/1555642501 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f80cc001670 con 0x7f80e00ce9a0 2024-04-16T16:13:31.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80df400640 1 -- 172.21.15.45:0/1555642501 >> v1:172.21.15.45:6789/0 conn(0x7f80e00ce9a0 legacy=0x7f80e0162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:31.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80df400640 1 --2- 172.21.15.45:0/1555642501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80c8003a00 0x7f80c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80df400640 1 --2- 172.21.15.45:0/1555642501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80c8004800 0x7f80c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80df400640 1 --2- 172.21.15.45:0/1555642501 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80c80051c0 0x7f80c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80df400640 1 -- 172.21.15.45:0/1555642501 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f80e00ce2e0 con 0x7f80c8004800 2024-04-16T16:13:31.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80df400640 1 -- 172.21.15.45:0/1555642501 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f80c800c0c0 con 0x7f80c8003a00 2024-04-16T16:13:31.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.251+0000 7f80df400640 1 -- 172.21.15.45:0/1555642501 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f80c800c2c0 con 0x7f80c80051c0 2024-04-16T16:13:31.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.252+0000 7f80dfe00640 1 --2- 172.21.15.45:0/1555642501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80c8004800 0x7f80c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:31.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.252+0000 7f80e5400640 1 --2- 172.21.15.45:0/1555642501 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80c80051c0 0x7f80c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:31.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.252+0000 7f80dfe00640 1 -- 172.21.15.45:0/1555642501 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80c80051c0 msgr2=0x7f80c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:31.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.252+0000 7f80dfe00640 1 --2- 172.21.15.45:0/1555642501 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80c80051c0 0x7f80c800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.252+0000 7f80dfe00640 1 -- 172.21.15.45:0/1555642501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80c8003a00 msgr2=0x7f80c8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:31.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.252+0000 7f80dfe00640 1 --2- 172.21.15.45:0/1555642501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80c8003a00 0x7f80c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.252+0000 7f80dfe00640 1 -- 172.21.15.45:0/1555642501 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c8003520 con 0x7f80c8004800 2024-04-16T16:13:31.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.252+0000 7f80e5400640 1 -- 172.21.15.45:0/1555642501 reap_dead start 2024-04-16T16:13:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.254+0000 7f80dfe00640 1 --2- 172.21.15.45:0/1555642501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80c8004800 0x7f80c8004c80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f80cc00c160 tx=0x7f80c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f9ae7c8078ea43 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.254+0000 7f80df400640 1 -- 172.21.15.45:0/1555642501 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc007c60 con 0x7f80c8004800 2024-04-16T16:13:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.254+0000 7f80df400640 1 -- 172.21.15.45:0/1555642501 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80cc030070 con 0x7f80c8004800 2024-04-16T16:13:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.254+0000 7f80df400640 1 -- 172.21.15.45:0/1555642501 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc005db0 con 0x7f80c8004800 2024-04-16T16:13:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.255+0000 7f80e5e00640 1 -- 172.21.15.45:0/1555642501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80c8004800 msgr2=0x7f80c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.255+0000 7f80e5e00640 1 --2- 172.21.15.45:0/1555642501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80c8004800 0x7f80c8004c80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f80cc00c160 tx=0x7f80c8005830 comp rx=0 tx=0).stop 2024-04-16T16:13:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.255+0000 7f80e5e00640 1 -- 172.21.15.45:0/1555642501 shutdown_connections 2024-04-16T16:13:31.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.255+0000 7f80e5e00640 1 --2- 172.21.15.45:0/1555642501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80c8004800 0x7f80c8004c80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.255+0000 7f80e5e00640 1 -- 172.21.15.45:0/1555642501 >> 172.21.15.45:0/1555642501 conn(0x7f80e006b930 msgr2=0x7f80e006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:31.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.255+0000 7f80e5e00640 1 -- 172.21.15.45:0/1555642501 shutdown_connections 2024-04-16T16:13:31.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.255+0000 7f80e5e00640 1 -- 172.21.15.45:0/1555642501 wait complete. 2024-04-16T16:13:31.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.257+0000 7f80e5e00640 1 Processor -- start 2024-04-16T16:13:31.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.257+0000 7f80e5e00640 1 -- start start 2024-04-16T16:13:31.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.258+0000 7f80e5e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80c8004800 0x7f80e0161880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.258+0000 7f80e5e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80e014ebd0 0x7f80e015cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.258+0000 7f80e5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80e015d010 0x7f80e015f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.258+0000 7f80e5e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f80e00cdf70 con 0x7f80e015d010 2024-04-16T16:13:31.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.258+0000 7f80e5e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f80e00cdc90 con 0x7f80e014ebd0 2024-04-16T16:13:31.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.258+0000 7f80e5e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f80e00cd360 con 0x7f80c8004800 2024-04-16T16:13:31.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.258+0000 7f80e5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80e015d010 0x7f80e015f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:31.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.258+0000 7f80e5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80e015d010 0x7f80e015f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49700/0 (socket says 172.21.15.45:49700) 2024-04-16T16:13:31.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.258+0000 7f80e5400640 1 -- 172.21.15.45:0/3457272131 learned_addr learned my addr 172.21.15.45:0/3457272131 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.259+0000 7f80e4a00640 1 --2- 172.21.15.45:0/3457272131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80c8004800 0x7f80e0161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.259+0000 7f80dfe00640 1 --2- 172.21.15.45:0/3457272131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80e014ebd0 0x7f80e015cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.259+0000 7f80e5400640 1 -- 172.21.15.45:0/3457272131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80c8004800 msgr2=0x7f80e0161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.259+0000 7f80e5400640 1 --2- 172.21.15.45:0/3457272131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80c8004800 0x7f80e0161880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.259+0000 7f80e5400640 1 -- 172.21.15.45:0/3457272131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80e014ebd0 msgr2=0x7f80e015cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.259+0000 7f80e5400640 1 --2- 172.21.15.45:0/3457272131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80e014ebd0 0x7f80e015cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.259+0000 7f80e5400640 1 -- 172.21.15.45:0/3457272131 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e0155570 con 0x7f80e015d010 2024-04-16T16:13:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.259+0000 7f80e5400640 1 --2- 172.21.15.45:0/3457272131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80e015d010 0x7f80e015f1e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f80e0070030 tx=0x7f80d800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.259+0000 7f80dcc00640 1 -- 172.21.15.45:0/3457272131 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d8003b70 con 0x7f80e015d010 2024-04-16T16:13:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.259+0000 7f80dcc00640 1 -- 172.21.15.45:0/3457272131 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80d8017070 con 0x7f80e015d010 2024-04-16T16:13:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.259+0000 7f80dcc00640 1 -- 172.21.15.45:0/3457272131 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d8005070 con 0x7f80e015d010 2024-04-16T16:13:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.259+0000 7f80e5e00640 1 -- 172.21.15.45:0/3457272131 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e0156190 con 0x7f80e015d010 2024-04-16T16:13:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.260+0000 7f80e5e00640 1 -- 172.21.15.45:0/3457272131 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e0154fb0 con 0x7f80e015d010 2024-04-16T16:13:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.261+0000 7f80e5e00640 1 -- 172.21.15.45:0/3457272131 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80e0075780 con 0x7f80e015d010 2024-04-16T16:13:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.262+0000 7f80dcc00640 1 -- 172.21.15.45:0/3457272131 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f80d8005210 con 0x7f80e015d010 2024-04-16T16:13:31.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.263+0000 7f80dcc00640 1 --2- 172.21.15.45:0/3457272131 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f80c006c8f0 0x7f80c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.263+0000 7f80dcc00640 1 -- 172.21.15.45:0/3457272131 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f80d808b9f0 con 0x7f80e015d010 2024-04-16T16:13:31.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.263+0000 7f80e4a00640 1 --2- 172.21.15.45:0/3457272131 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f80c006c8f0 0x7f80c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:31.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.265+0000 7f80e4a00640 1 --2- 172.21.15.45:0/3457272131 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f80c006c8f0 0x7f80c006edb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f80e006a990 tx=0x7f80d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:31.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.268+0000 7f80dcc00640 1 -- 172.21.15.45:0/3457272131 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f80d8059d80 con 0x7f80e015d010 2024-04-16T16:13:31.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.523+0000 7f80e5e00640 1 -- 172.21.15.45:0/3457272131 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f80e00b5b90 con 0x7f80e015d010 2024-04-16T16:13:31.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.532+0000 7f80dcc00640 1 -- 172.21.15.45:0/3457272131 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+174144 (secure 0 0 0) 0x7f80d801ae40 con 0x7f80e015d010 2024-04-16T16:13:31.534 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.536+0000 7f80e5e00640 1 -- 172.21.15.45:0/3457272131 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f80c006c8f0 msgr2=0x7f80c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.536+0000 7f80e5e00640 1 --2- 172.21.15.45:0/3457272131 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f80c006c8f0 0x7f80c006edb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f80e006a990 tx=0x7f80d4008040 comp rx=0 tx=0).stop 2024-04-16T16:13:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.536+0000 7f80e5e00640 1 -- 172.21.15.45:0/3457272131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80e015d010 msgr2=0x7f80e015f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.536+0000 7f80e5e00640 1 --2- 172.21.15.45:0/3457272131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80e015d010 0x7f80e015f1e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f80e0070030 tx=0x7f80d800dc00 comp rx=0 tx=0).stop 2024-04-16T16:13:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.536+0000 7f80e5e00640 1 -- 172.21.15.45:0/3457272131 shutdown_connections 2024-04-16T16:13:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.536+0000 7f80e5e00640 1 --2- 172.21.15.45:0/3457272131 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f80c006c8f0 0x7f80c006edb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.536+0000 7f80e5e00640 1 --2- 172.21.15.45:0/3457272131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80e015d010 0x7f80e015f1e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.536+0000 7f80e5e00640 1 --2- 172.21.15.45:0/3457272131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80e014ebd0 0x7f80e015cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.536+0000 7f80e5e00640 1 --2- 172.21.15.45:0/3457272131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80c8004800 0x7f80e0161880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.536+0000 7f80e5e00640 1 -- 172.21.15.45:0/3457272131 >> 172.21.15.45:0/3457272131 conn(0x7f80e006b930 msgr2=0x7f80e015dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.536+0000 7f80e5e00640 1 -- 172.21.15.45:0/3457272131 shutdown_connections 2024-04-16T16:13:31.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.536+0000 7f80e5e00640 1 -- 172.21.15.45:0/3457272131 wait complete. 2024-04-16T16:13:31.556 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2711977587},{"type":"v1","addr":"172.21.15.103:6833","nonce":2711977587}]},"active_addr":"172.21.15.103:6833/2711977587","active_change":"2024-04-16T16:13:11.070237+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":3554107853}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":2679792854}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":3010240888}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":2318159422}]}]} 2024-04-16T16:13:31.559 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-16T16:13:31.559 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-16T16:13:31.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:13:31.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.778+0000 7fa175400640 1 Processor -- start 2024-04-16T16:13:31.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.778+0000 7fa175400640 1 -- start start 2024-04-16T16:13:31.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.780+0000 7fa175400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa170144ff0 0x7fa1700cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.780+0000 7fa175400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa17015c700 con 0x7fa1700c5d80 2024-04-16T16:13:31.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.780+0000 7fa175400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa17015c9c0 con 0x7fa17006dfb0 2024-04-16T16:13:31.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.780+0000 7fa175400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa17015d2f0 con 0x7fa170144ff0 2024-04-16T16:13:31.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.780+0000 7fa174a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa1700c5d80 0x7fa17015d780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40326/0 (socket says 172.21.15.45:40326) 2024-04-16T16:13:31.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.780+0000 7fa174a00640 1 -- 172.21.15.45:0/2916878551 learned_addr learned my addr 172.21.15.45:0/2916878551 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:31.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16fe00640 1 --2- 172.21.15.45:0/2916878551 >> v2:172.21.15.103:3300/0 conn(0x7fa170144ff0 0x7fa1700cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:31.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 260126614 0 0) 0x7fa17015c700 con 0x7fa1700c5d80 2024-04-16T16:13:31.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa154003520 con 0x7fa1700c5d80 2024-04-16T16:13:31.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4105468325 0 0) 0x7fa17015c9c0 con 0x7fa17006dfb0 2024-04-16T16:13:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa17015c700 con 0x7fa17006dfb0 2024-04-16T16:13:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3918410924 0 0) 0x7fa154003520 con 0x7fa1700c5d80 2024-04-16T16:13:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 >> v2:172.21.15.103:3300/0 conn(0x7fa170144ff0 msgr2=0x7fa1700cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 --2- 172.21.15.45:0/2916878551 >> v2:172.21.15.103:3300/0 conn(0x7fa170144ff0 0x7fa1700cfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 >> v1:172.21.15.103:6789/0 conn(0x7fa17006dfb0 legacy=0x7fa1700c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa17015cca0 con 0x7fa1700c5d80 2024-04-16T16:13:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fa1680019f0 con 0x7fa1700c5d80 2024-04-16T16:13:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 >> v1:172.21.15.45:6789/0 conn(0x7fa1700c5d80 legacy=0x7fa17015d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 --2- 172.21.15.45:0/2916878551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa154003b40 0x7fa154003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 --2- 172.21.15.45:0/2916878551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa154004800 0x7fa154004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 --2- 172.21.15.45:0/2916878551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1540051c0 0x7fa15400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa17015d2f0 con 0x7fa154004800 2024-04-16T16:13:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa15400e320 con 0x7fa1540051c0 2024-04-16T16:13:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.781+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa15400e520 con 0x7fa154003b40 2024-04-16T16:13:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.782+0000 7fa167e00640 1 --2- 172.21.15.45:0/2916878551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa154004800 0x7fa154004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:31.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.782+0000 7fa16fe00640 1 --2- 172.21.15.45:0/2916878551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa154003b40 0x7fa154003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:31.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.782+0000 7fa174a00640 1 --2- 172.21.15.45:0/2916878551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1540051c0 0x7fa15400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:31.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.782+0000 7fa167e00640 1 -- 172.21.15.45:0/2916878551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa154003b40 msgr2=0x7fa154003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:31.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.782+0000 7fa167e00640 1 --2- 172.21.15.45:0/2916878551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa154003b40 0x7fa154003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.782+0000 7fa167e00640 1 -- 172.21.15.45:0/2916878551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1540051c0 msgr2=0x7fa15400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:31.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.782+0000 7fa167e00640 1 --2- 172.21.15.45:0/2916878551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1540051c0 0x7fa15400dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.782+0000 7fa167e00640 1 -- 172.21.15.45:0/2916878551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa154003520 con 0x7fa154004800 2024-04-16T16:13:31.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.782+0000 7fa174a00640 1 -- 172.21.15.45:0/2916878551 reap_dead start 2024-04-16T16:13:31.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.783+0000 7fa167e00640 1 --2- 172.21.15.45:0/2916878551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa154004800 0x7fa154004c80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa1580018a0 tx=0x7fa154005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ed3e0d855ae96 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:31.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.783+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa158037070 con 0x7fa154004800 2024-04-16T16:13:31.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.783+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1580052d0 con 0x7fa154004800 2024-04-16T16:13:31.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.783+0000 7fa16f400640 1 -- 172.21.15.45:0/2916878551 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1580014f0 con 0x7fa154004800 2024-04-16T16:13:31.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.784+0000 7fa175400640 1 -- 172.21.15.45:0/2916878551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa154004800 msgr2=0x7fa154004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:31.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.784+0000 7fa175400640 1 --2- 172.21.15.45:0/2916878551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa154004800 0x7fa154004c80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa1580018a0 tx=0x7fa154005830 comp rx=0 tx=0).stop 2024-04-16T16:13:31.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.784+0000 7fa175400640 1 -- 172.21.15.45:0/2916878551 shutdown_connections 2024-04-16T16:13:31.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.784+0000 7fa175400640 1 --2- 172.21.15.45:0/2916878551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa154004800 0x7fa154004c80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.784+0000 7fa175400640 1 -- 172.21.15.45:0/2916878551 >> 172.21.15.45:0/2916878551 conn(0x7fa17006b930 msgr2=0x7fa17006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:31.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.784+0000 7fa175400640 1 -- 172.21.15.45:0/2916878551 shutdown_connections 2024-04-16T16:13:31.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.784+0000 7fa175400640 1 -- 172.21.15.45:0/2916878551 wait complete. 2024-04-16T16:13:31.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.786+0000 7fa175400640 1 Processor -- start 2024-04-16T16:13:31.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.786+0000 7fa175400640 1 -- start start 2024-04-16T16:13:31.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.787+0000 7fa175400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa154004800 0x7fa1701f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.787+0000 7fa175400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1701e74a0 0x7fa1701e7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.787+0000 7fa175400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1700cd970 0x7fa1700cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.787+0000 7fa175400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa1700c5670 con 0x7fa1701e74a0 2024-04-16T16:13:31.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.787+0000 7fa175400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa1701453d0 con 0x7fa154004800 2024-04-16T16:13:31.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.787+0000 7fa175400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa1700c57f0 con 0x7fa1700cd970 2024-04-16T16:13:31.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.787+0000 7fa167e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1701e74a0 0x7fa1701e7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:31.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.787+0000 7fa167e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1701e74a0 0x7fa1701e7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49712/0 (socket says 172.21.15.45:49712) 2024-04-16T16:13:31.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.787+0000 7fa167e00640 1 -- 172.21.15.45:0/2143310727 learned_addr learned my addr 172.21.15.45:0/2143310727 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:31.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.787+0000 7fa174a00640 1 --2- 172.21.15.45:0/2143310727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1700cd970 0x7fa1700cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:31.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.787+0000 7fa167e00640 1 -- 172.21.15.45:0/2143310727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1700cd970 msgr2=0x7fa1700cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:31.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.787+0000 7fa167e00640 1 --2- 172.21.15.45:0/2143310727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1700cd970 0x7fa1700cddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.787+0000 7fa167e00640 1 -- 172.21.15.45:0/2143310727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa154004800 msgr2=0x7fa1701f3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:31.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.787+0000 7fa167e00640 1 --2- 172.21.15.45:0/2143310727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa154004800 0x7fa1701f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:31.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.787+0000 7fa167e00640 1 -- 172.21.15.45:0/2143310727 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1701e4ce0 con 0x7fa1701e74a0 2024-04-16T16:13:31.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.788+0000 7fa167e00640 1 --2- 172.21.15.45:0/2143310727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1701e74a0 0x7fa1701e7920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa158035040 tx=0x7fa1580012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:31.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.788+0000 7fa16cc00640 1 -- 172.21.15.45:0/2143310727 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa158042020 con 0x7fa1701e74a0 2024-04-16T16:13:31.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.788+0000 7fa16cc00640 1 -- 172.21.15.45:0/2143310727 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa158038040 con 0x7fa1701e74a0 2024-04-16T16:13:31.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.788+0000 7fa16cc00640 1 -- 172.21.15.45:0/2143310727 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa158037070 con 0x7fa1701e74a0 2024-04-16T16:13:31.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.788+0000 7fa175400640 1 -- 172.21.15.45:0/2143310727 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1700ce630 con 0x7fa1701e74a0 2024-04-16T16:13:31.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.788+0000 7fa175400640 1 -- 172.21.15.45:0/2143310727 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1701e5260 con 0x7fa1701e74a0 2024-04-16T16:13:31.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.789+0000 7fa175400640 1 -- 172.21.15.45:0/2143310727 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa170075780 con 0x7fa1701e74a0 2024-04-16T16:13:31.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.790+0000 7fa16cc00640 1 -- 172.21.15.45:0/2143310727 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa158045020 con 0x7fa1701e74a0 2024-04-16T16:13:31.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.791+0000 7fa16cc00640 1 --2- 172.21.15.45:0/2143310727 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa14c06c920 0x7fa14c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:31.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.791+0000 7fa16cc00640 1 -- 172.21.15.45:0/2143310727 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa1580ad330 con 0x7fa1701e74a0 2024-04-16T16:13:31.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.792+0000 7fa16fe00640 1 --2- 172.21.15.45:0/2143310727 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa14c06c920 0x7fa14c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:31.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.793+0000 7fa16fe00640 1 --2- 172.21.15.45:0/2143310727 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa14c06c920 0x7fa14c06ede0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa160006d40 tx=0x7fa160006bd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:31.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:31.796+0000 7fa16cc00640 1 -- 172.21.15.45:0/2143310727 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa15804c030 con 0x7fa1701e74a0 2024-04-16T16:13:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.002+0000 7fa175400640 1 -- 172.21.15.45:0/2143310727 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa1700b3cc0 con 0x7fa1701e74a0 2024-04-16T16:13:32.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.004+0000 7fa16cc00640 1 -- 172.21.15.45:0/2143310727 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7fa15807b740 con 0x7fa1701e74a0 2024-04-16T16:13:32.005 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:32.005 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":21,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:26.918617+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:13:32.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.007+0000 7fa175400640 1 -- 172.21.15.45:0/2143310727 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa14c06c920 msgr2=0x7fa14c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.007+0000 7fa175400640 1 --2- 172.21.15.45:0/2143310727 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa14c06c920 0x7fa14c06ede0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa160006d40 tx=0x7fa160006bd0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.007+0000 7fa175400640 1 -- 172.21.15.45:0/2143310727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1701e74a0 msgr2=0x7fa1701e7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.007+0000 7fa175400640 1 --2- 172.21.15.45:0/2143310727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1701e74a0 0x7fa1701e7920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa158035040 tx=0x7fa1580012d0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.008+0000 7fa175400640 1 -- 172.21.15.45:0/2143310727 shutdown_connections 2024-04-16T16:13:32.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.008+0000 7fa175400640 1 --2- 172.21.15.45:0/2143310727 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa14c06c920 0x7fa14c06ede0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.008+0000 7fa175400640 1 --2- 172.21.15.45:0/2143310727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1700cd970 0x7fa1700cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.008+0000 7fa175400640 1 --2- 172.21.15.45:0/2143310727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1701e74a0 0x7fa1701e7920 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.008+0000 7fa175400640 1 --2- 172.21.15.45:0/2143310727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa154004800 0x7fa1701f3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.008+0000 7fa175400640 1 -- 172.21.15.45:0/2143310727 >> 172.21.15.45:0/2143310727 conn(0x7fa17006b930 msgr2=0x7fa17015e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.008+0000 7fa175400640 1 -- 172.21.15.45:0/2143310727 shutdown_connections 2024-04-16T16:13:32.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.008+0000 7fa175400640 1 -- 172.21.15.45:0/2143310727 wait complete. 2024-04-16T16:13:32.028 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-16T16:13:32.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:13:32.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.252+0000 7f9d61800640 1 Processor -- start 2024-04-16T16:13:32.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.253+0000 7f9d61800640 1 -- start start 2024-04-16T16:13:32.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.254+0000 7f9d61800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9d5c161e90 0x7f9d5c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.254+0000 7f9d61800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d5c0d3080 con 0x7f9d5c0d52d0 2024-04-16T16:13:32.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.254+0000 7f9d61800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d5c0d3360 con 0x7f9d5c162270 2024-04-16T16:13:32.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.254+0000 7f9d61800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9d5c0d2e30 con 0x7f9d5c161e90 2024-04-16T16:13:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.255+0000 7f9d60e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9d5c0d52d0 0x7f9d5c06ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40328/0 (socket says 172.21.15.45:40328) 2024-04-16T16:13:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.255+0000 7f9d60e00640 1 -- 172.21.15.45:0/677832248 learned_addr learned my addr 172.21.15.45:0/677832248 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.255+0000 7f9d5be00640 1 --2- 172.21.15.45:0/677832248 >> v2:172.21.15.103:3300/0 conn(0x7f9d5c161e90 0x7f9d5c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.255+0000 7f9d5aa00640 1 -- 172.21.15.45:0/677832248 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2702135281 0 0) 0x7f9d5c0d3080 con 0x7f9d5c0d52d0 2024-04-16T16:13:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.255+0000 7f9d5aa00640 1 -- 172.21.15.45:0/677832248 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d3c003520 con 0x7f9d5c0d52d0 2024-04-16T16:13:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5aa00640 1 -- 172.21.15.45:0/677832248 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4170649778 0 0) 0x7f9d3c003520 con 0x7f9d5c0d52d0 2024-04-16T16:13:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5aa00640 1 -- 172.21.15.45:0/677832248 >> v2:172.21.15.103:3300/0 conn(0x7f9d5c161e90 msgr2=0x7f9d5c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5aa00640 1 --2- 172.21.15.45:0/677832248 >> v2:172.21.15.103:3300/0 conn(0x7f9d5c161e90 0x7f9d5c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5aa00640 1 -- 172.21.15.45:0/677832248 >> v1:172.21.15.103:6789/0 conn(0x7f9d5c162270 legacy=0x7f9d5c0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5aa00640 1 -- 172.21.15.45:0/677832248 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5c0d3640 con 0x7f9d5c0d52d0 2024-04-16T16:13:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5aa00640 1 -- 172.21.15.45:0/677832248 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f9d54001670 con 0x7f9d5c0d52d0 2024-04-16T16:13:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5aa00640 1 -- 172.21.15.45:0/677832248 >> v1:172.21.15.45:6789/0 conn(0x7f9d5c0d52d0 legacy=0x7f9d5c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5aa00640 1 --2- 172.21.15.45:0/677832248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d3c0039b0 0x7f9d3c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5aa00640 1 --2- 172.21.15.45:0/677832248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d3c004800 0x7f9d3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5aa00640 1 --2- 172.21.15.45:0/677832248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d3c0051c0 0x7f9d3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5aa00640 1 -- 172.21.15.45:0/677832248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5c0d2e30 con 0x7f9d3c004800 2024-04-16T16:13:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5aa00640 1 -- 172.21.15.45:0/677832248 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9d3c00c060 con 0x7f9d3c0051c0 2024-04-16T16:13:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5aa00640 1 -- 172.21.15.45:0/677832248 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9d3c00c260 con 0x7f9d3c0039b0 2024-04-16T16:13:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5b400640 1 --2- 172.21.15.45:0/677832248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d3c004800 0x7f9d3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d60e00640 1 --2- 172.21.15.45:0/677832248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d3c0051c0 0x7f9d3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5be00640 1 --2- 172.21.15.45:0/677832248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d3c0039b0 0x7f9d3c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5b400640 1 -- 172.21.15.45:0/677832248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d3c0039b0 msgr2=0x7f9d3c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5b400640 1 --2- 172.21.15.45:0/677832248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d3c0039b0 0x7f9d3c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.256+0000 7f9d5b400640 1 -- 172.21.15.45:0/677832248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d3c0051c0 msgr2=0x7f9d3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.257+0000 7f9d5b400640 1 --2- 172.21.15.45:0/677832248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d3c0051c0 0x7f9d3c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.257+0000 7f9d5b400640 1 -- 172.21.15.45:0/677832248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d3c003520 con 0x7f9d3c004800 2024-04-16T16:13:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.257+0000 7f9d60e00640 1 -- 172.21.15.45:0/677832248 reap_dead start 2024-04-16T16:13:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.258+0000 7f9d5b400640 1 --2- 172.21.15.45:0/677832248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d3c004800 0x7f9d3c004c80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9d44001e20 tx=0x7f9d3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59fd909ec452250 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.258+0000 7f9d5aa00640 1 -- 172.21.15.45:0/677832248 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44001a70 con 0x7f9d3c004800 2024-04-16T16:13:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.258+0000 7f9d5aa00640 1 -- 172.21.15.45:0/677832248 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d44003b70 con 0x7f9d3c004800 2024-04-16T16:13:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.258+0000 7f9d5aa00640 1 -- 172.21.15.45:0/677832248 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d440050d0 con 0x7f9d3c004800 2024-04-16T16:13:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.258+0000 7f9d61800640 1 -- 172.21.15.45:0/677832248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d3c004800 msgr2=0x7f9d3c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.258+0000 7f9d61800640 1 --2- 172.21.15.45:0/677832248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d3c004800 0x7f9d3c004c80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9d44001e20 tx=0x7f9d3c005830 comp rx=0 tx=0).stop 2024-04-16T16:13:32.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.259+0000 7f9d61800640 1 -- 172.21.15.45:0/677832248 shutdown_connections 2024-04-16T16:13:32.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.259+0000 7f9d61800640 1 --2- 172.21.15.45:0/677832248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d3c004800 0x7f9d3c004c80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.259+0000 7f9d61800640 1 -- 172.21.15.45:0/677832248 >> 172.21.15.45:0/677832248 conn(0x7f9d5c0d1940 msgr2=0x7f9d5c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.259+0000 7f9d61800640 1 -- 172.21.15.45:0/677832248 shutdown_connections 2024-04-16T16:13:32.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.259+0000 7f9d61800640 1 -- 172.21.15.45:0/677832248 wait complete. 2024-04-16T16:13:32.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.260+0000 7f9d61800640 1 Processor -- start 2024-04-16T16:13:32.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.261+0000 7f9d61800640 1 -- start start 2024-04-16T16:13:32.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d61800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d3c004800 0x7f9d5c0ce470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d61800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d5c1e94b0 0x7f9d5c152870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d61800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d5c1e9930 0x7f9d5c152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d61800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5c0cf950 con 0x7f9d5c1e9930 2024-04-16T16:13:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d61800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5c0cf7d0 con 0x7f9d5c1e94b0 2024-04-16T16:13:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d61800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9d5c0cfad0 con 0x7f9d3c004800 2024-04-16T16:13:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d60e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d5c1e9930 0x7f9d5c152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d60e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d5c1e9930 0x7f9d5c152db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49724/0 (socket says 172.21.15.45:49724) 2024-04-16T16:13:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d60e00640 1 -- 172.21.15.45:0/861001860 learned_addr learned my addr 172.21.15.45:0/861001860 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d5be00640 1 --2- 172.21.15.45:0/861001860 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d3c004800 0x7f9d5c0ce470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d5b400640 1 --2- 172.21.15.45:0/861001860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d5c1e94b0 0x7f9d5c152870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d60e00640 1 -- 172.21.15.45:0/861001860 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d3c004800 msgr2=0x7f9d5c0ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d60e00640 1 --2- 172.21.15.45:0/861001860 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d3c004800 0x7f9d5c0ce470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d60e00640 1 -- 172.21.15.45:0/861001860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d5c1e94b0 msgr2=0x7f9d5c152870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d60e00640 1 --2- 172.21.15.45:0/861001860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d5c1e94b0 0x7f9d5c152870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.262+0000 7f9d60e00640 1 -- 172.21.15.45:0/861001860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5c1e7ef0 con 0x7f9d5c1e9930 2024-04-16T16:13:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.263+0000 7f9d60e00640 1 --2- 172.21.15.45:0/861001860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d5c1e9930 0x7f9d5c152db0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0d03d0 tx=0x7f9d5400e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.263+0000 7f9d53e00640 1 -- 172.21.15.45:0/861001860 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d54019070 con 0x7f9d5c1e9930 2024-04-16T16:13:32.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.263+0000 7f9d53e00640 1 -- 172.21.15.45:0/861001860 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d54010510 con 0x7f9d5c1e9930 2024-04-16T16:13:32.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.263+0000 7f9d53e00640 1 -- 172.21.15.45:0/861001860 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d54010900 con 0x7f9d5c1e9930 2024-04-16T16:13:32.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.263+0000 7f9d61800640 1 -- 172.21.15.45:0/861001860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d5c1e8a50 con 0x7f9d5c1e9930 2024-04-16T16:13:32.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.263+0000 7f9d61800640 1 -- 172.21.15.45:0/861001860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d5c1e8490 con 0x7f9d5c1e9930 2024-04-16T16:13:32.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.264+0000 7f9d61800640 1 -- 172.21.15.45:0/861001860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d5c0d3e40 con 0x7f9d5c1e9930 2024-04-16T16:13:32.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.265+0000 7f9d53e00640 1 -- 172.21.15.45:0/861001860 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9d5401c070 con 0x7f9d5c1e9930 2024-04-16T16:13:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.266+0000 7f9d53e00640 1 --2- 172.21.15.45:0/861001860 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9d3806c920 0x7f9d3806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.266+0000 7f9d53e00640 1 -- 172.21.15.45:0/861001860 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9d5408afc0 con 0x7f9d5c1e9930 2024-04-16T16:13:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.267+0000 7f9d5be00640 1 --2- 172.21.15.45:0/861001860 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9d3806c920 0x7f9d3806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.268+0000 7f9d5be00640 1 --2- 172.21.15.45:0/861001860 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9d3806c920 0x7f9d3806ede0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c009a90 tx=0x7f9d4c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.271+0000 7f9d53e00640 1 -- 172.21.15.45:0/861001860 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9d5c0d3e40 con 0x7f9d5c1e9930 2024-04-16T16:13:32.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.478+0000 7f9d61800640 1 -- 172.21.15.45:0/861001860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9d5c075780 con 0x7f9d5c1e9930 2024-04-16T16:13:32.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.479+0000 7f9d53e00640 1 -- 172.21.15.45:0/861001860 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f9d540593e0 con 0x7f9d5c1e9930 2024-04-16T16:13:32.480 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:32.480 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":21,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:26.918617+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:13:32.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.482+0000 7f9d61800640 1 -- 172.21.15.45:0/861001860 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9d3806c920 msgr2=0x7f9d3806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.482+0000 7f9d61800640 1 --2- 172.21.15.45:0/861001860 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9d3806c920 0x7f9d3806ede0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c009a90 tx=0x7f9d4c008040 comp rx=0 tx=0).stop 2024-04-16T16:13:32.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.482+0000 7f9d61800640 1 -- 172.21.15.45:0/861001860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d5c1e9930 msgr2=0x7f9d5c152db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.482+0000 7f9d61800640 1 --2- 172.21.15.45:0/861001860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d5c1e9930 0x7f9d5c152db0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0d03d0 tx=0x7f9d5400e960 comp rx=0 tx=0).stop 2024-04-16T16:13:32.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.482+0000 7f9d61800640 1 -- 172.21.15.45:0/861001860 shutdown_connections 2024-04-16T16:13:32.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.482+0000 7f9d61800640 1 --2- 172.21.15.45:0/861001860 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9d3806c920 0x7f9d3806ede0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.482+0000 7f9d61800640 1 --2- 172.21.15.45:0/861001860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d5c1e9930 0x7f9d5c152db0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.482+0000 7f9d61800640 1 --2- 172.21.15.45:0/861001860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d5c1e94b0 0x7f9d5c152870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.482+0000 7f9d61800640 1 --2- 172.21.15.45:0/861001860 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d3c004800 0x7f9d5c0ce470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.482+0000 7f9d61800640 1 -- 172.21.15.45:0/861001860 >> 172.21.15.45:0/861001860 conn(0x7f9d5c0d1940 msgr2=0x7f9d5c0b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.482+0000 7f9d61800640 1 -- 172.21.15.45:0/861001860 shutdown_connections 2024-04-16T16:13:32.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.482+0000 7f9d61800640 1 -- 172.21.15.45:0/861001860 wait complete. 2024-04-16T16:13:32.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-16T16:13:32.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-16T16:13:32.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-16T16:13:32.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-16T16:13:32.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-16T16:13:32.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-16T16:13:32.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-16T16:13:32.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-16T16:13:32.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.811+0000 7f6b73e00640 1 Processor -- start 2024-04-16T16:13:32.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.811+0000 7f6b73e00640 1 -- start start 2024-04-16T16:13:32.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.812+0000 7fb635400640 1 Processor -- start 2024-04-16T16:13:32.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.812+0000 7fb635400640 1 -- start start 2024-04-16T16:13:32.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.813+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6b6c0eb060 0x7f6b6c055300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.813+0000 7f6b73e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b6c0387e0 con 0x7f6b6c0eb440 2024-04-16T16:13:32.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.813+0000 7f6b73e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b6c056d90 con 0x7f6b6c0061a0 2024-04-16T16:13:32.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.813+0000 7f6b73e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6b6c057e30 con 0x7f6b6c0eb060 2024-04-16T16:13:32.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.813+0000 7f6b72000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6b6c0eb440 0x7f6b6c0eb8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40336/0 (socket says 172.21.15.45:40336) 2024-04-16T16:13:32.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.813+0000 7f6b72000640 1 -- 172.21.15.45:0/3198504142 learned_addr learned my addr 172.21.15.45:0/3198504142 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:32.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.814+0000 7f6b72a00640 1 --2- 172.21.15.45:0/3198504142 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6b6c0eb060 0x7f6b6c055300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.815+0000 7f6b71600640 1 -- 172.21.15.45:0/3198504142 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3568175219 0 0) 0x7f6b64001110 con 0x7f6b6c0eb440 2024-04-16T16:13:32.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.815+0000 7f6b71600640 1 -- 172.21.15.45:0/3198504142 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b58003520 con 0x7f6b6c0eb440 2024-04-16T16:13:32.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.815+0000 7fb635400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb628034020 0x7fb62804fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.816+0000 7fb635400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb628006a50 0x7fb628006e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.816+0000 7fb635400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb628054c10 con 0x7fb62804b3c0 2024-04-16T16:13:32.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.816+0000 7f6b71600640 1 -- 172.21.15.45:0/3198504142 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2835465134 0 0) 0x7f6b58003520 con 0x7f6b6c0eb440 2024-04-16T16:13:32.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.816+0000 7f6b71600640 1 -- 172.21.15.45:0/3198504142 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6b6c0eb060 msgr2=0x7f6b6c055300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.816+0000 7f6b71600640 1 --2- 172.21.15.45:0/3198504142 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6b6c0eb060 0x7f6b6c055300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.816+0000 7f6b71600640 1 -- 172.21.15.45:0/3198504142 >> v1:172.21.15.103:6789/0 conn(0x7f6b6c0061a0 legacy=0x7f6b6c006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.816+0000 7f6b71600640 1 -- 172.21.15.45:0/3198504142 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6c057070 con 0x7f6b6c0eb440 2024-04-16T16:13:32.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.816+0000 7fb635400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb6280062e0 con 0x7fb628034020 2024-04-16T16:13:32.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.816+0000 7fb635400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb628050120 con 0x7fb628006a50 2024-04-16T16:13:32.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.816+0000 7fb62fe00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb62804b3c0 0x7fb62804f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:51494/0 (socket says 172.21.15.45:51494) 2024-04-16T16:13:32.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.816+0000 7fb62fe00640 1 -- 172.21.15.45:0/2526727617 learned_addr learned my addr 172.21.15.45:0/2526727617 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:32.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.816+0000 7f6b71600640 1 -- 172.21.15.45:0/3198504142 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f6b64001840 con 0x7f6b6c0eb440 2024-04-16T16:13:32.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.816+0000 7f6b71600640 1 -- 172.21.15.45:0/3198504142 >> v1:172.21.15.45:6789/0 conn(0x7f6b6c0eb440 legacy=0x7f6b6c0eb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.817+0000 7f6b71600640 1 --2- 172.21.15.45:0/3198504142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b580039e0 0x7f6b58003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.817+0000 7f6b71600640 1 --2- 172.21.15.45:0/3198504142 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6b58004800 0x7f6b58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.816+0000 7fb62f400640 1 --2- 172.21.15.45:0/2526727617 >> v2:172.21.15.45:3300/0 conn(0x7fb628034020 0x7fb62804fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.817+0000 7f6b71600640 1 --2- 172.21.15.45:0/3198504142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b58005290 0x7f6b5800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.817+0000 7f6b71600640 1 -- 172.21.15.45:0/3198504142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5800c180 con 0x7f6b58005290 2024-04-16T16:13:32.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.817+0000 7f6b71600640 1 -- 172.21.15.45:0/3198504142 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5800c330 con 0x7f6b580039e0 2024-04-16T16:13:32.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.817+0000 7f6b71600640 1 -- 172.21.15.45:0/3198504142 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6b5800c530 con 0x7f6b58004800 2024-04-16T16:13:32.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.817+0000 7fb62ea00640 1 -- 172.21.15.45:0/2526727617 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 712886613 0 0) 0x7fb628054c10 con 0x7fb62804b3c0 2024-04-16T16:13:32.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.817+0000 7fb62ea00640 1 -- 172.21.15.45:0/2526727617 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb618003520 con 0x7fb62804b3c0 2024-04-16T16:13:32.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7f6b72000640 1 --2- 172.21.15.45:0/3198504142 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6b58004800 0x7f6b58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.817+0000 7fb634a00640 1 --2- 172.21.15.45:0/2526727617 >> v2:172.21.15.103:3300/0 conn(0x7fb628006a50 0x7fb628006e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7fb62ea00640 1 -- 172.21.15.45:0/2526727617 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1169263936 0 0) 0x7fb618003520 con 0x7fb62804b3c0 2024-04-16T16:13:32.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7fb62ea00640 1 -- 172.21.15.45:0/2526727617 >> v2:172.21.15.103:3300/0 conn(0x7fb628006a50 msgr2=0x7fb628006e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7fb62ea00640 1 --2- 172.21.15.45:0/2526727617 >> v2:172.21.15.103:3300/0 conn(0x7fb628006a50 0x7fb628006e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7fb62ea00640 1 -- 172.21.15.45:0/2526727617 >> v2:172.21.15.45:3300/0 conn(0x7fb628034020 msgr2=0x7fb62804fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7f01ad800640 1 Processor -- start 2024-04-16T16:13:32.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7fb62ea00640 1 --2- 172.21.15.45:0/2526727617 >> v2:172.21.15.45:3300/0 conn(0x7fb628034020 0x7fb62804fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7f6b73400640 1 --2- 172.21.15.45:0/3198504142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b58005290 0x7f6b5800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7f6b72a00640 1 --2- 172.21.15.45:0/3198504142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b580039e0 0x7f6b58003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7fb62ea00640 1 -- 172.21.15.45:0/2526727617 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb628053a50 con 0x7fb62804b3c0 2024-04-16T16:13:32.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7fb62ea00640 1 -- 172.21.15.45:0/2526727617 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fb6300505e0 con 0x7fb62804b3c0 2024-04-16T16:13:32.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7fb62ea00640 1 -- 172.21.15.45:0/2526727617 >> v1:172.21.15.103:6789/0 conn(0x7fb62804b3c0 legacy=0x7fb62804f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7fb62ea00640 1 --2- 172.21.15.45:0/2526727617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6180039b0 0x7fb618003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7fb62ea00640 1 --2- 172.21.15.45:0/2526727617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb618004800 0x7fb618004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7fb62ea00640 1 --2- 172.21.15.45:0/2526727617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6180051c0 0x7fb61800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7fb62ea00640 1 -- 172.21.15.45:0/2526727617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6280062e0 con 0x7fb6180051c0 2024-04-16T16:13:32.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7fb62ea00640 1 -- 172.21.15.45:0/2526727617 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb628050120 con 0x7fb618004800 2024-04-16T16:13:32.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.819+0000 7f6b72a00640 1 -- 172.21.15.45:0/3198504142 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6b58004800 msgr2=0x7f6b58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7fb62ea00640 1 -- 172.21.15.45:0/2526727617 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb61800c0c0 con 0x7fb6180039b0 2024-04-16T16:13:32.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.819+0000 7f6b72a00640 1 --2- 172.21.15.45:0/3198504142 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6b58004800 0x7f6b58004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.819+0000 7f6b72a00640 1 -- 172.21.15.45:0/3198504142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b58005290 msgr2=0x7f6b5800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.819+0000 7f6b72a00640 1 --2- 172.21.15.45:0/3198504142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b58005290 0x7f6b5800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.819+0000 7f6b72a00640 1 -- 172.21.15.45:0/3198504142 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6c056d90 con 0x7f6b580039e0 2024-04-16T16:13:32.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.819+0000 7fb62f400640 1 --2- 172.21.15.45:0/2526727617 >> v2:172.21.15.45:3300/0 conn(0x7fb628034020 0x7fb62804fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:32.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.818+0000 7f01ad800640 1 -- start start 2024-04-16T16:13:32.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.819+0000 7fb634a00640 1 --2- 172.21.15.45:0/2526727617 >> v2:172.21.15.103:3300/0 conn(0x7fb628006a50 0x7fb628006e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:32.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.820+0000 7fb634a00640 1 --2- 172.21.15.45:0/2526727617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6180051c0 0x7fb61800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.820+0000 7f01ad800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01a8071f40 0x7f01a80723c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.820+0000 7f01ad800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f01a80cd7e0 0x7f01a80cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.820+0000 7f01ad800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01a80cecb0 con 0x7f01a8151ae0 2024-04-16T16:13:32.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.820+0000 7f01ad800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f01a815be60 con 0x7f01a80cd7e0 2024-04-16T16:13:32.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.820+0000 7f01ad800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f01a80cc500 con 0x7f01a8071f40 2024-04-16T16:13:32.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.820+0000 7fb62fe00640 1 --2- 172.21.15.45:0/2526727617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6180039b0 0x7fb618003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.820+0000 7fb634a00640 1 -- 172.21.15.45:0/2526727617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6180039b0 msgr2=0x7fb618003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.820+0000 7fb634a00640 1 --2- 172.21.15.45:0/2526727617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6180039b0 0x7fb618003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.820+0000 7fb634a00640 1 -- 172.21.15.45:0/2526727617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb618004800 msgr2=0x7fb618004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.820+0000 7fb634a00640 1 --2- 172.21.15.45:0/2526727617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb618004800 0x7fb618004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.820+0000 7fb634a00640 1 -- 172.21.15.45:0/2526727617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb628053a50 con 0x7fb6180051c0 2024-04-16T16:13:32.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.821+0000 7f01a7e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f01a8151ae0 0x7f01a815b750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40346/0 (socket says 172.21.15.45:40346) 2024-04-16T16:13:32.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.821+0000 7f01a7e00640 1 -- 172.21.15.45:0/3795057509 learned_addr learned my addr 172.21.15.45:0/3795057509 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:32.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.821+0000 7f01a7400640 1 --2- 172.21.15.45:0/3795057509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01a8071f40 0x7f01a80723c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.821+0000 7ff9d5a00640 1 Processor -- start 2024-04-16T16:13:32.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.821+0000 7f6b72a00640 1 --2- 172.21.15.45:0/3198504142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b580039e0 0x7f6b58003dc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6b74050b40 tx=0x7f6b58005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e1ce9401fb7df19 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.821+0000 7f0730e00640 1 Processor -- start 2024-04-16T16:13:32.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.821+0000 7ff9d5a00640 1 -- start start 2024-04-16T16:13:32.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.822+0000 7f0730e00640 1 -- start start 2024-04-16T16:13:32.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.822+0000 7fb634a00640 1 -- 172.21.15.45:0/2526727617 reap_dead start 2024-04-16T16:13:32.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.821+0000 7f01a6a00640 1 -- 172.21.15.45:0/3795057509 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 246538312 0 0) 0x7f01a80cecb0 con 0x7f01a8151ae0 2024-04-16T16:13:32.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.822+0000 7f01ace00640 1 --2- 172.21.15.45:0/3795057509 >> v2:172.21.15.103:3300/0 conn(0x7f01a80cd7e0 0x7f01a80cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.822+0000 7f01a6a00640 1 -- 172.21.15.45:0/3795057509 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f018c003520 con 0x7f01a8151ae0 2024-04-16T16:13:32.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.822+0000 7f07c8200640 1 Processor -- start 2024-04-16T16:13:32.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.823+0000 7f07c8200640 1 -- start start 2024-04-16T16:13:32.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.822+0000 7fb634a00640 1 --2- 172.21.15.45:0/2526727617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6180051c0 0x7fb61800b9c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb62000aa70 tx=0x7fb618005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57a606534b6e73f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.823+0000 7f0730e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f07240e3160 0x7f07240e7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.823+0000 7f0730e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f072403c0a0 0x7f0724058050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7f0730e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0724053cb0 0x7f0724054140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7f0730e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0724058590 con 0x7f07240e3160 2024-04-16T16:13:32.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7f0730e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0724054680 con 0x7f072403c0a0 2024-04-16T16:13:32.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7f0730e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0724058810 con 0x7f0724053cb0 2024-04-16T16:13:32.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.822+0000 7f01a7400640 1 -- 172.21.15.45:0/3795057509 >> v2:172.21.15.103:3300/0 conn(0x7f01a80cd7e0 msgr2=0x7f01a80cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.823+0000 7f6b71600640 1 -- 172.21.15.45:0/3198504142 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b740b5070 con 0x7f6b580039e0 2024-04-16T16:13:32.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7f6b71600640 1 -- 172.21.15.45:0/3198504142 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b740b0410 con 0x7f6b580039e0 2024-04-16T16:13:32.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7f6b71600640 1 -- 172.21.15.45:0/3198504142 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b74050c80 con 0x7f6b580039e0 2024-04-16T16:13:32.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.822+0000 7f01a7400640 1 --2- 172.21.15.45:0/3795057509 >> v2:172.21.15.103:3300/0 conn(0x7f01a80cd7e0 0x7f01a80cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.822+0000 7f01a7400640 1 -- 172.21.15.45:0/3795057509 >> v1:172.21.15.45:6789/0 conn(0x7f01a8151ae0 legacy=0x7f01a815b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.823+0000 7f01a7400640 1 -- 172.21.15.45:0/3795057509 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a80cc7e0 con 0x7f01a8071f40 2024-04-16T16:13:32.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.823+0000 7f01ace00640 1 --2- 172.21.15.45:0/3795057509 >> v2:172.21.15.103:3300/0 conn(0x7f01a80cd7e0 0x7f01a80cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:32.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7f072b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f07240e3160 0x7f07240e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7f072b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f07240e3160 0x7f07240e7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49760/0 (socket says 172.21.15.45:49760) 2024-04-16T16:13:32.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7f072b400640 1 -- 172.21.15.45:0/3109379311 learned_addr learned my addr 172.21.15.45:0/3109379311 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:32.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7f072be00640 1 --2- 172.21.15.45:0/3109379311 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0724053cb0 0x7f0724054140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f01a7400640 1 --2- 172.21.15.45:0/3795057509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01a8071f40 0x7f01a80723c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f01a000aea0 tx=0x7f01a002cf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5aceb79d07b7ce20 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f07c8200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07c00afe90 0x7f07c00b5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f07c8200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f07c0072890 0x7f07c0160160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f07c8200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07c0164440 con 0x7f07c00b5640 2024-04-16T16:13:32.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f07c8200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f07c006ef10 con 0x7f07c0072890 2024-04-16T16:13:32.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f07c8200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f07c00b5ca0 con 0x7f07c00afe90 2024-04-16T16:13:32.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7ff9d5a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d015ed30 0x7ff9d00d2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f072b400640 1 -- 172.21.15.45:0/3109379311 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0724053cb0 msgr2=0x7f0724054140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f072b400640 1 --2- 172.21.15.45:0/3109379311 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0724053cb0 0x7f0724054140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f072b400640 1 -- 172.21.15.45:0/3109379311 >> v2:172.21.15.103:3300/0 conn(0x7f072403c0a0 msgr2=0x7f0724058050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f072b400640 1 --2- 172.21.15.45:0/3109379311 >> v2:172.21.15.103:3300/0 conn(0x7f072403c0a0 0x7f0724058050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f072b400640 1 -- 172.21.15.45:0/3109379311 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0724058af0 con 0x7f07240e3160 2024-04-16T16:13:32.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7fb62ea00640 1 -- 172.21.15.45:0/2526727617 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62000a640 con 0x7fb6180051c0 2024-04-16T16:13:32.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7fb62ea00640 1 -- 172.21.15.45:0/2526727617 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb62002d650 con 0x7fb6180051c0 2024-04-16T16:13:32.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7fb62ea00640 1 -- 172.21.15.45:0/2526727617 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62002da40 con 0x7fb6180051c0 2024-04-16T16:13:32.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7fb635400640 1 -- 172.21.15.45:0/2526727617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6180051c0 msgr2=0x7fb61800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7fb635400640 1 --2- 172.21.15.45:0/2526727617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6180051c0 0x7fb61800b9c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb62000aa70 tx=0x7fb618005830 comp rx=0 tx=0).stop 2024-04-16T16:13:32.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f07c4e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f07c00b5640 0x7f07c0072180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40354/0 (socket says 172.21.15.45:40354) 2024-04-16T16:13:32.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f07c4e00640 1 -- 172.21.15.45:0/3718573783 learned_addr learned my addr 172.21.15.45:0/3718573783 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:32.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f6b73e00640 1 -- 172.21.15.45:0/3198504142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b580039e0 msgr2=0x7f6b58003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f6b73e00640 1 --2- 172.21.15.45:0/3198504142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b580039e0 0x7f6b58003dc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6b74050b40 tx=0x7f6b58005830 comp rx=0 tx=0).stop 2024-04-16T16:13:32.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f6b73400640 1 -- 172.21.15.45:0/3198504142 reap_dead start 2024-04-16T16:13:32.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f6b73e00640 1 -- 172.21.15.45:0/3198504142 shutdown_connections 2024-04-16T16:13:32.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f6b73e00640 1 -- 172.21.15.45:0/3198504142 >> 172.21.15.45:0/3198504142 conn(0x7f6b6c0559e0 msgr2=0x7f6b6c055df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.826+0000 7f6b73e00640 1 -- 172.21.15.45:0/3198504142 shutdown_connections 2024-04-16T16:13:32.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.826+0000 7f6b73e00640 1 -- 172.21.15.45:0/3198504142 wait complete. 2024-04-16T16:13:32.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f01a6a00640 1 -- 172.21.15.45:0/3795057509 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a002dab0 con 0x7f01a8071f40 2024-04-16T16:13:32.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.825+0000 7f07c5800640 1 --2- 172.21.15.45:0/3718573783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07c00afe90 0x7f07c00b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.826+0000 7f01a6a00640 1 -- 172.21.15.45:0/3795057509 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01a002dc50 con 0x7f01a8071f40 2024-04-16T16:13:32.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.826+0000 7f01a6a00640 1 -- 172.21.15.45:0/3795057509 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a0008440 con 0x7f01a8071f40 2024-04-16T16:13:32.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.826+0000 7f07c6200640 1 --2- 172.21.15.45:0/3718573783 >> v2:172.21.15.103:3300/0 conn(0x7f07c0072890 0x7f07c0160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7ff9d5a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9d00d3480 con 0x7ff9d00d2fe0 2024-04-16T16:13:32.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7ff9d5a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9d00d3760 con 0x7ff9d00cfe90 2024-04-16T16:13:32.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.824+0000 7ff9d5a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d015f110 con 0x7ff9d015ed30 2024-04-16T16:13:32.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.826+0000 7ff9cf400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7ff9d00cfe90 0x7ff9d00b7ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:51496/0 (socket says 172.21.15.45:51496) 2024-04-16T16:13:32.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.826+0000 7ff9cf400640 1 -- 172.21.15.45:0/2665310350 learned_addr learned my addr 172.21.15.45:0/2665310350 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:32.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.826+0000 7ff9cea00640 1 --2- 172.21.15.45:0/2665310350 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d015ed30 0x7ff9d00d2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.827+0000 7f01ad800640 1 -- 172.21.15.45:0/3795057509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01a8071f40 msgr2=0x7f01a80723c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.827+0000 7f01ad800640 1 --2- 172.21.15.45:0/3795057509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01a8071f40 0x7f01a80723c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f01a000aea0 tx=0x7f01a002cf10 comp rx=0 tx=0).stop 2024-04-16T16:13:32.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.827+0000 7ff9cd600640 1 -- 172.21.15.45:0/2665310350 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 848453566 0 0) 0x7ff9d00d3760 con 0x7ff9d00cfe90 2024-04-16T16:13:32.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.826+0000 7f072b400640 1 --2- 172.21.15.45:0/3109379311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240e3160 0x7f07240e7780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0724053700 tx=0x7f072002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8755337f37fd3612 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.827+0000 7f072a000640 1 -- 172.21.15.45:0/3109379311 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720034070 con 0x7f07240e3160 2024-04-16T16:13:32.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.827+0000 7ff9cd600640 1 -- 172.21.15.45:0/2665310350 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9b4003520 con 0x7ff9d00cfe90 2024-04-16T16:13:32.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.827+0000 7f01ad800640 1 -- 172.21.15.45:0/3795057509 shutdown_connections 2024-04-16T16:13:32.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.827+0000 7f072a000640 1 -- 172.21.15.45:0/3109379311 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0720030610 con 0x7f07240e3160 2024-04-16T16:13:32.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.827+0000 7f01ad800640 1 --2- 172.21.15.45:0/3795057509 >> v2:172.21.15.103:3300/0 conn(0x7f01a80cd7e0 0x7f01a80cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.827+0000 7f01ad800640 1 --2- 172.21.15.45:0/3795057509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01a8071f40 0x7f01a80723c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.827+0000 7f01ad800640 1 -- 172.21.15.45:0/3795057509 >> 172.21.15.45:0/3795057509 conn(0x7f01a80677d0 msgr2=0x7f01a8067bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.827+0000 7f072a000640 1 -- 172.21.15.45:0/3109379311 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720034070 con 0x7f07240e3160 2024-04-16T16:13:32.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.828+0000 7ff9cea00640 1 -- 172.21.15.45:0/2665310350 >> v1:172.21.15.103:6789/0 conn(0x7ff9d00cfe90 legacy=0x7ff9d00b7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.827+0000 7f07bfe00640 1 -- 172.21.15.45:0/3718573783 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 241996676 0 0) 0x7f07c0164440 con 0x7f07c00b5640 2024-04-16T16:13:32.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.827+0000 7f07bfe00640 1 -- 172.21.15.45:0/3718573783 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07a8003520 con 0x7f07c00b5640 2024-04-16T16:13:32.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.828+0000 7f07c5800640 1 -- 172.21.15.45:0/3718573783 >> v2:172.21.15.103:3300/0 conn(0x7f07c0072890 msgr2=0x7f07c0160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.828+0000 7f07c5800640 1 --2- 172.21.15.45:0/3718573783 >> v2:172.21.15.103:3300/0 conn(0x7f07c0072890 0x7f07c0160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.828+0000 7f07c5800640 1 -- 172.21.15.45:0/3718573783 >> v1:172.21.15.45:6789/0 conn(0x7f07c00b5640 legacy=0x7f07c0072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.828+0000 7f07c5800640 1 -- 172.21.15.45:0/3718573783 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07c0166a60 con 0x7f07c00afe90 2024-04-16T16:13:32.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.828+0000 7fb635400640 1 -- 172.21.15.45:0/2526727617 shutdown_connections 2024-04-16T16:13:32.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.828+0000 7fb635400640 1 --2- 172.21.15.45:0/2526727617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6180051c0 0x7fb61800b9c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.828+0000 7fb635400640 1 -- 172.21.15.45:0/2526727617 >> 172.21.15.45:0/2526727617 conn(0x7fb62800a400 msgr2=0x7fb62800a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.828+0000 7ff9cea00640 1 -- 172.21.15.45:0/2665310350 >> v1:172.21.15.45:6789/0 conn(0x7ff9d00d2fe0 legacy=0x7ff9d00cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.828+0000 7ff9cea00640 1 -- 172.21.15.45:0/2665310350 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d00d3a40 con 0x7ff9d015ed30 2024-04-16T16:13:32.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.828+0000 7f07c6200640 1 --2- 172.21.15.45:0/3718573783 >> v2:172.21.15.103:3300/0 conn(0x7f07c0072890 0x7f07c0160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:32.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.828+0000 7fb635400640 1 -- 172.21.15.45:0/2526727617 shutdown_connections 2024-04-16T16:13:32.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.829+0000 7f6b73e00640 1 Processor -- start 2024-04-16T16:13:32.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.829+0000 7f6b73e00640 1 -- start start 2024-04-16T16:13:32.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.828+0000 7f01ad800640 1 -- 172.21.15.45:0/3795057509 shutdown_connections 2024-04-16T16:13:32.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.829+0000 7f0730e00640 1 -- 172.21.15.45:0/3109379311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240e3160 msgr2=0x7f07240e7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.829+0000 7f0730e00640 1 --2- 172.21.15.45:0/3109379311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240e3160 0x7f07240e7780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0724053700 tx=0x7f072002b960 comp rx=0 tx=0).stop 2024-04-16T16:13:32.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.829+0000 7f01ad800640 1 -- 172.21.15.45:0/3795057509 wait complete. 2024-04-16T16:13:32.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.829+0000 7ff9cea00640 1 --2- 172.21.15.45:0/2665310350 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d015ed30 0x7ff9d00d2aa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff9c00097d0 tx=0x7ff9c002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56155fa6d2217a2c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.828+0000 7fb635400640 1 -- 172.21.15.45:0/2526727617 wait complete. 2024-04-16T16:13:32.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.830+0000 7f0730e00640 1 -- 172.21.15.45:0/3109379311 shutdown_connections 2024-04-16T16:13:32.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.830+0000 7f0730e00640 1 --2- 172.21.15.45:0/3109379311 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0724053cb0 0x7f0724054140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.830+0000 7f0730e00640 1 --2- 172.21.15.45:0/3109379311 >> v2:172.21.15.103:3300/0 conn(0x7f072403c0a0 0x7f0724058050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.830+0000 7f0730e00640 1 --2- 172.21.15.45:0/3109379311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240e3160 0x7f07240e7780 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.830+0000 7f0730e00640 1 -- 172.21.15.45:0/3109379311 >> 172.21.15.45:0/3109379311 conn(0x7f0724058ee0 msgr2=0x7f07240592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.830+0000 7ff9cd600640 1 -- 172.21.15.45:0/2665310350 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0034070 con 0x7ff9d015ed30 2024-04-16T16:13:32.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.830+0000 7f0730e00640 1 -- 172.21.15.45:0/3109379311 shutdown_connections 2024-04-16T16:13:32.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.830+0000 7f0730e00640 1 -- 172.21.15.45:0/3109379311 wait complete. 2024-04-16T16:13:32.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.830+0000 7ff9cd600640 1 -- 172.21.15.45:0/2665310350 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c0030800 con 0x7ff9d015ed30 2024-04-16T16:13:32.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.830+0000 7ff9cd600640 1 -- 172.21.15.45:0/2665310350 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0030bb0 con 0x7ff9d015ed30 2024-04-16T16:13:32.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.830+0000 7f07c5800640 1 --2- 172.21.15.45:0/3718573783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07c00afe90 0x7f07c00b5100 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f07b40097a0 tx=0x7f07b402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30ca54eba3f1c3eb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.831+0000 7f07bfe00640 1 -- 172.21.15.45:0/3718573783 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b4031780 con 0x7f07c00afe90 2024-04-16T16:13:32.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.831+0000 7f07bfe00640 1 -- 172.21.15.45:0/3718573783 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07b4037070 con 0x7f07c00afe90 2024-04-16T16:13:32.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.831+0000 7f07bfe00640 1 -- 172.21.15.45:0/3718573783 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b4031b40 con 0x7f07c00afe90 2024-04-16T16:13:32.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.831+0000 7f01ad800640 1 Processor -- start 2024-04-16T16:13:32.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.831+0000 7ff9d5a00640 1 -- 172.21.15.45:0/2665310350 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d015ed30 msgr2=0x7ff9d00d2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.831+0000 7ff9d5a00640 1 --2- 172.21.15.45:0/2665310350 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d015ed30 0x7ff9d00d2aa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff9c00097d0 tx=0x7ff9c002b940 comp rx=0 tx=0).stop 2024-04-16T16:13:32.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.831+0000 7fb814000640 1 Processor -- start 2024-04-16T16:13:32.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.831+0000 7f01ad800640 1 -- start start 2024-04-16T16:13:32.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.832+0000 7fb814000640 1 -- start start 2024-04-16T16:13:32.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.831+0000 7fb635400640 1 Processor -- start 2024-04-16T16:13:32.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.832+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b580039e0 0x7f6b6c04e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.832+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6b6c171b20 0x7f6b6c0e1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.832+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b6c0e9b00 0x7f6b6c0e9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.832+0000 7fb635400640 1 -- start start 2024-04-16T16:13:32.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.832+0000 7ff9d5a00640 1 -- 172.21.15.45:0/2665310350 shutdown_connections 2024-04-16T16:13:32.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.832+0000 7ff9d5a00640 1 --2- 172.21.15.45:0/2665310350 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d015ed30 0x7ff9d00d2aa0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.832+0000 7ff9d5a00640 1 -- 172.21.15.45:0/2665310350 >> 172.21.15.45:0/2665310350 conn(0x7ff9d00d1610 msgr2=0x7ff9d00d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f0730e00640 1 Processor -- start 2024-04-16T16:13:32.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7ff9d5a00640 1 -- 172.21.15.45:0/2665310350 shutdown_connections 2024-04-16T16:13:32.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f6b72000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6b6c171b20 0x7f6b6c0e1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f6b73400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b6c0e9b00 0x7f6b6c0e9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f6b72000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6b6c171b20 0x7f6b6c0e1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:36342/0 (socket says 172.21.15.45:36342) 2024-04-16T16:13:32.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f6b72000640 1 -- 172.21.15.45:0/2581286272 learned_addr learned my addr 172.21.15.45:0/2581286272 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:32.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f01ad800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a8071f40 0x7f01a814c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f01ad800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01a80cd7e0 0x7f01a814da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f01ad800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01a8151ae0 0x7f01a80b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f01ad800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f01a80cd080 con 0x7f01a8071f40 2024-04-16T16:13:32.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f01ad800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f01a80ccda0 con 0x7f01a80cd7e0 2024-04-16T16:13:32.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f01ad800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f01a815b180 con 0x7f01a8151ae0 2024-04-16T16:13:32.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f07c8200640 1 -- 172.21.15.45:0/3718573783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07c00afe90 msgr2=0x7f07c00b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f07c8200640 1 --2- 172.21.15.45:0/3718573783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07c00afe90 0x7f07c00b5100 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f07b40097a0 tx=0x7f07b402b820 comp rx=0 tx=0).stop 2024-04-16T16:13:32.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f0730e00640 1 -- start start 2024-04-16T16:13:32.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f6b73e00640 1 -- 172.21.15.45:0/2581286272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6c0579a0 con 0x7f6b6c0e9b00 2024-04-16T16:13:32.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f6b72a00640 1 --2- 172.21.15.45:0/2581286272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b580039e0 0x7f6b6c04e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f6b73e00640 1 -- 172.21.15.45:0/2581286272 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6c057630 con 0x7f6b580039e0 2024-04-16T16:13:32.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7f6b73e00640 1 -- 172.21.15.45:0/2581286272 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6b6c0e1e10 con 0x7f6b6c171b20 2024-04-16T16:13:32.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7fb814000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb80c0cfe90 0x7fb80c1697d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7fb814000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb80c0d31b0 con 0x7fb80c15ed30 2024-04-16T16:13:32.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7fb814000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb80c0d3480 con 0x7fb80c0ba710 2024-04-16T16:13:32.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7f01a7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a8071f40 0x7f01a814c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7f6dc3e00640 1 Processor -- start 2024-04-16T16:13:32.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7fb635400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6180051c0 0x7fb6280d6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7fb635400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb628037e10 0x7fb62804a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7fb635400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6280d4080 0x7fb6280d4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7fb635400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb62804b7a0 con 0x7fb6180051c0 2024-04-16T16:13:32.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7fb635400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb628006530 con 0x7fb628037e10 2024-04-16T16:13:32.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7fb635400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb628054830 con 0x7fb6280d4080 2024-04-16T16:13:32.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7f01a7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a8071f40 0x7f01a814c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49774/0 (socket says 172.21.15.45:49774) 2024-04-16T16:13:32.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7f01a7e00640 1 -- 172.21.15.45:0/554374418 learned_addr learned my addr 172.21.15.45:0/554374418 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:32.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7fb814000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb80c15f110 con 0x7fb80c0cfe90 2024-04-16T16:13:32.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.833+0000 7ff9d5a00640 1 -- 172.21.15.45:0/2665310350 wait complete. 2024-04-16T16:13:32.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7fb62fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6180051c0 0x7fb6280d6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7fb62fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6180051c0 0x7fb6280d6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49788/0 (socket says 172.21.15.45:49788) 2024-04-16T16:13:32.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7fb62fe00640 1 -- 172.21.15.45:0/1259113255 learned_addr learned my addr 172.21.15.45:0/1259113255 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:32.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7fb811600640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb80c15ed30 0x7fb80c0d2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40376/0 (socket says 172.21.15.45:40376) 2024-04-16T16:13:32.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7fb811600640 1 -- 172.21.15.45:0/1701002880 learned_addr learned my addr 172.21.15.45:0/1701002880 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:32.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7fb812000640 1 --2- 172.21.15.45:0/1701002880 >> v2:172.21.15.45:3300/0 conn(0x7fb80c0cfe90 0x7fb80c1697d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7f6b72000640 1 -- 172.21.15.45:0/2581286272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b580039e0 msgr2=0x7f6b6c04e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7f6b72000640 1 --2- 172.21.15.45:0/2581286272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b580039e0 0x7f6b6c04e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7f6b72000640 1 -- 172.21.15.45:0/2581286272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b6c0e9b00 msgr2=0x7f6b6c0e9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7f6b72000640 1 --2- 172.21.15.45:0/2581286272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b6c0e9b00 0x7f6b6c0e9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7f6b72000640 1 -- 172.21.15.45:0/2581286272 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6c17d8a0 con 0x7f6b6c171b20 2024-04-16T16:13:32.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7fb62f400640 1 --2- 172.21.15.45:0/1259113255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb628037e10 0x7fb62804a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.834+0000 7f01ace00640 1 --2- 172.21.15.45:0/554374418 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01a8151ae0 0x7f01a80b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7fb634a00640 1 --2- 172.21.15.45:0/1259113255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6280d4080 0x7fb6280d4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7f01a7400640 1 --2- 172.21.15.45:0/554374418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01a80cd7e0 0x7f01a814da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7f01a7e00640 1 -- 172.21.15.45:0/554374418 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01a8151ae0 msgr2=0x7f01a80b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7f0730e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f072403c0a0 0x7f07240d74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7f0730e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0724053cb0 0x7f07240522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7f0730e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240e3160 0x7f072403da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7f01a7e00640 1 --2- 172.21.15.45:0/554374418 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01a8151ae0 0x7f01a80b44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7f07c8200640 1 -- 172.21.15.45:0/3718573783 shutdown_connections 2024-04-16T16:13:32.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7f07c8200640 1 --2- 172.21.15.45:0/3718573783 >> v2:172.21.15.103:3300/0 conn(0x7f07c0072890 0x7f07c0160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7fb62fe00640 1 -- 172.21.15.45:0/1259113255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6280d4080 msgr2=0x7fb6280d4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7f07c8200640 1 --2- 172.21.15.45:0/3718573783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07c00afe90 0x7f07c00b5100 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7f07c8200640 1 -- 172.21.15.45:0/3718573783 >> 172.21.15.45:0/3718573783 conn(0x7f07c006b930 msgr2=0x7f07c006bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7f07c8200640 1 -- 172.21.15.45:0/3718573783 shutdown_connections 2024-04-16T16:13:32.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7f07c8200640 1 -- 172.21.15.45:0/3718573783 wait complete. 2024-04-16T16:13:32.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7fb62fe00640 1 --2- 172.21.15.45:0/1259113255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6280d4080 0x7fb6280d4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7f0730e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f07240e3540 con 0x7f07240e3160 2024-04-16T16:13:32.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7f0730e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0724006640 con 0x7f0724053cb0 2024-04-16T16:13:32.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7fb62fe00640 1 -- 172.21.15.45:0/1259113255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb628037e10 msgr2=0x7fb62804a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7fb62fe00640 1 --2- 172.21.15.45:0/1259113255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb628037e10 0x7fb62804a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7fb62fe00640 1 -- 172.21.15.45:0/1259113255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6280d76c0 con 0x7fb6180051c0 2024-04-16T16:13:32.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7f6b72000640 1 --2- 172.21.15.45:0/2581286272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6b6c171b20 0x7f6b6c0e1330 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6b640013d0 tx=0x7f6b64008430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7f072aa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0724053cb0 0x7f07240522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7f072b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f072403c0a0 0x7f07240d74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7f072b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f072403c0a0 0x7f07240d74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:36376/0 (socket says 172.21.15.45:36376) 2024-04-16T16:13:32.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7f072b400640 1 -- 172.21.15.45:0/1151590068 learned_addr learned my addr 172.21.15.45:0/1151590068 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:32.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7f0730e00640 1 -- 172.21.15.45:0/1151590068 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0724056e10 con 0x7f072403c0a0 2024-04-16T16:13:32.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7fb62f400640 1 --2- 172.21.15.45:0/1259113255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb628037e10 0x7fb62804a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:32.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7f01a7e00640 1 -- 172.21.15.45:0/554374418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01a80cd7e0 msgr2=0x7f01a814da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7f01a7e00640 1 --2- 172.21.15.45:0/554374418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01a80cd7e0 0x7f01a814da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7f01a7e00640 1 -- 172.21.15.45:0/554374418 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a815e5d0 con 0x7f01a8071f40 2024-04-16T16:13:32.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7f01a7400640 1 --2- 172.21.15.45:0/554374418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01a80cd7e0 0x7f01a814da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:13:32.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7f6dc3e00640 1 -- start start 2024-04-16T16:13:32.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7f072be00640 1 --2- 172.21.15.45:0/1151590068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240e3160 0x7f072403da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7fb62fe00640 1 --2- 172.21.15.45:0/1259113255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6180051c0 0x7fb6280d6fc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb6300502b0 tx=0x7fb63006dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.837+0000 7f01a7e00640 1 --2- 172.21.15.45:0/554374418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a8071f40 0x7f01a814c9b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f019400c8d0 tx=0x7f019400cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7ff9d5a00640 1 Processor -- start 2024-04-16T16:13:32.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.837+0000 7ff9d5a00640 1 -- start start 2024-04-16T16:13:32.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.837+0000 7fb61fe00640 1 -- 172.21.15.45:0/1259113255 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb630075830 con 0x7fb6180051c0 2024-04-16T16:13:32.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.837+0000 7fb61fe00640 1 -- 172.21.15.45:0/1259113255 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb63006b930 con 0x7fb6180051c0 2024-04-16T16:13:32.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.837+0000 7fb61fe00640 1 -- 172.21.15.45:0/1259113255 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63006bce0 con 0x7fb6180051c0 2024-04-16T16:13:32.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.835+0000 7fb812000640 1 -- 172.21.15.45:0/1701002880 >> v1:172.21.15.103:6789/0 conn(0x7fb80c0ba710 legacy=0x7fb80c0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7fb812000640 1 -- 172.21.15.45:0/1701002880 >> v1:172.21.15.45:6789/0 conn(0x7fb80c15ed30 legacy=0x7fb80c0d2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.836+0000 7fb812000640 1 -- 172.21.15.45:0/1701002880 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb80c0d3760 con 0x7fb80c0cfe90 2024-04-16T16:13:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.837+0000 7f072b400640 1 -- 172.21.15.45:0/1151590068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0724053cb0 msgr2=0x7f07240522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.837+0000 7f072b400640 1 --2- 172.21.15.45:0/1151590068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0724053cb0 0x7f07240522a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.837+0000 7f072b400640 1 -- 172.21.15.45:0/1151590068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240e3160 msgr2=0x7f072403da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.837+0000 7f072b400640 1 --2- 172.21.15.45:0/1151590068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240e3160 0x7f072403da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.837+0000 7f072b400640 1 -- 172.21.15.45:0/1151590068 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0724051550 con 0x7f072403c0a0 2024-04-16T16:13:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.837+0000 7f072b400640 1 --2- 172.21.15.45:0/1151590068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f072403c0a0 0x7f07240d74f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f072002b9a0 tx=0x7f07200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.837+0000 7fb812000640 1 --2- 172.21.15.45:0/1701002880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb80c0cfe90 0x7fb80c1697d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb804009d90 tx=0x7fb80402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13a2f1680eff6d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.837+0000 7f019fe00640 1 -- 172.21.15.45:0/554374418 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0194008040 con 0x7f01a8071f40 2024-04-16T16:13:32.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7f6b6b400640 1 -- 172.21.15.45:0/2581286272 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b64005070 con 0x7f6b6c171b20 2024-04-16T16:13:32.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7f071f400640 1 -- 172.21.15.45:0/1151590068 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720034040 con 0x7f072403c0a0 2024-04-16T16:13:32.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7f6b6b400640 1 -- 172.21.15.45:0/2581286272 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b64019070 con 0x7f6b6c171b20 2024-04-16T16:13:32.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7fb635400640 1 -- 172.21.15.45:0/1259113255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6280d79a0 con 0x7fb6180051c0 2024-04-16T16:13:32.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7fb635400640 1 -- 172.21.15.45:0/1259113255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb628038290 con 0x7fb6180051c0 2024-04-16T16:13:32.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7fb80be00640 1 -- 172.21.15.45:0/1701002880 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb804039070 con 0x7fb80c0cfe90 2024-04-16T16:13:32.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7fb80be00640 1 -- 172.21.15.45:0/1701002880 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb80402cc80 con 0x7fb80c0cfe90 2024-04-16T16:13:32.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7f019fe00640 1 -- 172.21.15.45:0/554374418 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0194007950 con 0x7f01a8071f40 2024-04-16T16:13:32.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7fb80be00640 1 -- 172.21.15.45:0/1701002880 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb804034660 con 0x7fb80c0cfe90 2024-04-16T16:13:32.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7f019fe00640 1 -- 172.21.15.45:0/554374418 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0194008040 con 0x7f01a8071f40 2024-04-16T16:13:32.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7f6dc3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dbc0ba710 0x7f6dbc0cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7f6dc3e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dbc0d31b0 con 0x7f6dbc15ed30 2024-04-16T16:13:32.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7f6dc3e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dbc0d3480 con 0x7f6dbc0cfcc0 2024-04-16T16:13:32.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7f6dc3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6dbc15f110 con 0x7f6dbc0ba710 2024-04-16T16:13:32.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7f6dc1400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6dbc15ed30 0x7f6dbc0d2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40382/0 (socket says 172.21.15.45:40382) 2024-04-16T16:13:32.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7f6dc1400640 1 -- 172.21.15.45:0/469695925 learned_addr learned my addr 172.21.15.45:0/469695925 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:32.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.839+0000 7fb635400640 1 -- 172.21.15.45:0/1259113255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb6280057e0 con 0x7fb6180051c0 2024-04-16T16:13:32.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7ff9d5a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d00cfe90 0x7ff9d0156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7ff9d5a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9d00d2fe0 0x7ff9d0151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7ff9d5a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d015ed30 0x7ff9d015cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7ff9d5a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d00d4520 con 0x7ff9d00d2fe0 2024-04-16T16:13:32.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7ff9d5a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d00d4000 con 0x7ff9d015ed30 2024-04-16T16:13:32.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7ff9d5a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d01578e0 con 0x7ff9d00cfe90 2024-04-16T16:13:32.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.839+0000 7ff9cf400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d015ed30 0x7ff9d015cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.839+0000 7ff9cea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d00cfe90 0x7ff9d0156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.839+0000 7ff9cf400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d015ed30 0x7ff9d015cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:37706/0 (socket says 172.21.15.45:37706) 2024-04-16T16:13:32.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.839+0000 7ff9cf400640 1 -- 172.21.15.45:0/3065376034 learned_addr learned my addr 172.21.15.45:0/3065376034 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:32.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7fb814000640 1 -- 172.21.15.45:0/1701002880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb80c0cfe90 msgr2=0x7fb80c1697d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7fb814000640 1 --2- 172.21.15.45:0/1701002880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb80c0cfe90 0x7fb80c1697d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb804009d90 tx=0x7fb80402bed0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7f07c8200640 1 Processor -- start 2024-04-16T16:13:32.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.839+0000 7f6dc0a00640 1 --2- 172.21.15.45:0/469695925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dbc0ba710 0x7f6dbc0cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.839+0000 7f01ad800640 1 -- 172.21.15.45:0/554374418 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01a814bbf0 con 0x7f01a8071f40 2024-04-16T16:13:32.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7fb814000640 1 -- 172.21.15.45:0/1701002880 shutdown_connections 2024-04-16T16:13:32.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7fb814000640 1 --2- 172.21.15.45:0/1701002880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb80c0cfe90 0x7fb80c1697d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7f07c8200640 1 -- start start 2024-04-16T16:13:32.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7fb814000640 1 -- 172.21.15.45:0/1701002880 >> 172.21.15.45:0/1701002880 conn(0x7fb80c0d1610 msgr2=0x7fb80c0d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7fb814000640 1 -- 172.21.15.45:0/1701002880 shutdown_connections 2024-04-16T16:13:32.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7f6dc0a00640 1 -- 172.21.15.45:0/469695925 >> v1:172.21.15.103:6789/0 conn(0x7f6dbc0cfcc0 legacy=0x7f6dbc0b7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7f6dc0a00640 1 -- 172.21.15.45:0/469695925 >> v1:172.21.15.45:6789/0 conn(0x7f6dbc15ed30 legacy=0x7f6dbc0d2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7fb814000640 1 -- 172.21.15.45:0/1701002880 wait complete. 2024-04-16T16:13:32.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7f01ad800640 1 -- 172.21.15.45:0/554374418 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01a814c1b0 con 0x7f01a8071f40 2024-04-16T16:13:32.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7f0730e00640 1 -- 172.21.15.45:0/1151590068 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07240504b0 con 0x7f072403c0a0 2024-04-16T16:13:32.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7f0730e00640 1 -- 172.21.15.45:0/1151590068 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0724050a70 con 0x7f072403c0a0 2024-04-16T16:13:32.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7f071f400640 1 -- 172.21.15.45:0/1151590068 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0720030610 con 0x7f072403c0a0 2024-04-16T16:13:32.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.841+0000 7f6dc0a00640 1 -- 172.21.15.45:0/469695925 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dbc0d3760 con 0x7f6dbc0ba710 2024-04-16T16:13:32.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.838+0000 7f071f400640 1 -- 172.21.15.45:0/1151590068 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720003a40 con 0x7f072403c0a0 2024-04-16T16:13:32.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7f0730e00640 1 -- 172.21.15.45:0/1151590068 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0708001030 con 0x7f072403c0a0 2024-04-16T16:13:32.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.839+0000 7f6b73e00640 1 -- 172.21.15.45:0/2581286272 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b6c04eea0 con 0x7f6b6c171b20 2024-04-16T16:13:32.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.839+0000 7f6b73e00640 1 -- 172.21.15.45:0/2581286272 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b6c04f460 con 0x7f6b6c171b20 2024-04-16T16:13:32.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7f6b6b400640 1 -- 172.21.15.45:0/2581286272 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b64005450 con 0x7f6b6c171b20 2024-04-16T16:13:32.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7ff9ce000640 1 --2- 172.21.15.45:0/3065376034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9d00d2fe0 0x7ff9d0151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7ff9cf400640 1 -- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d00cfe90 msgr2=0x7ff9d0156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7ff9cf400640 1 --2- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d00cfe90 0x7ff9d0156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7ff9cf400640 1 -- 172.21.15.45:0/3065376034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9d00d2fe0 msgr2=0x7ff9d0151eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7ff9cf400640 1 --2- 172.21.15.45:0/3065376034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9d00d2fe0 0x7ff9d0151eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7ff9cf400640 1 -- 172.21.15.45:0/3065376034 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d015d120 con 0x7ff9d015ed30 2024-04-16T16:13:32.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.840+0000 7ff9ce000640 1 --2- 172.21.15.45:0/3065376034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9d00d2fe0 0x7ff9d0151eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:13:32.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.841+0000 7ff9cea00640 1 --2- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d00cfe90 0x7ff9d0156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:32.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.841+0000 7ff9cf400640 1 --2- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d015ed30 0x7ff9d015cbe0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800cae0 tx=0x7ff9c800a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.842+0000 7f01ad800640 1 -- 172.21.15.45:0/554374418 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f01a8075780 con 0x7f01a8071f40 2024-04-16T16:13:32.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.842+0000 7f07c8200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c0072890 0x7f07c0146230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.842+0000 7f07c8200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07c00afe90 0x7f07c015d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.843+0000 7f07c8200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07c00b5640 0x7f07c015dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.842+0000 7ff9c6a00640 1 -- 172.21.15.45:0/3065376034 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8017070 con 0x7ff9d015ed30 2024-04-16T16:13:32.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.842+0000 7ff9c6a00640 1 -- 172.21.15.45:0/3065376034 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c800ac60 con 0x7ff9d015ed30 2024-04-16T16:13:32.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.842+0000 7ff9c6a00640 1 -- 172.21.15.45:0/3065376034 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8012640 con 0x7ff9d015ed30 2024-04-16T16:13:32.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.842+0000 7ff9d5a00640 1 -- 172.21.15.45:0/3065376034 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9d0156cd0 con 0x7ff9d015ed30 2024-04-16T16:13:32.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.842+0000 7ff9d5a00640 1 -- 172.21.15.45:0/3065376034 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9d0157290 con 0x7ff9d015ed30 2024-04-16T16:13:32.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.843+0000 7f07c8200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f07c0167300 con 0x7f07c0072890 2024-04-16T16:13:32.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.843+0000 7f07c8200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f07c0167020 con 0x7f07c00afe90 2024-04-16T16:13:32.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.843+0000 7fb814000640 1 Processor -- start 2024-04-16T16:13:32.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.843+0000 7f6dc0a00640 1 --2- 172.21.15.45:0/469695925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dbc0ba710 0x7f6dbc0cf780 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6da80097d0 tx=0x7f6da802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9abc6e78a702443e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.842+0000 7f6b73e00640 1 -- 172.21.15.45:0/2581286272 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6b4c001030 con 0x7f6b6c171b20 2024-04-16T16:13:32.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.843+0000 7f07c6200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07c00b5640 0x7f07c015dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.844+0000 7f07c6200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07c00b5640 0x7f07c015dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:36402/0 (socket says 172.21.15.45:36402) 2024-04-16T16:13:32.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.844+0000 7f07c6200640 1 -- 172.21.15.45:0/1929531094 learned_addr learned my addr 172.21.15.45:0/1929531094 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:32.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.844+0000 7ff9d5a00640 1 -- 172.21.15.45:0/3065376034 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff9a8001030 con 0x7ff9d015ed30 2024-04-16T16:13:32.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.844+0000 7fb61fe00640 1 -- 172.21.15.45:0/1259113255 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb63006d0d0 con 0x7fb6180051c0 2024-04-16T16:13:32.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.844+0000 7f07c8200640 1 -- 172.21.15.45:0/1929531094 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f07c00b0270 con 0x7f07c00b5640 2024-04-16T16:13:32.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.844+0000 7f6dbbe00640 1 -- 172.21.15.45:0/469695925 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da8034070 con 0x7f6dbc0ba710 2024-04-16T16:13:32.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.844+0000 7f6dbbe00640 1 -- 172.21.15.45:0/469695925 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6da8030800 con 0x7f6dbc0ba710 2024-04-16T16:13:32.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.844+0000 7f6dbbe00640 1 -- 172.21.15.45:0/469695925 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da8030bd0 con 0x7f6dbc0ba710 2024-04-16T16:13:32.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.844+0000 7f071f400640 1 -- 172.21.15.45:0/1151590068 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0720037070 con 0x7f072403c0a0 2024-04-16T16:13:32.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7f07c6200640 1 -- 172.21.15.45:0/1929531094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07c00afe90 msgr2=0x7f07c015d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7fb61fe00640 1 --2- 172.21.15.45:0/1259113255 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb61406c9c0 0x7fb61406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7f07c4e00640 1 --2- 172.21.15.45:0/1929531094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07c00afe90 0x7f07c015d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.843+0000 7fb814000640 1 -- start start 2024-04-16T16:13:32.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7fb814000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb80c0ba710 0x7fb80c15a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7fb61fe00640 1 -- 172.21.15.45:0/1259113255 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb630107670 con 0x7fb6180051c0 2024-04-16T16:13:32.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7f6dc3e00640 1 -- 172.21.15.45:0/469695925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dbc0ba710 msgr2=0x7f6dbc0cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7f6dc3e00640 1 --2- 172.21.15.45:0/469695925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dbc0ba710 0x7f6dbc0cf780 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6da80097d0 tx=0x7f6da802b940 comp rx=0 tx=0).stop 2024-04-16T16:13:32.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7f6b6b400640 1 -- 172.21.15.45:0/2581286272 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6b6401e460 con 0x7f6b6c171b20 2024-04-16T16:13:32.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7fb61fe00640 1 --2- 172.21.15.45:0/1259113255 >> [v2:172.21.15.45:6822/3039880665,v1:172.21.15.45:6824/3039880665] conn(0x7fb614074be0 0x7fb614077000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7fb61fe00640 1 -- 172.21.15.45:0/1259113255 --> [v2:172.21.15.45:6822/3039880665,v1:172.21.15.45:6824/3039880665] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6140776b0 con 0x7fb614074be0 2024-04-16T16:13:32.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7fb61fe00640 1 -- 172.21.15.45:0/1259113255 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb630107a60 con 0x7fb6180051c0 2024-04-16T16:13:32.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7f07c6200640 1 --2- 172.21.15.45:0/1929531094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07c00afe90 0x7f07c015d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7f07c6200640 1 -- 172.21.15.45:0/1929531094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c0072890 msgr2=0x7f07c0146230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7f07c6200640 1 --2- 172.21.15.45:0/1929531094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c0072890 0x7f07c0146230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7f07c6200640 1 -- 172.21.15.45:0/1929531094 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07c015e670 con 0x7f07c00b5640 2024-04-16T16:13:32.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7f07c4e00640 1 --2- 172.21.15.45:0/1929531094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07c00afe90 0x7f07c015d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:32.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.845+0000 7fb814000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb80c0cfe90 0x7fb80c0cd920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.846+0000 7f6dc3e00640 1 -- 172.21.15.45:0/469695925 shutdown_connections 2024-04-16T16:13:32.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.846+0000 7f6dc3e00640 1 --2- 172.21.15.45:0/469695925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dbc0ba710 0x7f6dbc0cf780 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.846+0000 7f6dc3e00640 1 -- 172.21.15.45:0/469695925 >> 172.21.15.45:0/469695925 conn(0x7f6dbc0d1610 msgr2=0x7f6dbc0d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.846+0000 7fb62f400640 1 --2- 172.21.15.45:0/1259113255 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb61406c9c0 0x7fb61406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.846+0000 7fb634a00640 1 --2- 172.21.15.45:0/1259113255 >> [v2:172.21.15.45:6822/3039880665,v1:172.21.15.45:6824/3039880665] conn(0x7fb614074be0 0x7fb614077000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.846+0000 7f6dc3e00640 1 -- 172.21.15.45:0/469695925 shutdown_connections 2024-04-16T16:13:32.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.846+0000 7f6dc3e00640 1 -- 172.21.15.45:0/469695925 wait complete. 2024-04-16T16:13:32.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.846+0000 7f07c6200640 1 --2- 172.21.15.45:0/1929531094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07c00b5640 0x7f07c015dd80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f07b800efb0 tx=0x7f07b800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7f6b6b400640 1 --2- 172.21.15.45:0/2581286272 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6b5406c8f0 0x7f6b5406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7f07bd600640 1 -- 172.21.15.45:0/1929531094 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8018020 con 0x7f07c00b5640 2024-04-16T16:13:32.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.846+0000 7f071f400640 1 --2- 172.21.15.45:0/1151590068 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f070c06c8f0 0x7f070c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.846+0000 7f071f400640 1 -- 172.21.15.45:0/1151590068 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f07200ae5e0 con 0x7f072403c0a0 2024-04-16T16:13:32.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7f071f400640 1 --2- 172.21.15.45:0/1151590068 >> [v2:172.21.15.103:6800/4229500350,v1:172.21.15.103:6801/4229500350] conn(0x7f070c074af0 0x7f070c076f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7f071f400640 1 -- 172.21.15.45:0/1151590068 --> [v2:172.21.15.103:6800/4229500350,v1:172.21.15.103:6801/4229500350] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0720001d30 con 0x7f070c074af0 2024-04-16T16:13:32.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7f071f400640 1 -- 172.21.15.45:0/1151590068 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f07200ae990 con 0x7f072403c0a0 2024-04-16T16:13:32.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7fb811600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb80c0ba710 0x7fb80c15a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7fb811600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb80c0ba710 0x7fb80c15a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:36404/0 (socket says 172.21.15.45:36404) 2024-04-16T16:13:32.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7f072aa00640 1 --2- 172.21.15.45:0/1151590068 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f070c06c8f0 0x7f070c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7f07bd600640 1 -- 172.21.15.45:0/1929531094 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07b8014480 con 0x7f07c00b5640 2024-04-16T16:13:32.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7f07bd600640 1 -- 172.21.15.45:0/1929531094 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8014850 con 0x7f07c00b5640 2024-04-16T16:13:32.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.846+0000 7f019fe00640 1 -- 172.21.15.45:0/554374418 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0194007b40 con 0x7f01a8071f40 2024-04-16T16:13:32.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7f07c8200640 1 -- 172.21.15.45:0/1929531094 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07c015e950 con 0x7f07c00b5640 2024-04-16T16:13:32.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7f6b6b400640 1 -- 172.21.15.45:0/2581286272 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6b6408c8d0 con 0x7f6b6c171b20 2024-04-16T16:13:32.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7fb814000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb80c15ed30 0x7fb80c1eceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7fb814000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c0d4000 con 0x7fb80c0cfe90 2024-04-16T16:13:32.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7f6b72a00640 1 --2- 172.21.15.45:0/2581286272 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6b5406c8f0 0x7f6b5406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7f6b6b400640 1 --2- 172.21.15.45:0/2581286272 >> [v2:172.21.15.45:6800/1377072140,v1:172.21.15.45:6801/1377072140] conn(0x7f6b54074af0 0x7f6b54076f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7f07c8200640 1 -- 172.21.15.45:0/1929531094 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f07c015e2c0 con 0x7f07c00b5640 2024-04-16T16:13:32.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7f6b6b400640 1 -- 172.21.15.45:0/2581286272 --> [v2:172.21.15.45:6800/1377072140,v1:172.21.15.45:6801/1377072140] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6b64029e50 con 0x7f6b54074af0 2024-04-16T16:13:32.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7f6b6b400640 1 -- 172.21.15.45:0/2581286272 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6b6408cc80 con 0x7f6b6c171b20 2024-04-16T16:13:32.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7fb811600640 1 -- 172.21.15.45:0/1949576262 learned_addr learned my addr 172.21.15.45:0/1949576262 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:32.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.848+0000 7fb814000640 1 -- 172.21.15.45:0/1949576262 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c0d3d20 con 0x7fb80c15ed30 2024-04-16T16:13:32.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.848+0000 7fb814000640 1 -- 172.21.15.45:0/1949576262 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb80c0b8af0 con 0x7fb80c0ba710 2024-04-16T16:13:32.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.848+0000 7f6b73400640 1 --2- 172.21.15.45:0/2581286272 >> [v2:172.21.15.45:6800/1377072140,v1:172.21.15.45:6801/1377072140] conn(0x7f6b54074af0 0x7f6b54076f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.848+0000 7fb810c00640 1 --2- 172.21.15.45:0/1949576262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb80c0cfe90 0x7fb80c0cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.848+0000 7fb812000640 1 --2- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb80c15ed30 0x7fb80c1eceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.849+0000 7f072be00640 1 --2- 172.21.15.45:0/1151590068 >> [v2:172.21.15.103:6800/4229500350,v1:172.21.15.103:6801/4229500350] conn(0x7f070c074af0 0x7f070c076f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.849+0000 7fb634a00640 1 --2- 172.21.15.45:0/1259113255 >> [v2:172.21.15.45:6822/3039880665,v1:172.21.15.45:6824/3039880665] conn(0x7fb614074be0 0x7fb614077000 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.847+0000 7ff9c6a00640 1 -- 172.21.15.45:0/3065376034 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff9c801a070 con 0x7ff9d015ed30 2024-04-16T16:13:32.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.848+0000 7ff9c6a00640 1 --2- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff9b006c9c0 0x7ff9b006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.848+0000 7ff9c6a00640 1 -- 172.21.15.45:0/3065376034 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff9c808bf00 con 0x7ff9d015ed30 2024-04-16T16:13:32.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.849+0000 7f07c8200640 1 -- 172.21.15.45:0/1929531094 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f07c0075780 con 0x7f07c00b5640 2024-04-16T16:13:32.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.849+0000 7ff9c6a00640 1 --2- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:6822/2127910843,v1:172.21.15.103:6824/2127910843] conn(0x7ff9b0074be0 0x7ff9b0077000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.849+0000 7ff9c6a00640 1 -- 172.21.15.45:0/3065376034 --> [v2:172.21.15.103:6822/2127910843,v1:172.21.15.103:6824/2127910843] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff9c8020ec0 con 0x7ff9b0074be0 2024-04-16T16:13:32.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.849+0000 7ff9c6a00640 1 -- 172.21.15.45:0/3065376034 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff9c808c2f0 con 0x7ff9d015ed30 2024-04-16T16:13:32.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.849+0000 7f6b73400640 1 --2- 172.21.15.45:0/2581286272 >> [v2:172.21.15.45:6800/1377072140,v1:172.21.15.45:6801/1377072140] conn(0x7f6b54074af0 0x7f6b54076f10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.849+0000 7f019fe00640 1 --2- 172.21.15.45:0/554374418 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f018806caf0 0x7f018806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.849+0000 7f6b72a00640 1 --2- 172.21.15.45:0/2581286272 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6b5406c8f0 0x7f6b5406edb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800c500 tx=0x7f6b740ad000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.850+0000 7f019fe00640 1 -- 172.21.15.45:0/554374418 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f019408b700 con 0x7f01a8071f40 2024-04-16T16:13:32.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.850+0000 7ff9ce000640 1 --2- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:6822/2127910843,v1:172.21.15.103:6824/2127910843] conn(0x7ff9b0074be0 0x7ff9b0077000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.849+0000 7fb811600640 1 -- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb80c15ed30 msgr2=0x7fb80c1eceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.849+0000 7fb811600640 1 --2- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb80c15ed30 0x7fb80c1eceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.849+0000 7fb811600640 1 -- 172.21.15.45:0/1949576262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb80c0cfe90 msgr2=0x7fb80c0cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.849+0000 7fb811600640 1 --2- 172.21.15.45:0/1949576262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb80c0cfe90 0x7fb80c0cd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.849+0000 7fb811600640 1 -- 172.21.15.45:0/1949576262 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb80c160410 con 0x7fb80c0ba710 2024-04-16T16:13:32.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.849+0000 7fb812000640 1 --2- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb80c15ed30 0x7fb80c1eceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:32.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.850+0000 7f01a7400640 1 --2- 172.21.15.45:0/554374418 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f018806caf0 0x7f018806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.850+0000 7f019fe00640 1 --2- 172.21.15.45:0/554374418 >> [v2:172.21.15.45:6808/3148487306,v1:172.21.15.45:6809/3148487306] conn(0x7f0188074c90 0x7f01880770b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.850+0000 7f019fe00640 1 -- 172.21.15.45:0/554374418 --> [v2:172.21.15.45:6808/3148487306,v1:172.21.15.45:6809/3148487306] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0188077760 con 0x7f0188074c90 2024-04-16T16:13:32.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.850+0000 7f019fe00640 1 -- 172.21.15.45:0/554374418 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0194092050 con 0x7f01a8071f40 2024-04-16T16:13:32.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.850+0000 7f6b6b400640 1 -- 172.21.15.45:0/2581286272 <== osd.0 v2:172.21.15.45:6800/1377072140 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6b64029e50 con 0x7f6b54074af0 2024-04-16T16:13:32.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.850+0000 7ff9cea00640 1 --2- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff9b006c9c0 0x7ff9b006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.850+0000 7fb811600640 1 --2- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb80c0ba710 0x7fb80c15a030 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb80000c480 tx=0x7fb80000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.850+0000 7fb62f400640 1 --2- 172.21.15.45:0/1259113255 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb61406c9c0 0x7fb61406ee80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb6281806f0 tx=0x7fb624008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.850+0000 7f072be00640 1 --2- 172.21.15.45:0/1151590068 >> [v2:172.21.15.103:6800/4229500350,v1:172.21.15.103:6801/4229500350] conn(0x7f070c074af0 0x7f070c076f10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.850+0000 7f6dc3e00640 1 Processor -- start 2024-04-16T16:13:32.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.850+0000 7f01ace00640 1 --2- 172.21.15.45:0/554374418 >> [v2:172.21.15.45:6808/3148487306,v1:172.21.15.45:6809/3148487306] conn(0x7f0188074c90 0x7f01880770b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.851+0000 7f6dc3e00640 1 -- start start 2024-04-16T16:13:32.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.851+0000 7fb809600640 1 -- 172.21.15.45:0/1949576262 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb800017070 con 0x7fb80c0ba710 2024-04-16T16:13:32.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.851+0000 7fb809600640 1 -- 172.21.15.45:0/1949576262 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb800005ce0 con 0x7fb80c0ba710 2024-04-16T16:13:32.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.851+0000 7fb809600640 1 -- 172.21.15.45:0/1949576262 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8000072f0 con 0x7fb80c0ba710 2024-04-16T16:13:32.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.851+0000 7fb814000640 1 -- 172.21.15.45:0/1949576262 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb80c0ce300 con 0x7fb80c0ba710 2024-04-16T16:13:32.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.851+0000 7fb814000640 1 -- 172.21.15.45:0/1949576262 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb80c0ce8c0 con 0x7fb80c0ba710 2024-04-16T16:13:32.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.851+0000 7fb61fe00640 1 -- 172.21.15.45:0/1259113255 <== osd.1 v2:172.21.15.45:6822/3039880665 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb6140776b0 con 0x7fb614074be0 2024-04-16T16:13:32.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.851+0000 7ff9ce000640 1 --2- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:6822/2127910843,v1:172.21.15.103:6824/2127910843] conn(0x7ff9b0074be0 0x7ff9b0077000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.851+0000 7f01ace00640 1 --2- 172.21.15.45:0/554374418 >> [v2:172.21.15.45:6808/3148487306,v1:172.21.15.45:6809/3148487306] conn(0x7f0188074c90 0x7f01880770b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.852+0000 7f01a7400640 1 --2- 172.21.15.45:0/554374418 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f018806caf0 0x7f018806efb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f01a0036000 tx=0x7f01a0031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.850+0000 7f072aa00640 1 --2- 172.21.15.45:0/1151590068 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f070c06c8f0 0x7f070c06edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f072403e630 tx=0x7f0718009110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.852+0000 7ff9cea00640 1 --2- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff9b006c9c0 0x7ff9b006ee80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0036000 tx=0x7ff9c0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.852+0000 7f07bd600640 1 -- 172.21.15.45:0/1929531094 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f07b8005d10 con 0x7f07c00b5640 2024-04-16T16:13:32.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.853+0000 7f07bd600640 1 --2- 172.21.15.45:0/1929531094 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f07a006c940 0x7f07a006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.853+0000 7f6dc3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dbc0ba710 0x7f6dbc14aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.853+0000 7f6dc3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dbc0cfcc0 0x7f6dbc0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.853+0000 7f6dc3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dbc15ed30 0x7f6dbc0cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.853+0000 7f6dc3e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc0d3d20 con 0x7f6dbc15ed30 2024-04-16T16:13:32.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.853+0000 7f6dc3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc0d0110 con 0x7f6dbc0ba710 2024-04-16T16:13:32.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.853+0000 7f6dc3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6dbc0d4000 con 0x7f6dbc0cfcc0 2024-04-16T16:13:32.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.853+0000 7f019fe00640 1 -- 172.21.15.45:0/554374418 <== osd.2 v2:172.21.15.45:6808/3148487306 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0188077760 con 0x7f0188074c90 2024-04-16T16:13:32.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.853+0000 7f07bd600640 1 -- 172.21.15.45:0/1929531094 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f07b808d2c0 con 0x7f07c00b5640 2024-04-16T16:13:32.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.852+0000 7fb814000640 1 -- 172.21.15.45:0/1949576262 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb80c1607b0 con 0x7fb80c0ba710 2024-04-16T16:13:32.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.853+0000 7f07bd600640 1 --2- 172.21.15.45:0/1929531094 >> [v2:172.21.15.45:6816/707516363,v1:172.21.15.45:6817/707516363] conn(0x7f07a0074b60 0x7f07a0076f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.853+0000 7f07bd600640 1 -- 172.21.15.45:0/1929531094 --> [v2:172.21.15.45:6816/707516363,v1:172.21.15.45:6817/707516363] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07b80020b0 con 0x7f07a0074b60 2024-04-16T16:13:32.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.853+0000 7f07bd600640 1 -- 172.21.15.45:0/1929531094 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f07b808d670 con 0x7f07c00b5640 2024-04-16T16:13:32.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.854+0000 7f07c5800640 1 --2- 172.21.15.45:0/1929531094 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f07a006c940 0x7f07a006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.854+0000 7f6dc1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dbc15ed30 0x7f6dbc0cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.854+0000 7f6dc1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dbc15ed30 0x7f6dbc0cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49834/0 (socket says 172.21.15.45:49834) 2024-04-16T16:13:32.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.854+0000 7f6dc1e00640 1 -- 172.21.15.45:0/354001199 learned_addr learned my addr 172.21.15.45:0/354001199 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:32.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.854+0000 7f6dc1400640 1 --2- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dbc0ba710 0x7f6dbc14aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.854+0000 7f6dc0a00640 1 --2- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dbc0cfcc0 0x7f6dbc0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.855+0000 7ff9c6a00640 1 -- 172.21.15.45:0/3065376034 <== osd.5 v2:172.21.15.103:6822/2127910843 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff9c8020ec0 con 0x7ff9b0074be0 2024-04-16T16:13:32.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.854+0000 7f071f400640 1 -- 172.21.15.45:0/1151590068 <== osd.4 v2:172.21.15.103:6800/4229500350 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0720001d30 con 0x7f070c074af0 2024-04-16T16:13:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.855+0000 7f6dc1e00640 1 -- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dbc0cfcc0 msgr2=0x7f6dbc0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.855+0000 7f6dc1e00640 1 --2- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dbc0cfcc0 0x7f6dbc0cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.855+0000 7f6dc1e00640 1 -- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dbc0ba710 msgr2=0x7f6dbc14aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.855+0000 7f6dc1e00640 1 --2- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dbc0ba710 0x7f6dbc14aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.855+0000 7f6dc1e00640 1 -- 172.21.15.45:0/354001199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dbc14b030 con 0x7f6dbc15ed30 2024-04-16T16:13:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.855+0000 7f07c4e00640 1 --2- 172.21.15.45:0/1929531094 >> [v2:172.21.15.45:6816/707516363,v1:172.21.15.45:6817/707516363] conn(0x7f07a0074b60 0x7f07a0076f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.856+0000 7f6dc1e00640 1 --2- 172.21.15.45:0/354001199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dbc15ed30 0x7f6dbc0cbfc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6db400c3c0 tx=0x7f6db400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.856+0000 7f07c5800640 1 --2- 172.21.15.45:0/1929531094 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f07a006c940 0x7f07a006ee00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f07b4036000 tx=0x7f07b4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.856+0000 7f6db9600640 1 -- 172.21.15.45:0/354001199 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db40052d0 con 0x7f6dbc15ed30 2024-04-16T16:13:32.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.856+0000 7f6db9600640 1 -- 172.21.15.45:0/354001199 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6db4005470 con 0x7f6dbc15ed30 2024-04-16T16:13:32.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.856+0000 7f6dc3e00640 1 -- 172.21.15.45:0/354001199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dbc0ca200 con 0x7f6dbc15ed30 2024-04-16T16:13:32.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.856+0000 7f6dc3e00640 1 -- 172.21.15.45:0/354001199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dbc0ca7c0 con 0x7f6dbc15ed30 2024-04-16T16:13:32.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.857+0000 7f6db9600640 1 -- 172.21.15.45:0/354001199 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db4007260 con 0x7f6dbc15ed30 2024-04-16T16:13:32.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.858+0000 7f6dc3e00640 1 -- 172.21.15.45:0/354001199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6d98001030 con 0x7f6dbc15ed30 2024-04-16T16:13:32.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.856+0000 7fb809600640 1 -- 172.21.15.45:0/1949576262 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb80001a070 con 0x7fb80c0ba710 2024-04-16T16:13:32.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.858+0000 7fb809600640 1 --2- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb7ec06c8f0 0x7fb7ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.858+0000 7fb809600640 1 -- 172.21.15.45:0/1949576262 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb80008b670 con 0x7fb80c0ba710 2024-04-16T16:13:32.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.858+0000 7fb809600640 1 --2- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:6808/3613143327,v1:172.21.15.103:6809/3613143327] conn(0x7fb7ec074af0 0x7fb7ec076f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.858+0000 7fb809600640 1 -- 172.21.15.45:0/1949576262 --> [v2:172.21.15.103:6808/3613143327,v1:172.21.15.103:6809/3613143327] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb80000ee50 con 0x7fb7ec074af0 2024-04-16T16:13:32.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.858+0000 7fb809600640 1 -- 172.21.15.45:0/1949576262 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb80008ba20 con 0x7fb80c0ba710 2024-04-16T16:13:32.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.858+0000 7f07c4e00640 1 --2- 172.21.15.45:0/1929531094 >> [v2:172.21.15.45:6816/707516363,v1:172.21.15.45:6817/707516363] conn(0x7f07a0074b60 0x7f07a0076f80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.859+0000 7fb810c00640 1 --2- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb7ec06c8f0 0x7fb7ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.859+0000 7fb812000640 1 --2- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:6808/3613143327,v1:172.21.15.103:6809/3613143327] conn(0x7fb7ec074af0 0x7fb7ec076f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.861+0000 7fb810c00640 1 --2- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb7ec06c8f0 0x7fb7ec06edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb80c15fdc0 tx=0x7fb7f8007e50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.861+0000 7fb812000640 1 --2- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:6808/3613143327,v1:172.21.15.103:6809/3613143327] conn(0x7fb7ec074af0 0x7fb7ec076f10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.861+0000 7f6db9600640 1 -- 172.21.15.45:0/354001199 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6db4007400 con 0x7f6dbc15ed30 2024-04-16T16:13:32.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.862+0000 7f07bd600640 1 -- 172.21.15.45:0/1929531094 <== osd.3 v2:172.21.15.45:6816/707516363 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07b80020b0 con 0x7f07a0074b60 2024-04-16T16:13:32.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.863+0000 7f6db9600640 1 --2- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6d9c06c940 0x7f6d9c06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.863+0000 7f6db9600640 1 -- 172.21.15.45:0/354001199 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6db408b920 con 0x7f6dbc15ed30 2024-04-16T16:13:32.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.864+0000 7f6db9600640 1 --2- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:6816/1303888027,v1:172.21.15.103:6817/1303888027] conn(0x7f6d9c074b60 0x7f6d9c076f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:32.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.864+0000 7f6db9600640 1 -- 172.21.15.45:0/354001199 --> [v2:172.21.15.103:6816/1303888027,v1:172.21.15.103:6817/1303888027] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6db4017e80 con 0x7f6d9c074b60 2024-04-16T16:13:32.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.864+0000 7f6db9600640 1 -- 172.21.15.45:0/354001199 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6db408be00 con 0x7f6dbc15ed30 2024-04-16T16:13:32.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.867+0000 7f6dc1400640 1 --2- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6d9c06c940 0x7f6d9c06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.867+0000 7fb809600640 1 -- 172.21.15.45:0/1949576262 <== osd.6 v2:172.21.15.103:6808/3613143327 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb80000ee50 con 0x7fb7ec074af0 2024-04-16T16:13:32.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.867+0000 7f6dc0a00640 1 --2- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:6816/1303888027,v1:172.21.15.103:6817/1303888027] conn(0x7f6d9c074b60 0x7f6d9c076f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:32.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.868+0000 7f6dc1400640 1 --2- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6d9c06c940 0x7f6d9c06ee00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6db00012e0 tx=0x7f6db0001990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.869+0000 7f6dc0a00640 1 --2- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:6816/1303888027,v1:172.21.15.103:6817/1303888027] conn(0x7f6d9c074b60 0x7f6d9c076f80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:32.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.871+0000 7f6db9600640 1 -- 172.21.15.45:0/354001199 <== osd.7 v2:172.21.15.103:6816/1303888027 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6db4017e80 con 0x7f6d9c074b60 2024-04-16T16:13:32.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.878+0000 7f0730e00640 1 -- 172.21.15.45:0/1151590068 --> [v2:172.21.15.103:6800/4229500350,v1:172.21.15.103:6801/4229500350] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0708002dd0 con 0x7f070c074af0 2024-04-16T16:13:32.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.878+0000 7f6b73e00640 1 -- 172.21.15.45:0/2581286272 --> [v2:172.21.15.45:6800/1377072140,v1:172.21.15.45:6801/1377072140] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6b4c002dd0 con 0x7f6b54074af0 2024-04-16T16:13:32.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.879+0000 7f071f400640 1 -- 172.21.15.45:0/1151590068 <== osd.4 v2:172.21.15.103:6800/4229500350 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0708002dd0 con 0x7f070c074af0 2024-04-16T16:13:32.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.879+0000 7f6b6b400640 1 -- 172.21.15.45:0/2581286272 <== osd.0 v2:172.21.15.45:6800/1377072140 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6b4c002dd0 con 0x7f6b54074af0 2024-04-16T16:13:32.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.880+0000 7f6b73e00640 1 -- 172.21.15.45:0/2581286272 >> [v2:172.21.15.45:6800/1377072140,v1:172.21.15.45:6801/1377072140] conn(0x7f6b54074af0 msgr2=0x7f6b54076f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.880+0000 7f6b73e00640 1 --2- 172.21.15.45:0/2581286272 >> [v2:172.21.15.45:6800/1377072140,v1:172.21.15.45:6801/1377072140] conn(0x7f6b54074af0 0x7f6b54076f10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.880+0000 7f6b73e00640 1 -- 172.21.15.45:0/2581286272 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6b5406c8f0 msgr2=0x7f6b5406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.880+0000 7f6b73e00640 1 --2- 172.21.15.45:0/2581286272 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6b5406c8f0 0x7f6b5406edb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800c500 tx=0x7f6b740ad000 comp rx=0 tx=0).stop 2024-04-16T16:13:32.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.880+0000 7f6b73e00640 1 -- 172.21.15.45:0/2581286272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6b6c171b20 msgr2=0x7f6b6c0e1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.880+0000 7f6b73e00640 1 --2- 172.21.15.45:0/2581286272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6b6c171b20 0x7f6b6c0e1330 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6b640013d0 tx=0x7f6b64008430 comp rx=0 tx=0).stop 2024-04-16T16:13:32.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.880+0000 7f6b73400640 1 -- 172.21.15.45:0/2581286272 reap_dead start 2024-04-16T16:13:32.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.880+0000 7f6b73e00640 1 -- 172.21.15.45:0/2581286272 shutdown_connections 2024-04-16T16:13:32.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.880+0000 7f6b73e00640 1 -- 172.21.15.45:0/2581286272 >> 172.21.15.45:0/2581286272 conn(0x7f6b6c0559e0 msgr2=0x7f6b6c0392d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.881+0000 7f071cc00640 1 -- 172.21.15.45:0/1151590068 >> [v2:172.21.15.103:6800/4229500350,v1:172.21.15.103:6801/4229500350] conn(0x7f070c074af0 msgr2=0x7f070c076f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.881+0000 7f071cc00640 1 --2- 172.21.15.45:0/1151590068 >> [v2:172.21.15.103:6800/4229500350,v1:172.21.15.103:6801/4229500350] conn(0x7f070c074af0 0x7f070c076f10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.881+0000 7f071cc00640 1 -- 172.21.15.45:0/1151590068 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f070c06c8f0 msgr2=0x7f070c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.881+0000 7f071cc00640 1 --2- 172.21.15.45:0/1151590068 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f070c06c8f0 0x7f070c06edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f072403e630 tx=0x7f0718009110 comp rx=0 tx=0).stop 2024-04-16T16:13:32.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.881+0000 7f071cc00640 1 -- 172.21.15.45:0/1151590068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f072403c0a0 msgr2=0x7f07240d74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.881+0000 7f071cc00640 1 --2- 172.21.15.45:0/1151590068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f072403c0a0 0x7f07240d74f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f072002b9a0 tx=0x7f07200046c0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.881+0000 7f072be00640 1 -- 172.21.15.45:0/1151590068 reap_dead start 2024-04-16T16:13:32.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.882+0000 7f071cc00640 1 -- 172.21.15.45:0/1151590068 shutdown_connections 2024-04-16T16:13:32.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.882+0000 7f071cc00640 1 -- 172.21.15.45:0/1151590068 >> 172.21.15.45:0/1151590068 conn(0x7f0724058ee0 msgr2=0x7f0724039dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.882+0000 7f6b73e00640 1 -- 172.21.15.45:0/2581286272 shutdown_connections 2024-04-16T16:13:32.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.882+0000 7f6b73e00640 1 -- 172.21.15.45:0/2581286272 wait complete. 2024-04-16T16:13:32.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.883+0000 7f071cc00640 1 -- 172.21.15.45:0/1151590068 shutdown_connections 2024-04-16T16:13:32.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.883+0000 7f071cc00640 1 -- 172.21.15.45:0/1151590068 wait complete. 2024-04-16T16:13:32.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.887+0000 7f01ad800640 1 -- 172.21.15.45:0/554374418 --> [v2:172.21.15.45:6808/3148487306,v1:172.21.15.45:6809/3148487306] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f01a814e220 con 0x7f0188074c90 2024-04-16T16:13:32.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.887+0000 7f019fe00640 1 -- 172.21.15.45:0/554374418 <== osd.2 v2:172.21.15.45:6808/3148487306 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f01a814e220 con 0x7f0188074c90 2024-04-16T16:13:32.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.889+0000 7f01ad800640 1 -- 172.21.15.45:0/554374418 >> [v2:172.21.15.45:6808/3148487306,v1:172.21.15.45:6809/3148487306] conn(0x7f0188074c90 msgr2=0x7f01880770b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.889+0000 7f01ad800640 1 --2- 172.21.15.45:0/554374418 >> [v2:172.21.15.45:6808/3148487306,v1:172.21.15.45:6809/3148487306] conn(0x7f0188074c90 0x7f01880770b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.890+0000 7f01ad800640 1 -- 172.21.15.45:0/554374418 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f018806caf0 msgr2=0x7f018806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.890+0000 7f01ad800640 1 --2- 172.21.15.45:0/554374418 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f018806caf0 0x7f018806efb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f01a0036000 tx=0x7f01a0031000 comp rx=0 tx=0).stop 2024-04-16T16:13:32.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.890+0000 7f01ad800640 1 -- 172.21.15.45:0/554374418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a8071f40 msgr2=0x7f01a814c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.890+0000 7f01ad800640 1 --2- 172.21.15.45:0/554374418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a8071f40 0x7f01a814c9b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f019400c8d0 tx=0x7f019400cd90 comp rx=0 tx=0).stop 2024-04-16T16:13:32.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.890+0000 7f01ace00640 1 -- 172.21.15.45:0/554374418 reap_dead start 2024-04-16T16:13:32.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.890+0000 7f01ad800640 1 -- 172.21.15.45:0/554374418 shutdown_connections 2024-04-16T16:13:32.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.890+0000 7fb635400640 1 -- 172.21.15.45:0/1259113255 --> [v2:172.21.15.45:6822/3039880665,v1:172.21.15.45:6824/3039880665] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb628054c10 con 0x7fb614074be0 2024-04-16T16:13:32.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.891+0000 7ff9d5a00640 1 -- 172.21.15.45:0/3065376034 --> [v2:172.21.15.103:6822/2127910843,v1:172.21.15.103:6824/2127910843] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff9a8002dd0 con 0x7ff9b0074be0 2024-04-16T16:13:32.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.890+0000 7f01ad800640 1 -- 172.21.15.45:0/554374418 >> 172.21.15.45:0/554374418 conn(0x7f01a80677d0 msgr2=0x7f01a80b21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.890+0000 7f01ad800640 1 -- 172.21.15.45:0/554374418 shutdown_connections 2024-04-16T16:13:32.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.891+0000 7f01ad800640 1 -- 172.21.15.45:0/554374418 wait complete. 2024-04-16T16:13:32.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.891+0000 7fb61fe00640 1 -- 172.21.15.45:0/1259113255 <== osd.1 v2:172.21.15.45:6822/3039880665 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb628054c10 con 0x7fb614074be0 2024-04-16T16:13:32.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.892+0000 7ff9c6a00640 1 -- 172.21.15.45:0/3065376034 <== osd.5 v2:172.21.15.103:6822/2127910843 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff9a8002dd0 con 0x7ff9b0074be0 2024-04-16T16:13:32.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.892+0000 7fb635400640 1 -- 172.21.15.45:0/1259113255 >> [v2:172.21.15.45:6822/3039880665,v1:172.21.15.45:6824/3039880665] conn(0x7fb614074be0 msgr2=0x7fb614077000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.892+0000 7fb635400640 1 --2- 172.21.15.45:0/1259113255 >> [v2:172.21.15.45:6822/3039880665,v1:172.21.15.45:6824/3039880665] conn(0x7fb614074be0 0x7fb614077000 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.892+0000 7fb635400640 1 -- 172.21.15.45:0/1259113255 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb61406c9c0 msgr2=0x7fb61406ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.892+0000 7fb635400640 1 --2- 172.21.15.45:0/1259113255 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb61406c9c0 0x7fb61406ee80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb6281806f0 tx=0x7fb624008040 comp rx=0 tx=0).stop 2024-04-16T16:13:32.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.892+0000 7fb635400640 1 -- 172.21.15.45:0/1259113255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6180051c0 msgr2=0x7fb6280d6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.892+0000 7fb635400640 1 --2- 172.21.15.45:0/1259113255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6180051c0 0x7fb6280d6fc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb6300502b0 tx=0x7fb63006dde0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.893+0000 7fb634a00640 1 -- 172.21.15.45:0/1259113255 reap_dead start 2024-04-16T16:13:32.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.893+0000 7fb635400640 1 -- 172.21.15.45:0/1259113255 shutdown_connections 2024-04-16T16:13:32.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.893+0000 7fb635400640 1 -- 172.21.15.45:0/1259113255 >> 172.21.15.45:0/1259113255 conn(0x7fb62800a400 msgr2=0x7fb628035fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.893+0000 7ff9d5a00640 1 -- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:6822/2127910843,v1:172.21.15.103:6824/2127910843] conn(0x7ff9b0074be0 msgr2=0x7ff9b0077000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.893+0000 7ff9d5a00640 1 --2- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:6822/2127910843,v1:172.21.15.103:6824/2127910843] conn(0x7ff9b0074be0 0x7ff9b0077000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.893+0000 7ff9d5a00640 1 -- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff9b006c9c0 msgr2=0x7ff9b006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.893+0000 7fb635400640 1 -- 172.21.15.45:0/1259113255 shutdown_connections 2024-04-16T16:13:32.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.893+0000 7fb635400640 1 -- 172.21.15.45:0/1259113255 wait complete. 2024-04-16T16:13:32.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.893+0000 7ff9d5a00640 1 --2- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff9b006c9c0 0x7ff9b006ee80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0036000 tx=0x7ff9c0008030 comp rx=0 tx=0).stop 2024-04-16T16:13:32.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.893+0000 7ff9d5a00640 1 -- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d015ed30 msgr2=0x7ff9d015cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.893+0000 7ff9d5a00640 1 --2- 172.21.15.45:0/3065376034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d015ed30 0x7ff9d015cbe0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800cae0 tx=0x7ff9c800a060 comp rx=0 tx=0).stop 2024-04-16T16:13:32.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.893+0000 7ff9cf400640 1 -- 172.21.15.45:0/3065376034 reap_dead start 2024-04-16T16:13:32.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.894+0000 7ff9d5a00640 1 -- 172.21.15.45:0/3065376034 shutdown_connections 2024-04-16T16:13:32.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.894+0000 7ff9d5a00640 1 -- 172.21.15.45:0/3065376034 >> 172.21.15.45:0/3065376034 conn(0x7ff9d00d1610 msgr2=0x7ff9d00b9360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.894+0000 7ff9d5a00640 1 -- 172.21.15.45:0/3065376034 shutdown_connections 2024-04-16T16:13:32.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.894+0000 7ff9d5a00640 1 -- 172.21.15.45:0/3065376034 wait complete. 2024-04-16T16:13:32.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.898+0000 7f6dc3e00640 1 -- 172.21.15.45:0/354001199 --> [v2:172.21.15.103:6816/1303888027,v1:172.21.15.103:6817/1303888027] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6d98002dd0 con 0x7f6d9c074b60 2024-04-16T16:13:32.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.899+0000 7f6db9600640 1 -- 172.21.15.45:0/354001199 <== osd.7 v2:172.21.15.103:6816/1303888027 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6d98002dd0 con 0x7f6d9c074b60 2024-04-16T16:13:32.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.899+0000 7f6dc3e00640 1 -- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:6816/1303888027,v1:172.21.15.103:6817/1303888027] conn(0x7f6d9c074b60 msgr2=0x7f6d9c076f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.899+0000 7f6dc3e00640 1 --2- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:6816/1303888027,v1:172.21.15.103:6817/1303888027] conn(0x7f6d9c074b60 0x7f6d9c076f80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.900+0000 7f6dc3e00640 1 -- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6d9c06c940 msgr2=0x7f6d9c06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.900+0000 7f07c8200640 1 -- 172.21.15.45:0/1929531094 --> [v2:172.21.15.45:6816/707516363,v1:172.21.15.45:6817/707516363] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f07c0070020 con 0x7f07a0074b60 2024-04-16T16:13:32.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.900+0000 7f6dc3e00640 1 --2- 172.21.15.45:0/354001199 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6d9c06c940 0x7f6d9c06ee00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6db00012e0 tx=0x7f6db0001990 comp rx=0 tx=0).stop 2024-04-16T16:13:32.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.900+0000 7f6dc3e00640 1 -- 172.21.15.45:0/354001199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dbc15ed30 msgr2=0x7f6dbc0cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.900+0000 7f6dc3e00640 1 --2- 172.21.15.45:0/354001199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dbc15ed30 0x7f6dbc0cbfc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6db400c3c0 tx=0x7f6db400c880 comp rx=0 tx=0).stop 2024-04-16T16:13:32.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.900+0000 7f6dc1e00640 1 -- 172.21.15.45:0/354001199 reap_dead start 2024-04-16T16:13:32.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.900+0000 7f6dc3e00640 1 -- 172.21.15.45:0/354001199 shutdown_connections 2024-04-16T16:13:32.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.901+0000 7f07bd600640 1 -- 172.21.15.45:0/1929531094 <== osd.3 v2:172.21.15.45:6816/707516363 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f07c0070020 con 0x7f07a0074b60 2024-04-16T16:13:32.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.900+0000 7f6dc3e00640 1 -- 172.21.15.45:0/354001199 >> 172.21.15.45:0/354001199 conn(0x7f6dbc0d1610 msgr2=0x7f6dbc0d19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.901+0000 7f6dc3e00640 1 -- 172.21.15.45:0/354001199 shutdown_connections 2024-04-16T16:13:32.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.901+0000 7f6dc3e00640 1 -- 172.21.15.45:0/354001199 wait complete. 2024-04-16T16:13:32.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.902+0000 7f07c8200640 1 -- 172.21.15.45:0/1929531094 >> [v2:172.21.15.45:6816/707516363,v1:172.21.15.45:6817/707516363] conn(0x7f07a0074b60 msgr2=0x7f07a0076f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.902+0000 7f07c8200640 1 --2- 172.21.15.45:0/1929531094 >> [v2:172.21.15.45:6816/707516363,v1:172.21.15.45:6817/707516363] conn(0x7f07a0074b60 0x7f07a0076f80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.902+0000 7f07c8200640 1 -- 172.21.15.45:0/1929531094 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f07a006c940 msgr2=0x7f07a006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.902+0000 7f07c8200640 1 --2- 172.21.15.45:0/1929531094 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f07a006c940 0x7f07a006ee00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f07b4036000 tx=0x7f07b4030000 comp rx=0 tx=0).stop 2024-04-16T16:13:32.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.902+0000 7f07c8200640 1 -- 172.21.15.45:0/1929531094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07c00b5640 msgr2=0x7f07c015dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.902+0000 7f07c8200640 1 --2- 172.21.15.45:0/1929531094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07c00b5640 0x7f07c015dd80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f07b800efb0 tx=0x7f07b800c490 comp rx=0 tx=0).stop 2024-04-16T16:13:32.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.903+0000 7f07c6200640 1 -- 172.21.15.45:0/1929531094 reap_dead start 2024-04-16T16:13:32.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.903+0000 7f07c8200640 1 -- 172.21.15.45:0/1929531094 shutdown_connections 2024-04-16T16:13:32.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.903+0000 7f07c8200640 1 -- 172.21.15.45:0/1929531094 >> 172.21.15.45:0/1929531094 conn(0x7f07c006b930 msgr2=0x7f07c00b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.904+0000 7f07c8200640 1 -- 172.21.15.45:0/1929531094 shutdown_connections 2024-04-16T16:13:32.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.906+0000 7f07c8200640 1 -- 172.21.15.45:0/1929531094 wait complete. 2024-04-16T16:13:32.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.906+0000 7fb814000640 1 -- 172.21.15.45:0/1949576262 --> [v2:172.21.15.103:6808/3613143327,v1:172.21.15.103:6809/3613143327] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb80c075780 con 0x7fb7ec074af0 2024-04-16T16:13:32.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.907+0000 7fb809600640 1 -- 172.21.15.45:0/1949576262 <== osd.6 v2:172.21.15.103:6808/3613143327 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb80c075780 con 0x7fb7ec074af0 2024-04-16T16:13:32.925 INFO:teuthology.orchestra.run.smithi045.stdout:55834574853 2024-04-16T16:13:32.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.908+0000 7fb814000640 1 -- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:6808/3613143327,v1:172.21.15.103:6809/3613143327] conn(0x7fb7ec074af0 msgr2=0x7fb7ec076f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.908+0000 7fb814000640 1 --2- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:6808/3613143327,v1:172.21.15.103:6809/3613143327] conn(0x7fb7ec074af0 0x7fb7ec076f10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:32.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T16:13:32.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.908+0000 7fb814000640 1 -- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb7ec06c8f0 msgr2=0x7fb7ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.908+0000 7fb814000640 1 --2- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb7ec06c8f0 0x7fb7ec06edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb80c15fdc0 tx=0x7fb7f8007e50 comp rx=0 tx=0).stop 2024-04-16T16:13:32.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.908+0000 7fb814000640 1 -- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb80c0ba710 msgr2=0x7fb80c15a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:32.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.908+0000 7fb814000640 1 --2- 172.21.15.45:0/1949576262 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb80c0ba710 0x7fb80c15a030 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb80000c480 tx=0x7fb80000c940 comp rx=0 tx=0).stop 2024-04-16T16:13:32.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.908+0000 7fb812000640 1 -- 172.21.15.45:0/1949576262 reap_dead start 2024-04-16T16:13:32.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.908+0000 7fb814000640 1 -- 172.21.15.45:0/1949576262 shutdown_connections 2024-04-16T16:13:32.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.908+0000 7fb814000640 1 -- 172.21.15.45:0/1949576262 >> 172.21.15.45:0/1949576262 conn(0x7fb80c0d1610 msgr2=0x7fb80c0d4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:32.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.909+0000 7fb814000640 1 -- 172.21.15.45:0/1949576262 shutdown_connections 2024-04-16T16:13:32.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:32.909+0000 7fb814000640 1 -- 172.21.15.45:0/1949576262 wait complete. 2024-04-16T16:13:32.928 INFO:teuthology.orchestra.run.smithi045.stdout:55834574854 2024-04-16T16:13:32.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-16T16:13:32.929 INFO:teuthology.orchestra.run.smithi045.stdout:55834574853 2024-04-16T16:13:32.930 INFO:teuthology.orchestra.run.smithi045.stdout:55834574852 2024-04-16T16:13:32.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T16:13:32.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T16:13:32.931 INFO:teuthology.orchestra.run.smithi045.stdout:55834574853 2024-04-16T16:13:32.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-16T16:13:32.932 INFO:teuthology.orchestra.run.smithi045.stdout:55834574852 2024-04-16T16:13:32.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-16T16:13:32.934 INFO:teuthology.orchestra.run.smithi045.stdout:55834574853 2024-04-16T16:13:32.934 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-16T16:13:32.935 INFO:teuthology.orchestra.run.smithi045.stdout:55834574852 2024-04-16T16:13:32.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T16:13:33.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.267+0000 7f7986a00640 1 Processor -- start 2024-04-16T16:13:33.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.268+0000 7f7986a00640 1 -- start start 2024-04-16T16:13:33.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.270+0000 7f7986a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f79780589c0 0x7f7978053530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.271+0000 7f7986a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7978057d90 con 0x7f7978056ab0 2024-04-16T16:13:33.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.271+0000 7f7986a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7978006260 con 0x7f79780e7710 2024-04-16T16:13:33.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.271+0000 7f7986a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7978006540 con 0x7f79780589c0 2024-04-16T16:13:33.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.271+0000 7f7986000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7978056ab0 0x7f7978007020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40392/0 (socket says 172.21.15.45:40392) 2024-04-16T16:13:33.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.271+0000 7f7986000640 1 -- 172.21.15.45:0/313332119 learned_addr learned my addr 172.21.15.45:0/313332119 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:33.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.272+0000 7f7984c00640 1 --2- 172.21.15.45:0/313332119 >> v2:172.21.15.103:3300/0 conn(0x7f79780589c0 0x7f7978053530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.272+0000 7f9077200640 1 Processor -- start 2024-04-16T16:13:33.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.273+0000 7f797fe00640 1 -- 172.21.15.45:0/313332119 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3311972307 0 0) 0x7f7978006260 con 0x7f79780e7710 2024-04-16T16:13:33.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.273+0000 7f797fe00640 1 -- 172.21.15.45:0/313332119 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7968003520 con 0x7f79780e7710 2024-04-16T16:13:33.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.273+0000 7f797fe00640 1 -- 172.21.15.45:0/313332119 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1121634033 0 0) 0x7f7978057d90 con 0x7f7978056ab0 2024-04-16T16:13:33.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.273+0000 7f797fe00640 1 -- 172.21.15.45:0/313332119 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7978006260 con 0x7f7978056ab0 2024-04-16T16:13:33.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.273+0000 7f9077200640 1 -- start start 2024-04-16T16:13:33.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.275+0000 7f9077200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f907806f410 0x7f9078163010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.274+0000 7f7984c00640 1 -- 172.21.15.45:0/313332119 >> v1:172.21.15.103:6789/0 conn(0x7f79780e7710 legacy=0x7f797804ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.274+0000 7f7984c00640 1 -- 172.21.15.45:0/313332119 >> v1:172.21.15.45:6789/0 conn(0x7f7978056ab0 legacy=0x7f7978007020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.274+0000 7f7984c00640 1 -- 172.21.15.45:0/313332119 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7978006820 con 0x7f79780589c0 2024-04-16T16:13:33.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.275+0000 7f9077200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f90780ce710 0x7f9078163ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.275+0000 7f9077200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90780cd210 con 0x7f90780b1ce0 2024-04-16T16:13:33.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.275+0000 7f9077200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f90780d02f0 con 0x7f90780ce710 2024-04-16T16:13:33.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.275+0000 7f9077200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f90780cd4f0 con 0x7f907806f410 2024-04-16T16:13:33.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.276+0000 7f7984c00640 1 --2- 172.21.15.45:0/313332119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79780589c0 0x7f7978053530 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f79740097a0 tx=0x7f797402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aff5cba124430138 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.276+0000 7f9076800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f90780ce710 0x7f9078163ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.276+0000 7f9075e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f90780b1ce0 0x7f90780b20c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40400/0 (socket says 172.21.15.45:40400) 2024-04-16T16:13:33.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.276+0000 7f9075e00640 1 -- 172.21.15.45:0/2017722296 learned_addr learned my addr 172.21.15.45:0/2017722296 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:33.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.276+0000 7f9075400640 1 --2- 172.21.15.45:0/2017722296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f907806f410 0x7f9078163010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.277+0000 7f797fe00640 1 -- 172.21.15.45:0/313332119 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79740315a0 con 0x7f79780589c0 2024-04-16T16:13:33.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.277+0000 7f797fe00640 1 -- 172.21.15.45:0/313332119 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7974037070 con 0x7f79780589c0 2024-04-16T16:13:33.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.278+0000 7f7986a00640 1 -- 172.21.15.45:0/313332119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79780589c0 msgr2=0x7f7978053530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.278+0000 7f7986a00640 1 --2- 172.21.15.45:0/313332119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79780589c0 0x7f7978053530 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f79740097a0 tx=0x7f797402b840 comp rx=0 tx=0).stop 2024-04-16T16:13:33.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.278+0000 7f9074a00640 1 -- 172.21.15.45:0/2017722296 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1519802846 0 0) 0x7f90780cd210 con 0x7f90780b1ce0 2024-04-16T16:13:33.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.278+0000 7f7986a00640 1 -- 172.21.15.45:0/313332119 shutdown_connections 2024-04-16T16:13:33.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.278+0000 7f7986a00640 1 --2- 172.21.15.45:0/313332119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79780589c0 0x7f7978053530 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f79740097a0 tx=0x7f797402b840 comp rx=0 tx=0).stop 2024-04-16T16:13:33.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.278+0000 7f7986a00640 1 -- 172.21.15.45:0/313332119 >> 172.21.15.45:0/313332119 conn(0x7f7978058ee0 msgr2=0x7f79780592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.278+0000 7f9075400640 1 -- 172.21.15.45:0/2017722296 >> v2:172.21.15.103:3300/0 conn(0x7f90780ce710 msgr2=0x7f9078163ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:33.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.278+0000 7f9075400640 1 --2- 172.21.15.45:0/2017722296 >> v2:172.21.15.103:3300/0 conn(0x7f90780ce710 0x7f9078163ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.278+0000 7f9075400640 1 -- 172.21.15.45:0/2017722296 >> v1:172.21.15.45:6789/0 conn(0x7f90780b1ce0 legacy=0x7f90780b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.278+0000 7f9075400640 1 -- 172.21.15.45:0/2017722296 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9078163680 con 0x7f907806f410 2024-04-16T16:13:33.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.279+0000 7f7986a00640 1 -- 172.21.15.45:0/313332119 shutdown_connections 2024-04-16T16:13:33.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.280+0000 7f6151800640 1 Processor -- start 2024-04-16T16:13:33.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.280+0000 7f9075400640 1 --2- 172.21.15.45:0/2017722296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f907806f410 0x7f9078163010 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f90600097a0 tx=0x7f906002b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c66a9a87e800c0e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.280+0000 7f6151800640 1 -- start start 2024-04-16T16:13:33.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.281+0000 7f9074a00640 1 -- 172.21.15.45:0/2017722296 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9060038070 con 0x7f907806f410 2024-04-16T16:13:33.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.280+0000 7f7986a00640 1 -- 172.21.15.45:0/313332119 wait complete. 2024-04-16T16:13:33.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.282+0000 7ff052a00640 1 Processor -- start 2024-04-16T16:13:33.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.282+0000 7ff052a00640 1 -- start start 2024-04-16T16:13:33.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.282+0000 7f6151800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614c06f730 0x7f614c0c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.282+0000 7f6151800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f614c0c6320 0x7f614c06df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.282+0000 7f6151800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f614c15d600 con 0x7f614c0c6fc0 2024-04-16T16:13:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.282+0000 7f6151800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f614c0c5670 con 0x7f614c0c6320 2024-04-16T16:13:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.282+0000 7f6151800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f614c15d8e0 con 0x7f614c06f730 2024-04-16T16:13:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.283+0000 7f9074a00640 1 -- 172.21.15.45:0/2017722296 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9060031520 con 0x7f907806f410 2024-04-16T16:13:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.283+0000 7f9074a00640 1 -- 172.21.15.45:0/2017722296 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90600318f0 con 0x7f907806f410 2024-04-16T16:13:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.283+0000 7fc61ee00640 1 Processor -- start 2024-04-16T16:13:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.283+0000 7f614b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614c06f730 0x7f614c0c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.283+0000 7f614b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614c06f730 0x7f614c0c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:36420/0 (socket says 172.21.15.45:36420) 2024-04-16T16:13:33.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.283+0000 7f614b400640 1 -- 172.21.15.45:0/489624419 learned_addr learned my addr 172.21.15.45:0/489624419 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:33.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.283+0000 7f51a0a00640 1 Processor -- start 2024-04-16T16:13:33.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.283+0000 7ff052a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff04c0afe90 0x7ff04c0b5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.284+0000 7f51a0a00640 1 -- start start 2024-04-16T16:13:33.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.283+0000 7f7986a00640 1 Processor -- start 2024-04-16T16:13:33.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.284+0000 7fc61ee00640 1 -- start start 2024-04-16T16:13:33.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.284+0000 7ff050a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff04c072890 0x7ff04c160160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40416/0 (socket says 172.21.15.45:40416) 2024-04-16T16:13:33.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.284+0000 7ff050a00640 1 -- 172.21.15.45:0/1640988929 learned_addr learned my addr 172.21.15.45:0/1640988929 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:33.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.284+0000 7f6150e00640 1 --2- 172.21.15.45:0/489624419 >> v2:172.21.15.103:3300/0 conn(0x7f614c0c6320 0x7f614c06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.284+0000 7f614b400640 1 -- 172.21.15.45:0/489624419 >> v2:172.21.15.103:3300/0 conn(0x7f614c0c6320 msgr2=0x7f614c06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:33.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.284+0000 7f614b400640 1 --2- 172.21.15.45:0/489624419 >> v2:172.21.15.103:3300/0 conn(0x7f614c0c6320 0x7f614c06df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.284+0000 7f614b400640 1 -- 172.21.15.45:0/489624419 >> v1:172.21.15.103:6789/0 conn(0x7f614c0c6fc0 legacy=0x7f614c15cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:33.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.284+0000 7f614b400640 1 -- 172.21.15.45:0/489624419 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614c15dbc0 con 0x7f614c06f730 2024-04-16T16:13:33.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.284+0000 7ff052a00640 1 -- 172.21.15.45:0/1640988929 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff04c164440 con 0x7ff04c072890 2024-04-16T16:13:33.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.284+0000 7ff04be00640 1 --2- 172.21.15.45:0/1640988929 >> v2:172.21.15.45:3300/0 conn(0x7ff04c0afe90 0x7ff04c0b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.284+0000 7f7986a00640 1 -- start start 2024-04-16T16:13:33.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.285+0000 7ff052a00640 1 -- 172.21.15.45:0/1640988929 --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff04c0b5ca0 con 0x7ff04c0b5640 2024-04-16T16:13:33.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.285+0000 7ff052a00640 1 -- 172.21.15.45:0/1640988929 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff04c06ef10 con 0x7ff04c0afe90 2024-04-16T16:13:33.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.285+0000 7f9077200640 1 -- 172.21.15.45:0/2017722296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f907806f410 msgr2=0x7f9078163010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.285+0000 7f9077200640 1 --2- 172.21.15.45:0/2017722296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f907806f410 0x7f9078163010 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f90600097a0 tx=0x7f906002b7c0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.285+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1916064951 0 0) 0x7ff04c164440 con 0x7ff04c072890 2024-04-16T16:13:33.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff030003520 con 0x7ff04c072890 2024-04-16T16:13:33.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.285+0000 7f9077200640 1 -- 172.21.15.45:0/2017722296 shutdown_connections 2024-04-16T16:13:33.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.285+0000 7f9077200640 1 --2- 172.21.15.45:0/2017722296 >> v2:172.21.15.103:3300/0 conn(0x7f90780ce710 0x7f9078163ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.285+0000 7f9077200640 1 --2- 172.21.15.45:0/2017722296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f907806f410 0x7f9078163010 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.285+0000 7f9077200640 1 -- 172.21.15.45:0/2017722296 >> 172.21.15.45:0/2017722296 conn(0x7f907806b930 msgr2=0x7f907806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1418728906 0 0) 0x7ff04c0b5ca0 con 0x7ff04c0b5640 2024-04-16T16:13:33.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff04c164440 con 0x7ff04c0b5640 2024-04-16T16:13:33.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7f7986a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7978056ab0 0x7f79780e7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7fc61ee00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc61815ea70 0x7fc6181626f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7f614b400640 1 --2- 172.21.15.45:0/489624419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614c06f730 0x7f614c0c5de0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6138009870 tx=0x7f613802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d04c014d9aa66f0d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7f7986a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79780e7710 0x7f79780ddd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7f7986a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7978052150 0x7f797817d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7f51a0a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f519c0cec90 0x7f519c06e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7f9077200640 1 -- 172.21.15.45:0/2017722296 shutdown_connections 2024-04-16T16:13:33.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7f7986a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7978057b20 con 0x7f7978056ab0 2024-04-16T16:13:33.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7f51a0a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f519c15e7c0 con 0x7f519c06f700 2024-04-16T16:13:33.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7f51a0a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f519c15eaa0 con 0x7f519c06c040 2024-04-16T16:13:33.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7f51a0a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f519c15f3d0 con 0x7f519c0cec90 2024-04-16T16:13:33.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7fc61ee00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc618162f40 con 0x7fc6180ba5a0 2024-04-16T16:13:33.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7fc61ee00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc618163220 con 0x7fc6180cfdf0 2024-04-16T16:13:33.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7fc61ee00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc618163500 con 0x7fc61815ea70 2024-04-16T16:13:33.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7f7986a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79780579a0 con 0x7f79780e7710 2024-04-16T16:13:33.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.287+0000 7f5199400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f519c0cec90 0x7f519c06e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7f7986a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7978007730 con 0x7f7978052150 2024-04-16T16:13:33.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.287+0000 7fc61d000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc6180ba5a0 0x7fc6180cf6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40418/0 (socket says 172.21.15.45:40418) 2024-04-16T16:13:33.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.287+0000 7fc61d000640 1 -- 172.21.15.45:0/786848719 learned_addr learned my addr 172.21.15.45:0/786848719 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:33.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.287+0000 7f5199400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f519c0cec90 0x7f519c06e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49864/0 (socket says 172.21.15.45:49864) 2024-04-16T16:13:33.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.287+0000 7f5199400640 1 -- 172.21.15.45:0/545975415 learned_addr learned my addr 172.21.15.45:0/545975415 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:33.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.287+0000 7f614aa00640 1 -- 172.21.15.45:0/489624419 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61380317c0 con 0x7f614c06f730 2024-04-16T16:13:33.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.287+0000 7f614aa00640 1 -- 172.21.15.45:0/489624419 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6138037070 con 0x7f614c06f730 2024-04-16T16:13:33.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.287+0000 7f614aa00640 1 -- 172.21.15.45:0/489624419 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6138031b80 con 0x7f614c06f730 2024-04-16T16:13:33.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.287+0000 7f7984c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79780e7710 0x7f79780ddd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.287+0000 7f7984c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79780e7710 0x7f79780ddd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:37760/0 (socket says 172.21.15.45:37760) 2024-04-16T16:13:33.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.287+0000 7f7984c00640 1 -- 172.21.15.45:0/2371806977 learned_addr learned my addr 172.21.15.45:0/2371806977 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:33.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.287+0000 7f7985600640 1 --2- 172.21.15.45:0/2371806977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7978056ab0 0x7f79780e7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.287+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 338792369 0 0) 0x7ff04c164440 con 0x7ff04c0b5640 2024-04-16T16:13:33.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.287+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 >> v2:172.21.15.45:3300/0 conn(0x7ff04c0afe90 msgr2=0x7ff04c0b5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:33.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7f7984c00640 1 -- 172.21.15.45:0/2371806977 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7978052150 msgr2=0x7f797817d150 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-16T16:13:33.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7f7984c00640 1 --2- 172.21.15.45:0/2371806977 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7978052150 0x7f797817d150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7f7984c00640 1 -- 172.21.15.45:0/2371806977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7978056ab0 msgr2=0x7f79780e7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7fc61da00640 1 --2- 172.21.15.45:0/786848719 >> v2:172.21.15.45:3300/0 conn(0x7fc61815ea70 0x7fc6181626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7fc617e00640 1 -- 172.21.15.45:0/786848719 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2857486471 0 0) 0x7fc618162f40 con 0x7fc6180ba5a0 2024-04-16T16:13:33.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.286+0000 7f9077200640 1 -- 172.21.15.45:0/2017722296 wait complete. 2024-04-16T16:13:33.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7f6151800640 1 -- 172.21.15.45:0/489624419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614c06f730 msgr2=0x7f614c0c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7f6151800640 1 --2- 172.21.15.45:0/489624419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614c06f730 0x7f614c0c5de0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6138009870 tx=0x7f613802b860 comp rx=0 tx=0).stop 2024-04-16T16:13:33.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7f7984c00640 1 --2- 172.21.15.45:0/2371806977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7978056ab0 0x7f79780e7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7f7984c00640 1 -- 172.21.15.45:0/2371806977 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79780e8030 con 0x7f79780e7710 2024-04-16T16:13:33.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7f7985600640 1 --2- 172.21.15.45:0/2371806977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7978056ab0 0x7f79780e7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:33.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7f5199400640 1 -- 172.21.15.45:0/545975415 >> v1:172.21.15.103:6789/0 conn(0x7f519c06c040 legacy=0x7f519c06df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7ff04aa00640 1 --2- 172.21.15.45:0/1640988929 >> v2:172.21.15.45:3300/0 conn(0x7ff04c0afe90 0x7ff04c0b5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 >> v1:172.21.15.45:6789/0 conn(0x7ff04c072890 legacy=0x7ff04c160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7f5199400640 1 -- 172.21.15.45:0/545975415 >> v1:172.21.15.45:6789/0 conn(0x7f519c06f700 legacy=0x7f519c165810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:33.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7f5199400640 1 -- 172.21.15.45:0/545975415 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f519c15ed80 con 0x7f519c0cec90 2024-04-16T16:13:33.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04c166a60 con 0x7ff04c0b5640 2024-04-16T16:13:33.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7ff03c0015a0 con 0x7ff04c0b5640 2024-04-16T16:13:33.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 >> v1:172.21.15.103:6789/0 conn(0x7ff04c0b5640 legacy=0x7ff04c072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7ff04aa00640 1 --2- 172.21.15.45:0/1640988929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff030003b40 0x7ff030003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7ff04aa00640 1 --2- 172.21.15.45:0/1640988929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff030004800 0x7ff030004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7ff04aa00640 1 --2- 172.21.15.45:0/1640988929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff030005230 0x7ff03000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c06ef10 con 0x7ff030005230 2024-04-16T16:13:33.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff03000c190 con 0x7ff030004800 2024-04-16T16:13:33.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.288+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff03000c390 con 0x7ff030003b40 2024-04-16T16:13:33.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.289+0000 7fc617e00640 1 -- 172.21.15.45:0/786848719 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc600003520 con 0x7fc6180ba5a0 2024-04-16T16:13:33.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.289+0000 7fc617e00640 1 -- 172.21.15.45:0/786848719 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1312733425 0 0) 0x7fc618163220 con 0x7fc6180cfdf0 2024-04-16T16:13:33.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.289+0000 7fc617e00640 1 -- 172.21.15.45:0/786848719 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc618162f40 con 0x7fc6180cfdf0 2024-04-16T16:13:33.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.289+0000 7f7984c00640 1 --2- 172.21.15.45:0/2371806977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79780e7710 0x7f79780ddd50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7974009890 tx=0x7f79740046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.289+0000 7ff04be00640 1 --2- 172.21.15.45:0/1640988929 >> v2:172.21.15.45:3300/0 conn(0x7ff04c0afe90 0x7ff04c0b5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:13:33.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.289+0000 7f9077200640 1 Processor -- start 2024-04-16T16:13:33.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.289+0000 7f9077200640 1 -- start start 2024-04-16T16:13:33.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.289+0000 7fc61da00640 1 -- 172.21.15.45:0/786848719 >> v1:172.21.15.103:6789/0 conn(0x7fc6180cfdf0 legacy=0x7fc6180d3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.289+0000 7fc61da00640 1 -- 172.21.15.45:0/786848719 >> v1:172.21.15.45:6789/0 conn(0x7fc6180ba5a0 legacy=0x7fc6180cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.289+0000 7fc61da00640 1 -- 172.21.15.45:0/786848719 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6180d2d50 con 0x7fc61815ea70 2024-04-16T16:13:33.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.290+0000 7ff050a00640 1 --2- 172.21.15.45:0/1640988929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff030005230 0x7ff03000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.290+0000 7f797d600640 1 -- 172.21.15.45:0/2371806977 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7974032940 con 0x7f79780e7710 2024-04-16T16:13:33.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.290+0000 7f797d600640 1 -- 172.21.15.45:0/2371806977 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7974032ae0 con 0x7f79780e7710 2024-04-16T16:13:33.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.290+0000 7f797d600640 1 -- 172.21.15.45:0/2371806977 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7974045400 con 0x7f79780e7710 2024-04-16T16:13:33.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.290+0000 7ff04be00640 1 --2- 172.21.15.45:0/1640988929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff030003b40 0x7ff030003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.290+0000 7f7986a00640 1 -- 172.21.15.45:0/2371806977 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79780e8300 con 0x7f79780e7710 2024-04-16T16:13:33.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.290+0000 7f6151800640 1 -- 172.21.15.45:0/489624419 shutdown_connections 2024-04-16T16:13:33.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.290+0000 7f6151800640 1 --2- 172.21.15.45:0/489624419 >> v2:172.21.15.103:3300/0 conn(0x7f614c0c6320 0x7f614c06df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.290+0000 7f6151800640 1 --2- 172.21.15.45:0/489624419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614c06f730 0x7f614c0c5de0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.290+0000 7f6151800640 1 -- 172.21.15.45:0/489624419 >> 172.21.15.45:0/489624419 conn(0x7f614c06b930 msgr2=0x7f614c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.290+0000 7f7986a00640 1 -- 172.21.15.45:0/2371806977 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79780e88c0 con 0x7f79780e7710 2024-04-16T16:13:33.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.290+0000 7ff04b400640 1 --2- 172.21.15.45:0/1640988929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff030004800 0x7ff030004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.291+0000 7f6151800640 1 -- 172.21.15.45:0/489624419 shutdown_connections 2024-04-16T16:13:33.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.291+0000 7ff050a00640 1 -- 172.21.15.45:0/1640988929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff030003b40 msgr2=0x7ff030003f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:33.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.291+0000 7ff050a00640 1 --2- 172.21.15.45:0/1640988929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff030003b40 0x7ff030003f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.291+0000 7f6151800640 1 -- 172.21.15.45:0/489624419 wait complete. 2024-04-16T16:13:33.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.291+0000 7f9077200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f907806f410 0x7f9078144b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.291+0000 7f9077200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90780b1ce0 0x7f907815d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.291+0000 7f9077200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90780ce710 0x7f907814d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.291+0000 7f9077200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f90781640e0 con 0x7f907806f410 2024-04-16T16:13:33.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.291+0000 7f9077200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f90780d00d0 con 0x7f90780b1ce0 2024-04-16T16:13:33.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.291+0000 7f9077200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9078164260 con 0x7f90780ce710 2024-04-16T16:13:33.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.291+0000 7fc61da00640 1 --2- 172.21.15.45:0/786848719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc61815ea70 0x7fc6181626f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc60c0097a0 tx=0x7fc60c02ba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80f7d4d5998f160c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.292+0000 7fc617e00640 1 -- 172.21.15.45:0/786848719 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60c02f820 con 0x7fc61815ea70 2024-04-16T16:13:33.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.291+0000 7f9075e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f907806f410 0x7f9078144b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.292+0000 7f9075e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f907806f410 0x7f9078144b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49896/0 (socket says 172.21.15.45:49896) 2024-04-16T16:13:33.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.292+0000 7f9075e00640 1 -- 172.21.15.45:0/124492984 learned_addr learned my addr 172.21.15.45:0/124492984 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:33.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.292+0000 7fc617e00640 1 -- 172.21.15.45:0/786848719 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc60c02f9c0 con 0x7fc61815ea70 2024-04-16T16:13:33.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.292+0000 7fc617e00640 1 -- 172.21.15.45:0/786848719 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60c02fd90 con 0x7fc61815ea70 2024-04-16T16:13:33.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.292+0000 7f9076800640 1 --2- 172.21.15.45:0/124492984 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90780ce710 0x7f907814d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.292+0000 7f5199400640 1 --2- 172.21.15.45:0/545975415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c0cec90 0x7f519c06e610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5188009870 tx=0x7f518802ff40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c68d95ebd8b85ca server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.292+0000 7f9075e00640 1 -- 172.21.15.45:0/124492984 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90780ce710 msgr2=0x7f907814d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.292+0000 7f9075400640 1 --2- 172.21.15.45:0/124492984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90780b1ce0 0x7f907815d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.292+0000 7f9075e00640 1 --2- 172.21.15.45:0/124492984 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90780ce710 0x7f907814d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.292+0000 7f9075e00640 1 -- 172.21.15.45:0/124492984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90780b1ce0 msgr2=0x7f907815d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.292+0000 7f9075e00640 1 --2- 172.21.15.45:0/124492984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90780b1ce0 0x7f907815d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.292+0000 7f9075e00640 1 -- 172.21.15.45:0/124492984 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90780c6680 con 0x7f907806f410 2024-04-16T16:13:33.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.293+0000 7f7986a00640 1 -- 172.21.15.45:0/2371806977 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7978005390 con 0x7f79780e7710 2024-04-16T16:13:33.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.293+0000 7f9075400640 1 --2- 172.21.15.45:0/124492984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90780b1ce0 0x7f907815d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:33.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.293+0000 7f5bf1200640 1 Processor -- start 2024-04-16T16:13:33.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.293+0000 7f5bf1200640 1 -- start start 2024-04-16T16:13:33.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.293+0000 7fc61ee00640 1 -- 172.21.15.45:0/786848719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc61815ea70 msgr2=0x7fc6181626f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.293+0000 7fc61ee00640 1 --2- 172.21.15.45:0/786848719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc61815ea70 0x7fc6181626f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc60c0097a0 tx=0x7fc60c02ba00 comp rx=0 tx=0).stop 2024-04-16T16:13:33.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.293+0000 7f6151800640 1 Processor -- start 2024-04-16T16:13:33.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.293+0000 7f9075e00640 1 --2- 172.21.15.45:0/124492984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f907806f410 0x7f9078144b30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f906800c8c0 tx=0x7f906800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7f906e000640 1 -- 172.21.15.45:0/124492984 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9068018070 con 0x7f907806f410 2024-04-16T16:13:33.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7f6151800640 1 -- start start 2024-04-16T16:13:33.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7fc61ee00640 1 -- 172.21.15.45:0/786848719 shutdown_connections 2024-04-16T16:13:33.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7fc61ee00640 1 --2- 172.21.15.45:0/786848719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc61815ea70 0x7fc6181626f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7fc61ee00640 1 -- 172.21.15.45:0/786848719 >> 172.21.15.45:0/786848719 conn(0x7fc6180d1780 msgr2=0x7fc6180d1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7f906e000640 1 -- 172.21.15.45:0/124492984 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f906800a640 con 0x7f907806f410 2024-04-16T16:13:33.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7f906e000640 1 -- 172.21.15.45:0/124492984 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906800aa70 con 0x7f907806f410 2024-04-16T16:13:33.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7f9077200640 1 -- 172.21.15.45:0/124492984 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90780c8080 con 0x7f907806f410 2024-04-16T16:13:33.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7f5198a00640 1 -- 172.21.15.45:0/545975415 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518803d070 con 0x7f519c0cec90 2024-04-16T16:13:33.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.293+0000 7ff050a00640 1 -- 172.21.15.45:0/1640988929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff030004800 msgr2=0x7ff030004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:33.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.293+0000 7ff050a00640 1 --2- 172.21.15.45:0/1640988929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff030004800 0x7ff030004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.293+0000 7ff050a00640 1 -- 172.21.15.45:0/1640988929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04c166a60 con 0x7ff030005230 2024-04-16T16:13:33.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.293+0000 7ff04b400640 1 --2- 172.21.15.45:0/1640988929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff030004800 0x7ff030004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:13:33.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7f5198a00640 1 -- 172.21.15.45:0/545975415 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5188030cf0 con 0x7f519c0cec90 2024-04-16T16:13:33.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7f5198a00640 1 -- 172.21.15.45:0/545975415 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5188038660 con 0x7f519c0cec90 2024-04-16T16:13:33.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7f9077200640 1 -- 172.21.15.45:0/124492984 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90780c5de0 con 0x7f907806f410 2024-04-16T16:13:33.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7f419e600640 1 Processor -- start 2024-04-16T16:13:33.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7fc61ee00640 1 -- 172.21.15.45:0/786848719 shutdown_connections 2024-04-16T16:13:33.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.295+0000 7ff050a00640 1 --2- 172.21.15.45:0/1640988929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff030005230 0x7ff03000b9c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff03800c160 tx=0x7ff030005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b68a33077c125f19 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.295+0000 7f5bf1200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5be40398d0 0x7f5be40dfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.295+0000 7f5bf1200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5be4054350 0x7f5be4052600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.295+0000 7f5bf1200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5be40e6da0 con 0x7f5be4052b40 2024-04-16T16:13:33.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.295+0000 7f5bf1200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5be40e0400 con 0x7f5be40398d0 2024-04-16T16:13:33.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.295+0000 7f5bf1200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5be4039dd0 con 0x7f5be4054350 2024-04-16T16:13:33.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.295+0000 7ff050a00640 1 -- 172.21.15.45:0/1640988929 reap_dead start 2024-04-16T16:13:33.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7fc61ee00640 1 -- 172.21.15.45:0/786848719 wait complete. 2024-04-16T16:13:33.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f5beaa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5be4054350 0x7f5be4052600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f51a0a00640 1 -- 172.21.15.45:0/545975415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c0cec90 msgr2=0x7f519c06e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f5beaa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5be4054350 0x7f5be4052600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:36456/0 (socket says 172.21.15.45:36456) 2024-04-16T16:13:33.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f5beaa00640 1 -- 172.21.15.45:0/2376640295 learned_addr learned my addr 172.21.15.45:0/2376640295 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:33.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f51a0a00640 1 --2- 172.21.15.45:0/545975415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c0cec90 0x7f519c06e610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5188009870 tx=0x7f518802ff40 comp rx=0 tx=0).stop 2024-04-16T16:13:33.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f797d600640 1 -- 172.21.15.45:0/2371806977 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f797404e430 con 0x7f79780e7710 2024-04-16T16:13:33.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f6151800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f614c06f730 0x7f614c0cc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f6151800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f614c0c6320 0x7f614c1490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f6151800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614c0c6fc0 0x7f614c0c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f6151800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f614c0c7520 con 0x7f614c06f730 2024-04-16T16:13:33.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f6151800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f614c0c73a0 con 0x7f614c0c6320 2024-04-16T16:13:33.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f6151800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f614c06e480 con 0x7f614c0c6fc0 2024-04-16T16:13:33.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f51a0a00640 1 -- 172.21.15.45:0/545975415 shutdown_connections 2024-04-16T16:13:33.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f51a0a00640 1 --2- 172.21.15.45:0/545975415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c0cec90 0x7f519c06e610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f51a0a00640 1 -- 172.21.15.45:0/545975415 >> 172.21.15.45:0/545975415 conn(0x7f519c067e00 msgr2=0x7f519c06b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.294+0000 7f419e600640 1 -- start start 2024-04-16T16:13:33.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f419e600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f41980afe90 0x7f41980b5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f419e600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4198164440 con 0x7f4198072890 2024-04-16T16:13:33.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f419e600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41980b5ca0 con 0x7f41980b5640 2024-04-16T16:13:33.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f419e600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f419806ef10 con 0x7f41980afe90 2024-04-16T16:13:33.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f6150e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614c0c6fc0 0x7f614c0c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f614b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f614c0c6320 0x7f614c1490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f5beaa00640 1 -- 172.21.15.45:0/2376640295 >> v2:172.21.15.45:3300/0 conn(0x7f5be40398d0 msgr2=0x7f5be40dfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:33.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f5beaa00640 1 --2- 172.21.15.45:0/2376640295 >> v2:172.21.15.45:3300/0 conn(0x7f5be40398d0 0x7f5be40dfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.296+0000 7f5beaa00640 1 -- 172.21.15.45:0/2376640295 >> v1:172.21.15.103:6789/0 conn(0x7f5be4052b40 legacy=0x7f5be404fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f5beaa00640 1 -- 172.21.15.45:0/2376640295 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be403a0b0 con 0x7f5be4054350 2024-04-16T16:13:33.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f51a0a00640 1 -- 172.21.15.45:0/545975415 shutdown_connections 2024-04-16T16:13:33.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f614b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f614c0c6320 0x7f614c1490a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:37782/0 (socket says 172.21.15.45:37782) 2024-04-16T16:13:33.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f614b400640 1 -- 172.21.15.45:0/204063217 learned_addr learned my addr 172.21.15.45:0/204063217 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:33.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff038007c20 con 0x7ff030005230 2024-04-16T16:13:33.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f4197e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4198072890 0x7f4198160160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40432/0 (socket says 172.21.15.45:40432) 2024-04-16T16:13:33.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f797d600640 1 --2- 172.21.15.45:0/2371806977 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f796006c8f0 0x7f796006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.298+0000 7f797d600640 1 -- 172.21.15.45:0/2371806977 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f797402f030 con 0x7f79780e7710 2024-04-16T16:13:33.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.298+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff038030070 con 0x7ff030005230 2024-04-16T16:13:33.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f4197e00640 1 -- 172.21.15.45:0/3769072953 learned_addr learned my addr 172.21.15.45:0/3769072953 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:33.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.298+0000 7fc61ee00640 1 Processor -- start 2024-04-16T16:13:33.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f51a0a00640 1 -- 172.21.15.45:0/545975415 wait complete. 2024-04-16T16:13:33.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.298+0000 7f614be00640 1 --2- 172.21.15.45:0/204063217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f614c06f730 0x7f614c0cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.297+0000 7f9077200640 1 -- 172.21.15.45:0/124492984 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9078075780 con 0x7f907806f410 2024-04-16T16:13:33.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.298+0000 7ff04aa00640 1 -- 172.21.15.45:0/1640988929 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff038005cf0 con 0x7ff030005230 2024-04-16T16:13:33.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.298+0000 7f4197400640 1 --2- 172.21.15.45:0/3769072953 >> v2:172.21.15.103:3300/0 conn(0x7f41980afe90 0x7f41980b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.298+0000 7f6150e00640 1 -- 172.21.15.45:0/204063217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f614c0c6320 msgr2=0x7f614c1490a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.298+0000 7f7985600640 1 --2- 172.21.15.45:0/2371806977 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f796006c8f0 0x7f796006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.298+0000 7f6150e00640 1 --2- 172.21.15.45:0/204063217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f614c0c6320 0x7f614c1490a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.298+0000 7f6150e00640 1 -- 172.21.15.45:0/204063217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f614c06f730 msgr2=0x7f614c0cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.298+0000 7f6150e00640 1 --2- 172.21.15.45:0/204063217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f614c06f730 0x7f614c0cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.298+0000 7f6150e00640 1 -- 172.21.15.45:0/204063217 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614c0cca10 con 0x7f614c0c6fc0 2024-04-16T16:13:33.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.299+0000 7f5beaa00640 1 --2- 172.21.15.45:0/2376640295 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5be4054350 0x7f5be4052600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009870 tx=0x7f5bd802b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9c966e7e3e3d689 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.299+0000 7f4196000640 1 -- 172.21.15.45:0/3769072953 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1088592945 0 0) 0x7f4198164440 con 0x7f4198072890 2024-04-16T16:13:33.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.299+0000 7f4196000640 1 -- 172.21.15.45:0/3769072953 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f417c003520 con 0x7f4198072890 2024-04-16T16:13:33.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.299+0000 7ff052a00640 1 -- 172.21.15.45:0/1640988929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff030005230 msgr2=0x7ff03000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.299+0000 7ff052a00640 1 --2- 172.21.15.45:0/1640988929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff030005230 0x7ff03000b9c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff03800c160 tx=0x7ff030005830 comp rx=0 tx=0).stop 2024-04-16T16:13:33.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.298+0000 7fc61ee00640 1 -- start start 2024-04-16T16:13:33.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.299+0000 7ff052a00640 1 -- 172.21.15.45:0/1640988929 shutdown_connections 2024-04-16T16:13:33.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.299+0000 7ff052a00640 1 --2- 172.21.15.45:0/1640988929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff030005230 0x7ff03000b9c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.299+0000 7ff052a00640 1 -- 172.21.15.45:0/1640988929 >> 172.21.15.45:0/1640988929 conn(0x7ff04c06b930 msgr2=0x7ff04c06bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.299+0000 7f5bea000640 1 -- 172.21.15.45:0/2376640295 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8035020 con 0x7f5be4054350 2024-04-16T16:13:33.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.299+0000 7f5bea000640 1 -- 172.21.15.45:0/2376640295 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bd8031870 con 0x7f5be4054350 2024-04-16T16:13:33.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.299+0000 7f5bea000640 1 -- 172.21.15.45:0/2376640295 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8035020 con 0x7f5be4054350 2024-04-16T16:13:33.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.299+0000 7f51a0a00640 1 Processor -- start 2024-04-16T16:13:33.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.299+0000 7f4196000640 1 -- 172.21.15.45:0/3769072953 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3152259458 0 0) 0x7f417c003520 con 0x7f4198072890 2024-04-16T16:13:33.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.299+0000 7f4196000640 1 -- 172.21.15.45:0/3769072953 >> v2:172.21.15.103:3300/0 conn(0x7f41980afe90 msgr2=0x7f41980b5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:33.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.299+0000 7f4196000640 1 --2- 172.21.15.45:0/3769072953 >> v2:172.21.15.103:3300/0 conn(0x7f41980afe90 0x7f41980b5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7f4196000640 1 -- 172.21.15.45:0/3769072953 >> v1:172.21.15.103:6789/0 conn(0x7f41980b5640 legacy=0x7f4198072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7f4196000640 1 -- 172.21.15.45:0/3769072953 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4198166a60 con 0x7f4198072890 2024-04-16T16:13:33.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7f51a0a00640 1 -- start start 2024-04-16T16:13:33.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7ff052a00640 1 -- 172.21.15.45:0/1640988929 shutdown_connections 2024-04-16T16:13:33.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7ff052a00640 1 -- 172.21.15.45:0/1640988929 wait complete. 2024-04-16T16:13:33.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7f906e000640 1 -- 172.21.15.45:0/124492984 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f906801b070 con 0x7f907806f410 2024-04-16T16:13:33.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7f4196000640 1 -- 172.21.15.45:0/3769072953 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f41900019f0 con 0x7f4198072890 2024-04-16T16:13:33.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7f4196000640 1 -- 172.21.15.45:0/3769072953 >> v1:172.21.15.45:6789/0 conn(0x7f4198072890 legacy=0x7f4198160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7f4196000640 1 --2- 172.21.15.45:0/3769072953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f417c0039b0 0x7f417c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7f4196000640 1 --2- 172.21.15.45:0/3769072953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f417c004800 0x7f417c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7f4196000640 1 --2- 172.21.15.45:0/3769072953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417c0051c0 0x7f417c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7f4196000640 1 -- 172.21.15.45:0/3769072953 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f419806ef10 con 0x7f417c0051c0 2024-04-16T16:13:33.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7f4196000640 1 -- 172.21.15.45:0/3769072953 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f417c00c0c0 con 0x7f417c004800 2024-04-16T16:13:33.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7f4196000640 1 -- 172.21.15.45:0/3769072953 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f417c00c2c0 con 0x7f417c0039b0 2024-04-16T16:13:33.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7f5bf1200640 1 -- 172.21.15.45:0/2376640295 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5be4054350 msgr2=0x7f5be4052600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7f5bf1200640 1 --2- 172.21.15.45:0/2376640295 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5be4054350 0x7f5be4052600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009870 tx=0x7f5bd802b910 comp rx=0 tx=0).stop 2024-04-16T16:13:33.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7fc61ee00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6180ba5a0 0x7fc6181f87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7fc61ee00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6180cfdf0 0x7fc6180b81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7fc61ee00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc61815ea70 0x7fc61806f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7fc61ee00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc6180b7ab0 con 0x7fc6180ba5a0 2024-04-16T16:13:33.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7fc61ee00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc618162c30 con 0x7fc6180cfdf0 2024-04-16T16:13:33.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7fc61ee00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc6180b7c30 con 0x7fc61815ea70 2024-04-16T16:13:33.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.300+0000 7f6150e00640 1 --2- 172.21.15.45:0/204063217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614c0c6fc0 0x7f614c0c9dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6144007d90 tx=0x7f614400e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7f4197e00640 1 --2- 172.21.15.45:0/3769072953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417c0051c0 0x7f417c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7f5bf1200640 1 -- 172.21.15.45:0/2376640295 shutdown_connections 2024-04-16T16:13:33.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7f5bf1200640 1 --2- 172.21.15.45:0/2376640295 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5be4054350 0x7f5be4052600 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7f5bf1200640 1 --2- 172.21.15.45:0/2376640295 >> v2:172.21.15.45:3300/0 conn(0x7f5be40398d0 0x7f5be40dfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7f5bf1200640 1 -- 172.21.15.45:0/2376640295 >> 172.21.15.45:0/2376640295 conn(0x7f5be4009fb0 msgr2=0x7f5be400a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7f6143e00640 1 -- 172.21.15.45:0/204063217 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614400edd0 con 0x7f614c0c6fc0 2024-04-16T16:13:33.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7f51a0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c06c040 0x7f519c145080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7f5bf1200640 1 -- 172.21.15.45:0/2376640295 shutdown_connections 2024-04-16T16:13:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7fc61d000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6180cfdf0 0x7fc6180b81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7fc61d000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6180cfdf0 0x7fc6180b81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:37810/0 (socket says 172.21.15.45:37810) 2024-04-16T16:13:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7fc61d000640 1 -- 172.21.15.45:0/2326022950 learned_addr learned my addr 172.21.15.45:0/2326022950 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7fc61e400640 1 --2- 172.21.15.45:0/2326022950 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc61815ea70 0x7fc61806f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7f51a0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f519c06f700 0x7f519c158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7f51a0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f519c0cec90 0x7f519c158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7f51a0a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f519c06eb50 con 0x7f519c06c040 2024-04-16T16:13:33.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7f51a0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f519c0b4e80 con 0x7f519c0cec90 2024-04-16T16:13:33.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7f51a0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f519c06ecd0 con 0x7f519c06f700 2024-04-16T16:13:33.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7f4197400640 1 --2- 172.21.15.45:0/3769072953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f417c0039b0 0x7f417c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7f4196a00640 1 --2- 172.21.15.45:0/3769072953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f417c004800 0x7f417c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7fc61d000640 1 -- 172.21.15.45:0/2326022950 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc61815ea70 msgr2=0x7fc61806f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7fc61d000640 1 --2- 172.21.15.45:0/2326022950 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc61815ea70 0x7fc61806f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7fc61d000640 1 -- 172.21.15.45:0/2326022950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6180ba5a0 msgr2=0x7fc6181f87d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:13:33.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7f7985600640 1 --2- 172.21.15.45:0/2371806977 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f796006c8f0 0x7f796006edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f796c000f60 tx=0x7f796c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7f5199e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c06c040 0x7f519c145080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7f5199e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c06c040 0x7f519c145080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49932/0 (socket says 172.21.15.45:49932) 2024-04-16T16:13:33.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7f5199e00640 1 -- 172.21.15.45:0/3400788114 learned_addr learned my addr 172.21.15.45:0/3400788114 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:33.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7f5199e00640 1 -- 172.21.15.45:0/3400788114 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f519c06f700 msgr2=0x7f519c158aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7f5199e00640 1 --2- 172.21.15.45:0/3400788114 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f519c06f700 0x7f519c158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7f5199e00640 1 -- 172.21.15.45:0/3400788114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f519c0cec90 msgr2=0x7f519c158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7f5199e00640 1 --2- 172.21.15.45:0/3400788114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f519c0cec90 0x7f519c158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7f5199e00640 1 -- 172.21.15.45:0/3400788114 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f519c151a20 con 0x7f519c06c040 2024-04-16T16:13:33.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7f5199e00640 1 --2- 172.21.15.45:0/3400788114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c06c040 0x7f519c145080 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f518c00c760 tx=0x7f518c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7f906e000640 1 --2- 172.21.15.45:0/124492984 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f905406c940 0x7f905406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7f906e000640 1 -- 172.21.15.45:0/124492984 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f906808b310 con 0x7f907806f410 2024-04-16T16:13:33.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7f9075400640 1 --2- 172.21.15.45:0/124492984 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f905406c940 0x7f905406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7f4197e00640 1 -- 172.21.15.45:0/3769072953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f417c0039b0 msgr2=0x7f417c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:33.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7f5bf1200640 1 -- 172.21.15.45:0/2376640295 wait complete. 2024-04-16T16:13:33.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7f6143e00640 1 -- 172.21.15.45:0/204063217 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6144003990 con 0x7f614c0c6fc0 2024-04-16T16:13:33.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.301+0000 7f6143e00640 1 -- 172.21.15.45:0/204063217 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6144003d80 con 0x7f614c0c6fc0 2024-04-16T16:13:33.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7f6151800640 1 -- 172.21.15.45:0/204063217 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f614c0cccf0 con 0x7f614c0c6fc0 2024-04-16T16:13:33.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.302+0000 7f6151800640 1 -- 172.21.15.45:0/204063217 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f614c0cebf0 con 0x7f614c0c6fc0 2024-04-16T16:13:33.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7f4197e00640 1 --2- 172.21.15.45:0/3769072953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f417c0039b0 0x7f417c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7f4197e00640 1 -- 172.21.15.45:0/3769072953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f417c004800 msgr2=0x7f417c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:33.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7f4197e00640 1 --2- 172.21.15.45:0/3769072953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f417c004800 0x7f417c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7f4197e00640 1 -- 172.21.15.45:0/3769072953 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f417c003520 con 0x7f417c0051c0 2024-04-16T16:13:33.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7f4197400640 1 --2- 172.21.15.45:0/3769072953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f417c0039b0 0x7f417c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:33.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.304+0000 7ff052a00640 1 Processor -- start 2024-04-16T16:13:33.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7fc61da00640 1 --2- 172.21.15.45:0/2326022950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6180ba5a0 0x7fc6181f87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7fc61d000640 1 --2- 172.21.15.45:0/2326022950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6180ba5a0 0x7fc6181f87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7fc61d000640 1 -- 172.21.15.45:0/2326022950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc618072330 con 0x7fc6180cfdf0 2024-04-16T16:13:33.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.303+0000 7fc61da00640 1 --2- 172.21.15.45:0/2326022950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6180ba5a0 0x7fc6181f87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:33.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.304+0000 7ff052a00640 1 -- start start 2024-04-16T16:13:33.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.304+0000 7f6151800640 1 -- 172.21.15.45:0/204063217 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f614c075780 con 0x7f614c0c6fc0 2024-04-16T16:13:33.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.304+0000 7fc61d000640 1 --2- 172.21.15.45:0/2326022950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6180cfdf0 0x7fc6180b81e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc60400c4d0 tx=0x7fc60400c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.304+0000 7f4197e00640 1 -- 172.21.15.45:0/3769072953 reap_dead start 2024-04-16T16:13:33.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.305+0000 7f5bf1200640 1 Processor -- start 2024-04-16T16:13:33.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.305+0000 7f9075400640 1 --2- 172.21.15.45:0/124492984 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f905406c940 0x7f905406ee00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9060035000 tx=0x7f9060030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.305+0000 7f5bf1200640 1 -- start start 2024-04-16T16:13:33.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.306+0000 7f4197e00640 1 --2- 172.21.15.45:0/3769072953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417c0051c0 0x7f417c00b9c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f41980cd550 tx=0x7f417c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dbe22a664b197ca server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.306+0000 7ff052a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff030005230 0x7ff04c15a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.306+0000 7ff052a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff04c15a840 0x7ff04c15dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.306+0000 7ff04be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff030005230 0x7ff04c15a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.306+0000 7f5bf1200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5be40398d0 0x7f5be40d55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.306+0000 7ff052a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff04c15d0d0 0x7ff04c1f8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.306+0000 7f5bf1200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5be4052b40 0x7f5be40e2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.306+0000 7f5bf1200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5be4054350 0x7f5be40de500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.306+0000 7f5bf1200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5be404f640 con 0x7f5be4052b40 2024-04-16T16:13:33.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.306+0000 7f5bf1200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5be404f360 con 0x7f5be40398d0 2024-04-16T16:13:33.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.306+0000 7f5bf1200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5be4050270 con 0x7f5be4054350 2024-04-16T16:13:33.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.307+0000 7ff052a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c167300 con 0x7ff030005230 2024-04-16T16:13:33.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.307+0000 7ff052a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c167020 con 0x7ff04c15a840 2024-04-16T16:13:33.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.307+0000 7ff052a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff04c0b0270 con 0x7ff04c15d0d0 2024-04-16T16:13:33.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.307+0000 7f6143e00640 1 -- 172.21.15.45:0/204063217 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f614401b020 con 0x7f614c0c6fc0 2024-04-16T16:13:33.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.307+0000 7ff04be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff030005230 0x7ff04c15a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43200/0 (socket says 172.21.15.45:43200) 2024-04-16T16:13:33.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.307+0000 7ff04be00640 1 -- 172.21.15.45:0/2418689899 learned_addr learned my addr 172.21.15.45:0/2418689899 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:33.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.307+0000 7ff04b400640 1 --2- 172.21.15.45:0/2418689899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff04c15a840 0x7ff04c15dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.307+0000 7f5beaa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5be4052b40 0x7f5be40e2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.307+0000 7f5bebe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5be4054350 0x7f5be40de500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.307+0000 7f5beaa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5be4052b40 0x7f5be40e2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43206/0 (socket says 172.21.15.45:43206) 2024-04-16T16:13:33.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.307+0000 7f5beaa00640 1 -- 172.21.15.45:0/3389497682 learned_addr learned my addr 172.21.15.45:0/3389497682 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:33.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.307+0000 7ff050a00640 1 --2- 172.21.15.45:0/2418689899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff04c15d0d0 0x7ff04c1f8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.308+0000 7ff04be00640 1 -- 172.21.15.45:0/2418689899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff04c15d0d0 msgr2=0x7ff04c1f8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.308+0000 7ff04be00640 1 --2- 172.21.15.45:0/2418689899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff04c15d0d0 0x7ff04c1f8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.308+0000 7ff04be00640 1 -- 172.21.15.45:0/2418689899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff04c15a840 msgr2=0x7ff04c15dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.308+0000 7ff04be00640 1 --2- 172.21.15.45:0/2418689899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff04c15a840 0x7ff04c15dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.308+0000 7ff04be00640 1 -- 172.21.15.45:0/2418689899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04c15e530 con 0x7ff030005230 2024-04-16T16:13:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.307+0000 7f5beb400640 1 --2- 172.21.15.45:0/3389497682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5be40398d0 0x7f5be40d55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.308+0000 7f5beaa00640 1 -- 172.21.15.45:0/3389497682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5be4054350 msgr2=0x7f5be40de500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.308+0000 7f5beaa00640 1 --2- 172.21.15.45:0/3389497682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5be4054350 0x7f5be40de500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.308+0000 7f5beaa00640 1 -- 172.21.15.45:0/3389497682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5be40398d0 msgr2=0x7f5be40d55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.308+0000 7f5beaa00640 1 --2- 172.21.15.45:0/3389497682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5be40398d0 0x7f5be40d55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.308+0000 7f5beaa00640 1 -- 172.21.15.45:0/3389497682 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be40e3140 con 0x7f5be4052b40 2024-04-16T16:13:33.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.308+0000 7f6143e00640 1 --2- 172.21.15.45:0/204063217 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f612c06c9c0 0x7f612c06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.308+0000 7f6143e00640 1 -- 172.21.15.45:0/204063217 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f614408c2f0 con 0x7f614c0c6fc0 2024-04-16T16:13:33.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.308+0000 7f5beaa00640 1 --2- 172.21.15.45:0/3389497682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5be4052b40 0x7f5be40e2680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80090b0 tx=0x7f5bd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.308+0000 7ff04be00640 1 --2- 172.21.15.45:0/2418689899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff030005230 0x7ff04c15a150 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff044007fa0 tx=0x7ff0440105f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.309+0000 7f614be00640 1 --2- 172.21.15.45:0/204063217 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f612c06c9c0 0x7f612c06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.309+0000 7f5192000640 1 -- 172.21.15.45:0/3400788114 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c016020 con 0x7f519c06c040 2024-04-16T16:13:33.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.310+0000 7ff043e00640 1 -- 172.21.15.45:0/2418689899 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff044003990 con 0x7ff030005230 2024-04-16T16:13:33.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.310+0000 7f4196000640 1 -- 172.21.15.45:0/3769072953 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4188009690 con 0x7f417c0051c0 2024-04-16T16:13:33.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.310+0000 7f4196000640 1 -- 172.21.15.45:0/3769072953 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4190031030 con 0x7f417c0051c0 2024-04-16T16:13:33.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.310+0000 7f4196000640 1 -- 172.21.15.45:0/3769072953 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190005640 con 0x7f417c0051c0 2024-04-16T16:13:33.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.310+0000 7ff052a00640 1 -- 172.21.15.45:0/2418689899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff04c15c310 con 0x7ff030005230 2024-04-16T16:13:33.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.310+0000 7f5192000640 1 -- 172.21.15.45:0/3400788114 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f518c006d50 con 0x7f519c06c040 2024-04-16T16:13:33.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.310+0000 7f5192000640 1 -- 172.21.15.45:0/3400788114 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c003b70 con 0x7f519c06c040 2024-04-16T16:13:33.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.310+0000 7f51a0a00640 1 -- 172.21.15.45:0/3400788114 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f519c151d00 con 0x7f519c06c040 2024-04-16T16:13:33.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.310+0000 7ff052a00640 1 -- 172.21.15.45:0/2418689899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff04c15c8d0 con 0x7ff030005230 2024-04-16T16:13:33.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.310+0000 7f5bdf400640 1 -- 172.21.15.45:0/3389497682 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8035050 con 0x7f5be4052b40 2024-04-16T16:13:33.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.310+0000 7f5bdf400640 1 -- 172.21.15.45:0/3389497682 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bd8003c50 con 0x7f5be4052b40 2024-04-16T16:13:33.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.310+0000 7f906e000640 1 -- 172.21.15.45:0/124492984 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9068059720 con 0x7f907806f410 2024-04-16T16:13:33.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.310+0000 7f5bdf400640 1 -- 172.21.15.45:0/3389497682 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8045480 con 0x7f5be4052b40 2024-04-16T16:13:33.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.310+0000 7f51a0a00640 1 -- 172.21.15.45:0/3400788114 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f519c1522c0 con 0x7f519c06c040 2024-04-16T16:13:33.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.311+0000 7f5bf1200640 1 -- 172.21.15.45:0/3389497682 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5be4048790 con 0x7f5be4052b40 2024-04-16T16:13:33.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.311+0000 7f5bf1200640 1 -- 172.21.15.45:0/3389497682 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5be4048d50 con 0x7f5be4052b40 2024-04-16T16:13:33.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.311+0000 7ff043e00640 1 -- 172.21.15.45:0/2418689899 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff044003b30 con 0x7ff030005230 2024-04-16T16:13:33.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.311+0000 7ff043e00640 1 -- 172.21.15.45:0/2418689899 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff044003d20 con 0x7ff030005230 2024-04-16T16:13:33.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.311+0000 7f614be00640 1 --2- 172.21.15.45:0/204063217 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f612c06c9c0 0x7f612c06ee80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f613c006fd0 tx=0x7f613c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.312+0000 7f51a0a00640 1 -- 172.21.15.45:0/3400788114 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f519c075780 con 0x7f519c06c040 2024-04-16T16:13:33.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.312+0000 7f419e600640 1 -- 172.21.15.45:0/3769072953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417c0051c0 msgr2=0x7f417c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.312+0000 7f419e600640 1 --2- 172.21.15.45:0/3769072953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417c0051c0 0x7f417c00b9c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f41980cd550 tx=0x7f417c005830 comp rx=0 tx=0).stop 2024-04-16T16:13:33.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.312+0000 7f419e600640 1 -- 172.21.15.45:0/3769072953 shutdown_connections 2024-04-16T16:13:33.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.312+0000 7f419e600640 1 --2- 172.21.15.45:0/3769072953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417c0051c0 0x7f417c00b9c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.312+0000 7f419e600640 1 -- 172.21.15.45:0/3769072953 >> 172.21.15.45:0/3769072953 conn(0x7f419806b930 msgr2=0x7f419806bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.312+0000 7ff052a00640 1 -- 172.21.15.45:0/2418689899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff024005290 con 0x7ff030005230 2024-04-16T16:13:33.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.314+0000 7f5192000640 1 -- 172.21.15.45:0/3400788114 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f518c003d10 con 0x7f519c06c040 2024-04-16T16:13:33.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.314+0000 7f419e600640 1 -- 172.21.15.45:0/3769072953 shutdown_connections 2024-04-16T16:13:33.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.314+0000 7f419e600640 1 -- 172.21.15.45:0/3769072953 wait complete. 2024-04-16T16:13:33.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.314+0000 7f5bdf400640 1 -- 172.21.15.45:0/3389497682 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5bd8038020 con 0x7f5be4052b40 2024-04-16T16:13:33.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.314+0000 7f5bf1200640 1 -- 172.21.15.45:0/3389497682 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5be4048f50 con 0x7f5be4052b40 2024-04-16T16:13:33.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.315+0000 7f5192000640 1 --2- 172.21.15.45:0/3400788114 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f517c06c8f0 0x7f517c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.315+0000 7f5192000640 1 -- 172.21.15.45:0/3400788114 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f518c08b7d0 con 0x7f519c06c040 2024-04-16T16:13:33.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.315+0000 7ff043e00640 1 -- 172.21.15.45:0/2418689899 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff04401e030 con 0x7ff030005230 2024-04-16T16:13:33.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.316+0000 7f5bdf400640 1 --2- 172.21.15.45:0/3389497682 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5bcc06c9f0 0x7f5bcc06eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.316+0000 7f5199400640 1 --2- 172.21.15.45:0/3400788114 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f517c06c8f0 0x7f517c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.316+0000 7f5bdf400640 1 -- 172.21.15.45:0/3389497682 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5bd80ae050 con 0x7f5be4052b40 2024-04-16T16:13:33.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.316+0000 7f5beb400640 1 --2- 172.21.15.45:0/3389497682 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5bcc06c9f0 0x7f5bcc06eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.316+0000 7ff043e00640 1 --2- 172.21.15.45:0/2418689899 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff02c06c920 0x7ff02c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.317+0000 7ff043e00640 1 -- 172.21.15.45:0/2418689899 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff04408c450 con 0x7ff030005230 2024-04-16T16:13:33.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.317+0000 7ff04b400640 1 --2- 172.21.15.45:0/2418689899 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff02c06c920 0x7ff02c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.317+0000 7f5199400640 1 --2- 172.21.15.45:0/3400788114 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f517c06c8f0 0x7f517c06edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f518803a000 tx=0x7f5188032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.317+0000 7f419e600640 1 Processor -- start 2024-04-16T16:13:33.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.318+0000 7f419e600640 1 -- start start 2024-04-16T16:13:33.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.318+0000 7f5beb400640 1 --2- 172.21.15.45:0/3389497682 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5bcc06c9f0 0x7f5bcc06eeb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5be4050020 tx=0x7f5be0009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.318+0000 7ff04b400640 1 --2- 172.21.15.45:0/2418689899 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff02c06c920 0x7ff02c06ede0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0b0ea0 tx=0x7ff03c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.319+0000 7f419e600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f417c0051c0 0x7f4198156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.319+0000 7f419e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4198157d80 0x7f419815dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.319+0000 7f419e600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f419815e0e0 0x7f4198161510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.319+0000 7f419e600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4198167300 con 0x7f4198157d80 2024-04-16T16:13:33.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.319+0000 7f419e600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4198167020 con 0x7f419815e0e0 2024-04-16T16:13:33.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.319+0000 7f419e600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f41980b0270 con 0x7f417c0051c0 2024-04-16T16:13:33.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.320+0000 7f4196a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4198157d80 0x7f419815dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.320+0000 7f4196a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4198157d80 0x7f419815dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43216/0 (socket says 172.21.15.45:43216) 2024-04-16T16:13:33.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.320+0000 7f4196a00640 1 -- 172.21.15.45:0/3734287390 learned_addr learned my addr 172.21.15.45:0/3734287390 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:33.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.320+0000 7fc615600640 1 -- 172.21.15.45:0/2326022950 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc604017070 con 0x7fc6180cfdf0 2024-04-16T16:13:33.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.320+0000 7fc61ee00640 1 -- 172.21.15.45:0/2326022950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc618072610 con 0x7fc6180cfdf0 2024-04-16T16:13:33.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.320+0000 7f4197400640 1 --2- 172.21.15.45:0/3734287390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f417c0051c0 0x7f4198156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.320+0000 7fc61ee00640 1 -- 172.21.15.45:0/2326022950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc618072bd0 con 0x7fc6180cfdf0 2024-04-16T16:13:33.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.320+0000 7fc615600640 1 -- 172.21.15.45:0/2326022950 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc604005ce0 con 0x7fc6180cfdf0 2024-04-16T16:13:33.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.320+0000 7f4196a00640 1 -- 172.21.15.45:0/3734287390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f417c0051c0 msgr2=0x7f4198156e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.320+0000 7f4196a00640 1 --2- 172.21.15.45:0/3734287390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f417c0051c0 0x7f4198156e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.320+0000 7f4196a00640 1 -- 172.21.15.45:0/3734287390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f419815e0e0 msgr2=0x7f4198161510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.320+0000 7f4196a00640 1 --2- 172.21.15.45:0/3734287390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f419815e0e0 0x7f4198161510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.320+0000 7f4196a00640 1 -- 172.21.15.45:0/3734287390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419815e610 con 0x7f4198157d80 2024-04-16T16:13:33.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.321+0000 7f4197400640 1 --2- 172.21.15.45:0/3734287390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f417c0051c0 0x7f4198156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:13:33.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.321+0000 7f4196a00640 1 --2- 172.21.15.45:0/3734287390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4198157d80 0x7f419815dba0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f418400f5f0 tx=0x7f418400fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.320+0000 7fc615600640 1 -- 172.21.15.45:0/2326022950 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc604007270 con 0x7fc6180cfdf0 2024-04-16T16:13:33.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.321+0000 7f797d600640 1 -- 172.21.15.45:0/2371806977 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7974040070 con 0x7f79780e7710 2024-04-16T16:13:33.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.321+0000 7fc61ee00640 1 -- 172.21.15.45:0/2326022950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5f4005290 con 0x7fc6180cfdf0 2024-04-16T16:13:33.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.323+0000 7f5192000640 1 -- 172.21.15.45:0/3400788114 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f518c020020 con 0x7f519c06c040 2024-04-16T16:13:33.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.324+0000 7fc615600640 1 -- 172.21.15.45:0/2326022950 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc60401a070 con 0x7fc6180cfdf0 2024-04-16T16:13:33.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.326+0000 7fc615600640 1 --2- 172.21.15.45:0/2326022950 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc5f806c8f0 0x7fc5f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.326+0000 7fc615600640 1 -- 172.21.15.45:0/2326022950 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc60408b2e0 con 0x7fc6180cfdf0 2024-04-16T16:13:33.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.326+0000 7f6143e00640 1 -- 172.21.15.45:0/204063217 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f614405a700 con 0x7f614c0c6fc0 2024-04-16T16:13:33.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.326+0000 7fc61da00640 1 --2- 172.21.15.45:0/2326022950 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc5f806c8f0 0x7fc5f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.328+0000 7fc61da00640 1 --2- 172.21.15.45:0/2326022950 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc5f806c8f0 0x7fc5f806edb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc60c036000 tx=0x7fc60c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.328+0000 7f418f400640 1 -- 172.21.15.45:0/3734287390 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41840052e0 con 0x7f4198157d80 2024-04-16T16:13:33.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.330+0000 7f419e600640 1 -- 172.21.15.45:0/3734287390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f419815e8f0 con 0x7f4198157d80 2024-04-16T16:13:33.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.330+0000 7f419e600640 1 -- 172.21.15.45:0/3734287390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41981573d0 con 0x7f4198157d80 2024-04-16T16:13:33.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.330+0000 7f5bdf400640 1 -- 172.21.15.45:0/3389497682 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5be4048f50 con 0x7f5be4052b40 2024-04-16T16:13:33.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.331+0000 7f419e600640 1 -- 172.21.15.45:0/3734287390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4198075780 con 0x7f4198157d80 2024-04-16T16:13:33.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.334+0000 7f418f400640 1 -- 172.21.15.45:0/3734287390 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4184018070 con 0x7f4198157d80 2024-04-16T16:13:33.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.335+0000 7ff043e00640 1 -- 172.21.15.45:0/2418689899 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff044025020 con 0x7ff030005230 2024-04-16T16:13:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.334+0000 7f418f400640 1 -- 172.21.15.45:0/3734287390 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41840052e0 con 0x7f4198157d80 2024-04-16T16:13:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.337+0000 7fc615600640 1 -- 172.21.15.45:0/2326022950 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc604059130 con 0x7fc6180cfdf0 2024-04-16T16:13:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.339+0000 7f418f400640 1 -- 172.21.15.45:0/3734287390 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f41840035f0 con 0x7f4198157d80 2024-04-16T16:13:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.341+0000 7f418f400640 1 --2- 172.21.15.45:0/3734287390 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f417806caf0 0x7f417806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.341+0000 7f418f400640 1 -- 172.21.15.45:0/3734287390 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f418408c000 con 0x7f4198157d80 2024-04-16T16:13:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.341+0000 7f4197400640 1 --2- 172.21.15.45:0/3734287390 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f417806caf0 0x7f417806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.345+0000 7f418f400640 1 -- 172.21.15.45:0/3734287390 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f418405a4a0 con 0x7f4198157d80 2024-04-16T16:13:33.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.345+0000 7f4197400640 1 --2- 172.21.15.45:0/3734287390 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f417806caf0 0x7f417806efb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f41981624f0 tx=0x7f4188008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.756+0000 7f9077200640 1 -- 172.21.15.45:0/124492984 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f907806c090 con 0x7f907806f410 2024-04-16T16:13:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.757+0000 7f906e000640 1 -- 172.21.15.45:0/124492984 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9068059020 con 0x7f907806f410 2024-04-16T16:13:33.757 INFO:teuthology.orchestra.run.smithi045.stdout:55834574853 2024-04-16T16:13:33.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.761+0000 7f9077200640 1 -- 172.21.15.45:0/124492984 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f905406c940 msgr2=0x7f905406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.761+0000 7f9077200640 1 --2- 172.21.15.45:0/124492984 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f905406c940 0x7f905406ee00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9060035000 tx=0x7f9060030000 comp rx=0 tx=0).stop 2024-04-16T16:13:33.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.762+0000 7f9077200640 1 -- 172.21.15.45:0/124492984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f907806f410 msgr2=0x7f9078144b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.762+0000 7f9077200640 1 --2- 172.21.15.45:0/124492984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f907806f410 0x7f9078144b30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f906800c8c0 tx=0x7f906800cd80 comp rx=0 tx=0).stop 2024-04-16T16:13:33.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.762+0000 7f9077200640 1 -- 172.21.15.45:0/124492984 shutdown_connections 2024-04-16T16:13:33.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.762+0000 7f9077200640 1 --2- 172.21.15.45:0/124492984 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f905406c940 0x7f905406ee00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.762+0000 7f9077200640 1 --2- 172.21.15.45:0/124492984 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90780ce710 0x7f907814d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.762+0000 7f9077200640 1 --2- 172.21.15.45:0/124492984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90780b1ce0 0x7f907815d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.762+0000 7f9077200640 1 --2- 172.21.15.45:0/124492984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f907806f410 0x7f9078144b30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.762+0000 7f9077200640 1 -- 172.21.15.45:0/124492984 >> 172.21.15.45:0/124492984 conn(0x7f907806b930 msgr2=0x7f9078160340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.763+0000 7f9077200640 1 -- 172.21.15.45:0/124492984 shutdown_connections 2024-04-16T16:13:33.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.763+0000 7f9077200640 1 -- 172.21.15.45:0/124492984 wait complete. 2024-04-16T16:13:33.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.775+0000 7f51a0a00640 1 -- 172.21.15.45:0/3400788114 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f519c0b0bd0 con 0x7f519c06c040 2024-04-16T16:13:33.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.775+0000 7f5192000640 1 -- 172.21.15.45:0/3400788114 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f518c019310 con 0x7f519c06c040 2024-04-16T16:13:33.776 INFO:teuthology.orchestra.run.smithi045.stdout:55834574851 2024-04-16T16:13:33.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.777+0000 7f7986a00640 1 -- 172.21.15.45:0/2371806977 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f797804f600 con 0x7f79780e7710 2024-04-16T16:13:33.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.779+0000 7f797d600640 1 -- 172.21.15.45:0/2371806977 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f797407bdd0 con 0x7f79780e7710 2024-04-16T16:13:33.780 INFO:teuthology.orchestra.run.smithi045.stdout:55834574852 2024-04-16T16:13:33.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.779+0000 7f51a0a00640 1 -- 172.21.15.45:0/3400788114 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f517c06c8f0 msgr2=0x7f517c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.779+0000 7f51a0a00640 1 --2- 172.21.15.45:0/3400788114 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f517c06c8f0 0x7f517c06edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f518803a000 tx=0x7f5188032000 comp rx=0 tx=0).stop 2024-04-16T16:13:33.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.779+0000 7f51a0a00640 1 -- 172.21.15.45:0/3400788114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c06c040 msgr2=0x7f519c145080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.779+0000 7f51a0a00640 1 --2- 172.21.15.45:0/3400788114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c06c040 0x7f519c145080 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f518c00c760 tx=0x7f518c00cc20 comp rx=0 tx=0).stop 2024-04-16T16:13:33.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.780+0000 7f51a0a00640 1 -- 172.21.15.45:0/3400788114 shutdown_connections 2024-04-16T16:13:33.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.780+0000 7f51a0a00640 1 --2- 172.21.15.45:0/3400788114 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f517c06c8f0 0x7f517c06edb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.780+0000 7f51a0a00640 1 --2- 172.21.15.45:0/3400788114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f519c0cec90 0x7f519c158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.780+0000 7f51a0a00640 1 --2- 172.21.15.45:0/3400788114 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f519c06f700 0x7f519c158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.780+0000 7f51a0a00640 1 --2- 172.21.15.45:0/3400788114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c06c040 0x7f519c145080 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.780+0000 7f51a0a00640 1 -- 172.21.15.45:0/3400788114 >> 172.21.15.45:0/3400788114 conn(0x7f519c067e00 msgr2=0x7f519c0cfa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.780+0000 7f51a0a00640 1 -- 172.21.15.45:0/3400788114 shutdown_connections 2024-04-16T16:13:33.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.780+0000 7f51a0a00640 1 -- 172.21.15.45:0/3400788114 wait complete. 2024-04-16T16:13:33.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.780+0000 7f6151800640 1 -- 172.21.15.45:0/204063217 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f614c15ee10 con 0x7f614c0c6fc0 2024-04-16T16:13:33.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.781+0000 7f6143e00640 1 -- 172.21.15.45:0/204063217 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f614405a000 con 0x7f614c0c6fc0 2024-04-16T16:13:33.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.782+0000 7f7986a00640 1 -- 172.21.15.45:0/2371806977 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f796006c8f0 msgr2=0x7f796006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.782+0000 7f7986a00640 1 --2- 172.21.15.45:0/2371806977 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f796006c8f0 0x7f796006edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f796c000f60 tx=0x7f796c009040 comp rx=0 tx=0).stop 2024-04-16T16:13:33.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.782+0000 7f7986a00640 1 -- 172.21.15.45:0/2371806977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79780e7710 msgr2=0x7f79780ddd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.782+0000 7f7986a00640 1 --2- 172.21.15.45:0/2371806977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79780e7710 0x7f79780ddd50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7974009890 tx=0x7f79740046c0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.782+0000 7f7986a00640 1 -- 172.21.15.45:0/2371806977 shutdown_connections 2024-04-16T16:13:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.783+0000 7f7986a00640 1 --2- 172.21.15.45:0/2371806977 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f796006c8f0 0x7f796006edb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.783+0000 7f7986a00640 1 --2- 172.21.15.45:0/2371806977 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7978052150 0x7f797817d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.783+0000 7f7986a00640 1 --2- 172.21.15.45:0/2371806977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79780e7710 0x7f79780ddd50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.783+0000 7f7986a00640 1 --2- 172.21.15.45:0/2371806977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7978056ab0 0x7f79780e7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.783+0000 7f7986a00640 1 -- 172.21.15.45:0/2371806977 >> 172.21.15.45:0/2371806977 conn(0x7f7978058ee0 msgr2=0x7f79780393d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.783+0000 7f7986a00640 1 -- 172.21.15.45:0/2371806977 shutdown_connections 2024-04-16T16:13:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.783+0000 7f7986a00640 1 -- 172.21.15.45:0/2371806977 wait complete. 2024-04-16T16:13:33.785 INFO:teuthology.orchestra.run.smithi045.stdout:55834574852 2024-04-16T16:13:33.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.785+0000 7f6151800640 1 -- 172.21.15.45:0/204063217 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f612c06c9c0 msgr2=0x7f612c06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.785+0000 7f6151800640 1 --2- 172.21.15.45:0/204063217 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f612c06c9c0 0x7f612c06ee80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f613c006fd0 tx=0x7f613c008040 comp rx=0 tx=0).stop 2024-04-16T16:13:33.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.785+0000 7f6151800640 1 -- 172.21.15.45:0/204063217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614c0c6fc0 msgr2=0x7f614c0c9dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.785+0000 7f6151800640 1 --2- 172.21.15.45:0/204063217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614c0c6fc0 0x7f614c0c9dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6144007d90 tx=0x7f614400e4b0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.785+0000 7f6151800640 1 -- 172.21.15.45:0/204063217 shutdown_connections 2024-04-16T16:13:33.786 INFO:teuthology.orchestra.run.smithi045.stdout:55834574851 2024-04-16T16:13:33.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.785+0000 7f6151800640 1 --2- 172.21.15.45:0/204063217 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f612c06c9c0 0x7f612c06ee80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.785+0000 7f6151800640 1 --2- 172.21.15.45:0/204063217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614c0c6fc0 0x7f614c0c9dd0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.785+0000 7f6151800640 1 --2- 172.21.15.45:0/204063217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f614c0c6320 0x7f614c1490a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.784+0000 7f5bf1200640 1 -- 172.21.15.45:0/3389497682 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f5be4005390 con 0x7f5be4052b40 2024-04-16T16:13:33.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.785+0000 7f5bdf400640 1 -- 172.21.15.45:0/3389497682 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5bd807be10 con 0x7f5be4052b40 2024-04-16T16:13:33.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.785+0000 7f6151800640 1 --2- 172.21.15.45:0/204063217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f614c06f730 0x7f614c0cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.785+0000 7f6151800640 1 -- 172.21.15.45:0/204063217 >> 172.21.15.45:0/204063217 conn(0x7f614c06b930 msgr2=0x7f614c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.786+0000 7f6151800640 1 -- 172.21.15.45:0/204063217 shutdown_connections 2024-04-16T16:13:33.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.786+0000 7f6151800640 1 -- 172.21.15.45:0/204063217 wait complete. 2024-04-16T16:13:33.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.789+0000 7ff052a00640 1 -- 172.21.15.45:0/2418689899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7ff024005580 con 0x7ff030005230 2024-04-16T16:13:33.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.789+0000 7f5bf1200640 1 -- 172.21.15.45:0/3389497682 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5bcc06c9f0 msgr2=0x7f5bcc06eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.789+0000 7f5bf1200640 1 --2- 172.21.15.45:0/3389497682 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5bcc06c9f0 0x7f5bcc06eeb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5be4050020 tx=0x7f5be0009060 comp rx=0 tx=0).stop 2024-04-16T16:13:33.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.789+0000 7f5bf1200640 1 -- 172.21.15.45:0/3389497682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5be4052b40 msgr2=0x7f5be40e2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.789+0000 7f5bf1200640 1 --2- 172.21.15.45:0/3389497682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5be4052b40 0x7f5be40e2680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80090b0 tx=0x7f5bd80046c0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.789+0000 7f5bf1200640 1 -- 172.21.15.45:0/3389497682 shutdown_connections 2024-04-16T16:13:33.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.789+0000 7f5bf1200640 1 --2- 172.21.15.45:0/3389497682 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5bcc06c9f0 0x7f5bcc06eeb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.789+0000 7f5bf1200640 1 --2- 172.21.15.45:0/3389497682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5be4054350 0x7f5be40de500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.789+0000 7f5bf1200640 1 --2- 172.21.15.45:0/3389497682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5be4052b40 0x7f5be40e2680 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.789+0000 7f5bf1200640 1 --2- 172.21.15.45:0/3389497682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5be40398d0 0x7f5be40d55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.791 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.4 2024-04-16T16:13:33.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.790+0000 7fc61ee00640 1 -- 172.21.15.45:0/2326022950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc5f4005850 con 0x7fc6180cfdf0 2024-04-16T16:13:33.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.789+0000 7f5bf1200640 1 -- 172.21.15.45:0/3389497682 >> 172.21.15.45:0/3389497682 conn(0x7f5be4009fb0 msgr2=0x7f5be4038d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.789+0000 7f5bf1200640 1 -- 172.21.15.45:0/3389497682 shutdown_connections 2024-04-16T16:13:33.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.789+0000 7f5bf1200640 1 -- 172.21.15.45:0/3389497682 wait complete. 2024-04-16T16:13:33.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.790+0000 7ff043e00640 1 -- 172.21.15.45:0/2418689899 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff04405a220 con 0x7ff030005230 2024-04-16T16:13:33.792 INFO:teuthology.orchestra.run.smithi045.stdout:55834574852 2024-04-16T16:13:33.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.791+0000 7fc615600640 1 -- 172.21.15.45:0/2326022950 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc60405d430 con 0x7fc6180cfdf0 2024-04-16T16:13:33.793 INFO:teuthology.orchestra.run.smithi045.stdout:55834574851 2024-04-16T16:13:33.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.794+0000 7ff052a00640 1 -- 172.21.15.45:0/2418689899 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff02c06c920 msgr2=0x7ff02c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.794+0000 7ff052a00640 1 --2- 172.21.15.45:0/2418689899 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff02c06c920 0x7ff02c06ede0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0b0ea0 tx=0x7ff03c009040 comp rx=0 tx=0).stop 2024-04-16T16:13:33.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.794+0000 7ff052a00640 1 -- 172.21.15.45:0/2418689899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff030005230 msgr2=0x7ff04c15a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.794+0000 7ff052a00640 1 --2- 172.21.15.45:0/2418689899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff030005230 0x7ff04c15a150 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff044007fa0 tx=0x7ff0440105f0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.795+0000 7ff052a00640 1 -- 172.21.15.45:0/2418689899 shutdown_connections 2024-04-16T16:13:33.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.795+0000 7ff052a00640 1 --2- 172.21.15.45:0/2418689899 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff02c06c920 0x7ff02c06ede0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.795+0000 7ff052a00640 1 --2- 172.21.15.45:0/2418689899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff04c15d0d0 0x7ff04c1f8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.795+0000 7ff052a00640 1 --2- 172.21.15.45:0/2418689899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff04c15a840 0x7ff04c15dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.795+0000 7ff052a00640 1 --2- 172.21.15.45:0/2418689899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff030005230 0x7ff04c15a150 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.795+0000 7ff052a00640 1 -- 172.21.15.45:0/2418689899 >> 172.21.15.45:0/2418689899 conn(0x7ff04c06b930 msgr2=0x7ff04c0cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.795+0000 7fc61ee00640 1 -- 172.21.15.45:0/2326022950 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc5f806c8f0 msgr2=0x7fc5f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.795+0000 7fc61ee00640 1 --2- 172.21.15.45:0/2326022950 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc5f806c8f0 0x7fc5f806edb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc60c036000 tx=0x7fc60c008030 comp rx=0 tx=0).stop 2024-04-16T16:13:33.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.795+0000 7fc61ee00640 1 -- 172.21.15.45:0/2326022950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6180cfdf0 msgr2=0x7fc6180b81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.795+0000 7fc61ee00640 1 --2- 172.21.15.45:0/2326022950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6180cfdf0 0x7fc6180b81e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc60400c4d0 tx=0x7fc60400c990 comp rx=0 tx=0).stop 2024-04-16T16:13:33.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.795+0000 7ff052a00640 1 -- 172.21.15.45:0/2418689899 shutdown_connections 2024-04-16T16:13:33.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.795+0000 7ff052a00640 1 -- 172.21.15.45:0/2418689899 wait complete. 2024-04-16T16:13:33.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.796+0000 7fc61ee00640 1 -- 172.21.15.45:0/2326022950 shutdown_connections 2024-04-16T16:13:33.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.796+0000 7fc61ee00640 1 --2- 172.21.15.45:0/2326022950 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc5f806c8f0 0x7fc5f806edb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.796+0000 7f419e600640 1 -- 172.21.15.45:0/3734287390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f419806f4b0 con 0x7f4198157d80 2024-04-16T16:13:33.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.796+0000 7fc61ee00640 1 --2- 172.21.15.45:0/2326022950 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc61815ea70 0x7fc61806f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.796+0000 7fc61ee00640 1 --2- 172.21.15.45:0/2326022950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6180cfdf0 0x7fc6180b81e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.796+0000 7fc61ee00640 1 --2- 172.21.15.45:0/2326022950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6180ba5a0 0x7fc6181f87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.796+0000 7fc61ee00640 1 -- 172.21.15.45:0/2326022950 >> 172.21.15.45:0/2326022950 conn(0x7fc6180d1780 msgr2=0x7fc6180b6610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.796+0000 7fc61ee00640 1 -- 172.21.15.45:0/2326022950 shutdown_connections 2024-04-16T16:13:33.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.796+0000 7fc61ee00640 1 -- 172.21.15.45:0/2326022950 wait complete. 2024-04-16T16:13:33.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.796+0000 7f418f400640 1 -- 172.21.15.45:0/3734287390 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4184059da0 con 0x7f4198157d80 2024-04-16T16:13:33.799 INFO:teuthology.orchestra.run.smithi045.stdout:55834574852 2024-04-16T16:13:33.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.800+0000 7f419e600640 1 -- 172.21.15.45:0/3734287390 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f417806caf0 msgr2=0x7f417806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.800+0000 7f419e600640 1 --2- 172.21.15.45:0/3734287390 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f417806caf0 0x7f417806efb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f41981624f0 tx=0x7f4188008040 comp rx=0 tx=0).stop 2024-04-16T16:13:33.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.800+0000 7f419e600640 1 -- 172.21.15.45:0/3734287390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4198157d80 msgr2=0x7f419815dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:33.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.800+0000 7f419e600640 1 --2- 172.21.15.45:0/3734287390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4198157d80 0x7f419815dba0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f418400f5f0 tx=0x7f418400fab0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.800+0000 7f419e600640 1 -- 172.21.15.45:0/3734287390 shutdown_connections 2024-04-16T16:13:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.800+0000 7f419e600640 1 --2- 172.21.15.45:0/3734287390 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f417806caf0 0x7f417806efb0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f41981624f0 tx=0x7f4188008040 comp rx=0 tx=0).stop 2024-04-16T16:13:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.800+0000 7f419e600640 1 --2- 172.21.15.45:0/3734287390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f419815e0e0 0x7f4198161510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.800+0000 7f419e600640 1 --2- 172.21.15.45:0/3734287390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4198157d80 0x7f419815dba0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.800+0000 7f419e600640 1 --2- 172.21.15.45:0/3734287390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f417c0051c0 0x7f4198156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.800+0000 7f419e600640 1 -- 172.21.15.45:0/3734287390 >> 172.21.15.45:0/3734287390 conn(0x7f419806b930 msgr2=0x7f41980cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.800+0000 7f419e600640 1 -- 172.21.15.45:0/3734287390 shutdown_connections 2024-04-16T16:13:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:33.800+0000 7f419e600640 1 -- 172.21.15.45:0/3734287390 wait complete. 2024-04-16T16:13:33.807 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.7 2024-04-16T16:13:33.812 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2024-04-16T16:13:33.819 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.1 2024-04-16T16:13:33.819 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.2 2024-04-16T16:13:33.820 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.3 2024-04-16T16:13:33.825 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.5 2024-04-16T16:13:33.826 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.6 2024-04-16T16:13:34.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-16T16:13:34.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-16T16:13:34.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T16:13:34.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T16:13:34.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T16:13:34.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T16:13:34.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-16T16:13:34.827 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-16T16:13:35.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.097+0000 7fa52ac00640 1 Processor -- start 2024-04-16T16:13:35.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.097+0000 7fa52ac00640 1 -- start start 2024-04-16T16:13:35.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.100+0000 7fa52ac00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa51c0386b0 0x7fa51c058600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.100+0000 7fa52ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c058b40 0x7fa51c053920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.101+0000 7fa52ac00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa51c007150 con 0x7fa51c053e60 2024-04-16T16:13:35.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.101+0000 7fa52ac00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa51c007430 con 0x7fa51c0386b0 2024-04-16T16:13:35.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.101+0000 7fa52ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa51c007710 con 0x7fa51c058b40 2024-04-16T16:13:35.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.101+0000 7fa523400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c058b40 0x7fa51c053920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.101+0000 7fa529600640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa51c053e60 0x7fa51c0e5d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:33772/0 (socket says 172.21.15.45:33772) 2024-04-16T16:13:35.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.102+0000 7fa529600640 1 -- 172.21.15.45:0/1190513030 learned_addr learned my addr 172.21.15.45:0/1190513030 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:35.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.102+0000 7fa522a00640 1 -- 172.21.15.45:0/1190513030 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2938525267 0 0) 0x7fa51c007150 con 0x7fa51c053e60 2024-04-16T16:13:35.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.102+0000 7fa523e00640 1 --2- 172.21.15.45:0/1190513030 >> v2:172.21.15.103:3300/0 conn(0x7fa51c0386b0 0x7fa51c058600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.103+0000 7fa522a00640 1 -- 172.21.15.45:0/1190513030 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa50c003520 con 0x7fa51c053e60 2024-04-16T16:13:35.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.103+0000 7fa523400640 1 -- 172.21.15.45:0/1190513030 >> v2:172.21.15.103:3300/0 conn(0x7fa51c0386b0 msgr2=0x7fa51c058600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:35.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.103+0000 7fa523400640 1 --2- 172.21.15.45:0/1190513030 >> v2:172.21.15.103:3300/0 conn(0x7fa51c0386b0 0x7fa51c058600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.103+0000 7fa523400640 1 -- 172.21.15.45:0/1190513030 >> v1:172.21.15.103:6789/0 conn(0x7fa51c053e60 legacy=0x7fa51c0e5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.103+0000 7fa523400640 1 -- 172.21.15.45:0/1190513030 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa51c0079f0 con 0x7fa51c058b40 2024-04-16T16:13:35.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.106+0000 7fa523400640 1 --2- 172.21.15.45:0/1190513030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c058b40 0x7fa51c053920 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa5180097a0 tx=0x7fa51802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44cd6dd80298959c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.106+0000 7fa522a00640 1 -- 172.21.15.45:0/1190513030 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5180315a0 con 0x7fa51c058b40 2024-04-16T16:13:35.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.106+0000 7fa522a00640 1 -- 172.21.15.45:0/1190513030 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa518037070 con 0x7fa51c058b40 2024-04-16T16:13:35.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.107+0000 7fa52ac00640 1 -- 172.21.15.45:0/1190513030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c058b40 msgr2=0x7fa51c053920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.107+0000 7fa52ac00640 1 --2- 172.21.15.45:0/1190513030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c058b40 0x7fa51c053920 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa5180097a0 tx=0x7fa51802b840 comp rx=0 tx=0).stop 2024-04-16T16:13:35.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.108+0000 7fa52ac00640 1 -- 172.21.15.45:0/1190513030 shutdown_connections 2024-04-16T16:13:35.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.108+0000 7fa52ac00640 1 --2- 172.21.15.45:0/1190513030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c058b40 0x7fa51c053920 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.108+0000 7fa52ac00640 1 --2- 172.21.15.45:0/1190513030 >> v2:172.21.15.103:3300/0 conn(0x7fa51c0386b0 0x7fa51c058600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.109+0000 7fa52ac00640 1 -- 172.21.15.45:0/1190513030 >> 172.21.15.45:0/1190513030 conn(0x7fa51c0590c0 msgr2=0x7fa51c0594d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.109+0000 7fa52ac00640 1 -- 172.21.15.45:0/1190513030 shutdown_connections 2024-04-16T16:13:35.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.109+0000 7fa52ac00640 1 -- 172.21.15.45:0/1190513030 wait complete. 2024-04-16T16:13:35.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.112+0000 7fa52ac00640 1 Processor -- start 2024-04-16T16:13:35.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.112+0000 7fa52ac00640 1 -- start start 2024-04-16T16:13:35.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.115+0000 7fa52ac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa51c0386b0 0x7fa51c178df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.115+0000 7fa52ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c053e60 0x7fa51c0d7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.115+0000 7fa52ac00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c058b40 0x7fa51c03d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.116+0000 7fa523400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c053e60 0x7fa51c0d7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.116+0000 7fa523400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c053e60 0x7fa51c0d7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:53684/0 (socket says 172.21.15.45:53684) 2024-04-16T16:13:35.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.116+0000 7fa529600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c058b40 0x7fa51c03d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.117+0000 7fa529600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c058b40 0x7fa51c03d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43222/0 (socket says 172.21.15.45:43222) 2024-04-16T16:13:35.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.117+0000 7fa523400640 1 -- 172.21.15.45:0/1650595802 learned_addr learned my addr 172.21.15.45:0/1650595802 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:35.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.117+0000 7fa523e00640 1 --2- 172.21.15.45:0/1650595802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa51c0386b0 0x7fa51c178df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.117+0000 7fa52ac00640 1 -- 172.21.15.45:0/1650595802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c0e65d0 con 0x7fa51c058b40 2024-04-16T16:13:35.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.117+0000 7fa52ac00640 1 -- 172.21.15.45:0/1650595802 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c0e6450 con 0x7fa51c0386b0 2024-04-16T16:13:35.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.117+0000 7fa52ac00640 1 -- 172.21.15.45:0/1650595802 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa51c0e6750 con 0x7fa51c053e60 2024-04-16T16:13:35.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.119+0000 7fa529600640 1 -- 172.21.15.45:0/1650595802 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c053e60 msgr2=0x7fa51c0d7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.119+0000 7fa529600640 1 --2- 172.21.15.45:0/1650595802 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c053e60 0x7fa51c0d7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.119+0000 7fa529600640 1 -- 172.21.15.45:0/1650595802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa51c0386b0 msgr2=0x7fa51c178df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.119+0000 7fa529600640 1 --2- 172.21.15.45:0/1650595802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa51c0386b0 0x7fa51c178df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.119+0000 7fa529600640 1 -- 172.21.15.45:0/1650595802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa51c02fc50 con 0x7fa51c058b40 2024-04-16T16:13:35.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.121+0000 7fa529600640 1 --2- 172.21.15.45:0/1650595802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c058b40 0x7fa51c03d480 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa51000c3c0 tx=0x7fa51000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.122+0000 7fa517e00640 1 -- 172.21.15.45:0/1650595802 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa510015e30 con 0x7fa51c058b40 2024-04-16T16:13:35.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.122+0000 7fa523400640 1 --2- 172.21.15.45:0/1650595802 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c053e60 0x7fa51c0d7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:35.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.122+0000 7fa517e00640 1 -- 172.21.15.45:0/1650595802 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa510019030 con 0x7fa51c058b40 2024-04-16T16:13:35.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.123+0000 7fa52ac00640 1 -- 172.21.15.45:0/1650595802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa51c02ff30 con 0x7fa51c058b40 2024-04-16T16:13:35.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.124+0000 7fa52ac00640 1 -- 172.21.15.45:0/1650595802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa51c0304f0 con 0x7fa51c058b40 2024-04-16T16:13:35.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.127+0000 7fa52ac00640 1 -- 172.21.15.45:0/1650595802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa51c005570 con 0x7fa51c058b40 2024-04-16T16:13:35.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.127+0000 7fa517e00640 1 -- 172.21.15.45:0/1650595802 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa510005ad0 con 0x7fa51c058b40 2024-04-16T16:13:35.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.129+0000 7fa517e00640 1 -- 172.21.15.45:0/1650595802 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa510005cf0 con 0x7fa51c058b40 2024-04-16T16:13:35.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.130+0000 7fa517e00640 1 --2- 172.21.15.45:0/1650595802 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa50406c8f0 0x7fa50406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.135+0000 7fa523e00640 1 --2- 172.21.15.45:0/1650595802 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa50406c8f0 0x7fa50406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.135+0000 7fa517e00640 1 -- 172.21.15.45:0/1650595802 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa5100076e0 con 0x7fa51c058b40 2024-04-16T16:13:35.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.140+0000 7fa523e00640 1 --2- 172.21.15.45:0/1650595802 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa50406c8f0 0x7fa50406edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa524065790 tx=0x7fa52406f040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.141+0000 7fa517e00640 1 -- 172.21.15.45:0/1650595802 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa510061e60 con 0x7fa51c058b40 2024-04-16T16:13:35.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.145+0000 7f3506a00640 1 Processor -- start 2024-04-16T16:13:35.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.145+0000 7f3506a00640 1 -- start start 2024-04-16T16:13:35.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.147+0000 7f3506a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f34f80ebb50 0x7f34f8055300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.148+0000 7f3506a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34f80387e0 con 0x7f34f80eb060 2024-04-16T16:13:35.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.148+0000 7f3506a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34f8056d90 con 0x7f34f80061a0 2024-04-16T16:13:35.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.148+0000 7f3506a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f34f8057e30 con 0x7f34f80ebb50 2024-04-16T16:13:35.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.148+0000 7f34ffe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f34f80eb060 0x7f34f80eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39016/0 (socket says 172.21.15.45:39016) 2024-04-16T16:13:35.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.148+0000 7f34ffe00640 1 -- 172.21.15.45:0/541311044 learned_addr learned my addr 172.21.15.45:0/541311044 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.150+0000 7f34ff400640 1 --2- 172.21.15.45:0/541311044 >> v2:172.21.15.45:3300/0 conn(0x7f34f80ebb50 0x7f34f8055300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.150+0000 7f34fea00640 1 -- 172.21.15.45:0/541311044 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3210152659 0 0) 0x7f34f8056d90 con 0x7f34f80061a0 2024-04-16T16:13:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.150+0000 7f34fea00640 1 -- 172.21.15.45:0/541311044 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34e8003520 con 0x7f34f80061a0 2024-04-16T16:13:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.151+0000 7f34fea00640 1 -- 172.21.15.45:0/541311044 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2792839909 0 0) 0x7f34e8003520 con 0x7f34f80061a0 2024-04-16T16:13:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.151+0000 7f0456800640 1 Processor -- start 2024-04-16T16:13:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.151+0000 7f34fea00640 1 -- 172.21.15.45:0/541311044 >> v2:172.21.15.45:3300/0 conn(0x7f34f80ebb50 msgr2=0x7f34f8055300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:35.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.151+0000 7f0456800640 1 -- start start 2024-04-16T16:13:35.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.152+0000 7f34fea00640 1 --2- 172.21.15.45:0/541311044 >> v2:172.21.15.45:3300/0 conn(0x7f34f80ebb50 0x7f34f8055300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.152+0000 7f34fea00640 1 -- 172.21.15.45:0/541311044 >> v1:172.21.15.45:6789/0 conn(0x7f34f80eb060 legacy=0x7f34f80eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.152+0000 7f34fea00640 1 -- 172.21.15.45:0/541311044 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f8057070 con 0x7f34f80061a0 2024-04-16T16:13:35.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.152+0000 7f34fea00640 1 -- 172.21.15.45:0/541311044 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f3500075f00 con 0x7f34f80061a0 2024-04-16T16:13:35.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.152+0000 7f34fea00640 1 -- 172.21.15.45:0/541311044 >> v1:172.21.15.103:6789/0 conn(0x7f34f80061a0 legacy=0x7f34f8006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.152+0000 7f34fea00640 1 --2- 172.21.15.45:0/541311044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e80039b0 0x7f34e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.152+0000 7f34fea00640 1 --2- 172.21.15.45:0/541311044 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34e8004800 0x7f34e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.152+0000 7f34fea00640 1 --2- 172.21.15.45:0/541311044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34e80052e0 0x7f34e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.152+0000 7f34fea00640 1 -- 172.21.15.45:0/541311044 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f34e800c180 con 0x7f34e80039b0 2024-04-16T16:13:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.152+0000 7f34fea00640 1 -- 172.21.15.45:0/541311044 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34e800c330 con 0x7f34e80052e0 2024-04-16T16:13:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.152+0000 7f34fea00640 1 -- 172.21.15.45:0/541311044 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f34e800c530 con 0x7f34e8004800 2024-04-16T16:13:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.152+0000 7f34ff400640 1 --2- 172.21.15.45:0/541311044 >> v2:172.21.15.45:3300/0 conn(0x7f34f80ebb50 0x7f34f8055300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.153+0000 7f9aca800640 1 Processor -- start 2024-04-16T16:13:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.153+0000 7f9aca800640 1 -- start start 2024-04-16T16:13:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.153+0000 7f3504a00640 1 --2- 172.21.15.45:0/541311044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34e80052e0 0x7f34e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.153+0000 7f34ffe00640 1 --2- 172.21.15.45:0/541311044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e80039b0 0x7f34e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.154+0000 7f3504a00640 1 -- 172.21.15.45:0/541311044 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34e8004800 msgr2=0x7f34e8004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-16T16:13:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.154+0000 7f3504a00640 1 --2- 172.21.15.45:0/541311044 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34e8004800 0x7f34e8004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.154+0000 7f3504a00640 1 -- 172.21.15.45:0/541311044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e80039b0 msgr2=0x7f34e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.154+0000 7f3504a00640 1 --2- 172.21.15.45:0/541311044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e80039b0 0x7f34e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.155+0000 7f3504a00640 1 -- 172.21.15.45:0/541311044 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f8057070 con 0x7f34e80052e0 2024-04-16T16:13:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.155+0000 7f9aca800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9ac415ed30 0x7f9ac40d2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.156+0000 7f3504a00640 1 -- 172.21.15.45:0/541311044 reap_dead start 2024-04-16T16:13:35.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.155+0000 7f9aca800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac40d2fe0 0x7f9ac40cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.154+0000 7f0456800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f04500cd350 0x7f045009abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.154+0000 7f0456800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f045009b100 0x7f04501668e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.154+0000 7f0456800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04500cd830 con 0x7f045015cb50 2024-04-16T16:13:35.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.154+0000 7f0456800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f045015cf30 con 0x7f04500cd350 2024-04-16T16:13:35.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.154+0000 7f0456800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f04500ce440 con 0x7f045009b100 2024-04-16T16:13:35.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.156+0000 7f9aca800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac40d3480 con 0x7f9ac40ba710 2024-04-16T16:13:35.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.156+0000 7f9aca800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9ac415f110 con 0x7f9ac415ed30 2024-04-16T16:13:35.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.155+0000 7f0446a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f04500cd350 0x7f045009abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.155+0000 7f0446a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f04500cd350 0x7f045009abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43252/0 (socket says 172.21.15.45:43252) 2024-04-16T16:13:35.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.155+0000 7f044fe00640 1 --2- 172.21.15.45:0/558270770 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f045009b100 0x7f04501668e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.155+0000 7f0446a00640 1 -- 172.21.15.45:0/558270770 learned_addr learned my addr 172.21.15.45:0/558270770 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:35.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.156+0000 7f9aca800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac40d3760 con 0x7f9ac40d2fe0 2024-04-16T16:13:35.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.156+0000 7f9ac3e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9ac40ba710 0x7f9ac406ce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39032/0 (socket says 172.21.15.45:39032) 2024-04-16T16:13:35.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.156+0000 7f9ac3e00640 1 -- 172.21.15.45:0/2127214078 learned_addr learned my addr 172.21.15.45:0/2127214078 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:35.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.155+0000 7fb226e00640 1 Processor -- start 2024-04-16T16:13:35.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.155+0000 7fb226e00640 1 -- start start 2024-04-16T16:13:35.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.157+0000 7f9abaa00640 1 --2- 172.21.15.45:0/2127214078 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac40d2fe0 0x7f9ac40cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.157+0000 7f044fe00640 1 -- 172.21.15.45:0/558270770 >> v2:172.21.15.45:3300/0 conn(0x7f04500cd350 msgr2=0x7f045009abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:35.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.157+0000 7f9ac3400640 1 --2- 172.21.15.45:0/2127214078 >> v2:172.21.15.103:3300/0 conn(0x7f9ac415ed30 0x7f9ac40d2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.157+0000 7f044fe00640 1 --2- 172.21.15.45:0/558270770 >> v2:172.21.15.45:3300/0 conn(0x7f04500cd350 0x7f045009abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.157+0000 7f044fe00640 1 -- 172.21.15.45:0/558270770 >> v1:172.21.15.45:6789/0 conn(0x7f045015cb50 legacy=0x7f04500d0410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-16T16:13:35.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.157+0000 7f044fe00640 1 -- 172.21.15.45:0/558270770 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04500cdb10 con 0x7f045009b100 2024-04-16T16:13:35.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.158+0000 7f9ac3400640 1 -- 172.21.15.45:0/2127214078 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac40d2fe0 msgr2=0x7f9ac40cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:35.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.158+0000 7f9ac3400640 1 --2- 172.21.15.45:0/2127214078 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac40d2fe0 0x7f9ac40cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.158+0000 7f9ac3400640 1 -- 172.21.15.45:0/2127214078 >> v1:172.21.15.45:6789/0 conn(0x7f9ac40ba710 legacy=0x7f9ac406ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:35.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.159+0000 7fa9a9c00640 1 Processor -- start 2024-04-16T16:13:35.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.158+0000 7f9ac3400640 1 -- 172.21.15.45:0/2127214078 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac40d3a40 con 0x7f9ac415ed30 2024-04-16T16:13:35.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.159+0000 7fb226e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb22015edf0 0x7fb2200d1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.159+0000 7f044fe00640 1 --2- 172.21.15.45:0/558270770 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f045009b100 0x7f04501668e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f04480097a0 tx=0x7f044802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c619e97405f73339 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.159+0000 7fb226e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2200b8c20 0x7fb2200b90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.159+0000 7fb226e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb22015f1d0 con 0x7fb2200d1f70 2024-04-16T16:13:35.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.159+0000 7fb226e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb2200ba060 con 0x7fb22015edf0 2024-04-16T16:13:35.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.159+0000 7fb226e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb2200d2d50 con 0x7fb2200b8c20 2024-04-16T16:13:35.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.159+0000 7fa9a9c00640 1 -- start start 2024-04-16T16:13:35.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.159+0000 7f34ffe00640 1 --2- 172.21.15.45:0/541311044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e80039b0 0x7f34e8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:35.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.159+0000 7f38d5600640 1 Processor -- start 2024-04-16T16:13:35.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.159+0000 7f38d5600640 1 -- start start 2024-04-16T16:13:35.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.160+0000 7fb224e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2200b8c20 0x7fb2200b90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.160+0000 7fb224e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2200b8c20 0x7fb2200b90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:53744/0 (socket says 172.21.15.45:53744) 2024-04-16T16:13:35.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.160+0000 7fb224e00640 1 -- 172.21.15.45:0/1092260086 learned_addr learned my addr 172.21.15.45:0/1092260086 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:35.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.160+0000 7fb21fe00640 1 --2- 172.21.15.45:0/1092260086 >> v2:172.21.15.103:3300/0 conn(0x7fb22015edf0 0x7fb2200d1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.160+0000 7f3504a00640 1 --2- 172.21.15.45:0/541311044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34e80052e0 0x7f34e800b9c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f350006a8f0 tx=0x7f34e8005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f34f6de8fa37cc10 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.160+0000 7f9ac3400640 1 --2- 172.21.15.45:0/2127214078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ac415ed30 0x7f9ac40d2aa0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0097a0 tx=0x7f9aac02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b716d2a19a64502d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.160+0000 7fb224e00640 1 -- 172.21.15.45:0/1092260086 >> v2:172.21.15.103:3300/0 conn(0x7fb22015edf0 msgr2=0x7fb2200d1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:35.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.160+0000 7fb224e00640 1 --2- 172.21.15.45:0/1092260086 >> v2:172.21.15.103:3300/0 conn(0x7fb22015edf0 0x7fb2200d1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7fb224e00640 1 -- 172.21.15.45:0/1092260086 >> v1:172.21.15.103:6789/0 conn(0x7fb2200d1f70 legacy=0x7fb2200d31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7fb224e00640 1 -- 172.21.15.45:0/1092260086 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2200b97c0 con 0x7fb2200b8c20 2024-04-16T16:13:35.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7f38d5600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f38d015ed30 0x7f38d00d2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7f38d5600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38d00cfe90 0x7f38d00b7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7f38d5600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38d00d3480 con 0x7f38d00d2fe0 2024-04-16T16:13:35.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7f38d5600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f38d015f110 con 0x7f38d015ed30 2024-04-16T16:13:35.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7f38d5600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f38d00d3760 con 0x7f38d00cfe90 2024-04-16T16:13:35.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7fa9a9c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9a415e150 0x7fa9a40b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7fa9a9c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa9a406f700 0x7fa9a40b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7fa9a9c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9a415e7c0 con 0x7fa9a40cec90 2024-04-16T16:13:35.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7fa9a9c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa9a415f3d0 con 0x7fa9a406f700 2024-04-16T16:13:35.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7fa9a9c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa9a415eaa0 con 0x7fa9a415e150 2024-04-16T16:13:35.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7f38cea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f38d015ed30 0x7f38d00d2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7f38cea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f38d015ed30 0x7f38d00d2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43256/0 (socket says 172.21.15.45:43256) 2024-04-16T16:13:35.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7f38cea00640 1 -- 172.21.15.45:0/4212336792 learned_addr learned my addr 172.21.15.45:0/4212336792 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:35.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7fa9a2000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa9a40cec90 0x7fa9a406df00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39042/0 (socket says 172.21.15.45:39042) 2024-04-16T16:13:35.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.161+0000 7fa9a2000640 1 -- 172.21.15.45:0/2847972368 learned_addr learned my addr 172.21.15.45:0/2847972368 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:35.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.162+0000 7f38cf400640 1 --2- 172.21.15.45:0/4212336792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38d00cfe90 0x7f38d00b7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.162+0000 7f38cea00640 1 -- 172.21.15.45:0/4212336792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38d00cfe90 msgr2=0x7f38d00b7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:35.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.162+0000 7fa9a2a00640 1 --2- 172.21.15.45:0/2847972368 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9a415e150 0x7fa9a40b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.162+0000 7f38cea00640 1 --2- 172.21.15.45:0/4212336792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38d00cfe90 0x7f38d00b7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.162+0000 7f38cea00640 1 -- 172.21.15.45:0/4212336792 >> v1:172.21.15.103:6789/0 conn(0x7f38d00d2fe0 legacy=0x7f38d00cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:35.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.162+0000 7f38cea00640 1 -- 172.21.15.45:0/4212336792 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d00d3a40 con 0x7f38d015ed30 2024-04-16T16:13:35.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.162+0000 7fa9a3400640 1 --2- 172.21.15.45:0/2847972368 >> v2:172.21.15.103:3300/0 conn(0x7fa9a406f700 0x7fa9a40b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.162+0000 7fa9a1600640 1 -- 172.21.15.45:0/2847972368 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2496823199 0 0) 0x7fa9a415e7c0 con 0x7fa9a40cec90 2024-04-16T16:13:35.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.162+0000 7fa9a1600640 1 -- 172.21.15.45:0/2847972368 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa988003520 con 0x7fa9a40cec90 2024-04-16T16:13:35.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.162+0000 7fa9a2a00640 1 -- 172.21.15.45:0/2847972368 >> v2:172.21.15.103:3300/0 conn(0x7fa9a406f700 msgr2=0x7fa9a40b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:35.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.163+0000 7fa9a2a00640 1 --2- 172.21.15.45:0/2847972368 >> v2:172.21.15.103:3300/0 conn(0x7fa9a406f700 0x7fa9a40b58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.163+0000 7fa9a2a00640 1 -- 172.21.15.45:0/2847972368 >> v1:172.21.15.45:6789/0 conn(0x7fa9a40cec90 legacy=0x7fa9a406df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.163+0000 7fa9a2a00640 1 -- 172.21.15.45:0/2847972368 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a415ed80 con 0x7fa9a415e150 2024-04-16T16:13:35.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.162+0000 7fb224e00640 1 --2- 172.21.15.45:0/1092260086 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2200b8c20 0x7fb2200b90b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb210009870 tx=0x7fb21002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d800a9de472163df server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.163+0000 7f044ea00640 1 -- 172.21.15.45:0/558270770 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448031780 con 0x7f045009b100 2024-04-16T16:13:35.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.163+0000 7f044ea00640 1 -- 172.21.15.45:0/558270770 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0448037070 con 0x7f045009b100 2024-04-16T16:13:35.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.162+0000 7f38cf400640 1 --2- 172.21.15.45:0/4212336792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38d00cfe90 0x7f38d00b7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:35.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.163+0000 7fb21ea00640 1 -- 172.21.15.45:0/1092260086 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210034070 con 0x7fb2200b8c20 2024-04-16T16:13:35.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.163+0000 7fb21ea00640 1 -- 172.21.15.45:0/1092260086 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb210030840 con 0x7fb2200b8c20 2024-04-16T16:13:35.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.163+0000 7fb21ea00640 1 -- 172.21.15.45:0/1092260086 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210030c10 con 0x7fb2200b8c20 2024-04-16T16:13:35.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.163+0000 7f38cea00640 1 --2- 172.21.15.45:0/4212336792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38d015ed30 0x7f38d00d2aa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f38b8009870 tx=0x7f38b802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e048c87693a24df3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7f38ce000640 1 -- 172.21.15.45:0/4212336792 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b8035020 con 0x7f38d015ed30 2024-04-16T16:13:35.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7f38ce000640 1 -- 172.21.15.45:0/4212336792 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38b8031640 con 0x7f38d015ed30 2024-04-16T16:13:35.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7f38ce000640 1 -- 172.21.15.45:0/4212336792 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b802d040 con 0x7f38d015ed30 2024-04-16T16:13:35.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7f0456800640 1 -- 172.21.15.45:0/558270770 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f045009b100 msgr2=0x7f04501668e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7f0456800640 1 --2- 172.21.15.45:0/558270770 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f045009b100 0x7f04501668e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f04480097a0 tx=0x7f044802b820 comp rx=0 tx=0).stop 2024-04-16T16:13:35.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7f0456800640 1 -- 172.21.15.45:0/558270770 shutdown_connections 2024-04-16T16:13:35.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7f0456800640 1 --2- 172.21.15.45:0/558270770 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f045009b100 0x7f04501668e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7f0456800640 1 --2- 172.21.15.45:0/558270770 >> v2:172.21.15.45:3300/0 conn(0x7f04500cd350 0x7f045009abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7f0456800640 1 -- 172.21.15.45:0/558270770 >> 172.21.15.45:0/558270770 conn(0x7f04500b4330 msgr2=0x7f04500b4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7fb226e00640 1 -- 172.21.15.45:0/1092260086 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2200b8c20 msgr2=0x7fb2200b90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7fb226e00640 1 --2- 172.21.15.45:0/1092260086 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2200b8c20 0x7fb2200b90b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb210009870 tx=0x7fb21002b980 comp rx=0 tx=0).stop 2024-04-16T16:13:35.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7fb226e00640 1 -- 172.21.15.45:0/1092260086 shutdown_connections 2024-04-16T16:13:35.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7fb226e00640 1 --2- 172.21.15.45:0/1092260086 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2200b8c20 0x7fb2200b90b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7fb226e00640 1 --2- 172.21.15.45:0/1092260086 >> v2:172.21.15.103:3300/0 conn(0x7fb22015edf0 0x7fb2200d1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7fb226e00640 1 -- 172.21.15.45:0/1092260086 >> 172.21.15.45:0/1092260086 conn(0x7fb220073480 msgr2=0x7fb220073890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7fa9a2a00640 1 --2- 172.21.15.45:0/2847972368 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9a415e150 0x7fa9a40b53b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa99400a970 tx=0x7fa99402dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af3bbe62b9b430e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.165+0000 7fb226e00640 1 -- 172.21.15.45:0/1092260086 shutdown_connections 2024-04-16T16:13:35.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.165+0000 7f38d5600640 1 -- 172.21.15.45:0/4212336792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38d015ed30 msgr2=0x7f38d00d2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.165+0000 7fb226e00640 1 -- 172.21.15.45:0/1092260086 wait complete. 2024-04-16T16:13:35.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.165+0000 7f38d5600640 1 --2- 172.21.15.45:0/4212336792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38d015ed30 0x7f38d00d2aa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f38b8009870 tx=0x7f38b802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.165+0000 7f38d5600640 1 -- 172.21.15.45:0/4212336792 shutdown_connections 2024-04-16T16:13:35.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.165+0000 7f38d5600640 1 --2- 172.21.15.45:0/4212336792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38d00cfe90 0x7f38d00b7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.165+0000 7f38d5600640 1 --2- 172.21.15.45:0/4212336792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38d015ed30 0x7f38d00d2aa0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.165+0000 7f38d5600640 1 -- 172.21.15.45:0/4212336792 >> 172.21.15.45:0/4212336792 conn(0x7f38d00d1610 msgr2=0x7f38d00d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7f0456800640 1 -- 172.21.15.45:0/558270770 shutdown_connections 2024-04-16T16:13:35.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.164+0000 7f0456800640 1 -- 172.21.15.45:0/558270770 wait complete. 2024-04-16T16:13:35.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.165+0000 7f38d5600640 1 -- 172.21.15.45:0/4212336792 shutdown_connections 2024-04-16T16:13:35.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.165+0000 7fa9a1600640 1 -- 172.21.15.45:0/2847972368 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99403b070 con 0x7fa9a415e150 2024-04-16T16:13:35.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.165+0000 7fa9a1600640 1 -- 172.21.15.45:0/2847972368 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa99402eb90 con 0x7fa9a415e150 2024-04-16T16:13:35.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.165+0000 7fa9a1600640 1 -- 172.21.15.45:0/2847972368 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99400b040 con 0x7fa9a415e150 2024-04-16T16:13:35.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.166+0000 7f34fea00640 1 -- 172.21.15.45:0/541311044 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35000b5070 con 0x7f34e80052e0 2024-04-16T16:13:35.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.166+0000 7f34fea00640 1 -- 172.21.15.45:0/541311044 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35000a8760 con 0x7f34e80052e0 2024-04-16T16:13:35.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.166+0000 7f34fea00640 1 -- 172.21.15.45:0/541311044 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350006dcb0 con 0x7f34e80052e0 2024-04-16T16:13:35.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.165+0000 7f38d5600640 1 -- 172.21.15.45:0/4212336792 wait complete. 2024-04-16T16:13:35.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.167+0000 7f0456800640 1 Processor -- start 2024-04-16T16:13:35.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.167+0000 7f38d5600640 1 Processor -- start 2024-04-16T16:13:35.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.167+0000 7f9ac2a00640 1 -- 172.21.15.45:0/2127214078 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac031780 con 0x7f9ac415ed30 2024-04-16T16:13:35.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.167+0000 7f0456800640 1 -- start start 2024-04-16T16:13:35.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.167+0000 7f9ac2a00640 1 -- 172.21.15.45:0/2127214078 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aac037070 con 0x7f9ac415ed30 2024-04-16T16:13:35.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.168+0000 7f3506a00640 1 -- 172.21.15.45:0/541311044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34e80052e0 msgr2=0x7f34e800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.168+0000 7f3506a00640 1 --2- 172.21.15.45:0/541311044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34e80052e0 0x7f34e800b9c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f350006a8f0 tx=0x7f34e8005830 comp rx=0 tx=0).stop 2024-04-16T16:13:35.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.168+0000 7f9ac2a00640 1 -- 172.21.15.45:0/2127214078 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac031b40 con 0x7f9ac415ed30 2024-04-16T16:13:35.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.168+0000 7fb226e00640 1 Processor -- start 2024-04-16T16:13:35.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.167+0000 7fa9a9c00640 1 -- 172.21.15.45:0/2847972368 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9a415e150 msgr2=0x7fa9a40b53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.167+0000 7fa9a9c00640 1 --2- 172.21.15.45:0/2847972368 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9a415e150 0x7fa9a40b53b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa99400a970 tx=0x7fa99402dde0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.168+0000 7fa9a9c00640 1 -- 172.21.15.45:0/2847972368 shutdown_connections 2024-04-16T16:13:35.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.168+0000 7fa9a9c00640 1 --2- 172.21.15.45:0/2847972368 >> v2:172.21.15.103:3300/0 conn(0x7fa9a406f700 0x7fa9a40b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.168+0000 7fa9a9c00640 1 --2- 172.21.15.45:0/2847972368 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9a415e150 0x7fa9a40b53b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.168+0000 7fa9a9c00640 1 -- 172.21.15.45:0/2847972368 >> 172.21.15.45:0/2847972368 conn(0x7fa9a4067e00 msgr2=0x7fa9a406b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.167+0000 7f38d5600640 1 -- start start 2024-04-16T16:13:35.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.168+0000 7f3506a00640 1 -- 172.21.15.45:0/541311044 shutdown_connections 2024-04-16T16:13:35.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.168+0000 7f3506a00640 1 --2- 172.21.15.45:0/541311044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34e80052e0 0x7f34e800b9c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.168+0000 7f3506a00640 1 -- 172.21.15.45:0/541311044 >> 172.21.15.45:0/541311044 conn(0x7f34f80559e0 msgr2=0x7f34f8055df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.168+0000 7f9aca800640 1 -- 172.21.15.45:0/2127214078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ac415ed30 msgr2=0x7f9ac40d2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.168+0000 7f9aca800640 1 --2- 172.21.15.45:0/2127214078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ac415ed30 0x7f9ac40d2aa0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0097a0 tx=0x7f9aac02b820 comp rx=0 tx=0).stop 2024-04-16T16:13:35.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.169+0000 7f3506a00640 1 -- 172.21.15.45:0/541311044 shutdown_connections 2024-04-16T16:13:35.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.169+0000 7fb226e00640 1 -- start start 2024-04-16T16:13:35.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.169+0000 7f0456800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f045009b100 0x7f04500c8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.169+0000 7f0456800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04500cd350 0x7f04500c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.169+0000 7f38d5600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38d00cfe90 0x7f38d0156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.169+0000 7f38d5600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38d00d2fe0 0x7f38d015d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.169+0000 7f38d5600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38d015ed30 0x7f38d0159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.169+0000 7f38d5600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f38d00d4520 con 0x7f38d00d2fe0 2024-04-16T16:13:35.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.169+0000 7f38d5600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f38d00d4000 con 0x7f38d00cfe90 2024-04-16T16:13:35.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.169+0000 7f38d5600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f38d0201a60 con 0x7f38d015ed30 2024-04-16T16:13:35.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.169+0000 7f0456800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f045015cb50 0x7f045014f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.169+0000 7f044f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f045009b100 0x7f04500c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.169+0000 7f044f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f045009b100 0x7f04500c8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:54982/0 (socket says 172.21.15.45:54982) 2024-04-16T16:13:35.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.169+0000 7f044f400640 1 -- 172.21.15.45:0/3803415431 learned_addr learned my addr 172.21.15.45:0/3803415431 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:35.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7f0456800640 1 -- 172.21.15.45:0/3803415431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f04500ce0d0 con 0x7f045015cb50 2024-04-16T16:13:35.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7f0456800640 1 -- 172.21.15.45:0/3803415431 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f04500d0b20 con 0x7f045009b100 2024-04-16T16:13:35.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7f0456800640 1 -- 172.21.15.45:0/3803415431 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0450072450 con 0x7f04500cd350 2024-04-16T16:13:35.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7fb226e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2200b8c20 0x7fb22015d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7fb226e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2200d1f70 0x7fb2201607b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7fb226e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb22015edf0 0x7fb2200ac640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7fb226e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb2200ba9b0 con 0x7fb2200d1f70 2024-04-16T16:13:35.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7fb226e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2200b9d80 con 0x7fb2200b8c20 2024-04-16T16:13:35.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7fb226e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb2200d38f0 con 0x7fb22015edf0 2024-04-16T16:13:35.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7f0446a00640 1 --2- 172.21.15.45:0/3803415431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04500cd350 0x7f04500c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7f044fe00640 1 --2- 172.21.15.45:0/3803415431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f045015cb50 0x7f045014f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7f38c6000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38d00d2fe0 0x7f38d015d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7f38cea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38d00cfe90 0x7f38d0156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7f38cf400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38d015ed30 0x7f38d0159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7f38cf400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38d015ed30 0x7f38d0159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:53764/0 (socket says 172.21.15.45:53764) 2024-04-16T16:13:35.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.170+0000 7f38cf400640 1 -- 172.21.15.45:0/3419675152 learned_addr learned my addr 172.21.15.45:0/3419675152 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:35.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7fb224e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb22015edf0 0x7fb2200ac640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7fb224e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb22015edf0 0x7fb2200ac640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:53772/0 (socket says 172.21.15.45:53772) 2024-04-16T16:13:35.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7fb224e00640 1 -- 172.21.15.45:0/2010020504 learned_addr learned my addr 172.21.15.45:0/2010020504 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:35.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7f9aca800640 1 -- 172.21.15.45:0/2127214078 shutdown_connections 2024-04-16T16:13:35.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7f9aca800640 1 --2- 172.21.15.45:0/2127214078 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac40d2fe0 0x7f9ac40cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7f9aca800640 1 --2- 172.21.15.45:0/2127214078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ac415ed30 0x7f9ac40d2aa0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7f9aca800640 1 -- 172.21.15.45:0/2127214078 >> 172.21.15.45:0/2127214078 conn(0x7f9ac40d1610 msgr2=0x7f9ac40d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.169+0000 7f3506a00640 1 -- 172.21.15.45:0/541311044 wait complete. 2024-04-16T16:13:35.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7fa9a9c00640 1 -- 172.21.15.45:0/2847972368 shutdown_connections 2024-04-16T16:13:35.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7fb21f400640 1 --2- 172.21.15.45:0/2010020504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2200d1f70 0x7fb2201607b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7fa9a9c00640 1 -- 172.21.15.45:0/2847972368 wait complete. 2024-04-16T16:13:35.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7f044f400640 1 -- 172.21.15.45:0/3803415431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04500cd350 msgr2=0x7f04500c8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7f9aca800640 1 -- 172.21.15.45:0/2127214078 shutdown_connections 2024-04-16T16:13:35.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7f044f400640 1 --2- 172.21.15.45:0/3803415431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04500cd350 0x7f04500c8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7f044f400640 1 -- 172.21.15.45:0/3803415431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f045015cb50 msgr2=0x7f045014f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7f044f400640 1 --2- 172.21.15.45:0/3803415431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f045015cb50 0x7f045014f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7f044f400640 1 -- 172.21.15.45:0/3803415431 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04500c92d0 con 0x7f045009b100 2024-04-16T16:13:35.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.172+0000 7f044fe00640 1 --2- 172.21.15.45:0/3803415431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f045015cb50 0x7f045014f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:13:35.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.172+0000 7fb224e00640 1 -- 172.21.15.45:0/2010020504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2200b8c20 msgr2=0x7fb22015d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.172+0000 7fb224e00640 1 --2- 172.21.15.45:0/2010020504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2200b8c20 0x7fb22015d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.172+0000 7fb224e00640 1 -- 172.21.15.45:0/2010020504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2200d1f70 msgr2=0x7fb2201607b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.172+0000 7fb224e00640 1 --2- 172.21.15.45:0/2010020504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2200d1f70 0x7fb2201607b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.172+0000 7f044f400640 1 --2- 172.21.15.45:0/3803415431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f045009b100 0x7f04500c8850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f044000ba50 tx=0x7f044000bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.172+0000 7f3506a00640 1 Processor -- start 2024-04-16T16:13:35.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.172+0000 7fb224e00640 1 -- 172.21.15.45:0/2010020504 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22015f710 con 0x7fb22015edf0 2024-04-16T16:13:35.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.172+0000 7f3506a00640 1 -- start start 2024-04-16T16:13:35.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.172+0000 7f38cea00640 1 -- 172.21.15.45:0/3419675152 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38d015ed30 msgr2=0x7f38d0159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.172+0000 7f38cea00640 1 --2- 172.21.15.45:0/3419675152 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38d015ed30 0x7f38d0159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.172+0000 7f38cea00640 1 -- 172.21.15.45:0/3419675152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38d00d2fe0 msgr2=0x7f38d015d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.172+0000 7f38cea00640 1 --2- 172.21.15.45:0/3419675152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38d00d2fe0 0x7f38d015d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.172+0000 7f38cea00640 1 -- 172.21.15.45:0/3419675152 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d0201d20 con 0x7f38d00cfe90 2024-04-16T16:13:35.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.173+0000 7f0447e00640 1 -- 172.21.15.45:0/3803415431 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0440019070 con 0x7f045009b100 2024-04-16T16:13:35.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.173+0000 7f38cea00640 1 --2- 172.21.15.45:0/3419675152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38d00cfe90 0x7f38d0156cb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f38b8033000 tx=0x7f38b80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.173+0000 7fb224e00640 1 --2- 172.21.15.45:0/2010020504 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb22015edf0 0x7fb2200ac640 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb210009870 tx=0x7fb210004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.171+0000 7f9aca800640 1 -- 172.21.15.45:0/2127214078 wait complete. 2024-04-16T16:13:35.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.173+0000 7f0447e00640 1 -- 172.21.15.45:0/3803415431 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f044000cc20 con 0x7f045009b100 2024-04-16T16:13:35.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.173+0000 7f0447e00640 1 -- 172.21.15.45:0/3803415431 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0440014440 con 0x7f045009b100 2024-04-16T16:13:35.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.173+0000 7fb217e00640 1 -- 172.21.15.45:0/2010020504 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210048070 con 0x7fb22015edf0 2024-04-16T16:13:35.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.173+0000 7fb217e00640 1 -- 172.21.15.45:0/2010020504 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb210030640 con 0x7fb22015edf0 2024-04-16T16:13:35.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7fa9a9c00640 1 Processor -- start 2024-04-16T16:13:35.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.173+0000 7fb226e00640 1 -- 172.21.15.45:0/2010020504 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb22015f9f0 con 0x7fb22015edf0 2024-04-16T16:13:35.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7fb217e00640 1 -- 172.21.15.45:0/2010020504 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210034040 con 0x7fb22015edf0 2024-04-16T16:13:35.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7fb226e00640 1 -- 172.21.15.45:0/2010020504 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb22015ffb0 con 0x7fb22015edf0 2024-04-16T16:13:35.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7fa9a9c00640 1 -- start start 2024-04-16T16:13:35.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7f9aca800640 1 Processor -- start 2024-04-16T16:13:35.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7f38c7400640 1 -- 172.21.15.45:0/3419675152 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b8040070 con 0x7f38d00cfe90 2024-04-16T16:13:35.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7f38c7400640 1 -- 172.21.15.45:0/3419675152 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38b8039cf0 con 0x7f38d00cfe90 2024-04-16T16:13:35.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7f38c7400640 1 -- 172.21.15.45:0/3419675152 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b8035040 con 0x7f38d00cfe90 2024-04-16T16:13:35.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7f0456800640 1 -- 172.21.15.45:0/3803415431 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f045015db60 con 0x7f045009b100 2024-04-16T16:13:35.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.175+0000 7f0456800640 1 -- 172.21.15.45:0/3803415431 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f045015e120 con 0x7f045009b100 2024-04-16T16:13:35.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7f3506a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34e80052e0 0x7f34f80526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7f3506a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34f8007720 0x7f34f81859c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7f3506a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34f804b2d0 0x7f34f8185f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7f3506a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f34f8057630 con 0x7f34f8007720 2024-04-16T16:13:35.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7f3506a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34f80579a0 con 0x7f34e80052e0 2024-04-16T16:13:35.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7f3506a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f34f80e1e10 con 0x7f34f804b2d0 2024-04-16T16:13:35.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.175+0000 7f34ffe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34e80052e0 0x7f34f80526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.175+0000 7f34ff400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34f8007720 0x7f34f81859c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.174+0000 7fe50b400640 1 Processor -- start 2024-04-16T16:13:35.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.175+0000 7fa9a9c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9a406f700 0x7fa9a4147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.175+0000 7f34ffe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34e80052e0 0x7f34f80526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:55008/0 (socket says 172.21.15.45:55008) 2024-04-16T16:13:35.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.175+0000 7f34ffe00640 1 -- 172.21.15.45:0/2783082448 learned_addr learned my addr 172.21.15.45:0/2783082448 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:35.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7fa9a9c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9a40cec90 0x7fa9a40aacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.175+0000 7f9aca800640 1 -- start start 2024-04-16T16:13:35.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7fa9a9c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9a415e150 0x7fa9a41475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7fa9a9c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a406c040 con 0x7fa9a406f700 2024-04-16T16:13:35.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7fa9a9c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a40b4e80 con 0x7fa9a415e150 2024-04-16T16:13:35.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7fa9a9c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa9a406c1c0 con 0x7fa9a40cec90 2024-04-16T16:13:35.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7fa9a2a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9a406f700 0x7fa9a4147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7fa9a2a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9a406f700 0x7fa9a4147f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43310/0 (socket says 172.21.15.45:43310) 2024-04-16T16:13:35.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7fa9a2a00640 1 -- 172.21.15.45:0/839906431 learned_addr learned my addr 172.21.15.45:0/839906431 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:35.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7f34ffe00640 1 -- 172.21.15.45:0/2783082448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34f804b2d0 msgr2=0x7f34f8185f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7f34ffe00640 1 --2- 172.21.15.45:0/2783082448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34f804b2d0 0x7f34f8185f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7f34ffe00640 1 -- 172.21.15.45:0/2783082448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34f8007720 msgr2=0x7f34f81859c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7f34ffe00640 1 --2- 172.21.15.45:0/2783082448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34f8007720 0x7f34f81859c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.175+0000 7fe50b400640 1 -- start start 2024-04-16T16:13:35.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7f0456800640 1 -- 172.21.15.45:0/3803415431 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0450075780 con 0x7f045009b100 2024-04-16T16:13:35.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7fb226e00640 1 -- 172.21.15.45:0/2010020504 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1f8005290 con 0x7fb22015edf0 2024-04-16T16:13:35.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7f34ffe00640 1 -- 172.21.15.45:0/2783082448 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f804fcc0 con 0x7f34e80052e0 2024-04-16T16:13:35.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.175+0000 7f38d5600640 1 -- 172.21.15.45:0/3419675152 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38d0202000 con 0x7f38d00cfe90 2024-04-16T16:13:35.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.177+0000 7f38d5600640 1 -- 172.21.15.45:0/3419675152 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38d02025c0 con 0x7f38d00cfe90 2024-04-16T16:13:35.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7fa9a2000640 1 --2- 172.21.15.45:0/839906431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9a40cec90 0x7fa9a40aacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.176+0000 7fa9a3400640 1 --2- 172.21.15.45:0/839906431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9a415e150 0x7fa9a41475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.177+0000 7f34ffe00640 1 --2- 172.21.15.45:0/2783082448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34e80052e0 0x7f34f80526a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f34f40018a0 tx=0x7f34f4001d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.177+0000 7fa9a2a00640 1 -- 172.21.15.45:0/839906431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9a40cec90 msgr2=0x7fa9a40aacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.177+0000 7fa9a2a00640 1 --2- 172.21.15.45:0/839906431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9a40cec90 0x7fa9a40aacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.177+0000 7fa9a2a00640 1 -- 172.21.15.45:0/839906431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9a415e150 msgr2=0x7fa9a41475f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.177+0000 7fa9a2a00640 1 --2- 172.21.15.45:0/839906431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9a415e150 0x7fa9a41475f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.177+0000 7fa9a2a00640 1 -- 172.21.15.45:0/839906431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a40ab200 con 0x7fa9a406f700 2024-04-16T16:13:35.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.177+0000 7fe50b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe5040b9330 0x7fe504169900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.177+0000 7fe50b400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe50415f1d0 con 0x7fe5040d2140 2024-04-16T16:13:35.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.177+0000 7fe50b400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5040b97c0 con 0x7fe50415edf0 2024-04-16T16:13:35.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.177+0000 7fe50b400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe5040ba060 con 0x7fe5040b9330 2024-04-16T16:13:35.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7fa9a2a00640 1 --2- 172.21.15.45:0/839906431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9a406f700 0x7fa9a4147f60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa99400aa90 tx=0x7fa9940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7f9aca800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ac40ba710 0x7f9ac41609e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7f9aca800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac40d2fe0 0x7f9ac4201450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7f9aca800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ac415ed30 0x7f9ac40cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7f9aca800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac40b8af0 con 0x7f9ac40ba710 2024-04-16T16:13:35.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7f9aca800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac40d4000 con 0x7f9ac415ed30 2024-04-16T16:13:35.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7f9aca800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac40d4520 con 0x7f9ac40d2fe0 2024-04-16T16:13:35.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7fe509400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe5040b9330 0x7fe504169900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7fe509400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe5040b9330 0x7fe504169900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43320/0 (socket says 172.21.15.45:43320) 2024-04-16T16:13:35.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7fe509400640 1 -- 172.21.15.45:0/70947591 learned_addr learned my addr 172.21.15.45:0/70947591 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:35.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7f38d5600640 1 -- 172.21.15.45:0/3419675152 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38d0075780 con 0x7f38d00cfe90 2024-04-16T16:13:35.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7f34f3e00640 1 -- 172.21.15.45:0/2783082448 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f401a070 con 0x7f34e80052e0 2024-04-16T16:13:35.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7f34f3e00640 1 -- 172.21.15.45:0/2783082448 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34f400f960 con 0x7f34e80052e0 2024-04-16T16:13:35.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7f34f3e00640 1 -- 172.21.15.45:0/2783082448 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4001440 con 0x7f34e80052e0 2024-04-16T16:13:35.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7fa99aa00640 1 -- 172.21.15.45:0/839906431 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa994044070 con 0x7fa9a406f700 2024-04-16T16:13:35.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7fa99aa00640 1 -- 172.21.15.45:0/839906431 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa99403f420 con 0x7fa9a406f700 2024-04-16T16:13:35.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7fa99aa00640 1 -- 172.21.15.45:0/839906431 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99403b040 con 0x7fa9a406f700 2024-04-16T16:13:35.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7f3506a00640 1 -- 172.21.15.45:0/2783082448 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34f804f9e0 con 0x7f34e80052e0 2024-04-16T16:13:35.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.179+0000 7fb217e00640 1 -- 172.21.15.45:0/2010020504 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb210037070 con 0x7fb22015edf0 2024-04-16T16:13:35.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7fa9a9c00640 1 -- 172.21.15.45:0/839906431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9a4148540 con 0x7fa9a406f700 2024-04-16T16:13:35.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.180+0000 7fa9a9c00640 1 -- 172.21.15.45:0/839906431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9a4148b00 con 0x7fa9a406f700 2024-04-16T16:13:35.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.180+0000 7fb217e00640 1 --2- 172.21.15.45:0/2010020504 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb20006c8f0 0x7fb20006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.179+0000 7fe509400640 1 -- 172.21.15.45:0/70947591 >> v1:172.21.15.103:6789/0 conn(0x7fe50415edf0 legacy=0x7fe5040d1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:35.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.179+0000 7fe509400640 1 -- 172.21.15.45:0/70947591 >> v1:172.21.15.45:6789/0 conn(0x7fe5040d2140 legacy=0x7fe5040b7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:35.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.179+0000 7fe509400640 1 -- 172.21.15.45:0/70947591 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5040b9aa0 con 0x7fe5040b9330 2024-04-16T16:13:35.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.178+0000 7f3506a00640 1 -- 172.21.15.45:0/2783082448 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f8187390 con 0x7f34e80052e0 2024-04-16T16:13:35.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.181+0000 7f3506a00640 1 -- 172.21.15.45:0/2783082448 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34f8187590 con 0x7f34e80052e0 2024-04-16T16:13:35.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.179+0000 7f9ac3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ac40ba710 0x7f9ac41609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.180+0000 7f9abaa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac40d2fe0 0x7f9ac4201450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.180+0000 7f9abaa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac40d2fe0 0x7f9ac4201450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:53792/0 (socket says 172.21.15.45:53792) 2024-04-16T16:13:35.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.180+0000 7f9abaa00640 1 -- 172.21.15.45:0/1884767985 learned_addr learned my addr 172.21.15.45:0/1884767985 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:35.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.181+0000 7f9ac3e00640 1 --2- 172.21.15.45:0/1884767985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ac415ed30 0x7f9ac40cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.182+0000 7f9abaa00640 1 -- 172.21.15.45:0/1884767985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ac415ed30 msgr2=0x7f9ac40cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.182+0000 7f9abaa00640 1 --2- 172.21.15.45:0/1884767985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ac415ed30 0x7f9ac40cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.181+0000 7f38c7400640 1 -- 172.21.15.45:0/3419675152 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f38b804b060 con 0x7f38d00cfe90 2024-04-16T16:13:35.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.181+0000 7fb217e00640 1 -- 172.21.15.45:0/2010020504 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb2100ad790 con 0x7fb22015edf0 2024-04-16T16:13:35.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.181+0000 7fe509400640 1 --2- 172.21.15.45:0/70947591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5040b9330 0x7fe504169900 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4009870 tx=0x7fe4f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0cd1fccd2947eed server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.182+0000 7f9abaa00640 1 -- 172.21.15.45:0/1884767985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ac40ba710 msgr2=0x7f9ac41609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.182+0000 7f9abaa00640 1 --2- 172.21.15.45:0/1884767985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ac40ba710 0x7f9ac41609e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.182+0000 7f9abaa00640 1 -- 172.21.15.45:0/1884767985 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac41604e0 con 0x7f9ac40d2fe0 2024-04-16T16:13:35.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.181+0000 7fa9a9c00640 1 -- 172.21.15.45:0/839906431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9a4075780 con 0x7fa9a406f700 2024-04-16T16:13:35.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.182+0000 7f0447e00640 1 -- 172.21.15.45:0/3803415431 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f044001c070 con 0x7f045009b100 2024-04-16T16:13:35.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.182+0000 7f9ac3400640 1 --2- 172.21.15.45:0/1884767985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ac40ba710 0x7f9ac41609e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:13:35.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.182+0000 7f9abaa00640 1 --2- 172.21.15.45:0/1884767985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac40d2fe0 0x7f9ac4201450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400e9b0 tx=0x7f9ab400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.182+0000 7f38c7400640 1 --2- 172.21.15.45:0/3419675152 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f38ac06c9c0 0x7f38ac06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.181+0000 7fb21fe00640 1 --2- 172.21.15.45:0/2010020504 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb20006c8f0 0x7fb20006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.182+0000 7fe503400640 1 -- 172.21.15.45:0/70947591 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f4035020 con 0x7fe5040b9330 2024-04-16T16:13:35.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.182+0000 7fe503400640 1 -- 172.21.15.45:0/70947591 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4f4031870 con 0x7fe5040b9330 2024-04-16T16:13:35.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.182+0000 7fe503400640 1 -- 172.21.15.45:0/70947591 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f4031c40 con 0x7fe5040b9330 2024-04-16T16:13:35.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.182+0000 7f38c7400640 1 -- 172.21.15.45:0/3419675152 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f38b80adb10 con 0x7f38d00cfe90 2024-04-16T16:13:35.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.183+0000 7f0447e00640 1 --2- 172.21.15.45:0/3803415431 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f042c06c9c0 0x7f042c06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.183+0000 7fb21fe00640 1 --2- 172.21.15.45:0/2010020504 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb20006c8f0 0x7fb20006edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb20c006a90 tx=0x7fb20c006a20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.183+0000 7f0447e00640 1 -- 172.21.15.45:0/3803415431 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f044008c040 con 0x7f045009b100 2024-04-16T16:13:35.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.183+0000 7fe50b400640 1 -- 172.21.15.45:0/70947591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5040b9330 msgr2=0x7fe504169900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.183+0000 7fe50b400640 1 --2- 172.21.15.45:0/70947591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5040b9330 0x7fe504169900 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4009870 tx=0x7fe4f402b910 comp rx=0 tx=0).stop 2024-04-16T16:13:35.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.183+0000 7f38c6000640 1 --2- 172.21.15.45:0/3419675152 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f38ac06c9c0 0x7f38ac06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.183+0000 7fe50b400640 1 -- 172.21.15.45:0/70947591 shutdown_connections 2024-04-16T16:13:35.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.183+0000 7fe50b400640 1 --2- 172.21.15.45:0/70947591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5040b9330 0x7fe504169900 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.183+0000 7fe50b400640 1 -- 172.21.15.45:0/70947591 >> 172.21.15.45:0/70947591 conn(0x7fe504073480 msgr2=0x7fe504073890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.184+0000 7f0446a00640 1 --2- 172.21.15.45:0/3803415431 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f042c06c9c0 0x7f042c06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.184+0000 7fa99aa00640 1 -- 172.21.15.45:0/839906431 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa994047070 con 0x7fa9a406f700 2024-04-16T16:13:35.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.184+0000 7fe50b400640 1 -- 172.21.15.45:0/70947591 shutdown_connections 2024-04-16T16:13:35.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.185+0000 7f38c6000640 1 --2- 172.21.15.45:0/3419675152 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f38ac06c9c0 0x7f38ac06ee80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f38d0157a70 tx=0x7f38c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.185+0000 7fa99aa00640 1 --2- 172.21.15.45:0/839906431 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa98406c940 0x7fa98406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.185+0000 7f0446a00640 1 --2- 172.21.15.45:0/3803415431 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f042c06c9c0 0x7f042c06ee80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0450153c50 tx=0x7f0438008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.185+0000 7fa99aa00640 1 -- 172.21.15.45:0/839906431 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa9940ad2f0 con 0x7fa9a406f700 2024-04-16T16:13:35.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.184+0000 7fe50b400640 1 -- 172.21.15.45:0/70947591 wait complete. 2024-04-16T16:13:35.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.186+0000 7fa9a2000640 1 --2- 172.21.15.45:0/839906431 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa98406c940 0x7fa98406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.186+0000 7fe50b400640 1 Processor -- start 2024-04-16T16:13:35.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.186+0000 7fe50b400640 1 -- start start 2024-04-16T16:13:35.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.187+0000 7fa9a2000640 1 --2- 172.21.15.45:0/839906431 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa98406c940 0x7fa98406ee00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4159440 tx=0x7fa99000b230 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.187+0000 7fe50b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5040d2140 0x7fe5040ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.187+0000 7fe50b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe50415edf0 0x7fe504157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.187+0000 7fe50b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe504156100 0x7fe504161060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.187+0000 7fe50b400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe5040ba9b0 con 0x7fe50415edf0 2024-04-16T16:13:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.187+0000 7fe50b400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe5040d2d50 con 0x7fe504156100 2024-04-16T16:13:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.187+0000 7fe50b400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe5040d41f0 con 0x7fe5040d2140 2024-04-16T16:13:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.188+0000 7fe503e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe50415edf0 0x7fe504157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.188+0000 7fe503e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe50415edf0 0x7fe504157a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43338/0 (socket says 172.21.15.45:43338) 2024-04-16T16:13:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.188+0000 7fe503e00640 1 -- 172.21.15.45:0/3866465324 learned_addr learned my addr 172.21.15.45:0/3866465324 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.188+0000 7fe508a00640 1 --2- 172.21.15.45:0/3866465324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5040d2140 0x7fe5040ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.188+0000 7fe509400640 1 --2- 172.21.15.45:0/3866465324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe504156100 0x7fe504161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.188+0000 7fe503e00640 1 -- 172.21.15.45:0/3866465324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5040d2140 msgr2=0x7fe5040ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.188+0000 7fe503e00640 1 --2- 172.21.15.45:0/3866465324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5040d2140 0x7fe5040ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.188+0000 7fe503e00640 1 -- 172.21.15.45:0/3866465324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe504156100 msgr2=0x7fe504161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.188+0000 7fe503e00640 1 --2- 172.21.15.45:0/3866465324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe504156100 0x7fe504161060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.188+0000 7fe503e00640 1 -- 172.21.15.45:0/3866465324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe504157f50 con 0x7fe50415edf0 2024-04-16T16:13:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.189+0000 7fe503e00640 1 --2- 172.21.15.45:0/3866465324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe50415edf0 0x7fe504157a10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000c780 tx=0x7fe4f000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.190+0000 7fe500c00640 1 -- 172.21.15.45:0/3866465324 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0017070 con 0x7fe50415edf0 2024-04-16T16:13:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.190+0000 7fe500c00640 1 -- 172.21.15.45:0/3866465324 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4f0005290 con 0x7fe50415edf0 2024-04-16T16:13:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.190+0000 7fe500c00640 1 -- 172.21.15.45:0/3866465324 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f000a880 con 0x7fe50415edf0 2024-04-16T16:13:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.190+0000 7fe50b400640 1 -- 172.21.15.45:0/3866465324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5041581d0 con 0x7fe50415edf0 2024-04-16T16:13:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.190+0000 7fe50b400640 1 -- 172.21.15.45:0/3866465324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe504158790 con 0x7fe50415edf0 2024-04-16T16:13:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.190+0000 7fa99aa00640 1 -- 172.21.15.45:0/839906431 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa99407b020 con 0x7fa9a406f700 2024-04-16T16:13:35.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.191+0000 7fe50b400640 1 -- 172.21.15.45:0/3866465324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe504075780 con 0x7fe50415edf0 2024-04-16T16:13:35.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.193+0000 7fe500c00640 1 -- 172.21.15.45:0/3866465324 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe4f001a070 con 0x7fe50415edf0 2024-04-16T16:13:35.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.193+0000 7fe500c00640 1 --2- 172.21.15.45:0/3866465324 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe4e406c940 0x7fe4e406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.193+0000 7fe500c00640 1 -- 172.21.15.45:0/3866465324 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe4f008af30 con 0x7fe50415edf0 2024-04-16T16:13:35.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.194+0000 7f9abbe00640 1 -- 172.21.15.45:0/1884767985 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4017070 con 0x7f9ac40d2fe0 2024-04-16T16:13:35.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.194+0000 7f9abbe00640 1 -- 172.21.15.45:0/1884767985 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ab400c9a0 con 0x7f9ac40d2fe0 2024-04-16T16:13:35.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.194+0000 7f9abbe00640 1 -- 172.21.15.45:0/1884767985 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab400cd90 con 0x7f9ac40d2fe0 2024-04-16T16:13:35.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.194+0000 7f9aca800640 1 -- 172.21.15.45:0/1884767985 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac4155740 con 0x7f9ac40d2fe0 2024-04-16T16:13:35.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.194+0000 7f9aca800640 1 -- 172.21.15.45:0/1884767985 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac4201990 con 0x7f9ac40d2fe0 2024-04-16T16:13:35.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.194+0000 7fe508a00640 1 --2- 172.21.15.45:0/3866465324 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe4e406c940 0x7fe4e406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.195+0000 7fb217e00640 1 -- 172.21.15.45:0/2010020504 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb21007bbb0 con 0x7fb22015edf0 2024-04-16T16:13:35.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.195+0000 7fe508a00640 1 --2- 172.21.15.45:0/3866465324 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe4e406c940 0x7fe4e406ee00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc009f60 tx=0x7fe4fc0094f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.195+0000 7f9aca800640 1 -- 172.21.15.45:0/1884767985 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac4075780 con 0x7f9ac40d2fe0 2024-04-16T16:13:35.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.197+0000 7f0447e00640 1 -- 172.21.15.45:0/3803415431 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0440059e90 con 0x7f045009b100 2024-04-16T16:13:35.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.197+0000 7f9abbe00640 1 -- 172.21.15.45:0/1884767985 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9ab401a070 con 0x7f9ac40d2fe0 2024-04-16T16:13:35.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.198+0000 7f9abbe00640 1 --2- 172.21.15.45:0/1884767985 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9aa006c9c0 0x7f9aa006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.198+0000 7f9abbe00640 1 -- 172.21.15.45:0/1884767985 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9ab408c320 con 0x7f9ac40d2fe0 2024-04-16T16:13:35.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.199+0000 7f9ac3400640 1 --2- 172.21.15.45:0/1884767985 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9aa006c9c0 0x7f9aa006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.200+0000 7f9ac3400640 1 --2- 172.21.15.45:0/1884767985 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9aa006c9c0 0x7f9aa006ee80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9aac036000 tx=0x7f9aac03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.203+0000 7fe500c00640 1 -- 172.21.15.45:0/3866465324 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe504157f50 con 0x7fe50415edf0 2024-04-16T16:13:35.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.211+0000 7f38c7400640 1 -- 172.21.15.45:0/3419675152 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f38b807b840 con 0x7f38d00cfe90 2024-04-16T16:13:35.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.212+0000 7f9abbe00640 1 -- 172.21.15.45:0/1884767985 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9ab405a0e0 con 0x7f9ac40d2fe0 2024-04-16T16:13:35.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.216+0000 7f34f3e00640 1 -- 172.21.15.45:0/2783082448 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f34f401d070 con 0x7f34e80052e0 2024-04-16T16:13:35.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.217+0000 7f34f3e00640 1 --2- 172.21.15.45:0/2783082448 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f34e006c9c0 0x7f34e006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.217+0000 7f34f3e00640 1 -- 172.21.15.45:0/2783082448 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f34f408bb00 con 0x7f34e80052e0 2024-04-16T16:13:35.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.219+0000 7f34ff400640 1 --2- 172.21.15.45:0/2783082448 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f34e006c9c0 0x7f34e006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.220+0000 7f34ff400640 1 --2- 172.21.15.45:0/2783082448 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f34e006c9c0 0x7f34e006ee80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f34ec006fd0 tx=0x7f34ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.225+0000 7f34f3e00640 1 -- 172.21.15.45:0/2783082448 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f34f8187590 con 0x7f34e80052e0 2024-04-16T16:13:35.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.573+0000 7fa52ac00640 1 -- 172.21.15.45:0/1650595802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fa51c04ff00 con 0x7fa51c058b40 2024-04-16T16:13:35.574 INFO:teuthology.orchestra.run.smithi045.stdout:55834574855 2024-04-16T16:13:35.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.574+0000 7fa517e00640 1 -- 172.21.15.45:0/1650595802 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa510061760 con 0x7fa51c058b40 2024-04-16T16:13:35.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.578+0000 7fa52ac00640 1 -- 172.21.15.45:0/1650595802 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa50406c8f0 msgr2=0x7fa50406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.578+0000 7fa52ac00640 1 --2- 172.21.15.45:0/1650595802 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa50406c8f0 0x7fa50406edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa524065790 tx=0x7fa52406f040 comp rx=0 tx=0).stop 2024-04-16T16:13:35.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.579+0000 7fa52ac00640 1 -- 172.21.15.45:0/1650595802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c058b40 msgr2=0x7fa51c03d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.579+0000 7fa52ac00640 1 --2- 172.21.15.45:0/1650595802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c058b40 0x7fa51c03d480 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa51000c3c0 tx=0x7fa51000c880 comp rx=0 tx=0).stop 2024-04-16T16:13:35.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.579+0000 7fa52ac00640 1 -- 172.21.15.45:0/1650595802 shutdown_connections 2024-04-16T16:13:35.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.579+0000 7fa52ac00640 1 --2- 172.21.15.45:0/1650595802 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa50406c8f0 0x7fa50406edb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.579+0000 7fa52ac00640 1 --2- 172.21.15.45:0/1650595802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c058b40 0x7fa51c03d480 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.579+0000 7fa52ac00640 1 --2- 172.21.15.45:0/1650595802 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c053e60 0x7fa51c0d7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.579+0000 7fa52ac00640 1 --2- 172.21.15.45:0/1650595802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa51c0386b0 0x7fa51c178df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.579+0000 7fa52ac00640 1 -- 172.21.15.45:0/1650595802 >> 172.21.15.45:0/1650595802 conn(0x7fa51c0590c0 msgr2=0x7fa51c0062e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.580+0000 7fa52ac00640 1 -- 172.21.15.45:0/1650595802 shutdown_connections 2024-04-16T16:13:35.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.580+0000 7fa52ac00640 1 -- 172.21.15.45:0/1650595802 wait complete. 2024-04-16T16:13:35.608 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.4 2024-04-16T16:13:35.608 DEBUG:teuthology.parallel:result is None 2024-04-16T16:13:35.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.618+0000 7fa9a9c00640 1 -- 172.21.15.45:0/839906431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa9a4072bb0 con 0x7fa9a406f700 2024-04-16T16:13:35.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.619+0000 7fa99aa00640 1 -- 172.21.15.45:0/839906431 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa99407f320 con 0x7fa9a406f700 2024-04-16T16:13:35.619 INFO:teuthology.orchestra.run.smithi045.stdout:55834574854 2024-04-16T16:13:35.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.620+0000 7f0456800640 1 -- 172.21.15.45:0/3803415431 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f04500d0230 con 0x7f045009b100 2024-04-16T16:13:35.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.621+0000 7f0447e00640 1 -- 172.21.15.45:0/3803415431 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f044005e190 con 0x7f045009b100 2024-04-16T16:13:35.621 INFO:teuthology.orchestra.run.smithi045.stdout:55834574853 2024-04-16T16:13:35.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.622+0000 7fa9a9c00640 1 -- 172.21.15.45:0/839906431 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa98406c940 msgr2=0x7fa98406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.622+0000 7fa9a9c00640 1 --2- 172.21.15.45:0/839906431 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa98406c940 0x7fa98406ee00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4159440 tx=0x7fa99000b230 comp rx=0 tx=0).stop 2024-04-16T16:13:35.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.622+0000 7fa9a9c00640 1 -- 172.21.15.45:0/839906431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9a406f700 msgr2=0x7fa9a4147f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.622+0000 7fa9a9c00640 1 --2- 172.21.15.45:0/839906431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9a406f700 0x7fa9a4147f60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa99400aa90 tx=0x7fa9940046c0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.623+0000 7fa9a9c00640 1 -- 172.21.15.45:0/839906431 shutdown_connections 2024-04-16T16:13:35.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.623+0000 7fa9a9c00640 1 --2- 172.21.15.45:0/839906431 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa98406c940 0x7fa98406ee00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.623+0000 7fa9a9c00640 1 --2- 172.21.15.45:0/839906431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9a415e150 0x7fa9a41475f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.623+0000 7fa9a9c00640 1 --2- 172.21.15.45:0/839906431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9a40cec90 0x7fa9a40aacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.623+0000 7fa9a9c00640 1 --2- 172.21.15.45:0/839906431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9a406f700 0x7fa9a4147f60 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.623+0000 7fa9a9c00640 1 -- 172.21.15.45:0/839906431 >> 172.21.15.45:0/839906431 conn(0x7fa9a4067e00 msgr2=0x7fa9a406e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.623+0000 7fa9a9c00640 1 -- 172.21.15.45:0/839906431 shutdown_connections 2024-04-16T16:13:35.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.623+0000 7fa9a9c00640 1 -- 172.21.15.45:0/839906431 wait complete. 2024-04-16T16:13:35.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.624+0000 7f0456800640 1 -- 172.21.15.45:0/3803415431 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f042c06c9c0 msgr2=0x7f042c06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.624+0000 7f0456800640 1 --2- 172.21.15.45:0/3803415431 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f042c06c9c0 0x7f042c06ee80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0450153c50 tx=0x7f0438008040 comp rx=0 tx=0).stop 2024-04-16T16:13:35.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.624+0000 7f0456800640 1 -- 172.21.15.45:0/3803415431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f045009b100 msgr2=0x7f04500c8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.624+0000 7f0456800640 1 --2- 172.21.15.45:0/3803415431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f045009b100 0x7f04500c8850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f044000ba50 tx=0x7f044000bf10 comp rx=0 tx=0).stop 2024-04-16T16:13:35.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.625+0000 7f0456800640 1 -- 172.21.15.45:0/3803415431 shutdown_connections 2024-04-16T16:13:35.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.625+0000 7f0456800640 1 --2- 172.21.15.45:0/3803415431 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f042c06c9c0 0x7f042c06ee80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.625+0000 7f0456800640 1 --2- 172.21.15.45:0/3803415431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f045015cb50 0x7f045014f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.625+0000 7f0456800640 1 --2- 172.21.15.45:0/3803415431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04500cd350 0x7f04500c8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.625+0000 7f0456800640 1 --2- 172.21.15.45:0/3803415431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f045009b100 0x7f04500c8850 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.625+0000 7f0456800640 1 -- 172.21.15.45:0/3803415431 >> 172.21.15.45:0/3803415431 conn(0x7f04500b4330 msgr2=0x7f04500b3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.625+0000 7f0456800640 1 -- 172.21.15.45:0/3803415431 shutdown_connections 2024-04-16T16:13:35.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.625+0000 7f0456800640 1 -- 172.21.15.45:0/3803415431 wait complete. 2024-04-16T16:13:35.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.630+0000 7fb226e00640 1 -- 172.21.15.45:0/2010020504 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb1f8005850 con 0x7fb22015edf0 2024-04-16T16:13:35.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.631+0000 7fb217e00640 1 -- 172.21.15.45:0/2010020504 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb21007b4b0 con 0x7fb22015edf0 2024-04-16T16:13:35.631 INFO:teuthology.orchestra.run.smithi045.stdout:55834574853 2024-04-16T16:13:35.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.634+0000 7fb226e00640 1 -- 172.21.15.45:0/2010020504 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb20006c8f0 msgr2=0x7fb20006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.634+0000 7fb226e00640 1 --2- 172.21.15.45:0/2010020504 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb20006c8f0 0x7fb20006edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb20c006a90 tx=0x7fb20c006a20 comp rx=0 tx=0).stop 2024-04-16T16:13:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.634+0000 7fb226e00640 1 -- 172.21.15.45:0/2010020504 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb22015edf0 msgr2=0x7fb2200ac640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.634+0000 7fb226e00640 1 --2- 172.21.15.45:0/2010020504 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb22015edf0 0x7fb2200ac640 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb210009870 tx=0x7fb210004730 comp rx=0 tx=0).stop 2024-04-16T16:13:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.635+0000 7fb226e00640 1 -- 172.21.15.45:0/2010020504 shutdown_connections 2024-04-16T16:13:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.635+0000 7fb226e00640 1 --2- 172.21.15.45:0/2010020504 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb20006c8f0 0x7fb20006edb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.635+0000 7fb226e00640 1 --2- 172.21.15.45:0/2010020504 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb22015edf0 0x7fb2200ac640 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.635+0000 7fb226e00640 1 --2- 172.21.15.45:0/2010020504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2200d1f70 0x7fb2201607b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.635+0000 7fb226e00640 1 --2- 172.21.15.45:0/2010020504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2200b8c20 0x7fb22015d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.635+0000 7fb226e00640 1 -- 172.21.15.45:0/2010020504 >> 172.21.15.45:0/2010020504 conn(0x7fb220073480 msgr2=0x7fb2200cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.635+0000 7fb226e00640 1 -- 172.21.15.45:0/2010020504 shutdown_connections 2024-04-16T16:13:35.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.635+0000 7fb226e00640 1 -- 172.21.15.45:0/2010020504 wait complete. 2024-04-16T16:13:35.645 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.7 2024-04-16T16:13:35.645 DEBUG:teuthology.parallel:result is None 2024-04-16T16:13:35.649 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.5 2024-04-16T16:13:35.649 DEBUG:teuthology.parallel:result is None 2024-04-16T16:13:35.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.653+0000 7fe50b400640 1 -- 172.21.15.45:0/3866465324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fe5040d0750 con 0x7fe50415edf0 2024-04-16T16:13:35.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.654+0000 7fe500c00640 1 -- 172.21.15.45:0/3866465324 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe4f0059340 con 0x7fe50415edf0 2024-04-16T16:13:35.654 INFO:teuthology.orchestra.run.smithi045.stdout:55834574854 2024-04-16T16:13:35.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.656+0000 7f38d5600640 1 -- 172.21.15.45:0/3419675152 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f38d00b8ba0 con 0x7f38d00cfe90 2024-04-16T16:13:35.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.657+0000 7fe50b400640 1 -- 172.21.15.45:0/3866465324 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe4e406c940 msgr2=0x7fe4e406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.657+0000 7fe50b400640 1 --2- 172.21.15.45:0/3866465324 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe4e406c940 0x7fe4e406ee00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc009f60 tx=0x7fe4fc0094f0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.657+0000 7fe50b400640 1 -- 172.21.15.45:0/3866465324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe50415edf0 msgr2=0x7fe504157a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.657+0000 7fe50b400640 1 --2- 172.21.15.45:0/3866465324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe50415edf0 0x7fe504157a10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000c780 tx=0x7fe4f000cc40 comp rx=0 tx=0).stop 2024-04-16T16:13:35.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.657+0000 7fe50b400640 1 -- 172.21.15.45:0/3866465324 shutdown_connections 2024-04-16T16:13:35.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.657+0000 7fe50b400640 1 --2- 172.21.15.45:0/3866465324 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe4e406c940 0x7fe4e406ee00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.657+0000 7fe50b400640 1 --2- 172.21.15.45:0/3866465324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe504156100 0x7fe504161060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.657+0000 7fe50b400640 1 --2- 172.21.15.45:0/3866465324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe50415edf0 0x7fe504157a10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.657+0000 7fe50b400640 1 --2- 172.21.15.45:0/3866465324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5040d2140 0x7fe5040ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.657+0000 7fe50b400640 1 -- 172.21.15.45:0/3866465324 >> 172.21.15.45:0/3866465324 conn(0x7fe504073480 msgr2=0x7fe5040b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.657+0000 7fe50b400640 1 -- 172.21.15.45:0/3866465324 shutdown_connections 2024-04-16T16:13:35.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.657+0000 7fe50b400640 1 -- 172.21.15.45:0/3866465324 wait complete. 2024-04-16T16:13:35.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.657+0000 7f38c7400640 1 -- 172.21.15.45:0/3419675152 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f38b807fb40 con 0x7f38d00cfe90 2024-04-16T16:13:35.658 INFO:teuthology.orchestra.run.smithi045.stdout:55834574853 2024-04-16T16:13:35.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.659+0000 7f3506a00640 1 -- 172.21.15.45:0/2783082448 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f34f80057e0 con 0x7f34e80052e0 2024-04-16T16:13:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.660+0000 7f34f3e00640 1 -- 172.21.15.45:0/2783082448 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f34f80387e0 con 0x7f34e80052e0 2024-04-16T16:13:35.660 INFO:teuthology.orchestra.run.smithi045.stdout:55834574854 2024-04-16T16:13:35.661 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.3 2024-04-16T16:13:35.661 DEBUG:teuthology.parallel:result is None 2024-04-16T16:13:35.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.661+0000 7f38d5600640 1 -- 172.21.15.45:0/3419675152 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f38ac06c9c0 msgr2=0x7f38ac06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.661+0000 7f38d5600640 1 --2- 172.21.15.45:0/3419675152 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f38ac06c9c0 0x7f38ac06ee80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f38d0157a70 tx=0x7f38c0009040 comp rx=0 tx=0).stop 2024-04-16T16:13:35.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.661+0000 7f38d5600640 1 -- 172.21.15.45:0/3419675152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38d00cfe90 msgr2=0x7f38d0156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.661+0000 7f38d5600640 1 --2- 172.21.15.45:0/3419675152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38d00cfe90 0x7f38d0156cb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f38b8033000 tx=0x7f38b80046c0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.661+0000 7f38d5600640 1 -- 172.21.15.45:0/3419675152 shutdown_connections 2024-04-16T16:13:35.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.661+0000 7f38d5600640 1 --2- 172.21.15.45:0/3419675152 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f38ac06c9c0 0x7f38ac06ee80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.661+0000 7f38d5600640 1 --2- 172.21.15.45:0/3419675152 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38d015ed30 0x7f38d0159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.661+0000 7f38d5600640 1 --2- 172.21.15.45:0/3419675152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38d00d2fe0 0x7f38d015d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.661+0000 7f38d5600640 1 --2- 172.21.15.45:0/3419675152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38d00cfe90 0x7f38d0156cb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.661+0000 7f38d5600640 1 -- 172.21.15.45:0/3419675152 >> 172.21.15.45:0/3419675152 conn(0x7f38d00d1610 msgr2=0x7f38d00b90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.661+0000 7f38d5600640 1 -- 172.21.15.45:0/3419675152 shutdown_connections 2024-04-16T16:13:35.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.661+0000 7f38d5600640 1 -- 172.21.15.45:0/3419675152 wait complete. 2024-04-16T16:13:35.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.663+0000 7f3506a00640 1 -- 172.21.15.45:0/2783082448 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f34e006c9c0 msgr2=0x7f34e006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.663+0000 7f3506a00640 1 --2- 172.21.15.45:0/2783082448 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f34e006c9c0 0x7f34e006ee80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f34ec006fd0 tx=0x7f34ec008040 comp rx=0 tx=0).stop 2024-04-16T16:13:35.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.663+0000 7f3506a00640 1 -- 172.21.15.45:0/2783082448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34e80052e0 msgr2=0x7f34f80526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.663+0000 7f3506a00640 1 --2- 172.21.15.45:0/2783082448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34e80052e0 0x7f34f80526a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f34f40018a0 tx=0x7f34f4001d60 comp rx=0 tx=0).stop 2024-04-16T16:13:35.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.663+0000 7f3506a00640 1 -- 172.21.15.45:0/2783082448 shutdown_connections 2024-04-16T16:13:35.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.663+0000 7f3506a00640 1 --2- 172.21.15.45:0/2783082448 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f34e006c9c0 0x7f34e006ee80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.663+0000 7f3506a00640 1 --2- 172.21.15.45:0/2783082448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34f804b2d0 0x7f34f8185f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.663+0000 7f3506a00640 1 --2- 172.21.15.45:0/2783082448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34f8007720 0x7f34f81859c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.663+0000 7f3506a00640 1 --2- 172.21.15.45:0/2783082448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34e80052e0 0x7f34f80526a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.663+0000 7f3506a00640 1 -- 172.21.15.45:0/2783082448 >> 172.21.15.45:0/2783082448 conn(0x7f34f80559e0 msgr2=0x7f34f8006c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.663+0000 7f3506a00640 1 -- 172.21.15.45:0/2783082448 shutdown_connections 2024-04-16T16:13:35.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.663+0000 7f3506a00640 1 -- 172.21.15.45:0/2783082448 wait complete. 2024-04-16T16:13:35.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.664+0000 7f9aca800640 1 -- 172.21.15.45:0/1884767985 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f9ac406de10 con 0x7f9ac40d2fe0 2024-04-16T16:13:35.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.665+0000 7f9abbe00640 1 -- 172.21.15.45:0/1884767985 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9ab405e3e0 con 0x7f9ac40d2fe0 2024-04-16T16:13:35.666 INFO:teuthology.orchestra.run.smithi045.stdout:55834574854 2024-04-16T16:13:35.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.670+0000 7f9aca800640 1 -- 172.21.15.45:0/1884767985 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9aa006c9c0 msgr2=0x7f9aa006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.670+0000 7f9aca800640 1 --2- 172.21.15.45:0/1884767985 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9aa006c9c0 0x7f9aa006ee80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9aac036000 tx=0x7f9aac03d040 comp rx=0 tx=0).stop 2024-04-16T16:13:35.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.670+0000 7f9aca800640 1 -- 172.21.15.45:0/1884767985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac40d2fe0 msgr2=0x7f9ac4201450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.670+0000 7f9aca800640 1 --2- 172.21.15.45:0/1884767985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac40d2fe0 0x7f9ac4201450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400e9b0 tx=0x7f9ab400ee70 comp rx=0 tx=0).stop 2024-04-16T16:13:35.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.670+0000 7f9aca800640 1 -- 172.21.15.45:0/1884767985 shutdown_connections 2024-04-16T16:13:35.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.670+0000 7f9aca800640 1 --2- 172.21.15.45:0/1884767985 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9aa006c9c0 0x7f9aa006ee80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.670+0000 7f9aca800640 1 --2- 172.21.15.45:0/1884767985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ac415ed30 0x7f9ac40cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.670+0000 7f9aca800640 1 --2- 172.21.15.45:0/1884767985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac40d2fe0 0x7f9ac4201450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.670+0000 7f9aca800640 1 --2- 172.21.15.45:0/1884767985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ac40ba710 0x7f9ac41609e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.670+0000 7f9aca800640 1 -- 172.21.15.45:0/1884767985 >> 172.21.15.45:0/1884767985 conn(0x7f9ac40d1610 msgr2=0x7f9ac40d4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.670+0000 7f9aca800640 1 -- 172.21.15.45:0/1884767985 shutdown_connections 2024-04-16T16:13:35.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.671+0000 7f9aca800640 1 -- 172.21.15.45:0/1884767985 wait complete. 2024-04-16T16:13:35.679 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.6 2024-04-16T16:13:35.679 DEBUG:teuthology.parallel:result is None 2024-04-16T16:13:35.682 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.2 2024-04-16T16:13:35.682 DEBUG:teuthology.parallel:result is None 2024-04-16T16:13:35.687 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.0 2024-04-16T16:13:35.687 DEBUG:teuthology.parallel:result is None 2024-04-16T16:13:35.692 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.1 2024-04-16T16:13:35.692 DEBUG:teuthology.parallel:result is None 2024-04-16T16:13:35.693 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T16:13:35.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:13:35.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.951+0000 7fd2d7c00640 1 Processor -- start 2024-04-16T16:13:35.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.952+0000 7fd2d7c00640 1 -- start start 2024-04-16T16:13:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.953+0000 7fd2d7c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd2d015ec60 0x7fd2d00d2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.953+0000 7fd2d7c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd2d00d1ac0 0x7fd2d00b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.953+0000 7fd2d7c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2d00d34c0 con 0x7fd2d00d4560 2024-04-16T16:13:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.953+0000 7fd2d7c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd2d00d37a0 con 0x7fd2d015ec60 2024-04-16T16:13:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.953+0000 7fd2d7c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd2d00d3a80 con 0x7fd2d00d1ac0 2024-04-16T16:13:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.953+0000 7fd2d5200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd2d015ec60 0x7fd2d00d2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.954+0000 7fd2d5200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd2d015ec60 0x7fd2d00d2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43350/0 (socket says 172.21.15.45:43350) 2024-04-16T16:13:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.954+0000 7fd2d5200640 1 -- 172.21.15.45:0/624938219 learned_addr learned my addr 172.21.15.45:0/624938219 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.954+0000 7fd2d5c00640 1 --2- 172.21.15.45:0/624938219 >> v2:172.21.15.103:3300/0 conn(0x7fd2d00d1ac0 0x7fd2d00b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.954+0000 7fd2d5200640 1 -- 172.21.15.45:0/624938219 >> v2:172.21.15.103:3300/0 conn(0x7fd2d00d1ac0 msgr2=0x7fd2d00b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.954+0000 7fd2d5200640 1 --2- 172.21.15.45:0/624938219 >> v2:172.21.15.103:3300/0 conn(0x7fd2d00d1ac0 0x7fd2d00b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.954+0000 7fd2d5200640 1 -- 172.21.15.45:0/624938219 >> v1:172.21.15.45:6789/0 conn(0x7fd2d00d4560 legacy=0x7fd2d00d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.954+0000 7fd2d5200640 1 -- 172.21.15.45:0/624938219 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d00d3d60 con 0x7fd2d015ec60 2024-04-16T16:13:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.955+0000 7fd2d5200640 1 --2- 172.21.15.45:0/624938219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2d015ec60 0x7fd2d00d2a00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd2d006e000 tx=0x7fd2c002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f068383ed6d52a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.955+0000 7fd2cf400640 1 -- 172.21.15.45:0/624938219 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0034070 con 0x7fd2d015ec60 2024-04-16T16:13:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.955+0000 7fd2cf400640 1 -- 172.21.15.45:0/624938219 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2c0030870 con 0x7fd2d015ec60 2024-04-16T16:13:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.955+0000 7fd2cf400640 1 -- 172.21.15.45:0/624938219 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0030c40 con 0x7fd2d015ec60 2024-04-16T16:13:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.956+0000 7fd2d7c00640 1 -- 172.21.15.45:0/624938219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2d015ec60 msgr2=0x7fd2d00d2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.956+0000 7fd2d7c00640 1 --2- 172.21.15.45:0/624938219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2d015ec60 0x7fd2d00d2a00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd2d006e000 tx=0x7fd2c002b9b0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.956+0000 7fd2d7c00640 1 -- 172.21.15.45:0/624938219 shutdown_connections 2024-04-16T16:13:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.956+0000 7fd2d7c00640 1 --2- 172.21.15.45:0/624938219 >> v2:172.21.15.103:3300/0 conn(0x7fd2d00d1ac0 0x7fd2d00b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.956+0000 7fd2d7c00640 1 --2- 172.21.15.45:0/624938219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2d015ec60 0x7fd2d00d2a00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.956+0000 7fd2d7c00640 1 -- 172.21.15.45:0/624938219 >> 172.21.15.45:0/624938219 conn(0x7fd2d0073690 msgr2=0x7fd2d006cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:35.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.956+0000 7fd2d7c00640 1 -- 172.21.15.45:0/624938219 shutdown_connections 2024-04-16T16:13:35.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.956+0000 7fd2d7c00640 1 -- 172.21.15.45:0/624938219 wait complete. 2024-04-16T16:13:35.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.958+0000 7fd2d7c00640 1 Processor -- start 2024-04-16T16:13:35.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.958+0000 7fd2d7c00640 1 -- start start 2024-04-16T16:13:35.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.959+0000 7fd2d7c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2d00d1ac0 0x7fd2d0153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.959+0000 7fd2d7c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2d00d4560 0x7fd2d015b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.959+0000 7fd2d7c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2d015ec60 0x7fd2d0160c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.959+0000 7fd2d7c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d00d2f40 con 0x7fd2d015ec60 2024-04-16T16:13:35.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.959+0000 7fd2d7c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d00ba460 con 0x7fd2d00d4560 2024-04-16T16:13:35.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.959+0000 7fd2d7c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd2d00d30c0 con 0x7fd2d00d1ac0 2024-04-16T16:13:35.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.959+0000 7fd2d5c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2d015ec60 0x7fd2d0160c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.959+0000 7fd2d5c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2d015ec60 0x7fd2d0160c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43364/0 (socket says 172.21.15.45:43364) 2024-04-16T16:13:35.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.959+0000 7fd2d5c00640 1 -- 172.21.15.45:0/1639457776 learned_addr learned my addr 172.21.15.45:0/1639457776 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:35.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.960+0000 7fd2d5200640 1 --2- 172.21.15.45:0/1639457776 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2d00d1ac0 0x7fd2d0153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.960+0000 7fd2cfe00640 1 --2- 172.21.15.45:0/1639457776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2d00d4560 0x7fd2d015b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.960+0000 7fd2d5c00640 1 -- 172.21.15.45:0/1639457776 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2d00d1ac0 msgr2=0x7fd2d0153f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.960+0000 7fd2d5c00640 1 --2- 172.21.15.45:0/1639457776 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2d00d1ac0 0x7fd2d0153f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.960+0000 7fd2d5c00640 1 -- 172.21.15.45:0/1639457776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2d00d4560 msgr2=0x7fd2d015b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.960+0000 7fd2d5c00640 1 --2- 172.21.15.45:0/1639457776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2d00d4560 0x7fd2d015b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.960+0000 7fd2d5c00640 1 -- 172.21.15.45:0/1639457776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d01613e0 con 0x7fd2d015ec60 2024-04-16T16:13:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.960+0000 7fd2cfe00640 1 --2- 172.21.15.45:0/1639457776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2d00d4560 0x7fd2d015b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.960+0000 7fd2d5c00640 1 --2- 172.21.15.45:0/1639457776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2d015ec60 0x7fd2d0160c70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800d5d0 tx=0x7fd2c800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.960+0000 7fd2ccc00640 1 -- 172.21.15.45:0/1639457776 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80125f0 con 0x7fd2d015ec60 2024-04-16T16:13:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.960+0000 7fd2ccc00640 1 -- 172.21.15.45:0/1639457776 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2c8019070 con 0x7fd2d015ec60 2024-04-16T16:13:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.960+0000 7fd2ccc00640 1 -- 172.21.15.45:0/1639457776 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80129d0 con 0x7fd2d015ec60 2024-04-16T16:13:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.960+0000 7fd2d7c00640 1 -- 172.21.15.45:0/1639457776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d01616c0 con 0x7fd2d015ec60 2024-04-16T16:13:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.960+0000 7fd2d7c00640 1 -- 172.21.15.45:0/1639457776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d0155080 con 0x7fd2d015ec60 2024-04-16T16:13:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.961+0000 7fd2d7c00640 1 -- 172.21.15.45:0/1639457776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2d0075780 con 0x7fd2d015ec60 2024-04-16T16:13:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.962+0000 7fd2ccc00640 1 -- 172.21.15.45:0/1639457776 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd2c8012b70 con 0x7fd2d015ec60 2024-04-16T16:13:35.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.963+0000 7fd2ccc00640 1 --2- 172.21.15.45:0/1639457776 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd2ac06c920 0x7fd2ac06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:35.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.963+0000 7fd2ccc00640 1 -- 172.21.15.45:0/1639457776 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd2c808c0b0 con 0x7fd2d015ec60 2024-04-16T16:13:35.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.964+0000 7fd2d5200640 1 --2- 172.21.15.45:0/1639457776 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd2ac06c920 0x7fd2ac06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:35.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.965+0000 7fd2d5200640 1 --2- 172.21.15.45:0/1639457776 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd2ac06c920 0x7fd2ac06ede0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0036000 tx=0x7fd2c003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:35.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:35.968+0000 7fd2ccc00640 1 -- 172.21.15.45:0/1639457776 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd2c805a3b0 con 0x7fd2d015ec60 2024-04-16T16:13:36.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.160+0000 7fd2d7c00640 1 -- 172.21.15.45:0/1639457776 --> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd2d00ba960 con 0x7fd2ac06c920 2024-04-16T16:13:36.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.182+0000 7fd2ccc00640 1 -- 172.21.15.45:0/1639457776 <== mgr.4105 v2:172.21.15.103:6832/2711977587 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251838 (secure 0 0 0) 0x7fd2d00ba960 con 0x7fd2ac06c920 2024-04-16T16:13:36.184 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:13:36.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.188+0000 7fd2d7c00640 1 -- 172.21.15.45:0/1639457776 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd2ac06c920 msgr2=0x7fd2ac06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:36.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.188+0000 7fd2d7c00640 1 --2- 172.21.15.45:0/1639457776 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd2ac06c920 0x7fd2ac06ede0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0036000 tx=0x7fd2c003b040 comp rx=0 tx=0).stop 2024-04-16T16:13:36.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.188+0000 7fd2d7c00640 1 -- 172.21.15.45:0/1639457776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2d015ec60 msgr2=0x7fd2d0160c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:36.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.188+0000 7fd2d7c00640 1 --2- 172.21.15.45:0/1639457776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2d015ec60 0x7fd2d0160c70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800d5d0 tx=0x7fd2c800da90 comp rx=0 tx=0).stop 2024-04-16T16:13:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.188+0000 7fd2d7c00640 1 -- 172.21.15.45:0/1639457776 shutdown_connections 2024-04-16T16:13:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.188+0000 7fd2d7c00640 1 --2- 172.21.15.45:0/1639457776 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd2ac06c920 0x7fd2ac06ede0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.188+0000 7fd2d7c00640 1 --2- 172.21.15.45:0/1639457776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2d015ec60 0x7fd2d0160c70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.188+0000 7fd2d7c00640 1 --2- 172.21.15.45:0/1639457776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2d00d4560 0x7fd2d015b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.188+0000 7fd2d7c00640 1 --2- 172.21.15.45:0/1639457776 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2d00d1ac0 0x7fd2d0153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.188+0000 7fd2d7c00640 1 -- 172.21.15.45:0/1639457776 >> 172.21.15.45:0/1639457776 conn(0x7fd2d0073690 msgr2=0x7fd2d00c96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.188+0000 7fd2d7c00640 1 -- 172.21.15.45:0/1639457776 shutdown_connections 2024-04-16T16:13:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.188+0000 7fd2d7c00640 1 -- 172.21.15.45:0/1639457776 wait complete. 2024-04-16T16:13:36.210 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-16T16:13:35.095120+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235756,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738964,"statfs":{"total":773094113280,"available":772852699136,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.191477"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943618+0000","last_change":"2024-04-16T16:13:25.934978+0000","last_active":"2024-04-16T16:13:26.943618+0000","last_peered":"2024-04-16T16:13:26.943618+0000","last_clean":"2024-04-16T16:13:26.943618+0000","last_became_active":"2024-04-16T16:13:25.934469+0000","last_became_peered":"2024-04-16T16:13:25.934469+0000","last_unstale":"2024-04-16T16:13:26.943618+0000","last_undegraded":"2024-04-16T16:13:26.943618+0000","last_fullsized":"2024-04-16T16:13:26.943618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:40:10.575471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955303+0000","last_change":"2024-04-16T16:13:25.936302+0000","last_active":"2024-04-16T16:13:26.955303+0000","last_peered":"2024-04-16T16:13:26.955303+0000","last_clean":"2024-04-16T16:13:26.955303+0000","last_became_active":"2024-04-16T16:13:25.935907+0000","last_became_peered":"2024-04-16T16:13:25.935907+0000","last_unstale":"2024-04-16T16:13:26.955303+0000","last_undegraded":"2024-04-16T16:13:26.955303+0000","last_fullsized":"2024-04-16T16:13:26.955303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.574975+0000","last_change":"2024-04-16T16:13:25.935281+0000","last_active":"2024-04-16T16:13:28.574975+0000","last_peered":"2024-04-16T16:13:28.574975+0000","last_clean":"2024-04-16T16:13:28.574975+0000","last_became_active":"2024-04-16T16:13:25.934986+0000","last_became_peered":"2024-04-16T16:13:25.934986+0000","last_unstale":"2024-04-16T16:13:28.574975+0000","last_undegraded":"2024-04-16T16:13:28.574975+0000","last_fullsized":"2024-04-16T16:13:28.574975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:10.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970937+0000","last_change":"2024-04-16T16:13:25.952485+0000","last_active":"2024-04-16T16:13:26.970937+0000","last_peered":"2024-04-16T16:13:26.970937+0000","last_clean":"2024-04-16T16:13:26.970937+0000","last_became_active":"2024-04-16T16:13:25.952189+0000","last_became_peered":"2024-04-16T16:13:25.952189+0000","last_unstale":"2024-04-16T16:13:26.970937+0000","last_undegraded":"2024-04-16T16:13:26.970937+0000","last_fullsized":"2024-04-16T16:13:26.970937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:47.987635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942605+0000","last_change":"2024-04-16T16:13:25.930237+0000","last_active":"2024-04-16T16:13:26.942605+0000","last_peered":"2024-04-16T16:13:26.942605+0000","last_clean":"2024-04-16T16:13:26.942605+0000","last_became_active":"2024-04-16T16:13:25.929635+0000","last_became_peered":"2024-04-16T16:13:25.929635+0000","last_unstale":"2024-04-16T16:13:26.942605+0000","last_undegraded":"2024-04-16T16:13:26.942605+0000","last_fullsized":"2024-04-16T16:13:26.942605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:30:58.937717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970737+0000","last_change":"2024-04-16T16:13:25.947705+0000","last_active":"2024-04-16T16:13:26.970737+0000","last_peered":"2024-04-16T16:13:26.970737+0000","last_clean":"2024-04-16T16:13:26.970737+0000","last_became_active":"2024-04-16T16:13:25.947433+0000","last_became_peered":"2024-04-16T16:13:25.947433+0000","last_unstale":"2024-04-16T16:13:26.970737+0000","last_undegraded":"2024-04-16T16:13:26.970737+0000","last_fullsized":"2024-04-16T16:13:26.970737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941969+0000","last_change":"2024-04-16T16:13:25.930905+0000","last_active":"2024-04-16T16:13:26.941969+0000","last_peered":"2024-04-16T16:13:26.941969+0000","last_clean":"2024-04-16T16:13:26.941969+0000","last_became_active":"2024-04-16T16:13:25.929782+0000","last_became_peered":"2024-04-16T16:13:25.929782+0000","last_unstale":"2024-04-16T16:13:26.941969+0000","last_undegraded":"2024-04-16T16:13:26.941969+0000","last_fullsized":"2024-04-16T16:13:26.941969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.572328+0000","last_change":"2024-04-16T16:13:25.947751+0000","last_active":"2024-04-16T16:13:28.572328+0000","last_peered":"2024-04-16T16:13:28.572328+0000","last_clean":"2024-04-16T16:13:28.572328+0000","last_became_active":"2024-04-16T16:13:25.947110+0000","last_became_peered":"2024-04-16T16:13:25.947110+0000","last_unstale":"2024-04-16T16:13:28.572328+0000","last_undegraded":"2024-04-16T16:13:28.572328+0000","last_fullsized":"2024-04-16T16:13:28.572328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:51:20.451559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943813+0000","last_change":"2024-04-16T16:13:25.950917+0000","last_active":"2024-04-16T16:13:26.943813+0000","last_peered":"2024-04-16T16:13:26.943813+0000","last_clean":"2024-04-16T16:13:26.943813+0000","last_became_active":"2024-04-16T16:13:25.946716+0000","last_became_peered":"2024-04-16T16:13:25.946716+0000","last_unstale":"2024-04-16T16:13:26.943813+0000","last_undegraded":"2024-04-16T16:13:26.943813+0000","last_fullsized":"2024-04-16T16:13:26.943813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:52:47.172342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961550+0000","last_change":"2024-04-16T16:13:25.941148+0000","last_active":"2024-04-16T16:13:26.961550+0000","last_peered":"2024-04-16T16:13:26.961550+0000","last_clean":"2024-04-16T16:13:26.961550+0000","last_became_active":"2024-04-16T16:13:25.940818+0000","last_became_peered":"2024-04-16T16:13:25.940818+0000","last_unstale":"2024-04-16T16:13:26.961550+0000","last_undegraded":"2024-04-16T16:13:26.961550+0000","last_fullsized":"2024-04-16T16:13:26.961550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960975+0000","last_change":"2024-04-16T16:13:25.943177+0000","last_active":"2024-04-16T16:13:26.960975+0000","last_peered":"2024-04-16T16:13:26.960975+0000","last_clean":"2024-04-16T16:13:26.960975+0000","last_became_active":"2024-04-16T16:13:25.942902+0000","last_became_peered":"2024-04-16T16:13:25.942902+0000","last_unstale":"2024-04-16T16:13:26.960975+0000","last_undegraded":"2024-04-16T16:13:26.960975+0000","last_fullsized":"2024-04-16T16:13:26.960975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:40:58.670474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941834+0000","last_change":"2024-04-16T16:13:25.938811+0000","last_active":"2024-04-16T16:13:26.941834+0000","last_peered":"2024-04-16T16:13:26.941834+0000","last_clean":"2024-04-16T16:13:26.941834+0000","last_became_active":"2024-04-16T16:13:25.938479+0000","last_became_peered":"2024-04-16T16:13:25.938479+0000","last_unstale":"2024-04-16T16:13:26.941834+0000","last_undegraded":"2024-04-16T16:13:26.941834+0000","last_fullsized":"2024-04-16T16:13:26.941834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:41.335228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.573197+0000","last_change":"2024-04-16T16:13:25.950387+0000","last_active":"2024-04-16T16:13:28.573197+0000","last_peered":"2024-04-16T16:13:28.573197+0000","last_clean":"2024-04-16T16:13:28.573197+0000","last_became_active":"2024-04-16T16:13:25.949164+0000","last_became_peered":"2024-04-16T16:13:25.949164+0000","last_unstale":"2024-04-16T16:13:28.573197+0000","last_undegraded":"2024-04-16T16:13:28.573197+0000","last_fullsized":"2024-04-16T16:13:28.573197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:40:58.670474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.572204+0000","last_change":"2024-04-16T16:13:25.939438+0000","last_active":"2024-04-16T16:13:28.572204+0000","last_peered":"2024-04-16T16:13:28.572204+0000","last_clean":"2024-04-16T16:13:28.572204+0000","last_became_active":"2024-04-16T16:13:25.939066+0000","last_became_peered":"2024-04-16T16:13:25.939066+0000","last_unstale":"2024-04-16T16:13:28.572204+0000","last_undegraded":"2024-04-16T16:13:28.572204+0000","last_fullsized":"2024-04-16T16:13:28.572204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:07.860908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954570+0000","last_change":"2024-04-16T16:13:25.937157+0000","last_active":"2024-04-16T16:13:26.954570+0000","last_peered":"2024-04-16T16:13:26.954570+0000","last_clean":"2024-04-16T16:13:26.954570+0000","last_became_active":"2024-04-16T16:13:25.936851+0000","last_became_peered":"2024-04-16T16:13:25.936851+0000","last_unstale":"2024-04-16T16:13:26.954570+0000","last_undegraded":"2024-04-16T16:13:26.954570+0000","last_fullsized":"2024-04-16T16:13:26.954570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:55.948136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970498+0000","last_change":"2024-04-16T16:13:25.949240+0000","last_active":"2024-04-16T16:13:26.970498+0000","last_peered":"2024-04-16T16:13:26.970498+0000","last_clean":"2024-04-16T16:13:26.970498+0000","last_became_active":"2024-04-16T16:13:25.948913+0000","last_became_peered":"2024-04-16T16:13:25.948913+0000","last_unstale":"2024-04-16T16:13:26.970498+0000","last_undegraded":"2024-04-16T16:13:26.970498+0000","last_fullsized":"2024-04-16T16:13:26.970498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.972475+0000","last_change":"2024-04-16T16:13:25.947113+0000","last_active":"2024-04-16T16:13:26.972475+0000","last_peered":"2024-04-16T16:13:26.972475+0000","last_clean":"2024-04-16T16:13:26.972475+0000","last_became_active":"2024-04-16T16:13:25.943174+0000","last_became_peered":"2024-04-16T16:13:25.943174+0000","last_unstale":"2024-04-16T16:13:26.972475+0000","last_undegraded":"2024-04-16T16:13:26.972475+0000","last_fullsized":"2024-04-16T16:13:26.972475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:08.801668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.937163+0000","last_change":"2024-04-16T16:13:25.942311+0000","last_active":"2024-04-16T16:13:26.937163+0000","last_peered":"2024-04-16T16:13:26.937163+0000","last_clean":"2024-04-16T16:13:26.937163+0000","last_became_active":"2024-04-16T16:13:25.942015+0000","last_became_peered":"2024-04-16T16:13:25.942015+0000","last_unstale":"2024-04-16T16:13:26.937163+0000","last_undegraded":"2024-04-16T16:13:26.937163+0000","last_fullsized":"2024-04-16T16:13:26.937163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:25.740193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943340+0000","last_change":"2024-04-16T16:13:25.929976+0000","last_active":"2024-04-16T16:13:26.943340+0000","last_peered":"2024-04-16T16:13:26.943340+0000","last_clean":"2024-04-16T16:13:26.943340+0000","last_became_active":"2024-04-16T16:13:25.929347+0000","last_became_peered":"2024-04-16T16:13:25.929347+0000","last_unstale":"2024-04-16T16:13:26.943340+0000","last_undegraded":"2024-04-16T16:13:26.943340+0000","last_fullsized":"2024-04-16T16:13:26.943340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:19:05.123994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960773+0000","last_change":"2024-04-16T16:13:25.945575+0000","last_active":"2024-04-16T16:13:26.960773+0000","last_peered":"2024-04-16T16:13:26.960773+0000","last_clean":"2024-04-16T16:13:26.960773+0000","last_became_active":"2024-04-16T16:13:25.945217+0000","last_became_peered":"2024-04-16T16:13:25.945217+0000","last_unstale":"2024-04-16T16:13:26.960773+0000","last_undegraded":"2024-04-16T16:13:26.960773+0000","last_fullsized":"2024-04-16T16:13:26.960773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:55.948136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.574655+0000","last_change":"2024-04-16T16:13:25.945082+0000","last_active":"2024-04-16T16:13:28.574655+0000","last_peered":"2024-04-16T16:13:28.574655+0000","last_clean":"2024-04-16T16:13:28.574655+0000","last_became_active":"2024-04-16T16:13:25.944781+0000","last_became_peered":"2024-04-16T16:13:25.944781+0000","last_unstale":"2024-04-16T16:13:28.574655+0000","last_undegraded":"2024-04-16T16:13:28.574655+0000","last_fullsized":"2024-04-16T16:13:28.574655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:51:20.451559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970886+0000","last_change":"2024-04-16T16:13:25.943741+0000","last_active":"2024-04-16T16:13:26.970886+0000","last_peered":"2024-04-16T16:13:26.970886+0000","last_clean":"2024-04-16T16:13:26.970886+0000","last_became_active":"2024-04-16T16:13:25.942843+0000","last_became_peered":"2024-04-16T16:13:25.942843+0000","last_unstale":"2024-04-16T16:13:26.970886+0000","last_undegraded":"2024-04-16T16:13:26.970886+0000","last_fullsized":"2024-04-16T16:13:26.970886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:30:58.937717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.572988+0000","last_change":"2024-04-16T16:13:25.943232+0000","last_active":"2024-04-16T16:13:28.572988+0000","last_peered":"2024-04-16T16:13:28.572988+0000","last_clean":"2024-04-16T16:13:28.572988+0000","last_became_active":"2024-04-16T16:13:25.940853+0000","last_became_peered":"2024-04-16T16:13:25.940853+0000","last_unstale":"2024-04-16T16:13:28.572988+0000","last_undegraded":"2024-04-16T16:13:28.572988+0000","last_fullsized":"2024-04-16T16:13:28.572988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:55.948136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970797+0000","last_change":"2024-04-16T16:13:25.943328+0000","last_active":"2024-04-16T16:13:26.970797+0000","last_peered":"2024-04-16T16:13:26.970797+0000","last_clean":"2024-04-16T16:13:26.970797+0000","last_became_active":"2024-04-16T16:13:25.941855+0000","last_became_peered":"2024-04-16T16:13:25.941855+0000","last_unstale":"2024-04-16T16:13:26.970797+0000","last_undegraded":"2024-04-16T16:13:26.970797+0000","last_fullsized":"2024-04-16T16:13:26.970797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:41.243756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.574357+0000","last_change":"2024-04-16T16:13:25.942627+0000","last_active":"2024-04-16T16:13:28.574357+0000","last_peered":"2024-04-16T16:13:28.574357+0000","last_clean":"2024-04-16T16:13:28.574357+0000","last_became_active":"2024-04-16T16:13:25.942285+0000","last_became_peered":"2024-04-16T16:13:25.942285+0000","last_unstale":"2024-04-16T16:13:28.574357+0000","last_undegraded":"2024-04-16T16:13:28.574357+0000","last_fullsized":"2024-04-16T16:13:28.574357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:39.592952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961635+0000","last_change":"2024-04-16T16:13:25.950401+0000","last_active":"2024-04-16T16:13:26.961635+0000","last_peered":"2024-04-16T16:13:26.961635+0000","last_clean":"2024-04-16T16:13:26.961635+0000","last_became_active":"2024-04-16T16:13:25.948651+0000","last_became_peered":"2024-04-16T16:13:25.948651+0000","last_unstale":"2024-04-16T16:13:26.961635+0000","last_undegraded":"2024-04-16T16:13:26.961635+0000","last_fullsized":"2024-04-16T16:13:26.961635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955096+0000","last_change":"2024-04-16T16:13:25.945071+0000","last_active":"2024-04-16T16:13:26.955096+0000","last_peered":"2024-04-16T16:13:26.955096+0000","last_clean":"2024-04-16T16:13:26.955096+0000","last_became_active":"2024-04-16T16:13:25.944708+0000","last_became_peered":"2024-04-16T16:13:25.944708+0000","last_unstale":"2024-04-16T16:13:26.955096+0000","last_undegraded":"2024-04-16T16:13:26.955096+0000","last_fullsized":"2024-04-16T16:13:26.955096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.573821+0000","last_change":"2024-04-16T16:13:25.944654+0000","last_active":"2024-04-16T16:13:28.573821+0000","last_peered":"2024-04-16T16:13:28.573821+0000","last_clean":"2024-04-16T16:13:28.573821+0000","last_became_active":"2024-04-16T16:13:25.941914+0000","last_became_peered":"2024-04-16T16:13:25.941914+0000","last_unstale":"2024-04-16T16:13:28.573821+0000","last_undegraded":"2024-04-16T16:13:28.573821+0000","last_fullsized":"2024-04-16T16:13:28.573821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.574823+0000","last_change":"2024-04-16T16:13:25.947081+0000","last_active":"2024-04-16T16:13:28.574823+0000","last_peered":"2024-04-16T16:13:28.574823+0000","last_clean":"2024-04-16T16:13:28.574823+0000","last_became_active":"2024-04-16T16:13:25.946791+0000","last_became_peered":"2024-04-16T16:13:25.946791+0000","last_unstale":"2024-04-16T16:13:28.574823+0000","last_undegraded":"2024-04-16T16:13:28.574823+0000","last_fullsized":"2024-04-16T16:13:28.574823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:11:32.536402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971641+0000","last_change":"2024-04-16T16:13:25.952506+0000","last_active":"2024-04-16T16:13:26.971641+0000","last_peered":"2024-04-16T16:13:26.971641+0000","last_clean":"2024-04-16T16:13:26.971641+0000","last_became_active":"2024-04-16T16:13:25.952184+0000","last_became_peered":"2024-04-16T16:13:25.952184+0000","last_unstale":"2024-04-16T16:13:26.971641+0000","last_undegraded":"2024-04-16T16:13:26.971641+0000","last_fullsized":"2024-04-16T16:13:26.971641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:19:05.123994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936714+0000","last_change":"2024-04-16T16:13:25.936689+0000","last_active":"2024-04-16T16:13:26.936714+0000","last_peered":"2024-04-16T16:13:26.936714+0000","last_clean":"2024-04-16T16:13:26.936714+0000","last_became_active":"2024-04-16T16:13:25.936385+0000","last_became_peered":"2024-04-16T16:13:25.936385+0000","last_unstale":"2024-04-16T16:13:26.936714+0000","last_undegraded":"2024-04-16T16:13:26.936714+0000","last_fullsized":"2024-04-16T16:13:26.936714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:12.895579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960388+0000","last_change":"2024-04-16T16:13:25.953671+0000","last_active":"2024-04-16T16:13:26.960388+0000","last_peered":"2024-04-16T16:13:26.960388+0000","last_clean":"2024-04-16T16:13:26.960388+0000","last_became_active":"2024-04-16T16:13:25.953325+0000","last_became_peered":"2024-04-16T16:13:25.953325+0000","last_unstale":"2024-04-16T16:13:26.960388+0000","last_undegraded":"2024-04-16T16:13:26.960388+0000","last_fullsized":"2024-04-16T16:13:26.960388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:51:20.451559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941086+0000","last_change":"2024-04-16T16:13:25.936138+0000","last_active":"2024-04-16T16:13:26.941086+0000","last_peered":"2024-04-16T16:13:26.941086+0000","last_clean":"2024-04-16T16:13:26.941086+0000","last_became_active":"2024-04-16T16:13:25.935759+0000","last_became_peered":"2024-04-16T16:13:25.935759+0000","last_unstale":"2024-04-16T16:13:26.941086+0000","last_undegraded":"2024-04-16T16:13:26.941086+0000","last_fullsized":"2024-04-16T16:13:26.941086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:26.052535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.573477+0000","last_change":"2024-04-16T16:13:24.912677+0000","last_active":"2024-04-16T16:13:28.573477+0000","last_peered":"2024-04-16T16:13:28.573477+0000","last_clean":"2024-04-16T16:13:28.573477+0000","last_became_active":"2024-04-16T16:13:24.912364+0000","last_became_peered":"2024-04-16T16:13:24.912364+0000","last_unstale":"2024-04-16T16:13:28.573477+0000","last_undegraded":"2024-04-16T16:13:28.573477+0000","last_fullsized":"2024-04-16T16:13:28.573477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.575151+0000","last_change":"2024-04-16T16:13:25.947853+0000","last_active":"2024-04-16T16:13:28.575151+0000","last_peered":"2024-04-16T16:13:28.575151+0000","last_clean":"2024-04-16T16:13:28.575151+0000","last_became_active":"2024-04-16T16:13:25.947545+0000","last_became_peered":"2024-04-16T16:13:25.947545+0000","last_unstale":"2024-04-16T16:13:28.575151+0000","last_undegraded":"2024-04-16T16:13:28.575151+0000","last_fullsized":"2024-04-16T16:13:28.575151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960762+0000","last_change":"2024-04-16T16:13:24.917069+0000","last_active":"2024-04-16T16:13:26.960762+0000","last_peered":"2024-04-16T16:13:26.960762+0000","last_clean":"2024-04-16T16:13:26.960762+0000","last_became_active":"2024-04-16T16:13:24.916846+0000","last_became_peered":"2024-04-16T16:13:24.916846+0000","last_unstale":"2024-04-16T16:13:26.960762+0000","last_undegraded":"2024-04-16T16:13:26.960762+0000","last_fullsized":"2024-04-16T16:13:26.960762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.573484+0000","last_change":"2024-04-16T16:13:24.919388+0000","last_active":"2024-04-16T16:13:28.573484+0000","last_peered":"2024-04-16T16:13:28.573484+0000","last_clean":"2024-04-16T16:13:28.573484+0000","last_became_active":"2024-04-16T16:13:24.918937+0000","last_became_peered":"2024-04-16T16:13:24.918937+0000","last_unstale":"2024-04-16T16:13:28.573484+0000","last_undegraded":"2024-04-16T16:13:28.573484+0000","last_fullsized":"2024-04-16T16:13:28.573484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:46:08.208026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.572516+0000","last_change":"2024-04-16T16:13:25.954029+0000","last_active":"2024-04-16T16:13:28.572516+0000","last_peered":"2024-04-16T16:13:28.572516+0000","last_clean":"2024-04-16T16:13:28.572516+0000","last_became_active":"2024-04-16T16:13:25.953740+0000","last_became_peered":"2024-04-16T16:13:25.953740+0000","last_unstale":"2024-04-16T16:13:28.572516+0000","last_undegraded":"2024-04-16T16:13:28.572516+0000","last_fullsized":"2024-04-16T16:13:28.572516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:11:32.536402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971052+0000","last_change":"2024-04-16T16:13:24.923031+0000","last_active":"2024-04-16T16:13:26.971052+0000","last_peered":"2024-04-16T16:13:26.971052+0000","last_clean":"2024-04-16T16:13:26.971052+0000","last_became_active":"2024-04-16T16:13:24.922708+0000","last_became_peered":"2024-04-16T16:13:24.922708+0000","last_unstale":"2024-04-16T16:13:26.971052+0000","last_undegraded":"2024-04-16T16:13:26.971052+0000","last_fullsized":"2024-04-16T16:13:26.971052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954570+0000","last_change":"2024-04-16T16:13:25.949124+0000","last_active":"2024-04-16T16:13:26.954570+0000","last_peered":"2024-04-16T16:13:26.954570+0000","last_clean":"2024-04-16T16:13:26.954570+0000","last_became_active":"2024-04-16T16:13:25.948436+0000","last_became_peered":"2024-04-16T16:13:25.948436+0000","last_unstale":"2024-04-16T16:13:26.954570+0000","last_undegraded":"2024-04-16T16:13:26.954570+0000","last_fullsized":"2024-04-16T16:13:26.954570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:10.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961547+0000","last_change":"2024-04-16T16:13:24.925547+0000","last_active":"2024-04-16T16:13:26.961547+0000","last_peered":"2024-04-16T16:13:26.961547+0000","last_clean":"2024-04-16T16:13:26.961547+0000","last_became_active":"2024-04-16T16:13:24.925182+0000","last_became_peered":"2024-04-16T16:13:24.925182+0000","last_unstale":"2024-04-16T16:13:26.961547+0000","last_undegraded":"2024-04-16T16:13:26.961547+0000","last_fullsized":"2024-04-16T16:13:26.961547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.972351+0000","last_change":"2024-04-16T16:13:25.945204+0000","last_active":"2024-04-16T16:13:26.972351+0000","last_peered":"2024-04-16T16:13:26.972351+0000","last_clean":"2024-04-16T16:13:26.972351+0000","last_became_active":"2024-04-16T16:13:25.944902+0000","last_became_peered":"2024-04-16T16:13:25.944902+0000","last_unstale":"2024-04-16T16:13:26.972351+0000","last_undegraded":"2024-04-16T16:13:26.972351+0000","last_fullsized":"2024-04-16T16:13:26.972351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:52:47.172342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971774+0000","last_change":"2024-04-16T16:13:24.928041+0000","last_active":"2024-04-16T16:13:26.971774+0000","last_peered":"2024-04-16T16:13:26.971774+0000","last_clean":"2024-04-16T16:13:26.971774+0000","last_became_active":"2024-04-16T16:13:24.927555+0000","last_became_peered":"2024-04-16T16:13:24.927555+0000","last_unstale":"2024-04-16T16:13:26.971774+0000","last_undegraded":"2024-04-16T16:13:26.971774+0000","last_fullsized":"2024-04-16T16:13:26.971774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.575155+0000","last_change":"2024-04-16T16:13:25.943385+0000","last_active":"2024-04-16T16:13:28.575155+0000","last_peered":"2024-04-16T16:13:28.575155+0000","last_clean":"2024-04-16T16:13:28.575155+0000","last_became_active":"2024-04-16T16:13:25.943095+0000","last_became_peered":"2024-04-16T16:13:25.943095+0000","last_unstale":"2024-04-16T16:13:28.575155+0000","last_undegraded":"2024-04-16T16:13:28.575155+0000","last_fullsized":"2024-04-16T16:13:28.575155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:40:58.670474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955094+0000","last_change":"2024-04-16T16:13:24.922351+0000","last_active":"2024-04-16T16:13:26.955094+0000","last_peered":"2024-04-16T16:13:26.955094+0000","last_clean":"2024-04-16T16:13:26.955094+0000","last_became_active":"2024-04-16T16:13:24.922017+0000","last_became_peered":"2024-04-16T16:13:24.922017+0000","last_unstale":"2024-04-16T16:13:26.955094+0000","last_undegraded":"2024-04-16T16:13:26.955094+0000","last_fullsized":"2024-04-16T16:13:26.955094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:24.722256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942286+0000","last_change":"2024-04-16T16:13:25.944764+0000","last_active":"2024-04-16T16:13:26.942286+0000","last_peered":"2024-04-16T16:13:26.942286+0000","last_clean":"2024-04-16T16:13:26.942286+0000","last_became_active":"2024-04-16T16:13:25.944418+0000","last_became_peered":"2024-04-16T16:13:25.944418+0000","last_unstale":"2024-04-16T16:13:26.942286+0000","last_undegraded":"2024-04-16T16:13:26.942286+0000","last_fullsized":"2024-04-16T16:13:26.942286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:25.740193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960752+0000","last_change":"2024-04-16T16:13:24.916122+0000","last_active":"2024-04-16T16:13:26.960752+0000","last_peered":"2024-04-16T16:13:26.960752+0000","last_clean":"2024-04-16T16:13:26.960752+0000","last_became_active":"2024-04-16T16:13:24.915132+0000","last_became_peered":"2024-04-16T16:13:24.915132+0000","last_unstale":"2024-04-16T16:13:26.960752+0000","last_undegraded":"2024-04-16T16:13:26.960752+0000","last_fullsized":"2024-04-16T16:13:26.960752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942858+0000","last_change":"2024-04-16T16:13:25.954813+0000","last_active":"2024-04-16T16:13:26.942858+0000","last_peered":"2024-04-16T16:13:26.942858+0000","last_clean":"2024-04-16T16:13:26.942858+0000","last_became_active":"2024-04-16T16:13:25.952310+0000","last_became_peered":"2024-04-16T16:13:25.952310+0000","last_unstale":"2024-04-16T16:13:26.942858+0000","last_undegraded":"2024-04-16T16:13:26.942858+0000","last_fullsized":"2024-04-16T16:13:26.942858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:13:01.042309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.575148+0000","last_change":"2024-04-16T16:13:24.907364+0000","last_active":"2024-04-16T16:13:28.575148+0000","last_peered":"2024-04-16T16:13:28.575148+0000","last_clean":"2024-04-16T16:13:28.575148+0000","last_became_active":"2024-04-16T16:13:24.906972+0000","last_became_peered":"2024-04-16T16:13:24.906972+0000","last_unstale":"2024-04-16T16:13:28.575148+0000","last_undegraded":"2024-04-16T16:13:28.575148+0000","last_fullsized":"2024-04-16T16:13:28.575148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:22.075493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.572223+0000","last_change":"2024-04-16T16:13:25.946280+0000","last_active":"2024-04-16T16:13:28.572223+0000","last_peered":"2024-04-16T16:13:28.572223+0000","last_clean":"2024-04-16T16:13:28.572223+0000","last_became_active":"2024-04-16T16:13:25.945919+0000","last_became_peered":"2024-04-16T16:13:25.945919+0000","last_unstale":"2024-04-16T16:13:28.572223+0000","last_undegraded":"2024-04-16T16:13:28.572223+0000","last_fullsized":"2024-04-16T16:13:28.572223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:23.603931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971179+0000","last_change":"2024-04-16T16:13:24.921202+0000","last_active":"2024-04-16T16:13:26.971179+0000","last_peered":"2024-04-16T16:13:26.971179+0000","last_clean":"2024-04-16T16:13:26.971179+0000","last_became_active":"2024-04-16T16:13:24.920850+0000","last_became_peered":"2024-04-16T16:13:24.920850+0000","last_unstale":"2024-04-16T16:13:26.971179+0000","last_undegraded":"2024-04-16T16:13:26.971179+0000","last_fullsized":"2024-04-16T16:13:26.971179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942649+0000","last_change":"2024-04-16T16:13:25.947511+0000","last_active":"2024-04-16T16:13:26.942649+0000","last_peered":"2024-04-16T16:13:26.942649+0000","last_clean":"2024-04-16T16:13:26.942649+0000","last_became_active":"2024-04-16T16:13:25.947213+0000","last_became_peered":"2024-04-16T16:13:25.947213+0000","last_unstale":"2024-04-16T16:13:26.942649+0000","last_undegraded":"2024-04-16T16:13:26.942649+0000","last_fullsized":"2024-04-16T16:13:26.942649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:47.987635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960500+0000","last_change":"2024-04-16T16:13:24.923552+0000","last_active":"2024-04-16T16:13:26.960500+0000","last_peered":"2024-04-16T16:13:26.960500+0000","last_clean":"2024-04-16T16:13:26.960500+0000","last_became_active":"2024-04-16T16:13:24.922940+0000","last_became_peered":"2024-04-16T16:13:24.922940+0000","last_unstale":"2024-04-16T16:13:26.960500+0000","last_undegraded":"2024-04-16T16:13:26.960500+0000","last_fullsized":"2024-04-16T16:13:26.960500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960474+0000","last_change":"2024-04-16T16:13:25.954523+0000","last_active":"2024-04-16T16:13:26.960474+0000","last_peered":"2024-04-16T16:13:26.960474+0000","last_clean":"2024-04-16T16:13:26.960474+0000","last_became_active":"2024-04-16T16:13:25.954058+0000","last_became_peered":"2024-04-16T16:13:25.954058+0000","last_unstale":"2024-04-16T16:13:26.960474+0000","last_undegraded":"2024-04-16T16:13:26.960474+0000","last_fullsized":"2024-04-16T16:13:26.960474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:11:32.536402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936502+0000","last_change":"2024-04-16T16:13:24.904943+0000","last_active":"2024-04-16T16:13:26.936502+0000","last_peered":"2024-04-16T16:13:26.936502+0000","last_clean":"2024-04-16T16:13:26.936502+0000","last_became_active":"2024-04-16T16:13:24.904527+0000","last_became_peered":"2024-04-16T16:13:24.904527+0000","last_unstale":"2024-04-16T16:13:26.936502+0000","last_undegraded":"2024-04-16T16:13:26.936502+0000","last_fullsized":"2024-04-16T16:13:26.936502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:22.075493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942098+0000","last_change":"2024-04-16T16:13:25.937269+0000","last_active":"2024-04-16T16:13:26.942098+0000","last_peered":"2024-04-16T16:13:26.942098+0000","last_clean":"2024-04-16T16:13:26.942098+0000","last_became_active":"2024-04-16T16:13:25.936953+0000","last_became_peered":"2024-04-16T16:13:25.936953+0000","last_unstale":"2024-04-16T16:13:26.942098+0000","last_undegraded":"2024-04-16T16:13:26.942098+0000","last_fullsized":"2024-04-16T16:13:26.942098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:41.243756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941662+0000","last_change":"2024-04-16T16:13:24.909436+0000","last_active":"2024-04-16T16:13:26.941662+0000","last_peered":"2024-04-16T16:13:26.941662+0000","last_clean":"2024-04-16T16:13:26.941662+0000","last_became_active":"2024-04-16T16:13:24.908963+0000","last_became_peered":"2024-04-16T16:13:24.908963+0000","last_unstale":"2024-04-16T16:13:26.941662+0000","last_undegraded":"2024-04-16T16:13:26.941662+0000","last_fullsized":"2024-04-16T16:13:26.941662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:21.269919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943634+0000","last_change":"2024-04-16T16:13:25.939892+0000","last_active":"2024-04-16T16:13:26.943634+0000","last_peered":"2024-04-16T16:13:26.943634+0000","last_clean":"2024-04-16T16:13:26.943634+0000","last_became_active":"2024-04-16T16:13:25.939610+0000","last_became_peered":"2024-04-16T16:13:25.939610+0000","last_unstale":"2024-04-16T16:13:26.943634+0000","last_undegraded":"2024-04-16T16:13:26.943634+0000","last_fullsized":"2024-04-16T16:13:26.943634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:58.462911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.937081+0000","last_change":"2024-04-16T16:13:24.898787+0000","last_active":"2024-04-16T16:13:26.937081+0000","last_peered":"2024-04-16T16:13:26.937081+0000","last_clean":"2024-04-16T16:13:26.937081+0000","last_became_active":"2024-04-16T16:13:24.898427+0000","last_became_peered":"2024-04-16T16:13:24.898427+0000","last_unstale":"2024-04-16T16:13:26.937081+0000","last_undegraded":"2024-04-16T16:13:26.937081+0000","last_fullsized":"2024-04-16T16:13:26.937081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943828+0000","last_change":"2024-04-16T16:13:25.953661+0000","last_active":"2024-04-16T16:13:26.943828+0000","last_peered":"2024-04-16T16:13:26.943828+0000","last_clean":"2024-04-16T16:13:26.943828+0000","last_became_active":"2024-04-16T16:13:25.950243+0000","last_became_peered":"2024-04-16T16:13:25.950243+0000","last_unstale":"2024-04-16T16:13:26.943828+0000","last_undegraded":"2024-04-16T16:13:26.943828+0000","last_fullsized":"2024-04-16T16:13:26.943828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:08.801668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955224+0000","last_change":"2024-04-16T16:13:24.906785+0000","last_active":"2024-04-16T16:13:26.955224+0000","last_peered":"2024-04-16T16:13:26.955224+0000","last_clean":"2024-04-16T16:13:26.955224+0000","last_became_active":"2024-04-16T16:13:24.906478+0000","last_became_peered":"2024-04-16T16:13:24.906478+0000","last_unstale":"2024-04-16T16:13:26.955224+0000","last_undegraded":"2024-04-16T16:13:26.955224+0000","last_fullsized":"2024-04-16T16:13:26.955224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:41.549340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971400+0000","last_change":"2024-04-16T16:13:25.951170+0000","last_active":"2024-04-16T16:13:26.971400+0000","last_peered":"2024-04-16T16:13:26.971400+0000","last_clean":"2024-04-16T16:13:26.971400+0000","last_became_active":"2024-04-16T16:13:25.950423+0000","last_became_peered":"2024-04-16T16:13:25.950423+0000","last_unstale":"2024-04-16T16:13:26.971400+0000","last_undegraded":"2024-04-16T16:13:26.971400+0000","last_fullsized":"2024-04-16T16:13:26.971400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:13:01.042309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936858+0000","last_change":"2024-04-16T16:13:24.920097+0000","last_active":"2024-04-16T16:13:26.936858+0000","last_peered":"2024-04-16T16:13:26.936858+0000","last_clean":"2024-04-16T16:13:26.936858+0000","last_became_active":"2024-04-16T16:13:24.919848+0000","last_became_peered":"2024-04-16T16:13:24.919848+0000","last_unstale":"2024-04-16T16:13:26.936858+0000","last_undegraded":"2024-04-16T16:13:26.936858+0000","last_fullsized":"2024-04-16T16:13:26.936858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:46:08.208026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942819+0000","last_change":"2024-04-16T16:13:25.952599+0000","last_active":"2024-04-16T16:13:26.942819+0000","last_peered":"2024-04-16T16:13:26.942819+0000","last_clean":"2024-04-16T16:13:26.942819+0000","last_became_active":"2024-04-16T16:13:25.949217+0000","last_became_peered":"2024-04-16T16:13:25.949217+0000","last_unstale":"2024-04-16T16:13:26.942819+0000","last_undegraded":"2024-04-16T16:13:26.942819+0000","last_fullsized":"2024-04-16T16:13:26.942819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:39.592952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.940554+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:13:26.940554+0000","last_peered":"2024-04-16T16:13:26.940554+0000","last_clean":"2024-04-16T16:13:26.940554+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:13:26.940554+0000","last_undegraded":"2024-04-16T16:13:26.940554+0000","last_fullsized":"2024-04-16T16:13:26.940554+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970651+0000","last_change":"2024-04-16T16:13:24.919303+0000","last_active":"2024-04-16T16:13:26.970651+0000","last_peered":"2024-04-16T16:13:26.970651+0000","last_clean":"2024-04-16T16:13:26.970651+0000","last_became_active":"2024-04-16T16:13:24.918056+0000","last_became_peered":"2024-04-16T16:13:24.918056+0000","last_unstale":"2024-04-16T16:13:26.970651+0000","last_undegraded":"2024-04-16T16:13:26.970651+0000","last_fullsized":"2024-04-16T16:13:26.970651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.574008+0000","last_change":"2024-04-16T16:13:25.947227+0000","last_active":"2024-04-16T16:13:28.574008+0000","last_peered":"2024-04-16T16:13:28.574008+0000","last_clean":"2024-04-16T16:13:28.574008+0000","last_became_active":"2024-04-16T16:13:25.946951+0000","last_became_peered":"2024-04-16T16:13:25.946951+0000","last_unstale":"2024-04-16T16:13:28.574008+0000","last_undegraded":"2024-04-16T16:13:28.574008+0000","last_fullsized":"2024-04-16T16:13:28.574008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:47.987635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.940404+0000","last_change":"2024-04-16T16:13:24.912738+0000","last_active":"2024-04-16T16:13:26.940404+0000","last_peered":"2024-04-16T16:13:26.940404+0000","last_clean":"2024-04-16T16:13:26.940404+0000","last_became_active":"2024-04-16T16:13:24.912432+0000","last_became_peered":"2024-04-16T16:13:24.912432+0000","last_unstale":"2024-04-16T16:13:26.940404+0000","last_undegraded":"2024-04-16T16:13:26.940404+0000","last_fullsized":"2024-04-16T16:13:26.940404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.571983+0000","last_change":"2024-04-16T16:13:25.952202+0000","last_active":"2024-04-16T16:13:28.571983+0000","last_peered":"2024-04-16T16:13:28.571983+0000","last_clean":"2024-04-16T16:13:28.571983+0000","last_became_active":"2024-04-16T16:13:25.951932+0000","last_became_peered":"2024-04-16T16:13:25.951932+0000","last_unstale":"2024-04-16T16:13:28.571983+0000","last_undegraded":"2024-04-16T16:13:28.571983+0000","last_fullsized":"2024-04-16T16:13:28.571983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:38:46.469621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936531+0000","last_change":"2024-04-16T16:13:24.901093+0000","last_active":"2024-04-16T16:13:26.936531+0000","last_peered":"2024-04-16T16:13:26.936531+0000","last_clean":"2024-04-16T16:13:26.936531+0000","last_became_active":"2024-04-16T16:13:24.900772+0000","last_became_peered":"2024-04-16T16:13:24.900772+0000","last_unstale":"2024-04-16T16:13:26.936531+0000","last_undegraded":"2024-04-16T16:13:26.936531+0000","last_fullsized":"2024-04-16T16:13:26.936531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954397+0000","last_change":"2024-04-16T16:13:25.929184+0000","last_active":"2024-04-16T16:13:26.954397+0000","last_peered":"2024-04-16T16:13:26.954397+0000","last_clean":"2024-04-16T16:13:26.954397+0000","last_became_active":"2024-04-16T16:13:25.927325+0000","last_became_peered":"2024-04-16T16:13:25.927325+0000","last_unstale":"2024-04-16T16:13:26.954397+0000","last_undegraded":"2024-04-16T16:13:26.954397+0000","last_fullsized":"2024-04-16T16:13:26.954397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:11:32.536402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971426+0000","last_change":"2024-04-16T16:13:24.923465+0000","last_active":"2024-04-16T16:13:26.971426+0000","last_peered":"2024-04-16T16:13:26.971426+0000","last_clean":"2024-04-16T16:13:26.971426+0000","last_became_active":"2024-04-16T16:13:24.923145+0000","last_became_peered":"2024-04-16T16:13:24.923145+0000","last_unstale":"2024-04-16T16:13:26.971426+0000","last_undegraded":"2024-04-16T16:13:26.971426+0000","last_fullsized":"2024-04-16T16:13:26.971426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941524+0000","last_change":"2024-04-16T16:13:25.948469+0000","last_active":"2024-04-16T16:13:26.941524+0000","last_peered":"2024-04-16T16:13:26.941524+0000","last_clean":"2024-04-16T16:13:26.941524+0000","last_became_active":"2024-04-16T16:13:25.947992+0000","last_became_peered":"2024-04-16T16:13:25.947992+0000","last_unstale":"2024-04-16T16:13:26.941524+0000","last_undegraded":"2024-04-16T16:13:26.941524+0000","last_fullsized":"2024-04-16T16:13:26.941524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:00.025740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954768+0000","last_change":"2024-04-16T16:13:24.905663+0000","last_active":"2024-04-16T16:13:26.954768+0000","last_peered":"2024-04-16T16:13:26.954768+0000","last_clean":"2024-04-16T16:13:26.954768+0000","last_became_active":"2024-04-16T16:13:24.905306+0000","last_became_peered":"2024-04-16T16:13:24.905306+0000","last_unstale":"2024-04-16T16:13:26.954768+0000","last_undegraded":"2024-04-16T16:13:26.954768+0000","last_fullsized":"2024-04-16T16:13:26.954768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971908+0000","last_change":"2024-04-16T16:13:25.955137+0000","last_active":"2024-04-16T16:13:26.971908+0000","last_peered":"2024-04-16T16:13:26.971908+0000","last_clean":"2024-04-16T16:13:26.971908+0000","last_became_active":"2024-04-16T16:13:25.954852+0000","last_became_peered":"2024-04-16T16:13:25.954852+0000","last_unstale":"2024-04-16T16:13:26.971908+0000","last_undegraded":"2024-04-16T16:13:26.971908+0000","last_fullsized":"2024-04-16T16:13:26.971908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:58.462911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943455+0000","last_change":"2024-04-16T16:13:24.915400+0000","last_active":"2024-04-16T16:13:26.943455+0000","last_peered":"2024-04-16T16:13:26.943455+0000","last_clean":"2024-04-16T16:13:26.943455+0000","last_became_active":"2024-04-16T16:13:24.915096+0000","last_became_peered":"2024-04-16T16:13:24.915096+0000","last_unstale":"2024-04-16T16:13:26.943455+0000","last_undegraded":"2024-04-16T16:13:26.943455+0000","last_fullsized":"2024-04-16T16:13:26.943455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:41.549340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954886+0000","last_change":"2024-04-16T16:13:25.924316+0000","last_active":"2024-04-16T16:13:26.954886+0000","last_peered":"2024-04-16T16:13:26.954886+0000","last_clean":"2024-04-16T16:13:26.954886+0000","last_became_active":"2024-04-16T16:13:25.923986+0000","last_became_peered":"2024-04-16T16:13:25.923986+0000","last_unstale":"2024-04-16T16:13:26.954886+0000","last_undegraded":"2024-04-16T16:13:26.954886+0000","last_fullsized":"2024-04-16T16:13:26.954886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:40:58.670474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.958843+0000","last_change":"2024-04-16T16:13:24.920876+0000","last_active":"2024-04-16T16:13:26.958843+0000","last_peered":"2024-04-16T16:13:26.958843+0000","last_clean":"2024-04-16T16:13:26.958843+0000","last_became_active":"2024-04-16T16:13:24.920409+0000","last_became_peered":"2024-04-16T16:13:24.920409+0000","last_unstale":"2024-04-16T16:13:26.958843+0000","last_undegraded":"2024-04-16T16:13:26.958843+0000","last_fullsized":"2024-04-16T16:13:26.958843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.573941+0000","last_change":"2024-04-16T16:13:25.941909+0000","last_active":"2024-04-16T16:13:28.573941+0000","last_peered":"2024-04-16T16:13:28.573941+0000","last_clean":"2024-04-16T16:13:28.573941+0000","last_became_active":"2024-04-16T16:13:25.941614+0000","last_became_peered":"2024-04-16T16:13:25.941614+0000","last_unstale":"2024-04-16T16:13:28.573941+0000","last_undegraded":"2024-04-16T16:13:28.573941+0000","last_fullsized":"2024-04-16T16:13:28.573941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:41.243756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961615+0000","last_change":"2024-04-16T16:13:24.918600+0000","last_active":"2024-04-16T16:13:26.961615+0000","last_peered":"2024-04-16T16:13:26.961615+0000","last_clean":"2024-04-16T16:13:26.961615+0000","last_became_active":"2024-04-16T16:13:24.918273+0000","last_became_peered":"2024-04-16T16:13:24.918273+0000","last_unstale":"2024-04-16T16:13:26.961615+0000","last_undegraded":"2024-04-16T16:13:26.961615+0000","last_fullsized":"2024-04-16T16:13:26.961615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.575341+0000","last_change":"2024-04-16T16:13:25.937881+0000","last_active":"2024-04-16T16:13:28.575341+0000","last_peered":"2024-04-16T16:13:28.575341+0000","last_clean":"2024-04-16T16:13:28.575341+0000","last_became_active":"2024-04-16T16:13:25.936555+0000","last_became_peered":"2024-04-16T16:13:25.936555+0000","last_unstale":"2024-04-16T16:13:28.575341+0000","last_undegraded":"2024-04-16T16:13:28.575341+0000","last_fullsized":"2024-04-16T16:13:28.575341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.940400+0000","last_change":"2024-04-16T16:13:24.915314+0000","last_active":"2024-04-16T16:13:26.940400+0000","last_peered":"2024-04-16T16:13:26.940400+0000","last_clean":"2024-04-16T16:13:26.940400+0000","last_became_active":"2024-04-16T16:13:24.913831+0000","last_became_peered":"2024-04-16T16:13:24.913831+0000","last_unstale":"2024-04-16T16:13:26.940400+0000","last_undegraded":"2024-04-16T16:13:26.940400+0000","last_fullsized":"2024-04-16T16:13:26.940400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941863+0000","last_change":"2024-04-16T16:13:25.939017+0000","last_active":"2024-04-16T16:13:26.941863+0000","last_peered":"2024-04-16T16:13:26.941863+0000","last_clean":"2024-04-16T16:13:26.941863+0000","last_became_active":"2024-04-16T16:13:25.936578+0000","last_became_peered":"2024-04-16T16:13:25.936578+0000","last_unstale":"2024-04-16T16:13:26.941863+0000","last_undegraded":"2024-04-16T16:13:26.941863+0000","last_fullsized":"2024-04-16T16:13:26.941863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:12.895579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954750+0000","last_change":"2024-04-16T16:13:24.909354+0000","last_active":"2024-04-16T16:13:26.954750+0000","last_peered":"2024-04-16T16:13:26.954750+0000","last_clean":"2024-04-16T16:13:26.954750+0000","last_became_active":"2024-04-16T16:13:24.909073+0000","last_became_peered":"2024-04-16T16:13:24.909073+0000","last_unstale":"2024-04-16T16:13:26.954750+0000","last_undegraded":"2024-04-16T16:13:26.954750+0000","last_fullsized":"2024-04-16T16:13:26.954750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.572771+0000","last_change":"2024-04-16T16:13:25.940225+0000","last_active":"2024-04-16T16:13:28.572771+0000","last_peered":"2024-04-16T16:13:28.572771+0000","last_clean":"2024-04-16T16:13:28.572771+0000","last_became_active":"2024-04-16T16:13:25.939871+0000","last_became_peered":"2024-04-16T16:13:25.939871+0000","last_unstale":"2024-04-16T16:13:28.572771+0000","last_undegraded":"2024-04-16T16:13:28.572771+0000","last_fullsized":"2024-04-16T16:13:28.572771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:10.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954700+0000","last_change":"2024-04-16T16:13:24.907229+0000","last_active":"2024-04-16T16:13:26.954700+0000","last_peered":"2024-04-16T16:13:26.954700+0000","last_clean":"2024-04-16T16:13:26.954700+0000","last_became_active":"2024-04-16T16:13:24.906948+0000","last_became_peered":"2024-04-16T16:13:24.906948+0000","last_unstale":"2024-04-16T16:13:26.954700+0000","last_undegraded":"2024-04-16T16:13:26.954700+0000","last_fullsized":"2024-04-16T16:13:26.954700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:11.877642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.573289+0000","last_change":"2024-04-16T16:13:25.963968+0000","last_active":"2024-04-16T16:13:28.573289+0000","last_peered":"2024-04-16T16:13:28.573289+0000","last_clean":"2024-04-16T16:13:28.573289+0000","last_became_active":"2024-04-16T16:13:25.963699+0000","last_became_peered":"2024-04-16T16:13:25.963699+0000","last_unstale":"2024-04-16T16:13:28.573289+0000","last_undegraded":"2024-04-16T16:13:28.573289+0000","last_fullsized":"2024-04-16T16:13:28.573289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942898+0000","last_change":"2024-04-16T16:13:24.924622+0000","last_active":"2024-04-16T16:13:26.942898+0000","last_peered":"2024-04-16T16:13:26.942898+0000","last_clean":"2024-04-16T16:13:26.942898+0000","last_became_active":"2024-04-16T16:13:24.923313+0000","last_became_peered":"2024-04-16T16:13:24.923313+0000","last_unstale":"2024-04-16T16:13:26.942898+0000","last_undegraded":"2024-04-16T16:13:26.942898+0000","last_fullsized":"2024-04-16T16:13:26.942898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960479+0000","last_change":"2024-04-16T16:13:25.938982+0000","last_active":"2024-04-16T16:13:26.960479+0000","last_peered":"2024-04-16T16:13:26.960479+0000","last_clean":"2024-04-16T16:13:26.960479+0000","last_became_active":"2024-04-16T16:13:25.938694+0000","last_became_peered":"2024-04-16T16:13:25.938694+0000","last_unstale":"2024-04-16T16:13:26.960479+0000","last_undegraded":"2024-04-16T16:13:26.960479+0000","last_fullsized":"2024-04-16T16:13:26.960479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:10.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971847+0000","last_change":"2024-04-16T16:13:25.944450+0000","last_active":"2024-04-16T16:13:26.971847+0000","last_peered":"2024-04-16T16:13:26.971847+0000","last_clean":"2024-04-16T16:13:26.971847+0000","last_became_active":"2024-04-16T16:13:25.941384+0000","last_became_peered":"2024-04-16T16:13:25.941384+0000","last_unstale":"2024-04-16T16:13:26.971847+0000","last_undegraded":"2024-04-16T16:13:26.971847+0000","last_fullsized":"2024-04-16T16:13:26.971847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:40:10.575471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942812+0000","last_change":"2024-04-16T16:13:24.925176+0000","last_active":"2024-04-16T16:13:26.942812+0000","last_peered":"2024-04-16T16:13:26.942812+0000","last_clean":"2024-04-16T16:13:26.942812+0000","last_became_active":"2024-04-16T16:13:24.924863+0000","last_became_peered":"2024-04-16T16:13:24.924863+0000","last_unstale":"2024-04-16T16:13:26.942812+0000","last_undegraded":"2024-04-16T16:13:26.942812+0000","last_fullsized":"2024-04-16T16:13:26.942812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:11.877642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971280+0000","last_change":"2024-04-16T16:13:25.944098+0000","last_active":"2024-04-16T16:13:26.971280+0000","last_peered":"2024-04-16T16:13:26.971280+0000","last_clean":"2024-04-16T16:13:26.971280+0000","last_became_active":"2024-04-16T16:13:25.943843+0000","last_became_peered":"2024-04-16T16:13:25.943843+0000","last_unstale":"2024-04-16T16:13:26.971280+0000","last_undegraded":"2024-04-16T16:13:26.971280+0000","last_fullsized":"2024-04-16T16:13:26.971280+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:39.592952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943140+0000","last_change":"2024-04-16T16:13:24.923704+0000","last_active":"2024-04-16T16:13:26.943140+0000","last_peered":"2024-04-16T16:13:26.943140+0000","last_clean":"2024-04-16T16:13:26.943140+0000","last_became_active":"2024-04-16T16:13:24.923214+0000","last_became_peered":"2024-04-16T16:13:24.923214+0000","last_unstale":"2024-04-16T16:13:26.943140+0000","last_undegraded":"2024-04-16T16:13:26.943140+0000","last_fullsized":"2024-04-16T16:13:26.943140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:24.722256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.574982+0000","last_change":"2024-04-16T16:13:25.955067+0000","last_active":"2024-04-16T16:13:28.574982+0000","last_peered":"2024-04-16T16:13:28.574982+0000","last_clean":"2024-04-16T16:13:28.574982+0000","last_became_active":"2024-04-16T16:13:25.954780+0000","last_became_peered":"2024-04-16T16:13:25.954780+0000","last_unstale":"2024-04-16T16:13:28.574982+0000","last_undegraded":"2024-04-16T16:13:28.574982+0000","last_fullsized":"2024-04-16T16:13:28.574982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:55.948136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.574102+0000","last_change":"2024-04-16T16:13:24.913036+0000","last_active":"2024-04-16T16:13:28.574102+0000","last_peered":"2024-04-16T16:13:28.574102+0000","last_clean":"2024-04-16T16:13:28.574102+0000","last_became_active":"2024-04-16T16:13:24.911928+0000","last_became_peered":"2024-04-16T16:13:24.911928+0000","last_unstale":"2024-04-16T16:13:28.574102+0000","last_undegraded":"2024-04-16T16:13:28.574102+0000","last_fullsized":"2024-04-16T16:13:28.574102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.573997+0000","last_change":"2024-04-16T16:13:25.946331+0000","last_active":"2024-04-16T16:13:28.573997+0000","last_peered":"2024-04-16T16:13:28.573997+0000","last_clean":"2024-04-16T16:13:28.573997+0000","last_became_active":"2024-04-16T16:13:25.946053+0000","last_became_peered":"2024-04-16T16:13:25.946053+0000","last_unstale":"2024-04-16T16:13:28.573997+0000","last_undegraded":"2024-04-16T16:13:28.573997+0000","last_fullsized":"2024-04-16T16:13:28.573997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:30:58.937717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.575420+0000","last_change":"2024-04-16T16:13:24.909349+0000","last_active":"2024-04-16T16:13:28.575420+0000","last_peered":"2024-04-16T16:13:28.575420+0000","last_clean":"2024-04-16T16:13:28.575420+0000","last_became_active":"2024-04-16T16:13:24.909069+0000","last_became_peered":"2024-04-16T16:13:24.909069+0000","last_unstale":"2024-04-16T16:13:28.575420+0000","last_undegraded":"2024-04-16T16:13:28.575420+0000","last_fullsized":"2024-04-16T16:13:28.575420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:46:08.208026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:13:36.214 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:13:36.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.475+0000 7fadebe00640 1 Processor -- start 2024-04-16T16:13:36.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.476+0000 7fadebe00640 1 -- start start 2024-04-16T16:13:36.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.477+0000 7fadebe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fadec161e90 0x7fadec0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:36.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.477+0000 7fadebe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fadec162270 0x7fadec0d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:36.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.477+0000 7fadebe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadec0d3080 con 0x7fadec0d52d0 2024-04-16T16:13:36.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.477+0000 7fadebe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fadec0d2e30 con 0x7fadec162270 2024-04-16T16:13:36.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.477+0000 7fadebe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fadec0d3360 con 0x7fadec161e90 2024-04-16T16:13:36.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.478+0000 7fadeb400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fadec0d52d0 0x7fadec06ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39080/0 (socket says 172.21.15.45:39080) 2024-04-16T16:13:36.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.478+0000 7fadeb400640 1 -- 172.21.15.45:0/3414916172 learned_addr learned my addr 172.21.15.45:0/3414916172 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:36.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.478+0000 7fade2000640 1 --2- 172.21.15.45:0/3414916172 >> v2:172.21.15.45:3300/0 conn(0x7fadec162270 0x7fadec0d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:36.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.478+0000 7fadeaa00640 1 --2- 172.21.15.45:0/3414916172 >> v2:172.21.15.103:3300/0 conn(0x7fadec161e90 0x7fadec0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:36.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.478+0000 7fade2000640 1 -- 172.21.15.45:0/3414916172 >> v2:172.21.15.103:3300/0 conn(0x7fadec161e90 msgr2=0x7fadec0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:36.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.478+0000 7fade2000640 1 --2- 172.21.15.45:0/3414916172 >> v2:172.21.15.103:3300/0 conn(0x7fadec161e90 0x7fadec0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:36.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.478+0000 7fade2000640 1 -- 172.21.15.45:0/3414916172 >> v1:172.21.15.45:6789/0 conn(0x7fadec0d52d0 legacy=0x7fadec06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:36.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.478+0000 7fade2000640 1 -- 172.21.15.45:0/3414916172 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadec0d3640 con 0x7fadec162270 2024-04-16T16:13:36.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.480+0000 7fade2000640 1 --2- 172.21.15.45:0/3414916172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadec162270 0x7fadec0d3e40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fadd40097a0 tx=0x7fadd402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a7a43b537ffdcf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:36.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.480+0000 7fadea000640 1 -- 172.21.15.45:0/3414916172 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd4035020 con 0x7fadec162270 2024-04-16T16:13:36.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.480+0000 7fadea000640 1 -- 172.21.15.45:0/3414916172 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadd4031830 con 0x7fadec162270 2024-04-16T16:13:36.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.480+0000 7fadea000640 1 -- 172.21.15.45:0/3414916172 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd4031c00 con 0x7fadec162270 2024-04-16T16:13:36.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.481+0000 7fadebe00640 1 -- 172.21.15.45:0/3414916172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadec162270 msgr2=0x7fadec0d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:36.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.481+0000 7fadebe00640 1 --2- 172.21.15.45:0/3414916172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadec162270 0x7fadec0d3e40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fadd40097a0 tx=0x7fadd402b8d0 comp rx=0 tx=0).stop 2024-04-16T16:13:36.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.481+0000 7fadebe00640 1 -- 172.21.15.45:0/3414916172 shutdown_connections 2024-04-16T16:13:36.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.481+0000 7fadebe00640 1 --2- 172.21.15.45:0/3414916172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadec162270 0x7fadec0d3e40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:36.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.481+0000 7fadebe00640 1 --2- 172.21.15.45:0/3414916172 >> v2:172.21.15.103:3300/0 conn(0x7fadec161e90 0x7fadec0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:36.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.481+0000 7fadebe00640 1 -- 172.21.15.45:0/3414916172 >> 172.21.15.45:0/3414916172 conn(0x7fadec0d1940 msgr2=0x7fadec0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:36.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.481+0000 7fadebe00640 1 -- 172.21.15.45:0/3414916172 shutdown_connections 2024-04-16T16:13:36.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.481+0000 7fadebe00640 1 -- 172.21.15.45:0/3414916172 wait complete. 2024-04-16T16:13:36.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.483+0000 7fadebe00640 1 Processor -- start 2024-04-16T16:13:36.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.483+0000 7fadebe00640 1 -- start start 2024-04-16T16:13:36.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.485+0000 7fadebe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadec0d52d0 0x7fadec1f7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:36.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.485+0000 7fadebe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec161e90 0x7fadec0c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:36.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.485+0000 7fadebe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadec162270 0x7fadec0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:36.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.485+0000 7fadebe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fadec06d550 con 0x7fadec162270 2024-04-16T16:13:36.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.485+0000 7fadebe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fadec0cfd10 con 0x7fadec161e90 2024-04-16T16:13:36.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.485+0000 7fadebe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fadec06d6d0 con 0x7fadec0d52d0 2024-04-16T16:13:36.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.485+0000 7fadeb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadec162270 0x7fadec0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:36.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.485+0000 7fadeb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadec162270 0x7fadec0c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43378/0 (socket says 172.21.15.45:43378) 2024-04-16T16:13:36.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.485+0000 7fadeb400640 1 -- 172.21.15.45:0/217821345 learned_addr learned my addr 172.21.15.45:0/217821345 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:36.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.485+0000 7fade2000640 1 --2- 172.21.15.45:0/217821345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec161e90 0x7fadec0c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:36.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.485+0000 7fadeaa00640 1 --2- 172.21.15.45:0/217821345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadec0d52d0 0x7fadec1f7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:36.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.486+0000 7fadeb400640 1 -- 172.21.15.45:0/217821345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadec0d52d0 msgr2=0x7fadec1f7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:36.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.486+0000 7fadeb400640 1 --2- 172.21.15.45:0/217821345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadec0d52d0 0x7fadec1f7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:36.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.486+0000 7fadeb400640 1 -- 172.21.15.45:0/217821345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec161e90 msgr2=0x7fadec0c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:36.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.486+0000 7fadeb400640 1 --2- 172.21.15.45:0/217821345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec161e90 0x7fadec0c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:36.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.486+0000 7fadeb400640 1 -- 172.21.15.45:0/217821345 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadec0c93e0 con 0x7fadec162270 2024-04-16T16:13:36.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.486+0000 7fadeaa00640 1 --2- 172.21.15.45:0/217821345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadec0d52d0 0x7fadec1f7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:13:36.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.486+0000 7fadeb400640 1 --2- 172.21.15.45:0/217821345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadec162270 0x7fadec0c9a90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fade400c340 tx=0x7fade400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:36.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.486+0000 7fade3400640 1 -- 172.21.15.45:0/217821345 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade4017070 con 0x7fadec162270 2024-04-16T16:13:36.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.486+0000 7fade3400640 1 -- 172.21.15.45:0/217821345 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fade4005ce0 con 0x7fadec162270 2024-04-16T16:13:36.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.486+0000 7fade3400640 1 -- 172.21.15.45:0/217821345 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade40072d0 con 0x7fadec162270 2024-04-16T16:13:36.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.486+0000 7fadebe00640 1 -- 172.21.15.45:0/217821345 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadec0c96c0 con 0x7fadec162270 2024-04-16T16:13:36.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.487+0000 7fadebe00640 1 -- 172.21.15.45:0/217821345 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadec15ca50 con 0x7fadec162270 2024-04-16T16:13:36.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.488+0000 7fadebe00640 1 -- 172.21.15.45:0/217821345 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadec075780 con 0x7fadec162270 2024-04-16T16:13:36.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.489+0000 7fade3400640 1 -- 172.21.15.45:0/217821345 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fade401a070 con 0x7fadec162270 2024-04-16T16:13:36.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.490+0000 7fade3400640 1 --2- 172.21.15.45:0/217821345 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fadc806c8f0 0x7fadc806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:36.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.490+0000 7fade3400640 1 -- 172.21.15.45:0/217821345 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fade408ad50 con 0x7fadec162270 2024-04-16T16:13:36.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.490+0000 7fadeaa00640 1 --2- 172.21.15.45:0/217821345 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fadc806c8f0 0x7fadc806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:36.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.492+0000 7fadeaa00640 1 --2- 172.21.15.45:0/217821345 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fadc806c8f0 0x7fadc806edb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faddc009b20 tx=0x7faddc0039d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:36.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.494+0000 7fade3400640 1 -- 172.21.15.45:0/217821345 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fade4058b20 con 0x7fadec162270 2024-04-16T16:13:36.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.686+0000 7fadebe00640 1 -- 172.21.15.45:0/217821345 --> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fadec06dae0 con 0x7fadc806c8f0 2024-04-16T16:13:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.704+0000 7fade3400640 1 -- 172.21.15.45:0/217821345 <== mgr.4105 v2:172.21.15.103:6832/2711977587 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251838 (secure 0 0 0) 0x7fadec06dae0 con 0x7fadc806c8f0 2024-04-16T16:13:36.707 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:36.709 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:13:36.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.709+0000 7fadebe00640 1 -- 172.21.15.45:0/217821345 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fadc806c8f0 msgr2=0x7fadc806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:36.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.709+0000 7fadebe00640 1 --2- 172.21.15.45:0/217821345 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fadc806c8f0 0x7fadc806edb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faddc009b20 tx=0x7faddc0039d0 comp rx=0 tx=0).stop 2024-04-16T16:13:36.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.709+0000 7fadebe00640 1 -- 172.21.15.45:0/217821345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadec162270 msgr2=0x7fadec0c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:36.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.709+0000 7fadebe00640 1 --2- 172.21.15.45:0/217821345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadec162270 0x7fadec0c9a90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fade400c340 tx=0x7fade400c800 comp rx=0 tx=0).stop 2024-04-16T16:13:36.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.710+0000 7fadebe00640 1 -- 172.21.15.45:0/217821345 shutdown_connections 2024-04-16T16:13:36.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.710+0000 7fadebe00640 1 --2- 172.21.15.45:0/217821345 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fadc806c8f0 0x7fadc806edb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:36.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.710+0000 7fadebe00640 1 --2- 172.21.15.45:0/217821345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadec162270 0x7fadec0c9a90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:36.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.710+0000 7fadebe00640 1 --2- 172.21.15.45:0/217821345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec161e90 0x7fadec0c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:36.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.710+0000 7fadebe00640 1 --2- 172.21.15.45:0/217821345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadec0d52d0 0x7fadec1f7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:36.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.710+0000 7fadebe00640 1 -- 172.21.15.45:0/217821345 >> 172.21.15.45:0/217821345 conn(0x7fadec0d1940 msgr2=0x7fadec0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:36.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.710+0000 7fadebe00640 1 -- 172.21.15.45:0/217821345 shutdown_connections 2024-04-16T16:13:36.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.710+0000 7fadebe00640 1 -- 172.21.15.45:0/217821345 wait complete. 2024-04-16T16:13:36.727 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-16T16:13:35.095120+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235756,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738964,"statfs":{"total":773094113280,"available":772852699136,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.191477"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943618+0000","last_change":"2024-04-16T16:13:25.934978+0000","last_active":"2024-04-16T16:13:26.943618+0000","last_peered":"2024-04-16T16:13:26.943618+0000","last_clean":"2024-04-16T16:13:26.943618+0000","last_became_active":"2024-04-16T16:13:25.934469+0000","last_became_peered":"2024-04-16T16:13:25.934469+0000","last_unstale":"2024-04-16T16:13:26.943618+0000","last_undegraded":"2024-04-16T16:13:26.943618+0000","last_fullsized":"2024-04-16T16:13:26.943618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:40:10.575471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955303+0000","last_change":"2024-04-16T16:13:25.936302+0000","last_active":"2024-04-16T16:13:26.955303+0000","last_peered":"2024-04-16T16:13:26.955303+0000","last_clean":"2024-04-16T16:13:26.955303+0000","last_became_active":"2024-04-16T16:13:25.935907+0000","last_became_peered":"2024-04-16T16:13:25.935907+0000","last_unstale":"2024-04-16T16:13:26.955303+0000","last_undegraded":"2024-04-16T16:13:26.955303+0000","last_fullsized":"2024-04-16T16:13:26.955303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.574975+0000","last_change":"2024-04-16T16:13:25.935281+0000","last_active":"2024-04-16T16:13:28.574975+0000","last_peered":"2024-04-16T16:13:28.574975+0000","last_clean":"2024-04-16T16:13:28.574975+0000","last_became_active":"2024-04-16T16:13:25.934986+0000","last_became_peered":"2024-04-16T16:13:25.934986+0000","last_unstale":"2024-04-16T16:13:28.574975+0000","last_undegraded":"2024-04-16T16:13:28.574975+0000","last_fullsized":"2024-04-16T16:13:28.574975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:10.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970937+0000","last_change":"2024-04-16T16:13:25.952485+0000","last_active":"2024-04-16T16:13:26.970937+0000","last_peered":"2024-04-16T16:13:26.970937+0000","last_clean":"2024-04-16T16:13:26.970937+0000","last_became_active":"2024-04-16T16:13:25.952189+0000","last_became_peered":"2024-04-16T16:13:25.952189+0000","last_unstale":"2024-04-16T16:13:26.970937+0000","last_undegraded":"2024-04-16T16:13:26.970937+0000","last_fullsized":"2024-04-16T16:13:26.970937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:47.987635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942605+0000","last_change":"2024-04-16T16:13:25.930237+0000","last_active":"2024-04-16T16:13:26.942605+0000","last_peered":"2024-04-16T16:13:26.942605+0000","last_clean":"2024-04-16T16:13:26.942605+0000","last_became_active":"2024-04-16T16:13:25.929635+0000","last_became_peered":"2024-04-16T16:13:25.929635+0000","last_unstale":"2024-04-16T16:13:26.942605+0000","last_undegraded":"2024-04-16T16:13:26.942605+0000","last_fullsized":"2024-04-16T16:13:26.942605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:30:58.937717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970737+0000","last_change":"2024-04-16T16:13:25.947705+0000","last_active":"2024-04-16T16:13:26.970737+0000","last_peered":"2024-04-16T16:13:26.970737+0000","last_clean":"2024-04-16T16:13:26.970737+0000","last_became_active":"2024-04-16T16:13:25.947433+0000","last_became_peered":"2024-04-16T16:13:25.947433+0000","last_unstale":"2024-04-16T16:13:26.970737+0000","last_undegraded":"2024-04-16T16:13:26.970737+0000","last_fullsized":"2024-04-16T16:13:26.970737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941969+0000","last_change":"2024-04-16T16:13:25.930905+0000","last_active":"2024-04-16T16:13:26.941969+0000","last_peered":"2024-04-16T16:13:26.941969+0000","last_clean":"2024-04-16T16:13:26.941969+0000","last_became_active":"2024-04-16T16:13:25.929782+0000","last_became_peered":"2024-04-16T16:13:25.929782+0000","last_unstale":"2024-04-16T16:13:26.941969+0000","last_undegraded":"2024-04-16T16:13:26.941969+0000","last_fullsized":"2024-04-16T16:13:26.941969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.572328+0000","last_change":"2024-04-16T16:13:25.947751+0000","last_active":"2024-04-16T16:13:28.572328+0000","last_peered":"2024-04-16T16:13:28.572328+0000","last_clean":"2024-04-16T16:13:28.572328+0000","last_became_active":"2024-04-16T16:13:25.947110+0000","last_became_peered":"2024-04-16T16:13:25.947110+0000","last_unstale":"2024-04-16T16:13:28.572328+0000","last_undegraded":"2024-04-16T16:13:28.572328+0000","last_fullsized":"2024-04-16T16:13:28.572328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:51:20.451559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943813+0000","last_change":"2024-04-16T16:13:25.950917+0000","last_active":"2024-04-16T16:13:26.943813+0000","last_peered":"2024-04-16T16:13:26.943813+0000","last_clean":"2024-04-16T16:13:26.943813+0000","last_became_active":"2024-04-16T16:13:25.946716+0000","last_became_peered":"2024-04-16T16:13:25.946716+0000","last_unstale":"2024-04-16T16:13:26.943813+0000","last_undegraded":"2024-04-16T16:13:26.943813+0000","last_fullsized":"2024-04-16T16:13:26.943813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:52:47.172342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961550+0000","last_change":"2024-04-16T16:13:25.941148+0000","last_active":"2024-04-16T16:13:26.961550+0000","last_peered":"2024-04-16T16:13:26.961550+0000","last_clean":"2024-04-16T16:13:26.961550+0000","last_became_active":"2024-04-16T16:13:25.940818+0000","last_became_peered":"2024-04-16T16:13:25.940818+0000","last_unstale":"2024-04-16T16:13:26.961550+0000","last_undegraded":"2024-04-16T16:13:26.961550+0000","last_fullsized":"2024-04-16T16:13:26.961550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960975+0000","last_change":"2024-04-16T16:13:25.943177+0000","last_active":"2024-04-16T16:13:26.960975+0000","last_peered":"2024-04-16T16:13:26.960975+0000","last_clean":"2024-04-16T16:13:26.960975+0000","last_became_active":"2024-04-16T16:13:25.942902+0000","last_became_peered":"2024-04-16T16:13:25.942902+0000","last_unstale":"2024-04-16T16:13:26.960975+0000","last_undegraded":"2024-04-16T16:13:26.960975+0000","last_fullsized":"2024-04-16T16:13:26.960975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:40:58.670474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941834+0000","last_change":"2024-04-16T16:13:25.938811+0000","last_active":"2024-04-16T16:13:26.941834+0000","last_peered":"2024-04-16T16:13:26.941834+0000","last_clean":"2024-04-16T16:13:26.941834+0000","last_became_active":"2024-04-16T16:13:25.938479+0000","last_became_peered":"2024-04-16T16:13:25.938479+0000","last_unstale":"2024-04-16T16:13:26.941834+0000","last_undegraded":"2024-04-16T16:13:26.941834+0000","last_fullsized":"2024-04-16T16:13:26.941834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:41.335228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.573197+0000","last_change":"2024-04-16T16:13:25.950387+0000","last_active":"2024-04-16T16:13:28.573197+0000","last_peered":"2024-04-16T16:13:28.573197+0000","last_clean":"2024-04-16T16:13:28.573197+0000","last_became_active":"2024-04-16T16:13:25.949164+0000","last_became_peered":"2024-04-16T16:13:25.949164+0000","last_unstale":"2024-04-16T16:13:28.573197+0000","last_undegraded":"2024-04-16T16:13:28.573197+0000","last_fullsized":"2024-04-16T16:13:28.573197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:40:58.670474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.572204+0000","last_change":"2024-04-16T16:13:25.939438+0000","last_active":"2024-04-16T16:13:28.572204+0000","last_peered":"2024-04-16T16:13:28.572204+0000","last_clean":"2024-04-16T16:13:28.572204+0000","last_became_active":"2024-04-16T16:13:25.939066+0000","last_became_peered":"2024-04-16T16:13:25.939066+0000","last_unstale":"2024-04-16T16:13:28.572204+0000","last_undegraded":"2024-04-16T16:13:28.572204+0000","last_fullsized":"2024-04-16T16:13:28.572204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:07.860908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954570+0000","last_change":"2024-04-16T16:13:25.937157+0000","last_active":"2024-04-16T16:13:26.954570+0000","last_peered":"2024-04-16T16:13:26.954570+0000","last_clean":"2024-04-16T16:13:26.954570+0000","last_became_active":"2024-04-16T16:13:25.936851+0000","last_became_peered":"2024-04-16T16:13:25.936851+0000","last_unstale":"2024-04-16T16:13:26.954570+0000","last_undegraded":"2024-04-16T16:13:26.954570+0000","last_fullsized":"2024-04-16T16:13:26.954570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:55.948136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970498+0000","last_change":"2024-04-16T16:13:25.949240+0000","last_active":"2024-04-16T16:13:26.970498+0000","last_peered":"2024-04-16T16:13:26.970498+0000","last_clean":"2024-04-16T16:13:26.970498+0000","last_became_active":"2024-04-16T16:13:25.948913+0000","last_became_peered":"2024-04-16T16:13:25.948913+0000","last_unstale":"2024-04-16T16:13:26.970498+0000","last_undegraded":"2024-04-16T16:13:26.970498+0000","last_fullsized":"2024-04-16T16:13:26.970498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.972475+0000","last_change":"2024-04-16T16:13:25.947113+0000","last_active":"2024-04-16T16:13:26.972475+0000","last_peered":"2024-04-16T16:13:26.972475+0000","last_clean":"2024-04-16T16:13:26.972475+0000","last_became_active":"2024-04-16T16:13:25.943174+0000","last_became_peered":"2024-04-16T16:13:25.943174+0000","last_unstale":"2024-04-16T16:13:26.972475+0000","last_undegraded":"2024-04-16T16:13:26.972475+0000","last_fullsized":"2024-04-16T16:13:26.972475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:08.801668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.937163+0000","last_change":"2024-04-16T16:13:25.942311+0000","last_active":"2024-04-16T16:13:26.937163+0000","last_peered":"2024-04-16T16:13:26.937163+0000","last_clean":"2024-04-16T16:13:26.937163+0000","last_became_active":"2024-04-16T16:13:25.942015+0000","last_became_peered":"2024-04-16T16:13:25.942015+0000","last_unstale":"2024-04-16T16:13:26.937163+0000","last_undegraded":"2024-04-16T16:13:26.937163+0000","last_fullsized":"2024-04-16T16:13:26.937163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:25.740193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943340+0000","last_change":"2024-04-16T16:13:25.929976+0000","last_active":"2024-04-16T16:13:26.943340+0000","last_peered":"2024-04-16T16:13:26.943340+0000","last_clean":"2024-04-16T16:13:26.943340+0000","last_became_active":"2024-04-16T16:13:25.929347+0000","last_became_peered":"2024-04-16T16:13:25.929347+0000","last_unstale":"2024-04-16T16:13:26.943340+0000","last_undegraded":"2024-04-16T16:13:26.943340+0000","last_fullsized":"2024-04-16T16:13:26.943340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:19:05.123994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960773+0000","last_change":"2024-04-16T16:13:25.945575+0000","last_active":"2024-04-16T16:13:26.960773+0000","last_peered":"2024-04-16T16:13:26.960773+0000","last_clean":"2024-04-16T16:13:26.960773+0000","last_became_active":"2024-04-16T16:13:25.945217+0000","last_became_peered":"2024-04-16T16:13:25.945217+0000","last_unstale":"2024-04-16T16:13:26.960773+0000","last_undegraded":"2024-04-16T16:13:26.960773+0000","last_fullsized":"2024-04-16T16:13:26.960773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:55.948136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.574655+0000","last_change":"2024-04-16T16:13:25.945082+0000","last_active":"2024-04-16T16:13:28.574655+0000","last_peered":"2024-04-16T16:13:28.574655+0000","last_clean":"2024-04-16T16:13:28.574655+0000","last_became_active":"2024-04-16T16:13:25.944781+0000","last_became_peered":"2024-04-16T16:13:25.944781+0000","last_unstale":"2024-04-16T16:13:28.574655+0000","last_undegraded":"2024-04-16T16:13:28.574655+0000","last_fullsized":"2024-04-16T16:13:28.574655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:51:20.451559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970886+0000","last_change":"2024-04-16T16:13:25.943741+0000","last_active":"2024-04-16T16:13:26.970886+0000","last_peered":"2024-04-16T16:13:26.970886+0000","last_clean":"2024-04-16T16:13:26.970886+0000","last_became_active":"2024-04-16T16:13:25.942843+0000","last_became_peered":"2024-04-16T16:13:25.942843+0000","last_unstale":"2024-04-16T16:13:26.970886+0000","last_undegraded":"2024-04-16T16:13:26.970886+0000","last_fullsized":"2024-04-16T16:13:26.970886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:30:58.937717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.572988+0000","last_change":"2024-04-16T16:13:25.943232+0000","last_active":"2024-04-16T16:13:28.572988+0000","last_peered":"2024-04-16T16:13:28.572988+0000","last_clean":"2024-04-16T16:13:28.572988+0000","last_became_active":"2024-04-16T16:13:25.940853+0000","last_became_peered":"2024-04-16T16:13:25.940853+0000","last_unstale":"2024-04-16T16:13:28.572988+0000","last_undegraded":"2024-04-16T16:13:28.572988+0000","last_fullsized":"2024-04-16T16:13:28.572988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:55.948136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970797+0000","last_change":"2024-04-16T16:13:25.943328+0000","last_active":"2024-04-16T16:13:26.970797+0000","last_peered":"2024-04-16T16:13:26.970797+0000","last_clean":"2024-04-16T16:13:26.970797+0000","last_became_active":"2024-04-16T16:13:25.941855+0000","last_became_peered":"2024-04-16T16:13:25.941855+0000","last_unstale":"2024-04-16T16:13:26.970797+0000","last_undegraded":"2024-04-16T16:13:26.970797+0000","last_fullsized":"2024-04-16T16:13:26.970797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:41.243756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.574357+0000","last_change":"2024-04-16T16:13:25.942627+0000","last_active":"2024-04-16T16:13:28.574357+0000","last_peered":"2024-04-16T16:13:28.574357+0000","last_clean":"2024-04-16T16:13:28.574357+0000","last_became_active":"2024-04-16T16:13:25.942285+0000","last_became_peered":"2024-04-16T16:13:25.942285+0000","last_unstale":"2024-04-16T16:13:28.574357+0000","last_undegraded":"2024-04-16T16:13:28.574357+0000","last_fullsized":"2024-04-16T16:13:28.574357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:39.592952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961635+0000","last_change":"2024-04-16T16:13:25.950401+0000","last_active":"2024-04-16T16:13:26.961635+0000","last_peered":"2024-04-16T16:13:26.961635+0000","last_clean":"2024-04-16T16:13:26.961635+0000","last_became_active":"2024-04-16T16:13:25.948651+0000","last_became_peered":"2024-04-16T16:13:25.948651+0000","last_unstale":"2024-04-16T16:13:26.961635+0000","last_undegraded":"2024-04-16T16:13:26.961635+0000","last_fullsized":"2024-04-16T16:13:26.961635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955096+0000","last_change":"2024-04-16T16:13:25.945071+0000","last_active":"2024-04-16T16:13:26.955096+0000","last_peered":"2024-04-16T16:13:26.955096+0000","last_clean":"2024-04-16T16:13:26.955096+0000","last_became_active":"2024-04-16T16:13:25.944708+0000","last_became_peered":"2024-04-16T16:13:25.944708+0000","last_unstale":"2024-04-16T16:13:26.955096+0000","last_undegraded":"2024-04-16T16:13:26.955096+0000","last_fullsized":"2024-04-16T16:13:26.955096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.573821+0000","last_change":"2024-04-16T16:13:25.944654+0000","last_active":"2024-04-16T16:13:28.573821+0000","last_peered":"2024-04-16T16:13:28.573821+0000","last_clean":"2024-04-16T16:13:28.573821+0000","last_became_active":"2024-04-16T16:13:25.941914+0000","last_became_peered":"2024-04-16T16:13:25.941914+0000","last_unstale":"2024-04-16T16:13:28.573821+0000","last_undegraded":"2024-04-16T16:13:28.573821+0000","last_fullsized":"2024-04-16T16:13:28.573821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.574823+0000","last_change":"2024-04-16T16:13:25.947081+0000","last_active":"2024-04-16T16:13:28.574823+0000","last_peered":"2024-04-16T16:13:28.574823+0000","last_clean":"2024-04-16T16:13:28.574823+0000","last_became_active":"2024-04-16T16:13:25.946791+0000","last_became_peered":"2024-04-16T16:13:25.946791+0000","last_unstale":"2024-04-16T16:13:28.574823+0000","last_undegraded":"2024-04-16T16:13:28.574823+0000","last_fullsized":"2024-04-16T16:13:28.574823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:11:32.536402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971641+0000","last_change":"2024-04-16T16:13:25.952506+0000","last_active":"2024-04-16T16:13:26.971641+0000","last_peered":"2024-04-16T16:13:26.971641+0000","last_clean":"2024-04-16T16:13:26.971641+0000","last_became_active":"2024-04-16T16:13:25.952184+0000","last_became_peered":"2024-04-16T16:13:25.952184+0000","last_unstale":"2024-04-16T16:13:26.971641+0000","last_undegraded":"2024-04-16T16:13:26.971641+0000","last_fullsized":"2024-04-16T16:13:26.971641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:19:05.123994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936714+0000","last_change":"2024-04-16T16:13:25.936689+0000","last_active":"2024-04-16T16:13:26.936714+0000","last_peered":"2024-04-16T16:13:26.936714+0000","last_clean":"2024-04-16T16:13:26.936714+0000","last_became_active":"2024-04-16T16:13:25.936385+0000","last_became_peered":"2024-04-16T16:13:25.936385+0000","last_unstale":"2024-04-16T16:13:26.936714+0000","last_undegraded":"2024-04-16T16:13:26.936714+0000","last_fullsized":"2024-04-16T16:13:26.936714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:12.895579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960388+0000","last_change":"2024-04-16T16:13:25.953671+0000","last_active":"2024-04-16T16:13:26.960388+0000","last_peered":"2024-04-16T16:13:26.960388+0000","last_clean":"2024-04-16T16:13:26.960388+0000","last_became_active":"2024-04-16T16:13:25.953325+0000","last_became_peered":"2024-04-16T16:13:25.953325+0000","last_unstale":"2024-04-16T16:13:26.960388+0000","last_undegraded":"2024-04-16T16:13:26.960388+0000","last_fullsized":"2024-04-16T16:13:26.960388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:51:20.451559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941086+0000","last_change":"2024-04-16T16:13:25.936138+0000","last_active":"2024-04-16T16:13:26.941086+0000","last_peered":"2024-04-16T16:13:26.941086+0000","last_clean":"2024-04-16T16:13:26.941086+0000","last_became_active":"2024-04-16T16:13:25.935759+0000","last_became_peered":"2024-04-16T16:13:25.935759+0000","last_unstale":"2024-04-16T16:13:26.941086+0000","last_undegraded":"2024-04-16T16:13:26.941086+0000","last_fullsized":"2024-04-16T16:13:26.941086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:26.052535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.573477+0000","last_change":"2024-04-16T16:13:24.912677+0000","last_active":"2024-04-16T16:13:28.573477+0000","last_peered":"2024-04-16T16:13:28.573477+0000","last_clean":"2024-04-16T16:13:28.573477+0000","last_became_active":"2024-04-16T16:13:24.912364+0000","last_became_peered":"2024-04-16T16:13:24.912364+0000","last_unstale":"2024-04-16T16:13:28.573477+0000","last_undegraded":"2024-04-16T16:13:28.573477+0000","last_fullsized":"2024-04-16T16:13:28.573477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.575151+0000","last_change":"2024-04-16T16:13:25.947853+0000","last_active":"2024-04-16T16:13:28.575151+0000","last_peered":"2024-04-16T16:13:28.575151+0000","last_clean":"2024-04-16T16:13:28.575151+0000","last_became_active":"2024-04-16T16:13:25.947545+0000","last_became_peered":"2024-04-16T16:13:25.947545+0000","last_unstale":"2024-04-16T16:13:28.575151+0000","last_undegraded":"2024-04-16T16:13:28.575151+0000","last_fullsized":"2024-04-16T16:13:28.575151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960762+0000","last_change":"2024-04-16T16:13:24.917069+0000","last_active":"2024-04-16T16:13:26.960762+0000","last_peered":"2024-04-16T16:13:26.960762+0000","last_clean":"2024-04-16T16:13:26.960762+0000","last_became_active":"2024-04-16T16:13:24.916846+0000","last_became_peered":"2024-04-16T16:13:24.916846+0000","last_unstale":"2024-04-16T16:13:26.960762+0000","last_undegraded":"2024-04-16T16:13:26.960762+0000","last_fullsized":"2024-04-16T16:13:26.960762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.573484+0000","last_change":"2024-04-16T16:13:24.919388+0000","last_active":"2024-04-16T16:13:28.573484+0000","last_peered":"2024-04-16T16:13:28.573484+0000","last_clean":"2024-04-16T16:13:28.573484+0000","last_became_active":"2024-04-16T16:13:24.918937+0000","last_became_peered":"2024-04-16T16:13:24.918937+0000","last_unstale":"2024-04-16T16:13:28.573484+0000","last_undegraded":"2024-04-16T16:13:28.573484+0000","last_fullsized":"2024-04-16T16:13:28.573484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:46:08.208026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.572516+0000","last_change":"2024-04-16T16:13:25.954029+0000","last_active":"2024-04-16T16:13:28.572516+0000","last_peered":"2024-04-16T16:13:28.572516+0000","last_clean":"2024-04-16T16:13:28.572516+0000","last_became_active":"2024-04-16T16:13:25.953740+0000","last_became_peered":"2024-04-16T16:13:25.953740+0000","last_unstale":"2024-04-16T16:13:28.572516+0000","last_undegraded":"2024-04-16T16:13:28.572516+0000","last_fullsized":"2024-04-16T16:13:28.572516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:11:32.536402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971052+0000","last_change":"2024-04-16T16:13:24.923031+0000","last_active":"2024-04-16T16:13:26.971052+0000","last_peered":"2024-04-16T16:13:26.971052+0000","last_clean":"2024-04-16T16:13:26.971052+0000","last_became_active":"2024-04-16T16:13:24.922708+0000","last_became_peered":"2024-04-16T16:13:24.922708+0000","last_unstale":"2024-04-16T16:13:26.971052+0000","last_undegraded":"2024-04-16T16:13:26.971052+0000","last_fullsized":"2024-04-16T16:13:26.971052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954570+0000","last_change":"2024-04-16T16:13:25.949124+0000","last_active":"2024-04-16T16:13:26.954570+0000","last_peered":"2024-04-16T16:13:26.954570+0000","last_clean":"2024-04-16T16:13:26.954570+0000","last_became_active":"2024-04-16T16:13:25.948436+0000","last_became_peered":"2024-04-16T16:13:25.948436+0000","last_unstale":"2024-04-16T16:13:26.954570+0000","last_undegraded":"2024-04-16T16:13:26.954570+0000","last_fullsized":"2024-04-16T16:13:26.954570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:10.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961547+0000","last_change":"2024-04-16T16:13:24.925547+0000","last_active":"2024-04-16T16:13:26.961547+0000","last_peered":"2024-04-16T16:13:26.961547+0000","last_clean":"2024-04-16T16:13:26.961547+0000","last_became_active":"2024-04-16T16:13:24.925182+0000","last_became_peered":"2024-04-16T16:13:24.925182+0000","last_unstale":"2024-04-16T16:13:26.961547+0000","last_undegraded":"2024-04-16T16:13:26.961547+0000","last_fullsized":"2024-04-16T16:13:26.961547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.972351+0000","last_change":"2024-04-16T16:13:25.945204+0000","last_active":"2024-04-16T16:13:26.972351+0000","last_peered":"2024-04-16T16:13:26.972351+0000","last_clean":"2024-04-16T16:13:26.972351+0000","last_became_active":"2024-04-16T16:13:25.944902+0000","last_became_peered":"2024-04-16T16:13:25.944902+0000","last_unstale":"2024-04-16T16:13:26.972351+0000","last_undegraded":"2024-04-16T16:13:26.972351+0000","last_fullsized":"2024-04-16T16:13:26.972351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:52:47.172342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971774+0000","last_change":"2024-04-16T16:13:24.928041+0000","last_active":"2024-04-16T16:13:26.971774+0000","last_peered":"2024-04-16T16:13:26.971774+0000","last_clean":"2024-04-16T16:13:26.971774+0000","last_became_active":"2024-04-16T16:13:24.927555+0000","last_became_peered":"2024-04-16T16:13:24.927555+0000","last_unstale":"2024-04-16T16:13:26.971774+0000","last_undegraded":"2024-04-16T16:13:26.971774+0000","last_fullsized":"2024-04-16T16:13:26.971774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.575155+0000","last_change":"2024-04-16T16:13:25.943385+0000","last_active":"2024-04-16T16:13:28.575155+0000","last_peered":"2024-04-16T16:13:28.575155+0000","last_clean":"2024-04-16T16:13:28.575155+0000","last_became_active":"2024-04-16T16:13:25.943095+0000","last_became_peered":"2024-04-16T16:13:25.943095+0000","last_unstale":"2024-04-16T16:13:28.575155+0000","last_undegraded":"2024-04-16T16:13:28.575155+0000","last_fullsized":"2024-04-16T16:13:28.575155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:40:58.670474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955094+0000","last_change":"2024-04-16T16:13:24.922351+0000","last_active":"2024-04-16T16:13:26.955094+0000","last_peered":"2024-04-16T16:13:26.955094+0000","last_clean":"2024-04-16T16:13:26.955094+0000","last_became_active":"2024-04-16T16:13:24.922017+0000","last_became_peered":"2024-04-16T16:13:24.922017+0000","last_unstale":"2024-04-16T16:13:26.955094+0000","last_undegraded":"2024-04-16T16:13:26.955094+0000","last_fullsized":"2024-04-16T16:13:26.955094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:24.722256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942286+0000","last_change":"2024-04-16T16:13:25.944764+0000","last_active":"2024-04-16T16:13:26.942286+0000","last_peered":"2024-04-16T16:13:26.942286+0000","last_clean":"2024-04-16T16:13:26.942286+0000","last_became_active":"2024-04-16T16:13:25.944418+0000","last_became_peered":"2024-04-16T16:13:25.944418+0000","last_unstale":"2024-04-16T16:13:26.942286+0000","last_undegraded":"2024-04-16T16:13:26.942286+0000","last_fullsized":"2024-04-16T16:13:26.942286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:25.740193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960752+0000","last_change":"2024-04-16T16:13:24.916122+0000","last_active":"2024-04-16T16:13:26.960752+0000","last_peered":"2024-04-16T16:13:26.960752+0000","last_clean":"2024-04-16T16:13:26.960752+0000","last_became_active":"2024-04-16T16:13:24.915132+0000","last_became_peered":"2024-04-16T16:13:24.915132+0000","last_unstale":"2024-04-16T16:13:26.960752+0000","last_undegraded":"2024-04-16T16:13:26.960752+0000","last_fullsized":"2024-04-16T16:13:26.960752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942858+0000","last_change":"2024-04-16T16:13:25.954813+0000","last_active":"2024-04-16T16:13:26.942858+0000","last_peered":"2024-04-16T16:13:26.942858+0000","last_clean":"2024-04-16T16:13:26.942858+0000","last_became_active":"2024-04-16T16:13:25.952310+0000","last_became_peered":"2024-04-16T16:13:25.952310+0000","last_unstale":"2024-04-16T16:13:26.942858+0000","last_undegraded":"2024-04-16T16:13:26.942858+0000","last_fullsized":"2024-04-16T16:13:26.942858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:13:01.042309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.575148+0000","last_change":"2024-04-16T16:13:24.907364+0000","last_active":"2024-04-16T16:13:28.575148+0000","last_peered":"2024-04-16T16:13:28.575148+0000","last_clean":"2024-04-16T16:13:28.575148+0000","last_became_active":"2024-04-16T16:13:24.906972+0000","last_became_peered":"2024-04-16T16:13:24.906972+0000","last_unstale":"2024-04-16T16:13:28.575148+0000","last_undegraded":"2024-04-16T16:13:28.575148+0000","last_fullsized":"2024-04-16T16:13:28.575148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:22.075493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.572223+0000","last_change":"2024-04-16T16:13:25.946280+0000","last_active":"2024-04-16T16:13:28.572223+0000","last_peered":"2024-04-16T16:13:28.572223+0000","last_clean":"2024-04-16T16:13:28.572223+0000","last_became_active":"2024-04-16T16:13:25.945919+0000","last_became_peered":"2024-04-16T16:13:25.945919+0000","last_unstale":"2024-04-16T16:13:28.572223+0000","last_undegraded":"2024-04-16T16:13:28.572223+0000","last_fullsized":"2024-04-16T16:13:28.572223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:23.603931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971179+0000","last_change":"2024-04-16T16:13:24.921202+0000","last_active":"2024-04-16T16:13:26.971179+0000","last_peered":"2024-04-16T16:13:26.971179+0000","last_clean":"2024-04-16T16:13:26.971179+0000","last_became_active":"2024-04-16T16:13:24.920850+0000","last_became_peered":"2024-04-16T16:13:24.920850+0000","last_unstale":"2024-04-16T16:13:26.971179+0000","last_undegraded":"2024-04-16T16:13:26.971179+0000","last_fullsized":"2024-04-16T16:13:26.971179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942649+0000","last_change":"2024-04-16T16:13:25.947511+0000","last_active":"2024-04-16T16:13:26.942649+0000","last_peered":"2024-04-16T16:13:26.942649+0000","last_clean":"2024-04-16T16:13:26.942649+0000","last_became_active":"2024-04-16T16:13:25.947213+0000","last_became_peered":"2024-04-16T16:13:25.947213+0000","last_unstale":"2024-04-16T16:13:26.942649+0000","last_undegraded":"2024-04-16T16:13:26.942649+0000","last_fullsized":"2024-04-16T16:13:26.942649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:47.987635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960500+0000","last_change":"2024-04-16T16:13:24.923552+0000","last_active":"2024-04-16T16:13:26.960500+0000","last_peered":"2024-04-16T16:13:26.960500+0000","last_clean":"2024-04-16T16:13:26.960500+0000","last_became_active":"2024-04-16T16:13:24.922940+0000","last_became_peered":"2024-04-16T16:13:24.922940+0000","last_unstale":"2024-04-16T16:13:26.960500+0000","last_undegraded":"2024-04-16T16:13:26.960500+0000","last_fullsized":"2024-04-16T16:13:26.960500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960474+0000","last_change":"2024-04-16T16:13:25.954523+0000","last_active":"2024-04-16T16:13:26.960474+0000","last_peered":"2024-04-16T16:13:26.960474+0000","last_clean":"2024-04-16T16:13:26.960474+0000","last_became_active":"2024-04-16T16:13:25.954058+0000","last_became_peered":"2024-04-16T16:13:25.954058+0000","last_unstale":"2024-04-16T16:13:26.960474+0000","last_undegraded":"2024-04-16T16:13:26.960474+0000","last_fullsized":"2024-04-16T16:13:26.960474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:11:32.536402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936502+0000","last_change":"2024-04-16T16:13:24.904943+0000","last_active":"2024-04-16T16:13:26.936502+0000","last_peered":"2024-04-16T16:13:26.936502+0000","last_clean":"2024-04-16T16:13:26.936502+0000","last_became_active":"2024-04-16T16:13:24.904527+0000","last_became_peered":"2024-04-16T16:13:24.904527+0000","last_unstale":"2024-04-16T16:13:26.936502+0000","last_undegraded":"2024-04-16T16:13:26.936502+0000","last_fullsized":"2024-04-16T16:13:26.936502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:22.075493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942098+0000","last_change":"2024-04-16T16:13:25.937269+0000","last_active":"2024-04-16T16:13:26.942098+0000","last_peered":"2024-04-16T16:13:26.942098+0000","last_clean":"2024-04-16T16:13:26.942098+0000","last_became_active":"2024-04-16T16:13:25.936953+0000","last_became_peered":"2024-04-16T16:13:25.936953+0000","last_unstale":"2024-04-16T16:13:26.942098+0000","last_undegraded":"2024-04-16T16:13:26.942098+0000","last_fullsized":"2024-04-16T16:13:26.942098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:41.243756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941662+0000","last_change":"2024-04-16T16:13:24.909436+0000","last_active":"2024-04-16T16:13:26.941662+0000","last_peered":"2024-04-16T16:13:26.941662+0000","last_clean":"2024-04-16T16:13:26.941662+0000","last_became_active":"2024-04-16T16:13:24.908963+0000","last_became_peered":"2024-04-16T16:13:24.908963+0000","last_unstale":"2024-04-16T16:13:26.941662+0000","last_undegraded":"2024-04-16T16:13:26.941662+0000","last_fullsized":"2024-04-16T16:13:26.941662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:21.269919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943634+0000","last_change":"2024-04-16T16:13:25.939892+0000","last_active":"2024-04-16T16:13:26.943634+0000","last_peered":"2024-04-16T16:13:26.943634+0000","last_clean":"2024-04-16T16:13:26.943634+0000","last_became_active":"2024-04-16T16:13:25.939610+0000","last_became_peered":"2024-04-16T16:13:25.939610+0000","last_unstale":"2024-04-16T16:13:26.943634+0000","last_undegraded":"2024-04-16T16:13:26.943634+0000","last_fullsized":"2024-04-16T16:13:26.943634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:58.462911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.937081+0000","last_change":"2024-04-16T16:13:24.898787+0000","last_active":"2024-04-16T16:13:26.937081+0000","last_peered":"2024-04-16T16:13:26.937081+0000","last_clean":"2024-04-16T16:13:26.937081+0000","last_became_active":"2024-04-16T16:13:24.898427+0000","last_became_peered":"2024-04-16T16:13:24.898427+0000","last_unstale":"2024-04-16T16:13:26.937081+0000","last_undegraded":"2024-04-16T16:13:26.937081+0000","last_fullsized":"2024-04-16T16:13:26.937081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943828+0000","last_change":"2024-04-16T16:13:25.953661+0000","last_active":"2024-04-16T16:13:26.943828+0000","last_peered":"2024-04-16T16:13:26.943828+0000","last_clean":"2024-04-16T16:13:26.943828+0000","last_became_active":"2024-04-16T16:13:25.950243+0000","last_became_peered":"2024-04-16T16:13:25.950243+0000","last_unstale":"2024-04-16T16:13:26.943828+0000","last_undegraded":"2024-04-16T16:13:26.943828+0000","last_fullsized":"2024-04-16T16:13:26.943828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:08.801668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.955224+0000","last_change":"2024-04-16T16:13:24.906785+0000","last_active":"2024-04-16T16:13:26.955224+0000","last_peered":"2024-04-16T16:13:26.955224+0000","last_clean":"2024-04-16T16:13:26.955224+0000","last_became_active":"2024-04-16T16:13:24.906478+0000","last_became_peered":"2024-04-16T16:13:24.906478+0000","last_unstale":"2024-04-16T16:13:26.955224+0000","last_undegraded":"2024-04-16T16:13:26.955224+0000","last_fullsized":"2024-04-16T16:13:26.955224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:41.549340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971400+0000","last_change":"2024-04-16T16:13:25.951170+0000","last_active":"2024-04-16T16:13:26.971400+0000","last_peered":"2024-04-16T16:13:26.971400+0000","last_clean":"2024-04-16T16:13:26.971400+0000","last_became_active":"2024-04-16T16:13:25.950423+0000","last_became_peered":"2024-04-16T16:13:25.950423+0000","last_unstale":"2024-04-16T16:13:26.971400+0000","last_undegraded":"2024-04-16T16:13:26.971400+0000","last_fullsized":"2024-04-16T16:13:26.971400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:13:01.042309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936858+0000","last_change":"2024-04-16T16:13:24.920097+0000","last_active":"2024-04-16T16:13:26.936858+0000","last_peered":"2024-04-16T16:13:26.936858+0000","last_clean":"2024-04-16T16:13:26.936858+0000","last_became_active":"2024-04-16T16:13:24.919848+0000","last_became_peered":"2024-04-16T16:13:24.919848+0000","last_unstale":"2024-04-16T16:13:26.936858+0000","last_undegraded":"2024-04-16T16:13:26.936858+0000","last_fullsized":"2024-04-16T16:13:26.936858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:46:08.208026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942819+0000","last_change":"2024-04-16T16:13:25.952599+0000","last_active":"2024-04-16T16:13:26.942819+0000","last_peered":"2024-04-16T16:13:26.942819+0000","last_clean":"2024-04-16T16:13:26.942819+0000","last_became_active":"2024-04-16T16:13:25.949217+0000","last_became_peered":"2024-04-16T16:13:25.949217+0000","last_unstale":"2024-04-16T16:13:26.942819+0000","last_undegraded":"2024-04-16T16:13:26.942819+0000","last_fullsized":"2024-04-16T16:13:26.942819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:39.592952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.940554+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:13:26.940554+0000","last_peered":"2024-04-16T16:13:26.940554+0000","last_clean":"2024-04-16T16:13:26.940554+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:13:26.940554+0000","last_undegraded":"2024-04-16T16:13:26.940554+0000","last_fullsized":"2024-04-16T16:13:26.940554+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.970651+0000","last_change":"2024-04-16T16:13:24.919303+0000","last_active":"2024-04-16T16:13:26.970651+0000","last_peered":"2024-04-16T16:13:26.970651+0000","last_clean":"2024-04-16T16:13:26.970651+0000","last_became_active":"2024-04-16T16:13:24.918056+0000","last_became_peered":"2024-04-16T16:13:24.918056+0000","last_unstale":"2024-04-16T16:13:26.970651+0000","last_undegraded":"2024-04-16T16:13:26.970651+0000","last_fullsized":"2024-04-16T16:13:26.970651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.574008+0000","last_change":"2024-04-16T16:13:25.947227+0000","last_active":"2024-04-16T16:13:28.574008+0000","last_peered":"2024-04-16T16:13:28.574008+0000","last_clean":"2024-04-16T16:13:28.574008+0000","last_became_active":"2024-04-16T16:13:25.946951+0000","last_became_peered":"2024-04-16T16:13:25.946951+0000","last_unstale":"2024-04-16T16:13:28.574008+0000","last_undegraded":"2024-04-16T16:13:28.574008+0000","last_fullsized":"2024-04-16T16:13:28.574008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:47.987635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.940404+0000","last_change":"2024-04-16T16:13:24.912738+0000","last_active":"2024-04-16T16:13:26.940404+0000","last_peered":"2024-04-16T16:13:26.940404+0000","last_clean":"2024-04-16T16:13:26.940404+0000","last_became_active":"2024-04-16T16:13:24.912432+0000","last_became_peered":"2024-04-16T16:13:24.912432+0000","last_unstale":"2024-04-16T16:13:26.940404+0000","last_undegraded":"2024-04-16T16:13:26.940404+0000","last_fullsized":"2024-04-16T16:13:26.940404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.571983+0000","last_change":"2024-04-16T16:13:25.952202+0000","last_active":"2024-04-16T16:13:28.571983+0000","last_peered":"2024-04-16T16:13:28.571983+0000","last_clean":"2024-04-16T16:13:28.571983+0000","last_became_active":"2024-04-16T16:13:25.951932+0000","last_became_peered":"2024-04-16T16:13:25.951932+0000","last_unstale":"2024-04-16T16:13:28.571983+0000","last_undegraded":"2024-04-16T16:13:28.571983+0000","last_fullsized":"2024-04-16T16:13:28.571983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:38:46.469621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.936531+0000","last_change":"2024-04-16T16:13:24.901093+0000","last_active":"2024-04-16T16:13:26.936531+0000","last_peered":"2024-04-16T16:13:26.936531+0000","last_clean":"2024-04-16T16:13:26.936531+0000","last_became_active":"2024-04-16T16:13:24.900772+0000","last_became_peered":"2024-04-16T16:13:24.900772+0000","last_unstale":"2024-04-16T16:13:26.936531+0000","last_undegraded":"2024-04-16T16:13:26.936531+0000","last_fullsized":"2024-04-16T16:13:26.936531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:44.393669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954397+0000","last_change":"2024-04-16T16:13:25.929184+0000","last_active":"2024-04-16T16:13:26.954397+0000","last_peered":"2024-04-16T16:13:26.954397+0000","last_clean":"2024-04-16T16:13:26.954397+0000","last_became_active":"2024-04-16T16:13:25.927325+0000","last_became_peered":"2024-04-16T16:13:25.927325+0000","last_unstale":"2024-04-16T16:13:26.954397+0000","last_undegraded":"2024-04-16T16:13:26.954397+0000","last_fullsized":"2024-04-16T16:13:26.954397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:11:32.536402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971426+0000","last_change":"2024-04-16T16:13:24.923465+0000","last_active":"2024-04-16T16:13:26.971426+0000","last_peered":"2024-04-16T16:13:26.971426+0000","last_clean":"2024-04-16T16:13:26.971426+0000","last_became_active":"2024-04-16T16:13:24.923145+0000","last_became_peered":"2024-04-16T16:13:24.923145+0000","last_unstale":"2024-04-16T16:13:26.971426+0000","last_undegraded":"2024-04-16T16:13:26.971426+0000","last_fullsized":"2024-04-16T16:13:26.971426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941524+0000","last_change":"2024-04-16T16:13:25.948469+0000","last_active":"2024-04-16T16:13:26.941524+0000","last_peered":"2024-04-16T16:13:26.941524+0000","last_clean":"2024-04-16T16:13:26.941524+0000","last_became_active":"2024-04-16T16:13:25.947992+0000","last_became_peered":"2024-04-16T16:13:25.947992+0000","last_unstale":"2024-04-16T16:13:26.941524+0000","last_undegraded":"2024-04-16T16:13:26.941524+0000","last_fullsized":"2024-04-16T16:13:26.941524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:00.025740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954768+0000","last_change":"2024-04-16T16:13:24.905663+0000","last_active":"2024-04-16T16:13:26.954768+0000","last_peered":"2024-04-16T16:13:26.954768+0000","last_clean":"2024-04-16T16:13:26.954768+0000","last_became_active":"2024-04-16T16:13:24.905306+0000","last_became_peered":"2024-04-16T16:13:24.905306+0000","last_unstale":"2024-04-16T16:13:26.954768+0000","last_undegraded":"2024-04-16T16:13:26.954768+0000","last_fullsized":"2024-04-16T16:13:26.954768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971908+0000","last_change":"2024-04-16T16:13:25.955137+0000","last_active":"2024-04-16T16:13:26.971908+0000","last_peered":"2024-04-16T16:13:26.971908+0000","last_clean":"2024-04-16T16:13:26.971908+0000","last_became_active":"2024-04-16T16:13:25.954852+0000","last_became_peered":"2024-04-16T16:13:25.954852+0000","last_unstale":"2024-04-16T16:13:26.971908+0000","last_undegraded":"2024-04-16T16:13:26.971908+0000","last_fullsized":"2024-04-16T16:13:26.971908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:58.462911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943455+0000","last_change":"2024-04-16T16:13:24.915400+0000","last_active":"2024-04-16T16:13:26.943455+0000","last_peered":"2024-04-16T16:13:26.943455+0000","last_clean":"2024-04-16T16:13:26.943455+0000","last_became_active":"2024-04-16T16:13:24.915096+0000","last_became_peered":"2024-04-16T16:13:24.915096+0000","last_unstale":"2024-04-16T16:13:26.943455+0000","last_undegraded":"2024-04-16T16:13:26.943455+0000","last_fullsized":"2024-04-16T16:13:26.943455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:41.549340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954886+0000","last_change":"2024-04-16T16:13:25.924316+0000","last_active":"2024-04-16T16:13:26.954886+0000","last_peered":"2024-04-16T16:13:26.954886+0000","last_clean":"2024-04-16T16:13:26.954886+0000","last_became_active":"2024-04-16T16:13:25.923986+0000","last_became_peered":"2024-04-16T16:13:25.923986+0000","last_unstale":"2024-04-16T16:13:26.954886+0000","last_undegraded":"2024-04-16T16:13:26.954886+0000","last_fullsized":"2024-04-16T16:13:26.954886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:40:58.670474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.958843+0000","last_change":"2024-04-16T16:13:24.920876+0000","last_active":"2024-04-16T16:13:26.958843+0000","last_peered":"2024-04-16T16:13:26.958843+0000","last_clean":"2024-04-16T16:13:26.958843+0000","last_became_active":"2024-04-16T16:13:24.920409+0000","last_became_peered":"2024-04-16T16:13:24.920409+0000","last_unstale":"2024-04-16T16:13:26.958843+0000","last_undegraded":"2024-04-16T16:13:26.958843+0000","last_fullsized":"2024-04-16T16:13:26.958843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.573941+0000","last_change":"2024-04-16T16:13:25.941909+0000","last_active":"2024-04-16T16:13:28.573941+0000","last_peered":"2024-04-16T16:13:28.573941+0000","last_clean":"2024-04-16T16:13:28.573941+0000","last_became_active":"2024-04-16T16:13:25.941614+0000","last_became_peered":"2024-04-16T16:13:25.941614+0000","last_unstale":"2024-04-16T16:13:28.573941+0000","last_undegraded":"2024-04-16T16:13:28.573941+0000","last_fullsized":"2024-04-16T16:13:28.573941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:41.243756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.961615+0000","last_change":"2024-04-16T16:13:24.918600+0000","last_active":"2024-04-16T16:13:26.961615+0000","last_peered":"2024-04-16T16:13:26.961615+0000","last_clean":"2024-04-16T16:13:26.961615+0000","last_became_active":"2024-04-16T16:13:24.918273+0000","last_became_peered":"2024-04-16T16:13:24.918273+0000","last_unstale":"2024-04-16T16:13:26.961615+0000","last_undegraded":"2024-04-16T16:13:26.961615+0000","last_fullsized":"2024-04-16T16:13:26.961615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.575341+0000","last_change":"2024-04-16T16:13:25.937881+0000","last_active":"2024-04-16T16:13:28.575341+0000","last_peered":"2024-04-16T16:13:28.575341+0000","last_clean":"2024-04-16T16:13:28.575341+0000","last_became_active":"2024-04-16T16:13:25.936555+0000","last_became_peered":"2024-04-16T16:13:25.936555+0000","last_unstale":"2024-04-16T16:13:28.575341+0000","last_undegraded":"2024-04-16T16:13:28.575341+0000","last_fullsized":"2024-04-16T16:13:28.575341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:04:07.652593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.940400+0000","last_change":"2024-04-16T16:13:24.915314+0000","last_active":"2024-04-16T16:13:26.940400+0000","last_peered":"2024-04-16T16:13:26.940400+0000","last_clean":"2024-04-16T16:13:26.940400+0000","last_became_active":"2024-04-16T16:13:24.913831+0000","last_became_peered":"2024-04-16T16:13:24.913831+0000","last_unstale":"2024-04-16T16:13:26.940400+0000","last_undegraded":"2024-04-16T16:13:26.940400+0000","last_fullsized":"2024-04-16T16:13:26.940400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:25.034598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.941863+0000","last_change":"2024-04-16T16:13:25.939017+0000","last_active":"2024-04-16T16:13:26.941863+0000","last_peered":"2024-04-16T16:13:26.941863+0000","last_clean":"2024-04-16T16:13:26.941863+0000","last_became_active":"2024-04-16T16:13:25.936578+0000","last_became_peered":"2024-04-16T16:13:25.936578+0000","last_unstale":"2024-04-16T16:13:26.941863+0000","last_undegraded":"2024-04-16T16:13:26.941863+0000","last_fullsized":"2024-04-16T16:13:26.941863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:12.895579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954750+0000","last_change":"2024-04-16T16:13:24.909354+0000","last_active":"2024-04-16T16:13:26.954750+0000","last_peered":"2024-04-16T16:13:26.954750+0000","last_clean":"2024-04-16T16:13:26.954750+0000","last_became_active":"2024-04-16T16:13:24.909073+0000","last_became_peered":"2024-04-16T16:13:24.909073+0000","last_unstale":"2024-04-16T16:13:26.954750+0000","last_undegraded":"2024-04-16T16:13:26.954750+0000","last_fullsized":"2024-04-16T16:13:26.954750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:59.007803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.572771+0000","last_change":"2024-04-16T16:13:25.940225+0000","last_active":"2024-04-16T16:13:28.572771+0000","last_peered":"2024-04-16T16:13:28.572771+0000","last_clean":"2024-04-16T16:13:28.572771+0000","last_became_active":"2024-04-16T16:13:25.939871+0000","last_became_peered":"2024-04-16T16:13:25.939871+0000","last_unstale":"2024-04-16T16:13:28.572771+0000","last_undegraded":"2024-04-16T16:13:28.572771+0000","last_fullsized":"2024-04-16T16:13:28.572771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:10.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.954700+0000","last_change":"2024-04-16T16:13:24.907229+0000","last_active":"2024-04-16T16:13:26.954700+0000","last_peered":"2024-04-16T16:13:26.954700+0000","last_clean":"2024-04-16T16:13:26.954700+0000","last_became_active":"2024-04-16T16:13:24.906948+0000","last_became_peered":"2024-04-16T16:13:24.906948+0000","last_unstale":"2024-04-16T16:13:26.954700+0000","last_undegraded":"2024-04-16T16:13:26.954700+0000","last_fullsized":"2024-04-16T16:13:26.954700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:11.877642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.573289+0000","last_change":"2024-04-16T16:13:25.963968+0000","last_active":"2024-04-16T16:13:28.573289+0000","last_peered":"2024-04-16T16:13:28.573289+0000","last_clean":"2024-04-16T16:13:28.573289+0000","last_became_active":"2024-04-16T16:13:25.963699+0000","last_became_peered":"2024-04-16T16:13:25.963699+0000","last_unstale":"2024-04-16T16:13:28.573289+0000","last_undegraded":"2024-04-16T16:13:28.573289+0000","last_fullsized":"2024-04-16T16:13:28.573289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:15:42.180184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942898+0000","last_change":"2024-04-16T16:13:24.924622+0000","last_active":"2024-04-16T16:13:26.942898+0000","last_peered":"2024-04-16T16:13:26.942898+0000","last_clean":"2024-04-16T16:13:26.942898+0000","last_became_active":"2024-04-16T16:13:24.923313+0000","last_became_peered":"2024-04-16T16:13:24.923313+0000","last_unstale":"2024-04-16T16:13:26.942898+0000","last_undegraded":"2024-04-16T16:13:26.942898+0000","last_fullsized":"2024-04-16T16:13:26.942898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:52:40.317291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.960479+0000","last_change":"2024-04-16T16:13:25.938982+0000","last_active":"2024-04-16T16:13:26.960479+0000","last_peered":"2024-04-16T16:13:26.960479+0000","last_clean":"2024-04-16T16:13:26.960479+0000","last_became_active":"2024-04-16T16:13:25.938694+0000","last_became_peered":"2024-04-16T16:13:25.938694+0000","last_unstale":"2024-04-16T16:13:26.960479+0000","last_undegraded":"2024-04-16T16:13:26.960479+0000","last_fullsized":"2024-04-16T16:13:26.960479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:10.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971847+0000","last_change":"2024-04-16T16:13:25.944450+0000","last_active":"2024-04-16T16:13:26.971847+0000","last_peered":"2024-04-16T16:13:26.971847+0000","last_clean":"2024-04-16T16:13:26.971847+0000","last_became_active":"2024-04-16T16:13:25.941384+0000","last_became_peered":"2024-04-16T16:13:25.941384+0000","last_unstale":"2024-04-16T16:13:26.971847+0000","last_undegraded":"2024-04-16T16:13:26.971847+0000","last_fullsized":"2024-04-16T16:13:26.971847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:40:10.575471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.942812+0000","last_change":"2024-04-16T16:13:24.925176+0000","last_active":"2024-04-16T16:13:26.942812+0000","last_peered":"2024-04-16T16:13:26.942812+0000","last_clean":"2024-04-16T16:13:26.942812+0000","last_became_active":"2024-04-16T16:13:24.924863+0000","last_became_peered":"2024-04-16T16:13:24.924863+0000","last_unstale":"2024-04-16T16:13:26.942812+0000","last_undegraded":"2024-04-16T16:13:26.942812+0000","last_fullsized":"2024-04-16T16:13:26.942812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:11.877642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.971280+0000","last_change":"2024-04-16T16:13:25.944098+0000","last_active":"2024-04-16T16:13:26.971280+0000","last_peered":"2024-04-16T16:13:26.971280+0000","last_clean":"2024-04-16T16:13:26.971280+0000","last_became_active":"2024-04-16T16:13:25.943843+0000","last_became_peered":"2024-04-16T16:13:25.943843+0000","last_unstale":"2024-04-16T16:13:26.971280+0000","last_undegraded":"2024-04-16T16:13:26.971280+0000","last_fullsized":"2024-04-16T16:13:26.971280+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:39.592952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:26.943140+0000","last_change":"2024-04-16T16:13:24.923704+0000","last_active":"2024-04-16T16:13:26.943140+0000","last_peered":"2024-04-16T16:13:26.943140+0000","last_clean":"2024-04-16T16:13:26.943140+0000","last_became_active":"2024-04-16T16:13:24.923214+0000","last_became_peered":"2024-04-16T16:13:24.923214+0000","last_unstale":"2024-04-16T16:13:26.943140+0000","last_undegraded":"2024-04-16T16:13:26.943140+0000","last_fullsized":"2024-04-16T16:13:26.943140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:57:24.722256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.574982+0000","last_change":"2024-04-16T16:13:25.955067+0000","last_active":"2024-04-16T16:13:28.574982+0000","last_peered":"2024-04-16T16:13:28.574982+0000","last_clean":"2024-04-16T16:13:28.574982+0000","last_became_active":"2024-04-16T16:13:25.954780+0000","last_became_peered":"2024-04-16T16:13:25.954780+0000","last_unstale":"2024-04-16T16:13:28.574982+0000","last_undegraded":"2024-04-16T16:13:28.574982+0000","last_fullsized":"2024-04-16T16:13:28.574982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:55.948136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.574102+0000","last_change":"2024-04-16T16:13:24.913036+0000","last_active":"2024-04-16T16:13:28.574102+0000","last_peered":"2024-04-16T16:13:28.574102+0000","last_clean":"2024-04-16T16:13:28.574102+0000","last_became_active":"2024-04-16T16:13:24.911928+0000","last_became_peered":"2024-04-16T16:13:24.911928+0000","last_unstale":"2024-04-16T16:13:28.574102+0000","last_undegraded":"2024-04-16T16:13:28.574102+0000","last_fullsized":"2024-04-16T16:13:28.574102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:20.906326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.573997+0000","last_change":"2024-04-16T16:13:25.946331+0000","last_active":"2024-04-16T16:13:28.573997+0000","last_peered":"2024-04-16T16:13:28.573997+0000","last_clean":"2024-04-16T16:13:28.573997+0000","last_became_active":"2024-04-16T16:13:25.946053+0000","last_became_peered":"2024-04-16T16:13:25.946053+0000","last_unstale":"2024-04-16T16:13:28.573997+0000","last_undegraded":"2024-04-16T16:13:28.573997+0000","last_fullsized":"2024-04-16T16:13:28.573997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:24.867233+0000","last_clean_scrub_stamp":"2024-04-16T16:13:24.867233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:30:58.937717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-16T16:13:28.575420+0000","last_change":"2024-04-16T16:13:24.909349+0000","last_active":"2024-04-16T16:13:28.575420+0000","last_peered":"2024-04-16T16:13:28.575420+0000","last_clean":"2024-04-16T16:13:28.575420+0000","last_became_active":"2024-04-16T16:13:24.909069+0000","last_became_peered":"2024-04-16T16:13:24.909069+0000","last_unstale":"2024-04-16T16:13:28.575420+0000","last_undegraded":"2024-04-16T16:13:28.575420+0000","last_fullsized":"2024-04-16T16:13:28.575420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:23.849296+0000","last_clean_scrub_stamp":"2024-04-16T16:13:23.849296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:46:08.208026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:13:36.732 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-16T16:13:36.732 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-16T16:13:36.732 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-16T16:13:36.733 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-16T16:13:36.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.987+0000 7fb442e00640 1 Processor -- start 2024-04-16T16:13:37.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.987+0000 7fb442e00640 1 -- start start 2024-04-16T16:13:37.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.989+0000 7fb442e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb43c144ff0 0x7fb43c0cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.989+0000 7fb442e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb43c0c5d80 0x7fb43c15d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.989+0000 7fb442e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb43c15c700 con 0x7fb43c06dfb0 2024-04-16T16:13:37.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.989+0000 7fb442e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb43c15d2f0 con 0x7fb43c144ff0 2024-04-16T16:13:37.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.989+0000 7fb442e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb43c15c9c0 con 0x7fb43c0c5d80 2024-04-16T16:13:37.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.989+0000 7fb441a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb43c144ff0 0x7fb43c0cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.989+0000 7fb441000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb43c06dfb0 0x7fb43c0c5670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39086/0 (socket says 172.21.15.45:39086) 2024-04-16T16:13:37.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.989+0000 7fb441a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb43c144ff0 0x7fb43c0cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43386/0 (socket says 172.21.15.45:43386) 2024-04-16T16:13:37.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.989+0000 7fb441000640 1 -- 172.21.15.45:0/2443709879 learned_addr learned my addr 172.21.15.45:0/2443709879 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:37.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.989+0000 7fb442400640 1 --2- 172.21.15.45:0/2443709879 >> v2:172.21.15.103:3300/0 conn(0x7fb43c0c5d80 0x7fb43c15d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.989+0000 7fb441a00640 1 -- 172.21.15.45:0/2443709879 >> v2:172.21.15.103:3300/0 conn(0x7fb43c0c5d80 msgr2=0x7fb43c15d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:37.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.989+0000 7fb441a00640 1 --2- 172.21.15.45:0/2443709879 >> v2:172.21.15.103:3300/0 conn(0x7fb43c0c5d80 0x7fb43c15d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.990+0000 7fb441a00640 1 -- 172.21.15.45:0/2443709879 >> v1:172.21.15.45:6789/0 conn(0x7fb43c06dfb0 legacy=0x7fb43c0c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.990+0000 7fb441a00640 1 -- 172.21.15.45:0/2443709879 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43c15cca0 con 0x7fb43c144ff0 2024-04-16T16:13:37.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.990+0000 7fb43be00640 1 -- 172.21.15.45:0/2443709879 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1366358988 0 0) 0x7fb43c15c700 con 0x7fb43c06dfb0 2024-04-16T16:13:37.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.991+0000 7fb441a00640 1 --2- 172.21.15.45:0/2443709879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c144ff0 0x7fb43c0cfd30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb43c15d260 tx=0x7fb43002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b24201674de8026 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:37.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.991+0000 7fb43be00640 1 -- 172.21.15.45:0/2443709879 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430035020 con 0x7fb43c144ff0 2024-04-16T16:13:37.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.991+0000 7fb43be00640 1 -- 172.21.15.45:0/2443709879 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb430031830 con 0x7fb43c144ff0 2024-04-16T16:13:37.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.991+0000 7fb43be00640 1 -- 172.21.15.45:0/2443709879 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430031c00 con 0x7fb43c144ff0 2024-04-16T16:13:37.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.992+0000 7fb442e00640 1 -- 172.21.15.45:0/2443709879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c144ff0 msgr2=0x7fb43c0cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.992+0000 7fb442e00640 1 --2- 172.21.15.45:0/2443709879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c144ff0 0x7fb43c0cfd30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb43c15d260 tx=0x7fb43002b8d0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.992+0000 7fb442e00640 1 -- 172.21.15.45:0/2443709879 shutdown_connections 2024-04-16T16:13:37.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.992+0000 7fb442e00640 1 --2- 172.21.15.45:0/2443709879 >> v2:172.21.15.103:3300/0 conn(0x7fb43c0c5d80 0x7fb43c15d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.992+0000 7fb442e00640 1 --2- 172.21.15.45:0/2443709879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c144ff0 0x7fb43c0cfd30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.992+0000 7fb442e00640 1 -- 172.21.15.45:0/2443709879 >> 172.21.15.45:0/2443709879 conn(0x7fb43c06b930 msgr2=0x7fb43c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:37.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.992+0000 7fb442e00640 1 -- 172.21.15.45:0/2443709879 shutdown_connections 2024-04-16T16:13:37.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.992+0000 7fb442e00640 1 -- 172.21.15.45:0/2443709879 wait complete. 2024-04-16T16:13:37.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.994+0000 7fb442e00640 1 Processor -- start 2024-04-16T16:13:37.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.994+0000 7fb442e00640 1 -- start start 2024-04-16T16:13:37.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.995+0000 7fb442e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb43c06dfb0 0x7fb43c15b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.995+0000 7fb442e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb43c0c5d80 0x7fb43c0aa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.995+0000 7fb442e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c144ff0 0x7fb43c0ab9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.995+0000 7fb442e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c0d0270 con 0x7fb43c144ff0 2024-04-16T16:13:37.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.995+0000 7fb442e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c1453d0 con 0x7fb43c0c5d80 2024-04-16T16:13:37.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.995+0000 7fb442e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb43c0d03f0 con 0x7fb43c06dfb0 2024-04-16T16:13:37.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.995+0000 7fb442400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c144ff0 0x7fb43c0ab9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.995+0000 7fb442400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c144ff0 0x7fb43c0ab9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43398/0 (socket says 172.21.15.45:43398) 2024-04-16T16:13:37.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.995+0000 7fb442400640 1 -- 172.21.15.45:0/2493920267 learned_addr learned my addr 172.21.15.45:0/2493920267 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:37.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.995+0000 7fb441a00640 1 --2- 172.21.15.45:0/2493920267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb43c06dfb0 0x7fb43c15b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.995+0000 7fb442400640 1 -- 172.21.15.45:0/2493920267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb43c06dfb0 msgr2=0x7fb43c15b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.995+0000 7fb442400640 1 --2- 172.21.15.45:0/2493920267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb43c06dfb0 0x7fb43c15b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.995+0000 7fb442400640 1 -- 172.21.15.45:0/2493920267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb43c0c5d80 msgr2=0x7fb43c0aa3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.995+0000 7fb441000640 1 --2- 172.21.15.45:0/2493920267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb43c0c5d80 0x7fb43c0aa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.995+0000 7fb442400640 1 --2- 172.21.15.45:0/2493920267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb43c0c5d80 0x7fb43c0aa3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.996+0000 7fb442400640 1 -- 172.21.15.45:0/2493920267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43c0aad40 con 0x7fb43c144ff0 2024-04-16T16:13:37.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.996+0000 7fb441a00640 1 --2- 172.21.15.45:0/2493920267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb43c06dfb0 0x7fb43c15b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:37.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.996+0000 7fb442400640 1 --2- 172.21.15.45:0/2493920267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c144ff0 0x7fb43c0ab9e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb43400cd00 tx=0x7fb43400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:37.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.996+0000 7fb439600640 1 -- 172.21.15.45:0/2493920267 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb434003b90 con 0x7fb43c144ff0 2024-04-16T16:13:37.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.996+0000 7fb439600640 1 -- 172.21.15.45:0/2493920267 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb434003d50 con 0x7fb43c144ff0 2024-04-16T16:13:37.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.996+0000 7fb439600640 1 -- 172.21.15.45:0/2493920267 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb434004260 con 0x7fb43c144ff0 2024-04-16T16:13:37.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.996+0000 7fb442e00640 1 -- 172.21.15.45:0/2493920267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb43c0ab020 con 0x7fb43c144ff0 2024-04-16T16:13:37.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.996+0000 7fb442e00640 1 -- 172.21.15.45:0/2493920267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb43c142500 con 0x7fb43c144ff0 2024-04-16T16:13:37.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.998+0000 7fb442e00640 1 -- 172.21.15.45:0/2493920267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb43c0ceec0 con 0x7fb43c144ff0 2024-04-16T16:13:37.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.998+0000 7fb439600640 1 -- 172.21.15.45:0/2493920267 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb434004400 con 0x7fb43c144ff0 2024-04-16T16:13:37.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.999+0000 7fb439600640 1 --2- 172.21.15.45:0/2493920267 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb41c06c8f0 0x7fb41c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:36.999+0000 7fb439600640 1 -- 172.21.15.45:0/2493920267 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb43408c5a0 con 0x7fb43c144ff0 2024-04-16T16:13:37.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.000+0000 7fb441a00640 1 --2- 172.21.15.45:0/2493920267 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb41c06c8f0 0x7fb41c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.001+0000 7fb441a00640 1 --2- 172.21.15.45:0/2493920267 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb41c06c8f0 0x7fb41c06edb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb430036000 tx=0x7fb43003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:37.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.003+0000 7fb439600640 1 -- 172.21.15.45:0/2493920267 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb43c0ceec0 con 0x7fb43c144ff0 2024-04-16T16:13:37.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.250+0000 7fb442e00640 1 -- 172.21.15.45:0/2493920267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb43c1426d0 con 0x7fb43c144ff0 2024-04-16T16:13:37.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.250+0000 7fb439600640 1 -- 172.21.15.45:0/2493920267 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fb43405a9d0 con 0x7fb43c144ff0 2024-04-16T16:13:37.251 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:37.251 INFO:teuthology.orchestra.run.smithi045.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-16T16:13:37.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.253+0000 7fb442e00640 1 -- 172.21.15.45:0/2493920267 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb41c06c8f0 msgr2=0x7fb41c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.253+0000 7fb442e00640 1 --2- 172.21.15.45:0/2493920267 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb41c06c8f0 0x7fb41c06edb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb430036000 tx=0x7fb43003c040 comp rx=0 tx=0).stop 2024-04-16T16:13:37.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.253+0000 7fb442e00640 1 -- 172.21.15.45:0/2493920267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c144ff0 msgr2=0x7fb43c0ab9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.253+0000 7fb442e00640 1 --2- 172.21.15.45:0/2493920267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c144ff0 0x7fb43c0ab9e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb43400cd00 tx=0x7fb43400a660 comp rx=0 tx=0).stop 2024-04-16T16:13:37.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.253+0000 7fb442e00640 1 -- 172.21.15.45:0/2493920267 shutdown_connections 2024-04-16T16:13:37.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.253+0000 7fb442e00640 1 --2- 172.21.15.45:0/2493920267 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb41c06c8f0 0x7fb41c06edb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.253+0000 7fb442e00640 1 --2- 172.21.15.45:0/2493920267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c144ff0 0x7fb43c0ab9e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.253+0000 7fb442e00640 1 --2- 172.21.15.45:0/2493920267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb43c0c5d80 0x7fb43c0aa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.253+0000 7fb442e00640 1 --2- 172.21.15.45:0/2493920267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb43c06dfb0 0x7fb43c15b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.253+0000 7fb442e00640 1 -- 172.21.15.45:0/2493920267 >> 172.21.15.45:0/2493920267 conn(0x7fb43c06b930 msgr2=0x7fb43c06e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:37.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.253+0000 7fb442e00640 1 -- 172.21.15.45:0/2493920267 shutdown_connections 2024-04-16T16:13:37.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.253+0000 7fb442e00640 1 -- 172.21.15.45:0/2493920267 wait complete. 2024-04-16T16:13:37.273 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-16T16:13:37.274 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:13:37.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.490+0000 7f144fe00640 1 Processor -- start 2024-04-16T16:13:37.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.490+0000 7f144fe00640 1 -- start start 2024-04-16T16:13:37.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.492+0000 7f144fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1450146220 0x7f145015ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.492+0000 7f144fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f145006f7a0 0x7f145006df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.492+0000 7f144fe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f145015d360 con 0x7f145006e480 2024-04-16T16:13:37.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.492+0000 7f144fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1450160410 con 0x7f145006f7a0 2024-04-16T16:13:37.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.492+0000 7f144fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f145015d640 con 0x7f1450146220 2024-04-16T16:13:37.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.492+0000 7f144f400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f145006e480 0x7f14501608a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39088/0 (socket says 172.21.15.45:39088) 2024-04-16T16:13:37.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.492+0000 7f144f400640 1 -- 172.21.15.45:0/899324929 learned_addr learned my addr 172.21.15.45:0/899324929 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:37.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144ea00640 1 --2- 172.21.15.45:0/899324929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1450146220 0x7f145015ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144e000640 1 --2- 172.21.15.45:0/899324929 >> v2:172.21.15.103:3300/0 conn(0x7f145006f7a0 0x7f145006df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 -- 172.21.15.45:0/899324929 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1171432645 0 0) 0x7f145015d360 con 0x7f145006e480 2024-04-16T16:13:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 -- 172.21.15.45:0/899324929 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1434003520 con 0x7f145006e480 2024-04-16T16:13:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 -- 172.21.15.45:0/899324929 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3042729664 0 0) 0x7f1434003520 con 0x7f145006e480 2024-04-16T16:13:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 -- 172.21.15.45:0/899324929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1450146220 msgr2=0x7f145015ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 --2- 172.21.15.45:0/899324929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1450146220 0x7f145015ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 -- 172.21.15.45:0/899324929 >> v2:172.21.15.103:3300/0 conn(0x7f145006f7a0 msgr2=0x7f145006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 --2- 172.21.15.45:0/899324929 >> v2:172.21.15.103:3300/0 conn(0x7f145006f7a0 0x7f145006df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 -- 172.21.15.45:0/899324929 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145015d920 con 0x7f145006e480 2024-04-16T16:13:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 -- 172.21.15.45:0/899324929 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f1448001670 con 0x7f145006e480 2024-04-16T16:13:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 -- 172.21.15.45:0/899324929 >> v1:172.21.15.45:6789/0 conn(0x7f145006e480 legacy=0x7f14501608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 --2- 172.21.15.45:0/899324929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1450146220 0x7f14340039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 --2- 172.21.15.45:0/899324929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1434003f30 0x7f1434004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 --2- 172.21.15.45:0/899324929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1434004d40 0x7f14340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 -- 172.21.15.45:0/899324929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1450160410 con 0x7f1434003f30 2024-04-16T16:13:37.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 -- 172.21.15.45:0/899324929 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f145015d640 con 0x7f1434004d40 2024-04-16T16:13:37.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.493+0000 7f144d600640 1 -- 172.21.15.45:0/899324929 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f143400b9c0 con 0x7f1450146220 2024-04-16T16:13:37.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.494+0000 7f144e000640 1 --2- 172.21.15.45:0/899324929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1434003f30 0x7f1434004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.494+0000 7f144ea00640 1 --2- 172.21.15.45:0/899324929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1450146220 0x7f14340039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.494+0000 7f144e000640 1 -- 172.21.15.45:0/899324929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1450146220 msgr2=0x7f14340039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:37.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.494+0000 7f144e000640 1 --2- 172.21.15.45:0/899324929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1450146220 0x7f14340039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.494+0000 7f144e000640 1 -- 172.21.15.45:0/899324929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1434004d40 msgr2=0x7f14340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:37.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.494+0000 7f144e000640 1 --2- 172.21.15.45:0/899324929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1434004d40 0x7f14340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.494+0000 7f144e000640 1 -- 172.21.15.45:0/899324929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1434003520 con 0x7f1434003f30 2024-04-16T16:13:37.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.495+0000 7f144e000640 1 --2- 172.21.15.45:0/899324929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1434003f30 0x7f1434004800 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1438009ac0 tx=0x7f1434005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddcd052a06eb7024 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:37.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.495+0000 7f144d600640 1 -- 172.21.15.45:0/899324929 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1438039070 con 0x7f1434003f30 2024-04-16T16:13:37.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.495+0000 7f144d600640 1 -- 172.21.15.45:0/899324929 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f143802c6d0 con 0x7f1434003f30 2024-04-16T16:13:37.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.495+0000 7f144d600640 1 -- 172.21.15.45:0/899324929 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1438009690 con 0x7f1434003f30 2024-04-16T16:13:37.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.496+0000 7f144fe00640 1 -- 172.21.15.45:0/899324929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1434003f30 msgr2=0x7f1434004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.496+0000 7f144fe00640 1 --2- 172.21.15.45:0/899324929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1434003f30 0x7f1434004800 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1438009ac0 tx=0x7f1434005830 comp rx=0 tx=0).stop 2024-04-16T16:13:37.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.496+0000 7f144f400640 1 -- 172.21.15.45:0/899324929 reap_dead start 2024-04-16T16:13:37.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.496+0000 7f144fe00640 1 -- 172.21.15.45:0/899324929 shutdown_connections 2024-04-16T16:13:37.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.496+0000 7f144fe00640 1 -- 172.21.15.45:0/899324929 >> 172.21.15.45:0/899324929 conn(0x7f145006b930 msgr2=0x7f145006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:37.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.496+0000 7f144fe00640 1 -- 172.21.15.45:0/899324929 shutdown_connections 2024-04-16T16:13:37.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.496+0000 7f144fe00640 1 -- 172.21.15.45:0/899324929 wait complete. 2024-04-16T16:13:37.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.498+0000 7f144fe00640 1 Processor -- start 2024-04-16T16:13:37.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.498+0000 7f144fe00640 1 -- start start 2024-04-16T16:13:37.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.499+0000 7f144fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1434003f30 0x7f145015c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.499+0000 7f144fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1450143ad0 0x7f1450145020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.499+0000 7f144fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1450145560 0x7f14501348c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.499+0000 7f144fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f145006fda0 con 0x7f1450143ad0 2024-04-16T16:13:37.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.499+0000 7f144fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f145006fc20 con 0x7f1450145560 2024-04-16T16:13:37.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.499+0000 7f144fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1450160fb0 con 0x7f1434003f30 2024-04-16T16:13:37.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.499+0000 7f144e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1450143ad0 0x7f1450145020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.499+0000 7f144e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1450143ad0 0x7f1450145020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43420/0 (socket says 172.21.15.45:43420) 2024-04-16T16:13:37.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.499+0000 7f144e000640 1 -- 172.21.15.45:0/2483770728 learned_addr learned my addr 172.21.15.45:0/2483770728 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:37.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.499+0000 7f144e000640 1 -- 172.21.15.45:0/2483770728 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1434003f30 msgr2=0x7f145015c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.499+0000 7f144e000640 1 --2- 172.21.15.45:0/2483770728 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1434003f30 0x7f145015c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.499+0000 7f144e000640 1 -- 172.21.15.45:0/2483770728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1450145560 msgr2=0x7f14501348c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.499+0000 7f144e000640 1 --2- 172.21.15.45:0/2483770728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1450145560 0x7f14501348c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.499+0000 7f144e000640 1 -- 172.21.15.45:0/2483770728 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145015c5f0 con 0x7f1450143ad0 2024-04-16T16:13:37.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.500+0000 7f144e000640 1 --2- 172.21.15.45:0/2483770728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1450143ad0 0x7f1450145020 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1438000c00 tx=0x7f1438004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:37.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.500+0000 7f1446a00640 1 -- 172.21.15.45:0/2483770728 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1438039050 con 0x7f1450143ad0 2024-04-16T16:13:37.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.500+0000 7f1446a00640 1 -- 172.21.15.45:0/2483770728 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1438045070 con 0x7f1450143ad0 2024-04-16T16:13:37.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.500+0000 7f1446a00640 1 -- 172.21.15.45:0/2483770728 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14380044e0 con 0x7f1450143ad0 2024-04-16T16:13:37.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.500+0000 7f144fe00640 1 -- 172.21.15.45:0/2483770728 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1450142a30 con 0x7f1450143ad0 2024-04-16T16:13:37.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.500+0000 7f144fe00640 1 -- 172.21.15.45:0/2483770728 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1450142ff0 con 0x7f1450143ad0 2024-04-16T16:13:37.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.501+0000 7f144fe00640 1 -- 172.21.15.45:0/2483770728 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1450075780 con 0x7f1450143ad0 2024-04-16T16:13:37.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.502+0000 7f1446a00640 1 -- 172.21.15.45:0/2483770728 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1438003a50 con 0x7f1450143ad0 2024-04-16T16:13:37.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.503+0000 7f1446a00640 1 --2- 172.21.15.45:0/2483770728 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f142c06c920 0x7f142c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.503+0000 7f1446a00640 1 -- 172.21.15.45:0/2483770728 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f14380ad780 con 0x7f1450143ad0 2024-04-16T16:13:37.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.504+0000 7f144ea00640 1 --2- 172.21.15.45:0/2483770728 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f142c06c920 0x7f142c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.505+0000 7f144ea00640 1 --2- 172.21.15.45:0/2483770728 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f142c06c920 0x7f142c06ede0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1440009fc0 tx=0x7f1440009750 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:37.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.508+0000 7f1446a00640 1 -- 172.21.15.45:0/2483770728 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f143804e020 con 0x7f1450143ad0 2024-04-16T16:13:37.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.711+0000 7f144fe00640 1 -- 172.21.15.45:0/2483770728 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14500cff60 con 0x7f1450143ad0 2024-04-16T16:13:37.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.713+0000 7f1446a00640 1 -- 172.21.15.45:0/2483770728 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f143807bb10 con 0x7f1450143ad0 2024-04-16T16:13:37.714 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:37.714 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":21,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:26.918617+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:13:37.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.716+0000 7f144fe00640 1 -- 172.21.15.45:0/2483770728 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f142c06c920 msgr2=0x7f142c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.716+0000 7f144fe00640 1 --2- 172.21.15.45:0/2483770728 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f142c06c920 0x7f142c06ede0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1440009fc0 tx=0x7f1440009750 comp rx=0 tx=0).stop 2024-04-16T16:13:37.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.716+0000 7f144fe00640 1 -- 172.21.15.45:0/2483770728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1450143ad0 msgr2=0x7f1450145020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.716+0000 7f144fe00640 1 --2- 172.21.15.45:0/2483770728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1450143ad0 0x7f1450145020 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1438000c00 tx=0x7f1438004730 comp rx=0 tx=0).stop 2024-04-16T16:13:37.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.716+0000 7f144fe00640 1 -- 172.21.15.45:0/2483770728 shutdown_connections 2024-04-16T16:13:37.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.716+0000 7f144fe00640 1 --2- 172.21.15.45:0/2483770728 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f142c06c920 0x7f142c06ede0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.716+0000 7f144fe00640 1 --2- 172.21.15.45:0/2483770728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1450145560 0x7f14501348c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.716+0000 7f144fe00640 1 --2- 172.21.15.45:0/2483770728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1450143ad0 0x7f1450145020 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.716+0000 7f144fe00640 1 --2- 172.21.15.45:0/2483770728 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1434003f30 0x7f145015c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.716+0000 7f144fe00640 1 -- 172.21.15.45:0/2483770728 >> 172.21.15.45:0/2483770728 conn(0x7f145006b930 msgr2=0x7f14501411a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.717+0000 7f144fe00640 1 -- 172.21.15.45:0/2483770728 shutdown_connections 2024-04-16T16:13:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.717+0000 7f144fe00640 1 -- 172.21.15.45:0/2483770728 wait complete. 2024-04-16T16:13:37.738 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:13:23.600104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:13:24.368141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:13:37.738 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:13:37.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.964+0000 7fb6c4e00640 1 Processor -- start 2024-04-16T16:13:37.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.964+0000 7fb6c4e00640 1 -- start start 2024-04-16T16:13:37.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.966+0000 7fb6c4e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb6c00cfb50 0x7fb6c006dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.966+0000 7fb6c4e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6c00c6210 con 0x7fb6c0160e90 2024-04-16T16:13:37.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.966+0000 7fb6c4e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6c01603b0 con 0x7fb6c0145fe0 2024-04-16T16:13:37.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.966+0000 7fb6c4e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb6c0160690 con 0x7fb6c00cfb50 2024-04-16T16:13:37.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.967+0000 7fb6bfe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb6c00cfb50 0x7fb6c006dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.967+0000 7fb6b6a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb6c0160e90 0x7fb6c00cf440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39104/0 (socket says 172.21.15.45:39104) 2024-04-16T16:13:37.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.967+0000 7fb6b6a00640 1 -- 172.21.15.45:0/2752182548 learned_addr learned my addr 172.21.15.45:0/2752182548 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.967+0000 7fb6bea00640 1 -- 172.21.15.45:0/2752182548 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1793991812 0 0) 0x7fb6c00c6210 con 0x7fb6c0160e90 2024-04-16T16:13:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.967+0000 7fb6bea00640 1 -- 172.21.15.45:0/2752182548 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6a4003520 con 0x7fb6c0160e90 2024-04-16T16:13:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.967+0000 7fb6bfe00640 1 -- 172.21.15.45:0/2752182548 >> v1:172.21.15.103:6789/0 conn(0x7fb6c0145fe0 legacy=0x7fb6c00c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.967+0000 7fb6bfe00640 1 -- 172.21.15.45:0/2752182548 >> v1:172.21.15.45:6789/0 conn(0x7fb6c0160e90 legacy=0x7fb6c00cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.967+0000 7fb6bfe00640 1 -- 172.21.15.45:0/2752182548 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c0160970 con 0x7fb6c00cfb50 2024-04-16T16:13:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.968+0000 7fb6bfe00640 1 --2- 172.21.15.45:0/2752182548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c00cfb50 0x7fb6c006dfb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80097d0 tx=0x7fb6b802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c1d4d08fee63ade server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:37.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.969+0000 7fb6bea00640 1 -- 172.21.15.45:0/2752182548 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8035020 con 0x7fb6c00cfb50 2024-04-16T16:13:37.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.969+0000 7fb6bea00640 1 -- 172.21.15.45:0/2752182548 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6b8031810 con 0x7fb6c00cfb50 2024-04-16T16:13:37.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.969+0000 7fb6bea00640 1 -- 172.21.15.45:0/2752182548 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8031be0 con 0x7fb6c00cfb50 2024-04-16T16:13:37.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.969+0000 7fb6c4e00640 1 -- 172.21.15.45:0/2752182548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c00cfb50 msgr2=0x7fb6c006dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.969+0000 7fb6c4e00640 1 --2- 172.21.15.45:0/2752182548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c00cfb50 0x7fb6c006dfb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80097d0 tx=0x7fb6b802b8b0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.970+0000 7fb6c4e00640 1 -- 172.21.15.45:0/2752182548 shutdown_connections 2024-04-16T16:13:37.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.970+0000 7fb6c4e00640 1 --2- 172.21.15.45:0/2752182548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c00cfb50 0x7fb6c006dfb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.970+0000 7fb6c4e00640 1 -- 172.21.15.45:0/2752182548 >> 172.21.15.45:0/2752182548 conn(0x7fb6c0067e00 msgr2=0x7fb6c006b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:37.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.970+0000 7fb6c4e00640 1 -- 172.21.15.45:0/2752182548 shutdown_connections 2024-04-16T16:13:37.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.970+0000 7fb6c4e00640 1 -- 172.21.15.45:0/2752182548 wait complete. 2024-04-16T16:13:37.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.972+0000 7fb6c4e00640 1 Processor -- start 2024-04-16T16:13:37.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.972+0000 7fb6c4e00640 1 -- start start 2024-04-16T16:13:37.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6c4e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6c00cfb50 0x7fb6c0142ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6c4e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c0145fe0 0x7fb6c01f5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6c4e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6c0160e90 0x7fb6c01e84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6c4e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c00c5e60 con 0x7fb6c0145fe0 2024-04-16T16:13:37.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6c4e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01463c0 con 0x7fb6c0160e90 2024-04-16T16:13:37.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6c4e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb6c01f5940 con 0x7fb6c00cfb50 2024-04-16T16:13:37.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6b6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c0145fe0 0x7fb6c01f5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6b6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c0145fe0 0x7fb6c01f5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43440/0 (socket says 172.21.15.45:43440) 2024-04-16T16:13:37.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6b6a00640 1 -- 172.21.15.45:0/2979538143 learned_addr learned my addr 172.21.15.45:0/2979538143 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:37.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6bf400640 1 --2- 172.21.15.45:0/2979538143 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6c00cfb50 0x7fb6c0142ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6b6a00640 1 -- 172.21.15.45:0/2979538143 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6c00cfb50 msgr2=0x7fb6c0142ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6b6a00640 1 --2- 172.21.15.45:0/2979538143 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6c00cfb50 0x7fb6c0142ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6b6a00640 1 -- 172.21.15.45:0/2979538143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6c0160e90 msgr2=0x7fb6c01e84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:37.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6b6a00640 1 --2- 172.21.15.45:0/2979538143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6c0160e90 0x7fb6c01e84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:37.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6b6a00640 1 -- 172.21.15.45:0/2979538143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c01e8a30 con 0x7fb6c0145fe0 2024-04-16T16:13:37.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.973+0000 7fb6bf400640 1 --2- 172.21.15.45:0/2979538143 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6c00cfb50 0x7fb6c0142ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:37.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.974+0000 7fb6b6a00640 1 --2- 172.21.15.45:0/2979538143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c0145fe0 0x7fb6c01f5400 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800c410 tx=0x7fb6a800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:37.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.974+0000 7fb6b7e00640 1 -- 172.21.15.45:0/2979538143 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8017070 con 0x7fb6c0145fe0 2024-04-16T16:13:37.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.974+0000 7fb6b7e00640 1 -- 172.21.15.45:0/2979538143 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6a8005ce0 con 0x7fb6c0145fe0 2024-04-16T16:13:37.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.974+0000 7fb6c4e00640 1 -- 172.21.15.45:0/2979538143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6c01e7700 con 0x7fb6c0145fe0 2024-04-16T16:13:37.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.974+0000 7fb6b7e00640 1 -- 172.21.15.45:0/2979538143 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a80072f0 con 0x7fb6c0145fe0 2024-04-16T16:13:37.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.974+0000 7fb6c4e00640 1 -- 172.21.15.45:0/2979538143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c01e7cc0 con 0x7fb6c0145fe0 2024-04-16T16:13:37.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.976+0000 7fb6c4e00640 1 -- 172.21.15.45:0/2979538143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6c0075780 con 0x7fb6c0145fe0 2024-04-16T16:13:37.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.976+0000 7fb6b7e00640 1 -- 172.21.15.45:0/2979538143 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb6a801a070 con 0x7fb6c0145fe0 2024-04-16T16:13:37.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.978+0000 7fb6b7e00640 1 --2- 172.21.15.45:0/2979538143 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb69c06c8f0 0x7fb69c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:37.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.978+0000 7fb6b7e00640 1 -- 172.21.15.45:0/2979538143 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb6a808aff0 con 0x7fb6c0145fe0 2024-04-16T16:13:37.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.978+0000 7fb6bf400640 1 --2- 172.21.15.45:0/2979538143 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb69c06c8f0 0x7fb69c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:37.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.979+0000 7fb6bf400640 1 --2- 172.21.15.45:0/2979538143 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb69c06c8f0 0x7fb69c06edb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0001d70 tx=0x7fb6b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:37.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:37.982+0000 7fb6b7e00640 1 -- 172.21.15.45:0/2979538143 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb6a80594d0 con 0x7fb6c0145fe0 2024-04-16T16:13:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.183+0000 7fb6c4e00640 1 -- 172.21.15.45:0/2979538143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb6c00c6c80 con 0x7fb6c0145fe0 2024-04-16T16:13:38.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.184+0000 7fb6b7e00640 1 -- 172.21.15.45:0/2979538143 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fb6a8058dd0 con 0x7fb6c0145fe0 2024-04-16T16:13:38.184 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:13:38.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.186+0000 7fb6c4e00640 1 -- 172.21.15.45:0/2979538143 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb69c06c8f0 msgr2=0x7fb69c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:38.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.186+0000 7fb6c4e00640 1 --2- 172.21.15.45:0/2979538143 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb69c06c8f0 0x7fb69c06edb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0001d70 tx=0x7fb6b0009040 comp rx=0 tx=0).stop 2024-04-16T16:13:38.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.186+0000 7fb6c4e00640 1 -- 172.21.15.45:0/2979538143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c0145fe0 msgr2=0x7fb6c01f5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:38.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.186+0000 7fb6c4e00640 1 --2- 172.21.15.45:0/2979538143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c0145fe0 0x7fb6c01f5400 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800c410 tx=0x7fb6a800c8d0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.186+0000 7fb6c4e00640 1 -- 172.21.15.45:0/2979538143 shutdown_connections 2024-04-16T16:13:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.186+0000 7fb6c4e00640 1 --2- 172.21.15.45:0/2979538143 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb69c06c8f0 0x7fb69c06edb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.186+0000 7fb6c4e00640 1 --2- 172.21.15.45:0/2979538143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6c0160e90 0x7fb6c01e84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.186+0000 7fb6c4e00640 1 --2- 172.21.15.45:0/2979538143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c0145fe0 0x7fb6c01f5400 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.186+0000 7fb6c4e00640 1 --2- 172.21.15.45:0/2979538143 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6c00cfb50 0x7fb6c0142ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.186+0000 7fb6c4e00640 1 -- 172.21.15.45:0/2979538143 >> 172.21.15.45:0/2979538143 conn(0x7fb6c0067e00 msgr2=0x7fb6c015ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.186+0000 7fb6c4e00640 1 -- 172.21.15.45:0/2979538143 shutdown_connections 2024-04-16T16:13:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.186+0000 7fb6c4e00640 1 -- 172.21.15.45:0/2979538143 wait complete. 2024-04-16T16:13:38.205 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:13:38.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.424+0000 7f56fec00640 1 Processor -- start 2024-04-16T16:13:38.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.425+0000 7f56fec00640 1 -- start start 2024-04-16T16:13:38.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.426+0000 7f56fec00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56f80d18f0 0x7f56f80ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:38.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.426+0000 7f56fec00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56f80b6310 0x7f56f80d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:38.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.426+0000 7f56fec00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f56f80d1cd0 0x7f56f80d33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:38.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.426+0000 7f56fec00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f56f80d29f0 con 0x7f56f80d18f0 2024-04-16T16:13:38.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.426+0000 7f56fec00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f56f80b6810 con 0x7f56f80d1cd0 2024-04-16T16:13:38.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.426+0000 7f56fec00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f56f80b6af0 con 0x7f56f80b6310 2024-04-16T16:13:38.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.426+0000 7f56fd800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56f80d18f0 0x7f56f80ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:38.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.426+0000 7f56fd800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56f80d18f0 0x7f56f80ba220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43450/0 (socket says 172.21.15.45:43450) 2024-04-16T16:13:38.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.426+0000 7f56fd800640 1 -- 172.21.15.45:0/3037244395 learned_addr learned my addr 172.21.15.45:0/3037244395 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:38.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.427+0000 7f56fe200640 1 --2- 172.21.15.45:0/3037244395 >> v2:172.21.15.103:3300/0 conn(0x7f56f80d1cd0 0x7f56f80d33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.427+0000 7f56fce00640 1 --2- 172.21.15.45:0/3037244395 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56f80b6310 0x7f56f80d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.427+0000 7f56fd800640 1 -- 172.21.15.45:0/3037244395 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56f80b6310 msgr2=0x7f56f80d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.427+0000 7f56fd800640 1 --2- 172.21.15.45:0/3037244395 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56f80b6310 0x7f56f80d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.427+0000 7f56fd800640 1 -- 172.21.15.45:0/3037244395 >> v2:172.21.15.103:3300/0 conn(0x7f56f80d1cd0 msgr2=0x7f56f80d33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.427+0000 7f56fd800640 1 --2- 172.21.15.45:0/3037244395 >> v2:172.21.15.103:3300/0 conn(0x7f56f80d1cd0 0x7f56f80d33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.427+0000 7f56fd800640 1 -- 172.21.15.45:0/3037244395 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f80b6dd0 con 0x7f56f80d18f0 2024-04-16T16:13:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.427+0000 7f56fce00640 1 --2- 172.21.15.45:0/3037244395 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56f80b6310 0x7f56f80d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:38.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.428+0000 7f56fd800640 1 --2- 172.21.15.45:0/3037244395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56f80d18f0 0x7f56f80ba220 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f56f80b4140 tx=0x7f56ec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1908ff98d765888f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:38.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.428+0000 7f56f7e00640 1 -- 172.21.15.45:0/3037244395 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec02e070 con 0x7f56f80d18f0 2024-04-16T16:13:38.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.428+0000 7f56f7e00640 1 -- 172.21.15.45:0/3037244395 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ec037070 con 0x7f56f80d18f0 2024-04-16T16:13:38.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.428+0000 7f56f7e00640 1 -- 172.21.15.45:0/3037244395 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec031a00 con 0x7f56f80d18f0 2024-04-16T16:13:38.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.429+0000 7f56fec00640 1 -- 172.21.15.45:0/3037244395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56f80d18f0 msgr2=0x7f56f80ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:38.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.429+0000 7f56fec00640 1 --2- 172.21.15.45:0/3037244395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56f80d18f0 0x7f56f80ba220 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f56f80b4140 tx=0x7f56ec02b880 comp rx=0 tx=0).stop 2024-04-16T16:13:38.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.429+0000 7f56fec00640 1 -- 172.21.15.45:0/3037244395 shutdown_connections 2024-04-16T16:13:38.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.429+0000 7f56fec00640 1 --2- 172.21.15.45:0/3037244395 >> v2:172.21.15.103:3300/0 conn(0x7f56f80d1cd0 0x7f56f80d33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.429+0000 7f56fec00640 1 --2- 172.21.15.45:0/3037244395 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56f80b6310 0x7f56f80d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.429+0000 7f56fec00640 1 --2- 172.21.15.45:0/3037244395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56f80d18f0 0x7f56f80ba220 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.429+0000 7f56fec00640 1 -- 172.21.15.45:0/3037244395 >> 172.21.15.45:0/3037244395 conn(0x7f56f806cf90 msgr2=0x7f56f806d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:38.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.429+0000 7f56fec00640 1 -- 172.21.15.45:0/3037244395 shutdown_connections 2024-04-16T16:13:38.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.429+0000 7f56fec00640 1 -- 172.21.15.45:0/3037244395 wait complete. 2024-04-16T16:13:38.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.431+0000 7f56fec00640 1 Processor -- start 2024-04-16T16:13:38.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.431+0000 7f56fec00640 1 -- start start 2024-04-16T16:13:38.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.432+0000 7f56fec00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56f80b6310 0x7f56f80ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:38.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.432+0000 7f56fec00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56f80d18f0 0x7f56f815a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:38.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.432+0000 7f56fec00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56f80d1cd0 0x7f56f815a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:38.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.432+0000 7f56fec00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f56f80d3900 con 0x7f56f80d1cd0 2024-04-16T16:13:38.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.432+0000 7f56fec00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f56f80ba760 con 0x7f56f80d18f0 2024-04-16T16:13:38.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.432+0000 7f56fec00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f56f80d3a80 con 0x7f56f80b6310 2024-04-16T16:13:38.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.432+0000 7f56fe200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56f80d1cd0 0x7f56f815a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:38.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.432+0000 7f56fe200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56f80d1cd0 0x7f56f815a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43452/0 (socket says 172.21.15.45:43452) 2024-04-16T16:13:38.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.432+0000 7f56fe200640 1 -- 172.21.15.45:0/2486406235 learned_addr learned my addr 172.21.15.45:0/2486406235 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:38.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.433+0000 7f56fd800640 1 --2- 172.21.15.45:0/2486406235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56f80b6310 0x7f56f80ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:38.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.433+0000 7f56fe200640 1 -- 172.21.15.45:0/2486406235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56f80b6310 msgr2=0x7f56f80ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:38.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.433+0000 7f56fe200640 1 --2- 172.21.15.45:0/2486406235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56f80b6310 0x7f56f80ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.433+0000 7f56fe200640 1 -- 172.21.15.45:0/2486406235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56f80d18f0 msgr2=0x7f56f815a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:38.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.433+0000 7f56fe200640 1 --2- 172.21.15.45:0/2486406235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56f80d18f0 0x7f56f815a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.433+0000 7f56fe200640 1 -- 172.21.15.45:0/2486406235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f815aea0 con 0x7f56f80d1cd0 2024-04-16T16:13:38.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.433+0000 7f56fe200640 1 --2- 172.21.15.45:0/2486406235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56f80d1cd0 0x7f56f815a760 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f56f000cc60 tx=0x7f56f000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:38.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.433+0000 7f56f5600640 1 -- 172.21.15.45:0/2486406235 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f0003990 con 0x7f56f80d1cd0 2024-04-16T16:13:38.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.433+0000 7f56f5600640 1 -- 172.21.15.45:0/2486406235 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f0003b30 con 0x7f56f80d1cd0 2024-04-16T16:13:38.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.433+0000 7f56fec00640 1 -- 172.21.15.45:0/2486406235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f815b180 con 0x7f56f80d1cd0 2024-04-16T16:13:38.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.433+0000 7f56f5600640 1 -- 172.21.15.45:0/2486406235 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f0004020 con 0x7f56f80d1cd0 2024-04-16T16:13:38.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.433+0000 7f56fec00640 1 -- 172.21.15.45:0/2486406235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f80cd160 con 0x7f56f80d1cd0 2024-04-16T16:13:38.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.435+0000 7f56fec00640 1 -- 172.21.15.45:0/2486406235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f8075780 con 0x7f56f80d1cd0 2024-04-16T16:13:38.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.435+0000 7f56f5600640 1 -- 172.21.15.45:0/2486406235 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f56f00041c0 con 0x7f56f80d1cd0 2024-04-16T16:13:38.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.436+0000 7f56f5600640 1 --2- 172.21.15.45:0/2486406235 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f56d806c8f0 0x7f56d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:38.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.436+0000 7f56f5600640 1 -- 172.21.15.45:0/2486406235 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f56f008c0f0 con 0x7f56f80d1cd0 2024-04-16T16:13:38.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.437+0000 7f56fd800640 1 --2- 172.21.15.45:0/2486406235 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f56d806c8f0 0x7f56d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:38.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.438+0000 7f56fd800640 1 --2- 172.21.15.45:0/2486406235 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f56d806c8f0 0x7f56d806edb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f56ec036000 tx=0x7f56ec008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:38.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.441+0000 7f56f5600640 1 -- 172.21.15.45:0/2486406235 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f56f0014030 con 0x7f56f80d1cd0 2024-04-16T16:13:38.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.636+0000 7f56fec00640 1 -- 172.21.15.45:0/2486406235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f56f80d25a0 con 0x7f56f80d1cd0 2024-04-16T16:13:38.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.637+0000 7f56f5600640 1 -- 172.21.15.45:0/2486406235 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f56f005a490 con 0x7f56f80d1cd0 2024-04-16T16:13:38.637 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:13:38.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.639+0000 7f56fec00640 1 -- 172.21.15.45:0/2486406235 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f56d806c8f0 msgr2=0x7f56d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:38.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.639+0000 7f56fec00640 1 --2- 172.21.15.45:0/2486406235 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f56d806c8f0 0x7f56d806edb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f56ec036000 tx=0x7f56ec008030 comp rx=0 tx=0).stop 2024-04-16T16:13:38.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.639+0000 7f56fec00640 1 -- 172.21.15.45:0/2486406235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56f80d1cd0 msgr2=0x7f56f815a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:38.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.639+0000 7f56fec00640 1 --2- 172.21.15.45:0/2486406235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56f80d1cd0 0x7f56f815a760 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f56f000cc60 tx=0x7f56f000a5c0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.640+0000 7f56fec00640 1 -- 172.21.15.45:0/2486406235 shutdown_connections 2024-04-16T16:13:38.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.640+0000 7f56fec00640 1 --2- 172.21.15.45:0/2486406235 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f56d806c8f0 0x7f56d806edb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.640+0000 7f56fec00640 1 --2- 172.21.15.45:0/2486406235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56f80d1cd0 0x7f56f815a760 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.640+0000 7f56fec00640 1 --2- 172.21.15.45:0/2486406235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56f80d18f0 0x7f56f815a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.640+0000 7f56fec00640 1 --2- 172.21.15.45:0/2486406235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56f80b6310 0x7f56f80ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.640+0000 7f56fec00640 1 -- 172.21.15.45:0/2486406235 >> 172.21.15.45:0/2486406235 conn(0x7f56f806cf90 msgr2=0x7f56f806d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:38.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.640+0000 7f56fec00640 1 -- 172.21.15.45:0/2486406235 shutdown_connections 2024-04-16T16:13:38.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.640+0000 7f56fec00640 1 -- 172.21.15.45:0/2486406235 wait complete. 2024-04-16T16:13:38.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:13:38.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.878+0000 7f8fcca00640 1 Processor -- start 2024-04-16T16:13:38.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.878+0000 7f8fcca00640 1 -- start start 2024-04-16T16:13:38.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.879+0000 7f8fcca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8fc80d1030 0x7f8fc815f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:38.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.879+0000 7f8fcca00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc80b5e10 con 0x7f8fc80b5a30 2024-04-16T16:13:38.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.879+0000 7f8fcca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc8162e70 con 0x7f8fc815fda0 2024-04-16T16:13:38.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.879+0000 7f8fcca00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8fc815f3d0 con 0x7f8fc80d1030 2024-04-16T16:13:38.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.880+0000 7f8fc5400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8fc80d1030 0x7f8fc815f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:38.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.880+0000 7f8fc5e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8fc80b5a30 0x7f8fc81625e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39106/0 (socket says 172.21.15.45:39106) 2024-04-16T16:13:38.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.880+0000 7f8fc5e00640 1 -- 172.21.15.45:0/3208398459 learned_addr learned my addr 172.21.15.45:0/3208398459 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:38.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.880+0000 7f8fc4a00640 1 -- 172.21.15.45:0/3208398459 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1069057952 0 0) 0x7f8fc80b5e10 con 0x7f8fc80b5a30 2024-04-16T16:13:38.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.880+0000 7f8fc5400640 1 -- 172.21.15.45:0/3208398459 >> v1:172.21.15.103:6789/0 conn(0x7f8fc815fda0 legacy=0x7f8fc806e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:38.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.880+0000 7f8fc5400640 1 -- 172.21.15.45:0/3208398459 >> v1:172.21.15.45:6789/0 conn(0x7f8fc80b5a30 legacy=0x7f8fc81625e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:38.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.880+0000 7f8fc5400640 1 -- 172.21.15.45:0/3208398459 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc8163150 con 0x7f8fc80d1030 2024-04-16T16:13:38.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.881+0000 7f8fc5400640 1 --2- 172.21.15.45:0/3208398459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fc80d1030 0x7f8fc815f860 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40097a0 tx=0x7f8fb402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5395f703e6c1cf7e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:38.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.882+0000 7f8fc4a00640 1 -- 172.21.15.45:0/3208398459 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4035020 con 0x7f8fc80d1030 2024-04-16T16:13:38.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.882+0000 7f8fc4a00640 1 -- 172.21.15.45:0/3208398459 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fb4031830 con 0x7f8fc80d1030 2024-04-16T16:13:38.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.882+0000 7f8fc4a00640 1 -- 172.21.15.45:0/3208398459 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4031c00 con 0x7f8fc80d1030 2024-04-16T16:13:38.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.882+0000 7f8fcca00640 1 -- 172.21.15.45:0/3208398459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fc80d1030 msgr2=0x7f8fc815f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:38.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.882+0000 7f8fcca00640 1 --2- 172.21.15.45:0/3208398459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fc80d1030 0x7f8fc815f860 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40097a0 tx=0x7f8fb402b8d0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.882+0000 7f8fcca00640 1 -- 172.21.15.45:0/3208398459 shutdown_connections 2024-04-16T16:13:38.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.882+0000 7f8fcca00640 1 --2- 172.21.15.45:0/3208398459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fc80d1030 0x7f8fc815f860 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.882+0000 7f8fcca00640 1 -- 172.21.15.45:0/3208398459 >> 172.21.15.45:0/3208398459 conn(0x7f8fc80b3970 msgr2=0x7f8fc80b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:38.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.883+0000 7f8fcca00640 1 -- 172.21.15.45:0/3208398459 shutdown_connections 2024-04-16T16:13:38.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.883+0000 7f8fcca00640 1 -- 172.21.15.45:0/3208398459 wait complete. 2024-04-16T16:13:38.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.884+0000 7f8fcca00640 1 Processor -- start 2024-04-16T16:13:38.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.884+0000 7f8fcca00640 1 -- start start 2024-04-16T16:13:38.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.885+0000 7f8fcca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fc80b5a30 0x7f8fc815e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:38.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.885+0000 7f8fcca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fc80d1030 0x7f8fc8152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:38.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.885+0000 7f8fcca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8fc815eba0 0x7f8fc80ce690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:38.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.885+0000 7f8fcca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc806e9a0 con 0x7f8fc80b5a30 2024-04-16T16:13:38.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.885+0000 7f8fcca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc815eff0 con 0x7f8fc80d1030 2024-04-16T16:13:38.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.885+0000 7f8fcca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8fc806eb20 con 0x7f8fc815eba0 2024-04-16T16:13:38.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.886+0000 7f8fc5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fc80b5a30 0x7f8fc815e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:38.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.886+0000 7f8fc5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fc80b5a30 0x7f8fc815e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43484/0 (socket says 172.21.15.45:43484) 2024-04-16T16:13:38.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.886+0000 7f8fc5e00640 1 -- 172.21.15.45:0/3121059119 learned_addr learned my addr 172.21.15.45:0/3121059119 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.886+0000 7f8fc5400640 1 --2- 172.21.15.45:0/3121059119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fc80d1030 0x7f8fc8152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.886+0000 7f8fc5e00640 1 -- 172.21.15.45:0/3121059119 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8fc815eba0 msgr2=0x7f8fc80ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.886+0000 7f8fc5e00640 1 --2- 172.21.15.45:0/3121059119 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8fc815eba0 0x7f8fc80ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.886+0000 7f8fc5e00640 1 -- 172.21.15.45:0/3121059119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fc80d1030 msgr2=0x7f8fc8152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.886+0000 7f8fc5e00640 1 --2- 172.21.15.45:0/3121059119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fc80d1030 0x7f8fc8152620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.886+0000 7f8fc5e00640 1 -- 172.21.15.45:0/3121059119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc80cd5f0 con 0x7f8fc80b5a30 2024-04-16T16:13:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.886+0000 7f8fc5e00640 1 --2- 172.21.15.45:0/3121059119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fc80b5a30 0x7f8fc815e660 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8fb800c4d0 tx=0x7f8fb800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.886+0000 7f8fbe000640 1 -- 172.21.15.45:0/3121059119 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8018070 con 0x7f8fc80b5a30 2024-04-16T16:13:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.887+0000 7f8fbe000640 1 -- 172.21.15.45:0/3121059119 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fb8007500 con 0x7f8fc80b5a30 2024-04-16T16:13:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.887+0000 7f8fcca00640 1 -- 172.21.15.45:0/3121059119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fc80cd8d0 con 0x7f8fc80b5a30 2024-04-16T16:13:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.887+0000 7f8fbe000640 1 -- 172.21.15.45:0/3121059119 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8004a20 con 0x7f8fc80b5a30 2024-04-16T16:13:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.887+0000 7f8fcca00640 1 -- 172.21.15.45:0/3121059119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fc80ce170 con 0x7f8fc80b5a30 2024-04-16T16:13:38.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.888+0000 7f8fcca00640 1 -- 172.21.15.45:0/3121059119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fc8075780 con 0x7f8fc80b5a30 2024-04-16T16:13:38.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.889+0000 7f8fbe000640 1 -- 172.21.15.45:0/3121059119 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8fb801b070 con 0x7f8fc80b5a30 2024-04-16T16:13:38.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.890+0000 7f8fbe000640 1 --2- 172.21.15.45:0/3121059119 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8fa806c8f0 0x7f8fa806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:38.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.890+0000 7f8fbe000640 1 -- 172.21.15.45:0/3121059119 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8fb808ae90 con 0x7f8fc80b5a30 2024-04-16T16:13:38.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.890+0000 7f8fc5400640 1 --2- 172.21.15.45:0/3121059119 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8fa806c8f0 0x7f8fa806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:38.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.891+0000 7f8fc5400640 1 --2- 172.21.15.45:0/3121059119 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8fa806c8f0 0x7f8fa806edb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4036000 tx=0x7f8fb4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:38.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:38.894+0000 7f8fbe000640 1 -- 172.21.15.45:0/3121059119 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8fb80592c0 con 0x7f8fc80b5a30 2024-04-16T16:13:39.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.100+0000 7f8fcca00640 1 -- 172.21.15.45:0/3121059119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8fc80cdaa0 con 0x7f8fc80b5a30 2024-04-16T16:13:39.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.101+0000 7f8fbe000640 1 -- 172.21.15.45:0/3121059119 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f8fb8058bc0 con 0x7f8fc80b5a30 2024-04-16T16:13:39.101 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:13:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.104+0000 7f8fcca00640 1 -- 172.21.15.45:0/3121059119 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8fa806c8f0 msgr2=0x7f8fa806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.104+0000 7f8fcca00640 1 --2- 172.21.15.45:0/3121059119 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8fa806c8f0 0x7f8fa806edb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4036000 tx=0x7f8fb4008030 comp rx=0 tx=0).stop 2024-04-16T16:13:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.104+0000 7f8fcca00640 1 -- 172.21.15.45:0/3121059119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fc80b5a30 msgr2=0x7f8fc815e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.104+0000 7f8fcca00640 1 --2- 172.21.15.45:0/3121059119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fc80b5a30 0x7f8fc815e660 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8fb800c4d0 tx=0x7f8fb800c990 comp rx=0 tx=0).stop 2024-04-16T16:13:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.104+0000 7f8fcca00640 1 -- 172.21.15.45:0/3121059119 shutdown_connections 2024-04-16T16:13:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.104+0000 7f8fcca00640 1 --2- 172.21.15.45:0/3121059119 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8fa806c8f0 0x7f8fa806edb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.104+0000 7f8fcca00640 1 --2- 172.21.15.45:0/3121059119 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8fc815eba0 0x7f8fc80ce690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.104+0000 7f8fcca00640 1 --2- 172.21.15.45:0/3121059119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fc80d1030 0x7f8fc8152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.104+0000 7f8fcca00640 1 --2- 172.21.15.45:0/3121059119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fc80b5a30 0x7f8fc815e660 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.104+0000 7f8fcca00640 1 -- 172.21.15.45:0/3121059119 >> 172.21.15.45:0/3121059119 conn(0x7f8fc80b3970 msgr2=0x7f8fc8072250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:39.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.104+0000 7f8fcca00640 1 -- 172.21.15.45:0/3121059119 shutdown_connections 2024-04-16T16:13:39.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.104+0000 7f8fcca00640 1 -- 172.21.15.45:0/3121059119 wait complete. 2024-04-16T16:13:39.124 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:13:39.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.328+0000 7fb108e00640 1 Processor -- start 2024-04-16T16:13:39.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.328+0000 7fb108e00640 1 -- start start 2024-04-16T16:13:39.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.330+0000 7fb108e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb1040cdeb0 0x7fb1040cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:39.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.330+0000 7fb108e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb1040d0060 0x7fb1040b4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:39.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.330+0000 7fb108e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1040ce310 con 0x7fb10415b730 2024-04-16T16:13:39.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.330+0000 7fb108e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb1040ce5b0 con 0x7fb1040d0060 2024-04-16T16:13:39.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.330+0000 7fb108e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb104072420 con 0x7fb1040cdeb0 2024-04-16T16:13:39.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.330+0000 7fb101e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb10415b730 0x7fb1040b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39108/0 (socket says 172.21.15.45:39108) 2024-04-16T16:13:39.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.330+0000 7fb101e00640 1 -- 172.21.15.45:0/311404886 learned_addr learned my addr 172.21.15.45:0/311404886 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:39.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.330+0000 7fb102800640 1 --2- 172.21.15.45:0/311404886 >> v2:172.21.15.45:3300/0 conn(0x7fb1040d0060 0x7fb1040b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:39.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.331+0000 7fb0fbe00640 1 --2- 172.21.15.45:0/311404886 >> v2:172.21.15.103:3300/0 conn(0x7fb1040cdeb0 0x7fb1040cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:39.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.331+0000 7fb101400640 1 -- 172.21.15.45:0/311404886 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3699390196 0 0) 0x7fb1040ce310 con 0x7fb10415b730 2024-04-16T16:13:39.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.331+0000 7fb102800640 1 -- 172.21.15.45:0/311404886 >> v2:172.21.15.103:3300/0 conn(0x7fb1040cdeb0 msgr2=0x7fb1040cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:39.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.331+0000 7fb102800640 1 --2- 172.21.15.45:0/311404886 >> v2:172.21.15.103:3300/0 conn(0x7fb1040cdeb0 0x7fb1040cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.331+0000 7fb102800640 1 -- 172.21.15.45:0/311404886 >> v1:172.21.15.45:6789/0 conn(0x7fb10415b730 legacy=0x7fb1040b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:39.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.331+0000 7fb102800640 1 -- 172.21.15.45:0/311404886 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb104072700 con 0x7fb1040d0060 2024-04-16T16:13:39.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.332+0000 7fb102800640 1 --2- 172.21.15.45:0/311404886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1040d0060 0x7fb1040b4400 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb104070410 tx=0x7fb0fc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f5ce43cd9e3809 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:39.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.332+0000 7fb101400640 1 -- 172.21.15.45:0/311404886 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc039070 con 0x7fb1040d0060 2024-04-16T16:13:39.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.332+0000 7fb101400640 1 -- 172.21.15.45:0/311404886 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0fc02cc80 con 0x7fb1040d0060 2024-04-16T16:13:39.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.332+0000 7fb101400640 1 -- 172.21.15.45:0/311404886 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc034660 con 0x7fb1040d0060 2024-04-16T16:13:39.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.333+0000 7fb108e00640 1 -- 172.21.15.45:0/311404886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1040d0060 msgr2=0x7fb1040b4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:39.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.333+0000 7fb108e00640 1 --2- 172.21.15.45:0/311404886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1040d0060 0x7fb1040b4400 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb104070410 tx=0x7fb0fc02bed0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.333+0000 7fb108e00640 1 -- 172.21.15.45:0/311404886 shutdown_connections 2024-04-16T16:13:39.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.333+0000 7fb108e00640 1 --2- 172.21.15.45:0/311404886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1040d0060 0x7fb1040b4400 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.333+0000 7fb108e00640 1 --2- 172.21.15.45:0/311404886 >> v2:172.21.15.103:3300/0 conn(0x7fb1040cdeb0 0x7fb1040cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.333+0000 7fb108e00640 1 -- 172.21.15.45:0/311404886 >> 172.21.15.45:0/311404886 conn(0x7fb1040b2750 msgr2=0x7fb1040b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:39.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.333+0000 7fb108e00640 1 -- 172.21.15.45:0/311404886 shutdown_connections 2024-04-16T16:13:39.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.333+0000 7fb108e00640 1 -- 172.21.15.45:0/311404886 wait complete. 2024-04-16T16:13:39.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.335+0000 7fb108e00640 1 Processor -- start 2024-04-16T16:13:39.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.335+0000 7fb108e00640 1 -- start start 2024-04-16T16:13:39.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.336+0000 7fb108e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1040cdeb0 0x7fb104146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:39.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.336+0000 7fb108e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1040d0060 0x7fb104158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:39.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.336+0000 7fb108e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10415b730 0x7fb10406b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:39.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.336+0000 7fb108e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb1040d04b0 con 0x7fb1040d0060 2024-04-16T16:13:39.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.336+0000 7fb108e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb104072cc0 con 0x7fb10415b730 2024-04-16T16:13:39.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.336+0000 7fb108e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb10415bb10 con 0x7fb1040cdeb0 2024-04-16T16:13:39.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.336+0000 7fb0fbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1040d0060 0x7fb104158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:39.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.336+0000 7fb0fbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1040d0060 0x7fb104158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43494/0 (socket says 172.21.15.45:43494) 2024-04-16T16:13:39.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.336+0000 7fb0fbe00640 1 -- 172.21.15.45:0/3571446499 learned_addr learned my addr 172.21.15.45:0/3571446499 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:39.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.337+0000 7fb102800640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10415b730 0x7fb10406b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:39.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.337+0000 7fb0fbe00640 1 -- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1040cdeb0 msgr2=0x7fb104146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:39.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.337+0000 7fb101e00640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1040cdeb0 0x7fb104146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:39.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.337+0000 7fb0fbe00640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1040cdeb0 0x7fb104146470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.337+0000 7fb0fbe00640 1 -- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10415b730 msgr2=0x7fb10406b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:39.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.337+0000 7fb0fbe00640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10415b730 0x7fb10406b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.337+0000 7fb0fbe00640 1 -- 172.21.15.45:0/3571446499 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb104156fa0 con 0x7fb1040d0060 2024-04-16T16:13:39.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.337+0000 7fb102800640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10415b730 0x7fb10406b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:39.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.337+0000 7fb101e00640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1040cdeb0 0x7fb104146470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:39.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.337+0000 7fb0fbe00640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1040d0060 0x7fb104158d20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00d830 tx=0x7fb0ec00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:39.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.337+0000 7fb0fa000640 1 -- 172.21.15.45:0/3571446499 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec00b6c0 con 0x7fb1040d0060 2024-04-16T16:13:39.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.337+0000 7fb0fa000640 1 -- 172.21.15.45:0/3571446499 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0ec019070 con 0x7fb1040d0060 2024-04-16T16:13:39.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.337+0000 7fb0fa000640 1 -- 172.21.15.45:0/3571446499 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec00ba80 con 0x7fb1040d0060 2024-04-16T16:13:39.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.337+0000 7fb108e00640 1 -- 172.21.15.45:0/3571446499 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb104157280 con 0x7fb1040d0060 2024-04-16T16:13:39.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.338+0000 7fb108e00640 1 -- 172.21.15.45:0/3571446499 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb104157840 con 0x7fb1040d0060 2024-04-16T16:13:39.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.339+0000 7fb108e00640 1 -- 172.21.15.45:0/3571446499 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb104157a10 con 0x7fb1040d0060 2024-04-16T16:13:39.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.340+0000 7fb0fa000640 1 -- 172.21.15.45:0/3571446499 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb0ec01e460 con 0x7fb1040d0060 2024-04-16T16:13:39.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.341+0000 7fb0fa000640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0e006c940 0x7fb0e006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:39.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.341+0000 7fb0fa000640 1 -- 172.21.15.45:0/3571446499 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb0ec08c230 con 0x7fb1040d0060 2024-04-16T16:13:39.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.341+0000 7fb101e00640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0e006c940 0x7fb0e006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:39.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.342+0000 7fb101e00640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0e006c940 0x7fb0e006ee00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4000f60 tx=0x7fb0f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:39.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.345+0000 7fb0fa000640 1 -- 172.21.15.45:0/3571446499 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb104157a10 con 0x7fb1040d0060 2024-04-16T16:13:39.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.592+0000 7fb108e00640 1 -- 172.21.15.45:0/3571446499 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb104075780 con 0x7fb1040d0060 2024-04-16T16:13:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.593+0000 7fb0fa000640 1 -- 172.21.15.45:0/3571446499 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5178 (secure 0 0 0) 0x7fb0ec05a6d0 con 0x7fb1040d0060 2024-04-16T16:13:39.594 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:39.594 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2486891866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2486891866},{"type":"v1","addr":"172.21.15.103:6836","nonce":2486891866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1877281473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1877281473},{"type":"v1","addr":"172.21.15.103:6837","nonce":1877281473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/315848518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":315848518},{"type":"v1","addr":"172.21.15.45:6833","nonce":315848518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:13:25.369839+0000","modified":"2024-04-16T16:13:28.463144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2461453826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2461453826},{"type":"v1","addr":"172.21.15.45:6835","nonce":2461453826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-04-16T16:13:39.594 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 8 2024-04-16T16:13:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.596+0000 7fb108e00640 1 -- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0e006c940 msgr2=0x7fb0e006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.596+0000 7fb108e00640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0e006c940 0x7fb0e006ee00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4000f60 tx=0x7fb0f4009040 comp rx=0 tx=0).stop 2024-04-16T16:13:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.596+0000 7fb108e00640 1 -- 172.21.15.45:0/3571446499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1040d0060 msgr2=0x7fb104158d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:39.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.596+0000 7fb108e00640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1040d0060 0x7fb104158d20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00d830 tx=0x7fb0ec00dcf0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.596+0000 7fb108e00640 1 -- 172.21.15.45:0/3571446499 shutdown_connections 2024-04-16T16:13:39.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.596+0000 7fb108e00640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0e006c940 0x7fb0e006ee00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.596+0000 7fb108e00640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10415b730 0x7fb10406b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.596+0000 7fb108e00640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1040d0060 0x7fb104158d20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.596+0000 7fb108e00640 1 --2- 172.21.15.45:0/3571446499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1040cdeb0 0x7fb104146470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.596+0000 7fb108e00640 1 -- 172.21.15.45:0/3571446499 >> 172.21.15.45:0/3571446499 conn(0x7fb1040b2750 msgr2=0x7fb10415fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:39.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.596+0000 7fb108e00640 1 -- 172.21.15.45:0/3571446499 shutdown_connections 2024-04-16T16:13:39.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.596+0000 7fb108e00640 1 -- 172.21.15.45:0/3571446499 wait complete. 2024-04-16T16:13:39.614 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:13:39.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.835+0000 7f6187000640 1 Processor -- start 2024-04-16T16:13:39.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.835+0000 7f6187000640 1 -- start start 2024-04-16T16:13:39.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.836+0000 7f6187000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f618009af60 0x7f618006b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:39.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.836+0000 7f6187000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f618006be70 0x7f61800cf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:39.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.836+0000 7f6187000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f618006f490 con 0x7f618015ecc0 2024-04-16T16:13:39.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.836+0000 7f6187000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f61800703f0 con 0x7f618006be70 2024-04-16T16:13:39.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.836+0000 7f6187000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f61800b3f60 con 0x7f618009af60 2024-04-16T16:13:39.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.837+0000 7f6185000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f618006be70 0x7f61800cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:39.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.837+0000 7f617fe00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f618015ecc0 0x7f618006fce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:33812/0 (socket says 172.21.15.45:33812) 2024-04-16T16:13:39.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.837+0000 7f617fe00640 1 -- 172.21.15.45:0/2882835917 learned_addr learned my addr 172.21.15.45:0/2882835917 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:39.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.837+0000 7f6177400640 1 --2- 172.21.15.45:0/2882835917 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f618009af60 0x7f618006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:39.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.837+0000 7f6185000640 1 -- 172.21.15.45:0/2882835917 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f618009af60 msgr2=0x7f618006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:39.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.837+0000 7f6185000640 1 --2- 172.21.15.45:0/2882835917 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f618009af60 0x7f618006b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.837+0000 7f6185000640 1 -- 172.21.15.45:0/2882835917 >> v1:172.21.15.103:6789/0 conn(0x7f618015ecc0 legacy=0x7f618006fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:39.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.837+0000 7f6185000640 1 -- 172.21.15.45:0/2882835917 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618006f730 con 0x7f618006be70 2024-04-16T16:13:39.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.839+0000 7f6185000640 1 --2- 172.21.15.45:0/2882835917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618006be70 0x7f61800cf580 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6178009d90 tx=0x7f617802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c85d51b3abf188c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:39.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.839+0000 7f617f400640 1 -- 172.21.15.45:0/2882835917 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178039070 con 0x7f618006be70 2024-04-16T16:13:39.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.839+0000 7f617f400640 1 -- 172.21.15.45:0/2882835917 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f617802cc50 con 0x7f618006be70 2024-04-16T16:13:39.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.839+0000 7f617f400640 1 -- 172.21.15.45:0/2882835917 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178034440 con 0x7f618006be70 2024-04-16T16:13:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.840+0000 7f6187000640 1 -- 172.21.15.45:0/2882835917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618006be70 msgr2=0x7f61800cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.840+0000 7f6187000640 1 --2- 172.21.15.45:0/2882835917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618006be70 0x7f61800cf580 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6178009d90 tx=0x7f617802bea0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.840+0000 7f6187000640 1 -- 172.21.15.45:0/2882835917 shutdown_connections 2024-04-16T16:13:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.840+0000 7f6187000640 1 --2- 172.21.15.45:0/2882835917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618006be70 0x7f61800cf580 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.840+0000 7f6187000640 1 --2- 172.21.15.45:0/2882835917 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f618009af60 0x7f618006b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.840+0000 7f6187000640 1 -- 172.21.15.45:0/2882835917 >> 172.21.15.45:0/2882835917 conn(0x7f618009aa70 msgr2=0x7f61800b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.840+0000 7f6187000640 1 -- 172.21.15.45:0/2882835917 shutdown_connections 2024-04-16T16:13:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.840+0000 7f6187000640 1 -- 172.21.15.45:0/2882835917 wait complete. 2024-04-16T16:13:39.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.841+0000 7f6187000640 1 Processor -- start 2024-04-16T16:13:39.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.842+0000 7f6187000640 1 -- start start 2024-04-16T16:13:39.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.842+0000 7f6187000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618006be70 0x7f618015a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:39.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.842+0000 7f6187000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f618009af60 0x7f61800af900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:39.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.843+0000 7f6187000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f618015ecc0 0x7f618014ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:39.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.843+0000 7f6187000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f618015f220 con 0x7f618006be70 2024-04-16T16:13:39.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.843+0000 7f6187000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f618015f0a0 con 0x7f618009af60 2024-04-16T16:13:39.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.843+0000 7f6187000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f61800cfac0 con 0x7f618015ecc0 2024-04-16T16:13:39.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.843+0000 7f617fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618006be70 0x7f618015a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:39.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.843+0000 7f617fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618006be70 0x7f618015a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43518/0 (socket says 172.21.15.45:43518) 2024-04-16T16:13:39.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.843+0000 7f617fe00640 1 -- 172.21.15.45:0/1263881161 learned_addr learned my addr 172.21.15.45:0/1263881161 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:39.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.843+0000 7f617fe00640 1 -- 172.21.15.45:0/1263881161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f618015ecc0 msgr2=0x7f618014ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:39.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.843+0000 7f6185000640 1 --2- 172.21.15.45:0/1263881161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f618015ecc0 0x7f618014ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:39.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.843+0000 7f6177400640 1 --2- 172.21.15.45:0/1263881161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f618009af60 0x7f61800af900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:39.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.843+0000 7f617fe00640 1 --2- 172.21.15.45:0/1263881161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f618015ecc0 0x7f618014ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.843+0000 7f617fe00640 1 -- 172.21.15.45:0/1263881161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f618009af60 msgr2=0x7f61800af900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:39.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.843+0000 7f617fe00640 1 --2- 172.21.15.45:0/1263881161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f618009af60 0x7f61800af900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:39.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.843+0000 7f617fe00640 1 -- 172.21.15.45:0/1263881161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618015b0f0 con 0x7f618006be70 2024-04-16T16:13:39.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.844+0000 7f617fe00640 1 --2- 172.21.15.45:0/1263881161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618006be70 0x7f618015a750 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f617000c3c0 tx=0x7f617000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:39.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.844+0000 7f617cc00640 1 -- 172.21.15.45:0/1263881161 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170017070 con 0x7f618006be70 2024-04-16T16:13:39.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.844+0000 7f617cc00640 1 -- 172.21.15.45:0/1263881161 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6170005ce0 con 0x7f618006be70 2024-04-16T16:13:39.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.844+0000 7f617cc00640 1 -- 172.21.15.45:0/1263881161 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170007310 con 0x7f618006be70 2024-04-16T16:13:39.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.844+0000 7f6187000640 1 -- 172.21.15.45:0/1263881161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6180160fa0 con 0x7f618006be70 2024-04-16T16:13:39.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.844+0000 7f6187000640 1 -- 172.21.15.45:0/1263881161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6180161560 con 0x7f618006be70 2024-04-16T16:13:39.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.845+0000 7f6187000640 1 -- 172.21.15.45:0/1263881161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6180075780 con 0x7f618006be70 2024-04-16T16:13:39.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.846+0000 7f617cc00640 1 -- 172.21.15.45:0/1263881161 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f617001a070 con 0x7f618006be70 2024-04-16T16:13:39.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.847+0000 7f617cc00640 1 --2- 172.21.15.45:0/1263881161 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f615c06c8f0 0x7f615c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:39.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.847+0000 7f617cc00640 1 -- 172.21.15.45:0/1263881161 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f617008af50 con 0x7f618006be70 2024-04-16T16:13:39.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.848+0000 7f6177400640 1 --2- 172.21.15.45:0/1263881161 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f615c06c8f0 0x7f615c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:39.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.849+0000 7f6177400640 1 --2- 172.21.15.45:0/1263881161 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f615c06c8f0 0x7f615c06edb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6168009a10 tx=0x7f6168008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:39.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:39.852+0000 7f617cc00640 1 -- 172.21.15.45:0/1263881161 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f61700592e0 con 0x7f618006be70 2024-04-16T16:13:40.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.060+0000 7f6187000640 1 -- 172.21.15.45:0/1263881161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f61800b0b90 con 0x7f618006be70 2024-04-16T16:13:40.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.063+0000 7f617cc00640 1 -- 172.21.15.45:0/1263881161 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f6170058be0 con 0x7f618006be70 2024-04-16T16:13:40.063 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:40.064 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":21,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:26.918617+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:13:40.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.066+0000 7f6187000640 1 -- 172.21.15.45:0/1263881161 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f615c06c8f0 msgr2=0x7f615c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:40.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.066+0000 7f6187000640 1 --2- 172.21.15.45:0/1263881161 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f615c06c8f0 0x7f615c06edb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6168009a10 tx=0x7f6168008040 comp rx=0 tx=0).stop 2024-04-16T16:13:40.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.066+0000 7f6187000640 1 -- 172.21.15.45:0/1263881161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618006be70 msgr2=0x7f618015a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:40.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.066+0000 7f6187000640 1 --2- 172.21.15.45:0/1263881161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618006be70 0x7f618015a750 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f617000c3c0 tx=0x7f617000c880 comp rx=0 tx=0).stop 2024-04-16T16:13:40.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.066+0000 7f6187000640 1 -- 172.21.15.45:0/1263881161 shutdown_connections 2024-04-16T16:13:40.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.066+0000 7f6187000640 1 --2- 172.21.15.45:0/1263881161 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f615c06c8f0 0x7f615c06edb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.066+0000 7f6187000640 1 --2- 172.21.15.45:0/1263881161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f618015ecc0 0x7f618014ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.066+0000 7f6187000640 1 --2- 172.21.15.45:0/1263881161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f618009af60 0x7f61800af900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.066+0000 7f6187000640 1 --2- 172.21.15.45:0/1263881161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618006be70 0x7f618015a750 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.066+0000 7f6187000640 1 -- 172.21.15.45:0/1263881161 >> 172.21.15.45:0/1263881161 conn(0x7f618009aa70 msgr2=0x7f61800cd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:40.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.066+0000 7f6187000640 1 -- 172.21.15.45:0/1263881161 shutdown_connections 2024-04-16T16:13:40.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.066+0000 7f6187000640 1 -- 172.21.15.45:0/1263881161 wait complete. 2024-04-16T16:13:40.087 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:13:23.600104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:13:24.368141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:13:40.087 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:13:40.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.317+0000 7ff2ed800640 1 Processor -- start 2024-04-16T16:13:40.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.317+0000 7ff2ed800640 1 -- start start 2024-04-16T16:13:40.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.318+0000 7ff2ed800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff2e815f830 0x7ff2e806e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:40.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.318+0000 7ff2ed800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff2e815b930 0x7ff2e806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:40.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.319+0000 7ff2ed800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e80ce7b0 con 0x7ff2e815b550 2024-04-16T16:13:40.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.319+0000 7ff2ed800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff2e80ced90 con 0x7ff2e815b930 2024-04-16T16:13:40.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.319+0000 7ff2ed800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff2e80d0230 con 0x7ff2e815f830 2024-04-16T16:13:40.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.319+0000 7ff2ece00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff2e815b930 0x7ff2e806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:40.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.319+0000 7ff2ece00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff2e815b930 0x7ff2e806ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43532/0 (socket says 172.21.15.45:43532) 2024-04-16T16:13:40.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.319+0000 7ff2ece00640 1 -- 172.21.15.45:0/570100122 learned_addr learned my addr 172.21.15.45:0/570100122 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:40.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.319+0000 7ff2dfe00640 1 --2- 172.21.15.45:0/570100122 >> v2:172.21.15.103:3300/0 conn(0x7ff2e815f830 0x7ff2e806e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:40.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.319+0000 7ff2ece00640 1 -- 172.21.15.45:0/570100122 >> v2:172.21.15.103:3300/0 conn(0x7ff2e815f830 msgr2=0x7ff2e806e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:40.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.319+0000 7ff2ece00640 1 --2- 172.21.15.45:0/570100122 >> v2:172.21.15.103:3300/0 conn(0x7ff2e815f830 0x7ff2e806e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.319+0000 7ff2ece00640 1 -- 172.21.15.45:0/570100122 >> v1:172.21.15.45:6789/0 conn(0x7ff2e815b550 legacy=0x7ff2e815f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:40.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.319+0000 7ff2ece00640 1 -- 172.21.15.45:0/570100122 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e806f440 con 0x7ff2e815b930 2024-04-16T16:13:40.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.319+0000 7ff2e7400640 1 -- 172.21.15.45:0/570100122 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2034666851 0 0) 0x7ff2e80ce7b0 con 0x7ff2e815b550 2024-04-16T16:13:40.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.321+0000 7ff2ece00640 1 --2- 172.21.15.45:0/570100122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2e815b930 0x7ff2e806ef00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff2e80d0700 tx=0x7ff2e002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac876a592a9306e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:40.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.321+0000 7ff2e7400640 1 -- 172.21.15.45:0/570100122 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e002cb20 con 0x7ff2e815b930 2024-04-16T16:13:40.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.321+0000 7ff2e7400640 1 -- 172.21.15.45:0/570100122 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2e002ccc0 con 0x7ff2e815b930 2024-04-16T16:13:40.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.321+0000 7ff2e7400640 1 -- 172.21.15.45:0/570100122 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e0034610 con 0x7ff2e815b930 2024-04-16T16:13:40.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.321+0000 7ff2ed800640 1 -- 172.21.15.45:0/570100122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2e815b930 msgr2=0x7ff2e806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:40.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.321+0000 7ff2ed800640 1 --2- 172.21.15.45:0/570100122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2e815b930 0x7ff2e806ef00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff2e80d0700 tx=0x7ff2e002bf80 comp rx=0 tx=0).stop 2024-04-16T16:13:40.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.321+0000 7ff2ed800640 1 -- 172.21.15.45:0/570100122 shutdown_connections 2024-04-16T16:13:40.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.321+0000 7ff2ed800640 1 --2- 172.21.15.45:0/570100122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2e815b930 0x7ff2e806ef00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.322+0000 7ff2ed800640 1 --2- 172.21.15.45:0/570100122 >> v2:172.21.15.103:3300/0 conn(0x7ff2e815f830 0x7ff2e806e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.322+0000 7ff2ed800640 1 -- 172.21.15.45:0/570100122 >> 172.21.15.45:0/570100122 conn(0x7ff2e8067e00 msgr2=0x7ff2e806b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:40.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.322+0000 7ff2ed800640 1 -- 172.21.15.45:0/570100122 shutdown_connections 2024-04-16T16:13:40.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.322+0000 7ff2ed800640 1 -- 172.21.15.45:0/570100122 wait complete. 2024-04-16T16:13:40.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.323+0000 7ff2ed800640 1 Processor -- start 2024-04-16T16:13:40.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.324+0000 7ff2ed800640 1 -- start start 2024-04-16T16:13:40.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.324+0000 7ff2ed800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2e815b550 0x7ff2e81563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:40.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.324+0000 7ff2ed800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2e815b930 0x7ff2e81e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:40.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.324+0000 7ff2ed800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2e815f830 0x7ff2e814da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:40.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.324+0000 7ff2ed800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e806f990 con 0x7ff2e815b550 2024-04-16T16:13:40.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.324+0000 7ff2ed800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e80cea20 con 0x7ff2e815b930 2024-04-16T16:13:40.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.324+0000 7ff2ed800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff2e80cf220 con 0x7ff2e815f830 2024-04-16T16:13:40.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.325+0000 7ff2e7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2e815b550 0x7ff2e81563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:40.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.325+0000 7ff2e7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2e815b550 0x7ff2e81563b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43534/0 (socket says 172.21.15.45:43534) 2024-04-16T16:13:40.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.325+0000 7ff2e7e00640 1 -- 172.21.15.45:0/119355442 learned_addr learned my addr 172.21.15.45:0/119355442 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:40.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.325+0000 7ff2dfe00640 1 --2- 172.21.15.45:0/119355442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2e815b930 0x7ff2e81e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:40.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.325+0000 7ff2ece00640 1 --2- 172.21.15.45:0/119355442 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2e815f830 0x7ff2e814da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:40.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.325+0000 7ff2e7e00640 1 -- 172.21.15.45:0/119355442 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2e815f830 msgr2=0x7ff2e814da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:40.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.325+0000 7ff2e7e00640 1 --2- 172.21.15.45:0/119355442 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2e815f830 0x7ff2e814da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.325+0000 7ff2e7e00640 1 -- 172.21.15.45:0/119355442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2e815b930 msgr2=0x7ff2e81e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:40.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.325+0000 7ff2e7e00640 1 --2- 172.21.15.45:0/119355442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2e815b930 0x7ff2e81e71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.325+0000 7ff2e7e00640 1 -- 172.21.15.45:0/119355442 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e814df70 con 0x7ff2e815b550 2024-04-16T16:13:40.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.326+0000 7ff2e7e00640 1 --2- 172.21.15.45:0/119355442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2e815b550 0x7ff2e81563b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff2d800c4b0 tx=0x7ff2d800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:40.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.326+0000 7ff2e4c00640 1 -- 172.21.15.45:0/119355442 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8017070 con 0x7ff2e815b550 2024-04-16T16:13:40.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.326+0000 7ff2e4c00640 1 -- 172.21.15.45:0/119355442 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2d8005480 con 0x7ff2e815b550 2024-04-16T16:13:40.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.326+0000 7ff2e4c00640 1 -- 172.21.15.45:0/119355442 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8003830 con 0x7ff2e815b550 2024-04-16T16:13:40.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.326+0000 7ff2ed800640 1 -- 172.21.15.45:0/119355442 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2e81e7950 con 0x7ff2e815b550 2024-04-16T16:13:40.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.326+0000 7ff2ed800640 1 -- 172.21.15.45:0/119355442 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2e81e5340 con 0x7ff2e815b550 2024-04-16T16:13:40.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.327+0000 7ff2ed800640 1 -- 172.21.15.45:0/119355442 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2e8075780 con 0x7ff2e815b550 2024-04-16T16:13:40.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.328+0000 7ff2e4c00640 1 -- 172.21.15.45:0/119355442 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff2d801a070 con 0x7ff2e815b550 2024-04-16T16:13:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.330+0000 7ff2e4c00640 1 --2- 172.21.15.45:0/119355442 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff2c406c8f0 0x7ff2c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.330+0000 7ff2e4c00640 1 -- 172.21.15.45:0/119355442 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff2d808aee0 con 0x7ff2e815b550 2024-04-16T16:13:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.330+0000 7ff2dfe00640 1 --2- 172.21.15.45:0/119355442 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff2c406c8f0 0x7ff2c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:40.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.331+0000 7ff2dfe00640 1 --2- 172.21.15.45:0/119355442 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff2c406c8f0 0x7ff2c406edb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0006fd0 tx=0x7ff2d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:40.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.334+0000 7ff2e4c00640 1 -- 172.21.15.45:0/119355442 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff2d8059300 con 0x7ff2e815b550 2024-04-16T16:13:40.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.537+0000 7ff2ed800640 1 -- 172.21.15.45:0/119355442 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff2e815ee90 con 0x7ff2e815b550 2024-04-16T16:13:40.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.538+0000 7ff2e4c00640 1 -- 172.21.15.45:0/119355442 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7ff2d8058c00 con 0x7ff2e815b550 2024-04-16T16:13:40.538 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:13:40.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.540+0000 7ff2ed800640 1 -- 172.21.15.45:0/119355442 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff2c406c8f0 msgr2=0x7ff2c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:40.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.540+0000 7ff2ed800640 1 --2- 172.21.15.45:0/119355442 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff2c406c8f0 0x7ff2c406edb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0006fd0 tx=0x7ff2d0008040 comp rx=0 tx=0).stop 2024-04-16T16:13:40.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.540+0000 7ff2ed800640 1 -- 172.21.15.45:0/119355442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2e815b550 msgr2=0x7ff2e81563b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:40.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.540+0000 7ff2ed800640 1 --2- 172.21.15.45:0/119355442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2e815b550 0x7ff2e81563b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff2d800c4b0 tx=0x7ff2d800c970 comp rx=0 tx=0).stop 2024-04-16T16:13:40.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.541+0000 7ff2ed800640 1 -- 172.21.15.45:0/119355442 shutdown_connections 2024-04-16T16:13:40.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.541+0000 7ff2ed800640 1 --2- 172.21.15.45:0/119355442 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff2c406c8f0 0x7ff2c406edb0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.541+0000 7ff2ed800640 1 --2- 172.21.15.45:0/119355442 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2e815f830 0x7ff2e814da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.541+0000 7ff2ed800640 1 --2- 172.21.15.45:0/119355442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2e815b930 0x7ff2e81e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.541+0000 7ff2ed800640 1 --2- 172.21.15.45:0/119355442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2e815b550 0x7ff2e81563b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.541+0000 7ff2ed800640 1 -- 172.21.15.45:0/119355442 >> 172.21.15.45:0/119355442 conn(0x7ff2e8067e00 msgr2=0x7ff2e815f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:40.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.541+0000 7ff2ed800640 1 -- 172.21.15.45:0/119355442 shutdown_connections 2024-04-16T16:13:40.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.541+0000 7ff2ed800640 1 -- 172.21.15.45:0/119355442 wait complete. 2024-04-16T16:13:40.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:13:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.781+0000 7fba7a600640 1 Processor -- start 2024-04-16T16:13:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.781+0000 7fba7a600640 1 -- start start 2024-04-16T16:13:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.782+0000 7fba7a600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba7415b920 0x7fba740b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.782+0000 7fba7a600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fba740d0e80 0x7fba7406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.782+0000 7fba7a600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba740ccb70 con 0x7fba740afda0 2024-04-16T16:13:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.782+0000 7fba7a600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fba7406e7a0 con 0x7fba740d0e80 2024-04-16T16:13:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.782+0000 7fba7a600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fba7406f170 con 0x7fba7415b920 2024-04-16T16:13:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.782+0000 7fba73e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fba740d0e80 0x7fba7406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.782+0000 7fba73e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fba740d0e80 0x7fba7406ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43542/0 (socket says 172.21.15.45:43542) 2024-04-16T16:13:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.782+0000 7fba73e00640 1 -- 172.21.15.45:0/2952032906 learned_addr learned my addr 172.21.15.45:0/2952032906 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.783+0000 7fba73400640 1 --2- 172.21.15.45:0/2952032906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba7415b920 0x7fba740b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.783+0000 7fba73e00640 1 -- 172.21.15.45:0/2952032906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba7415b920 msgr2=0x7fba740b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.783+0000 7fba73e00640 1 --2- 172.21.15.45:0/2952032906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba7415b920 0x7fba740b4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.783+0000 7fba73e00640 1 -- 172.21.15.45:0/2952032906 >> v1:172.21.15.103:6789/0 conn(0x7fba740afda0 legacy=0x7fba740d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.783+0000 7fba73e00640 1 -- 172.21.15.45:0/2952032906 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba740cce50 con 0x7fba740d0e80 2024-04-16T16:13:40.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.784+0000 7fba73e00640 1 --2- 172.21.15.45:0/2952032906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740d0e80 0x7fba7406ec30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fba6c009870 tx=0x7fba6c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29510328e6099288 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:40.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.784+0000 7fba72a00640 1 -- 172.21.15.45:0/2952032906 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6c035020 con 0x7fba740d0e80 2024-04-16T16:13:40.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.784+0000 7fba72a00640 1 -- 172.21.15.45:0/2952032906 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba6c031640 con 0x7fba740d0e80 2024-04-16T16:13:40.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.784+0000 7fba72a00640 1 -- 172.21.15.45:0/2952032906 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6c02d040 con 0x7fba740d0e80 2024-04-16T16:13:40.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.785+0000 7fba7a600640 1 -- 172.21.15.45:0/2952032906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740d0e80 msgr2=0x7fba7406ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:40.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.785+0000 7fba7a600640 1 --2- 172.21.15.45:0/2952032906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740d0e80 0x7fba7406ec30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fba6c009870 tx=0x7fba6c02b8f0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.785+0000 7fba7a600640 1 -- 172.21.15.45:0/2952032906 shutdown_connections 2024-04-16T16:13:40.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.785+0000 7fba7a600640 1 --2- 172.21.15.45:0/2952032906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740d0e80 0x7fba7406ec30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.785+0000 7fba7a600640 1 --2- 172.21.15.45:0/2952032906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba7415b920 0x7fba740b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.785+0000 7fba7a600640 1 -- 172.21.15.45:0/2952032906 >> 172.21.15.45:0/2952032906 conn(0x7fba740b3970 msgr2=0x7fba740b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:40.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.785+0000 7fba7a600640 1 -- 172.21.15.45:0/2952032906 shutdown_connections 2024-04-16T16:13:40.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.785+0000 7fba7a600640 1 -- 172.21.15.45:0/2952032906 wait complete. 2024-04-16T16:13:40.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.786+0000 7fba7a600640 1 Processor -- start 2024-04-16T16:13:40.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.787+0000 7fba7a600640 1 -- start start 2024-04-16T16:13:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.787+0000 7fba7a600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba740afda0 0x7fba74152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.787+0000 7fba7a600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740d0e80 0x7fba74148ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.787+0000 7fba7a600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba7415b920 0x7fba74149530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.787+0000 7fba7a600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fba740ce760 con 0x7fba740d0e80 2024-04-16T16:13:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.787+0000 7fba7a600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fba740ce5e0 con 0x7fba7415b920 2024-04-16T16:13:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.787+0000 7fba7a600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fba740ce8e0 con 0x7fba740afda0 2024-04-16T16:13:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.788+0000 7fba6aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740d0e80 0x7fba74148ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.788+0000 7fba6aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740d0e80 0x7fba74148ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43556/0 (socket says 172.21.15.45:43556) 2024-04-16T16:13:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.788+0000 7fba6aa00640 1 -- 172.21.15.45:0/2590674 learned_addr learned my addr 172.21.15.45:0/2590674 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.788+0000 7fba73e00640 1 --2- 172.21.15.45:0/2590674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba7415b920 0x7fba74149530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.788+0000 7fba6aa00640 1 -- 172.21.15.45:0/2590674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba740afda0 msgr2=0x7fba74152db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.788+0000 7fba6aa00640 1 --2- 172.21.15.45:0/2590674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba740afda0 0x7fba74152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.788+0000 7fba6aa00640 1 -- 172.21.15.45:0/2590674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba7415b920 msgr2=0x7fba74149530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.788+0000 7fba6aa00640 1 --2- 172.21.15.45:0/2590674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba7415b920 0x7fba74149530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.788+0000 7fba6aa00640 1 -- 172.21.15.45:0/2590674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba7415e820 con 0x7fba740d0e80 2024-04-16T16:13:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.788+0000 7fba6aa00640 1 --2- 172.21.15.45:0/2590674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740d0e80 0x7fba74148ff0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fba5c00c460 tx=0x7fba5c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.789+0000 7fba6be00640 1 -- 172.21.15.45:0/2590674 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c016020 con 0x7fba740d0e80 2024-04-16T16:13:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.789+0000 7fba6be00640 1 -- 172.21.15.45:0/2590674 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba5c012870 con 0x7fba740d0e80 2024-04-16T16:13:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.789+0000 7fba6be00640 1 -- 172.21.15.45:0/2590674 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c012c60 con 0x7fba740d0e80 2024-04-16T16:13:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.789+0000 7fba7a600640 1 -- 172.21.15.45:0/2590674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba741532f0 con 0x7fba740d0e80 2024-04-16T16:13:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.789+0000 7fba7a600640 1 -- 172.21.15.45:0/2590674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba740b0760 con 0x7fba740d0e80 2024-04-16T16:13:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.790+0000 7fba7a600640 1 -- 172.21.15.45:0/2590674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba74075780 con 0x7fba740d0e80 2024-04-16T16:13:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.791+0000 7fba6be00640 1 -- 172.21.15.45:0/2590674 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fba5c019020 con 0x7fba740d0e80 2024-04-16T16:13:40.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.792+0000 7fba6be00640 1 --2- 172.21.15.45:0/2590674 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fba5006c8f0 0x7fba5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:40.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.792+0000 7fba6be00640 1 -- 172.21.15.45:0/2590674 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fba5c08b5e0 con 0x7fba740d0e80 2024-04-16T16:13:40.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.792+0000 7fba73400640 1 --2- 172.21.15.45:0/2590674 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fba5006c8f0 0x7fba5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:40.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.794+0000 7fba73400640 1 --2- 172.21.15.45:0/2590674 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fba5006c8f0 0x7fba5006edb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fba64007dc0 tx=0x7fba640072f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:40.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.797+0000 7fba6be00640 1 -- 172.21.15.45:0/2590674 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fba5c020070 con 0x7fba740d0e80 2024-04-16T16:13:40.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.994+0000 7fba7a600640 1 -- 172.21.15.45:0/2590674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fba740b45c0 con 0x7fba740d0e80 2024-04-16T16:13:40.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.995+0000 7fba6be00640 1 -- 172.21.15.45:0/2590674 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fba5c059a00 con 0x7fba740d0e80 2024-04-16T16:13:40.995 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:13:40.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.998+0000 7fba7a600640 1 -- 172.21.15.45:0/2590674 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fba5006c8f0 msgr2=0x7fba5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:40.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.998+0000 7fba7a600640 1 --2- 172.21.15.45:0/2590674 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fba5006c8f0 0x7fba5006edb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fba64007dc0 tx=0x7fba640072f0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.998+0000 7fba7a600640 1 -- 172.21.15.45:0/2590674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740d0e80 msgr2=0x7fba74148ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:40.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.998+0000 7fba7a600640 1 --2- 172.21.15.45:0/2590674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740d0e80 0x7fba74148ff0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fba5c00c460 tx=0x7fba5c00c920 comp rx=0 tx=0).stop 2024-04-16T16:13:40.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.998+0000 7fba7a600640 1 -- 172.21.15.45:0/2590674 shutdown_connections 2024-04-16T16:13:40.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.998+0000 7fba7a600640 1 --2- 172.21.15.45:0/2590674 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fba5006c8f0 0x7fba5006edb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.998+0000 7fba7a600640 1 --2- 172.21.15.45:0/2590674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba7415b920 0x7fba74149530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.998+0000 7fba7a600640 1 --2- 172.21.15.45:0/2590674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740d0e80 0x7fba74148ff0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.998+0000 7fba7a600640 1 --2- 172.21.15.45:0/2590674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba740afda0 0x7fba74152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:40.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.998+0000 7fba7a600640 1 -- 172.21.15.45:0/2590674 >> 172.21.15.45:0/2590674 conn(0x7fba740b3970 msgr2=0x7fba7406f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:40.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.998+0000 7fba7a600640 1 -- 172.21.15.45:0/2590674 shutdown_connections 2024-04-16T16:13:40.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:40.998+0000 7fba7a600640 1 -- 172.21.15.45:0/2590674 wait complete. 2024-04-16T16:13:41.016 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:13:41.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.238+0000 7fe372c00640 1 Processor -- start 2024-04-16T16:13:41.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.238+0000 7fe372c00640 1 -- start start 2024-04-16T16:13:41.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.240+0000 7fe372c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe36c1461b0 0x7fe36c160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:41.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.240+0000 7fe372c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe36c0b5d50 0x7fe36c0cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:41.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.240+0000 7fe372c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe36c146590 con 0x7fe36c0b58f0 2024-04-16T16:13:41.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.240+0000 7fe372c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe36c06dfd0 con 0x7fe36c0b5d50 2024-04-16T16:13:41.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.240+0000 7fe372c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe36c0c5750 con 0x7fe36c1461b0 2024-04-16T16:13:41.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.240+0000 7fe371800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe36c1461b0 0x7fe36c160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:41.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.240+0000 7fe372200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe36c0b5d50 0x7fe36c0cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:41.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.241+0000 7fe371800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe36c1461b0 0x7fe36c160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:53940/0 (socket says 172.21.15.45:53940) 2024-04-16T16:13:41.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.241+0000 7fe371800640 1 -- 172.21.15.45:0/1207941632 learned_addr learned my addr 172.21.15.45:0/1207941632 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:41.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.241+0000 7fe370e00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fe36c0b58f0 0x7fe36c0cf440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:33830/0 (socket says 172.21.15.45:33830) 2024-04-16T16:13:41.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.241+0000 7fe372200640 1 -- 172.21.15.45:0/1207941632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe36c1461b0 msgr2=0x7fe36c160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:41.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.241+0000 7fe372200640 1 --2- 172.21.15.45:0/1207941632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe36c1461b0 0x7fe36c160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.241+0000 7fe372200640 1 -- 172.21.15.45:0/1207941632 >> v1:172.21.15.103:6789/0 conn(0x7fe36c0b58f0 legacy=0x7fe36c0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:41.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.241+0000 7fe372200640 1 -- 172.21.15.45:0/1207941632 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe36c161230 con 0x7fe36c0b5d50 2024-04-16T16:13:41.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.243+0000 7fe372200640 1 --2- 172.21.15.45:0/1207941632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe36c0b5d50 0x7fe36c0cfb50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe364009d90 tx=0x7fe36402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65ddeefc8373fb39 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:41.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.243+0000 7fe36be00640 1 -- 172.21.15.45:0/1207941632 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364039070 con 0x7fe36c0b5d50 2024-04-16T16:13:41.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.243+0000 7fe36be00640 1 -- 172.21.15.45:0/1207941632 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36402cc50 con 0x7fe36c0b5d50 2024-04-16T16:13:41.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.243+0000 7fe36be00640 1 -- 172.21.15.45:0/1207941632 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364034440 con 0x7fe36c0b5d50 2024-04-16T16:13:41.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.244+0000 7fe372c00640 1 -- 172.21.15.45:0/1207941632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe36c0b5d50 msgr2=0x7fe36c0cfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:41.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.244+0000 7fe372c00640 1 --2- 172.21.15.45:0/1207941632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe36c0b5d50 0x7fe36c0cfb50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe364009d90 tx=0x7fe36402bea0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.244+0000 7fe372c00640 1 -- 172.21.15.45:0/1207941632 shutdown_connections 2024-04-16T16:13:41.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.244+0000 7fe372c00640 1 --2- 172.21.15.45:0/1207941632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe36c0b5d50 0x7fe36c0cfb50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.244+0000 7fe372c00640 1 --2- 172.21.15.45:0/1207941632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe36c1461b0 0x7fe36c160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.244+0000 7fe372c00640 1 -- 172.21.15.45:0/1207941632 >> 172.21.15.45:0/1207941632 conn(0x7fe36c06b930 msgr2=0x7fe36c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:41.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.244+0000 7fe372c00640 1 -- 172.21.15.45:0/1207941632 shutdown_connections 2024-04-16T16:13:41.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.244+0000 7fe372c00640 1 -- 172.21.15.45:0/1207941632 wait complete. 2024-04-16T16:13:41.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.246+0000 7fe372c00640 1 Processor -- start 2024-04-16T16:13:41.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.246+0000 7fe372c00640 1 -- start start 2024-04-16T16:13:41.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.247+0000 7fe372c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe36c0b58f0 0x7fe36c1f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:41.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.247+0000 7fe372c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe36c0b5d50 0x7fe36c0cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:41.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.247+0000 7fe372c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe36c1461b0 0x7fe36c1f43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:41.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.247+0000 7fe372c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c0c5cd0 con 0x7fe36c0b5d50 2024-04-16T16:13:41.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.247+0000 7fe372c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c0c59f0 con 0x7fe36c1461b0 2024-04-16T16:13:41.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.247+0000 7fe372c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe36c0c5fb0 con 0x7fe36c0b58f0 2024-04-16T16:13:41.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.247+0000 7fe370e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe36c0b5d50 0x7fe36c0cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:41.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.247+0000 7fe370e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe36c0b5d50 0x7fe36c0cc040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43568/0 (socket says 172.21.15.45:43568) 2024-04-16T16:13:41.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.247+0000 7fe370e00640 1 -- 172.21.15.45:0/1516283286 learned_addr learned my addr 172.21.15.45:0/1516283286 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:41.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.247+0000 7fe371800640 1 --2- 172.21.15.45:0/1516283286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe36c0b58f0 0x7fe36c1f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:41.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.248+0000 7fe370e00640 1 -- 172.21.15.45:0/1516283286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe36c0b58f0 msgr2=0x7fe36c1f55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:41.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.248+0000 7fe370e00640 1 --2- 172.21.15.45:0/1516283286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe36c0b58f0 0x7fe36c1f55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.248+0000 7fe372200640 1 --2- 172.21.15.45:0/1516283286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe36c1461b0 0x7fe36c1f43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:41.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.248+0000 7fe370e00640 1 -- 172.21.15.45:0/1516283286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe36c1461b0 msgr2=0x7fe36c1f43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:41.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.248+0000 7fe370e00640 1 --2- 172.21.15.45:0/1516283286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe36c1461b0 0x7fe36c1f43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.248+0000 7fe370e00640 1 -- 172.21.15.45:0/1516283286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe36c1f4f10 con 0x7fe36c0b5d50 2024-04-16T16:13:41.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.248+0000 7fe370e00640 1 --2- 172.21.15.45:0/1516283286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe36c0b5d50 0x7fe36c0cc040 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe35800c400 tx=0x7fe35800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:41.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.248+0000 7fe369600640 1 -- 172.21.15.45:0/1516283286 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3580074e0 con 0x7fe36c0b5d50 2024-04-16T16:13:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.248+0000 7fe369600640 1 -- 172.21.15.45:0/1516283286 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe358007680 con 0x7fe36c0b5d50 2024-04-16T16:13:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.248+0000 7fe369600640 1 -- 172.21.15.45:0/1516283286 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe358004a00 con 0x7fe36c0b5d50 2024-04-16T16:13:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.248+0000 7fe372c00640 1 -- 172.21.15.45:0/1516283286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe36c1f51f0 con 0x7fe36c0b5d50 2024-04-16T16:13:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.248+0000 7fe372c00640 1 -- 172.21.15.45:0/1516283286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe36c0cd720 con 0x7fe36c0b5d50 2024-04-16T16:13:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.249+0000 7fe372c00640 1 -- 172.21.15.45:0/1516283286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe36c075780 con 0x7fe36c0b5d50 2024-04-16T16:13:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.250+0000 7fe369600640 1 -- 172.21.15.45:0/1516283286 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe35801a020 con 0x7fe36c0b5d50 2024-04-16T16:13:41.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.251+0000 7fe369600640 1 --2- 172.21.15.45:0/1516283286 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe34c06c8f0 0x7fe34c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:41.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.251+0000 7fe369600640 1 -- 172.21.15.45:0/1516283286 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe35808b3c0 con 0x7fe36c0b5d50 2024-04-16T16:13:41.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.252+0000 7fe371800640 1 --2- 172.21.15.45:0/1516283286 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe34c06c8f0 0x7fe34c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:41.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.253+0000 7fe371800640 1 --2- 172.21.15.45:0/1516283286 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe34c06c8f0 0x7fe34c06edb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe360009ac0 tx=0x7fe360008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:41.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.256+0000 7fe369600640 1 -- 172.21.15.45:0/1516283286 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe3580597e0 con 0x7fe36c0b5d50 2024-04-16T16:13:41.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.462+0000 7fe372c00640 1 -- 172.21.15.45:0/1516283286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe36c0cd8f0 con 0x7fe36c0b5d50 2024-04-16T16:13:41.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.463+0000 7fe369600640 1 -- 172.21.15.45:0/1516283286 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fe3580590e0 con 0x7fe36c0b5d50 2024-04-16T16:13:41.463 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:13:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.465+0000 7fe372c00640 1 -- 172.21.15.45:0/1516283286 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe34c06c8f0 msgr2=0x7fe34c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.466+0000 7fe372c00640 1 --2- 172.21.15.45:0/1516283286 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe34c06c8f0 0x7fe34c06edb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe360009ac0 tx=0x7fe360008040 comp rx=0 tx=0).stop 2024-04-16T16:13:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.466+0000 7fe372c00640 1 -- 172.21.15.45:0/1516283286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe36c0b5d50 msgr2=0x7fe36c0cc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.466+0000 7fe372c00640 1 --2- 172.21.15.45:0/1516283286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe36c0b5d50 0x7fe36c0cc040 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe35800c400 tx=0x7fe35800c8c0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.466+0000 7fe372c00640 1 -- 172.21.15.45:0/1516283286 shutdown_connections 2024-04-16T16:13:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.466+0000 7fe372c00640 1 --2- 172.21.15.45:0/1516283286 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe34c06c8f0 0x7fe34c06edb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.466+0000 7fe372c00640 1 --2- 172.21.15.45:0/1516283286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe36c1461b0 0x7fe36c1f43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.466+0000 7fe372c00640 1 --2- 172.21.15.45:0/1516283286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe36c0b5d50 0x7fe36c0cc040 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.466+0000 7fe372c00640 1 --2- 172.21.15.45:0/1516283286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe36c0b58f0 0x7fe36c1f55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.466+0000 7fe372c00640 1 -- 172.21.15.45:0/1516283286 >> 172.21.15.45:0/1516283286 conn(0x7fe36c06b930 msgr2=0x7fe36c06e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.466+0000 7fe372c00640 1 -- 172.21.15.45:0/1516283286 shutdown_connections 2024-04-16T16:13:41.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.466+0000 7fe372c00640 1 -- 172.21.15.45:0/1516283286 wait complete. 2024-04-16T16:13:41.486 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:13:41.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.706+0000 7f9a1ca00640 1 Processor -- start 2024-04-16T16:13:41.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.707+0000 7f9a1ca00640 1 -- start start 2024-04-16T16:13:41.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.708+0000 7f9a1ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9a1815ccb0 0x7f9a1806df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:41.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.708+0000 7f9a1ca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9a1806e480 0x7f9a181608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:41.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.708+0000 7f9a1ca00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a1815d360 con 0x7f9a18146220 2024-04-16T16:13:41.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.708+0000 7f9a1ca00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9a1815d130 con 0x7f9a1815ccb0 2024-04-16T16:13:41.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.708+0000 7f9a1ca00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9a18160410 con 0x7f9a1806e480 2024-04-16T16:13:41.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.709+0000 7f9a17400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9a18146220 0x7f9a1806f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39128/0 (socket says 172.21.15.45:39128) 2024-04-16T16:13:41.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.709+0000 7f9a17400640 1 -- 172.21.15.45:0/3048299533 learned_addr learned my addr 172.21.15.45:0/3048299533 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:41.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.709+0000 7f9a0fe00640 1 --2- 172.21.15.45:0/3048299533 >> v2:172.21.15.45:3300/0 conn(0x7f9a1815ccb0 0x7f9a1806df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:41.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.709+0000 7f9a17e00640 1 --2- 172.21.15.45:0/3048299533 >> v2:172.21.15.103:3300/0 conn(0x7f9a1806e480 0x7f9a181608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:41.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.709+0000 7f9a16a00640 1 -- 172.21.15.45:0/3048299533 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1751682596 0 0) 0x7f9a1815d360 con 0x7f9a18146220 2024-04-16T16:13:41.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.709+0000 7f9a16a00640 1 -- 172.21.15.45:0/3048299533 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99fc003520 con 0x7f9a18146220 2024-04-16T16:13:41.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.709+0000 7f9a0fe00640 1 -- 172.21.15.45:0/3048299533 >> v2:172.21.15.103:3300/0 conn(0x7f9a1806e480 msgr2=0x7f9a181608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:41.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.709+0000 7f9a0fe00640 1 --2- 172.21.15.45:0/3048299533 >> v2:172.21.15.103:3300/0 conn(0x7f9a1806e480 0x7f9a181608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.709+0000 7f9a0fe00640 1 -- 172.21.15.45:0/3048299533 >> v1:172.21.15.45:6789/0 conn(0x7f9a18146220 legacy=0x7f9a1806f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:41.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.709+0000 7f9a0fe00640 1 -- 172.21.15.45:0/3048299533 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1815d640 con 0x7f9a1815ccb0 2024-04-16T16:13:41.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.710+0000 7f9a0fe00640 1 --2- 172.21.15.45:0/3048299533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a1815ccb0 0x7f9a1806df40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9a18160590 tx=0x7f9a0002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76e77f70d54dbbe server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:41.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.711+0000 7f9a16a00640 1 -- 172.21.15.45:0/3048299533 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a00035020 con 0x7f9a1815ccb0 2024-04-16T16:13:41.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.711+0000 7f9a16a00640 1 -- 172.21.15.45:0/3048299533 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a00031830 con 0x7f9a1815ccb0 2024-04-16T16:13:41.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.711+0000 7f9a16a00640 1 -- 172.21.15.45:0/3048299533 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a00031c00 con 0x7f9a1815ccb0 2024-04-16T16:13:41.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.711+0000 7f9a1ca00640 1 -- 172.21.15.45:0/3048299533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a1815ccb0 msgr2=0x7f9a1806df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:41.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.711+0000 7f9a1ca00640 1 --2- 172.21.15.45:0/3048299533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a1815ccb0 0x7f9a1806df40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9a18160590 tx=0x7f9a0002b8d0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.711+0000 7f9a1ca00640 1 -- 172.21.15.45:0/3048299533 shutdown_connections 2024-04-16T16:13:41.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.711+0000 7f9a1ca00640 1 --2- 172.21.15.45:0/3048299533 >> v2:172.21.15.103:3300/0 conn(0x7f9a1806e480 0x7f9a181608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.711+0000 7f9a1ca00640 1 --2- 172.21.15.45:0/3048299533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a1815ccb0 0x7f9a1806df40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.711+0000 7f9a1ca00640 1 -- 172.21.15.45:0/3048299533 >> 172.21.15.45:0/3048299533 conn(0x7f9a1806b930 msgr2=0x7f9a1806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:41.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.711+0000 7f9a1ca00640 1 -- 172.21.15.45:0/3048299533 shutdown_connections 2024-04-16T16:13:41.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.712+0000 7f9a1ca00640 1 -- 172.21.15.45:0/3048299533 wait complete. 2024-04-16T16:13:41.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.713+0000 7f9a1ca00640 1 Processor -- start 2024-04-16T16:13:41.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.713+0000 7f9a1ca00640 1 -- start start 2024-04-16T16:13:41.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.714+0000 7f9a1ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a1806e480 0x7f9a180c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:41.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.714+0000 7f9a1ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a18146220 0x7f9a18144be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:41.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.714+0000 7f9a1ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a18145120 0x7f9a18143470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:41.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.714+0000 7f9a1ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9a18160de0 con 0x7f9a18146220 2024-04-16T16:13:41.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.714+0000 7f9a1ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1815dc00 con 0x7f9a18145120 2024-04-16T16:13:41.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.714+0000 7f9a1ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9a18160f60 con 0x7f9a1806e480 2024-04-16T16:13:41.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.715+0000 7f9a0fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a18146220 0x7f9a18144be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:41.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.715+0000 7f9a0fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a18146220 0x7f9a18144be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43596/0 (socket says 172.21.15.45:43596) 2024-04-16T16:13:41.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.715+0000 7f9a0fe00640 1 -- 172.21.15.45:0/1052690197 learned_addr learned my addr 172.21.15.45:0/1052690197 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:41.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.715+0000 7f9a17400640 1 --2- 172.21.15.45:0/1052690197 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a1806e480 0x7f9a180c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:41.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.715+0000 7f9a0fe00640 1 -- 172.21.15.45:0/1052690197 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a1806e480 msgr2=0x7f9a180c6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:41.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.715+0000 7f9a0fe00640 1 --2- 172.21.15.45:0/1052690197 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a1806e480 0x7f9a180c6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.715+0000 7f9a0fe00640 1 -- 172.21.15.45:0/1052690197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a18145120 msgr2=0x7f9a18143470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:41.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.715+0000 7f9a0fe00640 1 --2- 172.21.15.45:0/1052690197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a18145120 0x7f9a18143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.715+0000 7f9a0fe00640 1 -- 172.21.15.45:0/1052690197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a180cc240 con 0x7f9a18146220 2024-04-16T16:13:41.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.715+0000 7f9a0fe00640 1 --2- 172.21.15.45:0/1052690197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a18146220 0x7f9a18144be0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9a0002be60 tx=0x7f9a000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:41.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.716+0000 7f9a0f400640 1 -- 172.21.15.45:0/1052690197 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a00035040 con 0x7f9a18146220 2024-04-16T16:13:41.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.716+0000 7f9a0f400640 1 -- 172.21.15.45:0/1052690197 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a0003a840 con 0x7f9a18146220 2024-04-16T16:13:41.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.716+0000 7f9a0f400640 1 -- 172.21.15.45:0/1052690197 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0003abf0 con 0x7f9a18146220 2024-04-16T16:13:41.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.716+0000 7f9a1ca00640 1 -- 172.21.15.45:0/1052690197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a180cc520 con 0x7f9a18146220 2024-04-16T16:13:41.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.716+0000 7f9a1ca00640 1 -- 172.21.15.45:0/1052690197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a181356a0 con 0x7f9a18146220 2024-04-16T16:13:41.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.717+0000 7f9a1ca00640 1 -- 172.21.15.45:0/1052690197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a18075780 con 0x7f9a18146220 2024-04-16T16:13:41.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.718+0000 7f9a0f400640 1 -- 172.21.15.45:0/1052690197 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9a00038020 con 0x7f9a18146220 2024-04-16T16:13:41.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.719+0000 7f9a0f400640 1 --2- 172.21.15.45:0/1052690197 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f99f406c8f0 0x7f99f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:41.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.719+0000 7f9a0f400640 1 -- 172.21.15.45:0/1052690197 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9a000ad7e0 con 0x7f9a18146220 2024-04-16T16:13:41.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.719+0000 7f9a17400640 1 --2- 172.21.15.45:0/1052690197 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f99f406c8f0 0x7f99f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:41.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.720+0000 7f9a17400640 1 --2- 172.21.15.45:0/1052690197 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f99f406c8f0 0x7f99f406edb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9a08000f60 tx=0x7f9a08009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:41.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.724+0000 7f9a0f400640 1 -- 172.21.15.45:0/1052690197 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9a0007bc00 con 0x7f9a18146220 2024-04-16T16:13:41.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.975+0000 7f9a1ca00640 1 -- 172.21.15.45:0/1052690197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a180cf4f0 con 0x7f9a18146220 2024-04-16T16:13:41.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.976+0000 7f9a0f400640 1 -- 172.21.15.45:0/1052690197 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5178 (secure 0 0 0) 0x7f9a0007b500 con 0x7f9a18146220 2024-04-16T16:13:41.976 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:41.976 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2486891866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2486891866},{"type":"v1","addr":"172.21.15.103:6836","nonce":2486891866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1877281473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1877281473},{"type":"v1","addr":"172.21.15.103:6837","nonce":1877281473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/315848518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":315848518},{"type":"v1","addr":"172.21.15.45:6833","nonce":315848518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:13:25.369839+0000","modified":"2024-04-16T16:13:28.463144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2461453826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2461453826},{"type":"v1","addr":"172.21.15.45:6835","nonce":2461453826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-04-16T16:13:41.976 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 8 2024-04-16T16:13:41.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.979+0000 7f9a1ca00640 1 -- 172.21.15.45:0/1052690197 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f99f406c8f0 msgr2=0x7f99f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:41.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.979+0000 7f9a1ca00640 1 --2- 172.21.15.45:0/1052690197 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f99f406c8f0 0x7f99f406edb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9a08000f60 tx=0x7f9a08009040 comp rx=0 tx=0).stop 2024-04-16T16:13:41.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.979+0000 7f9a1ca00640 1 -- 172.21.15.45:0/1052690197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a18146220 msgr2=0x7f9a18144be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:41.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.979+0000 7f9a1ca00640 1 --2- 172.21.15.45:0/1052690197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a18146220 0x7f9a18144be0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9a0002be60 tx=0x7f9a000046c0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.979+0000 7f9a1ca00640 1 -- 172.21.15.45:0/1052690197 shutdown_connections 2024-04-16T16:13:41.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.979+0000 7f9a1ca00640 1 --2- 172.21.15.45:0/1052690197 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f99f406c8f0 0x7f99f406edb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.979+0000 7f9a1ca00640 1 --2- 172.21.15.45:0/1052690197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a18145120 0x7f9a18143470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.979+0000 7f9a1ca00640 1 --2- 172.21.15.45:0/1052690197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a18146220 0x7f9a18144be0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.979+0000 7f9a1ca00640 1 --2- 172.21.15.45:0/1052690197 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a1806e480 0x7f9a180c6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:41.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.979+0000 7f9a1ca00640 1 -- 172.21.15.45:0/1052690197 >> 172.21.15.45:0/1052690197 conn(0x7f9a1806b930 msgr2=0x7f9a180c5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:41.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.979+0000 7f9a1ca00640 1 -- 172.21.15.45:0/1052690197 shutdown_connections 2024-04-16T16:13:41.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:41.979+0000 7f9a1ca00640 1 -- 172.21.15.45:0/1052690197 wait complete. 2024-04-16T16:13:41.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:13:42.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.205+0000 7fbc4de00640 1 Processor -- start 2024-04-16T16:13:42.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.206+0000 7fbc4de00640 1 -- start start 2024-04-16T16:13:42.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.207+0000 7fbc4de00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbc48161e90 0x7fbc480cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:42.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.207+0000 7fbc4de00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbc480d52d0 0x7fbc4806ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:42.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.207+0000 7fbc4de00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc480d3080 con 0x7fbc48162270 2024-04-16T16:13:42.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.207+0000 7fbc4de00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fbc480d2e30 con 0x7fbc48161e90 2024-04-16T16:13:42.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.207+0000 7fbc4de00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbc480d3360 con 0x7fbc480d52d0 2024-04-16T16:13:42.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.207+0000 7fbc4ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbc48161e90 0x7fbc480cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:42.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.207+0000 7fbc4ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbc48161e90 0x7fbc480cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43600/0 (socket says 172.21.15.45:43600) 2024-04-16T16:13:42.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.207+0000 7fbc4ca00640 1 -- 172.21.15.45:0/2547543348 learned_addr learned my addr 172.21.15.45:0/2547543348 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:42.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.208+0000 7fbc4d400640 1 --2- 172.21.15.45:0/2547543348 >> v2:172.21.15.103:3300/0 conn(0x7fbc480d52d0 0x7fbc4806ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:42.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.208+0000 7fbc4ca00640 1 -- 172.21.15.45:0/2547543348 >> v2:172.21.15.103:3300/0 conn(0x7fbc480d52d0 msgr2=0x7fbc4806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:42.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.208+0000 7fbc4ca00640 1 --2- 172.21.15.45:0/2547543348 >> v2:172.21.15.103:3300/0 conn(0x7fbc480d52d0 0x7fbc4806ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.208+0000 7fbc4ca00640 1 -- 172.21.15.45:0/2547543348 >> v1:172.21.15.103:6789/0 conn(0x7fbc48162270 legacy=0x7fbc480d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:42.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.208+0000 7fbc4ca00640 1 -- 172.21.15.45:0/2547543348 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc480d3640 con 0x7fbc48161e90 2024-04-16T16:13:42.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.208+0000 7fbc4d400640 1 --2- 172.21.15.45:0/2547543348 >> v2:172.21.15.103:3300/0 conn(0x7fbc480d52d0 0x7fbc4806ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:42.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.209+0000 7fbc4ca00640 1 --2- 172.21.15.45:0/2547543348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc48161e90 0x7fbc480cf7d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbc38009870 tx=0x7fbc3802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6e97050341735 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:42.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.210+0000 7fbc47400640 1 -- 172.21.15.45:0/2547543348 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc38035020 con 0x7fbc48161e90 2024-04-16T16:13:42.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.210+0000 7fbc47400640 1 -- 172.21.15.45:0/2547543348 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc38031870 con 0x7fbc48161e90 2024-04-16T16:13:42.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.210+0000 7fbc47400640 1 -- 172.21.15.45:0/2547543348 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc38031c40 con 0x7fbc48161e90 2024-04-16T16:13:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.211+0000 7fbc4de00640 1 -- 172.21.15.45:0/2547543348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc48161e90 msgr2=0x7fbc480cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.211+0000 7fbc4de00640 1 --2- 172.21.15.45:0/2547543348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc48161e90 0x7fbc480cf7d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbc38009870 tx=0x7fbc3802b910 comp rx=0 tx=0).stop 2024-04-16T16:13:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.211+0000 7fbc4de00640 1 -- 172.21.15.45:0/2547543348 shutdown_connections 2024-04-16T16:13:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.211+0000 7fbc4de00640 1 --2- 172.21.15.45:0/2547543348 >> v2:172.21.15.103:3300/0 conn(0x7fbc480d52d0 0x7fbc4806ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.211+0000 7fbc4de00640 1 --2- 172.21.15.45:0/2547543348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc48161e90 0x7fbc480cf7d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.211+0000 7fbc4de00640 1 -- 172.21.15.45:0/2547543348 >> 172.21.15.45:0/2547543348 conn(0x7fbc480d1940 msgr2=0x7fbc480d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.211+0000 7fbc4de00640 1 -- 172.21.15.45:0/2547543348 shutdown_connections 2024-04-16T16:13:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.211+0000 7fbc4de00640 1 -- 172.21.15.45:0/2547543348 wait complete. 2024-04-16T16:13:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.212+0000 7fbc4de00640 1 Processor -- start 2024-04-16T16:13:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.213+0000 7fbc4de00640 1 -- start start 2024-04-16T16:13:42.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.213+0000 7fbc4de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc480d52d0 0x7fbc480cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:42.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.213+0000 7fbc4de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc48161e90 0x7fbc48157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:42.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.213+0000 7fbc4de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc48162270 0x7fbc48157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.213+0000 7fbc4de00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbc480d4550 con 0x7fbc48161e90 2024-04-16T16:13:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.213+0000 7fbc4de00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbc480cfd10 con 0x7fbc48162270 2024-04-16T16:13:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.213+0000 7fbc4de00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbc480d46d0 con 0x7fbc480d52d0 2024-04-16T16:13:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.214+0000 7fbc47e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc48161e90 0x7fbc48157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.214+0000 7fbc47e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc48161e90 0x7fbc48157750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43608/0 (socket says 172.21.15.45:43608) 2024-04-16T16:13:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.214+0000 7fbc47e00640 1 -- 172.21.15.45:0/2109078918 learned_addr learned my addr 172.21.15.45:0/2109078918 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.214+0000 7fbc4d400640 1 --2- 172.21.15.45:0/2109078918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc48162270 0x7fbc48157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.214+0000 7fbc4ca00640 1 --2- 172.21.15.45:0/2109078918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc480d52d0 0x7fbc480cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.214+0000 7fbc47e00640 1 -- 172.21.15.45:0/2109078918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc480d52d0 msgr2=0x7fbc480cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.214+0000 7fbc47e00640 1 --2- 172.21.15.45:0/2109078918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc480d52d0 0x7fbc480cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.214+0000 7fbc47e00640 1 -- 172.21.15.45:0/2109078918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc48162270 msgr2=0x7fbc48157c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.214+0000 7fbc47e00640 1 --2- 172.21.15.45:0/2109078918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc48162270 0x7fbc48157c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.214+0000 7fbc47e00640 1 -- 172.21.15.45:0/2109078918 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc48159770 con 0x7fbc48161e90 2024-04-16T16:13:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.214+0000 7fbc4ca00640 1 --2- 172.21.15.45:0/2109078918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc480d52d0 0x7fbc480cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:42.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.215+0000 7fbc47e00640 1 --2- 172.21.15.45:0/2109078918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc48161e90 0x7fbc48157750 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbc3000b3c0 tx=0x7fbc3000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:42.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.215+0000 7fbc44c00640 1 -- 172.21.15.45:0/2109078918 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc30011830 con 0x7fbc48161e90 2024-04-16T16:13:42.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.215+0000 7fbc44c00640 1 -- 172.21.15.45:0/2109078918 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc30016070 con 0x7fbc48161e90 2024-04-16T16:13:42.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.215+0000 7fbc4de00640 1 -- 172.21.15.45:0/2109078918 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc48159a50 con 0x7fbc48161e90 2024-04-16T16:13:42.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.215+0000 7fbc44c00640 1 -- 172.21.15.45:0/2109078918 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc30011c10 con 0x7fbc48161e90 2024-04-16T16:13:42.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.215+0000 7fbc4de00640 1 -- 172.21.15.45:0/2109078918 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc481584a0 con 0x7fbc48161e90 2024-04-16T16:13:42.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.216+0000 7fbc4de00640 1 -- 172.21.15.45:0/2109078918 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc48075780 con 0x7fbc48161e90 2024-04-16T16:13:42.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.217+0000 7fbc44c00640 1 -- 172.21.15.45:0/2109078918 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbc3001b460 con 0x7fbc48161e90 2024-04-16T16:13:42.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.218+0000 7fbc44c00640 1 --2- 172.21.15.45:0/2109078918 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbc2406c8f0 0x7fbc2406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:42.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.218+0000 7fbc44c00640 1 -- 172.21.15.45:0/2109078918 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbc3008b590 con 0x7fbc48161e90 2024-04-16T16:13:42.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.219+0000 7fbc4ca00640 1 --2- 172.21.15.45:0/2109078918 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbc2406c8f0 0x7fbc2406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:42.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.220+0000 7fbc4ca00640 1 --2- 172.21.15.45:0/2109078918 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbc2406c8f0 0x7fbc2406edb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbc38036000 tx=0x7fbc38008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:42.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.222+0000 7fbc44c00640 1 -- 172.21.15.45:0/2109078918 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbc30020020 con 0x7fbc48161e90 2024-04-16T16:13:42.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.428+0000 7fbc4de00640 1 -- 172.21.15.45:0/2109078918 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc48157530 con 0x7fbc48161e90 2024-04-16T16:13:42.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.430+0000 7fbc44c00640 1 -- 172.21.15.45:0/2109078918 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7fbc30059a50 con 0x7fbc48161e90 2024-04-16T16:13:42.431 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:42.431 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":21,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:26.918617+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:13:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.433+0000 7fbc4de00640 1 -- 172.21.15.45:0/2109078918 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbc2406c8f0 msgr2=0x7fbc2406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.433+0000 7fbc4de00640 1 --2- 172.21.15.45:0/2109078918 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbc2406c8f0 0x7fbc2406edb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbc38036000 tx=0x7fbc38008030 comp rx=0 tx=0).stop 2024-04-16T16:13:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.433+0000 7fbc4de00640 1 -- 172.21.15.45:0/2109078918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc48161e90 msgr2=0x7fbc48157750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.433+0000 7fbc4de00640 1 --2- 172.21.15.45:0/2109078918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc48161e90 0x7fbc48157750 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbc3000b3c0 tx=0x7fbc3000b880 comp rx=0 tx=0).stop 2024-04-16T16:13:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.433+0000 7fbc4de00640 1 -- 172.21.15.45:0/2109078918 shutdown_connections 2024-04-16T16:13:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.433+0000 7fbc4de00640 1 --2- 172.21.15.45:0/2109078918 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbc2406c8f0 0x7fbc2406edb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.433+0000 7fbc4de00640 1 --2- 172.21.15.45:0/2109078918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc48162270 0x7fbc48157c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.433+0000 7fbc4de00640 1 --2- 172.21.15.45:0/2109078918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc48161e90 0x7fbc48157750 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.433+0000 7fbc4de00640 1 --2- 172.21.15.45:0/2109078918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc480d52d0 0x7fbc480cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.433+0000 7fbc4de00640 1 -- 172.21.15.45:0/2109078918 >> 172.21.15.45:0/2109078918 conn(0x7fbc480d1940 msgr2=0x7fbc480b5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:42.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.433+0000 7fbc4de00640 1 -- 172.21.15.45:0/2109078918 shutdown_connections 2024-04-16T16:13:42.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.433+0000 7fbc4de00640 1 -- 172.21.15.45:0/2109078918 wait complete. 2024-04-16T16:13:42.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:13:42.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.674+0000 7fdf15200640 1 Processor -- start 2024-04-16T16:13:42.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.674+0000 7fdf15200640 1 -- start start 2024-04-16T16:13:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.675+0000 7fdf15200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdf10162270 0x7fdf100d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.675+0000 7fdf15200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdf100d52d0 0x7fdf1006ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.675+0000 7fdf15200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf100d3080 con 0x7fdf10161e90 2024-04-16T16:13:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.675+0000 7fdf15200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fdf100d2e30 con 0x7fdf10162270 2024-04-16T16:13:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.675+0000 7fdf15200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fdf100d3360 con 0x7fdf100d52d0 2024-04-16T16:13:42.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.676+0000 7fdf0ea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdf10162270 0x7fdf100d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:42.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.676+0000 7fdf0ea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdf10162270 0x7fdf100d3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43612/0 (socket says 172.21.15.45:43612) 2024-04-16T16:13:42.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.676+0000 7fdf0ea00640 1 -- 172.21.15.45:0/1608845320 learned_addr learned my addr 172.21.15.45:0/1608845320 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:42.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.676+0000 7fdf0fe00640 1 --2- 172.21.15.45:0/1608845320 >> v2:172.21.15.103:3300/0 conn(0x7fdf100d52d0 0x7fdf1006ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:42.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.676+0000 7fdf0ea00640 1 -- 172.21.15.45:0/1608845320 >> v2:172.21.15.103:3300/0 conn(0x7fdf100d52d0 msgr2=0x7fdf1006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:42.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.676+0000 7fdf0ea00640 1 --2- 172.21.15.45:0/1608845320 >> v2:172.21.15.103:3300/0 conn(0x7fdf100d52d0 0x7fdf1006ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.676+0000 7fdf0ea00640 1 -- 172.21.15.45:0/1608845320 >> v1:172.21.15.103:6789/0 conn(0x7fdf10161e90 legacy=0x7fdf100cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:42.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.676+0000 7fdf0ea00640 1 -- 172.21.15.45:0/1608845320 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf100d3640 con 0x7fdf10162270 2024-04-16T16:13:42.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.677+0000 7fdf0ea00640 1 --2- 172.21.15.45:0/1608845320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf10162270 0x7fdf100d3e40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdefc009870 tx=0x7fdefc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38da201a0db026c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.678+0000 7fdf0e000640 1 -- 172.21.15.45:0/1608845320 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdefc035020 con 0x7fdf10162270 2024-04-16T16:13:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.678+0000 7fdf0e000640 1 -- 172.21.15.45:0/1608845320 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdefc031870 con 0x7fdf10162270 2024-04-16T16:13:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.678+0000 7fdf0e000640 1 -- 172.21.15.45:0/1608845320 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdefc031c40 con 0x7fdf10162270 2024-04-16T16:13:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.678+0000 7fdf15200640 1 -- 172.21.15.45:0/1608845320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf10162270 msgr2=0x7fdf100d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.678+0000 7fdf15200640 1 --2- 172.21.15.45:0/1608845320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf10162270 0x7fdf100d3e40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdefc009870 tx=0x7fdefc02b910 comp rx=0 tx=0).stop 2024-04-16T16:13:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.678+0000 7fdf15200640 1 -- 172.21.15.45:0/1608845320 shutdown_connections 2024-04-16T16:13:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.678+0000 7fdf15200640 1 --2- 172.21.15.45:0/1608845320 >> v2:172.21.15.103:3300/0 conn(0x7fdf100d52d0 0x7fdf1006ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.678+0000 7fdf15200640 1 --2- 172.21.15.45:0/1608845320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf10162270 0x7fdf100d3e40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.678+0000 7fdf15200640 1 -- 172.21.15.45:0/1608845320 >> 172.21.15.45:0/1608845320 conn(0x7fdf100d1940 msgr2=0x7fdf100d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.679+0000 7fdf15200640 1 -- 172.21.15.45:0/1608845320 shutdown_connections 2024-04-16T16:13:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.679+0000 7fdf15200640 1 -- 172.21.15.45:0/1608845320 wait complete. 2024-04-16T16:13:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.680+0000 7fdf15200640 1 Processor -- start 2024-04-16T16:13:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.680+0000 7fdf15200640 1 -- start start 2024-04-16T16:13:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.681+0000 7fdf15200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf100d52d0 0x7fdf1015f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.681+0000 7fdf15200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf10161e90 0x7fdf100ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.681+0000 7fdf15200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf10162270 0x7fdf101e90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.681+0000 7fdf15200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdf100d4500 con 0x7fdf10161e90 2024-04-16T16:13:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.681+0000 7fdf15200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf100d4380 con 0x7fdf100d52d0 2024-04-16T16:13:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.681+0000 7fdf15200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdf100d4680 con 0x7fdf10162270 2024-04-16T16:13:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.682+0000 7fdf0ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf10161e90 0x7fdf100ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.682+0000 7fdf0ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf10161e90 0x7fdf100ca6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43618/0 (socket says 172.21.15.45:43618) 2024-04-16T16:13:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.682+0000 7fdf0ea00640 1 -- 172.21.15.45:0/1770639167 learned_addr learned my addr 172.21.15.45:0/1770639167 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.682+0000 7fdf0f400640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf100d52d0 0x7fdf1015f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:42.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.682+0000 7fdf0fe00640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf10162270 0x7fdf101e90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:42.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.682+0000 7fdf0ea00640 1 -- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf10162270 msgr2=0x7fdf101e90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:42.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.682+0000 7fdf0ea00640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf10162270 0x7fdf101e90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.682+0000 7fdf0ea00640 1 -- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf100d52d0 msgr2=0x7fdf1015f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:42.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.682+0000 7fdf0ea00640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf100d52d0 0x7fdf1015f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.682+0000 7fdf0ea00640 1 -- 172.21.15.45:0/1770639167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf100c8bd0 con 0x7fdf10161e90 2024-04-16T16:13:42.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.682+0000 7fdf0fe00640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf10162270 0x7fdf101e90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:42.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.682+0000 7fdf0f400640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf100d52d0 0x7fdf1015f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:13:42.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.683+0000 7fdf0ea00640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf10161e90 0x7fdf100ca6f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdefc000c00 tx=0x7fdefc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:42.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.683+0000 7fdf07400640 1 -- 172.21.15.45:0/1770639167 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdefc040070 con 0x7fdf10161e90 2024-04-16T16:13:42.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.683+0000 7fdf07400640 1 -- 172.21.15.45:0/1770639167 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdefc030d00 con 0x7fdf10161e90 2024-04-16T16:13:42.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.683+0000 7fdf15200640 1 -- 172.21.15.45:0/1770639167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf100c8eb0 con 0x7fdf10161e90 2024-04-16T16:13:42.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.683+0000 7fdf07400640 1 -- 172.21.15.45:0/1770639167 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdefc035040 con 0x7fdf10161e90 2024-04-16T16:13:42.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.683+0000 7fdf15200640 1 -- 172.21.15.45:0/1770639167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf100c9470 con 0x7fdf10161e90 2024-04-16T16:13:42.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.684+0000 7fdf15200640 1 -- 172.21.15.45:0/1770639167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf10075780 con 0x7fdf10161e90 2024-04-16T16:13:42.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.685+0000 7fdf07400640 1 -- 172.21.15.45:0/1770639167 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdefc04b060 con 0x7fdf10161e90 2024-04-16T16:13:42.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.686+0000 7fdf07400640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdef006c8f0 0x7fdef006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:42.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.686+0000 7fdf07400640 1 -- 172.21.15.45:0/1770639167 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdefc0ad920 con 0x7fdf10161e90 2024-04-16T16:13:42.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.687+0000 7fdf0f400640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdef006c8f0 0x7fdef006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:42.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.688+0000 7fdf0f400640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdef006c8f0 0x7fdef006edb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf00003eb0 tx=0x7fdf00009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:42.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.691+0000 7fdf07400640 1 -- 172.21.15.45:0/1770639167 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdefc07bd40 con 0x7fdf10161e90 2024-04-16T16:13:42.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.939+0000 7fdf15200640 1 -- 172.21.15.45:0/1770639167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf100b9da0 con 0x7fdf10161e90 2024-04-16T16:13:42.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.940+0000 7fdf07400640 1 -- 172.21.15.45:0/1770639167 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5178 (secure 0 0 0) 0x7fdefc07b640 con 0x7fdf10161e90 2024-04-16T16:13:42.940 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:13:42.941 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2486891866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2486891866},{"type":"v1","addr":"172.21.15.103:6836","nonce":2486891866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1877281473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1877281473},{"type":"v1","addr":"172.21.15.103:6837","nonce":1877281473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/315848518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":315848518},{"type":"v1","addr":"172.21.15.45:6833","nonce":315848518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:13:25.369839+0000","modified":"2024-04-16T16:13:28.463144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2461453826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2461453826},{"type":"v1","addr":"172.21.15.45:6835","nonce":2461453826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-04-16T16:13:42.941 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 8 2024-04-16T16:13:42.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.943+0000 7fdf15200640 1 -- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdef006c8f0 msgr2=0x7fdef006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:42.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.943+0000 7fdf15200640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdef006c8f0 0x7fdef006edb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf00003eb0 tx=0x7fdf00009040 comp rx=0 tx=0).stop 2024-04-16T16:13:42.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.943+0000 7fdf15200640 1 -- 172.21.15.45:0/1770639167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf10161e90 msgr2=0x7fdf100ca6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:42.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.943+0000 7fdf15200640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf10161e90 0x7fdf100ca6f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdefc000c00 tx=0x7fdefc0046c0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.943+0000 7fdf15200640 1 -- 172.21.15.45:0/1770639167 shutdown_connections 2024-04-16T16:13:42.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.943+0000 7fdf15200640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdef006c8f0 0x7fdef006edb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.943+0000 7fdf15200640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf10162270 0x7fdf101e90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.943+0000 7fdf15200640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf10161e90 0x7fdf100ca6f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.943+0000 7fdf15200640 1 --2- 172.21.15.45:0/1770639167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf100d52d0 0x7fdf1015f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:42.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.943+0000 7fdf15200640 1 -- 172.21.15.45:0/1770639167 >> 172.21.15.45:0/1770639167 conn(0x7fdf100d1940 msgr2=0x7fdf1006ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:42.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.943+0000 7fdf15200640 1 -- 172.21.15.45:0/1770639167 shutdown_connections 2024-04-16T16:13:42.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:42.943+0000 7fdf15200640 1 -- 172.21.15.45:0/1770639167 wait complete. 2024-04-16T16:13:43.363 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:13:25.369839+0000', 'modified': '2024-04-16T16:13:28.463144+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4245}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4245': {'gid': 4245, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6835/2461453826', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6834', 'nonce': 2461453826}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 2461453826}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4245, 'qdb_cluster': [4245]} 2024-04-16T16:13:43.364 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-16T16:13:43.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-16T16:13:43.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.596+0000 7fcefdc00640 1 Processor -- start 2024-04-16T16:13:43.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.596+0000 7fcefdc00640 1 -- start start 2024-04-16T16:13:43.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.598+0000 7fcefdc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcef8145230 0x7fcef815bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.598+0000 7fcefdc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcef815cbe0 0x7fcef80cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.598+0000 7fcefdc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcef80c5670 0x7fcef80c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.598+0000 7fcefdc00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fcef80c6020 con 0x7fcef8145230 2024-04-16T16:13:43.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.598+0000 7fcefdc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcef815dfc0 con 0x7fcef80c5670 2024-04-16T16:13:43.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.598+0000 7fcefdc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcef815d3b0 con 0x7fcef815cbe0 2024-04-16T16:13:43.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.598+0000 7fcef7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcef8145230 0x7fcef815bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.599+0000 7fcef7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcef8145230 0x7fcef815bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52848/0 (socket says 172.21.15.45:52848) 2024-04-16T16:13:43.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.599+0000 7fcef7e00640 1 -- 172.21.15.45:0/718257441 learned_addr learned my addr 172.21.15.45:0/718257441 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:43.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.599+0000 7fcefd200640 1 --2- 172.21.15.45:0/718257441 >> v2:172.21.15.103:3300/0 conn(0x7fcef80c5670 0x7fcef80c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.599+0000 7fceefe00640 1 --2- 172.21.15.45:0/718257441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcef815cbe0 0x7fcef80cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.599+0000 7fcef7e00640 1 -- 172.21.15.45:0/718257441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcef815cbe0 msgr2=0x7fcef80cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:43.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.599+0000 7fcef7e00640 1 --2- 172.21.15.45:0/718257441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcef815cbe0 0x7fcef80cd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.599+0000 7fcef7e00640 1 -- 172.21.15.45:0/718257441 >> v2:172.21.15.103:3300/0 conn(0x7fcef80c5670 msgr2=0x7fcef80c5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:43.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.599+0000 7fcef7e00640 1 --2- 172.21.15.45:0/718257441 >> v2:172.21.15.103:3300/0 conn(0x7fcef80c5670 0x7fcef80c5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.599+0000 7fcef7e00640 1 -- 172.21.15.45:0/718257441 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef815d690 con 0x7fcef8145230 2024-04-16T16:13:43.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.600+0000 7fcef7e00640 1 --2- 172.21.15.45:0/718257441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef8145230 0x7fcef815bc80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcef8073730 tx=0x7fcee802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e13f0b2e5f19a6f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:43.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.601+0000 7fcef7400640 1 -- 172.21.15.45:0/718257441 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee80317d0 con 0x7fcef8145230 2024-04-16T16:13:43.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.601+0000 7fcef7400640 1 -- 172.21.15.45:0/718257441 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee8037070 con 0x7fcef8145230 2024-04-16T16:13:43.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.601+0000 7fcef7400640 1 -- 172.21.15.45:0/718257441 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee8031b90 con 0x7fcef8145230 2024-04-16T16:13:43.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.601+0000 7fcefdc00640 1 -- 172.21.15.45:0/718257441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef8145230 msgr2=0x7fcef815bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:43.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.601+0000 7fcefdc00640 1 --2- 172.21.15.45:0/718257441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef8145230 0x7fcef815bc80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcef8073730 tx=0x7fcee802b870 comp rx=0 tx=0).stop 2024-04-16T16:13:43.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.601+0000 7fcefdc00640 1 -- 172.21.15.45:0/718257441 shutdown_connections 2024-04-16T16:13:43.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.601+0000 7fcefdc00640 1 --2- 172.21.15.45:0/718257441 >> v2:172.21.15.103:3300/0 conn(0x7fcef80c5670 0x7fcef80c5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.601+0000 7fcefdc00640 1 --2- 172.21.15.45:0/718257441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcef815cbe0 0x7fcef80cd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.601+0000 7fcefdc00640 1 --2- 172.21.15.45:0/718257441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef8145230 0x7fcef815bc80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.601+0000 7fcefdc00640 1 -- 172.21.15.45:0/718257441 >> 172.21.15.45:0/718257441 conn(0x7fcef806b930 msgr2=0x7fcef806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:43.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.602+0000 7fcefdc00640 1 -- 172.21.15.45:0/718257441 shutdown_connections 2024-04-16T16:13:43.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.602+0000 7fcefdc00640 1 -- 172.21.15.45:0/718257441 wait complete. 2024-04-16T16:13:43.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.603+0000 7fcefdc00640 1 Processor -- start 2024-04-16T16:13:43.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.603+0000 7fcefdc00640 1 -- start start 2024-04-16T16:13:43.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.604+0000 7fcefdc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcef80c5670 0x7fcef815b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.604+0000 7fcefdc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef8145230 0x7fcef813b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.605+0000 7fcefdc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcef815cbe0 0x7fcef8136950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.605+0000 7fcefdc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcef80cd620 con 0x7fcef8145230 2024-04-16T16:13:43.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.605+0000 7fcefdc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcef815dc50 con 0x7fcef815cbe0 2024-04-16T16:13:43.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.605+0000 7fcefdc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcef815d040 con 0x7fcef80c5670 2024-04-16T16:13:43.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.605+0000 7fceefe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef8145230 0x7fcef813b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.605+0000 7fceefe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef8145230 0x7fcef813b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52858/0 (socket says 172.21.15.45:52858) 2024-04-16T16:13:43.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.605+0000 7fceefe00640 1 -- 172.21.15.45:0/3370973485 learned_addr learned my addr 172.21.15.45:0/3370973485 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:43.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.605+0000 7fcef7e00640 1 --2- 172.21.15.45:0/3370973485 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcef80c5670 0x7fcef815b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.605+0000 7fceefe00640 1 -- 172.21.15.45:0/3370973485 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcef80c5670 msgr2=0x7fcef815b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:43.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.605+0000 7fcefd200640 1 --2- 172.21.15.45:0/3370973485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcef815cbe0 0x7fcef8136950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.605+0000 7fceefe00640 1 --2- 172.21.15.45:0/3370973485 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcef80c5670 0x7fcef815b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.605+0000 7fceefe00640 1 -- 172.21.15.45:0/3370973485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcef815cbe0 msgr2=0x7fcef8136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:43.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.605+0000 7fceefe00640 1 --2- 172.21.15.45:0/3370973485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcef815cbe0 0x7fcef8136950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.605+0000 7fceefe00640 1 -- 172.21.15.45:0/3370973485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef815b690 con 0x7fcef8145230 2024-04-16T16:13:43.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.605+0000 7fcefd200640 1 --2- 172.21.15.45:0/3370973485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcef815cbe0 0x7fcef8136950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:43.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.606+0000 7fceefe00640 1 --2- 172.21.15.45:0/3370973485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef8145230 0x7fcef813b140 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcee000d5f0 tx=0x7fcee000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:43.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.606+0000 7fcef4c00640 1 -- 172.21.15.45:0/3370973485 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee00125f0 con 0x7fcef8145230 2024-04-16T16:13:43.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.606+0000 7fcef4c00640 1 -- 172.21.15.45:0/3370973485 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee0019070 con 0x7fcef8145230 2024-04-16T16:13:43.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.606+0000 7fcef4c00640 1 -- 172.21.15.45:0/3370973485 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee00129d0 con 0x7fcef8145230 2024-04-16T16:13:43.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.606+0000 7fcefdc00640 1 -- 172.21.15.45:0/3370973485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef813b970 con 0x7fcef8145230 2024-04-16T16:13:43.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.606+0000 7fcefdc00640 1 -- 172.21.15.45:0/3370973485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef813bf30 con 0x7fcef8145230 2024-04-16T16:13:43.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.608+0000 7fcef4c00640 1 -- 172.21.15.45:0/3370973485 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcee0012b70 con 0x7fcef8145230 2024-04-16T16:13:43.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.609+0000 7fcef4c00640 1 --2- 172.21.15.45:0/3370973485 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fced406c8f0 0x7fced406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.610+0000 7fcef4c00640 1 -- 172.21.15.45:0/3370973485 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcee008bfb0 con 0x7fcef8145230 2024-04-16T16:13:43.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.610+0000 7fcf01fe2740 1 Processor -- start 2024-04-16T16:13:43.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.610+0000 7fcf01fe2740 1 -- start start 2024-04-16T16:13:43.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.610+0000 7fcef7e00640 1 --2- 172.21.15.45:0/3370973485 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fced406c8f0 0x7fced406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.611+0000 7fcf01fe2740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db97b890 0x55e5db99a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.611+0000 7fcf01fe2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97bc70 0x55e5db954570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.611+0000 7fcf01fe2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db954ab0 0x55e5db952440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.611+0000 7fcf01fe2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e5db8b99c0 con 0x55e5db97bc70 2024-04-16T16:13:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.611+0000 7fcf01fe2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e5db860560 con 0x55e5db954ab0 2024-04-16T16:13:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.611+0000 7fcf01fe2740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e5db8b5980 con 0x55e5db97b890 2024-04-16T16:13:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.611+0000 7fceefe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97bc70 0x55e5db954570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.611+0000 7fceefe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97bc70 0x55e5db954570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52868/0 (socket says 172.21.15.45:52868) 2024-04-16T16:13:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.611+0000 7fceefe00640 1 -- 172.21.15.45:0/4087665167 learned_addr learned my addr 172.21.15.45:0/4087665167 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.612+0000 7fcef7e00640 1 --2- 172.21.15.45:0/4087665167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db97b890 0x55e5db99a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.612+0000 7fcef7e00640 1 --2- 172.21.15.45:0/3370973485 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fced406c8f0 0x7fced406edb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcee8036000 tx=0x7fcee8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.612+0000 7fceefe00640 1 -- 172.21.15.45:0/4087665167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db97b890 msgr2=0x55e5db99a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.612+0000 7fceefe00640 1 --2- 172.21.15.45:0/4087665167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db97b890 0x55e5db99a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.612+0000 7fcefd200640 1 --2- 172.21.15.45:0/4087665167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db954ab0 0x55e5db952440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.612+0000 7fceefe00640 1 -- 172.21.15.45:0/4087665167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db954ab0 msgr2=0x55e5db952440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.612+0000 7fceefe00640 1 --2- 172.21.15.45:0/4087665167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db954ab0 0x55e5db952440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.612+0000 7fceefe00640 1 -- 172.21.15.45:0/4087665167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5db5cc2d0 con 0x55e5db97bc70 2024-04-16T16:13:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.612+0000 7fceefe00640 1 --2- 172.21.15.45:0/4087665167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97bc70 0x55e5db954570 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcee008a440 tx=0x7fcee005a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a40626108f46f9e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.612+0000 7fceecc00640 1 -- 172.21.15.45:0/4087665167 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee0072020 con 0x55e5db97bc70 2024-04-16T16:13:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.612+0000 7fceecc00640 1 -- 172.21.15.45:0/4087665167 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee00628b0 con 0x55e5db97bc70 2024-04-16T16:13:43.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.612+0000 7fceecc00640 1 -- 172.21.15.45:0/4087665167 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee00721c0 con 0x55e5db97bc70 2024-04-16T16:13:43.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.613+0000 7fcf01fe2740 1 -- 172.21.15.45:0/4087665167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97bc70 msgr2=0x55e5db954570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.613+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/4087665167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97bc70 0x55e5db954570 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcee008a440 tx=0x7fcee005a490 comp rx=0 tx=0).stop 2024-04-16T16:13:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.613+0000 7fcf01fe2740 1 -- 172.21.15.45:0/4087665167 shutdown_connections 2024-04-16T16:13:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.613+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/4087665167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db954ab0 0x55e5db952440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.613+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/4087665167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97bc70 0x55e5db954570 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.613+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/4087665167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db97b890 0x55e5db99a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.613+0000 7fcf01fe2740 1 -- 172.21.15.45:0/4087665167 >> 172.21.15.45:0/4087665167 conn(0x55e5db950b50 msgr2=0x55e5db9996d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.613+0000 7fcf01fe2740 1 -- 172.21.15.45:0/4087665167 shutdown_connections 2024-04-16T16:13:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.613+0000 7fcf01fe2740 1 -- 172.21.15.45:0/4087665167 wait complete. 2024-04-16T16:13:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.614+0000 7fcf01fe2740 1 -- start start 2024-04-16T16:13:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.614+0000 7fcf01fe2740 1 Processor -- start 2024-04-16T16:13:43.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.615+0000 7fcf01fe2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db97b890 0x55e5db9addb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.615+0000 7fcf01fe2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97bc70 0x55e5db99c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.615+0000 7fcf01fe2740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db992b40 0x55e5db99ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.615+0000 7fcf01fe2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e5db94e980 con 0x55e5db97bc70 2024-04-16T16:13:43.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.615+0000 7fcf01fe2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e5db873920 con 0x55e5db97b890 2024-04-16T16:13:43.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.615+0000 7fcf01fe2740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e5db954f30 con 0x55e5db992b40 2024-04-16T16:13:43.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.616+0000 7fcf01fe2740 10 client.0 _ll_register_callbacks cb 0x55e5db873310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:13:43.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.616+0000 7fceefe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97bc70 0x55e5db99c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.616+0000 7fceefe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97bc70 0x55e5db99c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52872/0 (socket says 172.21.15.45:52872) 2024-04-16T16:13:43.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.616+0000 7fceefe00640 1 -- 172.21.15.45:0/434585133 learned_addr learned my addr 172.21.15.45:0/434585133 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:43.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.616+0000 7fcef7e00640 1 --2- 172.21.15.45:0/434585133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db97b890 0x55e5db9addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.616+0000 7fceefe00640 1 -- 172.21.15.45:0/434585133 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db992b40 msgr2=0x55e5db99ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.616+0000 7fcefd200640 1 --2- 172.21.15.45:0/434585133 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db992b40 0x55e5db99ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.616+0000 7fceefe00640 1 --2- 172.21.15.45:0/434585133 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db992b40 0x55e5db99ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.616+0000 7fceefe00640 1 -- 172.21.15.45:0/434585133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db97b890 msgr2=0x55e5db9addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.616+0000 7fceefe00640 1 --2- 172.21.15.45:0/434585133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db97b890 0x55e5db9addb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.616+0000 7fceefe00640 1 -- 172.21.15.45:0/434585133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5db6966b0 con 0x55e5db97bc70 2024-04-16T16:13:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.616+0000 7fcef7e00640 1 --2- 172.21.15.45:0/434585133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db97b890 0x55e5db9addb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.616+0000 7fceefe00640 1 --2- 172.21.15.45:0/434585133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97bc70 0x55e5db99c950 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcee0059980 tx=0x7fcee0020070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941c608fd59ffd18 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.616+0000 7fcee5600640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:13:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.617+0000 7fcee5600640 1 -- 172.21.15.45:0/434585133 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee0076020 con 0x55e5db97bc70 2024-04-16T16:13:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.617+0000 7fcee5600640 1 -- 172.21.15.45:0/434585133 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcee00721e0 con 0x55e5db97bc70 2024-04-16T16:13:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.617+0000 7fcf01fe2740 1 -- 172.21.15.45:0/434585133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcee00628b0 con 0x55e5db97bc70 2024-04-16T16:13:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.617+0000 7fcee5600640 1 -- 172.21.15.45:0/434585133 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fcee0072040 con 0x55e5db97bc70 2024-04-16T16:13:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.617+0000 7fcf01fe2740 10 client.4647 fetch_fsmap learned FSMap version 8 2024-04-16T16:13:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.617+0000 7fcf01fe2740 1 -- 172.21.15.45:0/434585133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55e5db981c90 con 0x55e5db97bc70 2024-04-16T16:13:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.617+0000 7fcee5600640 1 -- 172.21.15.45:0/434585133 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee00628b0 con 0x55e5db97bc70 2024-04-16T16:13:43.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.617+0000 7fcee5600640 1 -- 172.21.15.45:0/434585133 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 8) ==== 3494+0+0 (secure 0 0 0) 0x7fcee0013070 con 0x55e5db97bc70 2024-04-16T16:13:43.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.617+0000 7fcf01fe2740 10 client.4647 fetch_fsmap finished waiting for FSMap version 8 2024-04-16T16:13:43.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.617+0000 7fcf01fe2740 10 client.4647 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-16T16:13:43.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.617+0000 7fcf01fe2740 20 client.4647 populate_metadata read hostname 'smithi045' 2024-04-16T16:13:43.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.617+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/434585133 >> [v2:172.21.15.45:6834/2461453826,v1:172.21.15.45:6835/2461453826] conn(0x55e5db9aecd0 0x55e5db9984f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.618+0000 7fcf01fe2740 4 client.4647 mds_command: new command op to 4245 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-16T16:13:43.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.618+0000 7fcf01fe2740 1 -- 172.21.15.45:0/434585133 --> [v2:172.21.15.45:6834/2461453826,v1:172.21.15.45:6835/2461453826] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5db85fb70 con 0x55e5db9aecd0 2024-04-16T16:13:43.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.618+0000 7fcef7e00640 1 --2- 172.21.15.45:0/434585133 >> [v2:172.21.15.45:6834/2461453826,v1:172.21.15.45:6835/2461453826] conn(0x55e5db9aecd0 0x55e5db9984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.618+0000 7fcef7e00640 1 --2- 172.21.15.45:0/434585133 >> [v2:172.21.15.45:6834/2461453826,v1:172.21.15.45:6835/2461453826] conn(0x55e5db9aecd0 0x55e5db9984f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e88f334680a8ba6 server_cookie=93538a0755ca1f55 in_seq=0 out_seq=0 2024-04-16T16:13:43.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.618+0000 7fcee5600640 10 client.4647 ms_handle_connect on v2:172.21.15.45:6834/2461453826 2024-04-16T16:13:43.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.619+0000 7fcee5600640 1 -- 172.21.15.45:0/434585133 <== mds.0 v2:172.21.15.45:6834/2461453826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e5db85fb70 con 0x55e5db9aecd0 2024-04-16T16:13:43.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.619+0000 7fcee5600640 10 client.4647 handle_command_reply: tid=0 2024-04-16T16:13:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.619+0000 7fcf01fe2740 1 client.4647 shutdown 2024-04-16T16:13:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.620+0000 7fcf01fe2740 1 -- 172.21.15.45:0/434585133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97bc70 msgr2=0x55e5db99c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.620+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/434585133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97bc70 0x55e5db99c950 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcee0059980 tx=0x7fcee0020070 comp rx=0 tx=0).stop 2024-04-16T16:13:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.620+0000 7fcf01fe2740 1 -- 172.21.15.45:0/434585133 shutdown_connections 2024-04-16T16:13:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.620+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/434585133 >> [v2:172.21.15.45:6834/2461453826,v1:172.21.15.45:6835/2461453826] conn(0x55e5db9aecd0 0x55e5db9984f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.620+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/434585133 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db992b40 0x55e5db99ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.620+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/434585133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97bc70 0x55e5db99c950 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.620+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/434585133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db97b890 0x55e5db9addb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.620+0000 7fcee5600640 0 client.4647 ms_handle_reset on v2:172.21.15.45:6834/2461453826 2024-04-16T16:13:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.620+0000 7fcf01fe2740 1 -- 172.21.15.45:0/434585133 >> 172.21.15.45:0/434585133 conn(0x55e5db954ab0 msgr2=0x55e5db993310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.620+0000 7fcf01fe2740 1 -- 172.21.15.45:0/434585133 shutdown_connections 2024-04-16T16:13:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.620+0000 7fcf01fe2740 1 -- 172.21.15.45:0/434585133 wait complete. 2024-04-16T16:13:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.620+0000 7fcf01fe2740 20 client.4647 trim_cache size 0 max 16384 2024-04-16T16:13:43.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.640+0000 7fcf01fe2740 1 Processor -- start 2024-04-16T16:13:43.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.640+0000 7fcf01fe2740 1 -- start start 2024-04-16T16:13:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.641+0000 7fcf01fe2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db97b890 0x55e5db99c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.641+0000 7fcf01fe2740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db97bc70 0x55e5db99cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.641+0000 7fcf01fe2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db992b40 0x55e5db999660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.641+0000 7fcf01fe2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcef81fbb60 con 0x55e5db992b40 2024-04-16T16:13:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.641+0000 7fcf01fe2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcef81faec0 con 0x55e5db97b890 2024-04-16T16:13:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.641+0000 7fcf01fe2740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcef81fb510 con 0x55e5db97bc70 2024-04-16T16:13:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.641+0000 7fcefd200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db992b40 0x55e5db999660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.641+0000 7fcefd200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db992b40 0x55e5db999660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52888/0 (socket says 172.21.15.45:52888) 2024-04-16T16:13:43.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.641+0000 7fcefd200640 1 -- 172.21.15.45:0/3572574757 learned_addr learned my addr 172.21.15.45:0/3572574757 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:43.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.642+0000 7fcefd200640 1 -- 172.21.15.45:0/3572574757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db97bc70 msgr2=0x55e5db99cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:43.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.642+0000 7fcefd200640 1 --2- 172.21.15.45:0/3572574757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db97bc70 0x55e5db99cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.642+0000 7fcefd200640 1 -- 172.21.15.45:0/3572574757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db97b890 msgr2=0x55e5db99c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:43.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.642+0000 7fcefd200640 1 --2- 172.21.15.45:0/3572574757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db97b890 0x55e5db99c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.642+0000 7fcefd200640 1 -- 172.21.15.45:0/3572574757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5db94e7b0 con 0x55e5db992b40 2024-04-16T16:13:43.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.642+0000 7fcefd200640 1 --2- 172.21.15.45:0/3572574757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db992b40 0x55e5db999660 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcef000bc20 tx=0x7fcef0009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10f268fef209f07 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:43.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.642+0000 7fcee7e00640 1 -- 172.21.15.45:0/3572574757 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef001b070 con 0x55e5db992b40 2024-04-16T16:13:43.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.642+0000 7fcee7e00640 1 -- 172.21.15.45:0/3572574757 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcef0016600 con 0x55e5db992b40 2024-04-16T16:13:43.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.642+0000 7fcee7e00640 1 -- 172.21.15.45:0/3572574757 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef00169f0 con 0x55e5db992b40 2024-04-16T16:13:43.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.643+0000 7fcf01fe2740 1 -- 172.21.15.45:0/3572574757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db992b40 msgr2=0x55e5db999660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:43.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.643+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/3572574757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db992b40 0x55e5db999660 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcef000bc20 tx=0x7fcef0009700 comp rx=0 tx=0).stop 2024-04-16T16:13:43.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.643+0000 7fcf01fe2740 1 -- 172.21.15.45:0/3572574757 shutdown_connections 2024-04-16T16:13:43.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.643+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/3572574757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db992b40 0x55e5db999660 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.643+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/3572574757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db97bc70 0x55e5db99cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.643+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/3572574757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db97b890 0x55e5db99c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.643+0000 7fcf01fe2740 1 -- 172.21.15.45:0/3572574757 >> 172.21.15.45:0/3572574757 conn(0x55e5db954ab0 msgr2=0x55e5db998470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:43.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.643+0000 7fcf01fe2740 1 -- 172.21.15.45:0/3572574757 shutdown_connections 2024-04-16T16:13:43.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.643+0000 7fcf01fe2740 1 -- 172.21.15.45:0/3572574757 wait complete. 2024-04-16T16:13:43.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.644+0000 7fcf01fe2740 1 -- start start 2024-04-16T16:13:43.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.644+0000 7fcf01fe2740 1 Processor -- start 2024-04-16T16:13:43.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.645+0000 7fcf01fe2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97b890 0x55e5db96c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.645+0000 7fcf01fe2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db97bc70 0x55e5dba17790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.645+0000 7fcf01fe2740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db992b40 0x55e5db97ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.645+0000 7fcf01fe2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e5db94e980 con 0x55e5db97b890 2024-04-16T16:13:43.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.645+0000 7fcf01fe2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e5db873920 con 0x55e5db97bc70 2024-04-16T16:13:43.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.645+0000 7fcf01fe2740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e5db9af3d0 con 0x55e5db992b40 2024-04-16T16:13:43.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.645+0000 7fcf01fe2740 10 client.0 _ll_register_callbacks cb 0x7fcee000d540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:13:43.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.646+0000 7fcef7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97b890 0x55e5db96c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.646+0000 7fcef7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97b890 0x55e5db96c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52894/0 (socket says 172.21.15.45:52894) 2024-04-16T16:13:43.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.646+0000 7fcef7e00640 1 -- 172.21.15.45:0/1878838076 learned_addr learned my addr 172.21.15.45:0/1878838076 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:43.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.646+0000 7fcefd200640 1 --2- 172.21.15.45:0/1878838076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db992b40 0x55e5db97ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.646+0000 7fceefe00640 1 --2- 172.21.15.45:0/1878838076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db97bc70 0x55e5dba17790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.646+0000 7fcef7e00640 1 -- 172.21.15.45:0/1878838076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db992b40 msgr2=0x55e5db97ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:43.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.646+0000 7fcef7e00640 1 --2- 172.21.15.45:0/1878838076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db992b40 0x55e5db97ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.646+0000 7fcef7e00640 1 -- 172.21.15.45:0/1878838076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db97bc70 msgr2=0x55e5dba17790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:43.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.646+0000 7fcef7e00640 1 --2- 172.21.15.45:0/1878838076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db97bc70 0x55e5dba17790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.646+0000 7fcef7e00640 1 -- 172.21.15.45:0/1878838076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5dba17cd0 con 0x55e5db97b890 2024-04-16T16:13:43.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.646+0000 7fcef7e00640 1 --2- 172.21.15.45:0/1878838076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97b890 0x55e5db96c1c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcee8034040 tx=0x7fcee80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4006bb35f3bfc63c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:43.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.646+0000 7fcee5600640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:13:43.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.647+0000 7fcee5600640 1 -- 172.21.15.45:0/1878838076 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee8032d40 con 0x55e5db97b890 2024-04-16T16:13:43.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.647+0000 7fcee5600640 1 -- 172.21.15.45:0/1878838076 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcee8052080 con 0x55e5db97b890 2024-04-16T16:13:43.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.647+0000 7fcf01fe2740 1 -- 172.21.15.45:0/1878838076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcef0016600 con 0x55e5db97b890 2024-04-16T16:13:43.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.647+0000 7fcee5600640 1 -- 172.21.15.45:0/1878838076 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fcee805b050 con 0x55e5db97b890 2024-04-16T16:13:43.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.647+0000 7fcf01fe2740 10 client.4653 fetch_fsmap learned FSMap version 8 2024-04-16T16:13:43.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.647+0000 7fcf01fe2740 1 -- 172.21.15.45:0/1878838076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55e5db94f430 con 0x55e5db97b890 2024-04-16T16:13:43.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.647+0000 7fcee5600640 1 -- 172.21.15.45:0/1878838076 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcef0016600 con 0x55e5db97b890 2024-04-16T16:13:43.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.647+0000 7fcee5600640 1 -- 172.21.15.45:0/1878838076 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 8) ==== 3494+0+0 (secure 0 0 0) 0x7fcee805e020 con 0x55e5db97b890 2024-04-16T16:13:43.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.647+0000 7fcf01fe2740 10 client.4653 fetch_fsmap finished waiting for FSMap version 8 2024-04-16T16:13:43.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.647+0000 7fcf01fe2740 10 client.4653 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-16T16:13:43.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.647+0000 7fcf01fe2740 20 client.4653 populate_metadata read hostname 'smithi045' 2024-04-16T16:13:43.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.647+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/1878838076 >> [v2:172.21.15.45:6834/2461453826,v1:172.21.15.45:6835/2461453826] conn(0x55e5db999a30 0x55e5db975460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:43.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.647+0000 7fcf01fe2740 4 client.4653 mds_command: new command op to 4245 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-16T16:13:43.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.647+0000 7fcf01fe2740 1 -- 172.21.15.45:0/1878838076 --> [v2:172.21.15.45:6834/2461453826,v1:172.21.15.45:6835/2461453826] -- command(tid 0: {"prefix": "status"}) -- 0x55e5db99a6a0 con 0x55e5db999a30 2024-04-16T16:13:43.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.648+0000 7fceefe00640 1 --2- 172.21.15.45:0/1878838076 >> [v2:172.21.15.45:6834/2461453826,v1:172.21.15.45:6835/2461453826] conn(0x55e5db999a30 0x55e5db975460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:43.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.648+0000 7fceefe00640 1 --2- 172.21.15.45:0/1878838076 >> [v2:172.21.15.45:6834/2461453826,v1:172.21.15.45:6835/2461453826] conn(0x55e5db999a30 0x55e5db975460 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a64b480517a01836 server_cookie=3fbe773be412f007 in_seq=0 out_seq=0 2024-04-16T16:13:43.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.648+0000 7fcee5600640 10 client.4653 ms_handle_connect on v2:172.21.15.45:6834/2461453826 2024-04-16T16:13:43.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.648+0000 7fcee5600640 1 -- 172.21.15.45:0/1878838076 <== mds.0 v2:172.21.15.45:6834/2461453826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55e5db99a6a0 con 0x55e5db999a30 2024-04-16T16:13:43.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.648+0000 7fcee5600640 10 client.4653 handle_command_reply: tid=0 2024-04-16T16:13:43.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.648+0000 7fcf01fe2740 1 client.4653 shutdown 2024-04-16T16:13:43.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.649+0000 7fcf01fe2740 1 -- 172.21.15.45:0/1878838076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97b890 msgr2=0x55e5db96c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:43.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.649+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/1878838076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97b890 0x55e5db96c1c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcee8034040 tx=0x7fcee80046c0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.649+0000 7fcf01fe2740 1 -- 172.21.15.45:0/1878838076 shutdown_connections 2024-04-16T16:13:43.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.649+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/1878838076 >> [v2:172.21.15.45:6834/2461453826,v1:172.21.15.45:6835/2461453826] conn(0x55e5db999a30 0x55e5db975460 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.649+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/1878838076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e5db992b40 0x55e5db97ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.649+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/1878838076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5db97bc70 0x55e5dba17790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.649+0000 7fcee5600640 0 client.4653 ms_handle_reset on v2:172.21.15.45:6834/2461453826 2024-04-16T16:13:43.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.649+0000 7fcf01fe2740 1 --2- 172.21.15.45:0/1878838076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5db97b890 0x55e5db96c1c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.649+0000 7fcf01fe2740 1 -- 172.21.15.45:0/1878838076 >> 172.21.15.45:0/1878838076 conn(0x55e5db954ab0 msgr2=0x55e5db998450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:43.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.649+0000 7fcf01fe2740 1 -- 172.21.15.45:0/1878838076 shutdown_connections 2024-04-16T16:13:43.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.649+0000 7fcf01fe2740 1 -- 172.21.15.45:0/1878838076 wait complete. 2024-04-16T16:13:43.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.649+0000 7fcf01fe2740 20 client.4653 trim_cache size 0 max 16384 2024-04-16T16:13:43.656 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:13:43.656 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "748c4b85-0190-4c59-80ad-098a6a5007c0", 2024-04-16T16:13:43.656 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-04-16T16:13:43.656 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 4245, 2024-04-16T16:13:43.657 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-04-16T16:13:43.657 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-04-16T16:13:43.657 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-04-16T16:13:43.657 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 17.724970015, 2024-04-16T16:13:43.657 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 8, 2024-04-16T16:13:43.657 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 21, 2024-04-16T16:13:43.657 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 21, 2024-04-16T16:13:43.657 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 25.308143599000001 2024-04-16T16:13:43.657 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:13:43.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.650+0000 7fcefdc00640 1 -- 172.21.15.45:0/3370973485 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fced406c8f0 msgr2=0x7fced406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:43.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.650+0000 7fcefdc00640 1 --2- 172.21.15.45:0/3370973485 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fced406c8f0 0x7fced406edb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcee8036000 tx=0x7fcee8008030 comp rx=0 tx=0).stop 2024-04-16T16:13:43.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.650+0000 7fcefdc00640 1 -- 172.21.15.45:0/3370973485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef8145230 msgr2=0x7fcef813b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:43.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.650+0000 7fcefdc00640 1 --2- 172.21.15.45:0/3370973485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef8145230 0x7fcef813b140 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcee000d5f0 tx=0x7fcee000dab0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.650+0000 7fcefdc00640 1 -- 172.21.15.45:0/3370973485 shutdown_connections 2024-04-16T16:13:43.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.650+0000 7fcefdc00640 1 --2- 172.21.15.45:0/3370973485 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fced406c8f0 0x7fced406edb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.650+0000 7fcefdc00640 1 --2- 172.21.15.45:0/3370973485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcef815cbe0 0x7fcef8136950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.650+0000 7fcefdc00640 1 --2- 172.21.15.45:0/3370973485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef8145230 0x7fcef813b140 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.650+0000 7fcefdc00640 1 --2- 172.21.15.45:0/3370973485 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcef80c5670 0x7fcef815b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:43.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.650+0000 7fcefdc00640 1 -- 172.21.15.45:0/3370973485 >> 172.21.15.45:0/3370973485 conn(0x7fcef806b930 msgr2=0x7fcef806e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:43.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.650+0000 7fcefdc00640 1 -- 172.21.15.45:0/3370973485 shutdown_connections 2024-04-16T16:13:43.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:43.650+0000 7fcefdc00640 1 -- 172.21.15.45:0/3370973485 wait complete. 2024-04-16T16:13:43.667 INFO:teuthology.run_tasks:Running task sequential... 2024-04-16T16:13:43.677 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-16T16:13:43.677 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-16T16:13:43.678 INFO:teuthology.task.print:Enabling mgr modules 2024-04-16T16:13:43.678 INFO:teuthology.run_tasks:Running task kclient... 2024-04-16T16:13:43.700 INFO:tasks.kclient:Mounting kernel clients... 2024-04-16T16:13:43.701 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-16T16:13:43.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:43.701 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T16:13:43.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:43.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2024-04-16T16:13:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:Cannot find device "ceph-brx" 2024-04-16T16:13:43.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T16:13:43.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:43.866 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T16:13:43.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:43.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2024-04-16T16:13:44.023 INFO:teuthology.orchestra.run.smithi045.stderr:Cannot find device "ceph-brx" 2024-04-16T16:13:44.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T16:13:44.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:44.028 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-16T16:13:44.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:44.085 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-04-16T16:13:44.183 INFO:teuthology.orchestra.run.smithi103.stderr:Cannot find device "ceph-brx" 2024-04-16T16:13:44.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T16:13:44.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:44.187 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-16T16:13:44.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:44.236 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-04-16T16:13:44.323 INFO:teuthology.orchestra.run.smithi103.stderr:Cannot find device "ceph-brx" 2024-04-16T16:13:44.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T16:13:44.327 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-04-16T16:13:44.328 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-16T16:13:44.328 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T16:13:44.328 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:13:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.562+0000 7f8b2d600640 1 Processor -- start 2024-04-16T16:13:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.563+0000 7f8b2d600640 1 -- start start 2024-04-16T16:13:44.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.564+0000 7f8b2d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b28144ff0 0x7f8b280cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:44.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.564+0000 7f8b2d600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8b280c5d80 0x7f8b2815d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:44.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.564+0000 7f8b2d600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b2815c700 con 0x7f8b2806dfb0 2024-04-16T16:13:44.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.564+0000 7f8b2d600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f8b2815d2f0 con 0x7f8b280c5d80 2024-04-16T16:13:44.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.564+0000 7f8b2d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8b2815c9c0 con 0x7f8b28144ff0 2024-04-16T16:13:44.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.565+0000 7f8b27400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8b2806dfb0 0x7f8b280c5670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36474/0 (socket says 172.21.15.45:36474) 2024-04-16T16:13:44.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.565+0000 7f8b27400640 1 -- 172.21.15.45:0/4178751076 learned_addr learned my addr 172.21.15.45:0/4178751076 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:44.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.565+0000 7f8b27e00640 1 --2- 172.21.15.45:0/4178751076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b28144ff0 0x7f8b280cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:44.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.565+0000 7f8b2cc00640 1 --2- 172.21.15.45:0/4178751076 >> v2:172.21.15.103:3300/0 conn(0x7f8b280c5d80 0x7f8b2815d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:44.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.565+0000 7f8b26a00640 1 -- 172.21.15.45:0/4178751076 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1500441795 0 0) 0x7f8b2815c700 con 0x7f8b2806dfb0 2024-04-16T16:13:44.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.565+0000 7f8b26a00640 1 -- 172.21.15.45:0/4178751076 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b0c005730 con 0x7f8b2806dfb0 2024-04-16T16:13:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.566+0000 7f8b26a00640 1 -- 172.21.15.45:0/4178751076 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2963464601 0 0) 0x7f8b0c005730 con 0x7f8b2806dfb0 2024-04-16T16:13:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.566+0000 7f8b26a00640 1 -- 172.21.15.45:0/4178751076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b28144ff0 msgr2=0x7f8b280cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.566+0000 7f8b26a00640 1 --2- 172.21.15.45:0/4178751076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b28144ff0 0x7f8b280cfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.566+0000 7f8b26a00640 1 -- 172.21.15.45:0/4178751076 >> v2:172.21.15.103:3300/0 conn(0x7f8b280c5d80 msgr2=0x7f8b2815d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.566+0000 7f8b26a00640 1 --2- 172.21.15.45:0/4178751076 >> v2:172.21.15.103:3300/0 conn(0x7f8b280c5d80 0x7f8b2815d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.566+0000 7f8b26a00640 1 -- 172.21.15.45:0/4178751076 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b2815cca0 con 0x7f8b2806dfb0 2024-04-16T16:13:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.566+0000 7f8b26a00640 1 -- 172.21.15.45:0/4178751076 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f8b10001670 con 0x7f8b2806dfb0 2024-04-16T16:13:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.566+0000 7f8b26a00640 1 -- 172.21.15.45:0/4178751076 >> v1:172.21.15.45:6789/0 conn(0x7f8b2806dfb0 legacy=0x7f8b280c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.566+0000 7f8b26a00640 1 --2- 172.21.15.45:0/4178751076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b28144ff0 0x7f8b0c005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.566+0000 7f8b26a00640 1 --2- 172.21.15.45:0/4178751076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b0c006140 0x7f8b0c006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.566+0000 7f8b26a00640 1 --2- 172.21.15.45:0/4178751076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b0c006f50 0x7f8b0c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.566+0000 7f8b26a00640 1 -- 172.21.15.45:0/4178751076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2815d2f0 con 0x7f8b0c006f50 2024-04-16T16:13:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.566+0000 7f8b26a00640 1 -- 172.21.15.45:0/4178751076 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2815c9c0 con 0x7f8b0c006140 2024-04-16T16:13:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.566+0000 7f8b26a00640 1 -- 172.21.15.45:0/4178751076 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8b0c00dbd0 con 0x7f8b28144ff0 2024-04-16T16:13:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.566+0000 7f8b2cc00640 1 --2- 172.21.15.45:0/4178751076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b0c006f50 0x7f8b0c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.567+0000 7f8b27e00640 1 --2- 172.21.15.45:0/4178751076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b28144ff0 0x7f8b0c005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.567+0000 7f8b27400640 1 --2- 172.21.15.45:0/4178751076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b0c006140 0x7f8b0c006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.567+0000 7f8b2cc00640 1 -- 172.21.15.45:0/4178751076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b28144ff0 msgr2=0x7f8b0c005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.567+0000 7f8b2cc00640 1 --2- 172.21.15.45:0/4178751076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b28144ff0 0x7f8b0c005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.567+0000 7f8b2cc00640 1 -- 172.21.15.45:0/4178751076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b0c006140 msgr2=0x7f8b0c006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.567+0000 7f8b2cc00640 1 --2- 172.21.15.45:0/4178751076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b0c006140 0x7f8b0c006a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.567+0000 7f8b2cc00640 1 -- 172.21.15.45:0/4178751076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b0c005730 con 0x7f8b0c006f50 2024-04-16T16:13:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.568+0000 7f8b2cc00640 1 --2- 172.21.15.45:0/4178751076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b0c006f50 0x7f8b0c0073d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8b20009fc0 tx=0x7f8b0c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb62513cfa008008 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.568+0000 7f8b26a00640 1 -- 172.21.15.45:0/4178751076 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b20039070 con 0x7f8b0c006f50 2024-04-16T16:13:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.568+0000 7f8b26a00640 1 -- 172.21.15.45:0/4178751076 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b2002cc40 con 0x7f8b0c006f50 2024-04-16T16:13:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.568+0000 7f8b26a00640 1 -- 172.21.15.45:0/4178751076 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b20009c10 con 0x7f8b0c006f50 2024-04-16T16:13:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.569+0000 7f8b2d600640 1 -- 172.21.15.45:0/4178751076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b0c006f50 msgr2=0x7f8b0c0073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.569+0000 7f8b2d600640 1 --2- 172.21.15.45:0/4178751076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b0c006f50 0x7f8b0c0073d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8b20009fc0 tx=0x7f8b0c007a40 comp rx=0 tx=0).stop 2024-04-16T16:13:44.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.569+0000 7f8b2cc00640 1 -- 172.21.15.45:0/4178751076 reap_dead start 2024-04-16T16:13:44.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.569+0000 7f8b2d600640 1 -- 172.21.15.45:0/4178751076 shutdown_connections 2024-04-16T16:13:44.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.569+0000 7f8b2d600640 1 -- 172.21.15.45:0/4178751076 >> 172.21.15.45:0/4178751076 conn(0x7f8b2806b930 msgr2=0x7f8b2806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:44.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.569+0000 7f8b2d600640 1 -- 172.21.15.45:0/4178751076 shutdown_connections 2024-04-16T16:13:44.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.569+0000 7f8b2d600640 1 -- 172.21.15.45:0/4178751076 wait complete. 2024-04-16T16:13:44.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.571+0000 7f8b2d600640 1 Processor -- start 2024-04-16T16:13:44.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.571+0000 7f8b2d600640 1 -- start start 2024-04-16T16:13:44.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.572+0000 7f8b2d600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b0c006f50 0x7f8b28138ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:44.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.572+0000 7f8b2d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b281437f0 0x7f8b2815ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:44.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.572+0000 7f8b2d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b28140a80 0x7f8b28140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:44.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.572+0000 7f8b2d600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8b280cfd30 con 0x7f8b0c006f50 2024-04-16T16:13:44.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.572+0000 7f8b2d600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8b281453d0 con 0x7f8b281437f0 2024-04-16T16:13:44.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.572+0000 7f8b2d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8b280cfeb0 con 0x7f8b28140a80 2024-04-16T16:13:44.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.572+0000 7f8b27e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b0c006f50 0x7f8b28138ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:44.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.572+0000 7f8b27e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b0c006f50 0x7f8b28138ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52920/0 (socket says 172.21.15.45:52920) 2024-04-16T16:13:44.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.572+0000 7f8b27e00640 1 -- 172.21.15.45:0/148979075 learned_addr learned my addr 172.21.15.45:0/148979075 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:44.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.572+0000 7f8b2cc00640 1 --2- 172.21.15.45:0/148979075 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b28140a80 0x7f8b28140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:44.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.572+0000 7f8b27e00640 1 -- 172.21.15.45:0/148979075 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b28140a80 msgr2=0x7f8b28140f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:44.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.572+0000 7f8b27e00640 1 --2- 172.21.15.45:0/148979075 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b28140a80 0x7f8b28140f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:44.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.572+0000 7f8b27400640 1 --2- 172.21.15.45:0/148979075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b281437f0 0x7f8b2815ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:44.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.572+0000 7f8b27e00640 1 -- 172.21.15.45:0/148979075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b281437f0 msgr2=0x7f8b2815ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:44.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.573+0000 7f8b27e00640 1 --2- 172.21.15.45:0/148979075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b281437f0 0x7f8b2815ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:44.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.573+0000 7f8b27e00640 1 -- 172.21.15.45:0/148979075 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b281396d0 con 0x7f8b0c006f50 2024-04-16T16:13:44.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.573+0000 7f8b27e00640 1 --2- 172.21.15.45:0/148979075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b0c006f50 0x7f8b28138ea0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8b18011600 tx=0x7f8b18011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:44.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.573+0000 7f8b1fe00640 1 -- 172.21.15.45:0/148979075 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1800f920 con 0x7f8b0c006f50 2024-04-16T16:13:44.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.573+0000 7f8b1fe00640 1 -- 172.21.15.45:0/148979075 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b1800fac0 con 0x7f8b0c006f50 2024-04-16T16:13:44.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.573+0000 7f8b1fe00640 1 -- 172.21.15.45:0/148979075 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1800fe70 con 0x7f8b0c006f50 2024-04-16T16:13:44.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.573+0000 7f8b2d600640 1 -- 172.21.15.45:0/148979075 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b281399b0 con 0x7f8b0c006f50 2024-04-16T16:13:44.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.573+0000 7f8b2d600640 1 -- 172.21.15.45:0/148979075 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b28139f70 con 0x7f8b0c006f50 2024-04-16T16:13:44.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.575+0000 7f8b2d600640 1 -- 172.21.15.45:0/148979075 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b28075780 con 0x7f8b0c006f50 2024-04-16T16:13:44.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.575+0000 7f8b1fe00640 1 -- 172.21.15.45:0/148979075 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8b18015460 con 0x7f8b0c006f50 2024-04-16T16:13:44.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.577+0000 7f8b1fe00640 1 --2- 172.21.15.45:0/148979075 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8b0406c920 0x7f8b0406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:44.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.577+0000 7f8b1fe00640 1 -- 172.21.15.45:0/148979075 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8b1808c4d0 con 0x7f8b0c006f50 2024-04-16T16:13:44.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.578+0000 7f8b27400640 1 --2- 172.21.15.45:0/148979075 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8b0406c920 0x7f8b0406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:44.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.579+0000 7f8b27400640 1 --2- 172.21.15.45:0/148979075 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8b0406c920 0x7f8b0406ede0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8b10000f60 tx=0x7f8b10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:44.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.582+0000 7f8b1fe00640 1 -- 172.21.15.45:0/148979075 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b18019030 con 0x7f8b0c006f50 2024-04-16T16:13:44.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.830+0000 7f8b2d600640 1 -- 172.21.15.45:0/148979075 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f8b2813dfb0 con 0x7f8b0c006f50 2024-04-16T16:13:44.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.831+0000 7f8b1fe00640 1 -- 172.21.15.45:0/148979075 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f8b1808ae20 con 0x7f8b0c006f50 2024-04-16T16:13:44.831 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:13:44.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.833+0000 7f8b2d600640 1 -- 172.21.15.45:0/148979075 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8b0406c920 msgr2=0x7f8b0406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:44.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.833+0000 7f8b2d600640 1 --2- 172.21.15.45:0/148979075 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8b0406c920 0x7f8b0406ede0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8b10000f60 tx=0x7f8b10009040 comp rx=0 tx=0).stop 2024-04-16T16:13:44.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.833+0000 7f8b2d600640 1 -- 172.21.15.45:0/148979075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b0c006f50 msgr2=0x7f8b28138ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:44.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.834+0000 7f8b2d600640 1 --2- 172.21.15.45:0/148979075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b0c006f50 0x7f8b28138ea0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8b18011600 tx=0x7f8b18011ac0 comp rx=0 tx=0).stop 2024-04-16T16:13:44.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.834+0000 7f8b2d600640 1 -- 172.21.15.45:0/148979075 shutdown_connections 2024-04-16T16:13:44.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.834+0000 7f8b2d600640 1 --2- 172.21.15.45:0/148979075 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8b0406c920 0x7f8b0406ede0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:44.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.834+0000 7f8b2d600640 1 --2- 172.21.15.45:0/148979075 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b28140a80 0x7f8b28140f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:44.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.834+0000 7f8b2d600640 1 --2- 172.21.15.45:0/148979075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b281437f0 0x7f8b2815ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:44.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.834+0000 7f8b2d600640 1 --2- 172.21.15.45:0/148979075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b0c006f50 0x7f8b28138ea0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:44.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.834+0000 7f8b2d600640 1 -- 172.21.15.45:0/148979075 >> 172.21.15.45:0/148979075 conn(0x7f8b2806b930 msgr2=0x7f8b2806ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:44.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.834+0000 7f8b2d600640 1 -- 172.21.15.45:0/148979075 shutdown_connections 2024-04-16T16:13:44.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:44.834+0000 7f8b2d600640 1 -- 172.21.15.45:0/148979075 wait complete. 2024-04-16T16:13:44.851 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-16T16:13:44.851 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:13:44.851 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-04-16T16:13:44.852 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-16T16:13:44.852 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T16:13:44.852 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-16T16:13:44.852 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-16T16:13:44.852 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:13:44.852 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:13:44.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:44.852 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-04-16T16:13:44.905 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:13:44.906 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:13:44.906 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:13:44.906 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:44.906 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-04-16T16:13:44.906 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:44.906 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:13:44.906 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:44.906 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-04-16T16:13:44.906 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:13:44.906 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:44.906 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-04-16T16:13:44.907 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:13:44.907 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:44.907 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-04-16T16:13:44.907 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-16T16:13:44.907 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42576sec preferred_lft 42576sec 2024-04-16T16:13:44.907 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-04-16T16:13:44.907 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:44.907 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:13:44.907 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:44.907 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-04-16T16:13:44.908 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-16T16:13:44.908 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:44.908 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:13:44.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add name ceph-brx type bridge 2024-04-16T16:13:44.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr flush dev ceph-brx 2024-04-16T16:13:44.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set ceph-brx up 2024-04-16T16:13:44.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-16T16:13:44.908 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:13:45.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:45.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:45.452 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:13:45.536 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-04-16T16:13:45.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:45.541 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-04-16T16:13:45.606 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-04-16T16:13:45.606 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:13:45.606 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-16T16:13:45.606 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-16T16:13:45.606 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:13:45.607 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:45.607 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:13:45.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-16T16:13:45.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-16T16:13:45.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-16T16:13:45.607 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:13:45.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:46.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:46.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:46.435 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T16:13:46.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:46.482 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list-id 2024-04-16T16:13:46.525 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:46.525 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:13:46.525 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T16:13:46.525 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-16T16:13:46.525 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:13:46.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:46.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:46.764 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-16T16:13:46.765 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:46.765 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:13:46.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-16T16:13:46.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-16T16:13:46.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-16T16:13:46.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-16T16:13:46.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-16T16:13:46.765 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:13:46.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:47.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:47.082 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:47.082 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:13:47.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-04-16T16:13:47.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set dev brx.0 master ceph-brx 2024-04-16T16:13:47.082 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:13:47.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:47.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:47.313 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:13:47.313 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-16T16:13:47.361 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:13:47.361 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:13:47.361 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:13:47.407 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-16T16:13:47.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:47.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:13:47.566 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:13:47.567 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:13:48.282 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 0@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-16T16:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:13:48.290 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:13:48.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:48.290 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:13:48.392 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:13:48.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:13:48.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:48.535 INFO:teuthology.orchestra.run.smithi045.stdout:client4662 2024-04-16T16:13:48.536 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:13:48.536 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:13:48.590 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:13:48.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:13:48.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:48.727 INFO:teuthology.orchestra.run.smithi045.stdout:client4662 2024-04-16T16:13:48.728 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:13:48.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:13:48.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:48.871 INFO:teuthology.orchestra.run.smithi045.stdout:client4662 2024-04-16T16:13:48.871 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/748c4b85-0190-4c59-80ad-098a6a5007c0.client4662/status') 2024-04-16T16:13:48.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:49.031 INFO:teuthology.orchestra.run.smithi045.stdout:instance: client.4662 (3)192.168.0.1:0/42203625 2024-04-16T16:13:49.031 INFO:teuthology.orchestra.run.smithi045.stdout:blocklisted: false 2024-04-16T16:13:49.031 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records in 2024-04-16T16:13:49.031 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records out 2024-04-16T16:13:49.031 INFO:teuthology.orchestra.run.smithi045.stderr:67 bytes copied, 6.817e-05 s, 983 kB/s 2024-04-16T16:13:49.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:49.046 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-04-16T16:13:49.046 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-16T16:13:49.046 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-16T16:13:49.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:13:49.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.272+0000 7f2b42c00640 1 Processor -- start 2024-04-16T16:13:49.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.272+0000 7f2b42c00640 1 -- start start 2024-04-16T16:13:49.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.274+0000 7f2b42c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2b3c147840 0x7f2b3c15fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:49.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.274+0000 7f2b42c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b3c0aed90 0x7f2b3c072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:49.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.274+0000 7f2b42c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2b3c0728c0 0x7f2b3c162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:49.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.274+0000 7f2b42c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2b3c0afe80 con 0x7f2b3c0728c0 2024-04-16T16:13:49.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.274+0000 7f2b42c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2b3c147c20 con 0x7f2b3c147840 2024-04-16T16:13:49.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.274+0000 7f2b42c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2b3c0af270 con 0x7f2b3c0aed90 2024-04-16T16:13:49.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.274+0000 7f2b40c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2b3c0728c0 0x7f2b3c162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:49.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.274+0000 7f2b40c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2b3c0728c0 0x7f2b3c162610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52926/0 (socket says 172.21.15.45:52926) 2024-04-16T16:13:49.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.274+0000 7f2b40c00640 1 -- 172.21.15.45:0/127985596 learned_addr learned my addr 172.21.15.45:0/127985596 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:49.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.275+0000 7f2b3b400640 1 --2- 172.21.15.45:0/127985596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b3c0aed90 0x7f2b3c072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:49.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.275+0000 7f2b3be00640 1 --2- 172.21.15.45:0/127985596 >> v2:172.21.15.103:3300/0 conn(0x7f2b3c147840 0x7f2b3c15fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:49.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.275+0000 7f2b40c00640 1 -- 172.21.15.45:0/127985596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b3c0aed90 msgr2=0x7f2b3c072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:49.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.275+0000 7f2b40c00640 1 --2- 172.21.15.45:0/127985596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b3c0aed90 0x7f2b3c072380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:49.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.275+0000 7f2b40c00640 1 -- 172.21.15.45:0/127985596 >> v2:172.21.15.103:3300/0 conn(0x7f2b3c147840 msgr2=0x7f2b3c15fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:49.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.275+0000 7f2b40c00640 1 --2- 172.21.15.45:0/127985596 >> v2:172.21.15.103:3300/0 conn(0x7f2b3c147840 0x7f2b3c15fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:49.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.275+0000 7f2b40c00640 1 -- 172.21.15.45:0/127985596 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b3c0af550 con 0x7f2b3c0728c0 2024-04-16T16:13:49.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.275+0000 7f2b3be00640 1 --2- 172.21.15.45:0/127985596 >> v2:172.21.15.103:3300/0 conn(0x7f2b3c147840 0x7f2b3c15fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:49.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.276+0000 7f2b40c00640 1 --2- 172.21.15.45:0/127985596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b3c0728c0 0x7f2b3c162610 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c06ab80 tx=0x7f2b3402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f25edeb69b7b0c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:49.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.276+0000 7f2b3aa00640 1 -- 172.21.15.45:0/127985596 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3402ca80 con 0x7f2b3c0728c0 2024-04-16T16:13:49.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.276+0000 7f2b3aa00640 1 -- 172.21.15.45:0/127985596 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b3402cc20 con 0x7f2b3c0728c0 2024-04-16T16:13:49.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.276+0000 7f2b3aa00640 1 -- 172.21.15.45:0/127985596 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b340343f0 con 0x7f2b3c0728c0 2024-04-16T16:13:49.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.277+0000 7f2b42c00640 1 -- 172.21.15.45:0/127985596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b3c0728c0 msgr2=0x7f2b3c162610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:49.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.277+0000 7f2b42c00640 1 --2- 172.21.15.45:0/127985596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b3c0728c0 0x7f2b3c162610 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c06ab80 tx=0x7f2b3402bee0 comp rx=0 tx=0).stop 2024-04-16T16:13:49.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.277+0000 7f2b42c00640 1 -- 172.21.15.45:0/127985596 shutdown_connections 2024-04-16T16:13:49.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.277+0000 7f2b42c00640 1 --2- 172.21.15.45:0/127985596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b3c0728c0 0x7f2b3c162610 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:49.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.277+0000 7f2b42c00640 1 --2- 172.21.15.45:0/127985596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b3c0aed90 0x7f2b3c072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.277+0000 7f2b42c00640 1 --2- 172.21.15.45:0/127985596 >> v2:172.21.15.103:3300/0 conn(0x7f2b3c147840 0x7f2b3c15fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.277+0000 7f2b42c00640 1 -- 172.21.15.45:0/127985596 >> 172.21.15.45:0/127985596 conn(0x7f2b3c0b2750 msgr2=0x7f2b3c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.277+0000 7f2b42c00640 1 -- 172.21.15.45:0/127985596 shutdown_connections 2024-04-16T16:13:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.277+0000 7f2b42c00640 1 -- 172.21.15.45:0/127985596 wait complete. 2024-04-16T16:13:49.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.279+0000 7f2b42c00640 1 Processor -- start 2024-04-16T16:13:49.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.279+0000 7f2b42c00640 1 -- start start 2024-04-16T16:13:49.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.280+0000 7f2b42c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b3c0728c0 0x7f2b3c161eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:49.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.280+0000 7f2b42c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b3c0aed90 0x7f2b3c15e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:49.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.280+0000 7f2b42c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3c147840 0x7f2b3c160510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:49.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.280+0000 7f2b42c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3c160040 con 0x7f2b3c0728c0 2024-04-16T16:13:49.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.280+0000 7f2b42c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3c0afb10 con 0x7f2b3c147840 2024-04-16T16:13:49.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.280+0000 7f2b42c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2b3c1601c0 con 0x7f2b3c0aed90 2024-04-16T16:13:49.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.280+0000 7f2b3be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b3c0728c0 0x7f2b3c161eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:49.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.280+0000 7f2b3be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b3c0728c0 0x7f2b3c161eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52938/0 (socket says 172.21.15.45:52938) 2024-04-16T16:13:49.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.280+0000 7f2b3be00640 1 -- 172.21.15.45:0/1602662834 learned_addr learned my addr 172.21.15.45:0/1602662834 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:49.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.281+0000 7f2b3be00640 1 -- 172.21.15.45:0/1602662834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b3c0aed90 msgr2=0x7f2b3c15e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:49.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.281+0000 7f2b3b400640 1 --2- 172.21.15.45:0/1602662834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b3c0aed90 0x7f2b3c15e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:49.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.281+0000 7f2b40c00640 1 --2- 172.21.15.45:0/1602662834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3c147840 0x7f2b3c160510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:49.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.281+0000 7f2b3be00640 1 --2- 172.21.15.45:0/1602662834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b3c0aed90 0x7f2b3c15e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:49.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.281+0000 7f2b3be00640 1 -- 172.21.15.45:0/1602662834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3c147840 msgr2=0x7f2b3c160510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:49.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.281+0000 7f2b3be00640 1 --2- 172.21.15.45:0/1602662834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3c147840 0x7f2b3c160510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:49.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.281+0000 7f2b3be00640 1 -- 172.21.15.45:0/1602662834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b3c160e10 con 0x7f2b3c0728c0 2024-04-16T16:13:49.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.281+0000 7f2b40c00640 1 --2- 172.21.15.45:0/1602662834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3c147840 0x7f2b3c160510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:49.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.281+0000 7f2b3be00640 1 --2- 172.21.15.45:0/1602662834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b3c0728c0 0x7f2b3c161eb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00c7f0 tx=0x7f2b2c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.281+0000 7f2b33e00640 1 -- 172.21.15.45:0/1602662834 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c015070 con 0x7f2b3c0728c0 2024-04-16T16:13:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.281+0000 7f2b33e00640 1 -- 172.21.15.45:0/1602662834 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b2c007a30 con 0x7f2b3c0728c0 2024-04-16T16:13:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.281+0000 7f2b33e00640 1 -- 172.21.15.45:0/1602662834 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c007e00 con 0x7f2b3c0728c0 2024-04-16T16:13:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.281+0000 7f2b42c00640 1 -- 172.21.15.45:0/1602662834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b3c1610f0 con 0x7f2b3c0728c0 2024-04-16T16:13:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.282+0000 7f2b42c00640 1 -- 172.21.15.45:0/1602662834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b3c1616b0 con 0x7f2b3c0728c0 2024-04-16T16:13:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.283+0000 7f2b42c00640 1 -- 172.21.15.45:0/1602662834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b3c161af0 con 0x7f2b3c0728c0 2024-04-16T16:13:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.284+0000 7f2b33e00640 1 -- 172.21.15.45:0/1602662834 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2b2c018070 con 0x7f2b3c0728c0 2024-04-16T16:13:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.285+0000 7f2b33e00640 1 --2- 172.21.15.45:0/1602662834 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2b1c06c8f0 0x7f2b1c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.285+0000 7f2b33e00640 1 -- 172.21.15.45:0/1602662834 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2b2c08bbb0 con 0x7f2b3c0728c0 2024-04-16T16:13:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.285+0000 7f2b3b400640 1 --2- 172.21.15.45:0/1602662834 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2b1c06c8f0 0x7f2b1c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.287+0000 7f2b3b400640 1 --2- 172.21.15.45:0/1602662834 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2b1c06c8f0 0x7f2b1c06edb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c162660 tx=0x7f2b28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:49.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.290+0000 7f2b33e00640 1 -- 172.21.15.45:0/1602662834 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2b2c059f40 con 0x7f2b3c0728c0 2024-04-16T16:13:49.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.537+0000 7f2b42c00640 1 -- 172.21.15.45:0/1602662834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2b3c075780 con 0x7f2b3c0728c0 2024-04-16T16:13:49.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.538+0000 7f2b33e00640 1 -- 172.21.15.45:0/1602662834 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f2b2c059840 con 0x7f2b3c0728c0 2024-04-16T16:13:49.538 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:13:49.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.541+0000 7f2b42c00640 1 -- 172.21.15.45:0/1602662834 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2b1c06c8f0 msgr2=0x7f2b1c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:49.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.541+0000 7f2b42c00640 1 --2- 172.21.15.45:0/1602662834 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2b1c06c8f0 0x7f2b1c06edb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c162660 tx=0x7f2b28008040 comp rx=0 tx=0).stop 2024-04-16T16:13:49.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.541+0000 7f2b42c00640 1 -- 172.21.15.45:0/1602662834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b3c0728c0 msgr2=0x7f2b3c161eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:49.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.541+0000 7f2b42c00640 1 --2- 172.21.15.45:0/1602662834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b3c0728c0 0x7f2b3c161eb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00c7f0 tx=0x7f2b2c00cbf0 comp rx=0 tx=0).stop 2024-04-16T16:13:49.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.541+0000 7f2b42c00640 1 -- 172.21.15.45:0/1602662834 shutdown_connections 2024-04-16T16:13:49.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.541+0000 7f2b42c00640 1 --2- 172.21.15.45:0/1602662834 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2b1c06c8f0 0x7f2b1c06edb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:49.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.541+0000 7f2b42c00640 1 --2- 172.21.15.45:0/1602662834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3c147840 0x7f2b3c160510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:49.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.541+0000 7f2b42c00640 1 --2- 172.21.15.45:0/1602662834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b3c0aed90 0x7f2b3c15e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:49.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.541+0000 7f2b42c00640 1 --2- 172.21.15.45:0/1602662834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b3c0728c0 0x7f2b3c161eb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:49.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.541+0000 7f2b42c00640 1 -- 172.21.15.45:0/1602662834 >> 172.21.15.45:0/1602662834 conn(0x7f2b3c0b2750 msgr2=0x7f2b3c0b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:49.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.541+0000 7f2b42c00640 1 -- 172.21.15.45:0/1602662834 shutdown_connections 2024-04-16T16:13:49.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49.541+0000 7f2b42c00640 1 -- 172.21.15.45:0/1602662834 wait complete. 2024-04-16T16:13:49.561 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-16T16:13:49.561 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:13:49.561 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-04-16T16:13:49.561 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-16T16:13:49.561 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-16T16:13:49.561 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-16T16:13:49.561 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-16T16:13:49.561 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:13:49.561 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-16T16:13:49.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:49.562 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-04-16T16:13:49.615 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:13:49.615 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:13:49.615 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:13:49.615 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:49.615 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-04-16T16:13:49.615 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:49.615 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:13:49.615 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:49.615 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-04-16T16:13:49.615 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:13:49.615 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:49.615 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-04-16T16:13:49.615 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:13:49.616 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:49.616 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-04-16T16:13:49.616 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-16T16:13:49.616 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42572sec preferred_lft 42572sec 2024-04-16T16:13:49.616 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-04-16T16:13:49.616 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:49.616 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:13:49.616 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:49.616 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-04-16T16:13:49.616 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:13:49.616 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:49.616 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:13:49.616 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:49.616 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::50f1:99ff:fe2b:225b/64 scope link 2024-04-16T16:13:49.617 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:49.617 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:13:49.617 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T16:13:49.617 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::7007:c3ff:fe13:7bcd/64 scope link 2024-04-16T16:13:49.617 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:49.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:49.617 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:13:49.703 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-04-16T16:13:49.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:49.708 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-04-16T16:13:49.765 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-04-16T16:13:49.766 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:13:49.766 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-16T16:13:49.766 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-16T16:13:49.766 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:13:49.767 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:49.767 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:13:49.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-16T16:13:49.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-16T16:13:49.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-16T16:13:49.767 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:13:49.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:50.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:50.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:50.030 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T16:13:50.080 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-16T16:13:50.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:50.080 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list-id 2024-04-16T16:13:50.128 INFO:teuthology.orchestra.run.smithi045.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-16T16:13:50.128 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:50.128 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:13:50.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-16T16:13:50.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-16T16:13:50.128 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:13:50.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:50.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:50.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:50.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-16T16:13:50.427 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:13:50.427 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:13:50.427 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:13:50.427 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:50.427 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-04-16T16:13:50.427 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:50.427 INFO:teuthology.orchestra.run.smithi045.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:13:50.428 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether d2:b3:3b:79:e6:1d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-16T16:13:50.428 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-16T16:13:50.428 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:50.428 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::d0b3:3bff:fe79:e61d/64 scope link 2024-04-16T16:13:50.428 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:50.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:50.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-16T16:13:50.527 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:13:50.527 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:13:50.527 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:13:50.527 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:50.527 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-04-16T16:13:50.527 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:50.527 INFO:teuthology.orchestra.run.smithi045.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:13:50.527 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether d2:b3:3b:79:e6:1d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-16T16:13:50.527 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-16T16:13:50.527 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:50.528 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::d0b3:3bff:fe79:e61d/64 scope link 2024-04-16T16:13:50.528 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:50.531 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-16T16:13:50.531 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:50.531 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:13:50.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-16T16:13:50.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-16T16:13:50.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-16T16:13:50.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-16T16:13:50.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-16T16:13:50.532 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:13:50.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:50.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:50.885 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:50.885 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:13:50.885 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.1 up 2024-04-16T16:13:50.885 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set dev brx.1 master ceph-brx 2024-04-16T16:13:50.886 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:13:50.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:51.124 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:13:51.124 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-16T16:13:51.171 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-16T16:13:51.171 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:13:51.171 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-16T16:13:51.273 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-04-16T16:13:51.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:51.273 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:13:51.420 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:13:51.420 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:13:51.455 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:13:51.455 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:13:51.455 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:13:51.455 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 2@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:13:51.456 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:13:51.459 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-04-16T16:13:51.459 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:13:51.459 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:13:51.459 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:13:51.459 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:13:51.465 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:13:51.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:51.465 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-16T16:13:51.545 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:13:51.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:13:51.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:51.672 INFO:teuthology.orchestra.run.smithi045.stdout:client4418 2024-04-16T16:13:51.673 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:13:51.673 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:13:51.727 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:13:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:13:51.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:51.855 INFO:teuthology.orchestra.run.smithi045.stdout:client4418 2024-04-16T16:13:51.855 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:13:51.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:13:51.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:51.974 INFO:teuthology.orchestra.run.smithi045.stdout:client4418 2024-04-16T16:13:51.974 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/748c4b85-0190-4c59-80ad-098a6a5007c0.client4418/status') 2024-04-16T16:13:52.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:52.124 INFO:teuthology.orchestra.run.smithi045.stdout:instance: client.4418 (3)172.21.15.45:0/1733196857 2024-04-16T16:13:52.124 INFO:teuthology.orchestra.run.smithi045.stdout:blocklisted: false 2024-04-16T16:13:52.124 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records in 2024-04-16T16:13:52.125 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records out 2024-04-16T16:13:52.125 INFO:teuthology.orchestra.run.smithi045.stderr:70 bytes copied, 7.4977e-05 s, 934 kB/s 2024-04-16T16:13:52.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:52.138 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-04-16T16:13:52.138 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-16T16:13:52.138 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T16:13:52.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:13:52.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.356+0000 7ff82b200640 1 Processor -- start 2024-04-16T16:13:52.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.356+0000 7ff82b200640 1 -- start start 2024-04-16T16:13:52.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.358+0000 7ff82b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff82c146220 0x7ff82c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:52.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.358+0000 7ff82b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff82c1608a0 0x7ff82c160d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:52.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.358+0000 7ff82b200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff82c15d360 con 0x7ff82c06f7a0 2024-04-16T16:13:52.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.358+0000 7ff82b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff82c160410 con 0x7ff82c146220 2024-04-16T16:13:52.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.358+0000 7ff82b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff82c15d640 con 0x7ff82c1608a0 2024-04-16T16:13:52.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.358+0000 7ff823e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff82c06f7a0 0x7ff82c06df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36490/0 (socket says 172.21.15.45:36490) 2024-04-16T16:13:52.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.358+0000 7ff823e00640 1 -- 172.21.15.45:0/868255319 learned_addr learned my addr 172.21.15.45:0/868255319 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:52.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.358+0000 7ff82a800640 1 --2- 172.21.15.45:0/868255319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff82c1608a0 0x7ff82c160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:52.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.358+0000 7ff829e00640 1 --2- 172.21.15.45:0/868255319 >> v2:172.21.15.103:3300/0 conn(0x7ff82c146220 0x7ff82c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:52.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.359+0000 7ff829400640 1 -- 172.21.15.45:0/868255319 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3092358292 0 0) 0x7ff82c15d360 con 0x7ff82c06f7a0 2024-04-16T16:13:52.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.359+0000 7ff829400640 1 -- 172.21.15.45:0/868255319 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff810003520 con 0x7ff82c06f7a0 2024-04-16T16:13:52.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.359+0000 7ff82a800640 1 -- 172.21.15.45:0/868255319 >> v2:172.21.15.103:3300/0 conn(0x7ff82c146220 msgr2=0x7ff82c15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:52.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.359+0000 7ff82a800640 1 --2- 172.21.15.45:0/868255319 >> v2:172.21.15.103:3300/0 conn(0x7ff82c146220 0x7ff82c15ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:52.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.359+0000 7ff82a800640 1 -- 172.21.15.45:0/868255319 >> v1:172.21.15.45:6789/0 conn(0x7ff82c06f7a0 legacy=0x7ff82c06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:52.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.359+0000 7ff82a800640 1 -- 172.21.15.45:0/868255319 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82c15d920 con 0x7ff82c1608a0 2024-04-16T16:13:52.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.362+0000 7ff82a800640 1 --2- 172.21.15.45:0/868255319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff82c1608a0 0x7ff82c160d30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff824009d90 tx=0x7ff82402beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f30c40a186a11ea0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:52.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.362+0000 7ff829400640 1 -- 172.21.15.45:0/868255319 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff824039070 con 0x7ff82c1608a0 2024-04-16T16:13:52.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.362+0000 7ff829400640 1 -- 172.21.15.45:0/868255319 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff82402cc60 con 0x7ff82c1608a0 2024-04-16T16:13:52.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.362+0000 7ff829400640 1 -- 172.21.15.45:0/868255319 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff824034660 con 0x7ff82c1608a0 2024-04-16T16:13:52.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.363+0000 7ff82b200640 1 -- 172.21.15.45:0/868255319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff82c1608a0 msgr2=0x7ff82c160d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:52.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.363+0000 7ff82b200640 1 --2- 172.21.15.45:0/868255319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff82c1608a0 0x7ff82c160d30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff824009d90 tx=0x7ff82402beb0 comp rx=0 tx=0).stop 2024-04-16T16:13:52.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.363+0000 7ff82b200640 1 -- 172.21.15.45:0/868255319 shutdown_connections 2024-04-16T16:13:52.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.363+0000 7ff82b200640 1 --2- 172.21.15.45:0/868255319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff82c1608a0 0x7ff82c160d30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:52.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.363+0000 7ff82b200640 1 --2- 172.21.15.45:0/868255319 >> v2:172.21.15.103:3300/0 conn(0x7ff82c146220 0x7ff82c15ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:52.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.363+0000 7ff82b200640 1 -- 172.21.15.45:0/868255319 >> 172.21.15.45:0/868255319 conn(0x7ff82c06b930 msgr2=0x7ff82c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:52.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.363+0000 7ff82b200640 1 -- 172.21.15.45:0/868255319 shutdown_connections 2024-04-16T16:13:52.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.363+0000 7ff82b200640 1 -- 172.21.15.45:0/868255319 wait complete. 2024-04-16T16:13:52.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.365+0000 7ff82b200640 1 Processor -- start 2024-04-16T16:13:52.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.365+0000 7ff82b200640 1 -- start start 2024-04-16T16:13:52.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.365+0000 7ff82b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff82c06f7a0 0x7ff82c1425a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:52.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.365+0000 7ff82b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff82c146220 0x7ff82c142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:52.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.365+0000 7ff82b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff82c1608a0 0x7ff82c0ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:52.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.365+0000 7ff82b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c06fc20 con 0x7ff82c06f7a0 2024-04-16T16:13:52.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.365+0000 7ff82b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c15dc00 con 0x7ff82c146220 2024-04-16T16:13:52.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.365+0000 7ff82b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff82c06fda0 con 0x7ff82c1608a0 2024-04-16T16:13:52.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.365+0000 7ff829e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff82c06f7a0 0x7ff82c1425a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:52.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.365+0000 7ff829e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff82c06f7a0 0x7ff82c1425a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52948/0 (socket says 172.21.15.45:52948) 2024-04-16T16:13:52.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.365+0000 7ff829e00640 1 -- 172.21.15.45:0/2802339543 learned_addr learned my addr 172.21.15.45:0/2802339543 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:52.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.366+0000 7ff823e00640 1 --2- 172.21.15.45:0/2802339543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff82c146220 0x7ff82c142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:52.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.366+0000 7ff82a800640 1 --2- 172.21.15.45:0/2802339543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff82c1608a0 0x7ff82c0ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:52.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.366+0000 7ff829e00640 1 -- 172.21.15.45:0/2802339543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff82c1608a0 msgr2=0x7ff82c0ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:52.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.366+0000 7ff829e00640 1 --2- 172.21.15.45:0/2802339543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff82c1608a0 0x7ff82c0ceed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:52.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.366+0000 7ff829e00640 1 -- 172.21.15.45:0/2802339543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff82c146220 msgr2=0x7ff82c142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:52.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.366+0000 7ff829e00640 1 --2- 172.21.15.45:0/2802339543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff82c146220 0x7ff82c142ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:52.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.366+0000 7ff829e00640 1 -- 172.21.15.45:0/2802339543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82c13ba00 con 0x7ff82c06f7a0 2024-04-16T16:13:52.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.366+0000 7ff829e00640 1 --2- 172.21.15.45:0/2802339543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff82c06f7a0 0x7ff82c1425a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff81c00e9b0 tx=0x7ff81c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:52.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.366+0000 7ff822000640 1 -- 172.21.15.45:0/2802339543 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c017070 con 0x7ff82c06f7a0 2024-04-16T16:13:52.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.366+0000 7ff822000640 1 -- 172.21.15.45:0/2802339543 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff81c00c9a0 con 0x7ff82c06f7a0 2024-04-16T16:13:52.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.366+0000 7ff822000640 1 -- 172.21.15.45:0/2802339543 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c00cd90 con 0x7ff82c06f7a0 2024-04-16T16:13:52.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.366+0000 7ff82b200640 1 -- 172.21.15.45:0/2802339543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff82c13bce0 con 0x7ff82c06f7a0 2024-04-16T16:13:52.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.367+0000 7ff82b200640 1 -- 172.21.15.45:0/2802339543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff82c13c2a0 con 0x7ff82c06f7a0 2024-04-16T16:13:52.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.368+0000 7ff82b200640 1 -- 172.21.15.45:0/2802339543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff82c075780 con 0x7ff82c06f7a0 2024-04-16T16:13:52.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.370+0000 7ff822000640 1 -- 172.21.15.45:0/2802339543 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff81c01a070 con 0x7ff82c06f7a0 2024-04-16T16:13:52.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.370+0000 7ff822000640 1 --2- 172.21.15.45:0/2802339543 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff80806c8f0 0x7ff80806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:52.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.370+0000 7ff822000640 1 -- 172.21.15.45:0/2802339543 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff81c08c050 con 0x7ff82c06f7a0 2024-04-16T16:13:52.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.371+0000 7ff823e00640 1 --2- 172.21.15.45:0/2802339543 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff80806c8f0 0x7ff80806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:52.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.372+0000 7ff823e00640 1 --2- 172.21.15.45:0/2802339543 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff80806c8f0 0x7ff80806edb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff82c13b740 tx=0x7ff814009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:52.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.375+0000 7ff822000640 1 -- 172.21.15.45:0/2802339543 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff81c05a350 con 0x7ff82c06f7a0 2024-04-16T16:13:52.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.621+0000 7ff82b200640 1 -- 172.21.15.45:0/2802339543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff82c0cff60 con 0x7ff82c06f7a0 2024-04-16T16:13:52.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.622+0000 7ff822000640 1 -- 172.21.15.45:0/2802339543 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7ff81c059c50 con 0x7ff82c06f7a0 2024-04-16T16:13:52.622 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:13:52.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.625+0000 7ff82b200640 1 -- 172.21.15.45:0/2802339543 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff80806c8f0 msgr2=0x7ff80806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:52.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.625+0000 7ff82b200640 1 --2- 172.21.15.45:0/2802339543 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff80806c8f0 0x7ff80806edb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff82c13b740 tx=0x7ff814009040 comp rx=0 tx=0).stop 2024-04-16T16:13:52.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.625+0000 7ff82b200640 1 -- 172.21.15.45:0/2802339543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff82c06f7a0 msgr2=0x7ff82c1425a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:52.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.625+0000 7ff82b200640 1 --2- 172.21.15.45:0/2802339543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff82c06f7a0 0x7ff82c1425a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff81c00e9b0 tx=0x7ff81c00ee70 comp rx=0 tx=0).stop 2024-04-16T16:13:52.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.625+0000 7ff82b200640 1 -- 172.21.15.45:0/2802339543 shutdown_connections 2024-04-16T16:13:52.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.625+0000 7ff82b200640 1 --2- 172.21.15.45:0/2802339543 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff80806c8f0 0x7ff80806edb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:52.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.625+0000 7ff82b200640 1 --2- 172.21.15.45:0/2802339543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff82c1608a0 0x7ff82c0ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:52.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.625+0000 7ff82b200640 1 --2- 172.21.15.45:0/2802339543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff82c146220 0x7ff82c142ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:52.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.625+0000 7ff82b200640 1 --2- 172.21.15.45:0/2802339543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff82c06f7a0 0x7ff82c1425a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:52.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.625+0000 7ff82b200640 1 -- 172.21.15.45:0/2802339543 >> 172.21.15.45:0/2802339543 conn(0x7ff82c06b930 msgr2=0x7ff82c0c5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:52.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.625+0000 7ff82b200640 1 -- 172.21.15.45:0/2802339543 shutdown_connections 2024-04-16T16:13:52.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:52.626+0000 7ff82b200640 1 -- 172.21.15.45:0/2802339543 wait complete. 2024-04-16T16:13:52.647 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-16T16:13:52.647 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:13:52.647 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-04-16T16:13:52.647 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-16T16:13:52.647 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T16:13:52.647 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-16T16:13:52.647 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-16T16:13:52.647 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:13:52.647 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:13:52.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:52.647 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-04-16T16:13:52.704 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:13:52.704 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:13:52.704 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:13:52.704 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:52.704 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-16T16:13:52.704 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:52.705 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:13:52.705 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:52.705 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-04-16T16:13:52.705 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:13:52.705 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:52.705 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-04-16T16:13:52.705 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:13:52.705 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:52.705 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:13:52.705 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:52.705 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-16T16:13:52.705 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42592sec preferred_lft 42592sec 2024-04-16T16:13:52.705 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-04-16T16:13:52.705 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:52.706 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-16T16:13:52.707 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:52.707 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:13:52.707 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add name ceph-brx type bridge 2024-04-16T16:13:52.707 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip addr flush dev ceph-brx 2024-04-16T16:13:52.707 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set ceph-brx up 2024-04-16T16:13:52.707 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-16T16:13:52.707 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:13:52.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:53.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:13:53.041 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:13:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:53.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:53.236 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:13:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-16T16:13:53.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:53.315 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-04-16T16:13:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-04-16T16:13:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:13:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-16T16:13:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-16T16:13:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:13:53.373 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:53.373 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:13:53.374 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-16T16:13:53.374 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-16T16:13:53.374 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-16T16:13:53.374 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:13:53.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:54.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:54.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:54.163 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-16T16:13:54.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:54.216 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list-id 2024-04-16T16:13:54.268 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:54.268 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:13:54.268 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T16:13:54.268 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-16T16:13:54.268 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:13:54.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:54.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:54.472 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-16T16:13:54.472 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:54.473 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:13:54.473 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-16T16:13:54.473 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-16T16:13:54.473 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-16T16:13:54.473 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-16T16:13:54.473 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-16T16:13:54.473 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:13:54.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:54.822 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:54.831 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:54.831 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:13:54.831 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.0 up 2024-04-16T16:13:54.831 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set dev brx.0 master ceph-brx 2024-04-16T16:13:54.831 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:13:54.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:55.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:55.057 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:13:55.057 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-16T16:13:55.106 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:13:55.107 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:13:55.107 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:13:55.156 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-16T16:13:55.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:55.156 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:13:55.287 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:13:55.288 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:13:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:13:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:13:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:13:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with new device syntax: 1@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:13:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:13:56.000 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-16T16:13:56.000 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:13:56.000 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:13:56.000 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:13:56.000 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:13:56.006 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:13:56.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:56.006 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:13:56.094 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:13:56.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:13:56.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:56.217 INFO:teuthology.orchestra.run.smithi103.stdout:client4677 2024-04-16T16:13:56.217 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:13:56.217 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:13:56.268 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:13:56.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:13:56.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:56.386 INFO:teuthology.orchestra.run.smithi103.stdout:client4677 2024-04-16T16:13:56.386 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:13:56.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:13:56.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:56.499 INFO:teuthology.orchestra.run.smithi103.stdout:client4677 2024-04-16T16:13:56.500 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/748c4b85-0190-4c59-80ad-098a6a5007c0.client4677/status') 2024-04-16T16:13:56.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:instance: client.4677 (3)172.21.15.103:0/4040154876 2024-04-16T16:13:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:blocklisted: false 2024-04-16T16:13:56.645 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records in 2024-04-16T16:13:56.645 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records out 2024-04-16T16:13:56.646 INFO:teuthology.orchestra.run.smithi103.stderr:71 bytes copied, 6.991e-05 s, 1.0 MB/s 2024-04-16T16:13:56.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:56.659 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-04-16T16:13:56.660 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-16T16:13:56.660 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-16T16:13:56.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:13:56.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.895+0000 7f1569600640 1 Processor -- start 2024-04-16T16:13:56.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.895+0000 7f1569600640 1 -- start start 2024-04-16T16:13:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.897+0000 7f1569600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1564072a90 0x7f15640ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.897+0000 7f1569600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f156415ea70 0x7f15640cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.897+0000 7f1569600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15640cf140 con 0x7f15641462e0 2024-04-16T16:13:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.897+0000 7f1569600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f15640cf6d0 con 0x7f156415ea70 2024-04-16T16:13:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.897+0000 7f1569600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f15641466c0 con 0x7f1564072a90 2024-04-16T16:13:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.897+0000 7f1563400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f156415ea70 0x7f15640cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.897+0000 7f1562a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f15641462e0 0x7f1564072380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49880/0 (socket says 172.21.15.45:49880) 2024-04-16T16:13:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.897+0000 7f1562a00640 1 -- 172.21.15.45:0/2706091893 learned_addr learned my addr 172.21.15.45:0/2706091893 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.897+0000 7f1562000640 1 --2- 172.21.15.45:0/2706091893 >> v2:172.21.15.103:3300/0 conn(0x7f1564072a90 0x7f15640ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.898+0000 7f1563400640 1 -- 172.21.15.45:0/2706091893 >> v2:172.21.15.103:3300/0 conn(0x7f1564072a90 msgr2=0x7f15640ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.898+0000 7f1563400640 1 --2- 172.21.15.45:0/2706091893 >> v2:172.21.15.103:3300/0 conn(0x7f1564072a90 0x7f15640ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.898+0000 7f1563400640 1 -- 172.21.15.45:0/2706091893 >> v1:172.21.15.45:6789/0 conn(0x7f15641462e0 legacy=0x7f1564072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.898+0000 7f1563400640 1 -- 172.21.15.45:0/2706091893 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15640cf3f0 con 0x7f156415ea70 2024-04-16T16:13:56.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.898+0000 7f1561600640 1 -- 172.21.15.45:0/2706091893 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2672327787 0 0) 0x7f15640cf140 con 0x7f15641462e0 2024-04-16T16:13:56.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.899+0000 7f1563400640 1 --2- 172.21.15.45:0/2706091893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f156415ea70 0x7f15640cec00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f156406f9b0 tx=0x7f155c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f57638cc7344f8f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:56.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.899+0000 7f1561600640 1 -- 172.21.15.45:0/2706091893 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c039070 con 0x7f156415ea70 2024-04-16T16:13:56.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.899+0000 7f1561600640 1 -- 172.21.15.45:0/2706091893 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f155c02cbf0 con 0x7f156415ea70 2024-04-16T16:13:56.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.899+0000 7f1561600640 1 -- 172.21.15.45:0/2706091893 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c034440 con 0x7f156415ea70 2024-04-16T16:13:56.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.900+0000 7f1569600640 1 -- 172.21.15.45:0/2706091893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f156415ea70 msgr2=0x7f15640cec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:56.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.900+0000 7f1569600640 1 --2- 172.21.15.45:0/2706091893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f156415ea70 0x7f15640cec00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f156406f9b0 tx=0x7f155c02be40 comp rx=0 tx=0).stop 2024-04-16T16:13:56.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.900+0000 7f1569600640 1 -- 172.21.15.45:0/2706091893 shutdown_connections 2024-04-16T16:13:56.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.900+0000 7f1569600640 1 --2- 172.21.15.45:0/2706091893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f156415ea70 0x7f15640cec00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:56.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.900+0000 7f1569600640 1 --2- 172.21.15.45:0/2706091893 >> v2:172.21.15.103:3300/0 conn(0x7f1564072a90 0x7f15640ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:56.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.900+0000 7f1569600640 1 -- 172.21.15.45:0/2706091893 >> 172.21.15.45:0/2706091893 conn(0x7f15640b2750 msgr2=0x7f15640b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:56.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.900+0000 7f1569600640 1 -- 172.21.15.45:0/2706091893 shutdown_connections 2024-04-16T16:13:56.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.900+0000 7f1569600640 1 -- 172.21.15.45:0/2706091893 wait complete. 2024-04-16T16:13:56.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.902+0000 7f1569600640 1 Processor -- start 2024-04-16T16:13:56.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.902+0000 7f1569600640 1 -- start start 2024-04-16T16:13:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.903+0000 7f1569600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1564072a90 0x7f15640b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.903+0000 7f1569600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15641462e0 0x7f156415acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.903+0000 7f1569600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f156415ea70 0x7f15641445e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.903+0000 7f1569600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f156415d540 con 0x7f156415ea70 2024-04-16T16:13:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.903+0000 7f1569600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f156415d260 con 0x7f1564072a90 2024-04-16T16:13:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.903+0000 7f1569600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1564166490 con 0x7f15641462e0 2024-04-16T16:13:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.903+0000 7f1563400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f156415ea70 0x7f15641445e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.903+0000 7f1563400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f156415ea70 0x7f15641445e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54422/0 (socket says 172.21.15.45:54422) 2024-04-16T16:13:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.903+0000 7f1563400640 1 -- 172.21.15.45:0/3028800478 learned_addr learned my addr 172.21.15.45:0/3028800478 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.904+0000 7f1562000640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15641462e0 0x7f156415acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.904+0000 7f1562a00640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1564072a90 0x7f15640b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.904+0000 7f1563400640 1 -- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15641462e0 msgr2=0x7f156415acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.904+0000 7f1563400640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15641462e0 0x7f156415acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.904+0000 7f1563400640 1 -- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1564072a90 msgr2=0x7f15640b5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.904+0000 7f1563400640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1564072a90 0x7f15640b5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.904+0000 7f1563400640 1 -- 172.21.15.45:0/3028800478 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156415b5e0 con 0x7f156415ea70 2024-04-16T16:13:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.904+0000 7f1562000640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15641462e0 0x7f156415acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.904+0000 7f1562a00640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1564072a90 0x7f15640b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.904+0000 7f1563400640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f156415ea70 0x7f15641445e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f155c009e80 tx=0x7f155c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:56.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.905+0000 7f155aa00640 1 -- 172.21.15.45:0/3028800478 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c039050 con 0x7f156415ea70 2024-04-16T16:13:56.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.905+0000 7f155aa00640 1 -- 172.21.15.45:0/3028800478 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f155c03a040 con 0x7f156415ea70 2024-04-16T16:13:56.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.905+0000 7f1569600640 1 -- 172.21.15.45:0/3028800478 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f156415b8c0 con 0x7f156415ea70 2024-04-16T16:13:56.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.905+0000 7f155aa00640 1 -- 172.21.15.45:0/3028800478 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c03e610 con 0x7f156415ea70 2024-04-16T16:13:56.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.905+0000 7f1569600640 1 -- 172.21.15.45:0/3028800478 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f156406bc50 con 0x7f156415ea70 2024-04-16T16:13:56.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.906+0000 7f1569600640 1 -- 172.21.15.45:0/3028800478 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1564075780 con 0x7f156415ea70 2024-04-16T16:13:56.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.907+0000 7f155aa00640 1 -- 172.21.15.45:0/3028800478 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f155c047020 con 0x7f156415ea70 2024-04-16T16:13:56.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.909+0000 7f155aa00640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f154406c8f0 0x7f154406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:56.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.909+0000 7f155aa00640 1 -- 172.21.15.45:0/3028800478 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f155c0adef0 con 0x7f156415ea70 2024-04-16T16:13:56.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.910+0000 7f1562a00640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f154406c8f0 0x7f154406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:56.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.911+0000 7f1562a00640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f154406c8f0 0x7f154406edb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1554000f60 tx=0x7f1554009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:56.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:56.914+0000 7f155aa00640 1 -- 172.21.15.45:0/3028800478 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f155c04e020 con 0x7f156415ea70 2024-04-16T16:13:57.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:57.164+0000 7f1569600640 1 -- 172.21.15.45:0/3028800478 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f156413f6a0 con 0x7f156415ea70 2024-04-16T16:13:57.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:57.165+0000 7f155aa00640 1 -- 172.21.15.45:0/3028800478 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f155c07c310 con 0x7f156415ea70 2024-04-16T16:13:57.164 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:13:57.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:57.168+0000 7f1569600640 1 -- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f154406c8f0 msgr2=0x7f154406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:57.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:57.168+0000 7f1569600640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f154406c8f0 0x7f154406edb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1554000f60 tx=0x7f1554009040 comp rx=0 tx=0).stop 2024-04-16T16:13:57.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:57.168+0000 7f1569600640 1 -- 172.21.15.45:0/3028800478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f156415ea70 msgr2=0x7f15641445e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:57.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:57.168+0000 7f1569600640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f156415ea70 0x7f15641445e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f155c009e80 tx=0x7f155c004a40 comp rx=0 tx=0).stop 2024-04-16T16:13:57.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:57.168+0000 7f1569600640 1 -- 172.21.15.45:0/3028800478 shutdown_connections 2024-04-16T16:13:57.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:57.168+0000 7f1569600640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f154406c8f0 0x7f154406edb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:57.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:57.168+0000 7f1569600640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f156415ea70 0x7f15641445e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:57.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:57.168+0000 7f1569600640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15641462e0 0x7f156415acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:57.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:57.168+0000 7f1569600640 1 --2- 172.21.15.45:0/3028800478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1564072a90 0x7f15640b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:57.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:57.168+0000 7f1569600640 1 -- 172.21.15.45:0/3028800478 >> 172.21.15.45:0/3028800478 conn(0x7f15640b2750 msgr2=0x7f156406fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:57.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:57.168+0000 7f1569600640 1 -- 172.21.15.45:0/3028800478 shutdown_connections 2024-04-16T16:13:57.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:57.168+0000 7f1569600640 1 -- 172.21.15.45:0/3028800478 wait complete. 2024-04-16T16:13:57.188 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-16T16:13:57.188 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:13:57.188 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-04-16T16:13:57.188 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-16T16:13:57.188 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-16T16:13:57.188 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-16T16:13:57.188 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-16T16:13:57.188 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:13:57.189 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-16T16:13:57.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:57.189 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-04-16T16:13:57.245 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:13:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:13:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:13:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-16T16:13:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:13:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-04-16T16:13:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:13:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-04-16T16:13:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:13:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:13:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-16T16:13:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42588sec preferred_lft 42588sec 2024-04-16T16:13:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-04-16T16:13:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:13:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:13:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:13:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::74ce:ceff:fec6:2a00/64 scope link 2024-04-16T16:13:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:13:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T16:13:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::3cf7:ffff:feca:f879/64 scope link 2024-04-16T16:13:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:57.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:57.249 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:13:57.336 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-16T16:13:57.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:57.340 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-04-16T16:13:57.391 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-04-16T16:13:57.391 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:13:57.391 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-16T16:13:57.392 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-16T16:13:57.392 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:13:57.392 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:57.393 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:13:57.393 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-16T16:13:57.393 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-16T16:13:57.393 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-16T16:13:57.393 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:13:57.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:57.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:57.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:57.637 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-16T16:13:57.686 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-16T16:13:57.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:57.687 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list-id 2024-04-16T16:13:57.733 INFO:teuthology.orchestra.run.smithi103.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-16T16:13:57.735 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:57.735 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:13:57.735 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-16T16:13:57.735 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-16T16:13:57.735 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:13:57.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:57.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:57.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:57.934 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-16T16:13:58.022 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:13:58.022 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:13:58.022 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:13:58.022 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:58.022 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-16T16:13:58.022 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:58.022 INFO:teuthology.orchestra.run.smithi103.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:13:58.022 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 52:7d:63:be:9c:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-16T16:13:58.022 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-16T16:13:58.022 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:58.023 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::507d:63ff:febe:9c05/64 scope link 2024-04-16T16:13:58.023 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:58.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:58.026 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-16T16:13:58.116 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:13:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:13:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:13:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-16T16:13:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:58.117 INFO:teuthology.orchestra.run.smithi103.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:13:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 52:7d:63:be:9c:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-16T16:13:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-16T16:13:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::507d:63ff:febe:9c05/64 scope link 2024-04-16T16:13:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:13:58.121 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-16T16:13:58.121 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:58.121 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:13:58.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-16T16:13:58.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-16T16:13:58.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-16T16:13:58.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-16T16:13:58.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-16T16:13:58.122 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:13:58.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:58.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:58.450 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:13:58.450 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:13:58.451 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.1 up 2024-04-16T16:13:58.451 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set dev brx.1 master ceph-brx 2024-04-16T16:13:58.451 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:13:58.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:58.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:58.681 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:13:58.681 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-16T16:13:58.724 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-16T16:13:58.725 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:13:58.725 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-16T16:13:58.818 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-04-16T16:13:58.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:58.818 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:13:58.956 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:13:58.956 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:13:58.992 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:13:58.992 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:13:58.992 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:13:58.992 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with new device syntax: 3@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:13:58.992 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:13:58.996 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-04-16T16:13:58.996 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:13:58.996 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:13:58.996 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:13:58.996 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:13:59.001 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:13:59.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:13:59.002 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-16T16:13:59.081 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:13:59.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:13:59.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:59.199 INFO:teuthology.orchestra.run.smithi103.stdout:client4689 2024-04-16T16:13:59.199 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:13:59.199 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:13:59.248 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:13:59.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:13:59.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:59.414 INFO:teuthology.orchestra.run.smithi103.stdout:client4689 2024-04-16T16:13:59.414 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:13:59.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:13:59.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:59.533 INFO:teuthology.orchestra.run.smithi103.stdout:client4689 2024-04-16T16:13:59.533 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/748c4b85-0190-4c59-80ad-098a6a5007c0.client4689/status') 2024-04-16T16:13:59.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:13:59.672 INFO:teuthology.orchestra.run.smithi103.stdout:instance: client.4689 (3)172.21.15.103:0/999797205 2024-04-16T16:13:59.672 INFO:teuthology.orchestra.run.smithi103.stdout:blocklisted: false 2024-04-16T16:13:59.672 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records in 2024-04-16T16:13:59.672 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records out 2024-04-16T16:13:59.672 INFO:teuthology.orchestra.run.smithi103.stderr:70 bytes copied, 6.4101e-05 s, 1.1 MB/s 2024-04-16T16:13:59.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:13:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:13:59.685 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-16T16:13:59.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:13:59.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.933+0000 7f615dc00640 1 Processor -- start 2024-04-16T16:13:59.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.933+0000 7f615dc00640 1 -- start start 2024-04-16T16:13:59.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.935+0000 7f615dc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6158162270 0x7f61580d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:59.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.935+0000 7f615dc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f61580d52d0 0x7f615806ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:59.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.935+0000 7f615dc00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61580d3080 con 0x7f6158161e90 2024-04-16T16:13:59.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.935+0000 7f615dc00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f61580d2e30 con 0x7f61580d52d0 2024-04-16T16:13:59.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.935+0000 7f615dc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f61580d3360 con 0x7f6158162270 2024-04-16T16:13:59.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.935+0000 7f6157e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6158161e90 0x7f61580cf7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49896/0 (socket says 172.21.15.45:49896) 2024-04-16T16:13:59.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.935+0000 7f6157e00640 1 -- 172.21.15.45:0/2763544575 learned_addr learned my addr 172.21.15.45:0/2763544575 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:13:59.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.935+0000 7f615d200640 1 --2- 172.21.15.45:0/2763544575 >> v2:172.21.15.45:3300/0 conn(0x7f61580d52d0 0x7f615806ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:59.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.936+0000 7f6157400640 1 --2- 172.21.15.45:0/2763544575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6158162270 0x7f61580d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:59.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.936+0000 7f615d200640 1 -- 172.21.15.45:0/2763544575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6158162270 msgr2=0x7f61580d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:13:59.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.936+0000 7f615d200640 1 --2- 172.21.15.45:0/2763544575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6158162270 0x7f61580d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:59.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.936+0000 7f615d200640 1 -- 172.21.15.45:0/2763544575 >> v1:172.21.15.45:6789/0 conn(0x7f6158161e90 legacy=0x7f61580cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:59.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.936+0000 7f615d200640 1 -- 172.21.15.45:0/2763544575 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61580d3640 con 0x7f61580d52d0 2024-04-16T16:13:59.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.936+0000 7f6156a00640 1 -- 172.21.15.45:0/2763544575 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1921468569 0 0) 0x7f61580d3080 con 0x7f6158161e90 2024-04-16T16:13:59.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.936+0000 7f6157400640 1 --2- 172.21.15.45:0/2763544575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6158162270 0x7f61580d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:59.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.937+0000 7f615d200640 1 --2- 172.21.15.45:0/2763544575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580d52d0 0x7f615806ce40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6150009d90 tx=0x7f615002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9543cddb57f3389a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:59.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.937+0000 7f6156a00640 1 -- 172.21.15.45:0/2763544575 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150039070 con 0x7f61580d52d0 2024-04-16T16:13:59.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.937+0000 7f6156a00640 1 -- 172.21.15.45:0/2763544575 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f615002cce0 con 0x7f61580d52d0 2024-04-16T16:13:59.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.937+0000 7f6156a00640 1 -- 172.21.15.45:0/2763544575 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150034660 con 0x7f61580d52d0 2024-04-16T16:13:59.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.938+0000 7f615dc00640 1 -- 172.21.15.45:0/2763544575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580d52d0 msgr2=0x7f615806ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:59.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.938+0000 7f615dc00640 1 --2- 172.21.15.45:0/2763544575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580d52d0 0x7f615806ce40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6150009d90 tx=0x7f615002bf30 comp rx=0 tx=0).stop 2024-04-16T16:13:59.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.938+0000 7f615dc00640 1 -- 172.21.15.45:0/2763544575 shutdown_connections 2024-04-16T16:13:59.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.938+0000 7f615dc00640 1 --2- 172.21.15.45:0/2763544575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580d52d0 0x7f615806ce40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:59.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.938+0000 7f615dc00640 1 --2- 172.21.15.45:0/2763544575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6158162270 0x7f61580d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:59.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.938+0000 7f615dc00640 1 -- 172.21.15.45:0/2763544575 >> 172.21.15.45:0/2763544575 conn(0x7f61580d1940 msgr2=0x7f61580d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:13:59.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.938+0000 7f615dc00640 1 -- 172.21.15.45:0/2763544575 shutdown_connections 2024-04-16T16:13:59.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.938+0000 7f615dc00640 1 -- 172.21.15.45:0/2763544575 wait complete. 2024-04-16T16:13:59.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.940+0000 7f615dc00640 1 Processor -- start 2024-04-16T16:13:59.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.940+0000 7f615dc00640 1 -- start start 2024-04-16T16:13:59.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.941+0000 7f615dc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580d52d0 0x7f6158159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:59.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.941+0000 7f615dc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6158161e90 0x7f61580b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:59.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.941+0000 7f615dc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6158162270 0x7f61580ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:59.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.941+0000 7f615dc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f61580d4500 con 0x7f61580d52d0 2024-04-16T16:13:59.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.941+0000 7f615dc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f61580d4380 con 0x7f6158162270 2024-04-16T16:13:59.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.941+0000 7f615dc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f61580d4680 con 0x7f6158161e90 2024-04-16T16:13:59.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.941+0000 7f6157e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580d52d0 0x7f6158159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.941+0000 7f6157e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580d52d0 0x7f6158159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54438/0 (socket says 172.21.15.45:54438) 2024-04-16T16:13:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.941+0000 7f6157e00640 1 -- 172.21.15.45:0/3659256112 learned_addr learned my addr 172.21.15.45:0/3659256112 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:13:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.942+0000 7f6157400640 1 --2- 172.21.15.45:0/3659256112 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6158161e90 0x7f61580b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.942+0000 7f615d200640 1 --2- 172.21.15.45:0/3659256112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6158162270 0x7f61580ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.942+0000 7f6157e00640 1 -- 172.21.15.45:0/3659256112 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6158161e90 msgr2=0x7f61580b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.942+0000 7f6157e00640 1 --2- 172.21.15.45:0/3659256112 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6158161e90 0x7f61580b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.942+0000 7f6157e00640 1 -- 172.21.15.45:0/3659256112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6158162270 msgr2=0x7f61580ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:13:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.942+0000 7f6157e00640 1 --2- 172.21.15.45:0/3659256112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6158162270 0x7f61580ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:13:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.942+0000 7f6157e00640 1 -- 172.21.15.45:0/3659256112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61581f60f0 con 0x7f61580d52d0 2024-04-16T16:13:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.942+0000 7f615d200640 1 --2- 172.21.15.45:0/3659256112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6158162270 0x7f61580ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:13:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.942+0000 7f6157e00640 1 --2- 172.21.15.45:0/3659256112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580d52d0 0x7f6158159740 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f614800c4d0 tx=0x7f614800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.943+0000 7f614fe00640 1 -- 172.21.15.45:0/3659256112 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6148018070 con 0x7f61580d52d0 2024-04-16T16:13:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.943+0000 7f614fe00640 1 -- 172.21.15.45:0/3659256112 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6148007500 con 0x7f61580d52d0 2024-04-16T16:13:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.943+0000 7f614fe00640 1 -- 172.21.15.45:0/3659256112 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6148004a20 con 0x7f61580d52d0 2024-04-16T16:13:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.943+0000 7f615dc00640 1 -- 172.21.15.45:0/3659256112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61581f63d0 con 0x7f61580d52d0 2024-04-16T16:13:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.943+0000 7f615dc00640 1 -- 172.21.15.45:0/3659256112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61581f6990 con 0x7f61580d52d0 2024-04-16T16:13:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.944+0000 7f615dc00640 1 -- 172.21.15.45:0/3659256112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6158075780 con 0x7f61580d52d0 2024-04-16T16:13:59.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.945+0000 7f614fe00640 1 -- 172.21.15.45:0/3659256112 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f614801b070 con 0x7f61580d52d0 2024-04-16T16:13:59.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.946+0000 7f614fe00640 1 --2- 172.21.15.45:0/3659256112 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f613806c8f0 0x7f613806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:13:59.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.946+0000 7f614fe00640 1 -- 172.21.15.45:0/3659256112 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f614808af30 con 0x7f61580d52d0 2024-04-16T16:13:59.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.947+0000 7f6157400640 1 --2- 172.21.15.45:0/3659256112 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f613806c8f0 0x7f613806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:13:59.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.948+0000 7f6157400640 1 --2- 172.21.15.45:0/3659256112 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f613806c8f0 0x7f613806edb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f61580cec70 tx=0x7f6144008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:13:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:13:59.950+0000 7f614fe00640 1 -- 172.21.15.45:0/3659256112 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f61480592c0 con 0x7f61580d52d0 2024-04-16T16:14:00.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.160+0000 7f615dc00640 1 -- 172.21.15.45:0/3659256112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f615806dae0 con 0x7f61580d52d0 2024-04-16T16:14:00.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.162+0000 7f614fe00640 1 -- 172.21.15.45:0/3659256112 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f6148058bc0 con 0x7f61580d52d0 2024-04-16T16:14:00.162 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:00.162 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":21,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:26.918617+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:00.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.165+0000 7f615dc00640 1 -- 172.21.15.45:0/3659256112 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f613806c8f0 msgr2=0x7f613806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:00.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.165+0000 7f615dc00640 1 --2- 172.21.15.45:0/3659256112 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f613806c8f0 0x7f613806edb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f61580cec70 tx=0x7f6144008040 comp rx=0 tx=0).stop 2024-04-16T16:14:00.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.165+0000 7f615dc00640 1 -- 172.21.15.45:0/3659256112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580d52d0 msgr2=0x7f6158159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:00.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.165+0000 7f615dc00640 1 --2- 172.21.15.45:0/3659256112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580d52d0 0x7f6158159740 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f614800c4d0 tx=0x7f614800c990 comp rx=0 tx=0).stop 2024-04-16T16:14:00.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.165+0000 7f615dc00640 1 -- 172.21.15.45:0/3659256112 shutdown_connections 2024-04-16T16:14:00.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.165+0000 7f615dc00640 1 --2- 172.21.15.45:0/3659256112 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f613806c8f0 0x7f613806edb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.165+0000 7f615dc00640 1 --2- 172.21.15.45:0/3659256112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6158162270 0x7f61580ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.166+0000 7f615dc00640 1 --2- 172.21.15.45:0/3659256112 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6158161e90 0x7f61580b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.166+0000 7f615dc00640 1 --2- 172.21.15.45:0/3659256112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580d52d0 0x7f6158159740 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.166+0000 7f615dc00640 1 -- 172.21.15.45:0/3659256112 >> 172.21.15.45:0/3659256112 conn(0x7f61580d1940 msgr2=0x7f61580b9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:00.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.166+0000 7f615dc00640 1 -- 172.21.15.45:0/3659256112 shutdown_connections 2024-04-16T16:14:00.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.166+0000 7f615dc00640 1 -- 172.21.15.45:0/3659256112 wait complete. 2024-04-16T16:14:00.185 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:13:23.600104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:13:24.368141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:14:00.186 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:14:00.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.414+0000 7f7eff200640 1 Processor -- start 2024-04-16T16:14:00.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.414+0000 7f7eff200640 1 -- start start 2024-04-16T16:14:00.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.416+0000 7f7eff200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f00145fe0 0x7f7f000c5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:00.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.416+0000 7f7eff200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7f000cfb50 0x7f7f0006dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:00.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.416+0000 7f7eff200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f000c6210 con 0x7f7f00160e90 2024-04-16T16:14:00.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.416+0000 7f7eff200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f7f001603b0 con 0x7f7f000cfb50 2024-04-16T16:14:00.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.416+0000 7f7eff200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7f00160690 con 0x7f7f00145fe0 2024-04-16T16:14:00.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.416+0000 7f7efe800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7f000cfb50 0x7f7f0006dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:00.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.416+0000 7f7ef5400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7f00160e90 0x7f7f000cf440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49910/0 (socket says 172.21.15.45:49910) 2024-04-16T16:14:00.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.416+0000 7f7ef5400640 1 -- 172.21.15.45:0/3764062240 learned_addr learned my addr 172.21.15.45:0/3764062240 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:00.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.417+0000 7f7efde00640 1 --2- 172.21.15.45:0/3764062240 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f00145fe0 0x7f7f000c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:00.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.417+0000 7f7efe800640 1 -- 172.21.15.45:0/3764062240 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f00145fe0 msgr2=0x7f7f000c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:00.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.417+0000 7f7efe800640 1 --2- 172.21.15.45:0/3764062240 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f00145fe0 0x7f7f000c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.417+0000 7f7efe800640 1 -- 172.21.15.45:0/3764062240 >> v1:172.21.15.45:6789/0 conn(0x7f7f00160e90 legacy=0x7f7f000cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:00.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.417+0000 7f7efe800640 1 -- 172.21.15.45:0/3764062240 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f00160970 con 0x7f7f000cfb50 2024-04-16T16:14:00.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.418+0000 7f7efe800640 1 --2- 172.21.15.45:0/3764062240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f000cfb50 0x7f7f0006dfb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80097a0 tx=0x7f7ef802b660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22341d250dfb0214 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:00.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.418+0000 7f7efd400640 1 -- 172.21.15.45:0/3764062240 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8037070 con 0x7f7f000cfb50 2024-04-16T16:14:00.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.418+0000 7f7efd400640 1 -- 172.21.15.45:0/3764062240 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ef8030710 con 0x7f7f000cfb50 2024-04-16T16:14:00.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.418+0000 7f7efd400640 1 -- 172.21.15.45:0/3764062240 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef802be60 con 0x7f7f000cfb50 2024-04-16T16:14:00.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.419+0000 7f7eff200640 1 -- 172.21.15.45:0/3764062240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f000cfb50 msgr2=0x7f7f0006dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:00.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.419+0000 7f7eff200640 1 --2- 172.21.15.45:0/3764062240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f000cfb50 0x7f7f0006dfb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80097a0 tx=0x7f7ef802b660 comp rx=0 tx=0).stop 2024-04-16T16:14:00.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.419+0000 7f7eff200640 1 -- 172.21.15.45:0/3764062240 shutdown_connections 2024-04-16T16:14:00.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.419+0000 7f7eff200640 1 --2- 172.21.15.45:0/3764062240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f000cfb50 0x7f7f0006dfb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.419+0000 7f7eff200640 1 --2- 172.21.15.45:0/3764062240 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f00145fe0 0x7f7f000c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.419+0000 7f7eff200640 1 -- 172.21.15.45:0/3764062240 >> 172.21.15.45:0/3764062240 conn(0x7f7f00067e00 msgr2=0x7f7f0006b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:00.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.419+0000 7f7eff200640 1 -- 172.21.15.45:0/3764062240 shutdown_connections 2024-04-16T16:14:00.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.419+0000 7f7eff200640 1 -- 172.21.15.45:0/3764062240 wait complete. 2024-04-16T16:14:00.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.421+0000 7f7eff200640 1 Processor -- start 2024-04-16T16:14:00.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.421+0000 7f7eff200640 1 -- start start 2024-04-16T16:14:00.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.422+0000 7f7eff200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f000cfb50 0x7f7f0013bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:00.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.422+0000 7f7eff200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f00145fe0 0x7f7f00141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:00.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.422+0000 7f7eff200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f00160e90 0x7f7f0015b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:00.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.422+0000 7f7eff200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7f00146540 con 0x7f7f000cfb50 2024-04-16T16:14:00.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.422+0000 7f7eff200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001463c0 con 0x7f7f00160e90 2024-04-16T16:14:00.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.422+0000 7f7eff200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7f000c5c90 con 0x7f7f00145fe0 2024-04-16T16:14:00.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.423+0000 7f7efde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f000cfb50 0x7f7f0013bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:00.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.423+0000 7f7efde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f000cfb50 0x7f7f0013bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54456/0 (socket says 172.21.15.45:54456) 2024-04-16T16:14:00.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.423+0000 7f7efde00640 1 -- 172.21.15.45:0/1133169856 learned_addr learned my addr 172.21.15.45:0/1133169856 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:00.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.423+0000 7f7ef5400640 1 --2- 172.21.15.45:0/1133169856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f00145fe0 0x7f7f00141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:00.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.423+0000 7f7efe800640 1 --2- 172.21.15.45:0/1133169856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f00160e90 0x7f7f0015b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:00.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.423+0000 7f7efde00640 1 -- 172.21.15.45:0/1133169856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f00145fe0 msgr2=0x7f7f00141ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:00.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.423+0000 7f7efde00640 1 --2- 172.21.15.45:0/1133169856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f00145fe0 0x7f7f00141ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.423+0000 7f7efde00640 1 -- 172.21.15.45:0/1133169856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f00160e90 msgr2=0x7f7f0015b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:00.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.423+0000 7f7efde00640 1 --2- 172.21.15.45:0/1133169856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f00160e90 0x7f7f0015b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.423+0000 7f7efde00640 1 -- 172.21.15.45:0/1133169856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f001f4a70 con 0x7f7f000cfb50 2024-04-16T16:14:00.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.423+0000 7f7efde00640 1 --2- 172.21.15.45:0/1133169856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f000cfb50 0x7f7f0013bd30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0007ef0 tx=0x7f7ef000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:00.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.424+0000 7f7ef6a00640 1 -- 172.21.15.45:0/1133169856 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0014620 con 0x7f7f000cfb50 2024-04-16T16:14:00.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.424+0000 7f7ef6a00640 1 -- 172.21.15.45:0/1133169856 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ef0019070 con 0x7f7f000cfb50 2024-04-16T16:14:00.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.424+0000 7f7ef6a00640 1 -- 172.21.15.45:0/1133169856 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0014a00 con 0x7f7f000cfb50 2024-04-16T16:14:00.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.424+0000 7f7eff200640 1 -- 172.21.15.45:0/1133169856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f001f4d50 con 0x7f7f000cfb50 2024-04-16T16:14:00.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.424+0000 7f7eff200640 1 -- 172.21.15.45:0/1133169856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f001f5310 con 0x7f7f000cfb50 2024-04-16T16:14:00.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.425+0000 7f7eff200640 1 -- 172.21.15.45:0/1133169856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f00075780 con 0x7f7f000cfb50 2024-04-16T16:14:00.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.426+0000 7f7ef6a00640 1 -- 172.21.15.45:0/1133169856 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7ef0014ba0 con 0x7f7f000cfb50 2024-04-16T16:14:00.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.427+0000 7f7ef6a00640 1 --2- 172.21.15.45:0/1133169856 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f7edc06c920 0x7f7edc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:00.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.427+0000 7f7ef6a00640 1 -- 172.21.15.45:0/1133169856 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7ef008c3d0 con 0x7f7f000cfb50 2024-04-16T16:14:00.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.427+0000 7f7ef5400640 1 --2- 172.21.15.45:0/1133169856 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f7edc06c920 0x7f7edc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:00.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.429+0000 7f7ef5400640 1 --2- 172.21.15.45:0/1133169856 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f7edc06c920 0x7f7edc06ede0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7f000ce2b0 tx=0x7f7ee8007e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:00.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.432+0000 7f7ef6a00640 1 -- 172.21.15.45:0/1133169856 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7ef005a800 con 0x7f7f000cfb50 2024-04-16T16:14:00.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.636+0000 7f7eff200640 1 -- 172.21.15.45:0/1133169856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7f0015d0e0 con 0x7f7f000cfb50 2024-04-16T16:14:00.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.637+0000 7f7ef6a00640 1 -- 172.21.15.45:0/1133169856 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f7ef005a100 con 0x7f7f000cfb50 2024-04-16T16:14:00.636 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:14:00.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.639+0000 7f7eff200640 1 -- 172.21.15.45:0/1133169856 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f7edc06c920 msgr2=0x7f7edc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.639+0000 7f7eff200640 1 --2- 172.21.15.45:0/1133169856 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f7edc06c920 0x7f7edc06ede0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7f000ce2b0 tx=0x7f7ee8007e20 comp rx=0 tx=0).stop 2024-04-16T16:14:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.639+0000 7f7eff200640 1 -- 172.21.15.45:0/1133169856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f000cfb50 msgr2=0x7f7f0013bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.639+0000 7f7eff200640 1 --2- 172.21.15.45:0/1133169856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f000cfb50 0x7f7f0013bd30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0007ef0 tx=0x7f7ef000e4b0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.640+0000 7f7eff200640 1 -- 172.21.15.45:0/1133169856 shutdown_connections 2024-04-16T16:14:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.640+0000 7f7eff200640 1 --2- 172.21.15.45:0/1133169856 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f7edc06c920 0x7f7edc06ede0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.640+0000 7f7eff200640 1 --2- 172.21.15.45:0/1133169856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f00160e90 0x7f7f0015b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.640+0000 7f7eff200640 1 --2- 172.21.15.45:0/1133169856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f00145fe0 0x7f7f00141ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.640+0000 7f7eff200640 1 --2- 172.21.15.45:0/1133169856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f000cfb50 0x7f7f0013bd30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.640+0000 7f7eff200640 1 -- 172.21.15.45:0/1133169856 >> 172.21.15.45:0/1133169856 conn(0x7f7f00067e00 msgr2=0x7f7f0015e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.640+0000 7f7eff200640 1 -- 172.21.15.45:0/1133169856 shutdown_connections 2024-04-16T16:14:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.640+0000 7f7eff200640 1 -- 172.21.15.45:0/1133169856 wait complete. 2024-04-16T16:14:00.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:14:00.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.885+0000 7f3dbb200640 1 Processor -- start 2024-04-16T16:14:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.885+0000 7f3dbb200640 1 -- start start 2024-04-16T16:14:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.886+0000 7f3dbb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3db415b6b0 0x7f3db40b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.886+0000 7f3dbb200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3db40b2110 0x7f3db40b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.886+0000 7f3dbb200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3db40af720 con 0x7f3db415ba90 2024-04-16T16:14:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.886+0000 7f3dbb200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3db40b0010 con 0x7f3db40b2110 2024-04-16T16:14:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.887+0000 7f3dbb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3db406c950 con 0x7f3db415b6b0 2024-04-16T16:14:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.887+0000 7f3db3400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3db415ba90 0x7f3db415be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49914/0 (socket says 172.21.15.45:49914) 2024-04-16T16:14:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.887+0000 7f3db3400640 1 -- 172.21.15.45:0/568984702 learned_addr learned my addr 172.21.15.45:0/568984702 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.887+0000 7f3db3e00640 1 --2- 172.21.15.45:0/568984702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3db415b6b0 0x7f3db40b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.887+0000 7f3db9200640 1 --2- 172.21.15.45:0/568984702 >> v2:172.21.15.103:3300/0 conn(0x7f3db40b2110 0x7f3db40b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.887+0000 7f3db2a00640 1 -- 172.21.15.45:0/568984702 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 141598146 0 0) 0x7f3db40af720 con 0x7f3db415ba90 2024-04-16T16:14:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.887+0000 7f3db2a00640 1 -- 172.21.15.45:0/568984702 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d94003520 con 0x7f3db415ba90 2024-04-16T16:14:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.888+0000 7f3db2a00640 1 -- 172.21.15.45:0/568984702 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2909074520 0 0) 0x7f3d94003520 con 0x7f3db415ba90 2024-04-16T16:14:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.888+0000 7f3db2a00640 1 -- 172.21.15.45:0/568984702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3db415b6b0 msgr2=0x7f3db40b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.888+0000 7f3db2a00640 1 --2- 172.21.15.45:0/568984702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3db415b6b0 0x7f3db40b04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.888+0000 7f3db2a00640 1 -- 172.21.15.45:0/568984702 >> v2:172.21.15.103:3300/0 conn(0x7f3db40b2110 msgr2=0x7f3db40b25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.888+0000 7f3db2a00640 1 --2- 172.21.15.45:0/568984702 >> v2:172.21.15.103:3300/0 conn(0x7f3db40b2110 0x7f3db40b25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.888+0000 7f3db2a00640 1 -- 172.21.15.45:0/568984702 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db40af9c0 con 0x7f3db415ba90 2024-04-16T16:14:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.888+0000 7f3db2a00640 1 -- 172.21.15.45:0/568984702 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f3d9c0019f0 con 0x7f3db415ba90 2024-04-16T16:14:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.888+0000 7f3db2a00640 1 -- 172.21.15.45:0/568984702 >> v1:172.21.15.45:6789/0 conn(0x7f3db415ba90 legacy=0x7f3db415be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.888+0000 7f3db2a00640 1 --2- 172.21.15.45:0/568984702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3db415b6b0 0x7f3d940039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.888+0000 7f3db2a00640 1 --2- 172.21.15.45:0/568984702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d94003fa0 0x7f3d94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.888+0000 7f3db2a00640 1 --2- 172.21.15.45:0/568984702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d94004d40 0x7f3d940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.888+0000 7f3db2a00640 1 -- 172.21.15.45:0/568984702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3db40b0010 con 0x7f3d94004d40 2024-04-16T16:14:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.888+0000 7f3db2a00640 1 -- 172.21.15.45:0/568984702 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3db406c950 con 0x7f3d94003fa0 2024-04-16T16:14:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.888+0000 7f3db2a00640 1 -- 172.21.15.45:0/568984702 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3d9400b9f0 con 0x7f3db415b6b0 2024-04-16T16:14:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.888+0000 7f3db9200640 1 --2- 172.21.15.45:0/568984702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d94004d40 0x7f3d940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.889+0000 7f3db3e00640 1 --2- 172.21.15.45:0/568984702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3db415b6b0 0x7f3d940039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.889+0000 7f3db3400640 1 --2- 172.21.15.45:0/568984702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d94003fa0 0x7f3d94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.889+0000 7f3db9200640 1 -- 172.21.15.45:0/568984702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3db415b6b0 msgr2=0x7f3d940039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.889+0000 7f3db9200640 1 --2- 172.21.15.45:0/568984702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3db415b6b0 0x7f3d940039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.889+0000 7f3db9200640 1 -- 172.21.15.45:0/568984702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d94003fa0 msgr2=0x7f3d94004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.889+0000 7f3db9200640 1 --2- 172.21.15.45:0/568984702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d94003fa0 0x7f3d94004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.889+0000 7f3db9200640 1 -- 172.21.15.45:0/568984702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d94003520 con 0x7f3d94004d40 2024-04-16T16:14:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.889+0000 7f3db3400640 1 --2- 172.21.15.45:0/568984702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d94003fa0 0x7f3d94004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.890+0000 7f3db9200640 1 --2- 172.21.15.45:0/568984702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d94004d40 0x7f3d940051c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009ac0 tx=0x7f3d94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5f526890bba636 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.890+0000 7f3db2a00640 1 -- 172.21.15.45:0/568984702 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac039070 con 0x7f3d94004d40 2024-04-16T16:14:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.890+0000 7f3db2a00640 1 -- 172.21.15.45:0/568984702 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dac02c710 con 0x7f3d94004d40 2024-04-16T16:14:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.890+0000 7f3db2a00640 1 -- 172.21.15.45:0/568984702 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac009690 con 0x7f3d94004d40 2024-04-16T16:14:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.891+0000 7f3dbb200640 1 -- 172.21.15.45:0/568984702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d94004d40 msgr2=0x7f3d940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.891+0000 7f3dbb200640 1 --2- 172.21.15.45:0/568984702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d94004d40 0x7f3d940051c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009ac0 tx=0x7f3d94005830 comp rx=0 tx=0).stop 2024-04-16T16:14:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.891+0000 7f3db9200640 1 -- 172.21.15.45:0/568984702 reap_dead start 2024-04-16T16:14:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.891+0000 7f3dbb200640 1 -- 172.21.15.45:0/568984702 shutdown_connections 2024-04-16T16:14:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.891+0000 7f3dbb200640 1 -- 172.21.15.45:0/568984702 >> 172.21.15.45:0/568984702 conn(0x7f3db409aa70 msgr2=0x7f3db40b6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.891+0000 7f3dbb200640 1 -- 172.21.15.45:0/568984702 shutdown_connections 2024-04-16T16:14:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.891+0000 7f3dbb200640 1 -- 172.21.15.45:0/568984702 wait complete. 2024-04-16T16:14:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.893+0000 7f3dbb200640 1 Processor -- start 2024-04-16T16:14:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.893+0000 7f3dbb200640 1 -- start start 2024-04-16T16:14:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.894+0000 7f3dbb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d94004d40 0x7f3db414b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.894+0000 7f3dbb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3db414bfd0 0x7f3db406f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.894+0000 7f3dbb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3db406f680 0x7f3db41f7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.894+0000 7f3dbb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3db40b04a0 con 0x7f3db414bfd0 2024-04-16T16:14:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.894+0000 7f3dbb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3db40b2ae0 con 0x7f3d94004d40 2024-04-16T16:14:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.894+0000 7f3dbb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3db40b0620 con 0x7f3db406f680 2024-04-16T16:14:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.894+0000 7f3db3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3db414bfd0 0x7f3db406f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.895+0000 7f3db3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3db414bfd0 0x7f3db406f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54472/0 (socket says 172.21.15.45:54472) 2024-04-16T16:14:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.895+0000 7f3db3400640 1 -- 172.21.15.45:0/693958579 learned_addr learned my addr 172.21.15.45:0/693958579 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.895+0000 7f3db3e00640 1 --2- 172.21.15.45:0/693958579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d94004d40 0x7f3db414b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.895+0000 7f3db9200640 1 --2- 172.21.15.45:0/693958579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3db406f680 0x7f3db41f7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.895+0000 7f3db3400640 1 -- 172.21.15.45:0/693958579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3db406f680 msgr2=0x7f3db41f7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.895+0000 7f3db3400640 1 --2- 172.21.15.45:0/693958579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3db406f680 0x7f3db41f7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.895+0000 7f3db3400640 1 -- 172.21.15.45:0/693958579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d94004d40 msgr2=0x7f3db414b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.895+0000 7f3db3400640 1 --2- 172.21.15.45:0/693958579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d94004d40 0x7f3db414b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.895+0000 7f3db3400640 1 -- 172.21.15.45:0/693958579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db4152e90 con 0x7f3db414bfd0 2024-04-16T16:14:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.895+0000 7f3db9200640 1 --2- 172.21.15.45:0/693958579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3db406f680 0x7f3db41f7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.895+0000 7f3db3400640 1 --2- 172.21.15.45:0/693958579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3db414bfd0 0x7f3db406f140 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c009ae0 tx=0x7f3d9c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.896+0000 7f3dabe00640 1 -- 172.21.15.45:0/693958579 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c018070 con 0x7f3db414bfd0 2024-04-16T16:14:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.896+0000 7f3dabe00640 1 -- 172.21.15.45:0/693958579 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d9c009c90 con 0x7f3db414bfd0 2024-04-16T16:14:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.896+0000 7f3dabe00640 1 -- 172.21.15.45:0/693958579 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c00dd50 con 0x7f3db414bfd0 2024-04-16T16:14:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.896+0000 7f3dbb200640 1 -- 172.21.15.45:0/693958579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db4154860 con 0x7f3db414bfd0 2024-04-16T16:14:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.896+0000 7f3dbb200640 1 -- 172.21.15.45:0/693958579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db4154e20 con 0x7f3db414bfd0 2024-04-16T16:14:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.897+0000 7f3dbb200640 1 -- 172.21.15.45:0/693958579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3db406fa90 con 0x7f3db414bfd0 2024-04-16T16:14:00.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.898+0000 7f3dabe00640 1 -- 172.21.15.45:0/693958579 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3d9c01b070 con 0x7f3db414bfd0 2024-04-16T16:14:00.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.899+0000 7f3dabe00640 1 --2- 172.21.15.45:0/693958579 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3d9006c920 0x7f3d9006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:00.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.899+0000 7f3dabe00640 1 -- 172.21.15.45:0/693958579 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3d9c08b550 con 0x7f3db414bfd0 2024-04-16T16:14:00.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.900+0000 7f3db3e00640 1 --2- 172.21.15.45:0/693958579 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3d9006c920 0x7f3d9006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:00.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.901+0000 7f3db3e00640 1 --2- 172.21.15.45:0/693958579 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3d9006c920 0x7f3d9006ede0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3db41f7f40 tx=0x7f3da4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:00.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:00.904+0000 7f3dabe00640 1 -- 172.21.15.45:0/693958579 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3db406fa90 con 0x7f3db414bfd0 2024-04-16T16:14:01.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.108+0000 7f3dbb200640 1 -- 172.21.15.45:0/693958579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3db4075780 con 0x7f3db414bfd0 2024-04-16T16:14:01.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.108+0000 7f3dabe00640 1 -- 172.21.15.45:0/693958579 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f3d9c010070 con 0x7f3db414bfd0 2024-04-16T16:14:01.108 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:14:01.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.111+0000 7f3dbb200640 1 -- 172.21.15.45:0/693958579 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3d9006c920 msgr2=0x7f3d9006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:01.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.111+0000 7f3dbb200640 1 --2- 172.21.15.45:0/693958579 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3d9006c920 0x7f3d9006ede0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3db41f7f40 tx=0x7f3da4008040 comp rx=0 tx=0).stop 2024-04-16T16:14:01.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.111+0000 7f3dbb200640 1 -- 172.21.15.45:0/693958579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3db414bfd0 msgr2=0x7f3db406f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:01.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.111+0000 7f3dbb200640 1 --2- 172.21.15.45:0/693958579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3db414bfd0 0x7f3db406f140 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c009ae0 tx=0x7f3d9c00d4a0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.111+0000 7f3dbb200640 1 -- 172.21.15.45:0/693958579 shutdown_connections 2024-04-16T16:14:01.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.111+0000 7f3dbb200640 1 --2- 172.21.15.45:0/693958579 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3d9006c920 0x7f3d9006ede0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.111+0000 7f3dbb200640 1 --2- 172.21.15.45:0/693958579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3db406f680 0x7f3db41f7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.111+0000 7f3dbb200640 1 --2- 172.21.15.45:0/693958579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3db414bfd0 0x7f3db406f140 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.111+0000 7f3dbb200640 1 --2- 172.21.15.45:0/693958579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d94004d40 0x7f3db414b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.111+0000 7f3dbb200640 1 -- 172.21.15.45:0/693958579 >> 172.21.15.45:0/693958579 conn(0x7f3db409aa70 msgr2=0x7f3db4070560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:01.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.112+0000 7f3dbb200640 1 -- 172.21.15.45:0/693958579 shutdown_connections 2024-04-16T16:14:01.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.112+0000 7f3dbb200640 1 -- 172.21.15.45:0/693958579 wait complete. 2024-04-16T16:14:01.130 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:14:01.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.359+0000 7f4d2a200640 1 Processor -- start 2024-04-16T16:14:01.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.360+0000 7f4d2a200640 1 -- start start 2024-04-16T16:14:01.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.361+0000 7f4d2a200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4d241461b0 0x7f4d24160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.361+0000 7f4d2a200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d240d0260 con 0x7f4d240b5d50 2024-04-16T16:14:01.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.361+0000 7f4d2a200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d24146590 con 0x7f4d240b58f0 2024-04-16T16:14:01.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.361+0000 7f4d2a200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4d2406dfd0 con 0x7f4d241461b0 2024-04-16T16:14:01.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.362+0000 7f4d29800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4d240b5d50 0x7f4d240cfb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49928/0 (socket says 172.21.15.45:49928) 2024-04-16T16:14:01.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.362+0000 7f4d29800640 1 -- 172.21.15.45:0/721800870 learned_addr learned my addr 172.21.15.45:0/721800870 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:01.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.362+0000 7f4d23400640 1 -- 172.21.15.45:0/721800870 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1991332635 0 0) 0x7f4d240d0260 con 0x7f4d240b5d50 2024-04-16T16:14:01.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.362+0000 7f4d23400640 1 -- 172.21.15.45:0/721800870 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d0c003520 con 0x7f4d240b5d50 2024-04-16T16:14:01.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.362+0000 7f4d28e00640 1 --2- 172.21.15.45:0/721800870 >> v2:172.21.15.103:3300/0 conn(0x7f4d241461b0 0x7f4d24160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:01.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.363+0000 7f4d23400640 1 -- 172.21.15.45:0/721800870 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 266957029 0 0) 0x7f4d0c003520 con 0x7f4d240b5d50 2024-04-16T16:14:01.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.363+0000 7f4d23400640 1 -- 172.21.15.45:0/721800870 >> v2:172.21.15.103:3300/0 conn(0x7f4d241461b0 msgr2=0x7f4d24160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:01.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.363+0000 7f4d23400640 1 --2- 172.21.15.45:0/721800870 >> v2:172.21.15.103:3300/0 conn(0x7f4d241461b0 0x7f4d24160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.363+0000 7f4d23400640 1 -- 172.21.15.45:0/721800870 >> v1:172.21.15.103:6789/0 conn(0x7f4d240b58f0 legacy=0x7f4d240cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:01.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.363+0000 7f4d23400640 1 -- 172.21.15.45:0/721800870 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d240c5750 con 0x7f4d240b5d50 2024-04-16T16:14:01.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.363+0000 7f4d23400640 1 -- 172.21.15.45:0/721800870 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f4d1c0019f0 con 0x7f4d240b5d50 2024-04-16T16:14:01.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.363+0000 7f4d28e00640 1 --2- 172.21.15.45:0/721800870 >> v2:172.21.15.103:3300/0 conn(0x7f4d241461b0 0x7f4d24160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:01.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.363+0000 7f4d23400640 1 -- 172.21.15.45:0/721800870 >> v1:172.21.15.45:6789/0 conn(0x7f4d240b5d50 legacy=0x7f4d240cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:01.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.363+0000 7f4d23400640 1 --2- 172.21.15.45:0/721800870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d0c003a00 0x7f4d0c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.363+0000 7f4d23400640 1 --2- 172.21.15.45:0/721800870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d0c004800 0x7f4d0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.363+0000 7f4d23400640 1 --2- 172.21.15.45:0/721800870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c0052e0 0x7f4d0c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.363+0000 7f4d23400640 1 -- 172.21.15.45:0/721800870 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0c00c200 con 0x7f4d0c0052e0 2024-04-16T16:14:01.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.363+0000 7f4d23400640 1 -- 172.21.15.45:0/721800870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0c00c3b0 con 0x7f4d0c004800 2024-04-16T16:14:01.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.363+0000 7f4d23400640 1 -- 172.21.15.45:0/721800870 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4d0c00c5b0 con 0x7f4d0c003a00 2024-04-16T16:14:01.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.363+0000 7f4d29800640 1 --2- 172.21.15.45:0/721800870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c0052e0 0x7f4d0c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:01.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.364+0000 7f4d28e00640 1 --2- 172.21.15.45:0/721800870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d0c003a00 0x7f4d0c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:01.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.364+0000 7f4d23e00640 1 --2- 172.21.15.45:0/721800870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d0c004800 0x7f4d0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:01.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.364+0000 7f4d29800640 1 -- 172.21.15.45:0/721800870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d0c003a00 msgr2=0x7f4d0c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:01.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.364+0000 7f4d29800640 1 --2- 172.21.15.45:0/721800870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d0c003a00 0x7f4d0c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.364+0000 7f4d29800640 1 -- 172.21.15.45:0/721800870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d0c004800 msgr2=0x7f4d0c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:01.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.364+0000 7f4d29800640 1 --2- 172.21.15.45:0/721800870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d0c004800 0x7f4d0c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.364+0000 7f4d29800640 1 -- 172.21.15.45:0/721800870 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d0c003520 con 0x7f4d0c0052e0 2024-04-16T16:14:01.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.365+0000 7f4d29800640 1 -- 172.21.15.45:0/721800870 reap_dead start 2024-04-16T16:14:01.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.365+0000 7f4d29800640 1 --2- 172.21.15.45:0/721800870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c0052e0 0x7f4d0c00b9c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c002230 tx=0x7f4d0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22770052dbbb460 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:01.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.365+0000 7f4d23400640 1 -- 172.21.15.45:0/721800870 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c00a3f0 con 0x7f4d0c0052e0 2024-04-16T16:14:01.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.365+0000 7f4d23400640 1 -- 172.21.15.45:0/721800870 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d1c034650 con 0x7f4d0c0052e0 2024-04-16T16:14:01.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.365+0000 7f4d23400640 1 -- 172.21.15.45:0/721800870 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c034a40 con 0x7f4d0c0052e0 2024-04-16T16:14:01.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.366+0000 7f4d2a200640 1 -- 172.21.15.45:0/721800870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c0052e0 msgr2=0x7f4d0c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:01.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.366+0000 7f4d2a200640 1 --2- 172.21.15.45:0/721800870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c0052e0 0x7f4d0c00b9c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c002230 tx=0x7f4d0c005830 comp rx=0 tx=0).stop 2024-04-16T16:14:01.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.366+0000 7f4d2a200640 1 -- 172.21.15.45:0/721800870 shutdown_connections 2024-04-16T16:14:01.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.366+0000 7f4d2a200640 1 --2- 172.21.15.45:0/721800870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c0052e0 0x7f4d0c00b9c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.366+0000 7f4d2a200640 1 -- 172.21.15.45:0/721800870 >> 172.21.15.45:0/721800870 conn(0x7f4d2406b930 msgr2=0x7f4d2406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:01.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.366+0000 7f4d2a200640 1 -- 172.21.15.45:0/721800870 shutdown_connections 2024-04-16T16:14:01.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.366+0000 7f4d2a200640 1 -- 172.21.15.45:0/721800870 wait complete. 2024-04-16T16:14:01.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.368+0000 7f4d2a200640 1 Processor -- start 2024-04-16T16:14:01.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.368+0000 7f4d2a200640 1 -- start start 2024-04-16T16:14:01.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.369+0000 7f4d2a200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c0052e0 0x7f4d240cec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.369+0000 7f4d2a200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24142510 0x7f4d24142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.369+0000 7f4d2a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d240cbd00 0x7f4d240cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.369+0000 7f4d2a200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4d240c5cd0 con 0x7f4d0c0052e0 2024-04-16T16:14:01.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.369+0000 7f4d2a200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4d240c59f0 con 0x7f4d24142510 2024-04-16T16:14:01.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.369+0000 7f4d2a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4d240c5fb0 con 0x7f4d240cbd00 2024-04-16T16:14:01.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.369+0000 7f4d28e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c0052e0 0x7f4d240cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:01.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.369+0000 7f4d28e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c0052e0 0x7f4d240cec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54492/0 (socket says 172.21.15.45:54492) 2024-04-16T16:14:01.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.369+0000 7f4d28e00640 1 -- 172.21.15.45:0/2692531255 learned_addr learned my addr 172.21.15.45:0/2692531255 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:01.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.370+0000 7f4d23e00640 1 --2- 172.21.15.45:0/2692531255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24142510 0x7f4d24142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:01.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.370+0000 7f4d29800640 1 --2- 172.21.15.45:0/2692531255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d240cbd00 0x7f4d240cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:01.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.370+0000 7f4d28e00640 1 -- 172.21.15.45:0/2692531255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d240cbd00 msgr2=0x7f4d240cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:01.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.370+0000 7f4d28e00640 1 --2- 172.21.15.45:0/2692531255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d240cbd00 0x7f4d240cc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.370+0000 7f4d28e00640 1 -- 172.21.15.45:0/2692531255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24142510 msgr2=0x7f4d24142a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:01.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.370+0000 7f4d28e00640 1 --2- 172.21.15.45:0/2692531255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24142510 0x7f4d24142a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.370+0000 7f4d28e00640 1 -- 172.21.15.45:0/2692531255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d241f53c0 con 0x7f4d0c0052e0 2024-04-16T16:14:01.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.370+0000 7f4d23e00640 1 --2- 172.21.15.45:0/2692531255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24142510 0x7f4d24142a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:01.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.370+0000 7f4d28e00640 1 --2- 172.21.15.45:0/2692531255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c0052e0 0x7f4d240cec90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800e840 tx=0x7f4d1800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:01.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.371+0000 7f4d20c00640 1 -- 172.21.15.45:0/2692531255 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18019020 con 0x7f4d0c0052e0 2024-04-16T16:14:01.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.371+0000 7f4d20c00640 1 -- 172.21.15.45:0/2692531255 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d1800ce20 con 0x7f4d0c0052e0 2024-04-16T16:14:01.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.371+0000 7f4d20c00640 1 -- 172.21.15.45:0/2692531255 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1800ab40 con 0x7f4d0c0052e0 2024-04-16T16:14:01.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.371+0000 7f4d2a200640 1 -- 172.21.15.45:0/2692531255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d241f4600 con 0x7f4d0c0052e0 2024-04-16T16:14:01.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.371+0000 7f4d2a200640 1 -- 172.21.15.45:0/2692531255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d241f4bc0 con 0x7f4d0c0052e0 2024-04-16T16:14:01.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.372+0000 7f4d2a200640 1 -- 172.21.15.45:0/2692531255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d240b5690 con 0x7f4d0c0052e0 2024-04-16T16:14:01.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.373+0000 7f4d20c00640 1 -- 172.21.15.45:0/2692531255 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4d1801c020 con 0x7f4d0c0052e0 2024-04-16T16:14:01.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.375+0000 7f4d20c00640 1 --2- 172.21.15.45:0/2692531255 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f4d0406c8f0 0x7f4d0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.375+0000 7f4d20c00640 1 -- 172.21.15.45:0/2692531255 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4d1808c5a0 con 0x7f4d0c0052e0 2024-04-16T16:14:01.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.376+0000 7f4d23e00640 1 --2- 172.21.15.45:0/2692531255 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f4d0406c8f0 0x7f4d0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:01.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.377+0000 7f4d23e00640 1 --2- 172.21.15.45:0/2692531255 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f4d0406c8f0 0x7f4d0406edb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4d24142910 tx=0x7f4d10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:01.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.380+0000 7f4d20c00640 1 -- 172.21.15.45:0/2692531255 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d240b5690 con 0x7f4d0c0052e0 2024-04-16T16:14:01.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.585+0000 7f4d2a200640 1 -- 172.21.15.45:0/2692531255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4d24075780 con 0x7f4d0c0052e0 2024-04-16T16:14:01.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.586+0000 7f4d20c00640 1 -- 172.21.15.45:0/2692531255 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f4d18023020 con 0x7f4d0c0052e0 2024-04-16T16:14:01.585 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:14:01.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.589+0000 7f4d2a200640 1 -- 172.21.15.45:0/2692531255 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f4d0406c8f0 msgr2=0x7f4d0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:01.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.589+0000 7f4d2a200640 1 --2- 172.21.15.45:0/2692531255 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f4d0406c8f0 0x7f4d0406edb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4d24142910 tx=0x7f4d10009040 comp rx=0 tx=0).stop 2024-04-16T16:14:01.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.589+0000 7f4d2a200640 1 -- 172.21.15.45:0/2692531255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c0052e0 msgr2=0x7f4d240cec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:01.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.589+0000 7f4d2a200640 1 --2- 172.21.15.45:0/2692531255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c0052e0 0x7f4d240cec90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800e840 tx=0x7f4d1800ed00 comp rx=0 tx=0).stop 2024-04-16T16:14:01.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.589+0000 7f4d2a200640 1 -- 172.21.15.45:0/2692531255 shutdown_connections 2024-04-16T16:14:01.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.589+0000 7f4d2a200640 1 --2- 172.21.15.45:0/2692531255 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f4d0406c8f0 0x7f4d0406edb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.589+0000 7f4d2a200640 1 --2- 172.21.15.45:0/2692531255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d240cbd00 0x7f4d240cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.589+0000 7f4d2a200640 1 --2- 172.21.15.45:0/2692531255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24142510 0x7f4d24142a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.589+0000 7f4d2a200640 1 --2- 172.21.15.45:0/2692531255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c0052e0 0x7f4d240cec90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.589+0000 7f4d2a200640 1 -- 172.21.15.45:0/2692531255 >> 172.21.15.45:0/2692531255 conn(0x7f4d2406b930 msgr2=0x7f4d24072c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:01.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.589+0000 7f4d2a200640 1 -- 172.21.15.45:0/2692531255 shutdown_connections 2024-04-16T16:14:01.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.590+0000 7f4d2a200640 1 -- 172.21.15.45:0/2692531255 wait complete. 2024-04-16T16:14:01.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:01.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.829+0000 7f196d200640 1 Processor -- start 2024-04-16T16:14:01.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.830+0000 7f196d200640 1 -- start start 2024-04-16T16:14:01.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.832+0000 7f196d200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f19680ce9a0 0x7f19680cb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.832+0000 7f196d200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f196815ecd0 con 0x7f19680ccb20 2024-04-16T16:14:01.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.832+0000 7f196d200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19680cd6d0 con 0x7f196815e8f0 2024-04-16T16:14:01.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.832+0000 7f196d200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f19680ce2e0 con 0x7f19680ce9a0 2024-04-16T16:14:01.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.832+0000 7f1966a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f19680ccb20 0x7f19680cac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49944/0 (socket says 172.21.15.45:49944) 2024-04-16T16:14:01.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.832+0000 7f1966a00640 1 -- 172.21.15.45:0/3827264712 learned_addr learned my addr 172.21.15.45:0/3827264712 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:01.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.832+0000 7f1967e00640 1 --2- 172.21.15.45:0/3827264712 >> v2:172.21.15.103:3300/0 conn(0x7f19680ce9a0 0x7f19680cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:01.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966000640 1 -- 172.21.15.45:0/3827264712 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 578118925 0 0) 0x7f196815ecd0 con 0x7f19680ccb20 2024-04-16T16:14:01.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966000640 1 -- 172.21.15.45:0/3827264712 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f194c003520 con 0x7f19680ccb20 2024-04-16T16:14:01.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966000640 1 -- 172.21.15.45:0/3827264712 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 927705962 0 0) 0x7f194c003520 con 0x7f19680ccb20 2024-04-16T16:14:01.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966000640 1 -- 172.21.15.45:0/3827264712 >> v2:172.21.15.103:3300/0 conn(0x7f19680ce9a0 msgr2=0x7f19680cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:01.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966000640 1 --2- 172.21.15.45:0/3827264712 >> v2:172.21.15.103:3300/0 conn(0x7f19680ce9a0 0x7f19680cb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966000640 1 -- 172.21.15.45:0/3827264712 >> v1:172.21.15.103:6789/0 conn(0x7f196815e8f0 legacy=0x7f19680cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:01.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966000640 1 -- 172.21.15.45:0/3827264712 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19680cd9b0 con 0x7f19680ccb20 2024-04-16T16:14:01.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966000640 1 -- 172.21.15.45:0/3827264712 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f1954001670 con 0x7f19680ccb20 2024-04-16T16:14:01.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966000640 1 -- 172.21.15.45:0/3827264712 >> v1:172.21.15.45:6789/0 conn(0x7f19680ccb20 legacy=0x7f19680cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:01.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966000640 1 --2- 172.21.15.45:0/3827264712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f194c003a00 0x7f194c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966000640 1 --2- 172.21.15.45:0/3827264712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c004800 0x7f194c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966000640 1 --2- 172.21.15.45:0/3827264712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194c005210 0x7f194c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966000640 1 -- 172.21.15.45:0/3827264712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f19680ce2e0 con 0x7f194c004800 2024-04-16T16:14:01.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966000640 1 -- 172.21.15.45:0/3827264712 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f194c00c160 con 0x7f194c003a00 2024-04-16T16:14:01.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966000640 1 -- 172.21.15.45:0/3827264712 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f194c00c360 con 0x7f194c005210 2024-04-16T16:14:01.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.833+0000 7f1966a00640 1 --2- 172.21.15.45:0/3827264712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c004800 0x7f194c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:01.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.834+0000 7f1966a00640 1 -- 172.21.15.45:0/3827264712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194c005210 msgr2=0x7f194c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:01.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.834+0000 7f1966a00640 1 --2- 172.21.15.45:0/3827264712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194c005210 0x7f194c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.834+0000 7f1966a00640 1 -- 172.21.15.45:0/3827264712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f194c003a00 msgr2=0x7f194c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:01.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.834+0000 7f1967400640 1 --2- 172.21.15.45:0/3827264712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f194c003a00 0x7f194c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:01.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.834+0000 7f1966a00640 1 --2- 172.21.15.45:0/3827264712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f194c003a00 0x7f194c003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.834+0000 7f1967e00640 1 -- 172.21.15.45:0/3827264712 reap_dead start 2024-04-16T16:14:01.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.834+0000 7f1966a00640 1 -- 172.21.15.45:0/3827264712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194c003520 con 0x7f194c004800 2024-04-16T16:14:01.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.835+0000 7f1966a00640 1 --2- 172.21.15.45:0/3827264712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c004800 0x7f194c004c80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f195400c160 tx=0x7f194c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a15f55ec6eaf40f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:01.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.835+0000 7f1966000640 1 -- 172.21.15.45:0/3827264712 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954007c60 con 0x7f194c004800 2024-04-16T16:14:01.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.835+0000 7f1966000640 1 -- 172.21.15.45:0/3827264712 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1954030070 con 0x7f194c004800 2024-04-16T16:14:01.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.835+0000 7f1966000640 1 -- 172.21.15.45:0/3827264712 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954005d90 con 0x7f194c004800 2024-04-16T16:14:01.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.836+0000 7f196d200640 1 -- 172.21.15.45:0/3827264712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c004800 msgr2=0x7f194c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:01.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.836+0000 7f196d200640 1 --2- 172.21.15.45:0/3827264712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c004800 0x7f194c004c80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f195400c160 tx=0x7f194c005830 comp rx=0 tx=0).stop 2024-04-16T16:14:01.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.836+0000 7f196d200640 1 -- 172.21.15.45:0/3827264712 shutdown_connections 2024-04-16T16:14:01.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.836+0000 7f196d200640 1 --2- 172.21.15.45:0/3827264712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c004800 0x7f194c004c80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.836+0000 7f196d200640 1 -- 172.21.15.45:0/3827264712 >> 172.21.15.45:0/3827264712 conn(0x7f196806b930 msgr2=0x7f196806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:01.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.836+0000 7f196d200640 1 -- 172.21.15.45:0/3827264712 shutdown_connections 2024-04-16T16:14:01.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.836+0000 7f196d200640 1 -- 172.21.15.45:0/3827264712 wait complete. 2024-04-16T16:14:01.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.838+0000 7f196d200640 1 Processor -- start 2024-04-16T16:14:01.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.838+0000 7f196d200640 1 -- start start 2024-04-16T16:14:01.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.839+0000 7f196d200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194c004800 0x7f196815d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.839+0000 7f196d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f196815f8e0 0x7f19681f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.839+0000 7f196d200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19681f4fb0 0x7f19681f5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.839+0000 7f196d200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f19680cd360 con 0x7f196815f8e0 2024-04-16T16:14:01.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.839+0000 7f196d200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f19680cdf70 con 0x7f19681f4fb0 2024-04-16T16:14:01.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.839+0000 7f196d200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f19680cf080 con 0x7f194c004800 2024-04-16T16:14:01.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.839+0000 7f1966a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f196815f8e0 0x7f19681f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:01.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.840+0000 7f1966a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f196815f8e0 0x7f19681f4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54508/0 (socket says 172.21.15.45:54508) 2024-04-16T16:14:01.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.840+0000 7f1966a00640 1 -- 172.21.15.45:0/3429277665 learned_addr learned my addr 172.21.15.45:0/3429277665 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:01.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.840+0000 7f1967400640 1 --2- 172.21.15.45:0/3429277665 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194c004800 0x7f196815d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:01.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.840+0000 7f1966a00640 1 -- 172.21.15.45:0/3429277665 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194c004800 msgr2=0x7f196815d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:01.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.840+0000 7f1966a00640 1 --2- 172.21.15.45:0/3429277665 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194c004800 0x7f196815d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.840+0000 7f1966a00640 1 -- 172.21.15.45:0/3429277665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19681f4fb0 msgr2=0x7f19681f5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:01.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.840+0000 7f1967e00640 1 --2- 172.21.15.45:0/3429277665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19681f4fb0 0x7f19681f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:01.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.840+0000 7f1966a00640 1 --2- 172.21.15.45:0/3429277665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19681f4fb0 0x7f19681f5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:01.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.840+0000 7f1966a00640 1 -- 172.21.15.45:0/3429277665 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1968161d50 con 0x7f196815f8e0 2024-04-16T16:14:01.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.840+0000 7f1967400640 1 --2- 172.21.15.45:0/3429277665 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194c004800 0x7f196815d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:01.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.840+0000 7f1966a00640 1 --2- 172.21.15.45:0/3429277665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f196815f8e0 0x7f19681f4a70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f195402c3d0 tx=0x7f1954001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:01.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.841+0000 7f195f400640 1 -- 172.21.15.45:0/3429277665 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954045070 con 0x7f196815f8e0 2024-04-16T16:14:01.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.841+0000 7f195f400640 1 -- 172.21.15.45:0/3429277665 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1954005e30 con 0x7f196815f8e0 2024-04-16T16:14:01.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.841+0000 7f195f400640 1 -- 172.21.15.45:0/3429277665 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954007480 con 0x7f196815f8e0 2024-04-16T16:14:01.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.841+0000 7f196d200640 1 -- 172.21.15.45:0/3429277665 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19681f5970 con 0x7f196815f8e0 2024-04-16T16:14:01.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.841+0000 7f196d200640 1 -- 172.21.15.45:0/3429277665 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19681601d0 con 0x7f196815f8e0 2024-04-16T16:14:01.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.842+0000 7f196d200640 1 -- 172.21.15.45:0/3429277665 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19680b2dc0 con 0x7f196815f8e0 2024-04-16T16:14:01.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.843+0000 7f195f400640 1 -- 172.21.15.45:0/3429277665 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1954048070 con 0x7f196815f8e0 2024-04-16T16:14:01.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.844+0000 7f195f400640 1 --2- 172.21.15.45:0/3429277665 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f194806c920 0x7f194806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:01.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.844+0000 7f195f400640 1 -- 172.21.15.45:0/3429277665 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f19540ac790 con 0x7f196815f8e0 2024-04-16T16:14:01.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.845+0000 7f1967400640 1 --2- 172.21.15.45:0/3429277665 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f194806c920 0x7f194806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:01.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.846+0000 7f1967400640 1 --2- 172.21.15.45:0/3429277665 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f194806c920 0x7f194806ede0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f19580016f0 tx=0x7f1958009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:01.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:01.849+0000 7f195f400640 1 -- 172.21.15.45:0/3429277665 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f19680b2dc0 con 0x7f196815f8e0 2024-04-16T16:14:02.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.055+0000 7f196d200640 1 -- 172.21.15.45:0/3429277665 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f196815aaf0 con 0x7f196815f8e0 2024-04-16T16:14:02.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.058+0000 7f195f400640 1 -- 172.21.15.45:0/3429277665 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f19681f5970 con 0x7f196815f8e0 2024-04-16T16:14:02.057 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:02.057 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":21,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:26.918617+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:02.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.060+0000 7f196d200640 1 -- 172.21.15.45:0/3429277665 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f194806c920 msgr2=0x7f194806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:02.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.060+0000 7f196d200640 1 --2- 172.21.15.45:0/3429277665 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f194806c920 0x7f194806ede0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f19580016f0 tx=0x7f1958009040 comp rx=0 tx=0).stop 2024-04-16T16:14:02.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.060+0000 7f196d200640 1 -- 172.21.15.45:0/3429277665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f196815f8e0 msgr2=0x7f19681f4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:02.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.060+0000 7f196d200640 1 --2- 172.21.15.45:0/3429277665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f196815f8e0 0x7f19681f4a70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f195402c3d0 tx=0x7f1954001970 comp rx=0 tx=0).stop 2024-04-16T16:14:02.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.061+0000 7f196d200640 1 -- 172.21.15.45:0/3429277665 shutdown_connections 2024-04-16T16:14:02.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.061+0000 7f196d200640 1 --2- 172.21.15.45:0/3429277665 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f194806c920 0x7f194806ede0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.061+0000 7f196d200640 1 --2- 172.21.15.45:0/3429277665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19681f4fb0 0x7f19681f5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.061+0000 7f196d200640 1 --2- 172.21.15.45:0/3429277665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f196815f8e0 0x7f19681f4a70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.061+0000 7f196d200640 1 --2- 172.21.15.45:0/3429277665 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194c004800 0x7f196815d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.061+0000 7f196d200640 1 -- 172.21.15.45:0/3429277665 >> 172.21.15.45:0/3429277665 conn(0x7f196806b930 msgr2=0x7f19681591c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:02.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.061+0000 7f196d200640 1 -- 172.21.15.45:0/3429277665 shutdown_connections 2024-04-16T16:14:02.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.061+0000 7f196d200640 1 -- 172.21.15.45:0/3429277665 wait complete. 2024-04-16T16:14:02.078 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:13:23.600104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:13:24.368141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:14:02.079 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:14:02.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.305+0000 7f5dff200640 1 Processor -- start 2024-04-16T16:14:02.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.305+0000 7f5dff200640 1 -- start start 2024-04-16T16:14:02.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.307+0000 7f5dff200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5e0015ea70 0x7f5e000d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:02.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.307+0000 7f5dff200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5e000cfc40 0x7f5e00162560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:02.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.307+0000 7f5dff200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e0015ee50 con 0x7f5e000d4370 2024-04-16T16:14:02.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.307+0000 7f5dff200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5e000d1a10 con 0x7f5e0015ea70 2024-04-16T16:14:02.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.307+0000 7f5dff200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5e000b6480 con 0x7f5e000cfc40 2024-04-16T16:14:02.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.307+0000 7f5dfde00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5e0015ea70 0x7f5e000d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:02.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.307+0000 7f5dfde00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5e0015ea70 0x7f5e000d11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54520/0 (socket says 172.21.15.45:54520) 2024-04-16T16:14:02.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.307+0000 7f5dfde00640 1 -- 172.21.15.45:0/3184268526 learned_addr learned my addr 172.21.15.45:0/3184268526 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:02.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.308+0000 7f5dfde00640 1 -- 172.21.15.45:0/3184268526 >> v2:172.21.15.103:3300/0 conn(0x7f5e000cfc40 msgr2=0x7f5e00162560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:02.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.308+0000 7f5dfe800640 1 --2- 172.21.15.45:0/3184268526 >> v2:172.21.15.103:3300/0 conn(0x7f5e000cfc40 0x7f5e00162560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:02.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.308+0000 7f5dfde00640 1 --2- 172.21.15.45:0/3184268526 >> v2:172.21.15.103:3300/0 conn(0x7f5e000cfc40 0x7f5e00162560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.308+0000 7f5dfde00640 1 -- 172.21.15.45:0/3184268526 >> v1:172.21.15.103:6789/0 conn(0x7f5e000d4370 legacy=0x7f5e000cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:02.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.308+0000 7f5dfde00640 1 -- 172.21.15.45:0/3184268526 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e000b6760 con 0x7f5e0015ea70 2024-04-16T16:14:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.308+0000 7f5dfe800640 1 --2- 172.21.15.45:0/3184268526 >> v2:172.21.15.103:3300/0 conn(0x7f5e000cfc40 0x7f5e00162560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.309+0000 7f5dfde00640 1 --2- 172.21.15.45:0/3184268526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0015ea70 0x7f5e000d11a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5e00074010 tx=0x7f5df002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96b54e6657c3fb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.309+0000 7f5dfd400640 1 -- 172.21.15.45:0/3184268526 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df0035020 con 0x7f5e0015ea70 2024-04-16T16:14:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.309+0000 7f5dfd400640 1 -- 172.21.15.45:0/3184268526 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5df0031870 con 0x7f5e0015ea70 2024-04-16T16:14:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.309+0000 7f5dfd400640 1 -- 172.21.15.45:0/3184268526 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df0031c40 con 0x7f5e0015ea70 2024-04-16T16:14:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.310+0000 7f5dff200640 1 -- 172.21.15.45:0/3184268526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0015ea70 msgr2=0x7f5e000d11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.310+0000 7f5dff200640 1 --2- 172.21.15.45:0/3184268526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0015ea70 0x7f5e000d11a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5e00074010 tx=0x7f5df002b910 comp rx=0 tx=0).stop 2024-04-16T16:14:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.310+0000 7f5dff200640 1 -- 172.21.15.45:0/3184268526 shutdown_connections 2024-04-16T16:14:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.310+0000 7f5dff200640 1 --2- 172.21.15.45:0/3184268526 >> v2:172.21.15.103:3300/0 conn(0x7f5e000cfc40 0x7f5e00162560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.310+0000 7f5dff200640 1 --2- 172.21.15.45:0/3184268526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0015ea70 0x7f5e000d11a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.310+0000 7f5dff200640 1 -- 172.21.15.45:0/3184268526 >> 172.21.15.45:0/3184268526 conn(0x7f5e0006d3b0 msgr2=0x7f5e00073060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.310+0000 7f5dff200640 1 -- 172.21.15.45:0/3184268526 shutdown_connections 2024-04-16T16:14:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.310+0000 7f5dff200640 1 -- 172.21.15.45:0/3184268526 wait complete. 2024-04-16T16:14:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.312+0000 7f5dff200640 1 Processor -- start 2024-04-16T16:14:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.312+0000 7f5dff200640 1 -- start start 2024-04-16T16:14:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.313+0000 7f5dff200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e000cfc40 0x7f5e00153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.313+0000 7f5dff200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e000d4370 0x7f5e0015f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.313+0000 7f5dff200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0015ea70 0x7f5e000ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.313+0000 7f5dff200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5e000b6f60 con 0x7f5e0015ea70 2024-04-16T16:14:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.313+0000 7f5dff200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5e000d16e0 con 0x7f5e000cfc40 2024-04-16T16:14:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.313+0000 7f5dff200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5e0015fb50 con 0x7f5e000d4370 2024-04-16T16:14:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.313+0000 7f5dfe800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0015ea70 0x7f5e000ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.314+0000 7f5dfe800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0015ea70 0x7f5e000ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54528/0 (socket says 172.21.15.45:54528) 2024-04-16T16:14:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.314+0000 7f5dfe800640 1 -- 172.21.15.45:0/2694010258 learned_addr learned my addr 172.21.15.45:0/2694010258 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:02.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.314+0000 7f5df5400640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e000d4370 0x7f5e0015f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:02.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.314+0000 7f5dfde00640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e000cfc40 0x7f5e00153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:02.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.314+0000 7f5dfe800640 1 -- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e000d4370 msgr2=0x7f5e0015f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:02.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.314+0000 7f5dfe800640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e000d4370 0x7f5e0015f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.314+0000 7f5dfe800640 1 -- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e000cfc40 msgr2=0x7f5e00153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.314+0000 7f5dfe800640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e000cfc40 0x7f5e00153fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.314+0000 7f5dfe800640 1 -- 172.21.15.45:0/2694010258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0015fe30 con 0x7f5e0015ea70 2024-04-16T16:14:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.314+0000 7f5df5400640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e000d4370 0x7f5e0015f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.314+0000 7f5dfde00640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e000cfc40 0x7f5e00153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.314+0000 7f5dfe800640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0015ea70 0x7f5e000ccd90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5df800cd20 tx=0x7f5df800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.315+0000 7f5df6a00640 1 -- 172.21.15.45:0/2694010258 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df800f680 con 0x7f5e0015ea70 2024-04-16T16:14:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.315+0000 7f5df6a00640 1 -- 172.21.15.45:0/2694010258 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5df800f840 con 0x7f5e0015ea70 2024-04-16T16:14:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.315+0000 7f5df6a00640 1 -- 172.21.15.45:0/2694010258 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df800fc30 con 0x7f5e0015ea70 2024-04-16T16:14:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.315+0000 7f5dff200640 1 -- 172.21.15.45:0/2694010258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e00160110 con 0x7f5e0015ea70 2024-04-16T16:14:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.315+0000 7f5dff200640 1 -- 172.21.15.45:0/2694010258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e001544e0 con 0x7f5e0015ea70 2024-04-16T16:14:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.316+0000 7f5dff200640 1 -- 172.21.15.45:0/2694010258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e00075780 con 0x7f5e0015ea70 2024-04-16T16:14:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.317+0000 7f5df6a00640 1 -- 172.21.15.45:0/2694010258 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5df80213f0 con 0x7f5e0015ea70 2024-04-16T16:14:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.318+0000 7f5df6a00640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ddc06c8f0 0x7f5ddc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.318+0000 7f5df6a00640 1 -- 172.21.15.45:0/2694010258 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5df808c990 con 0x7f5e0015ea70 2024-04-16T16:14:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.319+0000 7f5dfde00640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ddc06c8f0 0x7f5ddc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.320+0000 7f5dfde00640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ddc06c8f0 0x7f5ddc06edb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5df0036000 tx=0x7f5df003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:02.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.322+0000 7f5df6a00640 1 -- 172.21.15.45:0/2694010258 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5df8014030 con 0x7f5e0015ea70 2024-04-16T16:14:02.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.521+0000 7f5dff200640 1 -- 172.21.15.45:0/2694010258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5e000c9540 con 0x7f5e0015ea70 2024-04-16T16:14:02.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.522+0000 7f5df6a00640 1 -- 172.21.15.45:0/2694010258 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f5df805ad20 con 0x7f5e0015ea70 2024-04-16T16:14:02.521 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:14:02.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.524+0000 7f5dff200640 1 -- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ddc06c8f0 msgr2=0x7f5ddc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:02.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.524+0000 7f5dff200640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ddc06c8f0 0x7f5ddc06edb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5df0036000 tx=0x7f5df003c040 comp rx=0 tx=0).stop 2024-04-16T16:14:02.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.524+0000 7f5dff200640 1 -- 172.21.15.45:0/2694010258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0015ea70 msgr2=0x7f5e000ccd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:02.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.524+0000 7f5dff200640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0015ea70 0x7f5e000ccd90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5df800cd20 tx=0x7f5df800a650 comp rx=0 tx=0).stop 2024-04-16T16:14:02.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.524+0000 7f5dff200640 1 -- 172.21.15.45:0/2694010258 shutdown_connections 2024-04-16T16:14:02.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.524+0000 7f5dff200640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ddc06c8f0 0x7f5ddc06edb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.524+0000 7f5dff200640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0015ea70 0x7f5e000ccd90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.524+0000 7f5dff200640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e000d4370 0x7f5e0015f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.524+0000 7f5dff200640 1 --2- 172.21.15.45:0/2694010258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e000cfc40 0x7f5e00153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.524+0000 7f5dff200640 1 -- 172.21.15.45:0/2694010258 >> 172.21.15.45:0/2694010258 conn(0x7f5e0006d3b0 msgr2=0x7f5e000b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:02.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.525+0000 7f5dff200640 1 -- 172.21.15.45:0/2694010258 shutdown_connections 2024-04-16T16:14:02.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.525+0000 7f5dff200640 1 -- 172.21.15.45:0/2694010258 wait complete. 2024-04-16T16:14:02.543 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:14:02.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.755+0000 7f34c3200640 1 Processor -- start 2024-04-16T16:14:02.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.755+0000 7f34c3200640 1 -- start start 2024-04-16T16:14:02.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.757+0000 7f34c3200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f34c40b58f0 0x7f34c40b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:02.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.757+0000 7f34c3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34c406e460 0x7f34c406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:02.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.757+0000 7f34c3200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34c4146590 con 0x7f34c41461b0 2024-04-16T16:14:02.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.757+0000 7f34c3200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f34c406dfd0 con 0x7f34c40b58f0 2024-04-16T16:14:02.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.757+0000 7f34c3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f34c40b6290 con 0x7f34c406e460 2024-04-16T16:14:02.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.757+0000 7f34c1e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f34c41461b0 0x7f34c4160cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49954/0 (socket says 172.21.15.45:49954) 2024-04-16T16:14:02.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.757+0000 7f34c1e00640 1 -- 172.21.15.45:0/3483963823 learned_addr learned my addr 172.21.15.45:0/3483963823 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:02.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c2800640 1 --2- 172.21.15.45:0/3483963823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34c406e460 0x7f34c406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:02.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c1400640 1 --2- 172.21.15.45:0/3483963823 >> v2:172.21.15.103:3300/0 conn(0x7f34c40b58f0 0x7f34c40b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:02.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 -- 172.21.15.45:0/3483963823 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4129694952 0 0) 0x7f34c4146590 con 0x7f34c41461b0 2024-04-16T16:14:02.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 -- 172.21.15.45:0/3483963823 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34a8003520 con 0x7f34c41461b0 2024-04-16T16:14:02.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 -- 172.21.15.45:0/3483963823 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 540693381 0 0) 0x7f34a8003520 con 0x7f34c41461b0 2024-04-16T16:14:02.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 -- 172.21.15.45:0/3483963823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34c406e460 msgr2=0x7f34c406e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:02.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 --2- 172.21.15.45:0/3483963823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34c406e460 0x7f34c406e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 -- 172.21.15.45:0/3483963823 >> v2:172.21.15.103:3300/0 conn(0x7f34c40b58f0 msgr2=0x7f34c40b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:02.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 --2- 172.21.15.45:0/3483963823 >> v2:172.21.15.103:3300/0 conn(0x7f34c40b58f0 0x7f34c40b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 -- 172.21.15.45:0/3483963823 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c40c5750 con 0x7f34c41461b0 2024-04-16T16:14:02.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 -- 172.21.15.45:0/3483963823 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f34b4001670 con 0x7f34c41461b0 2024-04-16T16:14:02.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 -- 172.21.15.45:0/3483963823 >> v1:172.21.15.45:6789/0 conn(0x7f34c41461b0 legacy=0x7f34c4160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:02.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 --2- 172.21.15.45:0/3483963823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34a80039b0 0x7f34a8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:02.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 --2- 172.21.15.45:0/3483963823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a8004800 0x7f34a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:02.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 --2- 172.21.15.45:0/3483963823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34c406e460 0x7f34a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:02.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 -- 172.21.15.45:0/3483963823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f34a800b9c0 con 0x7f34a8004800 2024-04-16T16:14:02.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 -- 172.21.15.45:0/3483963823 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34a800bb40 con 0x7f34a80039b0 2024-04-16T16:14:02.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.758+0000 7f34c0a00640 1 -- 172.21.15.45:0/3483963823 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f34a800bcc0 con 0x7f34c406e460 2024-04-16T16:14:02.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.759+0000 7f34c1400640 1 --2- 172.21.15.45:0/3483963823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a8004800 0x7f34a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:02.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.759+0000 7f34c1e00640 1 --2- 172.21.15.45:0/3483963823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34a80039b0 0x7f34a8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:02.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.759+0000 7f34c2800640 1 --2- 172.21.15.45:0/3483963823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34c406e460 0x7f34a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:02.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.759+0000 7f34c1400640 1 -- 172.21.15.45:0/3483963823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34c406e460 msgr2=0x7f34a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:02.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.759+0000 7f34c1400640 1 --2- 172.21.15.45:0/3483963823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34c406e460 0x7f34a80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.759+0000 7f34c1400640 1 -- 172.21.15.45:0/3483963823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34a80039b0 msgr2=0x7f34a8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:02.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.759+0000 7f34c1400640 1 --2- 172.21.15.45:0/3483963823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34a80039b0 0x7f34a8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.759+0000 7f34c1400640 1 -- 172.21.15.45:0/3483963823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a8003520 con 0x7f34a8004800 2024-04-16T16:14:02.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.760+0000 7f34c1400640 1 --2- 172.21.15.45:0/3483963823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a8004800 0x7f34a8004c80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009ac0 tx=0x7f34a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a578229ece73d43c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:02.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.761+0000 7f34c0a00640 1 -- 172.21.15.45:0/3483963823 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0039070 con 0x7f34a8004800 2024-04-16T16:14:02.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.761+0000 7f34c0a00640 1 -- 172.21.15.45:0/3483963823 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34b002c710 con 0x7f34a8004800 2024-04-16T16:14:02.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.761+0000 7f34c0a00640 1 -- 172.21.15.45:0/3483963823 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0009690 con 0x7f34a8004800 2024-04-16T16:14:02.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.761+0000 7f34c3200640 1 -- 172.21.15.45:0/3483963823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a8004800 msgr2=0x7f34a8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:02.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.761+0000 7f34c3200640 1 --2- 172.21.15.45:0/3483963823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a8004800 0x7f34a8004c80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009ac0 tx=0x7f34a8005830 comp rx=0 tx=0).stop 2024-04-16T16:14:02.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.761+0000 7f34c2800640 1 -- 172.21.15.45:0/3483963823 reap_dead start 2024-04-16T16:14:02.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.762+0000 7f34c3200640 1 -- 172.21.15.45:0/3483963823 shutdown_connections 2024-04-16T16:14:02.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.762+0000 7f34c3200640 1 -- 172.21.15.45:0/3483963823 >> 172.21.15.45:0/3483963823 conn(0x7f34c406b930 msgr2=0x7f34c406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:02.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.762+0000 7f34c3200640 1 -- 172.21.15.45:0/3483963823 shutdown_connections 2024-04-16T16:14:02.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.762+0000 7f34c3200640 1 -- 172.21.15.45:0/3483963823 wait complete. 2024-04-16T16:14:02.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.763+0000 7f34c3200640 1 Processor -- start 2024-04-16T16:14:02.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.764+0000 7f34c3200640 1 -- start start 2024-04-16T16:14:02.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.764+0000 7f34c3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a8004800 0x7f34c4144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:02.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.764+0000 7f34c3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34c4139fb0 0x7f34c4143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:02.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.765+0000 7f34c3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34c40b4830 0x7f34c40b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:02.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.765+0000 7f34c3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f34c40c5cd0 con 0x7f34a8004800 2024-04-16T16:14:02.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.765+0000 7f34c3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34c40c59f0 con 0x7f34c4139fb0 2024-04-16T16:14:02.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.765+0000 7f34c3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f34c40c5fb0 con 0x7f34c40b4830 2024-04-16T16:14:02.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.765+0000 7f34c1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a8004800 0x7f34c4144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:02.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.765+0000 7f34c1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a8004800 0x7f34c4144eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54542/0 (socket says 172.21.15.45:54542) 2024-04-16T16:14:02.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.765+0000 7f34c1e00640 1 -- 172.21.15.45:0/891807447 learned_addr learned my addr 172.21.15.45:0/891807447 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:02.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.765+0000 7f34c1400640 1 --2- 172.21.15.45:0/891807447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34c4139fb0 0x7f34c4143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:02.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.765+0000 7f34c1e00640 1 -- 172.21.15.45:0/891807447 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34c40b4830 msgr2=0x7f34c40b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:02.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.765+0000 7f34c1e00640 1 --2- 172.21.15.45:0/891807447 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34c40b4830 0x7f34c40b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.765+0000 7f34c1e00640 1 -- 172.21.15.45:0/891807447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34c4139fb0 msgr2=0x7f34c4143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:02.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.765+0000 7f34c1e00640 1 --2- 172.21.15.45:0/891807447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34c4139fb0 0x7f34c4143130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.765+0000 7f34c1e00640 1 -- 172.21.15.45:0/891807447 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c4144220 con 0x7f34a8004800 2024-04-16T16:14:02.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.765+0000 7f34c1400640 1 --2- 172.21.15.45:0/891807447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34c4139fb0 0x7f34c4143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:02.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.766+0000 7f34c1e00640 1 --2- 172.21.15.45:0/891807447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a8004800 0x7f34c4144eb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f34b400e730 tx=0x7f34b400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:02.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.766+0000 7f34ba000640 1 -- 172.21.15.45:0/891807447 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b400c640 con 0x7f34a8004800 2024-04-16T16:14:02.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.766+0000 7f34ba000640 1 -- 172.21.15.45:0/891807447 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34b4018070 con 0x7f34a8004800 2024-04-16T16:14:02.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.766+0000 7f34c3200640 1 -- 172.21.15.45:0/891807447 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c4143980 con 0x7f34a8004800 2024-04-16T16:14:02.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.766+0000 7f34ba000640 1 -- 172.21.15.45:0/891807447 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b400ca20 con 0x7f34a8004800 2024-04-16T16:14:02.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.766+0000 7f34c3200640 1 -- 172.21.15.45:0/891807447 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c4143f40 con 0x7f34a8004800 2024-04-16T16:14:02.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.767+0000 7f34c3200640 1 -- 172.21.15.45:0/891807447 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34c4075780 con 0x7f34a8004800 2024-04-16T16:14:02.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.768+0000 7f34ba000640 1 -- 172.21.15.45:0/891807447 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f34b400cbc0 con 0x7f34a8004800 2024-04-16T16:14:02.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.769+0000 7f34ba000640 1 --2- 172.21.15.45:0/891807447 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f34a406c8f0 0x7f34a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:02.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.769+0000 7f34ba000640 1 -- 172.21.15.45:0/891807447 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f34b408b8b0 con 0x7f34a8004800 2024-04-16T16:14:02.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.770+0000 7f34c1400640 1 --2- 172.21.15.45:0/891807447 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f34a406c8f0 0x7f34a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:02.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.771+0000 7f34c1400640 1 --2- 172.21.15.45:0/891807447 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f34a406c8f0 0x7f34a406edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f34b0036000 tx=0x7f34b0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:02.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.774+0000 7f34ba000640 1 -- 172.21.15.45:0/891807447 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f34b4059c40 con 0x7f34a8004800 2024-04-16T16:14:02.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.975+0000 7f34c3200640 1 -- 172.21.15.45:0/891807447 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f34c4072500 con 0x7f34a8004800 2024-04-16T16:14:02.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.977+0000 7f34ba000640 1 -- 172.21.15.45:0/891807447 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f34b4059540 con 0x7f34a8004800 2024-04-16T16:14:02.976 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:14:02.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.979+0000 7f34c3200640 1 -- 172.21.15.45:0/891807447 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f34a406c8f0 msgr2=0x7f34a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:02.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.979+0000 7f34c3200640 1 --2- 172.21.15.45:0/891807447 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f34a406c8f0 0x7f34a406edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f34b0036000 tx=0x7f34b0030000 comp rx=0 tx=0).stop 2024-04-16T16:14:02.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.979+0000 7f34c3200640 1 -- 172.21.15.45:0/891807447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a8004800 msgr2=0x7f34c4144eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:02.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.979+0000 7f34c3200640 1 --2- 172.21.15.45:0/891807447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a8004800 0x7f34c4144eb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f34b400e730 tx=0x7f34b400ebf0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.979+0000 7f34c3200640 1 -- 172.21.15.45:0/891807447 shutdown_connections 2024-04-16T16:14:02.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.979+0000 7f34c3200640 1 --2- 172.21.15.45:0/891807447 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f34a406c8f0 0x7f34a406edb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.979+0000 7f34c3200640 1 --2- 172.21.15.45:0/891807447 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34c40b4830 0x7f34c40b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.979+0000 7f34c3200640 1 --2- 172.21.15.45:0/891807447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34c4139fb0 0x7f34c4143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.979+0000 7f34c3200640 1 --2- 172.21.15.45:0/891807447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a8004800 0x7f34c4144eb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:02.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.979+0000 7f34c3200640 1 -- 172.21.15.45:0/891807447 >> 172.21.15.45:0/891807447 conn(0x7f34c406b930 msgr2=0x7f34c40b24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:02.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.979+0000 7f34c3200640 1 -- 172.21.15.45:0/891807447 shutdown_connections 2024-04-16T16:14:02.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:02.979+0000 7f34c3200640 1 -- 172.21.15.45:0/891807447 wait complete. 2024-04-16T16:14:02.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:14:03.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.224+0000 7f804da00640 1 Processor -- start 2024-04-16T16:14:03.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.225+0000 7f804da00640 1 -- start start 2024-04-16T16:14:03.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.226+0000 7f804da00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8048148190 0x7f80480b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:03.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.226+0000 7f804da00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f80480b5e30 0x7f80480ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:03.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.226+0000 7f804da00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8048148570 con 0x7f80480ceb50 2024-04-16T16:14:03.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.226+0000 7f804da00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f80480b6290 con 0x7f80480b5e30 2024-04-16T16:14:03.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.226+0000 7f804da00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f804815fbb0 con 0x7f8048148190 2024-04-16T16:14:03.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.227+0000 7f803fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f80480b5e30 0x7f80480ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:03.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.227+0000 7f8047e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8048148190 0x7f80480b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:03.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.227+0000 7f804d000640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f80480ceb50 0x7f8048071f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:48290/0 (socket says 172.21.15.45:48290) 2024-04-16T16:14:03.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.227+0000 7f804d000640 1 -- 172.21.15.45:0/3427139533 learned_addr learned my addr 172.21.15.45:0/3427139533 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:03.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.228+0000 7f8047400640 1 -- 172.21.15.45:0/3427139533 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3569384846 0 0) 0x7f8048148570 con 0x7f80480ceb50 2024-04-16T16:14:03.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.228+0000 7f8047400640 1 -- 172.21.15.45:0/3427139533 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f802c003520 con 0x7f80480ceb50 2024-04-16T16:14:03.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.228+0000 7f803fe00640 1 -- 172.21.15.45:0/3427139533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8048148190 msgr2=0x7f80480b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:03.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.228+0000 7f803fe00640 1 --2- 172.21.15.45:0/3427139533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8048148190 0x7f80480b58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.228+0000 7f803fe00640 1 -- 172.21.15.45:0/3427139533 >> v1:172.21.15.103:6789/0 conn(0x7f80480ceb50 legacy=0x7f8048071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:03.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.228+0000 7f803fe00640 1 -- 172.21.15.45:0/3427139533 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f804815efa0 con 0x7f80480b5e30 2024-04-16T16:14:03.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.228+0000 7f8047e00640 1 --2- 172.21.15.45:0/3427139533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8048148190 0x7f80480b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:14:03.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.230+0000 7f803fe00640 1 --2- 172.21.15.45:0/3427139533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80480b5e30 0x7f80480ce610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f80300097a0 tx=0x7f803002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3fc0539fd88f693 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:03.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.230+0000 7f8047400640 1 -- 172.21.15.45:0/3427139533 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8030035020 con 0x7f80480b5e30 2024-04-16T16:14:03.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.230+0000 7f8047400640 1 -- 172.21.15.45:0/3427139533 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8030031810 con 0x7f80480b5e30 2024-04-16T16:14:03.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.230+0000 7f8047400640 1 -- 172.21.15.45:0/3427139533 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8030031be0 con 0x7f80480b5e30 2024-04-16T16:14:03.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.231+0000 7f804da00640 1 -- 172.21.15.45:0/3427139533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80480b5e30 msgr2=0x7f80480ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:03.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.231+0000 7f804da00640 1 --2- 172.21.15.45:0/3427139533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80480b5e30 0x7f80480ce610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f80300097a0 tx=0x7f803002b8b0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.231+0000 7f804da00640 1 -- 172.21.15.45:0/3427139533 shutdown_connections 2024-04-16T16:14:03.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.231+0000 7f804da00640 1 --2- 172.21.15.45:0/3427139533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80480b5e30 0x7f80480ce610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.231+0000 7f804da00640 1 --2- 172.21.15.45:0/3427139533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8048148190 0x7f80480b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.231+0000 7f804da00640 1 -- 172.21.15.45:0/3427139533 >> 172.21.15.45:0/3427139533 conn(0x7f804806b930 msgr2=0x7f804806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:03.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.231+0000 7f804da00640 1 -- 172.21.15.45:0/3427139533 shutdown_connections 2024-04-16T16:14:03.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.231+0000 7f804da00640 1 -- 172.21.15.45:0/3427139533 wait complete. 2024-04-16T16:14:03.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.233+0000 7f804da00640 1 Processor -- start 2024-04-16T16:14:03.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.233+0000 7f804da00640 1 -- start start 2024-04-16T16:14:03.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.234+0000 7f804da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80480b5e30 0x7f80480ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:03.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.234+0000 7f804da00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80480ceb50 0x7f804815e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:03.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.234+0000 7f804da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8048148190 0x7f80480cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:03.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.234+0000 7f804da00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f804815f840 con 0x7f80480b5e30 2024-04-16T16:14:03.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.234+0000 7f804da00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f804815f560 con 0x7f8048148190 2024-04-16T16:14:03.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.234+0000 7f804da00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f80480cefe0 con 0x7f80480ceb50 2024-04-16T16:14:03.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.234+0000 7f8047e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80480b5e30 0x7f80480ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:03.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.234+0000 7f8047e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80480b5e30 0x7f80480ad2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54558/0 (socket says 172.21.15.45:54558) 2024-04-16T16:14:03.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.234+0000 7f8047e00640 1 -- 172.21.15.45:0/1659972097 learned_addr learned my addr 172.21.15.45:0/1659972097 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:03.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.235+0000 7f803fe00640 1 --2- 172.21.15.45:0/1659972097 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80480ceb50 0x7f804815e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:03.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.235+0000 7f804d000640 1 --2- 172.21.15.45:0/1659972097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8048148190 0x7f80480cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:03.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.235+0000 7f8047e00640 1 -- 172.21.15.45:0/1659972097 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80480ceb50 msgr2=0x7f804815e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:03.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.235+0000 7f8047e00640 1 --2- 172.21.15.45:0/1659972097 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80480ceb50 0x7f804815e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.235+0000 7f8047e00640 1 -- 172.21.15.45:0/1659972097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8048148190 msgr2=0x7f80480cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:03.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.235+0000 7f8047e00640 1 --2- 172.21.15.45:0/1659972097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8048148190 0x7f80480cbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.235+0000 7f8047e00640 1 -- 172.21.15.45:0/1659972097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80480ac220 con 0x7f80480b5e30 2024-04-16T16:14:03.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.235+0000 7f8047e00640 1 --2- 172.21.15.45:0/1659972097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80480b5e30 0x7f80480ad2c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f803800e960 tx=0x7f803800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:03.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.235+0000 7f8044c00640 1 -- 172.21.15.45:0/1659972097 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8038017070 con 0x7f80480b5e30 2024-04-16T16:14:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.235+0000 7f8044c00640 1 -- 172.21.15.45:0/1659972097 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f803800ca00 con 0x7f80480b5e30 2024-04-16T16:14:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.236+0000 7f804da00640 1 -- 172.21.15.45:0/1659972097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80480ac500 con 0x7f80480b5e30 2024-04-16T16:14:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.236+0000 7f8044c00640 1 -- 172.21.15.45:0/1659972097 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803800cdd0 con 0x7f80480b5e30 2024-04-16T16:14:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.236+0000 7f804da00640 1 -- 172.21.15.45:0/1659972097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80480acac0 con 0x7f80480b5e30 2024-04-16T16:14:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.237+0000 7f804da00640 1 -- 172.21.15.45:0/1659972097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8048075780 con 0x7f80480b5e30 2024-04-16T16:14:03.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.238+0000 7f8044c00640 1 -- 172.21.15.45:0/1659972097 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f803801a070 con 0x7f80480b5e30 2024-04-16T16:14:03.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.239+0000 7f8044c00640 1 --2- 172.21.15.45:0/1659972097 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f802406c920 0x7f802406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:03.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.239+0000 7f8044c00640 1 -- 172.21.15.45:0/1659972097 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f803808bd70 con 0x7f80480b5e30 2024-04-16T16:14:03.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.240+0000 7f803fe00640 1 --2- 172.21.15.45:0/1659972097 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f802406c920 0x7f802406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.241+0000 7f803fe00640 1 --2- 172.21.15.45:0/1659972097 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f802406c920 0x7f802406ede0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8030036000 tx=0x7f8030008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.243+0000 7f8044c00640 1 -- 172.21.15.45:0/1659972097 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f803805a1a0 con 0x7f80480b5e30 2024-04-16T16:14:03.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.454+0000 7f804da00640 1 -- 172.21.15.45:0/1659972097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f80480b40a0 con 0x7f80480b5e30 2024-04-16T16:14:03.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.455+0000 7f8044c00640 1 -- 172.21.15.45:0/1659972097 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f8038059aa0 con 0x7f80480b5e30 2024-04-16T16:14:03.454 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:14:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.458+0000 7f804da00640 1 -- 172.21.15.45:0/1659972097 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f802406c920 msgr2=0x7f802406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.458+0000 7f804da00640 1 --2- 172.21.15.45:0/1659972097 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f802406c920 0x7f802406ede0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8030036000 tx=0x7f8030008030 comp rx=0 tx=0).stop 2024-04-16T16:14:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.458+0000 7f804da00640 1 -- 172.21.15.45:0/1659972097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80480b5e30 msgr2=0x7f80480ad2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.458+0000 7f804da00640 1 --2- 172.21.15.45:0/1659972097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80480b5e30 0x7f80480ad2c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f803800e960 tx=0x7f803800ee20 comp rx=0 tx=0).stop 2024-04-16T16:14:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.458+0000 7f804da00640 1 -- 172.21.15.45:0/1659972097 shutdown_connections 2024-04-16T16:14:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.458+0000 7f804da00640 1 --2- 172.21.15.45:0/1659972097 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f802406c920 0x7f802406ede0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.458+0000 7f804da00640 1 --2- 172.21.15.45:0/1659972097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8048148190 0x7f80480cbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.458+0000 7f804da00640 1 --2- 172.21.15.45:0/1659972097 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80480ceb50 0x7f804815e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.458+0000 7f804da00640 1 --2- 172.21.15.45:0/1659972097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80480b5e30 0x7f80480ad2c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.458+0000 7f804da00640 1 -- 172.21.15.45:0/1659972097 >> 172.21.15.45:0/1659972097 conn(0x7f804806b930 msgr2=0x7f80480c9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:03.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.458+0000 7f804da00640 1 -- 172.21.15.45:0/1659972097 shutdown_connections 2024-04-16T16:14:03.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.458+0000 7f804da00640 1 -- 172.21.15.45:0/1659972097 wait complete. 2024-04-16T16:14:03.477 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.695+0000 7fb125e00640 1 Processor -- start 2024-04-16T16:14:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.696+0000 7fb125e00640 1 -- start start 2024-04-16T16:14:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.697+0000 7fb125e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1200b6c00 0x7fb1200cf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.697+0000 7fb125e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb1200cf880 0x7fb1200b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.697+0000 7fb125e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1200d2e20 con 0x7fb12015edf0 2024-04-16T16:14:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.697+0000 7fb125e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb1200d3be0 con 0x7fb1200cf880 2024-04-16T16:14:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.697+0000 7fb125e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb1200d3750 con 0x7fb1200b6c00 2024-04-16T16:14:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.698+0000 7fb11e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1200b6c00 0x7fb1200cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.698+0000 7fb11e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1200b6c00 0x7fb1200cf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:42218/0 (socket says 172.21.15.45:42218) 2024-04-16T16:14:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.698+0000 7fb11e000640 1 -- 172.21.15.45:0/2767406651 learned_addr learned my addr 172.21.15.45:0/2767406651 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.698+0000 7fb11f400640 1 --2- 172.21.15.45:0/2767406651 >> v2:172.21.15.103:3300/0 conn(0x7fb1200cf880 0x7fb1200b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.698+0000 7fb11e000640 1 -- 172.21.15.45:0/2767406651 >> v2:172.21.15.103:3300/0 conn(0x7fb1200cf880 msgr2=0x7fb1200b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.698+0000 7fb11e000640 1 --2- 172.21.15.45:0/2767406651 >> v2:172.21.15.103:3300/0 conn(0x7fb1200cf880 0x7fb1200b7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.698+0000 7fb11e000640 1 -- 172.21.15.45:0/2767406651 >> v1:172.21.15.103:6789/0 conn(0x7fb12015edf0 legacy=0x7fb1200b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.698+0000 7fb11e000640 1 -- 172.21.15.45:0/2767406651 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1200d3100 con 0x7fb1200b6c00 2024-04-16T16:14:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.700+0000 7fb11e000640 1 --2- 172.21.15.45:0/2767406651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1200b6c00 0x7fb1200cf340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb108009e50 tx=0x7fb10802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fdfd57860f112bd server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.700+0000 7fb11d600640 1 -- 172.21.15.45:0/2767406651 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10802cb00 con 0x7fb1200b6c00 2024-04-16T16:14:03.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.700+0000 7fb11d600640 1 -- 172.21.15.45:0/2767406651 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb10802cca0 con 0x7fb1200b6c00 2024-04-16T16:14:03.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.700+0000 7fb11d600640 1 -- 172.21.15.45:0/2767406651 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb108034610 con 0x7fb1200b6c00 2024-04-16T16:14:03.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.701+0000 7fb125e00640 1 -- 172.21.15.45:0/2767406651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1200b6c00 msgr2=0x7fb1200cf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:03.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.701+0000 7fb125e00640 1 --2- 172.21.15.45:0/2767406651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1200b6c00 0x7fb1200cf340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb108009e50 tx=0x7fb10802bf60 comp rx=0 tx=0).stop 2024-04-16T16:14:03.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.701+0000 7fb125e00640 1 -- 172.21.15.45:0/2767406651 shutdown_connections 2024-04-16T16:14:03.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.701+0000 7fb125e00640 1 --2- 172.21.15.45:0/2767406651 >> v2:172.21.15.103:3300/0 conn(0x7fb1200cf880 0x7fb1200b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.701+0000 7fb125e00640 1 --2- 172.21.15.45:0/2767406651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1200b6c00 0x7fb1200cf340 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.701+0000 7fb125e00640 1 -- 172.21.15.45:0/2767406651 >> 172.21.15.45:0/2767406651 conn(0x7fb120073690 msgr2=0x7fb1200d14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:03.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.701+0000 7fb125e00640 1 -- 172.21.15.45:0/2767406651 shutdown_connections 2024-04-16T16:14:03.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.701+0000 7fb125e00640 1 -- 172.21.15.45:0/2767406651 wait complete. 2024-04-16T16:14:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.703+0000 7fb125e00640 1 Processor -- start 2024-04-16T16:14:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.703+0000 7fb125e00640 1 -- start start 2024-04-16T16:14:03.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.704+0000 7fb125e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1200b6c00 0x7fb12014adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:03.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.704+0000 7fb125e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1200cf880 0x7fb1200b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:03.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.704+0000 7fb125e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb12015edf0 0x7fb1200b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:03.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.704+0000 7fb125e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb1200b7fb0 con 0x7fb12015edf0 2024-04-16T16:14:03.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.704+0000 7fb125e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb1200cfd30 con 0x7fb1200b6c00 2024-04-16T16:14:03.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.704+0000 7fb125e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb1200b8130 con 0x7fb1200cf880 2024-04-16T16:14:03.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.704+0000 7fb11f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb12015edf0 0x7fb1200b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:03.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.705+0000 7fb11f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb12015edf0 0x7fb1200b63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40228/0 (socket says 172.21.15.45:40228) 2024-04-16T16:14:03.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.705+0000 7fb11f400640 1 -- 172.21.15.45:0/4094643390 learned_addr learned my addr 172.21.15.45:0/4094643390 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:03.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.705+0000 7fb11e000640 1 --2- 172.21.15.45:0/4094643390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1200cf880 0x7fb1200b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:03.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.705+0000 7fb11ea00640 1 --2- 172.21.15.45:0/4094643390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1200b6c00 0x7fb12014adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:03.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.705+0000 7fb11f400640 1 -- 172.21.15.45:0/4094643390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1200cf880 msgr2=0x7fb1200b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:03.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.705+0000 7fb11f400640 1 --2- 172.21.15.45:0/4094643390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1200cf880 0x7fb1200b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.705+0000 7fb11f400640 1 -- 172.21.15.45:0/4094643390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1200b6c00 msgr2=0x7fb12014adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:03.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.705+0000 7fb11f400640 1 --2- 172.21.15.45:0/4094643390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1200b6c00 0x7fb12014adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.705+0000 7fb11f400640 1 -- 172.21.15.45:0/4094643390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12014b300 con 0x7fb12015edf0 2024-04-16T16:14:03.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.705+0000 7fb11f400640 1 --2- 172.21.15.45:0/4094643390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb12015edf0 0x7fb1200b63d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb11800e960 tx=0x7fb11800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:03.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.706+0000 7fb117400640 1 -- 172.21.15.45:0/4094643390 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb118017070 con 0x7fb12015edf0 2024-04-16T16:14:03.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.706+0000 7fb117400640 1 -- 172.21.15.45:0/4094643390 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb11800c980 con 0x7fb12015edf0 2024-04-16T16:14:03.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.706+0000 7fb117400640 1 -- 172.21.15.45:0/4094643390 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11800cd70 con 0x7fb12015edf0 2024-04-16T16:14:03.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.706+0000 7fb125e00640 1 -- 172.21.15.45:0/4094643390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1201f59d0 con 0x7fb12015edf0 2024-04-16T16:14:03.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.706+0000 7fb125e00640 1 -- 172.21.15.45:0/4094643390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb12014b8e0 con 0x7fb12015edf0 2024-04-16T16:14:03.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.707+0000 7fb125e00640 1 -- 172.21.15.45:0/4094643390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb120075780 con 0x7fb12015edf0 2024-04-16T16:14:03.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.708+0000 7fb117400640 1 -- 172.21.15.45:0/4094643390 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb11801a070 con 0x7fb12015edf0 2024-04-16T16:14:03.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.709+0000 7fb117400640 1 --2- 172.21.15.45:0/4094643390 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0fc06c8f0 0x7fb0fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:03.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.709+0000 7fb117400640 1 -- 172.21.15.45:0/4094643390 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb11808bb50 con 0x7fb12015edf0 2024-04-16T16:14:03.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.710+0000 7fb11ea00640 1 --2- 172.21.15.45:0/4094643390 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0fc06c8f0 0x7fb0fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:03.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.711+0000 7fb11ea00640 1 --2- 172.21.15.45:0/4094643390 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0fc06c8f0 0x7fb0fc06edb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1200b5b80 tx=0x7fb110009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:03.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.714+0000 7fb117400640 1 -- 172.21.15.45:0/4094643390 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb118059f80 con 0x7fb12015edf0 2024-04-16T16:14:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.924+0000 7fb125e00640 1 -- 172.21.15.45:0/4094643390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb1200b9630 con 0x7fb12015edf0 2024-04-16T16:14:03.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.926+0000 7fb117400640 1 -- 172.21.15.45:0/4094643390 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7fb118059880 con 0x7fb12015edf0 2024-04-16T16:14:03.925 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:03.926 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":21,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:26.918617+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:03.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.928+0000 7fb125e00640 1 -- 172.21.15.45:0/4094643390 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0fc06c8f0 msgr2=0x7fb0fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:03.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.928+0000 7fb125e00640 1 --2- 172.21.15.45:0/4094643390 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0fc06c8f0 0x7fb0fc06edb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1200b5b80 tx=0x7fb110009040 comp rx=0 tx=0).stop 2024-04-16T16:14:03.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.929+0000 7fb125e00640 1 -- 172.21.15.45:0/4094643390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb12015edf0 msgr2=0x7fb1200b63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:03.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.929+0000 7fb125e00640 1 --2- 172.21.15.45:0/4094643390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb12015edf0 0x7fb1200b63d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb11800e960 tx=0x7fb11800ee20 comp rx=0 tx=0).stop 2024-04-16T16:14:03.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.929+0000 7fb125e00640 1 -- 172.21.15.45:0/4094643390 shutdown_connections 2024-04-16T16:14:03.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.929+0000 7fb125e00640 1 --2- 172.21.15.45:0/4094643390 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb0fc06c8f0 0x7fb0fc06edb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.929+0000 7fb125e00640 1 --2- 172.21.15.45:0/4094643390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb12015edf0 0x7fb1200b63d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.929+0000 7fb125e00640 1 --2- 172.21.15.45:0/4094643390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1200cf880 0x7fb1200b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.929+0000 7fb125e00640 1 --2- 172.21.15.45:0/4094643390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1200b6c00 0x7fb12014adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:03.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.929+0000 7fb125e00640 1 -- 172.21.15.45:0/4094643390 >> 172.21.15.45:0/4094643390 conn(0x7fb120073690 msgr2=0x7fb1200b5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:03.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.929+0000 7fb125e00640 1 -- 172.21.15.45:0/4094643390 shutdown_connections 2024-04-16T16:14:03.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:03.929+0000 7fb125e00640 1 -- 172.21.15.45:0/4094643390 wait complete. 2024-04-16T16:14:03.949 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:13:23.600104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:13:24.368141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:14:03.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:14:04.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.180+0000 7fd195e00640 1 Processor -- start 2024-04-16T16:14:04.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.180+0000 7fd195e00640 1 -- start start 2024-04-16T16:14:04.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.182+0000 7fd195e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd190145fe0 0x7fd1900c5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:04.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.182+0000 7fd195e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd190160e90 0x7fd1900cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:04.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.182+0000 7fd195e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1900c5c90 0x7fd1900cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:04.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.182+0000 7fd195e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd1900c6210 con 0x7fd190160e90 2024-04-16T16:14:04.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.182+0000 7fd195e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd1901603b0 con 0x7fd190145fe0 2024-04-16T16:14:04.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.182+0000 7fd195e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd190160690 con 0x7fd1900c5c90 2024-04-16T16:14:04.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.183+0000 7fd18fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd190160e90 0x7fd1900cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:04.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.183+0000 7fd18fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd190160e90 0x7fd1900cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40242/0 (socket says 172.21.15.45:40242) 2024-04-16T16:14:04.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.183+0000 7fd18fe00640 1 -- 172.21.15.45:0/4271770290 learned_addr learned my addr 172.21.15.45:0/4271770290 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:04.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.183+0000 7fd195400640 1 --2- 172.21.15.45:0/4271770290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1900c5c90 0x7fd1900cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:04.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.183+0000 7fd194a00640 1 --2- 172.21.15.45:0/4271770290 >> v2:172.21.15.103:3300/0 conn(0x7fd190145fe0 0x7fd1900c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:04.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.183+0000 7fd18fe00640 1 -- 172.21.15.45:0/4271770290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1900c5c90 msgr2=0x7fd1900cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:04.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.183+0000 7fd18fe00640 1 --2- 172.21.15.45:0/4271770290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1900c5c90 0x7fd1900cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.183+0000 7fd18fe00640 1 -- 172.21.15.45:0/4271770290 >> v2:172.21.15.103:3300/0 conn(0x7fd190145fe0 msgr2=0x7fd1900c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:04.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.183+0000 7fd18fe00640 1 --2- 172.21.15.45:0/4271770290 >> v2:172.21.15.103:3300/0 conn(0x7fd190145fe0 0x7fd1900c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.183+0000 7fd18fe00640 1 -- 172.21.15.45:0/4271770290 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd190160970 con 0x7fd190160e90 2024-04-16T16:14:04.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.184+0000 7fd18fe00640 1 --2- 172.21.15.45:0/4271770290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd190160e90 0x7fd1900cf440 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009870 tx=0x7fd17c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c4ff86efb6a83a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:04.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.185+0000 7fd18f400640 1 -- 172.21.15.45:0/4271770290 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17c0317c0 con 0x7fd190160e90 2024-04-16T16:14:04.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.185+0000 7fd18f400640 1 -- 172.21.15.45:0/4271770290 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd17c037070 con 0x7fd190160e90 2024-04-16T16:14:04.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.185+0000 7fd18f400640 1 -- 172.21.15.45:0/4271770290 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17c031b80 con 0x7fd190160e90 2024-04-16T16:14:04.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.185+0000 7fd195e00640 1 -- 172.21.15.45:0/4271770290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd190160e90 msgr2=0x7fd1900cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:04.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.185+0000 7fd195e00640 1 --2- 172.21.15.45:0/4271770290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd190160e90 0x7fd1900cf440 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009870 tx=0x7fd17c02b860 comp rx=0 tx=0).stop 2024-04-16T16:14:04.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.186+0000 7fd195e00640 1 -- 172.21.15.45:0/4271770290 shutdown_connections 2024-04-16T16:14:04.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.186+0000 7fd195e00640 1 --2- 172.21.15.45:0/4271770290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1900c5c90 0x7fd1900cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.186+0000 7fd195e00640 1 --2- 172.21.15.45:0/4271770290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd190160e90 0x7fd1900cf440 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.186+0000 7fd195e00640 1 --2- 172.21.15.45:0/4271770290 >> v2:172.21.15.103:3300/0 conn(0x7fd190145fe0 0x7fd1900c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.186+0000 7fd195e00640 1 -- 172.21.15.45:0/4271770290 >> 172.21.15.45:0/4271770290 conn(0x7fd190067e00 msgr2=0x7fd19006b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:04.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.186+0000 7fd195e00640 1 -- 172.21.15.45:0/4271770290 shutdown_connections 2024-04-16T16:14:04.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.186+0000 7fd195e00640 1 -- 172.21.15.45:0/4271770290 wait complete. 2024-04-16T16:14:04.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.188+0000 7fd195e00640 1 Processor -- start 2024-04-16T16:14:04.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.188+0000 7fd195e00640 1 -- start start 2024-04-16T16:14:04.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.189+0000 7fd195e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1900c5c90 0x7fd19013f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:04.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.189+0000 7fd195e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd190145fe0 0x7fd19013d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:04.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.189+0000 7fd195e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd190160e90 0x7fd19013d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:04.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.189+0000 7fd195e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd190146540 con 0x7fd1900c5c90 2024-04-16T16:14:04.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.189+0000 7fd195e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1901463c0 con 0x7fd190160e90 2024-04-16T16:14:04.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.189+0000 7fd195e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd19006dfb0 con 0x7fd190145fe0 2024-04-16T16:14:04.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.189+0000 7fd194a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1900c5c90 0x7fd19013f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:04.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.189+0000 7fd194a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1900c5c90 0x7fd19013f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40254/0 (socket says 172.21.15.45:40254) 2024-04-16T16:14:04.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.189+0000 7fd194a00640 1 -- 172.21.15.45:0/4261768692 learned_addr learned my addr 172.21.15.45:0/4261768692 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:04.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.189+0000 7fd194a00640 1 -- 172.21.15.45:0/4261768692 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd190145fe0 msgr2=0x7fd19013d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:04.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.189+0000 7fd194a00640 1 --2- 172.21.15.45:0/4261768692 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd190145fe0 0x7fd19013d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.189+0000 7fd194a00640 1 -- 172.21.15.45:0/4261768692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd190160e90 msgr2=0x7fd19013d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:04.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.189+0000 7fd194a00640 1 --2- 172.21.15.45:0/4261768692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd190160e90 0x7fd19013d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.189+0000 7fd194a00640 1 -- 172.21.15.45:0/4261768692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd19013f220 con 0x7fd1900c5c90 2024-04-16T16:14:04.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.190+0000 7fd194a00640 1 --2- 172.21.15.45:0/4261768692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1900c5c90 0x7fd19013f9a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd18400d800 tx=0x7fd18400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:04.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.190+0000 7fd18cc00640 1 -- 172.21.15.45:0/4261768692 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18400b6c0 con 0x7fd1900c5c90 2024-04-16T16:14:04.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.190+0000 7fd18cc00640 1 -- 172.21.15.45:0/4261768692 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd184019070 con 0x7fd1900c5c90 2024-04-16T16:14:04.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.190+0000 7fd18cc00640 1 -- 172.21.15.45:0/4261768692 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18400ba80 con 0x7fd1900c5c90 2024-04-16T16:14:04.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.190+0000 7fd195e00640 1 -- 172.21.15.45:0/4261768692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd19013c4b0 con 0x7fd1900c5c90 2024-04-16T16:14:04.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.190+0000 7fd195e00640 1 -- 172.21.15.45:0/4261768692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd19013ca70 con 0x7fd1900c5c90 2024-04-16T16:14:04.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.192+0000 7fd195e00640 1 -- 172.21.15.45:0/4261768692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd190075780 con 0x7fd1900c5c90 2024-04-16T16:14:04.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.193+0000 7fd18cc00640 1 -- 172.21.15.45:0/4261768692 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd18400bc20 con 0x7fd1900c5c90 2024-04-16T16:14:04.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.193+0000 7fd18cc00640 1 --2- 172.21.15.45:0/4261768692 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd17006c8f0 0x7fd17006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:04.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.194+0000 7fd18cc00640 1 -- 172.21.15.45:0/4261768692 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd18408bfa0 con 0x7fd1900c5c90 2024-04-16T16:14:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.194+0000 7fd18fe00640 1 --2- 172.21.15.45:0/4261768692 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd17006c8f0 0x7fd17006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:04.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.196+0000 7fd18fe00640 1 --2- 172.21.15.45:0/4261768692 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd17006c8f0 0x7fd17006edb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd17c036000 tx=0x7fd17c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:04.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.198+0000 7fd18cc00640 1 -- 172.21.15.45:0/4261768692 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd18405a3d0 con 0x7fd1900c5c90 2024-04-16T16:14:04.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.401+0000 7fd195e00640 1 -- 172.21.15.45:0/4261768692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd1901404f0 con 0x7fd1900c5c90 2024-04-16T16:14:04.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.402+0000 7fd18cc00640 1 -- 172.21.15.45:0/4261768692 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fd184059cd0 con 0x7fd1900c5c90 2024-04-16T16:14:04.402 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:14:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.405+0000 7fd195e00640 1 -- 172.21.15.45:0/4261768692 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd17006c8f0 msgr2=0x7fd17006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.405+0000 7fd195e00640 1 --2- 172.21.15.45:0/4261768692 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd17006c8f0 0x7fd17006edb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd17c036000 tx=0x7fd17c030000 comp rx=0 tx=0).stop 2024-04-16T16:14:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.405+0000 7fd195e00640 1 -- 172.21.15.45:0/4261768692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1900c5c90 msgr2=0x7fd19013f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.405+0000 7fd195e00640 1 --2- 172.21.15.45:0/4261768692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1900c5c90 0x7fd19013f9a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd18400d800 tx=0x7fd18400dcc0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.405+0000 7fd195e00640 1 -- 172.21.15.45:0/4261768692 shutdown_connections 2024-04-16T16:14:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.405+0000 7fd195e00640 1 --2- 172.21.15.45:0/4261768692 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd17006c8f0 0x7fd17006edb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.405+0000 7fd195e00640 1 --2- 172.21.15.45:0/4261768692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd190160e90 0x7fd19013d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.405+0000 7fd195e00640 1 --2- 172.21.15.45:0/4261768692 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd190145fe0 0x7fd19013d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.405+0000 7fd195e00640 1 --2- 172.21.15.45:0/4261768692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1900c5c90 0x7fd19013f9a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.405+0000 7fd195e00640 1 -- 172.21.15.45:0/4261768692 >> 172.21.15.45:0/4261768692 conn(0x7fd190067e00 msgr2=0x7fd1900b4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.406+0000 7fd195e00640 1 -- 172.21.15.45:0/4261768692 shutdown_connections 2024-04-16T16:14:04.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.406+0000 7fd195e00640 1 -- 172.21.15.45:0/4261768692 wait complete. 2024-04-16T16:14:04.427 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:14:04.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.656+0000 7f4625a00640 1 Processor -- start 2024-04-16T16:14:04.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.656+0000 7f4625a00640 1 -- start start 2024-04-16T16:14:04.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.657+0000 7f4625a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46201491c0 0x7f4620166840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:04.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.657+0000 7f4625a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f46200b58f0 0x7f46200b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:04.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.658+0000 7f4625a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46201495a0 con 0x7f4620163cd0 2024-04-16T16:14:04.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.658+0000 7f4625a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f46200cd810 con 0x7f46200b58f0 2024-04-16T16:14:04.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.658+0000 7f4625a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f462006e090 con 0x7f46201491c0 2024-04-16T16:14:04.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.658+0000 7f461fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46201491c0 0x7f4620166840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:04.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.659+0000 7f461fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46201491c0 0x7f4620166840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:42250/0 (socket says 172.21.15.45:42250) 2024-04-16T16:14:04.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.659+0000 7f4617e00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4620163cd0 0x7f46200cd100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:56190/0 (socket says 172.21.15.45:56190) 2024-04-16T16:14:04.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.659+0000 7f461fe00640 1 -- 172.21.15.45:0/872374186 learned_addr learned my addr 172.21.15.45:0/872374186 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:04.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.659+0000 7f4625000640 1 --2- 172.21.15.45:0/872374186 >> v2:172.21.15.103:3300/0 conn(0x7f46200b58f0 0x7f46200b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:04.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.659+0000 7f461fe00640 1 -- 172.21.15.45:0/872374186 >> v2:172.21.15.103:3300/0 conn(0x7f46200b58f0 msgr2=0x7f46200b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:04.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.659+0000 7f461fe00640 1 --2- 172.21.15.45:0/872374186 >> v2:172.21.15.103:3300/0 conn(0x7f46200b58f0 0x7f46200b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.659+0000 7f461fe00640 1 -- 172.21.15.45:0/872374186 >> v1:172.21.15.103:6789/0 conn(0x7f4620163cd0 legacy=0x7f46200cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:04.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.659+0000 7f461fe00640 1 -- 172.21.15.45:0/872374186 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4620166e40 con 0x7f46201491c0 2024-04-16T16:14:04.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.661+0000 7f461fe00640 1 --2- 172.21.15.45:0/872374186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46201491c0 0x7f4620166840 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4610009870 tx=0x7f461002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69cbb7f65407c3df server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:04.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.662+0000 7f461f400640 1 -- 172.21.15.45:0/872374186 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4610035020 con 0x7f46201491c0 2024-04-16T16:14:04.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.662+0000 7f461f400640 1 -- 172.21.15.45:0/872374186 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4610031640 con 0x7f46201491c0 2024-04-16T16:14:04.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.662+0000 7f461f400640 1 -- 172.21.15.45:0/872374186 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461002d040 con 0x7f46201491c0 2024-04-16T16:14:04.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.662+0000 7f4625a00640 1 -- 172.21.15.45:0/872374186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46201491c0 msgr2=0x7f4620166840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:04.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.662+0000 7f4625a00640 1 --2- 172.21.15.45:0/872374186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46201491c0 0x7f4620166840 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4610009870 tx=0x7f461002b8f0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.663+0000 7f4625a00640 1 -- 172.21.15.45:0/872374186 shutdown_connections 2024-04-16T16:14:04.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.663+0000 7f4625a00640 1 --2- 172.21.15.45:0/872374186 >> v2:172.21.15.103:3300/0 conn(0x7f46200b58f0 0x7f46200b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.663+0000 7f4625a00640 1 --2- 172.21.15.45:0/872374186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46201491c0 0x7f4620166840 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.663+0000 7f4625a00640 1 -- 172.21.15.45:0/872374186 >> 172.21.15.45:0/872374186 conn(0x7f462006b930 msgr2=0x7f462006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:04.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.663+0000 7f4625a00640 1 -- 172.21.15.45:0/872374186 shutdown_connections 2024-04-16T16:14:04.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.663+0000 7f4625a00640 1 -- 172.21.15.45:0/872374186 wait complete. 2024-04-16T16:14:04.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.665+0000 7f4625a00640 1 Processor -- start 2024-04-16T16:14:04.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.665+0000 7f4625a00640 1 -- start start 2024-04-16T16:14:04.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.666+0000 7f4625a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46200b58f0 0x7f462013e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:04.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.667+0000 7f4625a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46201491c0 0x7f462014bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:04.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.667+0000 7f4625a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4620163cd0 0x7f462014c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:04.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.667+0000 7f4625a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4620164110 con 0x7f46200b58f0 2024-04-16T16:14:04.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.667+0000 7f4625a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4620167400 con 0x7f4620163cd0 2024-04-16T16:14:04.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.667+0000 7f4625a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f46201676e0 con 0x7f46201491c0 2024-04-16T16:14:04.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.667+0000 7f461fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46200b58f0 0x7f462013e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:04.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.667+0000 7f461fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46200b58f0 0x7f462013e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40262/0 (socket says 172.21.15.45:40262) 2024-04-16T16:14:04.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.667+0000 7f461fe00640 1 -- 172.21.15.45:0/759195415 learned_addr learned my addr 172.21.15.45:0/759195415 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:04.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.667+0000 7f4617e00640 1 --2- 172.21.15.45:0/759195415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46201491c0 0x7f462014bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:04.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.667+0000 7f4625000640 1 --2- 172.21.15.45:0/759195415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4620163cd0 0x7f462014c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:04.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.668+0000 7f461fe00640 1 -- 172.21.15.45:0/759195415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46201491c0 msgr2=0x7f462014bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:04.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.668+0000 7f461fe00640 1 --2- 172.21.15.45:0/759195415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46201491c0 0x7f462014bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.668+0000 7f461fe00640 1 -- 172.21.15.45:0/759195415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4620163cd0 msgr2=0x7f462014c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:04.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.668+0000 7f461fe00640 1 --2- 172.21.15.45:0/759195415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4620163cd0 0x7f462014c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.668+0000 7f461fe00640 1 -- 172.21.15.45:0/759195415 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f462014adc0 con 0x7f46200b58f0 2024-04-16T16:14:04.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.668+0000 7f4617e00640 1 --2- 172.21.15.45:0/759195415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46201491c0 0x7f462014bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:14:04.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.668+0000 7f461fe00640 1 --2- 172.21.15.45:0/759195415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46200b58f0 0x7f462013e210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4610000c00 tx=0x7f46100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:04.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.669+0000 7f461cc00640 1 -- 172.21.15.45:0/759195415 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4610040070 con 0x7f46200b58f0 2024-04-16T16:14:04.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.669+0000 7f461cc00640 1 -- 172.21.15.45:0/759195415 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46100399f0 con 0x7f46200b58f0 2024-04-16T16:14:04.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.669+0000 7f461cc00640 1 -- 172.21.15.45:0/759195415 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4610035040 con 0x7f46200b58f0 2024-04-16T16:14:04.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.669+0000 7f4625a00640 1 -- 172.21.15.45:0/759195415 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f462014b0a0 con 0x7f46200b58f0 2024-04-16T16:14:04.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.669+0000 7f4625a00640 1 -- 172.21.15.45:0/759195415 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f462014b660 con 0x7f46200b58f0 2024-04-16T16:14:04.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.670+0000 7f4625a00640 1 -- 172.21.15.45:0/759195415 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f462014b860 con 0x7f46200b58f0 2024-04-16T16:14:04.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.671+0000 7f461cc00640 1 -- 172.21.15.45:0/759195415 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f461004b060 con 0x7f46200b58f0 2024-04-16T16:14:04.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.672+0000 7f461cc00640 1 --2- 172.21.15.45:0/759195415 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f45fc06c920 0x7f45fc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:04.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.672+0000 7f461cc00640 1 -- 172.21.15.45:0/759195415 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f46100ad400 con 0x7f46200b58f0 2024-04-16T16:14:04.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.673+0000 7f4617e00640 1 --2- 172.21.15.45:0/759195415 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f45fc06c920 0x7f45fc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:04.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.674+0000 7f4617e00640 1 --2- 172.21.15.45:0/759195415 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f45fc06c920 0x7f45fc06ede0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4620146ee0 tx=0x7f4608007d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:04.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.677+0000 7f461cc00640 1 -- 172.21.15.45:0/759195415 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f462014b860 con 0x7f46200b58f0 2024-04-16T16:14:04.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.878+0000 7f4625a00640 1 -- 172.21.15.45:0/759195415 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4620075780 con 0x7f46200b58f0 2024-04-16T16:14:04.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.879+0000 7f461cc00640 1 -- 172.21.15.45:0/759195415 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f461007b790 con 0x7f46200b58f0 2024-04-16T16:14:04.878 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:14:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.881+0000 7f4625a00640 1 -- 172.21.15.45:0/759195415 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f45fc06c920 msgr2=0x7f45fc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.881+0000 7f4625a00640 1 --2- 172.21.15.45:0/759195415 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f45fc06c920 0x7f45fc06ede0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4620146ee0 tx=0x7f4608007d80 comp rx=0 tx=0).stop 2024-04-16T16:14:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.881+0000 7f4625a00640 1 -- 172.21.15.45:0/759195415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46200b58f0 msgr2=0x7f462013e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.881+0000 7f4625a00640 1 --2- 172.21.15.45:0/759195415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46200b58f0 0x7f462013e210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4610000c00 tx=0x7f46100046c0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.882+0000 7f4625a00640 1 -- 172.21.15.45:0/759195415 shutdown_connections 2024-04-16T16:14:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.882+0000 7f4625a00640 1 --2- 172.21.15.45:0/759195415 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f45fc06c920 0x7f45fc06ede0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.882+0000 7f4625a00640 1 --2- 172.21.15.45:0/759195415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4620163cd0 0x7f462014c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.882+0000 7f4625a00640 1 --2- 172.21.15.45:0/759195415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46201491c0 0x7f462014bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.882+0000 7f4625a00640 1 --2- 172.21.15.45:0/759195415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46200b58f0 0x7f462013e210 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.882+0000 7f4625a00640 1 -- 172.21.15.45:0/759195415 >> 172.21.15.45:0/759195415 conn(0x7f462006b930 msgr2=0x7f4620146f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.882+0000 7f4625a00640 1 -- 172.21.15.45:0/759195415 shutdown_connections 2024-04-16T16:14:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:04.882+0000 7f4625a00640 1 -- 172.21.15.45:0/759195415 wait complete. 2024-04-16T16:14:04.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:14:05.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.127+0000 7fc175200640 1 Processor -- start 2024-04-16T16:14:05.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.128+0000 7fc175200640 1 -- start start 2024-04-16T16:14:05.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.129+0000 7fc175200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc1701461b0 0x7fc170160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:05.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.129+0000 7fc175200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1700b58f0 0x7fc1700b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:05.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.129+0000 7fc175200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc170146590 con 0x7fc1700cf440 2024-04-16T16:14:05.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.130+0000 7fc175200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc17006dfd0 con 0x7fc1700b58f0 2024-04-16T16:14:05.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.130+0000 7fc175200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc1700b6290 con 0x7fc1701461b0 2024-04-16T16:14:05.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.130+0000 7fc16ea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1700b58f0 0x7fc1700b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:05.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.130+0000 7fc16ea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1700b58f0 0x7fc1700b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40266/0 (socket says 172.21.15.45:40266) 2024-04-16T16:14:05.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.130+0000 7fc16ea00640 1 -- 172.21.15.45:0/3192138253 learned_addr learned my addr 172.21.15.45:0/3192138253 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.130+0000 7fc16ea00640 1 -- 172.21.15.45:0/3192138253 >> v2:172.21.15.103:3300/0 conn(0x7fc1701461b0 msgr2=0x7fc170160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.130+0000 7fc16f400640 1 --2- 172.21.15.45:0/3192138253 >> v2:172.21.15.103:3300/0 conn(0x7fc1701461b0 0x7fc170160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.130+0000 7fc16ea00640 1 --2- 172.21.15.45:0/3192138253 >> v2:172.21.15.103:3300/0 conn(0x7fc1701461b0 0x7fc170160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.130+0000 7fc16ea00640 1 -- 172.21.15.45:0/3192138253 >> v1:172.21.15.103:6789/0 conn(0x7fc1700cf440 legacy=0x7fc1700cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:05.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.130+0000 7fc16ea00640 1 -- 172.21.15.45:0/3192138253 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1700c5750 con 0x7fc1700b58f0 2024-04-16T16:14:05.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.131+0000 7fc16ea00640 1 --2- 172.21.15.45:0/3192138253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1700b58f0 0x7fc1700b5d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc1700b6580 tx=0x7fc15c02b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6962bc957d0cb80f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:05.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.132+0000 7fc16e000640 1 -- 172.21.15.45:0/3192138253 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15c035020 con 0x7fc1700b58f0 2024-04-16T16:14:05.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.132+0000 7fc16e000640 1 -- 172.21.15.45:0/3192138253 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc15c031890 con 0x7fc1700b58f0 2024-04-16T16:14:05.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.132+0000 7fc16e000640 1 -- 172.21.15.45:0/3192138253 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15c031c60 con 0x7fc1700b58f0 2024-04-16T16:14:05.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.132+0000 7fc175200640 1 -- 172.21.15.45:0/3192138253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1700b58f0 msgr2=0x7fc1700b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:05.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.132+0000 7fc175200640 1 --2- 172.21.15.45:0/3192138253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1700b58f0 0x7fc1700b5d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc1700b6580 tx=0x7fc15c02b930 comp rx=0 tx=0).stop 2024-04-16T16:14:05.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.133+0000 7fc175200640 1 -- 172.21.15.45:0/3192138253 shutdown_connections 2024-04-16T16:14:05.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.133+0000 7fc175200640 1 --2- 172.21.15.45:0/3192138253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1700b58f0 0x7fc1700b5d50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.133+0000 7fc175200640 1 --2- 172.21.15.45:0/3192138253 >> v2:172.21.15.103:3300/0 conn(0x7fc1701461b0 0x7fc170160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.133+0000 7fc175200640 1 -- 172.21.15.45:0/3192138253 >> 172.21.15.45:0/3192138253 conn(0x7fc17006b930 msgr2=0x7fc17006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:05.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.133+0000 7fc175200640 1 -- 172.21.15.45:0/3192138253 shutdown_connections 2024-04-16T16:14:05.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.133+0000 7fc175200640 1 -- 172.21.15.45:0/3192138253 wait complete. 2024-04-16T16:14:05.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.134+0000 7fc175200640 1 Processor -- start 2024-04-16T16:14:05.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.135+0000 7fc175200640 1 -- start start 2024-04-16T16:14:05.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.135+0000 7fc175200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1700b58f0 0x7fc170145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:05.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.136+0000 7fc175200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1700cf440 0x7fc170142f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:05.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.136+0000 7fc175200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1701461b0 0x7fc1701f4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:05.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.136+0000 7fc175200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc1700c59f0 con 0x7fc1701461b0 2024-04-16T16:14:05.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.136+0000 7fc175200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc170161230 con 0x7fc1700b58f0 2024-04-16T16:14:05.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.136+0000 7fc175200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc1700c5cd0 con 0x7fc1700cf440 2024-04-16T16:14:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.136+0000 7fc16fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1701461b0 0x7fc1701f4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.136+0000 7fc16fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1701461b0 0x7fc1701f4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40272/0 (socket says 172.21.15.45:40272) 2024-04-16T16:14:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.136+0000 7fc16fe00640 1 -- 172.21.15.45:0/3055834070 learned_addr learned my addr 172.21.15.45:0/3055834070 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.136+0000 7fc16f400640 1 --2- 172.21.15.45:0/3055834070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1700b58f0 0x7fc170145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.136+0000 7fc16ea00640 1 --2- 172.21.15.45:0/3055834070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1700cf440 0x7fc170142f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:05.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.137+0000 7fc16fe00640 1 -- 172.21.15.45:0/3055834070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1700cf440 msgr2=0x7fc170142f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:05.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.137+0000 7fc16fe00640 1 --2- 172.21.15.45:0/3055834070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1700cf440 0x7fc170142f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.137+0000 7fc16fe00640 1 -- 172.21.15.45:0/3055834070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1700b58f0 msgr2=0x7fc170145370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:05.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.137+0000 7fc16fe00640 1 --2- 172.21.15.45:0/3055834070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1700b58f0 0x7fc170145370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.137+0000 7fc16fe00640 1 -- 172.21.15.45:0/3055834070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc170143490 con 0x7fc1701461b0 2024-04-16T16:14:05.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.137+0000 7fc16fe00640 1 --2- 172.21.15.45:0/3055834070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1701461b0 0x7fc1701f4480 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc16800c780 tx=0x7fc16800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:05.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.137+0000 7fc167400640 1 -- 172.21.15.45:0/3055834070 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc168004b60 con 0x7fc1701461b0 2024-04-16T16:14:05.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.137+0000 7fc167400640 1 -- 172.21.15.45:0/3055834070 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc168004d00 con 0x7fc1701461b0 2024-04-16T16:14:05.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.137+0000 7fc167400640 1 -- 172.21.15.45:0/3055834070 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16800aa90 con 0x7fc1701461b0 2024-04-16T16:14:05.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.137+0000 7fc175200640 1 -- 172.21.15.45:0/3055834070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc170143760 con 0x7fc1701461b0 2024-04-16T16:14:05.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.137+0000 7fc175200640 1 -- 172.21.15.45:0/3055834070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc170143d20 con 0x7fc1701461b0 2024-04-16T16:14:05.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.139+0000 7fc175200640 1 -- 172.21.15.45:0/3055834070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1701e60c0 con 0x7fc1701461b0 2024-04-16T16:14:05.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.140+0000 7fc167400640 1 -- 172.21.15.45:0/3055834070 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc16800ac30 con 0x7fc1701461b0 2024-04-16T16:14:05.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.141+0000 7fc167400640 1 --2- 172.21.15.45:0/3055834070 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc15006c8f0 0x7fc15006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:05.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.141+0000 7fc167400640 1 -- 172.21.15.45:0/3055834070 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc16808bd70 con 0x7fc1701461b0 2024-04-16T16:14:05.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.141+0000 7fc16f400640 1 --2- 172.21.15.45:0/3055834070 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc15006c8f0 0x7fc15006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:05.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.142+0000 7fc16f400640 1 --2- 172.21.15.45:0/3055834070 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc15006c8f0 0x7fc15006edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc16000aa20 tx=0x7fc160008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:05.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.145+0000 7fc167400640 1 -- 172.21.15.45:0/3055834070 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc168020070 con 0x7fc1701461b0 2024-04-16T16:14:05.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.345+0000 7fc175200640 1 -- 172.21.15.45:0/3055834070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc170075780 con 0x7fc1701461b0 2024-04-16T16:14:05.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.346+0000 7fc167400640 1 -- 172.21.15.45:0/3055834070 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fc16805a190 con 0x7fc1701461b0 2024-04-16T16:14:05.345 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:14:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.348+0000 7fc175200640 1 -- 172.21.15.45:0/3055834070 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc15006c8f0 msgr2=0x7fc15006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.348+0000 7fc175200640 1 --2- 172.21.15.45:0/3055834070 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc15006c8f0 0x7fc15006edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc16000aa20 tx=0x7fc160008040 comp rx=0 tx=0).stop 2024-04-16T16:14:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.348+0000 7fc175200640 1 -- 172.21.15.45:0/3055834070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1701461b0 msgr2=0x7fc1701f4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.348+0000 7fc175200640 1 --2- 172.21.15.45:0/3055834070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1701461b0 0x7fc1701f4480 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc16800c780 tx=0x7fc16800cc40 comp rx=0 tx=0).stop 2024-04-16T16:14:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.349+0000 7fc175200640 1 -- 172.21.15.45:0/3055834070 shutdown_connections 2024-04-16T16:14:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.349+0000 7fc175200640 1 --2- 172.21.15.45:0/3055834070 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc15006c8f0 0x7fc15006edb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.349+0000 7fc175200640 1 --2- 172.21.15.45:0/3055834070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1701461b0 0x7fc1701f4480 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.349+0000 7fc175200640 1 --2- 172.21.15.45:0/3055834070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1700cf440 0x7fc170142f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.349+0000 7fc175200640 1 --2- 172.21.15.45:0/3055834070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1700b58f0 0x7fc170145370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.349+0000 7fc175200640 1 -- 172.21.15.45:0/3055834070 >> 172.21.15.45:0/3055834070 conn(0x7fc17006b930 msgr2=0x7fc1700b15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.349+0000 7fc175200640 1 -- 172.21.15.45:0/3055834070 shutdown_connections 2024-04-16T16:14:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.349+0000 7fc175200640 1 -- 172.21.15.45:0/3055834070 wait complete. 2024-04-16T16:14:05.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:05.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.564+0000 7f80d1000640 1 Processor -- start 2024-04-16T16:14:05.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.564+0000 7f80d1000640 1 -- start start 2024-04-16T16:14:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.566+0000 7f80d1000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80cc162270 0x7f80cc0d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.566+0000 7f80d1000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f80cc0d52d0 0x7f80cc06ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.566+0000 7f80d1000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80cc0d3080 con 0x7f80cc161e90 2024-04-16T16:14:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.566+0000 7f80d1000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f80cc0d2e30 con 0x7f80cc0d52d0 2024-04-16T16:14:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.566+0000 7f80d1000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f80cc0d3360 con 0x7f80cc162270 2024-04-16T16:14:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.566+0000 7f80c2a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80cc162270 0x7f80cc0d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.567+0000 7f80cbe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f80cc0d52d0 0x7f80cc06ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.567+0000 7f80cb400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f80cc161e90 0x7f80cc0cf7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:56216/0 (socket says 172.21.15.45:56216) 2024-04-16T16:14:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.567+0000 7f80cb400640 1 -- 172.21.15.45:0/818990865 learned_addr learned my addr 172.21.15.45:0/818990865 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.567+0000 7f80c2a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80cc162270 0x7f80cc0d3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:42270/0 (socket says 172.21.15.45:42270) 2024-04-16T16:14:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.567+0000 7f80caa00640 1 -- 172.21.15.45:0/818990865 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3887219409 0 0) 0x7f80cc0d3080 con 0x7f80cc161e90 2024-04-16T16:14:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.567+0000 7f80caa00640 1 -- 172.21.15.45:0/818990865 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80b0003520 con 0x7f80cc161e90 2024-04-16T16:14:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.567+0000 7f80cbe00640 1 -- 172.21.15.45:0/818990865 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80cc162270 msgr2=0x7f80cc0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.567+0000 7f80cbe00640 1 --2- 172.21.15.45:0/818990865 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80cc162270 0x7f80cc0d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.567+0000 7f80cbe00640 1 -- 172.21.15.45:0/818990865 >> v1:172.21.15.103:6789/0 conn(0x7f80cc161e90 legacy=0x7f80cc0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:05.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.567+0000 7f80cbe00640 1 -- 172.21.15.45:0/818990865 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80cc0d3640 con 0x7f80cc0d52d0 2024-04-16T16:14:05.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.569+0000 7f80cbe00640 1 --2- 172.21.15.45:0/818990865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80cc0d52d0 0x7f80cc06ce40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f80c40097a0 tx=0x7f80c402b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf11c106d0eb3234 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:05.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.570+0000 7f80caa00640 1 -- 172.21.15.45:0/818990865 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c4034070 con 0x7f80cc0d52d0 2024-04-16T16:14:05.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.570+0000 7f80caa00640 1 -- 172.21.15.45:0/818990865 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80c4030800 con 0x7f80cc0d52d0 2024-04-16T16:14:05.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.570+0000 7f80caa00640 1 -- 172.21.15.45:0/818990865 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c4030bd0 con 0x7f80cc0d52d0 2024-04-16T16:14:05.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.570+0000 7f80d1000640 1 -- 172.21.15.45:0/818990865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80cc0d52d0 msgr2=0x7f80cc06ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:05.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.570+0000 7f80d1000640 1 --2- 172.21.15.45:0/818990865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80cc0d52d0 0x7f80cc06ce40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f80c40097a0 tx=0x7f80c402b940 comp rx=0 tx=0).stop 2024-04-16T16:14:05.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.570+0000 7f80d1000640 1 -- 172.21.15.45:0/818990865 shutdown_connections 2024-04-16T16:14:05.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.570+0000 7f80d1000640 1 --2- 172.21.15.45:0/818990865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80cc0d52d0 0x7f80cc06ce40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.570+0000 7f80d1000640 1 --2- 172.21.15.45:0/818990865 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80cc162270 0x7f80cc0d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.570+0000 7f80d1000640 1 -- 172.21.15.45:0/818990865 >> 172.21.15.45:0/818990865 conn(0x7f80cc0d1940 msgr2=0x7f80cc0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:05.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.571+0000 7f80d1000640 1 -- 172.21.15.45:0/818990865 shutdown_connections 2024-04-16T16:14:05.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.571+0000 7f80d1000640 1 -- 172.21.15.45:0/818990865 wait complete. 2024-04-16T16:14:05.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.572+0000 7f80d1000640 1 Processor -- start 2024-04-16T16:14:05.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.572+0000 7f80d1000640 1 -- start start 2024-04-16T16:14:05.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.573+0000 7f80d1000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc0d52d0 0x7f80cc150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:05.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.573+0000 7f80d1000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80cc161e90 0x7f80cc0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:05.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.573+0000 7f80d1000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80cc162270 0x7f80cc0b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:05.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.573+0000 7f80d1000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc0d4500 con 0x7f80cc0d52d0 2024-04-16T16:14:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.573+0000 7f80d1000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc0d4380 con 0x7f80cc162270 2024-04-16T16:14:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.573+0000 7f80d1000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f80cc0d4680 con 0x7f80cc161e90 2024-04-16T16:14:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.573+0000 7f80cb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc0d52d0 0x7f80cc150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.573+0000 7f80cb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc0d52d0 0x7f80cc150bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40276/0 (socket says 172.21.15.45:40276) 2024-04-16T16:14:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.573+0000 7f80cb400640 1 -- 172.21.15.45:0/3961333066 learned_addr learned my addr 172.21.15.45:0/3961333066 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.574+0000 7f80cbe00640 1 --2- 172.21.15.45:0/3961333066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80cc162270 0x7f80cc0b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.574+0000 7f80c2a00640 1 --2- 172.21.15.45:0/3961333066 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80cc161e90 0x7f80cc0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.574+0000 7f80cb400640 1 -- 172.21.15.45:0/3961333066 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80cc161e90 msgr2=0x7f80cc0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.574+0000 7f80cb400640 1 --2- 172.21.15.45:0/3961333066 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80cc161e90 0x7f80cc0b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.574+0000 7f80cb400640 1 -- 172.21.15.45:0/3961333066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80cc162270 msgr2=0x7f80cc0b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.574+0000 7f80cb400640 1 --2- 172.21.15.45:0/3961333066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80cc162270 0x7f80cc0b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.574+0000 7f80cb400640 1 -- 172.21.15.45:0/3961333066 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80cc151660 con 0x7f80cc0d52d0 2024-04-16T16:14:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.574+0000 7f80cb400640 1 --2- 172.21.15.45:0/3961333066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc0d52d0 0x7f80cc150bc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f80bc00cb70 tx=0x7f80bc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.574+0000 7f80c3e00640 1 -- 172.21.15.45:0/3961333066 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80bc017070 con 0x7f80cc0d52d0 2024-04-16T16:14:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.574+0000 7f80c3e00640 1 -- 172.21.15.45:0/3961333066 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80bc00ad40 con 0x7f80cc0d52d0 2024-04-16T16:14:05.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.574+0000 7f80d1000640 1 -- 172.21.15.45:0/3961333066 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80cc151940 con 0x7f80cc0d52d0 2024-04-16T16:14:05.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.575+0000 7f80c3e00640 1 -- 172.21.15.45:0/3961333066 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80bc012620 con 0x7f80cc0d52d0 2024-04-16T16:14:05.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.575+0000 7f80d1000640 1 -- 172.21.15.45:0/3961333066 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80cc151f00 con 0x7f80cc0d52d0 2024-04-16T16:14:05.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.576+0000 7f80d1000640 1 -- 172.21.15.45:0/3961333066 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80cc075780 con 0x7f80cc0d52d0 2024-04-16T16:14:05.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.577+0000 7f80c3e00640 1 -- 172.21.15.45:0/3961333066 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f80bc01a070 con 0x7f80cc0d52d0 2024-04-16T16:14:05.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.578+0000 7f80c3e00640 1 --2- 172.21.15.45:0/3961333066 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f80a806c8f0 0x7f80a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:05.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.578+0000 7f80c3e00640 1 -- 172.21.15.45:0/3961333066 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f80bc08b990 con 0x7f80cc0d52d0 2024-04-16T16:14:05.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.578+0000 7f80c2a00640 1 --2- 172.21.15.45:0/3961333066 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f80a806c8f0 0x7f80a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:05.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.580+0000 7f80c2a00640 1 --2- 172.21.15.45:0/3961333066 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f80a806c8f0 0x7f80a806edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f80cc0cac80 tx=0x7f80b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:05.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.582+0000 7f80c3e00640 1 -- 172.21.15.45:0/3961333066 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f80bc059dc0 con 0x7f80cc0d52d0 2024-04-16T16:14:05.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.778+0000 7f80d1000640 1 -- 172.21.15.45:0/3961333066 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f80cc1520d0 con 0x7f80cc0d52d0 2024-04-16T16:14:05.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.779+0000 7f80c3e00640 1 -- 172.21.15.45:0/3961333066 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f80bc0596c0 con 0x7f80cc0d52d0 2024-04-16T16:14:05.779 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:05.779 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":21,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:26.918617+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:05.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.782+0000 7f80d1000640 1 -- 172.21.15.45:0/3961333066 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f80a806c8f0 msgr2=0x7f80a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:05.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.782+0000 7f80d1000640 1 --2- 172.21.15.45:0/3961333066 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f80a806c8f0 0x7f80a806edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f80cc0cac80 tx=0x7f80b4008040 comp rx=0 tx=0).stop 2024-04-16T16:14:05.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.782+0000 7f80d1000640 1 -- 172.21.15.45:0/3961333066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc0d52d0 msgr2=0x7f80cc150bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:05.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.782+0000 7f80d1000640 1 --2- 172.21.15.45:0/3961333066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc0d52d0 0x7f80cc150bc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f80bc00cb70 tx=0x7f80bc00a430 comp rx=0 tx=0).stop 2024-04-16T16:14:05.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.782+0000 7f80d1000640 1 -- 172.21.15.45:0/3961333066 shutdown_connections 2024-04-16T16:14:05.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.782+0000 7f80d1000640 1 --2- 172.21.15.45:0/3961333066 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f80a806c8f0 0x7f80a806edb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.782+0000 7f80d1000640 1 --2- 172.21.15.45:0/3961333066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80cc162270 0x7f80cc0b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.782+0000 7f80d1000640 1 --2- 172.21.15.45:0/3961333066 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80cc161e90 0x7f80cc0b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.782+0000 7f80d1000640 1 --2- 172.21.15.45:0/3961333066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc0d52d0 0x7f80cc150bc0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:05.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.782+0000 7f80d1000640 1 -- 172.21.15.45:0/3961333066 >> 172.21.15.45:0/3961333066 conn(0x7f80cc0d1940 msgr2=0x7f80cc0cf730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:05.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.782+0000 7f80d1000640 1 -- 172.21.15.45:0/3961333066 shutdown_connections 2024-04-16T16:14:05.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:05.782+0000 7f80d1000640 1 -- 172.21.15.45:0/3961333066 wait complete. 2024-04-16T16:14:05.800 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:13:23.600104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:13:24.368141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:14:05.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:14:06.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.026+0000 7f5748400640 1 Processor -- start 2024-04-16T16:14:06.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.027+0000 7f5748400640 1 -- start start 2024-04-16T16:14:06.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.028+0000 7f5748400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f574006cf10 0x7f574006d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.028+0000 7f5748400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f57400d35d0 0x7f57400d3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.028+0000 7f5748400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f574006d8a0 con 0x7f574015ecc0 2024-04-16T16:14:06.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.028+0000 7f5748400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f57400d2b10 con 0x7f574006cf10 2024-04-16T16:14:06.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.028+0000 7f5748400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5740163210 con 0x7f57400d35d0 2024-04-16T16:14:06.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.029+0000 7f5745a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f574015ecc0 0x7f57401626a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49838/0 (socket says 172.21.15.45:49838) 2024-04-16T16:14:06.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.029+0000 7f5745a00640 1 -- 172.21.15.45:0/473204467 learned_addr learned my addr 172.21.15.45:0/473204467 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:06.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.029+0000 7f5745000640 1 --2- 172.21.15.45:0/473204467 >> v2:172.21.15.45:3300/0 conn(0x7f574006cf10 0x7f574006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.029+0000 7f5746400640 1 --2- 172.21.15.45:0/473204467 >> v2:172.21.15.103:3300/0 conn(0x7f57400d35d0 0x7f57400d3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.029+0000 7f5745000640 1 -- 172.21.15.45:0/473204467 >> v2:172.21.15.103:3300/0 conn(0x7f57400d35d0 msgr2=0x7f57400d3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:06.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.029+0000 7f5745000640 1 --2- 172.21.15.45:0/473204467 >> v2:172.21.15.103:3300/0 conn(0x7f57400d35d0 0x7f57400d3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.029+0000 7f5745000640 1 -- 172.21.15.45:0/473204467 >> v1:172.21.15.45:6789/0 conn(0x7f574015ecc0 legacy=0x7f57401626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.029+0000 7f573fe00640 1 -- 172.21.15.45:0/473204467 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4018753329 0 0) 0x7f574006d8a0 con 0x7f574015ecc0 2024-04-16T16:14:06.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.029+0000 7f5745000640 1 -- 172.21.15.45:0/473204467 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57401634f0 con 0x7f574006cf10 2024-04-16T16:14:06.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.029+0000 7f5746400640 1 --2- 172.21.15.45:0/473204467 >> v2:172.21.15.103:3300/0 conn(0x7f57400d35d0 0x7f57400d3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:06.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.030+0000 7f5745000640 1 --2- 172.21.15.45:0/473204467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574006cf10 0x7f574006d360 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f574008de50 tx=0x7f572c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe2b49379f1788f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:06.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.031+0000 7f573fe00640 1 -- 172.21.15.45:0/473204467 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c034070 con 0x7f574006cf10 2024-04-16T16:14:06.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.031+0000 7f573fe00640 1 -- 172.21.15.45:0/473204467 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f572c030610 con 0x7f574006cf10 2024-04-16T16:14:06.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.031+0000 7f573fe00640 1 -- 172.21.15.45:0/473204467 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c0309e0 con 0x7f574006cf10 2024-04-16T16:14:06.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.031+0000 7f5748400640 1 -- 172.21.15.45:0/473204467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574006cf10 msgr2=0x7f574006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.031+0000 7f5748400640 1 --2- 172.21.15.45:0/473204467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574006cf10 0x7f574006d360 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f574008de50 tx=0x7f572c02b960 comp rx=0 tx=0).stop 2024-04-16T16:14:06.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.031+0000 7f5748400640 1 -- 172.21.15.45:0/473204467 shutdown_connections 2024-04-16T16:14:06.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.031+0000 7f5748400640 1 --2- 172.21.15.45:0/473204467 >> v2:172.21.15.103:3300/0 conn(0x7f57400d35d0 0x7f57400d3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.031+0000 7f5748400640 1 --2- 172.21.15.45:0/473204467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574006cf10 0x7f574006d360 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.031+0000 7f5748400640 1 -- 172.21.15.45:0/473204467 >> 172.21.15.45:0/473204467 conn(0x7f5740073690 msgr2=0x7f57400d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:06.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.032+0000 7f5748400640 1 -- 172.21.15.45:0/473204467 shutdown_connections 2024-04-16T16:14:06.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.032+0000 7f5748400640 1 -- 172.21.15.45:0/473204467 wait complete. 2024-04-16T16:14:06.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.033+0000 7f5748400640 1 Processor -- start 2024-04-16T16:14:06.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.034+0000 7f5748400640 1 -- start start 2024-04-16T16:14:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.035+0000 7f5748400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f574006cf10 0x7f57400b8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.035+0000 7f5748400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57400d35d0 0x7f57401f77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.035+0000 7f5748400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574015ecc0 0x7f57400c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.035+0000 7f5748400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f57400ba540 con 0x7f574015ecc0 2024-04-16T16:14:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.035+0000 7f5748400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57400d30b0 con 0x7f574006cf10 2024-04-16T16:14:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.035+0000 7f5748400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5740162db0 con 0x7f57400d35d0 2024-04-16T16:14:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.035+0000 7f5746400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574015ecc0 0x7f57400c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.035+0000 7f5746400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574015ecc0 0x7f57400c99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40296/0 (socket says 172.21.15.45:40296) 2024-04-16T16:14:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.035+0000 7f5746400640 1 -- 172.21.15.45:0/2567554540 learned_addr learned my addr 172.21.15.45:0/2567554540 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:06.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.035+0000 7f5745a00640 1 --2- 172.21.15.45:0/2567554540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f574006cf10 0x7f57400b8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.035+0000 7f5745000640 1 --2- 172.21.15.45:0/2567554540 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57400d35d0 0x7f57401f77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.036+0000 7f5746400640 1 -- 172.21.15.45:0/2567554540 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57400d35d0 msgr2=0x7f57401f77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.036+0000 7f5746400640 1 --2- 172.21.15.45:0/2567554540 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57400d35d0 0x7f57401f77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.036+0000 7f5746400640 1 -- 172.21.15.45:0/2567554540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f574006cf10 msgr2=0x7f57400b8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.036+0000 7f5746400640 1 --2- 172.21.15.45:0/2567554540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f574006cf10 0x7f57400b8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.036+0000 7f5746400640 1 -- 172.21.15.45:0/2567554540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57401f7d10 con 0x7f574015ecc0 2024-04-16T16:14:06.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.036+0000 7f5746400640 1 --2- 172.21.15.45:0/2567554540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574015ecc0 0x7f57400c99e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f573800cf20 tx=0x7f573800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:06.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.036+0000 7f573d600640 1 -- 172.21.15.45:0/2567554540 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573801a070 con 0x7f574015ecc0 2024-04-16T16:14:06.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.036+0000 7f573d600640 1 -- 172.21.15.45:0/2567554540 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f573800d840 con 0x7f574015ecc0 2024-04-16T16:14:06.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.036+0000 7f573d600640 1 -- 172.21.15.45:0/2567554540 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573800dc30 con 0x7f574015ecc0 2024-04-16T16:14:06.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.036+0000 7f5748400640 1 -- 172.21.15.45:0/2567554540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5740161a60 con 0x7f574015ecc0 2024-04-16T16:14:06.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.036+0000 7f5748400640 1 -- 172.21.15.45:0/2567554540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5740156a10 con 0x7f574015ecc0 2024-04-16T16:14:06.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.038+0000 7f5748400640 1 -- 172.21.15.45:0/2567554540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5740075780 con 0x7f574015ecc0 2024-04-16T16:14:06.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.038+0000 7f573d600640 1 -- 172.21.15.45:0/2567554540 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f573801d070 con 0x7f574015ecc0 2024-04-16T16:14:06.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.039+0000 7f573d600640 1 --2- 172.21.15.45:0/2567554540 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f572006c8f0 0x7f572006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.039+0000 7f573d600640 1 -- 172.21.15.45:0/2567554540 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f573808c310 con 0x7f574015ecc0 2024-04-16T16:14:06.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.040+0000 7f5745a00640 1 --2- 172.21.15.45:0/2567554540 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f572006c8f0 0x7f572006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.041+0000 7f5745a00640 1 --2- 172.21.15.45:0/2567554540 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f572006c8f0 0x7f572006edb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5734000f60 tx=0x7f5734007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:06.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.044+0000 7f573d600640 1 -- 172.21.15.45:0/2567554540 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f573805a7d0 con 0x7f574015ecc0 2024-04-16T16:14:06.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.238+0000 7f5748400640 1 -- 172.21.15.45:0/2567554540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f574015d250 con 0x7f574015ecc0 2024-04-16T16:14:06.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.239+0000 7f573d600640 1 -- 172.21.15.45:0/2567554540 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f573805a0d0 con 0x7f574015ecc0 2024-04-16T16:14:06.238 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:14:06.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.242+0000 7f5748400640 1 -- 172.21.15.45:0/2567554540 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f572006c8f0 msgr2=0x7f572006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.242+0000 7f5748400640 1 --2- 172.21.15.45:0/2567554540 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f572006c8f0 0x7f572006edb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5734000f60 tx=0x7f5734007c00 comp rx=0 tx=0).stop 2024-04-16T16:14:06.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.242+0000 7f5748400640 1 -- 172.21.15.45:0/2567554540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574015ecc0 msgr2=0x7f57400c99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.242+0000 7f5748400640 1 --2- 172.21.15.45:0/2567554540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574015ecc0 0x7f57400c99e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f573800cf20 tx=0x7f573800a7d0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.242+0000 7f5748400640 1 -- 172.21.15.45:0/2567554540 shutdown_connections 2024-04-16T16:14:06.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.242+0000 7f5748400640 1 --2- 172.21.15.45:0/2567554540 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f572006c8f0 0x7f572006edb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.242+0000 7f5748400640 1 --2- 172.21.15.45:0/2567554540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574015ecc0 0x7f57400c99e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.242+0000 7f5748400640 1 --2- 172.21.15.45:0/2567554540 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57400d35d0 0x7f57401f77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.242+0000 7f5748400640 1 --2- 172.21.15.45:0/2567554540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f574006cf10 0x7f57400b8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.242+0000 7f5748400640 1 -- 172.21.15.45:0/2567554540 >> 172.21.15.45:0/2567554540 conn(0x7f5740073690 msgr2=0x7f574015c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:06.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.242+0000 7f5748400640 1 -- 172.21.15.45:0/2567554540 shutdown_connections 2024-04-16T16:14:06.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.242+0000 7f5748400640 1 -- 172.21.15.45:0/2567554540 wait complete. 2024-04-16T16:14:06.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:14:06.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.479+0000 7f6e23200640 1 Processor -- start 2024-04-16T16:14:06.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.479+0000 7f6e23200640 1 -- start start 2024-04-16T16:14:06.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.481+0000 7f6e23200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6e240d18f0 0x7f6e240ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.481+0000 7f6e23200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e240b6310 0x7f6e240d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.481+0000 7f6e23200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e240b6810 con 0x7f6e240d1cd0 2024-04-16T16:14:06.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.481+0000 7f6e23200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6e240d29f0 con 0x7f6e240d18f0 2024-04-16T16:14:06.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.481+0000 7f6e23200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6e240b6af0 con 0x7f6e240b6310 2024-04-16T16:14:06.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.481+0000 7f6e22800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6e240d1cd0 0x7f6e240d33c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49850/0 (socket says 172.21.15.45:49850) 2024-04-16T16:14:06.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.481+0000 7f6e22800640 1 -- 172.21.15.45:0/1503971555 learned_addr learned my addr 172.21.15.45:0/1503971555 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:06.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.481+0000 7f6e21400640 1 --2- 172.21.15.45:0/1503971555 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e240b6310 0x7f6e240d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.481+0000 7f6e21e00640 1 --2- 172.21.15.45:0/1503971555 >> v2:172.21.15.103:3300/0 conn(0x7f6e240d18f0 0x7f6e240ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 -- 172.21.15.45:0/1503971555 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1034392212 0 0) 0x7f6e240b6810 con 0x7f6e240d1cd0 2024-04-16T16:14:06.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 -- 172.21.15.45:0/1503971555 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e08003520 con 0x7f6e240d1cd0 2024-04-16T16:14:06.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 -- 172.21.15.45:0/1503971555 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2901954819 0 0) 0x7f6e08003520 con 0x7f6e240d1cd0 2024-04-16T16:14:06.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 -- 172.21.15.45:0/1503971555 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e240b6310 msgr2=0x7f6e240d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:06.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 --2- 172.21.15.45:0/1503971555 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e240b6310 0x7f6e240d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 -- 172.21.15.45:0/1503971555 >> v2:172.21.15.103:3300/0 conn(0x7f6e240d18f0 msgr2=0x7f6e240ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:06.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 --2- 172.21.15.45:0/1503971555 >> v2:172.21.15.103:3300/0 conn(0x7f6e240d18f0 0x7f6e240ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 -- 172.21.15.45:0/1503971555 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e240b6dd0 con 0x7f6e240d1cd0 2024-04-16T16:14:06.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 -- 172.21.15.45:0/1503971555 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f6e1c0019f0 con 0x7f6e240d1cd0 2024-04-16T16:14:06.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 -- 172.21.15.45:0/1503971555 >> v1:172.21.15.45:6789/0 conn(0x7f6e240d1cd0 legacy=0x7f6e240d33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 --2- 172.21.15.45:0/1503971555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e080039b0 0x7f6e08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 --2- 172.21.15.45:0/1503971555 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e240b6310 0x7f6e08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 --2- 172.21.15.45:0/1503971555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e08004d40 0x7f6e080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 -- 172.21.15.45:0/1503971555 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6e240d29f0 con 0x7f6e08004d40 2024-04-16T16:14:06.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 -- 172.21.15.45:0/1503971555 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6e240b6af0 con 0x7f6e080039b0 2024-04-16T16:14:06.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e20a00640 1 -- 172.21.15.45:0/1503971555 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6e0800b9c0 con 0x7f6e240b6310 2024-04-16T16:14:06.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.482+0000 7f6e22800640 1 --2- 172.21.15.45:0/1503971555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e08004d40 0x7f6e080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.483+0000 7f6e22800640 1 -- 172.21.15.45:0/1503971555 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e240b6310 msgr2=0x7f6e08004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:06.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.483+0000 7f6e21400640 1 --2- 172.21.15.45:0/1503971555 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e240b6310 0x7f6e08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.483+0000 7f6e22800640 1 --2- 172.21.15.45:0/1503971555 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e240b6310 0x7f6e08004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.483+0000 7f6e22800640 1 -- 172.21.15.45:0/1503971555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e080039b0 msgr2=0x7f6e08003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:06.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.483+0000 7f6e21e00640 1 --2- 172.21.15.45:0/1503971555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e080039b0 0x7f6e08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.483+0000 7f6e22800640 1 --2- 172.21.15.45:0/1503971555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e080039b0 0x7f6e08003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.483+0000 7f6e22800640 1 -- 172.21.15.45:0/1503971555 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e08003520 con 0x7f6e08004d40 2024-04-16T16:14:06.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.484+0000 7f6e22800640 1 --2- 172.21.15.45:0/1503971555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e08004d40 0x7f6e080051c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00a920 tx=0x7f6e08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67eaa63d1ce6f778 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:06.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.484+0000 7f6e20a00640 1 -- 172.21.15.45:0/1503971555 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c00a4c0 con 0x7f6e08004d40 2024-04-16T16:14:06.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.484+0000 7f6e20a00640 1 -- 172.21.15.45:0/1503971555 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e1c034650 con 0x7f6e08004d40 2024-04-16T16:14:06.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.484+0000 7f6e20a00640 1 -- 172.21.15.45:0/1503971555 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c034a40 con 0x7f6e08004d40 2024-04-16T16:14:06.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.485+0000 7f6e23200640 1 -- 172.21.15.45:0/1503971555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e08004d40 msgr2=0x7f6e080051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.485+0000 7f6e23200640 1 --2- 172.21.15.45:0/1503971555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e08004d40 0x7f6e080051c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00a920 tx=0x7f6e08005830 comp rx=0 tx=0).stop 2024-04-16T16:14:06.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.485+0000 7f6e22800640 1 -- 172.21.15.45:0/1503971555 reap_dead start 2024-04-16T16:14:06.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.485+0000 7f6e23200640 1 -- 172.21.15.45:0/1503971555 shutdown_connections 2024-04-16T16:14:06.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.485+0000 7f6e23200640 1 -- 172.21.15.45:0/1503971555 >> 172.21.15.45:0/1503971555 conn(0x7f6e2406cf90 msgr2=0x7f6e2406d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:06.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.485+0000 7f6e23200640 1 -- 172.21.15.45:0/1503971555 shutdown_connections 2024-04-16T16:14:06.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.485+0000 7f6e23200640 1 -- 172.21.15.45:0/1503971555 wait complete. 2024-04-16T16:14:06.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.487+0000 7f6e23200640 1 Processor -- start 2024-04-16T16:14:06.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.487+0000 7f6e23200640 1 -- start start 2024-04-16T16:14:06.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.488+0000 7f6e23200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e240ca1f0 0x7f6e24159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.488+0000 7f6e23200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e24157c40 0x7f6e2415a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.488+0000 7f6e23200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e240c84c0 0x7f6e240c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.488+0000 7f6e23200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6e240d2e80 con 0x7f6e240c84c0 2024-04-16T16:14:06.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.488+0000 7f6e23200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6e240ba760 con 0x7f6e24157c40 2024-04-16T16:14:06.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.488+0000 7f6e23200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6e240d3000 con 0x7f6e240ca1f0 2024-04-16T16:14:06.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.488+0000 7f6e22800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e240c84c0 0x7f6e240c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.488+0000 7f6e22800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e240c84c0 0x7f6e240c8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40324/0 (socket says 172.21.15.45:40324) 2024-04-16T16:14:06.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.488+0000 7f6e22800640 1 -- 172.21.15.45:0/659773331 learned_addr learned my addr 172.21.15.45:0/659773331 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:06.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.489+0000 7f6e21e00640 1 --2- 172.21.15.45:0/659773331 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e240ca1f0 0x7f6e24159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.489+0000 7f6e21400640 1 --2- 172.21.15.45:0/659773331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e24157c40 0x7f6e2415a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.489+0000 7f6e22800640 1 -- 172.21.15.45:0/659773331 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e240ca1f0 msgr2=0x7f6e24159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.489+0000 7f6e22800640 1 --2- 172.21.15.45:0/659773331 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e240ca1f0 0x7f6e24159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.489+0000 7f6e22800640 1 -- 172.21.15.45:0/659773331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e24157c40 msgr2=0x7f6e2415a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.489+0000 7f6e22800640 1 --2- 172.21.15.45:0/659773331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e24157c40 0x7f6e2415a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.489+0000 7f6e22800640 1 -- 172.21.15.45:0/659773331 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e241e7fa0 con 0x7f6e240c84c0 2024-04-16T16:14:06.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.489+0000 7f6e22800640 1 --2- 172.21.15.45:0/659773331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e240c84c0 0x7f6e240c8940 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00aa10 tx=0x7f6e240b77e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:06.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.489+0000 7f6e1a000640 1 -- 172.21.15.45:0/659773331 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c045070 con 0x7f6e240c84c0 2024-04-16T16:14:06.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.490+0000 7f6e1a000640 1 -- 172.21.15.45:0/659773331 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e1c03a040 con 0x7f6e240c84c0 2024-04-16T16:14:06.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.490+0000 7f6e1a000640 1 -- 172.21.15.45:0/659773331 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c00a4c0 con 0x7f6e240c84c0 2024-04-16T16:14:06.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.490+0000 7f6e23200640 1 -- 172.21.15.45:0/659773331 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e2415ac50 con 0x7f6e240c84c0 2024-04-16T16:14:06.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.490+0000 7f6e23200640 1 -- 172.21.15.45:0/659773331 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e2415b210 con 0x7f6e240c84c0 2024-04-16T16:14:06.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.491+0000 7f6e23200640 1 -- 172.21.15.45:0/659773331 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e2415b410 con 0x7f6e240c84c0 2024-04-16T16:14:06.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.492+0000 7f6e1a000640 1 -- 172.21.15.45:0/659773331 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6e1c048070 con 0x7f6e240c84c0 2024-04-16T16:14:06.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.493+0000 7f6e1a000640 1 --2- 172.21.15.45:0/659773331 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6e0406c8f0 0x7f6e0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.493+0000 7f6e1a000640 1 -- 172.21.15.45:0/659773331 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6e1c0ad720 con 0x7f6e240c84c0 2024-04-16T16:14:06.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.493+0000 7f6e21e00640 1 --2- 172.21.15.45:0/659773331 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6e0406c8f0 0x7f6e0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.495+0000 7f6e21e00640 1 --2- 172.21.15.45:0/659773331 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6e0406c8f0 0x7f6e0406edb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009a90 tx=0x7f6e14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:06.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.497+0000 7f6e1a000640 1 -- 172.21.15.45:0/659773331 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6e2415b410 con 0x7f6e240c84c0 2024-04-16T16:14:06.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.697+0000 7f6e23200640 1 -- 172.21.15.45:0/659773331 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6e24075780 con 0x7f6e240c84c0 2024-04-16T16:14:06.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.698+0000 7f6e1a000640 1 -- 172.21.15.45:0/659773331 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f6e1c07ba20 con 0x7f6e240c84c0 2024-04-16T16:14:06.697 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:14:06.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.700+0000 7f6e23200640 1 -- 172.21.15.45:0/659773331 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6e0406c8f0 msgr2=0x7f6e0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.700+0000 7f6e23200640 1 --2- 172.21.15.45:0/659773331 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6e0406c8f0 0x7f6e0406edb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009a90 tx=0x7f6e14008040 comp rx=0 tx=0).stop 2024-04-16T16:14:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.700+0000 7f6e23200640 1 -- 172.21.15.45:0/659773331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e240c84c0 msgr2=0x7f6e240c8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.700+0000 7f6e23200640 1 --2- 172.21.15.45:0/659773331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e240c84c0 0x7f6e240c8940 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00aa10 tx=0x7f6e240b77e0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.700+0000 7f6e23200640 1 -- 172.21.15.45:0/659773331 shutdown_connections 2024-04-16T16:14:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.700+0000 7f6e23200640 1 --2- 172.21.15.45:0/659773331 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f6e0406c8f0 0x7f6e0406edb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.700+0000 7f6e23200640 1 --2- 172.21.15.45:0/659773331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e240c84c0 0x7f6e240c8940 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.700+0000 7f6e23200640 1 --2- 172.21.15.45:0/659773331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e24157c40 0x7f6e2415a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.700+0000 7f6e23200640 1 --2- 172.21.15.45:0/659773331 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e240ca1f0 0x7f6e24159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.700+0000 7f6e23200640 1 -- 172.21.15.45:0/659773331 >> 172.21.15.45:0/659773331 conn(0x7f6e2406cf90 msgr2=0x7f6e240ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.701+0000 7f6e23200640 1 -- 172.21.15.45:0/659773331 shutdown_connections 2024-04-16T16:14:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.701+0000 7f6e23200640 1 -- 172.21.15.45:0/659773331 wait complete. 2024-04-16T16:14:06.717 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:14:06.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.939+0000 7fb9c0a00640 1 Processor -- start 2024-04-16T16:14:06.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.940+0000 7fb9c0a00640 1 -- start start 2024-04-16T16:14:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.941+0000 7fb9c0a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb9bc148190 0x7fb9bc0b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.941+0000 7fb9c0a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9bc148570 con 0x7fb9bc0b5e30 2024-04-16T16:14:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.941+0000 7fb9c0a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9bc15efa0 con 0x7fb9bc0ced20 2024-04-16T16:14:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.942+0000 7fb9c0a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb9bc0b6290 con 0x7fb9bc148190 2024-04-16T16:14:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.942+0000 7fb9b1e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb9bc0b5e30 0x7fb9bc0ce610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49866/0 (socket says 172.21.15.45:49866) 2024-04-16T16:14:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.942+0000 7fb9b1e00640 1 -- 172.21.15.45:0/1140311166 learned_addr learned my addr 172.21.15.45:0/1140311166 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.942+0000 7fb9ba800640 1 --2- 172.21.15.45:0/1140311166 >> v2:172.21.15.103:3300/0 conn(0x7fb9bc148190 0x7fb9bc0b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.942+0000 7fb9b9e00640 1 -- 172.21.15.45:0/1140311166 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1657883920 0 0) 0x7fb9bc148570 con 0x7fb9bc0b5e30 2024-04-16T16:14:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.943+0000 7fb9b9e00640 1 -- 172.21.15.45:0/1140311166 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9a0003520 con 0x7fb9bc0b5e30 2024-04-16T16:14:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.943+0000 7fb9b9e00640 1 -- 172.21.15.45:0/1140311166 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4192710926 0 0) 0x7fb9a0003520 con 0x7fb9bc0b5e30 2024-04-16T16:14:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.943+0000 7fb9ba800640 1 -- 172.21.15.45:0/1140311166 >> v1:172.21.15.103:6789/0 conn(0x7fb9bc0ced20 legacy=0x7fb9bc160040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.943+0000 7fb9ba800640 1 -- 172.21.15.45:0/1140311166 >> v1:172.21.15.45:6789/0 conn(0x7fb9bc0b5e30 legacy=0x7fb9bc0ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.943+0000 7fb9ba800640 1 -- 172.21.15.45:0/1140311166 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9bc15f280 con 0x7fb9bc148190 2024-04-16T16:14:06.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.945+0000 7fb9ba800640 1 --2- 172.21.15.45:0/1140311166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9bc148190 0x7fb9bc0b58f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac0097a0 tx=0x7fb9ac02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12c965447d09a839 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:06.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.945+0000 7fb9b9e00640 1 -- 172.21.15.45:0/1140311166 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac038070 con 0x7fb9bc148190 2024-04-16T16:14:06.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.945+0000 7fb9b9e00640 1 -- 172.21.15.45:0/1140311166 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9ac031500 con 0x7fb9bc148190 2024-04-16T16:14:06.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.945+0000 7fb9b9e00640 1 -- 172.21.15.45:0/1140311166 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a4001670 con 0x7fb9bc148190 2024-04-16T16:14:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.946+0000 7fb9c0a00640 1 -- 172.21.15.45:0/1140311166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9bc148190 msgr2=0x7fb9bc0b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.946+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/1140311166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9bc148190 0x7fb9bc0b58f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac0097a0 tx=0x7fb9ac02b820 comp rx=0 tx=0).stop 2024-04-16T16:14:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.946+0000 7fb9c0a00640 1 -- 172.21.15.45:0/1140311166 shutdown_connections 2024-04-16T16:14:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.946+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/1140311166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9bc148190 0x7fb9bc0b58f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.946+0000 7fb9c0a00640 1 -- 172.21.15.45:0/1140311166 >> 172.21.15.45:0/1140311166 conn(0x7fb9bc06b930 msgr2=0x7fb9bc06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.946+0000 7fb9c0a00640 1 -- 172.21.15.45:0/1140311166 shutdown_connections 2024-04-16T16:14:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.946+0000 7fb9c0a00640 1 -- 172.21.15.45:0/1140311166 wait complete. 2024-04-16T16:14:06.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.948+0000 7fb9c0a00640 1 Processor -- start 2024-04-16T16:14:06.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.948+0000 7fb9c0a00640 1 -- start start 2024-04-16T16:14:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.949+0000 7fb9c0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9bc0b5e30 0x7fb9bc13d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.949+0000 7fb9c0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9bc0ced20 0x7fb9bc161150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.949+0000 7fb9c0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc148190 0x7fb9bc161690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.949+0000 7fb9c0a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc15f840 con 0x7fb9bc148190 2024-04-16T16:14:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.949+0000 7fb9c0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc15fbb0 con 0x7fb9bc0b5e30 2024-04-16T16:14:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.949+0000 7fb9c0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb9bc160750 con 0x7fb9bc0ced20 2024-04-16T16:14:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.949+0000 7fb9bb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc148190 0x7fb9bc161690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.949+0000 7fb9bb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc148190 0x7fb9bc161690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40340/0 (socket says 172.21.15.45:40340) 2024-04-16T16:14:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.949+0000 7fb9bb200640 1 -- 172.21.15.45:0/499115360 learned_addr learned my addr 172.21.15.45:0/499115360 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.949+0000 7fb9ba800640 1 --2- 172.21.15.45:0/499115360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9bc0b5e30 0x7fb9bc13d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.950+0000 7fb9b1e00640 1 --2- 172.21.15.45:0/499115360 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9bc0ced20 0x7fb9bc161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.950+0000 7fb9bb200640 1 -- 172.21.15.45:0/499115360 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9bc0ced20 msgr2=0x7fb9bc161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.950+0000 7fb9bb200640 1 --2- 172.21.15.45:0/499115360 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9bc0ced20 0x7fb9bc161150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.950+0000 7fb9bb200640 1 -- 172.21.15.45:0/499115360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9bc0b5e30 msgr2=0x7fb9bc13d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:06.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.950+0000 7fb9bb200640 1 --2- 172.21.15.45:0/499115360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9bc0b5e30 0x7fb9bc13d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:06.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.950+0000 7fb9bb200640 1 -- 172.21.15.45:0/499115360 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9bc0cd250 con 0x7fb9bc148190 2024-04-16T16:14:06.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.950+0000 7fb9b1e00640 1 --2- 172.21.15.45:0/499115360 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9bc0ced20 0x7fb9bc161150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:06.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.950+0000 7fb9bb200640 1 --2- 172.21.15.45:0/499115360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc148190 0x7fb9bc161690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400c7d0 tx=0x7fb9b400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:06.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.950+0000 7fb9b3400640 1 -- 172.21.15.45:0/499115360 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4003b70 con 0x7fb9bc148190 2024-04-16T16:14:06.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.950+0000 7fb9b3400640 1 -- 172.21.15.45:0/499115360 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9b4003d30 con 0x7fb9bc148190 2024-04-16T16:14:06.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.950+0000 7fb9b3400640 1 -- 172.21.15.45:0/499115360 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b400aa90 con 0x7fb9bc148190 2024-04-16T16:14:06.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.950+0000 7fb9c0a00640 1 -- 172.21.15.45:0/499115360 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9bc0cd530 con 0x7fb9bc148190 2024-04-16T16:14:06.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.950+0000 7fb9c0a00640 1 -- 172.21.15.45:0/499115360 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9bc0cc520 con 0x7fb9bc148190 2024-04-16T16:14:06.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.952+0000 7fb9c0a00640 1 -- 172.21.15.45:0/499115360 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9bc075780 con 0x7fb9bc148190 2024-04-16T16:14:06.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.953+0000 7fb9b3400640 1 -- 172.21.15.45:0/499115360 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb9b400ac30 con 0x7fb9bc148190 2024-04-16T16:14:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.953+0000 7fb9b3400640 1 --2- 172.21.15.45:0/499115360 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb99806c8f0 0x7fb99806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.954+0000 7fb9b3400640 1 -- 172.21.15.45:0/499115360 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb9b408b9c0 con 0x7fb9bc148190 2024-04-16T16:14:06.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.954+0000 7fb9ba800640 1 --2- 172.21.15.45:0/499115360 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb99806c8f0 0x7fb99806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:06.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.955+0000 7fb9ba800640 1 --2- 172.21.15.45:0/499115360 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb99806c8f0 0x7fb99806edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac035000 tx=0x7fb9ac008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:06.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:06.958+0000 7fb9b3400640 1 -- 172.21.15.45:0/499115360 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb9b408de20 con 0x7fb9bc148190 2024-04-16T16:14:07.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.170+0000 7fb9c0a00640 1 -- 172.21.15.45:0/499115360 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb9bc0b6590 con 0x7fb9bc148190 2024-04-16T16:14:07.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.171+0000 7fb9b3400640 1 -- 172.21.15.45:0/499115360 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fb9b4059e80 con 0x7fb9bc148190 2024-04-16T16:14:07.170 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:14:07.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.174+0000 7fb9c0a00640 1 -- 172.21.15.45:0/499115360 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb99806c8f0 msgr2=0x7fb99806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:07.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.174+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/499115360 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb99806c8f0 0x7fb99806edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac035000 tx=0x7fb9ac008030 comp rx=0 tx=0).stop 2024-04-16T16:14:07.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.174+0000 7fb9c0a00640 1 -- 172.21.15.45:0/499115360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc148190 msgr2=0x7fb9bc161690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:07.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.174+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/499115360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc148190 0x7fb9bc161690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400c7d0 tx=0x7fb9b400cc90 comp rx=0 tx=0).stop 2024-04-16T16:14:07.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.174+0000 7fb9c0a00640 1 -- 172.21.15.45:0/499115360 shutdown_connections 2024-04-16T16:14:07.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.174+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/499115360 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb99806c8f0 0x7fb99806edb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.174+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/499115360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc148190 0x7fb9bc161690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.174+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/499115360 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9bc0ced20 0x7fb9bc161150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.174+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/499115360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9bc0b5e30 0x7fb9bc13d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.174+0000 7fb9c0a00640 1 -- 172.21.15.45:0/499115360 >> 172.21.15.45:0/499115360 conn(0x7fb9bc06b930 msgr2=0x7fb9bc0b4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:07.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.174+0000 7fb9c0a00640 1 -- 172.21.15.45:0/499115360 shutdown_connections 2024-04-16T16:14:07.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.174+0000 7fb9c0a00640 1 -- 172.21.15.45:0/499115360 wait complete. 2024-04-16T16:14:07.229 INFO:tasks.cephfs_test_runner:Starting test: test_client_metrics_and_metadata (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2024-04-16T16:14:07.230 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:07.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.459+0000 7f48fca00640 1 Processor -- start 2024-04-16T16:14:07.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.459+0000 7f48fca00640 1 -- start start 2024-04-16T16:14:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.461+0000 7f48fca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f48f8162ea0 0x7f48f8169a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.461+0000 7f48fca00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f8163530 con 0x7f48f8159150 2024-04-16T16:14:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.461+0000 7f48fca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f815f440 con 0x7f48f80b0ef0 2024-04-16T16:14:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.461+0000 7f48fca00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f48f815fa90 con 0x7f48f8162ea0 2024-04-16T16:14:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.461+0000 7f48f6800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f48f8159150 0x7f48f80b07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49876/0 (socket says 172.21.15.45:49876) 2024-04-16T16:14:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.461+0000 7f48f6800640 1 -- 172.21.15.45:0/3199672979 learned_addr learned my addr 172.21.15.45:0/3199672979 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.461+0000 7f48f5400640 1 -- 172.21.15.45:0/3199672979 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3712705411 0 0) 0x7f48f8163530 con 0x7f48f8159150 2024-04-16T16:14:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.461+0000 7f48f7200640 1 --2- 172.21.15.45:0/3199672979 >> v2:172.21.15.103:3300/0 conn(0x7f48f8162ea0 0x7f48f8169a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.462+0000 7f48f5400640 1 -- 172.21.15.45:0/3199672979 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48dc003520 con 0x7f48f8159150 2024-04-16T16:14:07.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.462+0000 7f48f5400640 1 -- 172.21.15.45:0/3199672979 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 538947024 0 0) 0x7f48dc003520 con 0x7f48f8159150 2024-04-16T16:14:07.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.462+0000 7f48f5400640 1 -- 172.21.15.45:0/3199672979 >> v2:172.21.15.103:3300/0 conn(0x7f48f8162ea0 msgr2=0x7f48f8169a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:07.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.462+0000 7f48f5400640 1 --2- 172.21.15.45:0/3199672979 >> v2:172.21.15.103:3300/0 conn(0x7f48f8162ea0 0x7f48f8169a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.462+0000 7f48f5400640 1 -- 172.21.15.45:0/3199672979 >> v1:172.21.15.103:6789/0 conn(0x7f48f80b0ef0 legacy=0x7f48f8162790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:07.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.462+0000 7f48f5400640 1 -- 172.21.15.45:0/3199672979 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f815f720 con 0x7f48f8159150 2024-04-16T16:14:07.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.462+0000 7f48f5400640 1 -- 172.21.15.45:0/3199672979 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f48e8001670 con 0x7f48f8159150 2024-04-16T16:14:07.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.462+0000 7f48f5400640 1 -- 172.21.15.45:0/3199672979 >> v1:172.21.15.45:6789/0 conn(0x7f48f8159150 legacy=0x7f48f80b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:07.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.462+0000 7f48f5400640 1 --2- 172.21.15.45:0/3199672979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f48dc003a00 0x7f48dc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:07.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.462+0000 7f48f5400640 1 --2- 172.21.15.45:0/3199672979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48dc004800 0x7f48dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:07.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.462+0000 7f48f5400640 1 --2- 172.21.15.45:0/3199672979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48dc0051c0 0x7f48dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:07.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.462+0000 7f48f5400640 1 -- 172.21.15.45:0/3199672979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f48f815fa90 con 0x7f48dc004800 2024-04-16T16:14:07.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.462+0000 7f48f5400640 1 -- 172.21.15.45:0/3199672979 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc00c0c0 con 0x7f48dc0051c0 2024-04-16T16:14:07.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.462+0000 7f48f5400640 1 -- 172.21.15.45:0/3199672979 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f48dc00c2c0 con 0x7f48dc003a00 2024-04-16T16:14:07.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.462+0000 7f48f5e00640 1 --2- 172.21.15.45:0/3199672979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48dc004800 0x7f48dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:07.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.463+0000 7f48f6800640 1 --2- 172.21.15.45:0/3199672979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f48dc003a00 0x7f48dc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:07.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.463+0000 7f48f7200640 1 --2- 172.21.15.45:0/3199672979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48dc0051c0 0x7f48dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:07.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.463+0000 7f48f5e00640 1 -- 172.21.15.45:0/3199672979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f48dc003a00 msgr2=0x7f48dc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:07.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.463+0000 7f48f5e00640 1 --2- 172.21.15.45:0/3199672979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f48dc003a00 0x7f48dc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.463+0000 7f48f5e00640 1 -- 172.21.15.45:0/3199672979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48dc0051c0 msgr2=0x7f48dc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:07.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.463+0000 7f48f5e00640 1 --2- 172.21.15.45:0/3199672979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48dc0051c0 0x7f48dc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.463+0000 7f48f5e00640 1 -- 172.21.15.45:0/3199672979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48dc003520 con 0x7f48dc004800 2024-04-16T16:14:07.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.463+0000 7f48f7200640 1 --2- 172.21.15.45:0/3199672979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48dc0051c0 0x7f48dc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:07.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.463+0000 7f48f7200640 1 -- 172.21.15.45:0/3199672979 reap_dead start 2024-04-16T16:14:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.464+0000 7f48f5e00640 1 --2- 172.21.15.45:0/3199672979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48dc004800 0x7f48dc004c80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f48e4001750 tx=0x7f48dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59928e5de37528b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.464+0000 7f48f5400640 1 -- 172.21.15.45:0/3199672979 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e4001370 con 0x7f48dc004800 2024-04-16T16:14:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.464+0000 7f48f5400640 1 -- 172.21.15.45:0/3199672979 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48e40052d0 con 0x7f48dc004800 2024-04-16T16:14:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.464+0000 7f48f5400640 1 -- 172.21.15.45:0/3199672979 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e4003830 con 0x7f48dc004800 2024-04-16T16:14:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.465+0000 7f48fca00640 1 -- 172.21.15.45:0/3199672979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48dc004800 msgr2=0x7f48dc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.465+0000 7f48fca00640 1 --2- 172.21.15.45:0/3199672979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48dc004800 0x7f48dc004c80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f48e4001750 tx=0x7f48dc005830 comp rx=0 tx=0).stop 2024-04-16T16:14:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.465+0000 7f48fca00640 1 -- 172.21.15.45:0/3199672979 shutdown_connections 2024-04-16T16:14:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.465+0000 7f48fca00640 1 --2- 172.21.15.45:0/3199672979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48dc004800 0x7f48dc004c80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.465+0000 7f48fca00640 1 -- 172.21.15.45:0/3199672979 >> 172.21.15.45:0/3199672979 conn(0x7f48f806f0e0 msgr2=0x7f48f806f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.465+0000 7f48fca00640 1 -- 172.21.15.45:0/3199672979 shutdown_connections 2024-04-16T16:14:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.465+0000 7f48fca00640 1 -- 172.21.15.45:0/3199672979 wait complete. 2024-04-16T16:14:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.467+0000 7f48fca00640 1 Processor -- start 2024-04-16T16:14:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.467+0000 7f48fca00640 1 -- start start 2024-04-16T16:14:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.468+0000 7f48fca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48dc004800 0x7f48f813a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.468+0000 7f48fca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48f815bf80 0x7f48f813d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.468+0000 7f48fca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f48f8161710 0x7f48f8161b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.468+0000 7f48fca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f48f815ff20 con 0x7f48dc004800 2024-04-16T16:14:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.468+0000 7f48fca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f48f815f0f0 con 0x7f48f815bf80 2024-04-16T16:14:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.468+0000 7f48fca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f48f81600a0 con 0x7f48f8161710 2024-04-16T16:14:07.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.468+0000 7f48f6800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48dc004800 0x7f48f813a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:07.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.469+0000 7f48f6800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48dc004800 0x7f48f813a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40358/0 (socket says 172.21.15.45:40358) 2024-04-16T16:14:07.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.469+0000 7f48f6800640 1 -- 172.21.15.45:0/4112055551 learned_addr learned my addr 172.21.15.45:0/4112055551 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:07.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.469+0000 7f48f5e00640 1 --2- 172.21.15.45:0/4112055551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48f815bf80 0x7f48f813d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:07.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.469+0000 7f48f7200640 1 --2- 172.21.15.45:0/4112055551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f48f8161710 0x7f48f8161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:07.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.469+0000 7f48f6800640 1 -- 172.21.15.45:0/4112055551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f48f8161710 msgr2=0x7f48f8161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:07.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.469+0000 7f48f6800640 1 --2- 172.21.15.45:0/4112055551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f48f8161710 0x7f48f8161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.469+0000 7f48f6800640 1 -- 172.21.15.45:0/4112055551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48f815bf80 msgr2=0x7f48f813d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:07.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.469+0000 7f48f6800640 1 --2- 172.21.15.45:0/4112055551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48f815bf80 0x7f48f813d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.469+0000 7f48f6800640 1 -- 172.21.15.45:0/4112055551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f815b4e0 con 0x7f48dc004800 2024-04-16T16:14:07.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.469+0000 7f48f6800640 1 --2- 172.21.15.45:0/4112055551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48dc004800 0x7f48f813a7f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f48e8001050 tx=0x7f48e800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:07.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.470+0000 7f48eea00640 1 -- 172.21.15.45:0/4112055551 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e800c660 con 0x7f48dc004800 2024-04-16T16:14:07.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.470+0000 7f48eea00640 1 -- 172.21.15.45:0/4112055551 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48e8018070 con 0x7f48dc004800 2024-04-16T16:14:07.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.470+0000 7f48eea00640 1 -- 172.21.15.45:0/4112055551 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e800ca60 con 0x7f48dc004800 2024-04-16T16:14:07.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.470+0000 7f48fca00640 1 -- 172.21.15.45:0/4112055551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48f8159ae0 con 0x7f48dc004800 2024-04-16T16:14:07.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.470+0000 7f48fca00640 1 -- 172.21.15.45:0/4112055551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48f815ba60 con 0x7f48dc004800 2024-04-16T16:14:07.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.471+0000 7f48fca00640 1 -- 172.21.15.45:0/4112055551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48f8075780 con 0x7f48dc004800 2024-04-16T16:14:07.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.472+0000 7f48eea00640 1 -- 172.21.15.45:0/4112055551 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f48e801d460 con 0x7f48dc004800 2024-04-16T16:14:07.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.473+0000 7f48eea00640 1 --2- 172.21.15.45:0/4112055551 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f48d806c920 0x7f48d806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:07.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.473+0000 7f48eea00640 1 -- 172.21.15.45:0/4112055551 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f48e808bc60 con 0x7f48dc004800 2024-04-16T16:14:07.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.474+0000 7f48f5e00640 1 --2- 172.21.15.45:0/4112055551 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f48d806c920 0x7f48d806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:07.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.475+0000 7f48f5e00640 1 --2- 172.21.15.45:0/4112055551 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f48d806c920 0x7f48d806ede0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f48e4033000 tx=0x7f48e4009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:07.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.477+0000 7f48eea00640 1 -- 172.21.15.45:0/4112055551 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f48e8059f60 con 0x7f48dc004800 2024-04-16T16:14:07.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.686+0000 7f48fca00640 1 -- 172.21.15.45:0/4112055551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48f806c820 con 0x7f48dc004800 2024-04-16T16:14:07.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.688+0000 7f48eea00640 1 -- 172.21.15.45:0/4112055551 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f48e8059860 con 0x7f48dc004800 2024-04-16T16:14:07.687 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:07.687 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":21,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:26.918617+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:07.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.691+0000 7f48fca00640 1 -- 172.21.15.45:0/4112055551 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f48d806c920 msgr2=0x7f48d806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:07.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.691+0000 7f48fca00640 1 --2- 172.21.15.45:0/4112055551 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f48d806c920 0x7f48d806ede0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f48e4033000 tx=0x7f48e4009030 comp rx=0 tx=0).stop 2024-04-16T16:14:07.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.691+0000 7f48fca00640 1 -- 172.21.15.45:0/4112055551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48dc004800 msgr2=0x7f48f813a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:07.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.691+0000 7f48fca00640 1 --2- 172.21.15.45:0/4112055551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48dc004800 0x7f48f813a7f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f48e8001050 tx=0x7f48e800eb90 comp rx=0 tx=0).stop 2024-04-16T16:14:07.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.691+0000 7f48fca00640 1 -- 172.21.15.45:0/4112055551 shutdown_connections 2024-04-16T16:14:07.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.691+0000 7f48fca00640 1 --2- 172.21.15.45:0/4112055551 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f48d806c920 0x7f48d806ede0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.691+0000 7f48fca00640 1 --2- 172.21.15.45:0/4112055551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f48f8161710 0x7f48f8161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.691+0000 7f48fca00640 1 --2- 172.21.15.45:0/4112055551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48f815bf80 0x7f48f813d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.691+0000 7f48fca00640 1 --2- 172.21.15.45:0/4112055551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48dc004800 0x7f48f813a7f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.691+0000 7f48fca00640 1 -- 172.21.15.45:0/4112055551 >> 172.21.15.45:0/4112055551 conn(0x7f48f806f0e0 msgr2=0x7f48f8160220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:07.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.691+0000 7f48fca00640 1 -- 172.21.15.45:0/4112055551 shutdown_connections 2024-04-16T16:14:07.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.691+0000 7f48fca00640 1 -- 172.21.15.45:0/4112055551 wait complete. 2024-04-16T16:14:07.709 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:13:23.600104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:13:24.368141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:14:07.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:14:07.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.934+0000 7fe2abe00640 1 Processor -- start 2024-04-16T16:14:07.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.934+0000 7fe2abe00640 1 -- start start 2024-04-16T16:14:07.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.936+0000 7fe2abe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe2ac15ea70 0x7fe2ac0d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:07.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.936+0000 7fe2abe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe2ac0d4370 0x7fe2ac0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.936+0000 7fe2abe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ac15ee50 con 0x7fe2ac0cfa70 2024-04-16T16:14:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.936+0000 7fe2abe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe2ac0d1a10 con 0x7fe2ac15ea70 2024-04-16T16:14:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.936+0000 7fe2abe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe2ac0b6480 con 0x7fe2ac0d4370 2024-04-16T16:14:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.936+0000 7fe2aaa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe2ac15ea70 0x7fe2ac0d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.936+0000 7fe2aaa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe2ac15ea70 0x7fe2ac0d11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40370/0 (socket says 172.21.15.45:40370) 2024-04-16T16:14:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.937+0000 7fe2aaa00640 1 -- 172.21.15.45:0/3702693431 learned_addr learned my addr 172.21.15.45:0/3702693431 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.937+0000 7fe2a2000640 1 --2- 172.21.15.45:0/3702693431 >> v2:172.21.15.103:3300/0 conn(0x7fe2ac0d4370 0x7fe2ac0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.937+0000 7fe2aaa00640 1 -- 172.21.15.45:0/3702693431 >> v2:172.21.15.103:3300/0 conn(0x7fe2ac0d4370 msgr2=0x7fe2ac0cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.937+0000 7fe2aaa00640 1 --2- 172.21.15.45:0/3702693431 >> v2:172.21.15.103:3300/0 conn(0x7fe2ac0d4370 0x7fe2ac0cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.937+0000 7fe2aaa00640 1 -- 172.21.15.45:0/3702693431 >> v1:172.21.15.45:6789/0 conn(0x7fe2ac0cfa70 legacy=0x7fe2ac0d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.937+0000 7fe2aaa00640 1 -- 172.21.15.45:0/3702693431 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2ac0b6760 con 0x7fe2ac15ea70 2024-04-16T16:14:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.937+0000 7fe2aa000640 1 -- 172.21.15.45:0/3702693431 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1876997079 0 0) 0x7fe2ac15ee50 con 0x7fe2ac0cfa70 2024-04-16T16:14:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.938+0000 7fe2aaa00640 1 --2- 172.21.15.45:0/3702693431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2ac15ea70 0x7fe2ac0d11a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac073e80 tx=0x7fe29c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b5d483e64f0a45a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.939+0000 7fe2aa000640 1 -- 172.21.15.45:0/3702693431 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29c02cb20 con 0x7fe2ac15ea70 2024-04-16T16:14:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.939+0000 7fe2aa000640 1 -- 172.21.15.45:0/3702693431 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe29c02ccc0 con 0x7fe2ac15ea70 2024-04-16T16:14:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.939+0000 7fe2aa000640 1 -- 172.21.15.45:0/3702693431 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29c034610 con 0x7fe2ac15ea70 2024-04-16T16:14:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.939+0000 7fe2abe00640 1 -- 172.21.15.45:0/3702693431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2ac15ea70 msgr2=0x7fe2ac0d11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.939+0000 7fe2abe00640 1 --2- 172.21.15.45:0/3702693431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2ac15ea70 0x7fe2ac0d11a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac073e80 tx=0x7fe29c02bf80 comp rx=0 tx=0).stop 2024-04-16T16:14:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.940+0000 7fe2abe00640 1 -- 172.21.15.45:0/3702693431 shutdown_connections 2024-04-16T16:14:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.940+0000 7fe2abe00640 1 --2- 172.21.15.45:0/3702693431 >> v2:172.21.15.103:3300/0 conn(0x7fe2ac0d4370 0x7fe2ac0cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.940+0000 7fe2abe00640 1 --2- 172.21.15.45:0/3702693431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2ac15ea70 0x7fe2ac0d11a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.940+0000 7fe2abe00640 1 -- 172.21.15.45:0/3702693431 >> 172.21.15.45:0/3702693431 conn(0x7fe2ac06d3b0 msgr2=0x7fe2ac073060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.940+0000 7fe2abe00640 1 -- 172.21.15.45:0/3702693431 shutdown_connections 2024-04-16T16:14:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.940+0000 7fe2abe00640 1 -- 172.21.15.45:0/3702693431 wait complete. 2024-04-16T16:14:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.941+0000 7fe2abe00640 1 Processor -- start 2024-04-16T16:14:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.942+0000 7fe2abe00640 1 -- start start 2024-04-16T16:14:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.942+0000 7fe2abe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2ac0cfa70 0x7fe2ac1f5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.942+0000 7fe2abe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac0d4370 0x7fe2ac0cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.943+0000 7fe2abe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2ac15ea70 0x7fe2ac1615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.943+0000 7fe2abe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac0b6f60 con 0x7fe2ac15ea70 2024-04-16T16:14:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.943+0000 7fe2abe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac0d16e0 con 0x7fe2ac0cfa70 2024-04-16T16:14:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.943+0000 7fe2abe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2ac1f5560 con 0x7fe2ac0d4370 2024-04-16T16:14:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.943+0000 7fe2ab400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2ac15ea70 0x7fe2ac1615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.943+0000 7fe2ab400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2ac15ea70 0x7fe2ac1615e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40376/0 (socket says 172.21.15.45:40376) 2024-04-16T16:14:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.943+0000 7fe2ab400640 1 -- 172.21.15.45:0/2977606554 learned_addr learned my addr 172.21.15.45:0/2977606554 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.943+0000 7fe2a2000640 1 --2- 172.21.15.45:0/2977606554 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac0d4370 0x7fe2ac0cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.943+0000 7fe2aaa00640 1 --2- 172.21.15.45:0/2977606554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2ac0cfa70 0x7fe2ac1f5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.943+0000 7fe2ab400640 1 -- 172.21.15.45:0/2977606554 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac0d4370 msgr2=0x7fe2ac0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.943+0000 7fe2ab400640 1 --2- 172.21.15.45:0/2977606554 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac0d4370 0x7fe2ac0cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.943+0000 7fe2ab400640 1 -- 172.21.15.45:0/2977606554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2ac0cfa70 msgr2=0x7fe2ac1f5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.943+0000 7fe2ab400640 1 --2- 172.21.15.45:0/2977606554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2ac0cfa70 0x7fe2ac1f5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.943+0000 7fe2ab400640 1 -- 172.21.15.45:0/2977606554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2ac0cc910 con 0x7fe2ac15ea70 2024-04-16T16:14:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.944+0000 7fe2ab400640 1 --2- 172.21.15.45:0/2977606554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2ac15ea70 0x7fe2ac1615e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400c410 tx=0x7fe2a400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.944+0000 7fe2a3400640 1 -- 172.21.15.45:0/2977606554 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a4017070 con 0x7fe2ac15ea70 2024-04-16T16:14:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.944+0000 7fe2a3400640 1 -- 172.21.15.45:0/2977606554 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2a40035f0 con 0x7fe2ac15ea70 2024-04-16T16:14:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.944+0000 7fe2a3400640 1 -- 172.21.15.45:0/2977606554 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a40039c0 con 0x7fe2ac15ea70 2024-04-16T16:14:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.944+0000 7fe2abe00640 1 -- 172.21.15.45:0/2977606554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2ac0ccbb0 con 0x7fe2ac15ea70 2024-04-16T16:14:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.944+0000 7fe2abe00640 1 -- 172.21.15.45:0/2977606554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2ac0cd170 con 0x7fe2ac15ea70 2024-04-16T16:14:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.945+0000 7fe2abe00640 1 -- 172.21.15.45:0/2977606554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2ac0737c0 con 0x7fe2ac15ea70 2024-04-16T16:14:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.946+0000 7fe2a3400640 1 -- 172.21.15.45:0/2977606554 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe2a401a070 con 0x7fe2ac15ea70 2024-04-16T16:14:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.947+0000 7fe2a3400640 1 --2- 172.21.15.45:0/2977606554 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe28806c920 0x7fe28806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.948+0000 7fe2a3400640 1 -- 172.21.15.45:0/2977606554 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe2a408ae50 con 0x7fe2ac15ea70 2024-04-16T16:14:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.948+0000 7fe2aaa00640 1 --2- 172.21.15.45:0/2977606554 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe28806c920 0x7fe28806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.949+0000 7fe2aaa00640 1 --2- 172.21.15.45:0/2977606554 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe28806c920 0x7fe28806ede0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe29c035000 tx=0x7fe29c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:07.952+0000 7fe2a3400640 1 -- 172.21.15.45:0/2977606554 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe2ac0737c0 con 0x7fe2ac15ea70 2024-04-16T16:14:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.153+0000 7fe2abe00640 1 -- 172.21.15.45:0/2977606554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe2ac075780 con 0x7fe2ac15ea70 2024-04-16T16:14:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.154+0000 7fe2a3400640 1 -- 172.21.15.45:0/2977606554 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fe2a4059270 con 0x7fe2ac15ea70 2024-04-16T16:14:08.155 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:14:08.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.157+0000 7fe2abe00640 1 -- 172.21.15.45:0/2977606554 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe28806c920 msgr2=0x7fe28806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:08.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.157+0000 7fe2abe00640 1 --2- 172.21.15.45:0/2977606554 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe28806c920 0x7fe28806ede0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe29c035000 tx=0x7fe29c030000 comp rx=0 tx=0).stop 2024-04-16T16:14:08.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.157+0000 7fe2abe00640 1 -- 172.21.15.45:0/2977606554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2ac15ea70 msgr2=0x7fe2ac1615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:08.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.157+0000 7fe2abe00640 1 --2- 172.21.15.45:0/2977606554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2ac15ea70 0x7fe2ac1615e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400c410 tx=0x7fe2a400c8d0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.157+0000 7fe2abe00640 1 -- 172.21.15.45:0/2977606554 shutdown_connections 2024-04-16T16:14:08.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.157+0000 7fe2abe00640 1 --2- 172.21.15.45:0/2977606554 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe28806c920 0x7fe28806ede0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.157+0000 7fe2abe00640 1 --2- 172.21.15.45:0/2977606554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2ac15ea70 0x7fe2ac1615e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.157+0000 7fe2abe00640 1 --2- 172.21.15.45:0/2977606554 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac0d4370 0x7fe2ac0cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.157+0000 7fe2abe00640 1 --2- 172.21.15.45:0/2977606554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2ac0cfa70 0x7fe2ac1f5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.157+0000 7fe2abe00640 1 -- 172.21.15.45:0/2977606554 >> 172.21.15.45:0/2977606554 conn(0x7fe2ac06d3b0 msgr2=0x7fe2ac0c7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:08.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.157+0000 7fe2abe00640 1 -- 172.21.15.45:0/2977606554 shutdown_connections 2024-04-16T16:14:08.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.157+0000 7fe2abe00640 1 -- 172.21.15.45:0/2977606554 wait complete. 2024-04-16T16:14:08.176 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:14:08.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.397+0000 7fd7fb200640 1 Processor -- start 2024-04-16T16:14:08.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.398+0000 7fd7fb200640 1 -- start start 2024-04-16T16:14:08.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.399+0000 7fd7fb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7f40d18f0 0x7fd7f40ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:08.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.399+0000 7fd7fb200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd7f40b6310 0x7fd7f40d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:08.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.399+0000 7fd7fb200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f40b6810 con 0x7fd7f40d1cd0 2024-04-16T16:14:08.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.399+0000 7fd7fb200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd7f40d29f0 con 0x7fd7f40b6310 2024-04-16T16:14:08.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.399+0000 7fd7fb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f40b6af0 con 0x7fd7f40d18f0 2024-04-16T16:14:08.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.400+0000 7fd7fa800640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd7f40d1cd0 0x7fd7f40d33c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:56256/0 (socket says 172.21.15.45:56256) 2024-04-16T16:14:08.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.400+0000 7fd7fa800640 1 -- 172.21.15.45:0/4275943060 learned_addr learned my addr 172.21.15.45:0/4275943060 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:08.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.400+0000 7fd7f9400640 1 --2- 172.21.15.45:0/4275943060 >> v2:172.21.15.103:3300/0 conn(0x7fd7f40b6310 0x7fd7f40d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:08.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.400+0000 7fd7f9e00640 1 --2- 172.21.15.45:0/4275943060 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7f40d18f0 0x7fd7f40ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.401+0000 7fd7f8a00640 1 -- 172.21.15.45:0/4275943060 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 597086759 0 0) 0x7fd7f40b6810 con 0x7fd7f40d1cd0 2024-04-16T16:14:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.401+0000 7fd7f8a00640 1 -- 172.21.15.45:0/4275943060 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7e0003520 con 0x7fd7f40d1cd0 2024-04-16T16:14:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.401+0000 7fd7f9e00640 1 -- 172.21.15.45:0/4275943060 >> v2:172.21.15.103:3300/0 conn(0x7fd7f40b6310 msgr2=0x7fd7f40d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.401+0000 7fd7f9e00640 1 --2- 172.21.15.45:0/4275943060 >> v2:172.21.15.103:3300/0 conn(0x7fd7f40b6310 0x7fd7f40d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.401+0000 7fd7f9e00640 1 -- 172.21.15.45:0/4275943060 >> v1:172.21.15.103:6789/0 conn(0x7fd7f40d1cd0 legacy=0x7fd7f40d33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.401+0000 7fd7f9e00640 1 -- 172.21.15.45:0/4275943060 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f40b6dd0 con 0x7fd7f40d18f0 2024-04-16T16:14:08.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.403+0000 7fd7f9e00640 1 --2- 172.21.15.45:0/4275943060 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7f40d18f0 0x7fd7f40ba220 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec009d90 tx=0x7fd7ec02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=accd8a21a11bd0a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:08.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.403+0000 7fd7f8a00640 1 -- 172.21.15.45:0/4275943060 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec039070 con 0x7fd7f40d18f0 2024-04-16T16:14:08.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.403+0000 7fd7f8a00640 1 -- 172.21.15.45:0/4275943060 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ec02cbc0 con 0x7fd7f40d18f0 2024-04-16T16:14:08.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.403+0000 7fd7f8a00640 1 -- 172.21.15.45:0/4275943060 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec034440 con 0x7fd7f40d18f0 2024-04-16T16:14:08.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.404+0000 7fd7fb200640 1 -- 172.21.15.45:0/4275943060 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7f40d18f0 msgr2=0x7fd7f40ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:08.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.404+0000 7fd7fb200640 1 --2- 172.21.15.45:0/4275943060 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7f40d18f0 0x7fd7f40ba220 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec009d90 tx=0x7fd7ec02be10 comp rx=0 tx=0).stop 2024-04-16T16:14:08.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.404+0000 7fd7fb200640 1 -- 172.21.15.45:0/4275943060 shutdown_connections 2024-04-16T16:14:08.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.404+0000 7fd7fb200640 1 --2- 172.21.15.45:0/4275943060 >> v2:172.21.15.103:3300/0 conn(0x7fd7f40b6310 0x7fd7f40d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.404+0000 7fd7fb200640 1 --2- 172.21.15.45:0/4275943060 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7f40d18f0 0x7fd7f40ba220 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.404+0000 7fd7fb200640 1 -- 172.21.15.45:0/4275943060 >> 172.21.15.45:0/4275943060 conn(0x7fd7f406cf90 msgr2=0x7fd7f406d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:08.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.404+0000 7fd7fb200640 1 -- 172.21.15.45:0/4275943060 shutdown_connections 2024-04-16T16:14:08.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.404+0000 7fd7fb200640 1 -- 172.21.15.45:0/4275943060 wait complete. 2024-04-16T16:14:08.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.406+0000 7fd7fb200640 1 Processor -- start 2024-04-16T16:14:08.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.406+0000 7fd7fb200640 1 -- start start 2024-04-16T16:14:08.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.407+0000 7fd7fb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7f40b6310 0x7fd7f41556c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:08.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.407+0000 7fd7fb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7f40d18f0 0x7fd7f415a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:08.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.407+0000 7fd7fb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7f40d1cd0 0x7fd7f415a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:08.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.407+0000 7fd7fb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f4162dc0 con 0x7fd7f40d18f0 2024-04-16T16:14:08.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.407+0000 7fd7fb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f40ba760 con 0x7fd7f40d1cd0 2024-04-16T16:14:08.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.407+0000 7fd7fb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f4162f40 con 0x7fd7f40b6310 2024-04-16T16:14:08.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.407+0000 7fd7f9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7f40d18f0 0x7fd7f415a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:08.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.407+0000 7fd7f9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7f40d18f0 0x7fd7f415a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40384/0 (socket says 172.21.15.45:40384) 2024-04-16T16:14:08.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.407+0000 7fd7f9400640 1 -- 172.21.15.45:0/3112200187 learned_addr learned my addr 172.21.15.45:0/3112200187 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:08.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.407+0000 7fd7f9e00640 1 --2- 172.21.15.45:0/3112200187 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7f40b6310 0x7fd7f41556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:08.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.408+0000 7fd7fa800640 1 --2- 172.21.15.45:0/3112200187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7f40d1cd0 0x7fd7f415a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.408+0000 7fd7f9400640 1 -- 172.21.15.45:0/3112200187 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7f40b6310 msgr2=0x7fd7f41556c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.408+0000 7fd7f9400640 1 --2- 172.21.15.45:0/3112200187 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7f40b6310 0x7fd7f41556c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.408+0000 7fd7f9400640 1 -- 172.21.15.45:0/3112200187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7f40d1cd0 msgr2=0x7fd7f415a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.408+0000 7fd7f9400640 1 --2- 172.21.15.45:0/3112200187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7f40d1cd0 0x7fd7f415a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.408+0000 7fd7f9400640 1 -- 172.21.15.45:0/3112200187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f40cc870 con 0x7fd7f40d18f0 2024-04-16T16:14:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.408+0000 7fd7f9400640 1 --2- 172.21.15.45:0/3112200187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7f40d18f0 0x7fd7f415a480 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd7e400e9b0 tx=0x7fd7e400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:08.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.408+0000 7fd7ea000640 1 -- 172.21.15.45:0/3112200187 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4017070 con 0x7fd7f40d18f0 2024-04-16T16:14:08.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.408+0000 7fd7ea000640 1 -- 172.21.15.45:0/3112200187 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e400c9a0 con 0x7fd7f40d18f0 2024-04-16T16:14:08.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.409+0000 7fd7fb200640 1 -- 172.21.15.45:0/3112200187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f40ccb50 con 0x7fd7f40d18f0 2024-04-16T16:14:08.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.409+0000 7fd7ea000640 1 -- 172.21.15.45:0/3112200187 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e400cd90 con 0x7fd7f40d18f0 2024-04-16T16:14:08.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.409+0000 7fd7fb200640 1 -- 172.21.15.45:0/3112200187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f40cd110 con 0x7fd7f40d18f0 2024-04-16T16:14:08.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.410+0000 7fd7fb200640 1 -- 172.21.15.45:0/3112200187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7f40d4150 con 0x7fd7f40d18f0 2024-04-16T16:14:08.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.411+0000 7fd7ea000640 1 -- 172.21.15.45:0/3112200187 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd7e401a070 con 0x7fd7f40d18f0 2024-04-16T16:14:08.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.412+0000 7fd7ea000640 1 --2- 172.21.15.45:0/3112200187 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd7d806c8f0 0x7fd7d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:08.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.412+0000 7fd7ea000640 1 -- 172.21.15.45:0/3112200187 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd7e408bf50 con 0x7fd7f40d18f0 2024-04-16T16:14:08.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.413+0000 7fd7f9e00640 1 --2- 172.21.15.45:0/3112200187 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd7d806c8f0 0x7fd7d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:08.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.414+0000 7fd7f9e00640 1 --2- 172.21.15.45:0/3112200187 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd7d806c8f0 0x7fd7d806edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec036000 tx=0x7fd7ec030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:08.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.416+0000 7fd7ea000640 1 -- 172.21.15.45:0/3112200187 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd7f40d4150 con 0x7fd7f40d18f0 2024-04-16T16:14:08.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.620+0000 7fd7fb200640 1 -- 172.21.15.45:0/3112200187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd7f40cd550 con 0x7fd7f40d18f0 2024-04-16T16:14:08.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.621+0000 7fd7ea000640 1 -- 172.21.15.45:0/3112200187 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fd7e405a390 con 0x7fd7f40d18f0 2024-04-16T16:14:08.620 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:14:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.623+0000 7fd7fb200640 1 -- 172.21.15.45:0/3112200187 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd7d806c8f0 msgr2=0x7fd7d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.624+0000 7fd7fb200640 1 --2- 172.21.15.45:0/3112200187 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd7d806c8f0 0x7fd7d806edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec036000 tx=0x7fd7ec030000 comp rx=0 tx=0).stop 2024-04-16T16:14:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.624+0000 7fd7fb200640 1 -- 172.21.15.45:0/3112200187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7f40d18f0 msgr2=0x7fd7f415a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.624+0000 7fd7fb200640 1 --2- 172.21.15.45:0/3112200187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7f40d18f0 0x7fd7f415a480 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd7e400e9b0 tx=0x7fd7e400ee70 comp rx=0 tx=0).stop 2024-04-16T16:14:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.624+0000 7fd7fb200640 1 -- 172.21.15.45:0/3112200187 shutdown_connections 2024-04-16T16:14:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.624+0000 7fd7fb200640 1 --2- 172.21.15.45:0/3112200187 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd7d806c8f0 0x7fd7d806edb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.624+0000 7fd7fb200640 1 --2- 172.21.15.45:0/3112200187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7f40d1cd0 0x7fd7f415a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.624+0000 7fd7fb200640 1 --2- 172.21.15.45:0/3112200187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7f40d18f0 0x7fd7f415a480 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.624+0000 7fd7fb200640 1 --2- 172.21.15.45:0/3112200187 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7f40b6310 0x7fd7f41556c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.624+0000 7fd7fb200640 1 -- 172.21.15.45:0/3112200187 >> 172.21.15.45:0/3112200187 conn(0x7fd7f406cf90 msgr2=0x7fd7f4161b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.625+0000 7fd7fb200640 1 -- 172.21.15.45:0/3112200187 shutdown_connections 2024-04-16T16:14:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.625+0000 7fd7fb200640 1 -- 172.21.15.45:0/3112200187 wait complete. 2024-04-16T16:14:08.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:14:08.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.874+0000 7f125f600640 1 Processor -- start 2024-04-16T16:14:08.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.875+0000 7f125f600640 1 -- start start 2024-04-16T16:14:08.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.876+0000 7f125f600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12580b4930 0x7f1258072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:08.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.876+0000 7f125f600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12580b5530 con 0x7f125815c7c0 2024-04-16T16:14:08.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.876+0000 7f125f600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12580b5810 con 0x7f12580d10e0 2024-04-16T16:14:08.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.876+0000 7f125f600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f12580b52e0 con 0x7f12580b4930 2024-04-16T16:14:08.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.877+0000 7f125cc00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f125815c7c0 0x7f12580d09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49898/0 (socket says 172.21.15.45:49898) 2024-04-16T16:14:08.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.877+0000 7f125cc00640 1 -- 172.21.15.45:0/2437400516 learned_addr learned my addr 172.21.15.45:0/2437400516 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:08.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.877+0000 7f125d600640 1 --2- 172.21.15.45:0/2437400516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12580b4930 0x7f1258072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:08.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.877+0000 7f1257400640 1 -- 172.21.15.45:0/2437400516 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1848916792 0 0) 0x7f12580b5530 con 0x7f125815c7c0 2024-04-16T16:14:08.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.877+0000 7f1257400640 1 -- 172.21.15.45:0/2437400516 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1240003520 con 0x7f125815c7c0 2024-04-16T16:14:08.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.878+0000 7f1257400640 1 -- 172.21.15.45:0/2437400516 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3691422863 0 0) 0x7f1240003520 con 0x7f125815c7c0 2024-04-16T16:14:08.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.878+0000 7f1257400640 1 -- 172.21.15.45:0/2437400516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12580b4930 msgr2=0x7f1258072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:08.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.878+0000 7f1257400640 1 --2- 172.21.15.45:0/2437400516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12580b4930 0x7f1258072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.878+0000 7f1257400640 1 -- 172.21.15.45:0/2437400516 >> v1:172.21.15.103:6789/0 conn(0x7f12580d10e0 legacy=0x7f125806f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:08.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.878+0000 7f1257400640 1 -- 172.21.15.45:0/2437400516 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12580d04b0 con 0x7f125815c7c0 2024-04-16T16:14:08.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.878+0000 7f1257400640 1 -- 172.21.15.45:0/2437400516 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f124c001690 con 0x7f125815c7c0 2024-04-16T16:14:08.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.878+0000 7f1257400640 1 -- 172.21.15.45:0/2437400516 >> v1:172.21.15.45:6789/0 conn(0x7f125815c7c0 legacy=0x7f12580d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:08.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.878+0000 7f1257400640 1 --2- 172.21.15.45:0/2437400516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12400039e0 0x7f1240003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:08.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.878+0000 7f1257400640 1 --2- 172.21.15.45:0/2437400516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1240004800 0x7f1240004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:08.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.878+0000 7f1257400640 1 --2- 172.21.15.45:0/2437400516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12580b4930 0x7f1240005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:08.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.878+0000 7f1257400640 1 -- 172.21.15.45:0/2437400516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f12580b52e0 con 0x7f1240004800 2024-04-16T16:14:08.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.878+0000 7f1257400640 1 -- 172.21.15.45:0/2437400516 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f124000bb80 con 0x7f12400039e0 2024-04-16T16:14:08.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.878+0000 7f1257400640 1 -- 172.21.15.45:0/2437400516 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f124000bd80 con 0x7f12580b4930 2024-04-16T16:14:08.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.878+0000 7f1257e00640 1 --2- 172.21.15.45:0/2437400516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1240004800 0x7f1240004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:08.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.878+0000 7f125d600640 1 --2- 172.21.15.45:0/2437400516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12580b4930 0x7f1240005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:08.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.879+0000 7f1257e00640 1 -- 172.21.15.45:0/2437400516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12580b4930 msgr2=0x7f1240005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:08.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.879+0000 7f1257e00640 1 --2- 172.21.15.45:0/2437400516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12580b4930 0x7f1240005290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.879+0000 7f125cc00640 1 --2- 172.21.15.45:0/2437400516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12400039e0 0x7f1240003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:08.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.879+0000 7f1257e00640 1 -- 172.21.15.45:0/2437400516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12400039e0 msgr2=0x7f1240003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:08.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.879+0000 7f1257e00640 1 --2- 172.21.15.45:0/2437400516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12400039e0 0x7f1240003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.879+0000 7f1257e00640 1 -- 172.21.15.45:0/2437400516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1240003520 con 0x7f1240004800 2024-04-16T16:14:08.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.880+0000 7f1257e00640 1 --2- 172.21.15.45:0/2437400516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1240004800 0x7f1240004c80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1248001750 tx=0x7f1240005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8b51a8f50fc7de server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:08.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.880+0000 7f1257400640 1 -- 172.21.15.45:0/2437400516 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248001370 con 0x7f1240004800 2024-04-16T16:14:08.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.880+0000 7f1257400640 1 -- 172.21.15.45:0/2437400516 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12480052d0 con 0x7f1240004800 2024-04-16T16:14:08.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.880+0000 7f1257400640 1 -- 172.21.15.45:0/2437400516 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248003830 con 0x7f1240004800 2024-04-16T16:14:08.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.881+0000 7f125f600640 1 -- 172.21.15.45:0/2437400516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1240004800 msgr2=0x7f1240004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:08.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.881+0000 7f125f600640 1 --2- 172.21.15.45:0/2437400516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1240004800 0x7f1240004c80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1248001750 tx=0x7f1240005830 comp rx=0 tx=0).stop 2024-04-16T16:14:08.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.881+0000 7f125d600640 1 -- 172.21.15.45:0/2437400516 reap_dead start 2024-04-16T16:14:08.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.881+0000 7f125f600640 1 -- 172.21.15.45:0/2437400516 shutdown_connections 2024-04-16T16:14:08.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.881+0000 7f125f600640 1 -- 172.21.15.45:0/2437400516 >> 172.21.15.45:0/2437400516 conn(0x7f12580b3970 msgr2=0x7f12580b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:08.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.881+0000 7f125f600640 1 -- 172.21.15.45:0/2437400516 shutdown_connections 2024-04-16T16:14:08.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.881+0000 7f125f600640 1 -- 172.21.15.45:0/2437400516 wait complete. 2024-04-16T16:14:08.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.883+0000 7f125f600640 1 Processor -- start 2024-04-16T16:14:08.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.883+0000 7f125f600640 1 -- start start 2024-04-16T16:14:08.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.884+0000 7f125f600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1240004800 0x7f12581527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:08.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.884+0000 7f125f600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f125815b360 0x7f125815b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:08.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.884+0000 7f125f600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12581540c0 0x7f1258154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:08.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.884+0000 7f125f600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f12580d0260 con 0x7f125815b360 2024-04-16T16:14:08.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.884+0000 7f125f600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f125806fac0 con 0x7f12581540c0 2024-04-16T16:14:08.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.884+0000 7f125f600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1258072e10 con 0x7f1240004800 2024-04-16T16:14:08.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.884+0000 7f1257e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f125815b360 0x7f125815b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:08.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.884+0000 7f1257e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f125815b360 0x7f125815b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40396/0 (socket says 172.21.15.45:40396) 2024-04-16T16:14:08.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.884+0000 7f1257e00640 1 -- 172.21.15.45:0/1357307880 learned_addr learned my addr 172.21.15.45:0/1357307880 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:08.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.885+0000 7f125cc00640 1 --2- 172.21.15.45:0/1357307880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1240004800 0x7f12581527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:08.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.885+0000 7f125d600640 1 --2- 172.21.15.45:0/1357307880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12581540c0 0x7f1258154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:08.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.885+0000 7f1257e00640 1 -- 172.21.15.45:0/1357307880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1240004800 msgr2=0x7f12581527a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:08.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.885+0000 7f1257e00640 1 --2- 172.21.15.45:0/1357307880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1240004800 0x7f12581527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.885+0000 7f1257e00640 1 -- 172.21.15.45:0/1357307880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12581540c0 msgr2=0x7f1258154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:08.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.885+0000 7f1257e00640 1 --2- 172.21.15.45:0/1357307880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12581540c0 0x7f1258154540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:08.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.885+0000 7f1257e00640 1 -- 172.21.15.45:0/1357307880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125815dd90 con 0x7f125815b360 2024-04-16T16:14:08.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.885+0000 7f125d600640 1 --2- 172.21.15.45:0/1357307880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12581540c0 0x7f1258154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:08.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.885+0000 7f1257e00640 1 --2- 172.21.15.45:0/1357307880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f125815b360 0x7f125815b7e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1248001870 tx=0x7f1248001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:08.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.885+0000 7f1254c00640 1 -- 172.21.15.45:0/1357307880 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248041070 con 0x7f125815b360 2024-04-16T16:14:08.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.885+0000 7f1254c00640 1 -- 172.21.15.45:0/1357307880 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1248005480 con 0x7f125815b360 2024-04-16T16:14:08.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.885+0000 7f125f600640 1 -- 172.21.15.45:0/1357307880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f125815e030 con 0x7f125815b360 2024-04-16T16:14:08.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.885+0000 7f1254c00640 1 -- 172.21.15.45:0/1357307880 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248001370 con 0x7f125815b360 2024-04-16T16:14:08.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.886+0000 7f125f600640 1 -- 172.21.15.45:0/1357307880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f125815e5f0 con 0x7f125815b360 2024-04-16T16:14:08.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.887+0000 7f125f600640 1 -- 172.21.15.45:0/1357307880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1258075780 con 0x7f125815b360 2024-04-16T16:14:08.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.888+0000 7f1254c00640 1 -- 172.21.15.45:0/1357307880 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1248044070 con 0x7f125815b360 2024-04-16T16:14:08.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.889+0000 7f1254c00640 1 --2- 172.21.15.45:0/1357307880 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f123c06c920 0x7f123c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:08.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.889+0000 7f1254c00640 1 -- 172.21.15.45:0/1357307880 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f12480ad570 con 0x7f125815b360 2024-04-16T16:14:08.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.890+0000 7f125cc00640 1 --2- 172.21.15.45:0/1357307880 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f123c06c920 0x7f123c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:08.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.891+0000 7f125cc00640 1 --2- 172.21.15.45:0/1357307880 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f123c06c920 0x7f123c06ede0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f124c001200 tx=0x7f124c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:08.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:08.894+0000 7f1254c00640 1 -- 172.21.15.45:0/1357307880 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f124807b9a0 con 0x7f125815b360 2024-04-16T16:14:09.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.098+0000 7f125f600640 1 -- 172.21.15.45:0/1357307880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f12580af950 con 0x7f125815b360 2024-04-16T16:14:09.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.099+0000 7f1254c00640 1 -- 172.21.15.45:0/1357307880 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f124807b2a0 con 0x7f125815b360 2024-04-16T16:14:09.098 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:14:09.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.102+0000 7f125f600640 1 -- 172.21.15.45:0/1357307880 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f123c06c920 msgr2=0x7f123c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:09.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.102+0000 7f125f600640 1 --2- 172.21.15.45:0/1357307880 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f123c06c920 0x7f123c06ede0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f124c001200 tx=0x7f124c009040 comp rx=0 tx=0).stop 2024-04-16T16:14:09.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.102+0000 7f125f600640 1 -- 172.21.15.45:0/1357307880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f125815b360 msgr2=0x7f125815b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:09.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.102+0000 7f125f600640 1 --2- 172.21.15.45:0/1357307880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f125815b360 0x7f125815b7e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1248001870 tx=0x7f1248001120 comp rx=0 tx=0).stop 2024-04-16T16:14:09.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.103+0000 7f125f600640 1 -- 172.21.15.45:0/1357307880 shutdown_connections 2024-04-16T16:14:09.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.103+0000 7f125f600640 1 --2- 172.21.15.45:0/1357307880 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f123c06c920 0x7f123c06ede0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:09.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.103+0000 7f125f600640 1 --2- 172.21.15.45:0/1357307880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12581540c0 0x7f1258154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:09.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.103+0000 7f125f600640 1 --2- 172.21.15.45:0/1357307880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f125815b360 0x7f125815b7e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:09.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.103+0000 7f125f600640 1 --2- 172.21.15.45:0/1357307880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1240004800 0x7f12581527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:09.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.103+0000 7f125f600640 1 -- 172.21.15.45:0/1357307880 >> 172.21.15.45:0/1357307880 conn(0x7f12580b3970 msgr2=0x7f12580b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:09.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.103+0000 7f125f600640 1 -- 172.21.15.45:0/1357307880 shutdown_connections 2024-04-16T16:14:09.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.103+0000 7f125f600640 1 -- 172.21.15.45:0/1357307880 wait complete. 2024-04-16T16:14:09.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_client_metrics_and_metadata' 2024-04-16T16:14:09.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.345+0000 7fa177e00640 1 Processor -- start 2024-04-16T16:14:09.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.345+0000 7fa177e00640 1 -- start start 2024-04-16T16:14:09.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.347+0000 7fa177e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa178146220 0x7fa17815ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:09.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.347+0000 7fa177e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa17815d360 con 0x7fa1781608a0 2024-04-16T16:14:09.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.347+0000 7fa177e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa17815d640 con 0x7fa17806f7a0 2024-04-16T16:14:09.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.347+0000 7fa177e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa178160410 con 0x7fa178146220 2024-04-16T16:14:09.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.347+0000 7fa176a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa178146220 0x7fa17815ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:09.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.347+0000 7fa176a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa178146220 0x7fa17815ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40400/0 (socket says 172.21.15.45:40400) 2024-04-16T16:14:09.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.347+0000 7fa176a00640 1 -- 172.21.15.45:0/641603092 learned_addr learned my addr 172.21.15.45:0/641603092 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:09.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.348+0000 7fa176a00640 1 -- 172.21.15.45:0/641603092 >> v1:172.21.15.103:6789/0 conn(0x7fa17806f7a0 legacy=0x7fa17806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:09.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.348+0000 7fa176a00640 1 -- 172.21.15.45:0/641603092 >> v1:172.21.15.45:6789/0 conn(0x7fa1781608a0 legacy=0x7fa1780cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:09.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.348+0000 7fa176a00640 1 -- 172.21.15.45:0/641603092 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa17815d920 con 0x7fa178146220 2024-04-16T16:14:09.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.349+0000 7fa176a00640 1 --2- 172.21.15.45:0/641603092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa178146220 0x7fa17815ccb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa168009870 tx=0x7fa16802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65ffecf972742dd server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:09.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.349+0000 7fa175600640 1 -- 172.21.15.45:0/641603092 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa168035020 con 0x7fa178146220 2024-04-16T16:14:09.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.349+0000 7fa175600640 1 -- 172.21.15.45:0/641603092 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa168031870 con 0x7fa178146220 2024-04-16T16:14:09.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.349+0000 7fa175600640 1 -- 172.21.15.45:0/641603092 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa168031c40 con 0x7fa178146220 2024-04-16T16:14:09.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.350+0000 7fa177e00640 1 -- 172.21.15.45:0/641603092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa178146220 msgr2=0x7fa17815ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:09.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.350+0000 7fa177e00640 1 --2- 172.21.15.45:0/641603092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa178146220 0x7fa17815ccb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa168009870 tx=0x7fa16802b910 comp rx=0 tx=0).stop 2024-04-16T16:14:09.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.350+0000 7fa177e00640 1 -- 172.21.15.45:0/641603092 shutdown_connections 2024-04-16T16:14:09.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.350+0000 7fa177e00640 1 --2- 172.21.15.45:0/641603092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa178146220 0x7fa17815ccb0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:09.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.350+0000 7fa177e00640 1 -- 172.21.15.45:0/641603092 >> 172.21.15.45:0/641603092 conn(0x7fa17806b930 msgr2=0x7fa17806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:09.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.350+0000 7fa177e00640 1 -- 172.21.15.45:0/641603092 shutdown_connections 2024-04-16T16:14:09.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.350+0000 7fa177e00640 1 -- 172.21.15.45:0/641603092 wait complete. 2024-04-16T16:14:09.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.352+0000 7fa177e00640 1 Processor -- start 2024-04-16T16:14:09.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.352+0000 7fa177e00640 1 -- start start 2024-04-16T16:14:09.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.353+0000 7fa177e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa17806f7a0 0x7fa178156c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:09.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.353+0000 7fa177e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa178146220 0x7fa1781f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:09.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.353+0000 7fa177e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1781608a0 0x7fa178143170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:09.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.353+0000 7fa177e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa17806fc20 con 0x7fa178146220 2024-04-16T16:14:09.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.353+0000 7fa177e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa17815dc00 con 0x7fa17806f7a0 2024-04-16T16:14:09.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.353+0000 7fa177e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa17806fda0 con 0x7fa1781608a0 2024-04-16T16:14:09.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.353+0000 7fa176000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa178146220 0x7fa1781f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:09.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.354+0000 7fa176000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa178146220 0x7fa1781f3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40408/0 (socket says 172.21.15.45:40408) 2024-04-16T16:14:09.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.354+0000 7fa176000640 1 -- 172.21.15.45:0/2687665224 learned_addr learned my addr 172.21.15.45:0/2687665224 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:09.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.354+0000 7fa176a00640 1 --2- 172.21.15.45:0/2687665224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa17806f7a0 0x7fa178156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:09.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.354+0000 7fa177400640 1 --2- 172.21.15.45:0/2687665224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1781608a0 0x7fa178143170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:09.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.354+0000 7fa176000640 1 -- 172.21.15.45:0/2687665224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1781608a0 msgr2=0x7fa178143170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:09.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.354+0000 7fa176000640 1 --2- 172.21.15.45:0/2687665224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1781608a0 0x7fa178143170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:09.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.354+0000 7fa176000640 1 -- 172.21.15.45:0/2687665224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa17806f7a0 msgr2=0x7fa178156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:09.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.354+0000 7fa176000640 1 --2- 172.21.15.45:0/2687665224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa17806f7a0 0x7fa178156c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:09.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.354+0000 7fa176000640 1 -- 172.21.15.45:0/2687665224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa178157f20 con 0x7fa178146220 2024-04-16T16:14:09.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.354+0000 7fa176000640 1 --2- 172.21.15.45:0/2687665224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa178146220 0x7fa1781f3140 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa16400c430 tx=0x7fa16400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:09.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.355+0000 7fa16ea00640 1 -- 172.21.15.45:0/2687665224 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164011790 con 0x7fa178146220 2024-04-16T16:14:09.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.355+0000 7fa16ea00640 1 -- 172.21.15.45:0/2687665224 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa164017070 con 0x7fa178146220 2024-04-16T16:14:09.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.355+0000 7fa177e00640 1 -- 172.21.15.45:0/2687665224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa178158200 con 0x7fa178146220 2024-04-16T16:14:09.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.355+0000 7fa16ea00640 1 -- 172.21.15.45:0/2687665224 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164011bb0 con 0x7fa178146220 2024-04-16T16:14:09.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.355+0000 7fa177e00640 1 -- 172.21.15.45:0/2687665224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa178158aa0 con 0x7fa178146220 2024-04-16T16:14:09.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.356+0000 7fa177e00640 1 -- 172.21.15.45:0/2687665224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa178158ca0 con 0x7fa178146220 2024-04-16T16:14:09.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.359+0000 7fa16ea00640 1 -- 172.21.15.45:0/2687665224 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa164026430 con 0x7fa178146220 2024-04-16T16:14:09.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.361+0000 7fa16ea00640 1 --2- 172.21.15.45:0/2687665224 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa15806c8f0 0x7fa15806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:09.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.361+0000 7fa16ea00640 1 -- 172.21.15.45:0/2687665224 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa16408c560 con 0x7fa178146220 2024-04-16T16:14:09.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.362+0000 7fa176a00640 1 --2- 172.21.15.45:0/2687665224 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa15806c8f0 0x7fa15806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:09.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.363+0000 7fa176a00640 1 --2- 172.21.15.45:0/2687665224 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa15806c8f0 0x7fa15806edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa168036000 tx=0x7fa16803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:09.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.364+0000 7fa16ea00640 1 -- 172.21.15.45:0/2687665224 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa178158ca0 con 0x7fa178146220 2024-04-16T16:14:09.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:09.640+0000 7fa177e00640 1 -- 172.21.15.45:0/2687665224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_client_metrics_and_metadata"]} v 0) -- 0x7fa178075780 con 0x7fa178146220 2024-04-16T16:14:10.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.359+0000 7fa16ea00640 1 -- 172.21.15.45:0/2687665224 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_client_metrics_and_metadata"]}]=0 v59) ==== 159+0+0 (secure 0 0 0) 0x7fa16405a970 con 0x7fa178146220 2024-04-16T16:14:10.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.362+0000 7fa177e00640 1 -- 172.21.15.45:0/2687665224 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa15806c8f0 msgr2=0x7fa15806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:10.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.362+0000 7fa177e00640 1 --2- 172.21.15.45:0/2687665224 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa15806c8f0 0x7fa15806edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa168036000 tx=0x7fa16803c040 comp rx=0 tx=0).stop 2024-04-16T16:14:10.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.362+0000 7fa177e00640 1 -- 172.21.15.45:0/2687665224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa178146220 msgr2=0x7fa1781f3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:10.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.362+0000 7fa177e00640 1 --2- 172.21.15.45:0/2687665224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa178146220 0x7fa1781f3140 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa16400c430 tx=0x7fa16400c8f0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.362+0000 7fa177e00640 1 -- 172.21.15.45:0/2687665224 shutdown_connections 2024-04-16T16:14:10.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.362+0000 7fa177e00640 1 --2- 172.21.15.45:0/2687665224 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa15806c8f0 0x7fa15806edb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.362+0000 7fa177e00640 1 --2- 172.21.15.45:0/2687665224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1781608a0 0x7fa178143170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.362+0000 7fa177e00640 1 --2- 172.21.15.45:0/2687665224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa178146220 0x7fa1781f3140 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.362+0000 7fa177e00640 1 --2- 172.21.15.45:0/2687665224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa17806f7a0 0x7fa178156c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.362+0000 7fa177e00640 1 -- 172.21.15.45:0/2687665224 >> 172.21.15.45:0/2687665224 conn(0x7fa17806b930 msgr2=0x7fa1780cf360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:10.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.363+0000 7fa177e00640 1 -- 172.21.15.45:0/2687665224 shutdown_connections 2024-04-16T16:14:10.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.363+0000 7fa177e00640 1 -- 172.21.15.45:0/2687665224 wait complete. 2024-04-16T16:14:10.384 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-16T16:14:10.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.611+0000 7f9cbac00640 1 Processor -- start 2024-04-16T16:14:10.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.612+0000 7f9cbac00640 1 -- start start 2024-04-16T16:14:10.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.613+0000 7f9cbac00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9cb4148190 0x7f9cb40b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:10.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.613+0000 7f9cbac00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9cb40b5e30 0x7f9cb40ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:10.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.613+0000 7f9cbac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cb40ceb50 0x7f9cb4071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:10.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.613+0000 7f9cbac00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9cb40b6290 con 0x7f9cb4148190 2024-04-16T16:14:10.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.613+0000 7f9cbac00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9cb415fbb0 con 0x7f9cb40b5e30 2024-04-16T16:14:10.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.613+0000 7f9cbac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9cb4148570 con 0x7f9cb40ceb50 2024-04-16T16:14:10.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.614+0000 7f9cb9800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9cb4148190 0x7f9cb40b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:10.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.614+0000 7f9cb9800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9cb4148190 0x7f9cb40b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40422/0 (socket says 172.21.15.45:40422) 2024-04-16T16:14:10.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.614+0000 7f9cb9800640 1 -- 172.21.15.45:0/3702862612 learned_addr learned my addr 172.21.15.45:0/3702862612 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:10.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.615+0000 7f9cb8e00640 1 --2- 172.21.15.45:0/3702862612 >> v2:172.21.15.103:3300/0 conn(0x7f9cb40b5e30 0x7f9cb40ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:10.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.615+0000 7f9cb9800640 1 -- 172.21.15.45:0/3702862612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cb40ceb50 msgr2=0x7f9cb4071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:10.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.615+0000 7f9cba200640 1 --2- 172.21.15.45:0/3702862612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cb40ceb50 0x7f9cb4071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:10.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.615+0000 7f9cb9800640 1 --2- 172.21.15.45:0/3702862612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cb40ceb50 0x7f9cb4071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.615+0000 7f9cb9800640 1 -- 172.21.15.45:0/3702862612 >> v2:172.21.15.103:3300/0 conn(0x7f9cb40b5e30 msgr2=0x7f9cb40ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:10.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.615+0000 7f9cb9800640 1 --2- 172.21.15.45:0/3702862612 >> v2:172.21.15.103:3300/0 conn(0x7f9cb40b5e30 0x7f9cb40ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.615+0000 7f9cb9800640 1 -- 172.21.15.45:0/3702862612 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cb415efa0 con 0x7f9cb4148190 2024-04-16T16:14:10.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.615+0000 7f9cb8e00640 1 --2- 172.21.15.45:0/3702862612 >> v2:172.21.15.103:3300/0 conn(0x7f9cb40b5e30 0x7f9cb40ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:10.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.615+0000 7f9cba200640 1 --2- 172.21.15.45:0/3702862612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cb40ceb50 0x7f9cb4071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:10.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.616+0000 7f9cb9800640 1 --2- 172.21.15.45:0/3702862612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cb4148190 0x7f9cb40b58f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40b24b0 tx=0x7f9ca802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1904d16f3290f40c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:10.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.617+0000 7f9cb3e00640 1 -- 172.21.15.45:0/3702862612 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8035020 con 0x7f9cb4148190 2024-04-16T16:14:10.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.617+0000 7f9cb3e00640 1 -- 172.21.15.45:0/3702862612 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ca8031870 con 0x7f9cb4148190 2024-04-16T16:14:10.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.617+0000 7f9cb3e00640 1 -- 172.21.15.45:0/3702862612 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8031c40 con 0x7f9cb4148190 2024-04-16T16:14:10.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.617+0000 7f9cbac00640 1 -- 172.21.15.45:0/3702862612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cb4148190 msgr2=0x7f9cb40b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:10.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.617+0000 7f9cbac00640 1 --2- 172.21.15.45:0/3702862612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cb4148190 0x7f9cb40b58f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40b24b0 tx=0x7f9ca802b910 comp rx=0 tx=0).stop 2024-04-16T16:14:10.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.617+0000 7f9cbac00640 1 -- 172.21.15.45:0/3702862612 shutdown_connections 2024-04-16T16:14:10.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.617+0000 7f9cbac00640 1 --2- 172.21.15.45:0/3702862612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cb40ceb50 0x7f9cb4071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.617+0000 7f9cbac00640 1 --2- 172.21.15.45:0/3702862612 >> v2:172.21.15.103:3300/0 conn(0x7f9cb40b5e30 0x7f9cb40ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.617+0000 7f9cbac00640 1 --2- 172.21.15.45:0/3702862612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cb4148190 0x7f9cb40b58f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.617+0000 7f9cbac00640 1 -- 172.21.15.45:0/3702862612 >> 172.21.15.45:0/3702862612 conn(0x7f9cb406b930 msgr2=0x7f9cb406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:10.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.618+0000 7f9cbac00640 1 -- 172.21.15.45:0/3702862612 shutdown_connections 2024-04-16T16:14:10.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.618+0000 7f9cbac00640 1 -- 172.21.15.45:0/3702862612 wait complete. 2024-04-16T16:14:10.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.619+0000 7f9cbac00640 1 Processor -- start 2024-04-16T16:14:10.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.619+0000 7f9cbac00640 1 -- start start 2024-04-16T16:14:10.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.620+0000 7f9cbac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cb40b5e30 0x7f9cb4133990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:10.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.620+0000 7f9cbac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cb40ceb50 0x7f9cb40cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:10.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.620+0000 7f9cbac00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cb4148190 0x7f9cb40cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:10.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.620+0000 7f9cbac00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb415f840 con 0x7f9cb4148190 2024-04-16T16:14:10.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.620+0000 7f9cbac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb415f560 con 0x7f9cb40b5e30 2024-04-16T16:14:10.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.620+0000 7f9cbac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9cb40cefc0 con 0x7f9cb40ceb50 2024-04-16T16:14:10.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.621+0000 7f9cba200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cb4148190 0x7f9cb40cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:10.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.621+0000 7f9cba200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cb4148190 0x7f9cb40cc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40436/0 (socket says 172.21.15.45:40436) 2024-04-16T16:14:10.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.621+0000 7f9cba200640 1 -- 172.21.15.45:0/3543611040 learned_addr learned my addr 172.21.15.45:0/3543611040 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:10.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.621+0000 7f9cb8e00640 1 --2- 172.21.15.45:0/3543611040 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cb40ceb50 0x7f9cb40cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:10.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.621+0000 7f9cb9800640 1 --2- 172.21.15.45:0/3543611040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cb40b5e30 0x7f9cb4133990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:10.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.621+0000 7f9cba200640 1 -- 172.21.15.45:0/3543611040 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cb40ceb50 msgr2=0x7f9cb40cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:10.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.621+0000 7f9cba200640 1 --2- 172.21.15.45:0/3543611040 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cb40ceb50 0x7f9cb40cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.621+0000 7f9cba200640 1 -- 172.21.15.45:0/3543611040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cb40b5e30 msgr2=0x7f9cb4133990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:10.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.621+0000 7f9cba200640 1 --2- 172.21.15.45:0/3543611040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cb40b5e30 0x7f9cb4133990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.621+0000 7f9cba200640 1 -- 172.21.15.45:0/3543611040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cb40cb4b0 con 0x7f9cb4148190 2024-04-16T16:14:10.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.621+0000 7f9cb9800640 1 --2- 172.21.15.45:0/3543611040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cb40b5e30 0x7f9cb4133990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:10.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.621+0000 7f9cba200640 1 --2- 172.21.15.45:0/3543611040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cb4148190 0x7f9cb40cc7b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00cef0 tx=0x7f9cac00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:10.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.622+0000 7f9cb1600640 1 -- 172.21.15.45:0/3543611040 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cac01a070 con 0x7f9cb4148190 2024-04-16T16:14:10.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.622+0000 7f9cb1600640 1 -- 172.21.15.45:0/3543611040 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cac00d860 con 0x7f9cb4148190 2024-04-16T16:14:10.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.622+0000 7f9cb1600640 1 -- 172.21.15.45:0/3543611040 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cac00dc50 con 0x7f9cb4148190 2024-04-16T16:14:10.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.622+0000 7f9cbac00640 1 -- 172.21.15.45:0/3543611040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cb40cb790 con 0x7f9cb4148190 2024-04-16T16:14:10.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.622+0000 7f9cbac00640 1 -- 172.21.15.45:0/3543611040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cb40cccf0 con 0x7f9cb4148190 2024-04-16T16:14:10.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.623+0000 7f9cbac00640 1 -- 172.21.15.45:0/3543611040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cb4075780 con 0x7f9cb4148190 2024-04-16T16:14:10.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.624+0000 7f9cb1600640 1 -- 172.21.15.45:0/3543611040 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9cac01d070 con 0x7f9cb4148190 2024-04-16T16:14:10.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.625+0000 7f9cb1600640 1 --2- 172.21.15.45:0/3543611040 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9c9406c8f0 0x7f9c9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:10.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.625+0000 7f9cb1600640 1 -- 172.21.15.45:0/3543611040 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9cac08c340 con 0x7f9cb4148190 2024-04-16T16:14:10.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.626+0000 7f9cb9800640 1 --2- 172.21.15.45:0/3543611040 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9c9406c8f0 0x7f9c9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:10.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.627+0000 7f9cb9800640 1 --2- 172.21.15.45:0/3543611040 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9c9406c8f0 0x7f9c9406edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8036000 tx=0x7f9ca8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:10.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.630+0000 7f9cb1600640 1 -- 172.21.15.45:0/3543611040 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9cac05a8c0 con 0x7f9cb4148190 2024-04-16T16:14:10.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.836+0000 7f9cbac00640 1 -- 172.21.15.45:0/3543611040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f9cb40b6590 con 0x7f9cb4148190 2024-04-16T16:14:10.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.841+0000 7f9cb1600640 1 -- 172.21.15.45:0/3543611040 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f9cac05a1c0 con 0x7f9cb4148190 2024-04-16T16:14:10.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.844+0000 7f9cbac00640 1 -- 172.21.15.45:0/3543611040 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9c9406c8f0 msgr2=0x7f9c9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:10.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.844+0000 7f9cbac00640 1 --2- 172.21.15.45:0/3543611040 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9c9406c8f0 0x7f9c9406edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8036000 tx=0x7f9ca8008030 comp rx=0 tx=0).stop 2024-04-16T16:14:10.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.844+0000 7f9cbac00640 1 -- 172.21.15.45:0/3543611040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cb4148190 msgr2=0x7f9cb40cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:10.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.844+0000 7f9cbac00640 1 --2- 172.21.15.45:0/3543611040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cb4148190 0x7f9cb40cc7b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00cef0 tx=0x7f9cac00a710 comp rx=0 tx=0).stop 2024-04-16T16:14:10.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.844+0000 7f9cbac00640 1 -- 172.21.15.45:0/3543611040 shutdown_connections 2024-04-16T16:14:10.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.844+0000 7f9cbac00640 1 --2- 172.21.15.45:0/3543611040 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9c9406c8f0 0x7f9c9406edb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.844+0000 7f9cbac00640 1 --2- 172.21.15.45:0/3543611040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cb4148190 0x7f9cb40cc7b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.844+0000 7f9cbac00640 1 --2- 172.21.15.45:0/3543611040 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cb40ceb50 0x7f9cb40cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.844+0000 7f9cbac00640 1 --2- 172.21.15.45:0/3543611040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cb40b5e30 0x7f9cb4133990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:10.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.844+0000 7f9cbac00640 1 -- 172.21.15.45:0/3543611040 >> 172.21.15.45:0/3543611040 conn(0x7f9cb406b930 msgr2=0x7f9cb40c9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:10.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.845+0000 7f9cbac00640 1 -- 172.21.15.45:0/3543611040 shutdown_connections 2024-04-16T16:14:10.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:10.845+0000 7f9cbac00640 1 -- 172.21.15.45:0/3543611040 wait complete. 2024-04-16T16:14:10.862 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-16T16:14:10.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:14:10.992 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:14:11.106 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-16T16:14:11.106 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:14:11.106 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:11.169 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:14:11.169 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:11.224 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:14:11.224 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:11.277 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:11.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.503+0000 7f3422c00640 1 Processor -- start 2024-04-16T16:14:11.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.503+0000 7f3422c00640 1 -- start start 2024-04-16T16:14:11.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.505+0000 7f3422c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f341c15ccb0 0x7f341c06df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:11.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.505+0000 7f3422c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f341c06e480 0x7f341c1608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:11.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.505+0000 7f3422c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f341c15d360 con 0x7f341c146220 2024-04-16T16:14:11.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.505+0000 7f3422c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f341c15d130 con 0x7f341c15ccb0 2024-04-16T16:14:11.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.505+0000 7f3422c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f341c160410 con 0x7f341c06e480 2024-04-16T16:14:11.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.505+0000 7f3420e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f341c15ccb0 0x7f341c06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:11.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.506+0000 7f3420e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f341c15ccb0 0x7f341c06df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40438/0 (socket says 172.21.15.45:40438) 2024-04-16T16:14:11.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.506+0000 7f3420e00640 1 -- 172.21.15.45:0/1882243866 learned_addr learned my addr 172.21.15.45:0/1882243866 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:11.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.506+0000 7f3422200640 1 --2- 172.21.15.45:0/1882243866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f341c06e480 0x7f341c1608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:11.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.506+0000 7f3420e00640 1 -- 172.21.15.45:0/1882243866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f341c06e480 msgr2=0x7f341c1608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:11.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.506+0000 7f3420e00640 1 --2- 172.21.15.45:0/1882243866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f341c06e480 0x7f341c1608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.506+0000 7f3420e00640 1 -- 172.21.15.45:0/1882243866 >> v1:172.21.15.103:6789/0 conn(0x7f341c146220 legacy=0x7f341c06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:11.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.506+0000 7f3420e00640 1 -- 172.21.15.45:0/1882243866 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341c15d640 con 0x7f341c15ccb0 2024-04-16T16:14:11.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.507+0000 7f3420e00640 1 --2- 172.21.15.45:0/1882243866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c15ccb0 0x7f341c06df40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3408009870 tx=0x7f340802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ffd0cfdbccc1de3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:11.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.508+0000 7f341be00640 1 -- 172.21.15.45:0/1882243866 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3408035020 con 0x7f341c15ccb0 2024-04-16T16:14:11.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.508+0000 7f341be00640 1 -- 172.21.15.45:0/1882243866 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3408031640 con 0x7f341c15ccb0 2024-04-16T16:14:11.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.508+0000 7f341be00640 1 -- 172.21.15.45:0/1882243866 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340802d040 con 0x7f341c15ccb0 2024-04-16T16:14:11.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.508+0000 7f3422c00640 1 -- 172.21.15.45:0/1882243866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c15ccb0 msgr2=0x7f341c06df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:11.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.508+0000 7f3422c00640 1 --2- 172.21.15.45:0/1882243866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c15ccb0 0x7f341c06df40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3408009870 tx=0x7f340802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.508+0000 7f3422c00640 1 -- 172.21.15.45:0/1882243866 shutdown_connections 2024-04-16T16:14:11.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.508+0000 7f3422c00640 1 --2- 172.21.15.45:0/1882243866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f341c06e480 0x7f341c1608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.508+0000 7f3422c00640 1 --2- 172.21.15.45:0/1882243866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c15ccb0 0x7f341c06df40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.508+0000 7f3422c00640 1 -- 172.21.15.45:0/1882243866 >> 172.21.15.45:0/1882243866 conn(0x7f341c06b930 msgr2=0x7f341c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.508+0000 7f3422c00640 1 -- 172.21.15.45:0/1882243866 shutdown_connections 2024-04-16T16:14:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.509+0000 7f3422c00640 1 -- 172.21.15.45:0/1882243866 wait complete. 2024-04-16T16:14:11.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.510+0000 7f3422c00640 1 Processor -- start 2024-04-16T16:14:11.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.510+0000 7f3422c00640 1 -- start start 2024-04-16T16:14:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.511+0000 7f3422c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f341c06e480 0x7f341c130d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.511+0000 7f3422c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f341c146220 0x7f341c0b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.511+0000 7f3422c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c15ccb0 0x7f341c13b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.511+0000 7f3422c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f341c160de0 con 0x7f341c15ccb0 2024-04-16T16:14:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.511+0000 7f3422c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f341c15dc00 con 0x7f341c06e480 2024-04-16T16:14:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.511+0000 7f3422c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f341c160f60 con 0x7f341c146220 2024-04-16T16:14:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.512+0000 7f3422200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c15ccb0 0x7f341c13b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.512+0000 7f3422200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c15ccb0 0x7f341c13b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40446/0 (socket says 172.21.15.45:40446) 2024-04-16T16:14:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.512+0000 7f3422200640 1 -- 172.21.15.45:0/1136701099 learned_addr learned my addr 172.21.15.45:0/1136701099 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.512+0000 7f3420e00640 1 --2- 172.21.15.45:0/1136701099 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f341c146220 0x7f341c0b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.512+0000 7f3421800640 1 --2- 172.21.15.45:0/1136701099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f341c06e480 0x7f341c130d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.512+0000 7f3422200640 1 -- 172.21.15.45:0/1136701099 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f341c146220 msgr2=0x7f341c0b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.512+0000 7f3422200640 1 --2- 172.21.15.45:0/1136701099 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f341c146220 0x7f341c0b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.512+0000 7f3422200640 1 -- 172.21.15.45:0/1136701099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f341c06e480 msgr2=0x7f341c130d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.512+0000 7f3422200640 1 --2- 172.21.15.45:0/1136701099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f341c06e480 0x7f341c130d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.512+0000 7f3422200640 1 -- 172.21.15.45:0/1136701099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341c0ce0d0 con 0x7f341c15ccb0 2024-04-16T16:14:11.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.513+0000 7f3422200640 1 --2- 172.21.15.45:0/1136701099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c15ccb0 0x7f341c13b1e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3414007de0 tx=0x7f341400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:11.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.513+0000 7f3419600640 1 -- 172.21.15.45:0/1136701099 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3414014620 con 0x7f341c15ccb0 2024-04-16T16:14:11.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.513+0000 7f3419600640 1 -- 172.21.15.45:0/1136701099 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3414019070 con 0x7f341c15ccb0 2024-04-16T16:14:11.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.513+0000 7f3419600640 1 -- 172.21.15.45:0/1136701099 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3414014a20 con 0x7f341c15ccb0 2024-04-16T16:14:11.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.513+0000 7f3422c00640 1 -- 172.21.15.45:0/1136701099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f341c0cee20 con 0x7f341c15ccb0 2024-04-16T16:14:11.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.513+0000 7f3422c00640 1 -- 172.21.15.45:0/1136701099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f341c0ceb40 con 0x7f341c15ccb0 2024-04-16T16:14:11.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.514+0000 7f3422c00640 1 -- 172.21.15.45:0/1136701099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f341c13cb60 con 0x7f341c15ccb0 2024-04-16T16:14:11.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.515+0000 7f3419600640 1 -- 172.21.15.45:0/1136701099 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3414014bc0 con 0x7f341c15ccb0 2024-04-16T16:14:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.516+0000 7f3419600640 1 --2- 172.21.15.45:0/1136701099 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f33fc06c920 0x7f33fc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.516+0000 7f3419600640 1 -- 172.21.15.45:0/1136701099 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f341408c540 con 0x7f341c15ccb0 2024-04-16T16:14:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.517+0000 7f3421800640 1 --2- 172.21.15.45:0/1136701099 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f33fc06c920 0x7f33fc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.518+0000 7f3421800640 1 --2- 172.21.15.45:0/1136701099 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f33fc06c920 0x7f33fc06ede0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f341c0b4fb0 tx=0x7f3410009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:11.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.521+0000 7f3419600640 1 -- 172.21.15.45:0/1136701099 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f341c13cb60 con 0x7f341c15ccb0 2024-04-16T16:14:11.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.724+0000 7f3422c00640 1 -- 172.21.15.45:0/1136701099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f341c075780 con 0x7f341c15ccb0 2024-04-16T16:14:11.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.727+0000 7f3419600640 1 -- 172.21.15.45:0/1136701099 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f341405a8d0 con 0x7f341c15ccb0 2024-04-16T16:14:11.726 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:11.726 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":21,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:13:26.918617+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.729+0000 7f3422c00640 1 -- 172.21.15.45:0/1136701099 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f33fc06c920 msgr2=0x7f33fc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.729+0000 7f3422c00640 1 --2- 172.21.15.45:0/1136701099 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f33fc06c920 0x7f33fc06ede0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f341c0b4fb0 tx=0x7f3410009040 comp rx=0 tx=0).stop 2024-04-16T16:14:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.729+0000 7f3422c00640 1 -- 172.21.15.45:0/1136701099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c15ccb0 msgr2=0x7f341c13b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.729+0000 7f3422c00640 1 --2- 172.21.15.45:0/1136701099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c15ccb0 0x7f341c13b1e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3414007de0 tx=0x7f341400e4b0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.729+0000 7f3422c00640 1 -- 172.21.15.45:0/1136701099 shutdown_connections 2024-04-16T16:14:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.729+0000 7f3422c00640 1 --2- 172.21.15.45:0/1136701099 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f33fc06c920 0x7f33fc06ede0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.729+0000 7f3422c00640 1 --2- 172.21.15.45:0/1136701099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c15ccb0 0x7f341c13b1e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.729+0000 7f3422c00640 1 --2- 172.21.15.45:0/1136701099 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f341c146220 0x7f341c0b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.729+0000 7f3422c00640 1 --2- 172.21.15.45:0/1136701099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f341c06e480 0x7f341c130d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.729+0000 7f3422c00640 1 -- 172.21.15.45:0/1136701099 >> 172.21.15.45:0/1136701099 conn(0x7f341c06b930 msgr2=0x7f341c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.730+0000 7f3422c00640 1 -- 172.21.15.45:0/1136701099 shutdown_connections 2024-04-16T16:14:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.730+0000 7f3422c00640 1 -- 172.21.15.45:0/1136701099 wait complete. 2024-04-16T16:14:11.749 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:13:23.600104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:13:24.368141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:14:11.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:14:11.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.982+0000 7f3446600640 1 Processor -- start 2024-04-16T16:14:11.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.982+0000 7f3446600640 1 -- start start 2024-04-16T16:14:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.984+0000 7f3446600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3440160e90 0x7f34400cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.984+0000 7f3446600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34400c6210 con 0x7f34400cf980 2024-04-16T16:14:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.984+0000 7f3446600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34401603b0 con 0x7f3440145fe0 2024-04-16T16:14:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.984+0000 7f3446600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3440160690 con 0x7f3440160e90 2024-04-16T16:14:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.984+0000 7f3445c00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f34400cf980 0x7f344006dfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49930/0 (socket says 172.21.15.45:49930) 2024-04-16T16:14:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.984+0000 7f3445c00640 1 -- 172.21.15.45:0/3162220619 learned_addr learned my addr 172.21.15.45:0/3162220619 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:11.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.985+0000 7f343fe00640 1 --2- 172.21.15.45:0/3162220619 >> v2:172.21.15.103:3300/0 conn(0x7f3440160e90 0x7f34400cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:11.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.985+0000 7f343f400640 1 -- 172.21.15.45:0/3162220619 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4294681462 0 0) 0x7f34400c6210 con 0x7f34400cf980 2024-04-16T16:14:11.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.985+0000 7f343f400640 1 -- 172.21.15.45:0/3162220619 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3428003520 con 0x7f34400cf980 2024-04-16T16:14:11.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.985+0000 7f343f400640 1 -- 172.21.15.45:0/3162220619 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2353010224 0 0) 0x7f3428003520 con 0x7f34400cf980 2024-04-16T16:14:11.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.985+0000 7f343f400640 1 -- 172.21.15.45:0/3162220619 >> v2:172.21.15.103:3300/0 conn(0x7f3440160e90 msgr2=0x7f34400cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:11.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.985+0000 7f343f400640 1 --2- 172.21.15.45:0/3162220619 >> v2:172.21.15.103:3300/0 conn(0x7f3440160e90 0x7f34400cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.985+0000 7f343f400640 1 -- 172.21.15.45:0/3162220619 >> v1:172.21.15.103:6789/0 conn(0x7f3440145fe0 legacy=0x7f34400c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:11.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.985+0000 7f343f400640 1 -- 172.21.15.45:0/3162220619 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3440160970 con 0x7f34400cf980 2024-04-16T16:14:11.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.985+0000 7f343f400640 1 -- 172.21.15.45:0/3162220619 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f34380019f0 con 0x7f34400cf980 2024-04-16T16:14:11.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.985+0000 7f343f400640 1 -- 172.21.15.45:0/3162220619 >> v1:172.21.15.45:6789/0 conn(0x7f34400cf980 legacy=0x7f344006dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:11.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.985+0000 7f343f400640 1 --2- 172.21.15.45:0/3162220619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3428003a30 0x7f3428003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:11.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.986+0000 7f343f400640 1 --2- 172.21.15.45:0/3162220619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3428004800 0x7f3428004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:11.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.986+0000 7f343f400640 1 --2- 172.21.15.45:0/3162220619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3428005230 0x7f342800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:11.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.986+0000 7f343f400640 1 -- 172.21.15.45:0/3162220619 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3440160690 con 0x7f3428003a30 2024-04-16T16:14:11.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.986+0000 7f343f400640 1 -- 172.21.15.45:0/3162220619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f342800c190 con 0x7f3428005230 2024-04-16T16:14:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.986+0000 7f343f400640 1 -- 172.21.15.45:0/3162220619 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f342800c390 con 0x7f3428004800 2024-04-16T16:14:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.986+0000 7f3445200640 1 --2- 172.21.15.45:0/3162220619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3428003a30 0x7f3428003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.986+0000 7f3445c00640 1 --2- 172.21.15.45:0/3162220619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3428005230 0x7f342800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.986+0000 7f343fe00640 1 --2- 172.21.15.45:0/3162220619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3428004800 0x7f3428004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.986+0000 7f3445200640 1 -- 172.21.15.45:0/3162220619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3428004800 msgr2=0x7f3428004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.986+0000 7f3445200640 1 --2- 172.21.15.45:0/3162220619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3428004800 0x7f3428004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.986+0000 7f3445200640 1 -- 172.21.15.45:0/3162220619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3428005230 msgr2=0x7f342800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.986+0000 7f3445200640 1 --2- 172.21.15.45:0/3162220619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3428005230 0x7f342800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.986+0000 7f3445200640 1 -- 172.21.15.45:0/3162220619 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3428003520 con 0x7f3428003a30 2024-04-16T16:14:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.986+0000 7f3445c00640 1 --2- 172.21.15.45:0/3162220619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3428005230 0x7f342800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:14:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.986+0000 7f3445c00640 1 -- 172.21.15.45:0/3162220619 reap_dead start 2024-04-16T16:14:11.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.988+0000 7f3445200640 1 --2- 172.21.15.45:0/3162220619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3428003a30 0x7f3428003e50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3434001750 tx=0x7f3428005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a253fb09e5bf20f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:11.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.988+0000 7f343f400640 1 -- 172.21.15.45:0/3162220619 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434001370 con 0x7f3428003a30 2024-04-16T16:14:11.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.988+0000 7f343f400640 1 -- 172.21.15.45:0/3162220619 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34340052d0 con 0x7f3428003a30 2024-04-16T16:14:11.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.988+0000 7f343f400640 1 -- 172.21.15.45:0/3162220619 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434003830 con 0x7f3428003a30 2024-04-16T16:14:11.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.988+0000 7f3446600640 1 -- 172.21.15.45:0/3162220619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3428003a30 msgr2=0x7f3428003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:11.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.988+0000 7f3446600640 1 --2- 172.21.15.45:0/3162220619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3428003a30 0x7f3428003e50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3434001750 tx=0x7f3428005830 comp rx=0 tx=0).stop 2024-04-16T16:14:11.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.989+0000 7f3446600640 1 -- 172.21.15.45:0/3162220619 shutdown_connections 2024-04-16T16:14:11.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.989+0000 7f3446600640 1 --2- 172.21.15.45:0/3162220619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3428003a30 0x7f3428003e50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.989+0000 7f3446600640 1 -- 172.21.15.45:0/3162220619 >> 172.21.15.45:0/3162220619 conn(0x7f3440067e00 msgr2=0x7f344006b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:11.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.989+0000 7f3446600640 1 -- 172.21.15.45:0/3162220619 shutdown_connections 2024-04-16T16:14:11.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.989+0000 7f3446600640 1 -- 172.21.15.45:0/3162220619 wait complete. 2024-04-16T16:14:11.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.990+0000 7f3446600640 1 Processor -- start 2024-04-16T16:14:11.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.991+0000 7f3446600640 1 -- start start 2024-04-16T16:14:11.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f3446600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3428003a30 0x7f34401e80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:11.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f3446600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34401562c0 0x7f34401f8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:11.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f3446600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f344015e8a0 0x7f344015ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:11.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f3446600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3440146540 con 0x7f34401562c0 2024-04-16T16:14:11.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f3446600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34401463c0 con 0x7f3428003a30 2024-04-16T16:14:11.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f3446600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f34400cf440 con 0x7f344015e8a0 2024-04-16T16:14:11.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f343fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34401562c0 0x7f34401f8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:11.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f343fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34401562c0 0x7f34401f8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40464/0 (socket says 172.21.15.45:40464) 2024-04-16T16:14:11.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f343fe00640 1 -- 172.21.15.45:0/772264899 learned_addr learned my addr 172.21.15.45:0/772264899 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:11.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f3445200640 1 --2- 172.21.15.45:0/772264899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3428003a30 0x7f34401e80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:11.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f343fe00640 1 -- 172.21.15.45:0/772264899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f344015e8a0 msgr2=0x7f344015ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:11.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f343fe00640 1 --2- 172.21.15.45:0/772264899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f344015e8a0 0x7f344015ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f343fe00640 1 -- 172.21.15.45:0/772264899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3428003a30 msgr2=0x7f34401e80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:11.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f343fe00640 1 --2- 172.21.15.45:0/772264899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3428003a30 0x7f34401e80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:11.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f343fe00640 1 -- 172.21.15.45:0/772264899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34401f4c80 con 0x7f34401562c0 2024-04-16T16:14:11.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.992+0000 7f3445200640 1 --2- 172.21.15.45:0/772264899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3428003a30 0x7f34401e80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:11.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.993+0000 7f343fe00640 1 --2- 172.21.15.45:0/772264899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34401562c0 0x7f34401f8990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3430011650 tx=0x7f3430011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:11.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.993+0000 7f343cc00640 1 -- 172.21.15.45:0/772264899 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343000f960 con 0x7f34401562c0 2024-04-16T16:14:11.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.993+0000 7f343cc00640 1 -- 172.21.15.45:0/772264899 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f343000fb00 con 0x7f34401562c0 2024-04-16T16:14:11.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.993+0000 7f343cc00640 1 -- 172.21.15.45:0/772264899 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430015460 con 0x7f34401562c0 2024-04-16T16:14:11.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.993+0000 7f3446600640 1 -- 172.21.15.45:0/772264899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34401f4f60 con 0x7f34401562c0 2024-04-16T16:14:11.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.993+0000 7f3446600640 1 -- 172.21.15.45:0/772264899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34401f5520 con 0x7f34401562c0 2024-04-16T16:14:11.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.995+0000 7f3446600640 1 -- 172.21.15.45:0/772264899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34400c7280 con 0x7f34401562c0 2024-04-16T16:14:11.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.996+0000 7f343cc00640 1 -- 172.21.15.45:0/772264899 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3430015600 con 0x7f34401562c0 2024-04-16T16:14:11.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.997+0000 7f343cc00640 1 --2- 172.21.15.45:0/772264899 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f342406c920 0x7f342406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:11.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.997+0000 7f343cc00640 1 -- 172.21.15.45:0/772264899 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f343008c3e0 con 0x7f34401562c0 2024-04-16T16:14:11.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.998+0000 7f3445200640 1 --2- 172.21.15.45:0/772264899 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f342406c920 0x7f342406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:11.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:11.999+0000 7f3445200640 1 --2- 172.21.15.45:0/772264899 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f342406c920 0x7f342406ede0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3434033000 tx=0x7f3434009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:12.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.004+0000 7f343cc00640 1 -- 172.21.15.45:0/772264899 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f34400c7280 con 0x7f34401562c0 2024-04-16T16:14:12.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.210+0000 7f3446600640 1 -- 172.21.15.45:0/772264899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3440075780 con 0x7f34401562c0 2024-04-16T16:14:12.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.211+0000 7f343cc00640 1 -- 172.21.15.45:0/772264899 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f3430019030 con 0x7f34401562c0 2024-04-16T16:14:12.210 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:14:12.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.214+0000 7f3446600640 1 -- 172.21.15.45:0/772264899 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f342406c920 msgr2=0x7f342406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:12.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.214+0000 7f3446600640 1 --2- 172.21.15.45:0/772264899 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f342406c920 0x7f342406ede0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3434033000 tx=0x7f3434009030 comp rx=0 tx=0).stop 2024-04-16T16:14:12.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.214+0000 7f3446600640 1 -- 172.21.15.45:0/772264899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34401562c0 msgr2=0x7f34401f8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:12.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.214+0000 7f3446600640 1 --2- 172.21.15.45:0/772264899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34401562c0 0x7f34401f8990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3430011650 tx=0x7f3430011b10 comp rx=0 tx=0).stop 2024-04-16T16:14:12.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.214+0000 7f3446600640 1 -- 172.21.15.45:0/772264899 shutdown_connections 2024-04-16T16:14:12.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.214+0000 7f3446600640 1 --2- 172.21.15.45:0/772264899 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f342406c920 0x7f342406ede0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.214+0000 7f3446600640 1 --2- 172.21.15.45:0/772264899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f344015e8a0 0x7f344015ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.214+0000 7f3446600640 1 --2- 172.21.15.45:0/772264899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34401562c0 0x7f34401f8990 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.214+0000 7f3446600640 1 --2- 172.21.15.45:0/772264899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3428003a30 0x7f34401e80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.214+0000 7f3446600640 1 -- 172.21.15.45:0/772264899 >> 172.21.15.45:0/772264899 conn(0x7f3440067e00 msgr2=0x7f34400b44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:12.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.214+0000 7f3446600640 1 -- 172.21.15.45:0/772264899 shutdown_connections 2024-04-16T16:14:12.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.215+0000 7f3446600640 1 -- 172.21.15.45:0/772264899 wait complete. 2024-04-16T16:14:12.234 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:14:12.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.463+0000 7fc30ca00640 1 Processor -- start 2024-04-16T16:14:12.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.463+0000 7fc30ca00640 1 -- start start 2024-04-16T16:14:12.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.465+0000 7fc30ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc308070d50 0x7fc3080b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:12.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.465+0000 7fc30ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3080b6720 0x7fc3080cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:12.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.465+0000 7fc30ca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3080b4ff0 con 0x7fc308162350 2024-04-16T16:14:12.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.465+0000 7fc30ca00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc3080b4db0 con 0x7fc308070d50 2024-04-16T16:14:12.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.465+0000 7fc30ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc3080b6b70 con 0x7fc3080b6720 2024-04-16T16:14:12.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.465+0000 7fc2fea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc308070d50 0x7fc3080b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:12.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.465+0000 7fc2fea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc308070d50 0x7fc3080b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40476/0 (socket says 172.21.15.45:40476) 2024-04-16T16:14:12.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.465+0000 7fc2fea00640 1 -- 172.21.15.45:0/925549353 learned_addr learned my addr 172.21.15.45:0/925549353 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:12.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.466+0000 7fc307e00640 1 --2- 172.21.15.45:0/925549353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3080b6720 0x7fc3080cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:12.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.466+0000 7fc2fea00640 1 -- 172.21.15.45:0/925549353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3080b6720 msgr2=0x7fc3080cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:12.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.466+0000 7fc2fea00640 1 --2- 172.21.15.45:0/925549353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3080b6720 0x7fc3080cfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.466+0000 7fc2fea00640 1 -- 172.21.15.45:0/925549353 >> v1:172.21.15.103:6789/0 conn(0x7fc308162350 legacy=0x7fc3080b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:12.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.466+0000 7fc2fea00640 1 -- 172.21.15.45:0/925549353 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3080b52d0 con 0x7fc308070d50 2024-04-16T16:14:12.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.466+0000 7fc307e00640 1 --2- 172.21.15.45:0/925549353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3080b6720 0x7fc3080cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:12.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.467+0000 7fc2fea00640 1 --2- 172.21.15.45:0/925549353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308070d50 0x7fc3080b61e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009870 tx=0x7fc2f002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206645d5b34dd9c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:12.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.467+0000 7fc306a00640 1 -- 172.21.15.45:0/925549353 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0035020 con 0x7fc308070d50 2024-04-16T16:14:12.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.467+0000 7fc306a00640 1 -- 172.21.15.45:0/925549353 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f0031640 con 0x7fc308070d50 2024-04-16T16:14:12.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.467+0000 7fc306a00640 1 -- 172.21.15.45:0/925549353 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f002d040 con 0x7fc308070d50 2024-04-16T16:14:12.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.468+0000 7fc30ca00640 1 -- 172.21.15.45:0/925549353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308070d50 msgr2=0x7fc3080b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:12.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.468+0000 7fc30ca00640 1 --2- 172.21.15.45:0/925549353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308070d50 0x7fc3080b61e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009870 tx=0x7fc2f002b8f0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.468+0000 7fc30ca00640 1 -- 172.21.15.45:0/925549353 shutdown_connections 2024-04-16T16:14:12.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.468+0000 7fc30ca00640 1 --2- 172.21.15.45:0/925549353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3080b6720 0x7fc3080cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.468+0000 7fc30ca00640 1 --2- 172.21.15.45:0/925549353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308070d50 0x7fc3080b61e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.468+0000 7fc30ca00640 1 -- 172.21.15.45:0/925549353 >> 172.21.15.45:0/925549353 conn(0x7fc30806f0c0 msgr2=0x7fc30806f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:12.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.468+0000 7fc30ca00640 1 -- 172.21.15.45:0/925549353 shutdown_connections 2024-04-16T16:14:12.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.468+0000 7fc30ca00640 1 -- 172.21.15.45:0/925549353 wait complete. 2024-04-16T16:14:12.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.471+0000 7fc30ca00640 1 Processor -- start 2024-04-16T16:14:12.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.471+0000 7fc30ca00640 1 -- start start 2024-04-16T16:14:12.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.472+0000 7fc30ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308070d50 0x7fc3080cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:12.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.473+0000 7fc30ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3080b6720 0x7fc3081f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:12.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.473+0000 7fc30ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc308162350 0x7fc3081f4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:12.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.473+0000 7fc30ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc3080cf6d0 con 0x7fc308070d50 2024-04-16T16:14:12.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.473+0000 7fc30ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc3080b5890 con 0x7fc308162350 2024-04-16T16:14:12.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.473+0000 7fc30ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc3080d0160 con 0x7fc3080b6720 2024-04-16T16:14:12.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.473+0000 7fc307400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308070d50 0x7fc3080cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:12.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.473+0000 7fc307400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308070d50 0x7fc3080cba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40488/0 (socket says 172.21.15.45:40488) 2024-04-16T16:14:12.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.473+0000 7fc307400640 1 -- 172.21.15.45:0/3226583527 learned_addr learned my addr 172.21.15.45:0/3226583527 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:12.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.473+0000 7fc2fea00640 1 --2- 172.21.15.45:0/3226583527 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3080b6720 0x7fc3081f4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:12.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.473+0000 7fc307400640 1 -- 172.21.15.45:0/3226583527 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3080b6720 msgr2=0x7fc3081f4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:12.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.473+0000 7fc307400640 1 --2- 172.21.15.45:0/3226583527 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3080b6720 0x7fc3081f4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.473+0000 7fc307e00640 1 --2- 172.21.15.45:0/3226583527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc308162350 0x7fc3081f4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:12.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.473+0000 7fc307400640 1 -- 172.21.15.45:0/3226583527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc308162350 msgr2=0x7fc3081f4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:12.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.473+0000 7fc307400640 1 --2- 172.21.15.45:0/3226583527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc308162350 0x7fc3081f4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.473+0000 7fc307400640 1 -- 172.21.15.45:0/3226583527 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3080cc450 con 0x7fc308070d50 2024-04-16T16:14:12.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.474+0000 7fc307400640 1 --2- 172.21.15.45:0/3226583527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308070d50 0x7fc3080cba70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800ba50 tx=0x7fc2f800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:12.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.474+0000 7fc2ffe00640 1 -- 172.21.15.45:0/3226583527 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8019070 con 0x7fc308070d50 2024-04-16T16:14:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.474+0000 7fc2ffe00640 1 -- 172.21.15.45:0/3226583527 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f800cc20 con 0x7fc308070d50 2024-04-16T16:14:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.474+0000 7fc2ffe00640 1 -- 172.21.15.45:0/3226583527 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8014680 con 0x7fc308070d50 2024-04-16T16:14:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.474+0000 7fc30ca00640 1 -- 172.21.15.45:0/3226583527 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3080cc730 con 0x7fc308070d50 2024-04-16T16:14:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.474+0000 7fc30ca00640 1 -- 172.21.15.45:0/3226583527 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3080cccf0 con 0x7fc308070d50 2024-04-16T16:14:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.476+0000 7fc30ca00640 1 -- 172.21.15.45:0/3226583527 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc308075780 con 0x7fc308070d50 2024-04-16T16:14:12.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.477+0000 7fc2ffe00640 1 -- 172.21.15.45:0/3226583527 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc2f801c070 con 0x7fc308070d50 2024-04-16T16:14:12.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.478+0000 7fc2ffe00640 1 --2- 172.21.15.45:0/3226583527 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc2e406c8f0 0x7fc2e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:12.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.479+0000 7fc2ffe00640 1 -- 172.21.15.45:0/3226583527 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7fc2f808b900 con 0x7fc308070d50 2024-04-16T16:14:12.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.479+0000 7fc2fea00640 1 --2- 172.21.15.45:0/3226583527 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc2e406c8f0 0x7fc2e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:12.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.480+0000 7fc2fea00640 1 --2- 172.21.15.45:0/3226583527 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc2e406c8f0 0x7fc2e406edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0036000 tx=0x7fc2f0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:12.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.483+0000 7fc2ffe00640 1 -- 172.21.15.45:0/3226583527 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc2f8059650 con 0x7fc308070d50 2024-04-16T16:14:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.688+0000 7fc30ca00640 1 -- 172.21.15.45:0/3226583527 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc30806e080 con 0x7fc308070d50 2024-04-16T16:14:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.688+0000 7fc2ffe00640 1 -- 172.21.15.45:0/3226583527 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7fc2f805d950 con 0x7fc308070d50 2024-04-16T16:14:12.688 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:14:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.691+0000 7fc30ca00640 1 -- 172.21.15.45:0/3226583527 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc2e406c8f0 msgr2=0x7fc2e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.691+0000 7fc30ca00640 1 --2- 172.21.15.45:0/3226583527 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc2e406c8f0 0x7fc2e406edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0036000 tx=0x7fc2f0008030 comp rx=0 tx=0).stop 2024-04-16T16:14:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.691+0000 7fc30ca00640 1 -- 172.21.15.45:0/3226583527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308070d50 msgr2=0x7fc3080cba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.691+0000 7fc30ca00640 1 --2- 172.21.15.45:0/3226583527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308070d50 0x7fc3080cba70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800ba50 tx=0x7fc2f800bf10 comp rx=0 tx=0).stop 2024-04-16T16:14:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.691+0000 7fc30ca00640 1 -- 172.21.15.45:0/3226583527 shutdown_connections 2024-04-16T16:14:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.691+0000 7fc30ca00640 1 --2- 172.21.15.45:0/3226583527 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc2e406c8f0 0x7fc2e406edb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.691+0000 7fc30ca00640 1 --2- 172.21.15.45:0/3226583527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc308162350 0x7fc3081f4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.691+0000 7fc30ca00640 1 --2- 172.21.15.45:0/3226583527 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3080b6720 0x7fc3081f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.691+0000 7fc30ca00640 1 --2- 172.21.15.45:0/3226583527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308070d50 0x7fc3080cba70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.691+0000 7fc30ca00640 1 -- 172.21.15.45:0/3226583527 >> 172.21.15.45:0/3226583527 conn(0x7fc30806f0c0 msgr2=0x7fc308161590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.692+0000 7fc30ca00640 1 -- 172.21.15.45:0/3226583527 shutdown_connections 2024-04-16T16:14:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.692+0000 7fc30ca00640 1 -- 172.21.15.45:0/3226583527 wait complete. 2024-04-16T16:14:12.711 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:14:12.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.946+0000 7f7fdae00640 1 Processor -- start 2024-04-16T16:14:12.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.946+0000 7f7fdae00640 1 -- start start 2024-04-16T16:14:12.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.948+0000 7f7fdae00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7fd40b5d50 0x7f7fd406eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:12.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.948+0000 7f7fdae00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fd4146590 con 0x7f7fd41461b0 2024-04-16T16:14:12.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.948+0000 7f7fdae00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fd40c5750 con 0x7f7fd40b58f0 2024-04-16T16:14:12.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.948+0000 7f7fdae00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f7fd406dfd0 con 0x7f7fd40b5d50 2024-04-16T16:14:12.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.948+0000 7f7fda400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7fd40b5d50 0x7f7fd406eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:12.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.949+0000 7f7fd9a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7fd41461b0 0x7f7fd4160cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49938/0 (socket says 172.21.15.45:49938) 2024-04-16T16:14:12.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.949+0000 7f7fd9a00640 1 -- 172.21.15.45:0/1026790199 learned_addr learned my addr 172.21.15.45:0/1026790199 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:12.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.949+0000 7f7fda400640 1 --2- 172.21.15.45:0/1026790199 >> v2:172.21.15.45:3300/0 conn(0x7f7fd40b5d50 0x7f7fd406eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40492/0 (socket says 172.21.15.45:40492) 2024-04-16T16:14:12.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.949+0000 7f7fda400640 1 -- 172.21.15.45:0/1026790199 >> v1:172.21.15.103:6789/0 conn(0x7f7fd40b58f0 legacy=0x7f7fd406e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:12.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.949+0000 7f7fda400640 1 -- 172.21.15.45:0/1026790199 >> v1:172.21.15.45:6789/0 conn(0x7f7fd41461b0 legacy=0x7f7fd4160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:12.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.949+0000 7f7fda400640 1 -- 172.21.15.45:0/1026790199 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd415ccb0 con 0x7f7fd40b5d50 2024-04-16T16:14:12.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.950+0000 7f7fda400640 1 --2- 172.21.15.45:0/1026790199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fd40b5d50 0x7f7fd406eb70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009d90 tx=0x7f7fcc02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecee14d2517d819 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:12.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.950+0000 7f7fd3e00640 1 -- 172.21.15.45:0/1026790199 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc039070 con 0x7f7fd40b5d50 2024-04-16T16:14:12.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.951+0000 7f7fd3e00640 1 -- 172.21.15.45:0/1026790199 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fcc02ccf0 con 0x7f7fd40b5d50 2024-04-16T16:14:12.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.951+0000 7f7fd3e00640 1 -- 172.21.15.45:0/1026790199 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc034660 con 0x7f7fd40b5d50 2024-04-16T16:14:12.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.951+0000 7f7fdae00640 1 -- 172.21.15.45:0/1026790199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fd40b5d50 msgr2=0x7f7fd406eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:12.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.951+0000 7f7fdae00640 1 --2- 172.21.15.45:0/1026790199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fd40b5d50 0x7f7fd406eb70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009d90 tx=0x7f7fcc02bf40 comp rx=0 tx=0).stop 2024-04-16T16:14:12.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.951+0000 7f7fdae00640 1 -- 172.21.15.45:0/1026790199 shutdown_connections 2024-04-16T16:14:12.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.951+0000 7f7fdae00640 1 --2- 172.21.15.45:0/1026790199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fd40b5d50 0x7f7fd406eb70 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.951+0000 7f7fdae00640 1 -- 172.21.15.45:0/1026790199 >> 172.21.15.45:0/1026790199 conn(0x7f7fd406b930 msgr2=0x7f7fd406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:12.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.951+0000 7f7fdae00640 1 -- 172.21.15.45:0/1026790199 shutdown_connections 2024-04-16T16:14:12.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.952+0000 7f7fdae00640 1 -- 172.21.15.45:0/1026790199 wait complete. 2024-04-16T16:14:12.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.953+0000 7f7fdae00640 1 Processor -- start 2024-04-16T16:14:12.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.953+0000 7f7fdae00640 1 -- start start 2024-04-16T16:14:12.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.954+0000 7f7fdae00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fd40b58f0 0x7f7fd4141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:12.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.954+0000 7f7fdae00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fd40b5d50 0x7f7fd415f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:12.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.954+0000 7f7fdae00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fd41461b0 0x7f7fd414fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:12.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.954+0000 7f7fdae00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd40c5fb0 con 0x7f7fd40b58f0 2024-04-16T16:14:12.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.954+0000 7f7fdae00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd40c5cd0 con 0x7f7fd40b5d50 2024-04-16T16:14:12.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.954+0000 7f7fdae00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fd40c6290 con 0x7f7fd41461b0 2024-04-16T16:14:12.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.955+0000 7f7fd9a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fd40b58f0 0x7f7fd4141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:12.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.955+0000 7f7fd9a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fd40b58f0 0x7f7fd4141f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40502/0 (socket says 172.21.15.45:40502) 2024-04-16T16:14:12.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.955+0000 7f7fd9a00640 1 -- 172.21.15.45:0/3454153050 learned_addr learned my addr 172.21.15.45:0/3454153050 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:12.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.955+0000 7f7fd9000640 1 --2- 172.21.15.45:0/3454153050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fd40b5d50 0x7f7fd415f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:12.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.955+0000 7f7fda400640 1 --2- 172.21.15.45:0/3454153050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fd41461b0 0x7f7fd414fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:12.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.955+0000 7f7fd9a00640 1 -- 172.21.15.45:0/3454153050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fd41461b0 msgr2=0x7f7fd414fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:12.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.955+0000 7f7fd9a00640 1 --2- 172.21.15.45:0/3454153050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fd41461b0 0x7f7fd414fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.955+0000 7f7fd9a00640 1 -- 172.21.15.45:0/3454153050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fd40b5d50 msgr2=0x7f7fd415f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:12.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.955+0000 7f7fd9a00640 1 --2- 172.21.15.45:0/3454153050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fd40b5d50 0x7f7fd415f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:12.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.955+0000 7f7fd9a00640 1 -- 172.21.15.45:0/3454153050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd4142820 con 0x7f7fd40b58f0 2024-04-16T16:14:12.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.955+0000 7f7fda400640 1 --2- 172.21.15.45:0/3454153050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fd41461b0 0x7f7fd414fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:12.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.956+0000 7f7fd9a00640 1 --2- 172.21.15.45:0/3454153050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fd40b58f0 0x7f7fd4141f20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7fc800c480 tx=0x7f7fc800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:12.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.956+0000 7f7fd1600640 1 -- 172.21.15.45:0/3454153050 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8017070 con 0x7f7fd40b58f0 2024-04-16T16:14:12.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.956+0000 7f7fd1600640 1 -- 172.21.15.45:0/3454153050 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fc8005ce0 con 0x7f7fd40b58f0 2024-04-16T16:14:12.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.956+0000 7f7fd1600640 1 -- 172.21.15.45:0/3454153050 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc80072f0 con 0x7f7fd40b58f0 2024-04-16T16:14:12.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.956+0000 7f7fdae00640 1 -- 172.21.15.45:0/3454153050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fd4142b00 con 0x7f7fd40b58f0 2024-04-16T16:14:12.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.956+0000 7f7fdae00640 1 -- 172.21.15.45:0/3454153050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fd41430c0 con 0x7f7fd40b58f0 2024-04-16T16:14:12.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.957+0000 7f7fdae00640 1 -- 172.21.15.45:0/3454153050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fd4143500 con 0x7f7fd40b58f0 2024-04-16T16:14:12.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.959+0000 7f7fd1600640 1 -- 172.21.15.45:0/3454153050 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7fc801a070 con 0x7f7fd40b58f0 2024-04-16T16:14:12.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.959+0000 7f7fd1600640 1 --2- 172.21.15.45:0/3454153050 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f7fb806c8f0 0x7f7fb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:12.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.960+0000 7f7fd1600640 1 -- 172.21.15.45:0/3454153050 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7fc808af40 con 0x7f7fd40b58f0 2024-04-16T16:14:12.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.960+0000 7f7fd9000640 1 --2- 172.21.15.45:0/3454153050 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f7fb806c8f0 0x7f7fb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:12.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.961+0000 7f7fd9000640 1 --2- 172.21.15.45:0/3454153050 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f7fb806c8f0 0x7f7fb806edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4152b70 tx=0x7f7fc4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:12.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:12.964+0000 7f7fd1600640 1 -- 172.21.15.45:0/3454153050 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7fc8058c90 con 0x7f7fd40b58f0 2024-04-16T16:14:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.175+0000 7f7fdae00640 1 -- 172.21.15.45:0/3454153050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7fd4072500 con 0x7f7fd40b58f0 2024-04-16T16:14:13.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.176+0000 7f7fd1600640 1 -- 172.21.15.45:0/3454153050 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7f7fc805cf90 con 0x7f7fd40b58f0 2024-04-16T16:14:13.175 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:14:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.179+0000 7f7fdae00640 1 -- 172.21.15.45:0/3454153050 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f7fb806c8f0 msgr2=0x7f7fb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.179+0000 7f7fdae00640 1 --2- 172.21.15.45:0/3454153050 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f7fb806c8f0 0x7f7fb806edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4152b70 tx=0x7f7fc4009040 comp rx=0 tx=0).stop 2024-04-16T16:14:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.179+0000 7f7fdae00640 1 -- 172.21.15.45:0/3454153050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fd40b58f0 msgr2=0x7f7fd4141f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.179+0000 7f7fdae00640 1 --2- 172.21.15.45:0/3454153050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fd40b58f0 0x7f7fd4141f20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7fc800c480 tx=0x7f7fc800c940 comp rx=0 tx=0).stop 2024-04-16T16:14:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.179+0000 7f7fdae00640 1 -- 172.21.15.45:0/3454153050 shutdown_connections 2024-04-16T16:14:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.179+0000 7f7fdae00640 1 --2- 172.21.15.45:0/3454153050 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f7fb806c8f0 0x7f7fb806edb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.179+0000 7f7fdae00640 1 --2- 172.21.15.45:0/3454153050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fd41461b0 0x7f7fd414fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.179+0000 7f7fdae00640 1 --2- 172.21.15.45:0/3454153050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fd40b5d50 0x7f7fd415f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.179+0000 7f7fdae00640 1 --2- 172.21.15.45:0/3454153050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fd40b58f0 0x7f7fd4141f20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.179+0000 7f7fdae00640 1 -- 172.21.15.45:0/3454153050 >> 172.21.15.45:0/3454153050 conn(0x7f7fd406b930 msgr2=0x7f7fd40b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.180+0000 7f7fdae00640 1 -- 172.21.15.45:0/3454153050 shutdown_connections 2024-04-16T16:14:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.180+0000 7f7fdae00640 1 -- 172.21.15.45:0/3454153050 wait complete. 2024-04-16T16:14:13.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:14:13.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.420+0000 7effbbe00640 1 Processor -- start 2024-04-16T16:14:13.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.420+0000 7effbbe00640 1 -- start start 2024-04-16T16:14:13.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.422+0000 7effbbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effb4072890 0x7effb40cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:13.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.422+0000 7effbbe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effb4146600 con 0x7effb40cd220 2024-04-16T16:14:13.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.422+0000 7effbbe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effb40cde00 con 0x7effb4146220 2024-04-16T16:14:13.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.422+0000 7effbbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7effb406f840 con 0x7effb4072890 2024-04-16T16:14:13.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.423+0000 7effb9e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7effb40cd220 0x7effb406df00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42950/0 (socket says 172.21.15.45:42950) 2024-04-16T16:14:13.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.423+0000 7effb9e00640 1 -- 172.21.15.45:0/3406389008 learned_addr learned my addr 172.21.15.45:0/3406389008 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.423+0000 7effb8a00640 1 --2- 172.21.15.45:0/3406389008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effb4072890 0x7effb40cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1656344664 0 0) 0x7effb4146600 con 0x7effb40cd220 2024-04-16T16:14:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff9c003520 con 0x7effb40cd220 2024-04-16T16:14:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2813474065 0 0) 0x7effb40cde00 con 0x7effb4146220 2024-04-16T16:14:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effb4146600 con 0x7effb4146220 2024-04-16T16:14:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2245815983 0 0) 0x7eff9c003520 con 0x7effb40cd220 2024-04-16T16:14:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effb4072890 msgr2=0x7effb40cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 --2- 172.21.15.45:0/3406389008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effb4072890 0x7effb40cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 >> v1:172.21.15.103:6789/0 conn(0x7effb4146220 legacy=0x7effb4072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb40ce630 con 0x7effb40cd220 2024-04-16T16:14:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7effac0019f0 con 0x7effb40cd220 2024-04-16T16:14:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 >> v1:172.21.15.45:6789/0 conn(0x7effb40cd220 legacy=0x7effb406df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 --2- 172.21.15.45:0/3406389008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff9c003b70 0x7eff9c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 --2- 172.21.15.45:0/3406389008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c004800 0x7eff9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 --2- 172.21.15.45:0/3406389008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effb4072890 0x7eff9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c00ba70 con 0x7eff9c004800 2024-04-16T16:14:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c00bbf0 con 0x7eff9c003b70 2024-04-16T16:14:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.424+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7eff9c00bdc0 con 0x7effb4072890 2024-04-16T16:14:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.425+0000 7effb8a00640 1 --2- 172.21.15.45:0/3406389008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c004800 0x7eff9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.425+0000 7effb9400640 1 --2- 172.21.15.45:0/3406389008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff9c003b70 0x7eff9c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.425+0000 7effb9e00640 1 --2- 172.21.15.45:0/3406389008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effb4072890 0x7eff9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.425+0000 7effb8a00640 1 -- 172.21.15.45:0/3406389008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effb4072890 msgr2=0x7eff9c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.425+0000 7effb8a00640 1 --2- 172.21.15.45:0/3406389008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effb4072890 0x7eff9c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.425+0000 7effb8a00640 1 -- 172.21.15.45:0/3406389008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff9c003b70 msgr2=0x7eff9c003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.425+0000 7effb8a00640 1 --2- 172.21.15.45:0/3406389008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff9c003b70 0x7eff9c003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.425+0000 7effb8a00640 1 -- 172.21.15.45:0/3406389008 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9c003520 con 0x7eff9c004800 2024-04-16T16:14:13.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.426+0000 7effb8a00640 1 --2- 172.21.15.45:0/3406389008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c004800 0x7eff9c004c80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7effa0009ac0 tx=0x7eff9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e65dec9ea73f3fc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:13.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.427+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa0039070 con 0x7eff9c004800 2024-04-16T16:14:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.427+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effa002c660 con 0x7eff9c004800 2024-04-16T16:14:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.427+0000 7effb3e00640 1 -- 172.21.15.45:0/3406389008 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa0009690 con 0x7eff9c004800 2024-04-16T16:14:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.427+0000 7effbbe00640 1 -- 172.21.15.45:0/3406389008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c004800 msgr2=0x7eff9c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.427+0000 7effbbe00640 1 --2- 172.21.15.45:0/3406389008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c004800 0x7eff9c004c80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7effa0009ac0 tx=0x7eff9c005830 comp rx=0 tx=0).stop 2024-04-16T16:14:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.427+0000 7effb9e00640 1 -- 172.21.15.45:0/3406389008 reap_dead start 2024-04-16T16:14:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.427+0000 7effbbe00640 1 -- 172.21.15.45:0/3406389008 shutdown_connections 2024-04-16T16:14:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.427+0000 7effbbe00640 1 -- 172.21.15.45:0/3406389008 >> 172.21.15.45:0/3406389008 conn(0x7effb406b930 msgr2=0x7effb406bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.427+0000 7effbbe00640 1 -- 172.21.15.45:0/3406389008 shutdown_connections 2024-04-16T16:14:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.427+0000 7effbbe00640 1 -- 172.21.15.45:0/3406389008 wait complete. 2024-04-16T16:14:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.429+0000 7effbbe00640 1 Processor -- start 2024-04-16T16:14:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.429+0000 7effbbe00640 1 -- start start 2024-04-16T16:14:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.430+0000 7effbbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c004800 0x7effb4153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.430+0000 7effbbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effb41548c0 0x7effb40c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.430+0000 7effbbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effb40c8da0 0x7effb40c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.430+0000 7effbbe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7effb40cebf0 con 0x7eff9c004800 2024-04-16T16:14:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.430+0000 7effbbe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7effb40ce0e0 con 0x7effb41548c0 2024-04-16T16:14:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.430+0000 7effbbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7effb40cf110 con 0x7effb40c8da0 2024-04-16T16:14:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.431+0000 7effb9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c004800 0x7effb4153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.431+0000 7effb9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c004800 0x7effb4153ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51988/0 (socket says 172.21.15.45:51988) 2024-04-16T16:14:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.431+0000 7effb9400640 1 -- 172.21.15.45:0/3465971909 learned_addr learned my addr 172.21.15.45:0/3465971909 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.431+0000 7effb8a00640 1 --2- 172.21.15.45:0/3465971909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effb41548c0 0x7effb40c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.431+0000 7effb9e00640 1 --2- 172.21.15.45:0/3465971909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effb40c8da0 0x7effb40c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.431+0000 7effb9400640 1 -- 172.21.15.45:0/3465971909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effb40c8da0 msgr2=0x7effb40c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.431+0000 7effb9400640 1 --2- 172.21.15.45:0/3465971909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effb40c8da0 0x7effb40c9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.431+0000 7effb9400640 1 -- 172.21.15.45:0/3465971909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effb41548c0 msgr2=0x7effb40c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.431+0000 7effb9400640 1 --2- 172.21.15.45:0/3465971909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effb41548c0 0x7effb40c8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.431+0000 7effb9400640 1 -- 172.21.15.45:0/3465971909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb41544b0 con 0x7eff9c004800 2024-04-16T16:14:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.432+0000 7effb9400640 1 --2- 172.21.15.45:0/3465971909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c004800 0x7effb4153ed0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7effa800e7e0 tx=0x7effa800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.432+0000 7effb1600640 1 -- 172.21.15.45:0/3465971909 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa800ee90 con 0x7eff9c004800 2024-04-16T16:14:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.432+0000 7effb1600640 1 -- 172.21.15.45:0/3465971909 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effa8018070 con 0x7eff9c004800 2024-04-16T16:14:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.432+0000 7effbbe00640 1 -- 172.21.15.45:0/3465971909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb40ca4f0 con 0x7eff9c004800 2024-04-16T16:14:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.432+0000 7effb1600640 1 -- 172.21.15.45:0/3465971909 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa800c6c0 con 0x7eff9c004800 2024-04-16T16:14:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.432+0000 7effbbe00640 1 -- 172.21.15.45:0/3465971909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb40caab0 con 0x7eff9c004800 2024-04-16T16:14:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.433+0000 7effbbe00640 1 -- 172.21.15.45:0/3465971909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb4075780 con 0x7eff9c004800 2024-04-16T16:14:13.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.434+0000 7effb1600640 1 -- 172.21.15.45:0/3465971909 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7effa800c860 con 0x7eff9c004800 2024-04-16T16:14:13.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.435+0000 7effb1600640 1 --2- 172.21.15.45:0/3465971909 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7eff9406ca90 0x7eff9406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:13.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.435+0000 7effb1600640 1 -- 172.21.15.45:0/3465971909 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7effa808ba50 con 0x7eff9c004800 2024-04-16T16:14:13.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.436+0000 7effb8a00640 1 --2- 172.21.15.45:0/3465971909 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7eff9406ca90 0x7eff9406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.437+0000 7effb8a00640 1 --2- 172.21.15.45:0/3465971909 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7eff9406ca90 0x7eff9406ef50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7effa0036000 tx=0x7effa0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:13.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.439+0000 7effb1600640 1 -- 172.21.15.45:0/3465971909 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7effa8059df0 con 0x7eff9c004800 2024-04-16T16:14:13.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.658+0000 7effbbe00640 1 -- 172.21.15.45:0/3465971909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7effb406f720 con 0x7eff9c004800 2024-04-16T16:14:13.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.660+0000 7effb1600640 1 -- 172.21.15.45:0/3465971909 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7effa80596f0 con 0x7eff9c004800 2024-04-16T16:14:13.659 INFO:teuthology.orchestra.run.smithi045.stderr:listed 0 entries 2024-04-16T16:14:13.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.662+0000 7effbbe00640 1 -- 172.21.15.45:0/3465971909 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7eff9406ca90 msgr2=0x7eff9406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:13.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.662+0000 7effbbe00640 1 --2- 172.21.15.45:0/3465971909 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7eff9406ca90 0x7eff9406ef50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7effa0036000 tx=0x7effa0030000 comp rx=0 tx=0).stop 2024-04-16T16:14:13.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.662+0000 7effbbe00640 1 -- 172.21.15.45:0/3465971909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c004800 msgr2=0x7effb4153ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:13.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.662+0000 7effbbe00640 1 --2- 172.21.15.45:0/3465971909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c004800 0x7effb4153ed0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7effa800e7e0 tx=0x7effa800eca0 comp rx=0 tx=0).stop 2024-04-16T16:14:13.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.662+0000 7effbbe00640 1 -- 172.21.15.45:0/3465971909 shutdown_connections 2024-04-16T16:14:13.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.662+0000 7effbbe00640 1 --2- 172.21.15.45:0/3465971909 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7eff9406ca90 0x7eff9406ef50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:13.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.662+0000 7effbbe00640 1 --2- 172.21.15.45:0/3465971909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effb40c8da0 0x7effb40c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:13.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.663+0000 7effbbe00640 1 --2- 172.21.15.45:0/3465971909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effb41548c0 0x7effb40c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:13.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.663+0000 7effbbe00640 1 --2- 172.21.15.45:0/3465971909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c004800 0x7effb4153ed0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:13.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.663+0000 7effbbe00640 1 -- 172.21.15.45:0/3465971909 >> 172.21.15.45:0/3465971909 conn(0x7effb406b930 msgr2=0x7effb4144e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:13.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.663+0000 7effbbe00640 1 -- 172.21.15.45:0/3465971909 shutdown_connections 2024-04-16T16:14:13.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:13.663+0000 7effbbe00640 1 -- 172.21.15.45:0/3465971909 wait complete. 2024-04-16T16:14:13.682 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-16T16:14:13.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:13.682 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-04-16T16:14:13.790 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:14:13.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:13.790 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-16T16:14:13.847 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:14:13.847 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:13.905 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:14:13.905 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:13.957 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:14:13.957 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:14.008 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:14.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.238+0000 7ff0b0a00640 1 Processor -- start 2024-04-16T16:14:14.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.239+0000 7ff0b0a00640 1 -- start start 2024-04-16T16:14:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.240+0000 7ff0b0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0ac071a40 0x7ff0ac0b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.241+0000 7ff0b0a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff0ac0b7750 0x7ff0ac15cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.241+0000 7ff0b0a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0ac1604d0 con 0x7ff0ac071e20 2024-04-16T16:14:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.241+0000 7ff0b0a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff0ac1607b0 con 0x7ff0ac0b7750 2024-04-16T16:14:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.241+0000 7ff0b0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff0ac160a90 con 0x7ff0ac071a40 2024-04-16T16:14:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.241+0000 7ff0aaa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff0ac0b7750 0x7ff0ac15cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.241+0000 7ff0aaa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff0ac0b7750 0x7ff0ac15cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52002/0 (socket says 172.21.15.45:52002) 2024-04-16T16:14:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.241+0000 7ff0aaa00640 1 -- 172.21.15.45:0/2327574647 learned_addr learned my addr 172.21.15.45:0/2327574647 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:14.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.241+0000 7ff0aaa00640 1 -- 172.21.15.45:0/2327574647 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0ac071a40 msgr2=0x7ff0ac0b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:14.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.241+0000 7ff0aaa00640 1 --2- 172.21.15.45:0/2327574647 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0ac071a40 0x7ff0ac0b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.241+0000 7ff0aaa00640 1 -- 172.21.15.45:0/2327574647 >> v1:172.21.15.103:6789/0 conn(0x7ff0ac071e20 legacy=0x7ff0ac0cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:14.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.241+0000 7ff0aaa00640 1 -- 172.21.15.45:0/2327574647 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0ac160d70 con 0x7ff0ac0b7750 2024-04-16T16:14:14.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.243+0000 7ff0aaa00640 1 --2- 172.21.15.45:0/2327574647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0ac0b7750 0x7ff0ac15cd40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff094009870 tx=0x7ff09402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14b32657a89e903 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:14.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.243+0000 7ff0aa000640 1 -- 172.21.15.45:0/2327574647 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff094035020 con 0x7ff0ac0b7750 2024-04-16T16:14:14.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.243+0000 7ff0aa000640 1 -- 172.21.15.45:0/2327574647 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff094031640 con 0x7ff0ac0b7750 2024-04-16T16:14:14.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.243+0000 7ff0aa000640 1 -- 172.21.15.45:0/2327574647 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09402d040 con 0x7ff0ac0b7750 2024-04-16T16:14:14.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.243+0000 7ff0b0a00640 1 -- 172.21.15.45:0/2327574647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0ac0b7750 msgr2=0x7ff0ac15cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:14.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.243+0000 7ff0b0a00640 1 --2- 172.21.15.45:0/2327574647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0ac0b7750 0x7ff0ac15cd40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff094009870 tx=0x7ff09402b8f0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.244+0000 7ff0b0a00640 1 -- 172.21.15.45:0/2327574647 shutdown_connections 2024-04-16T16:14:14.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.244+0000 7ff0b0a00640 1 --2- 172.21.15.45:0/2327574647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0ac0b7750 0x7ff0ac15cd40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.244+0000 7ff0b0a00640 1 --2- 172.21.15.45:0/2327574647 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0ac071a40 0x7ff0ac0b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.244+0000 7ff0b0a00640 1 -- 172.21.15.45:0/2327574647 >> 172.21.15.45:0/2327574647 conn(0x7ff0ac0b43a0 msgr2=0x7ff0ac0b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:14.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.244+0000 7ff0b0a00640 1 -- 172.21.15.45:0/2327574647 shutdown_connections 2024-04-16T16:14:14.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.244+0000 7ff0b0a00640 1 -- 172.21.15.45:0/2327574647 wait complete. 2024-04-16T16:14:14.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.245+0000 7ff0b0a00640 1 Processor -- start 2024-04-16T16:14:14.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.246+0000 7ff0b0a00640 1 -- start start 2024-04-16T16:14:14.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.247+0000 7ff0b0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0ac071a40 0x7ff0ac15cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:14.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.247+0000 7ff0b0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0ac071e20 0x7ff0ac06e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:14.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.247+0000 7ff0b0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0ac0b7750 0x7ff0ac148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:14.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.247+0000 7ff0b0a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ac072b20 con 0x7ff0ac071e20 2024-04-16T16:14:14.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.247+0000 7ff0b0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ac15a670 con 0x7ff0ac0b7750 2024-04-16T16:14:14.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.247+0000 7ff0b0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff0ac0722d0 con 0x7ff0ac071a40 2024-04-16T16:14:14.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.247+0000 7ff0aaa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0ac071e20 0x7ff0ac06e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:14.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.247+0000 7ff0aaa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0ac071e20 0x7ff0ac06e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52016/0 (socket says 172.21.15.45:52016) 2024-04-16T16:14:14.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.247+0000 7ff0aaa00640 1 -- 172.21.15.45:0/2049758662 learned_addr learned my addr 172.21.15.45:0/2049758662 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:14.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.247+0000 7ff0ab400640 1 --2- 172.21.15.45:0/2049758662 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0ac071a40 0x7ff0ac15cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:14.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.248+0000 7ff0aaa00640 1 -- 172.21.15.45:0/2049758662 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0ac071a40 msgr2=0x7ff0ac15cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:14.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.248+0000 7ff0abe00640 1 --2- 172.21.15.45:0/2049758662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0ac0b7750 0x7ff0ac148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:14.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.248+0000 7ff0aaa00640 1 --2- 172.21.15.45:0/2049758662 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0ac071a40 0x7ff0ac15cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.248+0000 7ff0aaa00640 1 -- 172.21.15.45:0/2049758662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0ac0b7750 msgr2=0x7ff0ac148080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:14.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.248+0000 7ff0aaa00640 1 --2- 172.21.15.45:0/2049758662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0ac0b7750 0x7ff0ac148080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.248+0000 7ff0aaa00640 1 -- 172.21.15.45:0/2049758662 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0ac06e820 con 0x7ff0ac071e20 2024-04-16T16:14:14.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.248+0000 7ff0ab400640 1 --2- 172.21.15.45:0/2049758662 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0ac071a40 0x7ff0ac15cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:14.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.248+0000 7ff0aaa00640 1 --2- 172.21.15.45:0/2049758662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0ac071e20 0x7ff0ac06e2e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff09402b930 tx=0x7ff0940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:14.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.248+0000 7ff0a3400640 1 -- 172.21.15.45:0/2049758662 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09403ac60 con 0x7ff0ac071e20 2024-04-16T16:14:14.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.248+0000 7ff0a3400640 1 -- 172.21.15.45:0/2049758662 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff09403ae20 con 0x7ff0ac071e20 2024-04-16T16:14:14.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.248+0000 7ff0b0a00640 1 -- 172.21.15.45:0/2049758662 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0ac06cbe0 con 0x7ff0ac071e20 2024-04-16T16:14:14.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.248+0000 7ff0a3400640 1 -- 172.21.15.45:0/2049758662 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff094044630 con 0x7ff0ac071e20 2024-04-16T16:14:14.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.249+0000 7ff0b0a00640 1 -- 172.21.15.45:0/2049758662 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0ac06d1a0 con 0x7ff0ac071e20 2024-04-16T16:14:14.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.250+0000 7ff0b0a00640 1 -- 172.21.15.45:0/2049758662 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0ac0b4f50 con 0x7ff0ac071e20 2024-04-16T16:14:14.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.251+0000 7ff0a3400640 1 -- 172.21.15.45:0/2049758662 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff094038020 con 0x7ff0ac071e20 2024-04-16T16:14:14.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.251+0000 7ff0a3400640 1 --2- 172.21.15.45:0/2049758662 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff08806c8f0 0x7ff08806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:14.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.252+0000 7ff0a3400640 1 -- 172.21.15.45:0/2049758662 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff0940ad9e0 con 0x7ff0ac071e20 2024-04-16T16:14:14.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.252+0000 7ff0ab400640 1 --2- 172.21.15.45:0/2049758662 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff08806c8f0 0x7ff08806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:14.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.253+0000 7ff0ab400640 1 --2- 172.21.15.45:0/2049758662 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff08806c8f0 0x7ff08806edb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff09c007eb0 tx=0x7ff09c0073c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:14.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.256+0000 7ff0a3400640 1 -- 172.21.15.45:0/2049758662 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff09403f070 con 0x7ff0ac071e20 2024-04-16T16:14:14.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.463+0000 7ff0b0a00640 1 -- 172.21.15.45:0/2049758662 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff0ac0008d0 con 0x7ff0ac071e20 2024-04-16T16:14:14.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.465+0000 7ff0a3400640 1 -- 172.21.15.45:0/2049758662 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16799 (secure 0 0 0) 0x7ff09407be10 con 0x7ff0ac071e20 2024-04-16T16:14:14.464 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:14.465 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":23,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:14:12.851304+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:14.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.468+0000 7ff0b0a00640 1 -- 172.21.15.45:0/2049758662 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff08806c8f0 msgr2=0x7ff08806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:14.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.468+0000 7ff0b0a00640 1 --2- 172.21.15.45:0/2049758662 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff08806c8f0 0x7ff08806edb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff09c007eb0 tx=0x7ff09c0073c0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.468+0000 7ff0b0a00640 1 -- 172.21.15.45:0/2049758662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0ac071e20 msgr2=0x7ff0ac06e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:14.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.468+0000 7ff0b0a00640 1 --2- 172.21.15.45:0/2049758662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0ac071e20 0x7ff0ac06e2e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff09402b930 tx=0x7ff0940046c0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.468+0000 7ff0b0a00640 1 -- 172.21.15.45:0/2049758662 shutdown_connections 2024-04-16T16:14:14.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.468+0000 7ff0b0a00640 1 --2- 172.21.15.45:0/2049758662 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff08806c8f0 0x7ff08806edb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.468+0000 7ff0b0a00640 1 --2- 172.21.15.45:0/2049758662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0ac0b7750 0x7ff0ac148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.468+0000 7ff0b0a00640 1 --2- 172.21.15.45:0/2049758662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0ac071e20 0x7ff0ac06e2e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.468+0000 7ff0b0a00640 1 --2- 172.21.15.45:0/2049758662 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0ac071a40 0x7ff0ac15cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.468+0000 7ff0b0a00640 1 -- 172.21.15.45:0/2049758662 >> 172.21.15.45:0/2049758662 conn(0x7ff0ac0b43a0 msgr2=0x7ff0ac151400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:14.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.468+0000 7ff0b0a00640 1 -- 172.21.15.45:0/2049758662 shutdown_connections 2024-04-16T16:14:14.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.468+0000 7ff0b0a00640 1 -- 172.21.15.45:0/2049758662 wait complete. 2024-04-16T16:14:14.488 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:13:23.600104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:13:24.368141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:14:14.489 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:14:14.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.715+0000 7fb1a3000640 1 Processor -- start 2024-04-16T16:14:14.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.715+0000 7fb1a3000640 1 -- start start 2024-04-16T16:14:14.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.717+0000 7fb1a3000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb19c0b5550 0x7fb19c06e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:14.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.717+0000 7fb1a3000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb19c0af530 con 0x7fb19c162df0 2024-04-16T16:14:14.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.717+0000 7fb1a3000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb19c0af250 con 0x7fb19c0b50c0 2024-04-16T16:14:14.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.717+0000 7fb1a3000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb19c06ede0 con 0x7fb19c0b5550 2024-04-16T16:14:14.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.717+0000 7fb1a1c00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb19c162df0 0x7fb19c0b49b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42954/0 (socket says 172.21.15.45:42954) 2024-04-16T16:14:14.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.717+0000 7fb1a1c00640 1 -- 172.21.15.45:0/778754275 learned_addr learned my addr 172.21.15.45:0/778754275 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:14.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.717+0000 7fb1a2600640 1 --2- 172.21.15.45:0/778754275 >> v2:172.21.15.45:3300/0 conn(0x7fb19c0b5550 0x7fb19c06e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:14.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.718+0000 7fb1a2600640 1 -- 172.21.15.45:0/778754275 >> v1:172.21.15.103:6789/0 conn(0x7fb19c0b50c0 legacy=0x7fb19c06c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:14.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.718+0000 7fb1a2600640 1 -- 172.21.15.45:0/778754275 >> v1:172.21.15.45:6789/0 conn(0x7fb19c162df0 legacy=0x7fb19c0b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:14.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.718+0000 7fb1a2600640 1 -- 172.21.15.45:0/778754275 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19c06bad0 con 0x7fb19c0b5550 2024-04-16T16:14:14.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.719+0000 7fb1a2600640 1 --2- 172.21.15.45:0/778754275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c0b5550 0x7fb19c06e8a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb1940097a0 tx=0x7fb19402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f5c01be458354c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:14.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.719+0000 7fb19be00640 1 -- 172.21.15.45:0/778754275 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194035020 con 0x7fb19c0b5550 2024-04-16T16:14:14.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.719+0000 7fb19be00640 1 -- 172.21.15.45:0/778754275 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb194031830 con 0x7fb19c0b5550 2024-04-16T16:14:14.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.719+0000 7fb19be00640 1 -- 172.21.15.45:0/778754275 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194031c00 con 0x7fb19c0b5550 2024-04-16T16:14:14.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.720+0000 7fb1a3000640 1 -- 172.21.15.45:0/778754275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c0b5550 msgr2=0x7fb19c06e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:14.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.720+0000 7fb1a3000640 1 --2- 172.21.15.45:0/778754275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c0b5550 0x7fb19c06e8a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb1940097a0 tx=0x7fb19402b8d0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.720+0000 7fb1a3000640 1 -- 172.21.15.45:0/778754275 shutdown_connections 2024-04-16T16:14:14.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.720+0000 7fb1a3000640 1 --2- 172.21.15.45:0/778754275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c0b5550 0x7fb19c06e8a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.720+0000 7fb1a3000640 1 -- 172.21.15.45:0/778754275 >> 172.21.15.45:0/778754275 conn(0x7fb19c0d0ed0 msgr2=0x7fb19c0d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:14.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.720+0000 7fb1a3000640 1 -- 172.21.15.45:0/778754275 shutdown_connections 2024-04-16T16:14:14.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.720+0000 7fb1a3000640 1 -- 172.21.15.45:0/778754275 wait complete. 2024-04-16T16:14:14.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.722+0000 7fb1a3000640 1 Processor -- start 2024-04-16T16:14:14.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.722+0000 7fb1a3000640 1 -- start start 2024-04-16T16:14:14.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.723+0000 7fb1a3000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb19c0b50c0 0x7fb19c1ed080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:14.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.723+0000 7fb1a3000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb19c0b5550 0x7fb19c15d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:14.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.723+0000 7fb1a3000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c162df0 0x7fb19c163260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:14.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.723+0000 7fb1a3000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb19c0aeeb0 con 0x7fb19c162df0 2024-04-16T16:14:14.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.723+0000 7fb1a3000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb19c06c090 con 0x7fb19c0b50c0 2024-04-16T16:14:14.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.723+0000 7fb1a3000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb19c0af030 con 0x7fb19c0b5550 2024-04-16T16:14:14.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.723+0000 7fb1a2600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c162df0 0x7fb19c163260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:14.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.723+0000 7fb1a2600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c162df0 0x7fb19c163260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52026/0 (socket says 172.21.15.45:52026) 2024-04-16T16:14:14.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.723+0000 7fb1a2600640 1 -- 172.21.15.45:0/2865045469 learned_addr learned my addr 172.21.15.45:0/2865045469 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:14.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.724+0000 7fb1a2600640 1 -- 172.21.15.45:0/2865045469 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb19c0b5550 msgr2=0x7fb19c15d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:14.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.724+0000 7fb1a2600640 1 --2- 172.21.15.45:0/2865045469 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb19c0b5550 0x7fb19c15d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.724+0000 7fb1a2600640 1 -- 172.21.15.45:0/2865045469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb19c0b50c0 msgr2=0x7fb19c1ed080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:14.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.724+0000 7fb1a2600640 1 --2- 172.21.15.45:0/2865045469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb19c0b50c0 0x7fb19c1ed080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.724+0000 7fb1a2600640 1 -- 172.21.15.45:0/2865045469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19c1653f0 con 0x7fb19c162df0 2024-04-16T16:14:14.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.724+0000 7fb1a2600640 1 --2- 172.21.15.45:0/2865045469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c162df0 0x7fb19c163260 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb1940098c0 tx=0x7fb1940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:14.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.724+0000 7fb199600640 1 -- 172.21.15.45:0/2865045469 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194041070 con 0x7fb19c162df0 2024-04-16T16:14:14.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.724+0000 7fb199600640 1 -- 172.21.15.45:0/2865045469 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb19403a8d0 con 0x7fb19c162df0 2024-04-16T16:14:14.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.724+0000 7fb1a3000640 1 -- 172.21.15.45:0/2865045469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb19c1656d0 con 0x7fb19c162df0 2024-04-16T16:14:14.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.724+0000 7fb199600640 1 -- 172.21.15.45:0/2865045469 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194035040 con 0x7fb19c162df0 2024-04-16T16:14:14.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.724+0000 7fb1a3000640 1 -- 172.21.15.45:0/2865045469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb19c165c90 con 0x7fb19c162df0 2024-04-16T16:14:14.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.726+0000 7fb1a3000640 1 -- 172.21.15.45:0/2865045469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb19c075780 con 0x7fb19c162df0 2024-04-16T16:14:14.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.727+0000 7fb199600640 1 -- 172.21.15.45:0/2865045469 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb194038020 con 0x7fb19c162df0 2024-04-16T16:14:14.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.728+0000 7fb199600640 1 --2- 172.21.15.45:0/2865045469 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb17c06c8f0 0x7fb17c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:14.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.728+0000 7fb199600640 1 -- 172.21.15.45:0/2865045469 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb1940ad700 con 0x7fb19c162df0 2024-04-16T16:14:14.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.728+0000 7fb1a1c00640 1 --2- 172.21.15.45:0/2865045469 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb17c06c8f0 0x7fb17c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:14.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.730+0000 7fb1a1c00640 1 --2- 172.21.15.45:0/2865045469 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb17c06c8f0 0x7fb17c06edb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb18c0012e0 tx=0x7fb18c001590 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:14.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.732+0000 7fb199600640 1 -- 172.21.15.45:0/2865045469 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb19407ba10 con 0x7fb19c162df0 2024-04-16T16:14:14.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.935+0000 7fb1a3000640 1 -- 172.21.15.45:0/2865045469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb19c166170 con 0x7fb19c162df0 2024-04-16T16:14:14.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.936+0000 7fb199600640 1 -- 172.21.15.45:0/2865045469 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7fb19c166170 con 0x7fb19c162df0 2024-04-16T16:14:14.935 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:14:14.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.939+0000 7fb1a3000640 1 -- 172.21.15.45:0/2865045469 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb17c06c8f0 msgr2=0x7fb17c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:14.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.939+0000 7fb1a3000640 1 --2- 172.21.15.45:0/2865045469 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb17c06c8f0 0x7fb17c06edb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb18c0012e0 tx=0x7fb18c001590 comp rx=0 tx=0).stop 2024-04-16T16:14:14.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.939+0000 7fb1a3000640 1 -- 172.21.15.45:0/2865045469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c162df0 msgr2=0x7fb19c163260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:14.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.939+0000 7fb1a3000640 1 --2- 172.21.15.45:0/2865045469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c162df0 0x7fb19c163260 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb1940098c0 tx=0x7fb1940046c0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.939+0000 7fb1a3000640 1 -- 172.21.15.45:0/2865045469 shutdown_connections 2024-04-16T16:14:14.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.939+0000 7fb1a3000640 1 --2- 172.21.15.45:0/2865045469 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb17c06c8f0 0x7fb17c06edb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.939+0000 7fb1a3000640 1 --2- 172.21.15.45:0/2865045469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c162df0 0x7fb19c163260 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.939+0000 7fb1a3000640 1 --2- 172.21.15.45:0/2865045469 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb19c0b5550 0x7fb19c15d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.939+0000 7fb1a3000640 1 --2- 172.21.15.45:0/2865045469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb19c0b50c0 0x7fb19c1ed080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:14.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.939+0000 7fb1a3000640 1 -- 172.21.15.45:0/2865045469 >> 172.21.15.45:0/2865045469 conn(0x7fb19c0d0ed0 msgr2=0x7fb19c0b49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:14.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.939+0000 7fb1a3000640 1 -- 172.21.15.45:0/2865045469 shutdown_connections 2024-04-16T16:14:14.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:14.939+0000 7fb1a3000640 1 -- 172.21.15.45:0/2865045469 wait complete. 2024-04-16T16:14:14.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:14:15.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.186+0000 7f96e3000640 1 Processor -- start 2024-04-16T16:14:15.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.186+0000 7f96e3000640 1 -- start start 2024-04-16T16:14:15.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.187+0000 7f96e3000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f96dc0d45e0 0x7f96dc169a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:15.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.187+0000 7f96e3000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96dc0d2b10 con 0x7f96dc15ecc0 2024-04-16T16:14:15.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.187+0000 7f96e3000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96dc163210 con 0x7f96dc06cf10 2024-04-16T16:14:15.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.187+0000 7f96e3000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f96dc1634f0 con 0x7f96dc0d45e0 2024-04-16T16:14:15.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.188+0000 7f96e1000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f96dc0d45e0 0x7f96dc169a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:15.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.188+0000 7f96e1000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f96dc0d45e0 0x7f96dc169a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52032/0 (socket says 172.21.15.45:52032) 2024-04-16T16:14:15.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.188+0000 7f96e1000640 1 -- 172.21.15.45:0/3364709528 learned_addr learned my addr 172.21.15.45:0/3364709528 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.188+0000 7f96e1000640 1 -- 172.21.15.45:0/3364709528 >> v1:172.21.15.103:6789/0 conn(0x7f96dc06cf10 legacy=0x7f96dc06d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.188+0000 7f96e1000640 1 -- 172.21.15.45:0/3364709528 >> v1:172.21.15.45:6789/0 conn(0x7f96dc15ecc0 legacy=0x7f96dc1626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.188+0000 7f96e1000640 1 -- 172.21.15.45:0/3364709528 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96dc0d2dd0 con 0x7f96dc0d45e0 2024-04-16T16:14:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.190+0000 7f96e1000640 1 --2- 172.21.15.45:0/3364709528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96dc0d45e0 0x7f96dc169a30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f96d4009e50 tx=0x7f96d402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3580816d39612c55 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:15.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.190+0000 7f96daa00640 1 -- 172.21.15.45:0/3364709528 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d402cbb0 con 0x7f96dc0d45e0 2024-04-16T16:14:15.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.190+0000 7f96daa00640 1 -- 172.21.15.45:0/3364709528 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96d402cd50 con 0x7f96dc0d45e0 2024-04-16T16:14:15.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.190+0000 7f96daa00640 1 -- 172.21.15.45:0/3364709528 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d4034610 con 0x7f96dc0d45e0 2024-04-16T16:14:15.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.190+0000 7f96e3000640 1 -- 172.21.15.45:0/3364709528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96dc0d45e0 msgr2=0x7f96dc169a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:15.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.190+0000 7f96e3000640 1 --2- 172.21.15.45:0/3364709528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96dc0d45e0 0x7f96dc169a30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f96d4009e50 tx=0x7f96d402c010 comp rx=0 tx=0).stop 2024-04-16T16:14:15.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.191+0000 7f96e3000640 1 -- 172.21.15.45:0/3364709528 shutdown_connections 2024-04-16T16:14:15.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.191+0000 7f96e3000640 1 --2- 172.21.15.45:0/3364709528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96dc0d45e0 0x7f96dc169a30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.191+0000 7f96e3000640 1 -- 172.21.15.45:0/3364709528 >> 172.21.15.45:0/3364709528 conn(0x7f96dc073690 msgr2=0x7f96dc0d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:15.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.191+0000 7f96e3000640 1 -- 172.21.15.45:0/3364709528 shutdown_connections 2024-04-16T16:14:15.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.191+0000 7f96e3000640 1 -- 172.21.15.45:0/3364709528 wait complete. 2024-04-16T16:14:15.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.192+0000 7f96e3000640 1 Processor -- start 2024-04-16T16:14:15.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.193+0000 7f96e3000640 1 -- start start 2024-04-16T16:14:15.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.193+0000 7f96e3000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f96dc06cf10 0x7f96dc0b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:15.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.193+0000 7f96e3000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96dc0d45e0 0x7f96dc0c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:15.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.193+0000 7f96e3000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96dc15ecc0 0x7f96dc0ca920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:15.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.193+0000 7f96e3000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc162db0 con 0x7f96dc15ecc0 2024-04-16T16:14:15.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.193+0000 7f96e3000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc0ba540 con 0x7f96dc0d45e0 2024-04-16T16:14:15.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.193+0000 7f96e3000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f96dc162f30 con 0x7f96dc06cf10 2024-04-16T16:14:15.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.194+0000 7f96e1000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96dc15ecc0 0x7f96dc0ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:15.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.194+0000 7f96e1000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96dc15ecc0 0x7f96dc0ca920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52036/0 (socket says 172.21.15.45:52036) 2024-04-16T16:14:15.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.194+0000 7f96e1000640 1 -- 172.21.15.45:0/1472056306 learned_addr learned my addr 172.21.15.45:0/1472056306 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:15.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.194+0000 7f96db400640 1 --2- 172.21.15.45:0/1472056306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96dc0d45e0 0x7f96dc0c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:15.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.194+0000 7f96dbe00640 1 --2- 172.21.15.45:0/1472056306 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f96dc06cf10 0x7f96dc0b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:15.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.194+0000 7f96e1000640 1 -- 172.21.15.45:0/1472056306 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f96dc06cf10 msgr2=0x7f96dc0b97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:15.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.194+0000 7f96e1000640 1 --2- 172.21.15.45:0/1472056306 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f96dc06cf10 0x7f96dc0b97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.194+0000 7f96e1000640 1 -- 172.21.15.45:0/1472056306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96dc0d45e0 msgr2=0x7f96dc0c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:15.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.194+0000 7f96e1000640 1 --2- 172.21.15.45:0/1472056306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96dc0d45e0 0x7f96dc0c84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.194+0000 7f96e1000640 1 -- 172.21.15.45:0/1472056306 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96dc0c8a50 con 0x7f96dc15ecc0 2024-04-16T16:14:15.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.195+0000 7f96e1000640 1 --2- 172.21.15.45:0/1472056306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96dc15ecc0 0x7f96dc0ca920 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f96d4036040 tx=0x7f96d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:15.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.195+0000 7f96d3e00640 1 -- 172.21.15.45:0/1472056306 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d4043020 con 0x7f96dc15ecc0 2024-04-16T16:14:15.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.195+0000 7f96d3e00640 1 -- 172.21.15.45:0/1472056306 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96d403e450 con 0x7f96dc15ecc0 2024-04-16T16:14:15.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.195+0000 7f96d3e00640 1 -- 172.21.15.45:0/1472056306 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d403e860 con 0x7f96dc15ecc0 2024-04-16T16:14:15.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.195+0000 7f96e3000640 1 -- 172.21.15.45:0/1472056306 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96dc0c8cc0 con 0x7f96dc15ecc0 2024-04-16T16:14:15.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.195+0000 7f96e3000640 1 -- 172.21.15.45:0/1472056306 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96dc0c9560 con 0x7f96dc15ecc0 2024-04-16T16:14:15.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.196+0000 7f96e3000640 1 -- 172.21.15.45:0/1472056306 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96dc075780 con 0x7f96dc15ecc0 2024-04-16T16:14:15.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.197+0000 7f96d3e00640 1 -- 172.21.15.45:0/1472056306 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f96d4046020 con 0x7f96dc15ecc0 2024-04-16T16:14:15.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.198+0000 7f96d3e00640 1 --2- 172.21.15.45:0/1472056306 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f96bc06c8f0 0x7f96bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:15.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.198+0000 7f96d3e00640 1 -- 172.21.15.45:0/1472056306 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f96d40ae5c0 con 0x7f96dc15ecc0 2024-04-16T16:14:15.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.199+0000 7f96dbe00640 1 --2- 172.21.15.45:0/1472056306 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f96bc06c8f0 0x7f96bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:15.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.200+0000 7f96dbe00640 1 --2- 172.21.15.45:0/1472056306 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f96bc06c8f0 0x7f96bc06edb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0015a0 tx=0x7f96cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:15.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.202+0000 7f96d3e00640 1 -- 172.21.15.45:0/1472056306 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f96d404d030 con 0x7f96dc15ecc0 2024-04-16T16:14:15.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.407+0000 7f96e3000640 1 -- 172.21.15.45:0/1472056306 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f96dc0d26c0 con 0x7f96dc15ecc0 2024-04-16T16:14:15.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.408+0000 7f96d3e00640 1 -- 172.21.15.45:0/1472056306 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7f96d407c8d0 con 0x7f96dc15ecc0 2024-04-16T16:14:15.407 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:14:15.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.411+0000 7f96e3000640 1 -- 172.21.15.45:0/1472056306 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f96bc06c8f0 msgr2=0x7f96bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:15.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.411+0000 7f96e3000640 1 --2- 172.21.15.45:0/1472056306 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f96bc06c8f0 0x7f96bc06edb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0015a0 tx=0x7f96cc009040 comp rx=0 tx=0).stop 2024-04-16T16:14:15.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.411+0000 7f96e3000640 1 -- 172.21.15.45:0/1472056306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96dc15ecc0 msgr2=0x7f96dc0ca920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:15.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.411+0000 7f96e3000640 1 --2- 172.21.15.45:0/1472056306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96dc15ecc0 0x7f96dc0ca920 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f96d4036040 tx=0x7f96d4004a40 comp rx=0 tx=0).stop 2024-04-16T16:14:15.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.411+0000 7f96e3000640 1 -- 172.21.15.45:0/1472056306 shutdown_connections 2024-04-16T16:14:15.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.411+0000 7f96e3000640 1 --2- 172.21.15.45:0/1472056306 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f96bc06c8f0 0x7f96bc06edb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.411+0000 7f96e3000640 1 --2- 172.21.15.45:0/1472056306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96dc15ecc0 0x7f96dc0ca920 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.411+0000 7f96e3000640 1 --2- 172.21.15.45:0/1472056306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96dc0d45e0 0x7f96dc0c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.411+0000 7f96e3000640 1 --2- 172.21.15.45:0/1472056306 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f96dc06cf10 0x7f96dc0b97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.411+0000 7f96e3000640 1 -- 172.21.15.45:0/1472056306 >> 172.21.15.45:0/1472056306 conn(0x7f96dc073690 msgr2=0x7f96dc15ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:15.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.412+0000 7f96e3000640 1 -- 172.21.15.45:0/1472056306 shutdown_connections 2024-04-16T16:14:15.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.412+0000 7f96e3000640 1 -- 172.21.15.45:0/1472056306 wait complete. 2024-04-16T16:14:15.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:14:15.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.657+0000 7febc4a00640 1 Processor -- start 2024-04-16T16:14:15.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.658+0000 7febc4a00640 1 -- start start 2024-04-16T16:14:15.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.659+0000 7febc4a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7febc01461b0 0x7febc0160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:15.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.659+0000 7febc4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febc00b58f0 0x7febc00b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:15.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.659+0000 7febc4a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febc0146590 con 0x7febc00cf440 2024-04-16T16:14:15.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.659+0000 7febc4a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7febc006dfd0 con 0x7febc01461b0 2024-04-16T16:14:15.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.659+0000 7febc4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7febc00b6290 con 0x7febc00b58f0 2024-04-16T16:14:15.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.660+0000 7febbf200640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7febc00cf440 0x7febc00cf8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:50870/0 (socket says 172.21.15.45:50870) 2024-04-16T16:14:15.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.660+0000 7febbf200640 1 -- 172.21.15.45:0/2288329394 learned_addr learned my addr 172.21.15.45:0/2288329394 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:15.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.660+0000 7febb7e00640 1 --2- 172.21.15.45:0/2288329394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febc00b58f0 0x7febc00b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:15.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.660+0000 7febbe800640 1 --2- 172.21.15.45:0/2288329394 >> v2:172.21.15.103:3300/0 conn(0x7febc01461b0 0x7febc0160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:15.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.661+0000 7febbde00640 1 -- 172.21.15.45:0/2288329394 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3578975750 0 0) 0x7febc0146590 con 0x7febc00cf440 2024-04-16T16:14:15.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.661+0000 7febb7e00640 1 -- 172.21.15.45:0/2288329394 >> v2:172.21.15.103:3300/0 conn(0x7febc01461b0 msgr2=0x7febc0160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:15.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.661+0000 7febb7e00640 1 --2- 172.21.15.45:0/2288329394 >> v2:172.21.15.103:3300/0 conn(0x7febc01461b0 0x7febc0160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.661+0000 7febb7e00640 1 -- 172.21.15.45:0/2288329394 >> v1:172.21.15.103:6789/0 conn(0x7febc00cf440 legacy=0x7febc00cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:15.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.661+0000 7febb7e00640 1 -- 172.21.15.45:0/2288329394 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc00c5750 con 0x7febc00b58f0 2024-04-16T16:14:15.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.663+0000 7febb7e00640 1 --2- 172.21.15.45:0/2288329394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febc00b58f0 0x7febc00b5d50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feba80097a0 tx=0x7feba802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7231385ab2125709 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:15.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.663+0000 7febbde00640 1 -- 172.21.15.45:0/2288329394 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba8035020 con 0x7febc00b58f0 2024-04-16T16:14:15.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.663+0000 7febbde00640 1 -- 172.21.15.45:0/2288329394 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feba8031810 con 0x7febc00b58f0 2024-04-16T16:14:15.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.663+0000 7febbde00640 1 -- 172.21.15.45:0/2288329394 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba8031be0 con 0x7febc00b58f0 2024-04-16T16:14:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.664+0000 7febc4a00640 1 -- 172.21.15.45:0/2288329394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febc00b58f0 msgr2=0x7febc00b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.664+0000 7febc4a00640 1 --2- 172.21.15.45:0/2288329394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febc00b58f0 0x7febc00b5d50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feba80097a0 tx=0x7feba802b8b0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.664+0000 7febc4a00640 1 -- 172.21.15.45:0/2288329394 shutdown_connections 2024-04-16T16:14:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.664+0000 7febc4a00640 1 --2- 172.21.15.45:0/2288329394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febc00b58f0 0x7febc00b5d50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.664+0000 7febc4a00640 1 --2- 172.21.15.45:0/2288329394 >> v2:172.21.15.103:3300/0 conn(0x7febc01461b0 0x7febc0160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.664+0000 7febc4a00640 1 -- 172.21.15.45:0/2288329394 >> 172.21.15.45:0/2288329394 conn(0x7febc006b930 msgr2=0x7febc006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.664+0000 7febc4a00640 1 -- 172.21.15.45:0/2288329394 shutdown_connections 2024-04-16T16:14:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.664+0000 7febc4a00640 1 -- 172.21.15.45:0/2288329394 wait complete. 2024-04-16T16:14:15.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.666+0000 7febc4a00640 1 Processor -- start 2024-04-16T16:14:15.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.666+0000 7febc4a00640 1 -- start start 2024-04-16T16:14:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.667+0000 7febc4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febc00b58f0 0x7febc015c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.667+0000 7febc4a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc00cf440 0x7febc00cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.667+0000 7febc4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febc01461b0 0x7febc00cc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.667+0000 7febc4a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7febc00c59f0 con 0x7febc00cf440 2024-04-16T16:14:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.667+0000 7febc4a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7febc0161230 con 0x7febc01461b0 2024-04-16T16:14:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.667+0000 7febc4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7febc00c5cd0 con 0x7febc00b58f0 2024-04-16T16:14:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.667+0000 7febb7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc00cf440 0x7febc00cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.667+0000 7febb7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc00cf440 0x7febc00cbdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52046/0 (socket says 172.21.15.45:52046) 2024-04-16T16:14:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.667+0000 7febb7e00640 1 -- 172.21.15.45:0/3811609900 learned_addr learned my addr 172.21.15.45:0/3811609900 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.668+0000 7febbe800640 1 --2- 172.21.15.45:0/3811609900 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febc00b58f0 0x7febc015c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.668+0000 7febbf200640 1 --2- 172.21.15.45:0/3811609900 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febc01461b0 0x7febc00cc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.668+0000 7febb7e00640 1 -- 172.21.15.45:0/3811609900 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febc00b58f0 msgr2=0x7febc015c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.668+0000 7febb7e00640 1 --2- 172.21.15.45:0/3811609900 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febc00b58f0 0x7febc015c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.668+0000 7febb7e00640 1 -- 172.21.15.45:0/3811609900 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febc01461b0 msgr2=0x7febc00cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.668+0000 7febb7e00640 1 --2- 172.21.15.45:0/3811609900 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febc01461b0 0x7febc00cc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.668+0000 7febb7e00640 1 -- 172.21.15.45:0/3811609900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc015c6f0 con 0x7febc00cf440 2024-04-16T16:14:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.668+0000 7febb7e00640 1 --2- 172.21.15.45:0/3811609900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc00cf440 0x7febc00cbdd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7feba80098c0 tx=0x7feba80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.668+0000 7febb6a00640 1 -- 172.21.15.45:0/3811609900 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba803bcf0 con 0x7febc00cf440 2024-04-16T16:14:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.669+0000 7febb6a00640 1 -- 172.21.15.45:0/3811609900 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feba80453f0 con 0x7febc00cf440 2024-04-16T16:14:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.669+0000 7febb6a00640 1 -- 172.21.15.45:0/3811609900 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba80457e0 con 0x7febc00cf440 2024-04-16T16:14:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.669+0000 7febc4a00640 1 -- 172.21.15.45:0/3811609900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febc01f47d0 con 0x7febc00cf440 2024-04-16T16:14:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.669+0000 7febc4a00640 1 -- 172.21.15.45:0/3811609900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febc01f4d90 con 0x7febc00cf440 2024-04-16T16:14:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.670+0000 7febc4a00640 1 -- 172.21.15.45:0/3811609900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febc0075780 con 0x7febc00cf440 2024-04-16T16:14:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.671+0000 7febb6a00640 1 -- 172.21.15.45:0/3811609900 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feba804c060 con 0x7febc00cf440 2024-04-16T16:14:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.672+0000 7febb6a00640 1 --2- 172.21.15.45:0/3811609900 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7feb9c06c8f0 0x7feb9c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.672+0000 7febb6a00640 1 -- 172.21.15.45:0/3811609900 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7feba80adaf0 con 0x7febc00cf440 2024-04-16T16:14:15.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.673+0000 7febbe800640 1 --2- 172.21.15.45:0/3811609900 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7feb9c06c8f0 0x7feb9c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.674+0000 7febbe800640 1 --2- 172.21.15.45:0/3811609900 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7feb9c06c8f0 0x7feb9c06edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7febb0009a30 tx=0x7febb0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.677+0000 7febb6a00640 1 -- 172.21.15.45:0/3811609900 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feba8040070 con 0x7febc00cf440 2024-04-16T16:14:15.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.877+0000 7febc4a00640 1 -- 172.21.15.45:0/3811609900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7febc015d270 con 0x7febc00cf440 2024-04-16T16:14:15.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.878+0000 7febb6a00640 1 -- 172.21.15.45:0/3811609900 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7feba804b2c0 con 0x7febc00cf440 2024-04-16T16:14:15.877 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:14:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.881+0000 7febc4a00640 1 -- 172.21.15.45:0/3811609900 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7feb9c06c8f0 msgr2=0x7feb9c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.881+0000 7febc4a00640 1 --2- 172.21.15.45:0/3811609900 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7feb9c06c8f0 0x7feb9c06edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7febb0009a30 tx=0x7febb0008040 comp rx=0 tx=0).stop 2024-04-16T16:14:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.881+0000 7febc4a00640 1 -- 172.21.15.45:0/3811609900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc00cf440 msgr2=0x7febc00cbdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.881+0000 7febc4a00640 1 --2- 172.21.15.45:0/3811609900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc00cf440 0x7febc00cbdd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7feba80098c0 tx=0x7feba80046c0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.881+0000 7febc4a00640 1 -- 172.21.15.45:0/3811609900 shutdown_connections 2024-04-16T16:14:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.881+0000 7febc4a00640 1 --2- 172.21.15.45:0/3811609900 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7feb9c06c8f0 0x7feb9c06edb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.881+0000 7febc4a00640 1 --2- 172.21.15.45:0/3811609900 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febc01461b0 0x7febc00cc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.881+0000 7febc4a00640 1 --2- 172.21.15.45:0/3811609900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc00cf440 0x7febc00cbdd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.881+0000 7febc4a00640 1 --2- 172.21.15.45:0/3811609900 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febc00b58f0 0x7febc015c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:15.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.881+0000 7febc4a00640 1 -- 172.21.15.45:0/3811609900 >> 172.21.15.45:0/3811609900 conn(0x7febc006b930 msgr2=0x7febc0145100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:15.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.881+0000 7febc4a00640 1 -- 172.21.15.45:0/3811609900 shutdown_connections 2024-04-16T16:14:15.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:15.881+0000 7febc4a00640 1 -- 172.21.15.45:0/3811609900 wait complete. 2024-04-16T16:14:15.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:14:16.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.112+0000 7ffa5f200640 1 Processor -- start 2024-04-16T16:14:16.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.112+0000 7ffa5f200640 1 -- start start 2024-04-16T16:14:16.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.113+0000 7ffa5f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa60162090 0x7ffa600cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:16.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.113+0000 7ffa5f200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffa600ba140 0x7ffa600cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.113+0000 7ffa5f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ffa600d5520 0x7ffa601628a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.113+0000 7ffa5f200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ffa600ba8b0 con 0x7ffa600ba140 2024-04-16T16:14:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.113+0000 7ffa5f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ffa600d2600 con 0x7ffa600d5520 2024-04-16T16:14:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.113+0000 7ffa5f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffa600ba5d0 con 0x7ffa60162090 2024-04-16T16:14:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.114+0000 7ffa5d400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffa600ba140 0x7ffa600cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.114+0000 7ffa5d400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffa600ba140 0x7ffa600cfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52054/0 (socket says 172.21.15.45:52054) 2024-04-16T16:14:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.114+0000 7ffa5d400640 1 -- 172.21.15.45:0/1873402386 learned_addr learned my addr 172.21.15.45:0/1873402386 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.114+0000 7ffa5de00640 1 --2- 172.21.15.45:0/1873402386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa60162090 0x7ffa600cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.114+0000 7ffa5e800640 1 --2- 172.21.15.45:0/1873402386 >> v2:172.21.15.103:3300/0 conn(0x7ffa600d5520 0x7ffa601628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.114+0000 7ffa5d400640 1 -- 172.21.15.45:0/1873402386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa60162090 msgr2=0x7ffa600cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.114+0000 7ffa5d400640 1 --2- 172.21.15.45:0/1873402386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa60162090 0x7ffa600cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.114+0000 7ffa5d400640 1 -- 172.21.15.45:0/1873402386 >> v2:172.21.15.103:3300/0 conn(0x7ffa600d5520 msgr2=0x7ffa601628a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.114+0000 7ffa5d400640 1 --2- 172.21.15.45:0/1873402386 >> v2:172.21.15.103:3300/0 conn(0x7ffa600d5520 0x7ffa601628a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.114+0000 7ffa5d400640 1 -- 172.21.15.45:0/1873402386 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa600d28a0 con 0x7ffa600ba140 2024-04-16T16:14:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.115+0000 7ffa5d400640 1 --2- 172.21.15.45:0/1873402386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa600ba140 0x7ffa600cfa50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffa600b7430 tx=0x7ffa4802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=939506f4eba5c4bf server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.116+0000 7ffa5ca00640 1 -- 172.21.15.45:0/1873402386 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4802ca70 con 0x7ffa600ba140 2024-04-16T16:14:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.116+0000 7ffa5ca00640 1 -- 172.21.15.45:0/1873402386 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa4802cc10 con 0x7ffa600ba140 2024-04-16T16:14:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.116+0000 7ffa5ca00640 1 -- 172.21.15.45:0/1873402386 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa480343f0 con 0x7ffa600ba140 2024-04-16T16:14:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.116+0000 7ffa5f200640 1 -- 172.21.15.45:0/1873402386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa600ba140 msgr2=0x7ffa600cfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.116+0000 7ffa5f200640 1 --2- 172.21.15.45:0/1873402386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa600ba140 0x7ffa600cfa50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffa600b7430 tx=0x7ffa4802bed0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.117+0000 7ffa5f200640 1 -- 172.21.15.45:0/1873402386 shutdown_connections 2024-04-16T16:14:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.117+0000 7ffa5f200640 1 --2- 172.21.15.45:0/1873402386 >> v2:172.21.15.103:3300/0 conn(0x7ffa600d5520 0x7ffa601628a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.117+0000 7ffa5f200640 1 --2- 172.21.15.45:0/1873402386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa600ba140 0x7ffa600cfa50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.117+0000 7ffa5f200640 1 --2- 172.21.15.45:0/1873402386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa60162090 0x7ffa600cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.117+0000 7ffa5f200640 1 -- 172.21.15.45:0/1873402386 >> 172.21.15.45:0/1873402386 conn(0x7ffa600d1940 msgr2=0x7ffa600d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.117+0000 7ffa5f200640 1 -- 172.21.15.45:0/1873402386 shutdown_connections 2024-04-16T16:14:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.117+0000 7ffa5f200640 1 -- 172.21.15.45:0/1873402386 wait complete. 2024-04-16T16:14:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.118+0000 7ffa5f200640 1 Processor -- start 2024-04-16T16:14:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.118+0000 7ffa5f200640 1 -- start start 2024-04-16T16:14:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.119+0000 7ffa5f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa600d5520 0x7ffa600cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:16.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.119+0000 7ffa5f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa60162090 0x7ffa601538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:16.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.119+0000 7ffa5f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa60153e30 0x7ffa600b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:16.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.119+0000 7ffa5f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffa601625f0 con 0x7ffa60153e30 2024-04-16T16:14:16.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.119+0000 7ffa5f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa60162470 con 0x7ffa60162090 2024-04-16T16:14:16.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.119+0000 7ffa5f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffa600b36c0 con 0x7ffa600d5520 2024-04-16T16:14:16.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.120+0000 7ffa5e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa60153e30 0x7ffa600b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:16.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.120+0000 7ffa5e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa60153e30 0x7ffa600b5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52064/0 (socket says 172.21.15.45:52064) 2024-04-16T16:14:16.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.120+0000 7ffa5e800640 1 -- 172.21.15.45:0/397792897 learned_addr learned my addr 172.21.15.45:0/397792897 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:16.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.120+0000 7ffa5d400640 1 --2- 172.21.15.45:0/397792897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa60162090 0x7ffa601538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:16.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.120+0000 7ffa5de00640 1 --2- 172.21.15.45:0/397792897 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa600d5520 0x7ffa600cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:16.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.120+0000 7ffa5e800640 1 -- 172.21.15.45:0/397792897 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa600d5520 msgr2=0x7ffa600cabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:16.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.120+0000 7ffa5e800640 1 --2- 172.21.15.45:0/397792897 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa600d5520 0x7ffa600cabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.120+0000 7ffa5e800640 1 -- 172.21.15.45:0/397792897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa60162090 msgr2=0x7ffa601538f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:16.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.120+0000 7ffa5e800640 1 --2- 172.21.15.45:0/397792897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa60162090 0x7ffa601538f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.120+0000 7ffa5e800640 1 -- 172.21.15.45:0/397792897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa600c83d0 con 0x7ffa60153e30 2024-04-16T16:14:16.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.121+0000 7ffa5e800640 1 --2- 172.21.15.45:0/397792897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa60153e30 0x7ffa600b5030 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800d5f0 tx=0x7ffa5800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:16.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.121+0000 7ffa56000640 1 -- 172.21.15.45:0/397792897 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa580125f0 con 0x7ffa60153e30 2024-04-16T16:14:16.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.121+0000 7ffa56000640 1 -- 172.21.15.45:0/397792897 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa58019070 con 0x7ffa60153e30 2024-04-16T16:14:16.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.121+0000 7ffa56000640 1 -- 172.21.15.45:0/397792897 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa58012a10 con 0x7ffa60153e30 2024-04-16T16:14:16.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.121+0000 7ffa5f200640 1 -- 172.21.15.45:0/397792897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa600b5cf0 con 0x7ffa60153e30 2024-04-16T16:14:16.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.121+0000 7ffa5f200640 1 -- 172.21.15.45:0/397792897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa600b5fd0 con 0x7ffa60153e30 2024-04-16T16:14:16.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.122+0000 7ffa5f200640 1 -- 172.21.15.45:0/397792897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa60075780 con 0x7ffa60153e30 2024-04-16T16:14:16.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.123+0000 7ffa56000640 1 -- 172.21.15.45:0/397792897 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffa58012bb0 con 0x7ffa60153e30 2024-04-16T16:14:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.125+0000 7ffa56000640 1 --2- 172.21.15.45:0/397792897 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ffa3c06c8f0 0x7ffa3c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.125+0000 7ffa56000640 1 -- 172.21.15.45:0/397792897 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ffa5808c070 con 0x7ffa60153e30 2024-04-16T16:14:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.126+0000 7ffa5de00640 1 --2- 172.21.15.45:0/397792897 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ffa3c06c8f0 0x7ffa3c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.127+0000 7ffa5de00640 1 --2- 172.21.15.45:0/397792897 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ffa3c06c8f0 0x7ffa3c06edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ffa50006fd0 tx=0x7ffa50008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:16.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.129+0000 7ffa56000640 1 -- 172.21.15.45:0/397792897 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffa5805a380 con 0x7ffa60153e30 2024-04-16T16:14:16.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.346+0000 7ffa5f200640 1 -- 172.21.15.45:0/397792897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ffa6006dab0 con 0x7ffa60153e30 2024-04-16T16:14:16.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.347+0000 7ffa56000640 1 -- 172.21.15.45:0/397792897 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7ffa58059c80 con 0x7ffa60153e30 2024-04-16T16:14:16.346 INFO:teuthology.orchestra.run.smithi045.stderr:listed 0 entries 2024-04-16T16:14:16.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.349+0000 7ffa5f200640 1 -- 172.21.15.45:0/397792897 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ffa3c06c8f0 msgr2=0x7ffa3c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:16.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.349+0000 7ffa5f200640 1 --2- 172.21.15.45:0/397792897 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ffa3c06c8f0 0x7ffa3c06edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ffa50006fd0 tx=0x7ffa50008040 comp rx=0 tx=0).stop 2024-04-16T16:14:16.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.349+0000 7ffa5f200640 1 -- 172.21.15.45:0/397792897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa60153e30 msgr2=0x7ffa600b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:16.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.349+0000 7ffa5f200640 1 --2- 172.21.15.45:0/397792897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa60153e30 0x7ffa600b5030 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800d5f0 tx=0x7ffa5800dab0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.350+0000 7ffa5f200640 1 -- 172.21.15.45:0/397792897 shutdown_connections 2024-04-16T16:14:16.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.350+0000 7ffa5f200640 1 --2- 172.21.15.45:0/397792897 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ffa3c06c8f0 0x7ffa3c06edb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.350+0000 7ffa5f200640 1 --2- 172.21.15.45:0/397792897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa60153e30 0x7ffa600b5030 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.350+0000 7ffa5f200640 1 --2- 172.21.15.45:0/397792897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa60162090 0x7ffa601538f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.350+0000 7ffa5f200640 1 --2- 172.21.15.45:0/397792897 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa600d5520 0x7ffa600cabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.350+0000 7ffa5f200640 1 -- 172.21.15.45:0/397792897 >> 172.21.15.45:0/397792897 conn(0x7ffa600d1940 msgr2=0x7ffa600b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:16.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.350+0000 7ffa5f200640 1 -- 172.21.15.45:0/397792897 shutdown_connections 2024-04-16T16:14:16.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.350+0000 7ffa5f200640 1 -- 172.21.15.45:0/397792897 wait complete. 2024-04-16T16:14:16.367 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-16T16:14:16.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:16.367 DEBUG:teuthology.orchestra.run.smithi103:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-04-16T16:14:16.474 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:14:16.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:16.474 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-16T16:14:16.528 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:14:16.528 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:16.587 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:14:16.588 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:16.644 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:14:16.644 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:16.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:16.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.920+0000 7fa99e000640 1 Processor -- start 2024-04-16T16:14:16.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.920+0000 7fa99e000640 1 -- start start 2024-04-16T16:14:16.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.921+0000 7fa99e000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa998161e90 0x7fa9980cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:16.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.921+0000 7fa99e000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa998162270 0x7fa9980d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:16.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.921+0000 7fa99e000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9980d3080 con 0x7fa9980d52d0 2024-04-16T16:14:16.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.921+0000 7fa99e000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa9980d2e30 con 0x7fa998161e90 2024-04-16T16:14:16.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.921+0000 7fa99e000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa9980d3360 con 0x7fa998162270 2024-04-16T16:14:16.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.921+0000 7fa99cc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa998161e90 0x7fa9980cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:16.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.921+0000 7fa99cc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa998161e90 0x7fa9980cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52068/0 (socket says 172.21.15.45:52068) 2024-04-16T16:14:16.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.921+0000 7fa99cc00640 1 -- 172.21.15.45:0/3334301392 learned_addr learned my addr 172.21.15.45:0/3334301392 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:16.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.922+0000 7fa99cc00640 1 -- 172.21.15.45:0/3334301392 >> v2:172.21.15.103:3300/0 conn(0x7fa998162270 msgr2=0x7fa9980d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:16.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.922+0000 7fa99cc00640 1 --2- 172.21.15.45:0/3334301392 >> v2:172.21.15.103:3300/0 conn(0x7fa998162270 0x7fa9980d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.922+0000 7fa99cc00640 1 -- 172.21.15.45:0/3334301392 >> v1:172.21.15.103:6789/0 conn(0x7fa9980d52d0 legacy=0x7fa99806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:16.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.922+0000 7fa99cc00640 1 -- 172.21.15.45:0/3334301392 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9980d3640 con 0x7fa998161e90 2024-04-16T16:14:16.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.923+0000 7fa99cc00640 1 --2- 172.21.15.45:0/3334301392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa998161e90 0x7fa9980cf7d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa988009870 tx=0x7fa98802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd256696505280ce server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:16.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.923+0000 7fa997400640 1 -- 172.21.15.45:0/3334301392 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa988035020 con 0x7fa998161e90 2024-04-16T16:14:16.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.923+0000 7fa997400640 1 -- 172.21.15.45:0/3334301392 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa988031870 con 0x7fa998161e90 2024-04-16T16:14:16.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.923+0000 7fa997400640 1 -- 172.21.15.45:0/3334301392 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa988031c40 con 0x7fa998161e90 2024-04-16T16:14:16.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.924+0000 7fa99e000640 1 -- 172.21.15.45:0/3334301392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa998161e90 msgr2=0x7fa9980cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:16.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.924+0000 7fa99e000640 1 --2- 172.21.15.45:0/3334301392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa998161e90 0x7fa9980cf7d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa988009870 tx=0x7fa98802b910 comp rx=0 tx=0).stop 2024-04-16T16:14:16.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.924+0000 7fa99e000640 1 -- 172.21.15.45:0/3334301392 shutdown_connections 2024-04-16T16:14:16.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.924+0000 7fa99e000640 1 --2- 172.21.15.45:0/3334301392 >> v2:172.21.15.103:3300/0 conn(0x7fa998162270 0x7fa9980d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.924+0000 7fa99e000640 1 --2- 172.21.15.45:0/3334301392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa998161e90 0x7fa9980cf7d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.924+0000 7fa99e000640 1 -- 172.21.15.45:0/3334301392 >> 172.21.15.45:0/3334301392 conn(0x7fa9980d1940 msgr2=0x7fa9980d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:16.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.924+0000 7fa99e000640 1 -- 172.21.15.45:0/3334301392 shutdown_connections 2024-04-16T16:14:16.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.924+0000 7fa99e000640 1 -- 172.21.15.45:0/3334301392 wait complete. 2024-04-16T16:14:16.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.926+0000 7fa99e000640 1 Processor -- start 2024-04-16T16:14:16.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.926+0000 7fa99e000640 1 -- start start 2024-04-16T16:14:16.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.927+0000 7fa99e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9980d52d0 0x7fa9981603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:16.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.927+0000 7fa99e000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa998161e90 0x7fa998152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:16.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.927+0000 7fa99e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa998162270 0x7fa9981e8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:16.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.927+0000 7fa99e000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa99806d550 con 0x7fa9980d52d0 2024-04-16T16:14:16.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.927+0000 7fa99e000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9980cfd10 con 0x7fa998161e90 2024-04-16T16:14:16.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.927+0000 7fa99e000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa99806d6d0 con 0x7fa998162270 2024-04-16T16:14:16.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.927+0000 7fa99cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9980d52d0 0x7fa9981603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:16.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.928+0000 7fa99cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9980d52d0 0x7fa9981603d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52070/0 (socket says 172.21.15.45:52070) 2024-04-16T16:14:16.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.928+0000 7fa99cc00640 1 -- 172.21.15.45:0/2713190873 learned_addr learned my addr 172.21.15.45:0/2713190873 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:16.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.928+0000 7fa997e00640 1 --2- 172.21.15.45:0/2713190873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa998161e90 0x7fa998152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:16.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.928+0000 7fa99cc00640 1 -- 172.21.15.45:0/2713190873 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa998162270 msgr2=0x7fa9981e8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:16.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.928+0000 7fa99cc00640 1 --2- 172.21.15.45:0/2713190873 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa998162270 0x7fa9981e8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.928+0000 7fa99cc00640 1 -- 172.21.15.45:0/2713190873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa998161e90 msgr2=0x7fa998152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:16.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.928+0000 7fa99cc00640 1 --2- 172.21.15.45:0/2713190873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa998161e90 0x7fa998152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:16.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.928+0000 7fa99cc00640 1 -- 172.21.15.45:0/2713190873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9981e7780 con 0x7fa9980d52d0 2024-04-16T16:14:16.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.928+0000 7fa99cc00640 1 --2- 172.21.15.45:0/2713190873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9980d52d0 0x7fa9981603d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa988009870 tx=0x7fa9880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:16.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.929+0000 7fa994c00640 1 -- 172.21.15.45:0/2713190873 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98803ad60 con 0x7fa9980d52d0 2024-04-16T16:14:16.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.929+0000 7fa994c00640 1 -- 172.21.15.45:0/2713190873 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9880443f0 con 0x7fa9980d52d0 2024-04-16T16:14:16.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.929+0000 7fa994c00640 1 -- 172.21.15.45:0/2713190873 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9880447e0 con 0x7fa9980d52d0 2024-04-16T16:14:16.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.929+0000 7fa99e000640 1 -- 172.21.15.45:0/2713190873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9981e7a60 con 0x7fa9980d52d0 2024-04-16T16:14:16.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.929+0000 7fa99e000640 1 -- 172.21.15.45:0/2713190873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9981e8020 con 0x7fa9980d52d0 2024-04-16T16:14:16.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.930+0000 7fa99e000640 1 -- 172.21.15.45:0/2713190873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa998075780 con 0x7fa9980d52d0 2024-04-16T16:14:16.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.931+0000 7fa994c00640 1 -- 172.21.15.45:0/2713190873 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa98804b060 con 0x7fa9980d52d0 2024-04-16T16:14:16.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.932+0000 7fa994c00640 1 --2- 172.21.15.45:0/2713190873 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa97406c920 0x7fa97406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:16.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.932+0000 7fa994c00640 1 -- 172.21.15.45:0/2713190873 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa9880ade10 con 0x7fa9980d52d0 2024-04-16T16:14:16.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.933+0000 7fa997e00640 1 --2- 172.21.15.45:0/2713190873 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa97406c920 0x7fa97406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:16.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.934+0000 7fa997e00640 1 --2- 172.21.15.45:0/2713190873 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa97406c920 0x7fa97406ede0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa9981e9280 tx=0x7fa9800094f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:16.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:16.936+0000 7fa994c00640 1 -- 172.21.15.45:0/2713190873 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa9880b1030 con 0x7fa9980d52d0 2024-04-16T16:14:17.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.142+0000 7fa99e000640 1 -- 172.21.15.45:0/2713190873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa9980b75a0 con 0x7fa9980d52d0 2024-04-16T16:14:17.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.145+0000 7fa994c00640 1 -- 172.21.15.45:0/2713190873 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16799 (secure 0 0 0) 0x7fa98803f070 con 0x7fa9980d52d0 2024-04-16T16:14:17.146 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:17.146 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":23,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:14:12.851304+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:17.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.147+0000 7fa99e000640 1 -- 172.21.15.45:0/2713190873 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa97406c920 msgr2=0x7fa97406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:17.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.147+0000 7fa99e000640 1 --2- 172.21.15.45:0/2713190873 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa97406c920 0x7fa97406ede0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa9981e9280 tx=0x7fa9800094f0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.148+0000 7fa99e000640 1 -- 172.21.15.45:0/2713190873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9980d52d0 msgr2=0x7fa9981603d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:17.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.148+0000 7fa99e000640 1 --2- 172.21.15.45:0/2713190873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9980d52d0 0x7fa9981603d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa988009870 tx=0x7fa9880046c0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.148+0000 7fa99e000640 1 -- 172.21.15.45:0/2713190873 shutdown_connections 2024-04-16T16:14:17.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.148+0000 7fa99e000640 1 --2- 172.21.15.45:0/2713190873 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa97406c920 0x7fa97406ede0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.148+0000 7fa99e000640 1 --2- 172.21.15.45:0/2713190873 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa998162270 0x7fa9981e8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.148+0000 7fa99e000640 1 --2- 172.21.15.45:0/2713190873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa998161e90 0x7fa998152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.148+0000 7fa99e000640 1 --2- 172.21.15.45:0/2713190873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9980d52d0 0x7fa9981603d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.148+0000 7fa99e000640 1 -- 172.21.15.45:0/2713190873 >> 172.21.15.45:0/2713190873 conn(0x7fa9980d1940 msgr2=0x7fa9980d3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:17.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.148+0000 7fa99e000640 1 -- 172.21.15.45:0/2713190873 shutdown_connections 2024-04-16T16:14:17.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.148+0000 7fa99e000640 1 -- 172.21.15.45:0/2713190873 wait complete. 2024-04-16T16:14:17.168 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:13:23.600104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:13:24.368141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:14:17.168 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:14:17.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.395+0000 7f2c67e00640 1 Processor -- start 2024-04-16T16:14:17.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.396+0000 7f2c67e00640 1 -- start start 2024-04-16T16:14:17.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.397+0000 7f2c67e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2c680d03b0 0x7f2c6809ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:17.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.397+0000 7f2c67e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c680d0920 con 0x7f2c6806d930 2024-04-16T16:14:17.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.398+0000 7f2c67e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6809b390 con 0x7f2c6809aa70 2024-04-16T16:14:17.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.398+0000 7f2c67e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2c6806cb70 con 0x7f2c680d03b0 2024-04-16T16:14:17.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.398+0000 7f2c66000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2c680d03b0 0x7f2c6809ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:17.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.398+0000 7f2c67400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2c6806d930 0x7f2c6806dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42968/0 (socket says 172.21.15.45:42968) 2024-04-16T16:14:17.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.398+0000 7f2c66000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2c680d03b0 0x7f2c6809ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52072/0 (socket says 172.21.15.45:52072) 2024-04-16T16:14:17.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.398+0000 7f2c67400640 1 -- 172.21.15.45:0/1299266138 learned_addr learned my addr 172.21.15.45:0/1299266138 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:17.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.399+0000 7f2c66000640 1 -- 172.21.15.45:0/1299266138 >> v1:172.21.15.103:6789/0 conn(0x7f2c6809aa70 legacy=0x7f2c6809b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:17.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.399+0000 7f2c66000640 1 -- 172.21.15.45:0/1299266138 >> v1:172.21.15.45:6789/0 conn(0x7f2c6806d930 legacy=0x7f2c6806dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:17.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.399+0000 7f2c66000640 1 -- 172.21.15.45:0/1299266138 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6806ce50 con 0x7f2c680d03b0 2024-04-16T16:14:17.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.400+0000 7f2c66000640 1 --2- 172.21.15.45:0/1299266138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c680d03b0 0x7f2c6809ae50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009d90 tx=0x7f2c5802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d904ce11b0da364d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:17.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.401+0000 7f2c65600640 1 -- 172.21.15.45:0/1299266138 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c58039070 con 0x7f2c680d03b0 2024-04-16T16:14:17.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.401+0000 7f2c65600640 1 -- 172.21.15.45:0/1299266138 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c5802cc60 con 0x7f2c680d03b0 2024-04-16T16:14:17.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.401+0000 7f2c65600640 1 -- 172.21.15.45:0/1299266138 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c58034660 con 0x7f2c680d03b0 2024-04-16T16:14:17.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.401+0000 7f2c67e00640 1 -- 172.21.15.45:0/1299266138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c680d03b0 msgr2=0x7f2c6809ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:17.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.401+0000 7f2c67e00640 1 --2- 172.21.15.45:0/1299266138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c680d03b0 0x7f2c6809ae50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009d90 tx=0x7f2c5802beb0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.402+0000 7f2c67e00640 1 -- 172.21.15.45:0/1299266138 shutdown_connections 2024-04-16T16:14:17.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.402+0000 7f2c67e00640 1 --2- 172.21.15.45:0/1299266138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c680d03b0 0x7f2c6809ae50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.402+0000 7f2c67e00640 1 -- 172.21.15.45:0/1299266138 >> 172.21.15.45:0/1299266138 conn(0x7f2c680b40c0 msgr2=0x7f2c680b44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:17.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.402+0000 7f2c67e00640 1 -- 172.21.15.45:0/1299266138 shutdown_connections 2024-04-16T16:14:17.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.402+0000 7f2c67e00640 1 -- 172.21.15.45:0/1299266138 wait complete. 2024-04-16T16:14:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.404+0000 7f2c67e00640 1 Processor -- start 2024-04-16T16:14:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.404+0000 7f2c67e00640 1 -- start start 2024-04-16T16:14:17.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.406+0000 7f2c67e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c6806d930 0x7f2c680ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:17.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.406+0000 7f2c67e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c6809aa70 0x7f2c6815b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:17.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.406+0000 7f2c67e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c680d03b0 0x7f2c68160310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:17.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.406+0000 7f2c67e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6815ede0 con 0x7f2c6809aa70 2024-04-16T16:14:17.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.406+0000 7f2c67e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6806d410 con 0x7f2c680d03b0 2024-04-16T16:14:17.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.406+0000 7f2c67e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2c6815ef60 con 0x7f2c6806d930 2024-04-16T16:14:17.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.406+0000 7f2c66000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c6809aa70 0x7f2c6815b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:17.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.406+0000 7f2c66000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c6809aa70 0x7f2c6815b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52074/0 (socket says 172.21.15.45:52074) 2024-04-16T16:14:17.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.406+0000 7f2c66000640 1 -- 172.21.15.45:0/3192248774 learned_addr learned my addr 172.21.15.45:0/3192248774 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:17.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.406+0000 7f2c66a00640 1 --2- 172.21.15.45:0/3192248774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c6806d930 0x7f2c680ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:17.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.406+0000 7f2c67400640 1 --2- 172.21.15.45:0/3192248774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c680d03b0 0x7f2c68160310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:17.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.407+0000 7f2c66000640 1 -- 172.21.15.45:0/3192248774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c6806d930 msgr2=0x7f2c680ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:17.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.407+0000 7f2c66000640 1 --2- 172.21.15.45:0/3192248774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c6806d930 0x7f2c680ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.407+0000 7f2c66000640 1 -- 172.21.15.45:0/3192248774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c680d03b0 msgr2=0x7f2c68160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:17.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.407+0000 7f2c66000640 1 --2- 172.21.15.45:0/3192248774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c680d03b0 0x7f2c68160310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.407+0000 7f2c66000640 1 -- 172.21.15.45:0/3192248774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c680cb9a0 con 0x7f2c6809aa70 2024-04-16T16:14:17.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.407+0000 7f2c66000640 1 --2- 172.21.15.45:0/3192248774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c6809aa70 0x7f2c6815b8d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009d90 tx=0x7f2c58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:17.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.408+0000 7f2c5ea00640 1 -- 172.21.15.45:0/3192248774 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c58039050 con 0x7f2c6809aa70 2024-04-16T16:14:17.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.408+0000 7f2c5ea00640 1 -- 172.21.15.45:0/3192248774 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c58045070 con 0x7f2c6809aa70 2024-04-16T16:14:17.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.408+0000 7f2c5ea00640 1 -- 172.21.15.45:0/3192248774 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5803ea50 con 0x7f2c6809aa70 2024-04-16T16:14:17.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.408+0000 7f2c67e00640 1 -- 172.21.15.45:0/3192248774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c680ca900 con 0x7f2c6809aa70 2024-04-16T16:14:17.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.408+0000 7f2c67e00640 1 -- 172.21.15.45:0/3192248774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c680caec0 con 0x7f2c6809aa70 2024-04-16T16:14:17.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.409+0000 7f2c67e00640 1 -- 172.21.15.45:0/3192248774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c680ce740 con 0x7f2c6809aa70 2024-04-16T16:14:17.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.410+0000 7f2c5ea00640 1 -- 172.21.15.45:0/3192248774 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2c5802cae0 con 0x7f2c6809aa70 2024-04-16T16:14:17.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.411+0000 7f2c5ea00640 1 --2- 172.21.15.45:0/3192248774 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2c4406c920 0x7f2c4406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:17.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.411+0000 7f2c5ea00640 1 -- 172.21.15.45:0/3192248774 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2c580ae480 con 0x7f2c6809aa70 2024-04-16T16:14:17.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.412+0000 7f2c66a00640 1 --2- 172.21.15.45:0/3192248774 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2c4406c920 0x7f2c4406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:17.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.413+0000 7f2c66a00640 1 --2- 172.21.15.45:0/3192248774 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2c4406c920 0x7f2c4406ede0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2c50000f60 tx=0x7f2c50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:17.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.418+0000 7f2c5ea00640 1 -- 172.21.15.45:0/3192248774 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2c5804e020 con 0x7f2c6809aa70 2024-04-16T16:14:17.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.622+0000 7f2c67e00640 1 -- 172.21.15.45:0/3192248774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2c680b1780 con 0x7f2c6809aa70 2024-04-16T16:14:17.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.623+0000 7f2c5ea00640 1 -- 172.21.15.45:0/3192248774 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7f2c5807c820 con 0x7f2c6809aa70 2024-04-16T16:14:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:14:17.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.626+0000 7f2c67e00640 1 -- 172.21.15.45:0/3192248774 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2c4406c920 msgr2=0x7f2c4406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:17.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.626+0000 7f2c67e00640 1 --2- 172.21.15.45:0/3192248774 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2c4406c920 0x7f2c4406ede0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2c50000f60 tx=0x7f2c50009040 comp rx=0 tx=0).stop 2024-04-16T16:14:17.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.626+0000 7f2c67e00640 1 -- 172.21.15.45:0/3192248774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c6809aa70 msgr2=0x7f2c6815b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:17.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.626+0000 7f2c67e00640 1 --2- 172.21.15.45:0/3192248774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c6809aa70 0x7f2c6815b8d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009d90 tx=0x7f2c58004a40 comp rx=0 tx=0).stop 2024-04-16T16:14:17.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.626+0000 7f2c67e00640 1 -- 172.21.15.45:0/3192248774 shutdown_connections 2024-04-16T16:14:17.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.626+0000 7f2c67e00640 1 --2- 172.21.15.45:0/3192248774 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2c4406c920 0x7f2c4406ede0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.626+0000 7f2c67e00640 1 --2- 172.21.15.45:0/3192248774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c680d03b0 0x7f2c68160310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.626+0000 7f2c67e00640 1 --2- 172.21.15.45:0/3192248774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c6809aa70 0x7f2c6815b8d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.626+0000 7f2c67e00640 1 --2- 172.21.15.45:0/3192248774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c6806d930 0x7f2c680ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.626+0000 7f2c67e00640 1 -- 172.21.15.45:0/3192248774 >> 172.21.15.45:0/3192248774 conn(0x7f2c680b40c0 msgr2=0x7f2c6809ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.626+0000 7f2c67e00640 1 -- 172.21.15.45:0/3192248774 shutdown_connections 2024-04-16T16:14:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.626+0000 7f2c67e00640 1 -- 172.21.15.45:0/3192248774 wait complete. 2024-04-16T16:14:17.645 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:14:17.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.869+0000 7f869da00640 1 Processor -- start 2024-04-16T16:14:17.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.869+0000 7f869da00640 1 -- start start 2024-04-16T16:14:17.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.870+0000 7f869da00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f86980d0010 0x7f8698166550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:17.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.870+0000 7f869da00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86980cdc90 con 0x7f869815b430 2024-04-16T16:14:17.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.870+0000 7f869da00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86980cdf60 con 0x7f869815b810 2024-04-16T16:14:17.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.870+0000 7f869da00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f86980ce240 con 0x7f86980d0010 2024-04-16T16:14:17.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.870+0000 7f8697e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f869815b430 0x7f86980cd580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42982/0 (socket says 172.21.15.45:42982) 2024-04-16T16:14:17.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.870+0000 7f8697e00640 1 -- 172.21.15.45:0/3725297292 learned_addr learned my addr 172.21.15.45:0/3725297292 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:17.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.871+0000 7f869d000640 1 --2- 172.21.15.45:0/3725297292 >> v2:172.21.15.45:3300/0 conn(0x7f86980d0010 0x7f8698166550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:17.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.871+0000 7f8697400640 1 -- 172.21.15.45:0/3725297292 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4023393908 0 0) 0x7f86980cdc90 con 0x7f869815b430 2024-04-16T16:14:17.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.871+0000 7f8697400640 1 -- 172.21.15.45:0/3725297292 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f867c003520 con 0x7f869815b430 2024-04-16T16:14:17.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.871+0000 7f869d000640 1 -- 172.21.15.45:0/3725297292 >> v1:172.21.15.103:6789/0 conn(0x7f869815b810 legacy=0x7f86980b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:17.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.871+0000 7f869d000640 1 -- 172.21.15.45:0/3725297292 >> v1:172.21.15.45:6789/0 conn(0x7f869815b430 legacy=0x7f86980cd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:17.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.871+0000 7f869d000640 1 -- 172.21.15.45:0/3725297292 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86980ce520 con 0x7f86980d0010 2024-04-16T16:14:17.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.872+0000 7f869d000640 1 --2- 172.21.15.45:0/3725297292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86980d0010 0x7f8698166550 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8690009d90 tx=0x7f869002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df4001264949f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:17.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.872+0000 7f8697400640 1 -- 172.21.15.45:0/3725297292 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8690039070 con 0x7f86980d0010 2024-04-16T16:14:17.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.872+0000 7f8697400640 1 -- 172.21.15.45:0/3725297292 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f869002cc60 con 0x7f86980d0010 2024-04-16T16:14:17.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.872+0000 7f8697400640 1 -- 172.21.15.45:0/3725297292 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8690034640 con 0x7f86980d0010 2024-04-16T16:14:17.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.873+0000 7f869da00640 1 -- 172.21.15.45:0/3725297292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86980d0010 msgr2=0x7f8698166550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:17.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.873+0000 7f869da00640 1 --2- 172.21.15.45:0/3725297292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86980d0010 0x7f8698166550 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8690009d90 tx=0x7f869002beb0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.873+0000 7f869da00640 1 -- 172.21.15.45:0/3725297292 shutdown_connections 2024-04-16T16:14:17.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.873+0000 7f869da00640 1 --2- 172.21.15.45:0/3725297292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86980d0010 0x7f8698166550 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.873+0000 7f869da00640 1 -- 172.21.15.45:0/3725297292 >> 172.21.15.45:0/3725297292 conn(0x7f869806b930 msgr2=0x7f869806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:17.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.873+0000 7f869da00640 1 -- 172.21.15.45:0/3725297292 shutdown_connections 2024-04-16T16:14:17.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.873+0000 7f869da00640 1 -- 172.21.15.45:0/3725297292 wait complete. 2024-04-16T16:14:17.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.875+0000 7f869da00640 1 Processor -- start 2024-04-16T16:14:17.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.875+0000 7f869da00640 1 -- start start 2024-04-16T16:14:17.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.876+0000 7f869da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86980d0010 0x7f86981f4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:17.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.876+0000 7f869da00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f869815b430 0x7f86980cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:17.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.876+0000 7f869da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f869815b810 0x7f8698156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:17.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.876+0000 7f869da00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f869815f310 con 0x7f869815b810 2024-04-16T16:14:17.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.876+0000 7f869da00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f86980ceae0 con 0x7f86980d0010 2024-04-16T16:14:17.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.876+0000 7f869da00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f869815f490 con 0x7f869815b430 2024-04-16T16:14:17.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.876+0000 7f869d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f869815b810 0x7f8698156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:17.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.877+0000 7f869d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f869815b810 0x7f8698156aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52104/0 (socket says 172.21.15.45:52104) 2024-04-16T16:14:17.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.877+0000 7f869d000640 1 -- 172.21.15.45:0/2735033651 learned_addr learned my addr 172.21.15.45:0/2735033651 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:17.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.877+0000 7f8697e00640 1 --2- 172.21.15.45:0/2735033651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86980d0010 0x7f86981f4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:17.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.877+0000 7f868f400640 1 --2- 172.21.15.45:0/2735033651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f869815b430 0x7f86980cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:17.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.877+0000 7f869d000640 1 -- 172.21.15.45:0/2735033651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f869815b430 msgr2=0x7f86980cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:17.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.877+0000 7f869d000640 1 --2- 172.21.15.45:0/2735033651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f869815b430 0x7f86980cc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.877+0000 7f869d000640 1 -- 172.21.15.45:0/2735033651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86980d0010 msgr2=0x7f86981f4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:17.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.877+0000 7f869d000640 1 --2- 172.21.15.45:0/2735033651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86980d0010 0x7f86981f4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:17.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.877+0000 7f869d000640 1 -- 172.21.15.45:0/2735033651 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f869814ba70 con 0x7f869815b810 2024-04-16T16:14:17.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.877+0000 7f8697e00640 1 --2- 172.21.15.45:0/2735033651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86980d0010 0x7f86981f4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:14:17.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.877+0000 7f869d000640 1 --2- 172.21.15.45:0/2735033651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f869815b810 0x7f8698156aa0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8690037040 tx=0x7f8690004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:17.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.878+0000 7f8694c00640 1 -- 172.21.15.45:0/2735033651 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8690039050 con 0x7f869815b810 2024-04-16T16:14:17.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.878+0000 7f8694c00640 1 -- 172.21.15.45:0/2735033651 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8690045070 con 0x7f869815b810 2024-04-16T16:14:17.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.878+0000 7f869da00640 1 -- 172.21.15.45:0/2735033651 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f869814bd50 con 0x7f869815b810 2024-04-16T16:14:17.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.878+0000 7f8694c00640 1 -- 172.21.15.45:0/2735033651 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8690009450 con 0x7f869815b810 2024-04-16T16:14:17.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.878+0000 7f869da00640 1 -- 172.21.15.45:0/2735033651 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f869814c310 con 0x7f869815b810 2024-04-16T16:14:17.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.879+0000 7f869da00640 1 -- 172.21.15.45:0/2735033651 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8698075780 con 0x7f869815b810 2024-04-16T16:14:17.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.880+0000 7f8694c00640 1 -- 172.21.15.45:0/2735033651 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f869003e830 con 0x7f869815b810 2024-04-16T16:14:17.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.881+0000 7f8694c00640 1 --2- 172.21.15.45:0/2735033651 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f867406c920 0x7f867406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:17.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.881+0000 7f8694c00640 1 -- 172.21.15.45:0/2735033651 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f86900ae0f0 con 0x7f869815b810 2024-04-16T16:14:17.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.882+0000 7f8697e00640 1 --2- 172.21.15.45:0/2735033651 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f867406c920 0x7f867406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:17.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.883+0000 7f8697e00640 1 --2- 172.21.15.45:0/2735033651 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f867406c920 0x7f867406ede0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8688007fd0 tx=0x7f8688009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:17.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:17.885+0000 7f8694c00640 1 -- 172.21.15.45:0/2735033651 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8690032e20 con 0x7f869815b810 2024-04-16T16:14:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.090+0000 7f869da00640 1 -- 172.21.15.45:0/2735033651 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8698154f10 con 0x7f869815b810 2024-04-16T16:14:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.091+0000 7f8694c00640 1 -- 172.21.15.45:0/2735033651 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7f869007c4a0 con 0x7f869815b810 2024-04-16T16:14:18.090 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:14:18.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.093+0000 7f869da00640 1 -- 172.21.15.45:0/2735033651 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f867406c920 msgr2=0x7f867406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:18.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.093+0000 7f869da00640 1 --2- 172.21.15.45:0/2735033651 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f867406c920 0x7f867406ede0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8688007fd0 tx=0x7f8688009040 comp rx=0 tx=0).stop 2024-04-16T16:14:18.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.093+0000 7f869da00640 1 -- 172.21.15.45:0/2735033651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f869815b810 msgr2=0x7f8698156aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:18.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.093+0000 7f869da00640 1 --2- 172.21.15.45:0/2735033651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f869815b810 0x7f8698156aa0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8690037040 tx=0x7f8690004a40 comp rx=0 tx=0).stop 2024-04-16T16:14:18.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.093+0000 7f869da00640 1 -- 172.21.15.45:0/2735033651 shutdown_connections 2024-04-16T16:14:18.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.093+0000 7f869da00640 1 --2- 172.21.15.45:0/2735033651 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f867406c920 0x7f867406ede0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.093+0000 7f869da00640 1 --2- 172.21.15.45:0/2735033651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f869815b810 0x7f8698156aa0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.093+0000 7f869da00640 1 --2- 172.21.15.45:0/2735033651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f869815b430 0x7f86980cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.093+0000 7f869da00640 1 --2- 172.21.15.45:0/2735033651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86980d0010 0x7f86981f4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.093+0000 7f869da00640 1 -- 172.21.15.45:0/2735033651 >> 172.21.15.45:0/2735033651 conn(0x7f869806b930 msgr2=0x7f86980b55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:18.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.093+0000 7f869da00640 1 -- 172.21.15.45:0/2735033651 shutdown_connections 2024-04-16T16:14:18.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.094+0000 7f869da00640 1 -- 172.21.15.45:0/2735033651 wait complete. 2024-04-16T16:14:18.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:14:18.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.329+0000 7fae05a00640 1 Processor -- start 2024-04-16T16:14:18.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.330+0000 7fae05a00640 1 -- start start 2024-04-16T16:14:18.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.332+0000 7fae05a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fae001491c0 0x7fae00166840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:18.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.332+0000 7fae05a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae00163cd0 0x7fae000cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:18.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.332+0000 7fae05a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fae000b58f0 0x7fae000b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:18.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.332+0000 7fae05a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fae0006e090 con 0x7fae000b58f0 2024-04-16T16:14:18.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.332+0000 7fae05a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fae001495a0 con 0x7fae001491c0 2024-04-16T16:14:18.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.332+0000 7fae05a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fae00166e40 con 0x7fae00163cd0 2024-04-16T16:14:18.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.332+0000 7fae05000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fae000b58f0 0x7fae000b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:18.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.332+0000 7fae05000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fae000b58f0 0x7fae000b5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52118/0 (socket says 172.21.15.45:52118) 2024-04-16T16:14:18.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.332+0000 7fae05000640 1 -- 172.21.15.45:0/845328323 learned_addr learned my addr 172.21.15.45:0/845328323 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:18.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.333+0000 7fadffe00640 1 --2- 172.21.15.45:0/845328323 >> v2:172.21.15.103:3300/0 conn(0x7fae001491c0 0x7fae00166840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:18.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.333+0000 7fae05000640 1 -- 172.21.15.45:0/845328323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae00163cd0 msgr2=0x7fae000cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:18.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.333+0000 7fae05000640 1 --2- 172.21.15.45:0/845328323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae00163cd0 0x7fae000cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.333+0000 7fae05000640 1 -- 172.21.15.45:0/845328323 >> v2:172.21.15.103:3300/0 conn(0x7fae001491c0 msgr2=0x7fae00166840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:18.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.333+0000 7fae05000640 1 --2- 172.21.15.45:0/845328323 >> v2:172.21.15.103:3300/0 conn(0x7fae001491c0 0x7fae00166840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.333+0000 7fae05000640 1 -- 172.21.15.45:0/845328323 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae00167120 con 0x7fae000b58f0 2024-04-16T16:14:18.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.333+0000 7fadffe00640 1 --2- 172.21.15.45:0/845328323 >> v2:172.21.15.103:3300/0 conn(0x7fae001491c0 0x7fae00166840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:18.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.334+0000 7fae05000640 1 --2- 172.21.15.45:0/845328323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae000b58f0 0x7fae000b5d80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fae00070230 tx=0x7fadf802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59cae2d146b827c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:18.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.334+0000 7fadff400640 1 -- 172.21.15.45:0/845328323 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf802c9f0 con 0x7fae000b58f0 2024-04-16T16:14:18.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.335+0000 7fadff400640 1 -- 172.21.15.45:0/845328323 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadf802cb90 con 0x7fae000b58f0 2024-04-16T16:14:18.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.335+0000 7fadff400640 1 -- 172.21.15.45:0/845328323 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf80343f0 con 0x7fae000b58f0 2024-04-16T16:14:18.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.335+0000 7fae05a00640 1 -- 172.21.15.45:0/845328323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae000b58f0 msgr2=0x7fae000b5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:18.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.335+0000 7fae05a00640 1 --2- 172.21.15.45:0/845328323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae000b58f0 0x7fae000b5d80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fae00070230 tx=0x7fadf802be50 comp rx=0 tx=0).stop 2024-04-16T16:14:18.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.335+0000 7fae05a00640 1 -- 172.21.15.45:0/845328323 shutdown_connections 2024-04-16T16:14:18.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.335+0000 7fae05a00640 1 --2- 172.21.15.45:0/845328323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae000b58f0 0x7fae000b5d80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.335+0000 7fae05a00640 1 --2- 172.21.15.45:0/845328323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae00163cd0 0x7fae000cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.335+0000 7fae05a00640 1 --2- 172.21.15.45:0/845328323 >> v2:172.21.15.103:3300/0 conn(0x7fae001491c0 0x7fae00166840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.335+0000 7fae05a00640 1 -- 172.21.15.45:0/845328323 >> 172.21.15.45:0/845328323 conn(0x7fae0006b930 msgr2=0x7fae0006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:18.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.335+0000 7fae05a00640 1 -- 172.21.15.45:0/845328323 shutdown_connections 2024-04-16T16:14:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.335+0000 7fae05a00640 1 -- 172.21.15.45:0/845328323 wait complete. 2024-04-16T16:14:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.337+0000 7fae05a00640 1 Processor -- start 2024-04-16T16:14:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.338+0000 7fae05a00640 1 -- start start 2024-04-16T16:14:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.339+0000 7fae05a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae000b58f0 0x7fae00160260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.339+0000 7fae05a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae001491c0 0x7fae0014b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.339+0000 7fae05a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae00163cd0 0x7fae000cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.339+0000 7fae05a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fae001676e0 con 0x7fae001491c0 2024-04-16T16:14:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.339+0000 7fae05a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fae00164110 con 0x7fae00163cd0 2024-04-16T16:14:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.339+0000 7fae05a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fae000cd640 con 0x7fae000b58f0 2024-04-16T16:14:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.339+0000 7fadf7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae001491c0 0x7fae0014b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.339+0000 7fadf7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae001491c0 0x7fae0014b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52122/0 (socket says 172.21.15.45:52122) 2024-04-16T16:14:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.339+0000 7fadf7e00640 1 -- 172.21.15.45:0/606723428 learned_addr learned my addr 172.21.15.45:0/606723428 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.339+0000 7fadf7e00640 1 -- 172.21.15.45:0/606723428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae000b58f0 msgr2=0x7fae00160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.339+0000 7fadf7e00640 1 --2- 172.21.15.45:0/606723428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae000b58f0 0x7fae00160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.339+0000 7fadf7e00640 1 -- 172.21.15.45:0/606723428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae00163cd0 msgr2=0x7fae000cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.339+0000 7fae05000640 1 --2- 172.21.15.45:0/606723428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae00163cd0 0x7fae000cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.340+0000 7fadf7e00640 1 --2- 172.21.15.45:0/606723428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae00163cd0 0x7fae000cbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.340+0000 7fadf7e00640 1 -- 172.21.15.45:0/606723428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae00137d80 con 0x7fae001491c0 2024-04-16T16:14:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.340+0000 7fadf7e00640 1 --2- 172.21.15.45:0/606723428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae001491c0 0x7fae0014b180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fade800b4a0 tx=0x7fade800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.340+0000 7fadfcc00640 1 -- 172.21.15.45:0/606723428 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8014070 con 0x7fae001491c0 2024-04-16T16:14:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.340+0000 7fadfcc00640 1 -- 172.21.15.45:0/606723428 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fade8010610 con 0x7fae001491c0 2024-04-16T16:14:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.340+0000 7fadfcc00640 1 -- 172.21.15.45:0/606723428 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8010a00 con 0x7fae001491c0 2024-04-16T16:14:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.340+0000 7fae05a00640 1 -- 172.21.15.45:0/606723428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae0014bc20 con 0x7fae001491c0 2024-04-16T16:14:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.340+0000 7fae05a00640 1 -- 172.21.15.45:0/606723428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae0014c1e0 con 0x7fae001491c0 2024-04-16T16:14:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.341+0000 7fae05a00640 1 -- 172.21.15.45:0/606723428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae00075780 con 0x7fae001491c0 2024-04-16T16:14:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.342+0000 7fadfcc00640 1 -- 172.21.15.45:0/606723428 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fade8017070 con 0x7fae001491c0 2024-04-16T16:14:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.343+0000 7fadfcc00640 1 --2- 172.21.15.45:0/606723428 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7faddc06c8f0 0x7faddc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:18.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.344+0000 7fadfcc00640 1 -- 172.21.15.45:0/606723428 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fade808b750 con 0x7fae001491c0 2024-04-16T16:14:18.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.344+0000 7fadffe00640 1 --2- 172.21.15.45:0/606723428 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7faddc06c8f0 0x7faddc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:18.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.345+0000 7fadffe00640 1 --2- 172.21.15.45:0/606723428 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7faddc06c8f0 0x7faddc06edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fadf0006fd0 tx=0x7fadf0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.348+0000 7fadfcc00640 1 -- 172.21.15.45:0/606723428 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fade80599d0 con 0x7fae001491c0 2024-04-16T16:14:18.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.550+0000 7fae05a00640 1 -- 172.21.15.45:0/606723428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fae000b20d0 con 0x7fae001491c0 2024-04-16T16:14:18.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.551+0000 7fadfcc00640 1 -- 172.21.15.45:0/606723428 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7fade80592d0 con 0x7fae001491c0 2024-04-16T16:14:18.550 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:14:18.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.554+0000 7fae05a00640 1 -- 172.21.15.45:0/606723428 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7faddc06c8f0 msgr2=0x7faddc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:18.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.554+0000 7fae05a00640 1 --2- 172.21.15.45:0/606723428 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7faddc06c8f0 0x7faddc06edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fadf0006fd0 tx=0x7fadf0008040 comp rx=0 tx=0).stop 2024-04-16T16:14:18.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.554+0000 7fae05a00640 1 -- 172.21.15.45:0/606723428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae001491c0 msgr2=0x7fae0014b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:18.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.554+0000 7fae05a00640 1 --2- 172.21.15.45:0/606723428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae001491c0 0x7fae0014b180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fade800b4a0 tx=0x7fade800b960 comp rx=0 tx=0).stop 2024-04-16T16:14:18.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.554+0000 7fae05a00640 1 -- 172.21.15.45:0/606723428 shutdown_connections 2024-04-16T16:14:18.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.554+0000 7fae05a00640 1 --2- 172.21.15.45:0/606723428 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7faddc06c8f0 0x7faddc06edb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.554+0000 7fae05a00640 1 --2- 172.21.15.45:0/606723428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae00163cd0 0x7fae000cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.554+0000 7fae05a00640 1 --2- 172.21.15.45:0/606723428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae001491c0 0x7fae0014b180 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.554+0000 7fae05a00640 1 --2- 172.21.15.45:0/606723428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae000b58f0 0x7fae00160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.554+0000 7fae05a00640 1 -- 172.21.15.45:0/606723428 >> 172.21.15.45:0/606723428 conn(0x7fae0006b930 msgr2=0x7fae00162580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:18.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.554+0000 7fae05a00640 1 -- 172.21.15.45:0/606723428 shutdown_connections 2024-04-16T16:14:18.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.554+0000 7fae05a00640 1 -- 172.21.15.45:0/606723428 wait complete. 2024-04-16T16:14:18.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:14:18.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.798+0000 7f054be00640 1 Processor -- start 2024-04-16T16:14:18.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.798+0000 7f054be00640 1 -- start start 2024-04-16T16:14:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.799+0000 7f054be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f054c162090 0x7f054c0cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.800+0000 7f054be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f054c0d5520 0x7f054c0cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.800+0000 7f054be00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f054c0d2600 con 0x7f054c0ba140 2024-04-16T16:14:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.800+0000 7f054be00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f054c0ba8b0 con 0x7f054c162090 2024-04-16T16:14:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.800+0000 7f054be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f054c0ba5d0 con 0x7f054c0d5520 2024-04-16T16:14:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.800+0000 7f054aa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f054c162090 0x7f054c0cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.800+0000 7f0543e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f054c0ba140 0x7f054c1628a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42998/0 (socket says 172.21.15.45:42998) 2024-04-16T16:14:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.800+0000 7f0543e00640 1 -- 172.21.15.45:0/2709240362 learned_addr learned my addr 172.21.15.45:0/2709240362 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:18.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.800+0000 7f054b400640 1 --2- 172.21.15.45:0/2709240362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f054c0d5520 0x7f054c0cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:18.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.800+0000 7f054aa00640 1 -- 172.21.15.45:0/2709240362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f054c0d5520 msgr2=0x7f054c0cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:18.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.800+0000 7f054aa00640 1 --2- 172.21.15.45:0/2709240362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f054c0d5520 0x7f054c0cfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.800+0000 7f054aa00640 1 -- 172.21.15.45:0/2709240362 >> v1:172.21.15.45:6789/0 conn(0x7f054c0ba140 legacy=0x7f054c1628a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:18.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.800+0000 7f054aa00640 1 -- 172.21.15.45:0/2709240362 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054c0d28a0 con 0x7f054c162090 2024-04-16T16:14:18.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.800+0000 7f054a000640 1 -- 172.21.15.45:0/2709240362 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 243704373 0 0) 0x7f054c0d2600 con 0x7f054c0ba140 2024-04-16T16:14:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.802+0000 7f054aa00640 1 --2- 172.21.15.45:0/2709240362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f054c162090 0x7f054c0cf510 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f053c0097a0 tx=0x7f053c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e85fb70027fd99 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.802+0000 7f054a000640 1 -- 172.21.15.45:0/2709240362 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c035020 con 0x7f054c162090 2024-04-16T16:14:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.802+0000 7f054a000640 1 -- 172.21.15.45:0/2709240362 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f053c031830 con 0x7f054c162090 2024-04-16T16:14:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.802+0000 7f054a000640 1 -- 172.21.15.45:0/2709240362 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c031c00 con 0x7f054c162090 2024-04-16T16:14:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.803+0000 7f054be00640 1 -- 172.21.15.45:0/2709240362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f054c162090 msgr2=0x7f054c0cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.803+0000 7f054be00640 1 --2- 172.21.15.45:0/2709240362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f054c162090 0x7f054c0cf510 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f053c0097a0 tx=0x7f053c02b8d0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.803+0000 7f054be00640 1 -- 172.21.15.45:0/2709240362 shutdown_connections 2024-04-16T16:14:18.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.803+0000 7f054be00640 1 --2- 172.21.15.45:0/2709240362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f054c0d5520 0x7f054c0cfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.803+0000 7f054be00640 1 --2- 172.21.15.45:0/2709240362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f054c162090 0x7f054c0cf510 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.803+0000 7f054be00640 1 -- 172.21.15.45:0/2709240362 >> 172.21.15.45:0/2709240362 conn(0x7f054c0d1940 msgr2=0x7f054c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:18.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.803+0000 7f054be00640 1 -- 172.21.15.45:0/2709240362 shutdown_connections 2024-04-16T16:14:18.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.803+0000 7f054be00640 1 -- 172.21.15.45:0/2709240362 wait complete. 2024-04-16T16:14:18.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.805+0000 7f054be00640 1 Processor -- start 2024-04-16T16:14:18.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.805+0000 7f054be00640 1 -- start start 2024-04-16T16:14:18.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.806+0000 7f054be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f054c0d5520 0x7f054c153db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:18.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.806+0000 7f054be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f054c162090 0x7f054c0caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:18.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.806+0000 7f054be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f054c0c9070 0x7f054c0cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:18.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.806+0000 7f054be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f054c1625f0 con 0x7f054c162090 2024-04-16T16:14:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.806+0000 7f054be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f054c162470 con 0x7f054c0d5520 2024-04-16T16:14:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.806+0000 7f054be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f054c0d40b0 con 0x7f054c0c9070 2024-04-16T16:14:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.806+0000 7f0543e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f054c162090 0x7f054c0caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.806+0000 7f0543e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f054c162090 0x7f054c0caee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52152/0 (socket says 172.21.15.45:52152) 2024-04-16T16:14:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.806+0000 7f0543e00640 1 -- 172.21.15.45:0/261107840 learned_addr learned my addr 172.21.15.45:0/261107840 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.806+0000 7f054b400640 1 --2- 172.21.15.45:0/261107840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f054c0c9070 0x7f054c0cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.806+0000 7f0543e00640 1 -- 172.21.15.45:0/261107840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f054c0c9070 msgr2=0x7f054c0cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.806+0000 7f054aa00640 1 --2- 172.21.15.45:0/261107840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f054c0d5520 0x7f054c153db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.806+0000 7f0543e00640 1 --2- 172.21.15.45:0/261107840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f054c0c9070 0x7f054c0cdd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.807+0000 7f0543e00640 1 -- 172.21.15.45:0/261107840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f054c0d5520 msgr2=0x7f054c153db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.807+0000 7f0543e00640 1 --2- 172.21.15.45:0/261107840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f054c0d5520 0x7f054c153db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:18.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.807+0000 7f0543e00640 1 -- 172.21.15.45:0/261107840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054c0cc310 con 0x7f054c162090 2024-04-16T16:14:18.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.807+0000 7f054aa00640 1 --2- 172.21.15.45:0/261107840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f054c0d5520 0x7f054c153db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:18.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.807+0000 7f0543e00640 1 --2- 172.21.15.45:0/261107840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f054c162090 0x7f054c0caee0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f053400c4d0 tx=0x7f053400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:18.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.807+0000 7f0542a00640 1 -- 172.21.15.45:0/261107840 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0534018070 con 0x7f054c162090 2024-04-16T16:14:18.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.807+0000 7f0542a00640 1 -- 172.21.15.45:0/261107840 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0534007500 con 0x7f054c162090 2024-04-16T16:14:18.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.807+0000 7f054be00640 1 -- 172.21.15.45:0/261107840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f054c0cc5f0 con 0x7f054c162090 2024-04-16T16:14:18.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.807+0000 7f0542a00640 1 -- 172.21.15.45:0/261107840 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0534004a20 con 0x7f054c162090 2024-04-16T16:14:18.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.807+0000 7f054be00640 1 -- 172.21.15.45:0/261107840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f054c0c9500 con 0x7f054c162090 2024-04-16T16:14:18.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.809+0000 7f054be00640 1 -- 172.21.15.45:0/261107840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f054c0d0ac0 con 0x7f054c162090 2024-04-16T16:14:18.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.810+0000 7f0542a00640 1 -- 172.21.15.45:0/261107840 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f053401b070 con 0x7f054c162090 2024-04-16T16:14:18.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.810+0000 7f0542a00640 1 --2- 172.21.15.45:0/261107840 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f052806c8f0 0x7f052806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:18.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.811+0000 7f0542a00640 1 -- 172.21.15.45:0/261107840 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f053408af20 con 0x7f054c162090 2024-04-16T16:14:18.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.811+0000 7f054aa00640 1 --2- 172.21.15.45:0/261107840 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f052806c8f0 0x7f052806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:18.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.813+0000 7f054aa00640 1 --2- 172.21.15.45:0/261107840 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f052806c8f0 0x7f052806edb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f053c036000 tx=0x7f053c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:18.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:18.815+0000 7f0542a00640 1 -- 172.21.15.45:0/261107840 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f054c0d0ac0 con 0x7f054c162090 2024-04-16T16:14:19.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.034+0000 7f054be00640 1 -- 172.21.15.45:0/261107840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f054c075780 con 0x7f054c162090 2024-04-16T16:14:19.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.035+0000 7f0542a00640 1 -- 172.21.15.45:0/261107840 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7f05340592c0 con 0x7f054c162090 2024-04-16T16:14:19.034 INFO:teuthology.orchestra.run.smithi045.stderr:listed 0 entries 2024-04-16T16:14:19.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.038+0000 7f054be00640 1 -- 172.21.15.45:0/261107840 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f052806c8f0 msgr2=0x7f052806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:19.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.038+0000 7f054be00640 1 --2- 172.21.15.45:0/261107840 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f052806c8f0 0x7f052806edb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f053c036000 tx=0x7f053c008030 comp rx=0 tx=0).stop 2024-04-16T16:14:19.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.038+0000 7f054be00640 1 -- 172.21.15.45:0/261107840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f054c162090 msgr2=0x7f054c0caee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:19.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.038+0000 7f054be00640 1 --2- 172.21.15.45:0/261107840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f054c162090 0x7f054c0caee0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f053400c4d0 tx=0x7f053400c990 comp rx=0 tx=0).stop 2024-04-16T16:14:19.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.038+0000 7f054be00640 1 -- 172.21.15.45:0/261107840 shutdown_connections 2024-04-16T16:14:19.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.038+0000 7f054be00640 1 --2- 172.21.15.45:0/261107840 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f052806c8f0 0x7f052806edb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:19.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.038+0000 7f054be00640 1 --2- 172.21.15.45:0/261107840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f054c0c9070 0x7f054c0cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:19.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.038+0000 7f054be00640 1 --2- 172.21.15.45:0/261107840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f054c162090 0x7f054c0caee0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:19.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.038+0000 7f054be00640 1 --2- 172.21.15.45:0/261107840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f054c0d5520 0x7f054c153db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:19.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.038+0000 7f054be00640 1 -- 172.21.15.45:0/261107840 >> 172.21.15.45:0/261107840 conn(0x7f054c0d1940 msgr2=0x7f054c0b55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:19.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.038+0000 7f054be00640 1 -- 172.21.15.45:0/261107840 shutdown_connections 2024-04-16T16:14:19.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.038+0000 7f054be00640 1 -- 172.21.15.45:0/261107840 wait complete. 2024-04-16T16:14:19.057 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-04-16T16:14:19.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:19.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-04-16T16:14:19.157 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:14:19.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:19.158 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-16T16:14:19.211 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:14:19.212 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:19.270 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:14:19.270 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:19.322 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:14:19.322 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:19.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.608+0000 7fc25b200640 1 Processor -- start 2024-04-16T16:14:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.609+0000 7fc25b200640 1 -- start start 2024-04-16T16:14:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.610+0000 7fc25b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc25c146220 0x7fc25c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.610+0000 7fc25b200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc25c06f7a0 0x7fc25c06df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.610+0000 7fc25b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc25c15d360 con 0x7fc25c06e480 2024-04-16T16:14:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.610+0000 7fc25b200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc25c160410 con 0x7fc25c06f7a0 2024-04-16T16:14:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.610+0000 7fc25b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc25c15d640 con 0x7fc25c146220 2024-04-16T16:14:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.611+0000 7fc259400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc25c06f7a0 0x7fc25c06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.611+0000 7fc259400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc25c06f7a0 0x7fc25c06df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52162/0 (socket says 172.21.15.45:52162) 2024-04-16T16:14:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.611+0000 7fc259400640 1 -- 172.21.15.45:0/2696721794 learned_addr learned my addr 172.21.15.45:0/2696721794 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:19.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.611+0000 7fc259e00640 1 --2- 172.21.15.45:0/2696721794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc25c146220 0x7fc25c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:19.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.611+0000 7fc259400640 1 -- 172.21.15.45:0/2696721794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc25c146220 msgr2=0x7fc25c15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:19.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.611+0000 7fc259400640 1 --2- 172.21.15.45:0/2696721794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc25c146220 0x7fc25c15ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:19.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.611+0000 7fc259400640 1 -- 172.21.15.45:0/2696721794 >> v1:172.21.15.103:6789/0 conn(0x7fc25c06e480 legacy=0x7fc25c1608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:19.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.611+0000 7fc259400640 1 -- 172.21.15.45:0/2696721794 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc25c15d920 con 0x7fc25c06f7a0 2024-04-16T16:14:19.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.613+0000 7fc259400640 1 --2- 172.21.15.45:0/2696721794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc25c06f7a0 0x7fc25c06df40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc244009870 tx=0x7fc24402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8931a6a014a266 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:19.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.613+0000 7fc258a00640 1 -- 172.21.15.45:0/2696721794 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc244035020 con 0x7fc25c06f7a0 2024-04-16T16:14:19.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.613+0000 7fc258a00640 1 -- 172.21.15.45:0/2696721794 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc244031640 con 0x7fc25c06f7a0 2024-04-16T16:14:19.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.613+0000 7fc258a00640 1 -- 172.21.15.45:0/2696721794 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24402d040 con 0x7fc25c06f7a0 2024-04-16T16:14:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.614+0000 7fc25b200640 1 -- 172.21.15.45:0/2696721794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc25c06f7a0 msgr2=0x7fc25c06df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.614+0000 7fc25b200640 1 --2- 172.21.15.45:0/2696721794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc25c06f7a0 0x7fc25c06df40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc244009870 tx=0x7fc24402b8f0 comp rx=0 tx=0).stop 2024-04-16T16:14:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.614+0000 7fc25b200640 1 -- 172.21.15.45:0/2696721794 shutdown_connections 2024-04-16T16:14:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.614+0000 7fc25b200640 1 --2- 172.21.15.45:0/2696721794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc25c06f7a0 0x7fc25c06df40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.614+0000 7fc25b200640 1 --2- 172.21.15.45:0/2696721794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc25c146220 0x7fc25c15ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.614+0000 7fc25b200640 1 -- 172.21.15.45:0/2696721794 >> 172.21.15.45:0/2696721794 conn(0x7fc25c06b930 msgr2=0x7fc25c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.614+0000 7fc25b200640 1 -- 172.21.15.45:0/2696721794 shutdown_connections 2024-04-16T16:14:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.614+0000 7fc25b200640 1 -- 172.21.15.45:0/2696721794 wait complete. 2024-04-16T16:14:19.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.616+0000 7fc25b200640 1 Processor -- start 2024-04-16T16:14:19.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.616+0000 7fc25b200640 1 -- start start 2024-04-16T16:14:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.617+0000 7fc25b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc25c06e480 0x7fc25c15c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.617+0000 7fc25b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc25c06f7a0 0x7fc25c137d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.617+0000 7fc25b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc25c146220 0x7fc25c138270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.617+0000 7fc25b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc25c06fda0 con 0x7fc25c06e480 2024-04-16T16:14:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.617+0000 7fc25b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc25c06fc20 con 0x7fc25c06f7a0 2024-04-16T16:14:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.617+0000 7fc25b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc25c160fb0 con 0x7fc25c146220 2024-04-16T16:14:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.617+0000 7fc259e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc25c06e480 0x7fc25c15c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.617+0000 7fc259e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc25c06e480 0x7fc25c15c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52174/0 (socket says 172.21.15.45:52174) 2024-04-16T16:14:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.617+0000 7fc259e00640 1 -- 172.21.15.45:0/3252422789 learned_addr learned my addr 172.21.15.45:0/3252422789 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.617+0000 7fc259e00640 1 -- 172.21.15.45:0/3252422789 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc25c146220 msgr2=0x7fc25c138270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.618+0000 7fc259e00640 1 --2- 172.21.15.45:0/3252422789 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc25c146220 0x7fc25c138270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.618+0000 7fc259e00640 1 -- 172.21.15.45:0/3252422789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc25c06f7a0 msgr2=0x7fc25c137d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.618+0000 7fc259e00640 1 --2- 172.21.15.45:0/3252422789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc25c06f7a0 0x7fc25c137d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.618+0000 7fc259e00640 1 -- 172.21.15.45:0/3252422789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc25c1e7720 con 0x7fc25c06e480 2024-04-16T16:14:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.618+0000 7fc259e00640 1 --2- 172.21.15.45:0/3252422789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc25c06e480 0x7fc25c15c080 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc24c00c7f0 tx=0x7fc24c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.618+0000 7fc252000640 1 -- 172.21.15.45:0/3252422789 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c015070 con 0x7fc25c06e480 2024-04-16T16:14:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.618+0000 7fc252000640 1 -- 172.21.15.45:0/3252422789 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc24c004220 con 0x7fc25c06e480 2024-04-16T16:14:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.618+0000 7fc252000640 1 -- 172.21.15.45:0/3252422789 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c007620 con 0x7fc25c06e480 2024-04-16T16:14:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.618+0000 7fc25b200640 1 -- 172.21.15.45:0/3252422789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc25c1e7a00 con 0x7fc25c06e480 2024-04-16T16:14:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.618+0000 7fc25b200640 1 -- 172.21.15.45:0/3252422789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc25c13c080 con 0x7fc25c06e480 2024-04-16T16:14:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.620+0000 7fc25b200640 1 -- 172.21.15.45:0/3252422789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc25c068400 con 0x7fc25c06e480 2024-04-16T16:14:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.621+0000 7fc252000640 1 -- 172.21.15.45:0/3252422789 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc24c0079d0 con 0x7fc25c06e480 2024-04-16T16:14:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.622+0000 7fc252000640 1 --2- 172.21.15.45:0/3252422789 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc23806c920 0x7fc23806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.622+0000 7fc252000640 1 -- 172.21.15.45:0/3252422789 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc24c08bbe0 con 0x7fc25c06e480 2024-04-16T16:14:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.623+0000 7fc259400640 1 --2- 172.21.15.45:0/3252422789 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc23806c920 0x7fc23806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:19.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.624+0000 7fc259400640 1 --2- 172.21.15.45:0/3252422789 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc23806c920 0x7fc23806ede0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc244036000 tx=0x7fc244030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:19.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.627+0000 7fc252000640 1 -- 172.21.15.45:0/3252422789 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc25c068400 con 0x7fc25c06e480 2024-04-16T16:14:19.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.833+0000 7fc25b200640 1 -- 172.21.15.45:0/3252422789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc25c075780 con 0x7fc25c06e480 2024-04-16T16:14:19.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.835+0000 7fc252000640 1 -- 172.21.15.45:0/3252422789 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16799 (secure 0 0 0) 0x7fc24c059f80 con 0x7fc25c06e480 2024-04-16T16:14:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":23,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:14:12.851304+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:19.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.837+0000 7fc25b200640 1 -- 172.21.15.45:0/3252422789 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc23806c920 msgr2=0x7fc23806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:19.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.837+0000 7fc25b200640 1 --2- 172.21.15.45:0/3252422789 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc23806c920 0x7fc23806ede0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc244036000 tx=0x7fc244030000 comp rx=0 tx=0).stop 2024-04-16T16:14:19.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.837+0000 7fc25b200640 1 -- 172.21.15.45:0/3252422789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc25c06e480 msgr2=0x7fc25c15c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:19.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.837+0000 7fc25b200640 1 --2- 172.21.15.45:0/3252422789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc25c06e480 0x7fc25c15c080 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc24c00c7f0 tx=0x7fc24c00ccb0 comp rx=0 tx=0).stop 2024-04-16T16:14:19.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.837+0000 7fc25b200640 1 -- 172.21.15.45:0/3252422789 shutdown_connections 2024-04-16T16:14:19.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.837+0000 7fc25b200640 1 --2- 172.21.15.45:0/3252422789 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc23806c920 0x7fc23806ede0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:19.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.837+0000 7fc25b200640 1 --2- 172.21.15.45:0/3252422789 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc25c146220 0x7fc25c138270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:19.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.838+0000 7fc25b200640 1 --2- 172.21.15.45:0/3252422789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc25c06f7a0 0x7fc25c137d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:19.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.838+0000 7fc25b200640 1 --2- 172.21.15.45:0/3252422789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc25c06e480 0x7fc25c15c080 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:19.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.838+0000 7fc25b200640 1 -- 172.21.15.45:0/3252422789 >> 172.21.15.45:0/3252422789 conn(0x7fc25c06b930 msgr2=0x7fc25c15ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:19.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.838+0000 7fc25b200640 1 -- 172.21.15.45:0/3252422789 shutdown_connections 2024-04-16T16:14:19.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:19.838+0000 7fc25b200640 1 -- 172.21.15.45:0/3252422789 wait complete. 2024-04-16T16:14:19.859 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:13:23.600104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:13:24.368141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:14:19.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:14:20.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.085+0000 7f78f4a00640 1 Processor -- start 2024-04-16T16:14:20.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.086+0000 7f78f4a00640 1 -- start start 2024-04-16T16:14:20.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.087+0000 7f78f4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f0160e90 0x7f78f00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:20.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.087+0000 7f78f4a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f78f00cf980 0x7f78f006dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:20.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.087+0000 7f78f4a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f00c6210 con 0x7f78f0145fe0 2024-04-16T16:14:20.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.088+0000 7f78f4a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f78f00cfdd0 con 0x7f78f00cf980 2024-04-16T16:14:20.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.088+0000 7f78f4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f78f01603b0 con 0x7f78f0160e90 2024-04-16T16:14:20.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.088+0000 7f78ef400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f78f0145fe0 0x7f78f00c5750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43006/0 (socket says 172.21.15.45:43006) 2024-04-16T16:14:20.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.088+0000 7f78ef400640 1 -- 172.21.15.45:0/1122974297 learned_addr learned my addr 172.21.15.45:0/1122974297 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:20.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.088+0000 7f78efe00640 1 --2- 172.21.15.45:0/1122974297 >> v2:172.21.15.45:3300/0 conn(0x7f78f00cf980 0x7f78f006dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:20.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.088+0000 7f78eea00640 1 --2- 172.21.15.45:0/1122974297 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f0160e90 0x7f78f00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:20.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.088+0000 7f78efe00640 1 -- 172.21.15.45:0/1122974297 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f0160e90 msgr2=0x7f78f00cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:20.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.088+0000 7f78efe00640 1 --2- 172.21.15.45:0/1122974297 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f0160e90 0x7f78f00cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.088+0000 7f78efe00640 1 -- 172.21.15.45:0/1122974297 >> v1:172.21.15.45:6789/0 conn(0x7f78f0145fe0 legacy=0x7f78f00c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:20.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.089+0000 7f78efe00640 1 -- 172.21.15.45:0/1122974297 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f0160690 con 0x7f78f00cf980 2024-04-16T16:14:20.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.089+0000 7f78ee000640 1 -- 172.21.15.45:0/1122974297 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3874366713 0 0) 0x7f78f00c6210 con 0x7f78f0145fe0 2024-04-16T16:14:20.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.090+0000 7f78efe00640 1 --2- 172.21.15.45:0/1122974297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78f00cf980 0x7f78f006dfb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009d90 tx=0x7f78e802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f5bca7c693ea9e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:20.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.090+0000 7f78ee000640 1 -- 172.21.15.45:0/1122974297 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e8039070 con 0x7f78f00cf980 2024-04-16T16:14:20.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.090+0000 7f78ee000640 1 -- 172.21.15.45:0/1122974297 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e802cbc0 con 0x7f78f00cf980 2024-04-16T16:14:20.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.090+0000 7f78ee000640 1 -- 172.21.15.45:0/1122974297 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e8034440 con 0x7f78f00cf980 2024-04-16T16:14:20.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.091+0000 7f78f4a00640 1 -- 172.21.15.45:0/1122974297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78f00cf980 msgr2=0x7f78f006dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:20.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.091+0000 7f78f4a00640 1 --2- 172.21.15.45:0/1122974297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78f00cf980 0x7f78f006dfb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009d90 tx=0x7f78e802be10 comp rx=0 tx=0).stop 2024-04-16T16:14:20.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.091+0000 7f78f4a00640 1 -- 172.21.15.45:0/1122974297 shutdown_connections 2024-04-16T16:14:20.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.091+0000 7f78f4a00640 1 --2- 172.21.15.45:0/1122974297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78f00cf980 0x7f78f006dfb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.091+0000 7f78f4a00640 1 --2- 172.21.15.45:0/1122974297 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f0160e90 0x7f78f00cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.091+0000 7f78f4a00640 1 -- 172.21.15.45:0/1122974297 >> 172.21.15.45:0/1122974297 conn(0x7f78f0067e00 msgr2=0x7f78f006b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:20.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.091+0000 7f78f4a00640 1 -- 172.21.15.45:0/1122974297 shutdown_connections 2024-04-16T16:14:20.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.091+0000 7f78f4a00640 1 -- 172.21.15.45:0/1122974297 wait complete. 2024-04-16T16:14:20.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.093+0000 7f78f4a00640 1 Processor -- start 2024-04-16T16:14:20.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.094+0000 7f78f4a00640 1 -- start start 2024-04-16T16:14:20.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.095+0000 7f78f4a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78f00cf980 0x7f78f01f2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.095+0000 7f78f4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78f0145fe0 0x7f78f0142660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.095+0000 7f78f4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f0160e90 0x7f78f01f52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.095+0000 7f78f4a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f78f00c5e60 con 0x7f78f00cf980 2024-04-16T16:14:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.095+0000 7f78f4a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f78f015dae0 con 0x7f78f0145fe0 2024-04-16T16:14:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.095+0000 7f78f4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f78f01f2c40 con 0x7f78f0160e90 2024-04-16T16:14:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.095+0000 7f78ef400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78f00cf980 0x7f78f01f2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.095+0000 7f78ef400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78f00cf980 0x7f78f01f2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52198/0 (socket says 172.21.15.45:52198) 2024-04-16T16:14:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.095+0000 7f78ef400640 1 -- 172.21.15.45:0/4177656096 learned_addr learned my addr 172.21.15.45:0/4177656096 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.095+0000 7f78efe00640 1 --2- 172.21.15.45:0/4177656096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f0160e90 0x7f78f01f52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.096+0000 7f78eea00640 1 --2- 172.21.15.45:0/4177656096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78f0145fe0 0x7f78f0142660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.096+0000 7f78ef400640 1 -- 172.21.15.45:0/4177656096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f0160e90 msgr2=0x7f78f01f52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.096+0000 7f78ef400640 1 --2- 172.21.15.45:0/4177656096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f0160e90 0x7f78f01f52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.096+0000 7f78ef400640 1 -- 172.21.15.45:0/4177656096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78f0145fe0 msgr2=0x7f78f0142660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:20.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.096+0000 7f78ef400640 1 --2- 172.21.15.45:0/4177656096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78f0145fe0 0x7f78f0142660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.096+0000 7f78ef400640 1 -- 172.21.15.45:0/4177656096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f01f1c70 con 0x7f78f00cf980 2024-04-16T16:14:20.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.096+0000 7f78ef400640 1 --2- 172.21.15.45:0/4177656096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78f00cf980 0x7f78f01f2700 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f78e000c4d0 tx=0x7f78e000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:20.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.096+0000 7f78e7400640 1 -- 172.21.15.45:0/4177656096 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e0018070 con 0x7f78f00cf980 2024-04-16T16:14:20.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.096+0000 7f78e7400640 1 -- 172.21.15.45:0/4177656096 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e0007500 con 0x7f78f00cf980 2024-04-16T16:14:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.096+0000 7f78f4a00640 1 -- 172.21.15.45:0/4177656096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f01f5c50 con 0x7f78f00cf980 2024-04-16T16:14:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.096+0000 7f78e7400640 1 -- 172.21.15.45:0/4177656096 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e0004a20 con 0x7f78f00cf980 2024-04-16T16:14:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.097+0000 7f78f4a00640 1 -- 172.21.15.45:0/4177656096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f01f1f00 con 0x7f78f00cf980 2024-04-16T16:14:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.098+0000 7f78f4a00640 1 -- 172.21.15.45:0/4177656096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f0075780 con 0x7f78f00cf980 2024-04-16T16:14:20.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.099+0000 7f78e7400640 1 -- 172.21.15.45:0/4177656096 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f78e001b070 con 0x7f78f00cf980 2024-04-16T16:14:20.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.100+0000 7f78e7400640 1 --2- 172.21.15.45:0/4177656096 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f78d006c8f0 0x7f78d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:20.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.100+0000 7f78e7400640 1 -- 172.21.15.45:0/4177656096 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f78e008aea0 con 0x7f78f00cf980 2024-04-16T16:14:20.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.101+0000 7f78eea00640 1 --2- 172.21.15.45:0/4177656096 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f78d006c8f0 0x7f78d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:20.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.102+0000 7f78eea00640 1 --2- 172.21.15.45:0/4177656096 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f78d006c8f0 0x7f78d006edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f78f015c5a0 tx=0x7f78dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:20.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.104+0000 7f78e7400640 1 -- 172.21.15.45:0/4177656096 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78e0059240 con 0x7f78f00cf980 2024-04-16T16:14:20.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.306+0000 7f78f4a00640 1 -- 172.21.15.45:0/4177656096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f78f006c110 con 0x7f78f00cf980 2024-04-16T16:14:20.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.307+0000 7f78e7400640 1 -- 172.21.15.45:0/4177656096 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7f78e0058b40 con 0x7f78f00cf980 2024-04-16T16:14:20.306 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:14:20.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.310+0000 7f78f4a00640 1 -- 172.21.15.45:0/4177656096 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f78d006c8f0 msgr2=0x7f78d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:20.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.310+0000 7f78f4a00640 1 --2- 172.21.15.45:0/4177656096 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f78d006c8f0 0x7f78d006edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f78f015c5a0 tx=0x7f78dc008040 comp rx=0 tx=0).stop 2024-04-16T16:14:20.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.310+0000 7f78f4a00640 1 -- 172.21.15.45:0/4177656096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78f00cf980 msgr2=0x7f78f01f2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:20.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.310+0000 7f78f4a00640 1 --2- 172.21.15.45:0/4177656096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78f00cf980 0x7f78f01f2700 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f78e000c4d0 tx=0x7f78e000c990 comp rx=0 tx=0).stop 2024-04-16T16:14:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.310+0000 7f78f4a00640 1 -- 172.21.15.45:0/4177656096 shutdown_connections 2024-04-16T16:14:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.310+0000 7f78f4a00640 1 --2- 172.21.15.45:0/4177656096 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f78d006c8f0 0x7f78d006edb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.310+0000 7f78f4a00640 1 --2- 172.21.15.45:0/4177656096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f0160e90 0x7f78f01f52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.310+0000 7f78f4a00640 1 --2- 172.21.15.45:0/4177656096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78f0145fe0 0x7f78f0142660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.310+0000 7f78f4a00640 1 --2- 172.21.15.45:0/4177656096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78f00cf980 0x7f78f01f2700 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.310+0000 7f78f4a00640 1 -- 172.21.15.45:0/4177656096 >> 172.21.15.45:0/4177656096 conn(0x7f78f0067e00 msgr2=0x7f78f00c5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.310+0000 7f78f4a00640 1 -- 172.21.15.45:0/4177656096 shutdown_connections 2024-04-16T16:14:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.310+0000 7f78f4a00640 1 -- 172.21.15.45:0/4177656096 wait complete. 2024-04-16T16:14:20.328 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:14:20.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.557+0000 7f638d800640 1 Processor -- start 2024-04-16T16:14:20.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.557+0000 7f638d800640 1 -- start start 2024-04-16T16:14:20.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.559+0000 7f638d800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f638815b920 0x7f63880b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:20.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.559+0000 7f638d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63880d0e80 0x7f638806ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:20.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.559+0000 7f638d800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63880ccb70 con 0x7f63880afda0 2024-04-16T16:14:20.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.559+0000 7f638d800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f638806e7a0 con 0x7f638815b920 2024-04-16T16:14:20.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.559+0000 7f638d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f638806f170 con 0x7f63880d0e80 2024-04-16T16:14:20.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.559+0000 7f637fe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f63880afda0 0x7f63880d0770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43010/0 (socket says 172.21.15.45:43010) 2024-04-16T16:14:20.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.559+0000 7f637fe00640 1 -- 172.21.15.45:0/2733093434 learned_addr learned my addr 172.21.15.45:0/2733093434 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:20.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.559+0000 7f6386a00640 1 --2- 172.21.15.45:0/2733093434 >> v2:172.21.15.45:3300/0 conn(0x7f638815b920 0x7f63880b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:20.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.560+0000 7f6387400640 1 --2- 172.21.15.45:0/2733093434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63880d0e80 0x7f638806ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:20.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.560+0000 7f6386000640 1 -- 172.21.15.45:0/2733093434 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1094838296 0 0) 0x7f63880ccb70 con 0x7f63880afda0 2024-04-16T16:14:20.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.560+0000 7f6386000640 1 -- 172.21.15.45:0/2733093434 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f636c003520 con 0x7f63880afda0 2024-04-16T16:14:20.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.560+0000 7f6386a00640 1 -- 172.21.15.45:0/2733093434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63880d0e80 msgr2=0x7f638806ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:20.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.560+0000 7f6386a00640 1 --2- 172.21.15.45:0/2733093434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63880d0e80 0x7f638806ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.560+0000 7f6386a00640 1 -- 172.21.15.45:0/2733093434 >> v1:172.21.15.45:6789/0 conn(0x7f63880afda0 legacy=0x7f63880d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:20.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.560+0000 7f6386a00640 1 -- 172.21.15.45:0/2733093434 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63880cce50 con 0x7f638815b920 2024-04-16T16:14:20.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.561+0000 7f6386a00640 1 --2- 172.21.15.45:0/2733093434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f638815b920 0x7f63880b4e20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f63780097a0 tx=0x7f637802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60d4a596ce718e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:20.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.561+0000 7f6386000640 1 -- 172.21.15.45:0/2733093434 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6378031780 con 0x7f638815b920 2024-04-16T16:14:20.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.561+0000 7f6386000640 1 -- 172.21.15.45:0/2733093434 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6378037070 con 0x7f638815b920 2024-04-16T16:14:20.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.561+0000 7f6386000640 1 -- 172.21.15.45:0/2733093434 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6378031b40 con 0x7f638815b920 2024-04-16T16:14:20.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.562+0000 7f638d800640 1 -- 172.21.15.45:0/2733093434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f638815b920 msgr2=0x7f63880b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:20.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.562+0000 7f638d800640 1 --2- 172.21.15.45:0/2733093434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f638815b920 0x7f63880b4e20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f63780097a0 tx=0x7f637802b820 comp rx=0 tx=0).stop 2024-04-16T16:14:20.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.562+0000 7f638d800640 1 -- 172.21.15.45:0/2733093434 shutdown_connections 2024-04-16T16:14:20.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.562+0000 7f638d800640 1 --2- 172.21.15.45:0/2733093434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63880d0e80 0x7f638806ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.562+0000 7f638d800640 1 --2- 172.21.15.45:0/2733093434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f638815b920 0x7f63880b4e20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.562+0000 7f638d800640 1 -- 172.21.15.45:0/2733093434 >> 172.21.15.45:0/2733093434 conn(0x7f63880b3970 msgr2=0x7f63880b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:20.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.562+0000 7f638d800640 1 -- 172.21.15.45:0/2733093434 shutdown_connections 2024-04-16T16:14:20.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.562+0000 7f638d800640 1 -- 172.21.15.45:0/2733093434 wait complete. 2024-04-16T16:14:20.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.564+0000 7f638d800640 1 Processor -- start 2024-04-16T16:14:20.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.564+0000 7f638d800640 1 -- start start 2024-04-16T16:14:20.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.565+0000 7f638d800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63880afda0 0x7f6388158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:20.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.565+0000 7f638d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63880d0e80 0x7f638814b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:20.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.565+0000 7f638d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f638815b920 0x7f638815d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:20.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.565+0000 7f638d800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f63880ce760 con 0x7f638815b920 2024-04-16T16:14:20.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.565+0000 7f638d800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63880ce5e0 con 0x7f63880afda0 2024-04-16T16:14:20.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.565+0000 7f638d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f63880ce8e0 con 0x7f63880d0e80 2024-04-16T16:14:20.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.565+0000 7f6387400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f638815b920 0x7f638815d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:20.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.565+0000 7f6387400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f638815b920 0x7f638815d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52220/0 (socket says 172.21.15.45:52220) 2024-04-16T16:14:20.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.565+0000 7f6387400640 1 -- 172.21.15.45:0/3167626945 learned_addr learned my addr 172.21.15.45:0/3167626945 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:20.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.566+0000 7f6386a00640 1 --2- 172.21.15.45:0/3167626945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63880afda0 0x7f6388158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:20.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.566+0000 7f637fe00640 1 --2- 172.21.15.45:0/3167626945 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63880d0e80 0x7f638814b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:20.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.566+0000 7f6387400640 1 -- 172.21.15.45:0/3167626945 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63880d0e80 msgr2=0x7f638814b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:20.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.566+0000 7f6387400640 1 --2- 172.21.15.45:0/3167626945 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63880d0e80 0x7f638814b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.566+0000 7f6387400640 1 -- 172.21.15.45:0/3167626945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63880afda0 msgr2=0x7f6388158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:20.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.566+0000 7f6387400640 1 --2- 172.21.15.45:0/3167626945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63880afda0 0x7f6388158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.566+0000 7f6387400640 1 -- 172.21.15.45:0/3167626945 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638814bc10 con 0x7f638815b920 2024-04-16T16:14:20.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.566+0000 7f637fe00640 1 --2- 172.21.15.45:0/3167626945 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63880d0e80 0x7f638814b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:20.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.566+0000 7f6387400640 1 --2- 172.21.15.45:0/3167626945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f638815b920 0x7f638815d720 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6380007e70 tx=0x7f638000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:20.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.566+0000 7f637ea00640 1 -- 172.21.15.45:0/3167626945 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6380014620 con 0x7f638815b920 2024-04-16T16:14:20.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.567+0000 7f637ea00640 1 -- 172.21.15.45:0/3167626945 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6380019070 con 0x7f638815b920 2024-04-16T16:14:20.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.567+0000 7f637ea00640 1 -- 172.21.15.45:0/3167626945 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6380014a20 con 0x7f638815b920 2024-04-16T16:14:20.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.567+0000 7f638d800640 1 -- 172.21.15.45:0/3167626945 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f638815e290 con 0x7f638815b920 2024-04-16T16:14:20.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.567+0000 7f638d800640 1 -- 172.21.15.45:0/3167626945 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63880adcf0 con 0x7f638815b920 2024-04-16T16:14:20.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.568+0000 7f638d800640 1 -- 172.21.15.45:0/3167626945 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6388075780 con 0x7f638815b920 2024-04-16T16:14:20.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.569+0000 7f637ea00640 1 -- 172.21.15.45:0/3167626945 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6380014bc0 con 0x7f638815b920 2024-04-16T16:14:20.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.570+0000 7f637ea00640 1 --2- 172.21.15.45:0/3167626945 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f636406c8f0 0x7f636406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:20.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.570+0000 7f637ea00640 1 -- 172.21.15.45:0/3167626945 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f638008c4f0 con 0x7f638815b920 2024-04-16T16:14:20.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.570+0000 7f6386a00640 1 --2- 172.21.15.45:0/3167626945 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f636406c8f0 0x7f636406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:20.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.572+0000 7f6386a00640 1 --2- 172.21.15.45:0/3167626945 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f636406c8f0 0x7f636406edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6378036000 tx=0x7f637803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:20.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.574+0000 7f637ea00640 1 -- 172.21.15.45:0/3167626945 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6380011260 con 0x7f638815b920 2024-04-16T16:14:20.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.782+0000 7f638d800640 1 -- 172.21.15.45:0/3167626945 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6388158680 con 0x7f638815b920 2024-04-16T16:14:20.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.783+0000 7f637ea00640 1 -- 172.21.15.45:0/3167626945 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7f638005a890 con 0x7f638815b920 2024-04-16T16:14:20.782 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:14:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.786+0000 7f638d800640 1 -- 172.21.15.45:0/3167626945 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f636406c8f0 msgr2=0x7f636406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.786+0000 7f638d800640 1 --2- 172.21.15.45:0/3167626945 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f636406c8f0 0x7f636406edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6378036000 tx=0x7f637803d040 comp rx=0 tx=0).stop 2024-04-16T16:14:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.786+0000 7f638d800640 1 -- 172.21.15.45:0/3167626945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f638815b920 msgr2=0x7f638815d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.786+0000 7f638d800640 1 --2- 172.21.15.45:0/3167626945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f638815b920 0x7f638815d720 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6380007e70 tx=0x7f638000e4b0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.786+0000 7f638d800640 1 -- 172.21.15.45:0/3167626945 shutdown_connections 2024-04-16T16:14:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.786+0000 7f638d800640 1 --2- 172.21.15.45:0/3167626945 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f636406c8f0 0x7f636406edb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.786+0000 7f638d800640 1 --2- 172.21.15.45:0/3167626945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f638815b920 0x7f638815d720 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.786+0000 7f638d800640 1 --2- 172.21.15.45:0/3167626945 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63880d0e80 0x7f638814b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.786+0000 7f638d800640 1 --2- 172.21.15.45:0/3167626945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63880afda0 0x7f6388158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.786+0000 7f638d800640 1 -- 172.21.15.45:0/3167626945 >> 172.21.15.45:0/3167626945 conn(0x7f63880b3970 msgr2=0x7f63880b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.786+0000 7f638d800640 1 -- 172.21.15.45:0/3167626945 shutdown_connections 2024-04-16T16:14:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:20.786+0000 7f638d800640 1 -- 172.21.15.45:0/3167626945 wait complete. 2024-04-16T16:14:20.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:14:21.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.038+0000 7fdacb200640 1 Processor -- start 2024-04-16T16:14:21.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.038+0000 7fdacb200640 1 -- start start 2024-04-16T16:14:21.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.040+0000 7fdacb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdac40b58f0 0x7fdac40b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:21.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.040+0000 7fdacb200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdac406e460 0x7fdac406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:21.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.040+0000 7fdacb200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac4146590 con 0x7fdac41461b0 2024-04-16T16:14:21.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.040+0000 7fdacb200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fdac406dfd0 con 0x7fdac406e460 2024-04-16T16:14:21.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.040+0000 7fdacb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdac40b6290 con 0x7fdac40b58f0 2024-04-16T16:14:21.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.040+0000 7fdaca800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdac406e460 0x7fdac406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:21.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.040+0000 7fdaca800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdac406e460 0x7fdac406e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52226/0 (socket says 172.21.15.45:52226) 2024-04-16T16:14:21.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.040+0000 7fdaca800640 1 -- 172.21.15.45:0/1510478152 learned_addr learned my addr 172.21.15.45:0/1510478152 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:21.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.041+0000 7fdac9400640 1 --2- 172.21.15.45:0/1510478152 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdac40b58f0 0x7fdac40b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:21.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.041+0000 7fdaca800640 1 -- 172.21.15.45:0/1510478152 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdac40b58f0 msgr2=0x7fdac40b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:21.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.041+0000 7fdaca800640 1 --2- 172.21.15.45:0/1510478152 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdac40b58f0 0x7fdac40b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.041+0000 7fdaca800640 1 -- 172.21.15.45:0/1510478152 >> v1:172.21.15.103:6789/0 conn(0x7fdac41461b0 legacy=0x7fdac4160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:21.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.041+0000 7fdaca800640 1 -- 172.21.15.45:0/1510478152 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac40c5750 con 0x7fdac406e460 2024-04-16T16:14:21.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.042+0000 7fdaca800640 1 --2- 172.21.15.45:0/1510478152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac406e460 0x7fdac406e8f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdac0009e50 tx=0x7fdac002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7225b74a4bb3d85e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:21.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.042+0000 7fdac8a00640 1 -- 172.21.15.45:0/1510478152 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac002cb00 con 0x7fdac406e460 2024-04-16T16:14:21.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.043+0000 7fdac8a00640 1 -- 172.21.15.45:0/1510478152 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdac002cca0 con 0x7fdac406e460 2024-04-16T16:14:21.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.043+0000 7fdac8a00640 1 -- 172.21.15.45:0/1510478152 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac0034610 con 0x7fdac406e460 2024-04-16T16:14:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.043+0000 7fdacb200640 1 -- 172.21.15.45:0/1510478152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac406e460 msgr2=0x7fdac406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.043+0000 7fdacb200640 1 --2- 172.21.15.45:0/1510478152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac406e460 0x7fdac406e8f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdac0009e50 tx=0x7fdac002bf60 comp rx=0 tx=0).stop 2024-04-16T16:14:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.043+0000 7fdacb200640 1 -- 172.21.15.45:0/1510478152 shutdown_connections 2024-04-16T16:14:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.043+0000 7fdacb200640 1 --2- 172.21.15.45:0/1510478152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac406e460 0x7fdac406e8f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.043+0000 7fdacb200640 1 --2- 172.21.15.45:0/1510478152 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdac40b58f0 0x7fdac40b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.043+0000 7fdacb200640 1 -- 172.21.15.45:0/1510478152 >> 172.21.15.45:0/1510478152 conn(0x7fdac406b930 msgr2=0x7fdac406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.044+0000 7fdacb200640 1 -- 172.21.15.45:0/1510478152 shutdown_connections 2024-04-16T16:14:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.044+0000 7fdacb200640 1 -- 172.21.15.45:0/1510478152 wait complete. 2024-04-16T16:14:21.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.045+0000 7fdacb200640 1 Processor -- start 2024-04-16T16:14:21.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.045+0000 7fdacb200640 1 -- start start 2024-04-16T16:14:21.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.046+0000 7fdacb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac406e460 0x7fdac415bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:21.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.046+0000 7fdacb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdac40b58f0 0x7fdac41e6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:21.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.046+0000 7fdacb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdac41461b0 0x7fdac41e7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:21.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.046+0000 7fdacb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdac40c5cd0 con 0x7fdac406e460 2024-04-16T16:14:21.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.046+0000 7fdacb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdac40c59f0 con 0x7fdac41461b0 2024-04-16T16:14:21.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.046+0000 7fdacb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdac40c5fb0 con 0x7fdac40b58f0 2024-04-16T16:14:21.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.047+0000 7fdac9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac406e460 0x7fdac415bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:21.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.047+0000 7fdac9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac406e460 0x7fdac415bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52240/0 (socket says 172.21.15.45:52240) 2024-04-16T16:14:21.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.047+0000 7fdac9e00640 1 -- 172.21.15.45:0/2955879263 learned_addr learned my addr 172.21.15.45:0/2955879263 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:21.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.047+0000 7fdaca800640 1 --2- 172.21.15.45:0/2955879263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdac41461b0 0x7fdac41e7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:21.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.047+0000 7fdac9400640 1 --2- 172.21.15.45:0/2955879263 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdac40b58f0 0x7fdac41e6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:21.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.047+0000 7fdac9e00640 1 -- 172.21.15.45:0/2955879263 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdac40b58f0 msgr2=0x7fdac41e6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:21.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.047+0000 7fdac9e00640 1 --2- 172.21.15.45:0/2955879263 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdac40b58f0 0x7fdac41e6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.047+0000 7fdac9e00640 1 -- 172.21.15.45:0/2955879263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdac41461b0 msgr2=0x7fdac41e7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:21.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.047+0000 7fdac9e00640 1 --2- 172.21.15.45:0/2955879263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdac41461b0 0x7fdac41e7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.047+0000 7fdac9e00640 1 -- 172.21.15.45:0/2955879263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac41e6c50 con 0x7fdac406e460 2024-04-16T16:14:21.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.048+0000 7fdac9e00640 1 --2- 172.21.15.45:0/2955879263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac406e460 0x7fdac415bd50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdab800c400 tx=0x7fdab800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:21.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.048+0000 7fdabe000640 1 -- 172.21.15.45:0/2955879263 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab8017070 con 0x7fdac406e460 2024-04-16T16:14:21.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.048+0000 7fdabe000640 1 -- 172.21.15.45:0/2955879263 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdab80052d0 con 0x7fdac406e460 2024-04-16T16:14:21.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.048+0000 7fdabe000640 1 -- 172.21.15.45:0/2955879263 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab8003830 con 0x7fdac406e460 2024-04-16T16:14:21.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.048+0000 7fdacb200640 1 -- 172.21.15.45:0/2955879263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac41e6f30 con 0x7fdac406e460 2024-04-16T16:14:21.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.048+0000 7fdacb200640 1 -- 172.21.15.45:0/2955879263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac41e74f0 con 0x7fdac406e460 2024-04-16T16:14:21.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.049+0000 7fdacb200640 1 -- 172.21.15.45:0/2955879263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdac4075780 con 0x7fdac406e460 2024-04-16T16:14:21.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.050+0000 7fdabe000640 1 -- 172.21.15.45:0/2955879263 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdab801a070 con 0x7fdac406e460 2024-04-16T16:14:21.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.051+0000 7fdabe000640 1 --2- 172.21.15.45:0/2955879263 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdaa806c940 0x7fdaa806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:21.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.051+0000 7fdabe000640 1 -- 172.21.15.45:0/2955879263 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fdab808af20 con 0x7fdac406e460 2024-04-16T16:14:21.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.052+0000 7fdac9400640 1 --2- 172.21.15.45:0/2955879263 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdaa806c940 0x7fdaa806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:21.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.053+0000 7fdac9400640 1 --2- 172.21.15.45:0/2955879263 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdaa806c940 0x7fdaa806ee00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdac40b3f20 tx=0x7fdab4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:21.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.056+0000 7fdabe000640 1 -- 172.21.15.45:0/2955879263 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdab8059340 con 0x7fdac406e460 2024-04-16T16:14:21.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.263+0000 7fdacb200640 1 -- 172.21.15.45:0/2955879263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdac41e97c0 con 0x7fdac406e460 2024-04-16T16:14:21.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.264+0000 7fdabe000640 1 -- 172.21.15.45:0/2955879263 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7fdab8058c40 con 0x7fdac406e460 2024-04-16T16:14:21.263 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:14:21.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.266+0000 7fdacb200640 1 -- 172.21.15.45:0/2955879263 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdaa806c940 msgr2=0x7fdaa806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:21.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.266+0000 7fdacb200640 1 --2- 172.21.15.45:0/2955879263 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdaa806c940 0x7fdaa806ee00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdac40b3f20 tx=0x7fdab4008040 comp rx=0 tx=0).stop 2024-04-16T16:14:21.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.266+0000 7fdacb200640 1 -- 172.21.15.45:0/2955879263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac406e460 msgr2=0x7fdac415bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:21.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.266+0000 7fdacb200640 1 --2- 172.21.15.45:0/2955879263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac406e460 0x7fdac415bd50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdab800c400 tx=0x7fdab800c8c0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.266+0000 7fdacb200640 1 -- 172.21.15.45:0/2955879263 shutdown_connections 2024-04-16T16:14:21.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.266+0000 7fdacb200640 1 --2- 172.21.15.45:0/2955879263 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdaa806c940 0x7fdaa806ee00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.266+0000 7fdacb200640 1 --2- 172.21.15.45:0/2955879263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdac41461b0 0x7fdac41e7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.266+0000 7fdacb200640 1 --2- 172.21.15.45:0/2955879263 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdac40b58f0 0x7fdac41e6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.266+0000 7fdacb200640 1 --2- 172.21.15.45:0/2955879263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac406e460 0x7fdac415bd50 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.266+0000 7fdacb200640 1 -- 172.21.15.45:0/2955879263 >> 172.21.15.45:0/2955879263 conn(0x7fdac406b930 msgr2=0x7fdac40b4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:21.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.267+0000 7fdacb200640 1 -- 172.21.15.45:0/2955879263 shutdown_connections 2024-04-16T16:14:21.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.267+0000 7fdacb200640 1 -- 172.21.15.45:0/2955879263 wait complete. 2024-04-16T16:14:21.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:14:21.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.499+0000 7f0125e00640 1 Processor -- start 2024-04-16T16:14:21.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.500+0000 7f0125e00640 1 -- start start 2024-04-16T16:14:21.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.501+0000 7f0125e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0120147550 0x7f0120072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:21.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.501+0000 7f0125e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f01200726c0 0x7f01200b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:21.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.501+0000 7f0125e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f012006df00 con 0x7f01200b58e0 2024-04-16T16:14:21.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.501+0000 7f0125e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f012006fc50 con 0x7f0120147550 2024-04-16T16:14:21.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.501+0000 7f0125e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f01200b5d30 con 0x7f01200726c0 2024-04-16T16:14:21.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.502+0000 7f011ea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0120147550 0x7f0120072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:21.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.502+0000 7f011ea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0120147550 0x7f0120072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52242/0 (socket says 172.21.15.45:52242) 2024-04-16T16:14:21.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.502+0000 7f011ea00640 1 -- 172.21.15.45:0/3795685117 learned_addr learned my addr 172.21.15.45:0/3795685117 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:21.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.502+0000 7f011ea00640 1 -- 172.21.15.45:0/3795685117 >> v2:172.21.15.103:3300/0 conn(0x7f01200726c0 msgr2=0x7f01200b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:21.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.502+0000 7f011ea00640 1 --2- 172.21.15.45:0/3795685117 >> v2:172.21.15.103:3300/0 conn(0x7f01200726c0 0x7f01200b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.502+0000 7f011ea00640 1 -- 172.21.15.45:0/3795685117 >> v1:172.21.15.103:6789/0 conn(0x7f01200b58e0 legacy=0x7f0120162670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:21.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.502+0000 7f011ea00640 1 -- 172.21.15.45:0/3795685117 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012006e1e0 con 0x7f0120147550 2024-04-16T16:14:21.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.503+0000 7f011ea00640 1 --2- 172.21.15.45:0/3795685117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0120147550 0x7f0120072180 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0120070510 tx=0x7f011002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732fb600b5eeddb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:21.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.504+0000 7f011d600640 1 -- 172.21.15.45:0/3795685117 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011002ca80 con 0x7f0120147550 2024-04-16T16:14:21.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.504+0000 7f011d600640 1 -- 172.21.15.45:0/3795685117 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f011002cc20 con 0x7f0120147550 2024-04-16T16:14:21.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.504+0000 7f011d600640 1 -- 172.21.15.45:0/3795685117 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01100343f0 con 0x7f0120147550 2024-04-16T16:14:21.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.504+0000 7f0125e00640 1 -- 172.21.15.45:0/3795685117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0120147550 msgr2=0x7f0120072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:21.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.504+0000 7f0125e00640 1 --2- 172.21.15.45:0/3795685117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0120147550 0x7f0120072180 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0120070510 tx=0x7f011002bee0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.504+0000 7f0125e00640 1 -- 172.21.15.45:0/3795685117 shutdown_connections 2024-04-16T16:14:21.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.504+0000 7f0125e00640 1 --2- 172.21.15.45:0/3795685117 >> v2:172.21.15.103:3300/0 conn(0x7f01200726c0 0x7f01200b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.504+0000 7f0125e00640 1 --2- 172.21.15.45:0/3795685117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0120147550 0x7f0120072180 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.504+0000 7f0125e00640 1 -- 172.21.15.45:0/3795685117 >> 172.21.15.45:0/3795685117 conn(0x7f012006b930 msgr2=0x7f012006bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:21.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.504+0000 7f0125e00640 1 -- 172.21.15.45:0/3795685117 shutdown_connections 2024-04-16T16:14:21.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.504+0000 7f0125e00640 1 -- 172.21.15.45:0/3795685117 wait complete. 2024-04-16T16:14:21.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.506+0000 7f0125e00640 1 Processor -- start 2024-04-16T16:14:21.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.506+0000 7f0125e00640 1 -- start start 2024-04-16T16:14:21.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.507+0000 7f0125e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01200726c0 0x7f012015abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:21.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.507+0000 7f0125e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01200b58e0 0x7f01200cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:21.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.507+0000 7f0125e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0120147550 0x7f01200ce520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:21.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.507+0000 7f0125e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f012006ea80 con 0x7f0120147550 2024-04-16T16:14:21.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.507+0000 7f0125e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f012006e7a0 con 0x7f01200726c0 2024-04-16T16:14:21.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.507+0000 7f0125e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f01200b48a0 con 0x7f01200b58e0 2024-04-16T16:14:21.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.508+0000 7f011f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0120147550 0x7f01200ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:21.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.508+0000 7f011f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0120147550 0x7f01200ce520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52256/0 (socket says 172.21.15.45:52256) 2024-04-16T16:14:21.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.508+0000 7f011f400640 1 -- 172.21.15.45:0/2680157220 learned_addr learned my addr 172.21.15.45:0/2680157220 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:21.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.508+0000 7f011ea00640 1 --2- 172.21.15.45:0/2680157220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01200726c0 0x7f012015abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:21.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.508+0000 7f011e000640 1 --2- 172.21.15.45:0/2680157220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01200b58e0 0x7f01200cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:21.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.508+0000 7f011f400640 1 -- 172.21.15.45:0/2680157220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01200b58e0 msgr2=0x7f01200cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:21.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.508+0000 7f011f400640 1 --2- 172.21.15.45:0/2680157220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01200b58e0 0x7f01200cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.508+0000 7f011f400640 1 -- 172.21.15.45:0/2680157220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01200726c0 msgr2=0x7f012015abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:21.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.508+0000 7f011f400640 1 --2- 172.21.15.45:0/2680157220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01200726c0 0x7f012015abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.508+0000 7f011f400640 1 -- 172.21.15.45:0/2680157220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012015b0f0 con 0x7f0120147550 2024-04-16T16:14:21.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.508+0000 7f011ea00640 1 --2- 172.21.15.45:0/2680157220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01200726c0 0x7f012015abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:14:21.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.509+0000 7f011f400640 1 --2- 172.21.15.45:0/2680157220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0120147550 0x7f01200ce520 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f011800c390 tx=0x7f011800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:21.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.509+0000 7f0117400640 1 -- 172.21.15.45:0/2680157220 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0118017070 con 0x7f0120147550 2024-04-16T16:14:21.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.509+0000 7f0117400640 1 -- 172.21.15.45:0/2680157220 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01180035f0 con 0x7f0120147550 2024-04-16T16:14:21.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.509+0000 7f0117400640 1 -- 172.21.15.45:0/2680157220 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01180039e0 con 0x7f0120147550 2024-04-16T16:14:21.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.509+0000 7f0125e00640 1 -- 172.21.15.45:0/2680157220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01200cd480 con 0x7f0120147550 2024-04-16T16:14:21.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.509+0000 7f0125e00640 1 -- 172.21.15.45:0/2680157220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01200cda40 con 0x7f0120147550 2024-04-16T16:14:21.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.510+0000 7f0125e00640 1 -- 172.21.15.45:0/2680157220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0120075780 con 0x7f0120147550 2024-04-16T16:14:21.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.511+0000 7f0117400640 1 -- 172.21.15.45:0/2680157220 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0118003b80 con 0x7f0120147550 2024-04-16T16:14:21.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.513+0000 7f0117400640 1 --2- 172.21.15.45:0/2680157220 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f00fc06c8f0 0x7f00fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:21.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.513+0000 7f0117400640 1 -- 172.21.15.45:0/2680157220 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f011808af00 con 0x7f0120147550 2024-04-16T16:14:21.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.513+0000 7f011ea00640 1 --2- 172.21.15.45:0/2680157220 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f00fc06c8f0 0x7f00fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:21.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.515+0000 7f011ea00640 1 --2- 172.21.15.45:0/2680157220 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f00fc06c8f0 0x7f00fc06edb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0110035000 tx=0x7f0110030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.517+0000 7f0117400640 1 -- 172.21.15.45:0/2680157220 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f01180592b0 con 0x7f0120147550 2024-04-16T16:14:21.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.735+0000 7f0125e00640 1 -- 172.21.15.45:0/2680157220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f012006f4b0 con 0x7f0120147550 2024-04-16T16:14:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.736+0000 7f0117400640 1 -- 172.21.15.45:0/2680157220 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7f0118058bb0 con 0x7f0120147550 2024-04-16T16:14:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:listed 0 entries 2024-04-16T16:14:21.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.739+0000 7f0125e00640 1 -- 172.21.15.45:0/2680157220 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f00fc06c8f0 msgr2=0x7f00fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:21.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.739+0000 7f0125e00640 1 --2- 172.21.15.45:0/2680157220 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f00fc06c8f0 0x7f00fc06edb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0110035000 tx=0x7f0110030000 comp rx=0 tx=0).stop 2024-04-16T16:14:21.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.739+0000 7f0125e00640 1 -- 172.21.15.45:0/2680157220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0120147550 msgr2=0x7f01200ce520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:21.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.739+0000 7f0125e00640 1 --2- 172.21.15.45:0/2680157220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0120147550 0x7f01200ce520 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f011800c390 tx=0x7f011800c850 comp rx=0 tx=0).stop 2024-04-16T16:14:21.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.739+0000 7f0125e00640 1 -- 172.21.15.45:0/2680157220 shutdown_connections 2024-04-16T16:14:21.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.739+0000 7f0125e00640 1 --2- 172.21.15.45:0/2680157220 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f00fc06c8f0 0x7f00fc06edb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.739+0000 7f0125e00640 1 --2- 172.21.15.45:0/2680157220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0120147550 0x7f01200ce520 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.739+0000 7f0125e00640 1 --2- 172.21.15.45:0/2680157220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01200b58e0 0x7f01200cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.739+0000 7f0125e00640 1 --2- 172.21.15.45:0/2680157220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01200726c0 0x7f012015abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:21.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.739+0000 7f0125e00640 1 -- 172.21.15.45:0/2680157220 >> 172.21.15.45:0/2680157220 conn(0x7f012006b930 msgr2=0x7f012006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:21.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.739+0000 7f0125e00640 1 -- 172.21.15.45:0/2680157220 shutdown_connections 2024-04-16T16:14:21.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:21.739+0000 7f0125e00640 1 -- 172.21.15.45:0/2680157220 wait complete. 2024-04-16T16:14:21.758 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-04-16T16:14:21.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:21.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-04-16T16:14:21.869 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:14:21.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:21.869 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-16T16:14:21.917 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-16T16:14:21.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:14:22.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.146+0000 7f1a99800640 1 Processor -- start 2024-04-16T16:14:22.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.147+0000 7f1a99800640 1 -- start start 2024-04-16T16:14:22.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.148+0000 7f1a99800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1a941462e0 0x7f1a94072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:22.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.148+0000 7f1a99800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a940cf6d0 con 0x7f1a9415ea70 2024-04-16T16:14:22.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.148+0000 7f1a99800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a941466c0 con 0x7f1a940728c0 2024-04-16T16:14:22.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.148+0000 7f1a99800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1a940cf3f0 con 0x7f1a941462e0 2024-04-16T16:14:22.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.149+0000 7f1a93400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1a9415ea70 0x7f1a94166490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43026/0 (socket says 172.21.15.45:43026) 2024-04-16T16:14:22.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.149+0000 7f1a93400640 1 -- 172.21.15.45:0/2275428920 learned_addr learned my addr 172.21.15.45:0/2275428920 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:22.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.149+0000 7f1a92a00640 1 --2- 172.21.15.45:0/2275428920 >> v2:172.21.15.103:3300/0 conn(0x7f1a941462e0 0x7f1a94072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:22.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.149+0000 7f1a92000640 1 -- 172.21.15.45:0/2275428920 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1417349249 0 0) 0x7f1a940cf6d0 con 0x7f1a9415ea70 2024-04-16T16:14:22.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.149+0000 7f1a92000640 1 -- 172.21.15.45:0/2275428920 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a78003520 con 0x7f1a9415ea70 2024-04-16T16:14:22.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.150+0000 7f1a92000640 1 -- 172.21.15.45:0/2275428920 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 191270399 0 0) 0x7f1a78003520 con 0x7f1a9415ea70 2024-04-16T16:14:22.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.150+0000 7f1a92000640 1 -- 172.21.15.45:0/2275428920 >> v2:172.21.15.103:3300/0 conn(0x7f1a941462e0 msgr2=0x7f1a94072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:22.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.150+0000 7f1a92000640 1 --2- 172.21.15.45:0/2275428920 >> v2:172.21.15.103:3300/0 conn(0x7f1a941462e0 0x7f1a94072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.150+0000 7f1a92000640 1 -- 172.21.15.45:0/2275428920 >> v1:172.21.15.103:6789/0 conn(0x7f1a940728c0 legacy=0x7f1a940ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:22.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.150+0000 7f1a92000640 1 -- 172.21.15.45:0/2275428920 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a9415cf80 con 0x7f1a9415ea70 2024-04-16T16:14:22.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.150+0000 7f1a92000640 1 -- 172.21.15.45:0/2275428920 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f1a8c0019f0 con 0x7f1a9415ea70 2024-04-16T16:14:22.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.150+0000 7f1a92000640 1 -- 172.21.15.45:0/2275428920 >> v1:172.21.15.45:6789/0 conn(0x7f1a9415ea70 legacy=0x7f1a94166490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:22.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.150+0000 7f1a92000640 1 --2- 172.21.15.45:0/2275428920 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a780039b0 0x7f1a78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:22.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.150+0000 7f1a92000640 1 --2- 172.21.15.45:0/2275428920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a78004800 0x7f1a78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:22.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.150+0000 7f1a92000640 1 --2- 172.21.15.45:0/2275428920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a780051c0 0x7f1a7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:22.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.150+0000 7f1a92000640 1 -- 172.21.15.45:0/2275428920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a940cf3f0 con 0x7f1a780051c0 2024-04-16T16:14:22.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.150+0000 7f1a92000640 1 -- 172.21.15.45:0/2275428920 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a7800c160 con 0x7f1a78004800 2024-04-16T16:14:22.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.150+0000 7f1a92000640 1 -- 172.21.15.45:0/2275428920 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1a7800c360 con 0x7f1a780039b0 2024-04-16T16:14:22.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.150+0000 7f1a93400640 1 --2- 172.21.15.45:0/2275428920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a780051c0 0x7f1a7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:22.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.150+0000 7f1a92a00640 1 --2- 172.21.15.45:0/2275428920 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a780039b0 0x7f1a78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:22.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.151+0000 7f1a8be00640 1 --2- 172.21.15.45:0/2275428920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a78004800 0x7f1a78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:22.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.151+0000 7f1a93400640 1 -- 172.21.15.45:0/2275428920 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a780039b0 msgr2=0x7f1a78003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:22.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.151+0000 7f1a93400640 1 --2- 172.21.15.45:0/2275428920 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a780039b0 0x7f1a78003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.151+0000 7f1a93400640 1 -- 172.21.15.45:0/2275428920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a78004800 msgr2=0x7f1a78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:22.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.151+0000 7f1a93400640 1 --2- 172.21.15.45:0/2275428920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a78004800 0x7f1a78004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.151+0000 7f1a93400640 1 -- 172.21.15.45:0/2275428920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a78003520 con 0x7f1a780051c0 2024-04-16T16:14:22.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.151+0000 7f1a8be00640 1 --2- 172.21.15.45:0/2275428920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a78004800 0x7f1a78004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:22.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.152+0000 7f1a93400640 1 -- 172.21.15.45:0/2275428920 reap_dead start 2024-04-16T16:14:22.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.152+0000 7f1a93400640 1 --2- 172.21.15.45:0/2275428920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a780051c0 0x7f1a7800b9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c00a840 tx=0x7f1a78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab7cc23e699241e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:22.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.152+0000 7f1a92000640 1 -- 172.21.15.45:0/2275428920 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c00a460 con 0x7f1a780051c0 2024-04-16T16:14:22.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.152+0000 7f1a92000640 1 -- 172.21.15.45:0/2275428920 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a8c034650 con 0x7f1a780051c0 2024-04-16T16:14:22.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.152+0000 7f1a92000640 1 -- 172.21.15.45:0/2275428920 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c034a40 con 0x7f1a780051c0 2024-04-16T16:14:22.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.153+0000 7f1a99800640 1 -- 172.21.15.45:0/2275428920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a780051c0 msgr2=0x7f1a7800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:22.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.153+0000 7f1a99800640 1 --2- 172.21.15.45:0/2275428920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a780051c0 0x7f1a7800b9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c00a840 tx=0x7f1a78005830 comp rx=0 tx=0).stop 2024-04-16T16:14:22.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.153+0000 7f1a99800640 1 -- 172.21.15.45:0/2275428920 shutdown_connections 2024-04-16T16:14:22.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.153+0000 7f1a99800640 1 --2- 172.21.15.45:0/2275428920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a780051c0 0x7f1a7800b9c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.153+0000 7f1a99800640 1 -- 172.21.15.45:0/2275428920 >> 172.21.15.45:0/2275428920 conn(0x7f1a940b2750 msgr2=0x7f1a940b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:22.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.153+0000 7f1a99800640 1 -- 172.21.15.45:0/2275428920 shutdown_connections 2024-04-16T16:14:22.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.153+0000 7f1a99800640 1 -- 172.21.15.45:0/2275428920 wait complete. 2024-04-16T16:14:22.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.155+0000 7f1a99800640 1 Processor -- start 2024-04-16T16:14:22.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.155+0000 7f1a99800640 1 -- start start 2024-04-16T16:14:22.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.156+0000 7f1a99800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a780051c0 0x7f1a9413dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:22.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.156+0000 7f1a99800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a94140390 0x7f1a94140810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:22.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.156+0000 7f1a99800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a94141720 0x7f1a94141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:22.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.156+0000 7f1a99800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a94072380 con 0x7f1a780051c0 2024-04-16T16:14:22.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.156+0000 7f1a99800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9415d540 con 0x7f1a94140390 2024-04-16T16:14:22.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.156+0000 7f1a99800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1a94072500 con 0x7f1a94141720 2024-04-16T16:14:22.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.156+0000 7f1a92a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a780051c0 0x7f1a9413dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:22.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.157+0000 7f1a92a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a780051c0 0x7f1a9413dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52266/0 (socket says 172.21.15.45:52266) 2024-04-16T16:14:22.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.157+0000 7f1a92a00640 1 -- 172.21.15.45:0/2445578719 learned_addr learned my addr 172.21.15.45:0/2445578719 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:22.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.157+0000 7f1a8be00640 1 --2- 172.21.15.45:0/2445578719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a94140390 0x7f1a94140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:22.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.157+0000 7f1a93400640 1 --2- 172.21.15.45:0/2445578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a94141720 0x7f1a94141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:22.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.157+0000 7f1a92a00640 1 -- 172.21.15.45:0/2445578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a94141720 msgr2=0x7f1a94141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:22.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.157+0000 7f1a92a00640 1 --2- 172.21.15.45:0/2445578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a94141720 0x7f1a94141ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.157+0000 7f1a92a00640 1 -- 172.21.15.45:0/2445578719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a94140390 msgr2=0x7f1a94140810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:22.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.157+0000 7f1a92a00640 1 --2- 172.21.15.45:0/2445578719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a94140390 0x7f1a94140810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.157+0000 7f1a92a00640 1 -- 172.21.15.45:0/2445578719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a941420e0 con 0x7f1a780051c0 2024-04-16T16:14:22.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.157+0000 7f1a92a00640 1 --2- 172.21.15.45:0/2445578719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a780051c0 0x7f1a9413dac0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1a84011570 tx=0x7f1a84011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:22.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.158+0000 7f1a8aa00640 1 -- 172.21.15.45:0/2445578719 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84015930 con 0x7f1a780051c0 2024-04-16T16:14:22.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.158+0000 7f1a8aa00640 1 -- 172.21.15.45:0/2445578719 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a84015ad0 con 0x7f1a780051c0 2024-04-16T16:14:22.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.158+0000 7f1a8aa00640 1 -- 172.21.15.45:0/2445578719 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8400f460 con 0x7f1a780051c0 2024-04-16T16:14:22.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.158+0000 7f1a99800640 1 -- 172.21.15.45:0/2445578719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a941423a0 con 0x7f1a780051c0 2024-04-16T16:14:22.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.158+0000 7f1a99800640 1 -- 172.21.15.45:0/2445578719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a9413e3f0 con 0x7f1a780051c0 2024-04-16T16:14:22.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.159+0000 7f1a99800640 1 -- 172.21.15.45:0/2445578719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a94075780 con 0x7f1a780051c0 2024-04-16T16:14:22.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.160+0000 7f1a8aa00640 1 -- 172.21.15.45:0/2445578719 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1a8400f600 con 0x7f1a780051c0 2024-04-16T16:14:22.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.161+0000 7f1a8aa00640 1 --2- 172.21.15.45:0/2445578719 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1a7006c8f0 0x7f1a7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:22.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.162+0000 7f1a8aa00640 1 -- 172.21.15.45:0/2445578719 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1a8408c410 con 0x7f1a780051c0 2024-04-16T16:14:22.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.162+0000 7f1a8be00640 1 --2- 172.21.15.45:0/2445578719 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1a7006c8f0 0x7f1a7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:22.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.164+0000 7f1a8be00640 1 --2- 172.21.15.45:0/2445578719 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1a7006c8f0 0x7f1a7006edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1a94142c10 tx=0x7f1a7c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:22.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.167+0000 7f1a8aa00640 1 -- 172.21.15.45:0/2445578719 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a84019030 con 0x7f1a780051c0 2024-04-16T16:14:22.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.412+0000 7f1a99800640 1 -- 172.21.15.45:0/2445578719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a940b0ae0 con 0x7f1a780051c0 2024-04-16T16:14:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.413+0000 7f1a8aa00640 1 -- 172.21.15.45:0/2445578719 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5178 (secure 0 0 0) 0x7f1a8405a850 con 0x7f1a780051c0 2024-04-16T16:14:22.412 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:22.412 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2486891866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2486891866},{"type":"v1","addr":"172.21.15.103:6836","nonce":2486891866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1877281473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1877281473},{"type":"v1","addr":"172.21.15.103:6837","nonce":1877281473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/315848518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":315848518},{"type":"v1","addr":"172.21.15.45:6833","nonce":315848518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:13:25.369839+0000","modified":"2024-04-16T16:13:28.463144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2461453826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2461453826},{"type":"v1","addr":"172.21.15.45:6835","nonce":2461453826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-04-16T16:14:22.413 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 8 2024-04-16T16:14:22.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.416+0000 7f1a99800640 1 -- 172.21.15.45:0/2445578719 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1a7006c8f0 msgr2=0x7f1a7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:22.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.416+0000 7f1a99800640 1 --2- 172.21.15.45:0/2445578719 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1a7006c8f0 0x7f1a7006edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1a94142c10 tx=0x7f1a7c009040 comp rx=0 tx=0).stop 2024-04-16T16:14:22.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.416+0000 7f1a99800640 1 -- 172.21.15.45:0/2445578719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a780051c0 msgr2=0x7f1a9413dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:22.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.416+0000 7f1a99800640 1 --2- 172.21.15.45:0/2445578719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a780051c0 0x7f1a9413dac0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1a84011570 tx=0x7f1a84011a30 comp rx=0 tx=0).stop 2024-04-16T16:14:22.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.416+0000 7f1a99800640 1 -- 172.21.15.45:0/2445578719 shutdown_connections 2024-04-16T16:14:22.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.416+0000 7f1a99800640 1 --2- 172.21.15.45:0/2445578719 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1a7006c8f0 0x7f1a7006edb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.416+0000 7f1a99800640 1 --2- 172.21.15.45:0/2445578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a94141720 0x7f1a94141ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.416+0000 7f1a99800640 1 --2- 172.21.15.45:0/2445578719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a94140390 0x7f1a94140810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.416+0000 7f1a99800640 1 --2- 172.21.15.45:0/2445578719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a780051c0 0x7f1a9413dac0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.416+0000 7f1a99800640 1 -- 172.21.15.45:0/2445578719 >> 172.21.15.45:0/2445578719 conn(0x7f1a940b2750 msgr2=0x7f1a9406e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:22.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.416+0000 7f1a99800640 1 -- 172.21.15.45:0/2445578719 shutdown_connections 2024-04-16T16:14:22.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.416+0000 7f1a99800640 1 -- 172.21.15.45:0/2445578719 wait complete. 2024-04-16T16:14:22.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:22.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.639+0000 7fe11fa00640 1 Processor -- start 2024-04-16T16:14:22.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.639+0000 7fe11fa00640 1 -- start start 2024-04-16T16:14:22.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.640+0000 7fe11fa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe11809ab50 0x7fe11809af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:22.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.640+0000 7fe11fa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe1180d1cc0 0x7fe11815f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:22.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.641+0000 7fe11fa00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe11809b470 con 0x7fe118162720 2024-04-16T16:14:22.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.641+0000 7fe11fa00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe11806dbf0 con 0x7fe11809ab50 2024-04-16T16:14:22.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.641+0000 7fe11fa00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe11806cfe0 con 0x7fe1180d1cc0 2024-04-16T16:14:22.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.641+0000 7fe11d000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe11809ab50 0x7fe11809af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:22.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.641+0000 7fe11d000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe11809ab50 0x7fe11809af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52280/0 (socket says 172.21.15.45:52280) 2024-04-16T16:14:22.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.641+0000 7fe11d000640 1 -- 172.21.15.45:0/2595054301 learned_addr learned my addr 172.21.15.45:0/2595054301 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:22.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.641+0000 7fe11d000640 1 -- 172.21.15.45:0/2595054301 >> v2:172.21.15.103:3300/0 conn(0x7fe1180d1cc0 msgr2=0x7fe11815f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:22.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.641+0000 7fe11d000640 1 --2- 172.21.15.45:0/2595054301 >> v2:172.21.15.103:3300/0 conn(0x7fe1180d1cc0 0x7fe11815f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.641+0000 7fe11d000640 1 -- 172.21.15.45:0/2595054301 >> v1:172.21.15.103:6789/0 conn(0x7fe118162720 legacy=0x7fe11815eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:22.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.641+0000 7fe11d000640 1 -- 172.21.15.45:0/2595054301 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11806d2c0 con 0x7fe11809ab50 2024-04-16T16:14:22.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.643+0000 7fe11d000640 1 --2- 172.21.15.45:0/2595054301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe11809ab50 0x7fe11809af30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe118070300 tx=0x7fe10802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74aae0372a584d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:22.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.643+0000 7fe117400640 1 -- 172.21.15.45:0/2595054301 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10802e070 con 0x7fe11809ab50 2024-04-16T16:14:22.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.643+0000 7fe117400640 1 -- 172.21.15.45:0/2595054301 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe108037070 con 0x7fe11809ab50 2024-04-16T16:14:22.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.643+0000 7fe117400640 1 -- 172.21.15.45:0/2595054301 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe108031a00 con 0x7fe11809ab50 2024-04-16T16:14:22.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.644+0000 7fe11fa00640 1 -- 172.21.15.45:0/2595054301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe11809ab50 msgr2=0x7fe11809af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:22.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.644+0000 7fe11fa00640 1 --2- 172.21.15.45:0/2595054301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe11809ab50 0x7fe11809af30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe118070300 tx=0x7fe10802b880 comp rx=0 tx=0).stop 2024-04-16T16:14:22.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.644+0000 7fe11fa00640 1 -- 172.21.15.45:0/2595054301 shutdown_connections 2024-04-16T16:14:22.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.644+0000 7fe11fa00640 1 --2- 172.21.15.45:0/2595054301 >> v2:172.21.15.103:3300/0 conn(0x7fe1180d1cc0 0x7fe11815f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.644+0000 7fe11fa00640 1 --2- 172.21.15.45:0/2595054301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe11809ab50 0x7fe11809af30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.644+0000 7fe11fa00640 1 -- 172.21.15.45:0/2595054301 >> 172.21.15.45:0/2595054301 conn(0x7fe1180d2890 msgr2=0x7fe1180d2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:22.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.644+0000 7fe11fa00640 1 -- 172.21.15.45:0/2595054301 shutdown_connections 2024-04-16T16:14:22.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.644+0000 7fe11fa00640 1 -- 172.21.15.45:0/2595054301 wait complete. 2024-04-16T16:14:22.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.646+0000 7fe11fa00640 1 Processor -- start 2024-04-16T16:14:22.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.646+0000 7fe11fa00640 1 -- start start 2024-04-16T16:14:22.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.647+0000 7fe11fa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe11809ab50 0x7fe1180c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:22.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.647+0000 7fe11fa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1180d1cc0 0x7fe11814e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:22.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.647+0000 7fe11fa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe118162720 0x7fe11814ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:22.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.647+0000 7fe11fa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe11809b8b0 con 0x7fe118162720 2024-04-16T16:14:22.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.647+0000 7fe11fa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe11806d880 con 0x7fe11809ab50 2024-04-16T16:14:22.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.647+0000 7fe11fa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe11806ca40 con 0x7fe1180d1cc0 2024-04-16T16:14:22.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.647+0000 7fe11da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe118162720 0x7fe11814ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:22.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.647+0000 7fe11da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe118162720 0x7fe11814ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52284/0 (socket says 172.21.15.45:52284) 2024-04-16T16:14:22.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.647+0000 7fe11da00640 1 -- 172.21.15.45:0/319904593 learned_addr learned my addr 172.21.15.45:0/319904593 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:22.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.647+0000 7fe11d000640 1 --2- 172.21.15.45:0/319904593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe11809ab50 0x7fe1180c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:22.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.647+0000 7fe11da00640 1 -- 172.21.15.45:0/319904593 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1180d1cc0 msgr2=0x7fe11814e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:22.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.647+0000 7fe117e00640 1 --2- 172.21.15.45:0/319904593 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1180d1cc0 0x7fe11814e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:22.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.647+0000 7fe11da00640 1 --2- 172.21.15.45:0/319904593 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1180d1cc0 0x7fe11814e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.647+0000 7fe11da00640 1 -- 172.21.15.45:0/319904593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe11809ab50 msgr2=0x7fe1180c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:22.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.648+0000 7fe11da00640 1 --2- 172.21.15.45:0/319904593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe11809ab50 0x7fe1180c80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.648+0000 7fe11da00640 1 -- 172.21.15.45:0/319904593 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1180c7220 con 0x7fe118162720 2024-04-16T16:14:22.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.648+0000 7fe11d000640 1 --2- 172.21.15.45:0/319904593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe11809ab50 0x7fe1180c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:22.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.648+0000 7fe11da00640 1 --2- 172.21.15.45:0/319904593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe118162720 0x7fe11814ed90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe11000b970 tx=0x7fe11000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:22.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.648+0000 7fe114c00640 1 -- 172.21.15.45:0/319904593 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe110019070 con 0x7fe118162720 2024-04-16T16:14:22.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.648+0000 7fe114c00640 1 -- 172.21.15.45:0/319904593 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe11000cb60 con 0x7fe118162720 2024-04-16T16:14:22.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.648+0000 7fe114c00640 1 -- 172.21.15.45:0/319904593 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe110014440 con 0x7fe118162720 2024-04-16T16:14:22.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.648+0000 7fe11fa00640 1 -- 172.21.15.45:0/319904593 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1180c7500 con 0x7fe118162720 2024-04-16T16:14:22.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.648+0000 7fe11fa00640 1 -- 172.21.15.45:0/319904593 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1180b5b40 con 0x7fe118162720 2024-04-16T16:14:22.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.650+0000 7fe11fa00640 1 -- 172.21.15.45:0/319904593 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1180b5d40 con 0x7fe118162720 2024-04-16T16:14:22.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.650+0000 7fe114c00640 1 -- 172.21.15.45:0/319904593 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe11001c070 con 0x7fe118162720 2024-04-16T16:14:22.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.651+0000 7fe114c00640 1 --2- 172.21.15.45:0/319904593 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe0f406c8f0 0x7fe0f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:22.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.652+0000 7fe114c00640 1 -- 172.21.15.45:0/319904593 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe11008b920 con 0x7fe118162720 2024-04-16T16:14:22.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.652+0000 7fe11d000640 1 --2- 172.21.15.45:0/319904593 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe0f406c8f0 0x7fe0f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:22.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.653+0000 7fe11d000640 1 --2- 172.21.15.45:0/319904593 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe0f406c8f0 0x7fe0f406edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe108036000 tx=0x7fe108008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:22.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.656+0000 7fe114c00640 1 -- 172.21.15.45:0/319904593 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe110059cc0 con 0x7fe118162720 2024-04-16T16:14:22.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.873+0000 7fe11fa00640 1 -- 172.21.15.45:0/319904593 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe118075780 con 0x7fe118162720 2024-04-16T16:14:22.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.875+0000 7fe114c00640 1 -- 172.21.15.45:0/319904593 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16799 (secure 0 0 0) 0x7fe1100595c0 con 0x7fe118162720 2024-04-16T16:14:22.874 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:22.874 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":23,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:14:12.851304+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:22.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.877+0000 7fe11fa00640 1 -- 172.21.15.45:0/319904593 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe0f406c8f0 msgr2=0x7fe0f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:22.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.877+0000 7fe11fa00640 1 --2- 172.21.15.45:0/319904593 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe0f406c8f0 0x7fe0f406edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe108036000 tx=0x7fe108008030 comp rx=0 tx=0).stop 2024-04-16T16:14:22.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.877+0000 7fe11fa00640 1 -- 172.21.15.45:0/319904593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe118162720 msgr2=0x7fe11814ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:22.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.877+0000 7fe11fa00640 1 --2- 172.21.15.45:0/319904593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe118162720 0x7fe11814ed90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe11000b970 tx=0x7fe11000be30 comp rx=0 tx=0).stop 2024-04-16T16:14:22.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.878+0000 7fe11fa00640 1 -- 172.21.15.45:0/319904593 shutdown_connections 2024-04-16T16:14:22.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.878+0000 7fe11fa00640 1 --2- 172.21.15.45:0/319904593 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe0f406c8f0 0x7fe0f406edb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.878+0000 7fe11fa00640 1 --2- 172.21.15.45:0/319904593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe118162720 0x7fe11814ed90 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.878+0000 7fe11fa00640 1 --2- 172.21.15.45:0/319904593 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1180d1cc0 0x7fe11814e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.878+0000 7fe11fa00640 1 --2- 172.21.15.45:0/319904593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe11809ab50 0x7fe1180c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:22.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.878+0000 7fe11fa00640 1 -- 172.21.15.45:0/319904593 >> 172.21.15.45:0/319904593 conn(0x7fe1180d2890 msgr2=0x7fe1180cf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:22.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.878+0000 7fe11fa00640 1 -- 172.21.15.45:0/319904593 shutdown_connections 2024-04-16T16:14:22.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:22.878+0000 7fe11fa00640 1 -- 172.21.15.45:0/319904593 wait complete. 2024-04-16T16:14:22.897 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:13:23.600104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:13:24.368141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:14:22.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:14:23.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.118+0000 7fc1aca00640 1 Processor -- start 2024-04-16T16:14:23.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.118+0000 7fc1aca00640 1 -- start start 2024-04-16T16:14:23.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.120+0000 7fc1aca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1a80d4370 0x7fc1a80cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:23.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.120+0000 7fc1aca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1a80cfa70 0x7fc1a8162560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:23.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.120+0000 7fc1aca00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a815ee50 con 0x7fc1a815ea70 2024-04-16T16:14:23.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.120+0000 7fc1aca00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc1a80d1a10 con 0x7fc1a80d4370 2024-04-16T16:14:23.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.120+0000 7fc1aca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc1a80b6480 con 0x7fc1a80cfa70 2024-04-16T16:14:23.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.121+0000 7fc19de00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1a80d4370 0x7fc1a80cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:23.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.121+0000 7fc19de00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1a80d4370 0x7fc1a80cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52288/0 (socket says 172.21.15.45:52288) 2024-04-16T16:14:23.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.121+0000 7fc19de00640 1 -- 172.21.15.45:0/1470822184 learned_addr learned my addr 172.21.15.45:0/1470822184 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:23.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.121+0000 7fc1a6800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc1a815ea70 0x7fc1a80d11a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43032/0 (socket says 172.21.15.45:43032) 2024-04-16T16:14:23.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.121+0000 7fc1a7200640 1 --2- 172.21.15.45:0/1470822184 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1a80cfa70 0x7fc1a8162560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:23.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.121+0000 7fc19de00640 1 -- 172.21.15.45:0/1470822184 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1a80cfa70 msgr2=0x7fc1a8162560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:23.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.121+0000 7fc19de00640 1 --2- 172.21.15.45:0/1470822184 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1a80cfa70 0x7fc1a8162560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.121+0000 7fc19de00640 1 -- 172.21.15.45:0/1470822184 >> v1:172.21.15.45:6789/0 conn(0x7fc1a815ea70 legacy=0x7fc1a80d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:23.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.121+0000 7fc19de00640 1 -- 172.21.15.45:0/1470822184 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a80b6760 con 0x7fc1a80d4370 2024-04-16T16:14:23.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.121+0000 7fc1a7200640 1 --2- 172.21.15.45:0/1470822184 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1a80cfa70 0x7fc1a8162560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:23.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.122+0000 7fc19de00640 1 --2- 172.21.15.45:0/1470822184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1a80d4370 0x7fc1a80cf530 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc190009870 tx=0x7fc19002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc46a5de1350a2ba server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:23.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.123+0000 7fc1a5e00640 1 -- 172.21.15.45:0/1470822184 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc190034070 con 0x7fc1a80d4370 2024-04-16T16:14:23.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.123+0000 7fc1a5e00640 1 -- 172.21.15.45:0/1470822184 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc190030840 con 0x7fc1a80d4370 2024-04-16T16:14:23.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.123+0000 7fc1a5e00640 1 -- 172.21.15.45:0/1470822184 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc190030c10 con 0x7fc1a80d4370 2024-04-16T16:14:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.123+0000 7fc1aca00640 1 -- 172.21.15.45:0/1470822184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1a80d4370 msgr2=0x7fc1a80cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.123+0000 7fc1aca00640 1 --2- 172.21.15.45:0/1470822184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1a80d4370 0x7fc1a80cf530 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc190009870 tx=0x7fc19002b980 comp rx=0 tx=0).stop 2024-04-16T16:14:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.124+0000 7fc1aca00640 1 -- 172.21.15.45:0/1470822184 shutdown_connections 2024-04-16T16:14:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.124+0000 7fc1aca00640 1 --2- 172.21.15.45:0/1470822184 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1a80cfa70 0x7fc1a8162560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.124+0000 7fc1aca00640 1 --2- 172.21.15.45:0/1470822184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1a80d4370 0x7fc1a80cf530 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.124+0000 7fc1aca00640 1 -- 172.21.15.45:0/1470822184 >> 172.21.15.45:0/1470822184 conn(0x7fc1a806d3b0 msgr2=0x7fc1a8073060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.124+0000 7fc1aca00640 1 -- 172.21.15.45:0/1470822184 shutdown_connections 2024-04-16T16:14:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.124+0000 7fc1aca00640 1 -- 172.21.15.45:0/1470822184 wait complete. 2024-04-16T16:14:23.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.125+0000 7fc1aca00640 1 Processor -- start 2024-04-16T16:14:23.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.126+0000 7fc1aca00640 1 -- start start 2024-04-16T16:14:23.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1aca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a80cfa70 0x7fc1a80c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:23.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1aca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1a80d4370 0x7fc1a8160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:23.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1aca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1a815ea70 0x7fc1a80b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:23.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1aca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a80b6f60 con 0x7fc1a815ea70 2024-04-16T16:14:23.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1aca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a8162aa0 con 0x7fc1a80cfa70 2024-04-16T16:14:23.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1aca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc1a8160990 con 0x7fc1a80d4370 2024-04-16T16:14:23.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1a7200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1a815ea70 0x7fc1a80b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:23.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1a7200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1a815ea70 0x7fc1a80b27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52300/0 (socket says 172.21.15.45:52300) 2024-04-16T16:14:23.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1a7200640 1 -- 172.21.15.45:0/3601049251 learned_addr learned my addr 172.21.15.45:0/3601049251 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc19de00640 1 --2- 172.21.15.45:0/3601049251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1a80d4370 0x7fc1a8160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1a6800640 1 --2- 172.21.15.45:0/3601049251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a80cfa70 0x7fc1a80c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1a7200640 1 -- 172.21.15.45:0/3601049251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1a80d4370 msgr2=0x7fc1a8160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1a7200640 1 --2- 172.21.15.45:0/3601049251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1a80d4370 0x7fc1a8160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1a7200640 1 -- 172.21.15.45:0/3601049251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a80cfa70 msgr2=0x7fc1a80c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1a7200640 1 --2- 172.21.15.45:0/3601049251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a80cfa70 0x7fc1a80c9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1a7200640 1 -- 172.21.15.45:0/3601049251 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a80c9da0 con 0x7fc1a815ea70 2024-04-16T16:14:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.127+0000 7fc1a6800640 1 --2- 172.21.15.45:0/3601049251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a80cfa70 0x7fc1a80c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.128+0000 7fc1a7200640 1 --2- 172.21.15.45:0/3601049251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1a815ea70 0x7fc1a80b27d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000cef0 tx=0x7fc1a000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.128+0000 7fc19f400640 1 -- 172.21.15.45:0/3601049251 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a001a070 con 0x7fc1a815ea70 2024-04-16T16:14:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.128+0000 7fc19f400640 1 -- 172.21.15.45:0/3601049251 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1a000d860 con 0x7fc1a815ea70 2024-04-16T16:14:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.128+0000 7fc19f400640 1 -- 172.21.15.45:0/3601049251 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a000dc50 con 0x7fc1a815ea70 2024-04-16T16:14:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.128+0000 7fc1aca00640 1 -- 172.21.15.45:0/3601049251 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1a80ca080 con 0x7fc1a815ea70 2024-04-16T16:14:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.128+0000 7fc1aca00640 1 -- 172.21.15.45:0/3601049251 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1a80ca640 con 0x7fc1a815ea70 2024-04-16T16:14:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.129+0000 7fc1aca00640 1 -- 172.21.15.45:0/3601049251 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1a8075780 con 0x7fc1a815ea70 2024-04-16T16:14:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.130+0000 7fc19f400640 1 -- 172.21.15.45:0/3601049251 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc1a001d070 con 0x7fc1a815ea70 2024-04-16T16:14:23.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.131+0000 7fc19f400640 1 --2- 172.21.15.45:0/3601049251 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc18406c8f0 0x7fc18406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:23.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.131+0000 7fc19f400640 1 -- 172.21.15.45:0/3601049251 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc1a008c4b0 con 0x7fc1a815ea70 2024-04-16T16:14:23.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.132+0000 7fc1a6800640 1 --2- 172.21.15.45:0/3601049251 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc18406c8f0 0x7fc18406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:23.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.133+0000 7fc1a6800640 1 --2- 172.21.15.45:0/3601049251 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc18406c8f0 0x7fc18406edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc198000f60 tx=0x7fc198009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:23.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.136+0000 7fc19f400640 1 -- 172.21.15.45:0/3601049251 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc1a005a910 con 0x7fc1a815ea70 2024-04-16T16:14:23.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:14:23.203 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:14:23.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.343+0000 7fc1aca00640 1 -- 172.21.15.45:0/3601049251 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc1a80d2500 con 0x7fc1a815ea70 2024-04-16T16:14:23.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.344+0000 7fc19f400640 1 -- 172.21.15.45:0/3601049251 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7fc1a005a210 con 0x7fc1a815ea70 2024-04-16T16:14:23.343 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:14:23.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.347+0000 7fc1aca00640 1 -- 172.21.15.45:0/3601049251 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc18406c8f0 msgr2=0x7fc18406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:23.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.347+0000 7fc1aca00640 1 --2- 172.21.15.45:0/3601049251 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc18406c8f0 0x7fc18406edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc198000f60 tx=0x7fc198009040 comp rx=0 tx=0).stop 2024-04-16T16:14:23.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.347+0000 7fc1aca00640 1 -- 172.21.15.45:0/3601049251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1a815ea70 msgr2=0x7fc1a80b27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:23.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.347+0000 7fc1aca00640 1 --2- 172.21.15.45:0/3601049251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1a815ea70 0x7fc1a80b27d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000cef0 tx=0x7fc1a000a710 comp rx=0 tx=0).stop 2024-04-16T16:14:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.347+0000 7fc1aca00640 1 -- 172.21.15.45:0/3601049251 shutdown_connections 2024-04-16T16:14:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.347+0000 7fc1aca00640 1 --2- 172.21.15.45:0/3601049251 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc18406c8f0 0x7fc18406edb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.347+0000 7fc1aca00640 1 --2- 172.21.15.45:0/3601049251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1a815ea70 0x7fc1a80b27d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.347+0000 7fc1aca00640 1 --2- 172.21.15.45:0/3601049251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1a80d4370 0x7fc1a8160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.347+0000 7fc1aca00640 1 --2- 172.21.15.45:0/3601049251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a80cfa70 0x7fc1a80c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.347+0000 7fc1aca00640 1 -- 172.21.15.45:0/3601049251 >> 172.21.15.45:0/3601049251 conn(0x7fc1a806d3b0 msgr2=0x7fc1a815dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.347+0000 7fc1aca00640 1 -- 172.21.15.45:0/3601049251 shutdown_connections 2024-04-16T16:14:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.347+0000 7fc1aca00640 1 -- 172.21.15.45:0/3601049251 wait complete. 2024-04-16T16:14:23.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:14:23.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.610+0000 7fd50fe00640 1 Processor -- start 2024-04-16T16:14:23.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.610+0000 7fd50fe00640 1 -- start start 2024-04-16T16:14:23.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.612+0000 7fd50fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5100d3e90 0x7fd510169930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:23.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.612+0000 7fd50fe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5100b6810 con 0x7fd5100b6310 2024-04-16T16:14:23.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.612+0000 7fd50fe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5100b6af0 con 0x7fd5100d18f0 2024-04-16T16:14:23.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.612+0000 7fd50fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd5100d29f0 con 0x7fd5100d3e90 2024-04-16T16:14:23.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.612+0000 7fd507e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd5100b6310 0x7fd510161db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45328/0 (socket says 172.21.15.45:45328) 2024-04-16T16:14:23.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.612+0000 7fd507e00640 1 -- 172.21.15.45:0/3140506344 learned_addr learned my addr 172.21.15.45:0/3140506344 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.612+0000 7fd50f400640 1 --2- 172.21.15.45:0/3140506344 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5100d3e90 0x7fd510169930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50e000640 1 -- 172.21.15.45:0/3140506344 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2836982082 0 0) 0x7fd5100b6810 con 0x7fd5100b6310 2024-04-16T16:14:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50e000640 1 -- 172.21.15.45:0/3140506344 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4f4003520 con 0x7fd5100b6310 2024-04-16T16:14:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50e000640 1 -- 172.21.15.45:0/3140506344 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3273184252 0 0) 0x7fd4f4003520 con 0x7fd5100b6310 2024-04-16T16:14:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50e000640 1 -- 172.21.15.45:0/3140506344 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5100d3e90 msgr2=0x7fd510169930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50e000640 1 --2- 172.21.15.45:0/3140506344 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5100d3e90 0x7fd510169930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50e000640 1 -- 172.21.15.45:0/3140506344 >> v1:172.21.15.103:6789/0 conn(0x7fd5100d18f0 legacy=0x7fd5100ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50e000640 1 -- 172.21.15.45:0/3140506344 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5100b6dd0 con 0x7fd5100b6310 2024-04-16T16:14:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50e000640 1 -- 172.21.15.45:0/3140506344 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fd4f8001670 con 0x7fd5100b6310 2024-04-16T16:14:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50e000640 1 -- 172.21.15.45:0/3140506344 >> v1:172.21.15.45:6789/0 conn(0x7fd5100b6310 legacy=0x7fd510161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50e000640 1 --2- 172.21.15.45:0/3140506344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039e0 0x7fd4f4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50e000640 1 --2- 172.21.15.45:0/3140506344 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5100d3e90 0x7fd4f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50e000640 1 --2- 172.21.15.45:0/3140506344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f4004d40 0x7fd4f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50e000640 1 -- 172.21.15.45:0/3140506344 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd5100d29f0 con 0x7fd4f40039e0 2024-04-16T16:14:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50e000640 1 -- 172.21.15.45:0/3140506344 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f400baf0 con 0x7fd4f4004d40 2024-04-16T16:14:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50e000640 1 -- 172.21.15.45:0/3140506344 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd4f400bcf0 con 0x7fd5100d3e90 2024-04-16T16:14:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.613+0000 7fd50ea00640 1 --2- 172.21.15.45:0/3140506344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039e0 0x7fd4f4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.614+0000 7fd507e00640 1 --2- 172.21.15.45:0/3140506344 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5100d3e90 0x7fd4f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.614+0000 7fd50ea00640 1 -- 172.21.15.45:0/3140506344 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5100d3e90 msgr2=0x7fd4f4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.614+0000 7fd50ea00640 1 --2- 172.21.15.45:0/3140506344 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5100d3e90 0x7fd4f4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.614+0000 7fd50ea00640 1 -- 172.21.15.45:0/3140506344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f4004d40 msgr2=0x7fd4f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.614+0000 7fd50ea00640 1 --2- 172.21.15.45:0/3140506344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f4004d40 0x7fd4f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.614+0000 7fd50ea00640 1 -- 172.21.15.45:0/3140506344 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f4003520 con 0x7fd4f40039e0 2024-04-16T16:14:23.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.614+0000 7fd507e00640 1 --2- 172.21.15.45:0/3140506344 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5100d3e90 0x7fd4f4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:23.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.615+0000 7fd50ea00640 1 --2- 172.21.15.45:0/3140506344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039e0 0x7fd4f4003e00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd500001750 tx=0x7fd4f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd2c425654e4eb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:23.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.615+0000 7fd50e000640 1 -- 172.21.15.45:0/3140506344 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd500001370 con 0x7fd4f40039e0 2024-04-16T16:14:23.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.615+0000 7fd50e000640 1 -- 172.21.15.45:0/3140506344 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5000052d0 con 0x7fd4f40039e0 2024-04-16T16:14:23.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.615+0000 7fd50e000640 1 -- 172.21.15.45:0/3140506344 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd500003830 con 0x7fd4f40039e0 2024-04-16T16:14:23.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.616+0000 7fd50fe00640 1 -- 172.21.15.45:0/3140506344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039e0 msgr2=0x7fd4f4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:23.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.616+0000 7fd50fe00640 1 --2- 172.21.15.45:0/3140506344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039e0 0x7fd4f4003e00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd500001750 tx=0x7fd4f4005830 comp rx=0 tx=0).stop 2024-04-16T16:14:23.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.616+0000 7fd50f400640 1 -- 172.21.15.45:0/3140506344 reap_dead start 2024-04-16T16:14:23.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.616+0000 7fd50fe00640 1 -- 172.21.15.45:0/3140506344 shutdown_connections 2024-04-16T16:14:23.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.616+0000 7fd50fe00640 1 -- 172.21.15.45:0/3140506344 >> 172.21.15.45:0/3140506344 conn(0x7fd51006cf90 msgr2=0x7fd51006d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:23.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.616+0000 7fd50fe00640 1 -- 172.21.15.45:0/3140506344 shutdown_connections 2024-04-16T16:14:23.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.616+0000 7fd50fe00640 1 -- 172.21.15.45:0/3140506344 wait complete. 2024-04-16T16:14:23.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.619+0000 7fd50fe00640 1 Processor -- start 2024-04-16T16:14:23.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.619+0000 7fd50fe00640 1 -- start start 2024-04-16T16:14:23.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.620+0000 7fd50fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4f40039e0 0x7fd5101e6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:23.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.620+0000 7fd50fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd510153d90 0x7fd51015b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:23.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.620+0000 7fd50fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5100b7d90 0x7fd5100b8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:23.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.620+0000 7fd50fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd5100d1e50 con 0x7fd5100b7d90 2024-04-16T16:14:23.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.620+0000 7fd50fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd5100d1cd0 con 0x7fd510153d90 2024-04-16T16:14:23.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.620+0000 7fd50fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd5100d1fd0 con 0x7fd4f40039e0 2024-04-16T16:14:23.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.621+0000 7fd50f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5100b7d90 0x7fd5100b8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:23.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.621+0000 7fd50f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5100b7d90 0x7fd5100b8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53222/0 (socket says 172.21.15.45:53222) 2024-04-16T16:14:23.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.621+0000 7fd50f400640 1 -- 172.21.15.45:0/4061416794 learned_addr learned my addr 172.21.15.45:0/4061416794 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:23.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.621+0000 7fd50ea00640 1 --2- 172.21.15.45:0/4061416794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4f40039e0 0x7fd5101e6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:23.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.621+0000 7fd507e00640 1 --2- 172.21.15.45:0/4061416794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd510153d90 0x7fd51015b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:23.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.621+0000 7fd50f400640 1 -- 172.21.15.45:0/4061416794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4f40039e0 msgr2=0x7fd5101e6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:23.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.621+0000 7fd50f400640 1 --2- 172.21.15.45:0/4061416794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4f40039e0 0x7fd5101e6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.621+0000 7fd50f400640 1 -- 172.21.15.45:0/4061416794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd510153d90 msgr2=0x7fd51015b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:23.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.621+0000 7fd50f400640 1 --2- 172.21.15.45:0/4061416794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd510153d90 0x7fd51015b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.621+0000 7fd50f400640 1 -- 172.21.15.45:0/4061416794 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5101e7200 con 0x7fd5100b7d90 2024-04-16T16:14:23.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.621+0000 7fd50ea00640 1 --2- 172.21.15.45:0/4061416794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4f40039e0 0x7fd5101e6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:23.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.621+0000 7fd50f400640 1 --2- 172.21.15.45:0/4061416794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5100b7d90 0x7fd5100b8210 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd5100baa00 tx=0x7fd50800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:23.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.622+0000 7fd506a00640 1 -- 172.21.15.45:0/4061416794 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50800bae0 con 0x7fd5100b7d90 2024-04-16T16:14:23.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.622+0000 7fd506a00640 1 -- 172.21.15.45:0/4061416794 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd50801a070 con 0x7fd5100b7d90 2024-04-16T16:14:23.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.622+0000 7fd506a00640 1 -- 172.21.15.45:0/4061416794 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd508014400 con 0x7fd5100b7d90 2024-04-16T16:14:23.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.622+0000 7fd50fe00640 1 -- 172.21.15.45:0/4061416794 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5101e74e0 con 0x7fd5100b7d90 2024-04-16T16:14:23.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.622+0000 7fd50fe00640 1 -- 172.21.15.45:0/4061416794 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5100ccaf0 con 0x7fd5100b7d90 2024-04-16T16:14:23.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.623+0000 7fd50fe00640 1 -- 172.21.15.45:0/4061416794 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5100cccf0 con 0x7fd5100b7d90 2024-04-16T16:14:23.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.624+0000 7fd506a00640 1 -- 172.21.15.45:0/4061416794 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd508013070 con 0x7fd5100b7d90 2024-04-16T16:14:23.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.625+0000 7fd506a00640 1 --2- 172.21.15.45:0/4061416794 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd4ec06c8f0 0x7fd4ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:23.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.625+0000 7fd506a00640 1 -- 172.21.15.45:0/4061416794 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd50808bd50 con 0x7fd5100b7d90 2024-04-16T16:14:23.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.626+0000 7fd50ea00640 1 --2- 172.21.15.45:0/4061416794 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd4ec06c8f0 0x7fd4ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:23.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.627+0000 7fd50ea00640 1 --2- 172.21.15.45:0/4061416794 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd4ec06c8f0 0x7fd4ec06edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd500033000 tx=0x7fd500009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:23.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.630+0000 7fd506a00640 1 -- 172.21.15.45:0/4061416794 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd5100cccf0 con 0x7fd5100b7d90 2024-04-16T16:14:23.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.826+0000 7fd50fe00640 1 -- 172.21.15.45:0/4061416794 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd510075780 con 0x7fd5100b7d90 2024-04-16T16:14:23.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.826+0000 7fd506a00640 1 -- 172.21.15.45:0/4061416794 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7fd50805a060 con 0x7fd5100b7d90 2024-04-16T16:14:23.826 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:14:23.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.829+0000 7fd50fe00640 1 -- 172.21.15.45:0/4061416794 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd4ec06c8f0 msgr2=0x7fd4ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:23.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.829+0000 7fd50fe00640 1 --2- 172.21.15.45:0/4061416794 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd4ec06c8f0 0x7fd4ec06edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd500033000 tx=0x7fd500009030 comp rx=0 tx=0).stop 2024-04-16T16:14:23.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.829+0000 7fd50fe00640 1 -- 172.21.15.45:0/4061416794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5100b7d90 msgr2=0x7fd5100b8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:23.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.829+0000 7fd50fe00640 1 --2- 172.21.15.45:0/4061416794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5100b7d90 0x7fd5100b8210 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd5100baa00 tx=0x7fd50800b480 comp rx=0 tx=0).stop 2024-04-16T16:14:23.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.829+0000 7fd50fe00640 1 -- 172.21.15.45:0/4061416794 shutdown_connections 2024-04-16T16:14:23.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.829+0000 7fd50fe00640 1 --2- 172.21.15.45:0/4061416794 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd4ec06c8f0 0x7fd4ec06edb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.829+0000 7fd50fe00640 1 --2- 172.21.15.45:0/4061416794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5100b7d90 0x7fd5100b8210 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.829+0000 7fd50fe00640 1 --2- 172.21.15.45:0/4061416794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd510153d90 0x7fd51015b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.829+0000 7fd50fe00640 1 --2- 172.21.15.45:0/4061416794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4f40039e0 0x7fd5101e6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:23.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.829+0000 7fd50fe00640 1 -- 172.21.15.45:0/4061416794 >> 172.21.15.45:0/4061416794 conn(0x7fd51006cf90 msgr2=0x7fd5101580c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:23.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.829+0000 7fd50fe00640 1 -- 172.21.15.45:0/4061416794 shutdown_connections 2024-04-16T16:14:23.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:23.829+0000 7fd50fe00640 1 -- 172.21.15.45:0/4061416794 wait complete. 2024-04-16T16:14:23.845 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:14:24.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.065+0000 7fbfabe00640 1 Processor -- start 2024-04-16T16:14:24.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.066+0000 7fbfabe00640 1 -- start start 2024-04-16T16:14:24.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.067+0000 7fbfabe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbfac0b5d50 0x7fbfac06eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:24.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.068+0000 7fbfabe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfac146590 con 0x7fbfac0b58f0 2024-04-16T16:14:24.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.068+0000 7fbfabe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfac0c5750 con 0x7fbfac1461b0 2024-04-16T16:14:24.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.068+0000 7fbfabe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbfac06dfd0 con 0x7fbfac0b5d50 2024-04-16T16:14:24.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.068+0000 7fbfaa000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbfac0b58f0 0x7fbfac06e460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45330/0 (socket says 172.21.15.45:45330) 2024-04-16T16:14:24.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.068+0000 7fbfaa000640 1 -- 172.21.15.45:0/3887520640 learned_addr learned my addr 172.21.15.45:0/3887520640 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:24.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.068+0000 7fbfab400640 1 --2- 172.21.15.45:0/3887520640 >> v2:172.21.15.103:3300/0 conn(0x7fbfac0b5d50 0x7fbfac06eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.068+0000 7fbfa9600640 1 -- 172.21.15.45:0/3887520640 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3625388804 0 0) 0x7fbfac146590 con 0x7fbfac0b58f0 2024-04-16T16:14:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.069+0000 7fbfa9600640 1 -- 172.21.15.45:0/3887520640 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf90003520 con 0x7fbfac0b58f0 2024-04-16T16:14:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.069+0000 7fbfa9600640 1 -- 172.21.15.45:0/3887520640 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 29013542 0 0) 0x7fbf90003520 con 0x7fbfac0b58f0 2024-04-16T16:14:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.069+0000 7fbfa9600640 1 -- 172.21.15.45:0/3887520640 >> v2:172.21.15.103:3300/0 conn(0x7fbfac0b5d50 msgr2=0x7fbfac06eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.069+0000 7fbfa9600640 1 --2- 172.21.15.45:0/3887520640 >> v2:172.21.15.103:3300/0 conn(0x7fbfac0b5d50 0x7fbfac06eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.069+0000 7fbfa9600640 1 -- 172.21.15.45:0/3887520640 >> v1:172.21.15.103:6789/0 conn(0x7fbfac1461b0 legacy=0x7fbfac160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.069+0000 7fbfa9600640 1 -- 172.21.15.45:0/3887520640 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfac15ccb0 con 0x7fbfac0b58f0 2024-04-16T16:14:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.069+0000 7fbfa9600640 1 -- 172.21.15.45:0/3887520640 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fbf98001670 con 0x7fbfac0b58f0 2024-04-16T16:14:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.069+0000 7fbfa9600640 1 -- 172.21.15.45:0/3887520640 >> v1:172.21.15.45:6789/0 conn(0x7fbfac0b58f0 legacy=0x7fbfac06e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.069+0000 7fbfa9600640 1 --2- 172.21.15.45:0/3887520640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf90003a00 0x7fbf90003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.069+0000 7fbfa9600640 1 --2- 172.21.15.45:0/3887520640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf90004800 0x7fbf90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.069+0000 7fbfa9600640 1 --2- 172.21.15.45:0/3887520640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf900051c0 0x7fbf9000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.069+0000 7fbfa9600640 1 -- 172.21.15.45:0/3887520640 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9000c090 con 0x7fbf900051c0 2024-04-16T16:14:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.069+0000 7fbfa9600640 1 -- 172.21.15.45:0/3887520640 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9000c240 con 0x7fbf90003a00 2024-04-16T16:14:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.069+0000 7fbfa9600640 1 -- 172.21.15.45:0/3887520640 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbf9000c440 con 0x7fbf90004800 2024-04-16T16:14:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.069+0000 7fbfab400640 1 --2- 172.21.15.45:0/3887520640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf900051c0 0x7fbf9000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.070+0000 7fbfaa000640 1 --2- 172.21.15.45:0/3887520640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf90004800 0x7fbf90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.070+0000 7fbfab400640 1 -- 172.21.15.45:0/3887520640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf90004800 msgr2=0x7fbf90004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.070+0000 7fbfab400640 1 --2- 172.21.15.45:0/3887520640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf90004800 0x7fbf90004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.070+0000 7fbfaaa00640 1 --2- 172.21.15.45:0/3887520640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf90003a00 0x7fbf90003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.070+0000 7fbfab400640 1 -- 172.21.15.45:0/3887520640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf90003a00 msgr2=0x7fbf90003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:24.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.070+0000 7fbfab400640 1 --2- 172.21.15.45:0/3887520640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf90003a00 0x7fbf90003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.070+0000 7fbfab400640 1 -- 172.21.15.45:0/3887520640 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf90003520 con 0x7fbf900051c0 2024-04-16T16:14:24.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.070+0000 7fbfaaa00640 1 --2- 172.21.15.45:0/3887520640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf90003a00 0x7fbf90003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:24.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.071+0000 7fbfab400640 1 -- 172.21.15.45:0/3887520640 reap_dead start 2024-04-16T16:14:24.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.071+0000 7fbfab400640 1 --2- 172.21.15.45:0/3887520640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf900051c0 0x7fbf9000b9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4009fc0 tx=0x7fbf90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a674dfd1ee2699 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:24.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.071+0000 7fbfa9600640 1 -- 172.21.15.45:0/3887520640 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4009c10 con 0x7fbf900051c0 2024-04-16T16:14:24.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.071+0000 7fbfa9600640 1 -- 172.21.15.45:0/3887520640 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfa402cca0 con 0x7fbf900051c0 2024-04-16T16:14:24.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.071+0000 7fbfa9600640 1 -- 172.21.15.45:0/3887520640 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4003bd0 con 0x7fbf900051c0 2024-04-16T16:14:24.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.072+0000 7fbfabe00640 1 -- 172.21.15.45:0/3887520640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf900051c0 msgr2=0x7fbf9000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:24.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.072+0000 7fbfabe00640 1 --2- 172.21.15.45:0/3887520640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf900051c0 0x7fbf9000b9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4009fc0 tx=0x7fbf90005830 comp rx=0 tx=0).stop 2024-04-16T16:14:24.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.072+0000 7fbfabe00640 1 -- 172.21.15.45:0/3887520640 shutdown_connections 2024-04-16T16:14:24.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.072+0000 7fbfabe00640 1 --2- 172.21.15.45:0/3887520640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf900051c0 0x7fbf9000b9c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.072+0000 7fbfabe00640 1 -- 172.21.15.45:0/3887520640 >> 172.21.15.45:0/3887520640 conn(0x7fbfac06b930 msgr2=0x7fbfac06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:24.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.072+0000 7fbfabe00640 1 -- 172.21.15.45:0/3887520640 shutdown_connections 2024-04-16T16:14:24.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.072+0000 7fbfabe00640 1 -- 172.21.15.45:0/3887520640 wait complete. 2024-04-16T16:14:24.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.074+0000 7fbfabe00640 1 Processor -- start 2024-04-16T16:14:24.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.074+0000 7fbfabe00640 1 -- start start 2024-04-16T16:14:24.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.075+0000 7fbfabe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf900051c0 0x7fbfac144c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:24.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.075+0000 7fbfabe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfac15c0d0 0x7fbfac143090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:24.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.075+0000 7fbfabe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbfac13ce60 0x7fbfac13d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:24.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.075+0000 7fbfabe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbfac0c5fb0 con 0x7fbfac15c0d0 2024-04-16T16:14:24.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.075+0000 7fbfabe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbfac0c5cd0 con 0x7fbf900051c0 2024-04-16T16:14:24.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.075+0000 7fbfabe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbfac0c6290 con 0x7fbfac13ce60 2024-04-16T16:14:24.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.075+0000 7fbfaa000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfac15c0d0 0x7fbfac143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:24.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.076+0000 7fbfaa000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfac15c0d0 0x7fbfac143090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53246/0 (socket says 172.21.15.45:53246) 2024-04-16T16:14:24.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.076+0000 7fbfaa000640 1 -- 172.21.15.45:0/1640848179 learned_addr learned my addr 172.21.15.45:0/1640848179 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:24.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.076+0000 7fbfab400640 1 --2- 172.21.15.45:0/1640848179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbfac13ce60 0x7fbfac13d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:24.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.076+0000 7fbfaa000640 1 -- 172.21.15.45:0/1640848179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbfac13ce60 msgr2=0x7fbfac13d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:24.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.076+0000 7fbfaa000640 1 --2- 172.21.15.45:0/1640848179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbfac13ce60 0x7fbfac13d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.076+0000 7fbfaa000640 1 -- 172.21.15.45:0/1640848179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf900051c0 msgr2=0x7fbfac144c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:24.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.076+0000 7fbfaaa00640 1 --2- 172.21.15.45:0/1640848179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf900051c0 0x7fbfac144c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:24.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.076+0000 7fbfaa000640 1 --2- 172.21.15.45:0/1640848179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf900051c0 0x7fbfac144c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.076+0000 7fbfaa000640 1 -- 172.21.15.45:0/1640848179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfac145820 con 0x7fbfac15c0d0 2024-04-16T16:14:24.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.076+0000 7fbfab400640 1 --2- 172.21.15.45:0/1640848179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbfac13ce60 0x7fbfac13d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:24.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.076+0000 7fbfaa000640 1 --2- 172.21.15.45:0/1640848179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfac15c0d0 0x7fbfac143090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800d7d0 tx=0x7fbf9800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:24.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.077+0000 7fbfa2a00640 1 -- 172.21.15.45:0/1640848179 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf98019070 con 0x7fbfac15c0d0 2024-04-16T16:14:24.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.077+0000 7fbfa2a00640 1 -- 172.21.15.45:0/1640848179 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf980072f0 con 0x7fbfac15c0d0 2024-04-16T16:14:24.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.077+0000 7fbfa2a00640 1 -- 172.21.15.45:0/1640848179 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf980047e0 con 0x7fbfac15c0d0 2024-04-16T16:14:24.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.077+0000 7fbfabe00640 1 -- 172.21.15.45:0/1640848179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfac13f8d0 con 0x7fbfac15c0d0 2024-04-16T16:14:24.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.077+0000 7fbfabe00640 1 -- 172.21.15.45:0/1640848179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfac145540 con 0x7fbfac15c0d0 2024-04-16T16:14:24.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.078+0000 7fbfabe00640 1 -- 172.21.15.45:0/1640848179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfac075780 con 0x7fbfac15c0d0 2024-04-16T16:14:24.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.079+0000 7fbfa2a00640 1 -- 172.21.15.45:0/1640848179 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbf9801c070 con 0x7fbfac15c0d0 2024-04-16T16:14:24.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.080+0000 7fbfa2a00640 1 --2- 172.21.15.45:0/1640848179 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbf8c06c920 0x7fbf8c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:24.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.080+0000 7fbfa2a00640 1 -- 172.21.15.45:0/1640848179 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fbf9808b440 con 0x7fbfac15c0d0 2024-04-16T16:14:24.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.081+0000 7fbfaaa00640 1 --2- 172.21.15.45:0/1640848179 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbf8c06c920 0x7fbf8c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:24.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.082+0000 7fbfaaa00640 1 --2- 172.21.15.45:0/1640848179 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbf8c06c920 0x7fbf8c06ede0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c001630 tx=0x7fbf9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:24.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.084+0000 7fbfa2a00640 1 -- 172.21.15.45:0/1640848179 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbf98059940 con 0x7fbfac15c0d0 2024-04-16T16:14:24.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.287+0000 7fbfabe00640 1 -- 172.21.15.45:0/1640848179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbfac072500 con 0x7fbfac15c0d0 2024-04-16T16:14:24.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.288+0000 7fbfa2a00640 1 -- 172.21.15.45:0/1640848179 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7fbf98059240 con 0x7fbfac15c0d0 2024-04-16T16:14:24.287 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:14:24.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.291+0000 7fbfabe00640 1 -- 172.21.15.45:0/1640848179 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbf8c06c920 msgr2=0x7fbf8c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:24.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.291+0000 7fbfabe00640 1 --2- 172.21.15.45:0/1640848179 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbf8c06c920 0x7fbf8c06ede0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c001630 tx=0x7fbf9c009040 comp rx=0 tx=0).stop 2024-04-16T16:14:24.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.291+0000 7fbfabe00640 1 -- 172.21.15.45:0/1640848179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfac15c0d0 msgr2=0x7fbfac143090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:24.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.291+0000 7fbfabe00640 1 --2- 172.21.15.45:0/1640848179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfac15c0d0 0x7fbfac143090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800d7d0 tx=0x7fbf9800dc90 comp rx=0 tx=0).stop 2024-04-16T16:14:24.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.291+0000 7fbfabe00640 1 -- 172.21.15.45:0/1640848179 shutdown_connections 2024-04-16T16:14:24.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.291+0000 7fbfabe00640 1 --2- 172.21.15.45:0/1640848179 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbf8c06c920 0x7fbf8c06ede0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.291+0000 7fbfabe00640 1 --2- 172.21.15.45:0/1640848179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbfac13ce60 0x7fbfac13d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.291+0000 7fbfabe00640 1 --2- 172.21.15.45:0/1640848179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfac15c0d0 0x7fbfac143090 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.291+0000 7fbfabe00640 1 --2- 172.21.15.45:0/1640848179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf900051c0 0x7fbfac144c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.291+0000 7fbfabe00640 1 -- 172.21.15.45:0/1640848179 >> 172.21.15.45:0/1640848179 conn(0x7fbfac06b930 msgr2=0x7fbfac15f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:24.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.291+0000 7fbfabe00640 1 -- 172.21.15.45:0/1640848179 shutdown_connections 2024-04-16T16:14:24.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.291+0000 7fbfabe00640 1 -- 172.21.15.45:0/1640848179 wait complete. 2024-04-16T16:14:24.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:14:24.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.523+0000 7f0b66a00640 1 Processor -- start 2024-04-16T16:14:24.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.523+0000 7f0b66a00640 1 -- start start 2024-04-16T16:14:24.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.525+0000 7f0b66a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0b60161e90 0x7f0b600cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:24.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.525+0000 7f0b66a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b60162270 0x7f0b600d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:24.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.525+0000 7f0b66a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b600d3080 con 0x7f0b600d52d0 2024-04-16T16:14:24.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.525+0000 7f0b66a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0b600d2e30 con 0x7f0b60161e90 2024-04-16T16:14:24.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.525+0000 7f0b66a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0b600d3360 con 0x7f0b60162270 2024-04-16T16:14:24.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.525+0000 7f0b65600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0b60161e90 0x7f0b600cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:24.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.525+0000 7f0b65600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0b60161e90 0x7f0b600cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53262/0 (socket says 172.21.15.45:53262) 2024-04-16T16:14:24.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.525+0000 7f0b65600640 1 -- 172.21.15.45:0/2352015951 learned_addr learned my addr 172.21.15.45:0/2352015951 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:24.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.526+0000 7f0b64c00640 1 --2- 172.21.15.45:0/2352015951 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b60162270 0x7f0b600d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:24.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.526+0000 7f0b65600640 1 -- 172.21.15.45:0/2352015951 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b60162270 msgr2=0x7f0b600d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:24.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.526+0000 7f0b65600640 1 --2- 172.21.15.45:0/2352015951 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b60162270 0x7f0b600d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.526+0000 7f0b65600640 1 -- 172.21.15.45:0/2352015951 >> v1:172.21.15.45:6789/0 conn(0x7f0b600d52d0 legacy=0x7f0b6006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:24.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.526+0000 7f0b65600640 1 -- 172.21.15.45:0/2352015951 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b600d3640 con 0x7f0b60161e90 2024-04-16T16:14:24.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.527+0000 7f0b65600640 1 --2- 172.21.15.45:0/2352015951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b60161e90 0x7f0b600cf7d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0b54009870 tx=0x7f0b5402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf6148c8ba8a521 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:24.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.527+0000 7f0b5fe00640 1 -- 172.21.15.45:0/2352015951 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5402f840 con 0x7f0b60161e90 2024-04-16T16:14:24.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.527+0000 7f0b5fe00640 1 -- 172.21.15.45:0/2352015951 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b5402f9e0 con 0x7f0b60161e90 2024-04-16T16:14:24.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.527+0000 7f0b5fe00640 1 -- 172.21.15.45:0/2352015951 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5402fdb0 con 0x7f0b60161e90 2024-04-16T16:14:24.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.528+0000 7f0b66a00640 1 -- 172.21.15.45:0/2352015951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b60161e90 msgr2=0x7f0b600cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:24.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.528+0000 7f0b66a00640 1 --2- 172.21.15.45:0/2352015951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b60161e90 0x7f0b600cf7d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0b54009870 tx=0x7f0b5402ba20 comp rx=0 tx=0).stop 2024-04-16T16:14:24.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.528+0000 7f0b66a00640 1 -- 172.21.15.45:0/2352015951 shutdown_connections 2024-04-16T16:14:24.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.528+0000 7f0b66a00640 1 --2- 172.21.15.45:0/2352015951 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b60162270 0x7f0b600d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.528+0000 7f0b66a00640 1 --2- 172.21.15.45:0/2352015951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b60161e90 0x7f0b600cf7d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.528+0000 7f0b66a00640 1 -- 172.21.15.45:0/2352015951 >> 172.21.15.45:0/2352015951 conn(0x7f0b600d1940 msgr2=0x7f0b600d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:24.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.528+0000 7f0b66a00640 1 -- 172.21.15.45:0/2352015951 shutdown_connections 2024-04-16T16:14:24.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.528+0000 7f0b66a00640 1 -- 172.21.15.45:0/2352015951 wait complete. 2024-04-16T16:14:24.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.530+0000 7f0b66a00640 1 Processor -- start 2024-04-16T16:14:24.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.530+0000 7f0b66a00640 1 -- start start 2024-04-16T16:14:24.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.531+0000 7f0b66a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b600d52d0 0x7f0b601590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:24.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.531+0000 7f0b66a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b60161e90 0x7f0b6014ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:24.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.531+0000 7f0b66a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b60162270 0x7f0b601f53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:24.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.531+0000 7f0b66a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6006d550 con 0x7f0b60162270 2024-04-16T16:14:24.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.531+0000 7f0b66a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0b600cfd10 con 0x7f0b60161e90 2024-04-16T16:14:24.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.531+0000 7f0b66a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0b6006d6d0 con 0x7f0b600d52d0 2024-04-16T16:14:24.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.531+0000 7f0b66000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b60162270 0x7f0b601f53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:24.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.531+0000 7f0b66000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b60162270 0x7f0b601f53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53278/0 (socket says 172.21.15.45:53278) 2024-04-16T16:14:24.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.531+0000 7f0b66000640 1 -- 172.21.15.45:0/2569241971 learned_addr learned my addr 172.21.15.45:0/2569241971 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:24.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.532+0000 7f0b65600640 1 --2- 172.21.15.45:0/2569241971 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b600d52d0 0x7f0b601590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:24.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.532+0000 7f0b66000640 1 -- 172.21.15.45:0/2569241971 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b600d52d0 msgr2=0x7f0b601590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:24.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.532+0000 7f0b66000640 1 --2- 172.21.15.45:0/2569241971 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b600d52d0 0x7f0b601590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.532+0000 7f0b66000640 1 -- 172.21.15.45:0/2569241971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b60161e90 msgr2=0x7f0b6014ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:24.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.532+0000 7f0b64c00640 1 --2- 172.21.15.45:0/2569241971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b60161e90 0x7f0b6014ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:24.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.532+0000 7f0b66000640 1 --2- 172.21.15.45:0/2569241971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b60161e90 0x7f0b6014ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.532+0000 7f0b66000640 1 -- 172.21.15.45:0/2569241971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b600cc170 con 0x7f0b60162270 2024-04-16T16:14:24.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.532+0000 7f0b66000640 1 --2- 172.21.15.45:0/2569241971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b60162270 0x7f0b601f53f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800c710 tx=0x7f0b5800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:24.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.532+0000 7f0b5d600640 1 -- 172.21.15.45:0/2569241971 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b58016020 con 0x7f0b60162270 2024-04-16T16:14:24.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.532+0000 7f0b5d600640 1 -- 172.21.15.45:0/2569241971 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b58006d50 con 0x7f0b60162270 2024-04-16T16:14:24.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.532+0000 7f0b5d600640 1 -- 172.21.15.45:0/2569241971 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b58003bb0 con 0x7f0b60162270 2024-04-16T16:14:24.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.532+0000 7f0b66a00640 1 -- 172.21.15.45:0/2569241971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b600cc450 con 0x7f0b60162270 2024-04-16T16:14:24.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.533+0000 7f0b66a00640 1 -- 172.21.15.45:0/2569241971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b600cbb70 con 0x7f0b60162270 2024-04-16T16:14:24.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.534+0000 7f0b66a00640 1 -- 172.21.15.45:0/2569241971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b600ba7a0 con 0x7f0b60162270 2024-04-16T16:14:24.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.535+0000 7f0b5d600640 1 -- 172.21.15.45:0/2569241971 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0b5800a850 con 0x7f0b60162270 2024-04-16T16:14:24.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.535+0000 7f0b5d600640 1 --2- 172.21.15.45:0/2569241971 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f0b4006c8f0 0x7f0b4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:24.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.536+0000 7f0b5d600640 1 -- 172.21.15.45:0/2569241971 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0b5808b8b0 con 0x7f0b60162270 2024-04-16T16:14:24.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.536+0000 7f0b65600640 1 --2- 172.21.15.45:0/2569241971 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f0b4006c8f0 0x7f0b4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:24.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.537+0000 7f0b65600640 1 --2- 172.21.15.45:0/2569241971 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f0b4006c8f0 0x7f0b4006edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0b54036000 tx=0x7f0b54039040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:24.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.540+0000 7f0b5d600640 1 -- 172.21.15.45:0/2569241971 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0b600ba7a0 con 0x7f0b60162270 2024-04-16T16:14:24.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.777+0000 7f0b66a00640 1 -- 172.21.15.45:0/2569241971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b60075780 con 0x7f0b60162270 2024-04-16T16:14:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.778+0000 7f0b5d600640 1 -- 172.21.15.45:0/2569241971 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5178 (secure 0 0 0) 0x7f0b58020020 con 0x7f0b60162270 2024-04-16T16:14:24.778 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:24.778 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2486891866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2486891866},{"type":"v1","addr":"172.21.15.103:6836","nonce":2486891866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1877281473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1877281473},{"type":"v1","addr":"172.21.15.103:6837","nonce":1877281473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/315848518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":315848518},{"type":"v1","addr":"172.21.15.45:6833","nonce":315848518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:13:25.369839+0000","modified":"2024-04-16T16:13:28.463144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2461453826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2461453826},{"type":"v1","addr":"172.21.15.45:6835","nonce":2461453826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-04-16T16:14:24.778 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 8 2024-04-16T16:14:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.781+0000 7f0b66a00640 1 -- 172.21.15.45:0/2569241971 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f0b4006c8f0 msgr2=0x7f0b4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.781+0000 7f0b66a00640 1 --2- 172.21.15.45:0/2569241971 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f0b4006c8f0 0x7f0b4006edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0b54036000 tx=0x7f0b54039040 comp rx=0 tx=0).stop 2024-04-16T16:14:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.781+0000 7f0b66a00640 1 -- 172.21.15.45:0/2569241971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b60162270 msgr2=0x7f0b601f53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.781+0000 7f0b66a00640 1 --2- 172.21.15.45:0/2569241971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b60162270 0x7f0b601f53f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800c710 tx=0x7f0b5800cbd0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.781+0000 7f0b66a00640 1 -- 172.21.15.45:0/2569241971 shutdown_connections 2024-04-16T16:14:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.781+0000 7f0b66a00640 1 --2- 172.21.15.45:0/2569241971 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f0b4006c8f0 0x7f0b4006edb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.781+0000 7f0b66a00640 1 --2- 172.21.15.45:0/2569241971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b60162270 0x7f0b601f53f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.781+0000 7f0b66a00640 1 --2- 172.21.15.45:0/2569241971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b60161e90 0x7f0b6014ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.781+0000 7f0b66a00640 1 --2- 172.21.15.45:0/2569241971 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b600d52d0 0x7f0b601590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.781+0000 7f0b66a00640 1 -- 172.21.15.45:0/2569241971 >> 172.21.15.45:0/2569241971 conn(0x7f0b600d1940 msgr2=0x7f0b600b9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.782+0000 7f0b66a00640 1 -- 172.21.15.45:0/2569241971 shutdown_connections 2024-04-16T16:14:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:24.782+0000 7f0b66a00640 1 -- 172.21.15.45:0/2569241971 wait complete. 2024-04-16T16:14:24.798 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:25.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.013+0000 7fa4b3400640 1 Processor -- start 2024-04-16T16:14:25.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.013+0000 7fa4b3400640 1 -- start start 2024-04-16T16:14:25.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.015+0000 7fa4b3400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa4ac147840 0x7fa4ac15fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:25.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.015+0000 7fa4b3400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa4ac072a90 0x7fa4ac06e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.015+0000 7fa4b3400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4ac147c20 con 0x7fa4ac0aed90 2024-04-16T16:14:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.015+0000 7fa4b3400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa4ac0afe80 con 0x7fa4ac072a90 2024-04-16T16:14:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.015+0000 7fa4b3400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa4ac0af270 con 0x7fa4ac147840 2024-04-16T16:14:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.015+0000 7fa4b1400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa4ac072a90 0x7fa4ac06e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.016+0000 7fa4b1400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa4ac072a90 0x7fa4ac06e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53286/0 (socket says 172.21.15.45:53286) 2024-04-16T16:14:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.016+0000 7fa4b1400640 1 -- 172.21.15.45:0/3846062115 learned_addr learned my addr 172.21.15.45:0/3846062115 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.016+0000 7fa4b1400640 1 -- 172.21.15.45:0/3846062115 >> v2:172.21.15.103:3300/0 conn(0x7fa4ac147840 msgr2=0x7fa4ac15fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.016+0000 7fa4b0a00640 1 --2- 172.21.15.45:0/3846062115 >> v2:172.21.15.103:3300/0 conn(0x7fa4ac147840 0x7fa4ac15fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.016+0000 7fa4b1400640 1 --2- 172.21.15.45:0/3846062115 >> v2:172.21.15.103:3300/0 conn(0x7fa4ac147840 0x7fa4ac15fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.016+0000 7fa4b1400640 1 -- 172.21.15.45:0/3846062115 >> v1:172.21.15.103:6789/0 conn(0x7fa4ac0aed90 legacy=0x7fa4ac072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.016+0000 7fa4b1400640 1 -- 172.21.15.45:0/3846062115 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4ac0af550 con 0x7fa4ac072a90 2024-04-16T16:14:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.016+0000 7fa4b0a00640 1 --2- 172.21.15.45:0/3846062115 >> v2:172.21.15.103:3300/0 conn(0x7fa4ac147840 0x7fa4ac15fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.017+0000 7fa4b1400640 1 --2- 172.21.15.45:0/3846062115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4ac072a90 0x7fa4ac06e2b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac06ab80 tx=0x7fa4a402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba3825040f2e8b5c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.018+0000 7fa4ab400640 1 -- 172.21.15.45:0/3846062115 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a402ca80 con 0x7fa4ac072a90 2024-04-16T16:14:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.018+0000 7fa4ab400640 1 -- 172.21.15.45:0/3846062115 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4a402cc20 con 0x7fa4ac072a90 2024-04-16T16:14:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.018+0000 7fa4ab400640 1 -- 172.21.15.45:0/3846062115 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a40343f0 con 0x7fa4ac072a90 2024-04-16T16:14:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.018+0000 7fa4b3400640 1 -- 172.21.15.45:0/3846062115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4ac072a90 msgr2=0x7fa4ac06e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.018+0000 7fa4b3400640 1 --2- 172.21.15.45:0/3846062115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4ac072a90 0x7fa4ac06e2b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac06ab80 tx=0x7fa4a402bee0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.019+0000 7fa4b3400640 1 -- 172.21.15.45:0/3846062115 shutdown_connections 2024-04-16T16:14:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.019+0000 7fa4b3400640 1 --2- 172.21.15.45:0/3846062115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4ac072a90 0x7fa4ac06e2b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.019+0000 7fa4b3400640 1 --2- 172.21.15.45:0/3846062115 >> v2:172.21.15.103:3300/0 conn(0x7fa4ac147840 0x7fa4ac15fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.019+0000 7fa4b3400640 1 -- 172.21.15.45:0/3846062115 >> 172.21.15.45:0/3846062115 conn(0x7fa4ac0b2750 msgr2=0x7fa4ac0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.019+0000 7fa4b3400640 1 -- 172.21.15.45:0/3846062115 shutdown_connections 2024-04-16T16:14:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.019+0000 7fa4b3400640 1 -- 172.21.15.45:0/3846062115 wait complete. 2024-04-16T16:14:25.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.020+0000 7fa4b3400640 1 Processor -- start 2024-04-16T16:14:25.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.021+0000 7fa4b3400640 1 -- start start 2024-04-16T16:14:25.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4b3400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac072a90 0x7fa4ac15e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:25.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4b3400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0aed90 0x7fa4ac12fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:25.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4b3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4ac147840 0x7fa4ac161d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:25.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4b3400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ac160040 con 0x7fa4ac147840 2024-04-16T16:14:25.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4b3400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ac0afb10 con 0x7fa4ac0aed90 2024-04-16T16:14:25.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4b3400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa4ac1601c0 con 0x7fa4ac072a90 2024-04-16T16:14:25.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4b1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4ac147840 0x7fa4ac161d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:25.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4b1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4ac147840 0x7fa4ac161d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53296/0 (socket says 172.21.15.45:53296) 2024-04-16T16:14:25.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4b1400640 1 -- 172.21.15.45:0/3262934051 learned_addr learned my addr 172.21.15.45:0/3262934051 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:25.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4abe00640 1 --2- 172.21.15.45:0/3262934051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0aed90 0x7fa4ac12fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:25.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4b0a00640 1 --2- 172.21.15.45:0/3262934051 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac072a90 0x7fa4ac15e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:25.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4b1400640 1 -- 172.21.15.45:0/3262934051 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac072a90 msgr2=0x7fa4ac15e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:25.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4b1400640 1 --2- 172.21.15.45:0/3262934051 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac072a90 0x7fa4ac15e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4b1400640 1 -- 172.21.15.45:0/3262934051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0aed90 msgr2=0x7fa4ac12fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:25.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4b1400640 1 --2- 172.21.15.45:0/3262934051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0aed90 0x7fa4ac12fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4b1400640 1 -- 172.21.15.45:0/3262934051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4ac130880 con 0x7fa4ac147840 2024-04-16T16:14:25.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.022+0000 7fa4abe00640 1 --2- 172.21.15.45:0/3262934051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0aed90 0x7fa4ac12fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:25.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.023+0000 7fa4b1400640 1 --2- 172.21.15.45:0/3262934051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4ac147840 0x7fa4ac161d60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa4a402c4f0 tx=0x7fa4a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:25.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.023+0000 7fa4a8c00640 1 -- 172.21.15.45:0/3262934051 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4043020 con 0x7fa4ac147840 2024-04-16T16:14:25.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.023+0000 7fa4a8c00640 1 -- 172.21.15.45:0/3262934051 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4a403e450 con 0x7fa4ac147840 2024-04-16T16:14:25.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.023+0000 7fa4a8c00640 1 -- 172.21.15.45:0/3262934051 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a403e820 con 0x7fa4ac147840 2024-04-16T16:14:25.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.023+0000 7fa4b3400640 1 -- 172.21.15.45:0/3262934051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4ac130b60 con 0x7fa4ac147840 2024-04-16T16:14:25.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.023+0000 7fa4b3400640 1 -- 172.21.15.45:0/3262934051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4ac12e240 con 0x7fa4ac147840 2024-04-16T16:14:25.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.025+0000 7fa4b3400640 1 -- 172.21.15.45:0/3262934051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4ac12e440 con 0x7fa4ac147840 2024-04-16T16:14:25.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.025+0000 7fa4a8c00640 1 -- 172.21.15.45:0/3262934051 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa4a4046020 con 0x7fa4ac147840 2024-04-16T16:14:25.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.026+0000 7fa4a8c00640 1 --2- 172.21.15.45:0/3262934051 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa48c06c920 0x7fa48c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:25.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.026+0000 7fa4a8c00640 1 -- 172.21.15.45:0/3262934051 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa4a40ae2d0 con 0x7fa4ac147840 2024-04-16T16:14:25.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.027+0000 7fa4b0a00640 1 --2- 172.21.15.45:0/3262934051 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa48c06c920 0x7fa48c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:25.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.028+0000 7fa4b0a00640 1 --2- 172.21.15.45:0/3262934051 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa48c06c920 0x7fa48c06ede0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000b840 tx=0x7fa4a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:25.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.031+0000 7fa4a8c00640 1 -- 172.21.15.45:0/3262934051 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa4ac12e440 con 0x7fa4ac147840 2024-04-16T16:14:25.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.237+0000 7fa4b3400640 1 -- 172.21.15.45:0/3262934051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa4ac075780 con 0x7fa4ac147840 2024-04-16T16:14:25.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.239+0000 7fa4a8c00640 1 -- 172.21.15.45:0/3262934051 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16799 (secure 0 0 0) 0x7fa4a404d070 con 0x7fa4ac147840 2024-04-16T16:14:25.238 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:25.238 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":23,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:14:12.851304+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:25.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.241+0000 7fa4b3400640 1 -- 172.21.15.45:0/3262934051 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa48c06c920 msgr2=0x7fa48c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.241+0000 7fa4b3400640 1 --2- 172.21.15.45:0/3262934051 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa48c06c920 0x7fa48c06ede0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000b840 tx=0x7fa4a0008040 comp rx=0 tx=0).stop 2024-04-16T16:14:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.241+0000 7fa4b3400640 1 -- 172.21.15.45:0/3262934051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4ac147840 msgr2=0x7fa4ac161d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.241+0000 7fa4b3400640 1 --2- 172.21.15.45:0/3262934051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4ac147840 0x7fa4ac161d60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa4a402c4f0 tx=0x7fa4a4004a40 comp rx=0 tx=0).stop 2024-04-16T16:14:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.241+0000 7fa4b3400640 1 -- 172.21.15.45:0/3262934051 shutdown_connections 2024-04-16T16:14:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.241+0000 7fa4b3400640 1 --2- 172.21.15.45:0/3262934051 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa48c06c920 0x7fa48c06ede0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.241+0000 7fa4b3400640 1 --2- 172.21.15.45:0/3262934051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4ac147840 0x7fa4ac161d60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.241+0000 7fa4b3400640 1 --2- 172.21.15.45:0/3262934051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0aed90 0x7fa4ac12fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.241+0000 7fa4b3400640 1 --2- 172.21.15.45:0/3262934051 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac072a90 0x7fa4ac15e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.241+0000 7fa4b3400640 1 -- 172.21.15.45:0/3262934051 >> 172.21.15.45:0/3262934051 conn(0x7fa4ac0b2750 msgr2=0x7fa4ac0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.242+0000 7fa4b3400640 1 -- 172.21.15.45:0/3262934051 shutdown_connections 2024-04-16T16:14:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.242+0000 7fa4b3400640 1 -- 172.21.15.45:0/3262934051 wait complete. 2024-04-16T16:14:25.261 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-16T16:14:25.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:14:25.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.488+0000 7f5bcec00640 1 Processor -- start 2024-04-16T16:14:25.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.488+0000 7f5bcec00640 1 -- start start 2024-04-16T16:14:25.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.490+0000 7f5bcec00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc8146220 0x7f5bc815ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:25.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.490+0000 7f5bcec00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5bc81608a0 0x7f5bc8160d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:25.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.490+0000 7f5bcec00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bc815d360 con 0x7f5bc806f7a0 2024-04-16T16:14:25.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.490+0000 7f5bcec00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5bc8160410 con 0x7f5bc81608a0 2024-04-16T16:14:25.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.490+0000 7f5bcec00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc815d640 con 0x7f5bc8146220 2024-04-16T16:14:25.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.491+0000 7f5bcd800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc8146220 0x7f5bc815ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:25.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.491+0000 7f5bcd800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc8146220 0x7f5bc815ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:48846/0 (socket says 172.21.15.45:48846) 2024-04-16T16:14:25.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.491+0000 7f5bcd800640 1 -- 172.21.15.45:0/690616828 learned_addr learned my addr 172.21.15.45:0/690616828 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:25.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.491+0000 7f5bce200640 1 --2- 172.21.15.45:0/690616828 >> v2:172.21.15.103:3300/0 conn(0x7f5bc81608a0 0x7f5bc8160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:25.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.491+0000 7f5bcd800640 1 -- 172.21.15.45:0/690616828 >> v2:172.21.15.103:3300/0 conn(0x7f5bc81608a0 msgr2=0x7f5bc8160d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:25.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.491+0000 7f5bcd800640 1 --2- 172.21.15.45:0/690616828 >> v2:172.21.15.103:3300/0 conn(0x7f5bc81608a0 0x7f5bc8160d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.491+0000 7f5bcd800640 1 -- 172.21.15.45:0/690616828 >> v1:172.21.15.103:6789/0 conn(0x7f5bc806f7a0 legacy=0x7f5bc806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:25.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.491+0000 7f5bcd800640 1 -- 172.21.15.45:0/690616828 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc815d920 con 0x7f5bc8146220 2024-04-16T16:14:25.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.492+0000 7f5bcd800640 1 --2- 172.21.15.45:0/690616828 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc8146220 0x7f5bc815ccb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009e50 tx=0x7f5bbc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4b12b49e4612e85 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:25.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.493+0000 7f5bc7e00640 1 -- 172.21.15.45:0/690616828 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc02cb00 con 0x7f5bc8146220 2024-04-16T16:14:25.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.493+0000 7f5bc7e00640 1 -- 172.21.15.45:0/690616828 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bbc02cca0 con 0x7f5bc8146220 2024-04-16T16:14:25.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.493+0000 7f5bc7e00640 1 -- 172.21.15.45:0/690616828 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc034610 con 0x7f5bc8146220 2024-04-16T16:14:25.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.494+0000 7f5bcec00640 1 -- 172.21.15.45:0/690616828 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc8146220 msgr2=0x7f5bc815ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:25.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.494+0000 7f5bcec00640 1 --2- 172.21.15.45:0/690616828 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc8146220 0x7f5bc815ccb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009e50 tx=0x7f5bbc02bf60 comp rx=0 tx=0).stop 2024-04-16T16:14:25.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.494+0000 7f5bcec00640 1 -- 172.21.15.45:0/690616828 shutdown_connections 2024-04-16T16:14:25.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.494+0000 7f5bcec00640 1 --2- 172.21.15.45:0/690616828 >> v2:172.21.15.103:3300/0 conn(0x7f5bc81608a0 0x7f5bc8160d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.494+0000 7f5bcec00640 1 --2- 172.21.15.45:0/690616828 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc8146220 0x7f5bc815ccb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.494+0000 7f5bcec00640 1 -- 172.21.15.45:0/690616828 >> 172.21.15.45:0/690616828 conn(0x7f5bc806b930 msgr2=0x7f5bc806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:25.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.494+0000 7f5bcec00640 1 -- 172.21.15.45:0/690616828 shutdown_connections 2024-04-16T16:14:25.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.494+0000 7f5bcec00640 1 -- 172.21.15.45:0/690616828 wait complete. 2024-04-16T16:14:25.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.496+0000 7f5bcec00640 1 Processor -- start 2024-04-16T16:14:25.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.496+0000 7f5bcec00640 1 -- start start 2024-04-16T16:14:25.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.497+0000 7f5bcec00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc806f7a0 0x7f5bc80b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:25.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.497+0000 7f5bcec00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc8146220 0x7f5bc813a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:25.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.497+0000 7f5bcec00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc81608a0 0x7f5bc815e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:25.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.497+0000 7f5bcec00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc806fc20 con 0x7f5bc81608a0 2024-04-16T16:14:25.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.497+0000 7f5bcec00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc815dc00 con 0x7f5bc8146220 2024-04-16T16:14:25.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.497+0000 7f5bcec00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc806fda0 con 0x7f5bc806f7a0 2024-04-16T16:14:25.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.497+0000 7f5bce200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc81608a0 0x7f5bc815e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:25.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.497+0000 7f5bce200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc81608a0 0x7f5bc815e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53306/0 (socket says 172.21.15.45:53306) 2024-04-16T16:14:25.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.497+0000 7f5bce200640 1 -- 172.21.15.45:0/693725698 learned_addr learned my addr 172.21.15.45:0/693725698 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:25.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.498+0000 7f5bcd800640 1 --2- 172.21.15.45:0/693725698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc806f7a0 0x7f5bc80b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:25.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.498+0000 7f5bcce00640 1 --2- 172.21.15.45:0/693725698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc8146220 0x7f5bc813a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:25.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.498+0000 7f5bce200640 1 -- 172.21.15.45:0/693725698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc806f7a0 msgr2=0x7f5bc80b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:25.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.498+0000 7f5bce200640 1 --2- 172.21.15.45:0/693725698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc806f7a0 0x7f5bc80b3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.498+0000 7f5bce200640 1 -- 172.21.15.45:0/693725698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc8146220 msgr2=0x7f5bc813a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:25.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.498+0000 7f5bce200640 1 --2- 172.21.15.45:0/693725698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc8146220 0x7f5bc813a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.498+0000 7f5bce200640 1 -- 172.21.15.45:0/693725698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc80b35c0 con 0x7f5bc81608a0 2024-04-16T16:14:25.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.498+0000 7f5bce200640 1 --2- 172.21.15.45:0/693725698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc81608a0 0x7f5bc815e2d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000d5d0 tx=0x7f5bc000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:25.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.499+0000 7f5bc5600640 1 -- 172.21.15.45:0/693725698 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc00125f0 con 0x7f5bc81608a0 2024-04-16T16:14:25.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.499+0000 7f5bc5600640 1 -- 172.21.15.45:0/693725698 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bc0019070 con 0x7f5bc81608a0 2024-04-16T16:14:25.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.499+0000 7f5bc5600640 1 -- 172.21.15.45:0/693725698 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc00129f0 con 0x7f5bc81608a0 2024-04-16T16:14:25.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.499+0000 7f5bcec00640 1 -- 172.21.15.45:0/693725698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc813c9d0 con 0x7f5bc81608a0 2024-04-16T16:14:25.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.499+0000 7f5bcec00640 1 -- 172.21.15.45:0/693725698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc813cf90 con 0x7f5bc81608a0 2024-04-16T16:14:25.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.500+0000 7f5bcec00640 1 -- 172.21.15.45:0/693725698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bc8075780 con 0x7f5bc81608a0 2024-04-16T16:14:25.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.501+0000 7f5bc5600640 1 -- 172.21.15.45:0/693725698 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5bc0012b90 con 0x7f5bc81608a0 2024-04-16T16:14:25.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.502+0000 7f5bc5600640 1 --2- 172.21.15.45:0/693725698 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ba806c8f0 0x7f5ba806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:25.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.502+0000 7f5bc5600640 1 -- 172.21.15.45:0/693725698 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5bc008c040 con 0x7f5bc81608a0 2024-04-16T16:14:25.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.503+0000 7f5bcd800640 1 --2- 172.21.15.45:0/693725698 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ba806c8f0 0x7f5ba806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:25.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.504+0000 7f5bcd800640 1 --2- 172.21.15.45:0/693725698 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ba806c8f0 0x7f5ba806edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc035000 tx=0x7f5bbc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:25.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.506+0000 7f5bc5600640 1 -- 172.21.15.45:0/693725698 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5bc005a360 con 0x7f5bc81608a0 2024-04-16T16:14:25.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.755+0000 7f5bcec00640 1 -- 172.21.15.45:0/693725698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bc80cff60 con 0x7f5bc81608a0 2024-04-16T16:14:25.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.756+0000 7f5bc5600640 1 -- 172.21.15.45:0/693725698 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5178 (secure 0 0 0) 0x7f5bc0059c60 con 0x7f5bc81608a0 2024-04-16T16:14:25.755 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:25.755 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2486891866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2486891866},{"type":"v1","addr":"172.21.15.103:6836","nonce":2486891866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1877281473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1877281473},{"type":"v1","addr":"172.21.15.103:6837","nonce":1877281473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/315848518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":315848518},{"type":"v1","addr":"172.21.15.45:6833","nonce":315848518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:13:25.369839+0000","modified":"2024-04-16T16:13:28.463144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2461453826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2461453826},{"type":"v1","addr":"172.21.15.45:6835","nonce":2461453826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-04-16T16:14:25.755 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 8 2024-04-16T16:14:25.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.758+0000 7f5bcec00640 1 -- 172.21.15.45:0/693725698 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ba806c8f0 msgr2=0x7f5ba806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:25.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.758+0000 7f5bcec00640 1 --2- 172.21.15.45:0/693725698 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ba806c8f0 0x7f5ba806edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc035000 tx=0x7f5bbc030000 comp rx=0 tx=0).stop 2024-04-16T16:14:25.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.758+0000 7f5bcec00640 1 -- 172.21.15.45:0/693725698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc81608a0 msgr2=0x7f5bc815e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:25.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.758+0000 7f5bcec00640 1 --2- 172.21.15.45:0/693725698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc81608a0 0x7f5bc815e2d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000d5d0 tx=0x7f5bc000da90 comp rx=0 tx=0).stop 2024-04-16T16:14:25.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.758+0000 7f5bcec00640 1 -- 172.21.15.45:0/693725698 shutdown_connections 2024-04-16T16:14:25.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.758+0000 7f5bcec00640 1 --2- 172.21.15.45:0/693725698 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ba806c8f0 0x7f5ba806edb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.758+0000 7f5bcec00640 1 --2- 172.21.15.45:0/693725698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc81608a0 0x7f5bc815e2d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.758+0000 7f5bcec00640 1 --2- 172.21.15.45:0/693725698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc8146220 0x7f5bc813a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.758+0000 7f5bcec00640 1 --2- 172.21.15.45:0/693725698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc806f7a0 0x7f5bc80b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:25.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.758+0000 7f5bcec00640 1 -- 172.21.15.45:0/693725698 >> 172.21.15.45:0/693725698 conn(0x7f5bc806b930 msgr2=0x7f5bc80b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:25.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.759+0000 7f5bcec00640 1 -- 172.21.15.45:0/693725698 shutdown_connections 2024-04-16T16:14:25.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:25.759+0000 7f5bcec00640 1 -- 172.21.15.45:0/693725698 wait complete. 2024-04-16T16:14:25.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:14:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.005+0000 7f8e48a00640 1 Processor -- start 2024-04-16T16:14:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.005+0000 7f8e48a00640 1 -- start start 2024-04-16T16:14:26.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.007+0000 7f8e48a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8e440c5d80 0x7f8e44166600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:26.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.007+0000 7f8e48a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e4415c700 con 0x7f8e44144ff0 2024-04-16T16:14:26.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.007+0000 7f8e48a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e4415c9c0 con 0x7f8e4406dfb0 2024-04-16T16:14:26.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.007+0000 7f8e48a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8e4415d2f0 con 0x7f8e440c5d80 2024-04-16T16:14:26.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.007+0000 7f8e43400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8e44144ff0 0x7f8e440cfd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45344/0 (socket says 172.21.15.45:45344) 2024-04-16T16:14:26.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.007+0000 7f8e43400640 1 -- 172.21.15.45:0/209723727 learned_addr learned my addr 172.21.15.45:0/209723727 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:26.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.007+0000 7f8e43e00640 1 --2- 172.21.15.45:0/209723727 >> v2:172.21.15.45:3300/0 conn(0x7f8e440c5d80 0x7f8e44166600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:26.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.008+0000 7f8e43e00640 1 -- 172.21.15.45:0/209723727 >> v1:172.21.15.103:6789/0 conn(0x7f8e4406dfb0 legacy=0x7f8e440c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:26.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.008+0000 7f8e43e00640 1 -- 172.21.15.45:0/209723727 >> v1:172.21.15.45:6789/0 conn(0x7f8e44144ff0 legacy=0x7f8e440cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:26.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.008+0000 7f8e42000640 1 -- 172.21.15.45:0/209723727 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1594680061 0 0) 0x7f8e4415c700 con 0x7f8e44144ff0 2024-04-16T16:14:26.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.008+0000 7f8e43e00640 1 -- 172.21.15.45:0/209723727 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4415cca0 con 0x7f8e440c5d80 2024-04-16T16:14:26.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.009+0000 7f8e43e00640 1 --2- 172.21.15.45:0/209723727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e440c5d80 0x7f8e44166600 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8e4415d470 tx=0x7f8e3c02e0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ebcbba216a6d6b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:26.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.009+0000 7f8e42000640 1 -- 172.21.15.45:0/209723727 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c03b070 con 0x7f8e440c5d80 2024-04-16T16:14:26.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.009+0000 7f8e42000640 1 -- 172.21.15.45:0/209723727 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e3c036440 con 0x7f8e440c5d80 2024-04-16T16:14:26.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.009+0000 7f8e42000640 1 -- 172.21.15.45:0/209723727 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c02ee60 con 0x7f8e440c5d80 2024-04-16T16:14:26.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.010+0000 7f8e48a00640 1 -- 172.21.15.45:0/209723727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e440c5d80 msgr2=0x7f8e44166600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:26.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.010+0000 7f8e48a00640 1 --2- 172.21.15.45:0/209723727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e440c5d80 0x7f8e44166600 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8e4415d470 tx=0x7f8e3c02e0b0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.010+0000 7f8e48a00640 1 -- 172.21.15.45:0/209723727 shutdown_connections 2024-04-16T16:14:26.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.010+0000 7f8e48a00640 1 --2- 172.21.15.45:0/209723727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e440c5d80 0x7f8e44166600 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.010+0000 7f8e48a00640 1 -- 172.21.15.45:0/209723727 >> 172.21.15.45:0/209723727 conn(0x7f8e4406b930 msgr2=0x7f8e4406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:26.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.010+0000 7f8e48a00640 1 -- 172.21.15.45:0/209723727 shutdown_connections 2024-04-16T16:14:26.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.010+0000 7f8e48a00640 1 -- 172.21.15.45:0/209723727 wait complete. 2024-04-16T16:14:26.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.012+0000 7f8e48a00640 1 Processor -- start 2024-04-16T16:14:26.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.012+0000 7f8e48a00640 1 -- start start 2024-04-16T16:14:26.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.013+0000 7f8e48a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e4406dfb0 0x7f8e4413ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:26.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.013+0000 7f8e48a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e440c5d80 0x7f8e44138a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.013+0000 7f8e48a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e44144ff0 0x7f8e44138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.013+0000 7f8e48a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4415e790 con 0x7f8e440c5d80 2024-04-16T16:14:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.013+0000 7f8e48a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8e441453d0 con 0x7f8e44144ff0 2024-04-16T16:14:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.013+0000 7f8e48a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8e4415e910 con 0x7f8e4406dfb0 2024-04-16T16:14:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.013+0000 7f8e42a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e440c5d80 0x7f8e44138a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.014+0000 7f8e42a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e440c5d80 0x7f8e44138a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53334/0 (socket says 172.21.15.45:53334) 2024-04-16T16:14:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.014+0000 7f8e42a00640 1 -- 172.21.15.45:0/1049924094 learned_addr learned my addr 172.21.15.45:0/1049924094 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.014+0000 7f8e43400640 1 --2- 172.21.15.45:0/1049924094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e4406dfb0 0x7f8e4413ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.014+0000 7f8e43e00640 1 --2- 172.21.15.45:0/1049924094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e44144ff0 0x7f8e44138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.014+0000 7f8e42a00640 1 -- 172.21.15.45:0/1049924094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e4406dfb0 msgr2=0x7f8e4413ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.014+0000 7f8e42a00640 1 --2- 172.21.15.45:0/1049924094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e4406dfb0 0x7f8e4413ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.014+0000 7f8e42a00640 1 -- 172.21.15.45:0/1049924094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e44144ff0 msgr2=0x7f8e44138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.014+0000 7f8e42a00640 1 --2- 172.21.15.45:0/1049924094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e44144ff0 0x7f8e44138fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.014+0000 7f8e42a00640 1 -- 172.21.15.45:0/1049924094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e44135690 con 0x7f8e440c5d80 2024-04-16T16:14:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.014+0000 7f8e43400640 1 --2- 172.21.15.45:0/1049924094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e4406dfb0 0x7f8e4413ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:14:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.014+0000 7f8e42a00640 1 --2- 172.21.15.45:0/1049924094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e440c5d80 0x7f8e44138a70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000c400 tx=0x7f8e3000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:26.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.015+0000 7f8e3b400640 1 -- 172.21.15.45:0/1049924094 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30011790 con 0x7f8e440c5d80 2024-04-16T16:14:26.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.015+0000 7f8e3b400640 1 -- 172.21.15.45:0/1049924094 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e30017070 con 0x7f8e440c5d80 2024-04-16T16:14:26.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.015+0000 7f8e3b400640 1 -- 172.21.15.45:0/1049924094 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30011bb0 con 0x7f8e440c5d80 2024-04-16T16:14:26.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.015+0000 7f8e48a00640 1 -- 172.21.15.45:0/1049924094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e44135970 con 0x7f8e440c5d80 2024-04-16T16:14:26.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.015+0000 7f8e48a00640 1 -- 172.21.15.45:0/1049924094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e44135f30 con 0x7f8e440c5d80 2024-04-16T16:14:26.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.016+0000 7f8e48a00640 1 -- 172.21.15.45:0/1049924094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e44075780 con 0x7f8e440c5d80 2024-04-16T16:14:26.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.017+0000 7f8e3b400640 1 -- 172.21.15.45:0/1049924094 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8e3001c460 con 0x7f8e440c5d80 2024-04-16T16:14:26.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.018+0000 7f8e3b400640 1 --2- 172.21.15.45:0/1049924094 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8e2406c8f0 0x7f8e2406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:26.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.018+0000 7f8e3b400640 1 -- 172.21.15.45:0/1049924094 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8e3008bb20 con 0x7f8e440c5d80 2024-04-16T16:14:26.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.019+0000 7f8e43400640 1 --2- 172.21.15.45:0/1049924094 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8e2406c8f0 0x7f8e2406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:26.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.020+0000 7f8e43400640 1 --2- 172.21.15.45:0/1049924094 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8e2406c8f0 0x7f8e2406edb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8e34000f60 tx=0x7f8e34007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:26.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.023+0000 7f8e3b400640 1 -- 172.21.15.45:0/1049924094 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8e30059e30 con 0x7f8e440c5d80 2024-04-16T16:14:26.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.276+0000 7f8e48a00640 1 -- 172.21.15.45:0/1049924094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e4415fb40 con 0x7f8e440c5d80 2024-04-16T16:14:26.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.278+0000 7f8e3b400640 1 -- 172.21.15.45:0/1049924094 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5178 (secure 0 0 0) 0x7f8e30059730 con 0x7f8e440c5d80 2024-04-16T16:14:26.277 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:26.277 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2486891866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2486891866},{"type":"v1","addr":"172.21.15.103:6836","nonce":2486891866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1877281473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1877281473},{"type":"v1","addr":"172.21.15.103:6837","nonce":1877281473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/315848518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":315848518},{"type":"v1","addr":"172.21.15.45:6833","nonce":315848518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:13:25.369839+0000","modified":"2024-04-16T16:13:28.463144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2461453826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2461453826},{"type":"v1","addr":"172.21.15.45:6835","nonce":2461453826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-04-16T16:14:26.277 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 8 2024-04-16T16:14:26.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.280+0000 7f8e48a00640 1 -- 172.21.15.45:0/1049924094 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8e2406c8f0 msgr2=0x7f8e2406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:26.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.280+0000 7f8e48a00640 1 --2- 172.21.15.45:0/1049924094 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8e2406c8f0 0x7f8e2406edb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8e34000f60 tx=0x7f8e34007ca0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.280+0000 7f8e48a00640 1 -- 172.21.15.45:0/1049924094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e440c5d80 msgr2=0x7f8e44138a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:26.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.280+0000 7f8e48a00640 1 --2- 172.21.15.45:0/1049924094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e440c5d80 0x7f8e44138a70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000c400 tx=0x7f8e3000c8c0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.281+0000 7f8e48a00640 1 -- 172.21.15.45:0/1049924094 shutdown_connections 2024-04-16T16:14:26.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.281+0000 7f8e48a00640 1 --2- 172.21.15.45:0/1049924094 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8e2406c8f0 0x7f8e2406edb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.281+0000 7f8e48a00640 1 --2- 172.21.15.45:0/1049924094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e44144ff0 0x7f8e44138fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.281+0000 7f8e48a00640 1 --2- 172.21.15.45:0/1049924094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e440c5d80 0x7f8e44138a70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.281+0000 7f8e48a00640 1 --2- 172.21.15.45:0/1049924094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e4406dfb0 0x7f8e4413ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.281+0000 7f8e48a00640 1 -- 172.21.15.45:0/1049924094 >> 172.21.15.45:0/1049924094 conn(0x7f8e4406b930 msgr2=0x7f8e440b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:26.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.281+0000 7f8e48a00640 1 -- 172.21.15.45:0/1049924094 shutdown_connections 2024-04-16T16:14:26.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.281+0000 7f8e48a00640 1 -- 172.21.15.45:0/1049924094 wait complete. 2024-04-16T16:14:26.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:26.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.523+0000 7f9133200640 1 Processor -- start 2024-04-16T16:14:26.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.524+0000 7f9133200640 1 -- start start 2024-04-16T16:14:26.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.526+0000 7f9133200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f91341608a0 0x7f9134160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:26.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.526+0000 7f9133200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f913415d360 con 0x7f9134146220 2024-04-16T16:14:26.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.526+0000 7f9133200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f913415d640 con 0x7f913415ccb0 2024-04-16T16:14:26.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.526+0000 7f9133200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f913415d130 con 0x7f91341608a0 2024-04-16T16:14:26.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.526+0000 7f9132800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f91341608a0 0x7f9134160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:26.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.526+0000 7f9131e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9134146220 0x7f913406f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45348/0 (socket says 172.21.15.45:45348) 2024-04-16T16:14:26.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.526+0000 7f9131e00640 1 -- 172.21.15.45:0/2198686163 learned_addr learned my addr 172.21.15.45:0/2198686163 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:26.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.527+0000 7f9132800640 1 -- 172.21.15.45:0/2198686163 >> v1:172.21.15.103:6789/0 conn(0x7f913415ccb0 legacy=0x7f913406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:26.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.527+0000 7f9132800640 1 -- 172.21.15.45:0/2198686163 >> v1:172.21.15.45:6789/0 conn(0x7f9134146220 legacy=0x7f913406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:26.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.527+0000 7f9132800640 1 -- 172.21.15.45:0/2198686163 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913415d920 con 0x7f91341608a0 2024-04-16T16:14:26.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.528+0000 7f9132800640 1 --2- 172.21.15.45:0/2198686163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91341608a0 0x7f9134160cf0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f912c0097a0 tx=0x7f912c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deed23bec9da875e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:26.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.528+0000 7f9130a00640 1 -- 172.21.15.45:0/2198686163 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c034070 con 0x7f91341608a0 2024-04-16T16:14:26.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.528+0000 7f9130a00640 1 -- 172.21.15.45:0/2198686163 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912c030610 con 0x7f91341608a0 2024-04-16T16:14:26.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.528+0000 7f9130a00640 1 -- 172.21.15.45:0/2198686163 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c0309e0 con 0x7f91341608a0 2024-04-16T16:14:26.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.529+0000 7f9133200640 1 -- 172.21.15.45:0/2198686163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91341608a0 msgr2=0x7f9134160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:26.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.529+0000 7f9133200640 1 --2- 172.21.15.45:0/2198686163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91341608a0 0x7f9134160cf0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f912c0097a0 tx=0x7f912c02b960 comp rx=0 tx=0).stop 2024-04-16T16:14:26.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.529+0000 7f9133200640 1 -- 172.21.15.45:0/2198686163 shutdown_connections 2024-04-16T16:14:26.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.529+0000 7f9133200640 1 --2- 172.21.15.45:0/2198686163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91341608a0 0x7f9134160cf0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.529+0000 7f9133200640 1 -- 172.21.15.45:0/2198686163 >> 172.21.15.45:0/2198686163 conn(0x7f913406b930 msgr2=0x7f913406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:26.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.529+0000 7f9133200640 1 -- 172.21.15.45:0/2198686163 shutdown_connections 2024-04-16T16:14:26.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.529+0000 7f9133200640 1 -- 172.21.15.45:0/2198686163 wait complete. 2024-04-16T16:14:26.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.531+0000 7f9133200640 1 Processor -- start 2024-04-16T16:14:26.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.531+0000 7f9133200640 1 -- start start 2024-04-16T16:14:26.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.532+0000 7f9133200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9134146220 0x7f913415bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:26.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.532+0000 7f9133200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f913415ccb0 0x7f91341f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:26.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.532+0000 7f9133200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91341608a0 0x7f91341f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:26.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.532+0000 7f9133200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9134160410 con 0x7f91341608a0 2024-04-16T16:14:26.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.532+0000 7f9133200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f913415dc00 con 0x7f9134146220 2024-04-16T16:14:26.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.532+0000 7f9133200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f91340cf360 con 0x7f913415ccb0 2024-04-16T16:14:26.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.532+0000 7f9132800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91341608a0 0x7f91341f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:26.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.532+0000 7f9132800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91341608a0 0x7f91341f2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53354/0 (socket says 172.21.15.45:53354) 2024-04-16T16:14:26.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.532+0000 7f9132800640 1 -- 172.21.15.45:0/617869280 learned_addr learned my addr 172.21.15.45:0/617869280 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:26.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.532+0000 7f9132800640 1 -- 172.21.15.45:0/617869280 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f913415ccb0 msgr2=0x7f91341f4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:26.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.532+0000 7f9132800640 1 --2- 172.21.15.45:0/617869280 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f913415ccb0 0x7f91341f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.533+0000 7f9131e00640 1 --2- 172.21.15.45:0/617869280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9134146220 0x7f913415bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:26.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.533+0000 7f9132800640 1 -- 172.21.15.45:0/617869280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9134146220 msgr2=0x7f913415bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:26.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.533+0000 7f9132800640 1 --2- 172.21.15.45:0/617869280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9134146220 0x7f913415bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.533+0000 7f9132800640 1 -- 172.21.15.45:0/617869280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91341f3e00 con 0x7f91341608a0 2024-04-16T16:14:26.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.533+0000 7f9132800640 1 --2- 172.21.15.45:0/617869280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91341608a0 0x7f91341f2e60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f912c0098c0 tx=0x7f912c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:26.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.533+0000 7f912a000640 1 -- 172.21.15.45:0/617869280 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c034040 con 0x7f91341608a0 2024-04-16T16:14:26.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.533+0000 7f912a000640 1 -- 172.21.15.45:0/617869280 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912c03ac20 con 0x7f91341608a0 2024-04-16T16:14:26.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.533+0000 7f912a000640 1 -- 172.21.15.45:0/617869280 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c0443f0 con 0x7f91341608a0 2024-04-16T16:14:26.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.533+0000 7f9133200640 1 -- 172.21.15.45:0/617869280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91341f3840 con 0x7f91341608a0 2024-04-16T16:14:26.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.533+0000 7f9133200640 1 -- 172.21.15.45:0/617869280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91341e6200 con 0x7f91341608a0 2024-04-16T16:14:26.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.535+0000 7f9133200640 1 -- 172.21.15.45:0/617869280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9134075780 con 0x7f91341608a0 2024-04-16T16:14:26.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.536+0000 7f912a000640 1 -- 172.21.15.45:0/617869280 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f912c037070 con 0x7f91341608a0 2024-04-16T16:14:26.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.536+0000 7f912a000640 1 --2- 172.21.15.45:0/617869280 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f911406c8f0 0x7f911406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:26.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.537+0000 7f912a000640 1 -- 172.21.15.45:0/617869280 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f912c0adae0 con 0x7f91341608a0 2024-04-16T16:14:26.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.537+0000 7f9131e00640 1 --2- 172.21.15.45:0/617869280 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f911406c8f0 0x7f911406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:26.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.538+0000 7f9131e00640 1 --2- 172.21.15.45:0/617869280 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f911406c8f0 0x7f911406edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9124005880 tx=0x7f9124009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:26.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.541+0000 7f912a000640 1 -- 172.21.15.45:0/617869280 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f912c03f070 con 0x7f91341608a0 2024-04-16T16:14:26.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.751+0000 7f9133200640 1 -- 172.21.15.45:0/617869280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f91340cff60 con 0x7f91341608a0 2024-04-16T16:14:26.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.754+0000 7f912a000640 1 -- 172.21.15.45:0/617869280 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16799 (secure 0 0 0) 0x7f912c07bf20 con 0x7f91341608a0 2024-04-16T16:14:26.754 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:26.754 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":23,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:14:12.851304+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:13:23.600104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-16T16:13:24.368141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:26.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.758+0000 7f9133200640 1 -- 172.21.15.45:0/617869280 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f911406c8f0 msgr2=0x7f911406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:26.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.758+0000 7f9133200640 1 --2- 172.21.15.45:0/617869280 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f911406c8f0 0x7f911406edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9124005880 tx=0x7f9124009040 comp rx=0 tx=0).stop 2024-04-16T16:14:26.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.758+0000 7f9133200640 1 -- 172.21.15.45:0/617869280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91341608a0 msgr2=0x7f91341f2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:26.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.758+0000 7f9133200640 1 --2- 172.21.15.45:0/617869280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91341608a0 0x7f91341f2e60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f912c0098c0 tx=0x7f912c0046c0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.758+0000 7f9133200640 1 -- 172.21.15.45:0/617869280 shutdown_connections 2024-04-16T16:14:26.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.758+0000 7f9133200640 1 --2- 172.21.15.45:0/617869280 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f911406c8f0 0x7f911406edb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.758+0000 7f9133200640 1 --2- 172.21.15.45:0/617869280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91341608a0 0x7f91341f2e60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.758+0000 7f9133200640 1 --2- 172.21.15.45:0/617869280 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f913415ccb0 0x7f91341f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.758+0000 7f9133200640 1 --2- 172.21.15.45:0/617869280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9134146220 0x7f913415bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.758+0000 7f9133200640 1 -- 172.21.15.45:0/617869280 >> 172.21.15.45:0/617869280 conn(0x7f913406b930 msgr2=0x7f913415eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:26.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.758+0000 7f9133200640 1 -- 172.21.15.45:0/617869280 shutdown_connections 2024-04-16T16:14:26.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.758+0000 7f9133200640 1 -- 172.21.15.45:0/617869280 wait complete. 2024-04-16T16:14:26.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-16T16:14:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.996+0000 7fc75de00640 1 Processor -- start 2024-04-16T16:14:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.996+0000 7fc75de00640 1 -- start start 2024-04-16T16:14:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.997+0000 7fc75de00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc7580c6fc0 0x7fc75806f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.997+0000 7fc75de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc75815cdc0 0x7fc7580c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.998+0000 7fc75de00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc75815d600 con 0x7fc7580c6320 2024-04-16T16:14:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.998+0000 7fc75de00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc7580c5670 con 0x7fc7580c6fc0 2024-04-16T16:14:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.998+0000 7fc75de00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc75815d8e0 con 0x7fc75815cdc0 2024-04-16T16:14:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.998+0000 7fc75ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc7580c6fc0 0x7fc75806f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.998+0000 7fc75ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc7580c6fc0 0x7fc75806f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53366/0 (socket says 172.21.15.45:53366) 2024-04-16T16:14:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.998+0000 7fc75ca00640 1 -- 172.21.15.45:0/2130581446 learned_addr learned my addr 172.21.15.45:0/2130581446 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.998+0000 7fc75ca00640 1 -- 172.21.15.45:0/2130581446 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc75815cdc0 msgr2=0x7fc7580c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.998+0000 7fc757e00640 1 --2- 172.21.15.45:0/2130581446 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc75815cdc0 0x7fc7580c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.998+0000 7fc75ca00640 1 --2- 172.21.15.45:0/2130581446 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc75815cdc0 0x7fc7580c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.998+0000 7fc75ca00640 1 -- 172.21.15.45:0/2130581446 >> v1:172.21.15.103:6789/0 conn(0x7fc7580c6320 legacy=0x7fc75806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:26.998+0000 7fc75ca00640 1 -- 172.21.15.45:0/2130581446 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75815dbc0 con 0x7fc7580c6fc0 2024-04-16T16:14:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.000+0000 7fc75ca00640 1 --2- 172.21.15.45:0/2130581446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7580c6fc0 0x7fc75806f730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009870 tx=0x7fc74c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdecbfb6ac8d31f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.000+0000 7fc757400640 1 -- 172.21.15.45:0/2130581446 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74c035020 con 0x7fc7580c6fc0 2024-04-16T16:14:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.000+0000 7fc757400640 1 -- 172.21.15.45:0/2130581446 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc74c031640 con 0x7fc7580c6fc0 2024-04-16T16:14:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.000+0000 7fc757400640 1 -- 172.21.15.45:0/2130581446 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74c02d040 con 0x7fc7580c6fc0 2024-04-16T16:14:27.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.001+0000 7fc75de00640 1 -- 172.21.15.45:0/2130581446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7580c6fc0 msgr2=0x7fc75806f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:27.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.001+0000 7fc75de00640 1 --2- 172.21.15.45:0/2130581446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7580c6fc0 0x7fc75806f730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009870 tx=0x7fc74c02b8f0 comp rx=0 tx=0).stop 2024-04-16T16:14:27.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.001+0000 7fc75de00640 1 -- 172.21.15.45:0/2130581446 shutdown_connections 2024-04-16T16:14:27.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.001+0000 7fc75de00640 1 --2- 172.21.15.45:0/2130581446 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc75815cdc0 0x7fc7580c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:27.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.001+0000 7fc75de00640 1 --2- 172.21.15.45:0/2130581446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7580c6fc0 0x7fc75806f730 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:27.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.001+0000 7fc75de00640 1 -- 172.21.15.45:0/2130581446 >> 172.21.15.45:0/2130581446 conn(0x7fc75806b930 msgr2=0x7fc75806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:27.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.001+0000 7fc75de00640 1 -- 172.21.15.45:0/2130581446 shutdown_connections 2024-04-16T16:14:27.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.001+0000 7fc75de00640 1 -- 172.21.15.45:0/2130581446 wait complete. 2024-04-16T16:14:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.003+0000 7fc75de00640 1 Processor -- start 2024-04-16T16:14:27.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.003+0000 7fc75de00640 1 -- start start 2024-04-16T16:14:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.004+0000 7fc75de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7580c6320 0x7fc7581541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.004+0000 7fc75de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7580c6fc0 0x7fc7580cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.004+0000 7fc75de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc75815cdc0 0x7fc7580cbe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.004+0000 7fc75de00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc7580c73a0 con 0x7fc7580c6fc0 2024-04-16T16:14:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.004+0000 7fc75de00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc75815d270 con 0x7fc75815cdc0 2024-04-16T16:14:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.004+0000 7fc75de00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc7580c7520 con 0x7fc7580c6320 2024-04-16T16:14:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.004+0000 7fc757e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7580c6fc0 0x7fc7580cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:27.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.004+0000 7fc757e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7580c6fc0 0x7fc7580cdb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53374/0 (socket says 172.21.15.45:53374) 2024-04-16T16:14:27.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.004+0000 7fc757e00640 1 -- 172.21.15.45:0/883768051 learned_addr learned my addr 172.21.15.45:0/883768051 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:27.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.005+0000 7fc75ca00640 1 --2- 172.21.15.45:0/883768051 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7580c6320 0x7fc7581541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:27.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.005+0000 7fc75d400640 1 --2- 172.21.15.45:0/883768051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc75815cdc0 0x7fc7580cbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:27.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.005+0000 7fc757e00640 1 -- 172.21.15.45:0/883768051 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7580c6320 msgr2=0x7fc7581541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:27.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.005+0000 7fc757e00640 1 --2- 172.21.15.45:0/883768051 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7580c6320 0x7fc7581541e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:27.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.005+0000 7fc757e00640 1 -- 172.21.15.45:0/883768051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc75815cdc0 msgr2=0x7fc7580cbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:27.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.005+0000 7fc757e00640 1 --2- 172.21.15.45:0/883768051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc75815cdc0 0x7fc7580cbe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:27.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.005+0000 7fc757e00640 1 -- 172.21.15.45:0/883768051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7580ce390 con 0x7fc7580c6fc0 2024-04-16T16:14:27.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.005+0000 7fc75d400640 1 --2- 172.21.15.45:0/883768051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc75815cdc0 0x7fc7580cbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:27.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.005+0000 7fc757e00640 1 --2- 172.21.15.45:0/883768051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7580c6fc0 0x7fc7580cdb30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc74400c7f0 tx=0x7fc74400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:27.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.005+0000 7fc754c00640 1 -- 172.21.15.45:0/883768051 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc744015070 con 0x7fc7580c6fc0 2024-04-16T16:14:27.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.005+0000 7fc754c00640 1 -- 172.21.15.45:0/883768051 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc744004220 con 0x7fc7580c6fc0 2024-04-16T16:14:27.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.005+0000 7fc754c00640 1 -- 172.21.15.45:0/883768051 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc744003610 con 0x7fc7580c6fc0 2024-04-16T16:14:27.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.005+0000 7fc75de00640 1 -- 172.21.15.45:0/883768051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7580ce670 con 0x7fc7580c6fc0 2024-04-16T16:14:27.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.006+0000 7fc75de00640 1 -- 172.21.15.45:0/883768051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7580cec30 con 0x7fc7580c6fc0 2024-04-16T16:14:27.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.007+0000 7fc75de00640 1 -- 172.21.15.45:0/883768051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc758075780 con 0x7fc7580c6fc0 2024-04-16T16:14:27.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.008+0000 7fc754c00640 1 -- 172.21.15.45:0/883768051 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc7440039c0 con 0x7fc7580c6fc0 2024-04-16T16:14:27.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.009+0000 7fc754c00640 1 --2- 172.21.15.45:0/883768051 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc73806c8f0 0x7fc73806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:27.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.009+0000 7fc754c00640 1 -- 172.21.15.45:0/883768051 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc74408be20 con 0x7fc7580c6fc0 2024-04-16T16:14:27.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.009+0000 7fc75ca00640 1 --2- 172.21.15.45:0/883768051 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc73806c8f0 0x7fc73806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.011+0000 7fc75ca00640 1 --2- 172.21.15.45:0/883768051 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc73806c8f0 0x7fc73806edb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc74c036000 tx=0x7fc74c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:27.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.014+0000 7fc754c00640 1 -- 172.21.15.45:0/883768051 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc74405a260 con 0x7fc7580c6fc0 2024-04-16T16:14:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.266+0000 7fc75de00640 1 -- 172.21.15.45:0/883768051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc7580cfd90 con 0x7fc7580c6fc0 2024-04-16T16:14:27.267 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-04-16T16:14:27.268+0000 7fe7a4800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T16:14:27.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.271+0000 7fc754c00640 1 -- 172.21.15.45:0/883768051 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7fc744059b60 con 0x7fc7580c6fc0 2024-04-16T16:14:27.270 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-16T16:14:27.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.275+0000 7fc75de00640 1 -- 172.21.15.45:0/883768051 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc73806c8f0 msgr2=0x7fc73806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:27.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.275+0000 7fc75de00640 1 --2- 172.21.15.45:0/883768051 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc73806c8f0 0x7fc73806edb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc74c036000 tx=0x7fc74c008030 comp rx=0 tx=0).stop 2024-04-16T16:14:27.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.275+0000 7fc75de00640 1 -- 172.21.15.45:0/883768051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7580c6fc0 msgr2=0x7fc7580cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:27.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.275+0000 7fc75de00640 1 --2- 172.21.15.45:0/883768051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7580c6fc0 0x7fc7580cdb30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc74400c7f0 tx=0x7fc74400ccb0 comp rx=0 tx=0).stop 2024-04-16T16:14:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.275+0000 7fc75de00640 1 -- 172.21.15.45:0/883768051 shutdown_connections 2024-04-16T16:14:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.275+0000 7fc75de00640 1 --2- 172.21.15.45:0/883768051 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc73806c8f0 0x7fc73806edb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.275+0000 7fc75de00640 1 --2- 172.21.15.45:0/883768051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc75815cdc0 0x7fc7580cbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.275+0000 7fc75de00640 1 --2- 172.21.15.45:0/883768051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7580c6fc0 0x7fc7580cdb30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.275+0000 7fc75de00640 1 --2- 172.21.15.45:0/883768051 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7580c6320 0x7fc7581541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.275+0000 7fc75de00640 1 -- 172.21.15.45:0/883768051 >> 172.21.15.45:0/883768051 conn(0x7fc75806b930 msgr2=0x7fc75806df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.276+0000 7fc75de00640 1 -- 172.21.15.45:0/883768051 shutdown_connections 2024-04-16T16:14:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.276+0000 7fc75de00640 1 -- 172.21.15.45:0/883768051 wait complete. 2024-04-16T16:14:27.287 INFO:tasks.ceph.mds.c.smithi045.stderr: -8535> 2024-04-16T16:13:29.358+0000 7f992ec00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:14:27.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-16T16:14:27.348 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2024-04-16T16:14:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.497+0000 7fc234c00640 1 Processor -- start 2024-04-16T16:14:27.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.498+0000 7fc234c00640 1 -- start start 2024-04-16T16:14:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.499+0000 7fc234c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc2300b2c70 0x7fc230166bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.499+0000 7fc234c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2300d0ee0 con 0x7fc2300af960 2024-04-16T16:14:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.499+0000 7fc234c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2300b4b20 con 0x7fc2300b5bc0 2024-04-16T16:14:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.499+0000 7fc234c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc2300b4e00 con 0x7fc2300b2c70 2024-04-16T16:14:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.500+0000 7fc22e800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc2300b2c70 0x7fc230166bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.500+0000 7fc22e800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc2300b2c70 0x7fc230166bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53400/0 (socket says 172.21.15.45:53400) 2024-04-16T16:14:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.500+0000 7fc22e800640 1 -- 172.21.15.45:0/3893364344 learned_addr learned my addr 172.21.15.45:0/3893364344 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.500+0000 7fc22d400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc2300af960 0x7fc2300b2560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45356/0 (socket says 172.21.15.45:45356) 2024-04-16T16:14:27.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.500+0000 7fc22e800640 1 -- 172.21.15.45:0/3893364344 >> v1:172.21.15.103:6789/0 conn(0x7fc2300b5bc0 legacy=0x7fc2300d07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:27.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.500+0000 7fc22e800640 1 -- 172.21.15.45:0/3893364344 >> v1:172.21.15.45:6789/0 conn(0x7fc2300af960 legacy=0x7fc2300b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:27.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.500+0000 7fc22ca00640 1 -- 172.21.15.45:0/3893364344 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4219162110 0 0) 0x7fc218001490 con 0x7fc2300af960 2024-04-16T16:14:27.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.500+0000 7fc22e800640 1 -- 172.21.15.45:0/3893364344 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2300b50e0 con 0x7fc2300b2c70 2024-04-16T16:14:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.502+0000 7fc22e800640 1 --2- 172.21.15.45:0/3893364344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2300b2c70 0x7fc230166bc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc22800bb40 tx=0x7fc22802e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820894db7aa9aae0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.502+0000 7fc22ca00640 1 -- 172.21.15.45:0/3893364344 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22803b070 con 0x7fc2300b2c70 2024-04-16T16:14:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.502+0000 7fc22ca00640 1 -- 172.21.15.45:0/3893364344 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc228036440 con 0x7fc2300b2c70 2024-04-16T16:14:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.502+0000 7fc22ca00640 1 -- 172.21.15.45:0/3893364344 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc228036810 con 0x7fc2300b2c70 2024-04-16T16:14:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.502+0000 7fc234c00640 1 -- 172.21.15.45:0/3893364344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2300b2c70 msgr2=0x7fc230166bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.502+0000 7fc234c00640 1 --2- 172.21.15.45:0/3893364344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2300b2c70 0x7fc230166bc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc22800bb40 tx=0x7fc22802e170 comp rx=0 tx=0).stop 2024-04-16T16:14:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.503+0000 7fc234c00640 1 -- 172.21.15.45:0/3893364344 shutdown_connections 2024-04-16T16:14:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.503+0000 7fc234c00640 1 --2- 172.21.15.45:0/3893364344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2300b2c70 0x7fc230166bc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.503+0000 7fc234c00640 1 -- 172.21.15.45:0/3893364344 >> 172.21.15.45:0/3893364344 conn(0x7fc2300b3970 msgr2=0x7fc2300b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.503+0000 7fc234c00640 1 -- 172.21.15.45:0/3893364344 shutdown_connections 2024-04-16T16:14:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.503+0000 7fc234c00640 1 -- 172.21.15.45:0/3893364344 wait complete. 2024-04-16T16:14:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.504+0000 7fc234c00640 1 Processor -- start 2024-04-16T16:14:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.504+0000 7fc234c00640 1 -- start start 2024-04-16T16:14:27.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.505+0000 7fc234c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2300af960 0x7fc23014f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:27.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.505+0000 7fc234c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2300b2c70 0x7fc2301f47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:27.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.505+0000 7fc234c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2300b5bc0 0x7fc2300b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:27.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.505+0000 7fc234c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc23006e440 con 0x7fc2300b5bc0 2024-04-16T16:14:27.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.505+0000 7fc234c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc2300b56a0 con 0x7fc2300af960 2024-04-16T16:14:27.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.505+0000 7fc234c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc2300b0040 con 0x7fc2300b2c70 2024-04-16T16:14:27.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.506+0000 7fc22e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2300b5bc0 0x7fc2300b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:27.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.506+0000 7fc22e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2300b5bc0 0x7fc2300b0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53402/0 (socket says 172.21.15.45:53402) 2024-04-16T16:14:27.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.506+0000 7fc22e800640 1 -- 172.21.15.45:0/3276913174 learned_addr learned my addr 172.21.15.45:0/3276913174 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:27.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.506+0000 7fc22de00640 1 --2- 172.21.15.45:0/3276913174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2300af960 0x7fc23014f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:27.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.506+0000 7fc22e800640 1 -- 172.21.15.45:0/3276913174 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2300b2c70 msgr2=0x7fc2301f47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:27.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.506+0000 7fc22e800640 1 --2- 172.21.15.45:0/3276913174 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2300b2c70 0x7fc2301f47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:27.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.506+0000 7fc22e800640 1 -- 172.21.15.45:0/3276913174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2300af960 msgr2=0x7fc23014f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:27.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.506+0000 7fc22e800640 1 --2- 172.21.15.45:0/3276913174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2300af960 0x7fc23014f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:27.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.506+0000 7fc22e800640 1 -- 172.21.15.45:0/3276913174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2300ada80 con 0x7fc2300b5bc0 2024-04-16T16:14:27.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.506+0000 7fc22de00640 1 --2- 172.21.15.45:0/3276913174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2300af960 0x7fc23014f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:27.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.507+0000 7fc22e800640 1 --2- 172.21.15.45:0/3276913174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2300b5bc0 0x7fc2300b0de0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc22800bb40 tx=0x7fc228004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:27.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.507+0000 7fc226000640 1 -- 172.21.15.45:0/3276913174 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc228046070 con 0x7fc2300b5bc0 2024-04-16T16:14:27.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.507+0000 7fc226000640 1 -- 172.21.15.45:0/3276913174 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc228004230 con 0x7fc2300b5bc0 2024-04-16T16:14:27.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.507+0000 7fc226000640 1 -- 172.21.15.45:0/3276913174 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22803b070 con 0x7fc2300b5bc0 2024-04-16T16:14:27.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.507+0000 7fc234c00640 1 -- 172.21.15.45:0/3276913174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2300add20 con 0x7fc2300b5bc0 2024-04-16T16:14:27.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.507+0000 7fc234c00640 1 -- 172.21.15.45:0/3276913174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2300ae2e0 con 0x7fc2300b5bc0 2024-04-16T16:14:27.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.508+0000 7fc234c00640 1 -- 172.21.15.45:0/3276913174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc23015b0d0 con 0x7fc2300b5bc0 2024-04-16T16:14:27.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.509+0000 7fc226000640 1 -- 172.21.15.45:0/3276913174 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc228034710 con 0x7fc2300b5bc0 2024-04-16T16:14:27.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.510+0000 7fc226000640 1 --2- 172.21.15.45:0/3276913174 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc20c06c8f0 0x7fc20c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:27.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.510+0000 7fc226000640 1 -- 172.21.15.45:0/3276913174 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc2280addf0 con 0x7fc2300b5bc0 2024-04-16T16:14:27.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.511+0000 7fc22de00640 1 --2- 172.21.15.45:0/3276913174 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc20c06c8f0 0x7fc20c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:27.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.512+0000 7fc22de00640 1 --2- 172.21.15.45:0/3276913174 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc20c06c8f0 0x7fc20c06edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc2200015e0 tx=0x7fc220009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.515+0000 7fc226000640 1 -- 172.21.15.45:0/3276913174 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc23015b0d0 con 0x7fc2300b5bc0 2024-04-16T16:14:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:27.770+0000 7fc234c00640 1 -- 172.21.15.45:0/3276913174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fc230075780 con 0x7fc2300b5bc0 2024-04-16T16:14:28.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.275+0000 7fc226000640 1 -- 172.21.15.45:0/3276913174 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7fc22807c140 con 0x7fc2300b5bc0 2024-04-16T16:14:28.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.279+0000 7fc234c00640 1 -- 172.21.15.45:0/3276913174 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc20c06c8f0 msgr2=0x7fc20c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:28.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.279+0000 7fc234c00640 1 --2- 172.21.15.45:0/3276913174 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc20c06c8f0 0x7fc20c06edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc2200015e0 tx=0x7fc220009040 comp rx=0 tx=0).stop 2024-04-16T16:14:28.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.279+0000 7fc234c00640 1 -- 172.21.15.45:0/3276913174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2300b5bc0 msgr2=0x7fc2300b0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:28.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.279+0000 7fc234c00640 1 --2- 172.21.15.45:0/3276913174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2300b5bc0 0x7fc2300b0de0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc22800bb40 tx=0x7fc228004730 comp rx=0 tx=0).stop 2024-04-16T16:14:28.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.279+0000 7fc234c00640 1 -- 172.21.15.45:0/3276913174 shutdown_connections 2024-04-16T16:14:28.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.279+0000 7fc234c00640 1 --2- 172.21.15.45:0/3276913174 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc20c06c8f0 0x7fc20c06edb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:28.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.279+0000 7fc234c00640 1 --2- 172.21.15.45:0/3276913174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2300b5bc0 0x7fc2300b0de0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:28.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.279+0000 7fc234c00640 1 --2- 172.21.15.45:0/3276913174 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2300b2c70 0x7fc2301f47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:28.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.279+0000 7fc234c00640 1 --2- 172.21.15.45:0/3276913174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2300af960 0x7fc23014f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:28.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.279+0000 7fc234c00640 1 -- 172.21.15.45:0/3276913174 >> 172.21.15.45:0/3276913174 conn(0x7fc2300b3970 msgr2=0x7fc2300ce4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:28.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.279+0000 7fc234c00640 1 -- 172.21.15.45:0/3276913174 shutdown_connections 2024-04-16T16:14:28.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.279+0000 7fc234c00640 1 -- 172.21.15.45:0/3276913174 wait complete. 2024-04-16T16:14:28.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-16T16:14:28.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.529+0000 7f554b200640 1 Processor -- start 2024-04-16T16:14:28.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.529+0000 7f554b200640 1 -- start start 2024-04-16T16:14:28.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.530+0000 7f554b200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f554c146180 0x7f554c06e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:28.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.530+0000 7f554b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f554c072650 0x7f554c15d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:28.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.530+0000 7f554b200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f554c072c70 con 0x7f554c06e970 2024-04-16T16:14:28.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.530+0000 7f554b200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f554c15cc90 con 0x7f554c146180 2024-04-16T16:14:28.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.530+0000 7f554b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f554c15d520 con 0x7f554c072650 2024-04-16T16:14:28.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.531+0000 7f5549e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f554c146180 0x7f554c06e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:28.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.531+0000 7f5549e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f554c146180 0x7f554c06e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53412/0 (socket says 172.21.15.45:53412) 2024-04-16T16:14:28.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.531+0000 7f5549e00640 1 -- 172.21.15.45:0/3988234986 learned_addr learned my addr 172.21.15.45:0/3988234986 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:28.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.531+0000 7f5549e00640 1 -- 172.21.15.45:0/3988234986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f554c072650 msgr2=0x7f554c15d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:28.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.531+0000 7f554a800640 1 --2- 172.21.15.45:0/3988234986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f554c072650 0x7f554c15d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:28.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.531+0000 7f5549e00640 1 --2- 172.21.15.45:0/3988234986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f554c072650 0x7f554c15d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:28.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.531+0000 7f5549e00640 1 -- 172.21.15.45:0/3988234986 >> v1:172.21.15.45:6789/0 conn(0x7f554c06e970 legacy=0x7f554c071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:28.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.531+0000 7f5549e00640 1 -- 172.21.15.45:0/3988234986 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f554c15ced0 con 0x7f554c146180 2024-04-16T16:14:28.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.531+0000 7f5548a00640 1 -- 172.21.15.45:0/3988234986 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 331730788 0 0) 0x7f554c072c70 con 0x7f554c06e970 2024-04-16T16:14:28.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.531+0000 7f554a800640 1 --2- 172.21.15.45:0/3988234986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f554c072650 0x7f554c15d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.533+0000 7f5549e00640 1 --2- 172.21.15.45:0/3988234986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f554c146180 0x7f554c06e430 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5538009870 tx=0x7f553802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4414a900c0756813 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.533+0000 7f5548a00640 1 -- 172.21.15.45:0/3988234986 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5538035020 con 0x7f554c146180 2024-04-16T16:14:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.533+0000 7f5548a00640 1 -- 172.21.15.45:0/3988234986 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5538031640 con 0x7f554c146180 2024-04-16T16:14:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.533+0000 7f5548a00640 1 -- 172.21.15.45:0/3988234986 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553802d040 con 0x7f554c146180 2024-04-16T16:14:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.533+0000 7f554b200640 1 -- 172.21.15.45:0/3988234986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f554c146180 msgr2=0x7f554c06e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.533+0000 7f554b200640 1 --2- 172.21.15.45:0/3988234986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f554c146180 0x7f554c06e430 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5538009870 tx=0x7f553802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:14:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.534+0000 7f554b200640 1 -- 172.21.15.45:0/3988234986 shutdown_connections 2024-04-16T16:14:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.534+0000 7f554b200640 1 --2- 172.21.15.45:0/3988234986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f554c072650 0x7f554c15d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.534+0000 7f554b200640 1 --2- 172.21.15.45:0/3988234986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f554c146180 0x7f554c06e430 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.534+0000 7f554b200640 1 -- 172.21.15.45:0/3988234986 >> 172.21.15.45:0/3988234986 conn(0x7f554c0b3bb0 msgr2=0x7f554c0b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.534+0000 7f554b200640 1 -- 172.21.15.45:0/3988234986 shutdown_connections 2024-04-16T16:14:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.534+0000 7f554b200640 1 -- 172.21.15.45:0/3988234986 wait complete. 2024-04-16T16:14:28.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.535+0000 7f554b200640 1 Processor -- start 2024-04-16T16:14:28.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.536+0000 7f554b200640 1 -- start start 2024-04-16T16:14:28.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.536+0000 7f554b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f554c06e970 0x7f554c15c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:28.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.537+0000 7f554b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f554c072650 0x7f554c0ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:28.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.537+0000 7f554b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f554c146180 0x7f554c0c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:28.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.537+0000 7f554b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f554c15e070 con 0x7f554c06e970 2024-04-16T16:14:28.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.537+0000 7f554b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f554c15def0 con 0x7f554c146180 2024-04-16T16:14:28.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.537+0000 7f554b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f554c15e1f0 con 0x7f554c072650 2024-04-16T16:14:28.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.537+0000 7f5549e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f554c06e970 0x7f554c15c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:28.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.537+0000 7f5549e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f554c06e970 0x7f554c15c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53418/0 (socket says 172.21.15.45:53418) 2024-04-16T16:14:28.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.537+0000 7f5549e00640 1 -- 172.21.15.45:0/2399824988 learned_addr learned my addr 172.21.15.45:0/2399824988 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:28.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.537+0000 7f5549e00640 1 -- 172.21.15.45:0/2399824988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f554c072650 msgr2=0x7f554c0ada20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.537+0000 7f5549e00640 1 --2- 172.21.15.45:0/2399824988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f554c072650 0x7f554c0ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.537+0000 7f5549e00640 1 -- 172.21.15.45:0/2399824988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f554c146180 msgr2=0x7f554c0c64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.537+0000 7f5549e00640 1 --2- 172.21.15.45:0/2399824988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f554c146180 0x7f554c0c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.537+0000 7f5549e00640 1 -- 172.21.15.45:0/2399824988 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f554c15c790 con 0x7f554c06e970 2024-04-16T16:14:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.538+0000 7f5549e00640 1 --2- 172.21.15.45:0/2399824988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f554c06e970 0x7f554c15c250 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5538000c00 tx=0x7f55380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.538+0000 7f5542000640 1 -- 172.21.15.45:0/2399824988 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5538040070 con 0x7f554c06e970 2024-04-16T16:14:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.538+0000 7f5542000640 1 -- 172.21.15.45:0/2399824988 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5538039bf0 con 0x7f554c06e970 2024-04-16T16:14:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.538+0000 7f5542000640 1 -- 172.21.15.45:0/2399824988 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5538035040 con 0x7f554c06e970 2024-04-16T16:14:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.538+0000 7f554b200640 1 -- 172.21.15.45:0/2399824988 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f554c0af3c0 con 0x7f554c06e970 2024-04-16T16:14:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.538+0000 7f554b200640 1 -- 172.21.15.45:0/2399824988 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f554c0ae600 con 0x7f554c06e970 2024-04-16T16:14:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.540+0000 7f554b200640 1 -- 172.21.15.45:0/2399824988 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f554c075780 con 0x7f554c06e970 2024-04-16T16:14:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.540+0000 7f5542000640 1 -- 172.21.15.45:0/2399824988 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f553804b060 con 0x7f554c06e970 2024-04-16T16:14:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.541+0000 7f5542000640 1 --2- 172.21.15.45:0/2399824988 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f552c06c8f0 0x7f552c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.541+0000 7f5542000640 1 -- 172.21.15.45:0/2399824988 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f55380ad600 con 0x7f554c06e970 2024-04-16T16:14:28.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.542+0000 7f5549400640 1 --2- 172.21.15.45:0/2399824988 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f552c06c8f0 0x7f552c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:28.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.543+0000 7f5549400640 1 --2- 172.21.15.45:0/2399824988 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f552c06c8f0 0x7f552c06edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f553c000f60 tx=0x7f553c007910 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:28.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.547+0000 7f5542000640 1 -- 172.21.15.45:0/2399824988 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f553802f300 con 0x7f554c06e970 2024-04-16T16:14:28.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:28.752+0000 7f554b200640 1 -- 172.21.15.45:0/2399824988 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f554c06cf70 con 0x7f554c06e970 2024-04-16T16:14:29.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.481+0000 7f5542000640 1 -- 172.21.15.45:0/2399824988 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) ==== 181+0+0 (secure 0 0 0) 0x7f553807b960 con 0x7f554c06e970 2024-04-16T16:14:29.480 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2024-04-16T16:14:29.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.485+0000 7f554b200640 1 -- 172.21.15.45:0/2399824988 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f552c06c8f0 msgr2=0x7f552c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:29.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.485+0000 7f554b200640 1 --2- 172.21.15.45:0/2399824988 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f552c06c8f0 0x7f552c06edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f553c000f60 tx=0x7f553c007910 comp rx=0 tx=0).stop 2024-04-16T16:14:29.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.485+0000 7f554b200640 1 -- 172.21.15.45:0/2399824988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f554c06e970 msgr2=0x7f554c15c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:29.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.485+0000 7f554b200640 1 --2- 172.21.15.45:0/2399824988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f554c06e970 0x7f554c15c250 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5538000c00 tx=0x7f55380046c0 comp rx=0 tx=0).stop 2024-04-16T16:14:29.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.487+0000 7f554b200640 1 -- 172.21.15.45:0/2399824988 shutdown_connections 2024-04-16T16:14:29.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.487+0000 7f554b200640 1 --2- 172.21.15.45:0/2399824988 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f552c06c8f0 0x7f552c06edb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:29.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.487+0000 7f554b200640 1 --2- 172.21.15.45:0/2399824988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f554c146180 0x7f554c0c64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:29.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.487+0000 7f554b200640 1 --2- 172.21.15.45:0/2399824988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f554c072650 0x7f554c0ada20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:29.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.487+0000 7f554b200640 1 --2- 172.21.15.45:0/2399824988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f554c06e970 0x7f554c15c250 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:29.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.487+0000 7f554b200640 1 -- 172.21.15.45:0/2399824988 >> 172.21.15.45:0/2399824988 conn(0x7f554c0b3bb0 msgr2=0x7f554c0ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:29.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.487+0000 7f554b200640 1 -- 172.21.15.45:0/2399824988 shutdown_connections 2024-04-16T16:14:29.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.487+0000 7f554b200640 1 -- 172.21.15.45:0/2399824988 wait complete. 2024-04-16T16:14:29.510 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-16T16:14:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.726+0000 7f865f800640 1 Processor -- start 2024-04-16T16:14:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.726+0000 7f865f800640 1 -- start start 2024-04-16T16:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.728+0000 7f865f800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86580b5a30 0x7f86581625e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.728+0000 7f865f800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86580b5e10 con 0x7f865806e290 2024-04-16T16:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.728+0000 7f865f800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f865815ff70 con 0x7f86580d1030 2024-04-16T16:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.728+0000 7f865f800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f865815f3d0 con 0x7f86580b5a30 2024-04-16T16:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.728+0000 7f865d800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f865806e290 0x7f86580b4920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45374/0 (socket says 172.21.15.45:45374) 2024-04-16T16:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.728+0000 7f865d800640 1 -- 172.21.15.45:0/1485823870 learned_addr learned my addr 172.21.15.45:0/1485823870 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.729+0000 7f865ce00640 1 --2- 172.21.15.45:0/1485823870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86580b5a30 0x7f86581625e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.729+0000 7f8657400640 1 -- 172.21.15.45:0/1485823870 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1931772777 0 0) 0x7f86580b5e10 con 0x7f865806e290 2024-04-16T16:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.729+0000 7f8657400640 1 -- 172.21.15.45:0/1485823870 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f863c003520 con 0x7f865806e290 2024-04-16T16:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.729+0000 7f8657400640 1 -- 172.21.15.45:0/1485823870 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 954865350 0 0) 0x7f863c003520 con 0x7f865806e290 2024-04-16T16:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.729+0000 7f8657400640 1 -- 172.21.15.45:0/1485823870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86580b5a30 msgr2=0x7f86581625e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.729+0000 7f8657400640 1 --2- 172.21.15.45:0/1485823870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86580b5a30 0x7f86581625e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.729+0000 7f8657400640 1 -- 172.21.15.45:0/1485823870 >> v1:172.21.15.103:6789/0 conn(0x7f86580d1030 legacy=0x7f865815f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.729+0000 7f8657400640 1 -- 172.21.15.45:0/1485823870 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8658162e70 con 0x7f865806e290 2024-04-16T16:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.729+0000 7f8657400640 1 -- 172.21.15.45:0/1485823870 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f86500019f0 con 0x7f865806e290 2024-04-16T16:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.729+0000 7f8657400640 1 -- 172.21.15.45:0/1485823870 >> v1:172.21.15.45:6789/0 conn(0x7f865806e290 legacy=0x7f86580b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.729+0000 7f8657400640 1 --2- 172.21.15.45:0/1485823870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86580b5a30 0x7f863c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.729+0000 7f8657400640 1 --2- 172.21.15.45:0/1485823870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c003f60 0x7f863c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.730+0000 7f8657400640 1 --2- 172.21.15.45:0/1485823870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c004d40 0x7f863c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.730+0000 7f8657400640 1 -- 172.21.15.45:0/1485823870 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f865815f3d0 con 0x7f863c003f60 2024-04-16T16:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.730+0000 7f8657400640 1 -- 172.21.15.45:0/1485823870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f863c00baa0 con 0x7f863c004d40 2024-04-16T16:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.730+0000 7f8657400640 1 -- 172.21.15.45:0/1485823870 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f863c00bca0 con 0x7f86580b5a30 2024-04-16T16:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.730+0000 7f8657e00640 1 --2- 172.21.15.45:0/1485823870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c003f60 0x7f863c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.730+0000 7f865d800640 1 --2- 172.21.15.45:0/1485823870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c004d40 0x7f863c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.730+0000 7f865ce00640 1 --2- 172.21.15.45:0/1485823870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86580b5a30 0x7f863c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.730+0000 7f8657e00640 1 -- 172.21.15.45:0/1485823870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86580b5a30 msgr2=0x7f863c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.730+0000 7f8657e00640 1 --2- 172.21.15.45:0/1485823870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86580b5a30 0x7f863c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.730+0000 7f8657e00640 1 -- 172.21.15.45:0/1485823870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c004d40 msgr2=0x7f863c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.730+0000 7f8657e00640 1 --2- 172.21.15.45:0/1485823870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c004d40 0x7f863c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.730+0000 7f8657e00640 1 -- 172.21.15.45:0/1485823870 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f863c003520 con 0x7f863c003f60 2024-04-16T16:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.730+0000 7f865ce00640 1 --2- 172.21.15.45:0/1485823870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86580b5a30 0x7f863c0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.732+0000 7f8657e00640 1 --2- 172.21.15.45:0/1485823870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c003f60 0x7f863c004800 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8644001750 tx=0x7f863c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d65b1dce96ad6df server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.732+0000 7f8657400640 1 -- 172.21.15.45:0/1485823870 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644001370 con 0x7f863c003f60 2024-04-16T16:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.732+0000 7f8657400640 1 -- 172.21.15.45:0/1485823870 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86440052d0 con 0x7f863c003f60 2024-04-16T16:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.732+0000 7f8657400640 1 -- 172.21.15.45:0/1485823870 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644003830 con 0x7f863c003f60 2024-04-16T16:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.732+0000 7f865f800640 1 -- 172.21.15.45:0/1485823870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c003f60 msgr2=0x7f863c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.732+0000 7f865f800640 1 --2- 172.21.15.45:0/1485823870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c003f60 0x7f863c004800 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8644001750 tx=0x7f863c005830 comp rx=0 tx=0).stop 2024-04-16T16:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.732+0000 7f865d800640 1 -- 172.21.15.45:0/1485823870 reap_dead start 2024-04-16T16:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.733+0000 7f865f800640 1 -- 172.21.15.45:0/1485823870 shutdown_connections 2024-04-16T16:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.733+0000 7f865f800640 1 -- 172.21.15.45:0/1485823870 >> 172.21.15.45:0/1485823870 conn(0x7f86580b3970 msgr2=0x7f86580b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.733+0000 7f865f800640 1 -- 172.21.15.45:0/1485823870 shutdown_connections 2024-04-16T16:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.733+0000 7f865f800640 1 -- 172.21.15.45:0/1485823870 wait complete. 2024-04-16T16:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.734+0000 7f865f800640 1 Processor -- start 2024-04-16T16:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.735+0000 7f865f800640 1 -- start start 2024-04-16T16:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.736+0000 7f865f800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c003f60 0x7f8658158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.736+0000 7f865f800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f865815d0c0 0x7f865815d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.736+0000 7f865f800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86580ce440 0x7f86580ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.736+0000 7f865f800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f865815eff0 con 0x7f863c003f60 2024-04-16T16:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.736+0000 7f865f800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8658163430 con 0x7f865815d0c0 2024-04-16T16:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.736+0000 7f865f800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f86580b5030 con 0x7f86580ce440 2024-04-16T16:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.736+0000 7f865ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c003f60 0x7f8658158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.736+0000 7f865ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c003f60 0x7f8658158460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53440/0 (socket says 172.21.15.45:53440) 2024-04-16T16:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.736+0000 7f865ce00640 1 -- 172.21.15.45:0/2707581411 learned_addr learned my addr 172.21.15.45:0/2707581411 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.736+0000 7f865ce00640 1 -- 172.21.15.45:0/2707581411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86580ce440 msgr2=0x7f86580ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.736+0000 7f865ce00640 1 --2- 172.21.15.45:0/2707581411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86580ce440 0x7f86580ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.736+0000 7f8657e00640 1 --2- 172.21.15.45:0/2707581411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f865815d0c0 0x7f865815d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.736+0000 7f865ce00640 1 -- 172.21.15.45:0/2707581411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f865815d0c0 msgr2=0x7f865815d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.736+0000 7f865ce00640 1 --2- 172.21.15.45:0/2707581411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f865815d0c0 0x7f865815d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.736+0000 7f865ce00640 1 -- 172.21.15.45:0/2707581411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865815b470 con 0x7f863c003f60 2024-04-16T16:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.737+0000 7f8657e00640 1 --2- 172.21.15.45:0/2707581411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f865815d0c0 0x7f865815d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.737+0000 7f865ce00640 1 --2- 172.21.15.45:0/2707581411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c003f60 0x7f8658158460 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f86480108b0 tx=0x7f8648010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.737+0000 7f8654c00640 1 -- 172.21.15.45:0/2707581411 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864800f040 con 0x7f863c003f60 2024-04-16T16:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.737+0000 7f8654c00640 1 -- 172.21.15.45:0/2707581411 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f864800eb60 con 0x7f863c003f60 2024-04-16T16:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.737+0000 7f8654c00640 1 -- 172.21.15.45:0/2707581411 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864801c400 con 0x7f863c003f60 2024-04-16T16:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.737+0000 7f865f800640 1 -- 172.21.15.45:0/2707581411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f865815b750 con 0x7f863c003f60 2024-04-16T16:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.737+0000 7f865f800640 1 -- 172.21.15.45:0/2707581411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f865815bd10 con 0x7f863c003f60 2024-04-16T16:14:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.739+0000 7f865f800640 1 -- 172.21.15.45:0/2707581411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8658075780 con 0x7f863c003f60 2024-04-16T16:14:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.739+0000 7f8654c00640 1 -- 172.21.15.45:0/2707581411 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f864801c5a0 con 0x7f863c003f60 2024-04-16T16:14:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.740+0000 7f8654c00640 1 --2- 172.21.15.45:0/2707581411 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f863806c8f0 0x7f863806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.740+0000 7f8654c00640 1 -- 172.21.15.45:0/2707581411 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5473+0+0 (secure 0 0 0) 0x7f864808c210 con 0x7f863c003f60 2024-04-16T16:14:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.741+0000 7f8657e00640 1 --2- 172.21.15.45:0/2707581411 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f863806c8f0 0x7f863806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.742+0000 7f8657e00640 1 --2- 172.21.15.45:0/2707581411 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f863806c8f0 0x7f863806edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8644033000 tx=0x7f8644009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:29.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.745+0000 7f8654c00640 1 -- 172.21.15.45:0/2707581411 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8648018030 con 0x7f863c003f60 2024-04-16T16:14:29.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:29.966+0000 7f865f800640 1 -- 172.21.15.45:0/2707581411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f86581f4650 con 0x7f863c003f60 2024-04-16T16:14:30.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.494+0000 7f8654c00640 1 -- 172.21.15.45:0/2707581411 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) ==== 169+0+0 (secure 0 0 0) 0x7f864805a030 con 0x7f863c003f60 2024-04-16T16:14:30.494 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2024-04-16T16:14:30.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.497+0000 7f865f800640 1 -- 172.21.15.45:0/2707581411 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f863806c8f0 msgr2=0x7f863806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:30.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.497+0000 7f865f800640 1 --2- 172.21.15.45:0/2707581411 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f863806c8f0 0x7f863806edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8644033000 tx=0x7f8644009030 comp rx=0 tx=0).stop 2024-04-16T16:14:30.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.497+0000 7f865f800640 1 -- 172.21.15.45:0/2707581411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c003f60 msgr2=0x7f8658158460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:30.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.497+0000 7f865f800640 1 --2- 172.21.15.45:0/2707581411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c003f60 0x7f8658158460 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f86480108b0 tx=0x7f8648010d70 comp rx=0 tx=0).stop 2024-04-16T16:14:30.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.498+0000 7f865f800640 1 -- 172.21.15.45:0/2707581411 shutdown_connections 2024-04-16T16:14:30.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.498+0000 7f865f800640 1 --2- 172.21.15.45:0/2707581411 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f863806c8f0 0x7f863806edb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:30.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.498+0000 7f865f800640 1 --2- 172.21.15.45:0/2707581411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86580ce440 0x7f86580ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:30.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.498+0000 7f865f800640 1 --2- 172.21.15.45:0/2707581411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f865815d0c0 0x7f865815d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:30.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.498+0000 7f865f800640 1 --2- 172.21.15.45:0/2707581411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c003f60 0x7f8658158460 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:30.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.498+0000 7f865f800640 1 -- 172.21.15.45:0/2707581411 >> 172.21.15.45:0/2707581411 conn(0x7f86580b3970 msgr2=0x7f86580aecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:30.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.498+0000 7f865f800640 1 -- 172.21.15.45:0/2707581411 shutdown_connections 2024-04-16T16:14:30.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.498+0000 7f865f800640 1 -- 172.21.15.45:0/2707581411 wait complete. 2024-04-16T16:14:30.523 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-16T16:14:30.523 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-16T16:14:30.523 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-16T16:14:30.524 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:14:30.524 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-16T16:14:30.524 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-16T16:14:30.524 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-16T16:14:30.524 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:14:30.524 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-16T16:14:30.524 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-16T16:14:30.525 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-16T16:14:30.525 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:14:30.525 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-16T16:14:30.525 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-16T16:14:30.525 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-16T16:14:30.525 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:14:30.541 INFO:tasks.ceph.mds.a:Stopped 2024-04-16T16:14:30.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-16T16:14:30.542 INFO:tasks.ceph.mds.c:Stopped 2024-04-16T16:14:30.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-16T16:14:30.550 INFO:tasks.ceph.mds.b:Stopped 2024-04-16T16:14:30.550 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-16T16:14:30.551 INFO:tasks.ceph.mds.d:Stopped 2024-04-16T16:14:30.551 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-16T16:14:30.589 INFO:tasks.ceph.mds.a:Started 2024-04-16T16:14:30.590 DEBUG:teuthology.parallel:result is None 2024-04-16T16:14:30.596 INFO:tasks.ceph.mds.c:Started 2024-04-16T16:14:30.596 DEBUG:teuthology.parallel:result is None 2024-04-16T16:14:30.597 INFO:tasks.ceph.mds.b:Started 2024-04-16T16:14:30.598 DEBUG:teuthology.parallel:result is None 2024-04-16T16:14:30.603 INFO:tasks.ceph.mds.d:Started 2024-04-16T16:14:30.603 DEBUG:teuthology.parallel:result is None 2024-04-16T16:14:30.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-16T16:14:30.750 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2024-04-16T16:14:30.751 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2024-04-16T16:14:30.762 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-16T16:14:30.762 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-16T16:14:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.794+0000 7f5bcac00640 1 Processor -- start 2024-04-16T16:14:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.795+0000 7f5bcac00640 1 -- start start 2024-04-16T16:14:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.796+0000 7f5bcac00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5bc4146220 0x7f5bc40cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.796+0000 7f5bcac00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bc4146600 con 0x7f5bc40cd930 2024-04-16T16:14:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.796+0000 7f5bcac00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bc40cde00 con 0x7f5bc4072180 2024-04-16T16:14:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.796+0000 7f5bcac00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5bc406f840 con 0x7f5bc4146220 2024-04-16T16:14:30.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.796+0000 7f5bc8c00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5bc40cd930 0x7f5bc406df00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45398/0 (socket says 172.21.15.45:45398) 2024-04-16T16:14:30.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.796+0000 7f5bc8c00640 1 -- 172.21.15.45:0/3355016399 learned_addr learned my addr 172.21.15.45:0/3355016399 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:30.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.796+0000 7f5bc3e00640 1 --2- 172.21.15.45:0/3355016399 >> v2:172.21.15.103:3300/0 conn(0x7f5bc4146220 0x7f5bc40cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:30.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.797+0000 7f5bc3400640 1 -- 172.21.15.45:0/3355016399 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4019595920 0 0) 0x7f5bc4146600 con 0x7f5bc40cd930 2024-04-16T16:14:30.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.797+0000 7f5bc3400640 1 -- 172.21.15.45:0/3355016399 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ba8003520 con 0x7f5bc40cd930 2024-04-16T16:14:30.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.797+0000 7f5bc3400640 1 -- 172.21.15.45:0/3355016399 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2706451402 0 0) 0x7f5bc40cde00 con 0x7f5bc4072180 2024-04-16T16:14:30.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.797+0000 7f5bc3400640 1 -- 172.21.15.45:0/3355016399 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bc4146600 con 0x7f5bc4072180 2024-04-16T16:14:30.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.797+0000 7f5bc3e00640 1 -- 172.21.15.45:0/3355016399 >> v1:172.21.15.103:6789/0 conn(0x7f5bc4072180 legacy=0x7f5bc40cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:30.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.797+0000 7f5bc3400640 1 -- 172.21.15.45:0/3355016399 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1004523129 0 0) 0x7f5ba8003520 con 0x7f5bc40cd930 2024-04-16T16:14:30.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.797+0000 7f5bc3e00640 1 -- 172.21.15.45:0/3355016399 >> v1:172.21.15.45:6789/0 conn(0x7f5bc40cd930 legacy=0x7f5bc406df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.797+0000 7f5bc3e00640 1 -- 172.21.15.45:0/3355016399 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc40ce630 con 0x7f5bc4146220 2024-04-16T16:14:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.798+0000 7f5bc3e00640 1 --2- 172.21.15.45:0/3355016399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc4146220 0x7f5bc40cf110 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40097a0 tx=0x7f5bb402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5034c44e2fdcfc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.798+0000 7f5bc3400640 1 -- 172.21.15.45:0/3355016399 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4038070 con 0x7f5bc4146220 2024-04-16T16:14:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.798+0000 7f5bc3400640 1 -- 172.21.15.45:0/3355016399 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bb4031500 con 0x7f5bc4146220 2024-04-16T16:14:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.798+0000 7f5bc3400640 1 -- 172.21.15.45:0/3355016399 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc001a10 con 0x7f5bc4146220 2024-04-16T16:14:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.799+0000 7f5bcac00640 1 -- 172.21.15.45:0/3355016399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc4146220 msgr2=0x7f5bc40cf110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.799+0000 7f5bcac00640 1 --2- 172.21.15.45:0/3355016399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc4146220 0x7f5bc40cf110 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40097a0 tx=0x7f5bb402b820 comp rx=0 tx=0).stop 2024-04-16T16:14:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.799+0000 7f5bcac00640 1 -- 172.21.15.45:0/3355016399 shutdown_connections 2024-04-16T16:14:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.799+0000 7f5bcac00640 1 --2- 172.21.15.45:0/3355016399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc4146220 0x7f5bc40cf110 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.799+0000 7f5bcac00640 1 -- 172.21.15.45:0/3355016399 >> 172.21.15.45:0/3355016399 conn(0x7f5bc406b930 msgr2=0x7f5bc406bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:30.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.799+0000 7f5bcac00640 1 -- 172.21.15.45:0/3355016399 shutdown_connections 2024-04-16T16:14:30.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.799+0000 7f5bcac00640 1 -- 172.21.15.45:0/3355016399 wait complete. 2024-04-16T16:14:30.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.801+0000 7f5bcac00640 1 Processor -- start 2024-04-16T16:14:30.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.801+0000 7f5bcac00640 1 -- start start 2024-04-16T16:14:30.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.802+0000 7f5bcac00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc4072180 0x7f5bc40a9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:30.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.802+0000 7f5bcac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc40cd930 0x7f5bc40ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:30.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.802+0000 7f5bcac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc4146220 0x7f5bc413cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:30.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.802+0000 7f5bcac00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc40cebf0 con 0x7f5bc4072180 2024-04-16T16:14:30.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.802+0000 7f5bcac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc40ce0e0 con 0x7f5bc4146220 2024-04-16T16:14:30.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.802+0000 7f5bcac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc40725f0 con 0x7f5bc40cd930 2024-04-16T16:14:30.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.802+0000 7f5bc3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc4072180 0x7f5bc40a9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:30.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.802+0000 7f5bc3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc4072180 0x7f5bc40a9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53486/0 (socket says 172.21.15.45:53486) 2024-04-16T16:14:30.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.802+0000 7f5bc3e00640 1 -- 172.21.15.45:0/3965925339 learned_addr learned my addr 172.21.15.45:0/3965925339 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:30.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.803+0000 7f5bbbe00640 1 --2- 172.21.15.45:0/3965925339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc40cd930 0x7f5bc40ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:30.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.803+0000 7f5bc8c00640 1 --2- 172.21.15.45:0/3965925339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc4146220 0x7f5bc413cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:30.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.803+0000 7f5bc3e00640 1 -- 172.21.15.45:0/3965925339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc40cd930 msgr2=0x7f5bc40ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:30.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.803+0000 7f5bc3e00640 1 --2- 172.21.15.45:0/3965925339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc40cd930 0x7f5bc40ab8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:30.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.803+0000 7f5bc3e00640 1 -- 172.21.15.45:0/3965925339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc4146220 msgr2=0x7f5bc413cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:30.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.803+0000 7f5bc3e00640 1 --2- 172.21.15.45:0/3965925339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc4146220 0x7f5bc413cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:30.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.803+0000 7f5bc3e00640 1 -- 172.21.15.45:0/3965925339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc40a9c90 con 0x7f5bc4072180 2024-04-16T16:14:30.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.803+0000 7f5bc3e00640 1 --2- 172.21.15.45:0/3965925339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc4072180 0x7f5bc40a9670 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40098c0 tx=0x7f5bb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:30.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.804+0000 7f5bc0c00640 1 -- 172.21.15.45:0/3965925339 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4043070 con 0x7f5bc4072180 2024-04-16T16:14:30.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.804+0000 7f5bc0c00640 1 -- 172.21.15.45:0/3965925339 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bb403f870 con 0x7f5bc4072180 2024-04-16T16:14:30.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.804+0000 7f5bc0c00640 1 -- 172.21.15.45:0/3965925339 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4038040 con 0x7f5bc4072180 2024-04-16T16:14:30.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.804+0000 7f5bcac00640 1 -- 172.21.15.45:0/3965925339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc40a9f70 con 0x7f5bc4072180 2024-04-16T16:14:30.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.804+0000 7f5bcac00640 1 -- 172.21.15.45:0/3965925339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc40aa530 con 0x7f5bc4072180 2024-04-16T16:14:30.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.804+0000 7f5bcac00640 1 -- 172.21.15.45:0/3965925339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bc4068d50 con 0x7f5bc4072180 2024-04-16T16:14:30.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.805+0000 7f5bc0c00640 1 -- 172.21.15.45:0/3965925339 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5bb4046070 con 0x7f5bc4072180 2024-04-16T16:14:30.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.806+0000 7f5bc0c00640 1 --2- 172.21.15.45:0/3965925339 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ba006c8f0 0x7f5ba006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:30.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.806+0000 7f5bc0c00640 1 -- 172.21.15.45:0/3965925339 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4943+0+0 (secure 0 0 0) 0x7f5bb40ad450 con 0x7f5bc4072180 2024-04-16T16:14:30.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.807+0000 7f5bbbe00640 1 --2- 172.21.15.45:0/3965925339 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ba006c8f0 0x7f5ba006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:30.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.807+0000 7f5bbbe00640 1 --2- 172.21.15.45:0/3965925339 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ba006c8f0 0x7f5ba006edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40b4670 tx=0x7f5bac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:30.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:30.811+0000 7f5bc0c00640 1 -- 172.21.15.45:0/3965925339 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5bc4068d50 con 0x7f5bc4072180 2024-04-16T16:14:31.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.045+0000 7f5bcac00640 1 -- 172.21.15.45:0/3965925339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f5bc40abe50 con 0x7f5bc4072180 2024-04-16T16:14:31.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.520+0000 7f5bc0c00640 1 -- 172.21.15.45:0/3965925339 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v28) ==== 97+0+0 (secure 0 0 0) 0x7f5bb4030070 con 0x7f5bc4072180 2024-04-16T16:14:31.521 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2024-04-16T16:14:31.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.525+0000 7f5bcac00640 1 -- 172.21.15.45:0/3965925339 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ba006c8f0 msgr2=0x7f5ba006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:31.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.525+0000 7f5bcac00640 1 --2- 172.21.15.45:0/3965925339 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ba006c8f0 0x7f5ba006edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40b4670 tx=0x7f5bac009040 comp rx=0 tx=0).stop 2024-04-16T16:14:31.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.525+0000 7f5bcac00640 1 -- 172.21.15.45:0/3965925339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc4072180 msgr2=0x7f5bc40a9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:31.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.525+0000 7f5bcac00640 1 --2- 172.21.15.45:0/3965925339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc4072180 0x7f5bc40a9670 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40098c0 tx=0x7f5bb40046c0 comp rx=0 tx=0).stop 2024-04-16T16:14:31.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.526+0000 7f5bcac00640 1 -- 172.21.15.45:0/3965925339 shutdown_connections 2024-04-16T16:14:31.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.526+0000 7f5bcac00640 1 --2- 172.21.15.45:0/3965925339 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ba006c8f0 0x7f5ba006edb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:31.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.526+0000 7f5bcac00640 1 --2- 172.21.15.45:0/3965925339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc4146220 0x7f5bc413cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:31.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.526+0000 7f5bcac00640 1 --2- 172.21.15.45:0/3965925339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc40cd930 0x7f5bc40ab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:31.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.526+0000 7f5bcac00640 1 --2- 172.21.15.45:0/3965925339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc4072180 0x7f5bc40a9670 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:31.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.526+0000 7f5bcac00640 1 -- 172.21.15.45:0/3965925339 >> 172.21.15.45:0/3965925339 conn(0x7f5bc406b930 msgr2=0x7f5bc406df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:31.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.526+0000 7f5bcac00640 1 -- 172.21.15.45:0/3965925339 shutdown_connections 2024-04-16T16:14:31.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.526+0000 7f5bcac00640 1 -- 172.21.15.45:0/3965925339 wait complete. 2024-04-16T16:14:31.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-16T16:14:31.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.751+0000 7fa9cde00640 1 Processor -- start 2024-04-16T16:14:31.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.751+0000 7fa9cde00640 1 -- start start 2024-04-16T16:14:31.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.753+0000 7fa9cde00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9c815ccb0 0x7fa9c806df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:31.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.753+0000 7fa9cde00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9c815d360 con 0x7fa9c806e480 2024-04-16T16:14:31.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.753+0000 7fa9cde00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9c815d640 con 0x7fa9c8146220 2024-04-16T16:14:31.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.753+0000 7fa9cde00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa9c815d130 con 0x7fa9c815ccb0 2024-04-16T16:14:31.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.753+0000 7fa9cd400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa9c806e480 0x7fa9c81608a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45406/0 (socket says 172.21.15.45:45406) 2024-04-16T16:14:31.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.753+0000 7fa9cd400640 1 -- 172.21.15.45:0/2477177224 learned_addr learned my addr 172.21.15.45:0/2477177224 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:31.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7e00640 1 --2- 172.21.15.45:0/2477177224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9c815ccb0 0x7fa9c806df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:31.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7400640 1 -- 172.21.15.45:0/2477177224 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3785241823 0 0) 0x7fa9c815d360 con 0x7fa9c806e480 2024-04-16T16:14:31.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7400640 1 -- 172.21.15.45:0/2477177224 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9b0003520 con 0x7fa9c806e480 2024-04-16T16:14:31.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7400640 1 -- 172.21.15.45:0/2477177224 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2452600083 0 0) 0x7fa9b0003520 con 0x7fa9c806e480 2024-04-16T16:14:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7400640 1 -- 172.21.15.45:0/2477177224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9c815ccb0 msgr2=0x7fa9c806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7400640 1 --2- 172.21.15.45:0/2477177224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9c815ccb0 0x7fa9c806df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7400640 1 -- 172.21.15.45:0/2477177224 >> v1:172.21.15.103:6789/0 conn(0x7fa9c8146220 legacy=0x7fa9c806f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7400640 1 -- 172.21.15.45:0/2477177224 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c815d920 con 0x7fa9c806e480 2024-04-16T16:14:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7400640 1 -- 172.21.15.45:0/2477177224 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fa9c00019f0 con 0x7fa9c806e480 2024-04-16T16:14:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7400640 1 -- 172.21.15.45:0/2477177224 >> v1:172.21.15.45:6789/0 conn(0x7fa9c806e480 legacy=0x7fa9c81608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7400640 1 --2- 172.21.15.45:0/2477177224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9c815ccb0 0x7fa9b00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7400640 1 --2- 172.21.15.45:0/2477177224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9b0004060 0x7fa9b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7400640 1 --2- 172.21.15.45:0/2477177224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b0004d40 0x7fa9b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7400640 1 -- 172.21.15.45:0/2477177224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b000ffe0 con 0x7fa9b0004060 2024-04-16T16:14:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7400640 1 -- 172.21.15.45:0/2477177224 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b0010190 con 0x7fa9b0004d40 2024-04-16T16:14:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.754+0000 7fa9c7400640 1 -- 172.21.15.45:0/2477177224 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b0010390 con 0x7fa9c815ccb0 2024-04-16T16:14:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.755+0000 7fa9c7e00640 1 --2- 172.21.15.45:0/2477177224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9b0004060 0x7fa9b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.755+0000 7fa9cca00640 1 --2- 172.21.15.45:0/2477177224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9c815ccb0 0x7fa9b00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.755+0000 7fa9cd400640 1 --2- 172.21.15.45:0/2477177224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b0004d40 0x7fa9b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:31.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.755+0000 7fa9c7e00640 1 -- 172.21.15.45:0/2477177224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9c815ccb0 msgr2=0x7fa9b00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:31.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.755+0000 7fa9c7e00640 1 --2- 172.21.15.45:0/2477177224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9c815ccb0 0x7fa9b00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:31.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.755+0000 7fa9c7e00640 1 -- 172.21.15.45:0/2477177224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b0004d40 msgr2=0x7fa9b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:31.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.755+0000 7fa9c7e00640 1 --2- 172.21.15.45:0/2477177224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b0004d40 0x7fa9b00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:31.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.755+0000 7fa9c7e00640 1 -- 172.21.15.45:0/2477177224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b0003520 con 0x7fa9b0004060 2024-04-16T16:14:31.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.755+0000 7fa9cd400640 1 --2- 172.21.15.45:0/2477177224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b0004d40 0x7fa9b00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:31.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.756+0000 7fa9c7e00640 1 --2- 172.21.15.45:0/2477177224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9b0004060 0x7fa9b0004800 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa9b400ba30 tx=0x7fa9b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57ad74cd1b5436 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:31.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.757+0000 7fa9c7400640 1 -- 172.21.15.45:0/2477177224 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4036020 con 0x7fa9b0004060 2024-04-16T16:14:31.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.757+0000 7fa9c7400640 1 -- 172.21.15.45:0/2477177224 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9b4032630 con 0x7fa9b0004060 2024-04-16T16:14:31.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.757+0000 7fa9c7400640 1 -- 172.21.15.45:0/2477177224 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b400b600 con 0x7fa9b0004060 2024-04-16T16:14:31.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.757+0000 7fa9cde00640 1 -- 172.21.15.45:0/2477177224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9b0004060 msgr2=0x7fa9b0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:31.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.757+0000 7fa9cde00640 1 --2- 172.21.15.45:0/2477177224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9b0004060 0x7fa9b0004800 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa9b400ba30 tx=0x7fa9b0005830 comp rx=0 tx=0).stop 2024-04-16T16:14:31.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.757+0000 7fa9cd400640 1 -- 172.21.15.45:0/2477177224 reap_dead start 2024-04-16T16:14:31.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.758+0000 7fa9cde00640 1 -- 172.21.15.45:0/2477177224 shutdown_connections 2024-04-16T16:14:31.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.758+0000 7fa9cde00640 1 -- 172.21.15.45:0/2477177224 >> 172.21.15.45:0/2477177224 conn(0x7fa9c806b930 msgr2=0x7fa9c806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:31.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.758+0000 7fa9cde00640 1 -- 172.21.15.45:0/2477177224 shutdown_connections 2024-04-16T16:14:31.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.758+0000 7fa9cde00640 1 -- 172.21.15.45:0/2477177224 wait complete. 2024-04-16T16:14:31.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.759+0000 7fa9cde00640 1 Processor -- start 2024-04-16T16:14:31.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.760+0000 7fa9cde00640 1 -- start start 2024-04-16T16:14:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9cde00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b0004060 0x7fa9c815ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9cde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9c80c8b90 0x7fa9c80caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9cde00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9c815efe0 0x7fa9c815f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9cde00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c8160fb0 con 0x7fa9c80c8b90 2024-04-16T16:14:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9cde00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c8160410 con 0x7fa9b0004060 2024-04-16T16:14:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9cde00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa9c8161130 con 0x7fa9c815efe0 2024-04-16T16:14:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9c7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9c80c8b90 0x7fa9c80caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9c7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9c80c8b90 0x7fa9c80caeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53512/0 (socket says 172.21.15.45:53512) 2024-04-16T16:14:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9c7e00640 1 -- 172.21.15.45:0/446466216 learned_addr learned my addr 172.21.15.45:0/446466216 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9cca00640 1 --2- 172.21.15.45:0/446466216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b0004060 0x7fa9c815ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9c7e00640 1 -- 172.21.15.45:0/446466216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9c815efe0 msgr2=0x7fa9c815f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9cd400640 1 --2- 172.21.15.45:0/446466216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9c815efe0 0x7fa9c815f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9c7e00640 1 --2- 172.21.15.45:0/446466216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9c815efe0 0x7fa9c815f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9c7e00640 1 -- 172.21.15.45:0/446466216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b0004060 msgr2=0x7fa9c815ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9c7e00640 1 --2- 172.21.15.45:0/446466216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b0004060 0x7fa9c815ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.761+0000 7fa9c7e00640 1 -- 172.21.15.45:0/446466216 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c815bf90 con 0x7fa9c80c8b90 2024-04-16T16:14:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.762+0000 7fa9cca00640 1 --2- 172.21.15.45:0/446466216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b0004060 0x7fa9c815ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.762+0000 7fa9c7e00640 1 --2- 172.21.15.45:0/446466216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9c80c8b90 0x7fa9c80caeb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa9b402bf80 tx=0x7fa9b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.762+0000 7fa9c4c00640 1 -- 172.21.15.45:0/446466216 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4041070 con 0x7fa9c80c8b90 2024-04-16T16:14:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.762+0000 7fa9c4c00640 1 -- 172.21.15.45:0/446466216 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9b4030a40 con 0x7fa9c80c8b90 2024-04-16T16:14:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.762+0000 7fa9c4c00640 1 -- 172.21.15.45:0/446466216 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4036040 con 0x7fa9c80c8b90 2024-04-16T16:14:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.762+0000 7fa9cde00640 1 -- 172.21.15.45:0/446466216 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9c80cb700 con 0x7fa9c80c8b90 2024-04-16T16:14:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.762+0000 7fa9cde00640 1 -- 172.21.15.45:0/446466216 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9c80cbcc0 con 0x7fa9c80c8b90 2024-04-16T16:14:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.764+0000 7fa9cde00640 1 -- 172.21.15.45:0/446466216 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9c8075780 con 0x7fa9c80c8b90 2024-04-16T16:14:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.764+0000 7fa9c4c00640 1 -- 172.21.15.45:0/446466216 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa9b4039020 con 0x7fa9c80c8b90 2024-04-16T16:14:31.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.766+0000 7fa9c4c00640 1 --2- 172.21.15.45:0/446466216 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa9a806c920 0x7fa9a806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:31.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.766+0000 7fa9c4c00640 1 -- 172.21.15.45:0/446466216 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fa9b40adb00 con 0x7fa9c80c8b90 2024-04-16T16:14:31.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.766+0000 7fa9cca00640 1 --2- 172.21.15.45:0/446466216 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa9a806c920 0x7fa9a806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:31.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.768+0000 7fa9cca00640 1 --2- 172.21.15.45:0/446466216 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa9a806c920 0x7fa9a806ede0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc000c00 tx=0x7fa9bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:31.770+0000 7fa9c4c00640 1 -- 172.21.15.45:0/446466216 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa9b407c150 con 0x7fa9c80c8b90 2024-04-16T16:14:32.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.047+0000 7fa9cde00640 1 -- 172.21.15.45:0/446466216 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fa9c80cff60 con 0x7fa9c80c8b90 2024-04-16T16:14:32.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.049+0000 7fa9c4c00640 1 -- 172.21.15.45:0/446466216 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7fa9b403ae50 con 0x7fa9c80c8b90 2024-04-16T16:14:32.049 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:32.049 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:14:32.049 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2024-04-16T16:14:32.049 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.049 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2024-04-16T16:14:32.049 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mpjReORAAJMoNf4qvfuxPSiV9+VlKfg==", 2024-04-16T16:14:32.049 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.049 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:14:32.049 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:14:32.049 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:14:32.049 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:14:32.049 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.050 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.050 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2024-04-16T16:14:32.050 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mxHeBIBAA3G8maaK65zaOaqXpAOA6xA==", 2024-04-16T16:14:32.050 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.050 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:14:32.050 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:14:32.050 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:14:32.050 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:14:32.050 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.050 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.050 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2024-04-16T16:14:32.050 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5m+AaODxAA08XSjc44/i2KYExYQSukaQ==", 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mEQ4UMRAAOslSsxQ17bOxPrI6QMlRRw==", 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:14:32.051 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.052 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2024-04-16T16:14:32.052 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBuox5m3l3OHxAA+UjwZx8Mn6tCu95TTYIG3w==", 2024-04-16T16:14:32.052 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.052 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:14:32.052 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:14:32.052 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:14:32.052 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.052 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2024-04-16T16:14:32.052 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxox5mozt1LhAAJL5NtMQdPVBtKOaVlMQBtw==", 2024-04-16T16:14:32.052 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.052 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB1ox5mfE7kABAAIvgZ5Wx+eIJ6QNKj2GVdpw==", 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2024-04-16T16:14:32.053 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB4ox5mEvxcDRAAO0yXBKvsC7hZ6bpOAJyuag==", 2024-04-16T16:14:32.054 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.054 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:14:32.054 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:14:32.054 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:14:32.054 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.054 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.054 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2024-04-16T16:14:32.054 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+ox5mZubrORAAcGJLyRN5Ob9aBnopumnEUg==", 2024-04-16T16:14:32.054 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.054 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:14:32.054 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:14:32.054 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:14:32.054 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.055 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.055 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2024-04-16T16:14:32.055 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCCox5mcRqsChAACktmuUHvqWkp/mHT3UcjRw==", 2024-04-16T16:14:32.055 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.055 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:14:32.055 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:14:32.055 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:14:32.055 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.055 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.055 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2024-04-16T16:14:32.055 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFox5mT52FFhAAChhDDfNZmIMgxE1j90LZpQ==", 2024-04-16T16:14:32.055 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.055 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:14:32.055 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:14:32.056 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:14:32.056 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.056 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.056 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2024-04-16T16:14:32.056 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCIox5mC+EDIxAAxtvM0on00lxTLaoRjN0Gfw==", 2024-04-16T16:14:32.056 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.056 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:14:32.056 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:14:32.056 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:14:32.056 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.056 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.056 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2024-04-16T16:14:32.056 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ==", 2024-04-16T16:14:32.057 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.057 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:14:32.057 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2024-04-16T16:14:32.057 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2024-04-16T16:14:32.057 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2024-04-16T16:14:32.057 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.057 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.057 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.057 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2024-04-16T16:14:32.057 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mYpVUGhAAYdR+40O6zbWqgbI55F0FYA==", 2024-04-16T16:14:32.057 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.057 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:14:32.057 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2024-04-16T16:14:32.057 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2024-04-16T16:14:32.057 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2024-04-16T16:14:32.058 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.058 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.058 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2024-04-16T16:14:32.058 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mWdlUDxAAuO2FsGH1/E+qP1xcyUEe3A==", 2024-04-16T16:14:32.058 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.058 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:14:32.058 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2024-04-16T16:14:32.058 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2024-04-16T16:14:32.058 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2024-04-16T16:14:32.058 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.058 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.058 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2024-04-16T16:14:32.058 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5miPdMJBAAqjKpctNYu9ot1lff3rouxg==", 2024-04-16T16:14:32.059 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.059 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:14:32.059 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2024-04-16T16:14:32.059 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2024-04-16T16:14:32.059 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2024-04-16T16:14:32.059 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.059 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.059 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2024-04-16T16:14:32.059 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBpox5mO4PMLBAA/0YbQCfHujgdEzpXs0nOLw==", 2024-04-16T16:14:32.059 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.059 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:14:32.059 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:14:32.059 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:14:32.059 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:14:32.060 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.060 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.060 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.060 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.bootstrap-mds", 2024-04-16T16:14:32.060 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCRox5mUg7dMRAAg4RzwRdNZCA2ECGTF6u5UA==", 2024-04-16T16:14:32.060 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.060 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile bootstrap-mds" 2024-04-16T16:14:32.060 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.060 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.060 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.060 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.bootstrap-mgr", 2024-04-16T16:14:32.060 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCRox5mHS3dMRAAtHAjffraXFgqAaMuVkNVeA==", 2024-04-16T16:14:32.060 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.060 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-16T16:14:32.060 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.bootstrap-osd", 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCRox5mK0fdMRAAa9U0PMGotRmIdag1E0bsDA==", 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile bootstrap-osd" 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.bootstrap-rbd", 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCRox5mebPdMRAA6DeFCbmiv0Up8QiphXlqGw==", 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.061 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.062 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-16T16:14:32.062 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCRox5mbM3dMRAA5glxRhCcD3Rc39luz3gokg==", 2024-04-16T16:14:32.062 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.062 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-16T16:14:32.062 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.062 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.062 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.bootstrap-rgw", 2024-04-16T16:14:32.062 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCRox5mE+jdMRAAH6iqYOcCwDWX6ep4ORwiAw==", 2024-04-16T16:14:32.062 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.062 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-16T16:14:32.062 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.062 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.063 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2024-04-16T16:14:32.063 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mGevaLBAAPpA6g+Fdi/XHqAqG0QZNag==", 2024-04-16T16:14:32.063 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.063 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:14:32.063 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:14:32.063 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:14:32.063 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.063 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:14:32.063 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:32.063 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2024-04-16T16:14:32.063 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5moiNRIRAAz5JzOrjKNLkcphT+iMo+zw==", 2024-04-16T16:14:32.063 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:14:32.063 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:14:32.063 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:14:32.064 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:14:32.064 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.064 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:32.064 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-16T16:14:32.064 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:14:32.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.051+0000 7fa9cde00640 1 -- 172.21.15.45:0/446466216 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa9a806c920 msgr2=0x7fa9a806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:32.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.051+0000 7fa9cde00640 1 --2- 172.21.15.45:0/446466216 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa9a806c920 0x7fa9a806ede0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc000c00 tx=0x7fa9bc009040 comp rx=0 tx=0).stop 2024-04-16T16:14:32.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.051+0000 7fa9cde00640 1 -- 172.21.15.45:0/446466216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9c80c8b90 msgr2=0x7fa9c80caeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:32.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.051+0000 7fa9cde00640 1 --2- 172.21.15.45:0/446466216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9c80c8b90 0x7fa9c80caeb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa9b402bf80 tx=0x7fa9b4004730 comp rx=0 tx=0).stop 2024-04-16T16:14:32.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.051+0000 7fa9cde00640 1 -- 172.21.15.45:0/446466216 shutdown_connections 2024-04-16T16:14:32.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.051+0000 7fa9cde00640 1 --2- 172.21.15.45:0/446466216 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fa9a806c920 0x7fa9a806ede0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.051+0000 7fa9cde00640 1 --2- 172.21.15.45:0/446466216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9c815efe0 0x7fa9c815f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.051+0000 7fa9cde00640 1 --2- 172.21.15.45:0/446466216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9c80c8b90 0x7fa9c80caeb0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.051+0000 7fa9cde00640 1 --2- 172.21.15.45:0/446466216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b0004060 0x7fa9c815ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.051+0000 7fa9cde00640 1 -- 172.21.15.45:0/446466216 >> 172.21.15.45:0/446466216 conn(0x7fa9c806b930 msgr2=0x7fa9c80c5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:32.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.052+0000 7fa9cde00640 1 -- 172.21.15.45:0/446466216 shutdown_connections 2024-04-16T16:14:32.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.052+0000 7fa9cde00640 1 -- 172.21.15.45:0/446466216 wait complete. 2024-04-16T16:14:32.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-16T16:14:32.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.292+0000 7fef4b400640 1 Processor -- start 2024-04-16T16:14:32.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.292+0000 7fef4b400640 1 -- start start 2024-04-16T16:14:32.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.294+0000 7fef4b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef44146220 0x7fef440cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:32.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.294+0000 7fef4b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fef440cd930 0x7fef440725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:32.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.294+0000 7fef4b400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef44146600 con 0x7fef44072180 2024-04-16T16:14:32.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.294+0000 7fef4b400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fef4406f840 con 0x7fef440cd930 2024-04-16T16:14:32.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.294+0000 7fef4b400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fef440ce0e0 con 0x7fef44146220 2024-04-16T16:14:32.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.294+0000 7fef49400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fef440cd930 0x7fef440725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:32.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.294+0000 7fef49400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fef440cd930 0x7fef440725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53518/0 (socket says 172.21.15.45:53518) 2024-04-16T16:14:32.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.294+0000 7fef49400640 1 -- 172.21.15.45:0/1055981534 learned_addr learned my addr 172.21.15.45:0/1055981534 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:32.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.295+0000 7fef48a00640 1 --2- 172.21.15.45:0/1055981534 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef44146220 0x7fef440cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:32.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.295+0000 7fef49400640 1 -- 172.21.15.45:0/1055981534 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef44146220 msgr2=0x7fef440cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:32.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.295+0000 7fef49400640 1 --2- 172.21.15.45:0/1055981534 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef44146220 0x7fef440cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.295+0000 7fef49400640 1 -- 172.21.15.45:0/1055981534 >> v1:172.21.15.103:6789/0 conn(0x7fef44072180 legacy=0x7fef440cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:32.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.295+0000 7fef49400640 1 -- 172.21.15.45:0/1055981534 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef440cde00 con 0x7fef440cd930 2024-04-16T16:14:32.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.295+0000 7fef48a00640 1 --2- 172.21.15.45:0/1055981534 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef44146220 0x7fef440cf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:32.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.296+0000 7fef49400640 1 --2- 172.21.15.45:0/1055981534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef440cd930 0x7fef440725c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009e50 tx=0x7fef3c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0e0ac87796837e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:32.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.297+0000 7fef43400640 1 -- 172.21.15.45:0/1055981534 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c02ca70 con 0x7fef440cd930 2024-04-16T16:14:32.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.297+0000 7fef43400640 1 -- 172.21.15.45:0/1055981534 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef3c02cc10 con 0x7fef440cd930 2024-04-16T16:14:32.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.297+0000 7fef43400640 1 -- 172.21.15.45:0/1055981534 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c0343f0 con 0x7fef440cd930 2024-04-16T16:14:32.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.297+0000 7fef4b400640 1 -- 172.21.15.45:0/1055981534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef440cd930 msgr2=0x7fef440725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:32.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.297+0000 7fef4b400640 1 --2- 172.21.15.45:0/1055981534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef440cd930 0x7fef440725c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009e50 tx=0x7fef3c02bed0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.298+0000 7fef4b400640 1 -- 172.21.15.45:0/1055981534 shutdown_connections 2024-04-16T16:14:32.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.298+0000 7fef4b400640 1 --2- 172.21.15.45:0/1055981534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef440cd930 0x7fef440725c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.298+0000 7fef4b400640 1 --2- 172.21.15.45:0/1055981534 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef44146220 0x7fef440cf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.298+0000 7fef4b400640 1 -- 172.21.15.45:0/1055981534 >> 172.21.15.45:0/1055981534 conn(0x7fef4406b930 msgr2=0x7fef4406bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:32.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.298+0000 7fef4b400640 1 -- 172.21.15.45:0/1055981534 shutdown_connections 2024-04-16T16:14:32.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.298+0000 7fef4b400640 1 -- 172.21.15.45:0/1055981534 wait complete. 2024-04-16T16:14:32.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.299+0000 7fef4b400640 1 Processor -- start 2024-04-16T16:14:32.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.300+0000 7fef4b400640 1 -- start start 2024-04-16T16:14:32.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.300+0000 7fef4b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef44072180 0x7fef44156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:32.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.301+0000 7fef4b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef440cd930 0x7fef44142490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:32.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.301+0000 7fef4b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef44146220 0x7fef441602c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:32.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.301+0000 7fef4b400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fef440ce910 con 0x7fef440cd930 2024-04-16T16:14:32.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.301+0000 7fef4b400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fef44072b00 con 0x7fef44146220 2024-04-16T16:14:32.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.301+0000 7fef4b400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fef440cebf0 con 0x7fef44072180 2024-04-16T16:14:32.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.301+0000 7fef43e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef440cd930 0x7fef44142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:32.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.301+0000 7fef43e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef440cd930 0x7fef44142490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53530/0 (socket says 172.21.15.45:53530) 2024-04-16T16:14:32.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.301+0000 7fef43e00640 1 -- 172.21.15.45:0/1972285267 learned_addr learned my addr 172.21.15.45:0/1972285267 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:32.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.301+0000 7fef48a00640 1 --2- 172.21.15.45:0/1972285267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef44072180 0x7fef44156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:32.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.301+0000 7fef49400640 1 --2- 172.21.15.45:0/1972285267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef44146220 0x7fef441602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:32.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.302+0000 7fef43e00640 1 -- 172.21.15.45:0/1972285267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef44072180 msgr2=0x7fef44156310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:32.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.302+0000 7fef43e00640 1 --2- 172.21.15.45:0/1972285267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef44072180 0x7fef44156310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.302+0000 7fef43e00640 1 -- 172.21.15.45:0/1972285267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef44146220 msgr2=0x7fef441602c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:32.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.302+0000 7fef43e00640 1 --2- 172.21.15.45:0/1972285267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef44146220 0x7fef441602c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.302+0000 7fef43e00640 1 -- 172.21.15.45:0/1972285267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef441f4970 con 0x7fef440cd930 2024-04-16T16:14:32.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.302+0000 7fef43e00640 1 --2- 172.21.15.45:0/1972285267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef440cd930 0x7fef44142490 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fef3000b3f0 tx=0x7fef3000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:32.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.302+0000 7fef40c00640 1 -- 172.21.15.45:0/1972285267 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef30015020 con 0x7fef440cd930 2024-04-16T16:14:32.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.302+0000 7fef40c00640 1 -- 172.21.15.45:0/1972285267 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef30011830 con 0x7fef440cd930 2024-04-16T16:14:32.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.302+0000 7fef40c00640 1 -- 172.21.15.45:0/1972285267 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef30011c20 con 0x7fef440cd930 2024-04-16T16:14:32.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.302+0000 7fef4b400640 1 -- 172.21.15.45:0/1972285267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef441f4c50 con 0x7fef440cd930 2024-04-16T16:14:32.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.302+0000 7fef4b400640 1 -- 172.21.15.45:0/1972285267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef441f5210 con 0x7fef440cd930 2024-04-16T16:14:32.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.304+0000 7fef4b400640 1 -- 172.21.15.45:0/1972285267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef44075780 con 0x7fef440cd930 2024-04-16T16:14:32.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.304+0000 7fef40c00640 1 -- 172.21.15.45:0/1972285267 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fef30018020 con 0x7fef440cd930 2024-04-16T16:14:32.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.305+0000 7fef40c00640 1 --2- 172.21.15.45:0/1972285267 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fef2406c8f0 0x7fef2406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:32.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.305+0000 7fef40c00640 1 -- 172.21.15.45:0/1972285267 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fef3008b410 con 0x7fef440cd930 2024-04-16T16:14:32.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.306+0000 7fef48a00640 1 --2- 172.21.15.45:0/1972285267 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fef2406c8f0 0x7fef2406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:32.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.307+0000 7fef48a00640 1 --2- 172.21.15.45:0/1972285267 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fef2406c8f0 0x7fef2406edb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fef4415dd80 tx=0x7fef34008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:32.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.310+0000 7fef40c00640 1 -- 172.21.15.45:0/1972285267 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fef3001f030 con 0x7fef440cd930 2024-04-16T16:14:32.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.594+0000 7fef4b400640 1 -- 172.21.15.45:0/1972285267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7fef440b4240 con 0x7fef440cd930 2024-04-16T16:14:32.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.598+0000 7fef40c00640 1 -- 172.21.15.45:0/1972285267 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7fef300172c0 con 0x7fef440cd930 2024-04-16T16:14:32.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.601+0000 7fef4b400640 1 -- 172.21.15.45:0/1972285267 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fef2406c8f0 msgr2=0x7fef2406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:32.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.601+0000 7fef4b400640 1 --2- 172.21.15.45:0/1972285267 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fef2406c8f0 0x7fef2406edb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fef4415dd80 tx=0x7fef34008040 comp rx=0 tx=0).stop 2024-04-16T16:14:32.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.601+0000 7fef4b400640 1 -- 172.21.15.45:0/1972285267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef440cd930 msgr2=0x7fef44142490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:32.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.601+0000 7fef4b400640 1 --2- 172.21.15.45:0/1972285267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef440cd930 0x7fef44142490 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fef3000b3f0 tx=0x7fef3000b8b0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.601+0000 7fef4b400640 1 -- 172.21.15.45:0/1972285267 shutdown_connections 2024-04-16T16:14:32.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.601+0000 7fef4b400640 1 --2- 172.21.15.45:0/1972285267 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fef2406c8f0 0x7fef2406edb0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.601+0000 7fef4b400640 1 --2- 172.21.15.45:0/1972285267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef44146220 0x7fef441602c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.601+0000 7fef4b400640 1 --2- 172.21.15.45:0/1972285267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef440cd930 0x7fef44142490 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.601+0000 7fef4b400640 1 --2- 172.21.15.45:0/1972285267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef44072180 0x7fef44156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.601+0000 7fef4b400640 1 -- 172.21.15.45:0/1972285267 >> 172.21.15.45:0/1972285267 conn(0x7fef4406b930 msgr2=0x7fef440cef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:32.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.601+0000 7fef4b400640 1 -- 172.21.15.45:0/1972285267 shutdown_connections 2024-04-16T16:14:32.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.601+0000 7fef4b400640 1 -- 172.21.15.45:0/1972285267 wait complete. 2024-04-16T16:14:32.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-16T16:14:32.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.835+0000 7f1d14c00640 1 Processor -- start 2024-04-16T16:14:32.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.836+0000 7f1d14c00640 1 -- start start 2024-04-16T16:14:32.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.837+0000 7f1d14c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1d100cb7d0 0x7f1d100cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:32.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.837+0000 7f1d14c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1d100cbc80 0x7f1d100cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:32.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.837+0000 7f1d14c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d1015f1b0 con 0x7f1d1015e830 2024-04-16T16:14:32.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.837+0000 7f1d14c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1d100cf280 con 0x7f1d100cbc80 2024-04-16T16:14:32.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.837+0000 7f1d14c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1d100ce190 con 0x7f1d100cb7d0 2024-04-16T16:14:32.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.838+0000 7f1d0fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1d100cbc80 0x7f1d100cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:32.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.838+0000 7f1d0fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1d100cbc80 0x7f1d100cdb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53536/0 (socket says 172.21.15.45:53536) 2024-04-16T16:14:32.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.838+0000 7f1d0fe00640 1 -- 172.21.15.45:0/175130491 learned_addr learned my addr 172.21.15.45:0/175130491 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:32.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.838+0000 7f1d07e00640 1 --2- 172.21.15.45:0/175130491 >> v2:172.21.15.103:3300/0 conn(0x7f1d100cb7d0 0x7f1d100cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:32.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.838+0000 7f1d0fe00640 1 -- 172.21.15.45:0/175130491 >> v2:172.21.15.103:3300/0 conn(0x7f1d100cb7d0 msgr2=0x7f1d100cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:32.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.838+0000 7f1d0fe00640 1 --2- 172.21.15.45:0/175130491 >> v2:172.21.15.103:3300/0 conn(0x7f1d100cb7d0 0x7f1d100cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.838+0000 7f1d0fe00640 1 -- 172.21.15.45:0/175130491 >> v1:172.21.15.103:6789/0 conn(0x7f1d1015e830 legacy=0x7f1d100ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:32.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.838+0000 7f1d0fe00640 1 -- 172.21.15.45:0/175130491 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d100ce470 con 0x7f1d100cbc80 2024-04-16T16:14:32.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.839+0000 7f1d0fe00640 1 --2- 172.21.15.45:0/175130491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d100cbc80 0x7f1d100cdb60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1d10073810 tx=0x7f1d0802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e05bef1e65f330f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:32.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.840+0000 7f1d0ea00640 1 -- 172.21.15.45:0/175130491 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0802c9f0 con 0x7f1d100cbc80 2024-04-16T16:14:32.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.840+0000 7f1d0ea00640 1 -- 172.21.15.45:0/175130491 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d0802cb90 con 0x7f1d100cbc80 2024-04-16T16:14:32.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.840+0000 7f1d0ea00640 1 -- 172.21.15.45:0/175130491 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d080343f0 con 0x7f1d100cbc80 2024-04-16T16:14:32.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.840+0000 7f1d14c00640 1 -- 172.21.15.45:0/175130491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d100cbc80 msgr2=0x7f1d100cdb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:32.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.840+0000 7f1d14c00640 1 --2- 172.21.15.45:0/175130491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d100cbc80 0x7f1d100cdb60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1d10073810 tx=0x7f1d0802be50 comp rx=0 tx=0).stop 2024-04-16T16:14:32.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.840+0000 7f1d14c00640 1 -- 172.21.15.45:0/175130491 shutdown_connections 2024-04-16T16:14:32.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.840+0000 7f1d14c00640 1 --2- 172.21.15.45:0/175130491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d100cbc80 0x7f1d100cdb60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.840+0000 7f1d14c00640 1 --2- 172.21.15.45:0/175130491 >> v2:172.21.15.103:3300/0 conn(0x7f1d100cb7d0 0x7f1d100cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.840+0000 7f1d14c00640 1 -- 172.21.15.45:0/175130491 >> 172.21.15.45:0/175130491 conn(0x7f1d1006b930 msgr2=0x7f1d1006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:32.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.840+0000 7f1d14c00640 1 -- 172.21.15.45:0/175130491 shutdown_connections 2024-04-16T16:14:32.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.840+0000 7f1d14c00640 1 -- 172.21.15.45:0/175130491 wait complete. 2024-04-16T16:14:32.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.842+0000 7f1d14c00640 1 Processor -- start 2024-04-16T16:14:32.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.842+0000 7f1d14c00640 1 -- start start 2024-04-16T16:14:32.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.843+0000 7f1d14c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d100cb7d0 0x7f1d10154790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:32.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.843+0000 7f1d14c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d100cbc80 0x7f1d10156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:32.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.843+0000 7f1d14c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d1015e830 0x7f1d10157c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:32.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.843+0000 7f1d14c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1015ec10 con 0x7f1d100cb7d0 2024-04-16T16:14:32.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.843+0000 7f1d14c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1d100cb2b0 con 0x7f1d1015e830 2024-04-16T16:14:32.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.843+0000 7f1d14c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1d100cac30 con 0x7f1d100cbc80 2024-04-16T16:14:32.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.843+0000 7f1d0f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d100cb7d0 0x7f1d10154790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:32.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.844+0000 7f1d0f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d100cb7d0 0x7f1d10154790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53550/0 (socket says 172.21.15.45:53550) 2024-04-16T16:14:32.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.844+0000 7f1d0f400640 1 -- 172.21.15.45:0/3268848389 learned_addr learned my addr 172.21.15.45:0/3268848389 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:32.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.844+0000 7f1d07e00640 1 --2- 172.21.15.45:0/3268848389 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d100cbc80 0x7f1d10156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:32.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.844+0000 7f1d0f400640 1 -- 172.21.15.45:0/3268848389 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d100cbc80 msgr2=0x7f1d10156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:32.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.844+0000 7f1d0f400640 1 --2- 172.21.15.45:0/3268848389 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d100cbc80 0x7f1d10156a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.844+0000 7f1d0f400640 1 -- 172.21.15.45:0/3268848389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d1015e830 msgr2=0x7f1d10157c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:32.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.844+0000 7f1d0f400640 1 --2- 172.21.15.45:0/3268848389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d1015e830 0x7f1d10157c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:32.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.844+0000 7f1d0f400640 1 -- 172.21.15.45:0/3268848389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d10154e20 con 0x7f1d100cb7d0 2024-04-16T16:14:32.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.844+0000 7f1d0f400640 1 --2- 172.21.15.45:0/3268848389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d100cb7d0 0x7f1d10154790 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000c430 tx=0x7f1d0000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:32.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.844+0000 7f1d07400640 1 -- 172.21.15.45:0/3268848389 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d00011790 con 0x7f1d100cb7d0 2024-04-16T16:14:32.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.845+0000 7f1d07400640 1 -- 172.21.15.45:0/3268848389 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d00017070 con 0x7f1d100cb7d0 2024-04-16T16:14:32.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.845+0000 7f1d07400640 1 -- 172.21.15.45:0/3268848389 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d00011bb0 con 0x7f1d100cb7d0 2024-04-16T16:14:32.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.845+0000 7f1d14c00640 1 -- 172.21.15.45:0/3268848389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d10155100 con 0x7f1d100cb7d0 2024-04-16T16:14:32.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.845+0000 7f1d14c00640 1 -- 172.21.15.45:0/3268848389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d10157430 con 0x7f1d100cb7d0 2024-04-16T16:14:32.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.846+0000 7f1d14c00640 1 -- 172.21.15.45:0/3268848389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d10075780 con 0x7f1d100cb7d0 2024-04-16T16:14:32.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.847+0000 7f1d07400640 1 -- 172.21.15.45:0/3268848389 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1d0001c460 con 0x7f1d100cb7d0 2024-04-16T16:14:32.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.848+0000 7f1d07400640 1 --2- 172.21.15.45:0/3268848389 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1cec06c8f0 0x7f1cec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:32.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.848+0000 7f1d07400640 1 -- 172.21.15.45:0/3268848389 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f1d0008b770 con 0x7f1d100cb7d0 2024-04-16T16:14:32.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.848+0000 7f1d07e00640 1 --2- 172.21.15.45:0/3268848389 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1cec06c8f0 0x7f1cec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:32.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.850+0000 7f1d07e00640 1 --2- 172.21.15.45:0/3268848389 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1cec06c8f0 0x7f1cec06edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1d10149ea0 tx=0x7f1cf8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:32.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:32.852+0000 7f1d07400640 1 -- 172.21.15.45:0/3268848389 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1d00059e50 con 0x7f1d100cb7d0 2024-04-16T16:14:33.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.121+0000 7f1d14c00640 1 -- 172.21.15.45:0/3268848389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f1d101552d0 con 0x7f1d100cb7d0 2024-04-16T16:14:33.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.125+0000 7f1d07400640 1 -- 172.21.15.45:0/3268848389 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7f1d00059750 con 0x7f1d100cb7d0 2024-04-16T16:14:33.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.128+0000 7f1d14c00640 1 -- 172.21.15.45:0/3268848389 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1cec06c8f0 msgr2=0x7f1cec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:33.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.128+0000 7f1d14c00640 1 --2- 172.21.15.45:0/3268848389 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1cec06c8f0 0x7f1cec06edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1d10149ea0 tx=0x7f1cf8008040 comp rx=0 tx=0).stop 2024-04-16T16:14:33.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.128+0000 7f1d14c00640 1 -- 172.21.15.45:0/3268848389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d100cb7d0 msgr2=0x7f1d10154790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:33.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.128+0000 7f1d14c00640 1 --2- 172.21.15.45:0/3268848389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d100cb7d0 0x7f1d10154790 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000c430 tx=0x7f1d0000c8f0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.128+0000 7f1d14c00640 1 -- 172.21.15.45:0/3268848389 shutdown_connections 2024-04-16T16:14:33.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.128+0000 7f1d14c00640 1 --2- 172.21.15.45:0/3268848389 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1cec06c8f0 0x7f1cec06edb0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.128+0000 7f1d14c00640 1 --2- 172.21.15.45:0/3268848389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d1015e830 0x7f1d10157c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.128+0000 7f1d14c00640 1 --2- 172.21.15.45:0/3268848389 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d100cbc80 0x7f1d10156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.128+0000 7f1d14c00640 1 --2- 172.21.15.45:0/3268848389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d100cb7d0 0x7f1d10154790 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.128+0000 7f1d14c00640 1 -- 172.21.15.45:0/3268848389 >> 172.21.15.45:0/3268848389 conn(0x7f1d1006b930 msgr2=0x7f1d100cd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:33.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.128+0000 7f1d14c00640 1 -- 172.21.15.45:0/3268848389 shutdown_connections 2024-04-16T16:14:33.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.128+0000 7f1d14c00640 1 -- 172.21.15.45:0/3268848389 wait complete. 2024-04-16T16:14:33.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-16T16:14:33.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.362+0000 7f8a5e200640 1 Processor -- start 2024-04-16T16:14:33.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.363+0000 7f8a5e200640 1 -- start start 2024-04-16T16:14:33.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.364+0000 7f8a5e200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a580afe90 0x7f8a580b5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:33.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.364+0000 7f8a5e200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a58164440 con 0x7f8a58072890 2024-04-16T16:14:33.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.364+0000 7f8a5e200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a580b5ca0 con 0x7f8a580b5640 2024-04-16T16:14:33.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.364+0000 7f8a5e200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a5806ef10 con 0x7f8a580afe90 2024-04-16T16:14:33.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.365+0000 7f8a57e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8a58072890 0x7f8a58160160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:34498/0 (socket says 172.21.15.45:34498) 2024-04-16T16:14:33.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.365+0000 7f8a57e00640 1 -- 172.21.15.45:0/648892287 learned_addr learned my addr 172.21.15.45:0/648892287 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:33.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.365+0000 7f8a57400640 1 --2- 172.21.15.45:0/648892287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a580afe90 0x7f8a580b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:33.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.365+0000 7f8a56a00640 1 -- 172.21.15.45:0/648892287 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3147587584 0 0) 0x7f8a58164440 con 0x7f8a58072890 2024-04-16T16:14:33.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.366+0000 7f8a56a00640 1 -- 172.21.15.45:0/648892287 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a3c003520 con 0x7f8a58072890 2024-04-16T16:14:33.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.366+0000 7f8a56a00640 1 -- 172.21.15.45:0/648892287 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1632644642 0 0) 0x7f8a3c003520 con 0x7f8a58072890 2024-04-16T16:14:33.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.366+0000 7f8a56a00640 1 -- 172.21.15.45:0/648892287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a580afe90 msgr2=0x7f8a580b5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:33.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.366+0000 7f8a56a00640 1 --2- 172.21.15.45:0/648892287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a580afe90 0x7f8a580b5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.366+0000 7f8a56a00640 1 -- 172.21.15.45:0/648892287 >> v1:172.21.15.103:6789/0 conn(0x7f8a580b5640 legacy=0x7f8a58072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.366+0000 7f8a56a00640 1 -- 172.21.15.45:0/648892287 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a58166a60 con 0x7f8a58072890 2024-04-16T16:14:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.366+0000 7f8a56a00640 1 -- 172.21.15.45:0/648892287 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f8a500019f0 con 0x7f8a58072890 2024-04-16T16:14:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.366+0000 7f8a56a00640 1 -- 172.21.15.45:0/648892287 >> v1:172.21.15.45:6789/0 conn(0x7f8a58072890 legacy=0x7f8a58160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.366+0000 7f8a56a00640 1 --2- 172.21.15.45:0/648892287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a3c0039e0 0x7f8a3c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.366+0000 7f8a56a00640 1 --2- 172.21.15.45:0/648892287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a3c004800 0x7f8a3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.366+0000 7f8a56a00640 1 --2- 172.21.15.45:0/648892287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a580afe90 0x7f8a3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.366+0000 7f8a56a00640 1 -- 172.21.15.45:0/648892287 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5806ef10 con 0x7f8a3c0039e0 2024-04-16T16:14:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.366+0000 7f8a56a00640 1 -- 172.21.15.45:0/648892287 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c00baa0 con 0x7f8a3c004800 2024-04-16T16:14:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.366+0000 7f8a56a00640 1 -- 172.21.15.45:0/648892287 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a3c00bca0 con 0x7f8a580afe90 2024-04-16T16:14:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.366+0000 7f8a57400640 1 --2- 172.21.15.45:0/648892287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a3c0039e0 0x7f8a3c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.367+0000 7f8a57400640 1 -- 172.21.15.45:0/648892287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a580afe90 msgr2=0x7f8a3c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:33.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.367+0000 7f8a57400640 1 --2- 172.21.15.45:0/648892287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a580afe90 0x7f8a3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.367+0000 7f8a57400640 1 -- 172.21.15.45:0/648892287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a3c004800 msgr2=0x7f8a3c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:33.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.367+0000 7f8a57400640 1 --2- 172.21.15.45:0/648892287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a3c004800 0x7f8a3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.367+0000 7f8a57400640 1 -- 172.21.15.45:0/648892287 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3c003520 con 0x7f8a3c0039e0 2024-04-16T16:14:33.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.368+0000 7f8a57400640 1 --2- 172.21.15.45:0/648892287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a3c0039e0 0x7f8a3c003e00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8a48009a90 tx=0x7f8a3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38f7ec5a559d5f65 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:33.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.368+0000 7f8a56a00640 1 -- 172.21.15.45:0/648892287 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a48039070 con 0x7f8a3c0039e0 2024-04-16T16:14:33.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.368+0000 7f8a56a00640 1 -- 172.21.15.45:0/648892287 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a4802c6b0 con 0x7f8a3c0039e0 2024-04-16T16:14:33.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.368+0000 7f8a56a00640 1 -- 172.21.15.45:0/648892287 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a48009690 con 0x7f8a3c0039e0 2024-04-16T16:14:33.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.369+0000 7f8a5e200640 1 -- 172.21.15.45:0/648892287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a3c0039e0 msgr2=0x7f8a3c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:33.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.369+0000 7f8a5e200640 1 --2- 172.21.15.45:0/648892287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a3c0039e0 0x7f8a3c003e00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8a48009a90 tx=0x7f8a3c005830 comp rx=0 tx=0).stop 2024-04-16T16:14:33.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.369+0000 7f8a57e00640 1 -- 172.21.15.45:0/648892287 reap_dead start 2024-04-16T16:14:33.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.369+0000 7f8a5e200640 1 -- 172.21.15.45:0/648892287 shutdown_connections 2024-04-16T16:14:33.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.369+0000 7f8a5e200640 1 -- 172.21.15.45:0/648892287 >> 172.21.15.45:0/648892287 conn(0x7f8a5806b930 msgr2=0x7f8a5806bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:33.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.369+0000 7f8a5e200640 1 -- 172.21.15.45:0/648892287 shutdown_connections 2024-04-16T16:14:33.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.369+0000 7f8a5e200640 1 -- 172.21.15.45:0/648892287 wait complete. 2024-04-16T16:14:33.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.372+0000 7f8a5e200640 1 Processor -- start 2024-04-16T16:14:33.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.372+0000 7f8a5e200640 1 -- start start 2024-04-16T16:14:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.373+0000 7f8a5e200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a3c0039e0 0x7f8a581498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.373+0000 7f8a5e200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a5815afb0 0x7f8a5815b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.373+0000 7f8a5e200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a581651d0 0x7f8a58165650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.373+0000 7f8a5e200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a58167300 con 0x7f8a581651d0 2024-04-16T16:14:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.373+0000 7f8a5e200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a58167020 con 0x7f8a3c0039e0 2024-04-16T16:14:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.373+0000 7f8a5e200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a580b0270 con 0x7f8a5815afb0 2024-04-16T16:14:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.374+0000 7f8a57e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a581651d0 0x7f8a58165650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.374+0000 7f8a57e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a581651d0 0x7f8a58165650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33264/0 (socket says 172.21.15.45:33264) 2024-04-16T16:14:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.374+0000 7f8a57e00640 1 -- 172.21.15.45:0/117843248 learned_addr learned my addr 172.21.15.45:0/117843248 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.374+0000 7f8a57400640 1 --2- 172.21.15.45:0/117843248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a3c0039e0 0x7f8a581498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.374+0000 7f8a4fe00640 1 --2- 172.21.15.45:0/117843248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a5815afb0 0x7f8a5815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.374+0000 7f8a57e00640 1 -- 172.21.15.45:0/117843248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a5815afb0 msgr2=0x7f8a5815b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.374+0000 7f8a57e00640 1 --2- 172.21.15.45:0/117843248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a5815afb0 0x7f8a5815b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.374+0000 7f8a57e00640 1 -- 172.21.15.45:0/117843248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a3c0039e0 msgr2=0x7f8a581498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.374+0000 7f8a57e00640 1 --2- 172.21.15.45:0/117843248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a3c0039e0 0x7f8a581498e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.374+0000 7f8a57e00640 1 -- 172.21.15.45:0/117843248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a58149e20 con 0x7f8a581651d0 2024-04-16T16:14:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.374+0000 7f8a4fe00640 1 --2- 172.21.15.45:0/117843248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a5815afb0 0x7f8a5815b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.374+0000 7f8a57e00640 1 --2- 172.21.15.45:0/117843248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a581651d0 0x7f8a58165650 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8a5806f460 tx=0x7f8a5000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.375+0000 7f8a4f400640 1 -- 172.21.15.45:0/117843248 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5000a5b0 con 0x7f8a581651d0 2024-04-16T16:14:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.375+0000 7f8a4f400640 1 -- 172.21.15.45:0/117843248 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a5000a750 con 0x7f8a581651d0 2024-04-16T16:14:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.375+0000 7f8a4f400640 1 -- 172.21.15.45:0/117843248 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5000ab40 con 0x7f8a581651d0 2024-04-16T16:14:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.375+0000 7f8a5e200640 1 -- 172.21.15.45:0/117843248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a5815bee0 con 0x7f8a581651d0 2024-04-16T16:14:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.375+0000 7f8a5e200640 1 -- 172.21.15.45:0/117843248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a5815c180 con 0x7f8a581651d0 2024-04-16T16:14:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.376+0000 7f8a5e200640 1 -- 172.21.15.45:0/117843248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a58075780 con 0x7f8a581651d0 2024-04-16T16:14:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.377+0000 7f8a4f400640 1 -- 172.21.15.45:0/117843248 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8a50011070 con 0x7f8a581651d0 2024-04-16T16:14:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.378+0000 7f8a4f400640 1 --2- 172.21.15.45:0/117843248 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a3406c8f0 0x7f8a3406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.378+0000 7f8a4f400640 1 -- 172.21.15.45:0/117843248 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f8a5008ae90 con 0x7f8a581651d0 2024-04-16T16:14:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.379+0000 7f8a57400640 1 --2- 172.21.15.45:0/117843248 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a3406c8f0 0x7f8a3406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:33.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.380+0000 7f8a57400640 1 --2- 172.21.15.45:0/117843248 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a3406c8f0 0x7f8a3406edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8a48036000 tx=0x7f8a48030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:33.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.382+0000 7f8a4f400640 1 -- 172.21.15.45:0/117843248 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8a58149e20 con 0x7f8a581651d0 2024-04-16T16:14:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.653+0000 7f8a5e200640 1 -- 172.21.15.45:0/117843248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f8a58070020 con 0x7f8a581651d0 2024-04-16T16:14:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.657+0000 7f8a4f400640 1 -- 172.21.15.45:0/117843248 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f8a5008c0a0 con 0x7f8a581651d0 2024-04-16T16:14:33.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.659+0000 7f8a5e200640 1 -- 172.21.15.45:0/117843248 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a3406c8f0 msgr2=0x7f8a3406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:33.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.659+0000 7f8a5e200640 1 --2- 172.21.15.45:0/117843248 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a3406c8f0 0x7f8a3406edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8a48036000 tx=0x7f8a48030000 comp rx=0 tx=0).stop 2024-04-16T16:14:33.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.659+0000 7f8a5e200640 1 -- 172.21.15.45:0/117843248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a581651d0 msgr2=0x7f8a58165650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:33.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.659+0000 7f8a5e200640 1 --2- 172.21.15.45:0/117843248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a581651d0 0x7f8a58165650 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8a5806f460 tx=0x7f8a5000ccd0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.660+0000 7f8a5e200640 1 -- 172.21.15.45:0/117843248 shutdown_connections 2024-04-16T16:14:33.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.660+0000 7f8a5e200640 1 --2- 172.21.15.45:0/117843248 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a3406c8f0 0x7f8a3406edb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.660+0000 7f8a5e200640 1 --2- 172.21.15.45:0/117843248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a581651d0 0x7f8a58165650 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.660+0000 7f8a5e200640 1 --2- 172.21.15.45:0/117843248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a5815afb0 0x7f8a5815b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.660+0000 7f8a5e200640 1 --2- 172.21.15.45:0/117843248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a3c0039e0 0x7f8a581498e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.660+0000 7f8a5e200640 1 -- 172.21.15.45:0/117843248 >> 172.21.15.45:0/117843248 conn(0x7f8a5806b930 msgr2=0x7f8a581634e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:33.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.660+0000 7f8a5e200640 1 -- 172.21.15.45:0/117843248 shutdown_connections 2024-04-16T16:14:33.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.660+0000 7f8a5e200640 1 -- 172.21.15.45:0/117843248 wait complete. 2024-04-16T16:14:33.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-16T16:14:33.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.881+0000 7fd87ca00640 1 Processor -- start 2024-04-16T16:14:33.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.881+0000 7fd87ca00640 1 -- start start 2024-04-16T16:14:33.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.883+0000 7fd87ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8780c6fc0 0x7fd87806f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:33.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.883+0000 7fd87ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd87815cdc0 0x7fd8780c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:33.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.883+0000 7fd87ca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd87815d600 con 0x7fd8780c6320 2024-04-16T16:14:33.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.883+0000 7fd87ca00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd8780c5670 con 0x7fd87815cdc0 2024-04-16T16:14:33.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.883+0000 7fd87ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd87815d8e0 con 0x7fd8780c6fc0 2024-04-16T16:14:33.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.883+0000 7fd876a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd87815cdc0 0x7fd8780c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:33.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.883+0000 7fd876a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd87815cdc0 0x7fd8780c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33272/0 (socket says 172.21.15.45:33272) 2024-04-16T16:14:33.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.883+0000 7fd876a00640 1 -- 172.21.15.45:0/1227459430 learned_addr learned my addr 172.21.15.45:0/1227459430 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:33.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.884+0000 7fd877400640 1 --2- 172.21.15.45:0/1227459430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8780c6fc0 0x7fd87806f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:33.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.884+0000 7fd876a00640 1 -- 172.21.15.45:0/1227459430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8780c6fc0 msgr2=0x7fd87806f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:33.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.884+0000 7fd876a00640 1 --2- 172.21.15.45:0/1227459430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8780c6fc0 0x7fd87806f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.884+0000 7fd876a00640 1 -- 172.21.15.45:0/1227459430 >> v1:172.21.15.103:6789/0 conn(0x7fd8780c6320 legacy=0x7fd87806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:33.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.884+0000 7fd876a00640 1 -- 172.21.15.45:0/1227459430 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87815dbc0 con 0x7fd87815cdc0 2024-04-16T16:14:33.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.884+0000 7fd877400640 1 --2- 172.21.15.45:0/1227459430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8780c6fc0 0x7fd87806f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:33.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.885+0000 7fd876a00640 1 --2- 172.21.15.45:0/1227459430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd87815cdc0 0x7fd8780c5de0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd864009870 tx=0x7fd86402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b7f5e60a037711 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:33.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.885+0000 7fd876000640 1 -- 172.21.15.45:0/1227459430 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd864035020 con 0x7fd87815cdc0 2024-04-16T16:14:33.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.885+0000 7fd876000640 1 -- 172.21.15.45:0/1227459430 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd864031640 con 0x7fd87815cdc0 2024-04-16T16:14:33.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.885+0000 7fd876000640 1 -- 172.21.15.45:0/1227459430 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86402d040 con 0x7fd87815cdc0 2024-04-16T16:14:33.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.886+0000 7fd87ca00640 1 -- 172.21.15.45:0/1227459430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd87815cdc0 msgr2=0x7fd8780c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:33.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.886+0000 7fd87ca00640 1 --2- 172.21.15.45:0/1227459430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd87815cdc0 0x7fd8780c5de0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd864009870 tx=0x7fd86402b8f0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.886+0000 7fd87ca00640 1 -- 172.21.15.45:0/1227459430 shutdown_connections 2024-04-16T16:14:33.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.886+0000 7fd87ca00640 1 --2- 172.21.15.45:0/1227459430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd87815cdc0 0x7fd8780c5de0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.886+0000 7fd87ca00640 1 --2- 172.21.15.45:0/1227459430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8780c6fc0 0x7fd87806f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.886+0000 7fd87ca00640 1 -- 172.21.15.45:0/1227459430 >> 172.21.15.45:0/1227459430 conn(0x7fd87806b930 msgr2=0x7fd87806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:33.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.886+0000 7fd87ca00640 1 -- 172.21.15.45:0/1227459430 shutdown_connections 2024-04-16T16:14:33.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.886+0000 7fd87ca00640 1 -- 172.21.15.45:0/1227459430 wait complete. 2024-04-16T16:14:33.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.888+0000 7fd87ca00640 1 Processor -- start 2024-04-16T16:14:33.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.888+0000 7fd87ca00640 1 -- start start 2024-04-16T16:14:33.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.889+0000 7fd87ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8780c6320 0x7fd878140af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:33.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.889+0000 7fd87ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8780c6fc0 0x7fd878143460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:33.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.889+0000 7fd87ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd87815cdc0 0x7fd8781439a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:33.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.889+0000 7fd87ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd87815d3c0 con 0x7fd8780c6fc0 2024-04-16T16:14:33.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.889+0000 7fd87ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd87815d240 con 0x7fd8780c6320 2024-04-16T16:14:33.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.889+0000 7fd87ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd8780c73a0 con 0x7fd87815cdc0 2024-04-16T16:14:33.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.889+0000 7fd876a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8780c6fc0 0x7fd878143460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:33.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.889+0000 7fd876a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8780c6fc0 0x7fd878143460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33286/0 (socket says 172.21.15.45:33286) 2024-04-16T16:14:33.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.889+0000 7fd876a00640 1 -- 172.21.15.45:0/3286570420 learned_addr learned my addr 172.21.15.45:0/3286570420 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:33.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.890+0000 7fd876a00640 1 -- 172.21.15.45:0/3286570420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd87815cdc0 msgr2=0x7fd8781439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:33.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.890+0000 7fd876a00640 1 --2- 172.21.15.45:0/3286570420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd87815cdc0 0x7fd8781439a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.890+0000 7fd876a00640 1 -- 172.21.15.45:0/3286570420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8780c6320 msgr2=0x7fd878140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:33.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.890+0000 7fd876a00640 1 --2- 172.21.15.45:0/3286570420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8780c6320 0x7fd878140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:33.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.890+0000 7fd876a00640 1 -- 172.21.15.45:0/3286570420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd878143fd0 con 0x7fd8780c6fc0 2024-04-16T16:14:33.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.890+0000 7fd876a00640 1 --2- 172.21.15.45:0/3286570420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8780c6fc0 0x7fd878143460 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd86402b930 tx=0x7fd8640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:33.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.890+0000 7fd86f400640 1 -- 172.21.15.45:0/3286570420 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd864040070 con 0x7fd8780c6fc0 2024-04-16T16:14:33.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.890+0000 7fd86f400640 1 -- 172.21.15.45:0/3286570420 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd864039bf0 con 0x7fd8780c6fc0 2024-04-16T16:14:33.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.890+0000 7fd86f400640 1 -- 172.21.15.45:0/3286570420 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd864035040 con 0x7fd8780c6fc0 2024-04-16T16:14:33.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.890+0000 7fd87ca00640 1 -- 172.21.15.45:0/3286570420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8781442b0 con 0x7fd8780c6fc0 2024-04-16T16:14:33.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.891+0000 7fd87ca00640 1 -- 172.21.15.45:0/3286570420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8781416b0 con 0x7fd8780c6fc0 2024-04-16T16:14:33.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.892+0000 7fd87ca00640 1 -- 172.21.15.45:0/3286570420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8780b2990 con 0x7fd8780c6fc0 2024-04-16T16:14:33.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.893+0000 7fd86f400640 1 -- 172.21.15.45:0/3286570420 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd86404b060 con 0x7fd8780c6fc0 2024-04-16T16:14:33.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.893+0000 7fd86f400640 1 --2- 172.21.15.45:0/3286570420 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd85806c8f0 0x7fd85806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:33.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.894+0000 7fd86f400640 1 -- 172.21.15.45:0/3286570420 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fd8640ad2f0 con 0x7fd8780c6fc0 2024-04-16T16:14:33.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.894+0000 7fd877400640 1 --2- 172.21.15.45:0/3286570420 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd85806c8f0 0x7fd85806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:33.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.895+0000 7fd877400640 1 --2- 172.21.15.45:0/3286570420 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd85806c8f0 0x7fd85806edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd868006fd0 tx=0x7fd868008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:33.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:33.898+0000 7fd86f400640 1 -- 172.21.15.45:0/3286570420 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd8780b2990 con 0x7fd8780c6fc0 2024-04-16T16:14:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.177+0000 7fd87ca00640 1 -- 172.21.15.45:0/3286570420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7fd878075780 con 0x7fd8780c6fc0 2024-04-16T16:14:34.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.181+0000 7fd86f400640 1 -- 172.21.15.45:0/3286570420 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7fd86407ba60 con 0x7fd8780c6fc0 2024-04-16T16:14:34.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.184+0000 7fd87ca00640 1 -- 172.21.15.45:0/3286570420 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd85806c8f0 msgr2=0x7fd85806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:34.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.184+0000 7fd87ca00640 1 --2- 172.21.15.45:0/3286570420 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd85806c8f0 0x7fd85806edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd868006fd0 tx=0x7fd868008040 comp rx=0 tx=0).stop 2024-04-16T16:14:34.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.184+0000 7fd87ca00640 1 -- 172.21.15.45:0/3286570420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8780c6fc0 msgr2=0x7fd878143460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:34.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.184+0000 7fd87ca00640 1 --2- 172.21.15.45:0/3286570420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8780c6fc0 0x7fd878143460 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd86402b930 tx=0x7fd8640046c0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.184+0000 7fd87ca00640 1 -- 172.21.15.45:0/3286570420 shutdown_connections 2024-04-16T16:14:34.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.184+0000 7fd87ca00640 1 --2- 172.21.15.45:0/3286570420 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fd85806c8f0 0x7fd85806edb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.184+0000 7fd87ca00640 1 --2- 172.21.15.45:0/3286570420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd87815cdc0 0x7fd8781439a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.184+0000 7fd87ca00640 1 --2- 172.21.15.45:0/3286570420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8780c6fc0 0x7fd878143460 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.184+0000 7fd87ca00640 1 --2- 172.21.15.45:0/3286570420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8780c6320 0x7fd878140af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.184+0000 7fd87ca00640 1 -- 172.21.15.45:0/3286570420 >> 172.21.15.45:0/3286570420 conn(0x7fd87806b930 msgr2=0x7fd8780cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:34.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.184+0000 7fd87ca00640 1 -- 172.21.15.45:0/3286570420 shutdown_connections 2024-04-16T16:14:34.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.184+0000 7fd87ca00640 1 -- 172.21.15.45:0/3286570420 wait complete. 2024-04-16T16:14:34.204 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-16T16:14:34.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.424+0000 7f465da00640 1 Processor -- start 2024-04-16T16:14:34.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.424+0000 7f465da00640 1 -- start start 2024-04-16T16:14:34.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.426+0000 7f465da00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f46580cb7d0 0x7f46580cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:34.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.426+0000 7f465da00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f46580cbc80 0x7f46580cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:34.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.426+0000 7f465da00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f465815f1b0 con 0x7f465815e830 2024-04-16T16:14:34.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.426+0000 7f465da00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f46580cf280 con 0x7f46580cbc80 2024-04-16T16:14:34.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.426+0000 7f465da00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f46580ce190 con 0x7f46580cb7d0 2024-04-16T16:14:34.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.426+0000 7f4657e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f465815e830 0x7f46580ce9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:34510/0 (socket says 172.21.15.45:34510) 2024-04-16T16:14:34.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.426+0000 7f4657e00640 1 -- 172.21.15.45:0/1863440834 learned_addr learned my addr 172.21.15.45:0/1863440834 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:34.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.426+0000 7f465d000640 1 --2- 172.21.15.45:0/1863440834 >> v2:172.21.15.45:3300/0 conn(0x7f46580cbc80 0x7f46580cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:34.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.427+0000 7f464f400640 1 --2- 172.21.15.45:0/1863440834 >> v2:172.21.15.103:3300/0 conn(0x7f46580cb7d0 0x7f46580cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:34.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.427+0000 7f465d000640 1 -- 172.21.15.45:0/1863440834 >> v2:172.21.15.103:3300/0 conn(0x7f46580cb7d0 msgr2=0x7f46580cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:34.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.427+0000 7f465d000640 1 --2- 172.21.15.45:0/1863440834 >> v2:172.21.15.103:3300/0 conn(0x7f46580cb7d0 0x7f46580cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.427+0000 7f465d000640 1 -- 172.21.15.45:0/1863440834 >> v1:172.21.15.45:6789/0 conn(0x7f465815e830 legacy=0x7f46580ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:34.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.427+0000 7f465d000640 1 -- 172.21.15.45:0/1863440834 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46580ce470 con 0x7f46580cbc80 2024-04-16T16:14:34.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.427+0000 7f4657400640 1 -- 172.21.15.45:0/1863440834 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4133718754 0 0) 0x7f4648001110 con 0x7f465815e830 2024-04-16T16:14:34.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.428+0000 7f465d000640 1 --2- 172.21.15.45:0/1863440834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46580cbc80 0x7f46580cdb60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4658073810 tx=0x7f465002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6afcdfd6d3a52d7d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:34.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.428+0000 7f4657400640 1 -- 172.21.15.45:0/1863440834 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4650039070 con 0x7f46580cbc80 2024-04-16T16:14:34.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.428+0000 7f4657400640 1 -- 172.21.15.45:0/1863440834 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f465002cbf0 con 0x7f46580cbc80 2024-04-16T16:14:34.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.428+0000 7f4657400640 1 -- 172.21.15.45:0/1863440834 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4650034440 con 0x7f46580cbc80 2024-04-16T16:14:34.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.429+0000 7f465da00640 1 -- 172.21.15.45:0/1863440834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46580cbc80 msgr2=0x7f46580cdb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:34.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.429+0000 7f465da00640 1 --2- 172.21.15.45:0/1863440834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46580cbc80 0x7f46580cdb60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4658073810 tx=0x7f465002be40 comp rx=0 tx=0).stop 2024-04-16T16:14:34.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.429+0000 7f465da00640 1 -- 172.21.15.45:0/1863440834 shutdown_connections 2024-04-16T16:14:34.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.429+0000 7f465da00640 1 --2- 172.21.15.45:0/1863440834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46580cbc80 0x7f46580cdb60 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.429+0000 7f465da00640 1 --2- 172.21.15.45:0/1863440834 >> v2:172.21.15.103:3300/0 conn(0x7f46580cb7d0 0x7f46580cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.429+0000 7f465da00640 1 -- 172.21.15.45:0/1863440834 >> 172.21.15.45:0/1863440834 conn(0x7f465806b930 msgr2=0x7f465806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:34.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.429+0000 7f465da00640 1 -- 172.21.15.45:0/1863440834 shutdown_connections 2024-04-16T16:14:34.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.429+0000 7f465da00640 1 -- 172.21.15.45:0/1863440834 wait complete. 2024-04-16T16:14:34.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.431+0000 7f465da00640 1 Processor -- start 2024-04-16T16:14:34.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.431+0000 7f465da00640 1 -- start start 2024-04-16T16:14:34.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.432+0000 7f465da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46580cb7d0 0x7f465815a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:34.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.432+0000 7f465da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46580cbc80 0x7f465815d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:34.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.432+0000 7f465da00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815e830 0x7f465815d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:34.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.432+0000 7f465da00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f465815ec10 con 0x7f46580cb7d0 2024-04-16T16:14:34.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.432+0000 7f465da00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f46580cb2b0 con 0x7f46580cbc80 2024-04-16T16:14:34.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.432+0000 7f465da00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f46580cac30 con 0x7f465815e830 2024-04-16T16:14:34.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.432+0000 7f4657e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46580cb7d0 0x7f465815a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.432+0000 7f4657e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46580cb7d0 0x7f465815a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33306/0 (socket says 172.21.15.45:33306) 2024-04-16T16:14:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.432+0000 7f4657e00640 1 -- 172.21.15.45:0/450752973 learned_addr learned my addr 172.21.15.45:0/450752973 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.432+0000 7f464f400640 1 --2- 172.21.15.45:0/450752973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46580cbc80 0x7f465815d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.433+0000 7f465d000640 1 --2- 172.21.15.45:0/450752973 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815e830 0x7f465815d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.433+0000 7f4657e00640 1 -- 172.21.15.45:0/450752973 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815e830 msgr2=0x7f465815d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.433+0000 7f4657e00640 1 --2- 172.21.15.45:0/450752973 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815e830 0x7f465815d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.433+0000 7f4657e00640 1 -- 172.21.15.45:0/450752973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46580cbc80 msgr2=0x7f465815d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.433+0000 7f4657e00640 1 --2- 172.21.15.45:0/450752973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46580cbc80 0x7f465815d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.433+0000 7f4657e00640 1 -- 172.21.15.45:0/450752973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46581f7e40 con 0x7f46580cb7d0 2024-04-16T16:14:34.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.433+0000 7f464f400640 1 --2- 172.21.15.45:0/450752973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46580cbc80 0x7f465815d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:34.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.433+0000 7f4657e00640 1 --2- 172.21.15.45:0/450752973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46580cb7d0 0x7f465815a4f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f464800c780 tx=0x7f464800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:34.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.433+0000 7f4654c00640 1 -- 172.21.15.45:0/450752973 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4648018070 con 0x7f46580cb7d0 2024-04-16T16:14:34.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.433+0000 7f4654c00640 1 -- 172.21.15.45:0/450752973 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46480072d0 con 0x7f46580cb7d0 2024-04-16T16:14:34.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.433+0000 7f465da00640 1 -- 172.21.15.45:0/450752973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46581f8120 con 0x7f46580cb7d0 2024-04-16T16:14:34.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.433+0000 7f4654c00640 1 -- 172.21.15.45:0/450752973 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46480047e0 con 0x7f46580cb7d0 2024-04-16T16:14:34.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.433+0000 7f465da00640 1 -- 172.21.15.45:0/450752973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46581f86e0 con 0x7f46580cb7d0 2024-04-16T16:14:34.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.434+0000 7f465da00640 1 -- 172.21.15.45:0/450752973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4658075780 con 0x7f46580cb7d0 2024-04-16T16:14:34.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.436+0000 7f4654c00640 1 -- 172.21.15.45:0/450752973 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f464801b070 con 0x7f46580cb7d0 2024-04-16T16:14:34.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.437+0000 7f4654c00640 1 --2- 172.21.15.45:0/450752973 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f463406c8f0 0x7f463406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:34.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.437+0000 7f4654c00640 1 -- 172.21.15.45:0/450752973 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f464808abf0 con 0x7f46580cb7d0 2024-04-16T16:14:34.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.437+0000 7f464f400640 1 --2- 172.21.15.45:0/450752973 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f463406c8f0 0x7f463406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:34.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.438+0000 7f464f400640 1 --2- 172.21.15.45:0/450752973 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f463406c8f0 0x7f463406edb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4640006fd0 tx=0x7f4640008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:34.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.441+0000 7f4654c00640 1 -- 172.21.15.45:0/450752973 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f465815f1b0 con 0x7f46580cb7d0 2024-04-16T16:14:34.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.717+0000 7f465da00640 1 -- 172.21.15.45:0/450752973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f46580b5b90 con 0x7f46580cb7d0 2024-04-16T16:14:34.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.722+0000 7f4654c00640 1 -- 172.21.15.45:0/450752973 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7f46480592d0 con 0x7f46580cb7d0 2024-04-16T16:14:34.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.724+0000 7f465da00640 1 -- 172.21.15.45:0/450752973 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f463406c8f0 msgr2=0x7f463406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:34.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.724+0000 7f465da00640 1 --2- 172.21.15.45:0/450752973 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f463406c8f0 0x7f463406edb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4640006fd0 tx=0x7f4640008040 comp rx=0 tx=0).stop 2024-04-16T16:14:34.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.724+0000 7f465da00640 1 -- 172.21.15.45:0/450752973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46580cb7d0 msgr2=0x7f465815a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:34.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.724+0000 7f465da00640 1 --2- 172.21.15.45:0/450752973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46580cb7d0 0x7f465815a4f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f464800c780 tx=0x7f464800cc40 comp rx=0 tx=0).stop 2024-04-16T16:14:34.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.724+0000 7f465da00640 1 -- 172.21.15.45:0/450752973 shutdown_connections 2024-04-16T16:14:34.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.724+0000 7f465da00640 1 --2- 172.21.15.45:0/450752973 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f463406c8f0 0x7f463406edb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.724+0000 7f465da00640 1 --2- 172.21.15.45:0/450752973 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815e830 0x7f465815d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.724+0000 7f465da00640 1 --2- 172.21.15.45:0/450752973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46580cbc80 0x7f465815d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.724+0000 7f465da00640 1 --2- 172.21.15.45:0/450752973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46580cb7d0 0x7f465815a4f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.724+0000 7f465da00640 1 -- 172.21.15.45:0/450752973 >> 172.21.15.45:0/450752973 conn(0x7f465806b930 msgr2=0x7f46580cd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:34.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.725+0000 7f465da00640 1 -- 172.21.15.45:0/450752973 shutdown_connections 2024-04-16T16:14:34.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.725+0000 7f465da00640 1 -- 172.21.15.45:0/450752973 wait complete. 2024-04-16T16:14:34.741 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-16T16:14:34.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.942+0000 7f43a5c00640 1 Processor -- start 2024-04-16T16:14:34.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.942+0000 7f43a5c00640 1 -- start start 2024-04-16T16:14:34.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.944+0000 7f43a5c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f43a006f730 0x7f43a00c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:34.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.944+0000 7f43a5c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f43a00c6320 0x7f43a006df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:34.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.944+0000 7f43a5c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43a015d600 con 0x7f43a00c6fc0 2024-04-16T16:14:34.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.944+0000 7f43a5c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f43a00c5670 con 0x7f43a00c6320 2024-04-16T16:14:34.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.944+0000 7f43a5c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f43a015d8e0 con 0x7f43a006f730 2024-04-16T16:14:34.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.944+0000 7f439fe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f43a00c6fc0 0x7f43a015cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:34516/0 (socket says 172.21.15.45:34516) 2024-04-16T16:14:34.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.944+0000 7f439fe00640 1 -- 172.21.15.45:0/180226204 learned_addr learned my addr 172.21.15.45:0/180226204 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:34.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.944+0000 7f43a5200640 1 --2- 172.21.15.45:0/180226204 >> v2:172.21.15.45:3300/0 conn(0x7f43a00c6320 0x7f43a006df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:34.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.945+0000 7f4397e00640 1 --2- 172.21.15.45:0/180226204 >> v2:172.21.15.103:3300/0 conn(0x7f43a006f730 0x7f43a00c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:34.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.945+0000 7f439f400640 1 -- 172.21.15.45:0/180226204 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1476877656 0 0) 0x7f43a015d600 con 0x7f43a00c6fc0 2024-04-16T16:14:34.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.945+0000 7f439f400640 1 -- 172.21.15.45:0/180226204 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4380003520 con 0x7f43a00c6fc0 2024-04-16T16:14:34.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.945+0000 7f43a5200640 1 -- 172.21.15.45:0/180226204 >> v2:172.21.15.103:3300/0 conn(0x7f43a006f730 msgr2=0x7f43a00c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:34.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.945+0000 7f43a5200640 1 --2- 172.21.15.45:0/180226204 >> v2:172.21.15.103:3300/0 conn(0x7f43a006f730 0x7f43a00c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.945+0000 7f43a5200640 1 -- 172.21.15.45:0/180226204 >> v1:172.21.15.45:6789/0 conn(0x7f43a00c6fc0 legacy=0x7f43a015cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:34.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.945+0000 7f43a5200640 1 -- 172.21.15.45:0/180226204 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a015dbc0 con 0x7f43a00c6320 2024-04-16T16:14:34.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.945+0000 7f4397e00640 1 --2- 172.21.15.45:0/180226204 >> v2:172.21.15.103:3300/0 conn(0x7f43a006f730 0x7f43a00c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:34.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.946+0000 7f43a5200640 1 --2- 172.21.15.45:0/180226204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43a00c6320 0x7f43a006df40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f43a0073370 tx=0x7f439802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a60b9c2ff04a6d63 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:34.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.946+0000 7f439f400640 1 -- 172.21.15.45:0/180226204 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4398039070 con 0x7f43a00c6320 2024-04-16T16:14:34.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.947+0000 7f439f400640 1 -- 172.21.15.45:0/180226204 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f439802cbf0 con 0x7f43a00c6320 2024-04-16T16:14:34.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.947+0000 7f439f400640 1 -- 172.21.15.45:0/180226204 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4398034440 con 0x7f43a00c6320 2024-04-16T16:14:34.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.947+0000 7f43a5c00640 1 -- 172.21.15.45:0/180226204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43a00c6320 msgr2=0x7f43a006df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:34.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.947+0000 7f43a5c00640 1 --2- 172.21.15.45:0/180226204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43a00c6320 0x7f43a006df40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f43a0073370 tx=0x7f439802be40 comp rx=0 tx=0).stop 2024-04-16T16:14:34.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.947+0000 7f43a5c00640 1 -- 172.21.15.45:0/180226204 shutdown_connections 2024-04-16T16:14:34.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.947+0000 7f43a5c00640 1 --2- 172.21.15.45:0/180226204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43a00c6320 0x7f43a006df40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.947+0000 7f43a5c00640 1 --2- 172.21.15.45:0/180226204 >> v2:172.21.15.103:3300/0 conn(0x7f43a006f730 0x7f43a00c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.947+0000 7f43a5c00640 1 -- 172.21.15.45:0/180226204 >> 172.21.15.45:0/180226204 conn(0x7f43a006b930 msgr2=0x7f43a006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:34.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.947+0000 7f43a5c00640 1 -- 172.21.15.45:0/180226204 shutdown_connections 2024-04-16T16:14:34.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.947+0000 7f43a5c00640 1 -- 172.21.15.45:0/180226204 wait complete. 2024-04-16T16:14:34.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.949+0000 7f43a5c00640 1 Processor -- start 2024-04-16T16:14:34.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.950+0000 7f43a5c00640 1 -- start start 2024-04-16T16:14:34.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.951+0000 7f43a5c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a006f730 0x7f43a01f8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:34.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.951+0000 7f43a5c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43a00c6320 0x7f43a01f1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:34.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.951+0000 7f43a5c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43a00c6fc0 0x7f43a01556c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:34.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.951+0000 7f43a5c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f43a00c7520 con 0x7f43a00c6fc0 2024-04-16T16:14:34.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.951+0000 7f43a5c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f43a00c73a0 con 0x7f43a00c6320 2024-04-16T16:14:34.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.951+0000 7f43a5c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f43a006e480 con 0x7f43a006f730 2024-04-16T16:14:34.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.951+0000 7f43a5200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43a00c6fc0 0x7f43a01556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:34.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.951+0000 7f43a5200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43a00c6fc0 0x7f43a01556c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33324/0 (socket says 172.21.15.45:33324) 2024-04-16T16:14:34.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.951+0000 7f43a5200640 1 -- 172.21.15.45:0/323251821 learned_addr learned my addr 172.21.15.45:0/323251821 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:34.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.951+0000 7f439fe00640 1 --2- 172.21.15.45:0/323251821 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a006f730 0x7f43a01f8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:34.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.952+0000 7f4397e00640 1 --2- 172.21.15.45:0/323251821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43a00c6320 0x7f43a01f1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:34.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.952+0000 7f43a5200640 1 -- 172.21.15.45:0/323251821 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a006f730 msgr2=0x7f43a01f8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:34.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.952+0000 7f43a5200640 1 --2- 172.21.15.45:0/323251821 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a006f730 0x7f43a01f8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.952+0000 7f43a5200640 1 -- 172.21.15.45:0/323251821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43a00c6320 msgr2=0x7f43a01f1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:34.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.952+0000 7f43a5200640 1 --2- 172.21.15.45:0/323251821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43a00c6320 0x7f43a01f1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:34.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.952+0000 7f43a5200640 1 -- 172.21.15.45:0/323251821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a01f8870 con 0x7f43a00c6fc0 2024-04-16T16:14:34.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.952+0000 7f4397e00640 1 --2- 172.21.15.45:0/323251821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43a00c6320 0x7f43a01f1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:34.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.952+0000 7f43a5200640 1 --2- 172.21.15.45:0/323251821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43a00c6fc0 0x7f43a01556c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4398009e80 tx=0x7f4398004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:34.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.953+0000 7f439cc00640 1 -- 172.21.15.45:0/323251821 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4398039050 con 0x7f43a00c6fc0 2024-04-16T16:14:34.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.953+0000 7f439cc00640 1 -- 172.21.15.45:0/323251821 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4398045070 con 0x7f43a00c6fc0 2024-04-16T16:14:34.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.953+0000 7f439cc00640 1 -- 172.21.15.45:0/323251821 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439803e830 con 0x7f43a00c6fc0 2024-04-16T16:14:34.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.953+0000 7f43a5c00640 1 -- 172.21.15.45:0/323251821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43a01f52e0 con 0x7f43a00c6fc0 2024-04-16T16:14:34.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.953+0000 7f43a5c00640 1 -- 172.21.15.45:0/323251821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43a01f49b0 con 0x7f43a00c6fc0 2024-04-16T16:14:34.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.954+0000 7f43a5c00640 1 -- 172.21.15.45:0/323251821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43a0075780 con 0x7f43a00c6fc0 2024-04-16T16:14:34.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.955+0000 7f439cc00640 1 -- 172.21.15.45:0/323251821 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f439803e9d0 con 0x7f43a00c6fc0 2024-04-16T16:14:34.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.956+0000 7f439cc00640 1 --2- 172.21.15.45:0/323251821 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f437c06c920 0x7f437c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:34.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.956+0000 7f439cc00640 1 -- 172.21.15.45:0/323251821 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f43980ae4f0 con 0x7f43a00c6fc0 2024-04-16T16:14:34.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.957+0000 7f439fe00640 1 --2- 172.21.15.45:0/323251821 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f437c06c920 0x7f437c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:34.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.958+0000 7f439fe00640 1 --2- 172.21.15.45:0/323251821 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f437c06c920 0x7f437c06ede0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4390007fd0 tx=0x7f4390009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:34.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:34.961+0000 7f439cc00640 1 -- 172.21.15.45:0/323251821 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f43a01f8870 con 0x7f43a00c6fc0 2024-04-16T16:14:35.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.244+0000 7f43a5c00640 1 -- 172.21.15.45:0/323251821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f43a00b3df0 con 0x7f43a00c6fc0 2024-04-16T16:14:35.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.249+0000 7f439cc00640 1 -- 172.21.15.45:0/323251821 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f4398032e20 con 0x7f43a00c6fc0 2024-04-16T16:14:35.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.252+0000 7f43a5c00640 1 -- 172.21.15.45:0/323251821 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f437c06c920 msgr2=0x7f437c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:35.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.252+0000 7f43a5c00640 1 --2- 172.21.15.45:0/323251821 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f437c06c920 0x7f437c06ede0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4390007fd0 tx=0x7f4390009040 comp rx=0 tx=0).stop 2024-04-16T16:14:35.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.252+0000 7f43a5c00640 1 -- 172.21.15.45:0/323251821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43a00c6fc0 msgr2=0x7f43a01556c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:35.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.252+0000 7f43a5c00640 1 --2- 172.21.15.45:0/323251821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43a00c6fc0 0x7f43a01556c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4398009e80 tx=0x7f4398004a40 comp rx=0 tx=0).stop 2024-04-16T16:14:35.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.252+0000 7f43a5c00640 1 -- 172.21.15.45:0/323251821 shutdown_connections 2024-04-16T16:14:35.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.252+0000 7f43a5c00640 1 --2- 172.21.15.45:0/323251821 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f437c06c920 0x7f437c06ede0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.252+0000 7f43a5c00640 1 --2- 172.21.15.45:0/323251821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43a00c6fc0 0x7f43a01556c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.252+0000 7f43a5c00640 1 --2- 172.21.15.45:0/323251821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43a00c6320 0x7f43a01f1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.252+0000 7f43a5c00640 1 --2- 172.21.15.45:0/323251821 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a006f730 0x7f43a01f8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.252+0000 7f43a5c00640 1 -- 172.21.15.45:0/323251821 >> 172.21.15.45:0/323251821 conn(0x7f43a006b930 msgr2=0x7f43a00b36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:35.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.252+0000 7f43a5c00640 1 -- 172.21.15.45:0/323251821 shutdown_connections 2024-04-16T16:14:35.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.252+0000 7f43a5c00640 1 -- 172.21.15.45:0/323251821 wait complete. 2024-04-16T16:14:35.271 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:35.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.489+0000 7fc136c00640 1 Processor -- start 2024-04-16T16:14:35.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.489+0000 7fc136c00640 1 -- start start 2024-04-16T16:14:35.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.490+0000 7fc136c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc130161e90 0x7fc1300cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:35.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.490+0000 7fc136c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1300d52d0 0x7fc13006ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:35.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.490+0000 7fc136c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1300d3080 con 0x7fc130162270 2024-04-16T16:14:35.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.490+0000 7fc136c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc1300d2e30 con 0x7fc130161e90 2024-04-16T16:14:35.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.490+0000 7fc136c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc1300d3360 con 0x7fc1300d52d0 2024-04-16T16:14:35.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.490+0000 7fc135800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc130161e90 0x7fc1300cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:35.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.490+0000 7fc134e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc130162270 0x7fc1300d3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:34518/0 (socket says 172.21.15.45:34518) 2024-04-16T16:14:35.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.490+0000 7fc134e00640 1 -- 172.21.15.45:0/3724308890 learned_addr learned my addr 172.21.15.45:0/3724308890 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:35.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.491+0000 7fc135800640 1 -- 172.21.15.45:0/3724308890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1300d52d0 msgr2=0x7fc13006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:35.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.491+0000 7fc136200640 1 --2- 172.21.15.45:0/3724308890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1300d52d0 0x7fc13006ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:35.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.491+0000 7fc12fe00640 1 -- 172.21.15.45:0/3724308890 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4020006501 0 0) 0x7fc1300d3080 con 0x7fc130162270 2024-04-16T16:14:35.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.491+0000 7fc135800640 1 --2- 172.21.15.45:0/3724308890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1300d52d0 0x7fc13006ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.491+0000 7fc135800640 1 -- 172.21.15.45:0/3724308890 >> v1:172.21.15.45:6789/0 conn(0x7fc130162270 legacy=0x7fc1300d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:35.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.491+0000 7fc135800640 1 -- 172.21.15.45:0/3724308890 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1300d3640 con 0x7fc130161e90 2024-04-16T16:14:35.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.491+0000 7fc136200640 1 --2- 172.21.15.45:0/3724308890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1300d52d0 0x7fc13006ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:35.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.492+0000 7fc135800640 1 --2- 172.21.15.45:0/3724308890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc130161e90 0x7fc1300cf7d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc1200097d0 tx=0x7fc12002b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7d5583e90013b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:35.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.493+0000 7fc12fe00640 1 -- 172.21.15.45:0/3724308890 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc120034070 con 0x7fc130161e90 2024-04-16T16:14:35.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.493+0000 7fc12fe00640 1 -- 172.21.15.45:0/3724308890 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc120030800 con 0x7fc130161e90 2024-04-16T16:14:35.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.493+0000 7fc12fe00640 1 -- 172.21.15.45:0/3724308890 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc120030bd0 con 0x7fc130161e90 2024-04-16T16:14:35.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.493+0000 7fc136c00640 1 -- 172.21.15.45:0/3724308890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc130161e90 msgr2=0x7fc1300cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:35.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.493+0000 7fc136c00640 1 --2- 172.21.15.45:0/3724308890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc130161e90 0x7fc1300cf7d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc1200097d0 tx=0x7fc12002b940 comp rx=0 tx=0).stop 2024-04-16T16:14:35.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.493+0000 7fc136c00640 1 -- 172.21.15.45:0/3724308890 shutdown_connections 2024-04-16T16:14:35.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.493+0000 7fc136c00640 1 --2- 172.21.15.45:0/3724308890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1300d52d0 0x7fc13006ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.493+0000 7fc136c00640 1 --2- 172.21.15.45:0/3724308890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc130161e90 0x7fc1300cf7d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.493+0000 7fc136c00640 1 -- 172.21.15.45:0/3724308890 >> 172.21.15.45:0/3724308890 conn(0x7fc1300d1940 msgr2=0x7fc1300d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:35.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.494+0000 7fc136c00640 1 -- 172.21.15.45:0/3724308890 shutdown_connections 2024-04-16T16:14:35.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.494+0000 7fc136c00640 1 -- 172.21.15.45:0/3724308890 wait complete. 2024-04-16T16:14:35.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.495+0000 7fc136c00640 1 Processor -- start 2024-04-16T16:14:35.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.495+0000 7fc136c00640 1 -- start start 2024-04-16T16:14:35.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.496+0000 7fc136c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1300d52d0 0x7fc13006d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:35.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.496+0000 7fc136c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc130161e90 0x7fc130150750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:35.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.496+0000 7fc136c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc130162270 0x7fc130150c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:35.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.496+0000 7fc136c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc1300d4550 con 0x7fc1300d52d0 2024-04-16T16:14:35.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.496+0000 7fc136c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc1300cfd10 con 0x7fc130161e90 2024-04-16T16:14:35.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.496+0000 7fc136c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc1300d46d0 con 0x7fc130162270 2024-04-16T16:14:35.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.497+0000 7fc135800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1300d52d0 0x7fc13006d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:35.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.497+0000 7fc135800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1300d52d0 0x7fc13006d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33346/0 (socket says 172.21.15.45:33346) 2024-04-16T16:14:35.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.497+0000 7fc135800640 1 -- 172.21.15.45:0/1010644507 learned_addr learned my addr 172.21.15.45:0/1010644507 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:35.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.497+0000 7fc135800640 1 -- 172.21.15.45:0/1010644507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc130162270 msgr2=0x7fc130150c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:35.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.497+0000 7fc136200640 1 --2- 172.21.15.45:0/1010644507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc130162270 0x7fc130150c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:35.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.497+0000 7fc134e00640 1 --2- 172.21.15.45:0/1010644507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc130161e90 0x7fc130150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:35.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.497+0000 7fc135800640 1 --2- 172.21.15.45:0/1010644507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc130162270 0x7fc130150c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.497+0000 7fc135800640 1 -- 172.21.15.45:0/1010644507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc130161e90 msgr2=0x7fc130150750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:35.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.497+0000 7fc135800640 1 --2- 172.21.15.45:0/1010644507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc130161e90 0x7fc130150750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.497+0000 7fc135800640 1 -- 172.21.15.45:0/1010644507 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1301f7c70 con 0x7fc1300d52d0 2024-04-16T16:14:35.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.498+0000 7fc135800640 1 --2- 172.21.15.45:0/1010644507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1300d52d0 0x7fc13006d0b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc12002bf50 tx=0x7fc1200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:35.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.498+0000 7fc12d600640 1 -- 172.21.15.45:0/1010644507 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc120034040 con 0x7fc1300d52d0 2024-04-16T16:14:35.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.498+0000 7fc12d600640 1 -- 172.21.15.45:0/1010644507 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc120039d10 con 0x7fc1300d52d0 2024-04-16T16:14:35.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.498+0000 7fc12d600640 1 -- 172.21.15.45:0/1010644507 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc120042610 con 0x7fc1300d52d0 2024-04-16T16:14:35.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.498+0000 7fc136c00640 1 -- 172.21.15.45:0/1010644507 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1301f7f50 con 0x7fc1300d52d0 2024-04-16T16:14:35.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.498+0000 7fc136c00640 1 -- 172.21.15.45:0/1010644507 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1301f8510 con 0x7fc1300d52d0 2024-04-16T16:14:35.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.499+0000 7fc136c00640 1 -- 172.21.15.45:0/1010644507 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc130075780 con 0x7fc1300d52d0 2024-04-16T16:14:35.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.500+0000 7fc12d600640 1 -- 172.21.15.45:0/1010644507 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc120037070 con 0x7fc1300d52d0 2024-04-16T16:14:35.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.501+0000 7fc12d600640 1 --2- 172.21.15.45:0/1010644507 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc11406c8f0 0x7fc11406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:35.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.502+0000 7fc12d600640 1 -- 172.21.15.45:0/1010644507 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fc1200adb20 con 0x7fc1300d52d0 2024-04-16T16:14:35.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.502+0000 7fc134e00640 1 --2- 172.21.15.45:0/1010644507 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc11406c8f0 0x7fc11406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:35.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.504+0000 7fc134e00640 1 --2- 172.21.15.45:0/1010644507 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc11406c8f0 0x7fc11406edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc130157480 tx=0x7fc124009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:35.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.506+0000 7fc12d600640 1 -- 172.21.15.45:0/1010644507 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc12007c320 con 0x7fc1300d52d0 2024-04-16T16:14:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.717+0000 7fc136c00640 1 -- 172.21.15.45:0/1010644507 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc13006dae0 con 0x7fc1300d52d0 2024-04-16T16:14:35.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.719+0000 7fc12d600640 1 -- 172.21.15.45:0/1010644507 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+12891 (secure 0 0 0) 0x7fc12007bc20 con 0x7fc1300d52d0 2024-04-16T16:14:35.719 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:35.719 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":30,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:14:31.538243+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:35.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.722+0000 7fc136c00640 1 -- 172.21.15.45:0/1010644507 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc11406c8f0 msgr2=0x7fc11406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:35.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.722+0000 7fc136c00640 1 --2- 172.21.15.45:0/1010644507 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc11406c8f0 0x7fc11406edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc130157480 tx=0x7fc124009040 comp rx=0 tx=0).stop 2024-04-16T16:14:35.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.722+0000 7fc136c00640 1 -- 172.21.15.45:0/1010644507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1300d52d0 msgr2=0x7fc13006d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:35.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.722+0000 7fc136c00640 1 --2- 172.21.15.45:0/1010644507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1300d52d0 0x7fc13006d0b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc12002bf50 tx=0x7fc1200046c0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.722+0000 7fc136c00640 1 -- 172.21.15.45:0/1010644507 shutdown_connections 2024-04-16T16:14:35.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.722+0000 7fc136c00640 1 --2- 172.21.15.45:0/1010644507 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc11406c8f0 0x7fc11406edb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.722+0000 7fc136c00640 1 --2- 172.21.15.45:0/1010644507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc130162270 0x7fc130150c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.722+0000 7fc136c00640 1 --2- 172.21.15.45:0/1010644507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc130161e90 0x7fc130150750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.722+0000 7fc136c00640 1 --2- 172.21.15.45:0/1010644507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1300d52d0 0x7fc13006d0b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.722+0000 7fc136c00640 1 -- 172.21.15.45:0/1010644507 >> 172.21.15.45:0/1010644507 conn(0x7fc1300d1940 msgr2=0x7fc1300b9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:35.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.722+0000 7fc136c00640 1 -- 172.21.15.45:0/1010644507 shutdown_connections 2024-04-16T16:14:35.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.722+0000 7fc136c00640 1 -- 172.21.15.45:0/1010644507 wait complete. 2024-04-16T16:14:35.743 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:14:35.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:14:35.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.969+0000 7ff012600640 1 Processor -- start 2024-04-16T16:14:35.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.970+0000 7ff012600640 1 -- start start 2024-04-16T16:14:35.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.971+0000 7ff012600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff00c0cfe00 0x7ff00c0d2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:35.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.971+0000 7ff012600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff00c0d4640 con 0x7ff00c0d1f10 2024-04-16T16:14:35.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.971+0000 7ff012600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff00c0ba4c0 con 0x7ff00c1622d0 2024-04-16T16:14:35.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.971+0000 7ff012600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff00c0d43f0 con 0x7ff00c0cfe00 2024-04-16T16:14:35.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.971+0000 7ff00aa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff00c0cfe00 0x7ff00c0d2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:35.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.971+0000 7ff00aa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff00c0cfe00 0x7ff00c0d2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33362/0 (socket says 172.21.15.45:33362) 2024-04-16T16:14:35.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.971+0000 7ff00aa00640 1 -- 172.21.15.45:0/462219862 learned_addr learned my addr 172.21.15.45:0/462219862 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:35.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.972+0000 7ff00aa00640 1 -- 172.21.15.45:0/462219862 >> v1:172.21.15.103:6789/0 conn(0x7ff00c1622d0 legacy=0x7ff00c0cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:35.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.972+0000 7ff00aa00640 1 -- 172.21.15.45:0/462219862 >> v1:172.21.15.45:6789/0 conn(0x7ff00c0d1f10 legacy=0x7ff00c0d2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:35.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.972+0000 7ff00aa00640 1 -- 172.21.15.45:0/462219862 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff00c0b7b50 con 0x7ff00c0cfe00 2024-04-16T16:14:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.973+0000 7ff00aa00640 1 --2- 172.21.15.45:0/462219862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0cfe00 0x7ff00c0d2680 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7feff4009870 tx=0x7feff402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5e63f2ec6fa089 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.973+0000 7ff00a000640 1 -- 172.21.15.45:0/462219862 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff402f850 con 0x7ff00c0cfe00 2024-04-16T16:14:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.973+0000 7ff00a000640 1 -- 172.21.15.45:0/462219862 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feff402f9f0 con 0x7ff00c0cfe00 2024-04-16T16:14:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.973+0000 7ff00a000640 1 -- 172.21.15.45:0/462219862 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff402fdc0 con 0x7ff00c0cfe00 2024-04-16T16:14:35.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.974+0000 7ff012600640 1 -- 172.21.15.45:0/462219862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0cfe00 msgr2=0x7ff00c0d2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:35.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.974+0000 7ff012600640 1 --2- 172.21.15.45:0/462219862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0cfe00 0x7ff00c0d2680 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7feff4009870 tx=0x7feff402ba30 comp rx=0 tx=0).stop 2024-04-16T16:14:35.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.974+0000 7ff012600640 1 -- 172.21.15.45:0/462219862 shutdown_connections 2024-04-16T16:14:35.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.974+0000 7ff012600640 1 --2- 172.21.15.45:0/462219862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0cfe00 0x7ff00c0d2680 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.974+0000 7ff012600640 1 -- 172.21.15.45:0/462219862 >> 172.21.15.45:0/462219862 conn(0x7ff00c06d400 msgr2=0x7ff00c06d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:35.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.974+0000 7ff012600640 1 -- 172.21.15.45:0/462219862 shutdown_connections 2024-04-16T16:14:35.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.974+0000 7ff012600640 1 -- 172.21.15.45:0/462219862 wait complete. 2024-04-16T16:14:35.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.976+0000 7ff012600640 1 Processor -- start 2024-04-16T16:14:35.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.976+0000 7ff012600640 1 -- start start 2024-04-16T16:14:35.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.977+0000 7ff012600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff00c0cfe00 0x7ff00c0b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:35.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.977+0000 7ff012600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0d1f10 0x7ff00c0cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:35.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.977+0000 7ff012600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff00c1622d0 0x7ff00c0cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:35.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.977+0000 7ff012600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c0b70e0 con 0x7ff00c0d1f10 2024-04-16T16:14:35.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.977+0000 7ff012600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c0b6f60 con 0x7ff00c0cfe00 2024-04-16T16:14:35.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.977+0000 7ff012600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff00c0d32d0 con 0x7ff00c1622d0 2024-04-16T16:14:35.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.977+0000 7ff00aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0d1f10 0x7ff00c0cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:35.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.977+0000 7ff00aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0d1f10 0x7ff00c0cd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33366/0 (socket says 172.21.15.45:33366) 2024-04-16T16:14:35.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.977+0000 7ff00aa00640 1 -- 172.21.15.45:0/4237902611 learned_addr learned my addr 172.21.15.45:0/4237902611 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.978+0000 7ff00aa00640 1 -- 172.21.15.45:0/4237902611 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff00c1622d0 msgr2=0x7ff00c0cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.978+0000 7ff00aa00640 1 --2- 172.21.15.45:0/4237902611 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff00c1622d0 0x7ff00c0cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.978+0000 7ff00aa00640 1 -- 172.21.15.45:0/4237902611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff00c0cfe00 msgr2=0x7ff00c0b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.978+0000 7ff00aa00640 1 --2- 172.21.15.45:0/4237902611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff00c0cfe00 0x7ff00c0b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.978+0000 7ff00aa00640 1 -- 172.21.15.45:0/4237902611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff00c0ce010 con 0x7ff00c0d1f10 2024-04-16T16:14:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.978+0000 7ff00aa00640 1 --2- 172.21.15.45:0/4237902611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0d1f10 0x7ff00c0cd980 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7feff4000c00 tx=0x7feff4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.978+0000 7ff003400640 1 -- 172.21.15.45:0/4237902611 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff402f840 con 0x7ff00c0d1f10 2024-04-16T16:14:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.978+0000 7ff003400640 1 -- 172.21.15.45:0/4237902611 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feff4046070 con 0x7ff00c0d1f10 2024-04-16T16:14:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.978+0000 7ff003400640 1 -- 172.21.15.45:0/4237902611 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff4041620 con 0x7ff00c0d1f10 2024-04-16T16:14:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.979+0000 7ff012600640 1 -- 172.21.15.45:0/4237902611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff00c0ce2f0 con 0x7ff00c0d1f10 2024-04-16T16:14:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.979+0000 7ff012600640 1 -- 172.21.15.45:0/4237902611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff00c156bb0 con 0x7ff00c0d1f10 2024-04-16T16:14:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.980+0000 7ff012600640 1 -- 172.21.15.45:0/4237902611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff00c075780 con 0x7ff00c0d1f10 2024-04-16T16:14:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.981+0000 7ff003400640 1 -- 172.21.15.45:0/4237902611 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feff4040070 con 0x7ff00c0d1f10 2024-04-16T16:14:35.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.982+0000 7ff003400640 1 --2- 172.21.15.45:0/4237902611 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fefe806cb10 0x7fefe806efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:35.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.982+0000 7ff003400640 1 -- 172.21.15.45:0/4237902611 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7feff40ae020 con 0x7ff00c0d1f10 2024-04-16T16:14:35.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.983+0000 7ff00b400640 1 --2- 172.21.15.45:0/4237902611 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fefe806cb10 0x7fefe806efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:35.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.984+0000 7ff00b400640 1 --2- 172.21.15.45:0/4237902611 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fefe806cb10 0x7fefe806efd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7feffc0012e0 tx=0x7feffc007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:35.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:35.987+0000 7ff003400640 1 -- 172.21.15.45:0/4237902611 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feff4033030 con 0x7ff00c0d1f10 2024-04-16T16:14:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.191+0000 7ff012600640 1 -- 172.21.15.45:0/4237902611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff00c0b72e0 con 0x7ff00c0d1f10 2024-04-16T16:14:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.192+0000 7ff003400640 1 -- 172.21.15.45:0/4237902611 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) ==== 93+0+10 (secure 0 0 0) 0x7feff407c880 con 0x7ff00c0d1f10 2024-04-16T16:14:36.192 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:14:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.196+0000 7ff012600640 1 -- 172.21.15.45:0/4237902611 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fefe806cb10 msgr2=0x7fefe806efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.196+0000 7ff012600640 1 --2- 172.21.15.45:0/4237902611 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fefe806cb10 0x7fefe806efd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7feffc0012e0 tx=0x7feffc007d30 comp rx=0 tx=0).stop 2024-04-16T16:14:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.196+0000 7ff012600640 1 -- 172.21.15.45:0/4237902611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0d1f10 msgr2=0x7ff00c0cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.196+0000 7ff012600640 1 --2- 172.21.15.45:0/4237902611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0d1f10 0x7ff00c0cd980 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7feff4000c00 tx=0x7feff4004730 comp rx=0 tx=0).stop 2024-04-16T16:14:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.196+0000 7ff012600640 1 -- 172.21.15.45:0/4237902611 shutdown_connections 2024-04-16T16:14:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.196+0000 7ff012600640 1 --2- 172.21.15.45:0/4237902611 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fefe806cb10 0x7fefe806efd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.196+0000 7ff012600640 1 --2- 172.21.15.45:0/4237902611 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff00c1622d0 0x7ff00c0cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.196+0000 7ff012600640 1 --2- 172.21.15.45:0/4237902611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0d1f10 0x7ff00c0cd980 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.196+0000 7ff012600640 1 --2- 172.21.15.45:0/4237902611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff00c0cfe00 0x7ff00c0b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.196+0000 7ff012600640 1 -- 172.21.15.45:0/4237902611 >> 172.21.15.45:0/4237902611 conn(0x7ff00c06d400 msgr2=0x7ff00c0b9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.196+0000 7ff012600640 1 -- 172.21.15.45:0/4237902611 shutdown_connections 2024-04-16T16:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.196+0000 7ff012600640 1 -- 172.21.15.45:0/4237902611 wait complete. 2024-04-16T16:14:36.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-16T16:14:36.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.446+0000 7f5549e00640 1 Processor -- start 2024-04-16T16:14:36.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.446+0000 7f5549e00640 1 -- start start 2024-04-16T16:14:36.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.447+0000 7f5549e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f55441462e0 0x7f5544072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.447+0000 7f5549e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f55440728c0 0x7f55440ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.447+0000 7f5549e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55440cf6d0 con 0x7f55440cec00 2024-04-16T16:14:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.447+0000 7f5549e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f55441466c0 con 0x7f55441462e0 2024-04-16T16:14:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.447+0000 7f5549e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f55440cf3f0 con 0x7f55440728c0 2024-04-16T16:14:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.448+0000 7f5542a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f55441462e0 0x7f5544072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.448+0000 7f5542a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f55441462e0 0x7f5544072380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33376/0 (socket says 172.21.15.45:33376) 2024-04-16T16:14:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.448+0000 7f5542a00640 1 -- 172.21.15.45:0/2791736848 learned_addr learned my addr 172.21.15.45:0/2791736848 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.448+0000 7f5542a00640 1 -- 172.21.15.45:0/2791736848 >> v2:172.21.15.103:3300/0 conn(0x7f55440728c0 msgr2=0x7f55440ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:36.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.448+0000 7f5542a00640 1 --2- 172.21.15.45:0/2791736848 >> v2:172.21.15.103:3300/0 conn(0x7f55440728c0 0x7f55440ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.448+0000 7f5542a00640 1 -- 172.21.15.45:0/2791736848 >> v1:172.21.15.103:6789/0 conn(0x7f55440cec00 legacy=0x7f554415da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:36.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.448+0000 7f5542a00640 1 -- 172.21.15.45:0/2791736848 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f554415cf80 con 0x7f55441462e0 2024-04-16T16:14:36.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.449+0000 7f5542a00640 1 --2- 172.21.15.45:0/2791736848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55441462e0 0x7f5544072380 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f554406fc10 tx=0x7f553402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938e546a7c59492f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:36.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.450+0000 7f5541600640 1 -- 172.21.15.45:0/2791736848 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553402e070 con 0x7f55441462e0 2024-04-16T16:14:36.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.450+0000 7f5541600640 1 -- 172.21.15.45:0/2791736848 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5534037070 con 0x7f55441462e0 2024-04-16T16:14:36.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.450+0000 7f5541600640 1 -- 172.21.15.45:0/2791736848 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5534031a00 con 0x7f55441462e0 2024-04-16T16:14:36.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.450+0000 7f5549e00640 1 -- 172.21.15.45:0/2791736848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55441462e0 msgr2=0x7f5544072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:36.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.450+0000 7f5549e00640 1 --2- 172.21.15.45:0/2791736848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55441462e0 0x7f5544072380 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f554406fc10 tx=0x7f553402b880 comp rx=0 tx=0).stop 2024-04-16T16:14:36.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.450+0000 7f5549e00640 1 -- 172.21.15.45:0/2791736848 shutdown_connections 2024-04-16T16:14:36.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.450+0000 7f5549e00640 1 --2- 172.21.15.45:0/2791736848 >> v2:172.21.15.103:3300/0 conn(0x7f55440728c0 0x7f55440ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.450+0000 7f5549e00640 1 --2- 172.21.15.45:0/2791736848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55441462e0 0x7f5544072380 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.451+0000 7f5549e00640 1 -- 172.21.15.45:0/2791736848 >> 172.21.15.45:0/2791736848 conn(0x7f55440b2750 msgr2=0x7f55440b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:36.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.451+0000 7f5549e00640 1 -- 172.21.15.45:0/2791736848 shutdown_connections 2024-04-16T16:14:36.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.451+0000 7f5549e00640 1 -- 172.21.15.45:0/2791736848 wait complete. 2024-04-16T16:14:36.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.452+0000 7f5549e00640 1 Processor -- start 2024-04-16T16:14:36.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.452+0000 7f5549e00640 1 -- start start 2024-04-16T16:14:36.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.453+0000 7f5549e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55440728c0 0x7f554415c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:36.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.453+0000 7f5549e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55440cec00 0x7f55440b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.453+0000 7f5549e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f55441462e0 0x7f554406e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.453+0000 7f5549e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f55440cf020 con 0x7f55440728c0 2024-04-16T16:14:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.453+0000 7f5549e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f554415d540 con 0x7f55440cec00 2024-04-16T16:14:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.453+0000 7f5549e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f55440cf1a0 con 0x7f55441462e0 2024-04-16T16:14:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.454+0000 7f5542a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55440728c0 0x7f554415c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.454+0000 7f5542a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55440728c0 0x7f554415c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33378/0 (socket says 172.21.15.45:33378) 2024-04-16T16:14:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.454+0000 7f5542a00640 1 -- 172.21.15.45:0/440508947 learned_addr learned my addr 172.21.15.45:0/440508947 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:36.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.454+0000 7f5542a00640 1 -- 172.21.15.45:0/440508947 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f55441462e0 msgr2=0x7f554406e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:36.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.454+0000 7f5542a00640 1 --2- 172.21.15.45:0/440508947 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f55441462e0 0x7f554406e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.454+0000 7f5542a00640 1 -- 172.21.15.45:0/440508947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55440cec00 msgr2=0x7f55440b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:36.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.454+0000 7f5542a00640 1 --2- 172.21.15.45:0/440508947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55440cec00 0x7f55440b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.454+0000 7f5542a00640 1 -- 172.21.15.45:0/440508947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f554413dc70 con 0x7f55440728c0 2024-04-16T16:14:36.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.454+0000 7f5542a00640 1 --2- 172.21.15.45:0/440508947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55440728c0 0x7f554415c2f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f55340090b0 tx=0x7f55340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:36.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.455+0000 7f553b400640 1 -- 172.21.15.45:0/440508947 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5534032880 con 0x7f55440728c0 2024-04-16T16:14:36.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.455+0000 7f553b400640 1 -- 172.21.15.45:0/440508947 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5534032a40 con 0x7f55440728c0 2024-04-16T16:14:36.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.455+0000 7f553b400640 1 -- 172.21.15.45:0/440508947 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5534032e30 con 0x7f55440728c0 2024-04-16T16:14:36.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.455+0000 7f5549e00640 1 -- 172.21.15.45:0/440508947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55440b1a00 con 0x7f55440728c0 2024-04-16T16:14:36.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.455+0000 7f5549e00640 1 -- 172.21.15.45:0/440508947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55440b1fc0 con 0x7f55440728c0 2024-04-16T16:14:36.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.456+0000 7f5549e00640 1 -- 172.21.15.45:0/440508947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5544075780 con 0x7f55440728c0 2024-04-16T16:14:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.458+0000 7f553b400640 1 -- 172.21.15.45:0/440508947 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f553404c060 con 0x7f55440728c0 2024-04-16T16:14:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.458+0000 7f553b400640 1 --2- 172.21.15.45:0/440508947 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f552406c8f0 0x7f552406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.459+0000 7f553b400640 1 -- 172.21.15.45:0/440508947 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f55340ad070 con 0x7f55440728c0 2024-04-16T16:14:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.459+0000 7f5542000640 1 --2- 172.21.15.45:0/440508947 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f552406c8f0 0x7f552406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:36.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.460+0000 7f5542000640 1 --2- 172.21.15.45:0/440508947 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f552406c8f0 0x7f552406edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f554415f360 tx=0x7f5530006c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.463+0000 7f553b400640 1 -- 172.21.15.45:0/440508947 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f554413dc70 con 0x7f55440728c0 2024-04-16T16:14:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.670+0000 7f5549e00640 1 -- 172.21.15.45:0/440508947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f55440b56e0 con 0x7f55440728c0 2024-04-16T16:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.671+0000 7f553b400640 1 -- 172.21.15.45:0/440508947 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) ==== 84+0+65 (secure 0 0 0) 0x7f5534040070 con 0x7f55440728c0 2024-04-16T16:14:36.671 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:36.671 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2024-04-16T16:14:36.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:14:36.671 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2024-04-16T16:14:36.671 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2024-04-16T16:14:36.671 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:14:36.671 INFO:teuthology.orchestra.run.smithi045.stdout:] 2024-04-16T16:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.674+0000 7f5549e00640 1 -- 172.21.15.45:0/440508947 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f552406c8f0 msgr2=0x7f552406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.674+0000 7f5549e00640 1 --2- 172.21.15.45:0/440508947 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f552406c8f0 0x7f552406edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f554415f360 tx=0x7f5530006c90 comp rx=0 tx=0).stop 2024-04-16T16:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.674+0000 7f5549e00640 1 -- 172.21.15.45:0/440508947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55440728c0 msgr2=0x7f554415c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.674+0000 7f5549e00640 1 --2- 172.21.15.45:0/440508947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55440728c0 0x7f554415c2f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f55340090b0 tx=0x7f55340046c0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.674+0000 7f5549e00640 1 -- 172.21.15.45:0/440508947 shutdown_connections 2024-04-16T16:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.674+0000 7f5549e00640 1 --2- 172.21.15.45:0/440508947 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f552406c8f0 0x7f552406edb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.674+0000 7f5549e00640 1 --2- 172.21.15.45:0/440508947 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f55441462e0 0x7f554406e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.674+0000 7f5549e00640 1 --2- 172.21.15.45:0/440508947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55440cec00 0x7f55440b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.674+0000 7f5549e00640 1 --2- 172.21.15.45:0/440508947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55440728c0 0x7f554415c2f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.674+0000 7f5549e00640 1 -- 172.21.15.45:0/440508947 >> 172.21.15.45:0/440508947 conn(0x7f55440b2750 msgr2=0x7f5544143a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:36.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.674+0000 7f5549e00640 1 -- 172.21.15.45:0/440508947 shutdown_connections 2024-04-16T16:14:36.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.674+0000 7f5549e00640 1 -- 172.21.15.45:0/440508947 wait complete. 2024-04-16T16:14:36.695 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-16T16:14:36.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-16T16:14:36.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.927+0000 7f075f000640 1 Processor -- start 2024-04-16T16:14:36.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.928+0000 7f075f000640 1 -- start start 2024-04-16T16:14:36.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.929+0000 7f075f000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07581452e0 0x7f075806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:36.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.930+0000 7f075f000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0758071e60 con 0x7f07580cecc0 2024-04-16T16:14:36.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.930+0000 7f075f000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0758072140 con 0x7f075806e920 2024-04-16T16:14:36.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.930+0000 7f075f000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0758072f00 con 0x7f07581452e0 2024-04-16T16:14:36.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.930+0000 7f075e600640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f07580cecc0 0x7f075815cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:34528/0 (socket says 172.21.15.45:34528) 2024-04-16T16:14:36.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.930+0000 7f075e600640 1 -- 172.21.15.45:0/3001813687 learned_addr learned my addr 172.21.15.45:0/3001813687 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:36.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.930+0000 7f075dc00640 1 --2- 172.21.15.45:0/3001813687 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07581452e0 0x7f075806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:36.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.931+0000 7f0757e00640 1 -- 172.21.15.45:0/3001813687 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3850415794 0 0) 0x7f0758071e60 con 0x7f07580cecc0 2024-04-16T16:14:36.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.931+0000 7f0757e00640 1 -- 172.21.15.45:0/3001813687 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0740003520 con 0x7f07580cecc0 2024-04-16T16:14:36.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.931+0000 7f0757e00640 1 -- 172.21.15.45:0/3001813687 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3058508560 0 0) 0x7f0758072140 con 0x7f075806e920 2024-04-16T16:14:36.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.931+0000 7f075dc00640 1 -- 172.21.15.45:0/3001813687 >> v1:172.21.15.103:6789/0 conn(0x7f075806e920 legacy=0x7f07580ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:36.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.931+0000 7f075dc00640 1 -- 172.21.15.45:0/3001813687 >> v1:172.21.15.45:6789/0 conn(0x7f07580cecc0 legacy=0x7f075815cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:36.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.931+0000 7f075dc00640 1 -- 172.21.15.45:0/3001813687 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0758072420 con 0x7f07581452e0 2024-04-16T16:14:36.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.933+0000 7f075dc00640 1 --2- 172.21.15.45:0/3001813687 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07581452e0 0x7f075806e3e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f074c0097d0 tx=0x7f074c02b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e08fd4ac836b77d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:36.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.933+0000 7f0757e00640 1 -- 172.21.15.45:0/3001813687 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c037070 con 0x7f07581452e0 2024-04-16T16:14:36.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.933+0000 7f0757e00640 1 -- 172.21.15.45:0/3001813687 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f074c030510 con 0x7f07581452e0 2024-04-16T16:14:36.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.933+0000 7f0757e00640 1 -- 172.21.15.45:0/3001813687 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0750001990 con 0x7f07581452e0 2024-04-16T16:14:36.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.934+0000 7f075f000640 1 -- 172.21.15.45:0/3001813687 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07581452e0 msgr2=0x7f075806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:36.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.934+0000 7f075f000640 1 --2- 172.21.15.45:0/3001813687 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07581452e0 0x7f075806e3e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f074c0097d0 tx=0x7f074c02b7b0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.934+0000 7f075f000640 1 -- 172.21.15.45:0/3001813687 shutdown_connections 2024-04-16T16:14:36.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.934+0000 7f075f000640 1 --2- 172.21.15.45:0/3001813687 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07581452e0 0x7f075806e3e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.934+0000 7f075f000640 1 -- 172.21.15.45:0/3001813687 >> 172.21.15.45:0/3001813687 conn(0x7f07580b3bb0 msgr2=0x7f07580b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:36.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.934+0000 7f075f000640 1 -- 172.21.15.45:0/3001813687 shutdown_connections 2024-04-16T16:14:36.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.934+0000 7f075f000640 1 -- 172.21.15.45:0/3001813687 wait complete. 2024-04-16T16:14:36.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.936+0000 7f075f000640 1 Processor -- start 2024-04-16T16:14:36.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.936+0000 7f075f000640 1 -- start start 2024-04-16T16:14:36.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.937+0000 7f075f000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f075806e920 0x7f07580aa330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:36.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.937+0000 7f075f000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07580cecc0 0x7f07581f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:36.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.937+0000 7f075f000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07581452e0 0x7f07581f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:36.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.937+0000 7f075f000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f07580729e0 con 0x7f075806e920 2024-04-16T16:14:36.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.937+0000 7f075f000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f075806ed60 con 0x7f07580cecc0 2024-04-16T16:14:36.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.937+0000 7f075f000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f07581456c0 con 0x7f07581452e0 2024-04-16T16:14:36.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.937+0000 7f075dc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f075806e920 0x7f07580aa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:36.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.937+0000 7f075dc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f075806e920 0x7f07580aa330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33386/0 (socket says 172.21.15.45:33386) 2024-04-16T16:14:36.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.937+0000 7f075dc00640 1 -- 172.21.15.45:0/145667598 learned_addr learned my addr 172.21.15.45:0/145667598 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:36.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.938+0000 7f075dc00640 1 -- 172.21.15.45:0/145667598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07581452e0 msgr2=0x7f07581f1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:36.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.938+0000 7f075dc00640 1 --2- 172.21.15.45:0/145667598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07581452e0 0x7f07581f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.938+0000 7f075dc00640 1 -- 172.21.15.45:0/145667598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07580cecc0 msgr2=0x7f07581f3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:36.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.938+0000 7f075d200640 1 --2- 172.21.15.45:0/145667598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07580cecc0 0x7f07581f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:36.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.938+0000 7f075dc00640 1 --2- 172.21.15.45:0/145667598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07580cecc0 0x7f07581f3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:36.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.938+0000 7f075dc00640 1 -- 172.21.15.45:0/145667598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07581f45d0 con 0x7f075806e920 2024-04-16T16:14:36.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.938+0000 7f075d200640 1 --2- 172.21.15.45:0/145667598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07580cecc0 0x7f07581f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:36.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.938+0000 7f075dc00640 1 --2- 172.21.15.45:0/145667598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f075806e920 0x7f07580aa330 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f074c0098f0 tx=0x7f074c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:36.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.938+0000 7f0755600640 1 -- 172.21.15.45:0/145667598 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c041070 con 0x7f075806e920 2024-04-16T16:14:36.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.938+0000 7f0755600640 1 -- 172.21.15.45:0/145667598 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f074c004230 con 0x7f075806e920 2024-04-16T16:14:36.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.938+0000 7f075f000640 1 -- 172.21.15.45:0/145667598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07581f48b0 con 0x7f075806e920 2024-04-16T16:14:36.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.939+0000 7f0755600640 1 -- 172.21.15.45:0/145667598 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c037040 con 0x7f075806e920 2024-04-16T16:14:36.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.939+0000 7f075f000640 1 -- 172.21.15.45:0/145667598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07581f4e70 con 0x7f075806e920 2024-04-16T16:14:36.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.940+0000 7f075f000640 1 -- 172.21.15.45:0/145667598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0758075780 con 0x7f075806e920 2024-04-16T16:14:36.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.941+0000 7f0755600640 1 -- 172.21.15.45:0/145667598 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f074c044070 con 0x7f075806e920 2024-04-16T16:14:36.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.941+0000 7f0755600640 1 --2- 172.21.15.45:0/145667598 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f073806c8f0 0x7f073806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:36.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.942+0000 7f0755600640 1 -- 172.21.15.45:0/145667598 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f074c0ad050 con 0x7f075806e920 2024-04-16T16:14:36.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.942+0000 7f075d200640 1 --2- 172.21.15.45:0/145667598 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f073806c8f0 0x7f073806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:36.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.944+0000 7f075d200640 1 --2- 172.21.15.45:0/145667598 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f073806c8f0 0x7f073806edb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f07580b5830 tx=0x7f0744009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:36.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:36.946+0000 7f0755600640 1 -- 172.21.15.45:0/145667598 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f074c07b7d0 con 0x7f075806e920 2024-04-16T16:14:37.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.153+0000 7f075f000640 1 -- 172.21.15.45:0/145667598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f07580b65f0 con 0x7f075806e920 2024-04-16T16:14:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.623+0000 7f0755600640 1 -- 172.21.15.45:0/145667598 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) ==== 138+0+0 (secure 0 0 0) 0x7f074c07b0d0 con 0x7f075806e920 2024-04-16T16:14:37.624 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2024-04-16T16:14:37.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.628+0000 7f075f000640 1 -- 172.21.15.45:0/145667598 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f073806c8f0 msgr2=0x7f073806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:37.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.628+0000 7f075f000640 1 --2- 172.21.15.45:0/145667598 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f073806c8f0 0x7f073806edb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f07580b5830 tx=0x7f0744009040 comp rx=0 tx=0).stop 2024-04-16T16:14:37.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.628+0000 7f075f000640 1 -- 172.21.15.45:0/145667598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f075806e920 msgr2=0x7f07580aa330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:37.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.628+0000 7f075f000640 1 --2- 172.21.15.45:0/145667598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f075806e920 0x7f07580aa330 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f074c0098f0 tx=0x7f074c0046c0 comp rx=0 tx=0).stop 2024-04-16T16:14:37.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.628+0000 7f075f000640 1 -- 172.21.15.45:0/145667598 shutdown_connections 2024-04-16T16:14:37.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.628+0000 7f075f000640 1 --2- 172.21.15.45:0/145667598 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f073806c8f0 0x7f073806edb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:37.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.628+0000 7f075f000640 1 --2- 172.21.15.45:0/145667598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07581452e0 0x7f07581f1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:37.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.628+0000 7f075f000640 1 --2- 172.21.15.45:0/145667598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07580cecc0 0x7f07581f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:37.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.628+0000 7f075f000640 1 --2- 172.21.15.45:0/145667598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f075806e920 0x7f07580aa330 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:37.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.628+0000 7f075f000640 1 -- 172.21.15.45:0/145667598 >> 172.21.15.45:0/145667598 conn(0x7f07580b3bb0 msgr2=0x7f07581459d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:37.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.629+0000 7f075f000640 1 -- 172.21.15.45:0/145667598 shutdown_connections 2024-04-16T16:14:37.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.629+0000 7f075f000640 1 -- 172.21.15.45:0/145667598 wait complete. 2024-04-16T16:14:37.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-16T16:14:37.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.875+0000 7f8a60200640 1 Processor -- start 2024-04-16T16:14:37.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.875+0000 7f8a60200640 1 -- start start 2024-04-16T16:14:37.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.877+0000 7f8a60200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8a58146350 0x7f8a5806b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:37.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.877+0000 7f8a60200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a5806be70 0x7f8a580d0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:37.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.877+0000 7f8a60200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a5806c310 con 0x7f8a580d0ec0 2024-04-16T16:14:37.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.877+0000 7f8a60200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f8a580cf2a0 con 0x7f8a58146350 2024-04-16T16:14:37.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.877+0000 7f8a60200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a5815d920 con 0x7f8a5806be70 2024-04-16T16:14:37.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.877+0000 7f8a5e200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8a580d0ec0 0x7f8a580ce0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:34544/0 (socket says 172.21.15.45:34544) 2024-04-16T16:14:37.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.877+0000 7f8a5e200640 1 -- 172.21.15.45:0/3462559159 learned_addr learned my addr 172.21.15.45:0/3462559159 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:37.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.878+0000 7f8a5ce00640 1 --2- 172.21.15.45:0/3462559159 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a5806be70 0x7f8a580d0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:37.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.878+0000 7f8a5d800640 1 --2- 172.21.15.45:0/3462559159 >> v2:172.21.15.103:3300/0 conn(0x7f8a58146350 0x7f8a5806b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:37.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.878+0000 7f8a57e00640 1 -- 172.21.15.45:0/3462559159 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 85646605 0 0) 0x7f8a5806c310 con 0x7f8a580d0ec0 2024-04-16T16:14:37.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.878+0000 7f8a57e00640 1 -- 172.21.15.45:0/3462559159 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a40003520 con 0x7f8a580d0ec0 2024-04-16T16:14:37.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.878+0000 7f8a5ce00640 1 -- 172.21.15.45:0/3462559159 >> v2:172.21.15.103:3300/0 conn(0x7f8a58146350 msgr2=0x7f8a5806b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:37.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.878+0000 7f8a5ce00640 1 --2- 172.21.15.45:0/3462559159 >> v2:172.21.15.103:3300/0 conn(0x7f8a58146350 0x7f8a5806b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:37.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.878+0000 7f8a5ce00640 1 -- 172.21.15.45:0/3462559159 >> v1:172.21.15.45:6789/0 conn(0x7f8a580d0ec0 legacy=0x7f8a580ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:37.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.878+0000 7f8a57e00640 1 -- 172.21.15.45:0/3462559159 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2694879339 0 0) 0x7f8a40003520 con 0x7f8a580d0ec0 2024-04-16T16:14:37.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.879+0000 7f8a5ce00640 1 -- 172.21.15.45:0/3462559159 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5815d2d0 con 0x7f8a5806be70 2024-04-16T16:14:37.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.880+0000 7f8a5ce00640 1 --2- 172.21.15.45:0/3462559159 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a5806be70 0x7f8a580d0980 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8a440097a0 tx=0x7f8a4402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2f99769b4a46b8a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:37.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.881+0000 7f8a57e00640 1 -- 172.21.15.45:0/3462559159 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44037070 con 0x7f8a5806be70 2024-04-16T16:14:37.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.881+0000 7f8a57e00640 1 -- 172.21.15.45:0/3462559159 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a44030510 con 0x7f8a5806be70 2024-04-16T16:14:37.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.881+0000 7f8a57e00640 1 -- 172.21.15.45:0/3462559159 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a50001670 con 0x7f8a5806be70 2024-04-16T16:14:37.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.881+0000 7f8a60200640 1 -- 172.21.15.45:0/3462559159 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a5806be70 msgr2=0x7f8a580d0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:37.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.881+0000 7f8a60200640 1 --2- 172.21.15.45:0/3462559159 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a5806be70 0x7f8a580d0980 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8a440097a0 tx=0x7f8a4402b790 comp rx=0 tx=0).stop 2024-04-16T16:14:37.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.881+0000 7f8a60200640 1 -- 172.21.15.45:0/3462559159 shutdown_connections 2024-04-16T16:14:37.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.881+0000 7f8a60200640 1 --2- 172.21.15.45:0/3462559159 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a5806be70 0x7f8a580d0980 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:37.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.881+0000 7f8a60200640 1 --2- 172.21.15.45:0/3462559159 >> v2:172.21.15.103:3300/0 conn(0x7f8a58146350 0x7f8a5806b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:37.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.882+0000 7f8a60200640 1 -- 172.21.15.45:0/3462559159 >> 172.21.15.45:0/3462559159 conn(0x7f8a5809aa70 msgr2=0x7f8a580b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:37.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.882+0000 7f8a60200640 1 -- 172.21.15.45:0/3462559159 shutdown_connections 2024-04-16T16:14:37.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.882+0000 7f8a60200640 1 -- 172.21.15.45:0/3462559159 wait complete. 2024-04-16T16:14:37.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.884+0000 7f8a60200640 1 Processor -- start 2024-04-16T16:14:37.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.884+0000 7f8a60200640 1 -- start start 2024-04-16T16:14:37.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.886+0000 7f8a60200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a5806be70 0x7f8a5815c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.886+0000 7f8a60200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a580d0ec0 0x7f8a580b69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.886+0000 7f8a60200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a58146350 0x7f8a580b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.886+0000 7f8a60200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5815ddb0 con 0x7f8a5806be70 2024-04-16T16:14:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.886+0000 7f8a60200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5815cf80 con 0x7f8a58146350 2024-04-16T16:14:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.886+0000 7f8a60200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a5815df30 con 0x7f8a580d0ec0 2024-04-16T16:14:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.886+0000 7f8a5d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a5806be70 0x7f8a5815c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.886+0000 7f8a5d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a5806be70 0x7f8a5815c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33388/0 (socket says 172.21.15.45:33388) 2024-04-16T16:14:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.886+0000 7f8a5d800640 1 -- 172.21.15.45:0/4006932042 learned_addr learned my addr 172.21.15.45:0/4006932042 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.887+0000 7f8a5e200640 1 --2- 172.21.15.45:0/4006932042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a58146350 0x7f8a580b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.887+0000 7f8a5ce00640 1 --2- 172.21.15.45:0/4006932042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a580d0ec0 0x7f8a580b69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.887+0000 7f8a5d800640 1 -- 172.21.15.45:0/4006932042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a580d0ec0 msgr2=0x7f8a580b69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.887+0000 7f8a5d800640 1 --2- 172.21.15.45:0/4006932042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a580d0ec0 0x7f8a580b69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.887+0000 7f8a5d800640 1 -- 172.21.15.45:0/4006932042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a58146350 msgr2=0x7f8a580b6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:37.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.887+0000 7f8a5d800640 1 --2- 172.21.15.45:0/4006932042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a58146350 0x7f8a580b6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:37.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.887+0000 7f8a5d800640 1 -- 172.21.15.45:0/4006932042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a580b6220 con 0x7f8a5806be70 2024-04-16T16:14:37.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.887+0000 7f8a5d800640 1 --2- 172.21.15.45:0/4006932042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a5806be70 0x7f8a5815c400 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00efc0 tx=0x7f8a4c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:37.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.888+0000 7f8a55600640 1 -- 172.21.15.45:0/4006932042 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c018020 con 0x7f8a5806be70 2024-04-16T16:14:37.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.888+0000 7f8a55600640 1 -- 172.21.15.45:0/4006932042 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a4c014480 con 0x7f8a5806be70 2024-04-16T16:14:37.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.888+0000 7f8a55600640 1 -- 172.21.15.45:0/4006932042 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c014850 con 0x7f8a5806be70 2024-04-16T16:14:37.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.888+0000 7f8a60200640 1 -- 172.21.15.45:0/4006932042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a580b5180 con 0x7f8a5806be70 2024-04-16T16:14:37.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.888+0000 7f8a60200640 1 -- 172.21.15.45:0/4006932042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a580b5740 con 0x7f8a5806be70 2024-04-16T16:14:37.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.889+0000 7f8a60200640 1 -- 172.21.15.45:0/4006932042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a58075780 con 0x7f8a5806be70 2024-04-16T16:14:37.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.890+0000 7f8a55600640 1 -- 172.21.15.45:0/4006932042 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8a4c005d10 con 0x7f8a5806be70 2024-04-16T16:14:37.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.891+0000 7f8a55600640 1 --2- 172.21.15.45:0/4006932042 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a3806c8f0 0x7f8a3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:37.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.891+0000 7f8a55600640 1 -- 172.21.15.45:0/4006932042 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5227+0+0 (secure 0 0 0) 0x7f8a4c08c4a0 con 0x7f8a5806be70 2024-04-16T16:14:37.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.892+0000 7f8a5ce00640 1 --2- 172.21.15.45:0/4006932042 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a3806c8f0 0x7f8a3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:37.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.893+0000 7f8a5ce00640 1 --2- 172.21.15.45:0/4006932042 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a3806c8f0 0x7f8a3806edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8a44034000 tx=0x7f8a4402f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:37.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:37.897+0000 7f8a55600640 1 -- 172.21.15.45:0/4006932042 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8a4c022070 con 0x7f8a5806be70 2024-04-16T16:14:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.103+0000 7f8a60200640 1 -- 172.21.15.45:0/4006932042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f8a581452d0 con 0x7f8a5806be70 2024-04-16T16:14:38.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.630+0000 7f8a55600640 1 -- 172.21.15.45:0/4006932042 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) ==== 170+0+0 (secure 0 0 0) 0x7f8a4c010040 con 0x7f8a5806be70 2024-04-16T16:14:38.631 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2024-04-16T16:14:38.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.634+0000 7f8a60200640 1 -- 172.21.15.45:0/4006932042 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a3806c8f0 msgr2=0x7f8a3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:38.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.634+0000 7f8a60200640 1 --2- 172.21.15.45:0/4006932042 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a3806c8f0 0x7f8a3806edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8a44034000 tx=0x7f8a4402f000 comp rx=0 tx=0).stop 2024-04-16T16:14:38.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.634+0000 7f8a60200640 1 -- 172.21.15.45:0/4006932042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a5806be70 msgr2=0x7f8a5815c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:38.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.634+0000 7f8a60200640 1 --2- 172.21.15.45:0/4006932042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a5806be70 0x7f8a5815c400 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00efc0 tx=0x7f8a4c00c490 comp rx=0 tx=0).stop 2024-04-16T16:14:38.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.635+0000 7f8a60200640 1 -- 172.21.15.45:0/4006932042 shutdown_connections 2024-04-16T16:14:38.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.635+0000 7f8a60200640 1 --2- 172.21.15.45:0/4006932042 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a3806c8f0 0x7f8a3806edb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:38.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.635+0000 7f8a60200640 1 --2- 172.21.15.45:0/4006932042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a58146350 0x7f8a580b6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:38.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.635+0000 7f8a60200640 1 --2- 172.21.15.45:0/4006932042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a580d0ec0 0x7f8a580b69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:38.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.635+0000 7f8a60200640 1 --2- 172.21.15.45:0/4006932042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a5806be70 0x7f8a5815c400 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:38.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.635+0000 7f8a60200640 1 -- 172.21.15.45:0/4006932042 >> 172.21.15.45:0/4006932042 conn(0x7f8a5809aa70 msgr2=0x7f8a580cdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:38.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.637+0000 7f8a60200640 1 -- 172.21.15.45:0/4006932042 shutdown_connections 2024-04-16T16:14:38.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.637+0000 7f8a60200640 1 -- 172.21.15.45:0/4006932042 wait complete. 2024-04-16T16:14:38.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-16T16:14:38.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.867+0000 7fb89ae00640 1 Processor -- start 2024-04-16T16:14:38.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.867+0000 7fb89ae00640 1 -- start start 2024-04-16T16:14:38.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.868+0000 7fb89ae00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb89415c7c0 0x7fb894072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:38.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.869+0000 7fb89ae00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb8940b4930 0x7fb8940d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:38.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.869+0000 7fb89ae00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb8940d0f10 0x7fb89406f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:38.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.869+0000 7fb89ae00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb8940b52e0 con 0x7fb89415c7c0 2024-04-16T16:14:38.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.869+0000 7fb89ae00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb8940d0260 con 0x7fb8940b4930 2024-04-16T16:14:38.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.869+0000 7fb89ae00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb894073350 con 0x7fb8940d0f10 2024-04-16T16:14:38.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.869+0000 7fb893e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb89415c7c0 0x7fb894072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:38.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.869+0000 7fb893e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb89415c7c0 0x7fb894072e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33394/0 (socket says 172.21.15.45:33394) 2024-04-16T16:14:38.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.869+0000 7fb893e00640 1 -- 172.21.15.45:0/900059714 learned_addr learned my addr 172.21.15.45:0/900059714 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:38.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.869+0000 7fb893e00640 1 -- 172.21.15.45:0/900059714 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb8940d0f10 msgr2=0x7fb89406f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:38.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.869+0000 7fb893e00640 1 --2- 172.21.15.45:0/900059714 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb8940d0f10 0x7fb89406f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:38.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.869+0000 7fb893e00640 1 -- 172.21.15.45:0/900059714 >> v2:172.21.15.103:3300/0 conn(0x7fb8940b4930 msgr2=0x7fb8940d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:38.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.869+0000 7fb88be00640 1 --2- 172.21.15.45:0/900059714 >> v2:172.21.15.103:3300/0 conn(0x7fb8940b4930 0x7fb8940d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:38.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.870+0000 7fb893e00640 1 --2- 172.21.15.45:0/900059714 >> v2:172.21.15.103:3300/0 conn(0x7fb8940b4930 0x7fb8940d09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:38.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.870+0000 7fb893e00640 1 -- 172.21.15.45:0/900059714 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8940d1360 con 0x7fb89415c7c0 2024-04-16T16:14:38.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.870+0000 7fb88be00640 1 --2- 172.21.15.45:0/900059714 >> v2:172.21.15.103:3300/0 conn(0x7fb8940b4930 0x7fb8940d09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:38.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.871+0000 7fb893e00640 1 --2- 172.21.15.45:0/900059714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb89415c7c0 0x7fb894072e10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb8940d03e0 tx=0x7fb88402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc6731f3c47ce72 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:38.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.871+0000 7fb893400640 1 -- 172.21.15.45:0/900059714 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb884038070 con 0x7fb89415c7c0 2024-04-16T16:14:38.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.871+0000 7fb893400640 1 -- 172.21.15.45:0/900059714 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb884031740 con 0x7fb89415c7c0 2024-04-16T16:14:38.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.871+0000 7fb893400640 1 -- 172.21.15.45:0/900059714 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88402f050 con 0x7fb89415c7c0 2024-04-16T16:14:38.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.872+0000 7fb89ae00640 1 -- 172.21.15.45:0/900059714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb89415c7c0 msgr2=0x7fb894072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:38.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.872+0000 7fb89ae00640 1 --2- 172.21.15.45:0/900059714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb89415c7c0 0x7fb894072e10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb8940d03e0 tx=0x7fb88402b7e0 comp rx=0 tx=0).stop 2024-04-16T16:14:38.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.872+0000 7fb89ae00640 1 -- 172.21.15.45:0/900059714 shutdown_connections 2024-04-16T16:14:38.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.872+0000 7fb89ae00640 1 --2- 172.21.15.45:0/900059714 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb8940d0f10 0x7fb89406f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:38.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.872+0000 7fb89ae00640 1 --2- 172.21.15.45:0/900059714 >> v2:172.21.15.103:3300/0 conn(0x7fb8940b4930 0x7fb8940d09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:38.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.872+0000 7fb89ae00640 1 --2- 172.21.15.45:0/900059714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb89415c7c0 0x7fb894072e10 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:38.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.872+0000 7fb89ae00640 1 -- 172.21.15.45:0/900059714 >> 172.21.15.45:0/900059714 conn(0x7fb8940b3970 msgr2=0x7fb8940b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:38.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.872+0000 7fb89ae00640 1 -- 172.21.15.45:0/900059714 shutdown_connections 2024-04-16T16:14:38.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.872+0000 7fb89ae00640 1 -- 172.21.15.45:0/900059714 wait complete. 2024-04-16T16:14:38.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.874+0000 7fb89ae00640 1 Processor -- start 2024-04-16T16:14:38.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.874+0000 7fb89ae00640 1 -- start start 2024-04-16T16:14:38.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.875+0000 7fb89ae00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8940b4930 0x7fb8941586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:38.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.875+0000 7fb89ae00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8940d0f10 0x7fb894143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:38.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.875+0000 7fb89ae00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89415c7c0 0x7fb894153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:38.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.875+0000 7fb89ae00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb8940d04b0 con 0x7fb8940d0f10 2024-04-16T16:14:38.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.875+0000 7fb89ae00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb8940b5810 con 0x7fb8940b4930 2024-04-16T16:14:38.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.875+0000 7fb89ae00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb89406f6a0 con 0x7fb89415c7c0 2024-04-16T16:14:38.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.875+0000 7fb88be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8940d0f10 0x7fb894143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:38.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.875+0000 7fb88be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8940d0f10 0x7fb894143300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33402/0 (socket says 172.21.15.45:33402) 2024-04-16T16:14:38.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.875+0000 7fb88be00640 1 -- 172.21.15.45:0/1841810035 learned_addr learned my addr 172.21.15.45:0/1841810035 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:38.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.876+0000 7fb893e00640 1 --2- 172.21.15.45:0/1841810035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8940b4930 0x7fb8941586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:38.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.876+0000 7fb88be00640 1 -- 172.21.15.45:0/1841810035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89415c7c0 msgr2=0x7fb894153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:38.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.876+0000 7fb898e00640 1 --2- 172.21.15.45:0/1841810035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89415c7c0 0x7fb894153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:38.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.876+0000 7fb88be00640 1 --2- 172.21.15.45:0/1841810035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89415c7c0 0x7fb894153b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:38.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.876+0000 7fb88be00640 1 -- 172.21.15.45:0/1841810035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8940b4930 msgr2=0x7fb8941586d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:38.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.876+0000 7fb88be00640 1 --2- 172.21.15.45:0/1841810035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8940b4930 0x7fb8941586d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:38.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.876+0000 7fb88be00640 1 -- 172.21.15.45:0/1841810035 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb894157630 con 0x7fb8940d0f10 2024-04-16T16:14:38.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.876+0000 7fb88be00640 1 --2- 172.21.15.45:0/1841810035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8940d0f10 0x7fb894143300 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb87c00c7f0 tx=0x7fb87c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:38.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.877+0000 7fb890c00640 1 -- 172.21.15.45:0/1841810035 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c015070 con 0x7fb8940d0f10 2024-04-16T16:14:38.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.877+0000 7fb890c00640 1 -- 172.21.15.45:0/1841810035 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb87c0079d0 con 0x7fb8940d0f10 2024-04-16T16:14:38.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.877+0000 7fb890c00640 1 -- 172.21.15.45:0/1841810035 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c007da0 con 0x7fb8940d0f10 2024-04-16T16:14:38.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.877+0000 7fb89ae00640 1 -- 172.21.15.45:0/1841810035 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb894157910 con 0x7fb8940d0f10 2024-04-16T16:14:38.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.877+0000 7fb89ae00640 1 -- 172.21.15.45:0/1841810035 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb894157ed0 con 0x7fb8940d0f10 2024-04-16T16:14:38.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.878+0000 7fb89ae00640 1 -- 172.21.15.45:0/1841810035 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb894075780 con 0x7fb8940d0f10 2024-04-16T16:14:38.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.879+0000 7fb890c00640 1 -- 172.21.15.45:0/1841810035 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb87c018070 con 0x7fb8940d0f10 2024-04-16T16:14:38.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.880+0000 7fb890c00640 1 --2- 172.21.15.45:0/1841810035 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb87006c8f0 0x7fb87006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:38.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.880+0000 7fb890c00640 1 -- 172.21.15.45:0/1841810035 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5609+0+0 (secure 0 0 0) 0x7fb87c08bb50 con 0x7fb8940d0f10 2024-04-16T16:14:38.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.881+0000 7fb893e00640 1 --2- 172.21.15.45:0/1841810035 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb87006c8f0 0x7fb87006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:38.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.882+0000 7fb893e00640 1 --2- 172.21.15.45:0/1841810035 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb87006c8f0 0x7fb87006edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb884035000 tx=0x7fb884008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:38.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:38.886+0000 7fb890c00640 1 -- 172.21.15.45:0/1841810035 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb87c059f40 con 0x7fb8940d0f10 2024-04-16T16:14:39.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.129+0000 7fb89ae00640 1 -- 172.21.15.45:0/1841810035 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb89406ff10 con 0x7fb8940d0f10 2024-04-16T16:14:39.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.657+0000 7fb890c00640 1 -- 172.21.15.45:0/1841810035 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) ==== 172+0+0 (secure 0 0 0) 0x7fb87c059840 con 0x7fb8940d0f10 2024-04-16T16:14:39.658 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 4 and data pool 5 2024-04-16T16:14:39.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.661+0000 7fb89ae00640 1 -- 172.21.15.45:0/1841810035 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb87006c8f0 msgr2=0x7fb87006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:39.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.661+0000 7fb89ae00640 1 --2- 172.21.15.45:0/1841810035 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb87006c8f0 0x7fb87006edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb884035000 tx=0x7fb884008030 comp rx=0 tx=0).stop 2024-04-16T16:14:39.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.661+0000 7fb89ae00640 1 -- 172.21.15.45:0/1841810035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8940d0f10 msgr2=0x7fb894143300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:39.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.661+0000 7fb89ae00640 1 --2- 172.21.15.45:0/1841810035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8940d0f10 0x7fb894143300 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb87c00c7f0 tx=0x7fb87c00cbf0 comp rx=0 tx=0).stop 2024-04-16T16:14:39.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.661+0000 7fb89ae00640 1 -- 172.21.15.45:0/1841810035 shutdown_connections 2024-04-16T16:14:39.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.661+0000 7fb89ae00640 1 --2- 172.21.15.45:0/1841810035 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fb87006c8f0 0x7fb87006edb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:39.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.661+0000 7fb89ae00640 1 --2- 172.21.15.45:0/1841810035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb89415c7c0 0x7fb894153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:39.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.661+0000 7fb89ae00640 1 --2- 172.21.15.45:0/1841810035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8940d0f10 0x7fb894143300 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:39.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.661+0000 7fb89ae00640 1 --2- 172.21.15.45:0/1841810035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8940b4930 0x7fb8941586d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:39.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.661+0000 7fb89ae00640 1 -- 172.21.15.45:0/1841810035 >> 172.21.15.45:0/1841810035 conn(0x7fb8940b3970 msgr2=0x7fb89415d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:39.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.662+0000 7fb89ae00640 1 -- 172.21.15.45:0/1841810035 shutdown_connections 2024-04-16T16:14:39.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.662+0000 7fb89ae00640 1 -- 172.21.15.45:0/1841810035 wait complete. 2024-04-16T16:14:39.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:39.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.890+0000 7f2e34c00640 1 Processor -- start 2024-04-16T16:14:39.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.890+0000 7f2e34c00640 1 -- start start 2024-04-16T16:14:39.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.892+0000 7f2e34c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2e30147840 0x7f2e3015fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:39.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.892+0000 7f2e34c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e300aed90 0x7f2e30072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:39.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.892+0000 7f2e34c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e30147c20 con 0x7f2e300728c0 2024-04-16T16:14:39.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.892+0000 7f2e34c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2e300afe80 con 0x7f2e30147840 2024-04-16T16:14:39.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.892+0000 7f2e34c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2e300af270 con 0x7f2e300aed90 2024-04-16T16:14:39.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.892+0000 7f2e2e800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2e300728c0 0x7f2e30162610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:34556/0 (socket says 172.21.15.45:34556) 2024-04-16T16:14:39.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.892+0000 7f2e2e800640 1 -- 172.21.15.45:0/545404652 learned_addr learned my addr 172.21.15.45:0/545404652 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:39.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e25400640 1 --2- 172.21.15.45:0/545404652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e300aed90 0x7f2e30072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:39.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 -- 172.21.15.45:0/545404652 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2064374363 0 0) 0x7f2e30147c20 con 0x7f2e300728c0 2024-04-16T16:14:39.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2de00640 1 --2- 172.21.15.45:0/545404652 >> v2:172.21.15.103:3300/0 conn(0x7f2e30147840 0x7f2e3015fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:39.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 -- 172.21.15.45:0/545404652 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e14003520 con 0x7f2e300728c0 2024-04-16T16:14:39.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 -- 172.21.15.45:0/545404652 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3625562943 0 0) 0x7f2e14003520 con 0x7f2e300728c0 2024-04-16T16:14:39.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 -- 172.21.15.45:0/545404652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e300aed90 msgr2=0x7f2e30072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:39.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 --2- 172.21.15.45:0/545404652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e300aed90 0x7f2e30072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:39.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 -- 172.21.15.45:0/545404652 >> v2:172.21.15.103:3300/0 conn(0x7f2e30147840 msgr2=0x7f2e3015fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:39.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 --2- 172.21.15.45:0/545404652 >> v2:172.21.15.103:3300/0 conn(0x7f2e30147840 0x7f2e3015fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:39.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 -- 172.21.15.45:0/545404652 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e300af550 con 0x7f2e300728c0 2024-04-16T16:14:39.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 -- 172.21.15.45:0/545404652 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f2e28001670 con 0x7f2e300728c0 2024-04-16T16:14:39.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 -- 172.21.15.45:0/545404652 >> v1:172.21.15.45:6789/0 conn(0x7f2e300728c0 legacy=0x7f2e30162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:39.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 --2- 172.21.15.45:0/545404652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e140039b0 0x7f2e14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:39.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 --2- 172.21.15.45:0/545404652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e14004800 0x7f2e14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:39.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 --2- 172.21.15.45:0/545404652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e300aed90 0x7f2e140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:39.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 -- 172.21.15.45:0/545404652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2e300afe80 con 0x7f2e14004800 2024-04-16T16:14:39.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 -- 172.21.15.45:0/545404652 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2e300af270 con 0x7f2e140039b0 2024-04-16T16:14:39.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.893+0000 7f2e2d400640 1 -- 172.21.15.45:0/545404652 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2e1400bac0 con 0x7f2e300aed90 2024-04-16T16:14:39.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.894+0000 7f2e25400640 1 --2- 172.21.15.45:0/545404652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e14004800 0x7f2e14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:39.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.894+0000 7f2e2de00640 1 --2- 172.21.15.45:0/545404652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e140039b0 0x7f2e14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:39.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.894+0000 7f2e2e800640 1 --2- 172.21.15.45:0/545404652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e300aed90 0x7f2e140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:39.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.894+0000 7f2e25400640 1 -- 172.21.15.45:0/545404652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e300aed90 msgr2=0x7f2e140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:39.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.894+0000 7f2e25400640 1 --2- 172.21.15.45:0/545404652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e300aed90 0x7f2e140051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:39.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.894+0000 7f2e25400640 1 -- 172.21.15.45:0/545404652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e140039b0 msgr2=0x7f2e14003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:39.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.894+0000 7f2e25400640 1 --2- 172.21.15.45:0/545404652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e140039b0 0x7f2e14003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:39.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.894+0000 7f2e25400640 1 -- 172.21.15.45:0/545404652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e14003520 con 0x7f2e14004800 2024-04-16T16:14:39.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.894+0000 7f2e2de00640 1 --2- 172.21.15.45:0/545404652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e140039b0 0x7f2e14003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:39.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.895+0000 7f2e25400640 1 --2- 172.21.15.45:0/545404652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e14004800 0x7f2e14004c80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800ba00 tx=0x7f2e14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cddc457f3947ac2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:39.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.895+0000 7f2e2d400640 1 -- 172.21.15.45:0/545404652 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18036020 con 0x7f2e14004800 2024-04-16T16:14:39.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.895+0000 7f2e2d400640 1 -- 172.21.15.45:0/545404652 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e180325f0 con 0x7f2e14004800 2024-04-16T16:14:39.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.895+0000 7f2e2d400640 1 -- 172.21.15.45:0/545404652 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1800b600 con 0x7f2e14004800 2024-04-16T16:14:39.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.896+0000 7f2e34c00640 1 -- 172.21.15.45:0/545404652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e14004800 msgr2=0x7f2e14004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:39.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.896+0000 7f2e34c00640 1 --2- 172.21.15.45:0/545404652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e14004800 0x7f2e14004c80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800ba00 tx=0x7f2e14005830 comp rx=0 tx=0).stop 2024-04-16T16:14:39.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.896+0000 7f2e2e800640 1 -- 172.21.15.45:0/545404652 reap_dead start 2024-04-16T16:14:39.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.896+0000 7f2e34c00640 1 -- 172.21.15.45:0/545404652 shutdown_connections 2024-04-16T16:14:39.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.896+0000 7f2e34c00640 1 -- 172.21.15.45:0/545404652 >> 172.21.15.45:0/545404652 conn(0x7f2e300b2750 msgr2=0x7f2e300b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:39.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.896+0000 7f2e34c00640 1 -- 172.21.15.45:0/545404652 shutdown_connections 2024-04-16T16:14:39.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.896+0000 7f2e34c00640 1 -- 172.21.15.45:0/545404652 wait complete. 2024-04-16T16:14:39.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.898+0000 7f2e34c00640 1 Processor -- start 2024-04-16T16:14:39.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.898+0000 7f2e34c00640 1 -- start start 2024-04-16T16:14:39.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.899+0000 7f2e34c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e14004800 0x7f2e3013c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:39.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.899+0000 7f2e34c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e3015e500 0x7f2e30138b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:39.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.899+0000 7f2e34c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e3013e0a0 0x7f2e3013e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:39.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.899+0000 7f2e34c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2e30160040 con 0x7f2e3013e0a0 2024-04-16T16:14:39.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.899+0000 7f2e34c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2e300afb10 con 0x7f2e3015e500 2024-04-16T16:14:39.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.899+0000 7f2e34c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2e301601c0 con 0x7f2e14004800 2024-04-16T16:14:39.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.899+0000 7f2e2e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e3013e0a0 0x7f2e3013e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:39.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.900+0000 7f2e2e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e3013e0a0 0x7f2e3013e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33426/0 (socket says 172.21.15.45:33426) 2024-04-16T16:14:39.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.900+0000 7f2e2e800640 1 -- 172.21.15.45:0/4248282688 learned_addr learned my addr 172.21.15.45:0/4248282688 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:39.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.900+0000 7f2e2de00640 1 --2- 172.21.15.45:0/4248282688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e14004800 0x7f2e3013c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:39.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.900+0000 7f2e25400640 1 --2- 172.21.15.45:0/4248282688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e3015e500 0x7f2e30138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:39.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.900+0000 7f2e2e800640 1 -- 172.21.15.45:0/4248282688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e14004800 msgr2=0x7f2e3013c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:39.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.900+0000 7f2e2e800640 1 --2- 172.21.15.45:0/4248282688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e14004800 0x7f2e3013c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:39.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.900+0000 7f2e2e800640 1 -- 172.21.15.45:0/4248282688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e3015e500 msgr2=0x7f2e30138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:39.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.900+0000 7f2e2e800640 1 --2- 172.21.15.45:0/4248282688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e3015e500 0x7f2e30138b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:39.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.900+0000 7f2e2e800640 1 -- 172.21.15.45:0/4248282688 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e30163160 con 0x7f2e3013e0a0 2024-04-16T16:14:39.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.900+0000 7f2e2e800640 1 --2- 172.21.15.45:0/4248282688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e3013e0a0 0x7f2e3013e520 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2e3006c960 tx=0x7f2e2800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:39.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.901+0000 7f2e26a00640 1 -- 172.21.15.45:0/4248282688 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e28007440 con 0x7f2e3013e0a0 2024-04-16T16:14:39.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.901+0000 7f2e26a00640 1 -- 172.21.15.45:0/4248282688 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e28007600 con 0x7f2e3013e0a0 2024-04-16T16:14:39.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.901+0000 7f2e26a00640 1 -- 172.21.15.45:0/4248282688 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e28004a20 con 0x7f2e3013e0a0 2024-04-16T16:14:39.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.901+0000 7f2e34c00640 1 -- 172.21.15.45:0/4248282688 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e30163440 con 0x7f2e3013e0a0 2024-04-16T16:14:39.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.901+0000 7f2e34c00640 1 -- 172.21.15.45:0/4248282688 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e3013f050 con 0x7f2e3013e0a0 2024-04-16T16:14:39.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.902+0000 7f2e34c00640 1 -- 172.21.15.45:0/4248282688 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e300b3060 con 0x7f2e3013e0a0 2024-04-16T16:14:39.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.903+0000 7f2e26a00640 1 -- 172.21.15.45:0/4248282688 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2e2801b020 con 0x7f2e3013e0a0 2024-04-16T16:14:39.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.904+0000 7f2e26a00640 1 --2- 172.21.15.45:0/4248282688 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2e0c06c920 0x7f2e0c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:39.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.904+0000 7f2e26a00640 1 -- 172.21.15.45:0/4248282688 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f2e2808b550 con 0x7f2e3013e0a0 2024-04-16T16:14:39.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.905+0000 7f2e2de00640 1 --2- 172.21.15.45:0/4248282688 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2e0c06c920 0x7f2e0c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:39.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.906+0000 7f2e2de00640 1 --2- 172.21.15.45:0/4248282688 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2e0c06c920 0x7f2e0c06ede0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2e30146b00 tx=0x7f2e2000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:39.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:39.909+0000 7f2e26a00640 1 -- 172.21.15.45:0/4248282688 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e300b3060 con 0x7f2e3013e0a0 2024-04-16T16:14:40.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.116+0000 7f2e34c00640 1 -- 172.21.15.45:0/4248282688 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2e30075780 con 0x7f2e3013e0a0 2024-04-16T16:14:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.118+0000 7f2e26a00640 1 -- 172.21.15.45:0/4248282688 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16682 (secure 0 0 0) 0x7f2e280598e0 con 0x7f2e3013e0a0 2024-04-16T16:14:40.119 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:40.119 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:14:39.132051+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:14:37.171798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-16T16:14:38.122496+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:40.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.121+0000 7f2e34c00640 1 -- 172.21.15.45:0/4248282688 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2e0c06c920 msgr2=0x7f2e0c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:40.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.121+0000 7f2e34c00640 1 --2- 172.21.15.45:0/4248282688 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2e0c06c920 0x7f2e0c06ede0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2e30146b00 tx=0x7f2e2000b040 comp rx=0 tx=0).stop 2024-04-16T16:14:40.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.121+0000 7f2e34c00640 1 -- 172.21.15.45:0/4248282688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e3013e0a0 msgr2=0x7f2e3013e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:40.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.121+0000 7f2e34c00640 1 --2- 172.21.15.45:0/4248282688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e3013e0a0 0x7f2e3013e520 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2e3006c960 tx=0x7f2e2800d9d0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.121+0000 7f2e34c00640 1 -- 172.21.15.45:0/4248282688 shutdown_connections 2024-04-16T16:14:40.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.121+0000 7f2e34c00640 1 --2- 172.21.15.45:0/4248282688 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f2e0c06c920 0x7f2e0c06ede0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.121+0000 7f2e34c00640 1 --2- 172.21.15.45:0/4248282688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e3013e0a0 0x7f2e3013e520 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.121+0000 7f2e34c00640 1 --2- 172.21.15.45:0/4248282688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e3015e500 0x7f2e30138b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.121+0000 7f2e34c00640 1 --2- 172.21.15.45:0/4248282688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e14004800 0x7f2e3013c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.121+0000 7f2e34c00640 1 -- 172.21.15.45:0/4248282688 >> 172.21.15.45:0/4248282688 conn(0x7f2e300b2750 msgr2=0x7f2e300b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:40.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.122+0000 7f2e34c00640 1 -- 172.21.15.45:0/4248282688 shutdown_connections 2024-04-16T16:14:40.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.122+0000 7f2e34c00640 1 -- 172.21.15.45:0/4248282688 wait complete. 2024-04-16T16:14:40.141 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:40.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.359+0000 7f31cd000640 1 Processor -- start 2024-04-16T16:14:40.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.360+0000 7f31cd000640 1 -- start start 2024-04-16T16:14:40.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.361+0000 7f31cd000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c8071a40 0x7f31c80b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:40.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.361+0000 7f31cd000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f31c815cd40 0x7f31c80d1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:40.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.361+0000 7f31cd000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31c81604d0 con 0x7f31c80b7750 2024-04-16T16:14:40.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.361+0000 7f31cd000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f31c81607b0 con 0x7f31c815cd40 2024-04-16T16:14:40.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.361+0000 7f31cd000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f31c8160a90 con 0x7f31c8071a40 2024-04-16T16:14:40.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.361+0000 7f31c6a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f31c80b7750 0x7f31c8071e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:34564/0 (socket says 172.21.15.45:34564) 2024-04-16T16:14:40.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.361+0000 7f31c7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f31c815cd40 0x7f31c80d1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:40.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.361+0000 7f31c6a00640 1 -- 172.21.15.45:0/352433362 learned_addr learned my addr 172.21.15.45:0/352433362 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:40.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.362+0000 7f31c7400640 1 --2- 172.21.15.45:0/352433362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c8071a40 0x7f31c80b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:40.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.362+0000 7f31c7e00640 1 -- 172.21.15.45:0/352433362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c8071a40 msgr2=0x7f31c80b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:40.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.362+0000 7f31c7e00640 1 --2- 172.21.15.45:0/352433362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c8071a40 0x7f31c80b56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.362+0000 7f31c7e00640 1 -- 172.21.15.45:0/352433362 >> v1:172.21.15.45:6789/0 conn(0x7f31c80b7750 legacy=0x7f31c8071e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:40.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.362+0000 7f31c6000640 1 -- 172.21.15.45:0/352433362 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 342309731 0 0) 0x7f31c81604d0 con 0x7f31c80b7750 2024-04-16T16:14:40.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.362+0000 7f31c7e00640 1 -- 172.21.15.45:0/352433362 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c8160d70 con 0x7f31c815cd40 2024-04-16T16:14:40.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.363+0000 7f31c7e00640 1 --2- 172.21.15.45:0/352433362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c815cd40 0x7f31c80d1500 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009d90 tx=0x7f31c002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36dcb768cdd38fe server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:40.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.363+0000 7f31c6000640 1 -- 172.21.15.45:0/352433362 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c0039070 con 0x7f31c815cd40 2024-04-16T16:14:40.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.363+0000 7f31c6000640 1 -- 172.21.15.45:0/352433362 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31c002cc50 con 0x7f31c815cd40 2024-04-16T16:14:40.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.364+0000 7f31c6000640 1 -- 172.21.15.45:0/352433362 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c0034440 con 0x7f31c815cd40 2024-04-16T16:14:40.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.364+0000 7f31cd000640 1 -- 172.21.15.45:0/352433362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c815cd40 msgr2=0x7f31c80d1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:40.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.364+0000 7f31cd000640 1 --2- 172.21.15.45:0/352433362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c815cd40 0x7f31c80d1500 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009d90 tx=0x7f31c002bea0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.364+0000 7f31cd000640 1 -- 172.21.15.45:0/352433362 shutdown_connections 2024-04-16T16:14:40.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.364+0000 7f31cd000640 1 --2- 172.21.15.45:0/352433362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c815cd40 0x7f31c80d1500 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.364+0000 7f31cd000640 1 --2- 172.21.15.45:0/352433362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c8071a40 0x7f31c80b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.364+0000 7f31cd000640 1 -- 172.21.15.45:0/352433362 >> 172.21.15.45:0/352433362 conn(0x7f31c80b43a0 msgr2=0x7f31c80b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:40.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.364+0000 7f31cd000640 1 -- 172.21.15.45:0/352433362 shutdown_connections 2024-04-16T16:14:40.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.364+0000 7f31cd000640 1 -- 172.21.15.45:0/352433362 wait complete. 2024-04-16T16:14:40.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.366+0000 7f31cd000640 1 Processor -- start 2024-04-16T16:14:40.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.366+0000 7f31cd000640 1 -- start start 2024-04-16T16:14:40.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.367+0000 7f31cd000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8071a40 0x7f31c8146f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:40.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.367+0000 7f31cd000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31c80b7750 0x7f31c8143a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:40.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.367+0000 7f31cd000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cd40 0x7f31c81f7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:40.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.367+0000 7f31cd000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f31c8072b20 con 0x7f31c8071a40 2024-04-16T16:14:40.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.367+0000 7f31cd000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f31c815a670 con 0x7f31c80b7750 2024-04-16T16:14:40.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.367+0000 7f31cd000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f31c80d1a40 con 0x7f31c815cd40 2024-04-16T16:14:40.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.367+0000 7f31c7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8071a40 0x7f31c8146f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:40.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.367+0000 7f31c7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8071a40 0x7f31c8146f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33444/0 (socket says 172.21.15.45:33444) 2024-04-16T16:14:40.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.367+0000 7f31c7400640 1 -- 172.21.15.45:0/576088635 learned_addr learned my addr 172.21.15.45:0/576088635 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:40.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.368+0000 7f31c7e00640 1 --2- 172.21.15.45:0/576088635 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cd40 0x7f31c81f7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:40.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.368+0000 7f31c6a00640 1 --2- 172.21.15.45:0/576088635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31c80b7750 0x7f31c8143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:40.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.368+0000 7f31c7400640 1 -- 172.21.15.45:0/576088635 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cd40 msgr2=0x7f31c81f7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:40.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.368+0000 7f31c7400640 1 --2- 172.21.15.45:0/576088635 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cd40 0x7f31c81f7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.368+0000 7f31c7400640 1 -- 172.21.15.45:0/576088635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31c80b7750 msgr2=0x7f31c8143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:40.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.368+0000 7f31c7400640 1 --2- 172.21.15.45:0/576088635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31c80b7750 0x7f31c8143a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.368+0000 7f31c7400640 1 -- 172.21.15.45:0/576088635 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c8155990 con 0x7f31c8071a40 2024-04-16T16:14:40.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.368+0000 7f31c6a00640 1 --2- 172.21.15.45:0/576088635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31c80b7750 0x7f31c8143a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:40.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.368+0000 7f31c7400640 1 --2- 172.21.15.45:0/576088635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8071a40 0x7f31c8146f30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f31b800d830 tx=0x7f31b800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:40.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.369+0000 7f31bf400640 1 -- 172.21.15.45:0/576088635 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b800b6c0 con 0x7f31c8071a40 2024-04-16T16:14:40.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.369+0000 7f31bf400640 1 -- 172.21.15.45:0/576088635 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31b8019070 con 0x7f31c8071a40 2024-04-16T16:14:40.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.369+0000 7f31bf400640 1 -- 172.21.15.45:0/576088635 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b800ba80 con 0x7f31c8071a40 2024-04-16T16:14:40.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.369+0000 7f31cd000640 1 -- 172.21.15.45:0/576088635 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c81f88e0 con 0x7f31c8071a40 2024-04-16T16:14:40.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.369+0000 7f31cd000640 1 -- 172.21.15.45:0/576088635 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c8146730 con 0x7f31c8071a40 2024-04-16T16:14:40.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.370+0000 7f31cd000640 1 -- 172.21.15.45:0/576088635 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31c81567a0 con 0x7f31c8071a40 2024-04-16T16:14:40.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.371+0000 7f31bf400640 1 -- 172.21.15.45:0/576088635 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f31b800bc20 con 0x7f31c8071a40 2024-04-16T16:14:40.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.372+0000 7f31bf400640 1 --2- 172.21.15.45:0/576088635 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f31a406c940 0x7f31a406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:40.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.372+0000 7f31bf400640 1 -- 172.21.15.45:0/576088635 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f31b808bfb0 con 0x7f31c8071a40 2024-04-16T16:14:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.373+0000 7f31c6a00640 1 --2- 172.21.15.45:0/576088635 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f31a406c940 0x7f31a406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:40.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.374+0000 7f31c6a00640 1 --2- 172.21.15.45:0/576088635 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f31a406c940 0x7f31a406ee00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f31c8156200 tx=0x7f31b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:40.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.378+0000 7f31bf400640 1 -- 172.21.15.45:0/576088635 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f31b805a450 con 0x7f31c8071a40 2024-04-16T16:14:40.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.584+0000 7f31cd000640 1 -- 172.21.15.45:0/576088635 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f31c80b60a0 con 0x7f31c8071a40 2024-04-16T16:14:40.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.588+0000 7f31bf400640 1 -- 172.21.15.45:0/576088635 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16682 (secure 0 0 0) 0x7f31b8059d50 con 0x7f31c8071a40 2024-04-16T16:14:40.588 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:40.588 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:14:39.132051+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:14:37.171798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-16T16:14:38.122496+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.590+0000 7f31cd000640 1 -- 172.21.15.45:0/576088635 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f31a406c940 msgr2=0x7f31a406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.590+0000 7f31cd000640 1 --2- 172.21.15.45:0/576088635 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f31a406c940 0x7f31a406ee00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f31c8156200 tx=0x7f31b0009040 comp rx=0 tx=0).stop 2024-04-16T16:14:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.590+0000 7f31cd000640 1 -- 172.21.15.45:0/576088635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8071a40 msgr2=0x7f31c8146f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.590+0000 7f31cd000640 1 --2- 172.21.15.45:0/576088635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8071a40 0x7f31c8146f30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f31b800d830 tx=0x7f31b800dcf0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.590+0000 7f31cd000640 1 -- 172.21.15.45:0/576088635 shutdown_connections 2024-04-16T16:14:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.590+0000 7f31cd000640 1 --2- 172.21.15.45:0/576088635 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f31a406c940 0x7f31a406ee00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.590+0000 7f31cd000640 1 --2- 172.21.15.45:0/576088635 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cd40 0x7f31c81f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.590+0000 7f31cd000640 1 --2- 172.21.15.45:0/576088635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31c80b7750 0x7f31c8143a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.590+0000 7f31cd000640 1 --2- 172.21.15.45:0/576088635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8071a40 0x7f31c8146f30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.591+0000 7f31cd000640 1 -- 172.21.15.45:0/576088635 >> 172.21.15.45:0/576088635 conn(0x7f31c80b43a0 msgr2=0x7f31c8071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.591+0000 7f31cd000640 1 -- 172.21.15.45:0/576088635 shutdown_connections 2024-04-16T16:14:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.591+0000 7f31cd000640 1 -- 172.21.15.45:0/576088635 wait complete. 2024-04-16T16:14:40.610 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-16T16:14:40.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.812+0000 7f54d9800640 1 Processor -- start 2024-04-16T16:14:40.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.812+0000 7f54d9800640 1 -- start start 2024-04-16T16:14:40.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.814+0000 7f54d9800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f54d40b6000 0x7f54d40ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:40.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.814+0000 7f54d9800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54d4148570 con 0x7f54d4148190 2024-04-16T16:14:40.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.814+0000 7f54d9800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54d415efa0 con 0x7f54d40ceb50 2024-04-16T16:14:40.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.814+0000 7f54d9800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f54d415fbb0 con 0x7f54d40b6000 2024-04-16T16:14:40.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.814+0000 7f54d3e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f54d4148190 0x7f54d40b58f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:34568/0 (socket says 172.21.15.45:34568) 2024-04-16T16:14:40.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.814+0000 7f54d3e00640 1 -- 172.21.15.45:0/114047893 learned_addr learned my addr 172.21.15.45:0/114047893 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:40.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.815+0000 7f54d3400640 1 --2- 172.21.15.45:0/114047893 >> v2:172.21.15.45:3300/0 conn(0x7f54d40b6000 0x7f54d40ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:40.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.815+0000 7f54d2a00640 1 -- 172.21.15.45:0/114047893 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3859201240 0 0) 0x7f54d4148570 con 0x7f54d4148190 2024-04-16T16:14:40.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.815+0000 7f54d2a00640 1 -- 172.21.15.45:0/114047893 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54b8003520 con 0x7f54d4148190 2024-04-16T16:14:40.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.815+0000 7f54d3400640 1 -- 172.21.15.45:0/114047893 >> v1:172.21.15.103:6789/0 conn(0x7f54d40ceb50 legacy=0x7f54d4160040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:40.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.815+0000 7f54d3400640 1 -- 172.21.15.45:0/114047893 >> v1:172.21.15.45:6789/0 conn(0x7f54d4148190 legacy=0x7f54d40b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:40.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.815+0000 7f54d3400640 1 -- 172.21.15.45:0/114047893 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d415f280 con 0x7f54d40b6000 2024-04-16T16:14:40.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.816+0000 7f54d3400640 1 --2- 172.21.15.45:0/114047893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54d40b6000 0x7f54d40ce610 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f54c00097a0 tx=0x7f54c002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f72e8f424a4de5 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:40.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.817+0000 7f54d2a00640 1 -- 172.21.15.45:0/114047893 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c0034070 con 0x7f54d40b6000 2024-04-16T16:14:40.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.817+0000 7f54d2a00640 1 -- 172.21.15.45:0/114047893 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54c0030610 con 0x7f54d40b6000 2024-04-16T16:14:40.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.817+0000 7f54d2a00640 1 -- 172.21.15.45:0/114047893 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c00309e0 con 0x7f54d40b6000 2024-04-16T16:14:40.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.817+0000 7f54d9800640 1 -- 172.21.15.45:0/114047893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54d40b6000 msgr2=0x7f54d40ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:40.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.817+0000 7f54d9800640 1 --2- 172.21.15.45:0/114047893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54d40b6000 0x7f54d40ce610 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f54c00097a0 tx=0x7f54c002b960 comp rx=0 tx=0).stop 2024-04-16T16:14:40.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.817+0000 7f54d9800640 1 -- 172.21.15.45:0/114047893 shutdown_connections 2024-04-16T16:14:40.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.817+0000 7f54d9800640 1 --2- 172.21.15.45:0/114047893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54d40b6000 0x7f54d40ce610 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.817+0000 7f54d9800640 1 -- 172.21.15.45:0/114047893 >> 172.21.15.45:0/114047893 conn(0x7f54d406b930 msgr2=0x7f54d406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:40.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.818+0000 7f54d9800640 1 -- 172.21.15.45:0/114047893 shutdown_connections 2024-04-16T16:14:40.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.818+0000 7f54d9800640 1 -- 172.21.15.45:0/114047893 wait complete. 2024-04-16T16:14:40.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.819+0000 7f54d9800640 1 Processor -- start 2024-04-16T16:14:40.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.819+0000 7f54d9800640 1 -- start start 2024-04-16T16:14:40.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.820+0000 7f54d9800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54d40b6000 0x7f54d40ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:40.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.820+0000 7f54d9800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f54d40ceb50 0x7f54d413f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:40.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.820+0000 7f54d9800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54d4148190 0x7f54d4143970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:40.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.820+0000 7f54d9800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f54d40cefc0 con 0x7f54d4148190 2024-04-16T16:14:40.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.820+0000 7f54d9800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f54d415f840 con 0x7f54d40b6000 2024-04-16T16:14:40.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.820+0000 7f54d9800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f54d40cf140 con 0x7f54d40ceb50 2024-04-16T16:14:40.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.821+0000 7f54d8e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54d4148190 0x7f54d4143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:40.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.821+0000 7f54d8e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54d4148190 0x7f54d4143970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33472/0 (socket says 172.21.15.45:33472) 2024-04-16T16:14:40.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.821+0000 7f54d8e00640 1 -- 172.21.15.45:0/863927834 learned_addr learned my addr 172.21.15.45:0/863927834 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:40.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.821+0000 7f54d3e00640 1 --2- 172.21.15.45:0/863927834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54d40b6000 0x7f54d40ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:40.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.821+0000 7f54d3400640 1 --2- 172.21.15.45:0/863927834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f54d40ceb50 0x7f54d413f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:40.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.821+0000 7f54d8e00640 1 -- 172.21.15.45:0/863927834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f54d40ceb50 msgr2=0x7f54d413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:40.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.821+0000 7f54d8e00640 1 --2- 172.21.15.45:0/863927834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f54d40ceb50 0x7f54d413f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.821+0000 7f54d8e00640 1 -- 172.21.15.45:0/863927834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54d40b6000 msgr2=0x7f54d40ccf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:40.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.821+0000 7f54d8e00640 1 --2- 172.21.15.45:0/863927834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54d40b6000 0x7f54d40ccf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:40.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.821+0000 7f54d8e00640 1 -- 172.21.15.45:0/863927834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d413f860 con 0x7f54d4148190 2024-04-16T16:14:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.822+0000 7f54d8e00640 1 --2- 172.21.15.45:0/863927834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54d4148190 0x7f54d4143970 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00c7a0 tx=0x7f54cc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.822+0000 7f54cbe00640 1 -- 172.21.15.45:0/863927834 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc003b70 con 0x7f54d4148190 2024-04-16T16:14:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.822+0000 7f54cbe00640 1 -- 172.21.15.45:0/863927834 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54cc003d30 con 0x7f54d4148190 2024-04-16T16:14:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.822+0000 7f54d9800640 1 -- 172.21.15.45:0/863927834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54d413fb40 con 0x7f54d4148190 2024-04-16T16:14:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.822+0000 7f54cbe00640 1 -- 172.21.15.45:0/863927834 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc00aa90 con 0x7f54d4148190 2024-04-16T16:14:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.822+0000 7f54d9800640 1 -- 172.21.15.45:0/863927834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54d4140100 con 0x7f54d4148190 2024-04-16T16:14:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.823+0000 7f54d9800640 1 -- 172.21.15.45:0/863927834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54d4075780 con 0x7f54d4148190 2024-04-16T16:14:40.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.824+0000 7f54cbe00640 1 -- 172.21.15.45:0/863927834 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f54cc00ac30 con 0x7f54d4148190 2024-04-16T16:14:40.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.825+0000 7f54cbe00640 1 --2- 172.21.15.45:0/863927834 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f54b406c920 0x7f54b406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:40.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.825+0000 7f54cbe00640 1 -- 172.21.15.45:0/863927834 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f54cc08ba60 con 0x7f54d4148190 2024-04-16T16:14:40.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.826+0000 7f54d3e00640 1 --2- 172.21.15.45:0/863927834 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f54b406c920 0x7f54b406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:40.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.827+0000 7f54d3e00640 1 --2- 172.21.15.45:0/863927834 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f54b406c920 0x7f54b406ede0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f54c40011e0 tx=0x7f54c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:40.830+0000 7f54cbe00640 1 -- 172.21.15.45:0/863927834 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f54cc059e80 con 0x7f54d4148190 2024-04-16T16:14:41.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.076+0000 7f54d9800640 1 -- 172.21.15.45:0/863927834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f54d40b6590 con 0x7f54d4148190 2024-04-16T16:14:41.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.703+0000 7f54cbe00640 1 -- 172.21.15.45:0/863927834 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f54cc059780 con 0x7f54d4148190 2024-04-16T16:14:41.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.707+0000 7f54d9800640 1 -- 172.21.15.45:0/863927834 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f54b406c920 msgr2=0x7f54b406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:41.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.707+0000 7f54d9800640 1 --2- 172.21.15.45:0/863927834 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f54b406c920 0x7f54b406ede0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f54c40011e0 tx=0x7f54c4009040 comp rx=0 tx=0).stop 2024-04-16T16:14:41.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.707+0000 7f54d9800640 1 -- 172.21.15.45:0/863927834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54d4148190 msgr2=0x7f54d4143970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:41.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.707+0000 7f54d9800640 1 --2- 172.21.15.45:0/863927834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54d4148190 0x7f54d4143970 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00c7a0 tx=0x7f54cc00cc60 comp rx=0 tx=0).stop 2024-04-16T16:14:41.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.707+0000 7f54d9800640 1 -- 172.21.15.45:0/863927834 shutdown_connections 2024-04-16T16:14:41.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.707+0000 7f54d9800640 1 --2- 172.21.15.45:0/863927834 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f54b406c920 0x7f54b406ede0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:41.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.707+0000 7f54d9800640 1 --2- 172.21.15.45:0/863927834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54d4148190 0x7f54d4143970 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:41.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.707+0000 7f54d9800640 1 --2- 172.21.15.45:0/863927834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f54d40ceb50 0x7f54d413f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:41.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.707+0000 7f54d9800640 1 --2- 172.21.15.45:0/863927834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54d40b6000 0x7f54d40ccf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:41.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.707+0000 7f54d9800640 1 -- 172.21.15.45:0/863927834 >> 172.21.15.45:0/863927834 conn(0x7f54d406b930 msgr2=0x7f54d40b5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:41.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.707+0000 7f54d9800640 1 -- 172.21.15.45:0/863927834 shutdown_connections 2024-04-16T16:14:41.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.708+0000 7f54d9800640 1 -- 172.21.15.45:0/863927834 wait complete. 2024-04-16T16:14:41.729 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-16T16:14:41.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-16T16:14:41.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.952+0000 7f830e400640 1 Processor -- start 2024-04-16T16:14:41.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.953+0000 7f830e400640 1 -- start start 2024-04-16T16:14:41.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.954+0000 7f830e400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83081461b0 0x7f8308160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:41.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.954+0000 7f830e400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f83080b58f0 0x7f83080b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:41.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.954+0000 7f830e400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f83080cf440 0x7f83080cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:41.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.954+0000 7f830e400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f830806dfd0 con 0x7f83080cf440 2024-04-16T16:14:41.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.954+0000 7f830e400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f83080b6290 con 0x7f83080b58f0 2024-04-16T16:14:41.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.954+0000 7f830e400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8308146590 con 0x7f83081461b0 2024-04-16T16:14:41.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.955+0000 7f830da00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f83080cf440 0x7f83080cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:41.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.955+0000 7f830da00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f83080cf440 0x7f83080cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33480/0 (socket says 172.21.15.45:33480) 2024-04-16T16:14:41.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.955+0000 7f830da00640 1 -- 172.21.15.45:0/743195383 learned_addr learned my addr 172.21.15.45:0/743195383 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.955+0000 7f830d000640 1 --2- 172.21.15.45:0/743195383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83081461b0 0x7f8308160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.955+0000 7f8307e00640 1 --2- 172.21.15.45:0/743195383 >> v2:172.21.15.103:3300/0 conn(0x7f83080b58f0 0x7f83080b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.955+0000 7f830da00640 1 -- 172.21.15.45:0/743195383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83081461b0 msgr2=0x7f8308160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.955+0000 7f830da00640 1 --2- 172.21.15.45:0/743195383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83081461b0 0x7f8308160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.955+0000 7f830da00640 1 -- 172.21.15.45:0/743195383 >> v2:172.21.15.103:3300/0 conn(0x7f83080b58f0 msgr2=0x7f83080b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.955+0000 7f830da00640 1 --2- 172.21.15.45:0/743195383 >> v2:172.21.15.103:3300/0 conn(0x7f83080b58f0 0x7f83080b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.955+0000 7f830da00640 1 -- 172.21.15.45:0/743195383 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83080c5750 con 0x7f83080cf440 2024-04-16T16:14:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.957+0000 7f830da00640 1 --2- 172.21.15.45:0/743195383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83080cf440 0x7f83080cf8d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f83080d04d0 tx=0x7f830002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ea51ea5daa299c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.957+0000 7f8307400640 1 -- 172.21.15.45:0/743195383 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830002ca80 con 0x7f83080cf440 2024-04-16T16:14:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.957+0000 7f8307400640 1 -- 172.21.15.45:0/743195383 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f830002cc20 con 0x7f83080cf440 2024-04-16T16:14:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.957+0000 7f8307400640 1 -- 172.21.15.45:0/743195383 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83000343f0 con 0x7f83080cf440 2024-04-16T16:14:41.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.958+0000 7f830e400640 1 -- 172.21.15.45:0/743195383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83080cf440 msgr2=0x7f83080cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:41.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.958+0000 7f830e400640 1 --2- 172.21.15.45:0/743195383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83080cf440 0x7f83080cf8d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f83080d04d0 tx=0x7f830002bee0 comp rx=0 tx=0).stop 2024-04-16T16:14:41.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.958+0000 7f830e400640 1 -- 172.21.15.45:0/743195383 shutdown_connections 2024-04-16T16:14:41.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.958+0000 7f830e400640 1 --2- 172.21.15.45:0/743195383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83080cf440 0x7f83080cf8d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:41.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.958+0000 7f830e400640 1 --2- 172.21.15.45:0/743195383 >> v2:172.21.15.103:3300/0 conn(0x7f83080b58f0 0x7f83080b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:41.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.958+0000 7f830e400640 1 --2- 172.21.15.45:0/743195383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83081461b0 0x7f8308160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:41.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.958+0000 7f830e400640 1 -- 172.21.15.45:0/743195383 >> 172.21.15.45:0/743195383 conn(0x7f830806b930 msgr2=0x7f830806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:41.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.958+0000 7f830e400640 1 -- 172.21.15.45:0/743195383 shutdown_connections 2024-04-16T16:14:41.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.958+0000 7f830e400640 1 -- 172.21.15.45:0/743195383 wait complete. 2024-04-16T16:14:41.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.960+0000 7f830e400640 1 Processor -- start 2024-04-16T16:14:41.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.961+0000 7f830e400640 1 -- start start 2024-04-16T16:14:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.962+0000 7f830e400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83080b58f0 0x7f830813ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.962+0000 7f830e400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83080cf440 0x7f83080c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.962+0000 7f830e400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83081461b0 0x7f83081e8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.962+0000 7f830e400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f83080c59f0 con 0x7f83081461b0 2024-04-16T16:14:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.962+0000 7f830e400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8308161230 con 0x7f83080b58f0 2024-04-16T16:14:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.962+0000 7f830e400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f83080c5cd0 con 0x7f83080cf440 2024-04-16T16:14:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.962+0000 7f830da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83081461b0 0x7f83081e8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.962+0000 7f830da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83081461b0 0x7f83081e8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33482/0 (socket says 172.21.15.45:33482) 2024-04-16T16:14:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.962+0000 7f830da00640 1 -- 172.21.15.45:0/2726279674 learned_addr learned my addr 172.21.15.45:0/2726279674 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.963+0000 7f8307e00640 1 --2- 172.21.15.45:0/2726279674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83080cf440 0x7f83080c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.963+0000 7f830da00640 1 -- 172.21.15.45:0/2726279674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83080cf440 msgr2=0x7f83080c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.963+0000 7f830da00640 1 --2- 172.21.15.45:0/2726279674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83080cf440 0x7f83080c89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.963+0000 7f830da00640 1 -- 172.21.15.45:0/2726279674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83080b58f0 msgr2=0x7f830813ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.963+0000 7f830da00640 1 --2- 172.21.15.45:0/2726279674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83080b58f0 0x7f830813ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.963+0000 7f830da00640 1 -- 172.21.15.45:0/2726279674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8308144680 con 0x7f83081461b0 2024-04-16T16:14:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.963+0000 7f8307e00640 1 --2- 172.21.15.45:0/2726279674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83080cf440 0x7f83080c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.963+0000 7f830da00640 1 --2- 172.21.15.45:0/2726279674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83081461b0 0x7f83081e8290 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8300009f40 tx=0x7f8300004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.963+0000 7f8304c00640 1 -- 172.21.15.45:0/2726279674 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8300043020 con 0x7f83081461b0 2024-04-16T16:14:41.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.963+0000 7f8304c00640 1 -- 172.21.15.45:0/2726279674 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f830003e450 con 0x7f83081461b0 2024-04-16T16:14:41.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.963+0000 7f8304c00640 1 -- 172.21.15.45:0/2726279674 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8300009510 con 0x7f83081461b0 2024-04-16T16:14:41.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.963+0000 7f830e400640 1 -- 172.21.15.45:0/2726279674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8308144990 con 0x7f83081461b0 2024-04-16T16:14:41.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.964+0000 7f830e400640 1 -- 172.21.15.45:0/2726279674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83081404e0 con 0x7f83081461b0 2024-04-16T16:14:41.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.965+0000 7f830e400640 1 -- 172.21.15.45:0/2726279674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8308075780 con 0x7f83081461b0 2024-04-16T16:14:41.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.966+0000 7f8304c00640 1 -- 172.21.15.45:0/2726279674 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8300046020 con 0x7f83081461b0 2024-04-16T16:14:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.967+0000 7f8304c00640 1 --2- 172.21.15.45:0/2726279674 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f82e806c8f0 0x7f82e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.967+0000 7f8304c00640 1 -- 172.21.15.45:0/2726279674 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f83000ae420 con 0x7f83081461b0 2024-04-16T16:14:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.968+0000 7f830d000640 1 --2- 172.21.15.45:0/2726279674 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f82e806c8f0 0x7f82e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:41.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.969+0000 7f830d000640 1 --2- 172.21.15.45:0/2726279674 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f82e806c8f0 0x7f82e806edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f82fc006fd0 tx=0x7f82fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:41.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:41.972+0000 7f8304c00640 1 -- 172.21.15.45:0/2726279674 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f830004d030 con 0x7f83081461b0 2024-04-16T16:14:42.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.223+0000 7f830e400640 1 -- 172.21.15.45:0/2726279674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f83081416b0 con 0x7f83081461b0 2024-04-16T16:14:42.538 INFO:tasks.ceph.mds.a.smithi045.stderr:2024-04-16T16:14:42.538+0000 7fdea9200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:14:42.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.706+0000 7f8304c00640 1 -- 172.21.15.45:0/2726279674 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f8308144990 con 0x7f83081461b0 2024-04-16T16:14:42.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.710+0000 7f830e400640 1 -- 172.21.15.45:0/2726279674 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f82e806c8f0 msgr2=0x7f82e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:42.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.710+0000 7f830e400640 1 --2- 172.21.15.45:0/2726279674 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f82e806c8f0 0x7f82e806edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f82fc006fd0 tx=0x7f82fc008040 comp rx=0 tx=0).stop 2024-04-16T16:14:42.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.710+0000 7f830e400640 1 -- 172.21.15.45:0/2726279674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83081461b0 msgr2=0x7f83081e8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:42.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.710+0000 7f830e400640 1 --2- 172.21.15.45:0/2726279674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83081461b0 0x7f83081e8290 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8300009f40 tx=0x7f8300004a40 comp rx=0 tx=0).stop 2024-04-16T16:14:42.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.710+0000 7f830e400640 1 -- 172.21.15.45:0/2726279674 shutdown_connections 2024-04-16T16:14:42.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.710+0000 7f830e400640 1 --2- 172.21.15.45:0/2726279674 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f82e806c8f0 0x7f82e806edb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:42.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.710+0000 7f830e400640 1 --2- 172.21.15.45:0/2726279674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83081461b0 0x7f83081e8290 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:42.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.710+0000 7f830e400640 1 --2- 172.21.15.45:0/2726279674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83080cf440 0x7f83080c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:42.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.710+0000 7f830e400640 1 --2- 172.21.15.45:0/2726279674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83080b58f0 0x7f830813ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:42.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.710+0000 7f830e400640 1 -- 172.21.15.45:0/2726279674 >> 172.21.15.45:0/2726279674 conn(0x7f830806b930 msgr2=0x7f83080b50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:42.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.710+0000 7f830e400640 1 -- 172.21.15.45:0/2726279674 shutdown_connections 2024-04-16T16:14:42.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.711+0000 7f830e400640 1 -- 172.21.15.45:0/2726279674 wait complete. 2024-04-16T16:14:42.735 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-16T16:14:42.735 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:14:42.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.954+0000 7f3f6a200640 1 Processor -- start 2024-04-16T16:14:42.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.955+0000 7f3f6a200640 1 -- start start 2024-04-16T16:14:42.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.956+0000 7f3f6a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f640b0630 0x7f3f640b5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:42.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.956+0000 7f3f6a200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3f6406c9d0 0x7f3f640b5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:42.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.956+0000 7f3f6a200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f640b40f0 con 0x7f3f64067c20 2024-04-16T16:14:42.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.956+0000 7f3f6a200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3f640b43d0 con 0x7f3f6406c9d0 2024-04-16T16:14:42.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.956+0000 7f3f6a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3f640b46b0 con 0x7f3f640b0630 2024-04-16T16:14:42.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.957+0000 7f3f63e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3f6406c9d0 0x7f3f640b5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:42.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.957+0000 7f3f63e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3f6406c9d0 0x7f3f640b5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33494/0 (socket says 172.21.15.45:33494) 2024-04-16T16:14:42.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.957+0000 7f3f63e00640 1 -- 172.21.15.45:0/3765748933 learned_addr learned my addr 172.21.15.45:0/3765748933 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:42.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.957+0000 7f3f63400640 1 --2- 172.21.15.45:0/3765748933 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f640b0630 0x7f3f640b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:42.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.957+0000 7f3f63e00640 1 -- 172.21.15.45:0/3765748933 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f640b0630 msgr2=0x7f3f640b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:42.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.957+0000 7f3f63e00640 1 --2- 172.21.15.45:0/3765748933 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f640b0630 0x7f3f640b5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:42.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.957+0000 7f3f63e00640 1 -- 172.21.15.45:0/3765748933 >> v1:172.21.15.103:6789/0 conn(0x7f3f64067c20 legacy=0x7f3f641626c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:42.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.957+0000 7f3f63e00640 1 -- 172.21.15.45:0/3765748933 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f640b4990 con 0x7f3f6406c9d0 2024-04-16T16:14:42.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.959+0000 7f3f63e00640 1 --2- 172.21.15.45:0/3765748933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6406c9d0 0x7f3f640b5d70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c009e50 tx=0x7f3f5c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf994362cffdb53 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:42.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.959+0000 7f3f62000640 1 -- 172.21.15.45:0/3765748933 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c02c9e0 con 0x7f3f6406c9d0 2024-04-16T16:14:42.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.959+0000 7f3f62000640 1 -- 172.21.15.45:0/3765748933 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f5c02cb80 con 0x7f3f6406c9d0 2024-04-16T16:14:42.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.959+0000 7f3f62000640 1 -- 172.21.15.45:0/3765748933 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c0343f0 con 0x7f3f6406c9d0 2024-04-16T16:14:42.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.960+0000 7f3f6a200640 1 -- 172.21.15.45:0/3765748933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6406c9d0 msgr2=0x7f3f640b5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:42.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.960+0000 7f3f6a200640 1 --2- 172.21.15.45:0/3765748933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6406c9d0 0x7f3f640b5d70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c009e50 tx=0x7f3f5c02be40 comp rx=0 tx=0).stop 2024-04-16T16:14:42.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.960+0000 7f3f6a200640 1 -- 172.21.15.45:0/3765748933 shutdown_connections 2024-04-16T16:14:42.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.960+0000 7f3f6a200640 1 --2- 172.21.15.45:0/3765748933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6406c9d0 0x7f3f640b5d70 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:42.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.960+0000 7f3f6a200640 1 --2- 172.21.15.45:0/3765748933 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f640b0630 0x7f3f640b5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:42.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.960+0000 7f3f6a200640 1 -- 172.21.15.45:0/3765748933 >> 172.21.15.45:0/3765748933 conn(0x7f3f6406e9b0 msgr2=0x7f3f6406edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:42.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.960+0000 7f3f6a200640 1 -- 172.21.15.45:0/3765748933 shutdown_connections 2024-04-16T16:14:42.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.960+0000 7f3f6a200640 1 -- 172.21.15.45:0/3765748933 wait complete. 2024-04-16T16:14:42.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.962+0000 7f3f6a200640 1 Processor -- start 2024-04-16T16:14:42.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.962+0000 7f3f6a200640 1 -- start start 2024-04-16T16:14:42.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.964+0000 7f3f6a200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f64067c20 0x7f3f64159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:42.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.964+0000 7f3f6a200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6406c9d0 0x7f3f6415d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:42.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.964+0000 7f3f6a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f640b0630 0x7f3f6415dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:42.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.964+0000 7f3f6a200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3f640680f0 con 0x7f3f6406c9d0 2024-04-16T16:14:42.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.964+0000 7f3f6a200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3f640b0a10 con 0x7f3f64067c20 2024-04-16T16:14:42.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.964+0000 7f3f6a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3f64068270 con 0x7f3f640b0630 2024-04-16T16:14:42.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.964+0000 7f3f62a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6406c9d0 0x7f3f6415d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:42.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.964+0000 7f3f62a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6406c9d0 0x7f3f6415d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33504/0 (socket says 172.21.15.45:33504) 2024-04-16T16:14:42.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.964+0000 7f3f62a00640 1 -- 172.21.15.45:0/3726705422 learned_addr learned my addr 172.21.15.45:0/3726705422 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:42.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.964+0000 7f3f63e00640 1 --2- 172.21.15.45:0/3726705422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f640b0630 0x7f3f6415dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:42.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.964+0000 7f3f62a00640 1 -- 172.21.15.45:0/3726705422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f640b0630 msgr2=0x7f3f6415dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:42.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.964+0000 7f3f62a00640 1 --2- 172.21.15.45:0/3726705422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f640b0630 0x7f3f6415dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:42.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.964+0000 7f3f62a00640 1 -- 172.21.15.45:0/3726705422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f64067c20 msgr2=0x7f3f64159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:42.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.964+0000 7f3f62a00640 1 --2- 172.21.15.45:0/3726705422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f64067c20 0x7f3f64159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:42.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.964+0000 7f3f62a00640 1 -- 172.21.15.45:0/3726705422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f64155f20 con 0x7f3f6406c9d0 2024-04-16T16:14:42.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.965+0000 7f3f62a00640 1 --2- 172.21.15.45:0/3726705422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6406c9d0 0x7f3f6415d730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00b3f0 tx=0x7f3f4c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:42.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.965+0000 7f3f5b400640 1 -- 172.21.15.45:0/3726705422 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c015020 con 0x7f3f6406c9d0 2024-04-16T16:14:42.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.965+0000 7f3f5b400640 1 -- 172.21.15.45:0/3726705422 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f4c011830 con 0x7f3f6406c9d0 2024-04-16T16:14:42.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.965+0000 7f3f5b400640 1 -- 172.21.15.45:0/3726705422 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c011c20 con 0x7f3f6406c9d0 2024-04-16T16:14:42.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.965+0000 7f3f6a200640 1 -- 172.21.15.45:0/3726705422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f64154840 con 0x7f3f6406c9d0 2024-04-16T16:14:42.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.965+0000 7f3f6a200640 1 -- 172.21.15.45:0/3726705422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f64156730 con 0x7f3f6406c9d0 2024-04-16T16:14:42.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.966+0000 7f3f6a200640 1 -- 172.21.15.45:0/3726705422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f64075780 con 0x7f3f6406c9d0 2024-04-16T16:14:42.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.967+0000 7f3f5b400640 1 -- 172.21.15.45:0/3726705422 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3f4c018020 con 0x7f3f6406c9d0 2024-04-16T16:14:42.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.968+0000 7f3f5b400640 1 --2- 172.21.15.45:0/3726705422 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3f4006c8f0 0x7f3f4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:42.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.968+0000 7f3f5b400640 1 -- 172.21.15.45:0/3726705422 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f3f4c08b7f0 con 0x7f3f6406c9d0 2024-04-16T16:14:42.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.969+0000 7f3f63400640 1 --2- 172.21.15.45:0/3726705422 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3f4006c8f0 0x7f3f4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:42.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.970+0000 7f3f63400640 1 --2- 172.21.15.45:0/3726705422 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3f4006c8f0 0x7f3f4006edb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3f54006fd0 tx=0x7f3f54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:42.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:42.973+0000 7f3f5b400640 1 -- 172.21.15.45:0/3726705422 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3f4c01f030 con 0x7f3f6406c9d0 2024-04-16T16:14:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.216+0000 7f3f6a200640 1 -- 172.21.15.45:0/3726705422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f6415e9f0 con 0x7f3f6406c9d0 2024-04-16T16:14:43.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.218+0000 7f3f5b400640 1 -- 172.21.15.45:0/3726705422 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7f3f4c0172c0 con 0x7f3f6406c9d0 2024-04-16T16:14:43.218 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:43.218 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/814837411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":814837411},{"type":"v1","addr":"172.21.15.103:6836","nonce":814837411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/401733314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":401733314},{"type":"v1","addr":"172.21.15.103:6837","nonce":401733314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5073,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/768816269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":768816269},{"type":"v1","addr":"172.21.15.45:6834","nonce":768816269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:14:39.132000+0000","modified":"2024-04-16T16:14:42.226029+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5079},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5079":{"gid":5079,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6835/2956262988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2956262988},{"type":"v1","addr":"172.21.15.45:6835","nonce":2956262988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5079,"qdb_cluster":[5079]},"id":2}]} 2024-04-16T16:14:43.218 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 16 2024-04-16T16:14:43.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.220+0000 7f3f6a200640 1 -- 172.21.15.45:0/3726705422 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3f4006c8f0 msgr2=0x7f3f4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:43.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.220+0000 7f3f6a200640 1 --2- 172.21.15.45:0/3726705422 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3f4006c8f0 0x7f3f4006edb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3f54006fd0 tx=0x7f3f54008040 comp rx=0 tx=0).stop 2024-04-16T16:14:43.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.220+0000 7f3f6a200640 1 -- 172.21.15.45:0/3726705422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6406c9d0 msgr2=0x7f3f6415d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:43.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.220+0000 7f3f6a200640 1 --2- 172.21.15.45:0/3726705422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6406c9d0 0x7f3f6415d730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00b3f0 tx=0x7f3f4c00b8b0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.220+0000 7f3f6a200640 1 -- 172.21.15.45:0/3726705422 shutdown_connections 2024-04-16T16:14:43.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.220+0000 7f3f6a200640 1 --2- 172.21.15.45:0/3726705422 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f3f4006c8f0 0x7f3f4006edb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.220+0000 7f3f6a200640 1 --2- 172.21.15.45:0/3726705422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f640b0630 0x7f3f6415dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.220+0000 7f3f6a200640 1 --2- 172.21.15.45:0/3726705422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6406c9d0 0x7f3f6415d730 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.220+0000 7f3f6a200640 1 --2- 172.21.15.45:0/3726705422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f64067c20 0x7f3f64159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.220+0000 7f3f6a200640 1 -- 172.21.15.45:0/3726705422 >> 172.21.15.45:0/3726705422 conn(0x7f3f6406e9b0 msgr2=0x7f3f641511a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:43.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.220+0000 7f3f6a200640 1 -- 172.21.15.45:0/3726705422 shutdown_connections 2024-04-16T16:14:43.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.220+0000 7f3f6a200640 1 -- 172.21.15.45:0/3726705422 wait complete. 2024-04-16T16:14:43.239 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:43.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.465+0000 7fe373e00640 1 Processor -- start 2024-04-16T16:14:43.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.465+0000 7fe373e00640 1 -- start start 2024-04-16T16:14:43.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.466+0000 7fe373e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe374070fa0 0x7fe374071400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:43.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.466+0000 7fe373e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe37415d340 0x7fe37415d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:43.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.466+0000 7fe373e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3741433d0 con 0x7fe374142ff0 2024-04-16T16:14:43.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.466+0000 7fe373e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe374071940 con 0x7fe37415d340 2024-04-16T16:14:43.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.466+0000 7fe373e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe3741605e0 con 0x7fe374070fa0 2024-04-16T16:14:43.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.467+0000 7fe372000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe374070fa0 0x7fe374071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:43.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.467+0000 7fe372000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe374070fa0 0x7fe374071400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:46688/0 (socket says 172.21.15.45:46688) 2024-04-16T16:14:43.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.467+0000 7fe372000640 1 -- 172.21.15.45:0/1732719287 learned_addr learned my addr 172.21.15.45:0/1732719287 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:43.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.468+0000 7fe373400640 1 --2- 172.21.15.45:0/1732719287 >> v2:172.21.15.103:3300/0 conn(0x7fe37415d340 0x7fe37415d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:43.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.468+0000 7fe372000640 1 -- 172.21.15.45:0/1732719287 >> v2:172.21.15.103:3300/0 conn(0x7fe37415d340 msgr2=0x7fe37415d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:43.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.468+0000 7fe372000640 1 --2- 172.21.15.45:0/1732719287 >> v2:172.21.15.103:3300/0 conn(0x7fe37415d340 0x7fe37415d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.468+0000 7fe372000640 1 -- 172.21.15.45:0/1732719287 >> v1:172.21.15.103:6789/0 conn(0x7fe374142ff0 legacy=0x7fe37406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:43.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.468+0000 7fe372000640 1 -- 172.21.15.45:0/1732719287 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3741608c0 con 0x7fe374070fa0 2024-04-16T16:14:43.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.468+0000 7fe373400640 1 --2- 172.21.15.45:0/1732719287 >> v2:172.21.15.103:3300/0 conn(0x7fe37415d340 0x7fe37415d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:43.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.469+0000 7fe372000640 1 --2- 172.21.15.45:0/1732719287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe374070fa0 0x7fe374071400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009870 tx=0x7fe35c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52fb292e82adcd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.470+0000 7fe371600640 1 -- 172.21.15.45:0/1732719287 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c034070 con 0x7fe374070fa0 2024-04-16T16:14:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.470+0000 7fe371600640 1 -- 172.21.15.45:0/1732719287 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe35c030840 con 0x7fe374070fa0 2024-04-16T16:14:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.470+0000 7fe371600640 1 -- 172.21.15.45:0/1732719287 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c030c10 con 0x7fe374070fa0 2024-04-16T16:14:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.470+0000 7fe373e00640 1 -- 172.21.15.45:0/1732719287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe374070fa0 msgr2=0x7fe374071400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.470+0000 7fe373e00640 1 --2- 172.21.15.45:0/1732719287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe374070fa0 0x7fe374071400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009870 tx=0x7fe35c02b980 comp rx=0 tx=0).stop 2024-04-16T16:14:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.470+0000 7fe373e00640 1 -- 172.21.15.45:0/1732719287 shutdown_connections 2024-04-16T16:14:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.470+0000 7fe373e00640 1 --2- 172.21.15.45:0/1732719287 >> v2:172.21.15.103:3300/0 conn(0x7fe37415d340 0x7fe37415d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.470+0000 7fe373e00640 1 --2- 172.21.15.45:0/1732719287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe374070fa0 0x7fe374071400 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.470+0000 7fe373e00640 1 -- 172.21.15.45:0/1732719287 >> 172.21.15.45:0/1732719287 conn(0x7fe37406f0e0 msgr2=0x7fe37406f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.471+0000 7fe373e00640 1 -- 172.21.15.45:0/1732719287 shutdown_connections 2024-04-16T16:14:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.471+0000 7fe373e00640 1 -- 172.21.15.45:0/1732719287 wait complete. 2024-04-16T16:14:43.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.472+0000 7fe373e00640 1 Processor -- start 2024-04-16T16:14:43.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.472+0000 7fe373e00640 1 -- start start 2024-04-16T16:14:43.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.473+0000 7fe373e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe374070fa0 0x7fe37412fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.473+0000 7fe373e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe374142ff0 0x7fe374159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.473+0000 7fe373e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe37415d340 0x7fe374157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.473+0000 7fe373e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe374161160 con 0x7fe374070fa0 2024-04-16T16:14:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.473+0000 7fe373e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe374160e80 con 0x7fe37415d340 2024-04-16T16:14:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.473+0000 7fe373e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe374161440 con 0x7fe374142ff0 2024-04-16T16:14:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.474+0000 7fe372a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe374070fa0 0x7fe37412fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.474+0000 7fe372a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe374070fa0 0x7fe37412fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50550/0 (socket says 172.21.15.45:50550) 2024-04-16T16:14:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.474+0000 7fe372a00640 1 -- 172.21.15.45:0/3850660266 learned_addr learned my addr 172.21.15.45:0/3850660266 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.474+0000 7fe372000640 1 --2- 172.21.15.45:0/3850660266 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe374142ff0 0x7fe374159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.474+0000 7fe372a00640 1 -- 172.21.15.45:0/3850660266 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe374142ff0 msgr2=0x7fe374159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.474+0000 7fe372a00640 1 --2- 172.21.15.45:0/3850660266 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe374142ff0 0x7fe374159950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.474+0000 7fe373400640 1 --2- 172.21.15.45:0/3850660266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe37415d340 0x7fe374157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:43.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.474+0000 7fe372a00640 1 -- 172.21.15.45:0/3850660266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe37415d340 msgr2=0x7fe374157ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:43.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.474+0000 7fe372a00640 1 --2- 172.21.15.45:0/3850660266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe37415d340 0x7fe374157ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.474+0000 7fe372a00640 1 -- 172.21.15.45:0/3850660266 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe374130140 con 0x7fe374070fa0 2024-04-16T16:14:43.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.475+0000 7fe372a00640 1 --2- 172.21.15.45:0/3850660266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe374070fa0 0x7fe37412fc00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe36400c7c0 tx=0x7fe36400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:43.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.475+0000 7fe36aa00640 1 -- 172.21.15.45:0/3850660266 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364016020 con 0x7fe374070fa0 2024-04-16T16:14:43.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.475+0000 7fe36aa00640 1 -- 172.21.15.45:0/3850660266 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe364006d50 con 0x7fe374070fa0 2024-04-16T16:14:43.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.475+0000 7fe36aa00640 1 -- 172.21.15.45:0/3850660266 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364003bb0 con 0x7fe374070fa0 2024-04-16T16:14:43.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.475+0000 7fe373e00640 1 -- 172.21.15.45:0/3850660266 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3741588b0 con 0x7fe374070fa0 2024-04-16T16:14:43.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.475+0000 7fe373e00640 1 -- 172.21.15.45:0/3850660266 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe374158e70 con 0x7fe374070fa0 2024-04-16T16:14:43.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.476+0000 7fe373e00640 1 -- 172.21.15.45:0/3850660266 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe374075780 con 0x7fe374070fa0 2024-04-16T16:14:43.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.477+0000 7fe36aa00640 1 -- 172.21.15.45:0/3850660266 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe36400a7f0 con 0x7fe374070fa0 2024-04-16T16:14:43.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.478+0000 7fe36aa00640 1 --2- 172.21.15.45:0/3850660266 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe35006c920 0x7fe35006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:43.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.478+0000 7fe36aa00640 1 -- 172.21.15.45:0/3850660266 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fe36408b880 con 0x7fe374070fa0 2024-04-16T16:14:43.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.479+0000 7fe372000640 1 --2- 172.21.15.45:0/3850660266 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe35006c920 0x7fe35006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:43.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.480+0000 7fe372000640 1 --2- 172.21.15.45:0/3850660266 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe35006c920 0x7fe35006ede0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe35c036000 tx=0x7fe35c03b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:43.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.483+0000 7fe36aa00640 1 -- 172.21.15.45:0/3850660266 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe364020020 con 0x7fe374070fa0 2024-04-16T16:14:43.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.687+0000 7fe373e00640 1 -- 172.21.15.45:0/3850660266 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe37406c500 con 0x7fe374070fa0 2024-04-16T16:14:43.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.688+0000 7fe36aa00640 1 -- 172.21.15.45:0/3850660266 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16669 (secure 0 0 0) 0x7fe364059ca0 con 0x7fe374070fa0 2024-04-16T16:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":34,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:14:40.675999+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:14:37.171798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-16T16:14:38.122496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:43.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.691+0000 7fe373e00640 1 -- 172.21.15.45:0/3850660266 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe35006c920 msgr2=0x7fe35006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:43.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.691+0000 7fe373e00640 1 --2- 172.21.15.45:0/3850660266 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe35006c920 0x7fe35006ede0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe35c036000 tx=0x7fe35c03b040 comp rx=0 tx=0).stop 2024-04-16T16:14:43.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.691+0000 7fe373e00640 1 -- 172.21.15.45:0/3850660266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe374070fa0 msgr2=0x7fe37412fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:43.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.691+0000 7fe373e00640 1 --2- 172.21.15.45:0/3850660266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe374070fa0 0x7fe37412fc00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe36400c7c0 tx=0x7fe36400cc80 comp rx=0 tx=0).stop 2024-04-16T16:14:43.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.691+0000 7fe373e00640 1 -- 172.21.15.45:0/3850660266 shutdown_connections 2024-04-16T16:14:43.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.691+0000 7fe373e00640 1 --2- 172.21.15.45:0/3850660266 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fe35006c920 0x7fe35006ede0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.691+0000 7fe373e00640 1 --2- 172.21.15.45:0/3850660266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe37415d340 0x7fe374157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.691+0000 7fe373e00640 1 --2- 172.21.15.45:0/3850660266 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe374142ff0 0x7fe374159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.691+0000 7fe373e00640 1 --2- 172.21.15.45:0/3850660266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe374070fa0 0x7fe37412fc00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.691+0000 7fe373e00640 1 -- 172.21.15.45:0/3850660266 >> 172.21.15.45:0/3850660266 conn(0x7fe37406f0e0 msgr2=0x7fe374144d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:43.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.691+0000 7fe373e00640 1 -- 172.21.15.45:0/3850660266 shutdown_connections 2024-04-16T16:14:43.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.691+0000 7fe373e00640 1 -- 172.21.15.45:0/3850660266 wait complete. 2024-04-16T16:14:43.712 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-16T16:14:43.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:14:43.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.927+0000 7fdccdc00640 1 Processor -- start 2024-04-16T16:14:43.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.927+0000 7fdccdc00640 1 -- start start 2024-04-16T16:14:43.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.928+0000 7fdccdc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdcc815b730 0x7fdcc80cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:43.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.929+0000 7fdccdc00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcc80ce5b0 con 0x7fdcc80d0230 2024-04-16T16:14:43.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.929+0000 7fdccdc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcc8072420 con 0x7fdcc80b5840 2024-04-16T16:14:43.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.929+0000 7fdccdc00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fdcc8072700 con 0x7fdcc815b730 2024-04-16T16:14:43.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.929+0000 7fdcc6a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdcc815b730 0x7fdcc80cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:43.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.929+0000 7fdcc6a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdcc815b730 0x7fdcc80cdeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50564/0 (socket says 172.21.15.45:50564) 2024-04-16T16:14:43.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.929+0000 7fdcc6a00640 1 -- 172.21.15.45:0/2308734126 learned_addr learned my addr 172.21.15.45:0/2308734126 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:43.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.929+0000 7fdcc6a00640 1 -- 172.21.15.45:0/2308734126 >> v1:172.21.15.103:6789/0 conn(0x7fdcc80b5840 legacy=0x7fdcc80cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:43.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.929+0000 7fdcc6a00640 1 -- 172.21.15.45:0/2308734126 >> v1:172.21.15.45:6789/0 conn(0x7fdcc80d0230 legacy=0x7fdcc80b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:43.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.929+0000 7fdcc6a00640 1 -- 172.21.15.45:0/2308734126 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc80729e0 con 0x7fdcc815b730 2024-04-16T16:14:43.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.931+0000 7fdcc6a00640 1 --2- 172.21.15.45:0/2308734126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc815b730 0x7fdcc80cdeb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009870 tx=0x7fdcb802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43399057638189b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:43.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.931+0000 7fdcc5600640 1 -- 172.21.15.45:0/2308734126 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8034070 con 0x7fdcc815b730 2024-04-16T16:14:43.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.931+0000 7fdcc5600640 1 -- 172.21.15.45:0/2308734126 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcb8030860 con 0x7fdcc815b730 2024-04-16T16:14:43.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.931+0000 7fdcc5600640 1 -- 172.21.15.45:0/2308734126 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8030c30 con 0x7fdcc815b730 2024-04-16T16:14:43.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.932+0000 7fdccdc00640 1 -- 172.21.15.45:0/2308734126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc815b730 msgr2=0x7fdcc80cdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:43.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.932+0000 7fdccdc00640 1 --2- 172.21.15.45:0/2308734126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc815b730 0x7fdcc80cdeb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009870 tx=0x7fdcb802b9a0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.932+0000 7fdccdc00640 1 -- 172.21.15.45:0/2308734126 shutdown_connections 2024-04-16T16:14:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.932+0000 7fdccdc00640 1 --2- 172.21.15.45:0/2308734126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc815b730 0x7fdcc80cdeb0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.932+0000 7fdccdc00640 1 -- 172.21.15.45:0/2308734126 >> 172.21.15.45:0/2308734126 conn(0x7fdcc80b2750 msgr2=0x7fdcc80b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.932+0000 7fdccdc00640 1 -- 172.21.15.45:0/2308734126 shutdown_connections 2024-04-16T16:14:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.932+0000 7fdccdc00640 1 -- 172.21.15.45:0/2308734126 wait complete. 2024-04-16T16:14:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.933+0000 7fdccdc00640 1 Processor -- start 2024-04-16T16:14:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.934+0000 7fdccdc00640 1 -- start start 2024-04-16T16:14:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.934+0000 7fdccdc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdcc80b5840 0x7fdcc81f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.935+0000 7fdccdc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc80d0230 0x7fdcc813c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.935+0000 7fdccdc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcc815b730 0x7fdcc806e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.935+0000 7fdccdc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc815bb10 con 0x7fdcc80d0230 2024-04-16T16:14:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.935+0000 7fdccdc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc80b5ca0 con 0x7fdcc815b730 2024-04-16T16:14:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.935+0000 7fdccdc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdcc80731e0 con 0x7fdcc80b5840 2024-04-16T16:14:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.935+0000 7fdcc6000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc80d0230 0x7fdcc813c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.935+0000 7fdcc6000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc80d0230 0x7fdcc813c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50574/0 (socket says 172.21.15.45:50574) 2024-04-16T16:14:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.935+0000 7fdcc6000640 1 -- 172.21.15.45:0/1299799937 learned_addr learned my addr 172.21.15.45:0/1299799937 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.935+0000 7fdcc6a00640 1 --2- 172.21.15.45:0/1299799937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdcc80b5840 0x7fdcc81f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.935+0000 7fdcc6000640 1 -- 172.21.15.45:0/1299799937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdcc80b5840 msgr2=0x7fdcc81f34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.935+0000 7fdcc6000640 1 --2- 172.21.15.45:0/1299799937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdcc80b5840 0x7fdcc81f34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.935+0000 7fdcc6000640 1 -- 172.21.15.45:0/1299799937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcc815b730 msgr2=0x7fdcc806e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.935+0000 7fdcc6000640 1 --2- 172.21.15.45:0/1299799937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcc815b730 0x7fdcc806e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.935+0000 7fdcc6000640 1 -- 172.21.15.45:0/1299799937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc81f6dc0 con 0x7fdcc80d0230 2024-04-16T16:14:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.936+0000 7fdcc6000640 1 --2- 172.21.15.45:0/1299799937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc80d0230 0x7fdcc813c240 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400b3e0 tx=0x7fdcb400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.936+0000 7fdcbea00640 1 -- 172.21.15.45:0/1299799937 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4015020 con 0x7fdcc80d0230 2024-04-16T16:14:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.936+0000 7fdcbea00640 1 -- 172.21.15.45:0/1299799937 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcb4011850 con 0x7fdcc80d0230 2024-04-16T16:14:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.936+0000 7fdcbea00640 1 -- 172.21.15.45:0/1299799937 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4011c40 con 0x7fdcc80d0230 2024-04-16T16:14:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.936+0000 7fdccdc00640 1 -- 172.21.15.45:0/1299799937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcc81f6ae0 con 0x7fdcc80d0230 2024-04-16T16:14:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.936+0000 7fdccdc00640 1 -- 172.21.15.45:0/1299799937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcc81f2cd0 con 0x7fdcc80d0230 2024-04-16T16:14:43.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.937+0000 7fdccdc00640 1 -- 172.21.15.45:0/1299799937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcc8075780 con 0x7fdcc80d0230 2024-04-16T16:14:43.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.938+0000 7fdcbea00640 1 -- 172.21.15.45:0/1299799937 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdcb4018020 con 0x7fdcc80d0230 2024-04-16T16:14:43.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.939+0000 7fdcbea00640 1 --2- 172.21.15.45:0/1299799937 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdca806c8f0 0x7fdca806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:43.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.939+0000 7fdcbea00640 1 -- 172.21.15.45:0/1299799937 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fdcb408bb30 con 0x7fdcc80d0230 2024-04-16T16:14:43.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.940+0000 7fdcc6a00640 1 --2- 172.21.15.45:0/1299799937 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdca806c8f0 0x7fdca806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:43.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.941+0000 7fdcc6a00640 1 --2- 172.21.15.45:0/1299799937 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdca806c8f0 0x7fdca806edb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8036000 tx=0x7fdcb803b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:43.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:43.944+0000 7fdcbea00640 1 -- 172.21.15.45:0/1299799937 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdcb401f020 con 0x7fdcc80d0230 2024-04-16T16:14:44.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.146+0000 7fdccdc00640 1 -- 172.21.15.45:0/1299799937 --> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdcc80b3d00 con 0x7fdca806c8f0 2024-04-16T16:14:44.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.169+0000 7fdcbea00640 1 -- 172.21.15.45:0/1299799937 <== mgr.4105 v2:172.21.15.103:6832/2711977587 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+239854 (secure 0 0 0) 0x7fdcc80b3d00 con 0x7fdca806c8f0 2024-04-16T16:14:44.172 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:44.173 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:14:44.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.175+0000 7fdccdc00640 1 -- 172.21.15.45:0/1299799937 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdca806c8f0 msgr2=0x7fdca806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:44.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.175+0000 7fdccdc00640 1 --2- 172.21.15.45:0/1299799937 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdca806c8f0 0x7fdca806edb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8036000 tx=0x7fdcb803b040 comp rx=0 tx=0).stop 2024-04-16T16:14:44.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.175+0000 7fdccdc00640 1 -- 172.21.15.45:0/1299799937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc80d0230 msgr2=0x7fdcc813c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:44.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.175+0000 7fdccdc00640 1 --2- 172.21.15.45:0/1299799937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc80d0230 0x7fdcc813c240 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400b3e0 tx=0x7fdcb400b8a0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.175+0000 7fdccdc00640 1 -- 172.21.15.45:0/1299799937 shutdown_connections 2024-04-16T16:14:44.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.175+0000 7fdccdc00640 1 --2- 172.21.15.45:0/1299799937 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fdca806c8f0 0x7fdca806edb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.175+0000 7fdccdc00640 1 --2- 172.21.15.45:0/1299799937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcc815b730 0x7fdcc806e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.175+0000 7fdccdc00640 1 --2- 172.21.15.45:0/1299799937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc80d0230 0x7fdcc813c240 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.175+0000 7fdccdc00640 1 --2- 172.21.15.45:0/1299799937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdcc80b5840 0x7fdcc81f34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.175+0000 7fdccdc00640 1 -- 172.21.15.45:0/1299799937 >> 172.21.15.45:0/1299799937 conn(0x7fdcc80b2750 msgr2=0x7fdcc815dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:44.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.175+0000 7fdccdc00640 1 -- 172.21.15.45:0/1299799937 shutdown_connections 2024-04-16T16:14:44.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.175+0000 7fdccdc00640 1 -- 172.21.15.45:0/1299799937 wait complete. 2024-04-16T16:14:44.196 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2024-04-16T16:14:43.118956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236676,"kb_used_data":3868,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738044,"statfs":{"total":773094113280,"available":772851757056,"internally_reserved":0,"allocated":3960832,"data_stored":2230415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.491583"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.645616+0000","last_change":"2024-04-16T16:14:37.645616+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.645616+0000","last_undegraded":"2024-04-16T16:14:37.645616+0000","last_fullsized":"2024-04-16T16:14:37.645616+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.633469+0000","last_change":"2024-04-16T16:14:37.633469+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.633469+0000","last_undegraded":"2024-04-16T16:14:37.633469+0000","last_fullsized":"2024-04-16T16:14:37.633469+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.637282+0000","last_change":"2024-04-16T16:14:37.637282+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.637282+0000","last_undegraded":"2024-04-16T16:14:37.637282+0000","last_fullsized":"2024-04-16T16:14:37.637282+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.643797+0000","last_change":"2024-04-16T16:14:37.643797+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.643797+0000","last_undegraded":"2024-04-16T16:14:37.643797+0000","last_fullsized":"2024-04-16T16:14:37.643797+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.634649+0000","last_change":"2024-04-16T16:14:37.634649+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.634649+0000","last_undegraded":"2024-04-16T16:14:37.634649+0000","last_fullsized":"2024-04-16T16:14:37.634649+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.633362+0000","last_change":"2024-04-16T16:14:37.633362+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.633362+0000","last_undegraded":"2024-04-16T16:14:37.633362+0000","last_fullsized":"2024-04-16T16:14:37.633362+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.638690+0000","last_change":"2024-04-16T16:14:37.638690+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.638690+0000","last_undegraded":"2024-04-16T16:14:37.638690+0000","last_fullsized":"2024-04-16T16:14:37.638690+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.633304+0000","last_change":"2024-04-16T16:14:37.633304+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.633304+0000","last_undegraded":"2024-04-16T16:14:37.633304+0000","last_fullsized":"2024-04-16T16:14:37.633304+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.639464+0000","last_change":"2024-04-16T16:14:37.639464+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.639464+0000","last_undegraded":"2024-04-16T16:14:37.639464+0000","last_fullsized":"2024-04-16T16:14:37.639464+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.632614+0000","last_change":"2024-04-16T16:14:37.632614+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.632614+0000","last_undegraded":"2024-04-16T16:14:37.632614+0000","last_fullsized":"2024-04-16T16:14:37.632614+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.640466+0000","last_change":"2024-04-16T16:14:37.640466+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.640466+0000","last_undegraded":"2024-04-16T16:14:37.640466+0000","last_fullsized":"2024-04-16T16:14:37.640466+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.636158+0000","last_change":"2024-04-16T16:14:37.636158+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.636158+0000","last_undegraded":"2024-04-16T16:14:37.636158+0000","last_fullsized":"2024-04-16T16:14:37.636158+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.633547+0000","last_change":"2024-04-16T16:14:37.633547+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.633547+0000","last_undegraded":"2024-04-16T16:14:37.633547+0000","last_fullsized":"2024-04-16T16:14:37.633547+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.641032+0000","last_change":"2024-04-16T16:14:37.641032+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.641032+0000","last_undegraded":"2024-04-16T16:14:37.641032+0000","last_fullsized":"2024-04-16T16:14:37.641032+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.638018+0000","last_change":"2024-04-16T16:14:37.638018+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.638018+0000","last_undegraded":"2024-04-16T16:14:37.638018+0000","last_fullsized":"2024-04-16T16:14:37.638018+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.639994+0000","last_change":"2024-04-16T16:14:37.639994+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.639994+0000","last_undegraded":"2024-04-16T16:14:37.639994+0000","last_fullsized":"2024-04-16T16:14:37.639994+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.636734+0000","last_change":"2024-04-16T16:14:37.636734+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.636734+0000","last_undegraded":"2024-04-16T16:14:37.636734+0000","last_fullsized":"2024-04-16T16:14:37.636734+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.636051+0000","last_change":"2024-04-16T16:14:37.636051+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.636051+0000","last_undegraded":"2024-04-16T16:14:37.636051+0000","last_fullsized":"2024-04-16T16:14:37.636051+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.633046+0000","last_change":"2024-04-16T16:14:37.633046+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.633046+0000","last_undegraded":"2024-04-16T16:14:37.633046+0000","last_fullsized":"2024-04-16T16:14:37.633046+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.634311+0000","last_change":"2024-04-16T16:14:37.634311+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.634311+0000","last_undegraded":"2024-04-16T16:14:37.634311+0000","last_fullsized":"2024-04-16T16:14:37.634311+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.632413+0000","last_change":"2024-04-16T16:14:37.632413+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.632413+0000","last_undegraded":"2024-04-16T16:14:37.632413+0000","last_fullsized":"2024-04-16T16:14:37.632413+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.642262+0000","last_change":"2024-04-16T16:14:37.642262+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.642262+0000","last_undegraded":"2024-04-16T16:14:37.642262+0000","last_fullsized":"2024-04-16T16:14:37.642262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.637155+0000","last_change":"2024-04-16T16:14:37.637155+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.637155+0000","last_undegraded":"2024-04-16T16:14:37.637155+0000","last_fullsized":"2024-04-16T16:14:37.637155+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.640150+0000","last_change":"2024-04-16T16:14:37.640150+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.640150+0000","last_undegraded":"2024-04-16T16:14:37.640150+0000","last_fullsized":"2024-04-16T16:14:37.640150+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.636025+0000","last_change":"2024-04-16T16:14:37.636025+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.636025+0000","last_undegraded":"2024-04-16T16:14:37.636025+0000","last_fullsized":"2024-04-16T16:14:37.636025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.631388+0000","last_change":"2024-04-16T16:14:37.631388+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.631388+0000","last_undegraded":"2024-04-16T16:14:37.631388+0000","last_fullsized":"2024-04-16T16:14:37.631388+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.638734+0000","last_change":"2024-04-16T16:14:37.638734+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.638734+0000","last_undegraded":"2024-04-16T16:14:37.638734+0000","last_fullsized":"2024-04-16T16:14:37.638734+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.631393+0000","last_change":"2024-04-16T16:14:37.631393+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.631393+0000","last_undegraded":"2024-04-16T16:14:37.631393+0000","last_fullsized":"2024-04-16T16:14:37.631393+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.639185+0000","last_change":"2024-04-16T16:14:37.639185+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.639185+0000","last_undegraded":"2024-04-16T16:14:37.639185+0000","last_fullsized":"2024-04-16T16:14:37.639185+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.632956+0000","last_change":"2024-04-16T16:14:37.632956+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.632956+0000","last_undegraded":"2024-04-16T16:14:37.632956+0000","last_fullsized":"2024-04-16T16:14:37.632956+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.635967+0000","last_change":"2024-04-16T16:14:37.635967+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.635967+0000","last_undegraded":"2024-04-16T16:14:37.635967+0000","last_fullsized":"2024-04-16T16:14:37.635967+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.644579+0000","last_change":"2024-04-16T16:14:37.644579+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.644579+0000","last_undegraded":"2024-04-16T16:14:37.644579+0000","last_fullsized":"2024-04-16T16:14:37.644579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":190,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-16T16:14:37.631699+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:14:37.631699+0000","last_peered":"2024-04-16T16:14:37.631699+0000","last_clean":"2024-04-16T16:14:37.631699+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:14:37.631699+0000","last_undegraded":"2024-04-16T16:14:37.631699+0000","last_fullsized":"2024-04-16T16:14:37.631699+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":722305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":131937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":131937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":131937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":131937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":131937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":131937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":929792,"data_stored":716488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:14:44.200 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:14:44.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.464+0000 7fda41800640 1 Processor -- start 2024-04-16T16:14:44.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.464+0000 7fda41800640 1 -- start start 2024-04-16T16:14:44.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.465+0000 7fda41800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fda3c161e90 0x7fda3c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:44.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.465+0000 7fda41800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fda3c162270 0x7fda3c0d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:44.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.465+0000 7fda41800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda3c0d52d0 0x7fda3c06ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:44.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.465+0000 7fda41800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fda3c0d2e30 con 0x7fda3c162270 2024-04-16T16:14:44.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.465+0000 7fda41800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fda3c0d3080 con 0x7fda3c161e90 2024-04-16T16:14:44.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.465+0000 7fda41800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda3c0d3360 con 0x7fda3c0d52d0 2024-04-16T16:14:44.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.466+0000 7fda3b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fda3c162270 0x7fda3c0d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:44.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.466+0000 7fda3b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fda3c162270 0x7fda3c0d3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50590/0 (socket says 172.21.15.45:50590) 2024-04-16T16:14:44.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.466+0000 7fda3b400640 1 -- 172.21.15.45:0/3195863919 learned_addr learned my addr 172.21.15.45:0/3195863919 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:44.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.466+0000 7fda3be00640 1 --2- 172.21.15.45:0/3195863919 >> v2:172.21.15.103:3300/0 conn(0x7fda3c161e90 0x7fda3c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:44.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.466+0000 7fda3b400640 1 -- 172.21.15.45:0/3195863919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda3c0d52d0 msgr2=0x7fda3c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:44.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.466+0000 7fda40e00640 1 --2- 172.21.15.45:0/3195863919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda3c0d52d0 0x7fda3c06ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:44.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.466+0000 7fda3b400640 1 --2- 172.21.15.45:0/3195863919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda3c0d52d0 0x7fda3c06ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.466+0000 7fda3b400640 1 -- 172.21.15.45:0/3195863919 >> v2:172.21.15.103:3300/0 conn(0x7fda3c161e90 msgr2=0x7fda3c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:44.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.466+0000 7fda3b400640 1 --2- 172.21.15.45:0/3195863919 >> v2:172.21.15.103:3300/0 conn(0x7fda3c161e90 0x7fda3c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.466+0000 7fda3b400640 1 -- 172.21.15.45:0/3195863919 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3c0d3640 con 0x7fda3c162270 2024-04-16T16:14:44.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.466+0000 7fda40e00640 1 --2- 172.21.15.45:0/3195863919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda3c0d52d0 0x7fda3c06ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:44.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.467+0000 7fda3b400640 1 --2- 172.21.15.45:0/3195863919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c162270 0x7fda3c0d3e40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fda28009870 tx=0x7fda2802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6a08bca32051fe server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:44.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.468+0000 7fda3aa00640 1 -- 172.21.15.45:0/3195863919 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda28034070 con 0x7fda3c162270 2024-04-16T16:14:44.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.468+0000 7fda3aa00640 1 -- 172.21.15.45:0/3195863919 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda28030850 con 0x7fda3c162270 2024-04-16T16:14:44.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.468+0000 7fda3aa00640 1 -- 172.21.15.45:0/3195863919 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda28030c20 con 0x7fda3c162270 2024-04-16T16:14:44.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.468+0000 7fda41800640 1 -- 172.21.15.45:0/3195863919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c162270 msgr2=0x7fda3c0d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:44.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.468+0000 7fda41800640 1 --2- 172.21.15.45:0/3195863919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c162270 0x7fda3c0d3e40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fda28009870 tx=0x7fda2802b990 comp rx=0 tx=0).stop 2024-04-16T16:14:44.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.469+0000 7fda41800640 1 -- 172.21.15.45:0/3195863919 shutdown_connections 2024-04-16T16:14:44.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.469+0000 7fda41800640 1 --2- 172.21.15.45:0/3195863919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda3c0d52d0 0x7fda3c06ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.469+0000 7fda41800640 1 --2- 172.21.15.45:0/3195863919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c162270 0x7fda3c0d3e40 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.469+0000 7fda41800640 1 --2- 172.21.15.45:0/3195863919 >> v2:172.21.15.103:3300/0 conn(0x7fda3c161e90 0x7fda3c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.469+0000 7fda41800640 1 -- 172.21.15.45:0/3195863919 >> 172.21.15.45:0/3195863919 conn(0x7fda3c0d1940 msgr2=0x7fda3c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:44.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.469+0000 7fda41800640 1 -- 172.21.15.45:0/3195863919 shutdown_connections 2024-04-16T16:14:44.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.469+0000 7fda41800640 1 -- 172.21.15.45:0/3195863919 wait complete. 2024-04-16T16:14:44.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.470+0000 7fda41800640 1 Processor -- start 2024-04-16T16:14:44.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.470+0000 7fda41800640 1 -- start start 2024-04-16T16:14:44.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.471+0000 7fda41800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c0d52d0 0x7fda3c151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:44.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.471+0000 7fda41800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda3c161e90 0x7fda3c0cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:44.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.471+0000 7fda41800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda3c162270 0x7fda3c0b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:44.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.471+0000 7fda41800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c0d4380 con 0x7fda3c0d52d0 2024-04-16T16:14:44.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.471+0000 7fda41800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c0cfd10 con 0x7fda3c162270 2024-04-16T16:14:44.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.471+0000 7fda41800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda3c0d4500 con 0x7fda3c161e90 2024-04-16T16:14:44.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.472+0000 7fda3be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c0d52d0 0x7fda3c151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:44.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.472+0000 7fda3be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c0d52d0 0x7fda3c151c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50592/0 (socket says 172.21.15.45:50592) 2024-04-16T16:14:44.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.472+0000 7fda3be00640 1 -- 172.21.15.45:0/1398920992 learned_addr learned my addr 172.21.15.45:0/1398920992 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:44.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.472+0000 7fda3b400640 1 --2- 172.21.15.45:0/1398920992 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda3c161e90 0x7fda3c0cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:44.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.472+0000 7fda40e00640 1 --2- 172.21.15.45:0/1398920992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda3c162270 0x7fda3c0b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:44.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.472+0000 7fda3be00640 1 -- 172.21.15.45:0/1398920992 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda3c161e90 msgr2=0x7fda3c0cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:44.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.472+0000 7fda3be00640 1 --2- 172.21.15.45:0/1398920992 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda3c161e90 0x7fda3c0cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.472+0000 7fda3be00640 1 -- 172.21.15.45:0/1398920992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda3c162270 msgr2=0x7fda3c0b53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:44.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.472+0000 7fda3be00640 1 --2- 172.21.15.45:0/1398920992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda3c162270 0x7fda3c0b53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.472+0000 7fda3be00640 1 -- 172.21.15.45:0/1398920992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3c0cda60 con 0x7fda3c0d52d0 2024-04-16T16:14:44.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.472+0000 7fda3b400640 1 --2- 172.21.15.45:0/1398920992 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda3c161e90 0x7fda3c0cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:44.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.472+0000 7fda3be00640 1 --2- 172.21.15.45:0/1398920992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c0d52d0 0x7fda3c151c90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fda2c00d830 tx=0x7fda2c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:44.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.473+0000 7fda33e00640 1 -- 172.21.15.45:0/1398920992 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2c00b6c0 con 0x7fda3c0d52d0 2024-04-16T16:14:44.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.473+0000 7fda33e00640 1 -- 172.21.15.45:0/1398920992 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda2c019070 con 0x7fda3c0d52d0 2024-04-16T16:14:44.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.473+0000 7fda33e00640 1 -- 172.21.15.45:0/1398920992 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2c00ba80 con 0x7fda3c0d52d0 2024-04-16T16:14:44.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.473+0000 7fda41800640 1 -- 172.21.15.45:0/1398920992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda3c0cdd40 con 0x7fda3c0d52d0 2024-04-16T16:14:44.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.473+0000 7fda41800640 1 -- 172.21.15.45:0/1398920992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda3c0ce300 con 0x7fda3c0d52d0 2024-04-16T16:14:44.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.474+0000 7fda41800640 1 -- 172.21.15.45:0/1398920992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda3c1600d0 con 0x7fda3c0d52d0 2024-04-16T16:14:44.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.475+0000 7fda33e00640 1 -- 172.21.15.45:0/1398920992 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fda2c01e460 con 0x7fda3c0d52d0 2024-04-16T16:14:44.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.476+0000 7fda33e00640 1 --2- 172.21.15.45:0/1398920992 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fda1c06c8f0 0x7fda1c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:44.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.476+0000 7fda33e00640 1 -- 172.21.15.45:0/1398920992 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fda2c08c2d0 con 0x7fda3c0d52d0 2024-04-16T16:14:44.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.477+0000 7fda3b400640 1 --2- 172.21.15.45:0/1398920992 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fda1c06c8f0 0x7fda1c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:44.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.478+0000 7fda3b400640 1 --2- 172.21.15.45:0/1398920992 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fda1c06c8f0 0x7fda1c06edb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fda28036000 tx=0x7fda2803b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:44.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.480+0000 7fda33e00640 1 -- 172.21.15.45:0/1398920992 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda3c1600d0 con 0x7fda3c0d52d0 2024-04-16T16:14:44.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.679+0000 7fda41800640 1 -- 172.21.15.45:0/1398920992 --> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fda3c06dae0 con 0x7fda1c06c8f0 2024-04-16T16:14:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.696+0000 7fda33e00640 1 -- 172.21.15.45:0/1398920992 <== mgr.4105 v2:172.21.15.103:6832/2711977587 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+239854 (secure 0 0 0) 0x7fda3c06dae0 con 0x7fda1c06c8f0 2024-04-16T16:14:44.698 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:14:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.700+0000 7fda41800640 1 -- 172.21.15.45:0/1398920992 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fda1c06c8f0 msgr2=0x7fda1c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.700+0000 7fda41800640 1 --2- 172.21.15.45:0/1398920992 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fda1c06c8f0 0x7fda1c06edb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fda28036000 tx=0x7fda2803b040 comp rx=0 tx=0).stop 2024-04-16T16:14:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.700+0000 7fda41800640 1 -- 172.21.15.45:0/1398920992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c0d52d0 msgr2=0x7fda3c151c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.700+0000 7fda41800640 1 --2- 172.21.15.45:0/1398920992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c0d52d0 0x7fda3c151c90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fda2c00d830 tx=0x7fda2c00dcf0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.701+0000 7fda41800640 1 -- 172.21.15.45:0/1398920992 shutdown_connections 2024-04-16T16:14:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.701+0000 7fda41800640 1 --2- 172.21.15.45:0/1398920992 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fda1c06c8f0 0x7fda1c06edb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.701+0000 7fda41800640 1 --2- 172.21.15.45:0/1398920992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda3c162270 0x7fda3c0b53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.701+0000 7fda41800640 1 --2- 172.21.15.45:0/1398920992 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda3c161e90 0x7fda3c0cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.701+0000 7fda41800640 1 --2- 172.21.15.45:0/1398920992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c0d52d0 0x7fda3c151c90 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.701+0000 7fda41800640 1 -- 172.21.15.45:0/1398920992 >> 172.21.15.45:0/1398920992 conn(0x7fda3c0d1940 msgr2=0x7fda3c0b79f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.701+0000 7fda41800640 1 -- 172.21.15.45:0/1398920992 shutdown_connections 2024-04-16T16:14:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.701+0000 7fda41800640 1 -- 172.21.15.45:0/1398920992 wait complete. 2024-04-16T16:14:44.720 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2024-04-16T16:14:43.118956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236676,"kb_used_data":3868,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738044,"statfs":{"total":773094113280,"available":772851757056,"internally_reserved":0,"allocated":3960832,"data_stored":2230415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.491583"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.645616+0000","last_change":"2024-04-16T16:14:37.645616+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.645616+0000","last_undegraded":"2024-04-16T16:14:37.645616+0000","last_fullsized":"2024-04-16T16:14:37.645616+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.633469+0000","last_change":"2024-04-16T16:14:37.633469+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.633469+0000","last_undegraded":"2024-04-16T16:14:37.633469+0000","last_fullsized":"2024-04-16T16:14:37.633469+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.637282+0000","last_change":"2024-04-16T16:14:37.637282+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.637282+0000","last_undegraded":"2024-04-16T16:14:37.637282+0000","last_fullsized":"2024-04-16T16:14:37.637282+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.643797+0000","last_change":"2024-04-16T16:14:37.643797+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.643797+0000","last_undegraded":"2024-04-16T16:14:37.643797+0000","last_fullsized":"2024-04-16T16:14:37.643797+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.634649+0000","last_change":"2024-04-16T16:14:37.634649+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.634649+0000","last_undegraded":"2024-04-16T16:14:37.634649+0000","last_fullsized":"2024-04-16T16:14:37.634649+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.633362+0000","last_change":"2024-04-16T16:14:37.633362+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.633362+0000","last_undegraded":"2024-04-16T16:14:37.633362+0000","last_fullsized":"2024-04-16T16:14:37.633362+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.638690+0000","last_change":"2024-04-16T16:14:37.638690+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.638690+0000","last_undegraded":"2024-04-16T16:14:37.638690+0000","last_fullsized":"2024-04-16T16:14:37.638690+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.633304+0000","last_change":"2024-04-16T16:14:37.633304+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.633304+0000","last_undegraded":"2024-04-16T16:14:37.633304+0000","last_fullsized":"2024-04-16T16:14:37.633304+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.639464+0000","last_change":"2024-04-16T16:14:37.639464+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.639464+0000","last_undegraded":"2024-04-16T16:14:37.639464+0000","last_fullsized":"2024-04-16T16:14:37.639464+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.632614+0000","last_change":"2024-04-16T16:14:37.632614+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.632614+0000","last_undegraded":"2024-04-16T16:14:37.632614+0000","last_fullsized":"2024-04-16T16:14:37.632614+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.640466+0000","last_change":"2024-04-16T16:14:37.640466+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.640466+0000","last_undegraded":"2024-04-16T16:14:37.640466+0000","last_fullsized":"2024-04-16T16:14:37.640466+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.636158+0000","last_change":"2024-04-16T16:14:37.636158+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.636158+0000","last_undegraded":"2024-04-16T16:14:37.636158+0000","last_fullsized":"2024-04-16T16:14:37.636158+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.633547+0000","last_change":"2024-04-16T16:14:37.633547+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.633547+0000","last_undegraded":"2024-04-16T16:14:37.633547+0000","last_fullsized":"2024-04-16T16:14:37.633547+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.641032+0000","last_change":"2024-04-16T16:14:37.641032+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.641032+0000","last_undegraded":"2024-04-16T16:14:37.641032+0000","last_fullsized":"2024-04-16T16:14:37.641032+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.638018+0000","last_change":"2024-04-16T16:14:37.638018+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.638018+0000","last_undegraded":"2024-04-16T16:14:37.638018+0000","last_fullsized":"2024-04-16T16:14:37.638018+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.639994+0000","last_change":"2024-04-16T16:14:37.639994+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.639994+0000","last_undegraded":"2024-04-16T16:14:37.639994+0000","last_fullsized":"2024-04-16T16:14:37.639994+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.636734+0000","last_change":"2024-04-16T16:14:37.636734+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.636734+0000","last_undegraded":"2024-04-16T16:14:37.636734+0000","last_fullsized":"2024-04-16T16:14:37.636734+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.636051+0000","last_change":"2024-04-16T16:14:37.636051+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.636051+0000","last_undegraded":"2024-04-16T16:14:37.636051+0000","last_fullsized":"2024-04-16T16:14:37.636051+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.633046+0000","last_change":"2024-04-16T16:14:37.633046+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.633046+0000","last_undegraded":"2024-04-16T16:14:37.633046+0000","last_fullsized":"2024-04-16T16:14:37.633046+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.634311+0000","last_change":"2024-04-16T16:14:37.634311+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.634311+0000","last_undegraded":"2024-04-16T16:14:37.634311+0000","last_fullsized":"2024-04-16T16:14:37.634311+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.632413+0000","last_change":"2024-04-16T16:14:37.632413+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.632413+0000","last_undegraded":"2024-04-16T16:14:37.632413+0000","last_fullsized":"2024-04-16T16:14:37.632413+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.642262+0000","last_change":"2024-04-16T16:14:37.642262+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.642262+0000","last_undegraded":"2024-04-16T16:14:37.642262+0000","last_fullsized":"2024-04-16T16:14:37.642262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.637155+0000","last_change":"2024-04-16T16:14:37.637155+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.637155+0000","last_undegraded":"2024-04-16T16:14:37.637155+0000","last_fullsized":"2024-04-16T16:14:37.637155+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.640150+0000","last_change":"2024-04-16T16:14:37.640150+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.640150+0000","last_undegraded":"2024-04-16T16:14:37.640150+0000","last_fullsized":"2024-04-16T16:14:37.640150+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.636025+0000","last_change":"2024-04-16T16:14:37.636025+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.636025+0000","last_undegraded":"2024-04-16T16:14:37.636025+0000","last_fullsized":"2024-04-16T16:14:37.636025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.631388+0000","last_change":"2024-04-16T16:14:37.631388+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.631388+0000","last_undegraded":"2024-04-16T16:14:37.631388+0000","last_fullsized":"2024-04-16T16:14:37.631388+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.638734+0000","last_change":"2024-04-16T16:14:37.638734+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.638734+0000","last_undegraded":"2024-04-16T16:14:37.638734+0000","last_fullsized":"2024-04-16T16:14:37.638734+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.631393+0000","last_change":"2024-04-16T16:14:37.631393+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.631393+0000","last_undegraded":"2024-04-16T16:14:37.631393+0000","last_fullsized":"2024-04-16T16:14:37.631393+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.639185+0000","last_change":"2024-04-16T16:14:37.639185+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.639185+0000","last_undegraded":"2024-04-16T16:14:37.639185+0000","last_fullsized":"2024-04-16T16:14:37.639185+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.632956+0000","last_change":"2024-04-16T16:14:37.632956+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.632956+0000","last_undegraded":"2024-04-16T16:14:37.632956+0000","last_fullsized":"2024-04-16T16:14:37.632956+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.635967+0000","last_change":"2024-04-16T16:14:37.635967+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.635967+0000","last_undegraded":"2024-04-16T16:14:37.635967+0000","last_fullsized":"2024-04-16T16:14:37.635967+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:14:38.624558+0000","last_change":"2024-04-16T16:14:38.624558+0000","last_active":"2024-04-16T16:14:38.624558+0000","last_peered":"2024-04-16T16:14:38.624558+0000","last_clean":"2024-04-16T16:14:38.624558+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:38.624558+0000","last_undegraded":"2024-04-16T16:14:38.624558+0000","last_fullsized":"2024-04-16T16:14:38.624558+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-16T16:14:37.644579+0000","last_change":"2024-04-16T16:14:37.644579+0000","last_active":"2024-04-16T16:14:37.612306+0000","last_peered":"2024-04-16T16:14:37.612306+0000","last_clean":"2024-04-16T16:14:37.612306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:14:37.644579+0000","last_undegraded":"2024-04-16T16:14:37.644579+0000","last_fullsized":"2024-04-16T16:14:37.644579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":190,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-16T16:14:37.631699+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:14:37.631699+0000","last_peered":"2024-04-16T16:14:37.631699+0000","last_clean":"2024-04-16T16:14:37.631699+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:14:37.631699+0000","last_undegraded":"2024-04-16T16:14:37.631699+0000","last_fullsized":"2024-04-16T16:14:37.631699+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":722305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":131937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":131937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":131937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":131937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":131937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":131937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":929792,"data_stored":716488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:14:44.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:14:44.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.990+0000 7fc30b200640 1 Processor -- start 2024-04-16T16:14:44.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.990+0000 7fc30b200640 1 -- start start 2024-04-16T16:14:44.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.991+0000 7fc30b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc30c15ccb0 0x7fc30c06df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:44.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.991+0000 7fc30b200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc30c06e480 0x7fc30c1608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:44.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.991+0000 7fc30b200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc30c15d360 con 0x7fc30c146220 2024-04-16T16:14:44.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.991+0000 7fc30b200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc30c15d130 con 0x7fc30c06e480 2024-04-16T16:14:44.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.991+0000 7fc30b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc30c160410 con 0x7fc30c15ccb0 2024-04-16T16:14:44.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.992+0000 7fc30a800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc30c06e480 0x7fc30c1608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:44.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.992+0000 7fc309e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc30c146220 0x7fc30c06f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42356/0 (socket says 172.21.15.45:42356) 2024-04-16T16:14:44.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.992+0000 7fc309e00640 1 -- 172.21.15.45:0/1628400161 learned_addr learned my addr 172.21.15.45:0/1628400161 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:44.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.992+0000 7fc30a800640 1 -- 172.21.15.45:0/1628400161 >> v2:172.21.15.103:3300/0 conn(0x7fc30c15ccb0 msgr2=0x7fc30c06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:44.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.992+0000 7fc30a800640 1 --2- 172.21.15.45:0/1628400161 >> v2:172.21.15.103:3300/0 conn(0x7fc30c15ccb0 0x7fc30c06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.992+0000 7fc30a800640 1 -- 172.21.15.45:0/1628400161 >> v1:172.21.15.45:6789/0 conn(0x7fc30c146220 legacy=0x7fc30c06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:44.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.992+0000 7fc30a800640 1 -- 172.21.15.45:0/1628400161 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c15d640 con 0x7fc30c06e480 2024-04-16T16:14:44.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.993+0000 7fc30a800640 1 --2- 172.21.15.45:0/1628400161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c06e480 0x7fc30c1608a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc30c160590 tx=0x7fc30402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=156cb59ad25b276f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:44.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.994+0000 7fc308a00640 1 -- 172.21.15.45:0/1628400161 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc304039070 con 0x7fc30c06e480 2024-04-16T16:14:44.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.994+0000 7fc308a00640 1 -- 172.21.15.45:0/1628400161 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc30402cc60 con 0x7fc30c06e480 2024-04-16T16:14:44.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.994+0000 7fc308a00640 1 -- 172.21.15.45:0/1628400161 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc304034660 con 0x7fc30c06e480 2024-04-16T16:14:44.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.994+0000 7fc30b200640 1 -- 172.21.15.45:0/1628400161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c06e480 msgr2=0x7fc30c1608a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:44.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.994+0000 7fc30b200640 1 --2- 172.21.15.45:0/1628400161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c06e480 0x7fc30c1608a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc30c160590 tx=0x7fc30402beb0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.994+0000 7fc30b200640 1 -- 172.21.15.45:0/1628400161 shutdown_connections 2024-04-16T16:14:44.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.994+0000 7fc30b200640 1 --2- 172.21.15.45:0/1628400161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c06e480 0x7fc30c1608a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.994+0000 7fc30b200640 1 --2- 172.21.15.45:0/1628400161 >> v2:172.21.15.103:3300/0 conn(0x7fc30c15ccb0 0x7fc30c06df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.994+0000 7fc30b200640 1 -- 172.21.15.45:0/1628400161 >> 172.21.15.45:0/1628400161 conn(0x7fc30c06b930 msgr2=0x7fc30c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:44.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.995+0000 7fc30b200640 1 -- 172.21.15.45:0/1628400161 shutdown_connections 2024-04-16T16:14:44.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.995+0000 7fc30b200640 1 -- 172.21.15.45:0/1628400161 wait complete. 2024-04-16T16:14:44.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.996+0000 7fc30b200640 1 Processor -- start 2024-04-16T16:14:44.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.996+0000 7fc30b200640 1 -- start start 2024-04-16T16:14:44.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.997+0000 7fc30b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c06e480 0x7fc30c1f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:44.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.997+0000 7fc30b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc30c146220 0x7fc30c0c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:44.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.997+0000 7fc30b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc30c15ccb0 0x7fc30c1f0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:44.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.997+0000 7fc30b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c160de0 con 0x7fc30c06e480 2024-04-16T16:14:44.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.997+0000 7fc30b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c15dc00 con 0x7fc30c146220 2024-04-16T16:14:44.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.997+0000 7fc30b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc30c160f60 con 0x7fc30c15ccb0 2024-04-16T16:14:44.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.998+0000 7fc309e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c06e480 0x7fc30c1f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:44.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.998+0000 7fc309e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c06e480 0x7fc30c1f39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50614/0 (socket says 172.21.15.45:50614) 2024-04-16T16:14:44.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.998+0000 7fc309e00640 1 -- 172.21.15.45:0/1629501494 learned_addr learned my addr 172.21.15.45:0/1629501494 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:44.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.998+0000 7fc309e00640 1 -- 172.21.15.45:0/1629501494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc30c15ccb0 msgr2=0x7fc30c1f0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:44.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.998+0000 7fc30a800640 1 --2- 172.21.15.45:0/1629501494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc30c15ccb0 0x7fc30c1f0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:44.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.998+0000 7fc309e00640 1 --2- 172.21.15.45:0/1629501494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc30c15ccb0 0x7fc30c1f0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.998+0000 7fc309e00640 1 -- 172.21.15.45:0/1629501494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc30c146220 msgr2=0x7fc30c0c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:44.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.998+0000 7fc309400640 1 --2- 172.21.15.45:0/1629501494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc30c146220 0x7fc30c0c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:44.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.998+0000 7fc309e00640 1 --2- 172.21.15.45:0/1629501494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc30c146220 0x7fc30c0c89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:44.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.998+0000 7fc309e00640 1 -- 172.21.15.45:0/1629501494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c1f19e0 con 0x7fc30c06e480 2024-04-16T16:14:45.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.998+0000 7fc309e00640 1 --2- 172.21.15.45:0/1629501494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c06e480 0x7fc30c1f39d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc00c400 tx=0x7fc2fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:45.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.999+0000 7fc302000640 1 -- 172.21.15.45:0/1629501494 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc017070 con 0x7fc30c06e480 2024-04-16T16:14:45.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.999+0000 7fc302000640 1 -- 172.21.15.45:0/1629501494 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2fc005ce0 con 0x7fc30c06e480 2024-04-16T16:14:45.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.999+0000 7fc302000640 1 -- 172.21.15.45:0/1629501494 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0072f0 con 0x7fc30c06e480 2024-04-16T16:14:45.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.999+0000 7fc30b200640 1 -- 172.21.15.45:0/1629501494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30c143ee0 con 0x7fc30c06e480 2024-04-16T16:14:45.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:44.999+0000 7fc30b200640 1 -- 172.21.15.45:0/1629501494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30c1444a0 con 0x7fc30c06e480 2024-04-16T16:14:45.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.000+0000 7fc30b200640 1 -- 172.21.15.45:0/1629501494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc30c075780 con 0x7fc30c06e480 2024-04-16T16:14:45.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.001+0000 7fc302000640 1 -- 172.21.15.45:0/1629501494 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc2fc01a070 con 0x7fc30c06e480 2024-04-16T16:14:45.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.002+0000 7fc302000640 1 --2- 172.21.15.45:0/1629501494 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc2ec06c8f0 0x7fc2ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:45.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.003+0000 7fc302000640 1 -- 172.21.15.45:0/1629501494 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fc2fc08af70 con 0x7fc30c06e480 2024-04-16T16:14:45.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.003+0000 7fc309400640 1 --2- 172.21.15.45:0/1629501494 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc2ec06c8f0 0x7fc2ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:45.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.004+0000 7fc309400640 1 --2- 172.21.15.45:0/1629501494 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc2ec06c8f0 0x7fc2ec06edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc30c1455a0 tx=0x7fc2f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:45.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.007+0000 7fc302000640 1 -- 172.21.15.45:0/1629501494 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc2fc059390 con 0x7fc30c06e480 2024-04-16T16:14:45.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.211+0000 7fc30b200640 1 -- 172.21.15.45:0/1629501494 --> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc30c0c74b0 con 0x7fc2ec06c8f0 2024-04-16T16:14:45.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.229+0000 7fc302000640 1 -- 172.21.15.45:0/1629501494 <== mgr.4105 v2:172.21.15.103:6832/2711977587 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251950 (secure 0 0 0) 0x7fc30c0c74b0 con 0x7fc2ec06c8f0 2024-04-16T16:14:45.233 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:45.234 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:14:45.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.235+0000 7fc30b200640 1 -- 172.21.15.45:0/1629501494 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc2ec06c8f0 msgr2=0x7fc2ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:45.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.235+0000 7fc30b200640 1 --2- 172.21.15.45:0/1629501494 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc2ec06c8f0 0x7fc2ec06edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc30c1455a0 tx=0x7fc2f8008040 comp rx=0 tx=0).stop 2024-04-16T16:14:45.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.235+0000 7fc30b200640 1 -- 172.21.15.45:0/1629501494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c06e480 msgr2=0x7fc30c1f39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:45.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.235+0000 7fc30b200640 1 --2- 172.21.15.45:0/1629501494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c06e480 0x7fc30c1f39d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc00c400 tx=0x7fc2fc00c8c0 comp rx=0 tx=0).stop 2024-04-16T16:14:45.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.235+0000 7fc30b200640 1 -- 172.21.15.45:0/1629501494 shutdown_connections 2024-04-16T16:14:45.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.235+0000 7fc30b200640 1 --2- 172.21.15.45:0/1629501494 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fc2ec06c8f0 0x7fc2ec06edb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:45.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.235+0000 7fc30b200640 1 --2- 172.21.15.45:0/1629501494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc30c15ccb0 0x7fc30c1f0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:45.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.235+0000 7fc30b200640 1 --2- 172.21.15.45:0/1629501494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc30c146220 0x7fc30c0c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:45.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.235+0000 7fc30b200640 1 --2- 172.21.15.45:0/1629501494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c06e480 0x7fc30c1f39d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:45.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.235+0000 7fc30b200640 1 -- 172.21.15.45:0/1629501494 >> 172.21.15.45:0/1629501494 conn(0x7fc30c06b930 msgr2=0x7fc30c0cd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:45.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.235+0000 7fc30b200640 1 -- 172.21.15.45:0/1629501494 shutdown_connections 2024-04-16T16:14:45.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.235+0000 7fc30b200640 1 -- 172.21.15.45:0/1629501494 wait complete. 2024-04-16T16:14:45.255 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-04-16T16:14:45.119527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":158,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237040,"kb_used_data":4232,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737680,"statfs":{"total":773094113280,"available":772851384320,"internally_reserved":0,"allocated":4333568,"data_stored":2395788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.481590"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996516+0000","last_change":"2024-04-16T16:14:39.708878+0000","last_active":"2024-04-16T16:14:40.996516+0000","last_peered":"2024-04-16T16:14:40.996516+0000","last_clean":"2024-04-16T16:14:40.996516+0000","last_became_active":"2024-04-16T16:14:39.708146+0000","last_became_peered":"2024-04-16T16:14:39.708146+0000","last_unstale":"2024-04-16T16:14:40.996516+0000","last_undegraded":"2024-04-16T16:14:40.996516+0000","last_fullsized":"2024-04-16T16:14:40.996516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:24:14.006571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.686276+0000","last_change":"2024-04-16T16:14:39.697128+0000","last_active":"2024-04-16T16:14:40.686276+0000","last_peered":"2024-04-16T16:14:40.686276+0000","last_clean":"2024-04-16T16:14:40.686276+0000","last_became_active":"2024-04-16T16:14:39.696806+0000","last_became_peered":"2024-04-16T16:14:39.696806+0000","last_unstale":"2024-04-16T16:14:40.686276+0000","last_undegraded":"2024-04-16T16:14:40.686276+0000","last_fullsized":"2024-04-16T16:14:40.686276+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:55:56.042534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691181+0000","last_change":"2024-04-16T16:14:39.707270+0000","last_active":"2024-04-16T16:14:40.691181+0000","last_peered":"2024-04-16T16:14:40.691181+0000","last_clean":"2024-04-16T16:14:40.691181+0000","last_became_active":"2024-04-16T16:14:39.706592+0000","last_became_peered":"2024-04-16T16:14:39.706592+0000","last_unstale":"2024-04-16T16:14:40.691181+0000","last_undegraded":"2024-04-16T16:14:40.691181+0000","last_fullsized":"2024-04-16T16:14:40.691181+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:10:23.378754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291332+0000","last_change":"2024-04-16T16:14:39.704396+0000","last_active":"2024-04-16T16:14:41.291332+0000","last_peered":"2024-04-16T16:14:41.291332+0000","last_clean":"2024-04-16T16:14:41.291332+0000","last_became_active":"2024-04-16T16:14:39.703644+0000","last_became_peered":"2024-04-16T16:14:39.703644+0000","last_unstale":"2024-04-16T16:14:41.291332+0000","last_undegraded":"2024-04-16T16:14:41.291332+0000","last_fullsized":"2024-04-16T16:14:41.291332+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:28:27.889354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004853+0000","last_change":"2024-04-16T16:14:39.691482+0000","last_active":"2024-04-16T16:14:41.004853+0000","last_peered":"2024-04-16T16:14:41.004853+0000","last_clean":"2024-04-16T16:14:41.004853+0000","last_became_active":"2024-04-16T16:14:39.690475+0000","last_became_peered":"2024-04-16T16:14:39.690475+0000","last_unstale":"2024-04-16T16:14:41.004853+0000","last_undegraded":"2024-04-16T16:14:41.004853+0000","last_fullsized":"2024-04-16T16:14:41.004853+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:29:05.538981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685032+0000","last_change":"2024-04-16T16:14:39.707402+0000","last_active":"2024-04-16T16:14:40.685032+0000","last_peered":"2024-04-16T16:14:40.685032+0000","last_clean":"2024-04-16T16:14:40.685032+0000","last_became_active":"2024-04-16T16:14:39.707125+0000","last_became_peered":"2024-04-16T16:14:39.707125+0000","last_unstale":"2024-04-16T16:14:40.685032+0000","last_undegraded":"2024-04-16T16:14:40.685032+0000","last_fullsized":"2024-04-16T16:14:40.685032+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:24:24.985004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690212+0000","last_change":"2024-04-16T16:14:39.696478+0000","last_active":"2024-04-16T16:14:40.690212+0000","last_peered":"2024-04-16T16:14:40.690212+0000","last_clean":"2024-04-16T16:14:40.690212+0000","last_became_active":"2024-04-16T16:14:39.696171+0000","last_became_peered":"2024-04-16T16:14:39.696171+0000","last_unstale":"2024-04-16T16:14:40.690212+0000","last_undegraded":"2024-04-16T16:14:40.690212+0000","last_fullsized":"2024-04-16T16:14:40.690212+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:35.916987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.689394+0000","last_change":"2024-04-16T16:14:39.686048+0000","last_active":"2024-04-16T16:14:40.689394+0000","last_peered":"2024-04-16T16:14:40.689394+0000","last_clean":"2024-04-16T16:14:40.689394+0000","last_became_active":"2024-04-16T16:14:39.682978+0000","last_became_peered":"2024-04-16T16:14:39.682978+0000","last_unstale":"2024-04-16T16:14:40.689394+0000","last_undegraded":"2024-04-16T16:14:40.689394+0000","last_fullsized":"2024-04-16T16:14:40.689394+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:16:57.270160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292049+0000","last_change":"2024-04-16T16:14:39.701196+0000","last_active":"2024-04-16T16:14:41.292049+0000","last_peered":"2024-04-16T16:14:41.292049+0000","last_clean":"2024-04-16T16:14:41.292049+0000","last_became_active":"2024-04-16T16:14:39.700442+0000","last_became_peered":"2024-04-16T16:14:39.700442+0000","last_unstale":"2024-04-16T16:14:41.292049+0000","last_undegraded":"2024-04-16T16:14:41.292049+0000","last_fullsized":"2024-04-16T16:14:41.292049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:54.548210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291929+0000","last_change":"2024-04-16T16:14:39.707318+0000","last_active":"2024-04-16T16:14:41.291929+0000","last_peered":"2024-04-16T16:14:41.291929+0000","last_clean":"2024-04-16T16:14:41.291929+0000","last_became_active":"2024-04-16T16:14:39.706939+0000","last_became_peered":"2024-04-16T16:14:39.706939+0000","last_unstale":"2024-04-16T16:14:41.291929+0000","last_undegraded":"2024-04-16T16:14:41.291929+0000","last_fullsized":"2024-04-16T16:14:41.291929+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:23.282939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691664+0000","last_change":"2024-04-16T16:14:39.702686+0000","last_active":"2024-04-16T16:14:40.691664+0000","last_peered":"2024-04-16T16:14:40.691664+0000","last_clean":"2024-04-16T16:14:40.691664+0000","last_became_active":"2024-04-16T16:14:39.702274+0000","last_became_peered":"2024-04-16T16:14:39.702274+0000","last_unstale":"2024-04-16T16:14:40.691664+0000","last_undegraded":"2024-04-16T16:14:40.691664+0000","last_fullsized":"2024-04-16T16:14:40.691664+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:10:23.378754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690786+0000","last_change":"2024-04-16T16:14:39.709482+0000","last_active":"2024-04-16T16:14:40.690786+0000","last_peered":"2024-04-16T16:14:40.690786+0000","last_clean":"2024-04-16T16:14:40.690786+0000","last_became_active":"2024-04-16T16:14:39.709110+0000","last_became_peered":"2024-04-16T16:14:39.709110+0000","last_unstale":"2024-04-16T16:14:40.690786+0000","last_undegraded":"2024-04-16T16:14:40.690786+0000","last_fullsized":"2024-04-16T16:14:40.690786+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:31.834009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004919+0000","last_change":"2024-04-16T16:14:39.697790+0000","last_active":"2024-04-16T16:14:41.004919+0000","last_peered":"2024-04-16T16:14:41.004919+0000","last_clean":"2024-04-16T16:14:41.004919+0000","last_became_active":"2024-04-16T16:14:39.696862+0000","last_became_peered":"2024-04-16T16:14:39.696862+0000","last_unstale":"2024-04-16T16:14:41.004919+0000","last_undegraded":"2024-04-16T16:14:41.004919+0000","last_fullsized":"2024-04-16T16:14:41.004919+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:09:31.099716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685180+0000","last_change":"2024-04-16T16:14:39.691105+0000","last_active":"2024-04-16T16:14:40.685180+0000","last_peered":"2024-04-16T16:14:40.685180+0000","last_clean":"2024-04-16T16:14:40.685180+0000","last_became_active":"2024-04-16T16:14:39.690771+0000","last_became_peered":"2024-04-16T16:14:39.690771+0000","last_unstale":"2024-04-16T16:14:40.685180+0000","last_undegraded":"2024-04-16T16:14:40.685180+0000","last_fullsized":"2024-04-16T16:14:40.685180+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:10:23.378754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690774+0000","last_change":"2024-04-16T16:14:39.707728+0000","last_active":"2024-04-16T16:14:40.690774+0000","last_peered":"2024-04-16T16:14:40.690774+0000","last_clean":"2024-04-16T16:14:40.690774+0000","last_became_active":"2024-04-16T16:14:39.707447+0000","last_became_peered":"2024-04-16T16:14:39.707447+0000","last_unstale":"2024-04-16T16:14:40.690774+0000","last_undegraded":"2024-04-16T16:14:40.690774+0000","last_fullsized":"2024-04-16T16:14:40.690774+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:13.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691686+0000","last_change":"2024-04-16T16:14:39.689522+0000","last_active":"2024-04-16T16:14:40.691686+0000","last_peered":"2024-04-16T16:14:40.691686+0000","last_clean":"2024-04-16T16:14:40.691686+0000","last_became_active":"2024-04-16T16:14:39.685321+0000","last_became_peered":"2024-04-16T16:14:39.685321+0000","last_unstale":"2024-04-16T16:14:40.691686+0000","last_undegraded":"2024-04-16T16:14:40.691686+0000","last_fullsized":"2024-04-16T16:14:40.691686+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:23:31.999680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.692050+0000","last_change":"2024-04-16T16:14:39.699366+0000","last_active":"2024-04-16T16:14:40.692050+0000","last_peered":"2024-04-16T16:14:40.692050+0000","last_clean":"2024-04-16T16:14:40.692050+0000","last_became_active":"2024-04-16T16:14:39.697476+0000","last_became_peered":"2024-04-16T16:14:39.697476+0000","last_unstale":"2024-04-16T16:14:40.692050+0000","last_undegraded":"2024-04-16T16:14:40.692050+0000","last_fullsized":"2024-04-16T16:14:40.692050+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:51:43.656920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.686048+0000","last_change":"2024-04-16T16:14:39.710331+0000","last_active":"2024-04-16T16:14:40.686048+0000","last_peered":"2024-04-16T16:14:40.686048+0000","last_clean":"2024-04-16T16:14:40.686048+0000","last_became_active":"2024-04-16T16:14:39.710065+0000","last_became_peered":"2024-04-16T16:14:39.710065+0000","last_unstale":"2024-04-16T16:14:40.686048+0000","last_undegraded":"2024-04-16T16:14:40.686048+0000","last_fullsized":"2024-04-16T16:14:40.686048+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:37:58.914102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685635+0000","last_change":"2024-04-16T16:14:39.700033+0000","last_active":"2024-04-16T16:14:40.685635+0000","last_peered":"2024-04-16T16:14:40.685635+0000","last_clean":"2024-04-16T16:14:40.685635+0000","last_became_active":"2024-04-16T16:14:39.699473+0000","last_became_peered":"2024-04-16T16:14:39.699473+0000","last_unstale":"2024-04-16T16:14:40.685635+0000","last_undegraded":"2024-04-16T16:14:40.685635+0000","last_fullsized":"2024-04-16T16:14:40.685635+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:51:43.656920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004721+0000","last_change":"2024-04-16T16:14:39.713504+0000","last_active":"2024-04-16T16:14:41.004721+0000","last_peered":"2024-04-16T16:14:41.004721+0000","last_clean":"2024-04-16T16:14:41.004721+0000","last_became_active":"2024-04-16T16:14:39.711524+0000","last_became_peered":"2024-04-16T16:14:39.711524+0000","last_unstale":"2024-04-16T16:14:41.004721+0000","last_undegraded":"2024-04-16T16:14:41.004721+0000","last_fullsized":"2024-04-16T16:14:41.004721+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:06:16.299983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290756+0000","last_change":"2024-04-16T16:14:39.698857+0000","last_active":"2024-04-16T16:14:41.290756+0000","last_peered":"2024-04-16T16:14:41.290756+0000","last_clean":"2024-04-16T16:14:41.290756+0000","last_became_active":"2024-04-16T16:14:39.693200+0000","last_became_peered":"2024-04-16T16:14:39.693200+0000","last_unstale":"2024-04-16T16:14:41.290756+0000","last_undegraded":"2024-04-16T16:14:41.290756+0000","last_fullsized":"2024-04-16T16:14:41.290756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:17.454131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290778+0000","last_change":"2024-04-16T16:14:39.694980+0000","last_active":"2024-04-16T16:14:41.290778+0000","last_peered":"2024-04-16T16:14:41.290778+0000","last_clean":"2024-04-16T16:14:41.290778+0000","last_became_active":"2024-04-16T16:14:39.691011+0000","last_became_peered":"2024-04-16T16:14:39.691011+0000","last_unstale":"2024-04-16T16:14:41.290778+0000","last_undegraded":"2024-04-16T16:14:41.290778+0000","last_fullsized":"2024-04-16T16:14:41.290778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:24:14.006571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.726768+0000","last_change":"2024-04-16T16:14:39.687391+0000","last_active":"2024-04-16T16:14:40.726768+0000","last_peered":"2024-04-16T16:14:40.726768+0000","last_clean":"2024-04-16T16:14:40.726768+0000","last_became_active":"2024-04-16T16:14:39.687092+0000","last_became_peered":"2024-04-16T16:14:39.687092+0000","last_unstale":"2024-04-16T16:14:40.726768+0000","last_undegraded":"2024-04-16T16:14:40.726768+0000","last_fullsized":"2024-04-16T16:14:40.726768+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:31:35.408519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.689686+0000","last_change":"2024-04-16T16:14:39.707655+0000","last_active":"2024-04-16T16:14:40.689686+0000","last_peered":"2024-04-16T16:14:40.689686+0000","last_clean":"2024-04-16T16:14:40.689686+0000","last_became_active":"2024-04-16T16:14:39.707273+0000","last_became_peered":"2024-04-16T16:14:39.707273+0000","last_unstale":"2024-04-16T16:14:40.689686+0000","last_undegraded":"2024-04-16T16:14:40.689686+0000","last_fullsized":"2024-04-16T16:14:40.689686+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:23.282939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292509+0000","last_change":"2024-04-16T16:14:39.685869+0000","last_active":"2024-04-16T16:14:41.292509+0000","last_peered":"2024-04-16T16:14:41.292509+0000","last_clean":"2024-04-16T16:14:41.292509+0000","last_became_active":"2024-04-16T16:14:39.685394+0000","last_became_peered":"2024-04-16T16:14:39.685394+0000","last_unstale":"2024-04-16T16:14:41.292509+0000","last_undegraded":"2024-04-16T16:14:41.292509+0000","last_fullsized":"2024-04-16T16:14:41.292509+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:31.834009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.686198+0000","last_change":"2024-04-16T16:14:39.696275+0000","last_active":"2024-04-16T16:14:40.686198+0000","last_peered":"2024-04-16T16:14:40.686198+0000","last_clean":"2024-04-16T16:14:40.686198+0000","last_became_active":"2024-04-16T16:14:39.695976+0000","last_became_peered":"2024-04-16T16:14:39.695976+0000","last_unstale":"2024-04-16T16:14:40.686198+0000","last_undegraded":"2024-04-16T16:14:40.686198+0000","last_fullsized":"2024-04-16T16:14:40.686198+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:46:36.132077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292347+0000","last_change":"2024-04-16T16:14:39.689971+0000","last_active":"2024-04-16T16:14:41.292347+0000","last_peered":"2024-04-16T16:14:41.292347+0000","last_clean":"2024-04-16T16:14:41.292347+0000","last_became_active":"2024-04-16T16:14:39.689482+0000","last_became_peered":"2024-04-16T16:14:39.689482+0000","last_unstale":"2024-04-16T16:14:41.292347+0000","last_undegraded":"2024-04-16T16:14:41.292347+0000","last_fullsized":"2024-04-16T16:14:41.292347+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:35.916987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691169+0000","last_change":"2024-04-16T16:14:39.701188+0000","last_active":"2024-04-16T16:14:40.691169+0000","last_peered":"2024-04-16T16:14:40.691169+0000","last_clean":"2024-04-16T16:14:40.691169+0000","last_became_active":"2024-04-16T16:14:39.697558+0000","last_became_peered":"2024-04-16T16:14:39.697558+0000","last_unstale":"2024-04-16T16:14:40.691169+0000","last_undegraded":"2024-04-16T16:14:40.691169+0000","last_fullsized":"2024-04-16T16:14:40.691169+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:24:24.985004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290891+0000","last_change":"2024-04-16T16:14:39.702421+0000","last_active":"2024-04-16T16:14:41.290891+0000","last_peered":"2024-04-16T16:14:41.290891+0000","last_clean":"2024-04-16T16:14:41.290891+0000","last_became_active":"2024-04-16T16:14:39.701406+0000","last_became_peered":"2024-04-16T16:14:39.701406+0000","last_unstale":"2024-04-16T16:14:41.290891+0000","last_undegraded":"2024-04-16T16:14:41.290891+0000","last_fullsized":"2024-04-16T16:14:41.290891+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:45:31.844610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690802+0000","last_change":"2024-04-16T16:14:39.700536+0000","last_active":"2024-04-16T16:14:40.690802+0000","last_peered":"2024-04-16T16:14:40.690802+0000","last_clean":"2024-04-16T16:14:40.690802+0000","last_became_active":"2024-04-16T16:14:39.697778+0000","last_became_peered":"2024-04-16T16:14:39.697778+0000","last_unstale":"2024-04-16T16:14:40.690802+0000","last_undegraded":"2024-04-16T16:14:40.690802+0000","last_fullsized":"2024-04-16T16:14:40.690802+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:31.834009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996750+0000","last_change":"2024-04-16T16:14:39.698727+0000","last_active":"2024-04-16T16:14:40.996750+0000","last_peered":"2024-04-16T16:14:40.996750+0000","last_clean":"2024-04-16T16:14:40.996750+0000","last_became_active":"2024-04-16T16:14:39.696171+0000","last_became_peered":"2024-04-16T16:14:39.696171+0000","last_unstale":"2024-04-16T16:14:40.996750+0000","last_undegraded":"2024-04-16T16:14:40.996750+0000","last_fullsized":"2024-04-16T16:14:40.996750+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:33.368730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.689600+0000","last_change":"2024-04-16T16:14:39.694912+0000","last_active":"2024-04-16T16:14:40.689600+0000","last_peered":"2024-04-16T16:14:40.689600+0000","last_clean":"2024-04-16T16:14:40.689600+0000","last_became_active":"2024-04-16T16:14:39.694584+0000","last_became_peered":"2024-04-16T16:14:39.694584+0000","last_unstale":"2024-04-16T16:14:40.689600+0000","last_undegraded":"2024-04-16T16:14:40.689600+0000","last_fullsized":"2024-04-16T16:14:40.689600+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:01:01.716267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.727191+0000","last_change":"2024-04-16T16:14:39.681803+0000","last_active":"2024-04-16T16:14:40.727191+0000","last_peered":"2024-04-16T16:14:40.727191+0000","last_clean":"2024-04-16T16:14:40.727191+0000","last_became_active":"2024-04-16T16:14:39.681310+0000","last_became_peered":"2024-04-16T16:14:39.681310+0000","last_unstale":"2024-04-16T16:14:40.727191+0000","last_undegraded":"2024-04-16T16:14:40.727191+0000","last_fullsized":"2024-04-16T16:14:40.727191+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:54.548210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996669+0000","last_change":"2024-04-16T16:14:38.671801+0000","last_active":"2024-04-16T16:14:40.996669+0000","last_peered":"2024-04-16T16:14:40.996669+0000","last_clean":"2024-04-16T16:14:40.996669+0000","last_became_active":"2024-04-16T16:14:38.671502+0000","last_became_peered":"2024-04-16T16:14:38.671502+0000","last_unstale":"2024-04-16T16:14:40.996669+0000","last_undegraded":"2024-04-16T16:14:40.996669+0000","last_fullsized":"2024-04-16T16:14:40.996669+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:19:47.312068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996555+0000","last_change":"2024-04-16T16:14:39.708496+0000","last_active":"2024-04-16T16:14:40.996555+0000","last_peered":"2024-04-16T16:14:40.996555+0000","last_clean":"2024-04-16T16:14:40.996555+0000","last_became_active":"2024-04-16T16:14:39.707725+0000","last_became_peered":"2024-04-16T16:14:39.707725+0000","last_unstale":"2024-04-16T16:14:40.996555+0000","last_undegraded":"2024-04-16T16:14:40.996555+0000","last_fullsized":"2024-04-16T16:14:40.996555+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:31:35.408519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.727037+0000","last_change":"2024-04-16T16:14:38.663415+0000","last_active":"2024-04-16T16:14:40.727037+0000","last_peered":"2024-04-16T16:14:40.727037+0000","last_clean":"2024-04-16T16:14:40.727037+0000","last_became_active":"2024-04-16T16:14:38.662731+0000","last_became_peered":"2024-04-16T16:14:38.662731+0000","last_unstale":"2024-04-16T16:14:40.727037+0000","last_undegraded":"2024-04-16T16:14:40.727037+0000","last_fullsized":"2024-04-16T16:14:40.727037+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:39:10.432469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291543+0000","last_change":"2024-04-16T16:14:39.706462+0000","last_active":"2024-04-16T16:14:41.291543+0000","last_peered":"2024-04-16T16:14:41.291543+0000","last_clean":"2024-04-16T16:14:41.291543+0000","last_became_active":"2024-04-16T16:14:39.705749+0000","last_became_peered":"2024-04-16T16:14:39.705749+0000","last_unstale":"2024-04-16T16:14:41.291543+0000","last_undegraded":"2024-04-16T16:14:41.291543+0000","last_fullsized":"2024-04-16T16:14:41.291543+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:33.368730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685349+0000","last_change":"2024-04-16T16:14:38.668822+0000","last_active":"2024-04-16T16:14:40.685349+0000","last_peered":"2024-04-16T16:14:40.685349+0000","last_clean":"2024-04-16T16:14:40.685349+0000","last_became_active":"2024-04-16T16:14:38.668018+0000","last_became_peered":"2024-04-16T16:14:38.668018+0000","last_unstale":"2024-04-16T16:14:40.685349+0000","last_undegraded":"2024-04-16T16:14:40.685349+0000","last_fullsized":"2024-04-16T16:14:40.685349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:16.441878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292426+0000","last_change":"2024-04-16T16:14:39.689746+0000","last_active":"2024-04-16T16:14:41.292426+0000","last_peered":"2024-04-16T16:14:41.292426+0000","last_clean":"2024-04-16T16:14:41.292426+0000","last_became_active":"2024-04-16T16:14:39.689162+0000","last_became_peered":"2024-04-16T16:14:39.689162+0000","last_unstale":"2024-04-16T16:14:41.292426+0000","last_undegraded":"2024-04-16T16:14:41.292426+0000","last_fullsized":"2024-04-16T16:14:41.292426+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:13.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004846+0000","last_change":"2024-04-16T16:14:38.673943+0000","last_active":"2024-04-16T16:14:41.004846+0000","last_peered":"2024-04-16T16:14:41.004846+0000","last_clean":"2024-04-16T16:14:41.004846+0000","last_became_active":"2024-04-16T16:14:38.672857+0000","last_became_peered":"2024-04-16T16:14:38.672857+0000","last_unstale":"2024-04-16T16:14:41.004846+0000","last_undegraded":"2024-04-16T16:14:41.004846+0000","last_fullsized":"2024-04-16T16:14:41.004846+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:38:56.751996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691157+0000","last_change":"2024-04-16T16:14:39.704670+0000","last_active":"2024-04-16T16:14:40.691157+0000","last_peered":"2024-04-16T16:14:40.691157+0000","last_clean":"2024-04-16T16:14:40.691157+0000","last_became_active":"2024-04-16T16:14:39.704321+0000","last_became_peered":"2024-04-16T16:14:39.704321+0000","last_unstale":"2024-04-16T16:14:40.691157+0000","last_undegraded":"2024-04-16T16:14:40.691157+0000","last_fullsized":"2024-04-16T16:14:40.691157+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:24:24.985004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290894+0000","last_change":"2024-04-16T16:14:38.662314+0000","last_active":"2024-04-16T16:14:41.290894+0000","last_peered":"2024-04-16T16:14:41.290894+0000","last_clean":"2024-04-16T16:14:41.290894+0000","last_became_active":"2024-04-16T16:14:38.662011+0000","last_became_peered":"2024-04-16T16:14:38.662011+0000","last_unstale":"2024-04-16T16:14:41.290894+0000","last_undegraded":"2024-04-16T16:14:41.290894+0000","last_fullsized":"2024-04-16T16:14:41.290894+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:49.878355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691057+0000","last_change":"2024-04-16T16:14:39.697339+0000","last_active":"2024-04-16T16:14:40.691057+0000","last_peered":"2024-04-16T16:14:40.691057+0000","last_clean":"2024-04-16T16:14:40.691057+0000","last_became_active":"2024-04-16T16:14:39.697037+0000","last_became_peered":"2024-04-16T16:14:39.697037+0000","last_unstale":"2024-04-16T16:14:40.691057+0000","last_undegraded":"2024-04-16T16:14:40.691057+0000","last_fullsized":"2024-04-16T16:14:40.691057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:27:36.166180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290724+0000","last_change":"2024-04-16T16:14:38.660477+0000","last_active":"2024-04-16T16:14:41.290724+0000","last_peered":"2024-04-16T16:14:41.290724+0000","last_clean":"2024-04-16T16:14:41.290724+0000","last_became_active":"2024-04-16T16:14:38.660156+0000","last_became_peered":"2024-04-16T16:14:38.660156+0000","last_unstale":"2024-04-16T16:14:41.290724+0000","last_undegraded":"2024-04-16T16:14:41.290724+0000","last_fullsized":"2024-04-16T16:14:41.290724+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:43:48.615033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291106+0000","last_change":"2024-04-16T16:14:39.698584+0000","last_active":"2024-04-16T16:14:41.291106+0000","last_peered":"2024-04-16T16:14:41.291106+0000","last_clean":"2024-04-16T16:14:41.291106+0000","last_became_active":"2024-04-16T16:14:39.697610+0000","last_became_peered":"2024-04-16T16:14:39.697610+0000","last_unstale":"2024-04-16T16:14:41.291106+0000","last_undegraded":"2024-04-16T16:14:41.291106+0000","last_fullsized":"2024-04-16T16:14:41.291106+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:31:35.408519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.684802+0000","last_change":"2024-04-16T16:14:38.657699+0000","last_active":"2024-04-16T16:14:40.684802+0000","last_peered":"2024-04-16T16:14:40.684802+0000","last_clean":"2024-04-16T16:14:40.684802+0000","last_became_active":"2024-04-16T16:14:38.657409+0000","last_became_peered":"2024-04-16T16:14:38.657409+0000","last_unstale":"2024-04-16T16:14:40.684802+0000","last_undegraded":"2024-04-16T16:14:40.684802+0000","last_fullsized":"2024-04-16T16:14:40.684802+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:25:42.846200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291805+0000","last_change":"2024-04-16T16:14:39.707695+0000","last_active":"2024-04-16T16:14:41.291805+0000","last_peered":"2024-04-16T16:14:41.291805+0000","last_clean":"2024-04-16T16:14:41.291805+0000","last_became_active":"2024-04-16T16:14:39.707352+0000","last_became_peered":"2024-04-16T16:14:39.707352+0000","last_unstale":"2024-04-16T16:14:41.291805+0000","last_undegraded":"2024-04-16T16:14:41.291805+0000","last_fullsized":"2024-04-16T16:14:41.291805+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:01:01.716267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290472+0000","last_change":"2024-04-16T16:14:38.664407+0000","last_active":"2024-04-16T16:14:41.290472+0000","last_peered":"2024-04-16T16:14:41.290472+0000","last_clean":"2024-04-16T16:14:41.290472+0000","last_became_active":"2024-04-16T16:14:38.664039+0000","last_became_peered":"2024-04-16T16:14:38.664039+0000","last_unstale":"2024-04-16T16:14:41.290472+0000","last_undegraded":"2024-04-16T16:14:41.290472+0000","last_fullsized":"2024-04-16T16:14:41.290472+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:41:56.284222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691494+0000","last_change":"2024-04-16T16:14:39.701237+0000","last_active":"2024-04-16T16:14:40.691494+0000","last_peered":"2024-04-16T16:14:40.691494+0000","last_clean":"2024-04-16T16:14:40.691494+0000","last_became_active":"2024-04-16T16:14:39.700904+0000","last_became_peered":"2024-04-16T16:14:39.700904+0000","last_unstale":"2024-04-16T16:14:40.691494+0000","last_undegraded":"2024-04-16T16:14:40.691494+0000","last_fullsized":"2024-04-16T16:14:40.691494+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:23:31.999680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.686037+0000","last_change":"2024-04-16T16:14:38.658747+0000","last_active":"2024-04-16T16:14:40.686037+0000","last_peered":"2024-04-16T16:14:40.686037+0000","last_clean":"2024-04-16T16:14:40.686037+0000","last_became_active":"2024-04-16T16:14:38.656842+0000","last_became_peered":"2024-04-16T16:14:38.656842+0000","last_unstale":"2024-04-16T16:14:40.686037+0000","last_undegraded":"2024-04-16T16:14:40.686037+0000","last_fullsized":"2024-04-16T16:14:40.686037+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:45:30.832358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.686289+0000","last_change":"2024-04-16T16:14:39.699558+0000","last_active":"2024-04-16T16:14:40.686289+0000","last_peered":"2024-04-16T16:14:40.686289+0000","last_clean":"2024-04-16T16:14:40.686289+0000","last_became_active":"2024-04-16T16:14:39.699265+0000","last_became_peered":"2024-04-16T16:14:39.699265+0000","last_unstale":"2024-04-16T16:14:40.686289+0000","last_undegraded":"2024-04-16T16:14:40.686289+0000","last_fullsized":"2024-04-16T16:14:40.686289+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:43:08.613864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.727282+0000","last_change":"2024-04-16T16:14:38.672532+0000","last_active":"2024-04-16T16:14:40.727282+0000","last_peered":"2024-04-16T16:14:40.727282+0000","last_clean":"2024-04-16T16:14:40.727282+0000","last_became_active":"2024-04-16T16:14:38.671881+0000","last_became_peered":"2024-04-16T16:14:38.671881+0000","last_unstale":"2024-04-16T16:14:40.727282+0000","last_undegraded":"2024-04-16T16:14:40.727282+0000","last_fullsized":"2024-04-16T16:14:40.727282+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:18:01.189685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685628+0000","last_change":"2024-04-16T16:14:39.697739+0000","last_active":"2024-04-16T16:14:40.685628+0000","last_peered":"2024-04-16T16:14:40.685628+0000","last_clean":"2024-04-16T16:14:40.685628+0000","last_became_active":"2024-04-16T16:14:39.697472+0000","last_became_peered":"2024-04-16T16:14:39.697472+0000","last_unstale":"2024-04-16T16:14:40.685628+0000","last_undegraded":"2024-04-16T16:14:40.685628+0000","last_fullsized":"2024-04-16T16:14:40.685628+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:02:01.054178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691347+0000","last_change":"2024-04-16T16:14:38.667807+0000","last_active":"2024-04-16T16:14:40.691347+0000","last_peered":"2024-04-16T16:14:40.691347+0000","last_clean":"2024-04-16T16:14:40.691347+0000","last_became_active":"2024-04-16T16:14:38.667504+0000","last_became_peered":"2024-04-16T16:14:38.667504+0000","last_unstale":"2024-04-16T16:14:40.691347+0000","last_undegraded":"2024-04-16T16:14:40.691347+0000","last_fullsized":"2024-04-16T16:14:40.691347+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:50.689231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690759+0000","last_change":"2024-04-16T16:14:39.707564+0000","last_active":"2024-04-16T16:14:40.690759+0000","last_peered":"2024-04-16T16:14:40.690759+0000","last_clean":"2024-04-16T16:14:40.690759+0000","last_became_active":"2024-04-16T16:14:39.706939+0000","last_became_peered":"2024-04-16T16:14:39.706939+0000","last_unstale":"2024-04-16T16:14:40.690759+0000","last_undegraded":"2024-04-16T16:14:40.690759+0000","last_fullsized":"2024-04-16T16:14:40.690759+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:35.916987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685723+0000","last_change":"2024-04-16T16:14:38.669527+0000","last_active":"2024-04-16T16:14:40.685723+0000","last_peered":"2024-04-16T16:14:40.685723+0000","last_clean":"2024-04-16T16:14:40.685723+0000","last_became_active":"2024-04-16T16:14:38.669189+0000","last_became_peered":"2024-04-16T16:14:38.669189+0000","last_unstale":"2024-04-16T16:14:40.685723+0000","last_undegraded":"2024-04-16T16:14:40.685723+0000","last_fullsized":"2024-04-16T16:14:40.685723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:44:46.897943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291278+0000","last_change":"2024-04-16T16:14:39.708697+0000","last_active":"2024-04-16T16:14:41.291278+0000","last_peered":"2024-04-16T16:14:41.291278+0000","last_clean":"2024-04-16T16:14:41.291278+0000","last_became_active":"2024-04-16T16:14:39.708356+0000","last_became_peered":"2024-04-16T16:14:39.708356+0000","last_unstale":"2024-04-16T16:14:41.291278+0000","last_undegraded":"2024-04-16T16:14:41.291278+0000","last_fullsized":"2024-04-16T16:14:41.291278+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:54.548210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291043+0000","last_change":"2024-04-16T16:14:38.665714+0000","last_active":"2024-04-16T16:14:41.291043+0000","last_peered":"2024-04-16T16:14:41.291043+0000","last_clean":"2024-04-16T16:14:41.291043+0000","last_became_active":"2024-04-16T16:14:38.665380+0000","last_became_peered":"2024-04-16T16:14:38.665380+0000","last_unstale":"2024-04-16T16:14:41.291043+0000","last_undegraded":"2024-04-16T16:14:41.291043+0000","last_fullsized":"2024-04-16T16:14:41.291043+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:32:37.861499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685516+0000","last_change":"2024-04-16T16:14:39.687320+0000","last_active":"2024-04-16T16:14:40.685516+0000","last_peered":"2024-04-16T16:14:40.685516+0000","last_clean":"2024-04-16T16:14:40.685516+0000","last_became_active":"2024-04-16T16:14:39.686484+0000","last_became_peered":"2024-04-16T16:14:39.686484+0000","last_unstale":"2024-04-16T16:14:40.685516+0000","last_undegraded":"2024-04-16T16:14:40.685516+0000","last_fullsized":"2024-04-16T16:14:40.685516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:23:31.999680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291313+0000","last_change":"2024-04-16T16:14:38.672232+0000","last_active":"2024-04-16T16:14:41.291313+0000","last_peered":"2024-04-16T16:14:41.291313+0000","last_clean":"2024-04-16T16:14:41.291313+0000","last_became_active":"2024-04-16T16:14:38.671940+0000","last_became_peered":"2024-04-16T16:14:38.671940+0000","last_unstale":"2024-04-16T16:14:41.291313+0000","last_undegraded":"2024-04-16T16:14:41.291313+0000","last_fullsized":"2024-04-16T16:14:41.291313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:46.463986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690402+0000","last_change":"2024-04-16T16:14:39.691527+0000","last_active":"2024-04-16T16:14:40.690402+0000","last_peered":"2024-04-16T16:14:40.690402+0000","last_clean":"2024-04-16T16:14:40.690402+0000","last_became_active":"2024-04-16T16:14:39.691235+0000","last_became_peered":"2024-04-16T16:14:39.691235+0000","last_unstale":"2024-04-16T16:14:40.690402+0000","last_undegraded":"2024-04-16T16:14:40.690402+0000","last_fullsized":"2024-04-16T16:14:40.690402+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:13.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.684902+0000","last_change":"2024-04-16T16:14:38.659068+0000","last_active":"2024-04-16T16:14:40.684902+0000","last_peered":"2024-04-16T16:14:40.684902+0000","last_clean":"2024-04-16T16:14:40.684902+0000","last_became_active":"2024-04-16T16:14:38.658666+0000","last_became_peered":"2024-04-16T16:14:38.658666+0000","last_unstale":"2024-04-16T16:14:40.684902+0000","last_undegraded":"2024-04-16T16:14:40.684902+0000","last_fullsized":"2024-04-16T16:14:40.684902+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:50.689231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996507+0000","last_change":"2024-04-16T16:14:39.703632+0000","last_active":"2024-04-16T16:14:40.996507+0000","last_peered":"2024-04-16T16:14:40.996507+0000","last_clean":"2024-04-16T16:14:40.996507+0000","last_became_active":"2024-04-16T16:14:39.703276+0000","last_became_peered":"2024-04-16T16:14:39.703276+0000","last_unstale":"2024-04-16T16:14:40.996507+0000","last_undegraded":"2024-04-16T16:14:40.996507+0000","last_fullsized":"2024-04-16T16:14:40.996507+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:45:31.844610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":22,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.684827+0000","last_change":"2024-04-16T16:14:38.659623+0000","last_active":"2024-04-16T16:14:40.684827+0000","last_peered":"2024-04-16T16:14:40.684827+0000","last_clean":"2024-04-16T16:14:40.684827+0000","last_became_active":"2024-04-16T16:14:38.659313+0000","last_became_peered":"2024-04-16T16:14:38.659313+0000","last_unstale":"2024-04-16T16:14:40.684827+0000","last_undegraded":"2024-04-16T16:14:40.684827+0000","last_fullsized":"2024-04-16T16:14:40.684827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:24:12.994319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291714+0000","last_change":"2024-04-16T16:14:39.702044+0000","last_active":"2024-04-16T16:14:41.291714+0000","last_peered":"2024-04-16T16:14:41.291714+0000","last_clean":"2024-04-16T16:14:41.291714+0000","last_became_active":"2024-04-16T16:14:39.701773+0000","last_became_peered":"2024-04-16T16:14:39.701773+0000","last_unstale":"2024-04-16T16:14:41.291714+0000","last_undegraded":"2024-04-16T16:14:41.291714+0000","last_fullsized":"2024-04-16T16:14:41.291714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:36.323418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292516+0000","last_change":"2024-04-16T16:14:38.675837+0000","last_active":"2024-04-16T16:14:41.292516+0000","last_peered":"2024-04-16T16:14:41.292516+0000","last_clean":"2024-04-16T16:14:41.292516+0000","last_became_active":"2024-04-16T16:14:38.675519+0000","last_became_peered":"2024-04-16T16:14:38.675519+0000","last_unstale":"2024-04-16T16:14:41.292516+0000","last_undegraded":"2024-04-16T16:14:41.292516+0000","last_fullsized":"2024-04-16T16:14:41.292516+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:39:10.432469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691937+0000","last_change":"2024-04-16T16:14:39.702120+0000","last_active":"2024-04-16T16:14:40.691937+0000","last_peered":"2024-04-16T16:14:40.691937+0000","last_clean":"2024-04-16T16:14:40.691937+0000","last_became_active":"2024-04-16T16:14:39.701813+0000","last_became_peered":"2024-04-16T16:14:39.701813+0000","last_unstale":"2024-04-16T16:14:40.691937+0000","last_undegraded":"2024-04-16T16:14:40.691937+0000","last_fullsized":"2024-04-16T16:14:40.691937+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:51:43.656920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291528+0000","last_change":"2024-04-16T16:14:38.666762+0000","last_active":"2024-04-16T16:14:41.291528+0000","last_peered":"2024-04-16T16:14:41.291528+0000","last_clean":"2024-04-16T16:14:41.291528+0000","last_became_active":"2024-04-16T16:14:38.666193+0000","last_became_peered":"2024-04-16T16:14:38.666193+0000","last_unstale":"2024-04-16T16:14:41.291528+0000","last_undegraded":"2024-04-16T16:14:41.291528+0000","last_fullsized":"2024-04-16T16:14:41.291528+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:08:32.600681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291561+0000","last_change":"2024-04-16T16:14:39.704971+0000","last_active":"2024-04-16T16:14:41.291561+0000","last_peered":"2024-04-16T16:14:41.291561+0000","last_clean":"2024-04-16T16:14:41.291561+0000","last_became_active":"2024-04-16T16:14:39.704693+0000","last_became_peered":"2024-04-16T16:14:39.704693+0000","last_unstale":"2024-04-16T16:14:41.291561+0000","last_undegraded":"2024-04-16T16:14:41.291561+0000","last_fullsized":"2024-04-16T16:14:41.291561+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:16:57.270160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291579+0000","last_change":"2024-04-16T16:14:38.657843+0000","last_active":"2024-04-16T16:14:41.291579+0000","last_peered":"2024-04-16T16:14:41.291579+0000","last_clean":"2024-04-16T16:14:41.291579+0000","last_became_active":"2024-04-16T16:14:38.657384+0000","last_became_peered":"2024-04-16T16:14:38.657384+0000","last_unstale":"2024-04-16T16:14:41.291579+0000","last_undegraded":"2024-04-16T16:14:41.291579+0000","last_fullsized":"2024-04-16T16:14:41.291579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:19:47.312068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.005465+0000","last_change":"2024-04-16T16:14:39.699276+0000","last_active":"2024-04-16T16:14:41.005465+0000","last_peered":"2024-04-16T16:14:41.005465+0000","last_clean":"2024-04-16T16:14:41.005465+0000","last_became_active":"2024-04-16T16:14:39.698973+0000","last_became_peered":"2024-04-16T16:14:39.698973+0000","last_unstale":"2024-04-16T16:14:41.005465+0000","last_undegraded":"2024-04-16T16:14:41.005465+0000","last_fullsized":"2024-04-16T16:14:41.005465+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:47.476239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291384+0000","last_change":"2024-04-16T16:14:38.663164+0000","last_active":"2024-04-16T16:14:41.291384+0000","last_peered":"2024-04-16T16:14:41.291384+0000","last_clean":"2024-04-16T16:14:41.291384+0000","last_became_active":"2024-04-16T16:14:38.662887+0000","last_became_peered":"2024-04-16T16:14:38.662887+0000","last_unstale":"2024-04-16T16:14:41.291384+0000","last_undegraded":"2024-04-16T16:14:41.291384+0000","last_fullsized":"2024-04-16T16:14:41.291384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:14:46.163376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004691+0000","last_change":"2024-04-16T16:14:39.704026+0000","last_active":"2024-04-16T16:14:41.004691+0000","last_peered":"2024-04-16T16:14:41.004691+0000","last_clean":"2024-04-16T16:14:41.004691+0000","last_became_active":"2024-04-16T16:14:39.703710+0000","last_became_peered":"2024-04-16T16:14:39.703710+0000","last_unstale":"2024-04-16T16:14:41.004691+0000","last_undegraded":"2024-04-16T16:14:41.004691+0000","last_fullsized":"2024-04-16T16:14:41.004691+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:41:57.296475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.726518+0000","last_change":"2024-04-16T16:14:38.665012+0000","last_active":"2024-04-16T16:14:40.726518+0000","last_peered":"2024-04-16T16:14:40.726518+0000","last_clean":"2024-04-16T16:14:40.726518+0000","last_became_active":"2024-04-16T16:14:38.664721+0000","last_became_peered":"2024-04-16T16:14:38.664721+0000","last_unstale":"2024-04-16T16:14:40.726518+0000","last_undegraded":"2024-04-16T16:14:40.726518+0000","last_fullsized":"2024-04-16T16:14:40.726518+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:46.463986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.005261+0000","last_change":"2024-04-16T16:14:39.681884+0000","last_active":"2024-04-16T16:14:41.005261+0000","last_peered":"2024-04-16T16:14:41.005261+0000","last_clean":"2024-04-16T16:14:41.005261+0000","last_became_active":"2024-04-16T16:14:39.681629+0000","last_became_peered":"2024-04-16T16:14:39.681629+0000","last_unstale":"2024-04-16T16:14:41.005261+0000","last_undegraded":"2024-04-16T16:14:41.005261+0000","last_fullsized":"2024-04-16T16:14:41.005261+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:19:48.324320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996523+0000","last_change":"2024-04-16T16:14:38.671342+0000","last_active":"2024-04-16T16:14:40.996523+0000","last_peered":"2024-04-16T16:14:40.996523+0000","last_clean":"2024-04-16T16:14:40.996523+0000","last_became_active":"2024-04-16T16:14:38.670918+0000","last_became_peered":"2024-04-16T16:14:38.670918+0000","last_unstale":"2024-04-16T16:14:40.996523+0000","last_undegraded":"2024-04-16T16:14:40.996523+0000","last_fullsized":"2024-04-16T16:14:40.996523+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:08:32.600681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004216+0000","last_change":"2024-04-16T16:14:39.681058+0000","last_active":"2024-04-16T16:14:41.004216+0000","last_peered":"2024-04-16T16:14:41.004216+0000","last_clean":"2024-04-16T16:14:41.004216+0000","last_became_active":"2024-04-16T16:14:39.678280+0000","last_became_peered":"2024-04-16T16:14:39.678280+0000","last_unstale":"2024-04-16T16:14:41.004216+0000","last_undegraded":"2024-04-16T16:14:41.004216+0000","last_fullsized":"2024-04-16T16:14:41.004216+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:34.458058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290500+0000","last_change":"2024-04-16T16:14:38.668645+0000","last_active":"2024-04-16T16:14:41.290500+0000","last_peered":"2024-04-16T16:14:41.290500+0000","last_clean":"2024-04-16T16:14:41.290500+0000","last_became_active":"2024-04-16T16:14:38.668376+0000","last_became_peered":"2024-04-16T16:14:38.668376+0000","last_unstale":"2024-04-16T16:14:41.290500+0000","last_undegraded":"2024-04-16T16:14:41.290500+0000","last_fullsized":"2024-04-16T16:14:41.290500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:33.445805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690968+0000","last_change":"2024-04-16T16:14:39.697865+0000","last_active":"2024-04-16T16:14:40.690968+0000","last_peered":"2024-04-16T16:14:40.690968+0000","last_clean":"2024-04-16T16:14:40.690968+0000","last_became_active":"2024-04-16T16:14:39.696925+0000","last_became_peered":"2024-04-16T16:14:39.696925+0000","last_unstale":"2024-04-16T16:14:40.690968+0000","last_undegraded":"2024-04-16T16:14:40.690968+0000","last_fullsized":"2024-04-16T16:14:40.690968+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:27:36.166180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.689554+0000","last_change":"2024-04-16T16:14:38.668164+0000","last_active":"2024-04-16T16:14:40.689554+0000","last_peered":"2024-04-16T16:14:40.689554+0000","last_clean":"2024-04-16T16:14:40.689554+0000","last_became_active":"2024-04-16T16:14:38.667856+0000","last_became_peered":"2024-04-16T16:14:38.667856+0000","last_unstale":"2024-04-16T16:14:40.689554+0000","last_undegraded":"2024-04-16T16:14:40.689554+0000","last_fullsized":"2024-04-16T16:14:40.689554+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:16.441878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004964+0000","last_change":"2024-04-16T16:14:39.694889+0000","last_active":"2024-04-16T16:14:41.004964+0000","last_peered":"2024-04-16T16:14:41.004964+0000","last_clean":"2024-04-16T16:14:41.004964+0000","last_became_active":"2024-04-16T16:14:39.694130+0000","last_became_peered":"2024-04-16T16:14:39.694130+0000","last_unstale":"2024-04-16T16:14:41.004964+0000","last_undegraded":"2024-04-16T16:14:41.004964+0000","last_fullsized":"2024-04-16T16:14:41.004964+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:08:33.612934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292617+0000","last_change":"2024-04-16T16:14:38.674547+0000","last_active":"2024-04-16T16:14:41.292617+0000","last_peered":"2024-04-16T16:14:41.292617+0000","last_clean":"2024-04-16T16:14:41.292617+0000","last_became_active":"2024-04-16T16:14:38.674247+0000","last_became_peered":"2024-04-16T16:14:38.674247+0000","last_unstale":"2024-04-16T16:14:41.292617+0000","last_undegraded":"2024-04-16T16:14:41.292617+0000","last_fullsized":"2024-04-16T16:14:41.292617+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:25:42.846200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.692158+0000","last_change":"2024-04-16T16:14:39.709800+0000","last_active":"2024-04-16T16:14:40.692158+0000","last_peered":"2024-04-16T16:14:40.692158+0000","last_clean":"2024-04-16T16:14:40.692158+0000","last_became_active":"2024-04-16T16:14:39.709509+0000","last_became_peered":"2024-04-16T16:14:39.709509+0000","last_unstale":"2024-04-16T16:14:40.692158+0000","last_undegraded":"2024-04-16T16:14:40.692158+0000","last_fullsized":"2024-04-16T16:14:40.692158+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:02:01.054178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.727281+0000","last_change":"2024-04-16T16:14:38.669482+0000","last_active":"2024-04-16T16:14:40.727281+0000","last_peered":"2024-04-16T16:14:40.727281+0000","last_clean":"2024-04-16T16:14:40.727281+0000","last_became_active":"2024-04-16T16:14:38.669175+0000","last_became_peered":"2024-04-16T16:14:38.669175+0000","last_unstale":"2024-04-16T16:14:40.727281+0000","last_undegraded":"2024-04-16T16:14:40.727281+0000","last_fullsized":"2024-04-16T16:14:40.727281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:25:42.846200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.726937+0000","last_change":"2024-04-16T16:14:39.684854+0000","last_active":"2024-04-16T16:14:40.726937+0000","last_peered":"2024-04-16T16:14:40.726937+0000","last_clean":"2024-04-16T16:14:40.726937+0000","last_became_active":"2024-04-16T16:14:39.684524+0000","last_became_peered":"2024-04-16T16:14:39.684524+0000","last_unstale":"2024-04-16T16:14:40.726937+0000","last_undegraded":"2024-04-16T16:14:40.726937+0000","last_fullsized":"2024-04-16T16:14:40.726937+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:36.323418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291797+0000","last_change":"2024-04-16T16:14:38.675135+0000","last_active":"2024-04-16T16:14:41.291797+0000","last_peered":"2024-04-16T16:14:41.291797+0000","last_clean":"2024-04-16T16:14:41.291797+0000","last_became_active":"2024-04-16T16:14:38.674821+0000","last_became_peered":"2024-04-16T16:14:38.674821+0000","last_unstale":"2024-04-16T16:14:41.291797+0000","last_undegraded":"2024-04-16T16:14:41.291797+0000","last_fullsized":"2024-04-16T16:14:41.291797+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:18:01.189685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996428+0000","last_change":"2024-04-16T16:14:39.692135+0000","last_active":"2024-04-16T16:14:40.996428+0000","last_peered":"2024-04-16T16:14:40.996428+0000","last_clean":"2024-04-16T16:14:40.996428+0000","last_became_active":"2024-04-16T16:14:39.691817+0000","last_became_peered":"2024-04-16T16:14:39.691817+0000","last_unstale":"2024-04-16T16:14:40.996428+0000","last_undegraded":"2024-04-16T16:14:40.996428+0000","last_fullsized":"2024-04-16T16:14:40.996428+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:17.454131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.726901+0000","last_change":"2024-04-16T16:14:38.669656+0000","last_active":"2024-04-16T16:14:40.726901+0000","last_peered":"2024-04-16T16:14:40.726901+0000","last_clean":"2024-04-16T16:14:40.726901+0000","last_became_active":"2024-04-16T16:14:38.669304+0000","last_became_peered":"2024-04-16T16:14:38.669304+0000","last_unstale":"2024-04-16T16:14:40.726901+0000","last_undegraded":"2024-04-16T16:14:40.726901+0000","last_fullsized":"2024-04-16T16:14:40.726901+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:26:06.671145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.726817+0000","last_change":"2024-04-16T16:14:39.692738+0000","last_active":"2024-04-16T16:14:40.726817+0000","last_peered":"2024-04-16T16:14:40.726817+0000","last_clean":"2024-04-16T16:14:40.726817+0000","last_became_active":"2024-04-16T16:14:39.692443+0000","last_became_peered":"2024-04-16T16:14:39.692443+0000","last_unstale":"2024-04-16T16:14:40.726817+0000","last_undegraded":"2024-04-16T16:14:40.726817+0000","last_fullsized":"2024-04-16T16:14:40.726817+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:33.368730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291411+0000","last_change":"2024-04-16T16:14:38.673858+0000","last_active":"2024-04-16T16:14:41.291411+0000","last_peered":"2024-04-16T16:14:41.291411+0000","last_clean":"2024-04-16T16:14:41.291411+0000","last_became_active":"2024-04-16T16:14:38.673531+0000","last_became_peered":"2024-04-16T16:14:38.673531+0000","last_unstale":"2024-04-16T16:14:41.291411+0000","last_undegraded":"2024-04-16T16:14:41.291411+0000","last_fullsized":"2024-04-16T16:14:41.291411+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:26:06.671145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.684803+0000","last_change":"2024-04-16T16:14:39.700548+0000","last_active":"2024-04-16T16:14:40.684803+0000","last_peered":"2024-04-16T16:14:40.684803+0000","last_clean":"2024-04-16T16:14:40.684803+0000","last_became_active":"2024-04-16T16:14:39.700278+0000","last_became_peered":"2024-04-16T16:14:39.700278+0000","last_unstale":"2024-04-16T16:14:40.684803+0000","last_undegraded":"2024-04-16T16:14:40.684803+0000","last_fullsized":"2024-04-16T16:14:40.684803+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:27:36.166180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290796+0000","last_change":"2024-04-16T16:14:38.671160+0000","last_active":"2024-04-16T16:14:41.290796+0000","last_peered":"2024-04-16T16:14:41.290796+0000","last_clean":"2024-04-16T16:14:41.290796+0000","last_became_active":"2024-04-16T16:14:38.670444+0000","last_became_peered":"2024-04-16T16:14:38.670444+0000","last_unstale":"2024-04-16T16:14:41.290796+0000","last_undegraded":"2024-04-16T16:14:41.290796+0000","last_fullsized":"2024-04-16T16:14:41.290796+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:06:15.287730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290655+0000","last_change":"2024-04-16T16:14:39.701562+0000","last_active":"2024-04-16T16:14:41.290655+0000","last_peered":"2024-04-16T16:14:41.290655+0000","last_clean":"2024-04-16T16:14:41.290655+0000","last_became_active":"2024-04-16T16:14:39.697627+0000","last_became_peered":"2024-04-16T16:14:39.697627+0000","last_unstale":"2024-04-16T16:14:41.290655+0000","last_undegraded":"2024-04-16T16:14:41.290655+0000","last_fullsized":"2024-04-16T16:14:41.290655+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:44:47.910196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685143+0000","last_change":"2024-04-16T16:14:38.664663+0000","last_active":"2024-04-16T16:14:40.685143+0000","last_peered":"2024-04-16T16:14:40.685143+0000","last_clean":"2024-04-16T16:14:40.685143+0000","last_became_active":"2024-04-16T16:14:38.663340+0000","last_became_peered":"2024-04-16T16:14:38.663340+0000","last_unstale":"2024-04-16T16:14:40.685143+0000","last_undegraded":"2024-04-16T16:14:40.685143+0000","last_fullsized":"2024-04-16T16:14:40.685143+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:31:34.396267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690125+0000","last_change":"2024-04-16T16:14:39.692510+0000","last_active":"2024-04-16T16:14:40.690125+0000","last_peered":"2024-04-16T16:14:40.690125+0000","last_clean":"2024-04-16T16:14:40.690125+0000","last_became_active":"2024-04-16T16:14:39.691697+0000","last_became_peered":"2024-04-16T16:14:39.691697+0000","last_unstale":"2024-04-16T16:14:40.690125+0000","last_undegraded":"2024-04-16T16:14:40.690125+0000","last_fullsized":"2024-04-16T16:14:40.690125+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:23.282939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996063+0000","last_change":"2024-04-16T16:14:38.672045+0000","last_active":"2024-04-16T16:14:40.996063+0000","last_peered":"2024-04-16T16:14:40.996063+0000","last_clean":"2024-04-16T16:14:40.996063+0000","last_became_active":"2024-04-16T16:14:38.671755+0000","last_became_peered":"2024-04-16T16:14:38.671755+0000","last_unstale":"2024-04-16T16:14:40.996063+0000","last_undegraded":"2024-04-16T16:14:40.996063+0000","last_fullsized":"2024-04-16T16:14:40.996063+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:46.463986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":196,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690537+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:14:40.690537+0000","last_peered":"2024-04-16T16:14:40.690537+0000","last_clean":"2024-04-16T16:14:40.690537+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:14:40.690537+0000","last_undegraded":"2024-04-16T16:14:40.690537+0000","last_fullsized":"2024-04-16T16:14:40.690537+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":983040,"data_stored":741737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":151895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":151951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":151383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":152913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":389120,"data_stored":151349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":152913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":978944,"data_stored":741647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:14:45.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T16:14:45.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.520+0000 7f04ffe00640 1 Processor -- start 2024-04-16T16:14:45.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.520+0000 7f04ffe00640 1 -- start start 2024-04-16T16:14:45.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.521+0000 7f04ffe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f050015ccb0 0x7f050006df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:45.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.521+0000 7f04ffe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f050006e480 0x7f05001608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:45.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.521+0000 7f04ffe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f050015d360 con 0x7f0500146220 2024-04-16T16:14:45.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.521+0000 7f04ffe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f050015d130 con 0x7f050006e480 2024-04-16T16:14:45.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.521+0000 7f04ffe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0500160410 con 0x7f050015ccb0 2024-04-16T16:14:45.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.522+0000 7f04ff400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f050006e480 0x7f05001608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:45.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.522+0000 7f04fea00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0500146220 0x7f050006f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42362/0 (socket says 172.21.15.45:42362) 2024-04-16T16:14:45.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.522+0000 7f04ff400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f050006e480 0x7f05001608a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50622/0 (socket says 172.21.15.45:50622) 2024-04-16T16:14:45.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.522+0000 7f04fea00640 1 -- 172.21.15.45:0/484326900 learned_addr learned my addr 172.21.15.45:0/484326900 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:45.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.522+0000 7f04ff400640 1 -- 172.21.15.45:0/484326900 >> v2:172.21.15.103:3300/0 conn(0x7f050015ccb0 msgr2=0x7f050006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:45.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.522+0000 7f04f6000640 1 --2- 172.21.15.45:0/484326900 >> v2:172.21.15.103:3300/0 conn(0x7f050015ccb0 0x7f050006df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:45.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.522+0000 7f04fe000640 1 -- 172.21.15.45:0/484326900 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 273082479 0 0) 0x7f050015d360 con 0x7f0500146220 2024-04-16T16:14:45.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.523+0000 7f04ff400640 1 --2- 172.21.15.45:0/484326900 >> v2:172.21.15.103:3300/0 conn(0x7f050015ccb0 0x7f050006df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:45.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.523+0000 7f04ff400640 1 -- 172.21.15.45:0/484326900 >> v1:172.21.15.45:6789/0 conn(0x7f0500146220 legacy=0x7f050006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:45.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.523+0000 7f04ff400640 1 -- 172.21.15.45:0/484326900 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050015d640 con 0x7f050006e480 2024-04-16T16:14:45.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.523+0000 7f04f6000640 1 --2- 172.21.15.45:0/484326900 >> v2:172.21.15.103:3300/0 conn(0x7f050015ccb0 0x7f050006df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:45.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.524+0000 7f04ff400640 1 --2- 172.21.15.45:0/484326900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f050006e480 0x7f05001608a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0500160590 tx=0x7f04f802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e3b208f7ecbbc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:45.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.524+0000 7f04fe000640 1 -- 172.21.15.45:0/484326900 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f8039070 con 0x7f050006e480 2024-04-16T16:14:45.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.524+0000 7f04fe000640 1 -- 172.21.15.45:0/484326900 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04f802cc60 con 0x7f050006e480 2024-04-16T16:14:45.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.524+0000 7f04fe000640 1 -- 172.21.15.45:0/484326900 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f8034660 con 0x7f050006e480 2024-04-16T16:14:45.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.524+0000 7f04ffe00640 1 -- 172.21.15.45:0/484326900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f050006e480 msgr2=0x7f05001608a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:45.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.524+0000 7f04ffe00640 1 --2- 172.21.15.45:0/484326900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f050006e480 0x7f05001608a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0500160590 tx=0x7f04f802beb0 comp rx=0 tx=0).stop 2024-04-16T16:14:45.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.525+0000 7f04ffe00640 1 -- 172.21.15.45:0/484326900 shutdown_connections 2024-04-16T16:14:45.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.525+0000 7f04ffe00640 1 --2- 172.21.15.45:0/484326900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f050006e480 0x7f05001608a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:45.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.525+0000 7f04ffe00640 1 --2- 172.21.15.45:0/484326900 >> v2:172.21.15.103:3300/0 conn(0x7f050015ccb0 0x7f050006df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:45.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.525+0000 7f04ffe00640 1 -- 172.21.15.45:0/484326900 >> 172.21.15.45:0/484326900 conn(0x7f050006b930 msgr2=0x7f050006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:45.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.525+0000 7f04ffe00640 1 -- 172.21.15.45:0/484326900 shutdown_connections 2024-04-16T16:14:45.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.525+0000 7f04ffe00640 1 -- 172.21.15.45:0/484326900 wait complete. 2024-04-16T16:14:45.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.527+0000 7f04ffe00640 1 Processor -- start 2024-04-16T16:14:45.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.527+0000 7f04ffe00640 1 -- start start 2024-04-16T16:14:45.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.528+0000 7f04ffe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f050006e480 0x7f05001f78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:45.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.528+0000 7f04ffe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0500146220 0x7f05000cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:45.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.528+0000 7f04ffe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f050015ccb0 0x7f0500140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:45.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.528+0000 7f04ffe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0500160de0 con 0x7f050006e480 2024-04-16T16:14:45.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.528+0000 7f04ffe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f050015dc00 con 0x7f0500146220 2024-04-16T16:14:45.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.528+0000 7f04ffe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0500160f60 con 0x7f050015ccb0 2024-04-16T16:14:45.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.528+0000 7f04fea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f050006e480 0x7f05001f78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:45.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.528+0000 7f04fea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f050006e480 0x7f05001f78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50636/0 (socket says 172.21.15.45:50636) 2024-04-16T16:14:45.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.528+0000 7f04fea00640 1 -- 172.21.15.45:0/1072351222 learned_addr learned my addr 172.21.15.45:0/1072351222 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:45.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.529+0000 7f04fea00640 1 -- 172.21.15.45:0/1072351222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f050015ccb0 msgr2=0x7f0500140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:45.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.529+0000 7f04fea00640 1 --2- 172.21.15.45:0/1072351222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f050015ccb0 0x7f0500140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:45.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.529+0000 7f04fea00640 1 -- 172.21.15.45:0/1072351222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0500146220 msgr2=0x7f05000cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:45.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.529+0000 7f04fea00640 1 --2- 172.21.15.45:0/1072351222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0500146220 0x7f05000cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:45.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.529+0000 7f04fea00640 1 -- 172.21.15.45:0/1072351222 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05001f7fd0 con 0x7f050006e480 2024-04-16T16:14:45.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.529+0000 7f04fea00640 1 --2- 172.21.15.45:0/1072351222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f050006e480 0x7f05001f78d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f04f000c4d0 tx=0x7f04f000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:45.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.529+0000 7f04f7400640 1 -- 172.21.15.45:0/1072351222 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f0018070 con 0x7f050006e480 2024-04-16T16:14:45.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.529+0000 7f04f7400640 1 -- 172.21.15.45:0/1072351222 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04f0007500 con 0x7f050006e480 2024-04-16T16:14:45.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.529+0000 7f04f7400640 1 -- 172.21.15.45:0/1072351222 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f0004a20 con 0x7f050006e480 2024-04-16T16:14:45.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.529+0000 7f04ffe00640 1 -- 172.21.15.45:0/1072351222 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05001f82b0 con 0x7f050006e480 2024-04-16T16:14:45.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.530+0000 7f04ffe00640 1 -- 172.21.15.45:0/1072351222 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05001f44f0 con 0x7f050006e480 2024-04-16T16:14:45.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.531+0000 7f04ffe00640 1 -- 172.21.15.45:0/1072351222 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05001f46f0 con 0x7f050006e480 2024-04-16T16:14:45.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.532+0000 7f04f7400640 1 -- 172.21.15.45:0/1072351222 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f04f0004bc0 con 0x7f050006e480 2024-04-16T16:14:45.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.533+0000 7f04f7400640 1 --2- 172.21.15.45:0/1072351222 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f04dc06c8f0 0x7f04dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:45.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.533+0000 7f04f7400640 1 -- 172.21.15.45:0/1072351222 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f04f008ae40 con 0x7f050006e480 2024-04-16T16:14:45.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.533+0000 7f04f6000640 1 --2- 172.21.15.45:0/1072351222 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f04dc06c8f0 0x7f04dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:45.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.535+0000 7f04f6000640 1 --2- 172.21.15.45:0/1072351222 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f04dc06c8f0 0x7f04dc06edb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f04e8006fd0 tx=0x7f04e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:45.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.537+0000 7f04f7400640 1 -- 172.21.15.45:0/1072351222 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f05001f46f0 con 0x7f050006e480 2024-04-16T16:14:45.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.822+0000 7f04ffe00640 1 -- 172.21.15.45:0/1072351222 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f05001f42e0 con 0x7f050006e480 2024-04-16T16:14:45.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.822+0000 7f04f7400640 1 -- 172.21.15.45:0/1072351222 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1173 (secure 0 0 0) 0x7f04f0059270 con 0x7f050006e480 2024-04-16T16:14:45.822 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:45.823 INFO:teuthology.orchestra.run.smithi045.stdout:{"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":99,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":8,"num_up_osds":8,"osd_up_since":1713283997,"num_in_osds":8,"osd_in_since":1713283989,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":242728960,"bytes_avail":772851384320,"bytes_total":773094113280,"write_bytes_sec":2211,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":16,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":5079}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-04-16T16:14:21.111926+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-16T16:14:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.825+0000 7f04ffe00640 1 -- 172.21.15.45:0/1072351222 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f04dc06c8f0 msgr2=0x7f04dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.825+0000 7f04ffe00640 1 --2- 172.21.15.45:0/1072351222 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f04dc06c8f0 0x7f04dc06edb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f04e8006fd0 tx=0x7f04e8008040 comp rx=0 tx=0).stop 2024-04-16T16:14:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.825+0000 7f04ffe00640 1 -- 172.21.15.45:0/1072351222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f050006e480 msgr2=0x7f05001f78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.825+0000 7f04ffe00640 1 --2- 172.21.15.45:0/1072351222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f050006e480 0x7f05001f78d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f04f000c4d0 tx=0x7f04f000c990 comp rx=0 tx=0).stop 2024-04-16T16:14:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.826+0000 7f04ffe00640 1 -- 172.21.15.45:0/1072351222 shutdown_connections 2024-04-16T16:14:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.826+0000 7f04ffe00640 1 --2- 172.21.15.45:0/1072351222 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f04dc06c8f0 0x7f04dc06edb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.826+0000 7f04ffe00640 1 --2- 172.21.15.45:0/1072351222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f050015ccb0 0x7f0500140f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.826+0000 7f04ffe00640 1 --2- 172.21.15.45:0/1072351222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0500146220 0x7f05000cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.826+0000 7f04ffe00640 1 --2- 172.21.15.45:0/1072351222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f050006e480 0x7f05001f78d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:45.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.826+0000 7f04ffe00640 1 -- 172.21.15.45:0/1072351222 >> 172.21.15.45:0/1072351222 conn(0x7f050006b930 msgr2=0x7f050015f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:45.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.826+0000 7f04ffe00640 1 -- 172.21.15.45:0/1072351222 shutdown_connections 2024-04-16T16:14:45.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:45.826+0000 7f04ffe00640 1 -- 172.21.15.45:0/1072351222 wait complete. 2024-04-16T16:14:45.847 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-16T16:14:45.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:14:46.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.072+0000 7f9b19200640 1 Processor -- start 2024-04-16T16:14:46.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.072+0000 7f9b19200640 1 -- start start 2024-04-16T16:14:46.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.074+0000 7f9b19200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9b14067c20 0x7f9b1406b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:46.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.074+0000 7f9b19200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b140b5f40 0x7f9b1406bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:46.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.074+0000 7f9b19200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b140b40f0 con 0x7f9b140b0630 2024-04-16T16:14:46.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.074+0000 7f9b19200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9b140b43d0 con 0x7f9b14067c20 2024-04-16T16:14:46.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.074+0000 7f9b19200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9b140b46b0 con 0x7f9b140b5f40 2024-04-16T16:14:46.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.074+0000 7f9b11e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9b140b0630 0x7f9b140b5830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42374/0 (socket says 172.21.15.45:42374) 2024-04-16T16:14:46.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.074+0000 7f9b11e00640 1 -- 172.21.15.45:0/3140702542 learned_addr learned my addr 172.21.15.45:0/3140702542 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:46.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.075+0000 7f9b09400640 1 --2- 172.21.15.45:0/3140702542 >> v2:172.21.15.103:3300/0 conn(0x7f9b14067c20 0x7f9b1406b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:46.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.075+0000 7f9b12800640 1 --2- 172.21.15.45:0/3140702542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b140b5f40 0x7f9b1406bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.075+0000 7f9b11400640 1 -- 172.21.15.45:0/3140702542 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4249994516 0 0) 0x7f9b140b40f0 con 0x7f9b140b0630 2024-04-16T16:14:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.075+0000 7f9b11400640 1 -- 172.21.15.45:0/3140702542 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9af8003520 con 0x7f9b140b0630 2024-04-16T16:14:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.075+0000 7f9b11400640 1 -- 172.21.15.45:0/3140702542 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2809089194 0 0) 0x7f9af8003520 con 0x7f9b140b0630 2024-04-16T16:14:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.075+0000 7f9b11400640 1 -- 172.21.15.45:0/3140702542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b140b5f40 msgr2=0x7f9b1406bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.075+0000 7f9b11400640 1 --2- 172.21.15.45:0/3140702542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b140b5f40 0x7f9b1406bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.075+0000 7f9b11400640 1 -- 172.21.15.45:0/3140702542 >> v2:172.21.15.103:3300/0 conn(0x7f9b14067c20 msgr2=0x7f9b1406b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.075+0000 7f9b11400640 1 --2- 172.21.15.45:0/3140702542 >> v2:172.21.15.103:3300/0 conn(0x7f9b14067c20 0x7f9b1406b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.075+0000 7f9b11400640 1 -- 172.21.15.45:0/3140702542 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b140b4990 con 0x7f9b140b0630 2024-04-16T16:14:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.075+0000 7f9b11400640 1 -- 172.21.15.45:0/3140702542 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f9b04001670 con 0x7f9b140b0630 2024-04-16T16:14:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.075+0000 7f9b11400640 1 -- 172.21.15.45:0/3140702542 >> v1:172.21.15.45:6789/0 conn(0x7f9b140b0630 legacy=0x7f9b140b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.075+0000 7f9b11400640 1 --2- 172.21.15.45:0/3140702542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af80039b0 0x7f9af8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.075+0000 7f9b11400640 1 --2- 172.21.15.45:0/3140702542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b140b5f40 0x7f9af8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.076+0000 7f9b11400640 1 --2- 172.21.15.45:0/3140702542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af8004d40 0x7f9af80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.076+0000 7f9b11400640 1 -- 172.21.15.45:0/3140702542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b140b43d0 con 0x7f9af80039b0 2024-04-16T16:14:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.076+0000 7f9b11400640 1 -- 172.21.15.45:0/3140702542 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9b140b46b0 con 0x7f9af8004d40 2024-04-16T16:14:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.076+0000 7f9b11400640 1 -- 172.21.15.45:0/3140702542 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9af800b9c0 con 0x7f9b140b5f40 2024-04-16T16:14:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.076+0000 7f9b11e00640 1 --2- 172.21.15.45:0/3140702542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af80039b0 0x7f9af8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.076+0000 7f9b09400640 1 --2- 172.21.15.45:0/3140702542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b140b5f40 0x7f9af8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.076+0000 7f9b12800640 1 --2- 172.21.15.45:0/3140702542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af8004d40 0x7f9af80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.076+0000 7f9b11e00640 1 -- 172.21.15.45:0/3140702542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b140b5f40 msgr2=0x7f9af8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.076+0000 7f9b11e00640 1 --2- 172.21.15.45:0/3140702542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b140b5f40 0x7f9af8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.076+0000 7f9b11e00640 1 -- 172.21.15.45:0/3140702542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af8004d40 msgr2=0x7f9af80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.076+0000 7f9b11e00640 1 --2- 172.21.15.45:0/3140702542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af8004d40 0x7f9af80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.076+0000 7f9b11e00640 1 -- 172.21.15.45:0/3140702542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af8003520 con 0x7f9af80039b0 2024-04-16T16:14:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.076+0000 7f9b12800640 1 --2- 172.21.15.45:0/3140702542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af8004d40 0x7f9af80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:46.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.077+0000 7f9b11e00640 1 --2- 172.21.15.45:0/3140702542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af80039b0 0x7f9af8003dd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9b0400c290 tx=0x7f9af8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c71ebeac4a787e74 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.078+0000 7f9b11400640 1 -- 172.21.15.45:0/3140702542 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b04007d50 con 0x7f9af80039b0 2024-04-16T16:14:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.078+0000 7f9b11400640 1 -- 172.21.15.45:0/3140702542 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b04030070 con 0x7f9af80039b0 2024-04-16T16:14:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.078+0000 7f9b11400640 1 -- 172.21.15.45:0/3140702542 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0402cd80 con 0x7f9af80039b0 2024-04-16T16:14:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.078+0000 7f9b19200640 1 -- 172.21.15.45:0/3140702542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af80039b0 msgr2=0x7f9af8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.078+0000 7f9b19200640 1 --2- 172.21.15.45:0/3140702542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af80039b0 0x7f9af8003dd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9b0400c290 tx=0x7f9af8005830 comp rx=0 tx=0).stop 2024-04-16T16:14:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.078+0000 7f9b12800640 1 -- 172.21.15.45:0/3140702542 reap_dead start 2024-04-16T16:14:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.079+0000 7f9b19200640 1 -- 172.21.15.45:0/3140702542 shutdown_connections 2024-04-16T16:14:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.079+0000 7f9b19200640 1 -- 172.21.15.45:0/3140702542 >> 172.21.15.45:0/3140702542 conn(0x7f9b1406e9b0 msgr2=0x7f9b1406edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.079+0000 7f9b19200640 1 -- 172.21.15.45:0/3140702542 shutdown_connections 2024-04-16T16:14:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.079+0000 7f9b19200640 1 -- 172.21.15.45:0/3140702542 wait complete. 2024-04-16T16:14:46.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.080+0000 7f9b19200640 1 Processor -- start 2024-04-16T16:14:46.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.081+0000 7f9b19200640 1 -- start start 2024-04-16T16:14:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.082+0000 7f9b19200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af80039b0 0x7f9b14151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.082+0000 7f9b19200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b1414e380 0x7f9b140cea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.082+0000 7f9b19200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b140cef80 0x7f9b1414e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.082+0000 7f9b19200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b140680f0 con 0x7f9af80039b0 2024-04-16T16:14:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.082+0000 7f9b19200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9b140b0a10 con 0x7f9b140cef80 2024-04-16T16:14:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.082+0000 7f9b19200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9b14068270 con 0x7f9b1414e380 2024-04-16T16:14:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.082+0000 7f9b11e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af80039b0 0x7f9b14151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.082+0000 7f9b11e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af80039b0 0x7f9b14151a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50658/0 (socket says 172.21.15.45:50658) 2024-04-16T16:14:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.082+0000 7f9b11e00640 1 -- 172.21.15.45:0/2457255137 learned_addr learned my addr 172.21.15.45:0/2457255137 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.082+0000 7f9b11e00640 1 -- 172.21.15.45:0/2457255137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b1414e380 msgr2=0x7f9b140cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.082+0000 7f9b11e00640 1 --2- 172.21.15.45:0/2457255137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b1414e380 0x7f9b140cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.082+0000 7f9b11e00640 1 -- 172.21.15.45:0/2457255137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b140cef80 msgr2=0x7f9b1414e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.082+0000 7f9b11e00640 1 --2- 172.21.15.45:0/2457255137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b140cef80 0x7f9b1414e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.082+0000 7f9b11e00640 1 -- 172.21.15.45:0/2457255137 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b14158110 con 0x7f9af80039b0 2024-04-16T16:14:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.083+0000 7f9b11e00640 1 --2- 172.21.15.45:0/2457255137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af80039b0 0x7f9b14151a20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9b04001970 tx=0x7f9b140b4b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.083+0000 7f9b0aa00640 1 -- 172.21.15.45:0/2457255137 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b04001c80 con 0x7f9af80039b0 2024-04-16T16:14:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.083+0000 7f9b0aa00640 1 -- 172.21.15.45:0/2457255137 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b04004bf0 con 0x7f9af80039b0 2024-04-16T16:14:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.083+0000 7f9b19200640 1 -- 172.21.15.45:0/2457255137 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b14152530 con 0x7f9af80039b0 2024-04-16T16:14:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.083+0000 7f9b0aa00640 1 -- 172.21.15.45:0/2457255137 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b04007d50 con 0x7f9af80039b0 2024-04-16T16:14:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.083+0000 7f9b19200640 1 -- 172.21.15.45:0/2457255137 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b14154fa0 con 0x7f9af80039b0 2024-04-16T16:14:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.084+0000 7f9b19200640 1 -- 172.21.15.45:0/2457255137 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b14075780 con 0x7f9af80039b0 2024-04-16T16:14:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.085+0000 7f9b0aa00640 1 -- 172.21.15.45:0/2457255137 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9b04047020 con 0x7f9af80039b0 2024-04-16T16:14:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.086+0000 7f9b0aa00640 1 --2- 172.21.15.45:0/2457255137 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9af006c920 0x7f9af006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.086+0000 7f9b0aa00640 1 -- 172.21.15.45:0/2457255137 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f9b040ace50 con 0x7f9af80039b0 2024-04-16T16:14:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.087+0000 7f9b09400640 1 --2- 172.21.15.45:0/2457255137 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9af006c920 0x7f9af006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:46.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.088+0000 7f9b09400640 1 --2- 172.21.15.45:0/2457255137 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9af006c920 0x7f9af006ede0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9b1406fa60 tx=0x7f9afc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:46.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.091+0000 7f9b0aa00640 1 -- 172.21.15.45:0/2457255137 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9b0407b150 con 0x7f9af80039b0 2024-04-16T16:14:46.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.291+0000 7f9b19200640 1 -- 172.21.15.45:0/2457255137 --> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9b14152b80 con 0x7f9af006c920 2024-04-16T16:14:46.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.310+0000 7f9b0aa00640 1 -- 172.21.15.45:0/2457255137 <== mgr.4105 v2:172.21.15.103:6832/2711977587 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251950 (secure 0 0 0) 0x7f9b14152b80 con 0x7f9af006c920 2024-04-16T16:14:46.314 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:46.315 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:14:46.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.316+0000 7f9b19200640 1 -- 172.21.15.45:0/2457255137 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9af006c920 msgr2=0x7f9af006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:46.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.316+0000 7f9b19200640 1 --2- 172.21.15.45:0/2457255137 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9af006c920 0x7f9af006ede0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9b1406fa60 tx=0x7f9afc008040 comp rx=0 tx=0).stop 2024-04-16T16:14:46.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.316+0000 7f9b19200640 1 -- 172.21.15.45:0/2457255137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af80039b0 msgr2=0x7f9b14151a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:46.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.316+0000 7f9b19200640 1 --2- 172.21.15.45:0/2457255137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af80039b0 0x7f9b14151a20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9b04001970 tx=0x7f9b140b4b60 comp rx=0 tx=0).stop 2024-04-16T16:14:46.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.316+0000 7f9b19200640 1 -- 172.21.15.45:0/2457255137 shutdown_connections 2024-04-16T16:14:46.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.316+0000 7f9b19200640 1 --2- 172.21.15.45:0/2457255137 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f9af006c920 0x7f9af006ede0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:46.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.316+0000 7f9b19200640 1 --2- 172.21.15.45:0/2457255137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b140cef80 0x7f9b1414e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:46.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.316+0000 7f9b19200640 1 --2- 172.21.15.45:0/2457255137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b1414e380 0x7f9b140cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:46.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.316+0000 7f9b19200640 1 --2- 172.21.15.45:0/2457255137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af80039b0 0x7f9b14151a20 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:46.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.316+0000 7f9b19200640 1 -- 172.21.15.45:0/2457255137 >> 172.21.15.45:0/2457255137 conn(0x7f9b1406e9b0 msgr2=0x7f9b140b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:46.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.316+0000 7f9b19200640 1 -- 172.21.15.45:0/2457255137 shutdown_connections 2024-04-16T16:14:46.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:46.316+0000 7f9b19200640 1 -- 172.21.15.45:0/2457255137 wait complete. 2024-04-16T16:14:46.338 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-04-16T16:14:45.119527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":158,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237040,"kb_used_data":4232,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737680,"statfs":{"total":773094113280,"available":772851384320,"internally_reserved":0,"allocated":4333568,"data_stored":2395788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.481590"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996516+0000","last_change":"2024-04-16T16:14:39.708878+0000","last_active":"2024-04-16T16:14:40.996516+0000","last_peered":"2024-04-16T16:14:40.996516+0000","last_clean":"2024-04-16T16:14:40.996516+0000","last_became_active":"2024-04-16T16:14:39.708146+0000","last_became_peered":"2024-04-16T16:14:39.708146+0000","last_unstale":"2024-04-16T16:14:40.996516+0000","last_undegraded":"2024-04-16T16:14:40.996516+0000","last_fullsized":"2024-04-16T16:14:40.996516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:24:14.006571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.686276+0000","last_change":"2024-04-16T16:14:39.697128+0000","last_active":"2024-04-16T16:14:40.686276+0000","last_peered":"2024-04-16T16:14:40.686276+0000","last_clean":"2024-04-16T16:14:40.686276+0000","last_became_active":"2024-04-16T16:14:39.696806+0000","last_became_peered":"2024-04-16T16:14:39.696806+0000","last_unstale":"2024-04-16T16:14:40.686276+0000","last_undegraded":"2024-04-16T16:14:40.686276+0000","last_fullsized":"2024-04-16T16:14:40.686276+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:55:56.042534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691181+0000","last_change":"2024-04-16T16:14:39.707270+0000","last_active":"2024-04-16T16:14:40.691181+0000","last_peered":"2024-04-16T16:14:40.691181+0000","last_clean":"2024-04-16T16:14:40.691181+0000","last_became_active":"2024-04-16T16:14:39.706592+0000","last_became_peered":"2024-04-16T16:14:39.706592+0000","last_unstale":"2024-04-16T16:14:40.691181+0000","last_undegraded":"2024-04-16T16:14:40.691181+0000","last_fullsized":"2024-04-16T16:14:40.691181+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:10:23.378754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291332+0000","last_change":"2024-04-16T16:14:39.704396+0000","last_active":"2024-04-16T16:14:41.291332+0000","last_peered":"2024-04-16T16:14:41.291332+0000","last_clean":"2024-04-16T16:14:41.291332+0000","last_became_active":"2024-04-16T16:14:39.703644+0000","last_became_peered":"2024-04-16T16:14:39.703644+0000","last_unstale":"2024-04-16T16:14:41.291332+0000","last_undegraded":"2024-04-16T16:14:41.291332+0000","last_fullsized":"2024-04-16T16:14:41.291332+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:28:27.889354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004853+0000","last_change":"2024-04-16T16:14:39.691482+0000","last_active":"2024-04-16T16:14:41.004853+0000","last_peered":"2024-04-16T16:14:41.004853+0000","last_clean":"2024-04-16T16:14:41.004853+0000","last_became_active":"2024-04-16T16:14:39.690475+0000","last_became_peered":"2024-04-16T16:14:39.690475+0000","last_unstale":"2024-04-16T16:14:41.004853+0000","last_undegraded":"2024-04-16T16:14:41.004853+0000","last_fullsized":"2024-04-16T16:14:41.004853+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:29:05.538981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685032+0000","last_change":"2024-04-16T16:14:39.707402+0000","last_active":"2024-04-16T16:14:40.685032+0000","last_peered":"2024-04-16T16:14:40.685032+0000","last_clean":"2024-04-16T16:14:40.685032+0000","last_became_active":"2024-04-16T16:14:39.707125+0000","last_became_peered":"2024-04-16T16:14:39.707125+0000","last_unstale":"2024-04-16T16:14:40.685032+0000","last_undegraded":"2024-04-16T16:14:40.685032+0000","last_fullsized":"2024-04-16T16:14:40.685032+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:24:24.985004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690212+0000","last_change":"2024-04-16T16:14:39.696478+0000","last_active":"2024-04-16T16:14:40.690212+0000","last_peered":"2024-04-16T16:14:40.690212+0000","last_clean":"2024-04-16T16:14:40.690212+0000","last_became_active":"2024-04-16T16:14:39.696171+0000","last_became_peered":"2024-04-16T16:14:39.696171+0000","last_unstale":"2024-04-16T16:14:40.690212+0000","last_undegraded":"2024-04-16T16:14:40.690212+0000","last_fullsized":"2024-04-16T16:14:40.690212+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:35.916987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.689394+0000","last_change":"2024-04-16T16:14:39.686048+0000","last_active":"2024-04-16T16:14:40.689394+0000","last_peered":"2024-04-16T16:14:40.689394+0000","last_clean":"2024-04-16T16:14:40.689394+0000","last_became_active":"2024-04-16T16:14:39.682978+0000","last_became_peered":"2024-04-16T16:14:39.682978+0000","last_unstale":"2024-04-16T16:14:40.689394+0000","last_undegraded":"2024-04-16T16:14:40.689394+0000","last_fullsized":"2024-04-16T16:14:40.689394+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:16:57.270160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292049+0000","last_change":"2024-04-16T16:14:39.701196+0000","last_active":"2024-04-16T16:14:41.292049+0000","last_peered":"2024-04-16T16:14:41.292049+0000","last_clean":"2024-04-16T16:14:41.292049+0000","last_became_active":"2024-04-16T16:14:39.700442+0000","last_became_peered":"2024-04-16T16:14:39.700442+0000","last_unstale":"2024-04-16T16:14:41.292049+0000","last_undegraded":"2024-04-16T16:14:41.292049+0000","last_fullsized":"2024-04-16T16:14:41.292049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:54.548210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291929+0000","last_change":"2024-04-16T16:14:39.707318+0000","last_active":"2024-04-16T16:14:41.291929+0000","last_peered":"2024-04-16T16:14:41.291929+0000","last_clean":"2024-04-16T16:14:41.291929+0000","last_became_active":"2024-04-16T16:14:39.706939+0000","last_became_peered":"2024-04-16T16:14:39.706939+0000","last_unstale":"2024-04-16T16:14:41.291929+0000","last_undegraded":"2024-04-16T16:14:41.291929+0000","last_fullsized":"2024-04-16T16:14:41.291929+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:23.282939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691664+0000","last_change":"2024-04-16T16:14:39.702686+0000","last_active":"2024-04-16T16:14:40.691664+0000","last_peered":"2024-04-16T16:14:40.691664+0000","last_clean":"2024-04-16T16:14:40.691664+0000","last_became_active":"2024-04-16T16:14:39.702274+0000","last_became_peered":"2024-04-16T16:14:39.702274+0000","last_unstale":"2024-04-16T16:14:40.691664+0000","last_undegraded":"2024-04-16T16:14:40.691664+0000","last_fullsized":"2024-04-16T16:14:40.691664+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:10:23.378754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690786+0000","last_change":"2024-04-16T16:14:39.709482+0000","last_active":"2024-04-16T16:14:40.690786+0000","last_peered":"2024-04-16T16:14:40.690786+0000","last_clean":"2024-04-16T16:14:40.690786+0000","last_became_active":"2024-04-16T16:14:39.709110+0000","last_became_peered":"2024-04-16T16:14:39.709110+0000","last_unstale":"2024-04-16T16:14:40.690786+0000","last_undegraded":"2024-04-16T16:14:40.690786+0000","last_fullsized":"2024-04-16T16:14:40.690786+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:31.834009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004919+0000","last_change":"2024-04-16T16:14:39.697790+0000","last_active":"2024-04-16T16:14:41.004919+0000","last_peered":"2024-04-16T16:14:41.004919+0000","last_clean":"2024-04-16T16:14:41.004919+0000","last_became_active":"2024-04-16T16:14:39.696862+0000","last_became_peered":"2024-04-16T16:14:39.696862+0000","last_unstale":"2024-04-16T16:14:41.004919+0000","last_undegraded":"2024-04-16T16:14:41.004919+0000","last_fullsized":"2024-04-16T16:14:41.004919+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:09:31.099716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685180+0000","last_change":"2024-04-16T16:14:39.691105+0000","last_active":"2024-04-16T16:14:40.685180+0000","last_peered":"2024-04-16T16:14:40.685180+0000","last_clean":"2024-04-16T16:14:40.685180+0000","last_became_active":"2024-04-16T16:14:39.690771+0000","last_became_peered":"2024-04-16T16:14:39.690771+0000","last_unstale":"2024-04-16T16:14:40.685180+0000","last_undegraded":"2024-04-16T16:14:40.685180+0000","last_fullsized":"2024-04-16T16:14:40.685180+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:10:23.378754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690774+0000","last_change":"2024-04-16T16:14:39.707728+0000","last_active":"2024-04-16T16:14:40.690774+0000","last_peered":"2024-04-16T16:14:40.690774+0000","last_clean":"2024-04-16T16:14:40.690774+0000","last_became_active":"2024-04-16T16:14:39.707447+0000","last_became_peered":"2024-04-16T16:14:39.707447+0000","last_unstale":"2024-04-16T16:14:40.690774+0000","last_undegraded":"2024-04-16T16:14:40.690774+0000","last_fullsized":"2024-04-16T16:14:40.690774+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:13.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691686+0000","last_change":"2024-04-16T16:14:39.689522+0000","last_active":"2024-04-16T16:14:40.691686+0000","last_peered":"2024-04-16T16:14:40.691686+0000","last_clean":"2024-04-16T16:14:40.691686+0000","last_became_active":"2024-04-16T16:14:39.685321+0000","last_became_peered":"2024-04-16T16:14:39.685321+0000","last_unstale":"2024-04-16T16:14:40.691686+0000","last_undegraded":"2024-04-16T16:14:40.691686+0000","last_fullsized":"2024-04-16T16:14:40.691686+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:23:31.999680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.692050+0000","last_change":"2024-04-16T16:14:39.699366+0000","last_active":"2024-04-16T16:14:40.692050+0000","last_peered":"2024-04-16T16:14:40.692050+0000","last_clean":"2024-04-16T16:14:40.692050+0000","last_became_active":"2024-04-16T16:14:39.697476+0000","last_became_peered":"2024-04-16T16:14:39.697476+0000","last_unstale":"2024-04-16T16:14:40.692050+0000","last_undegraded":"2024-04-16T16:14:40.692050+0000","last_fullsized":"2024-04-16T16:14:40.692050+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:51:43.656920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.686048+0000","last_change":"2024-04-16T16:14:39.710331+0000","last_active":"2024-04-16T16:14:40.686048+0000","last_peered":"2024-04-16T16:14:40.686048+0000","last_clean":"2024-04-16T16:14:40.686048+0000","last_became_active":"2024-04-16T16:14:39.710065+0000","last_became_peered":"2024-04-16T16:14:39.710065+0000","last_unstale":"2024-04-16T16:14:40.686048+0000","last_undegraded":"2024-04-16T16:14:40.686048+0000","last_fullsized":"2024-04-16T16:14:40.686048+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:37:58.914102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685635+0000","last_change":"2024-04-16T16:14:39.700033+0000","last_active":"2024-04-16T16:14:40.685635+0000","last_peered":"2024-04-16T16:14:40.685635+0000","last_clean":"2024-04-16T16:14:40.685635+0000","last_became_active":"2024-04-16T16:14:39.699473+0000","last_became_peered":"2024-04-16T16:14:39.699473+0000","last_unstale":"2024-04-16T16:14:40.685635+0000","last_undegraded":"2024-04-16T16:14:40.685635+0000","last_fullsized":"2024-04-16T16:14:40.685635+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:51:43.656920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004721+0000","last_change":"2024-04-16T16:14:39.713504+0000","last_active":"2024-04-16T16:14:41.004721+0000","last_peered":"2024-04-16T16:14:41.004721+0000","last_clean":"2024-04-16T16:14:41.004721+0000","last_became_active":"2024-04-16T16:14:39.711524+0000","last_became_peered":"2024-04-16T16:14:39.711524+0000","last_unstale":"2024-04-16T16:14:41.004721+0000","last_undegraded":"2024-04-16T16:14:41.004721+0000","last_fullsized":"2024-04-16T16:14:41.004721+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:06:16.299983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290756+0000","last_change":"2024-04-16T16:14:39.698857+0000","last_active":"2024-04-16T16:14:41.290756+0000","last_peered":"2024-04-16T16:14:41.290756+0000","last_clean":"2024-04-16T16:14:41.290756+0000","last_became_active":"2024-04-16T16:14:39.693200+0000","last_became_peered":"2024-04-16T16:14:39.693200+0000","last_unstale":"2024-04-16T16:14:41.290756+0000","last_undegraded":"2024-04-16T16:14:41.290756+0000","last_fullsized":"2024-04-16T16:14:41.290756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:17.454131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290778+0000","last_change":"2024-04-16T16:14:39.694980+0000","last_active":"2024-04-16T16:14:41.290778+0000","last_peered":"2024-04-16T16:14:41.290778+0000","last_clean":"2024-04-16T16:14:41.290778+0000","last_became_active":"2024-04-16T16:14:39.691011+0000","last_became_peered":"2024-04-16T16:14:39.691011+0000","last_unstale":"2024-04-16T16:14:41.290778+0000","last_undegraded":"2024-04-16T16:14:41.290778+0000","last_fullsized":"2024-04-16T16:14:41.290778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:24:14.006571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.726768+0000","last_change":"2024-04-16T16:14:39.687391+0000","last_active":"2024-04-16T16:14:40.726768+0000","last_peered":"2024-04-16T16:14:40.726768+0000","last_clean":"2024-04-16T16:14:40.726768+0000","last_became_active":"2024-04-16T16:14:39.687092+0000","last_became_peered":"2024-04-16T16:14:39.687092+0000","last_unstale":"2024-04-16T16:14:40.726768+0000","last_undegraded":"2024-04-16T16:14:40.726768+0000","last_fullsized":"2024-04-16T16:14:40.726768+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:31:35.408519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.689686+0000","last_change":"2024-04-16T16:14:39.707655+0000","last_active":"2024-04-16T16:14:40.689686+0000","last_peered":"2024-04-16T16:14:40.689686+0000","last_clean":"2024-04-16T16:14:40.689686+0000","last_became_active":"2024-04-16T16:14:39.707273+0000","last_became_peered":"2024-04-16T16:14:39.707273+0000","last_unstale":"2024-04-16T16:14:40.689686+0000","last_undegraded":"2024-04-16T16:14:40.689686+0000","last_fullsized":"2024-04-16T16:14:40.689686+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:23.282939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292509+0000","last_change":"2024-04-16T16:14:39.685869+0000","last_active":"2024-04-16T16:14:41.292509+0000","last_peered":"2024-04-16T16:14:41.292509+0000","last_clean":"2024-04-16T16:14:41.292509+0000","last_became_active":"2024-04-16T16:14:39.685394+0000","last_became_peered":"2024-04-16T16:14:39.685394+0000","last_unstale":"2024-04-16T16:14:41.292509+0000","last_undegraded":"2024-04-16T16:14:41.292509+0000","last_fullsized":"2024-04-16T16:14:41.292509+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:31.834009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.686198+0000","last_change":"2024-04-16T16:14:39.696275+0000","last_active":"2024-04-16T16:14:40.686198+0000","last_peered":"2024-04-16T16:14:40.686198+0000","last_clean":"2024-04-16T16:14:40.686198+0000","last_became_active":"2024-04-16T16:14:39.695976+0000","last_became_peered":"2024-04-16T16:14:39.695976+0000","last_unstale":"2024-04-16T16:14:40.686198+0000","last_undegraded":"2024-04-16T16:14:40.686198+0000","last_fullsized":"2024-04-16T16:14:40.686198+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:46:36.132077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292347+0000","last_change":"2024-04-16T16:14:39.689971+0000","last_active":"2024-04-16T16:14:41.292347+0000","last_peered":"2024-04-16T16:14:41.292347+0000","last_clean":"2024-04-16T16:14:41.292347+0000","last_became_active":"2024-04-16T16:14:39.689482+0000","last_became_peered":"2024-04-16T16:14:39.689482+0000","last_unstale":"2024-04-16T16:14:41.292347+0000","last_undegraded":"2024-04-16T16:14:41.292347+0000","last_fullsized":"2024-04-16T16:14:41.292347+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:35.916987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691169+0000","last_change":"2024-04-16T16:14:39.701188+0000","last_active":"2024-04-16T16:14:40.691169+0000","last_peered":"2024-04-16T16:14:40.691169+0000","last_clean":"2024-04-16T16:14:40.691169+0000","last_became_active":"2024-04-16T16:14:39.697558+0000","last_became_peered":"2024-04-16T16:14:39.697558+0000","last_unstale":"2024-04-16T16:14:40.691169+0000","last_undegraded":"2024-04-16T16:14:40.691169+0000","last_fullsized":"2024-04-16T16:14:40.691169+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:24:24.985004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290891+0000","last_change":"2024-04-16T16:14:39.702421+0000","last_active":"2024-04-16T16:14:41.290891+0000","last_peered":"2024-04-16T16:14:41.290891+0000","last_clean":"2024-04-16T16:14:41.290891+0000","last_became_active":"2024-04-16T16:14:39.701406+0000","last_became_peered":"2024-04-16T16:14:39.701406+0000","last_unstale":"2024-04-16T16:14:41.290891+0000","last_undegraded":"2024-04-16T16:14:41.290891+0000","last_fullsized":"2024-04-16T16:14:41.290891+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:45:31.844610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690802+0000","last_change":"2024-04-16T16:14:39.700536+0000","last_active":"2024-04-16T16:14:40.690802+0000","last_peered":"2024-04-16T16:14:40.690802+0000","last_clean":"2024-04-16T16:14:40.690802+0000","last_became_active":"2024-04-16T16:14:39.697778+0000","last_became_peered":"2024-04-16T16:14:39.697778+0000","last_unstale":"2024-04-16T16:14:40.690802+0000","last_undegraded":"2024-04-16T16:14:40.690802+0000","last_fullsized":"2024-04-16T16:14:40.690802+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:31.834009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996750+0000","last_change":"2024-04-16T16:14:39.698727+0000","last_active":"2024-04-16T16:14:40.996750+0000","last_peered":"2024-04-16T16:14:40.996750+0000","last_clean":"2024-04-16T16:14:40.996750+0000","last_became_active":"2024-04-16T16:14:39.696171+0000","last_became_peered":"2024-04-16T16:14:39.696171+0000","last_unstale":"2024-04-16T16:14:40.996750+0000","last_undegraded":"2024-04-16T16:14:40.996750+0000","last_fullsized":"2024-04-16T16:14:40.996750+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:33.368730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.689600+0000","last_change":"2024-04-16T16:14:39.694912+0000","last_active":"2024-04-16T16:14:40.689600+0000","last_peered":"2024-04-16T16:14:40.689600+0000","last_clean":"2024-04-16T16:14:40.689600+0000","last_became_active":"2024-04-16T16:14:39.694584+0000","last_became_peered":"2024-04-16T16:14:39.694584+0000","last_unstale":"2024-04-16T16:14:40.689600+0000","last_undegraded":"2024-04-16T16:14:40.689600+0000","last_fullsized":"2024-04-16T16:14:40.689600+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:01:01.716267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.727191+0000","last_change":"2024-04-16T16:14:39.681803+0000","last_active":"2024-04-16T16:14:40.727191+0000","last_peered":"2024-04-16T16:14:40.727191+0000","last_clean":"2024-04-16T16:14:40.727191+0000","last_became_active":"2024-04-16T16:14:39.681310+0000","last_became_peered":"2024-04-16T16:14:39.681310+0000","last_unstale":"2024-04-16T16:14:40.727191+0000","last_undegraded":"2024-04-16T16:14:40.727191+0000","last_fullsized":"2024-04-16T16:14:40.727191+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:54.548210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996669+0000","last_change":"2024-04-16T16:14:38.671801+0000","last_active":"2024-04-16T16:14:40.996669+0000","last_peered":"2024-04-16T16:14:40.996669+0000","last_clean":"2024-04-16T16:14:40.996669+0000","last_became_active":"2024-04-16T16:14:38.671502+0000","last_became_peered":"2024-04-16T16:14:38.671502+0000","last_unstale":"2024-04-16T16:14:40.996669+0000","last_undegraded":"2024-04-16T16:14:40.996669+0000","last_fullsized":"2024-04-16T16:14:40.996669+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:19:47.312068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996555+0000","last_change":"2024-04-16T16:14:39.708496+0000","last_active":"2024-04-16T16:14:40.996555+0000","last_peered":"2024-04-16T16:14:40.996555+0000","last_clean":"2024-04-16T16:14:40.996555+0000","last_became_active":"2024-04-16T16:14:39.707725+0000","last_became_peered":"2024-04-16T16:14:39.707725+0000","last_unstale":"2024-04-16T16:14:40.996555+0000","last_undegraded":"2024-04-16T16:14:40.996555+0000","last_fullsized":"2024-04-16T16:14:40.996555+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:31:35.408519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.727037+0000","last_change":"2024-04-16T16:14:38.663415+0000","last_active":"2024-04-16T16:14:40.727037+0000","last_peered":"2024-04-16T16:14:40.727037+0000","last_clean":"2024-04-16T16:14:40.727037+0000","last_became_active":"2024-04-16T16:14:38.662731+0000","last_became_peered":"2024-04-16T16:14:38.662731+0000","last_unstale":"2024-04-16T16:14:40.727037+0000","last_undegraded":"2024-04-16T16:14:40.727037+0000","last_fullsized":"2024-04-16T16:14:40.727037+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:39:10.432469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291543+0000","last_change":"2024-04-16T16:14:39.706462+0000","last_active":"2024-04-16T16:14:41.291543+0000","last_peered":"2024-04-16T16:14:41.291543+0000","last_clean":"2024-04-16T16:14:41.291543+0000","last_became_active":"2024-04-16T16:14:39.705749+0000","last_became_peered":"2024-04-16T16:14:39.705749+0000","last_unstale":"2024-04-16T16:14:41.291543+0000","last_undegraded":"2024-04-16T16:14:41.291543+0000","last_fullsized":"2024-04-16T16:14:41.291543+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:33.368730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685349+0000","last_change":"2024-04-16T16:14:38.668822+0000","last_active":"2024-04-16T16:14:40.685349+0000","last_peered":"2024-04-16T16:14:40.685349+0000","last_clean":"2024-04-16T16:14:40.685349+0000","last_became_active":"2024-04-16T16:14:38.668018+0000","last_became_peered":"2024-04-16T16:14:38.668018+0000","last_unstale":"2024-04-16T16:14:40.685349+0000","last_undegraded":"2024-04-16T16:14:40.685349+0000","last_fullsized":"2024-04-16T16:14:40.685349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:16.441878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292426+0000","last_change":"2024-04-16T16:14:39.689746+0000","last_active":"2024-04-16T16:14:41.292426+0000","last_peered":"2024-04-16T16:14:41.292426+0000","last_clean":"2024-04-16T16:14:41.292426+0000","last_became_active":"2024-04-16T16:14:39.689162+0000","last_became_peered":"2024-04-16T16:14:39.689162+0000","last_unstale":"2024-04-16T16:14:41.292426+0000","last_undegraded":"2024-04-16T16:14:41.292426+0000","last_fullsized":"2024-04-16T16:14:41.292426+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:13.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004846+0000","last_change":"2024-04-16T16:14:38.673943+0000","last_active":"2024-04-16T16:14:41.004846+0000","last_peered":"2024-04-16T16:14:41.004846+0000","last_clean":"2024-04-16T16:14:41.004846+0000","last_became_active":"2024-04-16T16:14:38.672857+0000","last_became_peered":"2024-04-16T16:14:38.672857+0000","last_unstale":"2024-04-16T16:14:41.004846+0000","last_undegraded":"2024-04-16T16:14:41.004846+0000","last_fullsized":"2024-04-16T16:14:41.004846+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:38:56.751996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691157+0000","last_change":"2024-04-16T16:14:39.704670+0000","last_active":"2024-04-16T16:14:40.691157+0000","last_peered":"2024-04-16T16:14:40.691157+0000","last_clean":"2024-04-16T16:14:40.691157+0000","last_became_active":"2024-04-16T16:14:39.704321+0000","last_became_peered":"2024-04-16T16:14:39.704321+0000","last_unstale":"2024-04-16T16:14:40.691157+0000","last_undegraded":"2024-04-16T16:14:40.691157+0000","last_fullsized":"2024-04-16T16:14:40.691157+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:24:24.985004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290894+0000","last_change":"2024-04-16T16:14:38.662314+0000","last_active":"2024-04-16T16:14:41.290894+0000","last_peered":"2024-04-16T16:14:41.290894+0000","last_clean":"2024-04-16T16:14:41.290894+0000","last_became_active":"2024-04-16T16:14:38.662011+0000","last_became_peered":"2024-04-16T16:14:38.662011+0000","last_unstale":"2024-04-16T16:14:41.290894+0000","last_undegraded":"2024-04-16T16:14:41.290894+0000","last_fullsized":"2024-04-16T16:14:41.290894+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:49.878355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691057+0000","last_change":"2024-04-16T16:14:39.697339+0000","last_active":"2024-04-16T16:14:40.691057+0000","last_peered":"2024-04-16T16:14:40.691057+0000","last_clean":"2024-04-16T16:14:40.691057+0000","last_became_active":"2024-04-16T16:14:39.697037+0000","last_became_peered":"2024-04-16T16:14:39.697037+0000","last_unstale":"2024-04-16T16:14:40.691057+0000","last_undegraded":"2024-04-16T16:14:40.691057+0000","last_fullsized":"2024-04-16T16:14:40.691057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:27:36.166180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290724+0000","last_change":"2024-04-16T16:14:38.660477+0000","last_active":"2024-04-16T16:14:41.290724+0000","last_peered":"2024-04-16T16:14:41.290724+0000","last_clean":"2024-04-16T16:14:41.290724+0000","last_became_active":"2024-04-16T16:14:38.660156+0000","last_became_peered":"2024-04-16T16:14:38.660156+0000","last_unstale":"2024-04-16T16:14:41.290724+0000","last_undegraded":"2024-04-16T16:14:41.290724+0000","last_fullsized":"2024-04-16T16:14:41.290724+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:43:48.615033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291106+0000","last_change":"2024-04-16T16:14:39.698584+0000","last_active":"2024-04-16T16:14:41.291106+0000","last_peered":"2024-04-16T16:14:41.291106+0000","last_clean":"2024-04-16T16:14:41.291106+0000","last_became_active":"2024-04-16T16:14:39.697610+0000","last_became_peered":"2024-04-16T16:14:39.697610+0000","last_unstale":"2024-04-16T16:14:41.291106+0000","last_undegraded":"2024-04-16T16:14:41.291106+0000","last_fullsized":"2024-04-16T16:14:41.291106+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:31:35.408519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.684802+0000","last_change":"2024-04-16T16:14:38.657699+0000","last_active":"2024-04-16T16:14:40.684802+0000","last_peered":"2024-04-16T16:14:40.684802+0000","last_clean":"2024-04-16T16:14:40.684802+0000","last_became_active":"2024-04-16T16:14:38.657409+0000","last_became_peered":"2024-04-16T16:14:38.657409+0000","last_unstale":"2024-04-16T16:14:40.684802+0000","last_undegraded":"2024-04-16T16:14:40.684802+0000","last_fullsized":"2024-04-16T16:14:40.684802+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:25:42.846200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291805+0000","last_change":"2024-04-16T16:14:39.707695+0000","last_active":"2024-04-16T16:14:41.291805+0000","last_peered":"2024-04-16T16:14:41.291805+0000","last_clean":"2024-04-16T16:14:41.291805+0000","last_became_active":"2024-04-16T16:14:39.707352+0000","last_became_peered":"2024-04-16T16:14:39.707352+0000","last_unstale":"2024-04-16T16:14:41.291805+0000","last_undegraded":"2024-04-16T16:14:41.291805+0000","last_fullsized":"2024-04-16T16:14:41.291805+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:01:01.716267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290472+0000","last_change":"2024-04-16T16:14:38.664407+0000","last_active":"2024-04-16T16:14:41.290472+0000","last_peered":"2024-04-16T16:14:41.290472+0000","last_clean":"2024-04-16T16:14:41.290472+0000","last_became_active":"2024-04-16T16:14:38.664039+0000","last_became_peered":"2024-04-16T16:14:38.664039+0000","last_unstale":"2024-04-16T16:14:41.290472+0000","last_undegraded":"2024-04-16T16:14:41.290472+0000","last_fullsized":"2024-04-16T16:14:41.290472+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:41:56.284222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691494+0000","last_change":"2024-04-16T16:14:39.701237+0000","last_active":"2024-04-16T16:14:40.691494+0000","last_peered":"2024-04-16T16:14:40.691494+0000","last_clean":"2024-04-16T16:14:40.691494+0000","last_became_active":"2024-04-16T16:14:39.700904+0000","last_became_peered":"2024-04-16T16:14:39.700904+0000","last_unstale":"2024-04-16T16:14:40.691494+0000","last_undegraded":"2024-04-16T16:14:40.691494+0000","last_fullsized":"2024-04-16T16:14:40.691494+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:23:31.999680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.686037+0000","last_change":"2024-04-16T16:14:38.658747+0000","last_active":"2024-04-16T16:14:40.686037+0000","last_peered":"2024-04-16T16:14:40.686037+0000","last_clean":"2024-04-16T16:14:40.686037+0000","last_became_active":"2024-04-16T16:14:38.656842+0000","last_became_peered":"2024-04-16T16:14:38.656842+0000","last_unstale":"2024-04-16T16:14:40.686037+0000","last_undegraded":"2024-04-16T16:14:40.686037+0000","last_fullsized":"2024-04-16T16:14:40.686037+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:45:30.832358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.686289+0000","last_change":"2024-04-16T16:14:39.699558+0000","last_active":"2024-04-16T16:14:40.686289+0000","last_peered":"2024-04-16T16:14:40.686289+0000","last_clean":"2024-04-16T16:14:40.686289+0000","last_became_active":"2024-04-16T16:14:39.699265+0000","last_became_peered":"2024-04-16T16:14:39.699265+0000","last_unstale":"2024-04-16T16:14:40.686289+0000","last_undegraded":"2024-04-16T16:14:40.686289+0000","last_fullsized":"2024-04-16T16:14:40.686289+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:43:08.613864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.727282+0000","last_change":"2024-04-16T16:14:38.672532+0000","last_active":"2024-04-16T16:14:40.727282+0000","last_peered":"2024-04-16T16:14:40.727282+0000","last_clean":"2024-04-16T16:14:40.727282+0000","last_became_active":"2024-04-16T16:14:38.671881+0000","last_became_peered":"2024-04-16T16:14:38.671881+0000","last_unstale":"2024-04-16T16:14:40.727282+0000","last_undegraded":"2024-04-16T16:14:40.727282+0000","last_fullsized":"2024-04-16T16:14:40.727282+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:18:01.189685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685628+0000","last_change":"2024-04-16T16:14:39.697739+0000","last_active":"2024-04-16T16:14:40.685628+0000","last_peered":"2024-04-16T16:14:40.685628+0000","last_clean":"2024-04-16T16:14:40.685628+0000","last_became_active":"2024-04-16T16:14:39.697472+0000","last_became_peered":"2024-04-16T16:14:39.697472+0000","last_unstale":"2024-04-16T16:14:40.685628+0000","last_undegraded":"2024-04-16T16:14:40.685628+0000","last_fullsized":"2024-04-16T16:14:40.685628+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:02:01.054178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691347+0000","last_change":"2024-04-16T16:14:38.667807+0000","last_active":"2024-04-16T16:14:40.691347+0000","last_peered":"2024-04-16T16:14:40.691347+0000","last_clean":"2024-04-16T16:14:40.691347+0000","last_became_active":"2024-04-16T16:14:38.667504+0000","last_became_peered":"2024-04-16T16:14:38.667504+0000","last_unstale":"2024-04-16T16:14:40.691347+0000","last_undegraded":"2024-04-16T16:14:40.691347+0000","last_fullsized":"2024-04-16T16:14:40.691347+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:50.689231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690759+0000","last_change":"2024-04-16T16:14:39.707564+0000","last_active":"2024-04-16T16:14:40.690759+0000","last_peered":"2024-04-16T16:14:40.690759+0000","last_clean":"2024-04-16T16:14:40.690759+0000","last_became_active":"2024-04-16T16:14:39.706939+0000","last_became_peered":"2024-04-16T16:14:39.706939+0000","last_unstale":"2024-04-16T16:14:40.690759+0000","last_undegraded":"2024-04-16T16:14:40.690759+0000","last_fullsized":"2024-04-16T16:14:40.690759+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:35.916987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685723+0000","last_change":"2024-04-16T16:14:38.669527+0000","last_active":"2024-04-16T16:14:40.685723+0000","last_peered":"2024-04-16T16:14:40.685723+0000","last_clean":"2024-04-16T16:14:40.685723+0000","last_became_active":"2024-04-16T16:14:38.669189+0000","last_became_peered":"2024-04-16T16:14:38.669189+0000","last_unstale":"2024-04-16T16:14:40.685723+0000","last_undegraded":"2024-04-16T16:14:40.685723+0000","last_fullsized":"2024-04-16T16:14:40.685723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:44:46.897943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291278+0000","last_change":"2024-04-16T16:14:39.708697+0000","last_active":"2024-04-16T16:14:41.291278+0000","last_peered":"2024-04-16T16:14:41.291278+0000","last_clean":"2024-04-16T16:14:41.291278+0000","last_became_active":"2024-04-16T16:14:39.708356+0000","last_became_peered":"2024-04-16T16:14:39.708356+0000","last_unstale":"2024-04-16T16:14:41.291278+0000","last_undegraded":"2024-04-16T16:14:41.291278+0000","last_fullsized":"2024-04-16T16:14:41.291278+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:54.548210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291043+0000","last_change":"2024-04-16T16:14:38.665714+0000","last_active":"2024-04-16T16:14:41.291043+0000","last_peered":"2024-04-16T16:14:41.291043+0000","last_clean":"2024-04-16T16:14:41.291043+0000","last_became_active":"2024-04-16T16:14:38.665380+0000","last_became_peered":"2024-04-16T16:14:38.665380+0000","last_unstale":"2024-04-16T16:14:41.291043+0000","last_undegraded":"2024-04-16T16:14:41.291043+0000","last_fullsized":"2024-04-16T16:14:41.291043+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:32:37.861499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685516+0000","last_change":"2024-04-16T16:14:39.687320+0000","last_active":"2024-04-16T16:14:40.685516+0000","last_peered":"2024-04-16T16:14:40.685516+0000","last_clean":"2024-04-16T16:14:40.685516+0000","last_became_active":"2024-04-16T16:14:39.686484+0000","last_became_peered":"2024-04-16T16:14:39.686484+0000","last_unstale":"2024-04-16T16:14:40.685516+0000","last_undegraded":"2024-04-16T16:14:40.685516+0000","last_fullsized":"2024-04-16T16:14:40.685516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:23:31.999680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291313+0000","last_change":"2024-04-16T16:14:38.672232+0000","last_active":"2024-04-16T16:14:41.291313+0000","last_peered":"2024-04-16T16:14:41.291313+0000","last_clean":"2024-04-16T16:14:41.291313+0000","last_became_active":"2024-04-16T16:14:38.671940+0000","last_became_peered":"2024-04-16T16:14:38.671940+0000","last_unstale":"2024-04-16T16:14:41.291313+0000","last_undegraded":"2024-04-16T16:14:41.291313+0000","last_fullsized":"2024-04-16T16:14:41.291313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:46.463986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690402+0000","last_change":"2024-04-16T16:14:39.691527+0000","last_active":"2024-04-16T16:14:40.690402+0000","last_peered":"2024-04-16T16:14:40.690402+0000","last_clean":"2024-04-16T16:14:40.690402+0000","last_became_active":"2024-04-16T16:14:39.691235+0000","last_became_peered":"2024-04-16T16:14:39.691235+0000","last_unstale":"2024-04-16T16:14:40.690402+0000","last_undegraded":"2024-04-16T16:14:40.690402+0000","last_fullsized":"2024-04-16T16:14:40.690402+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:13.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.684902+0000","last_change":"2024-04-16T16:14:38.659068+0000","last_active":"2024-04-16T16:14:40.684902+0000","last_peered":"2024-04-16T16:14:40.684902+0000","last_clean":"2024-04-16T16:14:40.684902+0000","last_became_active":"2024-04-16T16:14:38.658666+0000","last_became_peered":"2024-04-16T16:14:38.658666+0000","last_unstale":"2024-04-16T16:14:40.684902+0000","last_undegraded":"2024-04-16T16:14:40.684902+0000","last_fullsized":"2024-04-16T16:14:40.684902+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:50.689231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996507+0000","last_change":"2024-04-16T16:14:39.703632+0000","last_active":"2024-04-16T16:14:40.996507+0000","last_peered":"2024-04-16T16:14:40.996507+0000","last_clean":"2024-04-16T16:14:40.996507+0000","last_became_active":"2024-04-16T16:14:39.703276+0000","last_became_peered":"2024-04-16T16:14:39.703276+0000","last_unstale":"2024-04-16T16:14:40.996507+0000","last_undegraded":"2024-04-16T16:14:40.996507+0000","last_fullsized":"2024-04-16T16:14:40.996507+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:45:31.844610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":22,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.684827+0000","last_change":"2024-04-16T16:14:38.659623+0000","last_active":"2024-04-16T16:14:40.684827+0000","last_peered":"2024-04-16T16:14:40.684827+0000","last_clean":"2024-04-16T16:14:40.684827+0000","last_became_active":"2024-04-16T16:14:38.659313+0000","last_became_peered":"2024-04-16T16:14:38.659313+0000","last_unstale":"2024-04-16T16:14:40.684827+0000","last_undegraded":"2024-04-16T16:14:40.684827+0000","last_fullsized":"2024-04-16T16:14:40.684827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:24:12.994319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291714+0000","last_change":"2024-04-16T16:14:39.702044+0000","last_active":"2024-04-16T16:14:41.291714+0000","last_peered":"2024-04-16T16:14:41.291714+0000","last_clean":"2024-04-16T16:14:41.291714+0000","last_became_active":"2024-04-16T16:14:39.701773+0000","last_became_peered":"2024-04-16T16:14:39.701773+0000","last_unstale":"2024-04-16T16:14:41.291714+0000","last_undegraded":"2024-04-16T16:14:41.291714+0000","last_fullsized":"2024-04-16T16:14:41.291714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:36.323418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292516+0000","last_change":"2024-04-16T16:14:38.675837+0000","last_active":"2024-04-16T16:14:41.292516+0000","last_peered":"2024-04-16T16:14:41.292516+0000","last_clean":"2024-04-16T16:14:41.292516+0000","last_became_active":"2024-04-16T16:14:38.675519+0000","last_became_peered":"2024-04-16T16:14:38.675519+0000","last_unstale":"2024-04-16T16:14:41.292516+0000","last_undegraded":"2024-04-16T16:14:41.292516+0000","last_fullsized":"2024-04-16T16:14:41.292516+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:39:10.432469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691937+0000","last_change":"2024-04-16T16:14:39.702120+0000","last_active":"2024-04-16T16:14:40.691937+0000","last_peered":"2024-04-16T16:14:40.691937+0000","last_clean":"2024-04-16T16:14:40.691937+0000","last_became_active":"2024-04-16T16:14:39.701813+0000","last_became_peered":"2024-04-16T16:14:39.701813+0000","last_unstale":"2024-04-16T16:14:40.691937+0000","last_undegraded":"2024-04-16T16:14:40.691937+0000","last_fullsized":"2024-04-16T16:14:40.691937+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:51:43.656920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291528+0000","last_change":"2024-04-16T16:14:38.666762+0000","last_active":"2024-04-16T16:14:41.291528+0000","last_peered":"2024-04-16T16:14:41.291528+0000","last_clean":"2024-04-16T16:14:41.291528+0000","last_became_active":"2024-04-16T16:14:38.666193+0000","last_became_peered":"2024-04-16T16:14:38.666193+0000","last_unstale":"2024-04-16T16:14:41.291528+0000","last_undegraded":"2024-04-16T16:14:41.291528+0000","last_fullsized":"2024-04-16T16:14:41.291528+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:08:32.600681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291561+0000","last_change":"2024-04-16T16:14:39.704971+0000","last_active":"2024-04-16T16:14:41.291561+0000","last_peered":"2024-04-16T16:14:41.291561+0000","last_clean":"2024-04-16T16:14:41.291561+0000","last_became_active":"2024-04-16T16:14:39.704693+0000","last_became_peered":"2024-04-16T16:14:39.704693+0000","last_unstale":"2024-04-16T16:14:41.291561+0000","last_undegraded":"2024-04-16T16:14:41.291561+0000","last_fullsized":"2024-04-16T16:14:41.291561+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:16:57.270160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291579+0000","last_change":"2024-04-16T16:14:38.657843+0000","last_active":"2024-04-16T16:14:41.291579+0000","last_peered":"2024-04-16T16:14:41.291579+0000","last_clean":"2024-04-16T16:14:41.291579+0000","last_became_active":"2024-04-16T16:14:38.657384+0000","last_became_peered":"2024-04-16T16:14:38.657384+0000","last_unstale":"2024-04-16T16:14:41.291579+0000","last_undegraded":"2024-04-16T16:14:41.291579+0000","last_fullsized":"2024-04-16T16:14:41.291579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:19:47.312068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.005465+0000","last_change":"2024-04-16T16:14:39.699276+0000","last_active":"2024-04-16T16:14:41.005465+0000","last_peered":"2024-04-16T16:14:41.005465+0000","last_clean":"2024-04-16T16:14:41.005465+0000","last_became_active":"2024-04-16T16:14:39.698973+0000","last_became_peered":"2024-04-16T16:14:39.698973+0000","last_unstale":"2024-04-16T16:14:41.005465+0000","last_undegraded":"2024-04-16T16:14:41.005465+0000","last_fullsized":"2024-04-16T16:14:41.005465+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:47.476239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291384+0000","last_change":"2024-04-16T16:14:38.663164+0000","last_active":"2024-04-16T16:14:41.291384+0000","last_peered":"2024-04-16T16:14:41.291384+0000","last_clean":"2024-04-16T16:14:41.291384+0000","last_became_active":"2024-04-16T16:14:38.662887+0000","last_became_peered":"2024-04-16T16:14:38.662887+0000","last_unstale":"2024-04-16T16:14:41.291384+0000","last_undegraded":"2024-04-16T16:14:41.291384+0000","last_fullsized":"2024-04-16T16:14:41.291384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:14:46.163376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004691+0000","last_change":"2024-04-16T16:14:39.704026+0000","last_active":"2024-04-16T16:14:41.004691+0000","last_peered":"2024-04-16T16:14:41.004691+0000","last_clean":"2024-04-16T16:14:41.004691+0000","last_became_active":"2024-04-16T16:14:39.703710+0000","last_became_peered":"2024-04-16T16:14:39.703710+0000","last_unstale":"2024-04-16T16:14:41.004691+0000","last_undegraded":"2024-04-16T16:14:41.004691+0000","last_fullsized":"2024-04-16T16:14:41.004691+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:41:57.296475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.726518+0000","last_change":"2024-04-16T16:14:38.665012+0000","last_active":"2024-04-16T16:14:40.726518+0000","last_peered":"2024-04-16T16:14:40.726518+0000","last_clean":"2024-04-16T16:14:40.726518+0000","last_became_active":"2024-04-16T16:14:38.664721+0000","last_became_peered":"2024-04-16T16:14:38.664721+0000","last_unstale":"2024-04-16T16:14:40.726518+0000","last_undegraded":"2024-04-16T16:14:40.726518+0000","last_fullsized":"2024-04-16T16:14:40.726518+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:46.463986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.005261+0000","last_change":"2024-04-16T16:14:39.681884+0000","last_active":"2024-04-16T16:14:41.005261+0000","last_peered":"2024-04-16T16:14:41.005261+0000","last_clean":"2024-04-16T16:14:41.005261+0000","last_became_active":"2024-04-16T16:14:39.681629+0000","last_became_peered":"2024-04-16T16:14:39.681629+0000","last_unstale":"2024-04-16T16:14:41.005261+0000","last_undegraded":"2024-04-16T16:14:41.005261+0000","last_fullsized":"2024-04-16T16:14:41.005261+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:19:48.324320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996523+0000","last_change":"2024-04-16T16:14:38.671342+0000","last_active":"2024-04-16T16:14:40.996523+0000","last_peered":"2024-04-16T16:14:40.996523+0000","last_clean":"2024-04-16T16:14:40.996523+0000","last_became_active":"2024-04-16T16:14:38.670918+0000","last_became_peered":"2024-04-16T16:14:38.670918+0000","last_unstale":"2024-04-16T16:14:40.996523+0000","last_undegraded":"2024-04-16T16:14:40.996523+0000","last_fullsized":"2024-04-16T16:14:40.996523+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:08:32.600681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004216+0000","last_change":"2024-04-16T16:14:39.681058+0000","last_active":"2024-04-16T16:14:41.004216+0000","last_peered":"2024-04-16T16:14:41.004216+0000","last_clean":"2024-04-16T16:14:41.004216+0000","last_became_active":"2024-04-16T16:14:39.678280+0000","last_became_peered":"2024-04-16T16:14:39.678280+0000","last_unstale":"2024-04-16T16:14:41.004216+0000","last_undegraded":"2024-04-16T16:14:41.004216+0000","last_fullsized":"2024-04-16T16:14:41.004216+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:34.458058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290500+0000","last_change":"2024-04-16T16:14:38.668645+0000","last_active":"2024-04-16T16:14:41.290500+0000","last_peered":"2024-04-16T16:14:41.290500+0000","last_clean":"2024-04-16T16:14:41.290500+0000","last_became_active":"2024-04-16T16:14:38.668376+0000","last_became_peered":"2024-04-16T16:14:38.668376+0000","last_unstale":"2024-04-16T16:14:41.290500+0000","last_undegraded":"2024-04-16T16:14:41.290500+0000","last_fullsized":"2024-04-16T16:14:41.290500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:33.445805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690968+0000","last_change":"2024-04-16T16:14:39.697865+0000","last_active":"2024-04-16T16:14:40.690968+0000","last_peered":"2024-04-16T16:14:40.690968+0000","last_clean":"2024-04-16T16:14:40.690968+0000","last_became_active":"2024-04-16T16:14:39.696925+0000","last_became_peered":"2024-04-16T16:14:39.696925+0000","last_unstale":"2024-04-16T16:14:40.690968+0000","last_undegraded":"2024-04-16T16:14:40.690968+0000","last_fullsized":"2024-04-16T16:14:40.690968+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:27:36.166180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.689554+0000","last_change":"2024-04-16T16:14:38.668164+0000","last_active":"2024-04-16T16:14:40.689554+0000","last_peered":"2024-04-16T16:14:40.689554+0000","last_clean":"2024-04-16T16:14:40.689554+0000","last_became_active":"2024-04-16T16:14:38.667856+0000","last_became_peered":"2024-04-16T16:14:38.667856+0000","last_unstale":"2024-04-16T16:14:40.689554+0000","last_undegraded":"2024-04-16T16:14:40.689554+0000","last_fullsized":"2024-04-16T16:14:40.689554+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:16.441878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004964+0000","last_change":"2024-04-16T16:14:39.694889+0000","last_active":"2024-04-16T16:14:41.004964+0000","last_peered":"2024-04-16T16:14:41.004964+0000","last_clean":"2024-04-16T16:14:41.004964+0000","last_became_active":"2024-04-16T16:14:39.694130+0000","last_became_peered":"2024-04-16T16:14:39.694130+0000","last_unstale":"2024-04-16T16:14:41.004964+0000","last_undegraded":"2024-04-16T16:14:41.004964+0000","last_fullsized":"2024-04-16T16:14:41.004964+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:08:33.612934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292617+0000","last_change":"2024-04-16T16:14:38.674547+0000","last_active":"2024-04-16T16:14:41.292617+0000","last_peered":"2024-04-16T16:14:41.292617+0000","last_clean":"2024-04-16T16:14:41.292617+0000","last_became_active":"2024-04-16T16:14:38.674247+0000","last_became_peered":"2024-04-16T16:14:38.674247+0000","last_unstale":"2024-04-16T16:14:41.292617+0000","last_undegraded":"2024-04-16T16:14:41.292617+0000","last_fullsized":"2024-04-16T16:14:41.292617+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:25:42.846200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.692158+0000","last_change":"2024-04-16T16:14:39.709800+0000","last_active":"2024-04-16T16:14:40.692158+0000","last_peered":"2024-04-16T16:14:40.692158+0000","last_clean":"2024-04-16T16:14:40.692158+0000","last_became_active":"2024-04-16T16:14:39.709509+0000","last_became_peered":"2024-04-16T16:14:39.709509+0000","last_unstale":"2024-04-16T16:14:40.692158+0000","last_undegraded":"2024-04-16T16:14:40.692158+0000","last_fullsized":"2024-04-16T16:14:40.692158+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:02:01.054178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.727281+0000","last_change":"2024-04-16T16:14:38.669482+0000","last_active":"2024-04-16T16:14:40.727281+0000","last_peered":"2024-04-16T16:14:40.727281+0000","last_clean":"2024-04-16T16:14:40.727281+0000","last_became_active":"2024-04-16T16:14:38.669175+0000","last_became_peered":"2024-04-16T16:14:38.669175+0000","last_unstale":"2024-04-16T16:14:40.727281+0000","last_undegraded":"2024-04-16T16:14:40.727281+0000","last_fullsized":"2024-04-16T16:14:40.727281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:25:42.846200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.726937+0000","last_change":"2024-04-16T16:14:39.684854+0000","last_active":"2024-04-16T16:14:40.726937+0000","last_peered":"2024-04-16T16:14:40.726937+0000","last_clean":"2024-04-16T16:14:40.726937+0000","last_became_active":"2024-04-16T16:14:39.684524+0000","last_became_peered":"2024-04-16T16:14:39.684524+0000","last_unstale":"2024-04-16T16:14:40.726937+0000","last_undegraded":"2024-04-16T16:14:40.726937+0000","last_fullsized":"2024-04-16T16:14:40.726937+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:36.323418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291797+0000","last_change":"2024-04-16T16:14:38.675135+0000","last_active":"2024-04-16T16:14:41.291797+0000","last_peered":"2024-04-16T16:14:41.291797+0000","last_clean":"2024-04-16T16:14:41.291797+0000","last_became_active":"2024-04-16T16:14:38.674821+0000","last_became_peered":"2024-04-16T16:14:38.674821+0000","last_unstale":"2024-04-16T16:14:41.291797+0000","last_undegraded":"2024-04-16T16:14:41.291797+0000","last_fullsized":"2024-04-16T16:14:41.291797+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:18:01.189685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996428+0000","last_change":"2024-04-16T16:14:39.692135+0000","last_active":"2024-04-16T16:14:40.996428+0000","last_peered":"2024-04-16T16:14:40.996428+0000","last_clean":"2024-04-16T16:14:40.996428+0000","last_became_active":"2024-04-16T16:14:39.691817+0000","last_became_peered":"2024-04-16T16:14:39.691817+0000","last_unstale":"2024-04-16T16:14:40.996428+0000","last_undegraded":"2024-04-16T16:14:40.996428+0000","last_fullsized":"2024-04-16T16:14:40.996428+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:17.454131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.726901+0000","last_change":"2024-04-16T16:14:38.669656+0000","last_active":"2024-04-16T16:14:40.726901+0000","last_peered":"2024-04-16T16:14:40.726901+0000","last_clean":"2024-04-16T16:14:40.726901+0000","last_became_active":"2024-04-16T16:14:38.669304+0000","last_became_peered":"2024-04-16T16:14:38.669304+0000","last_unstale":"2024-04-16T16:14:40.726901+0000","last_undegraded":"2024-04-16T16:14:40.726901+0000","last_fullsized":"2024-04-16T16:14:40.726901+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:26:06.671145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.726817+0000","last_change":"2024-04-16T16:14:39.692738+0000","last_active":"2024-04-16T16:14:40.726817+0000","last_peered":"2024-04-16T16:14:40.726817+0000","last_clean":"2024-04-16T16:14:40.726817+0000","last_became_active":"2024-04-16T16:14:39.692443+0000","last_became_peered":"2024-04-16T16:14:39.692443+0000","last_unstale":"2024-04-16T16:14:40.726817+0000","last_undegraded":"2024-04-16T16:14:40.726817+0000","last_fullsized":"2024-04-16T16:14:40.726817+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:33.368730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291411+0000","last_change":"2024-04-16T16:14:38.673858+0000","last_active":"2024-04-16T16:14:41.291411+0000","last_peered":"2024-04-16T16:14:41.291411+0000","last_clean":"2024-04-16T16:14:41.291411+0000","last_became_active":"2024-04-16T16:14:38.673531+0000","last_became_peered":"2024-04-16T16:14:38.673531+0000","last_unstale":"2024-04-16T16:14:41.291411+0000","last_undegraded":"2024-04-16T16:14:41.291411+0000","last_fullsized":"2024-04-16T16:14:41.291411+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:26:06.671145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.684803+0000","last_change":"2024-04-16T16:14:39.700548+0000","last_active":"2024-04-16T16:14:40.684803+0000","last_peered":"2024-04-16T16:14:40.684803+0000","last_clean":"2024-04-16T16:14:40.684803+0000","last_became_active":"2024-04-16T16:14:39.700278+0000","last_became_peered":"2024-04-16T16:14:39.700278+0000","last_unstale":"2024-04-16T16:14:40.684803+0000","last_undegraded":"2024-04-16T16:14:40.684803+0000","last_fullsized":"2024-04-16T16:14:40.684803+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:27:36.166180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290796+0000","last_change":"2024-04-16T16:14:38.671160+0000","last_active":"2024-04-16T16:14:41.290796+0000","last_peered":"2024-04-16T16:14:41.290796+0000","last_clean":"2024-04-16T16:14:41.290796+0000","last_became_active":"2024-04-16T16:14:38.670444+0000","last_became_peered":"2024-04-16T16:14:38.670444+0000","last_unstale":"2024-04-16T16:14:41.290796+0000","last_undegraded":"2024-04-16T16:14:41.290796+0000","last_fullsized":"2024-04-16T16:14:41.290796+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:06:15.287730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290655+0000","last_change":"2024-04-16T16:14:39.701562+0000","last_active":"2024-04-16T16:14:41.290655+0000","last_peered":"2024-04-16T16:14:41.290655+0000","last_clean":"2024-04-16T16:14:41.290655+0000","last_became_active":"2024-04-16T16:14:39.697627+0000","last_became_peered":"2024-04-16T16:14:39.697627+0000","last_unstale":"2024-04-16T16:14:41.290655+0000","last_undegraded":"2024-04-16T16:14:41.290655+0000","last_fullsized":"2024-04-16T16:14:41.290655+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:44:47.910196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685143+0000","last_change":"2024-04-16T16:14:38.664663+0000","last_active":"2024-04-16T16:14:40.685143+0000","last_peered":"2024-04-16T16:14:40.685143+0000","last_clean":"2024-04-16T16:14:40.685143+0000","last_became_active":"2024-04-16T16:14:38.663340+0000","last_became_peered":"2024-04-16T16:14:38.663340+0000","last_unstale":"2024-04-16T16:14:40.685143+0000","last_undegraded":"2024-04-16T16:14:40.685143+0000","last_fullsized":"2024-04-16T16:14:40.685143+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:31:34.396267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690125+0000","last_change":"2024-04-16T16:14:39.692510+0000","last_active":"2024-04-16T16:14:40.690125+0000","last_peered":"2024-04-16T16:14:40.690125+0000","last_clean":"2024-04-16T16:14:40.690125+0000","last_became_active":"2024-04-16T16:14:39.691697+0000","last_became_peered":"2024-04-16T16:14:39.691697+0000","last_unstale":"2024-04-16T16:14:40.690125+0000","last_undegraded":"2024-04-16T16:14:40.690125+0000","last_fullsized":"2024-04-16T16:14:40.690125+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:23.282939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996063+0000","last_change":"2024-04-16T16:14:38.672045+0000","last_active":"2024-04-16T16:14:40.996063+0000","last_peered":"2024-04-16T16:14:40.996063+0000","last_clean":"2024-04-16T16:14:40.996063+0000","last_became_active":"2024-04-16T16:14:38.671755+0000","last_became_peered":"2024-04-16T16:14:38.671755+0000","last_unstale":"2024-04-16T16:14:40.996063+0000","last_undegraded":"2024-04-16T16:14:40.996063+0000","last_fullsized":"2024-04-16T16:14:40.996063+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:46.463986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":196,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690537+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:14:40.690537+0000","last_peered":"2024-04-16T16:14:40.690537+0000","last_clean":"2024-04-16T16:14:40.690537+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:14:40.690537+0000","last_undegraded":"2024-04-16T16:14:40.690537+0000","last_fullsized":"2024-04-16T16:14:40.690537+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":983040,"data_stored":741737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":151895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":151951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":151383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":152913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":389120,"data_stored":151349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":152913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":978944,"data_stored":741647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:14:49.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:14:49.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.577+0000 7fc00f200640 1 Processor -- start 2024-04-16T16:14:49.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.578+0000 7fc00f200640 1 -- start start 2024-04-16T16:14:49.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.579+0000 7fc00f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc0100d52d0 0x7fc01006ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:49.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.579+0000 7fc00f200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0100d3080 con 0x7fc010162270 2024-04-16T16:14:49.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.579+0000 7fc00f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0100d3360 con 0x7fc010161e90 2024-04-16T16:14:49.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.579+0000 7fc00f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc0100d2e30 con 0x7fc0100d52d0 2024-04-16T16:14:49.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.580+0000 7fc00d400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc010162270 0x7fc0100d3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42376/0 (socket says 172.21.15.45:42376) 2024-04-16T16:14:49.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.580+0000 7fc00d400640 1 -- 172.21.15.45:0/3775623613 learned_addr learned my addr 172.21.15.45:0/3775623613 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:49.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.580+0000 7fc00e800640 1 --2- 172.21.15.45:0/3775623613 >> v2:172.21.15.103:3300/0 conn(0x7fc0100d52d0 0x7fc01006ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:49.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.580+0000 7fc00ca00640 1 -- 172.21.15.45:0/3775623613 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4178772764 0 0) 0x7fc0100d3080 con 0x7fc010162270 2024-04-16T16:14:49.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.580+0000 7fc00ca00640 1 -- 172.21.15.45:0/3775623613 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbff4003520 con 0x7fc010162270 2024-04-16T16:14:49.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.580+0000 7fc00ca00640 1 -- 172.21.15.45:0/3775623613 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3996682603 0 0) 0x7fbff4003520 con 0x7fc010162270 2024-04-16T16:14:49.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.580+0000 7fc00ca00640 1 -- 172.21.15.45:0/3775623613 >> v2:172.21.15.103:3300/0 conn(0x7fc0100d52d0 msgr2=0x7fc01006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:49.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.580+0000 7fc00ca00640 1 --2- 172.21.15.45:0/3775623613 >> v2:172.21.15.103:3300/0 conn(0x7fc0100d52d0 0x7fc01006ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:49.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00ca00640 1 -- 172.21.15.45:0/3775623613 >> v1:172.21.15.103:6789/0 conn(0x7fc010161e90 legacy=0x7fc0100cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:49.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00ca00640 1 -- 172.21.15.45:0/3775623613 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0100d3640 con 0x7fc010162270 2024-04-16T16:14:49.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00ca00640 1 -- 172.21.15.45:0/3775623613 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fbff8001670 con 0x7fc010162270 2024-04-16T16:14:49.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00ca00640 1 -- 172.21.15.45:0/3775623613 >> v1:172.21.15.45:6789/0 conn(0x7fc010162270 legacy=0x7fc0100d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:49.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00ca00640 1 --2- 172.21.15.45:0/3775623613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff40039b0 0x7fbff4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:49.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00ca00640 1 --2- 172.21.15.45:0/3775623613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbff4004800 0x7fbff4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:49.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00ca00640 1 --2- 172.21.15.45:0/3775623613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff40051c0 0x7fbff400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:49.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00ca00640 1 -- 172.21.15.45:0/3775623613 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc0100d2e30 con 0x7fbff40051c0 2024-04-16T16:14:49.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00ca00640 1 -- 172.21.15.45:0/3775623613 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbff400c110 con 0x7fbff40039b0 2024-04-16T16:14:49.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00ca00640 1 -- 172.21.15.45:0/3775623613 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbff400c310 con 0x7fbff4004800 2024-04-16T16:14:49.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00e800640 1 --2- 172.21.15.45:0/3775623613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff40051c0 0x7fbff400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:49.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00de00640 1 --2- 172.21.15.45:0/3775623613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff40039b0 0x7fbff4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:49.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00e800640 1 -- 172.21.15.45:0/3775623613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbff4004800 msgr2=0x7fbff4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:49.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00e800640 1 --2- 172.21.15.45:0/3775623613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbff4004800 0x7fbff4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:49.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00e800640 1 -- 172.21.15.45:0/3775623613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff40039b0 msgr2=0x7fbff4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:49.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00e800640 1 --2- 172.21.15.45:0/3775623613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff40039b0 0x7fbff4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:49.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.581+0000 7fc00e800640 1 -- 172.21.15.45:0/3775623613 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff4003520 con 0x7fbff40051c0 2024-04-16T16:14:49.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.582+0000 7fc00e800640 1 -- 172.21.15.45:0/3775623613 reap_dead start 2024-04-16T16:14:49.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.583+0000 7fc00e800640 1 --2- 172.21.15.45:0/3775623613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff40051c0 0x7fbff400b9c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc008007330 tx=0x7fbff4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945d1f668a01feb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:49.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.583+0000 7fc00ca00640 1 -- 172.21.15.45:0/3775623613 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc008039070 con 0x7fbff40051c0 2024-04-16T16:14:49.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.583+0000 7fc00ca00640 1 -- 172.21.15.45:0/3775623613 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc00802ccd0 con 0x7fbff40051c0 2024-04-16T16:14:49.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.583+0000 7fc00ca00640 1 -- 172.21.15.45:0/3775623613 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc008009be0 con 0x7fbff40051c0 2024-04-16T16:14:49.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.583+0000 7fc00f200640 1 -- 172.21.15.45:0/3775623613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff40051c0 msgr2=0x7fbff400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:49.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.583+0000 7fc00f200640 1 --2- 172.21.15.45:0/3775623613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff40051c0 0x7fbff400b9c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc008007330 tx=0x7fbff4005830 comp rx=0 tx=0).stop 2024-04-16T16:14:49.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.584+0000 7fc00f200640 1 -- 172.21.15.45:0/3775623613 shutdown_connections 2024-04-16T16:14:49.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.584+0000 7fc00f200640 1 --2- 172.21.15.45:0/3775623613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff40051c0 0x7fbff400b9c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:49.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.584+0000 7fc00f200640 1 -- 172.21.15.45:0/3775623613 >> 172.21.15.45:0/3775623613 conn(0x7fc0100d1940 msgr2=0x7fc0100d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:49.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.584+0000 7fc00f200640 1 -- 172.21.15.45:0/3775623613 shutdown_connections 2024-04-16T16:14:49.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.584+0000 7fc00f200640 1 -- 172.21.15.45:0/3775623613 wait complete. 2024-04-16T16:14:49.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.585+0000 7fc00f200640 1 Processor -- start 2024-04-16T16:14:49.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.586+0000 7fc00f200640 1 -- start start 2024-04-16T16:14:49.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.586+0000 7fc00f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff40051c0 0x7fc0100cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:49.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.586+0000 7fc00f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0100abc20 0x7fc01015dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:49.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.587+0000 7fc00f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100c93b0 0x7fc0100c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:49.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.587+0000 7fc00f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc0100cf950 con 0x7fc0100abc20 2024-04-16T16:14:49.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.587+0000 7fc00f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc0100cf7d0 con 0x7fbff40051c0 2024-04-16T16:14:49.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.587+0000 7fc00f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc0100cfad0 con 0x7fc0100c93b0 2024-04-16T16:14:49.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.587+0000 7fc00d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0100abc20 0x7fc01015dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:49.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.587+0000 7fc00d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0100abc20 0x7fc01015dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50680/0 (socket says 172.21.15.45:50680) 2024-04-16T16:14:49.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.587+0000 7fc00d400640 1 -- 172.21.15.45:0/20542691 learned_addr learned my addr 172.21.15.45:0/20542691 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:49.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.587+0000 7fc00de00640 1 --2- 172.21.15.45:0/20542691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff40051c0 0x7fc0100cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:49.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.587+0000 7fc00d400640 1 -- 172.21.15.45:0/20542691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100c93b0 msgr2=0x7fc0100c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:49.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.587+0000 7fc00d400640 1 --2- 172.21.15.45:0/20542691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100c93b0 0x7fc0100c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:49.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.587+0000 7fc00d400640 1 -- 172.21.15.45:0/20542691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff40051c0 msgr2=0x7fc0100cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:49.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.587+0000 7fc00d400640 1 --2- 172.21.15.45:0/20542691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff40051c0 0x7fc0100cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:49.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.587+0000 7fc00d400640 1 -- 172.21.15.45:0/20542691 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0100ac190 con 0x7fc0100abc20 2024-04-16T16:14:49.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.588+0000 7fc00d400640 1 --2- 172.21.15.45:0/20542691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0100abc20 0x7fc01015dd30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbff800c7a0 tx=0x7fbff800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:49.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.588+0000 7fc006000640 1 -- 172.21.15.45:0/20542691 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8018070 con 0x7fc0100abc20 2024-04-16T16:14:49.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.588+0000 7fc006000640 1 -- 172.21.15.45:0/20542691 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff800ce50 con 0x7fc0100abc20 2024-04-16T16:14:49.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.588+0000 7fc006000640 1 -- 172.21.15.45:0/20542691 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff80047e0 con 0x7fc0100abc20 2024-04-16T16:14:49.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.588+0000 7fc00f200640 1 -- 172.21.15.45:0/20542691 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0100ac470 con 0x7fc0100abc20 2024-04-16T16:14:49.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.588+0000 7fc00f200640 1 -- 172.21.15.45:0/20542691 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0100aca30 con 0x7fc0100abc20 2024-04-16T16:14:49.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.589+0000 7fc00f200640 1 -- 172.21.15.45:0/20542691 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc010075780 con 0x7fc0100abc20 2024-04-16T16:14:49.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.590+0000 7fc006000640 1 -- 172.21.15.45:0/20542691 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbff801b070 con 0x7fc0100abc20 2024-04-16T16:14:49.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.591+0000 7fc006000640 1 --2- 172.21.15.45:0/20542691 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbfec06c8f0 0x7fbfec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:49.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.592+0000 7fc006000640 1 -- 172.21.15.45:0/20542691 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fbff808b070 con 0x7fc0100abc20 2024-04-16T16:14:49.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.592+0000 7fc00de00640 1 --2- 172.21.15.45:0/20542691 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbfec06c8f0 0x7fbfec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:49.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.593+0000 7fc00de00640 1 --2- 172.21.15.45:0/20542691 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbfec06c8f0 0x7fbfec06edb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc000007fd0 tx=0x7fc000009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:49.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.596+0000 7fc006000640 1 -- 172.21.15.45:0/20542691 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbff8059490 con 0x7fc0100abc20 2024-04-16T16:14:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.798+0000 7fc00f200640 1 -- 172.21.15.45:0/20542691 --> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc01006dae0 con 0x7fbfec06c8f0 2024-04-16T16:14:49.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.817+0000 7fc006000640 1 -- 172.21.15.45:0/20542691 <== mgr.4105 v2:172.21.15.103:6832/2711977587 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251839 (secure 0 0 0) 0x7fc01006dae0 con 0x7fbfec06c8f0 2024-04-16T16:14:49.820 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:49.821 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:14:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.821+0000 7fc00f200640 1 -- 172.21.15.45:0/20542691 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbfec06c8f0 msgr2=0x7fbfec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.821+0000 7fc00f200640 1 --2- 172.21.15.45:0/20542691 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbfec06c8f0 0x7fbfec06edb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc000007fd0 tx=0x7fc000009040 comp rx=0 tx=0).stop 2024-04-16T16:14:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.821+0000 7fc00f200640 1 -- 172.21.15.45:0/20542691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0100abc20 msgr2=0x7fc01015dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.822+0000 7fc00f200640 1 --2- 172.21.15.45:0/20542691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0100abc20 0x7fc01015dd30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbff800c7a0 tx=0x7fbff800cc60 comp rx=0 tx=0).stop 2024-04-16T16:14:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.822+0000 7fc00f200640 1 -- 172.21.15.45:0/20542691 shutdown_connections 2024-04-16T16:14:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.822+0000 7fc00f200640 1 --2- 172.21.15.45:0/20542691 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbfec06c8f0 0x7fbfec06edb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.822+0000 7fc00f200640 1 --2- 172.21.15.45:0/20542691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100c93b0 0x7fc0100c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.822+0000 7fc00f200640 1 --2- 172.21.15.45:0/20542691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0100abc20 0x7fc01015dd30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.822+0000 7fc00f200640 1 --2- 172.21.15.45:0/20542691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff40051c0 0x7fc0100cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.822+0000 7fc00f200640 1 -- 172.21.15.45:0/20542691 >> 172.21.15.45:0/20542691 conn(0x7fc0100d1940 msgr2=0x7fc0100b9380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.822+0000 7fc00f200640 1 -- 172.21.15.45:0/20542691 shutdown_connections 2024-04-16T16:14:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:49.822+0000 7fc00f200640 1 -- 172.21.15.45:0/20542691 wait complete. 2024-04-16T16:14:49.843 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-04-16T16:14:49.121048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237040,"kb_used_data":4232,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737680,"statfs":{"total":773094113280,"available":772851384320,"internally_reserved":0,"allocated":4333568,"data_stored":2395788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.458698"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996516+0000","last_change":"2024-04-16T16:14:39.708878+0000","last_active":"2024-04-16T16:14:40.996516+0000","last_peered":"2024-04-16T16:14:40.996516+0000","last_clean":"2024-04-16T16:14:40.996516+0000","last_became_active":"2024-04-16T16:14:39.708146+0000","last_became_peered":"2024-04-16T16:14:39.708146+0000","last_unstale":"2024-04-16T16:14:40.996516+0000","last_undegraded":"2024-04-16T16:14:40.996516+0000","last_fullsized":"2024-04-16T16:14:40.996516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:24:14.006571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.686276+0000","last_change":"2024-04-16T16:14:39.697128+0000","last_active":"2024-04-16T16:14:40.686276+0000","last_peered":"2024-04-16T16:14:40.686276+0000","last_clean":"2024-04-16T16:14:40.686276+0000","last_became_active":"2024-04-16T16:14:39.696806+0000","last_became_peered":"2024-04-16T16:14:39.696806+0000","last_unstale":"2024-04-16T16:14:40.686276+0000","last_undegraded":"2024-04-16T16:14:40.686276+0000","last_fullsized":"2024-04-16T16:14:40.686276+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:55:56.042534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691181+0000","last_change":"2024-04-16T16:14:39.707270+0000","last_active":"2024-04-16T16:14:40.691181+0000","last_peered":"2024-04-16T16:14:40.691181+0000","last_clean":"2024-04-16T16:14:40.691181+0000","last_became_active":"2024-04-16T16:14:39.706592+0000","last_became_peered":"2024-04-16T16:14:39.706592+0000","last_unstale":"2024-04-16T16:14:40.691181+0000","last_undegraded":"2024-04-16T16:14:40.691181+0000","last_fullsized":"2024-04-16T16:14:40.691181+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:10:23.378754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291332+0000","last_change":"2024-04-16T16:14:39.704396+0000","last_active":"2024-04-16T16:14:41.291332+0000","last_peered":"2024-04-16T16:14:41.291332+0000","last_clean":"2024-04-16T16:14:41.291332+0000","last_became_active":"2024-04-16T16:14:39.703644+0000","last_became_peered":"2024-04-16T16:14:39.703644+0000","last_unstale":"2024-04-16T16:14:41.291332+0000","last_undegraded":"2024-04-16T16:14:41.291332+0000","last_fullsized":"2024-04-16T16:14:41.291332+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:28:27.889354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004853+0000","last_change":"2024-04-16T16:14:39.691482+0000","last_active":"2024-04-16T16:14:41.004853+0000","last_peered":"2024-04-16T16:14:41.004853+0000","last_clean":"2024-04-16T16:14:41.004853+0000","last_became_active":"2024-04-16T16:14:39.690475+0000","last_became_peered":"2024-04-16T16:14:39.690475+0000","last_unstale":"2024-04-16T16:14:41.004853+0000","last_undegraded":"2024-04-16T16:14:41.004853+0000","last_fullsized":"2024-04-16T16:14:41.004853+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:29:05.538981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685032+0000","last_change":"2024-04-16T16:14:39.707402+0000","last_active":"2024-04-16T16:14:40.685032+0000","last_peered":"2024-04-16T16:14:40.685032+0000","last_clean":"2024-04-16T16:14:40.685032+0000","last_became_active":"2024-04-16T16:14:39.707125+0000","last_became_peered":"2024-04-16T16:14:39.707125+0000","last_unstale":"2024-04-16T16:14:40.685032+0000","last_undegraded":"2024-04-16T16:14:40.685032+0000","last_fullsized":"2024-04-16T16:14:40.685032+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:24:24.985004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690212+0000","last_change":"2024-04-16T16:14:39.696478+0000","last_active":"2024-04-16T16:14:40.690212+0000","last_peered":"2024-04-16T16:14:40.690212+0000","last_clean":"2024-04-16T16:14:40.690212+0000","last_became_active":"2024-04-16T16:14:39.696171+0000","last_became_peered":"2024-04-16T16:14:39.696171+0000","last_unstale":"2024-04-16T16:14:40.690212+0000","last_undegraded":"2024-04-16T16:14:40.690212+0000","last_fullsized":"2024-04-16T16:14:40.690212+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:35.916987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.689394+0000","last_change":"2024-04-16T16:14:39.686048+0000","last_active":"2024-04-16T16:14:40.689394+0000","last_peered":"2024-04-16T16:14:40.689394+0000","last_clean":"2024-04-16T16:14:40.689394+0000","last_became_active":"2024-04-16T16:14:39.682978+0000","last_became_peered":"2024-04-16T16:14:39.682978+0000","last_unstale":"2024-04-16T16:14:40.689394+0000","last_undegraded":"2024-04-16T16:14:40.689394+0000","last_fullsized":"2024-04-16T16:14:40.689394+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:16:57.270160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292049+0000","last_change":"2024-04-16T16:14:39.701196+0000","last_active":"2024-04-16T16:14:41.292049+0000","last_peered":"2024-04-16T16:14:41.292049+0000","last_clean":"2024-04-16T16:14:41.292049+0000","last_became_active":"2024-04-16T16:14:39.700442+0000","last_became_peered":"2024-04-16T16:14:39.700442+0000","last_unstale":"2024-04-16T16:14:41.292049+0000","last_undegraded":"2024-04-16T16:14:41.292049+0000","last_fullsized":"2024-04-16T16:14:41.292049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:54.548210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291929+0000","last_change":"2024-04-16T16:14:39.707318+0000","last_active":"2024-04-16T16:14:41.291929+0000","last_peered":"2024-04-16T16:14:41.291929+0000","last_clean":"2024-04-16T16:14:41.291929+0000","last_became_active":"2024-04-16T16:14:39.706939+0000","last_became_peered":"2024-04-16T16:14:39.706939+0000","last_unstale":"2024-04-16T16:14:41.291929+0000","last_undegraded":"2024-04-16T16:14:41.291929+0000","last_fullsized":"2024-04-16T16:14:41.291929+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:23.282939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691664+0000","last_change":"2024-04-16T16:14:39.702686+0000","last_active":"2024-04-16T16:14:40.691664+0000","last_peered":"2024-04-16T16:14:40.691664+0000","last_clean":"2024-04-16T16:14:40.691664+0000","last_became_active":"2024-04-16T16:14:39.702274+0000","last_became_peered":"2024-04-16T16:14:39.702274+0000","last_unstale":"2024-04-16T16:14:40.691664+0000","last_undegraded":"2024-04-16T16:14:40.691664+0000","last_fullsized":"2024-04-16T16:14:40.691664+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:10:23.378754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690786+0000","last_change":"2024-04-16T16:14:39.709482+0000","last_active":"2024-04-16T16:14:40.690786+0000","last_peered":"2024-04-16T16:14:40.690786+0000","last_clean":"2024-04-16T16:14:40.690786+0000","last_became_active":"2024-04-16T16:14:39.709110+0000","last_became_peered":"2024-04-16T16:14:39.709110+0000","last_unstale":"2024-04-16T16:14:40.690786+0000","last_undegraded":"2024-04-16T16:14:40.690786+0000","last_fullsized":"2024-04-16T16:14:40.690786+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:31.834009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004919+0000","last_change":"2024-04-16T16:14:39.697790+0000","last_active":"2024-04-16T16:14:41.004919+0000","last_peered":"2024-04-16T16:14:41.004919+0000","last_clean":"2024-04-16T16:14:41.004919+0000","last_became_active":"2024-04-16T16:14:39.696862+0000","last_became_peered":"2024-04-16T16:14:39.696862+0000","last_unstale":"2024-04-16T16:14:41.004919+0000","last_undegraded":"2024-04-16T16:14:41.004919+0000","last_fullsized":"2024-04-16T16:14:41.004919+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:09:31.099716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685180+0000","last_change":"2024-04-16T16:14:39.691105+0000","last_active":"2024-04-16T16:14:40.685180+0000","last_peered":"2024-04-16T16:14:40.685180+0000","last_clean":"2024-04-16T16:14:40.685180+0000","last_became_active":"2024-04-16T16:14:39.690771+0000","last_became_peered":"2024-04-16T16:14:39.690771+0000","last_unstale":"2024-04-16T16:14:40.685180+0000","last_undegraded":"2024-04-16T16:14:40.685180+0000","last_fullsized":"2024-04-16T16:14:40.685180+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:10:23.378754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690774+0000","last_change":"2024-04-16T16:14:39.707728+0000","last_active":"2024-04-16T16:14:40.690774+0000","last_peered":"2024-04-16T16:14:40.690774+0000","last_clean":"2024-04-16T16:14:40.690774+0000","last_became_active":"2024-04-16T16:14:39.707447+0000","last_became_peered":"2024-04-16T16:14:39.707447+0000","last_unstale":"2024-04-16T16:14:40.690774+0000","last_undegraded":"2024-04-16T16:14:40.690774+0000","last_fullsized":"2024-04-16T16:14:40.690774+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:13.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691686+0000","last_change":"2024-04-16T16:14:39.689522+0000","last_active":"2024-04-16T16:14:40.691686+0000","last_peered":"2024-04-16T16:14:40.691686+0000","last_clean":"2024-04-16T16:14:40.691686+0000","last_became_active":"2024-04-16T16:14:39.685321+0000","last_became_peered":"2024-04-16T16:14:39.685321+0000","last_unstale":"2024-04-16T16:14:40.691686+0000","last_undegraded":"2024-04-16T16:14:40.691686+0000","last_fullsized":"2024-04-16T16:14:40.691686+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:23:31.999680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.692050+0000","last_change":"2024-04-16T16:14:39.699366+0000","last_active":"2024-04-16T16:14:40.692050+0000","last_peered":"2024-04-16T16:14:40.692050+0000","last_clean":"2024-04-16T16:14:40.692050+0000","last_became_active":"2024-04-16T16:14:39.697476+0000","last_became_peered":"2024-04-16T16:14:39.697476+0000","last_unstale":"2024-04-16T16:14:40.692050+0000","last_undegraded":"2024-04-16T16:14:40.692050+0000","last_fullsized":"2024-04-16T16:14:40.692050+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:51:43.656920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.686048+0000","last_change":"2024-04-16T16:14:39.710331+0000","last_active":"2024-04-16T16:14:40.686048+0000","last_peered":"2024-04-16T16:14:40.686048+0000","last_clean":"2024-04-16T16:14:40.686048+0000","last_became_active":"2024-04-16T16:14:39.710065+0000","last_became_peered":"2024-04-16T16:14:39.710065+0000","last_unstale":"2024-04-16T16:14:40.686048+0000","last_undegraded":"2024-04-16T16:14:40.686048+0000","last_fullsized":"2024-04-16T16:14:40.686048+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:37:58.914102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685635+0000","last_change":"2024-04-16T16:14:39.700033+0000","last_active":"2024-04-16T16:14:40.685635+0000","last_peered":"2024-04-16T16:14:40.685635+0000","last_clean":"2024-04-16T16:14:40.685635+0000","last_became_active":"2024-04-16T16:14:39.699473+0000","last_became_peered":"2024-04-16T16:14:39.699473+0000","last_unstale":"2024-04-16T16:14:40.685635+0000","last_undegraded":"2024-04-16T16:14:40.685635+0000","last_fullsized":"2024-04-16T16:14:40.685635+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:51:43.656920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004721+0000","last_change":"2024-04-16T16:14:39.713504+0000","last_active":"2024-04-16T16:14:41.004721+0000","last_peered":"2024-04-16T16:14:41.004721+0000","last_clean":"2024-04-16T16:14:41.004721+0000","last_became_active":"2024-04-16T16:14:39.711524+0000","last_became_peered":"2024-04-16T16:14:39.711524+0000","last_unstale":"2024-04-16T16:14:41.004721+0000","last_undegraded":"2024-04-16T16:14:41.004721+0000","last_fullsized":"2024-04-16T16:14:41.004721+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:06:16.299983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290756+0000","last_change":"2024-04-16T16:14:39.698857+0000","last_active":"2024-04-16T16:14:41.290756+0000","last_peered":"2024-04-16T16:14:41.290756+0000","last_clean":"2024-04-16T16:14:41.290756+0000","last_became_active":"2024-04-16T16:14:39.693200+0000","last_became_peered":"2024-04-16T16:14:39.693200+0000","last_unstale":"2024-04-16T16:14:41.290756+0000","last_undegraded":"2024-04-16T16:14:41.290756+0000","last_fullsized":"2024-04-16T16:14:41.290756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:17.454131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290778+0000","last_change":"2024-04-16T16:14:39.694980+0000","last_active":"2024-04-16T16:14:41.290778+0000","last_peered":"2024-04-16T16:14:41.290778+0000","last_clean":"2024-04-16T16:14:41.290778+0000","last_became_active":"2024-04-16T16:14:39.691011+0000","last_became_peered":"2024-04-16T16:14:39.691011+0000","last_unstale":"2024-04-16T16:14:41.290778+0000","last_undegraded":"2024-04-16T16:14:41.290778+0000","last_fullsized":"2024-04-16T16:14:41.290778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:24:14.006571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.726768+0000","last_change":"2024-04-16T16:14:39.687391+0000","last_active":"2024-04-16T16:14:40.726768+0000","last_peered":"2024-04-16T16:14:40.726768+0000","last_clean":"2024-04-16T16:14:40.726768+0000","last_became_active":"2024-04-16T16:14:39.687092+0000","last_became_peered":"2024-04-16T16:14:39.687092+0000","last_unstale":"2024-04-16T16:14:40.726768+0000","last_undegraded":"2024-04-16T16:14:40.726768+0000","last_fullsized":"2024-04-16T16:14:40.726768+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:31:35.408519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.689686+0000","last_change":"2024-04-16T16:14:39.707655+0000","last_active":"2024-04-16T16:14:40.689686+0000","last_peered":"2024-04-16T16:14:40.689686+0000","last_clean":"2024-04-16T16:14:40.689686+0000","last_became_active":"2024-04-16T16:14:39.707273+0000","last_became_peered":"2024-04-16T16:14:39.707273+0000","last_unstale":"2024-04-16T16:14:40.689686+0000","last_undegraded":"2024-04-16T16:14:40.689686+0000","last_fullsized":"2024-04-16T16:14:40.689686+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:23.282939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292509+0000","last_change":"2024-04-16T16:14:39.685869+0000","last_active":"2024-04-16T16:14:41.292509+0000","last_peered":"2024-04-16T16:14:41.292509+0000","last_clean":"2024-04-16T16:14:41.292509+0000","last_became_active":"2024-04-16T16:14:39.685394+0000","last_became_peered":"2024-04-16T16:14:39.685394+0000","last_unstale":"2024-04-16T16:14:41.292509+0000","last_undegraded":"2024-04-16T16:14:41.292509+0000","last_fullsized":"2024-04-16T16:14:41.292509+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:31.834009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.686198+0000","last_change":"2024-04-16T16:14:39.696275+0000","last_active":"2024-04-16T16:14:40.686198+0000","last_peered":"2024-04-16T16:14:40.686198+0000","last_clean":"2024-04-16T16:14:40.686198+0000","last_became_active":"2024-04-16T16:14:39.695976+0000","last_became_peered":"2024-04-16T16:14:39.695976+0000","last_unstale":"2024-04-16T16:14:40.686198+0000","last_undegraded":"2024-04-16T16:14:40.686198+0000","last_fullsized":"2024-04-16T16:14:40.686198+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:46:36.132077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292347+0000","last_change":"2024-04-16T16:14:39.689971+0000","last_active":"2024-04-16T16:14:41.292347+0000","last_peered":"2024-04-16T16:14:41.292347+0000","last_clean":"2024-04-16T16:14:41.292347+0000","last_became_active":"2024-04-16T16:14:39.689482+0000","last_became_peered":"2024-04-16T16:14:39.689482+0000","last_unstale":"2024-04-16T16:14:41.292347+0000","last_undegraded":"2024-04-16T16:14:41.292347+0000","last_fullsized":"2024-04-16T16:14:41.292347+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:35.916987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691169+0000","last_change":"2024-04-16T16:14:39.701188+0000","last_active":"2024-04-16T16:14:40.691169+0000","last_peered":"2024-04-16T16:14:40.691169+0000","last_clean":"2024-04-16T16:14:40.691169+0000","last_became_active":"2024-04-16T16:14:39.697558+0000","last_became_peered":"2024-04-16T16:14:39.697558+0000","last_unstale":"2024-04-16T16:14:40.691169+0000","last_undegraded":"2024-04-16T16:14:40.691169+0000","last_fullsized":"2024-04-16T16:14:40.691169+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:24:24.985004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290891+0000","last_change":"2024-04-16T16:14:39.702421+0000","last_active":"2024-04-16T16:14:41.290891+0000","last_peered":"2024-04-16T16:14:41.290891+0000","last_clean":"2024-04-16T16:14:41.290891+0000","last_became_active":"2024-04-16T16:14:39.701406+0000","last_became_peered":"2024-04-16T16:14:39.701406+0000","last_unstale":"2024-04-16T16:14:41.290891+0000","last_undegraded":"2024-04-16T16:14:41.290891+0000","last_fullsized":"2024-04-16T16:14:41.290891+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:45:31.844610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690802+0000","last_change":"2024-04-16T16:14:39.700536+0000","last_active":"2024-04-16T16:14:40.690802+0000","last_peered":"2024-04-16T16:14:40.690802+0000","last_clean":"2024-04-16T16:14:40.690802+0000","last_became_active":"2024-04-16T16:14:39.697778+0000","last_became_peered":"2024-04-16T16:14:39.697778+0000","last_unstale":"2024-04-16T16:14:40.690802+0000","last_undegraded":"2024-04-16T16:14:40.690802+0000","last_fullsized":"2024-04-16T16:14:40.690802+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:31.834009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996750+0000","last_change":"2024-04-16T16:14:39.698727+0000","last_active":"2024-04-16T16:14:40.996750+0000","last_peered":"2024-04-16T16:14:40.996750+0000","last_clean":"2024-04-16T16:14:40.996750+0000","last_became_active":"2024-04-16T16:14:39.696171+0000","last_became_peered":"2024-04-16T16:14:39.696171+0000","last_unstale":"2024-04-16T16:14:40.996750+0000","last_undegraded":"2024-04-16T16:14:40.996750+0000","last_fullsized":"2024-04-16T16:14:40.996750+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:33.368730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.689600+0000","last_change":"2024-04-16T16:14:39.694912+0000","last_active":"2024-04-16T16:14:40.689600+0000","last_peered":"2024-04-16T16:14:40.689600+0000","last_clean":"2024-04-16T16:14:40.689600+0000","last_became_active":"2024-04-16T16:14:39.694584+0000","last_became_peered":"2024-04-16T16:14:39.694584+0000","last_unstale":"2024-04-16T16:14:40.689600+0000","last_undegraded":"2024-04-16T16:14:40.689600+0000","last_fullsized":"2024-04-16T16:14:40.689600+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:01:01.716267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.727191+0000","last_change":"2024-04-16T16:14:39.681803+0000","last_active":"2024-04-16T16:14:40.727191+0000","last_peered":"2024-04-16T16:14:40.727191+0000","last_clean":"2024-04-16T16:14:40.727191+0000","last_became_active":"2024-04-16T16:14:39.681310+0000","last_became_peered":"2024-04-16T16:14:39.681310+0000","last_unstale":"2024-04-16T16:14:40.727191+0000","last_undegraded":"2024-04-16T16:14:40.727191+0000","last_fullsized":"2024-04-16T16:14:40.727191+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:54.548210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996669+0000","last_change":"2024-04-16T16:14:38.671801+0000","last_active":"2024-04-16T16:14:40.996669+0000","last_peered":"2024-04-16T16:14:40.996669+0000","last_clean":"2024-04-16T16:14:40.996669+0000","last_became_active":"2024-04-16T16:14:38.671502+0000","last_became_peered":"2024-04-16T16:14:38.671502+0000","last_unstale":"2024-04-16T16:14:40.996669+0000","last_undegraded":"2024-04-16T16:14:40.996669+0000","last_fullsized":"2024-04-16T16:14:40.996669+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:19:47.312068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996555+0000","last_change":"2024-04-16T16:14:39.708496+0000","last_active":"2024-04-16T16:14:40.996555+0000","last_peered":"2024-04-16T16:14:40.996555+0000","last_clean":"2024-04-16T16:14:40.996555+0000","last_became_active":"2024-04-16T16:14:39.707725+0000","last_became_peered":"2024-04-16T16:14:39.707725+0000","last_unstale":"2024-04-16T16:14:40.996555+0000","last_undegraded":"2024-04-16T16:14:40.996555+0000","last_fullsized":"2024-04-16T16:14:40.996555+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:31:35.408519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.727037+0000","last_change":"2024-04-16T16:14:38.663415+0000","last_active":"2024-04-16T16:14:40.727037+0000","last_peered":"2024-04-16T16:14:40.727037+0000","last_clean":"2024-04-16T16:14:40.727037+0000","last_became_active":"2024-04-16T16:14:38.662731+0000","last_became_peered":"2024-04-16T16:14:38.662731+0000","last_unstale":"2024-04-16T16:14:40.727037+0000","last_undegraded":"2024-04-16T16:14:40.727037+0000","last_fullsized":"2024-04-16T16:14:40.727037+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:39:10.432469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291543+0000","last_change":"2024-04-16T16:14:39.706462+0000","last_active":"2024-04-16T16:14:41.291543+0000","last_peered":"2024-04-16T16:14:41.291543+0000","last_clean":"2024-04-16T16:14:41.291543+0000","last_became_active":"2024-04-16T16:14:39.705749+0000","last_became_peered":"2024-04-16T16:14:39.705749+0000","last_unstale":"2024-04-16T16:14:41.291543+0000","last_undegraded":"2024-04-16T16:14:41.291543+0000","last_fullsized":"2024-04-16T16:14:41.291543+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:33.368730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685349+0000","last_change":"2024-04-16T16:14:38.668822+0000","last_active":"2024-04-16T16:14:40.685349+0000","last_peered":"2024-04-16T16:14:40.685349+0000","last_clean":"2024-04-16T16:14:40.685349+0000","last_became_active":"2024-04-16T16:14:38.668018+0000","last_became_peered":"2024-04-16T16:14:38.668018+0000","last_unstale":"2024-04-16T16:14:40.685349+0000","last_undegraded":"2024-04-16T16:14:40.685349+0000","last_fullsized":"2024-04-16T16:14:40.685349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:16.441878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292426+0000","last_change":"2024-04-16T16:14:39.689746+0000","last_active":"2024-04-16T16:14:41.292426+0000","last_peered":"2024-04-16T16:14:41.292426+0000","last_clean":"2024-04-16T16:14:41.292426+0000","last_became_active":"2024-04-16T16:14:39.689162+0000","last_became_peered":"2024-04-16T16:14:39.689162+0000","last_unstale":"2024-04-16T16:14:41.292426+0000","last_undegraded":"2024-04-16T16:14:41.292426+0000","last_fullsized":"2024-04-16T16:14:41.292426+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:13.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004846+0000","last_change":"2024-04-16T16:14:38.673943+0000","last_active":"2024-04-16T16:14:41.004846+0000","last_peered":"2024-04-16T16:14:41.004846+0000","last_clean":"2024-04-16T16:14:41.004846+0000","last_became_active":"2024-04-16T16:14:38.672857+0000","last_became_peered":"2024-04-16T16:14:38.672857+0000","last_unstale":"2024-04-16T16:14:41.004846+0000","last_undegraded":"2024-04-16T16:14:41.004846+0000","last_fullsized":"2024-04-16T16:14:41.004846+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:38:56.751996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691157+0000","last_change":"2024-04-16T16:14:39.704670+0000","last_active":"2024-04-16T16:14:40.691157+0000","last_peered":"2024-04-16T16:14:40.691157+0000","last_clean":"2024-04-16T16:14:40.691157+0000","last_became_active":"2024-04-16T16:14:39.704321+0000","last_became_peered":"2024-04-16T16:14:39.704321+0000","last_unstale":"2024-04-16T16:14:40.691157+0000","last_undegraded":"2024-04-16T16:14:40.691157+0000","last_fullsized":"2024-04-16T16:14:40.691157+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:24:24.985004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290894+0000","last_change":"2024-04-16T16:14:38.662314+0000","last_active":"2024-04-16T16:14:41.290894+0000","last_peered":"2024-04-16T16:14:41.290894+0000","last_clean":"2024-04-16T16:14:41.290894+0000","last_became_active":"2024-04-16T16:14:38.662011+0000","last_became_peered":"2024-04-16T16:14:38.662011+0000","last_unstale":"2024-04-16T16:14:41.290894+0000","last_undegraded":"2024-04-16T16:14:41.290894+0000","last_fullsized":"2024-04-16T16:14:41.290894+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:49.878355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691057+0000","last_change":"2024-04-16T16:14:39.697339+0000","last_active":"2024-04-16T16:14:40.691057+0000","last_peered":"2024-04-16T16:14:40.691057+0000","last_clean":"2024-04-16T16:14:40.691057+0000","last_became_active":"2024-04-16T16:14:39.697037+0000","last_became_peered":"2024-04-16T16:14:39.697037+0000","last_unstale":"2024-04-16T16:14:40.691057+0000","last_undegraded":"2024-04-16T16:14:40.691057+0000","last_fullsized":"2024-04-16T16:14:40.691057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:27:36.166180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290724+0000","last_change":"2024-04-16T16:14:38.660477+0000","last_active":"2024-04-16T16:14:41.290724+0000","last_peered":"2024-04-16T16:14:41.290724+0000","last_clean":"2024-04-16T16:14:41.290724+0000","last_became_active":"2024-04-16T16:14:38.660156+0000","last_became_peered":"2024-04-16T16:14:38.660156+0000","last_unstale":"2024-04-16T16:14:41.290724+0000","last_undegraded":"2024-04-16T16:14:41.290724+0000","last_fullsized":"2024-04-16T16:14:41.290724+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:43:48.615033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291106+0000","last_change":"2024-04-16T16:14:39.698584+0000","last_active":"2024-04-16T16:14:41.291106+0000","last_peered":"2024-04-16T16:14:41.291106+0000","last_clean":"2024-04-16T16:14:41.291106+0000","last_became_active":"2024-04-16T16:14:39.697610+0000","last_became_peered":"2024-04-16T16:14:39.697610+0000","last_unstale":"2024-04-16T16:14:41.291106+0000","last_undegraded":"2024-04-16T16:14:41.291106+0000","last_fullsized":"2024-04-16T16:14:41.291106+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:31:35.408519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.684802+0000","last_change":"2024-04-16T16:14:38.657699+0000","last_active":"2024-04-16T16:14:40.684802+0000","last_peered":"2024-04-16T16:14:40.684802+0000","last_clean":"2024-04-16T16:14:40.684802+0000","last_became_active":"2024-04-16T16:14:38.657409+0000","last_became_peered":"2024-04-16T16:14:38.657409+0000","last_unstale":"2024-04-16T16:14:40.684802+0000","last_undegraded":"2024-04-16T16:14:40.684802+0000","last_fullsized":"2024-04-16T16:14:40.684802+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:25:42.846200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291805+0000","last_change":"2024-04-16T16:14:39.707695+0000","last_active":"2024-04-16T16:14:41.291805+0000","last_peered":"2024-04-16T16:14:41.291805+0000","last_clean":"2024-04-16T16:14:41.291805+0000","last_became_active":"2024-04-16T16:14:39.707352+0000","last_became_peered":"2024-04-16T16:14:39.707352+0000","last_unstale":"2024-04-16T16:14:41.291805+0000","last_undegraded":"2024-04-16T16:14:41.291805+0000","last_fullsized":"2024-04-16T16:14:41.291805+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:01:01.716267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290472+0000","last_change":"2024-04-16T16:14:38.664407+0000","last_active":"2024-04-16T16:14:41.290472+0000","last_peered":"2024-04-16T16:14:41.290472+0000","last_clean":"2024-04-16T16:14:41.290472+0000","last_became_active":"2024-04-16T16:14:38.664039+0000","last_became_peered":"2024-04-16T16:14:38.664039+0000","last_unstale":"2024-04-16T16:14:41.290472+0000","last_undegraded":"2024-04-16T16:14:41.290472+0000","last_fullsized":"2024-04-16T16:14:41.290472+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:41:56.284222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691494+0000","last_change":"2024-04-16T16:14:39.701237+0000","last_active":"2024-04-16T16:14:40.691494+0000","last_peered":"2024-04-16T16:14:40.691494+0000","last_clean":"2024-04-16T16:14:40.691494+0000","last_became_active":"2024-04-16T16:14:39.700904+0000","last_became_peered":"2024-04-16T16:14:39.700904+0000","last_unstale":"2024-04-16T16:14:40.691494+0000","last_undegraded":"2024-04-16T16:14:40.691494+0000","last_fullsized":"2024-04-16T16:14:40.691494+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:23:31.999680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.686037+0000","last_change":"2024-04-16T16:14:38.658747+0000","last_active":"2024-04-16T16:14:40.686037+0000","last_peered":"2024-04-16T16:14:40.686037+0000","last_clean":"2024-04-16T16:14:40.686037+0000","last_became_active":"2024-04-16T16:14:38.656842+0000","last_became_peered":"2024-04-16T16:14:38.656842+0000","last_unstale":"2024-04-16T16:14:40.686037+0000","last_undegraded":"2024-04-16T16:14:40.686037+0000","last_fullsized":"2024-04-16T16:14:40.686037+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:45:30.832358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.686289+0000","last_change":"2024-04-16T16:14:39.699558+0000","last_active":"2024-04-16T16:14:40.686289+0000","last_peered":"2024-04-16T16:14:40.686289+0000","last_clean":"2024-04-16T16:14:40.686289+0000","last_became_active":"2024-04-16T16:14:39.699265+0000","last_became_peered":"2024-04-16T16:14:39.699265+0000","last_unstale":"2024-04-16T16:14:40.686289+0000","last_undegraded":"2024-04-16T16:14:40.686289+0000","last_fullsized":"2024-04-16T16:14:40.686289+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:43:08.613864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.727282+0000","last_change":"2024-04-16T16:14:38.672532+0000","last_active":"2024-04-16T16:14:40.727282+0000","last_peered":"2024-04-16T16:14:40.727282+0000","last_clean":"2024-04-16T16:14:40.727282+0000","last_became_active":"2024-04-16T16:14:38.671881+0000","last_became_peered":"2024-04-16T16:14:38.671881+0000","last_unstale":"2024-04-16T16:14:40.727282+0000","last_undegraded":"2024-04-16T16:14:40.727282+0000","last_fullsized":"2024-04-16T16:14:40.727282+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:18:01.189685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685628+0000","last_change":"2024-04-16T16:14:39.697739+0000","last_active":"2024-04-16T16:14:40.685628+0000","last_peered":"2024-04-16T16:14:40.685628+0000","last_clean":"2024-04-16T16:14:40.685628+0000","last_became_active":"2024-04-16T16:14:39.697472+0000","last_became_peered":"2024-04-16T16:14:39.697472+0000","last_unstale":"2024-04-16T16:14:40.685628+0000","last_undegraded":"2024-04-16T16:14:40.685628+0000","last_fullsized":"2024-04-16T16:14:40.685628+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:02:01.054178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691347+0000","last_change":"2024-04-16T16:14:38.667807+0000","last_active":"2024-04-16T16:14:40.691347+0000","last_peered":"2024-04-16T16:14:40.691347+0000","last_clean":"2024-04-16T16:14:40.691347+0000","last_became_active":"2024-04-16T16:14:38.667504+0000","last_became_peered":"2024-04-16T16:14:38.667504+0000","last_unstale":"2024-04-16T16:14:40.691347+0000","last_undegraded":"2024-04-16T16:14:40.691347+0000","last_fullsized":"2024-04-16T16:14:40.691347+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:50.689231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690759+0000","last_change":"2024-04-16T16:14:39.707564+0000","last_active":"2024-04-16T16:14:40.690759+0000","last_peered":"2024-04-16T16:14:40.690759+0000","last_clean":"2024-04-16T16:14:40.690759+0000","last_became_active":"2024-04-16T16:14:39.706939+0000","last_became_peered":"2024-04-16T16:14:39.706939+0000","last_unstale":"2024-04-16T16:14:40.690759+0000","last_undegraded":"2024-04-16T16:14:40.690759+0000","last_fullsized":"2024-04-16T16:14:40.690759+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:35.916987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685723+0000","last_change":"2024-04-16T16:14:38.669527+0000","last_active":"2024-04-16T16:14:40.685723+0000","last_peered":"2024-04-16T16:14:40.685723+0000","last_clean":"2024-04-16T16:14:40.685723+0000","last_became_active":"2024-04-16T16:14:38.669189+0000","last_became_peered":"2024-04-16T16:14:38.669189+0000","last_unstale":"2024-04-16T16:14:40.685723+0000","last_undegraded":"2024-04-16T16:14:40.685723+0000","last_fullsized":"2024-04-16T16:14:40.685723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:44:46.897943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291278+0000","last_change":"2024-04-16T16:14:39.708697+0000","last_active":"2024-04-16T16:14:41.291278+0000","last_peered":"2024-04-16T16:14:41.291278+0000","last_clean":"2024-04-16T16:14:41.291278+0000","last_became_active":"2024-04-16T16:14:39.708356+0000","last_became_peered":"2024-04-16T16:14:39.708356+0000","last_unstale":"2024-04-16T16:14:41.291278+0000","last_undegraded":"2024-04-16T16:14:41.291278+0000","last_fullsized":"2024-04-16T16:14:41.291278+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:54.548210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291043+0000","last_change":"2024-04-16T16:14:38.665714+0000","last_active":"2024-04-16T16:14:41.291043+0000","last_peered":"2024-04-16T16:14:41.291043+0000","last_clean":"2024-04-16T16:14:41.291043+0000","last_became_active":"2024-04-16T16:14:38.665380+0000","last_became_peered":"2024-04-16T16:14:38.665380+0000","last_unstale":"2024-04-16T16:14:41.291043+0000","last_undegraded":"2024-04-16T16:14:41.291043+0000","last_fullsized":"2024-04-16T16:14:41.291043+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:32:37.861499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685516+0000","last_change":"2024-04-16T16:14:39.687320+0000","last_active":"2024-04-16T16:14:40.685516+0000","last_peered":"2024-04-16T16:14:40.685516+0000","last_clean":"2024-04-16T16:14:40.685516+0000","last_became_active":"2024-04-16T16:14:39.686484+0000","last_became_peered":"2024-04-16T16:14:39.686484+0000","last_unstale":"2024-04-16T16:14:40.685516+0000","last_undegraded":"2024-04-16T16:14:40.685516+0000","last_fullsized":"2024-04-16T16:14:40.685516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:23:31.999680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291313+0000","last_change":"2024-04-16T16:14:38.672232+0000","last_active":"2024-04-16T16:14:41.291313+0000","last_peered":"2024-04-16T16:14:41.291313+0000","last_clean":"2024-04-16T16:14:41.291313+0000","last_became_active":"2024-04-16T16:14:38.671940+0000","last_became_peered":"2024-04-16T16:14:38.671940+0000","last_unstale":"2024-04-16T16:14:41.291313+0000","last_undegraded":"2024-04-16T16:14:41.291313+0000","last_fullsized":"2024-04-16T16:14:41.291313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:46.463986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690402+0000","last_change":"2024-04-16T16:14:39.691527+0000","last_active":"2024-04-16T16:14:40.690402+0000","last_peered":"2024-04-16T16:14:40.690402+0000","last_clean":"2024-04-16T16:14:40.690402+0000","last_became_active":"2024-04-16T16:14:39.691235+0000","last_became_peered":"2024-04-16T16:14:39.691235+0000","last_unstale":"2024-04-16T16:14:40.690402+0000","last_undegraded":"2024-04-16T16:14:40.690402+0000","last_fullsized":"2024-04-16T16:14:40.690402+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:13.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.684902+0000","last_change":"2024-04-16T16:14:38.659068+0000","last_active":"2024-04-16T16:14:40.684902+0000","last_peered":"2024-04-16T16:14:40.684902+0000","last_clean":"2024-04-16T16:14:40.684902+0000","last_became_active":"2024-04-16T16:14:38.658666+0000","last_became_peered":"2024-04-16T16:14:38.658666+0000","last_unstale":"2024-04-16T16:14:40.684902+0000","last_undegraded":"2024-04-16T16:14:40.684902+0000","last_fullsized":"2024-04-16T16:14:40.684902+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:50.689231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996507+0000","last_change":"2024-04-16T16:14:39.703632+0000","last_active":"2024-04-16T16:14:40.996507+0000","last_peered":"2024-04-16T16:14:40.996507+0000","last_clean":"2024-04-16T16:14:40.996507+0000","last_became_active":"2024-04-16T16:14:39.703276+0000","last_became_peered":"2024-04-16T16:14:39.703276+0000","last_unstale":"2024-04-16T16:14:40.996507+0000","last_undegraded":"2024-04-16T16:14:40.996507+0000","last_fullsized":"2024-04-16T16:14:40.996507+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:45:31.844610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":22,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.684827+0000","last_change":"2024-04-16T16:14:38.659623+0000","last_active":"2024-04-16T16:14:40.684827+0000","last_peered":"2024-04-16T16:14:40.684827+0000","last_clean":"2024-04-16T16:14:40.684827+0000","last_became_active":"2024-04-16T16:14:38.659313+0000","last_became_peered":"2024-04-16T16:14:38.659313+0000","last_unstale":"2024-04-16T16:14:40.684827+0000","last_undegraded":"2024-04-16T16:14:40.684827+0000","last_fullsized":"2024-04-16T16:14:40.684827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:24:12.994319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291714+0000","last_change":"2024-04-16T16:14:39.702044+0000","last_active":"2024-04-16T16:14:41.291714+0000","last_peered":"2024-04-16T16:14:41.291714+0000","last_clean":"2024-04-16T16:14:41.291714+0000","last_became_active":"2024-04-16T16:14:39.701773+0000","last_became_peered":"2024-04-16T16:14:39.701773+0000","last_unstale":"2024-04-16T16:14:41.291714+0000","last_undegraded":"2024-04-16T16:14:41.291714+0000","last_fullsized":"2024-04-16T16:14:41.291714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:36.323418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292516+0000","last_change":"2024-04-16T16:14:38.675837+0000","last_active":"2024-04-16T16:14:41.292516+0000","last_peered":"2024-04-16T16:14:41.292516+0000","last_clean":"2024-04-16T16:14:41.292516+0000","last_became_active":"2024-04-16T16:14:38.675519+0000","last_became_peered":"2024-04-16T16:14:38.675519+0000","last_unstale":"2024-04-16T16:14:41.292516+0000","last_undegraded":"2024-04-16T16:14:41.292516+0000","last_fullsized":"2024-04-16T16:14:41.292516+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:39:10.432469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.691937+0000","last_change":"2024-04-16T16:14:39.702120+0000","last_active":"2024-04-16T16:14:40.691937+0000","last_peered":"2024-04-16T16:14:40.691937+0000","last_clean":"2024-04-16T16:14:40.691937+0000","last_became_active":"2024-04-16T16:14:39.701813+0000","last_became_peered":"2024-04-16T16:14:39.701813+0000","last_unstale":"2024-04-16T16:14:40.691937+0000","last_undegraded":"2024-04-16T16:14:40.691937+0000","last_fullsized":"2024-04-16T16:14:40.691937+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:51:43.656920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291528+0000","last_change":"2024-04-16T16:14:38.666762+0000","last_active":"2024-04-16T16:14:41.291528+0000","last_peered":"2024-04-16T16:14:41.291528+0000","last_clean":"2024-04-16T16:14:41.291528+0000","last_became_active":"2024-04-16T16:14:38.666193+0000","last_became_peered":"2024-04-16T16:14:38.666193+0000","last_unstale":"2024-04-16T16:14:41.291528+0000","last_undegraded":"2024-04-16T16:14:41.291528+0000","last_fullsized":"2024-04-16T16:14:41.291528+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:08:32.600681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291561+0000","last_change":"2024-04-16T16:14:39.704971+0000","last_active":"2024-04-16T16:14:41.291561+0000","last_peered":"2024-04-16T16:14:41.291561+0000","last_clean":"2024-04-16T16:14:41.291561+0000","last_became_active":"2024-04-16T16:14:39.704693+0000","last_became_peered":"2024-04-16T16:14:39.704693+0000","last_unstale":"2024-04-16T16:14:41.291561+0000","last_undegraded":"2024-04-16T16:14:41.291561+0000","last_fullsized":"2024-04-16T16:14:41.291561+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:16:57.270160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291579+0000","last_change":"2024-04-16T16:14:38.657843+0000","last_active":"2024-04-16T16:14:41.291579+0000","last_peered":"2024-04-16T16:14:41.291579+0000","last_clean":"2024-04-16T16:14:41.291579+0000","last_became_active":"2024-04-16T16:14:38.657384+0000","last_became_peered":"2024-04-16T16:14:38.657384+0000","last_unstale":"2024-04-16T16:14:41.291579+0000","last_undegraded":"2024-04-16T16:14:41.291579+0000","last_fullsized":"2024-04-16T16:14:41.291579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:19:47.312068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.005465+0000","last_change":"2024-04-16T16:14:39.699276+0000","last_active":"2024-04-16T16:14:41.005465+0000","last_peered":"2024-04-16T16:14:41.005465+0000","last_clean":"2024-04-16T16:14:41.005465+0000","last_became_active":"2024-04-16T16:14:39.698973+0000","last_became_peered":"2024-04-16T16:14:39.698973+0000","last_unstale":"2024-04-16T16:14:41.005465+0000","last_undegraded":"2024-04-16T16:14:41.005465+0000","last_fullsized":"2024-04-16T16:14:41.005465+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:47.476239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291384+0000","last_change":"2024-04-16T16:14:38.663164+0000","last_active":"2024-04-16T16:14:41.291384+0000","last_peered":"2024-04-16T16:14:41.291384+0000","last_clean":"2024-04-16T16:14:41.291384+0000","last_became_active":"2024-04-16T16:14:38.662887+0000","last_became_peered":"2024-04-16T16:14:38.662887+0000","last_unstale":"2024-04-16T16:14:41.291384+0000","last_undegraded":"2024-04-16T16:14:41.291384+0000","last_fullsized":"2024-04-16T16:14:41.291384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:14:46.163376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004691+0000","last_change":"2024-04-16T16:14:39.704026+0000","last_active":"2024-04-16T16:14:41.004691+0000","last_peered":"2024-04-16T16:14:41.004691+0000","last_clean":"2024-04-16T16:14:41.004691+0000","last_became_active":"2024-04-16T16:14:39.703710+0000","last_became_peered":"2024-04-16T16:14:39.703710+0000","last_unstale":"2024-04-16T16:14:41.004691+0000","last_undegraded":"2024-04-16T16:14:41.004691+0000","last_fullsized":"2024-04-16T16:14:41.004691+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:41:57.296475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.726518+0000","last_change":"2024-04-16T16:14:38.665012+0000","last_active":"2024-04-16T16:14:40.726518+0000","last_peered":"2024-04-16T16:14:40.726518+0000","last_clean":"2024-04-16T16:14:40.726518+0000","last_became_active":"2024-04-16T16:14:38.664721+0000","last_became_peered":"2024-04-16T16:14:38.664721+0000","last_unstale":"2024-04-16T16:14:40.726518+0000","last_undegraded":"2024-04-16T16:14:40.726518+0000","last_fullsized":"2024-04-16T16:14:40.726518+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:46.463986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.005261+0000","last_change":"2024-04-16T16:14:39.681884+0000","last_active":"2024-04-16T16:14:41.005261+0000","last_peered":"2024-04-16T16:14:41.005261+0000","last_clean":"2024-04-16T16:14:41.005261+0000","last_became_active":"2024-04-16T16:14:39.681629+0000","last_became_peered":"2024-04-16T16:14:39.681629+0000","last_unstale":"2024-04-16T16:14:41.005261+0000","last_undegraded":"2024-04-16T16:14:41.005261+0000","last_fullsized":"2024-04-16T16:14:41.005261+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:19:48.324320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996523+0000","last_change":"2024-04-16T16:14:38.671342+0000","last_active":"2024-04-16T16:14:40.996523+0000","last_peered":"2024-04-16T16:14:40.996523+0000","last_clean":"2024-04-16T16:14:40.996523+0000","last_became_active":"2024-04-16T16:14:38.670918+0000","last_became_peered":"2024-04-16T16:14:38.670918+0000","last_unstale":"2024-04-16T16:14:40.996523+0000","last_undegraded":"2024-04-16T16:14:40.996523+0000","last_fullsized":"2024-04-16T16:14:40.996523+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:08:32.600681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004216+0000","last_change":"2024-04-16T16:14:39.681058+0000","last_active":"2024-04-16T16:14:41.004216+0000","last_peered":"2024-04-16T16:14:41.004216+0000","last_clean":"2024-04-16T16:14:41.004216+0000","last_became_active":"2024-04-16T16:14:39.678280+0000","last_became_peered":"2024-04-16T16:14:39.678280+0000","last_unstale":"2024-04-16T16:14:41.004216+0000","last_undegraded":"2024-04-16T16:14:41.004216+0000","last_fullsized":"2024-04-16T16:14:41.004216+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:34.458058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290500+0000","last_change":"2024-04-16T16:14:38.668645+0000","last_active":"2024-04-16T16:14:41.290500+0000","last_peered":"2024-04-16T16:14:41.290500+0000","last_clean":"2024-04-16T16:14:41.290500+0000","last_became_active":"2024-04-16T16:14:38.668376+0000","last_became_peered":"2024-04-16T16:14:38.668376+0000","last_unstale":"2024-04-16T16:14:41.290500+0000","last_undegraded":"2024-04-16T16:14:41.290500+0000","last_fullsized":"2024-04-16T16:14:41.290500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:33.445805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690968+0000","last_change":"2024-04-16T16:14:39.697865+0000","last_active":"2024-04-16T16:14:40.690968+0000","last_peered":"2024-04-16T16:14:40.690968+0000","last_clean":"2024-04-16T16:14:40.690968+0000","last_became_active":"2024-04-16T16:14:39.696925+0000","last_became_peered":"2024-04-16T16:14:39.696925+0000","last_unstale":"2024-04-16T16:14:40.690968+0000","last_undegraded":"2024-04-16T16:14:40.690968+0000","last_fullsized":"2024-04-16T16:14:40.690968+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:27:36.166180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.689554+0000","last_change":"2024-04-16T16:14:38.668164+0000","last_active":"2024-04-16T16:14:40.689554+0000","last_peered":"2024-04-16T16:14:40.689554+0000","last_clean":"2024-04-16T16:14:40.689554+0000","last_became_active":"2024-04-16T16:14:38.667856+0000","last_became_peered":"2024-04-16T16:14:38.667856+0000","last_unstale":"2024-04-16T16:14:40.689554+0000","last_undegraded":"2024-04-16T16:14:40.689554+0000","last_fullsized":"2024-04-16T16:14:40.689554+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:16.441878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.004964+0000","last_change":"2024-04-16T16:14:39.694889+0000","last_active":"2024-04-16T16:14:41.004964+0000","last_peered":"2024-04-16T16:14:41.004964+0000","last_clean":"2024-04-16T16:14:41.004964+0000","last_became_active":"2024-04-16T16:14:39.694130+0000","last_became_peered":"2024-04-16T16:14:39.694130+0000","last_unstale":"2024-04-16T16:14:41.004964+0000","last_undegraded":"2024-04-16T16:14:41.004964+0000","last_fullsized":"2024-04-16T16:14:41.004964+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:08:33.612934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.292617+0000","last_change":"2024-04-16T16:14:38.674547+0000","last_active":"2024-04-16T16:14:41.292617+0000","last_peered":"2024-04-16T16:14:41.292617+0000","last_clean":"2024-04-16T16:14:41.292617+0000","last_became_active":"2024-04-16T16:14:38.674247+0000","last_became_peered":"2024-04-16T16:14:38.674247+0000","last_unstale":"2024-04-16T16:14:41.292617+0000","last_undegraded":"2024-04-16T16:14:41.292617+0000","last_fullsized":"2024-04-16T16:14:41.292617+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:25:42.846200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.692158+0000","last_change":"2024-04-16T16:14:39.709800+0000","last_active":"2024-04-16T16:14:40.692158+0000","last_peered":"2024-04-16T16:14:40.692158+0000","last_clean":"2024-04-16T16:14:40.692158+0000","last_became_active":"2024-04-16T16:14:39.709509+0000","last_became_peered":"2024-04-16T16:14:39.709509+0000","last_unstale":"2024-04-16T16:14:40.692158+0000","last_undegraded":"2024-04-16T16:14:40.692158+0000","last_fullsized":"2024-04-16T16:14:40.692158+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:02:01.054178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.727281+0000","last_change":"2024-04-16T16:14:38.669482+0000","last_active":"2024-04-16T16:14:40.727281+0000","last_peered":"2024-04-16T16:14:40.727281+0000","last_clean":"2024-04-16T16:14:40.727281+0000","last_became_active":"2024-04-16T16:14:38.669175+0000","last_became_peered":"2024-04-16T16:14:38.669175+0000","last_unstale":"2024-04-16T16:14:40.727281+0000","last_undegraded":"2024-04-16T16:14:40.727281+0000","last_fullsized":"2024-04-16T16:14:40.727281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:25:42.846200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.726937+0000","last_change":"2024-04-16T16:14:39.684854+0000","last_active":"2024-04-16T16:14:40.726937+0000","last_peered":"2024-04-16T16:14:40.726937+0000","last_clean":"2024-04-16T16:14:40.726937+0000","last_became_active":"2024-04-16T16:14:39.684524+0000","last_became_peered":"2024-04-16T16:14:39.684524+0000","last_unstale":"2024-04-16T16:14:40.726937+0000","last_undegraded":"2024-04-16T16:14:40.726937+0000","last_fullsized":"2024-04-16T16:14:40.726937+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:36.323418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291797+0000","last_change":"2024-04-16T16:14:38.675135+0000","last_active":"2024-04-16T16:14:41.291797+0000","last_peered":"2024-04-16T16:14:41.291797+0000","last_clean":"2024-04-16T16:14:41.291797+0000","last_became_active":"2024-04-16T16:14:38.674821+0000","last_became_peered":"2024-04-16T16:14:38.674821+0000","last_unstale":"2024-04-16T16:14:41.291797+0000","last_undegraded":"2024-04-16T16:14:41.291797+0000","last_fullsized":"2024-04-16T16:14:41.291797+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:18:01.189685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996428+0000","last_change":"2024-04-16T16:14:39.692135+0000","last_active":"2024-04-16T16:14:40.996428+0000","last_peered":"2024-04-16T16:14:40.996428+0000","last_clean":"2024-04-16T16:14:40.996428+0000","last_became_active":"2024-04-16T16:14:39.691817+0000","last_became_peered":"2024-04-16T16:14:39.691817+0000","last_unstale":"2024-04-16T16:14:40.996428+0000","last_undegraded":"2024-04-16T16:14:40.996428+0000","last_fullsized":"2024-04-16T16:14:40.996428+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:17.454131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.726901+0000","last_change":"2024-04-16T16:14:38.669656+0000","last_active":"2024-04-16T16:14:40.726901+0000","last_peered":"2024-04-16T16:14:40.726901+0000","last_clean":"2024-04-16T16:14:40.726901+0000","last_became_active":"2024-04-16T16:14:38.669304+0000","last_became_peered":"2024-04-16T16:14:38.669304+0000","last_unstale":"2024-04-16T16:14:40.726901+0000","last_undegraded":"2024-04-16T16:14:40.726901+0000","last_fullsized":"2024-04-16T16:14:40.726901+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:26:06.671145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.726817+0000","last_change":"2024-04-16T16:14:39.692738+0000","last_active":"2024-04-16T16:14:40.726817+0000","last_peered":"2024-04-16T16:14:40.726817+0000","last_clean":"2024-04-16T16:14:40.726817+0000","last_became_active":"2024-04-16T16:14:39.692443+0000","last_became_peered":"2024-04-16T16:14:39.692443+0000","last_unstale":"2024-04-16T16:14:40.726817+0000","last_undegraded":"2024-04-16T16:14:40.726817+0000","last_fullsized":"2024-04-16T16:14:40.726817+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:33.368730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.291411+0000","last_change":"2024-04-16T16:14:38.673858+0000","last_active":"2024-04-16T16:14:41.291411+0000","last_peered":"2024-04-16T16:14:41.291411+0000","last_clean":"2024-04-16T16:14:41.291411+0000","last_became_active":"2024-04-16T16:14:38.673531+0000","last_became_peered":"2024-04-16T16:14:38.673531+0000","last_unstale":"2024-04-16T16:14:41.291411+0000","last_undegraded":"2024-04-16T16:14:41.291411+0000","last_fullsized":"2024-04-16T16:14:41.291411+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:26:06.671145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.684803+0000","last_change":"2024-04-16T16:14:39.700548+0000","last_active":"2024-04-16T16:14:40.684803+0000","last_peered":"2024-04-16T16:14:40.684803+0000","last_clean":"2024-04-16T16:14:40.684803+0000","last_became_active":"2024-04-16T16:14:39.700278+0000","last_became_peered":"2024-04-16T16:14:39.700278+0000","last_unstale":"2024-04-16T16:14:40.684803+0000","last_undegraded":"2024-04-16T16:14:40.684803+0000","last_fullsized":"2024-04-16T16:14:40.684803+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:27:36.166180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290796+0000","last_change":"2024-04-16T16:14:38.671160+0000","last_active":"2024-04-16T16:14:41.290796+0000","last_peered":"2024-04-16T16:14:41.290796+0000","last_clean":"2024-04-16T16:14:41.290796+0000","last_became_active":"2024-04-16T16:14:38.670444+0000","last_became_peered":"2024-04-16T16:14:38.670444+0000","last_unstale":"2024-04-16T16:14:41.290796+0000","last_undegraded":"2024-04-16T16:14:41.290796+0000","last_fullsized":"2024-04-16T16:14:41.290796+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:06:15.287730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:41.290655+0000","last_change":"2024-04-16T16:14:39.701562+0000","last_active":"2024-04-16T16:14:41.290655+0000","last_peered":"2024-04-16T16:14:41.290655+0000","last_clean":"2024-04-16T16:14:41.290655+0000","last_became_active":"2024-04-16T16:14:39.697627+0000","last_became_peered":"2024-04-16T16:14:39.697627+0000","last_unstale":"2024-04-16T16:14:41.290655+0000","last_undegraded":"2024-04-16T16:14:41.290655+0000","last_fullsized":"2024-04-16T16:14:41.290655+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:44:47.910196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.685143+0000","last_change":"2024-04-16T16:14:38.664663+0000","last_active":"2024-04-16T16:14:40.685143+0000","last_peered":"2024-04-16T16:14:40.685143+0000","last_clean":"2024-04-16T16:14:40.685143+0000","last_became_active":"2024-04-16T16:14:38.663340+0000","last_became_peered":"2024-04-16T16:14:38.663340+0000","last_unstale":"2024-04-16T16:14:40.685143+0000","last_undegraded":"2024-04-16T16:14:40.685143+0000","last_fullsized":"2024-04-16T16:14:40.685143+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:31:34.396267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690125+0000","last_change":"2024-04-16T16:14:39.692510+0000","last_active":"2024-04-16T16:14:40.690125+0000","last_peered":"2024-04-16T16:14:40.690125+0000","last_clean":"2024-04-16T16:14:40.690125+0000","last_became_active":"2024-04-16T16:14:39.691697+0000","last_became_peered":"2024-04-16T16:14:39.691697+0000","last_unstale":"2024-04-16T16:14:40.690125+0000","last_undegraded":"2024-04-16T16:14:40.690125+0000","last_fullsized":"2024-04-16T16:14:40.690125+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:38.624558+0000","last_clean_scrub_stamp":"2024-04-16T16:14:38.624558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:09:23.282939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.996063+0000","last_change":"2024-04-16T16:14:38.672045+0000","last_active":"2024-04-16T16:14:40.996063+0000","last_peered":"2024-04-16T16:14:40.996063+0000","last_clean":"2024-04-16T16:14:40.996063+0000","last_became_active":"2024-04-16T16:14:38.671755+0000","last_became_peered":"2024-04-16T16:14:38.671755+0000","last_unstale":"2024-04-16T16:14:40.996063+0000","last_undegraded":"2024-04-16T16:14:40.996063+0000","last_fullsized":"2024-04-16T16:14:40.996063+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:14:37.612306+0000","last_clean_scrub_stamp":"2024-04-16T16:14:37.612306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:46.463986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":196,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-16T16:14:40.690537+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:14:40.690537+0000","last_peered":"2024-04-16T16:14:40.690537+0000","last_clean":"2024-04-16T16:14:40.690537+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:14:40.690537+0000","last_undegraded":"2024-04-16T16:14:40.690537+0000","last_fullsized":"2024-04-16T16:14:40.690537+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":983040,"data_stored":741737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":151895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":151951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":151383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":152913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":389120,"data_stored":151349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":152913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":978944,"data_stored":741647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:14:49.846 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-16T16:14:49.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-16T16:14:50.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.122+0000 7ff4cb800640 1 Processor -- start 2024-04-16T16:14:50.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.122+0000 7ff4cb800640 1 -- start start 2024-04-16T16:14:50.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.124+0000 7ff4cb800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4c415c7c0 0x7ff4c4072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:50.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.124+0000 7ff4cb800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff4c40b4930 0x7ff4c40d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:50.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.124+0000 7ff4cb800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff4c40d0f10 0x7ff4c406f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:50.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.124+0000 7ff4cb800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c40b52e0 con 0x7ff4c40b4930 2024-04-16T16:14:50.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.124+0000 7ff4cb800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c40d0260 con 0x7ff4c40d0f10 2024-04-16T16:14:50.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.124+0000 7ff4cb800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c40d1380 con 0x7ff4c415c7c0 2024-04-16T16:14:50.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.124+0000 7ff4c3e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff4c40b4930 0x7ff4c40d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:50.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.124+0000 7ff4c3e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff4c40b4930 0x7ff4c40d09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50682/0 (socket says 172.21.15.45:50682) 2024-04-16T16:14:50.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.124+0000 7ff4c3e00640 1 -- 172.21.15.45:0/3782373399 learned_addr learned my addr 172.21.15.45:0/3782373399 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:50.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.125+0000 7ff4c9800640 1 --2- 172.21.15.45:0/3782373399 >> v2:172.21.15.103:3300/0 conn(0x7ff4c40d0f10 0x7ff4c406f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:50.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.125+0000 7ff4c8e00640 1 --2- 172.21.15.45:0/3782373399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4c415c7c0 0x7ff4c4072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:50.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.125+0000 7ff4c3e00640 1 -- 172.21.15.45:0/3782373399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4c415c7c0 msgr2=0x7ff4c4072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:50.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.125+0000 7ff4c3e00640 1 --2- 172.21.15.45:0/3782373399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4c415c7c0 0x7ff4c4072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.125+0000 7ff4c3e00640 1 -- 172.21.15.45:0/3782373399 >> v2:172.21.15.103:3300/0 conn(0x7ff4c40d0f10 msgr2=0x7ff4c406f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:50.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.125+0000 7ff4c3e00640 1 --2- 172.21.15.45:0/3782373399 >> v2:172.21.15.103:3300/0 conn(0x7ff4c40d0f10 0x7ff4c406f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.125+0000 7ff4c3e00640 1 -- 172.21.15.45:0/3782373399 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c40b5530 con 0x7ff4c40b4930 2024-04-16T16:14:50.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.126+0000 7ff4c3e00640 1 --2- 172.21.15.45:0/3782373399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c40b4930 0x7ff4c40d09d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40afd50 tx=0x7ff4b002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f187418307a2c2c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:50.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.126+0000 7ff4c3400640 1 -- 172.21.15.45:0/3782373399 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b0038070 con 0x7ff4c40b4930 2024-04-16T16:14:50.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.126+0000 7ff4c3400640 1 -- 172.21.15.45:0/3782373399 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b0031740 con 0x7ff4c40b4930 2024-04-16T16:14:50.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.126+0000 7ff4c3400640 1 -- 172.21.15.45:0/3782373399 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b002f050 con 0x7ff4c40b4930 2024-04-16T16:14:50.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.127+0000 7ff4cb800640 1 -- 172.21.15.45:0/3782373399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c40b4930 msgr2=0x7ff4c40d09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:50.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.127+0000 7ff4cb800640 1 --2- 172.21.15.45:0/3782373399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c40b4930 0x7ff4c40d09d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40afd50 tx=0x7ff4b002b7e0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.127+0000 7ff4cb800640 1 -- 172.21.15.45:0/3782373399 shutdown_connections 2024-04-16T16:14:50.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.127+0000 7ff4cb800640 1 --2- 172.21.15.45:0/3782373399 >> v2:172.21.15.103:3300/0 conn(0x7ff4c40d0f10 0x7ff4c406f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.127+0000 7ff4cb800640 1 --2- 172.21.15.45:0/3782373399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c40b4930 0x7ff4c40d09d0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.127+0000 7ff4cb800640 1 --2- 172.21.15.45:0/3782373399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4c415c7c0 0x7ff4c4072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.127+0000 7ff4cb800640 1 -- 172.21.15.45:0/3782373399 >> 172.21.15.45:0/3782373399 conn(0x7ff4c40b3970 msgr2=0x7ff4c40b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:50.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.127+0000 7ff4cb800640 1 -- 172.21.15.45:0/3782373399 shutdown_connections 2024-04-16T16:14:50.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.127+0000 7ff4cb800640 1 -- 172.21.15.45:0/3782373399 wait complete. 2024-04-16T16:14:50.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.130+0000 7ff4cb800640 1 Processor -- start 2024-04-16T16:14:50.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.130+0000 7ff4cb800640 1 -- start start 2024-04-16T16:14:50.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.131+0000 7ff4cb800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4c40b4930 0x7ff4c415aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:50.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.131+0000 7ff4cb800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4c40d0f10 0x7ff4c414a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:50.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.131+0000 7ff4cb800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c415c7c0 0x7ff4c415e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:50.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.131+0000 7ff4cb800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c40d04b0 con 0x7ff4c415c7c0 2024-04-16T16:14:50.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.131+0000 7ff4cb800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c4073350 con 0x7ff4c40b4930 2024-04-16T16:14:50.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.131+0000 7ff4cb800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c406f6a0 con 0x7ff4c40d0f10 2024-04-16T16:14:50.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.132+0000 7ff4c9800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c415c7c0 0x7ff4c415e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:50.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.132+0000 7ff4c9800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c415c7c0 0x7ff4c415e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50698/0 (socket says 172.21.15.45:50698) 2024-04-16T16:14:50.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.132+0000 7ff4c9800640 1 -- 172.21.15.45:0/3845921339 learned_addr learned my addr 172.21.15.45:0/3845921339 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:50.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.132+0000 7ff4c3e00640 1 --2- 172.21.15.45:0/3845921339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4c40d0f10 0x7ff4c414a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:50.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.132+0000 7ff4c8e00640 1 --2- 172.21.15.45:0/3845921339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4c40b4930 0x7ff4c415aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:50.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.132+0000 7ff4c9800640 1 -- 172.21.15.45:0/3845921339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4c40d0f10 msgr2=0x7ff4c414a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:50.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.132+0000 7ff4c9800640 1 --2- 172.21.15.45:0/3845921339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4c40d0f10 0x7ff4c414a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.132+0000 7ff4c9800640 1 -- 172.21.15.45:0/3845921339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4c40b4930 msgr2=0x7ff4c415aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:50.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.132+0000 7ff4c9800640 1 --2- 172.21.15.45:0/3845921339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4c40b4930 0x7ff4c415aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.132+0000 7ff4c9800640 1 -- 172.21.15.45:0/3845921339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c414a9e0 con 0x7ff4c415c7c0 2024-04-16T16:14:50.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.133+0000 7ff4c9800640 1 --2- 172.21.15.45:0/3845921339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c415c7c0 0x7ff4c415e190 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc007e40 tx=0x7ff4bc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:50.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.133+0000 7ff4c0c00640 1 -- 172.21.15.45:0/3845921339 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc014620 con 0x7ff4c415c7c0 2024-04-16T16:14:50.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.133+0000 7ff4c0c00640 1 -- 172.21.15.45:0/3845921339 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4bc019070 con 0x7ff4c415c7c0 2024-04-16T16:14:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.133+0000 7ff4cb800640 1 -- 172.21.15.45:0/3845921339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c4157ec0 con 0x7ff4c415c7c0 2024-04-16T16:14:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.133+0000 7ff4c0c00640 1 -- 172.21.15.45:0/3845921339 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc014a20 con 0x7ff4c415c7c0 2024-04-16T16:14:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.133+0000 7ff4cb800640 1 -- 172.21.15.45:0/3845921339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c4158480 con 0x7ff4c415c7c0 2024-04-16T16:14:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.134+0000 7ff4cb800640 1 -- 172.21.15.45:0/3845921339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4c4075780 con 0x7ff4c415c7c0 2024-04-16T16:14:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.136+0000 7ff4c0c00640 1 -- 172.21.15.45:0/3845921339 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff4bc014bc0 con 0x7ff4c415c7c0 2024-04-16T16:14:50.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.137+0000 7ff4c0c00640 1 --2- 172.21.15.45:0/3845921339 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff4a406c8f0 0x7ff4a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:50.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.137+0000 7ff4c0c00640 1 -- 172.21.15.45:0/3845921339 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7ff4bc08c3b0 con 0x7ff4c415c7c0 2024-04-16T16:14:50.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.138+0000 7ff4c8e00640 1 --2- 172.21.15.45:0/3845921339 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff4a406c8f0 0x7ff4a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:50.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.139+0000 7ff4c8e00640 1 --2- 172.21.15.45:0/3845921339 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff4a406c8f0 0x7ff4a406edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4006fd0 tx=0x7ff4b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.141+0000 7ff4c0c00640 1 -- 172.21.15.45:0/3845921339 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff4bc05a870 con 0x7ff4c415c7c0 2024-04-16T16:14:50.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.428+0000 7ff4cb800640 1 -- 172.21.15.45:0/3845921339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7ff4c406f2c0 con 0x7ff4c415c7c0 2024-04-16T16:14:50.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.432+0000 7ff4c0c00640 1 -- 172.21.15.45:0/3845921339 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7ff4bc05a170 con 0x7ff4c415c7c0 2024-04-16T16:14:50.432 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-04-16T16:14:50.432 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ== 2024-04-16T16:14:50.432 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:14:50.432 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:14:50.433 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-16T16:14:50.433 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2024-04-16T16:14:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.435+0000 7ff4cb800640 1 -- 172.21.15.45:0/3845921339 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff4a406c8f0 msgr2=0x7ff4a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.435+0000 7ff4cb800640 1 --2- 172.21.15.45:0/3845921339 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff4a406c8f0 0x7ff4a406edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4006fd0 tx=0x7ff4b4008040 comp rx=0 tx=0).stop 2024-04-16T16:14:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.435+0000 7ff4cb800640 1 -- 172.21.15.45:0/3845921339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c415c7c0 msgr2=0x7ff4c415e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.435+0000 7ff4cb800640 1 --2- 172.21.15.45:0/3845921339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c415c7c0 0x7ff4c415e190 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc007e40 tx=0x7ff4bc00e4b0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.435+0000 7ff4cb800640 1 -- 172.21.15.45:0/3845921339 shutdown_connections 2024-04-16T16:14:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.435+0000 7ff4cb800640 1 --2- 172.21.15.45:0/3845921339 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff4a406c8f0 0x7ff4a406edb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.435+0000 7ff4cb800640 1 --2- 172.21.15.45:0/3845921339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c415c7c0 0x7ff4c415e190 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.435+0000 7ff4cb800640 1 --2- 172.21.15.45:0/3845921339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4c40d0f10 0x7ff4c414a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.435+0000 7ff4cb800640 1 --2- 172.21.15.45:0/3845921339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4c40b4930 0x7ff4c415aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.435+0000 7ff4cb800640 1 -- 172.21.15.45:0/3845921339 >> 172.21.15.45:0/3845921339 conn(0x7ff4c40b3970 msgr2=0x7ff4c4072a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.435+0000 7ff4cb800640 1 -- 172.21.15.45:0/3845921339 shutdown_connections 2024-04-16T16:14:50.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.435+0000 7ff4cb800640 1 -- 172.21.15.45:0/3845921339 wait complete. 2024-04-16T16:14:50.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:14:50.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.672+0000 7f353d600640 1 Processor -- start 2024-04-16T16:14:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.672+0000 7f353d600640 1 -- start start 2024-04-16T16:14:50.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.674+0000 7f353d600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3538161e90 0x7f35380cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:50.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.674+0000 7f353d600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3538162270 0x7f35380d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:50.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.674+0000 7f353d600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35380d3080 con 0x7f35380d52d0 2024-04-16T16:14:50.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.674+0000 7f353d600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f35380d2e30 con 0x7f3538161e90 2024-04-16T16:14:50.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.674+0000 7f353d600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f35380d3360 con 0x7f3538162270 2024-04-16T16:14:50.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.674+0000 7f3537e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3538161e90 0x7f35380cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:50.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.674+0000 7f353cc00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f35380d52d0 0x7f353806ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42382/0 (socket says 172.21.15.45:42382) 2024-04-16T16:14:50.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.674+0000 7f3537e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3538161e90 0x7f35380cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50714/0 (socket says 172.21.15.45:50714) 2024-04-16T16:14:50.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.674+0000 7f353cc00640 1 -- 172.21.15.45:0/1236013083 learned_addr learned my addr 172.21.15.45:0/1236013083 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:50.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.675+0000 7f352fe00640 1 --2- 172.21.15.45:0/1236013083 >> v2:172.21.15.103:3300/0 conn(0x7f3538162270 0x7f35380d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:50.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.675+0000 7f3537e00640 1 -- 172.21.15.45:0/1236013083 >> v2:172.21.15.103:3300/0 conn(0x7f3538162270 msgr2=0x7f35380d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:50.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.675+0000 7f3537e00640 1 --2- 172.21.15.45:0/1236013083 >> v2:172.21.15.103:3300/0 conn(0x7f3538162270 0x7f35380d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.675+0000 7f3537e00640 1 -- 172.21.15.45:0/1236013083 >> v1:172.21.15.45:6789/0 conn(0x7f35380d52d0 legacy=0x7f353806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:50.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.675+0000 7f3537e00640 1 -- 172.21.15.45:0/1236013083 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35380d3640 con 0x7f3538161e90 2024-04-16T16:14:50.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.675+0000 7f3537400640 1 -- 172.21.15.45:0/1236013083 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2338191202 0 0) 0x7f35380d3080 con 0x7f35380d52d0 2024-04-16T16:14:50.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.676+0000 7f3537e00640 1 --2- 172.21.15.45:0/1236013083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3538161e90 0x7f35380cf7d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f35280097a0 tx=0x7f352802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90371bc1632469e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:50.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.676+0000 7f3537400640 1 -- 172.21.15.45:0/1236013083 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528035020 con 0x7f3538161e90 2024-04-16T16:14:50.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.676+0000 7f3537400640 1 -- 172.21.15.45:0/1236013083 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3528031830 con 0x7f3538161e90 2024-04-16T16:14:50.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.676+0000 7f3537400640 1 -- 172.21.15.45:0/1236013083 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528031c00 con 0x7f3538161e90 2024-04-16T16:14:50.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.677+0000 7f353d600640 1 -- 172.21.15.45:0/1236013083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3538161e90 msgr2=0x7f35380cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.677+0000 7f353d600640 1 --2- 172.21.15.45:0/1236013083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3538161e90 0x7f35380cf7d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f35280097a0 tx=0x7f352802b8d0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.677+0000 7f353d600640 1 -- 172.21.15.45:0/1236013083 shutdown_connections 2024-04-16T16:14:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.677+0000 7f353d600640 1 --2- 172.21.15.45:0/1236013083 >> v2:172.21.15.103:3300/0 conn(0x7f3538162270 0x7f35380d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.677+0000 7f353d600640 1 --2- 172.21.15.45:0/1236013083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3538161e90 0x7f35380cf7d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.677+0000 7f353d600640 1 -- 172.21.15.45:0/1236013083 >> 172.21.15.45:0/1236013083 conn(0x7f35380d1940 msgr2=0x7f35380d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.677+0000 7f353d600640 1 -- 172.21.15.45:0/1236013083 shutdown_connections 2024-04-16T16:14:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.677+0000 7f353d600640 1 -- 172.21.15.45:0/1236013083 wait complete. 2024-04-16T16:14:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.679+0000 7f353d600640 1 Processor -- start 2024-04-16T16:14:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.679+0000 7f353d600640 1 -- start start 2024-04-16T16:14:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.680+0000 7f353d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35380d52d0 0x7f35381f8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.680+0000 7f353d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3538161e90 0x7f35380cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.680+0000 7f353d600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3538162270 0x7f353814e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.680+0000 7f353d600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f353806d550 con 0x7f3538162270 2024-04-16T16:14:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.680+0000 7f353d600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f35380cfd10 con 0x7f3538161e90 2024-04-16T16:14:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.680+0000 7f353d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f353806d6d0 con 0x7f35380d52d0 2024-04-16T16:14:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.680+0000 7f353cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3538162270 0x7f353814e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.680+0000 7f353cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3538162270 0x7f353814e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50720/0 (socket says 172.21.15.45:50720) 2024-04-16T16:14:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.680+0000 7f353cc00640 1 -- 172.21.15.45:0/609583462 learned_addr learned my addr 172.21.15.45:0/609583462 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.681+0000 7f352fe00640 1 --2- 172.21.15.45:0/609583462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3538161e90 0x7f35380cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.681+0000 7f3537e00640 1 --2- 172.21.15.45:0/609583462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35380d52d0 0x7f35381f8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.681+0000 7f353cc00640 1 -- 172.21.15.45:0/609583462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35380d52d0 msgr2=0x7f35381f8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.681+0000 7f353cc00640 1 --2- 172.21.15.45:0/609583462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35380d52d0 0x7f35381f8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.681+0000 7f353cc00640 1 -- 172.21.15.45:0/609583462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3538161e90 msgr2=0x7f35380cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.681+0000 7f353cc00640 1 --2- 172.21.15.45:0/609583462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3538161e90 0x7f35380cc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.681+0000 7f353cc00640 1 -- 172.21.15.45:0/609583462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35380cd320 con 0x7f3538162270 2024-04-16T16:14:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.681+0000 7f353cc00640 1 --2- 172.21.15.45:0/609583462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3538162270 0x7f353814e830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f353000c6d0 tx=0x7f353000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.682+0000 7f3534c00640 1 -- 172.21.15.45:0/609583462 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530018070 con 0x7f3538162270 2024-04-16T16:14:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.682+0000 7f3534c00640 1 -- 172.21.15.45:0/609583462 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3530004d60 con 0x7f3538162270 2024-04-16T16:14:50.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.682+0000 7f353d600640 1 -- 172.21.15.45:0/609583462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35380cd600 con 0x7f3538162270 2024-04-16T16:14:50.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.682+0000 7f3534c00640 1 -- 172.21.15.45:0/609583462 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353000a880 con 0x7f3538162270 2024-04-16T16:14:50.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.682+0000 7f353d600640 1 -- 172.21.15.45:0/609583462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35380cdbc0 con 0x7f3538162270 2024-04-16T16:14:50.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.683+0000 7f353d600640 1 -- 172.21.15.45:0/609583462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3538075780 con 0x7f3538162270 2024-04-16T16:14:50.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.684+0000 7f3534c00640 1 -- 172.21.15.45:0/609583462 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f353001b070 con 0x7f3538162270 2024-04-16T16:14:50.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.685+0000 7f3534c00640 1 --2- 172.21.15.45:0/609583462 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f351406c8f0 0x7f351406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:50.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.685+0000 7f3534c00640 1 -- 172.21.15.45:0/609583462 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f353008b2f0 con 0x7f3538162270 2024-04-16T16:14:50.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.686+0000 7f3537e00640 1 --2- 172.21.15.45:0/609583462 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f351406c8f0 0x7f351406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:50.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.687+0000 7f3537e00640 1 --2- 172.21.15.45:0/609583462 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f351406c8f0 0x7f351406edb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3528036000 tx=0x7f3528008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:50.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.690+0000 7f3534c00640 1 -- 172.21.15.45:0/609583462 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3530059710 con 0x7f3538162270 2024-04-16T16:14:50.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.940+0000 7f353d600640 1 -- 172.21.15.45:0/609583462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35380b9930 con 0x7f3538162270 2024-04-16T16:14:50.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.941+0000 7f3534c00640 1 -- 172.21.15.45:0/609583462 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7f3530059010 con 0x7f3538162270 2024-04-16T16:14:50.941 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:50.941 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/814837411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":814837411},{"type":"v1","addr":"172.21.15.103:6836","nonce":814837411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/401733314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":401733314},{"type":"v1","addr":"172.21.15.103:6837","nonce":401733314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5073,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/768816269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":768816269},{"type":"v1","addr":"172.21.15.45:6834","nonce":768816269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:14:39.132000+0000","modified":"2024-04-16T16:14:42.226029+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5079},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5079":{"gid":5079,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6835/2956262988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2956262988},{"type":"v1","addr":"172.21.15.45:6835","nonce":2956262988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5079,"qdb_cluster":[5079]},"id":2}]} 2024-04-16T16:14:50.942 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 16 2024-04-16T16:14:50.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.944+0000 7f353d600640 1 -- 172.21.15.45:0/609583462 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f351406c8f0 msgr2=0x7f351406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:50.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.944+0000 7f353d600640 1 --2- 172.21.15.45:0/609583462 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f351406c8f0 0x7f351406edb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3528036000 tx=0x7f3528008030 comp rx=0 tx=0).stop 2024-04-16T16:14:50.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.944+0000 7f353d600640 1 -- 172.21.15.45:0/609583462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3538162270 msgr2=0x7f353814e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:50.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.944+0000 7f353d600640 1 --2- 172.21.15.45:0/609583462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3538162270 0x7f353814e830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f353000c6d0 tx=0x7f353000cb90 comp rx=0 tx=0).stop 2024-04-16T16:14:50.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.944+0000 7f353d600640 1 -- 172.21.15.45:0/609583462 shutdown_connections 2024-04-16T16:14:50.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.944+0000 7f353d600640 1 --2- 172.21.15.45:0/609583462 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f351406c8f0 0x7f351406edb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.944+0000 7f353d600640 1 --2- 172.21.15.45:0/609583462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3538162270 0x7f353814e830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.944+0000 7f353d600640 1 --2- 172.21.15.45:0/609583462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3538161e90 0x7f35380cc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.944+0000 7f353d600640 1 --2- 172.21.15.45:0/609583462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35380d52d0 0x7f35381f8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:50.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.944+0000 7f353d600640 1 -- 172.21.15.45:0/609583462 >> 172.21.15.45:0/609583462 conn(0x7f35380d1940 msgr2=0x7f35380b9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:50.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.944+0000 7f353d600640 1 -- 172.21.15.45:0/609583462 shutdown_connections 2024-04-16T16:14:50.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:50.944+0000 7f353d600640 1 -- 172.21.15.45:0/609583462 wait complete. 2024-04-16T16:14:51.365 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:14:39.132000+0000', 'modified': '2024-04-16T16:14:42.226029+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5079}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5079': {'gid': 5079, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.45:6835/2956262988', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6833', 'nonce': 2956262988}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 2956262988}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5079, 'qdb_cluster': [5079]} 2024-04-16T16:14:51.365 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-16T16:14:51.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-16T16:14:51.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.599+0000 7f0194a00640 1 Processor -- start 2024-04-16T16:14:51.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.599+0000 7f0194a00640 1 -- start start 2024-04-16T16:14:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.601+0000 7f0194a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0190145230 0x7f019015bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.601+0000 7f0194a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f01900c5670 0x7f01900c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.601+0000 7f0194a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f019015d3b0 con 0x7f019015cbe0 2024-04-16T16:14:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.601+0000 7f0194a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f01900c6020 con 0x7f01900c5670 2024-04-16T16:14:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.601+0000 7f0194a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f019015dfc0 con 0x7f0190145230 2024-04-16T16:14:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.601+0000 7f018f200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f01900c5670 0x7f01900c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.601+0000 7f018f200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f01900c5670 0x7f01900c5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50728/0 (socket says 172.21.15.45:50728) 2024-04-16T16:14:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.601+0000 7f018f200640 1 -- 172.21.15.45:0/417282115 learned_addr learned my addr 172.21.15.45:0/417282115 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.601+0000 7f018e800640 1 --2- 172.21.15.45:0/417282115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0190145230 0x7f019015bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.602+0000 7f018f200640 1 -- 172.21.15.45:0/417282115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0190145230 msgr2=0x7f019015bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.602+0000 7f018f200640 1 --2- 172.21.15.45:0/417282115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0190145230 0x7f019015bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.602+0000 7f018f200640 1 -- 172.21.15.45:0/417282115 >> v1:172.21.15.45:6789/0 conn(0x7f019015cbe0 legacy=0x7f01900cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.602+0000 7f018de00640 1 -- 172.21.15.45:0/417282115 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2021542320 0 0) 0x7f019015d3b0 con 0x7f019015cbe0 2024-04-16T16:14:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.602+0000 7f018f200640 1 -- 172.21.15.45:0/417282115 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f019015d690 con 0x7f01900c5670 2024-04-16T16:14:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.603+0000 7f018f200640 1 --2- 172.21.15.45:0/417282115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01900c5670 0x7f01900c5ae0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0188009870 tx=0x7f018802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1bd110ec064b0e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.603+0000 7f018de00640 1 -- 172.21.15.45:0/417282115 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188035020 con 0x7f01900c5670 2024-04-16T16:14:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.603+0000 7f018de00640 1 -- 172.21.15.45:0/417282115 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0188031640 con 0x7f01900c5670 2024-04-16T16:14:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.603+0000 7f018de00640 1 -- 172.21.15.45:0/417282115 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f018802d040 con 0x7f01900c5670 2024-04-16T16:14:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.604+0000 7f0194a00640 1 -- 172.21.15.45:0/417282115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01900c5670 msgr2=0x7f01900c5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.604+0000 7f0194a00640 1 --2- 172.21.15.45:0/417282115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01900c5670 0x7f01900c5ae0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0188009870 tx=0x7f018802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.604+0000 7f0194a00640 1 -- 172.21.15.45:0/417282115 shutdown_connections 2024-04-16T16:14:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.604+0000 7f0194a00640 1 --2- 172.21.15.45:0/417282115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01900c5670 0x7f01900c5ae0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.604+0000 7f0194a00640 1 --2- 172.21.15.45:0/417282115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0190145230 0x7f019015bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.604+0000 7f0194a00640 1 -- 172.21.15.45:0/417282115 >> 172.21.15.45:0/417282115 conn(0x7f019006b930 msgr2=0x7f019006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.604+0000 7f0194a00640 1 -- 172.21.15.45:0/417282115 shutdown_connections 2024-04-16T16:14:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.604+0000 7f0194a00640 1 -- 172.21.15.45:0/417282115 wait complete. 2024-04-16T16:14:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.606+0000 7f0194a00640 1 Processor -- start 2024-04-16T16:14:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.606+0000 7f0194a00640 1 -- start start 2024-04-16T16:14:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.607+0000 7f0194a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01900c5670 0x7f019015b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.607+0000 7f0194a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0190145230 0x7f01900cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.607+0000 7f0194a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f019015cbe0 0x7f01900ce180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.607+0000 7f0194a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f019015d000 con 0x7f0190145230 2024-04-16T16:14:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.607+0000 7f0194a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f019015dc50 con 0x7f019015cbe0 2024-04-16T16:14:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.607+0000 7f0194a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f01900cd0e0 con 0x7f01900c5670 2024-04-16T16:14:51.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.607+0000 7f0185e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0190145230 0x7f01900cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.607+0000 7f0185e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0190145230 0x7f01900cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50738/0 (socket says 172.21.15.45:50738) 2024-04-16T16:14:51.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.607+0000 7f0185e00640 1 -- 172.21.15.45:0/895861404 learned_addr learned my addr 172.21.15.45:0/895861404 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:51.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.608+0000 7f0185e00640 1 -- 172.21.15.45:0/895861404 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01900c5670 msgr2=0x7f019015b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:51.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.608+0000 7f018e800640 1 --2- 172.21.15.45:0/895861404 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01900c5670 0x7f019015b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.608+0000 7f018f200640 1 --2- 172.21.15.45:0/895861404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f019015cbe0 0x7f01900ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.608+0000 7f0185e00640 1 --2- 172.21.15.45:0/895861404 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01900c5670 0x7f019015b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.608+0000 7f0185e00640 1 -- 172.21.15.45:0/895861404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f019015cbe0 msgr2=0x7f01900ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:51.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.608+0000 7f0185e00640 1 --2- 172.21.15.45:0/895861404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f019015cbe0 0x7f01900ce180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.608+0000 7f0185e00640 1 -- 172.21.15.45:0/895861404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01900cd3c0 con 0x7f0190145230 2024-04-16T16:14:51.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.608+0000 7f0185e00640 1 --2- 172.21.15.45:0/895861404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0190145230 0x7f01900cbee0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f017800c7f0 tx=0x7f017800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:51.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.608+0000 7f0187400640 1 -- 172.21.15.45:0/895861404 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178017070 con 0x7f0190145230 2024-04-16T16:14:51.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.608+0000 7f0187400640 1 -- 172.21.15.45:0/895861404 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0178003da0 con 0x7f0190145230 2024-04-16T16:14:51.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.608+0000 7f0187400640 1 -- 172.21.15.45:0/895861404 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178005290 con 0x7f0190145230 2024-04-16T16:14:51.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.608+0000 7f0194a00640 1 -- 172.21.15.45:0/895861404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01900cd6a0 con 0x7f0190145230 2024-04-16T16:14:51.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.609+0000 7f0194a00640 1 -- 172.21.15.45:0/895861404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01900cdc60 con 0x7f0190145230 2024-04-16T16:14:51.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.611+0000 7f0187400640 1 -- 172.21.15.45:0/895861404 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f017801a070 con 0x7f0190145230 2024-04-16T16:14:51.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.612+0000 7f0187400640 1 --2- 172.21.15.45:0/895861404 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f016c06c8f0 0x7f016c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.612+0000 7f0187400640 1 -- 172.21.15.45:0/895861404 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f017808b290 con 0x7f0190145230 2024-04-16T16:14:51.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.612+0000 7f0198058740 1 Processor -- start 2024-04-16T16:14:51.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.612+0000 7f018e800640 1 --2- 172.21.15.45:0/895861404 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f016c06c8f0 0x7f016c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.613+0000 7f0198058740 1 -- start start 2024-04-16T16:14:51.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.613+0000 7f0198058740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71319d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.613+0000 7f0198058740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac7112c70 0x564ac70eb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.613+0000 7f0198058740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac70ebab0 0x564ac70e9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.614+0000 7f0198058740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564ac70509c0 con 0x564ac7112890 2024-04-16T16:14:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.614+0000 7f0198058740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564ac6ff7560 con 0x564ac7112c70 2024-04-16T16:14:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.614+0000 7f0198058740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564ac704c980 con 0x564ac70ebab0 2024-04-16T16:14:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.614+0000 7f018e800640 1 --2- 172.21.15.45:0/895861404 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f016c06c8f0 0x7f016c06edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0180006fd0 tx=0x7f0180008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.614+0000 7f018e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71319d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.614+0000 7f018e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71319d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50752/0 (socket says 172.21.15.45:50752) 2024-04-16T16:14:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.614+0000 7f018e800640 1 -- 172.21.15.45:0/1967647575 learned_addr learned my addr 172.21.15.45:0/1967647575 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.614+0000 7f0185e00640 1 --2- 172.21.15.45:0/1967647575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac7112c70 0x564ac70eb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.614+0000 7f018f200640 1 --2- 172.21.15.45:0/1967647575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac70ebab0 0x564ac70e9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.614+0000 7f018e800640 1 -- 172.21.15.45:0/1967647575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac70ebab0 msgr2=0x564ac70e9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.614+0000 7f018e800640 1 --2- 172.21.15.45:0/1967647575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac70ebab0 0x564ac70e9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.614+0000 7f018e800640 1 -- 172.21.15.45:0/1967647575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac7112c70 msgr2=0x564ac70eb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.614+0000 7f018e800640 1 --2- 172.21.15.45:0/1967647575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac7112c70 0x564ac70eb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.614+0000 7f018e800640 1 -- 172.21.15.45:0/1967647575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ac6d632d0 con 0x564ac7112890 2024-04-16T16:14:51.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.615+0000 7f018e800640 1 --2- 172.21.15.45:0/1967647575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71319d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f018000e8f0 tx=0x7f0180011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0941facb73e763 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:51.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.615+0000 7f017f000640 1 -- 172.21.15.45:0/1967647575 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0180025070 con 0x564ac7112890 2024-04-16T16:14:51.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.615+0000 7f017f000640 1 -- 172.21.15.45:0/1967647575 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0180015640 con 0x564ac7112890 2024-04-16T16:14:51.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.615+0000 7f017f000640 1 -- 172.21.15.45:0/1967647575 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f018000f040 con 0x564ac7112890 2024-04-16T16:14:51.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.616+0000 7f0198058740 1 -- 172.21.15.45:0/1967647575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 msgr2=0x564ac71319d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:51.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.616+0000 7f0198058740 1 --2- 172.21.15.45:0/1967647575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71319d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f018000e8f0 tx=0x7f0180011000 comp rx=0 tx=0).stop 2024-04-16T16:14:51.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.616+0000 7f0198058740 1 -- 172.21.15.45:0/1967647575 shutdown_connections 2024-04-16T16:14:51.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.616+0000 7f0198058740 1 --2- 172.21.15.45:0/1967647575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac70ebab0 0x564ac70e9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.616+0000 7f0198058740 1 --2- 172.21.15.45:0/1967647575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac7112c70 0x564ac70eb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.616+0000 7f0198058740 1 --2- 172.21.15.45:0/1967647575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71319d0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.616+0000 7f0198058740 1 -- 172.21.15.45:0/1967647575 >> 172.21.15.45:0/1967647575 conn(0x564ac70e7b50 msgr2=0x564ac71306d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:51.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.616+0000 7f0198058740 1 -- 172.21.15.45:0/1967647575 shutdown_connections 2024-04-16T16:14:51.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.616+0000 7f0198058740 1 -- 172.21.15.45:0/1967647575 wait complete. 2024-04-16T16:14:51.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.617+0000 7f0198058740 1 -- start start 2024-04-16T16:14:51.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.617+0000 7f0198058740 1 Processor -- start 2024-04-16T16:14:51.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.618+0000 7f0198058740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71043e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.618+0000 7f0198058740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac7112c70 0x564ac71062f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.618+0000 7f0198058740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac71049f0 0x564ac7106830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.618+0000 7f0198058740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564ac70e5980 con 0x564ac7112890 2024-04-16T16:14:51.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.618+0000 7f0198058740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564ac700a920 con 0x564ac71049f0 2024-04-16T16:14:51.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.618+0000 7f0198058740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564ac70ebf30 con 0x564ac7112c70 2024-04-16T16:14:51.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.618+0000 7f0198058740 10 client.0 _ll_register_callbacks cb 0x564ac700a310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:14:51.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.618+0000 7f018e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71043e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.618+0000 7f018e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71043e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50764/0 (socket says 172.21.15.45:50764) 2024-04-16T16:14:51.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.618+0000 7f018e800640 1 -- 172.21.15.45:0/2782443671 learned_addr learned my addr 172.21.15.45:0/2782443671 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:51.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.619+0000 7f0185e00640 1 --2- 172.21.15.45:0/2782443671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac7112c70 0x564ac71062f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.619+0000 7f018f200640 1 --2- 172.21.15.45:0/2782443671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac71049f0 0x564ac7106830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.619+0000 7f018e800640 1 -- 172.21.15.45:0/2782443671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac7112c70 msgr2=0x564ac71062f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:51.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.619+0000 7f018e800640 1 --2- 172.21.15.45:0/2782443671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac7112c70 0x564ac71062f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.619+0000 7f018e800640 1 -- 172.21.15.45:0/2782443671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac71049f0 msgr2=0x564ac7106830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:51.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.619+0000 7f018e800640 1 --2- 172.21.15.45:0/2782443671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac71049f0 0x564ac7106830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.619+0000 7f018e800640 1 -- 172.21.15.45:0/2782443671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ac6e2d6b0 con 0x564ac7112890 2024-04-16T16:14:51.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.619+0000 7f0185e00640 1 --2- 172.21.15.45:0/2782443671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac7112c70 0x564ac71062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:51.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.619+0000 7f018e800640 1 --2- 172.21.15.45:0/2782443671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71043e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f01800154f0 tx=0x7f0180015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528c9f065207a6a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:51.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.619+0000 7f0173400640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:14:51.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.619+0000 7f0173400640 1 -- 172.21.15.45:0/2782443671 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0180025070 con 0x564ac7112890 2024-04-16T16:14:51.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.620+0000 7f0173400640 1 -- 172.21.15.45:0/2782443671 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f0180015d00 con 0x564ac7112890 2024-04-16T16:14:51.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.620+0000 7f0198058740 1 -- 172.21.15.45:0/2782443671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0180015640 con 0x564ac7112890 2024-04-16T16:14:51.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.620+0000 7f0173400640 1 -- 172.21.15.45:0/2782443671 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f0180037050 con 0x564ac7112890 2024-04-16T16:14:51.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.620+0000 7f0198058740 10 client.5295 fetch_fsmap learned FSMap version 16 2024-04-16T16:14:51.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.620+0000 7f0198058740 1 -- 172.21.15.45:0/2782443671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x564ac71333b0 con 0x564ac7112890 2024-04-16T16:14:51.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.620+0000 7f0173400640 1 -- 172.21.15.45:0/2782443671 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0180015640 con 0x564ac7112890 2024-04-16T16:14:51.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.620+0000 7f0173400640 1 -- 172.21.15.45:0/2782443671 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f018003a020 con 0x564ac7112890 2024-04-16T16:14:51.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.620+0000 7f0198058740 10 client.5295 fetch_fsmap finished waiting for FSMap version 16 2024-04-16T16:14:51.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.620+0000 7f0198058740 10 client.5295 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-16T16:14:51.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.620+0000 7f0198058740 20 client.5295 populate_metadata read hostname 'smithi045' 2024-04-16T16:14:51.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.621+0000 7f0198058740 1 --2- 172.21.15.45:0/2782443671 >> [v2:172.21.15.45:6833/2956262988,v1:172.21.15.45:6835/2956262988] conn(0x564ac7134200 0x564ac7144e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.621+0000 7f0198058740 4 client.5295 mds_command: new command op to 5079 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-16T16:14:51.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.621+0000 7f0198058740 1 -- 172.21.15.45:0/2782443671 --> [v2:172.21.15.45:6833/2956262988,v1:172.21.15.45:6835/2956262988] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ac6ff6b70 con 0x564ac7134200 2024-04-16T16:14:51.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.621+0000 7f0185e00640 1 --2- 172.21.15.45:0/2782443671 >> [v2:172.21.15.45:6833/2956262988,v1:172.21.15.45:6835/2956262988] conn(0x564ac7134200 0x564ac7144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.621+0000 7f0185e00640 1 --2- 172.21.15.45:0/2782443671 >> [v2:172.21.15.45:6833/2956262988,v1:172.21.15.45:6835/2956262988] conn(0x564ac7134200 0x564ac7144e80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27379c574b1461ac server_cookie=2cb880fbd756ee04 in_seq=0 out_seq=0 2024-04-16T16:14:51.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.621+0000 7f0173400640 10 client.5295 ms_handle_connect on v2:172.21.15.45:6833/2956262988 2024-04-16T16:14:51.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.622+0000 7f0173400640 1 -- 172.21.15.45:0/2782443671 <== mds.0 v2:172.21.15.45:6833/2956262988 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564ac6ff6b70 con 0x564ac7134200 2024-04-16T16:14:51.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.622+0000 7f0173400640 10 client.5295 handle_command_reply: tid=0 2024-04-16T16:14:51.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.622+0000 7f0198058740 1 client.5295 shutdown 2024-04-16T16:14:51.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.623+0000 7f0198058740 1 -- 172.21.15.45:0/2782443671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 msgr2=0x564ac71043e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:51.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.623+0000 7f0198058740 1 --2- 172.21.15.45:0/2782443671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71043e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f01800154f0 tx=0x7f0180015000 comp rx=0 tx=0).stop 2024-04-16T16:14:51.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.623+0000 7f0198058740 1 -- 172.21.15.45:0/2782443671 shutdown_connections 2024-04-16T16:14:51.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.623+0000 7f0198058740 1 --2- 172.21.15.45:0/2782443671 >> [v2:172.21.15.45:6833/2956262988,v1:172.21.15.45:6835/2956262988] conn(0x564ac7134200 0x564ac7144e80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.623+0000 7f0198058740 1 --2- 172.21.15.45:0/2782443671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac71049f0 0x564ac7106830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.623+0000 7f0198058740 1 --2- 172.21.15.45:0/2782443671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac7112c70 0x564ac71062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.623+0000 7f0198058740 1 --2- 172.21.15.45:0/2782443671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71043e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.623+0000 7f0173400640 0 client.5295 ms_handle_reset on v2:172.21.15.45:6833/2956262988 2024-04-16T16:14:51.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.623+0000 7f0198058740 1 -- 172.21.15.45:0/2782443671 >> 172.21.15.45:0/2782443671 conn(0x564ac70ebab0 msgr2=0x564ac70ea7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:51.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.623+0000 7f0198058740 1 -- 172.21.15.45:0/2782443671 shutdown_connections 2024-04-16T16:14:51.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.623+0000 7f0198058740 1 -- 172.21.15.45:0/2782443671 wait complete. 2024-04-16T16:14:51.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.623+0000 7f0198058740 20 client.5295 trim_cache size 0 max 16384 2024-04-16T16:14:51.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.643+0000 7f0198058740 1 Processor -- start 2024-04-16T16:14:51.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.643+0000 7f0198058740 1 -- start start 2024-04-16T16:14:51.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.644+0000 7f0198058740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac71049f0 0x564ac7106420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.644+0000 7f0198058740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac70e9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.644+0000 7f0198058740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac7112c70 0x564ac70e9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.644+0000 7f0198058740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f019015aa00 con 0x564ac7112890 2024-04-16T16:14:51.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.644+0000 7f0198058740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0190159d60 con 0x564ac71049f0 2024-04-16T16:14:51.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.644+0000 7f0198058740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f019015a3b0 con 0x564ac7112c70 2024-04-16T16:14:51.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.645+0000 7f0185e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac70e9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.645+0000 7f0185e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac70e9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50768/0 (socket says 172.21.15.45:50768) 2024-04-16T16:14:51.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.645+0000 7f0185e00640 1 -- 172.21.15.45:0/2012216919 learned_addr learned my addr 172.21.15.45:0/2012216919 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:51.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.645+0000 7f018f200640 1 --2- 172.21.15.45:0/2012216919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac7112c70 0x564ac70e9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.645+0000 7f0185e00640 1 -- 172.21.15.45:0/2012216919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac7112c70 msgr2=0x564ac70e9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:51.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.645+0000 7f0185e00640 1 --2- 172.21.15.45:0/2012216919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac7112c70 0x564ac70e9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.645+0000 7f0185e00640 1 -- 172.21.15.45:0/2012216919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac71049f0 msgr2=0x564ac7106420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:51.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.645+0000 7f0185e00640 1 --2- 172.21.15.45:0/2012216919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac71049f0 0x564ac7106420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.645+0000 7f0185e00640 1 -- 172.21.15.45:0/2012216919 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ac7107100 con 0x564ac7112890 2024-04-16T16:14:51.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.646+0000 7f0185e00640 1 --2- 172.21.15.45:0/2012216919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac70e9440 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0178058c20 tx=0x7f017808b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6303c0e58acce0a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:51.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.646+0000 7f017e600640 1 -- 172.21.15.45:0/2012216919 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178089830 con 0x564ac7112890 2024-04-16T16:14:51.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.646+0000 7f017e600640 1 -- 172.21.15.45:0/2012216919 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0178057220 con 0x564ac7112890 2024-04-16T16:14:51.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.646+0000 7f017e600640 1 -- 172.21.15.45:0/2012216919 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178055590 con 0x564ac7112890 2024-04-16T16:14:51.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.647+0000 7f0198058740 1 -- 172.21.15.45:0/2012216919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 msgr2=0x564ac70e9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:51.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.647+0000 7f0198058740 1 --2- 172.21.15.45:0/2012216919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac70e9440 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0178058c20 tx=0x7f017808b040 comp rx=0 tx=0).stop 2024-04-16T16:14:51.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.647+0000 7f0198058740 1 -- 172.21.15.45:0/2012216919 shutdown_connections 2024-04-16T16:14:51.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.647+0000 7f0198058740 1 --2- 172.21.15.45:0/2012216919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac7112c70 0x564ac70e9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.647+0000 7f0198058740 1 --2- 172.21.15.45:0/2012216919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac70e9440 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.647+0000 7f0198058740 1 --2- 172.21.15.45:0/2012216919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac71049f0 0x564ac7106420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.647+0000 7f0198058740 1 -- 172.21.15.45:0/2012216919 >> 172.21.15.45:0/2012216919 conn(0x564ac70ebab0 msgr2=0x564ac70e0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:51.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.647+0000 7f0198058740 1 -- 172.21.15.45:0/2012216919 shutdown_connections 2024-04-16T16:14:51.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.647+0000 7f0198058740 1 -- 172.21.15.45:0/2012216919 wait complete. 2024-04-16T16:14:51.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.648+0000 7f0198058740 1 -- start start 2024-04-16T16:14:51.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.648+0000 7f0198058740 1 Processor -- start 2024-04-16T16:14:51.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.649+0000 7f0198058740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac71049f0 0x564ac7105980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.649+0000 7f0198058740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.649+0000 7f0198058740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac7112c70 0x564ac71a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.649+0000 7f0198058740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564ac7134080 con 0x564ac7112890 2024-04-16T16:14:51.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.649+0000 7f0198058740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564ac7109150 con 0x564ac71049f0 2024-04-16T16:14:51.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.649+0000 7f0198058740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564ac71346a0 con 0x564ac7112c70 2024-04-16T16:14:51.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.649+0000 7f0198058740 10 client.0 _ll_register_callbacks cb 0x7f0180023070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:14:51.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.649+0000 7f0185e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.649+0000 7f0185e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71a44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50782/0 (socket says 172.21.15.45:50782) 2024-04-16T16:14:51.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.649+0000 7f0185e00640 1 -- 172.21.15.45:0/883378080 learned_addr learned my addr 172.21.15.45:0/883378080 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:51.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.650+0000 7f018e800640 1 --2- 172.21.15.45:0/883378080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac71049f0 0x564ac7105980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.650+0000 7f0185e00640 1 -- 172.21.15.45:0/883378080 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac7112c70 msgr2=0x564ac71a4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:51.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.650+0000 7f0185e00640 1 --2- 172.21.15.45:0/883378080 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac7112c70 0x564ac71a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.650+0000 7f0185e00640 1 -- 172.21.15.45:0/883378080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac71049f0 msgr2=0x564ac7105980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:51.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.650+0000 7f0185e00640 1 --2- 172.21.15.45:0/883378080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac71049f0 0x564ac7105980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.650+0000 7f0185e00640 1 -- 172.21.15.45:0/883378080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ac71aa9c0 con 0x564ac7112890 2024-04-16T16:14:51.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.650+0000 7f0185e00640 1 --2- 172.21.15.45:0/883378080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71a44d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0178058c20 tx=0x7f017800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6241403b4abc6f15 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:51.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.650+0000 7f0173400640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:14:51.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.650+0000 7f0173400640 1 -- 172.21.15.45:0/883378080 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017807d060 con 0x564ac7112890 2024-04-16T16:14:51.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.650+0000 7f0173400640 1 -- 172.21.15.45:0/883378080 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f0178089830 con 0x564ac7112890 2024-04-16T16:14:51.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.650+0000 7f0198058740 1 -- 172.21.15.45:0/883378080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0178057220 con 0x564ac7112890 2024-04-16T16:14:51.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.650+0000 7f0173400640 1 -- 172.21.15.45:0/883378080 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f0178055590 con 0x564ac7112890 2024-04-16T16:14:51.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.650+0000 7f0198058740 10 client.5301 fetch_fsmap learned FSMap version 16 2024-04-16T16:14:51.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.650+0000 7f0198058740 1 -- 172.21.15.45:0/883378080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x564ac7105ec0 con 0x564ac7112890 2024-04-16T16:14:51.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.651+0000 7f0173400640 1 -- 172.21.15.45:0/883378080 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0178057220 con 0x564ac7112890 2024-04-16T16:14:51.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.651+0000 7f0173400640 1 -- 172.21.15.45:0/883378080 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f0178061670 con 0x564ac7112890 2024-04-16T16:14:51.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.651+0000 7f0198058740 10 client.5301 fetch_fsmap finished waiting for FSMap version 16 2024-04-16T16:14:51.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.651+0000 7f0198058740 10 client.5301 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-16T16:14:51.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.651+0000 7f0198058740 20 client.5301 populate_metadata read hostname 'smithi045' 2024-04-16T16:14:51.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.651+0000 7f0198058740 1 --2- 172.21.15.45:0/883378080 >> [v2:172.21.15.45:6833/2956262988,v1:172.21.15.45:6835/2956262988] conn(0x564ac7153f60 0x564ac7154340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.651+0000 7f0198058740 4 client.5301 mds_command: new command op to 5079 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-16T16:14:51.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.651+0000 7f0198058740 1 -- 172.21.15.45:0/883378080 --> [v2:172.21.15.45:6833/2956262988,v1:172.21.15.45:6835/2956262988] -- command(tid 0: {"prefix": "status"}) -- 0x564ac705e3e0 con 0x564ac7153f60 2024-04-16T16:14:51.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.651+0000 7f018e800640 1 --2- 172.21.15.45:0/883378080 >> [v2:172.21.15.45:6833/2956262988,v1:172.21.15.45:6835/2956262988] conn(0x564ac7153f60 0x564ac7154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f018e800640 1 --2- 172.21.15.45:0/883378080 >> [v2:172.21.15.45:6833/2956262988,v1:172.21.15.45:6835/2956262988] conn(0x564ac7153f60 0x564ac7154340 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=517feffd06cb3054 server_cookie=55b4df5c549644c0 in_seq=0 out_seq=0 2024-04-16T16:14:51.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f0173400640 10 client.5301 ms_handle_connect on v2:172.21.15.45:6833/2956262988 2024-04-16T16:14:51.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f0173400640 1 -- 172.21.15.45:0/883378080 <== mds.0 v2:172.21.15.45:6833/2956262988 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x564ac705e3e0 con 0x564ac7153f60 2024-04-16T16:14:51.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f0173400640 10 client.5301 handle_command_reply: tid=0 2024-04-16T16:14:51.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f0198058740 1 client.5301 shutdown 2024-04-16T16:14:51.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f0198058740 1 -- 172.21.15.45:0/883378080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 msgr2=0x564ac71a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:51.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f0198058740 1 --2- 172.21.15.45:0/883378080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71a44d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0178058c20 tx=0x7f017800f000 comp rx=0 tx=0).stop 2024-04-16T16:14:51.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f0198058740 1 -- 172.21.15.45:0/883378080 shutdown_connections 2024-04-16T16:14:51.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f0198058740 1 --2- 172.21.15.45:0/883378080 >> [v2:172.21.15.45:6833/2956262988,v1:172.21.15.45:6835/2956262988] conn(0x564ac7153f60 0x564ac7154340 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f0198058740 1 --2- 172.21.15.45:0/883378080 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564ac7112c70 0x564ac71a4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f0198058740 1 --2- 172.21.15.45:0/883378080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564ac7112890 0x564ac71a44d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f0198058740 1 --2- 172.21.15.45:0/883378080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ac71049f0 0x564ac7105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f0173400640 0 client.5301 ms_handle_reset on v2:172.21.15.45:6833/2956262988 2024-04-16T16:14:51.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f0198058740 1 -- 172.21.15.45:0/883378080 >> 172.21.15.45:0/883378080 conn(0x564ac70ebab0 msgr2=0x564ac7103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:51.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f0198058740 1 -- 172.21.15.45:0/883378080 shutdown_connections 2024-04-16T16:14:51.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.652+0000 7f0198058740 1 -- 172.21.15.45:0/883378080 wait complete. 2024-04-16T16:14:51.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.653+0000 7f0198058740 20 client.5301 trim_cache size 0 max 16384 2024-04-16T16:14:51.655 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:14:51.656 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "748c4b85-0190-4c59-80ad-098a6a5007c0", 2024-04-16T16:14:51.656 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-04-16T16:14:51.656 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 5079, 2024-04-16T16:14:51.656 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-04-16T16:14:51.656 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-04-16T16:14:51.656 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-04-16T16:14:51.656 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 11.993032942999999, 2024-04-16T16:14:51.656 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 16, 2024-04-16T16:14:51.656 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 34, 2024-04-16T16:14:51.656 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 34, 2024-04-16T16:14:51.656 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 20.894613510999999 2024-04-16T16:14:51.656 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:14:51.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.653+0000 7f0194a00640 1 -- 172.21.15.45:0/895861404 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f016c06c8f0 msgr2=0x7f016c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:51.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.653+0000 7f0194a00640 1 --2- 172.21.15.45:0/895861404 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f016c06c8f0 0x7f016c06edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0180006fd0 tx=0x7f0180008040 comp rx=0 tx=0).stop 2024-04-16T16:14:51.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.653+0000 7f0194a00640 1 -- 172.21.15.45:0/895861404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0190145230 msgr2=0x7f01900cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:51.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.653+0000 7f0194a00640 1 --2- 172.21.15.45:0/895861404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0190145230 0x7f01900cbee0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f017800c7f0 tx=0x7f017800ccb0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.653+0000 7f0194a00640 1 -- 172.21.15.45:0/895861404 shutdown_connections 2024-04-16T16:14:51.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.653+0000 7f0194a00640 1 --2- 172.21.15.45:0/895861404 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f016c06c8f0 0x7f016c06edb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.653+0000 7f0194a00640 1 --2- 172.21.15.45:0/895861404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f019015cbe0 0x7f01900ce180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.653+0000 7f0194a00640 1 --2- 172.21.15.45:0/895861404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0190145230 0x7f01900cbee0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.653+0000 7f0194a00640 1 --2- 172.21.15.45:0/895861404 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01900c5670 0x7f019015b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.653+0000 7f0194a00640 1 -- 172.21.15.45:0/895861404 >> 172.21.15.45:0/895861404 conn(0x7f019006b930 msgr2=0x7f01900b3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:51.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.653+0000 7f0194a00640 1 -- 172.21.15.45:0/895861404 shutdown_connections 2024-04-16T16:14:51.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.654+0000 7f0194a00640 1 -- 172.21.15.45:0/895861404 wait complete. 2024-04-16T16:14:51.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:14:51.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.877+0000 7f5b0d400640 1 Processor -- start 2024-04-16T16:14:51.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.877+0000 7f5b0d400640 1 -- start start 2024-04-16T16:14:51.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.878+0000 7f5b0d400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5b0815ec60 0x7f5b080d2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.878+0000 7f5b0d400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b080d1ac0 0x7f5b080b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.878+0000 7f5b0d400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b080d34c0 con 0x7f5b080d4560 2024-04-16T16:14:51.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.878+0000 7f5b0d400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5b080d37a0 con 0x7f5b080d1ac0 2024-04-16T16:14:51.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.878+0000 7f5b0d400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5b080d3a80 con 0x7f5b0815ec60 2024-04-16T16:14:51.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.879+0000 7f5b07200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b080d1ac0 0x7f5b080b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.879+0000 7f5b07200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b080d1ac0 0x7f5b080b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50796/0 (socket says 172.21.15.45:50796) 2024-04-16T16:14:51.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.879+0000 7f5b07200640 1 -- 172.21.15.45:0/737241503 learned_addr learned my addr 172.21.15.45:0/737241503 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:51.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.879+0000 7f5b07200640 1 -- 172.21.15.45:0/737241503 >> v2:172.21.15.103:3300/0 conn(0x7f5b0815ec60 msgr2=0x7f5b080d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:51.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.879+0000 7f5b06800640 1 --2- 172.21.15.45:0/737241503 >> v2:172.21.15.103:3300/0 conn(0x7f5b0815ec60 0x7f5b080d2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.879+0000 7f5b07200640 1 --2- 172.21.15.45:0/737241503 >> v2:172.21.15.103:3300/0 conn(0x7f5b0815ec60 0x7f5b080d2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.879+0000 7f5b07200640 1 -- 172.21.15.45:0/737241503 >> v1:172.21.15.103:6789/0 conn(0x7f5b080d4560 legacy=0x7f5b080d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:51.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.879+0000 7f5b07200640 1 -- 172.21.15.45:0/737241503 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b080d3d60 con 0x7f5b080d1ac0 2024-04-16T16:14:51.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.880+0000 7f5b07200640 1 --2- 172.21.15.45:0/737241503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b080d1ac0 0x7f5b080b79a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5b080b7530 tx=0x7f5b0002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e498c3d44081e20 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:51.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.881+0000 7f5b05400640 1 -- 172.21.15.45:0/737241503 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00035020 con 0x7f5b080d1ac0 2024-04-16T16:14:51.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.881+0000 7f5b05400640 1 -- 172.21.15.45:0/737241503 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b00031870 con 0x7f5b080d1ac0 2024-04-16T16:14:51.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.881+0000 7f5b05400640 1 -- 172.21.15.45:0/737241503 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00031c40 con 0x7f5b080d1ac0 2024-04-16T16:14:51.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.881+0000 7f5b0d400640 1 -- 172.21.15.45:0/737241503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b080d1ac0 msgr2=0x7f5b080b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:51.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.881+0000 7f5b0d400640 1 --2- 172.21.15.45:0/737241503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b080d1ac0 0x7f5b080b79a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5b080b7530 tx=0x7f5b0002b910 comp rx=0 tx=0).stop 2024-04-16T16:14:51.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.881+0000 7f5b0d400640 1 -- 172.21.15.45:0/737241503 shutdown_connections 2024-04-16T16:14:51.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.881+0000 7f5b0d400640 1 --2- 172.21.15.45:0/737241503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b080d1ac0 0x7f5b080b79a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.881+0000 7f5b0d400640 1 --2- 172.21.15.45:0/737241503 >> v2:172.21.15.103:3300/0 conn(0x7f5b0815ec60 0x7f5b080d2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.881+0000 7f5b0d400640 1 -- 172.21.15.45:0/737241503 >> 172.21.15.45:0/737241503 conn(0x7f5b08073690 msgr2=0x7f5b0806cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:51.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.882+0000 7f5b0d400640 1 -- 172.21.15.45:0/737241503 shutdown_connections 2024-04-16T16:14:51.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.882+0000 7f5b0d400640 1 -- 172.21.15.45:0/737241503 wait complete. 2024-04-16T16:14:51.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.883+0000 7f5b0d400640 1 Processor -- start 2024-04-16T16:14:51.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.883+0000 7f5b0d400640 1 -- start start 2024-04-16T16:14:51.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.884+0000 7f5b0d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b080d1ac0 0x7f5b081f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.884+0000 7f5b0d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b080d4560 0x7f5b081e72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.884+0000 7f5b0d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b0815ec60 0x7f5b081e7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.884+0000 7f5b0d400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5b080d2f40 con 0x7f5b080d1ac0 2024-04-16T16:14:51.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.884+0000 7f5b0d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5b080ba460 con 0x7f5b0815ec60 2024-04-16T16:14:51.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.884+0000 7f5b0d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5b080d30c0 con 0x7f5b080d4560 2024-04-16T16:14:51.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.885+0000 7f5b06800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b080d1ac0 0x7f5b081f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.885+0000 7f5b06800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b080d1ac0 0x7f5b081f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50812/0 (socket says 172.21.15.45:50812) 2024-04-16T16:14:51.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.885+0000 7f5b06800640 1 -- 172.21.15.45:0/1929846736 learned_addr learned my addr 172.21.15.45:0/1929846736 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:51.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.885+0000 7f5b06800640 1 -- 172.21.15.45:0/1929846736 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b080d4560 msgr2=0x7f5b081e72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:51.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.885+0000 7f5b05e00640 1 --2- 172.21.15.45:0/1929846736 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b080d4560 0x7f5b081e72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.885+0000 7f5b06800640 1 --2- 172.21.15.45:0/1929846736 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b080d4560 0x7f5b081e72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.885+0000 7f5b06800640 1 -- 172.21.15.45:0/1929846736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b0815ec60 msgr2=0x7f5b081e7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:51.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.885+0000 7f5b06800640 1 --2- 172.21.15.45:0/1929846736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b0815ec60 0x7f5b081e7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:51.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.885+0000 7f5b06800640 1 -- 172.21.15.45:0/1929846736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b080b6b80 con 0x7f5b080d1ac0 2024-04-16T16:14:51.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.885+0000 7f5b06800640 1 --2- 172.21.15.45:0/1929846736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b080d1ac0 0x7f5b081f3e40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5af800ccd0 tx=0x7f5af800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:51.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.886+0000 7f5afea00640 1 -- 172.21.15.45:0/1929846736 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8003990 con 0x7f5b080d1ac0 2024-04-16T16:14:51.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.886+0000 7f5afea00640 1 -- 172.21.15.45:0/1929846736 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af8003b30 con 0x7f5b080d1ac0 2024-04-16T16:14:51.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.886+0000 7f5afea00640 1 -- 172.21.15.45:0/1929846736 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8004020 con 0x7f5b080d1ac0 2024-04-16T16:14:51.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.886+0000 7f5b0d400640 1 -- 172.21.15.45:0/1929846736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b080b6e60 con 0x7f5b080d1ac0 2024-04-16T16:14:51.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.886+0000 7f5b0d400640 1 -- 172.21.15.45:0/1929846736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b080cce70 con 0x7f5b080d1ac0 2024-04-16T16:14:51.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.887+0000 7f5b0d400640 1 -- 172.21.15.45:0/1929846736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b08075780 con 0x7f5b080d1ac0 2024-04-16T16:14:51.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.888+0000 7f5afea00640 1 -- 172.21.15.45:0/1929846736 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5af80041c0 con 0x7f5b080d1ac0 2024-04-16T16:14:51.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.889+0000 7f5afea00640 1 --2- 172.21.15.45:0/1929846736 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ae806c8f0 0x7f5ae806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:51.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.889+0000 7f5afea00640 1 -- 172.21.15.45:0/1929846736 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f5af808c080 con 0x7f5b080d1ac0 2024-04-16T16:14:51.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.890+0000 7f5b05e00640 1 --2- 172.21.15.45:0/1929846736 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ae806c8f0 0x7f5ae806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:51.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.891+0000 7f5b05e00640 1 --2- 172.21.15.45:0/1929846736 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ae806c8f0 0x7f5ae806edb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5b0814f430 tx=0x7f5af4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:51.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:51.893+0000 7f5afea00640 1 -- 172.21.15.45:0/1929846736 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5af8014030 con 0x7f5b080d1ac0 2024-04-16T16:14:52.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52.144+0000 7f5b0d400640 1 -- 172.21.15.45:0/1929846736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5b080ba960 con 0x7f5b080d1ac0 2024-04-16T16:14:52.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52.145+0000 7f5afea00640 1 -- 172.21.15.45:0/1929846736 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7f5af805a4a0 con 0x7f5b080d1ac0 2024-04-16T16:14:52.145 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:14:52.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52.147+0000 7f5b0d400640 1 -- 172.21.15.45:0/1929846736 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ae806c8f0 msgr2=0x7f5ae806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:52.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52.147+0000 7f5b0d400640 1 --2- 172.21.15.45:0/1929846736 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ae806c8f0 0x7f5ae806edb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5b0814f430 tx=0x7f5af4009040 comp rx=0 tx=0).stop 2024-04-16T16:14:52.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52.147+0000 7f5b0d400640 1 -- 172.21.15.45:0/1929846736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b080d1ac0 msgr2=0x7f5b081f3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:52.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52.147+0000 7f5b0d400640 1 --2- 172.21.15.45:0/1929846736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b080d1ac0 0x7f5b081f3e40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5af800ccd0 tx=0x7f5af800a5c0 comp rx=0 tx=0).stop 2024-04-16T16:14:52.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52.148+0000 7f5b0d400640 1 -- 172.21.15.45:0/1929846736 shutdown_connections 2024-04-16T16:14:52.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52.148+0000 7f5b0d400640 1 --2- 172.21.15.45:0/1929846736 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f5ae806c8f0 0x7f5ae806edb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:52.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52.148+0000 7f5b0d400640 1 --2- 172.21.15.45:0/1929846736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b0815ec60 0x7f5b081e7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:52.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52.148+0000 7f5b0d400640 1 --2- 172.21.15.45:0/1929846736 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b080d4560 0x7f5b081e72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:52.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52.148+0000 7f5b0d400640 1 --2- 172.21.15.45:0/1929846736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b080d1ac0 0x7f5b081f3e40 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:52.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52.148+0000 7f5b0d400640 1 -- 172.21.15.45:0/1929846736 >> 172.21.15.45:0/1929846736 conn(0x7f5b08073690 msgr2=0x7f5b0815c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:52.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52.148+0000 7f5b0d400640 1 -- 172.21.15.45:0/1929846736 shutdown_connections 2024-04-16T16:14:52.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52.148+0000 7f5b0d400640 1 -- 172.21.15.45:0/1929846736 wait complete. 2024-04-16T16:14:52.166 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-16T16:14:52.166 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:14:52.167 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-04-16T16:14:52.167 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-16T16:14:52.167 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T16:14:52.167 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-16T16:14:52.167 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:14:52.167 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:14:52.167 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:14:52.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:52.167 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-04-16T16:14:52.222 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:14:52.222 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:14:52.222 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:14:52.222 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:14:52.222 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-04-16T16:14:52.222 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:14:52.222 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:14:52.222 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:14:52.223 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-04-16T16:14:52.223 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:14:52.223 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:14:52.223 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-04-16T16:14:52.223 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:14:52.223 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-04-16T16:14:52.223 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-04-16T16:14:52.223 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-16T16:14:52.223 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42509sec preferred_lft 42509sec 2024-04-16T16:14:52.223 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-04-16T16:14:52.223 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:14:52.223 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:14:52.224 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-04-16T16:14:52.224 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-04-16T16:14:52.224 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:14:52.224 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff 2024-04-16T16:14:52.224 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:14:52.224 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:14:52.224 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::50f1:99ff:fe2b:225b/64 scope link 2024-04-16T16:14:52.224 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:14:52.224 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:14:52.224 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T16:14:52.224 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::7007:c3ff:fe13:7bcd/64 scope link 2024-04-16T16:14:52.224 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:14:52.224 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:14:52.225 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether ae:10:ea:fc:0a:13 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-16T16:14:52.225 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ac10:eaff:fefc:a13/64 scope link 2024-04-16T16:14:52.225 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:14:52.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:52.226 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:14:52.318 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-04-16T16:14:52.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:52.323 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-04-16T16:14:52.379 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-04-16T16:14:52.379 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:14:52.379 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-16T16:14:52.379 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-16T16:14:52.379 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:14:52.379 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:14:52.379 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:14:52.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-16T16:14:52.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-16T16:14:52.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-16T16:14:52.380 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:14:52.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:14:52.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:14:52.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:52.642 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T16:14:52.700 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-16T16:14:52.701 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-16T16:14:52.701 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:14:52.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:52.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-04-16T16:14:52.803 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:14:52.803 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-16T16:14:52.858 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:14:52.858 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:14:52.858 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:14:52.913 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-16T16:14:52.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:52.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:14:53.052 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:14:53.052 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:14:53.089 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:14:53.089 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:14:53.089 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:14:53.090 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 0@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:14:53.090 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:14:53.093 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-16T16:14:53.093 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:14:53.093 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:14:53.093 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:14:53.093 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:14:53.097 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:14:53.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:53.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:14:53.185 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:14:53.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:14:53.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:14:53.326 INFO:teuthology.orchestra.run.smithi045.stdout:client4846 2024-04-16T16:14:53.326 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:14:53.326 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:53.334 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:14:53.378 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:14:53.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:14:53.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.638+0000 7f05e7e00640 1 Processor -- start 2024-04-16T16:14:53.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.638+0000 7f05e7e00640 1 -- start start 2024-04-16T16:14:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.640+0000 7f05e7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05e815e8f0 0x7f05e80cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.640+0000 7f05e7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f05e80ce9a0 0x7f05e80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.640+0000 7f05e7e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05e815ecd0 con 0x7f05e80cf080 2024-04-16T16:14:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.640+0000 7f05e7e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f05e80ce2e0 con 0x7f05e80ce9a0 2024-04-16T16:14:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.640+0000 7f05e7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f05e80cd6d0 con 0x7f05e815e8f0 2024-04-16T16:14:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.641+0000 7f05e7400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f05e80cf080 0x7f05e8162480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57112/0 (socket says 172.21.15.45:57112) 2024-04-16T16:14:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.641+0000 7f05e7400640 1 -- 172.21.15.45:0/44114834 learned_addr learned my addr 172.21.15.45:0/44114834 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.641+0000 7f05e6a00640 1 --2- 172.21.15.45:0/44114834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05e815e8f0 0x7f05e80cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.641+0000 7f05e6000640 1 --2- 172.21.15.45:0/44114834 >> v2:172.21.15.103:3300/0 conn(0x7f05e80ce9a0 0x7f05e80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.641+0000 7f05e5600640 1 -- 172.21.15.45:0/44114834 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 228508413 0 0) 0x7f05e815ecd0 con 0x7f05e80cf080 2024-04-16T16:14:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.641+0000 7f05e5600640 1 -- 172.21.15.45:0/44114834 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05cc003520 con 0x7f05e80cf080 2024-04-16T16:14:53.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e5600640 1 -- 172.21.15.45:0/44114834 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3567883262 0 0) 0x7f05cc003520 con 0x7f05e80cf080 2024-04-16T16:14:53.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e5600640 1 -- 172.21.15.45:0/44114834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05e815e8f0 msgr2=0x7f05e80cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:53.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e5600640 1 --2- 172.21.15.45:0/44114834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05e815e8f0 0x7f05e80cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:53.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e5600640 1 -- 172.21.15.45:0/44114834 >> v2:172.21.15.103:3300/0 conn(0x7f05e80ce9a0 msgr2=0x7f05e80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:53.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e5600640 1 --2- 172.21.15.45:0/44114834 >> v2:172.21.15.103:3300/0 conn(0x7f05e80ce9a0 0x7f05e80cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:53.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e5600640 1 -- 172.21.15.45:0/44114834 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e80cd9b0 con 0x7f05e80cf080 2024-04-16T16:14:53.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e5600640 1 -- 172.21.15.45:0/44114834 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f05e0001670 con 0x7f05e80cf080 2024-04-16T16:14:53.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e5600640 1 -- 172.21.15.45:0/44114834 >> v1:172.21.15.45:6789/0 conn(0x7f05e80cf080 legacy=0x7f05e8162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e5600640 1 --2- 172.21.15.45:0/44114834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05cc0039b0 0x7f05cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e5600640 1 --2- 172.21.15.45:0/44114834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05e815e8f0 0x7f05cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e5600640 1 --2- 172.21.15.45:0/44114834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05cc004d40 0x7f05cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e5600640 1 -- 172.21.15.45:0/44114834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f05e80ce2e0 con 0x7f05cc004d40 2024-04-16T16:14:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e5600640 1 -- 172.21.15.45:0/44114834 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f05e80cd6d0 con 0x7f05cc0039b0 2024-04-16T16:14:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e5600640 1 -- 172.21.15.45:0/44114834 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f05cc00b9c0 con 0x7f05e815e8f0 2024-04-16T16:14:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e7400640 1 --2- 172.21.15.45:0/44114834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05cc004d40 0x7f05cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e6a00640 1 --2- 172.21.15.45:0/44114834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05cc0039b0 0x7f05cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.642+0000 7f05e6000640 1 --2- 172.21.15.45:0/44114834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05e815e8f0 0x7f05cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.643+0000 7f05e7400640 1 -- 172.21.15.45:0/44114834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05e815e8f0 msgr2=0x7f05cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.643+0000 7f05e7400640 1 --2- 172.21.15.45:0/44114834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05e815e8f0 0x7f05cc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.643+0000 7f05e7400640 1 -- 172.21.15.45:0/44114834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05cc0039b0 msgr2=0x7f05cc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.643+0000 7f05e7400640 1 --2- 172.21.15.45:0/44114834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05cc0039b0 0x7f05cc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.643+0000 7f05e7400640 1 -- 172.21.15.45:0/44114834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05cc003520 con 0x7f05cc004d40 2024-04-16T16:14:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.644+0000 7f05e7400640 1 --2- 172.21.15.45:0/44114834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05cc004d40 0x7f05cc0051c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f05e000c190 tx=0x7f05cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c156b8fb44636d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.645+0000 7f05e5600640 1 -- 172.21.15.45:0/44114834 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e0007c80 con 0x7f05cc004d40 2024-04-16T16:14:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.645+0000 7f05e5600640 1 -- 172.21.15.45:0/44114834 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05e0030070 con 0x7f05cc004d40 2024-04-16T16:14:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.645+0000 7f05e5600640 1 -- 172.21.15.45:0/44114834 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e0005d30 con 0x7f05cc004d40 2024-04-16T16:14:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.646+0000 7f05e7e00640 1 -- 172.21.15.45:0/44114834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05cc004d40 msgr2=0x7f05cc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.646+0000 7f05e7e00640 1 --2- 172.21.15.45:0/44114834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05cc004d40 0x7f05cc0051c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f05e000c190 tx=0x7f05cc005830 comp rx=0 tx=0).stop 2024-04-16T16:14:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.646+0000 7f05e7400640 1 -- 172.21.15.45:0/44114834 reap_dead start 2024-04-16T16:14:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.646+0000 7f05e7e00640 1 -- 172.21.15.45:0/44114834 shutdown_connections 2024-04-16T16:14:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.646+0000 7f05e7e00640 1 -- 172.21.15.45:0/44114834 >> 172.21.15.45:0/44114834 conn(0x7f05e806b930 msgr2=0x7f05e806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.646+0000 7f05e7e00640 1 -- 172.21.15.45:0/44114834 shutdown_connections 2024-04-16T16:14:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.646+0000 7f05e7e00640 1 -- 172.21.15.45:0/44114834 wait complete. 2024-04-16T16:14:53.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.648+0000 7f05e7e00640 1 Processor -- start 2024-04-16T16:14:53.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.649+0000 7f05e7e00640 1 -- start start 2024-04-16T16:14:53.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.650+0000 7f05e7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05e80b43d0 0x7f05e8152270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:53.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.650+0000 7f05e7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05e8150170 0x7f05e8144d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:53.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.650+0000 7f05e7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05e80b47b0 0x7f05e8145250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.650+0000 7f05e7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f05e80cd360 con 0x7f05e80b47b0 2024-04-16T16:14:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.650+0000 7f05e7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f05e80cdf70 con 0x7f05e80b43d0 2024-04-16T16:14:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.650+0000 7f05e7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f05e80cc410 con 0x7f05e8150170 2024-04-16T16:14:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.650+0000 7f05e7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05e80b47b0 0x7f05e8145250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.650+0000 7f05e7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05e80b47b0 0x7f05e8145250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41748/0 (socket says 172.21.15.45:41748) 2024-04-16T16:14:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.650+0000 7f05e7400640 1 -- 172.21.15.45:0/1818789299 learned_addr learned my addr 172.21.15.45:0/1818789299 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.651+0000 7f05e6000640 1 --2- 172.21.15.45:0/1818789299 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05e8150170 0x7f05e8144d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.651+0000 7f05e6a00640 1 --2- 172.21.15.45:0/1818789299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05e80b43d0 0x7f05e8152270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.651+0000 7f05e7400640 1 -- 172.21.15.45:0/1818789299 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05e8150170 msgr2=0x7f05e8144d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.651+0000 7f05e7400640 1 --2- 172.21.15.45:0/1818789299 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05e8150170 0x7f05e8144d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.651+0000 7f05e7400640 1 -- 172.21.15.45:0/1818789299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05e80b43d0 msgr2=0x7f05e8152270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.651+0000 7f05e7400640 1 --2- 172.21.15.45:0/1818789299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05e80b43d0 0x7f05e8152270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.651+0000 7f05e7400640 1 -- 172.21.15.45:0/1818789299 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e81506c0 con 0x7f05e80b47b0 2024-04-16T16:14:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.651+0000 7f05e6a00640 1 --2- 172.21.15.45:0/1818789299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05e80b43d0 0x7f05e8152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:14:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.651+0000 7f05e7400640 1 --2- 172.21.15.45:0/1818789299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05e80b47b0 0x7f05e8145250 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f05e000c190 tx=0x7f05e0001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.652+0000 7f05df400640 1 -- 172.21.15.45:0/1818789299 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e0001bc0 con 0x7f05e80b47b0 2024-04-16T16:14:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.652+0000 7f05df400640 1 -- 172.21.15.45:0/1818789299 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05e002ca50 con 0x7f05e80b47b0 2024-04-16T16:14:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.652+0000 7f05df400640 1 -- 172.21.15.45:0/1818789299 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e002ce00 con 0x7f05e80b47b0 2024-04-16T16:14:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.652+0000 7f05e7e00640 1 -- 172.21.15.45:0/1818789299 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05e81509a0 con 0x7f05e80b47b0 2024-04-16T16:14:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.652+0000 7f05e7e00640 1 -- 172.21.15.45:0/1818789299 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05e8150f60 con 0x7f05e80b47b0 2024-04-16T16:14:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.653+0000 7f05e7e00640 1 -- 172.21.15.45:0/1818789299 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05e8075780 con 0x7f05e80b47b0 2024-04-16T16:14:53.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.654+0000 7f05df400640 1 -- 172.21.15.45:0/1818789299 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f05e0007070 con 0x7f05e80b47b0 2024-04-16T16:14:53.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.655+0000 7f05df400640 1 --2- 172.21.15.45:0/1818789299 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f05c406c8f0 0x7f05c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:53.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.655+0000 7f05df400640 1 -- 172.21.15.45:0/1818789299 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f05e00ac740 con 0x7f05e80b47b0 2024-04-16T16:14:53.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.656+0000 7f05e6a00640 1 --2- 172.21.15.45:0/1818789299 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f05c406c8f0 0x7f05c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:53.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.657+0000 7f05e6a00640 1 --2- 172.21.15.45:0/1818789299 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f05c406c8f0 0x7f05c406edb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009b40 tx=0x7f05d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:53.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.660+0000 7f05df400640 1 -- 172.21.15.45:0/1818789299 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f05e007a480 con 0x7f05e80b47b0 2024-04-16T16:14:53.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.909+0000 7f05e7e00640 1 -- 172.21.15.45:0/1818789299 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f05e80b5b90 con 0x7f05e80b47b0 2024-04-16T16:14:53.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.909+0000 7f05df400640 1 -- 172.21.15.45:0/1818789299 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7f05e007e780 con 0x7f05e80b47b0 2024-04-16T16:14:53.909 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:14:53.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.912+0000 7f05e7e00640 1 -- 172.21.15.45:0/1818789299 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f05c406c8f0 msgr2=0x7f05c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:53.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.912+0000 7f05e7e00640 1 --2- 172.21.15.45:0/1818789299 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f05c406c8f0 0x7f05c406edb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009b40 tx=0x7f05d0008040 comp rx=0 tx=0).stop 2024-04-16T16:14:53.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.912+0000 7f05e7e00640 1 -- 172.21.15.45:0/1818789299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05e80b47b0 msgr2=0x7f05e8145250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:53.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.912+0000 7f05e7e00640 1 --2- 172.21.15.45:0/1818789299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05e80b47b0 0x7f05e8145250 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f05e000c190 tx=0x7f05e0001970 comp rx=0 tx=0).stop 2024-04-16T16:14:53.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.912+0000 7f05e7e00640 1 -- 172.21.15.45:0/1818789299 shutdown_connections 2024-04-16T16:14:53.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.912+0000 7f05e7e00640 1 --2- 172.21.15.45:0/1818789299 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f05c406c8f0 0x7f05c406edb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:53.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.912+0000 7f05e7e00640 1 --2- 172.21.15.45:0/1818789299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05e80b47b0 0x7f05e8145250 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:53.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.912+0000 7f05e7e00640 1 --2- 172.21.15.45:0/1818789299 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05e8150170 0x7f05e8144d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:53.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.912+0000 7f05e7e00640 1 --2- 172.21.15.45:0/1818789299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05e80b43d0 0x7f05e8152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:53.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.912+0000 7f05e7e00640 1 -- 172.21.15.45:0/1818789299 >> 172.21.15.45:0/1818789299 conn(0x7f05e806b930 msgr2=0x7f05e806f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:53.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.912+0000 7f05e7e00640 1 -- 172.21.15.45:0/1818789299 shutdown_connections 2024-04-16T16:14:53.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:53.912+0000 7f05e7e00640 1 -- 172.21.15.45:0/1818789299 wait complete. 2024-04-16T16:14:53.931 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-16T16:14:53.931 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:14:53.931 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-04-16T16:14:53.931 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-16T16:14:53.932 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T16:14:53.932 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-16T16:14:53.932 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:14:53.932 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:14:53.932 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:14:53.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:53.932 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-04-16T16:14:53.985 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:14:53.986 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:14:53.986 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:14:53.986 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:14:53.986 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-16T16:14:53.986 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:14:53.986 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:14:53.986 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:14:53.986 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-04-16T16:14:53.986 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:14:53.986 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:14:53.986 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-04-16T16:14:53.986 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:14:53.987 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-04-16T16:14:53.987 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:14:53.987 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-04-16T16:14:53.987 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-16T16:14:53.987 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42531sec preferred_lft 42531sec 2024-04-16T16:14:53.987 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-04-16T16:14:53.987 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:14:53.987 INFO:teuthology.orchestra.run.smithi103.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:14:53.987 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:14:53.987 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:14:53.987 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:14:53.987 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::74ce:ceff:fec6:2a00/64 scope link 2024-04-16T16:14:53.987 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:14:53.987 INFO:teuthology.orchestra.run.smithi103.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:14:53.987 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T16:14:53.988 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::3cf7:ffff:feca:f879/64 scope link 2024-04-16T16:14:53.988 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:14:53.988 INFO:teuthology.orchestra.run.smithi103.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:14:53.988 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 86:4c:6e:ce:c3:07 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-16T16:14:53.988 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::844c:6eff:fece:c307/64 scope link 2024-04-16T16:14:53.988 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:14:53.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:53.989 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:14:54.073 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-16T16:14:54.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:54.077 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-04-16T16:14:54.119 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-04-16T16:14:54.120 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:14:54.120 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-16T16:14:54.120 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-16T16:14:54.120 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:14:54.120 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:14:54.120 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:14:54.120 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-16T16:14:54.120 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-16T16:14:54.120 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-16T16:14:54.120 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:14:54.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:14:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:14:54.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:14:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:14:54.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:54.380 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-16T16:14:54.419 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-16T16:14:54.419 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-16T16:14:54.420 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:14:54.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:54.420 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.0 up 2024-04-16T16:14:54.550 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:14:54.551 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-16T16:14:54.599 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:14:54.599 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:14:54.600 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:14:54.691 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-16T16:14:54.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:54.692 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:14:54.810 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:14:54.810 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:14:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:14:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:14:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:14:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with new device syntax: 1@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:14:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:14:54.845 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-16T16:14:54.846 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:14:54.846 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:14:54.846 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:14:54.846 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:14:54.849 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:14:54.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:54.849 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:14:54.925 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:14:55.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:14:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:14:55.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:14:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:14:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:client4855 2024-04-16T16:14:55.037 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:14:55.037 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:55.085 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:14:55.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.319+0000 7fbb94e00640 1 Processor -- start 2024-04-16T16:14:55.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.319+0000 7fbb94e00640 1 -- start start 2024-04-16T16:14:55.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.321+0000 7fbb94e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb90146220 0x7fbb9015ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:55.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.321+0000 7fbb94e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbb9006f7a0 0x7fbb9006df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:55.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.321+0000 7fbb94e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb9015d360 con 0x7fbb9006e480 2024-04-16T16:14:55.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.321+0000 7fbb94e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbb90160410 con 0x7fbb9006f7a0 2024-04-16T16:14:55.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.321+0000 7fbb94e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbb9015d640 con 0x7fbb90146220 2024-04-16T16:14:55.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.322+0000 7fbb8ea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbb9006f7a0 0x7fbb9006df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:55.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.322+0000 7fbb8fe00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fbb9006e480 0x7fbb901608a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:42630/0 (socket says 172.21.15.45:42630) 2024-04-16T16:14:55.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.322+0000 7fbb8fe00640 1 -- 172.21.15.45:0/3773624688 learned_addr learned my addr 172.21.15.45:0/3773624688 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:55.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.322+0000 7fbb8f400640 1 --2- 172.21.15.45:0/3773624688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb90146220 0x7fbb9015ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:55.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.322+0000 7fbb8ea00640 1 -- 172.21.15.45:0/3773624688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb90146220 msgr2=0x7fbb9015ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:55.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.322+0000 7fbb8ea00640 1 --2- 172.21.15.45:0/3773624688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb90146220 0x7fbb9015ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.322+0000 7fbb8e000640 1 -- 172.21.15.45:0/3773624688 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3423384469 0 0) 0x7fbb9015d360 con 0x7fbb9006e480 2024-04-16T16:14:55.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.322+0000 7fbb8ea00640 1 -- 172.21.15.45:0/3773624688 >> v1:172.21.15.103:6789/0 conn(0x7fbb9006e480 legacy=0x7fbb901608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:55.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.322+0000 7fbb8ea00640 1 -- 172.21.15.45:0/3773624688 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9015d920 con 0x7fbb9006f7a0 2024-04-16T16:14:55.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.324+0000 7fbb8ea00640 1 --2- 172.21.15.45:0/3773624688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb9006f7a0 0x7fbb9006df40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c0097a0 tx=0x7fbb7c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12e476685ea59fe server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:55.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.324+0000 7fbb8e000640 1 -- 172.21.15.45:0/3773624688 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c035020 con 0x7fbb9006f7a0 2024-04-16T16:14:55.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.324+0000 7fbb8e000640 1 -- 172.21.15.45:0/3773624688 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb7c031810 con 0x7fbb9006f7a0 2024-04-16T16:14:55.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.324+0000 7fbb8e000640 1 -- 172.21.15.45:0/3773624688 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c031be0 con 0x7fbb9006f7a0 2024-04-16T16:14:55.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.325+0000 7fbb94e00640 1 -- 172.21.15.45:0/3773624688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb9006f7a0 msgr2=0x7fbb9006df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:55.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.325+0000 7fbb94e00640 1 --2- 172.21.15.45:0/3773624688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb9006f7a0 0x7fbb9006df40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c0097a0 tx=0x7fbb7c02b8b0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.325+0000 7fbb94e00640 1 -- 172.21.15.45:0/3773624688 shutdown_connections 2024-04-16T16:14:55.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.325+0000 7fbb94e00640 1 --2- 172.21.15.45:0/3773624688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb9006f7a0 0x7fbb9006df40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.325+0000 7fbb94e00640 1 --2- 172.21.15.45:0/3773624688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb90146220 0x7fbb9015ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.325+0000 7fbb94e00640 1 -- 172.21.15.45:0/3773624688 >> 172.21.15.45:0/3773624688 conn(0x7fbb9006b930 msgr2=0x7fbb9006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:55.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.325+0000 7fbb94e00640 1 -- 172.21.15.45:0/3773624688 shutdown_connections 2024-04-16T16:14:55.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.325+0000 7fbb94e00640 1 -- 172.21.15.45:0/3773624688 wait complete. 2024-04-16T16:14:55.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.327+0000 7fbb94e00640 1 Processor -- start 2024-04-16T16:14:55.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.327+0000 7fbb94e00640 1 -- start start 2024-04-16T16:14:55.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.328+0000 7fbb94e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9006e480 0x7fbb900b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:55.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.328+0000 7fbb94e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb9006f7a0 0x7fbb9015c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:55.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.328+0000 7fbb94e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb90146220 0x7fbb900cf170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:55.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.328+0000 7fbb94e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9006fda0 con 0x7fbb9006e480 2024-04-16T16:14:55.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.328+0000 7fbb94e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9006fc20 con 0x7fbb9006f7a0 2024-04-16T16:14:55.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.328+0000 7fbb94e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbb90160fb0 con 0x7fbb90146220 2024-04-16T16:14:55.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.329+0000 7fbb8f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9006e480 0x7fbb900b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:55.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.329+0000 7fbb8f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9006e480 0x7fbb900b2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41752/0 (socket says 172.21.15.45:41752) 2024-04-16T16:14:55.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.329+0000 7fbb8f400640 1 -- 172.21.15.45:0/2045597950 learned_addr learned my addr 172.21.15.45:0/2045597950 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:55.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.329+0000 7fbb8ea00640 1 --2- 172.21.15.45:0/2045597950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb9006f7a0 0x7fbb9015c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:55.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.329+0000 7fbb8fe00640 1 --2- 172.21.15.45:0/2045597950 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb90146220 0x7fbb900cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:55.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.329+0000 7fbb8f400640 1 -- 172.21.15.45:0/2045597950 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb90146220 msgr2=0x7fbb900cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:55.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.329+0000 7fbb8f400640 1 --2- 172.21.15.45:0/2045597950 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb90146220 0x7fbb900cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.329+0000 7fbb8f400640 1 -- 172.21.15.45:0/2045597950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb9006f7a0 msgr2=0x7fbb9015c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:55.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.329+0000 7fbb8f400640 1 --2- 172.21.15.45:0/2045597950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb9006f7a0 0x7fbb9015c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.329+0000 7fbb8f400640 1 -- 172.21.15.45:0/2045597950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb900cf6b0 con 0x7fbb9006e480 2024-04-16T16:14:55.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.329+0000 7fbb8ea00640 1 --2- 172.21.15.45:0/2045597950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb9006f7a0 0x7fbb9015c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:14:55.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.329+0000 7fbb8f400640 1 --2- 172.21.15.45:0/2045597950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9006e480 0x7fbb900b2180 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000d620 tx=0x7fbb8000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:55.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.330+0000 7fbb87400640 1 -- 172.21.15.45:0/2045597950 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8000b620 con 0x7fbb9006e480 2024-04-16T16:14:55.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.330+0000 7fbb87400640 1 -- 172.21.15.45:0/2045597950 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb80019070 con 0x7fbb9006e480 2024-04-16T16:14:55.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.330+0000 7fbb87400640 1 -- 172.21.15.45:0/2045597950 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8000ba00 con 0x7fbb9006e480 2024-04-16T16:14:55.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.330+0000 7fbb94e00640 1 -- 172.21.15.45:0/2045597950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb901420f0 con 0x7fbb9006e480 2024-04-16T16:14:55.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.330+0000 7fbb94e00640 1 -- 172.21.15.45:0/2045597950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb90142390 con 0x7fbb9006e480 2024-04-16T16:14:55.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.331+0000 7fbb94e00640 1 -- 172.21.15.45:0/2045597950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb90075780 con 0x7fbb9006e480 2024-04-16T16:14:55.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.332+0000 7fbb87400640 1 -- 172.21.15.45:0/2045597950 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbb8000bba0 con 0x7fbb9006e480 2024-04-16T16:14:55.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.333+0000 7fbb87400640 1 --2- 172.21.15.45:0/2045597950 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbb7006c8f0 0x7fbb7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:55.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.334+0000 7fbb87400640 1 -- 172.21.15.45:0/2045597950 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fbb8008bf60 con 0x7fbb9006e480 2024-04-16T16:14:55.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.334+0000 7fbb8ea00640 1 --2- 172.21.15.45:0/2045597950 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbb7006c8f0 0x7fbb7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:55.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.336+0000 7fbb8ea00640 1 --2- 172.21.15.45:0/2045597950 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbb7006c8f0 0x7fbb7006edb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c036000 tx=0x7fbb7c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:55.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.337+0000 7fbb87400640 1 -- 172.21.15.45:0/2045597950 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbb900cf6b0 con 0x7fbb9006e480 2024-04-16T16:14:55.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.587+0000 7fbb94e00640 1 -- 172.21.15.45:0/2045597950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb900cff60 con 0x7fbb9006e480 2024-04-16T16:14:55.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.588+0000 7fbb87400640 1 -- 172.21.15.45:0/2045597950 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7fbb8005a2f0 con 0x7fbb9006e480 2024-04-16T16:14:55.588 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:55.588 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/814837411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":814837411},{"type":"v1","addr":"172.21.15.103:6836","nonce":814837411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/401733314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":401733314},{"type":"v1","addr":"172.21.15.103:6837","nonce":401733314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5073,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/768816269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":768816269},{"type":"v1","addr":"172.21.15.45:6834","nonce":768816269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:14:39.132000+0000","modified":"2024-04-16T16:14:42.226029+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5079},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5079":{"gid":5079,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6835/2956262988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2956262988},{"type":"v1","addr":"172.21.15.45:6835","nonce":2956262988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5079,"qdb_cluster":[5079]},"id":2}]} 2024-04-16T16:14:55.588 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 16 2024-04-16T16:14:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.590+0000 7fbb94e00640 1 -- 172.21.15.45:0/2045597950 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbb7006c8f0 msgr2=0x7fbb7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.590+0000 7fbb94e00640 1 --2- 172.21.15.45:0/2045597950 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbb7006c8f0 0x7fbb7006edb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c036000 tx=0x7fbb7c008030 comp rx=0 tx=0).stop 2024-04-16T16:14:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.591+0000 7fbb94e00640 1 -- 172.21.15.45:0/2045597950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9006e480 msgr2=0x7fbb900b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.591+0000 7fbb94e00640 1 --2- 172.21.15.45:0/2045597950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9006e480 0x7fbb900b2180 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000d620 tx=0x7fbb8000dae0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.591+0000 7fbb94e00640 1 -- 172.21.15.45:0/2045597950 shutdown_connections 2024-04-16T16:14:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.591+0000 7fbb94e00640 1 --2- 172.21.15.45:0/2045597950 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7fbb7006c8f0 0x7fbb7006edb0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.591+0000 7fbb94e00640 1 --2- 172.21.15.45:0/2045597950 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb90146220 0x7fbb900cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.591+0000 7fbb94e00640 1 --2- 172.21.15.45:0/2045597950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb9006f7a0 0x7fbb9015c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.591+0000 7fbb94e00640 1 --2- 172.21.15.45:0/2045597950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9006e480 0x7fbb900b2180 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.591+0000 7fbb94e00640 1 -- 172.21.15.45:0/2045597950 >> 172.21.15.45:0/2045597950 conn(0x7fbb9006b930 msgr2=0x7fbb900c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:55.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.591+0000 7fbb94e00640 1 -- 172.21.15.45:0/2045597950 shutdown_connections 2024-04-16T16:14:55.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.591+0000 7fbb94e00640 1 -- 172.21.15.45:0/2045597950 wait complete. 2024-04-16T16:14:55.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable stats 2024-04-16T16:14:55.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.833+0000 7ff32cc00640 1 Processor -- start 2024-04-16T16:14:55.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.833+0000 7ff32cc00640 1 -- start start 2024-04-16T16:14:55.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.834+0000 7ff32cc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff3280afda0 0x7ff3280b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:55.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.834+0000 7ff32cc00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3280ccb70 con 0x7ff3280d0e80 2024-04-16T16:14:55.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.834+0000 7ff32cc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3280cce50 con 0x7ff32815b920 2024-04-16T16:14:55.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.834+0000 7ff32cc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff32806e7a0 con 0x7ff3280afda0 2024-04-16T16:14:55.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.835+0000 7ff325400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff3280d0e80 0x7ff3280ce5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57124/0 (socket says 172.21.15.45:57124) 2024-04-16T16:14:55.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.835+0000 7ff325400640 1 -- 172.21.15.45:0/2214443854 learned_addr learned my addr 172.21.15.45:0/2214443854 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:55.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.835+0000 7ff326800640 1 --2- 172.21.15.45:0/2214443854 >> v2:172.21.15.103:3300/0 conn(0x7ff3280afda0 0x7ff3280b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:55.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff324a00640 1 -- 172.21.15.45:0/2214443854 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 770633918 0 0) 0x7ff3280ccb70 con 0x7ff3280d0e80 2024-04-16T16:14:55.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff324a00640 1 -- 172.21.15.45:0/2214443854 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff30c003520 con 0x7ff3280d0e80 2024-04-16T16:14:55.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff324a00640 1 -- 172.21.15.45:0/2214443854 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2989929939 0 0) 0x7ff30c003520 con 0x7ff3280d0e80 2024-04-16T16:14:55.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff324a00640 1 -- 172.21.15.45:0/2214443854 >> v2:172.21.15.103:3300/0 conn(0x7ff3280afda0 msgr2=0x7ff3280b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:55.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff324a00640 1 --2- 172.21.15.45:0/2214443854 >> v2:172.21.15.103:3300/0 conn(0x7ff3280afda0 0x7ff3280b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff324a00640 1 -- 172.21.15.45:0/2214443854 >> v1:172.21.15.103:6789/0 conn(0x7ff32815b920 legacy=0x7ff3280d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:55.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff324a00640 1 -- 172.21.15.45:0/2214443854 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32806e430 con 0x7ff3280d0e80 2024-04-16T16:14:55.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff324a00640 1 -- 172.21.15.45:0/2214443854 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7ff310001670 con 0x7ff3280d0e80 2024-04-16T16:14:55.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff324a00640 1 -- 172.21.15.45:0/2214443854 >> v1:172.21.15.45:6789/0 conn(0x7ff3280d0e80 legacy=0x7ff3280ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:55.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff324a00640 1 --2- 172.21.15.45:0/2214443854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff30c003a00 0x7ff30c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:55.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff324a00640 1 --2- 172.21.15.45:0/2214443854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff30c004800 0x7ff30c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:55.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff324a00640 1 --2- 172.21.15.45:0/2214443854 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff30c0051c0 0x7ff30c00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:55.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff324a00640 1 -- 172.21.15.45:0/2214443854 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff32806e7a0 con 0x7ff30c004800 2024-04-16T16:14:55.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff324a00640 1 -- 172.21.15.45:0/2214443854 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff30c00e300 con 0x7ff30c003a00 2024-04-16T16:14:55.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff324a00640 1 -- 172.21.15.45:0/2214443854 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff30c00e500 con 0x7ff30c0051c0 2024-04-16T16:14:55.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.836+0000 7ff325400640 1 --2- 172.21.15.45:0/2214443854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff30c004800 0x7ff30c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:55.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.837+0000 7ff325e00640 1 --2- 172.21.15.45:0/2214443854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff30c003a00 0x7ff30c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:55.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.837+0000 7ff326800640 1 --2- 172.21.15.45:0/2214443854 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff30c0051c0 0x7ff30c00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:55.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.837+0000 7ff325400640 1 -- 172.21.15.45:0/2214443854 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff30c0051c0 msgr2=0x7ff30c00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:55.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.837+0000 7ff325400640 1 --2- 172.21.15.45:0/2214443854 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff30c0051c0 0x7ff30c00dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.837+0000 7ff325400640 1 -- 172.21.15.45:0/2214443854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff30c003a00 msgr2=0x7ff30c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:55.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.837+0000 7ff325400640 1 --2- 172.21.15.45:0/2214443854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff30c003a00 0x7ff30c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.837+0000 7ff325400640 1 -- 172.21.15.45:0/2214443854 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff30c003520 con 0x7ff30c004800 2024-04-16T16:14:55.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.837+0000 7ff326800640 1 --2- 172.21.15.45:0/2214443854 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff30c0051c0 0x7ff30c00dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:55.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.837+0000 7ff326800640 1 -- 172.21.15.45:0/2214443854 reap_dead start 2024-04-16T16:14:55.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.838+0000 7ff325400640 1 --2- 172.21.15.45:0/2214443854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff30c004800 0x7ff30c004c80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff310007fd0 tx=0x7ff30c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1086d92c481cdef8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:55.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.838+0000 7ff324a00640 1 -- 172.21.15.45:0/2214443854 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff310007c20 con 0x7ff30c004800 2024-04-16T16:14:55.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.838+0000 7ff324a00640 1 -- 172.21.15.45:0/2214443854 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff310030070 con 0x7ff30c004800 2024-04-16T16:14:55.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.838+0000 7ff324a00640 1 -- 172.21.15.45:0/2214443854 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff310005d70 con 0x7ff30c004800 2024-04-16T16:14:55.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.839+0000 7ff32cc00640 1 -- 172.21.15.45:0/2214443854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff30c004800 msgr2=0x7ff30c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:55.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.839+0000 7ff32cc00640 1 --2- 172.21.15.45:0/2214443854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff30c004800 0x7ff30c004c80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff310007fd0 tx=0x7ff30c005830 comp rx=0 tx=0).stop 2024-04-16T16:14:55.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.839+0000 7ff32cc00640 1 -- 172.21.15.45:0/2214443854 shutdown_connections 2024-04-16T16:14:55.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.839+0000 7ff32cc00640 1 --2- 172.21.15.45:0/2214443854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff30c004800 0x7ff30c004c80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.839+0000 7ff32cc00640 1 -- 172.21.15.45:0/2214443854 >> 172.21.15.45:0/2214443854 conn(0x7ff3280b3970 msgr2=0x7ff3280b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:55.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.839+0000 7ff32cc00640 1 -- 172.21.15.45:0/2214443854 shutdown_connections 2024-04-16T16:14:55.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.839+0000 7ff32cc00640 1 -- 172.21.15.45:0/2214443854 wait complete. 2024-04-16T16:14:55.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.841+0000 7ff32cc00640 1 Processor -- start 2024-04-16T16:14:55.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.841+0000 7ff32cc00640 1 -- start start 2024-04-16T16:14:55.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.842+0000 7ff32cc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff30c004800 0x7ff32813fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:55.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.842+0000 7ff32cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3281f35b0 0x7ff32815e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:55.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.842+0000 7ff32cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff32815e820 0x7ff32814e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:55.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.842+0000 7ff32cc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff3280b51b0 con 0x7ff32815e820 2024-04-16T16:14:55.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.842+0000 7ff32cc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff3280b5030 con 0x7ff3281f35b0 2024-04-16T16:14:55.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.842+0000 7ff32cc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff32806ec30 con 0x7ff30c004800 2024-04-16T16:14:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.842+0000 7ff326800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff32815e820 0x7ff32814e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.842+0000 7ff326800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff32815e820 0x7ff32814e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41758/0 (socket says 172.21.15.45:41758) 2024-04-16T16:14:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.842+0000 7ff326800640 1 -- 172.21.15.45:0/3259161104 learned_addr learned my addr 172.21.15.45:0/3259161104 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.842+0000 7ff325e00640 1 --2- 172.21.15.45:0/3259161104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff30c004800 0x7ff32813fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.842+0000 7ff325400640 1 --2- 172.21.15.45:0/3259161104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3281f35b0 0x7ff32815e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.843+0000 7ff326800640 1 -- 172.21.15.45:0/3259161104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff30c004800 msgr2=0x7ff32813fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.843+0000 7ff326800640 1 --2- 172.21.15.45:0/3259161104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff30c004800 0x7ff32813fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.843+0000 7ff326800640 1 -- 172.21.15.45:0/3259161104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3281f35b0 msgr2=0x7ff32815e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.843+0000 7ff326800640 1 --2- 172.21.15.45:0/3259161104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3281f35b0 0x7ff32815e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.843+0000 7ff326800640 1 -- 172.21.15.45:0/3259161104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3280b1780 con 0x7ff32815e820 2024-04-16T16:14:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.843+0000 7ff326800640 1 --2- 172.21.15.45:0/3259161104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff32815e820 0x7ff32814e7f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff3280b4580 tx=0x7ff32000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:55.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.843+0000 7ff31e000640 1 -- 172.21.15.45:0/3259161104 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff320016620 con 0x7ff32815e820 2024-04-16T16:14:55.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.843+0000 7ff31e000640 1 -- 172.21.15.45:0/3259161104 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff32001b070 con 0x7ff32815e820 2024-04-16T16:14:55.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.843+0000 7ff31e000640 1 -- 172.21.15.45:0/3259161104 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3200169e0 con 0x7ff32815e820 2024-04-16T16:14:55.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.843+0000 7ff32cc00640 1 -- 172.21.15.45:0/3259161104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3281f3da0 con 0x7ff32815e820 2024-04-16T16:14:55.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.843+0000 7ff32cc00640 1 -- 172.21.15.45:0/3259161104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3281f4360 con 0x7ff32815e820 2024-04-16T16:14:55.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.844+0000 7ff32cc00640 1 -- 172.21.15.45:0/3259161104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff328075780 con 0x7ff32815e820 2024-04-16T16:14:55.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.845+0000 7ff31e000640 1 -- 172.21.15.45:0/3259161104 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff320016b80 con 0x7ff32815e820 2024-04-16T16:14:55.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.846+0000 7ff31e000640 1 --2- 172.21.15.45:0/3259161104 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff30406c8f0 0x7ff30406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:55.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.846+0000 7ff31e000640 1 -- 172.21.15.45:0/3259161104 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7ff32008ca70 con 0x7ff32815e820 2024-04-16T16:14:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.847+0000 7ff325e00640 1 --2- 172.21.15.45:0/3259161104 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff30406c8f0 0x7ff30406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:55.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.849+0000 7ff325e00640 1 --2- 172.21.15.45:0/3259161104 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff30406c8f0 0x7ff30406edb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff318005490 tx=0x7ff318009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:55.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:55.852+0000 7ff31e000640 1 -- 172.21.15.45:0/3259161104 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff3280b1780 con 0x7ff32815e820 2024-04-16T16:14:56.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:56.102+0000 7ff32cc00640 1 -- 172.21.15.45:0/3259161104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "stats"} v 0) -- 0x7ff32814dc70 con 0x7ff32815e820 2024-04-16T16:14:56.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:56.792+0000 7ff31e000640 1 -- 172.21.15.45:0/3259161104 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "stats"}]=0 v6) ==== 84+0+0 (secure 0 0 0) 0x7ff32005ae00 con 0x7ff32815e820 2024-04-16T16:14:56.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:56.796+0000 7ff32cc00640 1 -- 172.21.15.45:0/3259161104 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff30406c8f0 msgr2=0x7ff30406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:56.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:56.796+0000 7ff32cc00640 1 --2- 172.21.15.45:0/3259161104 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff30406c8f0 0x7ff30406edb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff318005490 tx=0x7ff318009040 comp rx=0 tx=0).stop 2024-04-16T16:14:56.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:56.796+0000 7ff32cc00640 1 -- 172.21.15.45:0/3259161104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff32815e820 msgr2=0x7ff32814e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:56.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:56.796+0000 7ff32cc00640 1 --2- 172.21.15.45:0/3259161104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff32815e820 0x7ff32814e7f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff3280b4580 tx=0x7ff32000f520 comp rx=0 tx=0).stop 2024-04-16T16:14:56.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:56.797+0000 7ff32cc00640 1 -- 172.21.15.45:0/3259161104 shutdown_connections 2024-04-16T16:14:56.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:56.797+0000 7ff32cc00640 1 --2- 172.21.15.45:0/3259161104 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7ff30406c8f0 0x7ff30406edb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:56.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:56.797+0000 7ff32cc00640 1 --2- 172.21.15.45:0/3259161104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff32815e820 0x7ff32814e7f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:56.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:56.797+0000 7ff32cc00640 1 --2- 172.21.15.45:0/3259161104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3281f35b0 0x7ff32815e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:56.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:56.797+0000 7ff32cc00640 1 --2- 172.21.15.45:0/3259161104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff30c004800 0x7ff32813fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:56.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:56.797+0000 7ff32cc00640 1 -- 172.21.15.45:0/3259161104 >> 172.21.15.45:0/3259161104 conn(0x7ff3280b3970 msgr2=0x7ff3280cf940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:56.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:56.799+0000 7ff32cc00640 1 -- 172.21.15.45:0/3259161104 shutdown_connections 2024-04-16T16:14:56.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:56.799+0000 7ff32cc00640 1 -- 172.21.15.45:0/3259161104 wait complete. 2024-04-16T16:14:56.833 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:14:56.833 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:56.881 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:14:56.881 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:56.961 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.207+0000 7f8a4f600640 1 Processor -- start 2024-04-16T16:14:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.208+0000 7f8a4f600640 1 -- start start 2024-04-16T16:14:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.209+0000 7f8a4f600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8a481462e0 0x7f8a48072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.210+0000 7f8a4f600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8a4815ea70 0x7f8a480cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.210+0000 7f8a4f600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a480cf6d0 con 0x7f8a480728c0 2024-04-16T16:14:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.210+0000 7f8a4f600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8a481466c0 con 0x7f8a481462e0 2024-04-16T16:14:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.210+0000 7f8a4f600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f8a480cf3f0 con 0x7f8a4815ea70 2024-04-16T16:14:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.210+0000 7f8a4cc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8a481462e0 0x7f8a48072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.210+0000 7f8a4cc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8a481462e0 0x7f8a48072380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41786/0 (socket says 172.21.15.45:41786) 2024-04-16T16:14:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.210+0000 7f8a4cc00640 1 -- 172.21.15.45:0/3389031769 learned_addr learned my addr 172.21.15.45:0/3389031769 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.210+0000 7f8a4cc00640 1 -- 172.21.15.45:0/3389031769 >> v2:172.21.15.103:3300/0 conn(0x7f8a4815ea70 msgr2=0x7f8a480cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.210+0000 7f8a4d600640 1 --2- 172.21.15.45:0/3389031769 >> v2:172.21.15.103:3300/0 conn(0x7f8a4815ea70 0x7f8a480cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.211+0000 7f8a4cc00640 1 --2- 172.21.15.45:0/3389031769 >> v2:172.21.15.103:3300/0 conn(0x7f8a4815ea70 0x7f8a480cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.211+0000 7f8a4cc00640 1 -- 172.21.15.45:0/3389031769 >> v1:172.21.15.45:6789/0 conn(0x7f8a480728c0 legacy=0x7f8a480ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.211+0000 7f8a4cc00640 1 -- 172.21.15.45:0/3389031769 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4815cf80 con 0x7f8a481462e0 2024-04-16T16:14:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.211+0000 7f8a4d600640 1 --2- 172.21.15.45:0/3389031769 >> v2:172.21.15.103:3300/0 conn(0x7f8a4815ea70 0x7f8a480cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:57.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.212+0000 7f8a4cc00640 1 --2- 172.21.15.45:0/3389031769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a481462e0 0x7f8a48072380 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8a4806f9b0 tx=0x7f8a3c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3cdb3743c5849b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:57.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.212+0000 7f8a47400640 1 -- 172.21.15.45:0/3389031769 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c035020 con 0x7f8a481462e0 2024-04-16T16:14:57.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.212+0000 7f8a47400640 1 -- 172.21.15.45:0/3389031769 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a3c031870 con 0x7f8a481462e0 2024-04-16T16:14:57.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.212+0000 7f8a47400640 1 -- 172.21.15.45:0/3389031769 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c031c40 con 0x7f8a481462e0 2024-04-16T16:14:57.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.213+0000 7f8a4f600640 1 -- 172.21.15.45:0/3389031769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a481462e0 msgr2=0x7f8a48072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:57.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.213+0000 7f8a4f600640 1 --2- 172.21.15.45:0/3389031769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a481462e0 0x7f8a48072380 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8a4806f9b0 tx=0x7f8a3c02b910 comp rx=0 tx=0).stop 2024-04-16T16:14:57.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.213+0000 7f8a4f600640 1 -- 172.21.15.45:0/3389031769 shutdown_connections 2024-04-16T16:14:57.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.213+0000 7f8a4f600640 1 --2- 172.21.15.45:0/3389031769 >> v2:172.21.15.103:3300/0 conn(0x7f8a4815ea70 0x7f8a480cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.213+0000 7f8a4f600640 1 --2- 172.21.15.45:0/3389031769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a481462e0 0x7f8a48072380 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.213+0000 7f8a4f600640 1 -- 172.21.15.45:0/3389031769 >> 172.21.15.45:0/3389031769 conn(0x7f8a480b2750 msgr2=0x7f8a480b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:57.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.213+0000 7f8a4f600640 1 -- 172.21.15.45:0/3389031769 shutdown_connections 2024-04-16T16:14:57.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.213+0000 7f8a4f600640 1 -- 172.21.15.45:0/3389031769 wait complete. 2024-04-16T16:14:57.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.215+0000 7f8a4f600640 1 Processor -- start 2024-04-16T16:14:57.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.215+0000 7f8a4f600640 1 -- start start 2024-04-16T16:14:57.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.216+0000 7f8a4f600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a480728c0 0x7f8a480b5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:57.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.216+0000 7f8a4f600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a481462e0 0x7f8a4815c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:57.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.216+0000 7f8a4f600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a4815ea70 0x7f8a480ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:57.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.216+0000 7f8a4f600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a48166490 con 0x7f8a480728c0 2024-04-16T16:14:57.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.216+0000 7f8a4f600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4815d540 con 0x7f8a4815ea70 2024-04-16T16:14:57.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.216+0000 7f8a4f600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a48166610 con 0x7f8a481462e0 2024-04-16T16:14:57.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.216+0000 7f8a4cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a480728c0 0x7f8a480b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:57.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.216+0000 7f8a4cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a480728c0 0x7f8a480b5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41798/0 (socket says 172.21.15.45:41798) 2024-04-16T16:14:57.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.216+0000 7f8a4cc00640 1 -- 172.21.15.45:0/977558536 learned_addr learned my addr 172.21.15.45:0/977558536 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:57.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.216+0000 7f8a4cc00640 1 -- 172.21.15.45:0/977558536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a481462e0 msgr2=0x7f8a4815c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:57.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.216+0000 7f8a4cc00640 1 --2- 172.21.15.45:0/977558536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a481462e0 0x7f8a4815c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.216+0000 7f8a4cc00640 1 -- 172.21.15.45:0/977558536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a4815ea70 msgr2=0x7f8a480ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:57.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.216+0000 7f8a4cc00640 1 --2- 172.21.15.45:0/977558536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a4815ea70 0x7f8a480ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.217+0000 7f8a4cc00640 1 -- 172.21.15.45:0/977558536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a480b41f0 con 0x7f8a480728c0 2024-04-16T16:14:57.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.217+0000 7f8a4cc00640 1 --2- 172.21.15.45:0/977558536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a480728c0 0x7f8a480b5220 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c02b950 tx=0x7f8a3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:57.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.217+0000 7f8a44c00640 1 -- 172.21.15.45:0/977558536 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c040070 con 0x7f8a480728c0 2024-04-16T16:14:57.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.217+0000 7f8a44c00640 1 -- 172.21.15.45:0/977558536 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a3c03a420 con 0x7f8a480728c0 2024-04-16T16:14:57.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.217+0000 7f8a44c00640 1 -- 172.21.15.45:0/977558536 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c035040 con 0x7f8a480728c0 2024-04-16T16:14:57.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.217+0000 7f8a4f600640 1 -- 172.21.15.45:0/977558536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a480b5760 con 0x7f8a480728c0 2024-04-16T16:14:57.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.217+0000 7f8a4f600640 1 -- 172.21.15.45:0/977558536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a480b4740 con 0x7f8a480728c0 2024-04-16T16:14:57.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.219+0000 7f8a4f600640 1 -- 172.21.15.45:0/977558536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a48075780 con 0x7f8a480728c0 2024-04-16T16:14:57.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.219+0000 7f8a44c00640 1 -- 172.21.15.45:0/977558536 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 6) ==== 91049+0+0 (secure 0 0 0) 0x7f8a3c038020 con 0x7f8a480728c0 2024-04-16T16:14:57.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.220+0000 7f8a44c00640 1 --2- 172.21.15.45:0/977558536 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a2806c940 0x7f8a2806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:57.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.220+0000 7f8a44c00640 1 -- 172.21.15.45:0/977558536 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f8a3c0adbb0 con 0x7f8a480728c0 2024-04-16T16:14:57.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.220+0000 7f8a47e00640 1 -- 172.21.15.45:0/977558536 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a2806c940 msgr2=0x7f8a2806ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:14:57.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.221+0000 7f8a47e00640 1 --2- 172.21.15.45:0/977558536 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a2806c940 0x7f8a2806ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:14:57.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.225+0000 7f8a44c00640 1 -- 172.21.15.45:0/977558536 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8a3c07bfd0 con 0x7f8a480728c0 2024-04-16T16:14:57.369 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:57.369+0000 7f86c0a12040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:14:57.379 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:57.379+0000 7fdf7d7db040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:14:57.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.422+0000 7f8a47e00640 1 -- 172.21.15.45:0/977558536 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a2806c940 msgr2=0x7f8a2806ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:14:57.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.422+0000 7f8a47e00640 1 --2- 172.21.15.45:0/977558536 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a2806c940 0x7f8a2806ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:14:57.423 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:57.423+0000 7f86c0a12040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:14:57.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.430+0000 7f8a4f600640 1 -- 172.21.15.45:0/977558536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a48142e00 con 0x7f8a480728c0 2024-04-16T16:14:57.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.432+0000 7f8a44c00640 1 -- 172.21.15.45:0/977558536 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16669 (secure 0 0 0) 0x7f8a3c07b8d0 con 0x7f8a480728c0 2024-04-16T16:14:57.432 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:14:57.433 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":34,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:14:40.675999+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:14:37.171798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-16T16:14:38.122496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:14:57.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.435+0000 7f8a4f600640 1 -- 172.21.15.45:0/977558536 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a2806c940 msgr2=0x7f8a2806ee00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:14:57.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.435+0000 7f8a4f600640 1 --2- 172.21.15.45:0/977558536 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a2806c940 0x7f8a2806ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.435+0000 7f8a4f600640 1 -- 172.21.15.45:0/977558536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a480728c0 msgr2=0x7f8a480b5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:57.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.435+0000 7f8a4f600640 1 --2- 172.21.15.45:0/977558536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a480728c0 0x7f8a480b5220 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c02b950 tx=0x7f8a3c0046c0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.435+0000 7f8a4f600640 1 -- 172.21.15.45:0/977558536 shutdown_connections 2024-04-16T16:14:57.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.435+0000 7f8a4f600640 1 --2- 172.21.15.45:0/977558536 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f8a2806c940 0x7f8a2806ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.435+0000 7f8a4f600640 1 --2- 172.21.15.45:0/977558536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a4815ea70 0x7f8a480ae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.435+0000 7f8a4f600640 1 --2- 172.21.15.45:0/977558536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a481462e0 0x7f8a4815c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.435+0000 7f8a4f600640 1 --2- 172.21.15.45:0/977558536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a480728c0 0x7f8a480b5220 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.435+0000 7f8a4f600640 1 -- 172.21.15.45:0/977558536 >> 172.21.15.45:0/977558536 conn(0x7f8a480b2750 msgr2=0x7f8a48074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:57.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.435+0000 7f8a4f600640 1 -- 172.21.15.45:0/977558536 shutdown_connections 2024-04-16T16:14:57.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.435+0000 7f8a4f600640 1 -- 172.21.15.45:0/977558536 wait complete. 2024-04-16T16:14:57.437 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:57.438+0000 7fdf7d7db040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:14:57.454 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:14:37.171798+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:14:38.122496+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:14:57.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:14:57.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.663+0000 7f1c3e600640 1 Processor -- start 2024-04-16T16:14:57.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.663+0000 7f1c3e600640 1 -- start start 2024-04-16T16:14:57.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.664+0000 7f1c3e600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1c380b6270 0x7f1c380d2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:57.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.665+0000 7f1c3e600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c380ba2a0 0x7f1c380b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:57.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.665+0000 7f1c3e600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c380b6b90 con 0x7f1c38073060 2024-04-16T16:14:57.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.665+0000 7f1c3e600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1c380d32c0 con 0x7f1c380b6270 2024-04-16T16:14:57.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.665+0000 7f1c3e600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c380d35a0 con 0x7f1c380ba2a0 2024-04-16T16:14:57.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.665+0000 7f1c3dc00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f1c38073060 0x7f1c380734f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:42650/0 (socket says 172.21.15.45:42650) 2024-04-16T16:14:57.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.665+0000 7f1c3dc00640 1 -- 172.21.15.45:0/509899831 learned_addr learned my addr 172.21.15.45:0/509899831 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:57.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.665+0000 7f1c37e00640 1 --2- 172.21.15.45:0/509899831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c380ba2a0 0x7f1c380b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:57.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.665+0000 7f1c3d200640 1 --2- 172.21.15.45:0/509899831 >> v2:172.21.15.103:3300/0 conn(0x7f1c380b6270 0x7f1c380d2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:57.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.666+0000 7f1c37e00640 1 -- 172.21.15.45:0/509899831 >> v2:172.21.15.103:3300/0 conn(0x7f1c380b6270 msgr2=0x7f1c380d2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:57.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.666+0000 7f1c37e00640 1 --2- 172.21.15.45:0/509899831 >> v2:172.21.15.103:3300/0 conn(0x7f1c380b6270 0x7f1c380d2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.666+0000 7f1c37e00640 1 -- 172.21.15.45:0/509899831 >> v1:172.21.15.103:6789/0 conn(0x7f1c38073060 legacy=0x7f1c380734f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:57.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.666+0000 7f1c37e00640 1 -- 172.21.15.45:0/509899831 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c380d3880 con 0x7f1c380ba2a0 2024-04-16T16:14:57.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.668+0000 7f1c37e00640 1 --2- 172.21.15.45:0/509899831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c380ba2a0 0x7f1c380b6650 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1c240097a0 tx=0x7f1c2402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2178bb2074d3b52 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:57.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.668+0000 7f1c37400640 1 -- 172.21.15.45:0/509899831 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c24035020 con 0x7f1c380ba2a0 2024-04-16T16:14:57.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.669+0000 7f1c37400640 1 -- 172.21.15.45:0/509899831 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c24031810 con 0x7f1c380ba2a0 2024-04-16T16:14:57.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.669+0000 7f1c37400640 1 -- 172.21.15.45:0/509899831 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c24031be0 con 0x7f1c380ba2a0 2024-04-16T16:14:57.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.669+0000 7f1c3e600640 1 -- 172.21.15.45:0/509899831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c380ba2a0 msgr2=0x7f1c380b6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:57.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.669+0000 7f1c3e600640 1 --2- 172.21.15.45:0/509899831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c380ba2a0 0x7f1c380b6650 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1c240097a0 tx=0x7f1c2402b8b0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.669+0000 7f1c3e600640 1 -- 172.21.15.45:0/509899831 shutdown_connections 2024-04-16T16:14:57.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.669+0000 7f1c3e600640 1 --2- 172.21.15.45:0/509899831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c380ba2a0 0x7f1c380b6650 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.669+0000 7f1c3e600640 1 --2- 172.21.15.45:0/509899831 >> v2:172.21.15.103:3300/0 conn(0x7f1c380b6270 0x7f1c380d2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.669+0000 7f1c3e600640 1 -- 172.21.15.45:0/509899831 >> 172.21.15.45:0/509899831 conn(0x7f1c3806d300 msgr2=0x7f1c380d1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:57.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.669+0000 7f1c3e600640 1 -- 172.21.15.45:0/509899831 shutdown_connections 2024-04-16T16:14:57.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.669+0000 7f1c3e600640 1 -- 172.21.15.45:0/509899831 wait complete. 2024-04-16T16:14:57.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.671+0000 7f1c3e600640 1 Processor -- start 2024-04-16T16:14:57.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.672+0000 7f1c3e600640 1 -- start start 2024-04-16T16:14:57.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.673+0000 7f1c3e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c38073060 0x7f1c381609f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:57.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.673+0000 7f1c3e600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c380b6270 0x7f1c38150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:57.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.673+0000 7f1c3e600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c380ba2a0 0x7f1c381e7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:57.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.673+0000 7f1c3e600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1c380ba700 con 0x7f1c38073060 2024-04-16T16:14:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.673+0000 7f1c3e600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c380d3e40 con 0x7f1c380ba2a0 2024-04-16T16:14:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.673+0000 7f1c3e600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c380b8930 con 0x7f1c380b6270 2024-04-16T16:14:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.673+0000 7f1c3d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c38073060 0x7f1c381609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.673+0000 7f1c3d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c38073060 0x7f1c381609f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41802/0 (socket says 172.21.15.45:41802) 2024-04-16T16:14:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.673+0000 7f1c3d200640 1 -- 172.21.15.45:0/1146530297 learned_addr learned my addr 172.21.15.45:0/1146530297 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.673+0000 7f1c3d200640 1 -- 172.21.15.45:0/1146530297 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c380b6270 msgr2=0x7f1c38150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.673+0000 7f1c3d200640 1 --2- 172.21.15.45:0/1146530297 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c380b6270 0x7f1c38150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.673+0000 7f1c3d200640 1 -- 172.21.15.45:0/1146530297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c380ba2a0 msgr2=0x7f1c381e7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.673+0000 7f1c3dc00640 1 --2- 172.21.15.45:0/1146530297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c380ba2a0 0x7f1c381e7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.673+0000 7f1c3d200640 1 --2- 172.21.15.45:0/1146530297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c380ba2a0 0x7f1c381e7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.673+0000 7f1c3d200640 1 -- 172.21.15.45:0/1146530297 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c380cce90 con 0x7f1c38073060 2024-04-16T16:14:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.674+0000 7f1c3d200640 1 --2- 172.21.15.45:0/1146530297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c38073060 0x7f1c381609f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c00e9b0 tx=0x7f1c2c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.674+0000 7f1c34c00640 1 -- 172.21.15.45:0/1146530297 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c017070 con 0x7f1c38073060 2024-04-16T16:14:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.674+0000 7f1c34c00640 1 -- 172.21.15.45:0/1146530297 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c2c00c9a0 con 0x7f1c38073060 2024-04-16T16:14:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.674+0000 7f1c34c00640 1 -- 172.21.15.45:0/1146530297 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c00cd90 con 0x7f1c38073060 2024-04-16T16:14:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.674+0000 7f1c3e600640 1 -- 172.21.15.45:0/1146530297 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c380cd170 con 0x7f1c38073060 2024-04-16T16:14:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.674+0000 7f1c3e600640 1 -- 172.21.15.45:0/1146530297 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c38160f30 con 0x7f1c38073060 2024-04-16T16:14:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.676+0000 7f1c3e600640 1 -- 172.21.15.45:0/1146530297 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c38075780 con 0x7f1c38073060 2024-04-16T16:14:57.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.676+0000 7f1c34c00640 1 -- 172.21.15.45:0/1146530297 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 6) ==== 91049+0+0 (secure 0 0 0) 0x7f1c2c01a070 con 0x7f1c38073060 2024-04-16T16:14:57.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.678+0000 7f1c34c00640 1 --2- 172.21.15.45:0/1146530297 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1c1806c970 0x7f1c1806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:57.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.678+0000 7f1c34c00640 1 -- 172.21.15.45:0/1146530297 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f1c2c08bf70 con 0x7f1c38073060 2024-04-16T16:14:57.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.678+0000 7f1c37e00640 1 -- 172.21.15.45:0/1146530297 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1c1806c970 msgr2=0x7f1c1806ee30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:14:57.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.678+0000 7f1c37e00640 1 --2- 172.21.15.45:0/1146530297 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1c1806c970 0x7f1c1806ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:14:57.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.682+0000 7f1c34c00640 1 -- 172.21.15.45:0/1146530297 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1c2c05a390 con 0x7f1c38073060 2024-04-16T16:14:57.868 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:57.869+0000 7f86c0a12040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:14:57.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.879+0000 7f1c37e00640 1 -- 172.21.15.45:0/1146530297 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1c1806c970 msgr2=0x7f1c1806ee30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:14:57.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.879+0000 7f1c37e00640 1 --2- 172.21.15.45:0/1146530297 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1c1806c970 0x7f1c1806ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:14:57.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.892+0000 7f1c3e600640 1 -- 172.21.15.45:0/1146530297 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1c380d2f40 con 0x7f1c38073060 2024-04-16T16:14:57.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.893+0000 7f1c34c00640 1 -- 172.21.15.45:0/1146530297 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) ==== 93+0+10 (secure 0 0 0) 0x7f1c2c059c90 con 0x7f1c38073060 2024-04-16T16:14:57.892 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:14:57.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.895+0000 7f1c3e600640 1 -- 172.21.15.45:0/1146530297 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1c1806c970 msgr2=0x7f1c1806ee30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:14:57.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.895+0000 7f1c3e600640 1 --2- 172.21.15.45:0/1146530297 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1c1806c970 0x7f1c1806ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.896+0000 7f1c3e600640 1 -- 172.21.15.45:0/1146530297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c38073060 msgr2=0x7f1c381609f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:57.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.896+0000 7f1c3e600640 1 --2- 172.21.15.45:0/1146530297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c38073060 0x7f1c381609f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c00e9b0 tx=0x7f1c2c00ee70 comp rx=0 tx=0).stop 2024-04-16T16:14:57.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.896+0000 7f1c3e600640 1 -- 172.21.15.45:0/1146530297 shutdown_connections 2024-04-16T16:14:57.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.896+0000 7f1c3e600640 1 --2- 172.21.15.45:0/1146530297 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f1c1806c970 0x7f1c1806ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.896+0000 7f1c3e600640 1 --2- 172.21.15.45:0/1146530297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c380ba2a0 0x7f1c381e7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.896+0000 7f1c3e600640 1 --2- 172.21.15.45:0/1146530297 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c380b6270 0x7f1c38150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.896+0000 7f1c3e600640 1 --2- 172.21.15.45:0/1146530297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c38073060 0x7f1c381609f0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:57.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.896+0000 7f1c3e600640 1 -- 172.21.15.45:0/1146530297 >> 172.21.15.45:0/1146530297 conn(0x7f1c3806d300 msgr2=0x7f1c380d47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:57.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.896+0000 7f1c3e600640 1 -- 172.21.15.45:0/1146530297 shutdown_connections 2024-04-16T16:14:57.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:57.896+0000 7f1c3e600640 1 -- 172.21.15.45:0/1146530297 wait complete. 2024-04-16T16:14:57.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:14:57.921 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:57.922+0000 7f86c0a12040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:14:57.926 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:57.927+0000 7fdf7d7db040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:14:57.971 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:57.972+0000 7f86c0a12040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:14:57.984 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:57.985+0000 7fdf7d7db040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:14:58.033 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:58.033+0000 7f86c0a12040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:14:58.038 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:58.039+0000 7fdf7d7db040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:14:58.083 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:58.084+0000 7f86c0a12040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:14:58.102 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:58.104+0000 7fdf7d7db040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:14:58.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.116+0000 7f854f200640 1 Processor -- start 2024-04-16T16:14:58.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.117+0000 7f854f200640 1 -- start start 2024-04-16T16:14:58.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.118+0000 7f854f200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f85500d18f0 0x7f85500ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:58.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.118+0000 7f854f200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85500b6810 con 0x7f85500b6310 2024-04-16T16:14:58.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.118+0000 7f854f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85500b6af0 con 0x7f85500d1cd0 2024-04-16T16:14:58.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.118+0000 7f854f200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f85500d29f0 con 0x7f85500d18f0 2024-04-16T16:14:58.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.119+0000 7f854de00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f85500d18f0 0x7f85500ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:58.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.119+0000 7f8547e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f85500b6310 0x7f85500d2e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57154/0 (socket says 172.21.15.45:57154) 2024-04-16T16:14:58.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.119+0000 7f8547e00640 1 -- 172.21.15.45:0/3412843243 learned_addr learned my addr 172.21.15.45:0/3412843243 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:58.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.119+0000 7f854de00640 1 -- 172.21.15.45:0/3412843243 >> v1:172.21.15.103:6789/0 conn(0x7f85500d1cd0 legacy=0x7f8550162dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:58.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.119+0000 7f854de00640 1 -- 172.21.15.45:0/3412843243 >> v1:172.21.15.45:6789/0 conn(0x7f85500b6310 legacy=0x7f85500d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:58.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.119+0000 7f854de00640 1 -- 172.21.15.45:0/3412843243 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85500b6dd0 con 0x7f85500d18f0 2024-04-16T16:14:58.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.120+0000 7f854de00640 1 --2- 172.21.15.45:0/3412843243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85500d18f0 0x7f85500ba220 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f85500d2b70 tx=0x7f854002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912a8b003b6fd018 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:58.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.121+0000 7f854d400640 1 -- 172.21.15.45:0/3412843243 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540031780 con 0x7f85500d18f0 2024-04-16T16:14:58.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.121+0000 7f854d400640 1 -- 172.21.15.45:0/3412843243 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8540037070 con 0x7f85500d18f0 2024-04-16T16:14:58.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.121+0000 7f854d400640 1 -- 172.21.15.45:0/3412843243 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540031b40 con 0x7f85500d18f0 2024-04-16T16:14:58.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.121+0000 7f854f200640 1 -- 172.21.15.45:0/3412843243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85500d18f0 msgr2=0x7f85500ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:58.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.121+0000 7f854f200640 1 --2- 172.21.15.45:0/3412843243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85500d18f0 0x7f85500ba220 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f85500d2b70 tx=0x7f854002b820 comp rx=0 tx=0).stop 2024-04-16T16:14:58.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.122+0000 7f854f200640 1 -- 172.21.15.45:0/3412843243 shutdown_connections 2024-04-16T16:14:58.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.122+0000 7f854f200640 1 --2- 172.21.15.45:0/3412843243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85500d18f0 0x7f85500ba220 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.122+0000 7f854f200640 1 -- 172.21.15.45:0/3412843243 >> 172.21.15.45:0/3412843243 conn(0x7f855006cf90 msgr2=0x7f855006d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:58.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.122+0000 7f854f200640 1 -- 172.21.15.45:0/3412843243 shutdown_connections 2024-04-16T16:14:58.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.122+0000 7f854f200640 1 -- 172.21.15.45:0/3412843243 wait complete. 2024-04-16T16:14:58.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.123+0000 7f854f200640 1 Processor -- start 2024-04-16T16:14:58.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.123+0000 7f854f200640 1 -- start start 2024-04-16T16:14:58.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.124+0000 7f854f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85500b6310 0x7f85501e9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:58.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.124+0000 7f854f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85500d18f0 0x7f85500cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:58.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.124+0000 7f854f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85500d1cd0 0x7f85501f7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:58.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.125+0000 7f854f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f85500d3590 con 0x7f85500b6310 2024-04-16T16:14:58.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.125+0000 7f854f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f85500ba760 con 0x7f85500d1cd0 2024-04-16T16:14:58.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.125+0000 7f854f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f85500d3710 con 0x7f85500d18f0 2024-04-16T16:14:58.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.125+0000 7f854de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85500b6310 0x7f85501e9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:58.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.125+0000 7f854de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85500b6310 0x7f85501e9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41816/0 (socket says 172.21.15.45:41816) 2024-04-16T16:14:58.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.125+0000 7f854de00640 1 -- 172.21.15.45:0/2064464322 learned_addr learned my addr 172.21.15.45:0/2064464322 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:58.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.125+0000 7f854e800640 1 --2- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85500d1cd0 0x7f85501f7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:58.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.125+0000 7f8547e00640 1 --2- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85500d18f0 0x7f85500cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:58.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.125+0000 7f854de00640 1 -- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85500d18f0 msgr2=0x7f85500cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:58.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.125+0000 7f854de00640 1 --2- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85500d18f0 0x7f85500cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.125+0000 7f854de00640 1 -- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85500d1cd0 msgr2=0x7f85501f7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:58.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.125+0000 7f854de00640 1 --2- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85500d1cd0 0x7f85501f7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.125+0000 7f854de00640 1 -- 172.21.15.45:0/2064464322 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85500cdab0 con 0x7f85500b6310 2024-04-16T16:14:58.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.125+0000 7f8547e00640 1 --2- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85500d18f0 0x7f85500cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:58.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.126+0000 7f854de00640 1 --2- 172.21.15.45:0/2064464322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85500b6310 0x7f85501e9240 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8540034040 tx=0x7f85400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.126+0000 7f8546000640 1 -- 172.21.15.45:0/2064464322 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540032820 con 0x7f85500b6310 2024-04-16T16:14:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.126+0000 7f8546000640 1 -- 172.21.15.45:0/2064464322 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85400329c0 con 0x7f85500b6310 2024-04-16T16:14:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.126+0000 7f8546000640 1 -- 172.21.15.45:0/2064464322 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540032db0 con 0x7f85500b6310 2024-04-16T16:14:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.126+0000 7f854f200640 1 -- 172.21.15.45:0/2064464322 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85500cdd90 con 0x7f85500b6310 2024-04-16T16:14:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.126+0000 7f854f200640 1 -- 172.21.15.45:0/2064464322 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85500ce350 con 0x7f85500b6310 2024-04-16T16:14:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.127+0000 7f854f200640 1 -- 172.21.15.45:0/2064464322 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8550075780 con 0x7f85500b6310 2024-04-16T16:14:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.128+0000 7f8546000640 1 -- 172.21.15.45:0/2064464322 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 6) ==== 91049+0+0 (secure 0 0 0) 0x7f854004e440 con 0x7f85500b6310 2024-04-16T16:14:58.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.129+0000 7f8546000640 1 --2- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f852c06c940 0x7f852c06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:58.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.129+0000 7f8546000640 1 -- 172.21.15.45:0/2064464322 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f85400adbe0 con 0x7f85500b6310 2024-04-16T16:14:58.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.129+0000 7f8547e00640 1 -- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f852c06c940 msgr2=0x7f852c06ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:14:58.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.129+0000 7f8547e00640 1 --2- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f852c06c940 0x7f852c06ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:14:58.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.134+0000 7f8546000640 1 -- 172.21.15.45:0/2064464322 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8540040070 con 0x7f85500b6310 2024-04-16T16:14:58.157 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:58.158+0000 7fdf7d7db040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:14:58.214 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:58.215+0000 7f86c0a12040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:14:58.277 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:58.277+0000 7f86c0a12040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:14:58.294 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:58.296+0000 7fdf7d7db040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:14:58.327 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:58.327+0000 7f86c0a12040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:14:58.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.330+0000 7f8547e00640 1 -- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f852c06c940 msgr2=0x7f852c06ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:14:58.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.330+0000 7f8547e00640 1 --2- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f852c06c940 0x7f852c06ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:14:58.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.338+0000 7f854f200640 1 -- 172.21.15.45:0/2064464322 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f85500d25a0 con 0x7f85500b6310 2024-04-16T16:14:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.339+0000 7f8546000640 1 -- 172.21.15.45:0/2064464322 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) ==== 104+0+11 (secure 0 0 0) 0x7f854002f050 con 0x7f85500b6310 2024-04-16T16:14:58.338 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:14:58.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.341+0000 7f854f200640 1 -- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f852c06c940 msgr2=0x7f852c06ee00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:14:58.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.341+0000 7f854f200640 1 --2- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f852c06c940 0x7f852c06ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.342+0000 7f854f200640 1 -- 172.21.15.45:0/2064464322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85500b6310 msgr2=0x7f85501e9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:58.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.342+0000 7f854f200640 1 --2- 172.21.15.45:0/2064464322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85500b6310 0x7f85501e9240 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8540034040 tx=0x7f85400046c0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.342+0000 7f854f200640 1 -- 172.21.15.45:0/2064464322 shutdown_connections 2024-04-16T16:14:58.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.342+0000 7f854f200640 1 --2- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f852c06c940 0x7f852c06ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.342+0000 7f854f200640 1 --2- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85500d1cd0 0x7f85501f7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.342+0000 7f854f200640 1 --2- 172.21.15.45:0/2064464322 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85500d18f0 0x7f85500cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.342+0000 7f854f200640 1 --2- 172.21.15.45:0/2064464322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85500b6310 0x7f85501e9240 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.342+0000 7f854f200640 1 -- 172.21.15.45:0/2064464322 >> 172.21.15.45:0/2064464322 conn(0x7f855006cf90 msgr2=0x7f85500b87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.342+0000 7f854f200640 1 -- 172.21.15.45:0/2064464322 shutdown_connections 2024-04-16T16:14:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.342+0000 7f854f200640 1 -- 172.21.15.45:0/2064464322 wait complete. 2024-04-16T16:14:58.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:14:58.362 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:58.363+0000 7fdf7d7db040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:14:58.384 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:58.385+0000 7f86c0a12040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:14:58.415 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:58.416+0000 7fdf7d7db040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:14:58.435 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:58.435+0000 7f86c0a12040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:14:58.477 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:58.478+0000 7fdf7d7db040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:14:58.531 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:58.532+0000 7fdf7d7db040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:14:58.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.566+0000 7feb6fe00640 1 Processor -- start 2024-04-16T16:14:58.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.566+0000 7feb6fe00640 1 -- start start 2024-04-16T16:14:58.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.568+0000 7feb6fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7feb700ba500 0x7feb700d4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:58.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.568+0000 7feb6fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb700cf4d0 0x7feb700cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:58.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.568+0000 7feb6fe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb700cfec0 con 0x7feb7015e980 2024-04-16T16:14:58.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.568+0000 7feb6fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7feb700d2f50 con 0x7feb700ba500 2024-04-16T16:14:58.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.568+0000 7feb6fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feb7015ed60 con 0x7feb700cf4d0 2024-04-16T16:14:58.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.568+0000 7feb6ea00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7feb7015e980 0x7feb700d2840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57158/0 (socket says 172.21.15.45:57158) 2024-04-16T16:14:58.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.568+0000 7feb6ea00640 1 -- 172.21.15.45:0/1934249792 learned_addr learned my addr 172.21.15.45:0/1934249792 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:14:58.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6e000640 1 --2- 172.21.15.45:0/1934249792 >> v2:172.21.15.103:3300/0 conn(0x7feb700ba500 0x7feb700d4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:58.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6f400640 1 --2- 172.21.15.45:0/1934249792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb700cf4d0 0x7feb700cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:58.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 -- 172.21.15.45:0/1934249792 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 797837291 0 0) 0x7feb700cfec0 con 0x7feb7015e980 2024-04-16T16:14:58.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 -- 172.21.15.45:0/1934249792 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb54003520 con 0x7feb7015e980 2024-04-16T16:14:58.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 -- 172.21.15.45:0/1934249792 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4057072231 0 0) 0x7feb54003520 con 0x7feb7015e980 2024-04-16T16:14:58.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 -- 172.21.15.45:0/1934249792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb700cf4d0 msgr2=0x7feb700cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:58.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 --2- 172.21.15.45:0/1934249792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb700cf4d0 0x7feb700cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 -- 172.21.15.45:0/1934249792 >> v2:172.21.15.103:3300/0 conn(0x7feb700ba500 msgr2=0x7feb700d4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:58.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 --2- 172.21.15.45:0/1934249792 >> v2:172.21.15.103:3300/0 conn(0x7feb700ba500 0x7feb700d4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 -- 172.21.15.45:0/1934249792 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb700d3170 con 0x7feb7015e980 2024-04-16T16:14:58.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 -- 172.21.15.45:0/1934249792 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7feb60001670 con 0x7feb7015e980 2024-04-16T16:14:58.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 -- 172.21.15.45:0/1934249792 >> v1:172.21.15.45:6789/0 conn(0x7feb7015e980 legacy=0x7feb700d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:58.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 --2- 172.21.15.45:0/1934249792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb540039b0 0x7feb54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:58.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 --2- 172.21.15.45:0/1934249792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb54004800 0x7feb54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:58.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 --2- 172.21.15.45:0/1934249792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb700cf4d0 0x7feb540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:58.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 -- 172.21.15.45:0/1934249792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feb700d2f50 con 0x7feb54004800 2024-04-16T16:14:58.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 -- 172.21.15.45:0/1934249792 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feb7015ed60 con 0x7feb540039b0 2024-04-16T16:14:58.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6d600640 1 -- 172.21.15.45:0/1934249792 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feb5400b9f0 con 0x7feb700cf4d0 2024-04-16T16:14:58.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.569+0000 7feb6e000640 1 --2- 172.21.15.45:0/1934249792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb54004800 0x7feb54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:58.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.570+0000 7feb6e000640 1 -- 172.21.15.45:0/1934249792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb700cf4d0 msgr2=0x7feb540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:58.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.570+0000 7feb6e000640 1 --2- 172.21.15.45:0/1934249792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb700cf4d0 0x7feb540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.570+0000 7feb6e000640 1 -- 172.21.15.45:0/1934249792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb540039b0 msgr2=0x7feb54003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:58.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.570+0000 7feb6e000640 1 --2- 172.21.15.45:0/1934249792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb540039b0 0x7feb54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.570+0000 7feb6e000640 1 -- 172.21.15.45:0/1934249792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb54003520 con 0x7feb54004800 2024-04-16T16:14:58.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.571+0000 7feb6e000640 1 --2- 172.21.15.45:0/1934249792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb54004800 0x7feb54004c80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7feb5c009a90 tx=0x7feb54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ee07a990bd5d5f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:58.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.571+0000 7feb6d600640 1 -- 172.21.15.45:0/1934249792 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5c039070 con 0x7feb54004800 2024-04-16T16:14:58.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.571+0000 7feb6d600640 1 -- 172.21.15.45:0/1934249792 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb5c02c6b0 con 0x7feb54004800 2024-04-16T16:14:58.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.571+0000 7feb6d600640 1 -- 172.21.15.45:0/1934249792 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5c009690 con 0x7feb54004800 2024-04-16T16:14:58.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.572+0000 7feb6fe00640 1 -- 172.21.15.45:0/1934249792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb54004800 msgr2=0x7feb54004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:58.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.572+0000 7feb6fe00640 1 --2- 172.21.15.45:0/1934249792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb54004800 0x7feb54004c80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7feb5c009a90 tx=0x7feb54005830 comp rx=0 tx=0).stop 2024-04-16T16:14:58.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.572+0000 7feb6f400640 1 -- 172.21.15.45:0/1934249792 reap_dead start 2024-04-16T16:14:58.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.572+0000 7feb6fe00640 1 -- 172.21.15.45:0/1934249792 shutdown_connections 2024-04-16T16:14:58.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.572+0000 7feb6fe00640 1 -- 172.21.15.45:0/1934249792 >> 172.21.15.45:0/1934249792 conn(0x7feb7006d2f0 msgr2=0x7feb700d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:58.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.572+0000 7feb6fe00640 1 -- 172.21.15.45:0/1934249792 shutdown_connections 2024-04-16T16:14:58.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.572+0000 7feb6fe00640 1 -- 172.21.15.45:0/1934249792 wait complete. 2024-04-16T16:14:58.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.574+0000 7feb6fe00640 1 Processor -- start 2024-04-16T16:14:58.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.574+0000 7feb6fe00640 1 -- start start 2024-04-16T16:14:58.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.575+0000 7feb6fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700c9100 0x7feb7014e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:58.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.575+0000 7feb6fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb701f4790 0x7feb700cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:58.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.575+0000 7feb6fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb701e71e0 0x7feb700c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:58.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.575+0000 7feb6fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feb700d3a10 con 0x7feb701e71e0 2024-04-16T16:14:58.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.575+0000 7feb6fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feb700d3730 con 0x7feb700c9100 2024-04-16T16:14:58.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.575+0000 7feb6fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feb700d3cf0 con 0x7feb701f4790 2024-04-16T16:14:58.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.575+0000 7feb6f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb701e71e0 0x7feb700c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:58.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.575+0000 7feb6f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb701e71e0 0x7feb700c94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41828/0 (socket says 172.21.15.45:41828) 2024-04-16T16:14:58.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.575+0000 7feb6f400640 1 -- 172.21.15.45:0/2100620104 learned_addr learned my addr 172.21.15.45:0/2100620104 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:58.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.576+0000 7feb6e000640 1 --2- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb701f4790 0x7feb700cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:58.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.576+0000 7feb6ea00640 1 --2- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700c9100 0x7feb7014e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:58.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.576+0000 7feb6f400640 1 -- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb701f4790 msgr2=0x7feb700cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:58.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.576+0000 7feb6f400640 1 --2- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb701f4790 0x7feb700cc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.576+0000 7feb6f400640 1 -- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700c9100 msgr2=0x7feb7014e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:58.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.576+0000 7feb6f400640 1 --2- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700c9100 0x7feb7014e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.576+0000 7feb6f400640 1 -- 172.21.15.45:0/2100620104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb700cca60 con 0x7feb701e71e0 2024-04-16T16:14:58.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.576+0000 7feb6ea00640 1 --2- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700c9100 0x7feb7014e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:14:58.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.576+0000 7feb6f400640 1 --2- 172.21.15.45:0/2100620104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb701e71e0 0x7feb700c94e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7feb7006dbe0 tx=0x7feb6800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:58.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.576+0000 7feb67400640 1 -- 172.21.15.45:0/2100620104 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6800bae0 con 0x7feb701e71e0 2024-04-16T16:14:58.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.576+0000 7feb67400640 1 -- 172.21.15.45:0/2100620104 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb6801a070 con 0x7feb701e71e0 2024-04-16T16:14:58.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.576+0000 7feb67400640 1 -- 172.21.15.45:0/2100620104 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb68014400 con 0x7feb701e71e0 2024-04-16T16:14:58.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.576+0000 7feb6fe00640 1 -- 172.21.15.45:0/2100620104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb700ccd40 con 0x7feb701e71e0 2024-04-16T16:14:58.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.577+0000 7feb6fe00640 1 -- 172.21.15.45:0/2100620104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb700cd300 con 0x7feb701e71e0 2024-04-16T16:14:58.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.578+0000 7feb6fe00640 1 -- 172.21.15.45:0/2100620104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb700c7a70 con 0x7feb701e71e0 2024-04-16T16:14:58.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.579+0000 7feb67400640 1 -- 172.21.15.45:0/2100620104 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 6) ==== 91049+0+0 (secure 0 0 0) 0x7feb68005d10 con 0x7feb701e71e0 2024-04-16T16:14:58.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.579+0000 7feb67400640 1 --2- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7feb5006c940 0x7feb5006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:58.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.580+0000 7feb67400640 1 -- 172.21.15.45:0/2100620104 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7feb6808be00 con 0x7feb701e71e0 2024-04-16T16:14:58.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.580+0000 7feb6ea00640 1 -- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7feb5006c940 msgr2=0x7feb5006ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:14:58.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.580+0000 7feb6ea00640 1 --2- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7feb5006c940 0x7feb5006ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:14:58.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.584+0000 7feb67400640 1 -- 172.21.15.45:0/2100620104 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feb68024020 con 0x7feb701e71e0 2024-04-16T16:14:58.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.781+0000 7feb6ea00640 1 -- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7feb5006c940 msgr2=0x7feb5006ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:14:58.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.781+0000 7feb6ea00640 1 --2- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7feb5006c940 0x7feb5006ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:14:58.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.784+0000 7feb6fe00640 1 -- 172.21.15.45:0/2100620104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7feb700b6320 con 0x7feb701e71e0 2024-04-16T16:14:58.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.785+0000 7feb67400640 1 -- 172.21.15.45:0/2100620104 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) ==== 100+0+11 (secure 0 0 0) 0x7feb6805a190 con 0x7feb701e71e0 2024-04-16T16:14:58.784 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:14:58.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.787+0000 7feb6fe00640 1 -- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7feb5006c940 msgr2=0x7feb5006ee00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:14:58.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.787+0000 7feb6fe00640 1 --2- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7feb5006c940 0x7feb5006ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.787+0000 7feb6fe00640 1 -- 172.21.15.45:0/2100620104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb701e71e0 msgr2=0x7feb700c94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:58.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.787+0000 7feb6fe00640 1 --2- 172.21.15.45:0/2100620104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb701e71e0 0x7feb700c94e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7feb7006dbe0 tx=0x7feb6800b480 comp rx=0 tx=0).stop 2024-04-16T16:14:58.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.787+0000 7feb6fe00640 1 -- 172.21.15.45:0/2100620104 shutdown_connections 2024-04-16T16:14:58.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.787+0000 7feb6fe00640 1 --2- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7feb5006c940 0x7feb5006ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.787+0000 7feb6fe00640 1 --2- 172.21.15.45:0/2100620104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb701e71e0 0x7feb700c94e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.787+0000 7feb6fe00640 1 --2- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb701f4790 0x7feb700cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.787+0000 7feb6fe00640 1 --2- 172.21.15.45:0/2100620104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700c9100 0x7feb7014e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:58.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.787+0000 7feb6fe00640 1 -- 172.21.15.45:0/2100620104 >> 172.21.15.45:0/2100620104 conn(0x7feb7006d2f0 msgr2=0x7feb700b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:58.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.787+0000 7feb6fe00640 1 -- 172.21.15.45:0/2100620104 shutdown_connections 2024-04-16T16:14:58.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:58.787+0000 7feb6fe00640 1 -- 172.21.15.45:0/2100620104 wait complete. 2024-04-16T16:14:58.803 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:14:58.837 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:58.838+0000 7f86c0a12040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:14:58.947 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:58.948+0000 7fdf7d7db040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:14:58.997 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:58.998+0000 7f86c0a12040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:14:59.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.015+0000 7f0e81400640 1 Processor -- start 2024-04-16T16:14:59.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.016+0000 7f0e81400640 1 -- start start 2024-04-16T16:14:59.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.017+0000 7f0e81400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0e7c15ca00 0x7f0e7c0cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:59.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.017+0000 7f0e81400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e7c0cde10 0x7f0e7c0b4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:59.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.017+0000 7f0e81400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e7c06fab0 con 0x7f0e7c0afe20 2024-04-16T16:14:59.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.017+0000 7f0e81400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0e7c0ce240 con 0x7f0e7c15ca00 2024-04-16T16:14:59.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.017+0000 7f0e81400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0e7c15bf20 con 0x7f0e7c0cde10 2024-04-16T16:14:59.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.018+0000 7f0e7b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e7c0cde10 0x7f0e7c0b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:59.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.018+0000 7f0e7b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e7c0cde10 0x7f0e7c0b4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:59118/0 (socket says 172.21.15.45:59118) 2024-04-16T16:14:59.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.018+0000 7f0e7b200640 1 -- 172.21.15.45:0/4052057761 learned_addr learned my addr 172.21.15.45:0/4052057761 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:59.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.018+0000 7f0e73e00640 1 --2- 172.21.15.45:0/4052057761 >> v2:172.21.15.103:3300/0 conn(0x7f0e7c15ca00 0x7f0e7c0cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.018+0000 7f0e7b200640 1 -- 172.21.15.45:0/4052057761 >> v2:172.21.15.103:3300/0 conn(0x7f0e7c15ca00 msgr2=0x7f0e7c0cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.018+0000 7f0e7b200640 1 --2- 172.21.15.45:0/4052057761 >> v2:172.21.15.103:3300/0 conn(0x7f0e7c15ca00 0x7f0e7c0cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.018+0000 7f0e7b200640 1 -- 172.21.15.45:0/4052057761 >> v1:172.21.15.103:6789/0 conn(0x7f0e7c0afe20 legacy=0x7f0e7c06f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.019+0000 7f0e7b200640 1 -- 172.21.15.45:0/4052057761 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e7c15c200 con 0x7f0e7c0cde10 2024-04-16T16:14:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.020+0000 7f0e7b200640 1 --2- 172.21.15.45:0/4052057761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e7c0cde10 0x7f0e7c0b4160 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e74009e50 tx=0x7f0e7402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47078711624c0d5e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.020+0000 7f0e79e00640 1 -- 172.21.15.45:0/4052057761 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7402cb00 con 0x7f0e7c0cde10 2024-04-16T16:14:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.020+0000 7f0e79e00640 1 -- 172.21.15.45:0/4052057761 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e7402cca0 con 0x7f0e7c0cde10 2024-04-16T16:14:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.021+0000 7f0e79e00640 1 -- 172.21.15.45:0/4052057761 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e74034610 con 0x7f0e7c0cde10 2024-04-16T16:14:59.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.021+0000 7f0e81400640 1 -- 172.21.15.45:0/4052057761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e7c0cde10 msgr2=0x7f0e7c0b4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:59.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.021+0000 7f0e81400640 1 --2- 172.21.15.45:0/4052057761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e7c0cde10 0x7f0e7c0b4160 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e74009e50 tx=0x7f0e7402bf60 comp rx=0 tx=0).stop 2024-04-16T16:14:59.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.021+0000 7f0e81400640 1 -- 172.21.15.45:0/4052057761 shutdown_connections 2024-04-16T16:14:59.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.021+0000 7f0e81400640 1 --2- 172.21.15.45:0/4052057761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e7c0cde10 0x7f0e7c0b4160 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.021+0000 7f0e81400640 1 --2- 172.21.15.45:0/4052057761 >> v2:172.21.15.103:3300/0 conn(0x7f0e7c15ca00 0x7f0e7c0cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.021+0000 7f0e81400640 1 -- 172.21.15.45:0/4052057761 >> 172.21.15.45:0/4052057761 conn(0x7f0e7c06b930 msgr2=0x7f0e7c06bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:59.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.021+0000 7f0e81400640 1 -- 172.21.15.45:0/4052057761 shutdown_connections 2024-04-16T16:14:59.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.021+0000 7f0e81400640 1 -- 172.21.15.45:0/4052057761 wait complete. 2024-04-16T16:14:59.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.023+0000 7f0e81400640 1 Processor -- start 2024-04-16T16:14:59.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.023+0000 7f0e81400640 1 -- start start 2024-04-16T16:14:59.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.024+0000 7f0e81400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e7c0afe20 0x7f0e7c1ea8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:59.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.024+0000 7f0e81400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e7c0cde10 0x7f0e7c1f4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:59.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.024+0000 7f0e81400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e7c15ca00 0x7f0e7c157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:59.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.024+0000 7f0e81400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7c0b0200 con 0x7f0e7c0afe20 2024-04-16T16:14:59.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.024+0000 7f0e81400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7c06f720 con 0x7f0e7c0cde10 2024-04-16T16:14:59.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.024+0000 7f0e81400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0e7c0b0380 con 0x7f0e7c15ca00 2024-04-16T16:14:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.024+0000 7f0e7a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e7c0afe20 0x7f0e7c1ea8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.024+0000 7f0e7a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e7c0afe20 0x7f0e7c1ea8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41842/0 (socket says 172.21.15.45:41842) 2024-04-16T16:14:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.024+0000 7f0e7a800640 1 -- 172.21.15.45:0/3836423656 learned_addr learned my addr 172.21.15.45:0/3836423656 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.025+0000 7f0e73e00640 1 --2- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e7c0cde10 0x7f0e7c1f4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.025+0000 7f0e7b200640 1 --2- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e7c15ca00 0x7f0e7c157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.025+0000 7f0e7a800640 1 -- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e7c15ca00 msgr2=0x7f0e7c157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.025+0000 7f0e7a800640 1 --2- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e7c15ca00 0x7f0e7c157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.025+0000 7f0e7a800640 1 -- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e7c0cde10 msgr2=0x7f0e7c1f4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.025+0000 7f0e7a800640 1 --2- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e7c0cde10 0x7f0e7c1f4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.025+0000 7f0e7a800640 1 -- 172.21.15.45:0/3836423656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e7c1eade0 con 0x7f0e7c0afe20 2024-04-16T16:14:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.025+0000 7f0e7a800640 1 --2- 172.21.15.45:0/3836423656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e7c0afe20 0x7f0e7c1ea8a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c00c340 tx=0x7f0e6c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.025+0000 7f0e72a00640 1 -- 172.21.15.45:0/3836423656 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c017070 con 0x7f0e7c0afe20 2024-04-16T16:14:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.025+0000 7f0e72a00640 1 -- 172.21.15.45:0/3836423656 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e6c0052d0 con 0x7f0e7c0afe20 2024-04-16T16:14:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.025+0000 7f0e72a00640 1 -- 172.21.15.45:0/3836423656 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c003850 con 0x7f0e7c0afe20 2024-04-16T16:14:59.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.025+0000 7f0e81400640 1 -- 172.21.15.45:0/3836423656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e7c157c30 con 0x7f0e7c0afe20 2024-04-16T16:14:59.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.026+0000 7f0e81400640 1 -- 172.21.15.45:0/3836423656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e7c1584d0 con 0x7f0e7c0afe20 2024-04-16T16:14:59.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.027+0000 7f0e81400640 1 -- 172.21.15.45:0/3836423656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e7c075780 con 0x7f0e7c0afe20 2024-04-16T16:14:59.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.028+0000 7f0e72a00640 1 -- 172.21.15.45:0/3836423656 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 6) ==== 91049+0+0 (secure 0 0 0) 0x7f0e6c01a070 con 0x7f0e7c0afe20 2024-04-16T16:14:59.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.029+0000 7f0e72a00640 1 --2- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f0e5806c940 0x7f0e5806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:59.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.029+0000 7f0e72a00640 1 -- 172.21.15.45:0/3836423656 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f0e6c08afc0 con 0x7f0e7c0afe20 2024-04-16T16:14:59.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.029+0000 7f0e73e00640 1 -- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f0e5806c940 msgr2=0x7f0e5806ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:14:59.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.029+0000 7f0e73e00640 1 --2- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f0e5806c940 0x7f0e5806ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:14:59.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.033+0000 7f0e72a00640 1 -- 172.21.15.45:0/3836423656 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0e7c1eade0 con 0x7f0e7c0afe20 2024-04-16T16:14:59.110 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:59.110+0000 7f86c0a12040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:14:59.119 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:59.121+0000 7fdf7d7db040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:14:59.160 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:59.161+0000 7f86c0a12040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:14:59.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.230+0000 7f0e73e00640 1 -- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f0e5806c940 msgr2=0x7f0e5806ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:14:59.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.230+0000 7f0e73e00640 1 --2- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f0e5806c940 0x7f0e5806ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:14:59.234 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:59.235+0000 7fdf7d7db040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:14:59.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.249+0000 7f0e81400640 1 -- 172.21.15.45:0/3836423656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0e7c06eab0 con 0x7f0e7c0afe20 2024-04-16T16:14:59.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.250+0000 7f0e72a00640 1 -- 172.21.15.45:0/3836423656 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) ==== 80+0+0 (secure 0 0 0) 0x7f0e6c059350 con 0x7f0e7c0afe20 2024-04-16T16:14:59.249 INFO:teuthology.orchestra.run.smithi045.stderr:listed 0 entries 2024-04-16T16:14:59.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.252+0000 7f0e81400640 1 -- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f0e5806c940 msgr2=0x7f0e5806ee00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:14:59.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.252+0000 7f0e81400640 1 --2- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f0e5806c940 0x7f0e5806ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.252+0000 7f0e81400640 1 -- 172.21.15.45:0/3836423656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e7c0afe20 msgr2=0x7f0e7c1ea8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:59.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.252+0000 7f0e81400640 1 --2- 172.21.15.45:0/3836423656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e7c0afe20 0x7f0e7c1ea8a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c00c340 tx=0x7f0e6c00c800 comp rx=0 tx=0).stop 2024-04-16T16:14:59.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.252+0000 7f0e81400640 1 -- 172.21.15.45:0/3836423656 shutdown_connections 2024-04-16T16:14:59.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.252+0000 7f0e81400640 1 --2- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f0e5806c940 0x7f0e5806ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.252+0000 7f0e81400640 1 --2- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e7c15ca00 0x7f0e7c157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.253+0000 7f0e81400640 1 --2- 172.21.15.45:0/3836423656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e7c0cde10 0x7f0e7c1f4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.253+0000 7f0e81400640 1 --2- 172.21.15.45:0/3836423656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e7c0afe20 0x7f0e7c1ea8a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.253+0000 7f0e81400640 1 -- 172.21.15.45:0/3836423656 >> 172.21.15.45:0/3836423656 conn(0x7f0e7c06b930 msgr2=0x7f0e7c15adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:59.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.253+0000 7f0e81400640 1 -- 172.21.15.45:0/3836423656 shutdown_connections 2024-04-16T16:14:59.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.253+0000 7f0e81400640 1 -- 172.21.15.45:0/3836423656 wait complete. 2024-04-16T16:14:59.271 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-16T16:14:59.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:59.272 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-16T16:14:59.285 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:59.286+0000 7f86c0a12040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:14:59.289 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:59.291+0000 7fdf7d7db040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:14:59.367 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:14:59.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:14:59.367 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-16T16:14:59.392 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:59.393+0000 7f86c0a12040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:14:59.428 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:59.429+0000 7fdf7d7db040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:14:59.460 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:14:59.460 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:59.513 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:14:59.513 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:14:59.541 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:59.542+0000 7fdf7d7db040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:14:59.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:14:59.579 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:59.579+0000 7f86c0a12040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:14:59.629 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:59.630+0000 7f86c0a12040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:14:59.729 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:59.729+0000 7fdf7d7db040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:14:59.766 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:14:59.766+0000 7f86c0a12040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:14:59.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.770+0000 7f68bcc00640 1 Processor -- start 2024-04-16T16:14:59.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.770+0000 7f68bcc00640 1 -- start start 2024-04-16T16:14:59.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.772+0000 7f68bcc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b815ecc0 0x7f68b806fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:59.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.772+0000 7f68bcc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f68b809af60 0x7f68b806b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:59.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.772+0000 7f68bcc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68b806f490 con 0x7f68b806be70 2024-04-16T16:14:59.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.772+0000 7f68bcc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f68b80b3f60 con 0x7f68b809af60 2024-04-16T16:14:59.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.772+0000 7f68bcc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f68b806f730 con 0x7f68b815ecc0 2024-04-16T16:14:59.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.773+0000 7f68b5e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b815ecc0 0x7f68b806fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:59.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.773+0000 7f68ad400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f68b809af60 0x7f68b806b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:59.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.773+0000 7f68b5e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b815ecc0 0x7f68b806fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:59134/0 (socket says 172.21.15.45:59134) 2024-04-16T16:14:59.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.773+0000 7f68ad400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f68b809af60 0x7f68b806b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:43506/0 (socket says 172.21.15.45:43506) 2024-04-16T16:14:59.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.773+0000 7f68b5e00640 1 -- 172.21.15.45:0/2786023530 learned_addr learned my addr 172.21.15.45:0/2786023530 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:59.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.774+0000 7f68ad400640 1 -- 172.21.15.45:0/2786023530 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b815ecc0 msgr2=0x7f68b806fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:14:59.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.774+0000 7f68ad400640 1 --2- 172.21.15.45:0/2786023530 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b815ecc0 0x7f68b806fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.774+0000 7f68ad400640 1 -- 172.21.15.45:0/2786023530 >> v1:172.21.15.103:6789/0 conn(0x7f68b806be70 legacy=0x7f68b80ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:59.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.774+0000 7f68ad400640 1 -- 172.21.15.45:0/2786023530 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68b80b3bf0 con 0x7f68b809af60 2024-04-16T16:14:59.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.774+0000 7f68b5e00640 1 --2- 172.21.15.45:0/2786023530 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b815ecc0 0x7f68b806fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:14:59.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.775+0000 7f68ad400640 1 --2- 172.21.15.45:0/2786023530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68b809af60 0x7f68b806b930 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f68a0009d90 tx=0x7f68a002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88f60188f9e7e4a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:59.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.776+0000 7f68b5400640 1 -- 172.21.15.45:0/2786023530 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0039070 con 0x7f68b809af60 2024-04-16T16:14:59.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.776+0000 7f68b5400640 1 -- 172.21.15.45:0/2786023530 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68a002cc50 con 0x7f68b809af60 2024-04-16T16:14:59.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.776+0000 7f68b5400640 1 -- 172.21.15.45:0/2786023530 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0034440 con 0x7f68b809af60 2024-04-16T16:14:59.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.776+0000 7f68bcc00640 1 -- 172.21.15.45:0/2786023530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68b809af60 msgr2=0x7f68b806b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:59.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.776+0000 7f68bcc00640 1 --2- 172.21.15.45:0/2786023530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68b809af60 0x7f68b806b930 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f68a0009d90 tx=0x7f68a002bea0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.776+0000 7f68bcc00640 1 -- 172.21.15.45:0/2786023530 shutdown_connections 2024-04-16T16:14:59.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.776+0000 7f68bcc00640 1 --2- 172.21.15.45:0/2786023530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68b809af60 0x7f68b806b930 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.776+0000 7f68bcc00640 1 --2- 172.21.15.45:0/2786023530 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b815ecc0 0x7f68b806fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.776+0000 7f68bcc00640 1 -- 172.21.15.45:0/2786023530 >> 172.21.15.45:0/2786023530 conn(0x7f68b809aa70 msgr2=0x7f68b80b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:14:59.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.777+0000 7f68bcc00640 1 -- 172.21.15.45:0/2786023530 shutdown_connections 2024-04-16T16:14:59.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.777+0000 7f68bcc00640 1 -- 172.21.15.45:0/2786023530 wait complete. 2024-04-16T16:14:59.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.778+0000 7f68bcc00640 1 Processor -- start 2024-04-16T16:14:59.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.778+0000 7f68bcc00640 1 -- start start 2024-04-16T16:14:59.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.779+0000 7f68bcc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b806be70 0x7f68b8153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:59.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.779+0000 7f68bcc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68b809af60 0x7f68b815cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:59.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.779+0000 7f68bcc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b815ecc0 0x7f68b8141e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:59.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.779+0000 7f68bcc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f68b80703a0 con 0x7f68b815ecc0 2024-04-16T16:14:59.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.779+0000 7f68bcc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f68b8070220 con 0x7f68b809af60 2024-04-16T16:14:59.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.779+0000 7f68bcc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f68b80d0590 con 0x7f68b806be70 2024-04-16T16:14:59.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.780+0000 7f68b6800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b815ecc0 0x7f68b8141e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:59.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.780+0000 7f68b6800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b815ecc0 0x7f68b8141e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41846/0 (socket says 172.21.15.45:41846) 2024-04-16T16:14:59.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.780+0000 7f68b6800640 1 -- 172.21.15.45:0/3114107510 learned_addr learned my addr 172.21.15.45:0/3114107510 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:14:59.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.780+0000 7f68b5e00640 1 --2- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b806be70 0x7f68b8153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:59.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.780+0000 7f68ad400640 1 --2- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68b809af60 0x7f68b815cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:14:59.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.780+0000 7f68b6800640 1 -- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b806be70 msgr2=0x7f68b8153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:59.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.780+0000 7f68b6800640 1 --2- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b806be70 0x7f68b8153760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.780+0000 7f68b6800640 1 -- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68b809af60 msgr2=0x7f68b815cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:14:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.780+0000 7f68b6800640 1 --2- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68b809af60 0x7f68b815cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:14:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.780+0000 7f68b6800640 1 -- 172.21.15.45:0/3114107510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68b8153e00 con 0x7f68b815ecc0 2024-04-16T16:14:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.781+0000 7f68b6800640 1 --2- 172.21.15.45:0/3114107510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b815ecc0 0x7f68b8141e30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f68b000c3f0 tx=0x7f68b000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:14:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.781+0000 7f68aea00640 1 -- 172.21.15.45:0/3114107510 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b0017070 con 0x7f68b815ecc0 2024-04-16T16:14:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.781+0000 7f68aea00640 1 -- 172.21.15.45:0/3114107510 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68b00052d0 con 0x7f68b815ecc0 2024-04-16T16:14:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.781+0000 7f68aea00640 1 -- 172.21.15.45:0/3114107510 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b0003850 con 0x7f68b815ecc0 2024-04-16T16:14:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.781+0000 7f68bcc00640 1 -- 172.21.15.45:0/3114107510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68b81540e0 con 0x7f68b815ecc0 2024-04-16T16:14:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.781+0000 7f68bcc00640 1 -- 172.21.15.45:0/3114107510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68b81546a0 con 0x7f68b815ecc0 2024-04-16T16:14:59.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.782+0000 7f68bcc00640 1 -- 172.21.15.45:0/3114107510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68b8075780 con 0x7f68b815ecc0 2024-04-16T16:14:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.784+0000 7f68aea00640 1 -- 172.21.15.45:0/3114107510 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 6) ==== 91049+0+0 (secure 0 0 0) 0x7f68b001a070 con 0x7f68b815ecc0 2024-04-16T16:14:59.784 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:59.785+0000 7fdf7d7db040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:14:59.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.785+0000 7f68aea00640 1 --2- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f689406c940 0x7f689406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:14:59.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.785+0000 7f68aea00640 1 -- 172.21.15.45:0/3114107510 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f68b008aff0 con 0x7f68b815ecc0 2024-04-16T16:14:59.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.785+0000 7f68b5e00640 1 -- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f689406c940 msgr2=0x7f689406ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:14:59.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.785+0000 7f68b5e00640 1 --2- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f689406c940 0x7f689406ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:14:59.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.789+0000 7f68aea00640 1 -- 172.21.15.45:0/3114107510 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f68b0058ca0 con 0x7f68b815ecc0 2024-04-16T16:14:59.931 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:14:59.931+0000 7fdf7d7db040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:14:59.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.986+0000 7f68b5e00640 1 -- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f689406c940 msgr2=0x7f689406ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:14:59.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.986+0000 7f68b5e00640 1 --2- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f689406c940 0x7f689406ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:14:59.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:14:59.999+0000 7f68bcc00640 1 -- 172.21.15.45:0/3114107510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f68b815d640 con 0x7f68b815ecc0 2024-04-16T16:14:59.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.000+0000 7f68aea00640 1 -- 172.21.15.45:0/3114107510 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16669 (secure 0 0 0) 0x7f68b005cfa0 con 0x7f68b815ecc0 2024-04-16T16:15:00.000 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:00.000 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":34,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:14:40.675999+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:14:37.171798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-16T16:14:38.122496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:15:00.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.003+0000 7f68bcc00640 1 -- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f689406c940 msgr2=0x7f689406ee00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:15:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.003+0000 7f68bcc00640 1 --2- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f689406c940 0x7f689406ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.003+0000 7f68bcc00640 1 -- 172.21.15.45:0/3114107510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b815ecc0 msgr2=0x7f68b8141e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.003+0000 7f68bcc00640 1 --2- 172.21.15.45:0/3114107510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b815ecc0 0x7f68b8141e30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f68b000c3f0 tx=0x7f68b000c8b0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.003+0000 7f68bcc00640 1 -- 172.21.15.45:0/3114107510 shutdown_connections 2024-04-16T16:15:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.003+0000 7f68bcc00640 1 --2- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f689406c940 0x7f689406ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.003+0000 7f68bcc00640 1 --2- 172.21.15.45:0/3114107510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b815ecc0 0x7f68b8141e30 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.003+0000 7f68bcc00640 1 --2- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68b809af60 0x7f68b815cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.003+0000 7f68bcc00640 1 --2- 172.21.15.45:0/3114107510 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b806be70 0x7f68b8153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.003+0000 7f68bcc00640 1 -- 172.21.15.45:0/3114107510 >> 172.21.15.45:0/3114107510 conn(0x7f68b809aa70 msgr2=0x7f68b815db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.004+0000 7f68bcc00640 1 -- 172.21.15.45:0/3114107510 shutdown_connections 2024-04-16T16:15:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.004+0000 7f68bcc00640 1 -- 172.21.15.45:0/3114107510 wait complete. 2024-04-16T16:15:00.021 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:14:37.171798+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:14:38.122496+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:15:00.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:15:00.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.223+0000 7f7806c00640 1 Processor -- start 2024-04-16T16:15:00.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.223+0000 7f7806c00640 1 -- start start 2024-04-16T16:15:00.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.225+0000 7f7806c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f780015ea70 0x7f78000d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:00.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.225+0000 7f7806c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78000cfc40 0x7f7800162560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:00.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.225+0000 7f7806c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f780015ee50 con 0x7f78000d4370 2024-04-16T16:15:00.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.225+0000 7f7806c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f78000d1a10 con 0x7f780015ea70 2024-04-16T16:15:00.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.225+0000 7f7806c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f78000b6480 con 0x7f78000cfc40 2024-04-16T16:15:00.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.226+0000 7f7805800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f780015ea70 0x7f78000d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:00.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.226+0000 7f7804e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f78000d4370 0x7f78000cf530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57160/0 (socket says 172.21.15.45:57160) 2024-04-16T16:15:00.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.226+0000 7f7804e00640 1 -- 172.21.15.45:0/1283331518 learned_addr learned my addr 172.21.15.45:0/1283331518 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:00.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.226+0000 7f7806200640 1 --2- 172.21.15.45:0/1283331518 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78000cfc40 0x7f7800162560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:00.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.226+0000 7f7805800640 1 -- 172.21.15.45:0/1283331518 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78000cfc40 msgr2=0x7f7800162560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:00.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.226+0000 7f7805800640 1 --2- 172.21.15.45:0/1283331518 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78000cfc40 0x7f7800162560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.226+0000 7f7805800640 1 -- 172.21.15.45:0/1283331518 >> v1:172.21.15.45:6789/0 conn(0x7f78000d4370 legacy=0x7f78000cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:00.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.226+0000 7f77ffe00640 1 -- 172.21.15.45:0/1283331518 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 736583331 0 0) 0x7f780015ee50 con 0x7f78000d4370 2024-04-16T16:15:00.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.226+0000 7f7805800640 1 -- 172.21.15.45:0/1283331518 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78000b6760 con 0x7f780015ea70 2024-04-16T16:15:00.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.226+0000 7f7806200640 1 --2- 172.21.15.45:0/1283331518 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78000cfc40 0x7f7800162560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:15:00.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.227+0000 7f7805800640 1 --2- 172.21.15.45:0/1283331518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f780015ea70 0x7f78000d11a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f77f40097a0 tx=0x7f77f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a469a6119a348c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:00.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.228+0000 7f77ffe00640 1 -- 172.21.15.45:0/1283331518 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f4035020 con 0x7f780015ea70 2024-04-16T16:15:00.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.228+0000 7f77ffe00640 1 -- 172.21.15.45:0/1283331518 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f4031830 con 0x7f780015ea70 2024-04-16T16:15:00.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.228+0000 7f77ffe00640 1 -- 172.21.15.45:0/1283331518 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f4031c00 con 0x7f780015ea70 2024-04-16T16:15:00.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.228+0000 7f7806c00640 1 -- 172.21.15.45:0/1283331518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f780015ea70 msgr2=0x7f78000d11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:00.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.228+0000 7f7806c00640 1 --2- 172.21.15.45:0/1283331518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f780015ea70 0x7f78000d11a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f77f40097a0 tx=0x7f77f402b8d0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.229+0000 7f7806c00640 1 -- 172.21.15.45:0/1283331518 shutdown_connections 2024-04-16T16:15:00.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.229+0000 7f7806c00640 1 --2- 172.21.15.45:0/1283331518 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78000cfc40 0x7f7800162560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.229+0000 7f7806c00640 1 --2- 172.21.15.45:0/1283331518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f780015ea70 0x7f78000d11a0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.229+0000 7f7806c00640 1 -- 172.21.15.45:0/1283331518 >> 172.21.15.45:0/1283331518 conn(0x7f780006d3b0 msgr2=0x7f7800073060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:00.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.229+0000 7f7806c00640 1 -- 172.21.15.45:0/1283331518 shutdown_connections 2024-04-16T16:15:00.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.229+0000 7f7806c00640 1 -- 172.21.15.45:0/1283331518 wait complete. 2024-04-16T16:15:00.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.230+0000 7f7806c00640 1 Processor -- start 2024-04-16T16:15:00.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.230+0000 7f7806c00640 1 -- start start 2024-04-16T16:15:00.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.231+0000 7f7806c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78000cfc40 0x7f78000cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:00.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.231+0000 7f7806c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78000d4370 0x7f78000c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:00.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.231+0000 7f7806c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f780015ea70 0x7f780015a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:00.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.231+0000 7f7806c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f78000b6f60 con 0x7f78000d4370 2024-04-16T16:15:00.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.231+0000 7f7806c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f78000d16e0 con 0x7f78000cfc40 2024-04-16T16:15:00.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.231+0000 7f7806c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f78000c92a0 con 0x7f780015ea70 2024-04-16T16:15:00.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.232+0000 7f7804e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78000d4370 0x7f78000c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:00.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.232+0000 7f7804e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78000d4370 0x7f78000c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41868/0 (socket says 172.21.15.45:41868) 2024-04-16T16:15:00.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.232+0000 7f7804e00640 1 -- 172.21.15.45:0/3582831453 learned_addr learned my addr 172.21.15.45:0/3582831453 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:00.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.232+0000 7f7806200640 1 --2- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f780015ea70 0x7f780015a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:00.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.232+0000 7f7805800640 1 --2- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78000cfc40 0x7f78000cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:00.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.232+0000 7f7804e00640 1 -- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f780015ea70 msgr2=0x7f780015a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:00.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.232+0000 7f7804e00640 1 --2- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f780015ea70 0x7f780015a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.232+0000 7f7804e00640 1 -- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78000cfc40 msgr2=0x7f78000cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:00.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.232+0000 7f7804e00640 1 --2- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78000cfc40 0x7f78000cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.232+0000 7f7804e00640 1 -- 172.21.15.45:0/3582831453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78000cb320 con 0x7f78000d4370 2024-04-16T16:15:00.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.232+0000 7f7804e00640 1 --2- 172.21.15.45:0/3582831453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78000d4370 0x7f78000c9fb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f77ec00c430 tx=0x7f77ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:00.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.233+0000 7f77fd600640 1 -- 172.21.15.45:0/3582831453 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0074e0 con 0x7f78000d4370 2024-04-16T16:15:00.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.233+0000 7f77fd600640 1 -- 172.21.15.45:0/3582831453 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77ec0047e0 con 0x7f78000d4370 2024-04-16T16:15:00.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.233+0000 7f7806c00640 1 -- 172.21.15.45:0/3582831453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78000c94d0 con 0x7f78000d4370 2024-04-16T16:15:00.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.233+0000 7f77fd600640 1 -- 172.21.15.45:0/3582831453 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec004bd0 con 0x7f78000d4370 2024-04-16T16:15:00.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.233+0000 7f7806c00640 1 -- 172.21.15.45:0/3582831453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78000cdbd0 con 0x7f78000d4370 2024-04-16T16:15:00.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.234+0000 7f7806c00640 1 -- 172.21.15.45:0/3582831453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78000737c0 con 0x7f78000d4370 2024-04-16T16:15:00.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.235+0000 7f77fd600640 1 -- 172.21.15.45:0/3582831453 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 6) ==== 91049+0+0 (secure 0 0 0) 0x7f77ec011070 con 0x7f78000d4370 2024-04-16T16:15:00.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.236+0000 7f77fd600640 1 --2- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f77e006c970 0x7f77e006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:00.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.236+0000 7f77fd600640 1 -- 172.21.15.45:0/3582831453 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f77ec08b5f0 con 0x7f78000d4370 2024-04-16T16:15:00.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.236+0000 7f7805800640 1 -- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f77e006c970 msgr2=0x7f77e006ee30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:15:00.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.236+0000 7f7805800640 1 --2- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f77e006c970 0x7f77e006ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:15:00.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.240+0000 7f77fd600640 1 -- 172.21.15.45:0/3582831453 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78000737c0 con 0x7f78000d4370 2024-04-16T16:15:00.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.437+0000 7f7805800640 1 -- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f77e006c970 msgr2=0x7f77e006ee30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:15:00.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.437+0000 7f7805800640 1 --2- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f77e006c970 0x7f77e006ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:15:00.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.440+0000 7f7806c00640 1 -- 172.21.15.45:0/3582831453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7800075780 con 0x7f78000d4370 2024-04-16T16:15:00.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.441+0000 7f77fd600640 1 -- 172.21.15.45:0/3582831453 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) ==== 93+0+10 (secure 0 0 0) 0x7f77ec059460 con 0x7f78000d4370 2024-04-16T16:15:00.440 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:15:00.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.444+0000 7f7806c00640 1 -- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f77e006c970 msgr2=0x7f77e006ee30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:15:00.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.444+0000 7f7806c00640 1 --2- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f77e006c970 0x7f77e006ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.444+0000 7f7806c00640 1 -- 172.21.15.45:0/3582831453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78000d4370 msgr2=0x7f78000c9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:00.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.444+0000 7f7806c00640 1 --2- 172.21.15.45:0/3582831453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78000d4370 0x7f78000c9fb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f77ec00c430 tx=0x7f77ec00c8f0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.444+0000 7f7806c00640 1 -- 172.21.15.45:0/3582831453 shutdown_connections 2024-04-16T16:15:00.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.444+0000 7f7806c00640 1 --2- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f77e006c970 0x7f77e006ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.444+0000 7f7806c00640 1 --2- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f780015ea70 0x7f780015a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.444+0000 7f7806c00640 1 --2- 172.21.15.45:0/3582831453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78000d4370 0x7f78000c9fb0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.444+0000 7f7806c00640 1 --2- 172.21.15.45:0/3582831453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78000cfc40 0x7f78000cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.444+0000 7f7806c00640 1 -- 172.21.15.45:0/3582831453 >> 172.21.15.45:0/3582831453 conn(0x7f780006d3b0 msgr2=0x7f7800159940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:00.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.444+0000 7f7806c00640 1 -- 172.21.15.45:0/3582831453 shutdown_connections 2024-04-16T16:15:00.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.444+0000 7f7806c00640 1 -- 172.21.15.45:0/3582831453 wait complete. 2024-04-16T16:15:00.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:15:00.584 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:15:00.584+0000 7f86c0a12040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:15:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.659+0000 7f53d6c00640 1 Processor -- start 2024-04-16T16:15:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.659+0000 7f53d6c00640 1 -- start start 2024-04-16T16:15:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.661+0000 7f53d6c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f53d00d18f0 0x7f53d00ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.661+0000 7f53d6c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f53d00d1cd0 0x7f53d0162dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.661+0000 7f53d6c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d00b6810 con 0x7f53d00b6310 2024-04-16T16:15:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.661+0000 7f53d6c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f53d00d29f0 con 0x7f53d00d1cd0 2024-04-16T16:15:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.661+0000 7f53d6c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f53d00b6af0 con 0x7f53d00d18f0 2024-04-16T16:15:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.661+0000 7f53d6200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f53d00d1cd0 0x7f53d0162dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.661+0000 7f53d4e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f53d00b6310 0x7f53d00d2e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57168/0 (socket says 172.21.15.45:57168) 2024-04-16T16:15:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.661+0000 7f53d4e00640 1 -- 172.21.15.45:0/3022768851 learned_addr learned my addr 172.21.15.45:0/3022768851 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.662+0000 7f53d5800640 1 --2- 172.21.15.45:0/3022768851 >> v2:172.21.15.103:3300/0 conn(0x7f53d00d18f0 0x7f53d00ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.662+0000 7f53d6200640 1 -- 172.21.15.45:0/3022768851 >> v2:172.21.15.103:3300/0 conn(0x7f53d00d18f0 msgr2=0x7f53d00ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.662+0000 7f53d6200640 1 --2- 172.21.15.45:0/3022768851 >> v2:172.21.15.103:3300/0 conn(0x7f53d00d18f0 0x7f53d00ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.662+0000 7f53d6200640 1 -- 172.21.15.45:0/3022768851 >> v1:172.21.15.45:6789/0 conn(0x7f53d00b6310 legacy=0x7f53d00d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.662+0000 7f53d6200640 1 -- 172.21.15.45:0/3022768851 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d00b6dd0 con 0x7f53d00d1cd0 2024-04-16T16:15:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.662+0000 7f53cfe00640 1 -- 172.21.15.45:0/3022768851 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1378114001 0 0) 0x7f53d00b6810 con 0x7f53d00b6310 2024-04-16T16:15:00.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.663+0000 7f53d6200640 1 --2- 172.21.15.45:0/3022768851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53d00d1cd0 0x7f53d0162dc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f53d0073f70 tx=0x7f53c802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b059ee6cff19826 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:00.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.663+0000 7f53cfe00640 1 -- 172.21.15.45:0/3022768851 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c8038070 con 0x7f53d00d1cd0 2024-04-16T16:15:00.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.663+0000 7f53cfe00640 1 -- 172.21.15.45:0/3022768851 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53c8031710 con 0x7f53d00d1cd0 2024-04-16T16:15:00.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.663+0000 7f53cfe00640 1 -- 172.21.15.45:0/3022768851 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c8031ae0 con 0x7f53d00d1cd0 2024-04-16T16:15:00.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.664+0000 7f53d6c00640 1 -- 172.21.15.45:0/3022768851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53d00d1cd0 msgr2=0x7f53d0162dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:00.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.664+0000 7f53d6c00640 1 --2- 172.21.15.45:0/3022768851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53d00d1cd0 0x7f53d0162dc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f53d0073f70 tx=0x7f53c802b7b0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.664+0000 7f53d6c00640 1 -- 172.21.15.45:0/3022768851 shutdown_connections 2024-04-16T16:15:00.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.664+0000 7f53d6c00640 1 --2- 172.21.15.45:0/3022768851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53d00d1cd0 0x7f53d0162dc0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.664+0000 7f53d6c00640 1 --2- 172.21.15.45:0/3022768851 >> v2:172.21.15.103:3300/0 conn(0x7f53d00d18f0 0x7f53d00ba220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.664+0000 7f53d6c00640 1 -- 172.21.15.45:0/3022768851 >> 172.21.15.45:0/3022768851 conn(0x7f53d006cf90 msgr2=0x7f53d006d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:00.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.664+0000 7f53d6c00640 1 -- 172.21.15.45:0/3022768851 shutdown_connections 2024-04-16T16:15:00.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.664+0000 7f53d6c00640 1 -- 172.21.15.45:0/3022768851 wait complete. 2024-04-16T16:15:00.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.666+0000 7f53d6c00640 1 Processor -- start 2024-04-16T16:15:00.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.666+0000 7f53d6c00640 1 -- start start 2024-04-16T16:15:00.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.667+0000 7f53d6c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53d00b6310 0x7f53d014caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:00.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.667+0000 7f53d6c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53d00d18f0 0x7f53d0162380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:00.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.667+0000 7f53d6c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53d00d1cd0 0x7f53d01628c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:00.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.667+0000 7f53d6c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53d0163300 con 0x7f53d00d18f0 2024-04-16T16:15:00.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.667+0000 7f53d6c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f53d00ba760 con 0x7f53d00b6310 2024-04-16T16:15:00.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.667+0000 7f53d6c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f53d00c8380 con 0x7f53d00d1cd0 2024-04-16T16:15:00.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.668+0000 7f53d4e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53d00d18f0 0x7f53d0162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.668+0000 7f53d4e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53d00d18f0 0x7f53d0162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41890/0 (socket says 172.21.15.45:41890) 2024-04-16T16:15:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.668+0000 7f53d4e00640 1 -- 172.21.15.45:0/530182690 learned_addr learned my addr 172.21.15.45:0/530182690 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.668+0000 7f53d5800640 1 --2- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53d00b6310 0x7f53d014caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.668+0000 7f53d6200640 1 --2- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53d00d1cd0 0x7f53d01628c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.668+0000 7f53d4e00640 1 -- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53d00d1cd0 msgr2=0x7f53d01628c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.668+0000 7f53d4e00640 1 --2- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53d00d1cd0 0x7f53d01628c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.668+0000 7f53d4e00640 1 -- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53d00b6310 msgr2=0x7f53d014caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.668+0000 7f53d4e00640 1 --2- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53d00b6310 0x7f53d014caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.668+0000 7f53d4e00640 1 -- 172.21.15.45:0/530182690 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d00c85d0 con 0x7f53d00d18f0 2024-04-16T16:15:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.668+0000 7f53d4e00640 1 --2- 172.21.15.45:0/530182690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53d00d18f0 0x7f53d0162380 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00c4b0 tx=0x7f53bc00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:00.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.669+0000 7f53cd600640 1 -- 172.21.15.45:0/530182690 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53bc018070 con 0x7f53d00d18f0 2024-04-16T16:15:00.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.669+0000 7f53cd600640 1 -- 172.21.15.45:0/530182690 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53bc007500 con 0x7f53d00d18f0 2024-04-16T16:15:00.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.669+0000 7f53cd600640 1 -- 172.21.15.45:0/530182690 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53bc004a20 con 0x7f53d00d18f0 2024-04-16T16:15:00.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.669+0000 7f53d6c00640 1 -- 172.21.15.45:0/530182690 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53d00c88b0 con 0x7f53d00d18f0 2024-04-16T16:15:00.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.669+0000 7f53d6c00640 1 -- 172.21.15.45:0/530182690 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d00c8e70 con 0x7f53d00d18f0 2024-04-16T16:15:00.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.670+0000 7f53d6c00640 1 -- 172.21.15.45:0/530182690 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53d00c9070 con 0x7f53d00d18f0 2024-04-16T16:15:00.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.671+0000 7f53cd600640 1 -- 172.21.15.45:0/530182690 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 6) ==== 91049+0+0 (secure 0 0 0) 0x7f53bc01b070 con 0x7f53d00d18f0 2024-04-16T16:15:00.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.672+0000 7f53cd600640 1 --2- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f53b006c990 0x7f53b006ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:00.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.672+0000 7f53cd600640 1 -- 172.21.15.45:0/530182690 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f53bc08b070 con 0x7f53d00d18f0 2024-04-16T16:15:00.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.672+0000 7f53d5800640 1 -- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f53b006c990 msgr2=0x7f53b006ee50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:15:00.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.672+0000 7f53d5800640 1 --2- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f53b006c990 0x7f53b006ee50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:15:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.676+0000 7f53cd600640 1 -- 172.21.15.45:0/530182690 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f53d00c9070 con 0x7f53d00d18f0 2024-04-16T16:15:00.816 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:15:00.816+0000 7fdf7d7db040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:15:00.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.873+0000 7f53d5800640 1 -- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f53b006c990 msgr2=0x7f53b006ee50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2711977587 2024-04-16T16:15:00.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.873+0000 7f53d5800640 1 --2- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f53b006c990 0x7f53b006ee50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:15:00.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.881+0000 7f53d6c00640 1 -- 172.21.15.45:0/530182690 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f53d0075780 con 0x7f53d00d18f0 2024-04-16T16:15:00.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.881+0000 7f53cd600640 1 -- 172.21.15.45:0/530182690 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) ==== 104+0+11 (secure 0 0 0) 0x7f53bc059540 con 0x7f53d00d18f0 2024-04-16T16:15:00.880 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:15:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.884+0000 7f53d6c00640 1 -- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f53b006c990 msgr2=0x7f53b006ee50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:15:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.884+0000 7f53d6c00640 1 --2- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f53b006c990 0x7f53b006ee50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.884+0000 7f53d6c00640 1 -- 172.21.15.45:0/530182690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53d00d18f0 msgr2=0x7f53d0162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.884+0000 7f53d6c00640 1 --2- 172.21.15.45:0/530182690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53d00d18f0 0x7f53d0162380 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00c4b0 tx=0x7f53bc00c970 comp rx=0 tx=0).stop 2024-04-16T16:15:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.884+0000 7f53d6c00640 1 -- 172.21.15.45:0/530182690 shutdown_connections 2024-04-16T16:15:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.884+0000 7f53d6c00640 1 --2- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:6832/2711977587,v1:172.21.15.103:6833/2711977587] conn(0x7f53b006c990 0x7f53b006ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.884+0000 7f53d6c00640 1 --2- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53d00d1cd0 0x7f53d01628c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.884+0000 7f53d6c00640 1 --2- 172.21.15.45:0/530182690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53d00d18f0 0x7f53d0162380 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.884+0000 7f53d6c00640 1 --2- 172.21.15.45:0/530182690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53d00b6310 0x7f53d014caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.884+0000 7f53d6c00640 1 -- 172.21.15.45:0/530182690 >> 172.21.15.45:0/530182690 conn(0x7f53d006cf90 msgr2=0x7f53d0161b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.885+0000 7f53d6c00640 1 -- 172.21.15.45:0/530182690 shutdown_connections 2024-04-16T16:15:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:00.885+0000 7f53d6c00640 1 -- 172.21.15.45:0/530182690 wait complete. 2024-04-16T16:15:00.889 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:15:00.890+0000 7f86c0a12040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:15:00.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:15:00.947 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:15:00.947+0000 7f86c0a12040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:15:01.077 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:15:01.077+0000 7f86c0a12040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:15:01.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.112+0000 7fc1e2800640 1 Processor -- start 2024-04-16T16:15:01.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.112+0000 7fc1e2800640 1 -- start start 2024-04-16T16:15:01.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.114+0000 7fc1e2800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc1dc0b6c00 0x7fc1dc0d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:01.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.114+0000 7fc1e2800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1dc0b7ca0 con 0x7fc1dc15f060 2024-04-16T16:15:01.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.114+0000 7fc1e2800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1dc0d3450 con 0x7fc1dc15ec80 2024-04-16T16:15:01.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.114+0000 7fc1e2800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc1dc0d3220 con 0x7fc1dc0b6c00 2024-04-16T16:15:01.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.114+0000 7fc1daa00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc1dc15f060 0x7fc1dc166980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57182/0 (socket says 172.21.15.45:57182) 2024-04-16T16:15:01.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.114+0000 7fc1daa00640 1 -- 172.21.15.45:0/3233863369 learned_addr learned my addr 172.21.15.45:0/3233863369 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:01.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.114+0000 7fc1dbe00640 1 --2- 172.21.15.45:0/3233863369 >> v2:172.21.15.103:3300/0 conn(0x7fc1dc0b6c00 0x7fc1dc0d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:01.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.114+0000 7fc1da000640 1 -- 172.21.15.45:0/3233863369 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3385808018 0 0) 0x7fc1dc0b7ca0 con 0x7fc1dc15f060 2024-04-16T16:15:01.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.114+0000 7fc1dbe00640 1 -- 172.21.15.45:0/3233863369 >> v1:172.21.15.103:6789/0 conn(0x7fc1dc15ec80 legacy=0x7fc1dc0b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:01.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.114+0000 7fc1dbe00640 1 -- 172.21.15.45:0/3233863369 >> v1:172.21.15.45:6789/0 conn(0x7fc1dc15f060 legacy=0x7fc1dc166980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:01.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.114+0000 7fc1dbe00640 1 -- 172.21.15.45:0/3233863369 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1dc0d3730 con 0x7fc1dc0b6c00 2024-04-16T16:15:01.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.116+0000 7fc1dbe00640 1 --2- 172.21.15.45:0/3233863369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1dc0b6c00 0x7fc1dc0d2ce0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4009d90 tx=0x7fc1d402bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1757ef743ef42ec6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:01.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.116+0000 7fc1da000640 1 -- 172.21.15.45:0/3233863369 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4039070 con 0x7fc1dc0b6c00 2024-04-16T16:15:01.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.116+0000 7fc1da000640 1 -- 172.21.15.45:0/3233863369 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1d402cd80 con 0x7fc1dc0b6c00 2024-04-16T16:15:01.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.116+0000 7fc1da000640 1 -- 172.21.15.45:0/3233863369 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4034660 con 0x7fc1dc0b6c00 2024-04-16T16:15:01.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.117+0000 7fc1e2800640 1 -- 172.21.15.45:0/3233863369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1dc0b6c00 msgr2=0x7fc1dc0d2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:01.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.117+0000 7fc1e2800640 1 --2- 172.21.15.45:0/3233863369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1dc0b6c00 0x7fc1dc0d2ce0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4009d90 tx=0x7fc1d402bfd0 comp rx=0 tx=0).stop 2024-04-16T16:15:01.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.117+0000 7fc1e2800640 1 -- 172.21.15.45:0/3233863369 shutdown_connections 2024-04-16T16:15:01.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.117+0000 7fc1e2800640 1 --2- 172.21.15.45:0/3233863369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1dc0b6c00 0x7fc1dc0d2ce0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:01.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.117+0000 7fc1e2800640 1 -- 172.21.15.45:0/3233863369 >> 172.21.15.45:0/3233863369 conn(0x7fc1dc0d16d0 msgr2=0x7fc1dc0d1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:01.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.117+0000 7fc1e2800640 1 -- 172.21.15.45:0/3233863369 shutdown_connections 2024-04-16T16:15:01.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.117+0000 7fc1e2800640 1 -- 172.21.15.45:0/3233863369 wait complete. 2024-04-16T16:15:01.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.119+0000 7fc1e2800640 1 Processor -- start 2024-04-16T16:15:01.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.119+0000 7fc1e2800640 1 -- start start 2024-04-16T16:15:01.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.120+0000 7fc1e2800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1dc0b6c00 0x7fc1dc15d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:01.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.120+0000 7fc1e2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1dc15ec80 0x7fc1dc155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:01.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.120+0000 7fc1e2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1dc155ea0 0x7fc1dc15f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.120+0000 7fc1e2800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc0ba190 con 0x7fc1dc155ea0 2024-04-16T16:15:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.120+0000 7fc1e2800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc0d3cf0 con 0x7fc1dc15ec80 2024-04-16T16:15:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.120+0000 7fc1e2800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc1dc0ba310 con 0x7fc1dc0b6c00 2024-04-16T16:15:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.120+0000 7fc1dbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1dc155ea0 0x7fc1dc15f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.120+0000 7fc1dbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1dc155ea0 0x7fc1dc15f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41898/0 (socket says 172.21.15.45:41898) 2024-04-16T16:15:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.120+0000 7fc1dbe00640 1 -- 172.21.15.45:0/1290948330 learned_addr learned my addr 172.21.15.45:0/1290948330 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.120+0000 7fc1daa00640 1 --2- 172.21.15.45:0/1290948330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1dc15ec80 0x7fc1dc155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.121+0000 7fc1db400640 1 --2- 172.21.15.45:0/1290948330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1dc0b6c00 0x7fc1dc15d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.121+0000 7fc1dbe00640 1 -- 172.21.15.45:0/1290948330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1dc0b6c00 msgr2=0x7fc1dc15d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.121+0000 7fc1dbe00640 1 --2- 172.21.15.45:0/1290948330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1dc0b6c00 0x7fc1dc15d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.121+0000 7fc1dbe00640 1 -- 172.21.15.45:0/1290948330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1dc15ec80 msgr2=0x7fc1dc155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.121+0000 7fc1dbe00640 1 --2- 172.21.15.45:0/1290948330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1dc15ec80 0x7fc1dc155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.121+0000 7fc1dbe00640 1 -- 172.21.15.45:0/1290948330 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1dc1563b0 con 0x7fc1dc155ea0 2024-04-16T16:15:01.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.121+0000 7fc1dbe00640 1 --2- 172.21.15.45:0/1290948330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1dc155ea0 0x7fc1dc15f890 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4009d90 tx=0x7fc1d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:01.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.121+0000 7fc1d3400640 1 -- 172.21.15.45:0/1290948330 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4039050 con 0x7fc1dc155ea0 2024-04-16T16:15:01.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.121+0000 7fc1d3400640 1 -- 172.21.15.45:0/1290948330 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1d403a040 con 0x7fc1dc155ea0 2024-04-16T16:15:01.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.121+0000 7fc1d3400640 1 -- 172.21.15.45:0/1290948330 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d403e610 con 0x7fc1dc155ea0 2024-04-16T16:15:01.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.121+0000 7fc1e2800640 1 -- 172.21.15.45:0/1290948330 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1dc156690 con 0x7fc1dc155ea0 2024-04-16T16:15:01.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.122+0000 7fc1e2800640 1 -- 172.21.15.45:0/1290948330 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1dc156c50 con 0x7fc1dc155ea0 2024-04-16T16:15:01.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.123+0000 7fc1e2800640 1 -- 172.21.15.45:0/1290948330 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1dc075780 con 0x7fc1dc155ea0 2024-04-16T16:15:01.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.124+0000 7fc1d3400640 1 -- 172.21.15.45:0/1290948330 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 7) ==== 90760+0+0 (secure 0 0 0) 0x7fc1d4047020 con 0x7fc1dc155ea0 2024-04-16T16:15:01.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.125+0000 7fc1d3400640 1 -- 172.21.15.45:0/1290948330 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5967+0+0 (secure 0 0 0) 0x7fc1d40ae5e0 con 0x7fc1dc155ea0 2024-04-16T16:15:01.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.128+0000 7fc1d3400640 1 -- 172.21.15.45:0/1290948330 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc1d404e020 con 0x7fc1dc155ea0 2024-04-16T16:15:01.165 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:15:01.166+0000 7fdf7d7db040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:15:01.177 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:15:01.178+0000 7f8636800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:15:01.178 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:15:01.178+0000 7f8636800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:15:01.178 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:15:01.178+0000 7f8636800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:15:01.178 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:15:01.178+0000 7f8636800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:15:01.178 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:15:01.178+0000 7f8636800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:15:01.179 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:15:01.179+0000 7f863a400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:15:01.179 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:15:01.179+0000 7f863a400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:15:01.179 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:15:01.179+0000 7f863a400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:15:01.179 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:15:01.179+0000 7f863a400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:15:01.179 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:15:01.179+0000 7f863a400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:15:01.230 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:15:01.231+0000 7fdf7d7db040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:15:01.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.330+0000 7fc1e2800640 1 -- 172.21.15.45:0/1290948330 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc1dc06dd20 con 0x7fc1dc155ea0 2024-04-16T16:15:01.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.331+0000 7fc1d3400640 1 -- 172.21.15.45:0/1290948330 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v35) ==== 100+0+11 (secure 0 0 0) 0x7fc1d407cdb0 con 0x7fc1dc155ea0 2024-04-16T16:15:01.330 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:15:01.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.334+0000 7fc1e2800640 1 -- 172.21.15.45:0/1290948330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1dc155ea0 msgr2=0x7fc1dc15f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:01.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.334+0000 7fc1e2800640 1 --2- 172.21.15.45:0/1290948330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1dc155ea0 0x7fc1dc15f890 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4009d90 tx=0x7fc1d4004a40 comp rx=0 tx=0).stop 2024-04-16T16:15:01.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.334+0000 7fc1e2800640 1 -- 172.21.15.45:0/1290948330 shutdown_connections 2024-04-16T16:15:01.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.334+0000 7fc1e2800640 1 --2- 172.21.15.45:0/1290948330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1dc155ea0 0x7fc1dc15f890 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:01.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.334+0000 7fc1e2800640 1 --2- 172.21.15.45:0/1290948330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1dc15ec80 0x7fc1dc155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:01.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.334+0000 7fc1e2800640 1 --2- 172.21.15.45:0/1290948330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1dc0b6c00 0x7fc1dc15d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:01.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.334+0000 7fc1e2800640 1 -- 172.21.15.45:0/1290948330 >> 172.21.15.45:0/1290948330 conn(0x7fc1dc0d16d0 msgr2=0x7fc1dc0b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:01.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.334+0000 7fc1e2800640 1 -- 172.21.15.45:0/1290948330 shutdown_connections 2024-04-16T16:15:01.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.334+0000 7fc1e2800640 1 -- 172.21.15.45:0/1290948330 wait complete. 2024-04-16T16:15:01.353 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:15:01.369 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:15:01.370+0000 7fdf7d7db040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:15:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.554+0000 7fc5f6600640 1 Processor -- start 2024-04-16T16:15:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.555+0000 7fc5f6600640 1 -- start start 2024-04-16T16:15:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.556+0000 7fc5f6600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc5f0162090 0x7fc5f00cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.556+0000 7fc5f6600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc5f00d5520 0x7fc5f00cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.556+0000 7fc5f6600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f00d2600 con 0x7fc5f00ba140 2024-04-16T16:15:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.556+0000 7fc5f6600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc5f00ba8b0 con 0x7fc5f00d5520 2024-04-16T16:15:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.556+0000 7fc5f6600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc5f00ba5d0 con 0x7fc5f0162090 2024-04-16T16:15:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.556+0000 7fc5f5c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc5f00d5520 0x7fc5f00cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.557+0000 7fc5f5c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc5f00d5520 0x7fc5f00cfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41914/0 (socket says 172.21.15.45:41914) 2024-04-16T16:15:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.557+0000 7fc5f5c00640 1 -- 172.21.15.45:0/3845891405 learned_addr learned my addr 172.21.15.45:0/3845891405 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.557+0000 7fc5f5200640 1 --2- 172.21.15.45:0/3845891405 >> v2:172.21.15.103:3300/0 conn(0x7fc5f0162090 0x7fc5f00cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.557+0000 7fc5f5c00640 1 -- 172.21.15.45:0/3845891405 >> v2:172.21.15.103:3300/0 conn(0x7fc5f0162090 msgr2=0x7fc5f00cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.557+0000 7fc5f5c00640 1 --2- 172.21.15.45:0/3845891405 >> v2:172.21.15.103:3300/0 conn(0x7fc5f0162090 0x7fc5f00cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.557+0000 7fc5f5c00640 1 -- 172.21.15.45:0/3845891405 >> v1:172.21.15.103:6789/0 conn(0x7fc5f00ba140 legacy=0x7fc5f01628a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.557+0000 7fc5f5c00640 1 -- 172.21.15.45:0/3845891405 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f00d28a0 con 0x7fc5f00d5520 2024-04-16T16:15:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.558+0000 7fc5f5c00640 1 --2- 172.21.15.45:0/3845891405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5f00d5520 0x7fc5f00cfa50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc5f008d5c0 tx=0x7fc5e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65622852892c141 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.558+0000 7fc5ef400640 1 -- 172.21.15.45:0/3845891405 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8035020 con 0x7fc5f00d5520 2024-04-16T16:15:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.559+0000 7fc5ef400640 1 -- 172.21.15.45:0/3845891405 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5e8031870 con 0x7fc5f00d5520 2024-04-16T16:15:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.559+0000 7fc5ef400640 1 -- 172.21.15.45:0/3845891405 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8031c40 con 0x7fc5f00d5520 2024-04-16T16:15:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.559+0000 7fc5f6600640 1 -- 172.21.15.45:0/3845891405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5f00d5520 msgr2=0x7fc5f00cfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.559+0000 7fc5f6600640 1 --2- 172.21.15.45:0/3845891405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5f00d5520 0x7fc5f00cfa50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc5f008d5c0 tx=0x7fc5e802b910 comp rx=0 tx=0).stop 2024-04-16T16:15:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.559+0000 7fc5f6600640 1 -- 172.21.15.45:0/3845891405 shutdown_connections 2024-04-16T16:15:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.559+0000 7fc5f6600640 1 --2- 172.21.15.45:0/3845891405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5f00d5520 0x7fc5f00cfa50 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.559+0000 7fc5f6600640 1 --2- 172.21.15.45:0/3845891405 >> v2:172.21.15.103:3300/0 conn(0x7fc5f0162090 0x7fc5f00cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.559+0000 7fc5f6600640 1 -- 172.21.15.45:0/3845891405 >> 172.21.15.45:0/3845891405 conn(0x7fc5f00d1940 msgr2=0x7fc5f00d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.559+0000 7fc5f6600640 1 -- 172.21.15.45:0/3845891405 shutdown_connections 2024-04-16T16:15:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.559+0000 7fc5f6600640 1 -- 172.21.15.45:0/3845891405 wait complete. 2024-04-16T16:15:01.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.561+0000 7fc5f6600640 1 Processor -- start 2024-04-16T16:15:01.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.562+0000 7fc5f6600640 1 -- start start 2024-04-16T16:15:01.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.563+0000 7fc5f6600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5f00d5520 0x7fc5f00cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:01.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.563+0000 7fc5f6600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5f0162090 0x7fc5f015e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:01.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.563+0000 7fc5f6600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5f00cc6c0 0x7fc5f015e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:01.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.563+0000 7fc5f6600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f01625f0 con 0x7fc5f0162090 2024-04-16T16:15:01.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.563+0000 7fc5f6600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f0162470 con 0x7fc5f00d5520 2024-04-16T16:15:01.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.563+0000 7fc5f6600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc5f00d40b0 con 0x7fc5f00cc6c0 2024-04-16T16:15:01.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.563+0000 7fc5efe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5f0162090 0x7fc5f015e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:01.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.563+0000 7fc5efe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5f0162090 0x7fc5f015e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41920/0 (socket says 172.21.15.45:41920) 2024-04-16T16:15:01.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.563+0000 7fc5efe00640 1 -- 172.21.15.45:0/1864132187 learned_addr learned my addr 172.21.15.45:0/1864132187 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:01.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.563+0000 7fc5efe00640 1 -- 172.21.15.45:0/1864132187 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5f00cc6c0 msgr2=0x7fc5f015e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:01.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.563+0000 7fc5efe00640 1 --2- 172.21.15.45:0/1864132187 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5f00cc6c0 0x7fc5f015e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:01.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.563+0000 7fc5efe00640 1 -- 172.21.15.45:0/1864132187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5f00d5520 msgr2=0x7fc5f00cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:01.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.563+0000 7fc5efe00640 1 --2- 172.21.15.45:0/1864132187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5f00d5520 0x7fc5f00cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:01.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.563+0000 7fc5efe00640 1 -- 172.21.15.45:0/1864132187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f00b49f0 con 0x7fc5f0162090 2024-04-16T16:15:01.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.564+0000 7fc5efe00640 1 --2- 172.21.15.45:0/1864132187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5f0162090 0x7fc5f015e0e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800c7c0 tx=0x7fc5d800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:01.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.564+0000 7fc5ecc00640 1 -- 172.21.15.45:0/1864132187 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8016020 con 0x7fc5f0162090 2024-04-16T16:15:01.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.564+0000 7fc5ecc00640 1 -- 172.21.15.45:0/1864132187 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d800aa50 con 0x7fc5f0162090 2024-04-16T16:15:01.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.564+0000 7fc5ecc00640 1 -- 172.21.15.45:0/1864132187 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8012420 con 0x7fc5f0162090 2024-04-16T16:15:01.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.564+0000 7fc5f6600640 1 -- 172.21.15.45:0/1864132187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5f00b4c90 con 0x7fc5f0162090 2024-04-16T16:15:01.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.564+0000 7fc5f6600640 1 -- 172.21.15.45:0/1864132187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5f00b5250 con 0x7fc5f0162090 2024-04-16T16:15:01.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.565+0000 7fc5f6600640 1 -- 172.21.15.45:0/1864132187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5f0075780 con 0x7fc5f0162090 2024-04-16T16:15:01.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.566+0000 7fc5ecc00640 1 -- 172.21.15.45:0/1864132187 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 7) ==== 90760+0+0 (secure 0 0 0) 0x7fc5d8019020 con 0x7fc5f0162090 2024-04-16T16:15:01.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.567+0000 7fc5ecc00640 1 -- 172.21.15.45:0/1864132187 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5967+0+0 (secure 0 0 0) 0x7fc5d808b540 con 0x7fc5f0162090 2024-04-16T16:15:01.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.572+0000 7fc5ecc00640 1 -- 172.21.15.45:0/1864132187 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc5d8020070 con 0x7fc5f0162090 2024-04-16T16:15:01.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.781+0000 7fc5f6600640 1 -- 172.21.15.45:0/1864132187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc5f00c7f20 con 0x7fc5f0162090 2024-04-16T16:15:01.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.782+0000 7fc5ecc00640 1 -- 172.21.15.45:0/1864132187 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v35) ==== 80+0+360 (secure 0 0 0) 0x7fc5d8059bd0 con 0x7fc5f0162090 2024-04-16T16:15:01.781 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2679792854 2024-04-17T16:15:01.083409+0000 2024-04-16T16:15:01.781 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3554107853 2024-04-17T16:15:01.083409+0000 2024-04-16T16:15:01.782 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3010240888 2024-04-17T16:15:01.083409+0000 2024-04-16T16:15:01.782 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/2711977587 2024-04-17T16:15:01.083409+0000 2024-04-16T16:15:01.782 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/2711977587 2024-04-17T16:15:01.083409+0000 2024-04-16T16:15:01.782 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2318159422 2024-04-17T16:15:01.083409+0000 2024-04-16T16:15:01.782 INFO:teuthology.orchestra.run.smithi045.stderr:listed 6 entries 2024-04-16T16:15:01.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.785+0000 7fc5f6600640 1 -- 172.21.15.45:0/1864132187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5f0162090 msgr2=0x7fc5f015e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:01.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.785+0000 7fc5f6600640 1 --2- 172.21.15.45:0/1864132187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5f0162090 0x7fc5f015e0e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800c7c0 tx=0x7fc5d800cc80 comp rx=0 tx=0).stop 2024-04-16T16:15:01.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.785+0000 7fc5f6600640 1 -- 172.21.15.45:0/1864132187 shutdown_connections 2024-04-16T16:15:01.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.785+0000 7fc5f6600640 1 --2- 172.21.15.45:0/1864132187 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5f00cc6c0 0x7fc5f015e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:01.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.785+0000 7fc5f6600640 1 --2- 172.21.15.45:0/1864132187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5f0162090 0x7fc5f015e0e0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:01.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.785+0000 7fc5f6600640 1 --2- 172.21.15.45:0/1864132187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5f00d5520 0x7fc5f00cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:01.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.785+0000 7fc5f6600640 1 -- 172.21.15.45:0/1864132187 >> 172.21.15.45:0/1864132187 conn(0x7fc5f00d1940 msgr2=0x7fc5f00b8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:01.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.785+0000 7fc5f6600640 1 -- 172.21.15.45:0/1864132187 shutdown_connections 2024-04-16T16:15:01.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:01.785+0000 7fc5f6600640 1 -- 172.21.15.45:0/1864132187 wait complete. 2024-04-16T16:15:01.802 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-16T16:15:01.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:01.802 DEBUG:teuthology.orchestra.run.smithi103:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-04-16T16:15:01.916 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:15:01.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:01.916 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-16T16:15:01.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:15:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.205+0000 7f8bbf600640 1 Processor -- start 2024-04-16T16:15:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.206+0000 7f8bbf600640 1 -- start start 2024-04-16T16:15:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.207+0000 7f8bbf600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bb815ed60 0x7f8bb80ba2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.207+0000 7f8bbf600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8bb80cf800 0x7f8bb80d3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.207+0000 7f8bbf600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bb815f140 con 0x7f8bb80ba7f0 2024-04-16T16:15:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.207+0000 7f8bbf600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8bb80d34f0 con 0x7f8bb80cf800 2024-04-16T16:15:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.207+0000 7f8bbf600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8bb80cfc90 con 0x7f8bb815ed60 2024-04-16T16:15:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.208+0000 7f8bb7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8bb80cf800 0x7f8bb80d3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.208+0000 7f8bb7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8bb80cf800 0x7f8bb80d3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41928/0 (socket says 172.21.15.45:41928) 2024-04-16T16:15:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.208+0000 7f8bb7e00640 1 -- 172.21.15.45:0/3739769557 learned_addr learned my addr 172.21.15.45:0/3739769557 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.208+0000 7f8bbcc00640 1 --2- 172.21.15.45:0/3739769557 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bb815ed60 0x7f8bb80ba2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.208+0000 7f8bb7e00640 1 -- 172.21.15.45:0/3739769557 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bb815ed60 msgr2=0x7f8bb80ba2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.208+0000 7f8bb7e00640 1 --2- 172.21.15.45:0/3739769557 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bb815ed60 0x7f8bb80ba2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.208+0000 7f8bb7e00640 1 -- 172.21.15.45:0/3739769557 >> v1:172.21.15.103:6789/0 conn(0x7f8bb80ba7f0 legacy=0x7f8bb80b7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.208+0000 7f8bb7e00640 1 -- 172.21.15.45:0/3739769557 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb80d3790 con 0x7f8bb80cf800 2024-04-16T16:15:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.209+0000 7f8bb7e00640 1 --2- 172.21.15.45:0/3739769557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb80cf800 0x7f8bb80d3f90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4009870 tx=0x7f8ba402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc172e617b890e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.210+0000 7f8bb7400640 1 -- 172.21.15.45:0/3739769557 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4034070 con 0x7f8bb80cf800 2024-04-16T16:15:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.210+0000 7f8bb7400640 1 -- 172.21.15.45:0/3739769557 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba4030840 con 0x7f8bb80cf800 2024-04-16T16:15:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.210+0000 7f8bb7400640 1 -- 172.21.15.45:0/3739769557 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4030c10 con 0x7f8bb80cf800 2024-04-16T16:15:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.211+0000 7f8bbf600640 1 -- 172.21.15.45:0/3739769557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb80cf800 msgr2=0x7f8bb80d3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.211+0000 7f8bbf600640 1 --2- 172.21.15.45:0/3739769557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb80cf800 0x7f8bb80d3f90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4009870 tx=0x7f8ba402b980 comp rx=0 tx=0).stop 2024-04-16T16:15:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.211+0000 7f8bbf600640 1 -- 172.21.15.45:0/3739769557 shutdown_connections 2024-04-16T16:15:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.211+0000 7f8bbf600640 1 --2- 172.21.15.45:0/3739769557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb80cf800 0x7f8bb80d3f90 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.211+0000 7f8bbf600640 1 --2- 172.21.15.45:0/3739769557 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bb815ed60 0x7f8bb80ba2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.211+0000 7f8bbf600640 1 -- 172.21.15.45:0/3739769557 >> 172.21.15.45:0/3739769557 conn(0x7f8bb8073480 msgr2=0x7f8bb8073890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.211+0000 7f8bbf600640 1 -- 172.21.15.45:0/3739769557 shutdown_connections 2024-04-16T16:15:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.211+0000 7f8bbf600640 1 -- 172.21.15.45:0/3739769557 wait complete. 2024-04-16T16:15:02.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.213+0000 7f8bbf600640 1 Processor -- start 2024-04-16T16:15:02.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.213+0000 7f8bbf600640 1 -- start start 2024-04-16T16:15:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.214+0000 7f8bbf600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb80ba7f0 0x7f8bb814a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bbf600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bb815ed60 0x7f8bb8160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bbf600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bb8161230 0x7f8bb80cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bbf600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb80d00f0 con 0x7f8bb80ba7f0 2024-04-16T16:15:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bbf600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb80cff70 con 0x7f8bb815ed60 2024-04-16T16:15:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bbf600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8bb80b8160 con 0x7f8bb8161230 2024-04-16T16:15:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bbcc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb80ba7f0 0x7f8bb814a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bbcc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb80ba7f0 0x7f8bb814a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41940/0 (socket says 172.21.15.45:41940) 2024-04-16T16:15:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bbcc00640 1 -- 172.21.15.45:0/47659148 learned_addr learned my addr 172.21.15.45:0/47659148 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bb7e00640 1 --2- 172.21.15.45:0/47659148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bb815ed60 0x7f8bb8160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bbd600640 1 --2- 172.21.15.45:0/47659148 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bb8161230 0x7f8bb80cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bbcc00640 1 -- 172.21.15.45:0/47659148 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bb8161230 msgr2=0x7f8bb80cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bbcc00640 1 --2- 172.21.15.45:0/47659148 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bb8161230 0x7f8bb80cba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bbcc00640 1 -- 172.21.15.45:0/47659148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bb815ed60 msgr2=0x7f8bb8160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bbcc00640 1 --2- 172.21.15.45:0/47659148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bb815ed60 0x7f8bb8160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bbcc00640 1 -- 172.21.15.45:0/47659148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb8161740 con 0x7f8bb80ba7f0 2024-04-16T16:15:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.215+0000 7f8bb7e00640 1 --2- 172.21.15.45:0/47659148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bb815ed60 0x7f8bb8160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.216+0000 7f8bbcc00640 1 --2- 172.21.15.45:0/47659148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb80ba7f0 0x7f8bb814a6b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f8ba800d620 tx=0x7f8ba800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.216+0000 7f8bb4c00640 1 -- 172.21.15.45:0/47659148 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba800b620 con 0x7f8bb80ba7f0 2024-04-16T16:15:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.216+0000 7f8bb4c00640 1 -- 172.21.15.45:0/47659148 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba8019070 con 0x7f8bb80ba7f0 2024-04-16T16:15:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.216+0000 7f8bb4c00640 1 -- 172.21.15.45:0/47659148 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba800ba00 con 0x7f8bb80ba7f0 2024-04-16T16:15:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.216+0000 7f8bbf600640 1 -- 172.21.15.45:0/47659148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bb8161a20 con 0x7f8bb80ba7f0 2024-04-16T16:15:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.216+0000 7f8bbf600640 1 -- 172.21.15.45:0/47659148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bb8161fe0 con 0x7f8bb80ba7f0 2024-04-16T16:15:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.218+0000 7f8bbf600640 1 -- 172.21.15.45:0/47659148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bb8075760 con 0x7f8bb80ba7f0 2024-04-16T16:15:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.218+0000 7f8bb4c00640 1 -- 172.21.15.45:0/47659148 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) ==== 90887+0+0 (secure 0 0 0) 0x7f8ba800bba0 con 0x7f8bb80ba7f0 2024-04-16T16:15:02.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.219+0000 7f8bb4c00640 1 --2- 172.21.15.45:0/47659148 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f8b9806c790 0x7f8b9806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:02.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.219+0000 7f8bb4c00640 1 -- 172.21.15.45:0/47659148 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5967+0+0 (secure 0 0 0) 0x7f8ba808bcc0 con 0x7f8bb80ba7f0 2024-04-16T16:15:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.220+0000 7f8bb7e00640 1 --2- 172.21.15.45:0/47659148 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f8b9806c790 0x7f8b9806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:02.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.221+0000 7f8bb7e00640 1 --2- 172.21.15.45:0/47659148 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f8b9806c790 0x7f8b9806ec50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4036000 tx=0x7f8ba403b040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.226+0000 7f8bb4c00640 1 -- 172.21.15.45:0/47659148 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8ba805a330 con 0x7f8bb80ba7f0 2024-04-16T16:15:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.472+0000 7f8bbf600640 1 -- 172.21.15.45:0/47659148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bb80caf20 con 0x7f8bb80ba7f0 2024-04-16T16:15:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.473+0000 7f8bb4c00640 1 -- 172.21.15.45:0/47659148 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7f8ba8059c30 con 0x7f8bb80ba7f0 2024-04-16T16:15:02.473 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/814837411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":814837411},{"type":"v1","addr":"172.21.15.103:6836","nonce":814837411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/401733314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":401733314},{"type":"v1","addr":"172.21.15.103:6837","nonce":401733314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5073,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/768816269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":768816269},{"type":"v1","addr":"172.21.15.45:6834","nonce":768816269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:14:39.132000+0000","modified":"2024-04-16T16:14:42.226029+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5079},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5079":{"gid":5079,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6835/2956262988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2956262988},{"type":"v1","addr":"172.21.15.45:6835","nonce":2956262988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5079,"qdb_cluster":[5079]},"id":2}]} 2024-04-16T16:15:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 16 2024-04-16T16:15:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.476+0000 7f8bbf600640 1 -- 172.21.15.45:0/47659148 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f8b9806c790 msgr2=0x7f8b9806ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.476+0000 7f8bbf600640 1 --2- 172.21.15.45:0/47659148 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f8b9806c790 0x7f8b9806ec50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4036000 tx=0x7f8ba403b040 comp rx=0 tx=0).stop 2024-04-16T16:15:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.476+0000 7f8bbf600640 1 -- 172.21.15.45:0/47659148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb80ba7f0 msgr2=0x7f8bb814a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.476+0000 7f8bbf600640 1 --2- 172.21.15.45:0/47659148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb80ba7f0 0x7f8bb814a6b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f8ba800d620 tx=0x7f8ba800dae0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.476+0000 7f8bbf600640 1 -- 172.21.15.45:0/47659148 shutdown_connections 2024-04-16T16:15:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.476+0000 7f8bbf600640 1 --2- 172.21.15.45:0/47659148 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bb8161230 0x7f8bb80cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.476+0000 7f8bbf600640 1 --2- 172.21.15.45:0/47659148 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f8b9806c790 0x7f8b9806ec50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.476+0000 7f8bbf600640 1 --2- 172.21.15.45:0/47659148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bb815ed60 0x7f8bb8160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.476+0000 7f8bbf600640 1 --2- 172.21.15.45:0/47659148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb80ba7f0 0x7f8bb814a6b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.476+0000 7f8bbf600640 1 -- 172.21.15.45:0/47659148 >> 172.21.15.45:0/47659148 conn(0x7f8bb8073480 msgr2=0x7f8bb80ba030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.477+0000 7f8bbf600640 1 -- 172.21.15.45:0/47659148 shutdown_connections 2024-04-16T16:15:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.477+0000 7f8bbf600640 1 -- 172.21.15.45:0/47659148 wait complete. 2024-04-16T16:15:02.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:15:02.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.724+0000 7fdb95e00640 1 Processor -- start 2024-04-16T16:15:02.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.725+0000 7fdb95e00640 1 -- start start 2024-04-16T16:15:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.726+0000 7fdb95e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdb900d52d0 0x7fdb9006ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.726+0000 7fdb95e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb900d3080 con 0x7fdb90161e90 2024-04-16T16:15:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.726+0000 7fdb95e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb900d3360 con 0x7fdb90162270 2024-04-16T16:15:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.726+0000 7fdb95e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fdb900d2e30 con 0x7fdb900d52d0 2024-04-16T16:15:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.726+0000 7fdb94a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdb90161e90 0x7fdb900cf7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57186/0 (socket says 172.21.15.45:57186) 2024-04-16T16:15:02.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.726+0000 7fdb94a00640 1 -- 172.21.15.45:0/1751817711 learned_addr learned my addr 172.21.15.45:0/1751817711 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb8f400640 1 -- 172.21.15.45:0/1751817711 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 641108853 0 0) 0x7fdb900d3080 con 0x7fdb90161e90 2024-04-16T16:15:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb8f400640 1 -- 172.21.15.45:0/1751817711 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb74003520 con 0x7fdb90161e90 2024-04-16T16:15:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb95400640 1 --2- 172.21.15.45:0/1751817711 >> v2:172.21.15.103:3300/0 conn(0x7fdb900d52d0 0x7fdb9006ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb8f400640 1 -- 172.21.15.45:0/1751817711 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 846323561 0 0) 0x7fdb74003520 con 0x7fdb90161e90 2024-04-16T16:15:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb8f400640 1 -- 172.21.15.45:0/1751817711 >> v2:172.21.15.103:3300/0 conn(0x7fdb900d52d0 msgr2=0x7fdb9006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb8f400640 1 --2- 172.21.15.45:0/1751817711 >> v2:172.21.15.103:3300/0 conn(0x7fdb900d52d0 0x7fdb9006ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb8f400640 1 -- 172.21.15.45:0/1751817711 >> v1:172.21.15.103:6789/0 conn(0x7fdb90162270 legacy=0x7fdb900d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb8f400640 1 -- 172.21.15.45:0/1751817711 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb900d3640 con 0x7fdb90161e90 2024-04-16T16:15:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb8f400640 1 -- 172.21.15.45:0/1751817711 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fdb800019f0 con 0x7fdb90161e90 2024-04-16T16:15:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb8f400640 1 -- 172.21.15.45:0/1751817711 >> v1:172.21.15.45:6789/0 conn(0x7fdb90161e90 legacy=0x7fdb900cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb8f400640 1 --2- 172.21.15.45:0/1751817711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb740039b0 0x7fdb74003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb8f400640 1 --2- 172.21.15.45:0/1751817711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb74004800 0x7fdb74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb8f400640 1 --2- 172.21.15.45:0/1751817711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb740051c0 0x7fdb7400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb8f400640 1 -- 172.21.15.45:0/1751817711 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdb900d2e30 con 0x7fdb74004800 2024-04-16T16:15:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb8f400640 1 -- 172.21.15.45:0/1751817711 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7400c110 con 0x7fdb740051c0 2024-04-16T16:15:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.727+0000 7fdb8f400640 1 -- 172.21.15.45:0/1751817711 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdb7400c310 con 0x7fdb740039b0 2024-04-16T16:15:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.728+0000 7fdb8fe00640 1 --2- 172.21.15.45:0/1751817711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb74004800 0x7fdb74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.728+0000 7fdb94a00640 1 --2- 172.21.15.45:0/1751817711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb740039b0 0x7fdb74003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.728+0000 7fdb95400640 1 --2- 172.21.15.45:0/1751817711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb740051c0 0x7fdb7400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.728+0000 7fdb8fe00640 1 -- 172.21.15.45:0/1751817711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb740039b0 msgr2=0x7fdb74003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.728+0000 7fdb8fe00640 1 --2- 172.21.15.45:0/1751817711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb740039b0 0x7fdb74003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.728+0000 7fdb8fe00640 1 -- 172.21.15.45:0/1751817711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb740051c0 msgr2=0x7fdb7400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.728+0000 7fdb8fe00640 1 --2- 172.21.15.45:0/1751817711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb740051c0 0x7fdb7400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.728+0000 7fdb8fe00640 1 -- 172.21.15.45:0/1751817711 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb74003520 con 0x7fdb74004800 2024-04-16T16:15:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.728+0000 7fdb95400640 1 -- 172.21.15.45:0/1751817711 reap_dead start 2024-04-16T16:15:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.728+0000 7fdb94a00640 1 --2- 172.21.15.45:0/1751817711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb740039b0 0x7fdb74003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.729+0000 7fdb8fe00640 1 --2- 172.21.15.45:0/1751817711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb74004800 0x7fdb74004c80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c001720 tx=0x7fdb74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33d50fdf02b66bed server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.730+0000 7fdb8f400640 1 -- 172.21.15.45:0/1751817711 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c001370 con 0x7fdb74004800 2024-04-16T16:15:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.730+0000 7fdb8f400640 1 -- 172.21.15.45:0/1751817711 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb7c0052d0 con 0x7fdb74004800 2024-04-16T16:15:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.730+0000 7fdb8f400640 1 -- 172.21.15.45:0/1751817711 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c003830 con 0x7fdb74004800 2024-04-16T16:15:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.730+0000 7fdb95e00640 1 -- 172.21.15.45:0/1751817711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb74004800 msgr2=0x7fdb74004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.730+0000 7fdb95e00640 1 --2- 172.21.15.45:0/1751817711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb74004800 0x7fdb74004c80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c001720 tx=0x7fdb74005830 comp rx=0 tx=0).stop 2024-04-16T16:15:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.730+0000 7fdb95e00640 1 -- 172.21.15.45:0/1751817711 shutdown_connections 2024-04-16T16:15:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.730+0000 7fdb95e00640 1 --2- 172.21.15.45:0/1751817711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb74004800 0x7fdb74004c80 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.730+0000 7fdb95e00640 1 -- 172.21.15.45:0/1751817711 >> 172.21.15.45:0/1751817711 conn(0x7fdb900d1940 msgr2=0x7fdb900d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.731+0000 7fdb95e00640 1 -- 172.21.15.45:0/1751817711 shutdown_connections 2024-04-16T16:15:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.731+0000 7fdb95e00640 1 -- 172.21.15.45:0/1751817711 wait complete. 2024-04-16T16:15:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.732+0000 7fdb95e00640 1 Processor -- start 2024-04-16T16:15:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.733+0000 7fdb95e00640 1 -- start start 2024-04-16T16:15:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb95e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb74004800 0x7fdb9014cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb95e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb900cbcb0 0x7fdb900ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb95e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb900ccff0 0x7fdb9015c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb95e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdb900cf950 con 0x7fdb900ccff0 2024-04-16T16:15:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb95e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb900cf7d0 con 0x7fdb900cbcb0 2024-04-16T16:15:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb95e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdb900cfad0 con 0x7fdb74004800 2024-04-16T16:15:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb95400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb900ccff0 0x7fdb9015c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb95400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb900ccff0 0x7fdb9015c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41954/0 (socket says 172.21.15.45:41954) 2024-04-16T16:15:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb95400640 1 -- 172.21.15.45:0/2414311536 learned_addr learned my addr 172.21.15.45:0/2414311536 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb94a00640 1 --2- 172.21.15.45:0/2414311536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb74004800 0x7fdb9014cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb95400640 1 -- 172.21.15.45:0/2414311536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb74004800 msgr2=0x7fdb9014cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb8fe00640 1 --2- 172.21.15.45:0/2414311536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb900cbcb0 0x7fdb900ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb95400640 1 --2- 172.21.15.45:0/2414311536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb74004800 0x7fdb9014cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb95400640 1 -- 172.21.15.45:0/2414311536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb900cbcb0 msgr2=0x7fdb900ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb95400640 1 --2- 172.21.15.45:0/2414311536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb900cbcb0 0x7fdb900ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb95400640 1 -- 172.21.15.45:0/2414311536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb900cd4c0 con 0x7fdb900ccff0 2024-04-16T16:15:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.734+0000 7fdb94a00640 1 --2- 172.21.15.45:0/2414311536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb74004800 0x7fdb9014cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.735+0000 7fdb95400640 1 --2- 172.21.15.45:0/2414311536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb900ccff0 0x7fdb9015c2a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fdb900d5780 tx=0x7fdb8800f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.735+0000 7fdb8cc00640 1 -- 172.21.15.45:0/2414311536 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88014870 con 0x7fdb900ccff0 2024-04-16T16:15:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.735+0000 7fdb8cc00640 1 -- 172.21.15.45:0/2414311536 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb8801a070 con 0x7fdb900ccff0 2024-04-16T16:15:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.735+0000 7fdb8cc00640 1 -- 172.21.15.45:0/2414311536 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88014c70 con 0x7fdb900ccff0 2024-04-16T16:15:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.735+0000 7fdb95e00640 1 -- 172.21.15.45:0/2414311536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb90157c60 con 0x7fdb900ccff0 2024-04-16T16:15:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.735+0000 7fdb95e00640 1 -- 172.21.15.45:0/2414311536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb90158220 con 0x7fdb900ccff0 2024-04-16T16:15:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.737+0000 7fdb95e00640 1 -- 172.21.15.45:0/2414311536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb90075780 con 0x7fdb900ccff0 2024-04-16T16:15:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.738+0000 7fdb8cc00640 1 -- 172.21.15.45:0/2414311536 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) ==== 90887+0+0 (secure 0 0 0) 0x7fdb88012070 con 0x7fdb900ccff0 2024-04-16T16:15:02.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.738+0000 7fdb8cc00640 1 --2- 172.21.15.45:0/2414311536 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdb7006c7c0 0x7fdb7006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:02.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.739+0000 7fdb8cc00640 1 -- 172.21.15.45:0/2414311536 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5967+0+0 (secure 0 0 0) 0x7fdb8808b860 con 0x7fdb900ccff0 2024-04-16T16:15:02.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.739+0000 7fdb94a00640 1 --2- 172.21.15.45:0/2414311536 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdb7006c7c0 0x7fdb7006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.740+0000 7fdb94a00640 1 --2- 172.21.15.45:0/2414311536 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdb7006c7c0 0x7fdb7006ec80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdb80002230 tx=0x7fdb80009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:02.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.744+0000 7fdb8cc00640 1 -- 172.21.15.45:0/2414311536 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb88059e00 con 0x7fdb900ccff0 2024-04-16T16:15:02.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.951+0000 7fdb95e00640 1 -- 172.21.15.45:0/2414311536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdb900ba1d0 con 0x7fdb900ccff0 2024-04-16T16:15:02.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.953+0000 7fdb8cc00640 1 -- 172.21.15.45:0/2414311536 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+17052 (secure 0 0 0) 0x7fdb88059700 con 0x7fdb900ccff0 2024-04-16T16:15:02.953 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:02.953 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":35,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:15:01.083563+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:14:37.171798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-16T16:14:38.122496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:15:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.956+0000 7fdb95e00640 1 -- 172.21.15.45:0/2414311536 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdb7006c7c0 msgr2=0x7fdb7006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.956+0000 7fdb95e00640 1 --2- 172.21.15.45:0/2414311536 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdb7006c7c0 0x7fdb7006ec80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdb80002230 tx=0x7fdb80009040 comp rx=0 tx=0).stop 2024-04-16T16:15:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.956+0000 7fdb95e00640 1 -- 172.21.15.45:0/2414311536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb900ccff0 msgr2=0x7fdb9015c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.956+0000 7fdb95e00640 1 --2- 172.21.15.45:0/2414311536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb900ccff0 0x7fdb9015c2a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fdb900d5780 tx=0x7fdb8800f710 comp rx=0 tx=0).stop 2024-04-16T16:15:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.956+0000 7fdb95e00640 1 -- 172.21.15.45:0/2414311536 shutdown_connections 2024-04-16T16:15:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.956+0000 7fdb95e00640 1 --2- 172.21.15.45:0/2414311536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb900ccff0 0x7fdb9015c2a0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.956+0000 7fdb95e00640 1 --2- 172.21.15.45:0/2414311536 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdb7006c7c0 0x7fdb7006ec80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.956+0000 7fdb95e00640 1 --2- 172.21.15.45:0/2414311536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb900cbcb0 0x7fdb900ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.956+0000 7fdb95e00640 1 --2- 172.21.15.45:0/2414311536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb74004800 0x7fdb9014cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.956+0000 7fdb95e00640 1 -- 172.21.15.45:0/2414311536 >> 172.21.15.45:0/2414311536 conn(0x7fdb900d1940 msgr2=0x7fdb900b6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.957+0000 7fdb95e00640 1 -- 172.21.15.45:0/2414311536 shutdown_connections 2024-04-16T16:15:02.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:02.957+0000 7fdb95e00640 1 -- 172.21.15.45:0/2414311536 wait complete. 2024-04-16T16:15:02.977 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:14:37.171798+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:14:38.122496+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:15:02.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:15:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.211+0000 7fc8d1a00640 1 Processor -- start 2024-04-16T16:15:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.211+0000 7fc8d1a00640 1 -- start start 2024-04-16T16:15:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.213+0000 7fc8d1a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8cc15c7c0 0x7fc8cc072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.213+0000 7fc8d1a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc8cc0d10e0 0x7fc8cc06f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.213+0000 7fc8d1a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8cc0b5530 con 0x7fc8cc0b4930 2024-04-16T16:15:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.213+0000 7fc8d1a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc8cc0b52e0 con 0x7fc8cc0d10e0 2024-04-16T16:15:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.213+0000 7fc8d1a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc8cc0d0260 con 0x7fc8cc15c7c0 2024-04-16T16:15:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.213+0000 7fc8cb400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc8cc0d10e0 0x7fc8cc06f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.213+0000 7fc8ca000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc8cc0b4930 0x7fc8cc0d09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57190/0 (socket says 172.21.15.45:57190) 2024-04-16T16:15:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.213+0000 7fc8ca000640 1 -- 172.21.15.45:0/4051594156 learned_addr learned my addr 172.21.15.45:0/4051594156 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.214+0000 7fc8caa00640 1 --2- 172.21.15.45:0/4051594156 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8cc15c7c0 0x7fc8cc072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.214+0000 7fc8cb400640 1 -- 172.21.15.45:0/4051594156 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8cc15c7c0 msgr2=0x7fc8cc072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:03.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.214+0000 7fc8cb400640 1 --2- 172.21.15.45:0/4051594156 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8cc15c7c0 0x7fc8cc072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.214+0000 7fc8cb400640 1 -- 172.21.15.45:0/4051594156 >> v1:172.21.15.45:6789/0 conn(0x7fc8cc0b4930 legacy=0x7fc8cc0d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:03.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.214+0000 7fc8cb400640 1 -- 172.21.15.45:0/4051594156 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8cc0b5810 con 0x7fc8cc0d10e0 2024-04-16T16:15:03.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.215+0000 7fc8cb400640 1 --2- 172.21.15.45:0/4051594156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8cc0d10e0 0x7fc8cc06f160 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009d90 tx=0x7fc8c402bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1c37b306ac8674 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:03.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.215+0000 7fc8c9600640 1 -- 172.21.15.45:0/4051594156 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4039070 con 0x7fc8cc0d10e0 2024-04-16T16:15:03.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.215+0000 7fc8c9600640 1 -- 172.21.15.45:0/4051594156 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c402cb30 con 0x7fc8cc0d10e0 2024-04-16T16:15:03.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.215+0000 7fc8c9600640 1 -- 172.21.15.45:0/4051594156 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4034440 con 0x7fc8cc0d10e0 2024-04-16T16:15:03.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.216+0000 7fc8d1a00640 1 -- 172.21.15.45:0/4051594156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8cc0d10e0 msgr2=0x7fc8cc06f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:03.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.216+0000 7fc8d1a00640 1 --2- 172.21.15.45:0/4051594156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8cc0d10e0 0x7fc8cc06f160 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009d90 tx=0x7fc8c402bd80 comp rx=0 tx=0).stop 2024-04-16T16:15:03.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.216+0000 7fc8d1a00640 1 -- 172.21.15.45:0/4051594156 shutdown_connections 2024-04-16T16:15:03.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.216+0000 7fc8d1a00640 1 --2- 172.21.15.45:0/4051594156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8cc0d10e0 0x7fc8cc06f160 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.216+0000 7fc8d1a00640 1 --2- 172.21.15.45:0/4051594156 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8cc15c7c0 0x7fc8cc072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.216+0000 7fc8d1a00640 1 -- 172.21.15.45:0/4051594156 >> 172.21.15.45:0/4051594156 conn(0x7fc8cc0b3970 msgr2=0x7fc8cc0b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:03.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.216+0000 7fc8d1a00640 1 -- 172.21.15.45:0/4051594156 shutdown_connections 2024-04-16T16:15:03.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.216+0000 7fc8d1a00640 1 -- 172.21.15.45:0/4051594156 wait complete. 2024-04-16T16:15:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.218+0000 7fc8d1a00640 1 Processor -- start 2024-04-16T16:15:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.218+0000 7fc8d1a00640 1 -- start start 2024-04-16T16:15:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.219+0000 7fc8d1a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8cc0b4930 0x7fc8cc0ce430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.219+0000 7fc8d1a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8cc0d10e0 0x7fc8cc0ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.219+0000 7fc8d1a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8cc15c7c0 0x7fc8cc0ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.219+0000 7fc8d1a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc8cc06f6a0 con 0x7fc8cc0b4930 2024-04-16T16:15:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.219+0000 7fc8d1a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc8cc073350 con 0x7fc8cc15c7c0 2024-04-16T16:15:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.219+0000 7fc8d1a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc8cc06f820 con 0x7fc8cc0d10e0 2024-04-16T16:15:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.219+0000 7fc8caa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8cc0b4930 0x7fc8cc0ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.220+0000 7fc8caa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8cc0b4930 0x7fc8cc0ce430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41970/0 (socket says 172.21.15.45:41970) 2024-04-16T16:15:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.220+0000 7fc8caa00640 1 -- 172.21.15.45:0/722059542 learned_addr learned my addr 172.21.15.45:0/722059542 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.220+0000 7fc8caa00640 1 -- 172.21.15.45:0/722059542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8cc0d10e0 msgr2=0x7fc8cc0ae2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.220+0000 7fc8caa00640 1 --2- 172.21.15.45:0/722059542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8cc0d10e0 0x7fc8cc0ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.220+0000 7fc8caa00640 1 -- 172.21.15.45:0/722059542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8cc15c7c0 msgr2=0x7fc8cc0ae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:03.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.220+0000 7fc8caa00640 1 --2- 172.21.15.45:0/722059542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8cc15c7c0 0x7fc8cc0ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.220+0000 7fc8caa00640 1 -- 172.21.15.45:0/722059542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8cc0af250 con 0x7fc8cc0b4930 2024-04-16T16:15:03.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.220+0000 7fc8caa00640 1 --2- 172.21.15.45:0/722059542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8cc0b4930 0x7fc8cc0ce430 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc00d5f0 tx=0x7fc8bc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:03.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.221+0000 7fc8c2a00640 1 -- 172.21.15.45:0/722059542 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc0125f0 con 0x7fc8cc0b4930 2024-04-16T16:15:03.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.221+0000 7fc8c2a00640 1 -- 172.21.15.45:0/722059542 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8bc019070 con 0x7fc8cc0b4930 2024-04-16T16:15:03.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.221+0000 7fc8c2a00640 1 -- 172.21.15.45:0/722059542 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc0129d0 con 0x7fc8cc0b4930 2024-04-16T16:15:03.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.221+0000 7fc8d1a00640 1 -- 172.21.15.45:0/722059542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8cc0aef70 con 0x7fc8cc0b4930 2024-04-16T16:15:03.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.221+0000 7fc8d1a00640 1 -- 172.21.15.45:0/722059542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8cc0c5c40 con 0x7fc8cc0b4930 2024-04-16T16:15:03.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.222+0000 7fc8d1a00640 1 -- 172.21.15.45:0/722059542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8cc0c6080 con 0x7fc8cc0b4930 2024-04-16T16:15:03.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.223+0000 7fc8c2a00640 1 -- 172.21.15.45:0/722059542 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) ==== 91095+0+0 (secure 0 0 0) 0x7fc8bc012b70 con 0x7fc8cc0b4930 2024-04-16T16:15:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.224+0000 7fc8c2a00640 1 --2- 172.21.15.45:0/722059542 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc8ac06c9d0 0x7fc8ac06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:03.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.224+0000 7fc8c2a00640 1 -- 172.21.15.45:0/722059542 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5967+0+0 (secure 0 0 0) 0x7fc8bc08c1d0 con 0x7fc8cc0b4930 2024-04-16T16:15:03.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.224+0000 7fc8ca000640 1 --2- 172.21.15.45:0/722059542 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc8ac06c9d0 0x7fc8ac06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:03.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.225+0000 7fc8ca000640 1 --2- 172.21.15.45:0/722059542 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc8ac06c9d0 0x7fc8ac06ee90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8000f60 tx=0x7fc8b8007a80 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:03.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.228+0000 7fc8c2a00640 1 -- 172.21.15.45:0/722059542 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc8bc05a410 con 0x7fc8cc0b4930 2024-04-16T16:15:03.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.434+0000 7fc8d1a00640 1 -- 172.21.15.45:0/722059542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc8cc075780 con 0x7fc8cc0b4930 2024-04-16T16:15:03.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.435+0000 7fc8c2a00640 1 -- 172.21.15.45:0/722059542 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) ==== 93+0+10 (secure 0 0 0) 0x7fc8bc059d10 con 0x7fc8cc0b4930 2024-04-16T16:15:03.434 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:15:03.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.438+0000 7fc8d1a00640 1 -- 172.21.15.45:0/722059542 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc8ac06c9d0 msgr2=0x7fc8ac06ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:03.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.438+0000 7fc8d1a00640 1 --2- 172.21.15.45:0/722059542 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc8ac06c9d0 0x7fc8ac06ee90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8000f60 tx=0x7fc8b8007a80 comp rx=0 tx=0).stop 2024-04-16T16:15:03.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.438+0000 7fc8d1a00640 1 -- 172.21.15.45:0/722059542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8cc0b4930 msgr2=0x7fc8cc0ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:03.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.438+0000 7fc8d1a00640 1 --2- 172.21.15.45:0/722059542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8cc0b4930 0x7fc8cc0ce430 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc00d5f0 tx=0x7fc8bc00dab0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.438+0000 7fc8d1a00640 1 -- 172.21.15.45:0/722059542 shutdown_connections 2024-04-16T16:15:03.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.438+0000 7fc8d1a00640 1 --2- 172.21.15.45:0/722059542 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc8ac06c9d0 0x7fc8ac06ee90 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.438+0000 7fc8d1a00640 1 --2- 172.21.15.45:0/722059542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8cc15c7c0 0x7fc8cc0ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.438+0000 7fc8d1a00640 1 --2- 172.21.15.45:0/722059542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8cc0d10e0 0x7fc8cc0ae2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.438+0000 7fc8d1a00640 1 --2- 172.21.15.45:0/722059542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8cc0b4930 0x7fc8cc0ce430 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.438+0000 7fc8d1a00640 1 -- 172.21.15.45:0/722059542 >> 172.21.15.45:0/722059542 conn(0x7fc8cc0b3970 msgr2=0x7fc8cc15f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:03.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.439+0000 7fc8d1a00640 1 -- 172.21.15.45:0/722059542 shutdown_connections 2024-04-16T16:15:03.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.439+0000 7fc8d1a00640 1 -- 172.21.15.45:0/722059542 wait complete. 2024-04-16T16:15:03.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:15:03.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.688+0000 7fd4df800640 1 Processor -- start 2024-04-16T16:15:03.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.689+0000 7fd4df800640 1 -- start start 2024-04-16T16:15:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.690+0000 7fd4df800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd4d80cfe00 0x7fd4d80d2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.690+0000 7fd4df800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4d80d1f10 0x7fd4d80d2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.690+0000 7fd4df800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4d80d4640 con 0x7fd4d81622d0 2024-04-16T16:15:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.690+0000 7fd4df800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd4d80d43f0 con 0x7fd4d80cfe00 2024-04-16T16:15:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.690+0000 7fd4df800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d80ba4c0 con 0x7fd4d80d1f10 2024-04-16T16:15:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.691+0000 7fd4dce00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd4d81622d0 0x7fd4d80cf6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48680/0 (socket says 172.21.15.45:48680) 2024-04-16T16:15:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.691+0000 7fd4dce00640 1 -- 172.21.15.45:0/1090532668 learned_addr learned my addr 172.21.15.45:0/1090532668 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.691+0000 7fd4d7e00640 1 --2- 172.21.15.45:0/1090532668 >> v2:172.21.15.45:3300/0 conn(0x7fd4d80cfe00 0x7fd4d80d2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.691+0000 7fd4dd800640 1 --2- 172.21.15.45:0/1090532668 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4d80d1f10 0x7fd4d80d2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.691+0000 7fd4d7400640 1 -- 172.21.15.45:0/1090532668 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 342502230 0 0) 0x7fd4d80d4640 con 0x7fd4d81622d0 2024-04-16T16:15:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.691+0000 7fd4d7400640 1 -- 172.21.15.45:0/1090532668 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4c0003520 con 0x7fd4d81622d0 2024-04-16T16:15:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.692+0000 7fd4d7e00640 1 -- 172.21.15.45:0/1090532668 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4d80d1f10 msgr2=0x7fd4d80d2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.692+0000 7fd4d7e00640 1 --2- 172.21.15.45:0/1090532668 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4d80d1f10 0x7fd4d80d2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.692+0000 7fd4d7e00640 1 -- 172.21.15.45:0/1090532668 >> v1:172.21.15.45:6789/0 conn(0x7fd4d81622d0 legacy=0x7fd4d80cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.692+0000 7fd4d7e00640 1 -- 172.21.15.45:0/1090532668 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d80b7b50 con 0x7fd4d80cfe00 2024-04-16T16:15:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.693+0000 7fd4d7e00640 1 --2- 172.21.15.45:0/1090532668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4d80cfe00 0x7fd4d80d2680 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40097a0 tx=0x7fd4c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bef8a10a6e8d30 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.693+0000 7fd4d7400640 1 -- 172.21.15.45:0/1090532668 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c4035020 con 0x7fd4d80cfe00 2024-04-16T16:15:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.693+0000 7fd4d7400640 1 -- 172.21.15.45:0/1090532668 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4c4031830 con 0x7fd4d80cfe00 2024-04-16T16:15:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.693+0000 7fd4d7400640 1 -- 172.21.15.45:0/1090532668 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c4031c00 con 0x7fd4d80cfe00 2024-04-16T16:15:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.694+0000 7fd4df800640 1 -- 172.21.15.45:0/1090532668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4d80cfe00 msgr2=0x7fd4d80d2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.694+0000 7fd4df800640 1 --2- 172.21.15.45:0/1090532668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4d80cfe00 0x7fd4d80d2680 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40097a0 tx=0x7fd4c402b8d0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.694+0000 7fd4df800640 1 -- 172.21.15.45:0/1090532668 shutdown_connections 2024-04-16T16:15:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.694+0000 7fd4df800640 1 --2- 172.21.15.45:0/1090532668 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4d80d1f10 0x7fd4d80d2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.694+0000 7fd4df800640 1 --2- 172.21.15.45:0/1090532668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4d80cfe00 0x7fd4d80d2680 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.694+0000 7fd4df800640 1 -- 172.21.15.45:0/1090532668 >> 172.21.15.45:0/1090532668 conn(0x7fd4d806d400 msgr2=0x7fd4d806d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.694+0000 7fd4df800640 1 -- 172.21.15.45:0/1090532668 shutdown_connections 2024-04-16T16:15:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.694+0000 7fd4df800640 1 -- 172.21.15.45:0/1090532668 wait complete. 2024-04-16T16:15:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.696+0000 7fd4df800640 1 Processor -- start 2024-04-16T16:15:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.696+0000 7fd4df800640 1 -- start start 2024-04-16T16:15:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.697+0000 7fd4df800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4d80cfe00 0x7fd4d814bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.697+0000 7fd4df800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4d80d1f10 0x7fd4d80b5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.697+0000 7fd4df800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4d81622d0 0x7fd4d80b5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.697+0000 7fd4df800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d80b70e0 con 0x7fd4d81622d0 2024-04-16T16:15:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.697+0000 7fd4df800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d80b6f60 con 0x7fd4d80d1f10 2024-04-16T16:15:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.697+0000 7fd4df800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d80d3100 con 0x7fd4d80cfe00 2024-04-16T16:15:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.698+0000 7fd4dd800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4d81622d0 0x7fd4d80b5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.698+0000 7fd4dd800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4d81622d0 0x7fd4d80b5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60114/0 (socket says 172.21.15.45:60114) 2024-04-16T16:15:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.698+0000 7fd4dd800640 1 -- 172.21.15.45:0/3032882533 learned_addr learned my addr 172.21.15.45:0/3032882533 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.698+0000 7fd4dd800640 1 -- 172.21.15.45:0/3032882533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4d80cfe00 msgr2=0x7fd4d814bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.698+0000 7fd4dd800640 1 --2- 172.21.15.45:0/3032882533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4d80cfe00 0x7fd4d814bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.698+0000 7fd4dd800640 1 -- 172.21.15.45:0/3032882533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4d80d1f10 msgr2=0x7fd4d80b5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.698+0000 7fd4dd800640 1 --2- 172.21.15.45:0/3032882533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4d80d1f10 0x7fd4d80b5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.698+0000 7fd4dd800640 1 -- 172.21.15.45:0/3032882533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d80c8540 con 0x7fd4d81622d0 2024-04-16T16:15:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.699+0000 7fd4dd800640 1 --2- 172.21.15.45:0/3032882533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4d81622d0 0x7fd4d80b5fd0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0007e40 tx=0x7fd4d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.699+0000 7fd4d4c00640 1 -- 172.21.15.45:0/3032882533 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0014620 con 0x7fd4d81622d0 2024-04-16T16:15:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.699+0000 7fd4d4c00640 1 -- 172.21.15.45:0/3032882533 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4d0019070 con 0x7fd4d81622d0 2024-04-16T16:15:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.699+0000 7fd4d4c00640 1 -- 172.21.15.45:0/3032882533 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0014a20 con 0x7fd4d81622d0 2024-04-16T16:15:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.699+0000 7fd4df800640 1 -- 172.21.15.45:0/3032882533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4d80c8820 con 0x7fd4d81622d0 2024-04-16T16:15:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.699+0000 7fd4df800640 1 -- 172.21.15.45:0/3032882533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4d80c8de0 con 0x7fd4d81622d0 2024-04-16T16:15:03.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.701+0000 7fd4df800640 1 -- 172.21.15.45:0/3032882533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4d8075780 con 0x7fd4d81622d0 2024-04-16T16:15:03.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.702+0000 7fd4d4c00640 1 -- 172.21.15.45:0/3032882533 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) ==== 91095+0+0 (secure 0 0 0) 0x7fd4d0014bc0 con 0x7fd4d81622d0 2024-04-16T16:15:03.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.703+0000 7fd4d4c00640 1 --2- 172.21.15.45:0/3032882533 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fd4b806ca20 0x7fd4b806eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.703+0000 7fd4d4c00640 1 -- 172.21.15.45:0/3032882533 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5967+0+0 (secure 0 0 0) 0x7fd4d008c510 con 0x7fd4d81622d0 2024-04-16T16:15:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.703+0000 7fd4dce00640 1 --2- 172.21.15.45:0/3032882533 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fd4b806ca20 0x7fd4b806eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:03.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.705+0000 7fd4dce00640 1 --2- 172.21.15.45:0/3032882533 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fd4b806ca20 0x7fd4b806eee0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc000f60 tx=0x7fd4cc0078f0 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:03.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.708+0000 7fd4d4c00640 1 -- 172.21.15.45:0/3032882533 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd4d005a960 con 0x7fd4d81622d0 2024-04-16T16:15:03.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.913+0000 7fd4df800640 1 -- 172.21.15.45:0/3032882533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd4d80b72e0 con 0x7fd4d81622d0 2024-04-16T16:15:03.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.914+0000 7fd4d4c00640 1 -- 172.21.15.45:0/3032882533 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v35) ==== 104+0+11 (secure 0 0 0) 0x7fd4d005a260 con 0x7fd4d81622d0 2024-04-16T16:15:03.913 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:15:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.916+0000 7fd4df800640 1 -- 172.21.15.45:0/3032882533 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fd4b806ca20 msgr2=0x7fd4b806eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.916+0000 7fd4df800640 1 --2- 172.21.15.45:0/3032882533 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fd4b806ca20 0x7fd4b806eee0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc000f60 tx=0x7fd4cc0078f0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.916+0000 7fd4df800640 1 -- 172.21.15.45:0/3032882533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4d81622d0 msgr2=0x7fd4d80b5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.916+0000 7fd4df800640 1 --2- 172.21.15.45:0/3032882533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4d81622d0 0x7fd4d80b5fd0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0007e40 tx=0x7fd4d000e4b0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.917+0000 7fd4df800640 1 -- 172.21.15.45:0/3032882533 shutdown_connections 2024-04-16T16:15:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.917+0000 7fd4df800640 1 --2- 172.21.15.45:0/3032882533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4d81622d0 0x7fd4d80b5fd0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.917+0000 7fd4df800640 1 --2- 172.21.15.45:0/3032882533 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fd4b806ca20 0x7fd4b806eee0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.917+0000 7fd4df800640 1 --2- 172.21.15.45:0/3032882533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4d80d1f10 0x7fd4d80b5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.917+0000 7fd4df800640 1 --2- 172.21.15.45:0/3032882533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4d80cfe00 0x7fd4d814bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.917+0000 7fd4df800640 1 -- 172.21.15.45:0/3032882533 >> 172.21.15.45:0/3032882533 conn(0x7fd4d806d400 msgr2=0x7fd4d815fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.917+0000 7fd4df800640 1 -- 172.21.15.45:0/3032882533 shutdown_connections 2024-04-16T16:15:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:03.917+0000 7fd4df800640 1 -- 172.21.15.45:0/3032882533 wait complete. 2024-04-16T16:15:03.934 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:15:04.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.162+0000 7f1eb9e00640 1 Processor -- start 2024-04-16T16:15:04.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.162+0000 7f1eb9e00640 1 -- start start 2024-04-16T16:15:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.163+0000 7f1eb9e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1eb40b6000 0x7f1eb40ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:04.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.163+0000 7f1eb9e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb40ceb50 0x7f1eb4160040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:04.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.163+0000 7f1eb9e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eb4148570 con 0x7f1eb4148190 2024-04-16T16:15:04.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.163+0000 7f1eb9e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1eb415fbb0 con 0x7f1eb40b6000 2024-04-16T16:15:04.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.163+0000 7f1eb9e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1eb415efa0 con 0x7f1eb40ceb50 2024-04-16T16:15:04.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.164+0000 7f1eb8a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1eb4148190 0x7f1eb40b58f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48694/0 (socket says 172.21.15.45:48694) 2024-04-16T16:15:04.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.164+0000 7f1eb8a00640 1 -- 172.21.15.45:0/3538716706 learned_addr learned my addr 172.21.15.45:0/3538716706 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:04.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.164+0000 7f1eb3e00640 1 --2- 172.21.15.45:0/3538716706 >> v2:172.21.15.103:3300/0 conn(0x7f1eb40b6000 0x7f1eb40ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:04.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.164+0000 7f1eb9400640 1 --2- 172.21.15.45:0/3538716706 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb40ceb50 0x7f1eb4160040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:04.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.164+0000 7f1eb3400640 1 -- 172.21.15.45:0/3538716706 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 646081260 0 0) 0x7f1eb4148570 con 0x7f1eb4148190 2024-04-16T16:15:04.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.164+0000 7f1eb3400640 1 -- 172.21.15.45:0/3538716706 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e9c003520 con 0x7f1eb4148190 2024-04-16T16:15:04.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.165+0000 7f1eb3400640 1 -- 172.21.15.45:0/3538716706 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2422491419 0 0) 0x7f1e9c003520 con 0x7f1eb4148190 2024-04-16T16:15:04.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.165+0000 7f1eb3400640 1 -- 172.21.15.45:0/3538716706 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb40ceb50 msgr2=0x7f1eb4160040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:04.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.165+0000 7f1eb3400640 1 --2- 172.21.15.45:0/3538716706 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb40ceb50 0x7f1eb4160040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.165+0000 7f1eb3400640 1 -- 172.21.15.45:0/3538716706 >> v2:172.21.15.103:3300/0 conn(0x7f1eb40b6000 msgr2=0x7f1eb40ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:04.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.165+0000 7f1eb3400640 1 --2- 172.21.15.45:0/3538716706 >> v2:172.21.15.103:3300/0 conn(0x7f1eb40b6000 0x7f1eb40ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.165+0000 7f1eb3400640 1 -- 172.21.15.45:0/3538716706 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb415f280 con 0x7f1eb4148190 2024-04-16T16:15:04.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.165+0000 7f1eb3400640 1 -- 172.21.15.45:0/3538716706 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f1ea8001670 con 0x7f1eb4148190 2024-04-16T16:15:04.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.165+0000 7f1eb3400640 1 -- 172.21.15.45:0/3538716706 >> v1:172.21.15.45:6789/0 conn(0x7f1eb4148190 legacy=0x7f1eb40b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:04.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.165+0000 7f1eb3400640 1 --2- 172.21.15.45:0/3538716706 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb40ceb50 0x7f1e9c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:04.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.165+0000 7f1eb3400640 1 --2- 172.21.15.45:0/3538716706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e9c003f80 0x7f1e9c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:04.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.165+0000 7f1eb3400640 1 --2- 172.21.15.45:0/3538716706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e9c004d40 0x7f1e9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:04.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.165+0000 7f1eb3400640 1 -- 172.21.15.45:0/3538716706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb415fbb0 con 0x7f1e9c004d40 2024-04-16T16:15:04.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.165+0000 7f1eb3400640 1 -- 172.21.15.45:0/3538716706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb415efa0 con 0x7f1e9c003f80 2024-04-16T16:15:04.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.165+0000 7f1eb3400640 1 -- 172.21.15.45:0/3538716706 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1e9c00b9c0 con 0x7f1eb40ceb50 2024-04-16T16:15:04.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.165+0000 7f1eb9400640 1 --2- 172.21.15.45:0/3538716706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e9c004d40 0x7f1e9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:04.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.166+0000 7f1eb3e00640 1 --2- 172.21.15.45:0/3538716706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e9c003f80 0x7f1e9c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:04.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.166+0000 7f1eb8a00640 1 --2- 172.21.15.45:0/3538716706 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb40ceb50 0x7f1e9c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:04.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.166+0000 7f1eb9400640 1 -- 172.21.15.45:0/3538716706 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb40ceb50 msgr2=0x7f1e9c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:04.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.166+0000 7f1eb9400640 1 --2- 172.21.15.45:0/3538716706 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb40ceb50 0x7f1e9c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.166+0000 7f1eb9400640 1 -- 172.21.15.45:0/3538716706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e9c003f80 msgr2=0x7f1e9c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:04.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.166+0000 7f1eb9400640 1 --2- 172.21.15.45:0/3538716706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e9c003f80 0x7f1e9c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.166+0000 7f1eb9400640 1 -- 172.21.15.45:0/3538716706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9c003520 con 0x7f1e9c004d40 2024-04-16T16:15:04.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.167+0000 7f1eb9400640 1 --2- 172.21.15.45:0/3538716706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e9c004d40 0x7f1e9c0051c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00e170 tx=0x7f1e9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66bfc7d4d6b33290 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:04.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.167+0000 7f1eb3400640 1 -- 172.21.15.45:0/3538716706 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eac039070 con 0x7f1e9c004d40 2024-04-16T16:15:04.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.167+0000 7f1eb3400640 1 -- 172.21.15.45:0/3538716706 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1eac02cbf0 con 0x7f1e9c004d40 2024-04-16T16:15:04.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.167+0000 7f1eb3400640 1 -- 172.21.15.45:0/3538716706 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eac009c10 con 0x7f1e9c004d40 2024-04-16T16:15:04.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.168+0000 7f1eb9e00640 1 -- 172.21.15.45:0/3538716706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e9c004d40 msgr2=0x7f1e9c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:04.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.168+0000 7f1eb9e00640 1 --2- 172.21.15.45:0/3538716706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e9c004d40 0x7f1e9c0051c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00e170 tx=0x7f1e9c005830 comp rx=0 tx=0).stop 2024-04-16T16:15:04.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.168+0000 7f1eb9400640 1 -- 172.21.15.45:0/3538716706 reap_dead start 2024-04-16T16:15:04.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.168+0000 7f1eb9e00640 1 -- 172.21.15.45:0/3538716706 shutdown_connections 2024-04-16T16:15:04.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.168+0000 7f1eb9e00640 1 -- 172.21.15.45:0/3538716706 >> 172.21.15.45:0/3538716706 conn(0x7f1eb406b930 msgr2=0x7f1eb406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:04.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.168+0000 7f1eb9e00640 1 -- 172.21.15.45:0/3538716706 shutdown_connections 2024-04-16T16:15:04.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.168+0000 7f1eb9e00640 1 -- 172.21.15.45:0/3538716706 wait complete. 2024-04-16T16:15:04.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.170+0000 7f1eb9e00640 1 Processor -- start 2024-04-16T16:15:04.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.170+0000 7f1eb9e00640 1 -- start start 2024-04-16T16:15:04.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.171+0000 7f1eb9e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e9c004d40 0x7f1eb40ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:04.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.171+0000 7f1eb9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eb40caad0 0x7f1eb413d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:04.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.171+0000 7f1eb9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eb4140240 0x7f1eb41406c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:04.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.171+0000 7f1eb9e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb40cefc0 con 0x7f1eb40caad0 2024-04-16T16:15:04.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.171+0000 7f1eb9e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb415f840 con 0x7f1eb4140240 2024-04-16T16:15:04.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.171+0000 7f1eb9e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1eb40cf140 con 0x7f1e9c004d40 2024-04-16T16:15:04.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.171+0000 7f1eb3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eb40caad0 0x7f1eb413d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:04.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.172+0000 7f1eb3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eb40caad0 0x7f1eb413d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60126/0 (socket says 172.21.15.45:60126) 2024-04-16T16:15:04.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.172+0000 7f1eb3e00640 1 -- 172.21.15.45:0/3135319338 learned_addr learned my addr 172.21.15.45:0/3135319338 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:04.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.172+0000 7f1eb9400640 1 --2- 172.21.15.45:0/3135319338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eb4140240 0x7f1eb41406c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:04.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.172+0000 7f1eb8a00640 1 --2- 172.21.15.45:0/3135319338 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e9c004d40 0x7f1eb40ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:04.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.172+0000 7f1eb3e00640 1 -- 172.21.15.45:0/3135319338 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e9c004d40 msgr2=0x7f1eb40ccf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:04.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.172+0000 7f1eb3e00640 1 --2- 172.21.15.45:0/3135319338 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e9c004d40 0x7f1eb40ccf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.172+0000 7f1eb3e00640 1 -- 172.21.15.45:0/3135319338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eb4140240 msgr2=0x7f1eb41406c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:04.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.172+0000 7f1eb3e00640 1 --2- 172.21.15.45:0/3135319338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eb4140240 0x7f1eb41406c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.172+0000 7f1eb3e00640 1 -- 172.21.15.45:0/3135319338 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb4142c00 con 0x7f1eb40caad0 2024-04-16T16:15:04.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.172+0000 7f1eb9400640 1 --2- 172.21.15.45:0/3135319338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eb4140240 0x7f1eb41406c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:04.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.172+0000 7f1eb3e00640 1 --2- 172.21.15.45:0/3135319338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eb40caad0 0x7f1eb413d200 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0011600 tx=0x7f1ea0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:04.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.173+0000 7f1eb0c00640 1 -- 172.21.15.45:0/3135319338 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea000f920 con 0x7f1eb40caad0 2024-04-16T16:15:04.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.173+0000 7f1eb0c00640 1 -- 172.21.15.45:0/3135319338 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ea000fac0 con 0x7f1eb40caad0 2024-04-16T16:15:04.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.173+0000 7f1eb9e00640 1 -- 172.21.15.45:0/3135319338 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eb40cd6c0 con 0x7f1eb40caad0 2024-04-16T16:15:04.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.173+0000 7f1eb0c00640 1 -- 172.21.15.45:0/3135319338 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea000fe70 con 0x7f1eb40caad0 2024-04-16T16:15:04.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.173+0000 7f1eb9e00640 1 -- 172.21.15.45:0/3135319338 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eb4142ec0 con 0x7f1eb40caad0 2024-04-16T16:15:04.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.174+0000 7f1eb9e00640 1 -- 172.21.15.45:0/3135319338 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eb4075780 con 0x7f1eb40caad0 2024-04-16T16:15:04.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.175+0000 7f1eb0c00640 1 -- 172.21.15.45:0/3135319338 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) ==== 91095+0+0 (secure 0 0 0) 0x7f1ea0015460 con 0x7f1eb40caad0 2024-04-16T16:15:04.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.176+0000 7f1eb0c00640 1 --2- 172.21.15.45:0/3135319338 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1e9406ca00 0x7f1e9406eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:04.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.176+0000 7f1eb0c00640 1 -- 172.21.15.45:0/3135319338 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5967+0+0 (secure 0 0 0) 0x7f1ea008c810 con 0x7f1eb40caad0 2024-04-16T16:15:04.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.177+0000 7f1eb8a00640 1 --2- 172.21.15.45:0/3135319338 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1e9406ca00 0x7f1e9406eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:04.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.178+0000 7f1eb8a00640 1 --2- 172.21.15.45:0/3135319338 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1e9406ca00 0x7f1e9406eec0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8001eb0 tx=0x7f1ea8009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:04.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.180+0000 7f1eb0c00640 1 -- 172.21.15.45:0/3135319338 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1ea0019030 con 0x7f1eb40caad0 2024-04-16T16:15:04.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.382+0000 7f1eb9e00640 1 -- 172.21.15.45:0/3135319338 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1eb40b6590 con 0x7f1eb40caad0 2024-04-16T16:15:04.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.384+0000 7f1eb0c00640 1 -- 172.21.15.45:0/3135319338 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v35) ==== 100+0+11 (secure 0 0 0) 0x7f1ea005a9c0 con 0x7f1eb40caad0 2024-04-16T16:15:04.383 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:15:04.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.386+0000 7f1eb9e00640 1 -- 172.21.15.45:0/3135319338 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1e9406ca00 msgr2=0x7f1e9406eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:04.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.386+0000 7f1eb9e00640 1 --2- 172.21.15.45:0/3135319338 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1e9406ca00 0x7f1e9406eec0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8001eb0 tx=0x7f1ea8009040 comp rx=0 tx=0).stop 2024-04-16T16:15:04.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.386+0000 7f1eb9e00640 1 -- 172.21.15.45:0/3135319338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eb40caad0 msgr2=0x7f1eb413d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:04.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.386+0000 7f1eb9e00640 1 --2- 172.21.15.45:0/3135319338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eb40caad0 0x7f1eb413d200 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0011600 tx=0x7f1ea0011ac0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.386+0000 7f1eb9e00640 1 -- 172.21.15.45:0/3135319338 shutdown_connections 2024-04-16T16:15:04.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.386+0000 7f1eb9e00640 1 --2- 172.21.15.45:0/3135319338 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1e9406ca00 0x7f1e9406eec0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.386+0000 7f1eb9e00640 1 --2- 172.21.15.45:0/3135319338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eb4140240 0x7f1eb41406c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.386+0000 7f1eb9e00640 1 --2- 172.21.15.45:0/3135319338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eb40caad0 0x7f1eb413d200 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.386+0000 7f1eb9e00640 1 --2- 172.21.15.45:0/3135319338 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e9c004d40 0x7f1eb40ccf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.386+0000 7f1eb9e00640 1 -- 172.21.15.45:0/3135319338 >> 172.21.15.45:0/3135319338 conn(0x7f1eb406b930 msgr2=0x7f1eb40b24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:04.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.386+0000 7f1eb9e00640 1 -- 172.21.15.45:0/3135319338 shutdown_connections 2024-04-16T16:15:04.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.386+0000 7f1eb9e00640 1 -- 172.21.15.45:0/3135319338 wait complete. 2024-04-16T16:15:04.403 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:15:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.621+0000 7fdc9f200640 1 Processor -- start 2024-04-16T16:15:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.622+0000 7fdc9f200640 1 -- start start 2024-04-16T16:15:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.623+0000 7fdc9f200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdc98161ff0 0x7fdc980cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.623+0000 7fdc9f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdc980b6e40 0x7fdc980b7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.623+0000 7fdc9f200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc980d46e0 con 0x7fdc980cfd20 2024-04-16T16:15:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.623+0000 7fdc9f200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fdc980ba560 con 0x7fdc98161ff0 2024-04-16T16:15:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.623+0000 7fdc9f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fdc980d3640 con 0x7fdc980b6e40 2024-04-16T16:15:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.624+0000 7fdc9d400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdc980cfd20 0x7fdc980b6730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48698/0 (socket says 172.21.15.45:48698) 2024-04-16T16:15:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.624+0000 7fdc9d400640 1 -- 172.21.15.45:0/1850277269 learned_addr learned my addr 172.21.15.45:0/1850277269 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.624+0000 7fdc9de00640 1 --2- 172.21.15.45:0/1850277269 >> v2:172.21.15.45:3300/0 conn(0x7fdc98161ff0 0x7fdc980cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.624+0000 7fdc9e800640 1 --2- 172.21.15.45:0/1850277269 >> v2:172.21.15.103:3300/0 conn(0x7fdc980b6e40 0x7fdc980b7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.624+0000 7fdc9ca00640 1 -- 172.21.15.45:0/1850277269 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3447098228 0 0) 0x7fdc980d46e0 con 0x7fdc980cfd20 2024-04-16T16:15:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.624+0000 7fdc9ca00640 1 -- 172.21.15.45:0/1850277269 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc80003520 con 0x7fdc980cfd20 2024-04-16T16:15:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.624+0000 7fdc9de00640 1 -- 172.21.15.45:0/1850277269 >> v2:172.21.15.103:3300/0 conn(0x7fdc980b6e40 msgr2=0x7fdc980b7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.624+0000 7fdc9de00640 1 --2- 172.21.15.45:0/1850277269 >> v2:172.21.15.103:3300/0 conn(0x7fdc980b6e40 0x7fdc980b7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.624+0000 7fdc9de00640 1 -- 172.21.15.45:0/1850277269 >> v1:172.21.15.45:6789/0 conn(0x7fdc980cfd20 legacy=0x7fdc980b6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:04.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.624+0000 7fdc9de00640 1 -- 172.21.15.45:0/1850277269 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc980d3920 con 0x7fdc98161ff0 2024-04-16T16:15:04.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.624+0000 7fdc9e800640 1 --2- 172.21.15.45:0/1850277269 >> v2:172.21.15.103:3300/0 conn(0x7fdc980b6e40 0x7fdc980b7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:15:04.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.626+0000 7fdc9de00640 1 --2- 172.21.15.45:0/1850277269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc98161ff0 0x7fdc980cf7e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdc980b7500 tx=0x7fdc8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32740b0e773454be server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.626+0000 7fdc9ca00640 1 -- 172.21.15.45:0/1850277269 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c035020 con 0x7fdc98161ff0 2024-04-16T16:15:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.626+0000 7fdc9ca00640 1 -- 172.21.15.45:0/1850277269 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc8c031830 con 0x7fdc98161ff0 2024-04-16T16:15:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.626+0000 7fdc9ca00640 1 -- 172.21.15.45:0/1850277269 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c031c00 con 0x7fdc98161ff0 2024-04-16T16:15:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.626+0000 7fdc9f200640 1 -- 172.21.15.45:0/1850277269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc98161ff0 msgr2=0x7fdc980cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.626+0000 7fdc9f200640 1 --2- 172.21.15.45:0/1850277269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc98161ff0 0x7fdc980cf7e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdc980b7500 tx=0x7fdc8c02b8d0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.626+0000 7fdc9f200640 1 -- 172.21.15.45:0/1850277269 shutdown_connections 2024-04-16T16:15:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.626+0000 7fdc9f200640 1 --2- 172.21.15.45:0/1850277269 >> v2:172.21.15.103:3300/0 conn(0x7fdc980b6e40 0x7fdc980b7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.626+0000 7fdc9f200640 1 --2- 172.21.15.45:0/1850277269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc98161ff0 0x7fdc980cf7e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.626+0000 7fdc9f200640 1 -- 172.21.15.45:0/1850277269 >> 172.21.15.45:0/1850277269 conn(0x7fdc980d17b0 msgr2=0x7fdc980d1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.627+0000 7fdc9f200640 1 -- 172.21.15.45:0/1850277269 shutdown_connections 2024-04-16T16:15:04.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.627+0000 7fdc9f200640 1 -- 172.21.15.45:0/1850277269 wait complete. 2024-04-16T16:15:04.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.628+0000 7fdc9f200640 1 Processor -- start 2024-04-16T16:15:04.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.629+0000 7fdc9f200640 1 -- start start 2024-04-16T16:15:04.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.629+0000 7fdc9f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc980b6e40 0x7fdc980ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:04.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.630+0000 7fdc9f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc980cfd20 0x7fdc980abdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:04.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.630+0000 7fdc9f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc98161ff0 0x7fdc98155060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:04.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.630+0000 7fdc9f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdc980d41c0 con 0x7fdc98161ff0 2024-04-16T16:15:04.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.630+0000 7fdc9f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdc980d3ee0 con 0x7fdc980cfd20 2024-04-16T16:15:04.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.630+0000 7fdc9f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdc981623d0 con 0x7fdc980b6e40 2024-04-16T16:15:04.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.630+0000 7fdc9e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc98161ff0 0x7fdc98155060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:04.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.630+0000 7fdc9e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc98161ff0 0x7fdc98155060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60138/0 (socket says 172.21.15.45:60138) 2024-04-16T16:15:04.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.630+0000 7fdc9e800640 1 -- 172.21.15.45:0/3085585989 learned_addr learned my addr 172.21.15.45:0/3085585989 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:04.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.630+0000 7fdc9de00640 1 --2- 172.21.15.45:0/3085585989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc980b6e40 0x7fdc980ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:04.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.630+0000 7fdc9e800640 1 -- 172.21.15.45:0/3085585989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc980b6e40 msgr2=0x7fdc980ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:04.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.630+0000 7fdc9e800640 1 --2- 172.21.15.45:0/3085585989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc980b6e40 0x7fdc980ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.630+0000 7fdc9e800640 1 -- 172.21.15.45:0/3085585989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc980cfd20 msgr2=0x7fdc980abdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:04.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.630+0000 7fdc9d400640 1 --2- 172.21.15.45:0/3085585989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc980cfd20 0x7fdc980abdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:04.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.630+0000 7fdc9e800640 1 --2- 172.21.15.45:0/3085585989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc980cfd20 0x7fdc980abdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.630+0000 7fdc9e800640 1 -- 172.21.15.45:0/3085585989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc980cbf10 con 0x7fdc98161ff0 2024-04-16T16:15:04.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.631+0000 7fdc9e800640 1 --2- 172.21.15.45:0/3085585989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc98161ff0 0x7fdc98155060 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fdc94007e20 tx=0x7fdc9400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:04.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.631+0000 7fdc92000640 1 -- 172.21.15.45:0/3085585989 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9401a070 con 0x7fdc98161ff0 2024-04-16T16:15:04.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.631+0000 7fdc92000640 1 -- 172.21.15.45:0/3085585989 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc94003b90 con 0x7fdc98161ff0 2024-04-16T16:15:04.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.631+0000 7fdc92000640 1 -- 172.21.15.45:0/3085585989 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9400de80 con 0x7fdc98161ff0 2024-04-16T16:15:04.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.631+0000 7fdc9f200640 1 -- 172.21.15.45:0/3085585989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc980cc1f0 con 0x7fdc98161ff0 2024-04-16T16:15:04.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.631+0000 7fdc9f200640 1 -- 172.21.15.45:0/3085585989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc980cc7b0 con 0x7fdc98161ff0 2024-04-16T16:15:04.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.632+0000 7fdc9f200640 1 -- 172.21.15.45:0/3085585989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc98075780 con 0x7fdc98161ff0 2024-04-16T16:15:04.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.633+0000 7fdc92000640 1 -- 172.21.15.45:0/3085585989 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) ==== 91095+0+0 (secure 0 0 0) 0x7fdc9401d070 con 0x7fdc98161ff0 2024-04-16T16:15:04.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.634+0000 7fdc92000640 1 --2- 172.21.15.45:0/3085585989 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdc7c06c9d0 0x7fdc7c06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:04.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.635+0000 7fdc92000640 1 -- 172.21.15.45:0/3085585989 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5967+0+0 (secure 0 0 0) 0x7fdc9408c0f0 con 0x7fdc98161ff0 2024-04-16T16:15:04.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.635+0000 7fdc9de00640 1 --2- 172.21.15.45:0/3085585989 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdc7c06c9d0 0x7fdc7c06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:04.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.636+0000 7fdc9de00640 1 --2- 172.21.15.45:0/3085585989 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdc7c06c9d0 0x7fdc7c06ee90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c036000 tx=0x7fdc8c008030 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:04.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.639+0000 7fdc92000640 1 -- 172.21.15.45:0/3085585989 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdc94010070 con 0x7fdc98161ff0 2024-04-16T16:15:04.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.887+0000 7fdc9f200640 1 -- 172.21.15.45:0/3085585989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc9806bda0 con 0x7fdc98161ff0 2024-04-16T16:15:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.888+0000 7fdc92000640 1 -- 172.21.15.45:0/3085585989 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7fdc9405a2b0 con 0x7fdc98161ff0 2024-04-16T16:15:04.888 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:04.888 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/814837411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":814837411},{"type":"v1","addr":"172.21.15.103:6836","nonce":814837411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/401733314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":401733314},{"type":"v1","addr":"172.21.15.103:6837","nonce":401733314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5073,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/768816269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":768816269},{"type":"v1","addr":"172.21.15.45:6834","nonce":768816269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:14:39.132000+0000","modified":"2024-04-16T16:14:42.226029+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5079},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5079":{"gid":5079,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6835/2956262988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2956262988},{"type":"v1","addr":"172.21.15.45:6835","nonce":2956262988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5079,"qdb_cluster":[5079]},"id":2}]} 2024-04-16T16:15:04.888 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 16 2024-04-16T16:15:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.891+0000 7fdc9f200640 1 -- 172.21.15.45:0/3085585989 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdc7c06c9d0 msgr2=0x7fdc7c06ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.891+0000 7fdc9f200640 1 --2- 172.21.15.45:0/3085585989 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdc7c06c9d0 0x7fdc7c06ee90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c036000 tx=0x7fdc8c008030 comp rx=0 tx=0).stop 2024-04-16T16:15:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.891+0000 7fdc9f200640 1 -- 172.21.15.45:0/3085585989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc98161ff0 msgr2=0x7fdc98155060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.891+0000 7fdc9f200640 1 --2- 172.21.15.45:0/3085585989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc98161ff0 0x7fdc98155060 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fdc94007e20 tx=0x7fdc9400d4a0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.891+0000 7fdc9f200640 1 -- 172.21.15.45:0/3085585989 shutdown_connections 2024-04-16T16:15:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.891+0000 7fdc9f200640 1 --2- 172.21.15.45:0/3085585989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc98161ff0 0x7fdc98155060 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.891+0000 7fdc9f200640 1 --2- 172.21.15.45:0/3085585989 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdc7c06c9d0 0x7fdc7c06ee90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.891+0000 7fdc9f200640 1 --2- 172.21.15.45:0/3085585989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc980cfd20 0x7fdc980abdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.891+0000 7fdc9f200640 1 --2- 172.21.15.45:0/3085585989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc980b6e40 0x7fdc980ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.891+0000 7fdc9f200640 1 -- 172.21.15.45:0/3085585989 >> 172.21.15.45:0/3085585989 conn(0x7fdc980d17b0 msgr2=0x7fdc980b6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.891+0000 7fdc9f200640 1 -- 172.21.15.45:0/3085585989 shutdown_connections 2024-04-16T16:15:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:04.891+0000 7fdc9f200640 1 -- 172.21.15.45:0/3085585989 wait complete. 2024-04-16T16:15:04.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:15:05.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.124+0000 7f7abb200640 1 Processor -- start 2024-04-16T16:15:05.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.125+0000 7f7abb200640 1 -- start start 2024-04-16T16:15:05.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.126+0000 7f7abb200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7abc072150 0x7f7abc0b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:05.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.126+0000 7f7abb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7abc162f40 0x7f7abc06cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:05.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.126+0000 7f7abb200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7abc1633f0 con 0x7f7abc15eb00 2024-04-16T16:15:05.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.126+0000 7f7abb200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f7abc0b5d40 con 0x7f7abc072150 2024-04-16T16:15:05.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.126+0000 7f7abb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7abc0b6020 con 0x7f7abc162f40 2024-04-16T16:15:05.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.126+0000 7f7ab9e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7abc15eb00 0x7f7abc071a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48714/0 (socket says 172.21.15.45:48714) 2024-04-16T16:15:05.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.127+0000 7f7ab9e00640 1 -- 172.21.15.45:0/2077408080 learned_addr learned my addr 172.21.15.45:0/2077408080 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:05.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.127+0000 7f7ab9400640 1 --2- 172.21.15.45:0/2077408080 >> v2:172.21.15.45:3300/0 conn(0x7f7abc072150 0x7f7abc0b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:05.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.127+0000 7f7aba800640 1 --2- 172.21.15.45:0/2077408080 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7abc162f40 0x7f7abc06cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:05.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.127+0000 7f7ab9400640 1 -- 172.21.15.45:0/2077408080 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7abc162f40 msgr2=0x7f7abc06cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:05.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.127+0000 7f7ab9400640 1 --2- 172.21.15.45:0/2077408080 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7abc162f40 0x7f7abc06cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.127+0000 7f7ab8a00640 1 -- 172.21.15.45:0/2077408080 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 138182096 0 0) 0x7f7abc1633f0 con 0x7f7abc15eb00 2024-04-16T16:15:05.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.127+0000 7f7ab9400640 1 -- 172.21.15.45:0/2077408080 >> v1:172.21.15.45:6789/0 conn(0x7f7abc15eb00 legacy=0x7f7abc071a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:05.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.127+0000 7f7ab9400640 1 -- 172.21.15.45:0/2077408080 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7abc0b6300 con 0x7f7abc072150 2024-04-16T16:15:05.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.129+0000 7f7ab9400640 1 --2- 172.21.15.45:0/2077408080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7abc072150 0x7f7abc0b56a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4009d90 tx=0x7f7aa402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cde991e80b3a00d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:05.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.129+0000 7f7ab8a00640 1 -- 172.21.15.45:0/2077408080 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4039070 con 0x7f7abc072150 2024-04-16T16:15:05.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.130+0000 7f7ab8a00640 1 -- 172.21.15.45:0/2077408080 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aa402cbc0 con 0x7f7abc072150 2024-04-16T16:15:05.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.130+0000 7f7ab8a00640 1 -- 172.21.15.45:0/2077408080 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4034440 con 0x7f7abc072150 2024-04-16T16:15:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.131+0000 7f7abb200640 1 -- 172.21.15.45:0/2077408080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7abc072150 msgr2=0x7f7abc0b56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.131+0000 7f7abb200640 1 --2- 172.21.15.45:0/2077408080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7abc072150 0x7f7abc0b56a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4009d90 tx=0x7f7aa402be10 comp rx=0 tx=0).stop 2024-04-16T16:15:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.131+0000 7f7abb200640 1 -- 172.21.15.45:0/2077408080 shutdown_connections 2024-04-16T16:15:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.131+0000 7f7abb200640 1 --2- 172.21.15.45:0/2077408080 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7abc162f40 0x7f7abc06cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.131+0000 7f7abb200640 1 --2- 172.21.15.45:0/2077408080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7abc072150 0x7f7abc0b56a0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.131+0000 7f7abb200640 1 -- 172.21.15.45:0/2077408080 >> 172.21.15.45:0/2077408080 conn(0x7f7abc0b43a0 msgr2=0x7f7abc0b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.131+0000 7f7abb200640 1 -- 172.21.15.45:0/2077408080 shutdown_connections 2024-04-16T16:15:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.131+0000 7f7abb200640 1 -- 172.21.15.45:0/2077408080 wait complete. 2024-04-16T16:15:05.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.133+0000 7f7abb200640 1 Processor -- start 2024-04-16T16:15:05.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.133+0000 7f7abb200640 1 -- start start 2024-04-16T16:15:05.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.134+0000 7f7abb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7abc072150 0x7f7abc1f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:05.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.134+0000 7f7abb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7abc15eb00 0x7f7abc0cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:05.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.134+0000 7f7abb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7abc162f40 0x7f7abc160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:05.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.134+0000 7f7abb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7abc06d2f0 con 0x7f7abc072150 2024-04-16T16:15:05.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.134+0000 7f7abb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7abc0b68c0 con 0x7f7abc162f40 2024-04-16T16:15:05.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.134+0000 7f7abb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7abc06d470 con 0x7f7abc15eb00 2024-04-16T16:15:05.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.134+0000 7f7ab9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7abc072150 0x7f7abc1f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:05.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.134+0000 7f7ab9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7abc072150 0x7f7abc1f41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60158/0 (socket says 172.21.15.45:60158) 2024-04-16T16:15:05.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.134+0000 7f7ab9e00640 1 -- 172.21.15.45:0/659936380 learned_addr learned my addr 172.21.15.45:0/659936380 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:05.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.135+0000 7f7ab9e00640 1 -- 172.21.15.45:0/659936380 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7abc15eb00 msgr2=0x7f7abc0cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:05.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.135+0000 7f7ab9e00640 1 --2- 172.21.15.45:0/659936380 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7abc15eb00 0x7f7abc0cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.135+0000 7f7ab9e00640 1 -- 172.21.15.45:0/659936380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7abc162f40 msgr2=0x7f7abc160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.135+0000 7f7ab9e00640 1 --2- 172.21.15.45:0/659936380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7abc162f40 0x7f7abc160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.135+0000 7f7ab9e00640 1 -- 172.21.15.45:0/659936380 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7abc0d0630 con 0x7f7abc072150 2024-04-16T16:15:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.135+0000 7f7ab9e00640 1 --2- 172.21.15.45:0/659936380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7abc072150 0x7f7abc1f41a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7aac00c500 tx=0x7f7aac00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.135+0000 7f7ab2000640 1 -- 172.21.15.45:0/659936380 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aac018070 con 0x7f7abc072150 2024-04-16T16:15:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.135+0000 7f7ab2000640 1 -- 172.21.15.45:0/659936380 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aac007560 con 0x7f7abc072150 2024-04-16T16:15:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.135+0000 7f7abb200640 1 -- 172.21.15.45:0/659936380 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7abc0d0910 con 0x7f7abc072150 2024-04-16T16:15:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.135+0000 7f7ab2000640 1 -- 172.21.15.45:0/659936380 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aac004a20 con 0x7f7abc072150 2024-04-16T16:15:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.135+0000 7f7abb200640 1 -- 172.21.15.45:0/659936380 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7abc0d0ed0 con 0x7f7abc072150 2024-04-16T16:15:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.137+0000 7f7abb200640 1 -- 172.21.15.45:0/659936380 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7abc0b4f50 con 0x7f7abc072150 2024-04-16T16:15:05.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.138+0000 7f7ab2000640 1 -- 172.21.15.45:0/659936380 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) ==== 91095+0+0 (secure 0 0 0) 0x7f7aac01b070 con 0x7f7abc072150 2024-04-16T16:15:05.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.138+0000 7f7ab2000640 1 --2- 172.21.15.45:0/659936380 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7a9806c9d0 0x7f7a9806ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:05.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.139+0000 7f7ab2000640 1 -- 172.21.15.45:0/659936380 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5967+0+0 (secure 0 0 0) 0x7f7aac08b100 con 0x7f7abc072150 2024-04-16T16:15:05.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.139+0000 7f7ab9400640 1 --2- 172.21.15.45:0/659936380 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7a9806c9d0 0x7f7a9806ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:05.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.141+0000 7f7ab9400640 1 --2- 172.21.15.45:0/659936380 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7a9806c9d0 0x7f7a9806ee90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4036000 tx=0x7f7aa402e000 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:05.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.143+0000 7f7ab2000640 1 -- 172.21.15.45:0/659936380 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7aac0593d0 con 0x7f7abc072150 2024-04-16T16:15:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.149+0000 7f7ab2000640 1 -- 172.21.15.45:0/659936380 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f7aac055340 con 0x7f7abc072150 2024-04-16T16:15:05.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.342+0000 7f7abb200640 1 -- 172.21.15.45:0/659936380 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7abc1f46e0 con 0x7f7abc072150 2024-04-16T16:15:05.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.345+0000 7f7ab2000640 1 -- 172.21.15.45:0/659936380 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+17052 (secure 0 0 0) 0x7f7aac019ca0 con 0x7f7abc072150 2024-04-16T16:15:05.345 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:05.345 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":35,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:15:01.083563+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:14:37.171798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-16T16:14:38.122496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:15:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.348+0000 7f7abb200640 1 -- 172.21.15.45:0/659936380 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7a9806c9d0 msgr2=0x7f7a9806ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.348+0000 7f7abb200640 1 --2- 172.21.15.45:0/659936380 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7a9806c9d0 0x7f7a9806ee90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4036000 tx=0x7f7aa402e000 comp rx=0 tx=0).stop 2024-04-16T16:15:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.348+0000 7f7abb200640 1 -- 172.21.15.45:0/659936380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7abc072150 msgr2=0x7f7abc1f41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.348+0000 7f7abb200640 1 --2- 172.21.15.45:0/659936380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7abc072150 0x7f7abc1f41a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7aac00c500 tx=0x7f7aac00c9c0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.348+0000 7f7abb200640 1 -- 172.21.15.45:0/659936380 shutdown_connections 2024-04-16T16:15:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.348+0000 7f7abb200640 1 --2- 172.21.15.45:0/659936380 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7a9806c9d0 0x7f7a9806ee90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.348+0000 7f7abb200640 1 --2- 172.21.15.45:0/659936380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7abc162f40 0x7f7abc160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.348+0000 7f7abb200640 1 --2- 172.21.15.45:0/659936380 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7abc15eb00 0x7f7abc0cfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.348+0000 7f7abb200640 1 --2- 172.21.15.45:0/659936380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7abc072150 0x7f7abc1f41a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.348+0000 7f7abb200640 1 -- 172.21.15.45:0/659936380 >> 172.21.15.45:0/659936380 conn(0x7f7abc0b43a0 msgr2=0x7f7abc1624c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:05.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.348+0000 7f7abb200640 1 -- 172.21.15.45:0/659936380 shutdown_connections 2024-04-16T16:15:05.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.348+0000 7f7abb200640 1 -- 172.21.15.45:0/659936380 wait complete. 2024-04-16T16:15:05.365 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-16T16:15:05.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:15:05.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.571+0000 7f1a03e00640 1 Processor -- start 2024-04-16T16:15:05.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.572+0000 7f1a03e00640 1 -- start start 2024-04-16T16:15:05.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.573+0000 7f1a03e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1a04162270 0x7f1a040d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:05.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.573+0000 7f1a03e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1a040d52d0 0x7f1a0406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:05.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.573+0000 7f1a03e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a040d3080 con 0x7f1a04161e90 2024-04-16T16:15:05.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.573+0000 7f1a03e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1a040d2e30 con 0x7f1a040d52d0 2024-04-16T16:15:05.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.573+0000 7f1a03e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1a040d3360 con 0x7f1a04162270 2024-04-16T16:15:05.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.574+0000 7f1a03400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1a040d52d0 0x7f1a0406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:05.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.574+0000 7f1a03400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1a040d52d0 0x7f1a0406ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60168/0 (socket says 172.21.15.45:60168) 2024-04-16T16:15:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.574+0000 7f1a03400640 1 -- 172.21.15.45:0/2761420382 learned_addr learned my addr 172.21.15.45:0/2761420382 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.574+0000 7f19fbe00640 1 --2- 172.21.15.45:0/2761420382 >> v2:172.21.15.103:3300/0 conn(0x7f1a04162270 0x7f1a040d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.574+0000 7f1a03400640 1 -- 172.21.15.45:0/2761420382 >> v2:172.21.15.103:3300/0 conn(0x7f1a04162270 msgr2=0x7f1a040d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.574+0000 7f1a03400640 1 --2- 172.21.15.45:0/2761420382 >> v2:172.21.15.103:3300/0 conn(0x7f1a04162270 0x7f1a040d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.574+0000 7f1a03400640 1 -- 172.21.15.45:0/2761420382 >> v1:172.21.15.103:6789/0 conn(0x7f1a04161e90 legacy=0x7f1a040cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.574+0000 7f1a03400640 1 -- 172.21.15.45:0/2761420382 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a040d3640 con 0x7f1a040d52d0 2024-04-16T16:15:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.574+0000 7f19fbe00640 1 --2- 172.21.15.45:0/2761420382 >> v2:172.21.15.103:3300/0 conn(0x7f1a04162270 0x7f1a040d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:05.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.575+0000 7f1a03400640 1 --2- 172.21.15.45:0/2761420382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a040d52d0 0x7f1a0406ce40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009e50 tx=0x7f19fc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d14cbf26c84418 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:05.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.576+0000 7f1a02000640 1 -- 172.21.15.45:0/2761420382 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19fc02ca80 con 0x7f1a040d52d0 2024-04-16T16:15:05.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.576+0000 7f1a02000640 1 -- 172.21.15.45:0/2761420382 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19fc02cc20 con 0x7f1a040d52d0 2024-04-16T16:15:05.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.576+0000 7f1a02000640 1 -- 172.21.15.45:0/2761420382 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19fc0343f0 con 0x7f1a040d52d0 2024-04-16T16:15:05.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.576+0000 7f1a03e00640 1 -- 172.21.15.45:0/2761420382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a040d52d0 msgr2=0x7f1a0406ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:05.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.576+0000 7f1a03e00640 1 --2- 172.21.15.45:0/2761420382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a040d52d0 0x7f1a0406ce40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009e50 tx=0x7f19fc02bee0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.576+0000 7f1a03e00640 1 -- 172.21.15.45:0/2761420382 shutdown_connections 2024-04-16T16:15:05.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.576+0000 7f1a03e00640 1 --2- 172.21.15.45:0/2761420382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a040d52d0 0x7f1a0406ce40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.577+0000 7f1a03e00640 1 --2- 172.21.15.45:0/2761420382 >> v2:172.21.15.103:3300/0 conn(0x7f1a04162270 0x7f1a040d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.577+0000 7f1a03e00640 1 -- 172.21.15.45:0/2761420382 >> 172.21.15.45:0/2761420382 conn(0x7f1a040d1940 msgr2=0x7f1a040d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:05.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.577+0000 7f1a03e00640 1 -- 172.21.15.45:0/2761420382 shutdown_connections 2024-04-16T16:15:05.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.577+0000 7f1a03e00640 1 -- 172.21.15.45:0/2761420382 wait complete. 2024-04-16T16:15:05.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.579+0000 7f1a03e00640 1 Processor -- start 2024-04-16T16:15:05.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.579+0000 7f1a03e00640 1 -- start start 2024-04-16T16:15:05.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.580+0000 7f1a03e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a040d52d0 0x7f1a0414dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:05.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.580+0000 7f1a03e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a04161e90 0x7f1a041e7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:05.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.581+0000 7f1a03e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a04162270 0x7f1a0414e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:05.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.581+0000 7f1a03e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a040d4500 con 0x7f1a04162270 2024-04-16T16:15:05.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.581+0000 7f1a03e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a040d4380 con 0x7f1a04161e90 2024-04-16T16:15:05.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.581+0000 7f1a03e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1a040d4680 con 0x7f1a040d52d0 2024-04-16T16:15:05.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.581+0000 7f1a03400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a04162270 0x7f1a0414e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:05.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.581+0000 7f1a03400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a04162270 0x7f1a0414e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60170/0 (socket says 172.21.15.45:60170) 2024-04-16T16:15:05.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.581+0000 7f1a03400640 1 -- 172.21.15.45:0/788892244 learned_addr learned my addr 172.21.15.45:0/788892244 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:05.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.581+0000 7f1a02a00640 1 --2- 172.21.15.45:0/788892244 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a040d52d0 0x7f1a0414dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:05.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.581+0000 7f19fbe00640 1 --2- 172.21.15.45:0/788892244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a04161e90 0x7f1a041e7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:05.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.581+0000 7f1a03400640 1 -- 172.21.15.45:0/788892244 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a040d52d0 msgr2=0x7f1a0414dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:05.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.581+0000 7f1a03400640 1 --2- 172.21.15.45:0/788892244 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a040d52d0 0x7f1a0414dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.581+0000 7f1a03400640 1 -- 172.21.15.45:0/788892244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a04161e90 msgr2=0x7f1a041e7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:05.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.581+0000 7f1a03400640 1 --2- 172.21.15.45:0/788892244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a04161e90 0x7f1a041e7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.581+0000 7f1a03400640 1 -- 172.21.15.45:0/788892244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a040ccfa0 con 0x7f1a04162270 2024-04-16T16:15:05.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.582+0000 7f1a03400640 1 --2- 172.21.15.45:0/788892244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a04162270 0x7f1a0414e5f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009e50 tx=0x7f19fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:05.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.582+0000 7f19faa00640 1 -- 172.21.15.45:0/788892244 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19fc043020 con 0x7f1a04162270 2024-04-16T16:15:05.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.582+0000 7f19faa00640 1 -- 172.21.15.45:0/788892244 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19fc03e450 con 0x7f1a04162270 2024-04-16T16:15:05.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.582+0000 7f19faa00640 1 -- 172.21.15.45:0/788892244 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19fc03e820 con 0x7f1a04162270 2024-04-16T16:15:05.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.582+0000 7f1a03e00640 1 -- 172.21.15.45:0/788892244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a040cd280 con 0x7f1a04162270 2024-04-16T16:15:05.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.582+0000 7f1a03e00640 1 -- 172.21.15.45:0/788892244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a040cd840 con 0x7f1a04162270 2024-04-16T16:15:05.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.583+0000 7f1a03e00640 1 -- 172.21.15.45:0/788892244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a04075780 con 0x7f1a04162270 2024-04-16T16:15:05.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.584+0000 7f19faa00640 1 -- 172.21.15.45:0/788892244 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f19fc046020 con 0x7f1a04162270 2024-04-16T16:15:05.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.587+0000 7f19faa00640 1 --2- 172.21.15.45:0/788892244 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f19e006ca60 0x7f19e006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:05.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.587+0000 7f19faa00640 1 -- 172.21.15.45:0/788892244 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5967+0+0 (secure 0 0 0) 0x7f19fc0ae760 con 0x7f1a04162270 2024-04-16T16:15:05.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.587+0000 7f1a02a00640 1 --2- 172.21.15.45:0/788892244 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f19e006ca60 0x7f19e006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:05.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.589+0000 7f1a02a00640 1 --2- 172.21.15.45:0/788892244 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f19e006ca60 0x7f19e006ef20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f19f4007fd0 tx=0x7f19f4009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:05.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.593+0000 7f19faa00640 1 -- 172.21.15.45:0/788892244 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f19fc04d030 con 0x7f1a04162270 2024-04-16T16:15:05.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.843+0000 7f1a03e00640 1 -- 172.21.15.45:0/788892244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a0406dae0 con 0x7f1a04162270 2024-04-16T16:15:05.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.844+0000 7f19faa00640 1 -- 172.21.15.45:0/788892244 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7f19fc07c8e0 con 0x7f1a04162270 2024-04-16T16:15:05.844 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:05.844 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/814837411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":814837411},{"type":"v1","addr":"172.21.15.103:6836","nonce":814837411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/401733314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":401733314},{"type":"v1","addr":"172.21.15.103:6837","nonce":401733314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5073,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/768816269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":768816269},{"type":"v1","addr":"172.21.15.45:6834","nonce":768816269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:14:39.132000+0000","modified":"2024-04-16T16:14:42.226029+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5079},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5079":{"gid":5079,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6835/2956262988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2956262988},{"type":"v1","addr":"172.21.15.45:6835","nonce":2956262988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5079,"qdb_cluster":[5079]},"id":2}]} 2024-04-16T16:15:05.844 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 16 2024-04-16T16:15:05.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.848+0000 7f1a03e00640 1 -- 172.21.15.45:0/788892244 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f19e006ca60 msgr2=0x7f19e006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:05.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.848+0000 7f1a03e00640 1 --2- 172.21.15.45:0/788892244 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f19e006ca60 0x7f19e006ef20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f19f4007fd0 tx=0x7f19f4009040 comp rx=0 tx=0).stop 2024-04-16T16:15:05.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.848+0000 7f1a03e00640 1 -- 172.21.15.45:0/788892244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a04162270 msgr2=0x7f1a0414e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:05.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.848+0000 7f1a03e00640 1 --2- 172.21.15.45:0/788892244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a04162270 0x7f1a0414e5f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009e50 tx=0x7f19fc004a40 comp rx=0 tx=0).stop 2024-04-16T16:15:05.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.848+0000 7f1a03e00640 1 -- 172.21.15.45:0/788892244 shutdown_connections 2024-04-16T16:15:05.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.848+0000 7f1a03e00640 1 --2- 172.21.15.45:0/788892244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a04162270 0x7f1a0414e5f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.848+0000 7f1a03e00640 1 --2- 172.21.15.45:0/788892244 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f19e006ca60 0x7f19e006ef20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.848+0000 7f1a03e00640 1 --2- 172.21.15.45:0/788892244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a04161e90 0x7f1a041e7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.848+0000 7f1a03e00640 1 --2- 172.21.15.45:0/788892244 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a040d52d0 0x7f1a0414dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:05.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.848+0000 7f1a03e00640 1 -- 172.21.15.45:0/788892244 >> 172.21.15.45:0/788892244 conn(0x7f1a040d1940 msgr2=0x7f1a040b8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:05.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.849+0000 7f1a03e00640 1 -- 172.21.15.45:0/788892244 shutdown_connections 2024-04-16T16:15:05.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:05.849+0000 7f1a03e00640 1 -- 172.21.15.45:0/788892244 wait complete. 2024-04-16T16:15:05.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:15:06.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.082+0000 7f6534a00640 1 Processor -- start 2024-04-16T16:15:06.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.082+0000 7f6534a00640 1 -- start start 2024-04-16T16:15:06.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.084+0000 7f6534a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f653015ccb0 0x7f653006df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:06.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.084+0000 7f6534a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f653015d360 con 0x7f653006e480 2024-04-16T16:15:06.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.084+0000 7f6534a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f653015d640 con 0x7f6530146220 2024-04-16T16:15:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.084+0000 7f6534a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f653015d130 con 0x7f653015ccb0 2024-04-16T16:15:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.084+0000 7f652fe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f653006e480 0x7f65301608a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48730/0 (socket says 172.21.15.45:48730) 2024-04-16T16:15:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.084+0000 7f652fe00640 1 -- 172.21.15.45:0/2963047674 learned_addr learned my addr 172.21.15.45:0/2963047674 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:06.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652ea00640 1 --2- 172.21.15.45:0/2963047674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f653015ccb0 0x7f653006df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:06.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652e000640 1 -- 172.21.15.45:0/2963047674 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3349632421 0 0) 0x7f653015d360 con 0x7f653006e480 2024-04-16T16:15:06.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652e000640 1 -- 172.21.15.45:0/2963047674 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6514003520 con 0x7f653006e480 2024-04-16T16:15:06.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652e000640 1 -- 172.21.15.45:0/2963047674 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 648260320 0 0) 0x7f6514003520 con 0x7f653006e480 2024-04-16T16:15:06.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652e000640 1 -- 172.21.15.45:0/2963047674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f653015ccb0 msgr2=0x7f653006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:06.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652e000640 1 --2- 172.21.15.45:0/2963047674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f653015ccb0 0x7f653006df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652e000640 1 -- 172.21.15.45:0/2963047674 >> v1:172.21.15.103:6789/0 conn(0x7f6530146220 legacy=0x7f653006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652e000640 1 -- 172.21.15.45:0/2963047674 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653015d920 con 0x7f653006e480 2024-04-16T16:15:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652e000640 1 -- 172.21.15.45:0/2963047674 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f65280019f0 con 0x7f653006e480 2024-04-16T16:15:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652e000640 1 -- 172.21.15.45:0/2963047674 >> v1:172.21.15.45:6789/0 conn(0x7f653006e480 legacy=0x7f65301608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652e000640 1 --2- 172.21.15.45:0/2963047674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65140039e0 0x7f6514003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652e000640 1 --2- 172.21.15.45:0/2963047674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6514004800 0x7f6514004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652e000640 1 --2- 172.21.15.45:0/2963047674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f653015ccb0 0x7f65140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652e000640 1 -- 172.21.15.45:0/2963047674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f651400ba70 con 0x7f65140039e0 2024-04-16T16:15:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652e000640 1 -- 172.21.15.45:0/2963047674 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f651400bbf0 con 0x7f6514004800 2024-04-16T16:15:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.085+0000 7f652e000640 1 -- 172.21.15.45:0/2963047674 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f651400bdf0 con 0x7f653015ccb0 2024-04-16T16:15:06.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.086+0000 7f652f400640 1 --2- 172.21.15.45:0/2963047674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65140039e0 0x7f6514003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:06.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.086+0000 7f652ea00640 1 --2- 172.21.15.45:0/2963047674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6514004800 0x7f6514004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:06.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.086+0000 7f652fe00640 1 --2- 172.21.15.45:0/2963047674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f653015ccb0 0x7f65140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:06.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.086+0000 7f652f400640 1 -- 172.21.15.45:0/2963047674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f653015ccb0 msgr2=0x7f65140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:06.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.086+0000 7f652f400640 1 --2- 172.21.15.45:0/2963047674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f653015ccb0 0x7f65140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.086+0000 7f652f400640 1 -- 172.21.15.45:0/2963047674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6514004800 msgr2=0x7f6514004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:06.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.086+0000 7f652f400640 1 --2- 172.21.15.45:0/2963047674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6514004800 0x7f6514004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.086+0000 7f652f400640 1 -- 172.21.15.45:0/2963047674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6514003520 con 0x7f65140039e0 2024-04-16T16:15:06.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.087+0000 7f652f400640 1 --2- 172.21.15.45:0/2963047674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65140039e0 0x7f6514003e00 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6520001720 tx=0x7f6514005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737612a2cbac8acc server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:06.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.088+0000 7f652e000640 1 -- 172.21.15.45:0/2963047674 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6520001370 con 0x7f65140039e0 2024-04-16T16:15:06.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.088+0000 7f652e000640 1 -- 172.21.15.45:0/2963047674 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65200052d0 con 0x7f65140039e0 2024-04-16T16:15:06.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.088+0000 7f652e000640 1 -- 172.21.15.45:0/2963047674 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6520003830 con 0x7f65140039e0 2024-04-16T16:15:06.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.088+0000 7f6534a00640 1 -- 172.21.15.45:0/2963047674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65140039e0 msgr2=0x7f6514003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:06.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.088+0000 7f6534a00640 1 --2- 172.21.15.45:0/2963047674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65140039e0 0x7f6514003e00 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6520001720 tx=0x7f6514005830 comp rx=0 tx=0).stop 2024-04-16T16:15:06.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.088+0000 7f652fe00640 1 -- 172.21.15.45:0/2963047674 reap_dead start 2024-04-16T16:15:06.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.089+0000 7f6534a00640 1 -- 172.21.15.45:0/2963047674 shutdown_connections 2024-04-16T16:15:06.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.089+0000 7f6534a00640 1 -- 172.21.15.45:0/2963047674 >> 172.21.15.45:0/2963047674 conn(0x7f653006b930 msgr2=0x7f653006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:06.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.089+0000 7f6534a00640 1 -- 172.21.15.45:0/2963047674 shutdown_connections 2024-04-16T16:15:06.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.089+0000 7f6534a00640 1 -- 172.21.15.45:0/2963047674 wait complete. 2024-04-16T16:15:06.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.090+0000 7f6534a00640 1 Processor -- start 2024-04-16T16:15:06.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.091+0000 7f6534a00640 1 -- start start 2024-04-16T16:15:06.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.091+0000 7f6534a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65140039e0 0x7f653013d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:06.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.091+0000 7f6534a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f653015c050 0x7f65301e8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:06.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.092+0000 7f6534a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65301e8550 0x7f65301f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:06.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.092+0000 7f6534a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6530160fb0 con 0x7f65301e8550 2024-04-16T16:15:06.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.092+0000 7f6534a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6530160410 con 0x7f653015c050 2024-04-16T16:15:06.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.092+0000 7f6534a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6530161130 con 0x7f65140039e0 2024-04-16T16:15:06.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.092+0000 7f652fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65301e8550 0x7f65301f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:06.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.092+0000 7f652fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65301e8550 0x7f65301f3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60182/0 (socket says 172.21.15.45:60182) 2024-04-16T16:15:06.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.092+0000 7f652fe00640 1 -- 172.21.15.45:0/124412209 learned_addr learned my addr 172.21.15.45:0/124412209 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:06.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.092+0000 7f652f400640 1 --2- 172.21.15.45:0/124412209 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65140039e0 0x7f653013d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:06.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.092+0000 7f652ea00640 1 --2- 172.21.15.45:0/124412209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f653015c050 0x7f65301e8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:06.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.092+0000 7f652fe00640 1 -- 172.21.15.45:0/124412209 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65140039e0 msgr2=0x7f653013d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:06.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.092+0000 7f652fe00640 1 --2- 172.21.15.45:0/124412209 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65140039e0 0x7f653013d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.092+0000 7f652fe00640 1 -- 172.21.15.45:0/124412209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f653015c050 msgr2=0x7f65301e8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:06.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.092+0000 7f652fe00640 1 --2- 172.21.15.45:0/124412209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f653015c050 0x7f65301e8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.092+0000 7f652fe00640 1 -- 172.21.15.45:0/124412209 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653013db40 con 0x7f65301e8550 2024-04-16T16:15:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.092+0000 7f652ea00640 1 --2- 172.21.15.45:0/124412209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f653015c050 0x7f65301e8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.093+0000 7f652fe00640 1 --2- 172.21.15.45:0/124412209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65301e8550 0x7f65301f3ac0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f65300b37b0 tx=0x7f652800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.093+0000 7f6527400640 1 -- 172.21.15.45:0/124412209 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652800ebb0 con 0x7f65301e8550 2024-04-16T16:15:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.093+0000 7f6527400640 1 -- 172.21.15.45:0/124412209 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6528018070 con 0x7f65301e8550 2024-04-16T16:15:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.093+0000 7f6527400640 1 -- 172.21.15.45:0/124412209 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6528013420 con 0x7f65301e8550 2024-04-16T16:15:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.093+0000 7f6534a00640 1 -- 172.21.15.45:0/124412209 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f653013de20 con 0x7f65301e8550 2024-04-16T16:15:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.093+0000 7f6534a00640 1 -- 172.21.15.45:0/124412209 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65301e6220 con 0x7f65301e8550 2024-04-16T16:15:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.095+0000 7f6534a00640 1 -- 172.21.15.45:0/124412209 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65301e6420 con 0x7f65301e8550 2024-04-16T16:15:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.096+0000 7f6527400640 1 -- 172.21.15.45:0/124412209 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f65280135c0 con 0x7f65301e8550 2024-04-16T16:15:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.096+0000 7f6527400640 1 --2- 172.21.15.45:0/124412209 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f651006ca90 0x7f651006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.096+0000 7f6527400640 1 -- 172.21.15.45:0/124412209 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5967+0+0 (secure 0 0 0) 0x7f652808c1b0 con 0x7f65301e8550 2024-04-16T16:15:06.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.097+0000 7f652f400640 1 --2- 172.21.15.45:0/124412209 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f651006ca90 0x7f651006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:06.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.098+0000 7f652f400640 1 --2- 172.21.15.45:0/124412209 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f651006ca90 0x7f651006ef50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6520033000 tx=0x7f6520009030 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:06.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.102+0000 7f6527400640 1 -- 172.21.15.45:0/124412209 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f65301e6420 con 0x7f65301e8550 2024-04-16T16:15:06.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.341+0000 7f6534a00640 1 -- 172.21.15.45:0/124412209 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6530075780 con 0x7f65301e8550 2024-04-16T16:15:06.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.343+0000 7f6527400640 1 -- 172.21.15.45:0/124412209 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7f652805a330 con 0x7f65301e8550 2024-04-16T16:15:06.342 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:06.342 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/814837411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":814837411},{"type":"v1","addr":"172.21.15.103:6836","nonce":814837411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/401733314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":401733314},{"type":"v1","addr":"172.21.15.103:6837","nonce":401733314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5073,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/768816269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":768816269},{"type":"v1","addr":"172.21.15.45:6834","nonce":768816269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:14:39.132000+0000","modified":"2024-04-16T16:14:42.226029+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5079},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5079":{"gid":5079,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6835/2956262988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2956262988},{"type":"v1","addr":"172.21.15.45:6835","nonce":2956262988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5079,"qdb_cluster":[5079]},"id":2}]} 2024-04-16T16:15:06.342 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 16 2024-04-16T16:15:06.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.345+0000 7f6534a00640 1 -- 172.21.15.45:0/124412209 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f651006ca90 msgr2=0x7f651006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:06.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.345+0000 7f6534a00640 1 --2- 172.21.15.45:0/124412209 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f651006ca90 0x7f651006ef50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6520033000 tx=0x7f6520009030 comp rx=0 tx=0).stop 2024-04-16T16:15:06.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.345+0000 7f6534a00640 1 -- 172.21.15.45:0/124412209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65301e8550 msgr2=0x7f65301f3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:06.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.345+0000 7f6534a00640 1 --2- 172.21.15.45:0/124412209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65301e8550 0x7f65301f3ac0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f65300b37b0 tx=0x7f652800e530 comp rx=0 tx=0).stop 2024-04-16T16:15:06.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.345+0000 7f6534a00640 1 -- 172.21.15.45:0/124412209 shutdown_connections 2024-04-16T16:15:06.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.345+0000 7f6534a00640 1 --2- 172.21.15.45:0/124412209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65301e8550 0x7f65301f3ac0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.345+0000 7f6534a00640 1 --2- 172.21.15.45:0/124412209 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f651006ca90 0x7f651006ef50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.345+0000 7f6534a00640 1 --2- 172.21.15.45:0/124412209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f653015c050 0x7f65301e8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.345+0000 7f6534a00640 1 --2- 172.21.15.45:0/124412209 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65140039e0 0x7f653013d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.345+0000 7f6534a00640 1 -- 172.21.15.45:0/124412209 >> 172.21.15.45:0/124412209 conn(0x7f653006b930 msgr2=0x7f65300cf360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:06.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.345+0000 7f6534a00640 1 -- 172.21.15.45:0/124412209 shutdown_connections 2024-04-16T16:15:06.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.345+0000 7f6534a00640 1 -- 172.21.15.45:0/124412209 wait complete. 2024-04-16T16:15:06.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:15:06.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.579+0000 7fdb4ac00640 1 Processor -- start 2024-04-16T16:15:06.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.579+0000 7fdb4ac00640 1 -- start start 2024-04-16T16:15:06.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.580+0000 7fdb4ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb440d52d0 0x7fdb4406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:06.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.580+0000 7fdb4ac00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb440d3080 con 0x7fdb44162270 2024-04-16T16:15:06.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.580+0000 7fdb4ac00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb440d3360 con 0x7fdb44161e90 2024-04-16T16:15:06.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.580+0000 7fdb4ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdb440d2e30 con 0x7fdb440d52d0 2024-04-16T16:15:06.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.581+0000 7fdb48e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdb44162270 0x7fdb440d3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48736/0 (socket says 172.21.15.45:48736) 2024-04-16T16:15:06.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.581+0000 7fdb48e00640 1 -- 172.21.15.45:0/1453003911 learned_addr learned my addr 172.21.15.45:0/1453003911 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.581+0000 7fdb4a200640 1 --2- 172.21.15.45:0/1453003911 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb440d52d0 0x7fdb4406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.581+0000 7fdb43e00640 1 -- 172.21.15.45:0/1453003911 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2392822759 0 0) 0x7fdb440d3080 con 0x7fdb44162270 2024-04-16T16:15:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.582+0000 7fdb43e00640 1 -- 172.21.15.45:0/1453003911 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb2c003520 con 0x7fdb44162270 2024-04-16T16:15:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.582+0000 7fdb43e00640 1 -- 172.21.15.45:0/1453003911 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2731000964 0 0) 0x7fdb440d3360 con 0x7fdb44161e90 2024-04-16T16:15:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.582+0000 7fdb43e00640 1 -- 172.21.15.45:0/1453003911 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb440d3080 con 0x7fdb44161e90 2024-04-16T16:15:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.582+0000 7fdb4a200640 1 -- 172.21.15.45:0/1453003911 >> v1:172.21.15.103:6789/0 conn(0x7fdb44161e90 legacy=0x7fdb440cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.582+0000 7fdb43e00640 1 -- 172.21.15.45:0/1453003911 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 876606478 0 0) 0x7fdb2c003520 con 0x7fdb44162270 2024-04-16T16:15:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.582+0000 7fdb4a200640 1 -- 172.21.15.45:0/1453003911 >> v1:172.21.15.45:6789/0 conn(0x7fdb44162270 legacy=0x7fdb440d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.582+0000 7fdb4a200640 1 -- 172.21.15.45:0/1453003911 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb440d3640 con 0x7fdb440d52d0 2024-04-16T16:15:06.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.583+0000 7fdb4a200640 1 --2- 172.21.15.45:0/1453003911 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb440d52d0 0x7fdb4406ce40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009df0 tx=0x7fdb3c02be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10aabeb7b0f35a4a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:06.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.584+0000 7fdb43e00640 1 -- 172.21.15.45:0/1453003911 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c039070 con 0x7fdb440d52d0 2024-04-16T16:15:06.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.584+0000 7fdb43e00640 1 -- 172.21.15.45:0/1453003911 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb3c02c9a0 con 0x7fdb440d52d0 2024-04-16T16:15:06.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.584+0000 7fdb43e00640 1 -- 172.21.15.45:0/1453003911 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb34001670 con 0x7fdb440d52d0 2024-04-16T16:15:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.585+0000 7fdb4ac00640 1 -- 172.21.15.45:0/1453003911 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb440d52d0 msgr2=0x7fdb4406ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.585+0000 7fdb4ac00640 1 --2- 172.21.15.45:0/1453003911 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb440d52d0 0x7fdb4406ce40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009df0 tx=0x7fdb3c02be70 comp rx=0 tx=0).stop 2024-04-16T16:15:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.585+0000 7fdb4ac00640 1 -- 172.21.15.45:0/1453003911 shutdown_connections 2024-04-16T16:15:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.585+0000 7fdb4ac00640 1 --2- 172.21.15.45:0/1453003911 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb440d52d0 0x7fdb4406ce40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.585+0000 7fdb4ac00640 1 -- 172.21.15.45:0/1453003911 >> 172.21.15.45:0/1453003911 conn(0x7fdb440d1940 msgr2=0x7fdb440d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.585+0000 7fdb4ac00640 1 -- 172.21.15.45:0/1453003911 shutdown_connections 2024-04-16T16:15:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.585+0000 7fdb4ac00640 1 -- 172.21.15.45:0/1453003911 wait complete. 2024-04-16T16:15:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.586+0000 7fdb4ac00640 1 Processor -- start 2024-04-16T16:15:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.587+0000 7fdb4ac00640 1 -- start start 2024-04-16T16:15:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.588+0000 7fdb4ac00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb440d52d0 0x7fdb4414fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.588+0000 7fdb4ac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb44161e90 0x7fdb4415e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.588+0000 7fdb4ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb44162270 0x7fdb4415ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.588+0000 7fdb4ac00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdb440d46d0 con 0x7fdb440d52d0 2024-04-16T16:15:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.588+0000 7fdb4ac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb440d4550 con 0x7fdb44161e90 2024-04-16T16:15:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.588+0000 7fdb4ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdb4406d380 con 0x7fdb44162270 2024-04-16T16:15:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.588+0000 7fdb49800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb440d52d0 0x7fdb4414fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.588+0000 7fdb49800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb440d52d0 0x7fdb4414fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60190/0 (socket says 172.21.15.45:60190) 2024-04-16T16:15:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.588+0000 7fdb49800640 1 -- 172.21.15.45:0/3267075302 learned_addr learned my addr 172.21.15.45:0/3267075302 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:06.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.588+0000 7fdb48e00640 1 --2- 172.21.15.45:0/3267075302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb44161e90 0x7fdb4415e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.588+0000 7fdb49800640 1 -- 172.21.15.45:0/3267075302 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb44162270 msgr2=0x7fdb4415ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.588+0000 7fdb49800640 1 --2- 172.21.15.45:0/3267075302 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb44162270 0x7fdb4415ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.588+0000 7fdb49800640 1 -- 172.21.15.45:0/3267075302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb44161e90 msgr2=0x7fdb4415e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.588+0000 7fdb49800640 1 --2- 172.21.15.45:0/3267075302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb44161e90 0x7fdb4415e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.589+0000 7fdb49800640 1 -- 172.21.15.45:0/3267075302 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb440cacc0 con 0x7fdb440d52d0 2024-04-16T16:15:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.589+0000 7fdb49800640 1 --2- 172.21.15.45:0/3267075302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb440d52d0 0x7fdb4414fd50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdb3800c500 tx=0x7fdb3800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.589+0000 7fdb41600640 1 -- 172.21.15.45:0/3267075302 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb380052d0 con 0x7fdb440d52d0 2024-04-16T16:15:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.589+0000 7fdb41600640 1 -- 172.21.15.45:0/3267075302 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb38005490 con 0x7fdb440d52d0 2024-04-16T16:15:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.589+0000 7fdb4ac00640 1 -- 172.21.15.45:0/3267075302 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb440c8c40 con 0x7fdb440d52d0 2024-04-16T16:15:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.589+0000 7fdb41600640 1 -- 172.21.15.45:0/3267075302 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb38012870 con 0x7fdb440d52d0 2024-04-16T16:15:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.589+0000 7fdb4ac00640 1 -- 172.21.15.45:0/3267075302 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb440c8ee0 con 0x7fdb440d52d0 2024-04-16T16:15:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.590+0000 7fdb4ac00640 1 -- 172.21.15.45:0/3267075302 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb440c9b60 con 0x7fdb440d52d0 2024-04-16T16:15:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.592+0000 7fdb41600640 1 -- 172.21.15.45:0/3267075302 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fdb38012a10 con 0x7fdb440d52d0 2024-04-16T16:15:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.593+0000 7fdb41600640 1 --2- 172.21.15.45:0/3267075302 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdb2806ca60 0x7fdb2806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:06.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.593+0000 7fdb41600640 1 -- 172.21.15.45:0/3267075302 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5967+0+0 (secure 0 0 0) 0x7fdb3808bb40 con 0x7fdb440d52d0 2024-04-16T16:15:06.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.593+0000 7fdb48e00640 1 --2- 172.21.15.45:0/3267075302 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdb2806ca60 0x7fdb2806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.595+0000 7fdb48e00640 1 --2- 172.21.15.45:0/3267075302 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdb2806ca60 0x7fdb2806ef20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdb340011e0 tx=0x7fdb34009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.597+0000 7fdb41600640 1 -- 172.21.15.45:0/3267075302 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb440c9b60 con 0x7fdb440d52d0 2024-04-16T16:15:06.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.803+0000 7fdb4ac00640 1 -- 172.21.15.45:0/3267075302 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdb4406dae0 con 0x7fdb440d52d0 2024-04-16T16:15:06.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.804+0000 7fdb41600640 1 -- 172.21.15.45:0/3267075302 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+17052 (secure 0 0 0) 0x7fdb38059cd0 con 0x7fdb440d52d0 2024-04-16T16:15:06.804 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:06.804 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":35,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:15:01.083563+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:14:37.171798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-16T16:14:38.122496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:15:06.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.807+0000 7fdb4ac00640 1 -- 172.21.15.45:0/3267075302 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdb2806ca60 msgr2=0x7fdb2806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:06.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.807+0000 7fdb4ac00640 1 --2- 172.21.15.45:0/3267075302 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdb2806ca60 0x7fdb2806ef20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdb340011e0 tx=0x7fdb34009040 comp rx=0 tx=0).stop 2024-04-16T16:15:06.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.807+0000 7fdb4ac00640 1 -- 172.21.15.45:0/3267075302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb440d52d0 msgr2=0x7fdb4414fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:06.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.807+0000 7fdb4ac00640 1 --2- 172.21.15.45:0/3267075302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb440d52d0 0x7fdb4414fd50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdb3800c500 tx=0x7fdb3800c9c0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.807+0000 7fdb4ac00640 1 -- 172.21.15.45:0/3267075302 shutdown_connections 2024-04-16T16:15:06.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.807+0000 7fdb4ac00640 1 --2- 172.21.15.45:0/3267075302 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb44162270 0x7fdb4415ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.807+0000 7fdb4ac00640 1 --2- 172.21.15.45:0/3267075302 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdb2806ca60 0x7fdb2806ef20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.807+0000 7fdb4ac00640 1 --2- 172.21.15.45:0/3267075302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb44161e90 0x7fdb4415e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.807+0000 7fdb4ac00640 1 --2- 172.21.15.45:0/3267075302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb440d52d0 0x7fdb4414fd50 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:06.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.807+0000 7fdb4ac00640 1 -- 172.21.15.45:0/3267075302 >> 172.21.15.45:0/3267075302 conn(0x7fdb440d1940 msgr2=0x7fdb440b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:06.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.807+0000 7fdb4ac00640 1 -- 172.21.15.45:0/3267075302 shutdown_connections 2024-04-16T16:15:06.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:06.807+0000 7fdb4ac00640 1 -- 172.21.15.45:0/3267075302 wait complete. 2024-04-16T16:15:06.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-16T16:15:07.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.051+0000 7f2a6ac00640 1 Processor -- start 2024-04-16T16:15:07.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.051+0000 7f2a6ac00640 1 -- start start 2024-04-16T16:15:07.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.053+0000 7f2a6ac00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2a640cbc80 0x7f2a640cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:07.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.053+0000 7f2a6ac00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a6415f1b0 con 0x7f2a6415e830 2024-04-16T16:15:07.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.053+0000 7f2a6ac00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a640cf280 con 0x7f2a640cb7d0 2024-04-16T16:15:07.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.053+0000 7f2a6ac00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2a640ce190 con 0x7f2a640cbc80 2024-04-16T16:15:07.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.053+0000 7f2a6a200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2a640cbc80 0x7f2a640cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:07.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.053+0000 7f2a69800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2a6415e830 0x7f2a640ce9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48746/0 (socket says 172.21.15.45:48746) 2024-04-16T16:15:07.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.053+0000 7f2a69800640 1 -- 172.21.15.45:0/4192866338 learned_addr learned my addr 172.21.15.45:0/4192866338 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:07.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.054+0000 7f2a6a200640 1 -- 172.21.15.45:0/4192866338 >> v1:172.21.15.103:6789/0 conn(0x7f2a640cb7d0 legacy=0x7f2a640cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:07.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.054+0000 7f2a6a200640 1 -- 172.21.15.45:0/4192866338 >> v1:172.21.15.45:6789/0 conn(0x7f2a6415e830 legacy=0x7f2a640ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:07.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.054+0000 7f2a6a200640 1 -- 172.21.15.45:0/4192866338 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a640ce470 con 0x7f2a640cbc80 2024-04-16T16:15:07.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.054+0000 7f2a63e00640 1 -- 172.21.15.45:0/4192866338 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1356008630 0 0) 0x7f2a58001110 con 0x7f2a6415e830 2024-04-16T16:15:07.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.055+0000 7f2a6a200640 1 --2- 172.21.15.45:0/4192866338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640cbc80 0x7f2a640cc410 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c009d90 tx=0x7f2a5c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9c11650e0519bc server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:07.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.055+0000 7f2a63e00640 1 -- 172.21.15.45:0/4192866338 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c039070 con 0x7f2a640cbc80 2024-04-16T16:15:07.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.055+0000 7f2a63e00640 1 -- 172.21.15.45:0/4192866338 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a5c02cbd0 con 0x7f2a640cbc80 2024-04-16T16:15:07.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.055+0000 7f2a63e00640 1 -- 172.21.15.45:0/4192866338 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c034440 con 0x7f2a640cbc80 2024-04-16T16:15:07.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.056+0000 7f2a6ac00640 1 -- 172.21.15.45:0/4192866338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640cbc80 msgr2=0x7f2a640cc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:07.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.056+0000 7f2a6ac00640 1 --2- 172.21.15.45:0/4192866338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640cbc80 0x7f2a640cc410 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c009d90 tx=0x7f2a5c02be20 comp rx=0 tx=0).stop 2024-04-16T16:15:07.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.056+0000 7f2a6ac00640 1 -- 172.21.15.45:0/4192866338 shutdown_connections 2024-04-16T16:15:07.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.056+0000 7f2a6ac00640 1 --2- 172.21.15.45:0/4192866338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640cbc80 0x7f2a640cc410 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:07.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.056+0000 7f2a6ac00640 1 -- 172.21.15.45:0/4192866338 >> 172.21.15.45:0/4192866338 conn(0x7f2a6406b930 msgr2=0x7f2a6406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:07.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.056+0000 7f2a6ac00640 1 -- 172.21.15.45:0/4192866338 shutdown_connections 2024-04-16T16:15:07.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.056+0000 7f2a6ac00640 1 -- 172.21.15.45:0/4192866338 wait complete. 2024-04-16T16:15:07.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.058+0000 7f2a6ac00640 1 Processor -- start 2024-04-16T16:15:07.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.058+0000 7f2a6ac00640 1 -- start start 2024-04-16T16:15:07.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.059+0000 7f2a6ac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a640cb7d0 0x7f2a64146320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:07.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.059+0000 7f2a6ac00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640cbc80 0x7f2a64159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:07.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.059+0000 7f2a6ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a6415e830 0x7f2a64154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.059+0000 7f2a6ac00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6415ec10 con 0x7f2a640cbc80 2024-04-16T16:15:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.059+0000 7f2a6ac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2a640cb2b0 con 0x7f2a640cb7d0 2024-04-16T16:15:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.059+0000 7f2a6ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2a640cac30 con 0x7f2a6415e830 2024-04-16T16:15:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.059+0000 7f2a68e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640cbc80 0x7f2a64159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.059+0000 7f2a68e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640cbc80 0x7f2a64159d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60214/0 (socket says 172.21.15.45:60214) 2024-04-16T16:15:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.059+0000 7f2a68e00640 1 -- 172.21.15.45:0/2529596643 learned_addr learned my addr 172.21.15.45:0/2529596643 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.059+0000 7f2a68e00640 1 -- 172.21.15.45:0/2529596643 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a6415e830 msgr2=0x7f2a64154900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.059+0000 7f2a68e00640 1 --2- 172.21.15.45:0/2529596643 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a6415e830 0x7f2a64154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.059+0000 7f2a68e00640 1 -- 172.21.15.45:0/2529596643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a640cb7d0 msgr2=0x7f2a64146320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.059+0000 7f2a69800640 1 --2- 172.21.15.45:0/2529596643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a640cb7d0 0x7f2a64146320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.059+0000 7f2a68e00640 1 --2- 172.21.15.45:0/2529596643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a640cb7d0 0x7f2a64146320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.059+0000 7f2a68e00640 1 -- 172.21.15.45:0/2529596643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6415a260 con 0x7f2a640cbc80 2024-04-16T16:15:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.060+0000 7f2a68e00640 1 --2- 172.21.15.45:0/2529596643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640cbc80 0x7f2a64159d20 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000b3f0 tx=0x7f2a5000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:07.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.060+0000 7f2a61600640 1 -- 172.21.15.45:0/2529596643 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50015020 con 0x7f2a640cbc80 2024-04-16T16:15:07.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.060+0000 7f2a61600640 1 -- 172.21.15.45:0/2529596643 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a50011830 con 0x7f2a640cbc80 2024-04-16T16:15:07.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.060+0000 7f2a61600640 1 -- 172.21.15.45:0/2529596643 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50011c20 con 0x7f2a640cbc80 2024-04-16T16:15:07.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.060+0000 7f2a6ac00640 1 -- 172.21.15.45:0/2529596643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a64158c80 con 0x7f2a640cbc80 2024-04-16T16:15:07.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.060+0000 7f2a6ac00640 1 -- 172.21.15.45:0/2529596643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a64159240 con 0x7f2a640cbc80 2024-04-16T16:15:07.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.062+0000 7f2a6ac00640 1 -- 172.21.15.45:0/2529596643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a64161e60 con 0x7f2a640cbc80 2024-04-16T16:15:07.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.062+0000 7f2a61600640 1 -- 172.21.15.45:0/2529596643 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f2a50018020 con 0x7f2a640cbc80 2024-04-16T16:15:07.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.063+0000 7f2a61600640 1 --2- 172.21.15.45:0/2529596643 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2a4406ca60 0x7f2a4406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:07.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.063+0000 7f2a61600640 1 -- 172.21.15.45:0/2529596643 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5967+0+0 (secure 0 0 0) 0x7f2a5008ba00 con 0x7f2a640cbc80 2024-04-16T16:15:07.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.064+0000 7f2a69800640 1 --2- 172.21.15.45:0/2529596643 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2a4406ca60 0x7f2a4406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:07.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.065+0000 7f2a69800640 1 --2- 172.21.15.45:0/2529596643 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2a4406ca60 0x7f2a4406ef20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2a58000f60 tx=0x7f2a58009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:07.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.068+0000 7f2a61600640 1 -- 172.21.15.45:0/2529596643 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2a64161e60 con 0x7f2a640cbc80 2024-04-16T16:15:07.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.317+0000 7f2a6ac00640 1 -- 172.21.15.45:0/2529596643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f2a640b5b90 con 0x7f2a640cbc80 2024-04-16T16:15:07.317 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-04-16T16:15:07.318+0000 7fe7a4800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T16:15:07.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.322+0000 7f2a61600640 1 -- 172.21.15.45:0/2529596643 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) ==== 161+0+0 (secure 0 0 0) 0x7f2a5001f030 con 0x7f2a640cbc80 2024-04-16T16:15:07.321 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-16T16:15:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.325+0000 7f2a6ac00640 1 -- 172.21.15.45:0/2529596643 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2a4406ca60 msgr2=0x7f2a4406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.325+0000 7f2a6ac00640 1 --2- 172.21.15.45:0/2529596643 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2a4406ca60 0x7f2a4406ef20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2a58000f60 tx=0x7f2a58009040 comp rx=0 tx=0).stop 2024-04-16T16:15:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.325+0000 7f2a6ac00640 1 -- 172.21.15.45:0/2529596643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640cbc80 msgr2=0x7f2a64159d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.325+0000 7f2a6ac00640 1 --2- 172.21.15.45:0/2529596643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640cbc80 0x7f2a64159d20 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000b3f0 tx=0x7f2a5000b8b0 comp rx=0 tx=0).stop 2024-04-16T16:15:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.325+0000 7f2a6ac00640 1 -- 172.21.15.45:0/2529596643 shutdown_connections 2024-04-16T16:15:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.325+0000 7f2a6ac00640 1 --2- 172.21.15.45:0/2529596643 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a6415e830 0x7f2a64154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.325+0000 7f2a6ac00640 1 --2- 172.21.15.45:0/2529596643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640cbc80 0x7f2a64159d20 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.325+0000 7f2a6ac00640 1 --2- 172.21.15.45:0/2529596643 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2a4406ca60 0x7f2a4406ef20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.325+0000 7f2a6ac00640 1 --2- 172.21.15.45:0/2529596643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a640cb7d0 0x7f2a64146320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.325+0000 7f2a6ac00640 1 -- 172.21.15.45:0/2529596643 >> 172.21.15.45:0/2529596643 conn(0x7f2a6406b930 msgr2=0x7f2a640b4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.326+0000 7f2a6ac00640 1 -- 172.21.15.45:0/2529596643 shutdown_connections 2024-04-16T16:15:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.326+0000 7f2a6ac00640 1 -- 172.21.15.45:0/2529596643 wait complete. 2024-04-16T16:15:07.344 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-16T16:15:07.348 INFO:tasks.ceph.mds.a.smithi045.stderr: -3077> 2024-04-16T16:14:42.538+0000 7fdea9200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:15:07.415 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2024-04-16T16:15:07.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.538+0000 7fa23f800640 1 Processor -- start 2024-04-16T16:15:07.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.538+0000 7fa23f800640 1 -- start start 2024-04-16T16:15:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.539+0000 7fa23f800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2380afda0 0x7fa2380b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.540+0000 7fa23f800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2380ccb70 con 0x7fa23815b920 2024-04-16T16:15:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.540+0000 7fa23f800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2380cce50 con 0x7fa2380d0e80 2024-04-16T16:15:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.540+0000 7fa23f800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa23806e7a0 con 0x7fa2380afda0 2024-04-16T16:15:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.540+0000 7fa23ce00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa23815b920 0x7fa2380d0770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48758/0 (socket says 172.21.15.45:48758) 2024-04-16T16:15:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.540+0000 7fa23ce00640 1 -- 172.21.15.45:0/837884929 learned_addr learned my addr 172.21.15.45:0/837884929 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:07.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.540+0000 7fa23d800640 1 --2- 172.21.15.45:0/837884929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2380afda0 0x7fa2380b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:07.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.540+0000 7fa237400640 1 -- 172.21.15.45:0/837884929 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 679415099 0 0) 0x7fa2380ccb70 con 0x7fa23815b920 2024-04-16T16:15:07.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.540+0000 7fa237400640 1 -- 172.21.15.45:0/837884929 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa220005730 con 0x7fa23815b920 2024-04-16T16:15:07.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.541+0000 7fa237400640 1 -- 172.21.15.45:0/837884929 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 610424114 0 0) 0x7fa220005730 con 0x7fa23815b920 2024-04-16T16:15:07.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.541+0000 7fa237400640 1 -- 172.21.15.45:0/837884929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2380afda0 msgr2=0x7fa2380b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:07.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.541+0000 7fa237400640 1 --2- 172.21.15.45:0/837884929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2380afda0 0x7fa2380b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.541+0000 7fa237400640 1 -- 172.21.15.45:0/837884929 >> v1:172.21.15.103:6789/0 conn(0x7fa2380d0e80 legacy=0x7fa2380ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.541+0000 7fa237400640 1 -- 172.21.15.45:0/837884929 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa23806e430 con 0x7fa23815b920 2024-04-16T16:15:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.541+0000 7fa237400640 1 -- 172.21.15.45:0/837884929 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fa22c001690 con 0x7fa23815b920 2024-04-16T16:15:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.541+0000 7fa237400640 1 -- 172.21.15.45:0/837884929 >> v1:172.21.15.45:6789/0 conn(0x7fa23815b920 legacy=0x7fa2380d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.541+0000 7fa237400640 1 --2- 172.21.15.45:0/837884929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2380afda0 0x7fa220005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.541+0000 7fa237400640 1 --2- 172.21.15.45:0/837884929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa220006170 0x7fa220006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.541+0000 7fa237400640 1 --2- 172.21.15.45:0/837884929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa220006f50 0x7fa2200073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.541+0000 7fa237400640 1 -- 172.21.15.45:0/837884929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa23806e7a0 con 0x7fa220006170 2024-04-16T16:15:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.541+0000 7fa237400640 1 -- 172.21.15.45:0/837884929 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa22000dcb0 con 0x7fa220006f50 2024-04-16T16:15:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.541+0000 7fa237400640 1 -- 172.21.15.45:0/837884929 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa22000deb0 con 0x7fa2380afda0 2024-04-16T16:15:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.541+0000 7fa237e00640 1 --2- 172.21.15.45:0/837884929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa220006170 0x7fa220006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.542+0000 7fa23ce00640 1 --2- 172.21.15.45:0/837884929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2380afda0 0x7fa220005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.542+0000 7fa237e00640 1 -- 172.21.15.45:0/837884929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2380afda0 msgr2=0x7fa220005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:07.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.542+0000 7fa237e00640 1 --2- 172.21.15.45:0/837884929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2380afda0 0x7fa220005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:07.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.542+0000 7fa237e00640 1 -- 172.21.15.45:0/837884929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa220006f50 msgr2=0x7fa2200073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:07.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.542+0000 7fa237e00640 1 --2- 172.21.15.45:0/837884929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa220006f50 0x7fa2200073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:07.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.542+0000 7fa237e00640 1 -- 172.21.15.45:0/837884929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa220005730 con 0x7fa220006170 2024-04-16T16:15:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.543+0000 7fa237e00640 1 --2- 172.21.15.45:0/837884929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa220006170 0x7fa220006a10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa228001720 tx=0x7fa220007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f78895ce74a3c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.543+0000 7fa237400640 1 -- 172.21.15.45:0/837884929 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa228001370 con 0x7fa220006170 2024-04-16T16:15:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.543+0000 7fa237400640 1 -- 172.21.15.45:0/837884929 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2280052d0 con 0x7fa220006170 2024-04-16T16:15:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.543+0000 7fa237400640 1 -- 172.21.15.45:0/837884929 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa228003830 con 0x7fa220006170 2024-04-16T16:15:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.544+0000 7fa23f800640 1 -- 172.21.15.45:0/837884929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa220006170 msgr2=0x7fa220006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.544+0000 7fa23f800640 1 --2- 172.21.15.45:0/837884929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa220006170 0x7fa220006a10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa228001720 tx=0x7fa220007a40 comp rx=0 tx=0).stop 2024-04-16T16:15:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.544+0000 7fa23d800640 1 -- 172.21.15.45:0/837884929 reap_dead start 2024-04-16T16:15:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.544+0000 7fa23f800640 1 -- 172.21.15.45:0/837884929 shutdown_connections 2024-04-16T16:15:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.544+0000 7fa23f800640 1 -- 172.21.15.45:0/837884929 >> 172.21.15.45:0/837884929 conn(0x7fa2380b3970 msgr2=0x7fa2380b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.544+0000 7fa23f800640 1 -- 172.21.15.45:0/837884929 shutdown_connections 2024-04-16T16:15:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.544+0000 7fa23f800640 1 -- 172.21.15.45:0/837884929 wait complete. 2024-04-16T16:15:07.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.546+0000 7fa23f800640 1 Processor -- start 2024-04-16T16:15:07.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.546+0000 7fa23f800640 1 -- start start 2024-04-16T16:15:07.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.547+0000 7fa23f800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa220006170 0x7fa2381f7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:07.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.547+0000 7fa23f800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2380b1f70 0x7fa2380b23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:07.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.547+0000 7fa23f800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2380b0550 0x7fa2380b09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:07.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.547+0000 7fa23f800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa2380b51b0 con 0x7fa220006170 2024-04-16T16:15:07.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.547+0000 7fa23f800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa2380b5030 con 0x7fa2380b0550 2024-04-16T16:15:07.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.547+0000 7fa23f800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa23806ec30 con 0x7fa2380b1f70 2024-04-16T16:15:07.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.548+0000 7fa23ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa220006170 0x7fa2381f7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:07.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.548+0000 7fa23ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa220006170 0x7fa2381f7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60250/0 (socket says 172.21.15.45:60250) 2024-04-16T16:15:07.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.548+0000 7fa23ce00640 1 -- 172.21.15.45:0/2541234048 learned_addr learned my addr 172.21.15.45:0/2541234048 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:07.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.548+0000 7fa23d800640 1 --2- 172.21.15.45:0/2541234048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2380b0550 0x7fa2380b09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:07.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.548+0000 7fa23ce00640 1 -- 172.21.15.45:0/2541234048 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2380b1f70 msgr2=0x7fa2380b23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:07.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.548+0000 7fa237e00640 1 --2- 172.21.15.45:0/2541234048 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2380b1f70 0x7fa2380b23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:07.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.548+0000 7fa23ce00640 1 --2- 172.21.15.45:0/2541234048 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2380b1f70 0x7fa2380b23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:07.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.548+0000 7fa23ce00640 1 -- 172.21.15.45:0/2541234048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2380b0550 msgr2=0x7fa2380b09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:07.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.548+0000 7fa23ce00640 1 --2- 172.21.15.45:0/2541234048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2380b0550 0x7fa2380b09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:07.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.548+0000 7fa23ce00640 1 -- 172.21.15.45:0/2541234048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa23815c6e0 con 0x7fa220006170 2024-04-16T16:15:07.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.548+0000 7fa23d800640 1 --2- 172.21.15.45:0/2541234048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2380b0550 0x7fa2380b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:07.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.549+0000 7fa23ce00640 1 --2- 172.21.15.45:0/2541234048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa220006170 0x7fa2381f7600 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa22c00e760 tx=0x7fa22c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:07.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.549+0000 7fa234c00640 1 -- 172.21.15.45:0/2541234048 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22c00ee70 con 0x7fa220006170 2024-04-16T16:15:07.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.549+0000 7fa234c00640 1 -- 172.21.15.45:0/2541234048 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa22c018070 con 0x7fa220006170 2024-04-16T16:15:07.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.549+0000 7fa234c00640 1 -- 172.21.15.45:0/2541234048 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22c00ca50 con 0x7fa220006170 2024-04-16T16:15:07.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.549+0000 7fa23f800640 1 -- 172.21.15.45:0/2541234048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2380b0f10 con 0x7fa220006170 2024-04-16T16:15:07.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.549+0000 7fa23f800640 1 -- 172.21.15.45:0/2541234048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2380b1480 con 0x7fa220006170 2024-04-16T16:15:07.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.550+0000 7fa23f800640 1 -- 172.21.15.45:0/2541234048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa238144a90 con 0x7fa220006170 2024-04-16T16:15:07.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.551+0000 7fa234c00640 1 -- 172.21.15.45:0/2541234048 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fa22c00cbf0 con 0x7fa220006170 2024-04-16T16:15:07.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.552+0000 7fa234c00640 1 --2- 172.21.15.45:0/2541234048 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa21c06ca90 0x7fa21c06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:07.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.552+0000 7fa234c00640 1 -- 172.21.15.45:0/2541234048 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6053+0+0 (secure 0 0 0) 0x7fa22c08b940 con 0x7fa220006170 2024-04-16T16:15:07.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.553+0000 7fa237e00640 1 --2- 172.21.15.45:0/2541234048 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa21c06ca90 0x7fa21c06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:07.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.554+0000 7fa237e00640 1 --2- 172.21.15.45:0/2541234048 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa21c06ca90 0x7fa21c06ef50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa228033000 tx=0x7fa228009030 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:07.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.557+0000 7fa234c00640 1 -- 172.21.15.45:0/2541234048 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa23815c6e0 con 0x7fa220006170 2024-04-16T16:15:07.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:07.805+0000 7fa23f800640 1 -- 172.21.15.45:0/2541234048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fa238075780 con 0x7fa220006170 2024-04-16T16:15:08.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.325+0000 7fa234c00640 1 -- 172.21.15.45:0/2541234048 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) ==== 104+0+0 (secure 0 0 0) 0x7fa22c059af0 con 0x7fa220006170 2024-04-16T16:15:08.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.329+0000 7fa23f800640 1 -- 172.21.15.45:0/2541234048 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa21c06ca90 msgr2=0x7fa21c06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:08.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.329+0000 7fa23f800640 1 --2- 172.21.15.45:0/2541234048 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa21c06ca90 0x7fa21c06ef50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa228033000 tx=0x7fa228009030 comp rx=0 tx=0).stop 2024-04-16T16:15:08.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.329+0000 7fa23f800640 1 -- 172.21.15.45:0/2541234048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa220006170 msgr2=0x7fa2381f7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:08.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.329+0000 7fa23f800640 1 --2- 172.21.15.45:0/2541234048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa220006170 0x7fa2381f7600 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa22c00e760 tx=0x7fa22c00ec20 comp rx=0 tx=0).stop 2024-04-16T16:15:08.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.329+0000 7fa23f800640 1 -- 172.21.15.45:0/2541234048 shutdown_connections 2024-04-16T16:15:08.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.329+0000 7fa23f800640 1 --2- 172.21.15.45:0/2541234048 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa21c06ca90 0x7fa21c06ef50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:08.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.329+0000 7fa23f800640 1 --2- 172.21.15.45:0/2541234048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2380b0550 0x7fa2380b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:08.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.329+0000 7fa23f800640 1 --2- 172.21.15.45:0/2541234048 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2380b1f70 0x7fa2380b23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:08.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.329+0000 7fa23f800640 1 --2- 172.21.15.45:0/2541234048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa220006170 0x7fa2381f7600 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:08.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.329+0000 7fa23f800640 1 -- 172.21.15.45:0/2541234048 >> 172.21.15.45:0/2541234048 conn(0x7fa2380b3970 msgr2=0x7fa23806f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:08.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.330+0000 7fa23f800640 1 -- 172.21.15.45:0/2541234048 shutdown_connections 2024-04-16T16:15:08.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.330+0000 7fa23f800640 1 -- 172.21.15.45:0/2541234048 wait complete. 2024-04-16T16:15:08.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-16T16:15:08.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.577+0000 7f18dee00640 1 Processor -- start 2024-04-16T16:15:08.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.577+0000 7f18dee00640 1 -- start start 2024-04-16T16:15:08.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.579+0000 7f18dee00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f18d80b2f60 0x7f18d815f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:08.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.579+0000 7f18dee00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18d80b21a0 con 0x7f18d815f760 2024-04-16T16:15:08.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.579+0000 7f18dee00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18d80b2480 con 0x7f18d806e430 2024-04-16T16:15:08.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.579+0000 7f18dee00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f18d80b2760 con 0x7f18d80b2f60 2024-04-16T16:15:08.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.579+0000 7f18de400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f18d815f760 0x7f18d80addb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48772/0 (socket says 172.21.15.45:48772) 2024-04-16T16:15:08.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.579+0000 7f18dd000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f18d80b2f60 0x7f18d815f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:08.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.579+0000 7f18de400640 1 -- 172.21.15.45:0/3304092233 learned_addr learned my addr 172.21.15.45:0/3304092233 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:08.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.580+0000 7f18dd000640 1 -- 172.21.15.45:0/3304092233 >> v1:172.21.15.103:6789/0 conn(0x7f18d806e430 legacy=0x7f18d806e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:08.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.580+0000 7f18dd000640 1 -- 172.21.15.45:0/3304092233 >> v1:172.21.15.45:6789/0 conn(0x7f18d815f760 legacy=0x7f18d80addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:08.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.580+0000 7f18dd000640 1 -- 172.21.15.45:0/3304092233 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d80b2a40 con 0x7f18d80b2f60 2024-04-16T16:15:08.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.580+0000 7f18d7e00640 1 -- 172.21.15.45:0/3304092233 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3710689373 0 0) 0x7f18d80b21a0 con 0x7f18d815f760 2024-04-16T16:15:08.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.581+0000 7f18dd000640 1 --2- 172.21.15.45:0/3304092233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18d80b2f60 0x7f18d815f220 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f18c40097a0 tx=0x7f18c402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce37f619bf1f93e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:08.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.581+0000 7f18d7e00640 1 -- 172.21.15.45:0/3304092233 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4034070 con 0x7f18d80b2f60 2024-04-16T16:15:08.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.581+0000 7f18d7e00640 1 -- 172.21.15.45:0/3304092233 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18c4030610 con 0x7f18d80b2f60 2024-04-16T16:15:08.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.581+0000 7f18d7e00640 1 -- 172.21.15.45:0/3304092233 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c40309e0 con 0x7f18d80b2f60 2024-04-16T16:15:08.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.582+0000 7f18dee00640 1 -- 172.21.15.45:0/3304092233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18d80b2f60 msgr2=0x7f18d815f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:08.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.582+0000 7f18dee00640 1 --2- 172.21.15.45:0/3304092233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18d80b2f60 0x7f18d815f220 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f18c40097a0 tx=0x7f18c402b960 comp rx=0 tx=0).stop 2024-04-16T16:15:08.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.582+0000 7f18dee00640 1 -- 172.21.15.45:0/3304092233 shutdown_connections 2024-04-16T16:15:08.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.582+0000 7f18dee00640 1 --2- 172.21.15.45:0/3304092233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18d80b2f60 0x7f18d815f220 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:08.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.582+0000 7f18dee00640 1 -- 172.21.15.45:0/3304092233 >> 172.21.15.45:0/3304092233 conn(0x7f18d80b3bb0 msgr2=0x7f18d80b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:08.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.582+0000 7f18dee00640 1 -- 172.21.15.45:0/3304092233 shutdown_connections 2024-04-16T16:15:08.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.582+0000 7f18dee00640 1 -- 172.21.15.45:0/3304092233 wait complete. 2024-04-16T16:15:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.583+0000 7f18dee00640 1 Processor -- start 2024-04-16T16:15:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.584+0000 7f18dee00640 1 -- start start 2024-04-16T16:15:08.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.585+0000 7f18dee00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18d806e430 0x7f18d81436e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:08.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.585+0000 7f18dee00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18d80b2f60 0x7f18d81456d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:08.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.585+0000 7f18dee00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18d815f760 0x7f18d815e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:08.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.585+0000 7f18dee00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f18d80b1f40 con 0x7f18d80b2f60 2024-04-16T16:15:08.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.585+0000 7f18dee00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18d80b1dc0 con 0x7f18d806e430 2024-04-16T16:15:08.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.585+0000 7f18dee00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f18d80ae4c0 con 0x7f18d815f760 2024-04-16T16:15:08.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.585+0000 7f18dd000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18d80b2f60 0x7f18d81456d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:08.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.585+0000 7f18dd000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18d80b2f60 0x7f18d81456d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60268/0 (socket says 172.21.15.45:60268) 2024-04-16T16:15:08.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.585+0000 7f18dd000640 1 -- 172.21.15.45:0/3746156092 learned_addr learned my addr 172.21.15.45:0/3746156092 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:08.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.585+0000 7f18dd000640 1 -- 172.21.15.45:0/3746156092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18d815f760 msgr2=0x7f18d815e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:08.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.585+0000 7f18dd000640 1 --2- 172.21.15.45:0/3746156092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18d815f760 0x7f18d815e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:08.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.585+0000 7f18dd000640 1 -- 172.21.15.45:0/3746156092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18d806e430 msgr2=0x7f18d81436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:08.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.585+0000 7f18dd000640 1 --2- 172.21.15.45:0/3746156092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18d806e430 0x7f18d81436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:08.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.585+0000 7f18dd000640 1 -- 172.21.15.45:0/3746156092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d8145c10 con 0x7f18d80b2f60 2024-04-16T16:15:08.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.586+0000 7f18dd000640 1 --2- 172.21.15.45:0/3746156092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18d80b2f60 0x7f18d81456d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f18c4000c00 tx=0x7f18c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:08.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.586+0000 7f18d5600640 1 -- 172.21.15.45:0/3746156092 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4034040 con 0x7f18d80b2f60 2024-04-16T16:15:08.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.586+0000 7f18d5600640 1 -- 172.21.15.45:0/3746156092 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18c4004030 con 0x7f18d80b2f60 2024-04-16T16:15:08.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.586+0000 7f18d5600640 1 -- 172.21.15.45:0/3746156092 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4004420 con 0x7f18d80b2f60 2024-04-16T16:15:08.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.586+0000 7f18dee00640 1 -- 172.21.15.45:0/3746156092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18d815e5c0 con 0x7f18d80b2f60 2024-04-16T16:15:08.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.586+0000 7f18dee00640 1 -- 172.21.15.45:0/3746156092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18d8144030 con 0x7f18d80b2f60 2024-04-16T16:15:08.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.587+0000 7f18dee00640 1 -- 172.21.15.45:0/3746156092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18d8075780 con 0x7f18d80b2f60 2024-04-16T16:15:08.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.588+0000 7f18d5600640 1 -- 172.21.15.45:0/3746156092 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f18c404c060 con 0x7f18d80b2f60 2024-04-16T16:15:08.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.589+0000 7f18d5600640 1 --2- 172.21.15.45:0/3746156092 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f18b806ca60 0x7f18b806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:08.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.589+0000 7f18d5600640 1 -- 172.21.15.45:0/3746156092 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6053+0+0 (secure 0 0 0) 0x7f18c40adc10 con 0x7f18d80b2f60 2024-04-16T16:15:08.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.590+0000 7f18dda00640 1 --2- 172.21.15.45:0/3746156092 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f18b806ca60 0x7f18b806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:08.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.591+0000 7f18dda00640 1 --2- 172.21.15.45:0/3746156092 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f18b806ca60 0x7f18b806ef20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f18c8000f60 tx=0x7f18c80079b0 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:08.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.594+0000 7f18d5600640 1 -- 172.21.15.45:0/3746156092 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f18c407bcb0 con 0x7f18d80b2f60 2024-04-16T16:15:08.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:08.800+0000 7f18dee00640 1 -- 172.21.15.45:0/3746156092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f18d80adb90 con 0x7f18d80b2f60 2024-04-16T16:15:09.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.336+0000 7f18d5600640 1 -- 172.21.15.45:0/3746156092 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v37) ==== 181+0+0 (secure 0 0 0) 0x7f18c407b5b0 con 0x7f18d80b2f60 2024-04-16T16:15:09.336 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2024-04-16T16:15:09.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.339+0000 7f18dee00640 1 -- 172.21.15.45:0/3746156092 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f18b806ca60 msgr2=0x7f18b806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:09.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.339+0000 7f18dee00640 1 --2- 172.21.15.45:0/3746156092 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f18b806ca60 0x7f18b806ef20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f18c8000f60 tx=0x7f18c80079b0 comp rx=0 tx=0).stop 2024-04-16T16:15:09.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.339+0000 7f18dee00640 1 -- 172.21.15.45:0/3746156092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18d80b2f60 msgr2=0x7f18d81456d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:09.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.339+0000 7f18dee00640 1 --2- 172.21.15.45:0/3746156092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18d80b2f60 0x7f18d81456d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f18c4000c00 tx=0x7f18c40046c0 comp rx=0 tx=0).stop 2024-04-16T16:15:09.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.340+0000 7f18dee00640 1 -- 172.21.15.45:0/3746156092 shutdown_connections 2024-04-16T16:15:09.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.340+0000 7f18dee00640 1 --2- 172.21.15.45:0/3746156092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18d815f760 0x7f18d815e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:09.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.340+0000 7f18dee00640 1 --2- 172.21.15.45:0/3746156092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18d80b2f60 0x7f18d81456d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:09.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.340+0000 7f18dee00640 1 --2- 172.21.15.45:0/3746156092 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f18b806ca60 0x7f18b806ef20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:09.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.340+0000 7f18dee00640 1 --2- 172.21.15.45:0/3746156092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18d806e430 0x7f18d81436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:09.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.340+0000 7f18dee00640 1 -- 172.21.15.45:0/3746156092 >> 172.21.15.45:0/3746156092 conn(0x7f18d80b3bb0 msgr2=0x7f18d81476d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:09.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.340+0000 7f18dee00640 1 -- 172.21.15.45:0/3746156092 shutdown_connections 2024-04-16T16:15:09.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.340+0000 7f18dee00640 1 -- 172.21.15.45:0/3746156092 wait complete. 2024-04-16T16:15:09.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-16T16:15:09.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.590+0000 7f14ee400640 1 Processor -- start 2024-04-16T16:15:09.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.590+0000 7f14ee400640 1 -- start start 2024-04-16T16:15:09.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.592+0000 7f14ee400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f14e80d0050 0x7f14e80b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:09.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.592+0000 7f14ee400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14e806eaf0 0x7f14e815f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:09.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.592+0000 7f14ee400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14e80d0430 con 0x7f14e80cd8a0 2024-04-16T16:15:09.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.592+0000 7f14ee400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f14e815bbd0 con 0x7f14e80d0050 2024-04-16T16:15:09.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.592+0000 7f14ee400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f14e80b0fd0 con 0x7f14e806eaf0 2024-04-16T16:15:09.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.592+0000 7f14e7e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f14e80cd8a0 0x7f14e806e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48784/0 (socket says 172.21.15.45:48784) 2024-04-16T16:15:09.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.592+0000 7f14e7e00640 1 -- 172.21.15.45:0/1205429866 learned_addr learned my addr 172.21.15.45:0/1205429866 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:09.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.593+0000 7f14ed000640 1 --2- 172.21.15.45:0/1205429866 >> v2:172.21.15.103:3300/0 conn(0x7f14e80d0050 0x7f14e80b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:09.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.593+0000 7f14eda00640 1 --2- 172.21.15.45:0/1205429866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14e806eaf0 0x7f14e815f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:09.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.593+0000 7f14e7400640 1 -- 172.21.15.45:0/1205429866 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1450945902 0 0) 0x7f14e80d0430 con 0x7f14e80cd8a0 2024-04-16T16:15:09.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.593+0000 7f14e7400640 1 -- 172.21.15.45:0/1205429866 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14d0003520 con 0x7f14e80cd8a0 2024-04-16T16:15:09.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.593+0000 7f14e7400640 1 -- 172.21.15.45:0/1205429866 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 295866372 0 0) 0x7f14d0003520 con 0x7f14e80cd8a0 2024-04-16T16:15:09.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.593+0000 7f14e7400640 1 -- 172.21.15.45:0/1205429866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14e806eaf0 msgr2=0x7f14e815f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:09.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.593+0000 7f14e7400640 1 --2- 172.21.15.45:0/1205429866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14e806eaf0 0x7f14e815f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:09.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7400640 1 -- 172.21.15.45:0/1205429866 >> v2:172.21.15.103:3300/0 conn(0x7f14e80d0050 msgr2=0x7f14e80b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:09.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7400640 1 --2- 172.21.15.45:0/1205429866 >> v2:172.21.15.103:3300/0 conn(0x7f14e80d0050 0x7f14e80b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:09.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7400640 1 -- 172.21.15.45:0/1205429866 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e815be90 con 0x7f14e80cd8a0 2024-04-16T16:15:09.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7400640 1 -- 172.21.15.45:0/1205429866 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f14d4001670 con 0x7f14e80cd8a0 2024-04-16T16:15:09.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7400640 1 -- 172.21.15.45:0/1205429866 >> v1:172.21.15.45:6789/0 conn(0x7f14e80cd8a0 legacy=0x7f14e806e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:09.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7400640 1 --2- 172.21.15.45:0/1205429866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14e806eaf0 0x7f14d00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:09.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7400640 1 --2- 172.21.15.45:0/1205429866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d0003ef0 0x7f14d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:09.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7400640 1 --2- 172.21.15.45:0/1205429866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d0004370 0x7f14d0004d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:09.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7400640 1 -- 172.21.15.45:0/1205429866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f14e815bbd0 con 0x7f14d0003ef0 2024-04-16T16:15:09.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7400640 1 -- 172.21.15.45:0/1205429866 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f14e80b0fd0 con 0x7f14d0004370 2024-04-16T16:15:09.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7400640 1 -- 172.21.15.45:0/1205429866 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f14d0005600 con 0x7f14e806eaf0 2024-04-16T16:15:09.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7e00640 1 --2- 172.21.15.45:0/1205429866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d0003ef0 0x7f14d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:09.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7e00640 1 -- 172.21.15.45:0/1205429866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14e806eaf0 msgr2=0x7f14d00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:09.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7e00640 1 --2- 172.21.15.45:0/1205429866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14e806eaf0 0x7f14d00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:09.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7e00640 1 -- 172.21.15.45:0/1205429866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d0004370 msgr2=0x7f14d0004d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:09.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7e00640 1 --2- 172.21.15.45:0/1205429866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d0004370 0x7f14d0004d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:09.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.594+0000 7f14e7e00640 1 -- 172.21.15.45:0/1205429866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d0003520 con 0x7f14d0003ef0 2024-04-16T16:15:09.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.597+0000 7f14e7e00640 1 --2- 172.21.15.45:0/1205429866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d0003ef0 0x7f14d0004800 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f14d400c190 tx=0x7f14d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf51d3d2cd2002b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:09.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.597+0000 7f14e7400640 1 -- 172.21.15.45:0/1205429866 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d4007c80 con 0x7f14d0003ef0 2024-04-16T16:15:09.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.597+0000 7f14e7400640 1 -- 172.21.15.45:0/1205429866 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14d4030070 con 0x7f14d0003ef0 2024-04-16T16:15:09.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.597+0000 7f14e7400640 1 -- 172.21.15.45:0/1205429866 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d4005d70 con 0x7f14d0003ef0 2024-04-16T16:15:09.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.598+0000 7f14ee400640 1 -- 172.21.15.45:0/1205429866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d0003ef0 msgr2=0x7f14d0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:09.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.598+0000 7f14ee400640 1 --2- 172.21.15.45:0/1205429866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d0003ef0 0x7f14d0004800 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f14d400c190 tx=0x7f14d0005830 comp rx=0 tx=0).stop 2024-04-16T16:15:09.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.598+0000 7f14eda00640 1 -- 172.21.15.45:0/1205429866 reap_dead start 2024-04-16T16:15:09.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.598+0000 7f14ee400640 1 -- 172.21.15.45:0/1205429866 shutdown_connections 2024-04-16T16:15:09.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.598+0000 7f14ee400640 1 -- 172.21.15.45:0/1205429866 >> 172.21.15.45:0/1205429866 conn(0x7f14e80b3bb0 msgr2=0x7f14e80b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:09.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.598+0000 7f14ee400640 1 -- 172.21.15.45:0/1205429866 shutdown_connections 2024-04-16T16:15:09.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.598+0000 7f14ee400640 1 -- 172.21.15.45:0/1205429866 wait complete. 2024-04-16T16:15:09.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.600+0000 7f14ee400640 1 Processor -- start 2024-04-16T16:15:09.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.600+0000 7f14ee400640 1 -- start start 2024-04-16T16:15:09.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.601+0000 7f14ee400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14d0003ef0 0x7f14e8154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:09.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.601+0000 7f14ee400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e81454b0 0x7f14e81ef970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:09.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.601+0000 7f14ee400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14e80b5810 0x7f14e80b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:09.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.601+0000 7f14ee400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f14e815c730 con 0x7f14e81454b0 2024-04-16T16:15:09.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.601+0000 7f14ee400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f14e815c450 con 0x7f14e80b5810 2024-04-16T16:15:09.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.601+0000 7f14ee400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f14e80cdd50 con 0x7f14d0003ef0 2024-04-16T16:15:09.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.601+0000 7f14e7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e81454b0 0x7f14e81ef970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:09.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.601+0000 7f14e7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e81454b0 0x7f14e81ef970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60284/0 (socket says 172.21.15.45:60284) 2024-04-16T16:15:09.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.601+0000 7f14e7e00640 1 -- 172.21.15.45:0/1608065391 learned_addr learned my addr 172.21.15.45:0/1608065391 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:09.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.602+0000 7f14ed000640 1 --2- 172.21.15.45:0/1608065391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14d0003ef0 0x7f14e8154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:09.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.602+0000 7f14eda00640 1 --2- 172.21.15.45:0/1608065391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14e80b5810 0x7f14e80b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:09.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.602+0000 7f14e7e00640 1 -- 172.21.15.45:0/1608065391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14d0003ef0 msgr2=0x7f14e8154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:09.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.602+0000 7f14e7e00640 1 --2- 172.21.15.45:0/1608065391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14d0003ef0 0x7f14e8154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:09.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.602+0000 7f14e7e00640 1 -- 172.21.15.45:0/1608065391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14e80b5810 msgr2=0x7f14e80b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:09.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.602+0000 7f14e7e00640 1 --2- 172.21.15.45:0/1608065391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14e80b5810 0x7f14e80b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:09.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.602+0000 7f14e7e00640 1 -- 172.21.15.45:0/1608065391 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e81547c0 con 0x7f14e81454b0 2024-04-16T16:15:09.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.602+0000 7f14ed000640 1 --2- 172.21.15.45:0/1608065391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14d0003ef0 0x7f14e8154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:09.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.602+0000 7f14e7e00640 1 --2- 172.21.15.45:0/1608065391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e81454b0 0x7f14e81ef970 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f14d4001eb0 tx=0x7f14e815ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:09.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.603+0000 7f14e4c00640 1 -- 172.21.15.45:0/1608065391 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d4004bf0 con 0x7f14e81454b0 2024-04-16T16:15:09.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.603+0000 7f14e4c00640 1 -- 172.21.15.45:0/1608065391 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14d4004d90 con 0x7f14e81454b0 2024-04-16T16:15:09.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.603+0000 7f14e4c00640 1 -- 172.21.15.45:0/1608065391 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d40072b0 con 0x7f14e81454b0 2024-04-16T16:15:09.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.603+0000 7f14ee400640 1 -- 172.21.15.45:0/1608065391 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14e8151d10 con 0x7f14e81454b0 2024-04-16T16:15:09.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.603+0000 7f14ee400640 1 -- 172.21.15.45:0/1608065391 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14e8151230 con 0x7f14e81454b0 2024-04-16T16:15:09.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.604+0000 7f14ee400640 1 -- 172.21.15.45:0/1608065391 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14e8075780 con 0x7f14e81454b0 2024-04-16T16:15:09.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.605+0000 7f14e4c00640 1 -- 172.21.15.45:0/1608065391 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f14d4007450 con 0x7f14e81454b0 2024-04-16T16:15:09.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.606+0000 7f14e4c00640 1 --2- 172.21.15.45:0/1608065391 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f14c806ca60 0x7f14c806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:09.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.606+0000 7f14e4c00640 1 -- 172.21.15.45:0/1608065391 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5615+0+0 (secure 0 0 0) 0x7f14d40ad330 con 0x7f14e81454b0 2024-04-16T16:15:09.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.607+0000 7f14ed000640 1 --2- 172.21.15.45:0/1608065391 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f14c806ca60 0x7f14c806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:09.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.608+0000 7f14ed000640 1 --2- 172.21.15.45:0/1608065391 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f14c806ca60 0x7f14c806ef20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009a20 tx=0x7f14dc008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:09.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.611+0000 7f14e4c00640 1 -- 172.21.15.45:0/1608065391 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f14d407b740 con 0x7f14e81454b0 2024-04-16T16:15:09.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:09.828+0000 7f14ee400640 1 -- 172.21.15.45:0/1608065391 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f14e81efeb0 con 0x7f14e81454b0 2024-04-16T16:15:10.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.348+0000 7f14e4c00640 1 -- 172.21.15.45:0/1608065391 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v38) ==== 169+0+0 (secure 0 0 0) 0x7f14d4034e80 con 0x7f14e81454b0 2024-04-16T16:15:10.348 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2024-04-16T16:15:10.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.352+0000 7f14ee400640 1 -- 172.21.15.45:0/1608065391 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f14c806ca60 msgr2=0x7f14c806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:10.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.352+0000 7f14ee400640 1 --2- 172.21.15.45:0/1608065391 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f14c806ca60 0x7f14c806ef20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009a20 tx=0x7f14dc008040 comp rx=0 tx=0).stop 2024-04-16T16:15:10.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.352+0000 7f14ee400640 1 -- 172.21.15.45:0/1608065391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e81454b0 msgr2=0x7f14e81ef970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:10.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.352+0000 7f14ee400640 1 --2- 172.21.15.45:0/1608065391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e81454b0 0x7f14e81ef970 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f14d4001eb0 tx=0x7f14e815ca10 comp rx=0 tx=0).stop 2024-04-16T16:15:10.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.353+0000 7f14ee400640 1 -- 172.21.15.45:0/1608065391 shutdown_connections 2024-04-16T16:15:10.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.353+0000 7f14ee400640 1 --2- 172.21.15.45:0/1608065391 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f14c806ca60 0x7f14c806ef20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:10.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.353+0000 7f14ee400640 1 --2- 172.21.15.45:0/1608065391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14e80b5810 0x7f14e80b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:10.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.353+0000 7f14ee400640 1 --2- 172.21.15.45:0/1608065391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e81454b0 0x7f14e81ef970 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:10.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.353+0000 7f14ee400640 1 --2- 172.21.15.45:0/1608065391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14d0003ef0 0x7f14e8154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:10.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.353+0000 7f14ee400640 1 -- 172.21.15.45:0/1608065391 >> 172.21.15.45:0/1608065391 conn(0x7f14e80b3bb0 msgr2=0x7f14e80736d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:10.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.355+0000 7f14ee400640 1 -- 172.21.15.45:0/1608065391 shutdown_connections 2024-04-16T16:15:10.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.355+0000 7f14ee400640 1 -- 172.21.15.45:0/1608065391 wait complete. 2024-04-16T16:15:10.379 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-04-16T16:15:10.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.589+0000 7f29b3200640 1 Processor -- start 2024-04-16T16:15:10.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.589+0000 7f29b3200640 1 -- start start 2024-04-16T16:15:10.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.590+0000 7f29b3200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f29b4155ab0 0x7f29b4162ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:10.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.590+0000 7f29b3200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f29b40ba610 0x7f29b415efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:10.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.590+0000 7f29b3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29b40d18c0 0x7f29b40d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:10.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.590+0000 7f29b3200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f29b415ff00 con 0x7f29b4155ab0 2024-04-16T16:15:10.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.590+0000 7f29b3200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f29b4155e90 con 0x7f29b40ba610 2024-04-16T16:15:10.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.590+0000 7f29b3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f29b415f940 con 0x7f29b40d18c0 2024-04-16T16:15:10.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.591+0000 7f29b1e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f29b4155ab0 0x7f29b4162ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:10.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.591+0000 7f29b1e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f29b4155ab0 0x7f29b4162ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60300/0 (socket says 172.21.15.45:60300) 2024-04-16T16:15:10.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.591+0000 7f29b1e00640 1 -- 172.21.15.45:0/3488710319 learned_addr learned my addr 172.21.15.45:0/3488710319 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:10.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.591+0000 7f29b2800640 1 --2- 172.21.15.45:0/3488710319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29b40d18c0 0x7f29b40d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:10.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.591+0000 7f29b1e00640 1 -- 172.21.15.45:0/3488710319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29b40d18c0 msgr2=0x7f29b40d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:10.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.591+0000 7f29b1e00640 1 --2- 172.21.15.45:0/3488710319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29b40d18c0 0x7f29b40d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:10.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.591+0000 7f29b1400640 1 --2- 172.21.15.45:0/3488710319 >> v2:172.21.15.103:3300/0 conn(0x7f29b40ba610 0x7f29b415efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:10.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.591+0000 7f29b1e00640 1 -- 172.21.15.45:0/3488710319 >> v2:172.21.15.103:3300/0 conn(0x7f29b40ba610 msgr2=0x7f29b415efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:10.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.591+0000 7f29b1e00640 1 --2- 172.21.15.45:0/3488710319 >> v2:172.21.15.103:3300/0 conn(0x7f29b40ba610 0x7f29b415efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:10.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.591+0000 7f29b1e00640 1 -- 172.21.15.45:0/3488710319 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b415fc20 con 0x7f29b4155ab0 2024-04-16T16:15:10.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.593+0000 7f29b1e00640 1 --2- 172.21.15.45:0/3488710319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b4155ab0 0x7f29b4162ef0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f29b40b66c0 tx=0x7f29a402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=853c458e5b061941 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:10.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.593+0000 7f29b0a00640 1 -- 172.21.15.45:0/3488710319 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a402c9e0 con 0x7f29b4155ab0 2024-04-16T16:15:10.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.593+0000 7f29b0a00640 1 -- 172.21.15.45:0/3488710319 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29a402cb80 con 0x7f29b4155ab0 2024-04-16T16:15:10.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.593+0000 7f29b0a00640 1 -- 172.21.15.45:0/3488710319 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a40343f0 con 0x7f29b4155ab0 2024-04-16T16:15:10.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.593+0000 7f29b3200640 1 -- 172.21.15.45:0/3488710319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b4155ab0 msgr2=0x7f29b4162ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:10.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.593+0000 7f29b3200640 1 --2- 172.21.15.45:0/3488710319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b4155ab0 0x7f29b4162ef0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f29b40b66c0 tx=0x7f29a402be40 comp rx=0 tx=0).stop 2024-04-16T16:15:10.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.594+0000 7f29b3200640 1 -- 172.21.15.45:0/3488710319 shutdown_connections 2024-04-16T16:15:10.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.594+0000 7f29b3200640 1 --2- 172.21.15.45:0/3488710319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29b40d18c0 0x7f29b40d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:10.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.594+0000 7f29b3200640 1 --2- 172.21.15.45:0/3488710319 >> v2:172.21.15.103:3300/0 conn(0x7f29b40ba610 0x7f29b415efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:10.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.594+0000 7f29b3200640 1 --2- 172.21.15.45:0/3488710319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b4155ab0 0x7f29b4162ef0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:10.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.594+0000 7f29b3200640 1 -- 172.21.15.45:0/3488710319 >> 172.21.15.45:0/3488710319 conn(0x7f29b406d1a0 msgr2=0x7f29b406d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:10.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.594+0000 7f29b3200640 1 -- 172.21.15.45:0/3488710319 shutdown_connections 2024-04-16T16:15:10.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.594+0000 7f29b3200640 1 -- 172.21.15.45:0/3488710319 wait complete. 2024-04-16T16:15:10.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.595+0000 7f29b3200640 1 Processor -- start 2024-04-16T16:15:10.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.596+0000 7f29b3200640 1 -- start start 2024-04-16T16:15:10.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40ba610 0x7f29b414f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:10.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29b40d18c0 0x7f29b415de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:10.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29b4155ab0 0x7f29b415e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:10.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f29b4163430 con 0x7f29b40ba610 2024-04-16T16:15:10.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f29b41629d0 con 0x7f29b4155ab0 2024-04-16T16:15:10.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f29b415f510 con 0x7f29b40d18c0 2024-04-16T16:15:10.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40ba610 0x7f29b414f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:10.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40ba610 0x7f29b414f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60310/0 (socket says 172.21.15.45:60310) 2024-04-16T16:15:10.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b1e00640 1 -- 172.21.15.45:0/3458975238 learned_addr learned my addr 172.21.15.45:0/3458975238 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:10.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b1e00640 1 -- 172.21.15.45:0/3458975238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29b40d18c0 msgr2=0x7f29b415de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:10.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b1400640 1 --2- 172.21.15.45:0/3458975238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29b40d18c0 0x7f29b415de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:10.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b2800640 1 --2- 172.21.15.45:0/3458975238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29b4155ab0 0x7f29b415e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:10.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b1e00640 1 --2- 172.21.15.45:0/3458975238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29b40d18c0 0x7f29b415de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:10.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b1e00640 1 -- 172.21.15.45:0/3458975238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29b4155ab0 msgr2=0x7f29b415e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:10.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b1e00640 1 --2- 172.21.15.45:0/3458975238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29b4155ab0 0x7f29b415e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:10.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.597+0000 7f29b1e00640 1 -- 172.21.15.45:0/3458975238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b40ccaa0 con 0x7f29b40ba610 2024-04-16T16:15:10.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.598+0000 7f29b1e00640 1 --2- 172.21.15.45:0/3458975238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40ba610 0x7f29b414f930 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f29a4000f80 tx=0x7f29a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:10.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.598+0000 7f29aa000640 1 -- 172.21.15.45:0/3458975238 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a4043020 con 0x7f29b40ba610 2024-04-16T16:15:10.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.598+0000 7f29aa000640 1 -- 172.21.15.45:0/3458975238 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29a403e450 con 0x7f29b40ba610 2024-04-16T16:15:10.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.598+0000 7f29aa000640 1 -- 172.21.15.45:0/3458975238 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a4009510 con 0x7f29b40ba610 2024-04-16T16:15:10.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.598+0000 7f29b3200640 1 -- 172.21.15.45:0/3458975238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b40cba00 con 0x7f29b40ba610 2024-04-16T16:15:10.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.598+0000 7f29b3200640 1 -- 172.21.15.45:0/3458975238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b40cbfc0 con 0x7f29b40ba610 2024-04-16T16:15:10.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.600+0000 7f29b3200640 1 -- 172.21.15.45:0/3458975238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29b4075780 con 0x7f29b40ba610 2024-04-16T16:15:10.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.601+0000 7f29aa000640 1 -- 172.21.15.45:0/3458975238 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f29a4046020 con 0x7f29b40ba610 2024-04-16T16:15:10.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.601+0000 7f29aa000640 1 --2- 172.21.15.45:0/3458975238 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f299406ca60 0x7f299406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:10.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.602+0000 7f29aa000640 1 -- 172.21.15.45:0/3458975238 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5201+0+0 (secure 0 0 0) 0x7f29a40ae410 con 0x7f29b40ba610 2024-04-16T16:15:10.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.602+0000 7f29b1400640 1 --2- 172.21.15.45:0/3458975238 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f299406ca60 0x7f299406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:10.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.604+0000 7f29b1400640 1 --2- 172.21.15.45:0/3458975238 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f299406ca60 0x7f299406ef20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f29b4154ff0 tx=0x7f29a0008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:10.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.606+0000 7f29aa000640 1 -- 172.21.15.45:0/3458975238 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f29a404d030 con 0x7f29b40ba610 2024-04-16T16:15:10.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:10.853+0000 7f29b3200640 1 -- 172.21.15.45:0/3458975238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f29b40d4310 con 0x7f29b40ba610 2024-04-16T16:15:11.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.351+0000 7f29aa000640 1 -- 172.21.15.45:0/3458975238 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v19) ==== 136+0+0 (secure 0 0 0) 0x7f29a407c910 con 0x7f29b40ba610 2024-04-16T16:15:11.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.355+0000 7f29b3200640 1 -- 172.21.15.45:0/3458975238 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f299406ca60 msgr2=0x7f299406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:11.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.355+0000 7f29b3200640 1 --2- 172.21.15.45:0/3458975238 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f299406ca60 0x7f299406ef20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f29b4154ff0 tx=0x7f29a0008040 comp rx=0 tx=0).stop 2024-04-16T16:15:11.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.355+0000 7f29b3200640 1 -- 172.21.15.45:0/3458975238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40ba610 msgr2=0x7f29b414f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:11.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.355+0000 7f29b3200640 1 --2- 172.21.15.45:0/3458975238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40ba610 0x7f29b414f930 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f29a4000f80 tx=0x7f29a4004a40 comp rx=0 tx=0).stop 2024-04-16T16:15:11.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.355+0000 7f29b3200640 1 -- 172.21.15.45:0/3458975238 shutdown_connections 2024-04-16T16:15:11.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.355+0000 7f29b3200640 1 --2- 172.21.15.45:0/3458975238 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f299406ca60 0x7f299406ef20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:11.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.355+0000 7f29b3200640 1 --2- 172.21.15.45:0/3458975238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29b4155ab0 0x7f29b415e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:11.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.355+0000 7f29b3200640 1 --2- 172.21.15.45:0/3458975238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29b40d18c0 0x7f29b415de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:11.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.355+0000 7f29b3200640 1 --2- 172.21.15.45:0/3458975238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40ba610 0x7f29b414f930 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:11.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.355+0000 7f29b3200640 1 -- 172.21.15.45:0/3458975238 >> 172.21.15.45:0/3458975238 conn(0x7f29b406d1a0 msgr2=0x7f29b4161100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:11.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.355+0000 7f29b3200640 1 -- 172.21.15.45:0/3458975238 shutdown_connections 2024-04-16T16:15:11.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.355+0000 7f29b3200640 1 -- 172.21.15.45:0/3458975238 wait complete. 2024-04-16T16:15:11.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:15:11.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.609+0000 7fbba0e00640 1 Processor -- start 2024-04-16T16:15:11.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.609+0000 7fbba0e00640 1 -- start start 2024-04-16T16:15:11.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.611+0000 7fbba0e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbb9c146220 0x7fbb9c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:11.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.611+0000 7fbba0e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb9c15d360 con 0x7fbb9c06f7a0 2024-04-16T16:15:11.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.611+0000 7fbba0e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb9c15d640 con 0x7fbb9c1608a0 2024-04-16T16:15:11.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.611+0000 7fbba0e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fbb9c160410 con 0x7fbb9c146220 2024-04-16T16:15:11.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.611+0000 7fbb9b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbb9c146220 0x7fbb9c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:11.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.611+0000 7fbb9aa00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbb9c06f7a0 0x7fbb9c06df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48800/0 (socket says 172.21.15.45:48800) 2024-04-16T16:15:11.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.611+0000 7fbb9b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbb9c146220 0x7fbb9c15ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60322/0 (socket says 172.21.15.45:60322) 2024-04-16T16:15:11.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.611+0000 7fbb9aa00640 1 -- 172.21.15.45:0/241004072 learned_addr learned my addr 172.21.15.45:0/241004072 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:11.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.612+0000 7fbb9b400640 1 -- 172.21.15.45:0/241004072 >> v1:172.21.15.103:6789/0 conn(0x7fbb9c1608a0 legacy=0x7fbb9c0cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:11.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.612+0000 7fbb9b400640 1 -- 172.21.15.45:0/241004072 >> v1:172.21.15.45:6789/0 conn(0x7fbb9c06f7a0 legacy=0x7fbb9c06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:11.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.612+0000 7fbb9b400640 1 -- 172.21.15.45:0/241004072 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9c15d920 con 0x7fbb9c146220 2024-04-16T16:15:11.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.613+0000 7fbb9b400640 1 --2- 172.21.15.45:0/241004072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9c146220 0x7fbb9c15ccb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0097a0 tx=0x7fbb8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902a24b6bcfa5fac server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:11.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.613+0000 7fbb9a000640 1 -- 172.21.15.45:0/241004072 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c035020 con 0x7fbb9c146220 2024-04-16T16:15:11.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.613+0000 7fbb9a000640 1 -- 172.21.15.45:0/241004072 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb8c031830 con 0x7fbb9c146220 2024-04-16T16:15:11.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.613+0000 7fbb9a000640 1 -- 172.21.15.45:0/241004072 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c031c00 con 0x7fbb9c146220 2024-04-16T16:15:11.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.614+0000 7fbba0e00640 1 -- 172.21.15.45:0/241004072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9c146220 msgr2=0x7fbb9c15ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:11.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.614+0000 7fbba0e00640 1 --2- 172.21.15.45:0/241004072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9c146220 0x7fbb9c15ccb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0097a0 tx=0x7fbb8c02b8d0 comp rx=0 tx=0).stop 2024-04-16T16:15:11.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.614+0000 7fbba0e00640 1 -- 172.21.15.45:0/241004072 shutdown_connections 2024-04-16T16:15:11.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.614+0000 7fbba0e00640 1 --2- 172.21.15.45:0/241004072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9c146220 0x7fbb9c15ccb0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:11.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.614+0000 7fbba0e00640 1 -- 172.21.15.45:0/241004072 >> 172.21.15.45:0/241004072 conn(0x7fbb9c06b930 msgr2=0x7fbb9c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:11.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.614+0000 7fbba0e00640 1 -- 172.21.15.45:0/241004072 shutdown_connections 2024-04-16T16:15:11.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.614+0000 7fbba0e00640 1 -- 172.21.15.45:0/241004072 wait complete. 2024-04-16T16:15:11.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.616+0000 7fbba0e00640 1 Processor -- start 2024-04-16T16:15:11.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.616+0000 7fbba0e00640 1 -- start start 2024-04-16T16:15:11.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.617+0000 7fbba0e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9c06f7a0 0x7fbb9c140270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:11.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.617+0000 7fbba0e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb9c146220 0x7fbb9c15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:11.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.617+0000 7fbba0e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb9c1608a0 0x7fbb9c15c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:11.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.617+0000 7fbba0e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9c06fc20 con 0x7fbb9c06f7a0 2024-04-16T16:15:11.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.617+0000 7fbba0e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9c15dc00 con 0x7fbb9c146220 2024-04-16T16:15:11.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.617+0000 7fbba0e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbb9c06fda0 con 0x7fbb9c1608a0 2024-04-16T16:15:11.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.617+0000 7fbb9b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9c06f7a0 0x7fbb9c140270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:11.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.617+0000 7fbb9b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9c06f7a0 0x7fbb9c140270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60332/0 (socket says 172.21.15.45:60332) 2024-04-16T16:15:11.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.617+0000 7fbb9b400640 1 -- 172.21.15.45:0/2067305946 learned_addr learned my addr 172.21.15.45:0/2067305946 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:11.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.617+0000 7fbb9b400640 1 -- 172.21.15.45:0/2067305946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb9c1608a0 msgr2=0x7fbb9c15c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:11.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.617+0000 7fbb9b400640 1 --2- 172.21.15.45:0/2067305946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb9c1608a0 0x7fbb9c15c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:11.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.617+0000 7fbb9b400640 1 -- 172.21.15.45:0/2067305946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb9c146220 msgr2=0x7fbb9c15bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:11.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.617+0000 7fbb9b400640 1 --2- 172.21.15.45:0/2067305946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb9c146220 0x7fbb9c15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:11.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.617+0000 7fbb9b400640 1 -- 172.21.15.45:0/2067305946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9c1391c0 con 0x7fbb9c06f7a0 2024-04-16T16:15:11.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.618+0000 7fbb9b400640 1 --2- 172.21.15.45:0/2067305946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9c06f7a0 0x7fbb9c140270 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c02b910 tx=0x7fbb8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:11.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.618+0000 7fbb93400640 1 -- 172.21.15.45:0/2067305946 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c041070 con 0x7fbb9c06f7a0 2024-04-16T16:15:11.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.618+0000 7fbb93400640 1 -- 172.21.15.45:0/2067305946 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb8c03a970 con 0x7fbb9c06f7a0 2024-04-16T16:15:11.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.618+0000 7fbb93400640 1 -- 172.21.15.45:0/2067305946 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c035040 con 0x7fbb9c06f7a0 2024-04-16T16:15:11.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.618+0000 7fbba0e00640 1 -- 172.21.15.45:0/2067305946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb9c1394a0 con 0x7fbb9c06f7a0 2024-04-16T16:15:11.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.618+0000 7fbba0e00640 1 -- 172.21.15.45:0/2067305946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb9c139a60 con 0x7fbb9c06f7a0 2024-04-16T16:15:11.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.619+0000 7fbba0e00640 1 -- 172.21.15.45:0/2067305946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb9c075780 con 0x7fbb9c06f7a0 2024-04-16T16:15:11.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.620+0000 7fbb93400640 1 -- 172.21.15.45:0/2067305946 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fbb8c038020 con 0x7fbb9c06f7a0 2024-04-16T16:15:11.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.621+0000 7fbb93400640 1 --2- 172.21.15.45:0/2067305946 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbb7c06ca60 0x7fbb7c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:11.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.621+0000 7fbb93400640 1 -- 172.21.15.45:0/2067305946 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5201+0+0 (secure 0 0 0) 0x7fbb8c0ad6b0 con 0x7fbb9c06f7a0 2024-04-16T16:15:11.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.622+0000 7fbb9aa00640 1 --2- 172.21.15.45:0/2067305946 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbb7c06ca60 0x7fbb7c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:11.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.623+0000 7fbb9aa00640 1 --2- 172.21.15.45:0/2067305946 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbb7c06ca60 0x7fbb7c06ef20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbb88000f60 tx=0x7fbb88007a80 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:11.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.626+0000 7fbb93400640 1 -- 172.21.15.45:0/2067305946 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbb8c07bb20 con 0x7fbb9c06f7a0 2024-04-16T16:15:11.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.834+0000 7fbba0e00640 1 -- 172.21.15.45:0/2067305946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbb9c0cff60 con 0x7fbb9c06f7a0 2024-04-16T16:15:11.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.836+0000 7fbb93400640 1 -- 172.21.15.45:0/2067305946 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+13404 (secure 0 0 0) 0x7fbb8c07b420 con 0x7fbb9c06f7a0 2024-04-16T16:15:11.836 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:11.836 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":38,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:15:10.337283+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:15:11.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.839+0000 7fbba0e00640 1 -- 172.21.15.45:0/2067305946 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbb7c06ca60 msgr2=0x7fbb7c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:11.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.839+0000 7fbba0e00640 1 --2- 172.21.15.45:0/2067305946 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbb7c06ca60 0x7fbb7c06ef20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbb88000f60 tx=0x7fbb88007a80 comp rx=0 tx=0).stop 2024-04-16T16:15:11.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.839+0000 7fbba0e00640 1 -- 172.21.15.45:0/2067305946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9c06f7a0 msgr2=0x7fbb9c140270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:11.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.839+0000 7fbba0e00640 1 --2- 172.21.15.45:0/2067305946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9c06f7a0 0x7fbb9c140270 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c02b910 tx=0x7fbb8c0046c0 comp rx=0 tx=0).stop 2024-04-16T16:15:11.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.839+0000 7fbba0e00640 1 -- 172.21.15.45:0/2067305946 shutdown_connections 2024-04-16T16:15:11.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.839+0000 7fbba0e00640 1 --2- 172.21.15.45:0/2067305946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb9c1608a0 0x7fbb9c15c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:11.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.839+0000 7fbba0e00640 1 --2- 172.21.15.45:0/2067305946 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbb7c06ca60 0x7fbb7c06ef20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:11.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.839+0000 7fbba0e00640 1 --2- 172.21.15.45:0/2067305946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb9c146220 0x7fbb9c15bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:11.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.839+0000 7fbba0e00640 1 --2- 172.21.15.45:0/2067305946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb9c06f7a0 0x7fbb9c140270 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:11.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.839+0000 7fbba0e00640 1 -- 172.21.15.45:0/2067305946 >> 172.21.15.45:0/2067305946 conn(0x7fbb9c06b930 msgr2=0x7fbb9c0c6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:11.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.840+0000 7fbba0e00640 1 -- 172.21.15.45:0/2067305946 shutdown_connections 2024-04-16T16:15:11.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:11.840+0000 7fbba0e00640 1 -- 172.21.15.45:0/2067305946 wait complete. 2024-04-16T16:15:11.858 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:15:11.858 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:15:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.083+0000 7f7c03e00640 1 Processor -- start 2024-04-16T16:15:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.083+0000 7f7c03e00640 1 -- start start 2024-04-16T16:15:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.085+0000 7f7c03e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7c0415b550 0x7f7c0415f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.085+0000 7f7c03e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0415b930 0x7f7c0415f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.085+0000 7f7c03e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c040ce7b0 con 0x7f7c0406e9c0 2024-04-16T16:15:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.085+0000 7f7c03e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7c040ced90 con 0x7f7c0415b550 2024-04-16T16:15:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.085+0000 7f7c03e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7c040d0230 con 0x7f7c0415b930 2024-04-16T16:15:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.085+0000 7f7c03400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7c0406e9c0 0x7f7c0406ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48814/0 (socket says 172.21.15.45:48814) 2024-04-16T16:15:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.085+0000 7f7c03400640 1 -- 172.21.15.45:0/921147124 learned_addr learned my addr 172.21.15.45:0/921147124 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02a00640 1 --2- 172.21.15.45:0/921147124 >> v2:172.21.15.103:3300/0 conn(0x7f7c0415b550 0x7f7c0415f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7bfa000640 1 --2- 172.21.15.45:0/921147124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0415b930 0x7f7c0415f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 -- 172.21.15.45:0/921147124 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2715909855 0 0) 0x7f7c040ce7b0 con 0x7f7c0406e9c0 2024-04-16T16:15:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 -- 172.21.15.45:0/921147124 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7be8003520 con 0x7f7c0406e9c0 2024-04-16T16:15:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 -- 172.21.15.45:0/921147124 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 200175147 0 0) 0x7f7be8003520 con 0x7f7c0406e9c0 2024-04-16T16:15:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 -- 172.21.15.45:0/921147124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0415b930 msgr2=0x7f7c0415f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 --2- 172.21.15.45:0/921147124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0415b930 0x7f7c0415f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 -- 172.21.15.45:0/921147124 >> v2:172.21.15.103:3300/0 conn(0x7f7c0415b550 msgr2=0x7f7c0415f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 --2- 172.21.15.45:0/921147124 >> v2:172.21.15.103:3300/0 conn(0x7f7c0415b550 0x7f7c0415f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 -- 172.21.15.45:0/921147124 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0406f6b0 con 0x7f7c0406e9c0 2024-04-16T16:15:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 -- 172.21.15.45:0/921147124 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f7bfc0019f0 con 0x7f7c0406e9c0 2024-04-16T16:15:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 -- 172.21.15.45:0/921147124 >> v1:172.21.15.45:6789/0 conn(0x7f7c0406e9c0 legacy=0x7f7c0406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 --2- 172.21.15.45:0/921147124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0415b930 0x7f7be80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 --2- 172.21.15.45:0/921147124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7be8003f80 0x7f7be8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 --2- 172.21.15.45:0/921147124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be8004d40 0x7f7be80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 -- 172.21.15.45:0/921147124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7c040ced90 con 0x7f7be8003f80 2024-04-16T16:15:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 -- 172.21.15.45:0/921147124 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7c040d0230 con 0x7f7be8004d40 2024-04-16T16:15:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.086+0000 7f7c02000640 1 -- 172.21.15.45:0/921147124 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7be800dbd0 con 0x7f7c0415b930 2024-04-16T16:15:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.087+0000 7f7bfa000640 1 --2- 172.21.15.45:0/921147124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7be8003f80 0x7f7be8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.087+0000 7f7c02a00640 1 --2- 172.21.15.45:0/921147124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0415b930 0x7f7be80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.087+0000 7f7bfa000640 1 -- 172.21.15.45:0/921147124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0415b930 msgr2=0x7f7be80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.087+0000 7f7bfa000640 1 --2- 172.21.15.45:0/921147124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0415b930 0x7f7be80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.087+0000 7f7bfa000640 1 -- 172.21.15.45:0/921147124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be8004d40 msgr2=0x7f7be80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.087+0000 7f7bfa000640 1 --2- 172.21.15.45:0/921147124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be8004d40 0x7f7be80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.087+0000 7f7bfa000640 1 -- 172.21.15.45:0/921147124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be8003520 con 0x7f7be8003f80 2024-04-16T16:15:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.087+0000 7f7c02a00640 1 --2- 172.21.15.45:0/921147124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0415b930 0x7f7be80039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.088+0000 7f7bfa000640 1 --2- 172.21.15.45:0/921147124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7be8003f80 0x7f7be8004800 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009af0 tx=0x7f7be8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb6869687a78e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.088+0000 7f7c02000640 1 -- 172.21.15.45:0/921147124 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec039070 con 0x7f7be8003f80 2024-04-16T16:15:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.088+0000 7f7c02000640 1 -- 172.21.15.45:0/921147124 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bec02c6b0 con 0x7f7be8003f80 2024-04-16T16:15:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.088+0000 7f7c02000640 1 -- 172.21.15.45:0/921147124 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec009690 con 0x7f7be8003f80 2024-04-16T16:15:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.089+0000 7f7c03e00640 1 -- 172.21.15.45:0/921147124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7be8003f80 msgr2=0x7f7be8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.089+0000 7f7c03e00640 1 --2- 172.21.15.45:0/921147124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7be8003f80 0x7f7be8004800 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009af0 tx=0x7f7be8005830 comp rx=0 tx=0).stop 2024-04-16T16:15:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.089+0000 7f7c03400640 1 -- 172.21.15.45:0/921147124 reap_dead start 2024-04-16T16:15:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.089+0000 7f7c03e00640 1 -- 172.21.15.45:0/921147124 shutdown_connections 2024-04-16T16:15:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.089+0000 7f7c03e00640 1 -- 172.21.15.45:0/921147124 >> 172.21.15.45:0/921147124 conn(0x7f7c04067e00 msgr2=0x7f7c0406b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.089+0000 7f7c03e00640 1 -- 172.21.15.45:0/921147124 shutdown_connections 2024-04-16T16:15:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.089+0000 7f7c03e00640 1 -- 172.21.15.45:0/921147124 wait complete. 2024-04-16T16:15:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.091+0000 7f7c03e00640 1 Processor -- start 2024-04-16T16:15:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.091+0000 7f7c03e00640 1 -- start start 2024-04-16T16:15:12.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.092+0000 7f7c03e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7be8003f80 0x7f7c04158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.092+0000 7f7c03e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c04159d90 0x7f7c0415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.092+0000 7f7c03e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c040cbb50 0x7f7c040cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.092+0000 7f7c03e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0415f660 con 0x7f7c04159d90 2024-04-16T16:15:12.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.092+0000 7f7c03e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0406f990 con 0x7f7c040cbb50 2024-04-16T16:15:12.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.092+0000 7f7c03e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0415f7e0 con 0x7f7be8003f80 2024-04-16T16:15:12.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.092+0000 7f7bfa000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c04159d90 0x7f7c0415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:12.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.092+0000 7f7bfa000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c04159d90 0x7f7c0415bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60340/0 (socket says 172.21.15.45:60340) 2024-04-16T16:15:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.092+0000 7f7bfa000640 1 -- 172.21.15.45:0/1805118310 learned_addr learned my addr 172.21.15.45:0/1805118310 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.093+0000 7f7c03400640 1 --2- 172.21.15.45:0/1805118310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c040cbb50 0x7f7c040cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.093+0000 7f7c02a00640 1 --2- 172.21.15.45:0/1805118310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7be8003f80 0x7f7c04158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.093+0000 7f7bfa000640 1 -- 172.21.15.45:0/1805118310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7be8003f80 msgr2=0x7f7c04158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.093+0000 7f7bfa000640 1 --2- 172.21.15.45:0/1805118310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7be8003f80 0x7f7c04158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.093+0000 7f7bfa000640 1 -- 172.21.15.45:0/1805118310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c040cbb50 msgr2=0x7f7c040cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.093+0000 7f7bfa000640 1 --2- 172.21.15.45:0/1805118310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c040cbb50 0x7f7c040cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.093+0000 7f7bfa000640 1 -- 172.21.15.45:0/1805118310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0415b0c0 con 0x7f7c04159d90 2024-04-16T16:15:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.093+0000 7f7bfa000640 1 --2- 172.21.15.45:0/1805118310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c04159d90 0x7f7c0415bd10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009af0 tx=0x7f7bec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.093+0000 7f7bfb400640 1 -- 172.21.15.45:0/1805118310 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec042020 con 0x7f7c04159d90 2024-04-16T16:15:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.093+0000 7f7bfb400640 1 -- 172.21.15.45:0/1805118310 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bec03a040 con 0x7f7c04159d90 2024-04-16T16:15:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.093+0000 7f7bfb400640 1 -- 172.21.15.45:0/1805118310 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec039070 con 0x7f7c04159d90 2024-04-16T16:15:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.093+0000 7f7c03e00640 1 -- 172.21.15.45:0/1805118310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c0415a210 con 0x7f7c04159d90 2024-04-16T16:15:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.094+0000 7f7c03e00640 1 -- 172.21.15.45:0/1805118310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c040cad90 con 0x7f7c04159d90 2024-04-16T16:15:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.095+0000 7f7c03e00640 1 -- 172.21.15.45:0/1805118310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c04075780 con 0x7f7c04159d90 2024-04-16T16:15:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.096+0000 7f7bfb400640 1 -- 172.21.15.45:0/1805118310 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f7bec045020 con 0x7f7c04159d90 2024-04-16T16:15:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.097+0000 7f7bfb400640 1 --2- 172.21.15.45:0/1805118310 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7be006ca60 0x7f7be006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.097+0000 7f7bfb400640 1 -- 172.21.15.45:0/1805118310 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5201+0+0 (secure 0 0 0) 0x7f7bec0acf90 con 0x7f7c04159d90 2024-04-16T16:15:12.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.097+0000 7f7c02a00640 1 --2- 172.21.15.45:0/1805118310 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7be006ca60 0x7f7be006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:12.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.099+0000 7f7c02a00640 1 --2- 172.21.15.45:0/1805118310 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7be006ca60 0x7f7be006ef20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7c0415a690 tx=0x7f7bf4008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:12.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.101+0000 7f7bfb400640 1 -- 172.21.15.45:0/1805118310 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7bec04c070 con 0x7f7c04159d90 2024-04-16T16:15:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.296+0000 7f7c03e00640 1 -- 172.21.15.45:0/1805118310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7c040c6490 con 0x7f7c04159d90 2024-04-16T16:15:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.296+0000 7f7bfb400640 1 -- 172.21.15.45:0/1805118310 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) ==== 93+0+10 (secure 0 0 0) 0x7f7bec07b490 con 0x7f7c04159d90 2024-04-16T16:15:12.296 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:15:12.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.299+0000 7f7c03e00640 1 -- 172.21.15.45:0/1805118310 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7be006ca60 msgr2=0x7f7be006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:12.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.299+0000 7f7c03e00640 1 --2- 172.21.15.45:0/1805118310 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7be006ca60 0x7f7be006ef20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7c0415a690 tx=0x7f7bf4008040 comp rx=0 tx=0).stop 2024-04-16T16:15:12.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.299+0000 7f7c03e00640 1 -- 172.21.15.45:0/1805118310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c04159d90 msgr2=0x7f7c0415bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:12.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.299+0000 7f7c03e00640 1 --2- 172.21.15.45:0/1805118310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c04159d90 0x7f7c0415bd10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009af0 tx=0x7f7bec004730 comp rx=0 tx=0).stop 2024-04-16T16:15:12.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.299+0000 7f7c03e00640 1 -- 172.21.15.45:0/1805118310 shutdown_connections 2024-04-16T16:15:12.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.299+0000 7f7c03e00640 1 --2- 172.21.15.45:0/1805118310 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7be006ca60 0x7f7be006ef20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.299+0000 7f7c03e00640 1 --2- 172.21.15.45:0/1805118310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c040cbb50 0x7f7c040cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.299+0000 7f7c03e00640 1 --2- 172.21.15.45:0/1805118310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c04159d90 0x7f7c0415bd10 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.299+0000 7f7c03e00640 1 --2- 172.21.15.45:0/1805118310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7be8003f80 0x7f7c04158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.299+0000 7f7c03e00640 1 -- 172.21.15.45:0/1805118310 >> 172.21.15.45:0/1805118310 conn(0x7f7c04067e00 msgr2=0x7f7c040b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:12.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.299+0000 7f7c03e00640 1 -- 172.21.15.45:0/1805118310 shutdown_connections 2024-04-16T16:15:12.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.299+0000 7f7c03e00640 1 -- 172.21.15.45:0/1805118310 wait complete. 2024-04-16T16:15:12.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-16T16:15:12.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.532+0000 7f6e56c00640 1 Processor -- start 2024-04-16T16:15:12.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.532+0000 7f6e56c00640 1 -- start start 2024-04-16T16:15:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.534+0000 7f6e56c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6e50146220 0x7f6e5015ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.534+0000 7f6e56c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e5006f7a0 0x7f6e5006df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.534+0000 7f6e56c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6e5006e480 0x7f6e501608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.534+0000 7f6e56c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f6e50160410 con 0x7f6e50146220 2024-04-16T16:15:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.534+0000 7f6e56c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6e5015d360 con 0x7f6e5006e480 2024-04-16T16:15:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.534+0000 7f6e56c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6e5015d640 con 0x7f6e5006f7a0 2024-04-16T16:15:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.534+0000 7f6e55800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6e50146220 0x7f6e5015ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.534+0000 7f6e55800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6e50146220 0x7f6e5015ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60352/0 (socket says 172.21.15.45:60352) 2024-04-16T16:15:12.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.534+0000 7f6e55800640 1 -- 172.21.15.45:0/1039413468 learned_addr learned my addr 172.21.15.45:0/1039413468 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:12.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.534+0000 7f6e54e00640 1 --2- 172.21.15.45:0/1039413468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e5006f7a0 0x7f6e5006df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:12.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.534+0000 7f6e56200640 1 --2- 172.21.15.45:0/1039413468 >> v2:172.21.15.103:3300/0 conn(0x7f6e5006e480 0x7f6e501608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:12.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.535+0000 7f6e55800640 1 -- 172.21.15.45:0/1039413468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e5006f7a0 msgr2=0x7f6e5006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:12.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.535+0000 7f6e55800640 1 --2- 172.21.15.45:0/1039413468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e5006f7a0 0x7f6e5006df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.535+0000 7f6e55800640 1 -- 172.21.15.45:0/1039413468 >> v2:172.21.15.103:3300/0 conn(0x7f6e5006e480 msgr2=0x7f6e501608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:12.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.535+0000 7f6e55800640 1 --2- 172.21.15.45:0/1039413468 >> v2:172.21.15.103:3300/0 conn(0x7f6e5006e480 0x7f6e501608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.535+0000 7f6e55800640 1 -- 172.21.15.45:0/1039413468 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5015d920 con 0x7f6e50146220 2024-04-16T16:15:12.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.535+0000 7f6e56200640 1 --2- 172.21.15.45:0/1039413468 >> v2:172.21.15.103:3300/0 conn(0x7f6e5006e480 0x7f6e501608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:12.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.536+0000 7f6e55800640 1 --2- 172.21.15.45:0/1039413468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e50146220 0x7f6e5015ccb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6e500d05f0 tx=0x7f6e4402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247fad6562a376ee server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:12.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.536+0000 7f6e4fe00640 1 -- 172.21.15.45:0/1039413468 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4402ca80 con 0x7f6e50146220 2024-04-16T16:15:12.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.536+0000 7f6e4fe00640 1 -- 172.21.15.45:0/1039413468 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e4402cc20 con 0x7f6e50146220 2024-04-16T16:15:12.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.536+0000 7f6e4fe00640 1 -- 172.21.15.45:0/1039413468 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e440343f0 con 0x7f6e50146220 2024-04-16T16:15:12.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.537+0000 7f6e56c00640 1 -- 172.21.15.45:0/1039413468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e50146220 msgr2=0x7f6e5015ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:12.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.537+0000 7f6e56c00640 1 --2- 172.21.15.45:0/1039413468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e50146220 0x7f6e5015ccb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6e500d05f0 tx=0x7f6e4402bee0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.537+0000 7f6e56c00640 1 -- 172.21.15.45:0/1039413468 shutdown_connections 2024-04-16T16:15:12.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.537+0000 7f6e56c00640 1 --2- 172.21.15.45:0/1039413468 >> v2:172.21.15.103:3300/0 conn(0x7f6e5006e480 0x7f6e501608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.537+0000 7f6e56c00640 1 --2- 172.21.15.45:0/1039413468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e5006f7a0 0x7f6e5006df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.537+0000 7f6e56c00640 1 --2- 172.21.15.45:0/1039413468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e50146220 0x7f6e5015ccb0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.537+0000 7f6e56c00640 1 -- 172.21.15.45:0/1039413468 >> 172.21.15.45:0/1039413468 conn(0x7f6e5006b930 msgr2=0x7f6e5006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:12.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.537+0000 7f6e56c00640 1 -- 172.21.15.45:0/1039413468 shutdown_connections 2024-04-16T16:15:12.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.537+0000 7f6e56c00640 1 -- 172.21.15.45:0/1039413468 wait complete. 2024-04-16T16:15:12.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.539+0000 7f6e56c00640 1 Processor -- start 2024-04-16T16:15:12.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.539+0000 7f6e56c00640 1 -- start start 2024-04-16T16:15:12.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.541+0000 7f6e56c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e5006e480 0x7f6e5013f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.541+0000 7f6e56c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e5006f7a0 0x7f6e500ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.541+0000 7f6e56c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e50146220 0x7f6e500ca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.541+0000 7f6e56c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5006fda0 con 0x7f6e5006e480 2024-04-16T16:15:12.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.541+0000 7f6e56c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5006fc20 con 0x7f6e5006f7a0 2024-04-16T16:15:12.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.541+0000 7f6e56c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6e50160de0 con 0x7f6e50146220 2024-04-16T16:15:12.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.541+0000 7f6e55800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e5006e480 0x7f6e5013f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:12.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.541+0000 7f6e55800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e5006e480 0x7f6e5013f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60368/0 (socket says 172.21.15.45:60368) 2024-04-16T16:15:12.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.541+0000 7f6e55800640 1 -- 172.21.15.45:0/1869999158 learned_addr learned my addr 172.21.15.45:0/1869999158 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:12.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.541+0000 7f6e55800640 1 -- 172.21.15.45:0/1869999158 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e50146220 msgr2=0x7f6e500ca9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:12.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.541+0000 7f6e55800640 1 --2- 172.21.15.45:0/1869999158 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e50146220 0x7f6e500ca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.541+0000 7f6e55800640 1 -- 172.21.15.45:0/1869999158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e5006f7a0 msgr2=0x7f6e500ca480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:12.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.541+0000 7f6e54e00640 1 --2- 172.21.15.45:0/1869999158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e5006f7a0 0x7f6e500ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:12.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.542+0000 7f6e55800640 1 --2- 172.21.15.45:0/1869999158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e5006f7a0 0x7f6e500ca480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.542+0000 7f6e55800640 1 -- 172.21.15.45:0/1869999158 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e500c9b80 con 0x7f6e5006e480 2024-04-16T16:15:12.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.542+0000 7f6e55800640 1 --2- 172.21.15.45:0/1869999158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e5006e480 0x7f6e5013f040 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6e44000f80 tx=0x7f6e44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:12.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.542+0000 7f6e4d600640 1 -- 172.21.15.45:0/1869999158 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e44043020 con 0x7f6e5006e480 2024-04-16T16:15:12.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.542+0000 7f6e4d600640 1 -- 172.21.15.45:0/1869999158 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e4403e450 con 0x7f6e5006e480 2024-04-16T16:15:12.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.542+0000 7f6e56c00640 1 -- 172.21.15.45:0/1869999158 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e501458d0 con 0x7f6e5006e480 2024-04-16T16:15:12.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.542+0000 7f6e4d600640 1 -- 172.21.15.45:0/1869999158 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4403e840 con 0x7f6e5006e480 2024-04-16T16:15:12.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.542+0000 7f6e56c00640 1 -- 172.21.15.45:0/1869999158 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e501450d0 con 0x7f6e5006e480 2024-04-16T16:15:12.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.543+0000 7f6e56c00640 1 -- 172.21.15.45:0/1869999158 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e50075780 con 0x7f6e5006e480 2024-04-16T16:15:12.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.545+0000 7f6e4d600640 1 -- 172.21.15.45:0/1869999158 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f6e44046020 con 0x7f6e5006e480 2024-04-16T16:15:12.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.545+0000 7f6e4d600640 1 --2- 172.21.15.45:0/1869999158 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6e3006ca60 0x7f6e3006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:12.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.546+0000 7f6e4d600640 1 -- 172.21.15.45:0/1869999158 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5201+0+0 (secure 0 0 0) 0x7f6e440ae4e0 con 0x7f6e5006e480 2024-04-16T16:15:12.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.546+0000 7f6e54e00640 1 --2- 172.21.15.45:0/1869999158 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6e3006ca60 0x7f6e3006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:12.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.548+0000 7f6e54e00640 1 --2- 172.21.15.45:0/1869999158 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6e3006ca60 0x7f6e3006ef20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c006fd0 tx=0x7f6e3c008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:12.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.550+0000 7f6e4d600640 1 -- 172.21.15.45:0/1869999158 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6e4404d030 con 0x7f6e5006e480 2024-04-16T16:15:12.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.756+0000 7f6e56c00640 1 -- 172.21.15.45:0/1869999158 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f6e500cff60 con 0x7f6e5006e480 2024-04-16T16:15:12.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.757+0000 7f6e4d600640 1 -- 172.21.15.45:0/1869999158 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v38) ==== 84+0+65 (secure 0 0 0) 0x7f6e4407c9e0 con 0x7f6e5006e480 2024-04-16T16:15:12.757 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:12.757 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2024-04-16T16:15:12.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:15:12.757 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2024-04-16T16:15:12.757 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2024-04-16T16:15:12.757 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:15:12.757 INFO:teuthology.orchestra.run.smithi045.stdout:] 2024-04-16T16:15:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.760+0000 7f6e56c00640 1 -- 172.21.15.45:0/1869999158 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6e3006ca60 msgr2=0x7f6e3006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.760+0000 7f6e56c00640 1 --2- 172.21.15.45:0/1869999158 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6e3006ca60 0x7f6e3006ef20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c006fd0 tx=0x7f6e3c008040 comp rx=0 tx=0).stop 2024-04-16T16:15:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.760+0000 7f6e56c00640 1 -- 172.21.15.45:0/1869999158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e5006e480 msgr2=0x7f6e5013f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.760+0000 7f6e56c00640 1 --2- 172.21.15.45:0/1869999158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e5006e480 0x7f6e5013f040 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6e44000f80 tx=0x7f6e44004a40 comp rx=0 tx=0).stop 2024-04-16T16:15:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.760+0000 7f6e56c00640 1 -- 172.21.15.45:0/1869999158 shutdown_connections 2024-04-16T16:15:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.760+0000 7f6e56c00640 1 --2- 172.21.15.45:0/1869999158 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e50146220 0x7f6e500ca9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.760+0000 7f6e56c00640 1 --2- 172.21.15.45:0/1869999158 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6e3006ca60 0x7f6e3006ef20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.760+0000 7f6e56c00640 1 --2- 172.21.15.45:0/1869999158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e5006f7a0 0x7f6e500ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.760+0000 7f6e56c00640 1 --2- 172.21.15.45:0/1869999158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e5006e480 0x7f6e5013f040 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.760+0000 7f6e56c00640 1 -- 172.21.15.45:0/1869999158 >> 172.21.15.45:0/1869999158 conn(0x7f6e5006b930 msgr2=0x7f6e500c77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.760+0000 7f6e56c00640 1 -- 172.21.15.45:0/1869999158 shutdown_connections 2024-04-16T16:15:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:12.760+0000 7f6e56c00640 1 -- 172.21.15.45:0/1869999158 wait complete. 2024-04-16T16:15:12.779 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'fs1' 2024-04-16T16:15:12.779 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create fs1_metadata --pg_num_min 64 2024-04-16T16:15:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.009+0000 7fddcce00640 1 Processor -- start 2024-04-16T16:15:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.009+0000 7fddcce00640 1 -- start start 2024-04-16T16:15:13.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.010+0000 7fddcce00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fddc80ba610 0x7fddc815efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:13.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.010+0000 7fddcce00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fddc80d18c0 0x7fddc80d1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:13.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.010+0000 7fddcce00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddc8155e90 con 0x7fddc8155ab0 2024-04-16T16:15:13.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.010+0000 7fddcce00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fddc815ff00 con 0x7fddc80d18c0 2024-04-16T16:15:13.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.010+0000 7fddcce00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fddc815f940 con 0x7fddc80ba610 2024-04-16T16:15:13.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.011+0000 7fddc7400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fddc8155ab0 0x7fddc8162ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48830/0 (socket says 172.21.15.45:48830) 2024-04-16T16:15:13.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.011+0000 7fddc7400640 1 -- 172.21.15.45:0/122741600 learned_addr learned my addr 172.21.15.45:0/122741600 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:13.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.011+0000 7fddc7e00640 1 --2- 172.21.15.45:0/122741600 >> v2:172.21.15.45:3300/0 conn(0x7fddc80d18c0 0x7fddc80d1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.011+0000 7fddc6a00640 1 --2- 172.21.15.45:0/122741600 >> v2:172.21.15.103:3300/0 conn(0x7fddc80ba610 0x7fddc815efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.011+0000 7fddc6000640 1 -- 172.21.15.45:0/122741600 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2197423816 0 0) 0x7fddc8155e90 con 0x7fddc8155ab0 2024-04-16T16:15:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.011+0000 7fddc6000640 1 -- 172.21.15.45:0/122741600 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddac003520 con 0x7fddc8155ab0 2024-04-16T16:15:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.012+0000 7fddc7e00640 1 -- 172.21.15.45:0/122741600 >> v2:172.21.15.103:3300/0 conn(0x7fddc80ba610 msgr2=0x7fddc815efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.012+0000 7fddc7e00640 1 --2- 172.21.15.45:0/122741600 >> v2:172.21.15.103:3300/0 conn(0x7fddc80ba610 0x7fddc815efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.012+0000 7fddc7e00640 1 -- 172.21.15.45:0/122741600 >> v1:172.21.15.45:6789/0 conn(0x7fddc8155ab0 legacy=0x7fddc8162ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.012+0000 7fddc7e00640 1 -- 172.21.15.45:0/122741600 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc815fc20 con 0x7fddc80d18c0 2024-04-16T16:15:13.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.013+0000 7fddc7e00640 1 --2- 172.21.15.45:0/122741600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddc80d18c0 0x7fddc80d1d50 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fddc80b66c0 tx=0x7fddc002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=842c62ab6b54598c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:13.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.013+0000 7fddc6000640 1 -- 172.21.15.45:0/122741600 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc00315a0 con 0x7fddc80d18c0 2024-04-16T16:15:13.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.013+0000 7fddc6000640 1 -- 172.21.15.45:0/122741600 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddc0037070 con 0x7fddc80d18c0 2024-04-16T16:15:13.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.013+0000 7fddc6000640 1 -- 172.21.15.45:0/122741600 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc0031940 con 0x7fddc80d18c0 2024-04-16T16:15:13.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.014+0000 7fddcce00640 1 -- 172.21.15.45:0/122741600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddc80d18c0 msgr2=0x7fddc80d1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:13.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.014+0000 7fddcce00640 1 --2- 172.21.15.45:0/122741600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddc80d18c0 0x7fddc80d1d50 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fddc80b66c0 tx=0x7fddc002b840 comp rx=0 tx=0).stop 2024-04-16T16:15:13.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.014+0000 7fddcce00640 1 -- 172.21.15.45:0/122741600 shutdown_connections 2024-04-16T16:15:13.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.014+0000 7fddcce00640 1 --2- 172.21.15.45:0/122741600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddc80d18c0 0x7fddc80d1d50 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:13.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.014+0000 7fddcce00640 1 --2- 172.21.15.45:0/122741600 >> v2:172.21.15.103:3300/0 conn(0x7fddc80ba610 0x7fddc815efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:13.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.014+0000 7fddcce00640 1 -- 172.21.15.45:0/122741600 >> 172.21.15.45:0/122741600 conn(0x7fddc806d1a0 msgr2=0x7fddc806d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:13.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.014+0000 7fddcce00640 1 -- 172.21.15.45:0/122741600 shutdown_connections 2024-04-16T16:15:13.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.014+0000 7fddcce00640 1 -- 172.21.15.45:0/122741600 wait complete. 2024-04-16T16:15:13.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.016+0000 7fddcce00640 1 Processor -- start 2024-04-16T16:15:13.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.016+0000 7fddcce00640 1 -- start start 2024-04-16T16:15:13.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.017+0000 7fddcce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddc80ba610 0x7fddc80d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:13.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.017+0000 7fddcce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddc80d18c0 0x7fddc8152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:13.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.017+0000 7fddcce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddc8155ab0 0x7fddc81f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:13.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.017+0000 7fddcce00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fddc815f510 con 0x7fddc80ba610 2024-04-16T16:15:13.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.017+0000 7fddcce00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fddc81629d0 con 0x7fddc8155ab0 2024-04-16T16:15:13.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.017+0000 7fddcce00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fddc815f690 con 0x7fddc80d18c0 2024-04-16T16:15:13.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.017+0000 7fddc7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddc80ba610 0x7fddc80d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:13.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.017+0000 7fddc7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddc80ba610 0x7fddc80d2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60382/0 (socket says 172.21.15.45:60382) 2024-04-16T16:15:13.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.017+0000 7fddc7400640 1 -- 172.21.15.45:0/841523048 learned_addr learned my addr 172.21.15.45:0/841523048 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:13.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.017+0000 7fddc7400640 1 -- 172.21.15.45:0/841523048 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddc80d18c0 msgr2=0x7fddc8152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:13.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.017+0000 7fddc7400640 1 --2- 172.21.15.45:0/841523048 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddc80d18c0 0x7fddc8152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:13.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.017+0000 7fddc7400640 1 -- 172.21.15.45:0/841523048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddc8155ab0 msgr2=0x7fddc81f4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:13.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.017+0000 7fddc7e00640 1 --2- 172.21.15.45:0/841523048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddc8155ab0 0x7fddc81f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:13.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.018+0000 7fddc7400640 1 --2- 172.21.15.45:0/841523048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddc8155ab0 0x7fddc81f4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:13.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.018+0000 7fddc7400640 1 -- 172.21.15.45:0/841523048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc81f5220 con 0x7fddc80ba610 2024-04-16T16:15:13.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.018+0000 7fddc7400640 1 --2- 172.21.15.45:0/841523048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddc80ba610 0x7fddc80d2cf0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fddb800caa0 tx=0x7fddb800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:13.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.018+0000 7fddbf400640 1 -- 172.21.15.45:0/841523048 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb8017070 con 0x7fddc80ba610 2024-04-16T16:15:13.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.018+0000 7fddbf400640 1 -- 172.21.15.45:0/841523048 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddb800ac00 con 0x7fddc80ba610 2024-04-16T16:15:13.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.018+0000 7fddbf400640 1 -- 172.21.15.45:0/841523048 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb8012400 con 0x7fddc80ba610 2024-04-16T16:15:13.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.018+0000 7fddcce00640 1 -- 172.21.15.45:0/841523048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddc815c390 con 0x7fddc80ba610 2024-04-16T16:15:13.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.018+0000 7fddcce00640 1 -- 172.21.15.45:0/841523048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddc8157820 con 0x7fddc80ba610 2024-04-16T16:15:13.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.020+0000 7fddcce00640 1 -- 172.21.15.45:0/841523048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddc8075780 con 0x7fddc80ba610 2024-04-16T16:15:13.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.021+0000 7fddbf400640 1 -- 172.21.15.45:0/841523048 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fddb801a070 con 0x7fddc80ba610 2024-04-16T16:15:13.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.021+0000 7fddbf400640 1 --2- 172.21.15.45:0/841523048 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdda806ca60 0x7fdda806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:13.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.022+0000 7fddbf400640 1 -- 172.21.15.45:0/841523048 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5201+0+0 (secure 0 0 0) 0x7fddb808b9e0 con 0x7fddc80ba610 2024-04-16T16:15:13.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.022+0000 7fddc6a00640 1 --2- 172.21.15.45:0/841523048 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdda806ca60 0x7fdda806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:13.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.023+0000 7fddc6a00640 1 --2- 172.21.15.45:0/841523048 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdda806ca60 0x7fdda806ef20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fddb4006e20 tx=0x7fddb4008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:13.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.026+0000 7fddbf400640 1 -- 172.21.15.45:0/841523048 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fddc81f5220 con 0x7fddc80ba610 2024-04-16T16:15:13.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.232+0000 7fddcce00640 1 -- 172.21.15.45:0/841523048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fs1_metadata", "pg_num_min": 64} v 0) -- 0x7fddc815cd40 con 0x7fddc80ba610 2024-04-16T16:15:13.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.377+0000 7fddbf400640 1 -- 172.21.15.45:0/841523048 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fs1_metadata", "pg_num_min": 64}]=0 pool 'fs1_metadata' created v39) ==== 132+0+0 (secure 0 0 0) 0x7fddb8059e50 con 0x7fddc80ba610 2024-04-16T16:15:13.376 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'fs1_metadata' created 2024-04-16T16:15:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.380+0000 7fddcce00640 1 -- 172.21.15.45:0/841523048 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdda806ca60 msgr2=0x7fdda806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.380+0000 7fddcce00640 1 --2- 172.21.15.45:0/841523048 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdda806ca60 0x7fdda806ef20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fddb4006e20 tx=0x7fddb4008040 comp rx=0 tx=0).stop 2024-04-16T16:15:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.380+0000 7fddcce00640 1 -- 172.21.15.45:0/841523048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddc80ba610 msgr2=0x7fddc80d2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.380+0000 7fddcce00640 1 --2- 172.21.15.45:0/841523048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddc80ba610 0x7fddc80d2cf0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fddb800caa0 tx=0x7fddb800cf60 comp rx=0 tx=0).stop 2024-04-16T16:15:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.380+0000 7fddcce00640 1 -- 172.21.15.45:0/841523048 shutdown_connections 2024-04-16T16:15:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.380+0000 7fddcce00640 1 --2- 172.21.15.45:0/841523048 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdda806ca60 0x7fdda806ef20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.380+0000 7fddcce00640 1 --2- 172.21.15.45:0/841523048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddc8155ab0 0x7fddc81f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.380+0000 7fddcce00640 1 --2- 172.21.15.45:0/841523048 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddc80d18c0 0x7fddc8152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.380+0000 7fddcce00640 1 --2- 172.21.15.45:0/841523048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddc80ba610 0x7fddc80d2cf0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.380+0000 7fddcce00640 1 -- 172.21.15.45:0/841523048 >> 172.21.15.45:0/841523048 conn(0x7fddc806d1a0 msgr2=0x7fddc80d48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.381+0000 7fddcce00640 1 -- 172.21.15.45:0/841523048 shutdown_connections 2024-04-16T16:15:13.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.381+0000 7fddcce00640 1 -- 172.21.15.45:0/841523048 wait complete. 2024-04-16T16:15:13.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create fs1_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-16T16:15:13.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.616+0000 7f5697000640 1 Processor -- start 2024-04-16T16:15:13.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.616+0000 7f5697000640 1 -- start start 2024-04-16T16:15:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.618+0000 7f5697000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f56900d10e0 0x7f5690072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.618+0000 7f5697000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56900b5530 con 0x7f569015c7c0 2024-04-16T16:15:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.618+0000 7f5697000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56900b5810 con 0x7f56900b4930 2024-04-16T16:15:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.618+0000 7f5697000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f56900b52e0 con 0x7f56900d10e0 2024-04-16T16:15:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.618+0000 7f568fe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f569015c7c0 0x7f56900d09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42000/0 (socket says 172.21.15.45:42000) 2024-04-16T16:15:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.618+0000 7f568fe00640 1 -- 172.21.15.45:0/1728231819 learned_addr learned my addr 172.21.15.45:0/1728231819 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.619+0000 7f568f400640 1 --2- 172.21.15.45:0/1728231819 >> v2:172.21.15.103:3300/0 conn(0x7f56900d10e0 0x7f5690072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.619+0000 7f568ea00640 1 -- 172.21.15.45:0/1728231819 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1015537819 0 0) 0x7f56900b5530 con 0x7f569015c7c0 2024-04-16T16:15:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.619+0000 7f568ea00640 1 -- 172.21.15.45:0/1728231819 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5674003520 con 0x7f569015c7c0 2024-04-16T16:15:13.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.619+0000 7f568f400640 1 -- 172.21.15.45:0/1728231819 >> v1:172.21.15.103:6789/0 conn(0x7f56900b4930 legacy=0x7f569006f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:13.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.619+0000 7f568f400640 1 -- 172.21.15.45:0/1728231819 >> v1:172.21.15.45:6789/0 conn(0x7f569015c7c0 legacy=0x7f56900d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:13.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.619+0000 7f568f400640 1 -- 172.21.15.45:0/1728231819 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56900d04b0 con 0x7f56900d10e0 2024-04-16T16:15:13.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.621+0000 7f568f400640 1 --2- 172.21.15.45:0/1728231819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56900d10e0 0x7f5690072e10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f567c0097a0 tx=0x7f567c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=379787469c06ea33 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:13.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.621+0000 7f568ea00640 1 -- 172.21.15.45:0/1728231819 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c0315a0 con 0x7f56900d10e0 2024-04-16T16:15:13.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.621+0000 7f568ea00640 1 -- 172.21.15.45:0/1728231819 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f567c037070 con 0x7f56900d10e0 2024-04-16T16:15:13.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.621+0000 7f568ea00640 1 -- 172.21.15.45:0/1728231819 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c031960 con 0x7f56900d10e0 2024-04-16T16:15:13.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.622+0000 7f5697000640 1 -- 172.21.15.45:0/1728231819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56900d10e0 msgr2=0x7f5690072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:13.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.622+0000 7f5697000640 1 --2- 172.21.15.45:0/1728231819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56900d10e0 0x7f5690072e10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f567c0097a0 tx=0x7f567c02b840 comp rx=0 tx=0).stop 2024-04-16T16:15:13.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.622+0000 7f5697000640 1 -- 172.21.15.45:0/1728231819 shutdown_connections 2024-04-16T16:15:13.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.622+0000 7f5697000640 1 --2- 172.21.15.45:0/1728231819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56900d10e0 0x7f5690072e10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:13.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.622+0000 7f5697000640 1 -- 172.21.15.45:0/1728231819 >> 172.21.15.45:0/1728231819 conn(0x7f56900b3970 msgr2=0x7f56900b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:13.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.622+0000 7f5697000640 1 -- 172.21.15.45:0/1728231819 shutdown_connections 2024-04-16T16:15:13.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.622+0000 7f5697000640 1 -- 172.21.15.45:0/1728231819 wait complete. 2024-04-16T16:15:13.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.624+0000 7f5697000640 1 Processor -- start 2024-04-16T16:15:13.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.624+0000 7f5697000640 1 -- start start 2024-04-16T16:15:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.625+0000 7f5697000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56900b4930 0x7f56901e6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.625+0000 7f5697000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56900d10e0 0x7f5690145910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.625+0000 7f5697000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f569015c7c0 0x7f56901e9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.625+0000 7f5697000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f56900d0260 con 0x7f56900d10e0 2024-04-16T16:15:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.625+0000 7f5697000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f569006fac0 con 0x7f569015c7c0 2024-04-16T16:15:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.625+0000 7f5697000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5690073350 con 0x7f56900b4930 2024-04-16T16:15:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.626+0000 7f568f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56900d10e0 0x7f5690145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.626+0000 7f568f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56900d10e0 0x7f5690145910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56808/0 (socket says 172.21.15.45:56808) 2024-04-16T16:15:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.626+0000 7f568f400640 1 -- 172.21.15.45:0/1814041368 learned_addr learned my addr 172.21.15.45:0/1814041368 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.626+0000 7f568fe00640 1 --2- 172.21.15.45:0/1814041368 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56900b4930 0x7f56901e6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.626+0000 7f5695000640 1 --2- 172.21.15.45:0/1814041368 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f569015c7c0 0x7f56901e9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:13.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.626+0000 7f568f400640 1 -- 172.21.15.45:0/1814041368 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56900b4930 msgr2=0x7f56901e6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:13.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.626+0000 7f568f400640 1 --2- 172.21.15.45:0/1814041368 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56900b4930 0x7f56901e6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:13.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.626+0000 7f568f400640 1 -- 172.21.15.45:0/1814041368 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f569015c7c0 msgr2=0x7f56901e9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:13.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.626+0000 7f568f400640 1 --2- 172.21.15.45:0/1814041368 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f569015c7c0 0x7f56901e9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:13.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.626+0000 7f568f400640 1 -- 172.21.15.45:0/1814041368 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56901460e0 con 0x7f56900d10e0 2024-04-16T16:15:13.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.627+0000 7f568f400640 1 --2- 172.21.15.45:0/1814041368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56900d10e0 0x7f5690145910 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f567c034040 tx=0x7f567c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:13.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.627+0000 7f5687e00640 1 -- 172.21.15.45:0/1814041368 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c032840 con 0x7f56900d10e0 2024-04-16T16:15:13.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.627+0000 7f5687e00640 1 -- 172.21.15.45:0/1814041368 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f567c032a00 con 0x7f56900d10e0 2024-04-16T16:15:13.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.627+0000 7f5697000640 1 -- 172.21.15.45:0/1814041368 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56901463c0 con 0x7f56900d10e0 2024-04-16T16:15:13.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.627+0000 7f5687e00640 1 -- 172.21.15.45:0/1814041368 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c032df0 con 0x7f56900d10e0 2024-04-16T16:15:13.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.627+0000 7f5697000640 1 -- 172.21.15.45:0/1814041368 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5690146980 con 0x7f56900d10e0 2024-04-16T16:15:13.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.629+0000 7f5697000640 1 -- 172.21.15.45:0/1814041368 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5690075780 con 0x7f56900d10e0 2024-04-16T16:15:13.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.630+0000 7f5687e00640 1 -- 172.21.15.45:0/1814041368 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f567c04e430 con 0x7f56900d10e0 2024-04-16T16:15:13.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.630+0000 7f5687e00640 1 --2- 172.21.15.45:0/1814041368 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f567006ca60 0x7f567006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:13.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.631+0000 7f5687e00640 1 -- 172.21.15.45:0/1814041368 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 5568+0+0 (secure 0 0 0) 0x7f567c02f030 con 0x7f56900d10e0 2024-04-16T16:15:13.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.631+0000 7f568fe00640 1 --2- 172.21.15.45:0/1814041368 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f567006ca60 0x7f567006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:13.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.632+0000 7f568fe00640 1 --2- 172.21.15.45:0/1814041368 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f567006ca60 0x7f567006ef20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f56800011e0 tx=0x7f5680009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:13.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.636+0000 7f5687e00640 1 -- 172.21.15.45:0/1814041368 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f567c040070 con 0x7f56900d10e0 2024-04-16T16:15:13.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:13.839+0000 7f5697000640 1 -- 172.21.15.45:0/1814041368 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fs1_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f56900b04f0 con 0x7f56900d10e0 2024-04-16T16:15:14.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.383+0000 7f5687e00640 1 -- 172.21.15.45:0/1814041368 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fs1_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'fs1_data' created v40) ==== 164+0+0 (secure 0 0 0) 0x7f567c031e90 con 0x7f56900d10e0 2024-04-16T16:15:14.383 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'fs1_data' created 2024-04-16T16:15:14.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.387+0000 7f5697000640 1 -- 172.21.15.45:0/1814041368 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f567006ca60 msgr2=0x7f567006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:14.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.387+0000 7f5697000640 1 --2- 172.21.15.45:0/1814041368 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f567006ca60 0x7f567006ef20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f56800011e0 tx=0x7f5680009040 comp rx=0 tx=0).stop 2024-04-16T16:15:14.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.387+0000 7f5697000640 1 -- 172.21.15.45:0/1814041368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56900d10e0 msgr2=0x7f5690145910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:14.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.387+0000 7f5697000640 1 --2- 172.21.15.45:0/1814041368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56900d10e0 0x7f5690145910 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f567c034040 tx=0x7f567c0046c0 comp rx=0 tx=0).stop 2024-04-16T16:15:14.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.387+0000 7f5697000640 1 -- 172.21.15.45:0/1814041368 shutdown_connections 2024-04-16T16:15:14.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.387+0000 7f5697000640 1 --2- 172.21.15.45:0/1814041368 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f567006ca60 0x7f567006ef20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:14.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.387+0000 7f5697000640 1 --2- 172.21.15.45:0/1814041368 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f569015c7c0 0x7f56901e9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:14.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.387+0000 7f5697000640 1 --2- 172.21.15.45:0/1814041368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56900d10e0 0x7f5690145910 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:14.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.387+0000 7f5697000640 1 --2- 172.21.15.45:0/1814041368 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56900b4930 0x7f56901e6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:14.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.387+0000 7f5697000640 1 -- 172.21.15.45:0/1814041368 >> 172.21.15.45:0/1814041368 conn(0x7f56900b3970 msgr2=0x7f56900b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:14.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.388+0000 7f5697000640 1 -- 172.21.15.45:0/1814041368 shutdown_connections 2024-04-16T16:15:14.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.388+0000 7f5697000640 1 -- 172.21.15.45:0/1814041368 wait complete. 2024-04-16T16:15:14.420 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new fs1 fs1_metadata fs1_data 2024-04-16T16:15:14.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.612+0000 7f9d24c00640 1 Processor -- start 2024-04-16T16:15:14.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.612+0000 7f9d24c00640 1 -- start start 2024-04-16T16:15:14.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.614+0000 7f9d24c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9d2015f060 0x7f9d200d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:14.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.614+0000 7f9d24c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d200b7ca0 con 0x7f9d2015ec80 2024-04-16T16:15:14.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.614+0000 7f9d24c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d200d3450 con 0x7f9d200b6c00 2024-04-16T16:15:14.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.614+0000 7f9d24c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9d200d3220 con 0x7f9d2015f060 2024-04-16T16:15:14.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.614+0000 7f9d1d400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9d2015f060 0x7f9d200d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:14.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.614+0000 7f9d1de00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9d2015ec80 0x7f9d200b64f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42014/0 (socket says 172.21.15.45:42014) 2024-04-16T16:15:14.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.614+0000 7f9d1de00640 1 -- 172.21.15.45:0/3466294324 learned_addr learned my addr 172.21.15.45:0/3466294324 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:14.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.615+0000 7f9d1d400640 1 -- 172.21.15.45:0/3466294324 >> v1:172.21.15.103:6789/0 conn(0x7f9d200b6c00 legacy=0x7f9d20168990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:14.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.615+0000 7f9d1d400640 1 -- 172.21.15.45:0/3466294324 >> v1:172.21.15.45:6789/0 conn(0x7f9d2015ec80 legacy=0x7f9d200b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:14.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.615+0000 7f9d1d400640 1 -- 172.21.15.45:0/3466294324 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d200d3730 con 0x7f9d2015f060 2024-04-16T16:15:14.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.616+0000 7f9d1d400640 1 --2- 172.21.15.45:0/3466294324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d2015f060 0x7f9d200d2ce0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0097a0 tx=0x7f9d0c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2f4b560f3e891 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:14.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.616+0000 7f9d1ca00640 1 -- 172.21.15.45:0/3466294324 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c034070 con 0x7f9d2015f060 2024-04-16T16:15:14.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.617+0000 7f9d1ca00640 1 -- 172.21.15.45:0/3466294324 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d0c030610 con 0x7f9d2015f060 2024-04-16T16:15:14.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.617+0000 7f9d1ca00640 1 -- 172.21.15.45:0/3466294324 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c0309e0 con 0x7f9d2015f060 2024-04-16T16:15:14.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.617+0000 7f9d24c00640 1 -- 172.21.15.45:0/3466294324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d2015f060 msgr2=0x7f9d200d2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:14.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.617+0000 7f9d24c00640 1 --2- 172.21.15.45:0/3466294324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d2015f060 0x7f9d200d2ce0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0097a0 tx=0x7f9d0c02b960 comp rx=0 tx=0).stop 2024-04-16T16:15:14.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.617+0000 7f9d24c00640 1 -- 172.21.15.45:0/3466294324 shutdown_connections 2024-04-16T16:15:14.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.617+0000 7f9d24c00640 1 --2- 172.21.15.45:0/3466294324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d2015f060 0x7f9d200d2ce0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:14.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.617+0000 7f9d24c00640 1 -- 172.21.15.45:0/3466294324 >> 172.21.15.45:0/3466294324 conn(0x7f9d200d16d0 msgr2=0x7f9d200d1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:14.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.617+0000 7f9d24c00640 1 -- 172.21.15.45:0/3466294324 shutdown_connections 2024-04-16T16:15:14.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.617+0000 7f9d24c00640 1 -- 172.21.15.45:0/3466294324 wait complete. 2024-04-16T16:15:14.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.619+0000 7f9d24c00640 1 Processor -- start 2024-04-16T16:15:14.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.620+0000 7f9d24c00640 1 -- start start 2024-04-16T16:15:14.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.621+0000 7f9d24c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d200b6c00 0x7f9d2015d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:14.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.621+0000 7f9d24c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d2015ec80 0x7f9d200c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:14.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.621+0000 7f9d24c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d200ab8b0 0x7f9d200abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:14.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.621+0000 7f9d24c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9d200ba190 con 0x7f9d200ab8b0 2024-04-16T16:15:14.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.621+0000 7f9d24c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9d200d3cf0 con 0x7f9d200b6c00 2024-04-16T16:15:14.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.621+0000 7f9d24c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9d200ba310 con 0x7f9d2015ec80 2024-04-16T16:15:14.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.621+0000 7f9d1e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d200ab8b0 0x7f9d200abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:14.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.621+0000 7f9d1e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d200ab8b0 0x7f9d200abe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56836/0 (socket says 172.21.15.45:56836) 2024-04-16T16:15:14.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.621+0000 7f9d1e800640 1 -- 172.21.15.45:0/2084891291 learned_addr learned my addr 172.21.15.45:0/2084891291 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:14.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.622+0000 7f9d1de00640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d200b6c00 0x7f9d2015d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:14.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.622+0000 7f9d1d400640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d2015ec80 0x7f9d200c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:14.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.622+0000 7f9d1e800640 1 -- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d2015ec80 msgr2=0x7f9d200c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:14.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.622+0000 7f9d1e800640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d2015ec80 0x7f9d200c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:14.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.622+0000 7f9d1e800640 1 -- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d200b6c00 msgr2=0x7f9d2015d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:14.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.622+0000 7f9d1e800640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d200b6c00 0x7f9d2015d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:14.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.622+0000 7f9d1e800640 1 -- 172.21.15.45:0/2084891291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d2015ddb0 con 0x7f9d200ab8b0 2024-04-16T16:15:14.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.622+0000 7f9d1de00640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d200b6c00 0x7f9d2015d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:14.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.622+0000 7f9d1d400640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d2015ec80 0x7f9d200c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:14.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.622+0000 7f9d1e800640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d200ab8b0 0x7f9d200abe40 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800c730 tx=0x7f9d1800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:14.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.622+0000 7f9d16000640 1 -- 172.21.15.45:0/2084891291 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18016020 con 0x7f9d200ab8b0 2024-04-16T16:15:14.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.622+0000 7f9d16000640 1 -- 172.21.15.45:0/2084891291 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d1800aa50 con 0x7f9d200ab8b0 2024-04-16T16:15:14.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.622+0000 7f9d24c00640 1 -- 172.21.15.45:0/2084891291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d200c8df0 con 0x7f9d200ab8b0 2024-04-16T16:15:14.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.622+0000 7f9d16000640 1 -- 172.21.15.45:0/2084891291 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1800ce40 con 0x7f9d200ab8b0 2024-04-16T16:15:14.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.623+0000 7f9d24c00640 1 -- 172.21.15.45:0/2084891291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d200c93b0 con 0x7f9d200ab8b0 2024-04-16T16:15:14.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.624+0000 7f9d24c00640 1 -- 172.21.15.45:0/2084891291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d20075780 con 0x7f9d200ab8b0 2024-04-16T16:15:14.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.625+0000 7f9d16000640 1 -- 172.21.15.45:0/2084891291 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f9d18019020 con 0x7f9d200ab8b0 2024-04-16T16:15:14.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.626+0000 7f9d16000640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f9d0006ca90 0x7f9d0006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:14.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.626+0000 7f9d16000640 1 -- 172.21.15.45:0/2084891291 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 5947+0+0 (secure 0 0 0) 0x7f9d1808be60 con 0x7f9d200ab8b0 2024-04-16T16:15:14.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.627+0000 7f9d1de00640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f9d0006ca90 0x7f9d0006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:14.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.628+0000 7f9d1de00640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f9d0006ca90 0x7f9d0006ef50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9d10000f60 tx=0x7f9d10009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:14.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.632+0000 7f9d16000640 1 -- 172.21.15.45:0/2084891291 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9d18020030 con 0x7f9d200ab8b0 2024-04-16T16:15:14.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:14.877+0000 7f9d24c00640 1 -- 172.21.15.45:0/2084891291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "fs1", "metadata": "fs1_metadata", "data": "fs1_data"} v 0) -- 0x7f9d200d5540 con 0x7f9d200ab8b0 2024-04-16T16:15:15.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.435+0000 7f9d16000640 1 -- 172.21.15.45:0/2084891291 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "fs1", "metadata": "fs1_metadata", "data": "fs1_data"}]=0 new fs with metadata pool 6 and data pool 7 v20) ==== 163+0+0 (secure 0 0 0) 0x7f9d180182c0 con 0x7f9d200ab8b0 2024-04-16T16:15:15.440 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 6 and data pool 7 2024-04-16T16:15:15.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.443+0000 7f9d24c00640 1 -- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f9d0006ca90 msgr2=0x7f9d0006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:15.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.443+0000 7f9d24c00640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f9d0006ca90 0x7f9d0006ef50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9d10000f60 tx=0x7f9d10009040 comp rx=0 tx=0).stop 2024-04-16T16:15:15.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.443+0000 7f9d24c00640 1 -- 172.21.15.45:0/2084891291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d200ab8b0 msgr2=0x7f9d200abe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:15.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.443+0000 7f9d24c00640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d200ab8b0 0x7f9d200abe40 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800c730 tx=0x7f9d1800cbf0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.444+0000 7f9d24c00640 1 -- 172.21.15.45:0/2084891291 shutdown_connections 2024-04-16T16:15:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.444+0000 7f9d24c00640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d200ab8b0 0x7f9d200abe40 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.444+0000 7f9d24c00640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d2015ec80 0x7f9d200c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.444+0000 7f9d24c00640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f9d0006ca90 0x7f9d0006ef50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.444+0000 7f9d24c00640 1 --2- 172.21.15.45:0/2084891291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d200b6c00 0x7f9d2015d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.444+0000 7f9d24c00640 1 -- 172.21.15.45:0/2084891291 >> 172.21.15.45:0/2084891291 conn(0x7f9d200d16d0 msgr2=0x7f9d200b4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:15.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.445+0000 7f9d24c00640 1 -- 172.21.15.45:0/2084891291 shutdown_connections 2024-04-16T16:15:15.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.445+0000 7f9d24c00640 1 -- 172.21.15.45:0/2084891291 wait complete. 2024-04-16T16:15:15.476 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:15:15.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.659+0000 7fe726800640 1 Processor -- start 2024-04-16T16:15:15.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.659+0000 7fe726800640 1 -- start start 2024-04-16T16:15:15.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.660+0000 7fe726800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe720144ff0 0x7fe7200cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:15.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.660+0000 7fe726800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe72015c700 con 0x7fe7200c5d80 2024-04-16T16:15:15.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.660+0000 7fe726800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe72015c9c0 con 0x7fe72006dfb0 2024-04-16T16:15:15.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.660+0000 7fe726800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe72015d2f0 con 0x7fe720144ff0 2024-04-16T16:15:15.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.661+0000 7fe725e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe7200c5d80 0x7fe72015d780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42016/0 (socket says 172.21.15.45:42016) 2024-04-16T16:15:15.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.661+0000 7fe725e00640 1 -- 172.21.15.45:0/1244626923 learned_addr learned my addr 172.21.15.45:0/1244626923 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:15.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.661+0000 7fe725400640 1 --2- 172.21.15.45:0/1244626923 >> v2:172.21.15.103:3300/0 conn(0x7fe720144ff0 0x7fe7200cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:15.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.661+0000 7fe71fe00640 1 -- 172.21.15.45:0/1244626923 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2247118416 0 0) 0x7fe72015c700 con 0x7fe7200c5d80 2024-04-16T16:15:15.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.661+0000 7fe71fe00640 1 -- 172.21.15.45:0/1244626923 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe704003520 con 0x7fe7200c5d80 2024-04-16T16:15:15.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe71fe00640 1 -- 172.21.15.45:0/1244626923 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 507071540 0 0) 0x7fe704003520 con 0x7fe7200c5d80 2024-04-16T16:15:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe71fe00640 1 -- 172.21.15.45:0/1244626923 >> v2:172.21.15.103:3300/0 conn(0x7fe720144ff0 msgr2=0x7fe7200cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe71fe00640 1 --2- 172.21.15.45:0/1244626923 >> v2:172.21.15.103:3300/0 conn(0x7fe720144ff0 0x7fe7200cfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe71fe00640 1 -- 172.21.15.45:0/1244626923 >> v1:172.21.15.103:6789/0 conn(0x7fe72006dfb0 legacy=0x7fe7200c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe71fe00640 1 -- 172.21.15.45:0/1244626923 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72015cca0 con 0x7fe7200c5d80 2024-04-16T16:15:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe71fe00640 1 -- 172.21.15.45:0/1244626923 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fe7180019f0 con 0x7fe7200c5d80 2024-04-16T16:15:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe71fe00640 1 -- 172.21.15.45:0/1244626923 >> v1:172.21.15.45:6789/0 conn(0x7fe7200c5d80 legacy=0x7fe72015d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe71fe00640 1 --2- 172.21.15.45:0/1244626923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7040039b0 0x7fe704003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe71fe00640 1 --2- 172.21.15.45:0/1244626923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe704004800 0x7fe704004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe71fe00640 1 --2- 172.21.15.45:0/1244626923 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7040051c0 0x7fe70400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe71fe00640 1 -- 172.21.15.45:0/1244626923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe72015d2f0 con 0x7fe7040039b0 2024-04-16T16:15:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe71fe00640 1 -- 172.21.15.45:0/1244626923 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe70400e2a0 con 0x7fe704004800 2024-04-16T16:15:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe71fe00640 1 -- 172.21.15.45:0/1244626923 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe70400e4a0 con 0x7fe7040051c0 2024-04-16T16:15:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe725400640 1 --2- 172.21.15.45:0/1244626923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7040039b0 0x7fe704003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe725400640 1 -- 172.21.15.45:0/1244626923 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7040051c0 msgr2=0x7fe70400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe725400640 1 --2- 172.21.15.45:0/1244626923 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7040051c0 0x7fe70400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe725400640 1 -- 172.21.15.45:0/1244626923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe704004800 msgr2=0x7fe704004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe725400640 1 --2- 172.21.15.45:0/1244626923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe704004800 0x7fe704004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.662+0000 7fe725400640 1 -- 172.21.15.45:0/1244626923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe704003520 con 0x7fe7040039b0 2024-04-16T16:15:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.663+0000 7fe725e00640 1 -- 172.21.15.45:0/1244626923 reap_dead start 2024-04-16T16:15:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.664+0000 7fe725400640 1 --2- 172.21.15.45:0/1244626923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7040039b0 0x7fe704003df0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe710009ac0 tx=0x7fe704005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9a3805653f600e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.664+0000 7fe71fe00640 1 -- 172.21.15.45:0/1244626923 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe710039070 con 0x7fe7040039b0 2024-04-16T16:15:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.664+0000 7fe71fe00640 1 -- 172.21.15.45:0/1244626923 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe71002c6b0 con 0x7fe7040039b0 2024-04-16T16:15:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.664+0000 7fe71fe00640 1 -- 172.21.15.45:0/1244626923 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe710009690 con 0x7fe7040039b0 2024-04-16T16:15:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.664+0000 7fe726800640 1 -- 172.21.15.45:0/1244626923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7040039b0 msgr2=0x7fe704003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.664+0000 7fe726800640 1 --2- 172.21.15.45:0/1244626923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7040039b0 0x7fe704003df0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe710009ac0 tx=0x7fe704005830 comp rx=0 tx=0).stop 2024-04-16T16:15:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.664+0000 7fe726800640 1 -- 172.21.15.45:0/1244626923 shutdown_connections 2024-04-16T16:15:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.664+0000 7fe726800640 1 --2- 172.21.15.45:0/1244626923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7040039b0 0x7fe704003df0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.665+0000 7fe726800640 1 -- 172.21.15.45:0/1244626923 >> 172.21.15.45:0/1244626923 conn(0x7fe72006b930 msgr2=0x7fe72006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.665+0000 7fe726800640 1 -- 172.21.15.45:0/1244626923 shutdown_connections 2024-04-16T16:15:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.665+0000 7fe726800640 1 -- 172.21.15.45:0/1244626923 wait complete. 2024-04-16T16:15:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.666+0000 7fe726800640 1 Processor -- start 2024-04-16T16:15:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.667+0000 7fe726800640 1 -- start start 2024-04-16T16:15:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.667+0000 7fe726800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7040039b0 0x7fe72012f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.667+0000 7fe726800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7201f72f0 0x7fe7200cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.668+0000 7fe726800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7200cd780 0x7fe7201f7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.668+0000 7fe726800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe7200c5670 con 0x7fe7201f72f0 2024-04-16T16:15:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.668+0000 7fe726800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe7201453d0 con 0x7fe7040039b0 2024-04-16T16:15:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.668+0000 7fe726800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe7200c57f0 con 0x7fe7200cd780 2024-04-16T16:15:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.668+0000 7fe724a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7201f72f0 0x7fe7200cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.668+0000 7fe724a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7201f72f0 0x7fe7200cd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56868/0 (socket says 172.21.15.45:56868) 2024-04-16T16:15:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.668+0000 7fe724a00640 1 -- 172.21.15.45:0/1358059822 learned_addr learned my addr 172.21.15.45:0/1358059822 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.668+0000 7fe725400640 1 --2- 172.21.15.45:0/1358059822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7040039b0 0x7fe72012f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.668+0000 7fe724a00640 1 -- 172.21.15.45:0/1358059822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7200cd780 msgr2=0x7fe7201f7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.668+0000 7fe725e00640 1 --2- 172.21.15.45:0/1358059822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7200cd780 0x7fe7201f7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.668+0000 7fe724a00640 1 --2- 172.21.15.45:0/1358059822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7200cd780 0x7fe7201f7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.668+0000 7fe724a00640 1 -- 172.21.15.45:0/1358059822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7040039b0 msgr2=0x7fe72012f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.668+0000 7fe724a00640 1 --2- 172.21.15.45:0/1358059822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7040039b0 0x7fe72012f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.668+0000 7fe724a00640 1 -- 172.21.15.45:0/1358059822 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7201f8a50 con 0x7fe7201f72f0 2024-04-16T16:15:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.668+0000 7fe725400640 1 --2- 172.21.15.45:0/1358059822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7040039b0 0x7fe72012f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.669+0000 7fe724a00640 1 --2- 172.21.15.45:0/1358059822 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7201f72f0 0x7fe7200cd240 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe70c00c530 tx=0x7fe70c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.669+0000 7fe71d600640 1 -- 172.21.15.45:0/1358059822 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70c0052d0 con 0x7fe7201f72f0 2024-04-16T16:15:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.669+0000 7fe71d600640 1 -- 172.21.15.45:0/1358059822 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe70c005470 con 0x7fe7201f72f0 2024-04-16T16:15:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.669+0000 7fe71d600640 1 -- 172.21.15.45:0/1358059822 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70c012850 con 0x7fe7201f72f0 2024-04-16T16:15:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.669+0000 7fe726800640 1 -- 172.21.15.45:0/1358059822 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7201f8770 con 0x7fe7201f72f0 2024-04-16T16:15:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.669+0000 7fe726800640 1 -- 172.21.15.45:0/1358059822 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe72012f000 con 0x7fe7201f72f0 2024-04-16T16:15:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.671+0000 7fe726800640 1 -- 172.21.15.45:0/1358059822 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe720075780 con 0x7fe7201f72f0 2024-04-16T16:15:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.671+0000 7fe71d600640 1 -- 172.21.15.45:0/1358059822 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fe70c0129f0 con 0x7fe7201f72f0 2024-04-16T16:15:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.672+0000 7fe71d600640 1 --2- 172.21.15.45:0/1358059822 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe70006ca60 0x7fe70006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.673+0000 7fe71d600640 1 -- 172.21.15.45:0/1358059822 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6041+0+0 (secure 0 0 0) 0x7fe70c08baa0 con 0x7fe7201f72f0 2024-04-16T16:15:15.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.673+0000 7fe725400640 1 --2- 172.21.15.45:0/1358059822 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe70006ca60 0x7fe70006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.674+0000 7fe725400640 1 --2- 172.21.15.45:0/1358059822 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe70006ca60 0x7fe70006ef20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe710009ac0 tx=0x7fe710030000 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.678+0000 7fe71d600640 1 -- 172.21.15.45:0/1358059822 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe70c059c70 con 0x7fe7201f72f0 2024-04-16T16:15:15.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.879+0000 7fe726800640 1 -- 172.21.15.45:0/1358059822 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe7200cdb60 con 0x7fe7201f72f0 2024-04-16T16:15:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.882+0000 7fe71d600640 1 -- 172.21.15.45:0/1358059822 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+17127 (secure 0 0 0) 0x7fe70c059570 con 0x7fe7201f72f0 2024-04-16T16:15:15.882 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:15.882 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":41,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:15:14.879058+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"fs1_metadata","create_time":"2024-04-16T16:15:13.250782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"fs1_data","create_time":"2024-04-16T16:15:13.856986+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:15:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.885+0000 7fe726800640 1 -- 172.21.15.45:0/1358059822 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe70006ca60 msgr2=0x7fe70006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.885+0000 7fe726800640 1 --2- 172.21.15.45:0/1358059822 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe70006ca60 0x7fe70006ef20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe710009ac0 tx=0x7fe710030000 comp rx=0 tx=0).stop 2024-04-16T16:15:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.885+0000 7fe726800640 1 -- 172.21.15.45:0/1358059822 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7201f72f0 msgr2=0x7fe7200cd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.885+0000 7fe726800640 1 --2- 172.21.15.45:0/1358059822 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7201f72f0 0x7fe7200cd240 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe70c00c530 tx=0x7fe70c00c9f0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.885+0000 7fe726800640 1 -- 172.21.15.45:0/1358059822 shutdown_connections 2024-04-16T16:15:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.885+0000 7fe726800640 1 --2- 172.21.15.45:0/1358059822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7200cd780 0x7fe7201f7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.885+0000 7fe726800640 1 --2- 172.21.15.45:0/1358059822 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7201f72f0 0x7fe7200cd240 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.885+0000 7fe726800640 1 --2- 172.21.15.45:0/1358059822 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe70006ca60 0x7fe70006ef20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.885+0000 7fe726800640 1 --2- 172.21.15.45:0/1358059822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7040039b0 0x7fe72012f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:15.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.885+0000 7fe726800640 1 -- 172.21.15.45:0/1358059822 >> 172.21.15.45:0/1358059822 conn(0x7fe72006b930 msgr2=0x7fe72006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:15.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.885+0000 7fe726800640 1 -- 172.21.15.45:0/1358059822 shutdown_connections 2024-04-16T16:15:15.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:15.886+0000 7fe726800640 1 -- 172.21.15.45:0/1358059822 wait complete. 2024-04-16T16:15:15.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:15:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.119+0000 7fc78cc00640 1 Processor -- start 2024-04-16T16:15:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.119+0000 7fc78cc00640 1 -- start start 2024-04-16T16:15:16.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.121+0000 7fc78cc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc7881462e0 0x7fc788072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:16.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.121+0000 7fc78cc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7880728c0 0x7fc7880ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.121+0000 7fc78cc00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7880cf6d0 con 0x7fc7880cec00 2024-04-16T16:15:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.121+0000 7fc78cc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc788072d20 con 0x7fc7881462e0 2024-04-16T16:15:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.121+0000 7fc78cc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc7881466c0 con 0x7fc7880728c0 2024-04-16T16:15:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.121+0000 7fc786800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc7880cec00 0x7fc78815da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42020/0 (socket says 172.21.15.45:42020) 2024-04-16T16:15:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.121+0000 7fc786800640 1 -- 172.21.15.45:0/4232040125 learned_addr learned my addr 172.21.15.45:0/4232040125 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc785e00640 1 --2- 172.21.15.45:0/4232040125 >> v2:172.21.15.103:3300/0 conn(0x7fc7881462e0 0x7fc788072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc785400640 1 --2- 172.21.15.45:0/4232040125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7880728c0 0x7fc7880ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 -- 172.21.15.45:0/4232040125 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 226618965 0 0) 0x7fc7880cf6d0 con 0x7fc7880cec00 2024-04-16T16:15:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 -- 172.21.15.45:0/4232040125 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc76c003520 con 0x7fc7880cec00 2024-04-16T16:15:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 -- 172.21.15.45:0/4232040125 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4105920072 0 0) 0x7fc76c003520 con 0x7fc7880cec00 2024-04-16T16:15:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 -- 172.21.15.45:0/4232040125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7880728c0 msgr2=0x7fc7880ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 --2- 172.21.15.45:0/4232040125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7880728c0 0x7fc7880ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 -- 172.21.15.45:0/4232040125 >> v2:172.21.15.103:3300/0 conn(0x7fc7881462e0 msgr2=0x7fc788072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 --2- 172.21.15.45:0/4232040125 >> v2:172.21.15.103:3300/0 conn(0x7fc7881462e0 0x7fc788072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 -- 172.21.15.45:0/4232040125 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7880cf3f0 con 0x7fc7880cec00 2024-04-16T16:15:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 -- 172.21.15.45:0/4232040125 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fc7800019f0 con 0x7fc7880cec00 2024-04-16T16:15:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 -- 172.21.15.45:0/4232040125 >> v1:172.21.15.45:6789/0 conn(0x7fc7880cec00 legacy=0x7fc78815da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 --2- 172.21.15.45:0/4232040125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7880728c0 0x7fc76c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 --2- 172.21.15.45:0/4232040125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc76c003f30 0x7fc76c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 --2- 172.21.15.45:0/4232040125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc76c004d40 0x7fc76c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 -- 172.21.15.45:0/4232040125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc7881466c0 con 0x7fc76c004d40 2024-04-16T16:15:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 -- 172.21.15.45:0/4232040125 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c00b9c0 con 0x7fc76c003f30 2024-04-16T16:15:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.122+0000 7fc784a00640 1 -- 172.21.15.45:0/4232040125 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc76c00bb90 con 0x7fc7880728c0 2024-04-16T16:15:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.123+0000 7fc786800640 1 --2- 172.21.15.45:0/4232040125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc76c004d40 0x7fc76c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.123+0000 7fc785400640 1 --2- 172.21.15.45:0/4232040125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc76c003f30 0x7fc76c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.123+0000 7fc785e00640 1 --2- 172.21.15.45:0/4232040125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7880728c0 0x7fc76c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.123+0000 7fc786800640 1 -- 172.21.15.45:0/4232040125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7880728c0 msgr2=0x7fc76c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.123+0000 7fc786800640 1 --2- 172.21.15.45:0/4232040125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7880728c0 0x7fc76c0039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.123+0000 7fc786800640 1 -- 172.21.15.45:0/4232040125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc76c003f30 msgr2=0x7fc76c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.123+0000 7fc786800640 1 --2- 172.21.15.45:0/4232040125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc76c003f30 0x7fc76c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.123+0000 7fc786800640 1 -- 172.21.15.45:0/4232040125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c003520 con 0x7fc76c004d40 2024-04-16T16:15:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.123+0000 7fc785400640 1 --2- 172.21.15.45:0/4232040125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc76c003f30 0x7fc76c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:16.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.124+0000 7fc786800640 1 --2- 172.21.15.45:0/4232040125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc76c004d40 0x7fc76c0051c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc78000a950 tx=0x7fc76c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=299f34e500172730 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:16.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.124+0000 7fc784a00640 1 -- 172.21.15.45:0/4232040125 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78000a4c0 con 0x7fc76c004d40 2024-04-16T16:15:16.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.125+0000 7fc784a00640 1 -- 172.21.15.45:0/4232040125 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc780034650 con 0x7fc76c004d40 2024-04-16T16:15:16.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.125+0000 7fc784a00640 1 -- 172.21.15.45:0/4232040125 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc780034ac0 con 0x7fc76c004d40 2024-04-16T16:15:16.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.125+0000 7fc78cc00640 1 -- 172.21.15.45:0/4232040125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc76c004d40 msgr2=0x7fc76c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:16.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.125+0000 7fc78cc00640 1 --2- 172.21.15.45:0/4232040125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc76c004d40 0x7fc76c0051c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc78000a950 tx=0x7fc76c005830 comp rx=0 tx=0).stop 2024-04-16T16:15:16.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.125+0000 7fc786800640 1 -- 172.21.15.45:0/4232040125 reap_dead start 2024-04-16T16:15:16.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.125+0000 7fc78cc00640 1 -- 172.21.15.45:0/4232040125 shutdown_connections 2024-04-16T16:15:16.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.125+0000 7fc78cc00640 1 -- 172.21.15.45:0/4232040125 >> 172.21.15.45:0/4232040125 conn(0x7fc7880b2750 msgr2=0x7fc7880b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:16.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.125+0000 7fc78cc00640 1 -- 172.21.15.45:0/4232040125 shutdown_connections 2024-04-16T16:15:16.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.125+0000 7fc78cc00640 1 -- 172.21.15.45:0/4232040125 wait complete. 2024-04-16T16:15:16.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.127+0000 7fc78cc00640 1 Processor -- start 2024-04-16T16:15:16.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.127+0000 7fc78cc00640 1 -- start start 2024-04-16T16:15:16.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.128+0000 7fc78cc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc76c004d40 0x7fc788143410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:16.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.128+0000 7fc78cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7880abbb0 0x7fc78815bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:16.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.128+0000 7fc78cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7880ac030 0x7fc78815c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:16.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.128+0000 7fc78cc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc78815d540 con 0x7fc7880ac030 2024-04-16T16:15:16.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.128+0000 7fc78cc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc78815d260 con 0x7fc7880abbb0 2024-04-16T16:15:16.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.128+0000 7fc78cc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc7880cf090 con 0x7fc76c004d40 2024-04-16T16:15:16.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.128+0000 7fc786800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7880ac030 0x7fc78815c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:16.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.128+0000 7fc786800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7880ac030 0x7fc78815c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56888/0 (socket says 172.21.15.45:56888) 2024-04-16T16:15:16.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.128+0000 7fc786800640 1 -- 172.21.15.45:0/2417357549 learned_addr learned my addr 172.21.15.45:0/2417357549 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:16.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.129+0000 7fc785400640 1 --2- 172.21.15.45:0/2417357549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7880abbb0 0x7fc78815bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:16.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.129+0000 7fc785e00640 1 --2- 172.21.15.45:0/2417357549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc76c004d40 0x7fc788143410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:16.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.129+0000 7fc786800640 1 -- 172.21.15.45:0/2417357549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc76c004d40 msgr2=0x7fc788143410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:16.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.129+0000 7fc786800640 1 --2- 172.21.15.45:0/2417357549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc76c004d40 0x7fc788143410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.129+0000 7fc786800640 1 -- 172.21.15.45:0/2417357549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7880abbb0 msgr2=0x7fc78815bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:16.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.129+0000 7fc786800640 1 --2- 172.21.15.45:0/2417357549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7880abbb0 0x7fc78815bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.129+0000 7fc786800640 1 -- 172.21.15.45:0/2417357549 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc788143af0 con 0x7fc7880ac030 2024-04-16T16:15:16.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.129+0000 7fc785400640 1 --2- 172.21.15.45:0/2417357549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7880abbb0 0x7fc78815bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:16.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.129+0000 7fc786800640 1 --2- 172.21.15.45:0/2417357549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7880ac030 0x7fc78815c400 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc780000f80 tx=0x7fc7880cf5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:16.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.129+0000 7fc77e000640 1 -- 172.21.15.45:0/2417357549 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc780002000 con 0x7fc7880ac030 2024-04-16T16:15:16.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.129+0000 7fc77e000640 1 -- 172.21.15.45:0/2417357549 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc780001940 con 0x7fc7880ac030 2024-04-16T16:15:16.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.129+0000 7fc77e000640 1 -- 172.21.15.45:0/2417357549 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc780034800 con 0x7fc7880ac030 2024-04-16T16:15:16.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.130+0000 7fc78cc00640 1 -- 172.21.15.45:0/2417357549 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc788143dd0 con 0x7fc7880ac030 2024-04-16T16:15:16.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.130+0000 7fc78cc00640 1 -- 172.21.15.45:0/2417357549 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7880ae330 con 0x7fc7880ac030 2024-04-16T16:15:16.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.131+0000 7fc78cc00640 1 -- 172.21.15.45:0/2417357549 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc788075780 con 0x7fc7880ac030 2024-04-16T16:15:16.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.132+0000 7fc77e000640 1 -- 172.21.15.45:0/2417357549 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fc780046020 con 0x7fc7880ac030 2024-04-16T16:15:16.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.133+0000 7fc77e000640 1 --2- 172.21.15.45:0/2417357549 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc76806ca90 0x7fc76806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:16.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.133+0000 7fc77e000640 1 -- 172.21.15.45:0/2417357549 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6041+0+0 (secure 0 0 0) 0x7fc7800adb10 con 0x7fc7880ac030 2024-04-16T16:15:16.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.133+0000 7fc785e00640 1 --2- 172.21.15.45:0/2417357549 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc76806ca90 0x7fc76806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:16.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.135+0000 7fc785e00640 1 --2- 172.21.15.45:0/2417357549 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc76806ca90 0x7fc76806ef50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc7880b13d0 tx=0x7fc778008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:16.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.137+0000 7fc77e000640 1 -- 172.21.15.45:0/2417357549 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc78004d020 con 0x7fc7880ac030 2024-04-16T16:15:16.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.338+0000 7fc78cc00640 1 -- 172.21.15.45:0/2417357549 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc7881442f0 con 0x7fc7880ac030 2024-04-16T16:15:16.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.340+0000 7fc77e000640 1 -- 172.21.15.45:0/2417357549 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+17127 (secure 0 0 0) 0x7fc78007bbc0 con 0x7fc7880ac030 2024-04-16T16:15:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:16.339 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":41,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:15:14.879058+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"fs1_metadata","create_time":"2024-04-16T16:15:13.250782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"fs1_data","create_time":"2024-04-16T16:15:13.856986+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:15:16.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.342+0000 7fc78cc00640 1 -- 172.21.15.45:0/2417357549 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc76806ca90 msgr2=0x7fc76806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:16.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.342+0000 7fc78cc00640 1 --2- 172.21.15.45:0/2417357549 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc76806ca90 0x7fc76806ef50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc7880b13d0 tx=0x7fc778008040 comp rx=0 tx=0).stop 2024-04-16T16:15:16.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.342+0000 7fc78cc00640 1 -- 172.21.15.45:0/2417357549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7880ac030 msgr2=0x7fc78815c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:16.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.342+0000 7fc78cc00640 1 --2- 172.21.15.45:0/2417357549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7880ac030 0x7fc78815c400 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc780000f80 tx=0x7fc7880cf5c0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.342+0000 7fc78cc00640 1 -- 172.21.15.45:0/2417357549 shutdown_connections 2024-04-16T16:15:16.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.342+0000 7fc78cc00640 1 --2- 172.21.15.45:0/2417357549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7880ac030 0x7fc78815c400 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.342+0000 7fc78cc00640 1 --2- 172.21.15.45:0/2417357549 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc76806ca90 0x7fc76806ef50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.342+0000 7fc78cc00640 1 --2- 172.21.15.45:0/2417357549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7880abbb0 0x7fc78815bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.343+0000 7fc78cc00640 1 --2- 172.21.15.45:0/2417357549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc76c004d40 0x7fc788143410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.343+0000 7fc78cc00640 1 -- 172.21.15.45:0/2417357549 >> 172.21.15.45:0/2417357549 conn(0x7fc7880b2750 msgr2=0x7fc788145510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:16.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.343+0000 7fc78cc00640 1 -- 172.21.15.45:0/2417357549 shutdown_connections 2024-04-16T16:15:16.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.343+0000 7fc78cc00640 1 -- 172.21.15.45:0/2417357549 wait complete. 2024-04-16T16:15:16.361 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set fs1 standby_count_wanted 0 2024-04-16T16:15:16.463 INFO:tasks.ceph.mds.a.smithi045.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: In function 'void MDLog::trim()' thread 7fcc3e600640 time 2024-04-16T16:15:16.464935+0000 2024-04-16T16:15:16.463 INFO:tasks.ceph.mds.a.smithi045.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: 688: FAILED ceph_assert(num_total_segments >= num_replayed_segments) 2024-04-16T16:15:16.464 INFO:tasks.ceph.mds.a.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:15:16.464 INFO:tasks.ceph.mds.a.smithi045.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fcc4ad81f4c] 2024-04-16T16:15:16.464 INFO:tasks.ceph.mds.a.smithi045.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7fcc4ad8210a] 2024-04-16T16:15:16.464 INFO:tasks.ceph.mds.a.smithi045.stderr: 3: (MDLog::trim()+0xf2a) [0x56024c58f1aa] 2024-04-16T16:15:16.464 INFO:tasks.ceph.mds.a.smithi045.stderr: 4: (MDLog::log_trim_upkeep()+0x339) [0x56024c58f559] 2024-04-16T16:15:16.464 INFO:tasks.ceph.mds.a.smithi045.stderr: 5: /lib64/libstdc++.so.6(+0xdb924) [0x7fcc4a8db924] 2024-04-16T16:15:16.464 INFO:tasks.ceph.mds.a.smithi045.stderr: 6: /lib64/libc.so.6(+0x9f802) [0x7fcc4a49f802] 2024-04-16T16:15:16.464 INFO:tasks.ceph.mds.a.smithi045.stderr: 7: /lib64/libc.so.6(+0x3f450) [0x7fcc4a43f450] 2024-04-16T16:15:16.465 INFO:tasks.ceph.mds.a.smithi045.stderr:2024-04-16T16:15:16.465+0000 7fcc3e600640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: In function 'void MDLog::trim()' thread 7fcc3e600640 time 2024-04-16T16:15:16.464935+0000 2024-04-16T16:15:16.465 INFO:tasks.ceph.mds.a.smithi045.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: 688: FAILED ceph_assert(num_total_segments >= num_replayed_segments) 2024-04-16T16:15:16.465 INFO:tasks.ceph.mds.a.smithi045.stderr: 2024-04-16T16:15:16.465 INFO:tasks.ceph.mds.a.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:15:16.465 INFO:tasks.ceph.mds.a.smithi045.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fcc4ad81f4c] 2024-04-16T16:15:16.465 INFO:tasks.ceph.mds.a.smithi045.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7fcc4ad8210a] 2024-04-16T16:15:16.465 INFO:tasks.ceph.mds.a.smithi045.stderr: 3: (MDLog::trim()+0xf2a) [0x56024c58f1aa] 2024-04-16T16:15:16.465 INFO:tasks.ceph.mds.a.smithi045.stderr: 4: (MDLog::log_trim_upkeep()+0x339) [0x56024c58f559] 2024-04-16T16:15:16.465 INFO:tasks.ceph.mds.a.smithi045.stderr: 5: /lib64/libstdc++.so.6(+0xdb924) [0x7fcc4a8db924] 2024-04-16T16:15:16.465 INFO:tasks.ceph.mds.a.smithi045.stderr: 6: /lib64/libc.so.6(+0x9f802) [0x7fcc4a49f802] 2024-04-16T16:15:16.465 INFO:tasks.ceph.mds.a.smithi045.stderr: 7: /lib64/libc.so.6(+0x3f450) [0x7fcc4a43f450] 2024-04-16T16:15:16.465 INFO:tasks.ceph.mds.a.smithi045.stderr: 2024-04-16T16:15:16.465 INFO:tasks.ceph.mds.a.smithi045.stderr:*** Caught signal (Aborted) ** 2024-04-16T16:15:16.465 INFO:tasks.ceph.mds.a.smithi045.stderr: in thread 7fcc3e600640 thread_name:ms_dispatch 2024-04-16T16:15:16.466 INFO:tasks.ceph.mds.a.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:15:16.466 INFO:tasks.ceph.mds.a.smithi045.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fcc4a454db0] 2024-04-16T16:15:16.466 INFO:tasks.ceph.mds.a.smithi045.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fcc4a4a154c] 2024-04-16T16:15:16.466 INFO:tasks.ceph.mds.a.smithi045.stderr: 3: raise() 2024-04-16T16:15:16.466 INFO:tasks.ceph.mds.a.smithi045.stderr: 4: abort() 2024-04-16T16:15:16.466 INFO:tasks.ceph.mds.a.smithi045.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fcc4ad81fa6] 2024-04-16T16:15:16.466 INFO:tasks.ceph.mds.a.smithi045.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7fcc4ad8210a] 2024-04-16T16:15:16.466 INFO:tasks.ceph.mds.a.smithi045.stderr: 7: (MDLog::trim()+0xf2a) [0x56024c58f1aa] 2024-04-16T16:15:16.466 INFO:tasks.ceph.mds.a.smithi045.stderr: 8: (MDLog::log_trim_upkeep()+0x339) [0x56024c58f559] 2024-04-16T16:15:16.466 INFO:tasks.ceph.mds.a.smithi045.stderr: 9: /lib64/libstdc++.so.6(+0xdb924) [0x7fcc4a8db924] 2024-04-16T16:15:16.466 INFO:tasks.ceph.mds.a.smithi045.stderr: 10: /lib64/libc.so.6(+0x9f802) [0x7fcc4a49f802] 2024-04-16T16:15:16.467 INFO:tasks.ceph.mds.a.smithi045.stderr: 11: /lib64/libc.so.6(+0x3f450) [0x7fcc4a43f450] 2024-04-16T16:15:16.467 INFO:tasks.ceph.mds.a.smithi045.stderr:2024-04-16T16:15:16.468+0000 7fcc3e600640 -1 *** Caught signal (Aborted) ** 2024-04-16T16:15:16.467 INFO:tasks.ceph.mds.a.smithi045.stderr: in thread 7fcc3e600640 thread_name:ms_dispatch 2024-04-16T16:15:16.467 INFO:tasks.ceph.mds.a.smithi045.stderr: 2024-04-16T16:15:16.467 INFO:tasks.ceph.mds.a.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:15:16.467 INFO:tasks.ceph.mds.a.smithi045.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fcc4a454db0] 2024-04-16T16:15:16.467 INFO:tasks.ceph.mds.a.smithi045.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fcc4a4a154c] 2024-04-16T16:15:16.467 INFO:tasks.ceph.mds.a.smithi045.stderr: 3: raise() 2024-04-16T16:15:16.467 INFO:tasks.ceph.mds.a.smithi045.stderr: 4: abort() 2024-04-16T16:15:16.467 INFO:tasks.ceph.mds.a.smithi045.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fcc4ad81fa6] 2024-04-16T16:15:16.467 INFO:tasks.ceph.mds.a.smithi045.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7fcc4ad8210a] 2024-04-16T16:15:16.468 INFO:tasks.ceph.mds.a.smithi045.stderr: 7: (MDLog::trim()+0xf2a) [0x56024c58f1aa] 2024-04-16T16:15:16.468 INFO:tasks.ceph.mds.a.smithi045.stderr: 8: (MDLog::log_trim_upkeep()+0x339) [0x56024c58f559] 2024-04-16T16:15:16.468 INFO:tasks.ceph.mds.a.smithi045.stderr: 9: /lib64/libstdc++.so.6(+0xdb924) [0x7fcc4a8db924] 2024-04-16T16:15:16.468 INFO:tasks.ceph.mds.a.smithi045.stderr: 10: /lib64/libc.so.6(+0x9f802) [0x7fcc4a49f802] 2024-04-16T16:15:16.468 INFO:tasks.ceph.mds.a.smithi045.stderr: 11: /lib64/libc.so.6(+0x3f450) [0x7fcc4a43f450] 2024-04-16T16:15:16.468 INFO:tasks.ceph.mds.a.smithi045.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-16T16:15:16.468 INFO:tasks.ceph.mds.a.smithi045.stderr: 2024-04-16T16:15:16.480 INFO:tasks.ceph.mds.a.smithi045.stderr: -1> 2024-04-16T16:15:16.465+0000 7fcc3e600640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: In function 'void MDLog::trim()' thread 7fcc3e600640 time 2024-04-16T16:15:16.464935+0000 2024-04-16T16:15:16.480 INFO:tasks.ceph.mds.a.smithi045.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: 688: FAILED ceph_assert(num_total_segments >= num_replayed_segments) 2024-04-16T16:15:16.480 INFO:tasks.ceph.mds.a.smithi045.stderr: 2024-04-16T16:15:16.480 INFO:tasks.ceph.mds.a.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:15:16.480 INFO:tasks.ceph.mds.a.smithi045.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fcc4ad81f4c] 2024-04-16T16:15:16.480 INFO:tasks.ceph.mds.a.smithi045.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7fcc4ad8210a] 2024-04-16T16:15:16.480 INFO:tasks.ceph.mds.a.smithi045.stderr: 3: (MDLog::trim()+0xf2a) [0x56024c58f1aa] 2024-04-16T16:15:16.480 INFO:tasks.ceph.mds.a.smithi045.stderr: 4: (MDLog::log_trim_upkeep()+0x339) [0x56024c58f559] 2024-04-16T16:15:16.480 INFO:tasks.ceph.mds.a.smithi045.stderr: 5: /lib64/libstdc++.so.6(+0xdb924) [0x7fcc4a8db924] 2024-04-16T16:15:16.480 INFO:tasks.ceph.mds.a.smithi045.stderr: 6: /lib64/libc.so.6(+0x9f802) [0x7fcc4a49f802] 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: 7: /lib64/libc.so.6(+0x3f450) [0x7fcc4a43f450] 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: 0> 2024-04-16T16:15:16.468+0000 7fcc3e600640 -1 *** Caught signal (Aborted) ** 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: in thread 7fcc3e600640 thread_name:ms_dispatch 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fcc4a454db0] 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fcc4a4a154c] 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: 3: raise() 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: 4: abort() 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fcc4ad81fa6] 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7fcc4ad8210a] 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: 7: (MDLog::trim()+0xf2a) [0x56024c58f1aa] 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: 8: (MDLog::log_trim_upkeep()+0x339) [0x56024c58f559] 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: 9: /lib64/libstdc++.so.6(+0xdb924) [0x7fcc4a8db924] 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: 10: /lib64/libc.so.6(+0x9f802) [0x7fcc4a49f802] 2024-04-16T16:15:16.481 INFO:tasks.ceph.mds.a.smithi045.stderr: 11: /lib64/libc.so.6(+0x3f450) [0x7fcc4a43f450] 2024-04-16T16:15:16.482 INFO:tasks.ceph.mds.a.smithi045.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-16T16:15:16.482 INFO:tasks.ceph.mds.a.smithi045.stderr: 2024-04-16T16:15:16.491 INFO:tasks.ceph.mds.a.smithi045.stderr: -1> 2024-04-16T16:15:16.465+0000 7fcc3e600640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: In function 'void MDLog::trim()' thread 7fcc3e600640 time 2024-04-16T16:15:16.464935+0000 2024-04-16T16:15:16.491 INFO:tasks.ceph.mds.a.smithi045.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: 688: FAILED ceph_assert(num_total_segments >= num_replayed_segments) 2024-04-16T16:15:16.492 INFO:tasks.ceph.mds.a.smithi045.stderr: 2024-04-16T16:15:16.492 INFO:tasks.ceph.mds.a.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:15:16.492 INFO:tasks.ceph.mds.a.smithi045.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fcc4ad81f4c] 2024-04-16T16:15:16.492 INFO:tasks.ceph.mds.a.smithi045.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7fcc4ad8210a] 2024-04-16T16:15:16.492 INFO:tasks.ceph.mds.a.smithi045.stderr: 3: (MDLog::trim()+0xf2a) [0x56024c58f1aa] 2024-04-16T16:15:16.492 INFO:tasks.ceph.mds.a.smithi045.stderr: 4: (MDLog::log_trim_upkeep()+0x339) [0x56024c58f559] 2024-04-16T16:15:16.492 INFO:tasks.ceph.mds.a.smithi045.stderr: 5: /lib64/libstdc++.so.6(+0xdb924) [0x7fcc4a8db924] 2024-04-16T16:15:16.492 INFO:tasks.ceph.mds.a.smithi045.stderr: 6: /lib64/libc.so.6(+0x9f802) [0x7fcc4a49f802] 2024-04-16T16:15:16.492 INFO:tasks.ceph.mds.a.smithi045.stderr: 7: /lib64/libc.so.6(+0x3f450) [0x7fcc4a43f450] 2024-04-16T16:15:16.492 INFO:tasks.ceph.mds.a.smithi045.stderr: 2024-04-16T16:15:16.492 INFO:tasks.ceph.mds.a.smithi045.stderr: 0> 2024-04-16T16:15:16.468+0000 7fcc3e600640 -1 *** Caught signal (Aborted) ** 2024-04-16T16:15:16.492 INFO:tasks.ceph.mds.a.smithi045.stderr: in thread 7fcc3e600640 thread_name:ms_dispatch 2024-04-16T16:15:16.492 INFO:tasks.ceph.mds.a.smithi045.stderr: 2024-04-16T16:15:16.492 INFO:tasks.ceph.mds.a.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:15:16.493 INFO:tasks.ceph.mds.a.smithi045.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fcc4a454db0] 2024-04-16T16:15:16.493 INFO:tasks.ceph.mds.a.smithi045.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fcc4a4a154c] 2024-04-16T16:15:16.493 INFO:tasks.ceph.mds.a.smithi045.stderr: 3: raise() 2024-04-16T16:15:16.493 INFO:tasks.ceph.mds.a.smithi045.stderr: 4: abort() 2024-04-16T16:15:16.493 INFO:tasks.ceph.mds.a.smithi045.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fcc4ad81fa6] 2024-04-16T16:15:16.493 INFO:tasks.ceph.mds.a.smithi045.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7fcc4ad8210a] 2024-04-16T16:15:16.493 INFO:tasks.ceph.mds.a.smithi045.stderr: 7: (MDLog::trim()+0xf2a) [0x56024c58f1aa] 2024-04-16T16:15:16.493 INFO:tasks.ceph.mds.a.smithi045.stderr: 8: (MDLog::log_trim_upkeep()+0x339) [0x56024c58f559] 2024-04-16T16:15:16.493 INFO:tasks.ceph.mds.a.smithi045.stderr: 9: /lib64/libstdc++.so.6(+0xdb924) [0x7fcc4a8db924] 2024-04-16T16:15:16.493 INFO:tasks.ceph.mds.a.smithi045.stderr: 10: /lib64/libc.so.6(+0x9f802) [0x7fcc4a49f802] 2024-04-16T16:15:16.493 INFO:tasks.ceph.mds.a.smithi045.stderr: 11: /lib64/libc.so.6(+0x3f450) [0x7fcc4a43f450] 2024-04-16T16:15:16.493 INFO:tasks.ceph.mds.a.smithi045.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-16T16:15:16.493 INFO:tasks.ceph.mds.a.smithi045.stderr: 2024-04-16T16:15:16.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.574+0000 7f04dfe00640 1 Processor -- start 2024-04-16T16:15:16.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.574+0000 7f04dfe00640 1 -- start start 2024-04-16T16:15:16.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.576+0000 7f04dfe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f04e015e980 0x7f04e00d4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:16.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.576+0000 7f04dfe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f04e00ba500 0x7f04e00d2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:16.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.576+0000 7f04dfe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04e015ed60 con 0x7f04e00cf4d0 2024-04-16T16:15:16.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.576+0000 7f04dfe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f04e00d3170 con 0x7f04e015e980 2024-04-16T16:15:16.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.576+0000 7f04dfe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f04e00d3450 con 0x7f04e00ba500 2024-04-16T16:15:16.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.577+0000 7f04dea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f04e015e980 0x7f04e00d4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:16.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.577+0000 7f04dea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f04e015e980 0x7f04e00d4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56898/0 (socket says 172.21.15.45:56898) 2024-04-16T16:15:16.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.577+0000 7f04dea00640 1 -- 172.21.15.45:0/485784896 learned_addr learned my addr 172.21.15.45:0/485784896 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:16.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.577+0000 7f04d6000640 1 --2- 172.21.15.45:0/485784896 >> v2:172.21.15.103:3300/0 conn(0x7f04e00ba500 0x7f04e00d2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:16.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.577+0000 7f04dea00640 1 -- 172.21.15.45:0/485784896 >> v2:172.21.15.103:3300/0 conn(0x7f04e00ba500 msgr2=0x7f04e00d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:16.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.577+0000 7f04dea00640 1 --2- 172.21.15.45:0/485784896 >> v2:172.21.15.103:3300/0 conn(0x7f04e00ba500 0x7f04e00d2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.577+0000 7f04dea00640 1 -- 172.21.15.45:0/485784896 >> v1:172.21.15.103:6789/0 conn(0x7f04e00cf4d0 legacy=0x7f04e00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:16.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.577+0000 7f04dea00640 1 -- 172.21.15.45:0/485784896 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e00d3730 con 0x7f04e015e980 2024-04-16T16:15:16.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.578+0000 7f04dea00640 1 --2- 172.21.15.45:0/485784896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e015e980 0x7f04e00d4210 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f04e00d5860 tx=0x7f04d002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fcec3253fa15e17 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:16.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.579+0000 7f04de000640 1 -- 172.21.15.45:0/485784896 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0034070 con 0x7f04e015e980 2024-04-16T16:15:16.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.579+0000 7f04de000640 1 -- 172.21.15.45:0/485784896 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04d0030840 con 0x7f04e015e980 2024-04-16T16:15:16.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.579+0000 7f04de000640 1 -- 172.21.15.45:0/485784896 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0030c10 con 0x7f04e015e980 2024-04-16T16:15:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.580+0000 7f04dfe00640 1 -- 172.21.15.45:0/485784896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e015e980 msgr2=0x7f04e00d4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.580+0000 7f04dfe00640 1 --2- 172.21.15.45:0/485784896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e015e980 0x7f04e00d4210 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f04e00d5860 tx=0x7f04d002b980 comp rx=0 tx=0).stop 2024-04-16T16:15:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.580+0000 7f04dfe00640 1 -- 172.21.15.45:0/485784896 shutdown_connections 2024-04-16T16:15:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.580+0000 7f04dfe00640 1 --2- 172.21.15.45:0/485784896 >> v2:172.21.15.103:3300/0 conn(0x7f04e00ba500 0x7f04e00d2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.580+0000 7f04dfe00640 1 --2- 172.21.15.45:0/485784896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e015e980 0x7f04e00d4210 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.580+0000 7f04dfe00640 1 -- 172.21.15.45:0/485784896 >> 172.21.15.45:0/485784896 conn(0x7f04e006d2f0 msgr2=0x7f04e00d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.580+0000 7f04dfe00640 1 -- 172.21.15.45:0/485784896 shutdown_connections 2024-04-16T16:15:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.580+0000 7f04dfe00640 1 -- 172.21.15.45:0/485784896 wait complete. 2024-04-16T16:15:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.582+0000 7f04dfe00640 1 Processor -- start 2024-04-16T16:15:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.582+0000 7f04dfe00640 1 -- start start 2024-04-16T16:15:16.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.583+0000 7f04dfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04e00ba500 0x7f04e01f4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:16.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.583+0000 7f04dfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e00cf4d0 0x7f04e0161180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.583+0000 7f04dfe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04e015e980 0x7f04e0154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.583+0000 7f04dfe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f04e00d2d80 con 0x7f04e00cf4d0 2024-04-16T16:15:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.583+0000 7f04dfe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f04e00d3cf0 con 0x7f04e00ba500 2024-04-16T16:15:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.583+0000 7f04dfe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f04e00d2f00 con 0x7f04e015e980 2024-04-16T16:15:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.583+0000 7f04d6000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e00cf4d0 0x7f04e0161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.584+0000 7f04d6000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e00cf4d0 0x7f04e0161180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56912/0 (socket says 172.21.15.45:56912) 2024-04-16T16:15:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.584+0000 7f04d6000640 1 -- 172.21.15.45:0/175093151 learned_addr learned my addr 172.21.15.45:0/175093151 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.584+0000 7f04dea00640 1 --2- 172.21.15.45:0/175093151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04e00ba500 0x7f04e01f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.584+0000 7f04df400640 1 --2- 172.21.15.45:0/175093151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04e015e980 0x7f04e0154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:16.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.584+0000 7f04d6000640 1 -- 172.21.15.45:0/175093151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04e015e980 msgr2=0x7f04e0154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:16.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.584+0000 7f04d6000640 1 --2- 172.21.15.45:0/175093151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04e015e980 0x7f04e0154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.584+0000 7f04d6000640 1 -- 172.21.15.45:0/175093151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04e00ba500 msgr2=0x7f04e01f4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.584+0000 7f04d6000640 1 --2- 172.21.15.45:0/175093151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04e00ba500 0x7f04e01f4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.584+0000 7f04d6000640 1 -- 172.21.15.45:0/175093151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e0161ad0 con 0x7f04e00cf4d0 2024-04-16T16:15:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.584+0000 7f04d6000640 1 --2- 172.21.15.45:0/175093151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e00cf4d0 0x7f04e0161180 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f04c800d7d0 tx=0x7f04c800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.585+0000 7f04d7400640 1 -- 172.21.15.45:0/175093151 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c8004020 con 0x7f04e00cf4d0 2024-04-16T16:15:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.585+0000 7f04d7400640 1 -- 172.21.15.45:0/175093151 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04c80041e0 con 0x7f04e00cf4d0 2024-04-16T16:15:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.585+0000 7f04d7400640 1 -- 172.21.15.45:0/175093151 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c80043d0 con 0x7f04e00cf4d0 2024-04-16T16:15:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.585+0000 7f04dfe00640 1 -- 172.21.15.45:0/175093151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04e0161db0 con 0x7f04e00cf4d0 2024-04-16T16:15:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.585+0000 7f04dfe00640 1 -- 172.21.15.45:0/175093151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04e0154120 con 0x7f04e00cf4d0 2024-04-16T16:15:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.586+0000 7f04dfe00640 1 -- 172.21.15.45:0/175093151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04e0073dc0 con 0x7f04e00cf4d0 2024-04-16T16:15:16.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.587+0000 7f04d7400640 1 -- 172.21.15.45:0/175093151 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f04c801b020 con 0x7f04e00cf4d0 2024-04-16T16:15:16.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.588+0000 7f04d7400640 1 --2- 172.21.15.45:0/175093151 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f04bc06ca60 0x7f04bc06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:16.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.588+0000 7f04d7400640 1 -- 172.21.15.45:0/175093151 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6041+0+0 (secure 0 0 0) 0x7f04c808bcf0 con 0x7f04e00cf4d0 2024-04-16T16:15:16.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.589+0000 7f04dea00640 1 --2- 172.21.15.45:0/175093151 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f04bc06ca60 0x7f04bc06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:16.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.590+0000 7f04dea00640 1 --2- 172.21.15.45:0/175093151 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f04bc06ca60 0x7f04bc06ef20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f04d0036000 tx=0x7f04d003b040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:16.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.593+0000 7f04d7400640 1 -- 172.21.15.45:0/175093151 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04c8059eb0 con 0x7f04e00cf4d0 2024-04-16T16:15:16.626 INFO:tasks.ceph.mgr.x.smithi103.stderr:Exception in thread Thread-4: 2024-04-16T16:15:16.626 INFO:tasks.ceph.mgr.x.smithi103.stderr:Traceback (most recent call last): 2024-04-16T16:15:16.626 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner 2024-04-16T16:15:16.639 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.run() 2024-04-16T16:15:16.639 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 1306, in run 2024-04-16T16:15:16.639 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.function(*self.args, **self.kwargs) 2024-04-16T16:15:16.639 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/share/ceph/mgr/stats/fs/perf_stats.py", line 222, in re_register_queries 2024-04-16T16:15:16.640 INFO:tasks.ceph.mgr.x.smithi103.stderr: if self.mx_last_updated >= ua_last_updated: 2024-04-16T16:15:16.640 INFO:tasks.ceph.mgr.x.smithi103.stderr:AttributeError: 'FSPerfStats' object has no attribute 'mx_last_updated' 2024-04-16T16:15:16.779 INFO:tasks.ceph.mds.a.smithi045.stderr:daemon-helper: command crashed with signal 6 2024-04-16T16:15:16.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:16.837+0000 7f04dfe00640 1 -- 172.21.15.45:0/175093151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "fs1", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f04e0073ae0 con 0x7f04e00cf4d0 2024-04-16T16:15:17.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.428+0000 7f04d7400640 1 -- 172.21.15.45:0/175093151 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "fs1", "var": "standby_count_wanted", "val": "0"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7f04c8060080 con 0x7f04e00cf4d0 2024-04-16T16:15:17.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.432+0000 7f04dfe00640 1 -- 172.21.15.45:0/175093151 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f04bc06ca60 msgr2=0x7f04bc06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:17.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.432+0000 7f04dfe00640 1 --2- 172.21.15.45:0/175093151 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f04bc06ca60 0x7f04bc06ef20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f04d0036000 tx=0x7f04d003b040 comp rx=0 tx=0).stop 2024-04-16T16:15:17.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.432+0000 7f04dfe00640 1 -- 172.21.15.45:0/175093151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e00cf4d0 msgr2=0x7f04e0161180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:17.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.432+0000 7f04dfe00640 1 --2- 172.21.15.45:0/175093151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e00cf4d0 0x7f04e0161180 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f04c800d7d0 tx=0x7f04c800dc90 comp rx=0 tx=0).stop 2024-04-16T16:15:17.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.432+0000 7f04dfe00640 1 -- 172.21.15.45:0/175093151 shutdown_connections 2024-04-16T16:15:17.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.432+0000 7f04dfe00640 1 --2- 172.21.15.45:0/175093151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04e015e980 0x7f04e0154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:17.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.432+0000 7f04dfe00640 1 --2- 172.21.15.45:0/175093151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e00cf4d0 0x7f04e0161180 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:17.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.432+0000 7f04dfe00640 1 --2- 172.21.15.45:0/175093151 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f04bc06ca60 0x7f04bc06ef20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:17.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.433+0000 7f04dfe00640 1 --2- 172.21.15.45:0/175093151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04e00ba500 0x7f04e01f4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:17.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.433+0000 7f04dfe00640 1 -- 172.21.15.45:0/175093151 >> 172.21.15.45:0/175093151 conn(0x7f04e006d2f0 msgr2=0x7f04e00b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:17.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.433+0000 7f04dfe00640 1 -- 172.21.15.45:0/175093151 shutdown_connections 2024-04-16T16:15:17.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.433+0000 7f04dfe00640 1 -- 172.21.15.45:0/175093151 wait complete. 2024-04-16T16:15:17.455 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-16T16:15:17.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set fs1 allow_standby_replay false 2024-04-16T16:15:17.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.730+0000 7f79b7000640 1 Processor -- start 2024-04-16T16:15:17.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.731+0000 7f79b7000640 1 -- start start 2024-04-16T16:15:17.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.732+0000 7f79b7000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f79b015b920 0x7f79b00b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:17.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.732+0000 7f79b7000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79b00d0e80 0x7f79b006ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:17.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.732+0000 7f79b7000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79b00ccb70 con 0x7f79b00afda0 2024-04-16T16:15:17.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.732+0000 7f79b7000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f79b006e7a0 con 0x7f79b015b920 2024-04-16T16:15:17.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.732+0000 7f79b7000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f79b006f170 con 0x7f79b00d0e80 2024-04-16T16:15:17.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.733+0000 7f79af400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f79b00afda0 0x7f79b00d0770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42022/0 (socket says 172.21.15.45:42022) 2024-04-16T16:15:17.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.733+0000 7f79af400640 1 -- 172.21.15.45:0/3365614677 learned_addr learned my addr 172.21.15.45:0/3365614677 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:17.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.733+0000 7f79afe00640 1 --2- 172.21.15.45:0/3365614677 >> v2:172.21.15.103:3300/0 conn(0x7f79b015b920 0x7f79b00b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:17.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.733+0000 7f79b5000640 1 --2- 172.21.15.45:0/3365614677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79b00d0e80 0x7f79b006ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:17.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.733+0000 7f79aea00640 1 -- 172.21.15.45:0/3365614677 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2429739529 0 0) 0x7f79b00ccb70 con 0x7f79b00afda0 2024-04-16T16:15:17.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.733+0000 7f79aea00640 1 -- 172.21.15.45:0/3365614677 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7994003520 con 0x7f79b00afda0 2024-04-16T16:15:17.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.734+0000 7f79aea00640 1 -- 172.21.15.45:0/3365614677 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3480503779 0 0) 0x7f7994003520 con 0x7f79b00afda0 2024-04-16T16:15:17.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.734+0000 7f79aea00640 1 -- 172.21.15.45:0/3365614677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79b00d0e80 msgr2=0x7f79b006ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:17.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.734+0000 7f79aea00640 1 --2- 172.21.15.45:0/3365614677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79b00d0e80 0x7f79b006ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:17.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.734+0000 7f79aea00640 1 -- 172.21.15.45:0/3365614677 >> v2:172.21.15.103:3300/0 conn(0x7f79b015b920 msgr2=0x7f79b00b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:17.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.734+0000 7f79aea00640 1 --2- 172.21.15.45:0/3365614677 >> v2:172.21.15.103:3300/0 conn(0x7f79b015b920 0x7f79b00b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:17.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.734+0000 7f79aea00640 1 -- 172.21.15.45:0/3365614677 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b00cce50 con 0x7f79b00afda0 2024-04-16T16:15:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.734+0000 7f79aea00640 1 -- 172.21.15.45:0/3365614677 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f799c001670 con 0x7f79b00afda0 2024-04-16T16:15:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.734+0000 7f79aea00640 1 -- 172.21.15.45:0/3365614677 >> v1:172.21.15.45:6789/0 conn(0x7f79b00afda0 legacy=0x7f79b00d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.734+0000 7f79aea00640 1 --2- 172.21.15.45:0/3365614677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79940039b0 0x7f7994003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.734+0000 7f79aea00640 1 --2- 172.21.15.45:0/3365614677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7994004800 0x7f7994004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.734+0000 7f79aea00640 1 --2- 172.21.15.45:0/3365614677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79b00d0e80 0x7f79940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.734+0000 7f79aea00640 1 -- 172.21.15.45:0/3365614677 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f79b006e7a0 con 0x7f79940039b0 2024-04-16T16:15:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.734+0000 7f79aea00640 1 -- 172.21.15.45:0/3365614677 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79b006f170 con 0x7f7994004800 2024-04-16T16:15:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.734+0000 7f79aea00640 1 -- 172.21.15.45:0/3365614677 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f799400ba40 con 0x7f79b00d0e80 2024-04-16T16:15:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.734+0000 7f79afe00640 1 --2- 172.21.15.45:0/3365614677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79940039b0 0x7f7994003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.735+0000 7f79af400640 1 --2- 172.21.15.45:0/3365614677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7994004800 0x7f7994004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.735+0000 7f79b5000640 1 --2- 172.21.15.45:0/3365614677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79b00d0e80 0x7f79940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.735+0000 7f79afe00640 1 -- 172.21.15.45:0/3365614677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79b00d0e80 msgr2=0x7f79940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.735+0000 7f79afe00640 1 --2- 172.21.15.45:0/3365614677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79b00d0e80 0x7f79940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.735+0000 7f79afe00640 1 -- 172.21.15.45:0/3365614677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7994004800 msgr2=0x7f7994004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.735+0000 7f79afe00640 1 --2- 172.21.15.45:0/3365614677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7994004800 0x7f7994004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:17.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.735+0000 7f79afe00640 1 -- 172.21.15.45:0/3365614677 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7994003520 con 0x7f79940039b0 2024-04-16T16:15:17.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.735+0000 7f79b5000640 1 --2- 172.21.15.45:0/3365614677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79b00d0e80 0x7f79940051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:17.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.736+0000 7f79afe00640 1 --2- 172.21.15.45:0/3365614677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79940039b0 0x7f7994003dd0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f79a0009fc0 tx=0x7f7994005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7004603d9e85056c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:17.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.736+0000 7f79aea00640 1 -- 172.21.15.45:0/3365614677 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a0039070 con 0x7f79940039b0 2024-04-16T16:15:17.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.736+0000 7f79aea00640 1 -- 172.21.15.45:0/3365614677 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79a002cc60 con 0x7f79940039b0 2024-04-16T16:15:17.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.736+0000 7f79aea00640 1 -- 172.21.15.45:0/3365614677 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a0009c10 con 0x7f79940039b0 2024-04-16T16:15:17.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.737+0000 7f79b7000640 1 -- 172.21.15.45:0/3365614677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79940039b0 msgr2=0x7f7994003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:17.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.737+0000 7f79b7000640 1 --2- 172.21.15.45:0/3365614677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79940039b0 0x7f7994003dd0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f79a0009fc0 tx=0x7f7994005830 comp rx=0 tx=0).stop 2024-04-16T16:15:17.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.737+0000 7f79b5000640 1 -- 172.21.15.45:0/3365614677 reap_dead start 2024-04-16T16:15:17.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.737+0000 7f79b7000640 1 -- 172.21.15.45:0/3365614677 shutdown_connections 2024-04-16T16:15:17.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.737+0000 7f79b7000640 1 -- 172.21.15.45:0/3365614677 >> 172.21.15.45:0/3365614677 conn(0x7f79b00b3970 msgr2=0x7f79b00b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:17.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.737+0000 7f79b7000640 1 -- 172.21.15.45:0/3365614677 shutdown_connections 2024-04-16T16:15:17.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.737+0000 7f79b7000640 1 -- 172.21.15.45:0/3365614677 wait complete. 2024-04-16T16:15:17.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.739+0000 7f79b7000640 1 Processor -- start 2024-04-16T16:15:17.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.740+0000 7f79b7000640 1 -- start start 2024-04-16T16:15:17.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.741+0000 7f79b7000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79940039b0 0x7f79b00cf860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:17.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.741+0000 7f79b7000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79b00b0550 0x7f79b00b09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:17.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.741+0000 7f79b7000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79b015c8c0 0x7f79b015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:17.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.741+0000 7f79b7000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f79b006edb0 con 0x7f79940039b0 2024-04-16T16:15:17.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.741+0000 7f79b7000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79b006ec30 con 0x7f79b015c8c0 2024-04-16T16:15:17.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.741+0000 7f79b7000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f79b006ef30 con 0x7f79b00b0550 2024-04-16T16:15:17.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.741+0000 7f79afe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79940039b0 0x7f79b00cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:17.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.741+0000 7f79afe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79940039b0 0x7f79b00cf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56930/0 (socket says 172.21.15.45:56930) 2024-04-16T16:15:17.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.741+0000 7f79afe00640 1 -- 172.21.15.45:0/2286721891 learned_addr learned my addr 172.21.15.45:0/2286721891 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.742+0000 7f79b5000640 1 --2- 172.21.15.45:0/2286721891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79b015c8c0 0x7f79b015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.742+0000 7f79af400640 1 --2- 172.21.15.45:0/2286721891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79b00b0550 0x7f79b00b09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.742+0000 7f79afe00640 1 -- 172.21.15.45:0/2286721891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79b00b0550 msgr2=0x7f79b00b09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.742+0000 7f79afe00640 1 --2- 172.21.15.45:0/2286721891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79b00b0550 0x7f79b00b09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.742+0000 7f79afe00640 1 -- 172.21.15.45:0/2286721891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79b015c8c0 msgr2=0x7f79b015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.742+0000 7f79afe00640 1 --2- 172.21.15.45:0/2286721891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79b015c8c0 0x7f79b015cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.742+0000 7f79afe00640 1 -- 172.21.15.45:0/2286721891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b00b15b0 con 0x7f79940039b0 2024-04-16T16:15:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.742+0000 7f79afe00640 1 --2- 172.21.15.45:0/2286721891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79940039b0 0x7f79b00cf860 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f79a0037040 tx=0x7f79a0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.742+0000 7f79a7e00640 1 -- 172.21.15.45:0/2286721891 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a0039050 con 0x7f79940039b0 2024-04-16T16:15:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.742+0000 7f79a7e00640 1 -- 172.21.15.45:0/2286721891 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79a003a040 con 0x7f79940039b0 2024-04-16T16:15:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.742+0000 7f79b7000640 1 -- 172.21.15.45:0/2286721891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b00b0ff0 con 0x7f79940039b0 2024-04-16T16:15:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.742+0000 7f79a7e00640 1 -- 172.21.15.45:0/2286721891 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a0009c10 con 0x7f79940039b0 2024-04-16T16:15:17.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.743+0000 7f79b7000640 1 -- 172.21.15.45:0/2286721891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b00b1d80 con 0x7f79940039b0 2024-04-16T16:15:17.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.744+0000 7f79b7000640 1 -- 172.21.15.45:0/2286721891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79b00b13e0 con 0x7f79940039b0 2024-04-16T16:15:17.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.745+0000 7f79a7e00640 1 -- 172.21.15.45:0/2286721891 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f79a0047030 con 0x7f79940039b0 2024-04-16T16:15:17.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.746+0000 7f79a7e00640 1 --2- 172.21.15.45:0/2286721891 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f799006ca60 0x7f799006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:17.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.746+0000 7f79a7e00640 1 -- 172.21.15.45:0/2286721891 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6041+0+0 (secure 0 0 0) 0x7f79a00ae1d0 con 0x7f79940039b0 2024-04-16T16:15:17.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.746+0000 7f79af400640 1 --2- 172.21.15.45:0/2286721891 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f799006ca60 0x7f799006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:17.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.748+0000 7f79af400640 1 --2- 172.21.15.45:0/2286721891 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f799006ca60 0x7f799006ef20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f79b015c6b0 tx=0x7f799c000ff0 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:17.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.750+0000 7f79a7e00640 1 -- 172.21.15.45:0/2286721891 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f79a004e060 con 0x7f79940039b0 2024-04-16T16:15:17.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:17.997+0000 7f79b7000640 1 -- 172.21.15.45:0/2286721891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "fs1", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f79b00d1660 con 0x7f79940039b0 2024-04-16T16:15:18.429 INFO:tasks.ceph.mgr.x.smithi103.stderr:Exception in thread Thread-5: 2024-04-16T16:15:18.429 INFO:tasks.ceph.mgr.x.smithi103.stderr:Traceback (most recent call last): 2024-04-16T16:15:18.429 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner 2024-04-16T16:15:18.430 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.run() 2024-04-16T16:15:18.430 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 1306, in run 2024-04-16T16:15:18.430 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.function(*self.args, **self.kwargs) 2024-04-16T16:15:18.430 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/share/ceph/mgr/stats/fs/perf_stats.py", line 222, in re_register_queries 2024-04-16T16:15:18.430 INFO:tasks.ceph.mgr.x.smithi103.stderr: if self.mx_last_updated >= ua_last_updated: 2024-04-16T16:15:18.430 INFO:tasks.ceph.mgr.x.smithi103.stderr:AttributeError: 'FSPerfStats' object has no attribute 'mx_last_updated' 2024-04-16T16:15:18.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.434+0000 7f79a7e00640 1 -- 172.21.15.45:0/2286721891 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "fs1", "var": "allow_standby_replay", "val": "false"}]=0 v23) ==== 119+0+0 (secure 0 0 0) 0x7f79b00d1660 con 0x7f79940039b0 2024-04-16T16:15:18.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.438+0000 7f79b7000640 1 -- 172.21.15.45:0/2286721891 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f799006ca60 msgr2=0x7f799006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:18.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.438+0000 7f79b7000640 1 --2- 172.21.15.45:0/2286721891 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f799006ca60 0x7f799006ef20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f79b015c6b0 tx=0x7f799c000ff0 comp rx=0 tx=0).stop 2024-04-16T16:15:18.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.438+0000 7f79b7000640 1 -- 172.21.15.45:0/2286721891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79940039b0 msgr2=0x7f79b00cf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:18.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.438+0000 7f79b7000640 1 --2- 172.21.15.45:0/2286721891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79940039b0 0x7f79b00cf860 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f79a0037040 tx=0x7f79a0009750 comp rx=0 tx=0).stop 2024-04-16T16:15:18.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.438+0000 7f79b7000640 1 -- 172.21.15.45:0/2286721891 shutdown_connections 2024-04-16T16:15:18.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.438+0000 7f79b7000640 1 --2- 172.21.15.45:0/2286721891 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f799006ca60 0x7f799006ef20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:18.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.438+0000 7f79b7000640 1 --2- 172.21.15.45:0/2286721891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79b015c8c0 0x7f79b015cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:18.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.438+0000 7f79b7000640 1 --2- 172.21.15.45:0/2286721891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79b00b0550 0x7f79b00b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:18.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.438+0000 7f79b7000640 1 --2- 172.21.15.45:0/2286721891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79940039b0 0x7f79b00cf860 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:18.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.438+0000 7f79b7000640 1 -- 172.21.15.45:0/2286721891 >> 172.21.15.45:0/2286721891 conn(0x7f79b00b3970 msgr2=0x7f79b00b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:18.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.438+0000 7f79b7000640 1 -- 172.21.15.45:0/2286721891 shutdown_connections 2024-04-16T16:15:18.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.438+0000 7f79b7000640 1 -- 172.21.15.45:0/2286721891 wait complete. 2024-04-16T16:15:18.462 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'fs1' 2024-04-16T16:15:18.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:15:18.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.684+0000 7f3759000640 1 Processor -- start 2024-04-16T16:15:18.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.684+0000 7f3759000640 1 -- start start 2024-04-16T16:15:18.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.686+0000 7f3759000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f375406e550 0x7f375409abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:18.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.686+0000 7f3759000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f37540d1990 0x7f375409b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:18.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.686+0000 7f3759000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37540ccf50 con 0x7f3754070420 2024-04-16T16:15:18.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.686+0000 7f3759000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f37540cd880 con 0x7f37540d1990 2024-04-16T16:15:18.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.686+0000 7f3759000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f37540cd230 con 0x7f375406e550 2024-04-16T16:15:18.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.686+0000 7f3752800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f37540d1990 0x7f375409b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:18.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.686+0000 7f3752800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f37540d1990 0x7f375409b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56938/0 (socket says 172.21.15.45:56938) 2024-04-16T16:15:18.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.686+0000 7f3752800640 1 -- 172.21.15.45:0/521506925 learned_addr learned my addr 172.21.15.45:0/521506925 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:18.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.686+0000 7f3751e00640 1 --2- 172.21.15.45:0/521506925 >> v2:172.21.15.103:3300/0 conn(0x7f375406e550 0x7f375409abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:18.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.687+0000 7f3752800640 1 -- 172.21.15.45:0/521506925 >> v2:172.21.15.103:3300/0 conn(0x7f375406e550 msgr2=0x7f375409abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:18.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.687+0000 7f3752800640 1 --2- 172.21.15.45:0/521506925 >> v2:172.21.15.103:3300/0 conn(0x7f375406e550 0x7f375409abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:18.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.687+0000 7f3752800640 1 -- 172.21.15.45:0/521506925 >> v1:172.21.15.45:6789/0 conn(0x7f3754070420 legacy=0x7f37540cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:18.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.687+0000 7f3752800640 1 -- 172.21.15.45:0/521506925 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37540cd510 con 0x7f37540d1990 2024-04-16T16:15:18.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.688+0000 7f3752800640 1 --2- 172.21.15.45:0/521506925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37540d1990 0x7f375409b100 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f375406c800 tx=0x7f374c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa60e88b570556c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:18.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.688+0000 7f3750a00640 1 -- 172.21.15.45:0/521506925 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c02cb20 con 0x7f37540d1990 2024-04-16T16:15:18.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.688+0000 7f3750a00640 1 -- 172.21.15.45:0/521506925 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f374c02ccc0 con 0x7f37540d1990 2024-04-16T16:15:18.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.688+0000 7f3750a00640 1 -- 172.21.15.45:0/521506925 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c034610 con 0x7f37540d1990 2024-04-16T16:15:18.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.689+0000 7f3759000640 1 -- 172.21.15.45:0/521506925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37540d1990 msgr2=0x7f375409b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:18.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.689+0000 7f3759000640 1 --2- 172.21.15.45:0/521506925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37540d1990 0x7f375409b100 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f375406c800 tx=0x7f374c02bf80 comp rx=0 tx=0).stop 2024-04-16T16:15:18.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.689+0000 7f3759000640 1 -- 172.21.15.45:0/521506925 shutdown_connections 2024-04-16T16:15:18.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.689+0000 7f3759000640 1 --2- 172.21.15.45:0/521506925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37540d1990 0x7f375409b100 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:18.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.689+0000 7f3759000640 1 --2- 172.21.15.45:0/521506925 >> v2:172.21.15.103:3300/0 conn(0x7f375406e550 0x7f375409abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:18.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.689+0000 7f3759000640 1 -- 172.21.15.45:0/521506925 >> 172.21.15.45:0/521506925 conn(0x7f37540b4350 msgr2=0x7f37540b4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:18.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.689+0000 7f3759000640 1 -- 172.21.15.45:0/521506925 shutdown_connections 2024-04-16T16:15:18.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.689+0000 7f3759000640 1 -- 172.21.15.45:0/521506925 wait complete. 2024-04-16T16:15:18.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.691+0000 7f3759000640 1 Processor -- start 2024-04-16T16:15:18.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.691+0000 7f3759000640 1 -- start start 2024-04-16T16:15:18.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.692+0000 7f3759000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f375406e550 0x7f37541e1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:18.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.692+0000 7f3759000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3754070420 0x7f37540c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:18.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.692+0000 7f3759000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f37540d1990 0x7f37540b0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:18.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.692+0000 7f3759000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f375406ca40 con 0x7f3754070420 2024-04-16T16:15:18.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.692+0000 7f3759000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f375406e930 con 0x7f375406e550 2024-04-16T16:15:18.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.692+0000 7f3759000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f37541e86a0 con 0x7f37540d1990 2024-04-16T16:15:18.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.693+0000 7f3751400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3754070420 0x7f37540c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:18.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.693+0000 7f3752800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f37540d1990 0x7f37540b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:18.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.693+0000 7f3751400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3754070420 0x7f37540c9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56946/0 (socket says 172.21.15.45:56946) 2024-04-16T16:15:18.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.693+0000 7f3751400640 1 -- 172.21.15.45:0/1198154359 learned_addr learned my addr 172.21.15.45:0/1198154359 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:18.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.693+0000 7f3751e00640 1 --2- 172.21.15.45:0/1198154359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f375406e550 0x7f37541e1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:18.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.693+0000 7f3751400640 1 -- 172.21.15.45:0/1198154359 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f37540d1990 msgr2=0x7f37540b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:18.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.693+0000 7f3751400640 1 --2- 172.21.15.45:0/1198154359 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f37540d1990 0x7f37540b0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:18.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.693+0000 7f3751400640 1 -- 172.21.15.45:0/1198154359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f375406e550 msgr2=0x7f37541e1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:18.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.693+0000 7f3751400640 1 --2- 172.21.15.45:0/1198154359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f375406e550 0x7f37541e1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:18.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.693+0000 7f3751400640 1 -- 172.21.15.45:0/1198154359 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37541e7a90 con 0x7f3754070420 2024-04-16T16:15:18.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.693+0000 7f3751e00640 1 --2- 172.21.15.45:0/1198154359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f375406e550 0x7f37541e1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:15:18.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.694+0000 7f3751400640 1 --2- 172.21.15.45:0/1198154359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3754070420 0x7f37540c9930 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f374000c400 tx=0x7f374000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:18.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.694+0000 7f374a000640 1 -- 172.21.15.45:0/1198154359 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37400052d0 con 0x7f3754070420 2024-04-16T16:15:18.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.694+0000 7f374a000640 1 -- 172.21.15.45:0/1198154359 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3740005490 con 0x7f3754070420 2024-04-16T16:15:18.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.694+0000 7f374a000640 1 -- 172.21.15.45:0/1198154359 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3740007260 con 0x7f3754070420 2024-04-16T16:15:18.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.694+0000 7f3759000640 1 -- 172.21.15.45:0/1198154359 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37541e7d70 con 0x7f3754070420 2024-04-16T16:15:18.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.694+0000 7f3759000640 1 -- 172.21.15.45:0/1198154359 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37541e8330 con 0x7f3754070420 2024-04-16T16:15:18.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.696+0000 7f3759000640 1 -- 172.21.15.45:0/1198154359 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37540d0560 con 0x7f3754070420 2024-04-16T16:15:18.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.697+0000 7f374a000640 1 -- 172.21.15.45:0/1198154359 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f3740007400 con 0x7f3754070420 2024-04-16T16:15:18.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.698+0000 7f374a000640 1 --2- 172.21.15.45:0/1198154359 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f373406ca60 0x7f373406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:18.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.698+0000 7f374a000640 1 -- 172.21.15.45:0/1198154359 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6041+0+0 (secure 0 0 0) 0x7f374008b490 con 0x7f3754070420 2024-04-16T16:15:18.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.699+0000 7f3751e00640 1 --2- 172.21.15.45:0/1198154359 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f373406ca60 0x7f373406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:18.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.700+0000 7f3751e00640 1 --2- 172.21.15.45:0/1198154359 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f373406ca60 0x7f373406ef20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3744006fd0 tx=0x7f3744008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:18.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.703+0000 7f374a000640 1 -- 172.21.15.45:0/1198154359 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f37540d0560 con 0x7f3754070420 2024-04-16T16:15:18.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.956+0000 7f3759000640 1 -- 172.21.15.45:0/1198154359 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37541e8820 con 0x7f3754070420 2024-04-16T16:15:18.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.958+0000 7f374a000640 1 -- 172.21.15.45:0/1198154359 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5175 (secure 0 0 0) 0x7f3740058ee0 con 0x7f3754070420 2024-04-16T16:15:18.957 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:18.957 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/814837411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":814837411},{"type":"v1","addr":"172.21.15.103:6836","nonce":814837411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/401733314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":401733314},{"type":"v1","addr":"172.21.15.103:6837","nonce":401733314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5073,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/768816269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":768816269},{"type":"v1","addr":"172.21.15.45:6834","nonce":768816269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:14.878986+0000","modified":"2024-04-16T16:15:17.998813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5520":{"gid":5520,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6835/2276311244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2276311244},{"type":"v1","addr":"172.21.15.45:6835","nonce":2276311244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"fs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5520,"qdb_cluster":[5520]},"id":3}]} 2024-04-16T16:15:18.958 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 23 2024-04-16T16:15:18.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.961+0000 7f3759000640 1 -- 172.21.15.45:0/1198154359 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f373406ca60 msgr2=0x7f373406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:18.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.961+0000 7f3759000640 1 --2- 172.21.15.45:0/1198154359 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f373406ca60 0x7f373406ef20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3744006fd0 tx=0x7f3744008040 comp rx=0 tx=0).stop 2024-04-16T16:15:18.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.961+0000 7f3759000640 1 -- 172.21.15.45:0/1198154359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3754070420 msgr2=0x7f37540c9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:18.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.961+0000 7f3759000640 1 --2- 172.21.15.45:0/1198154359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3754070420 0x7f37540c9930 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f374000c400 tx=0x7f374000c8c0 comp rx=0 tx=0).stop 2024-04-16T16:15:18.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.961+0000 7f3759000640 1 -- 172.21.15.45:0/1198154359 shutdown_connections 2024-04-16T16:15:18.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.961+0000 7f3759000640 1 --2- 172.21.15.45:0/1198154359 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f37540d1990 0x7f37540b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:18.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.961+0000 7f3759000640 1 --2- 172.21.15.45:0/1198154359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3754070420 0x7f37540c9930 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:18.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.961+0000 7f3759000640 1 --2- 172.21.15.45:0/1198154359 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f373406ca60 0x7f373406ef20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:18.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.961+0000 7f3759000640 1 --2- 172.21.15.45:0/1198154359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f375406e550 0x7f37541e1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:18.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.961+0000 7f3759000640 1 -- 172.21.15.45:0/1198154359 >> 172.21.15.45:0/1198154359 conn(0x7f37540b4350 msgr2=0x7f37540cf720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:18.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.961+0000 7f3759000640 1 -- 172.21.15.45:0/1198154359 shutdown_connections 2024-04-16T16:15:18.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:18.961+0000 7f3759000640 1 -- 172.21.15.45:0/1198154359 wait complete. 2024-04-16T16:15:18.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:15:19.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.193+0000 7f400ae00640 1 Processor -- start 2024-04-16T16:15:19.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.194+0000 7f400ae00640 1 -- start start 2024-04-16T16:15:19.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.195+0000 7f400ae00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4004162270 0x7f40040d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:19.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.195+0000 7f400ae00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40040d3080 con 0x7f4004161e90 2024-04-16T16:15:19.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.195+0000 7f400ae00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40040d3360 con 0x7f40040d52d0 2024-04-16T16:15:19.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.195+0000 7f400ae00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f40040d2e30 con 0x7f4004162270 2024-04-16T16:15:19.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.196+0000 7f4009a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4004161e90 0x7f40040cf7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42046/0 (socket says 172.21.15.45:42046) 2024-04-16T16:15:19.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.196+0000 7f4009a00640 1 -- 172.21.15.45:0/3322702425 learned_addr learned my addr 172.21.15.45:0/3322702425 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:19.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.196+0000 7f4009000640 1 --2- 172.21.15.45:0/3322702425 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4004162270 0x7f40040d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:19.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.196+0000 7f4003e00640 1 -- 172.21.15.45:0/3322702425 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 828529384 0 0) 0x7f40040d3080 con 0x7f4004161e90 2024-04-16T16:15:19.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.196+0000 7f4003e00640 1 -- 172.21.15.45:0/3322702425 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fec003520 con 0x7f4004161e90 2024-04-16T16:15:19.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.197+0000 7f4003e00640 1 -- 172.21.15.45:0/3322702425 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4048415946 0 0) 0x7f3fec003520 con 0x7f4004161e90 2024-04-16T16:15:19.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.197+0000 7f4003e00640 1 -- 172.21.15.45:0/3322702425 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4004162270 msgr2=0x7f40040d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:19.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.197+0000 7f4003e00640 1 --2- 172.21.15.45:0/3322702425 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4004162270 0x7f40040d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.197+0000 7f4003e00640 1 -- 172.21.15.45:0/3322702425 >> v1:172.21.15.103:6789/0 conn(0x7f40040d52d0 legacy=0x7f400406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:19.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.197+0000 7f4003e00640 1 -- 172.21.15.45:0/3322702425 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40040d3640 con 0x7f4004161e90 2024-04-16T16:15:19.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.197+0000 7f4003e00640 1 -- 172.21.15.45:0/3322702425 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f3ff8001670 con 0x7f4004161e90 2024-04-16T16:15:19.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.197+0000 7f4003e00640 1 -- 172.21.15.45:0/3322702425 >> v1:172.21.15.45:6789/0 conn(0x7f4004161e90 legacy=0x7f40040cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:19.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.197+0000 7f4003e00640 1 --2- 172.21.15.45:0/3322702425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fec0039e0 0x7f3fec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:19.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.197+0000 7f4003e00640 1 --2- 172.21.15.45:0/3322702425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fec004800 0x7f3fec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:19.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.197+0000 7f4003e00640 1 --2- 172.21.15.45:0/3322702425 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4004162270 0x7f3fec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:19.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.197+0000 7f4003e00640 1 -- 172.21.15.45:0/3322702425 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f40040d2e30 con 0x7f3fec004800 2024-04-16T16:15:19.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.197+0000 7f4003e00640 1 -- 172.21.15.45:0/3322702425 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec00bb30 con 0x7f3fec0039e0 2024-04-16T16:15:19.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.197+0000 7f4003e00640 1 -- 172.21.15.45:0/3322702425 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3fec00bd30 con 0x7f4004162270 2024-04-16T16:15:19.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.197+0000 7f4009000640 1 --2- 172.21.15.45:0/3322702425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fec004800 0x7f3fec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:19.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.198+0000 7f4009a00640 1 --2- 172.21.15.45:0/3322702425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fec0039e0 0x7f3fec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:19.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.198+0000 7f4009000640 1 -- 172.21.15.45:0/3322702425 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4004162270 msgr2=0x7f3fec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.198+0000 7f400a400640 1 --2- 172.21.15.45:0/3322702425 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4004162270 0x7f3fec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.198+0000 7f4009000640 1 --2- 172.21.15.45:0/3322702425 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4004162270 0x7f3fec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.198+0000 7f4009000640 1 -- 172.21.15.45:0/3322702425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fec0039e0 msgr2=0x7f3fec003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.198+0000 7f4009000640 1 --2- 172.21.15.45:0/3322702425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fec0039e0 0x7f3fec003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.198+0000 7f4009000640 1 -- 172.21.15.45:0/3322702425 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40040d3640 con 0x7f3fec004800 2024-04-16T16:15:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.198+0000 7f400a400640 1 --2- 172.21.15.45:0/3322702425 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4004162270 0x7f3fec0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.198+0000 7f4009a00640 1 --2- 172.21.15.45:0/3322702425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fec0039e0 0x7f3fec003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.199+0000 7f4009000640 1 --2- 172.21.15.45:0/3322702425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fec004800 0x7f3fec004c80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0009ac0 tx=0x7f3fec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d894e7f34dc01589 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.199+0000 7f4003e00640 1 -- 172.21.15.45:0/3322702425 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0039070 con 0x7f3fec004800 2024-04-16T16:15:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.199+0000 7f4003e00640 1 -- 172.21.15.45:0/3322702425 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff002c6b0 con 0x7f3fec004800 2024-04-16T16:15:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.199+0000 7f4003e00640 1 -- 172.21.15.45:0/3322702425 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0009690 con 0x7f3fec004800 2024-04-16T16:15:19.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.200+0000 7f400ae00640 1 -- 172.21.15.45:0/3322702425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fec004800 msgr2=0x7f3fec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:19.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.200+0000 7f400ae00640 1 --2- 172.21.15.45:0/3322702425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fec004800 0x7f3fec004c80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0009ac0 tx=0x7f3fec005830 comp rx=0 tx=0).stop 2024-04-16T16:15:19.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.200+0000 7f400a400640 1 -- 172.21.15.45:0/3322702425 reap_dead start 2024-04-16T16:15:19.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.200+0000 7f400ae00640 1 -- 172.21.15.45:0/3322702425 shutdown_connections 2024-04-16T16:15:19.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.200+0000 7f400ae00640 1 -- 172.21.15.45:0/3322702425 >> 172.21.15.45:0/3322702425 conn(0x7f40040d1940 msgr2=0x7f40040d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:19.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.200+0000 7f400ae00640 1 -- 172.21.15.45:0/3322702425 shutdown_connections 2024-04-16T16:15:19.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.200+0000 7f400ae00640 1 -- 172.21.15.45:0/3322702425 wait complete. 2024-04-16T16:15:19.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.202+0000 7f400ae00640 1 Processor -- start 2024-04-16T16:15:19.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.202+0000 7f400ae00640 1 -- start start 2024-04-16T16:15:19.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.203+0000 7f400ae00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fec004800 0x7f4004156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:19.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.203+0000 7f400ae00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40041f4970 0x7f400415d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:19.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.203+0000 7f400ae00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f400415de50 0x7f40040cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:19.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.203+0000 7f400ae00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f400406d6d0 con 0x7f40041f4970 2024-04-16T16:15:19.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.203+0000 7f400ae00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f400406d550 con 0x7f400415de50 2024-04-16T16:15:19.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.203+0000 7f400ae00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f40040d3e40 con 0x7f3fec004800 2024-04-16T16:15:19.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.203+0000 7f4009000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40041f4970 0x7f400415d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:19.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.203+0000 7f4009000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40041f4970 0x7f400415d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56962/0 (socket says 172.21.15.45:56962) 2024-04-16T16:15:19.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.203+0000 7f4009000640 1 -- 172.21.15.45:0/1909397683 learned_addr learned my addr 172.21.15.45:0/1909397683 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:19.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.204+0000 7f4009a00640 1 --2- 172.21.15.45:0/1909397683 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fec004800 0x7f4004156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:19.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.204+0000 7f400a400640 1 --2- 172.21.15.45:0/1909397683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f400415de50 0x7f40040cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:19.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.204+0000 7f4009000640 1 -- 172.21.15.45:0/1909397683 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fec004800 msgr2=0x7f4004156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:19.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.204+0000 7f4009000640 1 --2- 172.21.15.45:0/1909397683 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fec004800 0x7f4004156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.204+0000 7f4009000640 1 -- 172.21.15.45:0/1909397683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f400415de50 msgr2=0x7f40040cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:19.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.204+0000 7f4009000640 1 --2- 172.21.15.45:0/1909397683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f400415de50 0x7f40040cd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.204+0000 7f4009000640 1 -- 172.21.15.45:0/1909397683 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400415e480 con 0x7f40041f4970 2024-04-16T16:15:19.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.204+0000 7f4009000640 1 --2- 172.21.15.45:0/1909397683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40041f4970 0x7f400415d910 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0000c00 tx=0x7f3ff0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:19.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.205+0000 7f4001600640 1 -- 172.21.15.45:0/1909397683 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0042020 con 0x7f40041f4970 2024-04-16T16:15:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.205+0000 7f4001600640 1 -- 172.21.15.45:0/1909397683 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff003a040 con 0x7f40041f4970 2024-04-16T16:15:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.205+0000 7f4001600640 1 -- 172.21.15.45:0/1909397683 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0039070 con 0x7f40041f4970 2024-04-16T16:15:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.205+0000 7f400ae00640 1 -- 172.21.15.45:0/1909397683 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f400415b480 con 0x7f40041f4970 2024-04-16T16:15:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.205+0000 7f400ae00640 1 -- 172.21.15.45:0/1909397683 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f400415ba40 con 0x7f40041f4970 2024-04-16T16:15:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.206+0000 7f400ae00640 1 -- 172.21.15.45:0/1909397683 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4004075780 con 0x7f40041f4970 2024-04-16T16:15:19.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.207+0000 7f4001600640 1 -- 172.21.15.45:0/1909397683 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f3ff0045020 con 0x7f40041f4970 2024-04-16T16:15:19.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.208+0000 7f4001600640 1 --2- 172.21.15.45:0/1909397683 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3fe406ca90 0x7f3fe406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:19.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.208+0000 7f4001600640 1 -- 172.21.15.45:0/1909397683 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6041+0+0 (secure 0 0 0) 0x7f3ff00ad340 con 0x7f40041f4970 2024-04-16T16:15:19.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.208+0000 7f4009a00640 1 --2- 172.21.15.45:0/1909397683 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3fe406ca90 0x7f3fe406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:19.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.210+0000 7f4009a00640 1 --2- 172.21.15.45:0/1909397683 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3fe406ca90 0x7f3fe406ef50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8001eb0 tx=0x7f3ff8009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:19.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.212+0000 7f4001600640 1 -- 172.21.15.45:0/1909397683 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3ff004c070 con 0x7f40041f4970 2024-04-16T16:15:19.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.420+0000 7f400ae00640 1 -- 172.21.15.45:0/1909397683 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f400406dae0 con 0x7f40041f4970 2024-04-16T16:15:19.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.421+0000 7f4001600640 1 -- 172.21.15.45:0/1909397683 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17114 (secure 0 0 0) 0x7f3ff007b500 con 0x7f40041f4970 2024-04-16T16:15:19.421 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:19.421 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":42,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:15:16.405477+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"fs1_metadata","create_time":"2024-04-16T16:15:13.250782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"fs1_data","create_time":"2024-04-16T16:15:13.856986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:15:19.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.425+0000 7f400ae00640 1 -- 172.21.15.45:0/1909397683 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3fe406ca90 msgr2=0x7f3fe406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:19.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.425+0000 7f400ae00640 1 --2- 172.21.15.45:0/1909397683 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3fe406ca90 0x7f3fe406ef50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8001eb0 tx=0x7f3ff8009040 comp rx=0 tx=0).stop 2024-04-16T16:15:19.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.425+0000 7f400ae00640 1 -- 172.21.15.45:0/1909397683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40041f4970 msgr2=0x7f400415d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:19.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.425+0000 7f400ae00640 1 --2- 172.21.15.45:0/1909397683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40041f4970 0x7f400415d910 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0000c00 tx=0x7f3ff0004730 comp rx=0 tx=0).stop 2024-04-16T16:15:19.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.425+0000 7f400ae00640 1 -- 172.21.15.45:0/1909397683 shutdown_connections 2024-04-16T16:15:19.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.425+0000 7f400ae00640 1 --2- 172.21.15.45:0/1909397683 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3fe406ca90 0x7f3fe406ef50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.425+0000 7f400ae00640 1 --2- 172.21.15.45:0/1909397683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f400415de50 0x7f40040cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.425+0000 7f400ae00640 1 --2- 172.21.15.45:0/1909397683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40041f4970 0x7f400415d910 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.425+0000 7f400ae00640 1 --2- 172.21.15.45:0/1909397683 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fec004800 0x7f4004156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.425+0000 7f400ae00640 1 -- 172.21.15.45:0/1909397683 >> 172.21.15.45:0/1909397683 conn(0x7f40040d1940 msgr2=0x7f40040d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:19.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.425+0000 7f400ae00640 1 -- 172.21.15.45:0/1909397683 shutdown_connections 2024-04-16T16:15:19.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.425+0000 7f400ae00640 1 -- 172.21.15.45:0/1909397683 wait complete. 2024-04-16T16:15:19.434 INFO:tasks.ceph.mgr.x.smithi103.stderr:Exception in thread Thread-6: 2024-04-16T16:15:19.435 INFO:tasks.ceph.mgr.x.smithi103.stderr:Traceback (most recent call last): 2024-04-16T16:15:19.435 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner 2024-04-16T16:15:19.435 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.run() 2024-04-16T16:15:19.435 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 1306, in run 2024-04-16T16:15:19.436 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.function(*self.args, **self.kwargs) 2024-04-16T16:15:19.436 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/share/ceph/mgr/stats/fs/perf_stats.py", line 222, in re_register_queries 2024-04-16T16:15:19.436 INFO:tasks.ceph.mgr.x.smithi103.stderr: if self.mx_last_updated >= ua_last_updated: 2024-04-16T16:15:19.436 INFO:tasks.ceph.mgr.x.smithi103.stderr:AttributeError: 'FSPerfStats' object has no attribute 'mx_last_updated' 2024-04-16T16:15:19.445 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-16T16:15:19.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:15:19.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.660+0000 7f1f5b200640 1 Processor -- start 2024-04-16T16:15:19.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.660+0000 7f1f5b200640 1 -- start start 2024-04-16T16:15:19.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.661+0000 7f1f5b200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1f5c146220 0x7f1f5c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:19.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.662+0000 7f1f5b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f5c06f7a0 0x7f1f5c06df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:19.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.662+0000 7f1f5b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1f5c06e480 0x7f1f5c1608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:19.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.662+0000 7f1f5b200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1f5c160410 con 0x7f1f5c146220 2024-04-16T16:15:19.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.662+0000 7f1f5b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1f5c15d360 con 0x7f1f5c06e480 2024-04-16T16:15:19.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.662+0000 7f1f5b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1f5c15d640 con 0x7f1f5c06f7a0 2024-04-16T16:15:19.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.662+0000 7f1f59e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1f5c146220 0x7f1f5c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.662+0000 7f1f59e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1f5c146220 0x7f1f5c15ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56974/0 (socket says 172.21.15.45:56974) 2024-04-16T16:15:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.662+0000 7f1f59e00640 1 -- 172.21.15.45:0/2451196855 learned_addr learned my addr 172.21.15.45:0/2451196855 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.662+0000 7f1f59e00640 1 -- 172.21.15.45:0/2451196855 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f5c06f7a0 msgr2=0x7f1f5c06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.662+0000 7f1f5a800640 1 --2- 172.21.15.45:0/2451196855 >> v2:172.21.15.103:3300/0 conn(0x7f1f5c06e480 0x7f1f5c1608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.662+0000 7f1f59400640 1 --2- 172.21.15.45:0/2451196855 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f5c06f7a0 0x7f1f5c06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.663+0000 7f1f59e00640 1 --2- 172.21.15.45:0/2451196855 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f5c06f7a0 0x7f1f5c06df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.663+0000 7f1f59e00640 1 -- 172.21.15.45:0/2451196855 >> v2:172.21.15.103:3300/0 conn(0x7f1f5c06e480 msgr2=0x7f1f5c1608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.663+0000 7f1f59e00640 1 --2- 172.21.15.45:0/2451196855 >> v2:172.21.15.103:3300/0 conn(0x7f1f5c06e480 0x7f1f5c1608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.663+0000 7f1f59e00640 1 -- 172.21.15.45:0/2451196855 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5c15d920 con 0x7f1f5c146220 2024-04-16T16:15:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.663+0000 7f1f5a800640 1 --2- 172.21.15.45:0/2451196855 >> v2:172.21.15.103:3300/0 conn(0x7f1f5c06e480 0x7f1f5c1608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.664+0000 7f1f59e00640 1 --2- 172.21.15.45:0/2451196855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f5c146220 0x7f1f5c15ccb0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c0d05f0 tx=0x7f1f4c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1c79a4667c103e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.664+0000 7f1f58a00640 1 -- 172.21.15.45:0/2451196855 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c02ca70 con 0x7f1f5c146220 2024-04-16T16:15:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.664+0000 7f1f58a00640 1 -- 172.21.15.45:0/2451196855 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f4c02cc10 con 0x7f1f5c146220 2024-04-16T16:15:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.664+0000 7f1f58a00640 1 -- 172.21.15.45:0/2451196855 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c0343f0 con 0x7f1f5c146220 2024-04-16T16:15:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.665+0000 7f1f5b200640 1 -- 172.21.15.45:0/2451196855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f5c146220 msgr2=0x7f1f5c15ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.665+0000 7f1f5b200640 1 --2- 172.21.15.45:0/2451196855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f5c146220 0x7f1f5c15ccb0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c0d05f0 tx=0x7f1f4c02bed0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.665+0000 7f1f5b200640 1 -- 172.21.15.45:0/2451196855 shutdown_connections 2024-04-16T16:15:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.665+0000 7f1f5b200640 1 --2- 172.21.15.45:0/2451196855 >> v2:172.21.15.103:3300/0 conn(0x7f1f5c06e480 0x7f1f5c1608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.665+0000 7f1f5b200640 1 --2- 172.21.15.45:0/2451196855 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f5c06f7a0 0x7f1f5c06df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.665+0000 7f1f5b200640 1 --2- 172.21.15.45:0/2451196855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f5c146220 0x7f1f5c15ccb0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.665+0000 7f1f5b200640 1 -- 172.21.15.45:0/2451196855 >> 172.21.15.45:0/2451196855 conn(0x7f1f5c06b930 msgr2=0x7f1f5c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.665+0000 7f1f5b200640 1 -- 172.21.15.45:0/2451196855 shutdown_connections 2024-04-16T16:15:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.665+0000 7f1f5b200640 1 -- 172.21.15.45:0/2451196855 wait complete. 2024-04-16T16:15:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.667+0000 7f1f5b200640 1 Processor -- start 2024-04-16T16:15:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.668+0000 7f1f5b200640 1 -- start start 2024-04-16T16:15:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.669+0000 7f1f5b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f5c06e480 0x7f1f5c0ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.669+0000 7f1f5b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f5c06f7a0 0x7f1f5c142320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.669+0000 7f1f5b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f5c146220 0x7f1f5c142860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.669+0000 7f1f5b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c06fda0 con 0x7f1f5c06e480 2024-04-16T16:15:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.669+0000 7f1f5b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c06fc20 con 0x7f1f5c06f7a0 2024-04-16T16:15:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.669+0000 7f1f5b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1f5c160de0 con 0x7f1f5c146220 2024-04-16T16:15:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.669+0000 7f1f59e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f5c06e480 0x7f1f5c0ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.669+0000 7f1f59e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f5c06e480 0x7f1f5c0ca670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56978/0 (socket says 172.21.15.45:56978) 2024-04-16T16:15:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.669+0000 7f1f59e00640 1 -- 172.21.15.45:0/1829860536 learned_addr learned my addr 172.21.15.45:0/1829860536 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.670+0000 7f1f59400640 1 --2- 172.21.15.45:0/1829860536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f5c06f7a0 0x7f1f5c142320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.670+0000 7f1f5a800640 1 --2- 172.21.15.45:0/1829860536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f5c146220 0x7f1f5c142860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.670+0000 7f1f59e00640 1 -- 172.21.15.45:0/1829860536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f5c146220 msgr2=0x7f1f5c142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.670+0000 7f1f59e00640 1 --2- 172.21.15.45:0/1829860536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f5c146220 0x7f1f5c142860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.670+0000 7f1f59e00640 1 -- 172.21.15.45:0/1829860536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f5c06f7a0 msgr2=0x7f1f5c142320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.670+0000 7f1f59e00640 1 --2- 172.21.15.45:0/1829860536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f5c06f7a0 0x7f1f5c142320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.670+0000 7f1f59e00640 1 -- 172.21.15.45:0/1829860536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5c142e70 con 0x7f1f5c06e480 2024-04-16T16:15:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.670+0000 7f1f59e00640 1 --2- 172.21.15.45:0/1829860536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f5c06e480 0x7f1f5c0ca670 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c000f80 tx=0x7f1f4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.670+0000 7f1f52000640 1 -- 172.21.15.45:0/1829860536 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c043020 con 0x7f1f5c06e480 2024-04-16T16:15:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.670+0000 7f1f52000640 1 -- 172.21.15.45:0/1829860536 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f4c03e450 con 0x7f1f5c06e480 2024-04-16T16:15:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.670+0000 7f1f52000640 1 -- 172.21.15.45:0/1829860536 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c03e840 con 0x7f1f5c06e480 2024-04-16T16:15:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.671+0000 7f1f5b200640 1 -- 172.21.15.45:0/1829860536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f5c143150 con 0x7f1f5c06e480 2024-04-16T16:15:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.671+0000 7f1f5b200640 1 -- 172.21.15.45:0/1829860536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f5c143710 con 0x7f1f5c06e480 2024-04-16T16:15:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.672+0000 7f1f5b200640 1 -- 172.21.15.45:0/1829860536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f5c075780 con 0x7f1f5c06e480 2024-04-16T16:15:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.673+0000 7f1f52000640 1 -- 172.21.15.45:0/1829860536 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f1f4c046020 con 0x7f1f5c06e480 2024-04-16T16:15:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.674+0000 7f1f52000640 1 --2- 172.21.15.45:0/1829860536 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1f3c06ca60 0x7f1f3c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.674+0000 7f1f52000640 1 -- 172.21.15.45:0/1829860536 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6041+0+0 (secure 0 0 0) 0x7f1f4c0ae8a0 con 0x7f1f5c06e480 2024-04-16T16:15:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.674+0000 7f1f59400640 1 --2- 172.21.15.45:0/1829860536 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1f3c06ca60 0x7f1f3c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:19.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.676+0000 7f1f59400640 1 --2- 172.21.15.45:0/1829860536 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1f3c06ca60 0x7f1f3c06ef20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c143f10 tx=0x7f1f48008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:19.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.678+0000 7f1f52000640 1 -- 172.21.15.45:0/1829860536 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f4c04d030 con 0x7f1f5c06e480 2024-04-16T16:15:19.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.878+0000 7f1f5b200640 1 -- 172.21.15.45:0/1829860536 --> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1f5c144dc0 con 0x7f1f3c06ca60 2024-04-16T16:15:19.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.902+0000 7f1f52000640 1 -- 172.21.15.45:0/1829860536 <== mgr.4658 v2:172.21.15.103:6832/280591707 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251956 (secure 0 0 0) 0x7f1f5c144dc0 con 0x7f1f3c06ca60 2024-04-16T16:15:19.905 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:19.907 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:15:19.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.907+0000 7f1f5b200640 1 -- 172.21.15.45:0/1829860536 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1f3c06ca60 msgr2=0x7f1f3c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:19.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.907+0000 7f1f5b200640 1 --2- 172.21.15.45:0/1829860536 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1f3c06ca60 0x7f1f3c06ef20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c143f10 tx=0x7f1f48008040 comp rx=0 tx=0).stop 2024-04-16T16:15:19.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.907+0000 7f1f5b200640 1 -- 172.21.15.45:0/1829860536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f5c06e480 msgr2=0x7f1f5c0ca670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:19.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.907+0000 7f1f5b200640 1 --2- 172.21.15.45:0/1829860536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f5c06e480 0x7f1f5c0ca670 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c000f80 tx=0x7f1f4c004a40 comp rx=0 tx=0).stop 2024-04-16T16:15:19.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.907+0000 7f1f5b200640 1 -- 172.21.15.45:0/1829860536 shutdown_connections 2024-04-16T16:15:19.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.907+0000 7f1f5b200640 1 --2- 172.21.15.45:0/1829860536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f5c146220 0x7f1f5c142860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.907+0000 7f1f5b200640 1 --2- 172.21.15.45:0/1829860536 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1f3c06ca60 0x7f1f3c06ef20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.907+0000 7f1f5b200640 1 --2- 172.21.15.45:0/1829860536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f5c06f7a0 0x7f1f5c142320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.907+0000 7f1f5b200640 1 --2- 172.21.15.45:0/1829860536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f5c06e480 0x7f1f5c0ca670 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:19.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.907+0000 7f1f5b200640 1 -- 172.21.15.45:0/1829860536 >> 172.21.15.45:0/1829860536 conn(0x7f1f5c06b930 msgr2=0x7f1f5c0b3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:19.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.907+0000 7f1f5b200640 1 -- 172.21.15.45:0/1829860536 shutdown_connections 2024-04-16T16:15:19.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:19.908+0000 7f1f5b200640 1 -- 172.21.15.45:0/1829860536 wait complete. 2024-04-16T16:15:19.926 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-04-16T16:15:19.127758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":223,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166,"ondisk_log_size":166,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":149,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237852,"kb_used_data":5040,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736868,"statfs":{"total":773094113280,"available":772850552832,"internally_reserved":0,"allocated":5160960,"data_stored":2793210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.748774"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.416715+0000","last_change":"2024-04-16T16:15:15.445346+0000","last_active":"2024-04-16T16:15:16.416715+0000","last_peered":"2024-04-16T16:15:16.416715+0000","last_clean":"2024-04-16T16:15:16.416715+0000","last_became_active":"2024-04-16T16:15:15.445023+0000","last_became_peered":"2024-04-16T16:15:15.445023+0000","last_unstale":"2024-04-16T16:15:16.416715+0000","last_undegraded":"2024-04-16T16:15:16.416715+0000","last_fullsized":"2024-04-16T16:15:16.416715+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:51.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.418591+0000","last_change":"2024-04-16T16:15:15.451103+0000","last_active":"2024-04-16T16:15:16.418591+0000","last_peered":"2024-04-16T16:15:16.418591+0000","last_clean":"2024-04-16T16:15:16.418591+0000","last_became_active":"2024-04-16T16:15:15.450511+0000","last_became_peered":"2024-04-16T16:15:15.450511+0000","last_unstale":"2024-04-16T16:15:16.418591+0000","last_undegraded":"2024-04-16T16:15:16.418591+0000","last_fullsized":"2024-04-16T16:15:16.418591+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:34.706031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800293+0000","last_change":"2024-04-16T16:15:15.454071+0000","last_active":"2024-04-16T16:15:16.800293+0000","last_peered":"2024-04-16T16:15:16.800293+0000","last_clean":"2024-04-16T16:15:16.800293+0000","last_became_active":"2024-04-16T16:15:15.453781+0000","last_became_peered":"2024-04-16T16:15:15.453781+0000","last_unstale":"2024-04-16T16:15:16.800293+0000","last_undegraded":"2024-04-16T16:15:16.800293+0000","last_fullsized":"2024-04-16T16:15:16.800293+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:03.163739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801186+0000","last_change":"2024-04-16T16:15:15.459192+0000","last_active":"2024-04-16T16:15:16.801186+0000","last_peered":"2024-04-16T16:15:16.801186+0000","last_clean":"2024-04-16T16:15:16.801186+0000","last_became_active":"2024-04-16T16:15:15.452689+0000","last_became_peered":"2024-04-16T16:15:15.452689+0000","last_unstale":"2024-04-16T16:15:16.801186+0000","last_undegraded":"2024-04-16T16:15:16.801186+0000","last_fullsized":"2024-04-16T16:15:16.801186+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:39:57.997770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417373+0000","last_change":"2024-04-16T16:15:15.443526+0000","last_active":"2024-04-16T16:15:16.417373+0000","last_peered":"2024-04-16T16:15:16.417373+0000","last_clean":"2024-04-16T16:15:16.417373+0000","last_became_active":"2024-04-16T16:15:15.443203+0000","last_became_peered":"2024-04-16T16:15:15.443203+0000","last_unstale":"2024-04-16T16:15:16.417373+0000","last_undegraded":"2024-04-16T16:15:16.417373+0000","last_fullsized":"2024-04-16T16:15:16.417373+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:42.434776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803362+0000","last_change":"2024-04-16T16:15:15.429037+0000","last_active":"2024-04-16T16:15:16.803362+0000","last_peered":"2024-04-16T16:15:16.803362+0000","last_clean":"2024-04-16T16:15:16.803362+0000","last_became_active":"2024-04-16T16:15:15.425570+0000","last_became_peered":"2024-04-16T16:15:15.425570+0000","last_unstale":"2024-04-16T16:15:16.803362+0000","last_undegraded":"2024-04-16T16:15:16.803362+0000","last_fullsized":"2024-04-16T16:15:16.803362+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:17:00.492572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.416489+0000","last_change":"2024-04-16T16:15:15.439872+0000","last_active":"2024-04-16T16:15:16.416489+0000","last_peered":"2024-04-16T16:15:16.416489+0000","last_clean":"2024-04-16T16:15:16.416489+0000","last_became_active":"2024-04-16T16:15:15.439553+0000","last_became_peered":"2024-04-16T16:15:15.439553+0000","last_unstale":"2024-04-16T16:15:16.416489+0000","last_undegraded":"2024-04-16T16:15:16.416489+0000","last_fullsized":"2024-04-16T16:15:16.416489+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:12.276411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.802555+0000","last_change":"2024-04-16T16:15:15.463187+0000","last_active":"2024-04-16T16:15:16.802555+0000","last_peered":"2024-04-16T16:15:16.802555+0000","last_clean":"2024-04-16T16:15:16.802555+0000","last_became_active":"2024-04-16T16:15:15.461753+0000","last_became_peered":"2024-04-16T16:15:15.461753+0000","last_unstale":"2024-04-16T16:15:16.802555+0000","last_undegraded":"2024-04-16T16:15:16.802555+0000","last_fullsized":"2024-04-16T16:15:16.802555+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:44.974637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804974+0000","last_change":"2024-04-16T16:15:15.451403+0000","last_active":"2024-04-16T16:15:16.804974+0000","last_peered":"2024-04-16T16:15:16.804974+0000","last_clean":"2024-04-16T16:15:16.804974+0000","last_became_active":"2024-04-16T16:15:15.450199+0000","last_became_peered":"2024-04-16T16:15:15.450199+0000","last_unstale":"2024-04-16T16:15:16.804974+0000","last_undegraded":"2024-04-16T16:15:16.804974+0000","last_fullsized":"2024-04-16T16:15:16.804974+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:32:52.123986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801619+0000","last_change":"2024-04-16T16:15:15.459905+0000","last_active":"2024-04-16T16:15:16.801619+0000","last_peered":"2024-04-16T16:15:16.801619+0000","last_clean":"2024-04-16T16:15:16.801619+0000","last_became_active":"2024-04-16T16:15:15.459387+0000","last_became_peered":"2024-04-16T16:15:15.459387+0000","last_unstale":"2024-04-16T16:15:16.801619+0000","last_undegraded":"2024-04-16T16:15:16.801619+0000","last_fullsized":"2024-04-16T16:15:16.801619+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:51.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801092+0000","last_change":"2024-04-16T16:15:15.463054+0000","last_active":"2024-04-16T16:15:16.801092+0000","last_peered":"2024-04-16T16:15:16.801092+0000","last_clean":"2024-04-16T16:15:16.801092+0000","last_became_active":"2024-04-16T16:15:15.462075+0000","last_became_peered":"2024-04-16T16:15:15.462075+0000","last_unstale":"2024-04-16T16:15:16.801092+0000","last_undegraded":"2024-04-16T16:15:16.801092+0000","last_fullsized":"2024-04-16T16:15:16.801092+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:56.074330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417550+0000","last_change":"2024-04-16T16:15:15.449849+0000","last_active":"2024-04-16T16:15:16.417550+0000","last_peered":"2024-04-16T16:15:16.417550+0000","last_clean":"2024-04-16T16:15:16.417550+0000","last_became_active":"2024-04-16T16:15:15.449585+0000","last_became_peered":"2024-04-16T16:15:15.449585+0000","last_unstale":"2024-04-16T16:15:16.417550+0000","last_undegraded":"2024-04-16T16:15:16.417550+0000","last_fullsized":"2024-04-16T16:15:16.417550+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:12.276411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.410962+0000","last_change":"2024-04-16T16:15:15.441331+0000","last_active":"2024-04-16T16:15:16.410962+0000","last_peered":"2024-04-16T16:15:16.410962+0000","last_clean":"2024-04-16T16:15:16.410962+0000","last_became_active":"2024-04-16T16:15:15.440983+0000","last_became_peered":"2024-04-16T16:15:15.440983+0000","last_unstale":"2024-04-16T16:15:16.410962+0000","last_undegraded":"2024-04-16T16:15:16.410962+0000","last_fullsized":"2024-04-16T16:15:16.410962+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:10:59.131225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801104+0000","last_change":"2024-04-16T16:15:15.459411+0000","last_active":"2024-04-16T16:15:16.801104+0000","last_peered":"2024-04-16T16:15:16.801104+0000","last_clean":"2024-04-16T16:15:16.801104+0000","last_became_active":"2024-04-16T16:15:15.452989+0000","last_became_peered":"2024-04-16T16:15:15.452989+0000","last_unstale":"2024-04-16T16:15:16.801104+0000","last_undegraded":"2024-04-16T16:15:16.801104+0000","last_fullsized":"2024-04-16T16:15:16.801104+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:27.522215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.802979+0000","last_change":"2024-04-16T16:15:15.448356+0000","last_active":"2024-04-16T16:15:16.802979+0000","last_peered":"2024-04-16T16:15:16.802979+0000","last_clean":"2024-04-16T16:15:16.802979+0000","last_became_active":"2024-04-16T16:15:15.448056+0000","last_became_peered":"2024-04-16T16:15:15.448056+0000","last_unstale":"2024-04-16T16:15:16.802979+0000","last_undegraded":"2024-04-16T16:15:16.802979+0000","last_fullsized":"2024-04-16T16:15:16.802979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:30:21.270376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.506646+0000","last_change":"2024-04-16T16:15:15.447460+0000","last_active":"2024-04-16T16:15:16.506646+0000","last_peered":"2024-04-16T16:15:16.506646+0000","last_clean":"2024-04-16T16:15:16.506646+0000","last_became_active":"2024-04-16T16:15:15.447155+0000","last_became_peered":"2024-04-16T16:15:15.447155+0000","last_unstale":"2024-04-16T16:15:16.506646+0000","last_undegraded":"2024-04-16T16:15:16.506646+0000","last_fullsized":"2024-04-16T16:15:16.506646+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:48:05.645437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800986+0000","last_change":"2024-04-16T16:15:15.445583+0000","last_active":"2024-04-16T16:15:16.800986+0000","last_peered":"2024-04-16T16:15:16.800986+0000","last_clean":"2024-04-16T16:15:16.800986+0000","last_became_active":"2024-04-16T16:15:15.445259+0000","last_became_peered":"2024-04-16T16:15:15.445259+0000","last_unstale":"2024-04-16T16:15:16.800986+0000","last_undegraded":"2024-04-16T16:15:16.800986+0000","last_fullsized":"2024-04-16T16:15:16.800986+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:56.074330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800427+0000","last_change":"2024-04-16T16:15:15.451682+0000","last_active":"2024-04-16T16:15:16.800427+0000","last_peered":"2024-04-16T16:15:16.800427+0000","last_clean":"2024-04-16T16:15:16.800427+0000","last_became_active":"2024-04-16T16:15:15.450999+0000","last_became_peered":"2024-04-16T16:15:15.450999+0000","last_unstale":"2024-04-16T16:15:16.800427+0000","last_undegraded":"2024-04-16T16:15:16.800427+0000","last_fullsized":"2024-04-16T16:15:16.800427+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:01:20.716612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.507503+0000","last_change":"2024-04-16T16:15:15.446449+0000","last_active":"2024-04-16T16:15:16.507503+0000","last_peered":"2024-04-16T16:15:16.507503+0000","last_clean":"2024-04-16T16:15:16.507503+0000","last_became_active":"2024-04-16T16:15:15.446135+0000","last_became_peered":"2024-04-16T16:15:15.446135+0000","last_unstale":"2024-04-16T16:15:16.507503+0000","last_undegraded":"2024-04-16T16:15:16.507503+0000","last_fullsized":"2024-04-16T16:15:16.507503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:49.654384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.411023+0000","last_change":"2024-04-16T16:15:15.456287+0000","last_active":"2024-04-16T16:15:16.411023+0000","last_peered":"2024-04-16T16:15:16.411023+0000","last_clean":"2024-04-16T16:15:16.411023+0000","last_became_active":"2024-04-16T16:15:15.455963+0000","last_became_peered":"2024-04-16T16:15:15.455963+0000","last_unstale":"2024-04-16T16:15:16.411023+0000","last_undegraded":"2024-04-16T16:15:16.411023+0000","last_fullsized":"2024-04-16T16:15:16.411023+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:07.752151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803726+0000","last_change":"2024-04-16T16:15:15.447751+0000","last_active":"2024-04-16T16:15:16.803726+0000","last_peered":"2024-04-16T16:15:16.803726+0000","last_clean":"2024-04-16T16:15:16.803726+0000","last_became_active":"2024-04-16T16:15:15.447195+0000","last_became_peered":"2024-04-16T16:15:15.447195+0000","last_unstale":"2024-04-16T16:15:16.803726+0000","last_undegraded":"2024-04-16T16:15:16.803726+0000","last_fullsized":"2024-04-16T16:15:16.803726+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:34:32.934841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801511+0000","last_change":"2024-04-16T16:15:15.462804+0000","last_active":"2024-04-16T16:15:16.801511+0000","last_peered":"2024-04-16T16:15:16.801511+0000","last_clean":"2024-04-16T16:15:16.801511+0000","last_became_active":"2024-04-16T16:15:15.461788+0000","last_became_peered":"2024-04-16T16:15:15.461788+0000","last_unstale":"2024-04-16T16:15:16.801511+0000","last_undegraded":"2024-04-16T16:15:16.801511+0000","last_fullsized":"2024-04-16T16:15:16.801511+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:12.276411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.506488+0000","last_change":"2024-04-16T16:15:15.443351+0000","last_active":"2024-04-16T16:15:16.506488+0000","last_peered":"2024-04-16T16:15:16.506488+0000","last_clean":"2024-04-16T16:15:16.506488+0000","last_became_active":"2024-04-16T16:15:15.442366+0000","last_became_peered":"2024-04-16T16:15:15.442366+0000","last_unstale":"2024-04-16T16:15:16.506488+0000","last_undegraded":"2024-04-16T16:15:16.506488+0000","last_fullsized":"2024-04-16T16:15:16.506488+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:53.599226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.507074+0000","last_change":"2024-04-16T16:15:15.447202+0000","last_active":"2024-04-16T16:15:16.507074+0000","last_peered":"2024-04-16T16:15:16.507074+0000","last_clean":"2024-04-16T16:15:16.507074+0000","last_became_active":"2024-04-16T16:15:15.444539+0000","last_became_peered":"2024-04-16T16:15:15.444539+0000","last_unstale":"2024-04-16T16:15:16.507074+0000","last_undegraded":"2024-04-16T16:15:16.507074+0000","last_fullsized":"2024-04-16T16:15:16.507074+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:33:41.701252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.507294+0000","last_change":"2024-04-16T16:15:15.454894+0000","last_active":"2024-04-16T16:15:16.507294+0000","last_peered":"2024-04-16T16:15:16.507294+0000","last_clean":"2024-04-16T16:15:16.507294+0000","last_became_active":"2024-04-16T16:15:15.454659+0000","last_became_peered":"2024-04-16T16:15:15.454659+0000","last_unstale":"2024-04-16T16:15:16.507294+0000","last_undegraded":"2024-04-16T16:15:16.507294+0000","last_fullsized":"2024-04-16T16:15:16.507294+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:44.518137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.410920+0000","last_change":"2024-04-16T16:15:15.453129+0000","last_active":"2024-04-16T16:15:16.410920+0000","last_peered":"2024-04-16T16:15:16.410920+0000","last_clean":"2024-04-16T16:15:16.410920+0000","last_became_active":"2024-04-16T16:15:15.452843+0000","last_became_peered":"2024-04-16T16:15:15.452843+0000","last_unstale":"2024-04-16T16:15:16.410920+0000","last_undegraded":"2024-04-16T16:15:16.410920+0000","last_fullsized":"2024-04-16T16:15:16.410920+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:28:11.918651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800504+0000","last_change":"2024-04-16T16:15:15.437158+0000","last_active":"2024-04-16T16:15:16.800504+0000","last_peered":"2024-04-16T16:15:16.800504+0000","last_clean":"2024-04-16T16:15:16.800504+0000","last_became_active":"2024-04-16T16:15:15.435103+0000","last_became_peered":"2024-04-16T16:15:15.435103+0000","last_unstale":"2024-04-16T16:15:16.800504+0000","last_undegraded":"2024-04-16T16:15:16.800504+0000","last_fullsized":"2024-04-16T16:15:16.800504+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:34.312744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417574+0000","last_change":"2024-04-16T16:15:15.450932+0000","last_active":"2024-04-16T16:15:16.417574+0000","last_peered":"2024-04-16T16:15:16.417574+0000","last_clean":"2024-04-16T16:15:16.417574+0000","last_became_active":"2024-04-16T16:15:15.447441+0000","last_became_peered":"2024-04-16T16:15:15.447441+0000","last_unstale":"2024-04-16T16:15:16.417574+0000","last_undegraded":"2024-04-16T16:15:16.417574+0000","last_fullsized":"2024-04-16T16:15:16.417574+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:51.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.506434+0000","last_change":"2024-04-16T16:15:15.445285+0000","last_active":"2024-04-16T16:15:16.506434+0000","last_peered":"2024-04-16T16:15:16.506434+0000","last_clean":"2024-04-16T16:15:16.506434+0000","last_became_active":"2024-04-16T16:15:15.444969+0000","last_became_peered":"2024-04-16T16:15:15.444969+0000","last_unstale":"2024-04-16T16:15:16.506434+0000","last_undegraded":"2024-04-16T16:15:16.506434+0000","last_fullsized":"2024-04-16T16:15:16.506434+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:38:30.263403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800018+0000","last_change":"2024-04-16T16:15:15.445371+0000","last_active":"2024-04-16T16:15:16.800018+0000","last_peered":"2024-04-16T16:15:16.800018+0000","last_clean":"2024-04-16T16:15:16.800018+0000","last_became_active":"2024-04-16T16:15:15.445063+0000","last_became_peered":"2024-04-16T16:15:15.445063+0000","last_unstale":"2024-04-16T16:15:16.800018+0000","last_undegraded":"2024-04-16T16:15:16.800018+0000","last_fullsized":"2024-04-16T16:15:16.800018+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:57.624924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803133+0000","last_change":"2024-04-16T16:15:15.434979+0000","last_active":"2024-04-16T16:15:16.803133+0000","last_peered":"2024-04-16T16:15:16.803133+0000","last_clean":"2024-04-16T16:15:16.803133+0000","last_became_active":"2024-04-16T16:15:15.434028+0000","last_became_peered":"2024-04-16T16:15:15.434028+0000","last_unstale":"2024-04-16T16:15:16.803133+0000","last_undegraded":"2024-04-16T16:15:16.803133+0000","last_fullsized":"2024-04-16T16:15:16.803133+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:57:00.347584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.410627+0000","last_change":"2024-04-16T16:15:15.448903+0000","last_active":"2024-04-16T16:15:16.410627+0000","last_peered":"2024-04-16T16:15:16.410627+0000","last_clean":"2024-04-16T16:15:16.410627+0000","last_became_active":"2024-04-16T16:15:15.448548+0000","last_became_peered":"2024-04-16T16:15:15.448548+0000","last_unstale":"2024-04-16T16:15:16.410627+0000","last_undegraded":"2024-04-16T16:15:16.410627+0000","last_fullsized":"2024-04-16T16:15:16.410627+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:49.734767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800296+0000","last_change":"2024-04-16T16:15:15.444805+0000","last_active":"2024-04-16T16:15:16.800296+0000","last_peered":"2024-04-16T16:15:16.800296+0000","last_clean":"2024-04-16T16:15:16.800296+0000","last_became_active":"2024-04-16T16:15:15.442030+0000","last_became_peered":"2024-04-16T16:15:15.442030+0000","last_unstale":"2024-04-16T16:15:16.800296+0000","last_undegraded":"2024-04-16T16:15:16.800296+0000","last_fullsized":"2024-04-16T16:15:16.800296+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:33:41.701252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"41'4","reported_seq":18,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804128+0000","last_change":"2024-04-16T16:15:14.426636+0000","last_active":"2024-04-16T16:15:16.804128+0000","last_peered":"2024-04-16T16:15:16.804128+0000","last_clean":"2024-04-16T16:15:16.804128+0000","last_became_active":"2024-04-16T16:15:14.426319+0000","last_became_peered":"2024-04-16T16:15:14.426319+0000","last_unstale":"2024-04-16T16:15:16.804128+0000","last_undegraded":"2024-04-16T16:15:16.804128+0000","last_fullsized":"2024-04-16T16:15:16.804128+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:55.060644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.507344+0000","last_change":"2024-04-16T16:15:15.449524+0000","last_active":"2024-04-16T16:15:16.507344+0000","last_peered":"2024-04-16T16:15:16.507344+0000","last_clean":"2024-04-16T16:15:16.507344+0000","last_became_active":"2024-04-16T16:15:15.448988+0000","last_became_peered":"2024-04-16T16:15:15.448988+0000","last_unstale":"2024-04-16T16:15:16.507344+0000","last_undegraded":"2024-04-16T16:15:16.507344+0000","last_fullsized":"2024-04-16T16:15:16.507344+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:57.624924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"41'1","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800795+0000","last_change":"2024-04-16T16:15:14.430354+0000","last_active":"2024-04-16T16:15:16.800795+0000","last_peered":"2024-04-16T16:15:16.800795+0000","last_clean":"2024-04-16T16:15:16.800795+0000","last_became_active":"2024-04-16T16:15:14.429965+0000","last_became_peered":"2024-04-16T16:15:14.429965+0000","last_unstale":"2024-04-16T16:15:16.800795+0000","last_undegraded":"2024-04-16T16:15:16.800795+0000","last_fullsized":"2024-04-16T16:15:16.800795+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:28:19.722523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417921+0000","last_change":"2024-04-16T16:15:15.432880+0000","last_active":"2024-04-16T16:15:16.417921+0000","last_peered":"2024-04-16T16:15:16.417921+0000","last_clean":"2024-04-16T16:15:16.417921+0000","last_became_active":"2024-04-16T16:15:15.432458+0000","last_became_peered":"2024-04-16T16:15:15.432458+0000","last_unstale":"2024-04-16T16:15:16.417921+0000","last_undegraded":"2024-04-16T16:15:16.417921+0000","last_fullsized":"2024-04-16T16:15:16.417921+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:47.467138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"41'2","reported_seq":16,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.799696+0000","last_change":"2024-04-16T16:15:14.420969+0000","last_active":"2024-04-16T16:15:16.799696+0000","last_peered":"2024-04-16T16:15:16.799696+0000","last_clean":"2024-04-16T16:15:16.799696+0000","last_became_active":"2024-04-16T16:15:14.420657+0000","last_became_peered":"2024-04-16T16:15:14.420657+0000","last_unstale":"2024-04-16T16:15:16.799696+0000","last_undegraded":"2024-04-16T16:15:16.799696+0000","last_fullsized":"2024-04-16T16:15:16.799696+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:43.299902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.799557+0000","last_change":"2024-04-16T16:15:15.455406+0000","last_active":"2024-04-16T16:15:16.799557+0000","last_peered":"2024-04-16T16:15:16.799557+0000","last_clean":"2024-04-16T16:15:16.799557+0000","last_became_active":"2024-04-16T16:15:15.455065+0000","last_became_peered":"2024-04-16T16:15:15.455065+0000","last_unstale":"2024-04-16T16:15:16.799557+0000","last_undegraded":"2024-04-16T16:15:16.799557+0000","last_fullsized":"2024-04-16T16:15:16.799557+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:48:05.645437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"41'2","reported_seq":16,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.415506+0000","last_change":"2024-04-16T16:15:14.426165+0000","last_active":"2024-04-16T16:15:16.415506+0000","last_peered":"2024-04-16T16:15:16.415506+0000","last_clean":"2024-04-16T16:15:16.415506+0000","last_became_active":"2024-04-16T16:15:14.424933+0000","last_became_peered":"2024-04-16T16:15:14.424933+0000","last_unstale":"2024-04-16T16:15:16.415506+0000","last_undegraded":"2024-04-16T16:15:16.415506+0000","last_fullsized":"2024-04-16T16:15:16.415506+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:59.375922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800022+0000","last_change":"2024-04-16T16:15:15.451450+0000","last_active":"2024-04-16T16:15:16.800022+0000","last_peered":"2024-04-16T16:15:16.800022+0000","last_clean":"2024-04-16T16:15:16.800022+0000","last_became_active":"2024-04-16T16:15:15.451103+0000","last_became_peered":"2024-04-16T16:15:15.451103+0000","last_unstale":"2024-04-16T16:15:16.800022+0000","last_undegraded":"2024-04-16T16:15:16.800022+0000","last_fullsized":"2024-04-16T16:15:16.800022+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:49.654384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"41'1","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803785+0000","last_change":"2024-04-16T16:15:14.419476+0000","last_active":"2024-04-16T16:15:16.803785+0000","last_peered":"2024-04-16T16:15:16.803785+0000","last_clean":"2024-04-16T16:15:16.803785+0000","last_became_active":"2024-04-16T16:15:14.419185+0000","last_became_peered":"2024-04-16T16:15:14.419185+0000","last_unstale":"2024-04-16T16:15:16.803785+0000","last_undegraded":"2024-04-16T16:15:16.803785+0000","last_fullsized":"2024-04-16T16:15:16.803785+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:26.508529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801967+0000","last_change":"2024-04-16T16:15:15.460542+0000","last_active":"2024-04-16T16:15:16.801967+0000","last_peered":"2024-04-16T16:15:16.801967+0000","last_clean":"2024-04-16T16:15:16.801967+0000","last_became_active":"2024-04-16T16:15:15.459809+0000","last_became_peered":"2024-04-16T16:15:15.459809+0000","last_unstale":"2024-04-16T16:15:16.801967+0000","last_undegraded":"2024-04-16T16:15:16.801967+0000","last_fullsized":"2024-04-16T16:15:16.801967+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:47.467138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"41'1","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800347+0000","last_change":"2024-04-16T16:15:14.429151+0000","last_active":"2024-04-16T16:15:16.800347+0000","last_peered":"2024-04-16T16:15:16.800347+0000","last_clean":"2024-04-16T16:15:16.800347+0000","last_became_active":"2024-04-16T16:15:14.428841+0000","last_became_peered":"2024-04-16T16:15:14.428841+0000","last_unstale":"2024-04-16T16:15:16.800347+0000","last_undegraded":"2024-04-16T16:15:16.800347+0000","last_fullsized":"2024-04-16T16:15:16.800347+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:16.172813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800986+0000","last_change":"2024-04-16T16:15:15.461073+0000","last_active":"2024-04-16T16:15:16.800986+0000","last_peered":"2024-04-16T16:15:16.800986+0000","last_clean":"2024-04-16T16:15:16.800986+0000","last_became_active":"2024-04-16T16:15:15.459255+0000","last_became_peered":"2024-04-16T16:15:15.459255+0000","last_unstale":"2024-04-16T16:15:16.800986+0000","last_undegraded":"2024-04-16T16:15:16.800986+0000","last_fullsized":"2024-04-16T16:15:16.800986+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:27:00.456168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417002+0000","last_change":"2024-04-16T16:15:14.418012+0000","last_active":"2024-04-16T16:15:16.417002+0000","last_peered":"2024-04-16T16:15:16.417002+0000","last_clean":"2024-04-16T16:15:16.417002+0000","last_became_active":"2024-04-16T16:15:14.417689+0000","last_became_peered":"2024-04-16T16:15:14.417689+0000","last_unstale":"2024-04-16T16:15:16.417002+0000","last_undegraded":"2024-04-16T16:15:16.417002+0000","last_fullsized":"2024-04-16T16:15:16.417002+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:33:40.687566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.506201+0000","last_change":"2024-04-16T16:15:15.438978+0000","last_active":"2024-04-16T16:15:16.506201+0000","last_peered":"2024-04-16T16:15:16.506201+0000","last_clean":"2024-04-16T16:15:16.506201+0000","last_became_active":"2024-04-16T16:15:15.438597+0000","last_became_peered":"2024-04-16T16:15:15.438597+0000","last_unstale":"2024-04-16T16:15:16.506201+0000","last_undegraded":"2024-04-16T16:15:16.506201+0000","last_fullsized":"2024-04-16T16:15:16.506201+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:19:06.087718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801256+0000","last_change":"2024-04-16T16:15:14.411837+0000","last_active":"2024-04-16T16:15:16.801256+0000","last_peered":"2024-04-16T16:15:16.801256+0000","last_clean":"2024-04-16T16:15:16.801256+0000","last_became_active":"2024-04-16T16:15:14.411512+0000","last_became_peered":"2024-04-16T16:15:14.411512+0000","last_unstale":"2024-04-16T16:15:16.801256+0000","last_undegraded":"2024-04-16T16:15:16.801256+0000","last_fullsized":"2024-04-16T16:15:16.801256+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:28:19.722523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800517+0000","last_change":"2024-04-16T16:15:15.451420+0000","last_active":"2024-04-16T16:15:16.800517+0000","last_peered":"2024-04-16T16:15:16.800517+0000","last_clean":"2024-04-16T16:15:16.800517+0000","last_became_active":"2024-04-16T16:15:15.450543+0000","last_became_peered":"2024-04-16T16:15:15.450543+0000","last_unstale":"2024-04-16T16:15:16.800517+0000","last_undegraded":"2024-04-16T16:15:16.800517+0000","last_fullsized":"2024-04-16T16:15:16.800517+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:27:00.456168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801092+0000","last_change":"2024-04-16T16:15:14.422188+0000","last_active":"2024-04-16T16:15:16.801092+0000","last_peered":"2024-04-16T16:15:16.801092+0000","last_clean":"2024-04-16T16:15:16.801092+0000","last_became_active":"2024-04-16T16:15:14.421842+0000","last_became_peered":"2024-04-16T16:15:14.421842+0000","last_unstale":"2024-04-16T16:15:16.801092+0000","last_undegraded":"2024-04-16T16:15:16.801092+0000","last_fullsized":"2024-04-16T16:15:16.801092+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:43.299902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.418792+0000","last_change":"2024-04-16T16:15:15.436333+0000","last_active":"2024-04-16T16:15:16.418792+0000","last_peered":"2024-04-16T16:15:16.418792+0000","last_clean":"2024-04-16T16:15:16.418792+0000","last_became_active":"2024-04-16T16:15:15.435961+0000","last_became_peered":"2024-04-16T16:15:15.435961+0000","last_unstale":"2024-04-16T16:15:16.418792+0000","last_undegraded":"2024-04-16T16:15:16.418792+0000","last_fullsized":"2024-04-16T16:15:16.418792+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:44.974637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.802286+0000","last_change":"2024-04-16T16:15:14.427378+0000","last_active":"2024-04-16T16:15:16.802286+0000","last_peered":"2024-04-16T16:15:16.802286+0000","last_clean":"2024-04-16T16:15:16.802286+0000","last_became_active":"2024-04-16T16:15:14.427036+0000","last_became_peered":"2024-04-16T16:15:14.427036+0000","last_unstale":"2024-04-16T16:15:16.802286+0000","last_undegraded":"2024-04-16T16:15:16.802286+0000","last_fullsized":"2024-04-16T16:15:16.802286+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:16.172813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804455+0000","last_change":"2024-04-16T16:15:15.453206+0000","last_active":"2024-04-16T16:15:16.804455+0000","last_peered":"2024-04-16T16:15:16.804455+0000","last_clean":"2024-04-16T16:15:16.804455+0000","last_became_active":"2024-04-16T16:15:15.453002+0000","last_became_peered":"2024-04-16T16:15:15.453002+0000","last_unstale":"2024-04-16T16:15:16.804455+0000","last_undegraded":"2024-04-16T16:15:16.804455+0000","last_fullsized":"2024-04-16T16:15:16.804455+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:37.628550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"41'3","reported_seq":17,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417985+0000","last_change":"2024-04-16T16:15:14.427833+0000","last_active":"2024-04-16T16:15:16.417985+0000","last_peered":"2024-04-16T16:15:16.417985+0000","last_clean":"2024-04-16T16:15:16.417985+0000","last_became_active":"2024-04-16T16:15:14.427492+0000","last_became_peered":"2024-04-16T16:15:14.427492+0000","last_unstale":"2024-04-16T16:15:16.417985+0000","last_undegraded":"2024-04-16T16:15:16.417985+0000","last_fullsized":"2024-04-16T16:15:16.417985+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:48.640698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417688+0000","last_change":"2024-04-16T16:15:15.435454+0000","last_active":"2024-04-16T16:15:16.417688+0000","last_peered":"2024-04-16T16:15:16.417688+0000","last_clean":"2024-04-16T16:15:16.417688+0000","last_became_active":"2024-04-16T16:15:15.435056+0000","last_became_peered":"2024-04-16T16:15:15.435056+0000","last_unstale":"2024-04-16T16:15:16.417688+0000","last_undegraded":"2024-04-16T16:15:16.417688+0000","last_fullsized":"2024-04-16T16:15:16.417688+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:34.706031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"41'6","reported_seq":20,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.410761+0000","last_change":"2024-04-16T16:15:14.419221+0000","last_active":"2024-04-16T16:15:16.410761+0000","last_peered":"2024-04-16T16:15:16.410761+0000","last_clean":"2024-04-16T16:15:16.410761+0000","last_became_active":"2024-04-16T16:15:14.418573+0000","last_became_peered":"2024-04-16T16:15:14.418573+0000","last_unstale":"2024-04-16T16:15:16.410761+0000","last_undegraded":"2024-04-16T16:15:16.410761+0000","last_fullsized":"2024-04-16T16:15:16.410761+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:08.107514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.506564+0000","last_change":"2024-04-16T16:15:15.447435+0000","last_active":"2024-04-16T16:15:16.506564+0000","last_peered":"2024-04-16T16:15:16.506564+0000","last_clean":"2024-04-16T16:15:16.506564+0000","last_became_active":"2024-04-16T16:15:15.447094+0000","last_became_peered":"2024-04-16T16:15:15.447094+0000","last_unstale":"2024-04-16T16:15:16.506564+0000","last_undegraded":"2024-04-16T16:15:16.506564+0000","last_fullsized":"2024-04-16T16:15:16.506564+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:16:44.917291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"41'2","reported_seq":16,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804649+0000","last_change":"2024-04-16T16:15:14.408195+0000","last_active":"2024-04-16T16:15:16.804649+0000","last_peered":"2024-04-16T16:15:16.804649+0000","last_clean":"2024-04-16T16:15:16.804649+0000","last_became_active":"2024-04-16T16:15:14.407584+0000","last_became_peered":"2024-04-16T16:15:14.407584+0000","last_unstale":"2024-04-16T16:15:16.804649+0000","last_undegraded":"2024-04-16T16:15:16.804649+0000","last_fullsized":"2024-04-16T16:15:16.804649+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:39:56.984084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.802042+0000","last_change":"2024-04-16T16:15:15.462282+0000","last_active":"2024-04-16T16:15:16.802042+0000","last_peered":"2024-04-16T16:15:16.802042+0000","last_clean":"2024-04-16T16:15:16.802042+0000","last_became_active":"2024-04-16T16:15:15.461980+0000","last_became_peered":"2024-04-16T16:15:15.461980+0000","last_unstale":"2024-04-16T16:15:16.802042+0000","last_undegraded":"2024-04-16T16:15:16.802042+0000","last_fullsized":"2024-04-16T16:15:16.802042+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:11.662435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.416025+0000","last_change":"2024-04-16T16:15:14.421692+0000","last_active":"2024-04-16T16:15:16.416025+0000","last_peered":"2024-04-16T16:15:16.416025+0000","last_clean":"2024-04-16T16:15:16.416025+0000","last_became_active":"2024-04-16T16:15:14.421361+0000","last_became_peered":"2024-04-16T16:15:14.421361+0000","last_unstale":"2024-04-16T16:15:16.416025+0000","last_undegraded":"2024-04-16T16:15:16.416025+0000","last_fullsized":"2024-04-16T16:15:16.416025+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:56.611238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803068+0000","last_change":"2024-04-16T16:15:15.455736+0000","last_active":"2024-04-16T16:15:16.803068+0000","last_peered":"2024-04-16T16:15:16.803068+0000","last_clean":"2024-04-16T16:15:16.803068+0000","last_became_active":"2024-04-16T16:15:15.455415+0000","last_became_peered":"2024-04-16T16:15:15.455415+0000","last_unstale":"2024-04-16T16:15:16.803068+0000","last_undegraded":"2024-04-16T16:15:16.803068+0000","last_fullsized":"2024-04-16T16:15:16.803068+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:29:30.311490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803257+0000","last_change":"2024-04-16T16:15:14.410824+0000","last_active":"2024-04-16T16:15:16.803257+0000","last_peered":"2024-04-16T16:15:16.803257+0000","last_clean":"2024-04-16T16:15:16.803257+0000","last_became_active":"2024-04-16T16:15:14.409874+0000","last_became_peered":"2024-04-16T16:15:14.409874+0000","last_unstale":"2024-04-16T16:15:16.803257+0000","last_undegraded":"2024-04-16T16:15:16.803257+0000","last_fullsized":"2024-04-16T16:15:16.803257+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:26:59.442481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.416927+0000","last_change":"2024-04-16T16:15:15.446100+0000","last_active":"2024-04-16T16:15:16.416927+0000","last_peered":"2024-04-16T16:15:16.416927+0000","last_clean":"2024-04-16T16:15:16.416927+0000","last_became_active":"2024-04-16T16:15:15.444480+0000","last_became_peered":"2024-04-16T16:15:15.444480+0000","last_unstale":"2024-04-16T16:15:16.416927+0000","last_undegraded":"2024-04-16T16:15:16.416927+0000","last_fullsized":"2024-04-16T16:15:16.416927+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:47.467138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"41'8","reported_seq":22,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801528+0000","last_change":"2024-04-16T16:15:14.421997+0000","last_active":"2024-04-16T16:15:16.801528+0000","last_peered":"2024-04-16T16:15:16.801528+0000","last_clean":"2024-04-16T16:15:16.801528+0000","last_became_active":"2024-04-16T16:15:14.421747+0000","last_became_peered":"2024-04-16T16:15:14.421747+0000","last_unstale":"2024-04-16T16:15:16.801528+0000","last_undegraded":"2024-04-16T16:15:16.801528+0000","last_fullsized":"2024-04-16T16:15:16.801528+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:56:30.781319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.410930+0000","last_change":"2024-04-16T16:15:15.444883+0000","last_active":"2024-04-16T16:15:16.410930+0000","last_peered":"2024-04-16T16:15:16.410930+0000","last_clean":"2024-04-16T16:15:16.410930+0000","last_became_active":"2024-04-16T16:15:15.438454+0000","last_became_peered":"2024-04-16T16:15:15.438454+0000","last_unstale":"2024-04-16T16:15:16.410930+0000","last_undegraded":"2024-04-16T16:15:16.410930+0000","last_fullsized":"2024-04-16T16:15:16.410930+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:25:00.737475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"41'1","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.410689+0000","last_change":"2024-04-16T16:15:14.406535+0000","last_active":"2024-04-16T16:15:16.410689+0000","last_peered":"2024-04-16T16:15:16.410689+0000","last_clean":"2024-04-16T16:15:16.410689+0000","last_became_active":"2024-04-16T16:15:14.405872+0000","last_became_peered":"2024-04-16T16:15:14.405872+0000","last_unstale":"2024-04-16T16:15:16.410689+0000","last_undegraded":"2024-04-16T16:15:16.410689+0000","last_fullsized":"2024-04-16T16:15:16.410689+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:12:11.062203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.805181+0000","last_change":"2024-04-16T16:15:15.442790+0000","last_active":"2024-04-16T16:15:16.805181+0000","last_peered":"2024-04-16T16:15:16.805181+0000","last_clean":"2024-04-16T16:15:16.805181+0000","last_became_active":"2024-04-16T16:15:15.439265+0000","last_became_peered":"2024-04-16T16:15:15.439265+0000","last_unstale":"2024-04-16T16:15:16.805181+0000","last_undegraded":"2024-04-16T16:15:16.805181+0000","last_fullsized":"2024-04-16T16:15:16.805181+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:40:41.651741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"41'2","reported_seq":16,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800708+0000","last_change":"2024-04-16T16:15:14.420778+0000","last_active":"2024-04-16T16:15:16.800708+0000","last_peered":"2024-04-16T16:15:16.800708+0000","last_clean":"2024-04-16T16:15:16.800708+0000","last_became_active":"2024-04-16T16:15:14.420315+0000","last_became_peered":"2024-04-16T16:15:14.420315+0000","last_unstale":"2024-04-16T16:15:16.800708+0000","last_undegraded":"2024-04-16T16:15:16.800708+0000","last_fullsized":"2024-04-16T16:15:16.800708+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:43:43.352649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.506779+0000","last_change":"2024-04-16T16:15:15.446817+0000","last_active":"2024-04-16T16:15:16.506779+0000","last_peered":"2024-04-16T16:15:16.506779+0000","last_clean":"2024-04-16T16:15:16.506779+0000","last_became_active":"2024-04-16T16:15:15.444908+0000","last_became_peered":"2024-04-16T16:15:15.444908+0000","last_unstale":"2024-04-16T16:15:16.506779+0000","last_undegraded":"2024-04-16T16:15:16.506779+0000","last_fullsized":"2024-04-16T16:15:16.506779+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:46.819298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"41'1","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417390+0000","last_change":"2024-04-16T16:15:14.417893+0000","last_active":"2024-04-16T16:15:16.417390+0000","last_peered":"2024-04-16T16:15:16.417390+0000","last_clean":"2024-04-16T16:15:16.417390+0000","last_became_active":"2024-04-16T16:15:14.416107+0000","last_became_peered":"2024-04-16T16:15:14.416107+0000","last_unstale":"2024-04-16T16:15:16.417390+0000","last_undegraded":"2024-04-16T16:15:16.417390+0000","last_fullsized":"2024-04-16T16:15:16.417390+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:59.375922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.507204+0000","last_change":"2024-04-16T16:15:15.451372+0000","last_active":"2024-04-16T16:15:16.507204+0000","last_peered":"2024-04-16T16:15:16.507204+0000","last_clean":"2024-04-16T16:15:16.507204+0000","last_became_active":"2024-04-16T16:15:15.450983+0000","last_became_peered":"2024-04-16T16:15:15.450983+0000","last_unstale":"2024-04-16T16:15:16.507204+0000","last_undegraded":"2024-04-16T16:15:16.507204+0000","last_fullsized":"2024-04-16T16:15:16.507204+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:05:00.389608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804856+0000","last_change":"2024-04-16T16:15:14.423513+0000","last_active":"2024-04-16T16:15:16.804856+0000","last_peered":"2024-04-16T16:15:16.804856+0000","last_clean":"2024-04-16T16:15:16.804856+0000","last_became_active":"2024-04-16T16:15:14.423209+0000","last_became_peered":"2024-04-16T16:15:14.423209+0000","last_unstale":"2024-04-16T16:15:16.804856+0000","last_undegraded":"2024-04-16T16:15:16.804856+0000","last_fullsized":"2024-04-16T16:15:16.804856+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:41.421090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803514+0000","last_change":"2024-04-16T16:15:15.443802+0000","last_active":"2024-04-16T16:15:16.803514+0000","last_peered":"2024-04-16T16:15:16.803514+0000","last_clean":"2024-04-16T16:15:16.803514+0000","last_became_active":"2024-04-16T16:15:15.443493+0000","last_became_peered":"2024-04-16T16:15:15.443493+0000","last_unstale":"2024-04-16T16:15:16.803514+0000","last_undegraded":"2024-04-16T16:15:16.803514+0000","last_fullsized":"2024-04-16T16:15:16.803514+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:31:00.851752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"41'2","reported_seq":16,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804551+0000","last_change":"2024-04-16T16:15:14.419251+0000","last_active":"2024-04-16T16:15:16.804551+0000","last_peered":"2024-04-16T16:15:16.804551+0000","last_clean":"2024-04-16T16:15:16.804551+0000","last_became_active":"2024-04-16T16:15:14.418965+0000","last_became_peered":"2024-04-16T16:15:14.418965+0000","last_unstale":"2024-04-16T16:15:16.804551+0000","last_undegraded":"2024-04-16T16:15:16.804551+0000","last_fullsized":"2024-04-16T16:15:16.804551+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:11.262725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.799964+0000","last_change":"2024-04-16T16:15:15.450303+0000","last_active":"2024-04-16T16:15:16.799964+0000","last_peered":"2024-04-16T16:15:16.799964+0000","last_clean":"2024-04-16T16:15:16.799964+0000","last_became_active":"2024-04-16T16:15:15.449983+0000","last_became_peered":"2024-04-16T16:15:15.449983+0000","last_unstale":"2024-04-16T16:15:16.799964+0000","last_undegraded":"2024-04-16T16:15:16.799964+0000","last_fullsized":"2024-04-16T16:15:16.799964+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:46.819298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417060+0000","last_change":"2024-04-16T16:15:14.423634+0000","last_active":"2024-04-16T16:15:16.417060+0000","last_peered":"2024-04-16T16:15:16.417060+0000","last_clean":"2024-04-16T16:15:16.417060+0000","last_became_active":"2024-04-16T16:15:14.422076+0000","last_became_peered":"2024-04-16T16:15:14.422076+0000","last_unstale":"2024-04-16T16:15:16.417060+0000","last_undegraded":"2024-04-16T16:15:16.417060+0000","last_fullsized":"2024-04-16T16:15:16.417060+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:43.504451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.411072+0000","last_change":"2024-04-16T16:15:15.440012+0000","last_active":"2024-04-16T16:15:16.411072+0000","last_peered":"2024-04-16T16:15:16.411072+0000","last_clean":"2024-04-16T16:15:16.411072+0000","last_became_active":"2024-04-16T16:15:15.439703+0000","last_became_peered":"2024-04-16T16:15:15.439703+0000","last_unstale":"2024-04-16T16:15:16.411072+0000","last_undegraded":"2024-04-16T16:15:16.411072+0000","last_fullsized":"2024-04-16T16:15:16.411072+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:10:07.586480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801336+0000","last_change":"2024-04-16T16:15:14.426381+0000","last_active":"2024-04-16T16:15:16.801336+0000","last_peered":"2024-04-16T16:15:16.801336+0000","last_clean":"2024-04-16T16:15:16.801336+0000","last_became_active":"2024-04-16T16:15:14.425924+0000","last_became_peered":"2024-04-16T16:15:14.425924+0000","last_unstale":"2024-04-16T16:15:16.801336+0000","last_undegraded":"2024-04-16T16:15:16.801336+0000","last_fullsized":"2024-04-16T16:15:16.801336+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:07:59.840158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800909+0000","last_change":"2024-04-16T16:15:15.463514+0000","last_active":"2024-04-16T16:15:16.800909+0000","last_peered":"2024-04-16T16:15:16.800909+0000","last_clean":"2024-04-16T16:15:16.800909+0000","last_became_active":"2024-04-16T16:15:15.461956+0000","last_became_peered":"2024-04-16T16:15:15.461956+0000","last_unstale":"2024-04-16T16:15:16.800909+0000","last_undegraded":"2024-04-16T16:15:16.800909+0000","last_fullsized":"2024-04-16T16:15:16.800909+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:34.312744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.415603+0000","last_change":"2024-04-16T16:15:14.419487+0000","last_active":"2024-04-16T16:15:16.415603+0000","last_peered":"2024-04-16T16:15:16.415603+0000","last_clean":"2024-04-16T16:15:16.415603+0000","last_became_active":"2024-04-16T16:15:14.419154+0000","last_became_peered":"2024-04-16T16:15:14.419154+0000","last_unstale":"2024-04-16T16:15:16.415603+0000","last_undegraded":"2024-04-16T16:15:16.415603+0000","last_fullsized":"2024-04-16T16:15:16.415603+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:01:19.702926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.507532+0000","last_change":"2024-04-16T16:15:15.463217+0000","last_active":"2024-04-16T16:15:16.507532+0000","last_peered":"2024-04-16T16:15:16.507532+0000","last_clean":"2024-04-16T16:15:16.507532+0000","last_became_active":"2024-04-16T16:15:15.462748+0000","last_became_peered":"2024-04-16T16:15:15.462748+0000","last_unstale":"2024-04-16T16:15:16.507532+0000","last_undegraded":"2024-04-16T16:15:16.507532+0000","last_fullsized":"2024-04-16T16:15:16.507532+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:03.163739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804294+0000","last_change":"2024-04-16T16:15:14.424803+0000","last_active":"2024-04-16T16:15:16.804294+0000","last_peered":"2024-04-16T16:15:16.804294+0000","last_clean":"2024-04-16T16:15:16.804294+0000","last_became_active":"2024-04-16T16:15:14.424493+0000","last_became_peered":"2024-04-16T16:15:14.424493+0000","last_unstale":"2024-04-16T16:15:16.804294+0000","last_undegraded":"2024-04-16T16:15:16.804294+0000","last_fullsized":"2024-04-16T16:15:16.804294+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:33.299058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801352+0000","last_change":"2024-04-16T16:15:15.451816+0000","last_active":"2024-04-16T16:15:16.801352+0000","last_peered":"2024-04-16T16:15:16.801352+0000","last_clean":"2024-04-16T16:15:16.801352+0000","last_became_active":"2024-04-16T16:15:15.450017+0000","last_became_peered":"2024-04-16T16:15:15.450017+0000","last_unstale":"2024-04-16T16:15:16.801352+0000","last_undegraded":"2024-04-16T16:15:16.801352+0000","last_fullsized":"2024-04-16T16:15:16.801352+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:42.434776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.418079+0000","last_change":"2024-04-16T16:15:14.426984+0000","last_active":"2024-04-16T16:15:16.418079+0000","last_peered":"2024-04-16T16:15:16.418079+0000","last_clean":"2024-04-16T16:15:16.418079+0000","last_became_active":"2024-04-16T16:15:14.426689+0000","last_became_peered":"2024-04-16T16:15:14.426689+0000","last_unstale":"2024-04-16T16:15:16.418079+0000","last_undegraded":"2024-04-16T16:15:16.418079+0000","last_fullsized":"2024-04-16T16:15:16.418079+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:02.150053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417841+0000","last_change":"2024-04-16T16:15:15.449317+0000","last_active":"2024-04-16T16:15:16.417841+0000","last_peered":"2024-04-16T16:15:16.417841+0000","last_clean":"2024-04-16T16:15:16.417841+0000","last_became_active":"2024-04-16T16:15:15.448646+0000","last_became_peered":"2024-04-16T16:15:15.448646+0000","last_unstale":"2024-04-16T16:15:16.417841+0000","last_undegraded":"2024-04-16T16:15:16.417841+0000","last_fullsized":"2024-04-16T16:15:16.417841+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:11.662435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.799570+0000","last_change":"2024-04-16T16:15:14.424278+0000","last_active":"2024-04-16T16:15:16.799570+0000","last_peered":"2024-04-16T16:15:16.799570+0000","last_clean":"2024-04-16T16:15:16.799570+0000","last_became_active":"2024-04-16T16:15:14.423174+0000","last_became_peered":"2024-04-16T16:15:14.423174+0000","last_unstale":"2024-04-16T16:15:16.799570+0000","last_undegraded":"2024-04-16T16:15:16.799570+0000","last_fullsized":"2024-04-16T16:15:16.799570+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:19:05.074032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804235+0000","last_change":"2024-04-16T16:15:15.443473+0000","last_active":"2024-04-16T16:15:16.804235+0000","last_peered":"2024-04-16T16:15:16.804235+0000","last_clean":"2024-04-16T16:15:16.804235+0000","last_became_active":"2024-04-16T16:15:15.439092+0000","last_became_peered":"2024-04-16T16:15:15.439092+0000","last_unstale":"2024-04-16T16:15:16.804235+0000","last_undegraded":"2024-04-16T16:15:16.804235+0000","last_fullsized":"2024-04-16T16:15:16.804235+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:49:28.175995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.416428+0000","last_change":"2024-04-16T16:15:14.427316+0000","last_active":"2024-04-16T16:15:16.416428+0000","last_peered":"2024-04-16T16:15:16.416428+0000","last_clean":"2024-04-16T16:15:16.416428+0000","last_became_active":"2024-04-16T16:15:14.426635+0000","last_became_peered":"2024-04-16T16:15:14.426635+0000","last_unstale":"2024-04-16T16:15:16.416428+0000","last_undegraded":"2024-04-16T16:15:16.416428+0000","last_fullsized":"2024-04-16T16:15:16.416428+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:43.504451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803680+0000","last_change":"2024-04-16T16:15:15.440497+0000","last_active":"2024-04-16T16:15:16.803680+0000","last_peered":"2024-04-16T16:15:16.803680+0000","last_clean":"2024-04-16T16:15:16.803680+0000","last_became_active":"2024-04-16T16:15:15.440205+0000","last_became_peered":"2024-04-16T16:15:15.440205+0000","last_unstale":"2024-04-16T16:15:16.803680+0000","last_undegraded":"2024-04-16T16:15:16.803680+0000","last_fullsized":"2024-04-16T16:15:16.803680+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:29:51.615992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800249+0000","last_change":"2024-04-16T16:15:14.414769+0000","last_active":"2024-04-16T16:15:16.800249+0000","last_peered":"2024-04-16T16:15:16.800249+0000","last_clean":"2024-04-16T16:15:16.800249+0000","last_became_active":"2024-04-16T16:15:14.414402+0000","last_became_peered":"2024-04-16T16:15:14.414402+0000","last_unstale":"2024-04-16T16:15:16.800249+0000","last_undegraded":"2024-04-16T16:15:16.800249+0000","last_fullsized":"2024-04-16T16:15:16.800249+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:07:59.840158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.415822+0000","last_change":"2024-04-16T16:15:15.446666+0000","last_active":"2024-04-16T16:15:16.415822+0000","last_peered":"2024-04-16T16:15:16.415822+0000","last_clean":"2024-04-16T16:15:16.415822+0000","last_became_active":"2024-04-16T16:15:15.444672+0000","last_became_peered":"2024-04-16T16:15:15.444672+0000","last_unstale":"2024-04-16T16:15:16.415822+0000","last_undegraded":"2024-04-16T16:15:16.415822+0000","last_fullsized":"2024-04-16T16:15:16.415822+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:39:57.997770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.799321+0000","last_change":"2024-04-16T16:15:14.425948+0000","last_active":"2024-04-16T16:15:16.799321+0000","last_peered":"2024-04-16T16:15:16.799321+0000","last_clean":"2024-04-16T16:15:16.799321+0000","last_became_active":"2024-04-16T16:15:14.425692+0000","last_became_peered":"2024-04-16T16:15:14.425692+0000","last_unstale":"2024-04-16T16:15:16.799321+0000","last_undegraded":"2024-04-16T16:15:16.799321+0000","last_fullsized":"2024-04-16T16:15:16.799321+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:16.172813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800003+0000","last_change":"2024-04-16T16:15:15.448491+0000","last_active":"2024-04-16T16:15:16.800003+0000","last_peered":"2024-04-16T16:15:16.800003+0000","last_clean":"2024-04-16T16:15:16.800003+0000","last_became_active":"2024-04-16T16:15:15.448176+0000","last_became_peered":"2024-04-16T16:15:15.448176+0000","last_unstale":"2024-04-16T16:15:16.800003+0000","last_undegraded":"2024-04-16T16:15:16.800003+0000","last_fullsized":"2024-04-16T16:15:16.800003+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:44.518137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801815+0000","last_change":"2024-04-16T16:15:14.413186+0000","last_active":"2024-04-16T16:15:16.801815+0000","last_peered":"2024-04-16T16:15:16.801815+0000","last_clean":"2024-04-16T16:15:16.801815+0000","last_became_active":"2024-04-16T16:15:14.412710+0000","last_became_peered":"2024-04-16T16:15:14.412710+0000","last_unstale":"2024-04-16T16:15:16.801815+0000","last_undegraded":"2024-04-16T16:15:16.801815+0000","last_fullsized":"2024-04-16T16:15:16.801815+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:43:43.352649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.802050+0000","last_change":"2024-04-16T16:15:15.444726+0000","last_active":"2024-04-16T16:15:16.802050+0000","last_peered":"2024-04-16T16:15:16.802050+0000","last_clean":"2024-04-16T16:15:16.802050+0000","last_became_active":"2024-04-16T16:15:15.444386+0000","last_became_peered":"2024-04-16T16:15:15.444386+0000","last_unstale":"2024-04-16T16:15:16.802050+0000","last_undegraded":"2024-04-16T16:15:16.802050+0000","last_fullsized":"2024-04-16T16:15:16.802050+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:34.706031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"41'2","reported_seq":16,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.418458+0000","last_change":"2024-04-16T16:15:14.420968+0000","last_active":"2024-04-16T16:15:16.418458+0000","last_peered":"2024-04-16T16:15:16.418458+0000","last_clean":"2024-04-16T16:15:16.418458+0000","last_became_active":"2024-04-16T16:15:14.418308+0000","last_became_peered":"2024-04-16T16:15:14.418308+0000","last_unstale":"2024-04-16T16:15:16.418458+0000","last_undegraded":"2024-04-16T16:15:16.418458+0000","last_fullsized":"2024-04-16T16:15:16.418458+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:56.611238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"35'128","reported_seq":251,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.415584+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:15:16.415584+0000","last_peered":"2024-04-16T16:15:16.415584+0000","last_clean":"2024-04-16T16:15:16.415584+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:15:16.415584+0000","last_undegraded":"2024-04-16T16:15:16.415584+0000","last_fullsized":"2024-04-16T16:15:16.415584+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343752,"statfs":{"total":96636764160,"available":96608002048,"internally_reserved":0,"allocated":1089536,"data_stored":793072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":499712,"data_stored":203286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":495616,"data_stored":202684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27512,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344328,"statfs":{"total":96636764160,"available":96608591872,"internally_reserved":0,"allocated":503808,"data_stored":204226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":203252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":507904,"data_stored":204248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":483328,"data_stored":196055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32232,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339608,"statfs":{"total":96636764160,"available":96603758592,"internally_reserved":0,"allocated":1077248,"data_stored":786387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:15:19.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:15:19.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-04-16T16:15:20.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.196+0000 7f196b600640 1 Processor -- start 2024-04-16T16:15:20.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.196+0000 7f196b600640 1 -- start start 2024-04-16T16:15:20.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.197+0000 7f196b600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1964072a90 0x7f19641699a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:20.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.197+0000 7f196b600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1964147c20 con 0x7f19640aed90 2024-04-16T16:15:20.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.197+0000 7f196b600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1964160210 con 0x7f1964147840 2024-04-16T16:15:20.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.197+0000 7f196b600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f19640afe80 con 0x7f1964072a90 2024-04-16T16:15:20.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.198+0000 7f1963e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f19640aed90 0x7f1964072380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42060/0 (socket says 172.21.15.45:42060) 2024-04-16T16:15:20.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.198+0000 7f1963e00640 1 -- 172.21.15.45:0/3296903914 learned_addr learned my addr 172.21.15.45:0/3296903914 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:20.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.198+0000 7f1969600640 1 --2- 172.21.15.45:0/3296903914 >> v2:172.21.15.103:3300/0 conn(0x7f1964072a90 0x7f19641699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:20.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.198+0000 7f1963400640 1 -- 172.21.15.45:0/3296903914 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3319480415 0 0) 0x7f1964147c20 con 0x7f19640aed90 2024-04-16T16:15:20.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.198+0000 7f1963400640 1 -- 172.21.15.45:0/3296903914 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f194c003520 con 0x7f19640aed90 2024-04-16T16:15:20.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1963400640 1 -- 172.21.15.45:0/3296903914 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1306985093 0 0) 0x7f194c003520 con 0x7f19640aed90 2024-04-16T16:15:20.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1963400640 1 -- 172.21.15.45:0/3296903914 >> v2:172.21.15.103:3300/0 conn(0x7f1964072a90 msgr2=0x7f19641699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:20.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1963400640 1 --2- 172.21.15.45:0/3296903914 >> v2:172.21.15.103:3300/0 conn(0x7f1964072a90 0x7f19641699a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:20.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1963400640 1 -- 172.21.15.45:0/3296903914 >> v1:172.21.15.103:6789/0 conn(0x7f1964147840 legacy=0x7f196415fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:20.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1963400640 1 -- 172.21.15.45:0/3296903914 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19640af270 con 0x7f19640aed90 2024-04-16T16:15:20.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1963400640 1 -- 172.21.15.45:0/3296903914 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f1950001670 con 0x7f19640aed90 2024-04-16T16:15:20.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1963400640 1 -- 172.21.15.45:0/3296903914 >> v1:172.21.15.45:6789/0 conn(0x7f19640aed90 legacy=0x7f1964072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:20.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1963400640 1 --2- 172.21.15.45:0/3296903914 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194c0039b0 0x7f194c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:20.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1963400640 1 --2- 172.21.15.45:0/3296903914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f194c004800 0x7f194c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1963400640 1 --2- 172.21.15.45:0/3296903914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c0051c0 0x7f194c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1963400640 1 -- 172.21.15.45:0/3296903914 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f19640afe80 con 0x7f194c0051c0 2024-04-16T16:15:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1963400640 1 -- 172.21.15.45:0/3296903914 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f194c00c0c0 con 0x7f194c004800 2024-04-16T16:15:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1963400640 1 -- 172.21.15.45:0/3296903914 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f194c00c2c0 con 0x7f194c0039b0 2024-04-16T16:15:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1969600640 1 --2- 172.21.15.45:0/3296903914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c0051c0 0x7f194c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1963e00640 1 --2- 172.21.15.45:0/3296903914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f194c004800 0x7f194c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.199+0000 7f1968c00640 1 --2- 172.21.15.45:0/3296903914 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194c0039b0 0x7f194c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.200+0000 7f1969600640 1 -- 172.21.15.45:0/3296903914 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194c0039b0 msgr2=0x7f194c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.200+0000 7f1969600640 1 --2- 172.21.15.45:0/3296903914 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194c0039b0 0x7f194c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.200+0000 7f1969600640 1 -- 172.21.15.45:0/3296903914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f194c004800 msgr2=0x7f194c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:20.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.200+0000 7f1969600640 1 --2- 172.21.15.45:0/3296903914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f194c004800 0x7f194c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:20.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.200+0000 7f1969600640 1 -- 172.21.15.45:0/3296903914 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19640af270 con 0x7f194c0051c0 2024-04-16T16:15:20.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.200+0000 7f1963e00640 1 --2- 172.21.15.45:0/3296903914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f194c004800 0x7f194c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:20.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.201+0000 7f1969600640 1 -- 172.21.15.45:0/3296903914 reap_dead start 2024-04-16T16:15:20.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.201+0000 7f1969600640 1 --2- 172.21.15.45:0/3296903914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c0051c0 0x7f194c00b9c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f195c007330 tx=0x7f194c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e5d17942a17cda server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:20.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.202+0000 7f1963400640 1 -- 172.21.15.45:0/3296903914 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c009be0 con 0x7f194c0051c0 2024-04-16T16:15:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.202+0000 7f1963400640 1 -- 172.21.15.45:0/3296903914 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f195c02ccd0 con 0x7f194c0051c0 2024-04-16T16:15:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.202+0000 7f1963400640 1 -- 172.21.15.45:0/3296903914 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c003bd0 con 0x7f194c0051c0 2024-04-16T16:15:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.202+0000 7f196b600640 1 -- 172.21.15.45:0/3296903914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c0051c0 msgr2=0x7f194c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.202+0000 7f196b600640 1 --2- 172.21.15.45:0/3296903914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c0051c0 0x7f194c00b9c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f195c007330 tx=0x7f194c005830 comp rx=0 tx=0).stop 2024-04-16T16:15:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.202+0000 7f196b600640 1 -- 172.21.15.45:0/3296903914 shutdown_connections 2024-04-16T16:15:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.202+0000 7f196b600640 1 --2- 172.21.15.45:0/3296903914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c0051c0 0x7f194c00b9c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.202+0000 7f196b600640 1 -- 172.21.15.45:0/3296903914 >> 172.21.15.45:0/3296903914 conn(0x7f19640b2750 msgr2=0x7f19640b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.202+0000 7f196b600640 1 -- 172.21.15.45:0/3296903914 shutdown_connections 2024-04-16T16:15:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.202+0000 7f196b600640 1 -- 172.21.15.45:0/3296903914 wait complete. 2024-04-16T16:15:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.204+0000 7f196b600640 1 Processor -- start 2024-04-16T16:15:20.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.204+0000 7f196b600640 1 -- start start 2024-04-16T16:15:20.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.205+0000 7f196b600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c0051c0 0x7f196415e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.205+0000 7f196b600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f196413a930 0x7f196413adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.205+0000 7f196b600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f196413cc50 0x7f196413d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.205+0000 7f196b600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f19640afb10 con 0x7f194c0051c0 2024-04-16T16:15:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.205+0000 7f196b600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f19640af830 con 0x7f196413cc50 2024-04-16T16:15:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.205+0000 7f196b600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1964072380 con 0x7f196413a930 2024-04-16T16:15:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.206+0000 7f1968c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c0051c0 0x7f196415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.206+0000 7f1968c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c0051c0 0x7f196415e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57002/0 (socket says 172.21.15.45:57002) 2024-04-16T16:15:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.206+0000 7f1968c00640 1 -- 172.21.15.45:0/4069049579 learned_addr learned my addr 172.21.15.45:0/4069049579 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.206+0000 7f1968c00640 1 -- 172.21.15.45:0/4069049579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f196413a930 msgr2=0x7f196413adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.206+0000 7f1968c00640 1 --2- 172.21.15.45:0/4069049579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f196413a930 0x7f196413adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.206+0000 7f1968c00640 1 -- 172.21.15.45:0/4069049579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f196413cc50 msgr2=0x7f196413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.206+0000 7f1968c00640 1 --2- 172.21.15.45:0/4069049579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f196413cc50 0x7f196413d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.206+0000 7f1968c00640 1 -- 172.21.15.45:0/4069049579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f196415e550 con 0x7f194c0051c0 2024-04-16T16:15:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.206+0000 7f1968c00640 1 --2- 172.21.15.45:0/4069049579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c0051c0 0x7f196415e010 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f195800e490 tx=0x7f195800e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.207+0000 7f1960c00640 1 -- 172.21.15.45:0/4069049579 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19580135f0 con 0x7f194c0051c0 2024-04-16T16:15:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.207+0000 7f1960c00640 1 -- 172.21.15.45:0/4069049579 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f195801a070 con 0x7f194c0051c0 2024-04-16T16:15:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.207+0000 7f1960c00640 1 -- 172.21.15.45:0/4069049579 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19580139d0 con 0x7f194c0051c0 2024-04-16T16:15:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.207+0000 7f196b600640 1 -- 172.21.15.45:0/4069049579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f196413b6f0 con 0x7f194c0051c0 2024-04-16T16:15:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.207+0000 7f196b600640 1 -- 172.21.15.45:0/4069049579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f196413c450 con 0x7f194c0051c0 2024-04-16T16:15:20.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.208+0000 7f196b600640 1 -- 172.21.15.45:0/4069049579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f196406c7c0 con 0x7f194c0051c0 2024-04-16T16:15:20.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.209+0000 7f1960c00640 1 -- 172.21.15.45:0/4069049579 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f1958013b70 con 0x7f194c0051c0 2024-04-16T16:15:20.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.210+0000 7f1960c00640 1 --2- 172.21.15.45:0/4069049579 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f194406ca90 0x7f194406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:20.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.210+0000 7f1960c00640 1 -- 172.21.15.45:0/4069049579 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6041+0+0 (secure 0 0 0) 0x7f195808bb80 con 0x7f194c0051c0 2024-04-16T16:15:20.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.211+0000 7f1963e00640 1 --2- 172.21.15.45:0/4069049579 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f194406ca90 0x7f194406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:20.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.212+0000 7f1963e00640 1 --2- 172.21.15.45:0/4069049579 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f194406ca90 0x7f194406ef50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f19500017e0 tx=0x7f1950009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:20.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.214+0000 7f1960c00640 1 -- 172.21.15.45:0/4069049579 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f196406c7c0 con 0x7f194c0051c0 2024-04-16T16:15:20.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.414+0000 7f196b600640 1 -- 172.21.15.45:0/4069049579 --> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1964075780 con 0x7f194406ca90 2024-04-16T16:15:20.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.433+0000 7f1960c00640 1 -- 172.21.15.45:0/4069049579 <== mgr.4658 v2:172.21.15.103:6832/280591707 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251956 (secure 0 0 0) 0x7f1964075780 con 0x7f194406ca90 2024-04-16T16:15:20.437 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:20.439 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:15:20.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.440+0000 7f196b600640 1 -- 172.21.15.45:0/4069049579 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f194406ca90 msgr2=0x7f194406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:20.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.440+0000 7f196b600640 1 --2- 172.21.15.45:0/4069049579 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f194406ca90 0x7f194406ef50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f19500017e0 tx=0x7f1950009040 comp rx=0 tx=0).stop 2024-04-16T16:15:20.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.440+0000 7f196b600640 1 -- 172.21.15.45:0/4069049579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c0051c0 msgr2=0x7f196415e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:20.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.440+0000 7f196b600640 1 --2- 172.21.15.45:0/4069049579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c0051c0 0x7f196415e010 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f195800e490 tx=0x7f195800e950 comp rx=0 tx=0).stop 2024-04-16T16:15:20.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.440+0000 7f196b600640 1 -- 172.21.15.45:0/4069049579 shutdown_connections 2024-04-16T16:15:20.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.440+0000 7f196b600640 1 --2- 172.21.15.45:0/4069049579 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f194406ca90 0x7f194406ef50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:20.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.440+0000 7f196b600640 1 --2- 172.21.15.45:0/4069049579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f196413cc50 0x7f196413d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:20.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.440+0000 7f196b600640 1 --2- 172.21.15.45:0/4069049579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f196413a930 0x7f196413adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:20.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.440+0000 7f196b600640 1 --2- 172.21.15.45:0/4069049579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194c0051c0 0x7f196415e010 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:20.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.440+0000 7f196b600640 1 -- 172.21.15.45:0/4069049579 >> 172.21.15.45:0/4069049579 conn(0x7f19640b2750 msgr2=0x7f19640b5550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:20.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.440+0000 7f196b600640 1 -- 172.21.15.45:0/4069049579 shutdown_connections 2024-04-16T16:15:20.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.440+0000 7f196b600640 1 -- 172.21.15.45:0/4069049579 wait complete. 2024-04-16T16:15:20.460 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-04-16T16:15:19.127758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":223,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166,"ondisk_log_size":166,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":149,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237852,"kb_used_data":5040,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736868,"statfs":{"total":773094113280,"available":772850552832,"internally_reserved":0,"allocated":5160960,"data_stored":2793210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.748774"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.416715+0000","last_change":"2024-04-16T16:15:15.445346+0000","last_active":"2024-04-16T16:15:16.416715+0000","last_peered":"2024-04-16T16:15:16.416715+0000","last_clean":"2024-04-16T16:15:16.416715+0000","last_became_active":"2024-04-16T16:15:15.445023+0000","last_became_peered":"2024-04-16T16:15:15.445023+0000","last_unstale":"2024-04-16T16:15:16.416715+0000","last_undegraded":"2024-04-16T16:15:16.416715+0000","last_fullsized":"2024-04-16T16:15:16.416715+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:51.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.418591+0000","last_change":"2024-04-16T16:15:15.451103+0000","last_active":"2024-04-16T16:15:16.418591+0000","last_peered":"2024-04-16T16:15:16.418591+0000","last_clean":"2024-04-16T16:15:16.418591+0000","last_became_active":"2024-04-16T16:15:15.450511+0000","last_became_peered":"2024-04-16T16:15:15.450511+0000","last_unstale":"2024-04-16T16:15:16.418591+0000","last_undegraded":"2024-04-16T16:15:16.418591+0000","last_fullsized":"2024-04-16T16:15:16.418591+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:34.706031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800293+0000","last_change":"2024-04-16T16:15:15.454071+0000","last_active":"2024-04-16T16:15:16.800293+0000","last_peered":"2024-04-16T16:15:16.800293+0000","last_clean":"2024-04-16T16:15:16.800293+0000","last_became_active":"2024-04-16T16:15:15.453781+0000","last_became_peered":"2024-04-16T16:15:15.453781+0000","last_unstale":"2024-04-16T16:15:16.800293+0000","last_undegraded":"2024-04-16T16:15:16.800293+0000","last_fullsized":"2024-04-16T16:15:16.800293+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:03.163739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801186+0000","last_change":"2024-04-16T16:15:15.459192+0000","last_active":"2024-04-16T16:15:16.801186+0000","last_peered":"2024-04-16T16:15:16.801186+0000","last_clean":"2024-04-16T16:15:16.801186+0000","last_became_active":"2024-04-16T16:15:15.452689+0000","last_became_peered":"2024-04-16T16:15:15.452689+0000","last_unstale":"2024-04-16T16:15:16.801186+0000","last_undegraded":"2024-04-16T16:15:16.801186+0000","last_fullsized":"2024-04-16T16:15:16.801186+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:39:57.997770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417373+0000","last_change":"2024-04-16T16:15:15.443526+0000","last_active":"2024-04-16T16:15:16.417373+0000","last_peered":"2024-04-16T16:15:16.417373+0000","last_clean":"2024-04-16T16:15:16.417373+0000","last_became_active":"2024-04-16T16:15:15.443203+0000","last_became_peered":"2024-04-16T16:15:15.443203+0000","last_unstale":"2024-04-16T16:15:16.417373+0000","last_undegraded":"2024-04-16T16:15:16.417373+0000","last_fullsized":"2024-04-16T16:15:16.417373+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:42.434776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803362+0000","last_change":"2024-04-16T16:15:15.429037+0000","last_active":"2024-04-16T16:15:16.803362+0000","last_peered":"2024-04-16T16:15:16.803362+0000","last_clean":"2024-04-16T16:15:16.803362+0000","last_became_active":"2024-04-16T16:15:15.425570+0000","last_became_peered":"2024-04-16T16:15:15.425570+0000","last_unstale":"2024-04-16T16:15:16.803362+0000","last_undegraded":"2024-04-16T16:15:16.803362+0000","last_fullsized":"2024-04-16T16:15:16.803362+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:17:00.492572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.416489+0000","last_change":"2024-04-16T16:15:15.439872+0000","last_active":"2024-04-16T16:15:16.416489+0000","last_peered":"2024-04-16T16:15:16.416489+0000","last_clean":"2024-04-16T16:15:16.416489+0000","last_became_active":"2024-04-16T16:15:15.439553+0000","last_became_peered":"2024-04-16T16:15:15.439553+0000","last_unstale":"2024-04-16T16:15:16.416489+0000","last_undegraded":"2024-04-16T16:15:16.416489+0000","last_fullsized":"2024-04-16T16:15:16.416489+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:12.276411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.802555+0000","last_change":"2024-04-16T16:15:15.463187+0000","last_active":"2024-04-16T16:15:16.802555+0000","last_peered":"2024-04-16T16:15:16.802555+0000","last_clean":"2024-04-16T16:15:16.802555+0000","last_became_active":"2024-04-16T16:15:15.461753+0000","last_became_peered":"2024-04-16T16:15:15.461753+0000","last_unstale":"2024-04-16T16:15:16.802555+0000","last_undegraded":"2024-04-16T16:15:16.802555+0000","last_fullsized":"2024-04-16T16:15:16.802555+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:44.974637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804974+0000","last_change":"2024-04-16T16:15:15.451403+0000","last_active":"2024-04-16T16:15:16.804974+0000","last_peered":"2024-04-16T16:15:16.804974+0000","last_clean":"2024-04-16T16:15:16.804974+0000","last_became_active":"2024-04-16T16:15:15.450199+0000","last_became_peered":"2024-04-16T16:15:15.450199+0000","last_unstale":"2024-04-16T16:15:16.804974+0000","last_undegraded":"2024-04-16T16:15:16.804974+0000","last_fullsized":"2024-04-16T16:15:16.804974+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:32:52.123986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801619+0000","last_change":"2024-04-16T16:15:15.459905+0000","last_active":"2024-04-16T16:15:16.801619+0000","last_peered":"2024-04-16T16:15:16.801619+0000","last_clean":"2024-04-16T16:15:16.801619+0000","last_became_active":"2024-04-16T16:15:15.459387+0000","last_became_peered":"2024-04-16T16:15:15.459387+0000","last_unstale":"2024-04-16T16:15:16.801619+0000","last_undegraded":"2024-04-16T16:15:16.801619+0000","last_fullsized":"2024-04-16T16:15:16.801619+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:51.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801092+0000","last_change":"2024-04-16T16:15:15.463054+0000","last_active":"2024-04-16T16:15:16.801092+0000","last_peered":"2024-04-16T16:15:16.801092+0000","last_clean":"2024-04-16T16:15:16.801092+0000","last_became_active":"2024-04-16T16:15:15.462075+0000","last_became_peered":"2024-04-16T16:15:15.462075+0000","last_unstale":"2024-04-16T16:15:16.801092+0000","last_undegraded":"2024-04-16T16:15:16.801092+0000","last_fullsized":"2024-04-16T16:15:16.801092+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:56.074330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417550+0000","last_change":"2024-04-16T16:15:15.449849+0000","last_active":"2024-04-16T16:15:16.417550+0000","last_peered":"2024-04-16T16:15:16.417550+0000","last_clean":"2024-04-16T16:15:16.417550+0000","last_became_active":"2024-04-16T16:15:15.449585+0000","last_became_peered":"2024-04-16T16:15:15.449585+0000","last_unstale":"2024-04-16T16:15:16.417550+0000","last_undegraded":"2024-04-16T16:15:16.417550+0000","last_fullsized":"2024-04-16T16:15:16.417550+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:12.276411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.410962+0000","last_change":"2024-04-16T16:15:15.441331+0000","last_active":"2024-04-16T16:15:16.410962+0000","last_peered":"2024-04-16T16:15:16.410962+0000","last_clean":"2024-04-16T16:15:16.410962+0000","last_became_active":"2024-04-16T16:15:15.440983+0000","last_became_peered":"2024-04-16T16:15:15.440983+0000","last_unstale":"2024-04-16T16:15:16.410962+0000","last_undegraded":"2024-04-16T16:15:16.410962+0000","last_fullsized":"2024-04-16T16:15:16.410962+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:10:59.131225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801104+0000","last_change":"2024-04-16T16:15:15.459411+0000","last_active":"2024-04-16T16:15:16.801104+0000","last_peered":"2024-04-16T16:15:16.801104+0000","last_clean":"2024-04-16T16:15:16.801104+0000","last_became_active":"2024-04-16T16:15:15.452989+0000","last_became_peered":"2024-04-16T16:15:15.452989+0000","last_unstale":"2024-04-16T16:15:16.801104+0000","last_undegraded":"2024-04-16T16:15:16.801104+0000","last_fullsized":"2024-04-16T16:15:16.801104+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:27.522215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.802979+0000","last_change":"2024-04-16T16:15:15.448356+0000","last_active":"2024-04-16T16:15:16.802979+0000","last_peered":"2024-04-16T16:15:16.802979+0000","last_clean":"2024-04-16T16:15:16.802979+0000","last_became_active":"2024-04-16T16:15:15.448056+0000","last_became_peered":"2024-04-16T16:15:15.448056+0000","last_unstale":"2024-04-16T16:15:16.802979+0000","last_undegraded":"2024-04-16T16:15:16.802979+0000","last_fullsized":"2024-04-16T16:15:16.802979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:30:21.270376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.506646+0000","last_change":"2024-04-16T16:15:15.447460+0000","last_active":"2024-04-16T16:15:16.506646+0000","last_peered":"2024-04-16T16:15:16.506646+0000","last_clean":"2024-04-16T16:15:16.506646+0000","last_became_active":"2024-04-16T16:15:15.447155+0000","last_became_peered":"2024-04-16T16:15:15.447155+0000","last_unstale":"2024-04-16T16:15:16.506646+0000","last_undegraded":"2024-04-16T16:15:16.506646+0000","last_fullsized":"2024-04-16T16:15:16.506646+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:48:05.645437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800986+0000","last_change":"2024-04-16T16:15:15.445583+0000","last_active":"2024-04-16T16:15:16.800986+0000","last_peered":"2024-04-16T16:15:16.800986+0000","last_clean":"2024-04-16T16:15:16.800986+0000","last_became_active":"2024-04-16T16:15:15.445259+0000","last_became_peered":"2024-04-16T16:15:15.445259+0000","last_unstale":"2024-04-16T16:15:16.800986+0000","last_undegraded":"2024-04-16T16:15:16.800986+0000","last_fullsized":"2024-04-16T16:15:16.800986+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:56.074330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800427+0000","last_change":"2024-04-16T16:15:15.451682+0000","last_active":"2024-04-16T16:15:16.800427+0000","last_peered":"2024-04-16T16:15:16.800427+0000","last_clean":"2024-04-16T16:15:16.800427+0000","last_became_active":"2024-04-16T16:15:15.450999+0000","last_became_peered":"2024-04-16T16:15:15.450999+0000","last_unstale":"2024-04-16T16:15:16.800427+0000","last_undegraded":"2024-04-16T16:15:16.800427+0000","last_fullsized":"2024-04-16T16:15:16.800427+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:01:20.716612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.507503+0000","last_change":"2024-04-16T16:15:15.446449+0000","last_active":"2024-04-16T16:15:16.507503+0000","last_peered":"2024-04-16T16:15:16.507503+0000","last_clean":"2024-04-16T16:15:16.507503+0000","last_became_active":"2024-04-16T16:15:15.446135+0000","last_became_peered":"2024-04-16T16:15:15.446135+0000","last_unstale":"2024-04-16T16:15:16.507503+0000","last_undegraded":"2024-04-16T16:15:16.507503+0000","last_fullsized":"2024-04-16T16:15:16.507503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:49.654384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.411023+0000","last_change":"2024-04-16T16:15:15.456287+0000","last_active":"2024-04-16T16:15:16.411023+0000","last_peered":"2024-04-16T16:15:16.411023+0000","last_clean":"2024-04-16T16:15:16.411023+0000","last_became_active":"2024-04-16T16:15:15.455963+0000","last_became_peered":"2024-04-16T16:15:15.455963+0000","last_unstale":"2024-04-16T16:15:16.411023+0000","last_undegraded":"2024-04-16T16:15:16.411023+0000","last_fullsized":"2024-04-16T16:15:16.411023+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:07.752151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803726+0000","last_change":"2024-04-16T16:15:15.447751+0000","last_active":"2024-04-16T16:15:16.803726+0000","last_peered":"2024-04-16T16:15:16.803726+0000","last_clean":"2024-04-16T16:15:16.803726+0000","last_became_active":"2024-04-16T16:15:15.447195+0000","last_became_peered":"2024-04-16T16:15:15.447195+0000","last_unstale":"2024-04-16T16:15:16.803726+0000","last_undegraded":"2024-04-16T16:15:16.803726+0000","last_fullsized":"2024-04-16T16:15:16.803726+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:34:32.934841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801511+0000","last_change":"2024-04-16T16:15:15.462804+0000","last_active":"2024-04-16T16:15:16.801511+0000","last_peered":"2024-04-16T16:15:16.801511+0000","last_clean":"2024-04-16T16:15:16.801511+0000","last_became_active":"2024-04-16T16:15:15.461788+0000","last_became_peered":"2024-04-16T16:15:15.461788+0000","last_unstale":"2024-04-16T16:15:16.801511+0000","last_undegraded":"2024-04-16T16:15:16.801511+0000","last_fullsized":"2024-04-16T16:15:16.801511+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:12.276411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.506488+0000","last_change":"2024-04-16T16:15:15.443351+0000","last_active":"2024-04-16T16:15:16.506488+0000","last_peered":"2024-04-16T16:15:16.506488+0000","last_clean":"2024-04-16T16:15:16.506488+0000","last_became_active":"2024-04-16T16:15:15.442366+0000","last_became_peered":"2024-04-16T16:15:15.442366+0000","last_unstale":"2024-04-16T16:15:16.506488+0000","last_undegraded":"2024-04-16T16:15:16.506488+0000","last_fullsized":"2024-04-16T16:15:16.506488+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:53.599226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.507074+0000","last_change":"2024-04-16T16:15:15.447202+0000","last_active":"2024-04-16T16:15:16.507074+0000","last_peered":"2024-04-16T16:15:16.507074+0000","last_clean":"2024-04-16T16:15:16.507074+0000","last_became_active":"2024-04-16T16:15:15.444539+0000","last_became_peered":"2024-04-16T16:15:15.444539+0000","last_unstale":"2024-04-16T16:15:16.507074+0000","last_undegraded":"2024-04-16T16:15:16.507074+0000","last_fullsized":"2024-04-16T16:15:16.507074+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:33:41.701252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.507294+0000","last_change":"2024-04-16T16:15:15.454894+0000","last_active":"2024-04-16T16:15:16.507294+0000","last_peered":"2024-04-16T16:15:16.507294+0000","last_clean":"2024-04-16T16:15:16.507294+0000","last_became_active":"2024-04-16T16:15:15.454659+0000","last_became_peered":"2024-04-16T16:15:15.454659+0000","last_unstale":"2024-04-16T16:15:16.507294+0000","last_undegraded":"2024-04-16T16:15:16.507294+0000","last_fullsized":"2024-04-16T16:15:16.507294+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:44.518137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.410920+0000","last_change":"2024-04-16T16:15:15.453129+0000","last_active":"2024-04-16T16:15:16.410920+0000","last_peered":"2024-04-16T16:15:16.410920+0000","last_clean":"2024-04-16T16:15:16.410920+0000","last_became_active":"2024-04-16T16:15:15.452843+0000","last_became_peered":"2024-04-16T16:15:15.452843+0000","last_unstale":"2024-04-16T16:15:16.410920+0000","last_undegraded":"2024-04-16T16:15:16.410920+0000","last_fullsized":"2024-04-16T16:15:16.410920+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:28:11.918651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800504+0000","last_change":"2024-04-16T16:15:15.437158+0000","last_active":"2024-04-16T16:15:16.800504+0000","last_peered":"2024-04-16T16:15:16.800504+0000","last_clean":"2024-04-16T16:15:16.800504+0000","last_became_active":"2024-04-16T16:15:15.435103+0000","last_became_peered":"2024-04-16T16:15:15.435103+0000","last_unstale":"2024-04-16T16:15:16.800504+0000","last_undegraded":"2024-04-16T16:15:16.800504+0000","last_fullsized":"2024-04-16T16:15:16.800504+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:34.312744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417574+0000","last_change":"2024-04-16T16:15:15.450932+0000","last_active":"2024-04-16T16:15:16.417574+0000","last_peered":"2024-04-16T16:15:16.417574+0000","last_clean":"2024-04-16T16:15:16.417574+0000","last_became_active":"2024-04-16T16:15:15.447441+0000","last_became_peered":"2024-04-16T16:15:15.447441+0000","last_unstale":"2024-04-16T16:15:16.417574+0000","last_undegraded":"2024-04-16T16:15:16.417574+0000","last_fullsized":"2024-04-16T16:15:16.417574+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:51.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.506434+0000","last_change":"2024-04-16T16:15:15.445285+0000","last_active":"2024-04-16T16:15:16.506434+0000","last_peered":"2024-04-16T16:15:16.506434+0000","last_clean":"2024-04-16T16:15:16.506434+0000","last_became_active":"2024-04-16T16:15:15.444969+0000","last_became_peered":"2024-04-16T16:15:15.444969+0000","last_unstale":"2024-04-16T16:15:16.506434+0000","last_undegraded":"2024-04-16T16:15:16.506434+0000","last_fullsized":"2024-04-16T16:15:16.506434+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:38:30.263403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800018+0000","last_change":"2024-04-16T16:15:15.445371+0000","last_active":"2024-04-16T16:15:16.800018+0000","last_peered":"2024-04-16T16:15:16.800018+0000","last_clean":"2024-04-16T16:15:16.800018+0000","last_became_active":"2024-04-16T16:15:15.445063+0000","last_became_peered":"2024-04-16T16:15:15.445063+0000","last_unstale":"2024-04-16T16:15:16.800018+0000","last_undegraded":"2024-04-16T16:15:16.800018+0000","last_fullsized":"2024-04-16T16:15:16.800018+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:57.624924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803133+0000","last_change":"2024-04-16T16:15:15.434979+0000","last_active":"2024-04-16T16:15:16.803133+0000","last_peered":"2024-04-16T16:15:16.803133+0000","last_clean":"2024-04-16T16:15:16.803133+0000","last_became_active":"2024-04-16T16:15:15.434028+0000","last_became_peered":"2024-04-16T16:15:15.434028+0000","last_unstale":"2024-04-16T16:15:16.803133+0000","last_undegraded":"2024-04-16T16:15:16.803133+0000","last_fullsized":"2024-04-16T16:15:16.803133+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:57:00.347584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.410627+0000","last_change":"2024-04-16T16:15:15.448903+0000","last_active":"2024-04-16T16:15:16.410627+0000","last_peered":"2024-04-16T16:15:16.410627+0000","last_clean":"2024-04-16T16:15:16.410627+0000","last_became_active":"2024-04-16T16:15:15.448548+0000","last_became_peered":"2024-04-16T16:15:15.448548+0000","last_unstale":"2024-04-16T16:15:16.410627+0000","last_undegraded":"2024-04-16T16:15:16.410627+0000","last_fullsized":"2024-04-16T16:15:16.410627+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:49.734767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800296+0000","last_change":"2024-04-16T16:15:15.444805+0000","last_active":"2024-04-16T16:15:16.800296+0000","last_peered":"2024-04-16T16:15:16.800296+0000","last_clean":"2024-04-16T16:15:16.800296+0000","last_became_active":"2024-04-16T16:15:15.442030+0000","last_became_peered":"2024-04-16T16:15:15.442030+0000","last_unstale":"2024-04-16T16:15:16.800296+0000","last_undegraded":"2024-04-16T16:15:16.800296+0000","last_fullsized":"2024-04-16T16:15:16.800296+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:33:41.701252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"41'4","reported_seq":18,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804128+0000","last_change":"2024-04-16T16:15:14.426636+0000","last_active":"2024-04-16T16:15:16.804128+0000","last_peered":"2024-04-16T16:15:16.804128+0000","last_clean":"2024-04-16T16:15:16.804128+0000","last_became_active":"2024-04-16T16:15:14.426319+0000","last_became_peered":"2024-04-16T16:15:14.426319+0000","last_unstale":"2024-04-16T16:15:16.804128+0000","last_undegraded":"2024-04-16T16:15:16.804128+0000","last_fullsized":"2024-04-16T16:15:16.804128+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:55.060644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.507344+0000","last_change":"2024-04-16T16:15:15.449524+0000","last_active":"2024-04-16T16:15:16.507344+0000","last_peered":"2024-04-16T16:15:16.507344+0000","last_clean":"2024-04-16T16:15:16.507344+0000","last_became_active":"2024-04-16T16:15:15.448988+0000","last_became_peered":"2024-04-16T16:15:15.448988+0000","last_unstale":"2024-04-16T16:15:16.507344+0000","last_undegraded":"2024-04-16T16:15:16.507344+0000","last_fullsized":"2024-04-16T16:15:16.507344+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:57.624924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"41'1","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800795+0000","last_change":"2024-04-16T16:15:14.430354+0000","last_active":"2024-04-16T16:15:16.800795+0000","last_peered":"2024-04-16T16:15:16.800795+0000","last_clean":"2024-04-16T16:15:16.800795+0000","last_became_active":"2024-04-16T16:15:14.429965+0000","last_became_peered":"2024-04-16T16:15:14.429965+0000","last_unstale":"2024-04-16T16:15:16.800795+0000","last_undegraded":"2024-04-16T16:15:16.800795+0000","last_fullsized":"2024-04-16T16:15:16.800795+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:28:19.722523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417921+0000","last_change":"2024-04-16T16:15:15.432880+0000","last_active":"2024-04-16T16:15:16.417921+0000","last_peered":"2024-04-16T16:15:16.417921+0000","last_clean":"2024-04-16T16:15:16.417921+0000","last_became_active":"2024-04-16T16:15:15.432458+0000","last_became_peered":"2024-04-16T16:15:15.432458+0000","last_unstale":"2024-04-16T16:15:16.417921+0000","last_undegraded":"2024-04-16T16:15:16.417921+0000","last_fullsized":"2024-04-16T16:15:16.417921+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:47.467138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"41'2","reported_seq":16,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.799696+0000","last_change":"2024-04-16T16:15:14.420969+0000","last_active":"2024-04-16T16:15:16.799696+0000","last_peered":"2024-04-16T16:15:16.799696+0000","last_clean":"2024-04-16T16:15:16.799696+0000","last_became_active":"2024-04-16T16:15:14.420657+0000","last_became_peered":"2024-04-16T16:15:14.420657+0000","last_unstale":"2024-04-16T16:15:16.799696+0000","last_undegraded":"2024-04-16T16:15:16.799696+0000","last_fullsized":"2024-04-16T16:15:16.799696+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:43.299902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.799557+0000","last_change":"2024-04-16T16:15:15.455406+0000","last_active":"2024-04-16T16:15:16.799557+0000","last_peered":"2024-04-16T16:15:16.799557+0000","last_clean":"2024-04-16T16:15:16.799557+0000","last_became_active":"2024-04-16T16:15:15.455065+0000","last_became_peered":"2024-04-16T16:15:15.455065+0000","last_unstale":"2024-04-16T16:15:16.799557+0000","last_undegraded":"2024-04-16T16:15:16.799557+0000","last_fullsized":"2024-04-16T16:15:16.799557+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:48:05.645437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"41'2","reported_seq":16,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.415506+0000","last_change":"2024-04-16T16:15:14.426165+0000","last_active":"2024-04-16T16:15:16.415506+0000","last_peered":"2024-04-16T16:15:16.415506+0000","last_clean":"2024-04-16T16:15:16.415506+0000","last_became_active":"2024-04-16T16:15:14.424933+0000","last_became_peered":"2024-04-16T16:15:14.424933+0000","last_unstale":"2024-04-16T16:15:16.415506+0000","last_undegraded":"2024-04-16T16:15:16.415506+0000","last_fullsized":"2024-04-16T16:15:16.415506+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:59.375922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800022+0000","last_change":"2024-04-16T16:15:15.451450+0000","last_active":"2024-04-16T16:15:16.800022+0000","last_peered":"2024-04-16T16:15:16.800022+0000","last_clean":"2024-04-16T16:15:16.800022+0000","last_became_active":"2024-04-16T16:15:15.451103+0000","last_became_peered":"2024-04-16T16:15:15.451103+0000","last_unstale":"2024-04-16T16:15:16.800022+0000","last_undegraded":"2024-04-16T16:15:16.800022+0000","last_fullsized":"2024-04-16T16:15:16.800022+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:49.654384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"41'1","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803785+0000","last_change":"2024-04-16T16:15:14.419476+0000","last_active":"2024-04-16T16:15:16.803785+0000","last_peered":"2024-04-16T16:15:16.803785+0000","last_clean":"2024-04-16T16:15:16.803785+0000","last_became_active":"2024-04-16T16:15:14.419185+0000","last_became_peered":"2024-04-16T16:15:14.419185+0000","last_unstale":"2024-04-16T16:15:16.803785+0000","last_undegraded":"2024-04-16T16:15:16.803785+0000","last_fullsized":"2024-04-16T16:15:16.803785+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:26.508529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801967+0000","last_change":"2024-04-16T16:15:15.460542+0000","last_active":"2024-04-16T16:15:16.801967+0000","last_peered":"2024-04-16T16:15:16.801967+0000","last_clean":"2024-04-16T16:15:16.801967+0000","last_became_active":"2024-04-16T16:15:15.459809+0000","last_became_peered":"2024-04-16T16:15:15.459809+0000","last_unstale":"2024-04-16T16:15:16.801967+0000","last_undegraded":"2024-04-16T16:15:16.801967+0000","last_fullsized":"2024-04-16T16:15:16.801967+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:47.467138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"41'1","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800347+0000","last_change":"2024-04-16T16:15:14.429151+0000","last_active":"2024-04-16T16:15:16.800347+0000","last_peered":"2024-04-16T16:15:16.800347+0000","last_clean":"2024-04-16T16:15:16.800347+0000","last_became_active":"2024-04-16T16:15:14.428841+0000","last_became_peered":"2024-04-16T16:15:14.428841+0000","last_unstale":"2024-04-16T16:15:16.800347+0000","last_undegraded":"2024-04-16T16:15:16.800347+0000","last_fullsized":"2024-04-16T16:15:16.800347+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:16.172813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800986+0000","last_change":"2024-04-16T16:15:15.461073+0000","last_active":"2024-04-16T16:15:16.800986+0000","last_peered":"2024-04-16T16:15:16.800986+0000","last_clean":"2024-04-16T16:15:16.800986+0000","last_became_active":"2024-04-16T16:15:15.459255+0000","last_became_peered":"2024-04-16T16:15:15.459255+0000","last_unstale":"2024-04-16T16:15:16.800986+0000","last_undegraded":"2024-04-16T16:15:16.800986+0000","last_fullsized":"2024-04-16T16:15:16.800986+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:27:00.456168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417002+0000","last_change":"2024-04-16T16:15:14.418012+0000","last_active":"2024-04-16T16:15:16.417002+0000","last_peered":"2024-04-16T16:15:16.417002+0000","last_clean":"2024-04-16T16:15:16.417002+0000","last_became_active":"2024-04-16T16:15:14.417689+0000","last_became_peered":"2024-04-16T16:15:14.417689+0000","last_unstale":"2024-04-16T16:15:16.417002+0000","last_undegraded":"2024-04-16T16:15:16.417002+0000","last_fullsized":"2024-04-16T16:15:16.417002+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:33:40.687566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.506201+0000","last_change":"2024-04-16T16:15:15.438978+0000","last_active":"2024-04-16T16:15:16.506201+0000","last_peered":"2024-04-16T16:15:16.506201+0000","last_clean":"2024-04-16T16:15:16.506201+0000","last_became_active":"2024-04-16T16:15:15.438597+0000","last_became_peered":"2024-04-16T16:15:15.438597+0000","last_unstale":"2024-04-16T16:15:16.506201+0000","last_undegraded":"2024-04-16T16:15:16.506201+0000","last_fullsized":"2024-04-16T16:15:16.506201+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:19:06.087718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801256+0000","last_change":"2024-04-16T16:15:14.411837+0000","last_active":"2024-04-16T16:15:16.801256+0000","last_peered":"2024-04-16T16:15:16.801256+0000","last_clean":"2024-04-16T16:15:16.801256+0000","last_became_active":"2024-04-16T16:15:14.411512+0000","last_became_peered":"2024-04-16T16:15:14.411512+0000","last_unstale":"2024-04-16T16:15:16.801256+0000","last_undegraded":"2024-04-16T16:15:16.801256+0000","last_fullsized":"2024-04-16T16:15:16.801256+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:28:19.722523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800517+0000","last_change":"2024-04-16T16:15:15.451420+0000","last_active":"2024-04-16T16:15:16.800517+0000","last_peered":"2024-04-16T16:15:16.800517+0000","last_clean":"2024-04-16T16:15:16.800517+0000","last_became_active":"2024-04-16T16:15:15.450543+0000","last_became_peered":"2024-04-16T16:15:15.450543+0000","last_unstale":"2024-04-16T16:15:16.800517+0000","last_undegraded":"2024-04-16T16:15:16.800517+0000","last_fullsized":"2024-04-16T16:15:16.800517+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:27:00.456168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801092+0000","last_change":"2024-04-16T16:15:14.422188+0000","last_active":"2024-04-16T16:15:16.801092+0000","last_peered":"2024-04-16T16:15:16.801092+0000","last_clean":"2024-04-16T16:15:16.801092+0000","last_became_active":"2024-04-16T16:15:14.421842+0000","last_became_peered":"2024-04-16T16:15:14.421842+0000","last_unstale":"2024-04-16T16:15:16.801092+0000","last_undegraded":"2024-04-16T16:15:16.801092+0000","last_fullsized":"2024-04-16T16:15:16.801092+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:43.299902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.418792+0000","last_change":"2024-04-16T16:15:15.436333+0000","last_active":"2024-04-16T16:15:16.418792+0000","last_peered":"2024-04-16T16:15:16.418792+0000","last_clean":"2024-04-16T16:15:16.418792+0000","last_became_active":"2024-04-16T16:15:15.435961+0000","last_became_peered":"2024-04-16T16:15:15.435961+0000","last_unstale":"2024-04-16T16:15:16.418792+0000","last_undegraded":"2024-04-16T16:15:16.418792+0000","last_fullsized":"2024-04-16T16:15:16.418792+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:44.974637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.802286+0000","last_change":"2024-04-16T16:15:14.427378+0000","last_active":"2024-04-16T16:15:16.802286+0000","last_peered":"2024-04-16T16:15:16.802286+0000","last_clean":"2024-04-16T16:15:16.802286+0000","last_became_active":"2024-04-16T16:15:14.427036+0000","last_became_peered":"2024-04-16T16:15:14.427036+0000","last_unstale":"2024-04-16T16:15:16.802286+0000","last_undegraded":"2024-04-16T16:15:16.802286+0000","last_fullsized":"2024-04-16T16:15:16.802286+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:16.172813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804455+0000","last_change":"2024-04-16T16:15:15.453206+0000","last_active":"2024-04-16T16:15:16.804455+0000","last_peered":"2024-04-16T16:15:16.804455+0000","last_clean":"2024-04-16T16:15:16.804455+0000","last_became_active":"2024-04-16T16:15:15.453002+0000","last_became_peered":"2024-04-16T16:15:15.453002+0000","last_unstale":"2024-04-16T16:15:16.804455+0000","last_undegraded":"2024-04-16T16:15:16.804455+0000","last_fullsized":"2024-04-16T16:15:16.804455+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:37.628550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"41'3","reported_seq":17,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417985+0000","last_change":"2024-04-16T16:15:14.427833+0000","last_active":"2024-04-16T16:15:16.417985+0000","last_peered":"2024-04-16T16:15:16.417985+0000","last_clean":"2024-04-16T16:15:16.417985+0000","last_became_active":"2024-04-16T16:15:14.427492+0000","last_became_peered":"2024-04-16T16:15:14.427492+0000","last_unstale":"2024-04-16T16:15:16.417985+0000","last_undegraded":"2024-04-16T16:15:16.417985+0000","last_fullsized":"2024-04-16T16:15:16.417985+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:48.640698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417688+0000","last_change":"2024-04-16T16:15:15.435454+0000","last_active":"2024-04-16T16:15:16.417688+0000","last_peered":"2024-04-16T16:15:16.417688+0000","last_clean":"2024-04-16T16:15:16.417688+0000","last_became_active":"2024-04-16T16:15:15.435056+0000","last_became_peered":"2024-04-16T16:15:15.435056+0000","last_unstale":"2024-04-16T16:15:16.417688+0000","last_undegraded":"2024-04-16T16:15:16.417688+0000","last_fullsized":"2024-04-16T16:15:16.417688+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:34.706031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"41'6","reported_seq":20,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.410761+0000","last_change":"2024-04-16T16:15:14.419221+0000","last_active":"2024-04-16T16:15:16.410761+0000","last_peered":"2024-04-16T16:15:16.410761+0000","last_clean":"2024-04-16T16:15:16.410761+0000","last_became_active":"2024-04-16T16:15:14.418573+0000","last_became_peered":"2024-04-16T16:15:14.418573+0000","last_unstale":"2024-04-16T16:15:16.410761+0000","last_undegraded":"2024-04-16T16:15:16.410761+0000","last_fullsized":"2024-04-16T16:15:16.410761+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:08.107514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.506564+0000","last_change":"2024-04-16T16:15:15.447435+0000","last_active":"2024-04-16T16:15:16.506564+0000","last_peered":"2024-04-16T16:15:16.506564+0000","last_clean":"2024-04-16T16:15:16.506564+0000","last_became_active":"2024-04-16T16:15:15.447094+0000","last_became_peered":"2024-04-16T16:15:15.447094+0000","last_unstale":"2024-04-16T16:15:16.506564+0000","last_undegraded":"2024-04-16T16:15:16.506564+0000","last_fullsized":"2024-04-16T16:15:16.506564+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:16:44.917291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"41'2","reported_seq":16,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804649+0000","last_change":"2024-04-16T16:15:14.408195+0000","last_active":"2024-04-16T16:15:16.804649+0000","last_peered":"2024-04-16T16:15:16.804649+0000","last_clean":"2024-04-16T16:15:16.804649+0000","last_became_active":"2024-04-16T16:15:14.407584+0000","last_became_peered":"2024-04-16T16:15:14.407584+0000","last_unstale":"2024-04-16T16:15:16.804649+0000","last_undegraded":"2024-04-16T16:15:16.804649+0000","last_fullsized":"2024-04-16T16:15:16.804649+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:39:56.984084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.802042+0000","last_change":"2024-04-16T16:15:15.462282+0000","last_active":"2024-04-16T16:15:16.802042+0000","last_peered":"2024-04-16T16:15:16.802042+0000","last_clean":"2024-04-16T16:15:16.802042+0000","last_became_active":"2024-04-16T16:15:15.461980+0000","last_became_peered":"2024-04-16T16:15:15.461980+0000","last_unstale":"2024-04-16T16:15:16.802042+0000","last_undegraded":"2024-04-16T16:15:16.802042+0000","last_fullsized":"2024-04-16T16:15:16.802042+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:11.662435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.416025+0000","last_change":"2024-04-16T16:15:14.421692+0000","last_active":"2024-04-16T16:15:16.416025+0000","last_peered":"2024-04-16T16:15:16.416025+0000","last_clean":"2024-04-16T16:15:16.416025+0000","last_became_active":"2024-04-16T16:15:14.421361+0000","last_became_peered":"2024-04-16T16:15:14.421361+0000","last_unstale":"2024-04-16T16:15:16.416025+0000","last_undegraded":"2024-04-16T16:15:16.416025+0000","last_fullsized":"2024-04-16T16:15:16.416025+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:56.611238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803068+0000","last_change":"2024-04-16T16:15:15.455736+0000","last_active":"2024-04-16T16:15:16.803068+0000","last_peered":"2024-04-16T16:15:16.803068+0000","last_clean":"2024-04-16T16:15:16.803068+0000","last_became_active":"2024-04-16T16:15:15.455415+0000","last_became_peered":"2024-04-16T16:15:15.455415+0000","last_unstale":"2024-04-16T16:15:16.803068+0000","last_undegraded":"2024-04-16T16:15:16.803068+0000","last_fullsized":"2024-04-16T16:15:16.803068+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:29:30.311490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803257+0000","last_change":"2024-04-16T16:15:14.410824+0000","last_active":"2024-04-16T16:15:16.803257+0000","last_peered":"2024-04-16T16:15:16.803257+0000","last_clean":"2024-04-16T16:15:16.803257+0000","last_became_active":"2024-04-16T16:15:14.409874+0000","last_became_peered":"2024-04-16T16:15:14.409874+0000","last_unstale":"2024-04-16T16:15:16.803257+0000","last_undegraded":"2024-04-16T16:15:16.803257+0000","last_fullsized":"2024-04-16T16:15:16.803257+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:26:59.442481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.416927+0000","last_change":"2024-04-16T16:15:15.446100+0000","last_active":"2024-04-16T16:15:16.416927+0000","last_peered":"2024-04-16T16:15:16.416927+0000","last_clean":"2024-04-16T16:15:16.416927+0000","last_became_active":"2024-04-16T16:15:15.444480+0000","last_became_peered":"2024-04-16T16:15:15.444480+0000","last_unstale":"2024-04-16T16:15:16.416927+0000","last_undegraded":"2024-04-16T16:15:16.416927+0000","last_fullsized":"2024-04-16T16:15:16.416927+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:47.467138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"41'8","reported_seq":22,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801528+0000","last_change":"2024-04-16T16:15:14.421997+0000","last_active":"2024-04-16T16:15:16.801528+0000","last_peered":"2024-04-16T16:15:16.801528+0000","last_clean":"2024-04-16T16:15:16.801528+0000","last_became_active":"2024-04-16T16:15:14.421747+0000","last_became_peered":"2024-04-16T16:15:14.421747+0000","last_unstale":"2024-04-16T16:15:16.801528+0000","last_undegraded":"2024-04-16T16:15:16.801528+0000","last_fullsized":"2024-04-16T16:15:16.801528+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:56:30.781319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.410930+0000","last_change":"2024-04-16T16:15:15.444883+0000","last_active":"2024-04-16T16:15:16.410930+0000","last_peered":"2024-04-16T16:15:16.410930+0000","last_clean":"2024-04-16T16:15:16.410930+0000","last_became_active":"2024-04-16T16:15:15.438454+0000","last_became_peered":"2024-04-16T16:15:15.438454+0000","last_unstale":"2024-04-16T16:15:16.410930+0000","last_undegraded":"2024-04-16T16:15:16.410930+0000","last_fullsized":"2024-04-16T16:15:16.410930+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:25:00.737475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"41'1","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.410689+0000","last_change":"2024-04-16T16:15:14.406535+0000","last_active":"2024-04-16T16:15:16.410689+0000","last_peered":"2024-04-16T16:15:16.410689+0000","last_clean":"2024-04-16T16:15:16.410689+0000","last_became_active":"2024-04-16T16:15:14.405872+0000","last_became_peered":"2024-04-16T16:15:14.405872+0000","last_unstale":"2024-04-16T16:15:16.410689+0000","last_undegraded":"2024-04-16T16:15:16.410689+0000","last_fullsized":"2024-04-16T16:15:16.410689+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:12:11.062203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.805181+0000","last_change":"2024-04-16T16:15:15.442790+0000","last_active":"2024-04-16T16:15:16.805181+0000","last_peered":"2024-04-16T16:15:16.805181+0000","last_clean":"2024-04-16T16:15:16.805181+0000","last_became_active":"2024-04-16T16:15:15.439265+0000","last_became_peered":"2024-04-16T16:15:15.439265+0000","last_unstale":"2024-04-16T16:15:16.805181+0000","last_undegraded":"2024-04-16T16:15:16.805181+0000","last_fullsized":"2024-04-16T16:15:16.805181+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:40:41.651741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"41'2","reported_seq":16,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800708+0000","last_change":"2024-04-16T16:15:14.420778+0000","last_active":"2024-04-16T16:15:16.800708+0000","last_peered":"2024-04-16T16:15:16.800708+0000","last_clean":"2024-04-16T16:15:16.800708+0000","last_became_active":"2024-04-16T16:15:14.420315+0000","last_became_peered":"2024-04-16T16:15:14.420315+0000","last_unstale":"2024-04-16T16:15:16.800708+0000","last_undegraded":"2024-04-16T16:15:16.800708+0000","last_fullsized":"2024-04-16T16:15:16.800708+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:43:43.352649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.506779+0000","last_change":"2024-04-16T16:15:15.446817+0000","last_active":"2024-04-16T16:15:16.506779+0000","last_peered":"2024-04-16T16:15:16.506779+0000","last_clean":"2024-04-16T16:15:16.506779+0000","last_became_active":"2024-04-16T16:15:15.444908+0000","last_became_peered":"2024-04-16T16:15:15.444908+0000","last_unstale":"2024-04-16T16:15:16.506779+0000","last_undegraded":"2024-04-16T16:15:16.506779+0000","last_fullsized":"2024-04-16T16:15:16.506779+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:46.819298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"41'1","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417390+0000","last_change":"2024-04-16T16:15:14.417893+0000","last_active":"2024-04-16T16:15:16.417390+0000","last_peered":"2024-04-16T16:15:16.417390+0000","last_clean":"2024-04-16T16:15:16.417390+0000","last_became_active":"2024-04-16T16:15:14.416107+0000","last_became_peered":"2024-04-16T16:15:14.416107+0000","last_unstale":"2024-04-16T16:15:16.417390+0000","last_undegraded":"2024-04-16T16:15:16.417390+0000","last_fullsized":"2024-04-16T16:15:16.417390+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:59.375922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.507204+0000","last_change":"2024-04-16T16:15:15.451372+0000","last_active":"2024-04-16T16:15:16.507204+0000","last_peered":"2024-04-16T16:15:16.507204+0000","last_clean":"2024-04-16T16:15:16.507204+0000","last_became_active":"2024-04-16T16:15:15.450983+0000","last_became_peered":"2024-04-16T16:15:15.450983+0000","last_unstale":"2024-04-16T16:15:16.507204+0000","last_undegraded":"2024-04-16T16:15:16.507204+0000","last_fullsized":"2024-04-16T16:15:16.507204+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:05:00.389608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804856+0000","last_change":"2024-04-16T16:15:14.423513+0000","last_active":"2024-04-16T16:15:16.804856+0000","last_peered":"2024-04-16T16:15:16.804856+0000","last_clean":"2024-04-16T16:15:16.804856+0000","last_became_active":"2024-04-16T16:15:14.423209+0000","last_became_peered":"2024-04-16T16:15:14.423209+0000","last_unstale":"2024-04-16T16:15:16.804856+0000","last_undegraded":"2024-04-16T16:15:16.804856+0000","last_fullsized":"2024-04-16T16:15:16.804856+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:41.421090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803514+0000","last_change":"2024-04-16T16:15:15.443802+0000","last_active":"2024-04-16T16:15:16.803514+0000","last_peered":"2024-04-16T16:15:16.803514+0000","last_clean":"2024-04-16T16:15:16.803514+0000","last_became_active":"2024-04-16T16:15:15.443493+0000","last_became_peered":"2024-04-16T16:15:15.443493+0000","last_unstale":"2024-04-16T16:15:16.803514+0000","last_undegraded":"2024-04-16T16:15:16.803514+0000","last_fullsized":"2024-04-16T16:15:16.803514+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:31:00.851752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"41'2","reported_seq":16,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804551+0000","last_change":"2024-04-16T16:15:14.419251+0000","last_active":"2024-04-16T16:15:16.804551+0000","last_peered":"2024-04-16T16:15:16.804551+0000","last_clean":"2024-04-16T16:15:16.804551+0000","last_became_active":"2024-04-16T16:15:14.418965+0000","last_became_peered":"2024-04-16T16:15:14.418965+0000","last_unstale":"2024-04-16T16:15:16.804551+0000","last_undegraded":"2024-04-16T16:15:16.804551+0000","last_fullsized":"2024-04-16T16:15:16.804551+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:11.262725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.799964+0000","last_change":"2024-04-16T16:15:15.450303+0000","last_active":"2024-04-16T16:15:16.799964+0000","last_peered":"2024-04-16T16:15:16.799964+0000","last_clean":"2024-04-16T16:15:16.799964+0000","last_became_active":"2024-04-16T16:15:15.449983+0000","last_became_peered":"2024-04-16T16:15:15.449983+0000","last_unstale":"2024-04-16T16:15:16.799964+0000","last_undegraded":"2024-04-16T16:15:16.799964+0000","last_fullsized":"2024-04-16T16:15:16.799964+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:46.819298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417060+0000","last_change":"2024-04-16T16:15:14.423634+0000","last_active":"2024-04-16T16:15:16.417060+0000","last_peered":"2024-04-16T16:15:16.417060+0000","last_clean":"2024-04-16T16:15:16.417060+0000","last_became_active":"2024-04-16T16:15:14.422076+0000","last_became_peered":"2024-04-16T16:15:14.422076+0000","last_unstale":"2024-04-16T16:15:16.417060+0000","last_undegraded":"2024-04-16T16:15:16.417060+0000","last_fullsized":"2024-04-16T16:15:16.417060+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:43.504451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.411072+0000","last_change":"2024-04-16T16:15:15.440012+0000","last_active":"2024-04-16T16:15:16.411072+0000","last_peered":"2024-04-16T16:15:16.411072+0000","last_clean":"2024-04-16T16:15:16.411072+0000","last_became_active":"2024-04-16T16:15:15.439703+0000","last_became_peered":"2024-04-16T16:15:15.439703+0000","last_unstale":"2024-04-16T16:15:16.411072+0000","last_undegraded":"2024-04-16T16:15:16.411072+0000","last_fullsized":"2024-04-16T16:15:16.411072+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:10:07.586480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801336+0000","last_change":"2024-04-16T16:15:14.426381+0000","last_active":"2024-04-16T16:15:16.801336+0000","last_peered":"2024-04-16T16:15:16.801336+0000","last_clean":"2024-04-16T16:15:16.801336+0000","last_became_active":"2024-04-16T16:15:14.425924+0000","last_became_peered":"2024-04-16T16:15:14.425924+0000","last_unstale":"2024-04-16T16:15:16.801336+0000","last_undegraded":"2024-04-16T16:15:16.801336+0000","last_fullsized":"2024-04-16T16:15:16.801336+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:07:59.840158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800909+0000","last_change":"2024-04-16T16:15:15.463514+0000","last_active":"2024-04-16T16:15:16.800909+0000","last_peered":"2024-04-16T16:15:16.800909+0000","last_clean":"2024-04-16T16:15:16.800909+0000","last_became_active":"2024-04-16T16:15:15.461956+0000","last_became_peered":"2024-04-16T16:15:15.461956+0000","last_unstale":"2024-04-16T16:15:16.800909+0000","last_undegraded":"2024-04-16T16:15:16.800909+0000","last_fullsized":"2024-04-16T16:15:16.800909+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:34.312744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.415603+0000","last_change":"2024-04-16T16:15:14.419487+0000","last_active":"2024-04-16T16:15:16.415603+0000","last_peered":"2024-04-16T16:15:16.415603+0000","last_clean":"2024-04-16T16:15:16.415603+0000","last_became_active":"2024-04-16T16:15:14.419154+0000","last_became_peered":"2024-04-16T16:15:14.419154+0000","last_unstale":"2024-04-16T16:15:16.415603+0000","last_undegraded":"2024-04-16T16:15:16.415603+0000","last_fullsized":"2024-04-16T16:15:16.415603+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:01:19.702926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.507532+0000","last_change":"2024-04-16T16:15:15.463217+0000","last_active":"2024-04-16T16:15:16.507532+0000","last_peered":"2024-04-16T16:15:16.507532+0000","last_clean":"2024-04-16T16:15:16.507532+0000","last_became_active":"2024-04-16T16:15:15.462748+0000","last_became_peered":"2024-04-16T16:15:15.462748+0000","last_unstale":"2024-04-16T16:15:16.507532+0000","last_undegraded":"2024-04-16T16:15:16.507532+0000","last_fullsized":"2024-04-16T16:15:16.507532+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:03.163739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804294+0000","last_change":"2024-04-16T16:15:14.424803+0000","last_active":"2024-04-16T16:15:16.804294+0000","last_peered":"2024-04-16T16:15:16.804294+0000","last_clean":"2024-04-16T16:15:16.804294+0000","last_became_active":"2024-04-16T16:15:14.424493+0000","last_became_peered":"2024-04-16T16:15:14.424493+0000","last_unstale":"2024-04-16T16:15:16.804294+0000","last_undegraded":"2024-04-16T16:15:16.804294+0000","last_fullsized":"2024-04-16T16:15:16.804294+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:33.299058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801352+0000","last_change":"2024-04-16T16:15:15.451816+0000","last_active":"2024-04-16T16:15:16.801352+0000","last_peered":"2024-04-16T16:15:16.801352+0000","last_clean":"2024-04-16T16:15:16.801352+0000","last_became_active":"2024-04-16T16:15:15.450017+0000","last_became_peered":"2024-04-16T16:15:15.450017+0000","last_unstale":"2024-04-16T16:15:16.801352+0000","last_undegraded":"2024-04-16T16:15:16.801352+0000","last_fullsized":"2024-04-16T16:15:16.801352+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:42.434776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.418079+0000","last_change":"2024-04-16T16:15:14.426984+0000","last_active":"2024-04-16T16:15:16.418079+0000","last_peered":"2024-04-16T16:15:16.418079+0000","last_clean":"2024-04-16T16:15:16.418079+0000","last_became_active":"2024-04-16T16:15:14.426689+0000","last_became_peered":"2024-04-16T16:15:14.426689+0000","last_unstale":"2024-04-16T16:15:16.418079+0000","last_undegraded":"2024-04-16T16:15:16.418079+0000","last_fullsized":"2024-04-16T16:15:16.418079+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:02.150053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.417841+0000","last_change":"2024-04-16T16:15:15.449317+0000","last_active":"2024-04-16T16:15:16.417841+0000","last_peered":"2024-04-16T16:15:16.417841+0000","last_clean":"2024-04-16T16:15:16.417841+0000","last_became_active":"2024-04-16T16:15:15.448646+0000","last_became_peered":"2024-04-16T16:15:15.448646+0000","last_unstale":"2024-04-16T16:15:16.417841+0000","last_undegraded":"2024-04-16T16:15:16.417841+0000","last_fullsized":"2024-04-16T16:15:16.417841+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:11.662435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.799570+0000","last_change":"2024-04-16T16:15:14.424278+0000","last_active":"2024-04-16T16:15:16.799570+0000","last_peered":"2024-04-16T16:15:16.799570+0000","last_clean":"2024-04-16T16:15:16.799570+0000","last_became_active":"2024-04-16T16:15:14.423174+0000","last_became_peered":"2024-04-16T16:15:14.423174+0000","last_unstale":"2024-04-16T16:15:16.799570+0000","last_undegraded":"2024-04-16T16:15:16.799570+0000","last_fullsized":"2024-04-16T16:15:16.799570+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:19:05.074032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.804235+0000","last_change":"2024-04-16T16:15:15.443473+0000","last_active":"2024-04-16T16:15:16.804235+0000","last_peered":"2024-04-16T16:15:16.804235+0000","last_clean":"2024-04-16T16:15:16.804235+0000","last_became_active":"2024-04-16T16:15:15.439092+0000","last_became_peered":"2024-04-16T16:15:15.439092+0000","last_unstale":"2024-04-16T16:15:16.804235+0000","last_undegraded":"2024-04-16T16:15:16.804235+0000","last_fullsized":"2024-04-16T16:15:16.804235+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:49:28.175995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.416428+0000","last_change":"2024-04-16T16:15:14.427316+0000","last_active":"2024-04-16T16:15:16.416428+0000","last_peered":"2024-04-16T16:15:16.416428+0000","last_clean":"2024-04-16T16:15:16.416428+0000","last_became_active":"2024-04-16T16:15:14.426635+0000","last_became_peered":"2024-04-16T16:15:14.426635+0000","last_unstale":"2024-04-16T16:15:16.416428+0000","last_undegraded":"2024-04-16T16:15:16.416428+0000","last_fullsized":"2024-04-16T16:15:16.416428+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:43.504451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.803680+0000","last_change":"2024-04-16T16:15:15.440497+0000","last_active":"2024-04-16T16:15:16.803680+0000","last_peered":"2024-04-16T16:15:16.803680+0000","last_clean":"2024-04-16T16:15:16.803680+0000","last_became_active":"2024-04-16T16:15:15.440205+0000","last_became_peered":"2024-04-16T16:15:15.440205+0000","last_unstale":"2024-04-16T16:15:16.803680+0000","last_undegraded":"2024-04-16T16:15:16.803680+0000","last_fullsized":"2024-04-16T16:15:16.803680+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:29:51.615992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800249+0000","last_change":"2024-04-16T16:15:14.414769+0000","last_active":"2024-04-16T16:15:16.800249+0000","last_peered":"2024-04-16T16:15:16.800249+0000","last_clean":"2024-04-16T16:15:16.800249+0000","last_became_active":"2024-04-16T16:15:14.414402+0000","last_became_peered":"2024-04-16T16:15:14.414402+0000","last_unstale":"2024-04-16T16:15:16.800249+0000","last_undegraded":"2024-04-16T16:15:16.800249+0000","last_fullsized":"2024-04-16T16:15:16.800249+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:07:59.840158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.415822+0000","last_change":"2024-04-16T16:15:15.446666+0000","last_active":"2024-04-16T16:15:16.415822+0000","last_peered":"2024-04-16T16:15:16.415822+0000","last_clean":"2024-04-16T16:15:16.415822+0000","last_became_active":"2024-04-16T16:15:15.444672+0000","last_became_peered":"2024-04-16T16:15:15.444672+0000","last_unstale":"2024-04-16T16:15:16.415822+0000","last_undegraded":"2024-04-16T16:15:16.415822+0000","last_fullsized":"2024-04-16T16:15:16.415822+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:39:57.997770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.799321+0000","last_change":"2024-04-16T16:15:14.425948+0000","last_active":"2024-04-16T16:15:16.799321+0000","last_peered":"2024-04-16T16:15:16.799321+0000","last_clean":"2024-04-16T16:15:16.799321+0000","last_became_active":"2024-04-16T16:15:14.425692+0000","last_became_peered":"2024-04-16T16:15:14.425692+0000","last_unstale":"2024-04-16T16:15:16.799321+0000","last_undegraded":"2024-04-16T16:15:16.799321+0000","last_fullsized":"2024-04-16T16:15:16.799321+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:16.172813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.800003+0000","last_change":"2024-04-16T16:15:15.448491+0000","last_active":"2024-04-16T16:15:16.800003+0000","last_peered":"2024-04-16T16:15:16.800003+0000","last_clean":"2024-04-16T16:15:16.800003+0000","last_became_active":"2024-04-16T16:15:15.448176+0000","last_became_peered":"2024-04-16T16:15:15.448176+0000","last_unstale":"2024-04-16T16:15:16.800003+0000","last_undegraded":"2024-04-16T16:15:16.800003+0000","last_fullsized":"2024-04-16T16:15:16.800003+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:44.518137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.801815+0000","last_change":"2024-04-16T16:15:14.413186+0000","last_active":"2024-04-16T16:15:16.801815+0000","last_peered":"2024-04-16T16:15:16.801815+0000","last_clean":"2024-04-16T16:15:16.801815+0000","last_became_active":"2024-04-16T16:15:14.412710+0000","last_became_peered":"2024-04-16T16:15:14.412710+0000","last_unstale":"2024-04-16T16:15:16.801815+0000","last_undegraded":"2024-04-16T16:15:16.801815+0000","last_fullsized":"2024-04-16T16:15:16.801815+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:43:43.352649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.802050+0000","last_change":"2024-04-16T16:15:15.444726+0000","last_active":"2024-04-16T16:15:16.802050+0000","last_peered":"2024-04-16T16:15:16.802050+0000","last_clean":"2024-04-16T16:15:16.802050+0000","last_became_active":"2024-04-16T16:15:15.444386+0000","last_became_peered":"2024-04-16T16:15:15.444386+0000","last_unstale":"2024-04-16T16:15:16.802050+0000","last_undegraded":"2024-04-16T16:15:16.802050+0000","last_fullsized":"2024-04-16T16:15:16.802050+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:34.706031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"41'2","reported_seq":16,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.418458+0000","last_change":"2024-04-16T16:15:14.420968+0000","last_active":"2024-04-16T16:15:16.418458+0000","last_peered":"2024-04-16T16:15:16.418458+0000","last_clean":"2024-04-16T16:15:16.418458+0000","last_became_active":"2024-04-16T16:15:14.418308+0000","last_became_peered":"2024-04-16T16:15:14.418308+0000","last_unstale":"2024-04-16T16:15:16.418458+0000","last_undegraded":"2024-04-16T16:15:16.418458+0000","last_fullsized":"2024-04-16T16:15:16.418458+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:56.611238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"35'128","reported_seq":251,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-16T16:15:16.415584+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:15:16.415584+0000","last_peered":"2024-04-16T16:15:16.415584+0000","last_clean":"2024-04-16T16:15:16.415584+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:15:16.415584+0000","last_undegraded":"2024-04-16T16:15:16.415584+0000","last_fullsized":"2024-04-16T16:15:16.415584+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343752,"statfs":{"total":96636764160,"available":96608002048,"internally_reserved":0,"allocated":1089536,"data_stored":793072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":499712,"data_stored":203286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":495616,"data_stored":202684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27512,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344328,"statfs":{"total":96636764160,"available":96608591872,"internally_reserved":0,"allocated":503808,"data_stored":204226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":203252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":507904,"data_stored":204248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":483328,"data_stored":196055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32232,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339608,"statfs":{"total":96636764160,"available":96603758592,"internally_reserved":0,"allocated":1077248,"data_stored":786387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:15:20.464 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-16T16:15:20.464 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-16T16:15:20.464 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-16T16:15:20.464 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-16T16:15:20.465 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:15:20.465 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-16T16:15:20.465 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-16T16:15:20.465 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-16T16:15:20.466 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:15:20.466 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-16T16:15:20.466 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-16T16:15:20.466 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-16T16:15:20.466 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:15:20.467 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-16T16:15:20.467 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-16T16:15:20.467 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-16T16:15:20.467 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:15:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T16:15:20.468 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2024-04-16T16:15:20.469 INFO:tasks.ceph.mds.a:Stopped 2024-04-16T16:15:20.469 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-16T16:15:20.488 INFO:tasks.ceph.mds.d:Stopped 2024-04-16T16:15:20.489 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-16T16:15:20.492 INFO:tasks.ceph.mds.b:Stopped 2024-04-16T16:15:20.492 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-16T16:15:20.507 INFO:tasks.ceph.mds.a:Started 2024-04-16T16:15:20.508 DEBUG:teuthology.parallel:result is None 2024-04-16T16:15:20.523 INFO:tasks.ceph.mds.c:Stopped 2024-04-16T16:15:20.524 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-16T16:15:20.539 INFO:tasks.ceph.mds.d:Started 2024-04-16T16:15:20.540 DEBUG:teuthology.parallel:result is None 2024-04-16T16:15:20.546 INFO:tasks.ceph.mds.b:Started 2024-04-16T16:15:20.546 DEBUG:teuthology.parallel:result is None 2024-04-16T16:15:20.571 INFO:tasks.ceph.mds.c:Started 2024-04-16T16:15:20.571 DEBUG:teuthology.parallel:result is None 2024-04-16T16:15:20.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:15:20.681 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2024-04-16T16:15:20.689 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-04-16T16:15:20.690+0000 7fe7a4800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T16:15:20.705 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-16T16:15:20.708 INFO:tasks.ceph.mds.a.smithi045.stderr:2024-04-16T16:15:20.709+0000 7fd51fe00640 -1 mds.0.openfiles _load_finish got (2) No such file or directory 2024-04-16T16:15:20.712 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-16T16:15:20.737 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2024-04-16T16:15:20.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.775+0000 7fafc5400640 1 Processor -- start 2024-04-16T16:15:20.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.775+0000 7fafc5400640 1 -- start start 2024-04-16T16:15:20.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.776+0000 7fafc5400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fafc0072a90 0x7fafc01699a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:20.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.776+0000 7fafc5400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafc0147c20 con 0x7fafc00aed90 2024-04-16T16:15:20.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.776+0000 7fafc5400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafc0160210 con 0x7fafc0147840 2024-04-16T16:15:20.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.776+0000 7fafc5400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fafc00afe80 con 0x7fafc0072a90 2024-04-16T16:15:20.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.777+0000 7fafbde00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fafc00aed90 0x7fafc0072380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42078/0 (socket says 172.21.15.45:42078) 2024-04-16T16:15:20.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.777+0000 7fafbde00640 1 -- 172.21.15.45:0/2135426528 learned_addr learned my addr 172.21.15.45:0/2135426528 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:20.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.777+0000 7fafbf200640 1 --2- 172.21.15.45:0/2135426528 >> v2:172.21.15.103:3300/0 conn(0x7fafc0072a90 0x7fafc01699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:20.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.777+0000 7fafbd400640 1 -- 172.21.15.45:0/2135426528 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3015789709 0 0) 0x7fafc0147c20 con 0x7fafc00aed90 2024-04-16T16:15:20.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.777+0000 7fafbd400640 1 -- 172.21.15.45:0/2135426528 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafa4003520 con 0x7fafc00aed90 2024-04-16T16:15:20.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.777+0000 7fafbd400640 1 -- 172.21.15.45:0/2135426528 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3748614071 0 0) 0x7fafc0160210 con 0x7fafc0147840 2024-04-16T16:15:20.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.777+0000 7fafbd400640 1 -- 172.21.15.45:0/2135426528 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafc0147c20 con 0x7fafc0147840 2024-04-16T16:15:20.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.777+0000 7fafbf200640 1 -- 172.21.15.45:0/2135426528 >> v1:172.21.15.103:6789/0 conn(0x7fafc0147840 legacy=0x7fafc015fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:20.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.777+0000 7fafbf200640 1 -- 172.21.15.45:0/2135426528 >> v1:172.21.15.45:6789/0 conn(0x7fafc00aed90 legacy=0x7fafc0072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:20.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.777+0000 7fafbf200640 1 -- 172.21.15.45:0/2135426528 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc00af270 con 0x7fafc0072a90 2024-04-16T16:15:20.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.779+0000 7fafbf200640 1 --2- 172.21.15.45:0/2135426528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc0072a90 0x7fafc01699a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fafc00b0000 tx=0x7fafb802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84e092d1540db741 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:20.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.779+0000 7fafbd400640 1 -- 172.21.15.45:0/2135426528 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb8039070 con 0x7fafc0072a90 2024-04-16T16:15:20.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.779+0000 7fafbd400640 1 -- 172.21.15.45:0/2135426528 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafb802cce0 con 0x7fafc0072a90 2024-04-16T16:15:20.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.779+0000 7fafbd400640 1 -- 172.21.15.45:0/2135426528 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb8034640 con 0x7fafc0072a90 2024-04-16T16:15:20.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.780+0000 7fafc5400640 1 -- 172.21.15.45:0/2135426528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc0072a90 msgr2=0x7fafc01699a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:20.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.780+0000 7fafc5400640 1 --2- 172.21.15.45:0/2135426528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc0072a90 0x7fafc01699a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fafc00b0000 tx=0x7fafb802bf30 comp rx=0 tx=0).stop 2024-04-16T16:15:20.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.780+0000 7fafc5400640 1 -- 172.21.15.45:0/2135426528 shutdown_connections 2024-04-16T16:15:20.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.780+0000 7fafc5400640 1 --2- 172.21.15.45:0/2135426528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc0072a90 0x7fafc01699a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:20.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.780+0000 7fafc5400640 1 -- 172.21.15.45:0/2135426528 >> 172.21.15.45:0/2135426528 conn(0x7fafc00b2750 msgr2=0x7fafc00b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:20.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.780+0000 7fafc5400640 1 -- 172.21.15.45:0/2135426528 shutdown_connections 2024-04-16T16:15:20.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.780+0000 7fafc5400640 1 -- 172.21.15.45:0/2135426528 wait complete. 2024-04-16T16:15:20.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.782+0000 7fafc5400640 1 Processor -- start 2024-04-16T16:15:20.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.782+0000 7fafc5400640 1 -- start start 2024-04-16T16:15:20.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafc5400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fafc0072a90 0x7fafc01e8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:20.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafc5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc00aed90 0x7fafc01f7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:20.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafc5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc0147840 0x7fafc01f7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:20.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafc5400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fafc00afb10 con 0x7fafc00aed90 2024-04-16T16:15:20.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafc5400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fafc00af830 con 0x7fafc0147840 2024-04-16T16:15:20.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafc5400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fafc006f2c0 con 0x7fafc0072a90 2024-04-16T16:15:20.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafbde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc00aed90 0x7fafc01f7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafbde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc00aed90 0x7fafc01f7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57024/0 (socket says 172.21.15.45:57024) 2024-04-16T16:15:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafbde00640 1 -- 172.21.15.45:0/289452098 learned_addr learned my addr 172.21.15.45:0/289452098 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafbe800640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fafc0072a90 0x7fafc01e8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafbf200640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc0147840 0x7fafc01f7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafbde00640 1 -- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fafc0072a90 msgr2=0x7fafc01e8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafbde00640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fafc0072a90 0x7fafc01e8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafbde00640 1 -- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc0147840 msgr2=0x7fafc01f7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafbde00640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc0147840 0x7fafc01f7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafbde00640 1 -- 172.21.15.45:0/289452098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc01613b0 con 0x7fafc00aed90 2024-04-16T16:15:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafbf200640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc0147840 0x7fafc01f7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:20.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.783+0000 7fafbe800640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fafc0072a90 0x7fafc01e8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.784+0000 7fafbde00640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc00aed90 0x7fafc01f7910 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fafac00c4d0 tx=0x7fafac00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.784+0000 7fafb6a00640 1 -- 172.21.15.45:0/289452098 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafac017070 con 0x7fafc00aed90 2024-04-16T16:15:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.784+0000 7fafb6a00640 1 -- 172.21.15.45:0/289452098 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafac005ce0 con 0x7fafc00aed90 2024-04-16T16:15:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.784+0000 7fafb6a00640 1 -- 172.21.15.45:0/289452098 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafac007270 con 0x7fafc00aed90 2024-04-16T16:15:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.784+0000 7fafc5400640 1 -- 172.21.15.45:0/289452098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc0161690 con 0x7fafc00aed90 2024-04-16T16:15:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.784+0000 7fafc5400640 1 -- 172.21.15.45:0/289452098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc0161c50 con 0x7fafc00aed90 2024-04-16T16:15:20.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.785+0000 7fafc5400640 1 -- 172.21.15.45:0/289452098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc0075780 con 0x7fafc00aed90 2024-04-16T16:15:20.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.786+0000 7fafb6a00640 1 -- 172.21.15.45:0/289452098 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fafac01a070 con 0x7fafc00aed90 2024-04-16T16:15:20.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.787+0000 7fafb6a00640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fafa006ca60 0x7fafa006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:20.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.787+0000 7fafb6a00640 1 -- 172.21.15.45:0/289452098 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6127+0+0 (secure 0 0 0) 0x7fafac08b1b0 con 0x7fafc00aed90 2024-04-16T16:15:20.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.788+0000 7fafbe800640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fafa006ca60 0x7fafa006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:20.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.789+0000 7fafbe800640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fafa006ca60 0x7fafa006ef20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fafb0001610 tx=0x7fafb0009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:20.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:20.792+0000 7fafb6a00640 1 -- 172.21.15.45:0/289452098 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fafac058cd0 con 0x7fafc00aed90 2024-04-16T16:15:21.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:21.038+0000 7fafc5400640 1 -- 172.21.15.45:0/289452098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafc0070220 con 0x7fafc00aed90 2024-04-16T16:15:21.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:21.040+0000 7fafb6a00640 1 -- 172.21.15.45:0/289452098 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5166 (secure 0 0 0) 0x7fafac05cfd0 con 0x7fafc00aed90 2024-04-16T16:15:21.039 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:21.040 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/814837411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":814837411},{"type":"v1","addr":"172.21.15.103:6836","nonce":814837411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/401733314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":401733314},{"type":"v1","addr":"172.21.15.103:6837","nonce":401733314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5073,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/768816269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":768816269},{"type":"v1","addr":"172.21.15.45:6834","nonce":768816269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:14.878986+0000","modified":"2024-04-16T16:15:20.697724+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5646},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5646":{"gid":5646,"name":"a","rank":0,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.45:6833/368835060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":368835060},{"type":"v1","addr":"172.21.15.45:6833","nonce":368835060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"fs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-04-16T16:15:21.040 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 25 2024-04-16T16:15:21.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:21.042+0000 7fafc5400640 1 -- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fafa006ca60 msgr2=0x7fafa006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:21.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:21.042+0000 7fafc5400640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fafa006ca60 0x7fafa006ef20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fafb0001610 tx=0x7fafb0009040 comp rx=0 tx=0).stop 2024-04-16T16:15:21.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:21.042+0000 7fafc5400640 1 -- 172.21.15.45:0/289452098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc00aed90 msgr2=0x7fafc01f7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:21.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:21.042+0000 7fafc5400640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc00aed90 0x7fafc01f7910 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fafac00c4d0 tx=0x7fafac00c990 comp rx=0 tx=0).stop 2024-04-16T16:15:21.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:21.042+0000 7fafc5400640 1 -- 172.21.15.45:0/289452098 shutdown_connections 2024-04-16T16:15:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:21.042+0000 7fafc5400640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fafa006ca60 0x7fafa006ef20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:21.042+0000 7fafc5400640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc0147840 0x7fafc01f7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:21.042+0000 7fafc5400640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc00aed90 0x7fafc01f7910 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:21.042+0000 7fafc5400640 1 --2- 172.21.15.45:0/289452098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fafc0072a90 0x7fafc01e8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:21.042+0000 7fafc5400640 1 -- 172.21.15.45:0/289452098 >> 172.21.15.45:0/289452098 conn(0x7fafc00b2750 msgr2=0x7fafc00b5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:21.043+0000 7fafc5400640 1 -- 172.21.15.45:0/289452098 shutdown_connections 2024-04-16T16:15:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:21.043+0000 7fafc5400640 1 -- 172.21.15.45:0/289452098 wait complete. 2024-04-16T16:15:21.463 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:15:14.878986+0000', 'modified': '2024-04-16T16:15:20.697724+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 43, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5646}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5646': {'gid': 5646, 'name': 'a', 'rank': 0, 'incarnation': 25, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.45:6833/368835060', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6832', 'nonce': 368835060}, {'type': 'v1', 'addr': '172.21.15.45:6833', 'nonce': 368835060}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'fs1', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-16T16:15:21.463 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5646:up:replay 2024-04-16T16:15:22.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:15:22.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.695+0000 7fda5be00640 1 Processor -- start 2024-04-16T16:15:22.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.695+0000 7fda5be00640 1 -- start start 2024-04-16T16:15:22.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.696+0000 7fda5be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda5c161e90 0x7fda5c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:22.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.696+0000 7fda5be00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda5c0d3080 con 0x7fda5c162270 2024-04-16T16:15:22.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.696+0000 7fda5be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda5c0d3360 con 0x7fda5c0d52d0 2024-04-16T16:15:22.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.696+0000 7fda5be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda5c0d2e30 con 0x7fda5c161e90 2024-04-16T16:15:22.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.696+0000 7fda5a000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fda5c162270 0x7fda5c0d3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42088/0 (socket says 172.21.15.45:42088) 2024-04-16T16:15:22.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.696+0000 7fda5a000640 1 -- 172.21.15.45:0/301977037 learned_addr learned my addr 172.21.15.45:0/301977037 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.697+0000 7fda5aa00640 1 --2- 172.21.15.45:0/301977037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda5c161e90 0x7fda5c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.697+0000 7fda59600640 1 -- 172.21.15.45:0/301977037 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 989495152 0 0) 0x7fda5c0d3080 con 0x7fda5c162270 2024-04-16T16:15:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.697+0000 7fda59600640 1 -- 172.21.15.45:0/301977037 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda40003520 con 0x7fda5c162270 2024-04-16T16:15:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.697+0000 7fda59600640 1 -- 172.21.15.45:0/301977037 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2365328327 0 0) 0x7fda40003520 con 0x7fda5c162270 2024-04-16T16:15:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.697+0000 7fda59600640 1 -- 172.21.15.45:0/301977037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda5c161e90 msgr2=0x7fda5c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.697+0000 7fda59600640 1 --2- 172.21.15.45:0/301977037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda5c161e90 0x7fda5c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.697+0000 7fda59600640 1 -- 172.21.15.45:0/301977037 >> v1:172.21.15.103:6789/0 conn(0x7fda5c0d52d0 legacy=0x7fda5c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.697+0000 7fda59600640 1 -- 172.21.15.45:0/301977037 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5c0d3640 con 0x7fda5c162270 2024-04-16T16:15:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.697+0000 7fda59600640 1 -- 172.21.15.45:0/301977037 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fda480019f0 con 0x7fda5c162270 2024-04-16T16:15:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda59600640 1 -- 172.21.15.45:0/301977037 >> v1:172.21.15.45:6789/0 conn(0x7fda5c162270 legacy=0x7fda5c0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda59600640 1 --2- 172.21.15.45:0/301977037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda400039e0 0x7fda40003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda59600640 1 --2- 172.21.15.45:0/301977037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda40004800 0x7fda40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda59600640 1 --2- 172.21.15.45:0/301977037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda5c161e90 0x7fda400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda59600640 1 -- 172.21.15.45:0/301977037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c0d2e30 con 0x7fda40004800 2024-04-16T16:15:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda59600640 1 -- 172.21.15.45:0/301977037 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda4000bb00 con 0x7fda400039e0 2024-04-16T16:15:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda59600640 1 -- 172.21.15.45:0/301977037 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda4000bd00 con 0x7fda5c161e90 2024-04-16T16:15:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda5a000640 1 --2- 172.21.15.45:0/301977037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda40004800 0x7fda40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda5aa00640 1 --2- 172.21.15.45:0/301977037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda400039e0 0x7fda40003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda5b400640 1 --2- 172.21.15.45:0/301977037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda5c161e90 0x7fda400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda5a000640 1 -- 172.21.15.45:0/301977037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda5c161e90 msgr2=0x7fda400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda5a000640 1 --2- 172.21.15.45:0/301977037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda5c161e90 0x7fda400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda5a000640 1 -- 172.21.15.45:0/301977037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda400039e0 msgr2=0x7fda40003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda5a000640 1 --2- 172.21.15.45:0/301977037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda400039e0 0x7fda40003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda5a000640 1 -- 172.21.15.45:0/301977037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda40003520 con 0x7fda40004800 2024-04-16T16:15:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.698+0000 7fda5b400640 1 --2- 172.21.15.45:0/301977037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda5c161e90 0x7fda400051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.700+0000 7fda5a000640 1 --2- 172.21.15.45:0/301977037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda40004800 0x7fda40004c80 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fda48002230 tx=0x7fda40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438b17f6b3795ca2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.700+0000 7fda59600640 1 -- 172.21.15.45:0/301977037 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4800a4a0 con 0x7fda40004800 2024-04-16T16:15:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.700+0000 7fda59600640 1 -- 172.21.15.45:0/301977037 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda48034650 con 0x7fda40004800 2024-04-16T16:15:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.700+0000 7fda59600640 1 -- 172.21.15.45:0/301977037 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48034a40 con 0x7fda40004800 2024-04-16T16:15:22.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.700+0000 7fda5be00640 1 -- 172.21.15.45:0/301977037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda40004800 msgr2=0x7fda40004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:22.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.700+0000 7fda5be00640 1 --2- 172.21.15.45:0/301977037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda40004800 0x7fda40004c80 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fda48002230 tx=0x7fda40005830 comp rx=0 tx=0).stop 2024-04-16T16:15:22.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.700+0000 7fda5b400640 1 -- 172.21.15.45:0/301977037 reap_dead start 2024-04-16T16:15:22.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.701+0000 7fda5be00640 1 -- 172.21.15.45:0/301977037 shutdown_connections 2024-04-16T16:15:22.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.701+0000 7fda5be00640 1 -- 172.21.15.45:0/301977037 >> 172.21.15.45:0/301977037 conn(0x7fda5c0d1940 msgr2=0x7fda5c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.701+0000 7fda5be00640 1 -- 172.21.15.45:0/301977037 shutdown_connections 2024-04-16T16:15:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.701+0000 7fda5be00640 1 -- 172.21.15.45:0/301977037 wait complete. 2024-04-16T16:15:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.702+0000 7fda5be00640 1 Processor -- start 2024-04-16T16:15:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.703+0000 7fda5be00640 1 -- start start 2024-04-16T16:15:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.704+0000 7fda5be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda40004800 0x7fda5c155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.704+0000 7fda5be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5c0abd40 0x7fda5c0ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.704+0000 7fda5be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda5c1f4910 0x7fda5c1f4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.704+0000 7fda5be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c0d46d0 con 0x7fda5c0abd40 2024-04-16T16:15:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.704+0000 7fda5be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c0d4550 con 0x7fda5c1f4910 2024-04-16T16:15:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.704+0000 7fda5be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda5c06d550 con 0x7fda40004800 2024-04-16T16:15:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.704+0000 7fda5a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5c0abd40 0x7fda5c0ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.704+0000 7fda5a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5c0abd40 0x7fda5c0ce280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57050/0 (socket says 172.21.15.45:57050) 2024-04-16T16:15:22.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.704+0000 7fda5a000640 1 -- 172.21.15.45:0/1727123469 learned_addr learned my addr 172.21.15.45:0/1727123469 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:22.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.704+0000 7fda5aa00640 1 --2- 172.21.15.45:0/1727123469 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda40004800 0x7fda5c155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:22.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.704+0000 7fda5b400640 1 --2- 172.21.15.45:0/1727123469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda5c1f4910 0x7fda5c1f4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:22.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.705+0000 7fda5a000640 1 -- 172.21.15.45:0/1727123469 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda40004800 msgr2=0x7fda5c155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:22.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.705+0000 7fda5a000640 1 --2- 172.21.15.45:0/1727123469 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda40004800 0x7fda5c155050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:22.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.705+0000 7fda5a000640 1 -- 172.21.15.45:0/1727123469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda5c1f4910 msgr2=0x7fda5c1f4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:22.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.705+0000 7fda5a000640 1 --2- 172.21.15.45:0/1727123469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda5c1f4910 0x7fda5c1f4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:22.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.705+0000 7fda5a000640 1 -- 172.21.15.45:0/1727123469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5c0ceac0 con 0x7fda5c0abd40 2024-04-16T16:15:22.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.705+0000 7fda5a000640 1 --2- 172.21.15.45:0/1727123469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5c0abd40 0x7fda5c0ce280 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fda4800a940 tx=0x7fda48001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:22.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.707+0000 7fda53400640 1 -- 172.21.15.45:0/1727123469 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48034800 con 0x7fda5c0abd40 2024-04-16T16:15:22.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.707+0000 7fda5be00640 1 -- 172.21.15.45:0/1727123469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda5c0ceda0 con 0x7fda5c0abd40 2024-04-16T16:15:22.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.708+0000 7fda5be00640 1 -- 172.21.15.45:0/1727123469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda5c0cf360 con 0x7fda5c0abd40 2024-04-16T16:15:22.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.708+0000 7fda53400640 1 -- 172.21.15.45:0/1727123469 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda48044070 con 0x7fda5c0abd40 2024-04-16T16:15:22.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.708+0000 7fda53400640 1 -- 172.21.15.45:0/1727123469 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48034ba0 con 0x7fda5c0abd40 2024-04-16T16:15:22.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.709+0000 7fda5be00640 1 -- 172.21.15.45:0/1727123469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda34005290 con 0x7fda5c0abd40 2024-04-16T16:15:22.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.710+0000 7fda53400640 1 -- 172.21.15.45:0/1727123469 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fda48049460 con 0x7fda5c0abd40 2024-04-16T16:15:22.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.711+0000 7fda53400640 1 --2- 172.21.15.45:0/1727123469 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fda3c06ca60 0x7fda3c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:22.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.711+0000 7fda53400640 1 -- 172.21.15.45:0/1727123469 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7fda480ae6c0 con 0x7fda5c0abd40 2024-04-16T16:15:22.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.712+0000 7fda5aa00640 1 --2- 172.21.15.45:0/1727123469 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fda3c06ca60 0x7fda3c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:22.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.713+0000 7fda5aa00640 1 --2- 172.21.15.45:0/1727123469 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fda3c06ca60 0x7fda3c06ef20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fda4c009a10 tx=0x7fda4c008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:22.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.716+0000 7fda53400640 1 -- 172.21.15.45:0/1727123469 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda480483c0 con 0x7fda5c0abd40 2024-04-16T16:15:22.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.965+0000 7fda5be00640 1 -- 172.21.15.45:0/1727123469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda34005850 con 0x7fda5c0abd40 2024-04-16T16:15:22.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.966+0000 7fda53400640 1 -- 172.21.15.45:0/1727123469 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5166 (secure 0 0 0) 0x7fda4807c7f0 con 0x7fda5c0abd40 2024-04-16T16:15:22.966 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:22.966 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5056,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3982331497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3982331497},{"type":"v1","addr":"172.21.15.103:6837","nonce":3982331497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5065,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/842795151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":842795151},{"type":"v1","addr":"172.21.15.45:6835","nonce":842795151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/88032513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":88032513},{"type":"v1","addr":"172.21.15.103:6835","nonce":88032513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:14.878986+0000","modified":"2024-04-16T16:15:21.711239+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5646},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5646":{"gid":5646,"name":"a","rank":0,"incarnation":25,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.45:6833/368835060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":368835060},{"type":"v1","addr":"172.21.15.45:6833","nonce":368835060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"fs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-04-16T16:15:22.966 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 27 2024-04-16T16:15:22.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.969+0000 7fda5be00640 1 -- 172.21.15.45:0/1727123469 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fda3c06ca60 msgr2=0x7fda3c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:22.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.969+0000 7fda5be00640 1 --2- 172.21.15.45:0/1727123469 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fda3c06ca60 0x7fda3c06ef20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fda4c009a10 tx=0x7fda4c008040 comp rx=0 tx=0).stop 2024-04-16T16:15:22.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.969+0000 7fda5be00640 1 -- 172.21.15.45:0/1727123469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5c0abd40 msgr2=0x7fda5c0ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:22.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.969+0000 7fda5be00640 1 --2- 172.21.15.45:0/1727123469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5c0abd40 0x7fda5c0ce280 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fda4800a940 tx=0x7fda48001dc0 comp rx=0 tx=0).stop 2024-04-16T16:15:22.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.970+0000 7fda5be00640 1 -- 172.21.15.45:0/1727123469 shutdown_connections 2024-04-16T16:15:22.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.970+0000 7fda5be00640 1 --2- 172.21.15.45:0/1727123469 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fda3c06ca60 0x7fda3c06ef20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:22.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.970+0000 7fda5be00640 1 --2- 172.21.15.45:0/1727123469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda5c1f4910 0x7fda5c1f4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:22.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.970+0000 7fda5be00640 1 --2- 172.21.15.45:0/1727123469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5c0abd40 0x7fda5c0ce280 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:22.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.970+0000 7fda5be00640 1 --2- 172.21.15.45:0/1727123469 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda40004800 0x7fda5c155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:22.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.970+0000 7fda5be00640 1 -- 172.21.15.45:0/1727123469 >> 172.21.15.45:0/1727123469 conn(0x7fda5c0d1940 msgr2=0x7fda5c0b98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:22.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.970+0000 7fda5be00640 1 -- 172.21.15.45:0/1727123469 shutdown_connections 2024-04-16T16:15:22.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:22.970+0000 7fda5be00640 1 -- 172.21.15.45:0/1727123469 wait complete. 2024-04-16T16:15:23.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:15:14.878986+0000', 'modified': '2024-04-16T16:15:21.711239+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 43, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5646}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5646': {'gid': 5646, 'name': 'a', 'rank': 0, 'incarnation': 25, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.45:6833/368835060', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6832', 'nonce': 368835060}, {'type': 'v1', 'addr': '172.21.15.45:6833', 'nonce': 368835060}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'fs1', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-16T16:15:23.390 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5646:up:rejoin 2024-04-16T16:15:23.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:15:23.524 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:15:24.391 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:15:24.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.623+0000 7f8a4ca00640 1 Processor -- start 2024-04-16T16:15:24.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.624+0000 7f8a4ca00640 1 -- start start 2024-04-16T16:15:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.625+0000 7f8a4ca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8a480cde00 0x7f8a480cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.625+0000 7f8a4ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a48163010 0x7f8a4806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.626+0000 7f8a4ca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a48163510 con 0x7f8a480ce1e0 2024-04-16T16:15:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.626+0000 7f8a4ca00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f8a481637f0 con 0x7f8a480cde00 2024-04-16T16:15:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.626+0000 7f8a4ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a48163ad0 con 0x7f8a48163010 2024-04-16T16:15:24.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.626+0000 7f8a45e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a48163010 0x7f8a4806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:24.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.627+0000 7f8a45e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a48163010 0x7f8a4806f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:43212/0 (socket says 172.21.15.45:43212) 2024-04-16T16:15:24.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.627+0000 7f8a45e00640 1 -- 172.21.15.45:0/2663777061 learned_addr learned my addr 172.21.15.45:0/2663777061 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:24.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.627+0000 7f8a46800640 1 --2- 172.21.15.45:0/2663777061 >> v2:172.21.15.103:3300/0 conn(0x7f8a480cde00 0x7f8a480cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:24.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.627+0000 7f8a45e00640 1 -- 172.21.15.45:0/2663777061 >> v2:172.21.15.103:3300/0 conn(0x7f8a480cde00 msgr2=0x7f8a480cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.627+0000 7f8a45e00640 1 --2- 172.21.15.45:0/2663777061 >> v2:172.21.15.103:3300/0 conn(0x7f8a480cde00 0x7f8a480cff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.627+0000 7f8a45e00640 1 -- 172.21.15.45:0/2663777061 >> v1:172.21.15.103:6789/0 conn(0x7f8a480ce1e0 legacy=0x7f8a4806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.627+0000 7f8a45e00640 1 -- 172.21.15.45:0/2663777061 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a48163db0 con 0x7f8a48163010 2024-04-16T16:15:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.629+0000 7f8a45e00640 1 --2- 172.21.15.45:0/2663777061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a48163010 0x7f8a4806f540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8a30009870 tx=0x7f8a3002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8702cf54f6eac39a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.629+0000 7f8a45400640 1 -- 172.21.15.45:0/2663777061 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30034070 con 0x7f8a48163010 2024-04-16T16:15:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.629+0000 7f8a45400640 1 -- 172.21.15.45:0/2663777061 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a30030840 con 0x7f8a48163010 2024-04-16T16:15:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.629+0000 7f8a45400640 1 -- 172.21.15.45:0/2663777061 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30030c10 con 0x7f8a48163010 2024-04-16T16:15:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.630+0000 7f8a4ca00640 1 -- 172.21.15.45:0/2663777061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a48163010 msgr2=0x7f8a4806f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.630+0000 7f8a4ca00640 1 --2- 172.21.15.45:0/2663777061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a48163010 0x7f8a4806f540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8a30009870 tx=0x7f8a3002b980 comp rx=0 tx=0).stop 2024-04-16T16:15:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.630+0000 7f8a4ca00640 1 -- 172.21.15.45:0/2663777061 shutdown_connections 2024-04-16T16:15:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.630+0000 7f8a4ca00640 1 --2- 172.21.15.45:0/2663777061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a48163010 0x7f8a4806f540 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.630+0000 7f8a4ca00640 1 --2- 172.21.15.45:0/2663777061 >> v2:172.21.15.103:3300/0 conn(0x7f8a480cde00 0x7f8a480cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.630+0000 7f8a4ca00640 1 -- 172.21.15.45:0/2663777061 >> 172.21.15.45:0/2663777061 conn(0x7f8a4806b930 msgr2=0x7f8a4806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.630+0000 7f8a4ca00640 1 -- 172.21.15.45:0/2663777061 shutdown_connections 2024-04-16T16:15:24.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.630+0000 7f8a4ca00640 1 -- 172.21.15.45:0/2663777061 wait complete. 2024-04-16T16:15:24.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.632+0000 7f8a4ca00640 1 Processor -- start 2024-04-16T16:15:24.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.632+0000 7f8a4ca00640 1 -- start start 2024-04-16T16:15:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.633+0000 7f8a4ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a480cde00 0x7f8a48148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.633+0000 7f8a4ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a480ce1e0 0x7f8a481ed9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.633+0000 7f8a4ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a48163010 0x7f8a481edf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.633+0000 7f8a4ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a480ce7b0 con 0x7f8a480ce1e0 2024-04-16T16:15:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.633+0000 7f8a4ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a480ce630 con 0x7f8a480cde00 2024-04-16T16:15:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.633+0000 7f8a4ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a481655c0 con 0x7f8a48163010 2024-04-16T16:15:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.633+0000 7f8a45e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a480ce1e0 0x7f8a481ed9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.633+0000 7f8a45e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a480ce1e0 0x7f8a481ed9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38704/0 (socket says 172.21.15.45:38704) 2024-04-16T16:15:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.633+0000 7f8a45e00640 1 -- 172.21.15.45:0/1739520943 learned_addr learned my addr 172.21.15.45:0/1739520943 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.634+0000 7f8a46800640 1 --2- 172.21.15.45:0/1739520943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a480cde00 0x7f8a48148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.634+0000 7f8a47200640 1 --2- 172.21.15.45:0/1739520943 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a48163010 0x7f8a481edf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.634+0000 7f8a45e00640 1 -- 172.21.15.45:0/1739520943 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a48163010 msgr2=0x7f8a481edf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.634+0000 7f8a45e00640 1 --2- 172.21.15.45:0/1739520943 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a48163010 0x7f8a481edf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:24.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.634+0000 7f8a45e00640 1 -- 172.21.15.45:0/1739520943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a480cde00 msgr2=0x7f8a48148690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:24.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.634+0000 7f8a45e00640 1 --2- 172.21.15.45:0/1739520943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a480cde00 0x7f8a48148690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:24.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.634+0000 7f8a45e00640 1 -- 172.21.15.45:0/1739520943 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4815c7e0 con 0x7f8a480ce1e0 2024-04-16T16:15:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.634+0000 7f8a45e00640 1 --2- 172.21.15.45:0/1739520943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a480ce1e0 0x7f8a481ed9c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8a300090b0 tx=0x7f8a300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.634+0000 7f8a3ea00640 1 -- 172.21.15.45:0/1739520943 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30034040 con 0x7f8a480ce1e0 2024-04-16T16:15:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.635+0000 7f8a3ea00640 1 -- 172.21.15.45:0/1739520943 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a3003ada0 con 0x7f8a480ce1e0 2024-04-16T16:15:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.635+0000 7f8a3ea00640 1 -- 172.21.15.45:0/1739520943 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30044610 con 0x7f8a480ce1e0 2024-04-16T16:15:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.635+0000 7f8a4ca00640 1 -- 172.21.15.45:0/1739520943 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a4815cac0 con 0x7f8a480ce1e0 2024-04-16T16:15:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.635+0000 7f8a4ca00640 1 -- 172.21.15.45:0/1739520943 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a4815d080 con 0x7f8a480ce1e0 2024-04-16T16:15:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.636+0000 7f8a4ca00640 1 -- 172.21.15.45:0/1739520943 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a48075780 con 0x7f8a480ce1e0 2024-04-16T16:15:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.637+0000 7f8a3ea00640 1 -- 172.21.15.45:0/1739520943 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f8a3004b060 con 0x7f8a480ce1e0 2024-04-16T16:15:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.638+0000 7f8a3ea00640 1 --2- 172.21.15.45:0/1739520943 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f8a2406ca60 0x7f8a2406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.638+0000 7f8a3ea00640 1 -- 172.21.15.45:0/1739520943 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7f8a300adfe0 con 0x7f8a480ce1e0 2024-04-16T16:15:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.639+0000 7f8a46800640 1 --2- 172.21.15.45:0/1739520943 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f8a2406ca60 0x7f8a2406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.640+0000 7f8a46800640 1 --2- 172.21.15.45:0/1739520943 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f8a2406ca60 0x7f8a2406ef20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8a380074a0 tx=0x7f8a3800b040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:24.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.643+0000 7f8a3ea00640 1 -- 172.21.15.45:0/1739520943 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8a3003f070 con 0x7f8a480ce1e0 2024-04-16T16:15:24.718 INFO:tasks.ceph.mgr.x.smithi103.stderr:Exception in thread Thread-7: 2024-04-16T16:15:24.718 INFO:tasks.ceph.mgr.x.smithi103.stderr:Traceback (most recent call last): 2024-04-16T16:15:24.719 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner 2024-04-16T16:15:24.719 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.run() 2024-04-16T16:15:24.719 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 1306, in run 2024-04-16T16:15:24.719 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.function(*self.args, **self.kwargs) 2024-04-16T16:15:24.719 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/share/ceph/mgr/stats/fs/perf_stats.py", line 222, in re_register_queries 2024-04-16T16:15:24.719 INFO:tasks.ceph.mgr.x.smithi103.stderr: if self.mx_last_updated >= ua_last_updated: 2024-04-16T16:15:24.720 INFO:tasks.ceph.mgr.x.smithi103.stderr:AttributeError: 'FSPerfStats' object has no attribute 'mx_last_updated' 2024-04-16T16:15:24.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.889+0000 7f8a4ca00640 1 -- 172.21.15.45:0/1739520943 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a4806fc20 con 0x7f8a480ce1e0 2024-04-16T16:15:24.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.890+0000 7f8a3ea00640 1 -- 172.21.15.45:0/1739520943 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5173 (secure 0 0 0) 0x7f8a3007c170 con 0x7f8a480ce1e0 2024-04-16T16:15:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:24.889 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5056,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3982331497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3982331497},{"type":"v1","addr":"172.21.15.103:6837","nonce":3982331497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5065,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/842795151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":842795151},{"type":"v1","addr":"172.21.15.45:6835","nonce":842795151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/88032513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":88032513},{"type":"v1","addr":"172.21.15.103:6835","nonce":88032513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:14.878986+0000","modified":"2024-04-16T16:15:23.706709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5646},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5646":{"gid":5646,"name":"a","rank":0,"incarnation":25,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6833/368835060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":368835060},{"type":"v1","addr":"172.21.15.45:6833","nonce":368835060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"fs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5646,"qdb_cluster":[5646]},"id":3}]} 2024-04-16T16:15:24.889 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 28 2024-04-16T16:15:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.893+0000 7f8a4ca00640 1 -- 172.21.15.45:0/1739520943 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f8a2406ca60 msgr2=0x7f8a2406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.893+0000 7f8a4ca00640 1 --2- 172.21.15.45:0/1739520943 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f8a2406ca60 0x7f8a2406ef20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8a380074a0 tx=0x7f8a3800b040 comp rx=0 tx=0).stop 2024-04-16T16:15:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.893+0000 7f8a4ca00640 1 -- 172.21.15.45:0/1739520943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a480ce1e0 msgr2=0x7f8a481ed9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.893+0000 7f8a4ca00640 1 --2- 172.21.15.45:0/1739520943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a480ce1e0 0x7f8a481ed9c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8a300090b0 tx=0x7f8a300046c0 comp rx=0 tx=0).stop 2024-04-16T16:15:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.893+0000 7f8a4ca00640 1 -- 172.21.15.45:0/1739520943 shutdown_connections 2024-04-16T16:15:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.893+0000 7f8a4ca00640 1 --2- 172.21.15.45:0/1739520943 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a48163010 0x7f8a481edf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.893+0000 7f8a4ca00640 1 --2- 172.21.15.45:0/1739520943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a480ce1e0 0x7f8a481ed9c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.893+0000 7f8a4ca00640 1 --2- 172.21.15.45:0/1739520943 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f8a2406ca60 0x7f8a2406ef20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.893+0000 7f8a4ca00640 1 --2- 172.21.15.45:0/1739520943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a480cde00 0x7f8a48148690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.893+0000 7f8a4ca00640 1 -- 172.21.15.45:0/1739520943 >> 172.21.15.45:0/1739520943 conn(0x7f8a4806b930 msgr2=0x7f8a4806e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.893+0000 7f8a4ca00640 1 -- 172.21.15.45:0/1739520943 shutdown_connections 2024-04-16T16:15:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:24.893+0000 7f8a4ca00640 1 -- 172.21.15.45:0/1739520943 wait complete. 2024-04-16T16:15:25.315 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:15:14.878986+0000', 'modified': '2024-04-16T16:15:23.706709+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 43, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5646}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5646': {'gid': 5646, 'name': 'a', 'rank': 0, 'incarnation': 25, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.45:6833/368835060', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6832', 'nonce': 368835060}, {'type': 'v1', 'addr': '172.21.15.45:6833', 'nonce': 368835060}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'fs1', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5646, 'qdb_cluster': [5646]} 2024-04-16T16:15:25.316 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-16T16:15:25.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-16T16:15:25.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.543+0000 7f89bbe00640 1 Processor -- start 2024-04-16T16:15:25.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.543+0000 7f89bbe00640 1 -- start start 2024-04-16T16:15:25.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.545+0000 7f89bbe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f89bc06f410 0x7f89bc163010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.545+0000 7f89bbe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89bc0cd210 con 0x7f89bc0ce710 2024-04-16T16:15:25.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.545+0000 7f89bbe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89bc0cd4f0 con 0x7f89bc0b1ce0 2024-04-16T16:15:25.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.545+0000 7f89bbe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f89bc0d02f0 con 0x7f89bc06f410 2024-04-16T16:15:25.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.545+0000 7f89bb400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f89bc0ce710 0x7f89bc163ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50052/0 (socket says 172.21.15.45:50052) 2024-04-16T16:15:25.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.545+0000 7f89bb400640 1 -- 172.21.15.45:0/2059160705 learned_addr learned my addr 172.21.15.45:0/2059160705 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:25.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.545+0000 7f89b3e00640 1 --2- 172.21.15.45:0/2059160705 >> v2:172.21.15.45:3300/0 conn(0x7f89bc06f410 0x7f89bc163010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.546+0000 7f89b3e00640 1 -- 172.21.15.45:0/2059160705 >> v1:172.21.15.103:6789/0 conn(0x7f89bc0b1ce0 legacy=0x7f89bc0b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:25.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.546+0000 7f89ba000640 1 -- 172.21.15.45:0/2059160705 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3043322348 0 0) 0x7f89bc0cd210 con 0x7f89bc0ce710 2024-04-16T16:15:25.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.546+0000 7f89b3e00640 1 -- 172.21.15.45:0/2059160705 >> v1:172.21.15.45:6789/0 conn(0x7f89bc0ce710 legacy=0x7f89bc163ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:25.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.546+0000 7f89b3e00640 1 -- 172.21.15.45:0/2059160705 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89bc163680 con 0x7f89bc06f410 2024-04-16T16:15:25.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.547+0000 7f89b3e00640 1 --2- 172.21.15.45:0/2059160705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc06f410 0x7f89bc163010 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f89a40097a0 tx=0x7f89a402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494d4969bce986c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:25.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.547+0000 7f89ba000640 1 -- 172.21.15.45:0/2059160705 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a4038070 con 0x7f89bc06f410 2024-04-16T16:15:25.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.547+0000 7f89ba000640 1 -- 172.21.15.45:0/2059160705 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a4031700 con 0x7f89bc06f410 2024-04-16T16:15:25.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.547+0000 7f89ba000640 1 -- 172.21.15.45:0/2059160705 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a4031ad0 con 0x7f89bc06f410 2024-04-16T16:15:25.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.548+0000 7f89bbe00640 1 -- 172.21.15.45:0/2059160705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc06f410 msgr2=0x7f89bc163010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:25.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.548+0000 7f89bbe00640 1 --2- 172.21.15.45:0/2059160705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc06f410 0x7f89bc163010 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f89a40097a0 tx=0x7f89a402b7a0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.548+0000 7f89bbe00640 1 -- 172.21.15.45:0/2059160705 shutdown_connections 2024-04-16T16:15:25.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.548+0000 7f89bbe00640 1 --2- 172.21.15.45:0/2059160705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc06f410 0x7f89bc163010 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.548+0000 7f89bbe00640 1 -- 172.21.15.45:0/2059160705 >> 172.21.15.45:0/2059160705 conn(0x7f89bc06b930 msgr2=0x7f89bc06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:25.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.548+0000 7f89bbe00640 1 -- 172.21.15.45:0/2059160705 shutdown_connections 2024-04-16T16:15:25.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.548+0000 7f89bbe00640 1 -- 172.21.15.45:0/2059160705 wait complete. 2024-04-16T16:15:25.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.550+0000 7f89bbe00640 1 Processor -- start 2024-04-16T16:15:25.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.550+0000 7f89bbe00640 1 -- start start 2024-04-16T16:15:25.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.551+0000 7f89bbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89bc06f410 0x7f89bc15ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.551+0000 7f89bbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89bc0b1ce0 0x7f89bc14ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.551+0000 7f89bbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc0ce710 0x7f89bc155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.551+0000 7f89bbe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f89bc164430 con 0x7f89bc0ce710 2024-04-16T16:15:25.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.551+0000 7f89bbe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89bc1642b0 con 0x7f89bc0b1ce0 2024-04-16T16:15:25.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.551+0000 7f89bbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f89bc1645b0 con 0x7f89bc06f410 2024-04-16T16:15:25.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.551+0000 7f89bb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc0ce710 0x7f89bc155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.551+0000 7f89bb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc0ce710 0x7f89bc155140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38730/0 (socket says 172.21.15.45:38730) 2024-04-16T16:15:25.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.551+0000 7f89bb400640 1 -- 172.21.15.45:0/948470784 learned_addr learned my addr 172.21.15.45:0/948470784 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:25.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.552+0000 7f89b3e00640 1 --2- 172.21.15.45:0/948470784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89bc0b1ce0 0x7f89bc14ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.552+0000 7f89baa00640 1 --2- 172.21.15.45:0/948470784 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89bc06f410 0x7f89bc15ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.552+0000 7f89bb400640 1 -- 172.21.15.45:0/948470784 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89bc06f410 msgr2=0x7f89bc15ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:25.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.552+0000 7f89bb400640 1 --2- 172.21.15.45:0/948470784 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89bc06f410 0x7f89bc15ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.552+0000 7f89bb400640 1 -- 172.21.15.45:0/948470784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89bc0b1ce0 msgr2=0x7f89bc14ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:25.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.552+0000 7f89bb400640 1 --2- 172.21.15.45:0/948470784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89bc0b1ce0 0x7f89bc14ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.552+0000 7f89bb400640 1 -- 172.21.15.45:0/948470784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89bc15ee60 con 0x7f89bc0ce710 2024-04-16T16:15:25.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.552+0000 7f89b3e00640 1 --2- 172.21.15.45:0/948470784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89bc0b1ce0 0x7f89bc14ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:25.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.552+0000 7f89bb400640 1 --2- 172.21.15.45:0/948470784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc0ce710 0x7f89bc155140 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f89b400c730 tx=0x7f89b400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:25.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.552+0000 7f89b2a00640 1 -- 172.21.15.45:0/948470784 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b4018070 con 0x7f89bc0ce710 2024-04-16T16:15:25.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.552+0000 7f89b2a00640 1 -- 172.21.15.45:0/948470784 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b400ce40 con 0x7f89bc0ce710 2024-04-16T16:15:25.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.552+0000 7f89b2a00640 1 -- 172.21.15.45:0/948470784 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b400a860 con 0x7f89bc0ce710 2024-04-16T16:15:25.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.552+0000 7f89bbe00640 1 -- 172.21.15.45:0/948470784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89bc15f140 con 0x7f89bc0ce710 2024-04-16T16:15:25.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.553+0000 7f89bbe00640 1 -- 172.21.15.45:0/948470784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89bc15f700 con 0x7f89bc0ce710 2024-04-16T16:15:25.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.555+0000 7f89b2a00640 1 -- 172.21.15.45:0/948470784 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f89b401b070 con 0x7f89bc0ce710 2024-04-16T16:15:25.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.556+0000 7f89b2a00640 1 --2- 172.21.15.45:0/948470784 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f899806ca90 0x7f899806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.556+0000 7f89b2a00640 1 -- 172.21.15.45:0/948470784 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7f89b408b500 con 0x7f89bc0ce710 2024-04-16T16:15:25.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.557+0000 7f89baa00640 1 --2- 172.21.15.45:0/948470784 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f899806ca90 0x7f899806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.557+0000 7f89c4375740 1 Processor -- start 2024-04-16T16:15:25.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.557+0000 7f89c4375740 1 -- start start 2024-04-16T16:15:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.558+0000 7f89baa00640 1 --2- 172.21.15.45:0/948470784 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f899806ca90 0x7f899806ef50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f89ac004ea0 tx=0x7f89ac009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.558+0000 7f89c4375740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908890 0x55b8619279d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.558+0000 7f89c4375740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908c70 0x55b8618e1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.558+0000 7f89c4375740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b8618e1ab0 0x55b8618df440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.558+0000 7f89c4375740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b8618469c0 con 0x55b861908c70 2024-04-16T16:15:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.558+0000 7f89c4375740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b8617ed560 con 0x55b8618e1ab0 2024-04-16T16:15:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.558+0000 7f89c4375740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b861842980 con 0x55b861908890 2024-04-16T16:15:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.558+0000 7f89b3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908c70 0x55b8618e1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.559+0000 7f89b3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908c70 0x55b8618e1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38744/0 (socket says 172.21.15.45:38744) 2024-04-16T16:15:25.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.559+0000 7f89b3e00640 1 -- 172.21.15.45:0/2428937730 learned_addr learned my addr 172.21.15.45:0/2428937730 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:25.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.559+0000 7f89baa00640 1 --2- 172.21.15.45:0/2428937730 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908890 0x55b8619279d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.559+0000 7f89bb400640 1 --2- 172.21.15.45:0/2428937730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b8618e1ab0 0x55b8618df440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.559+0000 7f89b3e00640 1 -- 172.21.15.45:0/2428937730 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908890 msgr2=0x55b8619279d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:25.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.559+0000 7f89b3e00640 1 --2- 172.21.15.45:0/2428937730 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908890 0x55b8619279d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.559+0000 7f89b3e00640 1 -- 172.21.15.45:0/2428937730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b8618e1ab0 msgr2=0x55b8618df440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:25.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.559+0000 7f89b3e00640 1 --2- 172.21.15.45:0/2428937730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b8618e1ab0 0x55b8618df440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.559+0000 7f89b3e00640 1 -- 172.21.15.45:0/2428937730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8615592d0 con 0x55b861908c70 2024-04-16T16:15:25.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.559+0000 7f89baa00640 1 --2- 172.21.15.45:0/2428937730 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908890 0x55b8619279d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:25.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.559+0000 7f89b3e00640 1 --2- 172.21.15.45:0/2428937730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908c70 0x55b8618e1570 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f89a402b7e0 tx=0x7f89a4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee30377486546ba2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.560+0000 7f89ab000640 1 -- 172.21.15.45:0/2428937730 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a4038040 con 0x55b861908c70 2024-04-16T16:15:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.560+0000 7f89ab000640 1 -- 172.21.15.45:0/2428937730 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a4031b20 con 0x55b861908c70 2024-04-16T16:15:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.560+0000 7f89ab000640 1 -- 172.21.15.45:0/2428937730 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a4031ed0 con 0x55b861908c70 2024-04-16T16:15:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.560+0000 7f89c4375740 1 -- 172.21.15.45:0/2428937730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908c70 msgr2=0x55b8618e1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.560+0000 7f89c4375740 1 --2- 172.21.15.45:0/2428937730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908c70 0x55b8618e1570 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f89a402b7e0 tx=0x7f89a4030000 comp rx=0 tx=0).stop 2024-04-16T16:15:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.561+0000 7f89c4375740 1 -- 172.21.15.45:0/2428937730 shutdown_connections 2024-04-16T16:15:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.561+0000 7f89c4375740 1 --2- 172.21.15.45:0/2428937730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b8618e1ab0 0x55b8618df440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.561+0000 7f89c4375740 1 --2- 172.21.15.45:0/2428937730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908c70 0x55b8618e1570 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.561+0000 7f89c4375740 1 --2- 172.21.15.45:0/2428937730 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908890 0x55b8619279d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.561+0000 7f89c4375740 1 -- 172.21.15.45:0/2428937730 >> 172.21.15.45:0/2428937730 conn(0x55b8618ddb50 msgr2=0x55b8619266d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.561+0000 7f89c4375740 1 -- 172.21.15.45:0/2428937730 shutdown_connections 2024-04-16T16:15:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.561+0000 7f89c4375740 1 -- 172.21.15.45:0/2428937730 wait complete. 2024-04-16T16:15:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.561+0000 7f89c4375740 1 -- start start 2024-04-16T16:15:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.562+0000 7f89c4375740 1 Processor -- start 2024-04-16T16:15:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89c4375740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b861908890 0x55b86190c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89c4375740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908c70 0x55b861939f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89c4375740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b86190cb20 0x55b86193a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89c4375740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b8618db980 con 0x55b86190cb20 2024-04-16T16:15:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89c4375740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b861800920 con 0x55b861908890 2024-04-16T16:15:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89c4375740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b8618e1f30 con 0x55b861908c70 2024-04-16T16:15:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89c4375740 10 client.0 _ll_register_callbacks cb 0x55b861800310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:15:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89bb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b86190cb20 0x55b86193a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89bb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b86190cb20 0x55b86193a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38752/0 (socket says 172.21.15.45:38752) 2024-04-16T16:15:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89bb400640 1 -- 172.21.15.45:0/2720382515 learned_addr learned my addr 172.21.15.45:0/2720382515 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89baa00640 1 --2- 172.21.15.45:0/2720382515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b861908890 0x55b86190c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89b3e00640 1 --2- 172.21.15.45:0/2720382515 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908c70 0x55b861939f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89bb400640 1 -- 172.21.15.45:0/2720382515 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908c70 msgr2=0x55b861939f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89bb400640 1 --2- 172.21.15.45:0/2720382515 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908c70 0x55b861939f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89bb400640 1 -- 172.21.15.45:0/2720382515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b861908890 msgr2=0x55b86190c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89bb400640 1 --2- 172.21.15.45:0/2720382515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b861908890 0x55b86190c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89bb400640 1 -- 172.21.15.45:0/2720382515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8616236b0 con 0x55b86190cb20 2024-04-16T16:15:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.563+0000 7f89baa00640 1 --2- 172.21.15.45:0/2720382515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b861908890 0x55b86190c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.564+0000 7f89bb400640 1 --2- 172.21.15.45:0/2720382515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b86190cb20 0x55b86193a4a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f89b400c700 tx=0x7f89b40596b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85a738cf34947a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.564+0000 7f899f400640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:15:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.564+0000 7f899f400640 1 -- 172.21.15.45:0/2720382515 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b408b2c0 con 0x55b86190cb20 2024-04-16T16:15:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.564+0000 7f89c4375740 1 -- 172.21.15.45:0/2720382515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89a4031b20 con 0x55b86190cb20 2024-04-16T16:15:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.564+0000 7f899f400640 1 -- 172.21.15.45:0/2720382515 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7f89b4058340 con 0x55b86190cb20 2024-04-16T16:15:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.564+0000 7f899f400640 1 -- 172.21.15.45:0/2720382515 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f89b4061b60 con 0x55b86190cb20 2024-04-16T16:15:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.564+0000 7f89c4375740 10 client.5691 fetch_fsmap learned FSMap version 28 2024-04-16T16:15:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.564+0000 7f89c4375740 1 -- 172.21.15.45:0/2720382515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) -- 0x55b861928b70 con 0x55b86190cb20 2024-04-16T16:15:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.565+0000 7f899f400640 1 -- 172.21.15.45:0/2720382515 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a4031b20 con 0x55b86190cb20 2024-04-16T16:15:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.565+0000 7f899f400640 1 -- 172.21.15.45:0/2720382515 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 28) ==== 3491+0+0 (secure 0 0 0) 0x7f89b4089150 con 0x55b86190cb20 2024-04-16T16:15:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.565+0000 7f89c4375740 10 client.5691 fetch_fsmap finished waiting for FSMap version 28 2024-04-16T16:15:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.565+0000 7f89c4375740 10 client.5691 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-16T16:15:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.565+0000 7f89c4375740 20 client.5691 populate_metadata read hostname 'smithi045' 2024-04-16T16:15:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.565+0000 7f89c4375740 1 --2- 172.21.15.45:0/2720382515 >> [v2:172.21.15.45:6832/368835060,v1:172.21.15.45:6833/368835060] conn(0x55b86190b450 0x55b8619371d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.565+0000 7f89c4375740 4 client.5691 mds_command: new command op to 5646 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-16T16:15:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.565+0000 7f89c4375740 1 -- 172.21.15.45:0/2720382515 --> [v2:172.21.15.45:6832/368835060,v1:172.21.15.45:6833/368835060] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8617ecb70 con 0x55b86190b450 2024-04-16T16:15:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.565+0000 7f89baa00640 1 --2- 172.21.15.45:0/2720382515 >> [v2:172.21.15.45:6832/368835060,v1:172.21.15.45:6833/368835060] conn(0x55b86190b450 0x55b8619371d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.566+0000 7f89baa00640 1 --2- 172.21.15.45:0/2720382515 >> [v2:172.21.15.45:6832/368835060,v1:172.21.15.45:6833/368835060] conn(0x55b86190b450 0x55b8619371d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e6db20d8dd419eb server_cookie=c8fc788b4aad57d4 in_seq=0 out_seq=0 2024-04-16T16:15:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.566+0000 7f899f400640 10 client.5691 ms_handle_connect on v2:172.21.15.45:6832/368835060 2024-04-16T16:15:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.567+0000 7f899f400640 1 -- 172.21.15.45:0/2720382515 <== mds.0 v2:172.21.15.45:6832/368835060 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b8617ecb70 con 0x55b86190b450 2024-04-16T16:15:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.567+0000 7f899f400640 10 client.5691 handle_command_reply: tid=0 2024-04-16T16:15:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.567+0000 7f89c4375740 1 client.5691 shutdown 2024-04-16T16:15:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.567+0000 7f89c4375740 1 -- 172.21.15.45:0/2720382515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b86190cb20 msgr2=0x55b86193a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.567+0000 7f89c4375740 1 --2- 172.21.15.45:0/2720382515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b86190cb20 0x55b86193a4a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f89b400c700 tx=0x7f89b40596b0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.567+0000 7f89c4375740 1 -- 172.21.15.45:0/2720382515 shutdown_connections 2024-04-16T16:15:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.567+0000 7f89c4375740 1 --2- 172.21.15.45:0/2720382515 >> [v2:172.21.15.45:6832/368835060,v1:172.21.15.45:6833/368835060] conn(0x55b86190b450 0x55b8619371d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.567+0000 7f89c4375740 1 --2- 172.21.15.45:0/2720382515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b86190cb20 0x55b86193a4a0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.567+0000 7f89c4375740 1 --2- 172.21.15.45:0/2720382515 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908c70 0x55b861939f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.567+0000 7f89c4375740 1 --2- 172.21.15.45:0/2720382515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b861908890 0x55b86190c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.567+0000 7f899f400640 0 client.5691 ms_handle_reset on v2:172.21.15.45:6832/368835060 2024-04-16T16:15:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.567+0000 7f89c4375740 1 -- 172.21.15.45:0/2720382515 >> 172.21.15.45:0/2720382515 conn(0x55b8618e1ab0 msgr2=0x55b8618da790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.567+0000 7f89c4375740 1 -- 172.21.15.45:0/2720382515 shutdown_connections 2024-04-16T16:15:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.567+0000 7f89c4375740 1 -- 172.21.15.45:0/2720382515 wait complete. 2024-04-16T16:15:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.567+0000 7f89c4375740 20 client.5691 trim_cache size 0 max 16384 2024-04-16T16:15:25.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.588+0000 7f89c4375740 1 Processor -- start 2024-04-16T16:15:25.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.588+0000 7f89c4375740 1 -- start start 2024-04-16T16:15:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.589+0000 7f89c4375740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908890 0x55b861939e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.589+0000 7f89c4375740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908c70 0x55b86193a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.589+0000 7f89c4375740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b86190cb20 0x55b861935e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.589+0000 7f89c4375740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f89bc204490 con 0x55b861908890 2024-04-16T16:15:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.589+0000 7f89c4375740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89bc153c90 con 0x55b86190cb20 2024-04-16T16:15:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.589+0000 7f89c4375740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f89bc1542e0 con 0x55b861908c70 2024-04-16T16:15:25.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.590+0000 7f89baa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908890 0x55b861939e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.590+0000 7f89baa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908890 0x55b861939e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38762/0 (socket says 172.21.15.45:38762) 2024-04-16T16:15:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.590+0000 7f89baa00640 1 -- 172.21.15.45:0/2543171670 learned_addr learned my addr 172.21.15.45:0/2543171670 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.590+0000 7f89bb400640 1 --2- 172.21.15.45:0/2543171670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b86190cb20 0x55b861935e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.590+0000 7f89b3e00640 1 --2- 172.21.15.45:0/2543171670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908c70 0x55b86193a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.590+0000 7f89baa00640 1 -- 172.21.15.45:0/2543171670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908c70 msgr2=0x55b86193a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.590+0000 7f89baa00640 1 --2- 172.21.15.45:0/2543171670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908c70 0x55b86193a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.590+0000 7f89baa00640 1 -- 172.21.15.45:0/2543171670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b86190cb20 msgr2=0x55b861935e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.590+0000 7f89baa00640 1 --2- 172.21.15.45:0/2543171670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b86190cb20 0x55b861935e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.590+0000 7f89baa00640 1 -- 172.21.15.45:0/2543171670 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8618db100 con 0x55b861908890 2024-04-16T16:15:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.590+0000 7f89bb400640 1 --2- 172.21.15.45:0/2543171670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b86190cb20 0x55b861935e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:15:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.591+0000 7f89baa00640 1 --2- 172.21.15.45:0/2543171670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908890 0x55b861939e70 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f89ac005610 tx=0x7f89ac014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36bc39f0240f4ad1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.591+0000 7f89aa600640 1 -- 172.21.15.45:0/2543171670 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac0159f0 con 0x55b861908890 2024-04-16T16:15:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.591+0000 7f89aa600640 1 -- 172.21.15.45:0/2543171670 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89ac02c070 con 0x55b861908890 2024-04-16T16:15:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.591+0000 7f89aa600640 1 -- 172.21.15.45:0/2543171670 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac015db0 con 0x55b861908890 2024-04-16T16:15:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.592+0000 7f89c4375740 1 -- 172.21.15.45:0/2543171670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908890 msgr2=0x55b861939e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.592+0000 7f89c4375740 1 --2- 172.21.15.45:0/2543171670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908890 0x55b861939e70 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f89ac005610 tx=0x7f89ac014000 comp rx=0 tx=0).stop 2024-04-16T16:15:25.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.592+0000 7f89c4375740 1 -- 172.21.15.45:0/2543171670 shutdown_connections 2024-04-16T16:15:25.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.592+0000 7f89c4375740 1 --2- 172.21.15.45:0/2543171670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b86190cb20 0x55b861935e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.592+0000 7f89c4375740 1 --2- 172.21.15.45:0/2543171670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b861908c70 0x55b86193a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.592+0000 7f89c4375740 1 --2- 172.21.15.45:0/2543171670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908890 0x55b861939e70 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.592+0000 7f89c4375740 1 -- 172.21.15.45:0/2543171670 >> 172.21.15.45:0/2543171670 conn(0x55b8618e1ab0 msgr2=0x55b861926660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:25.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.592+0000 7f89c4375740 1 -- 172.21.15.45:0/2543171670 shutdown_connections 2024-04-16T16:15:25.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.592+0000 7f89c4375740 1 -- 172.21.15.45:0/2543171670 wait complete. 2024-04-16T16:15:25.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.593+0000 7f89c4375740 1 -- start start 2024-04-16T16:15:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.593+0000 7f89c4375740 1 Processor -- start 2024-04-16T16:15:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.594+0000 7f89c4375740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908890 0x55b86194a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.594+0000 7f89c4375740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b861908c70 0x55b8618d7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.594+0000 7f89c4375740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b86190cb20 0x55b861905b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.594+0000 7f89c4375740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b86190b2d0 con 0x55b861908890 2024-04-16T16:15:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.594+0000 7f89c4375740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b86190d820 con 0x55b861908c70 2024-04-16T16:15:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.594+0000 7f89c4375740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b861936c80 con 0x55b86190cb20 2024-04-16T16:15:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.594+0000 7f89c4375740 10 client.0 _ll_register_callbacks cb 0x7f89b4058850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:15:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.594+0000 7f89baa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908890 0x55b86194a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.594+0000 7f89baa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908890 0x55b86194a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38768/0 (socket says 172.21.15.45:38768) 2024-04-16T16:15:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.594+0000 7f89baa00640 1 -- 172.21.15.45:0/1709337229 learned_addr learned my addr 172.21.15.45:0/1709337229 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.594+0000 7f89baa00640 1 -- 172.21.15.45:0/1709337229 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b86190cb20 msgr2=0x55b861905b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:25.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.594+0000 7f89baa00640 1 --2- 172.21.15.45:0/1709337229 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b86190cb20 0x55b861905b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.595+0000 7f89baa00640 1 -- 172.21.15.45:0/1709337229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b861908c70 msgr2=0x55b8618d7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:25.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.595+0000 7f89b3e00640 1 --2- 172.21.15.45:0/1709337229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b861908c70 0x55b8618d7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.595+0000 7f89baa00640 1 --2- 172.21.15.45:0/1709337229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b861908c70 0x55b8618d7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.595+0000 7f89baa00640 1 -- 172.21.15.45:0/1709337229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b861927af0 con 0x55b861908890 2024-04-16T16:15:25.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.595+0000 7f89baa00640 1 --2- 172.21.15.45:0/1709337229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908890 0x55b86194a880 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f89ac015800 tx=0x7f89ac0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50069953025f23ad server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:25.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.595+0000 7f899f400640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:15:25.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.595+0000 7f899f400640 1 -- 172.21.15.45:0/1709337229 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac007020 con 0x55b861908890 2024-04-16T16:15:25.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.595+0000 7f89c4375740 1 -- 172.21.15.45:0/1709337229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89ac02c070 con 0x55b861908890 2024-04-16T16:15:25.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.595+0000 7f899f400640 1 -- 172.21.15.45:0/1709337229 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7f89ac0071c0 con 0x55b861908890 2024-04-16T16:15:25.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.596+0000 7f899f400640 1 -- 172.21.15.45:0/1709337229 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f89ac03e050 con 0x55b861908890 2024-04-16T16:15:25.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.596+0000 7f89c4375740 10 client.5697 fetch_fsmap learned FSMap version 28 2024-04-16T16:15:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.596+0000 7f89c4375740 1 -- 172.21.15.45:0/1709337229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) -- 0x55b861928310 con 0x55b861908890 2024-04-16T16:15:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.596+0000 7f899f400640 1 -- 172.21.15.45:0/1709337229 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89ac02c070 con 0x55b861908890 2024-04-16T16:15:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.596+0000 7f899f400640 1 -- 172.21.15.45:0/1709337229 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 28) ==== 3491+0+0 (secure 0 0 0) 0x7f89ac041020 con 0x55b861908890 2024-04-16T16:15:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.596+0000 7f89c4375740 10 client.5697 fetch_fsmap finished waiting for FSMap version 28 2024-04-16T16:15:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.596+0000 7f89c4375740 10 client.5697 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-16T16:15:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.596+0000 7f89c4375740 20 client.5697 populate_metadata read hostname 'smithi045' 2024-04-16T16:15:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.596+0000 7f89c4375740 1 --2- 172.21.15.45:0/1709337229 >> [v2:172.21.15.45:6832/368835060,v1:172.21.15.45:6833/368835060] conn(0x55b8619247b0 0x55b8619a4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.596+0000 7f89c4375740 4 client.5697 mds_command: new command op to 5646 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-16T16:15:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.596+0000 7f89c4375740 1 -- 172.21.15.45:0/1709337229 --> [v2:172.21.15.45:6832/368835060,v1:172.21.15.45:6833/368835060] -- command(tid 0: {"prefix": "status"}) -- 0x55b8618543e0 con 0x55b8619247b0 2024-04-16T16:15:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.596+0000 7f89b3e00640 1 --2- 172.21.15.45:0/1709337229 >> [v2:172.21.15.45:6832/368835060,v1:172.21.15.45:6833/368835060] conn(0x55b8619247b0 0x55b8619a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.597+0000 7f89b3e00640 1 --2- 172.21.15.45:0/1709337229 >> [v2:172.21.15.45:6832/368835060,v1:172.21.15.45:6833/368835060] conn(0x55b8619247b0 0x55b8619a4710 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2854f299a0b3598a server_cookie=5d0b4d8fadf9958 in_seq=0 out_seq=0 2024-04-16T16:15:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.597+0000 7f899f400640 10 client.5697 ms_handle_connect on v2:172.21.15.45:6832/368835060 2024-04-16T16:15:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.597+0000 7f899f400640 1 -- 172.21.15.45:0/1709337229 <== mds.0 v2:172.21.15.45:6832/368835060 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x55b8618543e0 con 0x55b8619247b0 2024-04-16T16:15:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.597+0000 7f899f400640 10 client.5697 handle_command_reply: tid=0 2024-04-16T16:15:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.597+0000 7f89c4375740 1 client.5697 shutdown 2024-04-16T16:15:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.597+0000 7f89c4375740 1 -- 172.21.15.45:0/1709337229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908890 msgr2=0x55b86194a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.597+0000 7f89c4375740 1 --2- 172.21.15.45:0/1709337229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908890 0x55b86194a880 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f89ac015800 tx=0x7f89ac0036f0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.597+0000 7f89c4375740 1 -- 172.21.15.45:0/1709337229 shutdown_connections 2024-04-16T16:15:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.597+0000 7f89c4375740 1 --2- 172.21.15.45:0/1709337229 >> [v2:172.21.15.45:6832/368835060,v1:172.21.15.45:6833/368835060] conn(0x55b8619247b0 0x55b8619a4710 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.597+0000 7f89c4375740 1 --2- 172.21.15.45:0/1709337229 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b86190cb20 0x55b861905b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.597+0000 7f89c4375740 1 --2- 172.21.15.45:0/1709337229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b861908c70 0x55b8618d7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.597+0000 7f89c4375740 1 --2- 172.21.15.45:0/1709337229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b861908890 0x55b86194a880 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.597+0000 7f899f400640 0 client.5697 ms_handle_reset on v2:172.21.15.45:6832/368835060 2024-04-16T16:15:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.597+0000 7f89c4375740 1 -- 172.21.15.45:0/1709337229 >> 172.21.15.45:0/1709337229 conn(0x55b8618e1ab0 msgr2=0x55b8618da790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.598+0000 7f89c4375740 1 -- 172.21.15.45:0/1709337229 shutdown_connections 2024-04-16T16:15:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.598+0000 7f89c4375740 1 -- 172.21.15.45:0/1709337229 wait complete. 2024-04-16T16:15:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.598+0000 7f89c4375740 20 client.5697 trim_cache size 0 max 16384 2024-04-16T16:15:25.600 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:15:25.600 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "748c4b85-0190-4c59-80ad-098a6a5007c0", 2024-04-16T16:15:25.600 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-04-16T16:15:25.600 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 5646, 2024-04-16T16:15:25.600 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-04-16T16:15:25.600 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-04-16T16:15:25.600 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "fs1", 2024-04-16T16:15:25.600 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 4.8950503799999998, 2024-04-16T16:15:25.600 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 28, 2024-04-16T16:15:25.600 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 46, 2024-04-16T16:15:25.600 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 46, 2024-04-16T16:15:25.601 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 4.9107735119999996 2024-04-16T16:15:25.601 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:15:25.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.598+0000 7f89bbe00640 1 -- 172.21.15.45:0/948470784 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f899806ca90 msgr2=0x7f899806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:25.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.598+0000 7f89bbe00640 1 --2- 172.21.15.45:0/948470784 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f899806ca90 0x7f899806ef50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f89ac004ea0 tx=0x7f89ac009040 comp rx=0 tx=0).stop 2024-04-16T16:15:25.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.598+0000 7f89bbe00640 1 -- 172.21.15.45:0/948470784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc0ce710 msgr2=0x7f89bc155140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:25.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.598+0000 7f89bbe00640 1 --2- 172.21.15.45:0/948470784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc0ce710 0x7f89bc155140 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f89b400c730 tx=0x7f89b400cbf0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.599+0000 7f89bbe00640 1 -- 172.21.15.45:0/948470784 shutdown_connections 2024-04-16T16:15:25.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.599+0000 7f89bbe00640 1 --2- 172.21.15.45:0/948470784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc0ce710 0x7f89bc155140 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.599+0000 7f89bbe00640 1 --2- 172.21.15.45:0/948470784 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f899806ca90 0x7f899806ef50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.599+0000 7f89bbe00640 1 --2- 172.21.15.45:0/948470784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89bc0b1ce0 0x7f89bc14ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.599+0000 7f89bbe00640 1 --2- 172.21.15.45:0/948470784 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89bc06f410 0x7f89bc15ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.599+0000 7f89bbe00640 1 -- 172.21.15.45:0/948470784 >> 172.21.15.45:0/948470784 conn(0x7f89bc06b930 msgr2=0x7f89bc1657f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:25.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.599+0000 7f89bbe00640 1 -- 172.21.15.45:0/948470784 shutdown_connections 2024-04-16T16:15:25.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.599+0000 7f89bbe00640 1 -- 172.21.15.45:0/948470784 wait complete. 2024-04-16T16:15:25.615 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=fs1_data' 2024-04-16T16:15:25.703 INFO:tasks.ceph.mds.a.smithi045.stderr:2024-04-16T16:15:25.704+0000 7fd524e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:15:25.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.845+0000 7fba02400640 1 Processor -- start 2024-04-16T16:15:25.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.845+0000 7fba02400640 1 -- start start 2024-04-16T16:15:25.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.847+0000 7fba02400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb9fc1452e0 0x7fb9fc06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.847+0000 7fba02400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9fc06e920 0x7fb9fc0ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.847+0000 7fba02400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb9fc0ceaf0 0x7fb9fc160270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.847+0000 7fba02400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb9fc072f00 con 0x7fb9fc1452e0 2024-04-16T16:15:25.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.847+0000 7fba02400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb9fc06ed60 con 0x7fb9fc0ceaf0 2024-04-16T16:15:25.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.847+0000 7fba02400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb9fc071e60 con 0x7fb9fc06e920 2024-04-16T16:15:25.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.848+0000 7fba01000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb9fc1452e0 0x7fb9fc06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.848+0000 7fba01000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb9fc1452e0 0x7fb9fc06e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38782/0 (socket says 172.21.15.45:38782) 2024-04-16T16:15:25.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.848+0000 7fba01000640 1 -- 172.21.15.45:0/1068945684 learned_addr learned my addr 172.21.15.45:0/1068945684 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:25.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.848+0000 7fb9fbe00640 1 --2- 172.21.15.45:0/1068945684 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9fc06e920 0x7fb9fc0ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.848+0000 7fba01a00640 1 --2- 172.21.15.45:0/1068945684 >> v2:172.21.15.103:3300/0 conn(0x7fb9fc0ceaf0 0x7fb9fc160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.848+0000 7fba01000640 1 -- 172.21.15.45:0/1068945684 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9fc06e920 msgr2=0x7fb9fc0ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:25.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.848+0000 7fba01000640 1 --2- 172.21.15.45:0/1068945684 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9fc06e920 0x7fb9fc0ce5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.848+0000 7fba01000640 1 -- 172.21.15.45:0/1068945684 >> v2:172.21.15.103:3300/0 conn(0x7fb9fc0ceaf0 msgr2=0x7fb9fc160270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:25.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.848+0000 7fba01000640 1 --2- 172.21.15.45:0/1068945684 >> v2:172.21.15.103:3300/0 conn(0x7fb9fc0ceaf0 0x7fb9fc160270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.848+0000 7fba01000640 1 -- 172.21.15.45:0/1068945684 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9fc072140 con 0x7fb9fc1452e0 2024-04-16T16:15:25.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.850+0000 7fba01000640 1 --2- 172.21.15.45:0/1068945684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc1452e0 0x7fb9fc06e3e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc071c20 tx=0x7fb9f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8eac409b630e5d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:25.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.850+0000 7fb9fb400640 1 -- 172.21.15.45:0/1068945684 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0034070 con 0x7fb9fc1452e0 2024-04-16T16:15:25.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.850+0000 7fb9fb400640 1 -- 172.21.15.45:0/1068945684 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9f0030860 con 0x7fb9fc1452e0 2024-04-16T16:15:25.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.850+0000 7fb9fb400640 1 -- 172.21.15.45:0/1068945684 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0030c30 con 0x7fb9fc1452e0 2024-04-16T16:15:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.851+0000 7fba02400640 1 -- 172.21.15.45:0/1068945684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc1452e0 msgr2=0x7fb9fc06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.851+0000 7fba02400640 1 --2- 172.21.15.45:0/1068945684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc1452e0 0x7fb9fc06e3e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc071c20 tx=0x7fb9f002b9a0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.851+0000 7fba02400640 1 -- 172.21.15.45:0/1068945684 shutdown_connections 2024-04-16T16:15:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.851+0000 7fba02400640 1 --2- 172.21.15.45:0/1068945684 >> v2:172.21.15.103:3300/0 conn(0x7fb9fc0ceaf0 0x7fb9fc160270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.851+0000 7fba02400640 1 --2- 172.21.15.45:0/1068945684 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9fc06e920 0x7fb9fc0ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.851+0000 7fba02400640 1 --2- 172.21.15.45:0/1068945684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc1452e0 0x7fb9fc06e3e0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.851+0000 7fba02400640 1 -- 172.21.15.45:0/1068945684 >> 172.21.15.45:0/1068945684 conn(0x7fb9fc0b3bb0 msgr2=0x7fb9fc0b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:25.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.851+0000 7fba02400640 1 -- 172.21.15.45:0/1068945684 shutdown_connections 2024-04-16T16:15:25.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.851+0000 7fba02400640 1 -- 172.21.15.45:0/1068945684 wait complete. 2024-04-16T16:15:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.853+0000 7fba02400640 1 Processor -- start 2024-04-16T16:15:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.853+0000 7fba02400640 1 -- start start 2024-04-16T16:15:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.854+0000 7fba02400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9fc06e920 0x7fb9fc15bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.854+0000 7fba02400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9fc0ceaf0 0x7fb9fc0abb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.854+0000 7fba02400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc1452e0 0x7fb9fc0ac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.854+0000 7fba02400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc0729e0 con 0x7fb9fc1452e0 2024-04-16T16:15:25.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.854+0000 7fba02400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc072700 con 0x7fb9fc06e920 2024-04-16T16:15:25.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.854+0000 7fba02400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb9fc0cef60 con 0x7fb9fc0ceaf0 2024-04-16T16:15:25.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.855+0000 7fba01a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc1452e0 0x7fb9fc0ac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.855+0000 7fba01a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc1452e0 0x7fb9fc0ac0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38798/0 (socket says 172.21.15.45:38798) 2024-04-16T16:15:25.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.855+0000 7fba01a00640 1 -- 172.21.15.45:0/3128023675 learned_addr learned my addr 172.21.15.45:0/3128023675 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:25.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.855+0000 7fb9fbe00640 1 --2- 172.21.15.45:0/3128023675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9fc0ceaf0 0x7fb9fc0abb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.855+0000 7fba01000640 1 --2- 172.21.15.45:0/3128023675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9fc06e920 0x7fb9fc15bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.855+0000 7fba01a00640 1 -- 172.21.15.45:0/3128023675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9fc0ceaf0 msgr2=0x7fb9fc0abb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:25.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.855+0000 7fba01a00640 1 --2- 172.21.15.45:0/3128023675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9fc0ceaf0 0x7fb9fc0abb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.855+0000 7fba01a00640 1 -- 172.21.15.45:0/3128023675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9fc06e920 msgr2=0x7fb9fc15bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.855+0000 7fba01a00640 1 --2- 172.21.15.45:0/3128023675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9fc06e920 0x7fb9fc15bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.855+0000 7fba01a00640 1 -- 172.21.15.45:0/3128023675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9fc0aeb60 con 0x7fb9fc1452e0 2024-04-16T16:15:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.855+0000 7fba01a00640 1 --2- 172.21.15.45:0/3128023675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc1452e0 0x7fb9fc0ac0a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4007e40 tx=0x7fb9f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.856+0000 7fb9f8c00640 1 -- 172.21.15.45:0/3128023675 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4014620 con 0x7fb9fc1452e0 2024-04-16T16:15:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.856+0000 7fb9f8c00640 1 -- 172.21.15.45:0/3128023675 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9f4019070 con 0x7fb9fc1452e0 2024-04-16T16:15:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.856+0000 7fb9f8c00640 1 -- 172.21.15.45:0/3128023675 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4014a20 con 0x7fb9fc1452e0 2024-04-16T16:15:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.856+0000 7fba02400640 1 -- 172.21.15.45:0/3128023675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9fc0aee40 con 0x7fb9fc1452e0 2024-04-16T16:15:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.856+0000 7fba02400640 1 -- 172.21.15.45:0/3128023675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9fc0add30 con 0x7fb9fc1452e0 2024-04-16T16:15:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.857+0000 7fba02400640 1 -- 172.21.15.45:0/3128023675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9fc075780 con 0x7fb9fc1452e0 2024-04-16T16:15:25.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.858+0000 7fb9f8c00640 1 -- 172.21.15.45:0/3128023675 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fb9f4014bc0 con 0x7fb9fc1452e0 2024-04-16T16:15:25.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.859+0000 7fb9f8c00640 1 --2- 172.21.15.45:0/3128023675 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb9dc06ca60 0x7fb9dc06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:25.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.859+0000 7fb9f8c00640 1 -- 172.21.15.45:0/3128023675 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7fb9f408c7e0 con 0x7fb9fc1452e0 2024-04-16T16:15:25.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.860+0000 7fba01000640 1 --2- 172.21.15.45:0/3128023675 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb9dc06ca60 0x7fb9dc06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:25.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.861+0000 7fba01000640 1 --2- 172.21.15.45:0/3128023675 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb9dc06ca60 0x7fb9dc06ef20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0036000 tx=0x7fb9f003b040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:25.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:25.864+0000 7fb9f8c00640 1 -- 172.21.15.45:0/3128023675 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb9f405a8c0 con 0x7fb9fc1452e0 2024-04-16T16:15:26.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.145+0000 7fba02400640 1 -- 172.21.15.45:0/3128023675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs1_data"]} v 0) -- 0x7fb9fc0af360 con 0x7fb9fc1452e0 2024-04-16T16:15:26.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.151+0000 7fb9f8c00640 1 -- 172.21.15.45:0/3128023675 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs1_data"]}]=0 updated caps for client.0 v12) ==== 181+0+138 (secure 0 0 0) 0x7fb9f405a1c0 con 0x7fb9fc1452e0 2024-04-16T16:15:26.150 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-04-16T16:15:26.150 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ== 2024-04-16T16:15:26.150 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:15:26.150 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:15:26.151 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw pool=fs1_data" 2024-04-16T16:15:26.151 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2024-04-16T16:15:26.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.154+0000 7fba02400640 1 -- 172.21.15.45:0/3128023675 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb9dc06ca60 msgr2=0x7fb9dc06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:26.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.154+0000 7fba02400640 1 --2- 172.21.15.45:0/3128023675 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb9dc06ca60 0x7fb9dc06ef20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0036000 tx=0x7fb9f003b040 comp rx=0 tx=0).stop 2024-04-16T16:15:26.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.154+0000 7fba02400640 1 -- 172.21.15.45:0/3128023675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc1452e0 msgr2=0x7fb9fc0ac0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:26.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.154+0000 7fba02400640 1 --2- 172.21.15.45:0/3128023675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc1452e0 0x7fb9fc0ac0a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4007e40 tx=0x7fb9f400e4b0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.154+0000 7fba02400640 1 -- 172.21.15.45:0/3128023675 shutdown_connections 2024-04-16T16:15:26.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.154+0000 7fba02400640 1 --2- 172.21.15.45:0/3128023675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc1452e0 0x7fb9fc0ac0a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.154+0000 7fba02400640 1 --2- 172.21.15.45:0/3128023675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9fc0ceaf0 0x7fb9fc0abb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.154+0000 7fba02400640 1 --2- 172.21.15.45:0/3128023675 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb9dc06ca60 0x7fb9dc06ef20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.154+0000 7fba02400640 1 --2- 172.21.15.45:0/3128023675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9fc06e920 0x7fb9fc15bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.154+0000 7fba02400640 1 -- 172.21.15.45:0/3128023675 >> 172.21.15.45:0/3128023675 conn(0x7fb9fc0b3bb0 msgr2=0x7fb9fc0b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:26.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.154+0000 7fba02400640 1 -- 172.21.15.45:0/3128023675 shutdown_connections 2024-04-16T16:15:26.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.154+0000 7fba02400640 1 -- 172.21.15.45:0/3128023675 wait complete. 2024-04-16T16:15:26.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=fs1_data' 2024-04-16T16:15:26.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.403+0000 7f2ffc600640 1 Processor -- start 2024-04-16T16:15:26.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.403+0000 7f2ffc600640 1 -- start start 2024-04-16T16:15:26.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.404+0000 7f2ffc600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ff415c7c0 0x7f2ff4072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:26.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.404+0000 7f2ffc600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2ff40b4930 0x7f2ff40d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:26.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.405+0000 7f2ffc600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2ff40d0f10 0x7f2ff406f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:26.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.405+0000 7f2ffc600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2ff40b52e0 con 0x7f2ff40b4930 2024-04-16T16:15:26.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.405+0000 7f2ffc600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2ff40d0260 con 0x7f2ff40d0f10 2024-04-16T16:15:26.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.405+0000 7f2ffc600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2ff40d1380 con 0x7f2ff415c7c0 2024-04-16T16:15:26.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.405+0000 7f2ff9200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2ff40b4930 0x7f2ff40d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:26.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.405+0000 7f2ff9200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2ff40b4930 0x7f2ff40d09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38802/0 (socket says 172.21.15.45:38802) 2024-04-16T16:15:26.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.405+0000 7f2ff9200640 1 -- 172.21.15.45:0/1769841698 learned_addr learned my addr 172.21.15.45:0/1769841698 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:26.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.405+0000 7f2ffa600640 1 --2- 172.21.15.45:0/1769841698 >> v2:172.21.15.103:3300/0 conn(0x7f2ff40d0f10 0x7f2ff406f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:26.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.405+0000 7f2ff9c00640 1 --2- 172.21.15.45:0/1769841698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ff415c7c0 0x7f2ff4072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:26.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.405+0000 7f2ff9200640 1 -- 172.21.15.45:0/1769841698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ff415c7c0 msgr2=0x7f2ff4072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:26.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.405+0000 7f2ff9200640 1 --2- 172.21.15.45:0/1769841698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ff415c7c0 0x7f2ff4072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.405+0000 7f2ff9200640 1 -- 172.21.15.45:0/1769841698 >> v2:172.21.15.103:3300/0 conn(0x7f2ff40d0f10 msgr2=0x7f2ff406f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:26.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.405+0000 7f2ff9200640 1 --2- 172.21.15.45:0/1769841698 >> v2:172.21.15.103:3300/0 conn(0x7f2ff40d0f10 0x7f2ff406f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.405+0000 7f2ff9200640 1 -- 172.21.15.45:0/1769841698 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff40b5530 con 0x7f2ff40b4930 2024-04-16T16:15:26.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.405+0000 7f2ff9c00640 1 --2- 172.21.15.45:0/1769841698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ff415c7c0 0x7f2ff4072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:26.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.407+0000 7f2ff9200640 1 --2- 172.21.15.45:0/1769841698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ff40b4930 0x7f2ff40d09d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40afd50 tx=0x7f2fe402b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102b286cc229c8bf server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:26.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.407+0000 7f2ff3e00640 1 -- 172.21.15.45:0/1769841698 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4031760 con 0x7f2ff40b4930 2024-04-16T16:15:26.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.407+0000 7f2ff3e00640 1 -- 172.21.15.45:0/1769841698 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fe4037070 con 0x7f2ff40b4930 2024-04-16T16:15:26.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.407+0000 7f2ff3e00640 1 -- 172.21.15.45:0/1769841698 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4031b20 con 0x7f2ff40b4930 2024-04-16T16:15:26.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.408+0000 7f2ffc600640 1 -- 172.21.15.45:0/1769841698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ff40b4930 msgr2=0x7f2ff40d09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:26.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.408+0000 7f2ffc600640 1 --2- 172.21.15.45:0/1769841698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ff40b4930 0x7f2ff40d09d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40afd50 tx=0x7f2fe402b800 comp rx=0 tx=0).stop 2024-04-16T16:15:26.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.408+0000 7f2ffc600640 1 -- 172.21.15.45:0/1769841698 shutdown_connections 2024-04-16T16:15:26.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.408+0000 7f2ffc600640 1 --2- 172.21.15.45:0/1769841698 >> v2:172.21.15.103:3300/0 conn(0x7f2ff40d0f10 0x7f2ff406f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.408+0000 7f2ffc600640 1 --2- 172.21.15.45:0/1769841698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ff40b4930 0x7f2ff40d09d0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.408+0000 7f2ffc600640 1 --2- 172.21.15.45:0/1769841698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ff415c7c0 0x7f2ff4072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.408+0000 7f2ffc600640 1 -- 172.21.15.45:0/1769841698 >> 172.21.15.45:0/1769841698 conn(0x7f2ff40b3970 msgr2=0x7f2ff40b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:26.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.408+0000 7f2ffc600640 1 -- 172.21.15.45:0/1769841698 shutdown_connections 2024-04-16T16:15:26.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.408+0000 7f2ffc600640 1 -- 172.21.15.45:0/1769841698 wait complete. 2024-04-16T16:15:26.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.410+0000 7f2ffc600640 1 Processor -- start 2024-04-16T16:15:26.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.410+0000 7f2ffc600640 1 -- start start 2024-04-16T16:15:26.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.411+0000 7f2ffc600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ff40b4930 0x7f2ff4157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:26.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.411+0000 7f2ffc600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ff40d0f10 0x7f2ff4143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:26.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.411+0000 7f2ffc600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ff415c7c0 0x7f2ff4146830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:26.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.411+0000 7f2ffc600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff40d04b0 con 0x7f2ff415c7c0 2024-04-16T16:15:26.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.411+0000 7f2ffc600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff4073350 con 0x7f2ff40b4930 2024-04-16T16:15:26.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.411+0000 7f2ffc600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2ff406f6a0 con 0x7f2ff40d0f10 2024-04-16T16:15:26.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.411+0000 7f2ffa600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ff415c7c0 0x7f2ff4146830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:26.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.412+0000 7f2ffa600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ff415c7c0 0x7f2ff4146830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38814/0 (socket says 172.21.15.45:38814) 2024-04-16T16:15:26.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.412+0000 7f2ffa600640 1 -- 172.21.15.45:0/1922945925 learned_addr learned my addr 172.21.15.45:0/1922945925 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:26.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.412+0000 7f2ff9200640 1 --2- 172.21.15.45:0/1922945925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ff40d0f10 0x7f2ff4143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:26.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.412+0000 7f2ff9c00640 1 --2- 172.21.15.45:0/1922945925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ff40b4930 0x7f2ff4157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:26.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.412+0000 7f2ffa600640 1 -- 172.21.15.45:0/1922945925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ff40d0f10 msgr2=0x7f2ff4143ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:26.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.412+0000 7f2ffa600640 1 --2- 172.21.15.45:0/1922945925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ff40d0f10 0x7f2ff4143ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.412+0000 7f2ffa600640 1 -- 172.21.15.45:0/1922945925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ff40b4930 msgr2=0x7f2ff4157ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:26.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.412+0000 7f2ffa600640 1 --2- 172.21.15.45:0/1922945925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ff40b4930 0x7f2ff4157ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.412+0000 7f2ffa600640 1 -- 172.21.15.45:0/1922945925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff4154b10 con 0x7f2ff415c7c0 2024-04-16T16:15:26.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.412+0000 7f2ffa600640 1 --2- 172.21.15.45:0/1922945925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ff415c7c0 0x7f2ff4146830 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2fec007e40 tx=0x7f2fec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:26.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.413+0000 7f2ff1600640 1 -- 172.21.15.45:0/1922945925 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec014620 con 0x7f2ff415c7c0 2024-04-16T16:15:26.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.413+0000 7f2ff1600640 1 -- 172.21.15.45:0/1922945925 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fec019070 con 0x7f2ff415c7c0 2024-04-16T16:15:26.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.413+0000 7f2ff1600640 1 -- 172.21.15.45:0/1922945925 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec014a20 con 0x7f2ff415c7c0 2024-04-16T16:15:26.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.413+0000 7f2ffc600640 1 -- 172.21.15.45:0/1922945925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff4144840 con 0x7f2ff415c7c0 2024-04-16T16:15:26.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.413+0000 7f2ffc600640 1 -- 172.21.15.45:0/1922945925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ff4144e00 con 0x7f2ff415c7c0 2024-04-16T16:15:26.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.414+0000 7f2ffc600640 1 -- 172.21.15.45:0/1922945925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ff4075780 con 0x7f2ff415c7c0 2024-04-16T16:15:26.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.415+0000 7f2ff1600640 1 -- 172.21.15.45:0/1922945925 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f2fec014bc0 con 0x7f2ff415c7c0 2024-04-16T16:15:26.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.416+0000 7f2ff1600640 1 --2- 172.21.15.45:0/1922945925 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2fd806ca60 0x7f2fd806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:26.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.416+0000 7f2ff1600640 1 -- 172.21.15.45:0/1922945925 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7f2fec08c860 con 0x7f2ff415c7c0 2024-04-16T16:15:26.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.417+0000 7f2ff9c00640 1 --2- 172.21.15.45:0/1922945925 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2fd806ca60 0x7f2fd806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:26.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.418+0000 7f2ff9c00640 1 --2- 172.21.15.45:0/1922945925 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2fd806ca60 0x7f2fd806ef20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800a8d0 tx=0x7f2fe8008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:26.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.420+0000 7f2ff1600640 1 -- 172.21.15.45:0/1922945925 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2fec05a9d0 con 0x7f2ff415c7c0 2024-04-16T16:15:26.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.705+0000 7f2ffc600640 1 -- 172.21.15.45:0/1922945925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs1_data"]} v 0) -- 0x7f2ff406d210 con 0x7f2ff415c7c0 2024-04-16T16:15:26.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-04-16T16:15:26.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.709+0000 7f2ff1600640 1 -- 172.21.15.45:0/1922945925 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs1_data"]}]=0 updated caps for client.1 v13) ==== 181+0+138 (secure 0 0 0) 0x7f2fec05a2d0 con 0x7f2ff415c7c0 2024-04-16T16:15:26.708 INFO:teuthology.orchestra.run.smithi045.stdout:[client.1] 2024-04-16T16:15:26.708 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mYpVUGhAAYdR+40O6zbWqgbI55F0FYA== 2024-04-16T16:15:26.709 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:15:26.709 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:15:26.709 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw pool=fs1_data" 2024-04-16T16:15:26.709 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.1 2024-04-16T16:15:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.713+0000 7f2ffc600640 1 -- 172.21.15.45:0/1922945925 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2fd806ca60 msgr2=0x7f2fd806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.713+0000 7f2ffc600640 1 --2- 172.21.15.45:0/1922945925 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2fd806ca60 0x7f2fd806ef20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800a8d0 tx=0x7f2fe8008040 comp rx=0 tx=0).stop 2024-04-16T16:15:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.713+0000 7f2ffc600640 1 -- 172.21.15.45:0/1922945925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ff415c7c0 msgr2=0x7f2ff4146830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:26.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.713+0000 7f2ffc600640 1 --2- 172.21.15.45:0/1922945925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ff415c7c0 0x7f2ff4146830 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2fec007e40 tx=0x7f2fec00e4b0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.713+0000 7f2ffc600640 1 -- 172.21.15.45:0/1922945925 shutdown_connections 2024-04-16T16:15:26.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.713+0000 7f2ffc600640 1 --2- 172.21.15.45:0/1922945925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ff415c7c0 0x7f2ff4146830 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.713+0000 7f2ffc600640 1 --2- 172.21.15.45:0/1922945925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ff40d0f10 0x7f2ff4143ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.713+0000 7f2ffc600640 1 --2- 172.21.15.45:0/1922945925 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2fd806ca60 0x7f2fd806ef20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.713+0000 7f2ffc600640 1 --2- 172.21.15.45:0/1922945925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ff40b4930 0x7f2ff4157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.713+0000 7f2ffc600640 1 -- 172.21.15.45:0/1922945925 >> 172.21.15.45:0/1922945925 conn(0x7f2ff40b3970 msgr2=0x7f2ff4072a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:26.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.713+0000 7f2ffc600640 1 -- 172.21.15.45:0/1922945925 shutdown_connections 2024-04-16T16:15:26.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.713+0000 7f2ffc600640 1 -- 172.21.15.45:0/1922945925 wait complete. 2024-04-16T16:15:26.734 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=fs1_data' 2024-04-16T16:15:26.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.954+0000 7f3fac600640 1 Processor -- start 2024-04-16T16:15:26.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.955+0000 7f3fac600640 1 -- start start 2024-04-16T16:15:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.956+0000 7f3fac600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fa415ecc0 0x7f3fa41626a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.956+0000 7f3fac600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fa40d2b10 con 0x7f3fa4162be0 2024-04-16T16:15:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.956+0000 7f3fac600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fa4163210 con 0x7f3fa406d620 2024-04-16T16:15:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.956+0000 7f3fac600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3fa41634f0 con 0x7f3fa415ecc0 2024-04-16T16:15:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.957+0000 7f3fa9200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3fa4162be0 0x7f3fa406cf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50060/0 (socket says 172.21.15.45:50060) 2024-04-16T16:15:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.957+0000 7f3fa9200640 1 -- 172.21.15.45:0/2686485513 learned_addr learned my addr 172.21.15.45:0/2686485513 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.957+0000 7f3fa3e00640 1 -- 172.21.15.45:0/2686485513 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1888013859 0 0) 0x7f3fa40d2b10 con 0x7f3fa4162be0 2024-04-16T16:15:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.957+0000 7f3fa3e00640 1 -- 172.21.15.45:0/2686485513 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f88003520 con 0x7f3fa4162be0 2024-04-16T16:15:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.957+0000 7f3fa9c00640 1 --2- 172.21.15.45:0/2686485513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fa415ecc0 0x7f3fa41626a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3fa3e00640 1 -- 172.21.15.45:0/2686485513 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2261281459 0 0) 0x7f3f88003520 con 0x7f3fa4162be0 2024-04-16T16:15:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3fa3e00640 1 -- 172.21.15.45:0/2686485513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fa415ecc0 msgr2=0x7f3fa41626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3fa3e00640 1 --2- 172.21.15.45:0/2686485513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fa415ecc0 0x7f3fa41626a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3fa3e00640 1 -- 172.21.15.45:0/2686485513 >> v1:172.21.15.103:6789/0 conn(0x7f3fa406d620 legacy=0x7f3fa40d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3fa3e00640 1 -- 172.21.15.45:0/2686485513 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa40d2dd0 con 0x7f3fa4162be0 2024-04-16T16:15:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3fa3e00640 1 -- 172.21.15.45:0/2686485513 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f3f90001670 con 0x7f3fa4162be0 2024-04-16T16:15:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3fa9c00640 1 --2- 172.21.15.45:0/2686485513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fa415ecc0 0x7f3fa41626a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:15:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3fa3e00640 1 -- 172.21.15.45:0/2686485513 >> v1:172.21.15.45:6789/0 conn(0x7f3fa4162be0 legacy=0x7f3fa406cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3fa3e00640 1 --2- 172.21.15.45:0/2686485513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f880039e0 0x7f3f88003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3fa3e00640 1 --2- 172.21.15.45:0/2686485513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fa415ecc0 0x7f3f88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3fa3e00640 1 --2- 172.21.15.45:0/2686485513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f88004d40 0x7f3f880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3fa3e00640 1 -- 172.21.15.45:0/2686485513 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa41634f0 con 0x7f3f88004d40 2024-04-16T16:15:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3fa3e00640 1 -- 172.21.15.45:0/2686485513 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8800bb60 con 0x7f3f880039e0 2024-04-16T16:15:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3fa3e00640 1 -- 172.21.15.45:0/2686485513 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3f8800bd60 con 0x7f3fa415ecc0 2024-04-16T16:15:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3faa600640 1 --2- 172.21.15.45:0/2686485513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f88004d40 0x7f3f880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.958+0000 7f3fa9c00640 1 --2- 172.21.15.45:0/2686485513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f880039e0 0x7f3f88003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.959+0000 7f3fa9200640 1 --2- 172.21.15.45:0/2686485513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fa415ecc0 0x7f3f88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.959+0000 7f3faa600640 1 -- 172.21.15.45:0/2686485513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fa415ecc0 msgr2=0x7f3f88004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.959+0000 7f3faa600640 1 --2- 172.21.15.45:0/2686485513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fa415ecc0 0x7f3f88004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.959+0000 7f3faa600640 1 -- 172.21.15.45:0/2686485513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f880039e0 msgr2=0x7f3f88003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.959+0000 7f3faa600640 1 --2- 172.21.15.45:0/2686485513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f880039e0 0x7f3f88003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.959+0000 7f3faa600640 1 -- 172.21.15.45:0/2686485513 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f88003520 con 0x7f3f88004d40 2024-04-16T16:15:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.959+0000 7f3fa9c00640 1 --2- 172.21.15.45:0/2686485513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f880039e0 0x7f3f88003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.959+0000 7f3fa9200640 1 --2- 172.21.15.45:0/2686485513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fa415ecc0 0x7f3f88004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.960+0000 7f3faa600640 1 --2- 172.21.15.45:0/2686485513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f88004d40 0x7f3f880051c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c001720 tx=0x7f3f88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735dab8847dc2d8a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.960+0000 7f3fa3e00640 1 -- 172.21.15.45:0/2686485513 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c001370 con 0x7f3f88004d40 2024-04-16T16:15:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.960+0000 7f3fa3e00640 1 -- 172.21.15.45:0/2686485513 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f9c0052d0 con 0x7f3f88004d40 2024-04-16T16:15:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.960+0000 7f3fa3e00640 1 -- 172.21.15.45:0/2686485513 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c003830 con 0x7f3f88004d40 2024-04-16T16:15:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.961+0000 7f3fac600640 1 -- 172.21.15.45:0/2686485513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f88004d40 msgr2=0x7f3f880051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.961+0000 7f3fac600640 1 --2- 172.21.15.45:0/2686485513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f88004d40 0x7f3f880051c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c001720 tx=0x7f3f88005830 comp rx=0 tx=0).stop 2024-04-16T16:15:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.961+0000 7f3faa600640 1 -- 172.21.15.45:0/2686485513 reap_dead start 2024-04-16T16:15:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.961+0000 7f3fac600640 1 -- 172.21.15.45:0/2686485513 shutdown_connections 2024-04-16T16:15:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.961+0000 7f3fac600640 1 -- 172.21.15.45:0/2686485513 >> 172.21.15.45:0/2686485513 conn(0x7f3fa4073690 msgr2=0x7f3fa40d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:26.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.961+0000 7f3fac600640 1 -- 172.21.15.45:0/2686485513 shutdown_connections 2024-04-16T16:15:26.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.961+0000 7f3fac600640 1 -- 172.21.15.45:0/2686485513 wait complete. 2024-04-16T16:15:26.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.963+0000 7f3fac600640 1 Processor -- start 2024-04-16T16:15:26.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.963+0000 7f3fac600640 1 -- start start 2024-04-16T16:15:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.964+0000 7f3fac600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fa40ce2d0 0x7f3fa4160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.964+0000 7f3fac600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fa40cdb90 0x7f3fa4160950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.964+0000 7f3fac600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fa40ce6b0 0x7f3fa41e84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.964+0000 7f3fac600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa41626a0 con 0x7f3fa40ce2d0 2024-04-16T16:15:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.964+0000 7f3fac600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa40ba540 con 0x7f3fa40cdb90 2024-04-16T16:15:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.964+0000 7f3fac600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3fa4162820 con 0x7f3fa40ce6b0 2024-04-16T16:15:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.964+0000 7f3fa9c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fa40ce2d0 0x7f3fa4160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.964+0000 7f3fa9c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fa40ce2d0 0x7f3fa4160410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38822/0 (socket says 172.21.15.45:38822) 2024-04-16T16:15:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.964+0000 7f3fa9c00640 1 -- 172.21.15.45:0/1724166459 learned_addr learned my addr 172.21.15.45:0/1724166459 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.965+0000 7f3fa9c00640 1 -- 172.21.15.45:0/1724166459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fa40ce6b0 msgr2=0x7f3fa41e84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.965+0000 7f3fa9c00640 1 --2- 172.21.15.45:0/1724166459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fa40ce6b0 0x7f3fa41e84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.965+0000 7f3fa9c00640 1 -- 172.21.15.45:0/1724166459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fa40cdb90 msgr2=0x7f3fa4160950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.965+0000 7f3fa9c00640 1 --2- 172.21.15.45:0/1724166459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fa40cdb90 0x7f3fa4160950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.965+0000 7f3fa9c00640 1 -- 172.21.15.45:0/1724166459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa4146670 con 0x7f3fa40ce2d0 2024-04-16T16:15:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.965+0000 7f3fa9c00640 1 --2- 172.21.15.45:0/1724166459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fa40ce2d0 0x7f3fa4160410 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009fc0 tx=0x7f3f9400e5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.965+0000 7f3fa1600640 1 -- 172.21.15.45:0/1724166459 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f94015610 con 0x7f3fa40ce2d0 2024-04-16T16:15:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.965+0000 7f3fa1600640 1 -- 172.21.15.45:0/1724166459 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f9401a070 con 0x7f3fa40ce2d0 2024-04-16T16:15:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.965+0000 7f3fa1600640 1 -- 172.21.15.45:0/1724166459 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f940159b0 con 0x7f3fa40ce2d0 2024-04-16T16:15:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.965+0000 7f3fac600640 1 -- 172.21.15.45:0/1724166459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fa41610f0 con 0x7f3fa40ce2d0 2024-04-16T16:15:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.965+0000 7f3fac600640 1 -- 172.21.15.45:0/1724166459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fa41616b0 con 0x7f3fa40ce2d0 2024-04-16T16:15:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.967+0000 7f3fac600640 1 -- 172.21.15.45:0/1724166459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fa4075780 con 0x7f3fa40ce2d0 2024-04-16T16:15:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.968+0000 7f3fa1600640 1 -- 172.21.15.45:0/1724166459 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f3f94012030 con 0x7f3fa40ce2d0 2024-04-16T16:15:26.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.969+0000 7f3fa1600640 1 --2- 172.21.15.45:0/1724166459 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3f8406ca60 0x7f3f8406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.969+0000 7f3fa1600640 1 -- 172.21.15.45:0/1724166459 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7f3f9408cc70 con 0x7f3fa40ce2d0 2024-04-16T16:15:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.969+0000 7f3fa9200640 1 --2- 172.21.15.45:0/1724166459 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3f8406ca60 0x7f3f8406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.970+0000 7f3fa9200640 1 --2- 172.21.15.45:0/1724166459 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3f8406ca60 0x7f3f8406ef20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3f90001200 tx=0x7f3f90009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:26.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:26.973+0000 7f3fa1600640 1 -- 172.21.15.45:0/1724166459 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3f94010040 con 0x7f3fa40ce2d0 2024-04-16T16:15:27.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.255+0000 7f3fac600640 1 -- 172.21.15.45:0/1724166459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs1_data"]} v 0) -- 0x7f3fa40d26c0 con 0x7f3fa40ce2d0 2024-04-16T16:15:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.261+0000 7f3fa1600640 1 -- 172.21.15.45:0/1724166459 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs1_data"]}]=0 updated caps for client.2 v14) ==== 181+0+138 (secure 0 0 0) 0x7f3f9405ac40 con 0x7f3fa40ce2d0 2024-04-16T16:15:27.260 INFO:teuthology.orchestra.run.smithi045.stdout:[client.2] 2024-04-16T16:15:27.260 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mWdlUDxAAuO2FsGH1/E+qP1xcyUEe3A== 2024-04-16T16:15:27.260 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:15:27.260 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:15:27.260 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw pool=fs1_data" 2024-04-16T16:15:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.2 2024-04-16T16:15:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.263+0000 7f3fac600640 1 -- 172.21.15.45:0/1724166459 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3f8406ca60 msgr2=0x7f3f8406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.263+0000 7f3fac600640 1 --2- 172.21.15.45:0/1724166459 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3f8406ca60 0x7f3f8406ef20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3f90001200 tx=0x7f3f90009040 comp rx=0 tx=0).stop 2024-04-16T16:15:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.263+0000 7f3fac600640 1 -- 172.21.15.45:0/1724166459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fa40ce2d0 msgr2=0x7f3fa4160410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.263+0000 7f3fac600640 1 --2- 172.21.15.45:0/1724166459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fa40ce2d0 0x7f3fa4160410 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009fc0 tx=0x7f3f9400e5d0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.264+0000 7f3fac600640 1 -- 172.21.15.45:0/1724166459 shutdown_connections 2024-04-16T16:15:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.264+0000 7f3fac600640 1 --2- 172.21.15.45:0/1724166459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fa40ce6b0 0x7f3fa41e84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.264+0000 7f3fac600640 1 --2- 172.21.15.45:0/1724166459 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3f8406ca60 0x7f3f8406ef20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.264+0000 7f3fac600640 1 --2- 172.21.15.45:0/1724166459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fa40cdb90 0x7f3fa4160950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.264+0000 7f3fac600640 1 --2- 172.21.15.45:0/1724166459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fa40ce2d0 0x7f3fa4160410 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.264+0000 7f3fac600640 1 -- 172.21.15.45:0/1724166459 >> 172.21.15.45:0/1724166459 conn(0x7f3fa4073690 msgr2=0x7f3fa40c98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.264+0000 7f3fac600640 1 -- 172.21.15.45:0/1724166459 shutdown_connections 2024-04-16T16:15:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.264+0000 7f3fac600640 1 -- 172.21.15.45:0/1724166459 wait complete. 2024-04-16T16:15:27.282 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=fs1_data' 2024-04-16T16:15:27.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.511+0000 7f1fefe00640 1 Processor -- start 2024-04-16T16:15:27.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.511+0000 7f1fefe00640 1 -- start start 2024-04-16T16:15:27.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.512+0000 7f1fefe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1ff0146220 0x7f1ff00ae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.512+0000 7f1fefe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ff00b2ea0 con 0x7f1ff00b2930 2024-04-16T16:15:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.512+0000 7f1fefe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ff00b09e0 con 0x7f1ff006c040 2024-04-16T16:15:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.512+0000 7f1fefe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1ff0146600 con 0x7f1ff0146220 2024-04-16T16:15:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.513+0000 7f1fee000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1ff00b2930 0x7f1ff006b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50070/0 (socket says 172.21.15.45:50070) 2024-04-16T16:15:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.513+0000 7f1fee000640 1 -- 172.21.15.45:0/361568819 learned_addr learned my addr 172.21.15.45:0/361568819 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.513+0000 7f1feea00640 1 --2- 172.21.15.45:0/361568819 >> v2:172.21.15.103:3300/0 conn(0x7f1ff0146220 0x7f1ff00ae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.513+0000 7f1fed600640 1 -- 172.21.15.45:0/361568819 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1099357661 0 0) 0x7f1ff00b2ea0 con 0x7f1ff00b2930 2024-04-16T16:15:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.513+0000 7f1fed600640 1 -- 172.21.15.45:0/361568819 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fd4003520 con 0x7f1ff00b2930 2024-04-16T16:15:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.514+0000 7f1fed600640 1 -- 172.21.15.45:0/361568819 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3366269364 0 0) 0x7f1fd4003520 con 0x7f1ff00b2930 2024-04-16T16:15:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.514+0000 7f1fed600640 1 -- 172.21.15.45:0/361568819 >> v2:172.21.15.103:3300/0 conn(0x7f1ff0146220 msgr2=0x7f1ff00ae5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.514+0000 7f1fed600640 1 --2- 172.21.15.45:0/361568819 >> v2:172.21.15.103:3300/0 conn(0x7f1ff0146220 0x7f1ff00ae5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.514+0000 7f1fed600640 1 -- 172.21.15.45:0/361568819 >> v1:172.21.15.103:6789/0 conn(0x7f1ff006c040 legacy=0x7f1ff00b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.514+0000 7f1fed600640 1 -- 172.21.15.45:0/361568819 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff00b0cc0 con 0x7f1ff00b2930 2024-04-16T16:15:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.514+0000 7f1fed600640 1 -- 172.21.15.45:0/361568819 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f1fd8001670 con 0x7f1ff00b2930 2024-04-16T16:15:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.514+0000 7f1fed600640 1 -- 172.21.15.45:0/361568819 >> v1:172.21.15.45:6789/0 conn(0x7f1ff00b2930 legacy=0x7f1ff006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.514+0000 7f1fed600640 1 --2- 172.21.15.45:0/361568819 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fd40039b0 0x7f1fd4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.514+0000 7f1fed600640 1 --2- 172.21.15.45:0/361568819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fd4004800 0x7f1fd4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.514+0000 7f1fed600640 1 --2- 172.21.15.45:0/361568819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd40051c0 0x7f1fd400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.514+0000 7f1fed600640 1 -- 172.21.15.45:0/361568819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff0146600 con 0x7f1fd40051c0 2024-04-16T16:15:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.514+0000 7f1fed600640 1 -- 172.21.15.45:0/361568819 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd400c060 con 0x7f1fd4004800 2024-04-16T16:15:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.514+0000 7f1fed600640 1 -- 172.21.15.45:0/361568819 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1fd400c260 con 0x7f1fd40039b0 2024-04-16T16:15:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.514+0000 7f1fef400640 1 --2- 172.21.15.45:0/361568819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd40051c0 0x7f1fd400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.515+0000 7f1feea00640 1 --2- 172.21.15.45:0/361568819 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fd40039b0 0x7f1fd4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.515+0000 7f1fee000640 1 --2- 172.21.15.45:0/361568819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fd4004800 0x7f1fd4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.515+0000 7f1fef400640 1 -- 172.21.15.45:0/361568819 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fd40039b0 msgr2=0x7f1fd4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.515+0000 7f1fef400640 1 --2- 172.21.15.45:0/361568819 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fd40039b0 0x7f1fd4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.515+0000 7f1fef400640 1 -- 172.21.15.45:0/361568819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fd4004800 msgr2=0x7f1fd4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.515+0000 7f1fef400640 1 --2- 172.21.15.45:0/361568819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fd4004800 0x7f1fd4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.515+0000 7f1fef400640 1 -- 172.21.15.45:0/361568819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd4003520 con 0x7f1fd40051c0 2024-04-16T16:15:27.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.516+0000 7f1fef400640 1 -- 172.21.15.45:0/361568819 reap_dead start 2024-04-16T16:15:27.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.516+0000 7f1fef400640 1 --2- 172.21.15.45:0/361568819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd40051c0 0x7f1fd400b9c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8001750 tx=0x7f1fd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=156aa8e845442454 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:27.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.516+0000 7f1fed600640 1 -- 172.21.15.45:0/361568819 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8001370 con 0x7f1fd40051c0 2024-04-16T16:15:27.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.516+0000 7f1fed600640 1 -- 172.21.15.45:0/361568819 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fe8005310 con 0x7f1fd40051c0 2024-04-16T16:15:27.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.516+0000 7f1fed600640 1 -- 172.21.15.45:0/361568819 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8003830 con 0x7f1fd40051c0 2024-04-16T16:15:27.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.517+0000 7f1fefe00640 1 -- 172.21.15.45:0/361568819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd40051c0 msgr2=0x7f1fd400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:27.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.517+0000 7f1fefe00640 1 --2- 172.21.15.45:0/361568819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd40051c0 0x7f1fd400b9c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8001750 tx=0x7f1fd4005830 comp rx=0 tx=0).stop 2024-04-16T16:15:27.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.517+0000 7f1fefe00640 1 -- 172.21.15.45:0/361568819 shutdown_connections 2024-04-16T16:15:27.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.517+0000 7f1fefe00640 1 --2- 172.21.15.45:0/361568819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd40051c0 0x7f1fd400b9c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.517+0000 7f1fefe00640 1 -- 172.21.15.45:0/361568819 >> 172.21.15.45:0/361568819 conn(0x7f1ff0073620 msgr2=0x7f1ff009aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:27.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.517+0000 7f1fefe00640 1 -- 172.21.15.45:0/361568819 shutdown_connections 2024-04-16T16:15:27.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.517+0000 7f1fefe00640 1 -- 172.21.15.45:0/361568819 wait complete. 2024-04-16T16:15:27.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.519+0000 7f1fefe00640 1 Processor -- start 2024-04-16T16:15:27.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.519+0000 7f1fefe00640 1 -- start start 2024-04-16T16:15:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.520+0000 7f1fefe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd40051c0 0x7f1ff013dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.520+0000 7f1fefe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ff0141f10 0x7f1ff00cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.520+0000 7f1fefe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ff009b740 0x7f1ff009bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.520+0000 7f1fefe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff00b3180 con 0x7f1fd40051c0 2024-04-16T16:15:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.520+0000 7f1fefe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff006c6e0 con 0x7f1ff009b740 2024-04-16T16:15:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.520+0000 7f1fefe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1ff00b1030 con 0x7f1ff0141f10 2024-04-16T16:15:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.521+0000 7f1feea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd40051c0 0x7f1ff013dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.521+0000 7f1feea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd40051c0 0x7f1ff013dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38832/0 (socket says 172.21.15.45:38832) 2024-04-16T16:15:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.521+0000 7f1feea00640 1 -- 172.21.15.45:0/329216681 learned_addr learned my addr 172.21.15.45:0/329216681 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.521+0000 7f1fee000640 1 --2- 172.21.15.45:0/329216681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ff0141f10 0x7f1ff00cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.521+0000 7f1feea00640 1 -- 172.21.15.45:0/329216681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ff0141f10 msgr2=0x7f1ff00cfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.521+0000 7f1feea00640 1 --2- 172.21.15.45:0/329216681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ff0141f10 0x7f1ff00cfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.521+0000 7f1feea00640 1 -- 172.21.15.45:0/329216681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ff009b740 msgr2=0x7f1ff009bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.521+0000 7f1feea00640 1 --2- 172.21.15.45:0/329216681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ff009b740 0x7f1ff009bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.521+0000 7f1feea00640 1 -- 172.21.15.45:0/329216681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff00ceb40 con 0x7f1fd40051c0 2024-04-16T16:15:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.521+0000 7f1fee000640 1 --2- 172.21.15.45:0/329216681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ff0141f10 0x7f1ff00cfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.521+0000 7f1feea00640 1 --2- 172.21.15.45:0/329216681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd40051c0 0x7f1ff013dfa0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009fb0 tx=0x7f1fe00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.522+0000 7f1fe6a00640 1 -- 172.21.15.45:0/329216681 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe001a020 con 0x7f1fd40051c0 2024-04-16T16:15:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.522+0000 7f1fe6a00640 1 -- 172.21.15.45:0/329216681 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fe0016470 con 0x7f1fd40051c0 2024-04-16T16:15:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.522+0000 7f1fe6a00640 1 -- 172.21.15.45:0/329216681 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0016860 con 0x7f1fd40051c0 2024-04-16T16:15:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.522+0000 7f1fefe00640 1 -- 172.21.15.45:0/329216681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ff00cee20 con 0x7f1fd40051c0 2024-04-16T16:15:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.522+0000 7f1fefe00640 1 -- 172.21.15.45:0/329216681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ff00b4e20 con 0x7f1fd40051c0 2024-04-16T16:15:27.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.523+0000 7f1fefe00640 1 -- 172.21.15.45:0/329216681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ff0075780 con 0x7f1fd40051c0 2024-04-16T16:15:27.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.524+0000 7f1fe6a00640 1 -- 172.21.15.45:0/329216681 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f1fe0005d10 con 0x7f1fd40051c0 2024-04-16T16:15:27.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.525+0000 7f1fe6a00640 1 --2- 172.21.15.45:0/329216681 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1fcc06ca60 0x7f1fcc06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:27.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.525+0000 7f1fe6a00640 1 -- 172.21.15.45:0/329216681 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7f1fe008cf40 con 0x7f1fd40051c0 2024-04-16T16:15:27.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.526+0000 7f1fee000640 1 --2- 172.21.15.45:0/329216681 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1fcc06ca60 0x7f1fcc06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:27.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.527+0000 7f1fee000640 1 --2- 172.21.15.45:0/329216681 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1fcc06ca60 0x7f1fcc06ef20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1ff00b6750 tx=0x7f1fd8001470 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.530+0000 7f1fe6a00640 1 -- 172.21.15.45:0/329216681 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1fe0024030 con 0x7f1fd40051c0 2024-04-16T16:15:27.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.818+0000 7f1fefe00640 1 -- 172.21.15.45:0/329216681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs1_data"]} v 0) -- 0x7f1ff00ccfc0 con 0x7f1fd40051c0 2024-04-16T16:15:27.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.822+0000 7f1fe6a00640 1 -- 172.21.15.45:0/329216681 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs1_data"]}]=0 updated caps for client.3 v15) ==== 181+0+138 (secure 0 0 0) 0x7f1fe005af10 con 0x7f1fd40051c0 2024-04-16T16:15:27.822 INFO:teuthology.orchestra.run.smithi045.stdout:[client.3] 2024-04-16T16:15:27.822 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5miPdMJBAAqjKpctNYu9ot1lff3rouxg== 2024-04-16T16:15:27.822 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:15:27.822 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:15:27.822 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw pool=fs1_data" 2024-04-16T16:15:27.822 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.3 2024-04-16T16:15:27.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.825+0000 7f1fefe00640 1 -- 172.21.15.45:0/329216681 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1fcc06ca60 msgr2=0x7f1fcc06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:27.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.825+0000 7f1fefe00640 1 --2- 172.21.15.45:0/329216681 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1fcc06ca60 0x7f1fcc06ef20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1ff00b6750 tx=0x7f1fd8001470 comp rx=0 tx=0).stop 2024-04-16T16:15:27.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.826+0000 7f1fefe00640 1 -- 172.21.15.45:0/329216681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd40051c0 msgr2=0x7f1ff013dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:27.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.826+0000 7f1fefe00640 1 --2- 172.21.15.45:0/329216681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd40051c0 0x7f1ff013dfa0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009fb0 tx=0x7f1fe00115a0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.826+0000 7f1fefe00640 1 -- 172.21.15.45:0/329216681 shutdown_connections 2024-04-16T16:15:27.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.826+0000 7f1fefe00640 1 --2- 172.21.15.45:0/329216681 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1fcc06ca60 0x7f1fcc06ef20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.826+0000 7f1fefe00640 1 --2- 172.21.15.45:0/329216681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ff009b740 0x7f1ff009bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.826+0000 7f1fefe00640 1 --2- 172.21.15.45:0/329216681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ff0141f10 0x7f1ff00cfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.826+0000 7f1fefe00640 1 --2- 172.21.15.45:0/329216681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd40051c0 0x7f1ff013dfa0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:27.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.826+0000 7f1fefe00640 1 -- 172.21.15.45:0/329216681 >> 172.21.15.45:0/329216681 conn(0x7f1ff0073620 msgr2=0x7f1ff006b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:27.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.826+0000 7f1fefe00640 1 -- 172.21.15.45:0/329216681 shutdown_connections 2024-04-16T16:15:27.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:27.826+0000 7f1fefe00640 1 -- 172.21.15.45:0/329216681 wait complete. 2024-04-16T16:15:27.846 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:15:28.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.073+0000 7f3181a00640 1 Processor -- start 2024-04-16T16:15:28.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.073+0000 7f3181a00640 1 -- start start 2024-04-16T16:15:28.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.075+0000 7f3181a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f317c0cfdc0 0x7f317c1697f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:28.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.075+0000 7f3181a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f317c0b7fd0 con 0x7f317c15ead0 2024-04-16T16:15:28.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.075+0000 7f3181a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f317c0b82b0 con 0x7f317c0b7b10 2024-04-16T16:15:28.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.075+0000 7f3181a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f317c0b8590 con 0x7f317c0cfdc0 2024-04-16T16:15:28.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.075+0000 7f317aa00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f317c15ead0 0x7f317c0ba300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50082/0 (socket says 172.21.15.45:50082) 2024-04-16T16:15:28.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.075+0000 7f317aa00640 1 -- 172.21.15.45:0/2449462841 learned_addr learned my addr 172.21.15.45:0/2449462841 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:28.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.075+0000 7f3179600640 1 -- 172.21.15.45:0/2449462841 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3122982963 0 0) 0x7f317c0b7fd0 con 0x7f317c15ead0 2024-04-16T16:15:28.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f3179600640 1 -- 172.21.15.45:0/2449462841 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3160003520 con 0x7f317c15ead0 2024-04-16T16:15:28.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f317b400640 1 --2- 172.21.15.45:0/2449462841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f317c0cfdc0 0x7f317c1697f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:28.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f3179600640 1 -- 172.21.15.45:0/2449462841 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 185232351 0 0) 0x7f3160003520 con 0x7f317c15ead0 2024-04-16T16:15:28.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f3179600640 1 -- 172.21.15.45:0/2449462841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f317c0cfdc0 msgr2=0x7f317c1697f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:28.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f3179600640 1 --2- 172.21.15.45:0/2449462841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f317c0cfdc0 0x7f317c1697f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:28.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f3179600640 1 -- 172.21.15.45:0/2449462841 >> v1:172.21.15.103:6789/0 conn(0x7f317c0b7b10 legacy=0x7f317c0cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:28.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f3179600640 1 -- 172.21.15.45:0/2449462841 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f317c0b8870 con 0x7f317c15ead0 2024-04-16T16:15:28.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f3179600640 1 -- 172.21.15.45:0/2449462841 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f316c0019f0 con 0x7f317c15ead0 2024-04-16T16:15:28.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f3179600640 1 -- 172.21.15.45:0/2449462841 >> v1:172.21.15.45:6789/0 conn(0x7f317c15ead0 legacy=0x7f317c0ba300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:28.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f3179600640 1 --2- 172.21.15.45:0/2449462841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31600039e0 0x7f3160003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:28.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f3179600640 1 --2- 172.21.15.45:0/2449462841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3160004800 0x7f3160004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:28.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f3179600640 1 --2- 172.21.15.45:0/2449462841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3160005230 0x7f316000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:28.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f3179600640 1 -- 172.21.15.45:0/2449462841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f317c0b8590 con 0x7f3160005230 2024-04-16T16:15:28.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f3179600640 1 -- 172.21.15.45:0/2449462841 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f316000c140 con 0x7f3160004800 2024-04-16T16:15:28.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f3179600640 1 -- 172.21.15.45:0/2449462841 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f316000c340 con 0x7f31600039e0 2024-04-16T16:15:28.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.076+0000 7f317b400640 1 --2- 172.21.15.45:0/2449462841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f317c0cfdc0 0x7f317c1697f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:28.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.077+0000 7f317b400640 1 --2- 172.21.15.45:0/2449462841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3160005230 0x7f316000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:28.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.077+0000 7f317aa00640 1 --2- 172.21.15.45:0/2449462841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31600039e0 0x7f3160003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:28.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.077+0000 7f317a000640 1 --2- 172.21.15.45:0/2449462841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3160004800 0x7f3160004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:28.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.077+0000 7f317b400640 1 -- 172.21.15.45:0/2449462841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31600039e0 msgr2=0x7f3160003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:28.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.077+0000 7f317b400640 1 --2- 172.21.15.45:0/2449462841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31600039e0 0x7f3160003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:28.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.077+0000 7f317b400640 1 -- 172.21.15.45:0/2449462841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3160004800 msgr2=0x7f3160004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:28.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.077+0000 7f317b400640 1 --2- 172.21.15.45:0/2449462841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3160004800 0x7f3160004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:28.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.077+0000 7f317b400640 1 -- 172.21.15.45:0/2449462841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3160003520 con 0x7f3160005230 2024-04-16T16:15:28.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.077+0000 7f317aa00640 1 --2- 172.21.15.45:0/2449462841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31600039e0 0x7f3160003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:15:28.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.079+0000 7f317b400640 1 --2- 172.21.15.45:0/2449462841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3160005230 0x7f316000b9c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f317c0b6cb0 tx=0x7f3160005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657ee2b144afca84 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:28.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.079+0000 7f3179600640 1 -- 172.21.15.45:0/2449462841 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317400a4b0 con 0x7f3160005230 2024-04-16T16:15:28.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.079+0000 7f3179600640 1 -- 172.21.15.45:0/2449462841 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3174037070 con 0x7f3160005230 2024-04-16T16:15:28.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.079+0000 7f3179600640 1 -- 172.21.15.45:0/2449462841 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31740328c0 con 0x7f3160005230 2024-04-16T16:15:28.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.080+0000 7f3181a00640 1 -- 172.21.15.45:0/2449462841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3160005230 msgr2=0x7f316000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:28.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.080+0000 7f3181a00640 1 --2- 172.21.15.45:0/2449462841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3160005230 0x7f316000b9c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f317c0b6cb0 tx=0x7f3160005830 comp rx=0 tx=0).stop 2024-04-16T16:15:28.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.080+0000 7f317b400640 1 -- 172.21.15.45:0/2449462841 reap_dead start 2024-04-16T16:15:28.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.080+0000 7f3181a00640 1 -- 172.21.15.45:0/2449462841 shutdown_connections 2024-04-16T16:15:28.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.080+0000 7f3181a00640 1 -- 172.21.15.45:0/2449462841 >> 172.21.15.45:0/2449462841 conn(0x7f317c073480 msgr2=0x7f317c073890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:28.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.080+0000 7f3181a00640 1 -- 172.21.15.45:0/2449462841 shutdown_connections 2024-04-16T16:15:28.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.080+0000 7f3181a00640 1 -- 172.21.15.45:0/2449462841 wait complete. 2024-04-16T16:15:28.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.082+0000 7f3181a00640 1 Processor -- start 2024-04-16T16:15:28.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.082+0000 7f3181a00640 1 -- start start 2024-04-16T16:15:28.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.083+0000 7f3181a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f317c153a00 0x7f317c0caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:28.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.083+0000 7f3181a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f317c153de0 0x7f317c0ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:28.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.083+0000 7f3181a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f317c0cb030 0x7f317c0cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:28.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.083+0000 7f3181a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f317c15f030 con 0x7f317c153a00 2024-04-16T16:15:28.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.083+0000 7f3181a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f317c15eeb0 con 0x7f317c153de0 2024-04-16T16:15:28.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.083+0000 7f3181a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f317c0d2d40 con 0x7f317c0cb030 2024-04-16T16:15:28.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.083+0000 7f317aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f317c153a00 0x7f317c0caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:28.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.083+0000 7f317aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f317c153a00 0x7f317c0caaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38852/0 (socket says 172.21.15.45:38852) 2024-04-16T16:15:28.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.083+0000 7f317aa00640 1 -- 172.21.15.45:0/2044028508 learned_addr learned my addr 172.21.15.45:0/2044028508 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:28.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.083+0000 7f317a000640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f317c153de0 0x7f317c0ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:28.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.083+0000 7f317b400640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f317c0cb030 0x7f317c0cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:28.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.083+0000 7f317aa00640 1 -- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f317c0cb030 msgr2=0x7f317c0cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:28.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.084+0000 7f317aa00640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f317c0cb030 0x7f317c0cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:28.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.084+0000 7f317aa00640 1 -- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f317c153de0 msgr2=0x7f317c0ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:28.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.084+0000 7f317aa00640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f317c153de0 0x7f317c0ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:28.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.084+0000 7f317aa00640 1 -- 172.21.15.45:0/2044028508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f317c1544d0 con 0x7f317c153a00 2024-04-16T16:15:28.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.084+0000 7f317b400640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f317c0cb030 0x7f317c0cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:28.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.084+0000 7f317a000640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f317c153de0 0x7f317c0ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:28.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.084+0000 7f317aa00640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f317c153a00 0x7f317c0caaf0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f316c0020a0 tx=0x7f316c00e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:28.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.084+0000 7f3172a00640 1 -- 172.21.15.45:0/2044028508 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f316c00ee80 con 0x7f317c153a00 2024-04-16T16:15:28.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.084+0000 7f3172a00640 1 -- 172.21.15.45:0/2044028508 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f316c018070 con 0x7f317c153a00 2024-04-16T16:15:28.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.084+0000 7f3172a00640 1 -- 172.21.15.45:0/2044028508 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f316c013620 con 0x7f317c153a00 2024-04-16T16:15:28.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.084+0000 7f3181a00640 1 -- 172.21.15.45:0/2044028508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f317c15f2e0 con 0x7f317c153a00 2024-04-16T16:15:28.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.084+0000 7f3181a00640 1 -- 172.21.15.45:0/2044028508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f317c15f8a0 con 0x7f317c153a00 2024-04-16T16:15:28.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.086+0000 7f3181a00640 1 -- 172.21.15.45:0/2044028508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f317c0b6d40 con 0x7f317c153a00 2024-04-16T16:15:28.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.086+0000 7f3172a00640 1 -- 172.21.15.45:0/2044028508 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f316c0137c0 con 0x7f317c153a00 2024-04-16T16:15:28.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.088+0000 7f3172a00640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f315806ca60 0x7f315806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:28.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.088+0000 7f3172a00640 1 -- 172.21.15.45:0/2044028508 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7f316c08c610 con 0x7f317c153a00 2024-04-16T16:15:28.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.089+0000 7f317a000640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f315806ca60 0x7f315806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:28.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.090+0000 7f317a000640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f315806ca60 0x7f315806ef20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f317c1f81b0 tx=0x7f3164009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:28.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.093+0000 7f3172a00640 1 -- 172.21.15.45:0/2044028508 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f317c0b6d40 con 0x7f317c153a00 2024-04-16T16:15:28.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.338+0000 7f3181a00640 1 -- 172.21.15.45:0/2044028508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f317c0cf700 con 0x7f317c153a00 2024-04-16T16:15:28.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.339+0000 7f3172a00640 1 -- 172.21.15.45:0/2044028508 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+64 (secure 0 0 0) 0x7f316c05a780 con 0x7f317c153a00 2024-04-16T16:15:28.339 INFO:teuthology.orchestra.run.smithi045.stdout:name: fs1, metadata pool: fs1_metadata, data pools: [fs1_data ] 2024-04-16T16:15:28.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.342+0000 7f3181a00640 1 -- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f315806ca60 msgr2=0x7f315806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:28.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.342+0000 7f3181a00640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f315806ca60 0x7f315806ef20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f317c1f81b0 tx=0x7f3164009040 comp rx=0 tx=0).stop 2024-04-16T16:15:28.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.342+0000 7f3181a00640 1 -- 172.21.15.45:0/2044028508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f317c153a00 msgr2=0x7f317c0caaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:28.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.342+0000 7f3181a00640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f317c153a00 0x7f317c0caaf0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f316c0020a0 tx=0x7f316c00e4d0 comp rx=0 tx=0).stop 2024-04-16T16:15:28.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.342+0000 7f3181a00640 1 -- 172.21.15.45:0/2044028508 shutdown_connections 2024-04-16T16:15:28.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.342+0000 7f3181a00640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f317c0cb030 0x7f317c0cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:28.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.342+0000 7f3181a00640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f315806ca60 0x7f315806ef20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:28.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.342+0000 7f3181a00640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f317c153de0 0x7f317c0ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:28.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.342+0000 7f3181a00640 1 --2- 172.21.15.45:0/2044028508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f317c153a00 0x7f317c0caaf0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:28.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.342+0000 7f3181a00640 1 -- 172.21.15.45:0/2044028508 >> 172.21.15.45:0/2044028508 conn(0x7f317c073480 msgr2=0x7f317c0d4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:28.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.342+0000 7f3181a00640 1 -- 172.21.15.45:0/2044028508 shutdown_connections 2024-04-16T16:15:28.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28.343+0000 7f3181a00640 1 -- 172.21.15.45:0/2044028508 wait complete. 2024-04-16T16:15:28.361 INFO:tasks.cephfs.mount:Mounting Ceph FS fs1; just confirmed its presence on cluster 2024-04-16T16:15:28.361 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:15:28.361 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-04-16T16:15:28.361 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-16T16:15:28.361 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T16:15:28.361 INFO:tasks.cephfs.mount:self.cephfs_name = fs1 2024-04-16T16:15:28.361 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:15:28.361 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:15:28.361 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:15:28.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:28.362 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-04-16T16:15:28.415 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:15:28.415 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:15:28.415 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:15:28.415 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:15:28.415 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-04-16T16:15:28.415 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:15:28.415 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:15:28.415 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:15:28.415 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-04-16T16:15:28.415 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:15:28.415 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:15:28.415 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-04-16T16:15:28.416 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:15:28.416 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-04-16T16:15:28.416 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-04-16T16:15:28.416 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-16T16:15:28.416 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42473sec preferred_lft 42473sec 2024-04-16T16:15:28.416 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-04-16T16:15:28.416 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:15:28.416 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:15:28.416 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-04-16T16:15:28.416 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-04-16T16:15:28.416 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:15:28.416 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff 2024-04-16T16:15:28.416 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:15:28.417 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:15:28.417 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::50f1:99ff:fe2b:225b/64 scope link 2024-04-16T16:15:28.417 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:15:28.417 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:15:28.417 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T16:15:28.417 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::7007:c3ff:fe13:7bcd/64 scope link 2024-04-16T16:15:28.417 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:15:28.417 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:15:28.417 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether ae:10:ea:fc:0a:13 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-16T16:15:28.417 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ac10:eaff:fefc:a13/64 scope link 2024-04-16T16:15:28.417 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:15:28.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:28.418 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:15:28.510 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-04-16T16:15:28.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:28.515 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-04-16T16:15:28.573 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-04-16T16:15:28.573 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:15:28.573 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-16T16:15:28.573 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-16T16:15:28.573 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:15:28.574 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:15:28.575 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:15:28.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-16T16:15:28.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-16T16:15:28.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-16T16:15:28.575 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:15:28.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:15:28.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:15:28.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:28.838 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T16:15:28.888 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-16T16:15:28.888 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-16T16:15:28.888 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:15:28.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:28.888 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-04-16T16:15:28.979 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:15:28.979 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-16T16:15:29.033 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:15:29.033 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:15:29.034 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:15:29.088 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.fs1=/ 2024-04-16T16:15:29.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:29.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.fs1=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:15:29.232 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:15:29.232 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:15:29.267 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:15:29.267 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:15:29.267 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:15:29.267 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 0@748c4b85-0190-4c59-80ad-098a6a5007c0.fs1=/ 2024-04-16T16:15:29.267 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:15:29.270 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-16T16:15:29.270 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:15:29.270 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:15:29.271 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:15:29.271 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:15:29.275 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:15:29.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:29.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:15:29.366 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:15:29.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:15:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:15:29.517 INFO:teuthology.orchestra.run.smithi045.stdout:client5730 2024-04-16T16:15:29.518 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:15:29.518 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:15:29.572 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:15:29.572 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:15:29.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:29.620 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=pad.bin bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:15:29.784 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:15:29.784 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:15:29.784 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475423 s, 22.1 MB/s 2024-04-16T16:15:29.786 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:15:29.786 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:15:29.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:29.846 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test.bin bs=1M conv=fdatasync count=2 seek=0) 2024-04-16T16:15:29.953 INFO:teuthology.orchestra.run.smithi045.stderr:2+0 records in 2024-04-16T16:15:29.953 INFO:teuthology.orchestra.run.smithi045.stderr:2+0 records out 2024-04-16T16:15:29.953 INFO:teuthology.orchestra.run.smithi045.stderr:2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0488152 s, 43.0 MB/s 2024-04-16T16:15:29.955 DEBUG:teuthology.orchestra.run.smithi045:> stdin-killer -- python3 -c ' 2024-04-16T16:15:29.955 DEBUG:teuthology.orchestra.run.smithi045:> import os 2024-04-16T16:15:29.955 DEBUG:teuthology.orchestra.run.smithi045:> import stat 2024-04-16T16:15:29.955 DEBUG:teuthology.orchestra.run.smithi045:> 2024-04-16T16:15:29.955 DEBUG:teuthology.orchestra.run.smithi045:> print(os.stat("/home/ubuntu/cephtest/mnt.0/test.bin").st_ino) 2024-04-16T16:15:29.956 DEBUG:teuthology.orchestra.run.smithi045:> ' 2024-04-16T16:15:30.106 INFO:teuthology.orchestra.run.smithi045.stdout:1099511627777 2024-04-16T16:15:30.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:15:30.120 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:15:30.121 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:15:30.219 INFO:tasks.cephfs.mount:Creating file a 2024-04-16T16:15:30.219 DEBUG:teuthology.orchestra.run.smithi045:> touch /home/ubuntu/cephtest/mnt.0/a 2024-04-16T16:15:30.275 INFO:tasks.cephfs.mount:Creating file b 2024-04-16T16:15:30.275 DEBUG:teuthology.orchestra.run.smithi045:> touch /home/ubuntu/cephtest/mnt.0/b 2024-04-16T16:15:30.333 INFO:tasks.cephfs.mount:Creating file c 2024-04-16T16:15:30.334 DEBUG:teuthology.orchestra.run.smithi045:> touch /home/ubuntu/cephtest/mnt.0/c 2024-04-16T16:15:30.385 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:15:30.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.607+0000 7fbbfa600640 1 Processor -- start 2024-04-16T16:15:30.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.608+0000 7fbbfa600640 1 -- start start 2024-04-16T16:15:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.609+0000 7fbbfa600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbbf4161e90 0x7fbbf40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.609+0000 7fbbfa600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf40d52d0 0x7fbbf406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.609+0000 7fbbfa600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbf40d3080 con 0x7fbbf4162270 2024-04-16T16:15:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.609+0000 7fbbfa600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbbf40d2e30 con 0x7fbbf4161e90 2024-04-16T16:15:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.609+0000 7fbbfa600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf40d3360 con 0x7fbbf40d52d0 2024-04-16T16:15:30.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.609+0000 7fbbf3e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbbf4162270 0x7fbbf40d3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50098/0 (socket says 172.21.15.45:50098) 2024-04-16T16:15:30.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.609+0000 7fbbf3e00640 1 -- 172.21.15.45:0/3293795743 learned_addr learned my addr 172.21.15.45:0/3293795743 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.610+0000 7fbbf9200640 1 --2- 172.21.15.45:0/3293795743 >> v2:172.21.15.103:3300/0 conn(0x7fbbf4161e90 0x7fbbf40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.610+0000 7fbbf3400640 1 -- 172.21.15.45:0/3293795743 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 968342213 0 0) 0x7fbbf40d3080 con 0x7fbbf4162270 2024-04-16T16:15:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.610+0000 7fbbf9c00640 1 --2- 172.21.15.45:0/3293795743 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf40d52d0 0x7fbbf406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.610+0000 7fbbf3400640 1 -- 172.21.15.45:0/3293795743 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbd4003520 con 0x7fbbf4162270 2024-04-16T16:15:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.610+0000 7fbbf3400640 1 -- 172.21.15.45:0/3293795743 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2764214880 0 0) 0x7fbbd4003520 con 0x7fbbf4162270 2024-04-16T16:15:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.610+0000 7fbbf3400640 1 -- 172.21.15.45:0/3293795743 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf40d52d0 msgr2=0x7fbbf406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.610+0000 7fbbf3400640 1 --2- 172.21.15.45:0/3293795743 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf40d52d0 0x7fbbf406ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.610+0000 7fbbf3400640 1 -- 172.21.15.45:0/3293795743 >> v2:172.21.15.103:3300/0 conn(0x7fbbf4161e90 msgr2=0x7fbbf40cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.610+0000 7fbbf3400640 1 --2- 172.21.15.45:0/3293795743 >> v2:172.21.15.103:3300/0 conn(0x7fbbf4161e90 0x7fbbf40cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.610+0000 7fbbf3400640 1 -- 172.21.15.45:0/3293795743 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf40d3640 con 0x7fbbf4162270 2024-04-16T16:15:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.610+0000 7fbbf3400640 1 -- 172.21.15.45:0/3293795743 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fbbdc0019f0 con 0x7fbbf4162270 2024-04-16T16:15:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.611+0000 7fbbf3400640 1 -- 172.21.15.45:0/3293795743 >> v1:172.21.15.45:6789/0 conn(0x7fbbf4162270 legacy=0x7fbbf40d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.611+0000 7fbbf3400640 1 --2- 172.21.15.45:0/3293795743 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf40d52d0 0x7fbbd40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.611+0000 7fbbf3400640 1 --2- 172.21.15.45:0/3293795743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbd4003f30 0x7fbbd4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.611+0000 7fbbf3400640 1 --2- 172.21.15.45:0/3293795743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd4004d40 0x7fbbd40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.611+0000 7fbbf3400640 1 -- 172.21.15.45:0/3293795743 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf40d2e30 con 0x7fbbd4004d40 2024-04-16T16:15:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.611+0000 7fbbf3400640 1 -- 172.21.15.45:0/3293795743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf40d3360 con 0x7fbbd4003f30 2024-04-16T16:15:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.611+0000 7fbbf3400640 1 -- 172.21.15.45:0/3293795743 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbbd400b9f0 con 0x7fbbf40d52d0 2024-04-16T16:15:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.611+0000 7fbbf9c00640 1 --2- 172.21.15.45:0/3293795743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd4004d40 0x7fbbd40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.611+0000 7fbbf9200640 1 --2- 172.21.15.45:0/3293795743 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf40d52d0 0x7fbbd40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.611+0000 7fbbf9c00640 1 -- 172.21.15.45:0/3293795743 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf40d52d0 msgr2=0x7fbbd40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.611+0000 7fbbf9c00640 1 --2- 172.21.15.45:0/3293795743 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf40d52d0 0x7fbbd40039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.611+0000 7fbbf9c00640 1 -- 172.21.15.45:0/3293795743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbd4003f30 msgr2=0x7fbbd4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.611+0000 7fbbf9c00640 1 --2- 172.21.15.45:0/3293795743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbd4003f30 0x7fbbd4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.611+0000 7fbbf9c00640 1 -- 172.21.15.45:0/3293795743 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd4003520 con 0x7fbbd4004d40 2024-04-16T16:15:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.612+0000 7fbbf9c00640 1 --2- 172.21.15.45:0/3293795743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd4004d40 0x7fbbd40051c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fbbec009a90 tx=0x7fbbd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3a64da467d1190d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.613+0000 7fbbf3400640 1 -- 172.21.15.45:0/3293795743 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbec039070 con 0x7fbbd4004d40 2024-04-16T16:15:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.613+0000 7fbbf3400640 1 -- 172.21.15.45:0/3293795743 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbec02c680 con 0x7fbbd4004d40 2024-04-16T16:15:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.613+0000 7fbbf3400640 1 -- 172.21.15.45:0/3293795743 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbec009690 con 0x7fbbd4004d40 2024-04-16T16:15:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.613+0000 7fbbfa600640 1 -- 172.21.15.45:0/3293795743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd4004d40 msgr2=0x7fbbd40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.613+0000 7fbbfa600640 1 --2- 172.21.15.45:0/3293795743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd4004d40 0x7fbbd40051c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fbbec009a90 tx=0x7fbbd4005830 comp rx=0 tx=0).stop 2024-04-16T16:15:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.613+0000 7fbbf9c00640 1 -- 172.21.15.45:0/3293795743 reap_dead start 2024-04-16T16:15:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.613+0000 7fbbfa600640 1 -- 172.21.15.45:0/3293795743 shutdown_connections 2024-04-16T16:15:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.614+0000 7fbbfa600640 1 -- 172.21.15.45:0/3293795743 >> 172.21.15.45:0/3293795743 conn(0x7fbbf40d1940 msgr2=0x7fbbf40d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.614+0000 7fbbfa600640 1 -- 172.21.15.45:0/3293795743 shutdown_connections 2024-04-16T16:15:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.614+0000 7fbbfa600640 1 -- 172.21.15.45:0/3293795743 wait complete. 2024-04-16T16:15:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.615+0000 7fbbfa600640 1 Processor -- start 2024-04-16T16:15:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.616+0000 7fbbfa600640 1 -- start start 2024-04-16T16:15:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.616+0000 7fbbfa600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd4004d40 0x7fbbf41f4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.616+0000 7fbbfa600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbf415fc20 0x7fbbf40ac590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.617+0000 7fbbfa600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf40acad0 0x7fbbf40ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.617+0000 7fbbfa600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf40d4550 con 0x7fbbd4004d40 2024-04-16T16:15:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.617+0000 7fbbfa600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf40cfd10 con 0x7fbbf415fc20 2024-04-16T16:15:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.617+0000 7fbbfa600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf40d46d0 con 0x7fbbf40acad0 2024-04-16T16:15:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.617+0000 7fbbf9200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd4004d40 0x7fbbf41f4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.617+0000 7fbbf9200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd4004d40 0x7fbbf41f4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38868/0 (socket says 172.21.15.45:38868) 2024-04-16T16:15:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.617+0000 7fbbf9200640 1 -- 172.21.15.45:0/2411480719 learned_addr learned my addr 172.21.15.45:0/2411480719 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.617+0000 7fbbf9200640 1 -- 172.21.15.45:0/2411480719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf40acad0 msgr2=0x7fbbf40ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.617+0000 7fbbf3e00640 1 --2- 172.21.15.45:0/2411480719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbf415fc20 0x7fbbf40ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.617+0000 7fbbf9200640 1 --2- 172.21.15.45:0/2411480719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf40acad0 0x7fbbf40ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.617+0000 7fbbf9200640 1 -- 172.21.15.45:0/2411480719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbf415fc20 msgr2=0x7fbbf40ac590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.617+0000 7fbbf9200640 1 --2- 172.21.15.45:0/2411480719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbf415fc20 0x7fbbf40ac590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.617+0000 7fbbf9200640 1 -- 172.21.15.45:0/2411480719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf41f5490 con 0x7fbbd4004d40 2024-04-16T16:15:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.618+0000 7fbbf9200640 1 --2- 172.21.15.45:0/2411480719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd4004d40 0x7fbbf41f4f50 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4010860 tx=0x7fbbe4010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.618+0000 7fbbf0c00640 1 -- 172.21.15.45:0/2411480719 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe400f040 con 0x7fbbd4004d40 2024-04-16T16:15:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.618+0000 7fbbf0c00640 1 -- 172.21.15.45:0/2411480719 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe400eb60 con 0x7fbbd4004d40 2024-04-16T16:15:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.618+0000 7fbbf0c00640 1 -- 172.21.15.45:0/2411480719 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe401c400 con 0x7fbbd4004d40 2024-04-16T16:15:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.618+0000 7fbbfa600640 1 -- 172.21.15.45:0/2411480719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf40ab210 con 0x7fbbd4004d40 2024-04-16T16:15:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.618+0000 7fbbfa600640 1 -- 172.21.15.45:0/2411480719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf40ab7d0 con 0x7fbbd4004d40 2024-04-16T16:15:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.620+0000 7fbbfa600640 1 -- 172.21.15.45:0/2411480719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbf40ac1d0 con 0x7fbbd4004d40 2024-04-16T16:15:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.621+0000 7fbbf0c00640 1 -- 172.21.15.45:0/2411480719 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fbbe401c5a0 con 0x7fbbd4004d40 2024-04-16T16:15:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.621+0000 7fbbf0c00640 1 --2- 172.21.15.45:0/2411480719 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbbd006ca90 0x7fbbd006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.622+0000 7fbbf0c00640 1 -- 172.21.15.45:0/2411480719 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7fbbe408c610 con 0x7fbbd4004d40 2024-04-16T16:15:30.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.622+0000 7fbbf3e00640 1 --2- 172.21.15.45:0/2411480719 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbbd006ca90 0x7fbbd006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:30.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.623+0000 7fbbf3e00640 1 --2- 172.21.15.45:0/2411480719 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbbd006ca90 0x7fbbd006ef50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0012e0 tx=0x7fbbdc009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:30.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.626+0000 7fbbf0c00640 1 -- 172.21.15.45:0/2411480719 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbbe4018030 con 0x7fbbd4004d40 2024-04-16T16:15:30.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.837+0000 7fbbfa600640 1 -- 172.21.15.45:0/2411480719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbbf406dae0 con 0x7fbbd4004d40 2024-04-16T16:15:30.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.838+0000 7fbbf0c00640 1 -- 172.21.15.45:0/2411480719 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) ==== 74+0+17244 (secure 0 0 0) 0x7fbbe405a6f0 con 0x7fbbd4004d40 2024-04-16T16:15:30.838 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:30.838 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":46,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:15:21.475120+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"fs1_metadata","create_time":"2024-04-16T16:15:13.250782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"fs1_data","create_time":"2024-04-16T16:15:13.856986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.45:6833/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:15:30.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.841+0000 7fbbfa600640 1 -- 172.21.15.45:0/2411480719 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbbd006ca90 msgr2=0x7fbbd006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:30.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.841+0000 7fbbfa600640 1 --2- 172.21.15.45:0/2411480719 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbbd006ca90 0x7fbbd006ef50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0012e0 tx=0x7fbbdc009040 comp rx=0 tx=0).stop 2024-04-16T16:15:30.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.841+0000 7fbbfa600640 1 -- 172.21.15.45:0/2411480719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd4004d40 msgr2=0x7fbbf41f4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:30.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.841+0000 7fbbfa600640 1 --2- 172.21.15.45:0/2411480719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd4004d40 0x7fbbf41f4f50 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4010860 tx=0x7fbbe4010d20 comp rx=0 tx=0).stop 2024-04-16T16:15:30.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.841+0000 7fbbfa600640 1 -- 172.21.15.45:0/2411480719 shutdown_connections 2024-04-16T16:15:30.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.841+0000 7fbbfa600640 1 --2- 172.21.15.45:0/2411480719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf40acad0 0x7fbbf40ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:30.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.841+0000 7fbbfa600640 1 --2- 172.21.15.45:0/2411480719 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbbd006ca90 0x7fbbd006ef50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:30.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.841+0000 7fbbfa600640 1 --2- 172.21.15.45:0/2411480719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbf415fc20 0x7fbbf40ac590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:30.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.841+0000 7fbbfa600640 1 --2- 172.21.15.45:0/2411480719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd4004d40 0x7fbbf41f4f50 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:30.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.841+0000 7fbbfa600640 1 -- 172.21.15.45:0/2411480719 >> 172.21.15.45:0/2411480719 conn(0x7fbbf40d1940 msgr2=0x7fbbf406ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:30.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.842+0000 7fbbfa600640 1 -- 172.21.15.45:0/2411480719 shutdown_connections 2024-04-16T16:15:30.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:30.842+0000 7fbbfa600640 1 -- 172.21.15.45:0/2411480719 wait complete. 2024-04-16T16:15:30.862 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'fs1_metadata', 'create_time': '2024-04-16T16:15:13.250782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'fs1'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'fs1_data', 'create_time': '2024-04-16T16:15:13.856986+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'fs1'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:15:30.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:15:31.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.092+0000 7f6fad600640 1 Processor -- start 2024-04-16T16:15:31.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.092+0000 7f6fad600640 1 -- start start 2024-04-16T16:15:31.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.094+0000 7f6fad600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fa80ba7c0 0x7f6fa80d2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:31.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.094+0000 7f6fad600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6fa80d3030 0x7f6fa80b95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:31.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.094+0000 7f6fad600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fa80b8230 con 0x7f6fa815ec30 2024-04-16T16:15:31.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.094+0000 7f6fad600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6fa80b8500 con 0x7f6fa80d3030 2024-04-16T16:15:31.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.094+0000 7f6fad600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6fa80b87e0 con 0x7f6fa80ba7c0 2024-04-16T16:15:31.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.095+0000 7f6fa6000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fa80ba7c0 0x7f6fa80d2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:31.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.095+0000 7f6fa7400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6fa80d3030 0x7f6fa80b95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:31.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.095+0000 7f6fa6000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fa80ba7c0 0x7f6fa80d2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:43402/0 (socket says 172.21.15.45:43402) 2024-04-16T16:15:31.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.095+0000 7f6fa6000640 1 -- 172.21.15.45:0/3600343173 learned_addr learned my addr 172.21.15.45:0/3600343173 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:31.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.095+0000 7f6fa6a00640 1 --1- 172.21.15.45:0/3600343173 >> v1:172.21.15.103:6789/0 conn(0x7f6fa815ec30 0x7f6fa80b7b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:45188/0 (socket says 172.21.15.45:45188) 2024-04-16T16:15:31.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.096+0000 7f6fa6000640 1 -- 172.21.15.45:0/3600343173 >> v2:172.21.15.103:3300/0 conn(0x7f6fa80d3030 msgr2=0x7f6fa80b95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:31.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.096+0000 7f6fa6000640 1 --2- 172.21.15.45:0/3600343173 >> v2:172.21.15.103:3300/0 conn(0x7f6fa80d3030 0x7f6fa80b95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.096+0000 7f6fa6000640 1 -- 172.21.15.45:0/3600343173 >> v1:172.21.15.103:6789/0 conn(0x7f6fa815ec30 legacy=0x7f6fa80b7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:31.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.096+0000 7f6fa6000640 1 -- 172.21.15.45:0/3600343173 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa80b8ac0 con 0x7f6fa80ba7c0 2024-04-16T16:15:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.097+0000 7f6fa6000640 1 --2- 172.21.15.45:0/3600343173 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fa80ba7c0 0x7f6fa80d2af0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6f90009870 tx=0x7f6f9002ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb475c52c6e24859 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.098+0000 7f6fa5600640 1 -- 172.21.15.45:0/3600343173 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9002f830 con 0x7f6fa80ba7c0 2024-04-16T16:15:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.098+0000 7f6fa5600640 1 -- 172.21.15.45:0/3600343173 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f9002f9d0 con 0x7f6fa80ba7c0 2024-04-16T16:15:31.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.098+0000 7f6fa5600640 1 -- 172.21.15.45:0/3600343173 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9002fda0 con 0x7f6fa80ba7c0 2024-04-16T16:15:31.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.098+0000 7f6fad600640 1 -- 172.21.15.45:0/3600343173 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fa80ba7c0 msgr2=0x7f6fa80d2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:31.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.098+0000 7f6fad600640 1 --2- 172.21.15.45:0/3600343173 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fa80ba7c0 0x7f6fa80d2af0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6f90009870 tx=0x7f6f9002ba10 comp rx=0 tx=0).stop 2024-04-16T16:15:31.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.098+0000 7f6fad600640 1 -- 172.21.15.45:0/3600343173 shutdown_connections 2024-04-16T16:15:31.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.098+0000 7f6fad600640 1 --2- 172.21.15.45:0/3600343173 >> v2:172.21.15.103:3300/0 conn(0x7f6fa80d3030 0x7f6fa80b95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.098+0000 7f6fad600640 1 --2- 172.21.15.45:0/3600343173 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fa80ba7c0 0x7f6fa80d2af0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.098+0000 7f6fad600640 1 -- 172.21.15.45:0/3600343173 >> 172.21.15.45:0/3600343173 conn(0x7f6fa8073480 msgr2=0x7f6fa8073890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:31.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.099+0000 7f6fad600640 1 -- 172.21.15.45:0/3600343173 shutdown_connections 2024-04-16T16:15:31.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.099+0000 7f6fad600640 1 -- 172.21.15.45:0/3600343173 wait complete. 2024-04-16T16:15:31.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.100+0000 7f6fad600640 1 Processor -- start 2024-04-16T16:15:31.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.100+0000 7f6fad600640 1 -- start start 2024-04-16T16:15:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.101+0000 7f6fad600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fa80ba7c0 0x7f6fa8158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.101+0000 7f6fad600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fa80d3030 0x7f6fa80c9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.101+0000 7f6fad600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa815ec30 0x7f6fa8156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.101+0000 7f6fad600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa815f010 con 0x7f6fa815ec30 2024-04-16T16:15:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.101+0000 7f6fad600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa80b9080 con 0x7f6fa80d3030 2024-04-16T16:15:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.101+0000 7f6fad600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6fa80b9ae0 con 0x7f6fa80ba7c0 2024-04-16T16:15:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.102+0000 7f6fa7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa815ec30 0x7f6fa8156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.102+0000 7f6fa7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa815ec30 0x7f6fa8156fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38882/0 (socket says 172.21.15.45:38882) 2024-04-16T16:15:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.102+0000 7f6fa7400640 1 -- 172.21.15.45:0/2534222676 learned_addr learned my addr 172.21.15.45:0/2534222676 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.102+0000 7f6fa6a00640 1 --2- 172.21.15.45:0/2534222676 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fa80ba7c0 0x7f6fa8158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.102+0000 7f6fa6000640 1 --2- 172.21.15.45:0/2534222676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fa80d3030 0x7f6fa80c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.102+0000 7f6fa7400640 1 -- 172.21.15.45:0/2534222676 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fa80ba7c0 msgr2=0x7f6fa8158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.102+0000 7f6fa7400640 1 --2- 172.21.15.45:0/2534222676 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fa80ba7c0 0x7f6fa8158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.102+0000 7f6fa7400640 1 -- 172.21.15.45:0/2534222676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fa80d3030 msgr2=0x7f6fa80c9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.102+0000 7f6fa7400640 1 --2- 172.21.15.45:0/2534222676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fa80d3030 0x7f6fa80c9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.102+0000 7f6fa7400640 1 -- 172.21.15.45:0/2534222676 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa80c9b70 con 0x7f6fa815ec30 2024-04-16T16:15:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.103+0000 7f6fa7400640 1 --2- 172.21.15.45:0/2534222676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa815ec30 0x7f6fa8156fe0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000e850 tx=0x7f6fa000ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:31.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.103+0000 7f6f9ea00640 1 -- 172.21.15.45:0/2534222676 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0017070 con 0x7f6fa815ec30 2024-04-16T16:15:31.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.103+0000 7f6f9ea00640 1 -- 172.21.15.45:0/2534222676 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fa000c940 con 0x7f6fa815ec30 2024-04-16T16:15:31.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.103+0000 7f6f9ea00640 1 -- 172.21.15.45:0/2534222676 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa000cd30 con 0x7f6fa815ec30 2024-04-16T16:15:31.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.103+0000 7f6fad600640 1 -- 172.21.15.45:0/2534222676 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fa80c9df0 con 0x7f6fa815ec30 2024-04-16T16:15:31.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.103+0000 7f6fad600640 1 -- 172.21.15.45:0/2534222676 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fa80ca3b0 con 0x7f6fa815ec30 2024-04-16T16:15:31.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.104+0000 7f6fad600640 1 -- 172.21.15.45:0/2534222676 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fa8075780 con 0x7f6fa815ec30 2024-04-16T16:15:31.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.105+0000 7f6f9ea00640 1 -- 172.21.15.45:0/2534222676 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f6fa001a070 con 0x7f6fa815ec30 2024-04-16T16:15:31.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.106+0000 7f6f9ea00640 1 --2- 172.21.15.45:0/2534222676 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6f8406ca60 0x7f6f8406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:31.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.106+0000 7f6f9ea00640 1 -- 172.21.15.45:0/2534222676 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7f6fa008bf70 con 0x7f6fa815ec30 2024-04-16T16:15:31.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.107+0000 7f6fa6a00640 1 --2- 172.21.15.45:0/2534222676 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6f8406ca60 0x7f6f8406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:31.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.108+0000 7f6fa6a00640 1 --2- 172.21.15.45:0/2534222676 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6f8406ca60 0x7f6f8406ef20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6f980012e0 tx=0x7f6f98005b70 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:31.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.111+0000 7f6f9ea00640 1 -- 172.21.15.45:0/2534222676 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6fa80c9b70 con 0x7f6fa815ec30 2024-04-16T16:15:31.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.313+0000 7f6fad600640 1 -- 172.21.15.45:0/2534222676 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6fa80b4140 con 0x7f6fa815ec30 2024-04-16T16:15:31.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.314+0000 7f6f9ea00640 1 -- 172.21.15.45:0/2534222676 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) ==== 93+0+10 (secure 0 0 0) 0x7f6fa005a050 con 0x7f6fa815ec30 2024-04-16T16:15:31.313 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:15:31.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.316+0000 7f6fad600640 1 -- 172.21.15.45:0/2534222676 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6f8406ca60 msgr2=0x7f6f8406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:31.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.316+0000 7f6fad600640 1 --2- 172.21.15.45:0/2534222676 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6f8406ca60 0x7f6f8406ef20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6f980012e0 tx=0x7f6f98005b70 comp rx=0 tx=0).stop 2024-04-16T16:15:31.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.316+0000 7f6fad600640 1 -- 172.21.15.45:0/2534222676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa815ec30 msgr2=0x7f6fa8156fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:31.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.316+0000 7f6fad600640 1 --2- 172.21.15.45:0/2534222676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa815ec30 0x7f6fa8156fe0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000e850 tx=0x7f6fa000ed10 comp rx=0 tx=0).stop 2024-04-16T16:15:31.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.317+0000 7f6fad600640 1 -- 172.21.15.45:0/2534222676 shutdown_connections 2024-04-16T16:15:31.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.317+0000 7f6fad600640 1 --2- 172.21.15.45:0/2534222676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa815ec30 0x7f6fa8156fe0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.317+0000 7f6fad600640 1 --2- 172.21.15.45:0/2534222676 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6f8406ca60 0x7f6f8406ef20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.317+0000 7f6fad600640 1 --2- 172.21.15.45:0/2534222676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fa80d3030 0x7f6fa80c9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.317+0000 7f6fad600640 1 --2- 172.21.15.45:0/2534222676 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fa80ba7c0 0x7f6fa8158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.317+0000 7f6fad600640 1 -- 172.21.15.45:0/2534222676 >> 172.21.15.45:0/2534222676 conn(0x7f6fa8073480 msgr2=0x7f6fa80b5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:31.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.317+0000 7f6fad600640 1 -- 172.21.15.45:0/2534222676 shutdown_connections 2024-04-16T16:15:31.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.317+0000 7f6fad600640 1 -- 172.21.15.45:0/2534222676 wait complete. 2024-04-16T16:15:31.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs1_metadata pg_num 2024-04-16T16:15:31.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.561+0000 7f66cec00640 1 Processor -- start 2024-04-16T16:15:31.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.561+0000 7f66cec00640 1 -- start start 2024-04-16T16:15:31.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.562+0000 7f66cec00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f66c80b58f0 0x7f66c80b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:31.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.562+0000 7f66cec00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66c8146590 con 0x7f66c806e460 2024-04-16T16:15:31.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.562+0000 7f66cec00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66c80c5750 con 0x7f66c81461b0 2024-04-16T16:15:31.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.562+0000 7f66cec00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f66c806dfd0 con 0x7f66c80b58f0 2024-04-16T16:15:31.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.563+0000 7f66cce00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f66c80b58f0 0x7f66c80b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:31.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.563+0000 7f66cce00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f66c80b58f0 0x7f66c80b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38898/0 (socket says 172.21.15.45:38898) 2024-04-16T16:15:31.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.563+0000 7f66cce00640 1 -- 172.21.15.45:0/1672938642 learned_addr learned my addr 172.21.15.45:0/1672938642 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:31.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.563+0000 7f66ce200640 1 --1- 172.21.15.45:0/1672938642 >> v1:172.21.15.45:6789/0 conn(0x7f66c806e460 0x7f66c806e8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50106/0 (socket says 172.21.15.45:50106) 2024-04-16T16:15:31.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.563+0000 7f66cce00640 1 -- 172.21.15.45:0/1672938642 >> v1:172.21.15.103:6789/0 conn(0x7f66c81461b0 legacy=0x7f66c8160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:31.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.563+0000 7f66cce00640 1 -- 172.21.15.45:0/1672938642 >> v1:172.21.15.45:6789/0 conn(0x7f66c806e460 legacy=0x7f66c806e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:31.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.563+0000 7f66cce00640 1 -- 172.21.15.45:0/1672938642 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c815ccb0 con 0x7f66c80b58f0 2024-04-16T16:15:31.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.564+0000 7f66cce00640 1 --2- 172.21.15.45:0/1672938642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66c80b58f0 0x7f66c80b5d50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f66b4009870 tx=0x7f66b402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d65aa1aa5a33a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:31.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.565+0000 7f66c7e00640 1 -- 172.21.15.45:0/1672938642 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b402f850 con 0x7f66c80b58f0 2024-04-16T16:15:31.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.565+0000 7f66c7e00640 1 -- 172.21.15.45:0/1672938642 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b402f9f0 con 0x7f66c80b58f0 2024-04-16T16:15:31.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.565+0000 7f66c7e00640 1 -- 172.21.15.45:0/1672938642 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b402fdc0 con 0x7f66c80b58f0 2024-04-16T16:15:31.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.565+0000 7f66cec00640 1 -- 172.21.15.45:0/1672938642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66c80b58f0 msgr2=0x7f66c80b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:31.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.565+0000 7f66cec00640 1 --2- 172.21.15.45:0/1672938642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66c80b58f0 0x7f66c80b5d50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f66b4009870 tx=0x7f66b402ba30 comp rx=0 tx=0).stop 2024-04-16T16:15:31.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.565+0000 7f66cec00640 1 -- 172.21.15.45:0/1672938642 shutdown_connections 2024-04-16T16:15:31.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.565+0000 7f66cec00640 1 --2- 172.21.15.45:0/1672938642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66c80b58f0 0x7f66c80b5d50 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.565+0000 7f66cec00640 1 -- 172.21.15.45:0/1672938642 >> 172.21.15.45:0/1672938642 conn(0x7f66c806b930 msgr2=0x7f66c806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:31.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.565+0000 7f66cec00640 1 -- 172.21.15.45:0/1672938642 shutdown_connections 2024-04-16T16:15:31.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.566+0000 7f66cec00640 1 -- 172.21.15.45:0/1672938642 wait complete. 2024-04-16T16:15:31.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.567+0000 7f66cec00640 1 Processor -- start 2024-04-16T16:15:31.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.567+0000 7f66cec00640 1 -- start start 2024-04-16T16:15:31.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.568+0000 7f66cec00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66c806e460 0x7f66c8138ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:31.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.568+0000 7f66cec00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66c80b58f0 0x7f66c8144730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:31.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.568+0000 7f66cec00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66c81461b0 0x7f66c80cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:31.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.568+0000 7f66cec00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f66c80c5cd0 con 0x7f66c81461b0 2024-04-16T16:15:31.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.568+0000 7f66cec00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f66c80c5fb0 con 0x7f66c80b58f0 2024-04-16T16:15:31.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.568+0000 7f66cec00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f66c80c6290 con 0x7f66c806e460 2024-04-16T16:15:31.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.569+0000 7f66ce200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66c81461b0 0x7f66c80cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:31.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.569+0000 7f66ce200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66c81461b0 0x7f66c80cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38906/0 (socket says 172.21.15.45:38906) 2024-04-16T16:15:31.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.569+0000 7f66ce200640 1 -- 172.21.15.45:0/573059271 learned_addr learned my addr 172.21.15.45:0/573059271 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:31.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.569+0000 7f66cce00640 1 --2- 172.21.15.45:0/573059271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66c80b58f0 0x7f66c8144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:31.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.569+0000 7f66cd800640 1 --2- 172.21.15.45:0/573059271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66c806e460 0x7f66c8138ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:31.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.569+0000 7f66ce200640 1 -- 172.21.15.45:0/573059271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66c806e460 msgr2=0x7f66c8138ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:31.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.569+0000 7f66ce200640 1 --2- 172.21.15.45:0/573059271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66c806e460 0x7f66c8138ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.569+0000 7f66ce200640 1 -- 172.21.15.45:0/573059271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66c80b58f0 msgr2=0x7f66c8144730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:31.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.569+0000 7f66ce200640 1 --2- 172.21.15.45:0/573059271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66c80b58f0 0x7f66c8144730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.569+0000 7f66ce200640 1 -- 172.21.15.45:0/573059271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c81432e0 con 0x7f66c81461b0 2024-04-16T16:15:31.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.570+0000 7f66ce200640 1 --2- 172.21.15.45:0/573059271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66c81461b0 0x7f66c80cbd00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f66c000c780 tx=0x7f66c000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:31.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.570+0000 7f66c5600640 1 -- 172.21.15.45:0/573059271 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c0017070 con 0x7f66c81461b0 2024-04-16T16:15:31.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.570+0000 7f66c5600640 1 -- 172.21.15.45:0/573059271 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66c000a640 con 0x7f66c81461b0 2024-04-16T16:15:31.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.570+0000 7f66c5600640 1 -- 172.21.15.45:0/573059271 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c000aa30 con 0x7f66c81461b0 2024-04-16T16:15:31.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.570+0000 7f66cec00640 1 -- 172.21.15.45:0/573059271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c81435c0 con 0x7f66c81461b0 2024-04-16T16:15:31.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.570+0000 7f66cec00640 1 -- 172.21.15.45:0/573059271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c8143b80 con 0x7f66c81461b0 2024-04-16T16:15:31.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.571+0000 7f66cec00640 1 -- 172.21.15.45:0/573059271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c8075780 con 0x7f66c81461b0 2024-04-16T16:15:31.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.572+0000 7f66c5600640 1 -- 172.21.15.45:0/573059271 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f66c001a070 con 0x7f66c81461b0 2024-04-16T16:15:31.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.573+0000 7f66c5600640 1 --2- 172.21.15.45:0/573059271 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f66a806ca60 0x7f66a806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:31.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.573+0000 7f66c5600640 1 -- 172.21.15.45:0/573059271 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7f66c008b440 con 0x7f66c81461b0 2024-04-16T16:15:31.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.574+0000 7f66cd800640 1 --2- 172.21.15.45:0/573059271 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f66a806ca60 0x7f66a806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:31.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.575+0000 7f66cd800640 1 --2- 172.21.15.45:0/573059271 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f66a806ca60 0x7f66a806ef20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f66bc007fd0 tx=0x7f66bc009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:31.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.578+0000 7f66c5600640 1 -- 172.21.15.45:0/573059271 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f66c0058f70 con 0x7f66c81461b0 2024-04-16T16:15:31.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.784+0000 7f66cec00640 1 -- 172.21.15.45:0/573059271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs1_metadata", "var": "pg_num"} v 0) -- 0x7f66c8072500 con 0x7f66c81461b0 2024-04-16T16:15:31.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.785+0000 7f66c5600640 1 -- 172.21.15.45:0/573059271 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs1_metadata", "var": "pg_num"}]=0 v46) ==== 101+0+11 (secure 0 0 0) 0x7f66c005d270 con 0x7f66c81461b0 2024-04-16T16:15:31.784 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:15:31.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.788+0000 7f66cec00640 1 -- 172.21.15.45:0/573059271 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f66a806ca60 msgr2=0x7f66a806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:31.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.788+0000 7f66cec00640 1 --2- 172.21.15.45:0/573059271 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f66a806ca60 0x7f66a806ef20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f66bc007fd0 tx=0x7f66bc009040 comp rx=0 tx=0).stop 2024-04-16T16:15:31.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.788+0000 7f66cec00640 1 -- 172.21.15.45:0/573059271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66c81461b0 msgr2=0x7f66c80cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:31.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.788+0000 7f66cec00640 1 --2- 172.21.15.45:0/573059271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66c81461b0 0x7f66c80cbd00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f66c000c780 tx=0x7f66c000cc40 comp rx=0 tx=0).stop 2024-04-16T16:15:31.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.788+0000 7f66cec00640 1 -- 172.21.15.45:0/573059271 shutdown_connections 2024-04-16T16:15:31.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.788+0000 7f66cec00640 1 --2- 172.21.15.45:0/573059271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66c81461b0 0x7f66c80cbd00 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.788+0000 7f66cec00640 1 --2- 172.21.15.45:0/573059271 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f66a806ca60 0x7f66a806ef20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.788+0000 7f66cec00640 1 --2- 172.21.15.45:0/573059271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66c80b58f0 0x7f66c8144730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.788+0000 7f66cec00640 1 --2- 172.21.15.45:0/573059271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66c806e460 0x7f66c8138ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:31.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.788+0000 7f66cec00640 1 -- 172.21.15.45:0/573059271 >> 172.21.15.45:0/573059271 conn(0x7f66c806b930 msgr2=0x7f66c80b24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:31.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.789+0000 7f66cec00640 1 -- 172.21.15.45:0/573059271 shutdown_connections 2024-04-16T16:15:31.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:31.789+0000 7f66cec00640 1 -- 172.21.15.45:0/573059271 wait complete. 2024-04-16T16:15:31.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs1_data pg_num 2024-04-16T16:15:32.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.039+0000 7f715d400640 1 Processor -- start 2024-04-16T16:15:32.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.040+0000 7f715d400640 1 -- start start 2024-04-16T16:15:32.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.041+0000 7f715d400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f71580d18f0 0x7f71580ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.041+0000 7f715d400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f71580d1cd0 0x7f7158162dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.041+0000 7f715d400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71580b6810 con 0x7f71580b6310 2024-04-16T16:15:32.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.041+0000 7f715d400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f71580d29f0 con 0x7f71580d1cd0 2024-04-16T16:15:32.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.041+0000 7f715d400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f71580b6af0 con 0x7f71580d18f0 2024-04-16T16:15:32.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.042+0000 7f715ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f71580d1cd0 0x7f7158162dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.042+0000 7f715ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f71580d1cd0 0x7f7158162dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38910/0 (socket says 172.21.15.45:38910) 2024-04-16T16:15:32.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.042+0000 7f715ca00640 1 -- 172.21.15.45:0/3152372105 learned_addr learned my addr 172.21.15.45:0/3152372105 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:32.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.042+0000 7f715ca00640 1 -- 172.21.15.45:0/3152372105 >> v2:172.21.15.103:3300/0 conn(0x7f71580d18f0 msgr2=0x7f71580ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:32.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.042+0000 7f715ca00640 1 --2- 172.21.15.45:0/3152372105 >> v2:172.21.15.103:3300/0 conn(0x7f71580d18f0 0x7f71580ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.042+0000 7f715ca00640 1 -- 172.21.15.45:0/3152372105 >> v1:172.21.15.103:6789/0 conn(0x7f71580b6310 legacy=0x7f71580d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:32.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.042+0000 7f715ca00640 1 -- 172.21.15.45:0/3152372105 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71580b6dd0 con 0x7f71580d1cd0 2024-04-16T16:15:32.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.044+0000 7f715ca00640 1 --2- 172.21.15.45:0/3152372105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71580d1cd0 0x7f7158162dc0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7158073f70 tx=0x7f715002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463853fe1f44900c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:32.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.044+0000 7f7157400640 1 -- 172.21.15.45:0/3152372105 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715002c960 con 0x7f71580d1cd0 2024-04-16T16:15:32.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.044+0000 7f7157400640 1 -- 172.21.15.45:0/3152372105 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f715002cb00 con 0x7f71580d1cd0 2024-04-16T16:15:32.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.044+0000 7f7157400640 1 -- 172.21.15.45:0/3152372105 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71500343f0 con 0x7f71580d1cd0 2024-04-16T16:15:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.045+0000 7f715d400640 1 -- 172.21.15.45:0/3152372105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71580d1cd0 msgr2=0x7f7158162dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.045+0000 7f715d400640 1 --2- 172.21.15.45:0/3152372105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71580d1cd0 0x7f7158162dc0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7158073f70 tx=0x7f715002bdc0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.045+0000 7f715d400640 1 -- 172.21.15.45:0/3152372105 shutdown_connections 2024-04-16T16:15:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.045+0000 7f715d400640 1 --2- 172.21.15.45:0/3152372105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71580d1cd0 0x7f7158162dc0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.045+0000 7f715d400640 1 --2- 172.21.15.45:0/3152372105 >> v2:172.21.15.103:3300/0 conn(0x7f71580d18f0 0x7f71580ba220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.045+0000 7f715d400640 1 -- 172.21.15.45:0/3152372105 >> 172.21.15.45:0/3152372105 conn(0x7f715806cf90 msgr2=0x7f715806d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.045+0000 7f715d400640 1 -- 172.21.15.45:0/3152372105 shutdown_connections 2024-04-16T16:15:32.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.045+0000 7f715d400640 1 -- 172.21.15.45:0/3152372105 wait complete. 2024-04-16T16:15:32.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.048+0000 7f715d400640 1 Processor -- start 2024-04-16T16:15:32.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.048+0000 7f715d400640 1 -- start start 2024-04-16T16:15:32.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.049+0000 7f715d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71580b6310 0x7f715815af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.049+0000 7f715d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71580d18f0 0x7f71580cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.049+0000 7f715d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71580d1cd0 0x7f7158160de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.049+0000 7f715d400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7158163300 con 0x7f71580d18f0 2024-04-16T16:15:32.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.049+0000 7f715d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f71580ba760 con 0x7f71580d1cd0 2024-04-16T16:15:32.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.049+0000 7f715d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f71581f8b70 con 0x7f71580b6310 2024-04-16T16:15:32.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.050+0000 7f714fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71580d18f0 0x7f71580cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.050+0000 7f714fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71580d18f0 0x7f71580cad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38916/0 (socket says 172.21.15.45:38916) 2024-04-16T16:15:32.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.050+0000 7f714fe00640 1 -- 172.21.15.45:0/2120586342 learned_addr learned my addr 172.21.15.45:0/2120586342 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:32.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.050+0000 7f7157e00640 1 --2- 172.21.15.45:0/2120586342 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71580b6310 0x7f715815af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.050+0000 7f715ca00640 1 --2- 172.21.15.45:0/2120586342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71580d1cd0 0x7f7158160de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.050+0000 7f714fe00640 1 -- 172.21.15.45:0/2120586342 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71580b6310 msgr2=0x7f715815af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.050+0000 7f714fe00640 1 --2- 172.21.15.45:0/2120586342 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71580b6310 0x7f715815af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.050+0000 7f714fe00640 1 -- 172.21.15.45:0/2120586342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71580d1cd0 msgr2=0x7f7158160de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.050+0000 7f714fe00640 1 --2- 172.21.15.45:0/2120586342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71580d1cd0 0x7f7158160de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.050+0000 7f714fe00640 1 -- 172.21.15.45:0/2120586342 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71581f7ff0 con 0x7f71580d18f0 2024-04-16T16:15:32.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.051+0000 7f714fe00640 1 --2- 172.21.15.45:0/2120586342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71580d18f0 0x7f71580cad80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f714000c400 tx=0x7f714000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:32.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.051+0000 7f7154c00640 1 -- 172.21.15.45:0/2120586342 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7140011790 con 0x7f71580d18f0 2024-04-16T16:15:32.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.051+0000 7f7154c00640 1 -- 172.21.15.45:0/2120586342 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7140017070 con 0x7f71580d18f0 2024-04-16T16:15:32.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.051+0000 7f7154c00640 1 -- 172.21.15.45:0/2120586342 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7140011bb0 con 0x7f71580d18f0 2024-04-16T16:15:32.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.051+0000 7f715d400640 1 -- 172.21.15.45:0/2120586342 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71581f82d0 con 0x7f71580d18f0 2024-04-16T16:15:32.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.051+0000 7f715d400640 1 -- 172.21.15.45:0/2120586342 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71581f8890 con 0x7f71580d18f0 2024-04-16T16:15:32.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.052+0000 7f715d400640 1 -- 172.21.15.45:0/2120586342 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7158075780 con 0x7f71580d18f0 2024-04-16T16:15:32.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.053+0000 7f7154c00640 1 -- 172.21.15.45:0/2120586342 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f714001c460 con 0x7f71580d18f0 2024-04-16T16:15:32.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.054+0000 7f7154c00640 1 --2- 172.21.15.45:0/2120586342 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f713406ca60 0x7f713406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.054+0000 7f7154c00640 1 -- 172.21.15.45:0/2120586342 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7f714008be30 con 0x7f71580d18f0 2024-04-16T16:15:32.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.055+0000 7f7157e00640 1 --2- 172.21.15.45:0/2120586342 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f713406ca60 0x7f713406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.056+0000 7f7157e00640 1 --2- 172.21.15.45:0/2120586342 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f713406ca60 0x7f713406ef20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7148009890 tx=0x7f7148008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:32.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.059+0000 7f7154c00640 1 -- 172.21.15.45:0/2120586342 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7140059f10 con 0x7f71580d18f0 2024-04-16T16:15:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.264+0000 7f715d400640 1 -- 172.21.15.45:0/2120586342 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs1_data", "var": "pg_num"} v 0) -- 0x7f71580caab0 con 0x7f71580d18f0 2024-04-16T16:15:32.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.265+0000 7f7154c00640 1 -- 172.21.15.45:0/2120586342 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs1_data", "var": "pg_num"}]=0 v46) ==== 97+0+11 (secure 0 0 0) 0x7f7140059810 con 0x7f71580d18f0 2024-04-16T16:15:32.264 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:15:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.267+0000 7f715d400640 1 -- 172.21.15.45:0/2120586342 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f713406ca60 msgr2=0x7f713406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.268+0000 7f715d400640 1 --2- 172.21.15.45:0/2120586342 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f713406ca60 0x7f713406ef20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7148009890 tx=0x7f7148008040 comp rx=0 tx=0).stop 2024-04-16T16:15:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.268+0000 7f715d400640 1 -- 172.21.15.45:0/2120586342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71580d18f0 msgr2=0x7f71580cad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.268+0000 7f715d400640 1 --2- 172.21.15.45:0/2120586342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71580d18f0 0x7f71580cad80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f714000c400 tx=0x7f714000c8c0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.268+0000 7f715d400640 1 -- 172.21.15.45:0/2120586342 shutdown_connections 2024-04-16T16:15:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.268+0000 7f715d400640 1 --2- 172.21.15.45:0/2120586342 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f713406ca60 0x7f713406ef20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.268+0000 7f715d400640 1 --2- 172.21.15.45:0/2120586342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71580d1cd0 0x7f7158160de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.268+0000 7f715d400640 1 --2- 172.21.15.45:0/2120586342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71580d18f0 0x7f71580cad80 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.268+0000 7f715d400640 1 --2- 172.21.15.45:0/2120586342 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71580b6310 0x7f715815af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.268+0000 7f715d400640 1 -- 172.21.15.45:0/2120586342 >> 172.21.15.45:0/2120586342 conn(0x7f715806cf90 msgr2=0x7f71580b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.268+0000 7f715d400640 1 -- 172.21.15.45:0/2120586342 shutdown_connections 2024-04-16T16:15:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.268+0000 7f715d400640 1 -- 172.21.15.45:0/2120586342 wait complete. 2024-04-16T16:15:32.287 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-16T16:15:32.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.502+0000 7f684d800640 1 Processor -- start 2024-04-16T16:15:32.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.502+0000 7f684d800640 1 -- start start 2024-04-16T16:15:32.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.504+0000 7f684d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6848162270 0x7f68480d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.504+0000 7f684d800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f68480d52d0 0x7f684806ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.504+0000 7f684d800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68480d3080 con 0x7f6848161e90 2024-04-16T16:15:32.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.504+0000 7f684d800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f68480d2e30 con 0x7f68480d52d0 2024-04-16T16:15:32.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.504+0000 7f684d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f68480d3360 con 0x7f6848162270 2024-04-16T16:15:32.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.505+0000 7f6847400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6848162270 0x7f68480d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.505+0000 7f6847400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6848162270 0x7f68480d3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:43446/0 (socket says 172.21.15.45:43446) 2024-04-16T16:15:32.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.505+0000 7f684ce00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f68480d52d0 0x7f684806ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.505+0000 7f6847400640 1 -- 172.21.15.45:0/2224479811 learned_addr learned my addr 172.21.15.45:0/2224479811 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:32.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.506+0000 7f6847400640 1 -- 172.21.15.45:0/2224479811 >> v2:172.21.15.103:3300/0 conn(0x7f68480d52d0 msgr2=0x7f684806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:32.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.506+0000 7f6847400640 1 --2- 172.21.15.45:0/2224479811 >> v2:172.21.15.103:3300/0 conn(0x7f68480d52d0 0x7f684806ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.506+0000 7f6847400640 1 -- 172.21.15.45:0/2224479811 >> v1:172.21.15.103:6789/0 conn(0x7f6848161e90 legacy=0x7f68480cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.506+0000 7f6847400640 1 -- 172.21.15.45:0/2224479811 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68480d3640 con 0x7f6848162270 2024-04-16T16:15:32.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.508+0000 7f6847400640 1 --2- 172.21.15.45:0/2224479811 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6848162270 0x7f68480d3e40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6830009e50 tx=0x7f683002bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f0a926901fac0e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:32.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.509+0000 7f6846a00640 1 -- 172.21.15.45:0/2224479811 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683002cb90 con 0x7f6848162270 2024-04-16T16:15:32.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.509+0000 7f6846a00640 1 -- 172.21.15.45:0/2224479811 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f683002cd30 con 0x7f6848162270 2024-04-16T16:15:32.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.509+0000 7f6846a00640 1 -- 172.21.15.45:0/2224479811 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6830034610 con 0x7f6848162270 2024-04-16T16:15:32.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.509+0000 7f684d800640 1 -- 172.21.15.45:0/2224479811 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6848162270 msgr2=0x7f68480d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.509+0000 7f684d800640 1 --2- 172.21.15.45:0/2224479811 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6848162270 0x7f68480d3e40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6830009e50 tx=0x7f683002bff0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.509+0000 7f684d800640 1 -- 172.21.15.45:0/2224479811 shutdown_connections 2024-04-16T16:15:32.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.509+0000 7f684d800640 1 --2- 172.21.15.45:0/2224479811 >> v2:172.21.15.103:3300/0 conn(0x7f68480d52d0 0x7f684806ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.509+0000 7f684d800640 1 --2- 172.21.15.45:0/2224479811 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6848162270 0x7f68480d3e40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.509+0000 7f684d800640 1 -- 172.21.15.45:0/2224479811 >> 172.21.15.45:0/2224479811 conn(0x7f68480d1940 msgr2=0x7f68480d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:32.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.510+0000 7f684d800640 1 -- 172.21.15.45:0/2224479811 shutdown_connections 2024-04-16T16:15:32.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.510+0000 7f684d800640 1 -- 172.21.15.45:0/2224479811 wait complete. 2024-04-16T16:15:32.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.511+0000 7f684d800640 1 Processor -- start 2024-04-16T16:15:32.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.511+0000 7f684d800640 1 -- start start 2024-04-16T16:15:32.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.512+0000 7f684d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68480d52d0 0x7f684814ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.512+0000 7f684d800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6848161e90 0x7f684815b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.512+0000 7f684d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6848162270 0x7f684815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.512+0000 7f684d800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f68480d4500 con 0x7f68480d52d0 2024-04-16T16:15:32.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.512+0000 7f684d800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f68480d4380 con 0x7f6848161e90 2024-04-16T16:15:32.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.512+0000 7f684d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f68480d4680 con 0x7f6848162270 2024-04-16T16:15:32.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.513+0000 7f6847e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68480d52d0 0x7f684814ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.513+0000 7f6847e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68480d52d0 0x7f684814ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38920/0 (socket says 172.21.15.45:38920) 2024-04-16T16:15:32.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.513+0000 7f6847e00640 1 -- 172.21.15.45:0/4236817279 learned_addr learned my addr 172.21.15.45:0/4236817279 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:32.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.513+0000 7f684ce00640 1 --2- 172.21.15.45:0/4236817279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6848162270 0x7f684815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.513+0000 7f6847400640 1 --2- 172.21.15.45:0/4236817279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6848161e90 0x7f684815b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.513+0000 7f6847e00640 1 -- 172.21.15.45:0/4236817279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6848162270 msgr2=0x7f684815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.513+0000 7f6847e00640 1 --2- 172.21.15.45:0/4236817279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6848162270 0x7f684815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.513+0000 7f6847e00640 1 -- 172.21.15.45:0/4236817279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6848161e90 msgr2=0x7f684815b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.513+0000 7f6847e00640 1 --2- 172.21.15.45:0/4236817279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6848161e90 0x7f684815b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.513+0000 7f6847e00640 1 -- 172.21.15.45:0/4236817279 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68480ce9e0 con 0x7f68480d52d0 2024-04-16T16:15:32.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.514+0000 7f6847e00640 1 --2- 172.21.15.45:0/4236817279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68480d52d0 0x7f684814ff00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f683800c400 tx=0x7f683800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:32.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.514+0000 7f683fe00640 1 -- 172.21.15.45:0/4236817279 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68380052d0 con 0x7f68480d52d0 2024-04-16T16:15:32.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.514+0000 7f683fe00640 1 -- 172.21.15.45:0/4236817279 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6838005490 con 0x7f68480d52d0 2024-04-16T16:15:32.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.514+0000 7f684d800640 1 -- 172.21.15.45:0/4236817279 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68480cc510 con 0x7f68480d52d0 2024-04-16T16:15:32.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.514+0000 7f683fe00640 1 -- 172.21.15.45:0/4236817279 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838007260 con 0x7f68480d52d0 2024-04-16T16:15:32.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.514+0000 7f684d800640 1 -- 172.21.15.45:0/4236817279 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68481591b0 con 0x7f68480d52d0 2024-04-16T16:15:32.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.515+0000 7f684d800640 1 -- 172.21.15.45:0/4236817279 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6848075780 con 0x7f68480d52d0 2024-04-16T16:15:32.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.516+0000 7f683fe00640 1 -- 172.21.15.45:0/4236817279 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f6838007400 con 0x7f68480d52d0 2024-04-16T16:15:32.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.517+0000 7f683fe00640 1 --2- 172.21.15.45:0/4236817279 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f682406ca90 0x7f682406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.517+0000 7f683fe00640 1 -- 172.21.15.45:0/4236817279 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7f683808b400 con 0x7f68480d52d0 2024-04-16T16:15:32.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.518+0000 7f6847400640 1 --2- 172.21.15.45:0/4236817279 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f682406ca90 0x7f682406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.519+0000 7f6847400640 1 --2- 172.21.15.45:0/4236817279 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f682406ca90 0x7f682406ef50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6830035000 tx=0x7f6830030000 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:32.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.522+0000 7f683fe00640 1 -- 172.21.15.45:0/4236817279 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6838059570 con 0x7f68480d52d0 2024-04-16T16:15:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.730+0000 7f684d800640 1 -- 172.21.15.45:0/4236817279 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f684806dae0 con 0x7f68480d52d0 2024-04-16T16:15:32.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.731+0000 7f683fe00640 1 -- 172.21.15.45:0/4236817279 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v46) ==== 84+0+201 (secure 0 0 0) 0x7f6838058e70 con 0x7f68480d52d0 2024-04-16T16:15:32.730 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:32.730 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2024-04-16T16:15:32.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:15:32.730 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2024-04-16T16:15:32.730 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2024-04-16T16:15:32.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:15:32.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:15:32.731 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 6, 2024-04-16T16:15:32.731 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": "fs1_metadata" 2024-04-16T16:15:32.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:15:32.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:15:32.731 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 7, 2024-04-16T16:15:32.731 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": "fs1_data" 2024-04-16T16:15:32.731 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:15:32.731 INFO:teuthology.orchestra.run.smithi045.stdout:] 2024-04-16T16:15:32.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.734+0000 7f684d800640 1 -- 172.21.15.45:0/4236817279 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f682406ca90 msgr2=0x7f682406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.734+0000 7f684d800640 1 --2- 172.21.15.45:0/4236817279 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f682406ca90 0x7f682406ef50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6830035000 tx=0x7f6830030000 comp rx=0 tx=0).stop 2024-04-16T16:15:32.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.734+0000 7f684d800640 1 -- 172.21.15.45:0/4236817279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68480d52d0 msgr2=0x7f684814ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.734+0000 7f684d800640 1 --2- 172.21.15.45:0/4236817279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68480d52d0 0x7f684814ff00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f683800c400 tx=0x7f683800c8c0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.734+0000 7f684d800640 1 -- 172.21.15.45:0/4236817279 shutdown_connections 2024-04-16T16:15:32.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.734+0000 7f684d800640 1 --2- 172.21.15.45:0/4236817279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6848162270 0x7f684815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.734+0000 7f684d800640 1 --2- 172.21.15.45:0/4236817279 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f682406ca90 0x7f682406ef50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.734+0000 7f684d800640 1 --2- 172.21.15.45:0/4236817279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6848161e90 0x7f684815b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.734+0000 7f684d800640 1 --2- 172.21.15.45:0/4236817279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68480d52d0 0x7f684814ff00 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.734+0000 7f684d800640 1 -- 172.21.15.45:0/4236817279 >> 172.21.15.45:0/4236817279 conn(0x7f68480d1940 msgr2=0x7f68480b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:32.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.734+0000 7f684d800640 1 -- 172.21.15.45:0/4236817279 shutdown_connections 2024-04-16T16:15:32.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.734+0000 7f684d800640 1 -- 172.21.15.45:0/4236817279 wait complete. 2024-04-16T16:15:32.752 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'fs2' 2024-04-16T16:15:32.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create fs2_metadata --pg_num_min 64 2024-04-16T16:15:32.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.973+0000 7f3a06800640 1 Processor -- start 2024-04-16T16:15:32.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.973+0000 7f3a06800640 1 -- start start 2024-04-16T16:15:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.974+0000 7f3a06800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3a000d0e80 0x7f3a000b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.975+0000 7f3a06800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a000ccb70 con 0x7f3a0015b920 2024-04-16T16:15:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.975+0000 7f3a06800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a000cce50 con 0x7f3a000afda0 2024-04-16T16:15:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.975+0000 7f3a06800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3a0006e7a0 con 0x7f3a000d0e80 2024-04-16T16:15:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.975+0000 7f39ff400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3a0015b920 0x7f3a000d0770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50120/0 (socket says 172.21.15.45:50120) 2024-04-16T16:15:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.975+0000 7f39ff400640 1 -- 172.21.15.45:0/1016461956 learned_addr learned my addr 172.21.15.45:0/1016461956 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.975+0000 7f39fea00640 1 --2- 172.21.15.45:0/1016461956 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3a000d0e80 0x7f3a000b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.975+0000 7f39fe000640 1 -- 172.21.15.45:0/1016461956 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2443992564 0 0) 0x7f3a000ccb70 con 0x7f3a0015b920 2024-04-16T16:15:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.975+0000 7f39fe000640 1 -- 172.21.15.45:0/1016461956 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39e4003520 con 0x7f3a0015b920 2024-04-16T16:15:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.976+0000 7f39fe000640 1 -- 172.21.15.45:0/1016461956 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3207873074 0 0) 0x7f39e4003520 con 0x7f3a0015b920 2024-04-16T16:15:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.976+0000 7f39fe000640 1 -- 172.21.15.45:0/1016461956 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3a000d0e80 msgr2=0x7f3a000b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.976+0000 7f39fe000640 1 --2- 172.21.15.45:0/1016461956 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3a000d0e80 0x7f3a000b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.976+0000 7f39fe000640 1 -- 172.21.15.45:0/1016461956 >> v1:172.21.15.103:6789/0 conn(0x7f3a000afda0 legacy=0x7f3a000ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.976+0000 7f39fe000640 1 -- 172.21.15.45:0/1016461956 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0006e430 con 0x7f3a0015b920 2024-04-16T16:15:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.976+0000 7f39fe000640 1 -- 172.21.15.45:0/1016461956 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f39f0001670 con 0x7f3a0015b920 2024-04-16T16:15:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.976+0000 7f39fe000640 1 -- 172.21.15.45:0/1016461956 >> v1:172.21.15.45:6789/0 conn(0x7f3a0015b920 legacy=0x7f3a000d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.976+0000 7f39fe000640 1 --2- 172.21.15.45:0/1016461956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39e40039e0 0x7f39e4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.976+0000 7f39fe000640 1 --2- 172.21.15.45:0/1016461956 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3a000d0e80 0x7f39e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.976+0000 7f39fe000640 1 --2- 172.21.15.45:0/1016461956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e4004d40 0x7f39e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.976+0000 7f39fe000640 1 -- 172.21.15.45:0/1016461956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0006e7a0 con 0x7f39e4004d40 2024-04-16T16:15:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.976+0000 7f39fe000640 1 -- 172.21.15.45:0/1016461956 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f39e400ba70 con 0x7f39e40039e0 2024-04-16T16:15:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.976+0000 7f39fe000640 1 -- 172.21.15.45:0/1016461956 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f39e400bc70 con 0x7f3a000d0e80 2024-04-16T16:15:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.976+0000 7f39ffe00640 1 --2- 172.21.15.45:0/1016461956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e4004d40 0x7f39e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.977+0000 7f39fea00640 1 --2- 172.21.15.45:0/1016461956 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3a000d0e80 0x7f39e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.977+0000 7f39ffe00640 1 -- 172.21.15.45:0/1016461956 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3a000d0e80 msgr2=0x7f39e4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:32.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.977+0000 7f39ffe00640 1 --2- 172.21.15.45:0/1016461956 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3a000d0e80 0x7f39e4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.977+0000 7f39ffe00640 1 -- 172.21.15.45:0/1016461956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39e40039e0 msgr2=0x7f39e4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:32.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.977+0000 7f39ffe00640 1 --2- 172.21.15.45:0/1016461956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39e40039e0 0x7f39e4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.977+0000 7f39ffe00640 1 -- 172.21.15.45:0/1016461956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e4003520 con 0x7f39e4004d40 2024-04-16T16:15:32.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.977+0000 7f39fea00640 1 --2- 172.21.15.45:0/1016461956 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3a000d0e80 0x7f39e4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.978+0000 7f39ffe00640 1 --2- 172.21.15.45:0/1016461956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e4004d40 0x7f39e40051c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f39f8001720 tx=0x7f39e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a3b0426c86cced server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.978+0000 7f39fe000640 1 -- 172.21.15.45:0/1016461956 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f8001370 con 0x7f39e4004d40 2024-04-16T16:15:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.978+0000 7f39fe000640 1 -- 172.21.15.45:0/1016461956 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39f80052d0 con 0x7f39e4004d40 2024-04-16T16:15:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.978+0000 7f39fe000640 1 -- 172.21.15.45:0/1016461956 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f8003830 con 0x7f39e4004d40 2024-04-16T16:15:32.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.979+0000 7f3a06800640 1 -- 172.21.15.45:0/1016461956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e4004d40 msgr2=0x7f39e40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.979+0000 7f3a06800640 1 --2- 172.21.15.45:0/1016461956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e4004d40 0x7f39e40051c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f39f8001720 tx=0x7f39e4005830 comp rx=0 tx=0).stop 2024-04-16T16:15:32.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.979+0000 7f39ffe00640 1 -- 172.21.15.45:0/1016461956 reap_dead start 2024-04-16T16:15:32.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.979+0000 7f3a06800640 1 -- 172.21.15.45:0/1016461956 shutdown_connections 2024-04-16T16:15:32.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.979+0000 7f3a06800640 1 -- 172.21.15.45:0/1016461956 >> 172.21.15.45:0/1016461956 conn(0x7f3a000b3970 msgr2=0x7f3a000b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:32.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.979+0000 7f3a06800640 1 -- 172.21.15.45:0/1016461956 shutdown_connections 2024-04-16T16:15:32.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.979+0000 7f3a06800640 1 -- 172.21.15.45:0/1016461956 wait complete. 2024-04-16T16:15:32.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.981+0000 7f3a06800640 1 Processor -- start 2024-04-16T16:15:32.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.981+0000 7f3a06800640 1 -- start start 2024-04-16T16:15:32.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.982+0000 7f3a06800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39e4004d40 0x7f3a001f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.982+0000 7f3a06800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a0013f2a0 0x7f3a001ed490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.982+0000 7f3a06800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a00156410 0x7f3a0015a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.982+0000 7f3a06800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3a000b51f0 con 0x7f3a0013f2a0 2024-04-16T16:15:32.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.982+0000 7f3a06800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3a000b5070 con 0x7f3a00156410 2024-04-16T16:15:32.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.982+0000 7f3a06800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3a0006ec30 con 0x7f39e4004d40 2024-04-16T16:15:32.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.982+0000 7f39fea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a0013f2a0 0x7f3a001ed490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.982+0000 7f39fea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a0013f2a0 0x7f3a001ed490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38938/0 (socket says 172.21.15.45:38938) 2024-04-16T16:15:32.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.982+0000 7f39fea00640 1 -- 172.21.15.45:0/2686852185 learned_addr learned my addr 172.21.15.45:0/2686852185 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:32.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.982+0000 7f39fea00640 1 -- 172.21.15.45:0/2686852185 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39e4004d40 msgr2=0x7f3a001f4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.982+0000 7f39fea00640 1 --2- 172.21.15.45:0/2686852185 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39e4004d40 0x7f3a001f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.982+0000 7f39fea00640 1 -- 172.21.15.45:0/2686852185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a00156410 msgr2=0x7f3a0015a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:32.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.982+0000 7f39fea00640 1 --2- 172.21.15.45:0/2686852185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a00156410 0x7f3a0015a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:32.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.982+0000 7f39fea00640 1 -- 172.21.15.45:0/2686852185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a001f4580 con 0x7f3a0013f2a0 2024-04-16T16:15:32.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.983+0000 7f39fea00640 1 --2- 172.21.15.45:0/2686852185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a0013f2a0 0x7f3a001ed490 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f39ec00f350 tx=0x7f39ec00f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:32.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.983+0000 7f39f7400640 1 -- 172.21.15.45:0/2686852185 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ec003b90 con 0x7f3a0013f2a0 2024-04-16T16:15:32.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.983+0000 7f39f7400640 1 -- 172.21.15.45:0/2686852185 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39ec003d30 con 0x7f3a0013f2a0 2024-04-16T16:15:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.983+0000 7f39f7400640 1 -- 172.21.15.45:0/2686852185 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ec004220 con 0x7f3a0013f2a0 2024-04-16T16:15:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.983+0000 7f3a06800640 1 -- 172.21.15.45:0/2686852185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a0013f760 con 0x7f3a0013f2a0 2024-04-16T16:15:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.983+0000 7f3a06800640 1 -- 172.21.15.45:0/2686852185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a0013fd20 con 0x7f3a0013f2a0 2024-04-16T16:15:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.984+0000 7f3a06800640 1 -- 172.21.15.45:0/2686852185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a00075780 con 0x7f3a0013f2a0 2024-04-16T16:15:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.985+0000 7f39f7400640 1 -- 172.21.15.45:0/2686852185 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f39ec01e020 con 0x7f3a0013f2a0 2024-04-16T16:15:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.986+0000 7f39f7400640 1 --2- 172.21.15.45:0/2686852185 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f39e006ca90 0x7f39e006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.986+0000 7f39f7400640 1 -- 172.21.15.45:0/2686852185 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6127+0+0 (secure 0 0 0) 0x7f39ec08c960 con 0x7f3a0013f2a0 2024-04-16T16:15:32.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.987+0000 7f39ff400640 1 --2- 172.21.15.45:0/2686852185 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f39e006ca90 0x7f39e006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:32.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.988+0000 7f39ff400640 1 --2- 172.21.15.45:0/2686852185 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f39e006ca90 0x7f39e006ef50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3a000b57c0 tx=0x7f39f0001500 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:32.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:32.991+0000 7f39f7400640 1 -- 172.21.15.45:0/2686852185 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3a001f4580 con 0x7f3a0013f2a0 2024-04-16T16:15:33.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:33.197+0000 7f3a06800640 1 -- 172.21.15.45:0/2686852185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fs2_metadata", "pg_num_min": 64} v 0) -- 0x7f3a000d0a20 con 0x7f3a0013f2a0 2024-04-16T16:15:33.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:33.772+0000 7f39f7400640 1 -- 172.21.15.45:0/2686852185 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fs2_metadata", "pg_num_min": 64}]=0 pool 'fs2_metadata' created v47) ==== 132+0+0 (secure 0 0 0) 0x7f39ec012070 con 0x7f3a0013f2a0 2024-04-16T16:15:33.772 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'fs2_metadata' created 2024-04-16T16:15:33.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:33.776+0000 7f3a06800640 1 -- 172.21.15.45:0/2686852185 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f39e006ca90 msgr2=0x7f39e006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:33.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:33.776+0000 7f3a06800640 1 --2- 172.21.15.45:0/2686852185 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f39e006ca90 0x7f39e006ef50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3a000b57c0 tx=0x7f39f0001500 comp rx=0 tx=0).stop 2024-04-16T16:15:33.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:33.776+0000 7f3a06800640 1 -- 172.21.15.45:0/2686852185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a0013f2a0 msgr2=0x7f3a001ed490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:33.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:33.776+0000 7f3a06800640 1 --2- 172.21.15.45:0/2686852185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a0013f2a0 0x7f3a001ed490 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f39ec00f350 tx=0x7f39ec00f810 comp rx=0 tx=0).stop 2024-04-16T16:15:33.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:33.776+0000 7f3a06800640 1 -- 172.21.15.45:0/2686852185 shutdown_connections 2024-04-16T16:15:33.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:33.776+0000 7f3a06800640 1 --2- 172.21.15.45:0/2686852185 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f39e006ca90 0x7f39e006ef50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:33.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:33.776+0000 7f3a06800640 1 --2- 172.21.15.45:0/2686852185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a00156410 0x7f3a0015a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:33.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:33.776+0000 7f3a06800640 1 --2- 172.21.15.45:0/2686852185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a0013f2a0 0x7f3a001ed490 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:33.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:33.776+0000 7f3a06800640 1 --2- 172.21.15.45:0/2686852185 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39e4004d40 0x7f3a001f4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:33.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:33.776+0000 7f3a06800640 1 -- 172.21.15.45:0/2686852185 >> 172.21.15.45:0/2686852185 conn(0x7f3a000b3970 msgr2=0x7f3a000cfa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:33.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:33.776+0000 7f3a06800640 1 -- 172.21.15.45:0/2686852185 shutdown_connections 2024-04-16T16:15:33.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:33.776+0000 7f3a06800640 1 -- 172.21.15.45:0/2686852185 wait complete. 2024-04-16T16:15:33.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create fs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-16T16:15:34.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.029+0000 7ff487200640 1 Processor -- start 2024-04-16T16:15:34.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.029+0000 7ff487200640 1 -- start start 2024-04-16T16:15:34.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.031+0000 7ff487200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff48015c7c0 0x7ff480072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:34.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.031+0000 7ff487200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff4800d10e0 0x7ff48006f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:34.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.031+0000 7ff487200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4800b5530 con 0x7ff4800b4930 2024-04-16T16:15:34.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.031+0000 7ff487200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff4800b52e0 con 0x7ff4800d10e0 2024-04-16T16:15:34.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.031+0000 7ff487200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff4800d0260 con 0x7ff48015c7c0 2024-04-16T16:15:34.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.032+0000 7ff47fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff48015c7c0 0x7ff480072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:34.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.032+0000 7ff47fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff48015c7c0 0x7ff480072e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:37210/0 (socket says 172.21.15.45:37210) 2024-04-16T16:15:34.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.032+0000 7ff47fe00640 1 -- 172.21.15.45:0/3049415269 learned_addr learned my addr 172.21.15.45:0/3049415269 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:34.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.032+0000 7ff47f400640 1 --1- 172.21.15.45:0/3049415269 >> v1:172.21.15.103:6789/0 conn(0x7ff4800b4930 0x7ff4800d09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:60576/0 (socket says 172.21.15.45:60576) 2024-04-16T16:15:34.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.032+0000 7ff485200640 1 --2- 172.21.15.45:0/3049415269 >> v2:172.21.15.103:3300/0 conn(0x7ff4800d10e0 0x7ff48006f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:34.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.033+0000 7ff47fe00640 1 -- 172.21.15.45:0/3049415269 >> v2:172.21.15.103:3300/0 conn(0x7ff4800d10e0 msgr2=0x7ff48006f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:34.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.033+0000 7ff47fe00640 1 --2- 172.21.15.45:0/3049415269 >> v2:172.21.15.103:3300/0 conn(0x7ff4800d10e0 0x7ff48006f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:34.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.033+0000 7ff47fe00640 1 -- 172.21.15.45:0/3049415269 >> v1:172.21.15.103:6789/0 conn(0x7ff4800b4930 legacy=0x7ff4800d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:34.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.033+0000 7ff47ea00640 1 -- 172.21.15.45:0/3049415269 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2155405358 0 0) 0x7ff4800b5530 con 0x7ff4800b4930 2024-04-16T16:15:34.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.033+0000 7ff47fe00640 1 -- 172.21.15.45:0/3049415269 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4800b5810 con 0x7ff48015c7c0 2024-04-16T16:15:34.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.034+0000 7ff47fe00640 1 --2- 172.21.15.45:0/3049415269 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff48015c7c0 0x7ff480072e10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff470009e50 tx=0x7ff47002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c85bc23adef2d51 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:34.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.034+0000 7ff47ea00640 1 -- 172.21.15.45:0/3049415269 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47002c9e0 con 0x7ff48015c7c0 2024-04-16T16:15:34.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.035+0000 7ff47ea00640 1 -- 172.21.15.45:0/3049415269 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff47002cb80 con 0x7ff48015c7c0 2024-04-16T16:15:34.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.035+0000 7ff47ea00640 1 -- 172.21.15.45:0/3049415269 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4700343f0 con 0x7ff48015c7c0 2024-04-16T16:15:34.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.035+0000 7ff487200640 1 -- 172.21.15.45:0/3049415269 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff48015c7c0 msgr2=0x7ff480072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:34.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.035+0000 7ff487200640 1 --2- 172.21.15.45:0/3049415269 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff48015c7c0 0x7ff480072e10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff470009e50 tx=0x7ff47002be40 comp rx=0 tx=0).stop 2024-04-16T16:15:34.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.035+0000 7ff487200640 1 -- 172.21.15.45:0/3049415269 shutdown_connections 2024-04-16T16:15:34.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.035+0000 7ff487200640 1 --2- 172.21.15.45:0/3049415269 >> v2:172.21.15.103:3300/0 conn(0x7ff4800d10e0 0x7ff48006f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:34.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.035+0000 7ff487200640 1 --2- 172.21.15.45:0/3049415269 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff48015c7c0 0x7ff480072e10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:34.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.035+0000 7ff487200640 1 -- 172.21.15.45:0/3049415269 >> 172.21.15.45:0/3049415269 conn(0x7ff4800b3970 msgr2=0x7ff4800b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:34.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.035+0000 7ff487200640 1 -- 172.21.15.45:0/3049415269 shutdown_connections 2024-04-16T16:15:34.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.035+0000 7ff487200640 1 -- 172.21.15.45:0/3049415269 wait complete. 2024-04-16T16:15:34.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.037+0000 7ff487200640 1 Processor -- start 2024-04-16T16:15:34.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.037+0000 7ff487200640 1 -- start start 2024-04-16T16:15:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.038+0000 7ff487200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4800b4930 0x7ff48014e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.038+0000 7ff487200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4800d10e0 0x7ff4800aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.038+0000 7ff487200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff48015c7c0 0x7ff48014c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.038+0000 7ff487200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff48006f6a0 con 0x7ff4800d10e0 2024-04-16T16:15:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.038+0000 7ff487200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff480073350 con 0x7ff48015c7c0 2024-04-16T16:15:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.038+0000 7ff487200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff48006f820 con 0x7ff4800b4930 2024-04-16T16:15:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.039+0000 7ff47f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4800d10e0 0x7ff4800aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.039+0000 7ff47f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4800d10e0 0x7ff4800aea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45680/0 (socket says 172.21.15.45:45680) 2024-04-16T16:15:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.039+0000 7ff47f400640 1 -- 172.21.15.45:0/3328517333 learned_addr learned my addr 172.21.15.45:0/3328517333 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.039+0000 7ff47fe00640 1 --2- 172.21.15.45:0/3328517333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4800b4930 0x7ff48014e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.039+0000 7ff485200640 1 --2- 172.21.15.45:0/3328517333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff48015c7c0 0x7ff48014c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:34.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.039+0000 7ff47f400640 1 -- 172.21.15.45:0/3328517333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4800b4930 msgr2=0x7ff48014e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:34.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.039+0000 7ff47f400640 1 --2- 172.21.15.45:0/3328517333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4800b4930 0x7ff48014e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.039+0000 7ff47f400640 1 -- 172.21.15.45:0/3328517333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff48015c7c0 msgr2=0x7ff48014c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.039+0000 7ff47f400640 1 --2- 172.21.15.45:0/3328517333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff48015c7c0 0x7ff48014c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.039+0000 7ff47f400640 1 -- 172.21.15.45:0/3328517333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48014b010 con 0x7ff4800d10e0 2024-04-16T16:15:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.039+0000 7ff485200640 1 --2- 172.21.15.45:0/3328517333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff48015c7c0 0x7ff48014c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:15:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.040+0000 7ff47f400640 1 --2- 172.21.15.45:0/3328517333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4800d10e0 0x7ff4800aea00 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00c4d0 tx=0x7ff46c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.040+0000 7ff477e00640 1 -- 172.21.15.45:0/3328517333 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c018070 con 0x7ff4800d10e0 2024-04-16T16:15:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.040+0000 7ff477e00640 1 -- 172.21.15.45:0/3328517333 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff46c0074e0 con 0x7ff4800d10e0 2024-04-16T16:15:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.040+0000 7ff477e00640 1 -- 172.21.15.45:0/3328517333 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c004a20 con 0x7ff4800d10e0 2024-04-16T16:15:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.040+0000 7ff487200640 1 -- 172.21.15.45:0/3328517333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff48014b2f0 con 0x7ff4800d10e0 2024-04-16T16:15:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.040+0000 7ff487200640 1 -- 172.21.15.45:0/3328517333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff48014b8b0 con 0x7ff4800d10e0 2024-04-16T16:15:34.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.041+0000 7ff487200640 1 -- 172.21.15.45:0/3328517333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff480075780 con 0x7ff4800d10e0 2024-04-16T16:15:34.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.042+0000 7ff477e00640 1 -- 172.21.15.45:0/3328517333 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7ff46c004bc0 con 0x7ff4800d10e0 2024-04-16T16:15:34.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.043+0000 7ff477e00640 1 --2- 172.21.15.45:0/3328517333 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7ff46006ca60 0x7ff46006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:34.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.043+0000 7ff477e00640 1 -- 172.21.15.45:0/3328517333 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6494+0+0 (secure 0 0 0) 0x7ff46c08b320 con 0x7ff4800d10e0 2024-04-16T16:15:34.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.044+0000 7ff47fe00640 1 --2- 172.21.15.45:0/3328517333 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7ff46006ca60 0x7ff46006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:34.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.045+0000 7ff47fe00640 1 --2- 172.21.15.45:0/3328517333 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7ff46006ca60 0x7ff46006ef20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff470035000 tx=0x7ff470030000 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:34.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.048+0000 7ff477e00640 1 -- 172.21.15.45:0/3328517333 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff46c058ce0 con 0x7ff4800d10e0 2024-04-16T16:15:34.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.257+0000 7ff487200640 1 -- 172.21.15.45:0/3328517333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7ff48006e4e0 con 0x7ff4800d10e0 2024-04-16T16:15:34.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.791+0000 7ff477e00640 1 -- 172.21.15.45:0/3328517333 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'fs2_data' created v48) ==== 164+0+0 (secure 0 0 0) 0x7ff46c05cfe0 con 0x7ff4800d10e0 2024-04-16T16:15:34.791 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'fs2_data' created 2024-04-16T16:15:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.795+0000 7ff487200640 1 -- 172.21.15.45:0/3328517333 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7ff46006ca60 msgr2=0x7ff46006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.795+0000 7ff487200640 1 --2- 172.21.15.45:0/3328517333 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7ff46006ca60 0x7ff46006ef20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff470035000 tx=0x7ff470030000 comp rx=0 tx=0).stop 2024-04-16T16:15:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.795+0000 7ff487200640 1 -- 172.21.15.45:0/3328517333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4800d10e0 msgr2=0x7ff4800aea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.795+0000 7ff487200640 1 --2- 172.21.15.45:0/3328517333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4800d10e0 0x7ff4800aea00 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00c4d0 tx=0x7ff46c00c990 comp rx=0 tx=0).stop 2024-04-16T16:15:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.796+0000 7ff487200640 1 -- 172.21.15.45:0/3328517333 shutdown_connections 2024-04-16T16:15:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.796+0000 7ff487200640 1 --2- 172.21.15.45:0/3328517333 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7ff46006ca60 0x7ff46006ef20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.796+0000 7ff487200640 1 --2- 172.21.15.45:0/3328517333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff48015c7c0 0x7ff48014c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.796+0000 7ff487200640 1 --2- 172.21.15.45:0/3328517333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4800d10e0 0x7ff4800aea00 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.796+0000 7ff487200640 1 --2- 172.21.15.45:0/3328517333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4800b4930 0x7ff48014e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:34.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.796+0000 7ff487200640 1 -- 172.21.15.45:0/3328517333 >> 172.21.15.45:0/3328517333 conn(0x7ff4800b3970 msgr2=0x7ff48015ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:34.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.798+0000 7ff487200640 1 -- 172.21.15.45:0/3328517333 shutdown_connections 2024-04-16T16:15:34.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:34.799+0000 7ff487200640 1 -- 172.21.15.45:0/3328517333 wait complete. 2024-04-16T16:15:34.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new fs2 fs2_metadata fs2_data 2024-04-16T16:15:35.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.025+0000 7fa44a600640 1 Processor -- start 2024-04-16T16:15:35.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.025+0000 7fa44a600640 1 -- start start 2024-04-16T16:15:35.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.026+0000 7fa44a600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa4441461b0 0x7fa444160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:35.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.026+0000 7fa44a600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa4440b5d50 0x7fa4440cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:35.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.026+0000 7fa44a600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa444146590 con 0x7fa4440b58f0 2024-04-16T16:15:35.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.026+0000 7fa44a600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa44406dfd0 con 0x7fa4441461b0 2024-04-16T16:15:35.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.026+0000 7fa44a600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa4440c5750 con 0x7fa4440b5d50 2024-04-16T16:15:35.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.027+0000 7fa449200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa4441461b0 0x7fa444160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:35.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.027+0000 7fa443e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa4440b58f0 0x7fa4440cf440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58554/0 (socket says 172.21.15.45:58554) 2024-04-16T16:15:35.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.027+0000 7fa443e00640 1 -- 172.21.15.45:0/4037098830 learned_addr learned my addr 172.21.15.45:0/4037098830 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:35.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.027+0000 7fa449c00640 1 --2- 172.21.15.45:0/4037098830 >> v2:172.21.15.103:3300/0 conn(0x7fa4440b5d50 0x7fa4440cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:35.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.027+0000 7fa449200640 1 -- 172.21.15.45:0/4037098830 >> v2:172.21.15.103:3300/0 conn(0x7fa4440b5d50 msgr2=0x7fa4440cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:35.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.027+0000 7fa449200640 1 --2- 172.21.15.45:0/4037098830 >> v2:172.21.15.103:3300/0 conn(0x7fa4440b5d50 0x7fa4440cfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:35.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.027+0000 7fa449200640 1 -- 172.21.15.45:0/4037098830 >> v1:172.21.15.45:6789/0 conn(0x7fa4440b58f0 legacy=0x7fa4440cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:35.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.027+0000 7fa449200640 1 -- 172.21.15.45:0/4037098830 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa444161230 con 0x7fa4441461b0 2024-04-16T16:15:35.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.027+0000 7fa443400640 1 -- 172.21.15.45:0/4037098830 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 851128770 0 0) 0x7fa444146590 con 0x7fa4440b58f0 2024-04-16T16:15:35.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.029+0000 7fa449200640 1 --2- 172.21.15.45:0/4037098830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4441461b0 0x7fa444160cf0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa444071d00 tx=0x7fa43802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f5e2006bbd6413 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:35.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.029+0000 7fa443400640 1 -- 172.21.15.45:0/4037098830 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa438035020 con 0x7fa4441461b0 2024-04-16T16:15:35.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.029+0000 7fa443400640 1 -- 172.21.15.45:0/4037098830 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa438031830 con 0x7fa4441461b0 2024-04-16T16:15:35.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.029+0000 7fa443400640 1 -- 172.21.15.45:0/4037098830 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa438031c00 con 0x7fa4441461b0 2024-04-16T16:15:35.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.030+0000 7fa44a600640 1 -- 172.21.15.45:0/4037098830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4441461b0 msgr2=0x7fa444160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:35.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.030+0000 7fa44a600640 1 --2- 172.21.15.45:0/4037098830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4441461b0 0x7fa444160cf0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa444071d00 tx=0x7fa43802b8d0 comp rx=0 tx=0).stop 2024-04-16T16:15:35.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.030+0000 7fa44a600640 1 -- 172.21.15.45:0/4037098830 shutdown_connections 2024-04-16T16:15:35.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.030+0000 7fa44a600640 1 --2- 172.21.15.45:0/4037098830 >> v2:172.21.15.103:3300/0 conn(0x7fa4440b5d50 0x7fa4440cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:35.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.030+0000 7fa44a600640 1 --2- 172.21.15.45:0/4037098830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4441461b0 0x7fa444160cf0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:35.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.030+0000 7fa44a600640 1 -- 172.21.15.45:0/4037098830 >> 172.21.15.45:0/4037098830 conn(0x7fa44406b930 msgr2=0x7fa44406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:35.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.030+0000 7fa44a600640 1 -- 172.21.15.45:0/4037098830 shutdown_connections 2024-04-16T16:15:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.030+0000 7fa44a600640 1 -- 172.21.15.45:0/4037098830 wait complete. 2024-04-16T16:15:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.032+0000 7fa44a600640 1 Processor -- start 2024-04-16T16:15:35.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.032+0000 7fa44a600640 1 -- start start 2024-04-16T16:15:35.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.033+0000 7fa44a600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4440b58f0 0x7fa444131280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:35.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.033+0000 7fa44a600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4440b5d50 0x7fa4440a95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:35.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.033+0000 7fa44a600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4441461b0 0x7fa4441329b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.033+0000 7fa44a600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa4440c5cd0 con 0x7fa4440b5d50 2024-04-16T16:15:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.033+0000 7fa44a600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa4440c59f0 con 0x7fa4441461b0 2024-04-16T16:15:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.033+0000 7fa44a600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa4440c5fb0 con 0x7fa4440b58f0 2024-04-16T16:15:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.033+0000 7fa443e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4440b5d50 0x7fa4440a95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.033+0000 7fa443e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4440b5d50 0x7fa4440a95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45698/0 (socket says 172.21.15.45:45698) 2024-04-16T16:15:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.033+0000 7fa443e00640 1 -- 172.21.15.45:0/2427728337 learned_addr learned my addr 172.21.15.45:0/2427728337 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.034+0000 7fa449200640 1 --2- 172.21.15.45:0/2427728337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4440b58f0 0x7fa444131280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.034+0000 7fa449c00640 1 --2- 172.21.15.45:0/2427728337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4441461b0 0x7fa4441329b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.034+0000 7fa443e00640 1 -- 172.21.15.45:0/2427728337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4440b58f0 msgr2=0x7fa444131280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.034+0000 7fa443e00640 1 --2- 172.21.15.45:0/2427728337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4440b58f0 0x7fa444131280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.034+0000 7fa443e00640 1 -- 172.21.15.45:0/2427728337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4441461b0 msgr2=0x7fa4441329b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.034+0000 7fa443e00640 1 --2- 172.21.15.45:0/2427728337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4441461b0 0x7fa4441329b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.034+0000 7fa443e00640 1 -- 172.21.15.45:0/2427728337 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa444131910 con 0x7fa4440b5d50 2024-04-16T16:15:35.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.034+0000 7fa449c00640 1 --2- 172.21.15.45:0/2427728337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4441461b0 0x7fa4441329b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:35.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.034+0000 7fa443e00640 1 --2- 172.21.15.45:0/2427728337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4440b5d50 0x7fa4440a95f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa43000c4d0 tx=0x7fa43000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:35.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.034+0000 7fa440c00640 1 -- 172.21.15.45:0/2427728337 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa430018070 con 0x7fa4440b5d50 2024-04-16T16:15:35.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.034+0000 7fa440c00640 1 -- 172.21.15.45:0/2427728337 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa430007500 con 0x7fa4440b5d50 2024-04-16T16:15:35.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.034+0000 7fa440c00640 1 -- 172.21.15.45:0/2427728337 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa430004a20 con 0x7fa4440b5d50 2024-04-16T16:15:35.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.034+0000 7fa44a600640 1 -- 172.21.15.45:0/2427728337 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa444131bf0 con 0x7fa4440b5d50 2024-04-16T16:15:35.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.035+0000 7fa44a600640 1 -- 172.21.15.45:0/2427728337 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4441321b0 con 0x7fa4440b5d50 2024-04-16T16:15:35.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.036+0000 7fa44a600640 1 -- 172.21.15.45:0/2427728337 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa444072260 con 0x7fa4440b5d50 2024-04-16T16:15:35.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.037+0000 7fa440c00640 1 -- 172.21.15.45:0/2427728337 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fa43001b070 con 0x7fa4440b5d50 2024-04-16T16:15:35.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.038+0000 7fa440c00640 1 --2- 172.21.15.45:0/2427728337 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa42406ca60 0x7fa42406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:35.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.038+0000 7fa440c00640 1 -- 172.21.15.45:0/2427728337 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6873+0+0 (secure 0 0 0) 0x7fa43008b5b0 con 0x7fa4440b5d50 2024-04-16T16:15:35.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.039+0000 7fa449200640 1 --2- 172.21.15.45:0/2427728337 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa42406ca60 0x7fa42406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.040+0000 7fa449200640 1 --2- 172.21.15.45:0/2427728337 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa42406ca60 0x7fa42406ef20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa438036000 tx=0x7fa43803c040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.043+0000 7fa440c00640 1 -- 172.21.15.45:0/2427728337 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa444072260 con 0x7fa4440b5d50 2024-04-16T16:15:35.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.288+0000 7fa44a600640 1 -- 172.21.15.45:0/2427728337 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "fs2", "metadata": "fs2_metadata", "data": "fs2_data"} v 0) -- 0x7fa444072500 con 0x7fa4440b5d50 2024-04-16T16:15:35.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.794+0000 7fa440c00640 1 -- 172.21.15.45:0/2427728337 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "fs2", "metadata": "fs2_metadata", "data": "fs2_data"}]=0 new fs with metadata pool 8 and data pool 9 v29) ==== 163+0+0 (secure 0 0 0) 0x7fa4300593a0 con 0x7fa4440b5d50 2024-04-16T16:15:35.795 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 8 and data pool 9 2024-04-16T16:15:35.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.798+0000 7fa44a600640 1 -- 172.21.15.45:0/2427728337 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa42406ca60 msgr2=0x7fa42406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:35.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.798+0000 7fa44a600640 1 --2- 172.21.15.45:0/2427728337 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa42406ca60 0x7fa42406ef20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa438036000 tx=0x7fa43803c040 comp rx=0 tx=0).stop 2024-04-16T16:15:35.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.798+0000 7fa44a600640 1 -- 172.21.15.45:0/2427728337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4440b5d50 msgr2=0x7fa4440a95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:35.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.798+0000 7fa44a600640 1 --2- 172.21.15.45:0/2427728337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4440b5d50 0x7fa4440a95f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa43000c4d0 tx=0x7fa43000c990 comp rx=0 tx=0).stop 2024-04-16T16:15:35.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.799+0000 7fa44a600640 1 -- 172.21.15.45:0/2427728337 shutdown_connections 2024-04-16T16:15:35.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.799+0000 7fa44a600640 1 --2- 172.21.15.45:0/2427728337 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa42406ca60 0x7fa42406ef20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:35.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.799+0000 7fa44a600640 1 --2- 172.21.15.45:0/2427728337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4441461b0 0x7fa4441329b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:35.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.799+0000 7fa44a600640 1 --2- 172.21.15.45:0/2427728337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4440b5d50 0x7fa4440a95f0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:35.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.799+0000 7fa44a600640 1 --2- 172.21.15.45:0/2427728337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4440b58f0 0x7fa444131280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:35.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.799+0000 7fa44a600640 1 -- 172.21.15.45:0/2427728337 >> 172.21.15.45:0/2427728337 conn(0x7fa44406b930 msgr2=0x7fa4440b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:35.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.799+0000 7fa44a600640 1 -- 172.21.15.45:0/2427728337 shutdown_connections 2024-04-16T16:15:35.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:35.799+0000 7fa44a600640 1 -- 172.21.15.45:0/2427728337 wait complete. 2024-04-16T16:15:35.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:15:36.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.018+0000 7f13f1a00640 1 Processor -- start 2024-04-16T16:15:36.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.018+0000 7f13f1a00640 1 -- start start 2024-04-16T16:15:36.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.020+0000 7f13f1a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13ec15e8f0 0x7f13ec0cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.020+0000 7f13f1a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f13ec0cf080 0x7f13ec0cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.020+0000 7f13f1a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13ec15ecd0 con 0x7f13ec0ce9a0 2024-04-16T16:15:36.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.020+0000 7f13f1a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f13ec0ce2e0 con 0x7f13ec0cf080 2024-04-16T16:15:36.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.020+0000 7f13f1a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f13ec0cd6d0 con 0x7f13ec15e8f0 2024-04-16T16:15:36.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.020+0000 7f13f1000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f13ec0cf080 0x7f13ec0cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.020+0000 7f13f1000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f13ec0cf080 0x7f13ec0cbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45708/0 (socket says 172.21.15.45:45708) 2024-04-16T16:15:36.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.020+0000 7f13eb400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f13ec0ce9a0 0x7f13ec162480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58568/0 (socket says 172.21.15.45:58568) 2024-04-16T16:15:36.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.020+0000 7f13f1000640 1 -- 172.21.15.45:0/621157740 learned_addr learned my addr 172.21.15.45:0/621157740 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:36.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.021+0000 7f13f1000640 1 -- 172.21.15.45:0/621157740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13ec15e8f0 msgr2=0x7f13ec0cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:36.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.021+0000 7f13ebe00640 1 --2- 172.21.15.45:0/621157740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13ec15e8f0 0x7f13ec0cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.021+0000 7f13f1000640 1 --2- 172.21.15.45:0/621157740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13ec15e8f0 0x7f13ec0cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.021+0000 7f13f1000640 1 -- 172.21.15.45:0/621157740 >> v1:172.21.15.45:6789/0 conn(0x7f13ec0ce9a0 legacy=0x7f13ec162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.021+0000 7f13eaa00640 1 -- 172.21.15.45:0/621157740 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2227635939 0 0) 0x7f13ec15ecd0 con 0x7f13ec0ce9a0 2024-04-16T16:15:36.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.021+0000 7f13f1000640 1 -- 172.21.15.45:0/621157740 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ec0cd9b0 con 0x7f13ec0cf080 2024-04-16T16:15:36.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.021+0000 7f13ebe00640 1 --2- 172.21.15.45:0/621157740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13ec15e8f0 0x7f13ec0cc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:36.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.022+0000 7f13f1000640 1 --2- 172.21.15.45:0/621157740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13ec0cf080 0x7f13ec0cbc40 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f13e4009e50 tx=0x7f13e402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a2028fa0245294f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:36.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.022+0000 7f13eaa00640 1 -- 172.21.15.45:0/621157740 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e402cb00 con 0x7f13ec0cf080 2024-04-16T16:15:36.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.022+0000 7f13eaa00640 1 -- 172.21.15.45:0/621157740 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13e402cca0 con 0x7f13ec0cf080 2024-04-16T16:15:36.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.022+0000 7f13eaa00640 1 -- 172.21.15.45:0/621157740 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e4034610 con 0x7f13ec0cf080 2024-04-16T16:15:36.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.022+0000 7f13f1a00640 1 -- 172.21.15.45:0/621157740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13ec0cf080 msgr2=0x7f13ec0cbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.022+0000 7f13f1a00640 1 --2- 172.21.15.45:0/621157740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13ec0cf080 0x7f13ec0cbc40 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f13e4009e50 tx=0x7f13e402bf60 comp rx=0 tx=0).stop 2024-04-16T16:15:36.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.023+0000 7f13f1a00640 1 -- 172.21.15.45:0/621157740 shutdown_connections 2024-04-16T16:15:36.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.023+0000 7f13f1a00640 1 --2- 172.21.15.45:0/621157740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13ec0cf080 0x7f13ec0cbc40 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.023+0000 7f13f1a00640 1 --2- 172.21.15.45:0/621157740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13ec15e8f0 0x7f13ec0cc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.023+0000 7f13f1a00640 1 -- 172.21.15.45:0/621157740 >> 172.21.15.45:0/621157740 conn(0x7f13ec06b930 msgr2=0x7f13ec06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:36.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.023+0000 7f13f1a00640 1 -- 172.21.15.45:0/621157740 shutdown_connections 2024-04-16T16:15:36.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.023+0000 7f13f1a00640 1 -- 172.21.15.45:0/621157740 wait complete. 2024-04-16T16:15:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.024+0000 7f13f1a00640 1 Processor -- start 2024-04-16T16:15:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.024+0000 7f13f1a00640 1 -- start start 2024-04-16T16:15:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.025+0000 7f13f1a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13ec0ce9a0 0x7f13ec14e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.025+0000 7f13f1a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13ec0cf080 0x7f13ec1524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.025+0000 7f13f1a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13ec15e8f0 0x7f13ec152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.025+0000 7f13f1a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec0cd360 con 0x7f13ec0cf080 2024-04-16T16:15:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.025+0000 7f13f1a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec0cdf70 con 0x7f13ec15e8f0 2024-04-16T16:15:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.025+0000 7f13f1a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f13ec0cc950 con 0x7f13ec0ce9a0 2024-04-16T16:15:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.026+0000 7f13eb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13ec0cf080 0x7f13ec1524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.026+0000 7f13eb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13ec0cf080 0x7f13ec1524d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45718/0 (socket says 172.21.15.45:45718) 2024-04-16T16:15:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.026+0000 7f13eb400640 1 -- 172.21.15.45:0/3721603657 learned_addr learned my addr 172.21.15.45:0/3721603657 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.026+0000 7f13f1000640 1 --2- 172.21.15.45:0/3721603657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13ec15e8f0 0x7f13ec152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.026+0000 7f13eb400640 1 -- 172.21.15.45:0/3721603657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13ec0ce9a0 msgr2=0x7f13ec14e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.026+0000 7f13ebe00640 1 --2- 172.21.15.45:0/3721603657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13ec0ce9a0 0x7f13ec14e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.026+0000 7f13eb400640 1 --2- 172.21.15.45:0/3721603657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13ec0ce9a0 0x7f13ec14e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.026+0000 7f13eb400640 1 -- 172.21.15.45:0/3721603657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13ec15e8f0 msgr2=0x7f13ec152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.026+0000 7f13eb400640 1 --2- 172.21.15.45:0/3721603657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13ec15e8f0 0x7f13ec152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.026+0000 7f13eb400640 1 -- 172.21.15.45:0/3721603657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ec14ebc0 con 0x7f13ec0cf080 2024-04-16T16:15:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.026+0000 7f13f1000640 1 --2- 172.21.15.45:0/3721603657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13ec15e8f0 0x7f13ec152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.026+0000 7f13eb400640 1 --2- 172.21.15.45:0/3721603657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13ec0cf080 0x7f13ec1524d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f13d800c4d0 tx=0x7f13d800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.026+0000 7f13e3e00640 1 -- 172.21.15.45:0/3721603657 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d8018070 con 0x7f13ec0cf080 2024-04-16T16:15:36.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.027+0000 7f13e3e00640 1 -- 172.21.15.45:0/3721603657 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d8007500 con 0x7f13ec0cf080 2024-04-16T16:15:36.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.027+0000 7f13e3e00640 1 -- 172.21.15.45:0/3721603657 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d8004a20 con 0x7f13ec0cf080 2024-04-16T16:15:36.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.027+0000 7f13f1a00640 1 -- 172.21.15.45:0/3721603657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ec1516b0 con 0x7f13ec0cf080 2024-04-16T16:15:36.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.027+0000 7f13f1a00640 1 -- 172.21.15.45:0/3721603657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ec15fd90 con 0x7f13ec0cf080 2024-04-16T16:15:36.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.028+0000 7f13f1a00640 1 -- 172.21.15.45:0/3721603657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13ec075780 con 0x7f13ec0cf080 2024-04-16T16:15:36.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.029+0000 7f13e3e00640 1 -- 172.21.15.45:0/3721603657 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f13d8004bc0 con 0x7f13ec0cf080 2024-04-16T16:15:36.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.030+0000 7f13e3e00640 1 --2- 172.21.15.45:0/3721603657 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f13cc06ca60 0x7f13cc06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.030+0000 7f13e3e00640 1 -- 172.21.15.45:0/3721603657 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6967+0+0 (secure 0 0 0) 0x7f13d808b4e0 con 0x7f13ec0cf080 2024-04-16T16:15:36.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.030+0000 7f13ebe00640 1 --2- 172.21.15.45:0/3721603657 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f13cc06ca60 0x7f13cc06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.031+0000 7f13ebe00640 1 --2- 172.21.15.45:0/3721603657 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f13cc06ca60 0x7f13cc06ef20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00b840 tx=0x7f13dc008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:36.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.034+0000 7f13e3e00640 1 -- 172.21.15.45:0/3721603657 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f13d8059300 con 0x7f13ec0cf080 2024-04-16T16:15:36.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.235+0000 7f13f1a00640 1 -- 172.21.15.45:0/3721603657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f13ec15e110 con 0x7f13ec0cf080 2024-04-16T16:15:36.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.236+0000 7f13e3e00640 1 -- 172.21.15.45:0/3721603657 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+21019 (secure 0 0 0) 0x7f13d8058c00 con 0x7f13ec0cf080 2024-04-16T16:15:36.237 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:36.238 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":49,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:15:35.289970+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"fs1_metadata","create_time":"2024-04-16T16:15:13.250782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"fs1_data","create_time":"2024-04-16T16:15:13.856986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"fs2_metadata","create_time":"2024-04-16T16:15:33.215042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"fs2_data","create_time":"2024-04-16T16:15:34.271426+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.45:6833/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:15:36.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.239+0000 7f13f1a00640 1 -- 172.21.15.45:0/3721603657 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f13cc06ca60 msgr2=0x7f13cc06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.239+0000 7f13f1a00640 1 --2- 172.21.15.45:0/3721603657 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f13cc06ca60 0x7f13cc06ef20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00b840 tx=0x7f13dc008040 comp rx=0 tx=0).stop 2024-04-16T16:15:36.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.239+0000 7f13f1a00640 1 -- 172.21.15.45:0/3721603657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13ec0cf080 msgr2=0x7f13ec1524d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.239+0000 7f13f1a00640 1 --2- 172.21.15.45:0/3721603657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13ec0cf080 0x7f13ec1524d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f13d800c4d0 tx=0x7f13d800c990 comp rx=0 tx=0).stop 2024-04-16T16:15:36.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.240+0000 7f13f1a00640 1 -- 172.21.15.45:0/3721603657 shutdown_connections 2024-04-16T16:15:36.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.240+0000 7f13f1a00640 1 --2- 172.21.15.45:0/3721603657 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f13cc06ca60 0x7f13cc06ef20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.240+0000 7f13f1a00640 1 --2- 172.21.15.45:0/3721603657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13ec15e8f0 0x7f13ec152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.240+0000 7f13f1a00640 1 --2- 172.21.15.45:0/3721603657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13ec0cf080 0x7f13ec1524d0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.240+0000 7f13f1a00640 1 --2- 172.21.15.45:0/3721603657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13ec0ce9a0 0x7f13ec14e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.240+0000 7f13f1a00640 1 -- 172.21.15.45:0/3721603657 >> 172.21.15.45:0/3721603657 conn(0x7f13ec06b930 msgr2=0x7f13ec0cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.240+0000 7f13f1a00640 1 -- 172.21.15.45:0/3721603657 shutdown_connections 2024-04-16T16:15:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.240+0000 7f13f1a00640 1 -- 172.21.15.45:0/3721603657 wait complete. 2024-04-16T16:15:36.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:15:36.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.479+0000 7fa676800640 1 Processor -- start 2024-04-16T16:15:36.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.479+0000 7fa676800640 1 -- start start 2024-04-16T16:15:36.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.480+0000 7fa676800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa6700ce9a0 0x7fa6700cb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.480+0000 7fa676800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa67015ecd0 con 0x7fa6700ccb20 2024-04-16T16:15:36.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.481+0000 7fa676800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6700cd6d0 con 0x7fa67015e8f0 2024-04-16T16:15:36.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.481+0000 7fa676800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa6700ce2e0 con 0x7fa6700ce9a0 2024-04-16T16:15:36.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.481+0000 7fa675e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa6700ce9a0 0x7fa6700cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.481+0000 7fa675e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa6700ce9a0 0x7fa6700cb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45724/0 (socket says 172.21.15.45:45724) 2024-04-16T16:15:36.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.481+0000 7fa675e00640 1 -- 172.21.15.45:0/652975422 learned_addr learned my addr 172.21.15.45:0/652975422 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:36.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.481+0000 7fa675e00640 1 -- 172.21.15.45:0/652975422 >> v1:172.21.15.103:6789/0 conn(0x7fa67015e8f0 legacy=0x7fa6700cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.482+0000 7fa675e00640 1 -- 172.21.15.45:0/652975422 >> v1:172.21.15.45:6789/0 conn(0x7fa6700ccb20 legacy=0x7fa6700cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.482+0000 7fa675e00640 1 -- 172.21.15.45:0/652975422 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6700cd9b0 con 0x7fa6700ce9a0 2024-04-16T16:15:36.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.483+0000 7fa675e00640 1 --2- 172.21.15.45:0/652975422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6700ce9a0 0x7fa6700cb340 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa668009e50 tx=0x7fa66802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232a6b0d5f0b4020 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:36.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.483+0000 7fa66fe00640 1 -- 172.21.15.45:0/652975422 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66802cb20 con 0x7fa6700ce9a0 2024-04-16T16:15:36.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.483+0000 7fa66fe00640 1 -- 172.21.15.45:0/652975422 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa66802ccc0 con 0x7fa6700ce9a0 2024-04-16T16:15:36.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.483+0000 7fa66fe00640 1 -- 172.21.15.45:0/652975422 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa668034610 con 0x7fa6700ce9a0 2024-04-16T16:15:36.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.484+0000 7fa676800640 1 -- 172.21.15.45:0/652975422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6700ce9a0 msgr2=0x7fa6700cb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.484+0000 7fa676800640 1 --2- 172.21.15.45:0/652975422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6700ce9a0 0x7fa6700cb340 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa668009e50 tx=0x7fa66802bf80 comp rx=0 tx=0).stop 2024-04-16T16:15:36.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.484+0000 7fa676800640 1 -- 172.21.15.45:0/652975422 shutdown_connections 2024-04-16T16:15:36.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.484+0000 7fa676800640 1 --2- 172.21.15.45:0/652975422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6700ce9a0 0x7fa6700cb340 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.484+0000 7fa676800640 1 -- 172.21.15.45:0/652975422 >> 172.21.15.45:0/652975422 conn(0x7fa67006b930 msgr2=0x7fa67006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:36.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.484+0000 7fa676800640 1 -- 172.21.15.45:0/652975422 shutdown_connections 2024-04-16T16:15:36.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.484+0000 7fa676800640 1 -- 172.21.15.45:0/652975422 wait complete. 2024-04-16T16:15:36.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.486+0000 7fa676800640 1 Processor -- start 2024-04-16T16:15:36.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.486+0000 7fa676800640 1 -- start start 2024-04-16T16:15:36.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.487+0000 7fa676800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6700ccb20 0x7fa67014d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.487+0000 7fa676800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6700ce9a0 0x7fa670161360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.487+0000 7fa676800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa67015e8f0 0x7fa6701618a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.487+0000 7fa676800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa6700cd360 con 0x7fa6700ce9a0 2024-04-16T16:15:36.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.487+0000 7fa676800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa6700cdf70 con 0x7fa6700ccb20 2024-04-16T16:15:36.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.487+0000 7fa676800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa6700cf080 con 0x7fa67015e8f0 2024-04-16T16:15:36.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.487+0000 7fa674a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6700ce9a0 0x7fa670161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.487+0000 7fa674a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6700ce9a0 0x7fa670161360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45736/0 (socket says 172.21.15.45:45736) 2024-04-16T16:15:36.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.487+0000 7fa674a00640 1 -- 172.21.15.45:0/1313080835 learned_addr learned my addr 172.21.15.45:0/1313080835 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:36.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.488+0000 7fa675400640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6700ccb20 0x7fa67014d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.488+0000 7fa675e00640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa67015e8f0 0x7fa6701618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.488+0000 7fa674a00640 1 -- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa67015e8f0 msgr2=0x7fa6701618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.488+0000 7fa674a00640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa67015e8f0 0x7fa6701618a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.488+0000 7fa674a00640 1 -- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6700ccb20 msgr2=0x7fa67014d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.488+0000 7fa674a00640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6700ccb20 0x7fa67014d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.488+0000 7fa674a00640 1 -- 172.21.15.45:0/1313080835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa670151c80 con 0x7fa6700ce9a0 2024-04-16T16:15:36.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.488+0000 7fa675400640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6700ccb20 0x7fa67014d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:36.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.488+0000 7fa675e00640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa67015e8f0 0x7fa6701618a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:36.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.488+0000 7fa674a00640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6700ce9a0 0x7fa670161360 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa66000c450 tx=0x7fa66000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:36.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.488+0000 7fa66d600640 1 -- 172.21.15.45:0/1313080835 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6600052d0 con 0x7fa6700ce9a0 2024-04-16T16:15:36.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.488+0000 7fa66d600640 1 -- 172.21.15.45:0/1313080835 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa660005470 con 0x7fa6700ce9a0 2024-04-16T16:15:36.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.488+0000 7fa676800640 1 -- 172.21.15.45:0/1313080835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa67015fff0 con 0x7fa6700ce9a0 2024-04-16T16:15:36.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.488+0000 7fa66d600640 1 -- 172.21.15.45:0/1313080835 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660012870 con 0x7fa6700ce9a0 2024-04-16T16:15:36.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.489+0000 7fa676800640 1 -- 172.21.15.45:0/1313080835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6701605b0 con 0x7fa6700ce9a0 2024-04-16T16:15:36.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.490+0000 7fa676800640 1 -- 172.21.15.45:0/1313080835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6700c9ed0 con 0x7fa6700ce9a0 2024-04-16T16:15:36.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.491+0000 7fa66d600640 1 -- 172.21.15.45:0/1313080835 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fa660012a10 con 0x7fa6700ce9a0 2024-04-16T16:15:36.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.492+0000 7fa66d600640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa65406ca60 0x7fa65406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.492+0000 7fa66d600640 1 -- 172.21.15.45:0/1313080835 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6967+0+0 (secure 0 0 0) 0x7fa66008bf70 con 0x7fa6700ce9a0 2024-04-16T16:15:36.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.493+0000 7fa675400640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa65406ca60 0x7fa65406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.494+0000 7fa675400640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa65406ca60 0x7fa65406ef20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa664000f60 tx=0x7fa664009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:36.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.496+0000 7fa66d600640 1 -- 172.21.15.45:0/1313080835 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa6700c9ed0 con 0x7fa6700ce9a0 2024-04-16T16:15:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.694+0000 7fa676800640 1 -- 172.21.15.45:0/1313080835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa670075780 con 0x7fa6700ce9a0 2024-04-16T16:15:36.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.699+0000 7fa66d600640 1 -- 172.21.15.45:0/1313080835 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+21019 (secure 0 0 0) 0x7fa660059c70 con 0x7fa6700ce9a0 2024-04-16T16:15:36.699 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:36.699 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":49,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:15:35.289970+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"fs1_metadata","create_time":"2024-04-16T16:15:13.250782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"fs1_data","create_time":"2024-04-16T16:15:13.856986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"fs2_metadata","create_time":"2024-04-16T16:15:33.215042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"fs2_data","create_time":"2024-04-16T16:15:34.271426+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.45:6833/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:15:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.701+0000 7fa676800640 1 -- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa65406ca60 msgr2=0x7fa65406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.701+0000 7fa676800640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa65406ca60 0x7fa65406ef20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa664000f60 tx=0x7fa664009040 comp rx=0 tx=0).stop 2024-04-16T16:15:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.701+0000 7fa676800640 1 -- 172.21.15.45:0/1313080835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6700ce9a0 msgr2=0x7fa670161360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.701+0000 7fa676800640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6700ce9a0 0x7fa670161360 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa66000c450 tx=0x7fa66000c910 comp rx=0 tx=0).stop 2024-04-16T16:15:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.701+0000 7fa676800640 1 -- 172.21.15.45:0/1313080835 shutdown_connections 2024-04-16T16:15:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.701+0000 7fa676800640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa67015e8f0 0x7fa6701618a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.701+0000 7fa676800640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6700ce9a0 0x7fa670161360 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.701+0000 7fa676800640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa65406ca60 0x7fa65406ef20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.701+0000 7fa676800640 1 --2- 172.21.15.45:0/1313080835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6700ccb20 0x7fa67014d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.702+0000 7fa676800640 1 -- 172.21.15.45:0/1313080835 >> 172.21.15.45:0/1313080835 conn(0x7fa67006b930 msgr2=0x7fa6700b53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.702+0000 7fa676800640 1 -- 172.21.15.45:0/1313080835 shutdown_connections 2024-04-16T16:15:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.702+0000 7fa676800640 1 -- 172.21.15.45:0/1313080835 wait complete. 2024-04-16T16:15:36.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set fs2 standby_count_wanted 0 2024-04-16T16:15:36.797 INFO:tasks.ceph.mgr.x.smithi103.stderr:Exception in thread Thread-8: 2024-04-16T16:15:36.797 INFO:tasks.ceph.mgr.x.smithi103.stderr:Traceback (most recent call last): 2024-04-16T16:15:36.797 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner 2024-04-16T16:15:36.797 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.run() 2024-04-16T16:15:36.797 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 1306, in run 2024-04-16T16:15:36.797 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.function(*self.args, **self.kwargs) 2024-04-16T16:15:36.798 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/share/ceph/mgr/stats/fs/perf_stats.py", line 222, in re_register_queries 2024-04-16T16:15:36.798 INFO:tasks.ceph.mgr.x.smithi103.stderr: if self.mx_last_updated >= ua_last_updated: 2024-04-16T16:15:36.798 INFO:tasks.ceph.mgr.x.smithi103.stderr:AttributeError: 'FSPerfStats' object has no attribute 'mx_last_updated' 2024-04-16T16:15:36.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.946+0000 7fb832800640 1 Processor -- start 2024-04-16T16:15:36.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.946+0000 7fb832800640 1 -- start start 2024-04-16T16:15:36.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.948+0000 7fb832800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb82c0b6310 0x7fb82c161db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.948+0000 7fb832800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb82c0d3e90 0x7fb82c1622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.948+0000 7fb832800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb82c0b6810 con 0x7fb82c0d18f0 2024-04-16T16:15:36.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.948+0000 7fb832800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb82c0d29f0 con 0x7fb82c0d3e90 2024-04-16T16:15:36.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.948+0000 7fb832800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb82c0b6af0 con 0x7fb82c0b6310 2024-04-16T16:15:36.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.949+0000 7fb830a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb82c0b6310 0x7fb82c161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.949+0000 7fb830a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb82c0b6310 0x7fb82c161db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:37270/0 (socket says 172.21.15.45:37270) 2024-04-16T16:15:36.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.949+0000 7fb830a00640 1 -- 172.21.15.45:0/1740104585 learned_addr learned my addr 172.21.15.45:0/1740104585 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:36.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.949+0000 7fb831e00640 1 --2- 172.21.15.45:0/1740104585 >> v2:172.21.15.103:3300/0 conn(0x7fb82c0d3e90 0x7fb82c1622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.950+0000 7fb830a00640 1 -- 172.21.15.45:0/1740104585 >> v2:172.21.15.103:3300/0 conn(0x7fb82c0d3e90 msgr2=0x7fb82c1622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:36.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.950+0000 7fb830a00640 1 --2- 172.21.15.45:0/1740104585 >> v2:172.21.15.103:3300/0 conn(0x7fb82c0d3e90 0x7fb82c1622f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.950+0000 7fb830a00640 1 -- 172.21.15.45:0/1740104585 >> v1:172.21.15.103:6789/0 conn(0x7fb82c0d18f0 legacy=0x7fb82c0ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.950+0000 7fb830a00640 1 -- 172.21.15.45:0/1740104585 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb82c0b6dd0 con 0x7fb82c0b6310 2024-04-16T16:15:36.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.950+0000 7fb831e00640 1 --2- 172.21.15.45:0/1740104585 >> v2:172.21.15.103:3300/0 conn(0x7fb82c0d3e90 0x7fb82c1622f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:15:36.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.951+0000 7fb830a00640 1 --2- 172.21.15.45:0/1740104585 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb82c0b6310 0x7fb82c161db0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb818009870 tx=0x7fb81802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b21b01decdc66b58 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:36.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.952+0000 7fb82be00640 1 -- 172.21.15.45:0/1740104585 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb818035020 con 0x7fb82c0b6310 2024-04-16T16:15:36.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.952+0000 7fb82be00640 1 -- 172.21.15.45:0/1740104585 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb818031640 con 0x7fb82c0b6310 2024-04-16T16:15:36.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.952+0000 7fb82be00640 1 -- 172.21.15.45:0/1740104585 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81802d040 con 0x7fb82c0b6310 2024-04-16T16:15:36.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.953+0000 7fb832800640 1 -- 172.21.15.45:0/1740104585 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb82c0b6310 msgr2=0x7fb82c161db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.953+0000 7fb832800640 1 --2- 172.21.15.45:0/1740104585 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb82c0b6310 0x7fb82c161db0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb818009870 tx=0x7fb81802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.953+0000 7fb832800640 1 -- 172.21.15.45:0/1740104585 shutdown_connections 2024-04-16T16:15:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.953+0000 7fb832800640 1 --2- 172.21.15.45:0/1740104585 >> v2:172.21.15.103:3300/0 conn(0x7fb82c0d3e90 0x7fb82c1622f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.953+0000 7fb832800640 1 --2- 172.21.15.45:0/1740104585 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb82c0b6310 0x7fb82c161db0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.953+0000 7fb832800640 1 -- 172.21.15.45:0/1740104585 >> 172.21.15.45:0/1740104585 conn(0x7fb82c06cf90 msgr2=0x7fb82c06d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.953+0000 7fb832800640 1 -- 172.21.15.45:0/1740104585 shutdown_connections 2024-04-16T16:15:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.953+0000 7fb832800640 1 -- 172.21.15.45:0/1740104585 wait complete. 2024-04-16T16:15:36.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.955+0000 7fb832800640 1 Processor -- start 2024-04-16T16:15:36.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.955+0000 7fb832800640 1 -- start start 2024-04-16T16:15:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.956+0000 7fb832800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb82c0b6310 0x7fb82c160110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.956+0000 7fb832800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb82c0d18f0 0x7fb82c1e6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.956+0000 7fb832800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb82c0d3e90 0x7fb82c1e8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.956+0000 7fb832800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb82c0d1e50 con 0x7fb82c0d18f0 2024-04-16T16:15:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.956+0000 7fb832800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb82c0d1cd0 con 0x7fb82c0b6310 2024-04-16T16:15:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.956+0000 7fb832800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb82c0d1fd0 con 0x7fb82c0d3e90 2024-04-16T16:15:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.957+0000 7fb830a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb82c0d18f0 0x7fb82c1e6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.957+0000 7fb830a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb82c0d18f0 0x7fb82c1e6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45740/0 (socket says 172.21.15.45:45740) 2024-04-16T16:15:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.957+0000 7fb830a00640 1 -- 172.21.15.45:0/2197960411 learned_addr learned my addr 172.21.15.45:0/2197960411 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.957+0000 7fb831e00640 1 --2- 172.21.15.45:0/2197960411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb82c0d3e90 0x7fb82c1e8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.957+0000 7fb831400640 1 --2- 172.21.15.45:0/2197960411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb82c0b6310 0x7fb82c160110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.957+0000 7fb830a00640 1 -- 172.21.15.45:0/2197960411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb82c0d3e90 msgr2=0x7fb82c1e8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.957+0000 7fb830a00640 1 --2- 172.21.15.45:0/2197960411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb82c0d3e90 0x7fb82c1e8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.957+0000 7fb830a00640 1 -- 172.21.15.45:0/2197960411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb82c0b6310 msgr2=0x7fb82c160110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.957+0000 7fb830a00640 1 --2- 172.21.15.45:0/2197960411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb82c0b6310 0x7fb82c160110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.957+0000 7fb830a00640 1 -- 172.21.15.45:0/2197960411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb82c151b80 con 0x7fb82c0d18f0 2024-04-16T16:15:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.958+0000 7fb830a00640 1 --2- 172.21.15.45:0/2197960411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb82c0d18f0 0x7fb82c1e6d00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb818009990 tx=0x7fb8180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.958+0000 7fb829600640 1 -- 172.21.15.45:0/2197960411 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb818040070 con 0x7fb82c0d18f0 2024-04-16T16:15:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.958+0000 7fb829600640 1 -- 172.21.15.45:0/2197960411 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb81803e070 con 0x7fb82c0d18f0 2024-04-16T16:15:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.958+0000 7fb829600640 1 -- 172.21.15.45:0/2197960411 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb818035040 con 0x7fb82c0d18f0 2024-04-16T16:15:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.958+0000 7fb832800640 1 -- 172.21.15.45:0/2197960411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb82c14dae0 con 0x7fb82c0d18f0 2024-04-16T16:15:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.958+0000 7fb832800640 1 -- 172.21.15.45:0/2197960411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb82c14e0a0 con 0x7fb82c0d18f0 2024-04-16T16:15:36.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.960+0000 7fb832800640 1 -- 172.21.15.45:0/2197960411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb82c075780 con 0x7fb82c0d18f0 2024-04-16T16:15:36.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.961+0000 7fb829600640 1 -- 172.21.15.45:0/2197960411 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fb818038020 con 0x7fb82c0d18f0 2024-04-16T16:15:36.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.962+0000 7fb829600640 1 --2- 172.21.15.45:0/2197960411 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb80c06ca60 0x7fb80c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:36.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.962+0000 7fb829600640 1 -- 172.21.15.45:0/2197960411 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6967+0+0 (secure 0 0 0) 0x7fb8180adb80 con 0x7fb82c0d18f0 2024-04-16T16:15:36.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.963+0000 7fb831400640 1 --2- 172.21.15.45:0/2197960411 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb80c06ca60 0x7fb80c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:36.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.964+0000 7fb831400640 1 --2- 172.21.15.45:0/2197960411 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb80c06ca60 0x7fb80c06ef20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb820001640 tx=0x7fb820009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:36.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:36.968+0000 7fb829600640 1 -- 172.21.15.45:0/2197960411 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb81807b9a0 con 0x7fb82c0d18f0 2024-04-16T16:15:37.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:37.215+0000 7fb832800640 1 -- 172.21.15.45:0/2197960411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "fs2", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb82c0d25a0 con 0x7fb82c0d18f0 2024-04-16T16:15:37.824 INFO:tasks.ceph.mgr.x.smithi103.stderr:Exception in thread Thread-12: 2024-04-16T16:15:37.825 INFO:tasks.ceph.mgr.x.smithi103.stderr:Traceback (most recent call last): 2024-04-16T16:15:37.825 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner 2024-04-16T16:15:37.825 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.run() 2024-04-16T16:15:37.825 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 1306, in run 2024-04-16T16:15:37.825 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.function(*self.args, **self.kwargs) 2024-04-16T16:15:37.826 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/share/ceph/mgr/stats/fs/perf_stats.py", line 222, in re_register_queries 2024-04-16T16:15:37.826 INFO:tasks.ceph.mgr.x.smithi103.stderr: if self.mx_last_updated >= ua_last_updated: 2024-04-16T16:15:37.826 INFO:tasks.ceph.mgr.x.smithi103.stderr:AttributeError: 'FSPerfStats' object has no attribute 'mx_last_updated' 2024-04-16T16:15:37.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:37.827+0000 7fb829600640 1 -- 172.21.15.45:0/2197960411 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "fs2", "var": "standby_count_wanted", "val": "0"}]=0 v32) ==== 115+0+0 (secure 0 0 0) 0x7fb81807b2a0 con 0x7fb82c0d18f0 2024-04-16T16:15:37.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:37.831+0000 7fb832800640 1 -- 172.21.15.45:0/2197960411 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb80c06ca60 msgr2=0x7fb80c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:37.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:37.831+0000 7fb832800640 1 --2- 172.21.15.45:0/2197960411 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb80c06ca60 0x7fb80c06ef20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb820001640 tx=0x7fb820009040 comp rx=0 tx=0).stop 2024-04-16T16:15:37.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:37.831+0000 7fb832800640 1 -- 172.21.15.45:0/2197960411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb82c0d18f0 msgr2=0x7fb82c1e6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:37.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:37.831+0000 7fb832800640 1 --2- 172.21.15.45:0/2197960411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb82c0d18f0 0x7fb82c1e6d00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb818009990 tx=0x7fb8180046c0 comp rx=0 tx=0).stop 2024-04-16T16:15:37.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:37.831+0000 7fb832800640 1 -- 172.21.15.45:0/2197960411 shutdown_connections 2024-04-16T16:15:37.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:37.831+0000 7fb832800640 1 --2- 172.21.15.45:0/2197960411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb82c0d3e90 0x7fb82c1e8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:37.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:37.831+0000 7fb832800640 1 --2- 172.21.15.45:0/2197960411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb82c0d18f0 0x7fb82c1e6d00 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:37.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:37.831+0000 7fb832800640 1 --2- 172.21.15.45:0/2197960411 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb80c06ca60 0x7fb80c06ef20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:37.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:37.831+0000 7fb832800640 1 --2- 172.21.15.45:0/2197960411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb82c0b6310 0x7fb82c160110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:37.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:37.831+0000 7fb832800640 1 -- 172.21.15.45:0/2197960411 >> 172.21.15.45:0/2197960411 conn(0x7fb82c06cf90 msgr2=0x7fb82c1612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:37.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:37.831+0000 7fb832800640 1 -- 172.21.15.45:0/2197960411 shutdown_connections 2024-04-16T16:15:37.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:37.831+0000 7fb832800640 1 -- 172.21.15.45:0/2197960411 wait complete. 2024-04-16T16:15:37.854 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-16T16:15:37.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set fs2 allow_standby_replay false 2024-04-16T16:15:38.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.080+0000 7f3989600640 1 Processor -- start 2024-04-16T16:15:38.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.080+0000 7f3989600640 1 -- start start 2024-04-16T16:15:38.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.082+0000 7f3989600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39840b5d50 0x7f398406eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:38.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.082+0000 7f3989600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3984146590 con 0x7f39841461b0 2024-04-16T16:15:38.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.082+0000 7f3989600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39840c5750 con 0x7f39840b58f0 2024-04-16T16:15:38.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.082+0000 7f3989600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f398406dfd0 con 0x7f39840b5d50 2024-04-16T16:15:38.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.082+0000 7f3983e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f39841461b0 0x7f3984160cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58592/0 (socket says 172.21.15.45:58592) 2024-04-16T16:15:38.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.082+0000 7f3983e00640 1 -- 172.21.15.45:0/1485373656 learned_addr learned my addr 172.21.15.45:0/1485373656 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:38.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3988c00640 1 --2- 172.21.15.45:0/1485373656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39840b5d50 0x7f398406eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:38.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3982a00640 1 -- 172.21.15.45:0/1485373656 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4217607107 0 0) 0x7f3984146590 con 0x7f39841461b0 2024-04-16T16:15:38.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3982a00640 1 -- 172.21.15.45:0/1485373656 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3968003520 con 0x7f39841461b0 2024-04-16T16:15:38.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3982a00640 1 -- 172.21.15.45:0/1485373656 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 831321241 0 0) 0x7f3968003520 con 0x7f39841461b0 2024-04-16T16:15:38.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3982a00640 1 -- 172.21.15.45:0/1485373656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39840b5d50 msgr2=0x7f398406eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:38.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3982a00640 1 --2- 172.21.15.45:0/1485373656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39840b5d50 0x7f398406eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:38.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3982a00640 1 -- 172.21.15.45:0/1485373656 >> v1:172.21.15.103:6789/0 conn(0x7f39840b58f0 legacy=0x7f398406e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:38.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3982a00640 1 -- 172.21.15.45:0/1485373656 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f398415ccb0 con 0x7f39841461b0 2024-04-16T16:15:38.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3982a00640 1 -- 172.21.15.45:0/1485373656 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f39740019f0 con 0x7f39841461b0 2024-04-16T16:15:38.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3982a00640 1 -- 172.21.15.45:0/1485373656 >> v1:172.21.15.45:6789/0 conn(0x7f39841461b0 legacy=0x7f3984160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:38.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3982a00640 1 --2- 172.21.15.45:0/1485373656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39840b5d50 0x7f3968003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:38.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3982a00640 1 --2- 172.21.15.45:0/1485373656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3968003fd0 0x7f3968004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:38.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3982a00640 1 --2- 172.21.15.45:0/1485373656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3968004d40 0x7f39680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:38.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3982a00640 1 -- 172.21.15.45:0/1485373656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f396800baf0 con 0x7f3968004d40 2024-04-16T16:15:38.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3982a00640 1 -- 172.21.15.45:0/1485373656 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f396800bca0 con 0x7f3968003fd0 2024-04-16T16:15:38.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.083+0000 7f3982a00640 1 -- 172.21.15.45:0/1485373656 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f396800bea0 con 0x7f39840b5d50 2024-04-16T16:15:38.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.084+0000 7f3988c00640 1 --2- 172.21.15.45:0/1485373656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3968004d40 0x7f39680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:38.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.084+0000 7f3983e00640 1 --2- 172.21.15.45:0/1485373656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39840b5d50 0x7f3968003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:38.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.084+0000 7f3983400640 1 --2- 172.21.15.45:0/1485373656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3968003fd0 0x7f3968004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:38.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.084+0000 7f3988c00640 1 -- 172.21.15.45:0/1485373656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39840b5d50 msgr2=0x7f3968003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:38.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.084+0000 7f3988c00640 1 --2- 172.21.15.45:0/1485373656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39840b5d50 0x7f3968003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:38.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.084+0000 7f3988c00640 1 -- 172.21.15.45:0/1485373656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3968003fd0 msgr2=0x7f3968004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:38.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.084+0000 7f3988c00640 1 --2- 172.21.15.45:0/1485373656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3968003fd0 0x7f3968004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:38.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.084+0000 7f3988c00640 1 -- 172.21.15.45:0/1485373656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3968003520 con 0x7f3968004d40 2024-04-16T16:15:38.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.085+0000 7f3988c00640 1 --2- 172.21.15.45:0/1485373656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3968004d40 0x7f39680051c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f397c009ac0 tx=0x7f3968005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcca484f157d6aeb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:38.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.086+0000 7f3982a00640 1 -- 172.21.15.45:0/1485373656 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c039070 con 0x7f3968004d40 2024-04-16T16:15:38.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.086+0000 7f3982a00640 1 -- 172.21.15.45:0/1485373656 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f397c02c6b0 con 0x7f3968004d40 2024-04-16T16:15:38.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.086+0000 7f3982a00640 1 -- 172.21.15.45:0/1485373656 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c009690 con 0x7f3968004d40 2024-04-16T16:15:38.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.086+0000 7f3989600640 1 -- 172.21.15.45:0/1485373656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3968004d40 msgr2=0x7f39680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:38.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.086+0000 7f3989600640 1 --2- 172.21.15.45:0/1485373656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3968004d40 0x7f39680051c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f397c009ac0 tx=0x7f3968005830 comp rx=0 tx=0).stop 2024-04-16T16:15:38.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.086+0000 7f3988c00640 1 -- 172.21.15.45:0/1485373656 reap_dead start 2024-04-16T16:15:38.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.086+0000 7f3989600640 1 -- 172.21.15.45:0/1485373656 shutdown_connections 2024-04-16T16:15:38.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.086+0000 7f3989600640 1 -- 172.21.15.45:0/1485373656 >> 172.21.15.45:0/1485373656 conn(0x7f398406b930 msgr2=0x7f398406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:38.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.086+0000 7f3989600640 1 -- 172.21.15.45:0/1485373656 shutdown_connections 2024-04-16T16:15:38.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.086+0000 7f3989600640 1 -- 172.21.15.45:0/1485373656 wait complete. 2024-04-16T16:15:38.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.088+0000 7f3989600640 1 Processor -- start 2024-04-16T16:15:38.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.088+0000 7f3989600640 1 -- start start 2024-04-16T16:15:38.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.089+0000 7f3989600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3968004d40 0x7f39840cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:38.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.089+0000 7f3989600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39840b0a60 0x7f39840b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.089+0000 7f3989600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39840a9540 0x7f39840a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.089+0000 7f3989600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f39840c5fb0 con 0x7f39840a9540 2024-04-16T16:15:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.089+0000 7f3989600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f39840c5cd0 con 0x7f39840b0a60 2024-04-16T16:15:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.089+0000 7f3989600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f39840c6290 con 0x7f3968004d40 2024-04-16T16:15:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.090+0000 7f3988c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39840a9540 0x7f39840a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.090+0000 7f3988c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39840a9540 0x7f39840a99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45750/0 (socket says 172.21.15.45:45750) 2024-04-16T16:15:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.090+0000 7f3988c00640 1 -- 172.21.15.45:0/1079246067 learned_addr learned my addr 172.21.15.45:0/1079246067 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.090+0000 7f3983e00640 1 --2- 172.21.15.45:0/1079246067 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3968004d40 0x7f39840cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.090+0000 7f3983400640 1 --2- 172.21.15.45:0/1079246067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39840b0a60 0x7f39840b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.090+0000 7f3988c00640 1 -- 172.21.15.45:0/1079246067 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3968004d40 msgr2=0x7f39840cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.090+0000 7f3988c00640 1 --2- 172.21.15.45:0/1079246067 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3968004d40 0x7f39840cbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.090+0000 7f3988c00640 1 -- 172.21.15.45:0/1079246067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39840b0a60 msgr2=0x7f39840b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.090+0000 7f3988c00640 1 --2- 172.21.15.45:0/1079246067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39840b0a60 0x7f39840b0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.090+0000 7f3988c00640 1 -- 172.21.15.45:0/1079246067 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3984133f10 con 0x7f39840a9540 2024-04-16T16:15:38.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.090+0000 7f3983e00640 1 --2- 172.21.15.45:0/1079246067 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3968004d40 0x7f39840cbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:15:38.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.090+0000 7f3988c00640 1 --2- 172.21.15.45:0/1079246067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39840a9540 0x7f39840a99c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f397c009be0 tx=0x7f397c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:38.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.091+0000 7f397be00640 1 -- 172.21.15.45:0/1079246067 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c039070 con 0x7f39840a9540 2024-04-16T16:15:38.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.091+0000 7f397be00640 1 -- 172.21.15.45:0/1079246067 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f397c043070 con 0x7f39840a9540 2024-04-16T16:15:38.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.091+0000 7f397be00640 1 -- 172.21.15.45:0/1079246067 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c009690 con 0x7f39840a9540 2024-04-16T16:15:38.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.091+0000 7f3989600640 1 -- 172.21.15.45:0/1079246067 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39841341f0 con 0x7f39840a9540 2024-04-16T16:15:38.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.091+0000 7f3989600640 1 -- 172.21.15.45:0/1079246067 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39841347b0 con 0x7f39840a9540 2024-04-16T16:15:38.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.092+0000 7f3989600640 1 -- 172.21.15.45:0/1079246067 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3984068430 con 0x7f39840a9540 2024-04-16T16:15:38.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.093+0000 7f397be00640 1 -- 172.21.15.45:0/1079246067 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f397c003850 con 0x7f39840a9540 2024-04-16T16:15:38.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.095+0000 7f397be00640 1 --2- 172.21.15.45:0/1079246067 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f396406ca90 0x7f396406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:38.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.095+0000 7f397be00640 1 -- 172.21.15.45:0/1079246067 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6967+0+0 (secure 0 0 0) 0x7f397c0ad280 con 0x7f39840a9540 2024-04-16T16:15:38.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.095+0000 7f3983e00640 1 --2- 172.21.15.45:0/1079246067 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f396406ca90 0x7f396406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:38.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.096+0000 7f3983e00640 1 --2- 172.21.15.45:0/1079246067 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f396406ca90 0x7f396406ef50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f39740013d0 tx=0x7f3974009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:38.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.099+0000 7f397be00640 1 -- 172.21.15.45:0/1079246067 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3984068430 con 0x7f39840a9540 2024-04-16T16:15:38.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.345+0000 7f3989600640 1 -- 172.21.15.45:0/1079246067 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "fs2", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f39840c6950 con 0x7f39840a9540 2024-04-16T16:15:38.829 INFO:tasks.ceph.mgr.x.smithi103.stderr:Exception in thread Thread-14: 2024-04-16T16:15:38.830 INFO:tasks.ceph.mgr.x.smithi103.stderr:Traceback (most recent call last): 2024-04-16T16:15:38.830 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner 2024-04-16T16:15:38.830 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.run() 2024-04-16T16:15:38.830 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 1306, in run 2024-04-16T16:15:38.831 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.function(*self.args, **self.kwargs) 2024-04-16T16:15:38.831 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/share/ceph/mgr/stats/fs/perf_stats.py", line 222, in re_register_queries 2024-04-16T16:15:38.831 INFO:tasks.ceph.mgr.x.smithi103.stderr: if self.mx_last_updated >= ua_last_updated: 2024-04-16T16:15:38.831 INFO:tasks.ceph.mgr.x.smithi103.stderr:AttributeError: 'FSPerfStats' object has no attribute 'mx_last_updated' 2024-04-16T16:15:38.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.840+0000 7f397be00640 1 -- 172.21.15.45:0/1079246067 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "fs2", "var": "allow_standby_replay", "val": "false"}]=0 v33) ==== 119+0+0 (secure 0 0 0) 0x7f39840c6950 con 0x7f39840a9540 2024-04-16T16:15:38.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.845+0000 7f3989600640 1 -- 172.21.15.45:0/1079246067 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f396406ca90 msgr2=0x7f396406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:38.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.845+0000 7f3989600640 1 --2- 172.21.15.45:0/1079246067 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f396406ca90 0x7f396406ef50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f39740013d0 tx=0x7f3974009040 comp rx=0 tx=0).stop 2024-04-16T16:15:38.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.845+0000 7f3989600640 1 -- 172.21.15.45:0/1079246067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39840a9540 msgr2=0x7f39840a99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:38.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.845+0000 7f3989600640 1 --2- 172.21.15.45:0/1079246067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39840a9540 0x7f39840a99c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f397c009be0 tx=0x7f397c004730 comp rx=0 tx=0).stop 2024-04-16T16:15:38.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.845+0000 7f3989600640 1 -- 172.21.15.45:0/1079246067 shutdown_connections 2024-04-16T16:15:38.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.845+0000 7f3989600640 1 --2- 172.21.15.45:0/1079246067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39840a9540 0x7f39840a99c0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:38.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.845+0000 7f3989600640 1 --2- 172.21.15.45:0/1079246067 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f396406ca90 0x7f396406ef50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:38.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.845+0000 7f3989600640 1 --2- 172.21.15.45:0/1079246067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39840b0a60 0x7f39840b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:38.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.845+0000 7f3989600640 1 --2- 172.21.15.45:0/1079246067 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3968004d40 0x7f39840cbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:38.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.845+0000 7f3989600640 1 -- 172.21.15.45:0/1079246067 >> 172.21.15.45:0/1079246067 conn(0x7f398406b930 msgr2=0x7f39840b4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:38.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.845+0000 7f3989600640 1 -- 172.21.15.45:0/1079246067 shutdown_connections 2024-04-16T16:15:38.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:38.845+0000 7f3989600640 1 -- 172.21.15.45:0/1079246067 wait complete. 2024-04-16T16:15:38.871 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'fs2' 2024-04-16T16:15:38.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:15:39.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.088+0000 7f5cb3200640 1 Processor -- start 2024-04-16T16:15:39.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.088+0000 7f5cb3200640 1 -- start start 2024-04-16T16:15:39.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.090+0000 7f5cb3200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5cb406dfb0 0x7f5cb40c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:39.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.090+0000 7f5cb3200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5cb40c5bb0 0x7f5cb415d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:39.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.090+0000 7f5cb3200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cb415c700 con 0x7f5cb4144ff0 2024-04-16T16:15:39.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.090+0000 7f5cb3200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5cb415d2f0 con 0x7f5cb40c5bb0 2024-04-16T16:15:39.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.090+0000 7f5cb3200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5cb415c9c0 con 0x7f5cb406dfb0 2024-04-16T16:15:39.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.090+0000 7f5cb2800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5cb40c5bb0 0x7f5cb415d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:39.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.090+0000 7f5cb2800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5cb40c5bb0 0x7f5cb415d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45756/0 (socket says 172.21.15.45:45756) 2024-04-16T16:15:39.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.090+0000 7f5cb1e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5cb4144ff0 0x7f5cb40cfd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58606/0 (socket says 172.21.15.45:58606) 2024-04-16T16:15:39.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.090+0000 7f5cb2800640 1 -- 172.21.15.45:0/2938011365 learned_addr learned my addr 172.21.15.45:0/2938011365 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:39.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.091+0000 7f5cb2800640 1 -- 172.21.15.45:0/2938011365 >> v2:172.21.15.103:3300/0 conn(0x7f5cb406dfb0 msgr2=0x7f5cb40c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:39.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.091+0000 7f5cb2800640 1 --2- 172.21.15.45:0/2938011365 >> v2:172.21.15.103:3300/0 conn(0x7f5cb406dfb0 0x7f5cb40c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.091+0000 7f5cb2800640 1 -- 172.21.15.45:0/2938011365 >> v1:172.21.15.45:6789/0 conn(0x7f5cb4144ff0 legacy=0x7f5cb40cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:39.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.091+0000 7f5cb2800640 1 -- 172.21.15.45:0/2938011365 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cb415cca0 con 0x7f5cb40c5bb0 2024-04-16T16:15:39.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.091+0000 7f5cb0a00640 1 -- 172.21.15.45:0/2938011365 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1508146290 0 0) 0x7f5cb415c700 con 0x7f5cb4144ff0 2024-04-16T16:15:39.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.092+0000 7f5cb2800640 1 --2- 172.21.15.45:0/2938011365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb40c5bb0 0x7f5cb415d780 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40701c0 tx=0x7f5cac02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3faf18ef0260885b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:39.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.092+0000 7f5cb0a00640 1 -- 172.21.15.45:0/2938011365 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cac02cb20 con 0x7f5cb40c5bb0 2024-04-16T16:15:39.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.092+0000 7f5cb0a00640 1 -- 172.21.15.45:0/2938011365 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cac02ccc0 con 0x7f5cb40c5bb0 2024-04-16T16:15:39.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.092+0000 7f5cb0a00640 1 -- 172.21.15.45:0/2938011365 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cac034610 con 0x7f5cb40c5bb0 2024-04-16T16:15:39.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.093+0000 7f5cb3200640 1 -- 172.21.15.45:0/2938011365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb40c5bb0 msgr2=0x7f5cb415d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:39.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.093+0000 7f5cb3200640 1 --2- 172.21.15.45:0/2938011365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb40c5bb0 0x7f5cb415d780 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40701c0 tx=0x7f5cac02bf80 comp rx=0 tx=0).stop 2024-04-16T16:15:39.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.093+0000 7f5cb3200640 1 -- 172.21.15.45:0/2938011365 shutdown_connections 2024-04-16T16:15:39.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.093+0000 7f5cb3200640 1 --2- 172.21.15.45:0/2938011365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb40c5bb0 0x7f5cb415d780 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.093+0000 7f5cb3200640 1 --2- 172.21.15.45:0/2938011365 >> v2:172.21.15.103:3300/0 conn(0x7f5cb406dfb0 0x7f5cb40c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.093+0000 7f5cb3200640 1 -- 172.21.15.45:0/2938011365 >> 172.21.15.45:0/2938011365 conn(0x7f5cb406b930 msgr2=0x7f5cb406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:39.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.093+0000 7f5cb3200640 1 -- 172.21.15.45:0/2938011365 shutdown_connections 2024-04-16T16:15:39.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.093+0000 7f5cb3200640 1 -- 172.21.15.45:0/2938011365 wait complete. 2024-04-16T16:15:39.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.095+0000 7f5cb3200640 1 Processor -- start 2024-04-16T16:15:39.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.095+0000 7f5cb3200640 1 -- start start 2024-04-16T16:15:39.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.096+0000 7f5cb3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb406dfb0 0x7f5cb41660c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:39.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.096+0000 7f5cb3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5cb40c5bb0 0x7f5cb41f7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:39.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.096+0000 7f5cb3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5cb4144ff0 0x7f5cb41f83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:39.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.096+0000 7f5cb3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb415dcc0 con 0x7f5cb406dfb0 2024-04-16T16:15:39.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.096+0000 7f5cb3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb41453d0 con 0x7f5cb40c5bb0 2024-04-16T16:15:39.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.096+0000 7f5cb3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5cb415de40 con 0x7f5cb4144ff0 2024-04-16T16:15:39.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.096+0000 7f5cb1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb406dfb0 0x7f5cb41660c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:39.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.096+0000 7f5cb1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb406dfb0 0x7f5cb41660c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45770/0 (socket says 172.21.15.45:45770) 2024-04-16T16:15:39.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.096+0000 7f5cb1e00640 1 -- 172.21.15.45:0/3713889444 learned_addr learned my addr 172.21.15.45:0/3713889444 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.097+0000 7f5cb2800640 1 --2- 172.21.15.45:0/3713889444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5cb4144ff0 0x7f5cb41f83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.097+0000 7f5cb1e00640 1 -- 172.21.15.45:0/3713889444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5cb4144ff0 msgr2=0x7f5cb41f83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.097+0000 7f5cb1400640 1 --2- 172.21.15.45:0/3713889444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5cb40c5bb0 0x7f5cb41f7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.097+0000 7f5cb1e00640 1 --2- 172.21.15.45:0/3713889444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5cb4144ff0 0x7f5cb41f83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.097+0000 7f5cb1e00640 1 -- 172.21.15.45:0/3713889444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5cb40c5bb0 msgr2=0x7f5cb41f7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.097+0000 7f5cb1e00640 1 --2- 172.21.15.45:0/3713889444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5cb40c5bb0 0x7f5cb41f7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.097+0000 7f5cb1e00640 1 -- 172.21.15.45:0/3713889444 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cb4165020 con 0x7f5cb406dfb0 2024-04-16T16:15:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.097+0000 7f5cb1400640 1 --2- 172.21.15.45:0/3713889444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5cb40c5bb0 0x7f5cb41f7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.097+0000 7f5cb1e00640 1 --2- 172.21.15.45:0/3713889444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb406dfb0 0x7f5cb41660c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400c460 tx=0x7f5ca400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.097+0000 7f5caa000640 1 -- 172.21.15.45:0/3713889444 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4018070 con 0x7f5cb406dfb0 2024-04-16T16:15:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.097+0000 7f5caa000640 1 -- 172.21.15.45:0/3713889444 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ca4007500 con 0x7f5cb406dfb0 2024-04-16T16:15:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.097+0000 7f5caa000640 1 -- 172.21.15.45:0/3713889444 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4004a20 con 0x7f5cb406dfb0 2024-04-16T16:15:39.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.097+0000 7f5cb3200640 1 -- 172.21.15.45:0/3713889444 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cb4165300 con 0x7f5cb406dfb0 2024-04-16T16:15:39.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.098+0000 7f5cb3200640 1 -- 172.21.15.45:0/3713889444 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cb41658c0 con 0x7f5cb406dfb0 2024-04-16T16:15:39.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.099+0000 7f5cb3200640 1 -- 172.21.15.45:0/3713889444 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cb4075780 con 0x7f5cb406dfb0 2024-04-16T16:15:39.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.100+0000 7f5caa000640 1 -- 172.21.15.45:0/3713889444 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f5ca4004bc0 con 0x7f5cb406dfb0 2024-04-16T16:15:39.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.101+0000 7f5caa000640 1 --2- 172.21.15.45:0/3713889444 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f5c9406ca60 0x7f5c9406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:39.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.101+0000 7f5caa000640 1 -- 172.21.15.45:0/3713889444 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6967+0+0 (secure 0 0 0) 0x7f5ca408b5d0 con 0x7f5cb406dfb0 2024-04-16T16:15:39.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.101+0000 7f5cb1400640 1 --2- 172.21.15.45:0/3713889444 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f5c9406ca60 0x7f5c9406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.102+0000 7f5cb1400640 1 --2- 172.21.15.45:0/3713889444 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f5c9406ca60 0x7f5c9406ef20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0006fd0 tx=0x7f5ca0008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:39.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.105+0000 7f5caa000640 1 -- 172.21.15.45:0/3713889444 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5ca4059360 con 0x7f5cb406dfb0 2024-04-16T16:15:39.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.350+0000 7f5cb3200640 1 -- 172.21.15.45:0/3713889444 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5cb40b59c0 con 0x7f5cb406dfb0 2024-04-16T16:15:39.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.351+0000 7f5caa000640 1 -- 172.21.15.45:0/3713889444 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6484 (secure 0 0 0) 0x7f5ca4058c60 con 0x7f5cb406dfb0 2024-04-16T16:15:39.352 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:39.352 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5056,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3982331497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3982331497},{"type":"v1","addr":"172.21.15.103:6837","nonce":3982331497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5065,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/842795151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":842795151},{"type":"v1","addr":"172.21.15.45:6835","nonce":842795151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:14.878986+0000","modified":"2024-04-16T16:15:23.706709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5646},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5646":{"gid":5646,"name":"a","rank":0,"incarnation":25,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6833/368835060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":368835060},{"type":"v1","addr":"172.21.15.45:6833","nonce":368835060}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"fs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5646,"qdb_cluster":[5646]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:35.289901+0000","modified":"2024-04-16T16:15:38.348161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6835/88032513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":88032513},{"type":"v1","addr":"172.21.15.103:6835","nonce":88032513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"fs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5652,"qdb_cluster":[5652]},"id":4}]} 2024-04-16T16:15:39.352 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-04-16T16:15:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.354+0000 7f5cb3200640 1 -- 172.21.15.45:0/3713889444 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f5c9406ca60 msgr2=0x7f5c9406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.354+0000 7f5cb3200640 1 --2- 172.21.15.45:0/3713889444 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f5c9406ca60 0x7f5c9406ef20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0006fd0 tx=0x7f5ca0008040 comp rx=0 tx=0).stop 2024-04-16T16:15:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.354+0000 7f5cb3200640 1 -- 172.21.15.45:0/3713889444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb406dfb0 msgr2=0x7f5cb41660c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.354+0000 7f5cb3200640 1 --2- 172.21.15.45:0/3713889444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb406dfb0 0x7f5cb41660c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400c460 tx=0x7f5ca400c920 comp rx=0 tx=0).stop 2024-04-16T16:15:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.354+0000 7f5cb3200640 1 -- 172.21.15.45:0/3713889444 shutdown_connections 2024-04-16T16:15:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.354+0000 7f5cb3200640 1 --2- 172.21.15.45:0/3713889444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5cb4144ff0 0x7f5cb41f83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.354+0000 7f5cb3200640 1 --2- 172.21.15.45:0/3713889444 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f5c9406ca60 0x7f5c9406ef20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.354+0000 7f5cb3200640 1 --2- 172.21.15.45:0/3713889444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5cb40c5bb0 0x7f5cb41f7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.354+0000 7f5cb3200640 1 --2- 172.21.15.45:0/3713889444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb406dfb0 0x7f5cb41660c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.354+0000 7f5cb3200640 1 -- 172.21.15.45:0/3713889444 >> 172.21.15.45:0/3713889444 conn(0x7f5cb406b930 msgr2=0x7f5cb40b3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.355+0000 7f5cb3200640 1 -- 172.21.15.45:0/3713889444 shutdown_connections 2024-04-16T16:15:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.355+0000 7f5cb3200640 1 -- 172.21.15.45:0/3713889444 wait complete. 2024-04-16T16:15:39.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:15:39.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.604+0000 7fe79b200640 1 Processor -- start 2024-04-16T16:15:39.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.604+0000 7fe79b200640 1 -- start start 2024-04-16T16:15:39.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.605+0000 7fe79b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe79c161e90 0x7fe79c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:39.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.605+0000 7fe79b200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe79c162270 0x7fe79c0d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:39.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.605+0000 7fe79b200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe79c0d3080 con 0x7fe79c0d52d0 2024-04-16T16:15:39.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.605+0000 7fe79b200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe79c0d2e30 con 0x7fe79c162270 2024-04-16T16:15:39.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.605+0000 7fe79b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe79c0d3360 con 0x7fe79c161e90 2024-04-16T16:15:39.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.606+0000 7fe799400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe79c162270 0x7fe79c0d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:39.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.606+0000 7fe799400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe79c162270 0x7fe79c0d3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45782/0 (socket says 172.21.15.45:45782) 2024-04-16T16:15:39.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.606+0000 7fe799400640 1 -- 172.21.15.45:0/3464742479 learned_addr learned my addr 172.21.15.45:0/3464742479 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:39.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.606+0000 7fe79a800640 1 --1- 172.21.15.45:0/3464742479 >> v1:172.21.15.45:6789/0 conn(0x7fe79c0d52d0 0x7fe79c06ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58612/0 (socket says 172.21.15.45:58612) 2024-04-16T16:15:39.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.606+0000 7fe799e00640 1 --2- 172.21.15.45:0/3464742479 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe79c161e90 0x7fe79c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:39.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.606+0000 7fe799400640 1 -- 172.21.15.45:0/3464742479 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe79c161e90 msgr2=0x7fe79c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:39.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.606+0000 7fe799400640 1 --2- 172.21.15.45:0/3464742479 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe79c161e90 0x7fe79c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.606+0000 7fe799400640 1 -- 172.21.15.45:0/3464742479 >> v1:172.21.15.45:6789/0 conn(0x7fe79c0d52d0 legacy=0x7fe79c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:39.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.606+0000 7fe799400640 1 -- 172.21.15.45:0/3464742479 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79c0d3640 con 0x7fe79c162270 2024-04-16T16:15:39.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.608+0000 7fe799400640 1 --2- 172.21.15.45:0/3464742479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79c162270 0x7fe79c0d3e40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe784009870 tx=0x7fe78402ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3edae10aac6e71f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:39.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.608+0000 7fe798a00640 1 -- 172.21.15.45:0/3464742479 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78402f830 con 0x7fe79c162270 2024-04-16T16:15:39.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.608+0000 7fe798a00640 1 -- 172.21.15.45:0/3464742479 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe78402f9d0 con 0x7fe79c162270 2024-04-16T16:15:39.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.608+0000 7fe798a00640 1 -- 172.21.15.45:0/3464742479 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78402fda0 con 0x7fe79c162270 2024-04-16T16:15:39.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.609+0000 7fe79b200640 1 -- 172.21.15.45:0/3464742479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79c162270 msgr2=0x7fe79c0d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:39.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.609+0000 7fe79b200640 1 --2- 172.21.15.45:0/3464742479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79c162270 0x7fe79c0d3e40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe784009870 tx=0x7fe78402ba10 comp rx=0 tx=0).stop 2024-04-16T16:15:39.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.609+0000 7fe79b200640 1 -- 172.21.15.45:0/3464742479 shutdown_connections 2024-04-16T16:15:39.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.609+0000 7fe79b200640 1 --2- 172.21.15.45:0/3464742479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79c162270 0x7fe79c0d3e40 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.609+0000 7fe79b200640 1 --2- 172.21.15.45:0/3464742479 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe79c161e90 0x7fe79c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.609+0000 7fe79b200640 1 -- 172.21.15.45:0/3464742479 >> 172.21.15.45:0/3464742479 conn(0x7fe79c0d1940 msgr2=0x7fe79c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:39.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.609+0000 7fe79b200640 1 -- 172.21.15.45:0/3464742479 shutdown_connections 2024-04-16T16:15:39.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.609+0000 7fe79b200640 1 -- 172.21.15.45:0/3464742479 wait complete. 2024-04-16T16:15:39.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.611+0000 7fe79b200640 1 Processor -- start 2024-04-16T16:15:39.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.611+0000 7fe79b200640 1 -- start start 2024-04-16T16:15:39.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.612+0000 7fe79b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe79c0d52d0 0x7fe79c1f5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:39.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.612+0000 7fe79b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79c161e90 0x7fe79c14ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:39.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.612+0000 7fe79b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe79c162270 0x7fe79c0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:39.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.612+0000 7fe79b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe79c06d550 con 0x7fe79c161e90 2024-04-16T16:15:39.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.612+0000 7fe79b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe79c0cfd10 con 0x7fe79c0d52d0 2024-04-16T16:15:39.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.612+0000 7fe79b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe79c06d6d0 con 0x7fe79c162270 2024-04-16T16:15:39.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.612+0000 7fe799400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79c161e90 0x7fe79c14ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:39.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.612+0000 7fe799400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79c161e90 0x7fe79c14ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45784/0 (socket says 172.21.15.45:45784) 2024-04-16T16:15:39.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.612+0000 7fe799400640 1 -- 172.21.15.45:0/3937482812 learned_addr learned my addr 172.21.15.45:0/3937482812 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:39.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.612+0000 7fe79a800640 1 --2- 172.21.15.45:0/3937482812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe79c162270 0x7fe79c0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:39.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.612+0000 7fe799400640 1 -- 172.21.15.45:0/3937482812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe79c162270 msgr2=0x7fe79c0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:39.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.613+0000 7fe799400640 1 --2- 172.21.15.45:0/3937482812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe79c162270 0x7fe79c0ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.613+0000 7fe799400640 1 -- 172.21.15.45:0/3937482812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe79c0d52d0 msgr2=0x7fe79c1f5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:39.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.613+0000 7fe799e00640 1 --2- 172.21.15.45:0/3937482812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe79c0d52d0 0x7fe79c1f5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:39.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.613+0000 7fe799400640 1 --2- 172.21.15.45:0/3937482812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe79c0d52d0 0x7fe79c1f5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.613+0000 7fe799400640 1 -- 172.21.15.45:0/3937482812 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79c1f80c0 con 0x7fe79c161e90 2024-04-16T16:15:39.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.613+0000 7fe799400640 1 --2- 172.21.15.45:0/3937482812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79c161e90 0x7fe79c14ceb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe7840090b0 tx=0x7fe784004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.613+0000 7fe792000640 1 -- 172.21.15.45:0/3937482812 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe784046070 con 0x7fe79c161e90 2024-04-16T16:15:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.613+0000 7fe792000640 1 -- 172.21.15.45:0/3937482812 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe784037c90 con 0x7fe79c161e90 2024-04-16T16:15:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.613+0000 7fe792000640 1 -- 172.21.15.45:0/3937482812 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe784041680 con 0x7fe79c161e90 2024-04-16T16:15:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.613+0000 7fe79b200640 1 -- 172.21.15.45:0/3937482812 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe79c1f83a0 con 0x7fe79c161e90 2024-04-16T16:15:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.614+0000 7fe79b200640 1 -- 172.21.15.45:0/3937482812 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe79c1f8960 con 0x7fe79c161e90 2024-04-16T16:15:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.615+0000 7fe79b200640 1 -- 172.21.15.45:0/3937482812 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe79c075780 con 0x7fe79c161e90 2024-04-16T16:15:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.616+0000 7fe792000640 1 -- 172.21.15.45:0/3937482812 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fe784049070 con 0x7fe79c161e90 2024-04-16T16:15:39.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.617+0000 7fe792000640 1 --2- 172.21.15.45:0/3937482812 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe77806ca60 0x7fe77806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:39.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.617+0000 7fe792000640 1 -- 172.21.15.45:0/3937482812 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6967+0+0 (secure 0 0 0) 0x7fe7840ae160 con 0x7fe79c161e90 2024-04-16T16:15:39.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.617+0000 7fe799e00640 1 --2- 172.21.15.45:0/3937482812 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe77806ca60 0x7fe77806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:39.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.619+0000 7fe799e00640 1 --2- 172.21.15.45:0/3937482812 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe77806ca60 0x7fe77806ef20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe78c006fd0 tx=0x7fe78c008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:39.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.621+0000 7fe792000640 1 -- 172.21.15.45:0/3937482812 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe784033030 con 0x7fe79c161e90 2024-04-16T16:15:39.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.827+0000 7fe79b200640 1 -- 172.21.15.45:0/3937482812 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe79c06dae0 con 0x7fe79c161e90 2024-04-16T16:15:39.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.829+0000 7fe792000640 1 -- 172.21.15.45:0/3937482812 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+21006 (secure 0 0 0) 0x7fe78407be70 con 0x7fe79c161e90 2024-04-16T16:15:39.830 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:39.830 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":50,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:15:36.815083+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"fs1_metadata","create_time":"2024-04-16T16:15:13.250782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"fs1_data","create_time":"2024-04-16T16:15:13.856986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"fs2_metadata","create_time":"2024-04-16T16:15:33.215042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"fs2_data","create_time":"2024-04-16T16:15:34.271426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.45:6833/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:15:39.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.832+0000 7fe79b200640 1 -- 172.21.15.45:0/3937482812 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe77806ca60 msgr2=0x7fe77806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:39.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.832+0000 7fe79b200640 1 --2- 172.21.15.45:0/3937482812 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe77806ca60 0x7fe77806ef20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe78c006fd0 tx=0x7fe78c008040 comp rx=0 tx=0).stop 2024-04-16T16:15:39.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.832+0000 7fe79b200640 1 -- 172.21.15.45:0/3937482812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79c161e90 msgr2=0x7fe79c14ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:39.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.832+0000 7fe79b200640 1 --2- 172.21.15.45:0/3937482812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79c161e90 0x7fe79c14ceb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe7840090b0 tx=0x7fe784004730 comp rx=0 tx=0).stop 2024-04-16T16:15:39.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.832+0000 7fe79b200640 1 -- 172.21.15.45:0/3937482812 shutdown_connections 2024-04-16T16:15:39.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.832+0000 7fe79b200640 1 --2- 172.21.15.45:0/3937482812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe79c162270 0x7fe79c0ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.832+0000 7fe79b200640 1 --2- 172.21.15.45:0/3937482812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79c161e90 0x7fe79c14ceb0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.832+0000 7fe79b200640 1 --2- 172.21.15.45:0/3937482812 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe77806ca60 0x7fe77806ef20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.832+0000 7fe79b200640 1 --2- 172.21.15.45:0/3937482812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe79c0d52d0 0x7fe79c1f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:39.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.832+0000 7fe79b200640 1 -- 172.21.15.45:0/3937482812 >> 172.21.15.45:0/3937482812 conn(0x7fe79c0d1940 msgr2=0x7fe79c0b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:39.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.832+0000 7fe79b200640 1 -- 172.21.15.45:0/3937482812 shutdown_connections 2024-04-16T16:15:39.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:39.832+0000 7fe79b200640 1 -- 172.21.15.45:0/3937482812 wait complete. 2024-04-16T16:15:39.842 INFO:tasks.ceph.mgr.x.smithi103.stderr:Exception in thread Thread-16: 2024-04-16T16:15:39.842 INFO:tasks.ceph.mgr.x.smithi103.stderr:Traceback (most recent call last): 2024-04-16T16:15:39.842 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner 2024-04-16T16:15:39.843 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.run() 2024-04-16T16:15:39.843 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 1306, in run 2024-04-16T16:15:39.843 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.function(*self.args, **self.kwargs) 2024-04-16T16:15:39.844 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/share/ceph/mgr/stats/fs/perf_stats.py", line 222, in re_register_queries 2024-04-16T16:15:39.844 INFO:tasks.ceph.mgr.x.smithi103.stderr: if self.mx_last_updated >= ua_last_updated: 2024-04-16T16:15:39.844 INFO:tasks.ceph.mgr.x.smithi103.stderr:AttributeError: 'FSPerfStats' object has no attribute 'mx_last_updated' 2024-04-16T16:15:39.852 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-16T16:15:39.853 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:15:40.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.067+0000 7fc8d7200640 1 Processor -- start 2024-04-16T16:15:40.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.067+0000 7fc8d7200640 1 -- start start 2024-04-16T16:15:40.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.069+0000 7fc8d7200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc8d815ccb0 0x7fc8d806df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:40.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.069+0000 7fc8d7200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8d815d360 con 0x7fc8d8146220 2024-04-16T16:15:40.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.069+0000 7fc8d7200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8d815d640 con 0x7fc8d806e480 2024-04-16T16:15:40.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.069+0000 7fc8d7200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc8d815d130 con 0x7fc8d815ccb0 2024-04-16T16:15:40.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.069+0000 7fc8d5e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc8d8146220 0x7fc8d806f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58618/0 (socket says 172.21.15.45:58618) 2024-04-16T16:15:40.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.069+0000 7fc8d5e00640 1 -- 172.21.15.45:0/3151691715 learned_addr learned my addr 172.21.15.45:0/3151691715 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:40.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d5400640 1 --2- 172.21.15.45:0/3151691715 >> v2:172.21.15.103:3300/0 conn(0x7fc8d815ccb0 0x7fc8d806df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d4a00640 1 -- 172.21.15.45:0/3151691715 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2372926227 0 0) 0x7fc8d815d360 con 0x7fc8d8146220 2024-04-16T16:15:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d4a00640 1 -- 172.21.15.45:0/3151691715 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8bc003520 con 0x7fc8d8146220 2024-04-16T16:15:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d4a00640 1 -- 172.21.15.45:0/3151691715 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2836291092 0 0) 0x7fc8bc003520 con 0x7fc8d8146220 2024-04-16T16:15:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d4a00640 1 -- 172.21.15.45:0/3151691715 >> v2:172.21.15.103:3300/0 conn(0x7fc8d815ccb0 msgr2=0x7fc8d806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d4a00640 1 --2- 172.21.15.45:0/3151691715 >> v2:172.21.15.103:3300/0 conn(0x7fc8d815ccb0 0x7fc8d806df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d4a00640 1 -- 172.21.15.45:0/3151691715 >> v1:172.21.15.103:6789/0 conn(0x7fc8d806e480 legacy=0x7fc8d81608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d4a00640 1 -- 172.21.15.45:0/3151691715 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d815d920 con 0x7fc8d8146220 2024-04-16T16:15:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d4a00640 1 -- 172.21.15.45:0/3151691715 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fc8c8001670 con 0x7fc8d8146220 2024-04-16T16:15:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d4a00640 1 -- 172.21.15.45:0/3151691715 >> v1:172.21.15.45:6789/0 conn(0x7fc8d8146220 legacy=0x7fc8d806f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d4a00640 1 --2- 172.21.15.45:0/3151691715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8bc003a00 0x7fc8bc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d4a00640 1 --2- 172.21.15.45:0/3151691715 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8bc004800 0x7fc8bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d4a00640 1 --2- 172.21.15.45:0/3151691715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8bc0051c0 0x7fc8bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:40.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d4a00640 1 -- 172.21.15.45:0/3151691715 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc8bc00c090 con 0x7fc8bc0051c0 2024-04-16T16:15:40.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d4a00640 1 -- 172.21.15.45:0/3151691715 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc8bc00c240 con 0x7fc8bc003a00 2024-04-16T16:15:40.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.070+0000 7fc8d4a00640 1 -- 172.21.15.45:0/3151691715 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc8bc00c440 con 0x7fc8bc004800 2024-04-16T16:15:40.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.071+0000 7fc8d6800640 1 --2- 172.21.15.45:0/3151691715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8bc0051c0 0x7fc8bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:40.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.071+0000 7fc8d5400640 1 --2- 172.21.15.45:0/3151691715 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8bc004800 0x7fc8bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:40.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.071+0000 7fc8d5e00640 1 --2- 172.21.15.45:0/3151691715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8bc003a00 0x7fc8bc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:40.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.071+0000 7fc8d6800640 1 -- 172.21.15.45:0/3151691715 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8bc004800 msgr2=0x7fc8bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:40.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.071+0000 7fc8d6800640 1 --2- 172.21.15.45:0/3151691715 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8bc004800 0x7fc8bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.071+0000 7fc8d6800640 1 -- 172.21.15.45:0/3151691715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8bc003a00 msgr2=0x7fc8bc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:40.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.071+0000 7fc8d6800640 1 --2- 172.21.15.45:0/3151691715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8bc003a00 0x7fc8bc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.071+0000 7fc8d6800640 1 -- 172.21.15.45:0/3151691715 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8bc003520 con 0x7fc8bc0051c0 2024-04-16T16:15:40.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.071+0000 7fc8d5e00640 1 --2- 172.21.15.45:0/3151691715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8bc003a00 0x7fc8bc003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:40.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.071+0000 7fc8d5400640 1 --2- 172.21.15.45:0/3151691715 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8bc004800 0x7fc8bc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:40.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.072+0000 7fc8d6800640 1 -- 172.21.15.45:0/3151691715 reap_dead start 2024-04-16T16:15:40.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.072+0000 7fc8d6800640 1 --2- 172.21.15.45:0/3151691715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8bc0051c0 0x7fc8bc00b9c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0001e50 tx=0x7fc8bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc4551d32523768 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:40.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.073+0000 7fc8d4a00640 1 -- 172.21.15.45:0/3151691715 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0001a70 con 0x7fc8bc0051c0 2024-04-16T16:15:40.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.073+0000 7fc8d4a00640 1 -- 172.21.15.45:0/3151691715 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8d0003bb0 con 0x7fc8bc0051c0 2024-04-16T16:15:40.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.073+0000 7fc8d4a00640 1 -- 172.21.15.45:0/3151691715 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d00050d0 con 0x7fc8bc0051c0 2024-04-16T16:15:40.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.073+0000 7fc8d7200640 1 -- 172.21.15.45:0/3151691715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8bc0051c0 msgr2=0x7fc8bc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:40.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.073+0000 7fc8d7200640 1 --2- 172.21.15.45:0/3151691715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8bc0051c0 0x7fc8bc00b9c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0001e50 tx=0x7fc8bc005830 comp rx=0 tx=0).stop 2024-04-16T16:15:40.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.073+0000 7fc8d7200640 1 -- 172.21.15.45:0/3151691715 shutdown_connections 2024-04-16T16:15:40.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.073+0000 7fc8d7200640 1 --2- 172.21.15.45:0/3151691715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8bc0051c0 0x7fc8bc00b9c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.073+0000 7fc8d7200640 1 -- 172.21.15.45:0/3151691715 >> 172.21.15.45:0/3151691715 conn(0x7fc8d806b930 msgr2=0x7fc8d806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:40.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.073+0000 7fc8d7200640 1 -- 172.21.15.45:0/3151691715 shutdown_connections 2024-04-16T16:15:40.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.073+0000 7fc8d7200640 1 -- 172.21.15.45:0/3151691715 wait complete. 2024-04-16T16:15:40.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.075+0000 7fc8d7200640 1 Processor -- start 2024-04-16T16:15:40.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.075+0000 7fc8d7200640 1 -- start start 2024-04-16T16:15:40.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.076+0000 7fc8d7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8bc0051c0 0x7fc8d81e5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:40.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.076+0000 7fc8d7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d80c89d0 0x7fc8d815c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:40.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.076+0000 7fc8d7200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d80c9570 0x7fc8d80c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:40.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.076+0000 7fc8d7200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d806df40 con 0x7fc8d80c9570 2024-04-16T16:15:40.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.076+0000 7fc8d7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d8160410 con 0x7fc8bc0051c0 2024-04-16T16:15:40.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.076+0000 7fc8d7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d806e0c0 con 0x7fc8d80c89d0 2024-04-16T16:15:40.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.077+0000 7fc8d6800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d80c9570 0x7fc8d80c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:40.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.077+0000 7fc8d6800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d80c9570 0x7fc8d80c99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45812/0 (socket says 172.21.15.45:45812) 2024-04-16T16:15:40.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.077+0000 7fc8d6800640 1 -- 172.21.15.45:0/988558227 learned_addr learned my addr 172.21.15.45:0/988558227 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:40.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.077+0000 7fc8d5e00640 1 --2- 172.21.15.45:0/988558227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8bc0051c0 0x7fc8d81e5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:40.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.077+0000 7fc8d5400640 1 --2- 172.21.15.45:0/988558227 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d80c89d0 0x7fc8d815c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.077+0000 7fc8d6800640 1 -- 172.21.15.45:0/988558227 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d80c89d0 msgr2=0x7fc8d815c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.077+0000 7fc8d6800640 1 --2- 172.21.15.45:0/988558227 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d80c89d0 0x7fc8d815c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.077+0000 7fc8d6800640 1 -- 172.21.15.45:0/988558227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8bc0051c0 msgr2=0x7fc8d81e5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.077+0000 7fc8d6800640 1 --2- 172.21.15.45:0/988558227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8bc0051c0 0x7fc8d81e5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.077+0000 7fc8d6800640 1 -- 172.21.15.45:0/988558227 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d8142080 con 0x7fc8d80c9570 2024-04-16T16:15:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.078+0000 7fc8d6800640 1 --2- 172.21.15.45:0/988558227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d80c9570 0x7fc8d80c99f0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc8d002c060 tx=0x7fc8d00014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.078+0000 7fc8ce000640 1 -- 172.21.15.45:0/988558227 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0007db0 con 0x7fc8d80c9570 2024-04-16T16:15:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.078+0000 7fc8ce000640 1 -- 172.21.15.45:0/988558227 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8d0006d60 con 0x7fc8d80c9570 2024-04-16T16:15:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.078+0000 7fc8ce000640 1 -- 172.21.15.45:0/988558227 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0007230 con 0x7fc8d80c9570 2024-04-16T16:15:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.078+0000 7fc8d7200640 1 -- 172.21.15.45:0/988558227 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d81e6810 con 0x7fc8d80c9570 2024-04-16T16:15:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.078+0000 7fc8d7200640 1 -- 172.21.15.45:0/988558227 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d81e6dd0 con 0x7fc8d80c9570 2024-04-16T16:15:40.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.079+0000 7fc8d7200640 1 -- 172.21.15.45:0/988558227 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d8075780 con 0x7fc8d80c9570 2024-04-16T16:15:40.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.080+0000 7fc8ce000640 1 -- 172.21.15.45:0/988558227 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fc8d00073d0 con 0x7fc8d80c9570 2024-04-16T16:15:40.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.081+0000 7fc8ce000640 1 --2- 172.21.15.45:0/988558227 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc8b806ca90 0x7fc8b806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:40.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.081+0000 7fc8ce000640 1 -- 172.21.15.45:0/988558227 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6967+0+0 (secure 0 0 0) 0x7fc8d00ae3f0 con 0x7fc8d80c9570 2024-04-16T16:15:40.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.082+0000 7fc8d5e00640 1 --2- 172.21.15.45:0/988558227 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc8b806ca90 0x7fc8b806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:40.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.083+0000 7fc8d5e00640 1 --2- 172.21.15.45:0/988558227 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc8b806ca90 0x7fc8b806ef50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8001200 tx=0x7fc8c8009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:40.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.086+0000 7fc8ce000640 1 -- 172.21.15.45:0/988558227 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc8d0033030 con 0x7fc8d80c9570 2024-04-16T16:15:40.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.288+0000 7fc8d7200640 1 -- 172.21.15.45:0/988558227 --> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc8d80cff60 con 0x7fc8b806ca90 2024-04-16T16:15:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.331+0000 7fc8ce000640 1 -- 172.21.15.45:0/988558227 <== mgr.4658 v2:172.21.15.103:6832/280591707 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490476 (secure 0 0 0) 0x7fc8d80cff60 con 0x7fc8b806ca90 2024-04-16T16:15:40.334 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:40.341 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:15:40.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.342+0000 7fc8d7200640 1 -- 172.21.15.45:0/988558227 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc8b806ca90 msgr2=0x7fc8b806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:40.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.342+0000 7fc8d7200640 1 --2- 172.21.15.45:0/988558227 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc8b806ca90 0x7fc8b806ef50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8001200 tx=0x7fc8c8009040 comp rx=0 tx=0).stop 2024-04-16T16:15:40.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.342+0000 7fc8d7200640 1 -- 172.21.15.45:0/988558227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d80c9570 msgr2=0x7fc8d80c99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:40.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.342+0000 7fc8d7200640 1 --2- 172.21.15.45:0/988558227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d80c9570 0x7fc8d80c99f0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc8d002c060 tx=0x7fc8d00014a0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.342+0000 7fc8d7200640 1 -- 172.21.15.45:0/988558227 shutdown_connections 2024-04-16T16:15:40.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.342+0000 7fc8d7200640 1 --2- 172.21.15.45:0/988558227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d80c9570 0x7fc8d80c99f0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.342+0000 7fc8d7200640 1 --2- 172.21.15.45:0/988558227 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d80c89d0 0x7fc8d815c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.342+0000 7fc8d7200640 1 --2- 172.21.15.45:0/988558227 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc8b806ca90 0x7fc8b806ef50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.342+0000 7fc8d7200640 1 --2- 172.21.15.45:0/988558227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8bc0051c0 0x7fc8d81e5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.342+0000 7fc8d7200640 1 -- 172.21.15.45:0/988558227 >> 172.21.15.45:0/988558227 conn(0x7fc8d806b930 msgr2=0x7fc8d806e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:40.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.342+0000 7fc8d7200640 1 -- 172.21.15.45:0/988558227 shutdown_connections 2024-04-16T16:15:40.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.342+0000 7fc8d7200640 1 -- 172.21.15.45:0/988558227 wait complete. 2024-04-16T16:15:40.368 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-16T16:15:39.135904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3752738,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":289,"num_read_kb":386,"num_write":277,"num_write_kb":5532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":220,"ondisk_log_size":220,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":295,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244840,"kb_used_data":11992,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754729880,"statfs":{"total":773094113280,"available":772843397120,"internally_reserved":0,"allocated":12279808,"data_stored":9525738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.361336"},"pg_stats":[{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.828847+0000","last_change":"2024-04-16T16:15:35.833708+0000","last_active":"2024-04-16T16:15:36.828847+0000","last_peered":"2024-04-16T16:15:36.828847+0000","last_clean":"2024-04-16T16:15:36.828847+0000","last_became_active":"2024-04-16T16:15:35.833408+0000","last_became_peered":"2024-04-16T16:15:35.833408+0000","last_unstale":"2024-04-16T16:15:36.828847+0000","last_undegraded":"2024-04-16T16:15:36.828847+0000","last_fullsized":"2024-04-16T16:15:36.828847+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:05:58.589624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"35'128","reported_seq":268,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829661+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:15:36.829661+0000","last_peered":"2024-04-16T16:15:36.829661+0000","last_clean":"2024-04-16T16:15:36.829661+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:15:36.829661+0000","last_undegraded":"2024-04-16T16:15:36.829661+0000","last_fullsized":"2024-04-16T16:15:36.829661+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"41'2","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.823795+0000","last_change":"2024-04-16T16:15:14.420968+0000","last_active":"2024-04-16T16:15:35.823795+0000","last_peered":"2024-04-16T16:15:35.823795+0000","last_clean":"2024-04-16T16:15:35.823795+0000","last_became_active":"2024-04-16T16:15:14.418308+0000","last_became_peered":"2024-04-16T16:15:14.418308+0000","last_unstale":"2024-04-16T16:15:35.823795+0000","last_undegraded":"2024-04-16T16:15:35.823795+0000","last_fullsized":"2024-04-16T16:15:35.823795+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:56.611238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811608+0000","last_change":"2024-04-16T16:15:15.444726+0000","last_active":"2024-04-16T16:15:35.811608+0000","last_peered":"2024-04-16T16:15:35.811608+0000","last_clean":"2024-04-16T16:15:35.811608+0000","last_became_active":"2024-04-16T16:15:15.444386+0000","last_became_peered":"2024-04-16T16:15:15.444386+0000","last_unstale":"2024-04-16T16:15:35.811608+0000","last_undegraded":"2024-04-16T16:15:35.811608+0000","last_fullsized":"2024-04-16T16:15:35.811608+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:34.706031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818397+0000","last_change":"2024-04-16T16:15:34.828397+0000","last_active":"2024-04-16T16:15:35.818397+0000","last_peered":"2024-04-16T16:15:35.818397+0000","last_clean":"2024-04-16T16:15:35.818397+0000","last_became_active":"2024-04-16T16:15:34.828110+0000","last_became_peered":"2024-04-16T16:15:34.828110+0000","last_unstale":"2024-04-16T16:15:35.818397+0000","last_undegraded":"2024-04-16T16:15:35.818397+0000","last_fullsized":"2024-04-16T16:15:35.818397+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:29:06.848116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.842023+0000","last_change":"2024-04-16T16:15:35.837402+0000","last_active":"2024-04-16T16:15:35.842023+0000","last_peered":"2024-04-16T16:15:35.842023+0000","last_clean":"2024-04-16T16:15:35.842023+0000","last_became_active":"2024-04-16T16:15:35.834639+0000","last_became_peered":"2024-04-16T16:15:35.834639+0000","last_unstale":"2024-04-16T16:15:35.842023+0000","last_undegraded":"2024-04-16T16:15:35.842023+0000","last_fullsized":"2024-04-16T16:15:35.842023+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:04.283280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.805491+0000","last_change":"2024-04-16T16:15:14.413186+0000","last_active":"2024-04-16T16:15:35.805491+0000","last_peered":"2024-04-16T16:15:35.805491+0000","last_clean":"2024-04-16T16:15:35.805491+0000","last_became_active":"2024-04-16T16:15:14.412710+0000","last_became_peered":"2024-04-16T16:15:14.412710+0000","last_unstale":"2024-04-16T16:15:35.805491+0000","last_undegraded":"2024-04-16T16:15:35.805491+0000","last_fullsized":"2024-04-16T16:15:35.805491+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:43:43.352649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831550+0000","last_change":"2024-04-16T16:15:15.448491+0000","last_active":"2024-04-16T16:15:36.831550+0000","last_peered":"2024-04-16T16:15:36.831550+0000","last_clean":"2024-04-16T16:15:36.831550+0000","last_became_active":"2024-04-16T16:15:15.448176+0000","last_became_peered":"2024-04-16T16:15:15.448176+0000","last_unstale":"2024-04-16T16:15:36.831550+0000","last_undegraded":"2024-04-16T16:15:36.831550+0000","last_fullsized":"2024-04-16T16:15:36.831550+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:44.518137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"49'2","reported_seq":14,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.844887+0000","last_change":"2024-04-16T16:15:34.821153+0000","last_active":"2024-04-16T16:15:35.844887+0000","last_peered":"2024-04-16T16:15:35.844887+0000","last_clean":"2024-04-16T16:15:35.844887+0000","last_became_active":"2024-04-16T16:15:34.820846+0000","last_became_peered":"2024-04-16T16:15:34.820846+0000","last_unstale":"2024-04-16T16:15:35.844887+0000","last_undegraded":"2024-04-16T16:15:35.844887+0000","last_fullsized":"2024-04-16T16:15:35.844887+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:23:32.286883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830861+0000","last_change":"2024-04-16T16:15:35.828889+0000","last_active":"2024-04-16T16:15:36.830861+0000","last_peered":"2024-04-16T16:15:36.830861+0000","last_clean":"2024-04-16T16:15:36.830861+0000","last_became_active":"2024-04-16T16:15:35.828595+0000","last_became_peered":"2024-04-16T16:15:35.828595+0000","last_unstale":"2024-04-16T16:15:36.830861+0000","last_undegraded":"2024-04-16T16:15:36.830861+0000","last_fullsized":"2024-04-16T16:15:36.830861+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:37:51.603270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.808258+0000","last_change":"2024-04-16T16:15:14.425948+0000","last_active":"2024-04-16T16:15:35.808258+0000","last_peered":"2024-04-16T16:15:35.808258+0000","last_clean":"2024-04-16T16:15:35.808258+0000","last_became_active":"2024-04-16T16:15:14.425692+0000","last_became_peered":"2024-04-16T16:15:14.425692+0000","last_unstale":"2024-04-16T16:15:35.808258+0000","last_undegraded":"2024-04-16T16:15:35.808258+0000","last_fullsized":"2024-04-16T16:15:35.808258+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:16.172813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":25,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.817579+0000","last_change":"2024-04-16T16:15:15.446666+0000","last_active":"2024-04-16T16:15:35.817579+0000","last_peered":"2024-04-16T16:15:35.817579+0000","last_clean":"2024-04-16T16:15:35.817579+0000","last_became_active":"2024-04-16T16:15:15.444672+0000","last_became_peered":"2024-04-16T16:15:15.444672+0000","last_unstale":"2024-04-16T16:15:35.817579+0000","last_undegraded":"2024-04-16T16:15:35.817579+0000","last_fullsized":"2024-04-16T16:15:35.817579+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:39:57.997770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830972+0000","last_change":"2024-04-16T16:15:34.820186+0000","last_active":"2024-04-16T16:15:36.830972+0000","last_peered":"2024-04-16T16:15:36.830972+0000","last_clean":"2024-04-16T16:15:36.830972+0000","last_became_active":"2024-04-16T16:15:34.819904+0000","last_became_peered":"2024-04-16T16:15:34.819904+0000","last_unstale":"2024-04-16T16:15:36.830972+0000","last_undegraded":"2024-04-16T16:15:36.830972+0000","last_fullsized":"2024-04-16T16:15:36.830972+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:08:25.566876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831954+0000","last_change":"2024-04-16T16:15:35.828108+0000","last_active":"2024-04-16T16:15:36.831954+0000","last_peered":"2024-04-16T16:15:36.831954+0000","last_clean":"2024-04-16T16:15:36.831954+0000","last_became_active":"2024-04-16T16:15:35.827263+0000","last_became_peered":"2024-04-16T16:15:35.827263+0000","last_unstale":"2024-04-16T16:15:36.831954+0000","last_undegraded":"2024-04-16T16:15:36.831954+0000","last_fullsized":"2024-04-16T16:15:36.831954+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:08:26.582991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":29,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832113+0000","last_change":"2024-04-16T16:15:14.414769+0000","last_active":"2024-04-16T16:15:36.832113+0000","last_peered":"2024-04-16T16:15:36.832113+0000","last_clean":"2024-04-16T16:15:36.832113+0000","last_became_active":"2024-04-16T16:15:14.414402+0000","last_became_peered":"2024-04-16T16:15:14.414402+0000","last_unstale":"2024-04-16T16:15:36.832113+0000","last_undegraded":"2024-04-16T16:15:36.832113+0000","last_fullsized":"2024-04-16T16:15:36.832113+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:07:59.840158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829924+0000","last_change":"2024-04-16T16:15:15.440497+0000","last_active":"2024-04-16T16:15:36.829924+0000","last_peered":"2024-04-16T16:15:36.829924+0000","last_clean":"2024-04-16T16:15:36.829924+0000","last_became_active":"2024-04-16T16:15:15.440205+0000","last_became_peered":"2024-04-16T16:15:15.440205+0000","last_unstale":"2024-04-16T16:15:36.829924+0000","last_undegraded":"2024-04-16T16:15:36.829924+0000","last_fullsized":"2024-04-16T16:15:36.829924+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:29:51.615992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.812933+0000","last_change":"2024-04-16T16:15:34.817957+0000","last_active":"2024-04-16T16:15:35.812933+0000","last_peered":"2024-04-16T16:15:35.812933+0000","last_clean":"2024-04-16T16:15:35.812933+0000","last_became_active":"2024-04-16T16:15:34.817022+0000","last_became_peered":"2024-04-16T16:15:34.817022+0000","last_unstale":"2024-04-16T16:15:35.812933+0000","last_undegraded":"2024-04-16T16:15:35.812933+0000","last_fullsized":"2024-04-16T16:15:35.812933+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:21:52.222598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830407+0000","last_change":"2024-04-16T16:15:35.848145+0000","last_active":"2024-04-16T16:15:36.830407+0000","last_peered":"2024-04-16T16:15:36.830407+0000","last_clean":"2024-04-16T16:15:36.830407+0000","last_became_active":"2024-04-16T16:15:35.847853+0000","last_became_peered":"2024-04-16T16:15:35.847853+0000","last_unstale":"2024-04-16T16:15:36.830407+0000","last_undegraded":"2024-04-16T16:15:36.830407+0000","last_fullsized":"2024-04-16T16:15:36.830407+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:07:21.242733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":30,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831154+0000","last_change":"2024-04-16T16:15:14.427316+0000","last_active":"2024-04-16T16:15:36.831154+0000","last_peered":"2024-04-16T16:15:36.831154+0000","last_clean":"2024-04-16T16:15:36.831154+0000","last_became_active":"2024-04-16T16:15:14.426635+0000","last_became_peered":"2024-04-16T16:15:14.426635+0000","last_unstale":"2024-04-16T16:15:36.831154+0000","last_undegraded":"2024-04-16T16:15:36.831154+0000","last_fullsized":"2024-04-16T16:15:36.831154+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:43.504451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830660+0000","last_change":"2024-04-16T16:15:15.443473+0000","last_active":"2024-04-16T16:15:36.830660+0000","last_peered":"2024-04-16T16:15:36.830660+0000","last_clean":"2024-04-16T16:15:36.830660+0000","last_became_active":"2024-04-16T16:15:15.439092+0000","last_became_peered":"2024-04-16T16:15:15.439092+0000","last_unstale":"2024-04-16T16:15:36.830660+0000","last_undegraded":"2024-04-16T16:15:36.830660+0000","last_fullsized":"2024-04-16T16:15:36.830660+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:49:28.175995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"49'1","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.839533+0000","last_change":"2024-04-16T16:15:34.827831+0000","last_active":"2024-04-16T16:15:35.839533+0000","last_peered":"2024-04-16T16:15:35.839533+0000","last_clean":"2024-04-16T16:15:35.839533+0000","last_became_active":"2024-04-16T16:15:34.827496+0000","last_became_peered":"2024-04-16T16:15:34.827496+0000","last_unstale":"2024-04-16T16:15:35.839533+0000","last_undegraded":"2024-04-16T16:15:35.839533+0000","last_fullsized":"2024-04-16T16:15:35.839533+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:06:11.243617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.838650+0000","last_change":"2024-04-16T16:15:35.834522+0000","last_active":"2024-04-16T16:15:35.838650+0000","last_peered":"2024-04-16T16:15:35.838650+0000","last_clean":"2024-04-16T16:15:35.838650+0000","last_became_active":"2024-04-16T16:15:35.833902+0000","last_became_peered":"2024-04-16T16:15:35.833902+0000","last_unstale":"2024-04-16T16:15:35.838650+0000","last_undegraded":"2024-04-16T16:15:35.838650+0000","last_fullsized":"2024-04-16T16:15:35.838650+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:04.283280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.814049+0000","last_change":"2024-04-16T16:15:14.424278+0000","last_active":"2024-04-16T16:15:35.814049+0000","last_peered":"2024-04-16T16:15:35.814049+0000","last_clean":"2024-04-16T16:15:35.814049+0000","last_became_active":"2024-04-16T16:15:14.423174+0000","last_became_peered":"2024-04-16T16:15:14.423174+0000","last_unstale":"2024-04-16T16:15:35.814049+0000","last_undegraded":"2024-04-16T16:15:35.814049+0000","last_fullsized":"2024-04-16T16:15:35.814049+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:19:05.074032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832250+0000","last_change":"2024-04-16T16:15:15.449317+0000","last_active":"2024-04-16T16:15:36.832250+0000","last_peered":"2024-04-16T16:15:36.832250+0000","last_clean":"2024-04-16T16:15:36.832250+0000","last_became_active":"2024-04-16T16:15:15.448646+0000","last_became_peered":"2024-04-16T16:15:15.448646+0000","last_unstale":"2024-04-16T16:15:36.832250+0000","last_undegraded":"2024-04-16T16:15:36.832250+0000","last_fullsized":"2024-04-16T16:15:36.832250+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:11.662435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.807980+0000","last_change":"2024-04-16T16:15:34.815648+0000","last_active":"2024-04-16T16:15:35.807980+0000","last_peered":"2024-04-16T16:15:35.807980+0000","last_clean":"2024-04-16T16:15:35.807980+0000","last_became_active":"2024-04-16T16:15:34.815334+0000","last_became_peered":"2024-04-16T16:15:34.815334+0000","last_unstale":"2024-04-16T16:15:35.807980+0000","last_undegraded":"2024-04-16T16:15:35.807980+0000","last_fullsized":"2024-04-16T16:15:35.807980+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:10.753869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.841466+0000","last_change":"2024-04-16T16:15:35.833940+0000","last_active":"2024-04-16T16:15:35.841466+0000","last_peered":"2024-04-16T16:15:35.841466+0000","last_clean":"2024-04-16T16:15:35.841466+0000","last_became_active":"2024-04-16T16:15:35.833636+0000","last_became_peered":"2024-04-16T16:15:35.833636+0000","last_unstale":"2024-04-16T16:15:35.841466+0000","last_undegraded":"2024-04-16T16:15:35.841466+0000","last_fullsized":"2024-04-16T16:15:35.841466+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:21:53.238713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":30,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832426+0000","last_change":"2024-04-16T16:15:14.426984+0000","last_active":"2024-04-16T16:15:36.832426+0000","last_peered":"2024-04-16T16:15:36.832426+0000","last_clean":"2024-04-16T16:15:36.832426+0000","last_became_active":"2024-04-16T16:15:14.426689+0000","last_became_peered":"2024-04-16T16:15:14.426689+0000","last_unstale":"2024-04-16T16:15:36.832426+0000","last_undegraded":"2024-04-16T16:15:36.832426+0000","last_fullsized":"2024-04-16T16:15:36.832426+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:02.150053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.813125+0000","last_change":"2024-04-16T16:15:15.451816+0000","last_active":"2024-04-16T16:15:35.813125+0000","last_peered":"2024-04-16T16:15:35.813125+0000","last_clean":"2024-04-16T16:15:35.813125+0000","last_became_active":"2024-04-16T16:15:15.450017+0000","last_became_peered":"2024-04-16T16:15:15.450017+0000","last_unstale":"2024-04-16T16:15:35.813125+0000","last_undegraded":"2024-04-16T16:15:35.813125+0000","last_fullsized":"2024-04-16T16:15:35.813125+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:42.434776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830471+0000","last_change":"2024-04-16T16:15:34.832134+0000","last_active":"2024-04-16T16:15:36.830471+0000","last_peered":"2024-04-16T16:15:36.830471+0000","last_clean":"2024-04-16T16:15:36.830471+0000","last_became_active":"2024-04-16T16:15:34.831845+0000","last_became_peered":"2024-04-16T16:15:34.831845+0000","last_unstale":"2024-04-16T16:15:36.830471+0000","last_undegraded":"2024-04-16T16:15:36.830471+0000","last_fullsized":"2024-04-16T16:15:36.830471+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:13:15.224743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.837320+0000","last_change":"2024-04-16T16:15:35.831430+0000","last_active":"2024-04-16T16:15:35.837320+0000","last_peered":"2024-04-16T16:15:35.837320+0000","last_clean":"2024-04-16T16:15:35.837320+0000","last_became_active":"2024-04-16T16:15:35.831164+0000","last_became_peered":"2024-04-16T16:15:35.831164+0000","last_unstale":"2024-04-16T16:15:35.837320+0000","last_undegraded":"2024-04-16T16:15:35.837320+0000","last_fullsized":"2024-04-16T16:15:35.837320+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:42:41.959591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"46'2","reported_seq":30,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831015+0000","last_change":"2024-04-16T16:15:14.424803+0000","last_active":"2024-04-16T16:15:36.831015+0000","last_peered":"2024-04-16T16:15:36.831015+0000","last_clean":"2024-04-16T16:15:36.831015+0000","last_became_active":"2024-04-16T16:15:14.424493+0000","last_became_peered":"2024-04-16T16:15:14.424493+0000","last_unstale":"2024-04-16T16:15:36.831015+0000","last_undegraded":"2024-04-16T16:15:36.831015+0000","last_fullsized":"2024-04-16T16:15:36.831015+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:33.299058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"46'1","reported_seq":25,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818174+0000","last_change":"2024-04-16T16:15:15.463217+0000","last_active":"2024-04-16T16:15:35.818174+0000","last_peered":"2024-04-16T16:15:35.818174+0000","last_clean":"2024-04-16T16:15:35.818174+0000","last_became_active":"2024-04-16T16:15:15.462748+0000","last_became_peered":"2024-04-16T16:15:15.462748+0000","last_unstale":"2024-04-16T16:15:35.818174+0000","last_undegraded":"2024-04-16T16:15:35.818174+0000","last_fullsized":"2024-04-16T16:15:35.818174+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:03.163739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097152,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.817600+0000","last_change":"2024-04-16T16:15:34.825031+0000","last_active":"2024-04-16T16:15:35.817600+0000","last_peered":"2024-04-16T16:15:35.817600+0000","last_clean":"2024-04-16T16:15:35.817600+0000","last_became_active":"2024-04-16T16:15:34.824722+0000","last_became_peered":"2024-04-16T16:15:34.824722+0000","last_unstale":"2024-04-16T16:15:35.817600+0000","last_undegraded":"2024-04-16T16:15:35.817600+0000","last_fullsized":"2024-04-16T16:15:35.817600+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:49:47.556974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.841872+0000","last_change":"2024-04-16T16:15:35.836648+0000","last_active":"2024-04-16T16:15:35.841872+0000","last_peered":"2024-04-16T16:15:35.841872+0000","last_clean":"2024-04-16T16:15:35.841872+0000","last_became_active":"2024-04-16T16:15:35.836312+0000","last_became_peered":"2024-04-16T16:15:35.836312+0000","last_unstale":"2024-04-16T16:15:35.841872+0000","last_undegraded":"2024-04-16T16:15:35.841872+0000","last_fullsized":"2024-04-16T16:15:35.841872+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:02:19.857827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":30,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829661+0000","last_change":"2024-04-16T16:15:14.419487+0000","last_active":"2024-04-16T16:15:36.829661+0000","last_peered":"2024-04-16T16:15:36.829661+0000","last_clean":"2024-04-16T16:15:36.829661+0000","last_became_active":"2024-04-16T16:15:14.419154+0000","last_became_peered":"2024-04-16T16:15:14.419154+0000","last_unstale":"2024-04-16T16:15:36.829661+0000","last_undegraded":"2024-04-16T16:15:36.829661+0000","last_fullsized":"2024-04-16T16:15:36.829661+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:01:19.702926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.807009+0000","last_change":"2024-04-16T16:15:15.463514+0000","last_active":"2024-04-16T16:15:35.807009+0000","last_peered":"2024-04-16T16:15:35.807009+0000","last_clean":"2024-04-16T16:15:35.807009+0000","last_became_active":"2024-04-16T16:15:15.461956+0000","last_became_peered":"2024-04-16T16:15:15.461956+0000","last_unstale":"2024-04-16T16:15:35.807009+0000","last_undegraded":"2024-04-16T16:15:35.807009+0000","last_fullsized":"2024-04-16T16:15:35.807009+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:34.312744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811542+0000","last_change":"2024-04-16T16:15:34.827098+0000","last_active":"2024-04-16T16:15:35.811542+0000","last_peered":"2024-04-16T16:15:35.811542+0000","last_clean":"2024-04-16T16:15:35.811542+0000","last_became_active":"2024-04-16T16:15:34.826802+0000","last_became_peered":"2024-04-16T16:15:34.826802+0000","last_unstale":"2024-04-16T16:15:35.811542+0000","last_undegraded":"2024-04-16T16:15:35.811542+0000","last_fullsized":"2024-04-16T16:15:35.811542+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:06:11.243617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.841997+0000","last_change":"2024-04-16T16:15:35.836801+0000","last_active":"2024-04-16T16:15:35.841997+0000","last_peered":"2024-04-16T16:15:35.841997+0000","last_clean":"2024-04-16T16:15:35.841997+0000","last_became_active":"2024-04-16T16:15:35.836175+0000","last_became_peered":"2024-04-16T16:15:35.836175+0000","last_unstale":"2024-04-16T16:15:35.841997+0000","last_undegraded":"2024-04-16T16:15:35.841997+0000","last_fullsized":"2024-04-16T16:15:35.841997+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:15:06.383899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.807505+0000","last_change":"2024-04-16T16:15:14.426381+0000","last_active":"2024-04-16T16:15:35.807505+0000","last_peered":"2024-04-16T16:15:35.807505+0000","last_clean":"2024-04-16T16:15:35.807505+0000","last_became_active":"2024-04-16T16:15:14.425924+0000","last_became_peered":"2024-04-16T16:15:14.425924+0000","last_unstale":"2024-04-16T16:15:35.807505+0000","last_undegraded":"2024-04-16T16:15:35.807505+0000","last_fullsized":"2024-04-16T16:15:35.807505+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:07:59.840158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.820268+0000","last_change":"2024-04-16T16:15:15.440012+0000","last_active":"2024-04-16T16:15:35.820268+0000","last_peered":"2024-04-16T16:15:35.820268+0000","last_clean":"2024-04-16T16:15:35.820268+0000","last_became_active":"2024-04-16T16:15:15.439703+0000","last_became_peered":"2024-04-16T16:15:15.439703+0000","last_unstale":"2024-04-16T16:15:35.820268+0000","last_undegraded":"2024-04-16T16:15:35.820268+0000","last_fullsized":"2024-04-16T16:15:35.820268+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:10:07.586480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"49'2","reported_seq":14,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.852320+0000","last_change":"2024-04-16T16:15:34.816832+0000","last_active":"2024-04-16T16:15:35.852320+0000","last_peered":"2024-04-16T16:15:35.852320+0000","last_clean":"2024-04-16T16:15:35.852320+0000","last_became_active":"2024-04-16T16:15:34.816528+0000","last_became_peered":"2024-04-16T16:15:34.816528+0000","last_unstale":"2024-04-16T16:15:35.852320+0000","last_undegraded":"2024-04-16T16:15:35.852320+0000","last_fullsized":"2024-04-16T16:15:35.852320+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:54.087009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.830067+0000","last_change":"2024-04-16T16:15:35.826463+0000","last_active":"2024-04-16T16:15:35.830067+0000","last_peered":"2024-04-16T16:15:35.830067+0000","last_clean":"2024-04-16T16:15:35.830067+0000","last_became_active":"2024-04-16T16:15:35.826113+0000","last_became_peered":"2024-04-16T16:15:35.826113+0000","last_unstale":"2024-04-16T16:15:35.830067+0000","last_undegraded":"2024-04-16T16:15:35.830067+0000","last_fullsized":"2024-04-16T16:15:35.830067+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:58.338850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":27,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.817771+0000","last_change":"2024-04-16T16:15:14.423634+0000","last_active":"2024-04-16T16:15:35.817771+0000","last_peered":"2024-04-16T16:15:35.817771+0000","last_clean":"2024-04-16T16:15:35.817771+0000","last_became_active":"2024-04-16T16:15:14.422076+0000","last_became_peered":"2024-04-16T16:15:14.422076+0000","last_unstale":"2024-04-16T16:15:35.817771+0000","last_undegraded":"2024-04-16T16:15:35.817771+0000","last_fullsized":"2024-04-16T16:15:35.817771+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:43.504451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.812089+0000","last_change":"2024-04-16T16:15:15.450303+0000","last_active":"2024-04-16T16:15:35.812089+0000","last_peered":"2024-04-16T16:15:35.812089+0000","last_clean":"2024-04-16T16:15:35.812089+0000","last_became_active":"2024-04-16T16:15:15.449983+0000","last_became_peered":"2024-04-16T16:15:15.449983+0000","last_unstale":"2024-04-16T16:15:35.812089+0000","last_undegraded":"2024-04-16T16:15:35.812089+0000","last_fullsized":"2024-04-16T16:15:35.812089+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:46.819298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.821918+0000","last_change":"2024-04-16T16:15:34.834124+0000","last_active":"2024-04-16T16:15:35.821918+0000","last_peered":"2024-04-16T16:15:35.821918+0000","last_clean":"2024-04-16T16:15:35.821918+0000","last_became_active":"2024-04-16T16:15:34.833822+0000","last_became_peered":"2024-04-16T16:15:34.833822+0000","last_unstale":"2024-04-16T16:15:35.821918+0000","last_undegraded":"2024-04-16T16:15:35.821918+0000","last_fullsized":"2024-04-16T16:15:35.821918+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:49:30.804849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.840110+0000","last_change":"2024-04-16T16:15:35.832226+0000","last_active":"2024-04-16T16:15:35.840110+0000","last_peered":"2024-04-16T16:15:35.840110+0000","last_clean":"2024-04-16T16:15:35.840110+0000","last_became_active":"2024-04-16T16:15:35.831978+0000","last_became_peered":"2024-04-16T16:15:35.831978+0000","last_unstale":"2024-04-16T16:15:35.840110+0000","last_undegraded":"2024-04-16T16:15:35.840110+0000","last_fullsized":"2024-04-16T16:15:35.840110+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:05.371730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"43'3","reported_seq":32,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830727+0000","last_change":"2024-04-16T16:15:14.419251+0000","last_active":"2024-04-16T16:15:36.830727+0000","last_peered":"2024-04-16T16:15:36.830727+0000","last_clean":"2024-04-16T16:15:36.830727+0000","last_became_active":"2024-04-16T16:15:14.418965+0000","last_became_peered":"2024-04-16T16:15:14.418965+0000","last_unstale":"2024-04-16T16:15:36.830727+0000","last_undegraded":"2024-04-16T16:15:36.830727+0000","last_fullsized":"2024-04-16T16:15:36.830727+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:11.262725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830376+0000","last_change":"2024-04-16T16:15:15.443802+0000","last_active":"2024-04-16T16:15:36.830376+0000","last_peered":"2024-04-16T16:15:36.830376+0000","last_clean":"2024-04-16T16:15:36.830376+0000","last_became_active":"2024-04-16T16:15:15.443493+0000","last_became_peered":"2024-04-16T16:15:15.443493+0000","last_unstale":"2024-04-16T16:15:36.830376+0000","last_undegraded":"2024-04-16T16:15:36.830376+0000","last_fullsized":"2024-04-16T16:15:36.830376+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:31:00.851752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818139+0000","last_change":"2024-04-16T16:15:34.832847+0000","last_active":"2024-04-16T16:15:35.818139+0000","last_peered":"2024-04-16T16:15:35.818139+0000","last_clean":"2024-04-16T16:15:35.818139+0000","last_became_active":"2024-04-16T16:15:34.832270+0000","last_became_peered":"2024-04-16T16:15:34.832270+0000","last_unstale":"2024-04-16T16:15:35.818139+0000","last_undegraded":"2024-04-16T16:15:35.818139+0000","last_fullsized":"2024-04-16T16:15:35.818139+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:10.753869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830668+0000","last_change":"2024-04-16T16:15:35.830104+0000","last_active":"2024-04-16T16:15:36.830668+0000","last_peered":"2024-04-16T16:15:36.830668+0000","last_clean":"2024-04-16T16:15:36.830668+0000","last_became_active":"2024-04-16T16:15:35.829832+0000","last_became_peered":"2024-04-16T16:15:35.829832+0000","last_unstale":"2024-04-16T16:15:36.830668+0000","last_undegraded":"2024-04-16T16:15:36.830668+0000","last_fullsized":"2024-04-16T16:15:36.830668+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:40:24.612282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830098+0000","last_change":"2024-04-16T16:15:14.423513+0000","last_active":"2024-04-16T16:15:36.830098+0000","last_peered":"2024-04-16T16:15:36.830098+0000","last_clean":"2024-04-16T16:15:36.830098+0000","last_became_active":"2024-04-16T16:15:14.423209+0000","last_became_peered":"2024-04-16T16:15:14.423209+0000","last_unstale":"2024-04-16T16:15:36.830098+0000","last_undegraded":"2024-04-16T16:15:36.830098+0000","last_fullsized":"2024-04-16T16:15:36.830098+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:41.421090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818323+0000","last_change":"2024-04-16T16:15:15.451372+0000","last_active":"2024-04-16T16:15:35.818323+0000","last_peered":"2024-04-16T16:15:35.818323+0000","last_clean":"2024-04-16T16:15:35.818323+0000","last_became_active":"2024-04-16T16:15:15.450983+0000","last_became_peered":"2024-04-16T16:15:15.450983+0000","last_unstale":"2024-04-16T16:15:35.818323+0000","last_undegraded":"2024-04-16T16:15:35.818323+0000","last_fullsized":"2024-04-16T16:15:35.818323+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:05:00.389608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831791+0000","last_change":"2024-04-16T16:15:34.832040+0000","last_active":"2024-04-16T16:15:36.831791+0000","last_peered":"2024-04-16T16:15:36.831791+0000","last_clean":"2024-04-16T16:15:36.831791+0000","last_became_active":"2024-04-16T16:15:34.831671+0000","last_became_peered":"2024-04-16T16:15:34.831671+0000","last_unstale":"2024-04-16T16:15:36.831791+0000","last_undegraded":"2024-04-16T16:15:36.831791+0000","last_fullsized":"2024-04-16T16:15:36.831791+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:25:57.981780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829161+0000","last_change":"2024-04-16T16:15:35.836601+0000","last_active":"2024-04-16T16:15:36.829161+0000","last_peered":"2024-04-16T16:15:36.829161+0000","last_clean":"2024-04-16T16:15:36.829161+0000","last_became_active":"2024-04-16T16:15:35.834804+0000","last_became_peered":"2024-04-16T16:15:35.834804+0000","last_unstale":"2024-04-16T16:15:36.829161+0000","last_undegraded":"2024-04-16T16:15:36.829161+0000","last_fullsized":"2024-04-16T16:15:36.829161+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:53:28.308092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"41'1","reported_seq":29,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.822546+0000","last_change":"2024-04-16T16:15:14.417893+0000","last_active":"2024-04-16T16:15:35.822546+0000","last_peered":"2024-04-16T16:15:35.822546+0000","last_clean":"2024-04-16T16:15:35.822546+0000","last_became_active":"2024-04-16T16:15:14.416107+0000","last_became_peered":"2024-04-16T16:15:14.416107+0000","last_unstale":"2024-04-16T16:15:35.822546+0000","last_undegraded":"2024-04-16T16:15:35.822546+0000","last_fullsized":"2024-04-16T16:15:35.822546+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:59.375922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.817717+0000","last_change":"2024-04-16T16:15:15.446817+0000","last_active":"2024-04-16T16:15:35.817717+0000","last_peered":"2024-04-16T16:15:35.817717+0000","last_clean":"2024-04-16T16:15:35.817717+0000","last_became_active":"2024-04-16T16:15:15.444908+0000","last_became_peered":"2024-04-16T16:15:15.444908+0000","last_unstale":"2024-04-16T16:15:35.817717+0000","last_undegraded":"2024-04-16T16:15:35.817717+0000","last_fullsized":"2024-04-16T16:15:35.817717+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:46.819298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811893+0000","last_change":"2024-04-16T16:15:34.825582+0000","last_active":"2024-04-16T16:15:35.811893+0000","last_peered":"2024-04-16T16:15:35.811893+0000","last_clean":"2024-04-16T16:15:35.811893+0000","last_became_active":"2024-04-16T16:15:34.824775+0000","last_became_peered":"2024-04-16T16:15:34.824775+0000","last_unstale":"2024-04-16T16:15:35.811893+0000","last_undegraded":"2024-04-16T16:15:35.811893+0000","last_fullsized":"2024-04-16T16:15:35.811893+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:29:06.848116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.828847+0000","last_change":"2024-04-16T16:15:35.841752+0000","last_active":"2024-04-16T16:15:36.828847+0000","last_peered":"2024-04-16T16:15:36.828847+0000","last_clean":"2024-04-16T16:15:36.828847+0000","last_became_active":"2024-04-16T16:15:35.841011+0000","last_became_peered":"2024-04-16T16:15:35.841011+0000","last_unstale":"2024-04-16T16:15:36.828847+0000","last_undegraded":"2024-04-16T16:15:36.828847+0000","last_fullsized":"2024-04-16T16:15:36.828847+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:37:06.311207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"41'2","reported_seq":33,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.833293+0000","last_change":"2024-04-16T16:15:14.420778+0000","last_active":"2024-04-16T16:15:36.833293+0000","last_peered":"2024-04-16T16:15:36.833293+0000","last_clean":"2024-04-16T16:15:36.833293+0000","last_became_active":"2024-04-16T16:15:14.420315+0000","last_became_peered":"2024-04-16T16:15:14.420315+0000","last_unstale":"2024-04-16T16:15:36.833293+0000","last_undegraded":"2024-04-16T16:15:36.833293+0000","last_fullsized":"2024-04-16T16:15:36.833293+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:43:43.352649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811179+0000","last_change":"2024-04-16T16:15:15.442790+0000","last_active":"2024-04-16T16:15:35.811179+0000","last_peered":"2024-04-16T16:15:35.811179+0000","last_clean":"2024-04-16T16:15:35.811179+0000","last_became_active":"2024-04-16T16:15:15.439265+0000","last_became_peered":"2024-04-16T16:15:15.439265+0000","last_unstale":"2024-04-16T16:15:35.811179+0000","last_undegraded":"2024-04-16T16:15:35.811179+0000","last_fullsized":"2024-04-16T16:15:35.811179+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:40:41.651741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.828788+0000","last_change":"2024-04-16T16:15:34.826765+0000","last_active":"2024-04-16T16:15:36.828788+0000","last_peered":"2024-04-16T16:15:36.828788+0000","last_clean":"2024-04-16T16:15:36.828788+0000","last_became_active":"2024-04-16T16:15:34.826268+0000","last_became_peered":"2024-04-16T16:15:34.826268+0000","last_unstale":"2024-04-16T16:15:36.828788+0000","last_undegraded":"2024-04-16T16:15:36.828788+0000","last_fullsized":"2024-04-16T16:15:36.828788+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:52:19.989548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.840976+0000","last_change":"2024-04-16T16:15:35.835346+0000","last_active":"2024-04-16T16:15:35.840976+0000","last_peered":"2024-04-16T16:15:35.840976+0000","last_clean":"2024-04-16T16:15:35.840976+0000","last_became_active":"2024-04-16T16:15:35.834372+0000","last_became_peered":"2024-04-16T16:15:35.834372+0000","last_unstale":"2024-04-16T16:15:35.840976+0000","last_undegraded":"2024-04-16T16:15:35.840976+0000","last_fullsized":"2024-04-16T16:15:35.840976+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:02.048834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"41'1","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.825302+0000","last_change":"2024-04-16T16:15:14.406535+0000","last_active":"2024-04-16T16:15:35.825302+0000","last_peered":"2024-04-16T16:15:35.825302+0000","last_clean":"2024-04-16T16:15:35.825302+0000","last_became_active":"2024-04-16T16:15:14.405872+0000","last_became_peered":"2024-04-16T16:15:14.405872+0000","last_unstale":"2024-04-16T16:15:35.825302+0000","last_undegraded":"2024-04-16T16:15:35.825302+0000","last_fullsized":"2024-04-16T16:15:35.825302+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:12:11.062203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.824925+0000","last_change":"2024-04-16T16:15:15.444883+0000","last_active":"2024-04-16T16:15:35.824925+0000","last_peered":"2024-04-16T16:15:35.824925+0000","last_clean":"2024-04-16T16:15:35.824925+0000","last_became_active":"2024-04-16T16:15:15.438454+0000","last_became_peered":"2024-04-16T16:15:15.438454+0000","last_unstale":"2024-04-16T16:15:35.824925+0000","last_undegraded":"2024-04-16T16:15:35.824925+0000","last_fullsized":"2024-04-16T16:15:35.824925+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:25:00.737475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.807978+0000","last_change":"2024-04-16T16:15:34.815921+0000","last_active":"2024-04-16T16:15:35.807978+0000","last_peered":"2024-04-16T16:15:35.807978+0000","last_clean":"2024-04-16T16:15:35.807978+0000","last_became_active":"2024-04-16T16:15:34.814833+0000","last_became_peered":"2024-04-16T16:15:34.814833+0000","last_unstale":"2024-04-16T16:15:35.807978+0000","last_undegraded":"2024-04-16T16:15:35.807978+0000","last_fullsized":"2024-04-16T16:15:35.807978+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:31.657389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.838181+0000","last_change":"2024-04-16T16:15:35.831248+0000","last_active":"2024-04-16T16:15:35.838181+0000","last_peered":"2024-04-16T16:15:35.838181+0000","last_clean":"2024-04-16T16:15:35.838181+0000","last_became_active":"2024-04-16T16:15:35.830852+0000","last_became_peered":"2024-04-16T16:15:35.830852+0000","last_unstale":"2024-04-16T16:15:35.838181+0000","last_undegraded":"2024-04-16T16:15:35.838181+0000","last_fullsized":"2024-04-16T16:15:35.838181+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:32.059528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830725+0000","last_change":"2024-04-16T16:15:34.824918+0000","last_active":"2024-04-16T16:15:36.830725+0000","last_peered":"2024-04-16T16:15:36.830725+0000","last_clean":"2024-04-16T16:15:36.830725+0000","last_became_active":"2024-04-16T16:15:34.824549+0000","last_became_peered":"2024-04-16T16:15:34.824549+0000","last_unstale":"2024-04-16T16:15:36.830725+0000","last_undegraded":"2024-04-16T16:15:36.830725+0000","last_fullsized":"2024-04-16T16:15:36.830725+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:37:50.587155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"41'8","reported_seq":42,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.805552+0000","last_change":"2024-04-16T16:15:14.421997+0000","last_active":"2024-04-16T16:15:35.805552+0000","last_peered":"2024-04-16T16:15:35.805552+0000","last_clean":"2024-04-16T16:15:35.805552+0000","last_became_active":"2024-04-16T16:15:14.421747+0000","last_became_peered":"2024-04-16T16:15:14.421747+0000","last_unstale":"2024-04-16T16:15:35.805552+0000","last_undegraded":"2024-04-16T16:15:35.805552+0000","last_fullsized":"2024-04-16T16:15:35.805552+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:56:30.781319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829916+0000","last_change":"2024-04-16T16:15:15.446100+0000","last_active":"2024-04-16T16:15:36.829916+0000","last_peered":"2024-04-16T16:15:36.829916+0000","last_clean":"2024-04-16T16:15:36.829916+0000","last_became_active":"2024-04-16T16:15:15.444480+0000","last_became_peered":"2024-04-16T16:15:15.444480+0000","last_unstale":"2024-04-16T16:15:36.829916+0000","last_undegraded":"2024-04-16T16:15:36.829916+0000","last_fullsized":"2024-04-16T16:15:36.829916+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:47.467138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.840798+0000","last_change":"2024-04-16T16:15:35.834966+0000","last_active":"2024-04-16T16:15:35.840798+0000","last_peered":"2024-04-16T16:15:35.840798+0000","last_clean":"2024-04-16T16:15:35.840798+0000","last_became_active":"2024-04-16T16:15:35.830812+0000","last_became_peered":"2024-04-16T16:15:35.830812+0000","last_unstale":"2024-04-16T16:15:35.840798+0000","last_undegraded":"2024-04-16T16:15:35.840798+0000","last_fullsized":"2024-04-16T16:15:35.840798+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:16:18.955453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"49'4","reported_seq":16,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.849316+0000","last_change":"2024-04-16T16:15:34.826238+0000","last_active":"2024-04-16T16:15:35.849316+0000","last_peered":"2024-04-16T16:15:35.849316+0000","last_clean":"2024-04-16T16:15:35.849316+0000","last_became_active":"2024-04-16T16:15:34.825558+0000","last_became_peered":"2024-04-16T16:15:34.825558+0000","last_unstale":"2024-04-16T16:15:35.849316+0000","last_undegraded":"2024-04-16T16:15:35.849316+0000","last_fullsized":"2024-04-16T16:15:35.849316+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:49:30.804849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829794+0000","last_change":"2024-04-16T16:15:14.410824+0000","last_active":"2024-04-16T16:15:36.829794+0000","last_peered":"2024-04-16T16:15:36.829794+0000","last_clean":"2024-04-16T16:15:36.829794+0000","last_became_active":"2024-04-16T16:15:14.409874+0000","last_became_peered":"2024-04-16T16:15:14.409874+0000","last_unstale":"2024-04-16T16:15:36.829794+0000","last_undegraded":"2024-04-16T16:15:36.829794+0000","last_fullsized":"2024-04-16T16:15:36.829794+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:26:59.442481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829182+0000","last_change":"2024-04-16T16:15:15.455736+0000","last_active":"2024-04-16T16:15:36.829182+0000","last_peered":"2024-04-16T16:15:36.829182+0000","last_clean":"2024-04-16T16:15:36.829182+0000","last_became_active":"2024-04-16T16:15:15.455415+0000","last_became_peered":"2024-04-16T16:15:15.455415+0000","last_unstale":"2024-04-16T16:15:36.829182+0000","last_undegraded":"2024-04-16T16:15:36.829182+0000","last_fullsized":"2024-04-16T16:15:36.829182+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:29:30.311490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.847967+0000","last_change":"2024-04-16T16:15:35.842047+0000","last_active":"2024-04-16T16:15:35.847967+0000","last_peered":"2024-04-16T16:15:35.847967+0000","last_clean":"2024-04-16T16:15:35.847967+0000","last_became_active":"2024-04-16T16:15:35.836938+0000","last_became_peered":"2024-04-16T16:15:35.836938+0000","last_unstale":"2024-04-16T16:15:35.847967+0000","last_undegraded":"2024-04-16T16:15:35.847967+0000","last_fullsized":"2024-04-16T16:15:35.847967+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:13:16.240858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":30,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830345+0000","last_change":"2024-04-16T16:15:14.421692+0000","last_active":"2024-04-16T16:15:36.830345+0000","last_peered":"2024-04-16T16:15:36.830345+0000","last_clean":"2024-04-16T16:15:36.830345+0000","last_became_active":"2024-04-16T16:15:14.421361+0000","last_became_peered":"2024-04-16T16:15:14.421361+0000","last_unstale":"2024-04-16T16:15:36.830345+0000","last_undegraded":"2024-04-16T16:15:36.830345+0000","last_fullsized":"2024-04-16T16:15:36.830345+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:56.611238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.812609+0000","last_change":"2024-04-16T16:15:15.462282+0000","last_active":"2024-04-16T16:15:35.812609+0000","last_peered":"2024-04-16T16:15:35.812609+0000","last_clean":"2024-04-16T16:15:35.812609+0000","last_became_active":"2024-04-16T16:15:15.461980+0000","last_became_peered":"2024-04-16T16:15:15.461980+0000","last_unstale":"2024-04-16T16:15:35.812609+0000","last_undegraded":"2024-04-16T16:15:35.812609+0000","last_fullsized":"2024-04-16T16:15:35.812609+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:11.662435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"49'2","reported_seq":14,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.840601+0000","last_change":"2024-04-16T16:15:34.834352+0000","last_active":"2024-04-16T16:15:35.840601+0000","last_peered":"2024-04-16T16:15:35.840601+0000","last_clean":"2024-04-16T16:15:35.840601+0000","last_became_active":"2024-04-16T16:15:34.834044+0000","last_became_peered":"2024-04-16T16:15:34.834044+0000","last_unstale":"2024-04-16T16:15:35.840601+0000","last_undegraded":"2024-04-16T16:15:35.840601+0000","last_fullsized":"2024-04-16T16:15:35.840601+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:23:32.286883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.830576+0000","last_change":"2024-04-16T16:15:35.827425+0000","last_active":"2024-04-16T16:15:35.830576+0000","last_peered":"2024-04-16T16:15:35.830576+0000","last_clean":"2024-04-16T16:15:35.830576+0000","last_became_active":"2024-04-16T16:15:35.827140+0000","last_became_peered":"2024-04-16T16:15:35.827140+0000","last_unstale":"2024-04-16T16:15:35.830576+0000","last_undegraded":"2024-04-16T16:15:35.830576+0000","last_fullsized":"2024-04-16T16:15:35.830576+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:02:19.857827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"41'2","reported_seq":32,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830557+0000","last_change":"2024-04-16T16:15:14.408195+0000","last_active":"2024-04-16T16:15:36.830557+0000","last_peered":"2024-04-16T16:15:36.830557+0000","last_clean":"2024-04-16T16:15:36.830557+0000","last_became_active":"2024-04-16T16:15:14.407584+0000","last_became_peered":"2024-04-16T16:15:14.407584+0000","last_unstale":"2024-04-16T16:15:36.830557+0000","last_undegraded":"2024-04-16T16:15:36.830557+0000","last_fullsized":"2024-04-16T16:15:36.830557+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:39:56.984084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.817189+0000","last_change":"2024-04-16T16:15:15.447435+0000","last_active":"2024-04-16T16:15:35.817189+0000","last_peered":"2024-04-16T16:15:35.817189+0000","last_clean":"2024-04-16T16:15:35.817189+0000","last_became_active":"2024-04-16T16:15:15.447094+0000","last_became_peered":"2024-04-16T16:15:15.447094+0000","last_unstale":"2024-04-16T16:15:35.817189+0000","last_undegraded":"2024-04-16T16:15:35.817189+0000","last_fullsized":"2024-04-16T16:15:35.817189+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:16:44.917291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1d","version":"49'2","reported_seq":14,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.847919+0000","last_change":"2024-04-16T16:15:34.816070+0000","last_active":"2024-04-16T16:15:35.847919+0000","last_peered":"2024-04-16T16:15:35.847919+0000","last_clean":"2024-04-16T16:15:35.847919+0000","last_became_active":"2024-04-16T16:15:34.815753+0000","last_became_peered":"2024-04-16T16:15:34.815753+0000","last_unstale":"2024-04-16T16:15:35.847919+0000","last_undegraded":"2024-04-16T16:15:35.847919+0000","last_fullsized":"2024-04-16T16:15:35.847919+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:35:37.144432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829807+0000","last_change":"2024-04-16T16:15:35.850306+0000","last_active":"2024-04-16T16:15:36.829807+0000","last_peered":"2024-04-16T16:15:36.829807+0000","last_clean":"2024-04-16T16:15:36.829807+0000","last_became_active":"2024-04-16T16:15:35.849296+0000","last_became_peered":"2024-04-16T16:15:35.849296+0000","last_unstale":"2024-04-16T16:15:36.829807+0000","last_undegraded":"2024-04-16T16:15:36.829807+0000","last_fullsized":"2024-04-16T16:15:36.829807+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:06:37.061403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"46'12","reported_seq":46,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.825898+0000","last_change":"2024-04-16T16:15:14.419221+0000","last_active":"2024-04-16T16:15:35.825898+0000","last_peered":"2024-04-16T16:15:35.825898+0000","last_clean":"2024-04-16T16:15:35.825898+0000","last_became_active":"2024-04-16T16:15:14.418573+0000","last_became_peered":"2024-04-16T16:15:14.418573+0000","last_unstale":"2024-04-16T16:15:35.825898+0000","last_undegraded":"2024-04-16T16:15:35.825898+0000","last_fullsized":"2024-04-16T16:15:35.825898+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:08.107514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":11,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832100+0000","last_change":"2024-04-16T16:15:15.435454+0000","last_active":"2024-04-16T16:15:36.832100+0000","last_peered":"2024-04-16T16:15:36.832100+0000","last_clean":"2024-04-16T16:15:36.832100+0000","last_became_active":"2024-04-16T16:15:15.435056+0000","last_became_peered":"2024-04-16T16:15:15.435056+0000","last_unstale":"2024-04-16T16:15:36.832100+0000","last_undegraded":"2024-04-16T16:15:36.832100+0000","last_fullsized":"2024-04-16T16:15:36.832100+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:34.706031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1a","version":"49'1","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.845431+0000","last_change":"2024-04-16T16:15:34.827734+0000","last_active":"2024-04-16T16:15:35.845431+0000","last_peered":"2024-04-16T16:15:35.845431+0000","last_clean":"2024-04-16T16:15:35.845431+0000","last_became_active":"2024-04-16T16:15:34.827375+0000","last_became_peered":"2024-04-16T16:15:34.827375+0000","last_unstale":"2024-04-16T16:15:35.845431+0000","last_undegraded":"2024-04-16T16:15:35.845431+0000","last_fullsized":"2024-04-16T16:15:35.845431+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:49:22.456907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.828851+0000","last_change":"2024-04-16T16:15:35.850483+0000","last_active":"2024-04-16T16:15:36.828851+0000","last_peered":"2024-04-16T16:15:36.828851+0000","last_clean":"2024-04-16T16:15:36.828851+0000","last_became_active":"2024-04-16T16:15:35.845409+0000","last_became_peered":"2024-04-16T16:15:35.845409+0000","last_unstale":"2024-04-16T16:15:36.828851+0000","last_undegraded":"2024-04-16T16:15:36.828851+0000","last_fullsized":"2024-04-16T16:15:36.828851+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:06:43.224387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"41'3","reported_seq":36,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832278+0000","last_change":"2024-04-16T16:15:14.427833+0000","last_active":"2024-04-16T16:15:36.832278+0000","last_peered":"2024-04-16T16:15:36.832278+0000","last_clean":"2024-04-16T16:15:36.832278+0000","last_became_active":"2024-04-16T16:15:14.427492+0000","last_became_peered":"2024-04-16T16:15:14.427492+0000","last_unstale":"2024-04-16T16:15:36.832278+0000","last_undegraded":"2024-04-16T16:15:36.832278+0000","last_fullsized":"2024-04-16T16:15:36.832278+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:48.640698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829941+0000","last_change":"2024-04-16T16:15:15.453206+0000","last_active":"2024-04-16T16:15:36.829941+0000","last_peered":"2024-04-16T16:15:36.829941+0000","last_clean":"2024-04-16T16:15:36.829941+0000","last_became_active":"2024-04-16T16:15:15.453002+0000","last_became_peered":"2024-04-16T16:15:15.453002+0000","last_unstale":"2024-04-16T16:15:36.829941+0000","last_undegraded":"2024-04-16T16:15:36.829941+0000","last_fullsized":"2024-04-16T16:15:36.829941+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:37.628550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832456+0000","last_change":"2024-04-16T16:15:34.832901+0000","last_active":"2024-04-16T16:15:36.832456+0000","last_peered":"2024-04-16T16:15:36.832456+0000","last_clean":"2024-04-16T16:15:36.832456+0000","last_became_active":"2024-04-16T16:15:34.832392+0000","last_became_peered":"2024-04-16T16:15:34.832392+0000","last_unstale":"2024-04-16T16:15:36.832456+0000","last_undegraded":"2024-04-16T16:15:36.832456+0000","last_fullsized":"2024-04-16T16:15:36.832456+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:14:10.714795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829277+0000","last_change":"2024-04-16T16:15:35.842001+0000","last_active":"2024-04-16T16:15:36.829277+0000","last_peered":"2024-04-16T16:15:36.829277+0000","last_clean":"2024-04-16T16:15:36.829277+0000","last_became_active":"2024-04-16T16:15:35.841419+0000","last_became_peered":"2024-04-16T16:15:35.841419+0000","last_unstale":"2024-04-16T16:15:36.829277+0000","last_undegraded":"2024-04-16T16:15:36.829277+0000","last_fullsized":"2024-04-16T16:15:36.829277+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:12:17.054235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811731+0000","last_change":"2024-04-16T16:15:14.427378+0000","last_active":"2024-04-16T16:15:35.811731+0000","last_peered":"2024-04-16T16:15:35.811731+0000","last_clean":"2024-04-16T16:15:35.811731+0000","last_became_active":"2024-04-16T16:15:14.427036+0000","last_became_peered":"2024-04-16T16:15:14.427036+0000","last_unstale":"2024-04-16T16:15:35.811731+0000","last_undegraded":"2024-04-16T16:15:35.811731+0000","last_fullsized":"2024-04-16T16:15:35.811731+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:16.172813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831520+0000","last_change":"2024-04-16T16:15:15.436333+0000","last_active":"2024-04-16T16:15:36.831520+0000","last_peered":"2024-04-16T16:15:36.831520+0000","last_clean":"2024-04-16T16:15:36.831520+0000","last_became_active":"2024-04-16T16:15:15.435961+0000","last_became_peered":"2024-04-16T16:15:15.435961+0000","last_unstale":"2024-04-16T16:15:36.831520+0000","last_undegraded":"2024-04-16T16:15:36.831520+0000","last_fullsized":"2024-04-16T16:15:36.831520+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:44.974637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831296+0000","last_change":"2024-04-16T16:15:34.825679+0000","last_active":"2024-04-16T16:15:36.831296+0000","last_peered":"2024-04-16T16:15:36.831296+0000","last_clean":"2024-04-16T16:15:36.831296+0000","last_became_active":"2024-04-16T16:15:34.824988+0000","last_became_peered":"2024-04-16T16:15:34.824988+0000","last_unstale":"2024-04-16T16:15:36.831296+0000","last_undegraded":"2024-04-16T16:15:36.831296+0000","last_fullsized":"2024-04-16T16:15:36.831296+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:34:03.842188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.839887+0000","last_change":"2024-04-16T16:15:35.834848+0000","last_active":"2024-04-16T16:15:35.839887+0000","last_peered":"2024-04-16T16:15:35.839887+0000","last_clean":"2024-04-16T16:15:35.839887+0000","last_became_active":"2024-04-16T16:15:35.833387+0000","last_became_peered":"2024-04-16T16:15:35.833387+0000","last_unstale":"2024-04-16T16:15:35.839887+0000","last_undegraded":"2024-04-16T16:15:35.839887+0000","last_fullsized":"2024-04-16T16:15:35.839887+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:05.371730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":29,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.833213+0000","last_change":"2024-04-16T16:15:14.422188+0000","last_active":"2024-04-16T16:15:36.833213+0000","last_peered":"2024-04-16T16:15:36.833213+0000","last_clean":"2024-04-16T16:15:36.833213+0000","last_became_active":"2024-04-16T16:15:14.421842+0000","last_became_peered":"2024-04-16T16:15:14.421842+0000","last_unstale":"2024-04-16T16:15:36.833213+0000","last_undegraded":"2024-04-16T16:15:36.833213+0000","last_fullsized":"2024-04-16T16:15:36.833213+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:43.299902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.833046+0000","last_change":"2024-04-16T16:15:15.451420+0000","last_active":"2024-04-16T16:15:36.833046+0000","last_peered":"2024-04-16T16:15:36.833046+0000","last_clean":"2024-04-16T16:15:36.833046+0000","last_became_active":"2024-04-16T16:15:15.450543+0000","last_became_peered":"2024-04-16T16:15:15.450543+0000","last_unstale":"2024-04-16T16:15:36.833046+0000","last_undegraded":"2024-04-16T16:15:36.833046+0000","last_fullsized":"2024-04-16T16:15:36.833046+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:27:00.456168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.820095+0000","last_change":"2024-04-16T16:15:34.830670+0000","last_active":"2024-04-16T16:15:35.820095+0000","last_peered":"2024-04-16T16:15:35.820095+0000","last_clean":"2024-04-16T16:15:35.820095+0000","last_became_active":"2024-04-16T16:15:34.830389+0000","last_became_peered":"2024-04-16T16:15:34.830389+0000","last_unstale":"2024-04-16T16:15:35.820095+0000","last_undegraded":"2024-04-16T16:15:35.820095+0000","last_fullsized":"2024-04-16T16:15:35.820095+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:57.009529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831368+0000","last_change":"2024-04-16T16:15:35.829605+0000","last_active":"2024-04-16T16:15:36.831368+0000","last_peered":"2024-04-16T16:15:36.831368+0000","last_clean":"2024-04-16T16:15:36.831368+0000","last_became_active":"2024-04-16T16:15:35.828966+0000","last_became_peered":"2024-04-16T16:15:35.828966+0000","last_unstale":"2024-04-16T16:15:36.831368+0000","last_undegraded":"2024-04-16T16:15:36.831368+0000","last_fullsized":"2024-04-16T16:15:36.831368+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:43.515569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.805163+0000","last_change":"2024-04-16T16:15:14.411837+0000","last_active":"2024-04-16T16:15:35.805163+0000","last_peered":"2024-04-16T16:15:35.805163+0000","last_clean":"2024-04-16T16:15:35.805163+0000","last_became_active":"2024-04-16T16:15:14.411512+0000","last_became_peered":"2024-04-16T16:15:14.411512+0000","last_unstale":"2024-04-16T16:15:35.805163+0000","last_undegraded":"2024-04-16T16:15:35.805163+0000","last_fullsized":"2024-04-16T16:15:35.805163+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:28:19.722523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.816715+0000","last_change":"2024-04-16T16:15:15.438978+0000","last_active":"2024-04-16T16:15:35.816715+0000","last_peered":"2024-04-16T16:15:35.816715+0000","last_clean":"2024-04-16T16:15:35.816715+0000","last_became_active":"2024-04-16T16:15:15.438597+0000","last_became_peered":"2024-04-16T16:15:15.438597+0000","last_unstale":"2024-04-16T16:15:35.816715+0000","last_undegraded":"2024-04-16T16:15:35.816715+0000","last_fullsized":"2024-04-16T16:15:35.816715+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:19:06.087718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830878+0000","last_change":"2024-04-16T16:15:34.819768+0000","last_active":"2024-04-16T16:15:36.830878+0000","last_peered":"2024-04-16T16:15:36.830878+0000","last_clean":"2024-04-16T16:15:36.830878+0000","last_became_active":"2024-04-16T16:15:34.819470+0000","last_became_peered":"2024-04-16T16:15:34.819470+0000","last_unstale":"2024-04-16T16:15:36.830878+0000","last_undegraded":"2024-04-16T16:15:36.830878+0000","last_fullsized":"2024-04-16T16:15:36.830878+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:58:01.343104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832948+0000","last_change":"2024-04-16T16:15:35.827678+0000","last_active":"2024-04-16T16:15:36.832948+0000","last_peered":"2024-04-16T16:15:36.832948+0000","last_clean":"2024-04-16T16:15:36.832948+0000","last_became_active":"2024-04-16T16:15:35.827407+0000","last_became_peered":"2024-04-16T16:15:35.827407+0000","last_unstale":"2024-04-16T16:15:36.832948+0000","last_undegraded":"2024-04-16T16:15:36.832948+0000","last_fullsized":"2024-04-16T16:15:36.832948+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:23:33.302998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":27,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.820400+0000","last_change":"2024-04-16T16:15:14.418012+0000","last_active":"2024-04-16T16:15:35.820400+0000","last_peered":"2024-04-16T16:15:35.820400+0000","last_clean":"2024-04-16T16:15:35.820400+0000","last_became_active":"2024-04-16T16:15:14.417689+0000","last_became_peered":"2024-04-16T16:15:14.417689+0000","last_unstale":"2024-04-16T16:15:35.820400+0000","last_undegraded":"2024-04-16T16:15:35.820400+0000","last_fullsized":"2024-04-16T16:15:35.820400+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:33:40.687566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.804895+0000","last_change":"2024-04-16T16:15:15.461073+0000","last_active":"2024-04-16T16:15:35.804895+0000","last_peered":"2024-04-16T16:15:35.804895+0000","last_clean":"2024-04-16T16:15:35.804895+0000","last_became_active":"2024-04-16T16:15:15.459255+0000","last_became_peered":"2024-04-16T16:15:15.459255+0000","last_unstale":"2024-04-16T16:15:35.804895+0000","last_undegraded":"2024-04-16T16:15:35.804895+0000","last_fullsized":"2024-04-16T16:15:35.804895+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:27:00.456168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831188+0000","last_change":"2024-04-16T16:15:34.829788+0000","last_active":"2024-04-16T16:15:36.831188+0000","last_peered":"2024-04-16T16:15:36.831188+0000","last_clean":"2024-04-16T16:15:36.831188+0000","last_became_active":"2024-04-16T16:15:34.829489+0000","last_became_peered":"2024-04-16T16:15:34.829489+0000","last_unstale":"2024-04-16T16:15:36.831188+0000","last_undegraded":"2024-04-16T16:15:36.831188+0000","last_fullsized":"2024-04-16T16:15:36.831188+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:10.996970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832192+0000","last_change":"2024-04-16T16:15:35.830113+0000","last_active":"2024-04-16T16:15:36.832192+0000","last_peered":"2024-04-16T16:15:36.832192+0000","last_clean":"2024-04-16T16:15:36.832192+0000","last_became_active":"2024-04-16T16:15:35.829488+0000","last_became_peered":"2024-04-16T16:15:35.829488+0000","last_unstale":"2024-04-16T16:15:36.832192+0000","last_undegraded":"2024-04-16T16:15:36.832192+0000","last_fullsized":"2024-04-16T16:15:36.832192+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:13.599730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"43'2","reported_seq":31,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832200+0000","last_change":"2024-04-16T16:15:14.429151+0000","last_active":"2024-04-16T16:15:36.832200+0000","last_peered":"2024-04-16T16:15:36.832200+0000","last_clean":"2024-04-16T16:15:36.832200+0000","last_became_active":"2024-04-16T16:15:14.428841+0000","last_became_peered":"2024-04-16T16:15:14.428841+0000","last_unstale":"2024-04-16T16:15:36.832200+0000","last_undegraded":"2024-04-16T16:15:36.832200+0000","last_fullsized":"2024-04-16T16:15:36.832200+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:16.172813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.812373+0000","last_change":"2024-04-16T16:15:15.460542+0000","last_active":"2024-04-16T16:15:35.812373+0000","last_peered":"2024-04-16T16:15:35.812373+0000","last_clean":"2024-04-16T16:15:35.812373+0000","last_became_active":"2024-04-16T16:15:15.459809+0000","last_became_peered":"2024-04-16T16:15:15.459809+0000","last_unstale":"2024-04-16T16:15:35.812373+0000","last_undegraded":"2024-04-16T16:15:35.812373+0000","last_fullsized":"2024-04-16T16:15:35.812373+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:47.467138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"49'6","reported_seq":20,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832462+0000","last_change":"2024-04-16T16:15:34.831724+0000","last_active":"2024-04-16T16:15:36.832462+0000","last_peered":"2024-04-16T16:15:36.832462+0000","last_clean":"2024-04-16T16:15:36.832462+0000","last_became_active":"2024-04-16T16:15:34.831436+0000","last_became_peered":"2024-04-16T16:15:34.831436+0000","last_unstale":"2024-04-16T16:15:36.832462+0000","last_undegraded":"2024-04-16T16:15:36.832462+0000","last_fullsized":"2024-04-16T16:15:36.832462+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:01.032719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.842383+0000","last_change":"2024-04-16T16:15:35.835878+0000","last_active":"2024-04-16T16:15:35.842383+0000","last_peered":"2024-04-16T16:15:35.842383+0000","last_clean":"2024-04-16T16:15:35.842383+0000","last_became_active":"2024-04-16T16:15:35.828058+0000","last_became_peered":"2024-04-16T16:15:35.828058+0000","last_unstale":"2024-04-16T16:15:35.842383+0000","last_undegraded":"2024-04-16T16:15:35.842383+0000","last_fullsized":"2024-04-16T16:15:35.842383+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:14:11.730910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"43'2","reported_seq":30,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830474+0000","last_change":"2024-04-16T16:15:14.419476+0000","last_active":"2024-04-16T16:15:36.830474+0000","last_peered":"2024-04-16T16:15:36.830474+0000","last_clean":"2024-04-16T16:15:36.830474+0000","last_became_active":"2024-04-16T16:15:14.419185+0000","last_became_peered":"2024-04-16T16:15:14.419185+0000","last_unstale":"2024-04-16T16:15:36.830474+0000","last_undegraded":"2024-04-16T16:15:36.830474+0000","last_fullsized":"2024-04-16T16:15:36.830474+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:26.508529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831945+0000","last_change":"2024-04-16T16:15:15.451450+0000","last_active":"2024-04-16T16:15:36.831945+0000","last_peered":"2024-04-16T16:15:36.831945+0000","last_clean":"2024-04-16T16:15:36.831945+0000","last_became_active":"2024-04-16T16:15:15.451103+0000","last_became_peered":"2024-04-16T16:15:15.451103+0000","last_unstale":"2024-04-16T16:15:36.831945+0000","last_undegraded":"2024-04-16T16:15:36.831945+0000","last_fullsized":"2024-04-16T16:15:36.831945+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:49.654384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.15","version":"49'3","reported_seq":17,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832383+0000","last_change":"2024-04-16T16:15:34.827571+0000","last_active":"2024-04-16T16:15:36.832383+0000","last_peered":"2024-04-16T16:15:36.832383+0000","last_clean":"2024-04-16T16:15:36.832383+0000","last_became_active":"2024-04-16T16:15:34.827285+0000","last_became_peered":"2024-04-16T16:15:34.827285+0000","last_unstale":"2024-04-16T16:15:36.832383+0000","last_undegraded":"2024-04-16T16:15:36.832383+0000","last_fullsized":"2024-04-16T16:15:36.832383+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:34:52.315819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830990+0000","last_change":"2024-04-16T16:15:35.832303+0000","last_active":"2024-04-16T16:15:36.830990+0000","last_peered":"2024-04-16T16:15:36.830990+0000","last_clean":"2024-04-16T16:15:36.830990+0000","last_became_active":"2024-04-16T16:15:35.832022+0000","last_became_peered":"2024-04-16T16:15:35.832022+0000","last_unstale":"2024-04-16T16:15:36.830990+0000","last_undegraded":"2024-04-16T16:15:36.830990+0000","last_fullsized":"2024-04-16T16:15:36.830990+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:58:02.359219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1c","version":"41'2","reported_seq":34,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831565+0000","last_change":"2024-04-16T16:15:14.426165+0000","last_active":"2024-04-16T16:15:36.831565+0000","last_peered":"2024-04-16T16:15:36.831565+0000","last_clean":"2024-04-16T16:15:36.831565+0000","last_became_active":"2024-04-16T16:15:14.424933+0000","last_became_peered":"2024-04-16T16:15:14.424933+0000","last_unstale":"2024-04-16T16:15:36.831565+0000","last_undegraded":"2024-04-16T16:15:36.831565+0000","last_fullsized":"2024-04-16T16:15:36.831565+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:59.375922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818560+0000","last_change":"2024-04-16T16:15:15.455406+0000","last_active":"2024-04-16T16:15:35.818560+0000","last_peered":"2024-04-16T16:15:35.818560+0000","last_clean":"2024-04-16T16:15:35.818560+0000","last_became_active":"2024-04-16T16:15:15.455065+0000","last_became_peered":"2024-04-16T16:15:15.455065+0000","last_unstale":"2024-04-16T16:15:35.818560+0000","last_undegraded":"2024-04-16T16:15:35.818560+0000","last_fullsized":"2024-04-16T16:15:35.818560+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:48:05.645437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.817080+0000","last_change":"2024-04-16T16:15:34.824758+0000","last_active":"2024-04-16T16:15:35.817080+0000","last_peered":"2024-04-16T16:15:35.817080+0000","last_clean":"2024-04-16T16:15:35.817080+0000","last_became_active":"2024-04-16T16:15:34.824345+0000","last_became_peered":"2024-04-16T16:15:34.824345+0000","last_unstale":"2024-04-16T16:15:35.817080+0000","last_undegraded":"2024-04-16T16:15:35.817080+0000","last_fullsized":"2024-04-16T16:15:35.817080+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:31.657389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.840644+0000","last_change":"2024-04-16T16:15:35.833575+0000","last_active":"2024-04-16T16:15:35.840644+0000","last_peered":"2024-04-16T16:15:35.840644+0000","last_clean":"2024-04-16T16:15:35.840644+0000","last_became_active":"2024-04-16T16:15:35.833268+0000","last_became_peered":"2024-04-16T16:15:35.833268+0000","last_unstale":"2024-04-16T16:15:35.840644+0000","last_undegraded":"2024-04-16T16:15:35.840644+0000","last_fullsized":"2024-04-16T16:15:35.840644+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:35:38.160546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"41'2","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818688+0000","last_change":"2024-04-16T16:15:14.420969+0000","last_active":"2024-04-16T16:15:35.818688+0000","last_peered":"2024-04-16T16:15:35.818688+0000","last_clean":"2024-04-16T16:15:35.818688+0000","last_became_active":"2024-04-16T16:15:14.420657+0000","last_became_peered":"2024-04-16T16:15:14.420657+0000","last_unstale":"2024-04-16T16:15:35.818688+0000","last_undegraded":"2024-04-16T16:15:35.818688+0000","last_fullsized":"2024-04-16T16:15:35.818688+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:43.299902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":25,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.822149+0000","last_change":"2024-04-16T16:15:15.432880+0000","last_active":"2024-04-16T16:15:35.822149+0000","last_peered":"2024-04-16T16:15:35.822149+0000","last_clean":"2024-04-16T16:15:35.822149+0000","last_became_active":"2024-04-16T16:15:15.432458+0000","last_became_peered":"2024-04-16T16:15:15.432458+0000","last_unstale":"2024-04-16T16:15:35.822149+0000","last_undegraded":"2024-04-16T16:15:35.822149+0000","last_fullsized":"2024-04-16T16:15:35.822149+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:47.467138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829561+0000","last_change":"2024-04-16T16:15:34.817296+0000","last_active":"2024-04-16T16:15:36.829561+0000","last_peered":"2024-04-16T16:15:36.829561+0000","last_clean":"2024-04-16T16:15:36.829561+0000","last_became_active":"2024-04-16T16:15:34.817033+0000","last_became_peered":"2024-04-16T16:15:34.817033+0000","last_unstale":"2024-04-16T16:15:36.829561+0000","last_undegraded":"2024-04-16T16:15:36.829561+0000","last_fullsized":"2024-04-16T16:15:36.829561+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:40:23.596167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829591+0000","last_change":"2024-04-16T16:15:35.824339+0000","last_active":"2024-04-16T16:15:36.829591+0000","last_peered":"2024-04-16T16:15:36.829591+0000","last_clean":"2024-04-16T16:15:36.829591+0000","last_became_active":"2024-04-16T16:15:35.824047+0000","last_became_peered":"2024-04-16T16:15:35.824047+0000","last_unstale":"2024-04-16T16:15:36.829591+0000","last_undegraded":"2024-04-16T16:15:36.829591+0000","last_fullsized":"2024-04-16T16:15:36.829591+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:04.283280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1e","version":"43'2","reported_seq":31,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.833341+0000","last_change":"2024-04-16T16:15:14.430354+0000","last_active":"2024-04-16T16:15:36.833341+0000","last_peered":"2024-04-16T16:15:36.833341+0000","last_clean":"2024-04-16T16:15:36.833341+0000","last_became_active":"2024-04-16T16:15:14.429965+0000","last_became_peered":"2024-04-16T16:15:14.429965+0000","last_unstale":"2024-04-16T16:15:36.833341+0000","last_undegraded":"2024-04-16T16:15:36.833341+0000","last_fullsized":"2024-04-16T16:15:36.833341+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:28:19.722523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818063+0000","last_change":"2024-04-16T16:15:15.449524+0000","last_active":"2024-04-16T16:15:35.818063+0000","last_peered":"2024-04-16T16:15:35.818063+0000","last_clean":"2024-04-16T16:15:35.818063+0000","last_became_active":"2024-04-16T16:15:15.448988+0000","last_became_peered":"2024-04-16T16:15:15.448988+0000","last_unstale":"2024-04-16T16:15:35.818063+0000","last_undegraded":"2024-04-16T16:15:35.818063+0000","last_fullsized":"2024-04-16T16:15:35.818063+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:57.624924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.10","version":"49'8","reported_seq":22,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829433+0000","last_change":"2024-04-16T16:15:34.828466+0000","last_active":"2024-04-16T16:15:36.829433+0000","last_peered":"2024-04-16T16:15:36.829433+0000","last_clean":"2024-04-16T16:15:36.829433+0000","last_became_active":"2024-04-16T16:15:34.826881+0000","last_became_peered":"2024-04-16T16:15:34.826881+0000","last_unstale":"2024-04-16T16:15:36.829433+0000","last_undegraded":"2024-04-16T16:15:36.829433+0000","last_fullsized":"2024-04-16T16:15:36.829433+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:31:20.232731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831366+0000","last_change":"2024-04-16T16:15:35.837395+0000","last_active":"2024-04-16T16:15:36.831366+0000","last_peered":"2024-04-16T16:15:36.831366+0000","last_clean":"2024-04-16T16:15:36.831366+0000","last_became_active":"2024-04-16T16:15:35.837059+0000","last_became_peered":"2024-04-16T16:15:35.837059+0000","last_unstale":"2024-04-16T16:15:36.831366+0000","last_undegraded":"2024-04-16T16:15:36.831366+0000","last_fullsized":"2024-04-16T16:15:36.831366+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:49:23.473022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"43'5","reported_seq":35,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831355+0000","last_change":"2024-04-16T16:15:14.426636+0000","last_active":"2024-04-16T16:15:36.831355+0000","last_peered":"2024-04-16T16:15:36.831355+0000","last_clean":"2024-04-16T16:15:36.831355+0000","last_became_active":"2024-04-16T16:15:14.426319+0000","last_became_peered":"2024-04-16T16:15:14.426319+0000","last_unstale":"2024-04-16T16:15:36.831355+0000","last_undegraded":"2024-04-16T16:15:36.831355+0000","last_fullsized":"2024-04-16T16:15:36.831355+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:55.060644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.821834+0000","last_change":"2024-04-16T16:15:15.444805+0000","last_active":"2024-04-16T16:15:35.821834+0000","last_peered":"2024-04-16T16:15:35.821834+0000","last_clean":"2024-04-16T16:15:35.821834+0000","last_became_active":"2024-04-16T16:15:15.442030+0000","last_became_peered":"2024-04-16T16:15:15.442030+0000","last_unstale":"2024-04-16T16:15:35.821834+0000","last_undegraded":"2024-04-16T16:15:35.821834+0000","last_fullsized":"2024-04-16T16:15:35.821834+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:33:41.701252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.820275+0000","last_change":"2024-04-16T16:15:34.828471+0000","last_active":"2024-04-16T16:15:35.820275+0000","last_peered":"2024-04-16T16:15:35.820275+0000","last_clean":"2024-04-16T16:15:35.820275+0000","last_became_active":"2024-04-16T16:15:34.827085+0000","last_became_peered":"2024-04-16T16:15:34.827085+0000","last_unstale":"2024-04-16T16:15:35.820275+0000","last_undegraded":"2024-04-16T16:15:35.820275+0000","last_fullsized":"2024-04-16T16:15:35.820275+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:49:22.456907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830762+0000","last_change":"2024-04-16T16:15:35.827049+0000","last_active":"2024-04-16T16:15:36.830762+0000","last_peered":"2024-04-16T16:15:36.830762+0000","last_clean":"2024-04-16T16:15:36.830762+0000","last_became_active":"2024-04-16T16:15:35.826687+0000","last_became_peered":"2024-04-16T16:15:35.826687+0000","last_unstale":"2024-04-16T16:15:36.830762+0000","last_undegraded":"2024-04-16T16:15:36.830762+0000","last_fullsized":"2024-04-16T16:15:36.830762+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:13:16.240858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.819707+0000","last_change":"2024-04-16T16:15:15.448903+0000","last_active":"2024-04-16T16:15:35.819707+0000","last_peered":"2024-04-16T16:15:35.819707+0000","last_clean":"2024-04-16T16:15:35.819707+0000","last_became_active":"2024-04-16T16:15:15.448548+0000","last_became_peered":"2024-04-16T16:15:15.448548+0000","last_unstale":"2024-04-16T16:15:35.819707+0000","last_undegraded":"2024-04-16T16:15:35.819707+0000","last_fullsized":"2024-04-16T16:15:35.819707+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:49.734767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831307+0000","last_change":"2024-04-16T16:15:35.841869+0000","last_active":"2024-04-16T16:15:36.831307+0000","last_peered":"2024-04-16T16:15:36.831307+0000","last_clean":"2024-04-16T16:15:36.831307+0000","last_became_active":"2024-04-16T16:15:35.841098+0000","last_became_peered":"2024-04-16T16:15:35.841098+0000","last_unstale":"2024-04-16T16:15:36.831307+0000","last_undegraded":"2024-04-16T16:15:36.831307+0000","last_fullsized":"2024-04-16T16:15:36.831307+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:25:08.011289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829466+0000","last_change":"2024-04-16T16:15:15.434979+0000","last_active":"2024-04-16T16:15:36.829466+0000","last_peered":"2024-04-16T16:15:36.829466+0000","last_clean":"2024-04-16T16:15:36.829466+0000","last_became_active":"2024-04-16T16:15:15.434028+0000","last_became_peered":"2024-04-16T16:15:15.434028+0000","last_unstale":"2024-04-16T16:15:36.829466+0000","last_undegraded":"2024-04-16T16:15:36.829466+0000","last_fullsized":"2024-04-16T16:15:36.829466+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:57:00.347584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829684+0000","last_change":"2024-04-16T16:15:35.850706+0000","last_active":"2024-04-16T16:15:36.829684+0000","last_peered":"2024-04-16T16:15:36.829684+0000","last_clean":"2024-04-16T16:15:36.829684+0000","last_became_active":"2024-04-16T16:15:35.850376+0000","last_became_peered":"2024-04-16T16:15:35.850376+0000","last_unstale":"2024-04-16T16:15:36.829684+0000","last_undegraded":"2024-04-16T16:15:36.829684+0000","last_fullsized":"2024-04-16T16:15:36.829684+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:48.663991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831942+0000","last_change":"2024-04-16T16:15:15.445371+0000","last_active":"2024-04-16T16:15:36.831942+0000","last_peered":"2024-04-16T16:15:36.831942+0000","last_clean":"2024-04-16T16:15:36.831942+0000","last_became_active":"2024-04-16T16:15:15.445063+0000","last_became_peered":"2024-04-16T16:15:15.445063+0000","last_unstale":"2024-04-16T16:15:36.831942+0000","last_undegraded":"2024-04-16T16:15:36.831942+0000","last_fullsized":"2024-04-16T16:15:36.831942+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:57.624924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.848983+0000","last_change":"2024-04-16T16:15:35.846891+0000","last_active":"2024-04-16T16:15:35.848983+0000","last_peered":"2024-04-16T16:15:35.848983+0000","last_clean":"2024-04-16T16:15:35.848983+0000","last_became_active":"2024-04-16T16:15:35.845284+0000","last_became_peered":"2024-04-16T16:15:35.845284+0000","last_unstale":"2024-04-16T16:15:35.848983+0000","last_undegraded":"2024-04-16T16:15:35.848983+0000","last_fullsized":"2024-04-16T16:15:35.848983+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:19:26.484812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818531+0000","last_change":"2024-04-16T16:15:15.445285+0000","last_active":"2024-04-16T16:15:35.818531+0000","last_peered":"2024-04-16T16:15:35.818531+0000","last_clean":"2024-04-16T16:15:35.818531+0000","last_became_active":"2024-04-16T16:15:15.444969+0000","last_became_peered":"2024-04-16T16:15:15.444969+0000","last_unstale":"2024-04-16T16:15:35.818531+0000","last_undegraded":"2024-04-16T16:15:35.818531+0000","last_fullsized":"2024-04-16T16:15:35.818531+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:38:30.263403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830437+0000","last_change":"2024-04-16T16:15:35.842607+0000","last_active":"2024-04-16T16:15:36.830437+0000","last_peered":"2024-04-16T16:15:36.830437+0000","last_clean":"2024-04-16T16:15:36.830437+0000","last_became_active":"2024-04-16T16:15:35.842338+0000","last_became_peered":"2024-04-16T16:15:35.842338+0000","last_unstale":"2024-04-16T16:15:36.830437+0000","last_undegraded":"2024-04-16T16:15:36.830437+0000","last_fullsized":"2024-04-16T16:15:36.830437+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:53:27.681063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":25,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.822095+0000","last_change":"2024-04-16T16:15:15.450932+0000","last_active":"2024-04-16T16:15:35.822095+0000","last_peered":"2024-04-16T16:15:35.822095+0000","last_clean":"2024-04-16T16:15:35.822095+0000","last_became_active":"2024-04-16T16:15:15.447441+0000","last_became_peered":"2024-04-16T16:15:15.447441+0000","last_unstale":"2024-04-16T16:15:35.822095+0000","last_undegraded":"2024-04-16T16:15:35.822095+0000","last_fullsized":"2024-04-16T16:15:35.822095+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:51.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830286+0000","last_change":"2024-04-16T16:15:35.834241+0000","last_active":"2024-04-16T16:15:36.830286+0000","last_peered":"2024-04-16T16:15:36.830286+0000","last_clean":"2024-04-16T16:15:36.830286+0000","last_became_active":"2024-04-16T16:15:35.833847+0000","last_became_peered":"2024-04-16T16:15:35.833847+0000","last_unstale":"2024-04-16T16:15:36.830286+0000","last_undegraded":"2024-04-16T16:15:36.830286+0000","last_fullsized":"2024-04-16T16:15:36.830286+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:52:21.005662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832853+0000","last_change":"2024-04-16T16:15:15.437158+0000","last_active":"2024-04-16T16:15:36.832853+0000","last_peered":"2024-04-16T16:15:36.832853+0000","last_clean":"2024-04-16T16:15:36.832853+0000","last_became_active":"2024-04-16T16:15:15.435103+0000","last_became_peered":"2024-04-16T16:15:15.435103+0000","last_unstale":"2024-04-16T16:15:36.832853+0000","last_undegraded":"2024-04-16T16:15:36.832853+0000","last_fullsized":"2024-04-16T16:15:36.832853+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:34.312744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.836875+0000","last_change":"2024-04-16T16:15:35.829435+0000","last_active":"2024-04-16T16:15:35.836875+0000","last_peered":"2024-04-16T16:15:35.836875+0000","last_clean":"2024-04-16T16:15:35.836875+0000","last_became_active":"2024-04-16T16:15:35.829109+0000","last_became_peered":"2024-04-16T16:15:35.829109+0000","last_unstale":"2024-04-16T16:15:35.836875+0000","last_undegraded":"2024-04-16T16:15:35.836875+0000","last_fullsized":"2024-04-16T16:15:35.836875+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:55.103124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.824884+0000","last_change":"2024-04-16T16:15:15.453129+0000","last_active":"2024-04-16T16:15:35.824884+0000","last_peered":"2024-04-16T16:15:35.824884+0000","last_clean":"2024-04-16T16:15:35.824884+0000","last_became_active":"2024-04-16T16:15:15.452843+0000","last_became_peered":"2024-04-16T16:15:15.452843+0000","last_unstale":"2024-04-16T16:15:35.824884+0000","last_undegraded":"2024-04-16T16:15:35.824884+0000","last_fullsized":"2024-04-16T16:15:35.824884+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:28:11.918651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829566+0000","last_change":"2024-04-16T16:15:35.820622+0000","last_active":"2024-04-16T16:15:36.829566+0000","last_peered":"2024-04-16T16:15:36.829566+0000","last_clean":"2024-04-16T16:15:36.829566+0000","last_became_active":"2024-04-16T16:15:35.820294+0000","last_became_peered":"2024-04-16T16:15:35.820294+0000","last_unstale":"2024-04-16T16:15:36.829566+0000","last_undegraded":"2024-04-16T16:15:36.829566+0000","last_fullsized":"2024-04-16T16:15:36.829566+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:38:47.915574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.817939+0000","last_change":"2024-04-16T16:15:15.454894+0000","last_active":"2024-04-16T16:15:35.817939+0000","last_peered":"2024-04-16T16:15:35.817939+0000","last_clean":"2024-04-16T16:15:35.817939+0000","last_became_active":"2024-04-16T16:15:15.454659+0000","last_became_peered":"2024-04-16T16:15:15.454659+0000","last_unstale":"2024-04-16T16:15:35.817939+0000","last_undegraded":"2024-04-16T16:15:35.817939+0000","last_fullsized":"2024-04-16T16:15:35.817939+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:44.518137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.841425+0000","last_change":"2024-04-16T16:15:35.834489+0000","last_active":"2024-04-16T16:15:35.841425+0000","last_peered":"2024-04-16T16:15:35.841425+0000","last_clean":"2024-04-16T16:15:35.841425+0000","last_became_active":"2024-04-16T16:15:35.833330+0000","last_became_peered":"2024-04-16T16:15:35.833330+0000","last_unstale":"2024-04-16T16:15:35.841425+0000","last_undegraded":"2024-04-16T16:15:35.841425+0000","last_fullsized":"2024-04-16T16:15:35.841425+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:58.338850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.816103+0000","last_change":"2024-04-16T16:15:15.447202+0000","last_active":"2024-04-16T16:15:35.816103+0000","last_peered":"2024-04-16T16:15:35.816103+0000","last_clean":"2024-04-16T16:15:35.816103+0000","last_became_active":"2024-04-16T16:15:15.444539+0000","last_became_peered":"2024-04-16T16:15:15.444539+0000","last_unstale":"2024-04-16T16:15:35.816103+0000","last_undegraded":"2024-04-16T16:15:35.816103+0000","last_fullsized":"2024-04-16T16:15:35.816103+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:33:41.701252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.835169+0000","last_change":"2024-04-16T16:15:35.831665+0000","last_active":"2024-04-16T16:15:35.835169+0000","last_peered":"2024-04-16T16:15:35.835169+0000","last_clean":"2024-04-16T16:15:35.835169+0000","last_became_active":"2024-04-16T16:15:35.831062+0000","last_became_peered":"2024-04-16T16:15:35.831062+0000","last_unstale":"2024-04-16T16:15:35.835169+0000","last_undegraded":"2024-04-16T16:15:35.835169+0000","last_fullsized":"2024-04-16T16:15:35.835169+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:44:04.763428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.816820+0000","last_change":"2024-04-16T16:15:15.443351+0000","last_active":"2024-04-16T16:15:35.816820+0000","last_peered":"2024-04-16T16:15:35.816820+0000","last_clean":"2024-04-16T16:15:35.816820+0000","last_became_active":"2024-04-16T16:15:15.442366+0000","last_became_peered":"2024-04-16T16:15:15.442366+0000","last_unstale":"2024-04-16T16:15:35.816820+0000","last_undegraded":"2024-04-16T16:15:35.816820+0000","last_fullsized":"2024-04-16T16:15:35.816820+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:53.599226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.843523+0000","last_change":"2024-04-16T16:15:35.841197+0000","last_active":"2024-04-16T16:15:35.843523+0000","last_peered":"2024-04-16T16:15:35.843523+0000","last_clean":"2024-04-16T16:15:35.843523+0000","last_became_active":"2024-04-16T16:15:35.840934+0000","last_became_peered":"2024-04-16T16:15:35.840934+0000","last_unstale":"2024-04-16T16:15:35.843523+0000","last_undegraded":"2024-04-16T16:15:35.843523+0000","last_fullsized":"2024-04-16T16:15:35.843523+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:32:04.710682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811771+0000","last_change":"2024-04-16T16:15:15.462804+0000","last_active":"2024-04-16T16:15:35.811771+0000","last_peered":"2024-04-16T16:15:35.811771+0000","last_clean":"2024-04-16T16:15:35.811771+0000","last_became_active":"2024-04-16T16:15:15.461788+0000","last_became_peered":"2024-04-16T16:15:15.461788+0000","last_unstale":"2024-04-16T16:15:35.811771+0000","last_undegraded":"2024-04-16T16:15:35.811771+0000","last_fullsized":"2024-04-16T16:15:35.811771+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:12.276411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.841465+0000","last_change":"2024-04-16T16:15:35.829688+0000","last_active":"2024-04-16T16:15:35.841465+0000","last_peered":"2024-04-16T16:15:35.841465+0000","last_clean":"2024-04-16T16:15:35.841465+0000","last_became_active":"2024-04-16T16:15:35.828676+0000","last_became_peered":"2024-04-16T16:15:35.828676+0000","last_unstale":"2024-04-16T16:15:35.841465+0000","last_undegraded":"2024-04-16T16:15:35.841465+0000","last_fullsized":"2024-04-16T16:15:35.841465+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:15:06.383899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830237+0000","last_change":"2024-04-16T16:15:15.447751+0000","last_active":"2024-04-16T16:15:36.830237+0000","last_peered":"2024-04-16T16:15:36.830237+0000","last_clean":"2024-04-16T16:15:36.830237+0000","last_became_active":"2024-04-16T16:15:15.447195+0000","last_became_peered":"2024-04-16T16:15:15.447195+0000","last_unstale":"2024-04-16T16:15:36.830237+0000","last_undegraded":"2024-04-16T16:15:36.830237+0000","last_fullsized":"2024-04-16T16:15:36.830237+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:34:32.934841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830068+0000","last_change":"2024-04-16T16:15:35.831119+0000","last_active":"2024-04-16T16:15:36.830068+0000","last_peered":"2024-04-16T16:15:36.830068+0000","last_clean":"2024-04-16T16:15:36.830068+0000","last_became_active":"2024-04-16T16:15:35.830882+0000","last_became_peered":"2024-04-16T16:15:35.830882+0000","last_unstale":"2024-04-16T16:15:36.830068+0000","last_undegraded":"2024-04-16T16:15:36.830068+0000","last_fullsized":"2024-04-16T16:15:36.830068+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:16:18.955453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.825792+0000","last_change":"2024-04-16T16:15:15.456287+0000","last_active":"2024-04-16T16:15:35.825792+0000","last_peered":"2024-04-16T16:15:35.825792+0000","last_clean":"2024-04-16T16:15:35.825792+0000","last_became_active":"2024-04-16T16:15:15.455963+0000","last_became_peered":"2024-04-16T16:15:15.455963+0000","last_unstale":"2024-04-16T16:15:35.825792+0000","last_undegraded":"2024-04-16T16:15:35.825792+0000","last_fullsized":"2024-04-16T16:15:35.825792+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:07.752151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830334+0000","last_change":"2024-04-16T16:15:35.845320+0000","last_active":"2024-04-16T16:15:36.830334+0000","last_peered":"2024-04-16T16:15:36.830334+0000","last_clean":"2024-04-16T16:15:36.830334+0000","last_became_active":"2024-04-16T16:15:35.845067+0000","last_became_peered":"2024-04-16T16:15:35.845067+0000","last_unstale":"2024-04-16T16:15:36.830334+0000","last_undegraded":"2024-04-16T16:15:36.830334+0000","last_fullsized":"2024-04-16T16:15:36.830334+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:00:18.173149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818527+0000","last_change":"2024-04-16T16:15:15.446449+0000","last_active":"2024-04-16T16:15:35.818527+0000","last_peered":"2024-04-16T16:15:35.818527+0000","last_clean":"2024-04-16T16:15:35.818527+0000","last_became_active":"2024-04-16T16:15:15.446135+0000","last_became_peered":"2024-04-16T16:15:15.446135+0000","last_unstale":"2024-04-16T16:15:35.818527+0000","last_undegraded":"2024-04-16T16:15:35.818527+0000","last_fullsized":"2024-04-16T16:15:35.818527+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:49.654384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.838886+0000","last_change":"2024-04-16T16:15:35.836251+0000","last_active":"2024-04-16T16:15:35.838886+0000","last_peered":"2024-04-16T16:15:35.838886+0000","last_clean":"2024-04-16T16:15:35.838886+0000","last_became_active":"2024-04-16T16:15:35.835956+0000","last_became_peered":"2024-04-16T16:15:35.835956+0000","last_unstale":"2024-04-16T16:15:35.838886+0000","last_undegraded":"2024-04-16T16:15:35.838886+0000","last_fullsized":"2024-04-16T16:15:35.838886+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:52:21.005662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.833029+0000","last_change":"2024-04-16T16:15:15.451682+0000","last_active":"2024-04-16T16:15:36.833029+0000","last_peered":"2024-04-16T16:15:36.833029+0000","last_clean":"2024-04-16T16:15:36.833029+0000","last_became_active":"2024-04-16T16:15:15.450999+0000","last_became_peered":"2024-04-16T16:15:15.450999+0000","last_unstale":"2024-04-16T16:15:36.833029+0000","last_undegraded":"2024-04-16T16:15:36.833029+0000","last_fullsized":"2024-04-16T16:15:36.833029+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:01:20.716612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.835205+0000","last_change":"2024-04-16T16:15:35.832685+0000","last_active":"2024-04-16T16:15:35.835205+0000","last_peered":"2024-04-16T16:15:35.835205+0000","last_clean":"2024-04-16T16:15:35.835205+0000","last_became_active":"2024-04-16T16:15:35.832226+0000","last_became_peered":"2024-04-16T16:15:35.832226+0000","last_unstale":"2024-04-16T16:15:35.835205+0000","last_undegraded":"2024-04-16T16:15:35.835205+0000","last_fullsized":"2024-04-16T16:15:35.835205+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:21.250937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.833032+0000","last_change":"2024-04-16T16:15:15.445583+0000","last_active":"2024-04-16T16:15:36.833032+0000","last_peered":"2024-04-16T16:15:36.833032+0000","last_clean":"2024-04-16T16:15:36.833032+0000","last_became_active":"2024-04-16T16:15:15.445259+0000","last_became_peered":"2024-04-16T16:15:15.445259+0000","last_unstale":"2024-04-16T16:15:36.833032+0000","last_undegraded":"2024-04-16T16:15:36.833032+0000","last_fullsized":"2024-04-16T16:15:36.833032+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:56.074330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830022+0000","last_change":"2024-04-16T16:15:35.844725+0000","last_active":"2024-04-16T16:15:36.830022+0000","last_peered":"2024-04-16T16:15:36.830022+0000","last_clean":"2024-04-16T16:15:36.830022+0000","last_became_active":"2024-04-16T16:15:35.844257+0000","last_became_peered":"2024-04-16T16:15:35.844257+0000","last_unstale":"2024-04-16T16:15:36.830022+0000","last_undegraded":"2024-04-16T16:15:36.830022+0000","last_fullsized":"2024-04-16T16:15:36.830022+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:16:41.449544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"46'2","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.815986+0000","last_change":"2024-04-16T16:15:15.447460+0000","last_active":"2024-04-16T16:15:35.815986+0000","last_peered":"2024-04-16T16:15:35.815986+0000","last_clean":"2024-04-16T16:15:35.815986+0000","last_became_active":"2024-04-16T16:15:15.447155+0000","last_became_peered":"2024-04-16T16:15:15.447155+0000","last_unstale":"2024-04-16T16:15:35.815986+0000","last_undegraded":"2024-04-16T16:15:35.815986+0000","last_fullsized":"2024-04-16T16:15:35.815986+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:48:05.645437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.835386+0000","last_change":"2024-04-16T16:15:35.831883+0000","last_active":"2024-04-16T16:15:35.835386+0000","last_peered":"2024-04-16T16:15:35.835386+0000","last_clean":"2024-04-16T16:15:35.835386+0000","last_became_active":"2024-04-16T16:15:35.831376+0000","last_became_peered":"2024-04-16T16:15:35.831376+0000","last_unstale":"2024-04-16T16:15:35.835386+0000","last_undegraded":"2024-04-16T16:15:35.835386+0000","last_fullsized":"2024-04-16T16:15:35.835386+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:28:41.133302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829768+0000","last_change":"2024-04-16T16:15:15.448356+0000","last_active":"2024-04-16T16:15:36.829768+0000","last_peered":"2024-04-16T16:15:36.829768+0000","last_clean":"2024-04-16T16:15:36.829768+0000","last_became_active":"2024-04-16T16:15:15.448056+0000","last_became_peered":"2024-04-16T16:15:15.448056+0000","last_unstale":"2024-04-16T16:15:36.829768+0000","last_undegraded":"2024-04-16T16:15:36.829768+0000","last_fullsized":"2024-04-16T16:15:36.829768+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:30:21.270376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.842185+0000","last_change":"2024-04-16T16:15:35.836622+0000","last_active":"2024-04-16T16:15:35.842185+0000","last_peered":"2024-04-16T16:15:35.842185+0000","last_clean":"2024-04-16T16:15:35.842185+0000","last_became_active":"2024-04-16T16:15:35.836124+0000","last_became_peered":"2024-04-16T16:15:35.836124+0000","last_unstale":"2024-04-16T16:15:35.842185+0000","last_undegraded":"2024-04-16T16:15:35.842185+0000","last_fullsized":"2024-04-16T16:15:35.842185+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:38:47.915574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.810899+0000","last_change":"2024-04-16T16:15:15.459411+0000","last_active":"2024-04-16T16:15:35.810899+0000","last_peered":"2024-04-16T16:15:35.810899+0000","last_clean":"2024-04-16T16:15:35.810899+0000","last_became_active":"2024-04-16T16:15:15.452989+0000","last_became_peered":"2024-04-16T16:15:15.452989+0000","last_unstale":"2024-04-16T16:15:35.810899+0000","last_undegraded":"2024-04-16T16:15:35.810899+0000","last_fullsized":"2024-04-16T16:15:35.810899+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:27.522215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831873+0000","last_change":"2024-04-16T16:15:35.829883+0000","last_active":"2024-04-16T16:15:36.831873+0000","last_peered":"2024-04-16T16:15:36.831873+0000","last_clean":"2024-04-16T16:15:36.831873+0000","last_became_active":"2024-04-16T16:15:35.829096+0000","last_became_peered":"2024-04-16T16:15:35.829096+0000","last_unstale":"2024-04-16T16:15:36.831873+0000","last_undegraded":"2024-04-16T16:15:36.831873+0000","last_fullsized":"2024-04-16T16:15:36.831873+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:36:28.560037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.825192+0000","last_change":"2024-04-16T16:15:15.441331+0000","last_active":"2024-04-16T16:15:35.825192+0000","last_peered":"2024-04-16T16:15:35.825192+0000","last_clean":"2024-04-16T16:15:35.825192+0000","last_became_active":"2024-04-16T16:15:15.440983+0000","last_became_peered":"2024-04-16T16:15:15.440983+0000","last_unstale":"2024-04-16T16:15:35.825192+0000","last_undegraded":"2024-04-16T16:15:35.825192+0000","last_fullsized":"2024-04-16T16:15:35.825192+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:10:59.131225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.847638+0000","last_change":"2024-04-16T16:15:35.839790+0000","last_active":"2024-04-16T16:15:35.847638+0000","last_peered":"2024-04-16T16:15:35.847638+0000","last_clean":"2024-04-16T16:15:35.847638+0000","last_became_active":"2024-04-16T16:15:35.839457+0000","last_became_peered":"2024-04-16T16:15:35.839457+0000","last_unstale":"2024-04-16T16:15:35.847638+0000","last_undegraded":"2024-04-16T16:15:35.847638+0000","last_fullsized":"2024-04-16T16:15:35.847638+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:40:24.612282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":25,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.821980+0000","last_change":"2024-04-16T16:15:15.449849+0000","last_active":"2024-04-16T16:15:35.821980+0000","last_peered":"2024-04-16T16:15:35.821980+0000","last_clean":"2024-04-16T16:15:35.821980+0000","last_became_active":"2024-04-16T16:15:15.449585+0000","last_became_peered":"2024-04-16T16:15:15.449585+0000","last_unstale":"2024-04-16T16:15:35.821980+0000","last_undegraded":"2024-04-16T16:15:35.821980+0000","last_fullsized":"2024-04-16T16:15:35.821980+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:12.276411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.842576+0000","last_change":"2024-04-16T16:15:35.839438+0000","last_active":"2024-04-16T16:15:35.842576+0000","last_peered":"2024-04-16T16:15:35.842576+0000","last_clean":"2024-04-16T16:15:35.842576+0000","last_became_active":"2024-04-16T16:15:35.839164+0000","last_became_peered":"2024-04-16T16:15:35.839164+0000","last_unstale":"2024-04-16T16:15:35.842576+0000","last_undegraded":"2024-04-16T16:15:35.842576+0000","last_fullsized":"2024-04-16T16:15:35.842576+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:37.583592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811485+0000","last_change":"2024-04-16T16:15:15.463054+0000","last_active":"2024-04-16T16:15:35.811485+0000","last_peered":"2024-04-16T16:15:35.811485+0000","last_clean":"2024-04-16T16:15:35.811485+0000","last_became_active":"2024-04-16T16:15:15.462075+0000","last_became_peered":"2024-04-16T16:15:15.462075+0000","last_unstale":"2024-04-16T16:15:35.811485+0000","last_undegraded":"2024-04-16T16:15:35.811485+0000","last_fullsized":"2024-04-16T16:15:35.811485+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:56.074330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829106+0000","last_change":"2024-04-16T16:15:35.848184+0000","last_active":"2024-04-16T16:15:36.829106+0000","last_peered":"2024-04-16T16:15:36.829106+0000","last_clean":"2024-04-16T16:15:36.829106+0000","last_became_active":"2024-04-16T16:15:35.847403+0000","last_became_peered":"2024-04-16T16:15:35.847403+0000","last_unstale":"2024-04-16T16:15:36.829106+0000","last_undegraded":"2024-04-16T16:15:36.829106+0000","last_fullsized":"2024-04-16T16:15:36.829106+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:39:29.875935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811338+0000","last_change":"2024-04-16T16:15:15.459905+0000","last_active":"2024-04-16T16:15:35.811338+0000","last_peered":"2024-04-16T16:15:35.811338+0000","last_clean":"2024-04-16T16:15:35.811338+0000","last_became_active":"2024-04-16T16:15:15.459387+0000","last_became_peered":"2024-04-16T16:15:15.459387+0000","last_unstale":"2024-04-16T16:15:35.811338+0000","last_undegraded":"2024-04-16T16:15:35.811338+0000","last_fullsized":"2024-04-16T16:15:35.811338+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:51.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830953+0000","last_change":"2024-04-16T16:15:35.840856+0000","last_active":"2024-04-16T16:15:36.830953+0000","last_peered":"2024-04-16T16:15:36.830953+0000","last_clean":"2024-04-16T16:15:36.830953+0000","last_became_active":"2024-04-16T16:15:35.840559+0000","last_became_peered":"2024-04-16T16:15:35.840559+0000","last_unstale":"2024-04-16T16:15:36.830953+0000","last_undegraded":"2024-04-16T16:15:36.830953+0000","last_fullsized":"2024-04-16T16:15:36.830953+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:49:48.573089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811068+0000","last_change":"2024-04-16T16:15:15.451403+0000","last_active":"2024-04-16T16:15:35.811068+0000","last_peered":"2024-04-16T16:15:35.811068+0000","last_clean":"2024-04-16T16:15:35.811068+0000","last_became_active":"2024-04-16T16:15:15.450199+0000","last_became_peered":"2024-04-16T16:15:15.450199+0000","last_unstale":"2024-04-16T16:15:35.811068+0000","last_undegraded":"2024-04-16T16:15:35.811068+0000","last_fullsized":"2024-04-16T16:15:35.811068+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:32:52.123986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.841012+0000","last_change":"2024-04-16T16:15:35.833202+0000","last_active":"2024-04-16T16:15:35.841012+0000","last_peered":"2024-04-16T16:15:35.841012+0000","last_clean":"2024-04-16T16:15:35.841012+0000","last_became_active":"2024-04-16T16:15:35.832825+0000","last_became_peered":"2024-04-16T16:15:35.832825+0000","last_unstale":"2024-04-16T16:15:35.841012+0000","last_undegraded":"2024-04-16T16:15:35.841012+0000","last_fullsized":"2024-04-16T16:15:35.841012+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:12.521080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.804499+0000","last_change":"2024-04-16T16:15:15.463187+0000","last_active":"2024-04-16T16:15:35.804499+0000","last_peered":"2024-04-16T16:15:35.804499+0000","last_clean":"2024-04-16T16:15:35.804499+0000","last_became_active":"2024-04-16T16:15:15.461753+0000","last_became_peered":"2024-04-16T16:15:15.461753+0000","last_unstale":"2024-04-16T16:15:35.804499+0000","last_undegraded":"2024-04-16T16:15:35.804499+0000","last_fullsized":"2024-04-16T16:15:35.804499+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:44.974637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.833127+0000","last_change":"2024-04-16T16:15:35.835649+0000","last_active":"2024-04-16T16:15:36.833127+0000","last_peered":"2024-04-16T16:15:36.833127+0000","last_clean":"2024-04-16T16:15:36.833127+0000","last_became_active":"2024-04-16T16:15:35.835102+0000","last_became_peered":"2024-04-16T16:15:35.835102+0000","last_unstale":"2024-04-16T16:15:36.833127+0000","last_undegraded":"2024-04-16T16:15:36.833127+0000","last_fullsized":"2024-04-16T16:15:36.833127+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:06:12.259732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829806+0000","last_change":"2024-04-16T16:15:15.439872+0000","last_active":"2024-04-16T16:15:36.829806+0000","last_peered":"2024-04-16T16:15:36.829806+0000","last_clean":"2024-04-16T16:15:36.829806+0000","last_became_active":"2024-04-16T16:15:15.439553+0000","last_became_peered":"2024-04-16T16:15:15.439553+0000","last_unstale":"2024-04-16T16:15:36.829806+0000","last_undegraded":"2024-04-16T16:15:36.829806+0000","last_fullsized":"2024-04-16T16:15:36.829806+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:12.276411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.841875+0000","last_change":"2024-04-16T16:15:35.836818+0000","last_active":"2024-04-16T16:15:35.841875+0000","last_peered":"2024-04-16T16:15:35.841875+0000","last_clean":"2024-04-16T16:15:35.841875+0000","last_became_active":"2024-04-16T16:15:35.836499+0000","last_became_peered":"2024-04-16T16:15:35.836499+0000","last_unstale":"2024-04-16T16:15:35.841875+0000","last_undegraded":"2024-04-16T16:15:35.841875+0000","last_fullsized":"2024-04-16T16:15:35.841875+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:42:41.959591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830173+0000","last_change":"2024-04-16T16:15:15.429037+0000","last_active":"2024-04-16T16:15:36.830173+0000","last_peered":"2024-04-16T16:15:36.830173+0000","last_clean":"2024-04-16T16:15:36.830173+0000","last_became_active":"2024-04-16T16:15:15.425570+0000","last_became_peered":"2024-04-16T16:15:15.425570+0000","last_unstale":"2024-04-16T16:15:36.830173+0000","last_undegraded":"2024-04-16T16:15:36.830173+0000","last_fullsized":"2024-04-16T16:15:36.830173+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:17:00.492572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.850644+0000","last_change":"2024-04-16T16:15:35.849428+0000","last_active":"2024-04-16T16:15:35.850644+0000","last_peered":"2024-04-16T16:15:35.850644+0000","last_clean":"2024-04-16T16:15:35.850644+0000","last_became_active":"2024-04-16T16:15:35.849131+0000","last_became_peered":"2024-04-16T16:15:35.849131+0000","last_unstale":"2024-04-16T16:15:35.850644+0000","last_undegraded":"2024-04-16T16:15:35.850644+0000","last_fullsized":"2024-04-16T16:15:35.850644+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:25:58.997894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":25,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.822087+0000","last_change":"2024-04-16T16:15:15.443526+0000","last_active":"2024-04-16T16:15:35.822087+0000","last_peered":"2024-04-16T16:15:35.822087+0000","last_clean":"2024-04-16T16:15:35.822087+0000","last_became_active":"2024-04-16T16:15:15.443203+0000","last_became_peered":"2024-04-16T16:15:15.443203+0000","last_unstale":"2024-04-16T16:15:35.822087+0000","last_undegraded":"2024-04-16T16:15:35.822087+0000","last_fullsized":"2024-04-16T16:15:35.822087+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:42.434776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831653+0000","last_change":"2024-04-16T16:15:35.836249+0000","last_active":"2024-04-16T16:15:36.831653+0000","last_peered":"2024-04-16T16:15:36.831653+0000","last_clean":"2024-04-16T16:15:36.831653+0000","last_became_active":"2024-04-16T16:15:35.835942+0000","last_became_peered":"2024-04-16T16:15:35.835942+0000","last_unstale":"2024-04-16T16:15:36.831653+0000","last_undegraded":"2024-04-16T16:15:36.831653+0000","last_fullsized":"2024-04-16T16:15:36.831653+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:49:31.820963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.813048+0000","last_change":"2024-04-16T16:15:15.459192+0000","last_active":"2024-04-16T16:15:35.813048+0000","last_peered":"2024-04-16T16:15:35.813048+0000","last_clean":"2024-04-16T16:15:35.813048+0000","last_became_active":"2024-04-16T16:15:15.452689+0000","last_became_peered":"2024-04-16T16:15:15.452689+0000","last_unstale":"2024-04-16T16:15:35.813048+0000","last_undegraded":"2024-04-16T16:15:35.813048+0000","last_fullsized":"2024-04-16T16:15:35.813048+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:39:57.997770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.836088+0000","last_change":"2024-04-16T16:15:35.830836+0000","last_active":"2024-04-16T16:15:35.836088+0000","last_peered":"2024-04-16T16:15:35.836088+0000","last_clean":"2024-04-16T16:15:35.836088+0000","last_became_active":"2024-04-16T16:15:35.830543+0000","last_became_peered":"2024-04-16T16:15:35.830543+0000","last_unstale":"2024-04-16T16:15:35.836088+0000","last_undegraded":"2024-04-16T16:15:35.836088+0000","last_fullsized":"2024-04-16T16:15:35.836088+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:32.059528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832729+0000","last_change":"2024-04-16T16:15:15.454071+0000","last_active":"2024-04-16T16:15:36.832729+0000","last_peered":"2024-04-16T16:15:36.832729+0000","last_clean":"2024-04-16T16:15:36.832729+0000","last_became_active":"2024-04-16T16:15:15.453781+0000","last_became_peered":"2024-04-16T16:15:15.453781+0000","last_unstale":"2024-04-16T16:15:36.832729+0000","last_undegraded":"2024-04-16T16:15:36.832729+0000","last_fullsized":"2024-04-16T16:15:36.832729+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:03.163739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.838404+0000","last_change":"2024-04-16T16:15:35.831480+0000","last_active":"2024-04-16T16:15:35.838404+0000","last_peered":"2024-04-16T16:15:35.838404+0000","last_clean":"2024-04-16T16:15:35.838404+0000","last_became_active":"2024-04-16T16:15:35.831022+0000","last_became_peered":"2024-04-16T16:15:35.831022+0000","last_unstale":"2024-04-16T16:15:35.838404+0000","last_undegraded":"2024-04-16T16:15:35.838404+0000","last_fullsized":"2024-04-16T16:15:35.838404+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:38:47.915574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":25,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.822994+0000","last_change":"2024-04-16T16:15:15.451103+0000","last_active":"2024-04-16T16:15:35.822994+0000","last_peered":"2024-04-16T16:15:35.822994+0000","last_clean":"2024-04-16T16:15:35.822994+0000","last_became_active":"2024-04-16T16:15:15.450511+0000","last_became_peered":"2024-04-16T16:15:15.450511+0000","last_unstale":"2024-04-16T16:15:35.822994+0000","last_undegraded":"2024-04-16T16:15:35.822994+0000","last_fullsized":"2024-04-16T16:15:35.822994+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:34.706031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831345+0000","last_change":"2024-04-16T16:15:35.827011+0000","last_active":"2024-04-16T16:15:36.831345+0000","last_peered":"2024-04-16T16:15:36.831345+0000","last_clean":"2024-04-16T16:15:36.831345+0000","last_became_active":"2024-04-16T16:15:35.826042+0000","last_became_peered":"2024-04-16T16:15:35.826042+0000","last_unstale":"2024-04-16T16:15:36.831345+0000","last_undegraded":"2024-04-16T16:15:36.831345+0000","last_fullsized":"2024-04-16T16:15:36.831345+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:58.025643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3f","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831423+0000","last_change":"2024-04-16T16:15:15.445346+0000","last_active":"2024-04-16T16:15:36.831423+0000","last_peered":"2024-04-16T16:15:36.831423+0000","last_clean":"2024-04-16T16:15:36.831423+0000","last_became_active":"2024-04-16T16:15:15.445023+0000","last_became_peered":"2024-04-16T16:15:15.445023+0000","last_unstale":"2024-04-16T16:15:36.831423+0000","last_undegraded":"2024-04-16T16:15:36.831423+0000","last_fullsized":"2024-04-16T16:15:36.831423+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:51.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":3145728,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":14232,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":77,"num_read_kb":80,"num_write":52,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":28464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51,"ondisk_log_size":51,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28184,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343656,"statfs":{"total":96636764160,"available":96607903744,"internally_reserved":0,"allocated":1183744,"data_stored":841262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30668,"kb_used_data":3644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341172,"statfs":{"total":96636764160,"available":96605360128,"internally_reserved":0,"allocated":3731456,"data_stored":3397044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":589824,"data_stored":251366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":626688,"data_stored":273245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":610304,"data_stored":258893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":614400,"data_stored":264156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574878,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31756,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340084,"statfs":{"total":96636764160,"available":96604246016,"internally_reserved":0,"allocated":585728,"data_stored":250804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35424,"kb_used_data":4236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336416,"statfs":{"total":96636764160,"available":96600489984,"internally_reserved":0,"allocated":4337664,"data_stored":3988968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":13456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":13434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3145728,"data_stored":3145728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3145728,"data_stored":3145728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:15:40.377 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:15:40.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.636+0000 7f3d8ca00640 1 Processor -- start 2024-04-16T16:15:40.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.636+0000 7f3d8ca00640 1 -- start start 2024-04-16T16:15:40.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.637+0000 7f3d8ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3d8815e8f0 0x7f3d880cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:40.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.637+0000 7f3d8ca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3d880cf080 0x7f3d880cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:40.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.637+0000 7f3d8ca00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8815ecd0 con 0x7f3d880ce9a0 2024-04-16T16:15:40.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.637+0000 7f3d8ca00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3d880ce2e0 con 0x7f3d8815e8f0 2024-04-16T16:15:40.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.637+0000 7f3d8ca00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3d880cd6d0 con 0x7f3d880cf080 2024-04-16T16:15:40.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.637+0000 7f3d86800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3d8815e8f0 0x7f3d880cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:40.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.637+0000 7f3d85e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3d880ce9a0 0x7f3d88162480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58620/0 (socket says 172.21.15.45:58620) 2024-04-16T16:15:40.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.637+0000 7f3d85e00640 1 -- 172.21.15.45:0/3458951269 learned_addr learned my addr 172.21.15.45:0/3458951269 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:40.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.638+0000 7f3d87200640 1 --2- 172.21.15.45:0/3458951269 >> v2:172.21.15.103:3300/0 conn(0x7f3d880cf080 0x7f3d880cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:40.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.638+0000 7f3d86800640 1 -- 172.21.15.45:0/3458951269 >> v2:172.21.15.103:3300/0 conn(0x7f3d880cf080 msgr2=0x7f3d880cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:40.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.638+0000 7f3d86800640 1 --2- 172.21.15.45:0/3458951269 >> v2:172.21.15.103:3300/0 conn(0x7f3d880cf080 0x7f3d880cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.638+0000 7f3d86800640 1 -- 172.21.15.45:0/3458951269 >> v1:172.21.15.45:6789/0 conn(0x7f3d880ce9a0 legacy=0x7f3d88162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:40.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.638+0000 7f3d86800640 1 -- 172.21.15.45:0/3458951269 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d880cd9b0 con 0x7f3d8815e8f0 2024-04-16T16:15:40.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.639+0000 7f3d86800640 1 --2- 172.21.15.45:0/3458951269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d8815e8f0 0x7f3d880cc410 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3d880b3790 tx=0x7f3d7802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99c77850063c2068 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:40.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.639+0000 7f3d85400640 1 -- 172.21.15.45:0/3458951269 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78035020 con 0x7f3d8815e8f0 2024-04-16T16:15:40.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.639+0000 7f3d85400640 1 -- 172.21.15.45:0/3458951269 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d78031830 con 0x7f3d8815e8f0 2024-04-16T16:15:40.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.639+0000 7f3d85400640 1 -- 172.21.15.45:0/3458951269 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78031c00 con 0x7f3d8815e8f0 2024-04-16T16:15:40.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.640+0000 7f3d8ca00640 1 -- 172.21.15.45:0/3458951269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d8815e8f0 msgr2=0x7f3d880cc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:40.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.640+0000 7f3d8ca00640 1 --2- 172.21.15.45:0/3458951269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d8815e8f0 0x7f3d880cc410 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3d880b3790 tx=0x7f3d7802b8d0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.640+0000 7f3d8ca00640 1 -- 172.21.15.45:0/3458951269 shutdown_connections 2024-04-16T16:15:40.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.640+0000 7f3d8ca00640 1 --2- 172.21.15.45:0/3458951269 >> v2:172.21.15.103:3300/0 conn(0x7f3d880cf080 0x7f3d880cbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.640+0000 7f3d8ca00640 1 --2- 172.21.15.45:0/3458951269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d8815e8f0 0x7f3d880cc410 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.640+0000 7f3d8ca00640 1 -- 172.21.15.45:0/3458951269 >> 172.21.15.45:0/3458951269 conn(0x7f3d8806b930 msgr2=0x7f3d8806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:40.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.640+0000 7f3d8ca00640 1 -- 172.21.15.45:0/3458951269 shutdown_connections 2024-04-16T16:15:40.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.640+0000 7f3d8ca00640 1 -- 172.21.15.45:0/3458951269 wait complete. 2024-04-16T16:15:40.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.642+0000 7f3d8ca00640 1 Processor -- start 2024-04-16T16:15:40.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.642+0000 7f3d8ca00640 1 -- start start 2024-04-16T16:15:40.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.643+0000 7f3d8ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d880ce9a0 0x7f3d8814e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:40.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.643+0000 7f3d8ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d880cf080 0x7f3d88152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:40.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.643+0000 7f3d8ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d8815e8f0 0x7f3d88153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:40.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.643+0000 7f3d8ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3d880cd360 con 0x7f3d880ce9a0 2024-04-16T16:15:40.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.643+0000 7f3d8ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3d880cdf70 con 0x7f3d8815e8f0 2024-04-16T16:15:40.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.643+0000 7f3d8ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3d880cc950 con 0x7f3d880cf080 2024-04-16T16:15:40.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.643+0000 7f3d86800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d880ce9a0 0x7f3d8814e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:40.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.643+0000 7f3d86800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d880ce9a0 0x7f3d8814e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45838/0 (socket says 172.21.15.45:45838) 2024-04-16T16:15:40.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.643+0000 7f3d86800640 1 -- 172.21.15.45:0/2152638006 learned_addr learned my addr 172.21.15.45:0/2152638006 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:40.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.644+0000 7f3d87200640 1 --2- 172.21.15.45:0/2152638006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d8815e8f0 0x7f3d88153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:40.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.644+0000 7f3d86800640 1 -- 172.21.15.45:0/2152638006 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d880cf080 msgr2=0x7f3d88152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:40.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.644+0000 7f3d86800640 1 --2- 172.21.15.45:0/2152638006 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d880cf080 0x7f3d88152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.644+0000 7f3d86800640 1 -- 172.21.15.45:0/2152638006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d8815e8f0 msgr2=0x7f3d88153070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:40.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.644+0000 7f3d86800640 1 --2- 172.21.15.45:0/2152638006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d8815e8f0 0x7f3d88153070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.644+0000 7f3d86800640 1 -- 172.21.15.45:0/2152638006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d881569f0 con 0x7f3d880ce9a0 2024-04-16T16:15:40.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.644+0000 7f3d86800640 1 --2- 172.21.15.45:0/2152638006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d880ce9a0 0x7f3d8814e8f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009890 tx=0x7f3d780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:40.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.645+0000 7f3d7ea00640 1 -- 172.21.15.45:0/2152638006 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78040070 con 0x7f3d880ce9a0 2024-04-16T16:15:40.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.645+0000 7f3d7ea00640 1 -- 172.21.15.45:0/2152638006 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d7803e070 con 0x7f3d880ce9a0 2024-04-16T16:15:40.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.645+0000 7f3d7ea00640 1 -- 172.21.15.45:0/2152638006 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78035040 con 0x7f3d880ce9a0 2024-04-16T16:15:40.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.645+0000 7f3d8ca00640 1 -- 172.21.15.45:0/2152638006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d881610e0 con 0x7f3d880ce9a0 2024-04-16T16:15:40.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.645+0000 7f3d8ca00640 1 -- 172.21.15.45:0/2152638006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d881616a0 con 0x7f3d880ce9a0 2024-04-16T16:15:40.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.646+0000 7f3d8ca00640 1 -- 172.21.15.45:0/2152638006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d88075780 con 0x7f3d880ce9a0 2024-04-16T16:15:40.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.647+0000 7f3d7ea00640 1 -- 172.21.15.45:0/2152638006 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f3d78038020 con 0x7f3d880ce9a0 2024-04-16T16:15:40.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.648+0000 7f3d7ea00640 1 --2- 172.21.15.45:0/2152638006 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3d6406ca60 0x7f3d6406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:40.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.648+0000 7f3d7ea00640 1 -- 172.21.15.45:0/2152638006 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6967+0+0 (secure 0 0 0) 0x7f3d780adca0 con 0x7f3d880ce9a0 2024-04-16T16:15:40.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.649+0000 7f3d85e00640 1 --2- 172.21.15.45:0/2152638006 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3d6406ca60 0x7f3d6406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:40.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.650+0000 7f3d85e00640 1 --2- 172.21.15.45:0/2152638006 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3d6406ca60 0x7f3d6406ef20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3d70000f60 tx=0x7f3d70007a80 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:40.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.653+0000 7f3d7ea00640 1 -- 172.21.15.45:0/2152638006 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d7807bac0 con 0x7f3d880ce9a0 2024-04-16T16:15:40.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.854+0000 7f3d8ca00640 1 -- 172.21.15.45:0/2152638006 --> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3d880b5b90 con 0x7f3d6406ca60 2024-04-16T16:15:40.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.892+0000 7f3d7ea00640 1 -- 172.21.15.45:0/2152638006 <== mgr.4658 v2:172.21.15.103:6832/280591707 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490476 (secure 0 0 0) 0x7f3d7008dc00 con 0x7f3d6406ca60 2024-04-16T16:15:40.949 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:40.951 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:15:40.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.898+0000 7f3d8ca00640 1 -- 172.21.15.45:0/2152638006 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3d6406ca60 msgr2=0x7f3d6406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:40.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.898+0000 7f3d8ca00640 1 --2- 172.21.15.45:0/2152638006 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3d6406ca60 0x7f3d6406ef20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3d70000f60 tx=0x7f3d70007a80 comp rx=0 tx=0).stop 2024-04-16T16:15:40.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.898+0000 7f3d8ca00640 1 -- 172.21.15.45:0/2152638006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d880ce9a0 msgr2=0x7f3d8814e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:40.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.898+0000 7f3d8ca00640 1 --2- 172.21.15.45:0/2152638006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d880ce9a0 0x7f3d8814e8f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009890 tx=0x7f3d780046c0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.898+0000 7f3d8ca00640 1 -- 172.21.15.45:0/2152638006 shutdown_connections 2024-04-16T16:15:40.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.898+0000 7f3d8ca00640 1 --2- 172.21.15.45:0/2152638006 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3d6406ca60 0x7f3d6406ef20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.898+0000 7f3d8ca00640 1 --2- 172.21.15.45:0/2152638006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d8815e8f0 0x7f3d88153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.898+0000 7f3d8ca00640 1 --2- 172.21.15.45:0/2152638006 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d880cf080 0x7f3d88152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.898+0000 7f3d8ca00640 1 --2- 172.21.15.45:0/2152638006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d880ce9a0 0x7f3d8814e8f0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:40.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.898+0000 7f3d8ca00640 1 -- 172.21.15.45:0/2152638006 >> 172.21.15.45:0/2152638006 conn(0x7f3d8806b930 msgr2=0x7f3d8806df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:40.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.898+0000 7f3d8ca00640 1 -- 172.21.15.45:0/2152638006 shutdown_connections 2024-04-16T16:15:40.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:40.898+0000 7f3d8ca00640 1 -- 172.21.15.45:0/2152638006 wait complete. 2024-04-16T16:15:40.956 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-16T16:15:39.135904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3752738,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":289,"num_read_kb":386,"num_write":277,"num_write_kb":5532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":220,"ondisk_log_size":220,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":295,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244840,"kb_used_data":11992,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754729880,"statfs":{"total":773094113280,"available":772843397120,"internally_reserved":0,"allocated":12279808,"data_stored":9525738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.361336"},"pg_stats":[{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.828847+0000","last_change":"2024-04-16T16:15:35.833708+0000","last_active":"2024-04-16T16:15:36.828847+0000","last_peered":"2024-04-16T16:15:36.828847+0000","last_clean":"2024-04-16T16:15:36.828847+0000","last_became_active":"2024-04-16T16:15:35.833408+0000","last_became_peered":"2024-04-16T16:15:35.833408+0000","last_unstale":"2024-04-16T16:15:36.828847+0000","last_undegraded":"2024-04-16T16:15:36.828847+0000","last_fullsized":"2024-04-16T16:15:36.828847+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:05:58.589624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"35'128","reported_seq":268,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829661+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:15:36.829661+0000","last_peered":"2024-04-16T16:15:36.829661+0000","last_clean":"2024-04-16T16:15:36.829661+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:15:36.829661+0000","last_undegraded":"2024-04-16T16:15:36.829661+0000","last_fullsized":"2024-04-16T16:15:36.829661+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"41'2","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.823795+0000","last_change":"2024-04-16T16:15:14.420968+0000","last_active":"2024-04-16T16:15:35.823795+0000","last_peered":"2024-04-16T16:15:35.823795+0000","last_clean":"2024-04-16T16:15:35.823795+0000","last_became_active":"2024-04-16T16:15:14.418308+0000","last_became_peered":"2024-04-16T16:15:14.418308+0000","last_unstale":"2024-04-16T16:15:35.823795+0000","last_undegraded":"2024-04-16T16:15:35.823795+0000","last_fullsized":"2024-04-16T16:15:35.823795+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:56.611238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811608+0000","last_change":"2024-04-16T16:15:15.444726+0000","last_active":"2024-04-16T16:15:35.811608+0000","last_peered":"2024-04-16T16:15:35.811608+0000","last_clean":"2024-04-16T16:15:35.811608+0000","last_became_active":"2024-04-16T16:15:15.444386+0000","last_became_peered":"2024-04-16T16:15:15.444386+0000","last_unstale":"2024-04-16T16:15:35.811608+0000","last_undegraded":"2024-04-16T16:15:35.811608+0000","last_fullsized":"2024-04-16T16:15:35.811608+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:34.706031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818397+0000","last_change":"2024-04-16T16:15:34.828397+0000","last_active":"2024-04-16T16:15:35.818397+0000","last_peered":"2024-04-16T16:15:35.818397+0000","last_clean":"2024-04-16T16:15:35.818397+0000","last_became_active":"2024-04-16T16:15:34.828110+0000","last_became_peered":"2024-04-16T16:15:34.828110+0000","last_unstale":"2024-04-16T16:15:35.818397+0000","last_undegraded":"2024-04-16T16:15:35.818397+0000","last_fullsized":"2024-04-16T16:15:35.818397+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:29:06.848116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.842023+0000","last_change":"2024-04-16T16:15:35.837402+0000","last_active":"2024-04-16T16:15:35.842023+0000","last_peered":"2024-04-16T16:15:35.842023+0000","last_clean":"2024-04-16T16:15:35.842023+0000","last_became_active":"2024-04-16T16:15:35.834639+0000","last_became_peered":"2024-04-16T16:15:35.834639+0000","last_unstale":"2024-04-16T16:15:35.842023+0000","last_undegraded":"2024-04-16T16:15:35.842023+0000","last_fullsized":"2024-04-16T16:15:35.842023+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:04.283280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.805491+0000","last_change":"2024-04-16T16:15:14.413186+0000","last_active":"2024-04-16T16:15:35.805491+0000","last_peered":"2024-04-16T16:15:35.805491+0000","last_clean":"2024-04-16T16:15:35.805491+0000","last_became_active":"2024-04-16T16:15:14.412710+0000","last_became_peered":"2024-04-16T16:15:14.412710+0000","last_unstale":"2024-04-16T16:15:35.805491+0000","last_undegraded":"2024-04-16T16:15:35.805491+0000","last_fullsized":"2024-04-16T16:15:35.805491+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:43:43.352649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831550+0000","last_change":"2024-04-16T16:15:15.448491+0000","last_active":"2024-04-16T16:15:36.831550+0000","last_peered":"2024-04-16T16:15:36.831550+0000","last_clean":"2024-04-16T16:15:36.831550+0000","last_became_active":"2024-04-16T16:15:15.448176+0000","last_became_peered":"2024-04-16T16:15:15.448176+0000","last_unstale":"2024-04-16T16:15:36.831550+0000","last_undegraded":"2024-04-16T16:15:36.831550+0000","last_fullsized":"2024-04-16T16:15:36.831550+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:44.518137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"49'2","reported_seq":14,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.844887+0000","last_change":"2024-04-16T16:15:34.821153+0000","last_active":"2024-04-16T16:15:35.844887+0000","last_peered":"2024-04-16T16:15:35.844887+0000","last_clean":"2024-04-16T16:15:35.844887+0000","last_became_active":"2024-04-16T16:15:34.820846+0000","last_became_peered":"2024-04-16T16:15:34.820846+0000","last_unstale":"2024-04-16T16:15:35.844887+0000","last_undegraded":"2024-04-16T16:15:35.844887+0000","last_fullsized":"2024-04-16T16:15:35.844887+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:23:32.286883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830861+0000","last_change":"2024-04-16T16:15:35.828889+0000","last_active":"2024-04-16T16:15:36.830861+0000","last_peered":"2024-04-16T16:15:36.830861+0000","last_clean":"2024-04-16T16:15:36.830861+0000","last_became_active":"2024-04-16T16:15:35.828595+0000","last_became_peered":"2024-04-16T16:15:35.828595+0000","last_unstale":"2024-04-16T16:15:36.830861+0000","last_undegraded":"2024-04-16T16:15:36.830861+0000","last_fullsized":"2024-04-16T16:15:36.830861+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:37:51.603270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.808258+0000","last_change":"2024-04-16T16:15:14.425948+0000","last_active":"2024-04-16T16:15:35.808258+0000","last_peered":"2024-04-16T16:15:35.808258+0000","last_clean":"2024-04-16T16:15:35.808258+0000","last_became_active":"2024-04-16T16:15:14.425692+0000","last_became_peered":"2024-04-16T16:15:14.425692+0000","last_unstale":"2024-04-16T16:15:35.808258+0000","last_undegraded":"2024-04-16T16:15:35.808258+0000","last_fullsized":"2024-04-16T16:15:35.808258+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:16.172813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":25,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.817579+0000","last_change":"2024-04-16T16:15:15.446666+0000","last_active":"2024-04-16T16:15:35.817579+0000","last_peered":"2024-04-16T16:15:35.817579+0000","last_clean":"2024-04-16T16:15:35.817579+0000","last_became_active":"2024-04-16T16:15:15.444672+0000","last_became_peered":"2024-04-16T16:15:15.444672+0000","last_unstale":"2024-04-16T16:15:35.817579+0000","last_undegraded":"2024-04-16T16:15:35.817579+0000","last_fullsized":"2024-04-16T16:15:35.817579+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:39:57.997770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830972+0000","last_change":"2024-04-16T16:15:34.820186+0000","last_active":"2024-04-16T16:15:36.830972+0000","last_peered":"2024-04-16T16:15:36.830972+0000","last_clean":"2024-04-16T16:15:36.830972+0000","last_became_active":"2024-04-16T16:15:34.819904+0000","last_became_peered":"2024-04-16T16:15:34.819904+0000","last_unstale":"2024-04-16T16:15:36.830972+0000","last_undegraded":"2024-04-16T16:15:36.830972+0000","last_fullsized":"2024-04-16T16:15:36.830972+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:08:25.566876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831954+0000","last_change":"2024-04-16T16:15:35.828108+0000","last_active":"2024-04-16T16:15:36.831954+0000","last_peered":"2024-04-16T16:15:36.831954+0000","last_clean":"2024-04-16T16:15:36.831954+0000","last_became_active":"2024-04-16T16:15:35.827263+0000","last_became_peered":"2024-04-16T16:15:35.827263+0000","last_unstale":"2024-04-16T16:15:36.831954+0000","last_undegraded":"2024-04-16T16:15:36.831954+0000","last_fullsized":"2024-04-16T16:15:36.831954+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:08:26.582991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":29,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832113+0000","last_change":"2024-04-16T16:15:14.414769+0000","last_active":"2024-04-16T16:15:36.832113+0000","last_peered":"2024-04-16T16:15:36.832113+0000","last_clean":"2024-04-16T16:15:36.832113+0000","last_became_active":"2024-04-16T16:15:14.414402+0000","last_became_peered":"2024-04-16T16:15:14.414402+0000","last_unstale":"2024-04-16T16:15:36.832113+0000","last_undegraded":"2024-04-16T16:15:36.832113+0000","last_fullsized":"2024-04-16T16:15:36.832113+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:07:59.840158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829924+0000","last_change":"2024-04-16T16:15:15.440497+0000","last_active":"2024-04-16T16:15:36.829924+0000","last_peered":"2024-04-16T16:15:36.829924+0000","last_clean":"2024-04-16T16:15:36.829924+0000","last_became_active":"2024-04-16T16:15:15.440205+0000","last_became_peered":"2024-04-16T16:15:15.440205+0000","last_unstale":"2024-04-16T16:15:36.829924+0000","last_undegraded":"2024-04-16T16:15:36.829924+0000","last_fullsized":"2024-04-16T16:15:36.829924+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:29:51.615992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.812933+0000","last_change":"2024-04-16T16:15:34.817957+0000","last_active":"2024-04-16T16:15:35.812933+0000","last_peered":"2024-04-16T16:15:35.812933+0000","last_clean":"2024-04-16T16:15:35.812933+0000","last_became_active":"2024-04-16T16:15:34.817022+0000","last_became_peered":"2024-04-16T16:15:34.817022+0000","last_unstale":"2024-04-16T16:15:35.812933+0000","last_undegraded":"2024-04-16T16:15:35.812933+0000","last_fullsized":"2024-04-16T16:15:35.812933+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:21:52.222598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830407+0000","last_change":"2024-04-16T16:15:35.848145+0000","last_active":"2024-04-16T16:15:36.830407+0000","last_peered":"2024-04-16T16:15:36.830407+0000","last_clean":"2024-04-16T16:15:36.830407+0000","last_became_active":"2024-04-16T16:15:35.847853+0000","last_became_peered":"2024-04-16T16:15:35.847853+0000","last_unstale":"2024-04-16T16:15:36.830407+0000","last_undegraded":"2024-04-16T16:15:36.830407+0000","last_fullsized":"2024-04-16T16:15:36.830407+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:07:21.242733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":30,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831154+0000","last_change":"2024-04-16T16:15:14.427316+0000","last_active":"2024-04-16T16:15:36.831154+0000","last_peered":"2024-04-16T16:15:36.831154+0000","last_clean":"2024-04-16T16:15:36.831154+0000","last_became_active":"2024-04-16T16:15:14.426635+0000","last_became_peered":"2024-04-16T16:15:14.426635+0000","last_unstale":"2024-04-16T16:15:36.831154+0000","last_undegraded":"2024-04-16T16:15:36.831154+0000","last_fullsized":"2024-04-16T16:15:36.831154+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:43.504451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830660+0000","last_change":"2024-04-16T16:15:15.443473+0000","last_active":"2024-04-16T16:15:36.830660+0000","last_peered":"2024-04-16T16:15:36.830660+0000","last_clean":"2024-04-16T16:15:36.830660+0000","last_became_active":"2024-04-16T16:15:15.439092+0000","last_became_peered":"2024-04-16T16:15:15.439092+0000","last_unstale":"2024-04-16T16:15:36.830660+0000","last_undegraded":"2024-04-16T16:15:36.830660+0000","last_fullsized":"2024-04-16T16:15:36.830660+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:49:28.175995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"49'1","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.839533+0000","last_change":"2024-04-16T16:15:34.827831+0000","last_active":"2024-04-16T16:15:35.839533+0000","last_peered":"2024-04-16T16:15:35.839533+0000","last_clean":"2024-04-16T16:15:35.839533+0000","last_became_active":"2024-04-16T16:15:34.827496+0000","last_became_peered":"2024-04-16T16:15:34.827496+0000","last_unstale":"2024-04-16T16:15:35.839533+0000","last_undegraded":"2024-04-16T16:15:35.839533+0000","last_fullsized":"2024-04-16T16:15:35.839533+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:06:11.243617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.838650+0000","last_change":"2024-04-16T16:15:35.834522+0000","last_active":"2024-04-16T16:15:35.838650+0000","last_peered":"2024-04-16T16:15:35.838650+0000","last_clean":"2024-04-16T16:15:35.838650+0000","last_became_active":"2024-04-16T16:15:35.833902+0000","last_became_peered":"2024-04-16T16:15:35.833902+0000","last_unstale":"2024-04-16T16:15:35.838650+0000","last_undegraded":"2024-04-16T16:15:35.838650+0000","last_fullsized":"2024-04-16T16:15:35.838650+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:04.283280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.814049+0000","last_change":"2024-04-16T16:15:14.424278+0000","last_active":"2024-04-16T16:15:35.814049+0000","last_peered":"2024-04-16T16:15:35.814049+0000","last_clean":"2024-04-16T16:15:35.814049+0000","last_became_active":"2024-04-16T16:15:14.423174+0000","last_became_peered":"2024-04-16T16:15:14.423174+0000","last_unstale":"2024-04-16T16:15:35.814049+0000","last_undegraded":"2024-04-16T16:15:35.814049+0000","last_fullsized":"2024-04-16T16:15:35.814049+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:19:05.074032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832250+0000","last_change":"2024-04-16T16:15:15.449317+0000","last_active":"2024-04-16T16:15:36.832250+0000","last_peered":"2024-04-16T16:15:36.832250+0000","last_clean":"2024-04-16T16:15:36.832250+0000","last_became_active":"2024-04-16T16:15:15.448646+0000","last_became_peered":"2024-04-16T16:15:15.448646+0000","last_unstale":"2024-04-16T16:15:36.832250+0000","last_undegraded":"2024-04-16T16:15:36.832250+0000","last_fullsized":"2024-04-16T16:15:36.832250+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:11.662435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.807980+0000","last_change":"2024-04-16T16:15:34.815648+0000","last_active":"2024-04-16T16:15:35.807980+0000","last_peered":"2024-04-16T16:15:35.807980+0000","last_clean":"2024-04-16T16:15:35.807980+0000","last_became_active":"2024-04-16T16:15:34.815334+0000","last_became_peered":"2024-04-16T16:15:34.815334+0000","last_unstale":"2024-04-16T16:15:35.807980+0000","last_undegraded":"2024-04-16T16:15:35.807980+0000","last_fullsized":"2024-04-16T16:15:35.807980+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:10.753869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.841466+0000","last_change":"2024-04-16T16:15:35.833940+0000","last_active":"2024-04-16T16:15:35.841466+0000","last_peered":"2024-04-16T16:15:35.841466+0000","last_clean":"2024-04-16T16:15:35.841466+0000","last_became_active":"2024-04-16T16:15:35.833636+0000","last_became_peered":"2024-04-16T16:15:35.833636+0000","last_unstale":"2024-04-16T16:15:35.841466+0000","last_undegraded":"2024-04-16T16:15:35.841466+0000","last_fullsized":"2024-04-16T16:15:35.841466+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:21:53.238713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":30,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832426+0000","last_change":"2024-04-16T16:15:14.426984+0000","last_active":"2024-04-16T16:15:36.832426+0000","last_peered":"2024-04-16T16:15:36.832426+0000","last_clean":"2024-04-16T16:15:36.832426+0000","last_became_active":"2024-04-16T16:15:14.426689+0000","last_became_peered":"2024-04-16T16:15:14.426689+0000","last_unstale":"2024-04-16T16:15:36.832426+0000","last_undegraded":"2024-04-16T16:15:36.832426+0000","last_fullsized":"2024-04-16T16:15:36.832426+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:02.150053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.813125+0000","last_change":"2024-04-16T16:15:15.451816+0000","last_active":"2024-04-16T16:15:35.813125+0000","last_peered":"2024-04-16T16:15:35.813125+0000","last_clean":"2024-04-16T16:15:35.813125+0000","last_became_active":"2024-04-16T16:15:15.450017+0000","last_became_peered":"2024-04-16T16:15:15.450017+0000","last_unstale":"2024-04-16T16:15:35.813125+0000","last_undegraded":"2024-04-16T16:15:35.813125+0000","last_fullsized":"2024-04-16T16:15:35.813125+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:42.434776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830471+0000","last_change":"2024-04-16T16:15:34.832134+0000","last_active":"2024-04-16T16:15:36.830471+0000","last_peered":"2024-04-16T16:15:36.830471+0000","last_clean":"2024-04-16T16:15:36.830471+0000","last_became_active":"2024-04-16T16:15:34.831845+0000","last_became_peered":"2024-04-16T16:15:34.831845+0000","last_unstale":"2024-04-16T16:15:36.830471+0000","last_undegraded":"2024-04-16T16:15:36.830471+0000","last_fullsized":"2024-04-16T16:15:36.830471+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:13:15.224743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.837320+0000","last_change":"2024-04-16T16:15:35.831430+0000","last_active":"2024-04-16T16:15:35.837320+0000","last_peered":"2024-04-16T16:15:35.837320+0000","last_clean":"2024-04-16T16:15:35.837320+0000","last_became_active":"2024-04-16T16:15:35.831164+0000","last_became_peered":"2024-04-16T16:15:35.831164+0000","last_unstale":"2024-04-16T16:15:35.837320+0000","last_undegraded":"2024-04-16T16:15:35.837320+0000","last_fullsized":"2024-04-16T16:15:35.837320+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:42:41.959591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"46'2","reported_seq":30,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831015+0000","last_change":"2024-04-16T16:15:14.424803+0000","last_active":"2024-04-16T16:15:36.831015+0000","last_peered":"2024-04-16T16:15:36.831015+0000","last_clean":"2024-04-16T16:15:36.831015+0000","last_became_active":"2024-04-16T16:15:14.424493+0000","last_became_peered":"2024-04-16T16:15:14.424493+0000","last_unstale":"2024-04-16T16:15:36.831015+0000","last_undegraded":"2024-04-16T16:15:36.831015+0000","last_fullsized":"2024-04-16T16:15:36.831015+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:33.299058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"46'1","reported_seq":25,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818174+0000","last_change":"2024-04-16T16:15:15.463217+0000","last_active":"2024-04-16T16:15:35.818174+0000","last_peered":"2024-04-16T16:15:35.818174+0000","last_clean":"2024-04-16T16:15:35.818174+0000","last_became_active":"2024-04-16T16:15:15.462748+0000","last_became_peered":"2024-04-16T16:15:15.462748+0000","last_unstale":"2024-04-16T16:15:35.818174+0000","last_undegraded":"2024-04-16T16:15:35.818174+0000","last_fullsized":"2024-04-16T16:15:35.818174+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:03.163739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097152,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.817600+0000","last_change":"2024-04-16T16:15:34.825031+0000","last_active":"2024-04-16T16:15:35.817600+0000","last_peered":"2024-04-16T16:15:35.817600+0000","last_clean":"2024-04-16T16:15:35.817600+0000","last_became_active":"2024-04-16T16:15:34.824722+0000","last_became_peered":"2024-04-16T16:15:34.824722+0000","last_unstale":"2024-04-16T16:15:35.817600+0000","last_undegraded":"2024-04-16T16:15:35.817600+0000","last_fullsized":"2024-04-16T16:15:35.817600+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:49:47.556974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.841872+0000","last_change":"2024-04-16T16:15:35.836648+0000","last_active":"2024-04-16T16:15:35.841872+0000","last_peered":"2024-04-16T16:15:35.841872+0000","last_clean":"2024-04-16T16:15:35.841872+0000","last_became_active":"2024-04-16T16:15:35.836312+0000","last_became_peered":"2024-04-16T16:15:35.836312+0000","last_unstale":"2024-04-16T16:15:35.841872+0000","last_undegraded":"2024-04-16T16:15:35.841872+0000","last_fullsized":"2024-04-16T16:15:35.841872+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:02:19.857827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":30,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829661+0000","last_change":"2024-04-16T16:15:14.419487+0000","last_active":"2024-04-16T16:15:36.829661+0000","last_peered":"2024-04-16T16:15:36.829661+0000","last_clean":"2024-04-16T16:15:36.829661+0000","last_became_active":"2024-04-16T16:15:14.419154+0000","last_became_peered":"2024-04-16T16:15:14.419154+0000","last_unstale":"2024-04-16T16:15:36.829661+0000","last_undegraded":"2024-04-16T16:15:36.829661+0000","last_fullsized":"2024-04-16T16:15:36.829661+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:01:19.702926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.807009+0000","last_change":"2024-04-16T16:15:15.463514+0000","last_active":"2024-04-16T16:15:35.807009+0000","last_peered":"2024-04-16T16:15:35.807009+0000","last_clean":"2024-04-16T16:15:35.807009+0000","last_became_active":"2024-04-16T16:15:15.461956+0000","last_became_peered":"2024-04-16T16:15:15.461956+0000","last_unstale":"2024-04-16T16:15:35.807009+0000","last_undegraded":"2024-04-16T16:15:35.807009+0000","last_fullsized":"2024-04-16T16:15:35.807009+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:34.312744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811542+0000","last_change":"2024-04-16T16:15:34.827098+0000","last_active":"2024-04-16T16:15:35.811542+0000","last_peered":"2024-04-16T16:15:35.811542+0000","last_clean":"2024-04-16T16:15:35.811542+0000","last_became_active":"2024-04-16T16:15:34.826802+0000","last_became_peered":"2024-04-16T16:15:34.826802+0000","last_unstale":"2024-04-16T16:15:35.811542+0000","last_undegraded":"2024-04-16T16:15:35.811542+0000","last_fullsized":"2024-04-16T16:15:35.811542+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:06:11.243617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.841997+0000","last_change":"2024-04-16T16:15:35.836801+0000","last_active":"2024-04-16T16:15:35.841997+0000","last_peered":"2024-04-16T16:15:35.841997+0000","last_clean":"2024-04-16T16:15:35.841997+0000","last_became_active":"2024-04-16T16:15:35.836175+0000","last_became_peered":"2024-04-16T16:15:35.836175+0000","last_unstale":"2024-04-16T16:15:35.841997+0000","last_undegraded":"2024-04-16T16:15:35.841997+0000","last_fullsized":"2024-04-16T16:15:35.841997+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:15:06.383899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.807505+0000","last_change":"2024-04-16T16:15:14.426381+0000","last_active":"2024-04-16T16:15:35.807505+0000","last_peered":"2024-04-16T16:15:35.807505+0000","last_clean":"2024-04-16T16:15:35.807505+0000","last_became_active":"2024-04-16T16:15:14.425924+0000","last_became_peered":"2024-04-16T16:15:14.425924+0000","last_unstale":"2024-04-16T16:15:35.807505+0000","last_undegraded":"2024-04-16T16:15:35.807505+0000","last_fullsized":"2024-04-16T16:15:35.807505+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:07:59.840158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.820268+0000","last_change":"2024-04-16T16:15:15.440012+0000","last_active":"2024-04-16T16:15:35.820268+0000","last_peered":"2024-04-16T16:15:35.820268+0000","last_clean":"2024-04-16T16:15:35.820268+0000","last_became_active":"2024-04-16T16:15:15.439703+0000","last_became_peered":"2024-04-16T16:15:15.439703+0000","last_unstale":"2024-04-16T16:15:35.820268+0000","last_undegraded":"2024-04-16T16:15:35.820268+0000","last_fullsized":"2024-04-16T16:15:35.820268+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:10:07.586480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"49'2","reported_seq":14,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.852320+0000","last_change":"2024-04-16T16:15:34.816832+0000","last_active":"2024-04-16T16:15:35.852320+0000","last_peered":"2024-04-16T16:15:35.852320+0000","last_clean":"2024-04-16T16:15:35.852320+0000","last_became_active":"2024-04-16T16:15:34.816528+0000","last_became_peered":"2024-04-16T16:15:34.816528+0000","last_unstale":"2024-04-16T16:15:35.852320+0000","last_undegraded":"2024-04-16T16:15:35.852320+0000","last_fullsized":"2024-04-16T16:15:35.852320+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:54.087009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.830067+0000","last_change":"2024-04-16T16:15:35.826463+0000","last_active":"2024-04-16T16:15:35.830067+0000","last_peered":"2024-04-16T16:15:35.830067+0000","last_clean":"2024-04-16T16:15:35.830067+0000","last_became_active":"2024-04-16T16:15:35.826113+0000","last_became_peered":"2024-04-16T16:15:35.826113+0000","last_unstale":"2024-04-16T16:15:35.830067+0000","last_undegraded":"2024-04-16T16:15:35.830067+0000","last_fullsized":"2024-04-16T16:15:35.830067+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:58.338850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":27,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.817771+0000","last_change":"2024-04-16T16:15:14.423634+0000","last_active":"2024-04-16T16:15:35.817771+0000","last_peered":"2024-04-16T16:15:35.817771+0000","last_clean":"2024-04-16T16:15:35.817771+0000","last_became_active":"2024-04-16T16:15:14.422076+0000","last_became_peered":"2024-04-16T16:15:14.422076+0000","last_unstale":"2024-04-16T16:15:35.817771+0000","last_undegraded":"2024-04-16T16:15:35.817771+0000","last_fullsized":"2024-04-16T16:15:35.817771+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:43.504451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.812089+0000","last_change":"2024-04-16T16:15:15.450303+0000","last_active":"2024-04-16T16:15:35.812089+0000","last_peered":"2024-04-16T16:15:35.812089+0000","last_clean":"2024-04-16T16:15:35.812089+0000","last_became_active":"2024-04-16T16:15:15.449983+0000","last_became_peered":"2024-04-16T16:15:15.449983+0000","last_unstale":"2024-04-16T16:15:35.812089+0000","last_undegraded":"2024-04-16T16:15:35.812089+0000","last_fullsized":"2024-04-16T16:15:35.812089+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:46.819298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.821918+0000","last_change":"2024-04-16T16:15:34.834124+0000","last_active":"2024-04-16T16:15:35.821918+0000","last_peered":"2024-04-16T16:15:35.821918+0000","last_clean":"2024-04-16T16:15:35.821918+0000","last_became_active":"2024-04-16T16:15:34.833822+0000","last_became_peered":"2024-04-16T16:15:34.833822+0000","last_unstale":"2024-04-16T16:15:35.821918+0000","last_undegraded":"2024-04-16T16:15:35.821918+0000","last_fullsized":"2024-04-16T16:15:35.821918+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:49:30.804849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.840110+0000","last_change":"2024-04-16T16:15:35.832226+0000","last_active":"2024-04-16T16:15:35.840110+0000","last_peered":"2024-04-16T16:15:35.840110+0000","last_clean":"2024-04-16T16:15:35.840110+0000","last_became_active":"2024-04-16T16:15:35.831978+0000","last_became_peered":"2024-04-16T16:15:35.831978+0000","last_unstale":"2024-04-16T16:15:35.840110+0000","last_undegraded":"2024-04-16T16:15:35.840110+0000","last_fullsized":"2024-04-16T16:15:35.840110+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:05.371730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"43'3","reported_seq":32,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830727+0000","last_change":"2024-04-16T16:15:14.419251+0000","last_active":"2024-04-16T16:15:36.830727+0000","last_peered":"2024-04-16T16:15:36.830727+0000","last_clean":"2024-04-16T16:15:36.830727+0000","last_became_active":"2024-04-16T16:15:14.418965+0000","last_became_peered":"2024-04-16T16:15:14.418965+0000","last_unstale":"2024-04-16T16:15:36.830727+0000","last_undegraded":"2024-04-16T16:15:36.830727+0000","last_fullsized":"2024-04-16T16:15:36.830727+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:11.262725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830376+0000","last_change":"2024-04-16T16:15:15.443802+0000","last_active":"2024-04-16T16:15:36.830376+0000","last_peered":"2024-04-16T16:15:36.830376+0000","last_clean":"2024-04-16T16:15:36.830376+0000","last_became_active":"2024-04-16T16:15:15.443493+0000","last_became_peered":"2024-04-16T16:15:15.443493+0000","last_unstale":"2024-04-16T16:15:36.830376+0000","last_undegraded":"2024-04-16T16:15:36.830376+0000","last_fullsized":"2024-04-16T16:15:36.830376+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:31:00.851752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818139+0000","last_change":"2024-04-16T16:15:34.832847+0000","last_active":"2024-04-16T16:15:35.818139+0000","last_peered":"2024-04-16T16:15:35.818139+0000","last_clean":"2024-04-16T16:15:35.818139+0000","last_became_active":"2024-04-16T16:15:34.832270+0000","last_became_peered":"2024-04-16T16:15:34.832270+0000","last_unstale":"2024-04-16T16:15:35.818139+0000","last_undegraded":"2024-04-16T16:15:35.818139+0000","last_fullsized":"2024-04-16T16:15:35.818139+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:10.753869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830668+0000","last_change":"2024-04-16T16:15:35.830104+0000","last_active":"2024-04-16T16:15:36.830668+0000","last_peered":"2024-04-16T16:15:36.830668+0000","last_clean":"2024-04-16T16:15:36.830668+0000","last_became_active":"2024-04-16T16:15:35.829832+0000","last_became_peered":"2024-04-16T16:15:35.829832+0000","last_unstale":"2024-04-16T16:15:36.830668+0000","last_undegraded":"2024-04-16T16:15:36.830668+0000","last_fullsized":"2024-04-16T16:15:36.830668+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:40:24.612282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830098+0000","last_change":"2024-04-16T16:15:14.423513+0000","last_active":"2024-04-16T16:15:36.830098+0000","last_peered":"2024-04-16T16:15:36.830098+0000","last_clean":"2024-04-16T16:15:36.830098+0000","last_became_active":"2024-04-16T16:15:14.423209+0000","last_became_peered":"2024-04-16T16:15:14.423209+0000","last_unstale":"2024-04-16T16:15:36.830098+0000","last_undegraded":"2024-04-16T16:15:36.830098+0000","last_fullsized":"2024-04-16T16:15:36.830098+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:41.421090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818323+0000","last_change":"2024-04-16T16:15:15.451372+0000","last_active":"2024-04-16T16:15:35.818323+0000","last_peered":"2024-04-16T16:15:35.818323+0000","last_clean":"2024-04-16T16:15:35.818323+0000","last_became_active":"2024-04-16T16:15:15.450983+0000","last_became_peered":"2024-04-16T16:15:15.450983+0000","last_unstale":"2024-04-16T16:15:35.818323+0000","last_undegraded":"2024-04-16T16:15:35.818323+0000","last_fullsized":"2024-04-16T16:15:35.818323+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:05:00.389608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831791+0000","last_change":"2024-04-16T16:15:34.832040+0000","last_active":"2024-04-16T16:15:36.831791+0000","last_peered":"2024-04-16T16:15:36.831791+0000","last_clean":"2024-04-16T16:15:36.831791+0000","last_became_active":"2024-04-16T16:15:34.831671+0000","last_became_peered":"2024-04-16T16:15:34.831671+0000","last_unstale":"2024-04-16T16:15:36.831791+0000","last_undegraded":"2024-04-16T16:15:36.831791+0000","last_fullsized":"2024-04-16T16:15:36.831791+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:25:57.981780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829161+0000","last_change":"2024-04-16T16:15:35.836601+0000","last_active":"2024-04-16T16:15:36.829161+0000","last_peered":"2024-04-16T16:15:36.829161+0000","last_clean":"2024-04-16T16:15:36.829161+0000","last_became_active":"2024-04-16T16:15:35.834804+0000","last_became_peered":"2024-04-16T16:15:35.834804+0000","last_unstale":"2024-04-16T16:15:36.829161+0000","last_undegraded":"2024-04-16T16:15:36.829161+0000","last_fullsized":"2024-04-16T16:15:36.829161+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:53:28.308092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"41'1","reported_seq":29,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.822546+0000","last_change":"2024-04-16T16:15:14.417893+0000","last_active":"2024-04-16T16:15:35.822546+0000","last_peered":"2024-04-16T16:15:35.822546+0000","last_clean":"2024-04-16T16:15:35.822546+0000","last_became_active":"2024-04-16T16:15:14.416107+0000","last_became_peered":"2024-04-16T16:15:14.416107+0000","last_unstale":"2024-04-16T16:15:35.822546+0000","last_undegraded":"2024-04-16T16:15:35.822546+0000","last_fullsized":"2024-04-16T16:15:35.822546+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:59.375922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.817717+0000","last_change":"2024-04-16T16:15:15.446817+0000","last_active":"2024-04-16T16:15:35.817717+0000","last_peered":"2024-04-16T16:15:35.817717+0000","last_clean":"2024-04-16T16:15:35.817717+0000","last_became_active":"2024-04-16T16:15:15.444908+0000","last_became_peered":"2024-04-16T16:15:15.444908+0000","last_unstale":"2024-04-16T16:15:35.817717+0000","last_undegraded":"2024-04-16T16:15:35.817717+0000","last_fullsized":"2024-04-16T16:15:35.817717+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:46.819298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811893+0000","last_change":"2024-04-16T16:15:34.825582+0000","last_active":"2024-04-16T16:15:35.811893+0000","last_peered":"2024-04-16T16:15:35.811893+0000","last_clean":"2024-04-16T16:15:35.811893+0000","last_became_active":"2024-04-16T16:15:34.824775+0000","last_became_peered":"2024-04-16T16:15:34.824775+0000","last_unstale":"2024-04-16T16:15:35.811893+0000","last_undegraded":"2024-04-16T16:15:35.811893+0000","last_fullsized":"2024-04-16T16:15:35.811893+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:29:06.848116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.828847+0000","last_change":"2024-04-16T16:15:35.841752+0000","last_active":"2024-04-16T16:15:36.828847+0000","last_peered":"2024-04-16T16:15:36.828847+0000","last_clean":"2024-04-16T16:15:36.828847+0000","last_became_active":"2024-04-16T16:15:35.841011+0000","last_became_peered":"2024-04-16T16:15:35.841011+0000","last_unstale":"2024-04-16T16:15:36.828847+0000","last_undegraded":"2024-04-16T16:15:36.828847+0000","last_fullsized":"2024-04-16T16:15:36.828847+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:37:06.311207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"41'2","reported_seq":33,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.833293+0000","last_change":"2024-04-16T16:15:14.420778+0000","last_active":"2024-04-16T16:15:36.833293+0000","last_peered":"2024-04-16T16:15:36.833293+0000","last_clean":"2024-04-16T16:15:36.833293+0000","last_became_active":"2024-04-16T16:15:14.420315+0000","last_became_peered":"2024-04-16T16:15:14.420315+0000","last_unstale":"2024-04-16T16:15:36.833293+0000","last_undegraded":"2024-04-16T16:15:36.833293+0000","last_fullsized":"2024-04-16T16:15:36.833293+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:43:43.352649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811179+0000","last_change":"2024-04-16T16:15:15.442790+0000","last_active":"2024-04-16T16:15:35.811179+0000","last_peered":"2024-04-16T16:15:35.811179+0000","last_clean":"2024-04-16T16:15:35.811179+0000","last_became_active":"2024-04-16T16:15:15.439265+0000","last_became_peered":"2024-04-16T16:15:15.439265+0000","last_unstale":"2024-04-16T16:15:35.811179+0000","last_undegraded":"2024-04-16T16:15:35.811179+0000","last_fullsized":"2024-04-16T16:15:35.811179+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:40:41.651741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.828788+0000","last_change":"2024-04-16T16:15:34.826765+0000","last_active":"2024-04-16T16:15:36.828788+0000","last_peered":"2024-04-16T16:15:36.828788+0000","last_clean":"2024-04-16T16:15:36.828788+0000","last_became_active":"2024-04-16T16:15:34.826268+0000","last_became_peered":"2024-04-16T16:15:34.826268+0000","last_unstale":"2024-04-16T16:15:36.828788+0000","last_undegraded":"2024-04-16T16:15:36.828788+0000","last_fullsized":"2024-04-16T16:15:36.828788+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:52:19.989548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.840976+0000","last_change":"2024-04-16T16:15:35.835346+0000","last_active":"2024-04-16T16:15:35.840976+0000","last_peered":"2024-04-16T16:15:35.840976+0000","last_clean":"2024-04-16T16:15:35.840976+0000","last_became_active":"2024-04-16T16:15:35.834372+0000","last_became_peered":"2024-04-16T16:15:35.834372+0000","last_unstale":"2024-04-16T16:15:35.840976+0000","last_undegraded":"2024-04-16T16:15:35.840976+0000","last_fullsized":"2024-04-16T16:15:35.840976+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:02.048834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"41'1","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.825302+0000","last_change":"2024-04-16T16:15:14.406535+0000","last_active":"2024-04-16T16:15:35.825302+0000","last_peered":"2024-04-16T16:15:35.825302+0000","last_clean":"2024-04-16T16:15:35.825302+0000","last_became_active":"2024-04-16T16:15:14.405872+0000","last_became_peered":"2024-04-16T16:15:14.405872+0000","last_unstale":"2024-04-16T16:15:35.825302+0000","last_undegraded":"2024-04-16T16:15:35.825302+0000","last_fullsized":"2024-04-16T16:15:35.825302+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:12:11.062203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.824925+0000","last_change":"2024-04-16T16:15:15.444883+0000","last_active":"2024-04-16T16:15:35.824925+0000","last_peered":"2024-04-16T16:15:35.824925+0000","last_clean":"2024-04-16T16:15:35.824925+0000","last_became_active":"2024-04-16T16:15:15.438454+0000","last_became_peered":"2024-04-16T16:15:15.438454+0000","last_unstale":"2024-04-16T16:15:35.824925+0000","last_undegraded":"2024-04-16T16:15:35.824925+0000","last_fullsized":"2024-04-16T16:15:35.824925+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:25:00.737475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.807978+0000","last_change":"2024-04-16T16:15:34.815921+0000","last_active":"2024-04-16T16:15:35.807978+0000","last_peered":"2024-04-16T16:15:35.807978+0000","last_clean":"2024-04-16T16:15:35.807978+0000","last_became_active":"2024-04-16T16:15:34.814833+0000","last_became_peered":"2024-04-16T16:15:34.814833+0000","last_unstale":"2024-04-16T16:15:35.807978+0000","last_undegraded":"2024-04-16T16:15:35.807978+0000","last_fullsized":"2024-04-16T16:15:35.807978+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:31.657389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.838181+0000","last_change":"2024-04-16T16:15:35.831248+0000","last_active":"2024-04-16T16:15:35.838181+0000","last_peered":"2024-04-16T16:15:35.838181+0000","last_clean":"2024-04-16T16:15:35.838181+0000","last_became_active":"2024-04-16T16:15:35.830852+0000","last_became_peered":"2024-04-16T16:15:35.830852+0000","last_unstale":"2024-04-16T16:15:35.838181+0000","last_undegraded":"2024-04-16T16:15:35.838181+0000","last_fullsized":"2024-04-16T16:15:35.838181+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:32.059528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830725+0000","last_change":"2024-04-16T16:15:34.824918+0000","last_active":"2024-04-16T16:15:36.830725+0000","last_peered":"2024-04-16T16:15:36.830725+0000","last_clean":"2024-04-16T16:15:36.830725+0000","last_became_active":"2024-04-16T16:15:34.824549+0000","last_became_peered":"2024-04-16T16:15:34.824549+0000","last_unstale":"2024-04-16T16:15:36.830725+0000","last_undegraded":"2024-04-16T16:15:36.830725+0000","last_fullsized":"2024-04-16T16:15:36.830725+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:37:50.587155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"41'8","reported_seq":42,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.805552+0000","last_change":"2024-04-16T16:15:14.421997+0000","last_active":"2024-04-16T16:15:35.805552+0000","last_peered":"2024-04-16T16:15:35.805552+0000","last_clean":"2024-04-16T16:15:35.805552+0000","last_became_active":"2024-04-16T16:15:14.421747+0000","last_became_peered":"2024-04-16T16:15:14.421747+0000","last_unstale":"2024-04-16T16:15:35.805552+0000","last_undegraded":"2024-04-16T16:15:35.805552+0000","last_fullsized":"2024-04-16T16:15:35.805552+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:56:30.781319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829916+0000","last_change":"2024-04-16T16:15:15.446100+0000","last_active":"2024-04-16T16:15:36.829916+0000","last_peered":"2024-04-16T16:15:36.829916+0000","last_clean":"2024-04-16T16:15:36.829916+0000","last_became_active":"2024-04-16T16:15:15.444480+0000","last_became_peered":"2024-04-16T16:15:15.444480+0000","last_unstale":"2024-04-16T16:15:36.829916+0000","last_undegraded":"2024-04-16T16:15:36.829916+0000","last_fullsized":"2024-04-16T16:15:36.829916+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:47.467138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.840798+0000","last_change":"2024-04-16T16:15:35.834966+0000","last_active":"2024-04-16T16:15:35.840798+0000","last_peered":"2024-04-16T16:15:35.840798+0000","last_clean":"2024-04-16T16:15:35.840798+0000","last_became_active":"2024-04-16T16:15:35.830812+0000","last_became_peered":"2024-04-16T16:15:35.830812+0000","last_unstale":"2024-04-16T16:15:35.840798+0000","last_undegraded":"2024-04-16T16:15:35.840798+0000","last_fullsized":"2024-04-16T16:15:35.840798+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:16:18.955453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"49'4","reported_seq":16,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.849316+0000","last_change":"2024-04-16T16:15:34.826238+0000","last_active":"2024-04-16T16:15:35.849316+0000","last_peered":"2024-04-16T16:15:35.849316+0000","last_clean":"2024-04-16T16:15:35.849316+0000","last_became_active":"2024-04-16T16:15:34.825558+0000","last_became_peered":"2024-04-16T16:15:34.825558+0000","last_unstale":"2024-04-16T16:15:35.849316+0000","last_undegraded":"2024-04-16T16:15:35.849316+0000","last_fullsized":"2024-04-16T16:15:35.849316+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:49:30.804849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829794+0000","last_change":"2024-04-16T16:15:14.410824+0000","last_active":"2024-04-16T16:15:36.829794+0000","last_peered":"2024-04-16T16:15:36.829794+0000","last_clean":"2024-04-16T16:15:36.829794+0000","last_became_active":"2024-04-16T16:15:14.409874+0000","last_became_peered":"2024-04-16T16:15:14.409874+0000","last_unstale":"2024-04-16T16:15:36.829794+0000","last_undegraded":"2024-04-16T16:15:36.829794+0000","last_fullsized":"2024-04-16T16:15:36.829794+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:26:59.442481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829182+0000","last_change":"2024-04-16T16:15:15.455736+0000","last_active":"2024-04-16T16:15:36.829182+0000","last_peered":"2024-04-16T16:15:36.829182+0000","last_clean":"2024-04-16T16:15:36.829182+0000","last_became_active":"2024-04-16T16:15:15.455415+0000","last_became_peered":"2024-04-16T16:15:15.455415+0000","last_unstale":"2024-04-16T16:15:36.829182+0000","last_undegraded":"2024-04-16T16:15:36.829182+0000","last_fullsized":"2024-04-16T16:15:36.829182+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:29:30.311490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.847967+0000","last_change":"2024-04-16T16:15:35.842047+0000","last_active":"2024-04-16T16:15:35.847967+0000","last_peered":"2024-04-16T16:15:35.847967+0000","last_clean":"2024-04-16T16:15:35.847967+0000","last_became_active":"2024-04-16T16:15:35.836938+0000","last_became_peered":"2024-04-16T16:15:35.836938+0000","last_unstale":"2024-04-16T16:15:35.847967+0000","last_undegraded":"2024-04-16T16:15:35.847967+0000","last_fullsized":"2024-04-16T16:15:35.847967+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:13:16.240858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":30,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830345+0000","last_change":"2024-04-16T16:15:14.421692+0000","last_active":"2024-04-16T16:15:36.830345+0000","last_peered":"2024-04-16T16:15:36.830345+0000","last_clean":"2024-04-16T16:15:36.830345+0000","last_became_active":"2024-04-16T16:15:14.421361+0000","last_became_peered":"2024-04-16T16:15:14.421361+0000","last_unstale":"2024-04-16T16:15:36.830345+0000","last_undegraded":"2024-04-16T16:15:36.830345+0000","last_fullsized":"2024-04-16T16:15:36.830345+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:56.611238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.812609+0000","last_change":"2024-04-16T16:15:15.462282+0000","last_active":"2024-04-16T16:15:35.812609+0000","last_peered":"2024-04-16T16:15:35.812609+0000","last_clean":"2024-04-16T16:15:35.812609+0000","last_became_active":"2024-04-16T16:15:15.461980+0000","last_became_peered":"2024-04-16T16:15:15.461980+0000","last_unstale":"2024-04-16T16:15:35.812609+0000","last_undegraded":"2024-04-16T16:15:35.812609+0000","last_fullsized":"2024-04-16T16:15:35.812609+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:11.662435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"49'2","reported_seq":14,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.840601+0000","last_change":"2024-04-16T16:15:34.834352+0000","last_active":"2024-04-16T16:15:35.840601+0000","last_peered":"2024-04-16T16:15:35.840601+0000","last_clean":"2024-04-16T16:15:35.840601+0000","last_became_active":"2024-04-16T16:15:34.834044+0000","last_became_peered":"2024-04-16T16:15:34.834044+0000","last_unstale":"2024-04-16T16:15:35.840601+0000","last_undegraded":"2024-04-16T16:15:35.840601+0000","last_fullsized":"2024-04-16T16:15:35.840601+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:23:32.286883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.830576+0000","last_change":"2024-04-16T16:15:35.827425+0000","last_active":"2024-04-16T16:15:35.830576+0000","last_peered":"2024-04-16T16:15:35.830576+0000","last_clean":"2024-04-16T16:15:35.830576+0000","last_became_active":"2024-04-16T16:15:35.827140+0000","last_became_peered":"2024-04-16T16:15:35.827140+0000","last_unstale":"2024-04-16T16:15:35.830576+0000","last_undegraded":"2024-04-16T16:15:35.830576+0000","last_fullsized":"2024-04-16T16:15:35.830576+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:02:19.857827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"41'2","reported_seq":32,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830557+0000","last_change":"2024-04-16T16:15:14.408195+0000","last_active":"2024-04-16T16:15:36.830557+0000","last_peered":"2024-04-16T16:15:36.830557+0000","last_clean":"2024-04-16T16:15:36.830557+0000","last_became_active":"2024-04-16T16:15:14.407584+0000","last_became_peered":"2024-04-16T16:15:14.407584+0000","last_unstale":"2024-04-16T16:15:36.830557+0000","last_undegraded":"2024-04-16T16:15:36.830557+0000","last_fullsized":"2024-04-16T16:15:36.830557+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:39:56.984084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.817189+0000","last_change":"2024-04-16T16:15:15.447435+0000","last_active":"2024-04-16T16:15:35.817189+0000","last_peered":"2024-04-16T16:15:35.817189+0000","last_clean":"2024-04-16T16:15:35.817189+0000","last_became_active":"2024-04-16T16:15:15.447094+0000","last_became_peered":"2024-04-16T16:15:15.447094+0000","last_unstale":"2024-04-16T16:15:35.817189+0000","last_undegraded":"2024-04-16T16:15:35.817189+0000","last_fullsized":"2024-04-16T16:15:35.817189+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:16:44.917291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1d","version":"49'2","reported_seq":14,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.847919+0000","last_change":"2024-04-16T16:15:34.816070+0000","last_active":"2024-04-16T16:15:35.847919+0000","last_peered":"2024-04-16T16:15:35.847919+0000","last_clean":"2024-04-16T16:15:35.847919+0000","last_became_active":"2024-04-16T16:15:34.815753+0000","last_became_peered":"2024-04-16T16:15:34.815753+0000","last_unstale":"2024-04-16T16:15:35.847919+0000","last_undegraded":"2024-04-16T16:15:35.847919+0000","last_fullsized":"2024-04-16T16:15:35.847919+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:35:37.144432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829807+0000","last_change":"2024-04-16T16:15:35.850306+0000","last_active":"2024-04-16T16:15:36.829807+0000","last_peered":"2024-04-16T16:15:36.829807+0000","last_clean":"2024-04-16T16:15:36.829807+0000","last_became_active":"2024-04-16T16:15:35.849296+0000","last_became_peered":"2024-04-16T16:15:35.849296+0000","last_unstale":"2024-04-16T16:15:36.829807+0000","last_undegraded":"2024-04-16T16:15:36.829807+0000","last_fullsized":"2024-04-16T16:15:36.829807+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:06:37.061403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"46'12","reported_seq":46,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.825898+0000","last_change":"2024-04-16T16:15:14.419221+0000","last_active":"2024-04-16T16:15:35.825898+0000","last_peered":"2024-04-16T16:15:35.825898+0000","last_clean":"2024-04-16T16:15:35.825898+0000","last_became_active":"2024-04-16T16:15:14.418573+0000","last_became_peered":"2024-04-16T16:15:14.418573+0000","last_unstale":"2024-04-16T16:15:35.825898+0000","last_undegraded":"2024-04-16T16:15:35.825898+0000","last_fullsized":"2024-04-16T16:15:35.825898+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:08.107514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":11,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832100+0000","last_change":"2024-04-16T16:15:15.435454+0000","last_active":"2024-04-16T16:15:36.832100+0000","last_peered":"2024-04-16T16:15:36.832100+0000","last_clean":"2024-04-16T16:15:36.832100+0000","last_became_active":"2024-04-16T16:15:15.435056+0000","last_became_peered":"2024-04-16T16:15:15.435056+0000","last_unstale":"2024-04-16T16:15:36.832100+0000","last_undegraded":"2024-04-16T16:15:36.832100+0000","last_fullsized":"2024-04-16T16:15:36.832100+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:34.706031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1a","version":"49'1","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.845431+0000","last_change":"2024-04-16T16:15:34.827734+0000","last_active":"2024-04-16T16:15:35.845431+0000","last_peered":"2024-04-16T16:15:35.845431+0000","last_clean":"2024-04-16T16:15:35.845431+0000","last_became_active":"2024-04-16T16:15:34.827375+0000","last_became_peered":"2024-04-16T16:15:34.827375+0000","last_unstale":"2024-04-16T16:15:35.845431+0000","last_undegraded":"2024-04-16T16:15:35.845431+0000","last_fullsized":"2024-04-16T16:15:35.845431+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:49:22.456907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.828851+0000","last_change":"2024-04-16T16:15:35.850483+0000","last_active":"2024-04-16T16:15:36.828851+0000","last_peered":"2024-04-16T16:15:36.828851+0000","last_clean":"2024-04-16T16:15:36.828851+0000","last_became_active":"2024-04-16T16:15:35.845409+0000","last_became_peered":"2024-04-16T16:15:35.845409+0000","last_unstale":"2024-04-16T16:15:36.828851+0000","last_undegraded":"2024-04-16T16:15:36.828851+0000","last_fullsized":"2024-04-16T16:15:36.828851+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:06:43.224387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"41'3","reported_seq":36,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832278+0000","last_change":"2024-04-16T16:15:14.427833+0000","last_active":"2024-04-16T16:15:36.832278+0000","last_peered":"2024-04-16T16:15:36.832278+0000","last_clean":"2024-04-16T16:15:36.832278+0000","last_became_active":"2024-04-16T16:15:14.427492+0000","last_became_peered":"2024-04-16T16:15:14.427492+0000","last_unstale":"2024-04-16T16:15:36.832278+0000","last_undegraded":"2024-04-16T16:15:36.832278+0000","last_fullsized":"2024-04-16T16:15:36.832278+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:48.640698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829941+0000","last_change":"2024-04-16T16:15:15.453206+0000","last_active":"2024-04-16T16:15:36.829941+0000","last_peered":"2024-04-16T16:15:36.829941+0000","last_clean":"2024-04-16T16:15:36.829941+0000","last_became_active":"2024-04-16T16:15:15.453002+0000","last_became_peered":"2024-04-16T16:15:15.453002+0000","last_unstale":"2024-04-16T16:15:36.829941+0000","last_undegraded":"2024-04-16T16:15:36.829941+0000","last_fullsized":"2024-04-16T16:15:36.829941+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:37.628550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832456+0000","last_change":"2024-04-16T16:15:34.832901+0000","last_active":"2024-04-16T16:15:36.832456+0000","last_peered":"2024-04-16T16:15:36.832456+0000","last_clean":"2024-04-16T16:15:36.832456+0000","last_became_active":"2024-04-16T16:15:34.832392+0000","last_became_peered":"2024-04-16T16:15:34.832392+0000","last_unstale":"2024-04-16T16:15:36.832456+0000","last_undegraded":"2024-04-16T16:15:36.832456+0000","last_fullsized":"2024-04-16T16:15:36.832456+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:14:10.714795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829277+0000","last_change":"2024-04-16T16:15:35.842001+0000","last_active":"2024-04-16T16:15:36.829277+0000","last_peered":"2024-04-16T16:15:36.829277+0000","last_clean":"2024-04-16T16:15:36.829277+0000","last_became_active":"2024-04-16T16:15:35.841419+0000","last_became_peered":"2024-04-16T16:15:35.841419+0000","last_unstale":"2024-04-16T16:15:36.829277+0000","last_undegraded":"2024-04-16T16:15:36.829277+0000","last_fullsized":"2024-04-16T16:15:36.829277+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:12:17.054235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811731+0000","last_change":"2024-04-16T16:15:14.427378+0000","last_active":"2024-04-16T16:15:35.811731+0000","last_peered":"2024-04-16T16:15:35.811731+0000","last_clean":"2024-04-16T16:15:35.811731+0000","last_became_active":"2024-04-16T16:15:14.427036+0000","last_became_peered":"2024-04-16T16:15:14.427036+0000","last_unstale":"2024-04-16T16:15:35.811731+0000","last_undegraded":"2024-04-16T16:15:35.811731+0000","last_fullsized":"2024-04-16T16:15:35.811731+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:16.172813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831520+0000","last_change":"2024-04-16T16:15:15.436333+0000","last_active":"2024-04-16T16:15:36.831520+0000","last_peered":"2024-04-16T16:15:36.831520+0000","last_clean":"2024-04-16T16:15:36.831520+0000","last_became_active":"2024-04-16T16:15:15.435961+0000","last_became_peered":"2024-04-16T16:15:15.435961+0000","last_unstale":"2024-04-16T16:15:36.831520+0000","last_undegraded":"2024-04-16T16:15:36.831520+0000","last_fullsized":"2024-04-16T16:15:36.831520+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:44.974637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831296+0000","last_change":"2024-04-16T16:15:34.825679+0000","last_active":"2024-04-16T16:15:36.831296+0000","last_peered":"2024-04-16T16:15:36.831296+0000","last_clean":"2024-04-16T16:15:36.831296+0000","last_became_active":"2024-04-16T16:15:34.824988+0000","last_became_peered":"2024-04-16T16:15:34.824988+0000","last_unstale":"2024-04-16T16:15:36.831296+0000","last_undegraded":"2024-04-16T16:15:36.831296+0000","last_fullsized":"2024-04-16T16:15:36.831296+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:34:03.842188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.839887+0000","last_change":"2024-04-16T16:15:35.834848+0000","last_active":"2024-04-16T16:15:35.839887+0000","last_peered":"2024-04-16T16:15:35.839887+0000","last_clean":"2024-04-16T16:15:35.839887+0000","last_became_active":"2024-04-16T16:15:35.833387+0000","last_became_peered":"2024-04-16T16:15:35.833387+0000","last_unstale":"2024-04-16T16:15:35.839887+0000","last_undegraded":"2024-04-16T16:15:35.839887+0000","last_fullsized":"2024-04-16T16:15:35.839887+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:05.371730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":29,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.833213+0000","last_change":"2024-04-16T16:15:14.422188+0000","last_active":"2024-04-16T16:15:36.833213+0000","last_peered":"2024-04-16T16:15:36.833213+0000","last_clean":"2024-04-16T16:15:36.833213+0000","last_became_active":"2024-04-16T16:15:14.421842+0000","last_became_peered":"2024-04-16T16:15:14.421842+0000","last_unstale":"2024-04-16T16:15:36.833213+0000","last_undegraded":"2024-04-16T16:15:36.833213+0000","last_fullsized":"2024-04-16T16:15:36.833213+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:43.299902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.833046+0000","last_change":"2024-04-16T16:15:15.451420+0000","last_active":"2024-04-16T16:15:36.833046+0000","last_peered":"2024-04-16T16:15:36.833046+0000","last_clean":"2024-04-16T16:15:36.833046+0000","last_became_active":"2024-04-16T16:15:15.450543+0000","last_became_peered":"2024-04-16T16:15:15.450543+0000","last_unstale":"2024-04-16T16:15:36.833046+0000","last_undegraded":"2024-04-16T16:15:36.833046+0000","last_fullsized":"2024-04-16T16:15:36.833046+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:27:00.456168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.820095+0000","last_change":"2024-04-16T16:15:34.830670+0000","last_active":"2024-04-16T16:15:35.820095+0000","last_peered":"2024-04-16T16:15:35.820095+0000","last_clean":"2024-04-16T16:15:35.820095+0000","last_became_active":"2024-04-16T16:15:34.830389+0000","last_became_peered":"2024-04-16T16:15:34.830389+0000","last_unstale":"2024-04-16T16:15:35.820095+0000","last_undegraded":"2024-04-16T16:15:35.820095+0000","last_fullsized":"2024-04-16T16:15:35.820095+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:57.009529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831368+0000","last_change":"2024-04-16T16:15:35.829605+0000","last_active":"2024-04-16T16:15:36.831368+0000","last_peered":"2024-04-16T16:15:36.831368+0000","last_clean":"2024-04-16T16:15:36.831368+0000","last_became_active":"2024-04-16T16:15:35.828966+0000","last_became_peered":"2024-04-16T16:15:35.828966+0000","last_unstale":"2024-04-16T16:15:36.831368+0000","last_undegraded":"2024-04-16T16:15:36.831368+0000","last_fullsized":"2024-04-16T16:15:36.831368+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:43.515569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.805163+0000","last_change":"2024-04-16T16:15:14.411837+0000","last_active":"2024-04-16T16:15:35.805163+0000","last_peered":"2024-04-16T16:15:35.805163+0000","last_clean":"2024-04-16T16:15:35.805163+0000","last_became_active":"2024-04-16T16:15:14.411512+0000","last_became_peered":"2024-04-16T16:15:14.411512+0000","last_unstale":"2024-04-16T16:15:35.805163+0000","last_undegraded":"2024-04-16T16:15:35.805163+0000","last_fullsized":"2024-04-16T16:15:35.805163+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:28:19.722523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.816715+0000","last_change":"2024-04-16T16:15:15.438978+0000","last_active":"2024-04-16T16:15:35.816715+0000","last_peered":"2024-04-16T16:15:35.816715+0000","last_clean":"2024-04-16T16:15:35.816715+0000","last_became_active":"2024-04-16T16:15:15.438597+0000","last_became_peered":"2024-04-16T16:15:15.438597+0000","last_unstale":"2024-04-16T16:15:35.816715+0000","last_undegraded":"2024-04-16T16:15:35.816715+0000","last_fullsized":"2024-04-16T16:15:35.816715+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:19:06.087718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830878+0000","last_change":"2024-04-16T16:15:34.819768+0000","last_active":"2024-04-16T16:15:36.830878+0000","last_peered":"2024-04-16T16:15:36.830878+0000","last_clean":"2024-04-16T16:15:36.830878+0000","last_became_active":"2024-04-16T16:15:34.819470+0000","last_became_peered":"2024-04-16T16:15:34.819470+0000","last_unstale":"2024-04-16T16:15:36.830878+0000","last_undegraded":"2024-04-16T16:15:36.830878+0000","last_fullsized":"2024-04-16T16:15:36.830878+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:58:01.343104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832948+0000","last_change":"2024-04-16T16:15:35.827678+0000","last_active":"2024-04-16T16:15:36.832948+0000","last_peered":"2024-04-16T16:15:36.832948+0000","last_clean":"2024-04-16T16:15:36.832948+0000","last_became_active":"2024-04-16T16:15:35.827407+0000","last_became_peered":"2024-04-16T16:15:35.827407+0000","last_unstale":"2024-04-16T16:15:36.832948+0000","last_undegraded":"2024-04-16T16:15:36.832948+0000","last_fullsized":"2024-04-16T16:15:36.832948+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:23:33.302998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":27,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.820400+0000","last_change":"2024-04-16T16:15:14.418012+0000","last_active":"2024-04-16T16:15:35.820400+0000","last_peered":"2024-04-16T16:15:35.820400+0000","last_clean":"2024-04-16T16:15:35.820400+0000","last_became_active":"2024-04-16T16:15:14.417689+0000","last_became_peered":"2024-04-16T16:15:14.417689+0000","last_unstale":"2024-04-16T16:15:35.820400+0000","last_undegraded":"2024-04-16T16:15:35.820400+0000","last_fullsized":"2024-04-16T16:15:35.820400+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:33:40.687566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.804895+0000","last_change":"2024-04-16T16:15:15.461073+0000","last_active":"2024-04-16T16:15:35.804895+0000","last_peered":"2024-04-16T16:15:35.804895+0000","last_clean":"2024-04-16T16:15:35.804895+0000","last_became_active":"2024-04-16T16:15:15.459255+0000","last_became_peered":"2024-04-16T16:15:15.459255+0000","last_unstale":"2024-04-16T16:15:35.804895+0000","last_undegraded":"2024-04-16T16:15:35.804895+0000","last_fullsized":"2024-04-16T16:15:35.804895+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:27:00.456168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831188+0000","last_change":"2024-04-16T16:15:34.829788+0000","last_active":"2024-04-16T16:15:36.831188+0000","last_peered":"2024-04-16T16:15:36.831188+0000","last_clean":"2024-04-16T16:15:36.831188+0000","last_became_active":"2024-04-16T16:15:34.829489+0000","last_became_peered":"2024-04-16T16:15:34.829489+0000","last_unstale":"2024-04-16T16:15:36.831188+0000","last_undegraded":"2024-04-16T16:15:36.831188+0000","last_fullsized":"2024-04-16T16:15:36.831188+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:10.996970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832192+0000","last_change":"2024-04-16T16:15:35.830113+0000","last_active":"2024-04-16T16:15:36.832192+0000","last_peered":"2024-04-16T16:15:36.832192+0000","last_clean":"2024-04-16T16:15:36.832192+0000","last_became_active":"2024-04-16T16:15:35.829488+0000","last_became_peered":"2024-04-16T16:15:35.829488+0000","last_unstale":"2024-04-16T16:15:36.832192+0000","last_undegraded":"2024-04-16T16:15:36.832192+0000","last_fullsized":"2024-04-16T16:15:36.832192+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:13.599730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"43'2","reported_seq":31,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832200+0000","last_change":"2024-04-16T16:15:14.429151+0000","last_active":"2024-04-16T16:15:36.832200+0000","last_peered":"2024-04-16T16:15:36.832200+0000","last_clean":"2024-04-16T16:15:36.832200+0000","last_became_active":"2024-04-16T16:15:14.428841+0000","last_became_peered":"2024-04-16T16:15:14.428841+0000","last_unstale":"2024-04-16T16:15:36.832200+0000","last_undegraded":"2024-04-16T16:15:36.832200+0000","last_fullsized":"2024-04-16T16:15:36.832200+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:16.172813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.812373+0000","last_change":"2024-04-16T16:15:15.460542+0000","last_active":"2024-04-16T16:15:35.812373+0000","last_peered":"2024-04-16T16:15:35.812373+0000","last_clean":"2024-04-16T16:15:35.812373+0000","last_became_active":"2024-04-16T16:15:15.459809+0000","last_became_peered":"2024-04-16T16:15:15.459809+0000","last_unstale":"2024-04-16T16:15:35.812373+0000","last_undegraded":"2024-04-16T16:15:35.812373+0000","last_fullsized":"2024-04-16T16:15:35.812373+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:47.467138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"49'6","reported_seq":20,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832462+0000","last_change":"2024-04-16T16:15:34.831724+0000","last_active":"2024-04-16T16:15:36.832462+0000","last_peered":"2024-04-16T16:15:36.832462+0000","last_clean":"2024-04-16T16:15:36.832462+0000","last_became_active":"2024-04-16T16:15:34.831436+0000","last_became_peered":"2024-04-16T16:15:34.831436+0000","last_unstale":"2024-04-16T16:15:36.832462+0000","last_undegraded":"2024-04-16T16:15:36.832462+0000","last_fullsized":"2024-04-16T16:15:36.832462+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:01.032719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.842383+0000","last_change":"2024-04-16T16:15:35.835878+0000","last_active":"2024-04-16T16:15:35.842383+0000","last_peered":"2024-04-16T16:15:35.842383+0000","last_clean":"2024-04-16T16:15:35.842383+0000","last_became_active":"2024-04-16T16:15:35.828058+0000","last_became_peered":"2024-04-16T16:15:35.828058+0000","last_unstale":"2024-04-16T16:15:35.842383+0000","last_undegraded":"2024-04-16T16:15:35.842383+0000","last_fullsized":"2024-04-16T16:15:35.842383+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:14:11.730910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"43'2","reported_seq":30,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830474+0000","last_change":"2024-04-16T16:15:14.419476+0000","last_active":"2024-04-16T16:15:36.830474+0000","last_peered":"2024-04-16T16:15:36.830474+0000","last_clean":"2024-04-16T16:15:36.830474+0000","last_became_active":"2024-04-16T16:15:14.419185+0000","last_became_peered":"2024-04-16T16:15:14.419185+0000","last_unstale":"2024-04-16T16:15:36.830474+0000","last_undegraded":"2024-04-16T16:15:36.830474+0000","last_fullsized":"2024-04-16T16:15:36.830474+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:26.508529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831945+0000","last_change":"2024-04-16T16:15:15.451450+0000","last_active":"2024-04-16T16:15:36.831945+0000","last_peered":"2024-04-16T16:15:36.831945+0000","last_clean":"2024-04-16T16:15:36.831945+0000","last_became_active":"2024-04-16T16:15:15.451103+0000","last_became_peered":"2024-04-16T16:15:15.451103+0000","last_unstale":"2024-04-16T16:15:36.831945+0000","last_undegraded":"2024-04-16T16:15:36.831945+0000","last_fullsized":"2024-04-16T16:15:36.831945+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:49.654384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.15","version":"49'3","reported_seq":17,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832383+0000","last_change":"2024-04-16T16:15:34.827571+0000","last_active":"2024-04-16T16:15:36.832383+0000","last_peered":"2024-04-16T16:15:36.832383+0000","last_clean":"2024-04-16T16:15:36.832383+0000","last_became_active":"2024-04-16T16:15:34.827285+0000","last_became_peered":"2024-04-16T16:15:34.827285+0000","last_unstale":"2024-04-16T16:15:36.832383+0000","last_undegraded":"2024-04-16T16:15:36.832383+0000","last_fullsized":"2024-04-16T16:15:36.832383+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:34:52.315819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830990+0000","last_change":"2024-04-16T16:15:35.832303+0000","last_active":"2024-04-16T16:15:36.830990+0000","last_peered":"2024-04-16T16:15:36.830990+0000","last_clean":"2024-04-16T16:15:36.830990+0000","last_became_active":"2024-04-16T16:15:35.832022+0000","last_became_peered":"2024-04-16T16:15:35.832022+0000","last_unstale":"2024-04-16T16:15:36.830990+0000","last_undegraded":"2024-04-16T16:15:36.830990+0000","last_fullsized":"2024-04-16T16:15:36.830990+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:58:02.359219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1c","version":"41'2","reported_seq":34,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831565+0000","last_change":"2024-04-16T16:15:14.426165+0000","last_active":"2024-04-16T16:15:36.831565+0000","last_peered":"2024-04-16T16:15:36.831565+0000","last_clean":"2024-04-16T16:15:36.831565+0000","last_became_active":"2024-04-16T16:15:14.424933+0000","last_became_peered":"2024-04-16T16:15:14.424933+0000","last_unstale":"2024-04-16T16:15:36.831565+0000","last_undegraded":"2024-04-16T16:15:36.831565+0000","last_fullsized":"2024-04-16T16:15:36.831565+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:59.375922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818560+0000","last_change":"2024-04-16T16:15:15.455406+0000","last_active":"2024-04-16T16:15:35.818560+0000","last_peered":"2024-04-16T16:15:35.818560+0000","last_clean":"2024-04-16T16:15:35.818560+0000","last_became_active":"2024-04-16T16:15:15.455065+0000","last_became_peered":"2024-04-16T16:15:15.455065+0000","last_unstale":"2024-04-16T16:15:35.818560+0000","last_undegraded":"2024-04-16T16:15:35.818560+0000","last_fullsized":"2024-04-16T16:15:35.818560+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:48:05.645437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.817080+0000","last_change":"2024-04-16T16:15:34.824758+0000","last_active":"2024-04-16T16:15:35.817080+0000","last_peered":"2024-04-16T16:15:35.817080+0000","last_clean":"2024-04-16T16:15:35.817080+0000","last_became_active":"2024-04-16T16:15:34.824345+0000","last_became_peered":"2024-04-16T16:15:34.824345+0000","last_unstale":"2024-04-16T16:15:35.817080+0000","last_undegraded":"2024-04-16T16:15:35.817080+0000","last_fullsized":"2024-04-16T16:15:35.817080+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:31.657389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.840644+0000","last_change":"2024-04-16T16:15:35.833575+0000","last_active":"2024-04-16T16:15:35.840644+0000","last_peered":"2024-04-16T16:15:35.840644+0000","last_clean":"2024-04-16T16:15:35.840644+0000","last_became_active":"2024-04-16T16:15:35.833268+0000","last_became_peered":"2024-04-16T16:15:35.833268+0000","last_unstale":"2024-04-16T16:15:35.840644+0000","last_undegraded":"2024-04-16T16:15:35.840644+0000","last_fullsized":"2024-04-16T16:15:35.840644+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:35:38.160546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"41'2","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818688+0000","last_change":"2024-04-16T16:15:14.420969+0000","last_active":"2024-04-16T16:15:35.818688+0000","last_peered":"2024-04-16T16:15:35.818688+0000","last_clean":"2024-04-16T16:15:35.818688+0000","last_became_active":"2024-04-16T16:15:14.420657+0000","last_became_peered":"2024-04-16T16:15:14.420657+0000","last_unstale":"2024-04-16T16:15:35.818688+0000","last_undegraded":"2024-04-16T16:15:35.818688+0000","last_fullsized":"2024-04-16T16:15:35.818688+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:43.299902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":25,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.822149+0000","last_change":"2024-04-16T16:15:15.432880+0000","last_active":"2024-04-16T16:15:35.822149+0000","last_peered":"2024-04-16T16:15:35.822149+0000","last_clean":"2024-04-16T16:15:35.822149+0000","last_became_active":"2024-04-16T16:15:15.432458+0000","last_became_peered":"2024-04-16T16:15:15.432458+0000","last_unstale":"2024-04-16T16:15:35.822149+0000","last_undegraded":"2024-04-16T16:15:35.822149+0000","last_fullsized":"2024-04-16T16:15:35.822149+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:47.467138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829561+0000","last_change":"2024-04-16T16:15:34.817296+0000","last_active":"2024-04-16T16:15:36.829561+0000","last_peered":"2024-04-16T16:15:36.829561+0000","last_clean":"2024-04-16T16:15:36.829561+0000","last_became_active":"2024-04-16T16:15:34.817033+0000","last_became_peered":"2024-04-16T16:15:34.817033+0000","last_unstale":"2024-04-16T16:15:36.829561+0000","last_undegraded":"2024-04-16T16:15:36.829561+0000","last_fullsized":"2024-04-16T16:15:36.829561+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:40:23.596167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829591+0000","last_change":"2024-04-16T16:15:35.824339+0000","last_active":"2024-04-16T16:15:36.829591+0000","last_peered":"2024-04-16T16:15:36.829591+0000","last_clean":"2024-04-16T16:15:36.829591+0000","last_became_active":"2024-04-16T16:15:35.824047+0000","last_became_peered":"2024-04-16T16:15:35.824047+0000","last_unstale":"2024-04-16T16:15:36.829591+0000","last_undegraded":"2024-04-16T16:15:36.829591+0000","last_fullsized":"2024-04-16T16:15:36.829591+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:04.283280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1e","version":"43'2","reported_seq":31,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.833341+0000","last_change":"2024-04-16T16:15:14.430354+0000","last_active":"2024-04-16T16:15:36.833341+0000","last_peered":"2024-04-16T16:15:36.833341+0000","last_clean":"2024-04-16T16:15:36.833341+0000","last_became_active":"2024-04-16T16:15:14.429965+0000","last_became_peered":"2024-04-16T16:15:14.429965+0000","last_unstale":"2024-04-16T16:15:36.833341+0000","last_undegraded":"2024-04-16T16:15:36.833341+0000","last_fullsized":"2024-04-16T16:15:36.833341+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:28:19.722523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818063+0000","last_change":"2024-04-16T16:15:15.449524+0000","last_active":"2024-04-16T16:15:35.818063+0000","last_peered":"2024-04-16T16:15:35.818063+0000","last_clean":"2024-04-16T16:15:35.818063+0000","last_became_active":"2024-04-16T16:15:15.448988+0000","last_became_peered":"2024-04-16T16:15:15.448988+0000","last_unstale":"2024-04-16T16:15:35.818063+0000","last_undegraded":"2024-04-16T16:15:35.818063+0000","last_fullsized":"2024-04-16T16:15:35.818063+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:57.624924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.10","version":"49'8","reported_seq":22,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829433+0000","last_change":"2024-04-16T16:15:34.828466+0000","last_active":"2024-04-16T16:15:36.829433+0000","last_peered":"2024-04-16T16:15:36.829433+0000","last_clean":"2024-04-16T16:15:36.829433+0000","last_became_active":"2024-04-16T16:15:34.826881+0000","last_became_peered":"2024-04-16T16:15:34.826881+0000","last_unstale":"2024-04-16T16:15:36.829433+0000","last_undegraded":"2024-04-16T16:15:36.829433+0000","last_fullsized":"2024-04-16T16:15:36.829433+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:31:20.232731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831366+0000","last_change":"2024-04-16T16:15:35.837395+0000","last_active":"2024-04-16T16:15:36.831366+0000","last_peered":"2024-04-16T16:15:36.831366+0000","last_clean":"2024-04-16T16:15:36.831366+0000","last_became_active":"2024-04-16T16:15:35.837059+0000","last_became_peered":"2024-04-16T16:15:35.837059+0000","last_unstale":"2024-04-16T16:15:36.831366+0000","last_undegraded":"2024-04-16T16:15:36.831366+0000","last_fullsized":"2024-04-16T16:15:36.831366+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:49:23.473022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"43'5","reported_seq":35,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831355+0000","last_change":"2024-04-16T16:15:14.426636+0000","last_active":"2024-04-16T16:15:36.831355+0000","last_peered":"2024-04-16T16:15:36.831355+0000","last_clean":"2024-04-16T16:15:36.831355+0000","last_became_active":"2024-04-16T16:15:14.426319+0000","last_became_peered":"2024-04-16T16:15:14.426319+0000","last_unstale":"2024-04-16T16:15:36.831355+0000","last_undegraded":"2024-04-16T16:15:36.831355+0000","last_fullsized":"2024-04-16T16:15:36.831355+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:13.363343+0000","last_clean_scrub_stamp":"2024-04-16T16:15:13.363343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:55.060644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.821834+0000","last_change":"2024-04-16T16:15:15.444805+0000","last_active":"2024-04-16T16:15:35.821834+0000","last_peered":"2024-04-16T16:15:35.821834+0000","last_clean":"2024-04-16T16:15:35.821834+0000","last_became_active":"2024-04-16T16:15:15.442030+0000","last_became_peered":"2024-04-16T16:15:15.442030+0000","last_unstale":"2024-04-16T16:15:35.821834+0000","last_undegraded":"2024-04-16T16:15:35.821834+0000","last_fullsized":"2024-04-16T16:15:35.821834+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:33:41.701252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.820275+0000","last_change":"2024-04-16T16:15:34.828471+0000","last_active":"2024-04-16T16:15:35.820275+0000","last_peered":"2024-04-16T16:15:35.820275+0000","last_clean":"2024-04-16T16:15:35.820275+0000","last_became_active":"2024-04-16T16:15:34.827085+0000","last_became_peered":"2024-04-16T16:15:34.827085+0000","last_unstale":"2024-04-16T16:15:35.820275+0000","last_undegraded":"2024-04-16T16:15:35.820275+0000","last_fullsized":"2024-04-16T16:15:35.820275+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:33.758008+0000","last_clean_scrub_stamp":"2024-04-16T16:15:33.758008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:49:22.456907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830762+0000","last_change":"2024-04-16T16:15:35.827049+0000","last_active":"2024-04-16T16:15:36.830762+0000","last_peered":"2024-04-16T16:15:36.830762+0000","last_clean":"2024-04-16T16:15:36.830762+0000","last_became_active":"2024-04-16T16:15:35.826687+0000","last_became_peered":"2024-04-16T16:15:35.826687+0000","last_unstale":"2024-04-16T16:15:36.830762+0000","last_undegraded":"2024-04-16T16:15:36.830762+0000","last_fullsized":"2024-04-16T16:15:36.830762+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:13:16.240858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.819707+0000","last_change":"2024-04-16T16:15:15.448903+0000","last_active":"2024-04-16T16:15:35.819707+0000","last_peered":"2024-04-16T16:15:35.819707+0000","last_clean":"2024-04-16T16:15:35.819707+0000","last_became_active":"2024-04-16T16:15:15.448548+0000","last_became_peered":"2024-04-16T16:15:15.448548+0000","last_unstale":"2024-04-16T16:15:35.819707+0000","last_undegraded":"2024-04-16T16:15:35.819707+0000","last_fullsized":"2024-04-16T16:15:35.819707+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:49.734767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831307+0000","last_change":"2024-04-16T16:15:35.841869+0000","last_active":"2024-04-16T16:15:36.831307+0000","last_peered":"2024-04-16T16:15:36.831307+0000","last_clean":"2024-04-16T16:15:36.831307+0000","last_became_active":"2024-04-16T16:15:35.841098+0000","last_became_peered":"2024-04-16T16:15:35.841098+0000","last_unstale":"2024-04-16T16:15:36.831307+0000","last_undegraded":"2024-04-16T16:15:36.831307+0000","last_fullsized":"2024-04-16T16:15:36.831307+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:25:08.011289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829466+0000","last_change":"2024-04-16T16:15:15.434979+0000","last_active":"2024-04-16T16:15:36.829466+0000","last_peered":"2024-04-16T16:15:36.829466+0000","last_clean":"2024-04-16T16:15:36.829466+0000","last_became_active":"2024-04-16T16:15:15.434028+0000","last_became_peered":"2024-04-16T16:15:15.434028+0000","last_unstale":"2024-04-16T16:15:36.829466+0000","last_undegraded":"2024-04-16T16:15:36.829466+0000","last_fullsized":"2024-04-16T16:15:36.829466+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:57:00.347584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829684+0000","last_change":"2024-04-16T16:15:35.850706+0000","last_active":"2024-04-16T16:15:36.829684+0000","last_peered":"2024-04-16T16:15:36.829684+0000","last_clean":"2024-04-16T16:15:36.829684+0000","last_became_active":"2024-04-16T16:15:35.850376+0000","last_became_peered":"2024-04-16T16:15:35.850376+0000","last_unstale":"2024-04-16T16:15:36.829684+0000","last_undegraded":"2024-04-16T16:15:36.829684+0000","last_fullsized":"2024-04-16T16:15:36.829684+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:48.663991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831942+0000","last_change":"2024-04-16T16:15:15.445371+0000","last_active":"2024-04-16T16:15:36.831942+0000","last_peered":"2024-04-16T16:15:36.831942+0000","last_clean":"2024-04-16T16:15:36.831942+0000","last_became_active":"2024-04-16T16:15:15.445063+0000","last_became_peered":"2024-04-16T16:15:15.445063+0000","last_unstale":"2024-04-16T16:15:36.831942+0000","last_undegraded":"2024-04-16T16:15:36.831942+0000","last_fullsized":"2024-04-16T16:15:36.831942+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:14:57.624924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.848983+0000","last_change":"2024-04-16T16:15:35.846891+0000","last_active":"2024-04-16T16:15:35.848983+0000","last_peered":"2024-04-16T16:15:35.848983+0000","last_clean":"2024-04-16T16:15:35.848983+0000","last_became_active":"2024-04-16T16:15:35.845284+0000","last_became_peered":"2024-04-16T16:15:35.845284+0000","last_unstale":"2024-04-16T16:15:35.848983+0000","last_undegraded":"2024-04-16T16:15:35.848983+0000","last_fullsized":"2024-04-16T16:15:35.848983+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:19:26.484812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818531+0000","last_change":"2024-04-16T16:15:15.445285+0000","last_active":"2024-04-16T16:15:35.818531+0000","last_peered":"2024-04-16T16:15:35.818531+0000","last_clean":"2024-04-16T16:15:35.818531+0000","last_became_active":"2024-04-16T16:15:15.444969+0000","last_became_peered":"2024-04-16T16:15:15.444969+0000","last_unstale":"2024-04-16T16:15:35.818531+0000","last_undegraded":"2024-04-16T16:15:35.818531+0000","last_fullsized":"2024-04-16T16:15:35.818531+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:38:30.263403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830437+0000","last_change":"2024-04-16T16:15:35.842607+0000","last_active":"2024-04-16T16:15:36.830437+0000","last_peered":"2024-04-16T16:15:36.830437+0000","last_clean":"2024-04-16T16:15:36.830437+0000","last_became_active":"2024-04-16T16:15:35.842338+0000","last_became_peered":"2024-04-16T16:15:35.842338+0000","last_unstale":"2024-04-16T16:15:36.830437+0000","last_undegraded":"2024-04-16T16:15:36.830437+0000","last_fullsized":"2024-04-16T16:15:36.830437+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:53:27.681063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":25,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.822095+0000","last_change":"2024-04-16T16:15:15.450932+0000","last_active":"2024-04-16T16:15:35.822095+0000","last_peered":"2024-04-16T16:15:35.822095+0000","last_clean":"2024-04-16T16:15:35.822095+0000","last_became_active":"2024-04-16T16:15:15.447441+0000","last_became_peered":"2024-04-16T16:15:15.447441+0000","last_unstale":"2024-04-16T16:15:35.822095+0000","last_undegraded":"2024-04-16T16:15:35.822095+0000","last_fullsized":"2024-04-16T16:15:35.822095+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:51.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830286+0000","last_change":"2024-04-16T16:15:35.834241+0000","last_active":"2024-04-16T16:15:36.830286+0000","last_peered":"2024-04-16T16:15:36.830286+0000","last_clean":"2024-04-16T16:15:36.830286+0000","last_became_active":"2024-04-16T16:15:35.833847+0000","last_became_peered":"2024-04-16T16:15:35.833847+0000","last_unstale":"2024-04-16T16:15:36.830286+0000","last_undegraded":"2024-04-16T16:15:36.830286+0000","last_fullsized":"2024-04-16T16:15:36.830286+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:52:21.005662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832853+0000","last_change":"2024-04-16T16:15:15.437158+0000","last_active":"2024-04-16T16:15:36.832853+0000","last_peered":"2024-04-16T16:15:36.832853+0000","last_clean":"2024-04-16T16:15:36.832853+0000","last_became_active":"2024-04-16T16:15:15.435103+0000","last_became_peered":"2024-04-16T16:15:15.435103+0000","last_unstale":"2024-04-16T16:15:36.832853+0000","last_undegraded":"2024-04-16T16:15:36.832853+0000","last_fullsized":"2024-04-16T16:15:36.832853+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:34.312744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.836875+0000","last_change":"2024-04-16T16:15:35.829435+0000","last_active":"2024-04-16T16:15:35.836875+0000","last_peered":"2024-04-16T16:15:35.836875+0000","last_clean":"2024-04-16T16:15:35.836875+0000","last_became_active":"2024-04-16T16:15:35.829109+0000","last_became_peered":"2024-04-16T16:15:35.829109+0000","last_unstale":"2024-04-16T16:15:35.836875+0000","last_undegraded":"2024-04-16T16:15:35.836875+0000","last_fullsized":"2024-04-16T16:15:35.836875+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:55.103124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.824884+0000","last_change":"2024-04-16T16:15:15.453129+0000","last_active":"2024-04-16T16:15:35.824884+0000","last_peered":"2024-04-16T16:15:35.824884+0000","last_clean":"2024-04-16T16:15:35.824884+0000","last_became_active":"2024-04-16T16:15:15.452843+0000","last_became_peered":"2024-04-16T16:15:15.452843+0000","last_unstale":"2024-04-16T16:15:35.824884+0000","last_undegraded":"2024-04-16T16:15:35.824884+0000","last_fullsized":"2024-04-16T16:15:35.824884+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:28:11.918651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829566+0000","last_change":"2024-04-16T16:15:35.820622+0000","last_active":"2024-04-16T16:15:36.829566+0000","last_peered":"2024-04-16T16:15:36.829566+0000","last_clean":"2024-04-16T16:15:36.829566+0000","last_became_active":"2024-04-16T16:15:35.820294+0000","last_became_peered":"2024-04-16T16:15:35.820294+0000","last_unstale":"2024-04-16T16:15:36.829566+0000","last_undegraded":"2024-04-16T16:15:36.829566+0000","last_fullsized":"2024-04-16T16:15:36.829566+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:38:47.915574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.817939+0000","last_change":"2024-04-16T16:15:15.454894+0000","last_active":"2024-04-16T16:15:35.817939+0000","last_peered":"2024-04-16T16:15:35.817939+0000","last_clean":"2024-04-16T16:15:35.817939+0000","last_became_active":"2024-04-16T16:15:15.454659+0000","last_became_peered":"2024-04-16T16:15:15.454659+0000","last_unstale":"2024-04-16T16:15:35.817939+0000","last_undegraded":"2024-04-16T16:15:35.817939+0000","last_fullsized":"2024-04-16T16:15:35.817939+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:44.518137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.841425+0000","last_change":"2024-04-16T16:15:35.834489+0000","last_active":"2024-04-16T16:15:35.841425+0000","last_peered":"2024-04-16T16:15:35.841425+0000","last_clean":"2024-04-16T16:15:35.841425+0000","last_became_active":"2024-04-16T16:15:35.833330+0000","last_became_peered":"2024-04-16T16:15:35.833330+0000","last_unstale":"2024-04-16T16:15:35.841425+0000","last_undegraded":"2024-04-16T16:15:35.841425+0000","last_fullsized":"2024-04-16T16:15:35.841425+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:58.338850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.816103+0000","last_change":"2024-04-16T16:15:15.447202+0000","last_active":"2024-04-16T16:15:35.816103+0000","last_peered":"2024-04-16T16:15:35.816103+0000","last_clean":"2024-04-16T16:15:35.816103+0000","last_became_active":"2024-04-16T16:15:15.444539+0000","last_became_peered":"2024-04-16T16:15:15.444539+0000","last_unstale":"2024-04-16T16:15:35.816103+0000","last_undegraded":"2024-04-16T16:15:35.816103+0000","last_fullsized":"2024-04-16T16:15:35.816103+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:33:41.701252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.835169+0000","last_change":"2024-04-16T16:15:35.831665+0000","last_active":"2024-04-16T16:15:35.835169+0000","last_peered":"2024-04-16T16:15:35.835169+0000","last_clean":"2024-04-16T16:15:35.835169+0000","last_became_active":"2024-04-16T16:15:35.831062+0000","last_became_peered":"2024-04-16T16:15:35.831062+0000","last_unstale":"2024-04-16T16:15:35.835169+0000","last_undegraded":"2024-04-16T16:15:35.835169+0000","last_fullsized":"2024-04-16T16:15:35.835169+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:44:04.763428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.816820+0000","last_change":"2024-04-16T16:15:15.443351+0000","last_active":"2024-04-16T16:15:35.816820+0000","last_peered":"2024-04-16T16:15:35.816820+0000","last_clean":"2024-04-16T16:15:35.816820+0000","last_became_active":"2024-04-16T16:15:15.442366+0000","last_became_peered":"2024-04-16T16:15:15.442366+0000","last_unstale":"2024-04-16T16:15:35.816820+0000","last_undegraded":"2024-04-16T16:15:35.816820+0000","last_fullsized":"2024-04-16T16:15:35.816820+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:53.599226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.843523+0000","last_change":"2024-04-16T16:15:35.841197+0000","last_active":"2024-04-16T16:15:35.843523+0000","last_peered":"2024-04-16T16:15:35.843523+0000","last_clean":"2024-04-16T16:15:35.843523+0000","last_became_active":"2024-04-16T16:15:35.840934+0000","last_became_peered":"2024-04-16T16:15:35.840934+0000","last_unstale":"2024-04-16T16:15:35.843523+0000","last_undegraded":"2024-04-16T16:15:35.843523+0000","last_fullsized":"2024-04-16T16:15:35.843523+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:32:04.710682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811771+0000","last_change":"2024-04-16T16:15:15.462804+0000","last_active":"2024-04-16T16:15:35.811771+0000","last_peered":"2024-04-16T16:15:35.811771+0000","last_clean":"2024-04-16T16:15:35.811771+0000","last_became_active":"2024-04-16T16:15:15.461788+0000","last_became_peered":"2024-04-16T16:15:15.461788+0000","last_unstale":"2024-04-16T16:15:35.811771+0000","last_undegraded":"2024-04-16T16:15:35.811771+0000","last_fullsized":"2024-04-16T16:15:35.811771+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:12.276411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.841465+0000","last_change":"2024-04-16T16:15:35.829688+0000","last_active":"2024-04-16T16:15:35.841465+0000","last_peered":"2024-04-16T16:15:35.841465+0000","last_clean":"2024-04-16T16:15:35.841465+0000","last_became_active":"2024-04-16T16:15:35.828676+0000","last_became_peered":"2024-04-16T16:15:35.828676+0000","last_unstale":"2024-04-16T16:15:35.841465+0000","last_undegraded":"2024-04-16T16:15:35.841465+0000","last_fullsized":"2024-04-16T16:15:35.841465+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:15:06.383899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830237+0000","last_change":"2024-04-16T16:15:15.447751+0000","last_active":"2024-04-16T16:15:36.830237+0000","last_peered":"2024-04-16T16:15:36.830237+0000","last_clean":"2024-04-16T16:15:36.830237+0000","last_became_active":"2024-04-16T16:15:15.447195+0000","last_became_peered":"2024-04-16T16:15:15.447195+0000","last_unstale":"2024-04-16T16:15:36.830237+0000","last_undegraded":"2024-04-16T16:15:36.830237+0000","last_fullsized":"2024-04-16T16:15:36.830237+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:34:32.934841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830068+0000","last_change":"2024-04-16T16:15:35.831119+0000","last_active":"2024-04-16T16:15:36.830068+0000","last_peered":"2024-04-16T16:15:36.830068+0000","last_clean":"2024-04-16T16:15:36.830068+0000","last_became_active":"2024-04-16T16:15:35.830882+0000","last_became_peered":"2024-04-16T16:15:35.830882+0000","last_unstale":"2024-04-16T16:15:36.830068+0000","last_undegraded":"2024-04-16T16:15:36.830068+0000","last_fullsized":"2024-04-16T16:15:36.830068+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:16:18.955453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.825792+0000","last_change":"2024-04-16T16:15:15.456287+0000","last_active":"2024-04-16T16:15:35.825792+0000","last_peered":"2024-04-16T16:15:35.825792+0000","last_clean":"2024-04-16T16:15:35.825792+0000","last_became_active":"2024-04-16T16:15:15.455963+0000","last_became_peered":"2024-04-16T16:15:15.455963+0000","last_unstale":"2024-04-16T16:15:35.825792+0000","last_undegraded":"2024-04-16T16:15:35.825792+0000","last_fullsized":"2024-04-16T16:15:35.825792+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:07.752151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830334+0000","last_change":"2024-04-16T16:15:35.845320+0000","last_active":"2024-04-16T16:15:36.830334+0000","last_peered":"2024-04-16T16:15:36.830334+0000","last_clean":"2024-04-16T16:15:36.830334+0000","last_became_active":"2024-04-16T16:15:35.845067+0000","last_became_peered":"2024-04-16T16:15:35.845067+0000","last_unstale":"2024-04-16T16:15:36.830334+0000","last_undegraded":"2024-04-16T16:15:36.830334+0000","last_fullsized":"2024-04-16T16:15:36.830334+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:00:18.173149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.818527+0000","last_change":"2024-04-16T16:15:15.446449+0000","last_active":"2024-04-16T16:15:35.818527+0000","last_peered":"2024-04-16T16:15:35.818527+0000","last_clean":"2024-04-16T16:15:35.818527+0000","last_became_active":"2024-04-16T16:15:15.446135+0000","last_became_peered":"2024-04-16T16:15:15.446135+0000","last_unstale":"2024-04-16T16:15:35.818527+0000","last_undegraded":"2024-04-16T16:15:35.818527+0000","last_fullsized":"2024-04-16T16:15:35.818527+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:49.654384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.838886+0000","last_change":"2024-04-16T16:15:35.836251+0000","last_active":"2024-04-16T16:15:35.838886+0000","last_peered":"2024-04-16T16:15:35.838886+0000","last_clean":"2024-04-16T16:15:35.838886+0000","last_became_active":"2024-04-16T16:15:35.835956+0000","last_became_peered":"2024-04-16T16:15:35.835956+0000","last_unstale":"2024-04-16T16:15:35.838886+0000","last_undegraded":"2024-04-16T16:15:35.838886+0000","last_fullsized":"2024-04-16T16:15:35.838886+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:52:21.005662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.833029+0000","last_change":"2024-04-16T16:15:15.451682+0000","last_active":"2024-04-16T16:15:36.833029+0000","last_peered":"2024-04-16T16:15:36.833029+0000","last_clean":"2024-04-16T16:15:36.833029+0000","last_became_active":"2024-04-16T16:15:15.450999+0000","last_became_peered":"2024-04-16T16:15:15.450999+0000","last_unstale":"2024-04-16T16:15:36.833029+0000","last_undegraded":"2024-04-16T16:15:36.833029+0000","last_fullsized":"2024-04-16T16:15:36.833029+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:01:20.716612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.835205+0000","last_change":"2024-04-16T16:15:35.832685+0000","last_active":"2024-04-16T16:15:35.835205+0000","last_peered":"2024-04-16T16:15:35.835205+0000","last_clean":"2024-04-16T16:15:35.835205+0000","last_became_active":"2024-04-16T16:15:35.832226+0000","last_became_peered":"2024-04-16T16:15:35.832226+0000","last_unstale":"2024-04-16T16:15:35.835205+0000","last_undegraded":"2024-04-16T16:15:35.835205+0000","last_fullsized":"2024-04-16T16:15:35.835205+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:21.250937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.833032+0000","last_change":"2024-04-16T16:15:15.445583+0000","last_active":"2024-04-16T16:15:36.833032+0000","last_peered":"2024-04-16T16:15:36.833032+0000","last_clean":"2024-04-16T16:15:36.833032+0000","last_became_active":"2024-04-16T16:15:15.445259+0000","last_became_peered":"2024-04-16T16:15:15.445259+0000","last_unstale":"2024-04-16T16:15:36.833032+0000","last_undegraded":"2024-04-16T16:15:36.833032+0000","last_fullsized":"2024-04-16T16:15:36.833032+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:56.074330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830022+0000","last_change":"2024-04-16T16:15:35.844725+0000","last_active":"2024-04-16T16:15:36.830022+0000","last_peered":"2024-04-16T16:15:36.830022+0000","last_clean":"2024-04-16T16:15:36.830022+0000","last_became_active":"2024-04-16T16:15:35.844257+0000","last_became_peered":"2024-04-16T16:15:35.844257+0000","last_unstale":"2024-04-16T16:15:36.830022+0000","last_undegraded":"2024-04-16T16:15:36.830022+0000","last_fullsized":"2024-04-16T16:15:36.830022+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:16:41.449544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"46'2","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.815986+0000","last_change":"2024-04-16T16:15:15.447460+0000","last_active":"2024-04-16T16:15:35.815986+0000","last_peered":"2024-04-16T16:15:35.815986+0000","last_clean":"2024-04-16T16:15:35.815986+0000","last_became_active":"2024-04-16T16:15:15.447155+0000","last_became_peered":"2024-04-16T16:15:15.447155+0000","last_unstale":"2024-04-16T16:15:35.815986+0000","last_undegraded":"2024-04-16T16:15:35.815986+0000","last_fullsized":"2024-04-16T16:15:35.815986+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:48:05.645437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.835386+0000","last_change":"2024-04-16T16:15:35.831883+0000","last_active":"2024-04-16T16:15:35.835386+0000","last_peered":"2024-04-16T16:15:35.835386+0000","last_clean":"2024-04-16T16:15:35.835386+0000","last_became_active":"2024-04-16T16:15:35.831376+0000","last_became_peered":"2024-04-16T16:15:35.831376+0000","last_unstale":"2024-04-16T16:15:35.835386+0000","last_undegraded":"2024-04-16T16:15:35.835386+0000","last_fullsized":"2024-04-16T16:15:35.835386+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:28:41.133302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829768+0000","last_change":"2024-04-16T16:15:15.448356+0000","last_active":"2024-04-16T16:15:36.829768+0000","last_peered":"2024-04-16T16:15:36.829768+0000","last_clean":"2024-04-16T16:15:36.829768+0000","last_became_active":"2024-04-16T16:15:15.448056+0000","last_became_peered":"2024-04-16T16:15:15.448056+0000","last_unstale":"2024-04-16T16:15:36.829768+0000","last_undegraded":"2024-04-16T16:15:36.829768+0000","last_fullsized":"2024-04-16T16:15:36.829768+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:30:21.270376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.842185+0000","last_change":"2024-04-16T16:15:35.836622+0000","last_active":"2024-04-16T16:15:35.842185+0000","last_peered":"2024-04-16T16:15:35.842185+0000","last_clean":"2024-04-16T16:15:35.842185+0000","last_became_active":"2024-04-16T16:15:35.836124+0000","last_became_peered":"2024-04-16T16:15:35.836124+0000","last_unstale":"2024-04-16T16:15:35.842185+0000","last_undegraded":"2024-04-16T16:15:35.842185+0000","last_fullsized":"2024-04-16T16:15:35.842185+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:38:47.915574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.810899+0000","last_change":"2024-04-16T16:15:15.459411+0000","last_active":"2024-04-16T16:15:35.810899+0000","last_peered":"2024-04-16T16:15:35.810899+0000","last_clean":"2024-04-16T16:15:35.810899+0000","last_became_active":"2024-04-16T16:15:15.452989+0000","last_became_peered":"2024-04-16T16:15:15.452989+0000","last_unstale":"2024-04-16T16:15:35.810899+0000","last_undegraded":"2024-04-16T16:15:35.810899+0000","last_fullsized":"2024-04-16T16:15:35.810899+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:27.522215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831873+0000","last_change":"2024-04-16T16:15:35.829883+0000","last_active":"2024-04-16T16:15:36.831873+0000","last_peered":"2024-04-16T16:15:36.831873+0000","last_clean":"2024-04-16T16:15:36.831873+0000","last_became_active":"2024-04-16T16:15:35.829096+0000","last_became_peered":"2024-04-16T16:15:35.829096+0000","last_unstale":"2024-04-16T16:15:36.831873+0000","last_undegraded":"2024-04-16T16:15:36.831873+0000","last_fullsized":"2024-04-16T16:15:36.831873+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:36:28.560037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":26,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.825192+0000","last_change":"2024-04-16T16:15:15.441331+0000","last_active":"2024-04-16T16:15:35.825192+0000","last_peered":"2024-04-16T16:15:35.825192+0000","last_clean":"2024-04-16T16:15:35.825192+0000","last_became_active":"2024-04-16T16:15:15.440983+0000","last_became_peered":"2024-04-16T16:15:15.440983+0000","last_unstale":"2024-04-16T16:15:35.825192+0000","last_undegraded":"2024-04-16T16:15:35.825192+0000","last_fullsized":"2024-04-16T16:15:35.825192+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:10:59.131225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.847638+0000","last_change":"2024-04-16T16:15:35.839790+0000","last_active":"2024-04-16T16:15:35.847638+0000","last_peered":"2024-04-16T16:15:35.847638+0000","last_clean":"2024-04-16T16:15:35.847638+0000","last_became_active":"2024-04-16T16:15:35.839457+0000","last_became_peered":"2024-04-16T16:15:35.839457+0000","last_unstale":"2024-04-16T16:15:35.847638+0000","last_undegraded":"2024-04-16T16:15:35.847638+0000","last_fullsized":"2024-04-16T16:15:35.847638+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:40:24.612282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":25,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.821980+0000","last_change":"2024-04-16T16:15:15.449849+0000","last_active":"2024-04-16T16:15:35.821980+0000","last_peered":"2024-04-16T16:15:35.821980+0000","last_clean":"2024-04-16T16:15:35.821980+0000","last_became_active":"2024-04-16T16:15:15.449585+0000","last_became_peered":"2024-04-16T16:15:15.449585+0000","last_unstale":"2024-04-16T16:15:35.821980+0000","last_undegraded":"2024-04-16T16:15:35.821980+0000","last_fullsized":"2024-04-16T16:15:35.821980+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:12.276411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.842576+0000","last_change":"2024-04-16T16:15:35.839438+0000","last_active":"2024-04-16T16:15:35.842576+0000","last_peered":"2024-04-16T16:15:35.842576+0000","last_clean":"2024-04-16T16:15:35.842576+0000","last_became_active":"2024-04-16T16:15:35.839164+0000","last_became_peered":"2024-04-16T16:15:35.839164+0000","last_unstale":"2024-04-16T16:15:35.842576+0000","last_undegraded":"2024-04-16T16:15:35.842576+0000","last_fullsized":"2024-04-16T16:15:35.842576+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:37.583592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811485+0000","last_change":"2024-04-16T16:15:15.463054+0000","last_active":"2024-04-16T16:15:35.811485+0000","last_peered":"2024-04-16T16:15:35.811485+0000","last_clean":"2024-04-16T16:15:35.811485+0000","last_became_active":"2024-04-16T16:15:15.462075+0000","last_became_peered":"2024-04-16T16:15:15.462075+0000","last_unstale":"2024-04-16T16:15:35.811485+0000","last_undegraded":"2024-04-16T16:15:35.811485+0000","last_fullsized":"2024-04-16T16:15:35.811485+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:56.074330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829106+0000","last_change":"2024-04-16T16:15:35.848184+0000","last_active":"2024-04-16T16:15:36.829106+0000","last_peered":"2024-04-16T16:15:36.829106+0000","last_clean":"2024-04-16T16:15:36.829106+0000","last_became_active":"2024-04-16T16:15:35.847403+0000","last_became_peered":"2024-04-16T16:15:35.847403+0000","last_unstale":"2024-04-16T16:15:36.829106+0000","last_undegraded":"2024-04-16T16:15:36.829106+0000","last_fullsized":"2024-04-16T16:15:36.829106+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:39:29.875935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811338+0000","last_change":"2024-04-16T16:15:15.459905+0000","last_active":"2024-04-16T16:15:35.811338+0000","last_peered":"2024-04-16T16:15:35.811338+0000","last_clean":"2024-04-16T16:15:35.811338+0000","last_became_active":"2024-04-16T16:15:15.459387+0000","last_became_peered":"2024-04-16T16:15:15.459387+0000","last_unstale":"2024-04-16T16:15:35.811338+0000","last_undegraded":"2024-04-16T16:15:35.811338+0000","last_fullsized":"2024-04-16T16:15:35.811338+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:51.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830953+0000","last_change":"2024-04-16T16:15:35.840856+0000","last_active":"2024-04-16T16:15:36.830953+0000","last_peered":"2024-04-16T16:15:36.830953+0000","last_clean":"2024-04-16T16:15:36.830953+0000","last_became_active":"2024-04-16T16:15:35.840559+0000","last_became_peered":"2024-04-16T16:15:35.840559+0000","last_unstale":"2024-04-16T16:15:36.830953+0000","last_undegraded":"2024-04-16T16:15:36.830953+0000","last_fullsized":"2024-04-16T16:15:36.830953+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:49:48.573089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.811068+0000","last_change":"2024-04-16T16:15:15.451403+0000","last_active":"2024-04-16T16:15:35.811068+0000","last_peered":"2024-04-16T16:15:35.811068+0000","last_clean":"2024-04-16T16:15:35.811068+0000","last_became_active":"2024-04-16T16:15:15.450199+0000","last_became_peered":"2024-04-16T16:15:15.450199+0000","last_unstale":"2024-04-16T16:15:35.811068+0000","last_undegraded":"2024-04-16T16:15:35.811068+0000","last_fullsized":"2024-04-16T16:15:35.811068+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:32:52.123986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.841012+0000","last_change":"2024-04-16T16:15:35.833202+0000","last_active":"2024-04-16T16:15:35.841012+0000","last_peered":"2024-04-16T16:15:35.841012+0000","last_clean":"2024-04-16T16:15:35.841012+0000","last_became_active":"2024-04-16T16:15:35.832825+0000","last_became_peered":"2024-04-16T16:15:35.832825+0000","last_unstale":"2024-04-16T16:15:35.841012+0000","last_undegraded":"2024-04-16T16:15:35.841012+0000","last_fullsized":"2024-04-16T16:15:35.841012+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:12.521080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.804499+0000","last_change":"2024-04-16T16:15:15.463187+0000","last_active":"2024-04-16T16:15:35.804499+0000","last_peered":"2024-04-16T16:15:35.804499+0000","last_clean":"2024-04-16T16:15:35.804499+0000","last_became_active":"2024-04-16T16:15:15.461753+0000","last_became_peered":"2024-04-16T16:15:15.461753+0000","last_unstale":"2024-04-16T16:15:35.804499+0000","last_undegraded":"2024-04-16T16:15:35.804499+0000","last_fullsized":"2024-04-16T16:15:35.804499+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:00:44.974637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.833127+0000","last_change":"2024-04-16T16:15:35.835649+0000","last_active":"2024-04-16T16:15:36.833127+0000","last_peered":"2024-04-16T16:15:36.833127+0000","last_clean":"2024-04-16T16:15:36.833127+0000","last_became_active":"2024-04-16T16:15:35.835102+0000","last_became_peered":"2024-04-16T16:15:35.835102+0000","last_unstale":"2024-04-16T16:15:36.833127+0000","last_undegraded":"2024-04-16T16:15:36.833127+0000","last_fullsized":"2024-04-16T16:15:36.833127+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:06:12.259732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.829806+0000","last_change":"2024-04-16T16:15:15.439872+0000","last_active":"2024-04-16T16:15:36.829806+0000","last_peered":"2024-04-16T16:15:36.829806+0000","last_clean":"2024-04-16T16:15:36.829806+0000","last_became_active":"2024-04-16T16:15:15.439553+0000","last_became_peered":"2024-04-16T16:15:15.439553+0000","last_unstale":"2024-04-16T16:15:36.829806+0000","last_undegraded":"2024-04-16T16:15:36.829806+0000","last_fullsized":"2024-04-16T16:15:36.829806+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:12.276411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.841875+0000","last_change":"2024-04-16T16:15:35.836818+0000","last_active":"2024-04-16T16:15:35.841875+0000","last_peered":"2024-04-16T16:15:35.841875+0000","last_clean":"2024-04-16T16:15:35.841875+0000","last_became_active":"2024-04-16T16:15:35.836499+0000","last_became_peered":"2024-04-16T16:15:35.836499+0000","last_unstale":"2024-04-16T16:15:35.841875+0000","last_undegraded":"2024-04-16T16:15:35.841875+0000","last_fullsized":"2024-04-16T16:15:35.841875+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:42:41.959591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":26,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.830173+0000","last_change":"2024-04-16T16:15:15.429037+0000","last_active":"2024-04-16T16:15:36.830173+0000","last_peered":"2024-04-16T16:15:36.830173+0000","last_clean":"2024-04-16T16:15:36.830173+0000","last_became_active":"2024-04-16T16:15:15.425570+0000","last_became_peered":"2024-04-16T16:15:15.425570+0000","last_unstale":"2024-04-16T16:15:36.830173+0000","last_undegraded":"2024-04-16T16:15:36.830173+0000","last_fullsized":"2024-04-16T16:15:36.830173+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:17:00.492572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.850644+0000","last_change":"2024-04-16T16:15:35.849428+0000","last_active":"2024-04-16T16:15:35.850644+0000","last_peered":"2024-04-16T16:15:35.850644+0000","last_clean":"2024-04-16T16:15:35.850644+0000","last_became_active":"2024-04-16T16:15:35.849131+0000","last_became_peered":"2024-04-16T16:15:35.849131+0000","last_unstale":"2024-04-16T16:15:35.850644+0000","last_undegraded":"2024-04-16T16:15:35.850644+0000","last_fullsized":"2024-04-16T16:15:35.850644+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:25:58.997894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":25,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.822087+0000","last_change":"2024-04-16T16:15:15.443526+0000","last_active":"2024-04-16T16:15:35.822087+0000","last_peered":"2024-04-16T16:15:35.822087+0000","last_clean":"2024-04-16T16:15:35.822087+0000","last_became_active":"2024-04-16T16:15:15.443203+0000","last_became_peered":"2024-04-16T16:15:15.443203+0000","last_unstale":"2024-04-16T16:15:35.822087+0000","last_undegraded":"2024-04-16T16:15:35.822087+0000","last_fullsized":"2024-04-16T16:15:35.822087+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:42.434776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831653+0000","last_change":"2024-04-16T16:15:35.836249+0000","last_active":"2024-04-16T16:15:36.831653+0000","last_peered":"2024-04-16T16:15:36.831653+0000","last_clean":"2024-04-16T16:15:36.831653+0000","last_became_active":"2024-04-16T16:15:35.835942+0000","last_became_peered":"2024-04-16T16:15:35.835942+0000","last_unstale":"2024-04-16T16:15:36.831653+0000","last_undegraded":"2024-04-16T16:15:36.831653+0000","last_fullsized":"2024-04-16T16:15:36.831653+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:49:31.820963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":24,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.813048+0000","last_change":"2024-04-16T16:15:15.459192+0000","last_active":"2024-04-16T16:15:35.813048+0000","last_peered":"2024-04-16T16:15:35.813048+0000","last_clean":"2024-04-16T16:15:35.813048+0000","last_became_active":"2024-04-16T16:15:15.452689+0000","last_became_peered":"2024-04-16T16:15:15.452689+0000","last_unstale":"2024-04-16T16:15:35.813048+0000","last_undegraded":"2024-04-16T16:15:35.813048+0000","last_fullsized":"2024-04-16T16:15:35.813048+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:39:57.997770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.836088+0000","last_change":"2024-04-16T16:15:35.830836+0000","last_active":"2024-04-16T16:15:35.836088+0000","last_peered":"2024-04-16T16:15:35.836088+0000","last_clean":"2024-04-16T16:15:35.836088+0000","last_became_active":"2024-04-16T16:15:35.830543+0000","last_became_peered":"2024-04-16T16:15:35.830543+0000","last_unstale":"2024-04-16T16:15:35.836088+0000","last_undegraded":"2024-04-16T16:15:35.836088+0000","last_fullsized":"2024-04-16T16:15:35.836088+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:32.059528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":27,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.832729+0000","last_change":"2024-04-16T16:15:15.454071+0000","last_active":"2024-04-16T16:15:36.832729+0000","last_peered":"2024-04-16T16:15:36.832729+0000","last_clean":"2024-04-16T16:15:36.832729+0000","last_became_active":"2024-04-16T16:15:15.453781+0000","last_became_peered":"2024-04-16T16:15:15.453781+0000","last_unstale":"2024-04-16T16:15:36.832729+0000","last_undegraded":"2024-04-16T16:15:36.832729+0000","last_fullsized":"2024-04-16T16:15:36.832729+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:03.163739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.838404+0000","last_change":"2024-04-16T16:15:35.831480+0000","last_active":"2024-04-16T16:15:35.838404+0000","last_peered":"2024-04-16T16:15:35.838404+0000","last_clean":"2024-04-16T16:15:35.838404+0000","last_became_active":"2024-04-16T16:15:35.831022+0000","last_became_peered":"2024-04-16T16:15:35.831022+0000","last_unstale":"2024-04-16T16:15:35.838404+0000","last_undegraded":"2024-04-16T16:15:35.838404+0000","last_fullsized":"2024-04-16T16:15:35.838404+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:38:47.915574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":25,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-16T16:15:35.822994+0000","last_change":"2024-04-16T16:15:15.451103+0000","last_active":"2024-04-16T16:15:35.822994+0000","last_peered":"2024-04-16T16:15:35.822994+0000","last_clean":"2024-04-16T16:15:35.822994+0000","last_became_active":"2024-04-16T16:15:15.450511+0000","last_became_peered":"2024-04-16T16:15:15.450511+0000","last_unstale":"2024-04-16T16:15:35.822994+0000","last_undegraded":"2024-04-16T16:15:35.822994+0000","last_fullsized":"2024-04-16T16:15:35.822994+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:51:34.706031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831345+0000","last_change":"2024-04-16T16:15:35.827011+0000","last_active":"2024-04-16T16:15:36.831345+0000","last_peered":"2024-04-16T16:15:36.831345+0000","last_clean":"2024-04-16T16:15:36.831345+0000","last_became_active":"2024-04-16T16:15:35.826042+0000","last_became_peered":"2024-04-16T16:15:35.826042+0000","last_unstale":"2024-04-16T16:15:36.831345+0000","last_undegraded":"2024-04-16T16:15:36.831345+0000","last_fullsized":"2024-04-16T16:15:36.831345+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:34.774122+0000","last_clean_scrub_stamp":"2024-04-16T16:15:34.774122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:58.025643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3f","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-16T16:15:36.831423+0000","last_change":"2024-04-16T16:15:15.445346+0000","last_active":"2024-04-16T16:15:36.831423+0000","last_peered":"2024-04-16T16:15:36.831423+0000","last_clean":"2024-04-16T16:15:36.831423+0000","last_became_active":"2024-04-16T16:15:15.445023+0000","last_became_peered":"2024-04-16T16:15:15.445023+0000","last_unstale":"2024-04-16T16:15:36.831423+0000","last_undegraded":"2024-04-16T16:15:36.831423+0000","last_fullsized":"2024-04-16T16:15:36.831423+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:15:14.377029+0000","last_clean_scrub_stamp":"2024-04-16T16:15:14.377029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:51.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":3145728,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":14232,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":77,"num_read_kb":80,"num_write":52,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":28464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51,"ondisk_log_size":51,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28184,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343656,"statfs":{"total":96636764160,"available":96607903744,"internally_reserved":0,"allocated":1183744,"data_stored":841262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30668,"kb_used_data":3644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341172,"statfs":{"total":96636764160,"available":96605360128,"internally_reserved":0,"allocated":3731456,"data_stored":3397044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":589824,"data_stored":251366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":626688,"data_stored":273245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":610304,"data_stored":258893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":614400,"data_stored":264156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574878,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31756,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340084,"statfs":{"total":96636764160,"available":96604246016,"internally_reserved":0,"allocated":585728,"data_stored":250804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35424,"kb_used_data":4236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336416,"statfs":{"total":96636764160,"available":96600489984,"internally_reserved":0,"allocated":4337664,"data_stored":3988968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":13456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":13434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3145728,"data_stored":3145728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3145728,"data_stored":3145728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:15:40.963 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-16T16:15:40.963 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-16T16:15:40.963 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-16T16:15:40.963 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-16T16:15:40.963 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:15:40.963 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-16T16:15:40.963 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-16T16:15:40.964 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-16T16:15:40.964 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:15:40.964 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-16T16:15:40.964 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-16T16:15:40.964 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-16T16:15:40.964 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:15:40.964 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-16T16:15:40.964 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-16T16:15:40.965 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-16T16:15:40.965 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:15:40.985 INFO:tasks.ceph.mds.b:Stopped 2024-04-16T16:15:40.985 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-16T16:15:40.993 INFO:tasks.ceph.mds.d:Stopped 2024-04-16T16:15:40.993 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-16T16:15:41.016 INFO:tasks.ceph.mds.c:Stopped 2024-04-16T16:15:41.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-16T16:15:41.023 INFO:tasks.ceph.mds.a:Stopped 2024-04-16T16:15:41.023 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-16T16:15:41.039 INFO:tasks.ceph.mds.b:Started 2024-04-16T16:15:41.039 DEBUG:teuthology.parallel:result is None 2024-04-16T16:15:41.045 INFO:tasks.ceph.mds.d:Started 2024-04-16T16:15:41.046 DEBUG:teuthology.parallel:result is None 2024-04-16T16:15:41.069 INFO:tasks.ceph.mds.c:Started 2024-04-16T16:15:41.070 DEBUG:teuthology.parallel:result is None 2024-04-16T16:15:41.076 INFO:tasks.ceph.mds.a:Started 2024-04-16T16:15:41.076 DEBUG:teuthology.parallel:result is None 2024-04-16T16:15:41.077 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:15:41.212 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-16T16:15:41.212 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-16T16:15:41.220 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-04-16T16:15:41.220+0000 7fe7a4800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T16:15:41.240 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2024-04-16T16:15:41.242 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2024-04-16T16:15:41.255 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-04-16T16:15:41.255+0000 7fe7a4800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T16:15:41.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.291+0000 7f6cd0e00640 1 Processor -- start 2024-04-16T16:15:41.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.291+0000 7f6cd0e00640 1 -- start start 2024-04-16T16:15:41.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.292+0000 7f6cd0e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6ccc0aed90 0x7f6ccc072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:41.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.292+0000 7f6cd0e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ccc147c20 con 0x7f6ccc147840 2024-04-16T16:15:41.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.292+0000 7f6cd0e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ccc160210 con 0x7f6ccc0728c0 2024-04-16T16:15:41.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.292+0000 7f6cd0e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6ccc072d10 con 0x7f6ccc0aed90 2024-04-16T16:15:41.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.293+0000 7f6cc9e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6ccc147840 0x7f6ccc15fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58648/0 (socket says 172.21.15.45:58648) 2024-04-16T16:15:41.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.293+0000 7f6cc9e00640 1 -- 172.21.15.45:0/4128115116 learned_addr learned my addr 172.21.15.45:0/4128115116 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:41.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.293+0000 7f6cc9400640 1 --2- 172.21.15.45:0/4128115116 >> v2:172.21.15.103:3300/0 conn(0x7f6ccc0aed90 0x7f6ccc072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:41.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.293+0000 7f6cc8a00640 1 -- 172.21.15.45:0/4128115116 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3696723403 0 0) 0x7f6ccc147c20 con 0x7f6ccc147840 2024-04-16T16:15:41.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.293+0000 7f6cc8a00640 1 -- 172.21.15.45:0/4128115116 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cb0003520 con 0x7f6ccc147840 2024-04-16T16:15:41.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.293+0000 7f6cc9400640 1 -- 172.21.15.45:0/4128115116 >> v1:172.21.15.103:6789/0 conn(0x7f6ccc0728c0 legacy=0x7f6ccc06e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:41.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.293+0000 7f6cc9400640 1 -- 172.21.15.45:0/4128115116 >> v1:172.21.15.45:6789/0 conn(0x7f6ccc147840 legacy=0x7f6ccc15fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:41.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.293+0000 7f6cc9400640 1 -- 172.21.15.45:0/4128115116 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ccc0af270 con 0x7f6ccc0aed90 2024-04-16T16:15:41.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.295+0000 7f6cc9400640 1 --2- 172.21.15.45:0/4128115116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ccc0aed90 0x7f6ccc072380 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6cb80097a0 tx=0x7f6cb802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6a80ed24c2cc8e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:41.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.295+0000 7f6cc8a00640 1 -- 172.21.15.45:0/4128115116 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8034070 con 0x7f6ccc0aed90 2024-04-16T16:15:41.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.295+0000 7f6cc8a00640 1 -- 172.21.15.45:0/4128115116 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cb8030610 con 0x7f6ccc0aed90 2024-04-16T16:15:41.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.295+0000 7f6cc8a00640 1 -- 172.21.15.45:0/4128115116 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb80309e0 con 0x7f6ccc0aed90 2024-04-16T16:15:41.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.295+0000 7f6cd0e00640 1 -- 172.21.15.45:0/4128115116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ccc0aed90 msgr2=0x7f6ccc072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:41.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.296+0000 7f6cd0e00640 1 --2- 172.21.15.45:0/4128115116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ccc0aed90 0x7f6ccc072380 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6cb80097a0 tx=0x7f6cb802b960 comp rx=0 tx=0).stop 2024-04-16T16:15:41.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.296+0000 7f6cd0e00640 1 -- 172.21.15.45:0/4128115116 shutdown_connections 2024-04-16T16:15:41.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.296+0000 7f6cd0e00640 1 --2- 172.21.15.45:0/4128115116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ccc0aed90 0x7f6ccc072380 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:41.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.296+0000 7f6cd0e00640 1 -- 172.21.15.45:0/4128115116 >> 172.21.15.45:0/4128115116 conn(0x7f6ccc0b2750 msgr2=0x7f6ccc0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:41.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.296+0000 7f6cd0e00640 1 -- 172.21.15.45:0/4128115116 shutdown_connections 2024-04-16T16:15:41.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.296+0000 7f6cd0e00640 1 -- 172.21.15.45:0/4128115116 wait complete. 2024-04-16T16:15:41.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.298+0000 7f6cd0e00640 1 Processor -- start 2024-04-16T16:15:41.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.298+0000 7f6cd0e00640 1 -- start start 2024-04-16T16:15:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cd0e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ccc0728c0 0x7f6ccc145210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cd0e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ccc0aed90 0x7f6ccc1629d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cd0e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ccc147840 0x7f6ccc140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cd0e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6ccc0af830 con 0x7f6ccc147840 2024-04-16T16:15:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cd0e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6ccc0afe80 con 0x7f6ccc0aed90 2024-04-16T16:15:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cd0e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6ccc0afb10 con 0x7f6ccc0728c0 2024-04-16T16:15:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cca800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ccc147840 0x7f6ccc140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cca800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ccc147840 0x7f6ccc140ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45868/0 (socket says 172.21.15.45:45868) 2024-04-16T16:15:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cca800640 1 -- 172.21.15.45:0/35543005 learned_addr learned my addr 172.21.15.45:0/35543005 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cc9e00640 1 --2- 172.21.15.45:0/35543005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ccc0728c0 0x7f6ccc145210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cc9400640 1 --2- 172.21.15.45:0/35543005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ccc0aed90 0x7f6ccc1629d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cca800640 1 -- 172.21.15.45:0/35543005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ccc0728c0 msgr2=0x7f6ccc145210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cca800640 1 --2- 172.21.15.45:0/35543005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ccc0728c0 0x7f6ccc145210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:41.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cca800640 1 -- 172.21.15.45:0/35543005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ccc0aed90 msgr2=0x7f6ccc1629d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:41.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cca800640 1 --2- 172.21.15.45:0/35543005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ccc0aed90 0x7f6ccc1629d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:41.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cca800640 1 -- 172.21.15.45:0/35543005 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ccc1445e0 con 0x7f6ccc147840 2024-04-16T16:15:41.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.299+0000 7f6cc9400640 1 --2- 172.21.15.45:0/35543005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ccc0aed90 0x7f6ccc1629d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:15:41.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.300+0000 7f6cca800640 1 --2- 172.21.15.45:0/35543005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ccc147840 0x7f6ccc140ed0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400c7a0 tx=0x7f6cc400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.300+0000 7f6cc2000640 1 -- 172.21.15.45:0/35543005 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4003b70 con 0x7f6ccc147840 2024-04-16T16:15:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.300+0000 7f6cc2000640 1 -- 172.21.15.45:0/35543005 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc4003d30 con 0x7f6ccc147840 2024-04-16T16:15:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.300+0000 7f6cc2000640 1 -- 172.21.15.45:0/35543005 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc400aa90 con 0x7f6ccc147840 2024-04-16T16:15:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.300+0000 7f6cd0e00640 1 -- 172.21.15.45:0/35543005 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ccc1448c0 con 0x7f6ccc147840 2024-04-16T16:15:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.300+0000 7f6cd0e00640 1 -- 172.21.15.45:0/35543005 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ccc160640 con 0x7f6ccc147840 2024-04-16T16:15:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.301+0000 7f6cd0e00640 1 -- 172.21.15.45:0/35543005 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ccc075780 con 0x7f6ccc147840 2024-04-16T16:15:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.302+0000 7f6cc2000640 1 -- 172.21.15.45:0/35543005 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f6cc400ac30 con 0x7f6ccc147840 2024-04-16T16:15:41.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.303+0000 7f6cc2000640 1 --2- 172.21.15.45:0/35543005 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6cac06ca60 0x7f6cac06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:41.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.303+0000 7f6cc2000640 1 -- 172.21.15.45:0/35543005 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7f6cc408c1c0 con 0x7f6ccc147840 2024-04-16T16:15:41.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.304+0000 7f6cc9e00640 1 --2- 172.21.15.45:0/35543005 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6cac06ca60 0x7f6cac06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:41.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.304+0000 7f6cc9e00640 1 --2- 172.21.15.45:0/35543005 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6cac06ca60 0x7f6cac06ef20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc0011e0 tx=0x7f6cbc007d30 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:41.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.308+0000 7f6cc2000640 1 -- 172.21.15.45:0/35543005 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6cc4059eb0 con 0x7f6ccc147840 2024-04-16T16:15:41.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.544+0000 7f6cd0e00640 1 -- 172.21.15.45:0/35543005 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ccc0d0710 con 0x7f6ccc147840 2024-04-16T16:15:41.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.545+0000 7f6cc2000640 1 -- 172.21.15.45:0/35543005 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6477 (secure 0 0 0) 0x7f6cc4060080 con 0x7f6ccc147840 2024-04-16T16:15:41.545 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:41.546 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/1497040472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1497040472},{"type":"v1","addr":"172.21.15.45:6834","nonce":1497040472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":4907,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/93057991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":93057991},{"type":"v1","addr":"172.21.15.45:6835","nonce":93057991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:14.878986+0000","modified":"2024-04-16T16:15:41.262942+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5188":{"gid":5188,"name":"d","rank":0,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.103:6836/2626302738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2626302738},{"type":"v1","addr":"172.21.15.103:6836","nonce":2626302738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"fs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:35.289901+0000","modified":"2024-04-16T16:15:41.227141+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"b","rank":0,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.103:6837/1576112884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1576112884},{"type":"v1","addr":"172.21.15.103:6837","nonce":1576112884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"fs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":4}]} 2024-04-16T16:15:41.546 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 37 2024-04-16T16:15:41.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.548+0000 7f6cd0e00640 1 -- 172.21.15.45:0/35543005 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6cac06ca60 msgr2=0x7f6cac06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:41.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.548+0000 7f6cd0e00640 1 --2- 172.21.15.45:0/35543005 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6cac06ca60 0x7f6cac06ef20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc0011e0 tx=0x7f6cbc007d30 comp rx=0 tx=0).stop 2024-04-16T16:15:41.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.548+0000 7f6cd0e00640 1 -- 172.21.15.45:0/35543005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ccc147840 msgr2=0x7f6ccc140ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:41.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.548+0000 7f6cd0e00640 1 --2- 172.21.15.45:0/35543005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ccc147840 0x7f6ccc140ed0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400c7a0 tx=0x7f6cc400cc60 comp rx=0 tx=0).stop 2024-04-16T16:15:41.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.548+0000 7f6cd0e00640 1 -- 172.21.15.45:0/35543005 shutdown_connections 2024-04-16T16:15:41.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.548+0000 7f6cd0e00640 1 --2- 172.21.15.45:0/35543005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ccc147840 0x7f6ccc140ed0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:41.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.548+0000 7f6cd0e00640 1 --2- 172.21.15.45:0/35543005 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6cac06ca60 0x7f6cac06ef20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:41.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.548+0000 7f6cd0e00640 1 --2- 172.21.15.45:0/35543005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ccc0aed90 0x7f6ccc1629d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:41.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.548+0000 7f6cd0e00640 1 --2- 172.21.15.45:0/35543005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ccc0728c0 0x7f6ccc145210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:41.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.548+0000 7f6cd0e00640 1 -- 172.21.15.45:0/35543005 >> 172.21.15.45:0/35543005 conn(0x7f6ccc0b2750 msgr2=0x7f6ccc141fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:41.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.548+0000 7f6cd0e00640 1 -- 172.21.15.45:0/35543005 shutdown_connections 2024-04-16T16:15:41.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:41.548+0000 7f6cd0e00640 1 -- 172.21.15.45:0/35543005 wait complete. 2024-04-16T16:15:41.967 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:15:35.289901+0000', 'modified': '2024-04-16T16:15:41.227141+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5191}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5191': {'gid': 5191, 'name': 'b', 'rank': 0, 'incarnation': 35, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.103:6837/1576112884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6835', 'nonce': 1576112884}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 1576112884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'fs2', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-16T16:15:41.967 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5191:up:replay 2024-04-16T16:15:42.234 INFO:tasks.ceph.mgr.x.smithi103.stderr:Exception in thread Thread-18: 2024-04-16T16:15:42.234 INFO:tasks.ceph.mgr.x.smithi103.stderr:Traceback (most recent call last): 2024-04-16T16:15:42.235 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner 2024-04-16T16:15:42.235 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.run() 2024-04-16T16:15:42.235 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 1306, in run 2024-04-16T16:15:42.235 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.function(*self.args, **self.kwargs) 2024-04-16T16:15:42.235 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/share/ceph/mgr/stats/fs/perf_stats.py", line 222, in re_register_queries 2024-04-16T16:15:42.235 INFO:tasks.ceph.mgr.x.smithi103.stderr: if self.mx_last_updated >= ua_last_updated: 2024-04-16T16:15:42.235 INFO:tasks.ceph.mgr.x.smithi103.stderr:AttributeError: 'FSPerfStats' object has no attribute 'mx_last_updated' 2024-04-16T16:15:42.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:15:43.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.202+0000 7f5843200640 1 Processor -- start 2024-04-16T16:15:43.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.203+0000 7f5843200640 1 -- start start 2024-04-16T16:15:43.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.204+0000 7f5843200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f583c0d52d0 0x7f583c06ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:43.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.204+0000 7f5843200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f583c0d3080 con 0x7f583c162270 2024-04-16T16:15:43.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.204+0000 7f5843200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f583c0d3360 con 0x7f583c161e90 2024-04-16T16:15:43.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.204+0000 7f5843200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f583c0d2e30 con 0x7f583c0d52d0 2024-04-16T16:15:43.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.204+0000 7f5841400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f583c162270 0x7f583c0d3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58652/0 (socket says 172.21.15.45:58652) 2024-04-16T16:15:43.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.204+0000 7f5841400640 1 -- 172.21.15.45:0/2972739108 learned_addr learned my addr 172.21.15.45:0/2972739108 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.205+0000 7f5840a00640 1 -- 172.21.15.45:0/2972739108 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2842238696 0 0) 0x7f583c0d3080 con 0x7f583c162270 2024-04-16T16:15:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.205+0000 7f5840a00640 1 -- 172.21.15.45:0/2972739108 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5824003520 con 0x7f583c162270 2024-04-16T16:15:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.205+0000 7f5842800640 1 --2- 172.21.15.45:0/2972739108 >> v2:172.21.15.103:3300/0 conn(0x7f583c0d52d0 0x7f583c06ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.205+0000 7f5840a00640 1 -- 172.21.15.45:0/2972739108 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2790698079 0 0) 0x7f5824003520 con 0x7f583c162270 2024-04-16T16:15:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.205+0000 7f5840a00640 1 -- 172.21.15.45:0/2972739108 >> v2:172.21.15.103:3300/0 conn(0x7f583c0d52d0 msgr2=0x7f583c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.205+0000 7f5840a00640 1 --2- 172.21.15.45:0/2972739108 >> v2:172.21.15.103:3300/0 conn(0x7f583c0d52d0 0x7f583c06ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.205+0000 7f5840a00640 1 -- 172.21.15.45:0/2972739108 >> v1:172.21.15.103:6789/0 conn(0x7f583c161e90 legacy=0x7f583c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.205+0000 7f5840a00640 1 -- 172.21.15.45:0/2972739108 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f583c0d3640 con 0x7f583c162270 2024-04-16T16:15:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.205+0000 7f5840a00640 1 -- 172.21.15.45:0/2972739108 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f582c001670 con 0x7f583c162270 2024-04-16T16:15:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.205+0000 7f5840a00640 1 -- 172.21.15.45:0/2972739108 >> v1:172.21.15.45:6789/0 conn(0x7f583c162270 legacy=0x7f583c0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.205+0000 7f5840a00640 1 --2- 172.21.15.45:0/2972739108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5824003a00 0x7f5824003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.206+0000 7f5840a00640 1 --2- 172.21.15.45:0/2972739108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5824004800 0x7f5824004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.206+0000 7f5840a00640 1 --2- 172.21.15.45:0/2972739108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5824005210 0x7f582400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.206+0000 7f5840a00640 1 -- 172.21.15.45:0/2972739108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f583c0d2e30 con 0x7f5824003a00 2024-04-16T16:15:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.206+0000 7f5840a00640 1 -- 172.21.15.45:0/2972739108 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f582400c140 con 0x7f5824005210 2024-04-16T16:15:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.206+0000 7f5840a00640 1 -- 172.21.15.45:0/2972739108 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f582400c340 con 0x7f5824004800 2024-04-16T16:15:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.206+0000 7f5841e00640 1 --2- 172.21.15.45:0/2972739108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5824003a00 0x7f5824003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.206+0000 7f5841e00640 1 -- 172.21.15.45:0/2972739108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5824004800 msgr2=0x7f5824004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.206+0000 7f5841e00640 1 --2- 172.21.15.45:0/2972739108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5824004800 0x7f5824004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.206+0000 7f5841e00640 1 -- 172.21.15.45:0/2972739108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5824005210 msgr2=0x7f582400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.206+0000 7f5841e00640 1 --2- 172.21.15.45:0/2972739108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5824005210 0x7f582400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.206+0000 7f5841e00640 1 -- 172.21.15.45:0/2972739108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5824003520 con 0x7f5824003a00 2024-04-16T16:15:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.206+0000 7f5842800640 1 -- 172.21.15.45:0/2972739108 reap_dead start 2024-04-16T16:15:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.207+0000 7f5841e00640 1 --2- 172.21.15.45:0/2972739108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5824003a00 0x7f5824003e00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5830003d80 tx=0x7f5824005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb11406766f56816 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.208+0000 7f5840a00640 1 -- 172.21.15.45:0/2972739108 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5830037070 con 0x7f5824003a00 2024-04-16T16:15:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.208+0000 7f5840a00640 1 -- 172.21.15.45:0/2972739108 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f583002be50 con 0x7f5824003a00 2024-04-16T16:15:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.208+0000 7f5840a00640 1 -- 172.21.15.45:0/2972739108 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5830003980 con 0x7f5824003a00 2024-04-16T16:15:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.208+0000 7f5843200640 1 -- 172.21.15.45:0/2972739108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5824003a00 msgr2=0x7f5824003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.208+0000 7f5843200640 1 --2- 172.21.15.45:0/2972739108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5824003a00 0x7f5824003e00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5830003d80 tx=0x7f5824005830 comp rx=0 tx=0).stop 2024-04-16T16:15:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.208+0000 7f5843200640 1 -- 172.21.15.45:0/2972739108 shutdown_connections 2024-04-16T16:15:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.208+0000 7f5843200640 1 --2- 172.21.15.45:0/2972739108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5824003a00 0x7f5824003e00 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.208+0000 7f5843200640 1 -- 172.21.15.45:0/2972739108 >> 172.21.15.45:0/2972739108 conn(0x7f583c0d1940 msgr2=0x7f583c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.208+0000 7f5843200640 1 -- 172.21.15.45:0/2972739108 shutdown_connections 2024-04-16T16:15:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.208+0000 7f5843200640 1 -- 172.21.15.45:0/2972739108 wait complete. 2024-04-16T16:15:43.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.210+0000 7f5843200640 1 Processor -- start 2024-04-16T16:15:43.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.210+0000 7f5843200640 1 -- start start 2024-04-16T16:15:43.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.211+0000 7f5843200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5824003a00 0x7f583c14ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:43.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.211+0000 7f5843200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f583c0ce2d0 0x7f583c1f55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:43.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.211+0000 7f5843200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f583c15af60 0x7f583c15b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:43.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.211+0000 7f5843200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f583c0cf950 con 0x7f583c15af60 2024-04-16T16:15:43.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.211+0000 7f5843200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f583c0cf7d0 con 0x7f583c0ce2d0 2024-04-16T16:15:43.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.211+0000 7f5843200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f583c0cfad0 con 0x7f5824003a00 2024-04-16T16:15:43.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.212+0000 7f5842800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f583c15af60 0x7f583c15b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:43.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.212+0000 7f5842800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f583c15af60 0x7f583c15b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45900/0 (socket says 172.21.15.45:45900) 2024-04-16T16:15:43.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.212+0000 7f5842800640 1 -- 172.21.15.45:0/2604636588 learned_addr learned my addr 172.21.15.45:0/2604636588 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:43.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.212+0000 7f5841400640 1 --2- 172.21.15.45:0/2604636588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f583c0ce2d0 0x7f583c1f55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:43.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.212+0000 7f5841e00640 1 --2- 172.21.15.45:0/2604636588 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5824003a00 0x7f583c14ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:43.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.212+0000 7f5842800640 1 -- 172.21.15.45:0/2604636588 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5824003a00 msgr2=0x7f583c14ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:43.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.212+0000 7f5842800640 1 --2- 172.21.15.45:0/2604636588 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5824003a00 0x7f583c14ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:43.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.212+0000 7f5842800640 1 -- 172.21.15.45:0/2604636588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f583c0ce2d0 msgr2=0x7f583c1f55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:43.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.212+0000 7f5842800640 1 --2- 172.21.15.45:0/2604636588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f583c0ce2d0 0x7f583c1f55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:43.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.212+0000 7f5842800640 1 -- 172.21.15.45:0/2604636588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f583c1574c0 con 0x7f583c15af60 2024-04-16T16:15:43.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.212+0000 7f5842800640 1 --2- 172.21.15.45:0/2604636588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f583c15af60 0x7f583c15b3c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f583c0d03d0 tx=0x7f583800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:43.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.213+0000 7f5836000640 1 -- 172.21.15.45:0/2604636588 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583800ed90 con 0x7f583c15af60 2024-04-16T16:15:43.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.213+0000 7f5836000640 1 -- 172.21.15.45:0/2604636588 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5838019070 con 0x7f583c15af60 2024-04-16T16:15:43.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.213+0000 7f5836000640 1 -- 172.21.15.45:0/2604636588 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5838014660 con 0x7f583c15af60 2024-04-16T16:15:43.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.213+0000 7f5843200640 1 -- 172.21.15.45:0/2604636588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f583c0ccf50 con 0x7f583c15af60 2024-04-16T16:15:43.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.213+0000 7f5843200640 1 -- 172.21.15.45:0/2604636588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f583c0cd510 con 0x7f583c15af60 2024-04-16T16:15:43.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.214+0000 7f5843200640 1 -- 172.21.15.45:0/2604636588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f583c075780 con 0x7f583c15af60 2024-04-16T16:15:43.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.215+0000 7f5836000640 1 -- 172.21.15.45:0/2604636588 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f5838014800 con 0x7f583c15af60 2024-04-16T16:15:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.216+0000 7f5836000640 1 --2- 172.21.15.45:0/2604636588 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f582006ca60 0x7f582006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.216+0000 7f5836000640 1 -- 172.21.15.45:0/2604636588 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7f583808c810 con 0x7f583c15af60 2024-04-16T16:15:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.217+0000 7f5841e00640 1 --2- 172.21.15.45:0/2604636588 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f582006ca60 0x7f582006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:43.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.218+0000 7f5841e00640 1 --2- 172.21.15.45:0/2604636588 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f582006ca60 0x7f582006ef20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5830034000 tx=0x7f5830009030 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:43.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.221+0000 7f5836000640 1 -- 172.21.15.45:0/2604636588 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f583805a4f0 con 0x7f583c15af60 2024-04-16T16:15:43.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.467+0000 7f5843200640 1 -- 172.21.15.45:0/2604636588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f583c06dae0 con 0x7f583c15af60 2024-04-16T16:15:43.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.468+0000 7f5836000640 1 -- 172.21.15.45:0/2604636588 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6477 (secure 0 0 0) 0x7f5838059df0 con 0x7f583c15af60 2024-04-16T16:15:43.468 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:43.468 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":39,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/1497040472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1497040472},{"type":"v1","addr":"172.21.15.45:6834","nonce":1497040472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":4907,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/93057991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":93057991},{"type":"v1","addr":"172.21.15.45:6835","nonce":93057991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:14.878986+0000","modified":"2024-04-16T16:15:42.283401+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5188":{"gid":5188,"name":"d","rank":0,"incarnation":37,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.103:6836/2626302738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2626302738},{"type":"v1","addr":"172.21.15.103:6836","nonce":2626302738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"fs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3},{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:35.289901+0000","modified":"2024-04-16T16:15:42.275500+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"b","rank":0,"incarnation":35,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.103:6837/1576112884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1576112884},{"type":"v1","addr":"172.21.15.103:6837","nonce":1576112884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"fs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":4}]} 2024-04-16T16:15:43.469 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 39 2024-04-16T16:15:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.470+0000 7f5843200640 1 -- 172.21.15.45:0/2604636588 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f582006ca60 msgr2=0x7f582006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.470+0000 7f5843200640 1 --2- 172.21.15.45:0/2604636588 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f582006ca60 0x7f582006ef20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5830034000 tx=0x7f5830009030 comp rx=0 tx=0).stop 2024-04-16T16:15:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.470+0000 7f5843200640 1 -- 172.21.15.45:0/2604636588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f583c15af60 msgr2=0x7f583c15b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.470+0000 7f5843200640 1 --2- 172.21.15.45:0/2604636588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f583c15af60 0x7f583c15b3c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f583c0d03d0 tx=0x7f583800e4b0 comp rx=0 tx=0).stop 2024-04-16T16:15:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.471+0000 7f5843200640 1 -- 172.21.15.45:0/2604636588 shutdown_connections 2024-04-16T16:15:43.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.471+0000 7f5843200640 1 --2- 172.21.15.45:0/2604636588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f583c15af60 0x7f583c15b3c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:43.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.471+0000 7f5843200640 1 --2- 172.21.15.45:0/2604636588 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f582006ca60 0x7f582006ef20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:43.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.471+0000 7f5843200640 1 --2- 172.21.15.45:0/2604636588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f583c0ce2d0 0x7f583c1f55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:43.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.471+0000 7f5843200640 1 --2- 172.21.15.45:0/2604636588 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5824003a00 0x7f583c14ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:43.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.471+0000 7f5843200640 1 -- 172.21.15.45:0/2604636588 >> 172.21.15.45:0/2604636588 conn(0x7f583c0d1940 msgr2=0x7f583c06ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:43.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.471+0000 7f5843200640 1 -- 172.21.15.45:0/2604636588 shutdown_connections 2024-04-16T16:15:43.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:43.471+0000 7f5843200640 1 -- 172.21.15.45:0/2604636588 wait complete. 2024-04-16T16:15:43.895 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 39, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:15:35.289901+0000', 'modified': '2024-04-16T16:15:42.275500+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5191}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5191': {'gid': 5191, 'name': 'b', 'rank': 0, 'incarnation': 35, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.103:6837/1576112884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6835', 'nonce': 1576112884}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 1576112884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'fs2', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-16T16:15:43.895 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5191:up:rejoin 2024-04-16T16:15:44.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:15:45.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.122+0000 7fbec3200640 1 Processor -- start 2024-04-16T16:15:45.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.122+0000 7fbec3200640 1 -- start start 2024-04-16T16:15:45.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.123+0000 7fbec3200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbec4144ff0 0x7fbec40cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:45.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.123+0000 7fbec3200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbec40c5d80 0x7fbec415d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:45.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.124+0000 7fbec3200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec415c700 con 0x7fbec406dfb0 2024-04-16T16:15:45.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.124+0000 7fbec3200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fbec415d2f0 con 0x7fbec40c5d80 2024-04-16T16:15:45.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.124+0000 7fbec3200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbec415c9c0 con 0x7fbec4144ff0 2024-04-16T16:15:45.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.124+0000 7fbec2800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbec40c5d80 0x7fbec415d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:45.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.124+0000 7fbec2800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbec40c5d80 0x7fbec415d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43126/0 (socket says 172.21.15.45:43126) 2024-04-16T16:15:45.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.124+0000 7fbec2800640 1 -- 172.21.15.45:0/3507921275 learned_addr learned my addr 172.21.15.45:0/3507921275 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:45.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.124+0000 7fbec2800640 1 -- 172.21.15.45:0/3507921275 >> v2:172.21.15.103:3300/0 conn(0x7fbec4144ff0 msgr2=0x7fbec40cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:45.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.124+0000 7fbec2800640 1 --2- 172.21.15.45:0/3507921275 >> v2:172.21.15.103:3300/0 conn(0x7fbec4144ff0 0x7fbec40cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:45.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.124+0000 7fbec2800640 1 -- 172.21.15.45:0/3507921275 >> v1:172.21.15.103:6789/0 conn(0x7fbec406dfb0 legacy=0x7fbec40c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:45.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.124+0000 7fbec2800640 1 -- 172.21.15.45:0/3507921275 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec415cca0 con 0x7fbec40c5d80 2024-04-16T16:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.126+0000 7fbec2800640 1 --2- 172.21.15.45:0/3507921275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec40c5d80 0x7fbec415d780 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fbec40733e0 tx=0x7fbebc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69946e1d7942a192 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.126+0000 7fbec0a00640 1 -- 172.21.15.45:0/3507921275 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc02ca80 con 0x7fbec40c5d80 2024-04-16T16:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.126+0000 7fbec0a00640 1 -- 172.21.15.45:0/3507921275 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbebc02cc20 con 0x7fbec40c5d80 2024-04-16T16:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.126+0000 7fbec0a00640 1 -- 172.21.15.45:0/3507921275 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc0343f0 con 0x7fbec40c5d80 2024-04-16T16:15:45.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.127+0000 7fbec3200640 1 -- 172.21.15.45:0/3507921275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec40c5d80 msgr2=0x7fbec415d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:45.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.127+0000 7fbec3200640 1 --2- 172.21.15.45:0/3507921275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec40c5d80 0x7fbec415d780 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fbec40733e0 tx=0x7fbebc02bee0 comp rx=0 tx=0).stop 2024-04-16T16:15:45.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.127+0000 7fbec3200640 1 -- 172.21.15.45:0/3507921275 shutdown_connections 2024-04-16T16:15:45.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.127+0000 7fbec3200640 1 --2- 172.21.15.45:0/3507921275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec40c5d80 0x7fbec415d780 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:45.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.127+0000 7fbec3200640 1 --2- 172.21.15.45:0/3507921275 >> v2:172.21.15.103:3300/0 conn(0x7fbec4144ff0 0x7fbec40cfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:45.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.127+0000 7fbec3200640 1 -- 172.21.15.45:0/3507921275 >> 172.21.15.45:0/3507921275 conn(0x7fbec406b930 msgr2=0x7fbec406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:45.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.127+0000 7fbec3200640 1 -- 172.21.15.45:0/3507921275 shutdown_connections 2024-04-16T16:15:45.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.127+0000 7fbec3200640 1 -- 172.21.15.45:0/3507921275 wait complete. 2024-04-16T16:15:45.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.129+0000 7fbec3200640 1 Processor -- start 2024-04-16T16:15:45.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.129+0000 7fbec3200640 1 -- start start 2024-04-16T16:15:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.131+0000 7fbec3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec406dfb0 0x7fbec40cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.131+0000 7fbec3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbec40c5d80 0x7fbec41f8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.131+0000 7fbec3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbec4144ff0 0x7fbec41f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.131+0000 7fbec3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbec40d0270 con 0x7fbec406dfb0 2024-04-16T16:15:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.131+0000 7fbec3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbec41453d0 con 0x7fbec4144ff0 2024-04-16T16:15:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.131+0000 7fbec3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbec40d03f0 con 0x7fbec40c5d80 2024-04-16T16:15:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.131+0000 7fbec1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec406dfb0 0x7fbec40cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.131+0000 7fbec1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec406dfb0 0x7fbec40cf2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43134/0 (socket says 172.21.15.45:43134) 2024-04-16T16:15:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.131+0000 7fbec1e00640 1 -- 172.21.15.45:0/4074006112 learned_addr learned my addr 172.21.15.45:0/4074006112 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:45.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.131+0000 7fbec2800640 1 --2- 172.21.15.45:0/4074006112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbec4144ff0 0x7fbec41f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.131+0000 7fbec1e00640 1 -- 172.21.15.45:0/4074006112 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbec40c5d80 msgr2=0x7fbec41f8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.131+0000 7fbec1e00640 1 --2- 172.21.15.45:0/4074006112 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbec40c5d80 0x7fbec41f8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.131+0000 7fbec1e00640 1 -- 172.21.15.45:0/4074006112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbec4144ff0 msgr2=0x7fbec41f88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.131+0000 7fbec1e00640 1 --2- 172.21.15.45:0/4074006112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbec4144ff0 0x7fbec41f88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.131+0000 7fbec1e00640 1 -- 172.21.15.45:0/4074006112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec41f91d0 con 0x7fbec406dfb0 2024-04-16T16:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.132+0000 7fbec1e00640 1 --2- 172.21.15.45:0/4074006112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec406dfb0 0x7fbec40cf2d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400b410 tx=0x7fbeb400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.132+0000 7fbeba000640 1 -- 172.21.15.45:0/4074006112 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4015020 con 0x7fbec406dfb0 2024-04-16T16:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.132+0000 7fbeba000640 1 -- 172.21.15.45:0/4074006112 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeb4011850 con 0x7fbec406dfb0 2024-04-16T16:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.132+0000 7fbeba000640 1 -- 172.21.15.45:0/4074006112 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4011c40 con 0x7fbec406dfb0 2024-04-16T16:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.132+0000 7fbec3200640 1 -- 172.21.15.45:0/4074006112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbec4138d50 con 0x7fbec406dfb0 2024-04-16T16:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.132+0000 7fbec3200640 1 -- 172.21.15.45:0/4074006112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbec4139310 con 0x7fbec406dfb0 2024-04-16T16:15:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.134+0000 7fbec3200640 1 -- 172.21.15.45:0/4074006112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbec4075780 con 0x7fbec406dfb0 2024-04-16T16:15:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.134+0000 7fbeba000640 1 -- 172.21.15.45:0/4074006112 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fbeb4018020 con 0x7fbec406dfb0 2024-04-16T16:15:45.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.135+0000 7fbeba000640 1 --2- 172.21.15.45:0/4074006112 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbea406ca60 0x7fbea406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:45.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.136+0000 7fbeba000640 1 -- 172.21.15.45:0/4074006112 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7fbeb408c280 con 0x7fbec406dfb0 2024-04-16T16:15:45.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.136+0000 7fbec1400640 1 --2- 172.21.15.45:0/4074006112 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbea406ca60 0x7fbea406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:45.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.137+0000 7fbec1400640 1 --2- 172.21.15.45:0/4074006112 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbea406ca60 0x7fbea406ef20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbec406e6d0 tx=0x7fbeb0007990 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:45.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.140+0000 7fbeba000640 1 -- 172.21.15.45:0/4074006112 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbeb401f020 con 0x7fbec406dfb0 2024-04-16T16:15:45.276 INFO:tasks.ceph.mgr.x.smithi103.stderr:Exception in thread Thread-20: 2024-04-16T16:15:45.276 INFO:tasks.ceph.mgr.x.smithi103.stderr:Traceback (most recent call last): 2024-04-16T16:15:45.276 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner 2024-04-16T16:15:45.277 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.run() 2024-04-16T16:15:45.277 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/lib64/python3.9/threading.py", line 1306, in run 2024-04-16T16:15:45.277 INFO:tasks.ceph.mgr.x.smithi103.stderr: self.function(*self.args, **self.kwargs) 2024-04-16T16:15:45.277 INFO:tasks.ceph.mgr.x.smithi103.stderr: File "/usr/share/ceph/mgr/stats/fs/perf_stats.py", line 222, in re_register_queries 2024-04-16T16:15:45.277 INFO:tasks.ceph.mgr.x.smithi103.stderr: if self.mx_last_updated >= ua_last_updated: 2024-04-16T16:15:45.277 INFO:tasks.ceph.mgr.x.smithi103.stderr:AttributeError: 'FSPerfStats' object has no attribute 'mx_last_updated' 2024-04-16T16:15:45.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.384+0000 7fbec3200640 1 -- 172.21.15.45:0/4074006112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbec40cfd30 con 0x7fbec406dfb0 2024-04-16T16:15:45.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.385+0000 7fbeba000640 1 -- 172.21.15.45:0/4074006112 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6491 (secure 0 0 0) 0x7fbeb40172c0 con 0x7fbec406dfb0 2024-04-16T16:15:45.385 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:15:45.386 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":40,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/1497040472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1497040472},{"type":"v1","addr":"172.21.15.45:6834","nonce":1497040472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":4907,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/93057991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":93057991},{"type":"v1","addr":"172.21.15.45:6835","nonce":93057991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:14.878986+0000","modified":"2024-04-16T16:15:44.272257+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5188":{"gid":5188,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.103:6836/2626302738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2626302738},{"type":"v1","addr":"172.21.15.103:6836","nonce":2626302738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"fs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5188,"qdb_cluster":[5188]},"id":3},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:35.289901+0000","modified":"2024-04-16T16:15:44.272258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"b","rank":0,"incarnation":35,"state":"up:active","state_seq":4,"addr":"172.21.15.103:6837/1576112884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1576112884},{"type":"v1","addr":"172.21.15.103:6837","nonce":1576112884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"fs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":4}]} 2024-04-16T16:15:45.386 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 40 2024-04-16T16:15:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.388+0000 7fbec3200640 1 -- 172.21.15.45:0/4074006112 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbea406ca60 msgr2=0x7fbea406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.388+0000 7fbec3200640 1 --2- 172.21.15.45:0/4074006112 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbea406ca60 0x7fbea406ef20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbec406e6d0 tx=0x7fbeb0007990 comp rx=0 tx=0).stop 2024-04-16T16:15:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.388+0000 7fbec3200640 1 -- 172.21.15.45:0/4074006112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec406dfb0 msgr2=0x7fbec40cf2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.388+0000 7fbec3200640 1 --2- 172.21.15.45:0/4074006112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec406dfb0 0x7fbec40cf2d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400b410 tx=0x7fbeb400b8d0 comp rx=0 tx=0).stop 2024-04-16T16:15:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.388+0000 7fbec3200640 1 -- 172.21.15.45:0/4074006112 shutdown_connections 2024-04-16T16:15:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.388+0000 7fbec3200640 1 --2- 172.21.15.45:0/4074006112 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbea406ca60 0x7fbea406ef20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.388+0000 7fbec3200640 1 --2- 172.21.15.45:0/4074006112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbec4144ff0 0x7fbec41f88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.388+0000 7fbec3200640 1 --2- 172.21.15.45:0/4074006112 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbec40c5d80 0x7fbec41f8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.388+0000 7fbec3200640 1 --2- 172.21.15.45:0/4074006112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec406dfb0 0x7fbec40cf2d0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.388+0000 7fbec3200640 1 -- 172.21.15.45:0/4074006112 >> 172.21.15.45:0/4074006112 conn(0x7fbec406b930 msgr2=0x7fbec40b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.388+0000 7fbec3200640 1 -- 172.21.15.45:0/4074006112 shutdown_connections 2024-04-16T16:15:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:45.388+0000 7fbec3200640 1 -- 172.21.15.45:0/4074006112 wait complete. 2024-04-16T16:15:45.811 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:15:35.289901+0000', 'modified': '2024-04-16T16:15:44.272258+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5191}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5191': {'gid': 5191, 'name': 'b', 'rank': 0, 'incarnation': 35, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.103:6837/1576112884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6835', 'nonce': 1576112884}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 1576112884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'fs2', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5191, 'qdb_cluster': [5191]} 2024-04-16T16:15:45.811 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-16T16:15:45.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-16T16:15:46.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.028+0000 7f7d1f200640 1 Processor -- start 2024-04-16T16:15:46.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.028+0000 7f7d1f200640 1 -- start start 2024-04-16T16:15:46.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.030+0000 7f7d1f200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7d18162090 0x7f7d180cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.030+0000 7f7d1f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d180ba140 0x7f7d180cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.030+0000 7f7d1f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7d180d5520 0x7f7d181628a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.030+0000 7f7d1f200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f7d180ba8b0 con 0x7f7d18162090 2024-04-16T16:15:46.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.030+0000 7f7d1f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7d180d2600 con 0x7f7d180d5520 2024-04-16T16:15:46.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.030+0000 7f7d1f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7d180ba5d0 con 0x7f7d180ba140 2024-04-16T16:15:46.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.030+0000 7f7d1de00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7d18162090 0x7f7d180cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.030+0000 7f7d1de00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7d18162090 0x7f7d180cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43140/0 (socket says 172.21.15.45:43140) 2024-04-16T16:15:46.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.030+0000 7f7d1de00640 1 -- 172.21.15.45:0/838191711 learned_addr learned my addr 172.21.15.45:0/838191711 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:46.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.031+0000 7f7d1d400640 1 --2- 172.21.15.45:0/838191711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d180ba140 0x7f7d180cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.031+0000 7f7d1de00640 1 -- 172.21.15.45:0/838191711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d180ba140 msgr2=0x7f7d180cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:46.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.031+0000 7f7d1de00640 1 --2- 172.21.15.45:0/838191711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d180ba140 0x7f7d180cfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.031+0000 7f7d1de00640 1 -- 172.21.15.45:0/838191711 >> v2:172.21.15.103:3300/0 conn(0x7f7d180d5520 msgr2=0x7f7d181628a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:46.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.031+0000 7f7d1e800640 1 --2- 172.21.15.45:0/838191711 >> v2:172.21.15.103:3300/0 conn(0x7f7d180d5520 0x7f7d181628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.031+0000 7f7d1de00640 1 --2- 172.21.15.45:0/838191711 >> v2:172.21.15.103:3300/0 conn(0x7f7d180d5520 0x7f7d181628a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.031+0000 7f7d1de00640 1 -- 172.21.15.45:0/838191711 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d180d28a0 con 0x7f7d18162090 2024-04-16T16:15:46.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.032+0000 7f7d1de00640 1 --2- 172.21.15.45:0/838191711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d18162090 0x7f7d180cf510 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7d180b7430 tx=0x7f7d0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f120453136bfdca2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:46.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.032+0000 7f7d1ca00640 1 -- 172.21.15.45:0/838191711 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d04035020 con 0x7f7d18162090 2024-04-16T16:15:46.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.032+0000 7f7d1ca00640 1 -- 172.21.15.45:0/838191711 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d04031870 con 0x7f7d18162090 2024-04-16T16:15:46.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.032+0000 7f7d1ca00640 1 -- 172.21.15.45:0/838191711 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d04031c40 con 0x7f7d18162090 2024-04-16T16:15:46.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.033+0000 7f7d1f200640 1 -- 172.21.15.45:0/838191711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d18162090 msgr2=0x7f7d180cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.033+0000 7f7d1f200640 1 --2- 172.21.15.45:0/838191711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d18162090 0x7f7d180cf510 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7d180b7430 tx=0x7f7d0402b910 comp rx=0 tx=0).stop 2024-04-16T16:15:46.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.033+0000 7f7d1f200640 1 -- 172.21.15.45:0/838191711 shutdown_connections 2024-04-16T16:15:46.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.033+0000 7f7d1f200640 1 --2- 172.21.15.45:0/838191711 >> v2:172.21.15.103:3300/0 conn(0x7f7d180d5520 0x7f7d181628a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.033+0000 7f7d1f200640 1 --2- 172.21.15.45:0/838191711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d180ba140 0x7f7d180cfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.033+0000 7f7d1f200640 1 --2- 172.21.15.45:0/838191711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d18162090 0x7f7d180cf510 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.033+0000 7f7d1f200640 1 -- 172.21.15.45:0/838191711 >> 172.21.15.45:0/838191711 conn(0x7f7d180d1940 msgr2=0x7f7d180d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:46.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.033+0000 7f7d1f200640 1 -- 172.21.15.45:0/838191711 shutdown_connections 2024-04-16T16:15:46.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.033+0000 7f7d1f200640 1 -- 172.21.15.45:0/838191711 wait complete. 2024-04-16T16:15:46.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.035+0000 7f7d1f200640 1 Processor -- start 2024-04-16T16:15:46.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.035+0000 7f7d1f200640 1 -- start start 2024-04-16T16:15:46.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.036+0000 7f7d1f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d180d5520 0x7f7d181f9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.036+0000 7f7d1f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d18162090 0x7f7d180cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.036+0000 7f7d1f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d18155f60 0x7f7d1815aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.036+0000 7f7d1f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181625f0 con 0x7f7d180d5520 2024-04-16T16:15:46.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.036+0000 7f7d1f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7d18162470 con 0x7f7d18155f60 2024-04-16T16:15:46.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.036+0000 7f7d1f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7d180b36c0 con 0x7f7d18162090 2024-04-16T16:15:46.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.037+0000 7f7d1de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d180d5520 0x7f7d181f9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.037+0000 7f7d1de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d180d5520 0x7f7d181f9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43148/0 (socket says 172.21.15.45:43148) 2024-04-16T16:15:46.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.037+0000 7f7d1de00640 1 -- 172.21.15.45:0/2781652657 learned_addr learned my addr 172.21.15.45:0/2781652657 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:46.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.037+0000 7f7d1d400640 1 --2- 172.21.15.45:0/2781652657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d18162090 0x7f7d180cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.037+0000 7f7d1de00640 1 -- 172.21.15.45:0/2781652657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d18162090 msgr2=0x7f7d180cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.037+0000 7f7d1de00640 1 --2- 172.21.15.45:0/2781652657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d18162090 0x7f7d180cad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.037+0000 7f7d1de00640 1 -- 172.21.15.45:0/2781652657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d18155f60 msgr2=0x7f7d1815aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.037+0000 7f7d1de00640 1 --2- 172.21.15.45:0/2781652657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d18155f60 0x7f7d1815aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.037+0000 7f7d1de00640 1 -- 172.21.15.45:0/2781652657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d180c8460 con 0x7f7d180d5520 2024-04-16T16:15:46.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.038+0000 7f7d1de00640 1 --2- 172.21.15.45:0/2781652657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d180d5520 0x7f7d181f9340 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7d0402b950 tx=0x7f7d040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:46.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.038+0000 7f7d12000640 1 -- 172.21.15.45:0/2781652657 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d04040070 con 0x7f7d180d5520 2024-04-16T16:15:46.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.038+0000 7f7d12000640 1 -- 172.21.15.45:0/2781652657 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d0403a420 con 0x7f7d180d5520 2024-04-16T16:15:46.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.038+0000 7f7d1f200640 1 -- 172.21.15.45:0/2781652657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d180c8700 con 0x7f7d180d5520 2024-04-16T16:15:46.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.038+0000 7f7d12000640 1 -- 172.21.15.45:0/2781652657 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d04035040 con 0x7f7d180d5520 2024-04-16T16:15:46.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.038+0000 7f7d1f200640 1 -- 172.21.15.45:0/2781652657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d180c8cc0 con 0x7f7d180d5520 2024-04-16T16:15:46.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.040+0000 7f7d12000640 1 -- 172.21.15.45:0/2781652657 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f7d0404b060 con 0x7f7d180d5520 2024-04-16T16:15:46.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.041+0000 7f7d12000640 1 --2- 172.21.15.45:0/2781652657 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7cf806ca60 0x7f7cf806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.041+0000 7f7d12000640 1 -- 172.21.15.45:0/2781652657 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7f7d040ae330 con 0x7f7d180d5520 2024-04-16T16:15:46.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.042+0000 7f7d1d400640 1 --2- 172.21.15.45:0/2781652657 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7cf806ca60 0x7f7cf806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.042+0000 7f7d2357f740 1 Processor -- start 2024-04-16T16:15:46.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.042+0000 7f7d2357f740 1 -- start start 2024-04-16T16:15:46.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.043+0000 7f7d1d400640 1 --2- 172.21.15.45:0/2781652657 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7cf806ca60 0x7f7cf806ef20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d180b58f0 tx=0x7f7d0c008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:46.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.043+0000 7f7d2357f740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6475b890 0x561b6477a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.043+0000 7f7d2357f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475bc70 0x561b64734570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.043+0000 7f7d2357f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b64734ab0 0x561b64732440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.043+0000 7f7d2357f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561b646999c0 con 0x561b6475bc70 2024-04-16T16:15:46.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.043+0000 7f7d2357f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561b64640560 con 0x561b64734ab0 2024-04-16T16:15:46.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.043+0000 7f7d2357f740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561b64695980 con 0x561b6475b890 2024-04-16T16:15:46.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.044+0000 7f7d1d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475bc70 0x561b64734570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.044+0000 7f7d1d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475bc70 0x561b64734570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43164/0 (socket says 172.21.15.45:43164) 2024-04-16T16:15:46.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.044+0000 7f7d1d400640 1 -- 172.21.15.45:0/3974012757 learned_addr learned my addr 172.21.15.45:0/3974012757 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:46.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.044+0000 7f7d1de00640 1 --2- 172.21.15.45:0/3974012757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6475b890 0x561b6477a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.044+0000 7f7d1e800640 1 --2- 172.21.15.45:0/3974012757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b64734ab0 0x561b64732440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.044+0000 7f7d1d400640 1 -- 172.21.15.45:0/3974012757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6475b890 msgr2=0x561b6477a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:46.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.044+0000 7f7d1d400640 1 --2- 172.21.15.45:0/3974012757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6475b890 0x561b6477a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.044+0000 7f7d1d400640 1 -- 172.21.15.45:0/3974012757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b64734ab0 msgr2=0x561b64732440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:46.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.044+0000 7f7d1d400640 1 --2- 172.21.15.45:0/3974012757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b64734ab0 0x561b64732440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.044+0000 7f7d1d400640 1 -- 172.21.15.45:0/3974012757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b643ac2d0 con 0x561b6475bc70 2024-04-16T16:15:46.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.045+0000 7f7d1d400640 1 --2- 172.21.15.45:0/3974012757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475bc70 0x561b64734570 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c00ec30 tx=0x7f7d0c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930e1cf6bae926c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:46.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.045+0000 7f7d0a600640 1 -- 172.21.15.45:0/3974012757 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c01a050 con 0x561b6475bc70 2024-04-16T16:15:46.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.045+0000 7f7d0a600640 1 -- 172.21.15.45:0/3974012757 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d0c01b040 con 0x561b6475bc70 2024-04-16T16:15:46.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.045+0000 7f7d0a600640 1 -- 172.21.15.45:0/3974012757 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c013d60 con 0x561b6475bc70 2024-04-16T16:15:46.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.045+0000 7f7d2357f740 1 -- 172.21.15.45:0/3974012757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475bc70 msgr2=0x561b64734570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.045+0000 7f7d2357f740 1 --2- 172.21.15.45:0/3974012757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475bc70 0x561b64734570 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c00ec30 tx=0x7f7d0c011000 comp rx=0 tx=0).stop 2024-04-16T16:15:46.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.046+0000 7f7d2357f740 1 -- 172.21.15.45:0/3974012757 shutdown_connections 2024-04-16T16:15:46.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.046+0000 7f7d2357f740 1 --2- 172.21.15.45:0/3974012757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b64734ab0 0x561b64732440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.046+0000 7f7d2357f740 1 --2- 172.21.15.45:0/3974012757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475bc70 0x561b64734570 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.046+0000 7f7d2357f740 1 --2- 172.21.15.45:0/3974012757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6475b890 0x561b6477a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.046+0000 7f7d2357f740 1 -- 172.21.15.45:0/3974012757 >> 172.21.15.45:0/3974012757 conn(0x561b64730b50 msgr2=0x561b647796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:46.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.046+0000 7f7d2357f740 1 -- 172.21.15.45:0/3974012757 shutdown_connections 2024-04-16T16:15:46.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.046+0000 7f7d2357f740 1 -- 172.21.15.45:0/3974012757 wait complete. 2024-04-16T16:15:46.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.046+0000 7f7d2357f740 1 -- start start 2024-04-16T16:15:46.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.047+0000 7f7d2357f740 1 Processor -- start 2024-04-16T16:15:46.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.048+0000 7f7d2357f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475b890 0x561b6474bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.048+0000 7f7d2357f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6475bc70 0x561b64774fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.048+0000 7f7d2357f740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b64775500 0x561b6477c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.048+0000 7f7d2357f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561b6472e980 con 0x561b6475b890 2024-04-16T16:15:46.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.048+0000 7f7d2357f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561b64653920 con 0x561b6475bc70 2024-04-16T16:15:46.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.048+0000 7f7d2357f740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561b64734f30 con 0x561b64775500 2024-04-16T16:15:46.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.048+0000 7f7d2357f740 10 client.0 _ll_register_callbacks cb 0x561b64653310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:15:46.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.048+0000 7f7d1de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475b890 0x561b6474bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.048+0000 7f7d1de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475b890 0x561b6474bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43178/0 (socket says 172.21.15.45:43178) 2024-04-16T16:15:46.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.048+0000 7f7d1de00640 1 -- 172.21.15.45:0/748395353 learned_addr learned my addr 172.21.15.45:0/748395353 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:46.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.048+0000 7f7d1de00640 1 -- 172.21.15.45:0/748395353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b64775500 msgr2=0x561b6477c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:46.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.048+0000 7f7d1de00640 1 --2- 172.21.15.45:0/748395353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b64775500 0x561b6477c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.048+0000 7f7d1de00640 1 -- 172.21.15.45:0/748395353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6475bc70 msgr2=0x561b64774fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:46.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.048+0000 7f7d1de00640 1 --2- 172.21.15.45:0/748395353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6475bc70 0x561b64774fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.048+0000 7f7d1de00640 1 -- 172.21.15.45:0/748395353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b644766b0 con 0x561b6475b890 2024-04-16T16:15:46.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.049+0000 7f7d1de00640 1 --2- 172.21.15.45:0/748395353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475b890 0x561b6474bb50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7d040ac040 tx=0x7f7d0407b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e95fd999835bfd server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:46.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.049+0000 7f7cfea00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:15:46.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.049+0000 7f7cfea00640 1 -- 172.21.15.45:0/748395353 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d040aa060 con 0x561b6475b890 2024-04-16T16:15:46.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.049+0000 7f7cfea00640 1 -- 172.21.15.45:0/748395353 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7f7d040ac210 con 0x561b6475b890 2024-04-16T16:15:46.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.049+0000 7f7d2357f740 1 -- 172.21.15.45:0/748395353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d0c01b040 con 0x561b6475b890 2024-04-16T16:15:46.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.049+0000 7f7cfea00640 1 -- 172.21.15.45:0/748395353 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7d0407fcc0 con 0x561b6475b890 2024-04-16T16:15:46.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.049+0000 7f7d2357f740 10 client.5889 fetch_fsmap learned FSMap version 40 2024-04-16T16:15:46.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.049+0000 7f7d2357f740 1 -- 172.21.15.45:0/748395353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) -- 0x561b647765d0 con 0x561b6475b890 2024-04-16T16:15:46.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.049+0000 7f7cfea00640 1 -- 172.21.15.45:0/748395353 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d0c01b040 con 0x561b6475b890 2024-04-16T16:15:46.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.049+0000 7f7cfea00640 1 -- 172.21.15.45:0/748395353 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 40) ==== 4132+0+0 (secure 0 0 0) 0x7f7d04077f80 con 0x561b6475b890 2024-04-16T16:15:46.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.049+0000 7f7d2357f740 10 client.5889 fetch_fsmap finished waiting for FSMap version 40 2024-04-16T16:15:46.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.049+0000 7f7d2357f740 10 client.5889 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-16T16:15:46.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.050+0000 7f7d2357f740 20 client.5889 populate_metadata read hostname 'smithi045' 2024-04-16T16:15:46.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.050+0000 7f7d2357f740 1 --2- 172.21.15.45:0/748395353 >> [v2:172.21.15.103:6835/1576112884,v1:172.21.15.103:6837/1576112884] conn(0x561b64773f60 0x561b6478a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.050+0000 7f7d2357f740 4 client.5889 mds_command: new command op to 5191 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-16T16:15:46.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.050+0000 7f7d2357f740 1 -- 172.21.15.45:0/748395353 --> [v2:172.21.15.103:6835/1576112884,v1:172.21.15.103:6837/1576112884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b6463fb70 con 0x561b64773f60 2024-04-16T16:15:46.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.050+0000 7f7d1d400640 1 --2- 172.21.15.45:0/748395353 >> [v2:172.21.15.103:6835/1576112884,v1:172.21.15.103:6837/1576112884] conn(0x561b64773f60 0x561b6478a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.051+0000 7f7d1d400640 1 --2- 172.21.15.45:0/748395353 >> [v2:172.21.15.103:6835/1576112884,v1:172.21.15.103:6837/1576112884] conn(0x561b64773f60 0x561b6478a870 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=789df8d133fee507 server_cookie=93a090135626031c in_seq=0 out_seq=0 2024-04-16T16:15:46.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.051+0000 7f7cfea00640 10 client.5889 ms_handle_connect on v2:172.21.15.103:6835/1576112884 2024-04-16T16:15:46.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.054+0000 7f7cfea00640 1 -- 172.21.15.45:0/748395353 <== mds.0 v2:172.21.15.103:6835/1576112884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b6463fb70 con 0x561b64773f60 2024-04-16T16:15:46.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.054+0000 7f7cfea00640 10 client.5889 handle_command_reply: tid=0 2024-04-16T16:15:46.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.054+0000 7f7d2357f740 1 client.5889 shutdown 2024-04-16T16:15:46.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.054+0000 7f7d2357f740 1 -- 172.21.15.45:0/748395353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475b890 msgr2=0x561b6474bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.054+0000 7f7d2357f740 1 --2- 172.21.15.45:0/748395353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475b890 0x561b6474bb50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7d040ac040 tx=0x7f7d0407b330 comp rx=0 tx=0).stop 2024-04-16T16:15:46.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.054+0000 7f7d2357f740 1 -- 172.21.15.45:0/748395353 shutdown_connections 2024-04-16T16:15:46.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.054+0000 7f7d2357f740 1 --2- 172.21.15.45:0/748395353 >> [v2:172.21.15.103:6835/1576112884,v1:172.21.15.103:6837/1576112884] conn(0x561b64773f60 0x561b6478a870 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.054+0000 7f7d2357f740 1 --2- 172.21.15.45:0/748395353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b64775500 0x561b6477c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.054+0000 7f7d2357f740 1 --2- 172.21.15.45:0/748395353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6475bc70 0x561b64774fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.054+0000 7f7d2357f740 1 --2- 172.21.15.45:0/748395353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475b890 0x561b6474bb50 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.054+0000 7f7cfea00640 0 client.5889 ms_handle_reset on v2:172.21.15.103:6835/1576112884 2024-04-16T16:15:46.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.054+0000 7f7d2357f740 1 -- 172.21.15.45:0/748395353 >> 172.21.15.45:0/748395353 conn(0x561b64734ab0 msgr2=0x561b6472d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:46.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.054+0000 7f7d2357f740 1 -- 172.21.15.45:0/748395353 shutdown_connections 2024-04-16T16:15:46.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.054+0000 7f7d2357f740 1 -- 172.21.15.45:0/748395353 wait complete. 2024-04-16T16:15:46.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.054+0000 7f7d2357f740 20 client.5889 trim_cache size 0 max 16384 2024-04-16T16:15:46.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.074+0000 7f7d2357f740 1 Processor -- start 2024-04-16T16:15:46.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.074+0000 7f7d2357f740 1 -- start start 2024-04-16T16:15:46.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.075+0000 7f7d2357f740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6475b890 0x561b64735700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.075+0000 7f7d2357f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6475bc70 0x561b64779660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.075+0000 7f7d2357f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b64775500 0x561b64779ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.075+0000 7f7d2357f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7d18204470 con 0x561b64775500 2024-04-16T16:15:46.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.075+0000 7f7d2357f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7d18159370 con 0x561b6475bc70 2024-04-16T16:15:46.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.075+0000 7f7d2357f740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7d182026c0 con 0x561b6475b890 2024-04-16T16:15:46.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.075+0000 7f7d1e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b64775500 0x561b64779ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.075+0000 7f7d1e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b64775500 0x561b64779ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43192/0 (socket says 172.21.15.45:43192) 2024-04-16T16:15:46.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.075+0000 7f7d1e800640 1 -- 172.21.15.45:0/2838157302 learned_addr learned my addr 172.21.15.45:0/2838157302 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.076+0000 7f7d1d400640 1 --2- 172.21.15.45:0/2838157302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6475bc70 0x561b64779660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.076+0000 7f7d1e800640 1 -- 172.21.15.45:0/2838157302 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6475b890 msgr2=0x561b64735700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.076+0000 7f7d1de00640 1 --2- 172.21.15.45:0/2838157302 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6475b890 0x561b64735700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.076+0000 7f7d1e800640 1 --2- 172.21.15.45:0/2838157302 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6475b890 0x561b64735700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.076+0000 7f7d1e800640 1 -- 172.21.15.45:0/2838157302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6475bc70 msgr2=0x561b64779660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.076+0000 7f7d1e800640 1 --2- 172.21.15.45:0/2838157302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6475bc70 0x561b64779660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.076+0000 7f7d1e800640 1 -- 172.21.15.45:0/2838157302 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b64735c40 con 0x561b64775500 2024-04-16T16:15:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.076+0000 7f7d1d400640 1 --2- 172.21.15.45:0/2838157302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6475bc70 0x561b64779660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.076+0000 7f7d1e800640 1 --2- 172.21.15.45:0/2838157302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b64775500 0x561b64779ba0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7d14007db0 tx=0x7f7d1400d750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d487171893312a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.076+0000 7f7d09c00640 1 -- 172.21.15.45:0/2838157302 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d140165f0 con 0x561b64775500 2024-04-16T16:15:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.076+0000 7f7d09c00640 1 -- 172.21.15.45:0/2838157302 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d14016790 con 0x561b64775500 2024-04-16T16:15:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.076+0000 7f7d09c00640 1 -- 172.21.15.45:0/2838157302 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d14016b80 con 0x561b64775500 2024-04-16T16:15:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.077+0000 7f7d2357f740 1 -- 172.21.15.45:0/2838157302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b64775500 msgr2=0x561b64779ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.077+0000 7f7d2357f740 1 --2- 172.21.15.45:0/2838157302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b64775500 0x561b64779ba0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7d14007db0 tx=0x7f7d1400d750 comp rx=0 tx=0).stop 2024-04-16T16:15:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.077+0000 7f7d2357f740 1 -- 172.21.15.45:0/2838157302 shutdown_connections 2024-04-16T16:15:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.077+0000 7f7d2357f740 1 --2- 172.21.15.45:0/2838157302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b64775500 0x561b64779ba0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.077+0000 7f7d2357f740 1 --2- 172.21.15.45:0/2838157302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6475bc70 0x561b64779660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.077+0000 7f7d2357f740 1 --2- 172.21.15.45:0/2838157302 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6475b890 0x561b64735700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.077+0000 7f7d2357f740 1 -- 172.21.15.45:0/2838157302 >> 172.21.15.45:0/2838157302 conn(0x561b64734ab0 msgr2=0x561b6475ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:46.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.077+0000 7f7d2357f740 1 -- 172.21.15.45:0/2838157302 shutdown_connections 2024-04-16T16:15:46.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.077+0000 7f7d2357f740 1 -- 172.21.15.45:0/2838157302 wait complete. 2024-04-16T16:15:46.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.078+0000 7f7d2357f740 1 -- start start 2024-04-16T16:15:46.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.078+0000 7f7d2357f740 1 Processor -- start 2024-04-16T16:15:46.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.079+0000 7f7d2357f740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6475b890 0x561b64732d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.079+0000 7f7d2357f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475bc70 0x561b647767a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.079+0000 7f7d2357f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b64775500 0x561b647f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.079+0000 7f7d2357f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561b6472e980 con 0x561b6475bc70 2024-04-16T16:15:46.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.079+0000 7f7d2357f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561b64653920 con 0x561b64775500 2024-04-16T16:15:46.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.079+0000 7f7d2357f740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561b64774420 con 0x561b6475b890 2024-04-16T16:15:46.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.079+0000 7f7d2357f740 10 client.0 _ll_register_callbacks cb 0x7f7d04030f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:15:46.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.079+0000 7f7d1d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475bc70 0x561b647767a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.079+0000 7f7d1d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475bc70 0x561b647767a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43194/0 (socket says 172.21.15.45:43194) 2024-04-16T16:15:46.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.079+0000 7f7d1d400640 1 -- 172.21.15.45:0/2376589183 learned_addr learned my addr 172.21.15.45:0/2376589183 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:46.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.080+0000 7f7d1d400640 1 -- 172.21.15.45:0/2376589183 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6475b890 msgr2=0x561b64732d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:46.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.080+0000 7f7d1de00640 1 --2- 172.21.15.45:0/2376589183 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6475b890 0x561b64732d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.080+0000 7f7d1d400640 1 --2- 172.21.15.45:0/2376589183 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6475b890 0x561b64732d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.080+0000 7f7d1d400640 1 -- 172.21.15.45:0/2376589183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b64775500 msgr2=0x561b647f3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:46.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.080+0000 7f7d1d400640 1 --2- 172.21.15.45:0/2376589183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b64775500 0x561b647f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.080+0000 7f7d1d400640 1 -- 172.21.15.45:0/2376589183 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b647f4a20 con 0x561b6475bc70 2024-04-16T16:15:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.080+0000 7f7d1d400640 1 --2- 172.21.15.45:0/2376589183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475bc70 0x561b647767a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c00a850 tx=0x7f7d0c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0aff19afb80982c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.080+0000 7f7cfea00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:15:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.080+0000 7f7cfea00640 1 -- 172.21.15.45:0/2376589183 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c01a040 con 0x561b6475bc70 2024-04-16T16:15:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.080+0000 7f7cfea00640 1 -- 172.21.15.45:0/2376589183 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7f7d0c00e850 con 0x561b6475bc70 2024-04-16T16:15:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.080+0000 7f7d2357f740 1 -- 172.21.15.45:0/2376589183 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d14016790 con 0x561b6475bc70 2024-04-16T16:15:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.081+0000 7f7cfea00640 1 -- 172.21.15.45:0/2376589183 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7d0c041050 con 0x561b6475bc70 2024-04-16T16:15:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.081+0000 7f7d2357f740 10 client.5895 fetch_fsmap learned FSMap version 40 2024-04-16T16:15:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.081+0000 7f7d2357f740 1 -- 172.21.15.45:0/2376589183 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) -- 0x561b64777220 con 0x561b6475bc70 2024-04-16T16:15:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.081+0000 7f7cfea00640 1 -- 172.21.15.45:0/2376589183 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d14016790 con 0x561b6475bc70 2024-04-16T16:15:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.081+0000 7f7cfea00640 1 -- 172.21.15.45:0/2376589183 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 40) ==== 4132+0+0 (secure 0 0 0) 0x7f7d0c031bc0 con 0x561b6475bc70 2024-04-16T16:15:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.081+0000 7f7d2357f740 10 client.5895 fetch_fsmap finished waiting for FSMap version 40 2024-04-16T16:15:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.081+0000 7f7d2357f740 10 client.5895 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-16T16:15:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.081+0000 7f7d2357f740 20 client.5895 populate_metadata read hostname 'smithi045' 2024-04-16T16:15:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.081+0000 7f7d2357f740 1 --2- 172.21.15.45:0/2376589183 >> [v2:172.21.15.103:6835/1576112884,v1:172.21.15.103:6837/1576112884] conn(0x561b64759880 0x561b64759c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.081+0000 7f7d2357f740 4 client.5895 mds_command: new command op to 5191 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-16T16:15:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.081+0000 7f7d2357f740 1 -- 172.21.15.45:0/2376589183 --> [v2:172.21.15.103:6835/1576112884,v1:172.21.15.103:6837/1576112884] -- command(tid 0: {"prefix": "status"}) -- 0x561b6475d160 con 0x561b64759880 2024-04-16T16:15:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.082+0000 7f7d1de00640 1 --2- 172.21.15.45:0/2376589183 >> [v2:172.21.15.103:6835/1576112884,v1:172.21.15.103:6837/1576112884] conn(0x561b64759880 0x561b64759c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.083+0000 7f7d1de00640 1 --2- 172.21.15.45:0/2376589183 >> [v2:172.21.15.103:6835/1576112884,v1:172.21.15.103:6837/1576112884] conn(0x561b64759880 0x561b64759c60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f06cd3352fb09a4 server_cookie=53e6da766287266a in_seq=0 out_seq=0 2024-04-16T16:15:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.083+0000 7f7cfea00640 10 client.5895 ms_handle_connect on v2:172.21.15.103:6835/1576112884 2024-04-16T16:15:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.083+0000 7f7cfea00640 1 -- 172.21.15.45:0/2376589183 <== mds.0 v2:172.21.15.103:6835/1576112884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x561b6475d160 con 0x561b64759880 2024-04-16T16:15:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.083+0000 7f7cfea00640 10 client.5895 handle_command_reply: tid=0 2024-04-16T16:15:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.083+0000 7f7d2357f740 1 client.5895 shutdown 2024-04-16T16:15:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.083+0000 7f7d2357f740 1 -- 172.21.15.45:0/2376589183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475bc70 msgr2=0x561b647767a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.083+0000 7f7d2357f740 1 --2- 172.21.15.45:0/2376589183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475bc70 0x561b647767a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c00a850 tx=0x7f7d0c015000 comp rx=0 tx=0).stop 2024-04-16T16:15:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.083+0000 7f7d2357f740 1 -- 172.21.15.45:0/2376589183 shutdown_connections 2024-04-16T16:15:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.083+0000 7f7d2357f740 1 --2- 172.21.15.45:0/2376589183 >> [v2:172.21.15.103:6835/1576112884,v1:172.21.15.103:6837/1576112884] conn(0x561b64759880 0x561b64759c60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.083+0000 7f7d2357f740 1 --2- 172.21.15.45:0/2376589183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b64775500 0x561b647f3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.083+0000 7f7d2357f740 1 --2- 172.21.15.45:0/2376589183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561b6475bc70 0x561b647767a0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.083+0000 7f7d2357f740 1 --2- 172.21.15.45:0/2376589183 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6475b890 0x561b64732d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.083+0000 7f7cfea00640 0 client.5895 ms_handle_reset on v2:172.21.15.103:6835/1576112884 2024-04-16T16:15:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.083+0000 7f7d2357f740 1 -- 172.21.15.45:0/2376589183 >> 172.21.15.45:0/2376589183 conn(0x561b64734ab0 msgr2=0x561b64773080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.084+0000 7f7d2357f740 1 -- 172.21.15.45:0/2376589183 shutdown_connections 2024-04-16T16:15:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.084+0000 7f7d2357f740 1 -- 172.21.15.45:0/2376589183 wait complete. 2024-04-16T16:15:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.084+0000 7f7d2357f740 20 client.5895 trim_cache size 0 max 16384 2024-04-16T16:15:46.087 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:15:46.087 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "748c4b85-0190-4c59-80ad-098a6a5007c0", 2024-04-16T16:15:46.087 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-04-16T16:15:46.087 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 5191, 2024-04-16T16:15:46.087 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-04-16T16:15:46.087 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-04-16T16:15:46.087 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "fs2", 2024-04-16T16:15:46.087 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 4.8490650459999998, 2024-04-16T16:15:46.087 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 40, 2024-04-16T16:15:46.087 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 54, 2024-04-16T16:15:46.087 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 54, 2024-04-16T16:15:46.087 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 4.8664984950000001 2024-04-16T16:15:46.087 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:15:46.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.084+0000 7f7d1f200640 1 -- 172.21.15.45:0/2781652657 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7cf806ca60 msgr2=0x7f7cf806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.084+0000 7f7d1f200640 1 --2- 172.21.15.45:0/2781652657 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7cf806ca60 0x7f7cf806ef20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d180b58f0 tx=0x7f7d0c008040 comp rx=0 tx=0).stop 2024-04-16T16:15:46.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.084+0000 7f7d1f200640 1 -- 172.21.15.45:0/2781652657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d180d5520 msgr2=0x7f7d181f9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.084+0000 7f7d1f200640 1 --2- 172.21.15.45:0/2781652657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d180d5520 0x7f7d181f9340 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7d0402b950 tx=0x7f7d040046c0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.084+0000 7f7d1f200640 1 -- 172.21.15.45:0/2781652657 shutdown_connections 2024-04-16T16:15:46.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.084+0000 7f7d1f200640 1 --2- 172.21.15.45:0/2781652657 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f7cf806ca60 0x7f7cf806ef20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.084+0000 7f7d1f200640 1 --2- 172.21.15.45:0/2781652657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d18155f60 0x7f7d1815aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.084+0000 7f7d1f200640 1 --2- 172.21.15.45:0/2781652657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d18162090 0x7f7d180cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.084+0000 7f7d1f200640 1 --2- 172.21.15.45:0/2781652657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d180d5520 0x7f7d181f9340 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.085+0000 7f7d1f200640 1 -- 172.21.15.45:0/2781652657 >> 172.21.15.45:0/2781652657 conn(0x7f7d180d1940 msgr2=0x7f7d180cf130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:46.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.085+0000 7f7d1f200640 1 -- 172.21.15.45:0/2781652657 shutdown_connections 2024-04-16T16:15:46.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.085+0000 7f7d1f200640 1 -- 172.21.15.45:0/2781652657 wait complete. 2024-04-16T16:15:46.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=fs2_data' 2024-04-16T16:15:46.240 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-16T16:15:46.240+0000 7f5a38600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:15:46.242 INFO:tasks.ceph.mds.b.smithi103.stderr:2024-04-16T16:15:46.242+0000 7f94b5000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:15:46.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.330+0000 7f33bca00640 1 Processor -- start 2024-04-16T16:15:46.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.330+0000 7f33bca00640 1 -- start start 2024-04-16T16:15:46.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.332+0000 7f33bca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b815e970 0x7f33b80ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.332+0000 7f33bca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f33b806ce10 0x7f33b806e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.332+0000 7f33bca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33b806d2a0 con 0x7f33b806ec80 2024-04-16T16:15:46.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.332+0000 7f33bca00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f33b8070650 con 0x7f33b806ce10 2024-04-16T16:15:46.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.332+0000 7f33bca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f33b806bb60 con 0x7f33b815e970 2024-04-16T16:15:46.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.333+0000 7f33b7400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b815e970 0x7f33b80ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.333+0000 7f33b7400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b815e970 0x7f33b80ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:36316/0 (socket says 172.21.15.45:36316) 2024-04-16T16:15:46.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.333+0000 7f33b7400640 1 -- 172.21.15.45:0/1084543143 learned_addr learned my addr 172.21.15.45:0/1084543143 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:46.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.333+0000 7f33b7e00640 1 --1- 172.21.15.45:0/1084543143 >> v1:172.21.15.103:6789/0 conn(0x7f33b806ec80 0x7f33b806bff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:50666/0 (socket says 172.21.15.45:50666) 2024-04-16T16:15:46.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.333+0000 7f33b6a00640 1 --2- 172.21.15.45:0/1084543143 >> v2:172.21.15.103:3300/0 conn(0x7f33b806ce10 0x7f33b806e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.334+0000 7f33b7400640 1 -- 172.21.15.45:0/1084543143 >> v2:172.21.15.103:3300/0 conn(0x7f33b806ce10 msgr2=0x7f33b806e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:46.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.334+0000 7f33b7400640 1 --2- 172.21.15.45:0/1084543143 >> v2:172.21.15.103:3300/0 conn(0x7f33b806ce10 0x7f33b806e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.334+0000 7f33b7400640 1 -- 172.21.15.45:0/1084543143 >> v1:172.21.15.103:6789/0 conn(0x7f33b806ec80 legacy=0x7f33b806bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.334+0000 7f33b7400640 1 -- 172.21.15.45:0/1084543143 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b8070090 con 0x7f33b815e970 2024-04-16T16:15:46.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.334+0000 7f33b6a00640 1 --2- 172.21.15.45:0/1084543143 >> v2:172.21.15.103:3300/0 conn(0x7f33b806ce10 0x7f33b806e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:15:46.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.335+0000 7f33b7400640 1 --2- 172.21.15.45:0/1084543143 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b815e970 0x7f33b80ce850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f33a8009870 tx=0x7f33a802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=984233cc63a70656 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:46.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.336+0000 7f33b6000640 1 -- 172.21.15.45:0/1084543143 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a8034070 con 0x7f33b815e970 2024-04-16T16:15:46.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.336+0000 7f33b6000640 1 -- 172.21.15.45:0/1084543143 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a8030840 con 0x7f33b815e970 2024-04-16T16:15:46.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.336+0000 7f33b6000640 1 -- 172.21.15.45:0/1084543143 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a8030c10 con 0x7f33b815e970 2024-04-16T16:15:46.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.336+0000 7f33bca00640 1 -- 172.21.15.45:0/1084543143 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b815e970 msgr2=0x7f33b80ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.336+0000 7f33bca00640 1 --2- 172.21.15.45:0/1084543143 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b815e970 0x7f33b80ce850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f33a8009870 tx=0x7f33a802b980 comp rx=0 tx=0).stop 2024-04-16T16:15:46.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.337+0000 7f33bca00640 1 -- 172.21.15.45:0/1084543143 shutdown_connections 2024-04-16T16:15:46.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.337+0000 7f33bca00640 1 --2- 172.21.15.45:0/1084543143 >> v2:172.21.15.103:3300/0 conn(0x7f33b806ce10 0x7f33b806e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.337+0000 7f33bca00640 1 --2- 172.21.15.45:0/1084543143 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b815e970 0x7f33b80ce850 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.337+0000 7f33bca00640 1 -- 172.21.15.45:0/1084543143 >> 172.21.15.45:0/1084543143 conn(0x7f33b80b36b0 msgr2=0x7f33b80cd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:46.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.337+0000 7f33bca00640 1 -- 172.21.15.45:0/1084543143 shutdown_connections 2024-04-16T16:15:46.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.337+0000 7f33bca00640 1 -- 172.21.15.45:0/1084543143 wait complete. 2024-04-16T16:15:46.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.338+0000 7f33bca00640 1 Processor -- start 2024-04-16T16:15:46.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.339+0000 7f33bca00640 1 -- start start 2024-04-16T16:15:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.339+0000 7f33bca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b806ec80 0x7f33b815fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.340+0000 7f33bca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b815e970 0x7f33b815c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.340+0000 7f33bca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33b8161b50 0x7f33b815cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.340+0000 7f33bca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f33b815eed0 con 0x7f33b806ec80 2024-04-16T16:15:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.340+0000 7f33bca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33b815ed50 con 0x7f33b8161b50 2024-04-16T16:15:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.340+0000 7f33bca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f33b80b02d0 con 0x7f33b815e970 2024-04-16T16:15:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.340+0000 7f33b7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b806ec80 0x7f33b815fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.340+0000 7f33b7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b806ec80 0x7f33b815fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43210/0 (socket says 172.21.15.45:43210) 2024-04-16T16:15:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.340+0000 7f33b7400640 1 -- 172.21.15.45:0/2488745939 learned_addr learned my addr 172.21.15.45:0/2488745939 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:46.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.340+0000 7f33b7e00640 1 --2- 172.21.15.45:0/2488745939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33b8161b50 0x7f33b815cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.340+0000 7f33b6a00640 1 --2- 172.21.15.45:0/2488745939 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b815e970 0x7f33b815c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.340+0000 7f33b7400640 1 -- 172.21.15.45:0/2488745939 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b815e970 msgr2=0x7f33b815c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.340+0000 7f33b7400640 1 --2- 172.21.15.45:0/2488745939 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b815e970 0x7f33b815c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.340+0000 7f33b7400640 1 -- 172.21.15.45:0/2488745939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33b8161b50 msgr2=0x7f33b815cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.340+0000 7f33b7400640 1 --2- 172.21.15.45:0/2488745939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33b8161b50 0x7f33b815cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.340+0000 7f33b7400640 1 -- 172.21.15.45:0/2488745939 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b81610b0 con 0x7f33b806ec80 2024-04-16T16:15:46.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.341+0000 7f33b7400640 1 --2- 172.21.15.45:0/2488745939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b806ec80 0x7f33b815fb70 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f33a802b9c0 tx=0x7f33a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:46.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.341+0000 7f33af400640 1 -- 172.21.15.45:0/2488745939 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a8034040 con 0x7f33b806ec80 2024-04-16T16:15:46.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.341+0000 7f33af400640 1 -- 172.21.15.45:0/2488745939 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a803ada0 con 0x7f33b806ec80 2024-04-16T16:15:46.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.341+0000 7f33af400640 1 -- 172.21.15.45:0/2488745939 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a8044610 con 0x7f33b806ec80 2024-04-16T16:15:46.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.341+0000 7f33bca00640 1 -- 172.21.15.45:0/2488745939 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33b81e9340 con 0x7f33b806ec80 2024-04-16T16:15:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.341+0000 7f33bca00640 1 -- 172.21.15.45:0/2488745939 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33b814c600 con 0x7f33b806ec80 2024-04-16T16:15:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.342+0000 7f33bca00640 1 -- 172.21.15.45:0/2488745939 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33b8075780 con 0x7f33b806ec80 2024-04-16T16:15:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.343+0000 7f33af400640 1 -- 172.21.15.45:0/2488745939 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f33a804b060 con 0x7f33b806ec80 2024-04-16T16:15:46.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.344+0000 7f33af400640 1 --2- 172.21.15.45:0/2488745939 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f339806ca90 0x7f339806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.344+0000 7f33af400640 1 -- 172.21.15.45:0/2488745939 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7f33a80ae3a0 con 0x7f33b806ec80 2024-04-16T16:15:46.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.345+0000 7f33b6a00640 1 --2- 172.21.15.45:0/2488745939 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f339806ca90 0x7f339806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.346+0000 7f33b6a00640 1 --2- 172.21.15.45:0/2488745939 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f339806ca90 0x7f339806ef50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f33b815d630 tx=0x7f33a4008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:46.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.349+0000 7f33af400640 1 -- 172.21.15.45:0/2488745939 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f33a803f070 con 0x7f33b806ec80 2024-04-16T16:15:46.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.631+0000 7f33bca00640 1 -- 172.21.15.45:0/2488745939 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs2_data"]} v 0) -- 0x7f33b8159fb0 con 0x7f33b806ec80 2024-04-16T16:15:46.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.636+0000 7f33af400640 1 -- 172.21.15.45:0/2488745939 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs2_data"]}]=0 updated caps for client.0 v16) ==== 181+0+138 (secure 0 0 0) 0x7f33a807c010 con 0x7f33b806ec80 2024-04-16T16:15:46.636 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-04-16T16:15:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ== 2024-04-16T16:15:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:15:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:15:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw pool=fs2_data" 2024-04-16T16:15:46.637 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2024-04-16T16:15:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.639+0000 7f33bca00640 1 -- 172.21.15.45:0/2488745939 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f339806ca90 msgr2=0x7f339806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.639+0000 7f33bca00640 1 --2- 172.21.15.45:0/2488745939 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f339806ca90 0x7f339806ef50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f33b815d630 tx=0x7f33a4008040 comp rx=0 tx=0).stop 2024-04-16T16:15:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.639+0000 7f33bca00640 1 -- 172.21.15.45:0/2488745939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b806ec80 msgr2=0x7f33b815fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.639+0000 7f33bca00640 1 --2- 172.21.15.45:0/2488745939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b806ec80 0x7f33b815fb70 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f33a802b9c0 tx=0x7f33a80046c0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.639+0000 7f33bca00640 1 -- 172.21.15.45:0/2488745939 shutdown_connections 2024-04-16T16:15:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.639+0000 7f33bca00640 1 --2- 172.21.15.45:0/2488745939 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f339806ca90 0x7f339806ef50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.639+0000 7f33bca00640 1 --2- 172.21.15.45:0/2488745939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33b8161b50 0x7f33b815cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.639+0000 7f33bca00640 1 --2- 172.21.15.45:0/2488745939 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b815e970 0x7f33b815c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.639+0000 7f33bca00640 1 --2- 172.21.15.45:0/2488745939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b806ec80 0x7f33b815fb70 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.639+0000 7f33bca00640 1 -- 172.21.15.45:0/2488745939 >> 172.21.15.45:0/2488745939 conn(0x7f33b80b36b0 msgr2=0x7f33b80ae7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:46.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.639+0000 7f33bca00640 1 -- 172.21.15.45:0/2488745939 shutdown_connections 2024-04-16T16:15:46.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.639+0000 7f33bca00640 1 -- 172.21.15.45:0/2488745939 wait complete. 2024-04-16T16:15:46.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=fs2_data' 2024-04-16T16:15:46.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.880+0000 7fb21e800640 1 Processor -- start 2024-04-16T16:15:46.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.880+0000 7fb21e800640 1 -- start start 2024-04-16T16:15:46.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.882+0000 7fb21e800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb2181452e0 0x7fb21806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.882+0000 7fb21e800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb218071e60 con 0x7fb2180cecc0 2024-04-16T16:15:46.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.882+0000 7fb21e800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb218072140 con 0x7fb21806e920 2024-04-16T16:15:46.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.882+0000 7fb21e800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb218072f00 con 0x7fb2181452e0 2024-04-16T16:15:46.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.882+0000 7fb21d400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb2181452e0 0x7fb21806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.882+0000 7fb21de00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb2180cecc0 0x7fb21815cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53570/0 (socket says 172.21.15.45:53570) 2024-04-16T16:15:46.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.882+0000 7fb21d400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb2181452e0 0x7fb21806e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43226/0 (socket says 172.21.15.45:43226) 2024-04-16T16:15:46.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.882+0000 7fb21de00640 1 -- 172.21.15.45:0/2546495373 learned_addr learned my addr 172.21.15.45:0/2546495373 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:46.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.882+0000 7fb21d400640 1 -- 172.21.15.45:0/2546495373 >> v1:172.21.15.103:6789/0 conn(0x7fb21806e920 legacy=0x7fb2180ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:46.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.882+0000 7fb21d400640 1 -- 172.21.15.45:0/2546495373 >> v1:172.21.15.45:6789/0 conn(0x7fb2180cecc0 legacy=0x7fb21815cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.882+0000 7fb21d400640 1 -- 172.21.15.45:0/2546495373 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb218072420 con 0x7fb2181452e0 2024-04-16T16:15:46.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.884+0000 7fb21d400640 1 --2- 172.21.15.45:0/2546495373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2181452e0 0x7fb21806e3e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0097a0 tx=0x7fb20c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aee6fca4c214d50 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:46.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.884+0000 7fb217e00640 1 -- 172.21.15.45:0/2546495373 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c034070 con 0x7fb2181452e0 2024-04-16T16:15:46.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.884+0000 7fb217e00640 1 -- 172.21.15.45:0/2546495373 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb20c030610 con 0x7fb2181452e0 2024-04-16T16:15:46.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.884+0000 7fb217e00640 1 -- 172.21.15.45:0/2546495373 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c0309e0 con 0x7fb2181452e0 2024-04-16T16:15:46.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.884+0000 7fb21e800640 1 -- 172.21.15.45:0/2546495373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2181452e0 msgr2=0x7fb21806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.884+0000 7fb21e800640 1 --2- 172.21.15.45:0/2546495373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2181452e0 0x7fb21806e3e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0097a0 tx=0x7fb20c02b960 comp rx=0 tx=0).stop 2024-04-16T16:15:46.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.885+0000 7fb21e800640 1 -- 172.21.15.45:0/2546495373 shutdown_connections 2024-04-16T16:15:46.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.885+0000 7fb21e800640 1 --2- 172.21.15.45:0/2546495373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2181452e0 0x7fb21806e3e0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.885+0000 7fb21e800640 1 -- 172.21.15.45:0/2546495373 >> 172.21.15.45:0/2546495373 conn(0x7fb2180b3bb0 msgr2=0x7fb2180b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:46.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.885+0000 7fb21e800640 1 -- 172.21.15.45:0/2546495373 shutdown_connections 2024-04-16T16:15:46.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.885+0000 7fb21e800640 1 -- 172.21.15.45:0/2546495373 wait complete. 2024-04-16T16:15:46.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.886+0000 7fb21e800640 1 Processor -- start 2024-04-16T16:15:46.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.886+0000 7fb21e800640 1 -- start start 2024-04-16T16:15:46.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.887+0000 7fb21e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb21806e920 0x7fb21812e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.887+0000 7fb21e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2180cecc0 0x7fb21812ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.887+0000 7fb21e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2181452e0 0x7fb2181f7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.887+0000 7fb21e800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb21806ed90 con 0x7fb21806e920 2024-04-16T16:15:46.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.887+0000 7fb21e800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2180cf0e0 con 0x7fb2180cecc0 2024-04-16T16:15:46.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.887+0000 7fb21e800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb2180729e0 con 0x7fb2181452e0 2024-04-16T16:15:46.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.888+0000 7fb21d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb21806e920 0x7fb21812e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.888+0000 7fb21d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb21806e920 0x7fb21812e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43242/0 (socket says 172.21.15.45:43242) 2024-04-16T16:15:46.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.888+0000 7fb21d400640 1 -- 172.21.15.45:0/1110271274 learned_addr learned my addr 172.21.15.45:0/1110271274 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:46.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.888+0000 7fb21d400640 1 -- 172.21.15.45:0/1110271274 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2181452e0 msgr2=0x7fb2181f7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.888+0000 7fb21d400640 1 --2- 172.21.15.45:0/1110271274 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2181452e0 0x7fb2181f7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.888+0000 7fb21ca00640 1 --2- 172.21.15.45:0/1110271274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2180cecc0 0x7fb21812ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.888+0000 7fb21d400640 1 -- 172.21.15.45:0/1110271274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2180cecc0 msgr2=0x7fb21812ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:46.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.888+0000 7fb21d400640 1 --2- 172.21.15.45:0/1110271274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2180cecc0 0x7fb21812ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:46.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.888+0000 7fb21d400640 1 -- 172.21.15.45:0/1110271274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2181e8d80 con 0x7fb21806e920 2024-04-16T16:15:46.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.888+0000 7fb21ca00640 1 --2- 172.21.15.45:0/1110271274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2180cecc0 0x7fb21812ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:46.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.888+0000 7fb21d400640 1 --2- 172.21.15.45:0/1110271274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb21806e920 0x7fb21812e930 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0098c0 tx=0x7fb20c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:46.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.889+0000 7fb215600640 1 -- 172.21.15.45:0/1110271274 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c034040 con 0x7fb21806e920 2024-04-16T16:15:46.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.889+0000 7fb215600640 1 -- 172.21.15.45:0/1110271274 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb20c0399d0 con 0x7fb21806e920 2024-04-16T16:15:46.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.889+0000 7fb215600640 1 -- 172.21.15.45:0/1110271274 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c039de0 con 0x7fb21806e920 2024-04-16T16:15:46.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.889+0000 7fb21e800640 1 -- 172.21.15.45:0/1110271274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2181e87c0 con 0x7fb21806e920 2024-04-16T16:15:46.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.889+0000 7fb21e800640 1 -- 172.21.15.45:0/1110271274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb21812c9b0 con 0x7fb21806e920 2024-04-16T16:15:46.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.890+0000 7fb21e800640 1 -- 172.21.15.45:0/1110271274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb218075780 con 0x7fb21806e920 2024-04-16T16:15:46.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.891+0000 7fb215600640 1 -- 172.21.15.45:0/1110271274 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fb20c037070 con 0x7fb21806e920 2024-04-16T16:15:46.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.892+0000 7fb215600640 1 --2- 172.21.15.45:0/1110271274 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb1f806ca90 0x7fb1f806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:46.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.892+0000 7fb215600640 1 -- 172.21.15.45:0/1110271274 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7fb20c0ae500 con 0x7fb21806e920 2024-04-16T16:15:46.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.892+0000 7fb21ca00640 1 --2- 172.21.15.45:0/1110271274 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb1f806ca90 0x7fb1f806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:46.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.894+0000 7fb21ca00640 1 --2- 172.21.15.45:0/1110271274 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb1f806ca90 0x7fb1f806ef50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb218132fb0 tx=0x7fb204008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:46.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:46.897+0000 7fb215600640 1 -- 172.21.15.45:0/1110271274 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb20c07c0d0 con 0x7fb21806e920 2024-04-16T16:15:47.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.175+0000 7fb21e800640 1 -- 172.21.15.45:0/1110271274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs2_data"]} v 0) -- 0x7fb218141860 con 0x7fb21806e920 2024-04-16T16:15:47.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.179+0000 7fb215600640 1 -- 172.21.15.45:0/1110271274 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs2_data"]}]=0 updated caps for client.1 v17) ==== 181+0+138 (secure 0 0 0) 0x7fb20c07b9d0 con 0x7fb21806e920 2024-04-16T16:15:47.180 INFO:teuthology.orchestra.run.smithi045.stdout:[client.1] 2024-04-16T16:15:47.180 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mYpVUGhAAYdR+40O6zbWqgbI55F0FYA== 2024-04-16T16:15:47.180 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:15:47.180 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:15:47.180 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw pool=fs2_data" 2024-04-16T16:15:47.180 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.1 2024-04-16T16:15:47.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.182+0000 7fb21e800640 1 -- 172.21.15.45:0/1110271274 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb1f806ca90 msgr2=0x7fb1f806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:47.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.182+0000 7fb21e800640 1 --2- 172.21.15.45:0/1110271274 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb1f806ca90 0x7fb1f806ef50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb218132fb0 tx=0x7fb204008040 comp rx=0 tx=0).stop 2024-04-16T16:15:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.182+0000 7fb21e800640 1 -- 172.21.15.45:0/1110271274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb21806e920 msgr2=0x7fb21812e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.182+0000 7fb21e800640 1 --2- 172.21.15.45:0/1110271274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb21806e920 0x7fb21812e930 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0098c0 tx=0x7fb20c0046c0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.182+0000 7fb21e800640 1 -- 172.21.15.45:0/1110271274 shutdown_connections 2024-04-16T16:15:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.182+0000 7fb21e800640 1 --2- 172.21.15.45:0/1110271274 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2181452e0 0x7fb2181f7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.182+0000 7fb21e800640 1 --2- 172.21.15.45:0/1110271274 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb1f806ca90 0x7fb1f806ef50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.182+0000 7fb21e800640 1 --2- 172.21.15.45:0/1110271274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2180cecc0 0x7fb21812ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.182+0000 7fb21e800640 1 --2- 172.21.15.45:0/1110271274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb21806e920 0x7fb21812e930 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.182+0000 7fb21e800640 1 -- 172.21.15.45:0/1110271274 >> 172.21.15.45:0/1110271274 conn(0x7fb2180b3bb0 msgr2=0x7fb218143770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.182+0000 7fb21e800640 1 -- 172.21.15.45:0/1110271274 shutdown_connections 2024-04-16T16:15:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.182+0000 7fb21e800640 1 -- 172.21.15.45:0/1110271274 wait complete. 2024-04-16T16:15:47.200 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=fs2_data' 2024-04-16T16:15:47.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.426+0000 7fb76b200640 1 Processor -- start 2024-04-16T16:15:47.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.426+0000 7fb76b200640 1 -- start start 2024-04-16T16:15:47.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.428+0000 7fb76b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb76c1452e0 0x7fb76c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:47.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.428+0000 7fb76b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb76c06e920 0x7fb76c0ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:47.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.428+0000 7fb76b200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb76c0ceaf0 0x7fb76c160270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:47.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.428+0000 7fb76b200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb76c072f00 con 0x7fb76c0ceaf0 2024-04-16T16:15:47.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.428+0000 7fb76b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb76c06ed60 con 0x7fb76c1452e0 2024-04-16T16:15:47.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.428+0000 7fb76b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb76c071e60 con 0x7fb76c06e920 2024-04-16T16:15:47.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.429+0000 7fb76a800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb76c0ceaf0 0x7fb76c160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:47.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.429+0000 7fb76a800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb76c0ceaf0 0x7fb76c160270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43254/0 (socket says 172.21.15.45:43254) 2024-04-16T16:15:47.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.429+0000 7fb76a800640 1 -- 172.21.15.45:0/3766369560 learned_addr learned my addr 172.21.15.45:0/3766369560 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:47.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.429+0000 7fb769e00640 1 --2- 172.21.15.45:0/3766369560 >> v2:172.21.15.103:3300/0 conn(0x7fb76c1452e0 0x7fb76c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:47.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.429+0000 7fb769400640 1 --2- 172.21.15.45:0/3766369560 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb76c06e920 0x7fb76c0ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:47.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.429+0000 7fb76a800640 1 -- 172.21.15.45:0/3766369560 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb76c06e920 msgr2=0x7fb76c0ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:47.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.429+0000 7fb76a800640 1 --2- 172.21.15.45:0/3766369560 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb76c06e920 0x7fb76c0ce5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.429+0000 7fb76a800640 1 -- 172.21.15.45:0/3766369560 >> v2:172.21.15.103:3300/0 conn(0x7fb76c1452e0 msgr2=0x7fb76c06e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:47.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.429+0000 7fb76a800640 1 --2- 172.21.15.45:0/3766369560 >> v2:172.21.15.103:3300/0 conn(0x7fb76c1452e0 0x7fb76c06e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.429+0000 7fb76a800640 1 -- 172.21.15.45:0/3766369560 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb76c072140 con 0x7fb76c0ceaf0 2024-04-16T16:15:47.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.429+0000 7fb769400640 1 --2- 172.21.15.45:0/3766369560 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb76c06e920 0x7fb76c0ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.431+0000 7fb76a800640 1 --2- 172.21.15.45:0/3766369560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb76c0ceaf0 0x7fb76c160270 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0702a0 tx=0x7fb76402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88526c8c15615bd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.431+0000 7fb768a00640 1 -- 172.21.15.45:0/3766369560 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76402cb00 con 0x7fb76c0ceaf0 2024-04-16T16:15:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.431+0000 7fb768a00640 1 -- 172.21.15.45:0/3766369560 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb76402cca0 con 0x7fb76c0ceaf0 2024-04-16T16:15:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.431+0000 7fb768a00640 1 -- 172.21.15.45:0/3766369560 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb764034610 con 0x7fb76c0ceaf0 2024-04-16T16:15:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.432+0000 7fb76b200640 1 -- 172.21.15.45:0/3766369560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb76c0ceaf0 msgr2=0x7fb76c160270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.432+0000 7fb76b200640 1 --2- 172.21.15.45:0/3766369560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb76c0ceaf0 0x7fb76c160270 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0702a0 tx=0x7fb76402bf60 comp rx=0 tx=0).stop 2024-04-16T16:15:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.432+0000 7fb76b200640 1 -- 172.21.15.45:0/3766369560 shutdown_connections 2024-04-16T16:15:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.432+0000 7fb76b200640 1 --2- 172.21.15.45:0/3766369560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb76c0ceaf0 0x7fb76c160270 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.432+0000 7fb76b200640 1 --2- 172.21.15.45:0/3766369560 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb76c06e920 0x7fb76c0ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.432+0000 7fb76b200640 1 --2- 172.21.15.45:0/3766369560 >> v2:172.21.15.103:3300/0 conn(0x7fb76c1452e0 0x7fb76c06e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.432+0000 7fb76b200640 1 -- 172.21.15.45:0/3766369560 >> 172.21.15.45:0/3766369560 conn(0x7fb76c0b3bb0 msgr2=0x7fb76c0b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:47.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.432+0000 7fb76b200640 1 -- 172.21.15.45:0/3766369560 shutdown_connections 2024-04-16T16:15:47.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.432+0000 7fb76b200640 1 -- 172.21.15.45:0/3766369560 wait complete. 2024-04-16T16:15:47.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.434+0000 7fb76b200640 1 Processor -- start 2024-04-16T16:15:47.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.434+0000 7fb76b200640 1 -- start start 2024-04-16T16:15:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.435+0000 7fb76b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb76c06e920 0x7fb76c1f54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.435+0000 7fb76b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76c0ceaf0 0x7fb76c15bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.435+0000 7fb76b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb76c1452e0 0x7fb76c1f4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.435+0000 7fb76b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c0729e0 con 0x7fb76c1452e0 2024-04-16T16:15:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.435+0000 7fb76b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c072700 con 0x7fb76c0ceaf0 2024-04-16T16:15:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.435+0000 7fb76b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb76c0cef60 con 0x7fb76c06e920 2024-04-16T16:15:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.435+0000 7fb76a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb76c1452e0 0x7fb76c1f4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.435+0000 7fb76a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb76c1452e0 0x7fb76c1f4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43260/0 (socket says 172.21.15.45:43260) 2024-04-16T16:15:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.435+0000 7fb76a800640 1 -- 172.21.15.45:0/1813599232 learned_addr learned my addr 172.21.15.45:0/1813599232 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.436+0000 7fb769e00640 1 --2- 172.21.15.45:0/1813599232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb76c06e920 0x7fb76c1f54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.436+0000 7fb769400640 1 --2- 172.21.15.45:0/1813599232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76c0ceaf0 0x7fb76c15bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.436+0000 7fb76a800640 1 -- 172.21.15.45:0/1813599232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb76c06e920 msgr2=0x7fb76c1f54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.436+0000 7fb76a800640 1 --2- 172.21.15.45:0/1813599232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb76c06e920 0x7fb76c1f54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.436+0000 7fb76a800640 1 -- 172.21.15.45:0/1813599232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76c0ceaf0 msgr2=0x7fb76c15bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.436+0000 7fb76a800640 1 --2- 172.21.15.45:0/1813599232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76c0ceaf0 0x7fb76c15bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.436+0000 7fb76a800640 1 -- 172.21.15.45:0/1813599232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb76c130d30 con 0x7fb76c1452e0 2024-04-16T16:15:47.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.436+0000 7fb76a800640 1 --2- 172.21.15.45:0/1813599232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb76c1452e0 0x7fb76c1f4bc0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb76402c570 tx=0x7fb764004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:47.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.437+0000 7fb762000640 1 -- 172.21.15.45:0/1813599232 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb764043020 con 0x7fb76c1452e0 2024-04-16T16:15:47.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.437+0000 7fb762000640 1 -- 172.21.15.45:0/1813599232 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb76402ce50 con 0x7fb76c1452e0 2024-04-16T16:15:47.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.437+0000 7fb76b200640 1 -- 172.21.15.45:0/1813599232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb76c131010 con 0x7fb76c1452e0 2024-04-16T16:15:47.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.437+0000 7fb762000640 1 -- 172.21.15.45:0/1813599232 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb764009510 con 0x7fb76c1452e0 2024-04-16T16:15:47.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.437+0000 7fb76b200640 1 -- 172.21.15.45:0/1813599232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb76c1315d0 con 0x7fb76c1452e0 2024-04-16T16:15:47.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.438+0000 7fb76b200640 1 -- 172.21.15.45:0/1813599232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb76c075780 con 0x7fb76c1452e0 2024-04-16T16:15:47.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.439+0000 7fb762000640 1 -- 172.21.15.45:0/1813599232 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fb764046020 con 0x7fb76c1452e0 2024-04-16T16:15:47.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.441+0000 7fb762000640 1 --2- 172.21.15.45:0/1813599232 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb74c06ca60 0x7fb74c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:47.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.441+0000 7fb762000640 1 -- 172.21.15.45:0/1813599232 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7fb7640aeaa0 con 0x7fb76c1452e0 2024-04-16T16:15:47.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.442+0000 7fb769e00640 1 --2- 172.21.15.45:0/1813599232 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb74c06ca60 0x7fb74c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:47.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.443+0000 7fb769e00640 1 --2- 172.21.15.45:0/1813599232 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb74c06ca60 0x7fb74c06ef20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb75c006fd0 tx=0x7fb75c008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:47.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.446+0000 7fb762000640 1 -- 172.21.15.45:0/1813599232 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb76404d030 con 0x7fb76c1452e0 2024-04-16T16:15:47.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.728+0000 7fb76b200640 1 -- 172.21.15.45:0/1813599232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs2_data"]} v 0) -- 0x7fb76c0b2d70 con 0x7fb76c1452e0 2024-04-16T16:15:47.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.732+0000 7fb762000640 1 -- 172.21.15.45:0/1813599232 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs2_data"]}]=0 updated caps for client.2 v18) ==== 181+0+138 (secure 0 0 0) 0x7fb76407c780 con 0x7fb76c1452e0 2024-04-16T16:15:47.733 INFO:teuthology.orchestra.run.smithi045.stdout:[client.2] 2024-04-16T16:15:47.733 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mWdlUDxAAuO2FsGH1/E+qP1xcyUEe3A== 2024-04-16T16:15:47.733 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:15:47.733 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:15:47.733 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw pool=fs2_data" 2024-04-16T16:15:47.733 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.2 2024-04-16T16:15:47.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.736+0000 7fb76b200640 1 -- 172.21.15.45:0/1813599232 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb74c06ca60 msgr2=0x7fb74c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:47.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.736+0000 7fb76b200640 1 --2- 172.21.15.45:0/1813599232 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb74c06ca60 0x7fb74c06ef20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb75c006fd0 tx=0x7fb75c008040 comp rx=0 tx=0).stop 2024-04-16T16:15:47.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.736+0000 7fb76b200640 1 -- 172.21.15.45:0/1813599232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb76c1452e0 msgr2=0x7fb76c1f4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:47.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.736+0000 7fb76b200640 1 --2- 172.21.15.45:0/1813599232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb76c1452e0 0x7fb76c1f4bc0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb76402c570 tx=0x7fb764004a40 comp rx=0 tx=0).stop 2024-04-16T16:15:47.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.736+0000 7fb76b200640 1 -- 172.21.15.45:0/1813599232 shutdown_connections 2024-04-16T16:15:47.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.736+0000 7fb76b200640 1 --2- 172.21.15.45:0/1813599232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb76c1452e0 0x7fb76c1f4bc0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.736+0000 7fb76b200640 1 --2- 172.21.15.45:0/1813599232 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fb74c06ca60 0x7fb74c06ef20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.736+0000 7fb76b200640 1 --2- 172.21.15.45:0/1813599232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76c0ceaf0 0x7fb76c15bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.736+0000 7fb76b200640 1 --2- 172.21.15.45:0/1813599232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb76c06e920 0x7fb76c1f54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.736+0000 7fb76b200640 1 -- 172.21.15.45:0/1813599232 >> 172.21.15.45:0/1813599232 conn(0x7fb76c0b3bb0 msgr2=0x7fb76c15fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:47.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.736+0000 7fb76b200640 1 -- 172.21.15.45:0/1813599232 shutdown_connections 2024-04-16T16:15:47.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.736+0000 7fb76b200640 1 -- 172.21.15.45:0/1813599232 wait complete. 2024-04-16T16:15:47.755 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=fs2_data' 2024-04-16T16:15:47.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.988+0000 7f6d75c00640 1 Processor -- start 2024-04-16T16:15:47.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.988+0000 7f6d75c00640 1 -- start start 2024-04-16T16:15:47.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.990+0000 7f6d75c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6d70147ff0 0x7f6d7006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:47.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.990+0000 7f6d75c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d7006e920 0x7f6d7015eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:47.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.990+0000 7f6d75c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d7015fc90 con 0x7f6d7015f430 2024-04-16T16:15:47.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.990+0000 7f6d75c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f6d701624e0 con 0x7f6d70147ff0 2024-04-16T16:15:47.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.990+0000 7f6d75c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6d70162730 con 0x7f6d7006e920 2024-04-16T16:15:47.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.990+0000 7f6d6fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6d70147ff0 0x7f6d7006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:47.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.990+0000 7f6d6fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6d70147ff0 0x7f6d7006e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43270/0 (socket says 172.21.15.45:43270) 2024-04-16T16:15:47.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.990+0000 7f6d6fe00640 1 -- 172.21.15.45:0/442612557 learned_addr learned my addr 172.21.15.45:0/442612557 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:47.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.991+0000 7f6d6f400640 1 --2- 172.21.15.45:0/442612557 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d7006e920 0x7f6d7015eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:47.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.991+0000 7f6d6fe00640 1 -- 172.21.15.45:0/442612557 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d7006e920 msgr2=0x7f6d7015eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:47.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.991+0000 7f6d6fe00640 1 --2- 172.21.15.45:0/442612557 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d7006e920 0x7f6d7015eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.991+0000 7f6d6fe00640 1 -- 172.21.15.45:0/442612557 >> v1:172.21.15.103:6789/0 conn(0x7f6d7015f430 legacy=0x7f6d70071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:47.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.991+0000 7f6d6fe00640 1 -- 172.21.15.45:0/442612557 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d70162a10 con 0x7f6d70147ff0 2024-04-16T16:15:47.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.992+0000 7f6d6fe00640 1 --2- 172.21.15.45:0/442612557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d70147ff0 0x7f6d7006e3e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6d700b6990 tx=0x7f6d6002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dbbc017cb928126 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:47.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.993+0000 7f6d6ea00640 1 -- 172.21.15.45:0/442612557 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d600315a0 con 0x7f6d70147ff0 2024-04-16T16:15:47.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.993+0000 7f6d6ea00640 1 -- 172.21.15.45:0/442612557 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d60037070 con 0x7f6d70147ff0 2024-04-16T16:15:47.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.993+0000 7f6d6ea00640 1 -- 172.21.15.45:0/442612557 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d60031960 con 0x7f6d70147ff0 2024-04-16T16:15:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.994+0000 7f6d75c00640 1 -- 172.21.15.45:0/442612557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d70147ff0 msgr2=0x7f6d7006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.994+0000 7f6d75c00640 1 --2- 172.21.15.45:0/442612557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d70147ff0 0x7f6d7006e3e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6d700b6990 tx=0x7f6d6002b840 comp rx=0 tx=0).stop 2024-04-16T16:15:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.994+0000 7f6d75c00640 1 -- 172.21.15.45:0/442612557 shutdown_connections 2024-04-16T16:15:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.994+0000 7f6d75c00640 1 --2- 172.21.15.45:0/442612557 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d7006e920 0x7f6d7015eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.994+0000 7f6d75c00640 1 --2- 172.21.15.45:0/442612557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d70147ff0 0x7f6d7006e3e0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.994+0000 7f6d75c00640 1 -- 172.21.15.45:0/442612557 >> 172.21.15.45:0/442612557 conn(0x7f6d700b3bb0 msgr2=0x7f6d700b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.994+0000 7f6d75c00640 1 -- 172.21.15.45:0/442612557 shutdown_connections 2024-04-16T16:15:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.994+0000 7f6d75c00640 1 -- 172.21.15.45:0/442612557 wait complete. 2024-04-16T16:15:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.996+0000 7f6d75c00640 1 Processor -- start 2024-04-16T16:15:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.996+0000 7f6d75c00640 1 -- start start 2024-04-16T16:15:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.997+0000 7f6d75c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d7006e920 0x7f6d700c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.997+0000 7f6d75c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d70147ff0 0x7f6d700b2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.997+0000 7f6d75c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d7015f430 0x7f6d7015bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.997+0000 7f6d75c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6d70163210 con 0x7f6d70147ff0 2024-04-16T16:15:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.997+0000 7f6d75c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6d700b2e40 con 0x7f6d7015f430 2024-04-16T16:15:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.997+0000 7f6d75c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6d70144f40 con 0x7f6d7006e920 2024-04-16T16:15:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.997+0000 7f6d6f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d70147ff0 0x7f6d700b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.997+0000 7f6d6f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d70147ff0 0x7f6d700b2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43282/0 (socket says 172.21.15.45:43282) 2024-04-16T16:15:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.997+0000 7f6d6f400640 1 -- 172.21.15.45:0/1230486868 learned_addr learned my addr 172.21.15.45:0/1230486868 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.998+0000 7f6d6fe00640 1 --2- 172.21.15.45:0/1230486868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d7006e920 0x7f6d700c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.998+0000 7f6d75200640 1 --2- 172.21.15.45:0/1230486868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d7015f430 0x7f6d7015bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.998+0000 7f6d6f400640 1 -- 172.21.15.45:0/1230486868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d7006e920 msgr2=0x7f6d700c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.998+0000 7f6d6f400640 1 --2- 172.21.15.45:0/1230486868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d7006e920 0x7f6d700c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.998+0000 7f6d6f400640 1 -- 172.21.15.45:0/1230486868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d7015f430 msgr2=0x7f6d7015bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.998+0000 7f6d6f400640 1 --2- 172.21.15.45:0/1230486868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d7015f430 0x7f6d7015bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:48.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.998+0000 7f6d6f400640 1 -- 172.21.15.45:0/1230486868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d70144c60 con 0x7f6d70147ff0 2024-04-16T16:15:48.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.998+0000 7f6d75200640 1 --2- 172.21.15.45:0/1230486868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d7015f430 0x7f6d7015bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:48.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.998+0000 7f6d6f400640 1 --2- 172.21.15.45:0/1230486868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d70147ff0 0x7f6d700b2660 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00d800 tx=0x7f6d5c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:48.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.998+0000 7f6d67e00640 1 -- 172.21.15.45:0/1230486868 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c00b6c0 con 0x7f6d70147ff0 2024-04-16T16:15:48.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.999+0000 7f6d67e00640 1 -- 172.21.15.45:0/1230486868 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d5c019070 con 0x7f6d70147ff0 2024-04-16T16:15:48.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.999+0000 7f6d67e00640 1 -- 172.21.15.45:0/1230486868 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c00ba80 con 0x7f6d70147ff0 2024-04-16T16:15:48.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.999+0000 7f6d75c00640 1 -- 172.21.15.45:0/1230486868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d700c61f0 con 0x7f6d70147ff0 2024-04-16T16:15:48.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:47.999+0000 7f6d75c00640 1 -- 172.21.15.45:0/1230486868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d700c67b0 con 0x7f6d70147ff0 2024-04-16T16:15:48.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.000+0000 7f6d75c00640 1 -- 172.21.15.45:0/1230486868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d70075780 con 0x7f6d70147ff0 2024-04-16T16:15:48.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.001+0000 7f6d67e00640 1 -- 172.21.15.45:0/1230486868 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f6d5c00bc20 con 0x7f6d70147ff0 2024-04-16T16:15:48.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.002+0000 7f6d67e00640 1 --2- 172.21.15.45:0/1230486868 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6d5006ca60 0x7f6d5006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:48.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.002+0000 7f6d67e00640 1 -- 172.21.15.45:0/1230486868 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7f6d5c08c730 con 0x7f6d70147ff0 2024-04-16T16:15:48.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.003+0000 7f6d6fe00640 1 --2- 172.21.15.45:0/1230486868 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6d5006ca60 0x7f6d5006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:48.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.004+0000 7f6d6fe00640 1 --2- 172.21.15.45:0/1230486868 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6d5006ca60 0x7f6d5006ef20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6d60036000 tx=0x7f6d6003d040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:48.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.006+0000 7f6d67e00640 1 -- 172.21.15.45:0/1230486868 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6d5c05a540 con 0x7f6d70147ff0 2024-04-16T16:15:48.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.280+0000 7f6d75c00640 1 -- 172.21.15.45:0/1230486868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs2_data"]} v 0) -- 0x7f6d7015c7d0 con 0x7f6d70147ff0 2024-04-16T16:15:48.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.286+0000 7f6d67e00640 1 -- 172.21.15.45:0/1230486868 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=fs2_data"]}]=0 updated caps for client.3 v19) ==== 181+0+138 (secure 0 0 0) 0x7f6d5c059e40 con 0x7f6d70147ff0 2024-04-16T16:15:48.286 INFO:teuthology.orchestra.run.smithi045.stdout:[client.3] 2024-04-16T16:15:48.286 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5miPdMJBAAqjKpctNYu9ot1lff3rouxg== 2024-04-16T16:15:48.286 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:15:48.286 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:15:48.286 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw pool=fs2_data" 2024-04-16T16:15:48.286 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.3 2024-04-16T16:15:48.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.288+0000 7f6d75c00640 1 -- 172.21.15.45:0/1230486868 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6d5006ca60 msgr2=0x7f6d5006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:48.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.289+0000 7f6d75c00640 1 --2- 172.21.15.45:0/1230486868 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6d5006ca60 0x7f6d5006ef20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6d60036000 tx=0x7f6d6003d040 comp rx=0 tx=0).stop 2024-04-16T16:15:48.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.289+0000 7f6d75c00640 1 -- 172.21.15.45:0/1230486868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d70147ff0 msgr2=0x7f6d700b2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:48.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.289+0000 7f6d75c00640 1 --2- 172.21.15.45:0/1230486868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d70147ff0 0x7f6d700b2660 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00d800 tx=0x7f6d5c00dcc0 comp rx=0 tx=0).stop 2024-04-16T16:15:48.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.289+0000 7f6d75c00640 1 -- 172.21.15.45:0/1230486868 shutdown_connections 2024-04-16T16:15:48.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.289+0000 7f6d75c00640 1 --2- 172.21.15.45:0/1230486868 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6d5006ca60 0x7f6d5006ef20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:48.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.289+0000 7f6d75c00640 1 --2- 172.21.15.45:0/1230486868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d7015f430 0x7f6d7015bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:48.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.289+0000 7f6d75c00640 1 --2- 172.21.15.45:0/1230486868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d70147ff0 0x7f6d700b2660 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:48.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.289+0000 7f6d75c00640 1 --2- 172.21.15.45:0/1230486868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d7006e920 0x7f6d700c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:48.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.289+0000 7f6d75c00640 1 -- 172.21.15.45:0/1230486868 >> 172.21.15.45:0/1230486868 conn(0x7f6d700b3bb0 msgr2=0x7f6d70073150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:48.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.289+0000 7f6d75c00640 1 -- 172.21.15.45:0/1230486868 shutdown_connections 2024-04-16T16:15:48.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.289+0000 7f6d75c00640 1 -- 172.21.15.45:0/1230486868 wait complete. 2024-04-16T16:15:48.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:15:48.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.524+0000 7fdcbbe00640 1 Processor -- start 2024-04-16T16:15:48.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.524+0000 7fdcbbe00640 1 -- start start 2024-04-16T16:15:48.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.526+0000 7fdcbbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdcbc161e90 0x7fdcbc0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:48.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.526+0000 7fdcbbe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcbc0d3080 con 0x7fdcbc162270 2024-04-16T16:15:48.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.526+0000 7fdcbbe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcbc0d3360 con 0x7fdcbc0d52d0 2024-04-16T16:15:48.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.526+0000 7fdcbbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdcbc0d2e30 con 0x7fdcbc161e90 2024-04-16T16:15:48.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.526+0000 7fdcb3e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdcbc162270 0x7fdcbc0d3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53574/0 (socket says 172.21.15.45:53574) 2024-04-16T16:15:48.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.526+0000 7fdcb3e00640 1 -- 172.21.15.45:0/3828677708 learned_addr learned my addr 172.21.15.45:0/3828677708 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:48.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.527+0000 7fdcbaa00640 1 --2- 172.21.15.45:0/3828677708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdcbc161e90 0x7fdcbc0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:48.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.527+0000 7fdcba000640 1 -- 172.21.15.45:0/3828677708 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2676407275 0 0) 0x7fdcbc0d3080 con 0x7fdcbc162270 2024-04-16T16:15:48.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.527+0000 7fdcba000640 1 -- 172.21.15.45:0/3828677708 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdca0003520 con 0x7fdcbc162270 2024-04-16T16:15:48.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.527+0000 7fdcba000640 1 -- 172.21.15.45:0/3828677708 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3246760150 0 0) 0x7fdca0003520 con 0x7fdcbc162270 2024-04-16T16:15:48.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.527+0000 7fdcba000640 1 -- 172.21.15.45:0/3828677708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdcbc161e90 msgr2=0x7fdcbc0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:48.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.527+0000 7fdcba000640 1 --2- 172.21.15.45:0/3828677708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdcbc161e90 0x7fdcbc0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:48.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.527+0000 7fdcba000640 1 -- 172.21.15.45:0/3828677708 >> v1:172.21.15.103:6789/0 conn(0x7fdcbc0d52d0 legacy=0x7fdcbc06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:48.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.527+0000 7fdcba000640 1 -- 172.21.15.45:0/3828677708 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcbc0d3640 con 0x7fdcbc162270 2024-04-16T16:15:48.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.527+0000 7fdcba000640 1 -- 172.21.15.45:0/3828677708 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fdca4001670 con 0x7fdcbc162270 2024-04-16T16:15:48.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.527+0000 7fdcba000640 1 -- 172.21.15.45:0/3828677708 >> v1:172.21.15.45:6789/0 conn(0x7fdcbc162270 legacy=0x7fdcbc0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:48.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.528+0000 7fdcba000640 1 --2- 172.21.15.45:0/3828677708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdca00039e0 0x7fdca0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:48.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.528+0000 7fdcba000640 1 --2- 172.21.15.45:0/3828677708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdcbc161e90 0x7fdca0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:48.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.528+0000 7fdcba000640 1 --2- 172.21.15.45:0/3828677708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdca0004d40 0x7fdca00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:48.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.528+0000 7fdcba000640 1 -- 172.21.15.45:0/3828677708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc0d2e30 con 0x7fdca00039e0 2024-04-16T16:15:48.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.528+0000 7fdcba000640 1 -- 172.21.15.45:0/3828677708 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdca000bb00 con 0x7fdca0004d40 2024-04-16T16:15:48.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.528+0000 7fdcba000640 1 -- 172.21.15.45:0/3828677708 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdca000bd00 con 0x7fdcbc161e90 2024-04-16T16:15:48.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.528+0000 7fdcbaa00640 1 --2- 172.21.15.45:0/3828677708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdca00039e0 0x7fdca0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:48.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.528+0000 7fdcb3e00640 1 --2- 172.21.15.45:0/3828677708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdcbc161e90 0x7fdca0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:48.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.528+0000 7fdcbaa00640 1 -- 172.21.15.45:0/3828677708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdcbc161e90 msgr2=0x7fdca0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:48.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.528+0000 7fdcbaa00640 1 --2- 172.21.15.45:0/3828677708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdcbc161e90 0x7fdca0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:48.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.528+0000 7fdcbaa00640 1 -- 172.21.15.45:0/3828677708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdca0004d40 msgr2=0x7fdca00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:48.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.528+0000 7fdcbaa00640 1 --2- 172.21.15.45:0/3828677708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdca0004d40 0x7fdca00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:48.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.528+0000 7fdcbaa00640 1 -- 172.21.15.45:0/3828677708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca0003520 con 0x7fdca00039e0 2024-04-16T16:15:48.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.528+0000 7fdcb3e00640 1 --2- 172.21.15.45:0/3828677708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdcbc161e90 0x7fdca0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:48.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.529+0000 7fdcbaa00640 1 --2- 172.21.15.45:0/3828677708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdca00039e0 0x7fdca0003e00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009a90 tx=0x7fdca0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8364b1a88230f4e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:48.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.530+0000 7fdcba000640 1 -- 172.21.15.45:0/3828677708 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac039070 con 0x7fdca00039e0 2024-04-16T16:15:48.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.530+0000 7fdcba000640 1 -- 172.21.15.45:0/3828677708 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcac02c680 con 0x7fdca00039e0 2024-04-16T16:15:48.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.530+0000 7fdcba000640 1 -- 172.21.15.45:0/3828677708 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac009690 con 0x7fdca00039e0 2024-04-16T16:15:48.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.530+0000 7fdcbbe00640 1 -- 172.21.15.45:0/3828677708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdca00039e0 msgr2=0x7fdca0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:48.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.530+0000 7fdcbbe00640 1 --2- 172.21.15.45:0/3828677708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdca00039e0 0x7fdca0003e00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009a90 tx=0x7fdca0005830 comp rx=0 tx=0).stop 2024-04-16T16:15:48.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.530+0000 7fdcbb400640 1 -- 172.21.15.45:0/3828677708 reap_dead start 2024-04-16T16:15:48.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.530+0000 7fdcbbe00640 1 -- 172.21.15.45:0/3828677708 shutdown_connections 2024-04-16T16:15:48.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.530+0000 7fdcbbe00640 1 -- 172.21.15.45:0/3828677708 >> 172.21.15.45:0/3828677708 conn(0x7fdcbc0d1940 msgr2=0x7fdcbc0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:48.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.530+0000 7fdcbbe00640 1 -- 172.21.15.45:0/3828677708 shutdown_connections 2024-04-16T16:15:48.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.530+0000 7fdcbbe00640 1 -- 172.21.15.45:0/3828677708 wait complete. 2024-04-16T16:15:48.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.532+0000 7fdcbbe00640 1 Processor -- start 2024-04-16T16:15:48.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.532+0000 7fdcbbe00640 1 -- start start 2024-04-16T16:15:48.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.533+0000 7fdcbbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdca00039e0 0x7fdcbc1f5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:48.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.533+0000 7fdcbbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcbc0cb960 0x7fdcbc1f2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:48.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.533+0000 7fdcbbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcbc0ca160 0x7fdcbc0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:48.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.533+0000 7fdcbbe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc0d46d0 con 0x7fdcbc0ca160 2024-04-16T16:15:48.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.533+0000 7fdcbbe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc0d4550 con 0x7fdcbc0cb960 2024-04-16T16:15:48.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.533+0000 7fdcbbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdcbc06d550 con 0x7fdca00039e0 2024-04-16T16:15:48.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.534+0000 7fdcbb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcbc0ca160 0x7fdcbc0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:48.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.534+0000 7fdcbb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcbc0ca160 0x7fdcbc0ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43294/0 (socket says 172.21.15.45:43294) 2024-04-16T16:15:48.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.534+0000 7fdcbb400640 1 -- 172.21.15.45:0/192068293 learned_addr learned my addr 172.21.15.45:0/192068293 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:48.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.534+0000 7fdcb3e00640 1 --2- 172.21.15.45:0/192068293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcbc0cb960 0x7fdcbc1f2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:48.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.534+0000 7fdcbaa00640 1 --2- 172.21.15.45:0/192068293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdca00039e0 0x7fdcbc1f5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:48.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.534+0000 7fdcbb400640 1 -- 172.21.15.45:0/192068293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdca00039e0 msgr2=0x7fdcbc1f5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:48.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.534+0000 7fdcbb400640 1 --2- 172.21.15.45:0/192068293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdca00039e0 0x7fdcbc1f5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:48.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.534+0000 7fdcbb400640 1 -- 172.21.15.45:0/192068293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcbc0cb960 msgr2=0x7fdcbc1f2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:48.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.534+0000 7fdcbb400640 1 --2- 172.21.15.45:0/192068293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcbc0cb960 0x7fdcbc1f2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:48.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.534+0000 7fdcbb400640 1 -- 172.21.15.45:0/192068293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcbc1f5b70 con 0x7fdcbc0ca160 2024-04-16T16:15:48.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.534+0000 7fdcbaa00640 1 --2- 172.21.15.45:0/192068293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdca00039e0 0x7fdcbc1f5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:48.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.535+0000 7fdcbb400640 1 --2- 172.21.15.45:0/192068293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcbc0ca160 0x7fdcbc0ca5f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc0ba950 tx=0x7fdcb400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:48.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.535+0000 7fdcb2a00640 1 -- 172.21.15.45:0/192068293 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4017070 con 0x7fdcbc0ca160 2024-04-16T16:15:48.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.535+0000 7fdcb2a00640 1 -- 172.21.15.45:0/192068293 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcb4005d60 con 0x7fdcbc0ca160 2024-04-16T16:15:48.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.535+0000 7fdcb2a00640 1 -- 172.21.15.45:0/192068293 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb400a850 con 0x7fdcbc0ca160 2024-04-16T16:15:48.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.535+0000 7fdcbbe00640 1 -- 172.21.15.45:0/192068293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcbc0cbde0 con 0x7fdcbc0ca160 2024-04-16T16:15:48.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.535+0000 7fdcbbe00640 1 -- 172.21.15.45:0/192068293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcbc0cc3a0 con 0x7fdcbc0ca160 2024-04-16T16:15:48.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.536+0000 7fdcbbe00640 1 -- 172.21.15.45:0/192068293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcbc075780 con 0x7fdcbc0ca160 2024-04-16T16:15:48.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.537+0000 7fdcb2a00640 1 -- 172.21.15.45:0/192068293 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fdcb400a9f0 con 0x7fdcbc0ca160 2024-04-16T16:15:48.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.538+0000 7fdcb2a00640 1 --2- 172.21.15.45:0/192068293 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdc9806ca60 0x7fdc9806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:48.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.538+0000 7fdcb2a00640 1 -- 172.21.15.45:0/192068293 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7fdcb408b2e0 con 0x7fdcbc0ca160 2024-04-16T16:15:48.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.539+0000 7fdcbaa00640 1 --2- 172.21.15.45:0/192068293 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdc9806ca60 0x7fdc9806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:48.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.540+0000 7fdcbaa00640 1 --2- 172.21.15.45:0/192068293 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdc9806ca60 0x7fdc9806ef20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdcac036000 tx=0x7fdcac030000 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:48.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.543+0000 7fdcb2a00640 1 -- 172.21.15.45:0/192068293 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdcbc1f5b70 con 0x7fdcbc0ca160 2024-04-16T16:15:48.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.793+0000 7fdcbbe00640 1 -- 172.21.15.45:0/192068293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdcbc06dae0 con 0x7fdcbc0ca160 2024-04-16T16:15:48.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.794+0000 7fdcb2a00640 1 -- 172.21.15.45:0/192068293 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v40) ==== 53+0+128 (secure 0 0 0) 0x7fdcb4059080 con 0x7fdcbc0ca160 2024-04-16T16:15:48.794 INFO:teuthology.orchestra.run.smithi045.stdout:name: fs1, metadata pool: fs1_metadata, data pools: [fs1_data ] 2024-04-16T16:15:48.794 INFO:teuthology.orchestra.run.smithi045.stdout:name: fs2, metadata pool: fs2_metadata, data pools: [fs2_data ] 2024-04-16T16:15:48.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.797+0000 7fdcbbe00640 1 -- 172.21.15.45:0/192068293 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdc9806ca60 msgr2=0x7fdc9806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:48.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.797+0000 7fdcbbe00640 1 --2- 172.21.15.45:0/192068293 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdc9806ca60 0x7fdc9806ef20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdcac036000 tx=0x7fdcac030000 comp rx=0 tx=0).stop 2024-04-16T16:15:48.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.797+0000 7fdcbbe00640 1 -- 172.21.15.45:0/192068293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcbc0ca160 msgr2=0x7fdcbc0ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:48.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.797+0000 7fdcbbe00640 1 --2- 172.21.15.45:0/192068293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcbc0ca160 0x7fdcbc0ca5f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc0ba950 tx=0x7fdcb400cb90 comp rx=0 tx=0).stop 2024-04-16T16:15:48.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.797+0000 7fdcbbe00640 1 -- 172.21.15.45:0/192068293 shutdown_connections 2024-04-16T16:15:48.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.797+0000 7fdcbbe00640 1 --2- 172.21.15.45:0/192068293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcbc0ca160 0x7fdcbc0ca5f0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:48.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.797+0000 7fdcbbe00640 1 --2- 172.21.15.45:0/192068293 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fdc9806ca60 0x7fdc9806ef20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:48.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.797+0000 7fdcbbe00640 1 --2- 172.21.15.45:0/192068293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcbc0cb960 0x7fdcbc1f2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:48.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.797+0000 7fdcbbe00640 1 --2- 172.21.15.45:0/192068293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdca00039e0 0x7fdcbc1f5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:48.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.797+0000 7fdcbbe00640 1 -- 172.21.15.45:0/192068293 >> 172.21.15.45:0/192068293 conn(0x7fdcbc0d1940 msgr2=0x7fdcbc0b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:48.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.797+0000 7fdcbbe00640 1 -- 172.21.15.45:0/192068293 shutdown_connections 2024-04-16T16:15:48.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:48.797+0000 7fdcbbe00640 1 -- 172.21.15.45:0/192068293 wait complete. 2024-04-16T16:15:48.815 INFO:tasks.cephfs.mount:Mounting Ceph FS fs2; just confirmed its presence on cluster 2024-04-16T16:15:48.815 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:15:48.815 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-04-16T16:15:48.815 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-16T16:15:48.815 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T16:15:48.815 INFO:tasks.cephfs.mount:self.cephfs_name = fs2 2024-04-16T16:15:48.815 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:15:48.816 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:15:48.816 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:15:48.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:48.816 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-04-16T16:15:48.876 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:15:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:15:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:15:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:15:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-16T16:15:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:15:48.877 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:15:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:15:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-04-16T16:15:48.877 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:15:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:15:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-04-16T16:15:48.877 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:15:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-04-16T16:15:48.877 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:15:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-04-16T16:15:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-16T16:15:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42476sec preferred_lft 42476sec 2024-04-16T16:15:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-04-16T16:15:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:15:48.878 INFO:teuthology.orchestra.run.smithi103.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:15:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:15:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:15:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:15:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::74ce:ceff:fec6:2a00/64 scope link 2024-04-16T16:15:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:15:48.878 INFO:teuthology.orchestra.run.smithi103.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:15:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T16:15:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::3cf7:ffff:feca:f879/64 scope link 2024-04-16T16:15:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:15:48.879 INFO:teuthology.orchestra.run.smithi103.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:15:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 86:4c:6e:ce:c3:07 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-16T16:15:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::844c:6eff:fece:c307/64 scope link 2024-04-16T16:15:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:15:48.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:48.880 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:15:48.972 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-16T16:15:48.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:48.977 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-04-16T16:15:49.029 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-04-16T16:15:49.029 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:15:49.029 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-16T16:15:49.029 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-16T16:15:49.029 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:15:49.030 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:15:49.030 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:15:49.030 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-16T16:15:49.030 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-16T16:15:49.031 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-16T16:15:49.031 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:15:49.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:15:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:15:49.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:15:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:15:49.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:49.273 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-16T16:15:49.322 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-16T16:15:49.322 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-16T16:15:49.323 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:15:49.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:49.323 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.0 up 2024-04-16T16:15:49.413 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:15:49.413 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-16T16:15:49.466 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:15:49.467 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:15:49.467 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:15:49.516 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.fs2=/ 2024-04-16T16:15:49.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:49.517 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.fs2=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:15:49.648 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:15:49.648 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:15:49.688 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:15:49.688 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:15:49.688 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:15:49.688 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with new device syntax: 1@748c4b85-0190-4c59-80ad-098a6a5007c0.fs2=/ 2024-04-16T16:15:49.688 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:15:49.693 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-16T16:15:49.693 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:15:49.693 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:15:49.693 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:15:49.693 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:15:49.698 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:15:49.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:49.699 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:15:49.779 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:15:49.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:15:49 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:15:49.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:15:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:15:49.909 INFO:teuthology.orchestra.run.smithi103.stdout:client5928 2024-04-16T16:15:49.910 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:15:49.910 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:15:49.958 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:15:49.958 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:15:50.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:15:50.054 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec dd if=/dev/urandom of=test.bin bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:15:50.165 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-16T16:15:50.165 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-16T16:15:50.165 INFO:teuthology.orchestra.run.smithi103.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432768 s, 24.2 MB/s 2024-04-16T16:15:50.167 DEBUG:teuthology.orchestra.run.smithi103:> stdin-killer -- python3 -c ' 2024-04-16T16:15:50.167 DEBUG:teuthology.orchestra.run.smithi103:> import os 2024-04-16T16:15:50.167 DEBUG:teuthology.orchestra.run.smithi103:> import stat 2024-04-16T16:15:50.167 DEBUG:teuthology.orchestra.run.smithi103:> 2024-04-16T16:15:50.167 DEBUG:teuthology.orchestra.run.smithi103:> print(os.stat("/home/ubuntu/cephtest/mnt.1/test.bin").st_ino) 2024-04-16T16:15:50.167 DEBUG:teuthology.orchestra.run.smithi103:> ' 2024-04-16T16:15:50.316 INFO:teuthology.orchestra.run.smithi103.stdout:1099511627776 2024-04-16T16:15:50.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:15:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:15:50.329 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:15:50.330 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:15:50.425 INFO:tasks.cephfs.mount:Creating file a 2024-04-16T16:15:50.425 DEBUG:teuthology.orchestra.run.smithi103:> touch /home/ubuntu/cephtest/mnt.1/a 2024-04-16T16:15:50.471 INFO:tasks.cephfs.mount:Creating file b 2024-04-16T16:15:50.471 DEBUG:teuthology.orchestra.run.smithi103:> touch /home/ubuntu/cephtest/mnt.1/b 2024-04-16T16:15:50.570 INFO:tasks.cephfs.mount:Creating file c 2024-04-16T16:15:50.570 DEBUG:teuthology.orchestra.run.smithi103:> touch /home/ubuntu/cephtest/mnt.1/c 2024-04-16T16:15:50.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:15:50.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.857+0000 7f3c0b800640 1 Processor -- start 2024-04-16T16:15:50.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.858+0000 7f3c0b800640 1 -- start start 2024-04-16T16:15:50.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.859+0000 7f3c0b800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c04072890 0x7f3c040cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:50.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.859+0000 7f3c0b800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c04146600 con 0x7f3c040cd220 2024-04-16T16:15:50.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.859+0000 7f3c0b800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c040cde00 con 0x7f3c04146220 2024-04-16T16:15:50.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.859+0000 7f3c0b800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c0406f840 con 0x7f3c04072890 2024-04-16T16:15:50.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.860+0000 7f3c09800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3c040cd220 0x7f3c0406df00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53576/0 (socket says 172.21.15.45:53576) 2024-04-16T16:15:50.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.860+0000 7f3c09800640 1 -- 172.21.15.45:0/2549326675 learned_addr learned my addr 172.21.15.45:0/2549326675 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:50.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.860+0000 7f3c03400640 1 -- 172.21.15.45:0/2549326675 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2085878153 0 0) 0x7f3c04146600 con 0x7f3c040cd220 2024-04-16T16:15:50.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.860+0000 7f3c03400640 1 -- 172.21.15.45:0/2549326675 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bec003520 con 0x7f3c040cd220 2024-04-16T16:15:50.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.860+0000 7f3c03e00640 1 --2- 172.21.15.45:0/2549326675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c04072890 0x7f3c040cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:50.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c03400640 1 -- 172.21.15.45:0/2549326675 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 48630596 0 0) 0x7f3bec003520 con 0x7f3c040cd220 2024-04-16T16:15:50.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c03400640 1 -- 172.21.15.45:0/2549326675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c04072890 msgr2=0x7f3c040cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:50.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c03400640 1 --2- 172.21.15.45:0/2549326675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c04072890 0x7f3c040cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:50.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c03400640 1 -- 172.21.15.45:0/2549326675 >> v1:172.21.15.103:6789/0 conn(0x7f3c04146220 legacy=0x7f3c04072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:50.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c03400640 1 -- 172.21.15.45:0/2549326675 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c040ce630 con 0x7f3c040cd220 2024-04-16T16:15:50.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c03400640 1 -- 172.21.15.45:0/2549326675 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f3bfc0019f0 con 0x7f3c040cd220 2024-04-16T16:15:50.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c03400640 1 -- 172.21.15.45:0/2549326675 >> v1:172.21.15.45:6789/0 conn(0x7f3c040cd220 legacy=0x7f3c0406df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:50.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c03400640 1 --2- 172.21.15.45:0/2549326675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c04072890 0x7f3bec0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:50.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c03400640 1 --2- 172.21.15.45:0/2549326675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bec004060 0x7f3bec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:50.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c03400640 1 --2- 172.21.15.45:0/2549326675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bec004d40 0x7f3bec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:50.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c03400640 1 -- 172.21.15.45:0/2549326675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3bec00bbd0 con 0x7f3bec004d40 2024-04-16T16:15:50.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c03400640 1 -- 172.21.15.45:0/2549326675 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3bec00bd80 con 0x7f3bec004060 2024-04-16T16:15:50.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c03400640 1 -- 172.21.15.45:0/2549326675 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3bec00bf80 con 0x7f3c04072890 2024-04-16T16:15:50.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c09800640 1 --2- 172.21.15.45:0/2549326675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bec004d40 0x7f3bec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:50.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c03e00640 1 --2- 172.21.15.45:0/2549326675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bec004060 0x7f3bec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:50.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.861+0000 7f3c08e00640 1 --2- 172.21.15.45:0/2549326675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c04072890 0x7f3bec0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:50.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.862+0000 7f3c09800640 1 -- 172.21.15.45:0/2549326675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c04072890 msgr2=0x7f3bec0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:50.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.862+0000 7f3c09800640 1 --2- 172.21.15.45:0/2549326675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c04072890 0x7f3bec0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:50.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.862+0000 7f3c09800640 1 -- 172.21.15.45:0/2549326675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bec004060 msgr2=0x7f3bec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:50.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.862+0000 7f3c09800640 1 --2- 172.21.15.45:0/2549326675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bec004060 0x7f3bec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:50.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.862+0000 7f3c09800640 1 -- 172.21.15.45:0/2549326675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bec003520 con 0x7f3bec004d40 2024-04-16T16:15:50.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.862+0000 7f3c08e00640 1 --2- 172.21.15.45:0/2549326675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c04072890 0x7f3bec0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:50.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.863+0000 7f3c09800640 1 --2- 172.21.15.45:0/2549326675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bec004d40 0x7f3bec0051c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc00a7b0 tx=0x7f3bec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4870cbad28d5b7e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:50.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.863+0000 7f3c03400640 1 -- 172.21.15.45:0/2549326675 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc00a3d0 con 0x7f3bec004d40 2024-04-16T16:15:50.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.863+0000 7f3c03400640 1 -- 172.21.15.45:0/2549326675 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bfc034630 con 0x7f3bec004d40 2024-04-16T16:15:50.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.863+0000 7f3c03400640 1 -- 172.21.15.45:0/2549326675 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc034aa0 con 0x7f3bec004d40 2024-04-16T16:15:50.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.863+0000 7f3c0b800640 1 -- 172.21.15.45:0/2549326675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bec004d40 msgr2=0x7f3bec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:50.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.863+0000 7f3c0b800640 1 --2- 172.21.15.45:0/2549326675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bec004d40 0x7f3bec0051c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc00a7b0 tx=0x7f3bec005830 comp rx=0 tx=0).stop 2024-04-16T16:15:50.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.864+0000 7f3c09800640 1 -- 172.21.15.45:0/2549326675 reap_dead start 2024-04-16T16:15:50.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.864+0000 7f3c0b800640 1 -- 172.21.15.45:0/2549326675 shutdown_connections 2024-04-16T16:15:50.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.864+0000 7f3c0b800640 1 -- 172.21.15.45:0/2549326675 >> 172.21.15.45:0/2549326675 conn(0x7f3c0406b930 msgr2=0x7f3c0406bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:50.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.864+0000 7f3c0b800640 1 -- 172.21.15.45:0/2549326675 shutdown_connections 2024-04-16T16:15:50.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.864+0000 7f3c0b800640 1 -- 172.21.15.45:0/2549326675 wait complete. 2024-04-16T16:15:50.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.866+0000 7f3c0b800640 1 Processor -- start 2024-04-16T16:15:50.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.866+0000 7f3c0b800640 1 -- start start 2024-04-16T16:15:50.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.867+0000 7f3c0b800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bec004d40 0x7f3c04152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:50.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.867+0000 7f3c0b800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c04137a60 0x7f3c040ae1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:50.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.867+0000 7f3c0b800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c04137ee0 0x7f3c040ae710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:50.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.867+0000 7f3c0b800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3c040cebf0 con 0x7f3c04137ee0 2024-04-16T16:15:50.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.867+0000 7f3c0b800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c040ce0e0 con 0x7f3c04137a60 2024-04-16T16:15:50.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.867+0000 7f3c0b800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c040cf110 con 0x7f3bec004d40 2024-04-16T16:15:50.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.867+0000 7f3c09800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c04137ee0 0x7f3c040ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:50.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.867+0000 7f3c09800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c04137ee0 0x7f3c040ae710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43316/0 (socket says 172.21.15.45:43316) 2024-04-16T16:15:50.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.867+0000 7f3c09800640 1 -- 172.21.15.45:0/713199108 learned_addr learned my addr 172.21.15.45:0/713199108 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:50.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.867+0000 7f3c03e00640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c04137a60 0x7f3c040ae1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:50.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.867+0000 7f3c09800640 1 -- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bec004d40 msgr2=0x7f3c04152f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:50.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.867+0000 7f3c08e00640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bec004d40 0x7f3c04152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:50.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.867+0000 7f3c09800640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bec004d40 0x7f3c04152f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:50.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.868+0000 7f3c09800640 1 -- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c04137a60 msgr2=0x7f3c040ae1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:50.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.868+0000 7f3c09800640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c04137a60 0x7f3c040ae1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:50.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.868+0000 7f3c09800640 1 -- 172.21.15.45:0/713199108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c041369c0 con 0x7f3c04137ee0 2024-04-16T16:15:50.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.868+0000 7f3c03e00640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c04137a60 0x7f3c040ae1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:50.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.868+0000 7f3c08e00640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bec004d40 0x7f3c04152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:50.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.868+0000 7f3c09800640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c04137ee0 0x7f3c040ae710 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc00a7b0 tx=0x7f3bfc001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:50.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.868+0000 7f3c00c00640 1 -- 172.21.15.45:0/713199108 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc039040 con 0x7f3c04137ee0 2024-04-16T16:15:50.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.868+0000 7f3c00c00640 1 -- 172.21.15.45:0/713199108 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bfc043070 con 0x7f3c04137ee0 2024-04-16T16:15:50.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.868+0000 7f3c0b800640 1 -- 172.21.15.45:0/713199108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c04136ca0 con 0x7f3c04137ee0 2024-04-16T16:15:50.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.868+0000 7f3c00c00640 1 -- 172.21.15.45:0/713199108 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc00a3d0 con 0x7f3c04137ee0 2024-04-16T16:15:50.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.868+0000 7f3c0b800640 1 -- 172.21.15.45:0/713199108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c04137260 con 0x7f3c04137ee0 2024-04-16T16:15:50.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.870+0000 7f3c0b800640 1 -- 172.21.15.45:0/713199108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c04075780 con 0x7f3c04137ee0 2024-04-16T16:15:50.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.871+0000 7f3c00c00640 1 -- 172.21.15.45:0/713199108 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f3bfc034a00 con 0x7f3c04137ee0 2024-04-16T16:15:50.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.872+0000 7f3c00c00640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3be406ca60 0x7f3be406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:50.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.872+0000 7f3c00c00640 1 -- 172.21.15.45:0/713199108 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7f3bfc0ae360 con 0x7f3c04137ee0 2024-04-16T16:15:50.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.872+0000 7f3c08e00640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3be406ca60 0x7f3be406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:50.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.874+0000 7f3c08e00640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3be406ca60 0x7f3be406ef20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80016f0 tx=0x7f3bf8009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:50.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:50.877+0000 7f3c00c00640 1 -- 172.21.15.45:0/713199108 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3bfc07bf30 con 0x7f3c04137ee0 2024-04-16T16:15:51.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:51.105+0000 7f3c0b800640 1 -- 172.21.15.45:0/713199108 --> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f3c041538e0 con 0x7f3be406ca60 2024-04-16T16:15:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:51.115+0000 7f3c00c00640 1 -- 172.21.15.45:0/713199108 <== mgr.4658 v2:172.21.15.103:6832/280591707 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7f3c041538e0 con 0x7f3be406ca60 2024-04-16T16:15:51.115 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:15:51.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:51.117+0000 7f3c0b800640 1 -- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3be406ca60 msgr2=0x7f3be406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:51.117+0000 7f3c0b800640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3be406ca60 0x7f3be406ef20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80016f0 tx=0x7f3bf8009040 comp rx=0 tx=0).stop 2024-04-16T16:15:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:51.117+0000 7f3c0b800640 1 -- 172.21.15.45:0/713199108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c04137ee0 msgr2=0x7f3c040ae710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:51.117+0000 7f3c0b800640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c04137ee0 0x7f3c040ae710 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc00a7b0 tx=0x7f3bfc001cf0 comp rx=0 tx=0).stop 2024-04-16T16:15:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:51.117+0000 7f3c0b800640 1 -- 172.21.15.45:0/713199108 shutdown_connections 2024-04-16T16:15:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:51.117+0000 7f3c0b800640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c04137ee0 0x7f3c040ae710 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:51.117+0000 7f3c0b800640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f3be406ca60 0x7f3be406ef20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:51.117+0000 7f3c0b800640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c04137a60 0x7f3c040ae1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:51.117+0000 7f3c0b800640 1 --2- 172.21.15.45:0/713199108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bec004d40 0x7f3c04152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:51.117+0000 7f3c0b800640 1 -- 172.21.15.45:0/713199108 >> 172.21.15.45:0/713199108 conn(0x7f3c0406b930 msgr2=0x7f3c040cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:51.118+0000 7f3c0b800640 1 -- 172.21.15.45:0/713199108 shutdown_connections 2024-04-16T16:15:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:51.118+0000 7f3c0b800640 1 -- 172.21.15.45:0/713199108 wait complete. 2024-04-16T16:15:52.141 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:15:52.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.373+0000 7f832ae00640 1 Processor -- start 2024-04-16T16:15:52.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.373+0000 7f832ae00640 1 -- start start 2024-04-16T16:15:52.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.375+0000 7f832ae00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f83240b0e30 0x7f832406b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:52.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.375+0000 7f832ae00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f832406be70 0x7f83240b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:52.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.375+0000 7f832ae00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83240b6b60 con 0x7f83240b6740 2024-04-16T16:15:52.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.375+0000 7f832ae00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f832406c380 con 0x7f83240b0e30 2024-04-16T16:15:52.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.375+0000 7f832ae00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f83240cf6d0 con 0x7f832406be70 2024-04-16T16:15:52.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.375+0000 7f8329a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f83240b0e30 0x7f832406b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:52.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.376+0000 7f8329a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f83240b0e30 0x7f832406b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43326/0 (socket says 172.21.15.45:43326) 2024-04-16T16:15:52.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.376+0000 7f8329a00640 1 -- 172.21.15.45:0/4027870327 learned_addr learned my addr 172.21.15.45:0/4027870327 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:52.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.376+0000 7f8329000640 1 --2- 172.21.15.45:0/4027870327 >> v2:172.21.15.103:3300/0 conn(0x7f832406be70 0x7f83240b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:52.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.376+0000 7f8329a00640 1 -- 172.21.15.45:0/4027870327 >> v2:172.21.15.103:3300/0 conn(0x7f832406be70 msgr2=0x7f83240b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:52.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.376+0000 7f8329a00640 1 --2- 172.21.15.45:0/4027870327 >> v2:172.21.15.103:3300/0 conn(0x7f832406be70 0x7f83240b6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:52.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.376+0000 7f8329a00640 1 -- 172.21.15.45:0/4027870327 >> v1:172.21.15.103:6789/0 conn(0x7f83240b6740 legacy=0x7f8324073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:52.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.376+0000 7f8329a00640 1 -- 172.21.15.45:0/4027870327 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83241636b0 con 0x7f83240b0e30 2024-04-16T16:15:52.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.376+0000 7f8329000640 1 --2- 172.21.15.45:0/4027870327 >> v2:172.21.15.103:3300/0 conn(0x7f832406be70 0x7f83240b6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:52.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.378+0000 7f8329a00640 1 --2- 172.21.15.45:0/4027870327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83240b0e30 0x7f832406b930 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f832406c500 tx=0x7f831802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aad7686d9ccab39 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:52.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.378+0000 7f8323e00640 1 -- 172.21.15.45:0/4027870327 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831802c950 con 0x7f83240b0e30 2024-04-16T16:15:52.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.378+0000 7f8323e00640 1 -- 172.21.15.45:0/4027870327 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f831802caf0 con 0x7f83240b0e30 2024-04-16T16:15:52.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.378+0000 7f8323e00640 1 -- 172.21.15.45:0/4027870327 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83180343f0 con 0x7f83240b0e30 2024-04-16T16:15:52.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.379+0000 7f832ae00640 1 -- 172.21.15.45:0/4027870327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83240b0e30 msgr2=0x7f832406b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:52.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.379+0000 7f832ae00640 1 --2- 172.21.15.45:0/4027870327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83240b0e30 0x7f832406b930 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f832406c500 tx=0x7f831802bdb0 comp rx=0 tx=0).stop 2024-04-16T16:15:52.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.379+0000 7f832ae00640 1 -- 172.21.15.45:0/4027870327 shutdown_connections 2024-04-16T16:15:52.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.379+0000 7f832ae00640 1 --2- 172.21.15.45:0/4027870327 >> v2:172.21.15.103:3300/0 conn(0x7f832406be70 0x7f83240b6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:52.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.379+0000 7f832ae00640 1 --2- 172.21.15.45:0/4027870327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83240b0e30 0x7f832406b930 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:52.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.379+0000 7f832ae00640 1 -- 172.21.15.45:0/4027870327 >> 172.21.15.45:0/4027870327 conn(0x7f832406f0e0 msgr2=0x7f832406f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:52.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.379+0000 7f832ae00640 1 -- 172.21.15.45:0/4027870327 shutdown_connections 2024-04-16T16:15:52.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.379+0000 7f832ae00640 1 -- 172.21.15.45:0/4027870327 wait complete. 2024-04-16T16:15:52.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.381+0000 7f832ae00640 1 Processor -- start 2024-04-16T16:15:52.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.381+0000 7f832ae00640 1 -- start start 2024-04-16T16:15:52.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.382+0000 7f832ae00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f832406be70 0x7f832415c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:52.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.382+0000 7f832ae00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83240b0e30 0x7f8324159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:52.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.382+0000 7f832ae00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83240b6740 0x7f83241fc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:52.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.382+0000 7f832ae00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8324163f50 con 0x7f83240b0e30 2024-04-16T16:15:52.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.382+0000 7f832ae00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8324163c70 con 0x7f832406be70 2024-04-16T16:15:52.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.382+0000 7f832ae00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8324164470 con 0x7f83240b6740 2024-04-16T16:15:52.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.382+0000 7f8329000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83240b0e30 0x7f8324159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:52.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.383+0000 7f8329000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83240b0e30 0x7f8324159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43336/0 (socket says 172.21.15.45:43336) 2024-04-16T16:15:52.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.383+0000 7f8329000640 1 -- 172.21.15.45:0/317411550 learned_addr learned my addr 172.21.15.45:0/317411550 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:52.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.383+0000 7f832a400640 1 --2- 172.21.15.45:0/317411550 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83240b6740 0x7f83241fc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:52.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.383+0000 7f8329a00640 1 --2- 172.21.15.45:0/317411550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f832406be70 0x7f832415c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:52.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.383+0000 7f8329000640 1 -- 172.21.15.45:0/317411550 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83240b6740 msgr2=0x7f83241fc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:52.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.383+0000 7f8329000640 1 --2- 172.21.15.45:0/317411550 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83240b6740 0x7f83241fc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:52.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.383+0000 7f8329000640 1 -- 172.21.15.45:0/317411550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f832406be70 msgr2=0x7f832415c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:52.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.383+0000 7f8329000640 1 --2- 172.21.15.45:0/317411550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f832406be70 0x7f832415c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:52.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.383+0000 7f8329000640 1 -- 172.21.15.45:0/317411550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83241f9590 con 0x7f83240b0e30 2024-04-16T16:15:52.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.383+0000 7f8329a00640 1 --2- 172.21.15.45:0/317411550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f832406be70 0x7f832415c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:52.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.383+0000 7f8329000640 1 --2- 172.21.15.45:0/317411550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83240b0e30 0x7f8324159000 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f831000c7f0 tx=0x7f831000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:52.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.384+0000 7f8321600640 1 -- 172.21.15.45:0/317411550 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310015070 con 0x7f83240b0e30 2024-04-16T16:15:52.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.384+0000 7f8321600640 1 -- 172.21.15.45:0/317411550 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8310007820 con 0x7f83240b0e30 2024-04-16T16:15:52.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.384+0000 7f8321600640 1 -- 172.21.15.45:0/317411550 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310007bd0 con 0x7f83240b0e30 2024-04-16T16:15:52.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.384+0000 7f832ae00640 1 -- 172.21.15.45:0/317411550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83241fd3c0 con 0x7f83240b0e30 2024-04-16T16:15:52.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.384+0000 7f832ae00640 1 -- 172.21.15.45:0/317411550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f832415bd70 con 0x7f83240b0e30 2024-04-16T16:15:52.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.385+0000 7f832ae00640 1 -- 172.21.15.45:0/317411550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8324075780 con 0x7f83240b0e30 2024-04-16T16:15:52.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.386+0000 7f8321600640 1 -- 172.21.15.45:0/317411550 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f8310018070 con 0x7f83240b0e30 2024-04-16T16:15:52.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.387+0000 7f8321600640 1 --2- 172.21.15.45:0/317411550 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f830406ca60 0x7f830406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:52.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.387+0000 7f8321600640 1 -- 172.21.15.45:0/317411550 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7f831008c5e0 con 0x7f83240b0e30 2024-04-16T16:15:52.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.388+0000 7f8329a00640 1 --2- 172.21.15.45:0/317411550 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f830406ca60 0x7f830406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:52.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.389+0000 7f8329a00640 1 --2- 172.21.15.45:0/317411550 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f830406ca60 0x7f830406ef20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8318035000 tx=0x7f8318030000 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:52.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.392+0000 7f8321600640 1 -- 172.21.15.45:0/317411550 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f831005a1b0 con 0x7f83240b0e30 2024-04-16T16:15:52.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.625+0000 7f832ae00640 1 -- 172.21.15.45:0/317411550 --> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f8324160d50 con 0x7f830406ca60 2024-04-16T16:15:52.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.629+0000 7f8321600640 1 -- 172.21.15.45:0/317411550 <== mgr.4658 v2:172.21.15.103:6832/280591707 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7f8324160d50 con 0x7f830406ca60 2024-04-16T16:15:52.628 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:15:52.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.631+0000 7f832ae00640 1 -- 172.21.15.45:0/317411550 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f830406ca60 msgr2=0x7f830406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:52.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.631+0000 7f832ae00640 1 --2- 172.21.15.45:0/317411550 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f830406ca60 0x7f830406ef20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8318035000 tx=0x7f8318030000 comp rx=0 tx=0).stop 2024-04-16T16:15:52.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.631+0000 7f832ae00640 1 -- 172.21.15.45:0/317411550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83240b0e30 msgr2=0x7f8324159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:52.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.631+0000 7f832ae00640 1 --2- 172.21.15.45:0/317411550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83240b0e30 0x7f8324159000 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f831000c7f0 tx=0x7f831000ccb0 comp rx=0 tx=0).stop 2024-04-16T16:15:52.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.631+0000 7f832ae00640 1 -- 172.21.15.45:0/317411550 shutdown_connections 2024-04-16T16:15:52.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.631+0000 7f832ae00640 1 --2- 172.21.15.45:0/317411550 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83240b6740 0x7f83241fc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:52.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.631+0000 7f832ae00640 1 --2- 172.21.15.45:0/317411550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83240b0e30 0x7f8324159000 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:52.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.631+0000 7f832ae00640 1 --2- 172.21.15.45:0/317411550 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f830406ca60 0x7f830406ef20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:52.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.631+0000 7f832ae00640 1 --2- 172.21.15.45:0/317411550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f832406be70 0x7f832415c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:52.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.631+0000 7f832ae00640 1 -- 172.21.15.45:0/317411550 >> 172.21.15.45:0/317411550 conn(0x7f832406f0e0 msgr2=0x7f8324161250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:52.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.632+0000 7f832ae00640 1 -- 172.21.15.45:0/317411550 shutdown_connections 2024-04-16T16:15:52.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:52.632+0000 7f832ae00640 1 -- 172.21.15.45:0/317411550 wait complete. 2024-04-16T16:15:53.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:15:53.667 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:15:53.707 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:15:53.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.880+0000 7fbbcce00640 1 Processor -- start 2024-04-16T16:15:53.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.881+0000 7fbbcce00640 1 -- start start 2024-04-16T16:15:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.882+0000 7fbbcce00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbbc809aa70 0x7fbbc809ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.882+0000 7fbbcce00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbbc806e810 0x7fbbc80d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.882+0000 7fbbcce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbc8070c20 0x7fbbc8071090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.882+0000 7fbbcce00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fbbc806d580 con 0x7fbbc809aa70 2024-04-16T16:15:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.882+0000 7fbbcce00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbbc80d1aa0 con 0x7fbbc806e810 2024-04-16T16:15:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.882+0000 7fbbcce00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbbc809b390 con 0x7fbbc8070c20 2024-04-16T16:15:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.883+0000 7fbbc7400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbbc809aa70 0x7fbbc809ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.883+0000 7fbbc7400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbbc809aa70 0x7fbbc809ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59002/0 (socket says 172.21.15.45:59002) 2024-04-16T16:15:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.883+0000 7fbbc7400640 1 -- 172.21.15.45:0/1908764363 learned_addr learned my addr 172.21.15.45:0/1908764363 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:53.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.883+0000 7fbbc6a00640 1 --2- 172.21.15.45:0/1908764363 >> v2:172.21.15.103:3300/0 conn(0x7fbbc806e810 0x7fbbc80d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:53.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.883+0000 7fbbc7e00640 1 --2- 172.21.15.45:0/1908764363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbc8070c20 0x7fbbc8071090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:53.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.883+0000 7fbbc7400640 1 -- 172.21.15.45:0/1908764363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbc8070c20 msgr2=0x7fbbc8071090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:53.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.883+0000 7fbbc7400640 1 --2- 172.21.15.45:0/1908764363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbc8070c20 0x7fbbc8071090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:53.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.883+0000 7fbbc7400640 1 -- 172.21.15.45:0/1908764363 >> v2:172.21.15.103:3300/0 conn(0x7fbbc806e810 msgr2=0x7fbbc80d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:53.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.883+0000 7fbbc7400640 1 --2- 172.21.15.45:0/1908764363 >> v2:172.21.15.103:3300/0 conn(0x7fbbc806e810 0x7fbbc80d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:53.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.883+0000 7fbbc7400640 1 -- 172.21.15.45:0/1908764363 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc806ddf0 con 0x7fbbc809aa70 2024-04-16T16:15:53.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.883+0000 7fbbc6a00640 1 --2- 172.21.15.45:0/1908764363 >> v2:172.21.15.103:3300/0 conn(0x7fbbc806e810 0x7fbbc80d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:53.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.884+0000 7fbbc7400640 1 --2- 172.21.15.45:0/1908764363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc809aa70 0x7fbbc809ae50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8069300 tx=0x7fbbb802b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61fac69f262273b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:53.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.885+0000 7fbbc6000640 1 -- 172.21.15.45:0/1908764363 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8035020 con 0x7fbbc809aa70 2024-04-16T16:15:53.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.885+0000 7fbbc6000640 1 -- 172.21.15.45:0/1908764363 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb8031880 con 0x7fbbc809aa70 2024-04-16T16:15:53.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.885+0000 7fbbc6000640 1 -- 172.21.15.45:0/1908764363 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8031c50 con 0x7fbbc809aa70 2024-04-16T16:15:53.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.886+0000 7fbbcce00640 1 -- 172.21.15.45:0/1908764363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc809aa70 msgr2=0x7fbbc809ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:53.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.886+0000 7fbbcce00640 1 --2- 172.21.15.45:0/1908764363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc809aa70 0x7fbbc809ae50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8069300 tx=0x7fbbb802b920 comp rx=0 tx=0).stop 2024-04-16T16:15:53.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.886+0000 7fbbcce00640 1 -- 172.21.15.45:0/1908764363 shutdown_connections 2024-04-16T16:15:53.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.886+0000 7fbbcce00640 1 --2- 172.21.15.45:0/1908764363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbc8070c20 0x7fbbc8071090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:53.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.886+0000 7fbbcce00640 1 --2- 172.21.15.45:0/1908764363 >> v2:172.21.15.103:3300/0 conn(0x7fbbc806e810 0x7fbbc80d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:53.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.886+0000 7fbbcce00640 1 --2- 172.21.15.45:0/1908764363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc809aa70 0x7fbbc809ae50 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:53.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.886+0000 7fbbcce00640 1 -- 172.21.15.45:0/1908764363 >> 172.21.15.45:0/1908764363 conn(0x7fbbc80d2610 msgr2=0x7fbbc80d2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:53.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.886+0000 7fbbcce00640 1 -- 172.21.15.45:0/1908764363 shutdown_connections 2024-04-16T16:15:53.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.886+0000 7fbbcce00640 1 -- 172.21.15.45:0/1908764363 wait complete. 2024-04-16T16:15:53.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.888+0000 7fbbcce00640 1 Processor -- start 2024-04-16T16:15:53.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.888+0000 7fbbcce00640 1 -- start start 2024-04-16T16:15:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.889+0000 7fbbcce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc806e810 0x7fbbc8158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.889+0000 7fbbcce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbc8070c20 0x7fbbc80b7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.889+0000 7fbbcce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbc809aa70 0x7fbbc8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.889+0000 7fbbcce00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc806e0b0 con 0x7fbbc806e810 2024-04-16T16:15:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.889+0000 7fbbcce00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc8070590 con 0x7fbbc8070c20 2024-04-16T16:15:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.889+0000 7fbbcce00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbbc8070870 con 0x7fbbc809aa70 2024-04-16T16:15:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.889+0000 7fbbc7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc806e810 0x7fbbc8158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:53.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.889+0000 7fbbc7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc806e810 0x7fbbc8158270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59010/0 (socket says 172.21.15.45:59010) 2024-04-16T16:15:53.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.889+0000 7fbbc7400640 1 -- 172.21.15.45:0/2597972256 learned_addr learned my addr 172.21.15.45:0/2597972256 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:53.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.889+0000 7fbbc7400640 1 -- 172.21.15.45:0/2597972256 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbc809aa70 msgr2=0x7fbbc8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:53.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.889+0000 7fbbc7400640 1 --2- 172.21.15.45:0/2597972256 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbc809aa70 0x7fbbc8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:53.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.889+0000 7fbbc7400640 1 -- 172.21.15.45:0/2597972256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbc8070c20 msgr2=0x7fbbc80b7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:53.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.889+0000 7fbbc7400640 1 --2- 172.21.15.45:0/2597972256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbc8070c20 0x7fbbc80b7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:53.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.889+0000 7fbbc7400640 1 -- 172.21.15.45:0/2597972256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc815cbf0 con 0x7fbbc806e810 2024-04-16T16:15:53.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.890+0000 7fbbc7400640 1 --2- 172.21.15.45:0/2597972256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc806e810 0x7fbbc8158270 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8009960 tx=0x7fbbb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:53.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.890+0000 7fbbbf400640 1 -- 172.21.15.45:0/2597972256 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8040070 con 0x7fbbc806e810 2024-04-16T16:15:53.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.890+0000 7fbbbf400640 1 -- 172.21.15.45:0/2597972256 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb803a420 con 0x7fbbc806e810 2024-04-16T16:15:53.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.890+0000 7fbbbf400640 1 -- 172.21.15.45:0/2597972256 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8035040 con 0x7fbbc806e810 2024-04-16T16:15:53.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.890+0000 7fbbcce00640 1 -- 172.21.15.45:0/2597972256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc815ced0 con 0x7fbbc806e810 2024-04-16T16:15:53.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.890+0000 7fbbcce00640 1 -- 172.21.15.45:0/2597972256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc80b7500 con 0x7fbbc806e810 2024-04-16T16:15:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.892+0000 7fbbcce00640 1 -- 172.21.15.45:0/2597972256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbc8075780 con 0x7fbbc806e810 2024-04-16T16:15:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.893+0000 7fbbbf400640 1 -- 172.21.15.45:0/2597972256 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fbbb8038020 con 0x7fbbc806e810 2024-04-16T16:15:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.894+0000 7fbbbf400640 1 --2- 172.21.15.45:0/2597972256 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbba806ca60 0x7fbba806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:53.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.894+0000 7fbbbf400640 1 -- 172.21.15.45:0/2597972256 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7fbbb80ae270 con 0x7fbbc806e810 2024-04-16T16:15:53.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.894+0000 7fbbc6a00640 1 --2- 172.21.15.45:0/2597972256 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbba806ca60 0x7fbba806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.896+0000 7fbbc6a00640 1 --2- 172.21.15.45:0/2597972256 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbba806ca60 0x7fbba806ef20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4007130 tx=0x7fbbb4007300 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:53.899+0000 7fbbbf400640 1 -- 172.21.15.45:0/2597972256 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbbb807bfe0 con 0x7fbbc806e810 2024-04-16T16:15:54.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:54.129+0000 7fbbcce00640 1 -- 172.21.15.45:0/2597972256 --> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7fbbc809b850 con 0x7fbba806ca60 2024-04-16T16:15:54.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:54.133+0000 7fbbbf400640 1 -- 172.21.15.45:0/2597972256 <== mgr.4658 v2:172.21.15.103:6832/280591707 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7fbbc809b850 con 0x7fbba806ca60 2024-04-16T16:15:54.132 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:15:54.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:54.134+0000 7fbbcce00640 1 -- 172.21.15.45:0/2597972256 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbba806ca60 msgr2=0x7fbba806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:54.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:54.134+0000 7fbbcce00640 1 --2- 172.21.15.45:0/2597972256 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbba806ca60 0x7fbba806ef20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4007130 tx=0x7fbbb4007300 comp rx=0 tx=0).stop 2024-04-16T16:15:54.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:54.134+0000 7fbbcce00640 1 -- 172.21.15.45:0/2597972256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc806e810 msgr2=0x7fbbc8158270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:54.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:54.134+0000 7fbbcce00640 1 --2- 172.21.15.45:0/2597972256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc806e810 0x7fbbc8158270 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8009960 tx=0x7fbbb80046c0 comp rx=0 tx=0).stop 2024-04-16T16:15:54.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:54.134+0000 7fbbcce00640 1 -- 172.21.15.45:0/2597972256 shutdown_connections 2024-04-16T16:15:54.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:54.134+0000 7fbbcce00640 1 --2- 172.21.15.45:0/2597972256 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbc809aa70 0x7fbbc8159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:54.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:54.134+0000 7fbbcce00640 1 --2- 172.21.15.45:0/2597972256 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fbba806ca60 0x7fbba806ef20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:54.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:54.134+0000 7fbbcce00640 1 --2- 172.21.15.45:0/2597972256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbc8070c20 0x7fbbc80b7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:54.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:54.134+0000 7fbbcce00640 1 --2- 172.21.15.45:0/2597972256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc806e810 0x7fbbc8158270 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:54.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:54.134+0000 7fbbcce00640 1 -- 172.21.15.45:0/2597972256 >> 172.21.15.45:0/2597972256 conn(0x7fbbc80d2610 msgr2=0x7fbbc80afe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:54.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:54.134+0000 7fbbcce00640 1 -- 172.21.15.45:0/2597972256 shutdown_connections 2024-04-16T16:15:54.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:54.135+0000 7fbbcce00640 1 -- 172.21.15.45:0/2597972256 wait complete. 2024-04-16T16:15:55.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:15:55.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.380+0000 7f41b0e00640 1 Processor -- start 2024-04-16T16:15:55.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.381+0000 7f41b0e00640 1 -- start start 2024-04-16T16:15:55.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.382+0000 7f41b0e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f41ac0ba140 0x7f41ac0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:55.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.382+0000 7f41b0e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f41ac0cfc20 0x7f41ac0d35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:55.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.382+0000 7f41b0e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41ac0d2600 con 0x7f41ac162090 2024-04-16T16:15:55.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.382+0000 7f41b0e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f41ac0ba8b0 con 0x7f41ac0ba140 2024-04-16T16:15:55.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.382+0000 7f41b0e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f41ac0ba5d0 con 0x7f41ac0cfc20 2024-04-16T16:15:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.383+0000 7f41ab400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f41ac162090 0x7f41ac0cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55108/0 (socket says 172.21.15.45:55108) 2024-04-16T16:15:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.383+0000 7f41ab400640 1 -- 172.21.15.45:0/2487641161 learned_addr learned my addr 172.21.15.45:0/2487641161 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.383+0000 7f41aaa00640 1 --2- 172.21.15.45:0/2487641161 >> v2:172.21.15.45:3300/0 conn(0x7f41ac0ba140 0x7f41ac0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.383+0000 7f41aa000640 1 -- 172.21.15.45:0/2487641161 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2143075792 0 0) 0x7f41ac0d2600 con 0x7f41ac162090 2024-04-16T16:15:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.383+0000 7f41aaa00640 1 -- 172.21.15.45:0/2487641161 >> v2:172.21.15.103:3300/0 conn(0x7f41ac0cfc20 msgr2=0x7f41ac0d35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.383+0000 7f41abe00640 1 --2- 172.21.15.45:0/2487641161 >> v2:172.21.15.103:3300/0 conn(0x7f41ac0cfc20 0x7f41ac0d35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.383+0000 7f41aaa00640 1 --2- 172.21.15.45:0/2487641161 >> v2:172.21.15.103:3300/0 conn(0x7f41ac0cfc20 0x7f41ac0d35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.383+0000 7f41aaa00640 1 -- 172.21.15.45:0/2487641161 >> v1:172.21.15.45:6789/0 conn(0x7f41ac162090 legacy=0x7f41ac0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.383+0000 7f41aaa00640 1 -- 172.21.15.45:0/2487641161 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ac0d28a0 con 0x7f41ac0ba140 2024-04-16T16:15:55.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.384+0000 7f41aaa00640 1 --2- 172.21.15.45:0/2487641161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ac0ba140 0x7f41ac0d30a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f41ac0cb8f0 tx=0x7f419802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da26315b9a4fc8da server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:55.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.385+0000 7f41aa000640 1 -- 172.21.15.45:0/2487641161 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4198035020 con 0x7f41ac0ba140 2024-04-16T16:15:55.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.385+0000 7f41aa000640 1 -- 172.21.15.45:0/2487641161 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4198031830 con 0x7f41ac0ba140 2024-04-16T16:15:55.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.385+0000 7f41aa000640 1 -- 172.21.15.45:0/2487641161 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4198031c00 con 0x7f41ac0ba140 2024-04-16T16:15:55.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.385+0000 7f41b0e00640 1 -- 172.21.15.45:0/2487641161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ac0ba140 msgr2=0x7f41ac0d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:55.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.385+0000 7f41b0e00640 1 --2- 172.21.15.45:0/2487641161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ac0ba140 0x7f41ac0d30a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f41ac0cb8f0 tx=0x7f419802b8d0 comp rx=0 tx=0).stop 2024-04-16T16:15:55.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.385+0000 7f41b0e00640 1 -- 172.21.15.45:0/2487641161 shutdown_connections 2024-04-16T16:15:55.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.385+0000 7f41b0e00640 1 --2- 172.21.15.45:0/2487641161 >> v2:172.21.15.103:3300/0 conn(0x7f41ac0cfc20 0x7f41ac0d35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:55.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.385+0000 7f41b0e00640 1 --2- 172.21.15.45:0/2487641161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ac0ba140 0x7f41ac0d30a0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:55.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.385+0000 7f41b0e00640 1 -- 172.21.15.45:0/2487641161 >> 172.21.15.45:0/2487641161 conn(0x7f41ac0d1940 msgr2=0x7f41ac0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:55.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.385+0000 7f41b0e00640 1 -- 172.21.15.45:0/2487641161 shutdown_connections 2024-04-16T16:15:55.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.386+0000 7f41b0e00640 1 -- 172.21.15.45:0/2487641161 wait complete. 2024-04-16T16:15:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.387+0000 7f41b0e00640 1 Processor -- start 2024-04-16T16:15:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.387+0000 7f41b0e00640 1 -- start start 2024-04-16T16:15:55.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.388+0000 7f41b0e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41ac0cfc20 0x7f41ac0b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.388+0000 7f41b0e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ac162090 0x7f41ac0cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.388+0000 7f41b0e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ac0b3f60 0x7f41ac153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.388+0000 7f41b0e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac1625f0 con 0x7f41ac0b3f60 2024-04-16T16:15:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.388+0000 7f41b0e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac162470 con 0x7f41ac162090 2024-04-16T16:15:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.388+0000 7f41b0e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f41ac0d5520 con 0x7f41ac0cfc20 2024-04-16T16:15:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.389+0000 7f41abe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ac0b3f60 0x7f41ac153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.389+0000 7f41abe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ac0b3f60 0x7f41ac153c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59030/0 (socket says 172.21.15.45:59030) 2024-04-16T16:15:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.389+0000 7f41abe00640 1 -- 172.21.15.45:0/970277955 learned_addr learned my addr 172.21.15.45:0/970277955 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.389+0000 7f41ab400640 1 --2- 172.21.15.45:0/970277955 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41ac0cfc20 0x7f41ac0b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.389+0000 7f41aaa00640 1 --2- 172.21.15.45:0/970277955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ac162090 0x7f41ac0cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:55.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.389+0000 7f41abe00640 1 -- 172.21.15.45:0/970277955 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41ac0cfc20 msgr2=0x7f41ac0b55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.389+0000 7f41abe00640 1 --2- 172.21.15.45:0/970277955 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41ac0cfc20 0x7f41ac0b55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.389+0000 7f41abe00640 1 -- 172.21.15.45:0/970277955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ac162090 msgr2=0x7f41ac0cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.389+0000 7f41abe00640 1 --2- 172.21.15.45:0/970277955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ac162090 0x7f41ac0cdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.389+0000 7f41abe00640 1 -- 172.21.15.45:0/970277955 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ac0b49b0 con 0x7f41ac0b3f60 2024-04-16T16:15:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.390+0000 7f41abe00640 1 --2- 172.21.15.45:0/970277955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ac0b3f60 0x7f41ac153c00 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f41a400cd70 tx=0x7f41a400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.390+0000 7f41a3400640 1 -- 172.21.15.45:0/970277955 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4003b90 con 0x7f41ac0b3f60 2024-04-16T16:15:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.390+0000 7f41a3400640 1 -- 172.21.15.45:0/970277955 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a4003d30 con 0x7f41ac0b3f60 2024-04-16T16:15:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.390+0000 7f41a3400640 1 -- 172.21.15.45:0/970277955 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4004240 con 0x7f41ac0b3f60 2024-04-16T16:15:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.390+0000 7f41b0e00640 1 -- 172.21.15.45:0/970277955 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ac0b4c90 con 0x7f41ac0b3f60 2024-04-16T16:15:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.390+0000 7f41b0e00640 1 -- 172.21.15.45:0/970277955 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ac0c89e0 con 0x7f41ac0b3f60 2024-04-16T16:15:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.391+0000 7f41b0e00640 1 -- 172.21.15.45:0/970277955 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41ac075780 con 0x7f41ac0b3f60 2024-04-16T16:15:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.392+0000 7f41a3400640 1 -- 172.21.15.45:0/970277955 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f41a40043e0 con 0x7f41ac0b3f60 2024-04-16T16:15:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.393+0000 7f41a3400640 1 --2- 172.21.15.45:0/970277955 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f418c06ca90 0x7f418c06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.393+0000 7f41a3400640 1 -- 172.21.15.45:0/970277955 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7f41a408ce90 con 0x7f41ac0b3f60 2024-04-16T16:15:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.394+0000 7f41ab400640 1 --2- 172.21.15.45:0/970277955 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f418c06ca90 0x7f418c06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:55.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.395+0000 7f41ab400640 1 --2- 172.21.15.45:0/970277955 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f418c06ca90 0x7f418c06ef50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f419c000f60 tx=0x7f419c009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:55.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.398+0000 7f41a3400640 1 -- 172.21.15.45:0/970277955 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f41a405aaf0 con 0x7f41ac0b3f60 2024-04-16T16:15:55.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.632+0000 7f41b0e00640 1 -- 172.21.15.45:0/970277955 --> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f41ac0d3120 con 0x7f418c06ca90 2024-04-16T16:15:55.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.635+0000 7f41a3400640 1 -- 172.21.15.45:0/970277955 <== mgr.4658 v2:172.21.15.103:6832/280591707 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7f41ac0d3120 con 0x7f418c06ca90 2024-04-16T16:15:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:15:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.637+0000 7f41b0e00640 1 -- 172.21.15.45:0/970277955 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f418c06ca90 msgr2=0x7f418c06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.637+0000 7f41b0e00640 1 --2- 172.21.15.45:0/970277955 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f418c06ca90 0x7f418c06ef50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f419c000f60 tx=0x7f419c009040 comp rx=0 tx=0).stop 2024-04-16T16:15:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.637+0000 7f41b0e00640 1 -- 172.21.15.45:0/970277955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ac0b3f60 msgr2=0x7f41ac153c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.637+0000 7f41b0e00640 1 --2- 172.21.15.45:0/970277955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ac0b3f60 0x7f41ac153c00 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f41a400cd70 tx=0x7f41a400a660 comp rx=0 tx=0).stop 2024-04-16T16:15:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.638+0000 7f41b0e00640 1 -- 172.21.15.45:0/970277955 shutdown_connections 2024-04-16T16:15:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.638+0000 7f41b0e00640 1 --2- 172.21.15.45:0/970277955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ac0b3f60 0x7f41ac153c00 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.638+0000 7f41b0e00640 1 --2- 172.21.15.45:0/970277955 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f418c06ca90 0x7f418c06ef50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.638+0000 7f41b0e00640 1 --2- 172.21.15.45:0/970277955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ac162090 0x7f41ac0cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.638+0000 7f41b0e00640 1 --2- 172.21.15.45:0/970277955 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41ac0cfc20 0x7f41ac0b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.638+0000 7f41b0e00640 1 -- 172.21.15.45:0/970277955 >> 172.21.15.45:0/970277955 conn(0x7f41ac0d1940 msgr2=0x7f41ac15ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.638+0000 7f41b0e00640 1 -- 172.21.15.45:0/970277955 shutdown_connections 2024-04-16T16:15:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:55.638+0000 7f41b0e00640 1 -- 172.21.15.45:0/970277955 wait complete. 2024-04-16T16:15:56.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:15:56.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.890+0000 7f9cde000640 1 Processor -- start 2024-04-16T16:15:56.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.891+0000 7f9cde000640 1 -- start start 2024-04-16T16:15:56.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.892+0000 7f9cde000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cd80b1ce0 0x7f9cd8163010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.892+0000 7f9cde000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9cd806f410 0x7f9cd80b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.892+0000 7f9cde000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cd80b2600 con 0x7f9cd80ce710 2024-04-16T16:15:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.892+0000 7f9cde000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9cd80d02f0 con 0x7f9cd806f410 2024-04-16T16:15:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.892+0000 7f9cde000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd80cd210 con 0x7f9cd80b1ce0 2024-04-16T16:15:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.893+0000 7f9cdd600640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9cd80ce710 0x7f9cd806df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55122/0 (socket says 172.21.15.45:55122) 2024-04-16T16:15:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.893+0000 7f9cdd600640 1 -- 172.21.15.45:0/292876926 learned_addr learned my addr 172.21.15.45:0/292876926 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:56.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.893+0000 7f9cd7e00640 1 --2- 172.21.15.45:0/292876926 >> v2:172.21.15.103:3300/0 conn(0x7f9cd806f410 0x7f9cd80b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:56.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.893+0000 7f9cdcc00640 1 --2- 172.21.15.45:0/292876926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cd80b1ce0 0x7f9cd8163010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:56.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.893+0000 7f9cd7400640 1 -- 172.21.15.45:0/292876926 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3954698067 0 0) 0x7f9cd80b2600 con 0x7f9cd80ce710 2024-04-16T16:15:56.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.893+0000 7f9cd7400640 1 -- 172.21.15.45:0/292876926 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cc0003520 con 0x7f9cd80ce710 2024-04-16T16:15:56.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.894+0000 7f9cd7400640 1 -- 172.21.15.45:0/292876926 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3490709660 0 0) 0x7f9cc0003520 con 0x7f9cd80ce710 2024-04-16T16:15:56.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.894+0000 7f9cd7400640 1 -- 172.21.15.45:0/292876926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cd80b1ce0 msgr2=0x7f9cd8163010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:56.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.894+0000 7f9cd7400640 1 --2- 172.21.15.45:0/292876926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cd80b1ce0 0x7f9cd8163010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:56.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.894+0000 7f9cd7400640 1 -- 172.21.15.45:0/292876926 >> v2:172.21.15.103:3300/0 conn(0x7f9cd806f410 msgr2=0x7f9cd80b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:56.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.894+0000 7f9cd7400640 1 --2- 172.21.15.45:0/292876926 >> v2:172.21.15.103:3300/0 conn(0x7f9cd806f410 0x7f9cd80b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:56.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.894+0000 7f9cd7400640 1 -- 172.21.15.45:0/292876926 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd80cd4f0 con 0x7f9cd80ce710 2024-04-16T16:15:56.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.894+0000 7f9cd7400640 1 -- 172.21.15.45:0/292876926 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f9cd00019f0 con 0x7f9cd80ce710 2024-04-16T16:15:56.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.894+0000 7f9cd7400640 1 -- 172.21.15.45:0/292876926 >> v1:172.21.15.45:6789/0 conn(0x7f9cd80ce710 legacy=0x7f9cd806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:56.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.894+0000 7f9cd7400640 1 --2- 172.21.15.45:0/292876926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cc00039b0 0x7f9cc0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:56.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.894+0000 7f9cd7400640 1 --2- 172.21.15.45:0/292876926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc0004800 0x7f9cc0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:56.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.894+0000 7f9cd7400640 1 --2- 172.21.15.45:0/292876926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cd80b1ce0 0x7f9cc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.894+0000 7f9cd7400640 1 -- 172.21.15.45:0/292876926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd80d02f0 con 0x7f9cc00039b0 2024-04-16T16:15:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.894+0000 7f9cd7400640 1 -- 172.21.15.45:0/292876926 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd80cd210 con 0x7f9cc0004800 2024-04-16T16:15:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.894+0000 7f9cd7400640 1 -- 172.21.15.45:0/292876926 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc000ba40 con 0x7f9cd80b1ce0 2024-04-16T16:15:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.894+0000 7f9cdcc00640 1 --2- 172.21.15.45:0/292876926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cc00039b0 0x7f9cc0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.895+0000 7f9cd7e00640 1 --2- 172.21.15.45:0/292876926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc0004800 0x7f9cc0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.895+0000 7f9cdd600640 1 --2- 172.21.15.45:0/292876926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cd80b1ce0 0x7f9cc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.895+0000 7f9cdcc00640 1 -- 172.21.15.45:0/292876926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cd80b1ce0 msgr2=0x7f9cc00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.895+0000 7f9cdcc00640 1 --2- 172.21.15.45:0/292876926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cd80b1ce0 0x7f9cc00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.895+0000 7f9cdcc00640 1 -- 172.21.15.45:0/292876926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc0004800 msgr2=0x7f9cc0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.895+0000 7f9cdcc00640 1 --2- 172.21.15.45:0/292876926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc0004800 0x7f9cc0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.895+0000 7f9cdcc00640 1 -- 172.21.15.45:0/292876926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc0003520 con 0x7f9cc00039b0 2024-04-16T16:15:56.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.895+0000 7f9cd7e00640 1 --2- 172.21.15.45:0/292876926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc0004800 0x7f9cc0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:15:56.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.897+0000 7f9cdcc00640 1 --2- 172.21.15.45:0/292876926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cc00039b0 0x7f9cc0003dd0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009ac0 tx=0x7f9cc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1a445764945e93 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:56.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.897+0000 7f9cd7400640 1 -- 172.21.15.45:0/292876926 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc039070 con 0x7f9cc00039b0 2024-04-16T16:15:56.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.897+0000 7f9cd7400640 1 -- 172.21.15.45:0/292876926 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ccc02c6b0 con 0x7f9cc00039b0 2024-04-16T16:15:56.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.897+0000 7f9cd7400640 1 -- 172.21.15.45:0/292876926 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc009690 con 0x7f9cc00039b0 2024-04-16T16:15:56.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.898+0000 7f9cde000640 1 -- 172.21.15.45:0/292876926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cc00039b0 msgr2=0x7f9cc0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:56.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.898+0000 7f9cde000640 1 --2- 172.21.15.45:0/292876926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cc00039b0 0x7f9cc0003dd0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009ac0 tx=0x7f9cc0005830 comp rx=0 tx=0).stop 2024-04-16T16:15:56.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.898+0000 7f9cdd600640 1 -- 172.21.15.45:0/292876926 reap_dead start 2024-04-16T16:15:56.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.898+0000 7f9cde000640 1 -- 172.21.15.45:0/292876926 shutdown_connections 2024-04-16T16:15:56.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.898+0000 7f9cde000640 1 -- 172.21.15.45:0/292876926 >> 172.21.15.45:0/292876926 conn(0x7f9cd806b930 msgr2=0x7f9cd806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:56.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.898+0000 7f9cde000640 1 -- 172.21.15.45:0/292876926 shutdown_connections 2024-04-16T16:15:56.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.898+0000 7f9cde000640 1 -- 172.21.15.45:0/292876926 wait complete. 2024-04-16T16:15:56.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.900+0000 7f9cde000640 1 Processor -- start 2024-04-16T16:15:56.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.900+0000 7f9cde000640 1 -- start start 2024-04-16T16:15:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.901+0000 7f9cde000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc00039b0 0x7f9cd81576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.901+0000 7f9cde000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cd81562b0 0x7f9cd8158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.901+0000 7f9cde000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cd8161f30 0x7f9cd81623b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.901+0000 7f9cde000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd80d0040 con 0x7f9cd81562b0 2024-04-16T16:15:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.901+0000 7f9cde000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd80cfec0 con 0x7f9cc00039b0 2024-04-16T16:15:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.901+0000 7f9cde000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd8164bb0 con 0x7f9cd8161f30 2024-04-16T16:15:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.901+0000 7f9cd7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cd81562b0 0x7f9cd8158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.901+0000 7f9cd7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cd81562b0 0x7f9cd8158e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59046/0 (socket says 172.21.15.45:59046) 2024-04-16T16:15:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.902+0000 7f9cd7e00640 1 -- 172.21.15.45:0/597248674 learned_addr learned my addr 172.21.15.45:0/597248674 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.902+0000 7f9cdcc00640 1 --2- 172.21.15.45:0/597248674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc00039b0 0x7f9cd81576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.902+0000 7f9cdd600640 1 --2- 172.21.15.45:0/597248674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cd8161f30 0x7f9cd81623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:56.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.902+0000 7f9cd7e00640 1 -- 172.21.15.45:0/597248674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cd8161f30 msgr2=0x7f9cd81623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.902+0000 7f9cd7e00640 1 --2- 172.21.15.45:0/597248674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cd8161f30 0x7f9cd81623b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.902+0000 7f9cd7e00640 1 -- 172.21.15.45:0/597248674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc00039b0 msgr2=0x7f9cd81576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.902+0000 7f9cd7e00640 1 --2- 172.21.15.45:0/597248674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc00039b0 0x7f9cd81576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.902+0000 7f9cd7e00640 1 -- 172.21.15.45:0/597248674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd81593a0 con 0x7f9cd81562b0 2024-04-16T16:15:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.902+0000 7f9cdd600640 1 --2- 172.21.15.45:0/597248674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cd8161f30 0x7f9cd81623b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.902+0000 7f9cd7e00640 1 --2- 172.21.15.45:0/597248674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cd81562b0 0x7f9cd8158e60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f9cc80116d0 tx=0x7f9cc8011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.903+0000 7f9cd4c00640 1 -- 172.21.15.45:0/597248674 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc800f960 con 0x7f9cd81562b0 2024-04-16T16:15:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.903+0000 7f9cd4c00640 1 -- 172.21.15.45:0/597248674 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc800fb00 con 0x7f9cd81562b0 2024-04-16T16:15:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.903+0000 7f9cd4c00640 1 -- 172.21.15.45:0/597248674 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8015460 con 0x7f9cd81562b0 2024-04-16T16:15:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.903+0000 7f9cde000640 1 -- 172.21.15.45:0/597248674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cd815a080 con 0x7f9cd81562b0 2024-04-16T16:15:56.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.903+0000 7f9cde000640 1 -- 172.21.15.45:0/597248674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cd815a640 con 0x7f9cd81562b0 2024-04-16T16:15:56.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.904+0000 7f9cde000640 1 -- 172.21.15.45:0/597248674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cd81628f0 con 0x7f9cd81562b0 2024-04-16T16:15:56.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.905+0000 7f9cd4c00640 1 -- 172.21.15.45:0/597248674 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f9cc8015600 con 0x7f9cd81562b0 2024-04-16T16:15:56.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.906+0000 7f9cd4c00640 1 --2- 172.21.15.45:0/597248674 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f9cbc06ca90 0x7f9cbc06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:56.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.907+0000 7f9cd4c00640 1 -- 172.21.15.45:0/597248674 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7f9cc808cbd0 con 0x7f9cd81562b0 2024-04-16T16:15:56.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.907+0000 7f9cdcc00640 1 --2- 172.21.15.45:0/597248674 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f9cbc06ca90 0x7f9cbc06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:56.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.908+0000 7f9cdcc00640 1 --2- 172.21.15.45:0/597248674 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f9cbc06ca90 0x7f9cbc06ef50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc036000 tx=0x7f9ccc030000 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:56.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:56.912+0000 7f9cd4c00640 1 -- 172.21.15.45:0/597248674 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9cc8019030 con 0x7f9cd81562b0 2024-04-16T16:15:57.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:57.147+0000 7f9cde000640 1 -- 172.21.15.45:0/597248674 --> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f9cd8075780 con 0x7f9cbc06ca90 2024-04-16T16:15:57.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:57.151+0000 7f9cd4c00640 1 -- 172.21.15.45:0/597248674 <== mgr.4658 v2:172.21.15.103:6832/280591707 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7f9cd8075780 con 0x7f9cbc06ca90 2024-04-16T16:15:57.150 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:15:57.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:57.153+0000 7f9cde000640 1 -- 172.21.15.45:0/597248674 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f9cbc06ca90 msgr2=0x7f9cbc06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:57.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:57.153+0000 7f9cde000640 1 --2- 172.21.15.45:0/597248674 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f9cbc06ca90 0x7f9cbc06ef50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc036000 tx=0x7f9ccc030000 comp rx=0 tx=0).stop 2024-04-16T16:15:57.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:57.153+0000 7f9cde000640 1 -- 172.21.15.45:0/597248674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cd81562b0 msgr2=0x7f9cd8158e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:57.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:57.153+0000 7f9cde000640 1 --2- 172.21.15.45:0/597248674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cd81562b0 0x7f9cd8158e60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f9cc80116d0 tx=0x7f9cc8011b90 comp rx=0 tx=0).stop 2024-04-16T16:15:57.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:57.153+0000 7f9cde000640 1 -- 172.21.15.45:0/597248674 shutdown_connections 2024-04-16T16:15:57.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:57.153+0000 7f9cde000640 1 --2- 172.21.15.45:0/597248674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cd8161f30 0x7f9cd81623b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:57.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:57.153+0000 7f9cde000640 1 --2- 172.21.15.45:0/597248674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cd81562b0 0x7f9cd8158e60 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:57.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:57.153+0000 7f9cde000640 1 --2- 172.21.15.45:0/597248674 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f9cbc06ca90 0x7f9cbc06ef50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:57.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:57.153+0000 7f9cde000640 1 --2- 172.21.15.45:0/597248674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc00039b0 0x7f9cd81576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:57.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:57.153+0000 7f9cde000640 1 -- 172.21.15.45:0/597248674 >> 172.21.15.45:0/597248674 conn(0x7f9cd806b930 msgr2=0x7f9cd806e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:57.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:57.153+0000 7f9cde000640 1 -- 172.21.15.45:0/597248674 shutdown_connections 2024-04-16T16:15:57.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:57.153+0000 7f9cde000640 1 -- 172.21.15.45:0/597248674 wait complete. 2024-04-16T16:15:58.174 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:15:58.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.405+0000 7f6cba400640 1 Processor -- start 2024-04-16T16:15:58.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.405+0000 7f6cba400640 1 -- start start 2024-04-16T16:15:58.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.407+0000 7f6cba400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6cb4145230 0x7f6cb415bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:58.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.407+0000 7f6cba400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6cb415cbe0 0x7f6cb40cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:58.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.407+0000 7f6cba400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cb415d3b0 con 0x7f6cb40c5670 2024-04-16T16:15:58.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.407+0000 7f6cba400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f6cb415dfc0 con 0x7f6cb415cbe0 2024-04-16T16:15:58.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.407+0000 7f6cba400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6cb415d690 con 0x7f6cb4145230 2024-04-16T16:15:58.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.407+0000 7f6cb9a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6cb40c5670 0x7f6cb415e450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55134/0 (socket says 172.21.15.45:55134) 2024-04-16T16:15:58.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.407+0000 7f6cb9a00640 1 -- 172.21.15.45:0/3096432298 learned_addr learned my addr 172.21.15.45:0/3096432298 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:15:58.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.407+0000 7f6cb3e00640 1 --2- 172.21.15.45:0/3096432298 >> v2:172.21.15.45:3300/0 conn(0x7f6cb415cbe0 0x7f6cb40cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:58.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.408+0000 7f6cb9000640 1 --2- 172.21.15.45:0/3096432298 >> v2:172.21.15.103:3300/0 conn(0x7f6cb4145230 0x7f6cb415bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:58.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.408+0000 7f6cb3400640 1 -- 172.21.15.45:0/3096432298 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2224504888 0 0) 0x7f6cb415d3b0 con 0x7f6cb40c5670 2024-04-16T16:15:58.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.408+0000 7f6cb3400640 1 -- 172.21.15.45:0/3096432298 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c9c003520 con 0x7f6cb40c5670 2024-04-16T16:15:58.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.408+0000 7f6cb3e00640 1 -- 172.21.15.45:0/3096432298 >> v2:172.21.15.103:3300/0 conn(0x7f6cb4145230 msgr2=0x7f6cb415bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:58.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.408+0000 7f6cb3e00640 1 --2- 172.21.15.45:0/3096432298 >> v2:172.21.15.103:3300/0 conn(0x7f6cb4145230 0x7f6cb415bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:58.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.408+0000 7f6cb3e00640 1 -- 172.21.15.45:0/3096432298 >> v1:172.21.15.45:6789/0 conn(0x7f6cb40c5670 legacy=0x7f6cb415e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:58.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.408+0000 7f6cb3e00640 1 -- 172.21.15.45:0/3096432298 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb415d970 con 0x7f6cb415cbe0 2024-04-16T16:15:58.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.408+0000 7f6cb9000640 1 --2- 172.21.15.45:0/3096432298 >> v2:172.21.15.103:3300/0 conn(0x7f6cb4145230 0x7f6cb415bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:15:58.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.409+0000 7f6cb3e00640 1 --2- 172.21.15.45:0/3096432298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb415cbe0 0x7f6cb40cd0e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4073180 tx=0x7f6ca002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bad55cf3305611e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:58.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.410+0000 7f6cb3400640 1 -- 172.21.15.45:0/3096432298 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca00315a0 con 0x7f6cb415cbe0 2024-04-16T16:15:58.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.410+0000 7f6cb3400640 1 -- 172.21.15.45:0/3096432298 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ca0037070 con 0x7f6cb415cbe0 2024-04-16T16:15:58.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.410+0000 7f6cb3400640 1 -- 172.21.15.45:0/3096432298 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0031960 con 0x7f6cb415cbe0 2024-04-16T16:15:58.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.410+0000 7f6cba400640 1 -- 172.21.15.45:0/3096432298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb415cbe0 msgr2=0x7f6cb40cd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:58.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.410+0000 7f6cba400640 1 --2- 172.21.15.45:0/3096432298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb415cbe0 0x7f6cb40cd0e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4073180 tx=0x7f6ca002b840 comp rx=0 tx=0).stop 2024-04-16T16:15:58.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.410+0000 7f6cba400640 1 -- 172.21.15.45:0/3096432298 shutdown_connections 2024-04-16T16:15:58.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.410+0000 7f6cba400640 1 --2- 172.21.15.45:0/3096432298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb415cbe0 0x7f6cb40cd0e0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:58.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.410+0000 7f6cba400640 1 --2- 172.21.15.45:0/3096432298 >> v2:172.21.15.103:3300/0 conn(0x7f6cb4145230 0x7f6cb415bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:58.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.410+0000 7f6cba400640 1 -- 172.21.15.45:0/3096432298 >> 172.21.15.45:0/3096432298 conn(0x7f6cb406b930 msgr2=0x7f6cb406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:58.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.410+0000 7f6cba400640 1 -- 172.21.15.45:0/3096432298 shutdown_connections 2024-04-16T16:15:58.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.410+0000 7f6cba400640 1 -- 172.21.15.45:0/3096432298 wait complete. 2024-04-16T16:15:58.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.412+0000 7f6cba400640 1 Processor -- start 2024-04-16T16:15:58.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.412+0000 7f6cba400640 1 -- start start 2024-04-16T16:15:58.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.413+0000 7f6cba400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cb40c5670 0x7f6cb41f4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:58.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.413+0000 7f6cba400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4145230 0x7f6cb40cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:58.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.413+0000 7f6cba400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cb415cbe0 0x7f6cb40b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:58.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.413+0000 7f6cba400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb415d070 con 0x7f6cb4145230 2024-04-16T16:15:58.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.413+0000 7f6cba400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb40cd620 con 0x7f6cb415cbe0 2024-04-16T16:15:58.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.413+0000 7f6cba400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6cb415eb60 con 0x7f6cb40c5670 2024-04-16T16:15:58.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.413+0000 7f6cb3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4145230 0x7f6cb40cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:58.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.414+0000 7f6cb3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4145230 0x7f6cb40cdf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59062/0 (socket says 172.21.15.45:59062) 2024-04-16T16:15:58.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.414+0000 7f6cb3e00640 1 -- 172.21.15.45:0/4023823009 learned_addr learned my addr 172.21.15.45:0/4023823009 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:58.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.414+0000 7f6cb9000640 1 --2- 172.21.15.45:0/4023823009 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cb40c5670 0x7f6cb41f4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:58.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.414+0000 7f6cb9a00640 1 --2- 172.21.15.45:0/4023823009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cb415cbe0 0x7f6cb40b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:58.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.414+0000 7f6cb3e00640 1 -- 172.21.15.45:0/4023823009 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cb40c5670 msgr2=0x7f6cb41f4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:58.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.414+0000 7f6cb3e00640 1 --2- 172.21.15.45:0/4023823009 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cb40c5670 0x7f6cb41f4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:58.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.414+0000 7f6cb3e00640 1 -- 172.21.15.45:0/4023823009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cb415cbe0 msgr2=0x7f6cb40b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:58.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.414+0000 7f6cb3e00640 1 --2- 172.21.15.45:0/4023823009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cb415cbe0 0x7f6cb40b1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:58.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.414+0000 7f6cb3e00640 1 -- 172.21.15.45:0/4023823009 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb40ce760 con 0x7f6cb4145230 2024-04-16T16:15:58.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.414+0000 7f6cb9000640 1 --2- 172.21.15.45:0/4023823009 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cb40c5670 0x7f6cb41f4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:58.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.414+0000 7f6cb3e00640 1 --2- 172.21.15.45:0/4023823009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4145230 0x7f6cb40cdf10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0000c00 tx=0x7f6ca00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:58.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.415+0000 7f6cb0c00640 1 -- 172.21.15.45:0/4023823009 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0032630 con 0x7f6cb4145230 2024-04-16T16:15:58.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.415+0000 7f6cb0c00640 1 -- 172.21.15.45:0/4023823009 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ca00327f0 con 0x7f6cb4145230 2024-04-16T16:15:58.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.415+0000 7f6cb0c00640 1 -- 172.21.15.45:0/4023823009 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0032be0 con 0x7f6cb4145230 2024-04-16T16:15:58.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.415+0000 7f6cba400640 1 -- 172.21.15.45:0/4023823009 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cb40cea40 con 0x7f6cb4145230 2024-04-16T16:15:58.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.415+0000 7f6cba400640 1 -- 172.21.15.45:0/4023823009 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cb40cf000 con 0x7f6cb4145230 2024-04-16T16:15:58.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.416+0000 7f6cba400640 1 -- 172.21.15.45:0/4023823009 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cb4075780 con 0x7f6cb4145230 2024-04-16T16:15:58.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.417+0000 7f6cb0c00640 1 -- 172.21.15.45:0/4023823009 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f6ca004d3f0 con 0x7f6cb4145230 2024-04-16T16:15:58.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.418+0000 7f6cb0c00640 1 --2- 172.21.15.45:0/4023823009 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6c9406ca90 0x7f6c9406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:58.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.418+0000 7f6cb0c00640 1 -- 172.21.15.45:0/4023823009 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7f6ca002f030 con 0x7f6cb4145230 2024-04-16T16:15:58.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.419+0000 7f6cb9000640 1 --2- 172.21.15.45:0/4023823009 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6c9406ca90 0x7f6c9406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:58.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.420+0000 7f6cb9000640 1 --2- 172.21.15.45:0/4023823009 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6c9406ca90 0x7f6c9406ef50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8006fd0 tx=0x7f6ca8008040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:58.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.423+0000 7f6cb0c00640 1 -- 172.21.15.45:0/4023823009 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ca0040070 con 0x7f6cb4145230 2024-04-16T16:15:58.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.655+0000 7f6cba400640 1 -- 172.21.15.45:0/4023823009 --> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f6cb40b5950 con 0x7f6c9406ca90 2024-04-16T16:15:58.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.659+0000 7f6cb0c00640 1 -- 172.21.15.45:0/4023823009 <== mgr.4658 v2:172.21.15.103:6832/280591707 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7f6cb40b5950 con 0x7f6c9406ca90 2024-04-16T16:15:58.659 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:15:58.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.661+0000 7f6cba400640 1 -- 172.21.15.45:0/4023823009 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6c9406ca90 msgr2=0x7f6c9406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:58.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.661+0000 7f6cba400640 1 --2- 172.21.15.45:0/4023823009 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6c9406ca90 0x7f6c9406ef50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8006fd0 tx=0x7f6ca8008040 comp rx=0 tx=0).stop 2024-04-16T16:15:58.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.661+0000 7f6cba400640 1 -- 172.21.15.45:0/4023823009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4145230 msgr2=0x7f6cb40cdf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:58.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.661+0000 7f6cba400640 1 --2- 172.21.15.45:0/4023823009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4145230 0x7f6cb40cdf10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0000c00 tx=0x7f6ca00046c0 comp rx=0 tx=0).stop 2024-04-16T16:15:58.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.661+0000 7f6cba400640 1 -- 172.21.15.45:0/4023823009 shutdown_connections 2024-04-16T16:15:58.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.661+0000 7f6cba400640 1 --2- 172.21.15.45:0/4023823009 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f6c9406ca90 0x7f6c9406ef50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:58.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.661+0000 7f6cba400640 1 --2- 172.21.15.45:0/4023823009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cb415cbe0 0x7f6cb40b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:58.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.661+0000 7f6cba400640 1 --2- 172.21.15.45:0/4023823009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4145230 0x7f6cb40cdf10 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:58.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.661+0000 7f6cba400640 1 --2- 172.21.15.45:0/4023823009 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cb40c5670 0x7f6cb41f4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:58.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.661+0000 7f6cba400640 1 -- 172.21.15.45:0/4023823009 >> 172.21.15.45:0/4023823009 conn(0x7f6cb406b930 msgr2=0x7f6cb415eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:58.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.661+0000 7f6cba400640 1 -- 172.21.15.45:0/4023823009 shutdown_connections 2024-04-16T16:15:58.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:58.661+0000 7f6cba400640 1 -- 172.21.15.45:0/4023823009 wait complete. 2024-04-16T16:15:59.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:15:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.918+0000 7f0a2e400640 1 Processor -- start 2024-04-16T16:15:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.919+0000 7f0a2e400640 1 -- start start 2024-04-16T16:15:59.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.920+0000 7f0a2e400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0a280ba140 0x7f0a280d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:59.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.920+0000 7f0a2e400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a280cfc20 0x7f0a280d35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:59.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.920+0000 7f0a2e400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a280d2600 con 0x7f0a28162090 2024-04-16T16:15:59.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.920+0000 7f0a2e400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0a280ba8b0 con 0x7f0a280ba140 2024-04-16T16:15:59.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.920+0000 7f0a2e400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0a280ba5d0 con 0x7f0a280cfc20 2024-04-16T16:15:59.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.920+0000 7f0a27e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0a280ba140 0x7f0a280d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:59.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.921+0000 7f0a27e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0a280ba140 0x7f0a280d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59076/0 (socket says 172.21.15.45:59076) 2024-04-16T16:15:59.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.921+0000 7f0a27e00640 1 -- 172.21.15.45:0/1980546064 learned_addr learned my addr 172.21.15.45:0/1980546064 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:59.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.921+0000 7f0a2da00640 1 --2- 172.21.15.45:0/1980546064 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a280cfc20 0x7f0a280d35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:59.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.921+0000 7f0a27e00640 1 -- 172.21.15.45:0/1980546064 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a280cfc20 msgr2=0x7f0a280d35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:59.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.921+0000 7f0a27e00640 1 --2- 172.21.15.45:0/1980546064 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a280cfc20 0x7f0a280d35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:59.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.921+0000 7f0a27e00640 1 -- 172.21.15.45:0/1980546064 >> v1:172.21.15.103:6789/0 conn(0x7f0a28162090 legacy=0x7f0a280cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:15:59.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.921+0000 7f0a27e00640 1 -- 172.21.15.45:0/1980546064 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a280d28a0 con 0x7f0a280ba140 2024-04-16T16:15:59.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.922+0000 7f0a27e00640 1 --2- 172.21.15.45:0/1980546064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a280ba140 0x7f0a280d30a0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009870 tx=0x7f0a1002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3174853eec14d4c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:59.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.923+0000 7f0a27400640 1 -- 172.21.15.45:0/1980546064 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a10034070 con 0x7f0a280ba140 2024-04-16T16:15:59.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.923+0000 7f0a27400640 1 -- 172.21.15.45:0/1980546064 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a10030840 con 0x7f0a280ba140 2024-04-16T16:15:59.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.923+0000 7f0a27400640 1 -- 172.21.15.45:0/1980546064 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a10030c10 con 0x7f0a280ba140 2024-04-16T16:15:59.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.923+0000 7f0a2e400640 1 -- 172.21.15.45:0/1980546064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a280ba140 msgr2=0x7f0a280d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:59.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.923+0000 7f0a2e400640 1 --2- 172.21.15.45:0/1980546064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a280ba140 0x7f0a280d30a0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009870 tx=0x7f0a1002b980 comp rx=0 tx=0).stop 2024-04-16T16:15:59.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.923+0000 7f0a2e400640 1 -- 172.21.15.45:0/1980546064 shutdown_connections 2024-04-16T16:15:59.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.923+0000 7f0a2e400640 1 --2- 172.21.15.45:0/1980546064 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a280cfc20 0x7f0a280d35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:59.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.923+0000 7f0a2e400640 1 --2- 172.21.15.45:0/1980546064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a280ba140 0x7f0a280d30a0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:59.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.923+0000 7f0a2e400640 1 -- 172.21.15.45:0/1980546064 >> 172.21.15.45:0/1980546064 conn(0x7f0a280d1940 msgr2=0x7f0a280d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:15:59.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.923+0000 7f0a2e400640 1 -- 172.21.15.45:0/1980546064 shutdown_connections 2024-04-16T16:15:59.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.924+0000 7f0a2e400640 1 -- 172.21.15.45:0/1980546064 wait complete. 2024-04-16T16:15:59.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.925+0000 7f0a2e400640 1 Processor -- start 2024-04-16T16:15:59.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.925+0000 7f0a2e400640 1 -- start start 2024-04-16T16:15:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.926+0000 7f0a2e400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a280cfc20 0x7f0a280ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.926+0000 7f0a2e400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a28162090 0x7f0a280c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.927+0000 7f0a2e400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a280ca360 0x7f0a28156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.927+0000 7f0a2e400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281625f0 con 0x7f0a280ca360 2024-04-16T16:15:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.927+0000 7f0a2e400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0a28162470 con 0x7f0a280cfc20 2024-04-16T16:15:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.927+0000 7f0a2e400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0a280d5520 con 0x7f0a28162090 2024-04-16T16:15:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.927+0000 7f0a2da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a280ca360 0x7f0a28156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.927+0000 7f0a2da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a280ca360 0x7f0a28156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59082/0 (socket says 172.21.15.45:59082) 2024-04-16T16:15:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.927+0000 7f0a2da00640 1 -- 172.21.15.45:0/1120100935 learned_addr learned my addr 172.21.15.45:0/1120100935 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:15:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.927+0000 7f0a27e00640 1 --2- 172.21.15.45:0/1120100935 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a28162090 0x7f0a280c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.927+0000 7f0a2d000640 1 --2- 172.21.15.45:0/1120100935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a280cfc20 0x7f0a280ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.927+0000 7f0a2da00640 1 -- 172.21.15.45:0/1120100935 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a28162090 msgr2=0x7f0a280c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.927+0000 7f0a2da00640 1 --2- 172.21.15.45:0/1120100935 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a28162090 0x7f0a280c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.927+0000 7f0a2da00640 1 -- 172.21.15.45:0/1120100935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a280cfc20 msgr2=0x7f0a280ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:15:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.927+0000 7f0a2da00640 1 --2- 172.21.15.45:0/1120100935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a280cfc20 0x7f0a280ce080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:15:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.927+0000 7f0a2da00640 1 -- 172.21.15.45:0/1120100935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a280ca8e0 con 0x7f0a280ca360 2024-04-16T16:15:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.927+0000 7f0a27e00640 1 --2- 172.21.15.45:0/1120100935 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a28162090 0x7f0a280c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:15:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.928+0000 7f0a2da00640 1 --2- 172.21.15.45:0/1120100935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a280ca360 0x7f0a28156d00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000d600 tx=0x7f0a2000dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.928+0000 7f0a24c00640 1 -- 172.21.15.45:0/1120100935 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a200125f0 con 0x7f0a280ca360 2024-04-16T16:15:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.928+0000 7f0a24c00640 1 -- 172.21.15.45:0/1120100935 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a20019070 con 0x7f0a280ca360 2024-04-16T16:15:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.928+0000 7f0a2e400640 1 -- 172.21.15.45:0/1120100935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a280cabc0 con 0x7f0a280ca360 2024-04-16T16:15:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.928+0000 7f0a24c00640 1 -- 172.21.15.45:0/1120100935 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a200129d0 con 0x7f0a280ca360 2024-04-16T16:15:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.928+0000 7f0a2e400640 1 -- 172.21.15.45:0/1120100935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a280b4dc0 con 0x7f0a280ca360 2024-04-16T16:15:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.929+0000 7f0a2e400640 1 -- 172.21.15.45:0/1120100935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a280b48d0 con 0x7f0a280ca360 2024-04-16T16:15:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.930+0000 7f0a24c00640 1 -- 172.21.15.45:0/1120100935 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f0a20012b70 con 0x7f0a280ca360 2024-04-16T16:15:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.931+0000 7f0a24c00640 1 --2- 172.21.15.45:0/1120100935 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f0a0406ca60 0x7f0a0406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:15:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.931+0000 7f0a24c00640 1 -- 172.21.15.45:0/1120100935 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7f0a2008c870 con 0x7f0a280ca360 2024-04-16T16:15:59.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.932+0000 7f0a2d000640 1 --2- 172.21.15.45:0/1120100935 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f0a0406ca60 0x7f0a0406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:15:59.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.933+0000 7f0a2d000640 1 --2- 172.21.15.45:0/1120100935 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f0a0406ca60 0x7f0a0406ef20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0a18007fd0 tx=0x7f0a18009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:15:59.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:15:59.936+0000 7f0a24c00640 1 -- 172.21.15.45:0/1120100935 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a2005a4c0 con 0x7f0a280ca360 2024-04-16T16:16:00.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:00.167+0000 7f0a2e400640 1 -- 172.21.15.45:0/1120100935 --> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f0a28075780 con 0x7f0a0406ca60 2024-04-16T16:16:00.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:00.171+0000 7f0a24c00640 1 -- 172.21.15.45:0/1120100935 <== mgr.4658 v2:172.21.15.103:6832/280591707 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7f0a28075780 con 0x7f0a0406ca60 2024-04-16T16:16:00.170 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:16:00.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:00.173+0000 7f0a2e400640 1 -- 172.21.15.45:0/1120100935 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f0a0406ca60 msgr2=0x7f0a0406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:00.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:00.173+0000 7f0a2e400640 1 --2- 172.21.15.45:0/1120100935 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f0a0406ca60 0x7f0a0406ef20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0a18007fd0 tx=0x7f0a18009040 comp rx=0 tx=0).stop 2024-04-16T16:16:00.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:00.173+0000 7f0a2e400640 1 -- 172.21.15.45:0/1120100935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a280ca360 msgr2=0x7f0a28156d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:00.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:00.173+0000 7f0a2e400640 1 --2- 172.21.15.45:0/1120100935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a280ca360 0x7f0a28156d00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000d600 tx=0x7f0a2000dac0 comp rx=0 tx=0).stop 2024-04-16T16:16:00.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:00.173+0000 7f0a2e400640 1 -- 172.21.15.45:0/1120100935 shutdown_connections 2024-04-16T16:16:00.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:00.173+0000 7f0a2e400640 1 --2- 172.21.15.45:0/1120100935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a280ca360 0x7f0a28156d00 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:00.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:00.173+0000 7f0a2e400640 1 --2- 172.21.15.45:0/1120100935 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a28162090 0x7f0a280c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:00.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:00.173+0000 7f0a2e400640 1 --2- 172.21.15.45:0/1120100935 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f0a0406ca60 0x7f0a0406ef20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:00.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:00.173+0000 7f0a2e400640 1 --2- 172.21.15.45:0/1120100935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a280cfc20 0x7f0a280ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:00.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:00.173+0000 7f0a2e400640 1 -- 172.21.15.45:0/1120100935 >> 172.21.15.45:0/1120100935 conn(0x7f0a280d1940 msgr2=0x7f0a280d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:00.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:00.173+0000 7f0a2e400640 1 -- 172.21.15.45:0/1120100935 shutdown_connections 2024-04-16T16:16:00.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:00.173+0000 7f0a2e400640 1 -- 172.21.15.45:0/1120100935 wait complete. 2024-04-16T16:16:01.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:16:01.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.429+0000 7fec95000640 1 Processor -- start 2024-04-16T16:16:01.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.429+0000 7fec95000640 1 -- start start 2024-04-16T16:16:01.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.431+0000 7fec95000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fec900c5670 0x7fec900c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:01.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.431+0000 7fec95000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec9015d3b0 con 0x7fec9015bc80 2024-04-16T16:16:01.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.431+0000 7fec95000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec9015d690 con 0x7fec90145230 2024-04-16T16:16:01.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.431+0000 7fec95000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fec9015dfc0 con 0x7fec900c5670 2024-04-16T16:16:01.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.431+0000 7fec87e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fec9015bc80 0x7fec900cd0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55136/0 (socket says 172.21.15.45:55136) 2024-04-16T16:16:01.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.431+0000 7fec87e00640 1 -- 172.21.15.45:0/3827040142 learned_addr learned my addr 172.21.15.45:0/3827040142 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:01.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.431+0000 7fec8fe00640 1 --2- 172.21.15.45:0/3827040142 >> v2:172.21.15.45:3300/0 conn(0x7fec900c5670 0x7fec900c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:01.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.431+0000 7fec8ea00640 1 -- 172.21.15.45:0/3827040142 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 822840079 0 0) 0x7fec9015d3b0 con 0x7fec9015bc80 2024-04-16T16:16:01.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.431+0000 7fec8ea00640 1 -- 172.21.15.45:0/3827040142 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec74003520 con 0x7fec9015bc80 2024-04-16T16:16:01.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.431+0000 7fec8fe00640 1 -- 172.21.15.45:0/3827040142 >> v1:172.21.15.103:6789/0 conn(0x7fec90145230 legacy=0x7fec9015cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:01.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.431+0000 7fec8fe00640 1 -- 172.21.15.45:0/3827040142 >> v1:172.21.15.45:6789/0 conn(0x7fec9015bc80 legacy=0x7fec900cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:01.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.431+0000 7fec8fe00640 1 -- 172.21.15.45:0/3827040142 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9015d970 con 0x7fec900c5670 2024-04-16T16:16:01.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.432+0000 7fec8fe00640 1 --2- 172.21.15.45:0/3827040142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec900c5670 0x7fec900c5aa0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fec9015e140 tx=0x7fec8802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48af0565125b8f59 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:01.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.432+0000 7fec8ea00640 1 -- 172.21.15.45:0/3827040142 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec88039070 con 0x7fec900c5670 2024-04-16T16:16:01.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.432+0000 7fec8ea00640 1 -- 172.21.15.45:0/3827040142 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec8802cc60 con 0x7fec900c5670 2024-04-16T16:16:01.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.432+0000 7fec8ea00640 1 -- 172.21.15.45:0/3827040142 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec88034660 con 0x7fec900c5670 2024-04-16T16:16:01.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.433+0000 7fec95000640 1 -- 172.21.15.45:0/3827040142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec900c5670 msgr2=0x7fec900c5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:01.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.433+0000 7fec95000640 1 --2- 172.21.15.45:0/3827040142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec900c5670 0x7fec900c5aa0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fec9015e140 tx=0x7fec8802beb0 comp rx=0 tx=0).stop 2024-04-16T16:16:01.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.433+0000 7fec95000640 1 -- 172.21.15.45:0/3827040142 shutdown_connections 2024-04-16T16:16:01.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.433+0000 7fec95000640 1 --2- 172.21.15.45:0/3827040142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec900c5670 0x7fec900c5aa0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:01.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.433+0000 7fec95000640 1 -- 172.21.15.45:0/3827040142 >> 172.21.15.45:0/3827040142 conn(0x7fec9006b930 msgr2=0x7fec9006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:01.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.433+0000 7fec95000640 1 -- 172.21.15.45:0/3827040142 shutdown_connections 2024-04-16T16:16:01.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.433+0000 7fec95000640 1 -- 172.21.15.45:0/3827040142 wait complete. 2024-04-16T16:16:01.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.435+0000 7fec95000640 1 Processor -- start 2024-04-16T16:16:01.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.435+0000 7fec95000640 1 -- start start 2024-04-16T16:16:01.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.436+0000 7fec95000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec900c5670 0x7fec9015b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:01.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.436+0000 7fec95000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec90145230 0x7fec90139760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:01.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.436+0000 7fec95000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec9015bc80 0x7fec9015f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:01.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.436+0000 7fec95000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fec9015e430 con 0x7fec9015bc80 2024-04-16T16:16:01.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.436+0000 7fec95000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fec900c5fe0 con 0x7fec900c5670 2024-04-16T16:16:01.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.436+0000 7fec95000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fec9015e5b0 con 0x7fec90145230 2024-04-16T16:16:01.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.436+0000 7fec8fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec9015bc80 0x7fec9015f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:01.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.436+0000 7fec8fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec9015bc80 0x7fec9015f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59090/0 (socket says 172.21.15.45:59090) 2024-04-16T16:16:01.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.437+0000 7fec8fe00640 1 -- 172.21.15.45:0/462043740 learned_addr learned my addr 172.21.15.45:0/462043740 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:01.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.437+0000 7fec87e00640 1 --2- 172.21.15.45:0/462043740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec90145230 0x7fec90139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:01.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.437+0000 7fec8f400640 1 --2- 172.21.15.45:0/462043740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec900c5670 0x7fec9015b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:01.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.437+0000 7fec8fe00640 1 -- 172.21.15.45:0/462043740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec90145230 msgr2=0x7fec90139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:01.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.437+0000 7fec8fe00640 1 --2- 172.21.15.45:0/462043740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec90145230 0x7fec90139760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:01.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.437+0000 7fec8fe00640 1 -- 172.21.15.45:0/462043740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec900c5670 msgr2=0x7fec9015b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:01.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.437+0000 7fec8fe00640 1 --2- 172.21.15.45:0/462043740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec900c5670 0x7fec9015b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:01.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.437+0000 7fec8fe00640 1 -- 172.21.15.45:0/462043740 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec900b4e20 con 0x7fec9015bc80 2024-04-16T16:16:01.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.437+0000 7fec87e00640 1 --2- 172.21.15.45:0/462043740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec90145230 0x7fec90139760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:01.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.437+0000 7fec8fe00640 1 --2- 172.21.15.45:0/462043740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec9015bc80 0x7fec9015f690 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fec88037040 tx=0x7fec88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.438+0000 7fec87400640 1 -- 172.21.15.45:0/462043740 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec88044020 con 0x7fec9015bc80 2024-04-16T16:16:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.438+0000 7fec87400640 1 -- 172.21.15.45:0/462043740 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec8803f470 con 0x7fec9015bc80 2024-04-16T16:16:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.438+0000 7fec87400640 1 -- 172.21.15.45:0/462043740 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec88039040 con 0x7fec9015bc80 2024-04-16T16:16:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.438+0000 7fec95000640 1 -- 172.21.15.45:0/462043740 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec900b4580 con 0x7fec9015bc80 2024-04-16T16:16:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.438+0000 7fec95000640 1 -- 172.21.15.45:0/462043740 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec900b4b40 con 0x7fec9015bc80 2024-04-16T16:16:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.439+0000 7fec95000640 1 -- 172.21.15.45:0/462043740 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec90075780 con 0x7fec9015bc80 2024-04-16T16:16:01.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.440+0000 7fec87400640 1 -- 172.21.15.45:0/462043740 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7fec88005d10 con 0x7fec9015bc80 2024-04-16T16:16:01.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.441+0000 7fec87400640 1 --2- 172.21.15.45:0/462043740 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fec6c06ca60 0x7fec6c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:01.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.441+0000 7fec87400640 1 -- 172.21.15.45:0/462043740 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7139+0+0 (secure 0 0 0) 0x7fec880aeff0 con 0x7fec9015bc80 2024-04-16T16:16:01.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.443+0000 7fec8f400640 1 --2- 172.21.15.45:0/462043740 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fec6c06ca60 0x7fec6c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:01.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.444+0000 7fec8f400640 1 --2- 172.21.15.45:0/462043740 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fec6c06ca60 0x7fec6c06ef20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fec9015f130 tx=0x7fec80009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:01.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.446+0000 7fec87400640 1 -- 172.21.15.45:0/462043740 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fec8804e030 con 0x7fec9015bc80 2024-04-16T16:16:01.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.676+0000 7fec95000640 1 -- 172.21.15.45:0/462043740 --> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7fec900b5950 con 0x7fec6c06ca60 2024-04-16T16:16:01.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.680+0000 7fec87400640 1 -- 172.21.15.45:0/462043740 <== mgr.4658 v2:172.21.15.103:6832/280591707 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+1838 (secure 0 0 0) 0x7fec900b5950 con 0x7fec6c06ca60 2024-04-16T16:16:01.679 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"fs1": {"client.5730": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "192.168.0.1"}}, "fs2": {"client.5928": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"fs1": {"client.5730": [[41, 26], [0, 0], [0, 53067004], [0, 49561584], [1, 0], [0, 1], [1, 1], [0, 1], [0, 0], [2, 3145728], [0, 0], [0, 0], [0, 26533502], [4262621859602, 2], [0, 1376716], [69762652109872, 36]]}, "fs2": {"client.5928": [[29, 25], [0, 0], [0, 21808644], [0, 43447615], [1, 0], [0, 5], [5, 5], [0, 5], [0, 0], [1, 1048576], [0, 0], [0, 0], [0, 21808644], [0, 1], [0, 1277875], [90953602998469, 34]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.5730": [], "client.5928": []}}} 2024-04-16T16:16:01.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.682+0000 7fec95000640 1 -- 172.21.15.45:0/462043740 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fec6c06ca60 msgr2=0x7fec6c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:01.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.682+0000 7fec95000640 1 --2- 172.21.15.45:0/462043740 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fec6c06ca60 0x7fec6c06ef20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fec9015f130 tx=0x7fec80009040 comp rx=0 tx=0).stop 2024-04-16T16:16:01.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.682+0000 7fec95000640 1 -- 172.21.15.45:0/462043740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec9015bc80 msgr2=0x7fec9015f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:01.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.682+0000 7fec95000640 1 --2- 172.21.15.45:0/462043740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec9015bc80 0x7fec9015f690 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fec88037040 tx=0x7fec88004a40 comp rx=0 tx=0).stop 2024-04-16T16:16:01.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.682+0000 7fec95000640 1 -- 172.21.15.45:0/462043740 shutdown_connections 2024-04-16T16:16:01.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.682+0000 7fec95000640 1 --2- 172.21.15.45:0/462043740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec9015bc80 0x7fec9015f690 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:01.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.682+0000 7fec95000640 1 --2- 172.21.15.45:0/462043740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec90145230 0x7fec90139760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:01.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.682+0000 7fec95000640 1 --2- 172.21.15.45:0/462043740 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fec6c06ca60 0x7fec6c06ef20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:01.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.682+0000 7fec95000640 1 --2- 172.21.15.45:0/462043740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec900c5670 0x7fec9015b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:01.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.682+0000 7fec95000640 1 -- 172.21.15.45:0/462043740 >> 172.21.15.45:0/462043740 conn(0x7fec9006b930 msgr2=0x7fec900cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:01.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.682+0000 7fec95000640 1 -- 172.21.15.45:0/462043740 shutdown_connections 2024-04-16T16:16:01.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.682+0000 7fec95000640 1 -- 172.21.15.45:0/462043740 wait complete. 2024-04-16T16:16:01.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:16:01.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.916+0000 7f1a88600640 1 Processor -- start 2024-04-16T16:16:01.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.916+0000 7f1a88600640 1 -- start start 2024-04-16T16:16:01.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.918+0000 7f1a88600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a800aed90 0x7f1a80072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:01.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.918+0000 7f1a88600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a80147c20 con 0x7f1a80147840 2024-04-16T16:16:01.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.918+0000 7f1a88600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a80160210 con 0x7f1a800728c0 2024-04-16T16:16:01.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.918+0000 7f1a88600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1a800afe80 con 0x7f1a800aed90 2024-04-16T16:16:01.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.918+0000 7f1a85c00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1a80147840 0x7f1a8015fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55146/0 (socket says 172.21.15.45:55146) 2024-04-16T16:16:01.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.918+0000 7f1a85c00640 1 -- 172.21.15.45:0/1135296521 learned_addr learned my addr 172.21.15.45:0/1135296521 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:01.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.918+0000 7f1a85200640 1 --2- 172.21.15.45:0/1135296521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a800aed90 0x7f1a80072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:01.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.918+0000 7f1a7fe00640 1 -- 172.21.15.45:0/1135296521 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1617833100 0 0) 0x7f1a80147c20 con 0x7f1a80147840 2024-04-16T16:16:01.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.918+0000 7f1a7fe00640 1 -- 172.21.15.45:0/1135296521 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a68003520 con 0x7f1a80147840 2024-04-16T16:16:01.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.918+0000 7f1a7fe00640 1 -- 172.21.15.45:0/1135296521 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 436346482 0 0) 0x7f1a68003520 con 0x7f1a80147840 2024-04-16T16:16:01.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.918+0000 7f1a7fe00640 1 -- 172.21.15.45:0/1135296521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a800aed90 msgr2=0x7f1a80072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:01.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.918+0000 7f1a7fe00640 1 --2- 172.21.15.45:0/1135296521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a800aed90 0x7f1a80072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:01.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.918+0000 7f1a7fe00640 1 -- 172.21.15.45:0/1135296521 >> v1:172.21.15.103:6789/0 conn(0x7f1a800728c0 legacy=0x7f1a8006e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:01.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.918+0000 7f1a7fe00640 1 -- 172.21.15.45:0/1135296521 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a800af270 con 0x7f1a80147840 2024-04-16T16:16:01.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a7fe00640 1 -- 172.21.15.45:0/1135296521 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f1a74001670 con 0x7f1a80147840 2024-04-16T16:16:01.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a7fe00640 1 -- 172.21.15.45:0/1135296521 >> v1:172.21.15.45:6789/0 conn(0x7f1a80147840 legacy=0x7f1a8015fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:01.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a7fe00640 1 --2- 172.21.15.45:0/1135296521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a680039e0 0x7f1a68003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:01.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a7fe00640 1 --2- 172.21.15.45:0/1135296521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a68004800 0x7f1a68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:01.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a7fe00640 1 --2- 172.21.15.45:0/1135296521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a800aed90 0x7f1a680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:01.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a7fe00640 1 -- 172.21.15.45:0/1135296521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a800afe80 con 0x7f1a680039e0 2024-04-16T16:16:01.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a7fe00640 1 -- 172.21.15.45:0/1135296521 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6800baa0 con 0x7f1a68004800 2024-04-16T16:16:01.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a7fe00640 1 -- 172.21.15.45:0/1135296521 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1a6800bca0 con 0x7f1a800aed90 2024-04-16T16:16:01.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a85c00640 1 --2- 172.21.15.45:0/1135296521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a680039e0 0x7f1a68003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:01.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a85200640 1 --2- 172.21.15.45:0/1135296521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a68004800 0x7f1a68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:01.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a85c00640 1 -- 172.21.15.45:0/1135296521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a800aed90 msgr2=0x7f1a680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:01.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a85c00640 1 --2- 172.21.15.45:0/1135296521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a800aed90 0x7f1a680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:01.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a85c00640 1 -- 172.21.15.45:0/1135296521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a68004800 msgr2=0x7f1a68004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:01.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a85c00640 1 --2- 172.21.15.45:0/1135296521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a68004800 0x7f1a68004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:01.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a85c00640 1 -- 172.21.15.45:0/1135296521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a68003520 con 0x7f1a680039e0 2024-04-16T16:16:01.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.919+0000 7f1a85200640 1 --2- 172.21.15.45:0/1135296521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a68004800 0x7f1a68004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:01.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.921+0000 7f1a85c00640 1 --2- 172.21.15.45:0/1135296521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a680039e0 0x7f1a68003e00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1a7400c0e0 tx=0x7f1a68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b279e3adf6db4a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:01.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.921+0000 7f1a7fe00640 1 -- 172.21.15.45:0/1135296521 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a74007c40 con 0x7f1a680039e0 2024-04-16T16:16:01.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.921+0000 7f1a7fe00640 1 -- 172.21.15.45:0/1135296521 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a74030070 con 0x7f1a680039e0 2024-04-16T16:16:01.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.921+0000 7f1a7fe00640 1 -- 172.21.15.45:0/1135296521 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a74005d30 con 0x7f1a680039e0 2024-04-16T16:16:01.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.922+0000 7f1a88600640 1 -- 172.21.15.45:0/1135296521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a680039e0 msgr2=0x7f1a68003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:01.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.922+0000 7f1a88600640 1 --2- 172.21.15.45:0/1135296521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a680039e0 0x7f1a68003e00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1a7400c0e0 tx=0x7f1a68005830 comp rx=0 tx=0).stop 2024-04-16T16:16:01.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.922+0000 7f1a86600640 1 -- 172.21.15.45:0/1135296521 reap_dead start 2024-04-16T16:16:01.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.922+0000 7f1a88600640 1 -- 172.21.15.45:0/1135296521 shutdown_connections 2024-04-16T16:16:01.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.922+0000 7f1a88600640 1 -- 172.21.15.45:0/1135296521 >> 172.21.15.45:0/1135296521 conn(0x7f1a800b2750 msgr2=0x7f1a800b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:01.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.922+0000 7f1a88600640 1 -- 172.21.15.45:0/1135296521 shutdown_connections 2024-04-16T16:16:01.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.922+0000 7f1a88600640 1 -- 172.21.15.45:0/1135296521 wait complete. 2024-04-16T16:16:01.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.924+0000 7f1a88600640 1 Processor -- start 2024-04-16T16:16:01.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.924+0000 7f1a88600640 1 -- start start 2024-04-16T16:16:01.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.925+0000 7f1a88600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a680039e0 0x7f1a80140260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:01.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.925+0000 7f1a88600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a80141990 0x7f1a8013ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:01.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.925+0000 7f1a88600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a8013fa70 0x7f1a8013b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:01.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.925+0000 7f1a88600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a800afb10 con 0x7f1a8013fa70 2024-04-16T16:16:01.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.925+0000 7f1a88600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a800af830 con 0x7f1a680039e0 2024-04-16T16:16:01.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.925+0000 7f1a88600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1a80072380 con 0x7f1a80141990 2024-04-16T16:16:01.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.925+0000 7f1a86600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a8013fa70 0x7f1a8013b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:01.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.925+0000 7f1a86600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a8013fa70 0x7f1a8013b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59108/0 (socket says 172.21.15.45:59108) 2024-04-16T16:16:01.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.925+0000 7f1a86600640 1 -- 172.21.15.45:0/3603639567 learned_addr learned my addr 172.21.15.45:0/3603639567 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:01.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.925+0000 7f1a85c00640 1 --2- 172.21.15.45:0/3603639567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a680039e0 0x7f1a80140260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:01.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.925+0000 7f1a85200640 1 --2- 172.21.15.45:0/3603639567 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a80141990 0x7f1a8013ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:01.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.925+0000 7f1a86600640 1 -- 172.21.15.45:0/3603639567 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a80141990 msgr2=0x7f1a8013ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:01.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.925+0000 7f1a86600640 1 --2- 172.21.15.45:0/3603639567 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a80141990 0x7f1a8013ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:01.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.926+0000 7f1a86600640 1 -- 172.21.15.45:0/3603639567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a680039e0 msgr2=0x7f1a80140260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:01.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.926+0000 7f1a86600640 1 --2- 172.21.15.45:0/3603639567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a680039e0 0x7f1a80140260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:01.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.926+0000 7f1a86600640 1 -- 172.21.15.45:0/3603639567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8013ee50 con 0x7f1a8013fa70 2024-04-16T16:16:01.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.926+0000 7f1a85c00640 1 --2- 172.21.15.45:0/3603639567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a680039e0 0x7f1a80140260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:01.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.926+0000 7f1a86600640 1 --2- 172.21.15.45:0/3603639567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a8013fa70 0x7f1a8013b950 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1a8006c960 tx=0x7f1a7800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:01.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.926+0000 7f1a7d600640 1 -- 172.21.15.45:0/3603639567 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a78017070 con 0x7f1a8013fa70 2024-04-16T16:16:01.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.926+0000 7f1a7d600640 1 -- 172.21.15.45:0/3603639567 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a78005d60 con 0x7f1a8013fa70 2024-04-16T16:16:01.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.926+0000 7f1a7d600640 1 -- 172.21.15.45:0/3603639567 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7800a850 con 0x7f1a8013fa70 2024-04-16T16:16:01.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.926+0000 7f1a88600640 1 -- 172.21.15.45:0/3603639567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a801408f0 con 0x7f1a8013fa70 2024-04-16T16:16:01.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.926+0000 7f1a88600640 1 -- 172.21.15.45:0/3603639567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a80140eb0 con 0x7f1a8013fa70 2024-04-16T16:16:01.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.927+0000 7f1a88600640 1 -- 172.21.15.45:0/3603639567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a80075780 con 0x7f1a8013fa70 2024-04-16T16:16:01.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.928+0000 7f1a7d600640 1 -- 172.21.15.45:0/3603639567 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7f1a7800a9f0 con 0x7f1a8013fa70 2024-04-16T16:16:01.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.929+0000 7f1a7d600640 1 --2- 172.21.15.45:0/3603639567 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1a6006ca90 0x7f1a6006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:01.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.930+0000 7f1a7d600640 1 -- 172.21.15.45:0/3603639567 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7197+0+0 (secure 0 0 0) 0x7f1a7808b310 con 0x7f1a8013fa70 2024-04-16T16:16:01.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.930+0000 7f1a85c00640 1 --2- 172.21.15.45:0/3603639567 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1a6006ca90 0x7f1a6006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:01.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.931+0000 7f1a85c00640 1 --2- 172.21.15.45:0/3603639567 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1a6006ca90 0x7f1a6006ef50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1a74034000 tx=0x7f1a74009030 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:01.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:01.935+0000 7f1a7d600640 1 -- 172.21.15.45:0/3603639567 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a78059080 con 0x7f1a8013fa70 2024-04-16T16:16:02.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.178+0000 7f1a88600640 1 -- 172.21.15.45:0/3603639567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a80070220 con 0x7f1a8013fa70 2024-04-16T16:16:02.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.180+0000 7f1a7d600640 1 -- 172.21.15.45:0/3603639567 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6491 (secure 0 0 0) 0x7f1a78058980 con 0x7f1a8013fa70 2024-04-16T16:16:02.180 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:02.180 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":40,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/1497040472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1497040472},{"type":"v1","addr":"172.21.15.45:6834","nonce":1497040472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":4907,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/93057991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":93057991},{"type":"v1","addr":"172.21.15.45:6835","nonce":93057991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:14.878986+0000","modified":"2024-04-16T16:15:44.272257+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5188":{"gid":5188,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.103:6836/2626302738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2626302738},{"type":"v1","addr":"172.21.15.103:6836","nonce":2626302738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"fs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5188,"qdb_cluster":[5188]},"id":3},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:35.289901+0000","modified":"2024-04-16T16:15:44.272258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"b","rank":0,"incarnation":35,"state":"up:active","state_seq":4,"addr":"172.21.15.103:6837/1576112884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1576112884},{"type":"v1","addr":"172.21.15.103:6837","nonce":1576112884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"fs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":4}]} 2024-04-16T16:16:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 40 2024-04-16T16:16:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.182+0000 7f1a88600640 1 -- 172.21.15.45:0/3603639567 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1a6006ca90 msgr2=0x7f1a6006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.182+0000 7f1a88600640 1 --2- 172.21.15.45:0/3603639567 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1a6006ca90 0x7f1a6006ef50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1a74034000 tx=0x7f1a74009030 comp rx=0 tx=0).stop 2024-04-16T16:16:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.182+0000 7f1a88600640 1 -- 172.21.15.45:0/3603639567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a8013fa70 msgr2=0x7f1a8013b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.182+0000 7f1a88600640 1 --2- 172.21.15.45:0/3603639567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a8013fa70 0x7f1a8013b950 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1a8006c960 tx=0x7f1a7800cb90 comp rx=0 tx=0).stop 2024-04-16T16:16:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.182+0000 7f1a88600640 1 -- 172.21.15.45:0/3603639567 shutdown_connections 2024-04-16T16:16:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.182+0000 7f1a88600640 1 --2- 172.21.15.45:0/3603639567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a8013fa70 0x7f1a8013b950 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:02.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.182+0000 7f1a88600640 1 --2- 172.21.15.45:0/3603639567 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a80141990 0x7f1a8013ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:02.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.182+0000 7f1a88600640 1 --2- 172.21.15.45:0/3603639567 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f1a6006ca90 0x7f1a6006ef50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:02.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.182+0000 7f1a88600640 1 --2- 172.21.15.45:0/3603639567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a680039e0 0x7f1a80140260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:02.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.182+0000 7f1a88600640 1 -- 172.21.15.45:0/3603639567 >> 172.21.15.45:0/3603639567 conn(0x7f1a800b2750 msgr2=0x7f1a8006f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:02.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.182+0000 7f1a88600640 1 -- 172.21.15.45:0/3603639567 shutdown_connections 2024-04-16T16:16:02.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.182+0000 7f1a88600640 1 -- 172.21.15.45:0/3603639567 wait complete. 2024-04-16T16:16:02.203 DEBUG:tasks.cephfs.test_mds_metrics:metrics={'version': 2, 'global_counters': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'counters': [], 'client_metadata': {'fs1': {'client.5730': {'hostname': 'smithi045', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '192.168.0.1'}}, 'fs2': {'client.5928': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '172.21.15.103'}}}, 'global_metrics': {'fs1': {'client.5730': [[41, 26], [0, 0], [0, 53067004], [0, 49561584], [1, 0], [0, 1], [1, 1], [0, 1], [0, 0], [2, 3145728], [0, 0], [0, 0], [0, 26533502], [4262621859602, 2], [0, 1376716], [69762652109872, 36]]}, 'fs2': {'client.5928': [[29, 25], [0, 0], [0, 21808644], [0, 43447615], [1, 0], [0, 5], [5, 5], [0, 5], [0, 0], [1, 1048576], [0, 0], [0, 0], [0, 21808644], [0, 1], [0, 1277875], [90953602998469, 34]]}}, 'metrics': {'delayed_ranks': [], 'mds.0': {'client.5730': [], 'client.5928': []}}} 2024-04-16T16:16:02.203 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable stats 2024-04-16T16:16:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.421+0000 7efec9400640 1 Processor -- start 2024-04-16T16:16:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.421+0000 7efec9400640 1 -- start start 2024-04-16T16:16:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.423+0000 7efec9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efec40cf7d0 0x7efec40cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.423+0000 7efec9400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efec415f220 0x7efec415f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.423+0000 7efec9400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efec415bc00 con 0x7efec406f030 2024-04-16T16:16:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.423+0000 7efec9400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7efec415bed0 con 0x7efec415f220 2024-04-16T16:16:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.423+0000 7efec9400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efec415c1b0 con 0x7efec40cf7d0 2024-04-16T16:16:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.423+0000 7efec3e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7efec406f030 0x7efec406f410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55156/0 (socket says 172.21.15.45:55156) 2024-04-16T16:16:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.423+0000 7efec3e00640 1 -- 172.21.15.45:0/1192144567 learned_addr learned my addr 172.21.15.45:0/1192144567 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.424+0000 7efec3400640 1 --2- 172.21.15.45:0/1192144567 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efec40cf7d0 0x7efec40cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.424+0000 7efec8a00640 1 --2- 172.21.15.45:0/1192144567 >> v2:172.21.15.103:3300/0 conn(0x7efec415f220 0x7efec415f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.424+0000 7efec2a00640 1 -- 172.21.15.45:0/1192144567 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 591100424 0 0) 0x7efec415bc00 con 0x7efec406f030 2024-04-16T16:16:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.424+0000 7efec2a00640 1 -- 172.21.15.45:0/1192144567 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efea8003520 con 0x7efec406f030 2024-04-16T16:16:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.424+0000 7efec2a00640 1 -- 172.21.15.45:0/1192144567 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3536194070 0 0) 0x7efea8003520 con 0x7efec406f030 2024-04-16T16:16:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.424+0000 7efec2a00640 1 -- 172.21.15.45:0/1192144567 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efec40cf7d0 msgr2=0x7efec40cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.424+0000 7efec2a00640 1 --2- 172.21.15.45:0/1192144567 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efec40cf7d0 0x7efec40cfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.424+0000 7efec2a00640 1 -- 172.21.15.45:0/1192144567 >> v2:172.21.15.103:3300/0 conn(0x7efec415f220 msgr2=0x7efec415f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.424+0000 7efec2a00640 1 --2- 172.21.15.45:0/1192144567 >> v2:172.21.15.103:3300/0 conn(0x7efec415f220 0x7efec415f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.424+0000 7efec2a00640 1 -- 172.21.15.45:0/1192144567 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec415c490 con 0x7efec406f030 2024-04-16T16:16:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.424+0000 7efec2a00640 1 -- 172.21.15.45:0/1192144567 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7efeb4001670 con 0x7efec406f030 2024-04-16T16:16:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.425+0000 7efec2a00640 1 -- 172.21.15.45:0/1192144567 >> v1:172.21.15.45:6789/0 conn(0x7efec406f030 legacy=0x7efec406f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.425+0000 7efec2a00640 1 --2- 172.21.15.45:0/1192144567 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efec40cf7d0 0x7efea80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.425+0000 7efec2a00640 1 --2- 172.21.15.45:0/1192144567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea8003fb0 0x7efea8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.425+0000 7efec2a00640 1 --2- 172.21.15.45:0/1192144567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efea8004d40 0x7efea80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.425+0000 7efec2a00640 1 -- 172.21.15.45:0/1192144567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efec415bed0 con 0x7efea8004d40 2024-04-16T16:16:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.425+0000 7efec2a00640 1 -- 172.21.15.45:0/1192144567 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efec415c1b0 con 0x7efea8003fb0 2024-04-16T16:16:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.425+0000 7efec2a00640 1 -- 172.21.15.45:0/1192144567 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efea800fe00 con 0x7efec40cf7d0 2024-04-16T16:16:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.425+0000 7efec8a00640 1 --2- 172.21.15.45:0/1192144567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efea8004d40 0x7efea80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.426+0000 7efec3e00640 1 --2- 172.21.15.45:0/1192144567 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efec40cf7d0 0x7efea80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.426+0000 7efec8a00640 1 -- 172.21.15.45:0/1192144567 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efec40cf7d0 msgr2=0x7efea80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.426+0000 7efec8a00640 1 --2- 172.21.15.45:0/1192144567 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efec40cf7d0 0x7efea80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.426+0000 7efec3400640 1 --2- 172.21.15.45:0/1192144567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea8003fb0 0x7efea8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.426+0000 7efec8a00640 1 -- 172.21.15.45:0/1192144567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea8003fb0 msgr2=0x7efea8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.426+0000 7efec8a00640 1 --2- 172.21.15.45:0/1192144567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea8003fb0 0x7efea8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.426+0000 7efec8a00640 1 -- 172.21.15.45:0/1192144567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea8003520 con 0x7efea8004d40 2024-04-16T16:16:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.426+0000 7efec3e00640 1 --2- 172.21.15.45:0/1192144567 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efec40cf7d0 0x7efea80039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.426+0000 7efec3400640 1 --2- 172.21.15.45:0/1192144567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea8003fb0 0x7efea8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.427+0000 7efec8a00640 1 --2- 172.21.15.45:0/1192144567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efea8004d40 0x7efea80051c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7efebc009fc0 tx=0x7efea8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4bf749a98057a57 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.427+0000 7efec2a00640 1 -- 172.21.15.45:0/1192144567 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc039070 con 0x7efea8004d40 2024-04-16T16:16:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.427+0000 7efec2a00640 1 -- 172.21.15.45:0/1192144567 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efebc02cc40 con 0x7efea8004d40 2024-04-16T16:16:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.427+0000 7efec2a00640 1 -- 172.21.15.45:0/1192144567 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc009c10 con 0x7efea8004d40 2024-04-16T16:16:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.428+0000 7efec9400640 1 -- 172.21.15.45:0/1192144567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efea8004d40 msgr2=0x7efea80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.428+0000 7efec9400640 1 --2- 172.21.15.45:0/1192144567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efea8004d40 0x7efea80051c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7efebc009fc0 tx=0x7efea8005830 comp rx=0 tx=0).stop 2024-04-16T16:16:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.428+0000 7efec8a00640 1 -- 172.21.15.45:0/1192144567 reap_dead start 2024-04-16T16:16:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.428+0000 7efec9400640 1 -- 172.21.15.45:0/1192144567 shutdown_connections 2024-04-16T16:16:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.428+0000 7efec9400640 1 -- 172.21.15.45:0/1192144567 >> 172.21.15.45:0/1192144567 conn(0x7efec4067e00 msgr2=0x7efec406b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.428+0000 7efec9400640 1 -- 172.21.15.45:0/1192144567 shutdown_connections 2024-04-16T16:16:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.428+0000 7efec9400640 1 -- 172.21.15.45:0/1192144567 wait complete. 2024-04-16T16:16:02.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.430+0000 7efec9400640 1 Processor -- start 2024-04-16T16:16:02.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.430+0000 7efec9400640 1 -- start start 2024-04-16T16:16:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.431+0000 7efec9400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea8004d40 0x7efec4140090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.431+0000 7efec9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efec414e2a0 0x7efec4140720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.431+0000 7efec9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efec4158b80 0x7efec4156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.431+0000 7efec9400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efec40b3fe0 con 0x7efec414e2a0 2024-04-16T16:16:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.431+0000 7efec9400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efec40b3e60 con 0x7efea8004d40 2024-04-16T16:16:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.431+0000 7efec9400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efec4155ea0 con 0x7efec4158b80 2024-04-16T16:16:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.431+0000 7efec3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efec414e2a0 0x7efec4140720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.432+0000 7efec3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efec414e2a0 0x7efec4140720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59128/0 (socket says 172.21.15.45:59128) 2024-04-16T16:16:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.432+0000 7efec3400640 1 -- 172.21.15.45:0/825229460 learned_addr learned my addr 172.21.15.45:0/825229460 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.432+0000 7efec3e00640 1 --2- 172.21.15.45:0/825229460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea8004d40 0x7efec4140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.432+0000 7efec8a00640 1 --2- 172.21.15.45:0/825229460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efec4158b80 0x7efec4156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.432+0000 7efec3400640 1 -- 172.21.15.45:0/825229460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efec4158b80 msgr2=0x7efec4156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.432+0000 7efec3400640 1 --2- 172.21.15.45:0/825229460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efec4158b80 0x7efec4156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.432+0000 7efec3400640 1 -- 172.21.15.45:0/825229460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea8004d40 msgr2=0x7efec4140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.432+0000 7efec3400640 1 --2- 172.21.15.45:0/825229460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea8004d40 0x7efec4140090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.432+0000 7efec3400640 1 -- 172.21.15.45:0/825229460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec4159e30 con 0x7efec414e2a0 2024-04-16T16:16:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.432+0000 7efec3400640 1 --2- 172.21.15.45:0/825229460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efec414e2a0 0x7efec4140720 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7efeb0010960 tx=0x7efeb0010e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.433+0000 7efebbe00640 1 -- 172.21.15.45:0/825229460 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb000f040 con 0x7efec414e2a0 2024-04-16T16:16:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.433+0000 7efebbe00640 1 -- 172.21.15.45:0/825229460 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb0004220 con 0x7efec414e2a0 2024-04-16T16:16:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.433+0000 7efebbe00640 1 -- 172.21.15.45:0/825229460 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb0005990 con 0x7efec414e2a0 2024-04-16T16:16:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.433+0000 7efec9400640 1 -- 172.21.15.45:0/825229460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec415a110 con 0x7efec414e2a0 2024-04-16T16:16:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.433+0000 7efec9400640 1 -- 172.21.15.45:0/825229460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec415a6d0 con 0x7efec414e2a0 2024-04-16T16:16:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.434+0000 7efec9400640 1 -- 172.21.15.45:0/825229460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efec4075780 con 0x7efec414e2a0 2024-04-16T16:16:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.435+0000 7efebbe00640 1 -- 172.21.15.45:0/825229460 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) ==== 91151+0+0 (secure 0 0 0) 0x7efeb0005b30 con 0x7efec414e2a0 2024-04-16T16:16:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.436+0000 7efebbe00640 1 --2- 172.21.15.45:0/825229460 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7efea406ca60 0x7efea406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.436+0000 7efebbe00640 1 -- 172.21.15.45:0/825229460 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7197+0+0 (secure 0 0 0) 0x7efeb008c790 con 0x7efec414e2a0 2024-04-16T16:16:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.437+0000 7efec3e00640 1 --2- 172.21.15.45:0/825229460 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7efea406ca60 0x7efea406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:02.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.438+0000 7efec3e00640 1 --2- 172.21.15.45:0/825229460 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7efea406ca60 0x7efea406ef20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efeb4001200 tx=0x7efeb4009040 comp rx=0 tx=0).ready entity=mgr.4658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:02.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.441+0000 7efebbe00640 1 -- 172.21.15.45:0/825229460 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efeb0018070 con 0x7efec414e2a0 2024-04-16T16:16:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.700+0000 7efec9400640 1 -- 172.21.15.45:0/825229460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "stats"} v 0) -- 0x7efec40b0020 con 0x7efec414e2a0 2024-04-16T16:16:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.734+0000 7efebbe00640 1 -- 172.21.15.45:0/825229460 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "stats"}]=0 v11) ==== 85+0+0 (secure 0 0 0) 0x7efeb005a4d0 con 0x7efec414e2a0 2024-04-16T16:16:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.738+0000 7efec9400640 1 -- 172.21.15.45:0/825229460 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7efea406ca60 msgr2=0x7efea406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.738+0000 7efec9400640 1 --2- 172.21.15.45:0/825229460 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7efea406ca60 0x7efea406ef20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efeb4001200 tx=0x7efeb4009040 comp rx=0 tx=0).stop 2024-04-16T16:16:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.738+0000 7efec9400640 1 -- 172.21.15.45:0/825229460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efec414e2a0 msgr2=0x7efec4140720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.738+0000 7efec9400640 1 --2- 172.21.15.45:0/825229460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efec414e2a0 0x7efec4140720 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7efeb0010960 tx=0x7efeb0010e20 comp rx=0 tx=0).stop 2024-04-16T16:16:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.739+0000 7efec9400640 1 -- 172.21.15.45:0/825229460 shutdown_connections 2024-04-16T16:16:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.739+0000 7efec9400640 1 --2- 172.21.15.45:0/825229460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efec4158b80 0x7efec4156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.739+0000 7efec9400640 1 --2- 172.21.15.45:0/825229460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efec414e2a0 0x7efec4140720 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.739+0000 7efec9400640 1 --2- 172.21.15.45:0/825229460 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7efea406ca60 0x7efea406ef20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.739+0000 7efec9400640 1 --2- 172.21.15.45:0/825229460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea8004d40 0x7efec4140090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.739+0000 7efec9400640 1 -- 172.21.15.45:0/825229460 >> 172.21.15.45:0/825229460 conn(0x7efec4067e00 msgr2=0x7efec415ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:02.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.741+0000 7efec9400640 1 -- 172.21.15.45:0/825229460 shutdown_connections 2024-04-16T16:16:02.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:02.742+0000 7efec9400640 1 -- 172.21.15.45:0/825229460 wait complete. 2024-04-16T16:16:02.763 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-16T16:16:02.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:16:02.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:16:02.958 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-16T16:16:02.959 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:16:02.959 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:16:02.959 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:16:03.010 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:16:03.010 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:16:03.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:16:03.301 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:03.301+0000 7f152b60f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:16:03.309 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:03.310+0000 7f9fd5e39040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:16:03.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.313+0000 7f709b600640 1 Processor -- start 2024-04-16T16:16:03.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.313+0000 7f709b600640 1 -- start start 2024-04-16T16:16:03.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.314+0000 7f709b600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f709406b930 0x7f709415fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:03.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.315+0000 7f709b600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7094160cd0 0x7f7094161120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:03.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.315+0000 7f709b600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70941607f0 con 0x7f70940b1fe0 2024-04-16T16:16:03.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.315+0000 7f709b600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f70940b7290 con 0x7f709406b930 2024-04-16T16:16:03.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.315+0000 7f709b600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f70940b3060 con 0x7f7094160cd0 2024-04-16T16:16:03.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.315+0000 7f7098c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f709406b930 0x7f709415fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:03.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.315+0000 7f7098c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f709406b930 0x7f709415fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51092/0 (socket says 172.21.15.45:51092) 2024-04-16T16:16:03.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.315+0000 7f7098c00640 1 -- 172.21.15.45:0/2176172490 learned_addr learned my addr 172.21.15.45:0/2176172490 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:03.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.315+0000 7f7098c00640 1 -- 172.21.15.45:0/2176172490 >> v2:172.21.15.103:3300/0 conn(0x7f7094160cd0 msgr2=0x7f7094161120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:03.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.315+0000 7f7098c00640 1 --2- 172.21.15.45:0/2176172490 >> v2:172.21.15.103:3300/0 conn(0x7f7094160cd0 0x7f7094161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.315+0000 7f7098c00640 1 -- 172.21.15.45:0/2176172490 >> v1:172.21.15.103:6789/0 conn(0x7f70940b1fe0 legacy=0x7f709406bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:03.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.315+0000 7f7098c00640 1 -- 172.21.15.45:0/2176172490 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70940b27c0 con 0x7f709406b930 2024-04-16T16:16:03.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.316+0000 7f7098c00640 1 --2- 172.21.15.45:0/2176172490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709406b930 0x7f709415fe20 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f70941609c0 tx=0x7f708802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483b6f5fcb2d60cb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:03.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.317+0000 7f7093400640 1 -- 172.21.15.45:0/2176172490 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088035020 con 0x7f709406b930 2024-04-16T16:16:03.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.317+0000 7f7093400640 1 -- 172.21.15.45:0/2176172490 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7088031640 con 0x7f709406b930 2024-04-16T16:16:03.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.317+0000 7f7093400640 1 -- 172.21.15.45:0/2176172490 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708802d040 con 0x7f709406b930 2024-04-16T16:16:03.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.317+0000 7f709b600640 1 -- 172.21.15.45:0/2176172490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709406b930 msgr2=0x7f709415fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:03.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.317+0000 7f709b600640 1 --2- 172.21.15.45:0/2176172490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709406b930 0x7f709415fe20 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f70941609c0 tx=0x7f708802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.317+0000 7f709b600640 1 -- 172.21.15.45:0/2176172490 shutdown_connections 2024-04-16T16:16:03.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.317+0000 7f709b600640 1 --2- 172.21.15.45:0/2176172490 >> v2:172.21.15.103:3300/0 conn(0x7f7094160cd0 0x7f7094161120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.317+0000 7f709b600640 1 --2- 172.21.15.45:0/2176172490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709406b930 0x7f709415fe20 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.317+0000 7f709b600640 1 -- 172.21.15.45:0/2176172490 >> 172.21.15.45:0/2176172490 conn(0x7f7094073810 msgr2=0x7f709409aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:03.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.318+0000 7f709b600640 1 -- 172.21.15.45:0/2176172490 shutdown_connections 2024-04-16T16:16:03.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.318+0000 7f709b600640 1 -- 172.21.15.45:0/2176172490 wait complete. 2024-04-16T16:16:03.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.319+0000 7f709b600640 1 Processor -- start 2024-04-16T16:16:03.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.319+0000 7f709b600640 1 -- start start 2024-04-16T16:16:03.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.320+0000 7f709b600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709406b930 0x7f70941e9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:03.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.320+0000 7f709b600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70940b1fe0 0x7f70941ea740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:03.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.320+0000 7f709b600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7094160cd0 0x7f709415d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:03.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.320+0000 7f709b600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f70940b2470 con 0x7f709406b930 2024-04-16T16:16:03.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.320+0000 7f709b600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f70940b2d80 con 0x7f7094160cd0 2024-04-16T16:16:03.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.320+0000 7f709b600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f709415ef20 con 0x7f70940b1fe0 2024-04-16T16:16:03.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.321+0000 7f7098c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709406b930 0x7f70941e9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:03.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.321+0000 7f7098c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709406b930 0x7f70941e9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51104/0 (socket says 172.21.15.45:51104) 2024-04-16T16:16:03.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.321+0000 7f7098c00640 1 -- 172.21.15.45:0/2414497241 learned_addr learned my addr 172.21.15.45:0/2414497241 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:03.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.321+0000 7f7099600640 1 --2- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7094160cd0 0x7f709415d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:03.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.321+0000 7f7093e00640 1 --2- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70940b1fe0 0x7f70941ea740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:03.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.321+0000 7f7098c00640 1 -- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70940b1fe0 msgr2=0x7f70941ea740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:03.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.321+0000 7f7098c00640 1 --2- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70940b1fe0 0x7f70941ea740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.321+0000 7f7098c00640 1 -- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7094160cd0 msgr2=0x7f709415d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:03.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.321+0000 7f7098c00640 1 --2- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7094160cd0 0x7f709415d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.321+0000 7f7098c00640 1 -- 172.21.15.45:0/2414497241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70940b6980 con 0x7f709406b930 2024-04-16T16:16:03.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.321+0000 7f7098c00640 1 --2- 172.21.15.45:0/2414497241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709406b930 0x7f70941e9530 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7088000c00 tx=0x7f70880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:03.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.322+0000 7f7090c00640 1 -- 172.21.15.45:0/2414497241 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708803ac60 con 0x7f709406b930 2024-04-16T16:16:03.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.322+0000 7f7090c00640 1 -- 172.21.15.45:0/2414497241 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708803ae20 con 0x7f709406b930 2024-04-16T16:16:03.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.322+0000 7f709b600640 1 -- 172.21.15.45:0/2414497241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70941e8770 con 0x7f709406b930 2024-04-16T16:16:03.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.322+0000 7f7090c00640 1 -- 172.21.15.45:0/2414497241 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088044630 con 0x7f709406b930 2024-04-16T16:16:03.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.322+0000 7f709b600640 1 -- 172.21.15.45:0/2414497241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70941e8d30 con 0x7f709406b930 2024-04-16T16:16:03.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.323+0000 7f709b600640 1 -- 172.21.15.45:0/2414497241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7094075780 con 0x7f709406b930 2024-04-16T16:16:03.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.324+0000 7f7090c00640 1 -- 172.21.15.45:0/2414497241 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) ==== 91142+0+0 (secure 0 0 0) 0x7f708804b060 con 0x7f709406b930 2024-04-16T16:16:03.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.325+0000 7f7090c00640 1 --2- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f707406ca10 0x7f707406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:03.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.325+0000 7f7090c00640 1 -- 172.21.15.45:0/2414497241 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7197+0+0 (secure 0 0 0) 0x7f70880ae170 con 0x7f709406b930 2024-04-16T16:16:03.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.325+0000 7f7093e00640 1 -- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f707406ca10 msgr2=0x7f707406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/280591707 2024-04-16T16:16:03.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.325+0000 7f7093e00640 1 --2- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f707406ca10 0x7f707406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:16:03.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.330+0000 7f7090c00640 1 -- 172.21.15.45:0/2414497241 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f708803f070 con 0x7f709406b930 2024-04-16T16:16:03.357 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:03.357+0000 7f152b60f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:16:03.369 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:03.370+0000 7f9fd5e39040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:16:03.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.526+0000 7f7093e00640 1 -- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f707406ca10 msgr2=0x7f707406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/280591707 2024-04-16T16:16:03.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.526+0000 7f7093e00640 1 --2- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f707406ca10 0x7f707406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:16:03.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.540+0000 7f709b600640 1 -- 172.21.15.45:0/2414497241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70940b0540 con 0x7f709406b930 2024-04-16T16:16:03.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.543+0000 7f7090c00640 1 -- 172.21.15.45:0/2414497241 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+21413 (secure 0 0 0) 0x7f708807bd10 con 0x7f709406b930 2024-04-16T16:16:03.543 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:03.543 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":56,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:16:02.721562+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"fs1_metadata","create_time":"2024-04-16T16:15:13.250782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"fs1_data","create_time":"2024-04-16T16:15:13.856986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"fs2_metadata","create_time":"2024-04-16T16:15:33.215042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"fs2_data","create_time":"2024-04-16T16:15:34.271426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.2c","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6833/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6832/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.45:6833/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6834/88032513":"2024-04-17T16:15:41.221894+0000","172.21.15.45:6835/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6835/88032513":"2024-04-17T16:15:41.221894+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:16:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.545+0000 7f709b600640 1 -- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f707406ca10 msgr2=0x7f707406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:16:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.545+0000 7f709b600640 1 --2- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f707406ca10 0x7f707406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.545+0000 7f709b600640 1 -- 172.21.15.45:0/2414497241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709406b930 msgr2=0x7f70941e9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.545+0000 7f709b600640 1 --2- 172.21.15.45:0/2414497241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709406b930 0x7f70941e9530 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7088000c00 tx=0x7f70880046c0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.546+0000 7f709b600640 1 -- 172.21.15.45:0/2414497241 shutdown_connections 2024-04-16T16:16:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.546+0000 7f709b600640 1 --2- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f707406ca10 0x7f707406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.546+0000 7f709b600640 1 --2- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7094160cd0 0x7f709415d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.546+0000 7f709b600640 1 --2- 172.21.15.45:0/2414497241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70940b1fe0 0x7f70941ea740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.546+0000 7f709b600640 1 --2- 172.21.15.45:0/2414497241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709406b930 0x7f70941e9530 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.546+0000 7f709b600640 1 -- 172.21.15.45:0/2414497241 >> 172.21.15.45:0/2414497241 conn(0x7f7094073810 msgr2=0x7f709406ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.546+0000 7f709b600640 1 -- 172.21.15.45:0/2414497241 shutdown_connections 2024-04-16T16:16:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.546+0000 7f709b600640 1 -- 172.21.15.45:0/2414497241 wait complete. 2024-04-16T16:16:03.563 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'fs1_metadata', 'create_time': '2024-04-16T16:15:13.250782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'fs1'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'fs1_data', 'create_time': '2024-04-16T16:15:13.856986+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'fs1'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'fs2_metadata', 'create_time': '2024-04-16T16:15:33.215042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'fs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'fs2_data', 'create_time': '2024-04-16T16:15:34.271426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'fs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:16:03.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:16:03.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.764+0000 7fe16fe00640 1 Processor -- start 2024-04-16T16:16:03.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.765+0000 7fe16fe00640 1 -- start start 2024-04-16T16:16:03.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.766+0000 7fe16fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe17015c560 0x7fe17006f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:03.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.767+0000 7fe16fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe1700ce570 0x7fe17015f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:03.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.767+0000 7fe16fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17006fc60 0x7fe1700cd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:03.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.767+0000 7fe16fe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe17015f0b0 con 0x7fe17015c560 2024-04-16T16:16:03.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.767+0000 7fe16fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe17015edc0 con 0x7fe1700ce570 2024-04-16T16:16:03.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.767+0000 7fe16fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe1700cf2a0 con 0x7fe17006fc60 2024-04-16T16:16:03.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.767+0000 7fe16ea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe17015c560 0x7fe17006f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.767+0000 7fe16ea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe17015c560 0x7fe17006f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51106/0 (socket says 172.21.15.45:51106) 2024-04-16T16:16:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.767+0000 7fe16ea00640 1 -- 172.21.15.45:0/2498552439 learned_addr learned my addr 172.21.15.45:0/2498552439 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.767+0000 7fe16e000640 1 --2- 172.21.15.45:0/2498552439 >> v2:172.21.15.103:3300/0 conn(0x7fe1700ce570 0x7fe17015f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.767+0000 7fe16ea00640 1 -- 172.21.15.45:0/2498552439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17006fc60 msgr2=0x7fe1700cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.767+0000 7fe16ea00640 1 --2- 172.21.15.45:0/2498552439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17006fc60 0x7fe1700cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.767+0000 7fe16ea00640 1 -- 172.21.15.45:0/2498552439 >> v2:172.21.15.103:3300/0 conn(0x7fe1700ce570 msgr2=0x7fe17015f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.767+0000 7fe16ea00640 1 --2- 172.21.15.45:0/2498552439 >> v2:172.21.15.103:3300/0 conn(0x7fe1700ce570 0x7fe17015f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.767+0000 7fe16ea00640 1 -- 172.21.15.45:0/2498552439 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1700cea00 con 0x7fe17015c560 2024-04-16T16:16:03.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.769+0000 7fe16ea00640 1 --2- 172.21.15.45:0/2498552439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe17015c560 0x7fe17006f720 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe1700b1ba0 tx=0x7fe16002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50423dff191cc36 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:03.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.769+0000 7fe16d600640 1 -- 172.21.15.45:0/2498552439 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe160034070 con 0x7fe17015c560 2024-04-16T16:16:03.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.769+0000 7fe16d600640 1 -- 172.21.15.45:0/2498552439 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe160030850 con 0x7fe17015c560 2024-04-16T16:16:03.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.769+0000 7fe16d600640 1 -- 172.21.15.45:0/2498552439 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe160030c20 con 0x7fe17015c560 2024-04-16T16:16:03.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.770+0000 7fe16fe00640 1 -- 172.21.15.45:0/2498552439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe17015c560 msgr2=0x7fe17006f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:03.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.770+0000 7fe16fe00640 1 --2- 172.21.15.45:0/2498552439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe17015c560 0x7fe17006f720 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe1700b1ba0 tx=0x7fe16002b990 comp rx=0 tx=0).stop 2024-04-16T16:16:03.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.770+0000 7fe16fe00640 1 -- 172.21.15.45:0/2498552439 shutdown_connections 2024-04-16T16:16:03.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.770+0000 7fe16fe00640 1 --2- 172.21.15.45:0/2498552439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17006fc60 0x7fe1700cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.770+0000 7fe16fe00640 1 --2- 172.21.15.45:0/2498552439 >> v2:172.21.15.103:3300/0 conn(0x7fe1700ce570 0x7fe17015f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.770+0000 7fe16fe00640 1 --2- 172.21.15.45:0/2498552439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe17015c560 0x7fe17006f720 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.770+0000 7fe16fe00640 1 -- 172.21.15.45:0/2498552439 >> 172.21.15.45:0/2498552439 conn(0x7fe170067e00 msgr2=0x7fe17006b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:03.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.770+0000 7fe16fe00640 1 -- 172.21.15.45:0/2498552439 shutdown_connections 2024-04-16T16:16:03.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.770+0000 7fe16fe00640 1 -- 172.21.15.45:0/2498552439 wait complete. 2024-04-16T16:16:03.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.773+0000 7fe16fe00640 1 Processor -- start 2024-04-16T16:16:03.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.773+0000 7fe16fe00640 1 -- start start 2024-04-16T16:16:03.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.774+0000 7fe16fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17015c560 0x7fe170153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.774+0000 7fe16fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe170151be0 0x7fe170152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.774+0000 7fe16fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe170146f20 0x7fe170147360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.774+0000 7fe16fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe17015fa80 con 0x7fe170151be0 2024-04-16T16:16:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.774+0000 7fe16fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe1700cefc0 con 0x7fe170146f20 2024-04-16T16:16:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.774+0000 7fe16fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe17015fc00 con 0x7fe17015c560 2024-04-16T16:16:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.775+0000 7fe16e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe170151be0 0x7fe170152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.775+0000 7fe16e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe170151be0 0x7fe170152060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51120/0 (socket says 172.21.15.45:51120) 2024-04-16T16:16:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.775+0000 7fe16e000640 1 -- 172.21.15.45:0/2487106407 learned_addr learned my addr 172.21.15.45:0/2487106407 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:03.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.775+0000 7fe16ea00640 1 --2- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17015c560 0x7fe170153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:03.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.775+0000 7fe16f400640 1 --2- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe170146f20 0x7fe170147360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:03.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.775+0000 7fe16e000640 1 -- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17015c560 msgr2=0x7fe170153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:03.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.775+0000 7fe16e000640 1 --2- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17015c560 0x7fe170153c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.775+0000 7fe16e000640 1 -- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe170146f20 msgr2=0x7fe170147360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:03.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.775+0000 7fe16e000640 1 --2- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe170146f20 0x7fe170147360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.775+0000 7fe16e000640 1 -- 172.21.15.45:0/2487106407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17014d9b0 con 0x7fe170151be0 2024-04-16T16:16:03.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.775+0000 7fe16f400640 1 --2- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe170146f20 0x7fe170147360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:03.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.776+0000 7fe16e000640 1 --2- 172.21.15.45:0/2487106407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe170151be0 0x7fe170152060 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe158007f20 tx=0x7fe15800e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:03.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.776+0000 7fe167400640 1 -- 172.21.15.45:0/2487106407 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe158014620 con 0x7fe170151be0 2024-04-16T16:16:03.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.776+0000 7fe167400640 1 -- 172.21.15.45:0/2487106407 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe158019070 con 0x7fe170151be0 2024-04-16T16:16:03.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.776+0000 7fe167400640 1 -- 172.21.15.45:0/2487106407 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe158014a00 con 0x7fe170151be0 2024-04-16T16:16:03.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.776+0000 7fe16fe00640 1 -- 172.21.15.45:0/2487106407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1700b44d0 con 0x7fe170151be0 2024-04-16T16:16:03.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.776+0000 7fe16fe00640 1 -- 172.21.15.45:0/2487106407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1700b4a30 con 0x7fe170151be0 2024-04-16T16:16:03.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.777+0000 7fe16fe00640 1 -- 172.21.15.45:0/2487106407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe170075780 con 0x7fe170151be0 2024-04-16T16:16:03.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.778+0000 7fe167400640 1 -- 172.21.15.45:0/2487106407 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) ==== 91142+0+0 (secure 0 0 0) 0x7fe158014ba0 con 0x7fe170151be0 2024-04-16T16:16:03.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.779+0000 7fe167400640 1 --2- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe14c06ca10 0x7fe14c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:03.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.779+0000 7fe167400640 1 -- 172.21.15.45:0/2487106407 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7197+0+0 (secure 0 0 0) 0x7fe15808cb40 con 0x7fe170151be0 2024-04-16T16:16:03.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.779+0000 7fe16ea00640 1 -- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe14c06ca10 msgr2=0x7fe14c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/280591707 2024-04-16T16:16:03.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.779+0000 7fe16ea00640 1 --2- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe14c06ca10 0x7fe14c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:16:03.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.783+0000 7fe167400640 1 -- 172.21.15.45:0/2487106407 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe1700b44d0 con 0x7fe170151be0 2024-04-16T16:16:03.805 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:03.805+0000 7f152b60f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:16:03.856 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:03.857+0000 7f9fd5e39040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:16:03.857 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:03.857+0000 7f152b60f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:16:03.908 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:03.908+0000 7f152b60f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:16:03.912 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:03.913+0000 7f9fd5e39040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:16:03.966 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:03.967+0000 7f9fd5e39040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:16:03.968 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:03.969+0000 7f152b60f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:16:03.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.980+0000 7fe16ea00640 1 -- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe14c06ca10 msgr2=0x7fe14c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/280591707 2024-04-16T16:16:03.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.980+0000 7fe16ea00640 1 --2- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe14c06ca10 0x7fe14c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:16:03.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.984+0000 7fe16fe00640 1 -- 172.21.15.45:0/2487106407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe17015e5a0 con 0x7fe170151be0 2024-04-16T16:16:03.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.984+0000 7fe167400640 1 -- 172.21.15.45:0/2487106407 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7fe15805a920 con 0x7fe170151be0 2024-04-16T16:16:03.983 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:16:03.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.987+0000 7fe16fe00640 1 -- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe14c06ca10 msgr2=0x7fe14c06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:16:03.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.987+0000 7fe16fe00640 1 --2- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe14c06ca10 0x7fe14c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.987+0000 7fe16fe00640 1 -- 172.21.15.45:0/2487106407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe170151be0 msgr2=0x7fe170152060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:03.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.987+0000 7fe16fe00640 1 --2- 172.21.15.45:0/2487106407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe170151be0 0x7fe170152060 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe158007f20 tx=0x7fe15800e570 comp rx=0 tx=0).stop 2024-04-16T16:16:03.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.987+0000 7fe16fe00640 1 -- 172.21.15.45:0/2487106407 shutdown_connections 2024-04-16T16:16:03.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.987+0000 7fe16fe00640 1 --2- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fe14c06ca10 0x7fe14c06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.987+0000 7fe16fe00640 1 --2- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe170146f20 0x7fe170147360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.987+0000 7fe16fe00640 1 --2- 172.21.15.45:0/2487106407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe170151be0 0x7fe170152060 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.987+0000 7fe16fe00640 1 --2- 172.21.15.45:0/2487106407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17015c560 0x7fe170153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:03.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.987+0000 7fe16fe00640 1 -- 172.21.15.45:0/2487106407 >> 172.21.15.45:0/2487106407 conn(0x7fe170067e00 msgr2=0x7fe170161850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:03.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.987+0000 7fe16fe00640 1 -- 172.21.15.45:0/2487106407 shutdown_connections 2024-04-16T16:16:03.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:03.987+0000 7fe16fe00640 1 -- 172.21.15.45:0/2487106407 wait complete. 2024-04-16T16:16:04.005 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs1_metadata pg_num 2024-04-16T16:16:04.017 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:04.018+0000 7f152b60f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:16:04.034 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:04.035+0000 7f9fd5e39040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:16:04.088 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:04.089+0000 7f9fd5e39040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:16:04.144 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:04.145+0000 7f152b60f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:16:04.206 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:04.206+0000 7f152b60f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:16:04.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.210+0000 7fa9ebe00640 1 Processor -- start 2024-04-16T16:16:04.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.211+0000 7fa9ebe00640 1 -- start start 2024-04-16T16:16:04.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.212+0000 7fa9ebe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa9ec0d3e90 0x7fa9ec169930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:04.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.212+0000 7fa9ebe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9ec0b6810 con 0x7fa9ec0d18f0 2024-04-16T16:16:04.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.212+0000 7fa9ebe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9ec0b6af0 con 0x7fa9ec0b6310 2024-04-16T16:16:04.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.212+0000 7fa9ebe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa9ec0d29f0 con 0x7fa9ec0d3e90 2024-04-16T16:16:04.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.212+0000 7fa9eb400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa9ec0d3e90 0x7fa9ec169930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:04.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.212+0000 7fa9eaa00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa9ec0d18f0 0x7fa9ec0ba220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42400/0 (socket says 172.21.15.45:42400) 2024-04-16T16:16:04.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.212+0000 7fa9eaa00640 1 -- 172.21.15.45:0/2097691165 learned_addr learned my addr 172.21.15.45:0/2097691165 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:04.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.213+0000 7fa9eb400640 1 -- 172.21.15.45:0/2097691165 >> v1:172.21.15.103:6789/0 conn(0x7fa9ec0b6310 legacy=0x7fa9ec161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:04.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.213+0000 7fa9eb400640 1 -- 172.21.15.45:0/2097691165 >> v1:172.21.15.45:6789/0 conn(0x7fa9ec0d18f0 legacy=0x7fa9ec0ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:04.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.213+0000 7fa9e9600640 1 -- 172.21.15.45:0/2097691165 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3846259553 0 0) 0x7fa9ec0b6810 con 0x7fa9ec0d18f0 2024-04-16T16:16:04.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.213+0000 7fa9eb400640 1 -- 172.21.15.45:0/2097691165 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ec0b6dd0 con 0x7fa9ec0d3e90 2024-04-16T16:16:04.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.214+0000 7fa9eb400640 1 --2- 172.21.15.45:0/2097691165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9ec0d3e90 0x7fa9ec169930 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec0d2b70 tx=0x7fa9e402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37148b9f6f678f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:04.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.214+0000 7fa9e9600640 1 -- 172.21.15.45:0/2097691165 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4039070 con 0x7fa9ec0d3e90 2024-04-16T16:16:04.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.214+0000 7fa9e9600640 1 -- 172.21.15.45:0/2097691165 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9e402cbc0 con 0x7fa9ec0d3e90 2024-04-16T16:16:04.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.214+0000 7fa9e9600640 1 -- 172.21.15.45:0/2097691165 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4034440 con 0x7fa9ec0d3e90 2024-04-16T16:16:04.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.215+0000 7fa9ebe00640 1 -- 172.21.15.45:0/2097691165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9ec0d3e90 msgr2=0x7fa9ec169930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:04.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.215+0000 7fa9ebe00640 1 --2- 172.21.15.45:0/2097691165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9ec0d3e90 0x7fa9ec169930 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec0d2b70 tx=0x7fa9e402be10 comp rx=0 tx=0).stop 2024-04-16T16:16:04.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.215+0000 7fa9ebe00640 1 -- 172.21.15.45:0/2097691165 shutdown_connections 2024-04-16T16:16:04.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.215+0000 7fa9ebe00640 1 --2- 172.21.15.45:0/2097691165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9ec0d3e90 0x7fa9ec169930 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.215+0000 7fa9ebe00640 1 -- 172.21.15.45:0/2097691165 >> 172.21.15.45:0/2097691165 conn(0x7fa9ec06cf90 msgr2=0x7fa9ec06d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:04.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.215+0000 7fa9ebe00640 1 -- 172.21.15.45:0/2097691165 shutdown_connections 2024-04-16T16:16:04.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.215+0000 7fa9ebe00640 1 -- 172.21.15.45:0/2097691165 wait complete. 2024-04-16T16:16:04.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.217+0000 7fa9ebe00640 1 Processor -- start 2024-04-16T16:16:04.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.217+0000 7fa9ebe00640 1 -- start start 2024-04-16T16:16:04.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.218+0000 7fa9ebe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ec0b6310 0x7fa9ec0cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:04.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.218+0000 7fa9ebe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9ec0d18f0 0x7fa9ec0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:04.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.218+0000 7fa9ebe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9ec0d3e90 0x7fa9ec0ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:04.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.218+0000 7fa9ebe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ec0d1e50 con 0x7fa9ec0d18f0 2024-04-16T16:16:04.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.218+0000 7fa9ebe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ec0d1cd0 con 0x7fa9ec0b6310 2024-04-16T16:16:04.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.218+0000 7fa9ebe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa9ec0d1fd0 con 0x7fa9ec0d3e90 2024-04-16T16:16:04.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.218+0000 7fa9ea000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9ec0d18f0 0x7fa9ec0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:04.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.218+0000 7fa9ea000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9ec0d18f0 0x7fa9ec0ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51134/0 (socket says 172.21.15.45:51134) 2024-04-16T16:16:04.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.218+0000 7fa9ea000640 1 -- 172.21.15.45:0/2184044774 learned_addr learned my addr 172.21.15.45:0/2184044774 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:04.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.219+0000 7fa9eaa00640 1 --2- 172.21.15.45:0/2184044774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ec0b6310 0x7fa9ec0cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:04.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.219+0000 7fa9ea000640 1 -- 172.21.15.45:0/2184044774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9ec0d3e90 msgr2=0x7fa9ec0ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:04.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.219+0000 7fa9ea000640 1 --2- 172.21.15.45:0/2184044774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9ec0d3e90 0x7fa9ec0ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.219+0000 7fa9ea000640 1 -- 172.21.15.45:0/2184044774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ec0b6310 msgr2=0x7fa9ec0cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.219+0000 7fa9ea000640 1 --2- 172.21.15.45:0/2184044774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ec0b6310 0x7fa9ec0cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.219+0000 7fa9ea000640 1 -- 172.21.15.45:0/2184044774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ec156130 con 0x7fa9ec0d18f0 2024-04-16T16:16:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.219+0000 7fa9ea000640 1 --2- 172.21.15.45:0/2184044774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9ec0d18f0 0x7fa9ec0ce360 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800c400 tx=0x7fa9d800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.219+0000 7fa9e2a00640 1 -- 172.21.15.45:0/2184044774 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8011790 con 0x7fa9ec0d18f0 2024-04-16T16:16:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.219+0000 7fa9e2a00640 1 -- 172.21.15.45:0/2184044774 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9d8017070 con 0x7fa9ec0d18f0 2024-04-16T16:16:04.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.219+0000 7fa9e2a00640 1 -- 172.21.15.45:0/2184044774 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8011bb0 con 0x7fa9ec0d18f0 2024-04-16T16:16:04.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.219+0000 7fa9ebe00640 1 -- 172.21.15.45:0/2184044774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9ec0ce8a0 con 0x7fa9ec0d18f0 2024-04-16T16:16:04.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.220+0000 7fa9ebe00640 1 -- 172.21.15.45:0/2184044774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9ec0cc340 con 0x7fa9ec0d18f0 2024-04-16T16:16:04.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.221+0000 7fa9ebe00640 1 -- 172.21.15.45:0/2184044774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9ec075780 con 0x7fa9ec0d18f0 2024-04-16T16:16:04.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.223+0000 7fa9e2a00640 1 -- 172.21.15.45:0/2184044774 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) ==== 91142+0+0 (secure 0 0 0) 0x7fa9d801c460 con 0x7fa9ec0d18f0 2024-04-16T16:16:04.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.223+0000 7fa9e2a00640 1 --2- 172.21.15.45:0/2184044774 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa9cc06ca10 0x7fa9cc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:04.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.224+0000 7fa9e2a00640 1 -- 172.21.15.45:0/2184044774 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7197+0+0 (secure 0 0 0) 0x7fa9d808c120 con 0x7fa9ec0d18f0 2024-04-16T16:16:04.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.224+0000 7fa9eaa00640 1 -- 172.21.15.45:0/2184044774 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa9cc06ca10 msgr2=0x7fa9cc06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/280591707 2024-04-16T16:16:04.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.224+0000 7fa9eaa00640 1 --2- 172.21.15.45:0/2184044774 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa9cc06ca10 0x7fa9cc06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:16:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.228+0000 7fa9e2a00640 1 -- 172.21.15.45:0/2184044774 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa9d8059e60 con 0x7fa9ec0d18f0 2024-04-16T16:16:04.231 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:04.232+0000 7f9fd5e39040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:16:04.258 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:04.258+0000 7f152b60f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:16:04.299 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:04.300+0000 7f9fd5e39040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:16:04.315 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:04.316+0000 7f152b60f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:16:04.352 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:04.353+0000 7f9fd5e39040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:16:04.365 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:04.366+0000 7f152b60f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:16:04.414 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:04.414+0000 7f9fd5e39040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:16:04.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.425+0000 7fa9eaa00640 1 -- 172.21.15.45:0/2184044774 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa9cc06ca10 msgr2=0x7fa9cc06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/280591707 2024-04-16T16:16:04.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.425+0000 7fa9eaa00640 1 --2- 172.21.15.45:0/2184044774 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa9cc06ca10 0x7fa9cc06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:16:04.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.433+0000 7fa9ebe00640 1 -- 172.21.15.45:0/2184044774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs1_metadata", "var": "pg_num"} v 0) -- 0x7fa9ec0d25a0 con 0x7fa9ec0d18f0 2024-04-16T16:16:04.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.434+0000 7fa9e2a00640 1 -- 172.21.15.45:0/2184044774 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs1_metadata", "var": "pg_num"}]=0 v56) ==== 101+0+11 (secure 0 0 0) 0x7fa9d8059760 con 0x7fa9ec0d18f0 2024-04-16T16:16:04.433 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:16:04.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.436+0000 7fa9ebe00640 1 -- 172.21.15.45:0/2184044774 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa9cc06ca10 msgr2=0x7fa9cc06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:16:04.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.436+0000 7fa9ebe00640 1 --2- 172.21.15.45:0/2184044774 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa9cc06ca10 0x7fa9cc06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.436+0000 7fa9ebe00640 1 -- 172.21.15.45:0/2184044774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9ec0d18f0 msgr2=0x7fa9ec0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:04.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.436+0000 7fa9ebe00640 1 --2- 172.21.15.45:0/2184044774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9ec0d18f0 0x7fa9ec0ce360 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800c400 tx=0x7fa9d800c8c0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.437+0000 7fa9ebe00640 1 -- 172.21.15.45:0/2184044774 shutdown_connections 2024-04-16T16:16:04.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.437+0000 7fa9ebe00640 1 --2- 172.21.15.45:0/2184044774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9ec0d3e90 0x7fa9ec0ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.437+0000 7fa9ebe00640 1 --2- 172.21.15.45:0/2184044774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9ec0d18f0 0x7fa9ec0ce360 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.437+0000 7fa9ebe00640 1 --2- 172.21.15.45:0/2184044774 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fa9cc06ca10 0x7fa9cc06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.437+0000 7fa9ebe00640 1 --2- 172.21.15.45:0/2184044774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ec0b6310 0x7fa9ec0cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.437+0000 7fa9ebe00640 1 -- 172.21.15.45:0/2184044774 >> 172.21.15.45:0/2184044774 conn(0x7fa9ec06cf90 msgr2=0x7fa9ec161060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:04.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.437+0000 7fa9ebe00640 1 -- 172.21.15.45:0/2184044774 shutdown_connections 2024-04-16T16:16:04.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.437+0000 7fa9ebe00640 1 -- 172.21.15.45:0/2184044774 wait complete. 2024-04-16T16:16:04.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs1_data pg_num 2024-04-16T16:16:04.469 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:04.471+0000 7f9fd5e39040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:16:04.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.686+0000 7f685e800640 1 Processor -- start 2024-04-16T16:16:04.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.687+0000 7f685e800640 1 -- start start 2024-04-16T16:16:04.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.688+0000 7f685e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68580ba500 0x7f68580d4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:04.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.688+0000 7f685e800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f685815ed60 con 0x7f685815e980 2024-04-16T16:16:04.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.688+0000 7f685e800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68580d3170 con 0x7f68580cf4d0 2024-04-16T16:16:04.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.688+0000 7f685e800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f68580d2f50 con 0x7f68580ba500 2024-04-16T16:16:04.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.688+0000 7f685d400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f685815e980 0x7f68580d2840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42402/0 (socket says 172.21.15.45:42402) 2024-04-16T16:16:04.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.688+0000 7f685d400640 1 -- 172.21.15.45:0/4096070115 learned_addr learned my addr 172.21.15.45:0/4096070115 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:04.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.689+0000 7f685ca00640 1 --2- 172.21.15.45:0/4096070115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68580ba500 0x7f68580d4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:04.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.689+0000 7f6857e00640 1 -- 172.21.15.45:0/4096070115 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3801542041 0 0) 0x7f685815ed60 con 0x7f685815e980 2024-04-16T16:16:04.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.689+0000 7f6857e00640 1 -- 172.21.15.45:0/4096070115 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6840003520 con 0x7f685815e980 2024-04-16T16:16:04.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.689+0000 7f6857e00640 1 -- 172.21.15.45:0/4096070115 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4190006209 0 0) 0x7f6840003520 con 0x7f685815e980 2024-04-16T16:16:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.689+0000 7f6857e00640 1 -- 172.21.15.45:0/4096070115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68580ba500 msgr2=0x7f68580d4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.689+0000 7f6857e00640 1 --2- 172.21.15.45:0/4096070115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68580ba500 0x7f68580d4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.689+0000 7f6857e00640 1 -- 172.21.15.45:0/4096070115 >> v1:172.21.15.103:6789/0 conn(0x7f68580cf4d0 legacy=0x7f68580cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.689+0000 7f6857e00640 1 -- 172.21.15.45:0/4096070115 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68580d3450 con 0x7f685815e980 2024-04-16T16:16:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.689+0000 7f6857e00640 1 -- 172.21.15.45:0/4096070115 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f684c001670 con 0x7f685815e980 2024-04-16T16:16:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.689+0000 7f6857e00640 1 -- 172.21.15.45:0/4096070115 >> v1:172.21.15.45:6789/0 conn(0x7f685815e980 legacy=0x7f68580d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.690+0000 7f6857e00640 1 --2- 172.21.15.45:0/4096070115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68580ba500 0x7f68400039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.690+0000 7f6857e00640 1 --2- 172.21.15.45:0/4096070115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6840003fb0 0x7f6840004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.690+0000 7f6857e00640 1 --2- 172.21.15.45:0/4096070115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6840004d40 0x7f68400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.690+0000 7f6857e00640 1 -- 172.21.15.45:0/4096070115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f68580d2f50 con 0x7f6840004d40 2024-04-16T16:16:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.690+0000 7f6857e00640 1 -- 172.21.15.45:0/4096070115 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f684000bb90 con 0x7f6840003fb0 2024-04-16T16:16:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.690+0000 7f6857e00640 1 -- 172.21.15.45:0/4096070115 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f684000bd90 con 0x7f68580ba500 2024-04-16T16:16:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.690+0000 7f685de00640 1 --2- 172.21.15.45:0/4096070115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6840004d40 0x7f68400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.690+0000 7f685d400640 1 --2- 172.21.15.45:0/4096070115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68580ba500 0x7f68400039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.690+0000 7f685ca00640 1 --2- 172.21.15.45:0/4096070115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6840003fb0 0x7f6840004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.690+0000 7f685de00640 1 -- 172.21.15.45:0/4096070115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68580ba500 msgr2=0x7f68400039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.690+0000 7f685de00640 1 --2- 172.21.15.45:0/4096070115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68580ba500 0x7f68400039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.690+0000 7f685de00640 1 -- 172.21.15.45:0/4096070115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6840003fb0 msgr2=0x7f6840004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.690+0000 7f685de00640 1 --2- 172.21.15.45:0/4096070115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6840003fb0 0x7f6840004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.690+0000 7f685de00640 1 -- 172.21.15.45:0/4096070115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6840003520 con 0x7f6840004d40 2024-04-16T16:16:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.692+0000 7f685de00640 1 --2- 172.21.15.45:0/4096070115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6840004d40 0x7f68400051c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6850001e50 tx=0x7f6840005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c4c9711ff739889 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.692+0000 7f6857e00640 1 -- 172.21.15.45:0/4096070115 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850001a70 con 0x7f6840004d40 2024-04-16T16:16:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.692+0000 7f6857e00640 1 -- 172.21.15.45:0/4096070115 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6850003b70 con 0x7f6840004d40 2024-04-16T16:16:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.692+0000 7f6857e00640 1 -- 172.21.15.45:0/4096070115 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68500050d0 con 0x7f6840004d40 2024-04-16T16:16:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.692+0000 7f685e800640 1 -- 172.21.15.45:0/4096070115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6840004d40 msgr2=0x7f68400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.692+0000 7f685e800640 1 --2- 172.21.15.45:0/4096070115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6840004d40 0x7f68400051c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6850001e50 tx=0x7f6840005830 comp rx=0 tx=0).stop 2024-04-16T16:16:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.692+0000 7f685de00640 1 -- 172.21.15.45:0/4096070115 reap_dead start 2024-04-16T16:16:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.693+0000 7f685e800640 1 -- 172.21.15.45:0/4096070115 shutdown_connections 2024-04-16T16:16:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.693+0000 7f685e800640 1 -- 172.21.15.45:0/4096070115 >> 172.21.15.45:0/4096070115 conn(0x7f685806d2f0 msgr2=0x7f68580d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.693+0000 7f685e800640 1 -- 172.21.15.45:0/4096070115 shutdown_connections 2024-04-16T16:16:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.693+0000 7f685e800640 1 -- 172.21.15.45:0/4096070115 wait complete. 2024-04-16T16:16:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.694+0000 7f685e800640 1 Processor -- start 2024-04-16T16:16:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.695+0000 7f685e800640 1 -- start start 2024-04-16T16:16:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.696+0000 7f685e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6840004d40 0x7f68580c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.696+0000 7f685e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68580c98c0 0x7f685815ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.696+0000 7f685e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f685815f2a0 0x7f6858153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.696+0000 7f685e800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f68580d3cf0 con 0x7f685815f2a0 2024-04-16T16:16:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.696+0000 7f685e800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f68580d3a10 con 0x7f6840004d40 2024-04-16T16:16:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.696+0000 7f685e800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f68580d4210 con 0x7f68580c98c0 2024-04-16T16:16:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.696+0000 7f685de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f685815f2a0 0x7f6858153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.696+0000 7f685de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f685815f2a0 0x7f6858153590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51148/0 (socket says 172.21.15.45:51148) 2024-04-16T16:16:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.696+0000 7f685de00640 1 -- 172.21.15.45:0/1472584565 learned_addr learned my addr 172.21.15.45:0/1472584565 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.696+0000 7f685d400640 1 --2- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6840004d40 0x7f68580c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.696+0000 7f685ca00640 1 --2- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68580c98c0 0x7f685815ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.696+0000 7f685de00640 1 -- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68580c98c0 msgr2=0x7f685815ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.696+0000 7f685de00640 1 --2- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68580c98c0 0x7f685815ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.696+0000 7f685de00640 1 -- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6840004d40 msgr2=0x7f68580c9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.696+0000 7f685de00640 1 --2- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6840004d40 0x7f68580c9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.697+0000 7f685de00640 1 -- 172.21.15.45:0/1472584565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68580c9e60 con 0x7f685815f2a0 2024-04-16T16:16:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.697+0000 7f685de00640 1 --2- 172.21.15.45:0/1472584565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f685815f2a0 0x7f6858153590 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6850001f70 tx=0x7f68500014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.697+0000 7f6855600640 1 -- 172.21.15.45:0/1472584565 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850007db0 con 0x7f685815f2a0 2024-04-16T16:16:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.697+0000 7f6855600640 1 -- 172.21.15.45:0/1472584565 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6850003970 con 0x7f685815f2a0 2024-04-16T16:16:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.697+0000 7f685e800640 1 -- 172.21.15.45:0/1472584565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68580ca140 con 0x7f685815f2a0 2024-04-16T16:16:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.697+0000 7f6855600640 1 -- 172.21.15.45:0/1472584565 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850006d70 con 0x7f685815f2a0 2024-04-16T16:16:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.697+0000 7f685e800640 1 -- 172.21.15.45:0/1472584565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68580ca700 con 0x7f685815f2a0 2024-04-16T16:16:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.698+0000 7f685e800640 1 -- 172.21.15.45:0/1472584565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68580d25f0 con 0x7f685815f2a0 2024-04-16T16:16:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.699+0000 7f6855600640 1 -- 172.21.15.45:0/1472584565 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) ==== 91142+0+0 (secure 0 0 0) 0x7f6850041030 con 0x7f685815f2a0 2024-04-16T16:16:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.700+0000 7f6855600640 1 --2- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f683806ca10 0x7f683806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.700+0000 7f6855600640 1 -- 172.21.15.45:0/1472584565 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7197+0+0 (secure 0 0 0) 0x7f68500ad540 con 0x7f685815f2a0 2024-04-16T16:16:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.700+0000 7f685d400640 1 -- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f683806ca10 msgr2=0x7f683806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/280591707 2024-04-16T16:16:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.700+0000 7f685d400640 1 --2- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f683806ca10 0x7f683806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:16:04.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.705+0000 7f6855600640 1 -- 172.21.15.45:0/1472584565 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6850033020 con 0x7f685815f2a0 2024-04-16T16:16:04.748 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:04.749+0000 7f152b60f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:16:04.885 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:04.886+0000 7f9fd5e39040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:16:04.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.901+0000 7f685d400640 1 -- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f683806ca10 msgr2=0x7f683806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/280591707 2024-04-16T16:16:04.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.901+0000 7f685d400640 1 --2- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f683806ca10 0x7f683806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:16:04.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.907+0000 7f685e800640 1 -- 172.21.15.45:0/1472584565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs1_data", "var": "pg_num"} v 0) -- 0x7f68580d2310 con 0x7f685815f2a0 2024-04-16T16:16:04.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.908+0000 7f6855600640 1 -- 172.21.15.45:0/1472584565 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs1_data", "var": "pg_num"}]=0 v56) ==== 97+0+11 (secure 0 0 0) 0x7f6850003b40 con 0x7f685815f2a0 2024-04-16T16:16:04.907 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:16:04.909 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:04.909+0000 7f152b60f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:16:04.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.910+0000 7f685e800640 1 -- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f683806ca10 msgr2=0x7f683806eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:16:04.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.910+0000 7f685e800640 1 --2- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f683806ca10 0x7f683806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.910+0000 7f685e800640 1 -- 172.21.15.45:0/1472584565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f685815f2a0 msgr2=0x7f6858153590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:04.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.910+0000 7f685e800640 1 --2- 172.21.15.45:0/1472584565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f685815f2a0 0x7f6858153590 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6850001f70 tx=0x7f68500014a0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.910+0000 7f685e800640 1 -- 172.21.15.45:0/1472584565 shutdown_connections 2024-04-16T16:16:04.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.910+0000 7f685e800640 1 --2- 172.21.15.45:0/1472584565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f685815f2a0 0x7f6858153590 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.910+0000 7f685e800640 1 --2- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68580c98c0 0x7f685815ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.910+0000 7f685e800640 1 --2- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f683806ca10 0x7f683806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.910+0000 7f685e800640 1 --2- 172.21.15.45:0/1472584565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6840004d40 0x7f68580c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:04.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.910+0000 7f685e800640 1 -- 172.21.15.45:0/1472584565 >> 172.21.15.45:0/1472584565 conn(0x7f685806d2f0 msgr2=0x7f68580b44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:04.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.910+0000 7f685e800640 1 -- 172.21.15.45:0/1472584565 shutdown_connections 2024-04-16T16:16:04.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:04.911+0000 7f685e800640 1 -- 172.21.15.45:0/1472584565 wait complete. 2024-04-16T16:16:04.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs2_metadata pg_num 2024-04-16T16:16:05.016 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:05.016+0000 7f152b60f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:16:05.058 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:05.059+0000 7f9fd5e39040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:16:05.066 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:05.067+0000 7f152b60f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:16:05.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.144+0000 7fc56f200640 1 Processor -- start 2024-04-16T16:16:05.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.144+0000 7fc56f200640 1 -- start start 2024-04-16T16:16:05.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.145+0000 7fc56f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc5700cf5c0 0x7fc5700d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.145+0000 7fc56f200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5700d35a0 con 0x7fc57015eef0 2024-04-16T16:16:05.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.146+0000 7fc56f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5700d2740 con 0x7fc5700d42d0 2024-04-16T16:16:05.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.146+0000 7fc56f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc5700d2a20 con 0x7fc5700cf5c0 2024-04-16T16:16:05.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.146+0000 7fc56d400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc57015eef0 0x7fc5700cf9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42404/0 (socket says 172.21.15.45:42404) 2024-04-16T16:16:05.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.146+0000 7fc56d400640 1 -- 172.21.15.45:0/1969182218 learned_addr learned my addr 172.21.15.45:0/1969182218 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:05.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.146+0000 7fc56de00640 1 --2- 172.21.15.45:0/1969182218 >> v2:172.21.15.103:3300/0 conn(0x7fc5700cf5c0 0x7fc5700d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:05.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56ca00640 1 -- 172.21.15.45:0/1969182218 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2621602910 0 0) 0x7fc5700d35a0 con 0x7fc57015eef0 2024-04-16T16:16:05.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56ca00640 1 -- 172.21.15.45:0/1969182218 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc550003520 con 0x7fc57015eef0 2024-04-16T16:16:05.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56ca00640 1 -- 172.21.15.45:0/1969182218 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 96621117 0 0) 0x7fc550003520 con 0x7fc57015eef0 2024-04-16T16:16:05.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56ca00640 1 -- 172.21.15.45:0/1969182218 >> v2:172.21.15.103:3300/0 conn(0x7fc5700cf5c0 msgr2=0x7fc5700d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:05.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56ca00640 1 --2- 172.21.15.45:0/1969182218 >> v2:172.21.15.103:3300/0 conn(0x7fc5700cf5c0 0x7fc5700d3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56ca00640 1 -- 172.21.15.45:0/1969182218 >> v1:172.21.15.103:6789/0 conn(0x7fc5700d42d0 legacy=0x7fc5700b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:05.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56ca00640 1 -- 172.21.15.45:0/1969182218 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5700d2d00 con 0x7fc57015eef0 2024-04-16T16:16:05.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56ca00640 1 -- 172.21.15.45:0/1969182218 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fc5580019f0 con 0x7fc57015eef0 2024-04-16T16:16:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56ca00640 1 -- 172.21.15.45:0/1969182218 >> v1:172.21.15.45:6789/0 conn(0x7fc57015eef0 legacy=0x7fc5700cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56ca00640 1 --2- 172.21.15.45:0/1969182218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc550003a00 0x7fc550003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56ca00640 1 --2- 172.21.15.45:0/1969182218 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc550004800 0x7fc550004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56ca00640 1 --2- 172.21.15.45:0/1969182218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5500051c0 0x7fc55000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56ca00640 1 -- 172.21.15.45:0/1969182218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc5700d2a20 con 0x7fc5500051c0 2024-04-16T16:16:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56ca00640 1 -- 172.21.15.45:0/1969182218 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc55000c0c0 con 0x7fc550003a00 2024-04-16T16:16:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56ca00640 1 -- 172.21.15.45:0/1969182218 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc55000c2c0 con 0x7fc550004800 2024-04-16T16:16:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.147+0000 7fc56e800640 1 --2- 172.21.15.45:0/1969182218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5500051c0 0x7fc55000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.148+0000 7fc56e800640 1 -- 172.21.15.45:0/1969182218 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc550004800 msgr2=0x7fc550004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.148+0000 7fc56e800640 1 --2- 172.21.15.45:0/1969182218 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc550004800 0x7fc550004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.148+0000 7fc56de00640 1 --2- 172.21.15.45:0/1969182218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc550003a00 0x7fc550003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.148+0000 7fc56e800640 1 -- 172.21.15.45:0/1969182218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc550003a00 msgr2=0x7fc550003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.148+0000 7fc56e800640 1 --2- 172.21.15.45:0/1969182218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc550003a00 0x7fc550003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.148+0000 7fc56e800640 1 -- 172.21.15.45:0/1969182218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc550003520 con 0x7fc5500051c0 2024-04-16T16:16:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.148+0000 7fc56de00640 1 --2- 172.21.15.45:0/1969182218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc550003a00 0x7fc550003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:05.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.149+0000 7fc56e800640 1 -- 172.21.15.45:0/1969182218 reap_dead start 2024-04-16T16:16:05.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.149+0000 7fc56e800640 1 --2- 172.21.15.45:0/1969182218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5500051c0 0x7fc55000b9c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc568001720 tx=0x7fc550005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a0b8ce0b098a94 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:05.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.149+0000 7fc56ca00640 1 -- 172.21.15.45:0/1969182218 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568001370 con 0x7fc5500051c0 2024-04-16T16:16:05.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.149+0000 7fc56ca00640 1 -- 172.21.15.45:0/1969182218 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5680052d0 con 0x7fc5500051c0 2024-04-16T16:16:05.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.149+0000 7fc56ca00640 1 -- 172.21.15.45:0/1969182218 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568003830 con 0x7fc5500051c0 2024-04-16T16:16:05.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.150+0000 7fc56f200640 1 -- 172.21.15.45:0/1969182218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5500051c0 msgr2=0x7fc55000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:05.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.150+0000 7fc56f200640 1 --2- 172.21.15.45:0/1969182218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5500051c0 0x7fc55000b9c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc568001720 tx=0x7fc550005830 comp rx=0 tx=0).stop 2024-04-16T16:16:05.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.150+0000 7fc56f200640 1 -- 172.21.15.45:0/1969182218 shutdown_connections 2024-04-16T16:16:05.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.150+0000 7fc56f200640 1 --2- 172.21.15.45:0/1969182218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5500051c0 0x7fc55000b9c0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.150+0000 7fc56f200640 1 -- 172.21.15.45:0/1969182218 >> 172.21.15.45:0/1969182218 conn(0x7fc57006d300 msgr2=0x7fc5700d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:05.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.150+0000 7fc56f200640 1 -- 172.21.15.45:0/1969182218 shutdown_connections 2024-04-16T16:16:05.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.150+0000 7fc56f200640 1 -- 172.21.15.45:0/1969182218 wait complete. 2024-04-16T16:16:05.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.152+0000 7fc56f200640 1 Processor -- start 2024-04-16T16:16:05.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.152+0000 7fc56f200640 1 -- start start 2024-04-16T16:16:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.153+0000 7fc56f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5500051c0 0x7fc57014ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.153+0000 7fc56f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5700caae0 0x7fc5700b5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.153+0000 7fc56f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5700caf60 0x7fc5700b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.153+0000 7fc56f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc5700d3980 con 0x7fc5500051c0 2024-04-16T16:16:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.153+0000 7fc56f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc5700d32c0 con 0x7fc5700caf60 2024-04-16T16:16:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.153+0000 7fc56f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc5700d3b00 con 0x7fc5700caae0 2024-04-16T16:16:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.153+0000 7fc56de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5500051c0 0x7fc57014ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.153+0000 7fc56de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5500051c0 0x7fc57014ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51164/0 (socket says 172.21.15.45:51164) 2024-04-16T16:16:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.153+0000 7fc56de00640 1 -- 172.21.15.45:0/2202897734 learned_addr learned my addr 172.21.15.45:0/2202897734 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:05.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.154+0000 7fc56d400640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5700caae0 0x7fc5700b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:05.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.154+0000 7fc56e800640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5700caf60 0x7fc5700b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:05.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.154+0000 7fc56de00640 1 -- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5700caae0 msgr2=0x7fc5700b5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:05.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.154+0000 7fc56de00640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5700caae0 0x7fc5700b5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.154+0000 7fc56de00640 1 -- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5700caf60 msgr2=0x7fc5700b5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:05.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.154+0000 7fc56de00640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5700caf60 0x7fc5700b5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.154+0000 7fc56de00640 1 -- 172.21.15.45:0/2202897734 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5700c9a40 con 0x7fc5500051c0 2024-04-16T16:16:05.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.154+0000 7fc56d400640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5700caae0 0x7fc5700b5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:05.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.154+0000 7fc56e800640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5700caf60 0x7fc5700b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:05.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.154+0000 7fc56de00640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5500051c0 0x7fc57014ce40 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc5600108b0 tx=0x7fc560010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:05.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.154+0000 7fc566000640 1 -- 172.21.15.45:0/2202897734 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56000f040 con 0x7fc5500051c0 2024-04-16T16:16:05.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.155+0000 7fc566000640 1 -- 172.21.15.45:0/2202897734 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc56000eb60 con 0x7fc5500051c0 2024-04-16T16:16:05.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.155+0000 7fc566000640 1 -- 172.21.15.45:0/2202897734 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56001c400 con 0x7fc5500051c0 2024-04-16T16:16:05.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.155+0000 7fc56f200640 1 -- 172.21.15.45:0/2202897734 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5700c9d20 con 0x7fc5500051c0 2024-04-16T16:16:05.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.155+0000 7fc56f200640 1 -- 172.21.15.45:0/2202897734 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5700ca2e0 con 0x7fc5500051c0 2024-04-16T16:16:05.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.156+0000 7fc56f200640 1 -- 172.21.15.45:0/2202897734 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc57015b520 con 0x7fc5500051c0 2024-04-16T16:16:05.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.157+0000 7fc566000640 1 -- 172.21.15.45:0/2202897734 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) ==== 91142+0+0 (secure 0 0 0) 0x7fc56001c5a0 con 0x7fc5500051c0 2024-04-16T16:16:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.158+0000 7fc566000640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc54c06ca10 0x7fc54c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.158+0000 7fc566000640 1 -- 172.21.15.45:0/2202897734 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7197+0+0 (secure 0 0 0) 0x7fc56008c9d0 con 0x7fc5500051c0 2024-04-16T16:16:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.158+0000 7fc56d400640 1 -- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc54c06ca10 msgr2=0x7fc54c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/280591707 2024-04-16T16:16:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.158+0000 7fc56d400640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc54c06ca10 0x7fc54c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:16:05.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.162+0000 7fc566000640 1 -- 172.21.15.45:0/2202897734 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc560018030 con 0x7fc5500051c0 2024-04-16T16:16:05.179 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:05.180+0000 7f9fd5e39040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:16:05.194 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:05.194+0000 7f152b60f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:16:05.234 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:05.235+0000 7f9fd5e39040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:16:05.301 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:05.302+0000 7f152b60f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:16:05.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.359+0000 7fc56d400640 1 -- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc54c06ca10 msgr2=0x7fc54c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/280591707 2024-04-16T16:16:05.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.359+0000 7fc56d400640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc54c06ca10 0x7fc54c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:16:05.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.365+0000 7fc56f200640 1 -- 172.21.15.45:0/2202897734 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs2_metadata", "var": "pg_num"} v 0) -- 0x7fc5700d3fd0 con 0x7fc5500051c0 2024-04-16T16:16:05.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.365+0000 7fc566000640 1 -- 172.21.15.45:0/2202897734 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs2_metadata", "var": "pg_num"}]=0 v56) ==== 101+0+11 (secure 0 0 0) 0x7fc56005a0c0 con 0x7fc5500051c0 2024-04-16T16:16:05.364 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:16:05.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.368+0000 7fc56f200640 1 -- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc54c06ca10 msgr2=0x7fc54c06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:16:05.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.368+0000 7fc56f200640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc54c06ca10 0x7fc54c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.368+0000 7fc56f200640 1 -- 172.21.15.45:0/2202897734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5500051c0 msgr2=0x7fc57014ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:05.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.368+0000 7fc56f200640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5500051c0 0x7fc57014ce40 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc5600108b0 tx=0x7fc560010d70 comp rx=0 tx=0).stop 2024-04-16T16:16:05.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.368+0000 7fc56f200640 1 -- 172.21.15.45:0/2202897734 shutdown_connections 2024-04-16T16:16:05.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.368+0000 7fc56f200640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7fc54c06ca10 0x7fc54c06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.368+0000 7fc56f200640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5700caf60 0x7fc5700b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.368+0000 7fc56f200640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5700caae0 0x7fc5700b5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.368+0000 7fc56f200640 1 --2- 172.21.15.45:0/2202897734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5500051c0 0x7fc57014ce40 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.368+0000 7fc56f200640 1 -- 172.21.15.45:0/2202897734 >> 172.21.15.45:0/2202897734 conn(0x7fc57006d300 msgr2=0x7fc5700d4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:05.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.369+0000 7fc56f200640 1 -- 172.21.15.45:0/2202897734 shutdown_connections 2024-04-16T16:16:05.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.369+0000 7fc56f200640 1 -- 172.21.15.45:0/2202897734 wait complete. 2024-04-16T16:16:05.369 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:05.370+0000 7f9fd5e39040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:16:05.385 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs2_data pg_num 2024-04-16T16:16:05.475 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:05.475+0000 7f152b60f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:16:05.488 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:05.489+0000 7f9fd5e39040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:16:05.524 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:05.525+0000 7f152b60f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:16:05.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.599+0000 7f0a9be00640 1 Processor -- start 2024-04-16T16:16:05.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.599+0000 7f0a9be00640 1 -- start start 2024-04-16T16:16:05.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.601+0000 7f0a9be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a9c15e980 0x7f0a9c0d4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.601+0000 7f0a9be00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a9c15ed60 con 0x7f0a9c0cf4d0 2024-04-16T16:16:05.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.601+0000 7f0a9be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a9c0d3170 con 0x7f0a9c0ba500 2024-04-16T16:16:05.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.601+0000 7f0a9be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0a9c0d2f50 con 0x7f0a9c15e980 2024-04-16T16:16:05.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.601+0000 7f0a9b400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0a9c0cf4d0 0x7f0a9c0cf960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42406/0 (socket says 172.21.15.45:42406) 2024-04-16T16:16:05.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.601+0000 7f0a9b400640 1 -- 172.21.15.45:0/936183636 learned_addr learned my addr 172.21.15.45:0/936183636 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:05.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.602+0000 7f0a9aa00640 1 --2- 172.21.15.45:0/936183636 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a9c15e980 0x7f0a9c0d4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:05.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.602+0000 7f0a9a000640 1 -- 172.21.15.45:0/936183636 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2906272224 0 0) 0x7f0a9c15ed60 con 0x7f0a9c0cf4d0 2024-04-16T16:16:05.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.602+0000 7f0a9a000640 1 -- 172.21.15.45:0/936183636 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a80003520 con 0x7f0a9c0cf4d0 2024-04-16T16:16:05.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.602+0000 7f0a9a000640 1 -- 172.21.15.45:0/936183636 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1322540798 0 0) 0x7f0a80003520 con 0x7f0a9c0cf4d0 2024-04-16T16:16:05.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.602+0000 7f0a9a000640 1 -- 172.21.15.45:0/936183636 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a9c15e980 msgr2=0x7f0a9c0d4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:05.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.602+0000 7f0a9a000640 1 --2- 172.21.15.45:0/936183636 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a9c15e980 0x7f0a9c0d4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.602+0000 7f0a9a000640 1 -- 172.21.15.45:0/936183636 >> v1:172.21.15.103:6789/0 conn(0x7f0a9c0ba500 legacy=0x7f0a9c0d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:05.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.602+0000 7f0a9a000640 1 -- 172.21.15.45:0/936183636 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9c0d3450 con 0x7f0a9c0cf4d0 2024-04-16T16:16:05.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.602+0000 7f0a9a000640 1 -- 172.21.15.45:0/936183636 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f0a94001670 con 0x7f0a9c0cf4d0 2024-04-16T16:16:05.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.602+0000 7f0a9a000640 1 -- 172.21.15.45:0/936183636 >> v1:172.21.15.45:6789/0 conn(0x7f0a9c0cf4d0 legacy=0x7f0a9c0cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:05.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.602+0000 7f0a9a000640 1 --2- 172.21.15.45:0/936183636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a800039e0 0x7f0a80003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.602+0000 7f0a9a000640 1 --2- 172.21.15.45:0/936183636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a80004800 0x7f0a80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.602+0000 7f0a9a000640 1 --2- 172.21.15.45:0/936183636 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a9c15e980 0x7f0a800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.603+0000 7f0a9a000640 1 -- 172.21.15.45:0/936183636 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c0d2f50 con 0x7f0a80004800 2024-04-16T16:16:05.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.603+0000 7f0a9a000640 1 -- 172.21.15.45:0/936183636 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8000bb30 con 0x7f0a800039e0 2024-04-16T16:16:05.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.603+0000 7f0a9a000640 1 -- 172.21.15.45:0/936183636 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0a8000bd30 con 0x7f0a9c15e980 2024-04-16T16:16:05.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.603+0000 7f0a92000640 1 --2- 172.21.15.45:0/936183636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a80004800 0x7f0a80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:05.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.603+0000 7f0a9aa00640 1 --2- 172.21.15.45:0/936183636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a800039e0 0x7f0a80003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:05.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.603+0000 7f0a92000640 1 -- 172.21.15.45:0/936183636 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a9c15e980 msgr2=0x7f0a800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:05.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.603+0000 7f0a92000640 1 --2- 172.21.15.45:0/936183636 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a9c15e980 0x7f0a800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.603+0000 7f0a92000640 1 -- 172.21.15.45:0/936183636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a800039e0 msgr2=0x7f0a80003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:05.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.603+0000 7f0a92000640 1 --2- 172.21.15.45:0/936183636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a800039e0 0x7f0a80003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.603+0000 7f0a92000640 1 -- 172.21.15.45:0/936183636 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a80003520 con 0x7f0a80004800 2024-04-16T16:16:05.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.604+0000 7f0a92000640 1 --2- 172.21.15.45:0/936183636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a80004800 0x7f0a80004c80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f0a84001720 tx=0x7f0a80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df2ae06b735fe86 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:05.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.604+0000 7f0a9a000640 1 -- 172.21.15.45:0/936183636 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a84001370 con 0x7f0a80004800 2024-04-16T16:16:05.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.605+0000 7f0a9a000640 1 -- 172.21.15.45:0/936183636 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a840052d0 con 0x7f0a80004800 2024-04-16T16:16:05.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.605+0000 7f0a9a000640 1 -- 172.21.15.45:0/936183636 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a84003830 con 0x7f0a80004800 2024-04-16T16:16:05.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.605+0000 7f0a9be00640 1 -- 172.21.15.45:0/936183636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a80004800 msgr2=0x7f0a80004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:05.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.605+0000 7f0a9be00640 1 --2- 172.21.15.45:0/936183636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a80004800 0x7f0a80004c80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f0a84001720 tx=0x7f0a80005830 comp rx=0 tx=0).stop 2024-04-16T16:16:05.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.605+0000 7f0a9b400640 1 -- 172.21.15.45:0/936183636 reap_dead start 2024-04-16T16:16:05.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.605+0000 7f0a9be00640 1 -- 172.21.15.45:0/936183636 shutdown_connections 2024-04-16T16:16:05.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.605+0000 7f0a9be00640 1 -- 172.21.15.45:0/936183636 >> 172.21.15.45:0/936183636 conn(0x7f0a9c06d2f0 msgr2=0x7f0a9c0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:05.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.605+0000 7f0a9be00640 1 -- 172.21.15.45:0/936183636 shutdown_connections 2024-04-16T16:16:05.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.605+0000 7f0a9be00640 1 -- 172.21.15.45:0/936183636 wait complete. 2024-04-16T16:16:05.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.607+0000 7f0a9be00640 1 Processor -- start 2024-04-16T16:16:05.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.607+0000 7f0a9be00640 1 -- start start 2024-04-16T16:16:05.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.608+0000 7f0a9be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a80004800 0x7f0a9c154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.608+0000 7f0a9be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a9c15ed60 0x7f0a9c15bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.608+0000 7f0a9be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a9c159000 0x7f0a9c159480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.608+0000 7f0a9be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c0d3cf0 con 0x7f0a80004800 2024-04-16T16:16:05.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.608+0000 7f0a9be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c0d3a10 con 0x7f0a9c15ed60 2024-04-16T16:16:05.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.608+0000 7f0a9be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0a9c0d4210 con 0x7f0a9c159000 2024-04-16T16:16:05.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.608+0000 7f0a9aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a80004800 0x7f0a9c154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:05.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.608+0000 7f0a9aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a80004800 0x7f0a9c154ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51176/0 (socket says 172.21.15.45:51176) 2024-04-16T16:16:05.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.608+0000 7f0a9aa00640 1 -- 172.21.15.45:0/815755552 learned_addr learned my addr 172.21.15.45:0/815755552 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:05.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.609+0000 7f0a92000640 1 --2- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a9c15ed60 0x7f0a9c15bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:05.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.609+0000 7f0a9b400640 1 --2- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a9c159000 0x7f0a9c159480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:05.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.609+0000 7f0a9aa00640 1 -- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a9c159000 msgr2=0x7f0a9c159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:05.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.609+0000 7f0a9aa00640 1 --2- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a9c159000 0x7f0a9c159480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.609+0000 7f0a9aa00640 1 -- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a9c15ed60 msgr2=0x7f0a9c15bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:05.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.609+0000 7f0a9aa00640 1 --2- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a9c15ed60 0x7f0a9c15bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.609+0000 7f0a9aa00640 1 -- 172.21.15.45:0/815755552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9c15f300 con 0x7f0a80004800 2024-04-16T16:16:05.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.609+0000 7f0a9aa00640 1 --2- 172.21.15.45:0/815755552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a80004800 0x7f0a9c154ae0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c007f60 tx=0x7f0a8c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:05.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.609+0000 7f0a93400640 1 -- 172.21.15.45:0/815755552 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c01a020 con 0x7f0a80004800 2024-04-16T16:16:05.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.609+0000 7f0a93400640 1 -- 172.21.15.45:0/815755552 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a8c016470 con 0x7f0a80004800 2024-04-16T16:16:05.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.609+0000 7f0a93400640 1 -- 172.21.15.45:0/815755552 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c016860 con 0x7f0a80004800 2024-04-16T16:16:05.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.610+0000 7f0a9be00640 1 -- 172.21.15.45:0/815755552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a9c15f5e0 con 0x7f0a80004800 2024-04-16T16:16:05.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.610+0000 7f0a9be00640 1 -- 172.21.15.45:0/815755552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a9c15fba0 con 0x7f0a80004800 2024-04-16T16:16:05.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.611+0000 7f0a9be00640 1 -- 172.21.15.45:0/815755552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a9c0b4300 con 0x7f0a80004800 2024-04-16T16:16:05.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.612+0000 7f0a93400640 1 -- 172.21.15.45:0/815755552 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) ==== 91142+0+0 (secure 0 0 0) 0x7f0a8c005d10 con 0x7f0a80004800 2024-04-16T16:16:05.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.613+0000 7f0a93400640 1 --2- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f0a7806ca40 0x7f0a7806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:05.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.613+0000 7f0a93400640 1 -- 172.21.15.45:0/815755552 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7197+0+0 (secure 0 0 0) 0x7f0a8c08d260 con 0x7f0a80004800 2024-04-16T16:16:05.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.613+0000 7f0a92000640 1 -- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f0a7806ca40 msgr2=0x7f0a7806ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/280591707 2024-04-16T16:16:05.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.613+0000 7f0a92000640 1 --2- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f0a7806ca40 0x7f0a7806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:16:05.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.621+0000 7f0a93400640 1 -- 172.21.15.45:0/815755552 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a8c024030 con 0x7f0a80004800 2024-04-16T16:16:05.662 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:05.662+0000 7f152b60f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:16:05.683 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:05.683+0000 7f9fd5e39040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:16:05.736 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:05.737+0000 7f9fd5e39040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:16:05.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.814+0000 7f0a92000640 1 -- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f0a7806ca40 msgr2=0x7f0a7806ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/280591707 2024-04-16T16:16:05.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.814+0000 7f0a92000640 1 --2- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f0a7806ca40 0x7f0a7806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:16:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.823+0000 7f0a9be00640 1 -- 172.21.15.45:0/815755552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs2_data", "var": "pg_num"} v 0) -- 0x7f0a9c0cfc20 con 0x7f0a80004800 2024-04-16T16:16:05.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.824+0000 7f0a93400640 1 -- 172.21.15.45:0/815755552 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs2_data", "var": "pg_num"}]=0 v56) ==== 97+0+11 (secure 0 0 0) 0x7f0a8c05ae00 con 0x7f0a80004800 2024-04-16T16:16:05.823 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:16:05.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.826+0000 7f0a9be00640 1 -- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f0a7806ca40 msgr2=0x7f0a7806ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:16:05.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.827+0000 7f0a9be00640 1 --2- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f0a7806ca40 0x7f0a7806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.827+0000 7f0a9be00640 1 -- 172.21.15.45:0/815755552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a80004800 msgr2=0x7f0a9c154ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:05.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.827+0000 7f0a9be00640 1 --2- 172.21.15.45:0/815755552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a80004800 0x7f0a9c154ae0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c007f60 tx=0x7f0a8c007f90 comp rx=0 tx=0).stop 2024-04-16T16:16:05.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.827+0000 7f0a9be00640 1 -- 172.21.15.45:0/815755552 shutdown_connections 2024-04-16T16:16:05.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.827+0000 7f0a9be00640 1 --2- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a9c159000 0x7f0a9c159480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.827+0000 7f0a9be00640 1 --2- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f0a7806ca40 0x7f0a7806ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.827+0000 7f0a9be00640 1 --2- 172.21.15.45:0/815755552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a9c15ed60 0x7f0a9c15bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.827+0000 7f0a9be00640 1 --2- 172.21.15.45:0/815755552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a80004800 0x7f0a9c154ae0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:05.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.827+0000 7f0a9be00640 1 -- 172.21.15.45:0/815755552 >> 172.21.15.45:0/815755552 conn(0x7f0a9c06d2f0 msgr2=0x7f0a9c0b7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:05.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.827+0000 7f0a9be00640 1 -- 172.21.15.45:0/815755552 shutdown_connections 2024-04-16T16:16:05.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:05.827+0000 7f0a9be00640 1 -- 172.21.15.45:0/815755552 wait complete. 2024-04-16T16:16:05.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:16:05.885 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:05.886+0000 7f9fd5e39040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:16:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.059+0000 7f2da3200640 1 Processor -- start 2024-04-16T16:16:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.059+0000 7f2da3200640 1 -- start start 2024-04-16T16:16:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.061+0000 7f2da3200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2da40c6fc0 0x7f2da406f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.061+0000 7f2da3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2da406df40 0x7f2da406e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.061+0000 7f2da3200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da415d600 con 0x7f2da415cdc0 2024-04-16T16:16:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.061+0000 7f2da3200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2da40c5670 con 0x7f2da40c6fc0 2024-04-16T16:16:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.061+0000 7f2da3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2da415d8e0 con 0x7f2da406df40 2024-04-16T16:16:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.061+0000 7f2da1e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2da40c6fc0 0x7f2da406f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.061+0000 7f2da1e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2da40c6fc0 0x7f2da406f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51184/0 (socket says 172.21.15.45:51184) 2024-04-16T16:16:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.061+0000 7f2da1e00640 1 -- 172.21.15.45:0/1701138929 learned_addr learned my addr 172.21.15.45:0/1701138929 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.061+0000 7f2da2800640 1 --2- 172.21.15.45:0/1701138929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2da406df40 0x7f2da406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.061+0000 7f2da1e00640 1 -- 172.21.15.45:0/1701138929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2da406df40 msgr2=0x7f2da406e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.062+0000 7f2da1e00640 1 --2- 172.21.15.45:0/1701138929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2da406df40 0x7f2da406e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.062+0000 7f2da1e00640 1 -- 172.21.15.45:0/1701138929 >> v1:172.21.15.103:6789/0 conn(0x7f2da415cdc0 legacy=0x7f2da40c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.062+0000 7f2da1e00640 1 -- 172.21.15.45:0/1701138929 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da415dbc0 con 0x7f2da40c6fc0 2024-04-16T16:16:06.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.062+0000 7f2da2800640 1 --2- 172.21.15.45:0/1701138929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2da406df40 0x7f2da406e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.063+0000 7f2da1e00640 1 --2- 172.21.15.45:0/1701138929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da40c6fc0 0x7f2da406f730 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f2d94009870 tx=0x7f2d9402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb1cde527c515a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.063+0000 7f2da0a00640 1 -- 172.21.15.45:0/1701138929 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94034070 con 0x7f2da40c6fc0 2024-04-16T16:16:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.063+0000 7f2da0a00640 1 -- 172.21.15.45:0/1701138929 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d94030840 con 0x7f2da40c6fc0 2024-04-16T16:16:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.063+0000 7f2da0a00640 1 -- 172.21.15.45:0/1701138929 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94030c10 con 0x7f2da40c6fc0 2024-04-16T16:16:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.064+0000 7f2da3200640 1 -- 172.21.15.45:0/1701138929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da40c6fc0 msgr2=0x7f2da406f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.064+0000 7f2da3200640 1 --2- 172.21.15.45:0/1701138929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da40c6fc0 0x7f2da406f730 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f2d94009870 tx=0x7f2d9402b980 comp rx=0 tx=0).stop 2024-04-16T16:16:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.064+0000 7f2da3200640 1 -- 172.21.15.45:0/1701138929 shutdown_connections 2024-04-16T16:16:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.064+0000 7f2da3200640 1 --2- 172.21.15.45:0/1701138929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2da406df40 0x7f2da406e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.064+0000 7f2da3200640 1 --2- 172.21.15.45:0/1701138929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da40c6fc0 0x7f2da406f730 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.064+0000 7f2da3200640 1 -- 172.21.15.45:0/1701138929 >> 172.21.15.45:0/1701138929 conn(0x7f2da406b930 msgr2=0x7f2da406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.064+0000 7f2da3200640 1 -- 172.21.15.45:0/1701138929 shutdown_connections 2024-04-16T16:16:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.064+0000 7f2da3200640 1 -- 172.21.15.45:0/1701138929 wait complete. 2024-04-16T16:16:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.066+0000 7f2da3200640 1 Processor -- start 2024-04-16T16:16:06.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.066+0000 7f2da3200640 1 -- start start 2024-04-16T16:16:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.067+0000 7f2da3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2da406df40 0x7f2da4152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.067+0000 7f2da3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2da40c6fc0 0x7f2da4150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.067+0000 7f2da3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da415cdc0 0x7f2da414bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.067+0000 7f2da3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2da40c64f0 con 0x7f2da415cdc0 2024-04-16T16:16:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.067+0000 7f2da3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2da415d270 con 0x7f2da406df40 2024-04-16T16:16:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.067+0000 7f2da3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2da40c6670 con 0x7f2da40c6fc0 2024-04-16T16:16:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.067+0000 7f2da2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da415cdc0 0x7f2da414bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.067+0000 7f2da2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da415cdc0 0x7f2da414bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51198/0 (socket says 172.21.15.45:51198) 2024-04-16T16:16:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.067+0000 7f2da2800640 1 -- 172.21.15.45:0/4272430606 learned_addr learned my addr 172.21.15.45:0/4272430606 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.067+0000 7f2da1e00640 1 --2- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2da406df40 0x7f2da4152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.067+0000 7f2da1400640 1 --2- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2da40c6fc0 0x7f2da4150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.068+0000 7f2da2800640 1 -- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2da40c6fc0 msgr2=0x7f2da4150b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.068+0000 7f2da2800640 1 --2- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2da40c6fc0 0x7f2da4150b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.068+0000 7f2da2800640 1 -- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2da406df40 msgr2=0x7f2da4152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.068+0000 7f2da2800640 1 --2- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2da406df40 0x7f2da4152810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.068+0000 7f2da2800640 1 -- 172.21.15.45:0/4272430606 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da4151410 con 0x7f2da415cdc0 2024-04-16T16:16:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.068+0000 7f2da2800640 1 --2- 172.21.15.45:0/4272430606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da415cdc0 0x7f2da414bf80 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c00c7f0 tx=0x7f2d9c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.068+0000 7f2d9a000640 1 -- 172.21.15.45:0/4272430606 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c015070 con 0x7f2da415cdc0 2024-04-16T16:16:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.068+0000 7f2d9a000640 1 -- 172.21.15.45:0/4272430606 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d9c007820 con 0x7f2da415cdc0 2024-04-16T16:16:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.068+0000 7f2d9a000640 1 -- 172.21.15.45:0/4272430606 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c007c50 con 0x7f2da415cdc0 2024-04-16T16:16:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.068+0000 7f2da3200640 1 -- 172.21.15.45:0/4272430606 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da41516f0 con 0x7f2da415cdc0 2024-04-16T16:16:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.069+0000 7f2da3200640 1 -- 172.21.15.45:0/4272430606 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da4151cb0 con 0x7f2da415cdc0 2024-04-16T16:16:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.070+0000 7f2da3200640 1 -- 172.21.15.45:0/4272430606 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da40c8580 con 0x7f2da415cdc0 2024-04-16T16:16:06.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.071+0000 7f2d9a000640 1 -- 172.21.15.45:0/4272430606 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) ==== 91142+0+0 (secure 0 0 0) 0x7f2d9c018070 con 0x7f2da415cdc0 2024-04-16T16:16:06.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.072+0000 7f2d9a000640 1 --2- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2d8006ca10 0x7f2d8006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:06.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.072+0000 7f2d9a000640 1 -- 172.21.15.45:0/4272430606 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7197+0+0 (secure 0 0 0) 0x7f2d9c08c440 con 0x7f2da415cdc0 2024-04-16T16:16:06.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.072+0000 7f2da1e00640 1 -- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2d8006ca10 msgr2=0x7f2d8006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/280591707 2024-04-16T16:16:06.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.072+0000 7f2da1e00640 1 --2- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2d8006ca10 0x7f2d8006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:16:06.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.077+0000 7f2d9a000640 1 -- 172.21.15.45:0/4272430606 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2da40c8580 con 0x7f2da415cdc0 2024-04-16T16:16:06.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.273+0000 7f2da1e00640 1 -- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2d8006ca10 msgr2=0x7f2d8006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/280591707 2024-04-16T16:16:06.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.273+0000 7f2da1e00640 1 --2- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2d8006ca10 0x7f2d8006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:16:06.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.290+0000 7f2da3200640 1 -- 172.21.15.45:0/4272430606 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2da4075780 con 0x7f2da415cdc0 2024-04-16T16:16:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.291+0000 7f2d9a000640 1 -- 172.21.15.45:0/4272430606 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 14 entries v56) ==== 81+0+844 (secure 0 0 0) 0x7f2d9c059ac0 con 0x7f2da415cdc0 2024-04-16T16:16:06.290 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6833/368835060 2024-04-17T16:15:41.256698+0000 2024-04-16T16:16:06.291 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2318159422 2024-04-17T16:15:01.083409+0000 2024-04-16T16:16:06.291 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6832/368835060 2024-04-17T16:15:41.256698+0000 2024-04-16T16:16:06.291 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/2711977587 2024-04-17T16:15:01.083409+0000 2024-04-16T16:16:06.291 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/2711977587 2024-04-17T16:15:01.083409+0000 2024-04-16T16:16:06.291 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6833/2956262988 2024-04-17T16:15:07.318380+0000 2024-04-16T16:16:06.291 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3554107853 2024-04-17T16:15:01.083409+0000 2024-04-16T16:16:06.291 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6835/2956262988 2024-04-17T16:15:07.318380+0000 2024-04-16T16:16:06.291 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6833/2276311244 2024-04-17T16:15:20.691128+0000 2024-04-16T16:16:06.291 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3010240888 2024-04-17T16:15:01.083409+0000 2024-04-16T16:16:06.291 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6834/88032513 2024-04-17T16:15:41.221894+0000 2024-04-16T16:16:06.291 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6835/2276311244 2024-04-17T16:15:20.691128+0000 2024-04-16T16:16:06.291 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2679792854 2024-04-17T16:15:01.083409+0000 2024-04-16T16:16:06.291 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6835/88032513 2024-04-17T16:15:41.221894+0000 2024-04-16T16:16:06.292 INFO:teuthology.orchestra.run.smithi045.stderr:listed 14 entries 2024-04-16T16:16:06.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.294+0000 7f2da3200640 1 -- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2d8006ca10 msgr2=0x7f2d8006eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:16:06.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.294+0000 7f2da3200640 1 --2- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2d8006ca10 0x7f2d8006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.294+0000 7f2da3200640 1 -- 172.21.15.45:0/4272430606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da415cdc0 msgr2=0x7f2da414bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:06.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.294+0000 7f2da3200640 1 --2- 172.21.15.45:0/4272430606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da415cdc0 0x7f2da414bf80 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c00c7f0 tx=0x7f2d9c00ccb0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.294+0000 7f2da3200640 1 -- 172.21.15.45:0/4272430606 shutdown_connections 2024-04-16T16:16:06.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.294+0000 7f2da3200640 1 --2- 172.21.15.45:0/4272430606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da415cdc0 0x7f2da414bf80 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.294+0000 7f2da3200640 1 --2- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2da40c6fc0 0x7f2da4150b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.294+0000 7f2da3200640 1 --2- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f2d8006ca10 0x7f2d8006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.294+0000 7f2da3200640 1 --2- 172.21.15.45:0/4272430606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2da406df40 0x7f2da4152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.294+0000 7f2da3200640 1 -- 172.21.15.45:0/4272430606 >> 172.21.15.45:0/4272430606 conn(0x7f2da406b930 msgr2=0x7f2da406e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:06.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.294+0000 7f2da3200640 1 -- 172.21.15.45:0/4272430606 shutdown_connections 2024-04-16T16:16:06.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.294+0000 7f2da3200640 1 -- 172.21.15.45:0/4272430606 wait complete. 2024-04-16T16:16:06.313 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-16T16:16:06.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:06.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-16T16:16:06.442 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:16:06.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:06.442 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-16T16:16:06.482 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:16:06.482 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:16:06.482 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:16:06.498 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:06.499+0000 7f152b60f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:16:06.535 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:16:06.535 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:16:06.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:16:06.744 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:06.745+0000 7f9fd5e39040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:16:06.818 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:06.818+0000 7f152b60f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:16:06.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.846+0000 7f7048a00640 1 Processor -- start 2024-04-16T16:16:06.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.846+0000 7f7048a00640 1 -- start start 2024-04-16T16:16:06.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.847+0000 7f7048a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7044161e90 0x7f70440cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:06.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.847+0000 7f7048a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7044162270 0x7f70440d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:06.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.847+0000 7f7048a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f70440d52d0 0x7f704406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:06.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.847+0000 7f7048a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f70440d2e30 con 0x7f7044161e90 2024-04-16T16:16:06.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.847+0000 7f7048a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f70440d3080 con 0x7f70440d52d0 2024-04-16T16:16:06.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.847+0000 7f7048a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f70440d3360 con 0x7f7044162270 2024-04-16T16:16:06.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.848+0000 7f7042800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7044161e90 0x7f70440cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:06.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.848+0000 7f7042800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7044161e90 0x7f70440cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51204/0 (socket says 172.21.15.45:51204) 2024-04-16T16:16:06.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.848+0000 7f7042800640 1 -- 172.21.15.45:0/212185903 learned_addr learned my addr 172.21.15.45:0/212185903 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:06.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.848+0000 7f7043200640 1 --2- 172.21.15.45:0/212185903 >> v2:172.21.15.103:3300/0 conn(0x7f70440d52d0 0x7f704406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:06.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.848+0000 7f703be00640 1 --2- 172.21.15.45:0/212185903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7044162270 0x7f70440d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:06.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.848+0000 7f7042800640 1 -- 172.21.15.45:0/212185903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7044162270 msgr2=0x7f70440d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:06.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.848+0000 7f7042800640 1 --2- 172.21.15.45:0/212185903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7044162270 0x7f70440d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.848+0000 7f7042800640 1 -- 172.21.15.45:0/212185903 >> v2:172.21.15.103:3300/0 conn(0x7f70440d52d0 msgr2=0x7f704406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:06.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.848+0000 7f7042800640 1 --2- 172.21.15.45:0/212185903 >> v2:172.21.15.103:3300/0 conn(0x7f70440d52d0 0x7f704406ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.848+0000 7f7042800640 1 -- 172.21.15.45:0/212185903 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70440d3640 con 0x7f7044161e90 2024-04-16T16:16:06.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.848+0000 7f703be00640 1 --2- 172.21.15.45:0/212185903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7044162270 0x7f70440d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:06.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.849+0000 7f7042800640 1 --2- 172.21.15.45:0/212185903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7044161e90 0x7f70440cf7d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f70440ba980 tx=0x7f703402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef4fa0eaec811df server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:06.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.850+0000 7f7041e00640 1 -- 172.21.15.45:0/212185903 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034034070 con 0x7f7044161e90 2024-04-16T16:16:06.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.850+0000 7f7041e00640 1 -- 172.21.15.45:0/212185903 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7034030860 con 0x7f7044161e90 2024-04-16T16:16:06.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.850+0000 7f7041e00640 1 -- 172.21.15.45:0/212185903 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034030c30 con 0x7f7044161e90 2024-04-16T16:16:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.850+0000 7f7048a00640 1 -- 172.21.15.45:0/212185903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7044161e90 msgr2=0x7f70440cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.850+0000 7f7048a00640 1 --2- 172.21.15.45:0/212185903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7044161e90 0x7f70440cf7d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f70440ba980 tx=0x7f703402b9a0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.851+0000 7f7048a00640 1 -- 172.21.15.45:0/212185903 shutdown_connections 2024-04-16T16:16:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.851+0000 7f7048a00640 1 --2- 172.21.15.45:0/212185903 >> v2:172.21.15.103:3300/0 conn(0x7f70440d52d0 0x7f704406ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.851+0000 7f7048a00640 1 --2- 172.21.15.45:0/212185903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7044162270 0x7f70440d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.851+0000 7f7048a00640 1 --2- 172.21.15.45:0/212185903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7044161e90 0x7f70440cf7d0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.851+0000 7f7048a00640 1 -- 172.21.15.45:0/212185903 >> 172.21.15.45:0/212185903 conn(0x7f70440d1940 msgr2=0x7f70440d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.851+0000 7f7048a00640 1 -- 172.21.15.45:0/212185903 shutdown_connections 2024-04-16T16:16:06.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.851+0000 7f7048a00640 1 -- 172.21.15.45:0/212185903 wait complete. 2024-04-16T16:16:06.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.852+0000 7f7048a00640 1 Processor -- start 2024-04-16T16:16:06.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.853+0000 7f7048a00640 1 -- start start 2024-04-16T16:16:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.853+0000 7f7048a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70440d52d0 0x7f70440c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.853+0000 7f7048a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7044161e90 0x7f704414da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.854+0000 7f7048a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7044162270 0x7f70441e7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.854+0000 7f7048a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f70440d4380 con 0x7f70440d52d0 2024-04-16T16:16:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.854+0000 7f7048a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f70440cfd10 con 0x7f7044161e90 2024-04-16T16:16:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.854+0000 7f7048a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f70440d4500 con 0x7f7044162270 2024-04-16T16:16:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.854+0000 7f7042800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70440d52d0 0x7f70440c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:06.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.854+0000 7f7042800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70440d52d0 0x7f70440c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51220/0 (socket says 172.21.15.45:51220) 2024-04-16T16:16:06.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.854+0000 7f7042800640 1 -- 172.21.15.45:0/1672223786 learned_addr learned my addr 172.21.15.45:0/1672223786 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:06.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.854+0000 7f703be00640 1 --2- 172.21.15.45:0/1672223786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7044161e90 0x7f704414da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:06.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.854+0000 7f7042800640 1 -- 172.21.15.45:0/1672223786 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7044162270 msgr2=0x7f70441e7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:06.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.854+0000 7f7042800640 1 --2- 172.21.15.45:0/1672223786 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7044162270 0x7f70441e7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.854+0000 7f7042800640 1 -- 172.21.15.45:0/1672223786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7044161e90 msgr2=0x7f704414da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:06.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.854+0000 7f7042800640 1 --2- 172.21.15.45:0/1672223786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7044161e90 0x7f704414da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:06.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.854+0000 7f7042800640 1 -- 172.21.15.45:0/1672223786 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f704414b750 con 0x7f70440d52d0 2024-04-16T16:16:06.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.854+0000 7f703be00640 1 --2- 172.21.15.45:0/1672223786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7044161e90 0x7f704414da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:16:06.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.855+0000 7f7042800640 1 --2- 172.21.15.45:0/1672223786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70440d52d0 0x7f70440c8c10 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f7034000c00 tx=0x7f7034004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:06.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.855+0000 7f703aa00640 1 -- 172.21.15.45:0/1672223786 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034048070 con 0x7f70440d52d0 2024-04-16T16:16:06.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.855+0000 7f703aa00640 1 -- 172.21.15.45:0/1672223786 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7034030660 con 0x7f70440d52d0 2024-04-16T16:16:06.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.855+0000 7f7048a00640 1 -- 172.21.15.45:0/1672223786 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f704414ba30 con 0x7f70440d52d0 2024-04-16T16:16:06.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.855+0000 7f703aa00640 1 -- 172.21.15.45:0/1672223786 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034034040 con 0x7f70440d52d0 2024-04-16T16:16:06.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.855+0000 7f7048a00640 1 -- 172.21.15.45:0/1672223786 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f704414c2d0 con 0x7f70440d52d0 2024-04-16T16:16:06.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.856+0000 7f7048a00640 1 -- 172.21.15.45:0/1672223786 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7044075780 con 0x7f70440d52d0 2024-04-16T16:16:06.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.857+0000 7f703aa00640 1 -- 172.21.15.45:0/1672223786 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) ==== 91142+0+0 (secure 0 0 0) 0x7f703404b070 con 0x7f70440d52d0 2024-04-16T16:16:06.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.858+0000 7f703aa00640 1 --2- 172.21.15.45:0/1672223786 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f702006ca10 0x7f702006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:06.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.858+0000 7f703aa00640 1 -- 172.21.15.45:0/1672223786 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7197+0+0 (secure 0 0 0) 0x7f70340ae5d0 con 0x7f70440d52d0 2024-04-16T16:16:06.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.858+0000 7f703be00640 1 -- 172.21.15.45:0/1672223786 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f702006ca10 msgr2=0x7f702006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/280591707 2024-04-16T16:16:06.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.858+0000 7f703be00640 1 --2- 172.21.15.45:0/1672223786 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f702006ca10 0x7f702006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:16:06.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:06.866+0000 7f703aa00640 1 -- 172.21.15.45:0/1672223786 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f703407c0f0 con 0x7f70440d52d0 2024-04-16T16:16:06.880 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:06.880+0000 7f152b60f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:16:07.010 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.010+0000 7f152b60f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:16:07.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.030+0000 7f703aa00640 1 -- 172.21.15.45:0/1672223786 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 12) ==== 90751+0+0 (secure 0 0 0) 0x7f7034037070 con 0x7f70440d52d0 2024-04-16T16:16:07.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.030+0000 7f703aa00640 1 -- 172.21.15.45:0/1672223786 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f702006ca10 msgr2=0x7f702006eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:16:07.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.030+0000 7f703aa00640 1 --2- 172.21.15.45:0/1672223786 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f702006ca10 0x7f702006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.077+0000 7f7048a00640 1 -- 172.21.15.45:0/1672223786 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70440b39f0 con 0x7f70440d52d0 2024-04-16T16:16:07.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.079+0000 7f703aa00640 1 -- 172.21.15.45:0/1672223786 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+21917 (secure 0 0 0) 0x7f7034092350 con 0x7f70440d52d0 2024-04-16T16:16:07.079 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:07.079 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":57,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:16:07.016528+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"fs1_metadata","create_time":"2024-04-16T16:15:13.250782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"fs1_data","create_time":"2024-04-16T16:15:13.856986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"fs2_metadata","create_time":"2024-04-16T16:15:33.215042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"fs2_data","create_time":"2024-04-16T16:15:34.271426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.2c","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/349066714":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/388389014":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/589483918":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3353543342":"2024-04-17T16:16:07.016480+0000","172.21.15.103:6832/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3310077212":"2024-04-17T16:16:07.016480+0000","172.21.15.45:6833/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6832/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.45:6833/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:6833/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6834/88032513":"2024-04-17T16:15:41.221894+0000","172.21.15.45:6835/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:0/644374842":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6835/88032513":"2024-04-17T16:15:41.221894+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:16:07.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.082+0000 7f7048a00640 1 -- 172.21.15.45:0/1672223786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70440d52d0 msgr2=0x7f70440c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:07.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.082+0000 7f7048a00640 1 --2- 172.21.15.45:0/1672223786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70440d52d0 0x7f70440c8c10 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f7034000c00 tx=0x7f7034004730 comp rx=0 tx=0).stop 2024-04-16T16:16:07.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.082+0000 7f7048a00640 1 -- 172.21.15.45:0/1672223786 shutdown_connections 2024-04-16T16:16:07.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.082+0000 7f7048a00640 1 --2- 172.21.15.45:0/1672223786 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7044162270 0x7f70441e7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.082+0000 7f7048a00640 1 --2- 172.21.15.45:0/1672223786 >> [v2:172.21.15.103:6832/280591707,v1:172.21.15.103:6833/280591707] conn(0x7f702006ca10 0x7f702006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.082+0000 7f7048a00640 1 --2- 172.21.15.45:0/1672223786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7044161e90 0x7f704414da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.082+0000 7f7048a00640 1 --2- 172.21.15.45:0/1672223786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70440d52d0 0x7f70440c8c10 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.082+0000 7f7048a00640 1 -- 172.21.15.45:0/1672223786 >> 172.21.15.45:0/1672223786 conn(0x7f70440d1940 msgr2=0x7f70440d2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.082+0000 7f7048a00640 1 -- 172.21.15.45:0/1672223786 shutdown_connections 2024-04-16T16:16:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.082+0000 7f7048a00640 1 -- 172.21.15.45:0/1672223786 wait complete. 2024-04-16T16:16:07.090 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:07.092+0000 7f9fd5e39040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:16:07.099 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'fs1_metadata', 'create_time': '2024-04-16T16:15:13.250782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'fs1'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'fs1_data', 'create_time': '2024-04-16T16:15:13.856986+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'fs1'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'fs2_metadata', 'create_time': '2024-04-16T16:15:33.215042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'fs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'fs2_data', 'create_time': '2024-04-16T16:15:34.271426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'fs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:16:07.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:16:07.116 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.116+0000 7f14a5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.116 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.116+0000 7f14a5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.116 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.116+0000 7f14a5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.116 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.116+0000 7f14a5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.116 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.116+0000 7f14a5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.116 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.116+0000 7f14a8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.116 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.116+0000 7f14a8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.117 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.116+0000 7f14a8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.117 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.116+0000 7f14a8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.117 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.116+0000 7f14a8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.130 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.131+0000 7f14a8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.130 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.131+0000 7f14a8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.131 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.131+0000 7f14a8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.131 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.131+0000 7f14a8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.131 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.131+0000 7f14a8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.131 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.131+0000 7f14a5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.131 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.131+0000 7f14a5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.131 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.131+0000 7f14a5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.131 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.131+0000 7f14a5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.132 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:07.131+0000 7f14a5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:16:07.157 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:07.158+0000 7f9fd5e39040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:16:07.300 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:07.301+0000 7f9fd5e39040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:16:07.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.308+0000 7ff156400640 1 Processor -- start 2024-04-16T16:16:07.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.308+0000 7ff156400640 1 -- start start 2024-04-16T16:16:07.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.310+0000 7ff156400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff15015ec30 0x7ff1500b7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:07.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.310+0000 7ff156400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff1500ba7c0 0x7ff1500b95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:07.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.310+0000 7ff156400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1500b8500 con 0x7ff1500b8060 2024-04-16T16:16:07.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.310+0000 7ff156400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff1500b87e0 con 0x7ff1500ba7c0 2024-04-16T16:16:07.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.310+0000 7ff156400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff1500b8ac0 con 0x7ff15015ec30 2024-04-16T16:16:07.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.310+0000 7ff14ea00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff1500b8060 0x7ff1500d2af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42422/0 (socket says 172.21.15.45:42422) 2024-04-16T16:16:07.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.310+0000 7ff14ea00640 1 -- 172.21.15.45:0/1111370675 learned_addr learned my addr 172.21.15.45:0/1111370675 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:07.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.310+0000 7ff14fe00640 1 --2- 172.21.15.45:0/1111370675 >> v2:172.21.15.103:3300/0 conn(0x7ff1500ba7c0 0x7ff1500b95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:07.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 -- 172.21.15.45:0/1111370675 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2764687354 0 0) 0x7ff1500b8500 con 0x7ff1500b8060 2024-04-16T16:16:07.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 -- 172.21.15.45:0/1111370675 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff134003520 con 0x7ff1500b8060 2024-04-16T16:16:07.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14f400640 1 --2- 172.21.15.45:0/1111370675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff15015ec30 0x7ff1500b7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:07.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 -- 172.21.15.45:0/1111370675 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2385699187 0 0) 0x7ff134003520 con 0x7ff1500b8060 2024-04-16T16:16:07.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 -- 172.21.15.45:0/1111370675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff15015ec30 msgr2=0x7ff1500b7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:07.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 --2- 172.21.15.45:0/1111370675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff15015ec30 0x7ff1500b7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 -- 172.21.15.45:0/1111370675 >> v2:172.21.15.103:3300/0 conn(0x7ff1500ba7c0 msgr2=0x7ff1500b95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:07.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 --2- 172.21.15.45:0/1111370675 >> v2:172.21.15.103:3300/0 conn(0x7ff1500ba7c0 0x7ff1500b95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 -- 172.21.15.45:0/1111370675 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1500b8da0 con 0x7ff1500b8060 2024-04-16T16:16:07.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 -- 172.21.15.45:0/1111370675 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7ff138001670 con 0x7ff1500b8060 2024-04-16T16:16:07.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 -- 172.21.15.45:0/1111370675 >> v1:172.21.15.45:6789/0 conn(0x7ff1500b8060 legacy=0x7ff1500d2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:07.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 --2- 172.21.15.45:0/1111370675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1340039b0 0x7ff134003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:07.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 --2- 172.21.15.45:0/1111370675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff134004800 0x7ff134004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:07.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 --2- 172.21.15.45:0/1111370675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1340051c0 0x7ff13400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:07.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 -- 172.21.15.45:0/1111370675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff1500b87e0 con 0x7ff1340051c0 2024-04-16T16:16:07.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 -- 172.21.15.45:0/1111370675 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff1500b8ac0 con 0x7ff1340039b0 2024-04-16T16:16:07.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14e000640 1 -- 172.21.15.45:0/1111370675 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff13400bf80 con 0x7ff134004800 2024-04-16T16:16:07.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.311+0000 7ff14fe00640 1 --2- 172.21.15.45:0/1111370675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1340051c0 0x7ff13400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:07.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.312+0000 7ff14fe00640 1 -- 172.21.15.45:0/1111370675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff134004800 msgr2=0x7ff134004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:07.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.312+0000 7ff14fe00640 1 --2- 172.21.15.45:0/1111370675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff134004800 0x7ff134004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.312+0000 7ff14fe00640 1 -- 172.21.15.45:0/1111370675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1340039b0 msgr2=0x7ff134003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:07.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.312+0000 7ff14fe00640 1 --2- 172.21.15.45:0/1111370675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1340039b0 0x7ff134003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.312+0000 7ff14fe00640 1 -- 172.21.15.45:0/1111370675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff134003520 con 0x7ff1340051c0 2024-04-16T16:16:07.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.313+0000 7ff14fe00640 1 --2- 172.21.15.45:0/1111370675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1340051c0 0x7ff13400b9c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff140009ac0 tx=0x7ff134005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b481f524ae0a2fc server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:07.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.313+0000 7ff14e000640 1 -- 172.21.15.45:0/1111370675 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140039070 con 0x7ff1340051c0 2024-04-16T16:16:07.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.313+0000 7ff14e000640 1 -- 172.21.15.45:0/1111370675 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff14002c710 con 0x7ff1340051c0 2024-04-16T16:16:07.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.313+0000 7ff14e000640 1 -- 172.21.15.45:0/1111370675 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140009690 con 0x7ff1340051c0 2024-04-16T16:16:07.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.313+0000 7ff156400640 1 -- 172.21.15.45:0/1111370675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1340051c0 msgr2=0x7ff13400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:07.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.313+0000 7ff156400640 1 --2- 172.21.15.45:0/1111370675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1340051c0 0x7ff13400b9c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff140009ac0 tx=0x7ff134005830 comp rx=0 tx=0).stop 2024-04-16T16:16:07.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.313+0000 7ff14fe00640 1 -- 172.21.15.45:0/1111370675 reap_dead start 2024-04-16T16:16:07.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.314+0000 7ff156400640 1 -- 172.21.15.45:0/1111370675 shutdown_connections 2024-04-16T16:16:07.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.314+0000 7ff156400640 1 -- 172.21.15.45:0/1111370675 >> 172.21.15.45:0/1111370675 conn(0x7ff150073480 msgr2=0x7ff150073890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:07.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.314+0000 7ff156400640 1 -- 172.21.15.45:0/1111370675 shutdown_connections 2024-04-16T16:16:07.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.314+0000 7ff156400640 1 -- 172.21.15.45:0/1111370675 wait complete. 2024-04-16T16:16:07.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.315+0000 7ff156400640 1 Processor -- start 2024-04-16T16:16:07.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.316+0000 7ff156400640 1 -- start start 2024-04-16T16:16:07.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.316+0000 7ff156400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1340051c0 0x7ff150161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:07.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.316+0000 7ff156400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff150154300 0x7ff15015a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:07.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.316+0000 7ff156400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff15015ad60 0x7ff150160230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:07.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.316+0000 7ff156400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff15015f010 con 0x7ff150154300 2024-04-16T16:16:07.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.316+0000 7ff156400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff1500d3200 con 0x7ff1340051c0 2024-04-16T16:16:07.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.316+0000 7ff156400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff1500b7b20 con 0x7ff15015ad60 2024-04-16T16:16:07.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.317+0000 7ff14ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff150154300 0x7ff15015a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:07.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.317+0000 7ff14ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff150154300 0x7ff15015a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51238/0 (socket says 172.21.15.45:51238) 2024-04-16T16:16:07.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.317+0000 7ff14ea00640 1 -- 172.21.15.45:0/3018145124 learned_addr learned my addr 172.21.15.45:0/3018145124 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:07.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.317+0000 7ff14f400640 1 --2- 172.21.15.45:0/3018145124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1340051c0 0x7ff150161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:07.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.317+0000 7ff14fe00640 1 --2- 172.21.15.45:0/3018145124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff15015ad60 0x7ff150160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:07.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.317+0000 7ff14ea00640 1 -- 172.21.15.45:0/3018145124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff15015ad60 msgr2=0x7ff150160230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:07.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.317+0000 7ff14ea00640 1 --2- 172.21.15.45:0/3018145124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff15015ad60 0x7ff150160230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.317+0000 7ff14ea00640 1 -- 172.21.15.45:0/3018145124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1340051c0 msgr2=0x7ff150161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:07.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.317+0000 7ff14ea00640 1 --2- 172.21.15.45:0/3018145124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1340051c0 0x7ff150161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.318+0000 7ff14ea00640 1 -- 172.21.15.45:0/3018145124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff150158820 con 0x7ff150154300 2024-04-16T16:16:07.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.318+0000 7ff14ea00640 1 --2- 172.21.15.45:0/3018145124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff150154300 0x7ff15015a820 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff13800c7a0 tx=0x7ff13800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:07.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.318+0000 7ff147400640 1 -- 172.21.15.45:0/3018145124 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff138018070 con 0x7ff150154300 2024-04-16T16:16:07.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.318+0000 7ff156400640 1 -- 172.21.15.45:0/3018145124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1500cb1a0 con 0x7ff150154300 2024-04-16T16:16:07.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.318+0000 7ff147400640 1 -- 172.21.15.45:0/3018145124 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff13800ce50 con 0x7ff150154300 2024-04-16T16:16:07.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.318+0000 7ff147400640 1 -- 172.21.15.45:0/3018145124 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1380047e0 con 0x7ff150154300 2024-04-16T16:16:07.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.318+0000 7ff156400640 1 -- 172.21.15.45:0/3018145124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff150158da0 con 0x7ff150154300 2024-04-16T16:16:07.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.319+0000 7ff156400640 1 -- 172.21.15.45:0/3018145124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff150075780 con 0x7ff150154300 2024-04-16T16:16:07.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.321+0000 7ff147400640 1 -- 172.21.15.45:0/3018145124 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) ==== 90751+0+0 (secure 0 0 0) 0x7ff138004980 con 0x7ff150154300 2024-04-16T16:16:07.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.322+0000 7ff147400640 1 -- 172.21.15.45:0/3018145124 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7ff13808b1b0 con 0x7ff150154300 2024-04-16T16:16:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.326+0000 7ff147400640 1 -- 172.21.15.45:0/3018145124 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff138058c30 con 0x7ff150154300 2024-04-16T16:16:07.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.536+0000 7ff156400640 1 -- 172.21.15.45:0/3018145124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff1500ba580 con 0x7ff150154300 2024-04-16T16:16:07.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.537+0000 7ff147400640 1 -- 172.21.15.45:0/3018145124 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) ==== 93+0+10 (secure 0 0 0) 0x7ff13805cf30 con 0x7ff150154300 2024-04-16T16:16:07.536 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:16:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.539+0000 7ff156400640 1 -- 172.21.15.45:0/3018145124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff150154300 msgr2=0x7ff15015a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.540+0000 7ff156400640 1 --2- 172.21.15.45:0/3018145124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff150154300 0x7ff15015a820 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff13800c7a0 tx=0x7ff13800cc60 comp rx=0 tx=0).stop 2024-04-16T16:16:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.540+0000 7ff156400640 1 -- 172.21.15.45:0/3018145124 shutdown_connections 2024-04-16T16:16:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.540+0000 7ff156400640 1 --2- 172.21.15.45:0/3018145124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff15015ad60 0x7ff150160230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.540+0000 7ff156400640 1 --2- 172.21.15.45:0/3018145124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff150154300 0x7ff15015a820 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.540+0000 7ff156400640 1 --2- 172.21.15.45:0/3018145124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1340051c0 0x7ff150161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.540+0000 7ff156400640 1 -- 172.21.15.45:0/3018145124 >> 172.21.15.45:0/3018145124 conn(0x7ff150073480 msgr2=0x7ff15015e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.540+0000 7ff156400640 1 -- 172.21.15.45:0/3018145124 shutdown_connections 2024-04-16T16:16:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.540+0000 7ff156400640 1 -- 172.21.15.45:0/3018145124 wait complete. 2024-04-16T16:16:07.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs1_metadata pg_num 2024-04-16T16:16:07.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.786+0000 7f31c7200640 1 Processor -- start 2024-04-16T16:16:07.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.787+0000 7f31c7200640 1 -- start start 2024-04-16T16:16:07.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.788+0000 7f31c7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cce0 0x7f31c8160ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:07.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.788+0000 7f31c7200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f31c8072fc0 0x7f31c80b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:07.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.788+0000 7f31c7200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31c80cf6f0 con 0x7f31c80cf8f0 2024-04-16T16:16:07.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.788+0000 7f31c7200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f31c8141e40 con 0x7f31c8072fc0 2024-04-16T16:16:07.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.788+0000 7f31c7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f31c815d230 con 0x7f31c815cce0 2024-04-16T16:16:07.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.789+0000 7f31c5e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f31c80cf8f0 0x7f31c80cfcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42434/0 (socket says 172.21.15.45:42434) 2024-04-16T16:16:07.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.789+0000 7f31c5e00640 1 -- 172.21.15.45:0/2740913593 learned_addr learned my addr 172.21.15.45:0/2740913593 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:07.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.789+0000 7f31c6800640 1 --2- 172.21.15.45:0/2740913593 >> v2:172.21.15.45:3300/0 conn(0x7f31c8072fc0 0x7f31c80b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:07.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.789+0000 7f31c5400640 1 --2- 172.21.15.45:0/2740913593 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cce0 0x7f31c8160ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:07.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.789+0000 7f31c4a00640 1 -- 172.21.15.45:0/2740913593 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1986973831 0 0) 0x7f31c80cf6f0 con 0x7f31c80cf8f0 2024-04-16T16:16:07.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.789+0000 7f31c4a00640 1 -- 172.21.15.45:0/2740913593 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31ac003520 con 0x7f31c80cf8f0 2024-04-16T16:16:07.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.789+0000 7f31c6800640 1 -- 172.21.15.45:0/2740913593 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cce0 msgr2=0x7f31c8160ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:07.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.789+0000 7f31c6800640 1 --2- 172.21.15.45:0/2740913593 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cce0 0x7f31c8160ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.789+0000 7f31c6800640 1 -- 172.21.15.45:0/2740913593 >> v1:172.21.15.45:6789/0 conn(0x7f31c80cf8f0 legacy=0x7f31c80cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:07.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.789+0000 7f31c6800640 1 -- 172.21.15.45:0/2740913593 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c815d510 con 0x7f31c8072fc0 2024-04-16T16:16:07.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.789+0000 7f31c5400640 1 --2- 172.21.15.45:0/2740913593 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cce0 0x7f31c8160ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:16:07.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.791+0000 7f31c6800640 1 --2- 172.21.15.45:0/2740913593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8072fc0 0x7f31c80b36e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f31c00097a0 tx=0x7f31c002b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e24bbc562dbe40 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:07.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.791+0000 7f31c4a00640 1 -- 172.21.15.45:0/2740913593 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c0034070 con 0x7f31c8072fc0 2024-04-16T16:16:07.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.791+0000 7f31c4a00640 1 -- 172.21.15.45:0/2740913593 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31c0030800 con 0x7f31c8072fc0 2024-04-16T16:16:07.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.791+0000 7f31c4a00640 1 -- 172.21.15.45:0/2740913593 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c0030bd0 con 0x7f31c8072fc0 2024-04-16T16:16:07.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.792+0000 7f31c7200640 1 -- 172.21.15.45:0/2740913593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8072fc0 msgr2=0x7f31c80b36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:07.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.792+0000 7f31c7200640 1 --2- 172.21.15.45:0/2740913593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8072fc0 0x7f31c80b36e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f31c00097a0 tx=0x7f31c002b940 comp rx=0 tx=0).stop 2024-04-16T16:16:07.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.792+0000 7f31c7200640 1 -- 172.21.15.45:0/2740913593 shutdown_connections 2024-04-16T16:16:07.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.792+0000 7f31c7200640 1 --2- 172.21.15.45:0/2740913593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8072fc0 0x7f31c80b36e0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.792+0000 7f31c7200640 1 --2- 172.21.15.45:0/2740913593 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cce0 0x7f31c8160ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.792+0000 7f31c7200640 1 -- 172.21.15.45:0/2740913593 >> 172.21.15.45:0/2740913593 conn(0x7f31c80b6ae0 msgr2=0x7f31c80b6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:07.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.792+0000 7f31c7200640 1 -- 172.21.15.45:0/2740913593 shutdown_connections 2024-04-16T16:16:07.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.792+0000 7f31c7200640 1 -- 172.21.15.45:0/2740913593 wait complete. 2024-04-16T16:16:07.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.794+0000 7f31c7200640 1 Processor -- start 2024-04-16T16:16:07.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.794+0000 7f31c7200640 1 -- start start 2024-04-16T16:16:07.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.795+0000 7f31c7200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8072fc0 0x7f31c8152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:07.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.795+0000 7f31c7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31c80cf8f0 0x7f31c81e60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:07.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.795+0000 7f31c7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cce0 0x7f31c81e6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:07.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.795+0000 7f31c7200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f31c8073450 con 0x7f31c8072fc0 2024-04-16T16:16:07.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.795+0000 7f31c7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f31c815dad0 con 0x7f31c80cf8f0 2024-04-16T16:16:07.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.795+0000 7f31c7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f31c80735d0 con 0x7f31c815cce0 2024-04-16T16:16:07.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.795+0000 7f31c5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8072fc0 0x7f31c8152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:07.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.795+0000 7f31c5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8072fc0 0x7f31c8152f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51248/0 (socket says 172.21.15.45:51248) 2024-04-16T16:16:07.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.795+0000 7f31c5e00640 1 -- 172.21.15.45:0/2750272012 learned_addr learned my addr 172.21.15.45:0/2750272012 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:07.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.795+0000 7f31c6800640 1 --2- 172.21.15.45:0/2750272012 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cce0 0x7f31c81e6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:07.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.795+0000 7f31c5400640 1 --2- 172.21.15.45:0/2750272012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31c80cf8f0 0x7f31c81e60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:07.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.796+0000 7f31c5e00640 1 -- 172.21.15.45:0/2750272012 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cce0 msgr2=0x7f31c81e6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:07.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.796+0000 7f31c5e00640 1 --2- 172.21.15.45:0/2750272012 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cce0 0x7f31c81e6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.796+0000 7f31c5e00640 1 -- 172.21.15.45:0/2750272012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31c80cf8f0 msgr2=0x7f31c81e60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:07.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.796+0000 7f31c5e00640 1 --2- 172.21.15.45:0/2750272012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31c80cf8f0 0x7f31c81e60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:07.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.796+0000 7f31c5e00640 1 -- 172.21.15.45:0/2750272012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c813efb0 con 0x7f31c8072fc0 2024-04-16T16:16:07.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.796+0000 7f31c5400640 1 --2- 172.21.15.45:0/2750272012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31c80cf8f0 0x7f31c81e60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:16:07.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.796+0000 7f31c5e00640 1 --2- 172.21.15.45:0/2750272012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8072fc0 0x7f31c8152f60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f31b8001460 tx=0x7f31b800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:07.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.796+0000 7f31be000640 1 -- 172.21.15.45:0/2750272012 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8017070 con 0x7f31c8072fc0 2024-04-16T16:16:07.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.796+0000 7f31be000640 1 -- 172.21.15.45:0/2750272012 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31b800ad00 con 0x7f31c8072fc0 2024-04-16T16:16:07.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.796+0000 7f31be000640 1 -- 172.21.15.45:0/2750272012 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8012640 con 0x7f31c8072fc0 2024-04-16T16:16:07.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.796+0000 7f31c7200640 1 -- 172.21.15.45:0/2750272012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c81e6cc0 con 0x7f31c8072fc0 2024-04-16T16:16:07.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.797+0000 7f31c7200640 1 -- 172.21.15.45:0/2750272012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c81e7500 con 0x7f31c8072fc0 2024-04-16T16:16:07.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.798+0000 7f31c7200640 1 -- 172.21.15.45:0/2750272012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31c8075780 con 0x7f31c8072fc0 2024-04-16T16:16:07.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.799+0000 7f31be000640 1 -- 172.21.15.45:0/2750272012 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) ==== 90751+0+0 (secure 0 0 0) 0x7f31b801a070 con 0x7f31c8072fc0 2024-04-16T16:16:07.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.800+0000 7f31be000640 1 -- 172.21.15.45:0/2750272012 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7f31b808bc50 con 0x7f31c8072fc0 2024-04-16T16:16:07.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:07.804+0000 7f31be000640 1 -- 172.21.15.45:0/2750272012 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f31c81e6cc0 con 0x7f31c8072fc0 2024-04-16T16:16:08.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.018+0000 7f31c7200640 1 -- 172.21.15.45:0/2750272012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs1_metadata", "var": "pg_num"} v 0) -- 0x7f31c80b3f40 con 0x7f31c8072fc0 2024-04-16T16:16:08.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.019+0000 7f31be000640 1 -- 172.21.15.45:0/2750272012 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs1_metadata", "var": "pg_num"}]=0 v57) ==== 101+0+11 (secure 0 0 0) 0x7f31b80595d0 con 0x7f31c8072fc0 2024-04-16T16:16:08.019 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:16:08.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.022+0000 7f31c7200640 1 -- 172.21.15.45:0/2750272012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8072fc0 msgr2=0x7f31c8152f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:08.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.022+0000 7f31c7200640 1 --2- 172.21.15.45:0/2750272012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8072fc0 0x7f31c8152f60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f31b8001460 tx=0x7f31b800a430 comp rx=0 tx=0).stop 2024-04-16T16:16:08.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.022+0000 7f31c7200640 1 -- 172.21.15.45:0/2750272012 shutdown_connections 2024-04-16T16:16:08.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.022+0000 7f31c7200640 1 --2- 172.21.15.45:0/2750272012 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31c815cce0 0x7f31c81e6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.022+0000 7f31c7200640 1 --2- 172.21.15.45:0/2750272012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31c80cf8f0 0x7f31c81e60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.022+0000 7f31c7200640 1 --2- 172.21.15.45:0/2750272012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c8072fc0 0x7f31c8152f60 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.022+0000 7f31c7200640 1 -- 172.21.15.45:0/2750272012 >> 172.21.15.45:0/2750272012 conn(0x7f31c80b6ae0 msgr2=0x7f31c8140ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:08.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.023+0000 7f31c7200640 1 -- 172.21.15.45:0/2750272012 shutdown_connections 2024-04-16T16:16:08.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.023+0000 7f31c7200640 1 -- 172.21.15.45:0/2750272012 wait complete. 2024-04-16T16:16:08.042 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs1_data pg_num 2024-04-16T16:16:08.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.265+0000 7fa520a00640 1 Processor -- start 2024-04-16T16:16:08.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.266+0000 7fa520a00640 1 -- start start 2024-04-16T16:16:08.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.267+0000 7fa520a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa51c0b6310 0x7fa51c161db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:08.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.267+0000 7fa520a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa51c0b6810 con 0x7fa51c0d18f0 2024-04-16T16:16:08.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.267+0000 7fa520a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa51c0b6af0 con 0x7fa51c0d3e90 2024-04-16T16:16:08.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.267+0000 7fa520a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa51c0d29f0 con 0x7fa51c0b6310 2024-04-16T16:16:08.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.268+0000 7fa51a800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa51c0d18f0 0x7fa51c0ba220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42436/0 (socket says 172.21.15.45:42436) 2024-04-16T16:16:08.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.268+0000 7fa51a800640 1 -- 172.21.15.45:0/1658227739 learned_addr learned my addr 172.21.15.45:0/1658227739 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:08.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.268+0000 7fa513e00640 1 --2- 172.21.15.45:0/1658227739 >> v2:172.21.15.45:3300/0 conn(0x7fa51c0b6310 0x7fa51c161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:08.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.268+0000 7fa519e00640 1 -- 172.21.15.45:0/1658227739 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1321300908 0 0) 0x7fa51c0b6810 con 0x7fa51c0d18f0 2024-04-16T16:16:08.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.268+0000 7fa519e00640 1 -- 172.21.15.45:0/1658227739 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa500003520 con 0x7fa51c0d18f0 2024-04-16T16:16:08.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.268+0000 7fa513e00640 1 -- 172.21.15.45:0/1658227739 >> v1:172.21.15.103:6789/0 conn(0x7fa51c0d3e90 legacy=0x7fa51c1622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:08.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.268+0000 7fa513e00640 1 -- 172.21.15.45:0/1658227739 >> v1:172.21.15.45:6789/0 conn(0x7fa51c0d18f0 legacy=0x7fa51c0ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:08.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.268+0000 7fa513e00640 1 -- 172.21.15.45:0/1658227739 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa51c0b6dd0 con 0x7fa51c0b6310 2024-04-16T16:16:08.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.269+0000 7fa513e00640 1 --2- 172.21.15.45:0/1658227739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c0b6310 0x7fa51c161db0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fa51c0d2b70 tx=0x7fa50402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0dcaadb6b40d16 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:08.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.270+0000 7fa519e00640 1 -- 172.21.15.45:0/1658227739 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5040315a0 con 0x7fa51c0b6310 2024-04-16T16:16:08.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.270+0000 7fa519e00640 1 -- 172.21.15.45:0/1658227739 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa504037070 con 0x7fa51c0b6310 2024-04-16T16:16:08.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.270+0000 7fa519e00640 1 -- 172.21.15.45:0/1658227739 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa504031940 con 0x7fa51c0b6310 2024-04-16T16:16:08.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.270+0000 7fa520a00640 1 -- 172.21.15.45:0/1658227739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c0b6310 msgr2=0x7fa51c161db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:08.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.270+0000 7fa520a00640 1 --2- 172.21.15.45:0/1658227739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c0b6310 0x7fa51c161db0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fa51c0d2b70 tx=0x7fa50402b840 comp rx=0 tx=0).stop 2024-04-16T16:16:08.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.271+0000 7fa520a00640 1 -- 172.21.15.45:0/1658227739 shutdown_connections 2024-04-16T16:16:08.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.271+0000 7fa520a00640 1 --2- 172.21.15.45:0/1658227739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c0b6310 0x7fa51c161db0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.271+0000 7fa520a00640 1 -- 172.21.15.45:0/1658227739 >> 172.21.15.45:0/1658227739 conn(0x7fa51c06cf90 msgr2=0x7fa51c06d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:08.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.271+0000 7fa520a00640 1 -- 172.21.15.45:0/1658227739 shutdown_connections 2024-04-16T16:16:08.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.271+0000 7fa520a00640 1 -- 172.21.15.45:0/1658227739 wait complete. 2024-04-16T16:16:08.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.272+0000 7fa520a00640 1 Processor -- start 2024-04-16T16:16:08.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.273+0000 7fa520a00640 1 -- start start 2024-04-16T16:16:08.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.273+0000 7fa520a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa51c0b6310 0x7fa51c154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:08.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.274+0000 7fa520a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c0d18f0 0x7fa51c0cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:08.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.274+0000 7fa520a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c0d3e90 0x7fa51c0cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:08.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.274+0000 7fa520a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c0d1e50 con 0x7fa51c0d3e90 2024-04-16T16:16:08.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.274+0000 7fa520a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c0d1cd0 con 0x7fa51c0b6310 2024-04-16T16:16:08.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.274+0000 7fa520a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa51c0d1fd0 con 0x7fa51c0d18f0 2024-04-16T16:16:08.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.274+0000 7fa51b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c0d3e90 0x7fa51c0cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:08.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.274+0000 7fa51b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c0d3e90 0x7fa51c0cd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51268/0 (socket says 172.21.15.45:51268) 2024-04-16T16:16:08.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.274+0000 7fa51b200640 1 -- 172.21.15.45:0/3721633188 learned_addr learned my addr 172.21.15.45:0/3721633188 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:08.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.274+0000 7fa51a800640 1 --2- 172.21.15.45:0/3721633188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa51c0b6310 0x7fa51c154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:08.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.274+0000 7fa51b200640 1 -- 172.21.15.45:0/3721633188 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c0d18f0 msgr2=0x7fa51c0cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:08.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.274+0000 7fa51b200640 1 --2- 172.21.15.45:0/3721633188 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c0d18f0 0x7fa51c0cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.274+0000 7fa51b200640 1 -- 172.21.15.45:0/3721633188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa51c0b6310 msgr2=0x7fa51c154e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:08.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.274+0000 7fa51b200640 1 --2- 172.21.15.45:0/3721633188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa51c0b6310 0x7fa51c154e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.274+0000 7fa51b200640 1 -- 172.21.15.45:0/3721633188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa51c155350 con 0x7fa51c0d3e90 2024-04-16T16:16:08.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.275+0000 7fa51b200640 1 --2- 172.21.15.45:0/3721633188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c0d3e90 0x7fa51c0cd640 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa51400b9a0 tx=0x7fa51400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:08.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.275+0000 7fa512a00640 1 -- 172.21.15.45:0/3721633188 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa514019070 con 0x7fa51c0d3e90 2024-04-16T16:16:08.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.275+0000 7fa512a00640 1 -- 172.21.15.45:0/3721633188 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa51400cba0 con 0x7fa51c0d3e90 2024-04-16T16:16:08.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.275+0000 7fa512a00640 1 -- 172.21.15.45:0/3721633188 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa514014440 con 0x7fa51c0d3e90 2024-04-16T16:16:08.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.275+0000 7fa520a00640 1 -- 172.21.15.45:0/3721633188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa51c0cc970 con 0x7fa51c0d3e90 2024-04-16T16:16:08.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.275+0000 7fa520a00640 1 -- 172.21.15.45:0/3721633188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa51c0cc690 con 0x7fa51c0d3e90 2024-04-16T16:16:08.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.277+0000 7fa520a00640 1 -- 172.21.15.45:0/3721633188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa51c0d0290 con 0x7fa51c0d3e90 2024-04-16T16:16:08.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.278+0000 7fa512a00640 1 -- 172.21.15.45:0/3721633188 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) ==== 90878+0+0 (secure 0 0 0) 0x7fa51401c070 con 0x7fa51c0d3e90 2024-04-16T16:16:08.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.279+0000 7fa512a00640 1 --2- 172.21.15.45:0/3721633188 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa4f806c740 0x7fa4f806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:08.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.279+0000 7fa512a00640 1 -- 172.21.15.45:0/3721633188 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7fa51408bb20 con 0x7fa51c0d3e90 2024-04-16T16:16:08.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.279+0000 7fa51a800640 1 --2- 172.21.15.45:0/3721633188 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa4f806c740 0x7fa4f806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:08.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.281+0000 7fa51a800640 1 --2- 172.21.15.45:0/3721633188 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa4f806c740 0x7fa4f806ec00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa50c000f60 tx=0x7fa50c009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:08.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.284+0000 7fa512a00640 1 -- 172.21.15.45:0/3721633188 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa51c0d0290 con 0x7fa51c0d3e90 2024-04-16T16:16:08.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.496+0000 7fa520a00640 1 -- 172.21.15.45:0/3721633188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs1_data", "var": "pg_num"} v 0) -- 0x7fa51c0d25a0 con 0x7fa51c0d3e90 2024-04-16T16:16:08.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.497+0000 7fa512a00640 1 -- 172.21.15.45:0/3721633188 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs1_data", "var": "pg_num"}]=0 v57) ==== 97+0+11 (secure 0 0 0) 0x7fa51c155350 con 0x7fa51c0d3e90 2024-04-16T16:16:08.496 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:16:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.500+0000 7fa520a00640 1 -- 172.21.15.45:0/3721633188 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa4f806c740 msgr2=0x7fa4f806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.500+0000 7fa520a00640 1 --2- 172.21.15.45:0/3721633188 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa4f806c740 0x7fa4f806ec00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa50c000f60 tx=0x7fa50c009040 comp rx=0 tx=0).stop 2024-04-16T16:16:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.500+0000 7fa520a00640 1 -- 172.21.15.45:0/3721633188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c0d3e90 msgr2=0x7fa51c0cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.500+0000 7fa520a00640 1 --2- 172.21.15.45:0/3721633188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c0d3e90 0x7fa51c0cd640 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa51400b9a0 tx=0x7fa51400be60 comp rx=0 tx=0).stop 2024-04-16T16:16:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.500+0000 7fa520a00640 1 -- 172.21.15.45:0/3721633188 shutdown_connections 2024-04-16T16:16:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.500+0000 7fa520a00640 1 --2- 172.21.15.45:0/3721633188 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa4f806c740 0x7fa4f806ec00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.500+0000 7fa520a00640 1 --2- 172.21.15.45:0/3721633188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c0d3e90 0x7fa51c0cd640 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.500+0000 7fa520a00640 1 --2- 172.21.15.45:0/3721633188 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51c0d18f0 0x7fa51c0cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.500+0000 7fa520a00640 1 --2- 172.21.15.45:0/3721633188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa51c0b6310 0x7fa51c154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.500+0000 7fa520a00640 1 -- 172.21.15.45:0/3721633188 >> 172.21.15.45:0/3721633188 conn(0x7fa51c06cf90 msgr2=0x7fa51c15f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:08.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.500+0000 7fa520a00640 1 -- 172.21.15.45:0/3721633188 shutdown_connections 2024-04-16T16:16:08.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.501+0000 7fa520a00640 1 -- 172.21.15.45:0/3721633188 wait complete. 2024-04-16T16:16:08.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs2_metadata pg_num 2024-04-16T16:16:08.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.744+0000 7f89c7200640 1 Processor -- start 2024-04-16T16:16:08.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.744+0000 7f89c7200640 1 -- start start 2024-04-16T16:16:08.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.745+0000 7f89c7200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f89c81461b0 0x7f89c8160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:08.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.745+0000 7f89c7200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89c80d0260 con 0x7f89c80b5d50 2024-04-16T16:16:08.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.745+0000 7f89c7200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89c8146590 con 0x7f89c80b58f0 2024-04-16T16:16:08.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.746+0000 7f89c7200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f89c806dfd0 con 0x7f89c81461b0 2024-04-16T16:16:08.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.746+0000 7f89c5e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f89c81461b0 0x7f89c8160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:08.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.746+0000 7f89c5e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f89c81461b0 0x7f89c8160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51274/0 (socket says 172.21.15.45:51274) 2024-04-16T16:16:08.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.746+0000 7f89c5e00640 1 -- 172.21.15.45:0/4287616731 learned_addr learned my addr 172.21.15.45:0/4287616731 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:08.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.746+0000 7f89c5e00640 1 -- 172.21.15.45:0/4287616731 >> v1:172.21.15.103:6789/0 conn(0x7f89c80b58f0 legacy=0x7f89c80cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:08.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.746+0000 7f89c5e00640 1 -- 172.21.15.45:0/4287616731 >> v1:172.21.15.45:6789/0 conn(0x7f89c80b5d50 legacy=0x7f89c80cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:08.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.746+0000 7f89c5e00640 1 -- 172.21.15.45:0/4287616731 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c80c5750 con 0x7f89c81461b0 2024-04-16T16:16:08.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.748+0000 7f89c5e00640 1 --2- 172.21.15.45:0/4287616731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89c81461b0 0x7f89c8160cf0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f89b8009870 tx=0x7f89b802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f7ec2c19878d7c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:08.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.748+0000 7f89c5400640 1 -- 172.21.15.45:0/4287616731 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8034070 con 0x7f89c81461b0 2024-04-16T16:16:08.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.748+0000 7f89c5400640 1 -- 172.21.15.45:0/4287616731 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b8030860 con 0x7f89c81461b0 2024-04-16T16:16:08.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.748+0000 7f89c5400640 1 -- 172.21.15.45:0/4287616731 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8030c30 con 0x7f89c81461b0 2024-04-16T16:16:08.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.749+0000 7f89c7200640 1 -- 172.21.15.45:0/4287616731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89c81461b0 msgr2=0x7f89c8160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:08.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.749+0000 7f89c7200640 1 --2- 172.21.15.45:0/4287616731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89c81461b0 0x7f89c8160cf0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f89b8009870 tx=0x7f89b802b9a0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.749+0000 7f89c7200640 1 -- 172.21.15.45:0/4287616731 shutdown_connections 2024-04-16T16:16:08.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.749+0000 7f89c7200640 1 --2- 172.21.15.45:0/4287616731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89c81461b0 0x7f89c8160cf0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.749+0000 7f89c7200640 1 -- 172.21.15.45:0/4287616731 >> 172.21.15.45:0/4287616731 conn(0x7f89c806b930 msgr2=0x7f89c806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:08.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.749+0000 7f89c7200640 1 -- 172.21.15.45:0/4287616731 shutdown_connections 2024-04-16T16:16:08.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.749+0000 7f89c7200640 1 -- 172.21.15.45:0/4287616731 wait complete. 2024-04-16T16:16:08.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.751+0000 7f89c7200640 1 Processor -- start 2024-04-16T16:16:08.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.751+0000 7f89c7200640 1 -- start start 2024-04-16T16:16:08.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.753+0000 7f89c7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89c80b58f0 0x7f89c8144fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:08.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.753+0000 7f89c7200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89c80b5d50 0x7f89c80cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:08.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.753+0000 7f89c7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89c81461b0 0x7f89c815bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:08.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.753+0000 7f89c7200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f89c80c59f0 con 0x7f89c80b5d50 2024-04-16T16:16:08.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.753+0000 7f89c7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89c8161230 con 0x7f89c80b58f0 2024-04-16T16:16:08.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.753+0000 7f89c7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f89c80c5cd0 con 0x7f89c81461b0 2024-04-16T16:16:08.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.753+0000 7f89bd400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89c80b5d50 0x7f89c80cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:08.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.753+0000 7f89bd400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89c80b5d50 0x7f89c80cc040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51280/0 (socket says 172.21.15.45:51280) 2024-04-16T16:16:08.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.753+0000 7f89bd400640 1 -- 172.21.15.45:0/555691853 learned_addr learned my addr 172.21.15.45:0/555691853 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:08.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.753+0000 7f89c5e00640 1 --2- 172.21.15.45:0/555691853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89c80b58f0 0x7f89c8144fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:08.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.754+0000 7f89c6800640 1 --2- 172.21.15.45:0/555691853 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89c81461b0 0x7f89c815bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:08.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.754+0000 7f89bd400640 1 -- 172.21.15.45:0/555691853 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89c81461b0 msgr2=0x7f89c815bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:08.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.754+0000 7f89bd400640 1 --2- 172.21.15.45:0/555691853 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89c81461b0 0x7f89c815bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.754+0000 7f89bd400640 1 -- 172.21.15.45:0/555691853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89c80b58f0 msgr2=0x7f89c8144fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:08.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.754+0000 7f89bd400640 1 --2- 172.21.15.45:0/555691853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89c80b58f0 0x7f89c8144fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.754+0000 7f89bd400640 1 -- 172.21.15.45:0/555691853 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c80b3750 con 0x7f89c80b5d50 2024-04-16T16:16:08.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.754+0000 7f89c6800640 1 --2- 172.21.15.45:0/555691853 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89c81461b0 0x7f89c815bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:08.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.754+0000 7f89bd400640 1 --2- 172.21.15.45:0/555691853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89c80b5d50 0x7f89c80cc040 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f89b000c430 tx=0x7f89b000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:08.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.754+0000 7f89bea00640 1 -- 172.21.15.45:0/555691853 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0011790 con 0x7f89c80b5d50 2024-04-16T16:16:08.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.754+0000 7f89bea00640 1 -- 172.21.15.45:0/555691853 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b0017070 con 0x7f89c80b5d50 2024-04-16T16:16:08.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.754+0000 7f89bea00640 1 -- 172.21.15.45:0/555691853 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0011bb0 con 0x7f89c80b5d50 2024-04-16T16:16:08.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.754+0000 7f89c7200640 1 -- 172.21.15.45:0/555691853 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c80b3a30 con 0x7f89c80b5d50 2024-04-16T16:16:08.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.755+0000 7f89c7200640 1 -- 172.21.15.45:0/555691853 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c80b44f0 con 0x7f89c80b5d50 2024-04-16T16:16:08.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.756+0000 7f89c7200640 1 -- 172.21.15.45:0/555691853 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c8075780 con 0x7f89c80b5d50 2024-04-16T16:16:08.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.757+0000 7f89bea00640 1 -- 172.21.15.45:0/555691853 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) ==== 90878+0+0 (secure 0 0 0) 0x7f89b001c460 con 0x7f89c80b5d50 2024-04-16T16:16:08.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.758+0000 7f89bea00640 1 --2- 172.21.15.45:0/555691853 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f89a406c740 0x7f89a406ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:08.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.758+0000 7f89bea00640 1 -- 172.21.15.45:0/555691853 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7f89b008be70 con 0x7f89c80b5d50 2024-04-16T16:16:08.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.758+0000 7f89c5e00640 1 --2- 172.21.15.45:0/555691853 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f89a406c740 0x7f89a406ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:08.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.760+0000 7f89c5e00640 1 --2- 172.21.15.45:0/555691853 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f89a406c740 0x7f89a406ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f89b8036000 tx=0x7f89b803b040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:08.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.762+0000 7f89bea00640 1 -- 172.21.15.45:0/555691853 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f89b0059de0 con 0x7f89c80b5d50 2024-04-16T16:16:08.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.968+0000 7f89c7200640 1 -- 172.21.15.45:0/555691853 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs2_metadata", "var": "pg_num"} v 0) -- 0x7f89c8072500 con 0x7f89c80b5d50 2024-04-16T16:16:08.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.968+0000 7f89bea00640 1 -- 172.21.15.45:0/555691853 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs2_metadata", "var": "pg_num"}]=0 v57) ==== 101+0+11 (secure 0 0 0) 0x7f89b00596e0 con 0x7f89c80b5d50 2024-04-16T16:16:08.968 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:16:08.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.971+0000 7f89c7200640 1 -- 172.21.15.45:0/555691853 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f89a406c740 msgr2=0x7f89a406ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:08.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.971+0000 7f89c7200640 1 --2- 172.21.15.45:0/555691853 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f89a406c740 0x7f89a406ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f89b8036000 tx=0x7f89b803b040 comp rx=0 tx=0).stop 2024-04-16T16:16:08.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.971+0000 7f89c7200640 1 -- 172.21.15.45:0/555691853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89c80b5d50 msgr2=0x7f89c80cc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:08.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.971+0000 7f89c7200640 1 --2- 172.21.15.45:0/555691853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89c80b5d50 0x7f89c80cc040 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f89b000c430 tx=0x7f89b000c8f0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.971+0000 7f89c7200640 1 -- 172.21.15.45:0/555691853 shutdown_connections 2024-04-16T16:16:08.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.971+0000 7f89c7200640 1 --2- 172.21.15.45:0/555691853 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f89a406c740 0x7f89a406ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.971+0000 7f89c7200640 1 --2- 172.21.15.45:0/555691853 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89c81461b0 0x7f89c815bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.971+0000 7f89c7200640 1 --2- 172.21.15.45:0/555691853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89c80b5d50 0x7f89c80cc040 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.971+0000 7f89c7200640 1 --2- 172.21.15.45:0/555691853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89c80b58f0 0x7f89c8144fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:08.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.971+0000 7f89c7200640 1 -- 172.21.15.45:0/555691853 >> 172.21.15.45:0/555691853 conn(0x7f89c806b930 msgr2=0x7f89c815cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:08.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.971+0000 7f89c7200640 1 -- 172.21.15.45:0/555691853 shutdown_connections 2024-04-16T16:16:08.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:08.971+0000 7f89c7200640 1 -- 172.21.15.45:0/555691853 wait complete. 2024-04-16T16:16:08.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs2_data pg_num 2024-04-16T16:16:09.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.223+0000 7f6cd7200640 1 Processor -- start 2024-04-16T16:16:09.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.223+0000 7f6cd7200640 1 -- start start 2024-04-16T16:16:09.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.224+0000 7f6cd7200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6cd80b58f0 0x7f6cd80b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.224+0000 7f6cd7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cd806e460 0x7f6cd806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.224+0000 7f6cd7200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cd8146590 con 0x7f6cd81461b0 2024-04-16T16:16:09.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.224+0000 7f6cd7200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6cd806dfd0 con 0x7f6cd80b58f0 2024-04-16T16:16:09.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.224+0000 7f6cd7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd80b6290 con 0x7f6cd806e460 2024-04-16T16:16:09.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.225+0000 7f6cd5e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6cd81461b0 0x7f6cd8160cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42454/0 (socket says 172.21.15.45:42454) 2024-04-16T16:16:09.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.225+0000 7f6cd5e00640 1 -- 172.21.15.45:0/544558462 learned_addr learned my addr 172.21.15.45:0/544558462 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:09.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.225+0000 7f6cd6800640 1 --2- 172.21.15.45:0/544558462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cd806e460 0x7f6cd806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.225+0000 7f6cd5400640 1 --2- 172.21.15.45:0/544558462 >> v2:172.21.15.103:3300/0 conn(0x7f6cd80b58f0 0x7f6cd80b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.225+0000 7f6cd4a00640 1 -- 172.21.15.45:0/544558462 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2807583972 0 0) 0x7f6cd8146590 con 0x7f6cd81461b0 2024-04-16T16:16:09.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.225+0000 7f6cd4a00640 1 -- 172.21.15.45:0/544558462 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cb8003520 con 0x7f6cd81461b0 2024-04-16T16:16:09.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.226+0000 7f6cd4a00640 1 -- 172.21.15.45:0/544558462 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1361268201 0 0) 0x7f6cb8003520 con 0x7f6cd81461b0 2024-04-16T16:16:09.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.226+0000 7f6cd4a00640 1 -- 172.21.15.45:0/544558462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cd806e460 msgr2=0x7f6cd806e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:09.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.226+0000 7f6cd4a00640 1 --2- 172.21.15.45:0/544558462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cd806e460 0x7f6cd806e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.226+0000 7f6cd4a00640 1 -- 172.21.15.45:0/544558462 >> v2:172.21.15.103:3300/0 conn(0x7f6cd80b58f0 msgr2=0x7f6cd80b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:09.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.226+0000 7f6cd4a00640 1 --2- 172.21.15.45:0/544558462 >> v2:172.21.15.103:3300/0 conn(0x7f6cd80b58f0 0x7f6cd80b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.226+0000 7f6cd4a00640 1 -- 172.21.15.45:0/544558462 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd80c5750 con 0x7f6cd81461b0 2024-04-16T16:16:09.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.226+0000 7f6cd4a00640 1 -- 172.21.15.45:0/544558462 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f6cc8001670 con 0x7f6cd81461b0 2024-04-16T16:16:09.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.226+0000 7f6cd4a00640 1 -- 172.21.15.45:0/544558462 >> v1:172.21.15.45:6789/0 conn(0x7f6cd81461b0 legacy=0x7f6cd8160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:09.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.226+0000 7f6cd4a00640 1 --2- 172.21.15.45:0/544558462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb80039b0 0x7f6cb8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.226+0000 7f6cd4a00640 1 --2- 172.21.15.45:0/544558462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cd806e460 0x7f6cb8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.226+0000 7f6cd4a00640 1 --2- 172.21.15.45:0/544558462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cb8004d40 0x7f6cb80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.226+0000 7f6cd4a00640 1 -- 172.21.15.45:0/544558462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb800b9c0 con 0x7f6cb80039b0 2024-04-16T16:16:09.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.226+0000 7f6cd4a00640 1 -- 172.21.15.45:0/544558462 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb800bb40 con 0x7f6cb8004d40 2024-04-16T16:16:09.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.226+0000 7f6cd4a00640 1 -- 172.21.15.45:0/544558462 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6cb800bcc0 con 0x7f6cd806e460 2024-04-16T16:16:09.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.226+0000 7f6cd5e00640 1 --2- 172.21.15.45:0/544558462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb80039b0 0x7f6cb8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.227+0000 7f6cd5400640 1 --2- 172.21.15.45:0/544558462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cd806e460 0x7f6cb8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.227+0000 7f6cd6800640 1 --2- 172.21.15.45:0/544558462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cb8004d40 0x7f6cb80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.227+0000 7f6cd5e00640 1 -- 172.21.15.45:0/544558462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cd806e460 msgr2=0x7f6cb8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:09.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.227+0000 7f6cd5e00640 1 --2- 172.21.15.45:0/544558462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cd806e460 0x7f6cb8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.227+0000 7f6cd5e00640 1 -- 172.21.15.45:0/544558462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cb8004d40 msgr2=0x7f6cb80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:09.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.227+0000 7f6cd5e00640 1 --2- 172.21.15.45:0/544558462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cb8004d40 0x7f6cb80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.227+0000 7f6cd5e00640 1 -- 172.21.15.45:0/544558462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb8003520 con 0x7f6cb80039b0 2024-04-16T16:16:09.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.228+0000 7f6cd5e00640 1 --2- 172.21.15.45:0/544558462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb80039b0 0x7f6cb8003dd0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8007fd0 tx=0x7f6cb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4bd2232a72f1464 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:09.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.228+0000 7f6cd4a00640 1 -- 172.21.15.45:0/544558462 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc8007d50 con 0x7f6cb80039b0 2024-04-16T16:16:09.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.228+0000 7f6cd4a00640 1 -- 172.21.15.45:0/544558462 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc8030070 con 0x7f6cb80039b0 2024-04-16T16:16:09.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.228+0000 7f6cd4a00640 1 -- 172.21.15.45:0/544558462 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc802cd40 con 0x7f6cb80039b0 2024-04-16T16:16:09.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.229+0000 7f6cd7200640 1 -- 172.21.15.45:0/544558462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb80039b0 msgr2=0x7f6cb8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:09.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.229+0000 7f6cd7200640 1 --2- 172.21.15.45:0/544558462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb80039b0 0x7f6cb8003dd0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8007fd0 tx=0x7f6cb8005830 comp rx=0 tx=0).stop 2024-04-16T16:16:09.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.229+0000 7f6cd6800640 1 -- 172.21.15.45:0/544558462 reap_dead start 2024-04-16T16:16:09.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.229+0000 7f6cd7200640 1 -- 172.21.15.45:0/544558462 shutdown_connections 2024-04-16T16:16:09.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.229+0000 7f6cd7200640 1 -- 172.21.15.45:0/544558462 >> 172.21.15.45:0/544558462 conn(0x7f6cd806b930 msgr2=0x7f6cd806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:09.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.229+0000 7f6cd7200640 1 -- 172.21.15.45:0/544558462 shutdown_connections 2024-04-16T16:16:09.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.229+0000 7f6cd7200640 1 -- 172.21.15.45:0/544558462 wait complete. 2024-04-16T16:16:09.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.231+0000 7f6cd7200640 1 Processor -- start 2024-04-16T16:16:09.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.231+0000 7f6cd7200640 1 -- start start 2024-04-16T16:16:09.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.232+0000 7f6cd7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cb80039b0 0x7f6cd8158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.232+0000 7f6cd7200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cd813e250 0x7f6cd813e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.232+0000 7f6cd7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cd813a3c0 0x7f6cd813a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.232+0000 7f6cd7200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd80c5cd0 con 0x7f6cd813e250 2024-04-16T16:16:09.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.232+0000 7f6cd7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd80c59f0 con 0x7f6cd813a3c0 2024-04-16T16:16:09.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.232+0000 7f6cd7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd80c5fb0 con 0x7f6cb80039b0 2024-04-16T16:16:09.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.232+0000 7f6cd5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cd813e250 0x7f6cd813e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.232+0000 7f6cd5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cd813e250 0x7f6cd813e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51306/0 (socket says 172.21.15.45:51306) 2024-04-16T16:16:09.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.233+0000 7f6cd5400640 1 -- 172.21.15.45:0/1191997850 learned_addr learned my addr 172.21.15.45:0/1191997850 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:09.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.233+0000 7f6cd5e00640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cb80039b0 0x7f6cd8158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.233+0000 7f6cd6800640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cd813a3c0 0x7f6cd813a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.233+0000 7f6cd5400640 1 -- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cb80039b0 msgr2=0x7f6cd8158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:09.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.233+0000 7f6cd5400640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cb80039b0 0x7f6cd8158920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.233+0000 7f6cd5400640 1 -- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cd813a3c0 msgr2=0x7f6cd813a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:09.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.233+0000 7f6cd5400640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cd813a3c0 0x7f6cd813a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.233+0000 7f6cd5400640 1 -- 172.21.15.45:0/1191997850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd813ad80 con 0x7f6cd813e250 2024-04-16T16:16:09.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.233+0000 7f6cd5e00640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cb80039b0 0x7f6cd8158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:09.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.233+0000 7f6cd6800640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cd813a3c0 0x7f6cd813a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:09.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.233+0000 7f6cd5400640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cd813e250 0x7f6cd813e6d0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0007f90 tx=0x7f6cc0011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:09.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.233+0000 7f6cce000640 1 -- 172.21.15.45:0/1191997850 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc001a020 con 0x7f6cd813e250 2024-04-16T16:16:09.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.234+0000 7f6cce000640 1 -- 172.21.15.45:0/1191997850 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc0016470 con 0x7f6cd813e250 2024-04-16T16:16:09.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.234+0000 7f6cce000640 1 -- 172.21.15.45:0/1191997850 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0016860 con 0x7f6cd813e250 2024-04-16T16:16:09.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.234+0000 7f6cd7200640 1 -- 172.21.15.45:0/1191997850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cd813c280 con 0x7f6cd813e250 2024-04-16T16:16:09.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.234+0000 7f6cd7200640 1 -- 172.21.15.45:0/1191997850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cd813c800 con 0x7f6cd813e250 2024-04-16T16:16:09.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.235+0000 7f6cd7200640 1 -- 172.21.15.45:0/1191997850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cd8075780 con 0x7f6cd813e250 2024-04-16T16:16:09.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.236+0000 7f6cce000640 1 -- 172.21.15.45:0/1191997850 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) ==== 90878+0+0 (secure 0 0 0) 0x7f6cc0005d10 con 0x7f6cd813e250 2024-04-16T16:16:09.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.237+0000 7f6cce000640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6cb406c740 0x7f6cb406ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.237+0000 7f6cce000640 1 -- 172.21.15.45:0/1191997850 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7f6cc008ce70 con 0x7f6cd813e250 2024-04-16T16:16:09.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.237+0000 7f6cd5e00640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6cb406c740 0x7f6cb406ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.239+0000 7f6cd5e00640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6cb406c740 0x7f6cb406ec00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8034000 tx=0x7f6cc8039040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:09.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.241+0000 7f6cce000640 1 -- 172.21.15.45:0/1191997850 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6cc0024030 con 0x7f6cd813e250 2024-04-16T16:16:09.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.448+0000 7f6cd7200640 1 -- 172.21.15.45:0/1191997850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs2_data", "var": "pg_num"} v 0) -- 0x7f6cd80c6280 con 0x7f6cd813e250 2024-04-16T16:16:09.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.449+0000 7f6cce000640 1 -- 172.21.15.45:0/1191997850 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs2_data", "var": "pg_num"}]=0 v57) ==== 97+0+11 (secure 0 0 0) 0x7f6cd813ad80 con 0x7f6cd813e250 2024-04-16T16:16:09.448 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:16:09.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.452+0000 7f6cd7200640 1 -- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6cb406c740 msgr2=0x7f6cb406ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:09.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.452+0000 7f6cd7200640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6cb406c740 0x7f6cb406ec00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8034000 tx=0x7f6cc8039040 comp rx=0 tx=0).stop 2024-04-16T16:16:09.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.452+0000 7f6cd7200640 1 -- 172.21.15.45:0/1191997850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cd813e250 msgr2=0x7f6cd813e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:09.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.452+0000 7f6cd7200640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cd813e250 0x7f6cd813e6d0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0007f90 tx=0x7f6cc0011660 comp rx=0 tx=0).stop 2024-04-16T16:16:09.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.453+0000 7f6cd7200640 1 -- 172.21.15.45:0/1191997850 shutdown_connections 2024-04-16T16:16:09.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.453+0000 7f6cd7200640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6cb406c740 0x7f6cb406ec00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.453+0000 7f6cd7200640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cd813a3c0 0x7f6cd813a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.453+0000 7f6cd7200640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cd813e250 0x7f6cd813e6d0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.453+0000 7f6cd7200640 1 --2- 172.21.15.45:0/1191997850 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6cb80039b0 0x7f6cd8158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.453+0000 7f6cd7200640 1 -- 172.21.15.45:0/1191997850 >> 172.21.15.45:0/1191997850 conn(0x7f6cd806b930 msgr2=0x7f6cd80b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:09.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.453+0000 7f6cd7200640 1 -- 172.21.15.45:0/1191997850 shutdown_connections 2024-04-16T16:16:09.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.453+0000 7f6cd7200640 1 -- 172.21.15.45:0/1191997850 wait complete. 2024-04-16T16:16:09.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:16:09.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.700+0000 7f7d87200640 1 Processor -- start 2024-04-16T16:16:09.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.700+0000 7f7d87200640 1 -- start start 2024-04-16T16:16:09.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.702+0000 7f7d87200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d880cbc80 0x7f7d880cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.702+0000 7f7d87200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d8815f1b0 con 0x7f7d880cb7d0 2024-04-16T16:16:09.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.702+0000 7f7d87200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d880cf280 con 0x7f7d8815e830 2024-04-16T16:16:09.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.702+0000 7f7d87200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7d880ce190 con 0x7f7d880cbc80 2024-04-16T16:16:09.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.702+0000 7f7d85400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7d880cb7d0 0x7f7d880cd620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42456/0 (socket says 172.21.15.45:42456) 2024-04-16T16:16:09.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.702+0000 7f7d85400640 1 -- 172.21.15.45:0/4102391390 learned_addr learned my addr 172.21.15.45:0/4102391390 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:09.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.703+0000 7f7d86800640 1 --2- 172.21.15.45:0/4102391390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d880cbc80 0x7f7d880cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.703+0000 7f7d84a00640 1 -- 172.21.15.45:0/4102391390 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1245076331 0 0) 0x7f7d74001110 con 0x7f7d880cb7d0 2024-04-16T16:16:09.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.703+0000 7f7d84a00640 1 -- 172.21.15.45:0/4102391390 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d6c003520 con 0x7f7d880cb7d0 2024-04-16T16:16:09.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.703+0000 7f7d84a00640 1 -- 172.21.15.45:0/4102391390 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1317422603 0 0) 0x7f7d6c003520 con 0x7f7d880cb7d0 2024-04-16T16:16:09.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.703+0000 7f7d84a00640 1 -- 172.21.15.45:0/4102391390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d880cbc80 msgr2=0x7f7d880cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:09.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.703+0000 7f7d84a00640 1 --2- 172.21.15.45:0/4102391390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d880cbc80 0x7f7d880cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.703+0000 7f7d84a00640 1 -- 172.21.15.45:0/4102391390 >> v1:172.21.15.103:6789/0 conn(0x7f7d8815e830 legacy=0x7f7d880ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:09.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.703+0000 7f7d84a00640 1 -- 172.21.15.45:0/4102391390 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d880ce470 con 0x7f7d880cb7d0 2024-04-16T16:16:09.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.703+0000 7f7d84a00640 1 -- 172.21.15.45:0/4102391390 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f7d74001840 con 0x7f7d880cb7d0 2024-04-16T16:16:09.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.703+0000 7f7d84a00640 1 -- 172.21.15.45:0/4102391390 >> v1:172.21.15.45:6789/0 conn(0x7f7d880cb7d0 legacy=0x7f7d880cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:09.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.703+0000 7f7d84a00640 1 --2- 172.21.15.45:0/4102391390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d6c0039e0 0x7f7d6c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.704+0000 7f7d84a00640 1 --2- 172.21.15.45:0/4102391390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d6c004800 0x7f7d6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.704+0000 7f7d84a00640 1 --2- 172.21.15.45:0/4102391390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d880cbc80 0x7f7d6c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.704+0000 7f7d84a00640 1 -- 172.21.15.45:0/4102391390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7d880ce190 con 0x7f7d6c004800 2024-04-16T16:16:09.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.704+0000 7f7d84a00640 1 -- 172.21.15.45:0/4102391390 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c00baf0 con 0x7f7d6c0039e0 2024-04-16T16:16:09.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.704+0000 7f7d84a00640 1 -- 172.21.15.45:0/4102391390 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7d6c00bcf0 con 0x7f7d880cbc80 2024-04-16T16:16:09.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.704+0000 7f7d85400640 1 --2- 172.21.15.45:0/4102391390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d6c004800 0x7f7d6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.704+0000 7f7d85e00640 1 --2- 172.21.15.45:0/4102391390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d6c0039e0 0x7f7d6c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.704+0000 7f7d86800640 1 --2- 172.21.15.45:0/4102391390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d880cbc80 0x7f7d6c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.704+0000 7f7d85400640 1 -- 172.21.15.45:0/4102391390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d880cbc80 msgr2=0x7f7d6c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:09.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.704+0000 7f7d85400640 1 --2- 172.21.15.45:0/4102391390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d880cbc80 0x7f7d6c005200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.704+0000 7f7d85400640 1 -- 172.21.15.45:0/4102391390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d6c0039e0 msgr2=0x7f7d6c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:09.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.704+0000 7f7d85400640 1 --2- 172.21.15.45:0/4102391390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d6c0039e0 0x7f7d6c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.704+0000 7f7d85400640 1 -- 172.21.15.45:0/4102391390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6c003520 con 0x7f7d6c004800 2024-04-16T16:16:09.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.706+0000 7f7d85400640 1 --2- 172.21.15.45:0/4102391390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d6c004800 0x7f7d6c004c80 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400c4c0 tx=0x7f7d6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31da3f2d179301a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:09.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.706+0000 7f7d84a00640 1 -- 172.21.15.45:0/4102391390 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7400c0e0 con 0x7f7d6c004800 2024-04-16T16:16:09.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.706+0000 7f7d84a00640 1 -- 172.21.15.45:0/4102391390 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d74030070 con 0x7f7d6c004800 2024-04-16T16:16:09.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.706+0000 7f7d84a00640 1 -- 172.21.15.45:0/4102391390 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d740058d0 con 0x7f7d6c004800 2024-04-16T16:16:09.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.707+0000 7f7d87200640 1 -- 172.21.15.45:0/4102391390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d6c004800 msgr2=0x7f7d6c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:09.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.707+0000 7f7d87200640 1 --2- 172.21.15.45:0/4102391390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d6c004800 0x7f7d6c004c80 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400c4c0 tx=0x7f7d6c005830 comp rx=0 tx=0).stop 2024-04-16T16:16:09.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.707+0000 7f7d86800640 1 -- 172.21.15.45:0/4102391390 reap_dead start 2024-04-16T16:16:09.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.707+0000 7f7d87200640 1 -- 172.21.15.45:0/4102391390 shutdown_connections 2024-04-16T16:16:09.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.707+0000 7f7d87200640 1 -- 172.21.15.45:0/4102391390 >> 172.21.15.45:0/4102391390 conn(0x7f7d8806b930 msgr2=0x7f7d8806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:09.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.707+0000 7f7d87200640 1 -- 172.21.15.45:0/4102391390 shutdown_connections 2024-04-16T16:16:09.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.707+0000 7f7d87200640 1 -- 172.21.15.45:0/4102391390 wait complete. 2024-04-16T16:16:09.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.709+0000 7f7d87200640 1 Processor -- start 2024-04-16T16:16:09.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.709+0000 7f7d87200640 1 -- start start 2024-04-16T16:16:09.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.710+0000 7f7d87200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d6c004800 0x7f7d88157ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.710+0000 7f7d87200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d88159350 0x7f7d88149130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.710+0000 7f7d87200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d881589c0 0x7f7d88160a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.710+0000 7f7d87200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8815ec10 con 0x7f7d881589c0 2024-04-16T16:16:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.710+0000 7f7d87200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7d880cb2b0 con 0x7f7d88159350 2024-04-16T16:16:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.710+0000 7f7d87200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7d880cac30 con 0x7f7d6c004800 2024-04-16T16:16:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.710+0000 7f7d86800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d881589c0 0x7f7d88160a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.710+0000 7f7d86800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d881589c0 0x7f7d88160a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51320/0 (socket says 172.21.15.45:51320) 2024-04-16T16:16:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.710+0000 7f7d86800640 1 -- 172.21.15.45:0/1079539920 learned_addr learned my addr 172.21.15.45:0/1079539920 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.711+0000 7f7d85e00640 1 --2- 172.21.15.45:0/1079539920 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d6c004800 0x7f7d88157ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.711+0000 7f7d85400640 1 --2- 172.21.15.45:0/1079539920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d88159350 0x7f7d88149130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.711+0000 7f7d86800640 1 -- 172.21.15.45:0/1079539920 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d6c004800 msgr2=0x7f7d88157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.711+0000 7f7d86800640 1 --2- 172.21.15.45:0/1079539920 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d6c004800 0x7f7d88157ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.711+0000 7f7d86800640 1 -- 172.21.15.45:0/1079539920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d88159350 msgr2=0x7f7d88149130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.711+0000 7f7d86800640 1 --2- 172.21.15.45:0/1079539920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d88159350 0x7f7d88149130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.711+0000 7f7d86800640 1 -- 172.21.15.45:0/1079539920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d881598d0 con 0x7f7d881589c0 2024-04-16T16:16:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.711+0000 7f7d85400640 1 --2- 172.21.15.45:0/1079539920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d88159350 0x7f7d88149130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.711+0000 7f7d86800640 1 --2- 172.21.15.45:0/1079539920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d881589c0 0x7f7d88160a10 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7d88071ed0 tx=0x7f7d8000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.711+0000 7f7d7e000640 1 -- 172.21.15.45:0/1079539920 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8001c070 con 0x7f7d881589c0 2024-04-16T16:16:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.711+0000 7f7d7e000640 1 -- 172.21.15.45:0/1079539920 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d80017400 con 0x7f7d881589c0 2024-04-16T16:16:09.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.711+0000 7f7d7e000640 1 -- 172.21.15.45:0/1079539920 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d800177f0 con 0x7f7d881589c0 2024-04-16T16:16:09.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.711+0000 7f7d87200640 1 -- 172.21.15.45:0/1079539920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d88156fe0 con 0x7f7d881589c0 2024-04-16T16:16:09.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.712+0000 7f7d87200640 1 -- 172.21.15.45:0/1079539920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d881575a0 con 0x7f7d881589c0 2024-04-16T16:16:09.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.713+0000 7f7d87200640 1 -- 172.21.15.45:0/1079539920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d88156e00 con 0x7f7d881589c0 2024-04-16T16:16:09.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.714+0000 7f7d7e000640 1 -- 172.21.15.45:0/1079539920 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) ==== 90878+0+0 (secure 0 0 0) 0x7f7d8001f070 con 0x7f7d881589c0 2024-04-16T16:16:09.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.715+0000 7f7d7e000640 1 --2- 172.21.15.45:0/1079539920 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f7d6806c770 0x7f7d6806ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:09.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.715+0000 7f7d7e000640 1 -- 172.21.15.45:0/1079539920 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7f7d8008c050 con 0x7f7d881589c0 2024-04-16T16:16:09.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.715+0000 7f7d85e00640 1 --2- 172.21.15.45:0/1079539920 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f7d6806c770 0x7f7d6806ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:09.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.717+0000 7f7d85e00640 1 --2- 172.21.15.45:0/1079539920 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f7d6806c770 0x7f7d6806ec30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7d78007fd0 tx=0x7f7d78009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:09.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.719+0000 7f7d7e000640 1 -- 172.21.15.45:0/1079539920 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7d88156e00 con 0x7f7d881589c0 2024-04-16T16:16:09.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.762+0000 7f7d7e000640 1 -- 172.21.15.45:0/1079539920 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 14) ==== 91188+0+0 (secure 0 0 0) 0x7f7d80055fe0 con 0x7f7d881589c0 2024-04-16T16:16:09.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.945+0000 7f7d87200640 1 -- 172.21.15.45:0/1079539920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7d88075780 con 0x7f7d881589c0 2024-04-16T16:16:09.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.946+0000 7f7d7e000640 1 -- 172.21.15.45:0/1079539920 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 22 entries v57) ==== 81+0+1316 (secure 0 0 0) 0x7f7d80012070 con 0x7f7d881589c0 2024-04-16T16:16:09.945 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/349066714 2024-04-17T16:16:07.016480+0000 2024-04-16T16:16:09.945 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/388389014 2024-04-17T16:16:07.016480+0000 2024-04-16T16:16:09.945 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/589483918 2024-04-17T16:16:07.016480+0000 2024-04-16T16:16:09.945 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3353543342 2024-04-17T16:16:07.016480+0000 2024-04-16T16:16:09.945 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/280591707 2024-04-17T16:16:07.016480+0000 2024-04-16T16:16:09.945 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3310077212 2024-04-17T16:16:07.016480+0000 2024-04-16T16:16:09.945 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6833/368835060 2024-04-17T16:15:41.256698+0000 2024-04-16T16:16:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2318159422 2024-04-17T16:15:01.083409+0000 2024-04-16T16:16:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6832/368835060 2024-04-17T16:15:41.256698+0000 2024-04-16T16:16:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/2711977587 2024-04-17T16:15:01.083409+0000 2024-04-16T16:16:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/2711977587 2024-04-17T16:15:01.083409+0000 2024-04-16T16:16:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6833/2956262988 2024-04-17T16:15:07.318380+0000 2024-04-16T16:16:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3554107853 2024-04-17T16:15:01.083409+0000 2024-04-16T16:16:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6835/2956262988 2024-04-17T16:15:07.318380+0000 2024-04-16T16:16:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6833/2276311244 2024-04-17T16:15:20.691128+0000 2024-04-16T16:16:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/280591707 2024-04-17T16:16:07.016480+0000 2024-04-16T16:16:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3010240888 2024-04-17T16:15:01.083409+0000 2024-04-16T16:16:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6834/88032513 2024-04-17T16:15:41.221894+0000 2024-04-16T16:16:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6835/2276311244 2024-04-17T16:15:20.691128+0000 2024-04-16T16:16:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/644374842 2024-04-17T16:16:07.016480+0000 2024-04-16T16:16:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2679792854 2024-04-17T16:15:01.083409+0000 2024-04-16T16:16:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6835/88032513 2024-04-17T16:15:41.221894+0000 2024-04-16T16:16:09.947 INFO:teuthology.orchestra.run.smithi045.stderr:listed 22 entries 2024-04-16T16:16:09.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.949+0000 7f7d87200640 1 -- 172.21.15.45:0/1079539920 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f7d6806c770 msgr2=0x7f7d6806ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:09.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.949+0000 7f7d87200640 1 --2- 172.21.15.45:0/1079539920 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f7d6806c770 0x7f7d6806ec30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7d78007fd0 tx=0x7f7d78009040 comp rx=0 tx=0).stop 2024-04-16T16:16:09.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.949+0000 7f7d87200640 1 -- 172.21.15.45:0/1079539920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d881589c0 msgr2=0x7f7d88160a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:09.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.949+0000 7f7d87200640 1 --2- 172.21.15.45:0/1079539920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d881589c0 0x7f7d88160a10 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7d88071ed0 tx=0x7f7d8000f4a0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.949+0000 7f7d87200640 1 -- 172.21.15.45:0/1079539920 shutdown_connections 2024-04-16T16:16:09.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.949+0000 7f7d87200640 1 --2- 172.21.15.45:0/1079539920 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f7d6806c770 0x7f7d6806ec30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.949+0000 7f7d87200640 1 --2- 172.21.15.45:0/1079539920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d881589c0 0x7f7d88160a10 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.949+0000 7f7d87200640 1 --2- 172.21.15.45:0/1079539920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d88159350 0x7f7d88149130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.949+0000 7f7d87200640 1 --2- 172.21.15.45:0/1079539920 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d6c004800 0x7f7d88157ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:09.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.949+0000 7f7d87200640 1 -- 172.21.15.45:0/1079539920 >> 172.21.15.45:0/1079539920 conn(0x7f7d8806b930 msgr2=0x7f7d88161940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:09.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.949+0000 7f7d87200640 1 -- 172.21.15.45:0/1079539920 shutdown_connections 2024-04-16T16:16:09.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:09.949+0000 7f7d87200640 1 -- 172.21.15.45:0/1079539920 wait complete. 2024-04-16T16:16:09.968 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-16T16:16:09.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:09.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-04-16T16:16:10.119 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:16:10.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:10.120 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-16T16:16:10.172 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:16:10.172 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:16:10.172 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:16:10.212 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:16:10.212 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:16:10.212 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:16:10.212 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:16:10.251 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:16:10.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:16:10.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.460+0000 7f455e800640 1 Processor -- start 2024-04-16T16:16:10.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.461+0000 7f455e800640 1 -- start start 2024-04-16T16:16:10.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.462+0000 7f455e800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4558161e90 0x7f45580cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:10.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.462+0000 7f455e800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4558162270 0x7f45580d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:10.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.462+0000 7f455e800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45580d3080 con 0x7f45580d52d0 2024-04-16T16:16:10.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.462+0000 7f455e800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f45580d2e30 con 0x7f4558162270 2024-04-16T16:16:10.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.462+0000 7f455e800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f45580d3360 con 0x7f4558161e90 2024-04-16T16:16:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.462+0000 7f455ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4558162270 0x7f45580d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.463+0000 7f455ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4558162270 0x7f45580d3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51330/0 (socket says 172.21.15.45:51330) 2024-04-16T16:16:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.463+0000 7f455ca00640 1 -- 172.21.15.45:0/1984875786 learned_addr learned my addr 172.21.15.45:0/1984875786 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.463+0000 7f455d400640 1 --2- 172.21.15.45:0/1984875786 >> v2:172.21.15.103:3300/0 conn(0x7f4558161e90 0x7f45580cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.463+0000 7f455ca00640 1 -- 172.21.15.45:0/1984875786 >> v2:172.21.15.103:3300/0 conn(0x7f4558161e90 msgr2=0x7f45580cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.463+0000 7f455ca00640 1 --2- 172.21.15.45:0/1984875786 >> v2:172.21.15.103:3300/0 conn(0x7f4558161e90 0x7f45580cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.463+0000 7f455ca00640 1 -- 172.21.15.45:0/1984875786 >> v1:172.21.15.45:6789/0 conn(0x7f45580d52d0 legacy=0x7f455806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.463+0000 7f455ca00640 1 -- 172.21.15.45:0/1984875786 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45580d3640 con 0x7f4558162270 2024-04-16T16:16:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.464+0000 7f455ca00640 1 --2- 172.21.15.45:0/1984875786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4558162270 0x7f45580d3e40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f4544009870 tx=0x7f454402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5bcb6ed7a5676f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.465+0000 7f4557e00640 1 -- 172.21.15.45:0/1984875786 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4544034070 con 0x7f4558162270 2024-04-16T16:16:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.465+0000 7f4557e00640 1 -- 172.21.15.45:0/1984875786 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4544030860 con 0x7f4558162270 2024-04-16T16:16:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.465+0000 7f4557e00640 1 -- 172.21.15.45:0/1984875786 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4544030c30 con 0x7f4558162270 2024-04-16T16:16:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.465+0000 7f455e800640 1 -- 172.21.15.45:0/1984875786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4558162270 msgr2=0x7f45580d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.465+0000 7f455e800640 1 --2- 172.21.15.45:0/1984875786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4558162270 0x7f45580d3e40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f4544009870 tx=0x7f454402b9a0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.465+0000 7f455e800640 1 -- 172.21.15.45:0/1984875786 shutdown_connections 2024-04-16T16:16:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.465+0000 7f455e800640 1 --2- 172.21.15.45:0/1984875786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4558162270 0x7f45580d3e40 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.465+0000 7f455e800640 1 --2- 172.21.15.45:0/1984875786 >> v2:172.21.15.103:3300/0 conn(0x7f4558161e90 0x7f45580cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.465+0000 7f455e800640 1 -- 172.21.15.45:0/1984875786 >> 172.21.15.45:0/1984875786 conn(0x7f45580d1940 msgr2=0x7f45580d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.466+0000 7f455e800640 1 -- 172.21.15.45:0/1984875786 shutdown_connections 2024-04-16T16:16:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.466+0000 7f455e800640 1 -- 172.21.15.45:0/1984875786 wait complete. 2024-04-16T16:16:10.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.467+0000 7f455e800640 1 Processor -- start 2024-04-16T16:16:10.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.468+0000 7f455e800640 1 -- start start 2024-04-16T16:16:10.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.469+0000 7f455e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45580d52d0 0x7f45581f7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:10.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.469+0000 7f455e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4558161e90 0x7f45580c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:10.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.469+0000 7f455e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4558162270 0x7f45580c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:10.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.469+0000 7f455e800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f455806d550 con 0x7f4558162270 2024-04-16T16:16:10.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.469+0000 7f455e800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f45580cfd10 con 0x7f45580d52d0 2024-04-16T16:16:10.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.469+0000 7f455e800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f455806d6d0 con 0x7f4558161e90 2024-04-16T16:16:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.469+0000 7f455de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4558162270 0x7f45580c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.469+0000 7f455de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4558162270 0x7f45580c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51340/0 (socket says 172.21.15.45:51340) 2024-04-16T16:16:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.469+0000 7f455de00640 1 -- 172.21.15.45:0/4288958476 learned_addr learned my addr 172.21.15.45:0/4288958476 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.469+0000 7f455ca00640 1 --2- 172.21.15.45:0/4288958476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4558161e90 0x7f45580c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.469+0000 7f455d400640 1 --2- 172.21.15.45:0/4288958476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45580d52d0 0x7f45581f7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.470+0000 7f455de00640 1 -- 172.21.15.45:0/4288958476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4558161e90 msgr2=0x7f45580c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.470+0000 7f455de00640 1 --2- 172.21.15.45:0/4288958476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4558161e90 0x7f45580c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.470+0000 7f455de00640 1 -- 172.21.15.45:0/4288958476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45580d52d0 msgr2=0x7f45581f7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.470+0000 7f455de00640 1 --2- 172.21.15.45:0/4288958476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45580d52d0 0x7f45581f7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.470+0000 7f455de00640 1 -- 172.21.15.45:0/4288958476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45580c93e0 con 0x7f4558162270 2024-04-16T16:16:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.470+0000 7f455de00640 1 --2- 172.21.15.45:0/4288958476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4558162270 0x7f45580c9a90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f455000c680 tx=0x7f455000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.470+0000 7f4555600640 1 -- 172.21.15.45:0/4288958476 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4550003b70 con 0x7f4558162270 2024-04-16T16:16:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.470+0000 7f4555600640 1 -- 172.21.15.45:0/4288958476 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4550003d10 con 0x7f4558162270 2024-04-16T16:16:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.470+0000 7f455e800640 1 -- 172.21.15.45:0/4288958476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45580c96c0 con 0x7f4558162270 2024-04-16T16:16:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.470+0000 7f4555600640 1 -- 172.21.15.45:0/4288958476 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4550005310 con 0x7f4558162270 2024-04-16T16:16:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.471+0000 7f455e800640 1 -- 172.21.15.45:0/4288958476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f455815ca50 con 0x7f4558162270 2024-04-16T16:16:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.472+0000 7f455e800640 1 -- 172.21.15.45:0/4288958476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4558075780 con 0x7f4558162270 2024-04-16T16:16:10.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.473+0000 7f4555600640 1 -- 172.21.15.45:0/4288958476 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 14) ==== 91188+0+0 (secure 0 0 0) 0x7f45500054b0 con 0x7f4558162270 2024-04-16T16:16:10.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.474+0000 7f4555600640 1 --2- 172.21.15.45:0/4288958476 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f453806caa0 0x7f453806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:10.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.474+0000 7f4555600640 1 -- 172.21.15.45:0/4288958476 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7f455008c400 con 0x7f4558162270 2024-04-16T16:16:10.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.474+0000 7f455d400640 1 --2- 172.21.15.45:0/4288958476 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f453806caa0 0x7f453806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:10.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.476+0000 7f455d400640 1 --2- 172.21.15.45:0/4288958476 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f453806caa0 0x7f453806ef60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f454c006fd0 tx=0x7f454c008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:10.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.479+0000 7f4555600640 1 -- 172.21.15.45:0/4288958476 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4550059e70 con 0x7f4558162270 2024-04-16T16:16:10.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.726+0000 7f455e800640 1 -- 172.21.15.45:0/4288958476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45580b75a0 con 0x7f4558162270 2024-04-16T16:16:10.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.728+0000 7f4555600640 1 -- 172.21.15.45:0/4288958476 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6491 (secure 0 0 0) 0x7f4550059770 con 0x7f4558162270 2024-04-16T16:16:10.727 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:10.727 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":40,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/1497040472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1497040472},{"type":"v1","addr":"172.21.15.45:6834","nonce":1497040472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":4907,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/93057991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":93057991},{"type":"v1","addr":"172.21.15.45:6835","nonce":93057991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:14.878986+0000","modified":"2024-04-16T16:15:44.272257+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5188":{"gid":5188,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.103:6836/2626302738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2626302738},{"type":"v1","addr":"172.21.15.103:6836","nonce":2626302738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"fs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5188,"qdb_cluster":[5188]},"id":3},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:35.289901+0000","modified":"2024-04-16T16:15:44.272258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"b","rank":0,"incarnation":35,"state":"up:active","state_seq":4,"addr":"172.21.15.103:6837/1576112884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1576112884},{"type":"v1","addr":"172.21.15.103:6837","nonce":1576112884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"fs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":4}]} 2024-04-16T16:16:10.727 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 40 2024-04-16T16:16:10.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.730+0000 7f455e800640 1 -- 172.21.15.45:0/4288958476 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f453806caa0 msgr2=0x7f453806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:10.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.730+0000 7f455e800640 1 --2- 172.21.15.45:0/4288958476 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f453806caa0 0x7f453806ef60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f454c006fd0 tx=0x7f454c008040 comp rx=0 tx=0).stop 2024-04-16T16:16:10.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.730+0000 7f455e800640 1 -- 172.21.15.45:0/4288958476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4558162270 msgr2=0x7f45580c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:10.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.730+0000 7f455e800640 1 --2- 172.21.15.45:0/4288958476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4558162270 0x7f45580c9a90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f455000c680 tx=0x7f455000cb40 comp rx=0 tx=0).stop 2024-04-16T16:16:10.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.731+0000 7f455e800640 1 -- 172.21.15.45:0/4288958476 shutdown_connections 2024-04-16T16:16:10.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.731+0000 7f455e800640 1 --2- 172.21.15.45:0/4288958476 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f453806caa0 0x7f453806ef60 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.731+0000 7f455e800640 1 --2- 172.21.15.45:0/4288958476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4558162270 0x7f45580c9a90 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.731+0000 7f455e800640 1 --2- 172.21.15.45:0/4288958476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4558161e90 0x7f45580c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.731+0000 7f455e800640 1 --2- 172.21.15.45:0/4288958476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45580d52d0 0x7f45581f7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.731+0000 7f455e800640 1 -- 172.21.15.45:0/4288958476 >> 172.21.15.45:0/4288958476 conn(0x7f45580d1940 msgr2=0x7f45580b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:10.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.731+0000 7f455e800640 1 -- 172.21.15.45:0/4288958476 shutdown_connections 2024-04-16T16:16:10.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.731+0000 7f455e800640 1 -- 172.21.15.45:0/4288958476 wait complete. 2024-04-16T16:16:10.750 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:16:10.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.976+0000 7f4afc600640 1 Processor -- start 2024-04-16T16:16:10.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.976+0000 7f4afc600640 1 -- start start 2024-04-16T16:16:10.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.978+0000 7f4afc600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4af40d0e70 0x7f4af40d1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:10.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.978+0000 7f4afc600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4af415cd40 0x7f4af40d0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:10.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.978+0000 7f4afc600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4af4160560 0x7f4af4160990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:10.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.978+0000 7f4afc600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f4af40d1790 con 0x7f4af415cd40 2024-04-16T16:16:10.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.978+0000 7f4afc600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4af415d1e0 con 0x7f4af4160560 2024-04-16T16:16:10.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.978+0000 7f4afc600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4af415d4c0 con 0x7f4af40d0e70 2024-04-16T16:16:10.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.978+0000 7f4af9200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4af415cd40 0x7f4af40d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:10.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.978+0000 7f4af9200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4af415cd40 0x7f4af40d0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51348/0 (socket says 172.21.15.45:51348) 2024-04-16T16:16:10.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.978+0000 7f4af9200640 1 -- 172.21.15.45:0/3370086740 learned_addr learned my addr 172.21.15.45:0/3370086740 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:10.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.978+0000 7f4af9c00640 1 --2- 172.21.15.45:0/3370086740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4af40d0e70 0x7f4af40d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:10.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.979+0000 7f4af9200640 1 -- 172.21.15.45:0/3370086740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4af40d0e70 msgr2=0x7f4af40d1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:10.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.979+0000 7f4afa600640 1 --2- 172.21.15.45:0/3370086740 >> v2:172.21.15.103:3300/0 conn(0x7f4af4160560 0x7f4af4160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:10.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.979+0000 7f4af9200640 1 --2- 172.21.15.45:0/3370086740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4af40d0e70 0x7f4af40d1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.979+0000 7f4af9200640 1 -- 172.21.15.45:0/3370086740 >> v2:172.21.15.103:3300/0 conn(0x7f4af4160560 msgr2=0x7f4af4160990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.979+0000 7f4af9200640 1 --2- 172.21.15.45:0/3370086740 >> v2:172.21.15.103:3300/0 conn(0x7f4af4160560 0x7f4af4160990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.979+0000 7f4af9200640 1 -- 172.21.15.45:0/3370086740 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af415d7a0 con 0x7f4af415cd40 2024-04-16T16:16:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.979+0000 7f4af9c00640 1 --2- 172.21.15.45:0/3370086740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4af40d0e70 0x7f4af40d1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.979+0000 7f4afa600640 1 --2- 172.21.15.45:0/3370086740 >> v2:172.21.15.103:3300/0 conn(0x7f4af4160560 0x7f4af4160990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.980+0000 7f4af9200640 1 --2- 172.21.15.45:0/3370086740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af415cd40 0x7f4af40d0460 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4af4070300 tx=0x7f4ae002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=832575d41421dd59 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.980+0000 7f4af3e00640 1 -- 172.21.15.45:0/3370086740 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae002ca70 con 0x7f4af415cd40 2024-04-16T16:16:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.980+0000 7f4af3e00640 1 -- 172.21.15.45:0/3370086740 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ae002cc10 con 0x7f4af415cd40 2024-04-16T16:16:10.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.980+0000 7f4af3e00640 1 -- 172.21.15.45:0/3370086740 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae00343f0 con 0x7f4af415cd40 2024-04-16T16:16:10.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.981+0000 7f4afc600640 1 -- 172.21.15.45:0/3370086740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af415cd40 msgr2=0x7f4af40d0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:10.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.981+0000 7f4afc600640 1 --2- 172.21.15.45:0/3370086740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af415cd40 0x7f4af40d0460 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4af4070300 tx=0x7f4ae002bed0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.981+0000 7f4afc600640 1 -- 172.21.15.45:0/3370086740 shutdown_connections 2024-04-16T16:16:10.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.981+0000 7f4afc600640 1 --2- 172.21.15.45:0/3370086740 >> v2:172.21.15.103:3300/0 conn(0x7f4af4160560 0x7f4af4160990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.981+0000 7f4afc600640 1 --2- 172.21.15.45:0/3370086740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af415cd40 0x7f4af40d0460 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.981+0000 7f4afc600640 1 --2- 172.21.15.45:0/3370086740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4af40d0e70 0x7f4af40d1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.981+0000 7f4afc600640 1 -- 172.21.15.45:0/3370086740 >> 172.21.15.45:0/3370086740 conn(0x7f4af40d2890 msgr2=0x7f4af40d2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:10.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.981+0000 7f4afc600640 1 -- 172.21.15.45:0/3370086740 shutdown_connections 2024-04-16T16:16:10.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.981+0000 7f4afc600640 1 -- 172.21.15.45:0/3370086740 wait complete. 2024-04-16T16:16:10.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.983+0000 7f4afc600640 1 Processor -- start 2024-04-16T16:16:10.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.983+0000 7f4afc600640 1 -- start start 2024-04-16T16:16:10.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.984+0000 7f4afc600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af40d0e70 0x7f4af41f5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:10.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.984+0000 7f4afc600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4af415cd40 0x7f4af40cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:10.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.984+0000 7f4afc600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af4160560 0x7f4af41f5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:10.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.984+0000 7f4afc600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4af409b970 con 0x7f4af40d0e70 2024-04-16T16:16:10.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.984+0000 7f4afc600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4af409b7f0 con 0x7f4af4160560 2024-04-16T16:16:10.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.984+0000 7f4afc600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4af40d09a0 con 0x7f4af415cd40 2024-04-16T16:16:10.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.984+0000 7f4af9c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af40d0e70 0x7f4af41f5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:10.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.984+0000 7f4af9c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af40d0e70 0x7f4af41f5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51364/0 (socket says 172.21.15.45:51364) 2024-04-16T16:16:10.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.984+0000 7f4af9c00640 1 -- 172.21.15.45:0/3583783721 learned_addr learned my addr 172.21.15.45:0/3583783721 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.985+0000 7f4af9200640 1 --2- 172.21.15.45:0/3583783721 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4af415cd40 0x7f4af40cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.985+0000 7f4afa600640 1 --2- 172.21.15.45:0/3583783721 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af4160560 0x7f4af41f5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.985+0000 7f4af9c00640 1 -- 172.21.15.45:0/3583783721 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4af415cd40 msgr2=0x7f4af40cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.985+0000 7f4af9c00640 1 --2- 172.21.15.45:0/3583783721 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4af415cd40 0x7f4af40cb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.985+0000 7f4af9c00640 1 -- 172.21.15.45:0/3583783721 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af4160560 msgr2=0x7f4af41f5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.985+0000 7f4af9c00640 1 --2- 172.21.15.45:0/3583783721 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af4160560 0x7f4af41f5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.985+0000 7f4af9c00640 1 -- 172.21.15.45:0/3583783721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af40b5780 con 0x7f4af40d0e70 2024-04-16T16:16:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.985+0000 7f4af9c00640 1 --2- 172.21.15.45:0/3583783721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af40d0e70 0x7f4af41f5b00 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4ae400c7f0 tx=0x7f4ae400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.986+0000 7f4af1600640 1 -- 172.21.15.45:0/3583783721 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4015070 con 0x7f4af40d0e70 2024-04-16T16:16:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.986+0000 7f4af1600640 1 -- 172.21.15.45:0/3583783721 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ae4007a30 con 0x7f4af40d0e70 2024-04-16T16:16:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.986+0000 7f4af1600640 1 -- 172.21.15.45:0/3583783721 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4007e00 con 0x7f4af40d0e70 2024-04-16T16:16:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.986+0000 7f4afc600640 1 -- 172.21.15.45:0/3583783721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4af40b5a60 con 0x7f4af40d0e70 2024-04-16T16:16:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.986+0000 7f4afc600640 1 -- 172.21.15.45:0/3583783721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4af40cb820 con 0x7f4af40d0e70 2024-04-16T16:16:10.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.987+0000 7f4afc600640 1 -- 172.21.15.45:0/3583783721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4af4075780 con 0x7f4af40d0e70 2024-04-16T16:16:10.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.988+0000 7f4af1600640 1 -- 172.21.15.45:0/3583783721 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 14) ==== 91188+0+0 (secure 0 0 0) 0x7f4ae4018070 con 0x7f4af40d0e70 2024-04-16T16:16:10.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.989+0000 7f4af1600640 1 --2- 172.21.15.45:0/3583783721 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4ad406caa0 0x7f4ad406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:10.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.989+0000 7f4af1600640 1 -- 172.21.15.45:0/3583783721 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7f4ae408c560 con 0x7f4af40d0e70 2024-04-16T16:16:10.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.990+0000 7f4af9200640 1 --2- 172.21.15.45:0/3583783721 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4ad406caa0 0x7f4ad406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:10.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.991+0000 7f4af9200640 1 --2- 172.21.15.45:0/3583783721 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4ad406caa0 0x7f4ad406ef60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0035000 tx=0x7f4ae0030000 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:10.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:10.994+0000 7f4af1600640 1 -- 172.21.15.45:0/3583783721 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ae4059f60 con 0x7f4af40d0e70 2024-04-16T16:16:11.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.199+0000 7f4afc600640 1 -- 172.21.15.45:0/3583783721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4af40d3710 con 0x7f4af40d0e70 2024-04-16T16:16:11.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.203+0000 7f4af1600640 1 -- 172.21.15.45:0/3583783721 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+21917 (secure 0 0 0) 0x7f4ae4059860 con 0x7f4af40d0e70 2024-04-16T16:16:11.203 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:11.203 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":57,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:16:07.016528+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"fs1_metadata","create_time":"2024-04-16T16:15:13.250782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"fs1_data","create_time":"2024-04-16T16:15:13.856986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"fs2_metadata","create_time":"2024-04-16T16:15:33.215042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"fs2_data","create_time":"2024-04-16T16:15:34.271426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.2c","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/349066714":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/388389014":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/589483918":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3353543342":"2024-04-17T16:16:07.016480+0000","172.21.15.103:6832/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3310077212":"2024-04-17T16:16:07.016480+0000","172.21.15.45:6833/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6832/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.45:6833/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:6833/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6834/88032513":"2024-04-17T16:15:41.221894+0000","172.21.15.45:6835/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:0/644374842":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6835/88032513":"2024-04-17T16:15:41.221894+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:16:11.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.206+0000 7f4afc600640 1 -- 172.21.15.45:0/3583783721 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4ad406caa0 msgr2=0x7f4ad406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:11.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.206+0000 7f4afc600640 1 --2- 172.21.15.45:0/3583783721 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4ad406caa0 0x7f4ad406ef60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0035000 tx=0x7f4ae0030000 comp rx=0 tx=0).stop 2024-04-16T16:16:11.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.206+0000 7f4afc600640 1 -- 172.21.15.45:0/3583783721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af40d0e70 msgr2=0x7f4af41f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:11.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.206+0000 7f4afc600640 1 --2- 172.21.15.45:0/3583783721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af40d0e70 0x7f4af41f5b00 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4ae400c7f0 tx=0x7f4ae400ccb0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.206+0000 7f4afc600640 1 -- 172.21.15.45:0/3583783721 shutdown_connections 2024-04-16T16:16:11.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.206+0000 7f4afc600640 1 --2- 172.21.15.45:0/3583783721 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4ad406caa0 0x7f4ad406ef60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.206+0000 7f4afc600640 1 --2- 172.21.15.45:0/3583783721 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af4160560 0x7f4af41f5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.206+0000 7f4afc600640 1 --2- 172.21.15.45:0/3583783721 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4af415cd40 0x7f4af40cb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.206+0000 7f4afc600640 1 --2- 172.21.15.45:0/3583783721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af40d0e70 0x7f4af41f5b00 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.206+0000 7f4afc600640 1 -- 172.21.15.45:0/3583783721 >> 172.21.15.45:0/3583783721 conn(0x7f4af40d2890 msgr2=0x7f4af40cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:11.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.206+0000 7f4afc600640 1 -- 172.21.15.45:0/3583783721 shutdown_connections 2024-04-16T16:16:11.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.206+0000 7f4afc600640 1 -- 172.21.15.45:0/3583783721 wait complete. 2024-04-16T16:16:11.223 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'fs1_metadata', 'create_time': '2024-04-16T16:15:13.250782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'fs1'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'fs1_data', 'create_time': '2024-04-16T16:15:13.856986+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'fs1'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'fs2_metadata', 'create_time': '2024-04-16T16:15:33.215042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'fs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'fs2_data', 'create_time': '2024-04-16T16:15:34.271426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'fs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:16:11.224 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:16:11.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.456+0000 7fec03e00640 1 Processor -- start 2024-04-16T16:16:11.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.457+0000 7fec03e00640 1 -- start start 2024-04-16T16:16:11.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.458+0000 7fec03e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fec04145fe0 0x7fec040c5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:11.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.458+0000 7fec03e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fec040cfb50 0x7fec0406dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:11.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.458+0000 7fec03e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec040c6210 con 0x7fec04160e90 2024-04-16T16:16:11.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.458+0000 7fec03e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fec041603b0 con 0x7fec040cfb50 2024-04-16T16:16:11.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.458+0000 7fec03e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fec04160690 con 0x7fec04145fe0 2024-04-16T16:16:11.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.458+0000 7fec03400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fec040cfb50 0x7fec0406dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:11.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.458+0000 7fec03400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fec040cfb50 0x7fec0406dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51366/0 (socket says 172.21.15.45:51366) 2024-04-16T16:16:11.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.458+0000 7fec03400640 1 -- 172.21.15.45:0/2592366389 learned_addr learned my addr 172.21.15.45:0/2592366389 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:11.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.458+0000 7fec02000640 1 --1- 172.21.15.45:0/2592366389 >> v1:172.21.15.45:6789/0 conn(0x7fec04160e90 0x7fec040cf440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42474/0 (socket says 172.21.15.45:42474) 2024-04-16T16:16:11.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.459+0000 7fec02a00640 1 --2- 172.21.15.45:0/2592366389 >> v2:172.21.15.103:3300/0 conn(0x7fec04145fe0 0x7fec040c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:11.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.459+0000 7fec03400640 1 -- 172.21.15.45:0/2592366389 >> v2:172.21.15.103:3300/0 conn(0x7fec04145fe0 msgr2=0x7fec040c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:11.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.459+0000 7fec03400640 1 --2- 172.21.15.45:0/2592366389 >> v2:172.21.15.103:3300/0 conn(0x7fec04145fe0 0x7fec040c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.459+0000 7fec03400640 1 -- 172.21.15.45:0/2592366389 >> v1:172.21.15.45:6789/0 conn(0x7fec04160e90 legacy=0x7fec040cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:11.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.459+0000 7fec03400640 1 -- 172.21.15.45:0/2592366389 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec04160970 con 0x7fec040cfb50 2024-04-16T16:16:11.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.461+0000 7fec03400640 1 --2- 172.21.15.45:0/2592366389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec040cfb50 0x7fec0406dfb0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7febfc009870 tx=0x7febfc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5015e39e5b1530 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:11.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.461+0000 7fec01600640 1 -- 172.21.15.45:0/2592366389 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc035020 con 0x7fec040cfb50 2024-04-16T16:16:11.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.461+0000 7fec01600640 1 -- 172.21.15.45:0/2592366389 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febfc02d040 con 0x7fec040cfb50 2024-04-16T16:16:11.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.461+0000 7fec01600640 1 -- 172.21.15.45:0/2592366389 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc031a80 con 0x7fec040cfb50 2024-04-16T16:16:11.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.462+0000 7fec03e00640 1 -- 172.21.15.45:0/2592366389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec040cfb50 msgr2=0x7fec0406dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:11.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.462+0000 7fec03e00640 1 --2- 172.21.15.45:0/2592366389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec040cfb50 0x7fec0406dfb0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7febfc009870 tx=0x7febfc02b900 comp rx=0 tx=0).stop 2024-04-16T16:16:11.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.462+0000 7fec03e00640 1 -- 172.21.15.45:0/2592366389 shutdown_connections 2024-04-16T16:16:11.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.462+0000 7fec03e00640 1 --2- 172.21.15.45:0/2592366389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec040cfb50 0x7fec0406dfb0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.462+0000 7fec03e00640 1 --2- 172.21.15.45:0/2592366389 >> v2:172.21.15.103:3300/0 conn(0x7fec04145fe0 0x7fec040c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.462+0000 7fec03e00640 1 -- 172.21.15.45:0/2592366389 >> 172.21.15.45:0/2592366389 conn(0x7fec04067e00 msgr2=0x7fec0406b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:11.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.462+0000 7fec03e00640 1 -- 172.21.15.45:0/2592366389 shutdown_connections 2024-04-16T16:16:11.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.462+0000 7fec03e00640 1 -- 172.21.15.45:0/2592366389 wait complete. 2024-04-16T16:16:11.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.464+0000 7fec03e00640 1 Processor -- start 2024-04-16T16:16:11.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.464+0000 7fec03e00640 1 -- start start 2024-04-16T16:16:11.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.465+0000 7fec03e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec040cfb50 0x7fec041451d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:11.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.465+0000 7fec03e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec04145fe0 0x7fec04158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:11.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.465+0000 7fec03e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec04160e90 0x7fec040724c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:11.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.465+0000 7fec03e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fec04146540 con 0x7fec04145fe0 2024-04-16T16:16:11.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.465+0000 7fec03e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fec041463c0 con 0x7fec040cfb50 2024-04-16T16:16:11.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.465+0000 7fec03e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fec040c5c90 con 0x7fec04160e90 2024-04-16T16:16:11.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.465+0000 7fec02000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec04145fe0 0x7fec04158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:11.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.465+0000 7fec02000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec04145fe0 0x7fec04158b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51374/0 (socket says 172.21.15.45:51374) 2024-04-16T16:16:11.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.465+0000 7fec02000640 1 -- 172.21.15.45:0/63233378 learned_addr learned my addr 172.21.15.45:0/63233378 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.466+0000 7fec02a00640 1 --2- 172.21.15.45:0/63233378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec040cfb50 0x7fec041451d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.466+0000 7fec03400640 1 --2- 172.21.15.45:0/63233378 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec04160e90 0x7fec040724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.466+0000 7fec02000640 1 -- 172.21.15.45:0/63233378 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec04160e90 msgr2=0x7fec040724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.466+0000 7fec02000640 1 --2- 172.21.15.45:0/63233378 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec04160e90 0x7fec040724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.466+0000 7fec02000640 1 -- 172.21.15.45:0/63233378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec040cfb50 msgr2=0x7fec041451d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.466+0000 7fec02000640 1 --2- 172.21.15.45:0/63233378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec040cfb50 0x7fec041451d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.466+0000 7fec02000640 1 -- 172.21.15.45:0/63233378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec040cd3d0 con 0x7fec04145fe0 2024-04-16T16:16:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.466+0000 7fec02000640 1 --2- 172.21.15.45:0/63233378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec04145fe0 0x7fec04158b10 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7febec00c760 tx=0x7febec00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.466+0000 7febfaa00640 1 -- 172.21.15.45:0/63233378 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febec017070 con 0x7fec04145fe0 2024-04-16T16:16:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.467+0000 7febfaa00640 1 -- 172.21.15.45:0/63233378 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febec00a640 con 0x7fec04145fe0 2024-04-16T16:16:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.467+0000 7febfaa00640 1 -- 172.21.15.45:0/63233378 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febec00aa10 con 0x7fec04145fe0 2024-04-16T16:16:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.467+0000 7fec03e00640 1 -- 172.21.15.45:0/63233378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec040cd6b0 con 0x7fec04145fe0 2024-04-16T16:16:11.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.467+0000 7fec03e00640 1 -- 172.21.15.45:0/63233378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec040cdc70 con 0x7fec04145fe0 2024-04-16T16:16:11.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.468+0000 7fec03e00640 1 -- 172.21.15.45:0/63233378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec04075780 con 0x7fec04145fe0 2024-04-16T16:16:11.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.469+0000 7febfaa00640 1 -- 172.21.15.45:0/63233378 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 14) ==== 91188+0+0 (secure 0 0 0) 0x7febec01a070 con 0x7fec04145fe0 2024-04-16T16:16:11.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.470+0000 7febfaa00640 1 --2- 172.21.15.45:0/63233378 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7febe006caa0 0x7febe006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:11.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.470+0000 7febfaa00640 1 -- 172.21.15.45:0/63233378 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7febec08bcf0 con 0x7fec04145fe0 2024-04-16T16:16:11.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.471+0000 7fec02a00640 1 --2- 172.21.15.45:0/63233378 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7febe006caa0 0x7febe006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:11.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.472+0000 7fec02a00640 1 --2- 172.21.15.45:0/63233378 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7febe006caa0 0x7febe006ef60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7febf4006fd0 tx=0x7febf4008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:11.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.474+0000 7febfaa00640 1 -- 172.21.15.45:0/63233378 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7febec059640 con 0x7fec04145fe0 2024-04-16T16:16:11.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.680+0000 7fec03e00640 1 -- 172.21.15.45:0/63233378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fec040008d0 con 0x7fec04145fe0 2024-04-16T16:16:11.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.681+0000 7febfaa00640 1 -- 172.21.15.45:0/63233378 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) ==== 93+0+10 (secure 0 0 0) 0x7febec058f40 con 0x7fec04145fe0 2024-04-16T16:16:11.680 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:16:11.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.683+0000 7fec03e00640 1 -- 172.21.15.45:0/63233378 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7febe006caa0 msgr2=0x7febe006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:11.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.683+0000 7fec03e00640 1 --2- 172.21.15.45:0/63233378 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7febe006caa0 0x7febe006ef60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7febf4006fd0 tx=0x7febf4008040 comp rx=0 tx=0).stop 2024-04-16T16:16:11.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.683+0000 7fec03e00640 1 -- 172.21.15.45:0/63233378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec04145fe0 msgr2=0x7fec04158b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:11.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.683+0000 7fec03e00640 1 --2- 172.21.15.45:0/63233378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec04145fe0 0x7fec04158b10 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7febec00c760 tx=0x7febec00cc20 comp rx=0 tx=0).stop 2024-04-16T16:16:11.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.683+0000 7fec03e00640 1 -- 172.21.15.45:0/63233378 shutdown_connections 2024-04-16T16:16:11.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.683+0000 7fec03e00640 1 --2- 172.21.15.45:0/63233378 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7febe006caa0 0x7febe006ef60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.683+0000 7fec03e00640 1 --2- 172.21.15.45:0/63233378 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec04160e90 0x7fec040724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.683+0000 7fec03e00640 1 --2- 172.21.15.45:0/63233378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec04145fe0 0x7fec04158b10 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.683+0000 7fec03e00640 1 --2- 172.21.15.45:0/63233378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec040cfb50 0x7fec041451d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.683+0000 7fec03e00640 1 -- 172.21.15.45:0/63233378 >> 172.21.15.45:0/63233378 conn(0x7fec04067e00 msgr2=0x7fec0406f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:11.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.683+0000 7fec03e00640 1 -- 172.21.15.45:0/63233378 shutdown_connections 2024-04-16T16:16:11.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.684+0000 7fec03e00640 1 -- 172.21.15.45:0/63233378 wait complete. 2024-04-16T16:16:11.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs1_metadata pg_num 2024-04-16T16:16:11.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.933+0000 7fb7d5400640 1 Processor -- start 2024-04-16T16:16:11.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.933+0000 7fb7d5400640 1 -- start start 2024-04-16T16:16:11.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.934+0000 7fb7d5400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7d00d18f0 0x7fb7d00ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:11.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.934+0000 7fb7d5400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb7d00d1cd0 0x7fb7d0162dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:11.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.934+0000 7fb7d5400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7d00b6810 con 0x7fb7d00b6310 2024-04-16T16:16:11.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.934+0000 7fb7d5400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb7d00d29f0 con 0x7fb7d00d1cd0 2024-04-16T16:16:11.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.934+0000 7fb7d5400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb7d00b6af0 con 0x7fb7d00d18f0 2024-04-16T16:16:11.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.935+0000 7fb7d4a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb7d00d1cd0 0x7fb7d0162dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:11.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.935+0000 7fb7d4a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb7d00d1cd0 0x7fb7d0162dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51378/0 (socket says 172.21.15.45:51378) 2024-04-16T16:16:11.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.935+0000 7fb7d4a00640 1 -- 172.21.15.45:0/3696320752 learned_addr learned my addr 172.21.15.45:0/3696320752 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:11.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.935+0000 7fb7cfe00640 1 --2- 172.21.15.45:0/3696320752 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7d00d18f0 0x7fb7d00ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:11.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.935+0000 7fb7d4a00640 1 -- 172.21.15.45:0/3696320752 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7d00d18f0 msgr2=0x7fb7d00ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:11.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.935+0000 7fb7d4a00640 1 --2- 172.21.15.45:0/3696320752 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7d00d18f0 0x7fb7d00ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.935+0000 7fb7d4a00640 1 -- 172.21.15.45:0/3696320752 >> v1:172.21.15.45:6789/0 conn(0x7fb7d00b6310 legacy=0x7fb7d00d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:11.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.935+0000 7fb7d4a00640 1 -- 172.21.15.45:0/3696320752 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7d00b6dd0 con 0x7fb7d00d1cd0 2024-04-16T16:16:11.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.936+0000 7fb7d4a00640 1 --2- 172.21.15.45:0/3696320752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d00d1cd0 0x7fb7d0162dc0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009870 tx=0x7fb7c802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9decdbf3db44a4c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:11.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.937+0000 7fb7cea00640 1 -- 172.21.15.45:0/3696320752 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8035020 con 0x7fb7d00d1cd0 2024-04-16T16:16:11.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.937+0000 7fb7cea00640 1 -- 172.21.15.45:0/3696320752 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7c8031640 con 0x7fb7d00d1cd0 2024-04-16T16:16:11.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.937+0000 7fb7cea00640 1 -- 172.21.15.45:0/3696320752 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c802d040 con 0x7fb7d00d1cd0 2024-04-16T16:16:11.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.937+0000 7fb7d5400640 1 -- 172.21.15.45:0/3696320752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d00d1cd0 msgr2=0x7fb7d0162dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:11.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.937+0000 7fb7d5400640 1 --2- 172.21.15.45:0/3696320752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d00d1cd0 0x7fb7d0162dc0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009870 tx=0x7fb7c802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.937+0000 7fb7d5400640 1 -- 172.21.15.45:0/3696320752 shutdown_connections 2024-04-16T16:16:11.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.937+0000 7fb7d5400640 1 --2- 172.21.15.45:0/3696320752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d00d1cd0 0x7fb7d0162dc0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.937+0000 7fb7d5400640 1 --2- 172.21.15.45:0/3696320752 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7d00d18f0 0x7fb7d00ba220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.937+0000 7fb7d5400640 1 -- 172.21.15.45:0/3696320752 >> 172.21.15.45:0/3696320752 conn(0x7fb7d006cf90 msgr2=0x7fb7d006d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:11.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.938+0000 7fb7d5400640 1 -- 172.21.15.45:0/3696320752 shutdown_connections 2024-04-16T16:16:11.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.938+0000 7fb7d5400640 1 -- 172.21.15.45:0/3696320752 wait complete. 2024-04-16T16:16:11.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.939+0000 7fb7d5400640 1 Processor -- start 2024-04-16T16:16:11.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.939+0000 7fb7d5400640 1 -- start start 2024-04-16T16:16:11.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.940+0000 7fb7d5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d00b6310 0x7fb7d00ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:11.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.940+0000 7fb7d5400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7d00d18f0 0x7fb7d00c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:11.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.940+0000 7fb7d5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d00d1cd0 0x7fb7d00cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:11.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.940+0000 7fb7d5400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d0163300 con 0x7fb7d00b6310 2024-04-16T16:16:11.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.940+0000 7fb7d5400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d00ba760 con 0x7fb7d00d1cd0 2024-04-16T16:16:11.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.940+0000 7fb7d5400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb7d00cae20 con 0x7fb7d00d18f0 2024-04-16T16:16:11.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.941+0000 7fb7cfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d00b6310 0x7fb7d00ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:11.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.941+0000 7fb7cfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d00b6310 0x7fb7d00ca4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51388/0 (socket says 172.21.15.45:51388) 2024-04-16T16:16:11.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.941+0000 7fb7cfe00640 1 -- 172.21.15.45:0/1420479591 learned_addr learned my addr 172.21.15.45:0/1420479591 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:11.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.941+0000 7fb7d4a00640 1 --2- 172.21.15.45:0/1420479591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d00d1cd0 0x7fb7d00cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:11.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.941+0000 7fb7cfe00640 1 -- 172.21.15.45:0/1420479591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7d00d18f0 msgr2=0x7fb7d00c8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:11.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.941+0000 7fb7cf400640 1 --2- 172.21.15.45:0/1420479591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7d00d18f0 0x7fb7d00c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:11.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.941+0000 7fb7cfe00640 1 --2- 172.21.15.45:0/1420479591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7d00d18f0 0x7fb7d00c8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.941+0000 7fb7cfe00640 1 -- 172.21.15.45:0/1420479591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d00d1cd0 msgr2=0x7fb7d00cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:11.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.941+0000 7fb7cfe00640 1 --2- 172.21.15.45:0/1420479591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d00d1cd0 0x7fb7d00cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:11.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.941+0000 7fb7cfe00640 1 -- 172.21.15.45:0/1420479591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7d00c94d0 con 0x7fb7d00b6310 2024-04-16T16:16:11.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.941+0000 7fb7d4a00640 1 --2- 172.21.15.45:0/1420479591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d00d1cd0 0x7fb7d00cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:11.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.942+0000 7fb7cfe00640 1 --2- 172.21.15.45:0/1420479591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d00b6310 0x7fb7d00ca4d0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000d800 tx=0x7fb7c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:11.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.942+0000 7fb7c7e00640 1 -- 172.21.15.45:0/1420479591 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c000b6c0 con 0x7fb7d00b6310 2024-04-16T16:16:11.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.942+0000 7fb7c7e00640 1 -- 172.21.15.45:0/1420479591 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7c0019070 con 0x7fb7d00b6310 2024-04-16T16:16:11.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.942+0000 7fb7c7e00640 1 -- 172.21.15.45:0/1420479591 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c000ba80 con 0x7fb7d00b6310 2024-04-16T16:16:11.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.942+0000 7fb7d5400640 1 -- 172.21.15.45:0/1420479591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7d00c97b0 con 0x7fb7d00b6310 2024-04-16T16:16:11.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.942+0000 7fb7d5400640 1 -- 172.21.15.45:0/1420479591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7d00c9d70 con 0x7fb7d00b6310 2024-04-16T16:16:11.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.943+0000 7fb7d5400640 1 -- 172.21.15.45:0/1420479591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7d0161280 con 0x7fb7d00b6310 2024-04-16T16:16:11.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.944+0000 7fb7c7e00640 1 -- 172.21.15.45:0/1420479591 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fb7c000bc20 con 0x7fb7d00b6310 2024-04-16T16:16:11.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.945+0000 7fb7c7e00640 1 --2- 172.21.15.45:0/1420479591 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb7b006cb30 0x7fb7b006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:11.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.945+0000 7fb7c7e00640 1 -- 172.21.15.45:0/1420479591 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7fb7c008cc10 con 0x7fb7d00b6310 2024-04-16T16:16:11.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.946+0000 7fb7cf400640 1 --2- 172.21.15.45:0/1420479591 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb7b006cb30 0x7fb7b006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:11.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.947+0000 7fb7cf400640 1 --2- 172.21.15.45:0/1420479591 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb7b006cb30 0x7fb7b006eff0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc007fd0 tx=0x7fb7bc009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:11.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:11.950+0000 7fb7c7e00640 1 -- 172.21.15.45:0/1420479591 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb7d0161280 con 0x7fb7d00b6310 2024-04-16T16:16:12.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.156+0000 7fb7d5400640 1 -- 172.21.15.45:0/1420479591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs1_metadata", "var": "pg_num"} v 0) -- 0x7fb7d0075780 con 0x7fb7d00b6310 2024-04-16T16:16:12.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.157+0000 7fb7c7e00640 1 -- 172.21.15.45:0/1420479591 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs1_metadata", "var": "pg_num"}]=0 v57) ==== 101+0+11 (secure 0 0 0) 0x7fb7c005a5a0 con 0x7fb7d00b6310 2024-04-16T16:16:12.157 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:16:12.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.160+0000 7fb7d5400640 1 -- 172.21.15.45:0/1420479591 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb7b006cb30 msgr2=0x7fb7b006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:12.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.160+0000 7fb7d5400640 1 --2- 172.21.15.45:0/1420479591 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb7b006cb30 0x7fb7b006eff0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc007fd0 tx=0x7fb7bc009040 comp rx=0 tx=0).stop 2024-04-16T16:16:12.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.160+0000 7fb7d5400640 1 -- 172.21.15.45:0/1420479591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d00b6310 msgr2=0x7fb7d00ca4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:12.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.160+0000 7fb7d5400640 1 --2- 172.21.15.45:0/1420479591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d00b6310 0x7fb7d00ca4d0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000d800 tx=0x7fb7c000dcc0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.161+0000 7fb7d5400640 1 -- 172.21.15.45:0/1420479591 shutdown_connections 2024-04-16T16:16:12.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.161+0000 7fb7d5400640 1 --2- 172.21.15.45:0/1420479591 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb7b006cb30 0x7fb7b006eff0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.161+0000 7fb7d5400640 1 --2- 172.21.15.45:0/1420479591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d00d1cd0 0x7fb7d00cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.161+0000 7fb7d5400640 1 --2- 172.21.15.45:0/1420479591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7d00d18f0 0x7fb7d00c8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.161+0000 7fb7d5400640 1 --2- 172.21.15.45:0/1420479591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d00b6310 0x7fb7d00ca4d0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.161+0000 7fb7d5400640 1 -- 172.21.15.45:0/1420479591 >> 172.21.15.45:0/1420479591 conn(0x7fb7d006cf90 msgr2=0x7fb7d00b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:12.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.161+0000 7fb7d5400640 1 -- 172.21.15.45:0/1420479591 shutdown_connections 2024-04-16T16:16:12.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.161+0000 7fb7d5400640 1 -- 172.21.15.45:0/1420479591 wait complete. 2024-04-16T16:16:12.178 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs1_data pg_num 2024-04-16T16:16:12.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.398+0000 7f8405a00640 1 Processor -- start 2024-04-16T16:16:12.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.399+0000 7f8405a00640 1 -- start start 2024-04-16T16:16:12.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.400+0000 7f8405a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f840015c7c0 0x7f8400072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:12.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.400+0000 7f8405a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84000d10e0 0x7f840006f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:12.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.400+0000 7f8405a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84000b5530 con 0x7f84000b4930 2024-04-16T16:16:12.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.400+0000 7f8405a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f84000b52e0 con 0x7f840015c7c0 2024-04-16T16:16:12.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.400+0000 7f8405a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f84000d0260 con 0x7f84000d10e0 2024-04-16T16:16:12.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.400+0000 7f83fea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f840015c7c0 0x7f8400072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:12.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.400+0000 7f83fea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f840015c7c0 0x7f8400072e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51396/0 (socket says 172.21.15.45:51396) 2024-04-16T16:16:12.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.400+0000 7f83fea00640 1 -- 172.21.15.45:0/1653628507 learned_addr learned my addr 172.21.15.45:0/1653628507 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:12.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.401+0000 7f83ff400640 1 --2- 172.21.15.45:0/1653628507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84000d10e0 0x7f840006f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:12.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.401+0000 7f83fea00640 1 -- 172.21.15.45:0/1653628507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84000d10e0 msgr2=0x7f840006f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:12.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.401+0000 7f83fea00640 1 --2- 172.21.15.45:0/1653628507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84000d10e0 0x7f840006f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.401+0000 7f83fea00640 1 -- 172.21.15.45:0/1653628507 >> v1:172.21.15.45:6789/0 conn(0x7f84000b4930 legacy=0x7f84000d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:12.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.401+0000 7f83fea00640 1 -- 172.21.15.45:0/1653628507 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84000b5810 con 0x7f840015c7c0 2024-04-16T16:16:12.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.402+0000 7f83fea00640 1 --2- 172.21.15.45:0/1653628507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840015c7c0 0x7f8400072e10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f83f0009870 tx=0x7f83f002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d506ec74216805 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:12.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.402+0000 7f83fd600640 1 -- 172.21.15.45:0/1653628507 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f00317c0 con 0x7f840015c7c0 2024-04-16T16:16:12.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.402+0000 7f83fd600640 1 -- 172.21.15.45:0/1653628507 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f0037070 con 0x7f840015c7c0 2024-04-16T16:16:12.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.402+0000 7f83fd600640 1 -- 172.21.15.45:0/1653628507 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f0031b80 con 0x7f840015c7c0 2024-04-16T16:16:12.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.403+0000 7f8405a00640 1 -- 172.21.15.45:0/1653628507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840015c7c0 msgr2=0x7f8400072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:12.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.403+0000 7f8405a00640 1 --2- 172.21.15.45:0/1653628507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840015c7c0 0x7f8400072e10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f83f0009870 tx=0x7f83f002b860 comp rx=0 tx=0).stop 2024-04-16T16:16:12.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.403+0000 7f8405a00640 1 -- 172.21.15.45:0/1653628507 shutdown_connections 2024-04-16T16:16:12.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.403+0000 7f8405a00640 1 --2- 172.21.15.45:0/1653628507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84000d10e0 0x7f840006f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.403+0000 7f8405a00640 1 --2- 172.21.15.45:0/1653628507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840015c7c0 0x7f8400072e10 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.403+0000 7f8405a00640 1 -- 172.21.15.45:0/1653628507 >> 172.21.15.45:0/1653628507 conn(0x7f84000b3970 msgr2=0x7f84000b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:12.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.403+0000 7f8405a00640 1 -- 172.21.15.45:0/1653628507 shutdown_connections 2024-04-16T16:16:12.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.403+0000 7f8405a00640 1 -- 172.21.15.45:0/1653628507 wait complete. 2024-04-16T16:16:12.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.405+0000 7f8405a00640 1 Processor -- start 2024-04-16T16:16:12.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.405+0000 7f8405a00640 1 -- start start 2024-04-16T16:16:12.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.406+0000 7f8405a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84000b4930 0x7f84000cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:12.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.406+0000 7f8405a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84000d10e0 0x7f8400150470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:12.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.406+0000 7f8405a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840015c7c0 0x7f84000c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:12.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.406+0000 7f8405a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f840006f6a0 con 0x7f840015c7c0 2024-04-16T16:16:12.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.406+0000 7f8405a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8400073350 con 0x7f84000b4930 2024-04-16T16:16:12.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.406+0000 7f8405a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f840006f820 con 0x7f84000d10e0 2024-04-16T16:16:12.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.406+0000 7f83ff400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840015c7c0 0x7f84000c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:12.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.406+0000 7f83ff400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840015c7c0 0x7f84000c91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51404/0 (socket says 172.21.15.45:51404) 2024-04-16T16:16:12.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.406+0000 7f83ff400640 1 -- 172.21.15.45:0/3292928885 learned_addr learned my addr 172.21.15.45:0/3292928885 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:12.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.407+0000 7f83fe000640 1 --2- 172.21.15.45:0/3292928885 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84000d10e0 0x7f8400150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:12.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.407+0000 7f83ff400640 1 -- 172.21.15.45:0/3292928885 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84000d10e0 msgr2=0x7f8400150470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:12.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.407+0000 7f83fea00640 1 --2- 172.21.15.45:0/3292928885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84000b4930 0x7f84000cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:12.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.407+0000 7f83ff400640 1 --2- 172.21.15.45:0/3292928885 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84000d10e0 0x7f8400150470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.407+0000 7f83ff400640 1 -- 172.21.15.45:0/3292928885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84000b4930 msgr2=0x7f84000cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:12.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.407+0000 7f83ff400640 1 --2- 172.21.15.45:0/3292928885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84000b4930 0x7f84000cc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.407+0000 7f83ff400640 1 -- 172.21.15.45:0/3292928885 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84001509b0 con 0x7f840015c7c0 2024-04-16T16:16:12.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.407+0000 7f83fe000640 1 --2- 172.21.15.45:0/3292928885 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84000d10e0 0x7f8400150470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:12.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.407+0000 7f83ff400640 1 --2- 172.21.15.45:0/3292928885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840015c7c0 0x7f84000c91b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f83f800d800 tx=0x7f83f800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:12.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.407+0000 7f83f6a00640 1 -- 172.21.15.45:0/3292928885 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f800b6c0 con 0x7f840015c7c0 2024-04-16T16:16:12.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.407+0000 7f83f6a00640 1 -- 172.21.15.45:0/3292928885 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f8019070 con 0x7f840015c7c0 2024-04-16T16:16:12.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.407+0000 7f83f6a00640 1 -- 172.21.15.45:0/3292928885 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f800bac0 con 0x7f840015c7c0 2024-04-16T16:16:12.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.407+0000 7f8405a00640 1 -- 172.21.15.45:0/3292928885 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84000c9ab0 con 0x7f840015c7c0 2024-04-16T16:16:12.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.408+0000 7f8405a00640 1 -- 172.21.15.45:0/3292928885 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84000ca070 con 0x7f840015c7c0 2024-04-16T16:16:12.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.409+0000 7f8405a00640 1 -- 172.21.15.45:0/3292928885 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8400075780 con 0x7f840015c7c0 2024-04-16T16:16:12.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.410+0000 7f83f6a00640 1 -- 172.21.15.45:0/3292928885 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f83f800bc60 con 0x7f840015c7c0 2024-04-16T16:16:12.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.411+0000 7f83f6a00640 1 --2- 172.21.15.45:0/3292928885 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f83e006cb30 0x7f83e006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:12.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.411+0000 7f83f6a00640 1 -- 172.21.15.45:0/3292928885 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7f83f808cc20 con 0x7f840015c7c0 2024-04-16T16:16:12.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.411+0000 7f83fea00640 1 --2- 172.21.15.45:0/3292928885 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f83e006cb30 0x7f83e006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:12.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.412+0000 7f83fea00640 1 --2- 172.21.15.45:0/3292928885 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f83e006cb30 0x7f83e006eff0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f83f0036000 tx=0x7f83f0008030 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:12.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.415+0000 7f83f6a00640 1 -- 172.21.15.45:0/3292928885 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f83f805a520 con 0x7f840015c7c0 2024-04-16T16:16:12.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.618+0000 7f8405a00640 1 -- 172.21.15.45:0/3292928885 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs1_data", "var": "pg_num"} v 0) -- 0x7f840006e6f0 con 0x7f840015c7c0 2024-04-16T16:16:12.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.618+0000 7f83f6a00640 1 -- 172.21.15.45:0/3292928885 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs1_data", "var": "pg_num"}]=0 v57) ==== 97+0+11 (secure 0 0 0) 0x7f83f8059e20 con 0x7f840015c7c0 2024-04-16T16:16:12.617 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:16:12.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.621+0000 7f8405a00640 1 -- 172.21.15.45:0/3292928885 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f83e006cb30 msgr2=0x7f83e006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:12.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.621+0000 7f8405a00640 1 --2- 172.21.15.45:0/3292928885 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f83e006cb30 0x7f83e006eff0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f83f0036000 tx=0x7f83f0008030 comp rx=0 tx=0).stop 2024-04-16T16:16:12.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.621+0000 7f8405a00640 1 -- 172.21.15.45:0/3292928885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840015c7c0 msgr2=0x7f84000c91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:12.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.621+0000 7f8405a00640 1 --2- 172.21.15.45:0/3292928885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840015c7c0 0x7f84000c91b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f83f800d800 tx=0x7f83f800dcc0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.622+0000 7f8405a00640 1 -- 172.21.15.45:0/3292928885 shutdown_connections 2024-04-16T16:16:12.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.622+0000 7f8405a00640 1 --2- 172.21.15.45:0/3292928885 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f83e006cb30 0x7f83e006eff0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.622+0000 7f8405a00640 1 --2- 172.21.15.45:0/3292928885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840015c7c0 0x7f84000c91b0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.622+0000 7f8405a00640 1 --2- 172.21.15.45:0/3292928885 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84000d10e0 0x7f8400150470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.622+0000 7f8405a00640 1 --2- 172.21.15.45:0/3292928885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84000b4930 0x7f84000cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.622+0000 7f8405a00640 1 -- 172.21.15.45:0/3292928885 >> 172.21.15.45:0/3292928885 conn(0x7f84000b3970 msgr2=0x7f84000b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:12.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.622+0000 7f8405a00640 1 -- 172.21.15.45:0/3292928885 shutdown_connections 2024-04-16T16:16:12.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.622+0000 7f8405a00640 1 -- 172.21.15.45:0/3292928885 wait complete. 2024-04-16T16:16:12.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs2_metadata pg_num 2024-04-16T16:16:12.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.870+0000 7f6ba6c00640 1 Processor -- start 2024-04-16T16:16:12.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.870+0000 7f6ba6c00640 1 -- start start 2024-04-16T16:16:12.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.872+0000 7f6ba6c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ba01461b0 0x7f6ba0160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:12.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.872+0000 7f6ba6c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ba00d0260 con 0x7f6ba00b5d50 2024-04-16T16:16:12.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.872+0000 7f6ba6c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ba0146590 con 0x7f6ba00b58f0 2024-04-16T16:16:12.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.872+0000 7f6ba6c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6ba006dfd0 con 0x7f6ba01461b0 2024-04-16T16:16:12.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.872+0000 7f6ba6200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6ba00b5d50 0x7f6ba00cfb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42492/0 (socket says 172.21.15.45:42492) 2024-04-16T16:16:12.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.872+0000 7f6ba6200640 1 -- 172.21.15.45:0/576238669 learned_addr learned my addr 172.21.15.45:0/576238669 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:12.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.873+0000 7f6ba5800640 1 --2- 172.21.15.45:0/576238669 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ba01461b0 0x7f6ba0160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.873+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1408912581 0 0) 0x7f6ba00d0260 con 0x7f6ba00b5d50 2024-04-16T16:16:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.873+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b88003520 con 0x7f6ba00b5d50 2024-04-16T16:16:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.873+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 663518664 0 0) 0x7f6ba0146590 con 0x7f6ba00b58f0 2024-04-16T16:16:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.873+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ba00d0260 con 0x7f6ba00b58f0 2024-04-16T16:16:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.873+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3578107079 0 0) 0x7f6b88003520 con 0x7f6ba00b5d50 2024-04-16T16:16:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.873+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ba01461b0 msgr2=0x7f6ba0160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.873+0000 7f6b9fe00640 1 --2- 172.21.15.45:0/576238669 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ba01461b0 0x7f6ba0160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.873+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 >> v1:172.21.15.103:6789/0 conn(0x7f6ba00b58f0 legacy=0x7f6ba00cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.874+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba00c5750 con 0x7f6ba00b5d50 2024-04-16T16:16:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.874+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f6b980019f0 con 0x7f6ba00b5d50 2024-04-16T16:16:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.874+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 >> v1:172.21.15.45:6789/0 conn(0x7f6ba00b5d50 legacy=0x7f6ba00cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.874+0000 7f6b9fe00640 1 --2- 172.21.15.45:0/576238669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b88003b70 0x7f6b88003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.874+0000 7f6b9fe00640 1 --2- 172.21.15.45:0/576238669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b88004800 0x7f6b88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:12.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.874+0000 7f6b9fe00640 1 --2- 172.21.15.45:0/576238669 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ba01461b0 0x7f6b880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:12.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.874+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8800ba70 con 0x7f6b88004800 2024-04-16T16:16:12.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.874+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8800bbf0 con 0x7f6b88003b70 2024-04-16T16:16:12.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.874+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6b8800bdf0 con 0x7f6ba01461b0 2024-04-16T16:16:12.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.874+0000 7f6ba4e00640 1 --2- 172.21.15.45:0/576238669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b88004800 0x7f6b88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:12.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.874+0000 7f6ba6200640 1 --2- 172.21.15.45:0/576238669 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ba01461b0 0x7f6b880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:12.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.874+0000 7f6ba5800640 1 --2- 172.21.15.45:0/576238669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b88003b70 0x7f6b88003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:12.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.874+0000 7f6ba4e00640 1 -- 172.21.15.45:0/576238669 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ba01461b0 msgr2=0x7f6b880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:12.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.875+0000 7f6ba4e00640 1 --2- 172.21.15.45:0/576238669 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ba01461b0 0x7f6b880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.875+0000 7f6ba4e00640 1 -- 172.21.15.45:0/576238669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b88003b70 msgr2=0x7f6b88003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:12.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.875+0000 7f6ba4e00640 1 --2- 172.21.15.45:0/576238669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b88003b70 0x7f6b88003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.875+0000 7f6ba4e00640 1 -- 172.21.15.45:0/576238669 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b88003520 con 0x7f6b88004800 2024-04-16T16:16:12.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.875+0000 7f6ba5800640 1 --2- 172.21.15.45:0/576238669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b88003b70 0x7f6b88003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:12.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.876+0000 7f6ba4e00640 1 --2- 172.21.15.45:0/576238669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b88004800 0x7f6b88004c80 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c0018d0 tx=0x7f6b88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24fdf9cea83f8a6f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:12.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.876+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c037070 con 0x7f6b88004800 2024-04-16T16:16:12.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.876+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b8c0052d0 con 0x7f6b88004800 2024-04-16T16:16:12.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.876+0000 7f6b9fe00640 1 -- 172.21.15.45:0/576238669 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c0014f0 con 0x7f6b88004800 2024-04-16T16:16:12.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.877+0000 7f6ba6c00640 1 -- 172.21.15.45:0/576238669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b88004800 msgr2=0x7f6b88004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:12.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.877+0000 7f6ba6c00640 1 --2- 172.21.15.45:0/576238669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b88004800 0x7f6b88004c80 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c0018d0 tx=0x7f6b88005830 comp rx=0 tx=0).stop 2024-04-16T16:16:12.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.877+0000 7f6ba6200640 1 -- 172.21.15.45:0/576238669 reap_dead start 2024-04-16T16:16:12.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.877+0000 7f6ba6c00640 1 -- 172.21.15.45:0/576238669 shutdown_connections 2024-04-16T16:16:12.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.877+0000 7f6ba6c00640 1 -- 172.21.15.45:0/576238669 >> 172.21.15.45:0/576238669 conn(0x7f6ba006b930 msgr2=0x7f6ba006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:12.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.877+0000 7f6ba6c00640 1 -- 172.21.15.45:0/576238669 shutdown_connections 2024-04-16T16:16:12.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.877+0000 7f6ba6c00640 1 -- 172.21.15.45:0/576238669 wait complete. 2024-04-16T16:16:12.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.879+0000 7f6ba6c00640 1 Processor -- start 2024-04-16T16:16:12.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.879+0000 7f6ba6c00640 1 -- start start 2024-04-16T16:16:12.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.880+0000 7f6ba6c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b88004800 0x7f6ba0138d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:12.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.880+0000 7f6ba6c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ba01396c0 0x7f6ba015c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:12.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.880+0000 7f6ba6c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ba0142350 0x7f6ba0144ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:12.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.880+0000 7f6ba6c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba00c5cd0 con 0x7f6b88004800 2024-04-16T16:16:12.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.880+0000 7f6ba6c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba00c59f0 con 0x7f6ba01396c0 2024-04-16T16:16:12.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.880+0000 7f6ba6c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6ba00c5fb0 con 0x7f6ba0142350 2024-04-16T16:16:12.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.880+0000 7f6ba5800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b88004800 0x7f6ba0138d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:12.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.880+0000 7f6ba5800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b88004800 0x7f6ba0138d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51428/0 (socket says 172.21.15.45:51428) 2024-04-16T16:16:12.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.880+0000 7f6ba5800640 1 -- 172.21.15.45:0/1325749401 learned_addr learned my addr 172.21.15.45:0/1325749401 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.881+0000 7f6ba6200640 1 --2- 172.21.15.45:0/1325749401 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ba0142350 0x7f6ba0144ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.881+0000 7f6ba4e00640 1 --2- 172.21.15.45:0/1325749401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ba01396c0 0x7f6ba015c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.881+0000 7f6ba5800640 1 -- 172.21.15.45:0/1325749401 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ba0142350 msgr2=0x7f6ba0144ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.881+0000 7f6ba5800640 1 --2- 172.21.15.45:0/1325749401 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ba0142350 0x7f6ba0144ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.881+0000 7f6ba5800640 1 -- 172.21.15.45:0/1325749401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ba01396c0 msgr2=0x7f6ba015c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.881+0000 7f6ba5800640 1 --2- 172.21.15.45:0/1325749401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ba01396c0 0x7f6ba015c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.881+0000 7f6ba5800640 1 -- 172.21.15.45:0/1325749401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba01393b0 con 0x7f6b88004800 2024-04-16T16:16:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.881+0000 7f6ba5800640 1 --2- 172.21.15.45:0/1325749401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b88004800 0x7f6ba0138d90 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f6b94010910 tx=0x7f6b94010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.882+0000 7f6b9d600640 1 -- 172.21.15.45:0/1325749401 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9400f040 con 0x7f6b88004800 2024-04-16T16:16:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.882+0000 7f6b9d600640 1 -- 172.21.15.45:0/1325749401 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b9400ec20 con 0x7f6b88004800 2024-04-16T16:16:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.882+0000 7f6b9d600640 1 -- 172.21.15.45:0/1325749401 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9401c400 con 0x7f6b88004800 2024-04-16T16:16:12.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.882+0000 7f6ba6c00640 1 -- 172.21.15.45:0/1325749401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ba0143fb0 con 0x7f6b88004800 2024-04-16T16:16:12.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.882+0000 7f6ba6c00640 1 -- 172.21.15.45:0/1325749401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ba0144570 con 0x7f6b88004800 2024-04-16T16:16:12.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.883+0000 7f6ba6c00640 1 -- 172.21.15.45:0/1325749401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ba0075780 con 0x7f6b88004800 2024-04-16T16:16:12.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.884+0000 7f6b9d600640 1 -- 172.21.15.45:0/1325749401 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f6b9401c5a0 con 0x7f6b88004800 2024-04-16T16:16:12.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.885+0000 7f6b9d600640 1 --2- 172.21.15.45:0/1325749401 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6b8006cb30 0x7f6b8006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:12.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.885+0000 7f6b9d600640 1 -- 172.21.15.45:0/1325749401 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7f6b9408cd70 con 0x7f6b88004800 2024-04-16T16:16:12.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.886+0000 7f6ba4e00640 1 --2- 172.21.15.45:0/1325749401 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6b8006cb30 0x7f6b8006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:12.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.887+0000 7f6ba4e00640 1 --2- 172.21.15.45:0/1325749401 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6b8006cb30 0x7f6b8006eff0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c034000 tx=0x7f6b8c031000 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:12.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:12.889+0000 7f6b9d600640 1 -- 172.21.15.45:0/1325749401 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6b94018030 con 0x7f6b88004800 2024-04-16T16:16:13.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.100+0000 7f6ba6c00640 1 -- 172.21.15.45:0/1325749401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs2_metadata", "var": "pg_num"} v 0) -- 0x7f6ba0072500 con 0x7f6b88004800 2024-04-16T16:16:13.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.101+0000 7f6b9d600640 1 -- 172.21.15.45:0/1325749401 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs2_metadata", "var": "pg_num"}]=0 v57) ==== 101+0+11 (secure 0 0 0) 0x7f6b9405a670 con 0x7f6b88004800 2024-04-16T16:16:13.100 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:16:13.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.104+0000 7f6ba6c00640 1 -- 172.21.15.45:0/1325749401 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6b8006cb30 msgr2=0x7f6b8006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:13.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.104+0000 7f6ba6c00640 1 --2- 172.21.15.45:0/1325749401 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6b8006cb30 0x7f6b8006eff0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c034000 tx=0x7f6b8c031000 comp rx=0 tx=0).stop 2024-04-16T16:16:13.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.104+0000 7f6ba6c00640 1 -- 172.21.15.45:0/1325749401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b88004800 msgr2=0x7f6ba0138d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:13.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.104+0000 7f6ba6c00640 1 --2- 172.21.15.45:0/1325749401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b88004800 0x7f6ba0138d90 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f6b94010910 tx=0x7f6b94010dd0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.104+0000 7f6ba6c00640 1 -- 172.21.15.45:0/1325749401 shutdown_connections 2024-04-16T16:16:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.104+0000 7f6ba6c00640 1 --2- 172.21.15.45:0/1325749401 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6b8006cb30 0x7f6b8006eff0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.104+0000 7f6ba6c00640 1 --2- 172.21.15.45:0/1325749401 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6ba0142350 0x7f6ba0144ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.104+0000 7f6ba6c00640 1 --2- 172.21.15.45:0/1325749401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ba01396c0 0x7f6ba015c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.104+0000 7f6ba6c00640 1 --2- 172.21.15.45:0/1325749401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b88004800 0x7f6ba0138d90 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.104+0000 7f6ba6c00640 1 -- 172.21.15.45:0/1325749401 >> 172.21.15.45:0/1325749401 conn(0x7f6ba006b930 msgr2=0x7f6ba015dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.104+0000 7f6ba6c00640 1 -- 172.21.15.45:0/1325749401 shutdown_connections 2024-04-16T16:16:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.104+0000 7f6ba6c00640 1 -- 172.21.15.45:0/1325749401 wait complete. 2024-04-16T16:16:13.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs2_data pg_num 2024-04-16T16:16:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.346+0000 7fb953200640 1 Processor -- start 2024-04-16T16:16:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.346+0000 7fb953200640 1 -- start start 2024-04-16T16:16:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.347+0000 7fb953200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb9540d18f0 0x7fb9540ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.347+0000 7fb953200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9540b6810 con 0x7fb9540d1cd0 2024-04-16T16:16:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.347+0000 7fb953200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9540b6af0 con 0x7fb9540b6310 2024-04-16T16:16:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.347+0000 7fb953200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb9540d29f0 con 0x7fb9540d18f0 2024-04-16T16:16:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.348+0000 7fb952800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb9540d1cd0 0x7fb954162dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47064/0 (socket says 172.21.15.45:47064) 2024-04-16T16:16:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.348+0000 7fb952800640 1 -- 172.21.15.45:0/2958047541 learned_addr learned my addr 172.21.15.45:0/2958047541 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.348+0000 7fb951e00640 1 --2- 172.21.15.45:0/2958047541 >> v2:172.21.15.103:3300/0 conn(0x7fb9540d18f0 0x7fb9540ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.348+0000 7fb950a00640 1 -- 172.21.15.45:0/2958047541 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1433488885 0 0) 0x7fb9540b6810 con 0x7fb9540d1cd0 2024-04-16T16:16:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.348+0000 7fb950a00640 1 -- 172.21.15.45:0/2958047541 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb938003520 con 0x7fb9540d1cd0 2024-04-16T16:16:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.349+0000 7fb950a00640 1 -- 172.21.15.45:0/2958047541 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2828114241 0 0) 0x7fb938003520 con 0x7fb9540d1cd0 2024-04-16T16:16:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.349+0000 7fb950a00640 1 -- 172.21.15.45:0/2958047541 >> v2:172.21.15.103:3300/0 conn(0x7fb9540d18f0 msgr2=0x7fb9540ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.349+0000 7fb950a00640 1 --2- 172.21.15.45:0/2958047541 >> v2:172.21.15.103:3300/0 conn(0x7fb9540d18f0 0x7fb9540ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.349+0000 7fb950a00640 1 -- 172.21.15.45:0/2958047541 >> v1:172.21.15.103:6789/0 conn(0x7fb9540b6310 legacy=0x7fb9540d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.349+0000 7fb950a00640 1 -- 172.21.15.45:0/2958047541 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9540b6dd0 con 0x7fb9540d1cd0 2024-04-16T16:16:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.349+0000 7fb950a00640 1 -- 172.21.15.45:0/2958047541 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fb94c0019f0 con 0x7fb9540d1cd0 2024-04-16T16:16:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.349+0000 7fb950a00640 1 -- 172.21.15.45:0/2958047541 >> v1:172.21.15.45:6789/0 conn(0x7fb9540d1cd0 legacy=0x7fb954162dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.349+0000 7fb950a00640 1 --2- 172.21.15.45:0/2958047541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9380039b0 0x7fb938003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.349+0000 7fb950a00640 1 --2- 172.21.15.45:0/2958047541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb938004800 0x7fb938004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.349+0000 7fb950a00640 1 --2- 172.21.15.45:0/2958047541 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9380052e0 0x7fb93800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.349+0000 7fb950a00640 1 -- 172.21.15.45:0/2958047541 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb9540d29f0 con 0x7fb9380039b0 2024-04-16T16:16:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.349+0000 7fb950a00640 1 -- 172.21.15.45:0/2958047541 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb93800c1b0 con 0x7fb938004800 2024-04-16T16:16:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.349+0000 7fb950a00640 1 -- 172.21.15.45:0/2958047541 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb93800c3b0 con 0x7fb9380052e0 2024-04-16T16:16:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.349+0000 7fb951e00640 1 --2- 172.21.15.45:0/2958047541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9380039b0 0x7fb938003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.349+0000 7fb951400640 1 --2- 172.21.15.45:0/2958047541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb938004800 0x7fb938004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.350+0000 7fb951e00640 1 -- 172.21.15.45:0/2958047541 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9380052e0 msgr2=0x7fb93800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.350+0000 7fb951e00640 1 --2- 172.21.15.45:0/2958047541 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9380052e0 0x7fb93800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.350+0000 7fb951e00640 1 -- 172.21.15.45:0/2958047541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb938004800 msgr2=0x7fb938004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.350+0000 7fb951e00640 1 --2- 172.21.15.45:0/2958047541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb938004800 0x7fb938004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.350+0000 7fb952800640 1 -- 172.21.15.45:0/2958047541 reap_dead start 2024-04-16T16:16:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.350+0000 7fb951e00640 1 -- 172.21.15.45:0/2958047541 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb938003520 con 0x7fb9380039b0 2024-04-16T16:16:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.350+0000 7fb951400640 1 --2- 172.21.15.45:0/2958047541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb938004800 0x7fb938004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.351+0000 7fb951e00640 1 --2- 172.21.15.45:0/2958047541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9380039b0 0x7fb938003dd0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb944009a90 tx=0x7fb938005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766d8b08c9d18d53 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.351+0000 7fb950a00640 1 -- 172.21.15.45:0/2958047541 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb944039070 con 0x7fb9380039b0 2024-04-16T16:16:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.351+0000 7fb950a00640 1 -- 172.21.15.45:0/2958047541 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb94402c680 con 0x7fb9380039b0 2024-04-16T16:16:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.351+0000 7fb950a00640 1 -- 172.21.15.45:0/2958047541 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb944009690 con 0x7fb9380039b0 2024-04-16T16:16:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.352+0000 7fb953200640 1 -- 172.21.15.45:0/2958047541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9380039b0 msgr2=0x7fb938003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.352+0000 7fb953200640 1 --2- 172.21.15.45:0/2958047541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9380039b0 0x7fb938003dd0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb944009a90 tx=0x7fb938005830 comp rx=0 tx=0).stop 2024-04-16T16:16:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.352+0000 7fb953200640 1 -- 172.21.15.45:0/2958047541 shutdown_connections 2024-04-16T16:16:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.352+0000 7fb953200640 1 --2- 172.21.15.45:0/2958047541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9380039b0 0x7fb938003dd0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.352+0000 7fb953200640 1 -- 172.21.15.45:0/2958047541 >> 172.21.15.45:0/2958047541 conn(0x7fb95406cf90 msgr2=0x7fb95406d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.352+0000 7fb953200640 1 -- 172.21.15.45:0/2958047541 shutdown_connections 2024-04-16T16:16:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.352+0000 7fb953200640 1 -- 172.21.15.45:0/2958047541 wait complete. 2024-04-16T16:16:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.354+0000 7fb953200640 1 Processor -- start 2024-04-16T16:16:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.354+0000 7fb953200640 1 -- start start 2024-04-16T16:16:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.355+0000 7fb953200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9380039b0 0x7fb95414fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.355+0000 7fb953200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb95415d690 0x7fb9541f82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.355+0000 7fb953200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb95415c630 0x7fb9540b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.355+0000 7fb953200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb9540ba3a0 con 0x7fb95415d690 2024-04-16T16:16:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.355+0000 7fb953200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb9540ba220 con 0x7fb9380039b0 2024-04-16T16:16:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.355+0000 7fb953200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb9540ba520 con 0x7fb95415c630 2024-04-16T16:16:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.355+0000 7fb951400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb95415d690 0x7fb9541f82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.355+0000 7fb951400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb95415d690 0x7fb9541f82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49224/0 (socket says 172.21.15.45:49224) 2024-04-16T16:16:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.355+0000 7fb951400640 1 -- 172.21.15.45:0/1767507262 learned_addr learned my addr 172.21.15.45:0/1767507262 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.356+0000 7fb952800640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb95415c630 0x7fb9540b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.356+0000 7fb951e00640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9380039b0 0x7fb95414fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.356+0000 7fb951400640 1 -- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb95415c630 msgr2=0x7fb9540b8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:13.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.356+0000 7fb951400640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb95415c630 0x7fb9540b8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.356+0000 7fb951400640 1 -- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9380039b0 msgr2=0x7fb95414fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:13.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.356+0000 7fb951400640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9380039b0 0x7fb95414fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.356+0000 7fb951400640 1 -- 172.21.15.45:0/1767507262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb954150220 con 0x7fb95415d690 2024-04-16T16:16:13.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.356+0000 7fb952800640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb95415c630 0x7fb9540b8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:13.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.356+0000 7fb951e00640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9380039b0 0x7fb95414fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:13.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.356+0000 7fb951400640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb95415d690 0x7fb9541f82d0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb94000d460 tx=0x7fb94000d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:13.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.356+0000 7fb94a000640 1 -- 172.21.15.45:0/1767507262 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb940016030 con 0x7fb95415d690 2024-04-16T16:16:13.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.356+0000 7fb94a000640 1 -- 172.21.15.45:0/1767507262 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb940007020 con 0x7fb95415d690 2024-04-16T16:16:13.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.356+0000 7fb953200640 1 -- 172.21.15.45:0/1767507262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9541f8c10 con 0x7fb95415d690 2024-04-16T16:16:13.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.356+0000 7fb94a000640 1 -- 172.21.15.45:0/1767507262 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9400073f0 con 0x7fb95415d690 2024-04-16T16:16:13.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.356+0000 7fb953200640 1 -- 172.21.15.45:0/1767507262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9541f91d0 con 0x7fb95415d690 2024-04-16T16:16:13.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.358+0000 7fb953200640 1 -- 172.21.15.45:0/1767507262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9540d0b30 con 0x7fb95415d690 2024-04-16T16:16:13.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.358+0000 7fb94a000640 1 -- 172.21.15.45:0/1767507262 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fb940019070 con 0x7fb95415d690 2024-04-16T16:16:13.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.360+0000 7fb94a000640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb93406cb30 0x7fb93406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:13.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.360+0000 7fb94a000640 1 -- 172.21.15.45:0/1767507262 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7fb94008c3f0 con 0x7fb95415d690 2024-04-16T16:16:13.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.360+0000 7fb951e00640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb93406cb30 0x7fb93406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:13.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.362+0000 7fb951e00640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb93406cb30 0x7fb93406eff0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb944036000 tx=0x7fb944030000 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:13.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.364+0000 7fb94a000640 1 -- 172.21.15.45:0/1767507262 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb9540d0b30 con 0x7fb95415d690 2024-04-16T16:16:13.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.568+0000 7fb953200640 1 -- 172.21.15.45:0/1767507262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs2_data", "var": "pg_num"} v 0) -- 0x7fb9540d25a0 con 0x7fb95415d690 2024-04-16T16:16:13.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.569+0000 7fb94a000640 1 -- 172.21.15.45:0/1767507262 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs2_data", "var": "pg_num"}]=0 v57) ==== 97+0+11 (secure 0 0 0) 0x7fb954150220 con 0x7fb95415d690 2024-04-16T16:16:13.568 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:16:13.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.572+0000 7fb953200640 1 -- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb93406cb30 msgr2=0x7fb93406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:13.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.572+0000 7fb953200640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb93406cb30 0x7fb93406eff0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb944036000 tx=0x7fb944030000 comp rx=0 tx=0).stop 2024-04-16T16:16:13.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.572+0000 7fb953200640 1 -- 172.21.15.45:0/1767507262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb95415d690 msgr2=0x7fb9541f82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:13.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.572+0000 7fb953200640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb95415d690 0x7fb9541f82d0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb94000d460 tx=0x7fb94000d920 comp rx=0 tx=0).stop 2024-04-16T16:16:13.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.572+0000 7fb953200640 1 -- 172.21.15.45:0/1767507262 shutdown_connections 2024-04-16T16:16:13.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.572+0000 7fb953200640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb93406cb30 0x7fb93406eff0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.572+0000 7fb953200640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb95415c630 0x7fb9540b8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.572+0000 7fb953200640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb95415d690 0x7fb9541f82d0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.572+0000 7fb953200640 1 --2- 172.21.15.45:0/1767507262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9380039b0 0x7fb95414fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.572+0000 7fb953200640 1 -- 172.21.15.45:0/1767507262 >> 172.21.15.45:0/1767507262 conn(0x7fb95406cf90 msgr2=0x7fb9540d4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:13.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.572+0000 7fb953200640 1 -- 172.21.15.45:0/1767507262 shutdown_connections 2024-04-16T16:16:13.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.572+0000 7fb953200640 1 -- 172.21.15.45:0/1767507262 wait complete. 2024-04-16T16:16:13.590 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:16:13.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.796+0000 7f0acf200640 1 Processor -- start 2024-04-16T16:16:13.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.796+0000 7f0acf200640 1 -- start start 2024-04-16T16:16:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.798+0000 7f0acf200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0ad015ccb0 0x7f0ad006df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.798+0000 7f0acf200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ad015d360 con 0x7f0ad006e480 2024-04-16T16:16:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.798+0000 7f0acf200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ad015d640 con 0x7f0ad0146220 2024-04-16T16:16:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.798+0000 7f0acf200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0ad015d130 con 0x7f0ad015ccb0 2024-04-16T16:16:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.798+0000 7f0ace800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0ad006e480 0x7f0ad01608a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47080/0 (socket says 172.21.15.45:47080) 2024-04-16T16:16:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.798+0000 7f0ace800640 1 -- 172.21.15.45:0/1057233850 learned_addr learned my addr 172.21.15.45:0/1057233850 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:13.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.798+0000 7f0acd400640 1 --2- 172.21.15.45:0/1057233850 >> v2:172.21.15.103:3300/0 conn(0x7f0ad015ccb0 0x7f0ad006df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:13.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.799+0000 7f0acca00640 1 -- 172.21.15.45:0/1057233850 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1590479305 0 0) 0x7f0ad015d360 con 0x7f0ad006e480 2024-04-16T16:16:13.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.799+0000 7f0acca00640 1 -- 172.21.15.45:0/1057233850 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ab4003520 con 0x7f0ad006e480 2024-04-16T16:16:13.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.799+0000 7f0acd400640 1 -- 172.21.15.45:0/1057233850 >> v1:172.21.15.103:6789/0 conn(0x7f0ad0146220 legacy=0x7f0ad006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:13.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.799+0000 7f0acd400640 1 -- 172.21.15.45:0/1057233850 >> v1:172.21.15.45:6789/0 conn(0x7f0ad006e480 legacy=0x7f0ad01608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:13.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.799+0000 7f0acd400640 1 -- 172.21.15.45:0/1057233850 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad015d920 con 0x7f0ad015ccb0 2024-04-16T16:16:13.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.801+0000 7f0acd400640 1 --2- 172.21.15.45:0/1057233850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ad015ccb0 0x7f0ad006df40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0097a0 tx=0x7f0abc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e0466e4edee95f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:13.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.804+0000 7f0acca00640 1 -- 172.21.15.45:0/1057233850 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc034070 con 0x7f0ad015ccb0 2024-04-16T16:16:13.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.804+0000 7f0acca00640 1 -- 172.21.15.45:0/1057233850 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0abc030610 con 0x7f0ad015ccb0 2024-04-16T16:16:13.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.804+0000 7f0acca00640 1 -- 172.21.15.45:0/1057233850 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc0309e0 con 0x7f0ad015ccb0 2024-04-16T16:16:13.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.805+0000 7f0acf200640 1 -- 172.21.15.45:0/1057233850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ad015ccb0 msgr2=0x7f0ad006df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:13.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.805+0000 7f0acf200640 1 --2- 172.21.15.45:0/1057233850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ad015ccb0 0x7f0ad006df40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0097a0 tx=0x7f0abc02b960 comp rx=0 tx=0).stop 2024-04-16T16:16:13.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.805+0000 7f0acf200640 1 -- 172.21.15.45:0/1057233850 shutdown_connections 2024-04-16T16:16:13.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.805+0000 7f0acf200640 1 --2- 172.21.15.45:0/1057233850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ad015ccb0 0x7f0ad006df40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.805+0000 7f0acf200640 1 -- 172.21.15.45:0/1057233850 >> 172.21.15.45:0/1057233850 conn(0x7f0ad006b930 msgr2=0x7f0ad006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:13.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.805+0000 7f0acf200640 1 -- 172.21.15.45:0/1057233850 shutdown_connections 2024-04-16T16:16:13.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.805+0000 7f0acf200640 1 -- 172.21.15.45:0/1057233850 wait complete. 2024-04-16T16:16:13.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.807+0000 7f0acf200640 1 Processor -- start 2024-04-16T16:16:13.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.807+0000 7f0acf200640 1 -- start start 2024-04-16T16:16:13.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.808+0000 7f0acf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ad006e480 0x7f0ad01456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:13.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.808+0000 7f0acf200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad0146220 0x7f0ad015c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:13.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.808+0000 7f0acf200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ad015ccb0 0x7f0ad0142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:13.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.808+0000 7f0acf200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad0160fb0 con 0x7f0ad0146220 2024-04-16T16:16:13.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.808+0000 7f0acf200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad0160410 con 0x7f0ad015ccb0 2024-04-16T16:16:13.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.808+0000 7f0acf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0ad0161130 con 0x7f0ad006e480 2024-04-16T16:16:13.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.808+0000 7f0acd400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad0146220 0x7f0ad015c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:13.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.809+0000 7f0acd400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad0146220 0x7f0ad015c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49226/0 (socket says 172.21.15.45:49226) 2024-04-16T16:16:13.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.809+0000 7f0acd400640 1 -- 172.21.15.45:0/1139595698 learned_addr learned my addr 172.21.15.45:0/1139595698 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:13.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.809+0000 7f0acde00640 1 --2- 172.21.15.45:0/1139595698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ad006e480 0x7f0ad01456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:13.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.809+0000 7f0ace800640 1 --2- 172.21.15.45:0/1139595698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ad015ccb0 0x7f0ad0142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:13.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.809+0000 7f0acd400640 1 -- 172.21.15.45:0/1139595698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ad006e480 msgr2=0x7f0ad01456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:13.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.809+0000 7f0acd400640 1 --2- 172.21.15.45:0/1139595698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ad006e480 0x7f0ad01456f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.809+0000 7f0acd400640 1 -- 172.21.15.45:0/1139595698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ad015ccb0 msgr2=0x7f0ad0142220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:13.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.809+0000 7f0acd400640 1 --2- 172.21.15.45:0/1139595698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ad015ccb0 0x7f0ad0142220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:13.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.809+0000 7f0acd400640 1 -- 172.21.15.45:0/1139595698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad01f5680 con 0x7f0ad0146220 2024-04-16T16:16:13.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.809+0000 7f0acd400640 1 --2- 172.21.15.45:0/1139595698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad0146220 0x7f0ad015c3f0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0098c0 tx=0x7f0abc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:13.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.810+0000 7f0ac6000640 1 -- 172.21.15.45:0/1139595698 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc034040 con 0x7f0ad0146220 2024-04-16T16:16:13.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.810+0000 7f0ac6000640 1 -- 172.21.15.45:0/1139595698 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0abc03aaa0 con 0x7f0ad0146220 2024-04-16T16:16:13.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.810+0000 7f0ac6000640 1 -- 172.21.15.45:0/1139595698 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc0443f0 con 0x7f0ad0146220 2024-04-16T16:16:13.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.810+0000 7f0acf200640 1 -- 172.21.15.45:0/1139595698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ad01f53a0 con 0x7f0ad0146220 2024-04-16T16:16:13.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.810+0000 7f0acf200640 1 -- 172.21.15.45:0/1139595698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ad0145030 con 0x7f0ad0146220 2024-04-16T16:16:13.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.811+0000 7f0acf200640 1 -- 172.21.15.45:0/1139595698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ad0075780 con 0x7f0ad0146220 2024-04-16T16:16:13.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.812+0000 7f0ac6000640 1 -- 172.21.15.45:0/1139595698 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f0abc04b060 con 0x7f0ad0146220 2024-04-16T16:16:13.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.813+0000 7f0ac6000640 1 --2- 172.21.15.45:0/1139595698 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0ab006cb30 0x7f0ab006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:13.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.813+0000 7f0ac6000640 1 -- 172.21.15.45:0/1139595698 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7f0abc0ae630 con 0x7f0ad0146220 2024-04-16T16:16:13.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.814+0000 7f0acde00640 1 --2- 172.21.15.45:0/1139595698 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0ab006cb30 0x7f0ab006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:13.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.815+0000 7f0acde00640 1 --2- 172.21.15.45:0/1139595698 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0ab006cb30 0x7f0ab006eff0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0000f60 tx=0x7f0ac0009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:13.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:13.817+0000 7f0ac6000640 1 -- 172.21.15.45:0/1139595698 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0abc03f070 con 0x7f0ad0146220 2024-04-16T16:16:14.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.056+0000 7f0acf200640 1 -- 172.21.15.45:0/1139595698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ad00cff60 con 0x7f0ad0146220 2024-04-16T16:16:14.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.058+0000 7f0ac6000640 1 -- 172.21.15.45:0/1139595698 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6491 (secure 0 0 0) 0x7f0abc04b350 con 0x7f0ad0146220 2024-04-16T16:16:14.057 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:14.057 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":40,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/1497040472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1497040472},{"type":"v1","addr":"172.21.15.45:6834","nonce":1497040472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":4907,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/93057991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":93057991},{"type":"v1","addr":"172.21.15.45:6835","nonce":93057991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:14.878986+0000","modified":"2024-04-16T16:15:44.272257+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5188":{"gid":5188,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.103:6836/2626302738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2626302738},{"type":"v1","addr":"172.21.15.103:6836","nonce":2626302738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"fs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5188,"qdb_cluster":[5188]},"id":3},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:35.289901+0000","modified":"2024-04-16T16:15:44.272258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"b","rank":0,"incarnation":35,"state":"up:active","state_seq":4,"addr":"172.21.15.103:6837/1576112884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1576112884},{"type":"v1","addr":"172.21.15.103:6837","nonce":1576112884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"fs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":4}]} 2024-04-16T16:16:14.058 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 40 2024-04-16T16:16:14.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.061+0000 7f0acf200640 1 -- 172.21.15.45:0/1139595698 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0ab006cb30 msgr2=0x7f0ab006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:14.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.061+0000 7f0acf200640 1 --2- 172.21.15.45:0/1139595698 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0ab006cb30 0x7f0ab006eff0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0000f60 tx=0x7f0ac0009040 comp rx=0 tx=0).stop 2024-04-16T16:16:14.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.061+0000 7f0acf200640 1 -- 172.21.15.45:0/1139595698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad0146220 msgr2=0x7f0ad015c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:14.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.061+0000 7f0acf200640 1 --2- 172.21.15.45:0/1139595698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad0146220 0x7f0ad015c3f0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0098c0 tx=0x7f0abc0046c0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.061+0000 7f0acf200640 1 -- 172.21.15.45:0/1139595698 shutdown_connections 2024-04-16T16:16:14.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.061+0000 7f0acf200640 1 --2- 172.21.15.45:0/1139595698 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0ab006cb30 0x7f0ab006eff0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.061+0000 7f0acf200640 1 --2- 172.21.15.45:0/1139595698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ad015ccb0 0x7f0ad0142220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.061+0000 7f0acf200640 1 --2- 172.21.15.45:0/1139595698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad0146220 0x7f0ad015c3f0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.061+0000 7f0acf200640 1 --2- 172.21.15.45:0/1139595698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ad006e480 0x7f0ad01456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.061+0000 7f0acf200640 1 -- 172.21.15.45:0/1139595698 >> 172.21.15.45:0/1139595698 conn(0x7f0ad006b930 msgr2=0x7f0ad00c6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:14.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.061+0000 7f0acf200640 1 -- 172.21.15.45:0/1139595698 shutdown_connections 2024-04-16T16:16:14.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.061+0000 7f0acf200640 1 -- 172.21.15.45:0/1139595698 wait complete. 2024-04-16T16:16:14.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:16:14.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.294+0000 7fdd34c00640 1 Processor -- start 2024-04-16T16:16:14.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.295+0000 7fdd34c00640 1 -- start start 2024-04-16T16:16:14.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.296+0000 7fdd34c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd30147840 0x7fdd3015fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:14.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.296+0000 7fdd34c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdd300aed90 0x7fdd30072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:14.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.296+0000 7fdd34c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd30147c20 con 0x7fdd300728c0 2024-04-16T16:16:14.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.296+0000 7fdd34c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fdd300afe80 con 0x7fdd300aed90 2024-04-16T16:16:14.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.296+0000 7fdd34c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdd300af270 con 0x7fdd30147840 2024-04-16T16:16:14.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.297+0000 7fdd2e800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdd300728c0 0x7fdd30162610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47096/0 (socket says 172.21.15.45:47096) 2024-04-16T16:16:14.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.297+0000 7fdd2e800640 1 -- 172.21.15.45:0/1961038248 learned_addr learned my addr 172.21.15.45:0/1961038248 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:14.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.297+0000 7fdd27e00640 1 --2- 172.21.15.45:0/1961038248 >> v2:172.21.15.103:3300/0 conn(0x7fdd300aed90 0x7fdd30072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:14.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.297+0000 7fdd2de00640 1 --2- 172.21.15.45:0/1961038248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd30147840 0x7fdd3015fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:14.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.297+0000 7fdd2d400640 1 -- 172.21.15.45:0/1961038248 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 916145269 0 0) 0x7fdd30147c20 con 0x7fdd300728c0 2024-04-16T16:16:14.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.297+0000 7fdd2d400640 1 -- 172.21.15.45:0/1961038248 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd14003520 con 0x7fdd300728c0 2024-04-16T16:16:14.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.298+0000 7fdd2d400640 1 -- 172.21.15.45:0/1961038248 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2843899469 0 0) 0x7fdd14003520 con 0x7fdd300728c0 2024-04-16T16:16:14.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.298+0000 7fdd2d400640 1 -- 172.21.15.45:0/1961038248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd30147840 msgr2=0x7fdd3015fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:14.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.298+0000 7fdd2d400640 1 --2- 172.21.15.45:0/1961038248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd30147840 0x7fdd3015fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.298+0000 7fdd2d400640 1 -- 172.21.15.45:0/1961038248 >> v2:172.21.15.103:3300/0 conn(0x7fdd300aed90 msgr2=0x7fdd30072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:14.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.298+0000 7fdd2d400640 1 --2- 172.21.15.45:0/1961038248 >> v2:172.21.15.103:3300/0 conn(0x7fdd300aed90 0x7fdd30072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.298+0000 7fdd2d400640 1 -- 172.21.15.45:0/1961038248 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd300af550 con 0x7fdd300728c0 2024-04-16T16:16:14.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.298+0000 7fdd2d400640 1 -- 172.21.15.45:0/1961038248 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fdd280019f0 con 0x7fdd300728c0 2024-04-16T16:16:14.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.298+0000 7fdd2d400640 1 -- 172.21.15.45:0/1961038248 >> v1:172.21.15.45:6789/0 conn(0x7fdd300728c0 legacy=0x7fdd30162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:14.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.298+0000 7fdd2d400640 1 --2- 172.21.15.45:0/1961038248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd30147840 0x7fdd140039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:14.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.298+0000 7fdd2d400640 1 --2- 172.21.15.45:0/1961038248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd14003f30 0x7fdd14004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:14.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.298+0000 7fdd2d400640 1 --2- 172.21.15.45:0/1961038248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd14004d40 0x7fdd140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:14.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.298+0000 7fdd2d400640 1 -- 172.21.15.45:0/1961038248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdd300afe80 con 0x7fdd14004d40 2024-04-16T16:16:14.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.298+0000 7fdd2d400640 1 -- 172.21.15.45:0/1961038248 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdd300af270 con 0x7fdd14003f30 2024-04-16T16:16:14.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.298+0000 7fdd2d400640 1 -- 172.21.15.45:0/1961038248 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdd1400b9c0 con 0x7fdd30147840 2024-04-16T16:16:14.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.298+0000 7fdd2e800640 1 --2- 172.21.15.45:0/1961038248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd14004d40 0x7fdd140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:14.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.299+0000 7fdd2de00640 1 --2- 172.21.15.45:0/1961038248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd30147840 0x7fdd140039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:14.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.299+0000 7fdd27e00640 1 --2- 172.21.15.45:0/1961038248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd14003f30 0x7fdd14004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:14.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.299+0000 7fdd2e800640 1 -- 172.21.15.45:0/1961038248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd30147840 msgr2=0x7fdd140039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:14.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.299+0000 7fdd2e800640 1 --2- 172.21.15.45:0/1961038248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd30147840 0x7fdd140039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.299+0000 7fdd2e800640 1 -- 172.21.15.45:0/1961038248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd14003f30 msgr2=0x7fdd14004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:14.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.299+0000 7fdd2e800640 1 --2- 172.21.15.45:0/1961038248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd14003f30 0x7fdd14004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.299+0000 7fdd2e800640 1 -- 172.21.15.45:0/1961038248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd14003520 con 0x7fdd14004d40 2024-04-16T16:16:14.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.299+0000 7fdd27e00640 1 --2- 172.21.15.45:0/1961038248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd14003f30 0x7fdd14004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:14.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.300+0000 7fdd2e800640 1 --2- 172.21.15.45:0/1961038248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd14004d40 0x7fdd140051c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800a920 tx=0x7fdd14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b097fe00d2ecb19f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:14.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.300+0000 7fdd2d400640 1 -- 172.21.15.45:0/1961038248 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2800a4c0 con 0x7fdd14004d40 2024-04-16T16:16:14.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.300+0000 7fdd2d400640 1 -- 172.21.15.45:0/1961038248 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd28034650 con 0x7fdd14004d40 2024-04-16T16:16:14.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.300+0000 7fdd2d400640 1 -- 172.21.15.45:0/1961038248 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28034a80 con 0x7fdd14004d40 2024-04-16T16:16:14.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.301+0000 7fdd34c00640 1 -- 172.21.15.45:0/1961038248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd14004d40 msgr2=0x7fdd140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:14.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.301+0000 7fdd34c00640 1 --2- 172.21.15.45:0/1961038248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd14004d40 0x7fdd140051c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800a920 tx=0x7fdd14005830 comp rx=0 tx=0).stop 2024-04-16T16:16:14.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.301+0000 7fdd2e800640 1 -- 172.21.15.45:0/1961038248 reap_dead start 2024-04-16T16:16:14.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.301+0000 7fdd34c00640 1 -- 172.21.15.45:0/1961038248 shutdown_connections 2024-04-16T16:16:14.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.301+0000 7fdd34c00640 1 -- 172.21.15.45:0/1961038248 >> 172.21.15.45:0/1961038248 conn(0x7fdd300b2750 msgr2=0x7fdd300b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:14.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.301+0000 7fdd34c00640 1 -- 172.21.15.45:0/1961038248 shutdown_connections 2024-04-16T16:16:14.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.301+0000 7fdd34c00640 1 -- 172.21.15.45:0/1961038248 wait complete. 2024-04-16T16:16:14.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.303+0000 7fdd34c00640 1 Processor -- start 2024-04-16T16:16:14.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.303+0000 7fdd34c00640 1 -- start start 2024-04-16T16:16:14.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.304+0000 7fdd34c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd14004d40 0x7fdd30146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:14.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.304+0000 7fdd34c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd30143fb0 0x7fdd3015dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:14.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.304+0000 7fdd34c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30161690 0x7fdd30161b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:14.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.304+0000 7fdd34c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3006f2c0 con 0x7fdd30143fb0 2024-04-16T16:16:14.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.304+0000 7fdd34c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdd300afb10 con 0x7fdd30161690 2024-04-16T16:16:14.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.304+0000 7fdd34c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdd3006f440 con 0x7fdd14004d40 2024-04-16T16:16:14.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.305+0000 7fdd27e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd30143fb0 0x7fdd3015dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:14.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.305+0000 7fdd27e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd30143fb0 0x7fdd3015dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49248/0 (socket says 172.21.15.45:49248) 2024-04-16T16:16:14.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.305+0000 7fdd27e00640 1 -- 172.21.15.45:0/1395771953 learned_addr learned my addr 172.21.15.45:0/1395771953 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:14.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.305+0000 7fdd2de00640 1 --2- 172.21.15.45:0/1395771953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd14004d40 0x7fdd30146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:14.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.305+0000 7fdd2e800640 1 --2- 172.21.15.45:0/1395771953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30161690 0x7fdd30161b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:14.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.305+0000 7fdd27e00640 1 -- 172.21.15.45:0/1395771953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd14004d40 msgr2=0x7fdd30146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:14.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.305+0000 7fdd27e00640 1 --2- 172.21.15.45:0/1395771953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd14004d40 0x7fdd30146470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.305+0000 7fdd27e00640 1 -- 172.21.15.45:0/1395771953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30161690 msgr2=0x7fdd30161b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:14.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.305+0000 7fdd27e00640 1 --2- 172.21.15.45:0/1395771953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30161690 0x7fdd30161b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.305+0000 7fdd27e00640 1 -- 172.21.15.45:0/1395771953 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd30146bc0 con 0x7fdd30143fb0 2024-04-16T16:16:14.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.305+0000 7fdd27e00640 1 --2- 172.21.15.45:0/1395771953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd30143fb0 0x7fdd3015dee0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fdd18010a00 tx=0x7fdd18010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:14.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.306+0000 7fdd26000640 1 -- 172.21.15.45:0/1395771953 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1800f040 con 0x7fdd30143fb0 2024-04-16T16:16:14.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.306+0000 7fdd26000640 1 -- 172.21.15.45:0/1395771953 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd1800ec20 con 0x7fdd30143fb0 2024-04-16T16:16:14.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.306+0000 7fdd26000640 1 -- 172.21.15.45:0/1395771953 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1801c400 con 0x7fdd30143fb0 2024-04-16T16:16:14.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.306+0000 7fdd34c00640 1 -- 172.21.15.45:0/1395771953 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd30146ea0 con 0x7fdd30143fb0 2024-04-16T16:16:14.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.306+0000 7fdd34c00640 1 -- 172.21.15.45:0/1395771953 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd3015e420 con 0x7fdd30143fb0 2024-04-16T16:16:14.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.307+0000 7fdd34c00640 1 -- 172.21.15.45:0/1395771953 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd300b2570 con 0x7fdd30143fb0 2024-04-16T16:16:14.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.308+0000 7fdd26000640 1 -- 172.21.15.45:0/1395771953 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fdd1801c5a0 con 0x7fdd30143fb0 2024-04-16T16:16:14.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.309+0000 7fdd26000640 1 --2- 172.21.15.45:0/1395771953 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdd0c06cb60 0x7fdd0c06f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:14.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.309+0000 7fdd26000640 1 -- 172.21.15.45:0/1395771953 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7fdd1808cdf0 con 0x7fdd30143fb0 2024-04-16T16:16:14.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.310+0000 7fdd2de00640 1 --2- 172.21.15.45:0/1395771953 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdd0c06cb60 0x7fdd0c06f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:14.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.311+0000 7fdd2de00640 1 --2- 172.21.15.45:0/1395771953 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdd0c06cb60 0x7fdd0c06f020 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009950 tx=0x7fdd20008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:14.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.314+0000 7fdd26000640 1 -- 172.21.15.45:0/1395771953 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd300b2570 con 0x7fdd30143fb0 2024-04-16T16:16:14.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.523+0000 7fdd34c00640 1 -- 172.21.15.45:0/1395771953 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdd30075780 con 0x7fdd30143fb0 2024-04-16T16:16:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.525+0000 7fdd26000640 1 -- 172.21.15.45:0/1395771953 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+21917 (secure 0 0 0) 0x7fdd18018030 con 0x7fdd30143fb0 2024-04-16T16:16:14.525 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:14.525 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":57,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:16:07.016528+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"fs1_metadata","create_time":"2024-04-16T16:15:13.250782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"fs1_data","create_time":"2024-04-16T16:15:13.856986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"fs2_metadata","create_time":"2024-04-16T16:15:33.215042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"fs2_data","create_time":"2024-04-16T16:15:34.271426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.2c","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/349066714":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/388389014":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/589483918":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3353543342":"2024-04-17T16:16:07.016480+0000","172.21.15.103:6832/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3310077212":"2024-04-17T16:16:07.016480+0000","172.21.15.45:6833/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6832/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.45:6833/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:6833/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6834/88032513":"2024-04-17T16:15:41.221894+0000","172.21.15.45:6835/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:0/644374842":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6835/88032513":"2024-04-17T16:15:41.221894+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:16:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.528+0000 7fdd34c00640 1 -- 172.21.15.45:0/1395771953 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdd0c06cb60 msgr2=0x7fdd0c06f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.528+0000 7fdd34c00640 1 --2- 172.21.15.45:0/1395771953 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdd0c06cb60 0x7fdd0c06f020 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009950 tx=0x7fdd20008040 comp rx=0 tx=0).stop 2024-04-16T16:16:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.528+0000 7fdd34c00640 1 -- 172.21.15.45:0/1395771953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd30143fb0 msgr2=0x7fdd3015dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.528+0000 7fdd34c00640 1 --2- 172.21.15.45:0/1395771953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd30143fb0 0x7fdd3015dee0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fdd18010a00 tx=0x7fdd18010ec0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.528+0000 7fdd34c00640 1 -- 172.21.15.45:0/1395771953 shutdown_connections 2024-04-16T16:16:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.528+0000 7fdd34c00640 1 --2- 172.21.15.45:0/1395771953 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdd0c06cb60 0x7fdd0c06f020 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.528+0000 7fdd34c00640 1 --2- 172.21.15.45:0/1395771953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30161690 0x7fdd30161b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.528+0000 7fdd34c00640 1 --2- 172.21.15.45:0/1395771953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd30143fb0 0x7fdd3015dee0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.528+0000 7fdd34c00640 1 --2- 172.21.15.45:0/1395771953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd14004d40 0x7fdd30146470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.529+0000 7fdd34c00640 1 -- 172.21.15.45:0/1395771953 >> 172.21.15.45:0/1395771953 conn(0x7fdd300b2750 msgr2=0x7fdd30142630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.529+0000 7fdd34c00640 1 -- 172.21.15.45:0/1395771953 shutdown_connections 2024-04-16T16:16:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.529+0000 7fdd34c00640 1 -- 172.21.15.45:0/1395771953 wait complete. 2024-04-16T16:16:14.546 INFO:tasks.cephfs.filesystem:Destroying file system fs1 and related pools 2024-04-16T16:16:14.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:16:14.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.764+0000 7fc1df800640 1 Processor -- start 2024-04-16T16:16:14.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.765+0000 7fc1df800640 1 -- start start 2024-04-16T16:16:14.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.766+0000 7fc1df800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc1d809ae30 0x7fc1d80720d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:14.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.766+0000 7fc1df800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1d809b210 0x7fc1d815fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:14.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.766+0000 7fc1df800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1d809be50 con 0x7fc1d80717f0 2024-04-16T16:16:14.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.766+0000 7fc1df800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc1d809bc00 con 0x7fc1d809b210 2024-04-16T16:16:14.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.766+0000 7fc1df800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc1d80cdc00 con 0x7fc1d809ae30 2024-04-16T16:16:14.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.766+0000 7fc1d7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1d809b210 0x7fc1d815fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:14.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.767+0000 7fc1d7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1d809b210 0x7fc1d815fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49264/0 (socket says 172.21.15.45:49264) 2024-04-16T16:16:14.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.767+0000 7fc1d7e00640 1 -- 172.21.15.45:0/3551588216 learned_addr learned my addr 172.21.15.45:0/3551588216 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:14.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.767+0000 7fc1dce00640 1 --2- 172.21.15.45:0/3551588216 >> v2:172.21.15.103:3300/0 conn(0x7fc1d809ae30 0x7fc1d80720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:14.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.767+0000 7fc1d7e00640 1 -- 172.21.15.45:0/3551588216 >> v2:172.21.15.103:3300/0 conn(0x7fc1d809ae30 msgr2=0x7fc1d80720d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:14.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.767+0000 7fc1d7e00640 1 --2- 172.21.15.45:0/3551588216 >> v2:172.21.15.103:3300/0 conn(0x7fc1d809ae30 0x7fc1d80720d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.767+0000 7fc1d7e00640 1 -- 172.21.15.45:0/3551588216 >> v1:172.21.15.45:6789/0 conn(0x7fc1d80717f0 legacy=0x7fc1d8070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:14.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.767+0000 7fc1d7e00640 1 -- 172.21.15.45:0/3551588216 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d80cd5b0 con 0x7fc1d809b210 2024-04-16T16:16:14.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.768+0000 7fc1d7e00640 1 --2- 172.21.15.45:0/3551588216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d809b210 0x7fc1d815fd60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8073360 tx=0x7fc1c802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3539a2c260f34d9e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:14.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.769+0000 7fc1d7400640 1 -- 172.21.15.45:0/3551588216 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8034070 con 0x7fc1d809b210 2024-04-16T16:16:14.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.769+0000 7fc1d7400640 1 -- 172.21.15.45:0/3551588216 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1c8030860 con 0x7fc1d809b210 2024-04-16T16:16:14.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.769+0000 7fc1d7400640 1 -- 172.21.15.45:0/3551588216 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8030c30 con 0x7fc1d809b210 2024-04-16T16:16:14.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.769+0000 7fc1df800640 1 -- 172.21.15.45:0/3551588216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d809b210 msgr2=0x7fc1d815fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:14.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.769+0000 7fc1df800640 1 --2- 172.21.15.45:0/3551588216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d809b210 0x7fc1d815fd60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8073360 tx=0x7fc1c802b9a0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.769+0000 7fc1df800640 1 -- 172.21.15.45:0/3551588216 shutdown_connections 2024-04-16T16:16:14.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.769+0000 7fc1df800640 1 --2- 172.21.15.45:0/3551588216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d809b210 0x7fc1d815fd60 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.769+0000 7fc1df800640 1 --2- 172.21.15.45:0/3551588216 >> v2:172.21.15.103:3300/0 conn(0x7fc1d809ae30 0x7fc1d80720d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.769+0000 7fc1df800640 1 -- 172.21.15.45:0/3551588216 >> 172.21.15.45:0/3551588216 conn(0x7fc1d80b4350 msgr2=0x7fc1d80b4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:14.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.770+0000 7fc1df800640 1 -- 172.21.15.45:0/3551588216 shutdown_connections 2024-04-16T16:16:14.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.770+0000 7fc1df800640 1 -- 172.21.15.45:0/3551588216 wait complete. 2024-04-16T16:16:14.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.771+0000 7fc1df800640 1 Processor -- start 2024-04-16T16:16:14.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.771+0000 7fc1df800640 1 -- start start 2024-04-16T16:16:14.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.772+0000 7fc1df800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d80717f0 0x7fc1d80b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:14.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.772+0000 7fc1df800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d809ae30 0x7fc1d815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:14.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.772+0000 7fc1df800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1d809b210 0x7fc1d80b0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:14.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.772+0000 7fc1df800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d80af610 con 0x7fc1d809ae30 2024-04-16T16:16:14.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.772+0000 7fc1df800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d80cf0a0 con 0x7fc1d80717f0 2024-04-16T16:16:14.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.772+0000 7fc1df800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc1d80cd1d0 con 0x7fc1d809b210 2024-04-16T16:16:14.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.773+0000 7fc1d7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d809ae30 0x7fc1d815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:14.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.773+0000 7fc1d7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d809ae30 0x7fc1d815ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49268/0 (socket says 172.21.15.45:49268) 2024-04-16T16:16:14.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.773+0000 7fc1d7e00640 1 -- 172.21.15.45:0/4124636274 learned_addr learned my addr 172.21.15.45:0/4124636274 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:14.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.773+0000 7fc1dce00640 1 --2- 172.21.15.45:0/4124636274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d80717f0 0x7fc1d80b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:14.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.773+0000 7fc1dd800640 1 --2- 172.21.15.45:0/4124636274 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1d809b210 0x7fc1d80b0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:14.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.773+0000 7fc1d7e00640 1 -- 172.21.15.45:0/4124636274 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1d809b210 msgr2=0x7fc1d80b0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:14.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.773+0000 7fc1d7e00640 1 --2- 172.21.15.45:0/4124636274 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1d809b210 0x7fc1d80b0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.773+0000 7fc1d7e00640 1 -- 172.21.15.45:0/4124636274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d80717f0 msgr2=0x7fc1d80b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:14.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.773+0000 7fc1d7e00640 1 --2- 172.21.15.45:0/4124636274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d80717f0 0x7fc1d80b5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:14.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.773+0000 7fc1d7e00640 1 -- 172.21.15.45:0/4124636274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d815e2f0 con 0x7fc1d809ae30 2024-04-16T16:16:14.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.774+0000 7fc1d7e00640 1 --2- 172.21.15.45:0/4124636274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d809ae30 0x7fc1d815ddb0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc1c802b9e0 tx=0x7fc1c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:14.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.774+0000 7fc1d4c00640 1 -- 172.21.15.45:0/4124636274 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8034040 con 0x7fc1d809ae30 2024-04-16T16:16:14.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.774+0000 7fc1d4c00640 1 -- 172.21.15.45:0/4124636274 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1c8030660 con 0x7fc1d809ae30 2024-04-16T16:16:14.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.774+0000 7fc1d4c00640 1 -- 172.21.15.45:0/4124636274 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c80423f0 con 0x7fc1d809ae30 2024-04-16T16:16:14.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.774+0000 7fc1df800640 1 -- 172.21.15.45:0/4124636274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1d80c65e0 con 0x7fc1d809ae30 2024-04-16T16:16:14.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.774+0000 7fc1df800640 1 -- 172.21.15.45:0/4124636274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1d80c6ba0 con 0x7fc1d809ae30 2024-04-16T16:16:14.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.775+0000 7fc1df800640 1 -- 172.21.15.45:0/4124636274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1d8075780 con 0x7fc1d809ae30 2024-04-16T16:16:14.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.776+0000 7fc1d4c00640 1 -- 172.21.15.45:0/4124636274 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fc1c8037070 con 0x7fc1d809ae30 2024-04-16T16:16:14.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.777+0000 7fc1d4c00640 1 --2- 172.21.15.45:0/4124636274 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc1b406cb30 0x7fc1b406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:14.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.777+0000 7fc1d4c00640 1 -- 172.21.15.45:0/4124636274 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7fc1c80aea80 con 0x7fc1d809ae30 2024-04-16T16:16:14.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.778+0000 7fc1dce00640 1 --2- 172.21.15.45:0/4124636274 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc1b406cb30 0x7fc1b406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:14.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.779+0000 7fc1dce00640 1 --2- 172.21.15.45:0/4124636274 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc1b406cb30 0x7fc1b406eff0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0006fd0 tx=0x7fc1c0008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:14.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:14.782+0000 7fc1d4c00640 1 -- 172.21.15.45:0/4124636274 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc1c807c4c0 con 0x7fc1d809ae30 2024-04-16T16:16:15.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.020+0000 7fc1df800640 1 -- 172.21.15.45:0/4124636274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1d80cc0b0 con 0x7fc1d809ae30 2024-04-16T16:16:15.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.021+0000 7fc1d4c00640 1 -- 172.21.15.45:0/4124636274 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6491 (secure 0 0 0) 0x7fc1c807bdc0 con 0x7fc1d809ae30 2024-04-16T16:16:15.020 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:15.020 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":40,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/1497040472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1497040472},{"type":"v1","addr":"172.21.15.45:6834","nonce":1497040472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":4907,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/93057991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":93057991},{"type":"v1","addr":"172.21.15.45:6835","nonce":93057991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:14.878986+0000","modified":"2024-04-16T16:15:44.272257+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5188":{"gid":5188,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.103:6836/2626302738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2626302738},{"type":"v1","addr":"172.21.15.103:6836","nonce":2626302738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"fs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5188,"qdb_cluster":[5188]},"id":3},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:35.289901+0000","modified":"2024-04-16T16:15:44.272258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"b","rank":0,"incarnation":35,"state":"up:active","state_seq":4,"addr":"172.21.15.103:6837/1576112884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1576112884},{"type":"v1","addr":"172.21.15.103:6837","nonce":1576112884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"fs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":4}]} 2024-04-16T16:16:15.020 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 40 2024-04-16T16:16:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.023+0000 7fc1df800640 1 -- 172.21.15.45:0/4124636274 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc1b406cb30 msgr2=0x7fc1b406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.023+0000 7fc1df800640 1 --2- 172.21.15.45:0/4124636274 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc1b406cb30 0x7fc1b406eff0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0006fd0 tx=0x7fc1c0008040 comp rx=0 tx=0).stop 2024-04-16T16:16:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.023+0000 7fc1df800640 1 -- 172.21.15.45:0/4124636274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d809ae30 msgr2=0x7fc1d815ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.023+0000 7fc1df800640 1 --2- 172.21.15.45:0/4124636274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d809ae30 0x7fc1d815ddb0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc1c802b9e0 tx=0x7fc1c80046c0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.023+0000 7fc1df800640 1 -- 172.21.15.45:0/4124636274 shutdown_connections 2024-04-16T16:16:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.023+0000 7fc1df800640 1 --2- 172.21.15.45:0/4124636274 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc1b406cb30 0x7fc1b406eff0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.023+0000 7fc1df800640 1 --2- 172.21.15.45:0/4124636274 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1d809b210 0x7fc1d80b0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.023+0000 7fc1df800640 1 --2- 172.21.15.45:0/4124636274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d809ae30 0x7fc1d815ddb0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.023+0000 7fc1df800640 1 --2- 172.21.15.45:0/4124636274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d80717f0 0x7fc1d80b5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.023+0000 7fc1df800640 1 -- 172.21.15.45:0/4124636274 >> 172.21.15.45:0/4124636274 conn(0x7fc1d80b4350 msgr2=0x7fc1d8070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.024+0000 7fc1df800640 1 -- 172.21.15.45:0/4124636274 shutdown_connections 2024-04-16T16:16:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.024+0000 7fc1df800640 1 -- 172.21.15.45:0/4124636274 wait complete. 2024-04-16T16:16:15.042 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:16:15.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.264+0000 7f1d0a800640 1 Processor -- start 2024-04-16T16:16:15.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.264+0000 7f1d0a800640 1 -- start start 2024-04-16T16:16:15.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.266+0000 7f1d0a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d04146220 0x7f1d0415ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:15.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.266+0000 7f1d0a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1d0406f7a0 0x7f1d0406df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:15.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.266+0000 7f1d0a800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d0415d360 con 0x7f1d0406e480 2024-04-16T16:16:15.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.266+0000 7f1d0a800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1d04160410 con 0x7f1d0406f7a0 2024-04-16T16:16:15.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.266+0000 7f1d0a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1d0415d640 con 0x7f1d04146220 2024-04-16T16:16:15.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.267+0000 7f1d09400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d04146220 0x7f1d0415ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:15.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.267+0000 7f1d08a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1d0406f7a0 0x7f1d0406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:15.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.267+0000 7f1d08a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1d0406f7a0 0x7f1d0406df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:54868/0 (socket says 172.21.15.45:54868) 2024-04-16T16:16:15.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.267+0000 7f1d08a00640 1 -- 172.21.15.45:0/2134997741 learned_addr learned my addr 172.21.15.45:0/2134997741 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:15.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.267+0000 7f1d09400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d04146220 0x7f1d0415ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:40956/0 (socket says 172.21.15.45:40956) 2024-04-16T16:16:15.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.267+0000 7f1d08a00640 1 -- 172.21.15.45:0/2134997741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d04146220 msgr2=0x7f1d0415ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:15.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.267+0000 7f1d08a00640 1 --2- 172.21.15.45:0/2134997741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d04146220 0x7f1d0415ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.267+0000 7f1d08a00640 1 -- 172.21.15.45:0/2134997741 >> v1:172.21.15.103:6789/0 conn(0x7f1d0406e480 legacy=0x7f1d041608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:15.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.267+0000 7f1d08a00640 1 -- 172.21.15.45:0/2134997741 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d0415d920 con 0x7f1d0406f7a0 2024-04-16T16:16:15.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.269+0000 7f1d08a00640 1 --2- 172.21.15.45:0/2134997741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0406f7a0 0x7f1d0406df40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009870 tx=0x7f1cf002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e879b21df9569745 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:15.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.269+0000 7f1d03e00640 1 -- 172.21.15.45:0/2134997741 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0034070 con 0x7f1d0406f7a0 2024-04-16T16:16:15.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.269+0000 7f1d03e00640 1 -- 172.21.15.45:0/2134997741 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cf0030840 con 0x7f1d0406f7a0 2024-04-16T16:16:15.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.269+0000 7f1d03e00640 1 -- 172.21.15.45:0/2134997741 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0030c10 con 0x7f1d0406f7a0 2024-04-16T16:16:15.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.270+0000 7f1d0a800640 1 -- 172.21.15.45:0/2134997741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0406f7a0 msgr2=0x7f1d0406df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.270+0000 7f1d0a800640 1 --2- 172.21.15.45:0/2134997741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0406f7a0 0x7f1d0406df40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009870 tx=0x7f1cf002b980 comp rx=0 tx=0).stop 2024-04-16T16:16:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.270+0000 7f1d0a800640 1 -- 172.21.15.45:0/2134997741 shutdown_connections 2024-04-16T16:16:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.270+0000 7f1d0a800640 1 --2- 172.21.15.45:0/2134997741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0406f7a0 0x7f1d0406df40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.270+0000 7f1d0a800640 1 --2- 172.21.15.45:0/2134997741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d04146220 0x7f1d0415ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.270+0000 7f1d0a800640 1 -- 172.21.15.45:0/2134997741 >> 172.21.15.45:0/2134997741 conn(0x7f1d0406b930 msgr2=0x7f1d0406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.270+0000 7f1d0a800640 1 -- 172.21.15.45:0/2134997741 shutdown_connections 2024-04-16T16:16:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.271+0000 7f1d0a800640 1 -- 172.21.15.45:0/2134997741 wait complete. 2024-04-16T16:16:15.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.272+0000 7f1d0a800640 1 Processor -- start 2024-04-16T16:16:15.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.272+0000 7f1d0a800640 1 -- start start 2024-04-16T16:16:15.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.273+0000 7f1d0a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0406e480 0x7f1d040ae540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:15.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.273+0000 7f1d0a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d0406f7a0 0x7f1d040b09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:15.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.273+0000 7f1d0a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d04146220 0x7f1d0415c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:15.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.273+0000 7f1d0a800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0406fda0 con 0x7f1d0406f7a0 2024-04-16T16:16:15.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.273+0000 7f1d0a800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0406fc20 con 0x7f1d0406e480 2024-04-16T16:16:15.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.273+0000 7f1d0a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1d04160fb0 con 0x7f1d04146220 2024-04-16T16:16:15.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.273+0000 7f1d08a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d0406f7a0 0x7f1d040b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:15.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.274+0000 7f1d08a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d0406f7a0 0x7f1d040b09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49276/0 (socket says 172.21.15.45:49276) 2024-04-16T16:16:15.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.274+0000 7f1d08a00640 1 -- 172.21.15.45:0/1374160341 learned_addr learned my addr 172.21.15.45:0/1374160341 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:15.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.274+0000 7f1d09400640 1 --2- 172.21.15.45:0/1374160341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0406e480 0x7f1d040ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:15.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.274+0000 7f1d09e00640 1 --2- 172.21.15.45:0/1374160341 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d04146220 0x7f1d0415c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:15.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.274+0000 7f1d08a00640 1 -- 172.21.15.45:0/1374160341 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d04146220 msgr2=0x7f1d0415c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:15.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.274+0000 7f1d08a00640 1 --2- 172.21.15.45:0/1374160341 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d04146220 0x7f1d0415c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.274+0000 7f1d08a00640 1 -- 172.21.15.45:0/1374160341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0406e480 msgr2=0x7f1d040ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:15.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.274+0000 7f1d08a00640 1 --2- 172.21.15.45:0/1374160341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0406e480 0x7f1d040ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.274+0000 7f1d08a00640 1 -- 172.21.15.45:0/1374160341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d040b1100 con 0x7f1d0406f7a0 2024-04-16T16:16:15.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.274+0000 7f1d08a00640 1 --2- 172.21.15.45:0/1374160341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d0406f7a0 0x7f1d040b09b0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009990 tx=0x7f1cf0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:15.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.275+0000 7f1d01600640 1 -- 172.21.15.45:0/1374160341 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0048070 con 0x7f1d0406f7a0 2024-04-16T16:16:15.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.275+0000 7f1d01600640 1 -- 172.21.15.45:0/1374160341 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cf0030640 con 0x7f1d0406f7a0 2024-04-16T16:16:15.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.275+0000 7f1d0a800640 1 -- 172.21.15.45:0/1374160341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d040b13e0 con 0x7f1d0406f7a0 2024-04-16T16:16:15.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.275+0000 7f1d01600640 1 -- 172.21.15.45:0/1374160341 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0034040 con 0x7f1d0406f7a0 2024-04-16T16:16:15.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.275+0000 7f1d0a800640 1 -- 172.21.15.45:0/1374160341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d04143850 con 0x7f1d0406f7a0 2024-04-16T16:16:15.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.276+0000 7f1d0a800640 1 -- 172.21.15.45:0/1374160341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d04075780 con 0x7f1d0406f7a0 2024-04-16T16:16:15.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.277+0000 7f1d01600640 1 -- 172.21.15.45:0/1374160341 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f1cf004b070 con 0x7f1d0406f7a0 2024-04-16T16:16:15.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.278+0000 7f1d01600640 1 --2- 172.21.15.45:0/1374160341 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f1ce406cb30 0x7f1ce406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:15.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.278+0000 7f1d01600640 1 -- 172.21.15.45:0/1374160341 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7f1cf00ae630 con 0x7f1d0406f7a0 2024-04-16T16:16:15.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.278+0000 7f1d09400640 1 --2- 172.21.15.45:0/1374160341 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f1ce406cb30 0x7f1ce406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:15.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.280+0000 7f1d09400640 1 --2- 172.21.15.45:0/1374160341 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f1ce406cb30 0x7f1ce406eff0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8009ad0 tx=0x7f1cf8008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:15.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.282+0000 7f1d01600640 1 -- 172.21.15.45:0/1374160341 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1cf007beb0 con 0x7f1d0406f7a0 2024-04-16T16:16:15.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.519+0000 7f1d0a800640 1 -- 172.21.15.45:0/1374160341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d040cff60 con 0x7f1d0406f7a0 2024-04-16T16:16:15.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.520+0000 7f1d01600640 1 -- 172.21.15.45:0/1374160341 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6491 (secure 0 0 0) 0x7f1cf007b7b0 con 0x7f1d0406f7a0 2024-04-16T16:16:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:15.519 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":40,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/1497040472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1497040472},{"type":"v1","addr":"172.21.15.45:6834","nonce":1497040472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":4907,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/93057991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":93057991},{"type":"v1","addr":"172.21.15.45:6835","nonce":93057991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:14.878986+0000","modified":"2024-04-16T16:15:44.272257+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5188":{"gid":5188,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.103:6836/2626302738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2626302738},{"type":"v1","addr":"172.21.15.103:6836","nonce":2626302738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"fs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5188,"qdb_cluster":[5188]},"id":3},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:35.289901+0000","modified":"2024-04-16T16:15:44.272258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"b","rank":0,"incarnation":35,"state":"up:active","state_seq":4,"addr":"172.21.15.103:6837/1576112884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1576112884},{"type":"v1","addr":"172.21.15.103:6837","nonce":1576112884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"fs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":4}]} 2024-04-16T16:16:15.519 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 40 2024-04-16T16:16:15.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.522+0000 7f1d0a800640 1 -- 172.21.15.45:0/1374160341 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f1ce406cb30 msgr2=0x7f1ce406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:15.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.522+0000 7f1d0a800640 1 --2- 172.21.15.45:0/1374160341 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f1ce406cb30 0x7f1ce406eff0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8009ad0 tx=0x7f1cf8008040 comp rx=0 tx=0).stop 2024-04-16T16:16:15.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.522+0000 7f1d0a800640 1 -- 172.21.15.45:0/1374160341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d0406f7a0 msgr2=0x7f1d040b09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:15.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.522+0000 7f1d0a800640 1 --2- 172.21.15.45:0/1374160341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d0406f7a0 0x7f1d040b09b0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009990 tx=0x7f1cf0004730 comp rx=0 tx=0).stop 2024-04-16T16:16:15.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.523+0000 7f1d0a800640 1 -- 172.21.15.45:0/1374160341 shutdown_connections 2024-04-16T16:16:15.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.523+0000 7f1d0a800640 1 --2- 172.21.15.45:0/1374160341 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f1ce406cb30 0x7f1ce406eff0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.523+0000 7f1d0a800640 1 --2- 172.21.15.45:0/1374160341 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d04146220 0x7f1d0415c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.523+0000 7f1d0a800640 1 --2- 172.21.15.45:0/1374160341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d0406f7a0 0x7f1d040b09b0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.523+0000 7f1d0a800640 1 --2- 172.21.15.45:0/1374160341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0406e480 0x7f1d040ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.523+0000 7f1d0a800640 1 -- 172.21.15.45:0/1374160341 >> 172.21.15.45:0/1374160341 conn(0x7f1d0406b930 msgr2=0x7f1d040c5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:15.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.523+0000 7f1d0a800640 1 -- 172.21.15.45:0/1374160341 shutdown_connections 2024-04-16T16:16:15.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.523+0000 7f1d0a800640 1 -- 172.21.15.45:0/1374160341 wait complete. 2024-04-16T16:16:15.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:16:15.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.760+0000 7fdc7b200640 1 Processor -- start 2024-04-16T16:16:15.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.760+0000 7fdc7b200640 1 -- start start 2024-04-16T16:16:15.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.761+0000 7fdc7b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc7c146220 0x7fdc7c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:15.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.761+0000 7fdc7b200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdc7c1608a0 0x7fdc7c160d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:15.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.761+0000 7fdc7b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7c15d360 con 0x7fdc7c06f7a0 2024-04-16T16:16:15.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.761+0000 7fdc7b200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fdc7c160410 con 0x7fdc7c1608a0 2024-04-16T16:16:15.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.761+0000 7fdc7b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdc7c15d640 con 0x7fdc7c146220 2024-04-16T16:16:15.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.762+0000 7fdc7a800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdc7c1608a0 0x7fdc7c160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:15.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.762+0000 7fdc7a800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdc7c1608a0 0x7fdc7c160d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49290/0 (socket says 172.21.15.45:49290) 2024-04-16T16:16:15.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.762+0000 7fdc7a800640 1 -- 172.21.15.45:0/2826001449 learned_addr learned my addr 172.21.15.45:0/2826001449 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:15.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.762+0000 7fdc79e00640 1 --2- 172.21.15.45:0/2826001449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc7c146220 0x7fdc7c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:15.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.762+0000 7fdc7a800640 1 -- 172.21.15.45:0/2826001449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc7c146220 msgr2=0x7fdc7c15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:15.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.762+0000 7fdc7a800640 1 --2- 172.21.15.45:0/2826001449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc7c146220 0x7fdc7c15ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.762+0000 7fdc7a800640 1 -- 172.21.15.45:0/2826001449 >> v1:172.21.15.103:6789/0 conn(0x7fdc7c06f7a0 legacy=0x7fdc7c06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:15.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.762+0000 7fdc7a800640 1 -- 172.21.15.45:0/2826001449 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7c15d920 con 0x7fdc7c1608a0 2024-04-16T16:16:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.763+0000 7fdc7a800640 1 --2- 172.21.15.45:0/2826001449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc7c1608a0 0x7fdc7c160d30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009870 tx=0x7fdc7402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39af9a69890193b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.764+0000 7fdc78a00640 1 -- 172.21.15.45:0/2826001449 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc74035020 con 0x7fdc7c1608a0 2024-04-16T16:16:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.764+0000 7fdc78a00640 1 -- 172.21.15.45:0/2826001449 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc74031640 con 0x7fdc7c1608a0 2024-04-16T16:16:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.764+0000 7fdc78a00640 1 -- 172.21.15.45:0/2826001449 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7402d040 con 0x7fdc7c1608a0 2024-04-16T16:16:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.764+0000 7fdc7b200640 1 -- 172.21.15.45:0/2826001449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc7c1608a0 msgr2=0x7fdc7c160d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.764+0000 7fdc7b200640 1 --2- 172.21.15.45:0/2826001449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc7c1608a0 0x7fdc7c160d30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009870 tx=0x7fdc7402b8f0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.765+0000 7fdc7b200640 1 -- 172.21.15.45:0/2826001449 shutdown_connections 2024-04-16T16:16:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.765+0000 7fdc7b200640 1 --2- 172.21.15.45:0/2826001449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc7c1608a0 0x7fdc7c160d30 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.765+0000 7fdc7b200640 1 --2- 172.21.15.45:0/2826001449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc7c146220 0x7fdc7c15ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.765+0000 7fdc7b200640 1 -- 172.21.15.45:0/2826001449 >> 172.21.15.45:0/2826001449 conn(0x7fdc7c06b930 msgr2=0x7fdc7c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.765+0000 7fdc7b200640 1 -- 172.21.15.45:0/2826001449 shutdown_connections 2024-04-16T16:16:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.765+0000 7fdc7b200640 1 -- 172.21.15.45:0/2826001449 wait complete. 2024-04-16T16:16:15.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.766+0000 7fdc7b200640 1 Processor -- start 2024-04-16T16:16:15.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.766+0000 7fdc7b200640 1 -- start start 2024-04-16T16:16:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.767+0000 7fdc7b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc7c06f7a0 0x7fdc7c13eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.767+0000 7fdc7b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc7c146220 0x7fdc7c0cee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.767+0000 7fdc7b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc7c1608a0 0x7fdc7c0cf390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.767+0000 7fdc7b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7c06fc20 con 0x7fdc7c06f7a0 2024-04-16T16:16:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.767+0000 7fdc7b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7c15dc00 con 0x7fdc7c146220 2024-04-16T16:16:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.767+0000 7fdc7b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdc7c06fda0 con 0x7fdc7c1608a0 2024-04-16T16:16:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.768+0000 7fdc79e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc7c06f7a0 0x7fdc7c13eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.768+0000 7fdc79e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc7c06f7a0 0x7fdc7c13eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49298/0 (socket says 172.21.15.45:49298) 2024-04-16T16:16:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.768+0000 7fdc79e00640 1 -- 172.21.15.45:0/3997849684 learned_addr learned my addr 172.21.15.45:0/3997849684 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.768+0000 7fdc79e00640 1 -- 172.21.15.45:0/3997849684 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc7c1608a0 msgr2=0x7fdc7c0cf390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.768+0000 7fdc79e00640 1 --2- 172.21.15.45:0/3997849684 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc7c1608a0 0x7fdc7c0cf390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.768+0000 7fdc79400640 1 --2- 172.21.15.45:0/3997849684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc7c146220 0x7fdc7c0cee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.768+0000 7fdc79e00640 1 -- 172.21.15.45:0/3997849684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc7c146220 msgr2=0x7fdc7c0cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.768+0000 7fdc79e00640 1 --2- 172.21.15.45:0/3997849684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc7c146220 0x7fdc7c0cee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.768+0000 7fdc79e00640 1 -- 172.21.15.45:0/3997849684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7c13ddc0 con 0x7fdc7c06f7a0 2024-04-16T16:16:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.768+0000 7fdc79400640 1 --2- 172.21.15.45:0/3997849684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc7c146220 0x7fdc7c0cee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.768+0000 7fdc79e00640 1 --2- 172.21.15.45:0/3997849684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc7c06f7a0 0x7fdc7c13eb80 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c007ef0 tx=0x7fdc6c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.769+0000 7fdc72000640 1 -- 172.21.15.45:0/3997849684 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c014620 con 0x7fdc7c06f7a0 2024-04-16T16:16:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.769+0000 7fdc72000640 1 -- 172.21.15.45:0/3997849684 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc6c019070 con 0x7fdc7c06f7a0 2024-04-16T16:16:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.769+0000 7fdc72000640 1 -- 172.21.15.45:0/3997849684 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c014a00 con 0x7fdc7c06f7a0 2024-04-16T16:16:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.769+0000 7fdc7b200640 1 -- 172.21.15.45:0/3997849684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc7c13e0a0 con 0x7fdc7c06f7a0 2024-04-16T16:16:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.769+0000 7fdc7b200640 1 -- 172.21.15.45:0/3997849684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc7c1457d0 con 0x7fdc7c06f7a0 2024-04-16T16:16:15.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.770+0000 7fdc7b200640 1 -- 172.21.15.45:0/3997849684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc7c075780 con 0x7fdc7c06f7a0 2024-04-16T16:16:15.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.771+0000 7fdc72000640 1 -- 172.21.15.45:0/3997849684 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fdc6c014ba0 con 0x7fdc7c06f7a0 2024-04-16T16:16:15.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.772+0000 7fdc72000640 1 --2- 172.21.15.45:0/3997849684 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdc5806cb30 0x7fdc5806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:15.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.772+0000 7fdc72000640 1 -- 172.21.15.45:0/3997849684 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7fdc6c08cfd0 con 0x7fdc7c06f7a0 2024-04-16T16:16:15.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.772+0000 7fdc79400640 1 --2- 172.21.15.45:0/3997849684 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdc5806cb30 0x7fdc5806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.773+0000 7fdc79400640 1 --2- 172.21.15.45:0/3997849684 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdc5806cb30 0x7fdc5806eff0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c1f7f50 tx=0x7fdc64009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.776+0000 7fdc72000640 1 -- 172.21.15.45:0/3997849684 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdc6c05a9a0 con 0x7fdc7c06f7a0 2024-04-16T16:16:15.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.988+0000 7fdc7b200640 1 -- 172.21.15.45:0/3997849684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc7c13f180 con 0x7fdc7c06f7a0 2024-04-16T16:16:15.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.991+0000 7fdc72000640 1 -- 172.21.15.45:0/3997849684 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+21917 (secure 0 0 0) 0x7fdc6c05a2a0 con 0x7fdc7c06f7a0 2024-04-16T16:16:15.990 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:15.990 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":57,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:16:07.016528+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"fs1_metadata","create_time":"2024-04-16T16:15:13.250782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"fs1_data","create_time":"2024-04-16T16:15:13.856986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs1"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"fs2_metadata","create_time":"2024-04-16T16:15:33.215042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"fs2_data","create_time":"2024-04-16T16:15:34.271426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.2c","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/349066714":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/388389014":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/589483918":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3353543342":"2024-04-17T16:16:07.016480+0000","172.21.15.103:6832/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3310077212":"2024-04-17T16:16:07.016480+0000","172.21.15.45:6833/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6832/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.45:6833/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:6833/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6834/88032513":"2024-04-17T16:15:41.221894+0000","172.21.15.45:6835/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:0/644374842":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6835/88032513":"2024-04-17T16:15:41.221894+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:16:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.993+0000 7fdc7b200640 1 -- 172.21.15.45:0/3997849684 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdc5806cb30 msgr2=0x7fdc5806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.993+0000 7fdc7b200640 1 --2- 172.21.15.45:0/3997849684 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdc5806cb30 0x7fdc5806eff0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c1f7f50 tx=0x7fdc64009040 comp rx=0 tx=0).stop 2024-04-16T16:16:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.993+0000 7fdc7b200640 1 -- 172.21.15.45:0/3997849684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc7c06f7a0 msgr2=0x7fdc7c13eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.993+0000 7fdc7b200640 1 --2- 172.21.15.45:0/3997849684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc7c06f7a0 0x7fdc7c13eb80 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c007ef0 tx=0x7fdc6c00e4b0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.994+0000 7fdc7b200640 1 -- 172.21.15.45:0/3997849684 shutdown_connections 2024-04-16T16:16:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.994+0000 7fdc7b200640 1 --2- 172.21.15.45:0/3997849684 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdc5806cb30 0x7fdc5806eff0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.994+0000 7fdc7b200640 1 --2- 172.21.15.45:0/3997849684 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc7c1608a0 0x7fdc7c0cf390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.994+0000 7fdc7b200640 1 --2- 172.21.15.45:0/3997849684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc7c146220 0x7fdc7c0cee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.994+0000 7fdc7b200640 1 --2- 172.21.15.45:0/3997849684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc7c06f7a0 0x7fdc7c13eb80 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.994+0000 7fdc7b200640 1 -- 172.21.15.45:0/3997849684 >> 172.21.15.45:0/3997849684 conn(0x7fdc7c06b930 msgr2=0x7fdc7c06df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.994+0000 7fdc7b200640 1 -- 172.21.15.45:0/3997849684 shutdown_connections 2024-04-16T16:16:15.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:15.994+0000 7fdc7b200640 1 -- 172.21.15.45:0/3997849684 wait complete. 2024-04-16T16:16:16.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail fs1 2024-04-16T16:16:16.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.235+0000 7f5300a00640 1 Processor -- start 2024-04-16T16:16:16.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.236+0000 7f5300a00640 1 -- start start 2024-04-16T16:16:16.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.237+0000 7f5300a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f52fc0ba140 0x7f52fc0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:16.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.237+0000 7f5300a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52fc0d2600 con 0x7f52fc162090 2024-04-16T16:16:16.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.237+0000 7f5300a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52fc0ba5d0 con 0x7f52fc0cfc20 2024-04-16T16:16:16.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.237+0000 7f5300a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f52fc0ba8b0 con 0x7f52fc0ba140 2024-04-16T16:16:16.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.237+0000 7f52f3e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f52fc0ba140 0x7f52fc0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:16.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.238+0000 7f52f3e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f52fc0ba140 0x7f52fc0d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49304/0 (socket says 172.21.15.45:49304) 2024-04-16T16:16:16.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.238+0000 7f52f3e00640 1 -- 172.21.15.45:0/1739006791 learned_addr learned my addr 172.21.15.45:0/1739006791 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:16.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.238+0000 7f52f3e00640 1 -- 172.21.15.45:0/1739006791 >> v1:172.21.15.103:6789/0 conn(0x7f52fc0cfc20 legacy=0x7f52fc0d35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:16.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.238+0000 7f52f3e00640 1 -- 172.21.15.45:0/1739006791 >> v1:172.21.15.45:6789/0 conn(0x7f52fc162090 legacy=0x7f52fc0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:16.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.238+0000 7f52f3e00640 1 -- 172.21.15.45:0/1739006791 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52fc0d28a0 con 0x7f52fc0ba140 2024-04-16T16:16:16.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.239+0000 7f52f3e00640 1 --2- 172.21.15.45:0/1739006791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc0ba140 0x7f52fc0d30a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009870 tx=0x7f52e402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=717e840f8c27d7a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:16.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.240+0000 7f52f9e00640 1 -- 172.21.15.45:0/1739006791 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e4034070 con 0x7f52fc0ba140 2024-04-16T16:16:16.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.240+0000 7f52f9e00640 1 -- 172.21.15.45:0/1739006791 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52e4030850 con 0x7f52fc0ba140 2024-04-16T16:16:16.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.240+0000 7f52f9e00640 1 -- 172.21.15.45:0/1739006791 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e4030c20 con 0x7f52fc0ba140 2024-04-16T16:16:16.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.240+0000 7f5300a00640 1 -- 172.21.15.45:0/1739006791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc0ba140 msgr2=0x7f52fc0d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:16.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.240+0000 7f5300a00640 1 --2- 172.21.15.45:0/1739006791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc0ba140 0x7f52fc0d30a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009870 tx=0x7f52e402b990 comp rx=0 tx=0).stop 2024-04-16T16:16:16.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.240+0000 7f5300a00640 1 -- 172.21.15.45:0/1739006791 shutdown_connections 2024-04-16T16:16:16.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.240+0000 7f5300a00640 1 --2- 172.21.15.45:0/1739006791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc0ba140 0x7f52fc0d30a0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:16.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.240+0000 7f5300a00640 1 -- 172.21.15.45:0/1739006791 >> 172.21.15.45:0/1739006791 conn(0x7f52fc0d1940 msgr2=0x7f52fc0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:16.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.240+0000 7f5300a00640 1 -- 172.21.15.45:0/1739006791 shutdown_connections 2024-04-16T16:16:16.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.240+0000 7f5300a00640 1 -- 172.21.15.45:0/1739006791 wait complete. 2024-04-16T16:16:16.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.242+0000 7f5300a00640 1 Processor -- start 2024-04-16T16:16:16.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.242+0000 7f5300a00640 1 -- start start 2024-04-16T16:16:16.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.243+0000 7f5300a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52fc0cfc20 0x7f52fc0c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:16.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.243+0000 7f5300a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52fc162090 0x7f52fc0b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:16.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.244+0000 7f5300a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc154050 0x7f52fc1f4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:16.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.244+0000 7f5300a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc1625f0 con 0x7f52fc154050 2024-04-16T16:16:16.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.244+0000 7f5300a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc162470 con 0x7f52fc162090 2024-04-16T16:16:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.244+0000 7f5300a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f52fc0d5520 con 0x7f52fc0cfc20 2024-04-16T16:16:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.244+0000 7f52fb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc154050 0x7f52fc1f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.244+0000 7f52fb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc154050 0x7f52fc1f4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49318/0 (socket says 172.21.15.45:49318) 2024-04-16T16:16:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.244+0000 7f52fb200640 1 -- 172.21.15.45:0/743222291 learned_addr learned my addr 172.21.15.45:0/743222291 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.244+0000 7f52fa800640 1 --2- 172.21.15.45:0/743222291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52fc0cfc20 0x7f52fc0c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.244+0000 7f52fb200640 1 -- 172.21.15.45:0/743222291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52fc0cfc20 msgr2=0x7f52fc0c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.244+0000 7f52fb200640 1 --2- 172.21.15.45:0/743222291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52fc0cfc20 0x7f52fc0c9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.244+0000 7f52f3e00640 1 --2- 172.21.15.45:0/743222291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52fc162090 0x7f52fc0b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.244+0000 7f52fb200640 1 -- 172.21.15.45:0/743222291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52fc162090 msgr2=0x7f52fc0b6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.244+0000 7f52fb200640 1 --2- 172.21.15.45:0/743222291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52fc162090 0x7f52fc0b6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.244+0000 7f52fb200640 1 -- 172.21.15.45:0/743222291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52fc0b6b80 con 0x7f52fc154050 2024-04-16T16:16:16.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.245+0000 7f52fb200640 1 --2- 172.21.15.45:0/743222291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc154050 0x7f52fc1f4210 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f52f400c780 tx=0x7f52f400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:16.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.245+0000 7f52f2a00640 1 -- 172.21.15.45:0/743222291 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f4004b60 con 0x7f52fc154050 2024-04-16T16:16:16.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.245+0000 7f52f2a00640 1 -- 172.21.15.45:0/743222291 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f4004d00 con 0x7f52fc154050 2024-04-16T16:16:16.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.245+0000 7f52f2a00640 1 -- 172.21.15.45:0/743222291 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f400aa30 con 0x7f52fc154050 2024-04-16T16:16:16.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.245+0000 7f5300a00640 1 -- 172.21.15.45:0/743222291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52fc0b6e60 con 0x7f52fc154050 2024-04-16T16:16:16.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.245+0000 7f5300a00640 1 -- 172.21.15.45:0/743222291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52fc0cca30 con 0x7f52fc154050 2024-04-16T16:16:16.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.246+0000 7f5300a00640 1 -- 172.21.15.45:0/743222291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52fc075780 con 0x7f52fc154050 2024-04-16T16:16:16.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.247+0000 7f52f2a00640 1 -- 172.21.15.45:0/743222291 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f52f400abd0 con 0x7f52fc154050 2024-04-16T16:16:16.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.248+0000 7f52f2a00640 1 --2- 172.21.15.45:0/743222291 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f52d806cb60 0x7f52d806f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:16.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.249+0000 7f52f2a00640 1 -- 172.21.15.45:0/743222291 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7541+0+0 (secure 0 0 0) 0x7f52f408c9a0 con 0x7f52fc154050 2024-04-16T16:16:16.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.249+0000 7f52fa800640 1 --2- 172.21.15.45:0/743222291 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f52d806cb60 0x7f52d806f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:16.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.250+0000 7f52fa800640 1 --2- 172.21.15.45:0/743222291 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f52d806cb60 0x7f52d806f020 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f52ec000f60 tx=0x7f52ec009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:16.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.253+0000 7f52f2a00640 1 -- 172.21.15.45:0/743222291 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f52f4020070 con 0x7f52fc154050 2024-04-16T16:16:16.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.495+0000 7f5300a00640 1 -- 172.21.15.45:0/743222291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "fs1"} v 0) -- 0x7f52fc0b9b60 con 0x7f52fc154050 2024-04-16T16:16:16.496 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-04-16T16:16:16.497+0000 7fe7a4800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T16:16:16.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.501+0000 7f52f2a00640 1 -- 172.21.15.45:0/743222291 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "fs1"}]=0 fs1 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v41) ==== 155+0+0 (secure 0 0 0) 0x7f52f405a2a0 con 0x7f52fc154050 2024-04-16T16:16:16.500 INFO:teuthology.orchestra.run.smithi045.stderr:fs1 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-16T16:16:16.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.504+0000 7f5300a00640 1 -- 172.21.15.45:0/743222291 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f52d806cb60 msgr2=0x7f52d806f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:16.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.504+0000 7f5300a00640 1 --2- 172.21.15.45:0/743222291 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f52d806cb60 0x7f52d806f020 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f52ec000f60 tx=0x7f52ec009040 comp rx=0 tx=0).stop 2024-04-16T16:16:16.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.504+0000 7f5300a00640 1 -- 172.21.15.45:0/743222291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc154050 msgr2=0x7f52fc1f4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:16.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.504+0000 7f5300a00640 1 --2- 172.21.15.45:0/743222291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc154050 0x7f52fc1f4210 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f52f400c780 tx=0x7f52f400cc40 comp rx=0 tx=0).stop 2024-04-16T16:16:16.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.504+0000 7f5300a00640 1 -- 172.21.15.45:0/743222291 shutdown_connections 2024-04-16T16:16:16.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.504+0000 7f5300a00640 1 --2- 172.21.15.45:0/743222291 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f52d806cb60 0x7f52d806f020 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:16.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.504+0000 7f5300a00640 1 --2- 172.21.15.45:0/743222291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc154050 0x7f52fc1f4210 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:16.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.504+0000 7f5300a00640 1 --2- 172.21.15.45:0/743222291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52fc162090 0x7f52fc0b6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:16.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.504+0000 7f5300a00640 1 --2- 172.21.15.45:0/743222291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52fc0cfc20 0x7f52fc0c9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:16.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.504+0000 7f5300a00640 1 -- 172.21.15.45:0/743222291 >> 172.21.15.45:0/743222291 conn(0x7f52fc0d1940 msgr2=0x7f52fc0b8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:16.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.504+0000 7f5300a00640 1 -- 172.21.15.45:0/743222291 shutdown_connections 2024-04-16T16:16:16.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.504+0000 7f5300a00640 1 -- 172.21.15.45:0/743222291 wait complete. 2024-04-16T16:16:16.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm fs1 --yes-i-really-mean-it 2024-04-16T16:16:16.541 INFO:tasks.ceph.mds.d.smithi103.stderr: -2293> 2024-04-16T16:15:46.240+0000 7f5a38600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:16:16.614 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-16T16:16:16.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.753+0000 7f9f90200640 1 Processor -- start 2024-04-16T16:16:16.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.753+0000 7f9f90200640 1 -- start start 2024-04-16T16:16:16.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.755+0000 7f9f90200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f880d10e0 0x7f9f88072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:16.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.755+0000 7f9f90200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9f880b4930 0x7f9f8806f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:16.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.755+0000 7f9f90200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f880b5530 con 0x7f9f8815c7c0 2024-04-16T16:16:16.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.755+0000 7f9f90200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9f880b52e0 con 0x7f9f880b4930 2024-04-16T16:16:16.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.755+0000 7f9f90200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9f880d0260 con 0x7f9f880d10e0 2024-04-16T16:16:16.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.755+0000 7f9f8e200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9f880b4930 0x7f9f8806f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:16.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.755+0000 7f9f8e200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9f880b4930 0x7f9f8806f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49334/0 (socket says 172.21.15.45:49334) 2024-04-16T16:16:16.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.755+0000 7f9f8e200640 1 -- 172.21.15.45:0/3970122311 learned_addr learned my addr 172.21.15.45:0/3970122311 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:16.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.756+0000 7f9f8ce00640 1 --2- 172.21.15.45:0/3970122311 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f880d10e0 0x7f9f88072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:16.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.756+0000 7f9f8e200640 1 -- 172.21.15.45:0/3970122311 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f880d10e0 msgr2=0x7f9f88072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:16.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.756+0000 7f9f8e200640 1 --2- 172.21.15.45:0/3970122311 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f880d10e0 0x7f9f88072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:16.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.756+0000 7f9f8e200640 1 -- 172.21.15.45:0/3970122311 >> v1:172.21.15.45:6789/0 conn(0x7f9f8815c7c0 legacy=0x7f9f880d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:16.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.756+0000 7f9f8e200640 1 -- 172.21.15.45:0/3970122311 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f880b5810 con 0x7f9f880b4930 2024-04-16T16:16:16.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.757+0000 7f9f8e200640 1 --2- 172.21.15.45:0/3970122311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f880b4930 0x7f9f8806f160 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009870 tx=0x7f9f8002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d08689d39f6e7805 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:16.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.757+0000 7f9f87e00640 1 -- 172.21.15.45:0/3970122311 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f800317c0 con 0x7f9f880b4930 2024-04-16T16:16:16.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.758+0000 7f9f87e00640 1 -- 172.21.15.45:0/3970122311 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f80037070 con 0x7f9f880b4930 2024-04-16T16:16:16.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.758+0000 7f9f87e00640 1 -- 172.21.15.45:0/3970122311 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f80031b80 con 0x7f9f880b4930 2024-04-16T16:16:16.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.758+0000 7f9f90200640 1 -- 172.21.15.45:0/3970122311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f880b4930 msgr2=0x7f9f8806f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:16.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.758+0000 7f9f90200640 1 --2- 172.21.15.45:0/3970122311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f880b4930 0x7f9f8806f160 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009870 tx=0x7f9f8002b860 comp rx=0 tx=0).stop 2024-04-16T16:16:16.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.758+0000 7f9f90200640 1 -- 172.21.15.45:0/3970122311 shutdown_connections 2024-04-16T16:16:16.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.758+0000 7f9f90200640 1 --2- 172.21.15.45:0/3970122311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f880b4930 0x7f9f8806f160 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:16.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.758+0000 7f9f90200640 1 --2- 172.21.15.45:0/3970122311 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f880d10e0 0x7f9f88072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:16.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.758+0000 7f9f90200640 1 -- 172.21.15.45:0/3970122311 >> 172.21.15.45:0/3970122311 conn(0x7f9f880b3970 msgr2=0x7f9f880b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:16.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.759+0000 7f9f90200640 1 -- 172.21.15.45:0/3970122311 shutdown_connections 2024-04-16T16:16:16.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.759+0000 7f9f90200640 1 -- 172.21.15.45:0/3970122311 wait complete. 2024-04-16T16:16:16.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.760+0000 7f9f90200640 1 Processor -- start 2024-04-16T16:16:16.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.761+0000 7f9f90200640 1 -- start start 2024-04-16T16:16:16.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.761+0000 7f9f90200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f880b4930 0x7f9f880ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:16.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.761+0000 7f9f90200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f880d10e0 0x7f9f8814d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:16.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.761+0000 7f9f90200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8815c7c0 0x7f9f880c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:16.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.761+0000 7f9f90200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8806f6a0 con 0x7f9f8815c7c0 2024-04-16T16:16:16.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.761+0000 7f9f90200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9f88073350 con 0x7f9f880d10e0 2024-04-16T16:16:16.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.761+0000 7f9f90200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9f8806f820 con 0x7f9f880b4930 2024-04-16T16:16:16.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.762+0000 7f9f8e200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8815c7c0 0x7f9f880c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:16.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.762+0000 7f9f8e200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8815c7c0 0x7f9f880c6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49344/0 (socket says 172.21.15.45:49344) 2024-04-16T16:16:16.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.762+0000 7f9f8e200640 1 -- 172.21.15.45:0/4105993755 learned_addr learned my addr 172.21.15.45:0/4105993755 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:16.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.762+0000 7f9f8e200640 1 -- 172.21.15.45:0/4105993755 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f880b4930 msgr2=0x7f9f880ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:16.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.762+0000 7f9f8ce00640 1 --2- 172.21.15.45:0/4105993755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f880d10e0 0x7f9f8814d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:16.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.762+0000 7f9f8d800640 1 --2- 172.21.15.45:0/4105993755 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f880b4930 0x7f9f880ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:16.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.762+0000 7f9f8e200640 1 --2- 172.21.15.45:0/4105993755 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f880b4930 0x7f9f880ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:16.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.762+0000 7f9f8e200640 1 -- 172.21.15.45:0/4105993755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f880d10e0 msgr2=0x7f9f8814d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:16.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.762+0000 7f9f8e200640 1 --2- 172.21.15.45:0/4105993755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f880d10e0 0x7f9f8814d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:16.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.762+0000 7f9f8e200640 1 -- 172.21.15.45:0/4105993755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f880c66c0 con 0x7f9f8815c7c0 2024-04-16T16:16:16.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.763+0000 7f9f8e200640 1 --2- 172.21.15.45:0/4105993755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8815c7c0 0x7f9f880c6180 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009870 tx=0x7f9f800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:16.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.763+0000 7f9f85600640 1 -- 172.21.15.45:0/4105993755 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f80032860 con 0x7f9f8815c7c0 2024-04-16T16:16:16.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.763+0000 7f9f85600640 1 -- 172.21.15.45:0/4105993755 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f80032a20 con 0x7f9f8815c7c0 2024-04-16T16:16:16.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.763+0000 7f9f90200640 1 -- 172.21.15.45:0/4105993755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f880adfe0 con 0x7f9f8815c7c0 2024-04-16T16:16:16.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.763+0000 7f9f85600640 1 -- 172.21.15.45:0/4105993755 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f80032e10 con 0x7f9f8815c7c0 2024-04-16T16:16:16.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.763+0000 7f9f90200640 1 -- 172.21.15.45:0/4105993755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f880ae5a0 con 0x7f9f8815c7c0 2024-04-16T16:16:16.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.765+0000 7f9f90200640 1 -- 172.21.15.45:0/4105993755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f880b4f40 con 0x7f9f8815c7c0 2024-04-16T16:16:16.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.765+0000 7f9f85600640 1 -- 172.21.15.45:0/4105993755 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f9f8004c060 con 0x7f9f8815c7c0 2024-04-16T16:16:16.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.766+0000 7f9f85600640 1 --2- 172.21.15.45:0/4105993755 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f9f6806cb30 0x7f9f6806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:16.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.766+0000 7f9f85600640 1 -- 172.21.15.45:0/4105993755 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 7627+0+0 (secure 0 0 0) 0x7f9f800adea0 con 0x7f9f8815c7c0 2024-04-16T16:16:16.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.767+0000 7f9f8d800640 1 --2- 172.21.15.45:0/4105993755 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f9f6806cb30 0x7f9f6806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:16.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.768+0000 7f9f8d800640 1 --2- 172.21.15.45:0/4105993755 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f9f6806cb30 0x7f9f6806eff0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9f78007fd0 tx=0x7f9f78009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:16.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:16.771+0000 7f9f85600640 1 -- 172.21.15.45:0/4105993755 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9f880b4f40 con 0x7f9f8815c7c0 2024-04-16T16:16:17.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.025+0000 7f9f90200640 1 -- 172.21.15.45:0/4105993755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "fs1", "yes_i_really_mean_it": true} v 0) -- 0x7f9f88075780 con 0x7f9f8815c7c0 2024-04-16T16:16:17.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.505+0000 7f9f85600640 1 -- 172.21.15.45:0/4105993755 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "fs1", "yes_i_really_mean_it": true}]=0 v42) ==== 101+0+0 (secure 0 0 0) 0x7f9f80040070 con 0x7f9f8815c7c0 2024-04-16T16:16:17.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.508+0000 7f9f90200640 1 -- 172.21.15.45:0/4105993755 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f9f6806cb30 msgr2=0x7f9f6806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:17.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.508+0000 7f9f90200640 1 --2- 172.21.15.45:0/4105993755 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f9f6806cb30 0x7f9f6806eff0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9f78007fd0 tx=0x7f9f78009040 comp rx=0 tx=0).stop 2024-04-16T16:16:17.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.508+0000 7f9f90200640 1 -- 172.21.15.45:0/4105993755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8815c7c0 msgr2=0x7f9f880c6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:17.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.508+0000 7f9f90200640 1 --2- 172.21.15.45:0/4105993755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8815c7c0 0x7f9f880c6180 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009870 tx=0x7f9f800046c0 comp rx=0 tx=0).stop 2024-04-16T16:16:17.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.508+0000 7f9f90200640 1 -- 172.21.15.45:0/4105993755 shutdown_connections 2024-04-16T16:16:17.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.508+0000 7f9f90200640 1 --2- 172.21.15.45:0/4105993755 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f9f6806cb30 0x7f9f6806eff0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:17.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.508+0000 7f9f90200640 1 --2- 172.21.15.45:0/4105993755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8815c7c0 0x7f9f880c6180 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:17.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.508+0000 7f9f90200640 1 --2- 172.21.15.45:0/4105993755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f880d10e0 0x7f9f8814d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:17.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.508+0000 7f9f90200640 1 --2- 172.21.15.45:0/4105993755 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f880b4930 0x7f9f880ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:17.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.508+0000 7f9f90200640 1 -- 172.21.15.45:0/4105993755 >> 172.21.15.45:0/4105993755 conn(0x7f9f880b3970 msgr2=0x7f9f88074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:17.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.509+0000 7f9f90200640 1 -- 172.21.15.45:0/4105993755 shutdown_connections 2024-04-16T16:16:17.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.509+0000 7f9f90200640 1 -- 172.21.15.45:0/4105993755 wait complete. 2024-04-16T16:16:17.530 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm fs1_metadata fs1_metadata --yes-i-really-really-mean-it 2024-04-16T16:16:17.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.762+0000 7f4e05800640 1 Processor -- start 2024-04-16T16:16:17.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.763+0000 7f4e05800640 1 -- start start 2024-04-16T16:16:17.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.764+0000 7f4e05800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e00155ab0 0x7f4e00162ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:17.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.764+0000 7f4e05800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4e000ba610 0x7f4e0015efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:17.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.764+0000 7f4e05800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4e000d18c0 0x7f4e000d1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:17.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.764+0000 7f4e05800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f4e0015ff00 con 0x7f4e000ba610 2024-04-16T16:16:17.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.764+0000 7f4e05800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4e00155e90 con 0x7f4e000d18c0 2024-04-16T16:16:17.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.764+0000 7f4e05800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4e0015f940 con 0x7f4e00155ab0 2024-04-16T16:16:17.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.764+0000 7f4dff400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4e000ba610 0x7f4e0015efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:17.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.765+0000 7f4dff400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4e000ba610 0x7f4e0015efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49350/0 (socket says 172.21.15.45:49350) 2024-04-16T16:16:17.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.765+0000 7f4dff400640 1 -- 172.21.15.45:0/4184247167 learned_addr learned my addr 172.21.15.45:0/4184247167 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:17.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.765+0000 7f4e04e00640 1 --2- 172.21.15.45:0/4184247167 >> v2:172.21.15.103:3300/0 conn(0x7f4e000d18c0 0x7f4e000d1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:17.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.765+0000 7f4dffe00640 1 --2- 172.21.15.45:0/4184247167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e00155ab0 0x7f4e00162ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:17.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.765+0000 7f4dff400640 1 -- 172.21.15.45:0/4184247167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e00155ab0 msgr2=0x7f4e00162ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:17.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.765+0000 7f4dff400640 1 --2- 172.21.15.45:0/4184247167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e00155ab0 0x7f4e00162ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:17.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.765+0000 7f4dff400640 1 -- 172.21.15.45:0/4184247167 >> v2:172.21.15.103:3300/0 conn(0x7f4e000d18c0 msgr2=0x7f4e000d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:17.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.765+0000 7f4dff400640 1 --2- 172.21.15.45:0/4184247167 >> v2:172.21.15.103:3300/0 conn(0x7f4e000d18c0 0x7f4e000d1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:17.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.765+0000 7f4dff400640 1 -- 172.21.15.45:0/4184247167 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0015fc20 con 0x7f4e000ba610 2024-04-16T16:16:17.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.766+0000 7f4dff400640 1 --2- 172.21.15.45:0/4184247167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e000ba610 0x7f4e0015efd0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4e000baa70 tx=0x7f4dec02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65cd19066118fe66 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:17.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.767+0000 7f4dfea00640 1 -- 172.21.15.45:0/4184247167 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec0317c0 con 0x7f4e000ba610 2024-04-16T16:16:17.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.767+0000 7f4dfea00640 1 -- 172.21.15.45:0/4184247167 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dec037070 con 0x7f4e000ba610 2024-04-16T16:16:17.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.767+0000 7f4dfea00640 1 -- 172.21.15.45:0/4184247167 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec031b80 con 0x7f4e000ba610 2024-04-16T16:16:17.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.767+0000 7f4e05800640 1 -- 172.21.15.45:0/4184247167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e000ba610 msgr2=0x7f4e0015efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:17.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.767+0000 7f4e05800640 1 --2- 172.21.15.45:0/4184247167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e000ba610 0x7f4e0015efd0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4e000baa70 tx=0x7f4dec02b860 comp rx=0 tx=0).stop 2024-04-16T16:16:17.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.767+0000 7f4e05800640 1 -- 172.21.15.45:0/4184247167 shutdown_connections 2024-04-16T16:16:17.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.767+0000 7f4e05800640 1 --2- 172.21.15.45:0/4184247167 >> v2:172.21.15.103:3300/0 conn(0x7f4e000d18c0 0x7f4e000d1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:17.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.767+0000 7f4e05800640 1 --2- 172.21.15.45:0/4184247167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e000ba610 0x7f4e0015efd0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:17.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.767+0000 7f4e05800640 1 --2- 172.21.15.45:0/4184247167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e00155ab0 0x7f4e00162ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:17.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.767+0000 7f4e05800640 1 -- 172.21.15.45:0/4184247167 >> 172.21.15.45:0/4184247167 conn(0x7f4e0006d1a0 msgr2=0x7f4e0006d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:17.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.768+0000 7f4e05800640 1 -- 172.21.15.45:0/4184247167 shutdown_connections 2024-04-16T16:16:17.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.768+0000 7f4e05800640 1 -- 172.21.15.45:0/4184247167 wait complete. 2024-04-16T16:16:17.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.769+0000 7f4e05800640 1 Processor -- start 2024-04-16T16:16:17.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.769+0000 7f4e05800640 1 -- start start 2024-04-16T16:16:17.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.770+0000 7f4e05800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e000ba610 0x7f4e00144950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:17.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.770+0000 7f4e05800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e000d18c0 0x7f4e00158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:17.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.770+0000 7f4e05800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e00155ab0 0x7f4e00158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:17.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.770+0000 7f4e05800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4e00163430 con 0x7f4e000d18c0 2024-04-16T16:16:17.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.770+0000 7f4e05800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4e001629d0 con 0x7f4e000ba610 2024-04-16T16:16:17.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.770+0000 7f4e05800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4e0015f510 con 0x7f4e00155ab0 2024-04-16T16:16:17.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.771+0000 7f4dff400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e000d18c0 0x7f4e00158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:17.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.771+0000 7f4dff400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e000d18c0 0x7f4e00158a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49356/0 (socket says 172.21.15.45:49356) 2024-04-16T16:16:17.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.771+0000 7f4dff400640 1 -- 172.21.15.45:0/2097945480 learned_addr learned my addr 172.21.15.45:0/2097945480 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:17.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.771+0000 7f4dffe00640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e000ba610 0x7f4e00144950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.771+0000 7f4e04e00640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e00155ab0 0x7f4e00158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.771+0000 7f4dff400640 1 -- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e00155ab0 msgr2=0x7f4e00158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.771+0000 7f4dff400640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e00155ab0 0x7f4e00158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.771+0000 7f4dff400640 1 -- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e000ba610 msgr2=0x7f4e00144950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.771+0000 7f4dff400640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e000ba610 0x7f4e00144950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.771+0000 7f4dff400640 1 -- 172.21.15.45:0/2097945480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0015c690 con 0x7f4e000d18c0 2024-04-16T16:16:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.771+0000 7f4e04e00640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e00155ab0 0x7f4e00158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.771+0000 7f4dffe00640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e000ba610 0x7f4e00144950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.772+0000 7f4dff400640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e000d18c0 0x7f4e00158a90 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4dec000c00 tx=0x7f4dec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.772+0000 7f4df7e00640 1 -- 172.21.15.45:0/2097945480 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec032860 con 0x7f4e000d18c0 2024-04-16T16:16:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.772+0000 7f4df7e00640 1 -- 172.21.15.45:0/2097945480 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dec032a20 con 0x7f4e000d18c0 2024-04-16T16:16:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.772+0000 7f4e05800640 1 -- 172.21.15.45:0/2097945480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e000ccb80 con 0x7f4e000d18c0 2024-04-16T16:16:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.772+0000 7f4df7e00640 1 -- 172.21.15.45:0/2097945480 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec032e10 con 0x7f4e000d18c0 2024-04-16T16:16:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.772+0000 7f4e05800640 1 -- 172.21.15.45:0/2097945480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e000cd100 con 0x7f4e000d18c0 2024-04-16T16:16:17.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.773+0000 7f4e05800640 1 -- 172.21.15.45:0/2097945480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e000d1170 con 0x7f4e000d18c0 2024-04-16T16:16:17.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.774+0000 7f4df7e00640 1 -- 172.21.15.45:0/2097945480 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f4dec0453f0 con 0x7f4e000d18c0 2024-04-16T16:16:17.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.776+0000 7f4df7e00640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4de006cb30 0x7f4de006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:17.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.776+0000 7f4df7e00640 1 -- 172.21.15.45:0/2097945480 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 7627+0+0 (secure 0 0 0) 0x7f4dec0ae9a0 con 0x7f4e000d18c0 2024-04-16T16:16:17.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.776+0000 7f4dffe00640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4de006cb30 0x7f4de006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:17.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.778+0000 7f4dffe00640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4de006cb30 0x7f4de006eff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4df0006fd0 tx=0x7f4df0008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:17.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.780+0000 7f4df7e00640 1 -- 172.21.15.45:0/2097945480 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e000d1170 con 0x7f4e000d18c0 2024-04-16T16:16:17.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:17.990+0000 7f4e05800640 1 -- 172.21.15.45:0/2097945480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "fs1_metadata", "pool2": "fs1_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4e000d0bb0 con 0x7f4e000d18c0 2024-04-16T16:16:18.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:18.838+0000 7f4df7e00640 1 -- 172.21.15.45:0/2097945480 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "fs1_metadata", "pool2": "fs1_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'fs1_metadata' removed v59) ==== 172+0+0 (secure 0 0 0) 0x7f4e000d0bb0 con 0x7f4e000d18c0 2024-04-16T16:16:18.838 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'fs1_metadata' removed 2024-04-16T16:16:18.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:18.842+0000 7f4e05800640 1 -- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4de006cb30 msgr2=0x7f4de006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:18.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:18.842+0000 7f4e05800640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4de006cb30 0x7f4de006eff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4df0006fd0 tx=0x7f4df0008040 comp rx=0 tx=0).stop 2024-04-16T16:16:18.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:18.842+0000 7f4e05800640 1 -- 172.21.15.45:0/2097945480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e000d18c0 msgr2=0x7f4e00158a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:18.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:18.842+0000 7f4e05800640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e000d18c0 0x7f4e00158a90 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4dec000c00 tx=0x7f4dec0046c0 comp rx=0 tx=0).stop 2024-04-16T16:16:18.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:18.843+0000 7f4e05800640 1 -- 172.21.15.45:0/2097945480 shutdown_connections 2024-04-16T16:16:18.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:18.843+0000 7f4e05800640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4de006cb30 0x7f4de006eff0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:18.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:18.844+0000 7f4e05800640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e00155ab0 0x7f4e00158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:18.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:18.844+0000 7f4e05800640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e000d18c0 0x7f4e00158a90 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:18.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:18.844+0000 7f4e05800640 1 --2- 172.21.15.45:0/2097945480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e000ba610 0x7f4e00144950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:18.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:18.844+0000 7f4e05800640 1 -- 172.21.15.45:0/2097945480 >> 172.21.15.45:0/2097945480 conn(0x7f4e0006d1a0 msgr2=0x7f4e000d3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:18.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:18.845+0000 7f4e05800640 1 -- 172.21.15.45:0/2097945480 shutdown_connections 2024-04-16T16:16:18.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:18.845+0000 7f4e05800640 1 -- 172.21.15.45:0/2097945480 wait complete. 2024-04-16T16:16:18.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm fs1_data fs1_data --yes-i-really-really-mean-it 2024-04-16T16:16:19.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.100+0000 7f57fea00640 1 Processor -- start 2024-04-16T16:16:19.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.100+0000 7f57fea00640 1 -- start start 2024-04-16T16:16:19.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.101+0000 7f57fea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57f815b920 0x7f57f80b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:19.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.101+0000 7f57fea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f57f80d0e80 0x7f57f806ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:19.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.101+0000 7f57fea00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57f80ccb70 con 0x7f57f80afda0 2024-04-16T16:16:19.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.101+0000 7f57fea00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f57f806e7a0 con 0x7f57f80d0e80 2024-04-16T16:16:19.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.101+0000 7f57fea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f57f806f170 con 0x7f57f815b920 2024-04-16T16:16:19.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.102+0000 7f57f7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57f815b920 0x7f57f80b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:19.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.102+0000 7f57fca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f57f80d0e80 0x7f57f806ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:19.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.102+0000 7f57f7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57f815b920 0x7f57f80b4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:41048/0 (socket says 172.21.15.45:41048) 2024-04-16T16:16:19.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.102+0000 7f57f7e00640 1 -- 172.21.15.45:0/3542287826 learned_addr learned my addr 172.21.15.45:0/3542287826 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:19.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.103+0000 7f57f7e00640 1 -- 172.21.15.45:0/3542287826 >> v2:172.21.15.103:3300/0 conn(0x7f57f80d0e80 msgr2=0x7f57f806ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:19.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.103+0000 7f57f7e00640 1 --2- 172.21.15.45:0/3542287826 >> v2:172.21.15.103:3300/0 conn(0x7f57f80d0e80 0x7f57f806ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:19.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.103+0000 7f57f7e00640 1 -- 172.21.15.45:0/3542287826 >> v1:172.21.15.103:6789/0 conn(0x7f57f80afda0 legacy=0x7f57f80d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:19.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.103+0000 7f57f7e00640 1 -- 172.21.15.45:0/3542287826 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f80cce50 con 0x7f57f815b920 2024-04-16T16:16:19.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.104+0000 7f57f7e00640 1 --2- 172.21.15.45:0/3542287826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57f815b920 0x7f57f80b4e20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f57e8009870 tx=0x7f57e802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90e1b1b5df2fe0c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:19.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.104+0000 7f57f7400640 1 -- 172.21.15.45:0/3542287826 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8035020 con 0x7f57f815b920 2024-04-16T16:16:19.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.105+0000 7f57f7400640 1 -- 172.21.15.45:0/3542287826 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57e8031640 con 0x7f57f815b920 2024-04-16T16:16:19.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.105+0000 7f57f7400640 1 -- 172.21.15.45:0/3542287826 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e802d040 con 0x7f57f815b920 2024-04-16T16:16:19.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.105+0000 7f57fea00640 1 -- 172.21.15.45:0/3542287826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57f815b920 msgr2=0x7f57f80b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:19.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.105+0000 7f57fea00640 1 --2- 172.21.15.45:0/3542287826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57f815b920 0x7f57f80b4e20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f57e8009870 tx=0x7f57e802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:16:19.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.105+0000 7f57fea00640 1 -- 172.21.15.45:0/3542287826 shutdown_connections 2024-04-16T16:16:19.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.105+0000 7f57fea00640 1 --2- 172.21.15.45:0/3542287826 >> v2:172.21.15.103:3300/0 conn(0x7f57f80d0e80 0x7f57f806ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:19.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.105+0000 7f57fea00640 1 --2- 172.21.15.45:0/3542287826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57f815b920 0x7f57f80b4e20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:19.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.105+0000 7f57fea00640 1 -- 172.21.15.45:0/3542287826 >> 172.21.15.45:0/3542287826 conn(0x7f57f80b3970 msgr2=0x7f57f80b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:19.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.105+0000 7f57fea00640 1 -- 172.21.15.45:0/3542287826 shutdown_connections 2024-04-16T16:16:19.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.106+0000 7f57fea00640 1 -- 172.21.15.45:0/3542287826 wait complete. 2024-04-16T16:16:19.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.107+0000 7f57fea00640 1 Processor -- start 2024-04-16T16:16:19.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.107+0000 7f57fea00640 1 -- start start 2024-04-16T16:16:19.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.108+0000 7f57fea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57f80afda0 0x7f57f813e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:19.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.108+0000 7f57fea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f80d0e80 0x7f57f81e49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:19.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.108+0000 7f57fea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57f815b920 0x7f57f8141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:19.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.108+0000 7f57fea00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f57f80ce760 con 0x7f57f80d0e80 2024-04-16T16:16:19.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.108+0000 7f57fea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57f80ce5e0 con 0x7f57f80afda0 2024-04-16T16:16:19.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.108+0000 7f57fea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f57f80ce8e0 con 0x7f57f815b920 2024-04-16T16:16:19.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.109+0000 7f57ef400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f80d0e80 0x7f57f81e49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:19.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.109+0000 7f57ef400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f80d0e80 0x7f57f81e49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49372/0 (socket says 172.21.15.45:49372) 2024-04-16T16:16:19.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.109+0000 7f57ef400640 1 -- 172.21.15.45:0/3732743120 learned_addr learned my addr 172.21.15.45:0/3732743120 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:19.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.109+0000 7f57f7e00640 1 --2- 172.21.15.45:0/3732743120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57f80afda0 0x7f57f813e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:19.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.109+0000 7f57fca00640 1 --2- 172.21.15.45:0/3732743120 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57f815b920 0x7f57f8141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:19.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.109+0000 7f57ef400640 1 -- 172.21.15.45:0/3732743120 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57f815b920 msgr2=0x7f57f8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:19.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.109+0000 7f57ef400640 1 --2- 172.21.15.45:0/3732743120 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57f815b920 0x7f57f8141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:19.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.109+0000 7f57ef400640 1 -- 172.21.15.45:0/3732743120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57f80afda0 msgr2=0x7f57f813e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:19.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.109+0000 7f57ef400640 1 --2- 172.21.15.45:0/3732743120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57f80afda0 0x7f57f813e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:19.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.109+0000 7f57ef400640 1 -- 172.21.15.45:0/3732743120 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f813eb40 con 0x7f57f80d0e80 2024-04-16T16:16:19.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.110+0000 7f57ef400640 1 --2- 172.21.15.45:0/3732743120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f80d0e80 0x7f57f81e49c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f57e000c430 tx=0x7f57e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.110+0000 7f57f4c00640 1 -- 172.21.15.45:0/3732743120 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0011790 con 0x7f57f80d0e80 2024-04-16T16:16:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.110+0000 7f57f4c00640 1 -- 172.21.15.45:0/3732743120 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57e0017070 con 0x7f57f80d0e80 2024-04-16T16:16:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.110+0000 7f57f4c00640 1 -- 172.21.15.45:0/3732743120 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0011b70 con 0x7f57f80d0e80 2024-04-16T16:16:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.110+0000 7f57fea00640 1 -- 172.21.15.45:0/3732743120 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f81e5380 con 0x7f57f80d0e80 2024-04-16T16:16:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.110+0000 7f57fea00640 1 -- 172.21.15.45:0/3732743120 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f81e5940 con 0x7f57f80d0e80 2024-04-16T16:16:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.111+0000 7f57fea00640 1 -- 172.21.15.45:0/3732743120 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57f8075780 con 0x7f57f80d0e80 2024-04-16T16:16:19.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.112+0000 7f57f4c00640 1 -- 172.21.15.45:0/3732743120 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f57e0011d10 con 0x7f57f80d0e80 2024-04-16T16:16:19.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.114+0000 7f57f4c00640 1 --2- 172.21.15.45:0/3732743120 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f57d406cb30 0x7f57d406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:19.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.114+0000 7f57f4c00640 1 -- 172.21.15.45:0/3732743120 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 7195+0+0 (secure 0 0 0) 0x7f57e008c3d0 con 0x7f57f80d0e80 2024-04-16T16:16:19.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.115+0000 7f57f7e00640 1 --2- 172.21.15.45:0/3732743120 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f57d406cb30 0x7f57d406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.116+0000 7f57f7e00640 1 --2- 172.21.15.45:0/3732743120 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f57d406cb30 0x7f57d406eff0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f57e8036000 tx=0x7f57e803c040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.119+0000 7f57f4c00640 1 -- 172.21.15.45:0/3732743120 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f57e0059f00 con 0x7f57f80d0e80 2024-04-16T16:16:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.321+0000 7f57fea00640 1 -- 172.21.15.45:0/3732743120 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "fs1_data", "pool2": "fs1_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f57f80b0ee0 con 0x7f57f80d0e80 2024-04-16T16:16:19.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.842+0000 7f57f4c00640 1 -- 172.21.15.45:0/3732743120 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "fs1_data", "pool2": "fs1_data", "yes_i_really_really_mean_it": true}]=0 pool 'fs1_data' removed v60) ==== 160+0+0 (secure 0 0 0) 0x7f57e0059800 con 0x7f57f80d0e80 2024-04-16T16:16:19.842 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'fs1_data' removed 2024-04-16T16:16:19.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.846+0000 7f57fea00640 1 -- 172.21.15.45:0/3732743120 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f57d406cb30 msgr2=0x7f57d406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.846+0000 7f57fea00640 1 --2- 172.21.15.45:0/3732743120 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f57d406cb30 0x7f57d406eff0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f57e8036000 tx=0x7f57e803c040 comp rx=0 tx=0).stop 2024-04-16T16:16:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.846+0000 7f57fea00640 1 -- 172.21.15.45:0/3732743120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f80d0e80 msgr2=0x7f57f81e49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.846+0000 7f57fea00640 1 --2- 172.21.15.45:0/3732743120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f80d0e80 0x7f57f81e49c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f57e000c430 tx=0x7f57e000c8f0 comp rx=0 tx=0).stop 2024-04-16T16:16:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.846+0000 7f57fea00640 1 -- 172.21.15.45:0/3732743120 shutdown_connections 2024-04-16T16:16:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.846+0000 7f57fea00640 1 --2- 172.21.15.45:0/3732743120 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f57d406cb30 0x7f57d406eff0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.846+0000 7f57fea00640 1 --2- 172.21.15.45:0/3732743120 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57f815b920 0x7f57f8141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.846+0000 7f57fea00640 1 --2- 172.21.15.45:0/3732743120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f80d0e80 0x7f57f81e49c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.846+0000 7f57fea00640 1 --2- 172.21.15.45:0/3732743120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57f80afda0 0x7f57f813e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.846+0000 7f57fea00640 1 -- 172.21.15.45:0/3732743120 >> 172.21.15.45:0/3732743120 conn(0x7f57f80b3970 msgr2=0x7f57f8072410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.846+0000 7f57fea00640 1 -- 172.21.15.45:0/3732743120 shutdown_connections 2024-04-16T16:16:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:19.847+0000 7f57fea00640 1 -- 172.21.15.45:0/3732743120 wait complete. 2024-04-16T16:16:19.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:16:20.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.088+0000 7f4cc1600640 1 Processor -- start 2024-04-16T16:16:20.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.089+0000 7f4cc1600640 1 -- start start 2024-04-16T16:16:20.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.090+0000 7f4cc1600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cbc0b6730 0x7f4cbc0b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:20.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.090+0000 7f4cc1600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4cbc0b7ab0 0x7f4cbc0b7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:20.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.090+0000 7f4cc1600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cbc0d46e0 con 0x7f4cbc161ff0 2024-04-16T16:16:20.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.090+0000 7f4cc1600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f4cbc0b70d0 con 0x7f4cbc0b7ab0 2024-04-16T16:16:20.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.090+0000 7f4cc1600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4cbc0ba560 con 0x7f4cbc0b6730 2024-04-16T16:16:20.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.091+0000 7f4cc0c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4cbc0b7ab0 0x7f4cbc0b7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:20.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.091+0000 7f4cc0c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4cbc0b7ab0 0x7f4cbc0b7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49378/0 (socket says 172.21.15.45:49378) 2024-04-16T16:16:20.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.091+0000 7f4cc0c00640 1 -- 172.21.15.45:0/1398184707 learned_addr learned my addr 172.21.15.45:0/1398184707 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:20.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.091+0000 7f4cbb400640 1 --2- 172.21.15.45:0/1398184707 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cbc0b6730 0x7f4cbc0b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:20.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.091+0000 7f4cc0c00640 1 -- 172.21.15.45:0/1398184707 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cbc0b6730 msgr2=0x7f4cbc0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:20.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.091+0000 7f4cc0c00640 1 --2- 172.21.15.45:0/1398184707 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cbc0b6730 0x7f4cbc0b6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.091+0000 7f4cc0c00640 1 -- 172.21.15.45:0/1398184707 >> v1:172.21.15.103:6789/0 conn(0x7f4cbc161ff0 legacy=0x7f4cbc0cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:20.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.091+0000 7f4cc0c00640 1 -- 172.21.15.45:0/1398184707 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cbc0d3640 con 0x7f4cbc0b7ab0 2024-04-16T16:16:20.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.092+0000 7f4cc0c00640 1 --2- 172.21.15.45:0/1398184707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0b7ab0 0x7f4cbc0b7f00 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009e50 tx=0x7f4cb402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90083105d3769d05 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:20.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.092+0000 7f4cbaa00640 1 -- 172.21.15.45:0/1398184707 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb402ca70 con 0x7f4cbc0b7ab0 2024-04-16T16:16:20.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.093+0000 7f4cbaa00640 1 -- 172.21.15.45:0/1398184707 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cb402cc10 con 0x7f4cbc0b7ab0 2024-04-16T16:16:20.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.093+0000 7f4cbaa00640 1 -- 172.21.15.45:0/1398184707 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb40343f0 con 0x7f4cbc0b7ab0 2024-04-16T16:16:20.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.093+0000 7f4cc1600640 1 -- 172.21.15.45:0/1398184707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0b7ab0 msgr2=0x7f4cbc0b7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:20.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.093+0000 7f4cc1600640 1 --2- 172.21.15.45:0/1398184707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0b7ab0 0x7f4cbc0b7f00 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009e50 tx=0x7f4cb402bed0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.093+0000 7f4cc1600640 1 -- 172.21.15.45:0/1398184707 shutdown_connections 2024-04-16T16:16:20.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.093+0000 7f4cc1600640 1 --2- 172.21.15.45:0/1398184707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0b7ab0 0x7f4cbc0b7f00 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.093+0000 7f4cc1600640 1 --2- 172.21.15.45:0/1398184707 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cbc0b6730 0x7f4cbc0b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.093+0000 7f4cc1600640 1 -- 172.21.15.45:0/1398184707 >> 172.21.15.45:0/1398184707 conn(0x7f4cbc0d17b0 msgr2=0x7f4cbc0d1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:20.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.093+0000 7f4cc1600640 1 -- 172.21.15.45:0/1398184707 shutdown_connections 2024-04-16T16:16:20.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.093+0000 7f4cc1600640 1 -- 172.21.15.45:0/1398184707 wait complete. 2024-04-16T16:16:20.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.095+0000 7f4cc1600640 1 Processor -- start 2024-04-16T16:16:20.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.095+0000 7f4cc1600640 1 -- start start 2024-04-16T16:16:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.096+0000 7f4cc1600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0b6730 0x7f4cbc0abca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.096+0000 7f4cc1600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cbc0b7ab0 0x7f4cbc0c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.096+0000 7f4cc1600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cbc161ff0 0x7f4cbc1549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.096+0000 7f4cc1600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc0d3ee0 con 0x7f4cbc0b6730 2024-04-16T16:16:20.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.096+0000 7f4cc1600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc0d3c00 con 0x7f4cbc0b7ab0 2024-04-16T16:16:20.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.096+0000 7f4cc1600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4cbc0d41c0 con 0x7f4cbc161ff0 2024-04-16T16:16:20.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.096+0000 7f4cbbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0b6730 0x7f4cbc0abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:20.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.096+0000 7f4cbbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0b6730 0x7f4cbc0abca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49382/0 (socket says 172.21.15.45:49382) 2024-04-16T16:16:20.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.096+0000 7f4cbbe00640 1 -- 172.21.15.45:0/604714091 learned_addr learned my addr 172.21.15.45:0/604714091 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.097+0000 7f4cbb400640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cbc0b7ab0 0x7f4cbc0c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.097+0000 7f4cc0c00640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cbc161ff0 0x7f4cbc1549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.097+0000 7f4cbbe00640 1 -- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cbc161ff0 msgr2=0x7f4cbc1549a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.097+0000 7f4cbbe00640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cbc161ff0 0x7f4cbc1549a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.097+0000 7f4cbbe00640 1 -- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cbc0b7ab0 msgr2=0x7f4cbc0c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.097+0000 7f4cbbe00640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cbc0b7ab0 0x7f4cbc0c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.097+0000 7f4cbbe00640 1 -- 172.21.15.45:0/604714091 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cbc0cb9e0 con 0x7f4cbc0b6730 2024-04-16T16:16:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.097+0000 7f4cc0c00640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cbc161ff0 0x7f4cbc1549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:20.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.097+0000 7f4cbb400640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cbc0b7ab0 0x7f4cbc0c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:20.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.097+0000 7f4cbbe00640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0b6730 0x7f4cbc0abca0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00c460 tx=0x7f4cac00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:20.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.098+0000 7f4cb3e00640 1 -- 172.21.15.45:0/604714091 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac016020 con 0x7f4cbc0b6730 2024-04-16T16:16:20.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.098+0000 7f4cb3e00640 1 -- 172.21.15.45:0/604714091 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cac012870 con 0x7f4cbc0b6730 2024-04-16T16:16:20.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.098+0000 7f4cb3e00640 1 -- 172.21.15.45:0/604714091 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac012c60 con 0x7f4cbc0b6730 2024-04-16T16:16:20.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.098+0000 7f4cc1600640 1 -- 172.21.15.45:0/604714091 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cbc0cca60 con 0x7f4cbc0b6730 2024-04-16T16:16:20.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.098+0000 7f4cc1600640 1 -- 172.21.15.45:0/604714091 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cbc0cbf80 con 0x7f4cbc0b6730 2024-04-16T16:16:20.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.099+0000 7f4cc1600640 1 -- 172.21.15.45:0/604714091 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cbc075780 con 0x7f4cbc0b6730 2024-04-16T16:16:20.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.100+0000 7f4cb3e00640 1 -- 172.21.15.45:0/604714091 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f4cac019020 con 0x7f4cbc0b6730 2024-04-16T16:16:20.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.101+0000 7f4cb3e00640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4c9c06cb30 0x7f4c9c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:20.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.101+0000 7f4cb3e00640 1 -- 172.21.15.45:0/604714091 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6729+0+0 (secure 0 0 0) 0x7f4cac08bc90 con 0x7f4cbc0b6730 2024-04-16T16:16:20.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.102+0000 7f4cbb400640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4c9c06cb30 0x7f4c9c06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:20.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.103+0000 7f4cbb400640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4c9c06cb30 0x7f4c9c06eff0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0cccb0 tx=0x7f4ca8008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:20.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.105+0000 7f4cb3e00640 1 -- 172.21.15.45:0/604714091 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4cac020070 con 0x7f4cbc0b6730 2024-04-16T16:16:20.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.318+0000 7f4cc1600640 1 -- 172.21.15.45:0/604714091 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4cbc161570 con 0x7f4cbc0b6730 2024-04-16T16:16:20.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.321+0000 7f4cb3e00640 1 -- 172.21.15.45:0/604714091 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) ==== 74+0+18188 (secure 0 0 0) 0x7f4cac059ac0 con 0x7f4cbc0b6730 2024-04-16T16:16:20.321 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:20.321 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":60,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:16:19.838837+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"fs2_metadata","create_time":"2024-04-16T16:15:33.215042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"fs2_data","create_time":"2024-04-16T16:15:34.271426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6834/2626302738":"2024-04-17T16:16:16.497315+0000","172.21.15.103:0/349066714":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/388389014":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/589483918":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3353543342":"2024-04-17T16:16:07.016480+0000","172.21.15.103:6832/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3310077212":"2024-04-17T16:16:07.016480+0000","172.21.15.45:6833/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6832/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6836/2626302738":"2024-04-17T16:16:16.497315+0000","172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.45:6833/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:6833/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6834/88032513":"2024-04-17T16:15:41.221894+0000","172.21.15.45:6835/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:0/644374842":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6835/88032513":"2024-04-17T16:15:41.221894+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:16:20.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.324+0000 7f4cc1600640 1 -- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4c9c06cb30 msgr2=0x7f4c9c06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:20.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.324+0000 7f4cc1600640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4c9c06cb30 0x7f4c9c06eff0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0cccb0 tx=0x7f4ca8008040 comp rx=0 tx=0).stop 2024-04-16T16:16:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.324+0000 7f4cc1600640 1 -- 172.21.15.45:0/604714091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0b6730 msgr2=0x7f4cbc0abca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.324+0000 7f4cc1600640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0b6730 0x7f4cbc0abca0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00c460 tx=0x7f4cac00c920 comp rx=0 tx=0).stop 2024-04-16T16:16:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.324+0000 7f4cc1600640 1 -- 172.21.15.45:0/604714091 shutdown_connections 2024-04-16T16:16:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.324+0000 7f4cc1600640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f4c9c06cb30 0x7f4c9c06eff0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.324+0000 7f4cc1600640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cbc161ff0 0x7f4cbc1549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.324+0000 7f4cc1600640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cbc0b7ab0 0x7f4cbc0c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.324+0000 7f4cc1600640 1 --2- 172.21.15.45:0/604714091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0b6730 0x7f4cbc0abca0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.324+0000 7f4cc1600640 1 -- 172.21.15.45:0/604714091 >> 172.21.15.45:0/604714091 conn(0x7f4cbc0d17b0 msgr2=0x7f4cbc0b9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.325+0000 7f4cc1600640 1 -- 172.21.15.45:0/604714091 shutdown_connections 2024-04-16T16:16:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.325+0000 7f4cc1600640 1 -- 172.21.15.45:0/604714091 wait complete. 2024-04-16T16:16:20.343 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'fs2_metadata', 'create_time': '2024-04-16T16:15:33.215042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'fs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'fs2_data', 'create_time': '2024-04-16T16:15:34.271426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'fs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:16:20.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:16:20.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.570+0000 7f5ef4a00640 1 Processor -- start 2024-04-16T16:16:20.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.570+0000 7f5ef4a00640 1 -- start start 2024-04-16T16:16:20.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.572+0000 7f5ef4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ef006d3e0 0x7f5ef00d25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:20.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.572+0000 7f5ef4a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5ef01623d0 0x7f5ef00d2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:20.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.572+0000 7f5ef4a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef00d4170 con 0x7f5ef0161ff0 2024-04-16T16:16:20.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.572+0000 7f5ef4a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5ef00ba700 con 0x7f5ef01623d0 2024-04-16T16:16:20.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.572+0000 7f5ef4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5ef00d4450 con 0x7f5ef006d3e0 2024-04-16T16:16:20.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.572+0000 7f5eefe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5ef01623d0 0x7f5ef00d2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:20.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.572+0000 7f5eefe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5ef01623d0 0x7f5ef00d2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49388/0 (socket says 172.21.15.45:49388) 2024-04-16T16:16:20.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.572+0000 7f5eefe00640 1 -- 172.21.15.45:0/1208709584 learned_addr learned my addr 172.21.15.45:0/1208709584 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:20.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.573+0000 7f5eefe00640 1 -- 172.21.15.45:0/1208709584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ef006d3e0 msgr2=0x7f5ef00d25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:20.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.573+0000 7f5eefe00640 1 --2- 172.21.15.45:0/1208709584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ef006d3e0 0x7f5ef00d25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.573+0000 7f5eefe00640 1 -- 172.21.15.45:0/1208709584 >> v1:172.21.15.103:6789/0 conn(0x7f5ef0161ff0 legacy=0x7f5ef006ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:20.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.573+0000 7f5eefe00640 1 -- 172.21.15.45:0/1208709584 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef00ba390 con 0x7f5ef01623d0 2024-04-16T16:16:20.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.574+0000 7f5eefe00640 1 --2- 172.21.15.45:0/1208709584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef01623d0 0x7f5ef00d2ae0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009e50 tx=0x7f5ee802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3875f2faad6f6ac5 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:20.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.574+0000 7f5eee000640 1 -- 172.21.15.45:0/1208709584 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee802ca70 con 0x7f5ef01623d0 2024-04-16T16:16:20.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.574+0000 7f5eee000640 1 -- 172.21.15.45:0/1208709584 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ee802cc10 con 0x7f5ef01623d0 2024-04-16T16:16:20.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.574+0000 7f5eee000640 1 -- 172.21.15.45:0/1208709584 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee80343f0 con 0x7f5ef01623d0 2024-04-16T16:16:20.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.575+0000 7f5ef4a00640 1 -- 172.21.15.45:0/1208709584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef01623d0 msgr2=0x7f5ef00d2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:20.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.575+0000 7f5ef4a00640 1 --2- 172.21.15.45:0/1208709584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef01623d0 0x7f5ef00d2ae0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009e50 tx=0x7f5ee802bed0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.575+0000 7f5ef4a00640 1 -- 172.21.15.45:0/1208709584 shutdown_connections 2024-04-16T16:16:20.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.575+0000 7f5ef4a00640 1 --2- 172.21.15.45:0/1208709584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef01623d0 0x7f5ef00d2ae0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.575+0000 7f5ef4a00640 1 --2- 172.21.15.45:0/1208709584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ef006d3e0 0x7f5ef00d25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.575+0000 7f5ef4a00640 1 -- 172.21.15.45:0/1208709584 >> 172.21.15.45:0/1208709584 conn(0x7f5ef00d1910 msgr2=0x7f5ef00d4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:20.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.575+0000 7f5ef4a00640 1 -- 172.21.15.45:0/1208709584 shutdown_connections 2024-04-16T16:16:20.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.575+0000 7f5ef4a00640 1 -- 172.21.15.45:0/1208709584 wait complete. 2024-04-16T16:16:20.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.577+0000 7f5ef4a00640 1 Processor -- start 2024-04-16T16:16:20.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.577+0000 7f5ef4a00640 1 -- start start 2024-04-16T16:16:20.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.578+0000 7f5ef4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ef006d3e0 0x7f5ef00cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:20.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.578+0000 7f5ef4a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef0161ff0 0x7f5ef014ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:20.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.578+0000 7f5ef4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef01623d0 0x7f5ef01f49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:20.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.578+0000 7f5ef4a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef00d5540 con 0x7f5ef0161ff0 2024-04-16T16:16:20.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.578+0000 7f5ef4a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef00d53c0 con 0x7f5ef01623d0 2024-04-16T16:16:20.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.578+0000 7f5ef4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5ef00d56c0 con 0x7f5ef006d3e0 2024-04-16T16:16:20.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.578+0000 7f5eeea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef0161ff0 0x7f5ef014ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:20.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.578+0000 7f5eeea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef0161ff0 0x7f5ef014ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49400/0 (socket says 172.21.15.45:49400) 2024-04-16T16:16:20.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.578+0000 7f5eeea00640 1 -- 172.21.15.45:0/4086530943 learned_addr learned my addr 172.21.15.45:0/4086530943 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:20.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.579+0000 7f5eef400640 1 --2- 172.21.15.45:0/4086530943 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ef006d3e0 0x7f5ef00cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:20.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.579+0000 7f5eefe00640 1 --2- 172.21.15.45:0/4086530943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef01623d0 0x7f5ef01f49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:20.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.579+0000 7f5eeea00640 1 -- 172.21.15.45:0/4086530943 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ef006d3e0 msgr2=0x7f5ef00cac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:20.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.579+0000 7f5eeea00640 1 --2- 172.21.15.45:0/4086530943 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ef006d3e0 0x7f5ef00cac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.579+0000 7f5eeea00640 1 -- 172.21.15.45:0/4086530943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef01623d0 msgr2=0x7f5ef01f49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:20.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.579+0000 7f5eeea00640 1 --2- 172.21.15.45:0/4086530943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef01623d0 0x7f5ef01f49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.579+0000 7f5eeea00640 1 -- 172.21.15.45:0/4086530943 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef01f5250 con 0x7f5ef0161ff0 2024-04-16T16:16:20.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.579+0000 7f5eefe00640 1 --2- 172.21.15.45:0/4086530943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef01623d0 0x7f5ef01f49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:20.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.579+0000 7f5eeea00640 1 --2- 172.21.15.45:0/4086530943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef0161ff0 0x7f5ef014ed30 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800b4a0 tx=0x7f5ed800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:20.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.579+0000 7f5ee7400640 1 -- 172.21.15.45:0/4086530943 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8014070 con 0x7f5ef0161ff0 2024-04-16T16:16:20.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.579+0000 7f5ee7400640 1 -- 172.21.15.45:0/4086530943 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ed8010610 con 0x7f5ef0161ff0 2024-04-16T16:16:20.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.579+0000 7f5ee7400640 1 -- 172.21.15.45:0/4086530943 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8010a00 con 0x7f5ef0161ff0 2024-04-16T16:16:20.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.579+0000 7f5ef4a00640 1 -- 172.21.15.45:0/4086530943 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ef01f5530 con 0x7f5ef0161ff0 2024-04-16T16:16:20.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.580+0000 7f5ef4a00640 1 -- 172.21.15.45:0/4086530943 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ef01f5af0 con 0x7f5ef0161ff0 2024-04-16T16:16:20.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.581+0000 7f5ef4a00640 1 -- 172.21.15.45:0/4086530943 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ef0075780 con 0x7f5ef0161ff0 2024-04-16T16:16:20.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.582+0000 7f5ee7400640 1 -- 172.21.15.45:0/4086530943 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f5ed8017070 con 0x7f5ef0161ff0 2024-04-16T16:16:20.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.583+0000 7f5ee7400640 1 --2- 172.21.15.45:0/4086530943 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5ecc06cb60 0x7f5ecc06f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:20.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.583+0000 7f5ee7400640 1 -- 172.21.15.45:0/4086530943 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6729+0+0 (secure 0 0 0) 0x7f5ed808be50 con 0x7f5ef0161ff0 2024-04-16T16:16:20.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.584+0000 7f5eef400640 1 --2- 172.21.15.45:0/4086530943 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5ecc06cb60 0x7f5ecc06f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:20.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.585+0000 7f5eef400640 1 --2- 172.21.15.45:0/4086530943 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5ecc06cb60 0x7f5ecc06f020 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0007fd0 tx=0x7f5ee0009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:20.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.588+0000 7f5ee7400640 1 -- 172.21.15.45:0/4086530943 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5ed8059b20 con 0x7f5ef0161ff0 2024-04-16T16:16:20.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.808+0000 7f5ef4a00640 1 -- 172.21.15.45:0/4086530943 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5ef006db10 con 0x7f5ef0161ff0 2024-04-16T16:16:20.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.809+0000 7f5ee7400640 1 -- 172.21.15.45:0/4086530943 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) ==== 93+0+10 (secure 0 0 0) 0x7f5ed8059420 con 0x7f5ef0161ff0 2024-04-16T16:16:20.808 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:16:20.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.812+0000 7f5ef4a00640 1 -- 172.21.15.45:0/4086530943 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5ecc06cb60 msgr2=0x7f5ecc06f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:20.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.812+0000 7f5ef4a00640 1 --2- 172.21.15.45:0/4086530943 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5ecc06cb60 0x7f5ecc06f020 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0007fd0 tx=0x7f5ee0009040 comp rx=0 tx=0).stop 2024-04-16T16:16:20.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.812+0000 7f5ef4a00640 1 -- 172.21.15.45:0/4086530943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef0161ff0 msgr2=0x7f5ef014ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:20.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.812+0000 7f5ef4a00640 1 --2- 172.21.15.45:0/4086530943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef0161ff0 0x7f5ef014ed30 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800b4a0 tx=0x7f5ed800b960 comp rx=0 tx=0).stop 2024-04-16T16:16:20.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.812+0000 7f5ef4a00640 1 -- 172.21.15.45:0/4086530943 shutdown_connections 2024-04-16T16:16:20.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.812+0000 7f5ef4a00640 1 --2- 172.21.15.45:0/4086530943 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5ecc06cb60 0x7f5ecc06f020 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.812+0000 7f5ef4a00640 1 --2- 172.21.15.45:0/4086530943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef01623d0 0x7f5ef01f49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.812+0000 7f5ef4a00640 1 --2- 172.21.15.45:0/4086530943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef0161ff0 0x7f5ef014ed30 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.812+0000 7f5ef4a00640 1 --2- 172.21.15.45:0/4086530943 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ef006d3e0 0x7f5ef00cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:20.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.812+0000 7f5ef4a00640 1 -- 172.21.15.45:0/4086530943 >> 172.21.15.45:0/4086530943 conn(0x7f5ef00d1910 msgr2=0x7f5ef00b8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:20.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.812+0000 7f5ef4a00640 1 -- 172.21.15.45:0/4086530943 shutdown_connections 2024-04-16T16:16:20.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:20.813+0000 7f5ef4a00640 1 -- 172.21.15.45:0/4086530943 wait complete. 2024-04-16T16:16:20.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs2_metadata pg_num 2024-04-16T16:16:21.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.040+0000 7f5c91400640 1 Processor -- start 2024-04-16T16:16:21.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.040+0000 7f5c91400640 1 -- start start 2024-04-16T16:16:21.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.041+0000 7f5c91400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5c8c06cf10 0x7f5c8c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.041+0000 7f5c91400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c8c0d35d0 0x7f5c8c0d3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.041+0000 7f5c91400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c8c06d8a0 con 0x7f5c8c15ecc0 2024-04-16T16:16:21.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.041+0000 7f5c91400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5c8c0d2b10 con 0x7f5c8c06cf10 2024-04-16T16:16:21.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.041+0000 7f5c91400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5c8c163210 con 0x7f5c8c0d35d0 2024-04-16T16:16:21.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.042+0000 7f5c8a800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5c8c15ecc0 0x7f5c8c1626a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47138/0 (socket says 172.21.15.45:47138) 2024-04-16T16:16:21.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.042+0000 7f5c8a800640 1 -- 172.21.15.45:0/1782601001 learned_addr learned my addr 172.21.15.45:0/1782601001 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:21.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.042+0000 7f5c89400640 1 -- 172.21.15.45:0/1782601001 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3852266116 0 0) 0x7f5c8c06d8a0 con 0x7f5c8c15ecc0 2024-04-16T16:16:21.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.042+0000 7f5c89e00640 1 --2- 172.21.15.45:0/1782601001 >> v2:172.21.15.103:3300/0 conn(0x7f5c8c06cf10 0x7f5c8c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.042+0000 7f5c89400640 1 -- 172.21.15.45:0/1782601001 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c70003520 con 0x7f5c8c15ecc0 2024-04-16T16:16:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.042+0000 7f5c8b200640 1 --2- 172.21.15.45:0/1782601001 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c8c0d35d0 0x7f5c8c0d3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c89400640 1 -- 172.21.15.45:0/1782601001 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3938254914 0 0) 0x7f5c70003520 con 0x7f5c8c15ecc0 2024-04-16T16:16:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c89400640 1 -- 172.21.15.45:0/1782601001 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c8c0d35d0 msgr2=0x7f5c8c0d3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c89400640 1 --2- 172.21.15.45:0/1782601001 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c8c0d35d0 0x7f5c8c0d3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c89400640 1 -- 172.21.15.45:0/1782601001 >> v2:172.21.15.103:3300/0 conn(0x7f5c8c06cf10 msgr2=0x7f5c8c06d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c89400640 1 --2- 172.21.15.45:0/1782601001 >> v2:172.21.15.103:3300/0 conn(0x7f5c8c06cf10 0x7f5c8c06d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c89400640 1 -- 172.21.15.45:0/1782601001 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c8c1634f0 con 0x7f5c8c15ecc0 2024-04-16T16:16:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c89400640 1 -- 172.21.15.45:0/1782601001 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f5c7c0019f0 con 0x7f5c8c15ecc0 2024-04-16T16:16:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c89400640 1 -- 172.21.15.45:0/1782601001 >> v1:172.21.15.45:6789/0 conn(0x7f5c8c15ecc0 legacy=0x7f5c8c1626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c89400640 1 --2- 172.21.15.45:0/1782601001 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c8c0d35d0 0x7f5c700039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c89400640 1 --2- 172.21.15.45:0/1782601001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c70004030 0x7f5c70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c89400640 1 --2- 172.21.15.45:0/1782601001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c70004d40 0x7f5c700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c89400640 1 -- 172.21.15.45:0/1782601001 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8c0d2b10 con 0x7f5c70004030 2024-04-16T16:16:21.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c89400640 1 -- 172.21.15.45:0/1782601001 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8c163210 con 0x7f5c70004d40 2024-04-16T16:16:21.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c89400640 1 -- 172.21.15.45:0/1782601001 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5c7000bbb0 con 0x7f5c8c0d35d0 2024-04-16T16:16:21.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c89e00640 1 --2- 172.21.15.45:0/1782601001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c70004030 0x7f5c70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c8a800640 1 --2- 172.21.15.45:0/1782601001 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c8c0d35d0 0x7f5c700039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.043+0000 7f5c8b200640 1 --2- 172.21.15.45:0/1782601001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c70004d40 0x7f5c700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.044+0000 7f5c89e00640 1 -- 172.21.15.45:0/1782601001 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c8c0d35d0 msgr2=0x7f5c700039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:21.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.044+0000 7f5c89e00640 1 --2- 172.21.15.45:0/1782601001 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c8c0d35d0 0x7f5c700039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.044+0000 7f5c89e00640 1 -- 172.21.15.45:0/1782601001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c70004d40 msgr2=0x7f5c700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:21.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.044+0000 7f5c89e00640 1 --2- 172.21.15.45:0/1782601001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c70004d40 0x7f5c700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.044+0000 7f5c89e00640 1 -- 172.21.15.45:0/1782601001 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c70003520 con 0x7f5c70004030 2024-04-16T16:16:21.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.045+0000 7f5c89e00640 1 --2- 172.21.15.45:0/1782601001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c70004030 0x7f5c70004800 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400ba30 tx=0x7f5c70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19026fa0f9d402d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:21.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.045+0000 7f5c89400640 1 -- 172.21.15.45:0/1782601001 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c74036020 con 0x7f5c70004030 2024-04-16T16:16:21.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.045+0000 7f5c89400640 1 -- 172.21.15.45:0/1782601001 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c74032680 con 0x7f5c70004030 2024-04-16T16:16:21.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.045+0000 7f5c89400640 1 -- 172.21.15.45:0/1782601001 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7400b600 con 0x7f5c70004030 2024-04-16T16:16:21.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.046+0000 7f5c91400640 1 -- 172.21.15.45:0/1782601001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c70004030 msgr2=0x7f5c70004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:21.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.046+0000 7f5c91400640 1 --2- 172.21.15.45:0/1782601001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c70004030 0x7f5c70004800 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400ba30 tx=0x7f5c70005830 comp rx=0 tx=0).stop 2024-04-16T16:16:21.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.046+0000 7f5c8b200640 1 -- 172.21.15.45:0/1782601001 reap_dead start 2024-04-16T16:16:21.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.046+0000 7f5c91400640 1 -- 172.21.15.45:0/1782601001 shutdown_connections 2024-04-16T16:16:21.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.046+0000 7f5c91400640 1 -- 172.21.15.45:0/1782601001 >> 172.21.15.45:0/1782601001 conn(0x7f5c8c073690 msgr2=0x7f5c8c0d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:21.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.046+0000 7f5c91400640 1 -- 172.21.15.45:0/1782601001 shutdown_connections 2024-04-16T16:16:21.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.046+0000 7f5c91400640 1 -- 172.21.15.45:0/1782601001 wait complete. 2024-04-16T16:16:21.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.048+0000 7f5c91400640 1 Processor -- start 2024-04-16T16:16:21.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.048+0000 7f5c91400640 1 -- start start 2024-04-16T16:16:21.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.049+0000 7f5c91400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c70004030 0x7f5c8c15b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.049+0000 7f5c91400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c8c0c9170 0x7f5c8c0c9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.049+0000 7f5c91400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c8c0ca1e0 0x7f5c8c1f7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.049+0000 7f5c91400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8c0ba540 con 0x7f5c8c0ca1e0 2024-04-16T16:16:21.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.049+0000 7f5c91400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8c0d30b0 con 0x7f5c70004030 2024-04-16T16:16:21.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.049+0000 7f5c91400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5c8c162db0 con 0x7f5c8c0c9170 2024-04-16T16:16:21.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.049+0000 7f5c8b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c8c0ca1e0 0x7f5c8c1f7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.050+0000 7f5c8b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c8c0ca1e0 0x7f5c8c1f7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49414/0 (socket says 172.21.15.45:49414) 2024-04-16T16:16:21.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.050+0000 7f5c8b200640 1 -- 172.21.15.45:0/369714696 learned_addr learned my addr 172.21.15.45:0/369714696 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:21.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.050+0000 7f5c89e00640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c8c0c9170 0x7f5c8c0c9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.050+0000 7f5c8a800640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c70004030 0x7f5c8c15b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.050+0000 7f5c8b200640 1 -- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c8c0c9170 msgr2=0x7f5c8c0c9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:21.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.050+0000 7f5c8b200640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c8c0c9170 0x7f5c8c0c9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.050+0000 7f5c8b200640 1 -- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c70004030 msgr2=0x7f5c8c15b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:21.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.050+0000 7f5c8b200640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c70004030 0x7f5c8c15b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.050+0000 7f5c8b200640 1 -- 172.21.15.45:0/369714696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c8c1f8a20 con 0x7f5c8c0ca1e0 2024-04-16T16:16:21.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.050+0000 7f5c89e00640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c8c0c9170 0x7f5c8c0c9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:16:21.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.050+0000 7f5c8a800640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c70004030 0x7f5c8c15b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:16:21.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.050+0000 7f5c8b200640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c8c0ca1e0 0x7f5c8c1f7cf0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c06dd90 tx=0x7f5c8400f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:21.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.051+0000 7f5c82a00640 1 -- 172.21.15.45:0/369714696 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c84004440 con 0x7f5c8c0ca1e0 2024-04-16T16:16:21.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.051+0000 7f5c82a00640 1 -- 172.21.15.45:0/369714696 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c84019070 con 0x7f5c8c0ca1e0 2024-04-16T16:16:21.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.051+0000 7f5c91400640 1 -- 172.21.15.45:0/369714696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c8c0c8690 con 0x7f5c8c0ca1e0 2024-04-16T16:16:21.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.051+0000 7f5c82a00640 1 -- 172.21.15.45:0/369714696 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c84003850 con 0x7f5c8c0ca1e0 2024-04-16T16:16:21.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.051+0000 7f5c91400640 1 -- 172.21.15.45:0/369714696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c8c0c8c50 con 0x7f5c8c0ca1e0 2024-04-16T16:16:21.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.052+0000 7f5c91400640 1 -- 172.21.15.45:0/369714696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c8c0c8480 con 0x7f5c8c0ca1e0 2024-04-16T16:16:21.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.053+0000 7f5c82a00640 1 -- 172.21.15.45:0/369714696 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f5c84012070 con 0x7f5c8c0ca1e0 2024-04-16T16:16:21.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.054+0000 7f5c82a00640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5c6806cb30 0x7f5c6806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.054+0000 7f5c82a00640 1 -- 172.21.15.45:0/369714696 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6729+0+0 (secure 0 0 0) 0x7f5c8408bcb0 con 0x7f5c8c0ca1e0 2024-04-16T16:16:21.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.055+0000 7f5c8a800640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5c6806cb30 0x7f5c6806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.056+0000 7f5c8a800640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5c6806cb30 0x7f5c6806eff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c005fd0 tx=0x7f5c7c009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:21.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.059+0000 7f5c82a00640 1 -- 172.21.15.45:0/369714696 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c840598e0 con 0x7f5c8c0ca1e0 2024-04-16T16:16:21.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.266+0000 7f5c91400640 1 -- 172.21.15.45:0/369714696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs2_metadata", "var": "pg_num"} v 0) -- 0x7f5c8c075780 con 0x7f5c8c0ca1e0 2024-04-16T16:16:21.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.266+0000 7f5c82a00640 1 -- 172.21.15.45:0/369714696 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs2_metadata", "var": "pg_num"}]=0 v60) ==== 101+0+11 (secure 0 0 0) 0x7f5c840591e0 con 0x7f5c8c0ca1e0 2024-04-16T16:16:21.265 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:16:21.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.269+0000 7f5c91400640 1 -- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5c6806cb30 msgr2=0x7f5c6806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:21.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.269+0000 7f5c91400640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5c6806cb30 0x7f5c6806eff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c005fd0 tx=0x7f5c7c009040 comp rx=0 tx=0).stop 2024-04-16T16:16:21.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.269+0000 7f5c91400640 1 -- 172.21.15.45:0/369714696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c8c0ca1e0 msgr2=0x7f5c8c1f7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:21.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.269+0000 7f5c91400640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c8c0ca1e0 0x7f5c8c1f7cf0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c06dd90 tx=0x7f5c8400f8a0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.269+0000 7f5c91400640 1 -- 172.21.15.45:0/369714696 shutdown_connections 2024-04-16T16:16:21.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.269+0000 7f5c91400640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5c6806cb30 0x7f5c6806eff0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.269+0000 7f5c91400640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c8c0ca1e0 0x7f5c8c1f7cf0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.269+0000 7f5c91400640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c8c0c9170 0x7f5c8c0c9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.269+0000 7f5c91400640 1 --2- 172.21.15.45:0/369714696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c70004030 0x7f5c8c15b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.269+0000 7f5c91400640 1 -- 172.21.15.45:0/369714696 >> 172.21.15.45:0/369714696 conn(0x7f5c8c073690 msgr2=0x7f5c8c0cf330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:21.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.269+0000 7f5c91400640 1 -- 172.21.15.45:0/369714696 shutdown_connections 2024-04-16T16:16:21.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.269+0000 7f5c91400640 1 -- 172.21.15.45:0/369714696 wait complete. 2024-04-16T16:16:21.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get fs2_data pg_num 2024-04-16T16:16:21.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.512+0000 7fd264400640 1 Processor -- start 2024-04-16T16:16:21.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.513+0000 7fd264400640 1 -- start start 2024-04-16T16:16:21.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.514+0000 7fd264400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd25c09ac30 0x7fd25c09b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.514+0000 7fd264400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd25c0d0a00 con 0x7fd25c06d450 2024-04-16T16:16:21.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.514+0000 7fd264400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd25c0d0ca0 con 0x7fd25c06d8b0 2024-04-16T16:16:21.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.514+0000 7fd264400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd25c0d15d0 con 0x7fd25c09ac30 2024-04-16T16:16:21.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.515+0000 7fd261a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd25c09ac30 0x7fd25c09b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.515+0000 7fd261a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd25c09ac30 0x7fd25c09b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49428/0 (socket says 172.21.15.45:49428) 2024-04-16T16:16:21.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.515+0000 7fd261a00640 1 -- 172.21.15.45:0/438235122 learned_addr learned my addr 172.21.15.45:0/438235122 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:21.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.515+0000 7fd261a00640 1 -- 172.21.15.45:0/438235122 >> v1:172.21.15.103:6789/0 conn(0x7fd25c06d8b0 legacy=0x7fd25c0d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:21.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.515+0000 7fd261a00640 1 -- 172.21.15.45:0/438235122 >> v1:172.21.15.45:6789/0 conn(0x7fd25c06d450 legacy=0x7fd25c1607f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:21.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.515+0000 7fd261a00640 1 -- 172.21.15.45:0/438235122 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25c0d0f80 con 0x7fd25c09ac30 2024-04-16T16:16:21.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.516+0000 7fd261a00640 1 --2- 172.21.15.45:0/438235122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c09ac30 0x7fd25c09b010 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd250009870 tx=0x7fd25002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d502be25fa2ece0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.517+0000 7fd25be00640 1 -- 172.21.15.45:0/438235122 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd250034070 con 0x7fd25c09ac30 2024-04-16T16:16:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.517+0000 7fd25be00640 1 -- 172.21.15.45:0/438235122 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd250030860 con 0x7fd25c09ac30 2024-04-16T16:16:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.517+0000 7fd25be00640 1 -- 172.21.15.45:0/438235122 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd250030c30 con 0x7fd25c09ac30 2024-04-16T16:16:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.517+0000 7fd264400640 1 -- 172.21.15.45:0/438235122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c09ac30 msgr2=0x7fd25c09b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.517+0000 7fd264400640 1 --2- 172.21.15.45:0/438235122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c09ac30 0x7fd25c09b010 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd250009870 tx=0x7fd25002b9a0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.517+0000 7fd264400640 1 -- 172.21.15.45:0/438235122 shutdown_connections 2024-04-16T16:16:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.517+0000 7fd264400640 1 --2- 172.21.15.45:0/438235122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c09ac30 0x7fd25c09b010 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.517+0000 7fd264400640 1 -- 172.21.15.45:0/438235122 >> 172.21.15.45:0/438235122 conn(0x7fd25c0d2870 msgr2=0x7fd25c0d2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.518+0000 7fd264400640 1 -- 172.21.15.45:0/438235122 shutdown_connections 2024-04-16T16:16:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.518+0000 7fd264400640 1 -- 172.21.15.45:0/438235122 wait complete. 2024-04-16T16:16:21.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.519+0000 7fd264400640 1 Processor -- start 2024-04-16T16:16:21.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.520+0000 7fd264400640 1 -- start start 2024-04-16T16:16:21.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.520+0000 7fd264400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd25c06d450 0x7fd25c0cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd264400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd25c06d8b0 0x7fd25c155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd264400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c09ac30 0x7fd25c0c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd264400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c09b550 con 0x7fd25c09ac30 2024-04-16T16:16:21.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd264400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c06df10 con 0x7fd25c06d8b0 2024-04-16T16:16:21.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd264400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd25c160330 con 0x7fd25c06d450 2024-04-16T16:16:21.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd262400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c09ac30 0x7fd25c0c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd262400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c09ac30 0x7fd25c0c7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49430/0 (socket says 172.21.15.45:49430) 2024-04-16T16:16:21.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd262400640 1 -- 172.21.15.45:0/2147347232 learned_addr learned my addr 172.21.15.45:0/2147347232 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:21.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd261000640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd25c06d8b0 0x7fd25c155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd261a00640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd25c06d450 0x7fd25c0cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd262400640 1 -- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd25c06d450 msgr2=0x7fd25c0cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:21.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd262400640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd25c06d450 0x7fd25c0cc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd262400640 1 -- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd25c06d8b0 msgr2=0x7fd25c155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:21.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd262400640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd25c06d8b0 0x7fd25c155cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd262400640 1 -- 172.21.15.45:0/2147347232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25c0b5610 con 0x7fd25c09ac30 2024-04-16T16:16:21.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd261000640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd25c06d8b0 0x7fd25c155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:21.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.521+0000 7fd261a00640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd25c06d450 0x7fd25c0cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:21.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.522+0000 7fd262400640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c09ac30 0x7fd25c0c7ae0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd25400b970 tx=0x7fd25400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:21.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.522+0000 7fd259600640 1 -- 172.21.15.45:0/2147347232 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254019070 con 0x7fd25c09ac30 2024-04-16T16:16:21.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.522+0000 7fd259600640 1 -- 172.21.15.45:0/2147347232 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd25400cb40 con 0x7fd25c09ac30 2024-04-16T16:16:21.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.522+0000 7fd259600640 1 -- 172.21.15.45:0/2147347232 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254014440 con 0x7fd25c09ac30 2024-04-16T16:16:21.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.522+0000 7fd264400640 1 -- 172.21.15.45:0/2147347232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd25c0b58f0 con 0x7fd25c09ac30 2024-04-16T16:16:21.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.522+0000 7fd264400640 1 -- 172.21.15.45:0/2147347232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd25c0b5eb0 con 0x7fd25c09ac30 2024-04-16T16:16:21.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.524+0000 7fd264400640 1 -- 172.21.15.45:0/2147347232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd25c069280 con 0x7fd25c09ac30 2024-04-16T16:16:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.525+0000 7fd259600640 1 -- 172.21.15.45:0/2147347232 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fd25401c070 con 0x7fd25c09ac30 2024-04-16T16:16:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.525+0000 7fd259600640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd23c06cb30 0x7fd23c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.526+0000 7fd259600640 1 -- 172.21.15.45:0/2147347232 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6729+0+0 (secure 0 0 0) 0x7fd25408bf70 con 0x7fd25c09ac30 2024-04-16T16:16:21.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.526+0000 7fd261a00640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd23c06cb30 0x7fd23c06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.527+0000 7fd261a00640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd23c06cb30 0x7fd23c06eff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd250036000 tx=0x7fd25003b040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:21.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.531+0000 7fd259600640 1 -- 172.21.15.45:0/2147347232 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd25c069280 con 0x7fd25c09ac30 2024-04-16T16:16:21.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.733+0000 7fd264400640 1 -- 172.21.15.45:0/2147347232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "fs2_data", "var": "pg_num"} v 0) -- 0x7fd25c0d3760 con 0x7fd25c09ac30 2024-04-16T16:16:21.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.734+0000 7fd259600640 1 -- 172.21.15.45:0/2147347232 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "fs2_data", "var": "pg_num"}]=0 v60) ==== 97+0+11 (secure 0 0 0) 0x7fd254059c30 con 0x7fd25c09ac30 2024-04-16T16:16:21.733 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:16:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.736+0000 7fd264400640 1 -- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd23c06cb30 msgr2=0x7fd23c06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.736+0000 7fd264400640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd23c06cb30 0x7fd23c06eff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd250036000 tx=0x7fd25003b040 comp rx=0 tx=0).stop 2024-04-16T16:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.736+0000 7fd264400640 1 -- 172.21.15.45:0/2147347232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c09ac30 msgr2=0x7fd25c0c7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.736+0000 7fd264400640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c09ac30 0x7fd25c0c7ae0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd25400b970 tx=0x7fd25400be30 comp rx=0 tx=0).stop 2024-04-16T16:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.736+0000 7fd264400640 1 -- 172.21.15.45:0/2147347232 shutdown_connections 2024-04-16T16:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.736+0000 7fd264400640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd23c06cb30 0x7fd23c06eff0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.736+0000 7fd264400640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c09ac30 0x7fd25c0c7ae0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.736+0000 7fd264400640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd25c06d8b0 0x7fd25c155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.736+0000 7fd264400640 1 --2- 172.21.15.45:0/2147347232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd25c06d450 0x7fd25c0cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.736+0000 7fd264400640 1 -- 172.21.15.45:0/2147347232 >> 172.21.15.45:0/2147347232 conn(0x7fd25c0d2870 msgr2=0x7fd25c0d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.737+0000 7fd264400640 1 -- 172.21.15.45:0/2147347232 shutdown_connections 2024-04-16T16:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.737+0000 7fd264400640 1 -- 172.21.15.45:0/2147347232 wait complete. 2024-04-16T16:16:21.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:16:21.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.965+0000 7fceeac00640 1 Processor -- start 2024-04-16T16:16:21.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.965+0000 7fceeac00640 1 -- start start 2024-04-16T16:16:21.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.967+0000 7fceeac00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcee4073320 0x7fcee40b4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.967+0000 7fceeac00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcee406e2b0 0x7fcee415bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.967+0000 7fceeac00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcee406e870 con 0x7fcee4141540 2024-04-16T16:16:21.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.967+0000 7fceeac00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fcee4070240 con 0x7fcee4073320 2024-04-16T16:16:21.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.967+0000 7fceeac00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcee406eb50 con 0x7fcee406e2b0 2024-04-16T16:16:21.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.968+0000 7fcedbe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcee4073320 0x7fcee40b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.968+0000 7fcedbe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcee4073320 0x7fcee40b4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49436/0 (socket says 172.21.15.45:49436) 2024-04-16T16:16:21.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.968+0000 7fcedbe00640 1 -- 172.21.15.45:0/2703882590 learned_addr learned my addr 172.21.15.45:0/2703882590 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:21.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.968+0000 7fcee8c00640 1 --2- 172.21.15.45:0/2703882590 >> v2:172.21.15.103:3300/0 conn(0x7fcee406e2b0 0x7fcee415bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.968+0000 7fcedbe00640 1 -- 172.21.15.45:0/2703882590 >> v2:172.21.15.103:3300/0 conn(0x7fcee406e2b0 msgr2=0x7fcee415bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:21.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.968+0000 7fcedbe00640 1 --2- 172.21.15.45:0/2703882590 >> v2:172.21.15.103:3300/0 conn(0x7fcee406e2b0 0x7fcee415bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.968+0000 7fcedbe00640 1 -- 172.21.15.45:0/2703882590 >> v1:172.21.15.103:6789/0 conn(0x7fcee4141540 legacy=0x7fcee4072c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:21.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.968+0000 7fcedbe00640 1 -- 172.21.15.45:0/2703882590 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee406ee30 con 0x7fcee4073320 2024-04-16T16:16:21.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.969+0000 7fcedbe00640 1 --2- 172.21.15.45:0/2703882590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcee4073320 0x7fcee40b4020 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fcee40b5f20 tx=0x7fcecc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e936f10b873e512e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:21.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.970+0000 7fcee3400640 1 -- 172.21.15.45:0/2703882590 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc02e070 con 0x7fcee4073320 2024-04-16T16:16:21.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.970+0000 7fcee3400640 1 -- 172.21.15.45:0/2703882590 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcecc037070 con 0x7fcee4073320 2024-04-16T16:16:21.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.970+0000 7fcee3400640 1 -- 172.21.15.45:0/2703882590 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc031a00 con 0x7fcee4073320 2024-04-16T16:16:21.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.970+0000 7fceeac00640 1 -- 172.21.15.45:0/2703882590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcee4073320 msgr2=0x7fcee40b4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:21.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.970+0000 7fceeac00640 1 --2- 172.21.15.45:0/2703882590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcee4073320 0x7fcee40b4020 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fcee40b5f20 tx=0x7fcecc02b880 comp rx=0 tx=0).stop 2024-04-16T16:16:21.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.971+0000 7fceeac00640 1 -- 172.21.15.45:0/2703882590 shutdown_connections 2024-04-16T16:16:21.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.971+0000 7fceeac00640 1 --2- 172.21.15.45:0/2703882590 >> v2:172.21.15.103:3300/0 conn(0x7fcee406e2b0 0x7fcee415bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.971+0000 7fceeac00640 1 --2- 172.21.15.45:0/2703882590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcee4073320 0x7fcee40b4020 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.971+0000 7fceeac00640 1 -- 172.21.15.45:0/2703882590 >> 172.21.15.45:0/2703882590 conn(0x7fcee40b2750 msgr2=0x7fcee40b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:21.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.971+0000 7fceeac00640 1 -- 172.21.15.45:0/2703882590 shutdown_connections 2024-04-16T16:16:21.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.971+0000 7fceeac00640 1 -- 172.21.15.45:0/2703882590 wait complete. 2024-04-16T16:16:21.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.973+0000 7fceeac00640 1 Processor -- start 2024-04-16T16:16:21.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.973+0000 7fceeac00640 1 -- start start 2024-04-16T16:16:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.975+0000 7fceeac00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcee406e2b0 0x7fcee414f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.975+0000 7fceeac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcee4073320 0x7fcee414df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.975+0000 7fceeac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee4141540 0x7fcee4150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.975+0000 7fceeac00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcee4072380 con 0x7fcee406e2b0 2024-04-16T16:16:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.975+0000 7fceeac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcee40cf3b0 con 0x7fcee4141540 2024-04-16T16:16:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.975+0000 7fceeac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcee4072500 con 0x7fcee4073320 2024-04-16T16:16:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.975+0000 7fcee3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcee406e2b0 0x7fcee414f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.975+0000 7fcee3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcee406e2b0 0x7fcee414f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49448/0 (socket says 172.21.15.45:49448) 2024-04-16T16:16:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.975+0000 7fcee3e00640 1 -- 172.21.15.45:0/559228576 learned_addr learned my addr 172.21.15.45:0/559228576 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.976+0000 7fcee8c00640 1 --2- 172.21.15.45:0/559228576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee4141540 0x7fcee4150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.976+0000 7fcedbe00640 1 --2- 172.21.15.45:0/559228576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcee4073320 0x7fcee414df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.976+0000 7fcee3e00640 1 -- 172.21.15.45:0/559228576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcee4073320 msgr2=0x7fcee414df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:21.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.976+0000 7fcee3e00640 1 --2- 172.21.15.45:0/559228576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcee4073320 0x7fcee414df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.976+0000 7fcee3e00640 1 -- 172.21.15.45:0/559228576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee4141540 msgr2=0x7fcee4150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:21.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.976+0000 7fcee3e00640 1 --2- 172.21.15.45:0/559228576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee4141540 0x7fcee4150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:21.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.976+0000 7fcee3e00640 1 -- 172.21.15.45:0/559228576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee414e8e0 con 0x7fcee406e2b0 2024-04-16T16:16:21.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.976+0000 7fcee8c00640 1 --2- 172.21.15.45:0/559228576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee4141540 0x7fcee4150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:21.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.976+0000 7fcee3e00640 1 --2- 172.21.15.45:0/559228576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcee406e2b0 0x7fcee414f980 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fced400b3f0 tx=0x7fced400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.976+0000 7fcee0c00640 1 -- 172.21.15.45:0/559228576 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced4015020 con 0x7fcee406e2b0 2024-04-16T16:16:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.976+0000 7fcee0c00640 1 -- 172.21.15.45:0/559228576 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fced4011830 con 0x7fcee406e2b0 2024-04-16T16:16:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.976+0000 7fcee0c00640 1 -- 172.21.15.45:0/559228576 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced4011c20 con 0x7fcee406e2b0 2024-04-16T16:16:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.976+0000 7fceeac00640 1 -- 172.21.15.45:0/559228576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcee414ebc0 con 0x7fcee406e2b0 2024-04-16T16:16:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.977+0000 7fceeac00640 1 -- 172.21.15.45:0/559228576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcee414f180 con 0x7fcee406e2b0 2024-04-16T16:16:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.978+0000 7fceeac00640 1 -- 172.21.15.45:0/559228576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcee4075780 con 0x7fcee406e2b0 2024-04-16T16:16:21.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.979+0000 7fcee0c00640 1 -- 172.21.15.45:0/559228576 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fced4018020 con 0x7fcee406e2b0 2024-04-16T16:16:21.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.980+0000 7fcee0c00640 1 --2- 172.21.15.45:0/559228576 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fcec006cb30 0x7fcec006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:21.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.980+0000 7fcee0c00640 1 -- 172.21.15.45:0/559228576 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6729+0+0 (secure 0 0 0) 0x7fced408bff0 con 0x7fcee406e2b0 2024-04-16T16:16:21.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.981+0000 7fcedbe00640 1 --2- 172.21.15.45:0/559228576 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fcec006cb30 0x7fcec006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:21.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.982+0000 7fcedbe00640 1 --2- 172.21.15.45:0/559228576 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fcec006cb30 0x7fcec006eff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcecc036000 tx=0x7fcecc008030 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:21.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:21.984+0000 7fcee0c00640 1 -- 172.21.15.45:0/559228576 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fced401f030 con 0x7fcee406e2b0 2024-04-16T16:16:22.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.231+0000 7fceeac00640 1 -- 172.21.15.45:0/559228576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcee413df40 con 0x7fcee406e2b0 2024-04-16T16:16:22.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.233+0000 7fcee0c00640 1 -- 172.21.15.45:0/559228576 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5180 (secure 0 0 0) 0x7fced4059cd0 con 0x7fcee406e2b0 2024-04-16T16:16:22.232 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:22.232 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":42,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/1497040472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1497040472},{"type":"v1","addr":"172.21.15.45:6834","nonce":1497040472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":4907,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/93057991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":93057991},{"type":"v1","addr":"172.21.15.45:6835","nonce":93057991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37},{"gid":5404,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1631198273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1631198273},{"type":"v1","addr":"172.21.15.103:6836","nonce":1631198273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:35.289901+0000","modified":"2024-04-16T16:15:44.272258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"b","rank":0,"incarnation":35,"state":"up:active","state_seq":4,"addr":"172.21.15.103:6837/1576112884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1576112884},{"type":"v1","addr":"172.21.15.103:6837","nonce":1576112884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"fs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":4}]} 2024-04-16T16:16:22.232 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 42 2024-04-16T16:16:22.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.236+0000 7fceeac00640 1 -- 172.21.15.45:0/559228576 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fcec006cb30 msgr2=0x7fcec006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:22.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.236+0000 7fceeac00640 1 --2- 172.21.15.45:0/559228576 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fcec006cb30 0x7fcec006eff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcecc036000 tx=0x7fcecc008030 comp rx=0 tx=0).stop 2024-04-16T16:16:22.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.236+0000 7fceeac00640 1 -- 172.21.15.45:0/559228576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcee406e2b0 msgr2=0x7fcee414f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:22.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.236+0000 7fceeac00640 1 --2- 172.21.15.45:0/559228576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcee406e2b0 0x7fcee414f980 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fced400b3f0 tx=0x7fced400b8b0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.236+0000 7fceeac00640 1 -- 172.21.15.45:0/559228576 shutdown_connections 2024-04-16T16:16:22.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.236+0000 7fceeac00640 1 --2- 172.21.15.45:0/559228576 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fcec006cb30 0x7fcec006eff0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.236+0000 7fceeac00640 1 --2- 172.21.15.45:0/559228576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee4141540 0x7fcee4150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.236+0000 7fceeac00640 1 --2- 172.21.15.45:0/559228576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcee4073320 0x7fcee414df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.236+0000 7fceeac00640 1 --2- 172.21.15.45:0/559228576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcee406e2b0 0x7fcee414f980 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.236+0000 7fceeac00640 1 -- 172.21.15.45:0/559228576 >> 172.21.15.45:0/559228576 conn(0x7fcee40b2750 msgr2=0x7fcee415b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:22.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.236+0000 7fceeac00640 1 -- 172.21.15.45:0/559228576 shutdown_connections 2024-04-16T16:16:22.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.236+0000 7fceeac00640 1 -- 172.21.15.45:0/559228576 wait complete. 2024-04-16T16:16:22.255 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:16:22.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.481+0000 7f46d0a00640 1 Processor -- start 2024-04-16T16:16:22.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.481+0000 7f46d0a00640 1 -- start start 2024-04-16T16:16:22.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.483+0000 7f46d0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46cc146220 0x7f46cc15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.483+0000 7f46d0a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f46cc06f7a0 0x7f46cc06df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.483+0000 7f46d0a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f46cc06e480 0x7f46cc1608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.483+0000 7f46d0a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f46cc160410 con 0x7f46cc06e480 2024-04-16T16:16:22.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.483+0000 7f46d0a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f46cc15d360 con 0x7f46cc06f7a0 2024-04-16T16:16:22.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.483+0000 7f46d0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f46cc15d640 con 0x7f46cc146220 2024-04-16T16:16:22.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.483+0000 7f46cb200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f46cc06e480 0x7f46cc1608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.483+0000 7f46cb200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f46cc06e480 0x7f46cc1608a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49452/0 (socket says 172.21.15.45:49452) 2024-04-16T16:16:22.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.483+0000 7f46cb200640 1 -- 172.21.15.45:0/1857311763 learned_addr learned my addr 172.21.15.45:0/1857311763 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:22.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.483+0000 7f46c9e00640 1 --2- 172.21.15.45:0/1857311763 >> v2:172.21.15.103:3300/0 conn(0x7f46cc06f7a0 0x7f46cc06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.483+0000 7f46ca800640 1 --2- 172.21.15.45:0/1857311763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46cc146220 0x7f46cc15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.484+0000 7f46cb200640 1 -- 172.21.15.45:0/1857311763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46cc146220 msgr2=0x7f46cc15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:22.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.484+0000 7f46cb200640 1 --2- 172.21.15.45:0/1857311763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46cc146220 0x7f46cc15ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.484+0000 7f46cb200640 1 -- 172.21.15.45:0/1857311763 >> v2:172.21.15.103:3300/0 conn(0x7f46cc06f7a0 msgr2=0x7f46cc06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:22.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.484+0000 7f46cb200640 1 --2- 172.21.15.45:0/1857311763 >> v2:172.21.15.103:3300/0 conn(0x7f46cc06f7a0 0x7f46cc06df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.484+0000 7f46cb200640 1 -- 172.21.15.45:0/1857311763 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc15d920 con 0x7f46cc06e480 2024-04-16T16:16:22.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.484+0000 7f46ca800640 1 --2- 172.21.15.45:0/1857311763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46cc146220 0x7f46cc15ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.485+0000 7f46cb200640 1 --2- 172.21.15.45:0/1857311763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46cc06e480 0x7f46cc1608a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f46cc070330 tx=0x7f46c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee64abf66d5610b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.485+0000 7f46c9400640 1 -- 172.21.15.45:0/1857311763 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c4035020 con 0x7f46cc06e480 2024-04-16T16:16:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.485+0000 7f46c9400640 1 -- 172.21.15.45:0/1857311763 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46c4031870 con 0x7f46cc06e480 2024-04-16T16:16:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.485+0000 7f46c9400640 1 -- 172.21.15.45:0/1857311763 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c4031c40 con 0x7f46cc06e480 2024-04-16T16:16:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.486+0000 7f46d0a00640 1 -- 172.21.15.45:0/1857311763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46cc06e480 msgr2=0x7f46cc1608a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.486+0000 7f46d0a00640 1 --2- 172.21.15.45:0/1857311763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46cc06e480 0x7f46cc1608a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f46cc070330 tx=0x7f46c402b910 comp rx=0 tx=0).stop 2024-04-16T16:16:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.486+0000 7f46d0a00640 1 -- 172.21.15.45:0/1857311763 shutdown_connections 2024-04-16T16:16:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.486+0000 7f46d0a00640 1 --2- 172.21.15.45:0/1857311763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46cc06e480 0x7f46cc1608a0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.486+0000 7f46d0a00640 1 --2- 172.21.15.45:0/1857311763 >> v2:172.21.15.103:3300/0 conn(0x7f46cc06f7a0 0x7f46cc06df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.486+0000 7f46d0a00640 1 --2- 172.21.15.45:0/1857311763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46cc146220 0x7f46cc15ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.486+0000 7f46d0a00640 1 -- 172.21.15.45:0/1857311763 >> 172.21.15.45:0/1857311763 conn(0x7f46cc06b930 msgr2=0x7f46cc06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.486+0000 7f46d0a00640 1 -- 172.21.15.45:0/1857311763 shutdown_connections 2024-04-16T16:16:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.486+0000 7f46d0a00640 1 -- 172.21.15.45:0/1857311763 wait complete. 2024-04-16T16:16:22.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.488+0000 7f46d0a00640 1 Processor -- start 2024-04-16T16:16:22.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.488+0000 7f46d0a00640 1 -- start start 2024-04-16T16:16:22.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.489+0000 7f46d0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46cc06e480 0x7f46cc1449a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.489+0000 7f46d0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46cc06f7a0 0x7f46cc145020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.489+0000 7f46d0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46cc146220 0x7f46cc1426c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.489+0000 7f46d0a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc06fda0 con 0x7f46cc06f7a0 2024-04-16T16:16:22.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.489+0000 7f46d0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc06fc20 con 0x7f46cc146220 2024-04-16T16:16:22.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.489+0000 7f46d0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f46cc160de0 con 0x7f46cc06e480 2024-04-16T16:16:22.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.489+0000 7f46c9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46cc06f7a0 0x7f46cc145020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.490+0000 7f46c9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46cc06f7a0 0x7f46cc145020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49464/0 (socket says 172.21.15.45:49464) 2024-04-16T16:16:22.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.490+0000 7f46c9e00640 1 -- 172.21.15.45:0/974674829 learned_addr learned my addr 172.21.15.45:0/974674829 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:22.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.490+0000 7f46ca800640 1 --2- 172.21.15.45:0/974674829 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46cc06e480 0x7f46cc1449a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.490+0000 7f46cb200640 1 --2- 172.21.15.45:0/974674829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46cc146220 0x7f46cc1426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.490+0000 7f46c9e00640 1 -- 172.21.15.45:0/974674829 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46cc06e480 msgr2=0x7f46cc1449a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.490+0000 7f46c9e00640 1 --2- 172.21.15.45:0/974674829 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46cc06e480 0x7f46cc1449a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.490+0000 7f46c9e00640 1 -- 172.21.15.45:0/974674829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46cc146220 msgr2=0x7f46cc1426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.490+0000 7f46c9e00640 1 --2- 172.21.15.45:0/974674829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46cc146220 0x7f46cc1426c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.490+0000 7f46c9e00640 1 -- 172.21.15.45:0/974674829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc145560 con 0x7f46cc06f7a0 2024-04-16T16:16:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.490+0000 7f46cb200640 1 --2- 172.21.15.45:0/974674829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46cc146220 0x7f46cc1426c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.490+0000 7f46c9e00640 1 --2- 172.21.15.45:0/974674829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46cc06f7a0 0x7f46cc145020 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f46b4007e60 tx=0x7f46b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.491+0000 7f46c2a00640 1 -- 172.21.15.45:0/974674829 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b4014620 con 0x7f46cc06f7a0 2024-04-16T16:16:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.491+0000 7f46c2a00640 1 -- 172.21.15.45:0/974674829 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46b4019070 con 0x7f46cc06f7a0 2024-04-16T16:16:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.491+0000 7f46d0a00640 1 -- 172.21.15.45:0/974674829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46cc12de70 con 0x7f46cc06f7a0 2024-04-16T16:16:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.491+0000 7f46c2a00640 1 -- 172.21.15.45:0/974674829 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b4014a00 con 0x7f46cc06f7a0 2024-04-16T16:16:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.491+0000 7f46d0a00640 1 -- 172.21.15.45:0/974674829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46cc12e430 con 0x7f46cc06f7a0 2024-04-16T16:16:22.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.492+0000 7f46d0a00640 1 -- 172.21.15.45:0/974674829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46cc145d60 con 0x7f46cc06f7a0 2024-04-16T16:16:22.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.493+0000 7f46c2a00640 1 -- 172.21.15.45:0/974674829 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f46b4014ba0 con 0x7f46cc06f7a0 2024-04-16T16:16:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.495+0000 7f46c2a00640 1 --2- 172.21.15.45:0/974674829 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f46a806cb30 0x7f46a806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.495+0000 7f46c2a00640 1 -- 172.21.15.45:0/974674829 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6729+0+0 (secure 0 0 0) 0x7f46b408cb80 con 0x7f46cc06f7a0 2024-04-16T16:16:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.495+0000 7f46ca800640 1 --2- 172.21.15.45:0/974674829 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f46a806cb30 0x7f46a806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.496+0000 7f46ca800640 1 --2- 172.21.15.45:0/974674829 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f46a806cb30 0x7f46a806eff0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00a950 tx=0x7f46bc008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:22.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.499+0000 7f46c2a00640 1 -- 172.21.15.45:0/974674829 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f46b405a920 con 0x7f46cc06f7a0 2024-04-16T16:16:22.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.705+0000 7f46d0a00640 1 -- 172.21.15.45:0/974674829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46cc075780 con 0x7f46cc06f7a0 2024-04-16T16:16:22.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.707+0000 7f46c2a00640 1 -- 172.21.15.45:0/974674829 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) ==== 74+0+18188 (secure 0 0 0) 0x7f46b405a220 con 0x7f46cc06f7a0 2024-04-16T16:16:22.707 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:22.707 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":60,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:16:19.838837+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"fs2_metadata","create_time":"2024-04-16T16:15:33.215042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"fs2_data","create_time":"2024-04-16T16:15:34.271426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6834/2626302738":"2024-04-17T16:16:16.497315+0000","172.21.15.103:0/349066714":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/388389014":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/589483918":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3353543342":"2024-04-17T16:16:07.016480+0000","172.21.15.103:6832/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3310077212":"2024-04-17T16:16:07.016480+0000","172.21.15.45:6833/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6832/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6836/2626302738":"2024-04-17T16:16:16.497315+0000","172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.45:6833/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:6833/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6834/88032513":"2024-04-17T16:15:41.221894+0000","172.21.15.45:6835/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:0/644374842":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6835/88032513":"2024-04-17T16:15:41.221894+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:16:22.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.710+0000 7f46d0a00640 1 -- 172.21.15.45:0/974674829 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f46a806cb30 msgr2=0x7f46a806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:22.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.710+0000 7f46d0a00640 1 --2- 172.21.15.45:0/974674829 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f46a806cb30 0x7f46a806eff0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00a950 tx=0x7f46bc008040 comp rx=0 tx=0).stop 2024-04-16T16:16:22.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.710+0000 7f46d0a00640 1 -- 172.21.15.45:0/974674829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46cc06f7a0 msgr2=0x7f46cc145020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:22.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.710+0000 7f46d0a00640 1 --2- 172.21.15.45:0/974674829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46cc06f7a0 0x7f46cc145020 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f46b4007e60 tx=0x7f46b400e4b0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.710+0000 7f46d0a00640 1 -- 172.21.15.45:0/974674829 shutdown_connections 2024-04-16T16:16:22.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.710+0000 7f46d0a00640 1 --2- 172.21.15.45:0/974674829 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f46a806cb30 0x7f46a806eff0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.710+0000 7f46d0a00640 1 --2- 172.21.15.45:0/974674829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46cc146220 0x7f46cc1426c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.710+0000 7f46d0a00640 1 --2- 172.21.15.45:0/974674829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46cc06f7a0 0x7f46cc145020 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.710+0000 7f46d0a00640 1 --2- 172.21.15.45:0/974674829 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46cc06e480 0x7f46cc1449a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.710+0000 7f46d0a00640 1 -- 172.21.15.45:0/974674829 >> 172.21.15.45:0/974674829 conn(0x7f46cc06b930 msgr2=0x7f46cc0b4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:22.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.710+0000 7f46d0a00640 1 -- 172.21.15.45:0/974674829 shutdown_connections 2024-04-16T16:16:22.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.710+0000 7f46d0a00640 1 -- 172.21.15.45:0/974674829 wait complete. 2024-04-16T16:16:22.728 INFO:tasks.cephfs.filesystem:Destroying file system fs2 and related pools 2024-04-16T16:16:22.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:16:22.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.941+0000 7f0857e00640 1 Processor -- start 2024-04-16T16:16:22.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.941+0000 7f0857e00640 1 -- start start 2024-04-16T16:16:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.943+0000 7f0857e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08580ccb20 0x7f08580cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.943+0000 7f0857e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f08580ce9a0 0x7f08580cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.943+0000 7f0857e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f085815ecd0 con 0x7f085815e8f0 2024-04-16T16:16:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.943+0000 7f0857e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f08580ce2e0 con 0x7f08580ce9a0 2024-04-16T16:16:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.943+0000 7f0857e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f08580cd6d0 con 0x7f08580ccb20 2024-04-16T16:16:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.944+0000 7f0856a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f085815e8f0 0x7f08580cc410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47148/0 (socket says 172.21.15.45:47148) 2024-04-16T16:16:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.944+0000 7f0856a00640 1 -- 172.21.15.45:0/739042328 learned_addr learned my addr 172.21.15.45:0/739042328 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.944+0000 7f084e000640 1 --2- 172.21.15.45:0/739042328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08580ccb20 0x7f08580cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.944+0000 7f0857400640 1 --2- 172.21.15.45:0/739042328 >> v2:172.21.15.103:3300/0 conn(0x7f08580ce9a0 0x7f08580cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 -- 172.21.15.45:0/739042328 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1363662877 0 0) 0x7f085815ecd0 con 0x7f085815e8f0 2024-04-16T16:16:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 -- 172.21.15.45:0/739042328 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f083c003520 con 0x7f085815e8f0 2024-04-16T16:16:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 -- 172.21.15.45:0/739042328 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 28730819 0 0) 0x7f083c003520 con 0x7f085815e8f0 2024-04-16T16:16:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 -- 172.21.15.45:0/739042328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08580ccb20 msgr2=0x7f08580cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 --2- 172.21.15.45:0/739042328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08580ccb20 0x7f08580cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 -- 172.21.15.45:0/739042328 >> v2:172.21.15.103:3300/0 conn(0x7f08580ce9a0 msgr2=0x7f08580cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:22.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 --2- 172.21.15.45:0/739042328 >> v2:172.21.15.103:3300/0 conn(0x7f08580ce9a0 0x7f08580cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 -- 172.21.15.45:0/739042328 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08580cd9b0 con 0x7f085815e8f0 2024-04-16T16:16:22.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 -- 172.21.15.45:0/739042328 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f0848001670 con 0x7f085815e8f0 2024-04-16T16:16:22.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 -- 172.21.15.45:0/739042328 >> v1:172.21.15.45:6789/0 conn(0x7f085815e8f0 legacy=0x7f08580cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:22.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 --2- 172.21.15.45:0/739042328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f083c0039b0 0x7f083c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 --2- 172.21.15.45:0/739042328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08580ccb20 0x7f083c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 --2- 172.21.15.45:0/739042328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f083c004d40 0x7f083c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 -- 172.21.15.45:0/739042328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f08580ce2e0 con 0x7f083c004d40 2024-04-16T16:16:22.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 -- 172.21.15.45:0/739042328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f08580cd6d0 con 0x7f083c0039b0 2024-04-16T16:16:22.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.945+0000 7f0856000640 1 -- 172.21.15.45:0/739042328 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f083c00b9c0 con 0x7f08580ccb20 2024-04-16T16:16:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.946+0000 7f0857400640 1 --2- 172.21.15.45:0/739042328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f083c004d40 0x7f083c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.946+0000 7f0856a00640 1 --2- 172.21.15.45:0/739042328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f083c0039b0 0x7f083c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.946+0000 7f084e000640 1 --2- 172.21.15.45:0/739042328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08580ccb20 0x7f083c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.946+0000 7f0857400640 1 -- 172.21.15.45:0/739042328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08580ccb20 msgr2=0x7f083c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.946+0000 7f0857400640 1 --2- 172.21.15.45:0/739042328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08580ccb20 0x7f083c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.946+0000 7f0857400640 1 -- 172.21.15.45:0/739042328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f083c0039b0 msgr2=0x7f083c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.946+0000 7f0857400640 1 --2- 172.21.15.45:0/739042328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f083c0039b0 0x7f083c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.946+0000 7f0857400640 1 -- 172.21.15.45:0/739042328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083c003520 con 0x7f083c004d40 2024-04-16T16:16:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.946+0000 7f084e000640 1 --2- 172.21.15.45:0/739042328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08580ccb20 0x7f083c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.947+0000 7f0857400640 1 --2- 172.21.15.45:0/739042328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f083c004d40 0x7f083c0051c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0850009f90 tx=0x7f083c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1432423191b1fa server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.948+0000 7f0856000640 1 -- 172.21.15.45:0/739042328 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850039070 con 0x7f083c004d40 2024-04-16T16:16:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.948+0000 7f0856000640 1 -- 172.21.15.45:0/739042328 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f085002cc00 con 0x7f083c004d40 2024-04-16T16:16:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.948+0000 7f0856000640 1 -- 172.21.15.45:0/739042328 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850009c10 con 0x7f083c004d40 2024-04-16T16:16:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.948+0000 7f0857e00640 1 -- 172.21.15.45:0/739042328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f083c004d40 msgr2=0x7f083c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.948+0000 7f0857e00640 1 --2- 172.21.15.45:0/739042328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f083c004d40 0x7f083c0051c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0850009f90 tx=0x7f083c005830 comp rx=0 tx=0).stop 2024-04-16T16:16:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.948+0000 7f0857400640 1 -- 172.21.15.45:0/739042328 reap_dead start 2024-04-16T16:16:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.948+0000 7f0857e00640 1 -- 172.21.15.45:0/739042328 shutdown_connections 2024-04-16T16:16:22.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.948+0000 7f0857e00640 1 -- 172.21.15.45:0/739042328 >> 172.21.15.45:0/739042328 conn(0x7f085806b930 msgr2=0x7f085806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:22.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.949+0000 7f0857e00640 1 -- 172.21.15.45:0/739042328 shutdown_connections 2024-04-16T16:16:22.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.949+0000 7f0857e00640 1 -- 172.21.15.45:0/739042328 wait complete. 2024-04-16T16:16:22.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.950+0000 7f0857e00640 1 Processor -- start 2024-04-16T16:16:22.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.950+0000 7f0857e00640 1 -- start start 2024-04-16T16:16:22.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.951+0000 7f0857e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f083c004d40 0x7f0858144d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.951+0000 7f0857e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085815ef00 0x7f085815f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.951+0000 7f0857e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f085815c1f0 0x7f085815c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.951+0000 7f0857e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f08580cd360 con 0x7f085815ef00 2024-04-16T16:16:22.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.951+0000 7f0857e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f08580cdf70 con 0x7f083c004d40 2024-04-16T16:16:22.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.951+0000 7f0857e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f08580cf080 con 0x7f085815c1f0 2024-04-16T16:16:22.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.952+0000 7f084e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085815ef00 0x7f085815f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.952+0000 7f084e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085815ef00 0x7f085815f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49482/0 (socket says 172.21.15.45:49482) 2024-04-16T16:16:22.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.952+0000 7f084e000640 1 -- 172.21.15.45:0/4051122678 learned_addr learned my addr 172.21.15.45:0/4051122678 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:22.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.952+0000 7f0857400640 1 --2- 172.21.15.45:0/4051122678 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f085815c1f0 0x7f085815c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.952+0000 7f0856a00640 1 --2- 172.21.15.45:0/4051122678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f083c004d40 0x7f0858144d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.952+0000 7f084e000640 1 -- 172.21.15.45:0/4051122678 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f085815c1f0 msgr2=0x7f085815c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:22.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.952+0000 7f084e000640 1 --2- 172.21.15.45:0/4051122678 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f085815c1f0 0x7f085815c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.952+0000 7f084e000640 1 -- 172.21.15.45:0/4051122678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f083c004d40 msgr2=0x7f0858144d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:22.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.952+0000 7f084e000640 1 --2- 172.21.15.45:0/4051122678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f083c004d40 0x7f0858144d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:22.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.952+0000 7f084e000640 1 -- 172.21.15.45:0/4051122678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085815f9c0 con 0x7f085815ef00 2024-04-16T16:16:22.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.952+0000 7f0857400640 1 --2- 172.21.15.45:0/4051122678 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f085815c1f0 0x7f085815c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:16:22.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.953+0000 7f084e000640 1 --2- 172.21.15.45:0/4051122678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085815ef00 0x7f085815f380 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f08400108b0 tx=0x7f0840010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:22.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.953+0000 7f084fe00640 1 -- 172.21.15.45:0/4051122678 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084000f040 con 0x7f085815ef00 2024-04-16T16:16:22.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.953+0000 7f084fe00640 1 -- 172.21.15.45:0/4051122678 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f084000eb00 con 0x7f085815ef00 2024-04-16T16:16:22.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.953+0000 7f0857e00640 1 -- 172.21.15.45:0/4051122678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f085815fca0 con 0x7f085815ef00 2024-04-16T16:16:22.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.953+0000 7f084fe00640 1 -- 172.21.15.45:0/4051122678 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084001c400 con 0x7f085815ef00 2024-04-16T16:16:22.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.953+0000 7f0857e00640 1 -- 172.21.15.45:0/4051122678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0858160260 con 0x7f085815ef00 2024-04-16T16:16:22.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.954+0000 7f0857e00640 1 -- 172.21.15.45:0/4051122678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0858075780 con 0x7f085815ef00 2024-04-16T16:16:22.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.955+0000 7f084fe00640 1 -- 172.21.15.45:0/4051122678 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f084001c5a0 con 0x7f085815ef00 2024-04-16T16:16:22.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.956+0000 7f084fe00640 1 --2- 172.21.15.45:0/4051122678 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f083406cb60 0x7f083406f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:22.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.956+0000 7f084fe00640 1 -- 172.21.15.45:0/4051122678 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6729+0+0 (secure 0 0 0) 0x7f084008ca90 con 0x7f085815ef00 2024-04-16T16:16:22.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.957+0000 7f0856a00640 1 --2- 172.21.15.45:0/4051122678 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f083406cb60 0x7f083406f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:22.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.958+0000 7f0856a00640 1 --2- 172.21.15.45:0/4051122678 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f083406cb60 0x7f083406f020 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0848005ad0 tx=0x7f0848009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:22.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:22.961+0000 7f084fe00640 1 -- 172.21.15.45:0/4051122678 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0840018030 con 0x7f085815ef00 2024-04-16T16:16:23.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.205+0000 7f0857e00640 1 -- 172.21.15.45:0/4051122678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08580b5b90 con 0x7f085815ef00 2024-04-16T16:16:23.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.206+0000 7f084fe00640 1 -- 172.21.15.45:0/4051122678 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5180 (secure 0 0 0) 0x7f084005a6c0 con 0x7f085815ef00 2024-04-16T16:16:23.205 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:23.205 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":42,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/1497040472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1497040472},{"type":"v1","addr":"172.21.15.45:6834","nonce":1497040472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":4907,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/93057991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":93057991},{"type":"v1","addr":"172.21.15.45:6835","nonce":93057991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37},{"gid":5404,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1631198273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1631198273},{"type":"v1","addr":"172.21.15.103:6836","nonce":1631198273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:35.289901+0000","modified":"2024-04-16T16:15:44.272258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"b","rank":0,"incarnation":35,"state":"up:active","state_seq":4,"addr":"172.21.15.103:6837/1576112884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1576112884},{"type":"v1","addr":"172.21.15.103:6837","nonce":1576112884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"fs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":4}]} 2024-04-16T16:16:23.205 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 42 2024-04-16T16:16:23.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.209+0000 7f0857e00640 1 -- 172.21.15.45:0/4051122678 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f083406cb60 msgr2=0x7f083406f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:23.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.209+0000 7f0857e00640 1 --2- 172.21.15.45:0/4051122678 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f083406cb60 0x7f083406f020 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0848005ad0 tx=0x7f0848009040 comp rx=0 tx=0).stop 2024-04-16T16:16:23.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.209+0000 7f0857e00640 1 -- 172.21.15.45:0/4051122678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085815ef00 msgr2=0x7f085815f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:23.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.209+0000 7f0857e00640 1 --2- 172.21.15.45:0/4051122678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085815ef00 0x7f085815f380 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f08400108b0 tx=0x7f0840010d70 comp rx=0 tx=0).stop 2024-04-16T16:16:23.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.209+0000 7f0857e00640 1 -- 172.21.15.45:0/4051122678 shutdown_connections 2024-04-16T16:16:23.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.209+0000 7f0857e00640 1 --2- 172.21.15.45:0/4051122678 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f083406cb60 0x7f083406f020 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.209+0000 7f0857e00640 1 --2- 172.21.15.45:0/4051122678 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f085815c1f0 0x7f085815c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.209+0000 7f0857e00640 1 --2- 172.21.15.45:0/4051122678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085815ef00 0x7f085815f380 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.209+0000 7f0857e00640 1 --2- 172.21.15.45:0/4051122678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f083c004d40 0x7f0858144d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.209+0000 7f0857e00640 1 -- 172.21.15.45:0/4051122678 >> 172.21.15.45:0/4051122678 conn(0x7f085806b930 msgr2=0x7f085806e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:23.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.209+0000 7f0857e00640 1 -- 172.21.15.45:0/4051122678 shutdown_connections 2024-04-16T16:16:23.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.209+0000 7f0857e00640 1 -- 172.21.15.45:0/4051122678 wait complete. 2024-04-16T16:16:23.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:16:23.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.446+0000 7ff68ca00640 1 Processor -- start 2024-04-16T16:16:23.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.446+0000 7ff68ca00640 1 -- start start 2024-04-16T16:16:23.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.447+0000 7ff68ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff68806e550 0x7ff68809abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:23.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.447+0000 7ff68ca00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6880ccf50 con 0x7ff688070420 2024-04-16T16:16:23.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.447+0000 7ff68ca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6880cd230 con 0x7ff6880d1990 2024-04-16T16:16:23.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.447+0000 7ff68ca00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff6880cd880 con 0x7ff68806e550 2024-04-16T16:16:23.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.448+0000 7ff685e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff68806e550 0x7ff68809abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:23.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.448+0000 7ff685e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff68806e550 0x7ff68809abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46150/0 (socket says 172.21.15.45:46150) 2024-04-16T16:16:23.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.448+0000 7ff685e00640 1 -- 172.21.15.45:0/3580145363 learned_addr learned my addr 172.21.15.45:0/3580145363 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:23.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.448+0000 7ff685e00640 1 -- 172.21.15.45:0/3580145363 >> v1:172.21.15.103:6789/0 conn(0x7ff6880d1990 legacy=0x7ff6880ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:23.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.448+0000 7ff685e00640 1 -- 172.21.15.45:0/3580145363 >> v1:172.21.15.45:6789/0 conn(0x7ff688070420 legacy=0x7ff6880cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:23.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.449+0000 7ff685e00640 1 -- 172.21.15.45:0/3580145363 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6880cd510 con 0x7ff68806e550 2024-04-16T16:16:23.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.450+0000 7ff685e00640 1 --2- 172.21.15.45:0/3580145363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff68806e550 0x7ff68809abc0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff678009870 tx=0x7ff67802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca20c42d31f8c0b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:23.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.451+0000 7ff684a00640 1 -- 172.21.15.45:0/3580145363 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67802f840 con 0x7ff68806e550 2024-04-16T16:16:23.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.451+0000 7ff684a00640 1 -- 172.21.15.45:0/3580145363 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff67802f9e0 con 0x7ff68806e550 2024-04-16T16:16:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.451+0000 7ff684a00640 1 -- 172.21.15.45:0/3580145363 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67802fdb0 con 0x7ff68806e550 2024-04-16T16:16:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.451+0000 7ff68ca00640 1 -- 172.21.15.45:0/3580145363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff68806e550 msgr2=0x7ff68809abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.451+0000 7ff68ca00640 1 --2- 172.21.15.45:0/3580145363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff68806e550 0x7ff68809abc0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff678009870 tx=0x7ff67802ba20 comp rx=0 tx=0).stop 2024-04-16T16:16:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.451+0000 7ff68ca00640 1 -- 172.21.15.45:0/3580145363 shutdown_connections 2024-04-16T16:16:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.451+0000 7ff68ca00640 1 --2- 172.21.15.45:0/3580145363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff68806e550 0x7ff68809abc0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.451+0000 7ff68ca00640 1 -- 172.21.15.45:0/3580145363 >> 172.21.15.45:0/3580145363 conn(0x7ff6880b4350 msgr2=0x7ff6880b4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:23.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.451+0000 7ff68ca00640 1 -- 172.21.15.45:0/3580145363 shutdown_connections 2024-04-16T16:16:23.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.451+0000 7ff68ca00640 1 -- 172.21.15.45:0/3580145363 wait complete. 2024-04-16T16:16:23.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.453+0000 7ff68ca00640 1 Processor -- start 2024-04-16T16:16:23.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.453+0000 7ff68ca00640 1 -- start start 2024-04-16T16:16:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.454+0000 7ff68ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff68806e550 0x7ff6881ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.454+0000 7ff68ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff688070420 0x7ff68809e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.454+0000 7ff68ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6880d1990 0x7ff68809c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.454+0000 7ff68ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff68806ca40 con 0x7ff688070420 2024-04-16T16:16:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.454+0000 7ff68ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff68806e930 con 0x7ff68806e550 2024-04-16T16:16:23.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.454+0000 7ff68ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff6880b6860 con 0x7ff6880d1990 2024-04-16T16:16:23.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.455+0000 7ff685400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff688070420 0x7ff68809e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:23.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.455+0000 7ff685400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff688070420 0x7ff68809e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46160/0 (socket says 172.21.15.45:46160) 2024-04-16T16:16:23.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.455+0000 7ff685400640 1 -- 172.21.15.45:0/484111407 learned_addr learned my addr 172.21.15.45:0/484111407 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.455+0000 7ff685e00640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff68806e550 0x7ff6881ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.455+0000 7ff686800640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6880d1990 0x7ff68809c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.455+0000 7ff685400640 1 -- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6880d1990 msgr2=0x7ff68809c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.455+0000 7ff685400640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6880d1990 0x7ff68809c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.455+0000 7ff685400640 1 -- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff68806e550 msgr2=0x7ff6881ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.455+0000 7ff685400640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff68806e550 0x7ff6881ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.455+0000 7ff685400640 1 -- 172.21.15.45:0/484111407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6880b6b40 con 0x7ff688070420 2024-04-16T16:16:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.455+0000 7ff685e00640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff68806e550 0x7ff6881ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.455+0000 7ff686800640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6880d1990 0x7ff68809c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.455+0000 7ff685400640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff688070420 0x7ff68809e980 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff67400c450 tx=0x7ff67400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.456+0000 7ff67e000640 1 -- 172.21.15.45:0/484111407 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6740052d0 con 0x7ff688070420 2024-04-16T16:16:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.456+0000 7ff67e000640 1 -- 172.21.15.45:0/484111407 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff674005470 con 0x7ff688070420 2024-04-16T16:16:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.456+0000 7ff67e000640 1 -- 172.21.15.45:0/484111407 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff674012870 con 0x7ff688070420 2024-04-16T16:16:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.456+0000 7ff68ca00640 1 -- 172.21.15.45:0/484111407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6880b6e20 con 0x7ff688070420 2024-04-16T16:16:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.456+0000 7ff68ca00640 1 -- 172.21.15.45:0/484111407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff68809f0a0 con 0x7ff688070420 2024-04-16T16:16:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.457+0000 7ff68ca00640 1 -- 172.21.15.45:0/484111407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6880691e0 con 0x7ff688070420 2024-04-16T16:16:23.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.458+0000 7ff67e000640 1 -- 172.21.15.45:0/484111407 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7ff674012a10 con 0x7ff688070420 2024-04-16T16:16:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.459+0000 7ff67e000640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7ff66806cb30 0x7ff66806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.459+0000 7ff67e000640 1 -- 172.21.15.45:0/484111407 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6729+0+0 (secure 0 0 0) 0x7ff67408bfc0 con 0x7ff688070420 2024-04-16T16:16:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.460+0000 7ff685e00640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7ff66806cb30 0x7ff66806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:23.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.461+0000 7ff685e00640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7ff66806cb30 0x7ff66806eff0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff678036000 tx=0x7ff678039040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:23.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.463+0000 7ff67e000640 1 -- 172.21.15.45:0/484111407 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff6880691e0 con 0x7ff688070420 2024-04-16T16:16:23.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.709+0000 7ff68ca00640 1 -- 172.21.15.45:0/484111407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff688075780 con 0x7ff688070420 2024-04-16T16:16:23.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.711+0000 7ff67e000640 1 -- 172.21.15.45:0/484111407 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5180 (secure 0 0 0) 0x7ff674059c80 con 0x7ff688070420 2024-04-16T16:16:23.710 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:23.710 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":42,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/1497040472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1497040472},{"type":"v1","addr":"172.21.15.45:6834","nonce":1497040472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":4907,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/93057991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":93057991},{"type":"v1","addr":"172.21.15.45:6835","nonce":93057991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37},{"gid":5404,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1631198273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1631198273},{"type":"v1","addr":"172.21.15.103:6836","nonce":1631198273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:15:35.289901+0000","modified":"2024-04-16T16:15:44.272258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"b","rank":0,"incarnation":35,"state":"up:active","state_seq":4,"addr":"172.21.15.103:6837/1576112884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1576112884},{"type":"v1","addr":"172.21.15.103:6837","nonce":1576112884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"fs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":4}]} 2024-04-16T16:16:23.710 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 42 2024-04-16T16:16:23.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.713+0000 7ff68ca00640 1 -- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7ff66806cb30 msgr2=0x7ff66806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:23.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.713+0000 7ff68ca00640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7ff66806cb30 0x7ff66806eff0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff678036000 tx=0x7ff678039040 comp rx=0 tx=0).stop 2024-04-16T16:16:23.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.713+0000 7ff68ca00640 1 -- 172.21.15.45:0/484111407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff688070420 msgr2=0x7ff68809e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:23.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.713+0000 7ff68ca00640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff688070420 0x7ff68809e980 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff67400c450 tx=0x7ff67400c910 comp rx=0 tx=0).stop 2024-04-16T16:16:23.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.713+0000 7ff68ca00640 1 -- 172.21.15.45:0/484111407 shutdown_connections 2024-04-16T16:16:23.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.713+0000 7ff68ca00640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7ff66806cb30 0x7ff66806eff0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.713+0000 7ff68ca00640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6880d1990 0x7ff68809c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.713+0000 7ff68ca00640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff688070420 0x7ff68809e980 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.713+0000 7ff68ca00640 1 --2- 172.21.15.45:0/484111407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff68806e550 0x7ff6881ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.713+0000 7ff68ca00640 1 -- 172.21.15.45:0/484111407 >> 172.21.15.45:0/484111407 conn(0x7ff6880b4350 msgr2=0x7ff68809abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:23.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.714+0000 7ff68ca00640 1 -- 172.21.15.45:0/484111407 shutdown_connections 2024-04-16T16:16:23.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.714+0000 7ff68ca00640 1 -- 172.21.15.45:0/484111407 wait complete. 2024-04-16T16:16:23.732 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:16:23.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:16:23.846 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:16:23.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.950+0000 7fb6e9600640 1 Processor -- start 2024-04-16T16:16:23.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.950+0000 7fb6e9600640 1 -- start start 2024-04-16T16:16:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.952+0000 7fb6e9600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb6e4067bc0 0x7fb6e40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.952+0000 7fb6e9600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6e40d0dd0 con 0x7fb6e415b770 2024-04-16T16:16:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.952+0000 7fb6e9600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6e40d0530 con 0x7fb6e4073ba0 2024-04-16T16:16:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.952+0000 7fb6e9600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb6e40d0810 con 0x7fb6e4067bc0 2024-04-16T16:16:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.952+0000 7fb6e2a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb6e415b770 0x7fb6e4073490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41004/0 (socket says 172.21.15.45:41004) 2024-04-16T16:16:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.952+0000 7fb6e2a00640 1 -- 172.21.15.45:0/873371091 learned_addr learned my addr 172.21.15.45:0/873371091 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:23.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.952+0000 7fb6e2000640 1 -- 172.21.15.45:0/873371091 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 419970039 0 0) 0x7fb6e40d0dd0 con 0x7fb6e415b770 2024-04-16T16:16:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.952+0000 7fb6e2000640 1 -- 172.21.15.45:0/873371091 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6c8003520 con 0x7fb6e415b770 2024-04-16T16:16:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.952+0000 7fb6e3400640 1 --2- 172.21.15.45:0/873371091 >> v2:172.21.15.103:3300/0 conn(0x7fb6e4067bc0 0x7fb6e40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.953+0000 7fb6e2000640 1 -- 172.21.15.45:0/873371091 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 419253261 0 0) 0x7fb6c8003520 con 0x7fb6e415b770 2024-04-16T16:16:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.953+0000 7fb6e2000640 1 -- 172.21.15.45:0/873371091 >> v2:172.21.15.103:3300/0 conn(0x7fb6e4067bc0 msgr2=0x7fb6e40b4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.953+0000 7fb6e2000640 1 --2- 172.21.15.45:0/873371091 >> v2:172.21.15.103:3300/0 conn(0x7fb6e4067bc0 0x7fb6e40b4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.953+0000 7fb6e2000640 1 -- 172.21.15.45:0/873371091 >> v1:172.21.15.103:6789/0 conn(0x7fb6e4073ba0 legacy=0x7fb6e40b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.953+0000 7fb6e2000640 1 -- 172.21.15.45:0/873371091 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e40d0af0 con 0x7fb6e415b770 2024-04-16T16:16:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.953+0000 7fb6e2000640 1 -- 172.21.15.45:0/873371091 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fb6d4001670 con 0x7fb6e415b770 2024-04-16T16:16:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.953+0000 7fb6e2000640 1 -- 172.21.15.45:0/873371091 >> v1:172.21.15.45:6789/0 conn(0x7fb6e415b770 legacy=0x7fb6e4073490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.953+0000 7fb6e2000640 1 --2- 172.21.15.45:0/873371091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6c8003a00 0x7fb6c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.953+0000 7fb6e2000640 1 --2- 172.21.15.45:0/873371091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6c8004800 0x7fb6c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.953+0000 7fb6e2000640 1 --2- 172.21.15.45:0/873371091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c80051c0 0x7fb6c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.953+0000 7fb6e2000640 1 -- 172.21.15.45:0/873371091 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e40d0810 con 0x7fb6c80051c0 2024-04-16T16:16:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.953+0000 7fb6e2000640 1 -- 172.21.15.45:0/873371091 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c800c0c0 con 0x7fb6c8004800 2024-04-16T16:16:23.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.953+0000 7fb6e2000640 1 -- 172.21.15.45:0/873371091 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb6c800c2c0 con 0x7fb6c8003a00 2024-04-16T16:16:23.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.953+0000 7fb6e3400640 1 --2- 172.21.15.45:0/873371091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c80051c0 0x7fb6c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:23.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.953+0000 7fb6dbe00640 1 --2- 172.21.15.45:0/873371091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6c8004800 0x7fb6c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:23.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.954+0000 7fb6e3400640 1 -- 172.21.15.45:0/873371091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6c8003a00 msgr2=0x7fb6c8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:23.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.954+0000 7fb6e3400640 1 --2- 172.21.15.45:0/873371091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6c8003a00 0x7fb6c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.954+0000 7fb6e3400640 1 -- 172.21.15.45:0/873371091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6c8004800 msgr2=0x7fb6c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:23.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.954+0000 7fb6e3400640 1 --2- 172.21.15.45:0/873371091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6c8004800 0x7fb6c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.954+0000 7fb6e3400640 1 -- 172.21.15.45:0/873371091 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c8003520 con 0x7fb6c80051c0 2024-04-16T16:16:23.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.955+0000 7fb6e3400640 1 -- 172.21.15.45:0/873371091 reap_dead start 2024-04-16T16:16:23.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.955+0000 7fb6e3400640 1 --2- 172.21.15.45:0/873371091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c80051c0 0x7fb6c800b9c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009fc0 tx=0x7fb6c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff1dd7ac041c20a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:23.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.955+0000 7fb6e2000640 1 -- 172.21.15.45:0/873371091 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc009c10 con 0x7fb6c80051c0 2024-04-16T16:16:23.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.955+0000 7fb6e2000640 1 -- 172.21.15.45:0/873371091 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6dc02cca0 con 0x7fb6c80051c0 2024-04-16T16:16:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.955+0000 7fb6e2000640 1 -- 172.21.15.45:0/873371091 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc003bd0 con 0x7fb6c80051c0 2024-04-16T16:16:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.956+0000 7fb6e9600640 1 -- 172.21.15.45:0/873371091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c80051c0 msgr2=0x7fb6c800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.956+0000 7fb6e9600640 1 --2- 172.21.15.45:0/873371091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c80051c0 0x7fb6c800b9c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009fc0 tx=0x7fb6c8005830 comp rx=0 tx=0).stop 2024-04-16T16:16:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.956+0000 7fb6e9600640 1 -- 172.21.15.45:0/873371091 shutdown_connections 2024-04-16T16:16:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.956+0000 7fb6e9600640 1 --2- 172.21.15.45:0/873371091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6c80051c0 0x7fb6c800b9c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.956+0000 7fb6e9600640 1 -- 172.21.15.45:0/873371091 >> 172.21.15.45:0/873371091 conn(0x7fb6e4072970 msgr2=0x7fb6e4072d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.956+0000 7fb6e9600640 1 -- 172.21.15.45:0/873371091 shutdown_connections 2024-04-16T16:16:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.956+0000 7fb6e9600640 1 -- 172.21.15.45:0/873371091 wait complete. 2024-04-16T16:16:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.958+0000 7fb6e9600640 1 Processor -- start 2024-04-16T16:16:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.958+0000 7fb6e9600640 1 -- start start 2024-04-16T16:16:23.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e9600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6c80051c0 0x7fb6e416b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:23.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e9600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e414e6e0 0x7fb6e4145eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:23.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e9600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e4169590 0x7fb6e4169a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:23.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e9600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e415bb50 con 0x7fb6e4169590 2024-04-16T16:16:23.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e9600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e40d0150 con 0x7fb6e414e6e0 2024-04-16T16:16:23.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e9600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e41463f0 con 0x7fb6c80051c0 2024-04-16T16:16:23.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e4169590 0x7fb6e4169a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:23.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e4169590 0x7fb6e4169a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46184/0 (socket says 172.21.15.45:46184) 2024-04-16T16:16:23.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e3400640 1 -- 172.21.15.45:0/1836694159 learned_addr learned my addr 172.21.15.45:0/1836694159 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:23.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e2a00640 1 --2- 172.21.15.45:0/1836694159 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6c80051c0 0x7fb6e416b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:23.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e3400640 1 -- 172.21.15.45:0/1836694159 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6c80051c0 msgr2=0x7fb6e416b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:23.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6dbe00640 1 --2- 172.21.15.45:0/1836694159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e414e6e0 0x7fb6e4145eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:23.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e3400640 1 --2- 172.21.15.45:0/1836694159 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6c80051c0 0x7fb6e416b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e3400640 1 -- 172.21.15.45:0/1836694159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e414e6e0 msgr2=0x7fb6e4145eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:23.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e3400640 1 --2- 172.21.15.45:0/1836694159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e414e6e0 0x7fb6e4145eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:23.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e3400640 1 -- 172.21.15.45:0/1836694159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e416a3a0 con 0x7fb6e4169590 2024-04-16T16:16:23.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.959+0000 7fb6e2a00640 1 --2- 172.21.15.45:0/1836694159 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6c80051c0 0x7fb6e416b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:23.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.960+0000 7fb6e3400640 1 --2- 172.21.15.45:0/1836694159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e4169590 0x7fb6e4169a10 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc035d50 tx=0x7fb6dc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:23.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.960+0000 7fb6daa00640 1 -- 172.21.15.45:0/1836694159 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc003990 con 0x7fb6e4169590 2024-04-16T16:16:23.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.960+0000 7fb6daa00640 1 -- 172.21.15.45:0/1836694159 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6dc044070 con 0x7fb6e4169590 2024-04-16T16:16:23.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.960+0000 7fb6daa00640 1 -- 172.21.15.45:0/1836694159 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc003d90 con 0x7fb6e4169590 2024-04-16T16:16:23.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.960+0000 7fb6e9600640 1 -- 172.21.15.45:0/1836694159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e416a680 con 0x7fb6e4169590 2024-04-16T16:16:23.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.960+0000 7fb6e9600640 1 -- 172.21.15.45:0/1836694159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e414adc0 con 0x7fb6e4169590 2024-04-16T16:16:23.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.961+0000 7fb6e9600640 1 -- 172.21.15.45:0/1836694159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6e406d470 con 0x7fb6e4169590 2024-04-16T16:16:23.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.962+0000 7fb6daa00640 1 -- 172.21.15.45:0/1836694159 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fb6dc005d10 con 0x7fb6e4169590 2024-04-16T16:16:23.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.963+0000 7fb6daa00640 1 --2- 172.21.15.45:0/1836694159 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb6c006cb30 0x7fb6c006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:23.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.963+0000 7fb6daa00640 1 -- 172.21.15.45:0/1836694159 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6729+0+0 (secure 0 0 0) 0x7fb6dc0aeaa0 con 0x7fb6e4169590 2024-04-16T16:16:23.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.964+0000 7fb6e2a00640 1 --2- 172.21.15.45:0/1836694159 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb6c006cb30 0x7fb6c006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:23.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.965+0000 7fb6e2a00640 1 --2- 172.21.15.45:0/1836694159 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb6c006cb30 0x7fb6c006eff0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb6d40079f0 tx=0x7fb6d4009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:23.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:23.968+0000 7fb6daa00640 1 -- 172.21.15.45:0/1836694159 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb6dc0333a0 con 0x7fb6e4169590 2024-04-16T16:16:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.175+0000 7fb6e9600640 1 -- 172.21.15.45:0/1836694159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6e4075780 con 0x7fb6e4169590 2024-04-16T16:16:24.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.177+0000 7fb6daa00640 1 -- 172.21.15.45:0/1836694159 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) ==== 74+0+18188 (secure 0 0 0) 0x7fb6dc07c770 con 0x7fb6e4169590 2024-04-16T16:16:24.177 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:24.177 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":60,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:16:19.838837+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"fs2_metadata","create_time":"2024-04-16T16:15:33.215042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"fs2_data","create_time":"2024-04-16T16:15:34.271426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"fs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6834/2626302738":"2024-04-17T16:16:16.497315+0000","172.21.15.103:0/349066714":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/388389014":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/589483918":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3353543342":"2024-04-17T16:16:07.016480+0000","172.21.15.103:6832/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3310077212":"2024-04-17T16:16:07.016480+0000","172.21.15.45:6833/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6832/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6836/2626302738":"2024-04-17T16:16:16.497315+0000","172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.45:6833/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:6833/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6834/88032513":"2024-04-17T16:15:41.221894+0000","172.21.15.45:6835/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:0/644374842":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6835/88032513":"2024-04-17T16:15:41.221894+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:16:24.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.180+0000 7fb6e9600640 1 -- 172.21.15.45:0/1836694159 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb6c006cb30 msgr2=0x7fb6c006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:24.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.180+0000 7fb6e9600640 1 --2- 172.21.15.45:0/1836694159 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb6c006cb30 0x7fb6c006eff0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb6d40079f0 tx=0x7fb6d4009040 comp rx=0 tx=0).stop 2024-04-16T16:16:24.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.180+0000 7fb6e9600640 1 -- 172.21.15.45:0/1836694159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e4169590 msgr2=0x7fb6e4169a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:24.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.180+0000 7fb6e9600640 1 --2- 172.21.15.45:0/1836694159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e4169590 0x7fb6e4169a10 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc035d50 tx=0x7fb6dc009750 comp rx=0 tx=0).stop 2024-04-16T16:16:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.180+0000 7fb6e9600640 1 -- 172.21.15.45:0/1836694159 shutdown_connections 2024-04-16T16:16:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.180+0000 7fb6e9600640 1 --2- 172.21.15.45:0/1836694159 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb6c006cb30 0x7fb6c006eff0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.180+0000 7fb6e9600640 1 --2- 172.21.15.45:0/1836694159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e4169590 0x7fb6e4169a10 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.180+0000 7fb6e9600640 1 --2- 172.21.15.45:0/1836694159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e414e6e0 0x7fb6e4145eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.180+0000 7fb6e9600640 1 --2- 172.21.15.45:0/1836694159 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6c80051c0 0x7fb6e416b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.180+0000 7fb6e9600640 1 -- 172.21.15.45:0/1836694159 >> 172.21.15.45:0/1836694159 conn(0x7fb6e4072970 msgr2=0x7fb6e406c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.180+0000 7fb6e9600640 1 -- 172.21.15.45:0/1836694159 shutdown_connections 2024-04-16T16:16:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.180+0000 7fb6e9600640 1 -- 172.21.15.45:0/1836694159 wait complete. 2024-04-16T16:16:24.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail fs2 2024-04-16T16:16:24.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.473+0000 7f6f36400640 1 Processor -- start 2024-04-16T16:16:24.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.473+0000 7f6f36400640 1 -- start start 2024-04-16T16:16:24.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.475+0000 7f6f36400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f30146220 0x7f6f300cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:24.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.475+0000 7f6f36400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6f30072180 0x7f6f300725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:24.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.475+0000 7f6f36400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f30146600 con 0x7f6f300cd220 2024-04-16T16:16:24.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.475+0000 7f6f36400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f6f3006f840 con 0x7f6f30072180 2024-04-16T16:16:24.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.475+0000 7f6f36400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6f300ce0e0 con 0x7f6f30146220 2024-04-16T16:16:24.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.475+0000 7f6f2ea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6f30072180 0x7f6f300725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:24.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.475+0000 7f6f2ea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6f30072180 0x7f6f300725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46194/0 (socket says 172.21.15.45:46194) 2024-04-16T16:16:24.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.475+0000 7f6f2ea00640 1 -- 172.21.15.45:0/3294193598 learned_addr learned my addr 172.21.15.45:0/3294193598 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:24.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.476+0000 7f6f2f400640 1 --2- 172.21.15.45:0/3294193598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f30146220 0x7f6f300cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:24.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.476+0000 7f6f2ea00640 1 -- 172.21.15.45:0/3294193598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f30146220 msgr2=0x7f6f300cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:24.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.476+0000 7f6f2ea00640 1 --2- 172.21.15.45:0/3294193598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f30146220 0x7f6f300cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.476+0000 7f6f2ea00640 1 -- 172.21.15.45:0/3294193598 >> v1:172.21.15.45:6789/0 conn(0x7f6f300cd220 legacy=0x7f6f3006df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:24.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.476+0000 7f6f2ea00640 1 -- 172.21.15.45:0/3294193598 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f300cde00 con 0x7f6f30072180 2024-04-16T16:16:24.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.477+0000 7f6f2ea00640 1 --2- 172.21.15.45:0/3294193598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f30072180 0x7f6f300725c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f6f18009870 tx=0x7f6f1802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162bc4e84e6a3d1a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:24.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.477+0000 7f6f2e000640 1 -- 172.21.15.45:0/3294193598 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18034070 con 0x7f6f30072180 2024-04-16T16:16:24.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.477+0000 7f6f2e000640 1 -- 172.21.15.45:0/3294193598 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f18030840 con 0x7f6f30072180 2024-04-16T16:16:24.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.477+0000 7f6f2e000640 1 -- 172.21.15.45:0/3294193598 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18030c10 con 0x7f6f30072180 2024-04-16T16:16:24.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.478+0000 7f6f36400640 1 -- 172.21.15.45:0/3294193598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f30072180 msgr2=0x7f6f300725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:24.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.478+0000 7f6f36400640 1 --2- 172.21.15.45:0/3294193598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f30072180 0x7f6f300725c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f6f18009870 tx=0x7f6f1802b980 comp rx=0 tx=0).stop 2024-04-16T16:16:24.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.478+0000 7f6f36400640 1 -- 172.21.15.45:0/3294193598 shutdown_connections 2024-04-16T16:16:24.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.478+0000 7f6f36400640 1 --2- 172.21.15.45:0/3294193598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f30072180 0x7f6f300725c0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.478+0000 7f6f36400640 1 --2- 172.21.15.45:0/3294193598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f30146220 0x7f6f300cf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.478+0000 7f6f36400640 1 -- 172.21.15.45:0/3294193598 >> 172.21.15.45:0/3294193598 conn(0x7f6f3006b930 msgr2=0x7f6f3006bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:24.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.478+0000 7f6f36400640 1 -- 172.21.15.45:0/3294193598 shutdown_connections 2024-04-16T16:16:24.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.478+0000 7f6f36400640 1 -- 172.21.15.45:0/3294193598 wait complete. 2024-04-16T16:16:24.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.480+0000 7f6f36400640 1 Processor -- start 2024-04-16T16:16:24.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.480+0000 7f6f36400640 1 -- start start 2024-04-16T16:16:24.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.481+0000 7f6f36400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f30072180 0x7f6f301f8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:24.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.481+0000 7f6f36400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f300cd220 0x7f6f30160140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:24.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.481+0000 7f6f36400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f30146220 0x7f6f30160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:24.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.481+0000 7f6f36400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6f300ce910 con 0x7f6f300cd220 2024-04-16T16:16:24.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.481+0000 7f6f36400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6f30072b00 con 0x7f6f30146220 2024-04-16T16:16:24.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.481+0000 7f6f36400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6f300cebf0 con 0x7f6f30072180 2024-04-16T16:16:24.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.481+0000 7f6f2ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f300cd220 0x7f6f30160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:24.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.481+0000 7f6f2ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f300cd220 0x7f6f30160140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46204/0 (socket says 172.21.15.45:46204) 2024-04-16T16:16:24.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.481+0000 7f6f2ea00640 1 -- 172.21.15.45:0/1504029926 learned_addr learned my addr 172.21.15.45:0/1504029926 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:24.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.482+0000 7f6f2f400640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f30072180 0x7f6f301f8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:24.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.482+0000 7f6f2ea00640 1 -- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f30072180 msgr2=0x7f6f301f8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:24.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.482+0000 7f6f2ea00640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f30072180 0x7f6f301f8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.482+0000 7f6f2ea00640 1 -- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f30146220 msgr2=0x7f6f30160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:24.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.482+0000 7f6f2fe00640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f30146220 0x7f6f30160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:24.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.482+0000 7f6f2ea00640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f30146220 0x7f6f30160680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.482+0000 7f6f2ea00640 1 -- 172.21.15.45:0/1504029926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f301f8a90 con 0x7f6f300cd220 2024-04-16T16:16:24.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.482+0000 7f6f2f400640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f30072180 0x7f6f301f8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:24.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.482+0000 7f6f2fe00640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f30146220 0x7f6f30160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:24.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.482+0000 7f6f2ea00640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f300cd220 0x7f6f30160140 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f6f18000c00 tx=0x7f6f18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:24.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.482+0000 7f6f27400640 1 -- 172.21.15.45:0/1504029926 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18048070 con 0x7f6f300cd220 2024-04-16T16:16:24.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.483+0000 7f6f27400640 1 -- 172.21.15.45:0/1504029926 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f18030640 con 0x7f6f300cd220 2024-04-16T16:16:24.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.483+0000 7f6f36400640 1 -- 172.21.15.45:0/1504029926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f301e6460 con 0x7f6f300cd220 2024-04-16T16:16:24.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.483+0000 7f6f27400640 1 -- 172.21.15.45:0/1504029926 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18034040 con 0x7f6f300cd220 2024-04-16T16:16:24.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.483+0000 7f6f36400640 1 -- 172.21.15.45:0/1504029926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f301e6a20 con 0x7f6f300cd220 2024-04-16T16:16:24.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.484+0000 7f6f36400640 1 -- 172.21.15.45:0/1504029926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f30075780 con 0x7f6f300cd220 2024-04-16T16:16:24.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.485+0000 7f6f27400640 1 -- 172.21.15.45:0/1504029926 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f6f1804b070 con 0x7f6f300cd220 2024-04-16T16:16:24.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.486+0000 7f6f27400640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6f0c06cb30 0x7f6f0c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:24.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.486+0000 7f6f27400640 1 -- 172.21.15.45:0/1504029926 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6729+0+0 (secure 0 0 0) 0x7f6f180ae520 con 0x7f6f300cd220 2024-04-16T16:16:24.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.486+0000 7f6f2f400640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6f0c06cb30 0x7f6f0c06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:24.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.488+0000 7f6f2f400640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6f0c06cb30 0x7f6f0c06eff0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6f20006fd0 tx=0x7f6f20008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:24.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.490+0000 7f6f27400640 1 -- 172.21.15.45:0/1504029926 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6f301f8a90 con 0x7f6f300cd220 2024-04-16T16:16:24.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.737+0000 7f6f36400640 1 -- 172.21.15.45:0/1504029926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "fs2"} v 0) -- 0x7f6f300b54e0 con 0x7f6f300cd220 2024-04-16T16:16:24.737 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-04-16T16:16:24.738+0000 7fe7a4800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T16:16:24.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.742+0000 7f6f27400640 1 -- 172.21.15.45:0/1504029926 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "fs2"}]=0 fs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) ==== 155+0+0 (secure 0 0 0) 0x7f6f1807c040 con 0x7f6f300cd220 2024-04-16T16:16:24.741 INFO:teuthology.orchestra.run.smithi045.stderr:fs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-16T16:16:24.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.745+0000 7f6f36400640 1 -- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6f0c06cb30 msgr2=0x7f6f0c06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:24.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.745+0000 7f6f36400640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6f0c06cb30 0x7f6f0c06eff0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6f20006fd0 tx=0x7f6f20008040 comp rx=0 tx=0).stop 2024-04-16T16:16:24.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.745+0000 7f6f36400640 1 -- 172.21.15.45:0/1504029926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f300cd220 msgr2=0x7f6f30160140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:24.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.745+0000 7f6f36400640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f300cd220 0x7f6f30160140 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f6f18000c00 tx=0x7f6f18004730 comp rx=0 tx=0).stop 2024-04-16T16:16:24.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.745+0000 7f6f36400640 1 -- 172.21.15.45:0/1504029926 shutdown_connections 2024-04-16T16:16:24.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.745+0000 7f6f36400640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f6f0c06cb30 0x7f6f0c06eff0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.745+0000 7f6f36400640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f30146220 0x7f6f30160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.745+0000 7f6f36400640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f300cd220 0x7f6f30160140 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.745+0000 7f6f36400640 1 --2- 172.21.15.45:0/1504029926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f30072180 0x7f6f301f8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.745+0000 7f6f36400640 1 -- 172.21.15.45:0/1504029926 >> 172.21.15.45:0/1504029926 conn(0x7f6f3006b930 msgr2=0x7f6f3015d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:24.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.746+0000 7f6f36400640 1 -- 172.21.15.45:0/1504029926 shutdown_connections 2024-04-16T16:16:24.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.746+0000 7f6f36400640 1 -- 172.21.15.45:0/1504029926 wait complete. 2024-04-16T16:16:24.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm fs2 --yes-i-really-mean-it 2024-04-16T16:16:24.786 INFO:tasks.ceph.mds.b.smithi103.stderr: -6586> 2024-04-16T16:15:46.242+0000 7f94b5000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:16:24.860 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-16T16:16:24.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.976+0000 7f26b3e00640 1 Processor -- start 2024-04-16T16:16:24.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.977+0000 7f26b3e00640 1 -- start start 2024-04-16T16:16:24.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.978+0000 7f26b3e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f26b40cf5c0 0x7f26b40d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:24.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.978+0000 7f26b3e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f26b415eef0 0x7f26b40cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:24.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.978+0000 7f26b3e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26b40d35a0 con 0x7f26b40d42d0 2024-04-16T16:16:24.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.978+0000 7f26b3e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f26b40cfee0 con 0x7f26b415eef0 2024-04-16T16:16:24.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.978+0000 7f26b3e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f26b40d2740 con 0x7f26b40cf5c0 2024-04-16T16:16:24.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.979+0000 7f26aa000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f26b415eef0 0x7f26b40cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:24.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.979+0000 7f26aa000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f26b415eef0 0x7f26b40cf9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46220/0 (socket says 172.21.15.45:46220) 2024-04-16T16:16:24.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.979+0000 7f26aa000640 1 -- 172.21.15.45:0/2403003459 learned_addr learned my addr 172.21.15.45:0/2403003459 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:24.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.979+0000 7f26b2a00640 1 --2- 172.21.15.45:0/2403003459 >> v2:172.21.15.103:3300/0 conn(0x7f26b40cf5c0 0x7f26b40d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:24.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.979+0000 7f26aa000640 1 -- 172.21.15.45:0/2403003459 >> v2:172.21.15.103:3300/0 conn(0x7f26b40cf5c0 msgr2=0x7f26b40d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:24.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.979+0000 7f26aa000640 1 --2- 172.21.15.45:0/2403003459 >> v2:172.21.15.103:3300/0 conn(0x7f26b40cf5c0 0x7f26b40d3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.979+0000 7f26aa000640 1 -- 172.21.15.45:0/2403003459 >> v1:172.21.15.45:6789/0 conn(0x7f26b40d42d0 legacy=0x7f26b40b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:24.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.979+0000 7f26aa000640 1 -- 172.21.15.45:0/2403003459 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26b40d2a20 con 0x7f26b415eef0 2024-04-16T16:16:24.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.981+0000 7f26aa000640 1 --2- 172.21.15.45:0/2403003459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26b415eef0 0x7f26b40cf9a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f26b40d2200 tx=0x7f269c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a6c9b2736cc79dd server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:24.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.981+0000 7f26b2000640 1 -- 172.21.15.45:0/2403003459 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269c034070 con 0x7f26b415eef0 2024-04-16T16:16:24.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.981+0000 7f26b2000640 1 -- 172.21.15.45:0/2403003459 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f269c030860 con 0x7f26b415eef0 2024-04-16T16:16:24.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.981+0000 7f26b2000640 1 -- 172.21.15.45:0/2403003459 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269c030c30 con 0x7f26b415eef0 2024-04-16T16:16:24.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.982+0000 7f26b3e00640 1 -- 172.21.15.45:0/2403003459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26b415eef0 msgr2=0x7f26b40cf9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:24.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.982+0000 7f26b3e00640 1 --2- 172.21.15.45:0/2403003459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26b415eef0 0x7f26b40cf9a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f26b40d2200 tx=0x7f269c02b9a0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.982+0000 7f26b3e00640 1 -- 172.21.15.45:0/2403003459 shutdown_connections 2024-04-16T16:16:24.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.982+0000 7f26b3e00640 1 --2- 172.21.15.45:0/2403003459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26b415eef0 0x7f26b40cf9a0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.982+0000 7f26b3e00640 1 --2- 172.21.15.45:0/2403003459 >> v2:172.21.15.103:3300/0 conn(0x7f26b40cf5c0 0x7f26b40d3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.982+0000 7f26b3e00640 1 -- 172.21.15.45:0/2403003459 >> 172.21.15.45:0/2403003459 conn(0x7f26b406d300 msgr2=0x7f26b40d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:24.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.982+0000 7f26b3e00640 1 -- 172.21.15.45:0/2403003459 shutdown_connections 2024-04-16T16:16:24.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.982+0000 7f26b3e00640 1 -- 172.21.15.45:0/2403003459 wait complete. 2024-04-16T16:16:24.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.984+0000 7f26b3e00640 1 Processor -- start 2024-04-16T16:16:24.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.984+0000 7f26b3e00640 1 -- start start 2024-04-16T16:16:24.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.985+0000 7f26b3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26b40cf5c0 0x7f26b41f82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:24.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.985+0000 7f26b3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26b40d42d0 0x7f26b41f8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:24.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.985+0000 7f26b3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26b415eef0 0x7f26b40c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:24.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.985+0000 7f26b3e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f26b40d32c0 con 0x7f26b40d42d0 2024-04-16T16:16:24.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.985+0000 7f26b3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f26b40d2fe0 con 0x7f26b415eef0 2024-04-16T16:16:24.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.985+0000 7f26b3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f26b4162b50 con 0x7f26b40cf5c0 2024-04-16T16:16:24.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.985+0000 7f26aa000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26b40d42d0 0x7f26b41f8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:24.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.985+0000 7f26aa000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26b40d42d0 0x7f26b41f8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46232/0 (socket says 172.21.15.45:46232) 2024-04-16T16:16:24.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.985+0000 7f26aa000640 1 -- 172.21.15.45:0/2877289625 learned_addr learned my addr 172.21.15.45:0/2877289625 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:24.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.985+0000 7f26b2a00640 1 --2- 172.21.15.45:0/2877289625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26b40cf5c0 0x7f26b41f82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:24.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.985+0000 7f26aa000640 1 -- 172.21.15.45:0/2877289625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26b40cf5c0 msgr2=0x7f26b41f82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:24.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.985+0000 7f26aa000640 1 --2- 172.21.15.45:0/2877289625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26b40cf5c0 0x7f26b41f82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.985+0000 7f26aa000640 1 -- 172.21.15.45:0/2877289625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26b415eef0 msgr2=0x7f26b40c87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:24.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.985+0000 7f26aa000640 1 --2- 172.21.15.45:0/2877289625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26b415eef0 0x7f26b40c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:24.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.985+0000 7f26aa000640 1 -- 172.21.15.45:0/2877289625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26b41f5e10 con 0x7f26b40d42d0 2024-04-16T16:16:24.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.986+0000 7f26aa000640 1 --2- 172.21.15.45:0/2877289625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26b40d42d0 0x7f26b41f8830 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f269c009960 tx=0x7f269c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:24.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.986+0000 7f26ab400640 1 -- 172.21.15.45:0/2877289625 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269c034040 con 0x7f26b40d42d0 2024-04-16T16:16:24.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.986+0000 7f26ab400640 1 -- 172.21.15.45:0/2877289625 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f269c039d90 con 0x7f26b40d42d0 2024-04-16T16:16:24.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.986+0000 7f26ab400640 1 -- 172.21.15.45:0/2877289625 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269c043630 con 0x7f26b40d42d0 2024-04-16T16:16:24.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.986+0000 7f26b3e00640 1 -- 172.21.15.45:0/2877289625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26b40c96c0 con 0x7f26b40d42d0 2024-04-16T16:16:24.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.986+0000 7f26b3e00640 1 -- 172.21.15.45:0/2877289625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26b40c9c80 con 0x7f26b40d42d0 2024-04-16T16:16:24.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.987+0000 7f26b3e00640 1 -- 172.21.15.45:0/2877289625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26b4075780 con 0x7f26b40d42d0 2024-04-16T16:16:24.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.988+0000 7f26ab400640 1 -- 172.21.15.45:0/2877289625 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f269c037070 con 0x7f26b40d42d0 2024-04-16T16:16:24.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.989+0000 7f26ab400640 1 --2- 172.21.15.45:0/2877289625 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f269006cb80 0x7f269006f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:24.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.989+0000 7f26ab400640 1 -- 172.21.15.45:0/2877289625 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6815+0+0 (secure 0 0 0) 0x7f269c0ae600 con 0x7f26b40d42d0 2024-04-16T16:16:24.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.990+0000 7f26b2a00640 1 --2- 172.21.15.45:0/2877289625 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f269006cb80 0x7f269006f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:24.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.991+0000 7f26b2a00640 1 --2- 172.21.15.45:0/2877289625 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f269006cb80 0x7f269006f040 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f26a4006fd0 tx=0x7f26a4008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:24.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:24.994+0000 7f26ab400640 1 -- 172.21.15.45:0/2877289625 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f269c03e070 con 0x7f26b40d42d0 2024-04-16T16:16:25.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:25.239+0000 7f26b3e00640 1 -- 172.21.15.45:0/2877289625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "fs2", "yes_i_really_mean_it": true} v 0) -- 0x7f26b40d3fd0 con 0x7f26b40d42d0 2024-04-16T16:16:25.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:25.746+0000 7f26ab400640 1 -- 172.21.15.45:0/2877289625 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "fs2", "yes_i_really_mean_it": true}]=0 v44) ==== 101+0+0 (secure 0 0 0) 0x7f269c07c300 con 0x7f26b40d42d0 2024-04-16T16:16:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:25.750+0000 7f26b3e00640 1 -- 172.21.15.45:0/2877289625 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f269006cb80 msgr2=0x7f269006f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:25.750+0000 7f26b3e00640 1 --2- 172.21.15.45:0/2877289625 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f269006cb80 0x7f269006f040 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f26a4006fd0 tx=0x7f26a4008040 comp rx=0 tx=0).stop 2024-04-16T16:16:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:25.750+0000 7f26b3e00640 1 -- 172.21.15.45:0/2877289625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26b40d42d0 msgr2=0x7f26b41f8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:25.750+0000 7f26b3e00640 1 --2- 172.21.15.45:0/2877289625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26b40d42d0 0x7f26b41f8830 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f269c009960 tx=0x7f269c0046c0 comp rx=0 tx=0).stop 2024-04-16T16:16:25.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:25.750+0000 7f26b3e00640 1 -- 172.21.15.45:0/2877289625 shutdown_connections 2024-04-16T16:16:25.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:25.750+0000 7f26b3e00640 1 --2- 172.21.15.45:0/2877289625 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f269006cb80 0x7f269006f040 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:25.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:25.750+0000 7f26b3e00640 1 --2- 172.21.15.45:0/2877289625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26b415eef0 0x7f26b40c87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:25.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:25.750+0000 7f26b3e00640 1 --2- 172.21.15.45:0/2877289625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26b40d42d0 0x7f26b41f8830 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:25.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:25.750+0000 7f26b3e00640 1 --2- 172.21.15.45:0/2877289625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26b40cf5c0 0x7f26b41f82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:25.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:25.750+0000 7f26b3e00640 1 -- 172.21.15.45:0/2877289625 >> 172.21.15.45:0/2877289625 conn(0x7f26b406d300 msgr2=0x7f26b40cbdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:25.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:25.750+0000 7f26b3e00640 1 -- 172.21.15.45:0/2877289625 shutdown_connections 2024-04-16T16:16:25.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:25.750+0000 7f26b3e00640 1 -- 172.21.15.45:0/2877289625 wait complete. 2024-04-16T16:16:25.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm fs2_metadata fs2_metadata --yes-i-really-really-mean-it 2024-04-16T16:16:26.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.012+0000 7f5966a00640 1 Processor -- start 2024-04-16T16:16:26.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.012+0000 7f5966a00640 1 -- start start 2024-04-16T16:16:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.014+0000 7f5966a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f59600b4930 0x7f5960072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.014+0000 7f5966a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59600b5530 con 0x7f59600d10e0 2024-04-16T16:16:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.014+0000 7f5966a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59600b5810 con 0x7f596015c7c0 2024-04-16T16:16:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.014+0000 7f5966a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f59600b52e0 con 0x7f59600b4930 2024-04-16T16:16:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.014+0000 7f5957400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f59600d10e0 0x7f596006f160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41046/0 (socket says 172.21.15.45:41046) 2024-04-16T16:16:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.014+0000 7f5957400640 1 -- 172.21.15.45:0/3503200886 learned_addr learned my addr 172.21.15.45:0/3503200886 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.014+0000 7f5964a00640 1 --2- 172.21.15.45:0/3503200886 >> v2:172.21.15.45:3300/0 conn(0x7f59600b4930 0x7f5960072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.015+0000 7f5964a00640 1 -- 172.21.15.45:0/3503200886 >> v1:172.21.15.103:6789/0 conn(0x7f596015c7c0 legacy=0x7f59600d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.015+0000 7f5964a00640 1 -- 172.21.15.45:0/3503200886 >> v1:172.21.15.45:6789/0 conn(0x7f59600d10e0 legacy=0x7f596006f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.015+0000 7f5964a00640 1 -- 172.21.15.45:0/3503200886 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59600d04b0 con 0x7f59600b4930 2024-04-16T16:16:26.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.016+0000 7f5964a00640 1 --2- 172.21.15.45:0/3503200886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59600b4930 0x7f5960072e10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f59580097a0 tx=0x7f595802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa2b3750cdceea82 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:26.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.017+0000 7f595f400640 1 -- 172.21.15.45:0/3503200886 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5958038070 con 0x7f59600b4930 2024-04-16T16:16:26.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.017+0000 7f595f400640 1 -- 172.21.15.45:0/3503200886 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5958031700 con 0x7f59600b4930 2024-04-16T16:16:26.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.017+0000 7f595f400640 1 -- 172.21.15.45:0/3503200886 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5958031ad0 con 0x7f59600b4930 2024-04-16T16:16:26.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.017+0000 7f5966a00640 1 -- 172.21.15.45:0/3503200886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59600b4930 msgr2=0x7f5960072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:26.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.017+0000 7f5966a00640 1 --2- 172.21.15.45:0/3503200886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59600b4930 0x7f5960072e10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f59580097a0 tx=0x7f595802b7a0 comp rx=0 tx=0).stop 2024-04-16T16:16:26.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.017+0000 7f5966a00640 1 -- 172.21.15.45:0/3503200886 shutdown_connections 2024-04-16T16:16:26.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.017+0000 7f5966a00640 1 --2- 172.21.15.45:0/3503200886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59600b4930 0x7f5960072e10 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:26.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.017+0000 7f5966a00640 1 -- 172.21.15.45:0/3503200886 >> 172.21.15.45:0/3503200886 conn(0x7f59600b3970 msgr2=0x7f59600b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:26.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.018+0000 7f5966a00640 1 -- 172.21.15.45:0/3503200886 shutdown_connections 2024-04-16T16:16:26.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.018+0000 7f5966a00640 1 -- 172.21.15.45:0/3503200886 wait complete. 2024-04-16T16:16:26.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.019+0000 7f5966a00640 1 Processor -- start 2024-04-16T16:16:26.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.019+0000 7f5966a00640 1 -- start start 2024-04-16T16:16:26.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.020+0000 7f5966a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59600b4930 0x7f59601f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:26.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.020+0000 7f5966a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59600d10e0 0x7f596015b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:26.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.020+0000 7f5966a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f596015c7c0 0x7f596014c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:26.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.020+0000 7f5966a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f59600d0260 con 0x7f596015c7c0 2024-04-16T16:16:26.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.020+0000 7f5966a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f596006fac0 con 0x7f59600b4930 2024-04-16T16:16:26.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.020+0000 7f5966a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5960073350 con 0x7f59600d10e0 2024-04-16T16:16:26.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.021+0000 7f5964a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f596015c7c0 0x7f596014c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:26.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.021+0000 7f5964a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f596015c7c0 0x7f596014c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46260/0 (socket says 172.21.15.45:46260) 2024-04-16T16:16:26.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.021+0000 7f5964a00640 1 -- 172.21.15.45:0/1864654310 learned_addr learned my addr 172.21.15.45:0/1864654310 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:26.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.021+0000 7f595fe00640 1 --2- 172.21.15.45:0/1864654310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59600b4930 0x7f59601f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:26.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.021+0000 7f5964a00640 1 -- 172.21.15.45:0/1864654310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59600d10e0 msgr2=0x7f596015b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:26.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.021+0000 7f5957400640 1 --2- 172.21.15.45:0/1864654310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59600d10e0 0x7f596015b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:26.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.021+0000 7f5964a00640 1 --2- 172.21.15.45:0/1864654310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59600d10e0 0x7f596015b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:26.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.021+0000 7f5964a00640 1 -- 172.21.15.45:0/1864654310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59600b4930 msgr2=0x7f59601f2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:26.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.021+0000 7f5964a00640 1 --2- 172.21.15.45:0/1864654310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59600b4930 0x7f59601f2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:26.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.021+0000 7f5964a00640 1 -- 172.21.15.45:0/1864654310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59601f3340 con 0x7f596015c7c0 2024-04-16T16:16:26.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.021+0000 7f595fe00640 1 --2- 172.21.15.45:0/1864654310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59600b4930 0x7f59601f2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:26.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.022+0000 7f5964a00640 1 --2- 172.21.15.45:0/1864654310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f596015c7c0 0x7f596014c120 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f59580098c0 tx=0x7f59580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:26.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.022+0000 7f595cc00640 1 -- 172.21.15.45:0/1864654310 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5958038040 con 0x7f596015c7c0 2024-04-16T16:16:26.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.022+0000 7f595cc00640 1 -- 172.21.15.45:0/1864654310 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5958043070 con 0x7f596015c7c0 2024-04-16T16:16:26.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.022+0000 7f595cc00640 1 -- 172.21.15.45:0/1864654310 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5958033bf0 con 0x7f596015c7c0 2024-04-16T16:16:26.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.022+0000 7f5966a00640 1 -- 172.21.15.45:0/1864654310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59600a97c0 con 0x7f596015c7c0 2024-04-16T16:16:26.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.022+0000 7f5966a00640 1 -- 172.21.15.45:0/1864654310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59600ae8e0 con 0x7f596015c7c0 2024-04-16T16:16:26.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.023+0000 7f5966a00640 1 -- 172.21.15.45:0/1864654310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5960075780 con 0x7f596015c7c0 2024-04-16T16:16:26.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.024+0000 7f595cc00640 1 -- 172.21.15.45:0/1864654310 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f595803b070 con 0x7f596015c7c0 2024-04-16T16:16:26.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.025+0000 7f595cc00640 1 --2- 172.21.15.45:0/1864654310 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f593c06cb30 0x7f593c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:26.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.025+0000 7f595cc00640 1 -- 172.21.15.45:0/1864654310 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6815+0+0 (secure 0 0 0) 0x7f59580ada50 con 0x7f596015c7c0 2024-04-16T16:16:26.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.026+0000 7f595fe00640 1 --2- 172.21.15.45:0/1864654310 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f593c06cb30 0x7f593c06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:26.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.027+0000 7f595fe00640 1 --2- 172.21.15.45:0/1864654310 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f593c06cb30 0x7f593c06eff0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5950007e90 tx=0x7f59500073a0 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.030+0000 7f595cc00640 1 -- 172.21.15.45:0/1864654310 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f59601f3340 con 0x7f596015c7c0 2024-04-16T16:16:26.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.238+0000 7f5966a00640 1 -- 172.21.15.45:0/1864654310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "fs2_metadata", "pool2": "fs2_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5960072fb0 con 0x7f596015c7c0 2024-04-16T16:16:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.905+0000 7f595cc00640 1 -- 172.21.15.45:0/1864654310 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "fs2_metadata", "pool2": "fs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'fs2_metadata' removed v62) ==== 172+0+0 (secure 0 0 0) 0x7f5958030070 con 0x7f596015c7c0 2024-04-16T16:16:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'fs2_metadata' removed 2024-04-16T16:16:26.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.908+0000 7f5966a00640 1 -- 172.21.15.45:0/1864654310 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f593c06cb30 msgr2=0x7f593c06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.908+0000 7f5966a00640 1 --2- 172.21.15.45:0/1864654310 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f593c06cb30 0x7f593c06eff0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5950007e90 tx=0x7f59500073a0 comp rx=0 tx=0).stop 2024-04-16T16:16:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.908+0000 7f5966a00640 1 -- 172.21.15.45:0/1864654310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f596015c7c0 msgr2=0x7f596014c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.908+0000 7f5966a00640 1 --2- 172.21.15.45:0/1864654310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f596015c7c0 0x7f596014c120 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f59580098c0 tx=0x7f59580046c0 comp rx=0 tx=0).stop 2024-04-16T16:16:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.909+0000 7f5966a00640 1 -- 172.21.15.45:0/1864654310 shutdown_connections 2024-04-16T16:16:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.909+0000 7f5966a00640 1 --2- 172.21.15.45:0/1864654310 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f593c06cb30 0x7f593c06eff0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.909+0000 7f5966a00640 1 --2- 172.21.15.45:0/1864654310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f596015c7c0 0x7f596014c120 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.909+0000 7f5966a00640 1 --2- 172.21.15.45:0/1864654310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59600d10e0 0x7f596015b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.909+0000 7f5966a00640 1 --2- 172.21.15.45:0/1864654310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59600b4930 0x7f59601f2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.909+0000 7f5966a00640 1 -- 172.21.15.45:0/1864654310 >> 172.21.15.45:0/1864654310 conn(0x7f59600b3970 msgr2=0x7f59600d09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.909+0000 7f5966a00640 1 -- 172.21.15.45:0/1864654310 shutdown_connections 2024-04-16T16:16:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:26.910+0000 7f5966a00640 1 -- 172.21.15.45:0/1864654310 wait complete. 2024-04-16T16:16:26.932 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm fs2_data fs2_data --yes-i-really-really-mean-it 2024-04-16T16:16:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.256+0000 7f891ac00640 1 Processor -- start 2024-04-16T16:16:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.256+0000 7f891ac00640 1 -- start start 2024-04-16T16:16:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.258+0000 7f891ac00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f89141452e0 0x7f891406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.258+0000 7f891ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f891406e920 0x7f89140ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.258+0000 7f891ac00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8914071e60 con 0x7f89140ceaf0 2024-04-16T16:16:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.258+0000 7f891ac00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f8914072f00 con 0x7f89141452e0 2024-04-16T16:16:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.258+0000 7f891ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f891406ed60 con 0x7f891406e920 2024-04-16T16:16:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.258+0000 7f891a200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f89140ceaf0 0x7f8914160270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41050/0 (socket says 172.21.15.45:41050) 2024-04-16T16:16:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.258+0000 7f891a200640 1 -- 172.21.15.45:0/641462748 learned_addr learned my addr 172.21.15.45:0/641462748 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8919800640 1 --2- 172.21.15.45:0/641462748 >> v2:172.21.15.103:3300/0 conn(0x7f89141452e0 0x7f891406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8918e00640 1 --2- 172.21.15.45:0/641462748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f891406e920 0x7f89140ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 -- 172.21.15.45:0/641462748 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1418032101 0 0) 0x7f8914071e60 con 0x7f89140ceaf0 2024-04-16T16:16:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 -- 172.21.15.45:0/641462748 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88fc003520 con 0x7f89140ceaf0 2024-04-16T16:16:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 -- 172.21.15.45:0/641462748 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 941217228 0 0) 0x7f88fc003520 con 0x7f89140ceaf0 2024-04-16T16:16:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 -- 172.21.15.45:0/641462748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f891406e920 msgr2=0x7f89140ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 --2- 172.21.15.45:0/641462748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f891406e920 0x7f89140ce5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 -- 172.21.15.45:0/641462748 >> v2:172.21.15.103:3300/0 conn(0x7f89141452e0 msgr2=0x7f891406e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 --2- 172.21.15.45:0/641462748 >> v2:172.21.15.103:3300/0 conn(0x7f89141452e0 0x7f891406e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 -- 172.21.15.45:0/641462748 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8914072140 con 0x7f89140ceaf0 2024-04-16T16:16:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 -- 172.21.15.45:0/641462748 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f890c001a10 con 0x7f89140ceaf0 2024-04-16T16:16:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 -- 172.21.15.45:0/641462748 >> v1:172.21.15.45:6789/0 conn(0x7f89140ceaf0 legacy=0x7f8914160270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 --2- 172.21.15.45:0/641462748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc0039b0 0x7f88fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 --2- 172.21.15.45:0/641462748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f891406e920 0x7f88fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 --2- 172.21.15.45:0/641462748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88fc004d40 0x7f88fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 -- 172.21.15.45:0/641462748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8914072f00 con 0x7f88fc0039b0 2024-04-16T16:16:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 -- 172.21.15.45:0/641462748 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f891406ed60 con 0x7f88fc004d40 2024-04-16T16:16:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8913e00640 1 -- 172.21.15.45:0/641462748 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f88fc00ba80 con 0x7f891406e920 2024-04-16T16:16:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.259+0000 7f8919800640 1 --2- 172.21.15.45:0/641462748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc0039b0 0x7f88fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.260+0000 7f8919800640 1 -- 172.21.15.45:0/641462748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f891406e920 msgr2=0x7f88fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.260+0000 7f8919800640 1 --2- 172.21.15.45:0/641462748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f891406e920 0x7f88fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.260+0000 7f8919800640 1 -- 172.21.15.45:0/641462748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88fc004d40 msgr2=0x7f88fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.260+0000 7f8919800640 1 --2- 172.21.15.45:0/641462748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88fc004d40 0x7f88fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.260+0000 7f8919800640 1 -- 172.21.15.45:0/641462748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88fc003520 con 0x7f88fc0039b0 2024-04-16T16:16:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.261+0000 7f8919800640 1 --2- 172.21.15.45:0/641462748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc0039b0 0x7f88fc003dd0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8908009ac0 tx=0x7f88fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aedb8d9fe4f290cd server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.261+0000 7f8913e00640 1 -- 172.21.15.45:0/641462748 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8908039070 con 0x7f88fc0039b0 2024-04-16T16:16:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.261+0000 7f8913e00640 1 -- 172.21.15.45:0/641462748 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f890802c710 con 0x7f88fc0039b0 2024-04-16T16:16:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.261+0000 7f8913e00640 1 -- 172.21.15.45:0/641462748 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8908009690 con 0x7f88fc0039b0 2024-04-16T16:16:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.262+0000 7f891ac00640 1 -- 172.21.15.45:0/641462748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc0039b0 msgr2=0x7f88fc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.262+0000 7f891ac00640 1 --2- 172.21.15.45:0/641462748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc0039b0 0x7f88fc003dd0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8908009ac0 tx=0x7f88fc005830 comp rx=0 tx=0).stop 2024-04-16T16:16:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.262+0000 7f891a200640 1 -- 172.21.15.45:0/641462748 reap_dead start 2024-04-16T16:16:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.262+0000 7f891ac00640 1 -- 172.21.15.45:0/641462748 shutdown_connections 2024-04-16T16:16:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.262+0000 7f891ac00640 1 -- 172.21.15.45:0/641462748 >> 172.21.15.45:0/641462748 conn(0x7f89140b3bb0 msgr2=0x7f89140b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.262+0000 7f891ac00640 1 -- 172.21.15.45:0/641462748 shutdown_connections 2024-04-16T16:16:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.262+0000 7f891ac00640 1 -- 172.21.15.45:0/641462748 wait complete. 2024-04-16T16:16:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.264+0000 7f891ac00640 1 Processor -- start 2024-04-16T16:16:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.264+0000 7f891ac00640 1 -- start start 2024-04-16T16:16:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.265+0000 7f891ac00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc0039b0 0x7f891414dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.265+0000 7f891ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f891414fb40 0x7f8914141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.265+0000 7f891ac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8914142430 0x7f89140a9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.265+0000 7f891ac00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f89140729e0 con 0x7f88fc0039b0 2024-04-16T16:16:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.265+0000 7f891ac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8914072700 con 0x7f8914142430 2024-04-16T16:16:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.265+0000 7f891ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f89140cef60 con 0x7f891414fb40 2024-04-16T16:16:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.265+0000 7f8919800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc0039b0 0x7f891414dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.265+0000 7f8919800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc0039b0 0x7f891414dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46292/0 (socket says 172.21.15.45:46292) 2024-04-16T16:16:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.265+0000 7f8919800640 1 -- 172.21.15.45:0/1718366804 learned_addr learned my addr 172.21.15.45:0/1718366804 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.265+0000 7f8919800640 1 -- 172.21.15.45:0/1718366804 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f891414fb40 msgr2=0x7f8914141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.265+0000 7f8919800640 1 --2- 172.21.15.45:0/1718366804 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f891414fb40 0x7f8914141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.265+0000 7f8919800640 1 -- 172.21.15.45:0/1718366804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8914142430 msgr2=0x7f89140a9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.265+0000 7f8919800640 1 --2- 172.21.15.45:0/1718366804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8914142430 0x7f89140a9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.265+0000 7f8919800640 1 -- 172.21.15.45:0/1718366804 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89140aa490 con 0x7f88fc0039b0 2024-04-16T16:16:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.266+0000 7f8919800640 1 --2- 172.21.15.45:0/1718366804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc0039b0 0x7f891414dd20 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8908000c00 tx=0x7f8908004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:27.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.266+0000 7f8911600640 1 -- 172.21.15.45:0/1718366804 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8908039050 con 0x7f88fc0039b0 2024-04-16T16:16:27.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.266+0000 7f8911600640 1 -- 172.21.15.45:0/1718366804 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8908045070 con 0x7f88fc0039b0 2024-04-16T16:16:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.266+0000 7f8911600640 1 -- 172.21.15.45:0/1718366804 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89080044e0 con 0x7f88fc0039b0 2024-04-16T16:16:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.266+0000 7f891ac00640 1 -- 172.21.15.45:0/1718366804 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89140af900 con 0x7f88fc0039b0 2024-04-16T16:16:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.266+0000 7f891ac00640 1 -- 172.21.15.45:0/1718366804 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89140afec0 con 0x7f88fc0039b0 2024-04-16T16:16:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.268+0000 7f891ac00640 1 -- 172.21.15.45:0/1718366804 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8914142850 con 0x7f88fc0039b0 2024-04-16T16:16:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.269+0000 7f8911600640 1 -- 172.21.15.45:0/1718366804 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f8908003a50 con 0x7f88fc0039b0 2024-04-16T16:16:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.270+0000 7f8911600640 1 --2- 172.21.15.45:0/1718366804 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f88f406cb30 0x7f88f406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.270+0000 7f8911600640 1 -- 172.21.15.45:0/1718366804 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6383+0+0 (secure 0 0 0) 0x7f89080ade70 con 0x7f88fc0039b0 2024-04-16T16:16:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.271+0000 7f8918e00640 1 --2- 172.21.15.45:0/1718366804 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f88f406cb30 0x7f88f406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:27.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.272+0000 7f8918e00640 1 --2- 172.21.15.45:0/1718366804 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f88f406cb30 0x7f88f406eff0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8900006f00 tx=0x7f8900008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:27.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.275+0000 7f8911600640 1 -- 172.21.15.45:0/1718366804 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f890804e020 con 0x7f88fc0039b0 2024-04-16T16:16:27.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.483+0000 7f891ac00640 1 -- 172.21.15.45:0/1718366804 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "fs2_data", "pool2": "fs2_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8914075780 con 0x7f88fc0039b0 2024-04-16T16:16:27.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.917+0000 7f8911600640 1 -- 172.21.15.45:0/1718366804 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "fs2_data", "pool2": "fs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'fs2_data' removed v63) ==== 160+0+0 (secure 0 0 0) 0x7f890807bc00 con 0x7f88fc0039b0 2024-04-16T16:16:27.917 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'fs2_data' removed 2024-04-16T16:16:27.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.922+0000 7f891ac00640 1 -- 172.21.15.45:0/1718366804 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f88f406cb30 msgr2=0x7f88f406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:27.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.922+0000 7f891ac00640 1 --2- 172.21.15.45:0/1718366804 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f88f406cb30 0x7f88f406eff0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8900006f00 tx=0x7f8900008040 comp rx=0 tx=0).stop 2024-04-16T16:16:27.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.922+0000 7f891ac00640 1 -- 172.21.15.45:0/1718366804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc0039b0 msgr2=0x7f891414dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:27.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.922+0000 7f891ac00640 1 --2- 172.21.15.45:0/1718366804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc0039b0 0x7f891414dd20 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8908000c00 tx=0x7f8908004730 comp rx=0 tx=0).stop 2024-04-16T16:16:27.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.922+0000 7f891ac00640 1 -- 172.21.15.45:0/1718366804 shutdown_connections 2024-04-16T16:16:27.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.922+0000 7f891ac00640 1 --2- 172.21.15.45:0/1718366804 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f88f406cb30 0x7f88f406eff0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:27.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.922+0000 7f891ac00640 1 --2- 172.21.15.45:0/1718366804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8914142430 0x7f89140a9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:27.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.922+0000 7f891ac00640 1 --2- 172.21.15.45:0/1718366804 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f891414fb40 0x7f8914141ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:27.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.922+0000 7f891ac00640 1 --2- 172.21.15.45:0/1718366804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc0039b0 0x7f891414dd20 secure :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8908000c00 tx=0x7f8908004730 comp rx=0 tx=0).stop 2024-04-16T16:16:27.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.922+0000 7f891ac00640 1 -- 172.21.15.45:0/1718366804 >> 172.21.15.45:0/1718366804 conn(0x7f89140b3bb0 msgr2=0x7f8914140fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:27.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.924+0000 7f891ac00640 1 -- 172.21.15.45:0/1718366804 shutdown_connections 2024-04-16T16:16:27.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:27.924+0000 7f891ac00640 1 -- 172.21.15.45:0/1718366804 wait complete. 2024-04-16T16:16:27.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-16T16:16:28.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.176+0000 7f26a4a00640 1 Processor -- start 2024-04-16T16:16:28.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.176+0000 7f26a4a00640 1 -- start start 2024-04-16T16:16:28.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.178+0000 7f26a4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26a015eef0 0x7f26a00d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:28.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.178+0000 7f26a4a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f26a00d42d0 0x7f26a00b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:28.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.178+0000 7f26a4a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26a00d35a0 con 0x7f26a00cf5c0 2024-04-16T16:16:28.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.178+0000 7f26a4a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f26a00d2740 con 0x7f26a00d42d0 2024-04-16T16:16:28.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.178+0000 7f26a4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f26a00d2a20 con 0x7f26a015eef0 2024-04-16T16:16:28.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.178+0000 7f269fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f26a00d42d0 0x7f26a00b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:28.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.179+0000 7f269fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f26a00d42d0 0x7f26a00b79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46302/0 (socket says 172.21.15.45:46302) 2024-04-16T16:16:28.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.179+0000 7f269fe00640 1 -- 172.21.15.45:0/2743266204 learned_addr learned my addr 172.21.15.45:0/2743266204 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:28.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.179+0000 7f2696a00640 1 --2- 172.21.15.45:0/2743266204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26a015eef0 0x7f26a00d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:28.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.179+0000 7f269fe00640 1 -- 172.21.15.45:0/2743266204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26a015eef0 msgr2=0x7f26a00d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:28.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.179+0000 7f269fe00640 1 --2- 172.21.15.45:0/2743266204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26a015eef0 0x7f26a00d3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.179+0000 7f269fe00640 1 -- 172.21.15.45:0/2743266204 >> v1:172.21.15.103:6789/0 conn(0x7f26a00cf5c0 legacy=0x7f26a00cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:28.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.179+0000 7f269fe00640 1 -- 172.21.15.45:0/2743266204 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a00d2d00 con 0x7f26a00d42d0 2024-04-16T16:16:28.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.180+0000 7f269fe00640 1 --2- 172.21.15.45:0/2743266204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a00d42d0 0x7f26a00b79d0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f2698009870 tx=0x7f269802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee4acb7527e31a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:28.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.180+0000 7f269ea00640 1 -- 172.21.15.45:0/2743266204 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2698035020 con 0x7f26a00d42d0 2024-04-16T16:16:28.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.181+0000 7f269ea00640 1 -- 172.21.15.45:0/2743266204 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2698031640 con 0x7f26a00d42d0 2024-04-16T16:16:28.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.181+0000 7f269ea00640 1 -- 172.21.15.45:0/2743266204 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269802d040 con 0x7f26a00d42d0 2024-04-16T16:16:28.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.181+0000 7f26a4a00640 1 -- 172.21.15.45:0/2743266204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a00d42d0 msgr2=0x7f26a00b79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:28.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.181+0000 7f26a4a00640 1 --2- 172.21.15.45:0/2743266204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a00d42d0 0x7f26a00b79d0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f2698009870 tx=0x7f269802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.181+0000 7f26a4a00640 1 -- 172.21.15.45:0/2743266204 shutdown_connections 2024-04-16T16:16:28.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.181+0000 7f26a4a00640 1 --2- 172.21.15.45:0/2743266204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a00d42d0 0x7f26a00b79d0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.181+0000 7f26a4a00640 1 --2- 172.21.15.45:0/2743266204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26a015eef0 0x7f26a00d3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.181+0000 7f26a4a00640 1 -- 172.21.15.45:0/2743266204 >> 172.21.15.45:0/2743266204 conn(0x7f26a006d300 msgr2=0x7f26a00d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:28.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.181+0000 7f26a4a00640 1 -- 172.21.15.45:0/2743266204 shutdown_connections 2024-04-16T16:16:28.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.181+0000 7f26a4a00640 1 -- 172.21.15.45:0/2743266204 wait complete. 2024-04-16T16:16:28.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.183+0000 7f26a4a00640 1 Processor -- start 2024-04-16T16:16:28.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.183+0000 7f26a4a00640 1 -- start start 2024-04-16T16:16:28.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.184+0000 7f26a4a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a00cf5c0 0x7f26a00ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:28.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.184+0000 7f26a4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26a00d42d0 0x7f26a00cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:28.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.184+0000 7f26a4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26a015eef0 0x7f26a015a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:28.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.184+0000 7f26a4a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f26a00b7f10 con 0x7f26a00cf5c0 2024-04-16T16:16:28.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.184+0000 7f26a4a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f26a00d32c0 con 0x7f26a00d42d0 2024-04-16T16:16:28.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.184+0000 7f26a4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f26a00b8090 con 0x7f26a015eef0 2024-04-16T16:16:28.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.184+0000 7f269f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a00cf5c0 0x7f26a00ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:28.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.184+0000 7f269f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a00cf5c0 0x7f26a00ca4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46304/0 (socket says 172.21.15.45:46304) 2024-04-16T16:16:28.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.185+0000 7f269f400640 1 -- 172.21.15.45:0/3545341946 learned_addr learned my addr 172.21.15.45:0/3545341946 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:28.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.185+0000 7f269fe00640 1 --2- 172.21.15.45:0/3545341946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26a015eef0 0x7f26a015a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:28.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.185+0000 7f2696a00640 1 --2- 172.21.15.45:0/3545341946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26a00d42d0 0x7f26a00cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:28.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.185+0000 7f269f400640 1 -- 172.21.15.45:0/3545341946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26a015eef0 msgr2=0x7f26a015a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:28.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.185+0000 7f269f400640 1 --2- 172.21.15.45:0/3545341946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26a015eef0 0x7f26a015a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.185+0000 7f269f400640 1 -- 172.21.15.45:0/3545341946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26a00d42d0 msgr2=0x7f26a00cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:28.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.185+0000 7f269f400640 1 --2- 172.21.15.45:0/3545341946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26a00d42d0 0x7f26a00cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.185+0000 7f269f400640 1 -- 172.21.15.45:0/3545341946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a0161f80 con 0x7f26a00cf5c0 2024-04-16T16:16:28.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.185+0000 7f2696a00640 1 --2- 172.21.15.45:0/3545341946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26a00d42d0 0x7f26a00cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:28.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.185+0000 7f269f400640 1 --2- 172.21.15.45:0/3545341946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a00cf5c0 0x7f26a00ca4a0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f269000c7f0 tx=0x7f269000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:28.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.186+0000 7f2697e00640 1 -- 172.21.15.45:0/3545341946 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2690016020 con 0x7f26a00cf5c0 2024-04-16T16:16:28.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.186+0000 7f2697e00640 1 -- 172.21.15.45:0/3545341946 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f269000aa50 con 0x7f26a00cf5c0 2024-04-16T16:16:28.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.186+0000 7f2697e00640 1 -- 172.21.15.45:0/3545341946 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269000ae40 con 0x7f26a00cf5c0 2024-04-16T16:16:28.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.186+0000 7f26a4a00640 1 -- 172.21.15.45:0/3545341946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26a00c99c0 con 0x7f26a00cf5c0 2024-04-16T16:16:28.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.186+0000 7f26a4a00640 1 -- 172.21.15.45:0/3545341946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26a00cb260 con 0x7f26a00cf5c0 2024-04-16T16:16:28.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.187+0000 7f26a4a00640 1 -- 172.21.15.45:0/3545341946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26a00d0450 con 0x7f26a00cf5c0 2024-04-16T16:16:28.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.188+0000 7f2697e00640 1 -- 172.21.15.45:0/3545341946 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f2690019020 con 0x7f26a00cf5c0 2024-04-16T16:16:28.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.189+0000 7f2697e00640 1 --2- 172.21.15.45:0/3545341946 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f267c06cb30 0x7f267c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:28.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.189+0000 7f2697e00640 1 -- 172.21.15.45:0/3545341946 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5975+0+0 (secure 0 0 0) 0x7f269008c110 con 0x7f26a00cf5c0 2024-04-16T16:16:28.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.190+0000 7f2696a00640 1 --2- 172.21.15.45:0/3545341946 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f267c06cb30 0x7f267c06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:28.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.191+0000 7f2696a00640 1 --2- 172.21.15.45:0/3545341946 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f267c06cb30 0x7f267c06eff0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2688006fd0 tx=0x7f2688008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:28.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.194+0000 7f2697e00640 1 -- 172.21.15.45:0/3545341946 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f26a00d0450 con 0x7f26a00cf5c0 2024-04-16T16:16:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.396+0000 7f26a4a00640 1 -- 172.21.15.45:0/3545341946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f26a0075780 con 0x7f26a00cf5c0 2024-04-16T16:16:28.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.403+0000 7f2697e00640 1 -- 172.21.15.45:0/3545341946 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f2690020030 con 0x7f26a00cf5c0 2024-04-16T16:16:28.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.406+0000 7f26a4a00640 1 -- 172.21.15.45:0/3545341946 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f267c06cb30 msgr2=0x7f267c06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:28.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.406+0000 7f26a4a00640 1 --2- 172.21.15.45:0/3545341946 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f267c06cb30 0x7f267c06eff0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2688006fd0 tx=0x7f2688008040 comp rx=0 tx=0).stop 2024-04-16T16:16:28.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.406+0000 7f26a4a00640 1 -- 172.21.15.45:0/3545341946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a00cf5c0 msgr2=0x7f26a00ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:28.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.406+0000 7f26a4a00640 1 --2- 172.21.15.45:0/3545341946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a00cf5c0 0x7f26a00ca4a0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f269000c7f0 tx=0x7f269000ccb0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.406+0000 7f26a4a00640 1 -- 172.21.15.45:0/3545341946 shutdown_connections 2024-04-16T16:16:28.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.406+0000 7f26a4a00640 1 --2- 172.21.15.45:0/3545341946 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f267c06cb30 0x7f267c06eff0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.406+0000 7f26a4a00640 1 --2- 172.21.15.45:0/3545341946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26a015eef0 0x7f26a015a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.406+0000 7f26a4a00640 1 --2- 172.21.15.45:0/3545341946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26a00d42d0 0x7f26a00cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.406+0000 7f26a4a00640 1 --2- 172.21.15.45:0/3545341946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a00cf5c0 0x7f26a00ca4a0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.406+0000 7f26a4a00640 1 -- 172.21.15.45:0/3545341946 >> 172.21.15.45:0/3545341946 conn(0x7f26a006d300 msgr2=0x7f26a00d50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:28.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.407+0000 7f26a4a00640 1 -- 172.21.15.45:0/3545341946 shutdown_connections 2024-04-16T16:16:28.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.407+0000 7f26a4a00640 1 -- 172.21.15.45:0/3545341946 wait complete. 2024-04-16T16:16:28.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_client_metrics_and_metadata' 2024-04-16T16:16:28.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.649+0000 7f68d2200640 1 Processor -- start 2024-04-16T16:16:28.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.649+0000 7f68d2200640 1 -- start start 2024-04-16T16:16:28.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.651+0000 7f68d2200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f68cc1462e0 0x7f68cc072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:28.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.651+0000 7f68d2200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68cc0cf6d0 con 0x7f68cc15ea70 2024-04-16T16:16:28.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.651+0000 7f68d2200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68cc1466c0 con 0x7f68cc0728c0 2024-04-16T16:16:28.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.651+0000 7f68d2200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f68cc0cf3f0 con 0x7f68cc1462e0 2024-04-16T16:16:28.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.651+0000 7f68cbe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f68cc15ea70 0x7f68cc166490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41064/0 (socket says 172.21.15.45:41064) 2024-04-16T16:16:28.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.651+0000 7f68cbe00640 1 -- 172.21.15.45:0/3363133544 learned_addr learned my addr 172.21.15.45:0/3363133544 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:28.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68cb400640 1 --2- 172.21.15.45:0/3363133544 >> v2:172.21.15.103:3300/0 conn(0x7f68cc1462e0 0x7f68cc072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:28.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68ca000640 1 -- 172.21.15.45:0/3363133544 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3371115809 0 0) 0x7f68cc0cf6d0 con 0x7f68cc15ea70 2024-04-16T16:16:28.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68ca000640 1 -- 172.21.15.45:0/3363133544 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68b0003520 con 0x7f68cc15ea70 2024-04-16T16:16:28.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68ca000640 1 -- 172.21.15.45:0/3363133544 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2926534473 0 0) 0x7f68b0003520 con 0x7f68cc15ea70 2024-04-16T16:16:28.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68ca000640 1 -- 172.21.15.45:0/3363133544 >> v2:172.21.15.103:3300/0 conn(0x7f68cc1462e0 msgr2=0x7f68cc072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:28.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68ca000640 1 --2- 172.21.15.45:0/3363133544 >> v2:172.21.15.103:3300/0 conn(0x7f68cc1462e0 0x7f68cc072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68ca000640 1 -- 172.21.15.45:0/3363133544 >> v1:172.21.15.103:6789/0 conn(0x7f68cc0728c0 legacy=0x7f68cc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:28.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68ca000640 1 -- 172.21.15.45:0/3363133544 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68cc15cf80 con 0x7f68cc15ea70 2024-04-16T16:16:28.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68ca000640 1 -- 172.21.15.45:0/3363133544 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f68c40019f0 con 0x7f68cc15ea70 2024-04-16T16:16:28.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68ca000640 1 -- 172.21.15.45:0/3363133544 >> v1:172.21.15.45:6789/0 conn(0x7f68cc15ea70 legacy=0x7f68cc166490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:28.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68ca000640 1 --2- 172.21.15.45:0/3363133544 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b0003a00 0x7f68b0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:28.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68ca000640 1 --2- 172.21.15.45:0/3363133544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68b0004800 0x7f68b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:28.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68ca000640 1 --2- 172.21.15.45:0/3363133544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b0005210 0x7f68b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:28.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68ca000640 1 -- 172.21.15.45:0/3363133544 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc0cf3f0 con 0x7f68b0005210 2024-04-16T16:16:28.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68ca000640 1 -- 172.21.15.45:0/3363133544 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f68b000c160 con 0x7f68b0004800 2024-04-16T16:16:28.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.652+0000 7f68ca000640 1 -- 172.21.15.45:0/3363133544 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f68b000c360 con 0x7f68b0003a00 2024-04-16T16:16:28.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.653+0000 7f68cbe00640 1 --2- 172.21.15.45:0/3363133544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b0005210 0x7f68b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:28.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.653+0000 7f68cb400640 1 --2- 172.21.15.45:0/3363133544 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b0003a00 0x7f68b0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:28.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.653+0000 7f68caa00640 1 --2- 172.21.15.45:0/3363133544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68b0004800 0x7f68b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:28.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.653+0000 7f68cbe00640 1 -- 172.21.15.45:0/3363133544 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b0003a00 msgr2=0x7f68b0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:28.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.653+0000 7f68cbe00640 1 --2- 172.21.15.45:0/3363133544 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68b0003a00 0x7f68b0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.653+0000 7f68cbe00640 1 -- 172.21.15.45:0/3363133544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68b0004800 msgr2=0x7f68b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:28.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.653+0000 7f68cbe00640 1 --2- 172.21.15.45:0/3363133544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68b0004800 0x7f68b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.653+0000 7f68cbe00640 1 -- 172.21.15.45:0/3363133544 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68b0003520 con 0x7f68b0005210 2024-04-16T16:16:28.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.653+0000 7f68caa00640 1 --2- 172.21.15.45:0/3363133544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68b0004800 0x7f68b0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:28.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.654+0000 7f68cbe00640 1 -- 172.21.15.45:0/3363133544 reap_dead start 2024-04-16T16:16:28.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.654+0000 7f68cbe00640 1 --2- 172.21.15.45:0/3363133544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b0005210 0x7f68b000b9c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f68c400a740 tx=0x7f68b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399dd9386dc9723b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:28.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.655+0000 7f68ca000640 1 -- 172.21.15.45:0/3363133544 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c400a390 con 0x7f68b0005210 2024-04-16T16:16:28.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.655+0000 7f68ca000640 1 -- 172.21.15.45:0/3363133544 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68c4034650 con 0x7f68b0005210 2024-04-16T16:16:28.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.655+0000 7f68ca000640 1 -- 172.21.15.45:0/3363133544 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c4034a40 con 0x7f68b0005210 2024-04-16T16:16:28.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.655+0000 7f68d2200640 1 -- 172.21.15.45:0/3363133544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b0005210 msgr2=0x7f68b000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:28.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.655+0000 7f68d2200640 1 --2- 172.21.15.45:0/3363133544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b0005210 0x7f68b000b9c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f68c400a740 tx=0x7f68b0005830 comp rx=0 tx=0).stop 2024-04-16T16:16:28.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.655+0000 7f68d2200640 1 -- 172.21.15.45:0/3363133544 shutdown_connections 2024-04-16T16:16:28.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.655+0000 7f68d2200640 1 --2- 172.21.15.45:0/3363133544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b0005210 0x7f68b000b9c0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.655+0000 7f68d2200640 1 -- 172.21.15.45:0/3363133544 >> 172.21.15.45:0/3363133544 conn(0x7f68cc0b2750 msgr2=0x7f68cc0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:28.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.655+0000 7f68d2200640 1 -- 172.21.15.45:0/3363133544 shutdown_connections 2024-04-16T16:16:28.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.655+0000 7f68d2200640 1 -- 172.21.15.45:0/3363133544 wait complete. 2024-04-16T16:16:28.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.657+0000 7f68d2200640 1 Processor -- start 2024-04-16T16:16:28.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.657+0000 7f68d2200640 1 -- start start 2024-04-16T16:16:28.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.658+0000 7f68d2200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b0005210 0x7f68cc139bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:28.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.658+0000 7f68d2200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68cc13f770 0x7f68cc13a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:28.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.658+0000 7f68d2200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68cc14d090 0x7f68cc14d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:28.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.658+0000 7f68d2200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc072380 con 0x7f68b0005210 2024-04-16T16:16:28.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.658+0000 7f68d2200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc15d540 con 0x7f68cc14d090 2024-04-16T16:16:28.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.658+0000 7f68d2200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f68cc072500 con 0x7f68cc13f770 2024-04-16T16:16:28.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.658+0000 7f68cb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b0005210 0x7f68cc139bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:28.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.659+0000 7f68cb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b0005210 0x7f68cc139bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46324/0 (socket says 172.21.15.45:46324) 2024-04-16T16:16:28.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.659+0000 7f68cb400640 1 -- 172.21.15.45:0/1739046651 learned_addr learned my addr 172.21.15.45:0/1739046651 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:28.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.659+0000 7f68caa00640 1 --2- 172.21.15.45:0/1739046651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68cc13f770 0x7f68cc13a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.659+0000 7f68cbe00640 1 --2- 172.21.15.45:0/1739046651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68cc14d090 0x7f68cc14d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.659+0000 7f68cb400640 1 -- 172.21.15.45:0/1739046651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68cc13f770 msgr2=0x7f68cc13a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.659+0000 7f68cb400640 1 --2- 172.21.15.45:0/1739046651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68cc13f770 0x7f68cc13a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.659+0000 7f68cb400640 1 -- 172.21.15.45:0/1739046651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68cc14d090 msgr2=0x7f68cc14d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.659+0000 7f68cb400640 1 --2- 172.21.15.45:0/1739046651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68cc14d090 0x7f68cc14d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.659+0000 7f68cb400640 1 -- 172.21.15.45:0/1739046651 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68cc13fc00 con 0x7f68b0005210 2024-04-16T16:16:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.659+0000 7f68cb400640 1 --2- 172.21.15.45:0/1739046651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b0005210 0x7f68cc139bf0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f68bc011570 tx=0x7f68bc011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.660+0000 7f68c3400640 1 -- 172.21.15.45:0/1739046651 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68bc015930 con 0x7f68b0005210 2024-04-16T16:16:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.660+0000 7f68c3400640 1 -- 172.21.15.45:0/1739046651 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68bc015ad0 con 0x7f68b0005210 2024-04-16T16:16:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.660+0000 7f68d2200640 1 -- 172.21.15.45:0/1739046651 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68cc13fee0 con 0x7f68b0005210 2024-04-16T16:16:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.660+0000 7f68c3400640 1 -- 172.21.15.45:0/1739046651 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68bc00f460 con 0x7f68b0005210 2024-04-16T16:16:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.660+0000 7f68d2200640 1 -- 172.21.15.45:0/1739046651 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68cc1404a0 con 0x7f68b0005210 2024-04-16T16:16:28.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.661+0000 7f68d2200640 1 -- 172.21.15.45:0/1739046651 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68cc075780 con 0x7f68b0005210 2024-04-16T16:16:28.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.662+0000 7f68c3400640 1 -- 172.21.15.45:0/1739046651 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f68bc00f600 con 0x7f68b0005210 2024-04-16T16:16:28.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.663+0000 7f68c3400640 1 --2- 172.21.15.45:0/1739046651 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f68ac06cb60 0x7f68ac06f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:28.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.663+0000 7f68c3400640 1 -- 172.21.15.45:0/1739046651 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5975+0+0 (secure 0 0 0) 0x7f68bc08ca00 con 0x7f68b0005210 2024-04-16T16:16:28.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.664+0000 7f68caa00640 1 --2- 172.21.15.45:0/1739046651 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f68ac06cb60 0x7f68ac06f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:28.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.665+0000 7f68caa00640 1 --2- 172.21.15.45:0/1739046651 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f68ac06cb60 0x7f68ac06f020 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f68cc15dea0 tx=0x7f68b8009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:28.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.668+0000 7f68c3400640 1 -- 172.21.15.45:0/1739046651 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f68bc019030 con 0x7f68b0005210 2024-04-16T16:16:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:28.969+0000 7f68d2200640 1 -- 172.21.15.45:0/1739046651 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_client_metrics_and_metadata"]} v 0) -- 0x7f68cc144af0 con 0x7f68b0005210 2024-04-16T16:16:29.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:29.923+0000 7f68c3400640 1 -- 172.21.15.45:0/1739046651 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_client_metrics_and_metadata"]}]=0 v191) ==== 156+0+0 (secure 0 0 0) 0x7f68bc05a920 con 0x7f68b0005210 2024-04-16T16:16:29.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:29.928+0000 7f68d2200640 1 -- 172.21.15.45:0/1739046651 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f68ac06cb60 msgr2=0x7f68ac06f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:29.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:29.928+0000 7f68d2200640 1 --2- 172.21.15.45:0/1739046651 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f68ac06cb60 0x7f68ac06f020 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f68cc15dea0 tx=0x7f68b8009040 comp rx=0 tx=0).stop 2024-04-16T16:16:29.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:29.928+0000 7f68d2200640 1 -- 172.21.15.45:0/1739046651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b0005210 msgr2=0x7f68cc139bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:29.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:29.928+0000 7f68d2200640 1 --2- 172.21.15.45:0/1739046651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b0005210 0x7f68cc139bf0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f68bc011570 tx=0x7f68bc011a30 comp rx=0 tx=0).stop 2024-04-16T16:16:29.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:29.928+0000 7f68d2200640 1 -- 172.21.15.45:0/1739046651 shutdown_connections 2024-04-16T16:16:29.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:29.928+0000 7f68d2200640 1 --2- 172.21.15.45:0/1739046651 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f68ac06cb60 0x7f68ac06f020 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:29.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:29.928+0000 7f68d2200640 1 --2- 172.21.15.45:0/1739046651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68cc14d090 0x7f68cc14d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:29.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:29.928+0000 7f68d2200640 1 --2- 172.21.15.45:0/1739046651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68cc13f770 0x7f68cc13a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:29.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:29.928+0000 7f68d2200640 1 --2- 172.21.15.45:0/1739046651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68b0005210 0x7f68cc139bf0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:29.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:29.928+0000 7f68d2200640 1 -- 172.21.15.45:0/1739046651 >> 172.21.15.45:0/1739046651 conn(0x7f68cc0b2750 msgr2=0x7f68cc1455a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:29.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:29.928+0000 7f68d2200640 1 -- 172.21.15.45:0/1739046651 shutdown_connections 2024-04-16T16:16:29.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:29.928+0000 7f68d2200640 1 -- 172.21.15.45:0/1739046651 wait complete. 2024-04-16T16:16:29.954 INFO:tasks.cephfs_test_runner:test_client_metrics_and_metadata (tasks.cephfs.test_mds_metrics.TestMDSMetrics) ... ok 2024-04-16T16:16:29.955 INFO:tasks.cephfs_test_runner:Starting test: test_delayed_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2024-04-16T16:16:29.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:16:30.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.181+0000 7f2afca00640 1 Processor -- start 2024-04-16T16:16:30.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.181+0000 7f2afca00640 1 -- start start 2024-04-16T16:16:30.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.182+0000 7f2afca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2af80d4560 0x7f2af80d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:30.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.182+0000 7f2afca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2af80d18f0 0x7f2af80b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:30.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.182+0000 7f2afca00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2af80d34c0 con 0x7f2af815ec60 2024-04-16T16:16:30.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.182+0000 7f2afca00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2af80d37a0 con 0x7f2af80d18f0 2024-04-16T16:16:30.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.182+0000 7f2afca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2af80d3a80 con 0x7f2af80d4560 2024-04-16T16:16:30.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.182+0000 7f2af5e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2af815ec60 0x7f2af80d2a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41072/0 (socket says 172.21.15.45:41072) 2024-04-16T16:16:30.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.182+0000 7f2af5e00640 1 -- 172.21.15.45:0/1797677623 learned_addr learned my addr 172.21.15.45:0/1797677623 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.183+0000 7f2af6800640 1 --2- 172.21.15.45:0/1797677623 >> v2:172.21.15.103:3300/0 conn(0x7f2af80d18f0 0x7f2af80b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.183+0000 7f2af5400640 1 --2- 172.21.15.45:0/1797677623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2af80d4560 0x7f2af80d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.183+0000 7f2af4a00640 1 -- 172.21.15.45:0/1797677623 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4071758505 0 0) 0x7f2af80d34c0 con 0x7f2af815ec60 2024-04-16T16:16:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.183+0000 7f2af4a00640 1 -- 172.21.15.45:0/1797677623 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2adc003520 con 0x7f2af815ec60 2024-04-16T16:16:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.183+0000 7f2af4a00640 1 -- 172.21.15.45:0/1797677623 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2452036861 0 0) 0x7f2adc003520 con 0x7f2af815ec60 2024-04-16T16:16:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.183+0000 7f2af4a00640 1 -- 172.21.15.45:0/1797677623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2af80d4560 msgr2=0x7f2af80d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.183+0000 7f2af4a00640 1 --2- 172.21.15.45:0/1797677623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2af80d4560 0x7f2af80d13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.183+0000 7f2af4a00640 1 -- 172.21.15.45:0/1797677623 >> v2:172.21.15.103:3300/0 conn(0x7f2af80d18f0 msgr2=0x7f2af80b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.183+0000 7f2af4a00640 1 --2- 172.21.15.45:0/1797677623 >> v2:172.21.15.103:3300/0 conn(0x7f2af80d18f0 0x7f2af80b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.183+0000 7f2af4a00640 1 -- 172.21.15.45:0/1797677623 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2af80d3d60 con 0x7f2af815ec60 2024-04-16T16:16:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.183+0000 7f2af4a00640 1 -- 172.21.15.45:0/1797677623 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f2ae80019f0 con 0x7f2af815ec60 2024-04-16T16:16:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.184+0000 7f2af4a00640 1 -- 172.21.15.45:0/1797677623 >> v1:172.21.15.45:6789/0 conn(0x7f2af815ec60 legacy=0x7f2af80d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.184+0000 7f2af4a00640 1 --2- 172.21.15.45:0/1797677623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2adc0039b0 0x7f2adc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.184+0000 7f2af4a00640 1 --2- 172.21.15.45:0/1797677623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2adc004800 0x7f2adc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.184+0000 7f2af4a00640 1 --2- 172.21.15.45:0/1797677623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2af80d4560 0x7f2adc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.184+0000 7f2af4a00640 1 -- 172.21.15.45:0/1797677623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2af80d37a0 con 0x7f2adc004800 2024-04-16T16:16:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.184+0000 7f2af4a00640 1 -- 172.21.15.45:0/1797677623 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2af80d3a80 con 0x7f2adc0039b0 2024-04-16T16:16:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.184+0000 7f2af4a00640 1 -- 172.21.15.45:0/1797677623 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2adc00b9c0 con 0x7f2af80d4560 2024-04-16T16:16:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.184+0000 7f2af5400640 1 --2- 172.21.15.45:0/1797677623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2adc004800 0x7f2adc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.184+0000 7f2af5e00640 1 --2- 172.21.15.45:0/1797677623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2adc0039b0 0x7f2adc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.184+0000 7f2af5400640 1 -- 172.21.15.45:0/1797677623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2af80d4560 msgr2=0x7f2adc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.184+0000 7f2af5400640 1 --2- 172.21.15.45:0/1797677623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2af80d4560 0x7f2adc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.184+0000 7f2af5400640 1 -- 172.21.15.45:0/1797677623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2adc0039b0 msgr2=0x7f2adc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.184+0000 7f2af5400640 1 --2- 172.21.15.45:0/1797677623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2adc0039b0 0x7f2adc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.184+0000 7f2af5400640 1 -- 172.21.15.45:0/1797677623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2adc003520 con 0x7f2adc004800 2024-04-16T16:16:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.184+0000 7f2af5e00640 1 --2- 172.21.15.45:0/1797677623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2adc0039b0 0x7f2adc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.186+0000 7f2af5400640 1 --2- 172.21.15.45:0/1797677623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2adc004800 0x7f2adc004c80 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0009a90 tx=0x7f2adc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3572d2a8b4fe19fa server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.186+0000 7f2af4a00640 1 -- 172.21.15.45:0/1797677623 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0039070 con 0x7f2adc004800 2024-04-16T16:16:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.186+0000 7f2af4a00640 1 -- 172.21.15.45:0/1797677623 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ae002c6b0 con 0x7f2adc004800 2024-04-16T16:16:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.186+0000 7f2af4a00640 1 -- 172.21.15.45:0/1797677623 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0009690 con 0x7f2adc004800 2024-04-16T16:16:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.187+0000 7f2afca00640 1 -- 172.21.15.45:0/1797677623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2adc004800 msgr2=0x7f2adc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.187+0000 7f2afca00640 1 --2- 172.21.15.45:0/1797677623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2adc004800 0x7f2adc004c80 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0009a90 tx=0x7f2adc005830 comp rx=0 tx=0).stop 2024-04-16T16:16:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.187+0000 7f2af6800640 1 -- 172.21.15.45:0/1797677623 reap_dead start 2024-04-16T16:16:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.187+0000 7f2afca00640 1 -- 172.21.15.45:0/1797677623 shutdown_connections 2024-04-16T16:16:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.187+0000 7f2afca00640 1 -- 172.21.15.45:0/1797677623 >> 172.21.15.45:0/1797677623 conn(0x7f2af8073690 msgr2=0x7f2af806cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.187+0000 7f2afca00640 1 -- 172.21.15.45:0/1797677623 shutdown_connections 2024-04-16T16:16:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.187+0000 7f2afca00640 1 -- 172.21.15.45:0/1797677623 wait complete. 2024-04-16T16:16:30.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.189+0000 7f2afca00640 1 Processor -- start 2024-04-16T16:16:30.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.189+0000 7f2afca00640 1 -- start start 2024-04-16T16:16:30.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.190+0000 7f2afca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2adc004800 0x7f2af80cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:30.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.190+0000 7f2afca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2af81f8dc0 0x7f2af8158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:30.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.190+0000 7f2afca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2af8158e50 0x7f2af8156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.190+0000 7f2afca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2af80d3110 con 0x7f2adc004800 2024-04-16T16:16:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.190+0000 7f2afca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2af80ba460 con 0x7f2af81f8dc0 2024-04-16T16:16:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.190+0000 7f2afca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2af80ccef0 con 0x7f2af8158e50 2024-04-16T16:16:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.190+0000 7f2af5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2adc004800 0x7f2af80cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.190+0000 7f2af5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2adc004800 0x7f2af80cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46338/0 (socket says 172.21.15.45:46338) 2024-04-16T16:16:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.190+0000 7f2af5e00640 1 -- 172.21.15.45:0/468275840 learned_addr learned my addr 172.21.15.45:0/468275840 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.191+0000 7f2af5400640 1 --2- 172.21.15.45:0/468275840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2af81f8dc0 0x7f2af8158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.191+0000 7f2af6800640 1 --2- 172.21.15.45:0/468275840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2af8158e50 0x7f2af8156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.191+0000 7f2af5e00640 1 -- 172.21.15.45:0/468275840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2af8158e50 msgr2=0x7f2af8156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.191+0000 7f2af5e00640 1 --2- 172.21.15.45:0/468275840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2af8158e50 0x7f2af8156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.191+0000 7f2af5e00640 1 -- 172.21.15.45:0/468275840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2af81f8dc0 msgr2=0x7f2af8158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.191+0000 7f2af5e00640 1 --2- 172.21.15.45:0/468275840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2af81f8dc0 0x7f2af8158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.191+0000 7f2af5e00640 1 -- 172.21.15.45:0/468275840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2af80cc2e0 con 0x7f2adc004800 2024-04-16T16:16:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.191+0000 7f2af6800640 1 --2- 172.21.15.45:0/468275840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2af8158e50 0x7f2af8156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.191+0000 7f2af5e00640 1 --2- 172.21.15.45:0/468275840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2adc004800 0x7f2af80cbab0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f2ae80021a0 tx=0x7f2ae800b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.191+0000 7f2aee000640 1 -- 172.21.15.45:0/468275840 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8019070 con 0x7f2adc004800 2024-04-16T16:16:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.191+0000 7f2aee000640 1 -- 172.21.15.45:0/468275840 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ae8004d60 con 0x7f2adc004800 2024-04-16T16:16:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.191+0000 7f2aee000640 1 -- 172.21.15.45:0/468275840 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae80072d0 con 0x7f2adc004800 2024-04-16T16:16:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.192+0000 7f2afca00640 1 -- 172.21.15.45:0/468275840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2af80cc5c0 con 0x7f2adc004800 2024-04-16T16:16:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.192+0000 7f2afca00640 1 -- 172.21.15.45:0/468275840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2af80ccb80 con 0x7f2adc004800 2024-04-16T16:16:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.193+0000 7f2afca00640 1 -- 172.21.15.45:0/468275840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2af8075780 con 0x7f2adc004800 2024-04-16T16:16:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.194+0000 7f2aee000640 1 -- 172.21.15.45:0/468275840 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f2ae8007470 con 0x7f2adc004800 2024-04-16T16:16:30.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.195+0000 7f2aee000640 1 --2- 172.21.15.45:0/468275840 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f2ad406cb30 0x7f2ad406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:30.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.195+0000 7f2aee000640 1 -- 172.21.15.45:0/468275840 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5975+0+0 (secure 0 0 0) 0x7f2ae808c2d0 con 0x7f2adc004800 2024-04-16T16:16:30.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.196+0000 7f2af5400640 1 --2- 172.21.15.45:0/468275840 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f2ad406cb30 0x7f2ad406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:30.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.197+0000 7f2af5400640 1 --2- 172.21.15.45:0/468275840 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f2ad406cb30 0x7f2ad406eff0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0036000 tx=0x7f2ae0030000 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:30.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.199+0000 7f2aee000640 1 -- 172.21.15.45:0/468275840 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2ae8059a90 con 0x7f2adc004800 2024-04-16T16:16:30.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.408+0000 7f2afca00640 1 -- 172.21.15.45:0/468275840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2af8159230 con 0x7f2adc004800 2024-04-16T16:16:30.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.410+0000 7f2aee000640 1 -- 172.21.15.45:0/468275840 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) ==== 74+0+14558 (secure 0 0 0) 0x7f2ae805dd90 con 0x7f2adc004800 2024-04-16T16:16:30.410 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:30.410 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":63,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:16:27.906333+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6837/1576112884":"2024-04-17T16:16:24.738649+0000","172.21.15.103:6834/2626302738":"2024-04-17T16:16:16.497315+0000","172.21.15.103:0/349066714":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/388389014":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/589483918":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3353543342":"2024-04-17T16:16:07.016480+0000","172.21.15.103:6832/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3310077212":"2024-04-17T16:16:07.016480+0000","172.21.15.45:6833/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:0/2318159422":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6832/368835060":"2024-04-17T16:15:41.256698+0000","172.21.15.103:6832/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6835/1576112884":"2024-04-17T16:16:24.738649+0000","172.21.15.103:6833/2711977587":"2024-04-17T16:15:01.083409+0000","172.21.15.45:6833/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.103:0/3554107853":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6836/2626302738":"2024-04-17T16:16:16.497315+0000","172.21.15.45:6835/2956262988":"2024-04-17T16:15:07.318380+0000","172.21.15.45:6833/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:6833/280591707":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/3010240888":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6834/88032513":"2024-04-17T16:15:41.221894+0000","172.21.15.45:6835/2276311244":"2024-04-17T16:15:20.691128+0000","172.21.15.103:0/644374842":"2024-04-17T16:16:07.016480+0000","172.21.15.103:0/2679792854":"2024-04-17T16:15:01.083409+0000","172.21.15.103:6835/88032513":"2024-04-17T16:15:41.221894+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:16:30.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.413+0000 7f2afca00640 1 -- 172.21.15.45:0/468275840 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f2ad406cb30 msgr2=0x7f2ad406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:30.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.413+0000 7f2afca00640 1 --2- 172.21.15.45:0/468275840 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f2ad406cb30 0x7f2ad406eff0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0036000 tx=0x7f2ae0030000 comp rx=0 tx=0).stop 2024-04-16T16:16:30.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.413+0000 7f2afca00640 1 -- 172.21.15.45:0/468275840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2adc004800 msgr2=0x7f2af80cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:30.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.413+0000 7f2afca00640 1 --2- 172.21.15.45:0/468275840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2adc004800 0x7f2af80cbab0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f2ae80021a0 tx=0x7f2ae800b560 comp rx=0 tx=0).stop 2024-04-16T16:16:30.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.413+0000 7f2afca00640 1 -- 172.21.15.45:0/468275840 shutdown_connections 2024-04-16T16:16:30.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.413+0000 7f2afca00640 1 --2- 172.21.15.45:0/468275840 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f2ad406cb30 0x7f2ad406eff0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.413+0000 7f2afca00640 1 --2- 172.21.15.45:0/468275840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2af8158e50 0x7f2af8156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.413+0000 7f2afca00640 1 --2- 172.21.15.45:0/468275840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2af81f8dc0 0x7f2af8158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.413+0000 7f2afca00640 1 --2- 172.21.15.45:0/468275840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2adc004800 0x7f2af80cbab0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.413+0000 7f2afca00640 1 -- 172.21.15.45:0/468275840 >> 172.21.15.45:0/468275840 conn(0x7f2af8073690 msgr2=0x7f2af80cf360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:30.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.414+0000 7f2afca00640 1 -- 172.21.15.45:0/468275840 shutdown_connections 2024-04-16T16:16:30.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.414+0000 7f2afca00640 1 -- 172.21.15.45:0/468275840 wait complete. 2024-04-16T16:16:30.434 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:16:30.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:16:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.656+0000 7fd11fe00640 1 Processor -- start 2024-04-16T16:16:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.657+0000 7fd11fe00640 1 -- start start 2024-04-16T16:16:30.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.658+0000 7fd11fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd12015ea70 0x7fd1200d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:30.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.658+0000 7fd11fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd1200d4370 0x7fd1200cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:30.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.658+0000 7fd11fe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd12015ee50 con 0x7fd1200cfa70 2024-04-16T16:16:30.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.658+0000 7fd11fe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd1200d1a10 con 0x7fd1200d4370 2024-04-16T16:16:30.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.658+0000 7fd11fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd1200b6480 con 0x7fd12015ea70 2024-04-16T16:16:30.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.659+0000 7fd11e000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd1200d4370 0x7fd1200cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:30.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.659+0000 7fd11e000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd1200d4370 0x7fd1200cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46346/0 (socket says 172.21.15.45:46346) 2024-04-16T16:16:30.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.659+0000 7fd11e000640 1 -- 172.21.15.45:0/1863500981 learned_addr learned my addr 172.21.15.45:0/1863500981 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:30.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.659+0000 7fd11ea00640 1 --2- 172.21.15.45:0/1863500981 >> v2:172.21.15.103:3300/0 conn(0x7fd12015ea70 0x7fd1200d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:30.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.659+0000 7fd11e000640 1 -- 172.21.15.45:0/1863500981 >> v2:172.21.15.103:3300/0 conn(0x7fd12015ea70 msgr2=0x7fd1200d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:30.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.659+0000 7fd11e000640 1 --2- 172.21.15.45:0/1863500981 >> v2:172.21.15.103:3300/0 conn(0x7fd12015ea70 0x7fd1200d11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.659+0000 7fd11e000640 1 -- 172.21.15.45:0/1863500981 >> v1:172.21.15.45:6789/0 conn(0x7fd1200cfa70 legacy=0x7fd1200d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:30.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.659+0000 7fd11e000640 1 -- 172.21.15.45:0/1863500981 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1200b6760 con 0x7fd1200d4370 2024-04-16T16:16:30.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.661+0000 7fd11e000640 1 --2- 172.21.15.45:0/1863500981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1200d4370 0x7fd1200cf530 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd1200d5890 tx=0x7fd10c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bde01dfa92288fd server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:30.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.661+0000 7fd11d600640 1 -- 172.21.15.45:0/1863500981 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10c034070 con 0x7fd1200d4370 2024-04-16T16:16:30.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.661+0000 7fd11d600640 1 -- 172.21.15.45:0/1863500981 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd10c030860 con 0x7fd1200d4370 2024-04-16T16:16:30.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.661+0000 7fd11d600640 1 -- 172.21.15.45:0/1863500981 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10c030c30 con 0x7fd1200d4370 2024-04-16T16:16:30.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.661+0000 7fd11fe00640 1 -- 172.21.15.45:0/1863500981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1200d4370 msgr2=0x7fd1200cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:30.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.661+0000 7fd11fe00640 1 --2- 172.21.15.45:0/1863500981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1200d4370 0x7fd1200cf530 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd1200d5890 tx=0x7fd10c02b9a0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.662+0000 7fd11fe00640 1 -- 172.21.15.45:0/1863500981 shutdown_connections 2024-04-16T16:16:30.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.662+0000 7fd11fe00640 1 --2- 172.21.15.45:0/1863500981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1200d4370 0x7fd1200cf530 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.662+0000 7fd11fe00640 1 --2- 172.21.15.45:0/1863500981 >> v2:172.21.15.103:3300/0 conn(0x7fd12015ea70 0x7fd1200d11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.662+0000 7fd11fe00640 1 -- 172.21.15.45:0/1863500981 >> 172.21.15.45:0/1863500981 conn(0x7fd12006d3b0 msgr2=0x7fd120073060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:30.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.662+0000 7fd11fe00640 1 -- 172.21.15.45:0/1863500981 shutdown_connections 2024-04-16T16:16:30.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.662+0000 7fd11fe00640 1 -- 172.21.15.45:0/1863500981 wait complete. 2024-04-16T16:16:30.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.663+0000 7fd11fe00640 1 Processor -- start 2024-04-16T16:16:30.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.664+0000 7fd11fe00640 1 -- start start 2024-04-16T16:16:30.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.665+0000 7fd11fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1200cfa70 0x7fd120157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:30.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.665+0000 7fd11fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1200d4370 0x7fd1201f8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:30.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.665+0000 7fd11fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd12015ea70 0x7fd12015f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:30.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.665+0000 7fd11fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd1200b6f60 con 0x7fd1200cfa70 2024-04-16T16:16:30.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.665+0000 7fd11fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1200d16e0 con 0x7fd12015ea70 2024-04-16T16:16:30.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.665+0000 7fd11fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd1201f9440 con 0x7fd1200d4370 2024-04-16T16:16:30.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.665+0000 7fd11ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1200cfa70 0x7fd120157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:30.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.665+0000 7fd11ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1200cfa70 0x7fd120157200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46360/0 (socket says 172.21.15.45:46360) 2024-04-16T16:16:30.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.665+0000 7fd11ea00640 1 -- 172.21.15.45:0/3171521605 learned_addr learned my addr 172.21.15.45:0/3171521605 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:30.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.665+0000 7fd11ea00640 1 -- 172.21.15.45:0/3171521605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1200d4370 msgr2=0x7fd1201f8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:30.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.665+0000 7fd11ea00640 1 --2- 172.21.15.45:0/3171521605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1200d4370 0x7fd1201f8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.665+0000 7fd11ea00640 1 -- 172.21.15.45:0/3171521605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd12015ea70 msgr2=0x7fd12015f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:30.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.665+0000 7fd11ea00640 1 --2- 172.21.15.45:0/3171521605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd12015ea70 0x7fd12015f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.665+0000 7fd11ea00640 1 -- 172.21.15.45:0/3171521605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1201f4f80 con 0x7fd1200cfa70 2024-04-16T16:16:30.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.666+0000 7fd11ea00640 1 --2- 172.21.15.45:0/3171521605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1200cfa70 0x7fd120157200 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd11000d5f0 tx=0x7fd11000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:30.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.666+0000 7fd116a00640 1 -- 172.21.15.45:0/3171521605 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1100125f0 con 0x7fd1200cfa70 2024-04-16T16:16:30.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.666+0000 7fd116a00640 1 -- 172.21.15.45:0/3171521605 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd110019070 con 0x7fd1200cfa70 2024-04-16T16:16:30.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.666+0000 7fd116a00640 1 -- 172.21.15.45:0/3171521605 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1100129d0 con 0x7fd1200cfa70 2024-04-16T16:16:30.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.666+0000 7fd11fe00640 1 -- 172.21.15.45:0/3171521605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1200c9dc0 con 0x7fd1200cfa70 2024-04-16T16:16:30.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.666+0000 7fd11fe00640 1 -- 172.21.15.45:0/3171521605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1200ca380 con 0x7fd1200cfa70 2024-04-16T16:16:30.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.668+0000 7fd11fe00640 1 -- 172.21.15.45:0/3171521605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd120075780 con 0x7fd1200cfa70 2024-04-16T16:16:30.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.669+0000 7fd116a00640 1 -- 172.21.15.45:0/3171521605 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fd110012b70 con 0x7fd1200cfa70 2024-04-16T16:16:30.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.670+0000 7fd116a00640 1 --2- 172.21.15.45:0/3171521605 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd10006cb30 0x7fd10006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:30.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.670+0000 7fd116a00640 1 -- 172.21.15.45:0/3171521605 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5975+0+0 (secure 0 0 0) 0x7fd11008c4e0 con 0x7fd1200cfa70 2024-04-16T16:16:30.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.671+0000 7fd11e000640 1 --2- 172.21.15.45:0/3171521605 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd10006cb30 0x7fd10006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:30.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.672+0000 7fd11e000640 1 --2- 172.21.15.45:0/3171521605 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd10006cb30 0x7fd10006eff0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd10c036000 tx=0x7fd10c030000 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:30.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.676+0000 7fd116a00640 1 -- 172.21.15.45:0/3171521605 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd11005a4a0 con 0x7fd1200cfa70 2024-04-16T16:16:30.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.884+0000 7fd11fe00640 1 -- 172.21.15.45:0/3171521605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd1200b86d0 con 0x7fd1200cfa70 2024-04-16T16:16:30.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.884+0000 7fd116a00640 1 -- 172.21.15.45:0/3171521605 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) ==== 93+0+10 (secure 0 0 0) 0x7fd110059da0 con 0x7fd1200cfa70 2024-04-16T16:16:30.883 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:16:30.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.887+0000 7fd11fe00640 1 -- 172.21.15.45:0/3171521605 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd10006cb30 msgr2=0x7fd10006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.887+0000 7fd11fe00640 1 --2- 172.21.15.45:0/3171521605 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd10006cb30 0x7fd10006eff0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd10c036000 tx=0x7fd10c030000 comp rx=0 tx=0).stop 2024-04-16T16:16:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.887+0000 7fd11fe00640 1 -- 172.21.15.45:0/3171521605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1200cfa70 msgr2=0x7fd120157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.887+0000 7fd11fe00640 1 --2- 172.21.15.45:0/3171521605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1200cfa70 0x7fd120157200 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd11000d5f0 tx=0x7fd11000dab0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.887+0000 7fd11fe00640 1 -- 172.21.15.45:0/3171521605 shutdown_connections 2024-04-16T16:16:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.887+0000 7fd11fe00640 1 --2- 172.21.15.45:0/3171521605 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd10006cb30 0x7fd10006eff0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.887+0000 7fd11fe00640 1 --2- 172.21.15.45:0/3171521605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd12015ea70 0x7fd12015f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.887+0000 7fd11fe00640 1 --2- 172.21.15.45:0/3171521605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1200d4370 0x7fd1201f8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.887+0000 7fd11fe00640 1 --2- 172.21.15.45:0/3171521605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1200cfa70 0x7fd120157200 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.887+0000 7fd11fe00640 1 -- 172.21.15.45:0/3171521605 >> 172.21.15.45:0/3171521605 conn(0x7fd12006d3b0 msgr2=0x7fd12015cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.887+0000 7fd11fe00640 1 -- 172.21.15.45:0/3171521605 shutdown_connections 2024-04-16T16:16:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:30.887+0000 7fd11fe00640 1 -- 172.21.15.45:0/3171521605 wait complete. 2024-04-16T16:16:30.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_delayed_metrics' 2024-04-16T16:16:31.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.125+0000 7f84dec00640 1 Processor -- start 2024-04-16T16:16:31.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.125+0000 7f84dec00640 1 -- start start 2024-04-16T16:16:31.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.127+0000 7f84dec00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f84d81593a0 0x7f84d815cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:31.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.127+0000 7f84dec00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f84d8071f40 0x7f84d80723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:31.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.127+0000 7f84dec00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84d80ce7f0 con 0x7f84d80c57e0 2024-04-16T16:16:31.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.127+0000 7f84dec00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f84d80728f0 con 0x7f84d8071f40 2024-04-16T16:16:31.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.127+0000 7f84dec00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f84d80cf400 con 0x7f84d81593a0 2024-04-16T16:16:31.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.127+0000 7f84de200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f84d8071f40 0x7f84d80723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:31.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.128+0000 7f84de200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f84d8071f40 0x7f84d80723b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46374/0 (socket says 172.21.15.45:46374) 2024-04-16T16:16:31.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.128+0000 7f84de200640 1 -- 172.21.15.45:0/3766216898 learned_addr learned my addr 172.21.15.45:0/3766216898 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:31.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.128+0000 7f84dd800640 1 --2- 172.21.15.45:0/3766216898 >> v2:172.21.15.103:3300/0 conn(0x7f84d81593a0 0x7f84d815cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:31.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.128+0000 7f84de200640 1 -- 172.21.15.45:0/3766216898 >> v2:172.21.15.103:3300/0 conn(0x7f84d81593a0 msgr2=0x7f84d815cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:31.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.128+0000 7f84de200640 1 --2- 172.21.15.45:0/3766216898 >> v2:172.21.15.103:3300/0 conn(0x7f84d81593a0 0x7f84d815cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:31.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.128+0000 7f84de200640 1 -- 172.21.15.45:0/3766216898 >> v1:172.21.15.103:6789/0 conn(0x7f84d80c57e0 legacy=0x7f84d80c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:31.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.128+0000 7f84de200640 1 -- 172.21.15.45:0/3766216898 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d80cead0 con 0x7f84d8071f40 2024-04-16T16:16:31.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.129+0000 7f84de200640 1 --2- 172.21.15.45:0/3766216898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d8071f40 0x7f84d80723b0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f84d80700c0 tx=0x7f84d002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ec761484c84b17 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:31.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.130+0000 7f84d7e00640 1 -- 172.21.15.45:0/3766216898 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d002c9f0 con 0x7f84d8071f40 2024-04-16T16:16:31.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.130+0000 7f84d7e00640 1 -- 172.21.15.45:0/3766216898 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84d002cb90 con 0x7f84d8071f40 2024-04-16T16:16:31.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.130+0000 7f84d7e00640 1 -- 172.21.15.45:0/3766216898 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d00343f0 con 0x7f84d8071f40 2024-04-16T16:16:31.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.130+0000 7f84dec00640 1 -- 172.21.15.45:0/3766216898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d8071f40 msgr2=0x7f84d80723b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:31.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.130+0000 7f84dec00640 1 --2- 172.21.15.45:0/3766216898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d8071f40 0x7f84d80723b0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f84d80700c0 tx=0x7f84d002be50 comp rx=0 tx=0).stop 2024-04-16T16:16:31.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.131+0000 7f84dec00640 1 -- 172.21.15.45:0/3766216898 shutdown_connections 2024-04-16T16:16:31.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.131+0000 7f84dec00640 1 --2- 172.21.15.45:0/3766216898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d8071f40 0x7f84d80723b0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:31.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.131+0000 7f84dec00640 1 --2- 172.21.15.45:0/3766216898 >> v2:172.21.15.103:3300/0 conn(0x7f84d81593a0 0x7f84d815cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:31.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.131+0000 7f84dec00640 1 -- 172.21.15.45:0/3766216898 >> 172.21.15.45:0/3766216898 conn(0x7f84d806b930 msgr2=0x7f84d806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:31.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.131+0000 7f84dec00640 1 -- 172.21.15.45:0/3766216898 shutdown_connections 2024-04-16T16:16:31.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.131+0000 7f84dec00640 1 -- 172.21.15.45:0/3766216898 wait complete. 2024-04-16T16:16:31.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.132+0000 7f84dec00640 1 Processor -- start 2024-04-16T16:16:31.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.133+0000 7f84dec00640 1 -- start start 2024-04-16T16:16:31.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84dec00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84d8071f40 0x7f84d814acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:31.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84dec00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d80c57e0 0x7f84d8150050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:31.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84dec00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84d81593a0 0x7f84d815ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:31.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84dec00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f84d80cf090 con 0x7f84d80c57e0 2024-04-16T16:16:31.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84dec00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f84d80c6460 con 0x7f84d8071f40 2024-04-16T16:16:31.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84dec00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f84d815d330 con 0x7f84d81593a0 2024-04-16T16:16:31.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84dce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d80c57e0 0x7f84d8150050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:31.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84dce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d80c57e0 0x7f84d8150050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46388/0 (socket says 172.21.15.45:46388) 2024-04-16T16:16:31.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84dce00640 1 -- 172.21.15.45:0/2964453732 learned_addr learned my addr 172.21.15.45:0/2964453732 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:31.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84de200640 1 --2- 172.21.15.45:0/2964453732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84d81593a0 0x7f84d815ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:31.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84dd800640 1 --2- 172.21.15.45:0/2964453732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84d8071f40 0x7f84d814acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:31.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84dce00640 1 -- 172.21.15.45:0/2964453732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84d81593a0 msgr2=0x7f84d815ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:31.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84dce00640 1 --2- 172.21.15.45:0/2964453732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84d81593a0 0x7f84d815ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:31.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84dce00640 1 -- 172.21.15.45:0/2964453732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84d8071f40 msgr2=0x7f84d814acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:31.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84dce00640 1 --2- 172.21.15.45:0/2964453732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84d8071f40 0x7f84d814acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:31.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84dce00640 1 -- 172.21.15.45:0/2964453732 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d8134eb0 con 0x7f84d80c57e0 2024-04-16T16:16:31.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.134+0000 7f84de200640 1 --2- 172.21.15.45:0/2964453732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84d81593a0 0x7f84d815ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:31.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.135+0000 7f84dce00640 1 --2- 172.21.15.45:0/2964453732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d80c57e0 0x7f84d8150050 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c3f0 tx=0x7f84c400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:31.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.135+0000 7f84d5600640 1 -- 172.21.15.45:0/2964453732 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c4007020 con 0x7f84d80c57e0 2024-04-16T16:16:31.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.135+0000 7f84d5600640 1 -- 172.21.15.45:0/2964453732 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84c40071c0 con 0x7f84d80c57e0 2024-04-16T16:16:31.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.135+0000 7f84d5600640 1 -- 172.21.15.45:0/2964453732 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c4007590 con 0x7f84d80c57e0 2024-04-16T16:16:31.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.135+0000 7f84dec00640 1 -- 172.21.15.45:0/2964453732 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d8135470 con 0x7f84d80c57e0 2024-04-16T16:16:31.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.135+0000 7f84dec00640 1 -- 172.21.15.45:0/2964453732 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d814b230 con 0x7f84d80c57e0 2024-04-16T16:16:31.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.136+0000 7f84dec00640 1 -- 172.21.15.45:0/2964453732 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84d8150bf0 con 0x7f84d80c57e0 2024-04-16T16:16:31.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.137+0000 7f84d5600640 1 -- 172.21.15.45:0/2964453732 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f84c4019020 con 0x7f84d80c57e0 2024-04-16T16:16:31.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.138+0000 7f84d5600640 1 --2- 172.21.15.45:0/2964453732 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f84b806cb30 0x7f84b806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:31.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.138+0000 7f84d5600640 1 -- 172.21.15.45:0/2964453732 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5975+0+0 (secure 0 0 0) 0x7f84c408b890 con 0x7f84d80c57e0 2024-04-16T16:16:31.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.139+0000 7f84dd800640 1 --2- 172.21.15.45:0/2964453732 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f84b806cb30 0x7f84b806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:31.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.140+0000 7f84dd800640 1 --2- 172.21.15.45:0/2964453732 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f84b806cb30 0x7f84b806eff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f84cc006fd0 tx=0x7f84cc008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:31.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.144+0000 7f84d5600640 1 -- 172.21.15.45:0/2964453732 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f84c40597f0 con 0x7f84d80c57e0 2024-04-16T16:16:31.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.423+0000 7f84dec00640 1 -- 172.21.15.45:0/2964453732 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_delayed_metrics"]} v 0) -- 0x7f84d8075780 con 0x7f84d80c57e0 2024-04-16T16:16:31.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.929+0000 7f84d5600640 1 -- 172.21.15.45:0/2964453732 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_delayed_metrics"]}]=0 v193) ==== 147+0+0 (secure 0 0 0) 0x7f84c40590f0 con 0x7f84d80c57e0 2024-04-16T16:16:31.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.933+0000 7f84dec00640 1 -- 172.21.15.45:0/2964453732 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f84b806cb30 msgr2=0x7f84b806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:31.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.933+0000 7f84dec00640 1 --2- 172.21.15.45:0/2964453732 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f84b806cb30 0x7f84b806eff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f84cc006fd0 tx=0x7f84cc008040 comp rx=0 tx=0).stop 2024-04-16T16:16:31.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.933+0000 7f84dec00640 1 -- 172.21.15.45:0/2964453732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d80c57e0 msgr2=0x7f84d8150050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:31.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.933+0000 7f84dec00640 1 --2- 172.21.15.45:0/2964453732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d80c57e0 0x7f84d8150050 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c3f0 tx=0x7f84c400c8b0 comp rx=0 tx=0).stop 2024-04-16T16:16:31.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.933+0000 7f84dec00640 1 -- 172.21.15.45:0/2964453732 shutdown_connections 2024-04-16T16:16:31.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.933+0000 7f84dec00640 1 --2- 172.21.15.45:0/2964453732 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f84b806cb30 0x7f84b806eff0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:31.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.933+0000 7f84dec00640 1 --2- 172.21.15.45:0/2964453732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84d81593a0 0x7f84d815ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:31.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.933+0000 7f84dec00640 1 --2- 172.21.15.45:0/2964453732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d80c57e0 0x7f84d8150050 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:31.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.933+0000 7f84dec00640 1 --2- 172.21.15.45:0/2964453732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84d8071f40 0x7f84d814acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:31.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.933+0000 7f84dec00640 1 -- 172.21.15.45:0/2964453732 >> 172.21.15.45:0/2964453732 conn(0x7f84d806b930 msgr2=0x7f84d80b3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:31.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.933+0000 7f84dec00640 1 -- 172.21.15.45:0/2964453732 shutdown_connections 2024-04-16T16:16:31.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:31.933+0000 7f84dec00640 1 -- 172.21.15.45:0/2964453732 wait complete. 2024-04-16T16:16:31.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-16T16:16:32.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.191+0000 7fd4e3200640 1 Processor -- start 2024-04-16T16:16:32.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.192+0000 7fd4e3200640 1 -- start start 2024-04-16T16:16:32.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.193+0000 7fd4e3200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd4e41461b0 0x7fd4e4160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:32.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.193+0000 7fd4e3200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd4e40b58f0 0x7fd4e40b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:32.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.193+0000 7fd4e3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4e40cf440 0x7fd4e40cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:32.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.193+0000 7fd4e3200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd4e406dfd0 con 0x7fd4e41461b0 2024-04-16T16:16:32.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.193+0000 7fd4e3200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd4e40b6290 con 0x7fd4e40b58f0 2024-04-16T16:16:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.193+0000 7fd4e3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd4e4146590 con 0x7fd4e40cf440 2024-04-16T16:16:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.193+0000 7fd4e1e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd4e41461b0 0x7fd4e4160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.193+0000 7fd4e1e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd4e41461b0 0x7fd4e4160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46402/0 (socket says 172.21.15.45:46402) 2024-04-16T16:16:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.193+0000 7fd4e1e00640 1 -- 172.21.15.45:0/2667652781 learned_addr learned my addr 172.21.15.45:0/2667652781 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.194+0000 7fd4e2800640 1 --2- 172.21.15.45:0/2667652781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4e40cf440 0x7fd4e40cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.194+0000 7fd4dbe00640 1 --2- 172.21.15.45:0/2667652781 >> v2:172.21.15.103:3300/0 conn(0x7fd4e40b58f0 0x7fd4e40b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.194+0000 7fd4e1e00640 1 -- 172.21.15.45:0/2667652781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4e40cf440 msgr2=0x7fd4e40cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.194+0000 7fd4e1e00640 1 --2- 172.21.15.45:0/2667652781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4e40cf440 0x7fd4e40cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.194+0000 7fd4e1e00640 1 -- 172.21.15.45:0/2667652781 >> v2:172.21.15.103:3300/0 conn(0x7fd4e40b58f0 msgr2=0x7fd4e40b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.194+0000 7fd4e1e00640 1 --2- 172.21.15.45:0/2667652781 >> v2:172.21.15.103:3300/0 conn(0x7fd4e40b58f0 0x7fd4e40b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.194+0000 7fd4e1e00640 1 -- 172.21.15.45:0/2667652781 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e40c5750 con 0x7fd4e41461b0 2024-04-16T16:16:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.194+0000 7fd4e2800640 1 --2- 172.21.15.45:0/2667652781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4e40cf440 0x7fd4e40cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.195+0000 7fd4e1e00640 1 --2- 172.21.15.45:0/2667652781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e41461b0 0x7fd4e4160cf0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40b68c0 tx=0x7fd4d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6e60a03f5c8a42 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.195+0000 7fd4e1400640 1 -- 172.21.15.45:0/2667652781 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4035020 con 0x7fd4e41461b0 2024-04-16T16:16:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.196+0000 7fd4e1400640 1 -- 172.21.15.45:0/2667652781 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4d4031870 con 0x7fd4e41461b0 2024-04-16T16:16:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.196+0000 7fd4e1400640 1 -- 172.21.15.45:0/2667652781 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4031c40 con 0x7fd4e41461b0 2024-04-16T16:16:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.196+0000 7fd4e3200640 1 -- 172.21.15.45:0/2667652781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e41461b0 msgr2=0x7fd4e4160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.196+0000 7fd4e3200640 1 --2- 172.21.15.45:0/2667652781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e41461b0 0x7fd4e4160cf0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40b68c0 tx=0x7fd4d402b910 comp rx=0 tx=0).stop 2024-04-16T16:16:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.196+0000 7fd4e3200640 1 -- 172.21.15.45:0/2667652781 shutdown_connections 2024-04-16T16:16:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.196+0000 7fd4e3200640 1 --2- 172.21.15.45:0/2667652781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4e40cf440 0x7fd4e40cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.196+0000 7fd4e3200640 1 --2- 172.21.15.45:0/2667652781 >> v2:172.21.15.103:3300/0 conn(0x7fd4e40b58f0 0x7fd4e40b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.196+0000 7fd4e3200640 1 --2- 172.21.15.45:0/2667652781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e41461b0 0x7fd4e4160cf0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.196+0000 7fd4e3200640 1 -- 172.21.15.45:0/2667652781 >> 172.21.15.45:0/2667652781 conn(0x7fd4e406b930 msgr2=0x7fd4e406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.196+0000 7fd4e3200640 1 -- 172.21.15.45:0/2667652781 shutdown_connections 2024-04-16T16:16:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.196+0000 7fd4e3200640 1 -- 172.21.15.45:0/2667652781 wait complete. 2024-04-16T16:16:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.198+0000 7fd4e3200640 1 Processor -- start 2024-04-16T16:16:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.198+0000 7fd4e3200640 1 -- start start 2024-04-16T16:16:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.199+0000 7fd4e3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4e40b58f0 0x7fd4e41454b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.199+0000 7fd4e3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4e40cf440 0x7fd4e4161ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.199+0000 7fd4e3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e41461b0 0x7fd4e4162530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.199+0000 7fd4e3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e40c59f0 con 0x7fd4e41461b0 2024-04-16T16:16:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.199+0000 7fd4e3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e4161230 con 0x7fd4e40cf440 2024-04-16T16:16:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.199+0000 7fd4e3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd4e40c5cd0 con 0x7fd4e40b58f0 2024-04-16T16:16:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.200+0000 7fd4e2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e41461b0 0x7fd4e4162530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.200+0000 7fd4e2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e41461b0 0x7fd4e4162530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46416/0 (socket says 172.21.15.45:46416) 2024-04-16T16:16:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.200+0000 7fd4e2800640 1 -- 172.21.15.45:0/669278404 learned_addr learned my addr 172.21.15.45:0/669278404 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.200+0000 7fd4e1e00640 1 --2- 172.21.15.45:0/669278404 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4e40b58f0 0x7fd4e41454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.200+0000 7fd4dbe00640 1 --2- 172.21.15.45:0/669278404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4e40cf440 0x7fd4e4161ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.200+0000 7fd4e2800640 1 -- 172.21.15.45:0/669278404 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4e40b58f0 msgr2=0x7fd4e41454b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.200+0000 7fd4e2800640 1 --2- 172.21.15.45:0/669278404 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4e40b58f0 0x7fd4e41454b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.200+0000 7fd4e2800640 1 -- 172.21.15.45:0/669278404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4e40cf440 msgr2=0x7fd4e4161ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.200+0000 7fd4e2800640 1 --2- 172.21.15.45:0/669278404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4e40cf440 0x7fd4e4161ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.200+0000 7fd4e2800640 1 -- 172.21.15.45:0/669278404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e40c9350 con 0x7fd4e41461b0 2024-04-16T16:16:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.200+0000 7fd4dbe00640 1 --2- 172.21.15.45:0/669278404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4e40cf440 0x7fd4e4161ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.200+0000 7fd4e2800640 1 --2- 172.21.15.45:0/669278404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e41461b0 0x7fd4e4162530 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00cef0 tx=0x7fd4dc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.201+0000 7fd4da000640 1 -- 172.21.15.45:0/669278404 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc01a070 con 0x7fd4e41461b0 2024-04-16T16:16:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.201+0000 7fd4da000640 1 -- 172.21.15.45:0/669278404 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4dc00d860 con 0x7fd4e41461b0 2024-04-16T16:16:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.201+0000 7fd4da000640 1 -- 172.21.15.45:0/669278404 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc00dc50 con 0x7fd4e41461b0 2024-04-16T16:16:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.201+0000 7fd4e3200640 1 -- 172.21.15.45:0/669278404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4e40c95f0 con 0x7fd4e41461b0 2024-04-16T16:16:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.201+0000 7fd4e3200640 1 -- 172.21.15.45:0/669278404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4e40c9bb0 con 0x7fd4e41461b0 2024-04-16T16:16:32.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.202+0000 7fd4e3200640 1 -- 172.21.15.45:0/669278404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4e4075780 con 0x7fd4e41461b0 2024-04-16T16:16:32.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.203+0000 7fd4da000640 1 -- 172.21.15.45:0/669278404 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fd4dc01d070 con 0x7fd4e41461b0 2024-04-16T16:16:32.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.204+0000 7fd4da000640 1 --2- 172.21.15.45:0/669278404 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd4c006cb30 0x7fd4c006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:32.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.204+0000 7fd4da000640 1 -- 172.21.15.45:0/669278404 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5975+0+0 (secure 0 0 0) 0x7fd4dc08c970 con 0x7fd4e41461b0 2024-04-16T16:16:32.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.205+0000 7fd4e1e00640 1 --2- 172.21.15.45:0/669278404 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd4c006cb30 0x7fd4c006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:32.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.206+0000 7fd4e1e00640 1 --2- 172.21.15.45:0/669278404 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd4c006cb30 0x7fd4c006eff0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4036000 tx=0x7fd4d403c040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:32.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.209+0000 7fd4da000640 1 -- 172.21.15.45:0/669278404 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd4dc05a9e0 con 0x7fd4e41461b0 2024-04-16T16:16:32.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.412+0000 7fd4e3200640 1 -- 172.21.15.45:0/669278404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fd4e4072500 con 0x7fd4e41461b0 2024-04-16T16:16:32.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.418+0000 7fd4da000640 1 -- 172.21.15.45:0/669278404 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7fd4dc05a2e0 con 0x7fd4e41461b0 2024-04-16T16:16:32.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.421+0000 7fd4e3200640 1 -- 172.21.15.45:0/669278404 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd4c006cb30 msgr2=0x7fd4c006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:32.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.421+0000 7fd4e3200640 1 --2- 172.21.15.45:0/669278404 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd4c006cb30 0x7fd4c006eff0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4036000 tx=0x7fd4d403c040 comp rx=0 tx=0).stop 2024-04-16T16:16:32.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.421+0000 7fd4e3200640 1 -- 172.21.15.45:0/669278404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e41461b0 msgr2=0x7fd4e4162530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:32.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.421+0000 7fd4e3200640 1 --2- 172.21.15.45:0/669278404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e41461b0 0x7fd4e4162530 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00cef0 tx=0x7fd4dc00a710 comp rx=0 tx=0).stop 2024-04-16T16:16:32.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.422+0000 7fd4e3200640 1 -- 172.21.15.45:0/669278404 shutdown_connections 2024-04-16T16:16:32.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.422+0000 7fd4e3200640 1 --2- 172.21.15.45:0/669278404 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fd4c006cb30 0x7fd4c006eff0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:32.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.422+0000 7fd4e3200640 1 --2- 172.21.15.45:0/669278404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e41461b0 0x7fd4e4162530 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:32.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.422+0000 7fd4e3200640 1 --2- 172.21.15.45:0/669278404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4e40cf440 0x7fd4e4161ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:32.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.422+0000 7fd4e3200640 1 --2- 172.21.15.45:0/669278404 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4e40b58f0 0x7fd4e41454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:32.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.422+0000 7fd4e3200640 1 -- 172.21.15.45:0/669278404 >> 172.21.15.45:0/669278404 conn(0x7fd4e406b930 msgr2=0x7fd4e413b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:32.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.422+0000 7fd4e3200640 1 -- 172.21.15.45:0/669278404 shutdown_connections 2024-04-16T16:16:32.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:32.422+0000 7fd4e3200640 1 -- 172.21.15.45:0/669278404 wait complete. 2024-04-16T16:16:32.440 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-16T16:16:32.440 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:16:32.562 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:16:32.683 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-16T16:16:32.684 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:16:32.684 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:16:32.744 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:16:32.744 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:16:32.744 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:16:32.802 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:16:32.803 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:16:32.803 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:16:32.863 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:16:32.863 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:16:32.863 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:16:32.922 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:16:32.922 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-16T16:16:32.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:16:33.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.154+0000 7f0c2be00640 1 Processor -- start 2024-04-16T16:16:33.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.154+0000 7f0c2be00640 1 -- start start 2024-04-16T16:16:33.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.156+0000 7f0c2be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c2c06dfb0 0x7f0c2c0c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:33.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.156+0000 7f0c2be00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c2c15c700 con 0x7f0c2c144ff0 2024-04-16T16:16:33.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.156+0000 7f0c2be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c2c15c9c0 con 0x7f0c2c0c5bb0 2024-04-16T16:16:33.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.156+0000 7f0c2be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0c2c15d2f0 con 0x7f0c2c06dfb0 2024-04-16T16:16:33.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.156+0000 7f0c2aa00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0c2c144ff0 0x7f0c2c0cfd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41088/0 (socket says 172.21.15.45:41088) 2024-04-16T16:16:33.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.156+0000 7f0c2aa00640 1 -- 172.21.15.45:0/510443516 learned_addr learned my addr 172.21.15.45:0/510443516 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:33.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.157+0000 7f0c23e00640 1 --2- 172.21.15.45:0/510443516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c2c06dfb0 0x7f0c2c0c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:33.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.157+0000 7f0c2a000640 1 -- 172.21.15.45:0/510443516 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 620270344 0 0) 0x7f0c2c15c700 con 0x7f0c2c144ff0 2024-04-16T16:16:33.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.157+0000 7f0c2a000640 1 -- 172.21.15.45:0/510443516 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c0c003520 con 0x7f0c2c144ff0 2024-04-16T16:16:33.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.157+0000 7f0c2a000640 1 -- 172.21.15.45:0/510443516 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3134880593 0 0) 0x7f0c0c003520 con 0x7f0c2c144ff0 2024-04-16T16:16:33.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.157+0000 7f0c2a000640 1 -- 172.21.15.45:0/510443516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c2c06dfb0 msgr2=0x7f0c2c0c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:33.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.157+0000 7f0c2a000640 1 --2- 172.21.15.45:0/510443516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c2c06dfb0 0x7f0c2c0c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:33.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.157+0000 7f0c2a000640 1 -- 172.21.15.45:0/510443516 >> v1:172.21.15.103:6789/0 conn(0x7f0c2c0c5bb0 legacy=0x7f0c2c15d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:33.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.157+0000 7f0c2a000640 1 -- 172.21.15.45:0/510443516 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c2c15cca0 con 0x7f0c2c144ff0 2024-04-16T16:16:33.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.157+0000 7f0c2a000640 1 -- 172.21.15.45:0/510443516 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f0c1c001670 con 0x7f0c2c144ff0 2024-04-16T16:16:33.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.157+0000 7f0c2a000640 1 -- 172.21.15.45:0/510443516 >> v1:172.21.15.45:6789/0 conn(0x7f0c2c144ff0 legacy=0x7f0c2c0cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:33.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.157+0000 7f0c2a000640 1 --2- 172.21.15.45:0/510443516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c0c0039e0 0x7f0c0c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:33.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.157+0000 7f0c2a000640 1 --2- 172.21.15.45:0/510443516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c0c004800 0x7f0c0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:33.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.157+0000 7f0c2a000640 1 --2- 172.21.15.45:0/510443516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c2c06dfb0 0x7f0c0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:33.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.158+0000 7f0c2a000640 1 -- 172.21.15.45:0/510443516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2c15d2f0 con 0x7f0c0c0039e0 2024-04-16T16:16:33.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.158+0000 7f0c2a000640 1 -- 172.21.15.45:0/510443516 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0c00ba70 con 0x7f0c0c004800 2024-04-16T16:16:33.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.158+0000 7f0c2a000640 1 -- 172.21.15.45:0/510443516 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0c0c00de80 con 0x7f0c2c06dfb0 2024-04-16T16:16:33.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.158+0000 7f0c2aa00640 1 --2- 172.21.15.45:0/510443516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c0c0039e0 0x7f0c0c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:33.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.158+0000 7f0c2b400640 1 --2- 172.21.15.45:0/510443516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c2c06dfb0 0x7f0c0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:33.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.158+0000 7f0c23e00640 1 --2- 172.21.15.45:0/510443516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c0c004800 0x7f0c0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:33.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.158+0000 7f0c2aa00640 1 -- 172.21.15.45:0/510443516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c2c06dfb0 msgr2=0x7f0c0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:33.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.158+0000 7f0c2aa00640 1 --2- 172.21.15.45:0/510443516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c2c06dfb0 0x7f0c0c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:33.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.158+0000 7f0c2aa00640 1 -- 172.21.15.45:0/510443516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c0c004800 msgr2=0x7f0c0c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:33.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.158+0000 7f0c2aa00640 1 --2- 172.21.15.45:0/510443516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c0c004800 0x7f0c0c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:33.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.158+0000 7f0c2aa00640 1 -- 172.21.15.45:0/510443516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0c003520 con 0x7f0c0c0039e0 2024-04-16T16:16:33.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.158+0000 7f0c2b400640 1 --2- 172.21.15.45:0/510443516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c2c06dfb0 0x7f0c0c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:33.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.158+0000 7f0c23e00640 1 --2- 172.21.15.45:0/510443516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c0c004800 0x7f0c0c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:33.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.159+0000 7f0c2aa00640 1 --2- 172.21.15.45:0/510443516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c0c0039e0 0x7f0c0c003e00 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c00c210 tx=0x7f0c0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b128f5392b5e887 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:33.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.160+0000 7f0c2a000640 1 -- 172.21.15.45:0/510443516 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c007d10 con 0x7f0c0c0039e0 2024-04-16T16:16:33.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.160+0000 7f0c2a000640 1 -- 172.21.15.45:0/510443516 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c1c030070 con 0x7f0c0c0039e0 2024-04-16T16:16:33.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.160+0000 7f0c2a000640 1 -- 172.21.15.45:0/510443516 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c005d30 con 0x7f0c0c0039e0 2024-04-16T16:16:33.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.160+0000 7f0c2be00640 1 -- 172.21.15.45:0/510443516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c0c0039e0 msgr2=0x7f0c0c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:33.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.160+0000 7f0c2be00640 1 --2- 172.21.15.45:0/510443516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c0c0039e0 0x7f0c0c003e00 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c00c210 tx=0x7f0c0c005830 comp rx=0 tx=0).stop 2024-04-16T16:16:33.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.160+0000 7f0c2b400640 1 -- 172.21.15.45:0/510443516 reap_dead start 2024-04-16T16:16:33.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.160+0000 7f0c2be00640 1 -- 172.21.15.45:0/510443516 shutdown_connections 2024-04-16T16:16:33.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.160+0000 7f0c2be00640 1 -- 172.21.15.45:0/510443516 >> 172.21.15.45:0/510443516 conn(0x7f0c2c06b930 msgr2=0x7f0c2c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:33.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.161+0000 7f0c2be00640 1 -- 172.21.15.45:0/510443516 shutdown_connections 2024-04-16T16:16:33.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.161+0000 7f0c2be00640 1 -- 172.21.15.45:0/510443516 wait complete. 2024-04-16T16:16:33.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.162+0000 7f0c2be00640 1 Processor -- start 2024-04-16T16:16:33.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.163+0000 7f0c2be00640 1 -- start start 2024-04-16T16:16:33.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.163+0000 7f0c2be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c0c0039e0 0x7f0c2c15b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:33.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.163+0000 7f0c2be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c2c135d80 0x7f0c2c141780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:33.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.163+0000 7f0c2be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c2c0cefc0 0x7f0c2c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:33.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.163+0000 7f0c2be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2c0c5670 con 0x7f0c2c0cefc0 2024-04-16T16:16:33.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.163+0000 7f0c2be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2c1453d0 con 0x7f0c0c0039e0 2024-04-16T16:16:33.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.163+0000 7f0c2be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0c2c0c57f0 con 0x7f0c2c135d80 2024-04-16T16:16:33.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.164+0000 7f0c2b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c2c0cefc0 0x7f0c2c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:33.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.164+0000 7f0c2b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c2c0cefc0 0x7f0c2c0cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46436/0 (socket says 172.21.15.45:46436) 2024-04-16T16:16:33.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.164+0000 7f0c2b400640 1 -- 172.21.15.45:0/1843260288 learned_addr learned my addr 172.21.15.45:0/1843260288 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:33.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.164+0000 7f0c2aa00640 1 --2- 172.21.15.45:0/1843260288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c0c0039e0 0x7f0c2c15b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:33.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.164+0000 7f0c23e00640 1 --2- 172.21.15.45:0/1843260288 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c2c135d80 0x7f0c2c141780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:33.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.164+0000 7f0c2b400640 1 -- 172.21.15.45:0/1843260288 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c2c135d80 msgr2=0x7f0c2c141780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:33.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.164+0000 7f0c2b400640 1 --2- 172.21.15.45:0/1843260288 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c2c135d80 0x7f0c2c141780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:33.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.164+0000 7f0c2b400640 1 -- 172.21.15.45:0/1843260288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c0c0039e0 msgr2=0x7f0c2c15b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:33.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.164+0000 7f0c2b400640 1 --2- 172.21.15.45:0/1843260288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c0c0039e0 0x7f0c2c15b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:33.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.164+0000 7f0c2b400640 1 -- 172.21.15.45:0/1843260288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c2c15b540 con 0x7f0c2c0cefc0 2024-04-16T16:16:33.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.164+0000 7f0c23e00640 1 --2- 172.21.15.45:0/1843260288 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c2c135d80 0x7f0c2c141780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:33.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.165+0000 7f0c2b400640 1 --2- 172.21.15.45:0/1843260288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c2c0cefc0 0x7f0c2c0cf440 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c070030 tx=0x7f0c2400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:33.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.165+0000 7f0c23400640 1 -- 172.21.15.45:0/1843260288 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c24006d50 con 0x7f0c2c0cefc0 2024-04-16T16:16:33.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.165+0000 7f0c23400640 1 -- 172.21.15.45:0/1843260288 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c24003970 con 0x7f0c2c0cefc0 2024-04-16T16:16:33.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.165+0000 7f0c23400640 1 -- 172.21.15.45:0/1843260288 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c24003da0 con 0x7f0c2c0cefc0 2024-04-16T16:16:33.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.165+0000 7f0c2be00640 1 -- 172.21.15.45:0/1843260288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c2c0cbef0 con 0x7f0c2c0cefc0 2024-04-16T16:16:33.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.165+0000 7f0c2be00640 1 -- 172.21.15.45:0/1843260288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c2c0cc4b0 con 0x7f0c2c0cefc0 2024-04-16T16:16:33.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.166+0000 7f0c2be00640 1 -- 172.21.15.45:0/1843260288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c2c0cf980 con 0x7f0c2c0cefc0 2024-04-16T16:16:33.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.167+0000 7f0c23400640 1 -- 172.21.15.45:0/1843260288 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f0c2401b020 con 0x7f0c2c0cefc0 2024-04-16T16:16:33.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.168+0000 7f0c23400640 1 --2- 172.21.15.45:0/1843260288 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0c0806cb30 0x7f0c0806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:33.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.168+0000 7f0c23400640 1 -- 172.21.15.45:0/1843260288 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5975+0+0 (secure 0 0 0) 0x7f0c2408bc90 con 0x7f0c2c0cefc0 2024-04-16T16:16:33.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.169+0000 7f0c2aa00640 1 --2- 172.21.15.45:0/1843260288 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0c0806cb30 0x7f0c0806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:33.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.170+0000 7f0c2aa00640 1 --2- 172.21.15.45:0/1843260288 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0c0806cb30 0x7f0c0806eff0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c034000 tx=0x7f0c1c009030 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:33.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.173+0000 7f0c23400640 1 -- 172.21.15.45:0/1843260288 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0c2c15b540 con 0x7f0c2c0cefc0 2024-04-16T16:16:33.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.418+0000 7f0c2be00640 1 -- 172.21.15.45:0/1843260288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c2c0b59c0 con 0x7f0c2c0cefc0 2024-04-16T16:16:33.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.419+0000 7f0c23400640 1 -- 172.21.15.45:0/1843260288 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+3869 (secure 0 0 0) 0x7f0c24059b40 con 0x7f0c2c0cefc0 2024-04-16T16:16:33.419 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:33.419 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":44,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/1497040472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1497040472},{"type":"v1","addr":"172.21.15.45:6834","nonce":1497040472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":4907,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/93057991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":93057991},{"type":"v1","addr":"172.21.15.45:6835","nonce":93057991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37},{"gid":5404,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1631198273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1631198273},{"type":"v1","addr":"172.21.15.103:6836","nonce":1631198273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5443,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1601605971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1601605971},{"type":"v1","addr":"172.21.15.103:6837","nonce":1601605971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":44}],"filesystems":[]} 2024-04-16T16:16:33.419 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 44 2024-04-16T16:16:33.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.422+0000 7f0c2be00640 1 -- 172.21.15.45:0/1843260288 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0c0806cb30 msgr2=0x7f0c0806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:33.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.422+0000 7f0c2be00640 1 --2- 172.21.15.45:0/1843260288 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0c0806cb30 0x7f0c0806eff0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c034000 tx=0x7f0c1c009030 comp rx=0 tx=0).stop 2024-04-16T16:16:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.422+0000 7f0c2be00640 1 -- 172.21.15.45:0/1843260288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c2c0cefc0 msgr2=0x7f0c2c0cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.422+0000 7f0c2be00640 1 --2- 172.21.15.45:0/1843260288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c2c0cefc0 0x7f0c2c0cf440 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c070030 tx=0x7f0c2400dc00 comp rx=0 tx=0).stop 2024-04-16T16:16:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.422+0000 7f0c2be00640 1 -- 172.21.15.45:0/1843260288 shutdown_connections 2024-04-16T16:16:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.422+0000 7f0c2be00640 1 --2- 172.21.15.45:0/1843260288 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0c0806cb30 0x7f0c0806eff0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.422+0000 7f0c2be00640 1 --2- 172.21.15.45:0/1843260288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c2c0cefc0 0x7f0c2c0cf440 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.422+0000 7f0c2be00640 1 --2- 172.21.15.45:0/1843260288 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c2c135d80 0x7f0c2c141780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.422+0000 7f0c2be00640 1 --2- 172.21.15.45:0/1843260288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c0c0039e0 0x7f0c2c15b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.422+0000 7f0c2be00640 1 -- 172.21.15.45:0/1843260288 >> 172.21.15.45:0/1843260288 conn(0x7f0c2c06b930 msgr2=0x7f0c2c06e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.423+0000 7f0c2be00640 1 -- 172.21.15.45:0/1843260288 shutdown_connections 2024-04-16T16:16:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.423+0000 7f0c2be00640 1 -- 172.21.15.45:0/1843260288 wait complete. 2024-04-16T16:16:33.443 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-16T16:16:33.443 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-16T16:16:33.443 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-16T16:16:33.443 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:16:33.444 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-16T16:16:33.444 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-16T16:16:33.444 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-16T16:16:33.444 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:16:33.444 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-16T16:16:33.444 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-16T16:16:33.444 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-16T16:16:33.445 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:16:33.445 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-16T16:16:33.445 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-16T16:16:33.445 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-16T16:16:33.445 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:16:33.465 INFO:tasks.ceph.mds.c:Stopped 2024-04-16T16:16:33.465 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-16T16:16:33.466 INFO:tasks.ceph.mds.a:Stopped 2024-04-16T16:16:33.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-16T16:16:33.471 INFO:tasks.ceph.mds.b:Stopped 2024-04-16T16:16:33.471 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-16T16:16:33.475 INFO:tasks.ceph.mds.d:Stopped 2024-04-16T16:16:33.475 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-16T16:16:33.512 INFO:tasks.ceph.mds.c:Started 2024-04-16T16:16:33.512 DEBUG:teuthology.parallel:result is None 2024-04-16T16:16:33.518 INFO:tasks.ceph.mds.a:Started 2024-04-16T16:16:33.518 DEBUG:teuthology.parallel:result is None 2024-04-16T16:16:33.522 INFO:tasks.ceph.mds.b:Started 2024-04-16T16:16:33.522 DEBUG:teuthology.parallel:result is None 2024-04-16T16:16:33.528 INFO:tasks.ceph.mds.d:Started 2024-04-16T16:16:33.528 DEBUG:teuthology.parallel:result is None 2024-04-16T16:16:33.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-16T16:16:33.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-04-16T16:16:33.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-04-16T16:16:33.678 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2024-04-16T16:16:33.678 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2024-04-16T16:16:33.692 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-16T16:16:33.696 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-16T16:16:33.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.721+0000 7f42c6600640 1 Processor -- start 2024-04-16T16:16:33.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.721+0000 7f42c6600640 1 -- start start 2024-04-16T16:16:33.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.722+0000 7f42c6600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42c015ed30 0x7f42c00d2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:33.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.722+0000 7f42c6600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f42c00d2fe0 0x7f42c00cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:33.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.722+0000 7f42c6600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42c00d3480 con 0x7f42c00ba710 2024-04-16T16:16:33.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.722+0000 7f42c6600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f42c015f110 con 0x7f42c00d2fe0 2024-04-16T16:16:33.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.722+0000 7f42c6600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f42c00d3760 con 0x7f42c015ed30 2024-04-16T16:16:33.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.722+0000 7f42bea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f42c00d2fe0 0x7f42c00cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:33.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.722+0000 7f42bea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f42c00d2fe0 0x7f42c00cf780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44988/0 (socket says 172.21.15.45:44988) 2024-04-16T16:16:33.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.722+0000 7f42bea00640 1 -- 172.21.15.45:0/1809760890 learned_addr learned my addr 172.21.15.45:0/1809760890 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:33.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.722+0000 7f42bf400640 1 --2- 172.21.15.45:0/1809760890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42c015ed30 0x7f42c00d2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:33.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.723+0000 7f42bea00640 1 -- 172.21.15.45:0/1809760890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42c015ed30 msgr2=0x7f42c00d2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:33.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.723+0000 7f42bea00640 1 --2- 172.21.15.45:0/1809760890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42c015ed30 0x7f42c00d2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:33.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.723+0000 7f42be000640 1 -- 172.21.15.45:0/1809760890 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 132749498 0 0) 0x7f42c00d3480 con 0x7f42c00ba710 2024-04-16T16:16:33.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.723+0000 7f42bea00640 1 -- 172.21.15.45:0/1809760890 >> v1:172.21.15.103:6789/0 conn(0x7f42c00ba710 legacy=0x7f42c006ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:33.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.723+0000 7f42bea00640 1 -- 172.21.15.45:0/1809760890 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c00d3a40 con 0x7f42c00d2fe0 2024-04-16T16:16:33.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.723+0000 7f42bf400640 1 --2- 172.21.15.45:0/1809760890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42c015ed30 0x7f42c00d2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:16:33.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.724+0000 7f42bea00640 1 --2- 172.21.15.45:0/1809760890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c00d2fe0 0x7f42c00cf780 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f42ac009870 tx=0x7f42ac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81656ce34e0bf2e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:33.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.724+0000 7f42be000640 1 -- 172.21.15.45:0/1809760890 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac034070 con 0x7f42c00d2fe0 2024-04-16T16:16:33.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.724+0000 7f42be000640 1 -- 172.21.15.45:0/1809760890 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42ac030840 con 0x7f42c00d2fe0 2024-04-16T16:16:33.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.724+0000 7f42be000640 1 -- 172.21.15.45:0/1809760890 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac030c10 con 0x7f42c00d2fe0 2024-04-16T16:16:33.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.725+0000 7f42c6600640 1 -- 172.21.15.45:0/1809760890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c00d2fe0 msgr2=0x7f42c00cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:33.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.725+0000 7f42c6600640 1 --2- 172.21.15.45:0/1809760890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c00d2fe0 0x7f42c00cf780 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f42ac009870 tx=0x7f42ac02b980 comp rx=0 tx=0).stop 2024-04-16T16:16:33.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.725+0000 7f42c6600640 1 -- 172.21.15.45:0/1809760890 shutdown_connections 2024-04-16T16:16:33.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.725+0000 7f42c6600640 1 --2- 172.21.15.45:0/1809760890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c00d2fe0 0x7f42c00cf780 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:33.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.725+0000 7f42c6600640 1 --2- 172.21.15.45:0/1809760890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42c015ed30 0x7f42c00d2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:33.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.725+0000 7f42c6600640 1 -- 172.21.15.45:0/1809760890 >> 172.21.15.45:0/1809760890 conn(0x7f42c00d1610 msgr2=0x7f42c00d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:33.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.725+0000 7f42c6600640 1 -- 172.21.15.45:0/1809760890 shutdown_connections 2024-04-16T16:16:33.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.725+0000 7f42c6600640 1 -- 172.21.15.45:0/1809760890 wait complete. 2024-04-16T16:16:33.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.727+0000 7f42c6600640 1 Processor -- start 2024-04-16T16:16:33.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.727+0000 7f42c6600640 1 -- start start 2024-04-16T16:16:33.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.728+0000 7f42c6600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42c00ba710 0x7f42c0153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:33.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.728+0000 7f42c6600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42c00d2fe0 0x7f42c00abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:33.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.728+0000 7f42c6600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c015ed30 0x7f42c00cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:33.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.728+0000 7f42c6600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f42c00b8af0 con 0x7f42c015ed30 2024-04-16T16:16:33.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.728+0000 7f42c6600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f42c00d4000 con 0x7f42c00ba710 2024-04-16T16:16:33.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.728+0000 7f42c6600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f42c00d4520 con 0x7f42c00d2fe0 2024-04-16T16:16:33.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.728+0000 7f42bfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c015ed30 0x7f42c00cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:33.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.728+0000 7f42bfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c015ed30 0x7f42c00cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44992/0 (socket says 172.21.15.45:44992) 2024-04-16T16:16:33.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.728+0000 7f42bfe00640 1 -- 172.21.15.45:0/1827053650 learned_addr learned my addr 172.21.15.45:0/1827053650 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:33.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.729+0000 7f42bea00640 1 --2- 172.21.15.45:0/1827053650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42c00d2fe0 0x7f42c00abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:33.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.729+0000 7f42bf400640 1 --2- 172.21.15.45:0/1827053650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42c00ba710 0x7f42c0153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:33.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.729+0000 7f42bfe00640 1 -- 172.21.15.45:0/1827053650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42c00d2fe0 msgr2=0x7f42c00abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:33.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.729+0000 7f42bfe00640 1 --2- 172.21.15.45:0/1827053650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42c00d2fe0 0x7f42c00abdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:33.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.729+0000 7f42bfe00640 1 -- 172.21.15.45:0/1827053650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42c00ba710 msgr2=0x7f42c0153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:33.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.729+0000 7f42bfe00640 1 --2- 172.21.15.45:0/1827053650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42c00ba710 0x7f42c0153c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:33.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.729+0000 7f42bfe00640 1 -- 172.21.15.45:0/1827053650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c00ad0b0 con 0x7f42c015ed30 2024-04-16T16:16:33.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.729+0000 7f42bf400640 1 --2- 172.21.15.45:0/1827053650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42c00ba710 0x7f42c0153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:16:33.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.729+0000 7f42bfe00640 1 --2- 172.21.15.45:0/1827053650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c015ed30 0x7f42c00cdc50 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f42b800c780 tx=0x7f42b800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:33.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.729+0000 7f42b7400640 1 -- 172.21.15.45:0/1827053650 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b8003da0 con 0x7f42c015ed30 2024-04-16T16:16:33.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.729+0000 7f42b7400640 1 -- 172.21.15.45:0/1827053650 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42b800ce30 con 0x7f42c015ed30 2024-04-16T16:16:33.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.729+0000 7f42b7400640 1 -- 172.21.15.45:0/1827053650 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b8005290 con 0x7f42c015ed30 2024-04-16T16:16:33.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.730+0000 7f42c6600640 1 -- 172.21.15.45:0/1827053650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c00ac810 con 0x7f42c015ed30 2024-04-16T16:16:33.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.730+0000 7f42c6600640 1 -- 172.21.15.45:0/1827053650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c00acdd0 con 0x7f42c015ed30 2024-04-16T16:16:33.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.731+0000 7f42c6600640 1 -- 172.21.15.45:0/1827053650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42c01f4d80 con 0x7f42c015ed30 2024-04-16T16:16:33.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.732+0000 7f42b7400640 1 -- 172.21.15.45:0/1827053650 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f42b8005430 con 0x7f42c015ed30 2024-04-16T16:16:33.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.733+0000 7f42b7400640 1 --2- 172.21.15.45:0/1827053650 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f42a006cb30 0x7f42a006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:33.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.733+0000 7f42b7400640 1 -- 172.21.15.45:0/1827053650 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5975+0+0 (secure 0 0 0) 0x7f42b808bde0 con 0x7f42c015ed30 2024-04-16T16:16:33.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.733+0000 7f42bf400640 1 --2- 172.21.15.45:0/1827053650 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f42a006cb30 0x7f42a006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:33.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.734+0000 7f42bf400640 1 --2- 172.21.15.45:0/1827053650 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f42a006cb30 0x7f42a006eff0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009ba0 tx=0x7f42b0008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:33.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.737+0000 7f42b7400640 1 -- 172.21.15.45:0/1827053650 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f42b8059e30 con 0x7f42c015ed30 2024-04-16T16:16:33.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:33.961+0000 7f42c6600640 1 -- 172.21.15.45:0/1827053650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f42c01f4950 con 0x7f42c015ed30 2024-04-16T16:16:34.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:34.975+0000 7f42b7400640 1 -- 172.21.15.45:0/1827053650 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v68) ==== 97+0+0 (secure 0 0 0) 0x7f42b8059730 con 0x7f42c015ed30 2024-04-16T16:16:34.974 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2024-04-16T16:16:34.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:34.979+0000 7f42c6600640 1 -- 172.21.15.45:0/1827053650 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f42a006cb30 msgr2=0x7f42a006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:34.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:34.979+0000 7f42c6600640 1 --2- 172.21.15.45:0/1827053650 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f42a006cb30 0x7f42a006eff0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009ba0 tx=0x7f42b0008040 comp rx=0 tx=0).stop 2024-04-16T16:16:34.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:34.979+0000 7f42c6600640 1 -- 172.21.15.45:0/1827053650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c015ed30 msgr2=0x7f42c00cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:34.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:34.979+0000 7f42c6600640 1 --2- 172.21.15.45:0/1827053650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c015ed30 0x7f42c00cdc50 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f42b800c780 tx=0x7f42b800cc40 comp rx=0 tx=0).stop 2024-04-16T16:16:34.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:34.979+0000 7f42c6600640 1 -- 172.21.15.45:0/1827053650 shutdown_connections 2024-04-16T16:16:34.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:34.979+0000 7f42c6600640 1 --2- 172.21.15.45:0/1827053650 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f42a006cb30 0x7f42a006eff0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:34.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:34.979+0000 7f42c6600640 1 --2- 172.21.15.45:0/1827053650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c015ed30 0x7f42c00cdc50 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:34.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:34.979+0000 7f42c6600640 1 --2- 172.21.15.45:0/1827053650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42c00d2fe0 0x7f42c00abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:34.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:34.979+0000 7f42c6600640 1 --2- 172.21.15.45:0/1827053650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42c00ba710 0x7f42c0153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:34.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:34.979+0000 7f42c6600640 1 -- 172.21.15.45:0/1827053650 >> 172.21.15.45:0/1827053650 conn(0x7f42c00d1610 msgr2=0x7f42c01612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:34.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:34.979+0000 7f42c6600640 1 -- 172.21.15.45:0/1827053650 shutdown_connections 2024-04-16T16:16:34.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:34.979+0000 7f42c6600640 1 -- 172.21.15.45:0/1827053650 wait complete. 2024-04-16T16:16:34.999 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-16T16:16:35.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.222+0000 7fc6d5c00640 1 Processor -- start 2024-04-16T16:16:35.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.222+0000 7fc6d5c00640 1 -- start start 2024-04-16T16:16:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.224+0000 7fc6d5c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc6d0162270 0x7fc6d00d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.224+0000 7fc6d5c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6d00d52d0 0x7fc6d006ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.224+0000 7fc6d5c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d00d3080 con 0x7fc6d0161e90 2024-04-16T16:16:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.224+0000 7fc6d5c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc6d00d2e30 con 0x7fc6d0162270 2024-04-16T16:16:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.224+0000 7fc6d5c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc6d00d3360 con 0x7fc6d00d52d0 2024-04-16T16:16:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.224+0000 7fc6cf400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc6d0162270 0x7fc6d00d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.224+0000 7fc6cf400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc6d0162270 0x7fc6d00d3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44994/0 (socket says 172.21.15.45:44994) 2024-04-16T16:16:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.224+0000 7fc6cf400640 1 -- 172.21.15.45:0/53992438 learned_addr learned my addr 172.21.15.45:0/53992438 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.225+0000 7fc6d5200640 1 --2- 172.21.15.45:0/53992438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6d00d52d0 0x7fc6d006ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.225+0000 7fc6cf400640 1 -- 172.21.15.45:0/53992438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6d00d52d0 msgr2=0x7fc6d006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:35.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.225+0000 7fc6cf400640 1 --2- 172.21.15.45:0/53992438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6d00d52d0 0x7fc6d006ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:35.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.225+0000 7fc6cf400640 1 -- 172.21.15.45:0/53992438 >> v1:172.21.15.103:6789/0 conn(0x7fc6d0161e90 legacy=0x7fc6d00cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:35.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.225+0000 7fc6cf400640 1 -- 172.21.15.45:0/53992438 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d00d3640 con 0x7fc6d0162270 2024-04-16T16:16:35.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.226+0000 7fc6cf400640 1 --2- 172.21.15.45:0/53992438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d0162270 0x7fc6d00d3e40 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc0098a0 tx=0x7fc6bc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc8a41ef3fc0e95 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:35.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.227+0000 7fc6cea00640 1 -- 172.21.15.45:0/53992438 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc034070 con 0x7fc6d0162270 2024-04-16T16:16:35.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.227+0000 7fc6cea00640 1 -- 172.21.15.45:0/53992438 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6bc030840 con 0x7fc6d0162270 2024-04-16T16:16:35.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.227+0000 7fc6cea00640 1 -- 172.21.15.45:0/53992438 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc030c10 con 0x7fc6d0162270 2024-04-16T16:16:35.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.227+0000 7fc6d5c00640 1 -- 172.21.15.45:0/53992438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d0162270 msgr2=0x7fc6d00d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:35.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.227+0000 7fc6d5c00640 1 --2- 172.21.15.45:0/53992438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d0162270 0x7fc6d00d3e40 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc0098a0 tx=0x7fc6bc02b980 comp rx=0 tx=0).stop 2024-04-16T16:16:35.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.227+0000 7fc6d5c00640 1 -- 172.21.15.45:0/53992438 shutdown_connections 2024-04-16T16:16:35.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.227+0000 7fc6d5c00640 1 --2- 172.21.15.45:0/53992438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6d00d52d0 0x7fc6d006ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:35.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.227+0000 7fc6d5c00640 1 --2- 172.21.15.45:0/53992438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d0162270 0x7fc6d00d3e40 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:35.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.227+0000 7fc6d5c00640 1 -- 172.21.15.45:0/53992438 >> 172.21.15.45:0/53992438 conn(0x7fc6d00d1940 msgr2=0x7fc6d00d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:35.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.228+0000 7fc6d5c00640 1 -- 172.21.15.45:0/53992438 shutdown_connections 2024-04-16T16:16:35.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.228+0000 7fc6d5c00640 1 -- 172.21.15.45:0/53992438 wait complete. 2024-04-16T16:16:35.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.229+0000 7fc6d5c00640 1 Processor -- start 2024-04-16T16:16:35.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.229+0000 7fc6d5c00640 1 -- start start 2024-04-16T16:16:35.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.230+0000 7fc6d5c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6d00d52d0 0x7fc6d01f5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:35.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.230+0000 7fc6d5c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6d0161e90 0x7fc6d01f4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:35.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.230+0000 7fc6d5c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d0162270 0x7fc6d01f8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:35.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.231+0000 7fc6d5c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d00d4500 con 0x7fc6d0162270 2024-04-16T16:16:35.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.231+0000 7fc6d5c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d00d4380 con 0x7fc6d00d52d0 2024-04-16T16:16:35.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.231+0000 7fc6d5c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc6d00d4680 con 0x7fc6d0161e90 2024-04-16T16:16:35.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.231+0000 7fc6d5200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d0162270 0x7fc6d01f8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:35.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.231+0000 7fc6d5200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d0162270 0x7fc6d01f8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45010/0 (socket says 172.21.15.45:45010) 2024-04-16T16:16:35.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.231+0000 7fc6d5200640 1 -- 172.21.15.45:0/4228123339 learned_addr learned my addr 172.21.15.45:0/4228123339 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:35.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.231+0000 7fc6cf400640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6d0161e90 0x7fc6d01f4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:35.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.231+0000 7fc6cfe00640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6d00d52d0 0x7fc6d01f5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:35.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.232+0000 7fc6d5200640 1 -- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6d0161e90 msgr2=0x7fc6d01f4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:35.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.232+0000 7fc6d5200640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6d0161e90 0x7fc6d01f4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:35.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.232+0000 7fc6d5200640 1 -- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6d00d52d0 msgr2=0x7fc6d01f5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:35.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.232+0000 7fc6d5200640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6d00d52d0 0x7fc6d01f5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:35.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.232+0000 7fc6d5200640 1 -- 172.21.15.45:0/4228123339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d01f9890 con 0x7fc6d0162270 2024-04-16T16:16:35.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.232+0000 7fc6cfe00640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6d00d52d0 0x7fc6d01f5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:16:35.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.232+0000 7fc6cf400640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6d0161e90 0x7fc6d01f4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:16:35.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.232+0000 7fc6d5200640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d0162270 0x7fc6d01f8960 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8007e30 tx=0x7fc6c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:35.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.232+0000 7fc6c7e00640 1 -- 172.21.15.45:0/4228123339 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8014620 con 0x7fc6d0162270 2024-04-16T16:16:35.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.232+0000 7fc6c7e00640 1 -- 172.21.15.45:0/4228123339 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6c8019070 con 0x7fc6d0162270 2024-04-16T16:16:35.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.232+0000 7fc6d5c00640 1 -- 172.21.15.45:0/4228123339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d01f92d0 con 0x7fc6d0162270 2024-04-16T16:16:35.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.232+0000 7fc6c7e00640 1 -- 172.21.15.45:0/4228123339 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8014a00 con 0x7fc6d0162270 2024-04-16T16:16:35.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.233+0000 7fc6d5c00640 1 -- 172.21.15.45:0/4228123339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d01f57f0 con 0x7fc6d0162270 2024-04-16T16:16:35.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.234+0000 7fc6d5c00640 1 -- 172.21.15.45:0/4228123339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6d0075780 con 0x7fc6d0162270 2024-04-16T16:16:35.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.235+0000 7fc6c7e00640 1 -- 172.21.15.45:0/4228123339 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fc6c8014ba0 con 0x7fc6d0162270 2024-04-16T16:16:35.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.236+0000 7fc6c7e00640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc6b006cb30 0x7fc6b006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:35.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.236+0000 7fc6c7e00640 1 -- 172.21.15.45:0/4228123339 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 4857+0+0 (secure 0 0 0) 0x7fc6c808c3f0 con 0x7fc6d0162270 2024-04-16T16:16:35.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.237+0000 7fc6cfe00640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc6b006cb30 0x7fc6b006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:35.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.238+0000 7fc6cfe00640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc6b006cb30 0x7fc6b006eff0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc6d01f88c0 tx=0x7fc6c0009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:35.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.240+0000 7fc6c7e00640 1 -- 172.21.15.45:0/4228123339 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc6c805a8e0 con 0x7fc6d0162270 2024-04-16T16:16:35.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.526+0000 7fc6d5c00640 1 -- 172.21.15.45:0/4228123339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fc6d00c7ec0 con 0x7fc6d0162270 2024-04-16T16:16:35.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.527+0000 7fc6c7e00640 1 -- 172.21.15.45:0/4228123339 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) ==== 80+0+5229 (secure 0 0 0) 0x7fc6c805a1e0 con 0x7fc6d0162270 2024-04-16T16:16:35.526 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:35.526 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:16:35.526 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2024-04-16T16:16:35.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.526 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2024-04-16T16:16:35.526 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mpjReORAAJMoNf4qvfuxPSiV9+VlKfg==", 2024-04-16T16:16:35.526 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:16:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:16:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:16:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:16:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2024-04-16T16:16:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mxHeBIBAA3G8maaK65zaOaqXpAOA6xA==", 2024-04-16T16:16:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:16:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:16:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:16:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:16:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2024-04-16T16:16:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5m+AaODxAA08XSjc44/i2KYExYQSukaQ==", 2024-04-16T16:16:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:16:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:16:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:16:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:16:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2024-04-16T16:16:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mEQ4UMRAAOslSsxQ17bOxPrI6QMlRRw==", 2024-04-16T16:16:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:16:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:16:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:16:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:16:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2024-04-16T16:16:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBuox5m3l3OHxAA+UjwZx8Mn6tCu95TTYIG3w==", 2024-04-16T16:16:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:16:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:16:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:16:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2024-04-16T16:16:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxox5mozt1LhAAJL5NtMQdPVBtKOaVlMQBtw==", 2024-04-16T16:16:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:16:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:16:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:16:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2024-04-16T16:16:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB1ox5mfE7kABAAIvgZ5Wx+eIJ6QNKj2GVdpw==", 2024-04-16T16:16:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:16:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:16:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:16:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2024-04-16T16:16:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB4ox5mEvxcDRAAO0yXBKvsC7hZ6bpOAJyuag==", 2024-04-16T16:16:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:16:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:16:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:16:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2024-04-16T16:16:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+ox5mZubrORAAcGJLyRN5Ob9aBnopumnEUg==", 2024-04-16T16:16:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:16:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:16:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:16:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2024-04-16T16:16:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCCox5mcRqsChAACktmuUHvqWkp/mHT3UcjRw==", 2024-04-16T16:16:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:16:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:16:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:16:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2024-04-16T16:16:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFox5mT52FFhAAChhDDfNZmIMgxE1j90LZpQ==", 2024-04-16T16:16:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:16:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:16:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:16:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2024-04-16T16:16:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCIox5mC+EDIxAAxtvM0on00lxTLaoRjN0Gfw==", 2024-04-16T16:16:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:16:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:16:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:16:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2024-04-16T16:16:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ==", 2024-04-16T16:16:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:16:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:16:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mYpVUGhAAYdR+40O6zbWqgbI55F0FYA==", 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mWdlUDxAAuO2FsGH1/E+qP1xcyUEe3A==", 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:16:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:16:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:16:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2024-04-16T16:16:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5miPdMJBAAqjKpctNYu9ot1lff3rouxg==", 2024-04-16T16:16:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:16:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:16:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:16:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2024-04-16T16:16:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBpox5mO4PMLBAA/0YbQCfHujgdEzpXs0nOLw==", 2024-04-16T16:16:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:16:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:16:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:16:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:16:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2024-04-16T16:16:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mGevaLBAAPpA6g+Fdi/XHqAqG0QZNag==", 2024-04-16T16:16:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:16:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:16:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:16:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:16:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2024-04-16T16:16:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5moiNRIRAAz5JzOrjKNLkcphT+iMo+zw==", 2024-04-16T16:16:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:16:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:16:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:16:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:16:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-16T16:16:35.538 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:16:35.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.530+0000 7fc6d5c00640 1 -- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc6b006cb30 msgr2=0x7fc6b006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:35.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.530+0000 7fc6d5c00640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc6b006cb30 0x7fc6b006eff0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc6d01f88c0 tx=0x7fc6c0009040 comp rx=0 tx=0).stop 2024-04-16T16:16:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.530+0000 7fc6d5c00640 1 -- 172.21.15.45:0/4228123339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d0162270 msgr2=0x7fc6d01f8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.530+0000 7fc6d5c00640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d0162270 0x7fc6d01f8960 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8007e30 tx=0x7fc6c800e4b0 comp rx=0 tx=0).stop 2024-04-16T16:16:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.530+0000 7fc6d5c00640 1 -- 172.21.15.45:0/4228123339 shutdown_connections 2024-04-16T16:16:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.530+0000 7fc6d5c00640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc6b006cb30 0x7fc6b006eff0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.530+0000 7fc6d5c00640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d0162270 0x7fc6d01f8960 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.530+0000 7fc6d5c00640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6d0161e90 0x7fc6d01f4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.530+0000 7fc6d5c00640 1 --2- 172.21.15.45:0/4228123339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6d00d52d0 0x7fc6d01f5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.530+0000 7fc6d5c00640 1 -- 172.21.15.45:0/4228123339 >> 172.21.15.45:0/4228123339 conn(0x7fc6d00d1940 msgr2=0x7fc6d00b5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.530+0000 7fc6d5c00640 1 -- 172.21.15.45:0/4228123339 shutdown_connections 2024-04-16T16:16:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.530+0000 7fc6d5c00640 1 -- 172.21.15.45:0/4228123339 wait complete. 2024-04-16T16:16:35.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:16:35.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.776+0000 7ff21d400640 1 Processor -- start 2024-04-16T16:16:35.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.776+0000 7ff21d400640 1 -- start start 2024-04-16T16:16:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.777+0000 7ff21d400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff218144ff0 0x7ff2180cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.777+0000 7ff21d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff21806dfb0 0x7ff2180c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.777+0000 7ff21d400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff21815c700 con 0x7ff2180c5bb0 2024-04-16T16:16:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.777+0000 7ff21d400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff21815d2f0 con 0x7ff218144ff0 2024-04-16T16:16:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.778+0000 7ff21d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff21815c9c0 con 0x7ff21806dfb0 2024-04-16T16:16:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.778+0000 7ff217e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff218144ff0 0x7ff2180cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.778+0000 7ff217e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff218144ff0 0x7ff2180cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45018/0 (socket says 172.21.15.45:45018) 2024-04-16T16:16:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.778+0000 7ff217e00640 1 -- 172.21.15.45:0/596237194 learned_addr learned my addr 172.21.15.45:0/596237194 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.778+0000 7ff21ca00640 1 --1- 172.21.15.45:0/596237194 >> v1:172.21.15.45:6789/0 conn(0x7ff2180c5bb0 0x7ff21815f270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42354/0 (socket says 172.21.15.45:42354) 2024-04-16T16:16:35.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.778+0000 7ff217400640 1 --2- 172.21.15.45:0/596237194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff21806dfb0 0x7ff2180c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:35.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.778+0000 7ff217e00640 1 -- 172.21.15.45:0/596237194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff21806dfb0 msgr2=0x7ff2180c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:35.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.778+0000 7ff217e00640 1 --2- 172.21.15.45:0/596237194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff21806dfb0 0x7ff2180c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:35.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.778+0000 7ff217e00640 1 -- 172.21.15.45:0/596237194 >> v1:172.21.15.45:6789/0 conn(0x7ff2180c5bb0 legacy=0x7ff21815f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:35.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.778+0000 7ff217e00640 1 -- 172.21.15.45:0/596237194 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21815cca0 con 0x7ff218144ff0 2024-04-16T16:16:35.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.780+0000 7ff217e00640 1 --2- 172.21.15.45:0/596237194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff218144ff0 0x7ff2180cfd30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff208009e50 tx=0x7ff20802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800043d6a2ccae58 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:35.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.780+0000 7ff216a00640 1 -- 172.21.15.45:0/596237194 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20802cb00 con 0x7ff218144ff0 2024-04-16T16:16:35.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.780+0000 7ff216a00640 1 -- 172.21.15.45:0/596237194 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff20802cca0 con 0x7ff218144ff0 2024-04-16T16:16:35.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.780+0000 7ff216a00640 1 -- 172.21.15.45:0/596237194 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff208034610 con 0x7ff218144ff0 2024-04-16T16:16:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.781+0000 7ff21d400640 1 -- 172.21.15.45:0/596237194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff218144ff0 msgr2=0x7ff2180cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.781+0000 7ff21d400640 1 --2- 172.21.15.45:0/596237194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff218144ff0 0x7ff2180cfd30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff208009e50 tx=0x7ff20802bf60 comp rx=0 tx=0).stop 2024-04-16T16:16:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.781+0000 7ff21d400640 1 -- 172.21.15.45:0/596237194 shutdown_connections 2024-04-16T16:16:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.781+0000 7ff21d400640 1 --2- 172.21.15.45:0/596237194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff21806dfb0 0x7ff2180c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.781+0000 7ff21d400640 1 --2- 172.21.15.45:0/596237194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff218144ff0 0x7ff2180cfd30 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.781+0000 7ff21d400640 1 -- 172.21.15.45:0/596237194 >> 172.21.15.45:0/596237194 conn(0x7ff21806b930 msgr2=0x7ff21806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.781+0000 7ff21d400640 1 -- 172.21.15.45:0/596237194 shutdown_connections 2024-04-16T16:16:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.781+0000 7ff21d400640 1 -- 172.21.15.45:0/596237194 wait complete. 2024-04-16T16:16:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.783+0000 7ff21d400640 1 Processor -- start 2024-04-16T16:16:35.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.783+0000 7ff21d400640 1 -- start start 2024-04-16T16:16:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.784+0000 7ff21d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff21806dfb0 0x7ff2181f5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.784+0000 7ff21d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2180c5bb0 0x7ff21815a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.784+0000 7ff21d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff218144ff0 0x7ff21815a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.784+0000 7ff21d400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff2180d0270 con 0x7ff2180c5bb0 2024-04-16T16:16:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.784+0000 7ff21d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff2181453d0 con 0x7ff218144ff0 2024-04-16T16:16:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.784+0000 7ff21d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff2180d03f0 con 0x7ff21806dfb0 2024-04-16T16:16:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.784+0000 7ff217400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2180c5bb0 0x7ff21815a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.784+0000 7ff217400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2180c5bb0 0x7ff21815a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45028/0 (socket says 172.21.15.45:45028) 2024-04-16T16:16:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.784+0000 7ff217400640 1 -- 172.21.15.45:0/496116569 learned_addr learned my addr 172.21.15.45:0/496116569 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.784+0000 7ff217400640 1 -- 172.21.15.45:0/496116569 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff21806dfb0 msgr2=0x7ff2181f5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.784+0000 7ff217400640 1 --2- 172.21.15.45:0/496116569 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff21806dfb0 0x7ff2181f5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.784+0000 7ff217400640 1 -- 172.21.15.45:0/496116569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff218144ff0 msgr2=0x7ff21815a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.784+0000 7ff217400640 1 --2- 172.21.15.45:0/496116569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff218144ff0 0x7ff21815a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.784+0000 7ff217400640 1 -- 172.21.15.45:0/496116569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff218166160 con 0x7ff2180c5bb0 2024-04-16T16:16:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.785+0000 7ff217400640 1 --2- 172.21.15.45:0/496116569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2180c5bb0 0x7ff21815a250 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7ff20000d7d0 tx=0x7ff20000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.785+0000 7ff20fe00640 1 -- 172.21.15.45:0/496116569 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff200004020 con 0x7ff2180c5bb0 2024-04-16T16:16:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.785+0000 7ff20fe00640 1 -- 172.21.15.45:0/496116569 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2000041e0 con 0x7ff2180c5bb0 2024-04-16T16:16:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.785+0000 7ff20fe00640 1 -- 172.21.15.45:0/496116569 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2000043d0 con 0x7ff2180c5bb0 2024-04-16T16:16:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.785+0000 7ff21d400640 1 -- 172.21.15.45:0/496116569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff218166440 con 0x7ff2180c5bb0 2024-04-16T16:16:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.785+0000 7ff21d400640 1 -- 172.21.15.45:0/496116569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff218166a00 con 0x7ff2180c5bb0 2024-04-16T16:16:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.786+0000 7ff21d400640 1 -- 172.21.15.45:0/496116569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff218075780 con 0x7ff2180c5bb0 2024-04-16T16:16:35.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.788+0000 7ff20fe00640 1 -- 172.21.15.45:0/496116569 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7ff20001b020 con 0x7ff2180c5bb0 2024-04-16T16:16:35.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.788+0000 7ff20fe00640 1 --2- 172.21.15.45:0/496116569 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7ff1f406cb30 0x7ff1f406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:35.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.789+0000 7ff20fe00640 1 -- 172.21.15.45:0/496116569 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 4857+0+0 (secure 0 0 0) 0x7ff20008bb40 con 0x7ff2180c5bb0 2024-04-16T16:16:35.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.789+0000 7ff217e00640 1 --2- 172.21.15.45:0/496116569 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7ff1f406cb30 0x7ff1f406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:35.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.791+0000 7ff217e00640 1 --2- 172.21.15.45:0/496116569 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7ff1f406cb30 0x7ff1f406eff0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff208035000 tx=0x7ff20802e000 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:35.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:35.794+0000 7ff20fe00640 1 -- 172.21.15.45:0/496116569 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff200059ec0 con 0x7ff2180c5bb0 2024-04-16T16:16:36.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.003+0000 7ff21d400640 1 -- 172.21.15.45:0/496116569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff21806eb80 con 0x7ff2180c5bb0 2024-04-16T16:16:36.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.005+0000 7ff20fe00640 1 -- 172.21.15.45:0/496116569 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) ==== 74+0+12891 (secure 0 0 0) 0x7ff2000597c0 con 0x7ff2180c5bb0 2024-04-16T16:16:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:36.004 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":68,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:16:34.972366+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:16:36.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.008+0000 7ff21d400640 1 -- 172.21.15.45:0/496116569 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7ff1f406cb30 msgr2=0x7ff1f406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:36.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.008+0000 7ff21d400640 1 --2- 172.21.15.45:0/496116569 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7ff1f406cb30 0x7ff1f406eff0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff208035000 tx=0x7ff20802e000 comp rx=0 tx=0).stop 2024-04-16T16:16:36.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.008+0000 7ff21d400640 1 -- 172.21.15.45:0/496116569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2180c5bb0 msgr2=0x7ff21815a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:36.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.008+0000 7ff21d400640 1 --2- 172.21.15.45:0/496116569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2180c5bb0 0x7ff21815a250 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7ff20000d7d0 tx=0x7ff20000dc90 comp rx=0 tx=0).stop 2024-04-16T16:16:36.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.008+0000 7ff21d400640 1 -- 172.21.15.45:0/496116569 shutdown_connections 2024-04-16T16:16:36.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.008+0000 7ff21d400640 1 --2- 172.21.15.45:0/496116569 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7ff1f406cb30 0x7ff1f406eff0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.008+0000 7ff21d400640 1 --2- 172.21.15.45:0/496116569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff218144ff0 0x7ff21815a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.008+0000 7ff21d400640 1 --2- 172.21.15.45:0/496116569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2180c5bb0 0x7ff21815a250 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.008+0000 7ff21d400640 1 --2- 172.21.15.45:0/496116569 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff21806dfb0 0x7ff2181f5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.008+0000 7ff21d400640 1 -- 172.21.15.45:0/496116569 >> 172.21.15.45:0/496116569 conn(0x7ff21806b930 msgr2=0x7ff2180b4c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:36.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.009+0000 7ff21d400640 1 -- 172.21.15.45:0/496116569 shutdown_connections 2024-04-16T16:16:36.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.009+0000 7ff21d400640 1 -- 172.21.15.45:0/496116569 wait complete. 2024-04-16T16:16:36.028 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:16:36.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:16:36.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.257+0000 7f5dd4c00640 1 Processor -- start 2024-04-16T16:16:36.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.257+0000 7f5dd4c00640 1 -- start start 2024-04-16T16:16:36.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.259+0000 7f5dd4c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5dd0072630 0x7f5dd0167490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:36.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.259+0000 7f5dd4c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dd0163fa0 con 0x7f5dd015d400 2024-04-16T16:16:36.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.259+0000 7f5dd4c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dd0164280 con 0x7f5dd0163360 2024-04-16T16:16:36.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.259+0000 7f5dd4c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5dd01669b0 con 0x7f5dd0072630 2024-04-16T16:16:36.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.260+0000 7f5dcf400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5dd015d400 0x7f5dd0071f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42360/0 (socket says 172.21.15.45:42360) 2024-04-16T16:16:36.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.260+0000 7f5dcf400640 1 -- 172.21.15.45:0/2512970948 learned_addr learned my addr 172.21.15.45:0/2512970948 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:36.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.260+0000 7f5dcea00640 1 --2- 172.21.15.45:0/2512970948 >> v2:172.21.15.103:3300/0 conn(0x7f5dd0072630 0x7f5dd0167490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:36.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.260+0000 7f5dce000640 1 -- 172.21.15.45:0/2512970948 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 541218950 0 0) 0x7f5dd0163fa0 con 0x7f5dd015d400 2024-04-16T16:16:36.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.260+0000 7f5dce000640 1 -- 172.21.15.45:0/2512970948 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5db4003520 con 0x7f5dd015d400 2024-04-16T16:16:36.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.261+0000 7f5dcea00640 1 -- 172.21.15.45:0/2512970948 >> v1:172.21.15.103:6789/0 conn(0x7f5dd0163360 legacy=0x7f5dd01637f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:36.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.261+0000 7f5dce000640 1 -- 172.21.15.45:0/2512970948 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3642645838 0 0) 0x7f5db4003520 con 0x7f5dd015d400 2024-04-16T16:16:36.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.261+0000 7f5dcea00640 1 -- 172.21.15.45:0/2512970948 >> v1:172.21.15.45:6789/0 conn(0x7f5dd015d400 legacy=0x7f5dd0071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:36.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.261+0000 7f5dcea00640 1 -- 172.21.15.45:0/2512970948 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd0166c90 con 0x7f5dd0072630 2024-04-16T16:16:36.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.262+0000 7f5dcea00640 1 --2- 172.21.15.45:0/2512970948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dd0072630 0x7f5dd0167490 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0097a0 tx=0x7f5dbc02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=534c70deb1dbd7db server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:36.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.263+0000 7f5dce000640 1 -- 172.21.15.45:0/2512970948 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc037070 con 0x7f5dd0072630 2024-04-16T16:16:36.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.263+0000 7f5dce000640 1 -- 172.21.15.45:0/2512970948 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dbc030510 con 0x7f5dd0072630 2024-04-16T16:16:36.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.263+0000 7f5dce000640 1 -- 172.21.15.45:0/2512970948 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0001670 con 0x7f5dd0072630 2024-04-16T16:16:36.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.264+0000 7f5dd4c00640 1 -- 172.21.15.45:0/2512970948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dd0072630 msgr2=0x7f5dd0167490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:36.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.264+0000 7f5dd4c00640 1 --2- 172.21.15.45:0/2512970948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dd0072630 0x7f5dd0167490 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0097a0 tx=0x7f5dbc02b790 comp rx=0 tx=0).stop 2024-04-16T16:16:36.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.264+0000 7f5dd4c00640 1 -- 172.21.15.45:0/2512970948 shutdown_connections 2024-04-16T16:16:36.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.264+0000 7f5dd4c00640 1 --2- 172.21.15.45:0/2512970948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dd0072630 0x7f5dd0167490 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.264+0000 7f5dd4c00640 1 -- 172.21.15.45:0/2512970948 >> 172.21.15.45:0/2512970948 conn(0x7f5dd0067e00 msgr2=0x7f5dd006b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:36.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.264+0000 7f5dd4c00640 1 -- 172.21.15.45:0/2512970948 shutdown_connections 2024-04-16T16:16:36.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.264+0000 7f5dd4c00640 1 -- 172.21.15.45:0/2512970948 wait complete. 2024-04-16T16:16:36.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.266+0000 7f5dd4c00640 1 Processor -- start 2024-04-16T16:16:36.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.266+0000 7f5dd4c00640 1 -- start start 2024-04-16T16:16:36.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.267+0000 7f5dd4c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd0072630 0x7f5dd01450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:36.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.267+0000 7f5dd4c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5dd015d400 0x7f5dd0159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:36.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.267+0000 7f5dd4c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dd0163360 0x7f5dd01596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:36.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.267+0000 7f5dd4c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd00cd0f0 con 0x7f5dd0072630 2024-04-16T16:16:36.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.267+0000 7f5dd4c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd015d7e0 con 0x7f5dd0163360 2024-04-16T16:16:36.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.267+0000 7f5dd4c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd00cd270 con 0x7f5dd015d400 2024-04-16T16:16:36.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.267+0000 7f5dcf400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd0072630 0x7f5dd01450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:36.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.267+0000 7f5dcf400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd0072630 0x7f5dd01450b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45032/0 (socket says 172.21.15.45:45032) 2024-04-16T16:16:36.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.267+0000 7f5dcf400640 1 -- 172.21.15.45:0/720110501 learned_addr learned my addr 172.21.15.45:0/720110501 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:36.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.268+0000 7f5dcea00640 1 --2- 172.21.15.45:0/720110501 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5dd015d400 0x7f5dd0159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:36.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.268+0000 7f5dcfe00640 1 --2- 172.21.15.45:0/720110501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dd0163360 0x7f5dd01596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:36.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.268+0000 7f5dcf400640 1 -- 172.21.15.45:0/720110501 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5dd015d400 msgr2=0x7f5dd0159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:36.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.268+0000 7f5dcf400640 1 --2- 172.21.15.45:0/720110501 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5dd015d400 0x7f5dd0159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.268+0000 7f5dcf400640 1 -- 172.21.15.45:0/720110501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dd0163360 msgr2=0x7f5dd01596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:36.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.268+0000 7f5dcf400640 1 --2- 172.21.15.45:0/720110501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dd0163360 0x7f5dd01596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.268+0000 7f5dcf400640 1 -- 172.21.15.45:0/720110501 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd01f9c00 con 0x7f5dd0072630 2024-04-16T16:16:36.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.268+0000 7f5dcf400640 1 --2- 172.21.15.45:0/720110501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd0072630 0x7f5dd01450b0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f5dc000c730 tx=0x7f5dc000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:36.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.269+0000 7f5dc7400640 1 -- 172.21.15.45:0/720110501 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0017070 con 0x7f5dd0072630 2024-04-16T16:16:36.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.269+0000 7f5dc7400640 1 -- 172.21.15.45:0/720110501 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc0005ad0 con 0x7f5dd0072630 2024-04-16T16:16:36.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.269+0000 7f5dc7400640 1 -- 172.21.15.45:0/720110501 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0007070 con 0x7f5dd0072630 2024-04-16T16:16:36.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.269+0000 7f5dd4c00640 1 -- 172.21.15.45:0/720110501 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd01f8e60 con 0x7f5dd0072630 2024-04-16T16:16:36.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.269+0000 7f5dd4c00640 1 -- 172.21.15.45:0/720110501 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd01f9400 con 0x7f5dd0072630 2024-04-16T16:16:36.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.270+0000 7f5dd4c00640 1 -- 172.21.15.45:0/720110501 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dd0167490 con 0x7f5dd0072630 2024-04-16T16:16:36.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.271+0000 7f5dc7400640 1 -- 172.21.15.45:0/720110501 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f5dc001a070 con 0x7f5dd0072630 2024-04-16T16:16:36.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.272+0000 7f5dc7400640 1 --2- 172.21.15.45:0/720110501 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5db006cb60 0x7f5db006f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:36.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.272+0000 7f5dc7400640 1 -- 172.21.15.45:0/720110501 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 4857+0+0 (secure 0 0 0) 0x7f5dc008b1f0 con 0x7f5dd0072630 2024-04-16T16:16:36.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.273+0000 7f5dcea00640 1 --2- 172.21.15.45:0/720110501 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5db006cb60 0x7f5db006f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:36.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.274+0000 7f5dcea00640 1 --2- 172.21.15.45:0/720110501 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5db006cb60 0x7f5db006f020 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc034000 tx=0x7f5dbc02f000 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:36.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.277+0000 7f5dc7400640 1 -- 172.21.15.45:0/720110501 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5dd0167490 con 0x7f5dd0072630 2024-04-16T16:16:36.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.482+0000 7f5dd4c00640 1 -- 172.21.15.45:0/720110501 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5dd0075780 con 0x7f5dd0072630 2024-04-16T16:16:36.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.482+0000 7f5dc7400640 1 -- 172.21.15.45:0/720110501 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v68) ==== 93+0+10 (secure 0 0 0) 0x7f5dc0059570 con 0x7f5dd0072630 2024-04-16T16:16:36.481 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:16:36.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.485+0000 7f5dd4c00640 1 -- 172.21.15.45:0/720110501 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5db006cb60 msgr2=0x7f5db006f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:36.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.485+0000 7f5dd4c00640 1 --2- 172.21.15.45:0/720110501 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5db006cb60 0x7f5db006f020 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc034000 tx=0x7f5dbc02f000 comp rx=0 tx=0).stop 2024-04-16T16:16:36.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.485+0000 7f5dd4c00640 1 -- 172.21.15.45:0/720110501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd0072630 msgr2=0x7f5dd01450b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:36.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.485+0000 7f5dd4c00640 1 --2- 172.21.15.45:0/720110501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd0072630 0x7f5dd01450b0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f5dc000c730 tx=0x7f5dc000cbf0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.485+0000 7f5dd4c00640 1 -- 172.21.15.45:0/720110501 shutdown_connections 2024-04-16T16:16:36.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.485+0000 7f5dd4c00640 1 --2- 172.21.15.45:0/720110501 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f5db006cb60 0x7f5db006f020 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.485+0000 7f5dd4c00640 1 --2- 172.21.15.45:0/720110501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dd0163360 0x7f5dd01596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.485+0000 7f5dd4c00640 1 --2- 172.21.15.45:0/720110501 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5dd015d400 0x7f5dd0159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.485+0000 7f5dd4c00640 1 --2- 172.21.15.45:0/720110501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd0072630 0x7f5dd01450b0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.485+0000 7f5dd4c00640 1 -- 172.21.15.45:0/720110501 >> 172.21.15.45:0/720110501 conn(0x7f5dd0067e00 msgr2=0x7f5dd0165720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:36.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.485+0000 7f5dd4c00640 1 -- 172.21.15.45:0/720110501 shutdown_connections 2024-04-16T16:16:36.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.485+0000 7f5dd4c00640 1 -- 172.21.15.45:0/720110501 wait complete. 2024-04-16T16:16:36.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-16T16:16:36.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.726+0000 7fe4ecc00640 1 Processor -- start 2024-04-16T16:16:36.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.726+0000 7fe4ecc00640 1 -- start start 2024-04-16T16:16:36.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.727+0000 7fe4ecc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4e80d52d0 0x7fe4e806ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:36.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.727+0000 7fe4ecc00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4e80d3080 con 0x7fe4e8161e90 2024-04-16T16:16:36.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.728+0000 7fe4ecc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4e80d3360 con 0x7fe4e8162270 2024-04-16T16:16:36.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.728+0000 7fe4ecc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe4e80d2e30 con 0x7fe4e80d52d0 2024-04-16T16:16:36.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.728+0000 7fe4e7400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe4e8161e90 0x7fe4e80cf7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42372/0 (socket says 172.21.15.45:42372) 2024-04-16T16:16:36.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.728+0000 7fe4e7400640 1 -- 172.21.15.45:0/1724222581 learned_addr learned my addr 172.21.15.45:0/1724222581 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:36.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.728+0000 7fe4e7e00640 1 --2- 172.21.15.45:0/1724222581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4e80d52d0 0x7fe4e806ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:36.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.728+0000 7fe4e6000640 1 -- 172.21.15.45:0/1724222581 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 952828451 0 0) 0x7fe4e80d3080 con 0x7fe4e8161e90 2024-04-16T16:16:36.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.728+0000 7fe4e6000640 1 -- 172.21.15.45:0/1724222581 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4cc003520 con 0x7fe4e8161e90 2024-04-16T16:16:36.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.729+0000 7fe4e6000640 1 -- 172.21.15.45:0/1724222581 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3917023540 0 0) 0x7fe4cc003520 con 0x7fe4e8161e90 2024-04-16T16:16:36.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.729+0000 7fe4e6000640 1 -- 172.21.15.45:0/1724222581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4e80d52d0 msgr2=0x7fe4e806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.729+0000 7fe4e6000640 1 --2- 172.21.15.45:0/1724222581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4e80d52d0 0x7fe4e806ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.729+0000 7fe4e6000640 1 -- 172.21.15.45:0/1724222581 >> v1:172.21.15.103:6789/0 conn(0x7fe4e8162270 legacy=0x7fe4e80d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.729+0000 7fe4e6000640 1 -- 172.21.15.45:0/1724222581 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e80d3640 con 0x7fe4e8161e90 2024-04-16T16:16:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.729+0000 7fe4e6000640 1 -- 172.21.15.45:0/1724222581 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fe4d80019f0 con 0x7fe4e8161e90 2024-04-16T16:16:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.729+0000 7fe4e6000640 1 -- 172.21.15.45:0/1724222581 >> v1:172.21.15.45:6789/0 conn(0x7fe4e8161e90 legacy=0x7fe4e80cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.729+0000 7fe4e6000640 1 --2- 172.21.15.45:0/1724222581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4e80d52d0 0x7fe4cc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.729+0000 7fe4e6000640 1 --2- 172.21.15.45:0/1724222581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4cc003f60 0x7fe4cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.729+0000 7fe4e6000640 1 --2- 172.21.15.45:0/1724222581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4cc004d40 0x7fe4cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.729+0000 7fe4e6000640 1 -- 172.21.15.45:0/1724222581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e80d2e30 con 0x7fe4cc003f60 2024-04-16T16:16:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.729+0000 7fe4e6000640 1 -- 172.21.15.45:0/1724222581 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe4cc00bb30 con 0x7fe4cc004d40 2024-04-16T16:16:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.729+0000 7fe4e6000640 1 -- 172.21.15.45:0/1724222581 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe4cc00bd30 con 0x7fe4e80d52d0 2024-04-16T16:16:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.730+0000 7fe4e6a00640 1 --2- 172.21.15.45:0/1724222581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4cc003f60 0x7fe4cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.730+0000 7fe4e7400640 1 --2- 172.21.15.45:0/1724222581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4e80d52d0 0x7fe4cc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.730+0000 7fe4e7e00640 1 --2- 172.21.15.45:0/1724222581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4cc004d40 0x7fe4cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:36.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.730+0000 7fe4e6a00640 1 -- 172.21.15.45:0/1724222581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4e80d52d0 msgr2=0x7fe4cc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:36.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.730+0000 7fe4e6a00640 1 --2- 172.21.15.45:0/1724222581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4e80d52d0 0x7fe4cc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.730+0000 7fe4e6a00640 1 -- 172.21.15.45:0/1724222581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4cc004d40 msgr2=0x7fe4cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:36.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.730+0000 7fe4e6a00640 1 --2- 172.21.15.45:0/1724222581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4cc004d40 0x7fe4cc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.730+0000 7fe4e6a00640 1 -- 172.21.15.45:0/1724222581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4cc003520 con 0x7fe4cc003f60 2024-04-16T16:16:36.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.730+0000 7fe4e7e00640 1 --2- 172.21.15.45:0/1724222581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4cc004d40 0x7fe4cc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.731+0000 7fe4e6a00640 1 --2- 172.21.15.45:0/1724222581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4cc003f60 0x7fe4cc004800 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0001720 tx=0x7fe4cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804612adefd27dc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.732+0000 7fe4e6000640 1 -- 172.21.15.45:0/1724222581 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0001370 con 0x7fe4cc003f60 2024-04-16T16:16:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.732+0000 7fe4e6000640 1 -- 172.21.15.45:0/1724222581 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4d00052d0 con 0x7fe4cc003f60 2024-04-16T16:16:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.732+0000 7fe4e6000640 1 -- 172.21.15.45:0/1724222581 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0003830 con 0x7fe4cc003f60 2024-04-16T16:16:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.732+0000 7fe4ecc00640 1 -- 172.21.15.45:0/1724222581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4cc003f60 msgr2=0x7fe4cc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.732+0000 7fe4ecc00640 1 --2- 172.21.15.45:0/1724222581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4cc003f60 0x7fe4cc004800 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0001720 tx=0x7fe4cc005830 comp rx=0 tx=0).stop 2024-04-16T16:16:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.732+0000 7fe4e7e00640 1 -- 172.21.15.45:0/1724222581 reap_dead start 2024-04-16T16:16:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.732+0000 7fe4ecc00640 1 -- 172.21.15.45:0/1724222581 shutdown_connections 2024-04-16T16:16:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.732+0000 7fe4ecc00640 1 -- 172.21.15.45:0/1724222581 >> 172.21.15.45:0/1724222581 conn(0x7fe4e80d1940 msgr2=0x7fe4e80d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.732+0000 7fe4ecc00640 1 -- 172.21.15.45:0/1724222581 shutdown_connections 2024-04-16T16:16:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.732+0000 7fe4ecc00640 1 -- 172.21.15.45:0/1724222581 wait complete. 2024-04-16T16:16:36.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.734+0000 7fe4ecc00640 1 Processor -- start 2024-04-16T16:16:36.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.734+0000 7fe4ecc00640 1 -- start start 2024-04-16T16:16:36.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.735+0000 7fe4ecc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4cc003f60 0x7fe4e8155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:36.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.735+0000 7fe4ecc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e80ac0e0 0x7fe4e80aca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:36.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.735+0000 7fe4ecc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4e80acfd0 0x7fe4e80cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:36.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.735+0000 7fe4ecc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e80d46d0 con 0x7fe4e80ac0e0 2024-04-16T16:16:36.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.735+0000 7fe4ecc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e80d4550 con 0x7fe4e80acfd0 2024-04-16T16:16:36.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.735+0000 7fe4ecc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe4e806ce40 con 0x7fe4cc003f60 2024-04-16T16:16:36.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.736+0000 7fe4e6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e80ac0e0 0x7fe4e80aca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:36.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.736+0000 7fe4e6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e80ac0e0 0x7fe4e80aca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45056/0 (socket says 172.21.15.45:45056) 2024-04-16T16:16:36.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.736+0000 7fe4e6a00640 1 -- 172.21.15.45:0/636521788 learned_addr learned my addr 172.21.15.45:0/636521788 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:36.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.736+0000 7fe4e7e00640 1 --2- 172.21.15.45:0/636521788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4e80acfd0 0x7fe4e80cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:36.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.736+0000 7fe4e7400640 1 --2- 172.21.15.45:0/636521788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4cc003f60 0x7fe4e8155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:36.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.736+0000 7fe4e6a00640 1 -- 172.21.15.45:0/636521788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4cc003f60 msgr2=0x7fe4e8155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:36.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.736+0000 7fe4e6a00640 1 --2- 172.21.15.45:0/636521788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4cc003f60 0x7fe4e8155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.736+0000 7fe4e6a00640 1 -- 172.21.15.45:0/636521788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4e80acfd0 msgr2=0x7fe4e80cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:36.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.736+0000 7fe4e6a00640 1 --2- 172.21.15.45:0/636521788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4e80acfd0 0x7fe4e80cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.736+0000 7fe4e6a00640 1 -- 172.21.15.45:0/636521788 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e81f53a0 con 0x7fe4e80ac0e0 2024-04-16T16:16:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.737+0000 7fe4e6a00640 1 --2- 172.21.15.45:0/636521788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e80ac0e0 0x7fe4e80aca90 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0005040 tx=0x7fe4d0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.737+0000 7fe4df400640 1 -- 172.21.15.45:0/636521788 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0041070 con 0x7fe4e80ac0e0 2024-04-16T16:16:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.737+0000 7fe4df400640 1 -- 172.21.15.45:0/636521788 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4d0005480 con 0x7fe4e80ac0e0 2024-04-16T16:16:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.737+0000 7fe4ecc00640 1 -- 172.21.15.45:0/636521788 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4e80ab320 con 0x7fe4e80ac0e0 2024-04-16T16:16:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.737+0000 7fe4df400640 1 -- 172.21.15.45:0/636521788 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0007a30 con 0x7fe4e80ac0e0 2024-04-16T16:16:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.737+0000 7fe4ecc00640 1 -- 172.21.15.45:0/636521788 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4e80ab8e0 con 0x7fe4e80ac0e0 2024-04-16T16:16:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.738+0000 7fe4ecc00640 1 -- 172.21.15.45:0/636521788 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4e80ad3e0 con 0x7fe4e80ac0e0 2024-04-16T16:16:36.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.740+0000 7fe4df400640 1 -- 172.21.15.45:0/636521788 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fe4d0044070 con 0x7fe4e80ac0e0 2024-04-16T16:16:36.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.741+0000 7fe4df400640 1 --2- 172.21.15.45:0/636521788 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe4c406cb60 0x7fe4c406f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:36.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.741+0000 7fe4df400640 1 -- 172.21.15.45:0/636521788 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 4857+0+0 (secure 0 0 0) 0x7fe4d00adb40 con 0x7fe4e80ac0e0 2024-04-16T16:16:36.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.741+0000 7fe4e7400640 1 --2- 172.21.15.45:0/636521788 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe4c406cb60 0x7fe4c406f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:36.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.743+0000 7fe4e7400640 1 --2- 172.21.15.45:0/636521788 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe4c406cb60 0x7fe4c406f020 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8002050 tx=0x7fe4d8009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:36.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.745+0000 7fe4df400640 1 -- 172.21.15.45:0/636521788 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe4e80ad3e0 con 0x7fe4e80ac0e0 2024-04-16T16:16:36.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.954+0000 7fe4ecc00640 1 -- 172.21.15.45:0/636521788 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe4e8075780 con 0x7fe4e80ac0e0 2024-04-16T16:16:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.955+0000 7fe4df400640 1 -- 172.21.15.45:0/636521788 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v68) ==== 84+0+65 (secure 0 0 0) 0x7fe4d00052a0 con 0x7fe4e80ac0e0 2024-04-16T16:16:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:36.954 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2024-04-16T16:16:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:16:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2024-04-16T16:16:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2024-04-16T16:16:36.955 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:16:36.955 INFO:teuthology.orchestra.run.smithi045.stdout:] 2024-04-16T16:16:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.958+0000 7fe4ecc00640 1 -- 172.21.15.45:0/636521788 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe4c406cb60 msgr2=0x7fe4c406f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.958+0000 7fe4ecc00640 1 --2- 172.21.15.45:0/636521788 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe4c406cb60 0x7fe4c406f020 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8002050 tx=0x7fe4d8009040 comp rx=0 tx=0).stop 2024-04-16T16:16:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.958+0000 7fe4ecc00640 1 -- 172.21.15.45:0/636521788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e80ac0e0 msgr2=0x7fe4e80aca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.958+0000 7fe4ecc00640 1 --2- 172.21.15.45:0/636521788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e80ac0e0 0x7fe4e80aca90 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0005040 tx=0x7fe4d0001120 comp rx=0 tx=0).stop 2024-04-16T16:16:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.958+0000 7fe4ecc00640 1 -- 172.21.15.45:0/636521788 shutdown_connections 2024-04-16T16:16:36.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.958+0000 7fe4ecc00640 1 --2- 172.21.15.45:0/636521788 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe4c406cb60 0x7fe4c406f020 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.958+0000 7fe4ecc00640 1 --2- 172.21.15.45:0/636521788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4e80acfd0 0x7fe4e80cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.958+0000 7fe4ecc00640 1 --2- 172.21.15.45:0/636521788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e80ac0e0 0x7fe4e80aca90 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.958+0000 7fe4ecc00640 1 --2- 172.21.15.45:0/636521788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4cc003f60 0x7fe4e8155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:36.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.958+0000 7fe4ecc00640 1 -- 172.21.15.45:0/636521788 >> 172.21.15.45:0/636521788 conn(0x7fe4e80d1940 msgr2=0x7fe4e80b9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:36.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.958+0000 7fe4ecc00640 1 -- 172.21.15.45:0/636521788 shutdown_connections 2024-04-16T16:16:36.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:36.958+0000 7fe4ecc00640 1 -- 172.21.15.45:0/636521788 wait complete. 2024-04-16T16:16:36.977 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-16T16:16:36.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-16T16:16:37.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.209+0000 7f2f23000640 1 Processor -- start 2024-04-16T16:16:37.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.209+0000 7f2f23000640 1 -- start start 2024-04-16T16:16:37.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.210+0000 7f2f23000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2f1c0ceab0 0x7f2f1c0b3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:37.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.210+0000 7f2f23000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f1c06d450 con 0x7f2f1c09b2d0 2024-04-16T16:16:37.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.210+0000 7f2f23000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f1c0d0980 con 0x7f2f1c0ce6d0 2024-04-16T16:16:37.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.210+0000 7f2f23000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2f1c0cceb0 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.211+0000 7f2f1b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2f1c0ceab0 0x7f2f1c0b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:37.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.211+0000 7f2f1b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2f1c0ceab0 0x7f2f1c0b3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45072/0 (socket says 172.21.15.45:45072) 2024-04-16T16:16:37.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.211+0000 7f2f1b400640 1 -- 172.21.15.45:0/1338518667 learned_addr learned my addr 172.21.15.45:0/1338518667 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:37.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.211+0000 7f2f1b400640 1 -- 172.21.15.45:0/1338518667 >> v1:172.21.15.103:6789/0 conn(0x7f2f1c0ce6d0 legacy=0x7f2f1c09abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:37.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.211+0000 7f2f1b400640 1 -- 172.21.15.45:0/1338518667 >> v1:172.21.15.45:6789/0 conn(0x7f2f1c09b2d0 legacy=0x7f2f1c0d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:37.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.211+0000 7f2f1b400640 1 -- 172.21.15.45:0/1338518667 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1c09bf10 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.213+0000 7f2f1b400640 1 --2- 172.21.15.45:0/1338518667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c0ceab0 0x7f2f1c0b3370 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009870 tx=0x7f2f0402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d262de7769ce72 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:37.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.213+0000 7f2f1aa00640 1 -- 172.21.15.45:0/1338518667 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f04034070 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.213+0000 7f2f1aa00640 1 -- 172.21.15.45:0/1338518667 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f04030850 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.213+0000 7f2f1aa00640 1 -- 172.21.15.45:0/1338518667 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f04030c20 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.214+0000 7f2f23000640 1 -- 172.21.15.45:0/1338518667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c0ceab0 msgr2=0x7f2f1c0b3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:37.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.214+0000 7f2f23000640 1 --2- 172.21.15.45:0/1338518667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c0ceab0 0x7f2f1c0b3370 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009870 tx=0x7f2f0402b990 comp rx=0 tx=0).stop 2024-04-16T16:16:37.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.214+0000 7f2f23000640 1 -- 172.21.15.45:0/1338518667 shutdown_connections 2024-04-16T16:16:37.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.214+0000 7f2f23000640 1 --2- 172.21.15.45:0/1338518667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c0ceab0 0x7f2f1c0b3370 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:37.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.214+0000 7f2f23000640 1 -- 172.21.15.45:0/1338518667 >> 172.21.15.45:0/1338518667 conn(0x7f2f1c0b4350 msgr2=0x7f2f1c0b4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:37.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.214+0000 7f2f23000640 1 -- 172.21.15.45:0/1338518667 shutdown_connections 2024-04-16T16:16:37.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.214+0000 7f2f23000640 1 -- 172.21.15.45:0/1338518667 wait complete. 2024-04-16T16:16:37.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.216+0000 7f2f23000640 1 Processor -- start 2024-04-16T16:16:37.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.216+0000 7f2f23000640 1 -- start start 2024-04-16T16:16:37.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.217+0000 7f2f23000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f1c09b2d0 0x7f2f1c0b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:37.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.217+0000 7f2f23000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f1c0ce6d0 0x7f2f1c0affb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:37.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.217+0000 7f2f23000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c0ceab0 0x7f2f1c09fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:37.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.217+0000 7f2f23000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c06dd40 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.217+0000 7f2f23000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c0b40b0 con 0x7f2f1c09b2d0 2024-04-16T16:16:37.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.217+0000 7f2f23000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2f1c06d6f0 con 0x7f2f1c0ce6d0 2024-04-16T16:16:37.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.218+0000 7f2f21000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c0ceab0 0x7f2f1c09fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:37.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.218+0000 7f2f21000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c0ceab0 0x7f2f1c09fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45084/0 (socket says 172.21.15.45:45084) 2024-04-16T16:16:37.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.218+0000 7f2f21000640 1 -- 172.21.15.45:0/1249340840 learned_addr learned my addr 172.21.15.45:0/1249340840 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:37.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.218+0000 7f2f1be00640 1 --2- 172.21.15.45:0/1249340840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f1c09b2d0 0x7f2f1c0b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:37.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.218+0000 7f2f1b400640 1 --2- 172.21.15.45:0/1249340840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f1c0ce6d0 0x7f2f1c0affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:37.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.218+0000 7f2f21000640 1 -- 172.21.15.45:0/1249340840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f1c0ce6d0 msgr2=0x7f2f1c0affb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:37.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.218+0000 7f2f21000640 1 --2- 172.21.15.45:0/1249340840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f1c0ce6d0 0x7f2f1c0affb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:37.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.218+0000 7f2f21000640 1 -- 172.21.15.45:0/1249340840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f1c09b2d0 msgr2=0x7f2f1c0b1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:37.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.218+0000 7f2f21000640 1 --2- 172.21.15.45:0/1249340840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f1c09b2d0 0x7f2f1c0b1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:37.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.218+0000 7f2f21000640 1 -- 172.21.15.45:0/1249340840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1c09e0a0 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.218+0000 7f2f1b400640 1 --2- 172.21.15.45:0/1249340840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f1c0ce6d0 0x7f2f1c0affb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:37.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.218+0000 7f2f21000640 1 --2- 172.21.15.45:0/1249340840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c0ceab0 0x7f2f1c09fe80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f2f1400c730 tx=0x7f2f1400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:37.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.219+0000 7f2f13e00640 1 -- 172.21.15.45:0/1249340840 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f14003b90 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.219+0000 7f2f13e00640 1 -- 172.21.15.45:0/1249340840 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f14003d30 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.219+0000 7f2f13e00640 1 -- 172.21.15.45:0/1249340840 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f14005310 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.219+0000 7f2f23000640 1 -- 172.21.15.45:0/1249340840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f1c09e380 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.219+0000 7f2f23000640 1 -- 172.21.15.45:0/1249340840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f1c0b2310 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.220+0000 7f2f23000640 1 -- 172.21.15.45:0/1249340840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f1c075780 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.221+0000 7f2f13e00640 1 -- 172.21.15.45:0/1249340840 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f2f140056c0 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.222+0000 7f2f13e00640 1 --2- 172.21.15.45:0/1249340840 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f2ef806cb30 0x7f2ef806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:37.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.222+0000 7f2f13e00640 1 -- 172.21.15.45:0/1249340840 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 4857+0+0 (secure 0 0 0) 0x7f2f1408bd20 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.223+0000 7f2f1be00640 1 --2- 172.21.15.45:0/1249340840 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f2ef806cb30 0x7f2ef806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:37.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.224+0000 7f2f1be00640 1 --2- 172.21.15.45:0/1249340840 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f2ef806cb30 0x7f2ef806eff0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c007fd0 tx=0x7f2f0c009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:37.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.227+0000 7f2f13e00640 1 -- 172.21.15.45:0/1249340840 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2f1405a0a0 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.434+0000 7f2f23000640 1 -- 172.21.15.45:0/1249340840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f2f1c0cdd80 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:37.997+0000 7f2f13e00640 1 -- 172.21.15.45:0/1249340840 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v69) ==== 138+0+0 (secure 0 0 0) 0x7f2f140599a0 con 0x7f2f1c0ceab0 2024-04-16T16:16:37.996 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2024-04-16T16:16:37.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.000+0000 7f2f23000640 1 -- 172.21.15.45:0/1249340840 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f2ef806cb30 msgr2=0x7f2ef806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:38.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.000+0000 7f2f23000640 1 --2- 172.21.15.45:0/1249340840 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f2ef806cb30 0x7f2ef806eff0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c007fd0 tx=0x7f2f0c009040 comp rx=0 tx=0).stop 2024-04-16T16:16:38.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.000+0000 7f2f23000640 1 -- 172.21.15.45:0/1249340840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c0ceab0 msgr2=0x7f2f1c09fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:38.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.000+0000 7f2f23000640 1 --2- 172.21.15.45:0/1249340840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c0ceab0 0x7f2f1c09fe80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f2f1400c730 tx=0x7f2f1400cbf0 comp rx=0 tx=0).stop 2024-04-16T16:16:38.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.001+0000 7f2f23000640 1 -- 172.21.15.45:0/1249340840 shutdown_connections 2024-04-16T16:16:38.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.001+0000 7f2f23000640 1 --2- 172.21.15.45:0/1249340840 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f2ef806cb30 0x7f2ef806eff0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:38.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.001+0000 7f2f23000640 1 --2- 172.21.15.45:0/1249340840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c0ceab0 0x7f2f1c09fe80 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:38.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.001+0000 7f2f23000640 1 --2- 172.21.15.45:0/1249340840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f1c0ce6d0 0x7f2f1c0affb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:38.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.001+0000 7f2f23000640 1 --2- 172.21.15.45:0/1249340840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f1c09b2d0 0x7f2f1c0b1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:38.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.001+0000 7f2f23000640 1 -- 172.21.15.45:0/1249340840 >> 172.21.15.45:0/1249340840 conn(0x7f2f1c0b4350 msgr2=0x7f2f1c0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:38.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.001+0000 7f2f23000640 1 -- 172.21.15.45:0/1249340840 shutdown_connections 2024-04-16T16:16:38.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.001+0000 7f2f23000640 1 -- 172.21.15.45:0/1249340840 wait complete. 2024-04-16T16:16:38.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-16T16:16:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.245+0000 7fe90e400640 1 Processor -- start 2024-04-16T16:16:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.246+0000 7fe90e400640 1 -- start start 2024-04-16T16:16:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.247+0000 7fe90e400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90815ec80 0x7fe9080d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.247+0000 7fe90e400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe90815f060 0x7fe9080b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.247+0000 7fe90e400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9080b7ca0 con 0x7fe9080ba190 2024-04-16T16:16:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.247+0000 7fe90e400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe9080d3220 con 0x7fe90815f060 2024-04-16T16:16:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.247+0000 7fe90e400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe9080d3450 con 0x7fe90815ec80 2024-04-16T16:16:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.248+0000 7fe906a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe90815f060 0x7fe9080b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.248+0000 7fe906a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe90815f060 0x7fe9080b64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45092/0 (socket says 172.21.15.45:45092) 2024-04-16T16:16:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.248+0000 7fe906a00640 1 -- 172.21.15.45:0/3908156292 learned_addr learned my addr 172.21.15.45:0/3908156292 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.248+0000 7fe907400640 1 --2- 172.21.15.45:0/3908156292 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90815ec80 0x7fe9080d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.248+0000 7fe906a00640 1 -- 172.21.15.45:0/3908156292 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90815ec80 msgr2=0x7fe9080d2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.248+0000 7fe906a00640 1 --2- 172.21.15.45:0/3908156292 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90815ec80 0x7fe9080d2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.248+0000 7fe906a00640 1 -- 172.21.15.45:0/3908156292 >> v1:172.21.15.103:6789/0 conn(0x7fe9080ba190 legacy=0x7fe90806cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.248+0000 7fe906a00640 1 -- 172.21.15.45:0/3908156292 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9080d3730 con 0x7fe90815f060 2024-04-16T16:16:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.250+0000 7fe906a00640 1 --2- 172.21.15.45:0/3908156292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90815f060 0x7fe9080b64f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4009870 tx=0x7fe8f402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24aece760e5b57e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:38.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.250+0000 7fe906000640 1 -- 172.21.15.45:0/3908156292 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f4034070 con 0x7fe90815f060 2024-04-16T16:16:38.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.250+0000 7fe906000640 1 -- 172.21.15.45:0/3908156292 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8f4030840 con 0x7fe90815f060 2024-04-16T16:16:38.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.250+0000 7fe906000640 1 -- 172.21.15.45:0/3908156292 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f4030c10 con 0x7fe90815f060 2024-04-16T16:16:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.251+0000 7fe90e400640 1 -- 172.21.15.45:0/3908156292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90815f060 msgr2=0x7fe9080b64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.251+0000 7fe90e400640 1 --2- 172.21.15.45:0/3908156292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90815f060 0x7fe9080b64f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4009870 tx=0x7fe8f402b980 comp rx=0 tx=0).stop 2024-04-16T16:16:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.251+0000 7fe90e400640 1 -- 172.21.15.45:0/3908156292 shutdown_connections 2024-04-16T16:16:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.251+0000 7fe90e400640 1 --2- 172.21.15.45:0/3908156292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90815f060 0x7fe9080b64f0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.251+0000 7fe90e400640 1 --2- 172.21.15.45:0/3908156292 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90815ec80 0x7fe9080d2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.251+0000 7fe90e400640 1 -- 172.21.15.45:0/3908156292 >> 172.21.15.45:0/3908156292 conn(0x7fe9080d16d0 msgr2=0x7fe9080d1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.251+0000 7fe90e400640 1 -- 172.21.15.45:0/3908156292 shutdown_connections 2024-04-16T16:16:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.251+0000 7fe90e400640 1 -- 172.21.15.45:0/3908156292 wait complete. 2024-04-16T16:16:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.253+0000 7fe90e400640 1 Processor -- start 2024-04-16T16:16:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.253+0000 7fe90e400640 1 -- start start 2024-04-16T16:16:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.254+0000 7fe90e400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9080ba190 0x7fe9081f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.254+0000 7fe90e400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90815ec80 0x7fe908150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.254+0000 7fe90e400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080c85a0 0x7fe9080cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.254+0000 7fe90e400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe9080d4160 con 0x7fe9080c85a0 2024-04-16T16:16:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.254+0000 7fe90e400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe9080d3cf0 con 0x7fe9080ba190 2024-04-16T16:16:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.254+0000 7fe90e400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe9080d42e0 con 0x7fe90815ec80 2024-04-16T16:16:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.255+0000 7fe907e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080c85a0 0x7fe9080cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.255+0000 7fe907e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080c85a0 0x7fe9080cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45096/0 (socket says 172.21.15.45:45096) 2024-04-16T16:16:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.255+0000 7fe907e00640 1 -- 172.21.15.45:0/2443712903 learned_addr learned my addr 172.21.15.45:0/2443712903 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.255+0000 7fe907400640 1 --2- 172.21.15.45:0/2443712903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9080ba190 0x7fe9081f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.255+0000 7fe906a00640 1 --2- 172.21.15.45:0/2443712903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90815ec80 0x7fe908150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.255+0000 7fe907e00640 1 -- 172.21.15.45:0/2443712903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90815ec80 msgr2=0x7fe908150da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.255+0000 7fe907e00640 1 --2- 172.21.15.45:0/2443712903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90815ec80 0x7fe908150da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.255+0000 7fe907e00640 1 -- 172.21.15.45:0/2443712903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9080ba190 msgr2=0x7fe9081f44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.255+0000 7fe907e00640 1 --2- 172.21.15.45:0/2443712903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9080ba190 0x7fe9081f44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.255+0000 7fe907e00640 1 -- 172.21.15.45:0/2443712903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9080cb6f0 con 0x7fe9080c85a0 2024-04-16T16:16:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.255+0000 7fe906a00640 1 --2- 172.21.15.45:0/2443712903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90815ec80 0x7fe908150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:16:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.256+0000 7fe907e00640 1 --2- 172.21.15.45:0/2443712903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080c85a0 0x7fe9080cd130 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe90000c700 tx=0x7fe90000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.256+0000 7fe8ff400640 1 -- 172.21.15.45:0/2443712903 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe900016020 con 0x7fe9080c85a0 2024-04-16T16:16:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.256+0000 7fe8ff400640 1 -- 172.21.15.45:0/2443712903 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe90000aa50 con 0x7fe9080c85a0 2024-04-16T16:16:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.256+0000 7fe8ff400640 1 -- 172.21.15.45:0/2443712903 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90000ce10 con 0x7fe9080c85a0 2024-04-16T16:16:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.256+0000 7fe90e400640 1 -- 172.21.15.45:0/2443712903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe908160b10 con 0x7fe9080c85a0 2024-04-16T16:16:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.256+0000 7fe90e400640 1 -- 172.21.15.45:0/2443712903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9081610d0 con 0x7fe9080c85a0 2024-04-16T16:16:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.257+0000 7fe90e400640 1 -- 172.21.15.45:0/2443712903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe908075780 con 0x7fe9080c85a0 2024-04-16T16:16:38.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.258+0000 7fe8ff400640 1 -- 172.21.15.45:0/2443712903 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fe900019020 con 0x7fe9080c85a0 2024-04-16T16:16:38.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.259+0000 7fe8ff400640 1 --2- 172.21.15.45:0/2443712903 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe8e806cb60 0x7fe8e806f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:38.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.259+0000 7fe8ff400640 1 -- 172.21.15.45:0/2443712903 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5227+0+0 (secure 0 0 0) 0x7fe90008c1a0 con 0x7fe9080c85a0 2024-04-16T16:16:38.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.260+0000 7fe907400640 1 --2- 172.21.15.45:0/2443712903 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe8e806cb60 0x7fe8e806f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:38.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.261+0000 7fe907400640 1 --2- 172.21.15.45:0/2443712903 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe8e806cb60 0x7fe8e806f020 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8006fd0 tx=0x7fe8f8008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:38.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.264+0000 7fe8ff400640 1 -- 172.21.15.45:0/2443712903 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe900020020 con 0x7fe9080c85a0 2024-04-16T16:16:38.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:38.469+0000 7fe90e400640 1 -- 172.21.15.45:0/2443712903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe9081512e0 con 0x7fe9080c85a0 2024-04-16T16:16:39.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.016+0000 7fe8ff400640 1 -- 172.21.15.45:0/2443712903 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v70) ==== 170+0+0 (secure 0 0 0) 0x7fe90005a3b0 con 0x7fe9080c85a0 2024-04-16T16:16:39.015 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2024-04-16T16:16:39.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.020+0000 7fe90e400640 1 -- 172.21.15.45:0/2443712903 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe8e806cb60 msgr2=0x7fe8e806f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:39.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.020+0000 7fe90e400640 1 --2- 172.21.15.45:0/2443712903 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe8e806cb60 0x7fe8e806f020 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8006fd0 tx=0x7fe8f8008040 comp rx=0 tx=0).stop 2024-04-16T16:16:39.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.020+0000 7fe90e400640 1 -- 172.21.15.45:0/2443712903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080c85a0 msgr2=0x7fe9080cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:39.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.020+0000 7fe90e400640 1 --2- 172.21.15.45:0/2443712903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080c85a0 0x7fe9080cd130 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe90000c700 tx=0x7fe90000cbc0 comp rx=0 tx=0).stop 2024-04-16T16:16:39.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.021+0000 7fe90e400640 1 -- 172.21.15.45:0/2443712903 shutdown_connections 2024-04-16T16:16:39.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.021+0000 7fe90e400640 1 --2- 172.21.15.45:0/2443712903 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe8e806cb60 0x7fe8e806f020 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:39.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.022+0000 7fe90e400640 1 --2- 172.21.15.45:0/2443712903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080c85a0 0x7fe9080cd130 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:39.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.022+0000 7fe90e400640 1 --2- 172.21.15.45:0/2443712903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90815ec80 0x7fe908150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:39.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.022+0000 7fe90e400640 1 --2- 172.21.15.45:0/2443712903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9080ba190 0x7fe9081f44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:39.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.022+0000 7fe90e400640 1 -- 172.21.15.45:0/2443712903 >> 172.21.15.45:0/2443712903 conn(0x7fe9080d16d0 msgr2=0x7fe9080b8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:39.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.023+0000 7fe90e400640 1 -- 172.21.15.45:0/2443712903 shutdown_connections 2024-04-16T16:16:39.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.023+0000 7fe90e400640 1 -- 172.21.15.45:0/2443712903 wait complete. 2024-04-16T16:16:39.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-16T16:16:39.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.250+0000 7fc89d600640 1 Processor -- start 2024-04-16T16:16:39.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.250+0000 7fc89d600640 1 -- start start 2024-04-16T16:16:39.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.251+0000 7fc89d600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc8981622d0 0x7fc8980cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:39.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.251+0000 7fc89d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8980d1f10 0x7fc8980cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:39.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.251+0000 7fc89d600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8980d4640 con 0x7fc898162a50 2024-04-16T16:16:39.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.251+0000 7fc89d600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc8980d43f0 con 0x7fc8981622d0 2024-04-16T16:16:39.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.251+0000 7fc89d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc8980ba4c0 con 0x7fc8980d1f10 2024-04-16T16:16:39.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.251+0000 7fc896a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc8981622d0 0x7fc8980cf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:39.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.252+0000 7fc896a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc8981622d0 0x7fc8980cf6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45106/0 (socket says 172.21.15.45:45106) 2024-04-16T16:16:39.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.252+0000 7fc896a00640 1 -- 172.21.15.45:0/3236249238 learned_addr learned my addr 172.21.15.45:0/3236249238 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:39.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.252+0000 7fc896000640 1 --2- 172.21.15.45:0/3236249238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8980d1f10 0x7fc8980cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:39.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.252+0000 7fc896a00640 1 -- 172.21.15.45:0/3236249238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8980d1f10 msgr2=0x7fc8980cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:39.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.252+0000 7fc896a00640 1 --2- 172.21.15.45:0/3236249238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8980d1f10 0x7fc8980cfc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:39.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.252+0000 7fc896a00640 1 -- 172.21.15.45:0/3236249238 >> v1:172.21.15.103:6789/0 conn(0x7fc898162a50 legacy=0x7fc898162e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:39.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.252+0000 7fc896a00640 1 -- 172.21.15.45:0/3236249238 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8980b7b50 con 0x7fc8981622d0 2024-04-16T16:16:39.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.253+0000 7fc896a00640 1 --2- 172.21.15.45:0/3236249238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8981622d0 0x7fc8980cf6f0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc888009e50 tx=0x7fc88802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e801ba37f2d5ac2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:39.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.254+0000 7fc895600640 1 -- 172.21.15.45:0/3236249238 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88802ca70 con 0x7fc8981622d0 2024-04-16T16:16:39.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.254+0000 7fc895600640 1 -- 172.21.15.45:0/3236249238 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc88802cc10 con 0x7fc8981622d0 2024-04-16T16:16:39.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.254+0000 7fc895600640 1 -- 172.21.15.45:0/3236249238 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8880343f0 con 0x7fc8981622d0 2024-04-16T16:16:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.255+0000 7fc89d600640 1 -- 172.21.15.45:0/3236249238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8981622d0 msgr2=0x7fc8980cf6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.255+0000 7fc89d600640 1 --2- 172.21.15.45:0/3236249238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8981622d0 0x7fc8980cf6f0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc888009e50 tx=0x7fc88802bed0 comp rx=0 tx=0).stop 2024-04-16T16:16:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.255+0000 7fc89d600640 1 -- 172.21.15.45:0/3236249238 shutdown_connections 2024-04-16T16:16:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.255+0000 7fc89d600640 1 --2- 172.21.15.45:0/3236249238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8980d1f10 0x7fc8980cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.255+0000 7fc89d600640 1 --2- 172.21.15.45:0/3236249238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8981622d0 0x7fc8980cf6f0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.255+0000 7fc89d600640 1 -- 172.21.15.45:0/3236249238 >> 172.21.15.45:0/3236249238 conn(0x7fc89806d400 msgr2=0x7fc89806d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.255+0000 7fc89d600640 1 -- 172.21.15.45:0/3236249238 shutdown_connections 2024-04-16T16:16:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.255+0000 7fc89d600640 1 -- 172.21.15.45:0/3236249238 wait complete. 2024-04-16T16:16:39.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.256+0000 7fc89d600640 1 Processor -- start 2024-04-16T16:16:39.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.257+0000 7fc89d600640 1 -- start start 2024-04-16T16:16:39.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.257+0000 7fc89d600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8980d1f10 0x7fc89815ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.258+0000 7fc89d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8981622d0 0x7fc8980ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.258+0000 7fc89d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc898162a50 0x7fc89814a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.258+0000 7fc89d600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc8980b70e0 con 0x7fc8980d1f10 2024-04-16T16:16:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.258+0000 7fc89d600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc8980b6f60 con 0x7fc898162a50 2024-04-16T16:16:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.258+0000 7fc89d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc898163590 con 0x7fc8981622d0 2024-04-16T16:16:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.258+0000 7fc896a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8980d1f10 0x7fc89815ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.258+0000 7fc896a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8980d1f10 0x7fc89815ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45108/0 (socket says 172.21.15.45:45108) 2024-04-16T16:16:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.258+0000 7fc896a00640 1 -- 172.21.15.45:0/4017431289 learned_addr learned my addr 172.21.15.45:0/4017431289 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.258+0000 7fc896000640 1 --2- 172.21.15.45:0/4017431289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8981622d0 0x7fc8980ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.258+0000 7fc896a00640 1 -- 172.21.15.45:0/4017431289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8981622d0 msgr2=0x7fc8980ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.258+0000 7fc896a00640 1 --2- 172.21.15.45:0/4017431289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8981622d0 0x7fc8980ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.258+0000 7fc896a00640 1 -- 172.21.15.45:0/4017431289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc898162a50 msgr2=0x7fc89814a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.258+0000 7fc896a00640 1 --2- 172.21.15.45:0/4017431289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc898162a50 0x7fc89814a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.258+0000 7fc896a00640 1 -- 172.21.15.45:0/4017431289 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8980cd010 con 0x7fc8980d1f10 2024-04-16T16:16:39.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.259+0000 7fc896a00640 1 --2- 172.21.15.45:0/4017431289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8980d1f10 0x7fc89815ea30 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc888009e50 tx=0x7fc888004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:39.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.259+0000 7fc88ea00640 1 -- 172.21.15.45:0/4017431289 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc888043020 con 0x7fc8980d1f10 2024-04-16T16:16:39.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.259+0000 7fc88ea00640 1 -- 172.21.15.45:0/4017431289 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc88803e450 con 0x7fc8980d1f10 2024-04-16T16:16:39.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.259+0000 7fc88ea00640 1 -- 172.21.15.45:0/4017431289 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88803e840 con 0x7fc8980d1f10 2024-04-16T16:16:39.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.259+0000 7fc89d600640 1 -- 172.21.15.45:0/4017431289 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc89814ad00 con 0x7fc8980d1f10 2024-04-16T16:16:39.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.259+0000 7fc89d600640 1 -- 172.21.15.45:0/4017431289 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8980cd590 con 0x7fc8980d1f10 2024-04-16T16:16:39.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.261+0000 7fc89d600640 1 -- 172.21.15.45:0/4017431289 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc898075780 con 0x7fc8980d1f10 2024-04-16T16:16:39.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.262+0000 7fc88ea00640 1 -- 172.21.15.45:0/4017431289 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fc888046020 con 0x7fc8980d1f10 2024-04-16T16:16:39.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.263+0000 7fc88ea00640 1 --2- 172.21.15.45:0/4017431289 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc87806cb30 0x7fc87806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:39.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.263+0000 7fc88ea00640 1 -- 172.21.15.45:0/4017431289 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5609+0+0 (secure 0 0 0) 0x7fc8880ae880 con 0x7fc8980d1f10 2024-04-16T16:16:39.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.264+0000 7fc896000640 1 --2- 172.21.15.45:0/4017431289 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc87806cb30 0x7fc87806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:39.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.265+0000 7fc896000640 1 --2- 172.21.15.45:0/4017431289 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc87806cb30 0x7fc87806eff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc8981f8240 tx=0x7fc884008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:39.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.268+0000 7fc88ea00640 1 -- 172.21.15.45:0/4017431289 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc88804d030 con 0x7fc8980d1f10 2024-04-16T16:16:39.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:39.512+0000 7fc89d600640 1 -- 172.21.15.45:0/4017431289 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc8980b4db0 con 0x7fc8980d1f10 2024-04-16T16:16:40.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.047+0000 7fc88ea00640 1 -- 172.21.15.45:0/4017431289 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v47) ==== 174+0+0 (secure 0 0 0) 0x7fc88807c9a0 con 0x7fc8980d1f10 2024-04-16T16:16:40.050 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 10 and data pool 11 2024-04-16T16:16:40.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.057+0000 7fc89d600640 1 -- 172.21.15.45:0/4017431289 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc87806cb30 msgr2=0x7fc87806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:40.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.057+0000 7fc89d600640 1 --2- 172.21.15.45:0/4017431289 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc87806cb30 0x7fc87806eff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc8981f8240 tx=0x7fc884008040 comp rx=0 tx=0).stop 2024-04-16T16:16:40.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.057+0000 7fc89d600640 1 -- 172.21.15.45:0/4017431289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8980d1f10 msgr2=0x7fc89815ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:40.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.057+0000 7fc89d600640 1 --2- 172.21.15.45:0/4017431289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8980d1f10 0x7fc89815ea30 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc888009e50 tx=0x7fc888004a40 comp rx=0 tx=0).stop 2024-04-16T16:16:40.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.057+0000 7fc89d600640 1 -- 172.21.15.45:0/4017431289 shutdown_connections 2024-04-16T16:16:40.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.057+0000 7fc89d600640 1 --2- 172.21.15.45:0/4017431289 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc87806cb30 0x7fc87806eff0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.057+0000 7fc89d600640 1 --2- 172.21.15.45:0/4017431289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc898162a50 0x7fc89814a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.057+0000 7fc89d600640 1 --2- 172.21.15.45:0/4017431289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8981622d0 0x7fc8980ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.057+0000 7fc89d600640 1 --2- 172.21.15.45:0/4017431289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8980d1f10 0x7fc89815ea30 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.057+0000 7fc89d600640 1 -- 172.21.15.45:0/4017431289 >> 172.21.15.45:0/4017431289 conn(0x7fc89806d400 msgr2=0x7fc89806d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:40.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.057+0000 7fc89d600640 1 -- 172.21.15.45:0/4017431289 shutdown_connections 2024-04-16T16:16:40.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.057+0000 7fc89d600640 1 -- 172.21.15.45:0/4017431289 wait complete. 2024-04-16T16:16:40.090 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:16:40.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.276+0000 7fa419c00640 1 Processor -- start 2024-04-16T16:16:40.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.277+0000 7fa419c00640 1 -- start start 2024-04-16T16:16:40.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.278+0000 7fa419c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa414072a90 0x7fa4140ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:40.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.278+0000 7fa419c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa41415ea70 0x7fa4140cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:40.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.278+0000 7fa419c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4140cf140 con 0x7fa4141462e0 2024-04-16T16:16:40.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.278+0000 7fa419c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa4140cf6d0 con 0x7fa414072a90 2024-04-16T16:16:40.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.278+0000 7fa419c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa4141466c0 con 0x7fa41415ea70 2024-04-16T16:16:40.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.278+0000 7fa40be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa414072a90 0x7fa4140ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:40.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.279+0000 7fa40be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa414072a90 0x7fa4140ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45118/0 (socket says 172.21.15.45:45118) 2024-04-16T16:16:40.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.279+0000 7fa40be00640 1 -- 172.21.15.45:0/2173576607 learned_addr learned my addr 172.21.15.45:0/2173576607 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:40.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.279+0000 7fa413400640 1 --2- 172.21.15.45:0/2173576607 >> v2:172.21.15.103:3300/0 conn(0x7fa41415ea70 0x7fa4140cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:40.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.279+0000 7fa40be00640 1 -- 172.21.15.45:0/2173576607 >> v2:172.21.15.103:3300/0 conn(0x7fa41415ea70 msgr2=0x7fa4140cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:40.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.279+0000 7fa40be00640 1 --2- 172.21.15.45:0/2173576607 >> v2:172.21.15.103:3300/0 conn(0x7fa41415ea70 0x7fa4140cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.279+0000 7fa40be00640 1 -- 172.21.15.45:0/2173576607 >> v1:172.21.15.103:6789/0 conn(0x7fa4141462e0 legacy=0x7fa414072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:40.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.279+0000 7fa40be00640 1 -- 172.21.15.45:0/2173576607 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4140cf3f0 con 0x7fa414072a90 2024-04-16T16:16:40.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.280+0000 7fa40be00640 1 --2- 172.21.15.45:0/2173576607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa414072a90 0x7fa4140ce6c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fa4140b0660 tx=0x7fa3fc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e030098a62c817fc server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:40.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.280+0000 7fa412000640 1 -- 172.21.15.45:0/2173576607 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc02e070 con 0x7fa414072a90 2024-04-16T16:16:40.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.280+0000 7fa412000640 1 -- 172.21.15.45:0/2173576607 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3fc037070 con 0x7fa414072a90 2024-04-16T16:16:40.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.280+0000 7fa412000640 1 -- 172.21.15.45:0/2173576607 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc031a00 con 0x7fa414072a90 2024-04-16T16:16:40.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.281+0000 7fa419c00640 1 -- 172.21.15.45:0/2173576607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa414072a90 msgr2=0x7fa4140ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:40.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.281+0000 7fa419c00640 1 --2- 172.21.15.45:0/2173576607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa414072a90 0x7fa4140ce6c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fa4140b0660 tx=0x7fa3fc02b880 comp rx=0 tx=0).stop 2024-04-16T16:16:40.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.281+0000 7fa419c00640 1 -- 172.21.15.45:0/2173576607 shutdown_connections 2024-04-16T16:16:40.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.281+0000 7fa419c00640 1 --2- 172.21.15.45:0/2173576607 >> v2:172.21.15.103:3300/0 conn(0x7fa41415ea70 0x7fa4140cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.281+0000 7fa419c00640 1 --2- 172.21.15.45:0/2173576607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa414072a90 0x7fa4140ce6c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.281+0000 7fa419c00640 1 -- 172.21.15.45:0/2173576607 >> 172.21.15.45:0/2173576607 conn(0x7fa4140b2750 msgr2=0x7fa4140b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:40.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.281+0000 7fa419c00640 1 -- 172.21.15.45:0/2173576607 shutdown_connections 2024-04-16T16:16:40.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.282+0000 7fa419c00640 1 -- 172.21.15.45:0/2173576607 wait complete. 2024-04-16T16:16:40.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.283+0000 7fa419c00640 1 Processor -- start 2024-04-16T16:16:40.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.283+0000 7fa419c00640 1 -- start start 2024-04-16T16:16:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.284+0000 7fa419c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa414072a90 0x7fa41415c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.284+0000 7fa419c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4141462e0 0x7fa4140aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.284+0000 7fa419c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa41415ea70 0x7fa41413da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.284+0000 7fa419c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa41415d540 con 0x7fa4141462e0 2024-04-16T16:16:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.284+0000 7fa419c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa41415d260 con 0x7fa41415ea70 2024-04-16T16:16:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.284+0000 7fa419c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa414166490 con 0x7fa414072a90 2024-04-16T16:16:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.285+0000 7fa40be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4141462e0 0x7fa4140aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.285+0000 7fa40be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4141462e0 0x7fa4140aaf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45122/0 (socket says 172.21.15.45:45122) 2024-04-16T16:16:40.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.285+0000 7fa40be00640 1 -- 172.21.15.45:0/388707747 learned_addr learned my addr 172.21.15.45:0/388707747 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:40.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.285+0000 7fa413400640 1 --2- 172.21.15.45:0/388707747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa41415ea70 0x7fa41413da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:40.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.285+0000 7fa412a00640 1 --2- 172.21.15.45:0/388707747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa414072a90 0x7fa41415c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:40.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.285+0000 7fa40be00640 1 -- 172.21.15.45:0/388707747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa414072a90 msgr2=0x7fa41415c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:40.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.285+0000 7fa40be00640 1 --2- 172.21.15.45:0/388707747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa414072a90 0x7fa41415c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.285+0000 7fa40be00640 1 -- 172.21.15.45:0/388707747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa41415ea70 msgr2=0x7fa41413da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:40.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.285+0000 7fa40be00640 1 --2- 172.21.15.45:0/388707747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa41415ea70 0x7fa41413da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.285+0000 7fa40be00640 1 -- 172.21.15.45:0/388707747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4140ab4c0 con 0x7fa4141462e0 2024-04-16T16:16:40.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.286+0000 7fa40be00640 1 --2- 172.21.15.45:0/388707747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4141462e0 0x7fa4140aaf80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc000c00 tx=0x7fa3fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:40.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.286+0000 7fa40aa00640 1 -- 172.21.15.45:0/388707747 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc032880 con 0x7fa4141462e0 2024-04-16T16:16:40.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.286+0000 7fa40aa00640 1 -- 172.21.15.45:0/388707747 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3fc032a20 con 0x7fa4141462e0 2024-04-16T16:16:40.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.286+0000 7fa40aa00640 1 -- 172.21.15.45:0/388707747 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc032e10 con 0x7fa4141462e0 2024-04-16T16:16:40.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.286+0000 7fa419c00640 1 -- 172.21.15.45:0/388707747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa41415b770 con 0x7fa4141462e0 2024-04-16T16:16:40.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.286+0000 7fa419c00640 1 -- 172.21.15.45:0/388707747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa41415bd30 con 0x7fa4141462e0 2024-04-16T16:16:40.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.287+0000 7fa419c00640 1 -- 172.21.15.45:0/388707747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa414075780 con 0x7fa4141462e0 2024-04-16T16:16:40.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.288+0000 7fa40aa00640 1 -- 172.21.15.45:0/388707747 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fa3fc0453f0 con 0x7fa4141462e0 2024-04-16T16:16:40.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.289+0000 7fa40aa00640 1 --2- 172.21.15.45:0/388707747 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa3f006cb30 0x7fa3f006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:40.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.289+0000 7fa40aa00640 1 -- 172.21.15.45:0/388707747 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5709+0+0 (secure 0 0 0) 0x7fa3fc0adf70 con 0x7fa4141462e0 2024-04-16T16:16:40.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.290+0000 7fa412a00640 1 --2- 172.21.15.45:0/388707747 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa3f006cb30 0x7fa3f006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:40.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.291+0000 7fa412a00640 1 --2- 172.21.15.45:0/388707747 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa3f006cb30 0x7fa3f006eff0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa40400a750 tx=0x7fa404008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:40.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.294+0000 7fa40aa00640 1 -- 172.21.15.45:0/388707747 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa3fc040070 con 0x7fa4141462e0 2024-04-16T16:16:40.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-04-16T16:16:40.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-04-16T16:16:40.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.498+0000 7fa419c00640 1 -- 172.21.15.45:0/388707747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa4140af880 con 0x7fa4141462e0 2024-04-16T16:16:40.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.501+0000 7fa40aa00640 1 -- 172.21.15.45:0/388707747 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+16681 (secure 0 0 0) 0x7fa3fc07be90 con 0x7fa4141462e0 2024-04-16T16:16:40.500 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:40.500 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":71,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:16:39.513410+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:16:37.449973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-16T16:16:38.485027+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:16:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.503+0000 7fa419c00640 1 -- 172.21.15.45:0/388707747 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa3f006cb30 msgr2=0x7fa3f006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.503+0000 7fa419c00640 1 --2- 172.21.15.45:0/388707747 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa3f006cb30 0x7fa3f006eff0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa40400a750 tx=0x7fa404008040 comp rx=0 tx=0).stop 2024-04-16T16:16:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.503+0000 7fa419c00640 1 -- 172.21.15.45:0/388707747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4141462e0 msgr2=0x7fa4140aaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.503+0000 7fa419c00640 1 --2- 172.21.15.45:0/388707747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4141462e0 0x7fa4140aaf80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc000c00 tx=0x7fa3fc0046c0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.504+0000 7fa419c00640 1 -- 172.21.15.45:0/388707747 shutdown_connections 2024-04-16T16:16:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.504+0000 7fa419c00640 1 --2- 172.21.15.45:0/388707747 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa3f006cb30 0x7fa3f006eff0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.504+0000 7fa419c00640 1 --2- 172.21.15.45:0/388707747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa41415ea70 0x7fa41413da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.504+0000 7fa419c00640 1 --2- 172.21.15.45:0/388707747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4141462e0 0x7fa4140aaf80 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.504+0000 7fa419c00640 1 --2- 172.21.15.45:0/388707747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa414072a90 0x7fa41415c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.504+0000 7fa419c00640 1 -- 172.21.15.45:0/388707747 >> 172.21.15.45:0/388707747 conn(0x7fa4140b2750 msgr2=0x7fa41406e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.504+0000 7fa419c00640 1 -- 172.21.15.45:0/388707747 shutdown_connections 2024-04-16T16:16:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.504+0000 7fa419c00640 1 -- 172.21.15.45:0/388707747 wait complete. 2024-04-16T16:16:40.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:16:40.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.737+0000 7f7b80a00640 1 Processor -- start 2024-04-16T16:16:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.737+0000 7f7b80a00640 1 -- start start 2024-04-16T16:16:40.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.739+0000 7f7b80a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7b7c162270 0x7f7b7c0d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:40.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.739+0000 7f7b80a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b7c0d52d0 0x7f7b7c06ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:40.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.739+0000 7f7b80a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b7c0d3080 con 0x7f7b7c161e90 2024-04-16T16:16:40.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.739+0000 7f7b80a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7b7c0d2e30 con 0x7f7b7c162270 2024-04-16T16:16:40.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.739+0000 7f7b80a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7b7c0d3360 con 0x7f7b7c0d52d0 2024-04-16T16:16:40.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.739+0000 7f7b7a800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7b7c161e90 0x7f7b7c0cf7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42394/0 (socket says 172.21.15.45:42394) 2024-04-16T16:16:40.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.739+0000 7f7b7a800640 1 -- 172.21.15.45:0/4164603615 learned_addr learned my addr 172.21.15.45:0/4164603615 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:40.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b71e00640 1 --2- 172.21.15.45:0/4164603615 >> v2:172.21.15.103:3300/0 conn(0x7f7b7c162270 0x7f7b7c0d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:40.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b7b200640 1 --2- 172.21.15.45:0/4164603615 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b7c0d52d0 0x7f7b7c06ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:40.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 -- 172.21.15.45:0/4164603615 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2128464798 0 0) 0x7f7b7c0d3080 con 0x7f7b7c161e90 2024-04-16T16:16:40.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 -- 172.21.15.45:0/4164603615 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b60003520 con 0x7f7b7c161e90 2024-04-16T16:16:40.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 -- 172.21.15.45:0/4164603615 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 86890760 0 0) 0x7f7b60003520 con 0x7f7b7c161e90 2024-04-16T16:16:40.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 -- 172.21.15.45:0/4164603615 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b7c0d52d0 msgr2=0x7f7b7c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:40.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 --2- 172.21.15.45:0/4164603615 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b7c0d52d0 0x7f7b7c06ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 -- 172.21.15.45:0/4164603615 >> v2:172.21.15.103:3300/0 conn(0x7f7b7c162270 msgr2=0x7f7b7c0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:40.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 --2- 172.21.15.45:0/4164603615 >> v2:172.21.15.103:3300/0 conn(0x7f7b7c162270 0x7f7b7c0d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 -- 172.21.15.45:0/4164603615 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7c0d3640 con 0x7f7b7c161e90 2024-04-16T16:16:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 -- 172.21.15.45:0/4164603615 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f7b6c0019f0 con 0x7f7b7c161e90 2024-04-16T16:16:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 -- 172.21.15.45:0/4164603615 >> v1:172.21.15.45:6789/0 conn(0x7f7b7c161e90 legacy=0x7f7b7c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 --2- 172.21.15.45:0/4164603615 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b7c0d52d0 0x7f7b600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 --2- 172.21.15.45:0/4164603615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b60003f30 0x7f7b60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 --2- 172.21.15.45:0/4164603615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b60004d40 0x7f7b600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 -- 172.21.15.45:0/4164603615 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7c0d2e30 con 0x7f7b60004d40 2024-04-16T16:16:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 -- 172.21.15.45:0/4164603615 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7c0d3360 con 0x7f7b60003f30 2024-04-16T16:16:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.740+0000 7f7b79e00640 1 -- 172.21.15.45:0/4164603615 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7b6000baa0 con 0x7f7b7c0d52d0 2024-04-16T16:16:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.741+0000 7f7b7b200640 1 --2- 172.21.15.45:0/4164603615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b60004d40 0x7f7b600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.741+0000 7f7b71e00640 1 --2- 172.21.15.45:0/4164603615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b60003f30 0x7f7b60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.741+0000 7f7b7b200640 1 -- 172.21.15.45:0/4164603615 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b7c0d52d0 msgr2=0x7f7b600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.741+0000 7f7b7b200640 1 --2- 172.21.15.45:0/4164603615 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b7c0d52d0 0x7f7b600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.741+0000 7f7b7b200640 1 -- 172.21.15.45:0/4164603615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b60003f30 msgr2=0x7f7b60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.741+0000 7f7b7b200640 1 --2- 172.21.15.45:0/4164603615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b60003f30 0x7f7b60004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.741+0000 7f7b7b200640 1 -- 172.21.15.45:0/4164603615 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b60003520 con 0x7f7b60004d40 2024-04-16T16:16:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.741+0000 7f7b71e00640 1 --2- 172.21.15.45:0/4164603615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b60003f30 0x7f7b60004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.742+0000 7f7b7b200640 1 --2- 172.21.15.45:0/4164603615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b60004d40 0x7f7b600051c0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009a90 tx=0x7f7b60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9380f04d9e3f81bf server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.743+0000 7f7b79e00640 1 -- 172.21.15.45:0/4164603615 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b74039070 con 0x7f7b60004d40 2024-04-16T16:16:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.743+0000 7f7b79e00640 1 -- 172.21.15.45:0/4164603615 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b7402c6b0 con 0x7f7b60004d40 2024-04-16T16:16:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.743+0000 7f7b79e00640 1 -- 172.21.15.45:0/4164603615 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b74009690 con 0x7f7b60004d40 2024-04-16T16:16:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.743+0000 7f7b80a00640 1 -- 172.21.15.45:0/4164603615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b60004d40 msgr2=0x7f7b600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.743+0000 7f7b80a00640 1 --2- 172.21.15.45:0/4164603615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b60004d40 0x7f7b600051c0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009a90 tx=0x7f7b60005830 comp rx=0 tx=0).stop 2024-04-16T16:16:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.743+0000 7f7b7b200640 1 -- 172.21.15.45:0/4164603615 reap_dead start 2024-04-16T16:16:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.743+0000 7f7b80a00640 1 -- 172.21.15.45:0/4164603615 shutdown_connections 2024-04-16T16:16:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.743+0000 7f7b80a00640 1 -- 172.21.15.45:0/4164603615 >> 172.21.15.45:0/4164603615 conn(0x7f7b7c0d1940 msgr2=0x7f7b7c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.743+0000 7f7b80a00640 1 -- 172.21.15.45:0/4164603615 shutdown_connections 2024-04-16T16:16:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.743+0000 7f7b80a00640 1 -- 172.21.15.45:0/4164603615 wait complete. 2024-04-16T16:16:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.745+0000 7f7b80a00640 1 Processor -- start 2024-04-16T16:16:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.745+0000 7f7b80a00640 1 -- start start 2024-04-16T16:16:40.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.746+0000 7f7b80a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b60004d40 0x7f7b7c15bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:40.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.746+0000 7f7b80a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c0c95b0 0x7f7b7c159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:40.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.746+0000 7f7b80a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b7c15da00 0x7f7b7c15de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:40.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.746+0000 7f7b80a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7c0d4500 con 0x7f7b7c15da00 2024-04-16T16:16:40.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.746+0000 7f7b80a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7c0d4380 con 0x7f7b7c0c95b0 2024-04-16T16:16:40.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.746+0000 7f7b80a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7b7c0d4680 con 0x7f7b60004d40 2024-04-16T16:16:40.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.746+0000 7f7b7b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b7c15da00 0x7f7b7c15de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:40.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.746+0000 7f7b7b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b7c15da00 0x7f7b7c15de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45142/0 (socket says 172.21.15.45:45142) 2024-04-16T16:16:40.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.746+0000 7f7b7b200640 1 -- 172.21.15.45:0/2227880841 learned_addr learned my addr 172.21.15.45:0/2227880841 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:40.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.747+0000 7f7b7a800640 1 --2- 172.21.15.45:0/2227880841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b60004d40 0x7f7b7c15bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:40.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.747+0000 7f7b7b200640 1 -- 172.21.15.45:0/2227880841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b60004d40 msgr2=0x7f7b7c15bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:40.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.747+0000 7f7b7b200640 1 --2- 172.21.15.45:0/2227880841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b60004d40 0x7f7b7c15bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.747+0000 7f7b71e00640 1 --2- 172.21.15.45:0/2227880841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c0c95b0 0x7f7b7c159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:40.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.747+0000 7f7b7b200640 1 -- 172.21.15.45:0/2227880841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c0c95b0 msgr2=0x7f7b7c159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:40.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.747+0000 7f7b7b200640 1 --2- 172.21.15.45:0/2227880841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c0c95b0 0x7f7b7c159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.747+0000 7f7b7b200640 1 -- 172.21.15.45:0/2227880841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7c0c8270 con 0x7f7b7c15da00 2024-04-16T16:16:40.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.747+0000 7f7b71e00640 1 --2- 172.21.15.45:0/2227880841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c0c95b0 0x7f7b7c159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:40.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.747+0000 7f7b7b200640 1 --2- 172.21.15.45:0/2227880841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b7c15da00 0x7f7b7c15de90 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009bb0 tx=0x7f7b74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:40.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.747+0000 7f7b73400640 1 -- 172.21.15.45:0/2227880841 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b74039070 con 0x7f7b7c15da00 2024-04-16T16:16:40.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.747+0000 7f7b73400640 1 -- 172.21.15.45:0/2227880841 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b74043070 con 0x7f7b7c15da00 2024-04-16T16:16:40.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.747+0000 7f7b73400640 1 -- 172.21.15.45:0/2227880841 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b74009690 con 0x7f7b7c15da00 2024-04-16T16:16:40.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.747+0000 7f7b80a00640 1 -- 172.21.15.45:0/2227880841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b7c0c8510 con 0x7f7b7c15da00 2024-04-16T16:16:40.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.748+0000 7f7b80a00640 1 -- 172.21.15.45:0/2227880841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b7c0c8ad0 con 0x7f7b7c15da00 2024-04-16T16:16:40.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.749+0000 7f7b80a00640 1 -- 172.21.15.45:0/2227880841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b7c0b34a0 con 0x7f7b7c15da00 2024-04-16T16:16:40.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.750+0000 7f7b73400640 1 -- 172.21.15.45:0/2227880841 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f7b74034a70 con 0x7f7b7c15da00 2024-04-16T16:16:40.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.750+0000 7f7b73400640 1 --2- 172.21.15.45:0/2227880841 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f7b5806cb60 0x7f7b5806f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:40.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.751+0000 7f7b73400640 1 -- 172.21.15.45:0/2227880841 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5709+0+0 (secure 0 0 0) 0x7f7b740ad1c0 con 0x7f7b7c15da00 2024-04-16T16:16:40.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.751+0000 7f7b7a800640 1 --2- 172.21.15.45:0/2227880841 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f7b5806cb60 0x7f7b5806f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:40.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.752+0000 7f7b7a800640 1 --2- 172.21.15.45:0/2227880841 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f7b5806cb60 0x7f7b5806f020 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0012e0 tx=0x7f7b6c009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:40.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.755+0000 7f7b73400640 1 -- 172.21.15.45:0/2227880841 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7b7c0b34a0 con 0x7f7b7c15da00 2024-04-16T16:16:40.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.955+0000 7f7b80a00640 1 -- 172.21.15.45:0/2227880841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b7c06dae0 con 0x7f7b7c15da00 2024-04-16T16:16:40.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.958+0000 7f7b73400640 1 -- 172.21.15.45:0/2227880841 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+16681 (secure 0 0 0) 0x7f7b7404c020 con 0x7f7b7c15da00 2024-04-16T16:16:40.958 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:40.959 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":71,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:16:39.513410+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:16:37.449973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-16T16:16:38.485027+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:16:40.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.960+0000 7f7b80a00640 1 -- 172.21.15.45:0/2227880841 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f7b5806cb60 msgr2=0x7f7b5806f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:40.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.960+0000 7f7b80a00640 1 --2- 172.21.15.45:0/2227880841 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f7b5806cb60 0x7f7b5806f020 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0012e0 tx=0x7f7b6c009040 comp rx=0 tx=0).stop 2024-04-16T16:16:40.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.960+0000 7f7b80a00640 1 -- 172.21.15.45:0/2227880841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b7c15da00 msgr2=0x7f7b7c15de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:40.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.960+0000 7f7b80a00640 1 --2- 172.21.15.45:0/2227880841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b7c15da00 0x7f7b7c15de90 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009bb0 tx=0x7f7b74004730 comp rx=0 tx=0).stop 2024-04-16T16:16:40.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.961+0000 7f7b80a00640 1 -- 172.21.15.45:0/2227880841 shutdown_connections 2024-04-16T16:16:40.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.961+0000 7f7b80a00640 1 --2- 172.21.15.45:0/2227880841 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f7b5806cb60 0x7f7b5806f020 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.961+0000 7f7b80a00640 1 --2- 172.21.15.45:0/2227880841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b7c15da00 0x7f7b7c15de90 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.961+0000 7f7b80a00640 1 --2- 172.21.15.45:0/2227880841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c0c95b0 0x7f7b7c159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.961+0000 7f7b80a00640 1 --2- 172.21.15.45:0/2227880841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b60004d40 0x7f7b7c15bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:40.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.961+0000 7f7b80a00640 1 -- 172.21.15.45:0/2227880841 >> 172.21.15.45:0/2227880841 conn(0x7f7b7c0d1940 msgr2=0x7f7b7c0b5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:40.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.961+0000 7f7b80a00640 1 -- 172.21.15.45:0/2227880841 shutdown_connections 2024-04-16T16:16:40.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:40.961+0000 7f7b80a00640 1 -- 172.21.15.45:0/2227880841 wait complete. 2024-04-16T16:16:40.980 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-16T16:16:41.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.199+0000 7fa05b000640 1 Processor -- start 2024-04-16T16:16:41.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.200+0000 7fa05b000640 1 -- start start 2024-04-16T16:16:41.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.201+0000 7fa05b000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa0540d18f0 0x7fa0540ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:41.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.201+0000 7fa05b000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0540d1cd0 0x7fa054162dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:41.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.201+0000 7fa05b000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0540b6810 con 0x7fa0540b6310 2024-04-16T16:16:41.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.201+0000 7fa05b000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa0540d29f0 con 0x7fa0540d18f0 2024-04-16T16:16:41.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.201+0000 7fa05b000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa0540b6af0 con 0x7fa0540d1cd0 2024-04-16T16:16:41.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.202+0000 7fa059c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa0540d18f0 0x7fa0540ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:41.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.202+0000 7fa059c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa0540d18f0 0x7fa0540ba220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45146/0 (socket says 172.21.15.45:45146) 2024-04-16T16:16:41.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.202+0000 7fa059c00640 1 -- 172.21.15.45:0/845226019 learned_addr learned my addr 172.21.15.45:0/845226019 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:41.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.202+0000 7fa05a600640 1 --2- 172.21.15.45:0/845226019 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0540d1cd0 0x7fa054162dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:41.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.202+0000 7fa059c00640 1 -- 172.21.15.45:0/845226019 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0540d1cd0 msgr2=0x7fa054162dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:41.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.202+0000 7fa059c00640 1 --2- 172.21.15.45:0/845226019 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0540d1cd0 0x7fa054162dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:41.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.202+0000 7fa059c00640 1 -- 172.21.15.45:0/845226019 >> v1:172.21.15.103:6789/0 conn(0x7fa0540b6310 legacy=0x7fa0540d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:41.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.202+0000 7fa059c00640 1 -- 172.21.15.45:0/845226019 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0540b6dd0 con 0x7fa0540d18f0 2024-04-16T16:16:41.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.203+0000 7fa059c00640 1 --2- 172.21.15.45:0/845226019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0540d18f0 0x7fa0540ba220 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa044009870 tx=0x7fa04402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d5e178f636d153 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:41.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.204+0000 7fa053e00640 1 -- 172.21.15.45:0/845226019 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0440317c0 con 0x7fa0540d18f0 2024-04-16T16:16:41.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.204+0000 7fa053e00640 1 -- 172.21.15.45:0/845226019 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa044037070 con 0x7fa0540d18f0 2024-04-16T16:16:41.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.204+0000 7fa053e00640 1 -- 172.21.15.45:0/845226019 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa044031b80 con 0x7fa0540d18f0 2024-04-16T16:16:41.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.204+0000 7fa05b000640 1 -- 172.21.15.45:0/845226019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0540d18f0 msgr2=0x7fa0540ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:41.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.204+0000 7fa05b000640 1 --2- 172.21.15.45:0/845226019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0540d18f0 0x7fa0540ba220 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa044009870 tx=0x7fa04402b860 comp rx=0 tx=0).stop 2024-04-16T16:16:41.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.204+0000 7fa05b000640 1 -- 172.21.15.45:0/845226019 shutdown_connections 2024-04-16T16:16:41.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.204+0000 7fa05b000640 1 --2- 172.21.15.45:0/845226019 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0540d1cd0 0x7fa054162dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:41.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.204+0000 7fa05b000640 1 --2- 172.21.15.45:0/845226019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0540d18f0 0x7fa0540ba220 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:41.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.204+0000 7fa05b000640 1 -- 172.21.15.45:0/845226019 >> 172.21.15.45:0/845226019 conn(0x7fa05406cf90 msgr2=0x7fa05406d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:41.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.205+0000 7fa05b000640 1 -- 172.21.15.45:0/845226019 shutdown_connections 2024-04-16T16:16:41.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.205+0000 7fa05b000640 1 -- 172.21.15.45:0/845226019 wait complete. 2024-04-16T16:16:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.206+0000 7fa05b000640 1 Processor -- start 2024-04-16T16:16:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.206+0000 7fa05b000640 1 -- start start 2024-04-16T16:16:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.207+0000 7fa05b000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0540b6310 0x7fa0540c8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.207+0000 7fa05b000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0540d18f0 0x7fa0541587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.207+0000 7fa05b000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0540d1cd0 0x7fa054158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.207+0000 7fa05b000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa054163300 con 0x7fa0540b6310 2024-04-16T16:16:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.207+0000 7fa05b000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa0540ba760 con 0x7fa0540d18f0 2024-04-16T16:16:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.207+0000 7fa05b000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa0540c98e0 con 0x7fa0540d1cd0 2024-04-16T16:16:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.208+0000 7fa059c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0540b6310 0x7fa0540c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.208+0000 7fa059c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0540b6310 0x7fa0540c8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45148/0 (socket says 172.21.15.45:45148) 2024-04-16T16:16:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.208+0000 7fa059c00640 1 -- 172.21.15.45:0/2463817247 learned_addr learned my addr 172.21.15.45:0/2463817247 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.208+0000 7fa059200640 1 --2- 172.21.15.45:0/2463817247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0540d18f0 0x7fa0541587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.208+0000 7fa05a600640 1 --2- 172.21.15.45:0/2463817247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0540d1cd0 0x7fa054158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.208+0000 7fa059c00640 1 -- 172.21.15.45:0/2463817247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0540d1cd0 msgr2=0x7fa054158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.208+0000 7fa059c00640 1 --2- 172.21.15.45:0/2463817247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0540d1cd0 0x7fa054158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.208+0000 7fa059c00640 1 -- 172.21.15.45:0/2463817247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0540d18f0 msgr2=0x7fa0541587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.208+0000 7fa059c00640 1 --2- 172.21.15.45:0/2463817247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0540d18f0 0x7fa0541587e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.208+0000 7fa059c00640 1 -- 172.21.15.45:0/2463817247 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0540c8cd0 con 0x7fa0540b6310 2024-04-16T16:16:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.208+0000 7fa059c00640 1 --2- 172.21.15.45:0/2463817247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0540b6310 0x7fa0540c8480 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa044000c00 tx=0x7fa0440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.209+0000 7fa051600640 1 -- 172.21.15.45:0/2463817247 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa044032650 con 0x7fa0540b6310 2024-04-16T16:16:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.209+0000 7fa051600640 1 -- 172.21.15.45:0/2463817247 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0440327f0 con 0x7fa0540b6310 2024-04-16T16:16:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.209+0000 7fa05b000640 1 -- 172.21.15.45:0/2463817247 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0540c8fb0 con 0x7fa0540b6310 2024-04-16T16:16:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.209+0000 7fa051600640 1 -- 172.21.15.45:0/2463817247 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa044032be0 con 0x7fa0540b6310 2024-04-16T16:16:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.209+0000 7fa05b000640 1 -- 172.21.15.45:0/2463817247 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0540c89c0 con 0x7fa0540b6310 2024-04-16T16:16:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.210+0000 7fa05b000640 1 -- 172.21.15.45:0/2463817247 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0540c9180 con 0x7fa0540b6310 2024-04-16T16:16:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.212+0000 7fa051600640 1 -- 172.21.15.45:0/2463817247 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fa04404c060 con 0x7fa0540b6310 2024-04-16T16:16:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.213+0000 7fa051600640 1 --2- 172.21.15.45:0/2463817247 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa03406cb60 0x7fa03406f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.213+0000 7fa051600640 1 -- 172.21.15.45:0/2463817247 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7fa0440ad8b0 con 0x7fa0540b6310 2024-04-16T16:16:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.214+0000 7fa059200640 1 --2- 172.21.15.45:0/2463817247 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa03406cb60 0x7fa03406f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.215+0000 7fa059200640 1 --2- 172.21.15.45:0/2463817247 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa03406cb60 0x7fa03406f020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa054068f20 tx=0x7fa040009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.217+0000 7fa051600640 1 -- 172.21.15.45:0/2463817247 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa044040070 con 0x7fa0540b6310 2024-04-16T16:16:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:41.468+0000 7fa05b000640 1 -- 172.21.15.45:0/2463817247 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa054075780 con 0x7fa0540b6310 2024-04-16T16:16:42.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.068+0000 7fa051600640 1 -- 172.21.15.45:0/2463817247 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v49) ==== 118+0+0 (secure 0 0 0) 0x7fa04407b970 con 0x7fa0540b6310 2024-04-16T16:16:42.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.071+0000 7fa05b000640 1 -- 172.21.15.45:0/2463817247 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa03406cb60 msgr2=0x7fa03406f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:42.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.071+0000 7fa05b000640 1 --2- 172.21.15.45:0/2463817247 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa03406cb60 0x7fa03406f020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa054068f20 tx=0x7fa040009040 comp rx=0 tx=0).stop 2024-04-16T16:16:42.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.071+0000 7fa05b000640 1 -- 172.21.15.45:0/2463817247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0540b6310 msgr2=0x7fa0540c8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:42.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.071+0000 7fa05b000640 1 --2- 172.21.15.45:0/2463817247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0540b6310 0x7fa0540c8480 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa044000c00 tx=0x7fa0440046c0 comp rx=0 tx=0).stop 2024-04-16T16:16:42.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.072+0000 7fa05b000640 1 -- 172.21.15.45:0/2463817247 shutdown_connections 2024-04-16T16:16:42.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.072+0000 7fa05b000640 1 --2- 172.21.15.45:0/2463817247 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fa03406cb60 0x7fa03406f020 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:42.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.072+0000 7fa05b000640 1 --2- 172.21.15.45:0/2463817247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0540d1cd0 0x7fa054158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:42.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.072+0000 7fa05b000640 1 --2- 172.21.15.45:0/2463817247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0540d18f0 0x7fa0541587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:42.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.072+0000 7fa05b000640 1 --2- 172.21.15.45:0/2463817247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0540b6310 0x7fa0540c8480 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:42.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.072+0000 7fa05b000640 1 -- 172.21.15.45:0/2463817247 >> 172.21.15.45:0/2463817247 conn(0x7fa05406cf90 msgr2=0x7fa0540b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:42.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.072+0000 7fa05b000640 1 -- 172.21.15.45:0/2463817247 shutdown_connections 2024-04-16T16:16:42.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.072+0000 7fa05b000640 1 -- 172.21.15.45:0/2463817247 wait complete. 2024-04-16T16:16:42.097 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-16T16:16:42.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-16T16:16:42.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.324+0000 7f0456200640 1 Processor -- start 2024-04-16T16:16:42.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.324+0000 7f0456200640 1 -- start start 2024-04-16T16:16:42.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.326+0000 7f0456200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f04501461b0 0x7f0450160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:42.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.326+0000 7f0456200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f04500b58f0 0x7f04500b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:42.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.326+0000 7f0456200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04500cf440 0x7f04500cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:42.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.326+0000 7f0456200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f045006dfd0 con 0x7f04501461b0 2024-04-16T16:16:42.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.326+0000 7f0456200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f04500b6290 con 0x7f04500b58f0 2024-04-16T16:16:42.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.326+0000 7f0456200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0450146590 con 0x7f04500cf440 2024-04-16T16:16:42.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.326+0000 7f0454e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f04501461b0 0x7f0450160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:42.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.326+0000 7f0454e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f04501461b0 0x7f0450160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45154/0 (socket says 172.21.15.45:45154) 2024-04-16T16:16:42.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.326+0000 7f0454e00640 1 -- 172.21.15.45:0/3214141777 learned_addr learned my addr 172.21.15.45:0/3214141777 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:42.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.327+0000 7f044fe00640 1 --2- 172.21.15.45:0/3214141777 >> v2:172.21.15.103:3300/0 conn(0x7f04500b58f0 0x7f04500b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:42.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.327+0000 7f0455800640 1 --2- 172.21.15.45:0/3214141777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04500cf440 0x7f04500cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:42.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.327+0000 7f0454e00640 1 -- 172.21.15.45:0/3214141777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04500cf440 msgr2=0x7f04500cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:42.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.327+0000 7f0454e00640 1 --2- 172.21.15.45:0/3214141777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04500cf440 0x7f04500cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:42.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.327+0000 7f0454e00640 1 -- 172.21.15.45:0/3214141777 >> v2:172.21.15.103:3300/0 conn(0x7f04500b58f0 msgr2=0x7f04500b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:42.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.327+0000 7f0454e00640 1 --2- 172.21.15.45:0/3214141777 >> v2:172.21.15.103:3300/0 conn(0x7f04500b58f0 0x7f04500b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:42.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.327+0000 7f0454e00640 1 -- 172.21.15.45:0/3214141777 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04500c5750 con 0x7f04501461b0 2024-04-16T16:16:42.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.328+0000 7f0454e00640 1 --2- 172.21.15.45:0/3214141777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04501461b0 0x7f0450160cf0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f04500b68c0 tx=0x7f044402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39ccc6dffef6db56 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:42.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.328+0000 7f044f400640 1 -- 172.21.15.45:0/3214141777 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0444035020 con 0x7f04501461b0 2024-04-16T16:16:42.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.328+0000 7f044f400640 1 -- 172.21.15.45:0/3214141777 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0444031870 con 0x7f04501461b0 2024-04-16T16:16:42.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.328+0000 7f044f400640 1 -- 172.21.15.45:0/3214141777 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0444031c40 con 0x7f04501461b0 2024-04-16T16:16:42.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.329+0000 7f0456200640 1 -- 172.21.15.45:0/3214141777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04501461b0 msgr2=0x7f0450160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:42.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.329+0000 7f0456200640 1 --2- 172.21.15.45:0/3214141777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04501461b0 0x7f0450160cf0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f04500b68c0 tx=0x7f044402b910 comp rx=0 tx=0).stop 2024-04-16T16:16:42.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.329+0000 7f0456200640 1 -- 172.21.15.45:0/3214141777 shutdown_connections 2024-04-16T16:16:42.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.329+0000 7f0456200640 1 --2- 172.21.15.45:0/3214141777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04500cf440 0x7f04500cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:42.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.329+0000 7f0456200640 1 --2- 172.21.15.45:0/3214141777 >> v2:172.21.15.103:3300/0 conn(0x7f04500b58f0 0x7f04500b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:42.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.329+0000 7f0456200640 1 --2- 172.21.15.45:0/3214141777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04501461b0 0x7f0450160cf0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:42.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.329+0000 7f0456200640 1 -- 172.21.15.45:0/3214141777 >> 172.21.15.45:0/3214141777 conn(0x7f045006b930 msgr2=0x7f045006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:42.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.329+0000 7f0456200640 1 -- 172.21.15.45:0/3214141777 shutdown_connections 2024-04-16T16:16:42.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.329+0000 7f0456200640 1 -- 172.21.15.45:0/3214141777 wait complete. 2024-04-16T16:16:42.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.331+0000 7f0456200640 1 Processor -- start 2024-04-16T16:16:42.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.332+0000 7f0456200640 1 -- start start 2024-04-16T16:16:42.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f0456200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04500b58f0 0x7f0450144c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:42.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f0456200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04500cf440 0x7f04500cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:42.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f0456200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04501461b0 0x7f045015c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:42.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f0456200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f04500c59f0 con 0x7f04500b58f0 2024-04-16T16:16:42.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f0456200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0450161230 con 0x7f04500cf440 2024-04-16T16:16:42.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f0456200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f04500c5cd0 con 0x7f04501461b0 2024-04-16T16:16:42.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f0454e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04500b58f0 0x7f0450144c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:42.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f0454e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04500b58f0 0x7f0450144c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45158/0 (socket says 172.21.15.45:45158) 2024-04-16T16:16:42.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f0454e00640 1 -- 172.21.15.45:0/4082187063 learned_addr learned my addr 172.21.15.45:0/4082187063 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:42.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f044fe00640 1 --2- 172.21.15.45:0/4082187063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04500cf440 0x7f04500cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:42.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f0455800640 1 --2- 172.21.15.45:0/4082187063 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04501461b0 0x7f045015c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:42.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f0454e00640 1 -- 172.21.15.45:0/4082187063 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04501461b0 msgr2=0x7f045015c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:42.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f0454e00640 1 --2- 172.21.15.45:0/4082187063 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04501461b0 0x7f045015c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:42.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f0454e00640 1 -- 172.21.15.45:0/4082187063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04500cf440 msgr2=0x7f04500cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:42.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f0454e00640 1 --2- 172.21.15.45:0/4082187063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04500cf440 0x7f04500cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:42.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f0454e00640 1 -- 172.21.15.45:0/4082187063 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04500cdb40 con 0x7f04500b58f0 2024-04-16T16:16:42.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.333+0000 7f044fe00640 1 --2- 172.21.15.45:0/4082187063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04500cf440 0x7f04500cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:42.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.334+0000 7f0454e00640 1 --2- 172.21.15.45:0/4082187063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04500b58f0 0x7f0450144c30 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0444000c00 tx=0x7f04440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:42.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.334+0000 7f044cc00640 1 -- 172.21.15.45:0/4082187063 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044403ad60 con 0x7f04500b58f0 2024-04-16T16:16:42.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.334+0000 7f044cc00640 1 -- 172.21.15.45:0/4082187063 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04440443f0 con 0x7f04500b58f0 2024-04-16T16:16:42.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.334+0000 7f044cc00640 1 -- 172.21.15.45:0/4082187063 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04440447e0 con 0x7f04500b58f0 2024-04-16T16:16:42.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.334+0000 7f0456200640 1 -- 172.21.15.45:0/4082187063 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04500ce1e0 con 0x7f04500b58f0 2024-04-16T16:16:42.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.334+0000 7f0456200640 1 -- 172.21.15.45:0/4082187063 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04500ce7a0 con 0x7f04500b58f0 2024-04-16T16:16:42.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.335+0000 7f0456200640 1 -- 172.21.15.45:0/4082187063 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0450075780 con 0x7f04500b58f0 2024-04-16T16:16:42.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.336+0000 7f044cc00640 1 -- 172.21.15.45:0/4082187063 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f044404b060 con 0x7f04500b58f0 2024-04-16T16:16:42.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.338+0000 7f044cc00640 1 --2- 172.21.15.45:0/4082187063 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f043006cb30 0x7f043006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:42.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.338+0000 7f044cc00640 1 -- 172.21.15.45:0/4082187063 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f04440ae2d0 con 0x7f04500b58f0 2024-04-16T16:16:42.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.338+0000 7f044fe00640 1 --2- 172.21.15.45:0/4082187063 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f043006cb30 0x7f043006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:42.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.340+0000 7f044fe00640 1 --2- 172.21.15.45:0/4082187063 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f043006cb30 0x7f043006eff0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f045015f1c0 tx=0x7f043c008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:42.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.343+0000 7f044cc00640 1 -- 172.21.15.45:0/4082187063 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f044403f070 con 0x7f04500b58f0 2024-04-16T16:16:42.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:42.593+0000 7f0456200640 1 -- 172.21.15.45:0/4082187063 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f0450072500 con 0x7f04500b58f0 2024-04-16T16:16:43.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.081+0000 7f044cc00640 1 -- 172.21.15.45:0/4082187063 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v50) ==== 122+0+0 (secure 0 0 0) 0x7f044402f320 con 0x7f04500b58f0 2024-04-16T16:16:43.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.085+0000 7f0456200640 1 -- 172.21.15.45:0/4082187063 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f043006cb30 msgr2=0x7f043006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:43.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.085+0000 7f0456200640 1 --2- 172.21.15.45:0/4082187063 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f043006cb30 0x7f043006eff0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f045015f1c0 tx=0x7f043c008040 comp rx=0 tx=0).stop 2024-04-16T16:16:43.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.085+0000 7f0456200640 1 -- 172.21.15.45:0/4082187063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04500b58f0 msgr2=0x7f0450144c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:43.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.085+0000 7f0456200640 1 --2- 172.21.15.45:0/4082187063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04500b58f0 0x7f0450144c30 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0444000c00 tx=0x7f04440046c0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.085+0000 7f0456200640 1 -- 172.21.15.45:0/4082187063 shutdown_connections 2024-04-16T16:16:43.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.085+0000 7f0456200640 1 --2- 172.21.15.45:0/4082187063 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f043006cb30 0x7f043006eff0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.085+0000 7f0456200640 1 --2- 172.21.15.45:0/4082187063 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04501461b0 0x7f045015c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.085+0000 7f0456200640 1 --2- 172.21.15.45:0/4082187063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04500cf440 0x7f04500cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.085+0000 7f0456200640 1 --2- 172.21.15.45:0/4082187063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04500b58f0 0x7f0450144c30 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.085+0000 7f0456200640 1 -- 172.21.15.45:0/4082187063 >> 172.21.15.45:0/4082187063 conn(0x7f045006b930 msgr2=0x7f04500b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:43.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.085+0000 7f0456200640 1 -- 172.21.15.45:0/4082187063 shutdown_connections 2024-04-16T16:16:43.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.085+0000 7f0456200640 1 -- 172.21.15.45:0/4082187063 wait complete. 2024-04-16T16:16:43.110 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-16T16:16:43.111 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:16:43.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.339+0000 7f915ec00640 1 Processor -- start 2024-04-16T16:16:43.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.339+0000 7f915ec00640 1 -- start start 2024-04-16T16:16:43.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.340+0000 7f915ec00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9158161ff0 0x7f91580cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:43.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.340+0000 7f915ec00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f91580cfd20 0x7f91580b6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:43.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.340+0000 7f915ec00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91580d46e0 con 0x7f91580b6c70 2024-04-16T16:16:43.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.340+0000 7f915ec00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f91580b70a0 con 0x7f9158161ff0 2024-04-16T16:16:43.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.340+0000 7f915ec00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f91580ba560 con 0x7f91580cfd20 2024-04-16T16:16:43.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.341+0000 7f915d800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9158161ff0 0x7f91580cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:43.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.341+0000 7f915e200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f91580b6c70 0x7f915806cd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35708/0 (socket says 172.21.15.45:35708) 2024-04-16T16:16:43.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.341+0000 7f915e200640 1 -- 172.21.15.45:0/4239025753 learned_addr learned my addr 172.21.15.45:0/4239025753 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:43.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.341+0000 7f915ce00640 1 --2- 172.21.15.45:0/4239025753 >> v2:172.21.15.103:3300/0 conn(0x7f91580cfd20 0x7f91580b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:43.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.341+0000 7f915d800640 1 -- 172.21.15.45:0/4239025753 >> v2:172.21.15.103:3300/0 conn(0x7f91580cfd20 msgr2=0x7f91580b6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:43.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.341+0000 7f915d800640 1 --2- 172.21.15.45:0/4239025753 >> v2:172.21.15.103:3300/0 conn(0x7f91580cfd20 0x7f91580b6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.342+0000 7f915d800640 1 -- 172.21.15.45:0/4239025753 >> v1:172.21.15.45:6789/0 conn(0x7f91580b6c70 legacy=0x7f915806cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:43.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.342+0000 7f915d800640 1 -- 172.21.15.45:0/4239025753 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91580d3640 con 0x7f9158161ff0 2024-04-16T16:16:43.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.342+0000 7f9157e00640 1 -- 172.21.15.45:0/4239025753 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 361418971 0 0) 0x7f91580d46e0 con 0x7f91580b6c70 2024-04-16T16:16:43.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.343+0000 7f915d800640 1 --2- 172.21.15.45:0/4239025753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9158161ff0 0x7f91580cf7e0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f91580b4320 tx=0x7f914802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f989664f71161394 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:43.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.343+0000 7f9157e00640 1 -- 172.21.15.45:0/4239025753 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148035020 con 0x7f9158161ff0 2024-04-16T16:16:43.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.343+0000 7f9157e00640 1 -- 172.21.15.45:0/4239025753 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9148031830 con 0x7f9158161ff0 2024-04-16T16:16:43.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.343+0000 7f9157e00640 1 -- 172.21.15.45:0/4239025753 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148031c00 con 0x7f9158161ff0 2024-04-16T16:16:43.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.344+0000 7f915ec00640 1 -- 172.21.15.45:0/4239025753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9158161ff0 msgr2=0x7f91580cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:43.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.344+0000 7f915ec00640 1 --2- 172.21.15.45:0/4239025753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9158161ff0 0x7f91580cf7e0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f91580b4320 tx=0x7f914802b8d0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.344+0000 7f915ec00640 1 -- 172.21.15.45:0/4239025753 shutdown_connections 2024-04-16T16:16:43.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.344+0000 7f915ec00640 1 --2- 172.21.15.45:0/4239025753 >> v2:172.21.15.103:3300/0 conn(0x7f91580cfd20 0x7f91580b6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.344+0000 7f915ec00640 1 --2- 172.21.15.45:0/4239025753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9158161ff0 0x7f91580cf7e0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.344+0000 7f915ec00640 1 -- 172.21.15.45:0/4239025753 >> 172.21.15.45:0/4239025753 conn(0x7f91580d17b0 msgr2=0x7f91580d1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:43.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.344+0000 7f915ec00640 1 -- 172.21.15.45:0/4239025753 shutdown_connections 2024-04-16T16:16:43.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.344+0000 7f915ec00640 1 -- 172.21.15.45:0/4239025753 wait complete. 2024-04-16T16:16:43.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.346+0000 7f915ec00640 1 Processor -- start 2024-04-16T16:16:43.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.346+0000 7f915ec00640 1 -- start start 2024-04-16T16:16:43.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.347+0000 7f915ec00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91580b6c70 0x7f91580c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:43.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.347+0000 7f915ec00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91580cfd20 0x7f91580cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:43.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.347+0000 7f915ec00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9158161ff0 0x7f91580cd560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:43.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.347+0000 7f915ec00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f91580d3ee0 con 0x7f9158161ff0 2024-04-16T16:16:43.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.347+0000 7f915ec00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f91580d3c00 con 0x7f91580b6c70 2024-04-16T16:16:43.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.347+0000 7f915ec00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f91580d41c0 con 0x7f91580cfd20 2024-04-16T16:16:43.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.348+0000 7f915e200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9158161ff0 0x7f91580cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:43.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.348+0000 7f915e200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9158161ff0 0x7f91580cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46434/0 (socket says 172.21.15.45:46434) 2024-04-16T16:16:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.348+0000 7f915e200640 1 -- 172.21.15.45:0/3911307474 learned_addr learned my addr 172.21.15.45:0/3911307474 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.348+0000 7f915d800640 1 --2- 172.21.15.45:0/3911307474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91580b6c70 0x7f91580c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.348+0000 7f915ce00640 1 --2- 172.21.15.45:0/3911307474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91580cfd20 0x7f91580cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.348+0000 7f915e200640 1 -- 172.21.15.45:0/3911307474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91580cfd20 msgr2=0x7f91580cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.348+0000 7f915e200640 1 --2- 172.21.15.45:0/3911307474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91580cfd20 0x7f91580cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.348+0000 7f915e200640 1 -- 172.21.15.45:0/3911307474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91580b6c70 msgr2=0x7f91580c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.348+0000 7f915e200640 1 --2- 172.21.15.45:0/3911307474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91580b6c70 0x7f91580c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.348+0000 7f915e200640 1 -- 172.21.15.45:0/3911307474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91580ce670 con 0x7f9158161ff0 2024-04-16T16:16:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.349+0000 7f915e200640 1 --2- 172.21.15.45:0/3911307474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9158161ff0 0x7f91580cd560 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f915000c700 tx=0x7f915000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.349+0000 7f9155600640 1 -- 172.21.15.45:0/3911307474 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9150018070 con 0x7f9158161ff0 2024-04-16T16:16:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.349+0000 7f9155600640 1 -- 172.21.15.45:0/3911307474 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f915000ce10 con 0x7f9158161ff0 2024-04-16T16:16:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.349+0000 7f915ec00640 1 -- 172.21.15.45:0/3911307474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91580cddd0 con 0x7f9158161ff0 2024-04-16T16:16:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.349+0000 7f9155600640 1 -- 172.21.15.45:0/3911307474 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915000a880 con 0x7f9158161ff0 2024-04-16T16:16:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.349+0000 7f915ec00640 1 -- 172.21.15.45:0/3911307474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91580ce390 con 0x7f9158161ff0 2024-04-16T16:16:43.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.350+0000 7f915ec00640 1 -- 172.21.15.45:0/3911307474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9158075780 con 0x7f9158161ff0 2024-04-16T16:16:43.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.352+0000 7f9155600640 1 -- 172.21.15.45:0/3911307474 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f915001b070 con 0x7f9158161ff0 2024-04-16T16:16:43.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.352+0000 7f9155600640 1 --2- 172.21.15.45:0/3911307474 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f913806cb60 0x7f913806f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:43.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.353+0000 7f9155600640 1 -- 172.21.15.45:0/3911307474 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f915008b760 con 0x7f9158161ff0 2024-04-16T16:16:43.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.353+0000 7f915d800640 1 --2- 172.21.15.45:0/3911307474 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f913806cb60 0x7f913806f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:43.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.355+0000 7f915d800640 1 --2- 172.21.15.45:0/3911307474 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f913806cb60 0x7f913806f020 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9148036000 tx=0x7f9148008030 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:43.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.356+0000 7f9155600640 1 -- 172.21.15.45:0/3911307474 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9150059910 con 0x7f9158161ff0 2024-04-16T16:16:43.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.607+0000 7f915ec00640 1 -- 172.21.15.45:0/3911307474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91580b3b10 con 0x7f9158161ff0 2024-04-16T16:16:43.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.609+0000 7f9155600640 1 -- 172.21.15.45:0/3911307474 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5180 (secure 0 0 0) 0x7f9150059210 con 0x7f9158161ff0 2024-04-16T16:16:43.609 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:43.609 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":50,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":6399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:16:42.595621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-04-16T16:16:43.609 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 50 2024-04-16T16:16:43.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.611+0000 7f915ec00640 1 -- 172.21.15.45:0/3911307474 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f913806cb60 msgr2=0x7f913806f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.611+0000 7f915ec00640 1 --2- 172.21.15.45:0/3911307474 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f913806cb60 0x7f913806f020 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9148036000 tx=0x7f9148008030 comp rx=0 tx=0).stop 2024-04-16T16:16:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.611+0000 7f915ec00640 1 -- 172.21.15.45:0/3911307474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9158161ff0 msgr2=0x7f91580cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.611+0000 7f915ec00640 1 --2- 172.21.15.45:0/3911307474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9158161ff0 0x7f91580cd560 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f915000c700 tx=0x7f915000cbc0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.611+0000 7f915ec00640 1 -- 172.21.15.45:0/3911307474 shutdown_connections 2024-04-16T16:16:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.611+0000 7f915ec00640 1 --2- 172.21.15.45:0/3911307474 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f913806cb60 0x7f913806f020 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.611+0000 7f915ec00640 1 --2- 172.21.15.45:0/3911307474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9158161ff0 0x7f91580cd560 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.611+0000 7f915ec00640 1 --2- 172.21.15.45:0/3911307474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91580cfd20 0x7f91580cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.611+0000 7f915ec00640 1 --2- 172.21.15.45:0/3911307474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91580b6c70 0x7f91580c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.611+0000 7f915ec00640 1 -- 172.21.15.45:0/3911307474 >> 172.21.15.45:0/3911307474 conn(0x7f91580d17b0 msgr2=0x7f91580d48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.612+0000 7f915ec00640 1 -- 172.21.15.45:0/3911307474 shutdown_connections 2024-04-16T16:16:43.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.612+0000 7f915ec00640 1 -- 172.21.15.45:0/3911307474 wait complete. 2024-04-16T16:16:43.632 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:16:43.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.852+0000 7f87d1c00640 1 Processor -- start 2024-04-16T16:16:43.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.852+0000 7f87d1c00640 1 -- start start 2024-04-16T16:16:43.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.854+0000 7f87d1c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87cc072450 0x7f87cc06d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:43.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.854+0000 7f87d1c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f87cc0b60c0 0x7f87cc06db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:43.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.854+0000 7f87d1c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87cc0728e0 con 0x7f87cc149460 2024-04-16T16:16:43.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.854+0000 7f87d1c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f87cc072bc0 con 0x7f87cc0b60c0 2024-04-16T16:16:43.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.854+0000 7f87d1c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f87cc06cb70 con 0x7f87cc072450 2024-04-16T16:16:43.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.854+0000 7f87d1200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f87cc0b60c0 0x7f87cc06db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:43.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.854+0000 7f87d1200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f87cc0b60c0 0x7f87cc06db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46448/0 (socket says 172.21.15.45:46448) 2024-04-16T16:16:43.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.854+0000 7f87d1200640 1 -- 172.21.15.45:0/4214145293 learned_addr learned my addr 172.21.15.45:0/4214145293 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:43.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.855+0000 7f87c3400640 1 --2- 172.21.15.45:0/4214145293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87cc072450 0x7f87cc06d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:43.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.855+0000 7f87d1200640 1 -- 172.21.15.45:0/4214145293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87cc072450 msgr2=0x7f87cc06d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:43.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.855+0000 7f87d1200640 1 --2- 172.21.15.45:0/4214145293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87cc072450 0x7f87cc06d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.855+0000 7f87d1200640 1 -- 172.21.15.45:0/4214145293 >> v1:172.21.15.103:6789/0 conn(0x7f87cc149460 legacy=0x7f87cc0b59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:43.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.855+0000 7f87d1200640 1 -- 172.21.15.45:0/4214145293 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87cc06ce50 con 0x7f87cc0b60c0 2024-04-16T16:16:43.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.856+0000 7f87d1200640 1 --2- 172.21.15.45:0/4214145293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc0b60c0 0x7f87cc06db90 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009e50 tx=0x7f87c402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccff31b1d836599 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:43.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.856+0000 7f87cb400640 1 -- 172.21.15.45:0/4214145293 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c402c9f0 con 0x7f87cc0b60c0 2024-04-16T16:16:43.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.856+0000 7f87cb400640 1 -- 172.21.15.45:0/4214145293 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87c402cb90 con 0x7f87cc0b60c0 2024-04-16T16:16:43.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.856+0000 7f87cb400640 1 -- 172.21.15.45:0/4214145293 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c40343f0 con 0x7f87cc0b60c0 2024-04-16T16:16:43.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.857+0000 7f87d1c00640 1 -- 172.21.15.45:0/4214145293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc0b60c0 msgr2=0x7f87cc06db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:43.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.857+0000 7f87d1c00640 1 --2- 172.21.15.45:0/4214145293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc0b60c0 0x7f87cc06db90 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009e50 tx=0x7f87c402be50 comp rx=0 tx=0).stop 2024-04-16T16:16:43.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.857+0000 7f87d1c00640 1 -- 172.21.15.45:0/4214145293 shutdown_connections 2024-04-16T16:16:43.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.857+0000 7f87d1c00640 1 --2- 172.21.15.45:0/4214145293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc0b60c0 0x7f87cc06db90 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.857+0000 7f87d1c00640 1 --2- 172.21.15.45:0/4214145293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87cc072450 0x7f87cc06d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.857+0000 7f87d1c00640 1 -- 172.21.15.45:0/4214145293 >> 172.21.15.45:0/4214145293 conn(0x7f87cc0b43a0 msgr2=0x7f87cc0b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:43.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.857+0000 7f87d1c00640 1 -- 172.21.15.45:0/4214145293 shutdown_connections 2024-04-16T16:16:43.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.857+0000 7f87d1c00640 1 -- 172.21.15.45:0/4214145293 wait complete. 2024-04-16T16:16:43.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.859+0000 7f87d1c00640 1 Processor -- start 2024-04-16T16:16:43.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.859+0000 7f87d1c00640 1 -- start start 2024-04-16T16:16:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.860+0000 7f87d1c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87cc0b60c0 0x7f87cc0c8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.860+0000 7f87d1c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc149460 0x7f87cc1449f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.860+0000 7f87d1c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87cc0d18e0 0x7f87cc0ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.860+0000 7f87d1c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc14c9d0 con 0x7f87cc149460 2024-04-16T16:16:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.860+0000 7f87d1c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc14c850 con 0x7f87cc0d18e0 2024-04-16T16:16:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.860+0000 7f87d1c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f87cc14c1b0 con 0x7f87cc0b60c0 2024-04-16T16:16:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.860+0000 7f87c3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc149460 0x7f87cc1449f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.860+0000 7f87c3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc149460 0x7f87cc1449f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46462/0 (socket says 172.21.15.45:46462) 2024-04-16T16:16:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.860+0000 7f87c3400640 1 -- 172.21.15.45:0/1346755149 learned_addr learned my addr 172.21.15.45:0/1346755149 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.861+0000 7f87cbe00640 1 --2- 172.21.15.45:0/1346755149 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87cc0b60c0 0x7f87cc0c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.861+0000 7f87d1200640 1 --2- 172.21.15.45:0/1346755149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87cc0d18e0 0x7f87cc0ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.861+0000 7f87c3400640 1 -- 172.21.15.45:0/1346755149 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87cc0b60c0 msgr2=0x7f87cc0c8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.861+0000 7f87c3400640 1 --2- 172.21.15.45:0/1346755149 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87cc0b60c0 0x7f87cc0c8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.861+0000 7f87c3400640 1 -- 172.21.15.45:0/1346755149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87cc0d18e0 msgr2=0x7f87cc0ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:43.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.861+0000 7f87c3400640 1 --2- 172.21.15.45:0/1346755149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87cc0d18e0 0x7f87cc0ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:43.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.861+0000 7f87c3400640 1 -- 172.21.15.45:0/1346755149 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87cc0c9570 con 0x7f87cc149460 2024-04-16T16:16:43.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.861+0000 7f87c3400640 1 --2- 172.21.15.45:0/1346755149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc149460 0x7f87cc1449f0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f87b400d800 tx=0x7f87b400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:43.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.862+0000 7f87c8c00640 1 -- 172.21.15.45:0/1346755149 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b400b6c0 con 0x7f87cc149460 2024-04-16T16:16:43.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.862+0000 7f87c8c00640 1 -- 172.21.15.45:0/1346755149 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b4019070 con 0x7f87cc149460 2024-04-16T16:16:43.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.862+0000 7f87c8c00640 1 -- 172.21.15.45:0/1346755149 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b400bac0 con 0x7f87cc149460 2024-04-16T16:16:43.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.862+0000 7f87d1c00640 1 -- 172.21.15.45:0/1346755149 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87cc0c8cd0 con 0x7f87cc149460 2024-04-16T16:16:43.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.862+0000 7f87d1c00640 1 -- 172.21.15.45:0/1346755149 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87cc0c9290 con 0x7f87cc149460 2024-04-16T16:16:43.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.863+0000 7f87d1c00640 1 -- 172.21.15.45:0/1346755149 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87cc06bd30 con 0x7f87cc149460 2024-04-16T16:16:43.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.864+0000 7f87c8c00640 1 -- 172.21.15.45:0/1346755149 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f87b400bc60 con 0x7f87cc149460 2024-04-16T16:16:43.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.865+0000 7f87c8c00640 1 --2- 172.21.15.45:0/1346755149 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f87a806cb30 0x7f87a806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:43.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.865+0000 7f87c8c00640 1 -- 172.21.15.45:0/1346755149 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f87b408c560 con 0x7f87cc149460 2024-04-16T16:16:43.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.866+0000 7f87cbe00640 1 --2- 172.21.15.45:0/1346755149 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f87a806cb30 0x7f87a806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:43.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.867+0000 7f87cbe00640 1 --2- 172.21.15.45:0/1346755149 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f87a806cb30 0x7f87a806eff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0d1f90 tx=0x7f87bc009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:43.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:43.870+0000 7f87c8c00640 1 -- 172.21.15.45:0/1346755149 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f87b405a590 con 0x7f87cc149460 2024-04-16T16:16:44.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.072+0000 7f87d1c00640 1 -- 172.21.15.45:0/1346755149 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f87cc0c89c0 con 0x7f87cc149460 2024-04-16T16:16:44.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.074+0000 7f87c8c00640 1 -- 172.21.15.45:0/1346755149 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+16668 (secure 0 0 0) 0x7f87b4059e90 con 0x7f87cc149460 2024-04-16T16:16:44.075 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:44.075 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":72,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:16:41.044663+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:16:37.449973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-16T16:16:38.485027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:16:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.077+0000 7f87d1c00640 1 -- 172.21.15.45:0/1346755149 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f87a806cb30 msgr2=0x7f87a806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.077+0000 7f87d1c00640 1 --2- 172.21.15.45:0/1346755149 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f87a806cb30 0x7f87a806eff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0d1f90 tx=0x7f87bc009040 comp rx=0 tx=0).stop 2024-04-16T16:16:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.077+0000 7f87d1c00640 1 -- 172.21.15.45:0/1346755149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc149460 msgr2=0x7f87cc1449f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.077+0000 7f87d1c00640 1 --2- 172.21.15.45:0/1346755149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc149460 0x7f87cc1449f0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f87b400d800 tx=0x7f87b400dcc0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.077+0000 7f87d1c00640 1 -- 172.21.15.45:0/1346755149 shutdown_connections 2024-04-16T16:16:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.077+0000 7f87d1c00640 1 --2- 172.21.15.45:0/1346755149 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f87a806cb30 0x7f87a806eff0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.077+0000 7f87d1c00640 1 --2- 172.21.15.45:0/1346755149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87cc0d18e0 0x7f87cc0ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.077+0000 7f87d1c00640 1 --2- 172.21.15.45:0/1346755149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc149460 0x7f87cc1449f0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.078+0000 7f87d1c00640 1 --2- 172.21.15.45:0/1346755149 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87cc0b60c0 0x7f87cc0c8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.078+0000 7f87d1c00640 1 -- 172.21.15.45:0/1346755149 >> 172.21.15.45:0/1346755149 conn(0x7f87cc0b43a0 msgr2=0x7f87cc0b4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.078+0000 7f87d1c00640 1 -- 172.21.15.45:0/1346755149 shutdown_connections 2024-04-16T16:16:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.078+0000 7f87d1c00640 1 -- 172.21.15.45:0/1346755149 wait complete. 2024-04-16T16:16:44.099 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-16T16:16:44.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:16:44.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.318+0000 7f6075800640 1 Processor -- start 2024-04-16T16:16:44.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.318+0000 7f6075800640 1 -- start start 2024-04-16T16:16:44.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.320+0000 7f6075800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6070144ff0 0x7f60700cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:44.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.320+0000 7f6075800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60700c5d80 0x7f607015d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:44.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.320+0000 7f6075800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f607015c700 con 0x7f607006dfb0 2024-04-16T16:16:44.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.320+0000 7f6075800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f607015d2f0 con 0x7f6070144ff0 2024-04-16T16:16:44.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.320+0000 7f6075800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f607015c9c0 con 0x7f60700c5d80 2024-04-16T16:16:44.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.321+0000 7f606fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6070144ff0 0x7f60700cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:44.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.321+0000 7f6074e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60700c5d80 0x7f607015d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:44.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.321+0000 7f6074e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60700c5d80 0x7f607015d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:33884/0 (socket says 172.21.15.45:33884) 2024-04-16T16:16:44.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.321+0000 7f606fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6070144ff0 0x7f60700cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:54726/0 (socket says 172.21.15.45:54726) 2024-04-16T16:16:44.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.321+0000 7f6074e00640 1 -- 172.21.15.45:0/3091454740 learned_addr learned my addr 172.21.15.45:0/3091454740 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:44.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.321+0000 7f6074e00640 1 -- 172.21.15.45:0/3091454740 >> v2:172.21.15.103:3300/0 conn(0x7f6070144ff0 msgr2=0x7f60700cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:44.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.321+0000 7f6074e00640 1 --2- 172.21.15.45:0/3091454740 >> v2:172.21.15.103:3300/0 conn(0x7f6070144ff0 0x7f60700cfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.321+0000 7f6074e00640 1 -- 172.21.15.45:0/3091454740 >> v1:172.21.15.103:6789/0 conn(0x7f607006dfb0 legacy=0x7f60700c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:44.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.321+0000 7f6074e00640 1 -- 172.21.15.45:0/3091454740 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607015cca0 con 0x7f60700c5d80 2024-04-16T16:16:44.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.323+0000 7f6074e00640 1 --2- 172.21.15.45:0/3091454740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60700c5d80 0x7f607015d780 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6068009e50 tx=0x7f606802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=397c4435b4d417e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:44.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.324+0000 7f606ea00640 1 -- 172.21.15.45:0/3091454740 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606802cb00 con 0x7f60700c5d80 2024-04-16T16:16:44.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.324+0000 7f606ea00640 1 -- 172.21.15.45:0/3091454740 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f606802cca0 con 0x7f60700c5d80 2024-04-16T16:16:44.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.324+0000 7f606ea00640 1 -- 172.21.15.45:0/3091454740 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068034610 con 0x7f60700c5d80 2024-04-16T16:16:44.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.324+0000 7f6075800640 1 -- 172.21.15.45:0/3091454740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60700c5d80 msgr2=0x7f607015d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:44.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.324+0000 7f6075800640 1 --2- 172.21.15.45:0/3091454740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60700c5d80 0x7f607015d780 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6068009e50 tx=0x7f606802bf60 comp rx=0 tx=0).stop 2024-04-16T16:16:44.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.324+0000 7f6075800640 1 -- 172.21.15.45:0/3091454740 shutdown_connections 2024-04-16T16:16:44.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.324+0000 7f6075800640 1 --2- 172.21.15.45:0/3091454740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60700c5d80 0x7f607015d780 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.324+0000 7f6075800640 1 --2- 172.21.15.45:0/3091454740 >> v2:172.21.15.103:3300/0 conn(0x7f6070144ff0 0x7f60700cfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.324+0000 7f6075800640 1 -- 172.21.15.45:0/3091454740 >> 172.21.15.45:0/3091454740 conn(0x7f607006b930 msgr2=0x7f607006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:44.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.324+0000 7f6075800640 1 -- 172.21.15.45:0/3091454740 shutdown_connections 2024-04-16T16:16:44.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.325+0000 7f6075800640 1 -- 172.21.15.45:0/3091454740 wait complete. 2024-04-16T16:16:44.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.326+0000 7f6075800640 1 Processor -- start 2024-04-16T16:16:44.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.326+0000 7f6075800640 1 -- start start 2024-04-16T16:16:44.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.327+0000 7f6075800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607006dfb0 0x7f60700b19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:44.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.327+0000 7f6075800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60700c5d80 0x7f6070132df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:44.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.327+0000 7f6075800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6070144ff0 0x7f607006e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:44.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.327+0000 7f6075800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f60700d0270 con 0x7f607006dfb0 2024-04-16T16:16:44.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.327+0000 7f6075800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60701453d0 con 0x7f60700c5d80 2024-04-16T16:16:44.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.327+0000 7f6075800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f60700d03f0 con 0x7f6070144ff0 2024-04-16T16:16:44.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.328+0000 7f606fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607006dfb0 0x7f60700b19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:44.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.328+0000 7f606fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607006dfb0 0x7f60700b19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46474/0 (socket says 172.21.15.45:46474) 2024-04-16T16:16:44.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.328+0000 7f606fe00640 1 -- 172.21.15.45:0/1293314964 learned_addr learned my addr 172.21.15.45:0/1293314964 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:44.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.328+0000 7f6074e00640 1 --2- 172.21.15.45:0/1293314964 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6070144ff0 0x7f607006e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:44.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.328+0000 7f606f400640 1 --2- 172.21.15.45:0/1293314964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60700c5d80 0x7f6070132df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:44.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.328+0000 7f606fe00640 1 -- 172.21.15.45:0/1293314964 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6070144ff0 msgr2=0x7f607006e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:44.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.328+0000 7f606fe00640 1 --2- 172.21.15.45:0/1293314964 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6070144ff0 0x7f607006e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.328+0000 7f606fe00640 1 -- 172.21.15.45:0/1293314964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60700c5d80 msgr2=0x7f6070132df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:44.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.328+0000 7f606fe00640 1 --2- 172.21.15.45:0/1293314964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60700c5d80 0x7f6070132df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.328+0000 7f606fe00640 1 -- 172.21.15.45:0/1293314964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6070133840 con 0x7f607006dfb0 2024-04-16T16:16:44.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.328+0000 7f606f400640 1 --2- 172.21.15.45:0/1293314964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60700c5d80 0x7f6070132df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:44.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.328+0000 7f606fe00640 1 --2- 172.21.15.45:0/1293314964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607006dfb0 0x7f60700b19a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f606000e870 tx=0x7f606000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:44.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.329+0000 7f6067e00640 1 -- 172.21.15.45:0/1293314964 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6060017070 con 0x7f607006dfb0 2024-04-16T16:16:44.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.329+0000 7f6067e00640 1 -- 172.21.15.45:0/1293314964 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f606000c940 con 0x7f607006dfb0 2024-04-16T16:16:44.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.329+0000 7f6067e00640 1 -- 172.21.15.45:0/1293314964 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606000cd30 con 0x7f607006dfb0 2024-04-16T16:16:44.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.329+0000 7f6075800640 1 -- 172.21.15.45:0/1293314964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6070159420 con 0x7f607006dfb0 2024-04-16T16:16:44.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.329+0000 7f6075800640 1 -- 172.21.15.45:0/1293314964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60701599e0 con 0x7f607006dfb0 2024-04-16T16:16:44.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.330+0000 7f6075800640 1 -- 172.21.15.45:0/1293314964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6070075780 con 0x7f607006dfb0 2024-04-16T16:16:44.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.331+0000 7f6067e00640 1 -- 172.21.15.45:0/1293314964 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f606001a070 con 0x7f607006dfb0 2024-04-16T16:16:44.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.332+0000 7f6067e00640 1 --2- 172.21.15.45:0/1293314964 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f605006cb30 0x7f605006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:44.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.332+0000 7f6067e00640 1 -- 172.21.15.45:0/1293314964 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f606008bef0 con 0x7f607006dfb0 2024-04-16T16:16:44.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.333+0000 7f606f400640 1 --2- 172.21.15.45:0/1293314964 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f605006cb30 0x7f605006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:44.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.334+0000 7f606f400640 1 --2- 172.21.15.45:0/1293314964 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f605006cb30 0x7f605006eff0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f60700b24f0 tx=0x7f605c009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:44.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.336+0000 7f6067e00640 1 -- 172.21.15.45:0/1293314964 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6060059f20 con 0x7f607006dfb0 2024-04-16T16:16:44.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.529+0000 7f6075800640 1 -- 172.21.15.45:0/1293314964 --> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f60700b59c0 con 0x7f605006cb30 2024-04-16T16:16:44.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.551+0000 7f6067e00640 1 -- 172.21.15.45:0/1293314964 <== mgr.4961 v2:172.21.15.103:6832/4155603915 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+244136 (secure 0 0 0) 0x7f60700b59c0 con 0x7f605006cb30 2024-04-16T16:16:44.554 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:44.555 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:16:44.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.556+0000 7f6075800640 1 -- 172.21.15.45:0/1293314964 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f605006cb30 msgr2=0x7f605006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:44.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.556+0000 7f6075800640 1 --2- 172.21.15.45:0/1293314964 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f605006cb30 0x7f605006eff0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f60700b24f0 tx=0x7f605c009040 comp rx=0 tx=0).stop 2024-04-16T16:16:44.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.556+0000 7f6075800640 1 -- 172.21.15.45:0/1293314964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607006dfb0 msgr2=0x7f60700b19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:44.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.556+0000 7f6075800640 1 --2- 172.21.15.45:0/1293314964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607006dfb0 0x7f60700b19a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f606000e870 tx=0x7f606000ed30 comp rx=0 tx=0).stop 2024-04-16T16:16:44.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.556+0000 7f6075800640 1 -- 172.21.15.45:0/1293314964 shutdown_connections 2024-04-16T16:16:44.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.556+0000 7f6075800640 1 --2- 172.21.15.45:0/1293314964 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f605006cb30 0x7f605006eff0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.556+0000 7f6075800640 1 --2- 172.21.15.45:0/1293314964 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6070144ff0 0x7f607006e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.556+0000 7f6075800640 1 --2- 172.21.15.45:0/1293314964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60700c5d80 0x7f6070132df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.556+0000 7f6075800640 1 --2- 172.21.15.45:0/1293314964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607006dfb0 0x7f60700b19a0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.556+0000 7f6075800640 1 -- 172.21.15.45:0/1293314964 >> 172.21.15.45:0/1293314964 conn(0x7f607006b930 msgr2=0x7f60700c5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:44.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.556+0000 7f6075800640 1 -- 172.21.15.45:0/1293314964 shutdown_connections 2024-04-16T16:16:44.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.556+0000 7f6075800640 1 -- 172.21.15.45:0/1293314964 wait complete. 2024-04-16T16:16:44.576 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-16T16:16:43.065817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592014,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":208,"num_write_kb":2426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":108,"acting":108,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":44,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248940,"kb_used_data":7800,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754725780,"statfs":{"total":773094113280,"available":772839198720,"internally_reserved":0,"allocated":7987200,"data_stored":4340213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1646,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.064145"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354998+0000","last_change":"2024-04-16T16:16:40.067374+0000","last_active":"2024-04-16T16:16:41.354998+0000","last_peered":"2024-04-16T16:16:41.354998+0000","last_clean":"2024-04-16T16:16:41.354998+0000","last_became_active":"2024-04-16T16:16:40.064314+0000","last_became_peered":"2024-04-16T16:16:40.064314+0000","last_unstale":"2024-04-16T16:16:41.354998+0000","last_undegraded":"2024-04-16T16:16:41.354998+0000","last_fullsized":"2024-04-16T16:16:41.354998+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:47:33.049888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052264+0000","last_change":"2024-04-16T16:16:40.053258+0000","last_active":"2024-04-16T16:16:41.052264+0000","last_peered":"2024-04-16T16:16:41.052264+0000","last_clean":"2024-04-16T16:16:41.052264+0000","last_became_active":"2024-04-16T16:16:40.051051+0000","last_became_peered":"2024-04-16T16:16:40.051051+0000","last_unstale":"2024-04-16T16:16:41.052264+0000","last_undegraded":"2024-04-16T16:16:41.052264+0000","last_fullsized":"2024-04-16T16:16:41.052264+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:16:10.608170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053974+0000","last_change":"2024-04-16T16:16:40.065283+0000","last_active":"2024-04-16T16:16:41.053974+0000","last_peered":"2024-04-16T16:16:41.053974+0000","last_clean":"2024-04-16T16:16:41.053974+0000","last_became_active":"2024-04-16T16:16:40.064211+0000","last_became_peered":"2024-04-16T16:16:40.064211+0000","last_unstale":"2024-04-16T16:16:41.053974+0000","last_undegraded":"2024-04-16T16:16:41.053974+0000","last_fullsized":"2024-04-16T16:16:41.053974+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:28.836553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354796+0000","last_change":"2024-04-16T16:16:40.070880+0000","last_active":"2024-04-16T16:16:41.354796+0000","last_peered":"2024-04-16T16:16:41.354796+0000","last_clean":"2024-04-16T16:16:41.354796+0000","last_became_active":"2024-04-16T16:16:40.070434+0000","last_became_peered":"2024-04-16T16:16:40.070434+0000","last_unstale":"2024-04-16T16:16:41.354796+0000","last_undegraded":"2024-04-16T16:16:41.354796+0000","last_fullsized":"2024-04-16T16:16:41.354796+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:13:47.015413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084447+0000","last_change":"2024-04-16T16:16:40.053229+0000","last_active":"2024-04-16T16:16:41.084447+0000","last_peered":"2024-04-16T16:16:41.084447+0000","last_clean":"2024-04-16T16:16:41.084447+0000","last_became_active":"2024-04-16T16:16:40.052968+0000","last_became_peered":"2024-04-16T16:16:40.052968+0000","last_unstale":"2024-04-16T16:16:41.084447+0000","last_undegraded":"2024-04-16T16:16:41.084447+0000","last_fullsized":"2024-04-16T16:16:41.084447+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:46.618649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052568+0000","last_change":"2024-04-16T16:16:40.059138+0000","last_active":"2024-04-16T16:16:41.052568+0000","last_peered":"2024-04-16T16:16:41.052568+0000","last_clean":"2024-04-16T16:16:41.052568+0000","last_became_active":"2024-04-16T16:16:40.056131+0000","last_became_peered":"2024-04-16T16:16:40.056131+0000","last_unstale":"2024-04-16T16:16:41.052568+0000","last_undegraded":"2024-04-16T16:16:41.052568+0000","last_fullsized":"2024-04-16T16:16:41.052568+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:39:52.139845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084236+0000","last_change":"2024-04-16T16:16:40.052923+0000","last_active":"2024-04-16T16:16:41.084236+0000","last_peered":"2024-04-16T16:16:41.084236+0000","last_clean":"2024-04-16T16:16:41.084236+0000","last_became_active":"2024-04-16T16:16:40.052609+0000","last_became_peered":"2024-04-16T16:16:40.052609+0000","last_unstale":"2024-04-16T16:16:41.084236+0000","last_undegraded":"2024-04-16T16:16:41.084236+0000","last_fullsized":"2024-04-16T16:16:41.084236+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:21.332023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054029+0000","last_change":"2024-04-16T16:16:40.069834+0000","last_active":"2024-04-16T16:16:41.054029+0000","last_peered":"2024-04-16T16:16:41.054029+0000","last_clean":"2024-04-16T16:16:41.054029+0000","last_became_active":"2024-04-16T16:16:40.062429+0000","last_became_peered":"2024-04-16T16:16:40.062429+0000","last_unstale":"2024-04-16T16:16:41.054029+0000","last_undegraded":"2024-04-16T16:16:41.054029+0000","last_fullsized":"2024-04-16T16:16:41.054029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:14:20.465129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355562+0000","last_change":"2024-04-16T16:16:40.055791+0000","last_active":"2024-04-16T16:16:41.355562+0000","last_peered":"2024-04-16T16:16:41.355562+0000","last_clean":"2024-04-16T16:16:41.355562+0000","last_became_active":"2024-04-16T16:16:40.055508+0000","last_became_peered":"2024-04-16T16:16:40.055508+0000","last_unstale":"2024-04-16T16:16:41.355562+0000","last_undegraded":"2024-04-16T16:16:41.355562+0000","last_fullsized":"2024-04-16T16:16:41.355562+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:44.062254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084256+0000","last_change":"2024-04-16T16:16:40.058979+0000","last_active":"2024-04-16T16:16:41.084256+0000","last_peered":"2024-04-16T16:16:41.084256+0000","last_clean":"2024-04-16T16:16:41.084256+0000","last_became_active":"2024-04-16T16:16:40.058648+0000","last_became_peered":"2024-04-16T16:16:40.058648+0000","last_unstale":"2024-04-16T16:16:41.084256+0000","last_undegraded":"2024-04-16T16:16:41.084256+0000","last_fullsized":"2024-04-16T16:16:41.084256+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:28:41.607987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.010949+0000","last_change":"2024-04-16T16:16:38.010949+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.010949+0000","last_undegraded":"2024-04-16T16:16:38.010949+0000","last_fullsized":"2024-04-16T16:16:38.010949+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054498+0000","last_change":"2024-04-16T16:16:40.063884+0000","last_active":"2024-04-16T16:16:41.054498+0000","last_peered":"2024-04-16T16:16:41.054498+0000","last_clean":"2024-04-16T16:16:41.054498+0000","last_became_active":"2024-04-16T16:16:40.061609+0000","last_became_peered":"2024-04-16T16:16:40.061609+0000","last_unstale":"2024-04-16T16:16:41.054498+0000","last_undegraded":"2024-04-16T16:16:41.054498+0000","last_fullsized":"2024-04-16T16:16:41.054498+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:55.827541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084571+0000","last_change":"2024-04-16T16:16:39.035506+0000","last_active":"2024-04-16T16:16:41.084571+0000","last_peered":"2024-04-16T16:16:41.084571+0000","last_clean":"2024-04-16T16:16:41.084571+0000","last_became_active":"2024-04-16T16:16:39.035196+0000","last_became_peered":"2024-04-16T16:16:39.035196+0000","last_unstale":"2024-04-16T16:16:41.084571+0000","last_undegraded":"2024-04-16T16:16:41.084571+0000","last_fullsized":"2024-04-16T16:16:41.084571+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:29:14.659131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.007452+0000","last_change":"2024-04-16T16:16:38.007452+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.007452+0000","last_undegraded":"2024-04-16T16:16:38.007452+0000","last_fullsized":"2024-04-16T16:16:38.007452+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.018161+0000","last_change":"2024-04-16T16:16:38.018161+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.018161+0000","last_undegraded":"2024-04-16T16:16:38.018161+0000","last_fullsized":"2024-04-16T16:16:38.018161+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.012103+0000","last_change":"2024-04-16T16:16:38.012103+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.012103+0000","last_undegraded":"2024-04-16T16:16:38.012103+0000","last_fullsized":"2024-04-16T16:16:38.012103+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.021097+0000","last_change":"2024-04-16T16:16:38.021097+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.021097+0000","last_undegraded":"2024-04-16T16:16:38.021097+0000","last_fullsized":"2024-04-16T16:16:38.021097+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.051911+0000","last_change":"2024-04-16T16:16:40.049943+0000","last_active":"2024-04-16T16:16:41.051911+0000","last_peered":"2024-04-16T16:16:41.051911+0000","last_clean":"2024-04-16T16:16:41.051911+0000","last_became_active":"2024-04-16T16:16:40.049466+0000","last_became_peered":"2024-04-16T16:16:40.049466+0000","last_unstale":"2024-04-16T16:16:41.051911+0000","last_undegraded":"2024-04-16T16:16:41.051911+0000","last_fullsized":"2024-04-16T16:16:41.051911+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:24.082098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353887+0000","last_change":"2024-04-16T16:16:39.048665+0000","last_active":"2024-04-16T16:16:41.353887+0000","last_peered":"2024-04-16T16:16:41.353887+0000","last_clean":"2024-04-16T16:16:41.353887+0000","last_became_active":"2024-04-16T16:16:39.046018+0000","last_became_peered":"2024-04-16T16:16:39.046018+0000","last_unstale":"2024-04-16T16:16:41.353887+0000","last_undegraded":"2024-04-16T16:16:41.353887+0000","last_fullsized":"2024-04-16T16:16:41.353887+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:37:31.768292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084230+0000","last_change":"2024-04-16T16:16:40.068846+0000","last_active":"2024-04-16T16:16:41.084230+0000","last_peered":"2024-04-16T16:16:41.084230+0000","last_clean":"2024-04-16T16:16:41.084230+0000","last_became_active":"2024-04-16T16:16:40.060719+0000","last_became_peered":"2024-04-16T16:16:40.060719+0000","last_unstale":"2024-04-16T16:16:41.084230+0000","last_undegraded":"2024-04-16T16:16:41.084230+0000","last_fullsized":"2024-04-16T16:16:41.084230+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:48:35.347056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.051833+0000","last_change":"2024-04-16T16:16:39.042020+0000","last_active":"2024-04-16T16:16:41.051833+0000","last_peered":"2024-04-16T16:16:41.051833+0000","last_clean":"2024-04-16T16:16:41.051833+0000","last_became_active":"2024-04-16T16:16:39.041447+0000","last_became_peered":"2024-04-16T16:16:39.041447+0000","last_unstale":"2024-04-16T16:16:41.051833+0000","last_undegraded":"2024-04-16T16:16:41.051833+0000","last_fullsized":"2024-04-16T16:16:41.051833+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:18:24.097920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.007668+0000","last_change":"2024-04-16T16:16:38.007668+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.007668+0000","last_undegraded":"2024-04-16T16:16:38.007668+0000","last_fullsized":"2024-04-16T16:16:38.007668+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.018790+0000","last_change":"2024-04-16T16:16:38.018790+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.018790+0000","last_undegraded":"2024-04-16T16:16:38.018790+0000","last_fullsized":"2024-04-16T16:16:38.018790+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.012305+0000","last_change":"2024-04-16T16:16:38.012305+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.012305+0000","last_undegraded":"2024-04-16T16:16:38.012305+0000","last_fullsized":"2024-04-16T16:16:38.012305+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053162+0000","last_change":"2024-04-16T16:16:40.056836+0000","last_active":"2024-04-16T16:16:41.053162+0000","last_peered":"2024-04-16T16:16:41.053162+0000","last_clean":"2024-04-16T16:16:41.053162+0000","last_became_active":"2024-04-16T16:16:40.056365+0000","last_became_peered":"2024-04-16T16:16:40.056365+0000","last_unstale":"2024-04-16T16:16:41.053162+0000","last_undegraded":"2024-04-16T16:16:41.053162+0000","last_fullsized":"2024-04-16T16:16:41.053162+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:17:23.179723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"71'6","reported_seq":20,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355016+0000","last_change":"2024-04-16T16:16:39.040106+0000","last_active":"2024-04-16T16:16:41.355016+0000","last_peered":"2024-04-16T16:16:41.355016+0000","last_clean":"2024-04-16T16:16:41.355016+0000","last_became_active":"2024-04-16T16:16:39.039334+0000","last_became_peered":"2024-04-16T16:16:39.039334+0000","last_unstale":"2024-04-16T16:16:41.355016+0000","last_undegraded":"2024-04-16T16:16:41.355016+0000","last_fullsized":"2024-04-16T16:16:41.355016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:29.791246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.13","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354451+0000","last_change":"2024-04-16T16:16:39.048257+0000","last_active":"2024-04-16T16:16:41.354451+0000","last_peered":"2024-04-16T16:16:41.354451+0000","last_clean":"2024-04-16T16:16:41.354451+0000","last_became_active":"2024-04-16T16:16:39.047835+0000","last_became_peered":"2024-04-16T16:16:39.047835+0000","last_unstale":"2024-04-16T16:16:41.354451+0000","last_undegraded":"2024-04-16T16:16:41.354451+0000","last_fullsized":"2024-04-16T16:16:41.354451+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:46.723824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.018786+0000","last_change":"2024-04-16T16:16:38.018786+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.018786+0000","last_undegraded":"2024-04-16T16:16:38.018786+0000","last_fullsized":"2024-04-16T16:16:38.018786+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084205+0000","last_change":"2024-04-16T16:16:40.055025+0000","last_active":"2024-04-16T16:16:41.084205+0000","last_peered":"2024-04-16T16:16:41.084205+0000","last_clean":"2024-04-16T16:16:41.084205+0000","last_became_active":"2024-04-16T16:16:40.054757+0000","last_became_peered":"2024-04-16T16:16:40.054757+0000","last_unstale":"2024-04-16T16:16:41.084205+0000","last_undegraded":"2024-04-16T16:16:41.084205+0000","last_fullsized":"2024-04-16T16:16:41.084205+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:31:33.590968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.014465+0000","last_change":"2024-04-16T16:16:38.014465+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.014465+0000","last_undegraded":"2024-04-16T16:16:38.014465+0000","last_fullsized":"2024-04-16T16:16:38.014465+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052671+0000","last_change":"2024-04-16T16:16:40.054326+0000","last_active":"2024-04-16T16:16:41.052671+0000","last_peered":"2024-04-16T16:16:41.052671+0000","last_clean":"2024-04-16T16:16:41.052671+0000","last_became_active":"2024-04-16T16:16:40.054013+0000","last_became_peered":"2024-04-16T16:16:40.054013+0000","last_unstale":"2024-04-16T16:16:41.052671+0000","last_undegraded":"2024-04-16T16:16:41.052671+0000","last_fullsized":"2024-04-16T16:16:41.052671+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:05:08.507551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.007127+0000","last_change":"2024-04-16T16:16:38.007127+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.007127+0000","last_undegraded":"2024-04-16T16:16:38.007127+0000","last_fullsized":"2024-04-16T16:16:38.007127+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.019474+0000","last_change":"2024-04-16T16:16:38.019474+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.019474+0000","last_undegraded":"2024-04-16T16:16:38.019474+0000","last_fullsized":"2024-04-16T16:16:38.019474+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.007563+0000","last_change":"2024-04-16T16:16:38.007563+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.007563+0000","last_undegraded":"2024-04-16T16:16:38.007563+0000","last_fullsized":"2024-04-16T16:16:38.007563+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354958+0000","last_change":"2024-04-16T16:16:40.066424+0000","last_active":"2024-04-16T16:16:41.354958+0000","last_peered":"2024-04-16T16:16:41.354958+0000","last_clean":"2024-04-16T16:16:41.354958+0000","last_became_active":"2024-04-16T16:16:40.063948+0000","last_became_peered":"2024-04-16T16:16:40.063948+0000","last_unstale":"2024-04-16T16:16:41.354958+0000","last_undegraded":"2024-04-16T16:16:41.354958+0000","last_fullsized":"2024-04-16T16:16:41.354958+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:59:04.285019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.017490+0000","last_change":"2024-04-16T16:16:38.017490+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.017490+0000","last_undegraded":"2024-04-16T16:16:38.017490+0000","last_fullsized":"2024-04-16T16:16:38.017490+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053204+0000","last_change":"2024-04-16T16:16:40.052810+0000","last_active":"2024-04-16T16:16:41.053204+0000","last_peered":"2024-04-16T16:16:41.053204+0000","last_clean":"2024-04-16T16:16:41.053204+0000","last_became_active":"2024-04-16T16:16:40.051522+0000","last_became_peered":"2024-04-16T16:16:40.051522+0000","last_unstale":"2024-04-16T16:16:41.053204+0000","last_undegraded":"2024-04-16T16:16:41.053204+0000","last_fullsized":"2024-04-16T16:16:41.053204+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:53:25.229933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.009735+0000","last_change":"2024-04-16T16:16:38.009735+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.009735+0000","last_undegraded":"2024-04-16T16:16:38.009735+0000","last_fullsized":"2024-04-16T16:16:38.009735+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.017425+0000","last_change":"2024-04-16T16:16:38.017425+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.017425+0000","last_undegraded":"2024-04-16T16:16:38.017425+0000","last_fullsized":"2024-04-16T16:16:38.017425+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.013089+0000","last_change":"2024-04-16T16:16:38.013089+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.013089+0000","last_undegraded":"2024-04-16T16:16:38.013089+0000","last_fullsized":"2024-04-16T16:16:38.013089+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.083960+0000","last_change":"2024-04-16T16:16:39.038427+0000","last_active":"2024-04-16T16:16:41.083960+0000","last_peered":"2024-04-16T16:16:41.083960+0000","last_clean":"2024-04-16T16:16:41.083960+0000","last_became_active":"2024-04-16T16:16:39.036287+0000","last_became_peered":"2024-04-16T16:16:39.036287+0000","last_unstale":"2024-04-16T16:16:41.083960+0000","last_undegraded":"2024-04-16T16:16:41.083960+0000","last_fullsized":"2024-04-16T16:16:41.083960+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:38.804660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084649+0000","last_change":"2024-04-16T16:16:40.050769+0000","last_active":"2024-04-16T16:16:41.084649+0000","last_peered":"2024-04-16T16:16:41.084649+0000","last_clean":"2024-04-16T16:16:41.084649+0000","last_became_active":"2024-04-16T16:16:40.050449+0000","last_became_peered":"2024-04-16T16:16:40.050449+0000","last_unstale":"2024-04-16T16:16:41.084649+0000","last_undegraded":"2024-04-16T16:16:41.084649+0000","last_fullsized":"2024-04-16T16:16:41.084649+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:29.398550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.018057+0000","last_change":"2024-04-16T16:16:38.018057+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.018057+0000","last_undegraded":"2024-04-16T16:16:38.018057+0000","last_fullsized":"2024-04-16T16:16:38.018057+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.013899+0000","last_change":"2024-04-16T16:16:38.013899+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.013899+0000","last_undegraded":"2024-04-16T16:16:38.013899+0000","last_fullsized":"2024-04-16T16:16:38.013899+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053323+0000","last_change":"2024-04-16T16:16:39.044030+0000","last_active":"2024-04-16T16:16:41.053323+0000","last_peered":"2024-04-16T16:16:41.053323+0000","last_clean":"2024-04-16T16:16:41.053323+0000","last_became_active":"2024-04-16T16:16:39.043710+0000","last_became_peered":"2024-04-16T16:16:39.043710+0000","last_unstale":"2024-04-16T16:16:41.053323+0000","last_undegraded":"2024-04-16T16:16:41.053323+0000","last_fullsized":"2024-04-16T16:16:41.053323+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:15.221340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084392+0000","last_change":"2024-04-16T16:16:39.048354+0000","last_active":"2024-04-16T16:16:41.084392+0000","last_peered":"2024-04-16T16:16:41.084392+0000","last_clean":"2024-04-16T16:16:41.084392+0000","last_became_active":"2024-04-16T16:16:39.048098+0000","last_became_peered":"2024-04-16T16:16:39.048098+0000","last_unstale":"2024-04-16T16:16:41.084392+0000","last_undegraded":"2024-04-16T16:16:41.084392+0000","last_fullsized":"2024-04-16T16:16:41.084392+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:37:41.504380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052443+0000","last_change":"2024-04-16T16:16:40.061326+0000","last_active":"2024-04-16T16:16:41.052443+0000","last_peered":"2024-04-16T16:16:41.052443+0000","last_clean":"2024-04-16T16:16:41.052443+0000","last_became_active":"2024-04-16T16:16:40.056522+0000","last_became_peered":"2024-04-16T16:16:40.056522+0000","last_unstale":"2024-04-16T16:16:41.052443+0000","last_undegraded":"2024-04-16T16:16:41.052443+0000","last_fullsized":"2024-04-16T16:16:41.052443+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:43:46.183862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052332+0000","last_change":"2024-04-16T16:16:39.051171+0000","last_active":"2024-04-16T16:16:41.052332+0000","last_peered":"2024-04-16T16:16:41.052332+0000","last_clean":"2024-04-16T16:16:41.052332+0000","last_became_active":"2024-04-16T16:16:39.050841+0000","last_became_peered":"2024-04-16T16:16:39.050841+0000","last_unstale":"2024-04-16T16:16:41.052332+0000","last_undegraded":"2024-04-16T16:16:41.052332+0000","last_fullsized":"2024-04-16T16:16:41.052332+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:32:24.457101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354394+0000","last_change":"2024-04-16T16:16:39.046999+0000","last_active":"2024-04-16T16:16:41.354394+0000","last_peered":"2024-04-16T16:16:41.354394+0000","last_clean":"2024-04-16T16:16:41.354394+0000","last_became_active":"2024-04-16T16:16:39.046741+0000","last_became_peered":"2024-04-16T16:16:39.046741+0000","last_unstale":"2024-04-16T16:16:41.354394+0000","last_undegraded":"2024-04-16T16:16:41.354394+0000","last_fullsized":"2024-04-16T16:16:41.354394+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:58:16.807985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354126+0000","last_change":"2024-04-16T16:16:40.057460+0000","last_active":"2024-04-16T16:16:41.354126+0000","last_peered":"2024-04-16T16:16:41.354126+0000","last_clean":"2024-04-16T16:16:41.354126+0000","last_became_active":"2024-04-16T16:16:40.057142+0000","last_became_peered":"2024-04-16T16:16:40.057142+0000","last_unstale":"2024-04-16T16:16:41.354126+0000","last_undegraded":"2024-04-16T16:16:41.354126+0000","last_fullsized":"2024-04-16T16:16:41.354126+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:44.350988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.007182+0000","last_change":"2024-04-16T16:16:38.007182+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.007182+0000","last_undegraded":"2024-04-16T16:16:38.007182+0000","last_fullsized":"2024-04-16T16:16:38.007182+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.016998+0000","last_change":"2024-04-16T16:16:38.016998+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.016998+0000","last_undegraded":"2024-04-16T16:16:38.016998+0000","last_fullsized":"2024-04-16T16:16:38.016998+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.017355+0000","last_change":"2024-04-16T16:16:38.017355+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.017355+0000","last_undegraded":"2024-04-16T16:16:38.017355+0000","last_fullsized":"2024-04-16T16:16:38.017355+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"57'139","reported_seq":345,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-16T16:16:38.009049+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:16:38.009049+0000","last_peered":"2024-04-16T16:16:38.009049+0000","last_clean":"2024-04-16T16:16:38.009049+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:16:38.009049+0000","last_undegraded":"2024-04-16T16:16:38.009049+0000","last_fullsized":"2024-04-16T16:16:38.009049+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":3},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":1646,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1421312,"data_stored":975385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":827392,"data_stored":385017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":827392,"data_stored":385017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":385017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":385017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32048,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339792,"statfs":{"total":96636764160,"available":96603947008,"internally_reserved":0,"allocated":888832,"data_stored":411796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":897024,"data_stored":412412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1470464,"data_stored":1000552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:16:44.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:16:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.841+0000 7f0d26a00640 1 Processor -- start 2024-04-16T16:16:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.841+0000 7f0d26a00640 1 -- start start 2024-04-16T16:16:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.843+0000 7f0d26a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0d20163010 0x7f0d200cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.843+0000 7f0d26a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d2006f540 0x7f0d201645b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.843+0000 7f0d26a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d20163510 con 0x7f0d200cde00 2024-04-16T16:16:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.843+0000 7f0d26a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0d201637f0 con 0x7f0d20163010 2024-04-16T16:16:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.843+0000 7f0d26a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0d20163ad0 con 0x7f0d2006f540 2024-04-16T16:16:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.844+0000 7f0d24c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0d20163010 0x7f0d200cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.844+0000 7f0d26000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d2006f540 0x7f0d201645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.844+0000 7f0d24c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0d20163010 0x7f0d200cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:54738/0 (socket says 172.21.15.45:54738) 2024-04-16T16:16:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.844+0000 7f0d24c00640 1 -- 172.21.15.45:0/1485530714 learned_addr learned my addr 172.21.15.45:0/1485530714 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.845+0000 7f0d24c00640 1 -- 172.21.15.45:0/1485530714 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d2006f540 msgr2=0x7f0d201645b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.845+0000 7f0d24c00640 1 --2- 172.21.15.45:0/1485530714 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d2006f540 0x7f0d201645b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.845+0000 7f0d24c00640 1 -- 172.21.15.45:0/1485530714 >> v1:172.21.15.103:6789/0 conn(0x7f0d200cde00 legacy=0x7f0d200ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.845+0000 7f0d24c00640 1 -- 172.21.15.45:0/1485530714 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d20163db0 con 0x7f0d20163010 2024-04-16T16:16:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.845+0000 7f0d26000640 1 --2- 172.21.15.45:0/1485530714 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d2006f540 0x7f0d201645b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:16:44.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.846+0000 7f0d24c00640 1 --2- 172.21.15.45:0/1485530714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d20163010 0x7f0d200cff30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c009870 tx=0x7f0d0c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd822900e29e2cde server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:44.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.847+0000 7f0d1fe00640 1 -- 172.21.15.45:0/1485530714 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c034070 con 0x7f0d20163010 2024-04-16T16:16:44.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.847+0000 7f0d1fe00640 1 -- 172.21.15.45:0/1485530714 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d0c030840 con 0x7f0d20163010 2024-04-16T16:16:44.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.847+0000 7f0d1fe00640 1 -- 172.21.15.45:0/1485530714 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c030c10 con 0x7f0d20163010 2024-04-16T16:16:44.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.848+0000 7f0d26a00640 1 -- 172.21.15.45:0/1485530714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d20163010 msgr2=0x7f0d200cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:44.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.848+0000 7f0d26a00640 1 --2- 172.21.15.45:0/1485530714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d20163010 0x7f0d200cff30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c009870 tx=0x7f0d0c02b980 comp rx=0 tx=0).stop 2024-04-16T16:16:44.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.848+0000 7f0d26a00640 1 -- 172.21.15.45:0/1485530714 shutdown_connections 2024-04-16T16:16:44.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.848+0000 7f0d26a00640 1 --2- 172.21.15.45:0/1485530714 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d2006f540 0x7f0d201645b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.848+0000 7f0d26a00640 1 --2- 172.21.15.45:0/1485530714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d20163010 0x7f0d200cff30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.848+0000 7f0d26a00640 1 -- 172.21.15.45:0/1485530714 >> 172.21.15.45:0/1485530714 conn(0x7f0d2006b930 msgr2=0x7f0d2006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:44.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.848+0000 7f0d26a00640 1 -- 172.21.15.45:0/1485530714 shutdown_connections 2024-04-16T16:16:44.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.848+0000 7f0d26a00640 1 -- 172.21.15.45:0/1485530714 wait complete. 2024-04-16T16:16:44.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.849+0000 7f0d26a00640 1 Processor -- start 2024-04-16T16:16:44.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.850+0000 7f0d26a00640 1 -- start start 2024-04-16T16:16:44.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.851+0000 7f0d26a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d2006f540 0x7f0d2013f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:44.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.851+0000 7f0d26a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d200cde00 0x7f0d2013ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:44.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.851+0000 7f0d26a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d20163010 0x7f0d200b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:44.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.851+0000 7f0d26a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20164af0 con 0x7f0d2006f540 2024-04-16T16:16:44.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.851+0000 7f0d26a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2006f970 con 0x7f0d20163010 2024-04-16T16:16:44.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.851+0000 7f0d26a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0d20164c70 con 0x7f0d200cde00 2024-04-16T16:16:44.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.851+0000 7f0d25600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d2006f540 0x7f0d2013f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:44.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.851+0000 7f0d25600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d2006f540 0x7f0d2013f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46490/0 (socket says 172.21.15.45:46490) 2024-04-16T16:16:44.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.851+0000 7f0d25600640 1 -- 172.21.15.45:0/2205427988 learned_addr learned my addr 172.21.15.45:0/2205427988 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:44.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.851+0000 7f0d24c00640 1 --2- 172.21.15.45:0/2205427988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d200cde00 0x7f0d2013ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:44.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.851+0000 7f0d26000640 1 --2- 172.21.15.45:0/2205427988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d20163010 0x7f0d200b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:44.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.852+0000 7f0d25600640 1 -- 172.21.15.45:0/2205427988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d200cde00 msgr2=0x7f0d2013ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:44.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.852+0000 7f0d25600640 1 --2- 172.21.15.45:0/2205427988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d200cde00 0x7f0d2013ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.852+0000 7f0d25600640 1 -- 172.21.15.45:0/2205427988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d20163010 msgr2=0x7f0d200b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:44.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.852+0000 7f0d25600640 1 --2- 172.21.15.45:0/2205427988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d20163010 0x7f0d200b4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:44.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.852+0000 7f0d25600640 1 -- 172.21.15.45:0/2205427988 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d2013f1a0 con 0x7f0d2006f540 2024-04-16T16:16:44.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.852+0000 7f0d26000640 1 --2- 172.21.15.45:0/2205427988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d20163010 0x7f0d200b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:16:44.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.852+0000 7f0d25600640 1 --2- 172.21.15.45:0/2205427988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d2006f540 0x7f0d2013f4b0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0d1400c7c0 tx=0x7f0d1400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.852+0000 7f0d1d600640 1 -- 172.21.15.45:0/2205427988 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d14003b90 con 0x7f0d2006f540 2024-04-16T16:16:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.852+0000 7f0d1d600640 1 -- 172.21.15.45:0/2205427988 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d14003d30 con 0x7f0d2006f540 2024-04-16T16:16:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.852+0000 7f0d1d600640 1 -- 172.21.15.45:0/2205427988 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1400aa90 con 0x7f0d2006f540 2024-04-16T16:16:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.852+0000 7f0d26a00640 1 -- 172.21.15.45:0/2205427988 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d2013fa90 con 0x7f0d2006f540 2024-04-16T16:16:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.853+0000 7f0d26a00640 1 -- 172.21.15.45:0/2205427988 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d20140050 con 0x7f0d2006f540 2024-04-16T16:16:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.854+0000 7f0d26a00640 1 -- 172.21.15.45:0/2205427988 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d20075780 con 0x7f0d2006f540 2024-04-16T16:16:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.855+0000 7f0d1d600640 1 -- 172.21.15.45:0/2205427988 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f0d1400ac30 con 0x7f0d2006f540 2024-04-16T16:16:44.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.856+0000 7f0d1d600640 1 --2- 172.21.15.45:0/2205427988 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0d0006cb30 0x7f0d0006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:44.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.856+0000 7f0d1d600640 1 -- 172.21.15.45:0/2205427988 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f0d1408bfa0 con 0x7f0d2006f540 2024-04-16T16:16:44.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.857+0000 7f0d24c00640 1 --2- 172.21.15.45:0/2205427988 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0d0006cb30 0x7f0d0006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:44.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.858+0000 7f0d24c00640 1 --2- 172.21.15.45:0/2205427988 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0d0006cb30 0x7f0d0006eff0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c036000 tx=0x7f0d0c03b040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:44.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:44.861+0000 7f0d1d600640 1 -- 172.21.15.45:0/2205427988 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d14059fd0 con 0x7f0d2006f540 2024-04-16T16:16:44.973 INFO:tasks.ceph.mds.a.smithi045.stderr:2024-04-16T16:16:44.973+0000 7fbe89a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:16:45.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.062+0000 7f0d26a00640 1 -- 172.21.15.45:0/2205427988 --> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0d2006c180 con 0x7f0d0006cb30 2024-04-16T16:16:45.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.082+0000 7f0d1d600640 1 -- 172.21.15.45:0/2205427988 <== mgr.4961 v2:172.21.15.103:6832/4155603915 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+244136 (secure 0 0 0) 0x7f0d2006c180 con 0x7f0d0006cb30 2024-04-16T16:16:45.083 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:45.087 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:16:45.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.090+0000 7f0d26a00640 1 -- 172.21.15.45:0/2205427988 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0d0006cb30 msgr2=0x7f0d0006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:45.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.090+0000 7f0d26a00640 1 --2- 172.21.15.45:0/2205427988 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0d0006cb30 0x7f0d0006eff0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c036000 tx=0x7f0d0c03b040 comp rx=0 tx=0).stop 2024-04-16T16:16:45.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.090+0000 7f0d26a00640 1 -- 172.21.15.45:0/2205427988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d2006f540 msgr2=0x7f0d2013f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:45.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.090+0000 7f0d26a00640 1 --2- 172.21.15.45:0/2205427988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d2006f540 0x7f0d2013f4b0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0d1400c7c0 tx=0x7f0d1400cc80 comp rx=0 tx=0).stop 2024-04-16T16:16:45.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.090+0000 7f0d26a00640 1 -- 172.21.15.45:0/2205427988 shutdown_connections 2024-04-16T16:16:45.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.090+0000 7f0d26a00640 1 --2- 172.21.15.45:0/2205427988 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f0d0006cb30 0x7f0d0006eff0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.090+0000 7f0d26a00640 1 --2- 172.21.15.45:0/2205427988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d20163010 0x7f0d200b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.090+0000 7f0d26a00640 1 --2- 172.21.15.45:0/2205427988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d200cde00 0x7f0d2013ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.090+0000 7f0d26a00640 1 --2- 172.21.15.45:0/2205427988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d2006f540 0x7f0d2013f4b0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.090+0000 7f0d26a00640 1 -- 172.21.15.45:0/2205427988 >> 172.21.15.45:0/2205427988 conn(0x7f0d2006b930 msgr2=0x7f0d20165d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:45.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.090+0000 7f0d26a00640 1 -- 172.21.15.45:0/2205427988 shutdown_connections 2024-04-16T16:16:45.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.090+0000 7f0d26a00640 1 -- 172.21.15.45:0/2205427988 wait complete. 2024-04-16T16:16:45.113 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-16T16:16:43.065817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592014,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":208,"num_write_kb":2426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":108,"acting":108,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":44,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248940,"kb_used_data":7800,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754725780,"statfs":{"total":773094113280,"available":772839198720,"internally_reserved":0,"allocated":7987200,"data_stored":4340213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1646,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.064145"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354998+0000","last_change":"2024-04-16T16:16:40.067374+0000","last_active":"2024-04-16T16:16:41.354998+0000","last_peered":"2024-04-16T16:16:41.354998+0000","last_clean":"2024-04-16T16:16:41.354998+0000","last_became_active":"2024-04-16T16:16:40.064314+0000","last_became_peered":"2024-04-16T16:16:40.064314+0000","last_unstale":"2024-04-16T16:16:41.354998+0000","last_undegraded":"2024-04-16T16:16:41.354998+0000","last_fullsized":"2024-04-16T16:16:41.354998+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:47:33.049888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052264+0000","last_change":"2024-04-16T16:16:40.053258+0000","last_active":"2024-04-16T16:16:41.052264+0000","last_peered":"2024-04-16T16:16:41.052264+0000","last_clean":"2024-04-16T16:16:41.052264+0000","last_became_active":"2024-04-16T16:16:40.051051+0000","last_became_peered":"2024-04-16T16:16:40.051051+0000","last_unstale":"2024-04-16T16:16:41.052264+0000","last_undegraded":"2024-04-16T16:16:41.052264+0000","last_fullsized":"2024-04-16T16:16:41.052264+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:16:10.608170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053974+0000","last_change":"2024-04-16T16:16:40.065283+0000","last_active":"2024-04-16T16:16:41.053974+0000","last_peered":"2024-04-16T16:16:41.053974+0000","last_clean":"2024-04-16T16:16:41.053974+0000","last_became_active":"2024-04-16T16:16:40.064211+0000","last_became_peered":"2024-04-16T16:16:40.064211+0000","last_unstale":"2024-04-16T16:16:41.053974+0000","last_undegraded":"2024-04-16T16:16:41.053974+0000","last_fullsized":"2024-04-16T16:16:41.053974+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:28.836553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354796+0000","last_change":"2024-04-16T16:16:40.070880+0000","last_active":"2024-04-16T16:16:41.354796+0000","last_peered":"2024-04-16T16:16:41.354796+0000","last_clean":"2024-04-16T16:16:41.354796+0000","last_became_active":"2024-04-16T16:16:40.070434+0000","last_became_peered":"2024-04-16T16:16:40.070434+0000","last_unstale":"2024-04-16T16:16:41.354796+0000","last_undegraded":"2024-04-16T16:16:41.354796+0000","last_fullsized":"2024-04-16T16:16:41.354796+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:13:47.015413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084447+0000","last_change":"2024-04-16T16:16:40.053229+0000","last_active":"2024-04-16T16:16:41.084447+0000","last_peered":"2024-04-16T16:16:41.084447+0000","last_clean":"2024-04-16T16:16:41.084447+0000","last_became_active":"2024-04-16T16:16:40.052968+0000","last_became_peered":"2024-04-16T16:16:40.052968+0000","last_unstale":"2024-04-16T16:16:41.084447+0000","last_undegraded":"2024-04-16T16:16:41.084447+0000","last_fullsized":"2024-04-16T16:16:41.084447+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:46.618649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052568+0000","last_change":"2024-04-16T16:16:40.059138+0000","last_active":"2024-04-16T16:16:41.052568+0000","last_peered":"2024-04-16T16:16:41.052568+0000","last_clean":"2024-04-16T16:16:41.052568+0000","last_became_active":"2024-04-16T16:16:40.056131+0000","last_became_peered":"2024-04-16T16:16:40.056131+0000","last_unstale":"2024-04-16T16:16:41.052568+0000","last_undegraded":"2024-04-16T16:16:41.052568+0000","last_fullsized":"2024-04-16T16:16:41.052568+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:39:52.139845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084236+0000","last_change":"2024-04-16T16:16:40.052923+0000","last_active":"2024-04-16T16:16:41.084236+0000","last_peered":"2024-04-16T16:16:41.084236+0000","last_clean":"2024-04-16T16:16:41.084236+0000","last_became_active":"2024-04-16T16:16:40.052609+0000","last_became_peered":"2024-04-16T16:16:40.052609+0000","last_unstale":"2024-04-16T16:16:41.084236+0000","last_undegraded":"2024-04-16T16:16:41.084236+0000","last_fullsized":"2024-04-16T16:16:41.084236+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:21.332023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054029+0000","last_change":"2024-04-16T16:16:40.069834+0000","last_active":"2024-04-16T16:16:41.054029+0000","last_peered":"2024-04-16T16:16:41.054029+0000","last_clean":"2024-04-16T16:16:41.054029+0000","last_became_active":"2024-04-16T16:16:40.062429+0000","last_became_peered":"2024-04-16T16:16:40.062429+0000","last_unstale":"2024-04-16T16:16:41.054029+0000","last_undegraded":"2024-04-16T16:16:41.054029+0000","last_fullsized":"2024-04-16T16:16:41.054029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:14:20.465129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355562+0000","last_change":"2024-04-16T16:16:40.055791+0000","last_active":"2024-04-16T16:16:41.355562+0000","last_peered":"2024-04-16T16:16:41.355562+0000","last_clean":"2024-04-16T16:16:41.355562+0000","last_became_active":"2024-04-16T16:16:40.055508+0000","last_became_peered":"2024-04-16T16:16:40.055508+0000","last_unstale":"2024-04-16T16:16:41.355562+0000","last_undegraded":"2024-04-16T16:16:41.355562+0000","last_fullsized":"2024-04-16T16:16:41.355562+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:44.062254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084256+0000","last_change":"2024-04-16T16:16:40.058979+0000","last_active":"2024-04-16T16:16:41.084256+0000","last_peered":"2024-04-16T16:16:41.084256+0000","last_clean":"2024-04-16T16:16:41.084256+0000","last_became_active":"2024-04-16T16:16:40.058648+0000","last_became_peered":"2024-04-16T16:16:40.058648+0000","last_unstale":"2024-04-16T16:16:41.084256+0000","last_undegraded":"2024-04-16T16:16:41.084256+0000","last_fullsized":"2024-04-16T16:16:41.084256+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:28:41.607987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.010949+0000","last_change":"2024-04-16T16:16:38.010949+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.010949+0000","last_undegraded":"2024-04-16T16:16:38.010949+0000","last_fullsized":"2024-04-16T16:16:38.010949+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054498+0000","last_change":"2024-04-16T16:16:40.063884+0000","last_active":"2024-04-16T16:16:41.054498+0000","last_peered":"2024-04-16T16:16:41.054498+0000","last_clean":"2024-04-16T16:16:41.054498+0000","last_became_active":"2024-04-16T16:16:40.061609+0000","last_became_peered":"2024-04-16T16:16:40.061609+0000","last_unstale":"2024-04-16T16:16:41.054498+0000","last_undegraded":"2024-04-16T16:16:41.054498+0000","last_fullsized":"2024-04-16T16:16:41.054498+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:55.827541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084571+0000","last_change":"2024-04-16T16:16:39.035506+0000","last_active":"2024-04-16T16:16:41.084571+0000","last_peered":"2024-04-16T16:16:41.084571+0000","last_clean":"2024-04-16T16:16:41.084571+0000","last_became_active":"2024-04-16T16:16:39.035196+0000","last_became_peered":"2024-04-16T16:16:39.035196+0000","last_unstale":"2024-04-16T16:16:41.084571+0000","last_undegraded":"2024-04-16T16:16:41.084571+0000","last_fullsized":"2024-04-16T16:16:41.084571+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:29:14.659131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.007452+0000","last_change":"2024-04-16T16:16:38.007452+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.007452+0000","last_undegraded":"2024-04-16T16:16:38.007452+0000","last_fullsized":"2024-04-16T16:16:38.007452+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.018161+0000","last_change":"2024-04-16T16:16:38.018161+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.018161+0000","last_undegraded":"2024-04-16T16:16:38.018161+0000","last_fullsized":"2024-04-16T16:16:38.018161+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.012103+0000","last_change":"2024-04-16T16:16:38.012103+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.012103+0000","last_undegraded":"2024-04-16T16:16:38.012103+0000","last_fullsized":"2024-04-16T16:16:38.012103+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.021097+0000","last_change":"2024-04-16T16:16:38.021097+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.021097+0000","last_undegraded":"2024-04-16T16:16:38.021097+0000","last_fullsized":"2024-04-16T16:16:38.021097+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.051911+0000","last_change":"2024-04-16T16:16:40.049943+0000","last_active":"2024-04-16T16:16:41.051911+0000","last_peered":"2024-04-16T16:16:41.051911+0000","last_clean":"2024-04-16T16:16:41.051911+0000","last_became_active":"2024-04-16T16:16:40.049466+0000","last_became_peered":"2024-04-16T16:16:40.049466+0000","last_unstale":"2024-04-16T16:16:41.051911+0000","last_undegraded":"2024-04-16T16:16:41.051911+0000","last_fullsized":"2024-04-16T16:16:41.051911+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:24.082098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353887+0000","last_change":"2024-04-16T16:16:39.048665+0000","last_active":"2024-04-16T16:16:41.353887+0000","last_peered":"2024-04-16T16:16:41.353887+0000","last_clean":"2024-04-16T16:16:41.353887+0000","last_became_active":"2024-04-16T16:16:39.046018+0000","last_became_peered":"2024-04-16T16:16:39.046018+0000","last_unstale":"2024-04-16T16:16:41.353887+0000","last_undegraded":"2024-04-16T16:16:41.353887+0000","last_fullsized":"2024-04-16T16:16:41.353887+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:37:31.768292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084230+0000","last_change":"2024-04-16T16:16:40.068846+0000","last_active":"2024-04-16T16:16:41.084230+0000","last_peered":"2024-04-16T16:16:41.084230+0000","last_clean":"2024-04-16T16:16:41.084230+0000","last_became_active":"2024-04-16T16:16:40.060719+0000","last_became_peered":"2024-04-16T16:16:40.060719+0000","last_unstale":"2024-04-16T16:16:41.084230+0000","last_undegraded":"2024-04-16T16:16:41.084230+0000","last_fullsized":"2024-04-16T16:16:41.084230+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:48:35.347056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.051833+0000","last_change":"2024-04-16T16:16:39.042020+0000","last_active":"2024-04-16T16:16:41.051833+0000","last_peered":"2024-04-16T16:16:41.051833+0000","last_clean":"2024-04-16T16:16:41.051833+0000","last_became_active":"2024-04-16T16:16:39.041447+0000","last_became_peered":"2024-04-16T16:16:39.041447+0000","last_unstale":"2024-04-16T16:16:41.051833+0000","last_undegraded":"2024-04-16T16:16:41.051833+0000","last_fullsized":"2024-04-16T16:16:41.051833+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:18:24.097920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.007668+0000","last_change":"2024-04-16T16:16:38.007668+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.007668+0000","last_undegraded":"2024-04-16T16:16:38.007668+0000","last_fullsized":"2024-04-16T16:16:38.007668+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.018790+0000","last_change":"2024-04-16T16:16:38.018790+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.018790+0000","last_undegraded":"2024-04-16T16:16:38.018790+0000","last_fullsized":"2024-04-16T16:16:38.018790+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.012305+0000","last_change":"2024-04-16T16:16:38.012305+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.012305+0000","last_undegraded":"2024-04-16T16:16:38.012305+0000","last_fullsized":"2024-04-16T16:16:38.012305+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053162+0000","last_change":"2024-04-16T16:16:40.056836+0000","last_active":"2024-04-16T16:16:41.053162+0000","last_peered":"2024-04-16T16:16:41.053162+0000","last_clean":"2024-04-16T16:16:41.053162+0000","last_became_active":"2024-04-16T16:16:40.056365+0000","last_became_peered":"2024-04-16T16:16:40.056365+0000","last_unstale":"2024-04-16T16:16:41.053162+0000","last_undegraded":"2024-04-16T16:16:41.053162+0000","last_fullsized":"2024-04-16T16:16:41.053162+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:17:23.179723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"71'6","reported_seq":20,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355016+0000","last_change":"2024-04-16T16:16:39.040106+0000","last_active":"2024-04-16T16:16:41.355016+0000","last_peered":"2024-04-16T16:16:41.355016+0000","last_clean":"2024-04-16T16:16:41.355016+0000","last_became_active":"2024-04-16T16:16:39.039334+0000","last_became_peered":"2024-04-16T16:16:39.039334+0000","last_unstale":"2024-04-16T16:16:41.355016+0000","last_undegraded":"2024-04-16T16:16:41.355016+0000","last_fullsized":"2024-04-16T16:16:41.355016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:29.791246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.13","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354451+0000","last_change":"2024-04-16T16:16:39.048257+0000","last_active":"2024-04-16T16:16:41.354451+0000","last_peered":"2024-04-16T16:16:41.354451+0000","last_clean":"2024-04-16T16:16:41.354451+0000","last_became_active":"2024-04-16T16:16:39.047835+0000","last_became_peered":"2024-04-16T16:16:39.047835+0000","last_unstale":"2024-04-16T16:16:41.354451+0000","last_undegraded":"2024-04-16T16:16:41.354451+0000","last_fullsized":"2024-04-16T16:16:41.354451+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:46.723824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.018786+0000","last_change":"2024-04-16T16:16:38.018786+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.018786+0000","last_undegraded":"2024-04-16T16:16:38.018786+0000","last_fullsized":"2024-04-16T16:16:38.018786+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084205+0000","last_change":"2024-04-16T16:16:40.055025+0000","last_active":"2024-04-16T16:16:41.084205+0000","last_peered":"2024-04-16T16:16:41.084205+0000","last_clean":"2024-04-16T16:16:41.084205+0000","last_became_active":"2024-04-16T16:16:40.054757+0000","last_became_peered":"2024-04-16T16:16:40.054757+0000","last_unstale":"2024-04-16T16:16:41.084205+0000","last_undegraded":"2024-04-16T16:16:41.084205+0000","last_fullsized":"2024-04-16T16:16:41.084205+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:31:33.590968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.014465+0000","last_change":"2024-04-16T16:16:38.014465+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.014465+0000","last_undegraded":"2024-04-16T16:16:38.014465+0000","last_fullsized":"2024-04-16T16:16:38.014465+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052671+0000","last_change":"2024-04-16T16:16:40.054326+0000","last_active":"2024-04-16T16:16:41.052671+0000","last_peered":"2024-04-16T16:16:41.052671+0000","last_clean":"2024-04-16T16:16:41.052671+0000","last_became_active":"2024-04-16T16:16:40.054013+0000","last_became_peered":"2024-04-16T16:16:40.054013+0000","last_unstale":"2024-04-16T16:16:41.052671+0000","last_undegraded":"2024-04-16T16:16:41.052671+0000","last_fullsized":"2024-04-16T16:16:41.052671+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:05:08.507551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.007127+0000","last_change":"2024-04-16T16:16:38.007127+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.007127+0000","last_undegraded":"2024-04-16T16:16:38.007127+0000","last_fullsized":"2024-04-16T16:16:38.007127+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.019474+0000","last_change":"2024-04-16T16:16:38.019474+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.019474+0000","last_undegraded":"2024-04-16T16:16:38.019474+0000","last_fullsized":"2024-04-16T16:16:38.019474+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.007563+0000","last_change":"2024-04-16T16:16:38.007563+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.007563+0000","last_undegraded":"2024-04-16T16:16:38.007563+0000","last_fullsized":"2024-04-16T16:16:38.007563+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354958+0000","last_change":"2024-04-16T16:16:40.066424+0000","last_active":"2024-04-16T16:16:41.354958+0000","last_peered":"2024-04-16T16:16:41.354958+0000","last_clean":"2024-04-16T16:16:41.354958+0000","last_became_active":"2024-04-16T16:16:40.063948+0000","last_became_peered":"2024-04-16T16:16:40.063948+0000","last_unstale":"2024-04-16T16:16:41.354958+0000","last_undegraded":"2024-04-16T16:16:41.354958+0000","last_fullsized":"2024-04-16T16:16:41.354958+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:59:04.285019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.017490+0000","last_change":"2024-04-16T16:16:38.017490+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.017490+0000","last_undegraded":"2024-04-16T16:16:38.017490+0000","last_fullsized":"2024-04-16T16:16:38.017490+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053204+0000","last_change":"2024-04-16T16:16:40.052810+0000","last_active":"2024-04-16T16:16:41.053204+0000","last_peered":"2024-04-16T16:16:41.053204+0000","last_clean":"2024-04-16T16:16:41.053204+0000","last_became_active":"2024-04-16T16:16:40.051522+0000","last_became_peered":"2024-04-16T16:16:40.051522+0000","last_unstale":"2024-04-16T16:16:41.053204+0000","last_undegraded":"2024-04-16T16:16:41.053204+0000","last_fullsized":"2024-04-16T16:16:41.053204+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:53:25.229933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.009735+0000","last_change":"2024-04-16T16:16:38.009735+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.009735+0000","last_undegraded":"2024-04-16T16:16:38.009735+0000","last_fullsized":"2024-04-16T16:16:38.009735+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.017425+0000","last_change":"2024-04-16T16:16:38.017425+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.017425+0000","last_undegraded":"2024-04-16T16:16:38.017425+0000","last_fullsized":"2024-04-16T16:16:38.017425+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.013089+0000","last_change":"2024-04-16T16:16:38.013089+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.013089+0000","last_undegraded":"2024-04-16T16:16:38.013089+0000","last_fullsized":"2024-04-16T16:16:38.013089+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.083960+0000","last_change":"2024-04-16T16:16:39.038427+0000","last_active":"2024-04-16T16:16:41.083960+0000","last_peered":"2024-04-16T16:16:41.083960+0000","last_clean":"2024-04-16T16:16:41.083960+0000","last_became_active":"2024-04-16T16:16:39.036287+0000","last_became_peered":"2024-04-16T16:16:39.036287+0000","last_unstale":"2024-04-16T16:16:41.083960+0000","last_undegraded":"2024-04-16T16:16:41.083960+0000","last_fullsized":"2024-04-16T16:16:41.083960+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:38.804660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084649+0000","last_change":"2024-04-16T16:16:40.050769+0000","last_active":"2024-04-16T16:16:41.084649+0000","last_peered":"2024-04-16T16:16:41.084649+0000","last_clean":"2024-04-16T16:16:41.084649+0000","last_became_active":"2024-04-16T16:16:40.050449+0000","last_became_peered":"2024-04-16T16:16:40.050449+0000","last_unstale":"2024-04-16T16:16:41.084649+0000","last_undegraded":"2024-04-16T16:16:41.084649+0000","last_fullsized":"2024-04-16T16:16:41.084649+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:29.398550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.018057+0000","last_change":"2024-04-16T16:16:38.018057+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.018057+0000","last_undegraded":"2024-04-16T16:16:38.018057+0000","last_fullsized":"2024-04-16T16:16:38.018057+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.013899+0000","last_change":"2024-04-16T16:16:38.013899+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.013899+0000","last_undegraded":"2024-04-16T16:16:38.013899+0000","last_fullsized":"2024-04-16T16:16:38.013899+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053323+0000","last_change":"2024-04-16T16:16:39.044030+0000","last_active":"2024-04-16T16:16:41.053323+0000","last_peered":"2024-04-16T16:16:41.053323+0000","last_clean":"2024-04-16T16:16:41.053323+0000","last_became_active":"2024-04-16T16:16:39.043710+0000","last_became_peered":"2024-04-16T16:16:39.043710+0000","last_unstale":"2024-04-16T16:16:41.053323+0000","last_undegraded":"2024-04-16T16:16:41.053323+0000","last_fullsized":"2024-04-16T16:16:41.053323+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:15.221340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084392+0000","last_change":"2024-04-16T16:16:39.048354+0000","last_active":"2024-04-16T16:16:41.084392+0000","last_peered":"2024-04-16T16:16:41.084392+0000","last_clean":"2024-04-16T16:16:41.084392+0000","last_became_active":"2024-04-16T16:16:39.048098+0000","last_became_peered":"2024-04-16T16:16:39.048098+0000","last_unstale":"2024-04-16T16:16:41.084392+0000","last_undegraded":"2024-04-16T16:16:41.084392+0000","last_fullsized":"2024-04-16T16:16:41.084392+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:37:41.504380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052443+0000","last_change":"2024-04-16T16:16:40.061326+0000","last_active":"2024-04-16T16:16:41.052443+0000","last_peered":"2024-04-16T16:16:41.052443+0000","last_clean":"2024-04-16T16:16:41.052443+0000","last_became_active":"2024-04-16T16:16:40.056522+0000","last_became_peered":"2024-04-16T16:16:40.056522+0000","last_unstale":"2024-04-16T16:16:41.052443+0000","last_undegraded":"2024-04-16T16:16:41.052443+0000","last_fullsized":"2024-04-16T16:16:41.052443+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:43:46.183862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052332+0000","last_change":"2024-04-16T16:16:39.051171+0000","last_active":"2024-04-16T16:16:41.052332+0000","last_peered":"2024-04-16T16:16:41.052332+0000","last_clean":"2024-04-16T16:16:41.052332+0000","last_became_active":"2024-04-16T16:16:39.050841+0000","last_became_peered":"2024-04-16T16:16:39.050841+0000","last_unstale":"2024-04-16T16:16:41.052332+0000","last_undegraded":"2024-04-16T16:16:41.052332+0000","last_fullsized":"2024-04-16T16:16:41.052332+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:32:24.457101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354394+0000","last_change":"2024-04-16T16:16:39.046999+0000","last_active":"2024-04-16T16:16:41.354394+0000","last_peered":"2024-04-16T16:16:41.354394+0000","last_clean":"2024-04-16T16:16:41.354394+0000","last_became_active":"2024-04-16T16:16:39.046741+0000","last_became_peered":"2024-04-16T16:16:39.046741+0000","last_unstale":"2024-04-16T16:16:41.354394+0000","last_undegraded":"2024-04-16T16:16:41.354394+0000","last_fullsized":"2024-04-16T16:16:41.354394+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:58:16.807985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354126+0000","last_change":"2024-04-16T16:16:40.057460+0000","last_active":"2024-04-16T16:16:41.354126+0000","last_peered":"2024-04-16T16:16:41.354126+0000","last_clean":"2024-04-16T16:16:41.354126+0000","last_became_active":"2024-04-16T16:16:40.057142+0000","last_became_peered":"2024-04-16T16:16:40.057142+0000","last_unstale":"2024-04-16T16:16:41.354126+0000","last_undegraded":"2024-04-16T16:16:41.354126+0000","last_fullsized":"2024-04-16T16:16:41.354126+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:44.350988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.007182+0000","last_change":"2024-04-16T16:16:38.007182+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.007182+0000","last_undegraded":"2024-04-16T16:16:38.007182+0000","last_fullsized":"2024-04-16T16:16:38.007182+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.016998+0000","last_change":"2024-04-16T16:16:38.016998+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.016998+0000","last_undegraded":"2024-04-16T16:16:38.016998+0000","last_fullsized":"2024-04-16T16:16:38.016998+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:16:38.998393+0000","last_change":"2024-04-16T16:16:38.998393+0000","last_active":"2024-04-16T16:16:38.998393+0000","last_peered":"2024-04-16T16:16:38.998393+0000","last_clean":"2024-04-16T16:16:38.998393+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.998393+0000","last_undegraded":"2024-04-16T16:16:38.998393+0000","last_fullsized":"2024-04-16T16:16:38.998393+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-04-16T16:16:38.017355+0000","last_change":"2024-04-16T16:16:38.017355+0000","last_active":"2024-04-16T16:16:37.982377+0000","last_peered":"2024-04-16T16:16:37.982377+0000","last_clean":"2024-04-16T16:16:37.982377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:16:38.017355+0000","last_undegraded":"2024-04-16T16:16:38.017355+0000","last_fullsized":"2024-04-16T16:16:38.017355+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"57'139","reported_seq":345,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-16T16:16:38.009049+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:16:38.009049+0000","last_peered":"2024-04-16T16:16:38.009049+0000","last_clean":"2024-04-16T16:16:38.009049+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:16:38.009049+0000","last_undegraded":"2024-04-16T16:16:38.009049+0000","last_fullsized":"2024-04-16T16:16:38.009049+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":3},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":1646,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1421312,"data_stored":975385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":827392,"data_stored":385017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":827392,"data_stored":385017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":385017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":385017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32048,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339792,"statfs":{"total":96636764160,"available":96603947008,"internally_reserved":0,"allocated":888832,"data_stored":411796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":897024,"data_stored":412412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1470464,"data_stored":1000552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:16:45.116 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:16:45.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.384+0000 7fb6eb200640 1 Processor -- start 2024-04-16T16:16:45.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.384+0000 7fb6eb200640 1 -- start start 2024-04-16T16:16:45.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.385+0000 7fb6eb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e4072150 0x7fb6e40b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.386+0000 7fb6eb200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6e41633c0 con 0x7fb6e415eb00 2024-04-16T16:16:45.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.386+0000 7fb6eb200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6e40b5d40 con 0x7fb6e4162f40 2024-04-16T16:16:45.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.386+0000 7fb6eb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e40b6020 con 0x7fb6e4072150 2024-04-16T16:16:45.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.386+0000 7fb6e9e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb6e415eb00 0x7fb6e4071a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35714/0 (socket says 172.21.15.45:35714) 2024-04-16T16:16:45.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.386+0000 7fb6e9e00640 1 -- 172.21.15.45:0/725620807 learned_addr learned my addr 172.21.15.45:0/725620807 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:45.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.386+0000 7fb6e9400640 1 --2- 172.21.15.45:0/725620807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e4072150 0x7fb6e40b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:45.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e8a00640 1 -- 172.21.15.45:0/725620807 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 316333359 0 0) 0x7fb6e41633c0 con 0x7fb6e415eb00 2024-04-16T16:16:45.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e8a00640 1 -- 172.21.15.45:0/725620807 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6cc003520 con 0x7fb6e415eb00 2024-04-16T16:16:45.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e8a00640 1 -- 172.21.15.45:0/725620807 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 489567570 0 0) 0x7fb6cc003520 con 0x7fb6e415eb00 2024-04-16T16:16:45.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e8a00640 1 -- 172.21.15.45:0/725620807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e4072150 msgr2=0x7fb6e40b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:45.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e8a00640 1 --2- 172.21.15.45:0/725620807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e4072150 0x7fb6e40b56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e8a00640 1 -- 172.21.15.45:0/725620807 >> v1:172.21.15.103:6789/0 conn(0x7fb6e4162f40 legacy=0x7fb6e406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:45.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e8a00640 1 -- 172.21.15.45:0/725620807 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e40b6300 con 0x7fb6e415eb00 2024-04-16T16:16:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e8a00640 1 -- 172.21.15.45:0/725620807 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fb6d8001670 con 0x7fb6e415eb00 2024-04-16T16:16:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e8a00640 1 -- 172.21.15.45:0/725620807 >> v1:172.21.15.45:6789/0 conn(0x7fb6e415eb00 legacy=0x7fb6e4071a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e8a00640 1 --2- 172.21.15.45:0/725620807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e4072150 0x7fb6cc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e8a00640 1 --2- 172.21.15.45:0/725620807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6cc003fb0 0x7fb6cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e8a00640 1 --2- 172.21.15.45:0/725620807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6cc004d40 0x7fb6cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e8a00640 1 -- 172.21.15.45:0/725620807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e40b6020 con 0x7fb6cc003fb0 2024-04-16T16:16:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e8a00640 1 -- 172.21.15.45:0/725620807 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb6cc00bbb0 con 0x7fb6cc004d40 2024-04-16T16:16:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e8a00640 1 -- 172.21.15.45:0/725620807 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb6cc00bdb0 con 0x7fb6e4072150 2024-04-16T16:16:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.387+0000 7fb6e9400640 1 --2- 172.21.15.45:0/725620807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6cc003fb0 0x7fb6cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.388+0000 7fb6e9e00640 1 --2- 172.21.15.45:0/725620807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e4072150 0x7fb6cc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.388+0000 7fb6e9400640 1 -- 172.21.15.45:0/725620807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e4072150 msgr2=0x7fb6cc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.388+0000 7fb6e9400640 1 --2- 172.21.15.45:0/725620807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e4072150 0x7fb6cc0039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.388+0000 7fb6e9400640 1 -- 172.21.15.45:0/725620807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6cc004d40 msgr2=0x7fb6cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:45.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.388+0000 7fb6e9400640 1 --2- 172.21.15.45:0/725620807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6cc004d40 0x7fb6cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.388+0000 7fb6e9400640 1 -- 172.21.15.45:0/725620807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6cc003520 con 0x7fb6cc003fb0 2024-04-16T16:16:45.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.389+0000 7fb6e9400640 1 --2- 172.21.15.45:0/725620807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6cc003fb0 0x7fb6cc004800 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb6e406fcf0 tx=0x7fb6cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdce728f82506275 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:45.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.390+0000 7fb6e8a00640 1 -- 172.21.15.45:0/725620807 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4039070 con 0x7fb6cc003fb0 2024-04-16T16:16:45.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.390+0000 7fb6e8a00640 1 -- 172.21.15.45:0/725620807 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6d402c660 con 0x7fb6cc003fb0 2024-04-16T16:16:45.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.390+0000 7fb6e8a00640 1 -- 172.21.15.45:0/725620807 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4009690 con 0x7fb6cc003fb0 2024-04-16T16:16:45.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.390+0000 7fb6eb200640 1 -- 172.21.15.45:0/725620807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6cc003fb0 msgr2=0x7fb6cc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:45.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.390+0000 7fb6eb200640 1 --2- 172.21.15.45:0/725620807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6cc003fb0 0x7fb6cc004800 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb6e406fcf0 tx=0x7fb6cc005830 comp rx=0 tx=0).stop 2024-04-16T16:16:45.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.390+0000 7fb6ea800640 1 -- 172.21.15.45:0/725620807 reap_dead start 2024-04-16T16:16:45.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.390+0000 7fb6eb200640 1 -- 172.21.15.45:0/725620807 shutdown_connections 2024-04-16T16:16:45.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.390+0000 7fb6eb200640 1 -- 172.21.15.45:0/725620807 >> 172.21.15.45:0/725620807 conn(0x7fb6e40b43a0 msgr2=0x7fb6e40b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:45.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.390+0000 7fb6eb200640 1 -- 172.21.15.45:0/725620807 shutdown_connections 2024-04-16T16:16:45.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.390+0000 7fb6eb200640 1 -- 172.21.15.45:0/725620807 wait complete. 2024-04-16T16:16:45.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.392+0000 7fb6eb200640 1 Processor -- start 2024-04-16T16:16:45.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.392+0000 7fb6eb200640 1 -- start start 2024-04-16T16:16:45.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.393+0000 7fb6eb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6cc003fb0 0x7fb6e41522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.393+0000 7fb6eb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e415ad40 0x7fb6e415b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.393+0000 7fb6eb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e40cc940 0x7fb6e40ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.393+0000 7fb6eb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e40b56a0 con 0x7fb6cc003fb0 2024-04-16T16:16:45.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.393+0000 7fb6eb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e40b68c0 con 0x7fb6e415ad40 2024-04-16T16:16:45.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.393+0000 7fb6eb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e40b5820 con 0x7fb6e40cc940 2024-04-16T16:16:45.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.393+0000 7fb6e9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6cc003fb0 0x7fb6e41522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:45.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.393+0000 7fb6e9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6cc003fb0 0x7fb6e41522d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46516/0 (socket says 172.21.15.45:46516) 2024-04-16T16:16:45.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.393+0000 7fb6e9e00640 1 -- 172.21.15.45:0/329519296 learned_addr learned my addr 172.21.15.45:0/329519296 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:45.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.394+0000 7fb6e9400640 1 --2- 172.21.15.45:0/329519296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e415ad40 0x7fb6e415b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:45.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.394+0000 7fb6e9e00640 1 -- 172.21.15.45:0/329519296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e40cc940 msgr2=0x7fb6e40ccdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:45.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.394+0000 7fb6ea800640 1 --2- 172.21.15.45:0/329519296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e40cc940 0x7fb6e40ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:45.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.394+0000 7fb6e9e00640 1 --2- 172.21.15.45:0/329519296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e40cc940 0x7fb6e40ccdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.394+0000 7fb6e9e00640 1 -- 172.21.15.45:0/329519296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e415ad40 msgr2=0x7fb6e415b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:45.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.394+0000 7fb6e9e00640 1 --2- 172.21.15.45:0/329519296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e415ad40 0x7fb6e415b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.394+0000 7fb6e9e00640 1 -- 172.21.15.45:0/329519296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e40cd910 con 0x7fb6cc003fb0 2024-04-16T16:16:45.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.394+0000 7fb6ea800640 1 --2- 172.21.15.45:0/329519296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e40cc940 0x7fb6e40ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:45.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.394+0000 7fb6e9e00640 1 --2- 172.21.15.45:0/329519296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6cc003fb0 0x7fb6e41522d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800d800 tx=0x7fb6d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:45.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.395+0000 7fb6de000640 1 -- 172.21.15.45:0/329519296 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8019070 con 0x7fb6cc003fb0 2024-04-16T16:16:45.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.395+0000 7fb6de000640 1 -- 172.21.15.45:0/329519296 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6d80074e0 con 0x7fb6cc003fb0 2024-04-16T16:16:45.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.395+0000 7fb6de000640 1 -- 172.21.15.45:0/329519296 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d80049e0 con 0x7fb6cc003fb0 2024-04-16T16:16:45.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.395+0000 7fb6eb200640 1 -- 172.21.15.45:0/329519296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e40cd630 con 0x7fb6cc003fb0 2024-04-16T16:16:45.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.395+0000 7fb6eb200640 1 -- 172.21.15.45:0/329519296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e40cfb20 con 0x7fb6cc003fb0 2024-04-16T16:16:45.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.396+0000 7fb6eb200640 1 -- 172.21.15.45:0/329519296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6e415b700 con 0x7fb6cc003fb0 2024-04-16T16:16:45.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.397+0000 7fb6de000640 1 -- 172.21.15.45:0/329519296 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fb6d801c070 con 0x7fb6cc003fb0 2024-04-16T16:16:45.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.398+0000 7fb6de000640 1 --2- 172.21.15.45:0/329519296 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb6c806cb60 0x7fb6c806f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.398+0000 7fb6de000640 1 -- 172.21.15.45:0/329519296 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7fb6d808bbc0 con 0x7fb6cc003fb0 2024-04-16T16:16:45.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.399+0000 7fb6e9400640 1 --2- 172.21.15.45:0/329519296 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb6c806cb60 0x7fb6c806f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:45.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.400+0000 7fb6e9400640 1 --2- 172.21.15.45:0/329519296 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb6c806cb60 0x7fb6c806f020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4036000 tx=0x7fb6d4030000 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:45.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.402+0000 7fb6de000640 1 -- 172.21.15.45:0/329519296 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb6d8059ae0 con 0x7fb6cc003fb0 2024-04-16T16:16:45.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.601+0000 7fb6eb200640 1 -- 172.21.15.45:0/329519296 --> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb6e40b4f50 con 0x7fb6c806cb60 2024-04-16T16:16:45.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.622+0000 7fb6de000640 1 -- 172.21.15.45:0/329519296 <== mgr.4961 v2:172.21.15.103:6832/4155603915 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252069 (secure 0 0 0) 0x7fb6e40b4f50 con 0x7fb6c806cb60 2024-04-16T16:16:45.624 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:45.627 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:16:45.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.629+0000 7fb6eb200640 1 -- 172.21.15.45:0/329519296 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb6c806cb60 msgr2=0x7fb6c806f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:45.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.629+0000 7fb6eb200640 1 --2- 172.21.15.45:0/329519296 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb6c806cb60 0x7fb6c806f020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4036000 tx=0x7fb6d4030000 comp rx=0 tx=0).stop 2024-04-16T16:16:45.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.629+0000 7fb6eb200640 1 -- 172.21.15.45:0/329519296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6cc003fb0 msgr2=0x7fb6e41522d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:45.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.629+0000 7fb6eb200640 1 --2- 172.21.15.45:0/329519296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6cc003fb0 0x7fb6e41522d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800d800 tx=0x7fb6d800dcc0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.629+0000 7fb6eb200640 1 -- 172.21.15.45:0/329519296 shutdown_connections 2024-04-16T16:16:45.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.629+0000 7fb6eb200640 1 --2- 172.21.15.45:0/329519296 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fb6c806cb60 0x7fb6c806f020 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.629+0000 7fb6eb200640 1 --2- 172.21.15.45:0/329519296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e40cc940 0x7fb6e40ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.629+0000 7fb6eb200640 1 --2- 172.21.15.45:0/329519296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e415ad40 0x7fb6e415b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.629+0000 7fb6eb200640 1 --2- 172.21.15.45:0/329519296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6cc003fb0 0x7fb6e41522d0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.629+0000 7fb6eb200640 1 -- 172.21.15.45:0/329519296 >> 172.21.15.45:0/329519296 conn(0x7fb6e40b43a0 msgr2=0x7fb6e406ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:45.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.629+0000 7fb6eb200640 1 -- 172.21.15.45:0/329519296 shutdown_connections 2024-04-16T16:16:45.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.629+0000 7fb6eb200640 1 -- 172.21.15.45:0/329519296 wait complete. 2024-04-16T16:16:45.651 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-16T16:16:45.082003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":244,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":178,"ondisk_log_size":178,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":161,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":253360,"kb_used_data":8072,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754721360,"statfs":{"total":773094113280,"available":772834672640,"internally_reserved":0,"allocated":8265728,"data_stored":4467028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.062676"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.059101+0000","last_change":"2024-04-16T16:16:40.048700+0000","last_active":"2024-04-16T16:16:41.059101+0000","last_peered":"2024-04-16T16:16:41.059101+0000","last_clean":"2024-04-16T16:16:41.059101+0000","last_became_active":"2024-04-16T16:16:40.046882+0000","last_became_peered":"2024-04-16T16:16:40.046882+0000","last_unstale":"2024-04-16T16:16:41.059101+0000","last_undegraded":"2024-04-16T16:16:41.059101+0000","last_fullsized":"2024-04-16T16:16:41.059101+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:41:52.714263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355645+0000","last_change":"2024-04-16T16:16:40.063532+0000","last_active":"2024-04-16T16:16:41.355645+0000","last_peered":"2024-04-16T16:16:41.355645+0000","last_clean":"2024-04-16T16:16:41.355645+0000","last_became_active":"2024-04-16T16:16:40.063023+0000","last_became_peered":"2024-04-16T16:16:40.063023+0000","last_unstale":"2024-04-16T16:16:41.355645+0000","last_undegraded":"2024-04-16T16:16:41.355645+0000","last_fullsized":"2024-04-16T16:16:41.355645+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:45.673814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353809+0000","last_change":"2024-04-16T16:16:40.058970+0000","last_active":"2024-04-16T16:16:41.353809+0000","last_peered":"2024-04-16T16:16:41.353809+0000","last_clean":"2024-04-16T16:16:41.353809+0000","last_became_active":"2024-04-16T16:16:40.058582+0000","last_became_peered":"2024-04-16T16:16:40.058582+0000","last_unstale":"2024-04-16T16:16:41.353809+0000","last_undegraded":"2024-04-16T16:16:41.353809+0000","last_fullsized":"2024-04-16T16:16:41.353809+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:04.907779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354998+0000","last_change":"2024-04-16T16:16:40.067374+0000","last_active":"2024-04-16T16:16:41.354998+0000","last_peered":"2024-04-16T16:16:41.354998+0000","last_clean":"2024-04-16T16:16:41.354998+0000","last_became_active":"2024-04-16T16:16:40.064314+0000","last_became_peered":"2024-04-16T16:16:40.064314+0000","last_unstale":"2024-04-16T16:16:41.354998+0000","last_undegraded":"2024-04-16T16:16:41.354998+0000","last_fullsized":"2024-04-16T16:16:41.354998+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:47:33.049888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052264+0000","last_change":"2024-04-16T16:16:40.053258+0000","last_active":"2024-04-16T16:16:41.052264+0000","last_peered":"2024-04-16T16:16:41.052264+0000","last_clean":"2024-04-16T16:16:41.052264+0000","last_became_active":"2024-04-16T16:16:40.051051+0000","last_became_peered":"2024-04-16T16:16:40.051051+0000","last_unstale":"2024-04-16T16:16:41.052264+0000","last_undegraded":"2024-04-16T16:16:41.052264+0000","last_fullsized":"2024-04-16T16:16:41.052264+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:16:10.608170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355834+0000","last_change":"2024-04-16T16:16:40.064127+0000","last_active":"2024-04-16T16:16:41.355834+0000","last_peered":"2024-04-16T16:16:41.355834+0000","last_clean":"2024-04-16T16:16:41.355834+0000","last_became_active":"2024-04-16T16:16:40.063617+0000","last_became_peered":"2024-04-16T16:16:40.063617+0000","last_unstale":"2024-04-16T16:16:41.355834+0000","last_undegraded":"2024-04-16T16:16:41.355834+0000","last_fullsized":"2024-04-16T16:16:41.355834+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:54:31.905334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079437+0000","last_change":"2024-04-16T16:16:40.060797+0000","last_active":"2024-04-16T16:16:41.079437+0000","last_peered":"2024-04-16T16:16:41.079437+0000","last_clean":"2024-04-16T16:16:41.079437+0000","last_became_active":"2024-04-16T16:16:40.060497+0000","last_became_peered":"2024-04-16T16:16:40.060497+0000","last_unstale":"2024-04-16T16:16:41.079437+0000","last_undegraded":"2024-04-16T16:16:41.079437+0000","last_fullsized":"2024-04-16T16:16:41.079437+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:43.199657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.356013+0000","last_change":"2024-04-16T16:16:40.067270+0000","last_active":"2024-04-16T16:16:41.356013+0000","last_peered":"2024-04-16T16:16:41.356013+0000","last_clean":"2024-04-16T16:16:41.356013+0000","last_became_active":"2024-04-16T16:16:40.066527+0000","last_became_peered":"2024-04-16T16:16:40.066527+0000","last_unstale":"2024-04-16T16:16:41.356013+0000","last_undegraded":"2024-04-16T16:16:41.356013+0000","last_fullsized":"2024-04-16T16:16:41.356013+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:06:06.933720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053974+0000","last_change":"2024-04-16T16:16:40.065283+0000","last_active":"2024-04-16T16:16:41.053974+0000","last_peered":"2024-04-16T16:16:41.053974+0000","last_clean":"2024-04-16T16:16:41.053974+0000","last_became_active":"2024-04-16T16:16:40.064211+0000","last_became_peered":"2024-04-16T16:16:40.064211+0000","last_unstale":"2024-04-16T16:16:41.053974+0000","last_undegraded":"2024-04-16T16:16:41.053974+0000","last_fullsized":"2024-04-16T16:16:41.053974+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:28.836553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.058144+0000","last_change":"2024-04-16T16:16:40.057046+0000","last_active":"2024-04-16T16:16:41.058144+0000","last_peered":"2024-04-16T16:16:41.058144+0000","last_clean":"2024-04-16T16:16:41.058144+0000","last_became_active":"2024-04-16T16:16:40.056773+0000","last_became_peered":"2024-04-16T16:16:40.056773+0000","last_unstale":"2024-04-16T16:16:41.058144+0000","last_undegraded":"2024-04-16T16:16:41.058144+0000","last_fullsized":"2024-04-16T16:16:41.058144+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:54:31.905334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.055911+0000","last_change":"2024-04-16T16:16:40.059929+0000","last_active":"2024-04-16T16:16:41.055911+0000","last_peered":"2024-04-16T16:16:41.055911+0000","last_clean":"2024-04-16T16:16:41.055911+0000","last_became_active":"2024-04-16T16:16:40.059666+0000","last_became_peered":"2024-04-16T16:16:40.059666+0000","last_unstale":"2024-04-16T16:16:41.055911+0000","last_undegraded":"2024-04-16T16:16:41.055911+0000","last_fullsized":"2024-04-16T16:16:41.055911+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:21.332023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354796+0000","last_change":"2024-04-16T16:16:40.070880+0000","last_active":"2024-04-16T16:16:41.354796+0000","last_peered":"2024-04-16T16:16:41.354796+0000","last_clean":"2024-04-16T16:16:41.354796+0000","last_became_active":"2024-04-16T16:16:40.070434+0000","last_became_peered":"2024-04-16T16:16:40.070434+0000","last_unstale":"2024-04-16T16:16:41.354796+0000","last_undegraded":"2024-04-16T16:16:41.354796+0000","last_fullsized":"2024-04-16T16:16:41.354796+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:13:47.015413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084447+0000","last_change":"2024-04-16T16:16:40.053229+0000","last_active":"2024-04-16T16:16:41.084447+0000","last_peered":"2024-04-16T16:16:41.084447+0000","last_clean":"2024-04-16T16:16:41.084447+0000","last_became_active":"2024-04-16T16:16:40.052968+0000","last_became_peered":"2024-04-16T16:16:40.052968+0000","last_unstale":"2024-04-16T16:16:41.084447+0000","last_undegraded":"2024-04-16T16:16:41.084447+0000","last_fullsized":"2024-04-16T16:16:41.084447+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:46.618649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052568+0000","last_change":"2024-04-16T16:16:40.059138+0000","last_active":"2024-04-16T16:16:41.052568+0000","last_peered":"2024-04-16T16:16:41.052568+0000","last_clean":"2024-04-16T16:16:41.052568+0000","last_became_active":"2024-04-16T16:16:40.056131+0000","last_became_peered":"2024-04-16T16:16:40.056131+0000","last_unstale":"2024-04-16T16:16:41.052568+0000","last_undegraded":"2024-04-16T16:16:41.052568+0000","last_fullsized":"2024-04-16T16:16:41.052568+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:39:52.139845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084236+0000","last_change":"2024-04-16T16:16:40.052923+0000","last_active":"2024-04-16T16:16:41.084236+0000","last_peered":"2024-04-16T16:16:41.084236+0000","last_clean":"2024-04-16T16:16:41.084236+0000","last_became_active":"2024-04-16T16:16:40.052609+0000","last_became_peered":"2024-04-16T16:16:40.052609+0000","last_unstale":"2024-04-16T16:16:41.084236+0000","last_undegraded":"2024-04-16T16:16:41.084236+0000","last_fullsized":"2024-04-16T16:16:41.084236+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:21.332023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057643+0000","last_change":"2024-04-16T16:16:40.055850+0000","last_active":"2024-04-16T16:16:41.057643+0000","last_peered":"2024-04-16T16:16:41.057643+0000","last_clean":"2024-04-16T16:16:41.057643+0000","last_became_active":"2024-04-16T16:16:40.055560+0000","last_became_peered":"2024-04-16T16:16:40.055560+0000","last_unstale":"2024-04-16T16:16:41.057643+0000","last_undegraded":"2024-04-16T16:16:41.057643+0000","last_fullsized":"2024-04-16T16:16:41.057643+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:22.397420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054029+0000","last_change":"2024-04-16T16:16:40.069834+0000","last_active":"2024-04-16T16:16:41.054029+0000","last_peered":"2024-04-16T16:16:41.054029+0000","last_clean":"2024-04-16T16:16:41.054029+0000","last_became_active":"2024-04-16T16:16:40.062429+0000","last_became_peered":"2024-04-16T16:16:40.062429+0000","last_unstale":"2024-04-16T16:16:41.054029+0000","last_undegraded":"2024-04-16T16:16:41.054029+0000","last_fullsized":"2024-04-16T16:16:41.054029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:14:20.465129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355562+0000","last_change":"2024-04-16T16:16:40.055791+0000","last_active":"2024-04-16T16:16:41.355562+0000","last_peered":"2024-04-16T16:16:41.355562+0000","last_clean":"2024-04-16T16:16:41.355562+0000","last_became_active":"2024-04-16T16:16:40.055508+0000","last_became_peered":"2024-04-16T16:16:40.055508+0000","last_unstale":"2024-04-16T16:16:41.355562+0000","last_undegraded":"2024-04-16T16:16:41.355562+0000","last_fullsized":"2024-04-16T16:16:41.355562+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:44.062254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355381+0000","last_change":"2024-04-16T16:16:40.067596+0000","last_active":"2024-04-16T16:16:41.355381+0000","last_peered":"2024-04-16T16:16:41.355381+0000","last_clean":"2024-04-16T16:16:41.355381+0000","last_became_active":"2024-04-16T16:16:40.067146+0000","last_became_peered":"2024-04-16T16:16:40.067146+0000","last_unstale":"2024-04-16T16:16:41.355381+0000","last_undegraded":"2024-04-16T16:16:41.355381+0000","last_fullsized":"2024-04-16T16:16:41.355381+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:07:41.285674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084256+0000","last_change":"2024-04-16T16:16:40.058979+0000","last_active":"2024-04-16T16:16:41.084256+0000","last_peered":"2024-04-16T16:16:41.084256+0000","last_clean":"2024-04-16T16:16:41.084256+0000","last_became_active":"2024-04-16T16:16:40.058648+0000","last_became_peered":"2024-04-16T16:16:40.058648+0000","last_unstale":"2024-04-16T16:16:41.084256+0000","last_undegraded":"2024-04-16T16:16:41.084256+0000","last_fullsized":"2024-04-16T16:16:41.084256+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:28:41.607987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056889+0000","last_change":"2024-04-16T16:16:40.065622+0000","last_active":"2024-04-16T16:16:41.056889+0000","last_peered":"2024-04-16T16:16:41.056889+0000","last_clean":"2024-04-16T16:16:41.056889+0000","last_became_active":"2024-04-16T16:16:40.064748+0000","last_became_peered":"2024-04-16T16:16:40.064748+0000","last_unstale":"2024-04-16T16:16:41.056889+0000","last_undegraded":"2024-04-16T16:16:41.056889+0000","last_fullsized":"2024-04-16T16:16:41.056889+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:12:52.888262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057883+0000","last_change":"2024-04-16T16:16:40.054692+0000","last_active":"2024-04-16T16:16:41.057883+0000","last_peered":"2024-04-16T16:16:41.057883+0000","last_clean":"2024-04-16T16:16:41.057883+0000","last_became_active":"2024-04-16T16:16:40.054392+0000","last_became_peered":"2024-04-16T16:16:40.054392+0000","last_unstale":"2024-04-16T16:16:41.057883+0000","last_undegraded":"2024-04-16T16:16:41.057883+0000","last_fullsized":"2024-04-16T16:16:41.057883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:25.467004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056193+0000","last_change":"2024-04-16T16:16:40.056002+0000","last_active":"2024-04-16T16:16:41.056193+0000","last_peered":"2024-04-16T16:16:41.056193+0000","last_clean":"2024-04-16T16:16:41.056193+0000","last_became_active":"2024-04-16T16:16:40.055212+0000","last_became_peered":"2024-04-16T16:16:40.055212+0000","last_unstale":"2024-04-16T16:16:41.056193+0000","last_undegraded":"2024-04-16T16:16:41.056193+0000","last_fullsized":"2024-04-16T16:16:41.056193+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:28:41.607987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054913+0000","last_change":"2024-04-16T16:16:40.054708+0000","last_active":"2024-04-16T16:16:41.054913+0000","last_peered":"2024-04-16T16:16:41.054913+0000","last_clean":"2024-04-16T16:16:41.054913+0000","last_became_active":"2024-04-16T16:16:40.053297+0000","last_became_peered":"2024-04-16T16:16:40.053297+0000","last_unstale":"2024-04-16T16:16:41.054913+0000","last_undegraded":"2024-04-16T16:16:41.054913+0000","last_fullsized":"2024-04-16T16:16:41.054913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:31:33.590968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.058757+0000","last_change":"2024-04-16T16:16:40.053469+0000","last_active":"2024-04-16T16:16:41.058757+0000","last_peered":"2024-04-16T16:16:41.058757+0000","last_clean":"2024-04-16T16:16:41.058757+0000","last_became_active":"2024-04-16T16:16:40.052617+0000","last_became_peered":"2024-04-16T16:16:40.052617+0000","last_unstale":"2024-04-16T16:16:41.058757+0000","last_undegraded":"2024-04-16T16:16:41.058757+0000","last_fullsized":"2024-04-16T16:16:41.058757+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:06:06.933720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.080143+0000","last_change":"2024-04-16T16:16:40.058621+0000","last_active":"2024-04-16T16:16:41.080143+0000","last_peered":"2024-04-16T16:16:41.080143+0000","last_clean":"2024-04-16T16:16:41.080143+0000","last_became_active":"2024-04-16T16:16:40.058250+0000","last_became_peered":"2024-04-16T16:16:40.058250+0000","last_unstale":"2024-04-16T16:16:41.080143+0000","last_undegraded":"2024-04-16T16:16:41.080143+0000","last_fullsized":"2024-04-16T16:16:41.080143+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:21.332023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353888+0000","last_change":"2024-04-16T16:16:40.066255+0000","last_active":"2024-04-16T16:16:41.353888+0000","last_peered":"2024-04-16T16:16:41.353888+0000","last_clean":"2024-04-16T16:16:41.353888+0000","last_became_active":"2024-04-16T16:16:40.065779+0000","last_became_peered":"2024-04-16T16:16:40.065779+0000","last_unstale":"2024-04-16T16:16:41.353888+0000","last_undegraded":"2024-04-16T16:16:41.353888+0000","last_fullsized":"2024-04-16T16:16:41.353888+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:26:05.697779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057071+0000","last_change":"2024-04-16T16:16:40.056911+0000","last_active":"2024-04-16T16:16:41.057071+0000","last_peered":"2024-04-16T16:16:41.057071+0000","last_clean":"2024-04-16T16:16:41.057071+0000","last_became_active":"2024-04-16T16:16:40.054450+0000","last_became_peered":"2024-04-16T16:16:40.054450+0000","last_unstale":"2024-04-16T16:16:41.057071+0000","last_undegraded":"2024-04-16T16:16:41.057071+0000","last_fullsized":"2024-04-16T16:16:41.057071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:07:41.285674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353512+0000","last_change":"2024-04-16T16:16:40.055812+0000","last_active":"2024-04-16T16:16:41.353512+0000","last_peered":"2024-04-16T16:16:41.353512+0000","last_clean":"2024-04-16T16:16:41.353512+0000","last_became_active":"2024-04-16T16:16:40.055488+0000","last_became_peered":"2024-04-16T16:16:40.055488+0000","last_unstale":"2024-04-16T16:16:41.353512+0000","last_undegraded":"2024-04-16T16:16:41.353512+0000","last_fullsized":"2024-04-16T16:16:41.353512+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:49:49.405418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079606+0000","last_change":"2024-04-16T16:16:40.056514+0000","last_active":"2024-04-16T16:16:41.079606+0000","last_peered":"2024-04-16T16:16:41.079606+0000","last_clean":"2024-04-16T16:16:41.079606+0000","last_became_active":"2024-04-16T16:16:40.056184+0000","last_became_peered":"2024-04-16T16:16:40.056184+0000","last_unstale":"2024-04-16T16:16:41.079606+0000","last_undegraded":"2024-04-16T16:16:41.079606+0000","last_fullsized":"2024-04-16T16:16:41.079606+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:08:39.372895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079878+0000","last_change":"2024-04-16T16:16:40.054600+0000","last_active":"2024-04-16T16:16:41.079878+0000","last_peered":"2024-04-16T16:16:41.079878+0000","last_clean":"2024-04-16T16:16:41.079878+0000","last_became_active":"2024-04-16T16:16:40.054303+0000","last_became_peered":"2024-04-16T16:16:40.054303+0000","last_unstale":"2024-04-16T16:16:41.079878+0000","last_undegraded":"2024-04-16T16:16:41.079878+0000","last_fullsized":"2024-04-16T16:16:41.079878+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:31:33.590968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355739+0000","last_change":"2024-04-16T16:16:40.066863+0000","last_active":"2024-04-16T16:16:41.355739+0000","last_peered":"2024-04-16T16:16:41.355739+0000","last_clean":"2024-04-16T16:16:41.355739+0000","last_became_active":"2024-04-16T16:16:40.064405+0000","last_became_peered":"2024-04-16T16:16:40.064405+0000","last_unstale":"2024-04-16T16:16:41.355739+0000","last_undegraded":"2024-04-16T16:16:41.355739+0000","last_fullsized":"2024-04-16T16:16:41.355739+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:25.467004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056006+0000","last_change":"2024-04-16T16:16:40.052907+0000","last_active":"2024-04-16T16:16:41.056006+0000","last_peered":"2024-04-16T16:16:41.056006+0000","last_clean":"2024-04-16T16:16:41.056006+0000","last_became_active":"2024-04-16T16:16:40.052066+0000","last_became_peered":"2024-04-16T16:16:40.052066+0000","last_unstale":"2024-04-16T16:16:41.056006+0000","last_undegraded":"2024-04-16T16:16:41.056006+0000","last_fullsized":"2024-04-16T16:16:41.056006+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:48:35.347056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"71'4","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057049+0000","last_change":"2024-04-16T16:16:39.052943+0000","last_active":"2024-04-16T16:16:41.057049+0000","last_peered":"2024-04-16T16:16:41.057049+0000","last_clean":"2024-04-16T16:16:41.057049+0000","last_became_active":"2024-04-16T16:16:39.052572+0000","last_became_peered":"2024-04-16T16:16:39.052572+0000","last_unstale":"2024-04-16T16:16:41.057049+0000","last_undegraded":"2024-04-16T16:16:41.057049+0000","last_fullsized":"2024-04-16T16:16:41.057049+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:54:30.889318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054498+0000","last_change":"2024-04-16T16:16:40.063884+0000","last_active":"2024-04-16T16:16:41.054498+0000","last_peered":"2024-04-16T16:16:41.054498+0000","last_clean":"2024-04-16T16:16:41.054498+0000","last_became_active":"2024-04-16T16:16:40.061609+0000","last_became_peered":"2024-04-16T16:16:40.061609+0000","last_unstale":"2024-04-16T16:16:41.054498+0000","last_undegraded":"2024-04-16T16:16:41.054498+0000","last_fullsized":"2024-04-16T16:16:41.054498+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:55.827541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084571+0000","last_change":"2024-04-16T16:16:39.035506+0000","last_active":"2024-04-16T16:16:41.084571+0000","last_peered":"2024-04-16T16:16:41.084571+0000","last_clean":"2024-04-16T16:16:41.084571+0000","last_became_active":"2024-04-16T16:16:39.035196+0000","last_became_peered":"2024-04-16T16:16:39.035196+0000","last_unstale":"2024-04-16T16:16:41.084571+0000","last_undegraded":"2024-04-16T16:16:41.084571+0000","last_fullsized":"2024-04-16T16:16:41.084571+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:29:14.659131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355912+0000","last_change":"2024-04-16T16:16:40.061879+0000","last_active":"2024-04-16T16:16:41.355912+0000","last_peered":"2024-04-16T16:16:41.355912+0000","last_clean":"2024-04-16T16:16:41.355912+0000","last_became_active":"2024-04-16T16:16:40.058652+0000","last_became_peered":"2024-04-16T16:16:40.058652+0000","last_unstale":"2024-04-16T16:16:41.355912+0000","last_undegraded":"2024-04-16T16:16:41.355912+0000","last_fullsized":"2024-04-16T16:16:41.355912+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:12.235559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057778+0000","last_change":"2024-04-16T16:16:39.057527+0000","last_active":"2024-04-16T16:16:41.057778+0000","last_peered":"2024-04-16T16:16:41.057778+0000","last_clean":"2024-04-16T16:16:41.057778+0000","last_became_active":"2024-04-16T16:16:39.056172+0000","last_became_peered":"2024-04-16T16:16:39.056172+0000","last_unstale":"2024-04-16T16:16:41.057778+0000","last_undegraded":"2024-04-16T16:16:41.057778+0000","last_fullsized":"2024-04-16T16:16:41.057778+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:13:45.999397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056353+0000","last_change":"2024-04-16T16:16:40.058328+0000","last_active":"2024-04-16T16:16:41.056353+0000","last_peered":"2024-04-16T16:16:41.056353+0000","last_clean":"2024-04-16T16:16:41.056353+0000","last_became_active":"2024-04-16T16:16:40.058025+0000","last_became_peered":"2024-04-16T16:16:40.058025+0000","last_unstale":"2024-04-16T16:16:41.056353+0000","last_undegraded":"2024-04-16T16:16:41.056353+0000","last_fullsized":"2024-04-16T16:16:41.056353+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:46.618649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355874+0000","last_change":"2024-04-16T16:16:39.042499+0000","last_active":"2024-04-16T16:16:41.355874+0000","last_peered":"2024-04-16T16:16:41.355874+0000","last_clean":"2024-04-16T16:16:41.355874+0000","last_became_active":"2024-04-16T16:16:39.042042+0000","last_became_peered":"2024-04-16T16:16:39.042042+0000","last_unstale":"2024-04-16T16:16:41.355874+0000","last_undegraded":"2024-04-16T16:16:41.355874+0000","last_fullsized":"2024-04-16T16:16:41.355874+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:40:02.778520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079720+0000","last_change":"2024-04-16T16:16:40.061501+0000","last_active":"2024-04-16T16:16:41.079720+0000","last_peered":"2024-04-16T16:16:41.079720+0000","last_clean":"2024-04-16T16:16:41.079720+0000","last_became_active":"2024-04-16T16:16:40.061046+0000","last_became_peered":"2024-04-16T16:16:40.061046+0000","last_unstale":"2024-04-16T16:16:41.079720+0000","last_undegraded":"2024-04-16T16:16:41.079720+0000","last_fullsized":"2024-04-16T16:16:41.079720+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:34:19.296917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353704+0000","last_change":"2024-04-16T16:16:39.051382+0000","last_active":"2024-04-16T16:16:41.353704+0000","last_peered":"2024-04-16T16:16:41.353704+0000","last_clean":"2024-04-16T16:16:41.353704+0000","last_became_active":"2024-04-16T16:16:39.050793+0000","last_became_peered":"2024-04-16T16:16:39.050793+0000","last_unstale":"2024-04-16T16:16:41.353704+0000","last_undegraded":"2024-04-16T16:16:41.353704+0000","last_fullsized":"2024-04-16T16:16:41.353704+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:55:57.308712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057316+0000","last_change":"2024-04-16T16:16:40.049380+0000","last_active":"2024-04-16T16:16:41.057316+0000","last_peered":"2024-04-16T16:16:41.057316+0000","last_clean":"2024-04-16T16:16:41.057316+0000","last_became_active":"2024-04-16T16:16:40.049071+0000","last_became_peered":"2024-04-16T16:16:40.049071+0000","last_unstale":"2024-04-16T16:16:41.057316+0000","last_undegraded":"2024-04-16T16:16:41.057316+0000","last_fullsized":"2024-04-16T16:16:41.057316+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:45.673814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354835+0000","last_change":"2024-04-16T16:16:39.046490+0000","last_active":"2024-04-16T16:16:41.354835+0000","last_peered":"2024-04-16T16:16:41.354835+0000","last_clean":"2024-04-16T16:16:41.354835+0000","last_became_active":"2024-04-16T16:16:39.045734+0000","last_became_peered":"2024-04-16T16:16:39.045734+0000","last_unstale":"2024-04-16T16:16:41.354835+0000","last_undegraded":"2024-04-16T16:16:41.354835+0000","last_fullsized":"2024-04-16T16:16:41.354835+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:37:31.768292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.051911+0000","last_change":"2024-04-16T16:16:40.049943+0000","last_active":"2024-04-16T16:16:41.051911+0000","last_peered":"2024-04-16T16:16:41.051911+0000","last_clean":"2024-04-16T16:16:41.051911+0000","last_became_active":"2024-04-16T16:16:40.049466+0000","last_became_peered":"2024-04-16T16:16:40.049466+0000","last_unstale":"2024-04-16T16:16:41.051911+0000","last_undegraded":"2024-04-16T16:16:41.051911+0000","last_fullsized":"2024-04-16T16:16:41.051911+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:24.082098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353887+0000","last_change":"2024-04-16T16:16:39.048665+0000","last_active":"2024-04-16T16:16:41.353887+0000","last_peered":"2024-04-16T16:16:41.353887+0000","last_clean":"2024-04-16T16:16:41.353887+0000","last_became_active":"2024-04-16T16:16:39.046018+0000","last_became_peered":"2024-04-16T16:16:39.046018+0000","last_unstale":"2024-04-16T16:16:41.353887+0000","last_undegraded":"2024-04-16T16:16:41.353887+0000","last_fullsized":"2024-04-16T16:16:41.353887+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:37:31.768292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084230+0000","last_change":"2024-04-16T16:16:40.068846+0000","last_active":"2024-04-16T16:16:41.084230+0000","last_peered":"2024-04-16T16:16:41.084230+0000","last_clean":"2024-04-16T16:16:41.084230+0000","last_became_active":"2024-04-16T16:16:40.060719+0000","last_became_peered":"2024-04-16T16:16:40.060719+0000","last_unstale":"2024-04-16T16:16:41.084230+0000","last_undegraded":"2024-04-16T16:16:41.084230+0000","last_fullsized":"2024-04-16T16:16:41.084230+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:48:35.347056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.051833+0000","last_change":"2024-04-16T16:16:39.042020+0000","last_active":"2024-04-16T16:16:41.051833+0000","last_peered":"2024-04-16T16:16:41.051833+0000","last_clean":"2024-04-16T16:16:41.051833+0000","last_became_active":"2024-04-16T16:16:39.041447+0000","last_became_peered":"2024-04-16T16:16:39.041447+0000","last_unstale":"2024-04-16T16:16:41.051833+0000","last_undegraded":"2024-04-16T16:16:41.051833+0000","last_fullsized":"2024-04-16T16:16:41.051833+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:18:24.097920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.080180+0000","last_change":"2024-04-16T16:16:40.060588+0000","last_active":"2024-04-16T16:16:41.080180+0000","last_peered":"2024-04-16T16:16:41.080180+0000","last_clean":"2024-04-16T16:16:41.080180+0000","last_became_active":"2024-04-16T16:16:40.060149+0000","last_became_peered":"2024-04-16T16:16:40.060149+0000","last_unstale":"2024-04-16T16:16:41.080180+0000","last_undegraded":"2024-04-16T16:16:41.080180+0000","last_fullsized":"2024-04-16T16:16:41.080180+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:48:35.347056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.058616+0000","last_change":"2024-04-16T16:16:39.052655+0000","last_active":"2024-04-16T16:16:41.058616+0000","last_peered":"2024-04-16T16:16:41.058616+0000","last_clean":"2024-04-16T16:16:41.058616+0000","last_became_active":"2024-04-16T16:16:39.051190+0000","last_became_peered":"2024-04-16T16:16:39.051190+0000","last_unstale":"2024-04-16T16:16:41.058616+0000","last_undegraded":"2024-04-16T16:16:41.058616+0000","last_fullsized":"2024-04-16T16:16:41.058616+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:43.334972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056462+0000","last_change":"2024-04-16T16:16:40.051811+0000","last_active":"2024-04-16T16:16:41.056462+0000","last_peered":"2024-04-16T16:16:41.056462+0000","last_clean":"2024-04-16T16:16:41.056462+0000","last_became_active":"2024-04-16T16:16:40.051476+0000","last_became_peered":"2024-04-16T16:16:40.051476+0000","last_unstale":"2024-04-16T16:16:41.056462+0000","last_undegraded":"2024-04-16T16:16:41.056462+0000","last_fullsized":"2024-04-16T16:16:41.056462+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:29.398550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355529+0000","last_change":"2024-04-16T16:16:39.056709+0000","last_active":"2024-04-16T16:16:41.355529+0000","last_peered":"2024-04-16T16:16:41.355529+0000","last_clean":"2024-04-16T16:16:41.355529+0000","last_became_active":"2024-04-16T16:16:39.056334+0000","last_became_peered":"2024-04-16T16:16:39.056334+0000","last_unstale":"2024-04-16T16:16:41.355529+0000","last_undegraded":"2024-04-16T16:16:41.355529+0000","last_fullsized":"2024-04-16T16:16:41.355529+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:52:38.661658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355675+0000","last_change":"2024-04-16T16:16:40.055050+0000","last_active":"2024-04-16T16:16:41.355675+0000","last_peered":"2024-04-16T16:16:41.355675+0000","last_clean":"2024-04-16T16:16:41.355675+0000","last_became_active":"2024-04-16T16:16:40.054723+0000","last_became_peered":"2024-04-16T16:16:40.054723+0000","last_unstale":"2024-04-16T16:16:41.355675+0000","last_undegraded":"2024-04-16T16:16:41.355675+0000","last_fullsized":"2024-04-16T16:16:41.355675+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:22.397420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"71'3","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353950+0000","last_change":"2024-04-16T16:16:39.046939+0000","last_active":"2024-04-16T16:16:41.353950+0000","last_peered":"2024-04-16T16:16:41.353950+0000","last_clean":"2024-04-16T16:16:41.353950+0000","last_became_active":"2024-04-16T16:16:39.046609+0000","last_became_peered":"2024-04-16T16:16:39.046609+0000","last_unstale":"2024-04-16T16:16:41.353950+0000","last_undegraded":"2024-04-16T16:16:41.353950+0000","last_fullsized":"2024-04-16T16:16:41.353950+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:38:05.588116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053162+0000","last_change":"2024-04-16T16:16:40.056836+0000","last_active":"2024-04-16T16:16:41.053162+0000","last_peered":"2024-04-16T16:16:41.053162+0000","last_clean":"2024-04-16T16:16:41.053162+0000","last_became_active":"2024-04-16T16:16:40.056365+0000","last_became_peered":"2024-04-16T16:16:40.056365+0000","last_unstale":"2024-04-16T16:16:41.053162+0000","last_undegraded":"2024-04-16T16:16:41.053162+0000","last_fullsized":"2024-04-16T16:16:41.053162+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:17:23.179723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"71'6","reported_seq":20,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355016+0000","last_change":"2024-04-16T16:16:39.040106+0000","last_active":"2024-04-16T16:16:41.355016+0000","last_peered":"2024-04-16T16:16:41.355016+0000","last_clean":"2024-04-16T16:16:41.355016+0000","last_became_active":"2024-04-16T16:16:39.039334+0000","last_became_peered":"2024-04-16T16:16:39.039334+0000","last_unstale":"2024-04-16T16:16:41.355016+0000","last_undegraded":"2024-04-16T16:16:41.355016+0000","last_fullsized":"2024-04-16T16:16:41.355016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:29.791246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056901+0000","last_change":"2024-04-16T16:16:40.049290+0000","last_active":"2024-04-16T16:16:41.056901+0000","last_peered":"2024-04-16T16:16:41.056901+0000","last_clean":"2024-04-16T16:16:41.056901+0000","last_became_active":"2024-04-16T16:16:40.048726+0000","last_became_peered":"2024-04-16T16:16:40.048726+0000","last_unstale":"2024-04-16T16:16:41.056901+0000","last_undegraded":"2024-04-16T16:16:41.056901+0000","last_fullsized":"2024-04-16T16:16:41.056901+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:41:00.935635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354451+0000","last_change":"2024-04-16T16:16:39.048257+0000","last_active":"2024-04-16T16:16:41.354451+0000","last_peered":"2024-04-16T16:16:41.354451+0000","last_clean":"2024-04-16T16:16:41.354451+0000","last_became_active":"2024-04-16T16:16:39.047835+0000","last_became_peered":"2024-04-16T16:16:39.047835+0000","last_unstale":"2024-04-16T16:16:41.354451+0000","last_undegraded":"2024-04-16T16:16:41.354451+0000","last_fullsized":"2024-04-16T16:16:41.354451+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:46.723824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079769+0000","last_change":"2024-04-16T16:16:40.055833+0000","last_active":"2024-04-16T16:16:41.079769+0000","last_peered":"2024-04-16T16:16:41.079769+0000","last_clean":"2024-04-16T16:16:41.079769+0000","last_became_active":"2024-04-16T16:16:40.055564+0000","last_became_peered":"2024-04-16T16:16:40.055564+0000","last_unstale":"2024-04-16T16:16:41.079769+0000","last_undegraded":"2024-04-16T16:16:41.079769+0000","last_fullsized":"2024-04-16T16:16:41.079769+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:51:29.994460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355134+0000","last_change":"2024-04-16T16:16:39.046580+0000","last_active":"2024-04-16T16:16:41.355134+0000","last_peered":"2024-04-16T16:16:41.355134+0000","last_clean":"2024-04-16T16:16:41.355134+0000","last_became_active":"2024-04-16T16:16:39.046181+0000","last_became_peered":"2024-04-16T16:16:39.046181+0000","last_unstale":"2024-04-16T16:16:41.355134+0000","last_undegraded":"2024-04-16T16:16:41.355134+0000","last_fullsized":"2024-04-16T16:16:41.355134+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:29.791246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084205+0000","last_change":"2024-04-16T16:16:40.055025+0000","last_active":"2024-04-16T16:16:41.084205+0000","last_peered":"2024-04-16T16:16:41.084205+0000","last_clean":"2024-04-16T16:16:41.084205+0000","last_became_active":"2024-04-16T16:16:40.054757+0000","last_became_peered":"2024-04-16T16:16:40.054757+0000","last_unstale":"2024-04-16T16:16:41.084205+0000","last_undegraded":"2024-04-16T16:16:41.084205+0000","last_fullsized":"2024-04-16T16:16:41.084205+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:31:33.590968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079273+0000","last_change":"2024-04-16T16:16:39.052798+0000","last_active":"2024-04-16T16:16:41.079273+0000","last_peered":"2024-04-16T16:16:41.079273+0000","last_clean":"2024-04-16T16:16:41.079273+0000","last_became_active":"2024-04-16T16:16:39.052440+0000","last_became_peered":"2024-04-16T16:16:39.052440+0000","last_unstale":"2024-04-16T16:16:41.079273+0000","last_undegraded":"2024-04-16T16:16:41.079273+0000","last_fullsized":"2024-04-16T16:16:41.079273+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:44.657799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052671+0000","last_change":"2024-04-16T16:16:40.054326+0000","last_active":"2024-04-16T16:16:41.052671+0000","last_peered":"2024-04-16T16:16:41.052671+0000","last_clean":"2024-04-16T16:16:41.052671+0000","last_became_active":"2024-04-16T16:16:40.054013+0000","last_became_peered":"2024-04-16T16:16:40.054013+0000","last_unstale":"2024-04-16T16:16:41.052671+0000","last_undegraded":"2024-04-16T16:16:41.052671+0000","last_fullsized":"2024-04-16T16:16:41.052671+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:05:08.507551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"71'8","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057407+0000","last_change":"2024-04-16T16:16:39.053695+0000","last_active":"2024-04-16T16:16:41.057407+0000","last_peered":"2024-04-16T16:16:41.057407+0000","last_clean":"2024-04-16T16:16:41.057407+0000","last_became_active":"2024-04-16T16:16:39.053264+0000","last_became_peered":"2024-04-16T16:16:39.053264+0000","last_unstale":"2024-04-16T16:16:41.057407+0000","last_undegraded":"2024-04-16T16:16:41.057407+0000","last_fullsized":"2024-04-16T16:16:41.057407+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:39:32.200451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.058313+0000","last_change":"2024-04-16T16:16:40.056673+0000","last_active":"2024-04-16T16:16:41.058313+0000","last_peered":"2024-04-16T16:16:41.058313+0000","last_clean":"2024-04-16T16:16:41.058313+0000","last_became_active":"2024-04-16T16:16:40.056357+0000","last_became_peered":"2024-04-16T16:16:40.056357+0000","last_unstale":"2024-04-16T16:16:41.058313+0000","last_undegraded":"2024-04-16T16:16:41.058313+0000","last_fullsized":"2024-04-16T16:16:41.058313+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:12.235559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355020+0000","last_change":"2024-04-16T16:16:39.046833+0000","last_active":"2024-04-16T16:16:41.355020+0000","last_peered":"2024-04-16T16:16:41.355020+0000","last_clean":"2024-04-16T16:16:41.355020+0000","last_became_active":"2024-04-16T16:16:39.046543+0000","last_became_peered":"2024-04-16T16:16:39.046543+0000","last_unstale":"2024-04-16T16:16:41.355020+0000","last_undegraded":"2024-04-16T16:16:41.355020+0000","last_fullsized":"2024-04-16T16:16:41.355020+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:39:32.200451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056906+0000","last_change":"2024-04-16T16:16:40.059425+0000","last_active":"2024-04-16T16:16:41.056906+0000","last_peered":"2024-04-16T16:16:41.056906+0000","last_clean":"2024-04-16T16:16:41.056906+0000","last_became_active":"2024-04-16T16:16:40.059078+0000","last_became_peered":"2024-04-16T16:16:40.059078+0000","last_unstale":"2024-04-16T16:16:41.056906+0000","last_undegraded":"2024-04-16T16:16:41.056906+0000","last_fullsized":"2024-04-16T16:16:41.056906+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:50:55.122092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.058306+0000","last_change":"2024-04-16T16:16:39.057643+0000","last_active":"2024-04-16T16:16:41.058306+0000","last_peered":"2024-04-16T16:16:41.058306+0000","last_clean":"2024-04-16T16:16:41.058306+0000","last_became_active":"2024-04-16T16:16:39.057160+0000","last_became_peered":"2024-04-16T16:16:39.057160+0000","last_unstale":"2024-04-16T16:16:41.058306+0000","last_undegraded":"2024-04-16T16:16:41.058306+0000","last_fullsized":"2024-04-16T16:16:41.058306+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:59:03.269004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354958+0000","last_change":"2024-04-16T16:16:40.066424+0000","last_active":"2024-04-16T16:16:41.354958+0000","last_peered":"2024-04-16T16:16:41.354958+0000","last_clean":"2024-04-16T16:16:41.354958+0000","last_became_active":"2024-04-16T16:16:40.063948+0000","last_became_peered":"2024-04-16T16:16:40.063948+0000","last_unstale":"2024-04-16T16:16:41.354958+0000","last_undegraded":"2024-04-16T16:16:41.354958+0000","last_fullsized":"2024-04-16T16:16:41.354958+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:59:04.285019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355470+0000","last_change":"2024-04-16T16:16:39.054911+0000","last_active":"2024-04-16T16:16:41.355470+0000","last_peered":"2024-04-16T16:16:41.355470+0000","last_clean":"2024-04-16T16:16:41.355470+0000","last_became_active":"2024-04-16T16:16:39.053957+0000","last_became_peered":"2024-04-16T16:16:39.053957+0000","last_unstale":"2024-04-16T16:16:41.355470+0000","last_undegraded":"2024-04-16T16:16:41.355470+0000","last_fullsized":"2024-04-16T16:16:41.355470+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:46.723824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053204+0000","last_change":"2024-04-16T16:16:40.052810+0000","last_active":"2024-04-16T16:16:41.053204+0000","last_peered":"2024-04-16T16:16:41.053204+0000","last_clean":"2024-04-16T16:16:41.053204+0000","last_became_active":"2024-04-16T16:16:40.051522+0000","last_became_peered":"2024-04-16T16:16:40.051522+0000","last_unstale":"2024-04-16T16:16:41.053204+0000","last_undegraded":"2024-04-16T16:16:41.053204+0000","last_fullsized":"2024-04-16T16:16:41.053204+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:53:25.229933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056633+0000","last_change":"2024-04-16T16:16:39.054182+0000","last_active":"2024-04-16T16:16:41.056633+0000","last_peered":"2024-04-16T16:16:41.056633+0000","last_clean":"2024-04-16T16:16:41.056633+0000","last_became_active":"2024-04-16T16:16:39.053902+0000","last_became_peered":"2024-04-16T16:16:39.053902+0000","last_unstale":"2024-04-16T16:16:41.056633+0000","last_undegraded":"2024-04-16T16:16:41.056633+0000","last_fullsized":"2024-04-16T16:16:41.056633+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:11.219544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353546+0000","last_change":"2024-04-16T16:16:40.062784+0000","last_active":"2024-04-16T16:16:41.353546+0000","last_peered":"2024-04-16T16:16:41.353546+0000","last_clean":"2024-04-16T16:16:41.353546+0000","last_became_active":"2024-04-16T16:16:40.062465+0000","last_became_peered":"2024-04-16T16:16:40.062465+0000","last_unstale":"2024-04-16T16:16:41.353546+0000","last_undegraded":"2024-04-16T16:16:41.353546+0000","last_fullsized":"2024-04-16T16:16:41.353546+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:29:01.496515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355429+0000","last_change":"2024-04-16T16:16:39.045311+0000","last_active":"2024-04-16T16:16:41.355429+0000","last_peered":"2024-04-16T16:16:41.355429+0000","last_clean":"2024-04-16T16:16:41.355429+0000","last_became_active":"2024-04-16T16:16:39.044896+0000","last_became_peered":"2024-04-16T16:16:39.044896+0000","last_unstale":"2024-04-16T16:16:41.355429+0000","last_undegraded":"2024-04-16T16:16:41.355429+0000","last_fullsized":"2024-04-16T16:16:41.355429+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:58:16.807985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354225+0000","last_change":"2024-04-16T16:16:40.054819+0000","last_active":"2024-04-16T16:16:41.354225+0000","last_peered":"2024-04-16T16:16:41.354225+0000","last_clean":"2024-04-16T16:16:41.354225+0000","last_became_active":"2024-04-16T16:16:40.054521+0000","last_became_peered":"2024-04-16T16:16:40.054521+0000","last_unstale":"2024-04-16T16:16:41.354225+0000","last_undegraded":"2024-04-16T16:16:41.354225+0000","last_fullsized":"2024-04-16T16:16:41.354225+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:01:17.068261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.080016+0000","last_change":"2024-04-16T16:16:39.055458+0000","last_active":"2024-04-16T16:16:41.080016+0000","last_peered":"2024-04-16T16:16:41.080016+0000","last_clean":"2024-04-16T16:16:41.080016+0000","last_became_active":"2024-04-16T16:16:39.053012+0000","last_became_peered":"2024-04-16T16:16:39.053012+0000","last_unstale":"2024-04-16T16:16:41.080016+0000","last_undegraded":"2024-04-16T16:16:41.080016+0000","last_fullsized":"2024-04-16T16:16:41.080016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:21.381404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353660+0000","last_change":"2024-04-16T16:16:40.053963+0000","last_active":"2024-04-16T16:16:41.353660+0000","last_peered":"2024-04-16T16:16:41.353660+0000","last_clean":"2024-04-16T16:16:41.353660+0000","last_became_active":"2024-04-16T16:16:40.053682+0000","last_became_peered":"2024-04-16T16:16:40.053682+0000","last_unstale":"2024-04-16T16:16:41.353660+0000","last_undegraded":"2024-04-16T16:16:41.353660+0000","last_fullsized":"2024-04-16T16:16:41.353660+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:16.693760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.083960+0000","last_change":"2024-04-16T16:16:39.038427+0000","last_active":"2024-04-16T16:16:41.083960+0000","last_peered":"2024-04-16T16:16:41.083960+0000","last_clean":"2024-04-16T16:16:41.083960+0000","last_became_active":"2024-04-16T16:16:39.036287+0000","last_became_peered":"2024-04-16T16:16:39.036287+0000","last_unstale":"2024-04-16T16:16:41.083960+0000","last_undegraded":"2024-04-16T16:16:41.083960+0000","last_fullsized":"2024-04-16T16:16:41.083960+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:38.804660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084649+0000","last_change":"2024-04-16T16:16:40.050769+0000","last_active":"2024-04-16T16:16:41.084649+0000","last_peered":"2024-04-16T16:16:41.084649+0000","last_clean":"2024-04-16T16:16:41.084649+0000","last_became_active":"2024-04-16T16:16:40.050449+0000","last_became_peered":"2024-04-16T16:16:40.050449+0000","last_unstale":"2024-04-16T16:16:41.084649+0000","last_undegraded":"2024-04-16T16:16:41.084649+0000","last_fullsized":"2024-04-16T16:16:41.084649+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:29.398550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355074+0000","last_change":"2024-04-16T16:16:39.047443+0000","last_active":"2024-04-16T16:16:41.355074+0000","last_peered":"2024-04-16T16:16:41.355074+0000","last_clean":"2024-04-16T16:16:41.355074+0000","last_became_active":"2024-04-16T16:16:39.046833+0000","last_became_peered":"2024-04-16T16:16:39.046833+0000","last_unstale":"2024-04-16T16:16:41.355074+0000","last_undegraded":"2024-04-16T16:16:41.355074+0000","last_fullsized":"2024-04-16T16:16:41.355074+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:54:40.017503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.080259+0000","last_change":"2024-04-16T16:16:40.062706+0000","last_active":"2024-04-16T16:16:41.080259+0000","last_peered":"2024-04-16T16:16:41.080259+0000","last_clean":"2024-04-16T16:16:41.080259+0000","last_became_active":"2024-04-16T16:16:40.060527+0000","last_became_peered":"2024-04-16T16:16:40.060527+0000","last_unstale":"2024-04-16T16:16:41.080259+0000","last_undegraded":"2024-04-16T16:16:41.080259+0000","last_fullsized":"2024-04-16T16:16:41.080259+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:28:41.607987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"72'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.080437+0000","last_change":"2024-04-16T16:16:39.055132+0000","last_active":"2024-04-16T16:16:41.080437+0000","last_peered":"2024-04-16T16:16:41.080437+0000","last_clean":"2024-04-16T16:16:41.080437+0000","last_became_active":"2024-04-16T16:16:39.054859+0000","last_became_peered":"2024-04-16T16:16:39.054859+0000","last_unstale":"2024-04-16T16:16:41.080437+0000","last_undegraded":"2024-04-16T16:16:41.080437+0000","last_fullsized":"2024-04-16T16:16:41.080437+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:24.450988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056499+0000","last_change":"2024-04-16T16:16:40.056526+0000","last_active":"2024-04-16T16:16:41.056499+0000","last_peered":"2024-04-16T16:16:41.056499+0000","last_clean":"2024-04-16T16:16:41.056499+0000","last_became_active":"2024-04-16T16:16:40.055969+0000","last_became_peered":"2024-04-16T16:16:40.055969+0000","last_unstale":"2024-04-16T16:16:41.056499+0000","last_undegraded":"2024-04-16T16:16:41.056499+0000","last_fullsized":"2024-04-16T16:16:41.056499+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:59:46.671848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053323+0000","last_change":"2024-04-16T16:16:39.044030+0000","last_active":"2024-04-16T16:16:41.053323+0000","last_peered":"2024-04-16T16:16:41.053323+0000","last_clean":"2024-04-16T16:16:41.053323+0000","last_became_active":"2024-04-16T16:16:39.043710+0000","last_became_peered":"2024-04-16T16:16:39.043710+0000","last_unstale":"2024-04-16T16:16:41.053323+0000","last_undegraded":"2024-04-16T16:16:41.053323+0000","last_fullsized":"2024-04-16T16:16:41.053323+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:15.221340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056763+0000","last_change":"2024-04-16T16:16:40.056711+0000","last_active":"2024-04-16T16:16:41.056763+0000","last_peered":"2024-04-16T16:16:41.056763+0000","last_clean":"2024-04-16T16:16:41.056763+0000","last_became_active":"2024-04-16T16:16:40.053649+0000","last_became_peered":"2024-04-16T16:16:40.053649+0000","last_unstale":"2024-04-16T16:16:41.056763+0000","last_undegraded":"2024-04-16T16:16:41.056763+0000","last_fullsized":"2024-04-16T16:16:41.056763+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:45:10.434171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084392+0000","last_change":"2024-04-16T16:16:39.048354+0000","last_active":"2024-04-16T16:16:41.084392+0000","last_peered":"2024-04-16T16:16:41.084392+0000","last_clean":"2024-04-16T16:16:41.084392+0000","last_became_active":"2024-04-16T16:16:39.048098+0000","last_became_peered":"2024-04-16T16:16:39.048098+0000","last_unstale":"2024-04-16T16:16:41.084392+0000","last_undegraded":"2024-04-16T16:16:41.084392+0000","last_fullsized":"2024-04-16T16:16:41.084392+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:37:41.504380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052443+0000","last_change":"2024-04-16T16:16:40.061326+0000","last_active":"2024-04-16T16:16:41.052443+0000","last_peered":"2024-04-16T16:16:41.052443+0000","last_clean":"2024-04-16T16:16:41.052443+0000","last_became_active":"2024-04-16T16:16:40.056522+0000","last_became_peered":"2024-04-16T16:16:40.056522+0000","last_unstale":"2024-04-16T16:16:41.052443+0000","last_undegraded":"2024-04-16T16:16:41.052443+0000","last_fullsized":"2024-04-16T16:16:41.052443+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:43:46.183862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052332+0000","last_change":"2024-04-16T16:16:39.051171+0000","last_active":"2024-04-16T16:16:41.052332+0000","last_peered":"2024-04-16T16:16:41.052332+0000","last_clean":"2024-04-16T16:16:41.052332+0000","last_became_active":"2024-04-16T16:16:39.050841+0000","last_became_peered":"2024-04-16T16:16:39.050841+0000","last_unstale":"2024-04-16T16:16:41.052332+0000","last_undegraded":"2024-04-16T16:16:41.052332+0000","last_fullsized":"2024-04-16T16:16:41.052332+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:32:24.457101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354741+0000","last_change":"2024-04-16T16:16:40.062944+0000","last_active":"2024-04-16T16:16:41.354741+0000","last_peered":"2024-04-16T16:16:41.354741+0000","last_clean":"2024-04-16T16:16:41.354741+0000","last_became_active":"2024-04-16T16:16:40.057251+0000","last_became_peered":"2024-04-16T16:16:40.057251+0000","last_unstale":"2024-04-16T16:16:41.354741+0000","last_undegraded":"2024-04-16T16:16:41.354741+0000","last_fullsized":"2024-04-16T16:16:41.354741+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:12:52.888262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354394+0000","last_change":"2024-04-16T16:16:39.046999+0000","last_active":"2024-04-16T16:16:41.354394+0000","last_peered":"2024-04-16T16:16:41.354394+0000","last_clean":"2024-04-16T16:16:41.354394+0000","last_became_active":"2024-04-16T16:16:39.046741+0000","last_became_peered":"2024-04-16T16:16:39.046741+0000","last_unstale":"2024-04-16T16:16:41.354394+0000","last_undegraded":"2024-04-16T16:16:41.354394+0000","last_fullsized":"2024-04-16T16:16:41.354394+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:58:16.807985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354126+0000","last_change":"2024-04-16T16:16:40.057460+0000","last_active":"2024-04-16T16:16:41.354126+0000","last_peered":"2024-04-16T16:16:41.354126+0000","last_clean":"2024-04-16T16:16:41.354126+0000","last_became_active":"2024-04-16T16:16:40.057142+0000","last_became_peered":"2024-04-16T16:16:40.057142+0000","last_unstale":"2024-04-16T16:16:41.354126+0000","last_undegraded":"2024-04-16T16:16:41.354126+0000","last_fullsized":"2024-04-16T16:16:41.354126+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:44.350988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.055586+0000","last_change":"2024-04-16T16:16:39.053727+0000","last_active":"2024-04-16T16:16:41.055586+0000","last_peered":"2024-04-16T16:16:41.055586+0000","last_clean":"2024-04-16T16:16:41.055586+0000","last_became_active":"2024-04-16T16:16:39.053180+0000","last_became_peered":"2024-04-16T16:16:39.053180+0000","last_unstale":"2024-04-16T16:16:41.055586+0000","last_undegraded":"2024-04-16T16:16:41.055586+0000","last_fullsized":"2024-04-16T16:16:41.055586+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:54:40.017503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354595+0000","last_change":"2024-04-16T16:16:40.062043+0000","last_active":"2024-04-16T16:16:41.354595+0000","last_peered":"2024-04-16T16:16:41.354595+0000","last_clean":"2024-04-16T16:16:41.354595+0000","last_became_active":"2024-04-16T16:16:40.061740+0000","last_became_peered":"2024-04-16T16:16:40.061740+0000","last_unstale":"2024-04-16T16:16:41.354595+0000","last_undegraded":"2024-04-16T16:16:41.354595+0000","last_fullsized":"2024-04-16T16:16:41.354595+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:54:32.532363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355921+0000","last_change":"2024-04-16T16:16:39.055076+0000","last_active":"2024-04-16T16:16:41.355921+0000","last_peered":"2024-04-16T16:16:41.355921+0000","last_clean":"2024-04-16T16:16:41.355921+0000","last_became_active":"2024-04-16T16:16:39.054757+0000","last_became_peered":"2024-04-16T16:16:39.054757+0000","last_unstale":"2024-04-16T16:16:41.355921+0000","last_undegraded":"2024-04-16T16:16:41.355921+0000","last_fullsized":"2024-04-16T16:16:41.355921+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:54:24.777181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353278+0000","last_change":"2024-04-16T16:16:40.056758+0000","last_active":"2024-04-16T16:16:41.353278+0000","last_peered":"2024-04-16T16:16:41.353278+0000","last_clean":"2024-04-16T16:16:41.353278+0000","last_became_active":"2024-04-16T16:16:40.056150+0000","last_became_peered":"2024-04-16T16:16:40.056150+0000","last_unstale":"2024-04-16T16:16:41.353278+0000","last_undegraded":"2024-04-16T16:16:41.353278+0000","last_fullsized":"2024-04-16T16:16:41.353278+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:24.062608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354925+0000","last_change":"2024-04-16T16:16:39.048647+0000","last_active":"2024-04-16T16:16:41.354925+0000","last_peered":"2024-04-16T16:16:41.354925+0000","last_clean":"2024-04-16T16:16:41.354925+0000","last_became_active":"2024-04-16T16:16:39.048341+0000","last_became_peered":"2024-04-16T16:16:39.048341+0000","last_unstale":"2024-04-16T16:16:41.354925+0000","last_undegraded":"2024-04-16T16:16:41.354925+0000","last_fullsized":"2024-04-16T16:16:41.354925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:23:45.725727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"57'139","reported_seq":351,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054822+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:16:41.054822+0000","last_peered":"2024-04-16T16:16:41.054822+0000","last_clean":"2024-04-16T16:16:41.054822+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:16:41.054822+0000","last_undegraded":"2024-04-16T16:16:41.054822+0000","last_fullsized":"2024-04-16T16:16:41.054822+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32628,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339212,"statfs":{"total":96636764160,"available":96603353088,"internally_reserved":0,"allocated":1478656,"data_stored":1001168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":884736,"data_stored":410276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":410274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":880640,"data_stored":410254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":884736,"data_stored":410296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32048,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339792,"statfs":{"total":96636764160,"available":96603947008,"internally_reserved":0,"allocated":888832,"data_stored":411796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":897024,"data_stored":412412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1470464,"data_stored":1000552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:16:45.656 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T16:16:45.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.922+0000 7fda89600640 1 Processor -- start 2024-04-16T16:16:45.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.922+0000 7fda89600640 1 -- start start 2024-04-16T16:16:45.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.924+0000 7fda89600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fda8415ccb0 0x7fda8406df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.924+0000 7fda89600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda8415d360 con 0x7fda8406e480 2024-04-16T16:16:45.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.924+0000 7fda89600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda8415d640 con 0x7fda84146220 2024-04-16T16:16:45.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.924+0000 7fda89600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fda8415d130 con 0x7fda8415ccb0 2024-04-16T16:16:45.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.924+0000 7fda88c00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fda8406e480 0x7fda841608a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35730/0 (socket says 172.21.15.45:35730) 2024-04-16T16:16:45.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.924+0000 7fda88c00640 1 -- 172.21.15.45:0/2971961568 learned_addr learned my addr 172.21.15.45:0/2971961568 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:45.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.924+0000 7fda83400640 1 --2- 172.21.15.45:0/2971961568 >> v2:172.21.15.103:3300/0 conn(0x7fda8415ccb0 0x7fda8406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:45.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.925+0000 7fda82a00640 1 -- 172.21.15.45:0/2971961568 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2101095606 0 0) 0x7fda8415d360 con 0x7fda8406e480 2024-04-16T16:16:45.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.925+0000 7fda82a00640 1 -- 172.21.15.45:0/2971961568 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda68003520 con 0x7fda8406e480 2024-04-16T16:16:45.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.925+0000 7fda82a00640 1 -- 172.21.15.45:0/2971961568 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2455289286 0 0) 0x7fda68003520 con 0x7fda8406e480 2024-04-16T16:16:45.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.925+0000 7fda82a00640 1 -- 172.21.15.45:0/2971961568 >> v2:172.21.15.103:3300/0 conn(0x7fda8415ccb0 msgr2=0x7fda8406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:45.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.925+0000 7fda82a00640 1 --2- 172.21.15.45:0/2971961568 >> v2:172.21.15.103:3300/0 conn(0x7fda8415ccb0 0x7fda8406df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.925+0000 7fda82a00640 1 -- 172.21.15.45:0/2971961568 >> v1:172.21.15.103:6789/0 conn(0x7fda84146220 legacy=0x7fda8406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:45.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.925+0000 7fda82a00640 1 -- 172.21.15.45:0/2971961568 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8415d920 con 0x7fda8406e480 2024-04-16T16:16:45.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.925+0000 7fda82a00640 1 -- 172.21.15.45:0/2971961568 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fda7c001670 con 0x7fda8406e480 2024-04-16T16:16:45.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.925+0000 7fda82a00640 1 -- 172.21.15.45:0/2971961568 >> v1:172.21.15.45:6789/0 conn(0x7fda8406e480 legacy=0x7fda841608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:45.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.925+0000 7fda82a00640 1 --2- 172.21.15.45:0/2971961568 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda680039b0 0x7fda68003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.925+0000 7fda82a00640 1 --2- 172.21.15.45:0/2971961568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda68004800 0x7fda68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.925+0000 7fda82a00640 1 --2- 172.21.15.45:0/2971961568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda680051c0 0x7fda6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.925+0000 7fda82a00640 1 -- 172.21.15.45:0/2971961568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fda6800c030 con 0x7fda680051c0 2024-04-16T16:16:45.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.925+0000 7fda82a00640 1 -- 172.21.15.45:0/2971961568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda6800c1e0 con 0x7fda68004800 2024-04-16T16:16:45.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.925+0000 7fda82a00640 1 -- 172.21.15.45:0/2971961568 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda6800c3e0 con 0x7fda680039b0 2024-04-16T16:16:45.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.926+0000 7fda88c00640 1 --2- 172.21.15.45:0/2971961568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda680051c0 0x7fda6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:45.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.926+0000 7fda83e00640 1 --2- 172.21.15.45:0/2971961568 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda680039b0 0x7fda68003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:45.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.926+0000 7fda83400640 1 --2- 172.21.15.45:0/2971961568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda68004800 0x7fda68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:45.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.926+0000 7fda88c00640 1 -- 172.21.15.45:0/2971961568 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda680039b0 msgr2=0x7fda68003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:45.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.926+0000 7fda88c00640 1 --2- 172.21.15.45:0/2971961568 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda680039b0 0x7fda68003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.926+0000 7fda88c00640 1 -- 172.21.15.45:0/2971961568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda68004800 msgr2=0x7fda68004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:45.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.926+0000 7fda88c00640 1 --2- 172.21.15.45:0/2971961568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda68004800 0x7fda68004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.926+0000 7fda88c00640 1 -- 172.21.15.45:0/2971961568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda68003520 con 0x7fda680051c0 2024-04-16T16:16:45.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.927+0000 7fda88c00640 1 -- 172.21.15.45:0/2971961568 reap_dead start 2024-04-16T16:16:45.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.927+0000 7fda88c00640 1 --2- 172.21.15.45:0/2971961568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda680051c0 0x7fda6800b9c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00c240 tx=0x7fda68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2eb9c91693ccd57 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:45.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.928+0000 7fda82a00640 1 -- 172.21.15.45:0/2971961568 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c007d10 con 0x7fda680051c0 2024-04-16T16:16:45.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.928+0000 7fda82a00640 1 -- 172.21.15.45:0/2971961568 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda7c030070 con 0x7fda680051c0 2024-04-16T16:16:45.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.928+0000 7fda82a00640 1 -- 172.21.15.45:0/2971961568 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c02cd20 con 0x7fda680051c0 2024-04-16T16:16:45.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.928+0000 7fda89600640 1 -- 172.21.15.45:0/2971961568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda680051c0 msgr2=0x7fda6800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:45.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.928+0000 7fda89600640 1 --2- 172.21.15.45:0/2971961568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda680051c0 0x7fda6800b9c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00c240 tx=0x7fda68005830 comp rx=0 tx=0).stop 2024-04-16T16:16:45.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.928+0000 7fda89600640 1 -- 172.21.15.45:0/2971961568 shutdown_connections 2024-04-16T16:16:45.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.928+0000 7fda89600640 1 --2- 172.21.15.45:0/2971961568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda680051c0 0x7fda6800b9c0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.928+0000 7fda89600640 1 -- 172.21.15.45:0/2971961568 >> 172.21.15.45:0/2971961568 conn(0x7fda8406b930 msgr2=0x7fda8406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:45.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.929+0000 7fda89600640 1 -- 172.21.15.45:0/2971961568 shutdown_connections 2024-04-16T16:16:45.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.929+0000 7fda89600640 1 -- 172.21.15.45:0/2971961568 wait complete. 2024-04-16T16:16:45.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.930+0000 7fda89600640 1 Processor -- start 2024-04-16T16:16:45.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.931+0000 7fda89600640 1 -- start start 2024-04-16T16:16:45.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda89600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda680051c0 0x7fda841f7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda89600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda8413d110 0x7fda84159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda89600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda84159c70 0x7fda841f50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda89600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fda8406df40 con 0x7fda680051c0 2024-04-16T16:16:45.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda89600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda84160410 con 0x7fda8413d110 2024-04-16T16:16:45.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda89600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda8406e0c0 con 0x7fda84159c70 2024-04-16T16:16:45.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda83e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda680051c0 0x7fda841f7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:45.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda83e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda680051c0 0x7fda841f7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46528/0 (socket says 172.21.15.45:46528) 2024-04-16T16:16:45.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda83e00640 1 -- 172.21.15.45:0/1778626732 learned_addr learned my addr 172.21.15.45:0/1778626732 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:45.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda83400640 1 --2- 172.21.15.45:0/1778626732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda8413d110 0x7fda84159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:45.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda83e00640 1 -- 172.21.15.45:0/1778626732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda84159c70 msgr2=0x7fda841f50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:45.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda83e00640 1 --2- 172.21.15.45:0/1778626732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda84159c70 0x7fda841f50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda83e00640 1 -- 172.21.15.45:0/1778626732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda8413d110 msgr2=0x7fda84159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:45.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda83e00640 1 --2- 172.21.15.45:0/1778626732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda8413d110 0x7fda84159730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:45.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda83e00640 1 -- 172.21.15.45:0/1778626732 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda841f85d0 con 0x7fda680051c0 2024-04-16T16:16:45.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.932+0000 7fda83400640 1 --2- 172.21.15.45:0/1778626732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda8413d110 0x7fda84159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:45.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.933+0000 7fda83e00640 1 --2- 172.21.15.45:0/1778626732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda680051c0 0x7fda841f7f60 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fda7400e480 tx=0x7fda7400e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:45.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.933+0000 7fda7be00640 1 -- 172.21.15.45:0/1778626732 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda740052e0 con 0x7fda680051c0 2024-04-16T16:16:45.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.933+0000 7fda7be00640 1 -- 172.21.15.45:0/1778626732 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda74018070 con 0x7fda680051c0 2024-04-16T16:16:45.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.933+0000 7fda7be00640 1 -- 172.21.15.45:0/1778626732 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74003830 con 0x7fda680051c0 2024-04-16T16:16:45.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.933+0000 7fda89600640 1 -- 172.21.15.45:0/1778626732 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda841f88b0 con 0x7fda680051c0 2024-04-16T16:16:45.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.933+0000 7fda89600640 1 -- 172.21.15.45:0/1778626732 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda841f4ba0 con 0x7fda680051c0 2024-04-16T16:16:45.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.935+0000 7fda89600640 1 -- 172.21.15.45:0/1778626732 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda841f4990 con 0x7fda680051c0 2024-04-16T16:16:45.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.936+0000 7fda7be00640 1 -- 172.21.15.45:0/1778626732 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fda740039d0 con 0x7fda680051c0 2024-04-16T16:16:45.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.937+0000 7fda7be00640 1 --2- 172.21.15.45:0/1778626732 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fda6006cb30 0x7fda6006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:45.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.937+0000 7fda7be00640 1 -- 172.21.15.45:0/1778626732 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7fda7408bcf0 con 0x7fda680051c0 2024-04-16T16:16:45.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.938+0000 7fda83400640 1 --2- 172.21.15.45:0/1778626732 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fda6006cb30 0x7fda6006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:45.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.939+0000 7fda83400640 1 --2- 172.21.15.45:0/1778626732 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fda6006cb30 0x7fda6006eff0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fda840b1af0 tx=0x7fda6c00b040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:45.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:45.941+0000 7fda7be00640 1 -- 172.21.15.45:0/1778626732 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda74059db0 con 0x7fda680051c0 2024-04-16T16:16:46.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.224+0000 7fda89600640 1 -- 172.21.15.45:0/1778626732 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fda84075780 con 0x7fda680051c0 2024-04-16T16:16:46.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.225+0000 7fda7be00640 1 -- 172.21.15.45:0/1778626732 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+3208 (secure 0 0 0) 0x7fda740596b0 con 0x7fda680051c0 2024-04-16T16:16:46.225 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:46.225 INFO:teuthology.orchestra.run.smithi045.stdout:{"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":220,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":72,"num_osds":8,"num_up_osds":8,"osd_up_since":1713283997,"num_in_osds":8,"osd_in_since":1713283989,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":259440640,"bytes_avail":772834672640,"bytes_total":773094113280,"write_bytes_sec":2364,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":50,"id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"a","status":"up:active","gid":6402}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-16T16:16:25.060027+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"y":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"6":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"7":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-16T16:16:46.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.228+0000 7fda89600640 1 -- 172.21.15.45:0/1778626732 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fda6006cb30 msgr2=0x7fda6006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:46.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.228+0000 7fda89600640 1 --2- 172.21.15.45:0/1778626732 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fda6006cb30 0x7fda6006eff0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fda840b1af0 tx=0x7fda6c00b040 comp rx=0 tx=0).stop 2024-04-16T16:16:46.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.228+0000 7fda89600640 1 -- 172.21.15.45:0/1778626732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda680051c0 msgr2=0x7fda841f7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:46.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.228+0000 7fda89600640 1 --2- 172.21.15.45:0/1778626732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda680051c0 0x7fda841f7f60 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fda7400e480 tx=0x7fda7400e940 comp rx=0 tx=0).stop 2024-04-16T16:16:46.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.228+0000 7fda89600640 1 -- 172.21.15.45:0/1778626732 shutdown_connections 2024-04-16T16:16:46.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.228+0000 7fda89600640 1 --2- 172.21.15.45:0/1778626732 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fda6006cb30 0x7fda6006eff0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:46.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.228+0000 7fda89600640 1 --2- 172.21.15.45:0/1778626732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda84159c70 0x7fda841f50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:46.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.228+0000 7fda89600640 1 --2- 172.21.15.45:0/1778626732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda8413d110 0x7fda84159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:46.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.228+0000 7fda89600640 1 --2- 172.21.15.45:0/1778626732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda680051c0 0x7fda841f7f60 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:46.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.228+0000 7fda89600640 1 -- 172.21.15.45:0/1778626732 >> 172.21.15.45:0/1778626732 conn(0x7fda8406b930 msgr2=0x7fda840c6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:46.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.228+0000 7fda89600640 1 -- 172.21.15.45:0/1778626732 shutdown_connections 2024-04-16T16:16:46.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.228+0000 7fda89600640 1 -- 172.21.15.45:0/1778626732 wait complete. 2024-04-16T16:16:46.247 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-16T16:16:46.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:16:46.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.464+0000 7f3d93200640 1 Processor -- start 2024-04-16T16:16:46.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.465+0000 7f3d93200640 1 -- start start 2024-04-16T16:16:46.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.466+0000 7f3d93200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3d94162270 0x7f3d940d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:46.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.466+0000 7f3d93200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d940d3080 con 0x7f3d940d52d0 2024-04-16T16:16:46.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.466+0000 7f3d93200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d940d3360 con 0x7f3d94161e90 2024-04-16T16:16:46.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.466+0000 7f3d93200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3d940d2e30 con 0x7f3d94162270 2024-04-16T16:16:46.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.467+0000 7f3d92800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3d940d52d0 0x7f3d9406ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35740/0 (socket says 172.21.15.45:35740) 2024-04-16T16:16:46.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.467+0000 7f3d92800640 1 -- 172.21.15.45:0/1982705012 learned_addr learned my addr 172.21.15.45:0/1982705012 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:46.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.467+0000 7f3d8be00640 1 --2- 172.21.15.45:0/1982705012 >> v2:172.21.15.103:3300/0 conn(0x7f3d94162270 0x7f3d940d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:46.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.467+0000 7f3d91400640 1 -- 172.21.15.45:0/1982705012 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3920436394 0 0) 0x7f3d940d3080 con 0x7f3d940d52d0 2024-04-16T16:16:46.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.467+0000 7f3d91400640 1 -- 172.21.15.45:0/1982705012 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d78003520 con 0x7f3d940d52d0 2024-04-16T16:16:46.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.468+0000 7f3d91400640 1 -- 172.21.15.45:0/1982705012 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3194912215 0 0) 0x7f3d78003520 con 0x7f3d940d52d0 2024-04-16T16:16:46.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.468+0000 7f3d91400640 1 -- 172.21.15.45:0/1982705012 >> v2:172.21.15.103:3300/0 conn(0x7f3d94162270 msgr2=0x7f3d940d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:46.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.468+0000 7f3d91400640 1 --2- 172.21.15.45:0/1982705012 >> v2:172.21.15.103:3300/0 conn(0x7f3d94162270 0x7f3d940d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:46.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.468+0000 7f3d91400640 1 -- 172.21.15.45:0/1982705012 >> v1:172.21.15.103:6789/0 conn(0x7f3d94161e90 legacy=0x7f3d940cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:46.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.468+0000 7f3d91400640 1 -- 172.21.15.45:0/1982705012 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d940d3640 con 0x7f3d940d52d0 2024-04-16T16:16:46.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.468+0000 7f3d91400640 1 -- 172.21.15.45:0/1982705012 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f3d8c0019f0 con 0x7f3d940d52d0 2024-04-16T16:16:46.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.468+0000 7f3d91400640 1 -- 172.21.15.45:0/1982705012 >> v1:172.21.15.45:6789/0 conn(0x7f3d940d52d0 legacy=0x7f3d9406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:46.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.468+0000 7f3d91400640 1 --2- 172.21.15.45:0/1982705012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d780039e0 0x7f3d78003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:46.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.468+0000 7f3d91400640 1 --2- 172.21.15.45:0/1982705012 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d78004800 0x7f3d78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:46.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.468+0000 7f3d91400640 1 --2- 172.21.15.45:0/1982705012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d780051c0 0x7f3d7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:46.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.468+0000 7f3d91400640 1 -- 172.21.15.45:0/1982705012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3d940d2e30 con 0x7f3d780039e0 2024-04-16T16:16:46.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.468+0000 7f3d91400640 1 -- 172.21.15.45:0/1982705012 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7800c090 con 0x7f3d780051c0 2024-04-16T16:16:46.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.468+0000 7f3d91400640 1 -- 172.21.15.45:0/1982705012 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3d7800c290 con 0x7f3d78004800 2024-04-16T16:16:46.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.468+0000 7f3d91e00640 1 --2- 172.21.15.45:0/1982705012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d780039e0 0x7f3d78003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:46.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.469+0000 7f3d92800640 1 --2- 172.21.15.45:0/1982705012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d780051c0 0x7f3d7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:46.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.469+0000 7f3d91e00640 1 -- 172.21.15.45:0/1982705012 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d78004800 msgr2=0x7f3d78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:46.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.469+0000 7f3d91e00640 1 --2- 172.21.15.45:0/1982705012 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d78004800 0x7f3d78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:46.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.469+0000 7f3d91e00640 1 -- 172.21.15.45:0/1982705012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d780051c0 msgr2=0x7f3d7800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:46.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.469+0000 7f3d91e00640 1 --2- 172.21.15.45:0/1982705012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d780051c0 0x7f3d7800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:46.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.469+0000 7f3d91e00640 1 -- 172.21.15.45:0/1982705012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d78003520 con 0x7f3d780039e0 2024-04-16T16:16:46.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.469+0000 7f3d92800640 1 -- 172.21.15.45:0/1982705012 reap_dead start 2024-04-16T16:16:46.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.470+0000 7f3d91e00640 1 --2- 172.21.15.45:0/1982705012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d780039e0 0x7f3d78003e00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f3d84001750 tx=0x7f3d78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abbe9c4aaf160a64 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:46.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.470+0000 7f3d91400640 1 -- 172.21.15.45:0/1982705012 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d84001370 con 0x7f3d780039e0 2024-04-16T16:16:46.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.470+0000 7f3d91400640 1 -- 172.21.15.45:0/1982705012 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d840052d0 con 0x7f3d780039e0 2024-04-16T16:16:46.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.470+0000 7f3d91400640 1 -- 172.21.15.45:0/1982705012 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d84003830 con 0x7f3d780039e0 2024-04-16T16:16:46.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.470+0000 7f3d93200640 1 -- 172.21.15.45:0/1982705012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d780039e0 msgr2=0x7f3d78003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:46.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.470+0000 7f3d93200640 1 --2- 172.21.15.45:0/1982705012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d780039e0 0x7f3d78003e00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f3d84001750 tx=0x7f3d78005830 comp rx=0 tx=0).stop 2024-04-16T16:16:46.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.471+0000 7f3d93200640 1 -- 172.21.15.45:0/1982705012 shutdown_connections 2024-04-16T16:16:46.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.471+0000 7f3d93200640 1 --2- 172.21.15.45:0/1982705012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d780039e0 0x7f3d78003e00 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:46.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.471+0000 7f3d93200640 1 -- 172.21.15.45:0/1982705012 >> 172.21.15.45:0/1982705012 conn(0x7f3d940d1940 msgr2=0x7f3d940d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:46.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.471+0000 7f3d93200640 1 -- 172.21.15.45:0/1982705012 shutdown_connections 2024-04-16T16:16:46.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.471+0000 7f3d93200640 1 -- 172.21.15.45:0/1982705012 wait complete. 2024-04-16T16:16:46.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.472+0000 7f3d93200640 1 Processor -- start 2024-04-16T16:16:46.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.473+0000 7f3d93200640 1 -- start start 2024-04-16T16:16:46.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.473+0000 7f3d93200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d780039e0 0x7f3d94159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:46.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d93200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9414cb80 0x7f3d94158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:46.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d93200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d941589a0 0x7f3d940c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:46.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d93200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3d940cf950 con 0x7f3d941589a0 2024-04-16T16:16:46.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d93200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3d940cf7d0 con 0x7f3d9414cb80 2024-04-16T16:16:46.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d93200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3d940cfad0 con 0x7f3d780039e0 2024-04-16T16:16:46.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d92800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d941589a0 0x7f3d940c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:46.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d92800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d941589a0 0x7f3d940c8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46548/0 (socket says 172.21.15.45:46548) 2024-04-16T16:16:46.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d92800640 1 -- 172.21.15.45:0/3933100660 learned_addr learned my addr 172.21.15.45:0/3933100660 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:46.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d91e00640 1 --2- 172.21.15.45:0/3933100660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d780039e0 0x7f3d94159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:46.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d92800640 1 -- 172.21.15.45:0/3933100660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d780039e0 msgr2=0x7f3d94159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:46.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d92800640 1 --2- 172.21.15.45:0/3933100660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d780039e0 0x7f3d94159a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:46.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d8be00640 1 --2- 172.21.15.45:0/3933100660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9414cb80 0x7f3d94158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:46.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d92800640 1 -- 172.21.15.45:0/3933100660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9414cb80 msgr2=0x7f3d94158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:46.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d92800640 1 --2- 172.21.15.45:0/3933100660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9414cb80 0x7f3d94158460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:46.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d92800640 1 -- 172.21.15.45:0/3933100660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d94159fb0 con 0x7f3d941589a0 2024-04-16T16:16:46.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.474+0000 7f3d91e00640 1 --2- 172.21.15.45:0/3933100660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d780039e0 0x7f3d94159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:46.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.475+0000 7f3d92800640 1 --2- 172.21.15.45:0/3933100660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d941589a0 0x7f3d940c8e00 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c001330 tx=0x7f3d8c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:46.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.475+0000 7f3d8a000640 1 -- 172.21.15.45:0/3933100660 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c004d60 con 0x7f3d941589a0 2024-04-16T16:16:46.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.475+0000 7f3d8a000640 1 -- 172.21.15.45:0/3933100660 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d8c00bb80 con 0x7f3d941589a0 2024-04-16T16:16:46.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.475+0000 7f3d8a000640 1 -- 172.21.15.45:0/3933100660 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c007070 con 0x7f3d941589a0 2024-04-16T16:16:46.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.475+0000 7f3d93200640 1 -- 172.21.15.45:0/3933100660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d9415a230 con 0x7f3d941589a0 2024-04-16T16:16:46.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.475+0000 7f3d93200640 1 -- 172.21.15.45:0/3933100660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d9415a7f0 con 0x7f3d941589a0 2024-04-16T16:16:46.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.476+0000 7f3d93200640 1 -- 172.21.15.45:0/3933100660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d94075780 con 0x7f3d941589a0 2024-04-16T16:16:46.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.477+0000 7f3d8a000640 1 -- 172.21.15.45:0/3933100660 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f3d8c01b060 con 0x7f3d941589a0 2024-04-16T16:16:46.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.478+0000 7f3d8a000640 1 --2- 172.21.15.45:0/3933100660 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f3d7006cb30 0x7f3d7006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:46.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.478+0000 7f3d8a000640 1 -- 172.21.15.45:0/3933100660 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f3d8c08c1d0 con 0x7f3d941589a0 2024-04-16T16:16:46.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.479+0000 7f3d91e00640 1 --2- 172.21.15.45:0/3933100660 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f3d7006cb30 0x7f3d7006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:46.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.480+0000 7f3d91e00640 1 --2- 172.21.15.45:0/3933100660 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f3d7006cb30 0x7f3d7006eff0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3d84033000 tx=0x7f3d84009030 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:46.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.484+0000 7f3d8a000640 1 -- 172.21.15.45:0/3933100660 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d94159fb0 con 0x7f3d941589a0 2024-04-16T16:16:46.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.686+0000 7f3d93200640 1 -- 172.21.15.45:0/3933100660 --> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3d9406dae0 con 0x7f3d7006cb30 2024-04-16T16:16:46.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.705+0000 7f3d8a000640 1 -- 172.21.15.45:0/3933100660 <== mgr.4961 v2:172.21.15.103:6832/4155603915 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252069 (secure 0 0 0) 0x7f3d9406dae0 con 0x7f3d7006cb30 2024-04-16T16:16:46.709 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:46.710 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:16:46.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.711+0000 7f3d93200640 1 -- 172.21.15.45:0/3933100660 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f3d7006cb30 msgr2=0x7f3d7006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:46.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.711+0000 7f3d93200640 1 --2- 172.21.15.45:0/3933100660 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f3d7006cb30 0x7f3d7006eff0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3d84033000 tx=0x7f3d84009030 comp rx=0 tx=0).stop 2024-04-16T16:16:46.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.711+0000 7f3d93200640 1 -- 172.21.15.45:0/3933100660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d941589a0 msgr2=0x7f3d940c8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:46.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.711+0000 7f3d93200640 1 --2- 172.21.15.45:0/3933100660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d941589a0 0x7f3d940c8e00 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c001330 tx=0x7f3d8c00b4a0 comp rx=0 tx=0).stop 2024-04-16T16:16:46.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.711+0000 7f3d93200640 1 -- 172.21.15.45:0/3933100660 shutdown_connections 2024-04-16T16:16:46.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.711+0000 7f3d93200640 1 --2- 172.21.15.45:0/3933100660 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f3d7006cb30 0x7f3d7006eff0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:46.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.711+0000 7f3d93200640 1 --2- 172.21.15.45:0/3933100660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d941589a0 0x7f3d940c8e00 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:46.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.711+0000 7f3d93200640 1 --2- 172.21.15.45:0/3933100660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9414cb80 0x7f3d94158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:46.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.711+0000 7f3d93200640 1 --2- 172.21.15.45:0/3933100660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d780039e0 0x7f3d94159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:46.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.711+0000 7f3d93200640 1 -- 172.21.15.45:0/3933100660 >> 172.21.15.45:0/3933100660 conn(0x7f3d940d1940 msgr2=0x7f3d940b6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:46.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.712+0000 7f3d93200640 1 -- 172.21.15.45:0/3933100660 shutdown_connections 2024-04-16T16:16:46.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:46.712+0000 7f3d93200640 1 -- 172.21.15.45:0/3933100660 wait complete. 2024-04-16T16:16:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-16T16:16:45.082003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":244,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":178,"ondisk_log_size":178,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":161,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":253360,"kb_used_data":8072,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754721360,"statfs":{"total":773094113280,"available":772834672640,"internally_reserved":0,"allocated":8265728,"data_stored":4467028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.062676"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.059101+0000","last_change":"2024-04-16T16:16:40.048700+0000","last_active":"2024-04-16T16:16:41.059101+0000","last_peered":"2024-04-16T16:16:41.059101+0000","last_clean":"2024-04-16T16:16:41.059101+0000","last_became_active":"2024-04-16T16:16:40.046882+0000","last_became_peered":"2024-04-16T16:16:40.046882+0000","last_unstale":"2024-04-16T16:16:41.059101+0000","last_undegraded":"2024-04-16T16:16:41.059101+0000","last_fullsized":"2024-04-16T16:16:41.059101+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:41:52.714263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355645+0000","last_change":"2024-04-16T16:16:40.063532+0000","last_active":"2024-04-16T16:16:41.355645+0000","last_peered":"2024-04-16T16:16:41.355645+0000","last_clean":"2024-04-16T16:16:41.355645+0000","last_became_active":"2024-04-16T16:16:40.063023+0000","last_became_peered":"2024-04-16T16:16:40.063023+0000","last_unstale":"2024-04-16T16:16:41.355645+0000","last_undegraded":"2024-04-16T16:16:41.355645+0000","last_fullsized":"2024-04-16T16:16:41.355645+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:45.673814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353809+0000","last_change":"2024-04-16T16:16:40.058970+0000","last_active":"2024-04-16T16:16:41.353809+0000","last_peered":"2024-04-16T16:16:41.353809+0000","last_clean":"2024-04-16T16:16:41.353809+0000","last_became_active":"2024-04-16T16:16:40.058582+0000","last_became_peered":"2024-04-16T16:16:40.058582+0000","last_unstale":"2024-04-16T16:16:41.353809+0000","last_undegraded":"2024-04-16T16:16:41.353809+0000","last_fullsized":"2024-04-16T16:16:41.353809+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:04.907779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354998+0000","last_change":"2024-04-16T16:16:40.067374+0000","last_active":"2024-04-16T16:16:41.354998+0000","last_peered":"2024-04-16T16:16:41.354998+0000","last_clean":"2024-04-16T16:16:41.354998+0000","last_became_active":"2024-04-16T16:16:40.064314+0000","last_became_peered":"2024-04-16T16:16:40.064314+0000","last_unstale":"2024-04-16T16:16:41.354998+0000","last_undegraded":"2024-04-16T16:16:41.354998+0000","last_fullsized":"2024-04-16T16:16:41.354998+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:47:33.049888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052264+0000","last_change":"2024-04-16T16:16:40.053258+0000","last_active":"2024-04-16T16:16:41.052264+0000","last_peered":"2024-04-16T16:16:41.052264+0000","last_clean":"2024-04-16T16:16:41.052264+0000","last_became_active":"2024-04-16T16:16:40.051051+0000","last_became_peered":"2024-04-16T16:16:40.051051+0000","last_unstale":"2024-04-16T16:16:41.052264+0000","last_undegraded":"2024-04-16T16:16:41.052264+0000","last_fullsized":"2024-04-16T16:16:41.052264+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:16:10.608170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355834+0000","last_change":"2024-04-16T16:16:40.064127+0000","last_active":"2024-04-16T16:16:41.355834+0000","last_peered":"2024-04-16T16:16:41.355834+0000","last_clean":"2024-04-16T16:16:41.355834+0000","last_became_active":"2024-04-16T16:16:40.063617+0000","last_became_peered":"2024-04-16T16:16:40.063617+0000","last_unstale":"2024-04-16T16:16:41.355834+0000","last_undegraded":"2024-04-16T16:16:41.355834+0000","last_fullsized":"2024-04-16T16:16:41.355834+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:54:31.905334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079437+0000","last_change":"2024-04-16T16:16:40.060797+0000","last_active":"2024-04-16T16:16:41.079437+0000","last_peered":"2024-04-16T16:16:41.079437+0000","last_clean":"2024-04-16T16:16:41.079437+0000","last_became_active":"2024-04-16T16:16:40.060497+0000","last_became_peered":"2024-04-16T16:16:40.060497+0000","last_unstale":"2024-04-16T16:16:41.079437+0000","last_undegraded":"2024-04-16T16:16:41.079437+0000","last_fullsized":"2024-04-16T16:16:41.079437+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:43.199657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.356013+0000","last_change":"2024-04-16T16:16:40.067270+0000","last_active":"2024-04-16T16:16:41.356013+0000","last_peered":"2024-04-16T16:16:41.356013+0000","last_clean":"2024-04-16T16:16:41.356013+0000","last_became_active":"2024-04-16T16:16:40.066527+0000","last_became_peered":"2024-04-16T16:16:40.066527+0000","last_unstale":"2024-04-16T16:16:41.356013+0000","last_undegraded":"2024-04-16T16:16:41.356013+0000","last_fullsized":"2024-04-16T16:16:41.356013+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:06:06.933720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053974+0000","last_change":"2024-04-16T16:16:40.065283+0000","last_active":"2024-04-16T16:16:41.053974+0000","last_peered":"2024-04-16T16:16:41.053974+0000","last_clean":"2024-04-16T16:16:41.053974+0000","last_became_active":"2024-04-16T16:16:40.064211+0000","last_became_peered":"2024-04-16T16:16:40.064211+0000","last_unstale":"2024-04-16T16:16:41.053974+0000","last_undegraded":"2024-04-16T16:16:41.053974+0000","last_fullsized":"2024-04-16T16:16:41.053974+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:28.836553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.058144+0000","last_change":"2024-04-16T16:16:40.057046+0000","last_active":"2024-04-16T16:16:41.058144+0000","last_peered":"2024-04-16T16:16:41.058144+0000","last_clean":"2024-04-16T16:16:41.058144+0000","last_became_active":"2024-04-16T16:16:40.056773+0000","last_became_peered":"2024-04-16T16:16:40.056773+0000","last_unstale":"2024-04-16T16:16:41.058144+0000","last_undegraded":"2024-04-16T16:16:41.058144+0000","last_fullsized":"2024-04-16T16:16:41.058144+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:54:31.905334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.055911+0000","last_change":"2024-04-16T16:16:40.059929+0000","last_active":"2024-04-16T16:16:41.055911+0000","last_peered":"2024-04-16T16:16:41.055911+0000","last_clean":"2024-04-16T16:16:41.055911+0000","last_became_active":"2024-04-16T16:16:40.059666+0000","last_became_peered":"2024-04-16T16:16:40.059666+0000","last_unstale":"2024-04-16T16:16:41.055911+0000","last_undegraded":"2024-04-16T16:16:41.055911+0000","last_fullsized":"2024-04-16T16:16:41.055911+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:21.332023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354796+0000","last_change":"2024-04-16T16:16:40.070880+0000","last_active":"2024-04-16T16:16:41.354796+0000","last_peered":"2024-04-16T16:16:41.354796+0000","last_clean":"2024-04-16T16:16:41.354796+0000","last_became_active":"2024-04-16T16:16:40.070434+0000","last_became_peered":"2024-04-16T16:16:40.070434+0000","last_unstale":"2024-04-16T16:16:41.354796+0000","last_undegraded":"2024-04-16T16:16:41.354796+0000","last_fullsized":"2024-04-16T16:16:41.354796+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:13:47.015413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084447+0000","last_change":"2024-04-16T16:16:40.053229+0000","last_active":"2024-04-16T16:16:41.084447+0000","last_peered":"2024-04-16T16:16:41.084447+0000","last_clean":"2024-04-16T16:16:41.084447+0000","last_became_active":"2024-04-16T16:16:40.052968+0000","last_became_peered":"2024-04-16T16:16:40.052968+0000","last_unstale":"2024-04-16T16:16:41.084447+0000","last_undegraded":"2024-04-16T16:16:41.084447+0000","last_fullsized":"2024-04-16T16:16:41.084447+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:46.618649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052568+0000","last_change":"2024-04-16T16:16:40.059138+0000","last_active":"2024-04-16T16:16:41.052568+0000","last_peered":"2024-04-16T16:16:41.052568+0000","last_clean":"2024-04-16T16:16:41.052568+0000","last_became_active":"2024-04-16T16:16:40.056131+0000","last_became_peered":"2024-04-16T16:16:40.056131+0000","last_unstale":"2024-04-16T16:16:41.052568+0000","last_undegraded":"2024-04-16T16:16:41.052568+0000","last_fullsized":"2024-04-16T16:16:41.052568+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:39:52.139845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084236+0000","last_change":"2024-04-16T16:16:40.052923+0000","last_active":"2024-04-16T16:16:41.084236+0000","last_peered":"2024-04-16T16:16:41.084236+0000","last_clean":"2024-04-16T16:16:41.084236+0000","last_became_active":"2024-04-16T16:16:40.052609+0000","last_became_peered":"2024-04-16T16:16:40.052609+0000","last_unstale":"2024-04-16T16:16:41.084236+0000","last_undegraded":"2024-04-16T16:16:41.084236+0000","last_fullsized":"2024-04-16T16:16:41.084236+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:21.332023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057643+0000","last_change":"2024-04-16T16:16:40.055850+0000","last_active":"2024-04-16T16:16:41.057643+0000","last_peered":"2024-04-16T16:16:41.057643+0000","last_clean":"2024-04-16T16:16:41.057643+0000","last_became_active":"2024-04-16T16:16:40.055560+0000","last_became_peered":"2024-04-16T16:16:40.055560+0000","last_unstale":"2024-04-16T16:16:41.057643+0000","last_undegraded":"2024-04-16T16:16:41.057643+0000","last_fullsized":"2024-04-16T16:16:41.057643+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:22.397420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054029+0000","last_change":"2024-04-16T16:16:40.069834+0000","last_active":"2024-04-16T16:16:41.054029+0000","last_peered":"2024-04-16T16:16:41.054029+0000","last_clean":"2024-04-16T16:16:41.054029+0000","last_became_active":"2024-04-16T16:16:40.062429+0000","last_became_peered":"2024-04-16T16:16:40.062429+0000","last_unstale":"2024-04-16T16:16:41.054029+0000","last_undegraded":"2024-04-16T16:16:41.054029+0000","last_fullsized":"2024-04-16T16:16:41.054029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:14:20.465129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355562+0000","last_change":"2024-04-16T16:16:40.055791+0000","last_active":"2024-04-16T16:16:41.355562+0000","last_peered":"2024-04-16T16:16:41.355562+0000","last_clean":"2024-04-16T16:16:41.355562+0000","last_became_active":"2024-04-16T16:16:40.055508+0000","last_became_peered":"2024-04-16T16:16:40.055508+0000","last_unstale":"2024-04-16T16:16:41.355562+0000","last_undegraded":"2024-04-16T16:16:41.355562+0000","last_fullsized":"2024-04-16T16:16:41.355562+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:44.062254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355381+0000","last_change":"2024-04-16T16:16:40.067596+0000","last_active":"2024-04-16T16:16:41.355381+0000","last_peered":"2024-04-16T16:16:41.355381+0000","last_clean":"2024-04-16T16:16:41.355381+0000","last_became_active":"2024-04-16T16:16:40.067146+0000","last_became_peered":"2024-04-16T16:16:40.067146+0000","last_unstale":"2024-04-16T16:16:41.355381+0000","last_undegraded":"2024-04-16T16:16:41.355381+0000","last_fullsized":"2024-04-16T16:16:41.355381+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:07:41.285674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084256+0000","last_change":"2024-04-16T16:16:40.058979+0000","last_active":"2024-04-16T16:16:41.084256+0000","last_peered":"2024-04-16T16:16:41.084256+0000","last_clean":"2024-04-16T16:16:41.084256+0000","last_became_active":"2024-04-16T16:16:40.058648+0000","last_became_peered":"2024-04-16T16:16:40.058648+0000","last_unstale":"2024-04-16T16:16:41.084256+0000","last_undegraded":"2024-04-16T16:16:41.084256+0000","last_fullsized":"2024-04-16T16:16:41.084256+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:28:41.607987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056889+0000","last_change":"2024-04-16T16:16:40.065622+0000","last_active":"2024-04-16T16:16:41.056889+0000","last_peered":"2024-04-16T16:16:41.056889+0000","last_clean":"2024-04-16T16:16:41.056889+0000","last_became_active":"2024-04-16T16:16:40.064748+0000","last_became_peered":"2024-04-16T16:16:40.064748+0000","last_unstale":"2024-04-16T16:16:41.056889+0000","last_undegraded":"2024-04-16T16:16:41.056889+0000","last_fullsized":"2024-04-16T16:16:41.056889+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:12:52.888262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057883+0000","last_change":"2024-04-16T16:16:40.054692+0000","last_active":"2024-04-16T16:16:41.057883+0000","last_peered":"2024-04-16T16:16:41.057883+0000","last_clean":"2024-04-16T16:16:41.057883+0000","last_became_active":"2024-04-16T16:16:40.054392+0000","last_became_peered":"2024-04-16T16:16:40.054392+0000","last_unstale":"2024-04-16T16:16:41.057883+0000","last_undegraded":"2024-04-16T16:16:41.057883+0000","last_fullsized":"2024-04-16T16:16:41.057883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:25.467004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056193+0000","last_change":"2024-04-16T16:16:40.056002+0000","last_active":"2024-04-16T16:16:41.056193+0000","last_peered":"2024-04-16T16:16:41.056193+0000","last_clean":"2024-04-16T16:16:41.056193+0000","last_became_active":"2024-04-16T16:16:40.055212+0000","last_became_peered":"2024-04-16T16:16:40.055212+0000","last_unstale":"2024-04-16T16:16:41.056193+0000","last_undegraded":"2024-04-16T16:16:41.056193+0000","last_fullsized":"2024-04-16T16:16:41.056193+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:28:41.607987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054913+0000","last_change":"2024-04-16T16:16:40.054708+0000","last_active":"2024-04-16T16:16:41.054913+0000","last_peered":"2024-04-16T16:16:41.054913+0000","last_clean":"2024-04-16T16:16:41.054913+0000","last_became_active":"2024-04-16T16:16:40.053297+0000","last_became_peered":"2024-04-16T16:16:40.053297+0000","last_unstale":"2024-04-16T16:16:41.054913+0000","last_undegraded":"2024-04-16T16:16:41.054913+0000","last_fullsized":"2024-04-16T16:16:41.054913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:31:33.590968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.058757+0000","last_change":"2024-04-16T16:16:40.053469+0000","last_active":"2024-04-16T16:16:41.058757+0000","last_peered":"2024-04-16T16:16:41.058757+0000","last_clean":"2024-04-16T16:16:41.058757+0000","last_became_active":"2024-04-16T16:16:40.052617+0000","last_became_peered":"2024-04-16T16:16:40.052617+0000","last_unstale":"2024-04-16T16:16:41.058757+0000","last_undegraded":"2024-04-16T16:16:41.058757+0000","last_fullsized":"2024-04-16T16:16:41.058757+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:06:06.933720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.080143+0000","last_change":"2024-04-16T16:16:40.058621+0000","last_active":"2024-04-16T16:16:41.080143+0000","last_peered":"2024-04-16T16:16:41.080143+0000","last_clean":"2024-04-16T16:16:41.080143+0000","last_became_active":"2024-04-16T16:16:40.058250+0000","last_became_peered":"2024-04-16T16:16:40.058250+0000","last_unstale":"2024-04-16T16:16:41.080143+0000","last_undegraded":"2024-04-16T16:16:41.080143+0000","last_fullsized":"2024-04-16T16:16:41.080143+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:21.332023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353888+0000","last_change":"2024-04-16T16:16:40.066255+0000","last_active":"2024-04-16T16:16:41.353888+0000","last_peered":"2024-04-16T16:16:41.353888+0000","last_clean":"2024-04-16T16:16:41.353888+0000","last_became_active":"2024-04-16T16:16:40.065779+0000","last_became_peered":"2024-04-16T16:16:40.065779+0000","last_unstale":"2024-04-16T16:16:41.353888+0000","last_undegraded":"2024-04-16T16:16:41.353888+0000","last_fullsized":"2024-04-16T16:16:41.353888+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:26:05.697779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057071+0000","last_change":"2024-04-16T16:16:40.056911+0000","last_active":"2024-04-16T16:16:41.057071+0000","last_peered":"2024-04-16T16:16:41.057071+0000","last_clean":"2024-04-16T16:16:41.057071+0000","last_became_active":"2024-04-16T16:16:40.054450+0000","last_became_peered":"2024-04-16T16:16:40.054450+0000","last_unstale":"2024-04-16T16:16:41.057071+0000","last_undegraded":"2024-04-16T16:16:41.057071+0000","last_fullsized":"2024-04-16T16:16:41.057071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:07:41.285674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353512+0000","last_change":"2024-04-16T16:16:40.055812+0000","last_active":"2024-04-16T16:16:41.353512+0000","last_peered":"2024-04-16T16:16:41.353512+0000","last_clean":"2024-04-16T16:16:41.353512+0000","last_became_active":"2024-04-16T16:16:40.055488+0000","last_became_peered":"2024-04-16T16:16:40.055488+0000","last_unstale":"2024-04-16T16:16:41.353512+0000","last_undegraded":"2024-04-16T16:16:41.353512+0000","last_fullsized":"2024-04-16T16:16:41.353512+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:49:49.405418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079606+0000","last_change":"2024-04-16T16:16:40.056514+0000","last_active":"2024-04-16T16:16:41.079606+0000","last_peered":"2024-04-16T16:16:41.079606+0000","last_clean":"2024-04-16T16:16:41.079606+0000","last_became_active":"2024-04-16T16:16:40.056184+0000","last_became_peered":"2024-04-16T16:16:40.056184+0000","last_unstale":"2024-04-16T16:16:41.079606+0000","last_undegraded":"2024-04-16T16:16:41.079606+0000","last_fullsized":"2024-04-16T16:16:41.079606+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:08:39.372895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079878+0000","last_change":"2024-04-16T16:16:40.054600+0000","last_active":"2024-04-16T16:16:41.079878+0000","last_peered":"2024-04-16T16:16:41.079878+0000","last_clean":"2024-04-16T16:16:41.079878+0000","last_became_active":"2024-04-16T16:16:40.054303+0000","last_became_peered":"2024-04-16T16:16:40.054303+0000","last_unstale":"2024-04-16T16:16:41.079878+0000","last_undegraded":"2024-04-16T16:16:41.079878+0000","last_fullsized":"2024-04-16T16:16:41.079878+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:31:33.590968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355739+0000","last_change":"2024-04-16T16:16:40.066863+0000","last_active":"2024-04-16T16:16:41.355739+0000","last_peered":"2024-04-16T16:16:41.355739+0000","last_clean":"2024-04-16T16:16:41.355739+0000","last_became_active":"2024-04-16T16:16:40.064405+0000","last_became_peered":"2024-04-16T16:16:40.064405+0000","last_unstale":"2024-04-16T16:16:41.355739+0000","last_undegraded":"2024-04-16T16:16:41.355739+0000","last_fullsized":"2024-04-16T16:16:41.355739+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:25.467004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056006+0000","last_change":"2024-04-16T16:16:40.052907+0000","last_active":"2024-04-16T16:16:41.056006+0000","last_peered":"2024-04-16T16:16:41.056006+0000","last_clean":"2024-04-16T16:16:41.056006+0000","last_became_active":"2024-04-16T16:16:40.052066+0000","last_became_peered":"2024-04-16T16:16:40.052066+0000","last_unstale":"2024-04-16T16:16:41.056006+0000","last_undegraded":"2024-04-16T16:16:41.056006+0000","last_fullsized":"2024-04-16T16:16:41.056006+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:48:35.347056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"71'4","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057049+0000","last_change":"2024-04-16T16:16:39.052943+0000","last_active":"2024-04-16T16:16:41.057049+0000","last_peered":"2024-04-16T16:16:41.057049+0000","last_clean":"2024-04-16T16:16:41.057049+0000","last_became_active":"2024-04-16T16:16:39.052572+0000","last_became_peered":"2024-04-16T16:16:39.052572+0000","last_unstale":"2024-04-16T16:16:41.057049+0000","last_undegraded":"2024-04-16T16:16:41.057049+0000","last_fullsized":"2024-04-16T16:16:41.057049+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:54:30.889318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054498+0000","last_change":"2024-04-16T16:16:40.063884+0000","last_active":"2024-04-16T16:16:41.054498+0000","last_peered":"2024-04-16T16:16:41.054498+0000","last_clean":"2024-04-16T16:16:41.054498+0000","last_became_active":"2024-04-16T16:16:40.061609+0000","last_became_peered":"2024-04-16T16:16:40.061609+0000","last_unstale":"2024-04-16T16:16:41.054498+0000","last_undegraded":"2024-04-16T16:16:41.054498+0000","last_fullsized":"2024-04-16T16:16:41.054498+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:55.827541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084571+0000","last_change":"2024-04-16T16:16:39.035506+0000","last_active":"2024-04-16T16:16:41.084571+0000","last_peered":"2024-04-16T16:16:41.084571+0000","last_clean":"2024-04-16T16:16:41.084571+0000","last_became_active":"2024-04-16T16:16:39.035196+0000","last_became_peered":"2024-04-16T16:16:39.035196+0000","last_unstale":"2024-04-16T16:16:41.084571+0000","last_undegraded":"2024-04-16T16:16:41.084571+0000","last_fullsized":"2024-04-16T16:16:41.084571+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:29:14.659131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355912+0000","last_change":"2024-04-16T16:16:40.061879+0000","last_active":"2024-04-16T16:16:41.355912+0000","last_peered":"2024-04-16T16:16:41.355912+0000","last_clean":"2024-04-16T16:16:41.355912+0000","last_became_active":"2024-04-16T16:16:40.058652+0000","last_became_peered":"2024-04-16T16:16:40.058652+0000","last_unstale":"2024-04-16T16:16:41.355912+0000","last_undegraded":"2024-04-16T16:16:41.355912+0000","last_fullsized":"2024-04-16T16:16:41.355912+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:12.235559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057778+0000","last_change":"2024-04-16T16:16:39.057527+0000","last_active":"2024-04-16T16:16:41.057778+0000","last_peered":"2024-04-16T16:16:41.057778+0000","last_clean":"2024-04-16T16:16:41.057778+0000","last_became_active":"2024-04-16T16:16:39.056172+0000","last_became_peered":"2024-04-16T16:16:39.056172+0000","last_unstale":"2024-04-16T16:16:41.057778+0000","last_undegraded":"2024-04-16T16:16:41.057778+0000","last_fullsized":"2024-04-16T16:16:41.057778+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:13:45.999397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056353+0000","last_change":"2024-04-16T16:16:40.058328+0000","last_active":"2024-04-16T16:16:41.056353+0000","last_peered":"2024-04-16T16:16:41.056353+0000","last_clean":"2024-04-16T16:16:41.056353+0000","last_became_active":"2024-04-16T16:16:40.058025+0000","last_became_peered":"2024-04-16T16:16:40.058025+0000","last_unstale":"2024-04-16T16:16:41.056353+0000","last_undegraded":"2024-04-16T16:16:41.056353+0000","last_fullsized":"2024-04-16T16:16:41.056353+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:46.618649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355874+0000","last_change":"2024-04-16T16:16:39.042499+0000","last_active":"2024-04-16T16:16:41.355874+0000","last_peered":"2024-04-16T16:16:41.355874+0000","last_clean":"2024-04-16T16:16:41.355874+0000","last_became_active":"2024-04-16T16:16:39.042042+0000","last_became_peered":"2024-04-16T16:16:39.042042+0000","last_unstale":"2024-04-16T16:16:41.355874+0000","last_undegraded":"2024-04-16T16:16:41.355874+0000","last_fullsized":"2024-04-16T16:16:41.355874+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:40:02.778520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079720+0000","last_change":"2024-04-16T16:16:40.061501+0000","last_active":"2024-04-16T16:16:41.079720+0000","last_peered":"2024-04-16T16:16:41.079720+0000","last_clean":"2024-04-16T16:16:41.079720+0000","last_became_active":"2024-04-16T16:16:40.061046+0000","last_became_peered":"2024-04-16T16:16:40.061046+0000","last_unstale":"2024-04-16T16:16:41.079720+0000","last_undegraded":"2024-04-16T16:16:41.079720+0000","last_fullsized":"2024-04-16T16:16:41.079720+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:34:19.296917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353704+0000","last_change":"2024-04-16T16:16:39.051382+0000","last_active":"2024-04-16T16:16:41.353704+0000","last_peered":"2024-04-16T16:16:41.353704+0000","last_clean":"2024-04-16T16:16:41.353704+0000","last_became_active":"2024-04-16T16:16:39.050793+0000","last_became_peered":"2024-04-16T16:16:39.050793+0000","last_unstale":"2024-04-16T16:16:41.353704+0000","last_undegraded":"2024-04-16T16:16:41.353704+0000","last_fullsized":"2024-04-16T16:16:41.353704+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:55:57.308712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057316+0000","last_change":"2024-04-16T16:16:40.049380+0000","last_active":"2024-04-16T16:16:41.057316+0000","last_peered":"2024-04-16T16:16:41.057316+0000","last_clean":"2024-04-16T16:16:41.057316+0000","last_became_active":"2024-04-16T16:16:40.049071+0000","last_became_peered":"2024-04-16T16:16:40.049071+0000","last_unstale":"2024-04-16T16:16:41.057316+0000","last_undegraded":"2024-04-16T16:16:41.057316+0000","last_fullsized":"2024-04-16T16:16:41.057316+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:45.673814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354835+0000","last_change":"2024-04-16T16:16:39.046490+0000","last_active":"2024-04-16T16:16:41.354835+0000","last_peered":"2024-04-16T16:16:41.354835+0000","last_clean":"2024-04-16T16:16:41.354835+0000","last_became_active":"2024-04-16T16:16:39.045734+0000","last_became_peered":"2024-04-16T16:16:39.045734+0000","last_unstale":"2024-04-16T16:16:41.354835+0000","last_undegraded":"2024-04-16T16:16:41.354835+0000","last_fullsized":"2024-04-16T16:16:41.354835+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:37:31.768292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.051911+0000","last_change":"2024-04-16T16:16:40.049943+0000","last_active":"2024-04-16T16:16:41.051911+0000","last_peered":"2024-04-16T16:16:41.051911+0000","last_clean":"2024-04-16T16:16:41.051911+0000","last_became_active":"2024-04-16T16:16:40.049466+0000","last_became_peered":"2024-04-16T16:16:40.049466+0000","last_unstale":"2024-04-16T16:16:41.051911+0000","last_undegraded":"2024-04-16T16:16:41.051911+0000","last_fullsized":"2024-04-16T16:16:41.051911+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:24.082098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353887+0000","last_change":"2024-04-16T16:16:39.048665+0000","last_active":"2024-04-16T16:16:41.353887+0000","last_peered":"2024-04-16T16:16:41.353887+0000","last_clean":"2024-04-16T16:16:41.353887+0000","last_became_active":"2024-04-16T16:16:39.046018+0000","last_became_peered":"2024-04-16T16:16:39.046018+0000","last_unstale":"2024-04-16T16:16:41.353887+0000","last_undegraded":"2024-04-16T16:16:41.353887+0000","last_fullsized":"2024-04-16T16:16:41.353887+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:37:31.768292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084230+0000","last_change":"2024-04-16T16:16:40.068846+0000","last_active":"2024-04-16T16:16:41.084230+0000","last_peered":"2024-04-16T16:16:41.084230+0000","last_clean":"2024-04-16T16:16:41.084230+0000","last_became_active":"2024-04-16T16:16:40.060719+0000","last_became_peered":"2024-04-16T16:16:40.060719+0000","last_unstale":"2024-04-16T16:16:41.084230+0000","last_undegraded":"2024-04-16T16:16:41.084230+0000","last_fullsized":"2024-04-16T16:16:41.084230+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:48:35.347056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.051833+0000","last_change":"2024-04-16T16:16:39.042020+0000","last_active":"2024-04-16T16:16:41.051833+0000","last_peered":"2024-04-16T16:16:41.051833+0000","last_clean":"2024-04-16T16:16:41.051833+0000","last_became_active":"2024-04-16T16:16:39.041447+0000","last_became_peered":"2024-04-16T16:16:39.041447+0000","last_unstale":"2024-04-16T16:16:41.051833+0000","last_undegraded":"2024-04-16T16:16:41.051833+0000","last_fullsized":"2024-04-16T16:16:41.051833+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:18:24.097920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.080180+0000","last_change":"2024-04-16T16:16:40.060588+0000","last_active":"2024-04-16T16:16:41.080180+0000","last_peered":"2024-04-16T16:16:41.080180+0000","last_clean":"2024-04-16T16:16:41.080180+0000","last_became_active":"2024-04-16T16:16:40.060149+0000","last_became_peered":"2024-04-16T16:16:40.060149+0000","last_unstale":"2024-04-16T16:16:41.080180+0000","last_undegraded":"2024-04-16T16:16:41.080180+0000","last_fullsized":"2024-04-16T16:16:41.080180+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:48:35.347056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.058616+0000","last_change":"2024-04-16T16:16:39.052655+0000","last_active":"2024-04-16T16:16:41.058616+0000","last_peered":"2024-04-16T16:16:41.058616+0000","last_clean":"2024-04-16T16:16:41.058616+0000","last_became_active":"2024-04-16T16:16:39.051190+0000","last_became_peered":"2024-04-16T16:16:39.051190+0000","last_unstale":"2024-04-16T16:16:41.058616+0000","last_undegraded":"2024-04-16T16:16:41.058616+0000","last_fullsized":"2024-04-16T16:16:41.058616+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:43.334972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056462+0000","last_change":"2024-04-16T16:16:40.051811+0000","last_active":"2024-04-16T16:16:41.056462+0000","last_peered":"2024-04-16T16:16:41.056462+0000","last_clean":"2024-04-16T16:16:41.056462+0000","last_became_active":"2024-04-16T16:16:40.051476+0000","last_became_peered":"2024-04-16T16:16:40.051476+0000","last_unstale":"2024-04-16T16:16:41.056462+0000","last_undegraded":"2024-04-16T16:16:41.056462+0000","last_fullsized":"2024-04-16T16:16:41.056462+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:29.398550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355529+0000","last_change":"2024-04-16T16:16:39.056709+0000","last_active":"2024-04-16T16:16:41.355529+0000","last_peered":"2024-04-16T16:16:41.355529+0000","last_clean":"2024-04-16T16:16:41.355529+0000","last_became_active":"2024-04-16T16:16:39.056334+0000","last_became_peered":"2024-04-16T16:16:39.056334+0000","last_unstale":"2024-04-16T16:16:41.355529+0000","last_undegraded":"2024-04-16T16:16:41.355529+0000","last_fullsized":"2024-04-16T16:16:41.355529+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:52:38.661658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355675+0000","last_change":"2024-04-16T16:16:40.055050+0000","last_active":"2024-04-16T16:16:41.355675+0000","last_peered":"2024-04-16T16:16:41.355675+0000","last_clean":"2024-04-16T16:16:41.355675+0000","last_became_active":"2024-04-16T16:16:40.054723+0000","last_became_peered":"2024-04-16T16:16:40.054723+0000","last_unstale":"2024-04-16T16:16:41.355675+0000","last_undegraded":"2024-04-16T16:16:41.355675+0000","last_fullsized":"2024-04-16T16:16:41.355675+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:22.397420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"71'3","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353950+0000","last_change":"2024-04-16T16:16:39.046939+0000","last_active":"2024-04-16T16:16:41.353950+0000","last_peered":"2024-04-16T16:16:41.353950+0000","last_clean":"2024-04-16T16:16:41.353950+0000","last_became_active":"2024-04-16T16:16:39.046609+0000","last_became_peered":"2024-04-16T16:16:39.046609+0000","last_unstale":"2024-04-16T16:16:41.353950+0000","last_undegraded":"2024-04-16T16:16:41.353950+0000","last_fullsized":"2024-04-16T16:16:41.353950+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:38:05.588116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053162+0000","last_change":"2024-04-16T16:16:40.056836+0000","last_active":"2024-04-16T16:16:41.053162+0000","last_peered":"2024-04-16T16:16:41.053162+0000","last_clean":"2024-04-16T16:16:41.053162+0000","last_became_active":"2024-04-16T16:16:40.056365+0000","last_became_peered":"2024-04-16T16:16:40.056365+0000","last_unstale":"2024-04-16T16:16:41.053162+0000","last_undegraded":"2024-04-16T16:16:41.053162+0000","last_fullsized":"2024-04-16T16:16:41.053162+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:17:23.179723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"71'6","reported_seq":20,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355016+0000","last_change":"2024-04-16T16:16:39.040106+0000","last_active":"2024-04-16T16:16:41.355016+0000","last_peered":"2024-04-16T16:16:41.355016+0000","last_clean":"2024-04-16T16:16:41.355016+0000","last_became_active":"2024-04-16T16:16:39.039334+0000","last_became_peered":"2024-04-16T16:16:39.039334+0000","last_unstale":"2024-04-16T16:16:41.355016+0000","last_undegraded":"2024-04-16T16:16:41.355016+0000","last_fullsized":"2024-04-16T16:16:41.355016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:29.791246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056901+0000","last_change":"2024-04-16T16:16:40.049290+0000","last_active":"2024-04-16T16:16:41.056901+0000","last_peered":"2024-04-16T16:16:41.056901+0000","last_clean":"2024-04-16T16:16:41.056901+0000","last_became_active":"2024-04-16T16:16:40.048726+0000","last_became_peered":"2024-04-16T16:16:40.048726+0000","last_unstale":"2024-04-16T16:16:41.056901+0000","last_undegraded":"2024-04-16T16:16:41.056901+0000","last_fullsized":"2024-04-16T16:16:41.056901+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:41:00.935635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354451+0000","last_change":"2024-04-16T16:16:39.048257+0000","last_active":"2024-04-16T16:16:41.354451+0000","last_peered":"2024-04-16T16:16:41.354451+0000","last_clean":"2024-04-16T16:16:41.354451+0000","last_became_active":"2024-04-16T16:16:39.047835+0000","last_became_peered":"2024-04-16T16:16:39.047835+0000","last_unstale":"2024-04-16T16:16:41.354451+0000","last_undegraded":"2024-04-16T16:16:41.354451+0000","last_fullsized":"2024-04-16T16:16:41.354451+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:46.723824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079769+0000","last_change":"2024-04-16T16:16:40.055833+0000","last_active":"2024-04-16T16:16:41.079769+0000","last_peered":"2024-04-16T16:16:41.079769+0000","last_clean":"2024-04-16T16:16:41.079769+0000","last_became_active":"2024-04-16T16:16:40.055564+0000","last_became_peered":"2024-04-16T16:16:40.055564+0000","last_unstale":"2024-04-16T16:16:41.079769+0000","last_undegraded":"2024-04-16T16:16:41.079769+0000","last_fullsized":"2024-04-16T16:16:41.079769+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:51:29.994460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355134+0000","last_change":"2024-04-16T16:16:39.046580+0000","last_active":"2024-04-16T16:16:41.355134+0000","last_peered":"2024-04-16T16:16:41.355134+0000","last_clean":"2024-04-16T16:16:41.355134+0000","last_became_active":"2024-04-16T16:16:39.046181+0000","last_became_peered":"2024-04-16T16:16:39.046181+0000","last_unstale":"2024-04-16T16:16:41.355134+0000","last_undegraded":"2024-04-16T16:16:41.355134+0000","last_fullsized":"2024-04-16T16:16:41.355134+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:29.791246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084205+0000","last_change":"2024-04-16T16:16:40.055025+0000","last_active":"2024-04-16T16:16:41.084205+0000","last_peered":"2024-04-16T16:16:41.084205+0000","last_clean":"2024-04-16T16:16:41.084205+0000","last_became_active":"2024-04-16T16:16:40.054757+0000","last_became_peered":"2024-04-16T16:16:40.054757+0000","last_unstale":"2024-04-16T16:16:41.084205+0000","last_undegraded":"2024-04-16T16:16:41.084205+0000","last_fullsized":"2024-04-16T16:16:41.084205+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:31:33.590968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079273+0000","last_change":"2024-04-16T16:16:39.052798+0000","last_active":"2024-04-16T16:16:41.079273+0000","last_peered":"2024-04-16T16:16:41.079273+0000","last_clean":"2024-04-16T16:16:41.079273+0000","last_became_active":"2024-04-16T16:16:39.052440+0000","last_became_peered":"2024-04-16T16:16:39.052440+0000","last_unstale":"2024-04-16T16:16:41.079273+0000","last_undegraded":"2024-04-16T16:16:41.079273+0000","last_fullsized":"2024-04-16T16:16:41.079273+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:44.657799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052671+0000","last_change":"2024-04-16T16:16:40.054326+0000","last_active":"2024-04-16T16:16:41.052671+0000","last_peered":"2024-04-16T16:16:41.052671+0000","last_clean":"2024-04-16T16:16:41.052671+0000","last_became_active":"2024-04-16T16:16:40.054013+0000","last_became_peered":"2024-04-16T16:16:40.054013+0000","last_unstale":"2024-04-16T16:16:41.052671+0000","last_undegraded":"2024-04-16T16:16:41.052671+0000","last_fullsized":"2024-04-16T16:16:41.052671+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:05:08.507551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"71'8","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057407+0000","last_change":"2024-04-16T16:16:39.053695+0000","last_active":"2024-04-16T16:16:41.057407+0000","last_peered":"2024-04-16T16:16:41.057407+0000","last_clean":"2024-04-16T16:16:41.057407+0000","last_became_active":"2024-04-16T16:16:39.053264+0000","last_became_peered":"2024-04-16T16:16:39.053264+0000","last_unstale":"2024-04-16T16:16:41.057407+0000","last_undegraded":"2024-04-16T16:16:41.057407+0000","last_fullsized":"2024-04-16T16:16:41.057407+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:39:32.200451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.058313+0000","last_change":"2024-04-16T16:16:40.056673+0000","last_active":"2024-04-16T16:16:41.058313+0000","last_peered":"2024-04-16T16:16:41.058313+0000","last_clean":"2024-04-16T16:16:41.058313+0000","last_became_active":"2024-04-16T16:16:40.056357+0000","last_became_peered":"2024-04-16T16:16:40.056357+0000","last_unstale":"2024-04-16T16:16:41.058313+0000","last_undegraded":"2024-04-16T16:16:41.058313+0000","last_fullsized":"2024-04-16T16:16:41.058313+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:12.235559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355020+0000","last_change":"2024-04-16T16:16:39.046833+0000","last_active":"2024-04-16T16:16:41.355020+0000","last_peered":"2024-04-16T16:16:41.355020+0000","last_clean":"2024-04-16T16:16:41.355020+0000","last_became_active":"2024-04-16T16:16:39.046543+0000","last_became_peered":"2024-04-16T16:16:39.046543+0000","last_unstale":"2024-04-16T16:16:41.355020+0000","last_undegraded":"2024-04-16T16:16:41.355020+0000","last_fullsized":"2024-04-16T16:16:41.355020+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:39:32.200451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056906+0000","last_change":"2024-04-16T16:16:40.059425+0000","last_active":"2024-04-16T16:16:41.056906+0000","last_peered":"2024-04-16T16:16:41.056906+0000","last_clean":"2024-04-16T16:16:41.056906+0000","last_became_active":"2024-04-16T16:16:40.059078+0000","last_became_peered":"2024-04-16T16:16:40.059078+0000","last_unstale":"2024-04-16T16:16:41.056906+0000","last_undegraded":"2024-04-16T16:16:41.056906+0000","last_fullsized":"2024-04-16T16:16:41.056906+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:50:55.122092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.058306+0000","last_change":"2024-04-16T16:16:39.057643+0000","last_active":"2024-04-16T16:16:41.058306+0000","last_peered":"2024-04-16T16:16:41.058306+0000","last_clean":"2024-04-16T16:16:41.058306+0000","last_became_active":"2024-04-16T16:16:39.057160+0000","last_became_peered":"2024-04-16T16:16:39.057160+0000","last_unstale":"2024-04-16T16:16:41.058306+0000","last_undegraded":"2024-04-16T16:16:41.058306+0000","last_fullsized":"2024-04-16T16:16:41.058306+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:59:03.269004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354958+0000","last_change":"2024-04-16T16:16:40.066424+0000","last_active":"2024-04-16T16:16:41.354958+0000","last_peered":"2024-04-16T16:16:41.354958+0000","last_clean":"2024-04-16T16:16:41.354958+0000","last_became_active":"2024-04-16T16:16:40.063948+0000","last_became_peered":"2024-04-16T16:16:40.063948+0000","last_unstale":"2024-04-16T16:16:41.354958+0000","last_undegraded":"2024-04-16T16:16:41.354958+0000","last_fullsized":"2024-04-16T16:16:41.354958+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:59:04.285019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355470+0000","last_change":"2024-04-16T16:16:39.054911+0000","last_active":"2024-04-16T16:16:41.355470+0000","last_peered":"2024-04-16T16:16:41.355470+0000","last_clean":"2024-04-16T16:16:41.355470+0000","last_became_active":"2024-04-16T16:16:39.053957+0000","last_became_peered":"2024-04-16T16:16:39.053957+0000","last_unstale":"2024-04-16T16:16:41.355470+0000","last_undegraded":"2024-04-16T16:16:41.355470+0000","last_fullsized":"2024-04-16T16:16:41.355470+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:46.723824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053204+0000","last_change":"2024-04-16T16:16:40.052810+0000","last_active":"2024-04-16T16:16:41.053204+0000","last_peered":"2024-04-16T16:16:41.053204+0000","last_clean":"2024-04-16T16:16:41.053204+0000","last_became_active":"2024-04-16T16:16:40.051522+0000","last_became_peered":"2024-04-16T16:16:40.051522+0000","last_unstale":"2024-04-16T16:16:41.053204+0000","last_undegraded":"2024-04-16T16:16:41.053204+0000","last_fullsized":"2024-04-16T16:16:41.053204+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:53:25.229933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056633+0000","last_change":"2024-04-16T16:16:39.054182+0000","last_active":"2024-04-16T16:16:41.056633+0000","last_peered":"2024-04-16T16:16:41.056633+0000","last_clean":"2024-04-16T16:16:41.056633+0000","last_became_active":"2024-04-16T16:16:39.053902+0000","last_became_peered":"2024-04-16T16:16:39.053902+0000","last_unstale":"2024-04-16T16:16:41.056633+0000","last_undegraded":"2024-04-16T16:16:41.056633+0000","last_fullsized":"2024-04-16T16:16:41.056633+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:11.219544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353546+0000","last_change":"2024-04-16T16:16:40.062784+0000","last_active":"2024-04-16T16:16:41.353546+0000","last_peered":"2024-04-16T16:16:41.353546+0000","last_clean":"2024-04-16T16:16:41.353546+0000","last_became_active":"2024-04-16T16:16:40.062465+0000","last_became_peered":"2024-04-16T16:16:40.062465+0000","last_unstale":"2024-04-16T16:16:41.353546+0000","last_undegraded":"2024-04-16T16:16:41.353546+0000","last_fullsized":"2024-04-16T16:16:41.353546+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:29:01.496515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355429+0000","last_change":"2024-04-16T16:16:39.045311+0000","last_active":"2024-04-16T16:16:41.355429+0000","last_peered":"2024-04-16T16:16:41.355429+0000","last_clean":"2024-04-16T16:16:41.355429+0000","last_became_active":"2024-04-16T16:16:39.044896+0000","last_became_peered":"2024-04-16T16:16:39.044896+0000","last_unstale":"2024-04-16T16:16:41.355429+0000","last_undegraded":"2024-04-16T16:16:41.355429+0000","last_fullsized":"2024-04-16T16:16:41.355429+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:58:16.807985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354225+0000","last_change":"2024-04-16T16:16:40.054819+0000","last_active":"2024-04-16T16:16:41.354225+0000","last_peered":"2024-04-16T16:16:41.354225+0000","last_clean":"2024-04-16T16:16:41.354225+0000","last_became_active":"2024-04-16T16:16:40.054521+0000","last_became_peered":"2024-04-16T16:16:40.054521+0000","last_unstale":"2024-04-16T16:16:41.354225+0000","last_undegraded":"2024-04-16T16:16:41.354225+0000","last_fullsized":"2024-04-16T16:16:41.354225+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:01:17.068261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.080016+0000","last_change":"2024-04-16T16:16:39.055458+0000","last_active":"2024-04-16T16:16:41.080016+0000","last_peered":"2024-04-16T16:16:41.080016+0000","last_clean":"2024-04-16T16:16:41.080016+0000","last_became_active":"2024-04-16T16:16:39.053012+0000","last_became_peered":"2024-04-16T16:16:39.053012+0000","last_unstale":"2024-04-16T16:16:41.080016+0000","last_undegraded":"2024-04-16T16:16:41.080016+0000","last_fullsized":"2024-04-16T16:16:41.080016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:21.381404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353660+0000","last_change":"2024-04-16T16:16:40.053963+0000","last_active":"2024-04-16T16:16:41.353660+0000","last_peered":"2024-04-16T16:16:41.353660+0000","last_clean":"2024-04-16T16:16:41.353660+0000","last_became_active":"2024-04-16T16:16:40.053682+0000","last_became_peered":"2024-04-16T16:16:40.053682+0000","last_unstale":"2024-04-16T16:16:41.353660+0000","last_undegraded":"2024-04-16T16:16:41.353660+0000","last_fullsized":"2024-04-16T16:16:41.353660+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:16.693760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.083960+0000","last_change":"2024-04-16T16:16:39.038427+0000","last_active":"2024-04-16T16:16:41.083960+0000","last_peered":"2024-04-16T16:16:41.083960+0000","last_clean":"2024-04-16T16:16:41.083960+0000","last_became_active":"2024-04-16T16:16:39.036287+0000","last_became_peered":"2024-04-16T16:16:39.036287+0000","last_unstale":"2024-04-16T16:16:41.083960+0000","last_undegraded":"2024-04-16T16:16:41.083960+0000","last_fullsized":"2024-04-16T16:16:41.083960+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:38.804660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084649+0000","last_change":"2024-04-16T16:16:40.050769+0000","last_active":"2024-04-16T16:16:41.084649+0000","last_peered":"2024-04-16T16:16:41.084649+0000","last_clean":"2024-04-16T16:16:41.084649+0000","last_became_active":"2024-04-16T16:16:40.050449+0000","last_became_peered":"2024-04-16T16:16:40.050449+0000","last_unstale":"2024-04-16T16:16:41.084649+0000","last_undegraded":"2024-04-16T16:16:41.084649+0000","last_fullsized":"2024-04-16T16:16:41.084649+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:29.398550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355074+0000","last_change":"2024-04-16T16:16:39.047443+0000","last_active":"2024-04-16T16:16:41.355074+0000","last_peered":"2024-04-16T16:16:41.355074+0000","last_clean":"2024-04-16T16:16:41.355074+0000","last_became_active":"2024-04-16T16:16:39.046833+0000","last_became_peered":"2024-04-16T16:16:39.046833+0000","last_unstale":"2024-04-16T16:16:41.355074+0000","last_undegraded":"2024-04-16T16:16:41.355074+0000","last_fullsized":"2024-04-16T16:16:41.355074+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:54:40.017503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.080259+0000","last_change":"2024-04-16T16:16:40.062706+0000","last_active":"2024-04-16T16:16:41.080259+0000","last_peered":"2024-04-16T16:16:41.080259+0000","last_clean":"2024-04-16T16:16:41.080259+0000","last_became_active":"2024-04-16T16:16:40.060527+0000","last_became_peered":"2024-04-16T16:16:40.060527+0000","last_unstale":"2024-04-16T16:16:41.080259+0000","last_undegraded":"2024-04-16T16:16:41.080259+0000","last_fullsized":"2024-04-16T16:16:41.080259+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:28:41.607987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"72'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.080437+0000","last_change":"2024-04-16T16:16:39.055132+0000","last_active":"2024-04-16T16:16:41.080437+0000","last_peered":"2024-04-16T16:16:41.080437+0000","last_clean":"2024-04-16T16:16:41.080437+0000","last_became_active":"2024-04-16T16:16:39.054859+0000","last_became_peered":"2024-04-16T16:16:39.054859+0000","last_unstale":"2024-04-16T16:16:41.080437+0000","last_undegraded":"2024-04-16T16:16:41.080437+0000","last_fullsized":"2024-04-16T16:16:41.080437+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:24.450988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056499+0000","last_change":"2024-04-16T16:16:40.056526+0000","last_active":"2024-04-16T16:16:41.056499+0000","last_peered":"2024-04-16T16:16:41.056499+0000","last_clean":"2024-04-16T16:16:41.056499+0000","last_became_active":"2024-04-16T16:16:40.055969+0000","last_became_peered":"2024-04-16T16:16:40.055969+0000","last_unstale":"2024-04-16T16:16:41.056499+0000","last_undegraded":"2024-04-16T16:16:41.056499+0000","last_fullsized":"2024-04-16T16:16:41.056499+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:59:46.671848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053323+0000","last_change":"2024-04-16T16:16:39.044030+0000","last_active":"2024-04-16T16:16:41.053323+0000","last_peered":"2024-04-16T16:16:41.053323+0000","last_clean":"2024-04-16T16:16:41.053323+0000","last_became_active":"2024-04-16T16:16:39.043710+0000","last_became_peered":"2024-04-16T16:16:39.043710+0000","last_unstale":"2024-04-16T16:16:41.053323+0000","last_undegraded":"2024-04-16T16:16:41.053323+0000","last_fullsized":"2024-04-16T16:16:41.053323+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:15.221340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056763+0000","last_change":"2024-04-16T16:16:40.056711+0000","last_active":"2024-04-16T16:16:41.056763+0000","last_peered":"2024-04-16T16:16:41.056763+0000","last_clean":"2024-04-16T16:16:41.056763+0000","last_became_active":"2024-04-16T16:16:40.053649+0000","last_became_peered":"2024-04-16T16:16:40.053649+0000","last_unstale":"2024-04-16T16:16:41.056763+0000","last_undegraded":"2024-04-16T16:16:41.056763+0000","last_fullsized":"2024-04-16T16:16:41.056763+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:45:10.434171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084392+0000","last_change":"2024-04-16T16:16:39.048354+0000","last_active":"2024-04-16T16:16:41.084392+0000","last_peered":"2024-04-16T16:16:41.084392+0000","last_clean":"2024-04-16T16:16:41.084392+0000","last_became_active":"2024-04-16T16:16:39.048098+0000","last_became_peered":"2024-04-16T16:16:39.048098+0000","last_unstale":"2024-04-16T16:16:41.084392+0000","last_undegraded":"2024-04-16T16:16:41.084392+0000","last_fullsized":"2024-04-16T16:16:41.084392+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:37:41.504380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052443+0000","last_change":"2024-04-16T16:16:40.061326+0000","last_active":"2024-04-16T16:16:41.052443+0000","last_peered":"2024-04-16T16:16:41.052443+0000","last_clean":"2024-04-16T16:16:41.052443+0000","last_became_active":"2024-04-16T16:16:40.056522+0000","last_became_peered":"2024-04-16T16:16:40.056522+0000","last_unstale":"2024-04-16T16:16:41.052443+0000","last_undegraded":"2024-04-16T16:16:41.052443+0000","last_fullsized":"2024-04-16T16:16:41.052443+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:43:46.183862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052332+0000","last_change":"2024-04-16T16:16:39.051171+0000","last_active":"2024-04-16T16:16:41.052332+0000","last_peered":"2024-04-16T16:16:41.052332+0000","last_clean":"2024-04-16T16:16:41.052332+0000","last_became_active":"2024-04-16T16:16:39.050841+0000","last_became_peered":"2024-04-16T16:16:39.050841+0000","last_unstale":"2024-04-16T16:16:41.052332+0000","last_undegraded":"2024-04-16T16:16:41.052332+0000","last_fullsized":"2024-04-16T16:16:41.052332+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:32:24.457101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354741+0000","last_change":"2024-04-16T16:16:40.062944+0000","last_active":"2024-04-16T16:16:41.354741+0000","last_peered":"2024-04-16T16:16:41.354741+0000","last_clean":"2024-04-16T16:16:41.354741+0000","last_became_active":"2024-04-16T16:16:40.057251+0000","last_became_peered":"2024-04-16T16:16:40.057251+0000","last_unstale":"2024-04-16T16:16:41.354741+0000","last_undegraded":"2024-04-16T16:16:41.354741+0000","last_fullsized":"2024-04-16T16:16:41.354741+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:12:52.888262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354394+0000","last_change":"2024-04-16T16:16:39.046999+0000","last_active":"2024-04-16T16:16:41.354394+0000","last_peered":"2024-04-16T16:16:41.354394+0000","last_clean":"2024-04-16T16:16:41.354394+0000","last_became_active":"2024-04-16T16:16:39.046741+0000","last_became_peered":"2024-04-16T16:16:39.046741+0000","last_unstale":"2024-04-16T16:16:41.354394+0000","last_undegraded":"2024-04-16T16:16:41.354394+0000","last_fullsized":"2024-04-16T16:16:41.354394+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:58:16.807985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354126+0000","last_change":"2024-04-16T16:16:40.057460+0000","last_active":"2024-04-16T16:16:41.354126+0000","last_peered":"2024-04-16T16:16:41.354126+0000","last_clean":"2024-04-16T16:16:41.354126+0000","last_became_active":"2024-04-16T16:16:40.057142+0000","last_became_peered":"2024-04-16T16:16:40.057142+0000","last_unstale":"2024-04-16T16:16:41.354126+0000","last_undegraded":"2024-04-16T16:16:41.354126+0000","last_fullsized":"2024-04-16T16:16:41.354126+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:44.350988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.055586+0000","last_change":"2024-04-16T16:16:39.053727+0000","last_active":"2024-04-16T16:16:41.055586+0000","last_peered":"2024-04-16T16:16:41.055586+0000","last_clean":"2024-04-16T16:16:41.055586+0000","last_became_active":"2024-04-16T16:16:39.053180+0000","last_became_peered":"2024-04-16T16:16:39.053180+0000","last_unstale":"2024-04-16T16:16:41.055586+0000","last_undegraded":"2024-04-16T16:16:41.055586+0000","last_fullsized":"2024-04-16T16:16:41.055586+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:54:40.017503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354595+0000","last_change":"2024-04-16T16:16:40.062043+0000","last_active":"2024-04-16T16:16:41.354595+0000","last_peered":"2024-04-16T16:16:41.354595+0000","last_clean":"2024-04-16T16:16:41.354595+0000","last_became_active":"2024-04-16T16:16:40.061740+0000","last_became_peered":"2024-04-16T16:16:40.061740+0000","last_unstale":"2024-04-16T16:16:41.354595+0000","last_undegraded":"2024-04-16T16:16:41.354595+0000","last_fullsized":"2024-04-16T16:16:41.354595+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:54:32.532363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355921+0000","last_change":"2024-04-16T16:16:39.055076+0000","last_active":"2024-04-16T16:16:41.355921+0000","last_peered":"2024-04-16T16:16:41.355921+0000","last_clean":"2024-04-16T16:16:41.355921+0000","last_became_active":"2024-04-16T16:16:39.054757+0000","last_became_peered":"2024-04-16T16:16:39.054757+0000","last_unstale":"2024-04-16T16:16:41.355921+0000","last_undegraded":"2024-04-16T16:16:41.355921+0000","last_fullsized":"2024-04-16T16:16:41.355921+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:54:24.777181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353278+0000","last_change":"2024-04-16T16:16:40.056758+0000","last_active":"2024-04-16T16:16:41.353278+0000","last_peered":"2024-04-16T16:16:41.353278+0000","last_clean":"2024-04-16T16:16:41.353278+0000","last_became_active":"2024-04-16T16:16:40.056150+0000","last_became_peered":"2024-04-16T16:16:40.056150+0000","last_unstale":"2024-04-16T16:16:41.353278+0000","last_undegraded":"2024-04-16T16:16:41.353278+0000","last_fullsized":"2024-04-16T16:16:41.353278+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:24.062608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354925+0000","last_change":"2024-04-16T16:16:39.048647+0000","last_active":"2024-04-16T16:16:41.354925+0000","last_peered":"2024-04-16T16:16:41.354925+0000","last_clean":"2024-04-16T16:16:41.354925+0000","last_became_active":"2024-04-16T16:16:39.048341+0000","last_became_peered":"2024-04-16T16:16:39.048341+0000","last_unstale":"2024-04-16T16:16:41.354925+0000","last_undegraded":"2024-04-16T16:16:41.354925+0000","last_fullsized":"2024-04-16T16:16:41.354925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:23:45.725727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"57'139","reported_seq":351,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054822+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:16:41.054822+0000","last_peered":"2024-04-16T16:16:41.054822+0000","last_clean":"2024-04-16T16:16:41.054822+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:16:41.054822+0000","last_undegraded":"2024-04-16T16:16:41.054822+0000","last_fullsized":"2024-04-16T16:16:41.054822+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32628,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339212,"statfs":{"total":96636764160,"available":96603353088,"internally_reserved":0,"allocated":1478656,"data_stored":1001168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":884736,"data_stored":410276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":410274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":880640,"data_stored":410254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":884736,"data_stored":410296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32048,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339792,"statfs":{"total":96636764160,"available":96603947008,"internally_reserved":0,"allocated":888832,"data_stored":411796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":897024,"data_stored":412412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1470464,"data_stored":1000552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:16:49.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:16:49.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.964+0000 7fca1b000640 1 Processor -- start 2024-04-16T16:16:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.965+0000 7fca1b000640 1 -- start start 2024-04-16T16:16:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.966+0000 7fca1b000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca14161e90 0x7fca140cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.966+0000 7fca1b000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca140d3080 con 0x7fca14162270 2024-04-16T16:16:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.966+0000 7fca1b000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca140d3360 con 0x7fca140d52d0 2024-04-16T16:16:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.966+0000 7fca1b000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fca140d2e30 con 0x7fca14161e90 2024-04-16T16:16:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.966+0000 7fca19200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fca14162270 0x7fca140d3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35754/0 (socket says 172.21.15.45:35754) 2024-04-16T16:16:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.966+0000 7fca19200640 1 -- 172.21.15.45:0/1263454199 learned_addr learned my addr 172.21.15.45:0/1263454199 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca19c00640 1 --2- 172.21.15.45:0/1263454199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca14161e90 0x7fca140cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca13e00640 1 -- 172.21.15.45:0/1263454199 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1369279207 0 0) 0x7fca140d3080 con 0x7fca14162270 2024-04-16T16:16:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca13e00640 1 -- 172.21.15.45:0/1263454199 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9fc003520 con 0x7fca14162270 2024-04-16T16:16:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca13e00640 1 -- 172.21.15.45:0/1263454199 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 346660958 0 0) 0x7fc9fc003520 con 0x7fca14162270 2024-04-16T16:16:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca13e00640 1 -- 172.21.15.45:0/1263454199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca14161e90 msgr2=0x7fca140cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca13e00640 1 --2- 172.21.15.45:0/1263454199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca14161e90 0x7fca140cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca13e00640 1 -- 172.21.15.45:0/1263454199 >> v1:172.21.15.103:6789/0 conn(0x7fca140d52d0 legacy=0x7fca1406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca13e00640 1 -- 172.21.15.45:0/1263454199 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca140d3640 con 0x7fca14162270 2024-04-16T16:16:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca13e00640 1 -- 172.21.15.45:0/1263454199 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fca08001670 con 0x7fca14162270 2024-04-16T16:16:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca13e00640 1 -- 172.21.15.45:0/1263454199 >> v1:172.21.15.45:6789/0 conn(0x7fca14162270 legacy=0x7fca140d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca13e00640 1 --2- 172.21.15.45:0/1263454199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca14161e90 0x7fc9fc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca13e00640 1 --2- 172.21.15.45:0/1263454199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9fc003f60 0x7fc9fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca13e00640 1 --2- 172.21.15.45:0/1263454199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9fc004d40 0x7fc9fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca13e00640 1 -- 172.21.15.45:0/1263454199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fca140d2e30 con 0x7fc9fc004d40 2024-04-16T16:16:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca13e00640 1 -- 172.21.15.45:0/1263454199 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc9fc00bb30 con 0x7fc9fc003f60 2024-04-16T16:16:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.967+0000 7fca13e00640 1 -- 172.21.15.45:0/1263454199 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc9fc00bd30 con 0x7fca14161e90 2024-04-16T16:16:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.968+0000 7fca1a600640 1 --2- 172.21.15.45:0/1263454199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9fc004d40 0x7fc9fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:49.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.968+0000 7fca19200640 1 --2- 172.21.15.45:0/1263454199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9fc003f60 0x7fc9fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:49.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.968+0000 7fca19c00640 1 --2- 172.21.15.45:0/1263454199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca14161e90 0x7fc9fc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:49.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.968+0000 7fca1a600640 1 -- 172.21.15.45:0/1263454199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca14161e90 msgr2=0x7fc9fc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.968+0000 7fca1a600640 1 --2- 172.21.15.45:0/1263454199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca14161e90 0x7fc9fc0039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.968+0000 7fca1a600640 1 -- 172.21.15.45:0/1263454199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9fc003f60 msgr2=0x7fc9fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.968+0000 7fca1a600640 1 --2- 172.21.15.45:0/1263454199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9fc003f60 0x7fc9fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.968+0000 7fca1a600640 1 -- 172.21.15.45:0/1263454199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9fc003520 con 0x7fc9fc004d40 2024-04-16T16:16:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.969+0000 7fca1a600640 1 --2- 172.21.15.45:0/1263454199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9fc004d40 0x7fc9fc0051c0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fca0c001e50 tx=0x7fc9fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12bec10607ea329d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.970+0000 7fca13e00640 1 -- 172.21.15.45:0/1263454199 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c001a70 con 0x7fc9fc004d40 2024-04-16T16:16:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.970+0000 7fca13e00640 1 -- 172.21.15.45:0/1263454199 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca0c003b70 con 0x7fc9fc004d40 2024-04-16T16:16:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.970+0000 7fca13e00640 1 -- 172.21.15.45:0/1263454199 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0050d0 con 0x7fc9fc004d40 2024-04-16T16:16:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.970+0000 7fca1b000640 1 -- 172.21.15.45:0/1263454199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9fc004d40 msgr2=0x7fc9fc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.970+0000 7fca1b000640 1 --2- 172.21.15.45:0/1263454199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9fc004d40 0x7fc9fc0051c0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fca0c001e50 tx=0x7fc9fc005830 comp rx=0 tx=0).stop 2024-04-16T16:16:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.970+0000 7fca1a600640 1 -- 172.21.15.45:0/1263454199 reap_dead start 2024-04-16T16:16:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.970+0000 7fca1b000640 1 -- 172.21.15.45:0/1263454199 shutdown_connections 2024-04-16T16:16:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.970+0000 7fca1b000640 1 -- 172.21.15.45:0/1263454199 >> 172.21.15.45:0/1263454199 conn(0x7fca140d1940 msgr2=0x7fca140d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.970+0000 7fca1b000640 1 -- 172.21.15.45:0/1263454199 shutdown_connections 2024-04-16T16:16:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.971+0000 7fca1b000640 1 -- 172.21.15.45:0/1263454199 wait complete. 2024-04-16T16:16:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.972+0000 7fca1b000640 1 Processor -- start 2024-04-16T16:16:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.972+0000 7fca1b000640 1 -- start start 2024-04-16T16:16:49.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.973+0000 7fca1b000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca140b5370 0x7fca141e9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:49.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.973+0000 7fca1b000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca140b5750 0x7fca1414f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:49.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.973+0000 7fca1b000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca140cade0 0x7fca1414fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:49.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.973+0000 7fca1b000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fca140d46d0 con 0x7fca140b5370 2024-04-16T16:16:49.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.973+0000 7fca1b000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fca140d4550 con 0x7fca140cade0 2024-04-16T16:16:49.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.973+0000 7fca1b000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fca1406d550 con 0x7fca140b5750 2024-04-16T16:16:49.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.974+0000 7fca19c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca140b5370 0x7fca141e9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:49.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.974+0000 7fca19c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca140b5370 0x7fca141e9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46558/0 (socket says 172.21.15.45:46558) 2024-04-16T16:16:49.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.974+0000 7fca19c00640 1 -- 172.21.15.45:0/1212900741 learned_addr learned my addr 172.21.15.45:0/1212900741 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:49.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.974+0000 7fca1a600640 1 --2- 172.21.15.45:0/1212900741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca140cade0 0x7fca1414fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:49.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.974+0000 7fca19200640 1 --2- 172.21.15.45:0/1212900741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca140b5750 0x7fca1414f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:49.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.974+0000 7fca19c00640 1 -- 172.21.15.45:0/1212900741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca140b5750 msgr2=0x7fca1414f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:49.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.974+0000 7fca19c00640 1 --2- 172.21.15.45:0/1212900741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca140b5750 0x7fca1414f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:49.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.974+0000 7fca19c00640 1 -- 172.21.15.45:0/1212900741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca140cade0 msgr2=0x7fca1414fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:49.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.974+0000 7fca19c00640 1 --2- 172.21.15.45:0/1212900741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca140cade0 0x7fca1414fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:49.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.974+0000 7fca19c00640 1 -- 172.21.15.45:0/1212900741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca141e8c40 con 0x7fca140b5370 2024-04-16T16:16:49.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.975+0000 7fca19c00640 1 --2- 172.21.15.45:0/1212900741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca140b5370 0x7fca141e9ce0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fca04010860 tx=0x7fca04010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:49.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.975+0000 7fca11600640 1 -- 172.21.15.45:0/1212900741 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0400f040 con 0x7fca140b5370 2024-04-16T16:16:49.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.975+0000 7fca11600640 1 -- 172.21.15.45:0/1212900741 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca0400eb60 con 0x7fca140b5370 2024-04-16T16:16:49.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.975+0000 7fca1b000640 1 -- 172.21.15.45:0/1212900741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca141e8f20 con 0x7fca140b5370 2024-04-16T16:16:49.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.975+0000 7fca11600640 1 -- 172.21.15.45:0/1212900741 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0401c400 con 0x7fca140b5370 2024-04-16T16:16:49.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.975+0000 7fca1b000640 1 -- 172.21.15.45:0/1212900741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca141e94e0 con 0x7fca140b5370 2024-04-16T16:16:49.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.976+0000 7fca1b000640 1 -- 172.21.15.45:0/1212900741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca14075780 con 0x7fca140b5370 2024-04-16T16:16:49.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.977+0000 7fca11600640 1 -- 172.21.15.45:0/1212900741 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fca0401c5a0 con 0x7fca140b5370 2024-04-16T16:16:49.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.978+0000 7fca11600640 1 --2- 172.21.15.45:0/1212900741 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc9f806cb30 0x7fc9f806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:49.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.979+0000 7fca11600640 1 -- 172.21.15.45:0/1212900741 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7fca0408c660 con 0x7fca140b5370 2024-04-16T16:16:49.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.980+0000 7fca19200640 1 --2- 172.21.15.45:0/1212900741 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc9f806cb30 0x7fc9f806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:49.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.981+0000 7fca19200640 1 --2- 172.21.15.45:0/1212900741 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc9f806cb30 0x7fc9f806eff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fca08001cd0 tx=0x7fca08009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:49.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:49.983+0000 7fca11600640 1 -- 172.21.15.45:0/1212900741 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fca04018030 con 0x7fca140b5370 2024-04-16T16:16:50.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.185+0000 7fca1b000640 1 -- 172.21.15.45:0/1212900741 --> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fca1406dae0 con 0x7fc9f806cb30 2024-04-16T16:16:50.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.205+0000 7fca11600640 1 -- 172.21.15.45:0/1212900741 <== mgr.4961 v2:172.21.15.103:6832/4155603915 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251958 (secure 0 0 0) 0x7fca1406dae0 con 0x7fc9f806cb30 2024-04-16T16:16:50.209 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:50.210 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:16:50.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.211+0000 7fca1b000640 1 -- 172.21.15.45:0/1212900741 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc9f806cb30 msgr2=0x7fc9f806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:50.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.211+0000 7fca1b000640 1 --2- 172.21.15.45:0/1212900741 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc9f806cb30 0x7fc9f806eff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fca08001cd0 tx=0x7fca08009040 comp rx=0 tx=0).stop 2024-04-16T16:16:50.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.211+0000 7fca1b000640 1 -- 172.21.15.45:0/1212900741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca140b5370 msgr2=0x7fca141e9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:50.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.211+0000 7fca1b000640 1 --2- 172.21.15.45:0/1212900741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca140b5370 0x7fca141e9ce0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fca04010860 tx=0x7fca04010d20 comp rx=0 tx=0).stop 2024-04-16T16:16:50.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.211+0000 7fca1b000640 1 -- 172.21.15.45:0/1212900741 shutdown_connections 2024-04-16T16:16:50.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.211+0000 7fca1b000640 1 --2- 172.21.15.45:0/1212900741 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc9f806cb30 0x7fc9f806eff0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:50.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.211+0000 7fca1b000640 1 --2- 172.21.15.45:0/1212900741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca140cade0 0x7fca1414fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:50.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.211+0000 7fca1b000640 1 --2- 172.21.15.45:0/1212900741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca140b5750 0x7fca1414f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:50.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.211+0000 7fca1b000640 1 --2- 172.21.15.45:0/1212900741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca140b5370 0x7fca141e9ce0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:50.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.212+0000 7fca1b000640 1 -- 172.21.15.45:0/1212900741 >> 172.21.15.45:0/1212900741 conn(0x7fca140d1940 msgr2=0x7fca140b9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:50.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.212+0000 7fca1b000640 1 -- 172.21.15.45:0/1212900741 shutdown_connections 2024-04-16T16:16:50.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.212+0000 7fca1b000640 1 -- 172.21.15.45:0/1212900741 wait complete. 2024-04-16T16:16:50.231 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-16T16:16:49.083860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":244,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":178,"ondisk_log_size":178,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":253360,"kb_used_data":8072,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754721360,"statfs":{"total":773094113280,"available":772834672640,"internally_reserved":0,"allocated":8265728,"data_stored":4467028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.060353"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.059101+0000","last_change":"2024-04-16T16:16:40.048700+0000","last_active":"2024-04-16T16:16:41.059101+0000","last_peered":"2024-04-16T16:16:41.059101+0000","last_clean":"2024-04-16T16:16:41.059101+0000","last_became_active":"2024-04-16T16:16:40.046882+0000","last_became_peered":"2024-04-16T16:16:40.046882+0000","last_unstale":"2024-04-16T16:16:41.059101+0000","last_undegraded":"2024-04-16T16:16:41.059101+0000","last_fullsized":"2024-04-16T16:16:41.059101+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:41:52.714263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355645+0000","last_change":"2024-04-16T16:16:40.063532+0000","last_active":"2024-04-16T16:16:41.355645+0000","last_peered":"2024-04-16T16:16:41.355645+0000","last_clean":"2024-04-16T16:16:41.355645+0000","last_became_active":"2024-04-16T16:16:40.063023+0000","last_became_peered":"2024-04-16T16:16:40.063023+0000","last_unstale":"2024-04-16T16:16:41.355645+0000","last_undegraded":"2024-04-16T16:16:41.355645+0000","last_fullsized":"2024-04-16T16:16:41.355645+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:45.673814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353809+0000","last_change":"2024-04-16T16:16:40.058970+0000","last_active":"2024-04-16T16:16:41.353809+0000","last_peered":"2024-04-16T16:16:41.353809+0000","last_clean":"2024-04-16T16:16:41.353809+0000","last_became_active":"2024-04-16T16:16:40.058582+0000","last_became_peered":"2024-04-16T16:16:40.058582+0000","last_unstale":"2024-04-16T16:16:41.353809+0000","last_undegraded":"2024-04-16T16:16:41.353809+0000","last_fullsized":"2024-04-16T16:16:41.353809+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:04.907779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354998+0000","last_change":"2024-04-16T16:16:40.067374+0000","last_active":"2024-04-16T16:16:41.354998+0000","last_peered":"2024-04-16T16:16:41.354998+0000","last_clean":"2024-04-16T16:16:41.354998+0000","last_became_active":"2024-04-16T16:16:40.064314+0000","last_became_peered":"2024-04-16T16:16:40.064314+0000","last_unstale":"2024-04-16T16:16:41.354998+0000","last_undegraded":"2024-04-16T16:16:41.354998+0000","last_fullsized":"2024-04-16T16:16:41.354998+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:47:33.049888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052264+0000","last_change":"2024-04-16T16:16:40.053258+0000","last_active":"2024-04-16T16:16:41.052264+0000","last_peered":"2024-04-16T16:16:41.052264+0000","last_clean":"2024-04-16T16:16:41.052264+0000","last_became_active":"2024-04-16T16:16:40.051051+0000","last_became_peered":"2024-04-16T16:16:40.051051+0000","last_unstale":"2024-04-16T16:16:41.052264+0000","last_undegraded":"2024-04-16T16:16:41.052264+0000","last_fullsized":"2024-04-16T16:16:41.052264+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:16:10.608170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355834+0000","last_change":"2024-04-16T16:16:40.064127+0000","last_active":"2024-04-16T16:16:41.355834+0000","last_peered":"2024-04-16T16:16:41.355834+0000","last_clean":"2024-04-16T16:16:41.355834+0000","last_became_active":"2024-04-16T16:16:40.063617+0000","last_became_peered":"2024-04-16T16:16:40.063617+0000","last_unstale":"2024-04-16T16:16:41.355834+0000","last_undegraded":"2024-04-16T16:16:41.355834+0000","last_fullsized":"2024-04-16T16:16:41.355834+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:54:31.905334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079437+0000","last_change":"2024-04-16T16:16:40.060797+0000","last_active":"2024-04-16T16:16:41.079437+0000","last_peered":"2024-04-16T16:16:41.079437+0000","last_clean":"2024-04-16T16:16:41.079437+0000","last_became_active":"2024-04-16T16:16:40.060497+0000","last_became_peered":"2024-04-16T16:16:40.060497+0000","last_unstale":"2024-04-16T16:16:41.079437+0000","last_undegraded":"2024-04-16T16:16:41.079437+0000","last_fullsized":"2024-04-16T16:16:41.079437+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:43.199657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.356013+0000","last_change":"2024-04-16T16:16:40.067270+0000","last_active":"2024-04-16T16:16:41.356013+0000","last_peered":"2024-04-16T16:16:41.356013+0000","last_clean":"2024-04-16T16:16:41.356013+0000","last_became_active":"2024-04-16T16:16:40.066527+0000","last_became_peered":"2024-04-16T16:16:40.066527+0000","last_unstale":"2024-04-16T16:16:41.356013+0000","last_undegraded":"2024-04-16T16:16:41.356013+0000","last_fullsized":"2024-04-16T16:16:41.356013+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:06:06.933720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053974+0000","last_change":"2024-04-16T16:16:40.065283+0000","last_active":"2024-04-16T16:16:41.053974+0000","last_peered":"2024-04-16T16:16:41.053974+0000","last_clean":"2024-04-16T16:16:41.053974+0000","last_became_active":"2024-04-16T16:16:40.064211+0000","last_became_peered":"2024-04-16T16:16:40.064211+0000","last_unstale":"2024-04-16T16:16:41.053974+0000","last_undegraded":"2024-04-16T16:16:41.053974+0000","last_fullsized":"2024-04-16T16:16:41.053974+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:28.836553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.058144+0000","last_change":"2024-04-16T16:16:40.057046+0000","last_active":"2024-04-16T16:16:41.058144+0000","last_peered":"2024-04-16T16:16:41.058144+0000","last_clean":"2024-04-16T16:16:41.058144+0000","last_became_active":"2024-04-16T16:16:40.056773+0000","last_became_peered":"2024-04-16T16:16:40.056773+0000","last_unstale":"2024-04-16T16:16:41.058144+0000","last_undegraded":"2024-04-16T16:16:41.058144+0000","last_fullsized":"2024-04-16T16:16:41.058144+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:54:31.905334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.055911+0000","last_change":"2024-04-16T16:16:40.059929+0000","last_active":"2024-04-16T16:16:41.055911+0000","last_peered":"2024-04-16T16:16:41.055911+0000","last_clean":"2024-04-16T16:16:41.055911+0000","last_became_active":"2024-04-16T16:16:40.059666+0000","last_became_peered":"2024-04-16T16:16:40.059666+0000","last_unstale":"2024-04-16T16:16:41.055911+0000","last_undegraded":"2024-04-16T16:16:41.055911+0000","last_fullsized":"2024-04-16T16:16:41.055911+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:21.332023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354796+0000","last_change":"2024-04-16T16:16:40.070880+0000","last_active":"2024-04-16T16:16:41.354796+0000","last_peered":"2024-04-16T16:16:41.354796+0000","last_clean":"2024-04-16T16:16:41.354796+0000","last_became_active":"2024-04-16T16:16:40.070434+0000","last_became_peered":"2024-04-16T16:16:40.070434+0000","last_unstale":"2024-04-16T16:16:41.354796+0000","last_undegraded":"2024-04-16T16:16:41.354796+0000","last_fullsized":"2024-04-16T16:16:41.354796+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:13:47.015413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084447+0000","last_change":"2024-04-16T16:16:40.053229+0000","last_active":"2024-04-16T16:16:41.084447+0000","last_peered":"2024-04-16T16:16:41.084447+0000","last_clean":"2024-04-16T16:16:41.084447+0000","last_became_active":"2024-04-16T16:16:40.052968+0000","last_became_peered":"2024-04-16T16:16:40.052968+0000","last_unstale":"2024-04-16T16:16:41.084447+0000","last_undegraded":"2024-04-16T16:16:41.084447+0000","last_fullsized":"2024-04-16T16:16:41.084447+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:46.618649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052568+0000","last_change":"2024-04-16T16:16:40.059138+0000","last_active":"2024-04-16T16:16:41.052568+0000","last_peered":"2024-04-16T16:16:41.052568+0000","last_clean":"2024-04-16T16:16:41.052568+0000","last_became_active":"2024-04-16T16:16:40.056131+0000","last_became_peered":"2024-04-16T16:16:40.056131+0000","last_unstale":"2024-04-16T16:16:41.052568+0000","last_undegraded":"2024-04-16T16:16:41.052568+0000","last_fullsized":"2024-04-16T16:16:41.052568+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:39:52.139845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084236+0000","last_change":"2024-04-16T16:16:40.052923+0000","last_active":"2024-04-16T16:16:41.084236+0000","last_peered":"2024-04-16T16:16:41.084236+0000","last_clean":"2024-04-16T16:16:41.084236+0000","last_became_active":"2024-04-16T16:16:40.052609+0000","last_became_peered":"2024-04-16T16:16:40.052609+0000","last_unstale":"2024-04-16T16:16:41.084236+0000","last_undegraded":"2024-04-16T16:16:41.084236+0000","last_fullsized":"2024-04-16T16:16:41.084236+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:21.332023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057643+0000","last_change":"2024-04-16T16:16:40.055850+0000","last_active":"2024-04-16T16:16:41.057643+0000","last_peered":"2024-04-16T16:16:41.057643+0000","last_clean":"2024-04-16T16:16:41.057643+0000","last_became_active":"2024-04-16T16:16:40.055560+0000","last_became_peered":"2024-04-16T16:16:40.055560+0000","last_unstale":"2024-04-16T16:16:41.057643+0000","last_undegraded":"2024-04-16T16:16:41.057643+0000","last_fullsized":"2024-04-16T16:16:41.057643+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:22.397420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054029+0000","last_change":"2024-04-16T16:16:40.069834+0000","last_active":"2024-04-16T16:16:41.054029+0000","last_peered":"2024-04-16T16:16:41.054029+0000","last_clean":"2024-04-16T16:16:41.054029+0000","last_became_active":"2024-04-16T16:16:40.062429+0000","last_became_peered":"2024-04-16T16:16:40.062429+0000","last_unstale":"2024-04-16T16:16:41.054029+0000","last_undegraded":"2024-04-16T16:16:41.054029+0000","last_fullsized":"2024-04-16T16:16:41.054029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:14:20.465129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355562+0000","last_change":"2024-04-16T16:16:40.055791+0000","last_active":"2024-04-16T16:16:41.355562+0000","last_peered":"2024-04-16T16:16:41.355562+0000","last_clean":"2024-04-16T16:16:41.355562+0000","last_became_active":"2024-04-16T16:16:40.055508+0000","last_became_peered":"2024-04-16T16:16:40.055508+0000","last_unstale":"2024-04-16T16:16:41.355562+0000","last_undegraded":"2024-04-16T16:16:41.355562+0000","last_fullsized":"2024-04-16T16:16:41.355562+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:44.062254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355381+0000","last_change":"2024-04-16T16:16:40.067596+0000","last_active":"2024-04-16T16:16:41.355381+0000","last_peered":"2024-04-16T16:16:41.355381+0000","last_clean":"2024-04-16T16:16:41.355381+0000","last_became_active":"2024-04-16T16:16:40.067146+0000","last_became_peered":"2024-04-16T16:16:40.067146+0000","last_unstale":"2024-04-16T16:16:41.355381+0000","last_undegraded":"2024-04-16T16:16:41.355381+0000","last_fullsized":"2024-04-16T16:16:41.355381+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:07:41.285674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084256+0000","last_change":"2024-04-16T16:16:40.058979+0000","last_active":"2024-04-16T16:16:41.084256+0000","last_peered":"2024-04-16T16:16:41.084256+0000","last_clean":"2024-04-16T16:16:41.084256+0000","last_became_active":"2024-04-16T16:16:40.058648+0000","last_became_peered":"2024-04-16T16:16:40.058648+0000","last_unstale":"2024-04-16T16:16:41.084256+0000","last_undegraded":"2024-04-16T16:16:41.084256+0000","last_fullsized":"2024-04-16T16:16:41.084256+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:28:41.607987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056889+0000","last_change":"2024-04-16T16:16:40.065622+0000","last_active":"2024-04-16T16:16:41.056889+0000","last_peered":"2024-04-16T16:16:41.056889+0000","last_clean":"2024-04-16T16:16:41.056889+0000","last_became_active":"2024-04-16T16:16:40.064748+0000","last_became_peered":"2024-04-16T16:16:40.064748+0000","last_unstale":"2024-04-16T16:16:41.056889+0000","last_undegraded":"2024-04-16T16:16:41.056889+0000","last_fullsized":"2024-04-16T16:16:41.056889+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:12:52.888262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057883+0000","last_change":"2024-04-16T16:16:40.054692+0000","last_active":"2024-04-16T16:16:41.057883+0000","last_peered":"2024-04-16T16:16:41.057883+0000","last_clean":"2024-04-16T16:16:41.057883+0000","last_became_active":"2024-04-16T16:16:40.054392+0000","last_became_peered":"2024-04-16T16:16:40.054392+0000","last_unstale":"2024-04-16T16:16:41.057883+0000","last_undegraded":"2024-04-16T16:16:41.057883+0000","last_fullsized":"2024-04-16T16:16:41.057883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:25.467004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056193+0000","last_change":"2024-04-16T16:16:40.056002+0000","last_active":"2024-04-16T16:16:41.056193+0000","last_peered":"2024-04-16T16:16:41.056193+0000","last_clean":"2024-04-16T16:16:41.056193+0000","last_became_active":"2024-04-16T16:16:40.055212+0000","last_became_peered":"2024-04-16T16:16:40.055212+0000","last_unstale":"2024-04-16T16:16:41.056193+0000","last_undegraded":"2024-04-16T16:16:41.056193+0000","last_fullsized":"2024-04-16T16:16:41.056193+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:28:41.607987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054913+0000","last_change":"2024-04-16T16:16:40.054708+0000","last_active":"2024-04-16T16:16:41.054913+0000","last_peered":"2024-04-16T16:16:41.054913+0000","last_clean":"2024-04-16T16:16:41.054913+0000","last_became_active":"2024-04-16T16:16:40.053297+0000","last_became_peered":"2024-04-16T16:16:40.053297+0000","last_unstale":"2024-04-16T16:16:41.054913+0000","last_undegraded":"2024-04-16T16:16:41.054913+0000","last_fullsized":"2024-04-16T16:16:41.054913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:31:33.590968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.058757+0000","last_change":"2024-04-16T16:16:40.053469+0000","last_active":"2024-04-16T16:16:41.058757+0000","last_peered":"2024-04-16T16:16:41.058757+0000","last_clean":"2024-04-16T16:16:41.058757+0000","last_became_active":"2024-04-16T16:16:40.052617+0000","last_became_peered":"2024-04-16T16:16:40.052617+0000","last_unstale":"2024-04-16T16:16:41.058757+0000","last_undegraded":"2024-04-16T16:16:41.058757+0000","last_fullsized":"2024-04-16T16:16:41.058757+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:06:06.933720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.080143+0000","last_change":"2024-04-16T16:16:40.058621+0000","last_active":"2024-04-16T16:16:41.080143+0000","last_peered":"2024-04-16T16:16:41.080143+0000","last_clean":"2024-04-16T16:16:41.080143+0000","last_became_active":"2024-04-16T16:16:40.058250+0000","last_became_peered":"2024-04-16T16:16:40.058250+0000","last_unstale":"2024-04-16T16:16:41.080143+0000","last_undegraded":"2024-04-16T16:16:41.080143+0000","last_fullsized":"2024-04-16T16:16:41.080143+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:21.332023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353888+0000","last_change":"2024-04-16T16:16:40.066255+0000","last_active":"2024-04-16T16:16:41.353888+0000","last_peered":"2024-04-16T16:16:41.353888+0000","last_clean":"2024-04-16T16:16:41.353888+0000","last_became_active":"2024-04-16T16:16:40.065779+0000","last_became_peered":"2024-04-16T16:16:40.065779+0000","last_unstale":"2024-04-16T16:16:41.353888+0000","last_undegraded":"2024-04-16T16:16:41.353888+0000","last_fullsized":"2024-04-16T16:16:41.353888+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:26:05.697779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057071+0000","last_change":"2024-04-16T16:16:40.056911+0000","last_active":"2024-04-16T16:16:41.057071+0000","last_peered":"2024-04-16T16:16:41.057071+0000","last_clean":"2024-04-16T16:16:41.057071+0000","last_became_active":"2024-04-16T16:16:40.054450+0000","last_became_peered":"2024-04-16T16:16:40.054450+0000","last_unstale":"2024-04-16T16:16:41.057071+0000","last_undegraded":"2024-04-16T16:16:41.057071+0000","last_fullsized":"2024-04-16T16:16:41.057071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:07:41.285674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353512+0000","last_change":"2024-04-16T16:16:40.055812+0000","last_active":"2024-04-16T16:16:41.353512+0000","last_peered":"2024-04-16T16:16:41.353512+0000","last_clean":"2024-04-16T16:16:41.353512+0000","last_became_active":"2024-04-16T16:16:40.055488+0000","last_became_peered":"2024-04-16T16:16:40.055488+0000","last_unstale":"2024-04-16T16:16:41.353512+0000","last_undegraded":"2024-04-16T16:16:41.353512+0000","last_fullsized":"2024-04-16T16:16:41.353512+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:49:49.405418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079606+0000","last_change":"2024-04-16T16:16:40.056514+0000","last_active":"2024-04-16T16:16:41.079606+0000","last_peered":"2024-04-16T16:16:41.079606+0000","last_clean":"2024-04-16T16:16:41.079606+0000","last_became_active":"2024-04-16T16:16:40.056184+0000","last_became_peered":"2024-04-16T16:16:40.056184+0000","last_unstale":"2024-04-16T16:16:41.079606+0000","last_undegraded":"2024-04-16T16:16:41.079606+0000","last_fullsized":"2024-04-16T16:16:41.079606+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:08:39.372895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079878+0000","last_change":"2024-04-16T16:16:40.054600+0000","last_active":"2024-04-16T16:16:41.079878+0000","last_peered":"2024-04-16T16:16:41.079878+0000","last_clean":"2024-04-16T16:16:41.079878+0000","last_became_active":"2024-04-16T16:16:40.054303+0000","last_became_peered":"2024-04-16T16:16:40.054303+0000","last_unstale":"2024-04-16T16:16:41.079878+0000","last_undegraded":"2024-04-16T16:16:41.079878+0000","last_fullsized":"2024-04-16T16:16:41.079878+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:31:33.590968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355739+0000","last_change":"2024-04-16T16:16:40.066863+0000","last_active":"2024-04-16T16:16:41.355739+0000","last_peered":"2024-04-16T16:16:41.355739+0000","last_clean":"2024-04-16T16:16:41.355739+0000","last_became_active":"2024-04-16T16:16:40.064405+0000","last_became_peered":"2024-04-16T16:16:40.064405+0000","last_unstale":"2024-04-16T16:16:41.355739+0000","last_undegraded":"2024-04-16T16:16:41.355739+0000","last_fullsized":"2024-04-16T16:16:41.355739+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:25.467004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056006+0000","last_change":"2024-04-16T16:16:40.052907+0000","last_active":"2024-04-16T16:16:41.056006+0000","last_peered":"2024-04-16T16:16:41.056006+0000","last_clean":"2024-04-16T16:16:41.056006+0000","last_became_active":"2024-04-16T16:16:40.052066+0000","last_became_peered":"2024-04-16T16:16:40.052066+0000","last_unstale":"2024-04-16T16:16:41.056006+0000","last_undegraded":"2024-04-16T16:16:41.056006+0000","last_fullsized":"2024-04-16T16:16:41.056006+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:48:35.347056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"71'4","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057049+0000","last_change":"2024-04-16T16:16:39.052943+0000","last_active":"2024-04-16T16:16:41.057049+0000","last_peered":"2024-04-16T16:16:41.057049+0000","last_clean":"2024-04-16T16:16:41.057049+0000","last_became_active":"2024-04-16T16:16:39.052572+0000","last_became_peered":"2024-04-16T16:16:39.052572+0000","last_unstale":"2024-04-16T16:16:41.057049+0000","last_undegraded":"2024-04-16T16:16:41.057049+0000","last_fullsized":"2024-04-16T16:16:41.057049+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:54:30.889318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054498+0000","last_change":"2024-04-16T16:16:40.063884+0000","last_active":"2024-04-16T16:16:41.054498+0000","last_peered":"2024-04-16T16:16:41.054498+0000","last_clean":"2024-04-16T16:16:41.054498+0000","last_became_active":"2024-04-16T16:16:40.061609+0000","last_became_peered":"2024-04-16T16:16:40.061609+0000","last_unstale":"2024-04-16T16:16:41.054498+0000","last_undegraded":"2024-04-16T16:16:41.054498+0000","last_fullsized":"2024-04-16T16:16:41.054498+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:55.827541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084571+0000","last_change":"2024-04-16T16:16:39.035506+0000","last_active":"2024-04-16T16:16:41.084571+0000","last_peered":"2024-04-16T16:16:41.084571+0000","last_clean":"2024-04-16T16:16:41.084571+0000","last_became_active":"2024-04-16T16:16:39.035196+0000","last_became_peered":"2024-04-16T16:16:39.035196+0000","last_unstale":"2024-04-16T16:16:41.084571+0000","last_undegraded":"2024-04-16T16:16:41.084571+0000","last_fullsized":"2024-04-16T16:16:41.084571+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:29:14.659131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355912+0000","last_change":"2024-04-16T16:16:40.061879+0000","last_active":"2024-04-16T16:16:41.355912+0000","last_peered":"2024-04-16T16:16:41.355912+0000","last_clean":"2024-04-16T16:16:41.355912+0000","last_became_active":"2024-04-16T16:16:40.058652+0000","last_became_peered":"2024-04-16T16:16:40.058652+0000","last_unstale":"2024-04-16T16:16:41.355912+0000","last_undegraded":"2024-04-16T16:16:41.355912+0000","last_fullsized":"2024-04-16T16:16:41.355912+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:12.235559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057778+0000","last_change":"2024-04-16T16:16:39.057527+0000","last_active":"2024-04-16T16:16:41.057778+0000","last_peered":"2024-04-16T16:16:41.057778+0000","last_clean":"2024-04-16T16:16:41.057778+0000","last_became_active":"2024-04-16T16:16:39.056172+0000","last_became_peered":"2024-04-16T16:16:39.056172+0000","last_unstale":"2024-04-16T16:16:41.057778+0000","last_undegraded":"2024-04-16T16:16:41.057778+0000","last_fullsized":"2024-04-16T16:16:41.057778+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:13:45.999397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056353+0000","last_change":"2024-04-16T16:16:40.058328+0000","last_active":"2024-04-16T16:16:41.056353+0000","last_peered":"2024-04-16T16:16:41.056353+0000","last_clean":"2024-04-16T16:16:41.056353+0000","last_became_active":"2024-04-16T16:16:40.058025+0000","last_became_peered":"2024-04-16T16:16:40.058025+0000","last_unstale":"2024-04-16T16:16:41.056353+0000","last_undegraded":"2024-04-16T16:16:41.056353+0000","last_fullsized":"2024-04-16T16:16:41.056353+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:46.618649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355874+0000","last_change":"2024-04-16T16:16:39.042499+0000","last_active":"2024-04-16T16:16:41.355874+0000","last_peered":"2024-04-16T16:16:41.355874+0000","last_clean":"2024-04-16T16:16:41.355874+0000","last_became_active":"2024-04-16T16:16:39.042042+0000","last_became_peered":"2024-04-16T16:16:39.042042+0000","last_unstale":"2024-04-16T16:16:41.355874+0000","last_undegraded":"2024-04-16T16:16:41.355874+0000","last_fullsized":"2024-04-16T16:16:41.355874+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:40:02.778520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079720+0000","last_change":"2024-04-16T16:16:40.061501+0000","last_active":"2024-04-16T16:16:41.079720+0000","last_peered":"2024-04-16T16:16:41.079720+0000","last_clean":"2024-04-16T16:16:41.079720+0000","last_became_active":"2024-04-16T16:16:40.061046+0000","last_became_peered":"2024-04-16T16:16:40.061046+0000","last_unstale":"2024-04-16T16:16:41.079720+0000","last_undegraded":"2024-04-16T16:16:41.079720+0000","last_fullsized":"2024-04-16T16:16:41.079720+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:34:19.296917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353704+0000","last_change":"2024-04-16T16:16:39.051382+0000","last_active":"2024-04-16T16:16:41.353704+0000","last_peered":"2024-04-16T16:16:41.353704+0000","last_clean":"2024-04-16T16:16:41.353704+0000","last_became_active":"2024-04-16T16:16:39.050793+0000","last_became_peered":"2024-04-16T16:16:39.050793+0000","last_unstale":"2024-04-16T16:16:41.353704+0000","last_undegraded":"2024-04-16T16:16:41.353704+0000","last_fullsized":"2024-04-16T16:16:41.353704+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:55:57.308712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057316+0000","last_change":"2024-04-16T16:16:40.049380+0000","last_active":"2024-04-16T16:16:41.057316+0000","last_peered":"2024-04-16T16:16:41.057316+0000","last_clean":"2024-04-16T16:16:41.057316+0000","last_became_active":"2024-04-16T16:16:40.049071+0000","last_became_peered":"2024-04-16T16:16:40.049071+0000","last_unstale":"2024-04-16T16:16:41.057316+0000","last_undegraded":"2024-04-16T16:16:41.057316+0000","last_fullsized":"2024-04-16T16:16:41.057316+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:45.673814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354835+0000","last_change":"2024-04-16T16:16:39.046490+0000","last_active":"2024-04-16T16:16:41.354835+0000","last_peered":"2024-04-16T16:16:41.354835+0000","last_clean":"2024-04-16T16:16:41.354835+0000","last_became_active":"2024-04-16T16:16:39.045734+0000","last_became_peered":"2024-04-16T16:16:39.045734+0000","last_unstale":"2024-04-16T16:16:41.354835+0000","last_undegraded":"2024-04-16T16:16:41.354835+0000","last_fullsized":"2024-04-16T16:16:41.354835+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:37:31.768292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.051911+0000","last_change":"2024-04-16T16:16:40.049943+0000","last_active":"2024-04-16T16:16:41.051911+0000","last_peered":"2024-04-16T16:16:41.051911+0000","last_clean":"2024-04-16T16:16:41.051911+0000","last_became_active":"2024-04-16T16:16:40.049466+0000","last_became_peered":"2024-04-16T16:16:40.049466+0000","last_unstale":"2024-04-16T16:16:41.051911+0000","last_undegraded":"2024-04-16T16:16:41.051911+0000","last_fullsized":"2024-04-16T16:16:41.051911+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:24.082098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353887+0000","last_change":"2024-04-16T16:16:39.048665+0000","last_active":"2024-04-16T16:16:41.353887+0000","last_peered":"2024-04-16T16:16:41.353887+0000","last_clean":"2024-04-16T16:16:41.353887+0000","last_became_active":"2024-04-16T16:16:39.046018+0000","last_became_peered":"2024-04-16T16:16:39.046018+0000","last_unstale":"2024-04-16T16:16:41.353887+0000","last_undegraded":"2024-04-16T16:16:41.353887+0000","last_fullsized":"2024-04-16T16:16:41.353887+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:37:31.768292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084230+0000","last_change":"2024-04-16T16:16:40.068846+0000","last_active":"2024-04-16T16:16:41.084230+0000","last_peered":"2024-04-16T16:16:41.084230+0000","last_clean":"2024-04-16T16:16:41.084230+0000","last_became_active":"2024-04-16T16:16:40.060719+0000","last_became_peered":"2024-04-16T16:16:40.060719+0000","last_unstale":"2024-04-16T16:16:41.084230+0000","last_undegraded":"2024-04-16T16:16:41.084230+0000","last_fullsized":"2024-04-16T16:16:41.084230+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:48:35.347056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.051833+0000","last_change":"2024-04-16T16:16:39.042020+0000","last_active":"2024-04-16T16:16:41.051833+0000","last_peered":"2024-04-16T16:16:41.051833+0000","last_clean":"2024-04-16T16:16:41.051833+0000","last_became_active":"2024-04-16T16:16:39.041447+0000","last_became_peered":"2024-04-16T16:16:39.041447+0000","last_unstale":"2024-04-16T16:16:41.051833+0000","last_undegraded":"2024-04-16T16:16:41.051833+0000","last_fullsized":"2024-04-16T16:16:41.051833+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:18:24.097920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.080180+0000","last_change":"2024-04-16T16:16:40.060588+0000","last_active":"2024-04-16T16:16:41.080180+0000","last_peered":"2024-04-16T16:16:41.080180+0000","last_clean":"2024-04-16T16:16:41.080180+0000","last_became_active":"2024-04-16T16:16:40.060149+0000","last_became_peered":"2024-04-16T16:16:40.060149+0000","last_unstale":"2024-04-16T16:16:41.080180+0000","last_undegraded":"2024-04-16T16:16:41.080180+0000","last_fullsized":"2024-04-16T16:16:41.080180+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:48:35.347056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.058616+0000","last_change":"2024-04-16T16:16:39.052655+0000","last_active":"2024-04-16T16:16:41.058616+0000","last_peered":"2024-04-16T16:16:41.058616+0000","last_clean":"2024-04-16T16:16:41.058616+0000","last_became_active":"2024-04-16T16:16:39.051190+0000","last_became_peered":"2024-04-16T16:16:39.051190+0000","last_unstale":"2024-04-16T16:16:41.058616+0000","last_undegraded":"2024-04-16T16:16:41.058616+0000","last_fullsized":"2024-04-16T16:16:41.058616+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:43.334972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056462+0000","last_change":"2024-04-16T16:16:40.051811+0000","last_active":"2024-04-16T16:16:41.056462+0000","last_peered":"2024-04-16T16:16:41.056462+0000","last_clean":"2024-04-16T16:16:41.056462+0000","last_became_active":"2024-04-16T16:16:40.051476+0000","last_became_peered":"2024-04-16T16:16:40.051476+0000","last_unstale":"2024-04-16T16:16:41.056462+0000","last_undegraded":"2024-04-16T16:16:41.056462+0000","last_fullsized":"2024-04-16T16:16:41.056462+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:29.398550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355529+0000","last_change":"2024-04-16T16:16:39.056709+0000","last_active":"2024-04-16T16:16:41.355529+0000","last_peered":"2024-04-16T16:16:41.355529+0000","last_clean":"2024-04-16T16:16:41.355529+0000","last_became_active":"2024-04-16T16:16:39.056334+0000","last_became_peered":"2024-04-16T16:16:39.056334+0000","last_unstale":"2024-04-16T16:16:41.355529+0000","last_undegraded":"2024-04-16T16:16:41.355529+0000","last_fullsized":"2024-04-16T16:16:41.355529+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:52:38.661658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355675+0000","last_change":"2024-04-16T16:16:40.055050+0000","last_active":"2024-04-16T16:16:41.355675+0000","last_peered":"2024-04-16T16:16:41.355675+0000","last_clean":"2024-04-16T16:16:41.355675+0000","last_became_active":"2024-04-16T16:16:40.054723+0000","last_became_peered":"2024-04-16T16:16:40.054723+0000","last_unstale":"2024-04-16T16:16:41.355675+0000","last_undegraded":"2024-04-16T16:16:41.355675+0000","last_fullsized":"2024-04-16T16:16:41.355675+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:22.397420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"71'3","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353950+0000","last_change":"2024-04-16T16:16:39.046939+0000","last_active":"2024-04-16T16:16:41.353950+0000","last_peered":"2024-04-16T16:16:41.353950+0000","last_clean":"2024-04-16T16:16:41.353950+0000","last_became_active":"2024-04-16T16:16:39.046609+0000","last_became_peered":"2024-04-16T16:16:39.046609+0000","last_unstale":"2024-04-16T16:16:41.353950+0000","last_undegraded":"2024-04-16T16:16:41.353950+0000","last_fullsized":"2024-04-16T16:16:41.353950+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:38:05.588116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053162+0000","last_change":"2024-04-16T16:16:40.056836+0000","last_active":"2024-04-16T16:16:41.053162+0000","last_peered":"2024-04-16T16:16:41.053162+0000","last_clean":"2024-04-16T16:16:41.053162+0000","last_became_active":"2024-04-16T16:16:40.056365+0000","last_became_peered":"2024-04-16T16:16:40.056365+0000","last_unstale":"2024-04-16T16:16:41.053162+0000","last_undegraded":"2024-04-16T16:16:41.053162+0000","last_fullsized":"2024-04-16T16:16:41.053162+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:17:23.179723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"71'6","reported_seq":20,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355016+0000","last_change":"2024-04-16T16:16:39.040106+0000","last_active":"2024-04-16T16:16:41.355016+0000","last_peered":"2024-04-16T16:16:41.355016+0000","last_clean":"2024-04-16T16:16:41.355016+0000","last_became_active":"2024-04-16T16:16:39.039334+0000","last_became_peered":"2024-04-16T16:16:39.039334+0000","last_unstale":"2024-04-16T16:16:41.355016+0000","last_undegraded":"2024-04-16T16:16:41.355016+0000","last_fullsized":"2024-04-16T16:16:41.355016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:29.791246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056901+0000","last_change":"2024-04-16T16:16:40.049290+0000","last_active":"2024-04-16T16:16:41.056901+0000","last_peered":"2024-04-16T16:16:41.056901+0000","last_clean":"2024-04-16T16:16:41.056901+0000","last_became_active":"2024-04-16T16:16:40.048726+0000","last_became_peered":"2024-04-16T16:16:40.048726+0000","last_unstale":"2024-04-16T16:16:41.056901+0000","last_undegraded":"2024-04-16T16:16:41.056901+0000","last_fullsized":"2024-04-16T16:16:41.056901+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:41:00.935635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354451+0000","last_change":"2024-04-16T16:16:39.048257+0000","last_active":"2024-04-16T16:16:41.354451+0000","last_peered":"2024-04-16T16:16:41.354451+0000","last_clean":"2024-04-16T16:16:41.354451+0000","last_became_active":"2024-04-16T16:16:39.047835+0000","last_became_peered":"2024-04-16T16:16:39.047835+0000","last_unstale":"2024-04-16T16:16:41.354451+0000","last_undegraded":"2024-04-16T16:16:41.354451+0000","last_fullsized":"2024-04-16T16:16:41.354451+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:46.723824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079769+0000","last_change":"2024-04-16T16:16:40.055833+0000","last_active":"2024-04-16T16:16:41.079769+0000","last_peered":"2024-04-16T16:16:41.079769+0000","last_clean":"2024-04-16T16:16:41.079769+0000","last_became_active":"2024-04-16T16:16:40.055564+0000","last_became_peered":"2024-04-16T16:16:40.055564+0000","last_unstale":"2024-04-16T16:16:41.079769+0000","last_undegraded":"2024-04-16T16:16:41.079769+0000","last_fullsized":"2024-04-16T16:16:41.079769+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:51:29.994460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355134+0000","last_change":"2024-04-16T16:16:39.046580+0000","last_active":"2024-04-16T16:16:41.355134+0000","last_peered":"2024-04-16T16:16:41.355134+0000","last_clean":"2024-04-16T16:16:41.355134+0000","last_became_active":"2024-04-16T16:16:39.046181+0000","last_became_peered":"2024-04-16T16:16:39.046181+0000","last_unstale":"2024-04-16T16:16:41.355134+0000","last_undegraded":"2024-04-16T16:16:41.355134+0000","last_fullsized":"2024-04-16T16:16:41.355134+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:29.791246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084205+0000","last_change":"2024-04-16T16:16:40.055025+0000","last_active":"2024-04-16T16:16:41.084205+0000","last_peered":"2024-04-16T16:16:41.084205+0000","last_clean":"2024-04-16T16:16:41.084205+0000","last_became_active":"2024-04-16T16:16:40.054757+0000","last_became_peered":"2024-04-16T16:16:40.054757+0000","last_unstale":"2024-04-16T16:16:41.084205+0000","last_undegraded":"2024-04-16T16:16:41.084205+0000","last_fullsized":"2024-04-16T16:16:41.084205+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:31:33.590968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.079273+0000","last_change":"2024-04-16T16:16:39.052798+0000","last_active":"2024-04-16T16:16:41.079273+0000","last_peered":"2024-04-16T16:16:41.079273+0000","last_clean":"2024-04-16T16:16:41.079273+0000","last_became_active":"2024-04-16T16:16:39.052440+0000","last_became_peered":"2024-04-16T16:16:39.052440+0000","last_unstale":"2024-04-16T16:16:41.079273+0000","last_undegraded":"2024-04-16T16:16:41.079273+0000","last_fullsized":"2024-04-16T16:16:41.079273+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:44.657799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052671+0000","last_change":"2024-04-16T16:16:40.054326+0000","last_active":"2024-04-16T16:16:41.052671+0000","last_peered":"2024-04-16T16:16:41.052671+0000","last_clean":"2024-04-16T16:16:41.052671+0000","last_became_active":"2024-04-16T16:16:40.054013+0000","last_became_peered":"2024-04-16T16:16:40.054013+0000","last_unstale":"2024-04-16T16:16:41.052671+0000","last_undegraded":"2024-04-16T16:16:41.052671+0000","last_fullsized":"2024-04-16T16:16:41.052671+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:05:08.507551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"71'8","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.057407+0000","last_change":"2024-04-16T16:16:39.053695+0000","last_active":"2024-04-16T16:16:41.057407+0000","last_peered":"2024-04-16T16:16:41.057407+0000","last_clean":"2024-04-16T16:16:41.057407+0000","last_became_active":"2024-04-16T16:16:39.053264+0000","last_became_peered":"2024-04-16T16:16:39.053264+0000","last_unstale":"2024-04-16T16:16:41.057407+0000","last_undegraded":"2024-04-16T16:16:41.057407+0000","last_fullsized":"2024-04-16T16:16:41.057407+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:39:32.200451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.058313+0000","last_change":"2024-04-16T16:16:40.056673+0000","last_active":"2024-04-16T16:16:41.058313+0000","last_peered":"2024-04-16T16:16:41.058313+0000","last_clean":"2024-04-16T16:16:41.058313+0000","last_became_active":"2024-04-16T16:16:40.056357+0000","last_became_peered":"2024-04-16T16:16:40.056357+0000","last_unstale":"2024-04-16T16:16:41.058313+0000","last_undegraded":"2024-04-16T16:16:41.058313+0000","last_fullsized":"2024-04-16T16:16:41.058313+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:12.235559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355020+0000","last_change":"2024-04-16T16:16:39.046833+0000","last_active":"2024-04-16T16:16:41.355020+0000","last_peered":"2024-04-16T16:16:41.355020+0000","last_clean":"2024-04-16T16:16:41.355020+0000","last_became_active":"2024-04-16T16:16:39.046543+0000","last_became_peered":"2024-04-16T16:16:39.046543+0000","last_unstale":"2024-04-16T16:16:41.355020+0000","last_undegraded":"2024-04-16T16:16:41.355020+0000","last_fullsized":"2024-04-16T16:16:41.355020+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:39:32.200451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056906+0000","last_change":"2024-04-16T16:16:40.059425+0000","last_active":"2024-04-16T16:16:41.056906+0000","last_peered":"2024-04-16T16:16:41.056906+0000","last_clean":"2024-04-16T16:16:41.056906+0000","last_became_active":"2024-04-16T16:16:40.059078+0000","last_became_peered":"2024-04-16T16:16:40.059078+0000","last_unstale":"2024-04-16T16:16:41.056906+0000","last_undegraded":"2024-04-16T16:16:41.056906+0000","last_fullsized":"2024-04-16T16:16:41.056906+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:50:55.122092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.058306+0000","last_change":"2024-04-16T16:16:39.057643+0000","last_active":"2024-04-16T16:16:41.058306+0000","last_peered":"2024-04-16T16:16:41.058306+0000","last_clean":"2024-04-16T16:16:41.058306+0000","last_became_active":"2024-04-16T16:16:39.057160+0000","last_became_peered":"2024-04-16T16:16:39.057160+0000","last_unstale":"2024-04-16T16:16:41.058306+0000","last_undegraded":"2024-04-16T16:16:41.058306+0000","last_fullsized":"2024-04-16T16:16:41.058306+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:59:03.269004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354958+0000","last_change":"2024-04-16T16:16:40.066424+0000","last_active":"2024-04-16T16:16:41.354958+0000","last_peered":"2024-04-16T16:16:41.354958+0000","last_clean":"2024-04-16T16:16:41.354958+0000","last_became_active":"2024-04-16T16:16:40.063948+0000","last_became_peered":"2024-04-16T16:16:40.063948+0000","last_unstale":"2024-04-16T16:16:41.354958+0000","last_undegraded":"2024-04-16T16:16:41.354958+0000","last_fullsized":"2024-04-16T16:16:41.354958+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:59:04.285019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355470+0000","last_change":"2024-04-16T16:16:39.054911+0000","last_active":"2024-04-16T16:16:41.355470+0000","last_peered":"2024-04-16T16:16:41.355470+0000","last_clean":"2024-04-16T16:16:41.355470+0000","last_became_active":"2024-04-16T16:16:39.053957+0000","last_became_peered":"2024-04-16T16:16:39.053957+0000","last_unstale":"2024-04-16T16:16:41.355470+0000","last_undegraded":"2024-04-16T16:16:41.355470+0000","last_fullsized":"2024-04-16T16:16:41.355470+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:46.723824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053204+0000","last_change":"2024-04-16T16:16:40.052810+0000","last_active":"2024-04-16T16:16:41.053204+0000","last_peered":"2024-04-16T16:16:41.053204+0000","last_clean":"2024-04-16T16:16:41.053204+0000","last_became_active":"2024-04-16T16:16:40.051522+0000","last_became_peered":"2024-04-16T16:16:40.051522+0000","last_unstale":"2024-04-16T16:16:41.053204+0000","last_undegraded":"2024-04-16T16:16:41.053204+0000","last_fullsized":"2024-04-16T16:16:41.053204+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:53:25.229933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056633+0000","last_change":"2024-04-16T16:16:39.054182+0000","last_active":"2024-04-16T16:16:41.056633+0000","last_peered":"2024-04-16T16:16:41.056633+0000","last_clean":"2024-04-16T16:16:41.056633+0000","last_became_active":"2024-04-16T16:16:39.053902+0000","last_became_peered":"2024-04-16T16:16:39.053902+0000","last_unstale":"2024-04-16T16:16:41.056633+0000","last_undegraded":"2024-04-16T16:16:41.056633+0000","last_fullsized":"2024-04-16T16:16:41.056633+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:11.219544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353546+0000","last_change":"2024-04-16T16:16:40.062784+0000","last_active":"2024-04-16T16:16:41.353546+0000","last_peered":"2024-04-16T16:16:41.353546+0000","last_clean":"2024-04-16T16:16:41.353546+0000","last_became_active":"2024-04-16T16:16:40.062465+0000","last_became_peered":"2024-04-16T16:16:40.062465+0000","last_unstale":"2024-04-16T16:16:41.353546+0000","last_undegraded":"2024-04-16T16:16:41.353546+0000","last_fullsized":"2024-04-16T16:16:41.353546+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:29:01.496515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355429+0000","last_change":"2024-04-16T16:16:39.045311+0000","last_active":"2024-04-16T16:16:41.355429+0000","last_peered":"2024-04-16T16:16:41.355429+0000","last_clean":"2024-04-16T16:16:41.355429+0000","last_became_active":"2024-04-16T16:16:39.044896+0000","last_became_peered":"2024-04-16T16:16:39.044896+0000","last_unstale":"2024-04-16T16:16:41.355429+0000","last_undegraded":"2024-04-16T16:16:41.355429+0000","last_fullsized":"2024-04-16T16:16:41.355429+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:58:16.807985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354225+0000","last_change":"2024-04-16T16:16:40.054819+0000","last_active":"2024-04-16T16:16:41.354225+0000","last_peered":"2024-04-16T16:16:41.354225+0000","last_clean":"2024-04-16T16:16:41.354225+0000","last_became_active":"2024-04-16T16:16:40.054521+0000","last_became_peered":"2024-04-16T16:16:40.054521+0000","last_unstale":"2024-04-16T16:16:41.354225+0000","last_undegraded":"2024-04-16T16:16:41.354225+0000","last_fullsized":"2024-04-16T16:16:41.354225+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:01:17.068261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.080016+0000","last_change":"2024-04-16T16:16:39.055458+0000","last_active":"2024-04-16T16:16:41.080016+0000","last_peered":"2024-04-16T16:16:41.080016+0000","last_clean":"2024-04-16T16:16:41.080016+0000","last_became_active":"2024-04-16T16:16:39.053012+0000","last_became_peered":"2024-04-16T16:16:39.053012+0000","last_unstale":"2024-04-16T16:16:41.080016+0000","last_undegraded":"2024-04-16T16:16:41.080016+0000","last_fullsized":"2024-04-16T16:16:41.080016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:21.381404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353660+0000","last_change":"2024-04-16T16:16:40.053963+0000","last_active":"2024-04-16T16:16:41.353660+0000","last_peered":"2024-04-16T16:16:41.353660+0000","last_clean":"2024-04-16T16:16:41.353660+0000","last_became_active":"2024-04-16T16:16:40.053682+0000","last_became_peered":"2024-04-16T16:16:40.053682+0000","last_unstale":"2024-04-16T16:16:41.353660+0000","last_undegraded":"2024-04-16T16:16:41.353660+0000","last_fullsized":"2024-04-16T16:16:41.353660+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:16.693760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.083960+0000","last_change":"2024-04-16T16:16:39.038427+0000","last_active":"2024-04-16T16:16:41.083960+0000","last_peered":"2024-04-16T16:16:41.083960+0000","last_clean":"2024-04-16T16:16:41.083960+0000","last_became_active":"2024-04-16T16:16:39.036287+0000","last_became_peered":"2024-04-16T16:16:39.036287+0000","last_unstale":"2024-04-16T16:16:41.083960+0000","last_undegraded":"2024-04-16T16:16:41.083960+0000","last_fullsized":"2024-04-16T16:16:41.083960+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:38.804660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084649+0000","last_change":"2024-04-16T16:16:40.050769+0000","last_active":"2024-04-16T16:16:41.084649+0000","last_peered":"2024-04-16T16:16:41.084649+0000","last_clean":"2024-04-16T16:16:41.084649+0000","last_became_active":"2024-04-16T16:16:40.050449+0000","last_became_peered":"2024-04-16T16:16:40.050449+0000","last_unstale":"2024-04-16T16:16:41.084649+0000","last_undegraded":"2024-04-16T16:16:41.084649+0000","last_fullsized":"2024-04-16T16:16:41.084649+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:29.398550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355074+0000","last_change":"2024-04-16T16:16:39.047443+0000","last_active":"2024-04-16T16:16:41.355074+0000","last_peered":"2024-04-16T16:16:41.355074+0000","last_clean":"2024-04-16T16:16:41.355074+0000","last_became_active":"2024-04-16T16:16:39.046833+0000","last_became_peered":"2024-04-16T16:16:39.046833+0000","last_unstale":"2024-04-16T16:16:41.355074+0000","last_undegraded":"2024-04-16T16:16:41.355074+0000","last_fullsized":"2024-04-16T16:16:41.355074+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:54:40.017503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.080259+0000","last_change":"2024-04-16T16:16:40.062706+0000","last_active":"2024-04-16T16:16:41.080259+0000","last_peered":"2024-04-16T16:16:41.080259+0000","last_clean":"2024-04-16T16:16:41.080259+0000","last_became_active":"2024-04-16T16:16:40.060527+0000","last_became_peered":"2024-04-16T16:16:40.060527+0000","last_unstale":"2024-04-16T16:16:41.080259+0000","last_undegraded":"2024-04-16T16:16:41.080259+0000","last_fullsized":"2024-04-16T16:16:41.080259+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:28:41.607987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"72'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.080437+0000","last_change":"2024-04-16T16:16:39.055132+0000","last_active":"2024-04-16T16:16:41.080437+0000","last_peered":"2024-04-16T16:16:41.080437+0000","last_clean":"2024-04-16T16:16:41.080437+0000","last_became_active":"2024-04-16T16:16:39.054859+0000","last_became_peered":"2024-04-16T16:16:39.054859+0000","last_unstale":"2024-04-16T16:16:41.080437+0000","last_undegraded":"2024-04-16T16:16:41.080437+0000","last_fullsized":"2024-04-16T16:16:41.080437+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:24.450988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056499+0000","last_change":"2024-04-16T16:16:40.056526+0000","last_active":"2024-04-16T16:16:41.056499+0000","last_peered":"2024-04-16T16:16:41.056499+0000","last_clean":"2024-04-16T16:16:41.056499+0000","last_became_active":"2024-04-16T16:16:40.055969+0000","last_became_peered":"2024-04-16T16:16:40.055969+0000","last_unstale":"2024-04-16T16:16:41.056499+0000","last_undegraded":"2024-04-16T16:16:41.056499+0000","last_fullsized":"2024-04-16T16:16:41.056499+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:59:46.671848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.053323+0000","last_change":"2024-04-16T16:16:39.044030+0000","last_active":"2024-04-16T16:16:41.053323+0000","last_peered":"2024-04-16T16:16:41.053323+0000","last_clean":"2024-04-16T16:16:41.053323+0000","last_became_active":"2024-04-16T16:16:39.043710+0000","last_became_peered":"2024-04-16T16:16:39.043710+0000","last_unstale":"2024-04-16T16:16:41.053323+0000","last_undegraded":"2024-04-16T16:16:41.053323+0000","last_fullsized":"2024-04-16T16:16:41.053323+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:15.221340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.056763+0000","last_change":"2024-04-16T16:16:40.056711+0000","last_active":"2024-04-16T16:16:41.056763+0000","last_peered":"2024-04-16T16:16:41.056763+0000","last_clean":"2024-04-16T16:16:41.056763+0000","last_became_active":"2024-04-16T16:16:40.053649+0000","last_became_peered":"2024-04-16T16:16:40.053649+0000","last_unstale":"2024-04-16T16:16:41.056763+0000","last_undegraded":"2024-04-16T16:16:41.056763+0000","last_fullsized":"2024-04-16T16:16:41.056763+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:45:10.434171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.084392+0000","last_change":"2024-04-16T16:16:39.048354+0000","last_active":"2024-04-16T16:16:41.084392+0000","last_peered":"2024-04-16T16:16:41.084392+0000","last_clean":"2024-04-16T16:16:41.084392+0000","last_became_active":"2024-04-16T16:16:39.048098+0000","last_became_peered":"2024-04-16T16:16:39.048098+0000","last_unstale":"2024-04-16T16:16:41.084392+0000","last_undegraded":"2024-04-16T16:16:41.084392+0000","last_fullsized":"2024-04-16T16:16:41.084392+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:37:41.504380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052443+0000","last_change":"2024-04-16T16:16:40.061326+0000","last_active":"2024-04-16T16:16:41.052443+0000","last_peered":"2024-04-16T16:16:41.052443+0000","last_clean":"2024-04-16T16:16:41.052443+0000","last_became_active":"2024-04-16T16:16:40.056522+0000","last_became_peered":"2024-04-16T16:16:40.056522+0000","last_unstale":"2024-04-16T16:16:41.052443+0000","last_undegraded":"2024-04-16T16:16:41.052443+0000","last_fullsized":"2024-04-16T16:16:41.052443+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:43:46.183862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.052332+0000","last_change":"2024-04-16T16:16:39.051171+0000","last_active":"2024-04-16T16:16:41.052332+0000","last_peered":"2024-04-16T16:16:41.052332+0000","last_clean":"2024-04-16T16:16:41.052332+0000","last_became_active":"2024-04-16T16:16:39.050841+0000","last_became_peered":"2024-04-16T16:16:39.050841+0000","last_unstale":"2024-04-16T16:16:41.052332+0000","last_undegraded":"2024-04-16T16:16:41.052332+0000","last_fullsized":"2024-04-16T16:16:41.052332+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:32:24.457101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354741+0000","last_change":"2024-04-16T16:16:40.062944+0000","last_active":"2024-04-16T16:16:41.354741+0000","last_peered":"2024-04-16T16:16:41.354741+0000","last_clean":"2024-04-16T16:16:41.354741+0000","last_became_active":"2024-04-16T16:16:40.057251+0000","last_became_peered":"2024-04-16T16:16:40.057251+0000","last_unstale":"2024-04-16T16:16:41.354741+0000","last_undegraded":"2024-04-16T16:16:41.354741+0000","last_fullsized":"2024-04-16T16:16:41.354741+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:12:52.888262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354394+0000","last_change":"2024-04-16T16:16:39.046999+0000","last_active":"2024-04-16T16:16:41.354394+0000","last_peered":"2024-04-16T16:16:41.354394+0000","last_clean":"2024-04-16T16:16:41.354394+0000","last_became_active":"2024-04-16T16:16:39.046741+0000","last_became_peered":"2024-04-16T16:16:39.046741+0000","last_unstale":"2024-04-16T16:16:41.354394+0000","last_undegraded":"2024-04-16T16:16:41.354394+0000","last_fullsized":"2024-04-16T16:16:41.354394+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:58:16.807985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354126+0000","last_change":"2024-04-16T16:16:40.057460+0000","last_active":"2024-04-16T16:16:41.354126+0000","last_peered":"2024-04-16T16:16:41.354126+0000","last_clean":"2024-04-16T16:16:41.354126+0000","last_became_active":"2024-04-16T16:16:40.057142+0000","last_became_peered":"2024-04-16T16:16:40.057142+0000","last_unstale":"2024-04-16T16:16:41.354126+0000","last_undegraded":"2024-04-16T16:16:41.354126+0000","last_fullsized":"2024-04-16T16:16:41.354126+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:44.350988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.055586+0000","last_change":"2024-04-16T16:16:39.053727+0000","last_active":"2024-04-16T16:16:41.055586+0000","last_peered":"2024-04-16T16:16:41.055586+0000","last_clean":"2024-04-16T16:16:41.055586+0000","last_became_active":"2024-04-16T16:16:39.053180+0000","last_became_peered":"2024-04-16T16:16:39.053180+0000","last_unstale":"2024-04-16T16:16:41.055586+0000","last_undegraded":"2024-04-16T16:16:41.055586+0000","last_fullsized":"2024-04-16T16:16:41.055586+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:54:40.017503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354595+0000","last_change":"2024-04-16T16:16:40.062043+0000","last_active":"2024-04-16T16:16:41.354595+0000","last_peered":"2024-04-16T16:16:41.354595+0000","last_clean":"2024-04-16T16:16:41.354595+0000","last_became_active":"2024-04-16T16:16:40.061740+0000","last_became_peered":"2024-04-16T16:16:40.061740+0000","last_unstale":"2024-04-16T16:16:41.354595+0000","last_undegraded":"2024-04-16T16:16:41.354595+0000","last_fullsized":"2024-04-16T16:16:41.354595+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:54:32.532363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.355921+0000","last_change":"2024-04-16T16:16:39.055076+0000","last_active":"2024-04-16T16:16:41.355921+0000","last_peered":"2024-04-16T16:16:41.355921+0000","last_clean":"2024-04-16T16:16:41.355921+0000","last_became_active":"2024-04-16T16:16:39.054757+0000","last_became_peered":"2024-04-16T16:16:39.054757+0000","last_unstale":"2024-04-16T16:16:41.355921+0000","last_undegraded":"2024-04-16T16:16:41.355921+0000","last_fullsized":"2024-04-16T16:16:41.355921+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:54:24.777181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.353278+0000","last_change":"2024-04-16T16:16:40.056758+0000","last_active":"2024-04-16T16:16:41.353278+0000","last_peered":"2024-04-16T16:16:41.353278+0000","last_clean":"2024-04-16T16:16:41.353278+0000","last_became_active":"2024-04-16T16:16:40.056150+0000","last_became_peered":"2024-04-16T16:16:40.056150+0000","last_unstale":"2024-04-16T16:16:41.353278+0000","last_undegraded":"2024-04-16T16:16:41.353278+0000","last_fullsized":"2024-04-16T16:16:41.353278+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:38.998393+0000","last_clean_scrub_stamp":"2024-04-16T16:16:38.998393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:24.062608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.354925+0000","last_change":"2024-04-16T16:16:39.048647+0000","last_active":"2024-04-16T16:16:41.354925+0000","last_peered":"2024-04-16T16:16:41.354925+0000","last_clean":"2024-04-16T16:16:41.354925+0000","last_became_active":"2024-04-16T16:16:39.048341+0000","last_became_peered":"2024-04-16T16:16:39.048341+0000","last_unstale":"2024-04-16T16:16:41.354925+0000","last_undegraded":"2024-04-16T16:16:41.354925+0000","last_fullsized":"2024-04-16T16:16:41.354925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:16:37.982377+0000","last_clean_scrub_stamp":"2024-04-16T16:16:37.982377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:23:45.725727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"57'139","reported_seq":351,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-16T16:16:41.054822+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:16:41.054822+0000","last_peered":"2024-04-16T16:16:41.054822+0000","last_clean":"2024-04-16T16:16:41.054822+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:16:41.054822+0000","last_undegraded":"2024-04-16T16:16:41.054822+0000","last_fullsized":"2024-04-16T16:16:41.054822+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32628,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339212,"statfs":{"total":96636764160,"available":96603353088,"internally_reserved":0,"allocated":1478656,"data_stored":1001168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":884736,"data_stored":410276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574892,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":410274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574893,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":880640,"data_stored":410254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":884736,"data_stored":410296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32048,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339792,"statfs":{"total":96636764160,"available":96603947008,"internally_reserved":0,"allocated":888832,"data_stored":411796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":897024,"data_stored":412412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574892,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1470464,"data_stored":1000552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:16:50.234 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-16T16:16:50.234 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-16T16:16:50.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.507+0000 7f141ca00640 1 Processor -- start 2024-04-16T16:16:50.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.507+0000 7f141ca00640 1 -- start start 2024-04-16T16:16:50.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.509+0000 7f141ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f141806eaf0 0x7f14180ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:50.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.509+0000 7f141ca00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1418071e60 con 0x7f14180ceaf0 2024-04-16T16:16:50.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.509+0000 7f141ca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1418072140 con 0x7f14181452e0 2024-04-16T16:16:50.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.509+0000 7f141ca00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1418072f00 con 0x7f141806eaf0 2024-04-16T16:16:50.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.509+0000 7f1415e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f141806eaf0 0x7f14180ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:50.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.510+0000 7f1415e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f141806eaf0 0x7f14180ce5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46562/0 (socket says 172.21.15.45:46562) 2024-04-16T16:16:50.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.510+0000 7f1415e00640 1 -- 172.21.15.45:0/593578987 learned_addr learned my addr 172.21.15.45:0/593578987 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:50.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.510+0000 7f1417200640 1 --1- 172.21.15.45:0/593578987 >> v1:172.21.15.45:6789/0 conn(0x7f14180ceaf0 0x7f141815cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35766/0 (socket says 172.21.15.45:35766) 2024-04-16T16:16:50.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.510+0000 7f1415e00640 1 -- 172.21.15.45:0/593578987 >> v1:172.21.15.103:6789/0 conn(0x7f14181452e0 legacy=0x7f141806e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:50.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.510+0000 7f1415e00640 1 -- 172.21.15.45:0/593578987 >> v1:172.21.15.45:6789/0 conn(0x7f14180ceaf0 legacy=0x7f141815cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:50.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.510+0000 7f1415e00640 1 -- 172.21.15.45:0/593578987 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1418072420 con 0x7f141806eaf0 2024-04-16T16:16:50.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.511+0000 7f1415e00640 1 --2- 172.21.15.45:0/593578987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f141806eaf0 0x7f14180ce5b0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f1404009870 tx=0x7f140402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e2419ba4fd9a79 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:50.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.512+0000 7f1415400640 1 -- 172.21.15.45:0/593578987 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140402f850 con 0x7f141806eaf0 2024-04-16T16:16:50.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.512+0000 7f1415400640 1 -- 172.21.15.45:0/593578987 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f140402f9f0 con 0x7f141806eaf0 2024-04-16T16:16:50.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.512+0000 7f1415400640 1 -- 172.21.15.45:0/593578987 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140402fdc0 con 0x7f141806eaf0 2024-04-16T16:16:50.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.512+0000 7f141ca00640 1 -- 172.21.15.45:0/593578987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f141806eaf0 msgr2=0x7f14180ce5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:50.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.512+0000 7f141ca00640 1 --2- 172.21.15.45:0/593578987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f141806eaf0 0x7f14180ce5b0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f1404009870 tx=0x7f140402ba30 comp rx=0 tx=0).stop 2024-04-16T16:16:50.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.512+0000 7f141ca00640 1 -- 172.21.15.45:0/593578987 shutdown_connections 2024-04-16T16:16:50.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.512+0000 7f141ca00640 1 --2- 172.21.15.45:0/593578987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f141806eaf0 0x7f14180ce5b0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:50.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.512+0000 7f141ca00640 1 -- 172.21.15.45:0/593578987 >> 172.21.15.45:0/593578987 conn(0x7f14180b3bb0 msgr2=0x7f14180b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:50.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.513+0000 7f141ca00640 1 -- 172.21.15.45:0/593578987 shutdown_connections 2024-04-16T16:16:50.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.513+0000 7f141ca00640 1 -- 172.21.15.45:0/593578987 wait complete. 2024-04-16T16:16:50.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.514+0000 7f141ca00640 1 Processor -- start 2024-04-16T16:16:50.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.515+0000 7f141ca00640 1 -- start start 2024-04-16T16:16:50.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.515+0000 7f141ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f141806eaf0 0x7f141815bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:50.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.515+0000 7f141ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14180ceaf0 0x7f1418143f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:50.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.516+0000 7f141ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14181452e0 0x7f14181444c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:50.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.516+0000 7f141ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f14180cef60 con 0x7f14181452e0 2024-04-16T16:16:50.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.516+0000 7f141ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f14180729e0 con 0x7f14180ceaf0 2024-04-16T16:16:50.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.516+0000 7f141ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f14180cf0e0 con 0x7f141806eaf0 2024-04-16T16:16:50.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.516+0000 7f1417200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14181452e0 0x7f14181444c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:50.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.516+0000 7f1417200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14181452e0 0x7f14181444c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46576/0 (socket says 172.21.15.45:46576) 2024-04-16T16:16:50.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.516+0000 7f1417200640 1 -- 172.21.15.45:0/4122929255 learned_addr learned my addr 172.21.15.45:0/4122929255 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:50.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.516+0000 7f1415e00640 1 --2- 172.21.15.45:0/4122929255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14180ceaf0 0x7f1418143f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:50.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.516+0000 7f1417200640 1 -- 172.21.15.45:0/4122929255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f141806eaf0 msgr2=0x7f141815bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:50.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.516+0000 7f1417200640 1 --2- 172.21.15.45:0/4122929255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f141806eaf0 0x7f141815bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:50.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.516+0000 7f1417200640 1 -- 172.21.15.45:0/4122929255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14180ceaf0 msgr2=0x7f1418143f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:50.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.516+0000 7f1417200640 1 --2- 172.21.15.45:0/4122929255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14180ceaf0 0x7f1418143f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:50.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.516+0000 7f1417200640 1 -- 172.21.15.45:0/4122929255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14180b03e0 con 0x7f14181452e0 2024-04-16T16:16:50.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.517+0000 7f1417200640 1 --2- 172.21.15.45:0/4122929255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14181452e0 0x7f14181444c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f141000c7b0 tx=0x7f141000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:50.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.517+0000 7f140ea00640 1 -- 172.21.15.45:0/4122929255 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1410017070 con 0x7f14181452e0 2024-04-16T16:16:50.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.517+0000 7f140ea00640 1 -- 172.21.15.45:0/4122929255 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f141000a640 con 0x7f14181452e0 2024-04-16T16:16:50.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.517+0000 7f140ea00640 1 -- 172.21.15.45:0/4122929255 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141000a9f0 con 0x7f14181452e0 2024-04-16T16:16:50.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.517+0000 7f141ca00640 1 -- 172.21.15.45:0/4122929255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141815c4e0 con 0x7f14181452e0 2024-04-16T16:16:50.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.517+0000 7f141ca00640 1 -- 172.21.15.45:0/4122929255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14180af080 con 0x7f14181452e0 2024-04-16T16:16:50.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.518+0000 7f141ca00640 1 -- 172.21.15.45:0/4122929255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1418075780 con 0x7f14181452e0 2024-04-16T16:16:50.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.519+0000 7f140ea00640 1 -- 172.21.15.45:0/4122929255 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f141001a070 con 0x7f14181452e0 2024-04-16T16:16:50.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.520+0000 7f140ea00640 1 --2- 172.21.15.45:0/4122929255 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f13f806cb30 0x7f13f806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:50.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.520+0000 7f140ea00640 1 -- 172.21.15.45:0/4122929255 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f141008b490 con 0x7f14181452e0 2024-04-16T16:16:50.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.521+0000 7f1416800640 1 --2- 172.21.15.45:0/4122929255 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f13f806cb30 0x7f13f806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:50.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.522+0000 7f1416800640 1 --2- 172.21.15.45:0/4122929255 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f13f806cb30 0x7f13f806eff0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1408006fd0 tx=0x7f1408006bc0 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:50.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.525+0000 7f140ea00640 1 -- 172.21.15.45:0/4122929255 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1410058fa0 con 0x7f14181452e0 2024-04-16T16:16:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.807+0000 7f141ca00640 1 -- 172.21.15.45:0/4122929255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f141806e600 con 0x7f14181452e0 2024-04-16T16:16:50.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.812+0000 7f140ea00640 1 -- 172.21.15.45:0/4122929255 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) ==== 190+0+147 (secure 0 0 0) 0x7f141005d2a0 con 0x7f14181452e0 2024-04-16T16:16:50.812 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-04-16T16:16:50.812 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ== 2024-04-16T16:16:50.812 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:16:50.812 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:16:50.812 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-16T16:16:50.813 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2024-04-16T16:16:50.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.815+0000 7f141ca00640 1 -- 172.21.15.45:0/4122929255 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f13f806cb30 msgr2=0x7f13f806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:50.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.815+0000 7f141ca00640 1 --2- 172.21.15.45:0/4122929255 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f13f806cb30 0x7f13f806eff0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1408006fd0 tx=0x7f1408006bc0 comp rx=0 tx=0).stop 2024-04-16T16:16:50.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.815+0000 7f141ca00640 1 -- 172.21.15.45:0/4122929255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14181452e0 msgr2=0x7f14181444c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:50.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.815+0000 7f141ca00640 1 --2- 172.21.15.45:0/4122929255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14181452e0 0x7f14181444c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f141000c7b0 tx=0x7f141000cc70 comp rx=0 tx=0).stop 2024-04-16T16:16:50.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.815+0000 7f141ca00640 1 -- 172.21.15.45:0/4122929255 shutdown_connections 2024-04-16T16:16:50.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.815+0000 7f141ca00640 1 --2- 172.21.15.45:0/4122929255 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f13f806cb30 0x7f13f806eff0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:50.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.815+0000 7f141ca00640 1 --2- 172.21.15.45:0/4122929255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14181452e0 0x7f14181444c0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:50.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.815+0000 7f141ca00640 1 --2- 172.21.15.45:0/4122929255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14180ceaf0 0x7f1418143f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:50.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.815+0000 7f141ca00640 1 --2- 172.21.15.45:0/4122929255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f141806eaf0 0x7f141815bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:50.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.815+0000 7f141ca00640 1 -- 172.21.15.45:0/4122929255 >> 172.21.15.45:0/4122929255 conn(0x7f14180b3bb0 msgr2=0x7f1418145750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:50.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.815+0000 7f141ca00640 1 -- 172.21.15.45:0/4122929255 shutdown_connections 2024-04-16T16:16:50.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:50.815+0000 7f141ca00640 1 -- 172.21.15.45:0/4122929255 wait complete. 2024-04-16T16:16:50.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:16:51.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.051+0000 7f5851000640 1 Processor -- start 2024-04-16T16:16:51.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.051+0000 7f5851000640 1 -- start start 2024-04-16T16:16:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.053+0000 7f5851000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f584c146220 0x7f584c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.053+0000 7f5851000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f584c1608a0 0x7f584c160d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.053+0000 7f5851000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f584c15d360 con 0x7f584c06f7a0 2024-04-16T16:16:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.053+0000 7f5851000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f584c160410 con 0x7f584c146220 2024-04-16T16:16:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.053+0000 7f5851000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f584c15d640 con 0x7f584c1608a0 2024-04-16T16:16:51.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.053+0000 7f584aa00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f584c06f7a0 0x7f584c06df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35778/0 (socket says 172.21.15.45:35778) 2024-04-16T16:16:51.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.053+0000 7f584aa00640 1 -- 172.21.15.45:0/1911615637 learned_addr learned my addr 172.21.15.45:0/1911615637 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:51.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584be00640 1 --2- 172.21.15.45:0/1911615637 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f584c1608a0 0x7f584c160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584b400640 1 --2- 172.21.15.45:0/1911615637 >> v2:172.21.15.103:3300/0 conn(0x7f584c146220 0x7f584c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584a000640 1 -- 172.21.15.45:0/1911615637 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3423350838 0 0) 0x7f584c15d360 con 0x7f584c06f7a0 2024-04-16T16:16:51.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584a000640 1 -- 172.21.15.45:0/1911615637 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5830003520 con 0x7f584c06f7a0 2024-04-16T16:16:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584a000640 1 -- 172.21.15.45:0/1911615637 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2348159040 0 0) 0x7f5830003520 con 0x7f584c06f7a0 2024-04-16T16:16:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584a000640 1 -- 172.21.15.45:0/1911615637 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f584c1608a0 msgr2=0x7f584c160d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584a000640 1 --2- 172.21.15.45:0/1911615637 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f584c1608a0 0x7f584c160d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584a000640 1 -- 172.21.15.45:0/1911615637 >> v2:172.21.15.103:3300/0 conn(0x7f584c146220 msgr2=0x7f584c15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584a000640 1 --2- 172.21.15.45:0/1911615637 >> v2:172.21.15.103:3300/0 conn(0x7f584c146220 0x7f584c15ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584a000640 1 -- 172.21.15.45:0/1911615637 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584c15d920 con 0x7f584c06f7a0 2024-04-16T16:16:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584a000640 1 -- 172.21.15.45:0/1911615637 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f5838001670 con 0x7f584c06f7a0 2024-04-16T16:16:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584a000640 1 -- 172.21.15.45:0/1911615637 >> v1:172.21.15.45:6789/0 conn(0x7f584c06f7a0 legacy=0x7f584c06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584a000640 1 --2- 172.21.15.45:0/1911615637 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f584c1608a0 0x7f58300039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584a000640 1 --2- 172.21.15.45:0/1911615637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5830003f30 0x7f5830004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584a000640 1 --2- 172.21.15.45:0/1911615637 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5830004d40 0x7f58300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584a000640 1 -- 172.21.15.45:0/1911615637 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f584c160410 con 0x7f5830003f30 2024-04-16T16:16:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.054+0000 7f584a000640 1 -- 172.21.15.45:0/1911615637 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f584c15d640 con 0x7f5830004d40 2024-04-16T16:16:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.055+0000 7f584a000640 1 -- 172.21.15.45:0/1911615637 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f583000b9c0 con 0x7f584c1608a0 2024-04-16T16:16:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.055+0000 7f584aa00640 1 --2- 172.21.15.45:0/1911615637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5830003f30 0x7f5830004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.055+0000 7f584b400640 1 --2- 172.21.15.45:0/1911615637 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f584c1608a0 0x7f58300039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.055+0000 7f584be00640 1 --2- 172.21.15.45:0/1911615637 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5830004d40 0x7f58300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.055+0000 7f584aa00640 1 -- 172.21.15.45:0/1911615637 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f584c1608a0 msgr2=0x7f58300039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.055+0000 7f584aa00640 1 --2- 172.21.15.45:0/1911615637 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f584c1608a0 0x7f58300039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.055+0000 7f584aa00640 1 -- 172.21.15.45:0/1911615637 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5830004d40 msgr2=0x7f58300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.055+0000 7f584aa00640 1 --2- 172.21.15.45:0/1911615637 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5830004d40 0x7f58300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.055+0000 7f584aa00640 1 -- 172.21.15.45:0/1911615637 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5830003520 con 0x7f5830003f30 2024-04-16T16:16:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.055+0000 7f584be00640 1 --2- 172.21.15.45:0/1911615637 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5830004d40 0x7f58300051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:51.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.057+0000 7f584aa00640 1 --2- 172.21.15.45:0/1911615637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5830003f30 0x7f5830004800 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f5838001050 tx=0x7f5830005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9811d03312eff94 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:51.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.057+0000 7f584a000640 1 -- 172.21.15.45:0/1911615637 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5838007d70 con 0x7f5830003f30 2024-04-16T16:16:51.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.057+0000 7f584a000640 1 -- 172.21.15.45:0/1911615637 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5838030070 con 0x7f5830003f30 2024-04-16T16:16:51.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.057+0000 7f584a000640 1 -- 172.21.15.45:0/1911615637 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583802cd70 con 0x7f5830003f30 2024-04-16T16:16:51.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.058+0000 7f5851000640 1 -- 172.21.15.45:0/1911615637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5830003f30 msgr2=0x7f5830004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:51.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.058+0000 7f5851000640 1 --2- 172.21.15.45:0/1911615637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5830003f30 0x7f5830004800 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f5838001050 tx=0x7f5830005830 comp rx=0 tx=0).stop 2024-04-16T16:16:51.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.058+0000 7f584be00640 1 -- 172.21.15.45:0/1911615637 reap_dead start 2024-04-16T16:16:51.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.058+0000 7f5851000640 1 -- 172.21.15.45:0/1911615637 shutdown_connections 2024-04-16T16:16:51.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.058+0000 7f5851000640 1 -- 172.21.15.45:0/1911615637 >> 172.21.15.45:0/1911615637 conn(0x7f584c06b930 msgr2=0x7f584c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:51.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.058+0000 7f5851000640 1 -- 172.21.15.45:0/1911615637 shutdown_connections 2024-04-16T16:16:51.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.058+0000 7f5851000640 1 -- 172.21.15.45:0/1911615637 wait complete. 2024-04-16T16:16:51.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.060+0000 7f5851000640 1 Processor -- start 2024-04-16T16:16:51.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.060+0000 7f5851000640 1 -- start start 2024-04-16T16:16:51.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.061+0000 7f5851000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5830003f30 0x7f584c1421f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.061+0000 7f5851000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f584c142730 0x7f584c15f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.061+0000 7f5851000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f584c15bf10 0x7f584c15f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.061+0000 7f5851000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f584c06fc20 con 0x7f5830003f30 2024-04-16T16:16:51.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.061+0000 7f5851000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f584c15dc00 con 0x7f584c15bf10 2024-04-16T16:16:51.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.061+0000 7f5851000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f584c06fda0 con 0x7f584c142730 2024-04-16T16:16:51.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.061+0000 7f584b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5830003f30 0x7f584c1421f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.061+0000 7f584b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5830003f30 0x7f584c1421f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46596/0 (socket says 172.21.15.45:46596) 2024-04-16T16:16:51.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.061+0000 7f584b400640 1 -- 172.21.15.45:0/3811389946 learned_addr learned my addr 172.21.15.45:0/3811389946 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:51.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.062+0000 7f584aa00640 1 --2- 172.21.15.45:0/3811389946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f584c142730 0x7f584c15f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.062+0000 7f584be00640 1 --2- 172.21.15.45:0/3811389946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f584c15bf10 0x7f584c15f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.062+0000 7f584b400640 1 -- 172.21.15.45:0/3811389946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f584c142730 msgr2=0x7f584c15f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:51.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.062+0000 7f584b400640 1 --2- 172.21.15.45:0/3811389946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f584c142730 0x7f584c15f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.062+0000 7f584b400640 1 -- 172.21.15.45:0/3811389946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f584c15bf10 msgr2=0x7f584c15f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:51.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.062+0000 7f584b400640 1 --2- 172.21.15.45:0/3811389946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f584c15bf10 0x7f584c15f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.062+0000 7f584b400640 1 -- 172.21.15.45:0/3811389946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584c13ac70 con 0x7f5830003f30 2024-04-16T16:16:51.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.062+0000 7f584b400640 1 --2- 172.21.15.45:0/3811389946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5830003f30 0x7f584c1421f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f583c011660 tx=0x7f583c011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:51.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.062+0000 7f5843400640 1 -- 172.21.15.45:0/3811389946 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c00f960 con 0x7f5830003f30 2024-04-16T16:16:51.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.062+0000 7f5843400640 1 -- 172.21.15.45:0/3811389946 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f583c00fb00 con 0x7f5830003f30 2024-04-16T16:16:51.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.062+0000 7f5843400640 1 -- 172.21.15.45:0/3811389946 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c015460 con 0x7f5830003f30 2024-04-16T16:16:51.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.062+0000 7f5851000640 1 -- 172.21.15.45:0/3811389946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f584c13af50 con 0x7f5830003f30 2024-04-16T16:16:51.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.063+0000 7f5851000640 1 -- 172.21.15.45:0/3811389946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f584c13b510 con 0x7f5830003f30 2024-04-16T16:16:51.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.064+0000 7f5851000640 1 -- 172.21.15.45:0/3811389946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f584c075780 con 0x7f5830003f30 2024-04-16T16:16:51.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.065+0000 7f5843400640 1 -- 172.21.15.45:0/3811389946 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f583c015600 con 0x7f5830003f30 2024-04-16T16:16:51.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.066+0000 7f5843400640 1 --2- 172.21.15.45:0/3811389946 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f582c06cb30 0x7f582c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.066+0000 7f5843400640 1 -- 172.21.15.45:0/3811389946 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f583c08c910 con 0x7f5830003f30 2024-04-16T16:16:51.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.066+0000 7f584aa00640 1 --2- 172.21.15.45:0/3811389946 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f582c06cb30 0x7f582c06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.067+0000 7f584aa00640 1 --2- 172.21.15.45:0/3811389946 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f582c06cb30 0x7f582c06eff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5838034000 tx=0x7f5838007000 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:51.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.071+0000 7f5843400640 1 -- 172.21.15.45:0/3811389946 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f583c019030 con 0x7f5830003f30 2024-04-16T16:16:51.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.316+0000 7f5851000640 1 -- 172.21.15.45:0/3811389946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f584c0b42d0 con 0x7f5830003f30 2024-04-16T16:16:51.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.318+0000 7f5843400640 1 -- 172.21.15.45:0/3811389946 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5180 (secure 0 0 0) 0x7f583c05a9d0 con 0x7f5830003f30 2024-04-16T16:16:51.318 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:51.318 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":50,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":6399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:16:42.595621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-04-16T16:16:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 50 2024-04-16T16:16:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.320+0000 7f5851000640 1 -- 172.21.15.45:0/3811389946 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f582c06cb30 msgr2=0x7f582c06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.320+0000 7f5851000640 1 --2- 172.21.15.45:0/3811389946 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f582c06cb30 0x7f582c06eff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5838034000 tx=0x7f5838007000 comp rx=0 tx=0).stop 2024-04-16T16:16:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.320+0000 7f5851000640 1 -- 172.21.15.45:0/3811389946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5830003f30 msgr2=0x7f584c1421f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.320+0000 7f5851000640 1 --2- 172.21.15.45:0/3811389946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5830003f30 0x7f584c1421f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f583c011660 tx=0x7f583c011b20 comp rx=0 tx=0).stop 2024-04-16T16:16:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.320+0000 7f5851000640 1 -- 172.21.15.45:0/3811389946 shutdown_connections 2024-04-16T16:16:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.320+0000 7f5851000640 1 --2- 172.21.15.45:0/3811389946 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f582c06cb30 0x7f582c06eff0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.320+0000 7f5851000640 1 --2- 172.21.15.45:0/3811389946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f584c15bf10 0x7f584c15f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.321+0000 7f5851000640 1 --2- 172.21.15.45:0/3811389946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f584c142730 0x7f584c15f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.321+0000 7f5851000640 1 --2- 172.21.15.45:0/3811389946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5830003f30 0x7f584c1421f0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.321+0000 7f5851000640 1 -- 172.21.15.45:0/3811389946 >> 172.21.15.45:0/3811389946 conn(0x7f584c06b930 msgr2=0x7f584c13e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.321+0000 7f5851000640 1 -- 172.21.15.45:0/3811389946 shutdown_connections 2024-04-16T16:16:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.321+0000 7f5851000640 1 -- 172.21.15.45:0/3811389946 wait complete. 2024-04-16T16:16:51.744 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:16:39.513364+0000', 'modified': '2024-04-16T16:16:42.595621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6402}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6402': {'gid': 6402, 'name': 'a', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6835/268877398', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6833', 'nonce': 268877398}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 268877398}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6402, 'qdb_cluster': [6402]} 2024-04-16T16:16:51.744 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-16T16:16:51.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-16T16:16:51.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.982+0000 7fc939e00640 1 Processor -- start 2024-04-16T16:16:51.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.982+0000 7fc939e00640 1 -- start start 2024-04-16T16:16:51.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.984+0000 7fc939e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc9340cfc20 0x7fc9340d37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.984+0000 7fc939e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9340d2600 con 0x7fc934162090 2024-04-16T16:16:51.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.984+0000 7fc939e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9340ba5d0 con 0x7fc9340ba140 2024-04-16T16:16:51.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.984+0000 7fc939e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc9340ba8b0 con 0x7fc9340cfc20 2024-04-16T16:16:51.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.984+0000 7fc938a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc934162090 0x7fc9340cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35780/0 (socket says 172.21.15.45:35780) 2024-04-16T16:16:51.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.984+0000 7fc938a00640 1 -- 172.21.15.45:0/1881142787 learned_addr learned my addr 172.21.15.45:0/1881142787 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:51.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.985+0000 7fc939400640 1 --2- 172.21.15.45:0/1881142787 >> v2:172.21.15.103:3300/0 conn(0x7fc9340cfc20 0x7fc9340d37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.985+0000 7fc933400640 1 -- 172.21.15.45:0/1881142787 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 503025787 0 0) 0x7fc9340d2600 con 0x7fc934162090 2024-04-16T16:16:51.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.985+0000 7fc933400640 1 -- 172.21.15.45:0/1881142787 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc91c003520 con 0x7fc934162090 2024-04-16T16:16:51.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.985+0000 7fc933400640 1 -- 172.21.15.45:0/1881142787 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1011402596 0 0) 0x7fc91c003520 con 0x7fc934162090 2024-04-16T16:16:51.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.985+0000 7fc933400640 1 -- 172.21.15.45:0/1881142787 >> v2:172.21.15.103:3300/0 conn(0x7fc9340cfc20 msgr2=0x7fc9340d37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:51.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.985+0000 7fc933400640 1 --2- 172.21.15.45:0/1881142787 >> v2:172.21.15.103:3300/0 conn(0x7fc9340cfc20 0x7fc9340d37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.985+0000 7fc933400640 1 -- 172.21.15.45:0/1881142787 >> v1:172.21.15.103:6789/0 conn(0x7fc9340ba140 legacy=0x7fc9340d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:51.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.985+0000 7fc933400640 1 -- 172.21.15.45:0/1881142787 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9340d28a0 con 0x7fc934162090 2024-04-16T16:16:51.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc933400640 1 -- 172.21.15.45:0/1881142787 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fc928001690 con 0x7fc934162090 2024-04-16T16:16:51.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc933400640 1 -- 172.21.15.45:0/1881142787 >> v1:172.21.15.45:6789/0 conn(0x7fc934162090 legacy=0x7fc9340cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:51.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc933400640 1 --2- 172.21.15.45:0/1881142787 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc91c003a00 0x7fc91c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc933400640 1 --2- 172.21.15.45:0/1881142787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc91c004800 0x7fc91c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc933400640 1 --2- 172.21.15.45:0/1881142787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc91c0051c0 0x7fc91c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc933400640 1 -- 172.21.15.45:0/1881142787 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc9340ba8b0 con 0x7fc91c004800 2024-04-16T16:16:51.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc933400640 1 -- 172.21.15.45:0/1881142787 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc91c00c0c0 con 0x7fc91c0051c0 2024-04-16T16:16:51.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc933400640 1 -- 172.21.15.45:0/1881142787 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc91c00c2c0 con 0x7fc91c003a00 2024-04-16T16:16:51.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc933e00640 1 --2- 172.21.15.45:0/1881142787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc91c004800 0x7fc91c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc938a00640 1 --2- 172.21.15.45:0/1881142787 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc91c003a00 0x7fc91c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc933e00640 1 -- 172.21.15.45:0/1881142787 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc91c003a00 msgr2=0x7fc91c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:51.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc933e00640 1 --2- 172.21.15.45:0/1881142787 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc91c003a00 0x7fc91c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc939400640 1 --2- 172.21.15.45:0/1881142787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc91c0051c0 0x7fc91c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc933e00640 1 -- 172.21.15.45:0/1881142787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc91c0051c0 msgr2=0x7fc91c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:51.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc933e00640 1 --2- 172.21.15.45:0/1881142787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc91c0051c0 0x7fc91c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.986+0000 7fc933e00640 1 -- 172.21.15.45:0/1881142787 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc91c003520 con 0x7fc91c004800 2024-04-16T16:16:51.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.987+0000 7fc939400640 1 -- 172.21.15.45:0/1881142787 reap_dead start 2024-04-16T16:16:51.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.987+0000 7fc938a00640 1 --2- 172.21.15.45:0/1881142787 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc91c003a00 0x7fc91c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.988+0000 7fc933e00640 1 --2- 172.21.15.45:0/1881142787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc91c004800 0x7fc91c004c80 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc920001750 tx=0x7fc91c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffc799cd136d06f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.988+0000 7fc933400640 1 -- 172.21.15.45:0/1881142787 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc920001370 con 0x7fc91c004800 2024-04-16T16:16:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.988+0000 7fc933400640 1 -- 172.21.15.45:0/1881142787 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9200052d0 con 0x7fc91c004800 2024-04-16T16:16:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.988+0000 7fc933400640 1 -- 172.21.15.45:0/1881142787 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc920003830 con 0x7fc91c004800 2024-04-16T16:16:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.988+0000 7fc939e00640 1 -- 172.21.15.45:0/1881142787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc91c004800 msgr2=0x7fc91c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.988+0000 7fc939e00640 1 --2- 172.21.15.45:0/1881142787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc91c004800 0x7fc91c004c80 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc920001750 tx=0x7fc91c005830 comp rx=0 tx=0).stop 2024-04-16T16:16:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.989+0000 7fc939e00640 1 -- 172.21.15.45:0/1881142787 shutdown_connections 2024-04-16T16:16:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.989+0000 7fc939e00640 1 --2- 172.21.15.45:0/1881142787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc91c004800 0x7fc91c004c80 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.989+0000 7fc939e00640 1 -- 172.21.15.45:0/1881142787 >> 172.21.15.45:0/1881142787 conn(0x7fc9340d1940 msgr2=0x7fc9340d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:51.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.989+0000 7fc939e00640 1 -- 172.21.15.45:0/1881142787 shutdown_connections 2024-04-16T16:16:51.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.989+0000 7fc939e00640 1 -- 172.21.15.45:0/1881142787 wait complete. 2024-04-16T16:16:51.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.991+0000 7fc939e00640 1 Processor -- start 2024-04-16T16:16:51.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.991+0000 7fc939e00640 1 -- start start 2024-04-16T16:16:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.992+0000 7fc939e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc91c004800 0x7fc9340abb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.992+0000 7fc939e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9340ab3d0 0x7fc934153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.992+0000 7fc939e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9341f84a0 0x7fc9341f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.992+0000 7fc939e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc9341625f0 con 0x7fc9341f84a0 2024-04-16T16:16:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.992+0000 7fc939e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc934162470 con 0x7fc9340ab3d0 2024-04-16T16:16:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.992+0000 7fc939e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc9340d5520 con 0x7fc91c004800 2024-04-16T16:16:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.992+0000 7fc939400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9341f84a0 0x7fc9341f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.992+0000 7fc939400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9341f84a0 0x7fc9341f8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46620/0 (socket says 172.21.15.45:46620) 2024-04-16T16:16:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.992+0000 7fc939400640 1 -- 172.21.15.45:0/187954757 learned_addr learned my addr 172.21.15.45:0/187954757 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.993+0000 7fc933e00640 1 --2- 172.21.15.45:0/187954757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9340ab3d0 0x7fc934153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.993+0000 7fc938a00640 1 --2- 172.21.15.45:0/187954757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc91c004800 0x7fc9340abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.993+0000 7fc939400640 1 -- 172.21.15.45:0/187954757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc91c004800 msgr2=0x7fc9340abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.993+0000 7fc939400640 1 --2- 172.21.15.45:0/187954757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc91c004800 0x7fc9340abb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.993+0000 7fc939400640 1 -- 172.21.15.45:0/187954757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9340ab3d0 msgr2=0x7fc934153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.993+0000 7fc939400640 1 --2- 172.21.15.45:0/187954757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9340ab3d0 0x7fc934153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.993+0000 7fc939400640 1 -- 172.21.15.45:0/187954757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9340ac0c0 con 0x7fc9341f84a0 2024-04-16T16:16:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.993+0000 7fc933e00640 1 --2- 172.21.15.45:0/187954757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9340ab3d0 0x7fc934153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.993+0000 7fc939400640 1 --2- 172.21.15.45:0/187954757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9341f84a0 0x7fc9341f8930 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc9340d2300 tx=0x7fc92c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.993+0000 7fc930c00640 1 -- 172.21.15.45:0/187954757 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c016420 con 0x7fc9341f84a0 2024-04-16T16:16:51.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.994+0000 7fc930c00640 1 -- 172.21.15.45:0/187954757 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc92c01b070 con 0x7fc9341f84a0 2024-04-16T16:16:51.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.994+0000 7fc930c00640 1 -- 172.21.15.45:0/187954757 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c016820 con 0x7fc9341f84a0 2024-04-16T16:16:51.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.994+0000 7fc939e00640 1 -- 172.21.15.45:0/187954757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9340cc900 con 0x7fc9341f84a0 2024-04-16T16:16:51.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.994+0000 7fc939e00640 1 -- 172.21.15.45:0/187954757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9340ccec0 con 0x7fc9341f84a0 2024-04-16T16:16:51.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.996+0000 7fc930c00640 1 -- 172.21.15.45:0/187954757 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fc92c0169c0 con 0x7fc9341f84a0 2024-04-16T16:16:51.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.997+0000 7fc930c00640 1 --2- 172.21.15.45:0/187954757 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc91406cb60 0x7fc91406f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.997+0000 7fc930c00640 1 -- 172.21.15.45:0/187954757 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7fc92c08cfe0 con 0x7fc9341f84a0 2024-04-16T16:16:51.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.998+0000 7fc93e19d740 1 Processor -- start 2024-04-16T16:16:51.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.998+0000 7fc938a00640 1 --2- 172.21.15.45:0/187954757 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc91406cb60 0x7fc91406f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:51.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.998+0000 7fc93e19d740 1 -- start start 2024-04-16T16:16:51.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc93e19d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c0469f890 0x559c046be9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc93e19d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c04678570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:51.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc93e19d740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c04678ab0 0x559c04676440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc93e19d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559c045dd9c0 con 0x559c0469fc70 2024-04-16T16:16:52.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc93e19d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559c04584560 con 0x559c0469f890 2024-04-16T16:16:52.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc93e19d740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x559c045d9980 con 0x559c04678ab0 2024-04-16T16:16:52.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc933e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c04678570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:52.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc933e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c04678570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46628/0 (socket says 172.21.15.45:46628) 2024-04-16T16:16:52.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc933e00640 1 -- 172.21.15.45:0/1372829896 learned_addr learned my addr 172.21.15.45:0/1372829896 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:52.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc938a00640 1 --2- 172.21.15.45:0/187954757 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc91406cb60 0x7fc91406f020 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc928001eb0 tx=0x7fc928009040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:52.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc938a00640 1 --2- 172.21.15.45:0/1372829896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c0469f890 0x559c046be9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:52.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc933e00640 1 -- 172.21.15.45:0/1372829896 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c04678ab0 msgr2=0x559c04676440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:52.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc933e00640 1 --2- 172.21.15.45:0/1372829896 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c04678ab0 0x559c04676440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc933e00640 1 -- 172.21.15.45:0/1372829896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c0469f890 msgr2=0x559c046be9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:52.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc933e00640 1 --2- 172.21.15.45:0/1372829896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c0469f890 0x559c046be9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:51.999+0000 7fc933e00640 1 -- 172.21.15.45:0/1372829896 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c042f02d0 con 0x559c0469fc70 2024-04-16T16:16:52.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.000+0000 7fc933e00640 1 --2- 172.21.15.45:0/1372829896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c04678570 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc92002bc50 tx=0x7fc92002f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba84f0667174bda4 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:52.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.000+0000 7fc925200640 1 -- 172.21.15.45:0/1372829896 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc920005450 con 0x559c0469fc70 2024-04-16T16:16:52.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.000+0000 7fc925200640 1 -- 172.21.15.45:0/1372829896 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc920043070 con 0x559c0469fc70 2024-04-16T16:16:52.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.000+0000 7fc925200640 1 -- 172.21.15.45:0/1372829896 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc920007240 con 0x559c0469fc70 2024-04-16T16:16:52.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.001+0000 7fc93e19d740 1 -- 172.21.15.45:0/1372829896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 msgr2=0x559c04678570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:52.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.001+0000 7fc93e19d740 1 --2- 172.21.15.45:0/1372829896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c04678570 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc92002bc50 tx=0x7fc92002f000 comp rx=0 tx=0).stop 2024-04-16T16:16:52.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.001+0000 7fc93e19d740 1 -- 172.21.15.45:0/1372829896 shutdown_connections 2024-04-16T16:16:52.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.001+0000 7fc93e19d740 1 --2- 172.21.15.45:0/1372829896 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c04678ab0 0x559c04676440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.001+0000 7fc93e19d740 1 --2- 172.21.15.45:0/1372829896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c04678570 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.001+0000 7fc93e19d740 1 --2- 172.21.15.45:0/1372829896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c0469f890 0x559c046be9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.001+0000 7fc93e19d740 1 -- 172.21.15.45:0/1372829896 >> 172.21.15.45:0/1372829896 conn(0x559c04674b50 msgr2=0x559c046bd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:52.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.001+0000 7fc93e19d740 1 -- 172.21.15.45:0/1372829896 shutdown_connections 2024-04-16T16:16:52.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.001+0000 7fc93e19d740 1 -- 172.21.15.45:0/1372829896 wait complete. 2024-04-16T16:16:52.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.002+0000 7fc93e19d740 1 -- start start 2024-04-16T16:16:52.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.002+0000 7fc93e19d740 1 Processor -- start 2024-04-16T16:16:52.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.003+0000 7fc93e19d740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c0469f890 0x559c04699c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.003+0000 7fc93e19d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c046bb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.003+0000 7fc93e19d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c046b8860 0x559c046c01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.003+0000 7fc93e19d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559c04672980 con 0x559c0469fc70 2024-04-16T16:16:52.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.003+0000 7fc93e19d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559c04597920 con 0x559c046b8860 2024-04-16T16:16:52.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.003+0000 7fc93e19d740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x559c04678f30 con 0x559c0469f890 2024-04-16T16:16:52.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.003+0000 7fc93e19d740 10 client.0 _ll_register_callbacks cb 0x559c04597310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:16:52.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.003+0000 7fc933e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c046bb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:52.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.003+0000 7fc933e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c046bb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46630/0 (socket says 172.21.15.45:46630) 2024-04-16T16:16:52.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.003+0000 7fc933e00640 1 -- 172.21.15.45:0/1233504766 learned_addr learned my addr 172.21.15.45:0/1233504766 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:52.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.004+0000 7fc939400640 1 --2- 172.21.15.45:0/1233504766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c046b8860 0x559c046c01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:52.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.004+0000 7fc938a00640 1 --2- 172.21.15.45:0/1233504766 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c0469f890 0x559c04699c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:52.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.004+0000 7fc933e00640 1 -- 172.21.15.45:0/1233504766 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c0469f890 msgr2=0x559c04699c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:52.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.004+0000 7fc933e00640 1 --2- 172.21.15.45:0/1233504766 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c0469f890 0x559c04699c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.004+0000 7fc933e00640 1 -- 172.21.15.45:0/1233504766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c046b8860 msgr2=0x559c046c01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:52.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.004+0000 7fc933e00640 1 --2- 172.21.15.45:0/1233504766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c046b8860 0x559c046c01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.004+0000 7fc933e00640 1 -- 172.21.15.45:0/1233504766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c043ba6b0 con 0x559c0469fc70 2024-04-16T16:16:52.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.004+0000 7fc939400640 1 --2- 172.21.15.45:0/1233504766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c046b8860 0x559c046c01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:52.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.004+0000 7fc933e00640 1 --2- 172.21.15.45:0/1233504766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c046bb340 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fc920001840 tx=0x7fc920001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddec95cdb70f2bad server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:52.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.004+0000 7fc919600640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:16:52.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.004+0000 7fc919600640 1 -- 172.21.15.45:0/1233504766 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc920007240 con 0x559c0469fc70 2024-04-16T16:16:52.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.005+0000 7fc919600640 1 -- 172.21.15.45:0/1233504766 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7fc9200073e0 con 0x559c0469fc70 2024-04-16T16:16:52.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.005+0000 7fc93e19d740 1 -- 172.21.15.45:0/1233504766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc920043070 con 0x559c0469fc70 2024-04-16T16:16:52.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.005+0000 7fc919600640 1 -- 172.21.15.45:0/1233504766 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fc920057050 con 0x559c0469fc70 2024-04-16T16:16:52.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.005+0000 7fc93e19d740 10 client.6567 fetch_fsmap learned FSMap version 50 2024-04-16T16:16:52.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.005+0000 7fc93e19d740 1 -- 172.21.15.45:0/1233504766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) -- 0x559c046bc1c0 con 0x559c0469fc70 2024-04-16T16:16:52.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.005+0000 7fc919600640 1 -- 172.21.15.45:0/1233504766 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc920043070 con 0x559c0469fc70 2024-04-16T16:16:52.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.005+0000 7fc919600640 1 -- 172.21.15.45:0/1233504766 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 50) ==== 3494+0+0 (secure 0 0 0) 0x7fc92005a020 con 0x559c0469fc70 2024-04-16T16:16:52.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.005+0000 7fc93e19d740 10 client.6567 fetch_fsmap finished waiting for FSMap version 50 2024-04-16T16:16:52.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.005+0000 7fc93e19d740 10 client.6567 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-16T16:16:52.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.005+0000 7fc93e19d740 20 client.6567 populate_metadata read hostname 'smithi045' 2024-04-16T16:16:52.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.005+0000 7fc93e19d740 1 --2- 172.21.15.45:0/1233504766 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x559c046b99e0 0x559c046b9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.005+0000 7fc93e19d740 4 client.6567 mds_command: new command op to 6402 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.005+0000 7fc93e19d740 1 -- 172.21.15.45:0/1233504766 --> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c04583b70 con 0x559c046b99e0 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.006+0000 7fc938a00640 1 --2- 172.21.15.45:0/1233504766 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x559c046b99e0 0x559c046b9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.006+0000 7fc938a00640 1 --2- 172.21.15.45:0/1233504766 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x559c046b99e0 0x559c046b9dc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f970ff98ed90554 server_cookie=97cda2779b96844c in_seq=0 out_seq=0 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.006+0000 7fc919600640 10 client.6567 ms_handle_connect on v2:172.21.15.45:6833/268877398 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.007+0000 7fc919600640 1 -- 172.21.15.45:0/1233504766 <== mds.0 v2:172.21.15.45:6833/268877398 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559c04583b70 con 0x559c046b99e0 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.007+0000 7fc919600640 10 client.6567 handle_command_reply: tid=0 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.007+0000 7fc93e19d740 1 client.6567 shutdown 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.008+0000 7fc93e19d740 1 -- 172.21.15.45:0/1233504766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 msgr2=0x559c046bb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.008+0000 7fc93e19d740 1 --2- 172.21.15.45:0/1233504766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c046bb340 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fc920001840 tx=0x7fc920001190 comp rx=0 tx=0).stop 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.008+0000 7fc93e19d740 1 -- 172.21.15.45:0/1233504766 shutdown_connections 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.008+0000 7fc93e19d740 1 --2- 172.21.15.45:0/1233504766 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x559c046b99e0 0x559c046b9dc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.008+0000 7fc93e19d740 1 --2- 172.21.15.45:0/1233504766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c046b8860 0x559c046c01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.008+0000 7fc93e19d740 1 --2- 172.21.15.45:0/1233504766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c046bb340 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.008+0000 7fc93e19d740 1 --2- 172.21.15.45:0/1233504766 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c0469f890 0x559c04699c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.008+0000 7fc919600640 0 client.6567 ms_handle_reset on v2:172.21.15.45:6833/268877398 2024-04-16T16:16:52.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.008+0000 7fc93e19d740 1 -- 172.21.15.45:0/1233504766 >> 172.21.15.45:0/1233504766 conn(0x559c04678ab0 msgr2=0x559c0469cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:52.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.008+0000 7fc93e19d740 1 -- 172.21.15.45:0/1233504766 shutdown_connections 2024-04-16T16:16:52.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.008+0000 7fc93e19d740 1 -- 172.21.15.45:0/1233504766 wait complete. 2024-04-16T16:16:52.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.008+0000 7fc93e19d740 20 client.6567 trim_cache size 0 max 16384 2024-04-16T16:16:52.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.028+0000 7fc93e19d740 1 Processor -- start 2024-04-16T16:16:52.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.028+0000 7fc93e19d740 1 -- start start 2024-04-16T16:16:52.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.029+0000 7fc93e19d740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c0469f890 0x559c046a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.029+0000 7fc93e19d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c0469e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.029+0000 7fc93e19d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c046b8860 0x559c0469ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.029+0000 7fc93e19d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc934200810 con 0x559c0469fc70 2024-04-16T16:16:52.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.029+0000 7fc93e19d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc934160000 con 0x559c046b8860 2024-04-16T16:16:52.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.029+0000 7fc93e19d740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc9340ce680 con 0x559c0469f890 2024-04-16T16:16:52.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.030+0000 7fc933e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c0469e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:52.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.030+0000 7fc933e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c0469e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46632/0 (socket says 172.21.15.45:46632) 2024-04-16T16:16:52.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.030+0000 7fc933e00640 1 -- 172.21.15.45:0/3961599144 learned_addr learned my addr 172.21.15.45:0/3961599144 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:52.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.030+0000 7fc933e00640 1 -- 172.21.15.45:0/3961599144 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c0469f890 msgr2=0x559c046a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:52.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.030+0000 7fc933e00640 1 --2- 172.21.15.45:0/3961599144 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c0469f890 0x559c046a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.030+0000 7fc933e00640 1 -- 172.21.15.45:0/3961599144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c046b8860 msgr2=0x559c0469ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:52.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.030+0000 7fc933e00640 1 --2- 172.21.15.45:0/3961599144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c046b8860 0x559c0469ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.030+0000 7fc933e00640 1 -- 172.21.15.45:0/3961599144 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c0469a480 con 0x559c0469fc70 2024-04-16T16:16:52.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.030+0000 7fc933e00640 1 --2- 172.21.15.45:0/3961599144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c0469e6f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fc920007500 tx=0x7fc92004b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ff45e0648c0e3cf server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:52.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.031+0000 7fc91be00640 1 -- 172.21.15.45:0/3961599144 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc920007660 con 0x559c0469fc70 2024-04-16T16:16:52.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.031+0000 7fc91be00640 1 -- 172.21.15.45:0/3961599144 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc92004d020 con 0x559c0469fc70 2024-04-16T16:16:52.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.031+0000 7fc91be00640 1 -- 172.21.15.45:0/3961599144 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc920006a70 con 0x559c0469fc70 2024-04-16T16:16:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.031+0000 7fc93e19d740 1 -- 172.21.15.45:0/3961599144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 msgr2=0x559c0469e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.031+0000 7fc93e19d740 1 --2- 172.21.15.45:0/3961599144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c0469e6f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fc920007500 tx=0x7fc92004b070 comp rx=0 tx=0).stop 2024-04-16T16:16:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.031+0000 7fc93e19d740 1 -- 172.21.15.45:0/3961599144 shutdown_connections 2024-04-16T16:16:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.031+0000 7fc93e19d740 1 --2- 172.21.15.45:0/3961599144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c046b8860 0x559c0469ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.031+0000 7fc93e19d740 1 --2- 172.21.15.45:0/3961599144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c0469fc70 0x559c0469e6f0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.031+0000 7fc93e19d740 1 --2- 172.21.15.45:0/3961599144 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c0469f890 0x559c046a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.031+0000 7fc93e19d740 1 -- 172.21.15.45:0/3961599144 >> 172.21.15.45:0/3961599144 conn(0x559c04678ab0 msgr2=0x559c046b9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.031+0000 7fc93e19d740 1 -- 172.21.15.45:0/3961599144 shutdown_connections 2024-04-16T16:16:52.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.031+0000 7fc93e19d740 1 -- 172.21.15.45:0/3961599144 wait complete. 2024-04-16T16:16:52.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.032+0000 7fc93e19d740 1 -- start start 2024-04-16T16:16:52.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.032+0000 7fc93e19d740 1 Processor -- start 2024-04-16T16:16:52.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.033+0000 7fc93e19d740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c0469f890 0x559c04676440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.033+0000 7fc93e19d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c0469fc70 0x559c046a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.033+0000 7fc93e19d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c046b8860 0x559c046d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.033+0000 7fc93e19d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559c04672980 con 0x559c046b8860 2024-04-16T16:16:52.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.033+0000 7fc93e19d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559c04597920 con 0x559c0469fc70 2024-04-16T16:16:52.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.033+0000 7fc93e19d740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x559c046d12c0 con 0x559c0469f890 2024-04-16T16:16:52.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.033+0000 7fc93e19d740 10 client.0 _ll_register_callbacks cb 0x7fc9200015f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:16:52.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.034+0000 7fc939400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c046b8860 0x559c046d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:52.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.034+0000 7fc939400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c046b8860 0x559c046d00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46636/0 (socket says 172.21.15.45:46636) 2024-04-16T16:16:52.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.034+0000 7fc939400640 1 -- 172.21.15.45:0/3312330263 learned_addr learned my addr 172.21.15.45:0/3312330263 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:52.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.034+0000 7fc938a00640 1 --2- 172.21.15.45:0/3312330263 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c0469f890 0x559c04676440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.034+0000 7fc933e00640 1 --2- 172.21.15.45:0/3312330263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c0469fc70 0x559c046a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.034+0000 7fc939400640 1 -- 172.21.15.45:0/3312330263 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c0469f890 msgr2=0x559c04676440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.034+0000 7fc939400640 1 --2- 172.21.15.45:0/3312330263 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c0469f890 0x559c04676440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.034+0000 7fc939400640 1 -- 172.21.15.45:0/3312330263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c0469fc70 msgr2=0x559c046a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.034+0000 7fc939400640 1 --2- 172.21.15.45:0/3312330263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c0469fc70 0x559c046a02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.034+0000 7fc939400640 1 -- 172.21.15.45:0/3312330263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c046b7f30 con 0x559c046b8860 2024-04-16T16:16:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc939400640 1 --2- 172.21.15.45:0/3312330263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c046b8860 0x559c046d00a0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fc92c062ef0 tx=0x7fc92c01d310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96922dbc53ab13fb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc919600640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:16:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc919600640 1 -- 172.21.15.45:0/3312330263 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c089060 con 0x559c046b8860 2024-04-16T16:16:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc919600640 1 -- 172.21.15.45:0/3312330263 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7fc92c062f90 con 0x559c046b8860 2024-04-16T16:16:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc93e19d740 1 -- 172.21.15.45:0/3312330263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc92004d020 con 0x559c046b8860 2024-04-16T16:16:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc919600640 1 -- 172.21.15.45:0/3312330263 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fc92c05a840 con 0x559c046b8860 2024-04-16T16:16:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc93e19d740 10 client.6573 fetch_fsmap learned FSMap version 50 2024-04-16T16:16:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc93e19d740 1 -- 172.21.15.45:0/3312330263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) -- 0x559c046b71f0 con 0x559c046b8860 2024-04-16T16:16:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc919600640 1 -- 172.21.15.45:0/3312330263 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc92004d020 con 0x559c046b8860 2024-04-16T16:16:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc919600640 1 -- 172.21.15.45:0/3312330263 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 50) ==== 3494+0+0 (secure 0 0 0) 0x7fc92c071070 con 0x559c046b8860 2024-04-16T16:16:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc93e19d740 10 client.6573 fetch_fsmap finished waiting for FSMap version 50 2024-04-16T16:16:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc93e19d740 10 client.6573 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-16T16:16:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc93e19d740 20 client.6573 populate_metadata read hostname 'smithi045' 2024-04-16T16:16:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc93e19d740 1 --2- 172.21.15.45:0/3312330263 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x559c046be850 0x559c046bec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc93e19d740 4 client.6573 mds_command: new command op to 6402 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-16T16:16:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.035+0000 7fc93e19d740 1 -- 172.21.15.45:0/3312330263 --> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] -- command(tid 0: {"prefix": "status"}) -- 0x559c045eb3e0 con 0x559c046be850 2024-04-16T16:16:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.036+0000 7fc938a00640 1 --2- 172.21.15.45:0/3312330263 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x559c046be850 0x559c046bec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.036+0000 7fc938a00640 1 --2- 172.21.15.45:0/3312330263 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x559c046be850 0x559c046bec30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d73f493c1ac84c9f server_cookie=4f1a6d196378ed9c in_seq=0 out_seq=0 2024-04-16T16:16:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.036+0000 7fc919600640 10 client.6573 ms_handle_connect on v2:172.21.15.45:6833/268877398 2024-04-16T16:16:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.037+0000 7fc919600640 1 -- 172.21.15.45:0/3312330263 <== mds.0 v2:172.21.15.45:6833/268877398 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x559c045eb3e0 con 0x559c046be850 2024-04-16T16:16:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.037+0000 7fc919600640 10 client.6573 handle_command_reply: tid=0 2024-04-16T16:16:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.037+0000 7fc93e19d740 1 client.6573 shutdown 2024-04-16T16:16:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.037+0000 7fc93e19d740 1 -- 172.21.15.45:0/3312330263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c046b8860 msgr2=0x559c046d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.037+0000 7fc93e19d740 1 --2- 172.21.15.45:0/3312330263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c046b8860 0x559c046d00a0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fc92c062ef0 tx=0x7fc92c01d310 comp rx=0 tx=0).stop 2024-04-16T16:16:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.037+0000 7fc93e19d740 1 -- 172.21.15.45:0/3312330263 shutdown_connections 2024-04-16T16:16:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.037+0000 7fc93e19d740 1 --2- 172.21.15.45:0/3312330263 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x559c046be850 0x559c046bec30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.037+0000 7fc93e19d740 1 --2- 172.21.15.45:0/3312330263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c046b8860 0x559c046d00a0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.037+0000 7fc93e19d740 1 --2- 172.21.15.45:0/3312330263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c0469fc70 0x559c046a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.037+0000 7fc93e19d740 1 --2- 172.21.15.45:0/3312330263 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559c0469f890 0x559c04676440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.037+0000 7fc919600640 0 client.6573 ms_handle_reset on v2:172.21.15.45:6833/268877398 2024-04-16T16:16:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.037+0000 7fc93e19d740 1 -- 172.21.15.45:0/3312330263 >> 172.21.15.45:0/3312330263 conn(0x559c04678ab0 msgr2=0x559c046bfa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.037+0000 7fc93e19d740 1 -- 172.21.15.45:0/3312330263 shutdown_connections 2024-04-16T16:16:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.037+0000 7fc93e19d740 1 -- 172.21.15.45:0/3312330263 wait complete. 2024-04-16T16:16:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.037+0000 7fc93e19d740 20 client.6573 trim_cache size 0 max 16384 2024-04-16T16:16:52.039 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:16:52.040 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "748c4b85-0190-4c59-80ad-098a6a5007c0", 2024-04-16T16:16:52.040 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-04-16T16:16:52.040 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 6402, 2024-04-16T16:16:52.040 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-04-16T16:16:52.040 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-04-16T16:16:52.040 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-04-16T16:16:52.040 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 11.987522801000001, 2024-04-16T16:16:52.040 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 50, 2024-04-16T16:16:52.040 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 72, 2024-04-16T16:16:52.040 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 72, 2024-04-16T16:16:52.040 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 18.354863762000001 2024-04-16T16:16:52.040 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:16:52.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.038+0000 7fc939e00640 1 -- 172.21.15.45:0/187954757 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc91406cb60 msgr2=0x7fc91406f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:52.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.038+0000 7fc939e00640 1 --2- 172.21.15.45:0/187954757 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc91406cb60 0x7fc91406f020 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc928001eb0 tx=0x7fc928009040 comp rx=0 tx=0).stop 2024-04-16T16:16:52.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.038+0000 7fc939e00640 1 -- 172.21.15.45:0/187954757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9341f84a0 msgr2=0x7fc9341f8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:52.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.038+0000 7fc939e00640 1 --2- 172.21.15.45:0/187954757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9341f84a0 0x7fc9341f8930 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc9340d2300 tx=0x7fc92c00f520 comp rx=0 tx=0).stop 2024-04-16T16:16:52.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.038+0000 7fc939e00640 1 -- 172.21.15.45:0/187954757 shutdown_connections 2024-04-16T16:16:52.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.038+0000 7fc939e00640 1 --2- 172.21.15.45:0/187954757 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fc91406cb60 0x7fc91406f020 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.038+0000 7fc939e00640 1 --2- 172.21.15.45:0/187954757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9341f84a0 0x7fc9341f8930 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.038+0000 7fc939e00640 1 --2- 172.21.15.45:0/187954757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9340ab3d0 0x7fc934153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.038+0000 7fc939e00640 1 --2- 172.21.15.45:0/187954757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc91c004800 0x7fc9340abb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.038+0000 7fc939e00640 1 -- 172.21.15.45:0/187954757 >> 172.21.15.45:0/187954757 conn(0x7fc9340d1940 msgr2=0x7fc9340b4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.038+0000 7fc939e00640 1 -- 172.21.15.45:0/187954757 shutdown_connections 2024-04-16T16:16:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.038+0000 7fc939e00640 1 -- 172.21.15.45:0/187954757 wait complete. 2024-04-16T16:16:52.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:16:52.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.278+0000 7fdd36200640 1 Processor -- start 2024-04-16T16:16:52.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.278+0000 7fdd36200640 1 -- start start 2024-04-16T16:16:52.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.279+0000 7fdd36200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdd3006dfb0 0x7fdd300c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.279+0000 7fdd36200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd3015c700 con 0x7fdd30144ff0 2024-04-16T16:16:52.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.279+0000 7fdd36200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd3015c9c0 con 0x7fdd300c5bb0 2024-04-16T16:16:52.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.279+0000 7fdd36200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fdd3015d2f0 con 0x7fdd3006dfb0 2024-04-16T16:16:52.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.280+0000 7fdd2fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdd3006dfb0 0x7fdd300c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:52.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.280+0000 7fdd2fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdd3006dfb0 0x7fdd300c5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46652/0 (socket says 172.21.15.45:46652) 2024-04-16T16:16:52.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.280+0000 7fdd2fe00640 1 -- 172.21.15.45:0/3969457299 learned_addr learned my addr 172.21.15.45:0/3969457299 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:52.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.280+0000 7fdd34e00640 1 --1- 172.21.15.45:0/3969457299 >> v1:172.21.15.45:6789/0 conn(0x7fdd30144ff0 0x7fdd300cfd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35786/0 (socket says 172.21.15.45:35786) 2024-04-16T16:16:52.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.280+0000 7fdd2fe00640 1 -- 172.21.15.45:0/3969457299 >> v1:172.21.15.103:6789/0 conn(0x7fdd300c5bb0 legacy=0x7fdd3015d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:52.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.280+0000 7fdd2fe00640 1 -- 172.21.15.45:0/3969457299 >> v1:172.21.15.45:6789/0 conn(0x7fdd30144ff0 legacy=0x7fdd300cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:52.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.280+0000 7fdd2fe00640 1 -- 172.21.15.45:0/3969457299 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd3015cca0 con 0x7fdd3006dfb0 2024-04-16T16:16:52.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.281+0000 7fdd2fe00640 1 --2- 172.21.15.45:0/3969457299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd3006dfb0 0x7fdd300c5670 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009870 tx=0x7fdd1802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9afb01e1be879bd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:52.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.282+0000 7fdd2f400640 1 -- 172.21.15.45:0/3969457299 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18034070 con 0x7fdd3006dfb0 2024-04-16T16:16:52.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.282+0000 7fdd2f400640 1 -- 172.21.15.45:0/3969457299 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd18030850 con 0x7fdd3006dfb0 2024-04-16T16:16:52.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.282+0000 7fdd2f400640 1 -- 172.21.15.45:0/3969457299 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18030c20 con 0x7fdd3006dfb0 2024-04-16T16:16:52.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.282+0000 7fdd36200640 1 -- 172.21.15.45:0/3969457299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd3006dfb0 msgr2=0x7fdd300c5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:52.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.282+0000 7fdd36200640 1 --2- 172.21.15.45:0/3969457299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd3006dfb0 0x7fdd300c5670 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009870 tx=0x7fdd1802b990 comp rx=0 tx=0).stop 2024-04-16T16:16:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.282+0000 7fdd36200640 1 -- 172.21.15.45:0/3969457299 shutdown_connections 2024-04-16T16:16:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.282+0000 7fdd36200640 1 --2- 172.21.15.45:0/3969457299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd3006dfb0 0x7fdd300c5670 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.282+0000 7fdd36200640 1 -- 172.21.15.45:0/3969457299 >> 172.21.15.45:0/3969457299 conn(0x7fdd3006b930 msgr2=0x7fdd3006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.283+0000 7fdd36200640 1 -- 172.21.15.45:0/3969457299 shutdown_connections 2024-04-16T16:16:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.283+0000 7fdd36200640 1 -- 172.21.15.45:0/3969457299 wait complete. 2024-04-16T16:16:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.284+0000 7fdd36200640 1 Processor -- start 2024-04-16T16:16:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.285+0000 7fdd36200640 1 -- start start 2024-04-16T16:16:52.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.285+0000 7fdd36200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd300c5bb0 0x7fdd300a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.285+0000 7fdd36200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30144ff0 0x7fdd300aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.286+0000 7fdd36200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd301f8040 0x7fdd3015e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.286+0000 7fdd36200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3015de90 con 0x7fdd300c5bb0 2024-04-16T16:16:52.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.286+0000 7fdd36200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301453d0 con 0x7fdd30144ff0 2024-04-16T16:16:52.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.286+0000 7fdd36200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdd3015e010 con 0x7fdd301f8040 2024-04-16T16:16:52.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.286+0000 7fdd34e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd300c5bb0 0x7fdd300a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:52.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.286+0000 7fdd34e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd300c5bb0 0x7fdd300a94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46660/0 (socket says 172.21.15.45:46660) 2024-04-16T16:16:52.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.286+0000 7fdd34e00640 1 -- 172.21.15.45:0/34878396 learned_addr learned my addr 172.21.15.45:0/34878396 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:52.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.286+0000 7fdd2fe00640 1 --2- 172.21.15.45:0/34878396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30144ff0 0x7fdd300aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:52.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.286+0000 7fdd34e00640 1 -- 172.21.15.45:0/34878396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd301f8040 msgr2=0x7fdd3015e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:52.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.286+0000 7fdd34e00640 1 --2- 172.21.15.45:0/34878396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd301f8040 0x7fdd3015e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.286+0000 7fdd34e00640 1 -- 172.21.15.45:0/34878396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30144ff0 msgr2=0x7fdd300aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:52.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.286+0000 7fdd34e00640 1 --2- 172.21.15.45:0/34878396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30144ff0 0x7fdd300aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.286+0000 7fdd34e00640 1 -- 172.21.15.45:0/34878396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd301f84d0 con 0x7fdd300c5bb0 2024-04-16T16:16:52.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.287+0000 7fdd34e00640 1 --2- 172.21.15.45:0/34878396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd300c5bb0 0x7fdd300a94d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000ca70 tx=0x7fdd2000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:52.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.287+0000 7fdd2cc00640 1 -- 172.21.15.45:0/34878396 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20017070 con 0x7fdd300c5bb0 2024-04-16T16:16:52.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.287+0000 7fdd2cc00640 1 -- 172.21.15.45:0/34878396 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd2000a6a0 con 0x7fdd300c5bb0 2024-04-16T16:16:52.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.287+0000 7fdd2cc00640 1 -- 172.21.15.45:0/34878396 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2000aa70 con 0x7fdd300c5bb0 2024-04-16T16:16:52.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.287+0000 7fdd36200640 1 -- 172.21.15.45:0/34878396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd301f87b0 con 0x7fdd300c5bb0 2024-04-16T16:16:52.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.287+0000 7fdd36200640 1 -- 172.21.15.45:0/34878396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd301f8d70 con 0x7fdd300c5bb0 2024-04-16T16:16:52.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.288+0000 7fdd36200640 1 -- 172.21.15.45:0/34878396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd30075780 con 0x7fdd300c5bb0 2024-04-16T16:16:52.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.289+0000 7fdd2cc00640 1 -- 172.21.15.45:0/34878396 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fdd2001a070 con 0x7fdd300c5bb0 2024-04-16T16:16:52.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.290+0000 7fdd2cc00640 1 --2- 172.21.15.45:0/34878396 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdd0c06cb60 0x7fdd0c06f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:52.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.290+0000 7fdd2cc00640 1 -- 172.21.15.45:0/34878396 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7fdd2008b7b0 con 0x7fdd300c5bb0 2024-04-16T16:16:52.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.291+0000 7fdd2fe00640 1 --2- 172.21.15.45:0/34878396 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdd0c06cb60 0x7fdd0c06f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:52.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.292+0000 7fdd2fe00640 1 --2- 172.21.15.45:0/34878396 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdd0c06cb60 0x7fdd0c06f020 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdd18036000 tx=0x7fdd1803b040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:52.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.295+0000 7fdd2cc00640 1 -- 172.21.15.45:0/34878396 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd20059880 con 0x7fdd300c5bb0 2024-04-16T16:16:52.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.540+0000 7fdd36200640 1 -- 172.21.15.45:0/34878396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdd300b59c0 con 0x7fdd300c5bb0 2024-04-16T16:16:52.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.541+0000 7fdd2cc00640 1 -- 172.21.15.45:0/34878396 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v50) ==== 53+0+73 (secure 0 0 0) 0x7fdd20059180 con 0x7fdd300c5bb0 2024-04-16T16:16:52.541 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:16:52.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.543+0000 7fdd36200640 1 -- 172.21.15.45:0/34878396 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdd0c06cb60 msgr2=0x7fdd0c06f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:52.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.543+0000 7fdd36200640 1 --2- 172.21.15.45:0/34878396 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdd0c06cb60 0x7fdd0c06f020 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdd18036000 tx=0x7fdd1803b040 comp rx=0 tx=0).stop 2024-04-16T16:16:52.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.543+0000 7fdd36200640 1 -- 172.21.15.45:0/34878396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd300c5bb0 msgr2=0x7fdd300a94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:52.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.543+0000 7fdd36200640 1 --2- 172.21.15.45:0/34878396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd300c5bb0 0x7fdd300a94d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000ca70 tx=0x7fdd2000cf30 comp rx=0 tx=0).stop 2024-04-16T16:16:52.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.544+0000 7fdd36200640 1 -- 172.21.15.45:0/34878396 shutdown_connections 2024-04-16T16:16:52.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.544+0000 7fdd36200640 1 --2- 172.21.15.45:0/34878396 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdd0c06cb60 0x7fdd0c06f020 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.544+0000 7fdd36200640 1 --2- 172.21.15.45:0/34878396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdd301f8040 0x7fdd3015e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.544+0000 7fdd36200640 1 --2- 172.21.15.45:0/34878396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30144ff0 0x7fdd300aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.544+0000 7fdd36200640 1 --2- 172.21.15.45:0/34878396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd300c5bb0 0x7fdd300a94d0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:52.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.544+0000 7fdd36200640 1 -- 172.21.15.45:0/34878396 >> 172.21.15.45:0/34878396 conn(0x7fdd3006b930 msgr2=0x7fdd300b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:52.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.544+0000 7fdd36200640 1 -- 172.21.15.45:0/34878396 shutdown_connections 2024-04-16T16:16:52.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52.544+0000 7fdd36200640 1 -- 172.21.15.45:0/34878396 wait complete. 2024-04-16T16:16:52.563 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-16T16:16:52.563 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:16:52.563 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-04-16T16:16:52.563 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-16T16:16:52.564 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T16:16:52.564 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-16T16:16:52.564 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:16:52.564 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:16:52.564 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:16:52.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:52.564 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-04-16T16:16:52.614 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:16:52.614 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:16:52.614 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:16:52.614 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:16:52.614 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-04-16T16:16:52.614 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42389sec preferred_lft 42389sec 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-04-16T16:16:52.615 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-04-16T16:16:52.616 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:16:52.616 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff 2024-04-16T16:16:52.616 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:16:52.616 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:16:52.616 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::50f1:99ff:fe2b:225b/64 scope link 2024-04-16T16:16:52.616 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:16:52.616 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:16:52.616 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T16:16:52.616 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::7007:c3ff:fe13:7bcd/64 scope link 2024-04-16T16:16:52.616 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:16:52.616 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:16:52.616 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether ae:10:ea:fc:0a:13 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-16T16:16:52.616 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ac10:eaff:fefc:a13/64 scope link 2024-04-16T16:16:52.616 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:16:52.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:52.617 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:16:52.712 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-04-16T16:16:52.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:52.717 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-04-16T16:16:52.774 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-04-16T16:16:52.774 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:16:52.774 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-16T16:16:52.774 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-16T16:16:52.774 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:16:52.775 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:16:52.775 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:16:52.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-16T16:16:52.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-16T16:16:52.776 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-16T16:16:52.776 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:16:52.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:16:53.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:16:53.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:53.038 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T16:16:53.088 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-16T16:16:53.088 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-16T16:16:53.089 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:16:53.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:53.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-04-16T16:16:53.176 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:16:53.177 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-16T16:16:53.228 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:16:53.228 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:16:53.229 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:16:53.277 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-16T16:16:53.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:53.277 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:16:53.411 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:16:53.411 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:16:53.446 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:16:53.446 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:16:53.446 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:16:53.446 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 0@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:16:53.446 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:16:53.449 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-16T16:16:53.449 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:16:53.449 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:16:53.449 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:16:53.449 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:16:53.456 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:16:53.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:53.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:16:53.548 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:16:53.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:16:53.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:16:53.690 INFO:teuthology.orchestra.run.smithi045.stdout:client5243 2024-04-16T16:16:53.690 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:16:53.690 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:16:53.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:16:53.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.965+0000 7fdb11800640 1 Processor -- start 2024-04-16T16:16:53.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.966+0000 7fdb11800640 1 -- start start 2024-04-16T16:16:53.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.967+0000 7fdb11800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdb0c161e90 0x7fdb0c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:53.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.967+0000 7fdb11800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdb0c0d52d0 0x7fdb0c06ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:53.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.967+0000 7fdb11800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb0c0d3080 con 0x7fdb0c162270 2024-04-16T16:16:53.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.967+0000 7fdb11800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fdb0c0d2e30 con 0x7fdb0c161e90 2024-04-16T16:16:53.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.967+0000 7fdb11800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fdb0c0d3360 con 0x7fdb0c0d52d0 2024-04-16T16:16:53.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.968+0000 7fdb0be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdb0c161e90 0x7fdb0c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:53.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.968+0000 7fdb0be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdb0c161e90 0x7fdb0c0cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51090/0 (socket says 172.21.15.45:51090) 2024-04-16T16:16:53.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.968+0000 7fdb0be00640 1 -- 172.21.15.45:0/2810237432 learned_addr learned my addr 172.21.15.45:0/2810237432 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:53.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.968+0000 7fdb10e00640 1 --2- 172.21.15.45:0/2810237432 >> v2:172.21.15.103:3300/0 conn(0x7fdb0c0d52d0 0x7fdb0c06ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:53.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.968+0000 7fdb0be00640 1 -- 172.21.15.45:0/2810237432 >> v2:172.21.15.103:3300/0 conn(0x7fdb0c0d52d0 msgr2=0x7fdb0c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:53.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.968+0000 7fdb0be00640 1 --2- 172.21.15.45:0/2810237432 >> v2:172.21.15.103:3300/0 conn(0x7fdb0c0d52d0 0x7fdb0c06ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:53.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.968+0000 7fdb0be00640 1 -- 172.21.15.45:0/2810237432 >> v1:172.21.15.103:6789/0 conn(0x7fdb0c162270 legacy=0x7fdb0c0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:53.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.968+0000 7fdb0be00640 1 -- 172.21.15.45:0/2810237432 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb0c0d3640 con 0x7fdb0c161e90 2024-04-16T16:16:53.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.969+0000 7fdb0be00640 1 --2- 172.21.15.45:0/2810237432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb0c161e90 0x7fdb0c0cf7d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fdafc009870 tx=0x7fdafc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a1f1594952a45e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:53.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.970+0000 7fdb0b400640 1 -- 172.21.15.45:0/2810237432 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdafc035020 con 0x7fdb0c161e90 2024-04-16T16:16:53.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.970+0000 7fdb0b400640 1 -- 172.21.15.45:0/2810237432 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdafc031870 con 0x7fdb0c161e90 2024-04-16T16:16:53.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.970+0000 7fdb0b400640 1 -- 172.21.15.45:0/2810237432 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdafc031c40 con 0x7fdb0c161e90 2024-04-16T16:16:53.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.970+0000 7fdb11800640 1 -- 172.21.15.45:0/2810237432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb0c161e90 msgr2=0x7fdb0c0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:53.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.970+0000 7fdb11800640 1 --2- 172.21.15.45:0/2810237432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb0c161e90 0x7fdb0c0cf7d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fdafc009870 tx=0x7fdafc02b910 comp rx=0 tx=0).stop 2024-04-16T16:16:53.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.971+0000 7fdb11800640 1 -- 172.21.15.45:0/2810237432 shutdown_connections 2024-04-16T16:16:53.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.971+0000 7fdb11800640 1 --2- 172.21.15.45:0/2810237432 >> v2:172.21.15.103:3300/0 conn(0x7fdb0c0d52d0 0x7fdb0c06ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:53.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.971+0000 7fdb11800640 1 --2- 172.21.15.45:0/2810237432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb0c161e90 0x7fdb0c0cf7d0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:53.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.971+0000 7fdb11800640 1 -- 172.21.15.45:0/2810237432 >> 172.21.15.45:0/2810237432 conn(0x7fdb0c0d1940 msgr2=0x7fdb0c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:53.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:16:53.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.971+0000 7fdb11800640 1 -- 172.21.15.45:0/2810237432 shutdown_connections 2024-04-16T16:16:53.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.971+0000 7fdb11800640 1 -- 172.21.15.45:0/2810237432 wait complete. 2024-04-16T16:16:53.975 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:16:53.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.973+0000 7fdb11800640 1 Processor -- start 2024-04-16T16:16:53.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.973+0000 7fdb11800640 1 -- start start 2024-04-16T16:16:53.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.974+0000 7fdb11800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb0c0d52d0 0x7fdb0c1558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:53.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.974+0000 7fdb11800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb0c161e90 0x7fdb0c1f80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:53.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.974+0000 7fdb11800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb0c162270 0x7fdb0c1f8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:53.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.975+0000 7fdb11800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0c0d4550 con 0x7fdb0c161e90 2024-04-16T16:16:53.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.975+0000 7fdb11800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0c0cfd10 con 0x7fdb0c0d52d0 2024-04-16T16:16:53.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.975+0000 7fdb11800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdb0c0d46d0 con 0x7fdb0c162270 2024-04-16T16:16:53.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.975+0000 7fdb03e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb0c161e90 0x7fdb0c1f80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:53.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.975+0000 7fdb03e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb0c161e90 0x7fdb0c1f80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51108/0 (socket says 172.21.15.45:51108) 2024-04-16T16:16:53.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.975+0000 7fdb03e00640 1 -- 172.21.15.45:0/4212644649 learned_addr learned my addr 172.21.15.45:0/4212644649 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:53.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.975+0000 7fdb0be00640 1 --2- 172.21.15.45:0/4212644649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb0c0d52d0 0x7fdb0c1558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:53.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.975+0000 7fdb03e00640 1 -- 172.21.15.45:0/4212644649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb0c162270 msgr2=0x7fdb0c1f8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:53.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.975+0000 7fdb03e00640 1 --2- 172.21.15.45:0/4212644649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb0c162270 0x7fdb0c1f8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:53.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.975+0000 7fdb03e00640 1 -- 172.21.15.45:0/4212644649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb0c0d52d0 msgr2=0x7fdb0c1558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:53.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.975+0000 7fdb03e00640 1 --2- 172.21.15.45:0/4212644649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb0c0d52d0 0x7fdb0c1558f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:53.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.975+0000 7fdb03e00640 1 -- 172.21.15.45:0/4212644649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb0c15c180 con 0x7fdb0c161e90 2024-04-16T16:16:53.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.976+0000 7fdb03e00640 1 --2- 172.21.15.45:0/4212644649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb0c161e90 0x7fdb0c1f80c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400b3f0 tx=0x7fdaf400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:53.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.976+0000 7fdb08c00640 1 -- 172.21.15.45:0/4212644649 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4015020 con 0x7fdb0c161e90 2024-04-16T16:16:53.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.976+0000 7fdb08c00640 1 -- 172.21.15.45:0/4212644649 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaf4011830 con 0x7fdb0c161e90 2024-04-16T16:16:53.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.976+0000 7fdb08c00640 1 -- 172.21.15.45:0/4212644649 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4011c00 con 0x7fdb0c161e90 2024-04-16T16:16:53.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.976+0000 7fdb11800640 1 -- 172.21.15.45:0/4212644649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb0c15c460 con 0x7fdb0c161e90 2024-04-16T16:16:53.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.976+0000 7fdb11800640 1 -- 172.21.15.45:0/4212644649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb0c15ca20 con 0x7fdb0c161e90 2024-04-16T16:16:53.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.978+0000 7fdb11800640 1 -- 172.21.15.45:0/4212644649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb0c0ba7a0 con 0x7fdb0c161e90 2024-04-16T16:16:53.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.978+0000 7fdb08c00640 1 -- 172.21.15.45:0/4212644649 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fdaf4018020 con 0x7fdb0c161e90 2024-04-16T16:16:53.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.979+0000 7fdb08c00640 1 --2- 172.21.15.45:0/4212644649 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdae806cb30 0x7fdae806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:53.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.980+0000 7fdb08c00640 1 -- 172.21.15.45:0/4212644649 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7fdaf408bee0 con 0x7fdb0c161e90 2024-04-16T16:16:53.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.980+0000 7fdb0be00640 1 --2- 172.21.15.45:0/4212644649 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdae806cb30 0x7fdae806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:53.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.981+0000 7fdb0be00640 1 --2- 172.21.15.45:0/4212644649 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdae806cb30 0x7fdae806eff0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdafc036000 tx=0x7fdafc008030 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:53.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:53.984+0000 7fdb08c00640 1 -- 172.21.15.45:0/4212644649 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb0c0ba7a0 con 0x7fdb0c161e90 2024-04-16T16:16:54.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:54.235+0000 7fdb11800640 1 -- 172.21.15.45:0/4212644649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdb0c1593e0 con 0x7fdb0c161e90 2024-04-16T16:16:54.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:54.235+0000 7fdb08c00640 1 -- 172.21.15.45:0/4212644649 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v50) ==== 53+0+73 (secure 0 0 0) 0x7fdb0c1593e0 con 0x7fdb0c161e90 2024-04-16T16:16:54.235 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:16:54.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:54.238+0000 7fdb11800640 1 -- 172.21.15.45:0/4212644649 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdae806cb30 msgr2=0x7fdae806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:54.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:54.238+0000 7fdb11800640 1 --2- 172.21.15.45:0/4212644649 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdae806cb30 0x7fdae806eff0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdafc036000 tx=0x7fdafc008030 comp rx=0 tx=0).stop 2024-04-16T16:16:54.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:54.238+0000 7fdb11800640 1 -- 172.21.15.45:0/4212644649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb0c161e90 msgr2=0x7fdb0c1f80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:54.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:54.238+0000 7fdb11800640 1 --2- 172.21.15.45:0/4212644649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb0c161e90 0x7fdb0c1f80c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400b3f0 tx=0x7fdaf400b8b0 comp rx=0 tx=0).stop 2024-04-16T16:16:54.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:54.238+0000 7fdb11800640 1 -- 172.21.15.45:0/4212644649 shutdown_connections 2024-04-16T16:16:54.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:54.238+0000 7fdb11800640 1 --2- 172.21.15.45:0/4212644649 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fdae806cb30 0x7fdae806eff0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:54.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:54.238+0000 7fdb11800640 1 --2- 172.21.15.45:0/4212644649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb0c162270 0x7fdb0c1f8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:54.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:54.238+0000 7fdb11800640 1 --2- 172.21.15.45:0/4212644649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb0c161e90 0x7fdb0c1f80c0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:54.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:54.238+0000 7fdb11800640 1 --2- 172.21.15.45:0/4212644649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb0c0d52d0 0x7fdb0c1558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:54.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:54.238+0000 7fdb11800640 1 -- 172.21.15.45:0/4212644649 >> 172.21.15.45:0/4212644649 conn(0x7fdb0c0d1940 msgr2=0x7fdb0c0b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:54.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:54.239+0000 7fdb11800640 1 -- 172.21.15.45:0/4212644649 shutdown_connections 2024-04-16T16:16:54.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:54.239+0000 7fdb11800640 1 -- 172.21.15.45:0/4212644649 wait complete. 2024-04-16T16:16:54.259 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-16T16:16:54.259 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:16:54.259 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-04-16T16:16:54.259 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-16T16:16:54.259 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T16:16:54.259 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-16T16:16:54.259 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:16:54.259 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:16:54.259 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:16:54.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:54.259 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-04-16T16:16:54.318 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:16:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:16:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:16:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:16:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-16T16:16:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:16:54.318 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:16:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:16:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-04-16T16:16:54.318 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:16:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:16:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-04-16T16:16:54.318 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:16:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42411sec preferred_lft 42411sec 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::74ce:ceff:fec6:2a00/64 scope link 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::3cf7:ffff:feca:f879/64 scope link 2024-04-16T16:16:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:16:54.320 INFO:teuthology.orchestra.run.smithi103.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:16:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 86:4c:6e:ce:c3:07 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-16T16:16:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::844c:6eff:fece:c307/64 scope link 2024-04-16T16:16:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:16:54.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:54.321 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:16:54.413 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-16T16:16:54.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:54.418 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-04-16T16:16:54.471 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-04-16T16:16:54.471 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:16:54.471 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-16T16:16:54.471 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-16T16:16:54.471 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:16:54.471 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:16:54.471 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:16:54.471 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-16T16:16:54.471 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-16T16:16:54.471 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-16T16:16:54.472 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:16:54.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:16:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:16:54.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:16:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:16:54.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:54.720 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-16T16:16:54.769 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-16T16:16:54.769 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-16T16:16:54.770 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:16:54.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:54.770 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.0 up 2024-04-16T16:16:54.865 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:16:54.866 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-16T16:16:54.917 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:16:54.918 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:16:54.918 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:16:54.968 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-16T16:16:54.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:54.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:16:55.105 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:16:55.105 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:16:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:16:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:16:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:16:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with new device syntax: 1@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:16:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:16:55.146 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-16T16:16:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:16:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:16:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:16:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:16:55.151 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:16:55.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:16:55.151 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:16:55.239 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:16:55.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:16:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:16:55.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:16:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:16:55.359 INFO:teuthology.orchestra.run.smithi103.stdout:client6591 2024-04-16T16:16:55.359 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:16:55.359 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:16:55.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:16:55.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.649+0000 7fe2ab000640 1 Processor -- start 2024-04-16T16:16:55.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.649+0000 7fe2ab000640 1 -- start start 2024-04-16T16:16:55.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.651+0000 7fe2ab000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2a4162ec0 0x7fe2a40b6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:55.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.651+0000 7fe2ab000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe2a40b6790 0x7fe2a406b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:55.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.651+0000 7fe2ab000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2a406c300 con 0x7fe2a406be70 2024-04-16T16:16:55.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.651+0000 7fe2ab000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe2a40b6bd0 con 0x7fe2a40b6790 2024-04-16T16:16:55.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.651+0000 7fe2ab000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2a4166c90 con 0x7fe2a4162ec0 2024-04-16T16:16:55.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.651+0000 7fe2a9200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe2a40b6790 0x7fe2a406b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:55.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.652+0000 7fe2a9200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe2a40b6790 0x7fe2a406b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51114/0 (socket says 172.21.15.45:51114) 2024-04-16T16:16:55.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.652+0000 7fe2a9200640 1 -- 172.21.15.45:0/4125690744 learned_addr learned my addr 172.21.15.45:0/4125690744 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:55.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.652+0000 7fe2a9c00640 1 --2- 172.21.15.45:0/4125690744 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2a4162ec0 0x7fe2a40b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:55.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.652+0000 7fe2a9200640 1 -- 172.21.15.45:0/4125690744 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2a4162ec0 msgr2=0x7fe2a40b6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.652+0000 7fe2a9200640 1 --2- 172.21.15.45:0/4125690744 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2a4162ec0 0x7fe2a40b6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.652+0000 7fe2a9200640 1 -- 172.21.15.45:0/4125690744 >> v1:172.21.15.103:6789/0 conn(0x7fe2a406be70 legacy=0x7fe2a4073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.652+0000 7fe2a9200640 1 -- 172.21.15.45:0/4125690744 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a4166f70 con 0x7fe2a40b6790 2024-04-16T16:16:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.653+0000 7fe2a9200640 1 --2- 172.21.15.45:0/4125690744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a40b6790 0x7fe2a406b930 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe290009870 tx=0x7fe29002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5447c77d17d5fe2e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.654+0000 7fe2a3e00640 1 -- 172.21.15.45:0/4125690744 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe290038070 con 0x7fe2a40b6790 2024-04-16T16:16:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.654+0000 7fe2a3e00640 1 -- 172.21.15.45:0/4125690744 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe290031740 con 0x7fe2a40b6790 2024-04-16T16:16:55.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.654+0000 7fe2a3e00640 1 -- 172.21.15.45:0/4125690744 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29002f050 con 0x7fe2a40b6790 2024-04-16T16:16:55.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.654+0000 7fe2ab000640 1 -- 172.21.15.45:0/4125690744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a40b6790 msgr2=0x7fe2a406b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:55.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.654+0000 7fe2ab000640 1 --2- 172.21.15.45:0/4125690744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a40b6790 0x7fe2a406b930 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe290009870 tx=0x7fe29002b7e0 comp rx=0 tx=0).stop 2024-04-16T16:16:55.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.655+0000 7fe2ab000640 1 -- 172.21.15.45:0/4125690744 shutdown_connections 2024-04-16T16:16:55.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.655+0000 7fe2ab000640 1 --2- 172.21.15.45:0/4125690744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a40b6790 0x7fe2a406b930 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:55.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.655+0000 7fe2ab000640 1 --2- 172.21.15.45:0/4125690744 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2a4162ec0 0x7fe2a40b6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:55.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.655+0000 7fe2ab000640 1 -- 172.21.15.45:0/4125690744 >> 172.21.15.45:0/4125690744 conn(0x7fe2a406f0e0 msgr2=0x7fe2a406f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:55.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.655+0000 7fe2ab000640 1 -- 172.21.15.45:0/4125690744 shutdown_connections 2024-04-16T16:16:55.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.655+0000 7fe2ab000640 1 -- 172.21.15.45:0/4125690744 wait complete. 2024-04-16T16:16:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.656+0000 7fe2ab000640 1 Processor -- start 2024-04-16T16:16:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.657+0000 7fe2ab000640 1 -- start start 2024-04-16T16:16:55.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2ab000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2a406be70 0x7fe2a414c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:55.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2ab000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2a40b6790 0x7fe2a415bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:55.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2ab000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a4162ec0 0x7fe2a40cff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:55.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2ab000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a41632a0 con 0x7fe2a4162ec0 2024-04-16T16:16:55.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2ab000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a4167530 con 0x7fe2a406be70 2024-04-16T16:16:55.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2ab000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2a4163420 con 0x7fe2a40b6790 2024-04-16T16:16:55.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2aa600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a4162ec0 0x7fe2a40cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:55.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2aa600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a4162ec0 0x7fe2a40cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51126/0 (socket says 172.21.15.45:51126) 2024-04-16T16:16:55.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2aa600640 1 -- 172.21.15.45:0/3356745695 learned_addr learned my addr 172.21.15.45:0/3356745695 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:55.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2a9200640 1 --2- 172.21.15.45:0/3356745695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2a40b6790 0x7fe2a415bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:55.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2a9c00640 1 --2- 172.21.15.45:0/3356745695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2a406be70 0x7fe2a414c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:55.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2aa600640 1 -- 172.21.15.45:0/3356745695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2a40b6790 msgr2=0x7fe2a415bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:55.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2aa600640 1 --2- 172.21.15.45:0/3356745695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2a40b6790 0x7fe2a415bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:55.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2aa600640 1 -- 172.21.15.45:0/3356745695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2a406be70 msgr2=0x7fe2a414c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:55.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2aa600640 1 --2- 172.21.15.45:0/3356745695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2a406be70 0x7fe2a414c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:55.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.658+0000 7fe2aa600640 1 -- 172.21.15.45:0/3356745695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a4158f20 con 0x7fe2a4162ec0 2024-04-16T16:16:55.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.659+0000 7fe2aa600640 1 --2- 172.21.15.45:0/3356745695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a4162ec0 0x7fe2a40cff40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fe29c00c780 tx=0x7fe29c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:55.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.659+0000 7fe2a1600640 1 -- 172.21.15.45:0/3356745695 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29c004b60 con 0x7fe2a4162ec0 2024-04-16T16:16:55.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.659+0000 7fe2a1600640 1 -- 172.21.15.45:0/3356745695 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe29c004d00 con 0x7fe2a4162ec0 2024-04-16T16:16:55.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.659+0000 7fe2ab000640 1 -- 172.21.15.45:0/3356745695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2a41ecaf0 con 0x7fe2a4162ec0 2024-04-16T16:16:55.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.659+0000 7fe2a1600640 1 -- 172.21.15.45:0/3356745695 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29c00aa90 con 0x7fe2a4162ec0 2024-04-16T16:16:55.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.659+0000 7fe2ab000640 1 -- 172.21.15.45:0/3356745695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2a41ed0b0 con 0x7fe2a4162ec0 2024-04-16T16:16:55.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.660+0000 7fe2ab000640 1 -- 172.21.15.45:0/3356745695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2a4075780 con 0x7fe2a4162ec0 2024-04-16T16:16:55.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.661+0000 7fe2a1600640 1 -- 172.21.15.45:0/3356745695 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7fe29c00ac30 con 0x7fe2a4162ec0 2024-04-16T16:16:55.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.662+0000 7fe2a1600640 1 --2- 172.21.15.45:0/3356745695 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe28406cb30 0x7fe28406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:55.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.662+0000 7fe2a1600640 1 -- 172.21.15.45:0/3356745695 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7fe29c08c2d0 con 0x7fe2a4162ec0 2024-04-16T16:16:55.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.663+0000 7fe2a9c00640 1 --2- 172.21.15.45:0/3356745695 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe28406cb30 0x7fe28406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:55.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.664+0000 7fe2a9c00640 1 --2- 172.21.15.45:0/3356745695 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe28406cb30 0x7fe28406eff0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe298006fd0 tx=0x7fe298008040 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:55.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.667+0000 7fe2a1600640 1 -- 172.21.15.45:0/3356745695 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe29c020070 con 0x7fe2a4162ec0 2024-04-16T16:16:55.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.917+0000 7fe2ab000640 1 -- 172.21.15.45:0/3356745695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2a40b3010 con 0x7fe2a4162ec0 2024-04-16T16:16:55.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.918+0000 7fe2a1600640 1 -- 172.21.15.45:0/3356745695 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5180 (secure 0 0 0) 0x7fe29c05a390 con 0x7fe2a4162ec0 2024-04-16T16:16:55.918 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:16:55.918 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":50,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":6399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:16:42.595621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-04-16T16:16:55.919 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 50 2024-04-16T16:16:55.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.921+0000 7fe2ab000640 1 -- 172.21.15.45:0/3356745695 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe28406cb30 msgr2=0x7fe28406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:55.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.921+0000 7fe2ab000640 1 --2- 172.21.15.45:0/3356745695 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe28406cb30 0x7fe28406eff0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe298006fd0 tx=0x7fe298008040 comp rx=0 tx=0).stop 2024-04-16T16:16:55.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.921+0000 7fe2ab000640 1 -- 172.21.15.45:0/3356745695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a4162ec0 msgr2=0x7fe2a40cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:55.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.921+0000 7fe2ab000640 1 --2- 172.21.15.45:0/3356745695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a4162ec0 0x7fe2a40cff40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fe29c00c780 tx=0x7fe29c00cc40 comp rx=0 tx=0).stop 2024-04-16T16:16:55.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.921+0000 7fe2ab000640 1 -- 172.21.15.45:0/3356745695 shutdown_connections 2024-04-16T16:16:55.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.921+0000 7fe2ab000640 1 --2- 172.21.15.45:0/3356745695 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7fe28406cb30 0x7fe28406eff0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:55.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.921+0000 7fe2ab000640 1 --2- 172.21.15.45:0/3356745695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a4162ec0 0x7fe2a40cff40 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:55.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.921+0000 7fe2ab000640 1 --2- 172.21.15.45:0/3356745695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2a40b6790 0x7fe2a415bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:55.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.921+0000 7fe2ab000640 1 --2- 172.21.15.45:0/3356745695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2a406be70 0x7fe2a414c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:55.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.921+0000 7fe2ab000640 1 -- 172.21.15.45:0/3356745695 >> 172.21.15.45:0/3356745695 conn(0x7fe2a406f0e0 msgr2=0x7fe2a40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:55.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.921+0000 7fe2ab000640 1 -- 172.21.15.45:0/3356745695 shutdown_connections 2024-04-16T16:16:55.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:55.921+0000 7fe2ab000640 1 -- 172.21.15.45:0/3356745695 wait complete. 2024-04-16T16:16:55.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable stats 2024-04-16T16:16:56.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.169+0000 7f2543200640 1 Processor -- start 2024-04-16T16:16:56.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.169+0000 7f2543200640 1 -- start start 2024-04-16T16:16:56.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.170+0000 7f2543200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f253c15eef0 0x7f253c0d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:56.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.170+0000 7f2543200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f253c0d35a0 con 0x7f253c0cf5c0 2024-04-16T16:16:56.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.170+0000 7f2543200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f253c0d2740 con 0x7f253c0d42d0 2024-04-16T16:16:56.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.170+0000 7f2543200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f253c0d2a20 con 0x7f253c15eef0 2024-04-16T16:16:56.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.171+0000 7f2541e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f253c0cf5c0 0x7f253c0cf9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57504/0 (socket says 172.21.15.45:57504) 2024-04-16T16:16:56.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.171+0000 7f2541e00640 1 -- 172.21.15.45:0/3991932555 learned_addr learned my addr 172.21.15.45:0/3991932555 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:16:56.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.171+0000 7f2541400640 1 --2- 172.21.15.45:0/3991932555 >> v2:172.21.15.103:3300/0 conn(0x7f253c15eef0 0x7f253c0d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:56.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.171+0000 7f2540a00640 1 -- 172.21.15.45:0/3991932555 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 176617044 0 0) 0x7f253c0d35a0 con 0x7f253c0cf5c0 2024-04-16T16:16:56.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.171+0000 7f2540a00640 1 -- 172.21.15.45:0/3991932555 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2524003520 con 0x7f253c0cf5c0 2024-04-16T16:16:56.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.172+0000 7f2540a00640 1 -- 172.21.15.45:0/3991932555 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1532426385 0 0) 0x7f2524003520 con 0x7f253c0cf5c0 2024-04-16T16:16:56.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.172+0000 7f2540a00640 1 -- 172.21.15.45:0/3991932555 >> v2:172.21.15.103:3300/0 conn(0x7f253c15eef0 msgr2=0x7f253c0d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:56.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.172+0000 7f2540a00640 1 --2- 172.21.15.45:0/3991932555 >> v2:172.21.15.103:3300/0 conn(0x7f253c15eef0 0x7f253c0d3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:56.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.172+0000 7f2540a00640 1 -- 172.21.15.45:0/3991932555 >> v1:172.21.15.103:6789/0 conn(0x7f253c0d42d0 legacy=0x7f253c0b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:56.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.172+0000 7f2540a00640 1 -- 172.21.15.45:0/3991932555 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253c0d2d00 con 0x7f253c0cf5c0 2024-04-16T16:16:56.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.172+0000 7f2540a00640 1 -- 172.21.15.45:0/3991932555 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f2530001670 con 0x7f253c0cf5c0 2024-04-16T16:16:56.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.172+0000 7f2540a00640 1 -- 172.21.15.45:0/3991932555 >> v1:172.21.15.45:6789/0 conn(0x7f253c0cf5c0 legacy=0x7f253c0cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:56.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.172+0000 7f2540a00640 1 --2- 172.21.15.45:0/3991932555 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2524003a00 0x7f2524003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:56.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.172+0000 7f2540a00640 1 --2- 172.21.15.45:0/3991932555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2524004800 0x7f2524004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:56.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.172+0000 7f2540a00640 1 --2- 172.21.15.45:0/3991932555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25240051c0 0x7f252400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:56.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.172+0000 7f2540a00640 1 -- 172.21.15.45:0/3991932555 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f253c0d2a20 con 0x7f25240051c0 2024-04-16T16:16:56.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.172+0000 7f2540a00640 1 -- 172.21.15.45:0/3991932555 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f252400c0c0 con 0x7f2524004800 2024-04-16T16:16:56.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.172+0000 7f2540a00640 1 -- 172.21.15.45:0/3991932555 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f252400c2c0 con 0x7f2524003a00 2024-04-16T16:16:56.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.172+0000 7f2542800640 1 --2- 172.21.15.45:0/3991932555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25240051c0 0x7f252400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:56.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.173+0000 7f2541400640 1 --2- 172.21.15.45:0/3991932555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2524004800 0x7f2524004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:56.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.173+0000 7f2542800640 1 -- 172.21.15.45:0/3991932555 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2524003a00 msgr2=0x7f2524003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:56.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.173+0000 7f2542800640 1 --2- 172.21.15.45:0/3991932555 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2524003a00 0x7f2524003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:56.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.173+0000 7f2542800640 1 -- 172.21.15.45:0/3991932555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2524004800 msgr2=0x7f2524004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:56.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.173+0000 7f2542800640 1 --2- 172.21.15.45:0/3991932555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2524004800 0x7f2524004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:56.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.173+0000 7f2542800640 1 -- 172.21.15.45:0/3991932555 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2524003520 con 0x7f25240051c0 2024-04-16T16:16:56.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.173+0000 7f2541400640 1 --2- 172.21.15.45:0/3991932555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2524004800 0x7f2524004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:16:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.174+0000 7f2542800640 1 -- 172.21.15.45:0/3991932555 reap_dead start 2024-04-16T16:16:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.174+0000 7f2542800640 1 --2- 172.21.15.45:0/3991932555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25240051c0 0x7f252400b9c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f2538001e20 tx=0x7f2524005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185690d25521eef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.174+0000 7f2540a00640 1 -- 172.21.15.45:0/3991932555 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2538001a70 con 0x7f25240051c0 2024-04-16T16:16:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.174+0000 7f2540a00640 1 -- 172.21.15.45:0/3991932555 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2538003b70 con 0x7f25240051c0 2024-04-16T16:16:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.174+0000 7f2540a00640 1 -- 172.21.15.45:0/3991932555 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25380050d0 con 0x7f25240051c0 2024-04-16T16:16:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.175+0000 7f2543200640 1 -- 172.21.15.45:0/3991932555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25240051c0 msgr2=0x7f252400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.175+0000 7f2543200640 1 --2- 172.21.15.45:0/3991932555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25240051c0 0x7f252400b9c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f2538001e20 tx=0x7f2524005830 comp rx=0 tx=0).stop 2024-04-16T16:16:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.175+0000 7f2543200640 1 -- 172.21.15.45:0/3991932555 shutdown_connections 2024-04-16T16:16:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.175+0000 7f2543200640 1 --2- 172.21.15.45:0/3991932555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25240051c0 0x7f252400b9c0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.175+0000 7f2543200640 1 -- 172.21.15.45:0/3991932555 >> 172.21.15.45:0/3991932555 conn(0x7f253c06d300 msgr2=0x7f253c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.175+0000 7f2543200640 1 -- 172.21.15.45:0/3991932555 shutdown_connections 2024-04-16T16:16:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.175+0000 7f2543200640 1 -- 172.21.15.45:0/3991932555 wait complete. 2024-04-16T16:16:56.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.177+0000 7f2543200640 1 Processor -- start 2024-04-16T16:16:56.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.177+0000 7f2543200640 1 -- start start 2024-04-16T16:16:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.178+0000 7f2543200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25240051c0 0x7f253c0caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.178+0000 7f2543200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f253c14e8a0 0x7f253c0c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.178+0000 7f2543200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f253c0c9b20 0x7f253c0c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.178+0000 7f2543200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f253c0d3980 con 0x7f253c14e8a0 2024-04-16T16:16:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.178+0000 7f2543200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f253c0d32c0 con 0x7f25240051c0 2024-04-16T16:16:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.178+0000 7f2543200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f253c0d3b00 con 0x7f253c0c9b20 2024-04-16T16:16:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.178+0000 7f2541400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f253c14e8a0 0x7f253c0c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.178+0000 7f2541400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f253c14e8a0 0x7f253c0c8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51150/0 (socket says 172.21.15.45:51150) 2024-04-16T16:16:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.178+0000 7f2541400640 1 -- 172.21.15.45:0/241334101 learned_addr learned my addr 172.21.15.45:0/241334101 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.178+0000 7f2541400640 1 -- 172.21.15.45:0/241334101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f253c0c9b20 msgr2=0x7f253c0c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.178+0000 7f2541400640 1 --2- 172.21.15.45:0/241334101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f253c0c9b20 0x7f253c0c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.178+0000 7f2541400640 1 -- 172.21.15.45:0/241334101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25240051c0 msgr2=0x7f253c0caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.178+0000 7f2541400640 1 --2- 172.21.15.45:0/241334101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25240051c0 0x7f253c0caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.178+0000 7f2541400640 1 -- 172.21.15.45:0/241334101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253c14e3a0 con 0x7f253c14e8a0 2024-04-16T16:16:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.179+0000 7f2541400640 1 --2- 172.21.15.45:0/241334101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f253c14e8a0 0x7f253c0c8510 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f252c010880 tx=0x7f252c010d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.179+0000 7f2536000640 1 -- 172.21.15.45:0/241334101 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c00f040 con 0x7f253c14e8a0 2024-04-16T16:16:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.179+0000 7f2536000640 1 -- 172.21.15.45:0/241334101 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f252c00eb60 con 0x7f253c14e8a0 2024-04-16T16:16:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.179+0000 7f2536000640 1 -- 172.21.15.45:0/241334101 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c01c400 con 0x7f253c14e8a0 2024-04-16T16:16:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.179+0000 7f2543200640 1 -- 172.21.15.45:0/241334101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253c0cb280 con 0x7f253c14e8a0 2024-04-16T16:16:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.179+0000 7f2543200640 1 -- 172.21.15.45:0/241334101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f253c0c9320 con 0x7f253c14e8a0 2024-04-16T16:16:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.181+0000 7f2543200640 1 -- 172.21.15.45:0/241334101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f253c075780 con 0x7f253c14e8a0 2024-04-16T16:16:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.182+0000 7f2536000640 1 -- 172.21.15.45:0/241334101 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 91244+0+0 (secure 0 0 0) 0x7f252c01c5a0 con 0x7f253c14e8a0 2024-04-16T16:16:56.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.182+0000 7f2536000640 1 --2- 172.21.15.45:0/241334101 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f252006cb30 0x7f252006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:56.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.183+0000 7f2536000640 1 -- 172.21.15.45:0/241334101 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f252c08c650 con 0x7f253c14e8a0 2024-04-16T16:16:56.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.183+0000 7f2541e00640 1 --2- 172.21.15.45:0/241334101 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f252006cb30 0x7f252006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:56.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.184+0000 7f2541e00640 1 --2- 172.21.15.45:0/241334101 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f252006cb30 0x7f252006eff0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f25300078d0 tx=0x7f2530001520 comp rx=0 tx=0).ready entity=mgr.4961 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:56.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.187+0000 7f2536000640 1 -- 172.21.15.45:0/241334101 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f252c018030 con 0x7f253c14e8a0 2024-04-16T16:16:56.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:56.445+0000 7f2543200640 1 -- 172.21.15.45:0/241334101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "stats"} v 0) -- 0x7f253c0d2520 con 0x7f253c14e8a0 2024-04-16T16:16:57.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.168+0000 7f2536000640 1 -- 172.21.15.45:0/241334101 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "stats"}]=0 v16) ==== 84+0+0 (secure 0 0 0) 0x7f252c05a160 con 0x7f253c14e8a0 2024-04-16T16:16:57.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.173+0000 7f2543200640 1 -- 172.21.15.45:0/241334101 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f252006cb30 msgr2=0x7f252006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:57.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.173+0000 7f2543200640 1 --2- 172.21.15.45:0/241334101 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f252006cb30 0x7f252006eff0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f25300078d0 tx=0x7f2530001520 comp rx=0 tx=0).stop 2024-04-16T16:16:57.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.173+0000 7f2543200640 1 -- 172.21.15.45:0/241334101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f253c14e8a0 msgr2=0x7f253c0c8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:57.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.173+0000 7f2543200640 1 --2- 172.21.15.45:0/241334101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f253c14e8a0 0x7f253c0c8510 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f252c010880 tx=0x7f252c010d40 comp rx=0 tx=0).stop 2024-04-16T16:16:57.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.174+0000 7f2543200640 1 -- 172.21.15.45:0/241334101 shutdown_connections 2024-04-16T16:16:57.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.174+0000 7f2543200640 1 --2- 172.21.15.45:0/241334101 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f252006cb30 0x7f252006eff0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:57.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.174+0000 7f2543200640 1 --2- 172.21.15.45:0/241334101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f253c0c9b20 0x7f253c0c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:57.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.174+0000 7f2543200640 1 --2- 172.21.15.45:0/241334101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f253c14e8a0 0x7f253c0c8510 secure :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f252c010880 tx=0x7f252c010d40 comp rx=0 tx=0).stop 2024-04-16T16:16:57.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.174+0000 7f2543200640 1 --2- 172.21.15.45:0/241334101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25240051c0 0x7f253c0caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:57.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.174+0000 7f2543200640 1 -- 172.21.15.45:0/241334101 >> 172.21.15.45:0/241334101 conn(0x7f253c06d300 msgr2=0x7f253c0b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:57.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.175+0000 7f2543200640 1 -- 172.21.15.45:0/241334101 shutdown_connections 2024-04-16T16:16:57.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.175+0000 7f2543200640 1 -- 172.21.15.45:0/241334101 wait complete. 2024-04-16T16:16:57.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:16:57.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.401+0000 7f4931e00640 1 Processor -- start 2024-04-16T16:16:57.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.402+0000 7f4931e00640 1 -- start start 2024-04-16T16:16:57.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.404+0000 7f4931e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f492c1453c0 0x7f492c0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:57.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.404+0000 7f4931e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f492c06b930 0x7f492c06bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:57.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.404+0000 7f4931e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f492c0b6930 con 0x7f492c0b63c0 2024-04-16T16:16:57.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.404+0000 7f4931e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f492c0b6c10 con 0x7f492c06b930 2024-04-16T16:16:57.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.404+0000 7f4931e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f492c147590 con 0x7f492c1453c0 2024-04-16T16:16:57.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.404+0000 7f492b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f492c06b930 0x7f492c06bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:57.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.404+0000 7f492b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f492c06b930 0x7f492c06bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51168/0 (socket says 172.21.15.45:51168) 2024-04-16T16:16:57.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.404+0000 7f492b400640 1 -- 172.21.15.45:0/1582838695 learned_addr learned my addr 172.21.15.45:0/1582838695 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:57.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.405+0000 7f492aa00640 1 --2- 172.21.15.45:0/1582838695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f492c1453c0 0x7f492c0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:57.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.405+0000 7f492b400640 1 -- 172.21.15.45:0/1582838695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f492c1453c0 msgr2=0x7f492c0b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:16:57.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.405+0000 7f492b400640 1 --2- 172.21.15.45:0/1582838695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f492c1453c0 0x7f492c0b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:57.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.405+0000 7f492b400640 1 -- 172.21.15.45:0/1582838695 >> v1:172.21.15.103:6789/0 conn(0x7f492c0b63c0 legacy=0x7f492c15f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:57.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.405+0000 7f492b400640 1 -- 172.21.15.45:0/1582838695 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492c147870 con 0x7f492c06b930 2024-04-16T16:16:57.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.407+0000 7f492b400640 1 --2- 172.21.15.45:0/1582838695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f492c06b930 0x7f492c06bdc0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f4924009e50 tx=0x7f492402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807b86d0d9b3f91a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:57.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.407+0000 7f492a000640 1 -- 172.21.15.45:0/1582838695 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f492402cb00 con 0x7f492c06b930 2024-04-16T16:16:57.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.407+0000 7f492a000640 1 -- 172.21.15.45:0/1582838695 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f492402cca0 con 0x7f492c06b930 2024-04-16T16:16:57.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.407+0000 7f492a000640 1 -- 172.21.15.45:0/1582838695 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4924034610 con 0x7f492c06b930 2024-04-16T16:16:57.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.408+0000 7f4931e00640 1 -- 172.21.15.45:0/1582838695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f492c06b930 msgr2=0x7f492c06bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:57.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.408+0000 7f4931e00640 1 --2- 172.21.15.45:0/1582838695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f492c06b930 0x7f492c06bdc0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f4924009e50 tx=0x7f492402bf60 comp rx=0 tx=0).stop 2024-04-16T16:16:57.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.408+0000 7f4931e00640 1 -- 172.21.15.45:0/1582838695 shutdown_connections 2024-04-16T16:16:57.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.408+0000 7f4931e00640 1 --2- 172.21.15.45:0/1582838695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f492c06b930 0x7f492c06bdc0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:57.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.408+0000 7f4931e00640 1 --2- 172.21.15.45:0/1582838695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f492c1453c0 0x7f492c0b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:57.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.408+0000 7f4931e00640 1 -- 172.21.15.45:0/1582838695 >> 172.21.15.45:0/1582838695 conn(0x7f492c0b6ea0 msgr2=0x7f492c09aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:16:57.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.408+0000 7f4931e00640 1 -- 172.21.15.45:0/1582838695 shutdown_connections 2024-04-16T16:16:57.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.408+0000 7f4931e00640 1 -- 172.21.15.45:0/1582838695 wait complete. 2024-04-16T16:16:57.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.411+0000 7f4931e00640 1 Processor -- start 2024-04-16T16:16:57.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.411+0000 7f4931e00640 1 -- start start 2024-04-16T16:16:57.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.413+0000 7f4931e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f492c06b930 0x7f492c15ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:57.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.413+0000 7f4931e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f492c0b63c0 0x7f492c0caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:57.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.413+0000 7f4931e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f492c1453c0 0x7f492c0b1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:57.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.413+0000 7f4931e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f492c0b4d10 con 0x7f492c0b63c0 2024-04-16T16:16:57.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.413+0000 7f4931e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f492c15fa00 con 0x7f492c1453c0 2024-04-16T16:16:57.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.413+0000 7f4931e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f492c0b4e90 con 0x7f492c06b930 2024-04-16T16:16:57.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.413+0000 7f4923e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f492c0b63c0 0x7f492c0caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:57.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.413+0000 7f4923e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f492c0b63c0 0x7f492c0caae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51178/0 (socket says 172.21.15.45:51178) 2024-04-16T16:16:57.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.413+0000 7f4923e00640 1 -- 172.21.15.45:0/3354772558 learned_addr learned my addr 172.21.15.45:0/3354772558 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:16:57.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.413+0000 7f492b400640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f492c1453c0 0x7f492c0b1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:57.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.413+0000 7f492aa00640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f492c06b930 0x7f492c15ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:16:57.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.414+0000 7f4923e00640 1 -- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f492c06b930 msgr2=0x7f492c15ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:57.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.414+0000 7f4923e00640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f492c06b930 0x7f492c15ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:57.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.414+0000 7f4923e00640 1 -- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f492c1453c0 msgr2=0x7f492c0b1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:16:57.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.414+0000 7f4923e00640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f492c1453c0 0x7f492c0b1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:16:57.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.414+0000 7f4923e00640 1 -- 172.21.15.45:0/3354772558 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492c1f4c00 con 0x7f492c0b63c0 2024-04-16T16:16:57.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.414+0000 7f4923e00640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f492c0b63c0 0x7f492c0caae0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f491400c400 tx=0x7f491400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:16:57.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.414+0000 7f4923400640 1 -- 172.21.15.45:0/3354772558 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49140052d0 con 0x7f492c0b63c0 2024-04-16T16:16:57.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.414+0000 7f4923400640 1 -- 172.21.15.45:0/3354772558 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4914005490 con 0x7f492c0b63c0 2024-04-16T16:16:57.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.414+0000 7f4923400640 1 -- 172.21.15.45:0/3354772558 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4914007260 con 0x7f492c0b63c0 2024-04-16T16:16:57.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.414+0000 7f4931e00640 1 -- 172.21.15.45:0/3354772558 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f492c1f4ee0 con 0x7f492c0b63c0 2024-04-16T16:16:57.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.415+0000 7f4931e00640 1 -- 172.21.15.45:0/3354772558 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f492c1f54a0 con 0x7f492c0b63c0 2024-04-16T16:16:57.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.416+0000 7f4931e00640 1 -- 172.21.15.45:0/3354772558 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f492c1f5c80 con 0x7f492c0b63c0 2024-04-16T16:16:57.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.417+0000 7f4923400640 1 -- 172.21.15.45:0/3354772558 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) ==== 91253+0+0 (secure 0 0 0) 0x7f4914007400 con 0x7f492c0b63c0 2024-04-16T16:16:57.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.418+0000 7f4923400640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f490806cb80 0x7f490806f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:16:57.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.418+0000 7f4923400640 1 -- 172.21.15.45:0/3354772558 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f491408b550 con 0x7f492c0b63c0 2024-04-16T16:16:57.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.418+0000 7f492aa00640 1 -- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f490806cb80 msgr2=0x7f490806f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/4155603915 2024-04-16T16:16:57.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.418+0000 7f492aa00640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f490806cb80 0x7f490806f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:16:57.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.424+0000 7f4923400640 1 -- 172.21.15.45:0/3354772558 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f492c1f5c80 con 0x7f492c0b63c0 2024-04-16T16:16:57.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.619+0000 7f492aa00640 1 -- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f490806cb80 msgr2=0x7f490806f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/4155603915 2024-04-16T16:16:57.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.619+0000 7f492aa00640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f490806cb80 0x7f490806f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:16:57.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:57.655+0000 7f4931e00640 1 -- 172.21.15.45:0/3354772558 --> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f492c075780 con 0x7f490806cb80 2024-04-16T16:16:57.733 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:57.734+0000 7f033ae4b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:16:57.752 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:57.752+0000 7f9e58232040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:16:57.787 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:57.788+0000 7f033ae4b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:16:57.809 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:57.809+0000 7f9e58232040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:16:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:58.020+0000 7f492aa00640 1 -- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f490806cb80 msgr2=0x7f490806f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/4155603915 2024-04-16T16:16:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:58.020+0000 7f492aa00640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f490806cb80 0x7f490806f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T16:16:58.229 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:58.229+0000 7f033ae4b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:16:58.273 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:58.273+0000 7f9e58232040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:16:58.280 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:58.281+0000 7f033ae4b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:16:58.327 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:58.327+0000 7f9e58232040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:16:58.330 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:58.331+0000 7f033ae4b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:16:58.381 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:58.381+0000 7f9e58232040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:16:58.392 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:58.393+0000 7f033ae4b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:16:58.442 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:58.442+0000 7f033ae4b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:16:58.445 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:58.445+0000 7f9e58232040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:16:58.497 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:58.497+0000 7f9e58232040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:16:58.569 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:58.569+0000 7f033ae4b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:16:58.631 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:58.632+0000 7f033ae4b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:16:58.632 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:58.632+0000 7f9e58232040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:16:58.682 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:58.682+0000 7f033ae4b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:16:58.698 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:58.698+0000 7f9e58232040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:16:58.738 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:58.739+0000 7f033ae4b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:16:58.751 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:58.751+0000 7f9e58232040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:16:58.789 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:58.789+0000 7f033ae4b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:16:58.811 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:58.811+0000 7f9e58232040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:16:58.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:58.821+0000 7f492aa00640 1 -- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f490806cb80 msgr2=0x7f490806f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/4155603915 2024-04-16T16:16:58.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:16:58.821+0000 7f492aa00640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f490806cb80 0x7f490806f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T16:16:58.864 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:58.864+0000 7f9e58232040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:16:59.171 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:59.171+0000 7f033ae4b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:16:59.267 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:59.267+0000 7f9e58232040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:16:59.328 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:59.329+0000 7f033ae4b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:16:59.436 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:59.436+0000 7f9e58232040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:16:59.437 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:59.438+0000 7f033ae4b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:16:59.487 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:59.488+0000 7f033ae4b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:16:59.550 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:59.550+0000 7f9e58232040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:16:59.603 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:59.603+0000 7f9e58232040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:16:59.613 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:59.613+0000 7f033ae4b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:16:59.719 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:59.720+0000 7f033ae4b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:16:59.736 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:59.736+0000 7f9e58232040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:16:59.848 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:16:59.848+0000 7f9e58232040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:16:59.893 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:59.894+0000 7f033ae4b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:16:59.943 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:16:59.943+0000 7f033ae4b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:17:00.032 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:00.032+0000 7f9e58232040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:17:00.078 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:00.078+0000 7f033ae4b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:17:00.084 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:00.084+0000 7f9e58232040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:17:00.228 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:00.228+0000 7f9e58232040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:17:00.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:00.423+0000 7f492aa00640 1 -- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f490806cb80 msgr2=0x7f490806f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/4155603915 2024-04-16T16:17:00.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:00.423+0000 7f492aa00640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f490806cb80 0x7f490806f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T16:17:00.911 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:00.911+0000 7f033ae4b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:17:01.105 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:01.104+0000 7f9e58232040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:17:01.214 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:01.214+0000 7f033ae4b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:17:01.271 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:01.272+0000 7f033ae4b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:17:01.401 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:01.401+0000 7f033ae4b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:17:01.428 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:01.428+0000 7f9e58232040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:17:01.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:01.429+0000 7f4923400640 1 -- 172.21.15.45:0/3354772558 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 17) ==== 90760+0+0 (secure 0 0 0) 0x7f49140555c0 con 0x7f492c0b63c0 2024-04-16T16:17:01.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:01.429+0000 7f4923400640 1 -- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f490806cb80 msgr2=0x7f490806f040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:17:01.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:01.429+0000 7f4923400640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/4155603915,v1:172.21.15.103:6833/4155603915] conn(0x7f490806cb80 0x7f490806f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:01.492 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:01.492+0000 7f9e58232040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:17:01.504 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:01.504+0000 7f02b6a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:01.504 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:01.504+0000 7f02b6a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:01.504 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:01.504+0000 7f02b6a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:01.504 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:01.504+0000 7f02b6a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:01.504 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:01.504+0000 7f02b6a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:01.504 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:01.505+0000 7f02b2e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:01.504 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:01.505+0000 7f02b2e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:01.504 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:01.505+0000 7f02b2e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:01.504 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:01.505+0000 7f02b2e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:01.505 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:01.505+0000 7f02b2e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:01.634 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:01.634+0000 7f9e58232040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:17:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:02.438+0000 7f4923400640 1 -- 172.21.15.45:0/3354772558 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 18) ==== 90887+0+0 (secure 0 0 0) 0x7f49140591b0 con 0x7f492c0b63c0 2024-04-16T16:17:02.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:02.439+0000 7f4923400640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4908074a30 0x7f4908076e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:02.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:02.439+0000 7f4923400640 1 -- 172.21.15.45:0/3354772558 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f492c075780 con 0x7f4908074a30 2024-04-16T16:17:02.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:02.441+0000 7f492aa00640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4908074a30 0x7f4908076e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:02.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:02.459+0000 7f492aa00640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4908074a30 0x7f4908076e20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f491c006f30 tx=0x7f491c008040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:02.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:02.477+0000 7f4923400640 1 -- 172.21.15.45:0/3354772558 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7f492c075780 con 0x7f4908074a30 2024-04-16T16:17:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:17:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:02.480+0000 7f4931e00640 1 -- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4908074a30 msgr2=0x7f4908076e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:02.480+0000 7f4931e00640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4908074a30 0x7f4908076e20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f491c006f30 tx=0x7f491c008040 comp rx=0 tx=0).stop 2024-04-16T16:17:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:02.480+0000 7f4931e00640 1 -- 172.21.15.45:0/3354772558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f492c0b63c0 msgr2=0x7f492c0caae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:02.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:02.480+0000 7f4931e00640 1 --2- 172.21.15.45:0/3354772558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f492c0b63c0 0x7f492c0caae0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f491400c400 tx=0x7f491400c8c0 comp rx=0 tx=0).stop 2024-04-16T16:17:02.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:02.480+0000 7f492b400640 1 -- 172.21.15.45:0/3354772558 reap_dead start 2024-04-16T16:17:02.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:02.480+0000 7f4931e00640 1 -- 172.21.15.45:0/3354772558 shutdown_connections 2024-04-16T16:17:02.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:02.480+0000 7f4931e00640 1 -- 172.21.15.45:0/3354772558 >> 172.21.15.45:0/3354772558 conn(0x7f492c0b6ea0 msgr2=0x7f492c146b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:02.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:02.481+0000 7f4931e00640 1 -- 172.21.15.45:0/3354772558 shutdown_connections 2024-04-16T16:17:02.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:02.481+0000 7f4931e00640 1 -- 172.21.15.45:0/3354772558 wait complete. 2024-04-16T16:17:03.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:03.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.743+0000 7fbe9fe00640 1 Processor -- start 2024-04-16T16:17:03.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.743+0000 7fbe9fe00640 1 -- start start 2024-04-16T16:17:03.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.744+0000 7fbe9fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbea00c6fc0 0x7fbea006f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:03.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.745+0000 7fbe9fe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbea015d600 con 0x7fbea006df40 2024-04-16T16:17:03.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.745+0000 7fbe9fe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbea015d8e0 con 0x7fbea015cdc0 2024-04-16T16:17:03.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.745+0000 7fbe9fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbea00c5670 con 0x7fbea00c6fc0 2024-04-16T16:17:03.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.745+0000 7fbe9f400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbea006df40 0x7fbea00cf360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45116/0 (socket says 172.21.15.45:45116) 2024-04-16T16:17:03.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.745+0000 7fbe9f400640 1 -- 172.21.15.45:0/3694588386 learned_addr learned my addr 172.21.15.45:0/3694588386 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:03.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.745+0000 7fbe9ea00640 1 --2- 172.21.15.45:0/3694588386 >> v2:172.21.15.103:3300/0 conn(0x7fbea00c6fc0 0x7fbea006f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:03.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe9e000640 1 -- 172.21.15.45:0/3694588386 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3943416010 0 0) 0x7fbea015d600 con 0x7fbea006df40 2024-04-16T16:17:03.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe9e000640 1 -- 172.21.15.45:0/3694588386 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe84003520 con 0x7fbea006df40 2024-04-16T16:17:03.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe9e000640 1 -- 172.21.15.45:0/3694588386 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 93673882 0 0) 0x7fbe84003520 con 0x7fbea006df40 2024-04-16T16:17:03.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe9e000640 1 -- 172.21.15.45:0/3694588386 >> v2:172.21.15.103:3300/0 conn(0x7fbea00c6fc0 msgr2=0x7fbea006f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:03.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe9e000640 1 --2- 172.21.15.45:0/3694588386 >> v2:172.21.15.103:3300/0 conn(0x7fbea00c6fc0 0x7fbea006f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:03.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe9e000640 1 -- 172.21.15.45:0/3694588386 >> v1:172.21.15.103:6789/0 conn(0x7fbea015cdc0 legacy=0x7fbea00c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:03.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe9e000640 1 -- 172.21.15.45:0/3694588386 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea015dbc0 con 0x7fbea006df40 2024-04-16T16:17:03.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe9e000640 1 -- 172.21.15.45:0/3694588386 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fbe980019f0 con 0x7fbea006df40 2024-04-16T16:17:03.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe9e000640 1 -- 172.21.15.45:0/3694588386 >> v1:172.21.15.45:6789/0 conn(0x7fbea006df40 legacy=0x7fbea00cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:03.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe9e000640 1 --2- 172.21.15.45:0/3694588386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe840039b0 0x7fbe84003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:03.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe9e000640 1 --2- 172.21.15.45:0/3694588386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe84004800 0x7fbe84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:03.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe9e000640 1 --2- 172.21.15.45:0/3694588386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe840051c0 0x7fbe8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:03.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe9e000640 1 -- 172.21.15.45:0/3694588386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbea00c5670 con 0x7fbe84004800 2024-04-16T16:17:03.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe9e000640 1 -- 172.21.15.45:0/3694588386 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8400c0c0 con 0x7fbe840051c0 2024-04-16T16:17:03.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe9e000640 1 -- 172.21.15.45:0/3694588386 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbe8400c2c0 con 0x7fbe840039b0 2024-04-16T16:17:03.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.746+0000 7fbe97e00640 1 --2- 172.21.15.45:0/3694588386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe84004800 0x7fbe84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:03.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.747+0000 7fbe9ea00640 1 --2- 172.21.15.45:0/3694588386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe840039b0 0x7fbe84003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:03.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.747+0000 7fbe9f400640 1 --2- 172.21.15.45:0/3694588386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe840051c0 0x7fbe8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:03.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.747+0000 7fbe97e00640 1 -- 172.21.15.45:0/3694588386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe840039b0 msgr2=0x7fbe84003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:03.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.747+0000 7fbe97e00640 1 --2- 172.21.15.45:0/3694588386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe840039b0 0x7fbe84003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:03.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.747+0000 7fbe97e00640 1 -- 172.21.15.45:0/3694588386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe840051c0 msgr2=0x7fbe8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:03.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.747+0000 7fbe97e00640 1 --2- 172.21.15.45:0/3694588386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe840051c0 0x7fbe8400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:03.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.747+0000 7fbe9f400640 1 -- 172.21.15.45:0/3694588386 reap_dead start 2024-04-16T16:17:03.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.747+0000 7fbe97e00640 1 -- 172.21.15.45:0/3694588386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea015dbc0 con 0x7fbe84004800 2024-04-16T16:17:03.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.748+0000 7fbe97e00640 1 --2- 172.21.15.45:0/3694588386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe84004800 0x7fbe84004c80 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fbe88001750 tx=0x7fbe84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9cbfaea769c8ca server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:03.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.748+0000 7fbe9e000640 1 -- 172.21.15.45:0/3694588386 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe88001370 con 0x7fbe84004800 2024-04-16T16:17:03.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.748+0000 7fbe9e000640 1 -- 172.21.15.45:0/3694588386 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe880052d0 con 0x7fbe84004800 2024-04-16T16:17:03.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.748+0000 7fbe9e000640 1 -- 172.21.15.45:0/3694588386 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe88003830 con 0x7fbe84004800 2024-04-16T16:17:03.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.749+0000 7fbe9fe00640 1 -- 172.21.15.45:0/3694588386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe84004800 msgr2=0x7fbe84004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:03.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.749+0000 7fbe9fe00640 1 --2- 172.21.15.45:0/3694588386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe84004800 0x7fbe84004c80 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fbe88001750 tx=0x7fbe84005830 comp rx=0 tx=0).stop 2024-04-16T16:17:03.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.749+0000 7fbe9fe00640 1 -- 172.21.15.45:0/3694588386 shutdown_connections 2024-04-16T16:17:03.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.749+0000 7fbe9fe00640 1 --2- 172.21.15.45:0/3694588386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe84004800 0x7fbe84004c80 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:03.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.749+0000 7fbe9fe00640 1 -- 172.21.15.45:0/3694588386 >> 172.21.15.45:0/3694588386 conn(0x7fbea006b930 msgr2=0x7fbea006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:03.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.749+0000 7fbe9fe00640 1 -- 172.21.15.45:0/3694588386 shutdown_connections 2024-04-16T16:17:03.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.749+0000 7fbe9fe00640 1 -- 172.21.15.45:0/3694588386 wait complete. 2024-04-16T16:17:03.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.751+0000 7fbe9fe00640 1 Processor -- start 2024-04-16T16:17:03.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.751+0000 7fbe9fe00640 1 -- start start 2024-04-16T16:17:03.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.752+0000 7fbe9fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe84004800 0x7fbea0144e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:03.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.752+0000 7fbe9fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbea014ec90 0x7fbea01504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:03.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.752+0000 7fbe9fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbea0150a10 0x7fbea015ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:03.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.752+0000 7fbe9fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbea00c73a0 con 0x7fbe84004800 2024-04-16T16:17:03.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.752+0000 7fbe9fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbea015d270 con 0x7fbea0150a10 2024-04-16T16:17:03.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.752+0000 7fbe9fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbea00c7520 con 0x7fbea014ec90 2024-04-16T16:17:03.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.753+0000 7fbe9ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe84004800 0x7fbea0144e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.753+0000 7fbe9ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe84004800 0x7fbea0144e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44242/0 (socket says 172.21.15.45:44242) 2024-04-16T16:17:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.753+0000 7fbe9ea00640 1 -- 172.21.15.45:0/1105399658 learned_addr learned my addr 172.21.15.45:0/1105399658 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.753+0000 7fbe97e00640 1 --2- 172.21.15.45:0/1105399658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbea014ec90 0x7fbea01504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.753+0000 7fbe9f400640 1 --2- 172.21.15.45:0/1105399658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbea0150a10 0x7fbea015ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.753+0000 7fbe9ea00640 1 -- 172.21.15.45:0/1105399658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbea014ec90 msgr2=0x7fbea01504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.753+0000 7fbe9ea00640 1 --2- 172.21.15.45:0/1105399658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbea014ec90 0x7fbea01504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.753+0000 7fbe9ea00640 1 -- 172.21.15.45:0/1105399658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbea0150a10 msgr2=0x7fbea015ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.753+0000 7fbe9ea00640 1 --2- 172.21.15.45:0/1105399658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbea0150a10 0x7fbea015ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.753+0000 7fbe9ea00640 1 -- 172.21.15.45:0/1105399658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea01f7d90 con 0x7fbe84004800 2024-04-16T16:17:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.754+0000 7fbe9ea00640 1 --2- 172.21.15.45:0/1105399658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe84004800 0x7fbea0144e50 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fbe9000f560 tx=0x7fbe9000fa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:03.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.754+0000 7fbe96a00640 1 -- 172.21.15.45:0/1105399658 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe90019020 con 0x7fbe84004800 2024-04-16T16:17:03.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.754+0000 7fbe96a00640 1 -- 172.21.15.45:0/1105399658 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe90014870 con 0x7fbe84004800 2024-04-16T16:17:03.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.754+0000 7fbe96a00640 1 -- 172.21.15.45:0/1105399658 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe90014c60 con 0x7fbe84004800 2024-04-16T16:17:03.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.754+0000 7fbe9fe00640 1 -- 172.21.15.45:0/1105399658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbea01f8030 con 0x7fbe84004800 2024-04-16T16:17:03.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.754+0000 7fbe9fe00640 1 -- 172.21.15.45:0/1105399658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbea01f85f0 con 0x7fbe84004800 2024-04-16T16:17:03.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.755+0000 7fbe9fe00640 1 -- 172.21.15.45:0/1105399658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbea0068400 con 0x7fbe84004800 2024-04-16T16:17:03.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.756+0000 7fbe96a00640 1 -- 172.21.15.45:0/1105399658 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 91095+0+0 (secure 0 0 0) 0x7fbe9001c020 con 0x7fbe84004800 2024-04-16T16:17:03.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.757+0000 7fbe96a00640 1 --2- 172.21.15.45:0/1105399658 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fbe7c06c9d0 0x7fbe7c06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:03.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.757+0000 7fbe96a00640 1 -- 172.21.15.45:0/1105399658 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7fbe9008c610 con 0x7fbe84004800 2024-04-16T16:17:03.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.758+0000 7fbe97e00640 1 --2- 172.21.15.45:0/1105399658 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fbe7c06c9d0 0x7fbe7c06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:03.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.759+0000 7fbe97e00640 1 --2- 172.21.15.45:0/1105399658 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fbe7c06c9d0 0x7fbe7c06ee90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbe88033000 tx=0x7fbe88009030 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:03.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.763+0000 7fbe96a00640 1 -- 172.21.15.45:0/1105399658 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbea0068400 con 0x7fbe84004800 2024-04-16T16:17:03.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.992+0000 7fbe9fe00640 1 -- 172.21.15.45:0/1105399658 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7fbea0075780 con 0x7fbe7c06c9d0 2024-04-16T16:17:03.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.996+0000 7fbe96a00640 1 -- 172.21.15.45:0/1105399658 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7fbea0075780 con 0x7fbe7c06c9d0 2024-04-16T16:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:17:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.998+0000 7fbe9fe00640 1 -- 172.21.15.45:0/1105399658 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fbe7c06c9d0 msgr2=0x7fbe7c06ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.998+0000 7fbe9fe00640 1 --2- 172.21.15.45:0/1105399658 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fbe7c06c9d0 0x7fbe7c06ee90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbe88033000 tx=0x7fbe88009030 comp rx=0 tx=0).stop 2024-04-16T16:17:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.998+0000 7fbe9fe00640 1 -- 172.21.15.45:0/1105399658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe84004800 msgr2=0x7fbea0144e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:03.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.998+0000 7fbe9fe00640 1 --2- 172.21.15.45:0/1105399658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe84004800 0x7fbea0144e50 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fbe9000f560 tx=0x7fbe9000fa20 comp rx=0 tx=0).stop 2024-04-16T16:17:03.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.998+0000 7fbe9fe00640 1 -- 172.21.15.45:0/1105399658 shutdown_connections 2024-04-16T16:17:03.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.998+0000 7fbe9fe00640 1 --2- 172.21.15.45:0/1105399658 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fbe7c06c9d0 0x7fbe7c06ee90 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:03.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.998+0000 7fbe9fe00640 1 --2- 172.21.15.45:0/1105399658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbea0150a10 0x7fbea015ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:03.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.998+0000 7fbe9fe00640 1 --2- 172.21.15.45:0/1105399658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbea014ec90 0x7fbea01504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:03.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.998+0000 7fbe9fe00640 1 --2- 172.21.15.45:0/1105399658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe84004800 0x7fbea0144e50 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:03.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.998+0000 7fbe9fe00640 1 -- 172.21.15.45:0/1105399658 >> 172.21.15.45:0/1105399658 conn(0x7fbea006b930 msgr2=0x7fbea006ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:03.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.998+0000 7fbe9fe00640 1 -- 172.21.15.45:0/1105399658 shutdown_connections 2024-04-16T16:17:03.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:03.998+0000 7fbe9fe00640 1 -- 172.21.15.45:0/1105399658 wait complete. 2024-04-16T16:17:05.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:05.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.259+0000 7f6490a00640 1 Processor -- start 2024-04-16T16:17:05.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.259+0000 7f6490a00640 1 -- start start 2024-04-16T16:17:05.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.261+0000 7f6490a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c0cb7d0 0x7f648c0cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:05.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.261+0000 7f6490a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f648c0cd620 0x7f648c0cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:05.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.261+0000 7f6490a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f648c15f1b0 con 0x7f648c15e830 2024-04-16T16:17:05.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.261+0000 7f6490a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f648c0cf280 con 0x7f648c0cd620 2024-04-16T16:17:05.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.261+0000 7f6490a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f648c0ce190 con 0x7f648c0cb7d0 2024-04-16T16:17:05.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.261+0000 7f648aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c0cb7d0 0x7f648c0cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:05.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.262+0000 7f648b400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f648c15e830 0x7f648c0ce9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:47072/0 (socket says 172.21.15.45:47072) 2024-04-16T16:17:05.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.262+0000 7f648b400640 1 -- 172.21.15.45:0/379147180 learned_addr learned my addr 172.21.15.45:0/379147180 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:05.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.262+0000 7f648be00640 1 --2- 172.21.15.45:0/379147180 >> v2:172.21.15.103:3300/0 conn(0x7f648c0cd620 0x7f648c0cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:05.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.262+0000 7f648be00640 1 -- 172.21.15.45:0/379147180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c0cb7d0 msgr2=0x7f648c0cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:05.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.262+0000 7f648be00640 1 --2- 172.21.15.45:0/379147180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c0cb7d0 0x7f648c0cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:05.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.262+0000 7f648be00640 1 -- 172.21.15.45:0/379147180 >> v1:172.21.15.103:6789/0 conn(0x7f648c15e830 legacy=0x7f648c0ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:05.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.262+0000 7f648be00640 1 -- 172.21.15.45:0/379147180 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c0ce470 con 0x7f648c0cd620 2024-04-16T16:17:05.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.262+0000 7f648aa00640 1 --2- 172.21.15.45:0/379147180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c0cb7d0 0x7f648c0cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:17:05.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.264+0000 7f648be00640 1 --2- 172.21.15.45:0/379147180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f648c0cd620 0x7f648c0cda90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f64840097a0 tx=0x7f648402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c6691dc724ac07e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:05.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.264+0000 7f648a000640 1 -- 172.21.15.45:0/379147180 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6484031780 con 0x7f648c0cd620 2024-04-16T16:17:05.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.264+0000 7f648a000640 1 -- 172.21.15.45:0/379147180 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6484037070 con 0x7f648c0cd620 2024-04-16T16:17:05.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.264+0000 7f648a000640 1 -- 172.21.15.45:0/379147180 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6484031b40 con 0x7f648c0cd620 2024-04-16T16:17:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.265+0000 7f6490a00640 1 -- 172.21.15.45:0/379147180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f648c0cd620 msgr2=0x7f648c0cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.265+0000 7f6490a00640 1 --2- 172.21.15.45:0/379147180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f648c0cd620 0x7f648c0cda90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f64840097a0 tx=0x7f648402b820 comp rx=0 tx=0).stop 2024-04-16T16:17:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.265+0000 7f6490a00640 1 -- 172.21.15.45:0/379147180 shutdown_connections 2024-04-16T16:17:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.265+0000 7f6490a00640 1 --2- 172.21.15.45:0/379147180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f648c0cd620 0x7f648c0cda90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.265+0000 7f6490a00640 1 --2- 172.21.15.45:0/379147180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c0cb7d0 0x7f648c0cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.265+0000 7f6490a00640 1 -- 172.21.15.45:0/379147180 >> 172.21.15.45:0/379147180 conn(0x7f648c06b930 msgr2=0x7f648c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.265+0000 7f6490a00640 1 -- 172.21.15.45:0/379147180 shutdown_connections 2024-04-16T16:17:05.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.265+0000 7f6490a00640 1 -- 172.21.15.45:0/379147180 wait complete. 2024-04-16T16:17:05.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.267+0000 7f6490a00640 1 Processor -- start 2024-04-16T16:17:05.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.267+0000 7f6490a00640 1 -- start start 2024-04-16T16:17:05.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.268+0000 7f6490a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c0cb7d0 0x7f648c0b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:05.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.268+0000 7f6490a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f648c0cd620 0x7f648c1e70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:05.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.268+0000 7f6490a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c15e830 0x7f648c142df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:05.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.268+0000 7f6490a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f648c15ec10 con 0x7f648c0cb7d0 2024-04-16T16:17:05.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.268+0000 7f6490a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f648c0cb2b0 con 0x7f648c0cd620 2024-04-16T16:17:05.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.268+0000 7f6490a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f648c0cac30 con 0x7f648c15e830 2024-04-16T16:17:05.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.268+0000 7f648b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c0cb7d0 0x7f648c0b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:05.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.268+0000 7f648b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c0cb7d0 0x7f648c0b4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44246/0 (socket says 172.21.15.45:44246) 2024-04-16T16:17:05.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.268+0000 7f648b400640 1 -- 172.21.15.45:0/2797721000 learned_addr learned my addr 172.21.15.45:0/2797721000 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:05.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.269+0000 7f648aa00640 1 --2- 172.21.15.45:0/2797721000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f648c0cd620 0x7f648c1e70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:05.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.269+0000 7f648be00640 1 --2- 172.21.15.45:0/2797721000 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c15e830 0x7f648c142df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:05.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.269+0000 7f648b400640 1 -- 172.21.15.45:0/2797721000 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c15e830 msgr2=0x7f648c142df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:05.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.269+0000 7f648b400640 1 --2- 172.21.15.45:0/2797721000 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c15e830 0x7f648c142df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:05.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.269+0000 7f648b400640 1 -- 172.21.15.45:0/2797721000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f648c0cd620 msgr2=0x7f648c1e70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:05.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.269+0000 7f648b400640 1 --2- 172.21.15.45:0/2797721000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f648c0cd620 0x7f648c1e70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:05.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.269+0000 7f648b400640 1 -- 172.21.15.45:0/2797721000 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c158760 con 0x7f648c0cb7d0 2024-04-16T16:17:05.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.269+0000 7f648be00640 1 --2- 172.21.15.45:0/2797721000 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c15e830 0x7f648c142df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:05.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.269+0000 7f648b400640 1 --2- 172.21.15.45:0/2797721000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c0cb7d0 0x7f648c0b4310 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f647c00c400 tx=0x7f647c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:05.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.269+0000 7f6483400640 1 -- 172.21.15.45:0/2797721000 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c017070 con 0x7f648c0cb7d0 2024-04-16T16:17:05.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.269+0000 7f6483400640 1 -- 172.21.15.45:0/2797721000 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f647c005ce0 con 0x7f648c0cb7d0 2024-04-16T16:17:05.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.269+0000 7f6483400640 1 -- 172.21.15.45:0/2797721000 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c0072f0 con 0x7f648c0cb7d0 2024-04-16T16:17:05.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.269+0000 7f6490a00640 1 -- 172.21.15.45:0/2797721000 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f648c1e7a90 con 0x7f648c0cb7d0 2024-04-16T16:17:05.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.270+0000 7f6490a00640 1 -- 172.21.15.45:0/2797721000 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f648c158a10 con 0x7f648c0cb7d0 2024-04-16T16:17:05.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.271+0000 7f6490a00640 1 -- 172.21.15.45:0/2797721000 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f648c075780 con 0x7f648c0cb7d0 2024-04-16T16:17:05.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.272+0000 7f6483400640 1 -- 172.21.15.45:0/2797721000 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 91095+0+0 (secure 0 0 0) 0x7f647c01a070 con 0x7f648c0cb7d0 2024-04-16T16:17:05.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.273+0000 7f6483400640 1 --2- 172.21.15.45:0/2797721000 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f646c06c9d0 0x7f646c06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:05.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.273+0000 7f6483400640 1 -- 172.21.15.45:0/2797721000 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f647c08b210 con 0x7f648c0cb7d0 2024-04-16T16:17:05.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.274+0000 7f648aa00640 1 --2- 172.21.15.45:0/2797721000 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f646c06c9d0 0x7f646c06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:05.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.275+0000 7f648aa00640 1 --2- 172.21.15.45:0/2797721000 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f646c06c9d0 0x7f646c06ee90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6478007440 tx=0x7f647800b040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:05.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.277+0000 7f6483400640 1 -- 172.21.15.45:0/2797721000 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f648c15f1b0 con 0x7f648c0cb7d0 2024-04-16T16:17:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.514+0000 7f6483400640 1 -- 172.21.15.45:0/2797721000 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f647c0553a0 con 0x7f648c0cb7d0 2024-04-16T16:17:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.517+0000 7f6490a00640 1 -- 172.21.15.45:0/2797721000 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f648c0b5b90 con 0x7f646c06c9d0 2024-04-16T16:17:05.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.524+0000 7f6483400640 1 -- 172.21.15.45:0/2797721000 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7f648c0b5b90 con 0x7f646c06c9d0 2024-04-16T16:17:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:17:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.526+0000 7f6490a00640 1 -- 172.21.15.45:0/2797721000 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f646c06c9d0 msgr2=0x7f646c06ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.526+0000 7f6490a00640 1 --2- 172.21.15.45:0/2797721000 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f646c06c9d0 0x7f646c06ee90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6478007440 tx=0x7f647800b040 comp rx=0 tx=0).stop 2024-04-16T16:17:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.526+0000 7f6490a00640 1 -- 172.21.15.45:0/2797721000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c0cb7d0 msgr2=0x7f648c0b4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.526+0000 7f6490a00640 1 --2- 172.21.15.45:0/2797721000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c0cb7d0 0x7f648c0b4310 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f647c00c400 tx=0x7f647c00c8c0 comp rx=0 tx=0).stop 2024-04-16T16:17:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.527+0000 7f6490a00640 1 -- 172.21.15.45:0/2797721000 shutdown_connections 2024-04-16T16:17:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.527+0000 7f6490a00640 1 --2- 172.21.15.45:0/2797721000 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f646c06c9d0 0x7f646c06ee90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.527+0000 7f6490a00640 1 --2- 172.21.15.45:0/2797721000 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c15e830 0x7f648c142df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.527+0000 7f6490a00640 1 --2- 172.21.15.45:0/2797721000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f648c0cd620 0x7f648c1e70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.527+0000 7f6490a00640 1 --2- 172.21.15.45:0/2797721000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c0cb7d0 0x7f648c0b4310 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.527+0000 7f6490a00640 1 -- 172.21.15.45:0/2797721000 >> 172.21.15.45:0/2797721000 conn(0x7f648c06b930 msgr2=0x7f648c06f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.527+0000 7f6490a00640 1 -- 172.21.15.45:0/2797721000 shutdown_connections 2024-04-16T16:17:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:05.527+0000 7f6490a00640 1 -- 172.21.15.45:0/2797721000 wait complete. 2024-04-16T16:17:06.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:06.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.775+0000 7f5399c00640 1 Processor -- start 2024-04-16T16:17:06.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.775+0000 7f5399c00640 1 -- start start 2024-04-16T16:17:06.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5399c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53940cfc20 0x7f53940d37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:06.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5399c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53940d2600 con 0x7f5394162090 2024-04-16T16:17:06.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5399c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53940ba5d0 con 0x7f53940ba140 2024-04-16T16:17:06.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5399c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f53940ba8b0 con 0x7f53940cfc20 2024-04-16T16:17:06.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5393e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5394162090 0x7f53940cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45132/0 (socket says 172.21.15.45:45132) 2024-04-16T16:17:06.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5393e00640 1 -- 172.21.15.45:0/1307618535 learned_addr learned my addr 172.21.15.45:0/1307618535 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:06.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5399200640 1 --2- 172.21.15.45:0/1307618535 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53940cfc20 0x7f53940d37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:06.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5392a00640 1 -- 172.21.15.45:0/1307618535 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3757944518 0 0) 0x7f53940d2600 con 0x7f5394162090 2024-04-16T16:17:06.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5392a00640 1 -- 172.21.15.45:0/1307618535 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5378003520 con 0x7f5394162090 2024-04-16T16:17:06.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5392a00640 1 -- 172.21.15.45:0/1307618535 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 987246821 0 0) 0x7f5378003520 con 0x7f5394162090 2024-04-16T16:17:06.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5392a00640 1 -- 172.21.15.45:0/1307618535 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53940cfc20 msgr2=0x7f53940d37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:06.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5392a00640 1 --2- 172.21.15.45:0/1307618535 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53940cfc20 0x7f53940d37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:06.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5392a00640 1 -- 172.21.15.45:0/1307618535 >> v1:172.21.15.103:6789/0 conn(0x7f53940ba140 legacy=0x7f53940d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:06.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5392a00640 1 -- 172.21.15.45:0/1307618535 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53940d28a0 con 0x7f5394162090 2024-04-16T16:17:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5392a00640 1 -- 172.21.15.45:0/1307618535 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f5384001690 con 0x7f5394162090 2024-04-16T16:17:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5392a00640 1 -- 172.21.15.45:0/1307618535 >> v1:172.21.15.45:6789/0 conn(0x7f5394162090 legacy=0x7f53940cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5392a00640 1 --2- 172.21.15.45:0/1307618535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53780039e0 0x7f5378003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5392a00640 1 --2- 172.21.15.45:0/1307618535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5378004800 0x7f5378004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5392a00640 1 --2- 172.21.15.45:0/1307618535 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53940cfc20 0x7f53780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5392a00640 1 -- 172.21.15.45:0/1307618535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53940ba8b0 con 0x7f53780039e0 2024-04-16T16:17:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5392a00640 1 -- 172.21.15.45:0/1307618535 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f537800bad0 con 0x7f5378004800 2024-04-16T16:17:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.777+0000 7f5392a00640 1 -- 172.21.15.45:0/1307618535 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f537800bcd0 con 0x7f53940cfc20 2024-04-16T16:17:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.778+0000 7f5393e00640 1 --2- 172.21.15.45:0/1307618535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53780039e0 0x7f5378003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.778+0000 7f5393400640 1 --2- 172.21.15.45:0/1307618535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5378004800 0x7f5378004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.778+0000 7f5393e00640 1 -- 172.21.15.45:0/1307618535 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53940cfc20 msgr2=0x7f53780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.778+0000 7f5393e00640 1 --2- 172.21.15.45:0/1307618535 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53940cfc20 0x7f53780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.778+0000 7f5393e00640 1 -- 172.21.15.45:0/1307618535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5378004800 msgr2=0x7f5378004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.778+0000 7f5393e00640 1 --2- 172.21.15.45:0/1307618535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5378004800 0x7f5378004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.778+0000 7f5393e00640 1 -- 172.21.15.45:0/1307618535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5378003520 con 0x7f53780039e0 2024-04-16T16:17:06.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.778+0000 7f5393400640 1 --2- 172.21.15.45:0/1307618535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5378004800 0x7f5378004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:06.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.779+0000 7f5393e00640 1 --2- 172.21.15.45:0/1307618535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53780039e0 0x7f5378003e00 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5384001eb0 tx=0x7f5378005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1948c1ba8a2acaff server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.779+0000 7f5392a00640 1 -- 172.21.15.45:0/1307618535 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5384007c80 con 0x7f53780039e0 2024-04-16T16:17:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.780+0000 7f5392a00640 1 -- 172.21.15.45:0/1307618535 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5384030070 con 0x7f53780039e0 2024-04-16T16:17:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.780+0000 7f5392a00640 1 -- 172.21.15.45:0/1307618535 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5384005cf0 con 0x7f53780039e0 2024-04-16T16:17:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.780+0000 7f5399c00640 1 -- 172.21.15.45:0/1307618535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53780039e0 msgr2=0x7f5378003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.780+0000 7f5399c00640 1 --2- 172.21.15.45:0/1307618535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53780039e0 0x7f5378003e00 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5384001eb0 tx=0x7f5378005830 comp rx=0 tx=0).stop 2024-04-16T16:17:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.780+0000 7f5399200640 1 -- 172.21.15.45:0/1307618535 reap_dead start 2024-04-16T16:17:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.780+0000 7f5399c00640 1 -- 172.21.15.45:0/1307618535 shutdown_connections 2024-04-16T16:17:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.780+0000 7f5399c00640 1 -- 172.21.15.45:0/1307618535 >> 172.21.15.45:0/1307618535 conn(0x7f53940d1940 msgr2=0x7f53940d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.780+0000 7f5399c00640 1 -- 172.21.15.45:0/1307618535 shutdown_connections 2024-04-16T16:17:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.780+0000 7f5399c00640 1 -- 172.21.15.45:0/1307618535 wait complete. 2024-04-16T16:17:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.782+0000 7f5399c00640 1 Processor -- start 2024-04-16T16:17:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.782+0000 7f5399c00640 1 -- start start 2024-04-16T16:17:06.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.783+0000 7f5399c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53780039e0 0x7f53940cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:06.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.783+0000 7f5399c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53940ce540 0x7f5394157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:06.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.783+0000 7f5399c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5394154970 0x7f53940b59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:06.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.783+0000 7f5399c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53941625f0 con 0x7f53780039e0 2024-04-16T16:17:06.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.783+0000 7f5399c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5394162470 con 0x7f5394154970 2024-04-16T16:17:06.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.783+0000 7f5399c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f53940d5520 con 0x7f53940ce540 2024-04-16T16:17:06.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.783+0000 7f5393e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53780039e0 0x7f53940cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:06.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.783+0000 7f5393e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53780039e0 0x7f53940cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44264/0 (socket says 172.21.15.45:44264) 2024-04-16T16:17:06.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.783+0000 7f5393e00640 1 -- 172.21.15.45:0/242220062 learned_addr learned my addr 172.21.15.45:0/242220062 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:06.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.784+0000 7f5399200640 1 --2- 172.21.15.45:0/242220062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5394154970 0x7f53940b59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:06.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.784+0000 7f5393400640 1 --2- 172.21.15.45:0/242220062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53940ce540 0x7f5394157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:06.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.784+0000 7f5393e00640 1 -- 172.21.15.45:0/242220062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53940ce540 msgr2=0x7f5394157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:06.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.784+0000 7f5393e00640 1 --2- 172.21.15.45:0/242220062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53940ce540 0x7f5394157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:06.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.784+0000 7f5393e00640 1 -- 172.21.15.45:0/242220062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5394154970 msgr2=0x7f53940b59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:06.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.784+0000 7f5393e00640 1 --2- 172.21.15.45:0/242220062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5394154970 0x7f53940b59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:06.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.784+0000 7f5393e00640 1 -- 172.21.15.45:0/242220062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5394157b40 con 0x7f53780039e0 2024-04-16T16:17:06.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.784+0000 7f5393e00640 1 --2- 172.21.15.45:0/242220062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53780039e0 0x7f53940cd930 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f538402c430 tx=0x7f5384001990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:06.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.785+0000 7f538be00640 1 -- 172.21.15.45:0/242220062 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538402cbd0 con 0x7f53780039e0 2024-04-16T16:17:06.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.785+0000 7f538be00640 1 -- 172.21.15.45:0/242220062 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5384000ed0 con 0x7f53780039e0 2024-04-16T16:17:06.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.785+0000 7f538be00640 1 -- 172.21.15.45:0/242220062 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5384007c60 con 0x7f53780039e0 2024-04-16T16:17:06.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.785+0000 7f5399c00640 1 -- 172.21.15.45:0/242220062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53940c8460 con 0x7f53780039e0 2024-04-16T16:17:06.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.785+0000 7f5399c00640 1 -- 172.21.15.45:0/242220062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53940c8a20 con 0x7f53780039e0 2024-04-16T16:17:06.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.786+0000 7f5399c00640 1 -- 172.21.15.45:0/242220062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5394075780 con 0x7f53780039e0 2024-04-16T16:17:06.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.787+0000 7f538be00640 1 -- 172.21.15.45:0/242220062 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f5384047020 con 0x7f53780039e0 2024-04-16T16:17:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.788+0000 7f538be00640 1 --2- 172.21.15.45:0/242220062 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f537406ca60 0x7f537406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.789+0000 7f538be00640 1 -- 172.21.15.45:0/242220062 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f53840ad230 con 0x7f53780039e0 2024-04-16T16:17:06.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.789+0000 7f5393400640 1 --2- 172.21.15.45:0/242220062 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f537406ca60 0x7f537406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:06.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.790+0000 7f5393400640 1 --2- 172.21.15.45:0/242220062 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f537406ca60 0x7f537406ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f539406ccc0 tx=0x7f5380009040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:06.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:06.794+0000 7f538be00640 1 -- 172.21.15.45:0/242220062 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f538407b390 con 0x7f53780039e0 2024-04-16T16:17:07.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:07.025+0000 7f5399c00640 1 -- 172.21.15.45:0/242220062 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f539406dab0 con 0x7f537406ca60 2024-04-16T16:17:07.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:07.029+0000 7f538be00640 1 -- 172.21.15.45:0/242220062 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7f539406dab0 con 0x7f537406ca60 2024-04-16T16:17:07.029 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:17:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:07.031+0000 7f5399c00640 1 -- 172.21.15.45:0/242220062 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f537406ca60 msgr2=0x7f537406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:07.031+0000 7f5399c00640 1 --2- 172.21.15.45:0/242220062 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f537406ca60 0x7f537406ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f539406ccc0 tx=0x7f5380009040 comp rx=0 tx=0).stop 2024-04-16T16:17:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:07.031+0000 7f5399c00640 1 -- 172.21.15.45:0/242220062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53780039e0 msgr2=0x7f53940cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:07.031+0000 7f5399c00640 1 --2- 172.21.15.45:0/242220062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53780039e0 0x7f53940cd930 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f538402c430 tx=0x7f5384001990 comp rx=0 tx=0).stop 2024-04-16T16:17:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:07.031+0000 7f5399c00640 1 -- 172.21.15.45:0/242220062 shutdown_connections 2024-04-16T16:17:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:07.031+0000 7f5399c00640 1 --2- 172.21.15.45:0/242220062 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f537406ca60 0x7f537406ef20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:07.031+0000 7f5399c00640 1 --2- 172.21.15.45:0/242220062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5394154970 0x7f53940b59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:07.031+0000 7f5399c00640 1 --2- 172.21.15.45:0/242220062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53940ce540 0x7f5394157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:07.031+0000 7f5399c00640 1 --2- 172.21.15.45:0/242220062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53780039e0 0x7f53940cd930 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:07.031+0000 7f5399c00640 1 -- 172.21.15.45:0/242220062 >> 172.21.15.45:0/242220062 conn(0x7f53940d1940 msgr2=0x7f53940b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:07.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:07.032+0000 7f5399c00640 1 -- 172.21.15.45:0/242220062 shutdown_connections 2024-04-16T16:17:07.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:07.032+0000 7f5399c00640 1 -- 172.21.15.45:0/242220062 wait complete. 2024-04-16T16:17:08.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:08.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.284+0000 7fa733000640 1 Processor -- start 2024-04-16T16:17:08.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.284+0000 7fa733000640 1 -- start start 2024-04-16T16:17:08.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.286+0000 7fa733000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa72c0afe20 0x7fa72c06f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:08.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.286+0000 7fa733000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa72c06f550 0x7fa72c0cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:08.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.286+0000 7fa733000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa72c06fab0 con 0x7fa72c15ca00 2024-04-16T16:17:08.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.286+0000 7fa733000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa72c15bf20 con 0x7fa72c0afe20 2024-04-16T16:17:08.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.286+0000 7fa733000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa72c15c200 con 0x7fa72c06f550 2024-04-16T16:17:08.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.287+0000 7fa72be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa72c0afe20 0x7fa72c06f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:08.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.287+0000 7fa72be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa72c0afe20 0x7fa72c06f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44274/0 (socket says 172.21.15.45:44274) 2024-04-16T16:17:08.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.287+0000 7fa72be00640 1 -- 172.21.15.45:0/1808596473 learned_addr learned my addr 172.21.15.45:0/1808596473 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:08.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.287+0000 7fa72b400640 1 --2- 172.21.15.45:0/1808596473 >> v2:172.21.15.103:3300/0 conn(0x7fa72c06f550 0x7fa72c0cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:08.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.287+0000 7fa72be00640 1 -- 172.21.15.45:0/1808596473 >> v2:172.21.15.103:3300/0 conn(0x7fa72c06f550 msgr2=0x7fa72c0cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:08.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.287+0000 7fa72be00640 1 --2- 172.21.15.45:0/1808596473 >> v2:172.21.15.103:3300/0 conn(0x7fa72c06f550 0x7fa72c0cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.287+0000 7fa72be00640 1 -- 172.21.15.45:0/1808596473 >> v1:172.21.15.103:6789/0 conn(0x7fa72c15ca00 legacy=0x7fa72c0b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:08.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.287+0000 7fa72be00640 1 -- 172.21.15.45:0/1808596473 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa72c15c4e0 con 0x7fa72c0afe20 2024-04-16T16:17:08.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.288+0000 7fa72be00640 1 --2- 172.21.15.45:0/1808596473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa72c0afe20 0x7fa72c06f010 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fa72c06f9c0 tx=0x7fa71c02e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f46fe94bf0c86 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:08.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.289+0000 7fa72aa00640 1 -- 172.21.15.45:0/1808596473 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71c02ea90 con 0x7fa72c0afe20 2024-04-16T16:17:08.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.289+0000 7fa72aa00640 1 -- 172.21.15.45:0/1808596473 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa71c02ec30 con 0x7fa72c0afe20 2024-04-16T16:17:08.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.289+0000 7fa72aa00640 1 -- 172.21.15.45:0/1808596473 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71c009600 con 0x7fa72c0afe20 2024-04-16T16:17:08.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.290+0000 7fa733000640 1 -- 172.21.15.45:0/1808596473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa72c0afe20 msgr2=0x7fa72c06f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:08.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.290+0000 7fa733000640 1 --2- 172.21.15.45:0/1808596473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa72c0afe20 0x7fa72c06f010 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fa72c06f9c0 tx=0x7fa71c02e0f0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.290+0000 7fa733000640 1 -- 172.21.15.45:0/1808596473 shutdown_connections 2024-04-16T16:17:08.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.290+0000 7fa733000640 1 --2- 172.21.15.45:0/1808596473 >> v2:172.21.15.103:3300/0 conn(0x7fa72c06f550 0x7fa72c0cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.290+0000 7fa733000640 1 --2- 172.21.15.45:0/1808596473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa72c0afe20 0x7fa72c06f010 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.290+0000 7fa733000640 1 -- 172.21.15.45:0/1808596473 >> 172.21.15.45:0/1808596473 conn(0x7fa72c06b930 msgr2=0x7fa72c06bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:08.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.290+0000 7fa733000640 1 -- 172.21.15.45:0/1808596473 shutdown_connections 2024-04-16T16:17:08.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.290+0000 7fa733000640 1 -- 172.21.15.45:0/1808596473 wait complete. 2024-04-16T16:17:08.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.292+0000 7fa733000640 1 Processor -- start 2024-04-16T16:17:08.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.292+0000 7fa733000640 1 -- start start 2024-04-16T16:17:08.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.293+0000 7fa733000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa72c06f550 0x7fa72c0b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:08.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.293+0000 7fa733000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa72c0afe20 0x7fa72c1481d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:08.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.293+0000 7fa733000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa72c15ca00 0x7fa72c0b2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.293+0000 7fa733000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa72c0b59d0 con 0x7fa72c06f550 2024-04-16T16:17:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.293+0000 7fa733000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa72c0b5850 con 0x7fa72c0afe20 2024-04-16T16:17:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.293+0000 7fa733000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa72c0cde10 con 0x7fa72c15ca00 2024-04-16T16:17:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.293+0000 7fa72be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa72c06f550 0x7fa72c0b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.293+0000 7fa72be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa72c06f550 0x7fa72c0b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44290/0 (socket says 172.21.15.45:44290) 2024-04-16T16:17:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.293+0000 7fa72be00640 1 -- 172.21.15.45:0/680751198 learned_addr learned my addr 172.21.15.45:0/680751198 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.294+0000 7fa72b400640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa72c0afe20 0x7fa72c1481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.294+0000 7fa731000640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa72c15ca00 0x7fa72c0b2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.294+0000 7fa72be00640 1 -- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa72c15ca00 msgr2=0x7fa72c0b2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.294+0000 7fa72be00640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa72c15ca00 0x7fa72c0b2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.294+0000 7fa72be00640 1 -- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa72c0afe20 msgr2=0x7fa72c1481d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.294+0000 7fa72be00640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa72c0afe20 0x7fa72c1481d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.294+0000 7fa72be00640 1 -- 172.21.15.45:0/680751198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa72c0b2ce0 con 0x7fa72c06f550 2024-04-16T16:17:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.294+0000 7fa731000640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa72c15ca00 0x7fa72c0b2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:08.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.294+0000 7fa72b400640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa72c0afe20 0x7fa72c1481d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:08.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.294+0000 7fa72be00640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa72c06f550 0x7fa72c0b46c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fa71c038040 tx=0x7fa71c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:08.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.294+0000 7fa723e00640 1 -- 172.21.15.45:0/680751198 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71c00b720 con 0x7fa72c06f550 2024-04-16T16:17:08.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.294+0000 7fa723e00640 1 -- 172.21.15.45:0/680751198 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa71c045070 con 0x7fa72c06f550 2024-04-16T16:17:08.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.294+0000 7fa723e00640 1 -- 172.21.15.45:0/680751198 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71c02ea90 con 0x7fa72c06f550 2024-04-16T16:17:08.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.294+0000 7fa733000640 1 -- 172.21.15.45:0/680751198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa72c0b2fc0 con 0x7fa72c06f550 2024-04-16T16:17:08.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.295+0000 7fa733000640 1 -- 172.21.15.45:0/680751198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa72c0b3580 con 0x7fa72c06f550 2024-04-16T16:17:08.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.296+0000 7fa733000640 1 -- 172.21.15.45:0/680751198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa72c075780 con 0x7fa72c06f550 2024-04-16T16:17:08.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.297+0000 7fa723e00640 1 -- 172.21.15.45:0/680751198 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7fa71c036a60 con 0x7fa72c06f550 2024-04-16T16:17:08.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.298+0000 7fa723e00640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fa70806ca60 0x7fa70806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:08.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.298+0000 7fa723e00640 1 -- 172.21.15.45:0/680751198 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7fa71c0aeee0 con 0x7fa72c06f550 2024-04-16T16:17:08.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.298+0000 7fa72b400640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fa70806ca60 0x7fa70806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:08.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.300+0000 7fa72b400640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fa70806ca60 0x7fa70806ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa72c153ba0 tx=0x7fa714008040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:08.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.302+0000 7fa723e00640 1 -- 172.21.15.45:0/680751198 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa71c07cfc0 con 0x7fa72c06f550 2024-04-16T16:17:08.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.530+0000 7fa733000640 1 -- 172.21.15.45:0/680751198 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7fa72c0b38d0 con 0x7fa70806ca60 2024-04-16T16:17:08.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.534+0000 7fa723e00640 1 -- 172.21.15.45:0/680751198 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+2961 (secure 0 0 0) 0x7fa72c0b38d0 con 0x7fa70806ca60 2024-04-16T16:17:08.533 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[5, 19], [0, 0], [0, 0], [0, 21230274], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 884601], [17953202520157, 24]], "client.5285": [[2, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[2, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[24, 0], [0, 0], [0, 0], [0, 12985265], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3246317], [51775632992079, 4]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}}} 2024-04-16T16:17:08.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.537+0000 7fa733000640 1 -- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fa70806ca60 msgr2=0x7fa70806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:08.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.537+0000 7fa733000640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fa70806ca60 0x7fa70806ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa72c153ba0 tx=0x7fa714008040 comp rx=0 tx=0).stop 2024-04-16T16:17:08.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.537+0000 7fa733000640 1 -- 172.21.15.45:0/680751198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa72c06f550 msgr2=0x7fa72c0b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:08.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.537+0000 7fa733000640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa72c06f550 0x7fa72c0b46c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fa71c038040 tx=0x7fa71c004a40 comp rx=0 tx=0).stop 2024-04-16T16:17:08.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.537+0000 7fa733000640 1 -- 172.21.15.45:0/680751198 shutdown_connections 2024-04-16T16:17:08.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.537+0000 7fa733000640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fa70806ca60 0x7fa70806ef20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.537+0000 7fa733000640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa72c15ca00 0x7fa72c0b2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.537+0000 7fa733000640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa72c0afe20 0x7fa72c1481d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.537+0000 7fa733000640 1 --2- 172.21.15.45:0/680751198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa72c06f550 0x7fa72c0b46c0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.537+0000 7fa733000640 1 -- 172.21.15.45:0/680751198 >> 172.21.15.45:0/680751198 conn(0x7fa72c06b930 msgr2=0x7fa72c06df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:08.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.537+0000 7fa733000640 1 -- 172.21.15.45:0/680751198 shutdown_connections 2024-04-16T16:17:08.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.537+0000 7fa733000640 1 -- 172.21.15.45:0/680751198 wait complete. 2024-04-16T16:17:08.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:17:08.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.773+0000 7fe55f000640 1 Processor -- start 2024-04-16T16:17:08.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.773+0000 7fe55f000640 1 -- start start 2024-04-16T16:17:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.775+0000 7fe55f000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe55815b730 0x7fe5580cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.775+0000 7fe55f000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe5580d0230 0x7fe5580b4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.775+0000 7fe55f000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5580ce5b0 con 0x7fe5580b5840 2024-04-16T16:17:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.775+0000 7fe55f000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe558072420 con 0x7fe5580d0230 2024-04-16T16:17:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.775+0000 7fe55f000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe558072700 con 0x7fe55815b730 2024-04-16T16:17:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.775+0000 7fe55d000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe5580d0230 0x7fe5580b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.775+0000 7fe55d000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe5580d0230 0x7fe5580b4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44304/0 (socket says 172.21.15.45:44304) 2024-04-16T16:17:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.775+0000 7fe55d000640 1 -- 172.21.15.45:0/2398811539 learned_addr learned my addr 172.21.15.45:0/2398811539 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.776+0000 7fe557e00640 1 --2- 172.21.15.45:0/2398811539 >> v2:172.21.15.103:3300/0 conn(0x7fe55815b730 0x7fe5580cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.776+0000 7fe55d000640 1 -- 172.21.15.45:0/2398811539 >> v2:172.21.15.103:3300/0 conn(0x7fe55815b730 msgr2=0x7fe5580cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.776+0000 7fe55d000640 1 --2- 172.21.15.45:0/2398811539 >> v2:172.21.15.103:3300/0 conn(0x7fe55815b730 0x7fe5580cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.776+0000 7fe55d000640 1 -- 172.21.15.45:0/2398811539 >> v1:172.21.15.103:6789/0 conn(0x7fe5580b5840 legacy=0x7fe5580cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.776+0000 7fe55d000640 1 -- 172.21.15.45:0/2398811539 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5580729e0 con 0x7fe5580d0230 2024-04-16T16:17:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.776+0000 7fe557e00640 1 --2- 172.21.15.45:0/2398811539 >> v2:172.21.15.103:3300/0 conn(0x7fe55815b730 0x7fe5580cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:08.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.777+0000 7fe55d000640 1 --2- 172.21.15.45:0/2398811539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5580d0230 0x7fe5580b4400 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fe5580b3600 tx=0x7fe55002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde62219a7fbf97f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:08.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.777+0000 7fe556a00640 1 -- 172.21.15.45:0/2398811539 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55002ca80 con 0x7fe5580d0230 2024-04-16T16:17:08.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.777+0000 7fe556a00640 1 -- 172.21.15.45:0/2398811539 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe55002cc20 con 0x7fe5580d0230 2024-04-16T16:17:08.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.777+0000 7fe556a00640 1 -- 172.21.15.45:0/2398811539 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5500343f0 con 0x7fe5580d0230 2024-04-16T16:17:08.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.778+0000 7fe55f000640 1 -- 172.21.15.45:0/2398811539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5580d0230 msgr2=0x7fe5580b4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:08.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.778+0000 7fe55f000640 1 --2- 172.21.15.45:0/2398811539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5580d0230 0x7fe5580b4400 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fe5580b3600 tx=0x7fe55002bee0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.778+0000 7fe55f000640 1 -- 172.21.15.45:0/2398811539 shutdown_connections 2024-04-16T16:17:08.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.778+0000 7fe55f000640 1 --2- 172.21.15.45:0/2398811539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5580d0230 0x7fe5580b4400 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.778+0000 7fe55f000640 1 --2- 172.21.15.45:0/2398811539 >> v2:172.21.15.103:3300/0 conn(0x7fe55815b730 0x7fe5580cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.778+0000 7fe55f000640 1 -- 172.21.15.45:0/2398811539 >> 172.21.15.45:0/2398811539 conn(0x7fe5580b2750 msgr2=0x7fe5580b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:08.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.778+0000 7fe55f000640 1 -- 172.21.15.45:0/2398811539 shutdown_connections 2024-04-16T16:17:08.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.778+0000 7fe55f000640 1 -- 172.21.15.45:0/2398811539 wait complete. 2024-04-16T16:17:08.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.780+0000 7fe55f000640 1 Processor -- start 2024-04-16T16:17:08.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.780+0000 7fe55f000640 1 -- start start 2024-04-16T16:17:08.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.781+0000 7fe55f000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5580b5840 0x7fe55813a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:08.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.781+0000 7fe55f000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5580d0230 0x7fe558158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:08.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.781+0000 7fe55f000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe55815b730 0x7fe55814d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:08.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.781+0000 7fe55f000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe55815bb10 con 0x7fe5580d0230 2024-04-16T16:17:08.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.781+0000 7fe55f000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe5580b5ca0 con 0x7fe5580b5840 2024-04-16T16:17:08.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.781+0000 7fe55f000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe5580731e0 con 0x7fe55815b730 2024-04-16T16:17:08.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.782+0000 7fe557400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5580d0230 0x7fe558158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:08.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.782+0000 7fe557400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5580d0230 0x7fe558158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44308/0 (socket says 172.21.15.45:44308) 2024-04-16T16:17:08.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.782+0000 7fe557400640 1 -- 172.21.15.45:0/3521721907 learned_addr learned my addr 172.21.15.45:0/3521721907 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:08.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.782+0000 7fe55d000640 1 --2- 172.21.15.45:0/3521721907 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe55815b730 0x7fe55814d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:08.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.782+0000 7fe557e00640 1 --2- 172.21.15.45:0/3521721907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5580b5840 0x7fe55813a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:08.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.782+0000 7fe557400640 1 -- 172.21.15.45:0/3521721907 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe55815b730 msgr2=0x7fe55814d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:08.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.782+0000 7fe557400640 1 --2- 172.21.15.45:0/3521721907 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe55815b730 0x7fe55814d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.782+0000 7fe557400640 1 -- 172.21.15.45:0/3521721907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5580b5840 msgr2=0x7fe55813a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:08.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.782+0000 7fe557400640 1 --2- 172.21.15.45:0/3521721907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5580b5840 0x7fe55813a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:08.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.782+0000 7fe557400640 1 -- 172.21.15.45:0/3521721907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5581593e0 con 0x7fe5580d0230 2024-04-16T16:17:08.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.782+0000 7fe55d000640 1 --2- 172.21.15.45:0/3521721907 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe55815b730 0x7fe55814d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:17:08.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.782+0000 7fe557400640 1 --2- 172.21.15.45:0/3521721907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5580d0230 0x7fe558158d20 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fe54400b420 tx=0x7fe54400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:08.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.783+0000 7fe54fe00640 1 -- 172.21.15.45:0/3521721907 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe544011830 con 0x7fe5580d0230 2024-04-16T16:17:08.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.783+0000 7fe54fe00640 1 -- 172.21.15.45:0/3521721907 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe544016070 con 0x7fe5580d0230 2024-04-16T16:17:08.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.783+0000 7fe54fe00640 1 -- 172.21.15.45:0/3521721907 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe544011c10 con 0x7fe5580d0230 2024-04-16T16:17:08.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.783+0000 7fe55f000640 1 -- 172.21.15.45:0/3521721907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5581596c0 con 0x7fe5580d0230 2024-04-16T16:17:08.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.783+0000 7fe55f000640 1 -- 172.21.15.45:0/3521721907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe558159c80 con 0x7fe5580d0230 2024-04-16T16:17:08.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.784+0000 7fe55f000640 1 -- 172.21.15.45:0/3521721907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe558075780 con 0x7fe5580d0230 2024-04-16T16:17:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.785+0000 7fe54fe00640 1 -- 172.21.15.45:0/3521721907 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7fe54401b460 con 0x7fe5580d0230 2024-04-16T16:17:08.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.786+0000 7fe54fe00640 1 --2- 172.21.15.45:0/3521721907 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fe53806ca60 0x7fe53806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:08.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.786+0000 7fe54fe00640 1 -- 172.21.15.45:0/3521721907 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7fe54408b9d0 con 0x7fe5580d0230 2024-04-16T16:17:08.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.787+0000 7fe557e00640 1 --2- 172.21.15.45:0/3521721907 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fe53806ca60 0x7fe53806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:08.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.788+0000 7fe557e00640 1 --2- 172.21.15.45:0/3521721907 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fe53806ca60 0x7fe53806ef20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe54800a990 tx=0x7fe548008040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:08.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:08.790+0000 7fe54fe00640 1 -- 172.21.15.45:0/3521721907 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe544020020 con 0x7fe5580d0230 2024-04-16T16:17:09.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.037+0000 7fe55f000640 1 -- 172.21.15.45:0/3521721907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5580b3d00 con 0x7fe5580d0230 2024-04-16T16:17:09.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.038+0000 7fe54fe00640 1 -- 172.21.15.45:0/3521721907 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5180 (secure 0 0 0) 0x7fe544059b30 con 0x7fe5580d0230 2024-04-16T16:17:09.037 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:09.038 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":50,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":6399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:16:42.595621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-04-16T16:17:09.038 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 50 2024-04-16T16:17:09.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.041+0000 7fe55f000640 1 -- 172.21.15.45:0/3521721907 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fe53806ca60 msgr2=0x7fe53806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:09.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.041+0000 7fe55f000640 1 --2- 172.21.15.45:0/3521721907 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fe53806ca60 0x7fe53806ef20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe54800a990 tx=0x7fe548008040 comp rx=0 tx=0).stop 2024-04-16T16:17:09.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.041+0000 7fe55f000640 1 -- 172.21.15.45:0/3521721907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5580d0230 msgr2=0x7fe558158d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:09.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.041+0000 7fe55f000640 1 --2- 172.21.15.45:0/3521721907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5580d0230 0x7fe558158d20 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fe54400b420 tx=0x7fe54400b8e0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.041+0000 7fe55f000640 1 -- 172.21.15.45:0/3521721907 shutdown_connections 2024-04-16T16:17:09.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.041+0000 7fe55f000640 1 --2- 172.21.15.45:0/3521721907 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fe53806ca60 0x7fe53806ef20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.041+0000 7fe55f000640 1 --2- 172.21.15.45:0/3521721907 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe55815b730 0x7fe55814d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.041+0000 7fe55f000640 1 --2- 172.21.15.45:0/3521721907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5580d0230 0x7fe558158d20 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.041+0000 7fe55f000640 1 --2- 172.21.15.45:0/3521721907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5580b5840 0x7fe55813a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.041+0000 7fe55f000640 1 -- 172.21.15.45:0/3521721907 >> 172.21.15.45:0/3521721907 conn(0x7fe5580b2750 msgr2=0x7fe55806b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:09.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.041+0000 7fe55f000640 1 -- 172.21.15.45:0/3521721907 shutdown_connections 2024-04-16T16:17:09.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.041+0000 7fe55f000640 1 -- 172.21.15.45:0/3521721907 wait complete. 2024-04-16T16:17:09.059 DEBUG:tasks.cephfs.test_mds_metrics:metrics={'version': 2, 'global_counters': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'counters': [], 'client_metadata': {'cephfs': {'client.5243': {'hostname': 'smithi045', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '172.21.15.45'}, 'client.5285': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'IP': '172.21.15.103'}, 'client.5288': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'IP': '172.21.15.103'}, 'client.6591': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '172.21.15.103'}}}, 'global_metrics': {'cephfs': {'client.5243': [[5, 19], [0, 0], [0, 0], [0, 21230274], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 884601], [17953202520157, 24]], 'client.5285': [[2, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], 'client.5288': [[2, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], 'client.6591': [[24, 0], [0, 0], [0, 0], [0, 12985265], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3246317], [51775632992079, 4]]}}, 'metrics': {'delayed_ranks': [], 'mds.0': {'client.5243': [], 'client.5285': [], 'client.5288': [], 'client.6591': []}}} 2024-04-16T16:17:09.059 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:17:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.257+0000 7f11b8000640 1 Processor -- start 2024-04-16T16:17:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.257+0000 7f11b8000640 1 -- start start 2024-04-16T16:17:09.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.259+0000 7f11b8000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f11b01462e0 0x7f11b0072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:09.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.259+0000 7f11b8000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11b00728c0 0x7f11b00ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:09.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.259+0000 7f11b8000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f11b00cec00 0x7f11b015da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:09.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.259+0000 7f11b8000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f11b0072d20 con 0x7f11b01462e0 2024-04-16T16:17:09.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.259+0000 7f11b8000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f11b00cf6d0 con 0x7f11b00cec00 2024-04-16T16:17:09.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.259+0000 7f11b8000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f11b01466c0 con 0x7f11b00728c0 2024-04-16T16:17:09.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.259+0000 7f11b5600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f11b01462e0 0x7f11b0072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:09.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.259+0000 7f11b5600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f11b01462e0 0x7f11b0072380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44316/0 (socket says 172.21.15.45:44316) 2024-04-16T16:17:09.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.259+0000 7f11b5600640 1 -- 172.21.15.45:0/970718761 learned_addr learned my addr 172.21.15.45:0/970718761 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:09.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.260+0000 7f11b4c00640 1 --2- 172.21.15.45:0/970718761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11b00728c0 0x7f11b00ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:09.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.260+0000 7f11b6000640 1 --2- 172.21.15.45:0/970718761 >> v2:172.21.15.103:3300/0 conn(0x7f11b00cec00 0x7f11b015da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:09.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.260+0000 7f11b5600640 1 -- 172.21.15.45:0/970718761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11b00728c0 msgr2=0x7f11b00ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:09.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.260+0000 7f11b5600640 1 --2- 172.21.15.45:0/970718761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11b00728c0 0x7f11b00ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.260+0000 7f11b5600640 1 -- 172.21.15.45:0/970718761 >> v2:172.21.15.103:3300/0 conn(0x7f11b00cec00 msgr2=0x7f11b015da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:09.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.260+0000 7f11b5600640 1 --2- 172.21.15.45:0/970718761 >> v2:172.21.15.103:3300/0 conn(0x7f11b00cec00 0x7f11b015da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.260+0000 7f11b5600640 1 -- 172.21.15.45:0/970718761 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b00cf3f0 con 0x7f11b01462e0 2024-04-16T16:17:09.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.260+0000 7f11b6000640 1 --2- 172.21.15.45:0/970718761 >> v2:172.21.15.103:3300/0 conn(0x7f11b00cec00 0x7f11b015da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:17:09.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.261+0000 7f11b5600640 1 --2- 172.21.15.45:0/970718761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b01462e0 0x7f11b0072380 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f11b006f9e0 tx=0x7f11a402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71535dd7eb2064ed server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:09.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.262+0000 7f11afe00640 1 -- 172.21.15.45:0/970718761 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a402e070 con 0x7f11b01462e0 2024-04-16T16:17:09.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.262+0000 7f11afe00640 1 -- 172.21.15.45:0/970718761 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11a4037070 con 0x7f11b01462e0 2024-04-16T16:17:09.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.262+0000 7f11afe00640 1 -- 172.21.15.45:0/970718761 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a4031a00 con 0x7f11b01462e0 2024-04-16T16:17:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.262+0000 7f11b8000640 1 -- 172.21.15.45:0/970718761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b01462e0 msgr2=0x7f11b0072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.262+0000 7f11b8000640 1 --2- 172.21.15.45:0/970718761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b01462e0 0x7f11b0072380 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f11b006f9e0 tx=0x7f11a402b880 comp rx=0 tx=0).stop 2024-04-16T16:17:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.263+0000 7f11b8000640 1 -- 172.21.15.45:0/970718761 shutdown_connections 2024-04-16T16:17:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.263+0000 7f11b8000640 1 --2- 172.21.15.45:0/970718761 >> v2:172.21.15.103:3300/0 conn(0x7f11b00cec00 0x7f11b015da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.263+0000 7f11b8000640 1 --2- 172.21.15.45:0/970718761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11b00728c0 0x7f11b00ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.263+0000 7f11b8000640 1 --2- 172.21.15.45:0/970718761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b01462e0 0x7f11b0072380 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.263+0000 7f11b8000640 1 -- 172.21.15.45:0/970718761 >> 172.21.15.45:0/970718761 conn(0x7f11b00b2750 msgr2=0x7f11b00b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.263+0000 7f11b8000640 1 -- 172.21.15.45:0/970718761 shutdown_connections 2024-04-16T16:17:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.263+0000 7f11b8000640 1 -- 172.21.15.45:0/970718761 wait complete. 2024-04-16T16:17:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.264+0000 7f11b8000640 1 Processor -- start 2024-04-16T16:17:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.265+0000 7f11b8000640 1 -- start start 2024-04-16T16:17:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.265+0000 7f11b8000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11b00728c0 0x7f11b015c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.265+0000 7f11b8000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b00cec00 0x7f11b0141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.265+0000 7f11b8000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11b01462e0 0x7f11b0142100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.266+0000 7f11b8000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f11b015d540 con 0x7f11b00cec00 2024-04-16T16:17:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.266+0000 7f11b8000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f11b015d260 con 0x7f11b01462e0 2024-04-16T16:17:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.266+0000 7f11b8000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f11b00cf090 con 0x7f11b00728c0 2024-04-16T16:17:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.266+0000 7f11b4c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b00cec00 0x7f11b0141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.266+0000 7f11b4c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b00cec00 0x7f11b0141bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44332/0 (socket says 172.21.15.45:44332) 2024-04-16T16:17:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.266+0000 7f11b4c00640 1 -- 172.21.15.45:0/3505845490 learned_addr learned my addr 172.21.15.45:0/3505845490 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:09.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.266+0000 7f11b5600640 1 --2- 172.21.15.45:0/3505845490 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11b00728c0 0x7f11b015c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:09.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.266+0000 7f11b4c00640 1 -- 172.21.15.45:0/3505845490 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11b00728c0 msgr2=0x7f11b015c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:09.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.266+0000 7f11b6000640 1 --2- 172.21.15.45:0/3505845490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11b01462e0 0x7f11b0142100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:09.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.266+0000 7f11b4c00640 1 --2- 172.21.15.45:0/3505845490 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11b00728c0 0x7f11b015c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.266+0000 7f11b4c00640 1 -- 172.21.15.45:0/3505845490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11b01462e0 msgr2=0x7f11b0142100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.266+0000 7f11b4c00640 1 --2- 172.21.15.45:0/3505845490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11b01462e0 0x7f11b0142100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.266+0000 7f11b4c00640 1 -- 172.21.15.45:0/3505845490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b01f1070 con 0x7f11b00cec00 2024-04-16T16:17:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.267+0000 7f11b4c00640 1 --2- 172.21.15.45:0/3505845490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b00cec00 0x7f11b0141bc0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f119c00d620 tx=0x7f119c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.267+0000 7f11ad600640 1 -- 172.21.15.45:0/3505845490 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119c00b620 con 0x7f11b00cec00 2024-04-16T16:17:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.267+0000 7f11ad600640 1 -- 172.21.15.45:0/3505845490 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f119c019070 con 0x7f11b00cec00 2024-04-16T16:17:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.267+0000 7f11ad600640 1 -- 172.21.15.45:0/3505845490 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119c00ba00 con 0x7f11b00cec00 2024-04-16T16:17:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.267+0000 7f11b8000640 1 -- 172.21.15.45:0/3505845490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11b01f1350 con 0x7f11b00cec00 2024-04-16T16:17:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.267+0000 7f11b8000640 1 -- 172.21.15.45:0/3505845490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11b01f1910 con 0x7f11b00cec00 2024-04-16T16:17:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.268+0000 7f11b8000640 1 -- 172.21.15.45:0/3505845490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11b0075780 con 0x7f11b00cec00 2024-04-16T16:17:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.269+0000 7f11ad600640 1 -- 172.21.15.45:0/3505845490 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f119c00bba0 con 0x7f11b00cec00 2024-04-16T16:17:09.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.270+0000 7f11ad600640 1 --2- 172.21.15.45:0/3505845490 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f119006ca60 0x7f119006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:09.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.270+0000 7f11ad600640 1 -- 172.21.15.45:0/3505845490 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f119c08c2c0 con 0x7f11b00cec00 2024-04-16T16:17:09.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.271+0000 7f11b5600640 1 --2- 172.21.15.45:0/3505845490 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f119006ca60 0x7f119006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:09.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.272+0000 7f11b5600640 1 --2- 172.21.15.45:0/3505845490 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f119006ca60 0x7f119006ef20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f11a4036000 tx=0x7f11a4008030 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:09.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.274+0000 7f11ad600640 1 -- 172.21.15.45:0/3505845490 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f119c05a4c0 con 0x7f11b00cec00 2024-04-16T16:17:09.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.509+0000 7f11b8000640 1 -- 172.21.15.45:0/3505845490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11b00b5a20 con 0x7f11b00cec00 2024-04-16T16:17:09.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.510+0000 7f11ad600640 1 -- 172.21.15.45:0/3505845490 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5180 (secure 0 0 0) 0x7f119c08c0e0 con 0x7f11b00cec00 2024-04-16T16:17:09.510 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:09.510 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":50,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":6399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:16:42.595621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-04-16T16:17:09.510 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 50 2024-04-16T16:17:09.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.513+0000 7f11b8000640 1 -- 172.21.15.45:0/3505845490 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f119006ca60 msgr2=0x7f119006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:09.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.513+0000 7f11b8000640 1 --2- 172.21.15.45:0/3505845490 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f119006ca60 0x7f119006ef20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f11a4036000 tx=0x7f11a4008030 comp rx=0 tx=0).stop 2024-04-16T16:17:09.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.513+0000 7f11b8000640 1 -- 172.21.15.45:0/3505845490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b00cec00 msgr2=0x7f11b0141bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:09.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.513+0000 7f11b8000640 1 --2- 172.21.15.45:0/3505845490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b00cec00 0x7f11b0141bc0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f119c00d620 tx=0x7f119c00dae0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.514+0000 7f11b8000640 1 -- 172.21.15.45:0/3505845490 shutdown_connections 2024-04-16T16:17:09.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.514+0000 7f11b8000640 1 --2- 172.21.15.45:0/3505845490 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f119006ca60 0x7f119006ef20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.514+0000 7f11b8000640 1 --2- 172.21.15.45:0/3505845490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11b01462e0 0x7f11b0142100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.514+0000 7f11b8000640 1 --2- 172.21.15.45:0/3505845490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b00cec00 0x7f11b0141bc0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.514+0000 7f11b8000640 1 --2- 172.21.15.45:0/3505845490 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11b00728c0 0x7f11b015c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.514+0000 7f11b8000640 1 -- 172.21.15.45:0/3505845490 >> 172.21.15.45:0/3505845490 conn(0x7f11b00b2750 msgr2=0x7f11b006fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:09.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.514+0000 7f11b8000640 1 -- 172.21.15.45:0/3505845490 shutdown_connections 2024-04-16T16:17:09.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.514+0000 7f11b8000640 1 -- 172.21.15.45:0/3505845490 wait complete. 2024-04-16T16:17:09.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-04-16T16:17:09.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.750+0000 7f3e0fe00640 1 Processor -- start 2024-04-16T16:17:09.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.751+0000 7f3e0fe00640 1 -- start start 2024-04-16T16:17:09.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.752+0000 7f3e0fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e100d18f0 0x7f3e100ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:09.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.752+0000 7f3e0fe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e100b6810 con 0x7f3e100d1cd0 2024-04-16T16:17:09.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.752+0000 7f3e0fe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e100b6af0 con 0x7f3e100b6310 2024-04-16T16:17:09.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.752+0000 7f3e0fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3e100d29f0 con 0x7f3e100d18f0 2024-04-16T16:17:09.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.753+0000 7f3e0f400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3e100d1cd0 0x7f3e10162dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45144/0 (socket says 172.21.15.45:45144) 2024-04-16T16:17:09.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.753+0000 7f3e0f400640 1 -- 172.21.15.45:0/3299625351 learned_addr learned my addr 172.21.15.45:0/3299625351 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:09.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.753+0000 7f3e0ea00640 1 --2- 172.21.15.45:0/3299625351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e100d18f0 0x7f3e100ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:09.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.753+0000 7f3e0d600640 1 -- 172.21.15.45:0/3299625351 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1285366540 0 0) 0x7f3e100b6810 con 0x7f3e100d1cd0 2024-04-16T16:17:09.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.753+0000 7f3e0d600640 1 -- 172.21.15.45:0/3299625351 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3df0003520 con 0x7f3e100d1cd0 2024-04-16T16:17:09.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.753+0000 7f3e0d600640 1 -- 172.21.15.45:0/3299625351 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1120404285 0 0) 0x7f3df0003520 con 0x7f3e100d1cd0 2024-04-16T16:17:09.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.753+0000 7f3e0d600640 1 -- 172.21.15.45:0/3299625351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e100d18f0 msgr2=0x7f3e100ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:09.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.753+0000 7f3e0d600640 1 --2- 172.21.15.45:0/3299625351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e100d18f0 0x7f3e100ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.753+0000 7f3e0d600640 1 -- 172.21.15.45:0/3299625351 >> v1:172.21.15.103:6789/0 conn(0x7f3e100b6310 legacy=0x7f3e100d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:09.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.753+0000 7f3e0d600640 1 -- 172.21.15.45:0/3299625351 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e100b6dd0 con 0x7f3e100d1cd0 2024-04-16T16:17:09.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.753+0000 7f3e0d600640 1 -- 172.21.15.45:0/3299625351 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f3e08001670 con 0x7f3e100d1cd0 2024-04-16T16:17:09.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.754+0000 7f3e0d600640 1 -- 172.21.15.45:0/3299625351 >> v1:172.21.15.45:6789/0 conn(0x7f3e100d1cd0 legacy=0x7f3e10162dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:09.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.754+0000 7f3e0d600640 1 --2- 172.21.15.45:0/3299625351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3df00039e0 0x7f3df0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:09.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.754+0000 7f3e0d600640 1 --2- 172.21.15.45:0/3299625351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3df0004800 0x7f3df0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:09.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.754+0000 7f3e0d600640 1 --2- 172.21.15.45:0/3299625351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e100d18f0 0x7f3df0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:09.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.754+0000 7f3e0d600640 1 -- 172.21.15.45:0/3299625351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3e100d29f0 con 0x7f3df0004800 2024-04-16T16:17:09.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.754+0000 7f3e0d600640 1 -- 172.21.15.45:0/3299625351 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3df000baf0 con 0x7f3df00039e0 2024-04-16T16:17:09.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.754+0000 7f3e0d600640 1 -- 172.21.15.45:0/3299625351 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3df000bcf0 con 0x7f3e100d18f0 2024-04-16T16:17:09.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.754+0000 7f3e0e000640 1 --2- 172.21.15.45:0/3299625351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3df0004800 0x7f3df0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:09.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.754+0000 7f3e0f400640 1 --2- 172.21.15.45:0/3299625351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e100d18f0 0x7f3df0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:09.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.754+0000 7f3e0ea00640 1 --2- 172.21.15.45:0/3299625351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3df00039e0 0x7f3df0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:09.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.754+0000 7f3e0e000640 1 -- 172.21.15.45:0/3299625351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e100d18f0 msgr2=0x7f3df0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:09.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.754+0000 7f3e0e000640 1 --2- 172.21.15.45:0/3299625351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e100d18f0 0x7f3df0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.755+0000 7f3e0e000640 1 -- 172.21.15.45:0/3299625351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3df00039e0 msgr2=0x7f3df0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:09.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.755+0000 7f3e0e000640 1 --2- 172.21.15.45:0/3299625351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3df00039e0 0x7f3df0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.755+0000 7f3e0e000640 1 -- 172.21.15.45:0/3299625351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df0003520 con 0x7f3df0004800 2024-04-16T16:17:09.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.756+0000 7f3e0e000640 1 --2- 172.21.15.45:0/3299625351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3df0004800 0x7f3df0004c80 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f3df8001e50 tx=0x7f3df0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e16b540d1549f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:09.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.756+0000 7f3e0d600640 1 -- 172.21.15.45:0/3299625351 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8001a70 con 0x7f3df0004800 2024-04-16T16:17:09.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.756+0000 7f3e0d600640 1 -- 172.21.15.45:0/3299625351 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3df8003b70 con 0x7f3df0004800 2024-04-16T16:17:09.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.756+0000 7f3e0d600640 1 -- 172.21.15.45:0/3299625351 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df80050d0 con 0x7f3df0004800 2024-04-16T16:17:09.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.756+0000 7f3e0fe00640 1 -- 172.21.15.45:0/3299625351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3df0004800 msgr2=0x7f3df0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:09.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.756+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/3299625351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3df0004800 0x7f3df0004c80 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f3df8001e50 tx=0x7f3df0005830 comp rx=0 tx=0).stop 2024-04-16T16:17:09.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.756+0000 7f3e0f400640 1 -- 172.21.15.45:0/3299625351 reap_dead start 2024-04-16T16:17:09.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.757+0000 7f3e0fe00640 1 -- 172.21.15.45:0/3299625351 shutdown_connections 2024-04-16T16:17:09.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.757+0000 7f3e0fe00640 1 -- 172.21.15.45:0/3299625351 >> 172.21.15.45:0/3299625351 conn(0x7f3e1006cf90 msgr2=0x7f3e1006d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:09.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.757+0000 7f3e0fe00640 1 -- 172.21.15.45:0/3299625351 shutdown_connections 2024-04-16T16:17:09.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.757+0000 7f3e0fe00640 1 -- 172.21.15.45:0/3299625351 wait complete. 2024-04-16T16:17:09.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.759+0000 7f3e0fe00640 1 Processor -- start 2024-04-16T16:17:09.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.759+0000 7f3e0fe00640 1 -- start start 2024-04-16T16:17:09.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.760+0000 7f3e0fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3df0004800 0x7f3e10157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:09.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.760+0000 7f3e0fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e100cdf70 0x7f3e1015c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:09.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.760+0000 7f3e0fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e101f4a60 0x7f3e1015d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:09.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.760+0000 7f3e0fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3e100ba3a0 con 0x7f3e100cdf70 2024-04-16T16:17:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.760+0000 7f3e0fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3e100ba220 con 0x7f3e101f4a60 2024-04-16T16:17:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.760+0000 7f3e0fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3e100ba520 con 0x7f3df0004800 2024-04-16T16:17:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.760+0000 7f3e0e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e100cdf70 0x7f3e1015c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.760+0000 7f3e0e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e100cdf70 0x7f3e1015c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44352/0 (socket says 172.21.15.45:44352) 2024-04-16T16:17:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.760+0000 7f3e0e000640 1 -- 172.21.15.45:0/2000644459 learned_addr learned my addr 172.21.15.45:0/2000644459 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.760+0000 7f3e0ea00640 1 --2- 172.21.15.45:0/2000644459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3df0004800 0x7f3e10157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.760+0000 7f3e0f400640 1 --2- 172.21.15.45:0/2000644459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e101f4a60 0x7f3e1015d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.761+0000 7f3e0e000640 1 -- 172.21.15.45:0/2000644459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3df0004800 msgr2=0x7f3e10157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.761+0000 7f3e0e000640 1 --2- 172.21.15.45:0/2000644459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3df0004800 0x7f3e10157cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.761+0000 7f3e0e000640 1 -- 172.21.15.45:0/2000644459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e101f4a60 msgr2=0x7f3e1015d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.761+0000 7f3e0e000640 1 --2- 172.21.15.45:0/2000644459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e101f4a60 0x7f3e1015d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.761+0000 7f3e0e000640 1 -- 172.21.15.45:0/2000644459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e100cba00 con 0x7f3e100cdf70 2024-04-16T16:17:09.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.761+0000 7f3e0e000640 1 --2- 172.21.15.45:0/2000644459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e100cdf70 0x7f3e1015c860 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f3df8001f70 tx=0x7f3df80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:09.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.761+0000 7f3e06a00640 1 -- 172.21.15.45:0/2000644459 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8003970 con 0x7f3e100cdf70 2024-04-16T16:17:09.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.761+0000 7f3e06a00640 1 -- 172.21.15.45:0/2000644459 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3df8006b50 con 0x7f3e100cdf70 2024-04-16T16:17:09.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.761+0000 7f3e06a00640 1 -- 172.21.15.45:0/2000644459 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8007030 con 0x7f3e100cdf70 2024-04-16T16:17:09.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.761+0000 7f3e0fe00640 1 -- 172.21.15.45:0/2000644459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e100cbce0 con 0x7f3e100cdf70 2024-04-16T16:17:09.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.761+0000 7f3e0fe00640 1 -- 172.21.15.45:0/2000644459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e100cc2a0 con 0x7f3e100cdf70 2024-04-16T16:17:09.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.762+0000 7f3e0fe00640 1 -- 172.21.15.45:0/2000644459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e100b3f60 con 0x7f3e100cdf70 2024-04-16T16:17:09.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.763+0000 7f3e06a00640 1 -- 172.21.15.45:0/2000644459 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f3df80071d0 con 0x7f3e100cdf70 2024-04-16T16:17:09.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.764+0000 7f3e06a00640 1 --2- 172.21.15.45:0/2000644459 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f3dec06ca60 0x7f3dec06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:09.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.764+0000 7f3e06a00640 1 -- 172.21.15.45:0/2000644459 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f3df80add00 con 0x7f3e100cdf70 2024-04-16T16:17:09.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.765+0000 7f3e0ea00640 1 --2- 172.21.15.45:0/2000644459 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f3dec06ca60 0x7f3dec06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:09.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.766+0000 7f3e0ea00640 1 --2- 172.21.15.45:0/2000644459 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f3dec06ca60 0x7f3dec06ef20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009a10 tx=0x7f3e00008040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:09.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:09.769+0000 7f3e06a00640 1 -- 172.21.15.45:0/2000644459 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3e100b3f60 con 0x7f3e100cdf70 2024-04-16T16:17:10.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.003+0000 7f3e0fe00640 1 -- 172.21.15.45:0/2000644459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) -- 0x7f3e10075780 con 0x7f3e100cdf70 2024-04-16T16:17:10.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.554+0000 7f3e06a00640 1 -- 172.21.15.45:0/2000644459 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v51) ==== 105+0+0 (secure 0 0 0) 0x7f3df8033020 con 0x7f3e100cdf70 2024-04-16T16:17:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.558+0000 7f3e0fe00640 1 -- 172.21.15.45:0/2000644459 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f3dec06ca60 msgr2=0x7f3dec06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.558+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/2000644459 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f3dec06ca60 0x7f3dec06ef20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009a10 tx=0x7f3e00008040 comp rx=0 tx=0).stop 2024-04-16T16:17:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.558+0000 7f3e0fe00640 1 -- 172.21.15.45:0/2000644459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e100cdf70 msgr2=0x7f3e1015c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.558+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/2000644459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e100cdf70 0x7f3e1015c860 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f3df8001f70 tx=0x7f3df80014a0 comp rx=0 tx=0).stop 2024-04-16T16:17:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.558+0000 7f3e0fe00640 1 -- 172.21.15.45:0/2000644459 shutdown_connections 2024-04-16T16:17:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.558+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/2000644459 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f3dec06ca60 0x7f3dec06ef20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.558+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/2000644459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e101f4a60 0x7f3e1015d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.558+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/2000644459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e100cdf70 0x7f3e1015c860 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.558+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/2000644459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3df0004800 0x7f3e10157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.558+0000 7f3e0fe00640 1 -- 172.21.15.45:0/2000644459 >> 172.21.15.45:0/2000644459 conn(0x7f3e1006cf90 msgr2=0x7f3e100b3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.559+0000 7f3e0fe00640 1 -- 172.21.15.45:0/2000644459 shutdown_connections 2024-04-16T16:17:10.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.559+0000 7f3e0fe00640 1 -- 172.21.15.45:0/2000644459 wait complete. 2024-04-16T16:17:10.590 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:17:10.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.778+0000 7fda55800640 1 Processor -- start 2024-04-16T16:17:10.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.778+0000 7fda55800640 1 -- start start 2024-04-16T16:17:10.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.780+0000 7fda55800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fda500d52d0 0x7fda5006ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:10.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.780+0000 7fda55800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda500d3080 con 0x7fda50162270 2024-04-16T16:17:10.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.780+0000 7fda55800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda500d3360 con 0x7fda50161e90 2024-04-16T16:17:10.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.780+0000 7fda55800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fda500d2e30 con 0x7fda500d52d0 2024-04-16T16:17:10.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.780+0000 7fda47400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fda50162270 0x7fda500d3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45148/0 (socket says 172.21.15.45:45148) 2024-04-16T16:17:10.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.780+0000 7fda47400640 1 -- 172.21.15.45:0/2402991231 learned_addr learned my addr 172.21.15.45:0/2402991231 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:10.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.780+0000 7fda54e00640 1 --2- 172.21.15.45:0/2402991231 >> v2:172.21.15.45:3300/0 conn(0x7fda500d52d0 0x7fda5006ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:10.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.781+0000 7fda54e00640 1 -- 172.21.15.45:0/2402991231 >> v1:172.21.15.103:6789/0 conn(0x7fda50161e90 legacy=0x7fda500cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:10.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.781+0000 7fda54e00640 1 -- 172.21.15.45:0/2402991231 >> v1:172.21.15.45:6789/0 conn(0x7fda50162270 legacy=0x7fda500d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:10.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.781+0000 7fda54e00640 1 -- 172.21.15.45:0/2402991231 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda500d3640 con 0x7fda500d52d0 2024-04-16T16:17:10.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.781+0000 7fda4f400640 1 -- 172.21.15.45:0/2402991231 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 465252527 0 0) 0x7fda500d3080 con 0x7fda50162270 2024-04-16T16:17:10.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.782+0000 7fda54e00640 1 --2- 172.21.15.45:0/2402991231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda500d52d0 0x7fda5006ce40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fda48009dc0 tx=0x7fda4802bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574acb7901bf1c1d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:10.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.782+0000 7fda4f400640 1 -- 172.21.15.45:0/2402991231 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48039070 con 0x7fda500d52d0 2024-04-16T16:17:10.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.782+0000 7fda4f400640 1 -- 172.21.15.45:0/2402991231 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda4802ccd0 con 0x7fda500d52d0 2024-04-16T16:17:10.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.782+0000 7fda4f400640 1 -- 172.21.15.45:0/2402991231 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48034660 con 0x7fda500d52d0 2024-04-16T16:17:10.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.783+0000 7fda55800640 1 -- 172.21.15.45:0/2402991231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda500d52d0 msgr2=0x7fda5006ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:10.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.783+0000 7fda55800640 1 --2- 172.21.15.45:0/2402991231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda500d52d0 0x7fda5006ce40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fda48009dc0 tx=0x7fda4802bf20 comp rx=0 tx=0).stop 2024-04-16T16:17:10.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.783+0000 7fda55800640 1 -- 172.21.15.45:0/2402991231 shutdown_connections 2024-04-16T16:17:10.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.783+0000 7fda55800640 1 --2- 172.21.15.45:0/2402991231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda500d52d0 0x7fda5006ce40 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:10.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.783+0000 7fda55800640 1 -- 172.21.15.45:0/2402991231 >> 172.21.15.45:0/2402991231 conn(0x7fda500d1940 msgr2=0x7fda500d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:10.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.783+0000 7fda55800640 1 -- 172.21.15.45:0/2402991231 shutdown_connections 2024-04-16T16:17:10.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.783+0000 7fda55800640 1 -- 172.21.15.45:0/2402991231 wait complete. 2024-04-16T16:17:10.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.785+0000 7fda55800640 1 Processor -- start 2024-04-16T16:17:10.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.785+0000 7fda55800640 1 -- start start 2024-04-16T16:17:10.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.786+0000 7fda55800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda500d52d0 0x7fda501f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:10.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.786+0000 7fda55800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda50161e90 0x7fda5015d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:10.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.786+0000 7fda55800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda50162270 0x7fda500b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:10.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.786+0000 7fda55800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fda500d46d0 con 0x7fda50161e90 2024-04-16T16:17:10.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.786+0000 7fda55800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda500d4550 con 0x7fda50162270 2024-04-16T16:17:10.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.786+0000 7fda55800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda5006d380 con 0x7fda500d52d0 2024-04-16T16:17:10.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.786+0000 7fda47400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda50161e90 0x7fda5015d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:10.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.786+0000 7fda47400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda50161e90 0x7fda5015d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44376/0 (socket says 172.21.15.45:44376) 2024-04-16T16:17:10.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.786+0000 7fda47400640 1 -- 172.21.15.45:0/3764796558 learned_addr learned my addr 172.21.15.45:0/3764796558 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:10.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.786+0000 7fda54e00640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda50162270 0x7fda500b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:10.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.786+0000 7fda4fe00640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda500d52d0 0x7fda501f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:10.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.787+0000 7fda47400640 1 -- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda500d52d0 msgr2=0x7fda501f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:10.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.787+0000 7fda47400640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda500d52d0 0x7fda501f3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:10.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.787+0000 7fda47400640 1 -- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda50162270 msgr2=0x7fda500b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:10.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.787+0000 7fda47400640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda50162270 0x7fda500b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:10.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.787+0000 7fda47400640 1 -- 172.21.15.45:0/3764796558 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda501f43e0 con 0x7fda50161e90 2024-04-16T16:17:10.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.787+0000 7fda54e00640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda50162270 0x7fda500b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:10.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.787+0000 7fda4fe00640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda500d52d0 0x7fda501f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:10.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.787+0000 7fda47400640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda50161e90 0x7fda5015d1f0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fda3800c3c0 tx=0x7fda3800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:10.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.787+0000 7fda4cc00640 1 -- 172.21.15.45:0/3764796558 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda380074e0 con 0x7fda50161e90 2024-04-16T16:17:10.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.787+0000 7fda4cc00640 1 -- 172.21.15.45:0/3764796558 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda380047e0 con 0x7fda50161e90 2024-04-16T16:17:10.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.787+0000 7fda4cc00640 1 -- 172.21.15.45:0/3764796558 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda38004bd0 con 0x7fda50161e90 2024-04-16T16:17:10.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.787+0000 7fda55800640 1 -- 172.21.15.45:0/3764796558 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda5015dc90 con 0x7fda50161e90 2024-04-16T16:17:10.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.787+0000 7fda55800640 1 -- 172.21.15.45:0/3764796558 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda5015e250 con 0x7fda50161e90 2024-04-16T16:17:10.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.789+0000 7fda55800640 1 -- 172.21.15.45:0/3764796558 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda50075780 con 0x7fda50161e90 2024-04-16T16:17:10.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.789+0000 7fda4cc00640 1 -- 172.21.15.45:0/3764796558 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7fda38011070 con 0x7fda50161e90 2024-04-16T16:17:10.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.790+0000 7fda4cc00640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fda2c06ca60 0x7fda2c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:10.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.790+0000 7fda4cc00640 1 -- 172.21.15.45:0/3764796558 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7fda3808ba10 con 0x7fda50161e90 2024-04-16T16:17:10.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.791+0000 7fda4fe00640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fda2c06ca60 0x7fda2c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:10.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.792+0000 7fda4fe00640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fda2c06ca60 0x7fda2c06ef20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fda40000f60 tx=0x7fda40009040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:10.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:10.795+0000 7fda4cc00640 1 -- 172.21.15.45:0/3764796558 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda38059c10 con 0x7fda50161e90 2024-04-16T16:17:11.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:11.029+0000 7fda55800640 1 -- 172.21.15.45:0/3764796558 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda500cae10 con 0x7fda50161e90 2024-04-16T16:17:11.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:11.030+0000 7fda4cc00640 1 -- 172.21.15.45:0/3764796558 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5196 (secure 0 0 0) 0x7fda3808b030 con 0x7fda50161e90 2024-04-16T16:17:11.029 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:11.029 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:17:10.559154+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6402,"mds_1":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"c","rank":1,"incarnation":52,"state":"up:creating","state_seq":1,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-04-16T16:17:11.029 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 52 2024-04-16T16:17:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:11.032+0000 7fda55800640 1 -- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fda2c06ca60 msgr2=0x7fda2c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:11.032+0000 7fda55800640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fda2c06ca60 0x7fda2c06ef20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fda40000f60 tx=0x7fda40009040 comp rx=0 tx=0).stop 2024-04-16T16:17:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:11.032+0000 7fda55800640 1 -- 172.21.15.45:0/3764796558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda50161e90 msgr2=0x7fda5015d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:11.032+0000 7fda55800640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda50161e90 0x7fda5015d1f0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fda3800c3c0 tx=0x7fda3800c880 comp rx=0 tx=0).stop 2024-04-16T16:17:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:11.033+0000 7fda55800640 1 -- 172.21.15.45:0/3764796558 shutdown_connections 2024-04-16T16:17:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:11.033+0000 7fda55800640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fda2c06ca60 0x7fda2c06ef20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:11.033+0000 7fda55800640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda50162270 0x7fda500b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:11.033+0000 7fda55800640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda50161e90 0x7fda5015d1f0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:11.033+0000 7fda55800640 1 --2- 172.21.15.45:0/3764796558 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda500d52d0 0x7fda501f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:11.033+0000 7fda55800640 1 -- 172.21.15.45:0/3764796558 >> 172.21.15.45:0/3764796558 conn(0x7fda500d1940 msgr2=0x7fda500d2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:11.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:11.033+0000 7fda55800640 1 -- 172.21.15.45:0/3764796558 shutdown_connections 2024-04-16T16:17:11.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:11.033+0000 7fda55800640 1 -- 172.21.15.45:0/3764796558 wait complete. 2024-04-16T16:17:11.455 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:16:39.513364+0000', 'modified': '2024-04-16T16:17:10.559154+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 6402, 'mds_1': 6399}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6399': {'gid': 6399, 'name': 'c', 'rank': 1, 'incarnation': 52, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.45:6834/883721507', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6832', 'nonce': 883721507}, {'type': 'v1', 'addr': '172.21.15.45:6834', 'nonce': 883721507}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_6402': {'gid': 6402, 'name': 'a', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6835/268877398', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6833', 'nonce': 268877398}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 268877398}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6402, 'qdb_cluster': [6402]} 2024-04-16T16:17:11.455 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6399:up:creating 2024-04-16T16:17:12.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:17:12.465 INFO:tasks.ceph.mds.a.smithi045.stderr:2024-04-16T16:17:12.461+0000 7fbe89a00640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-16T16:17:12.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.686+0000 7f17b3e00640 1 Processor -- start 2024-04-16T16:17:12.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.687+0000 7f17b3e00640 1 -- start start 2024-04-16T16:17:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.688+0000 7f17b3e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f17b4158a40 0x7f17b415be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.688+0000 7f17b3e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17b40cfff0 con 0x7f17b406eee0 2024-04-16T16:17:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.688+0000 7f17b3e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17b40d02d0 con 0x7f17b40b2de0 2024-04-16T16:17:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.688+0000 7f17b3e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f17b415c890 con 0x7f17b4158a40 2024-04-16T16:17:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.688+0000 7f17b2a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f17b4158a40 0x7f17b415be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.688+0000 7f17b3400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f17b406eee0 0x7f17b415cd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45158/0 (socket says 172.21.15.45:45158) 2024-04-16T16:17:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.688+0000 7f17b3400640 1 -- 172.21.15.45:0/6963520 learned_addr learned my addr 172.21.15.45:0/6963520 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.689+0000 7f17b2a00640 1 -- 172.21.15.45:0/6963520 >> v1:172.21.15.103:6789/0 conn(0x7f17b40b2de0 legacy=0x7f17b406e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.689+0000 7f17b2a00640 1 -- 172.21.15.45:0/6963520 >> v1:172.21.15.45:6789/0 conn(0x7f17b406eee0 legacy=0x7f17b415cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.689+0000 7f17b2a00640 1 -- 172.21.15.45:0/6963520 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b415c3b0 con 0x7f17b4158a40 2024-04-16T16:17:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.689+0000 7f17b1600640 1 -- 172.21.15.45:0/6963520 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2964148194 0 0) 0x7f17b40cfff0 con 0x7f17b406eee0 2024-04-16T16:17:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.690+0000 7f17b2a00640 1 --2- 172.21.15.45:0/6963520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17b4158a40 0x7f17b415be70 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f17a40097a0 tx=0x7f17a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de0afdfecf2a5cf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.690+0000 7f17b1600640 1 -- 172.21.15.45:0/6963520 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a4035020 con 0x7f17b4158a40 2024-04-16T16:17:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.690+0000 7f17b1600640 1 -- 172.21.15.45:0/6963520 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17a4031830 con 0x7f17b4158a40 2024-04-16T16:17:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.690+0000 7f17b1600640 1 -- 172.21.15.45:0/6963520 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a4031c00 con 0x7f17b4158a40 2024-04-16T16:17:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.691+0000 7f17b3e00640 1 -- 172.21.15.45:0/6963520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17b4158a40 msgr2=0x7f17b415be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.691+0000 7f17b3e00640 1 --2- 172.21.15.45:0/6963520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17b4158a40 0x7f17b415be70 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f17a40097a0 tx=0x7f17a402b8d0 comp rx=0 tx=0).stop 2024-04-16T16:17:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.691+0000 7f17b3e00640 1 -- 172.21.15.45:0/6963520 shutdown_connections 2024-04-16T16:17:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.691+0000 7f17b3e00640 1 --2- 172.21.15.45:0/6963520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17b4158a40 0x7f17b415be70 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.691+0000 7f17b3e00640 1 -- 172.21.15.45:0/6963520 >> 172.21.15.45:0/6963520 conn(0x7f17b406b930 msgr2=0x7f17b406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.691+0000 7f17b3e00640 1 -- 172.21.15.45:0/6963520 shutdown_connections 2024-04-16T16:17:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.691+0000 7f17b3e00640 1 -- 172.21.15.45:0/6963520 wait complete. 2024-04-16T16:17:12.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.693+0000 7f17b3e00640 1 Processor -- start 2024-04-16T16:17:12.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.693+0000 7f17b3e00640 1 -- start start 2024-04-16T16:17:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.694+0000 7f17b3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17b406eee0 0x7f17b4151520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.694+0000 7f17b3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17b40b2de0 0x7f17b41e66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.694+0000 7f17b3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17b41e60b0 0x7f17b41f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.694+0000 7f17b3e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f17b406e1d0 con 0x7f17b41e60b0 2024-04-16T16:17:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.694+0000 7f17b3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f17b415c5e0 con 0x7f17b406eee0 2024-04-16T16:17:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.694+0000 7f17b3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f17b406e350 con 0x7f17b40b2de0 2024-04-16T16:17:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.694+0000 7f17b3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17b41e60b0 0x7f17b41f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.695+0000 7f17b3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17b41e60b0 0x7f17b41f3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44394/0 (socket says 172.21.15.45:44394) 2024-04-16T16:17:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.695+0000 7f17b3400640 1 -- 172.21.15.45:0/562642680 learned_addr learned my addr 172.21.15.45:0/562642680 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.695+0000 7f17b2a00640 1 --2- 172.21.15.45:0/562642680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17b406eee0 0x7f17b4151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.695+0000 7f17b2000640 1 --2- 172.21.15.45:0/562642680 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17b40b2de0 0x7f17b41e66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.695+0000 7f17b3400640 1 -- 172.21.15.45:0/562642680 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17b40b2de0 msgr2=0x7f17b41e66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.695+0000 7f17b3400640 1 --2- 172.21.15.45:0/562642680 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17b40b2de0 0x7f17b41e66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.695+0000 7f17b3400640 1 -- 172.21.15.45:0/562642680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17b406eee0 msgr2=0x7f17b4151520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.695+0000 7f17b3400640 1 --2- 172.21.15.45:0/562642680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17b406eee0 0x7f17b4151520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.695+0000 7f17b3400640 1 -- 172.21.15.45:0/562642680 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b41e3980 con 0x7f17b41e60b0 2024-04-16T16:17:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.695+0000 7f17b3400640 1 --2- 172.21.15.45:0/562642680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17b41e60b0 0x7f17b41f3840 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f17ac00c7a0 tx=0x7f17ac00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:12.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.696+0000 7f17aaa00640 1 -- 172.21.15.45:0/562642680 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ac018070 con 0x7f17b41e60b0 2024-04-16T16:17:12.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.696+0000 7f17aaa00640 1 -- 172.21.15.45:0/562642680 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17ac00a640 con 0x7f17b41e60b0 2024-04-16T16:17:12.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.696+0000 7f17aaa00640 1 -- 172.21.15.45:0/562642680 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ac00aab0 con 0x7f17b41e60b0 2024-04-16T16:17:12.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.696+0000 7f17b3e00640 1 -- 172.21.15.45:0/562642680 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17b41e3c60 con 0x7f17b41e60b0 2024-04-16T16:17:12.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.696+0000 7f17b3e00640 1 -- 172.21.15.45:0/562642680 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17b41e4220 con 0x7f17b41e60b0 2024-04-16T16:17:12.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.697+0000 7f17b3e00640 1 -- 172.21.15.45:0/562642680 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17b4075780 con 0x7f17b41e60b0 2024-04-16T16:17:12.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.698+0000 7f17aaa00640 1 -- 172.21.15.45:0/562642680 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f17ac01b070 con 0x7f17b41e60b0 2024-04-16T16:17:12.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.699+0000 7f17aaa00640 1 --2- 172.21.15.45:0/562642680 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f179406ca60 0x7f179406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:12.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.699+0000 7f17aaa00640 1 -- 172.21.15.45:0/562642680 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f17ac08b4b0 con 0x7f17b41e60b0 2024-04-16T16:17:12.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.700+0000 7f17b2a00640 1 --2- 172.21.15.45:0/562642680 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f179406ca60 0x7f179406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:12.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.701+0000 7f17b2a00640 1 --2- 172.21.15.45:0/562642680 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f179406ca60 0x7f179406ef20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f17a4036000 tx=0x7f17a403c040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:12.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.704+0000 7f17aaa00640 1 -- 172.21.15.45:0/562642680 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f17ac0596b0 con 0x7f17b41e60b0 2024-04-16T16:17:12.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.948+0000 7f17b3e00640 1 -- 172.21.15.45:0/562642680 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17b415d840 con 0x7f17b41e60b0 2024-04-16T16:17:12.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.949+0000 7f17aaa00640 1 -- 172.21.15.45:0/562642680 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5200 (secure 0 0 0) 0x7f17ac058fb0 con 0x7f17b41e60b0 2024-04-16T16:17:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:12.948 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":53,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:17:11.564712+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6402,"mds_1":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"c","rank":1,"incarnation":52,"state":"up:active","state_seq":11,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6399,6402]},"id":5}]} 2024-04-16T16:17:12.948 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 53 2024-04-16T16:17:12.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.951+0000 7f17b3e00640 1 -- 172.21.15.45:0/562642680 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f179406ca60 msgr2=0x7f179406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:12.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.951+0000 7f17b3e00640 1 --2- 172.21.15.45:0/562642680 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f179406ca60 0x7f179406ef20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f17a4036000 tx=0x7f17a403c040 comp rx=0 tx=0).stop 2024-04-16T16:17:12.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.952+0000 7f17b3e00640 1 -- 172.21.15.45:0/562642680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17b41e60b0 msgr2=0x7f17b41f3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:12.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.952+0000 7f17b3e00640 1 --2- 172.21.15.45:0/562642680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17b41e60b0 0x7f17b41f3840 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f17ac00c7a0 tx=0x7f17ac00cc60 comp rx=0 tx=0).stop 2024-04-16T16:17:12.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.952+0000 7f17b3e00640 1 -- 172.21.15.45:0/562642680 shutdown_connections 2024-04-16T16:17:12.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.952+0000 7f17b3e00640 1 --2- 172.21.15.45:0/562642680 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f179406ca60 0x7f179406ef20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:12.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.952+0000 7f17b3e00640 1 --2- 172.21.15.45:0/562642680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17b41e60b0 0x7f17b41f3840 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:12.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.952+0000 7f17b3e00640 1 --2- 172.21.15.45:0/562642680 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17b40b2de0 0x7f17b41e66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:12.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.952+0000 7f17b3e00640 1 --2- 172.21.15.45:0/562642680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17b406eee0 0x7f17b4151520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:12.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.952+0000 7f17b3e00640 1 -- 172.21.15.45:0/562642680 >> 172.21.15.45:0/562642680 conn(0x7f17b406b930 msgr2=0x7f17b415e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:12.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.952+0000 7f17b3e00640 1 -- 172.21.15.45:0/562642680 shutdown_connections 2024-04-16T16:17:12.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:12.952+0000 7f17b3e00640 1 -- 172.21.15.45:0/562642680 wait complete. 2024-04-16T16:17:13.374 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:16:39.513364+0000', 'modified': '2024-04-16T16:17:11.564712+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 6402, 'mds_1': 6399}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6399': {'gid': 6399, 'name': 'c', 'rank': 1, 'incarnation': 52, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.45:6834/883721507', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6832', 'nonce': 883721507}, {'type': 'v1', 'addr': '172.21.15.45:6834', 'nonce': 883721507}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_6402': {'gid': 6402, 'name': 'a', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6835/268877398', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6833', 'nonce': 268877398}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 268877398}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6402, 'qdb_cluster': [6399, 6402]} 2024-04-16T16:17:13.374 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-04-16T16:17:13.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-16T16:17:13.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.616+0000 7fd84f200640 1 Processor -- start 2024-04-16T16:17:13.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.616+0000 7fd84f200640 1 -- start start 2024-04-16T16:17:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.618+0000 7fd84f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd8500c6fc0 0x7fd85006f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.618+0000 7fd84f200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd85006df40 0x7fd85006e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.618+0000 7fd84f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd85015d600 con 0x7fd85015cdc0 2024-04-16T16:17:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.618+0000 7fd84f200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd8500c5670 con 0x7fd85006df40 2024-04-16T16:17:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.618+0000 7fd84f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd85015d8e0 con 0x7fd8500c6fc0 2024-04-16T16:17:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.618+0000 7fd84e800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd85006df40 0x7fd85006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.618+0000 7fd84e800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd85006df40 0x7fd85006e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47134/0 (socket says 172.21.15.45:47134) 2024-04-16T16:17:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.618+0000 7fd84e800640 1 -- 172.21.15.45:0/1627004664 learned_addr learned my addr 172.21.15.45:0/1627004664 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.619+0000 7fd84de00640 1 --2- 172.21.15.45:0/1627004664 >> v2:172.21.15.103:3300/0 conn(0x7fd8500c6fc0 0x7fd85006f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.619+0000 7fd84e800640 1 -- 172.21.15.45:0/1627004664 >> v2:172.21.15.103:3300/0 conn(0x7fd8500c6fc0 msgr2=0x7fd85006f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.619+0000 7fd84e800640 1 --2- 172.21.15.45:0/1627004664 >> v2:172.21.15.103:3300/0 conn(0x7fd8500c6fc0 0x7fd85006f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.619+0000 7fd84e800640 1 -- 172.21.15.45:0/1627004664 >> v1:172.21.15.103:6789/0 conn(0x7fd85015cdc0 legacy=0x7fd8500c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.619+0000 7fd84e800640 1 -- 172.21.15.45:0/1627004664 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd85015dbc0 con 0x7fd85006df40 2024-04-16T16:17:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.619+0000 7fd84de00640 1 --2- 172.21.15.45:0/1627004664 >> v2:172.21.15.103:3300/0 conn(0x7fd8500c6fc0 0x7fd85006f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:13.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.620+0000 7fd84e800640 1 --2- 172.21.15.45:0/1627004664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd85006df40 0x7fd85006e350 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fd8500700f0 tx=0x7fd84802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ddebb6b7c84326 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:13.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.620+0000 7fd84d400640 1 -- 172.21.15.45:0/1627004664 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84802ca80 con 0x7fd85006df40 2024-04-16T16:17:13.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.620+0000 7fd84d400640 1 -- 172.21.15.45:0/1627004664 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd84802cc20 con 0x7fd85006df40 2024-04-16T16:17:13.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.620+0000 7fd84d400640 1 -- 172.21.15.45:0/1627004664 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8480343f0 con 0x7fd85006df40 2024-04-16T16:17:13.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.621+0000 7fd84f200640 1 -- 172.21.15.45:0/1627004664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd85006df40 msgr2=0x7fd85006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.621+0000 7fd84f200640 1 --2- 172.21.15.45:0/1627004664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd85006df40 0x7fd85006e350 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fd8500700f0 tx=0x7fd84802bee0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.621+0000 7fd84f200640 1 -- 172.21.15.45:0/1627004664 shutdown_connections 2024-04-16T16:17:13.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.621+0000 7fd84f200640 1 --2- 172.21.15.45:0/1627004664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd85006df40 0x7fd85006e350 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.621+0000 7fd84f200640 1 --2- 172.21.15.45:0/1627004664 >> v2:172.21.15.103:3300/0 conn(0x7fd8500c6fc0 0x7fd85006f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.621+0000 7fd84f200640 1 -- 172.21.15.45:0/1627004664 >> 172.21.15.45:0/1627004664 conn(0x7fd85006b930 msgr2=0x7fd85006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:13.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.621+0000 7fd84f200640 1 -- 172.21.15.45:0/1627004664 shutdown_connections 2024-04-16T16:17:13.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.621+0000 7fd84f200640 1 -- 172.21.15.45:0/1627004664 wait complete. 2024-04-16T16:17:13.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.623+0000 7fd84f200640 1 Processor -- start 2024-04-16T16:17:13.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.623+0000 7fd84f200640 1 -- start start 2024-04-16T16:17:13.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.624+0000 7fd84f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd85006df40 0x7fd8501521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.624+0000 7fd84f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8500c6fc0 0x7fd850157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.624+0000 7fd84f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd85015cdc0 0x7fd8500a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.624+0000 7fd84f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd8500c64f0 con 0x7fd8500c6fc0 2024-04-16T16:17:13.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.624+0000 7fd84f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd85015d270 con 0x7fd85015cdc0 2024-04-16T16:17:13.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.624+0000 7fd84f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd8500c6670 con 0x7fd85006df40 2024-04-16T16:17:13.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.624+0000 7fd847e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8500c6fc0 0x7fd850157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.625+0000 7fd847e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8500c6fc0 0x7fd850157490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47140/0 (socket says 172.21.15.45:47140) 2024-04-16T16:17:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.625+0000 7fd847e00640 1 -- 172.21.15.45:0/3814075420 learned_addr learned my addr 172.21.15.45:0/3814075420 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.625+0000 7fd84e800640 1 --2- 172.21.15.45:0/3814075420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd85015cdc0 0x7fd8500a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.625+0000 7fd84de00640 1 --2- 172.21.15.45:0/3814075420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd85006df40 0x7fd8501521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.625+0000 7fd847e00640 1 -- 172.21.15.45:0/3814075420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd85006df40 msgr2=0x7fd8501521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.625+0000 7fd847e00640 1 --2- 172.21.15.45:0/3814075420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd85006df40 0x7fd8501521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.625+0000 7fd847e00640 1 -- 172.21.15.45:0/3814075420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd85015cdc0 msgr2=0x7fd8500a94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.625+0000 7fd847e00640 1 --2- 172.21.15.45:0/3814075420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd85015cdc0 0x7fd8500a94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.625+0000 7fd847e00640 1 -- 172.21.15.45:0/3814075420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd850157e00 con 0x7fd8500c6fc0 2024-04-16T16:17:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.625+0000 7fd84e800640 1 --2- 172.21.15.45:0/3814075420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd85015cdc0 0x7fd8500a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:13.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.625+0000 7fd847e00640 1 --2- 172.21.15.45:0/3814075420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8500c6fc0 0x7fd850157490 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fd83800b3f0 tx=0x7fd83800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:13.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.626+0000 7fd846000640 1 -- 172.21.15.45:0/3814075420 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd838015020 con 0x7fd8500c6fc0 2024-04-16T16:17:13.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.626+0000 7fd846000640 1 -- 172.21.15.45:0/3814075420 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd838011830 con 0x7fd8500c6fc0 2024-04-16T16:17:13.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.626+0000 7fd846000640 1 -- 172.21.15.45:0/3814075420 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd838011c20 con 0x7fd8500c6fc0 2024-04-16T16:17:13.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.626+0000 7fd84f200640 1 -- 172.21.15.45:0/3814075420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8501580e0 con 0x7fd8500c6fc0 2024-04-16T16:17:13.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.626+0000 7fd84f200640 1 -- 172.21.15.45:0/3814075420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8501539e0 con 0x7fd8500c6fc0 2024-04-16T16:17:13.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.628+0000 7fd846000640 1 -- 172.21.15.45:0/3814075420 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7fd838018020 con 0x7fd8500c6fc0 2024-04-16T16:17:13.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.630+0000 7fd846000640 1 --2- 172.21.15.45:0/3814075420 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd82c06ca90 0x7fd82c06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.630+0000 7fd846000640 1 -- 172.21.15.45:0/3814075420 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7fd83808bb30 con 0x7fd8500c6fc0 2024-04-16T16:17:13.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.630+0000 7fd857fba740 1 Processor -- start 2024-04-16T16:17:13.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.630+0000 7fd84de00640 1 --2- 172.21.15.45:0/3814075420 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd82c06ca90 0x7fd82c06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.631+0000 7fd857fba740 1 -- start start 2024-04-16T16:17:13.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.631+0000 7fd857fba740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369b4890 0x5587369d39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.631+0000 7fd857fba740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4c70 0x55873698d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.631+0000 7fd857fba740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55873698dab0 0x55873698b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.631+0000 7fd857fba740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5587368f29c0 con 0x55873698dab0 2024-04-16T16:17:13.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.631+0000 7fd857fba740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558736899560 con 0x5587369b4c70 2024-04-16T16:17:13.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.631+0000 7fd857fba740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5587368ee980 con 0x5587369b4890 2024-04-16T16:17:13.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.632+0000 7fd84de00640 1 --2- 172.21.15.45:0/3814075420 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd82c06ca90 0x7fd82c06ef50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd840008850 tx=0x7fd840008040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:13.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.632+0000 7fd84e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55873698dab0 0x55873698b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.632+0000 7fd84e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55873698dab0 0x55873698b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47142/0 (socket says 172.21.15.45:47142) 2024-04-16T16:17:13.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.632+0000 7fd84e800640 1 -- 172.21.15.45:0/1908057675 learned_addr learned my addr 172.21.15.45:0/1908057675 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:13.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.632+0000 7fd847e00640 1 --2- 172.21.15.45:0/1908057675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4c70 0x55873698d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.632+0000 7fd84de00640 1 --2- 172.21.15.45:0/1908057675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369b4890 0x5587369d39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.632+0000 7fd84e800640 1 -- 172.21.15.45:0/1908057675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369b4890 msgr2=0x5587369d39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.632+0000 7fd84e800640 1 --2- 172.21.15.45:0/1908057675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369b4890 0x5587369d39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.632+0000 7fd84e800640 1 -- 172.21.15.45:0/1908057675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4c70 msgr2=0x55873698d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.632+0000 7fd84e800640 1 --2- 172.21.15.45:0/1908057675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4c70 0x55873698d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.632+0000 7fd84e800640 1 -- 172.21.15.45:0/1908057675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587366052d0 con 0x55873698dab0 2024-04-16T16:17:13.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.632+0000 7fd847e00640 1 --2- 172.21.15.45:0/1908057675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4c70 0x55873698d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:17:13.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.633+0000 7fd84e800640 1 --2- 172.21.15.45:0/1908057675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55873698dab0 0x55873698b440 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fd84802bf20 tx=0x7fd84802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a310034056cb6b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:13.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.633+0000 7fd83ea00640 1 -- 172.21.15.45:0/1908057675 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84802ca80 con 0x55873698dab0 2024-04-16T16:17:13.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.633+0000 7fd83ea00640 1 -- 172.21.15.45:0/1908057675 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd84802cc20 con 0x55873698dab0 2024-04-16T16:17:13.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.633+0000 7fd83ea00640 1 -- 172.21.15.45:0/1908057675 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8480406b0 con 0x55873698dab0 2024-04-16T16:17:13.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.634+0000 7fd857fba740 1 -- 172.21.15.45:0/1908057675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55873698dab0 msgr2=0x55873698b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.634+0000 7fd857fba740 1 --2- 172.21.15.45:0/1908057675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55873698dab0 0x55873698b440 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fd84802bf20 tx=0x7fd84802e000 comp rx=0 tx=0).stop 2024-04-16T16:17:13.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.634+0000 7fd857fba740 1 -- 172.21.15.45:0/1908057675 shutdown_connections 2024-04-16T16:17:13.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.634+0000 7fd857fba740 1 --2- 172.21.15.45:0/1908057675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55873698dab0 0x55873698b440 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.634+0000 7fd857fba740 1 --2- 172.21.15.45:0/1908057675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4c70 0x55873698d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.634+0000 7fd857fba740 1 --2- 172.21.15.45:0/1908057675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369b4890 0x5587369d39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.634+0000 7fd857fba740 1 -- 172.21.15.45:0/1908057675 >> 172.21.15.45:0/1908057675 conn(0x558736989b50 msgr2=0x5587369d26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:13.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.634+0000 7fd857fba740 1 -- 172.21.15.45:0/1908057675 shutdown_connections 2024-04-16T16:17:13.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.634+0000 7fd857fba740 1 -- 172.21.15.45:0/1908057675 wait complete. 2024-04-16T16:17:13.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.635+0000 7fd857fba740 1 -- start start 2024-04-16T16:17:13.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.635+0000 7fd857fba740 1 Processor -- start 2024-04-16T16:17:13.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd857fba740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369b4890 0x5587369ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd857fba740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4c70 0x5587369e2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd857fba740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369e6b10 0x5587369e5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd857fba740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x558736987980 con 0x5587369e6b10 2024-04-16T16:17:13.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd857fba740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5587368ac920 con 0x5587369b4c70 2024-04-16T16:17:13.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd857fba740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55873698df30 con 0x5587369b4890 2024-04-16T16:17:13.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd857fba740 10 client.0 _ll_register_callbacks cb 0x5587368ac310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:17:13.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd84e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369e6b10 0x5587369e5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd84e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369e6b10 0x5587369e5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47146/0 (socket says 172.21.15.45:47146) 2024-04-16T16:17:13.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd84e800640 1 -- 172.21.15.45:0/39223862 learned_addr learned my addr 172.21.15.45:0/39223862 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:13.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd847e00640 1 --2- 172.21.15.45:0/39223862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4c70 0x5587369e2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd84de00640 1 --2- 172.21.15.45:0/39223862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369b4890 0x5587369ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd84e800640 1 -- 172.21.15.45:0/39223862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369b4890 msgr2=0x5587369ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd84e800640 1 --2- 172.21.15.45:0/39223862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369b4890 0x5587369ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd84e800640 1 -- 172.21.15.45:0/39223862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4c70 msgr2=0x5587369e2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd84e800640 1 --2- 172.21.15.45:0/39223862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4c70 0x5587369e2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd84e800640 1 -- 172.21.15.45:0/39223862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587366cf6b0 con 0x5587369e6b10 2024-04-16T16:17:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.636+0000 7fd847e00640 1 --2- 172.21.15.45:0/39223862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4c70 0x5587369e2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.637+0000 7fd84e800640 1 --2- 172.21.15.45:0/39223862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369e6b10 0x5587369e5f00 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fd848009f40 tx=0x7fd848004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a6a6cb618673a31 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.637+0000 7fd833400640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:17:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.637+0000 7fd833400640 1 -- 172.21.15.45:0/39223862 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84804e030 con 0x5587369e6b10 2024-04-16T16:17:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.637+0000 7fd833400640 1 -- 172.21.15.45:0/39223862 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7fd848040a50 con 0x5587369e6b10 2024-04-16T16:17:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.637+0000 7fd857fba740 1 -- 172.21.15.45:0/39223862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd84802cc20 con 0x5587369e6b10 2024-04-16T16:17:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.637+0000 7fd833400640 1 -- 172.21.15.45:0/39223862 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fd848040c50 con 0x5587369e6b10 2024-04-16T16:17:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.637+0000 7fd857fba740 10 client.6717 fetch_fsmap learned FSMap version 53 2024-04-16T16:17:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.638+0000 7fd857fba740 1 -- 172.21.15.45:0/39223862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) -- 0x5587369cf2e0 con 0x5587369e6b10 2024-04-16T16:17:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.638+0000 7fd833400640 1 -- 172.21.15.45:0/39223862 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd84802cc20 con 0x5587369e6b10 2024-04-16T16:17:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.638+0000 7fd833400640 1 -- 172.21.15.45:0/39223862 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 53) ==== 3514+0+0 (secure 0 0 0) 0x7fd84805a020 con 0x5587369e6b10 2024-04-16T16:17:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.638+0000 7fd857fba740 10 client.6717 fetch_fsmap finished waiting for FSMap version 53 2024-04-16T16:17:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.638+0000 7fd857fba740 10 client.6717 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-16T16:17:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.638+0000 7fd857fba740 20 client.6717 populate_metadata read hostname 'smithi045' 2024-04-16T16:17:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.638+0000 7fd857fba740 1 --2- 172.21.15.45:0/39223862 >> [v2:172.21.15.45:6832/883721507,v1:172.21.15.45:6834/883721507] conn(0x5587369b5fb0 0x5587369d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.638+0000 7fd857fba740 4 client.6717 mds_command: new command op to 6399 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-16T16:17:13.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.638+0000 7fd857fba740 1 -- 172.21.15.45:0/39223862 --> [v2:172.21.15.45:6832/883721507,v1:172.21.15.45:6834/883721507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558736898b70 con 0x5587369b5fb0 2024-04-16T16:17:13.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.638+0000 7fd84de00640 1 --2- 172.21.15.45:0/39223862 >> [v2:172.21.15.45:6832/883721507,v1:172.21.15.45:6834/883721507] conn(0x5587369b5fb0 0x5587369d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.639+0000 7fd84de00640 1 --2- 172.21.15.45:0/39223862 >> [v2:172.21.15.45:6832/883721507,v1:172.21.15.45:6834/883721507] conn(0x5587369b5fb0 0x5587369d0930 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bfac1ef9fb9db410 server_cookie=82bb0b1380be7fc7 in_seq=0 out_seq=0 2024-04-16T16:17:13.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.639+0000 7fd833400640 10 client.6717 ms_handle_connect on v2:172.21.15.45:6832/883721507 2024-04-16T16:17:13.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.640+0000 7fd833400640 1 -- 172.21.15.45:0/39223862 <== mds.1 v2:172.21.15.45:6832/883721507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558736898b70 con 0x5587369b5fb0 2024-04-16T16:17:13.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.640+0000 7fd833400640 10 client.6717 handle_command_reply: tid=0 2024-04-16T16:17:13.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.640+0000 7fd857fba740 1 client.6717 shutdown 2024-04-16T16:17:13.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.640+0000 7fd857fba740 1 -- 172.21.15.45:0/39223862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369e6b10 msgr2=0x5587369e5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.640+0000 7fd857fba740 1 --2- 172.21.15.45:0/39223862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369e6b10 0x5587369e5f00 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fd848009f40 tx=0x7fd848004ab0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.640+0000 7fd857fba740 1 -- 172.21.15.45:0/39223862 shutdown_connections 2024-04-16T16:17:13.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.640+0000 7fd857fba740 1 --2- 172.21.15.45:0/39223862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369e6b10 0x5587369e5f00 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.640+0000 7fd857fba740 1 --2- 172.21.15.45:0/39223862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4c70 0x5587369e2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.640+0000 7fd857fba740 1 --2- 172.21.15.45:0/39223862 >> [v2:172.21.15.45:6832/883721507,v1:172.21.15.45:6834/883721507] conn(0x5587369b5fb0 0x5587369d0930 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.640+0000 7fd857fba740 1 --2- 172.21.15.45:0/39223862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369b4890 0x5587369ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.640+0000 7fd857fba740 1 -- 172.21.15.45:0/39223862 >> 172.21.15.45:0/39223862 conn(0x55873698dab0 msgr2=0x5587369d2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:13.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.640+0000 7fd833400640 0 client.6717 ms_handle_reset on v2:172.21.15.45:6832/883721507 2024-04-16T16:17:13.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.640+0000 7fd857fba740 1 -- 172.21.15.45:0/39223862 shutdown_connections 2024-04-16T16:17:13.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.640+0000 7fd857fba740 1 -- 172.21.15.45:0/39223862 wait complete. 2024-04-16T16:17:13.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.641+0000 7fd857fba740 20 client.6717 trim_cache size 0 max 16384 2024-04-16T16:17:13.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.660+0000 7fd857fba740 1 Processor -- start 2024-04-16T16:17:13.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.661+0000 7fd857fba740 1 -- start start 2024-04-16T16:17:13.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.661+0000 7fd857fba740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369b4890 0x5587369b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.661+0000 7fd857fba740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369b4c70 0x5587369b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.661+0000 7fd857fba740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369e6b10 0x5587369b75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.661+0000 7fd857fba740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd8502010d0 con 0x5587369b4890 2024-04-16T16:17:13.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.661+0000 7fd857fba740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd850159ad0 con 0x5587369e6b10 2024-04-16T16:17:13.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.661+0000 7fd857fba740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd8501ffb20 con 0x5587369b4c70 2024-04-16T16:17:13.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.662+0000 7fd84de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369b4890 0x5587369b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.662+0000 7fd84de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369b4890 0x5587369b0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47148/0 (socket says 172.21.15.45:47148) 2024-04-16T16:17:13.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.662+0000 7fd84de00640 1 -- 172.21.15.45:0/2852266957 learned_addr learned my addr 172.21.15.45:0/2852266957 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:13.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.662+0000 7fd847e00640 1 --2- 172.21.15.45:0/2852266957 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369b4c70 0x5587369b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.662+0000 7fd84e800640 1 --2- 172.21.15.45:0/2852266957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369e6b10 0x5587369b75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.662+0000 7fd84de00640 1 -- 172.21.15.45:0/2852266957 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369b4c70 msgr2=0x5587369b70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.662+0000 7fd84de00640 1 --2- 172.21.15.45:0/2852266957 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369b4c70 0x5587369b70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.662+0000 7fd84de00640 1 -- 172.21.15.45:0/2852266957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369e6b10 msgr2=0x5587369b75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.662+0000 7fd84de00640 1 --2- 172.21.15.45:0/2852266957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369e6b10 0x5587369b75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.662+0000 7fd84de00640 1 -- 172.21.15.45:0/2852266957 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55873698ef20 con 0x5587369b4890 2024-04-16T16:17:13.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.662+0000 7fd84de00640 1 --2- 172.21.15.45:0/2852266957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369b4890 0x5587369b0e00 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x5587369cf120 tx=0x7fd840013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc86cde21a9293f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:13.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.663+0000 7fd83e000640 1 -- 172.21.15.45:0/2852266957 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84002c070 con 0x5587369b4890 2024-04-16T16:17:13.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.663+0000 7fd83e000640 1 -- 172.21.15.45:0/2852266957 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd84001a040 con 0x5587369b4890 2024-04-16T16:17:13.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.663+0000 7fd83e000640 1 -- 172.21.15.45:0/2852266957 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd840019070 con 0x5587369b4890 2024-04-16T16:17:13.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.663+0000 7fd857fba740 1 -- 172.21.15.45:0/2852266957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369b4890 msgr2=0x5587369b0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.663+0000 7fd857fba740 1 --2- 172.21.15.45:0/2852266957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369b4890 0x5587369b0e00 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x5587369cf120 tx=0x7fd840013000 comp rx=0 tx=0).stop 2024-04-16T16:17:13.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.664+0000 7fd857fba740 1 -- 172.21.15.45:0/2852266957 shutdown_connections 2024-04-16T16:17:13.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.664+0000 7fd857fba740 1 --2- 172.21.15.45:0/2852266957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369e6b10 0x5587369b75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.664+0000 7fd857fba740 1 --2- 172.21.15.45:0/2852266957 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369b4c70 0x5587369b70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.664+0000 7fd857fba740 1 --2- 172.21.15.45:0/2852266957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369b4890 0x5587369b0e00 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.664+0000 7fd857fba740 1 -- 172.21.15.45:0/2852266957 >> 172.21.15.45:0/2852266957 conn(0x55873698dab0 msgr2=0x5587369b1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:13.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.664+0000 7fd857fba740 1 -- 172.21.15.45:0/2852266957 shutdown_connections 2024-04-16T16:17:13.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.664+0000 7fd857fba740 1 -- 172.21.15.45:0/2852266957 wait complete. 2024-04-16T16:17:13.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.664+0000 7fd857fba740 1 -- start start 2024-04-16T16:17:13.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.664+0000 7fd857fba740 1 Processor -- start 2024-04-16T16:17:13.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.665+0000 7fd857fba740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4890 0x5587369d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.665+0000 7fd857fba740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369b4c70 0x55873698d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.665+0000 7fd857fba740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369e6b10 0x5587369d0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.665+0000 7fd857fba740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5587369d13d0 con 0x5587369b4c70 2024-04-16T16:17:13.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.665+0000 7fd857fba740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558736987980 con 0x5587369b4890 2024-04-16T16:17:13.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.665+0000 7fd857fba740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5587369b6400 con 0x5587369e6b10 2024-04-16T16:17:13.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.666+0000 7fd857fba740 10 client.0 _ll_register_callbacks cb 0x7fd848007080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:17:13.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.666+0000 7fd847e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369b4c70 0x55873698d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.666+0000 7fd847e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369b4c70 0x55873698d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47158/0 (socket says 172.21.15.45:47158) 2024-04-16T16:17:13.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.666+0000 7fd847e00640 1 -- 172.21.15.45:0/1543124717 learned_addr learned my addr 172.21.15.45:0/1543124717 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:13.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.666+0000 7fd84de00640 1 --2- 172.21.15.45:0/1543124717 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4890 0x5587369d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.666+0000 7fd84e800640 1 --2- 172.21.15.45:0/1543124717 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369e6b10 0x5587369d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.666+0000 7fd847e00640 1 -- 172.21.15.45:0/1543124717 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369e6b10 msgr2=0x5587369d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.666+0000 7fd847e00640 1 --2- 172.21.15.45:0/1543124717 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369e6b10 0x5587369d0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.666+0000 7fd847e00640 1 -- 172.21.15.45:0/1543124717 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4890 msgr2=0x5587369d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.666+0000 7fd847e00640 1 --2- 172.21.15.45:0/1543124717 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4890 0x5587369d2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.666+0000 7fd847e00640 1 -- 172.21.15.45:0/1543124717 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587369b7800 con 0x5587369b4c70 2024-04-16T16:17:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.667+0000 7fd847e00640 1 --2- 172.21.15.45:0/1543124717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369b4c70 0x55873698d0e0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd838059520 tx=0x7fd838059650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b78d17be45e025 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.667+0000 7fd833400640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:17:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.667+0000 7fd833400640 1 -- 172.21.15.45:0/1543124717 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd838089e00 con 0x5587369b4c70 2024-04-16T16:17:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.667+0000 7fd833400640 1 -- 172.21.15.45:0/1543124717 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7fd838055c00 con 0x5587369b4c70 2024-04-16T16:17:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.667+0000 7fd857fba740 1 -- 172.21.15.45:0/1543124717 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd84001a040 con 0x5587369b4c70 2024-04-16T16:17:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.667+0000 7fd833400640 1 -- 172.21.15.45:0/1543124717 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fd838078050 con 0x5587369b4c70 2024-04-16T16:17:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.667+0000 7fd857fba740 10 client.6723 fetch_fsmap learned FSMap version 53 2024-04-16T16:17:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.667+0000 7fd857fba740 1 -- 172.21.15.45:0/1543124717 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) -- 0x558736a4dda0 con 0x5587369b4c70 2024-04-16T16:17:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.667+0000 7fd833400640 1 -- 172.21.15.45:0/1543124717 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd84001a040 con 0x5587369b4c70 2024-04-16T16:17:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.667+0000 7fd833400640 1 -- 172.21.15.45:0/1543124717 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 53) ==== 3514+0+0 (secure 0 0 0) 0x7fd8380589e0 con 0x5587369b4c70 2024-04-16T16:17:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.667+0000 7fd857fba740 10 client.6723 fetch_fsmap finished waiting for FSMap version 53 2024-04-16T16:17:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.667+0000 7fd857fba740 10 client.6723 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-16T16:17:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.667+0000 7fd857fba740 20 client.6723 populate_metadata read hostname 'smithi045' 2024-04-16T16:17:13.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.667+0000 7fd857fba740 1 --2- 172.21.15.45:0/1543124717 >> [v2:172.21.15.45:6832/883721507,v1:172.21.15.45:6834/883721507] conn(0x558736a4d840 0x5587369e0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.668+0000 7fd857fba740 4 client.6723 mds_command: new command op to 6399 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-16T16:17:13.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.668+0000 7fd857fba740 1 -- 172.21.15.45:0/1543124717 --> [v2:172.21.15.45:6832/883721507,v1:172.21.15.45:6834/883721507] -- command(tid 0: {"prefix": "status"}) -- 0x5587369003e0 con 0x558736a4d840 2024-04-16T16:17:13.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.668+0000 7fd84de00640 1 --2- 172.21.15.45:0/1543124717 >> [v2:172.21.15.45:6832/883721507,v1:172.21.15.45:6834/883721507] conn(0x558736a4d840 0x5587369e0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.668+0000 7fd84de00640 1 --2- 172.21.15.45:0/1543124717 >> [v2:172.21.15.45:6832/883721507,v1:172.21.15.45:6834/883721507] conn(0x558736a4d840 0x5587369e0bb0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=50a8ad5da17a909 server_cookie=18f63798359f76d7 in_seq=0 out_seq=0 2024-04-16T16:17:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.668+0000 7fd833400640 10 client.6723 ms_handle_connect on v2:172.21.15.45:6832/883721507 2024-04-16T16:17:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.669+0000 7fd833400640 1 -- 172.21.15.45:0/1543124717 <== mds.1 v2:172.21.15.45:6832/883721507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5587369003e0 con 0x558736a4d840 2024-04-16T16:17:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.669+0000 7fd833400640 10 client.6723 handle_command_reply: tid=0 2024-04-16T16:17:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.669+0000 7fd857fba740 1 client.6723 shutdown 2024-04-16T16:17:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.669+0000 7fd857fba740 1 -- 172.21.15.45:0/1543124717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369b4c70 msgr2=0x55873698d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.669+0000 7fd857fba740 1 --2- 172.21.15.45:0/1543124717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369b4c70 0x55873698d0e0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd838059520 tx=0x7fd838059650 comp rx=0 tx=0).stop 2024-04-16T16:17:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.669+0000 7fd857fba740 1 -- 172.21.15.45:0/1543124717 shutdown_connections 2024-04-16T16:17:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.669+0000 7fd857fba740 1 --2- 172.21.15.45:0/1543124717 >> [v2:172.21.15.45:6832/883721507,v1:172.21.15.45:6834/883721507] conn(0x558736a4d840 0x5587369e0bb0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.669+0000 7fd857fba740 1 --2- 172.21.15.45:0/1543124717 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5587369e6b10 0x5587369d0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.669+0000 7fd857fba740 1 --2- 172.21.15.45:0/1543124717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5587369b4c70 0x55873698d0e0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.669+0000 7fd857fba740 1 --2- 172.21.15.45:0/1543124717 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587369b4890 0x5587369d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.669+0000 7fd857fba740 1 -- 172.21.15.45:0/1543124717 >> 172.21.15.45:0/1543124717 conn(0x55873698dab0 msgr2=0x5587369e5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.669+0000 7fd833400640 0 client.6723 ms_handle_reset on v2:172.21.15.45:6832/883721507 2024-04-16T16:17:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.669+0000 7fd857fba740 1 -- 172.21.15.45:0/1543124717 shutdown_connections 2024-04-16T16:17:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.669+0000 7fd857fba740 1 -- 172.21.15.45:0/1543124717 wait complete. 2024-04-16T16:17:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.669+0000 7fd857fba740 20 client.6723 trim_cache size 0 max 16384 2024-04-16T16:17:13.671 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:17:13.672 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "748c4b85-0190-4c59-80ad-098a6a5007c0", 2024-04-16T16:17:13.672 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 1, 2024-04-16T16:17:13.672 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 6399, 2024-04-16T16:17:13.672 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-04-16T16:17:13.672 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-04-16T16:17:13.672 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-04-16T16:17:13.672 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 3.1035940179999999, 2024-04-16T16:17:13.672 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 53, 2024-04-16T16:17:13.672 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 73, 2024-04-16T16:17:13.672 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 73, 2024-04-16T16:17:13.672 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 39.986883484000003 2024-04-16T16:17:13.672 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:17:13.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.670+0000 7fd84f200640 1 -- 172.21.15.45:0/3814075420 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd82c06ca90 msgr2=0x7fd82c06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.670+0000 7fd84f200640 1 --2- 172.21.15.45:0/3814075420 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd82c06ca90 0x7fd82c06ef50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd840008850 tx=0x7fd840008040 comp rx=0 tx=0).stop 2024-04-16T16:17:13.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.670+0000 7fd84f200640 1 -- 172.21.15.45:0/3814075420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8500c6fc0 msgr2=0x7fd850157490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.670+0000 7fd84f200640 1 --2- 172.21.15.45:0/3814075420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8500c6fc0 0x7fd850157490 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fd83800b3f0 tx=0x7fd83800b8b0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.670+0000 7fd84f200640 1 -- 172.21.15.45:0/3814075420 shutdown_connections 2024-04-16T16:17:13.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.670+0000 7fd84f200640 1 --2- 172.21.15.45:0/3814075420 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd82c06ca90 0x7fd82c06ef50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.670+0000 7fd84f200640 1 --2- 172.21.15.45:0/3814075420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd85015cdc0 0x7fd8500a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.670+0000 7fd84f200640 1 --2- 172.21.15.45:0/3814075420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8500c6fc0 0x7fd850157490 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.670+0000 7fd84f200640 1 --2- 172.21.15.45:0/3814075420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd85006df40 0x7fd8501521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.670+0000 7fd84f200640 1 -- 172.21.15.45:0/3814075420 >> 172.21.15.45:0/3814075420 conn(0x7fd85006b930 msgr2=0x7fd8500cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.670+0000 7fd84f200640 1 -- 172.21.15.45:0/3814075420 shutdown_connections 2024-04-16T16:17:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.670+0000 7fd84f200640 1 -- 172.21.15.45:0/3814075420 wait complete. 2024-04-16T16:17:13.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-16T16:17:13.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.897+0000 7f513c800640 1 Processor -- start 2024-04-16T16:17:13.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.898+0000 7f513c800640 1 -- start start 2024-04-16T16:17:13.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.900+0000 7f513c800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5134147550 0x7f5134072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.900+0000 7f513c800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51340726c0 0x7f51340b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.900+0000 7f513c800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51340b5d00 con 0x7f51340b58e0 2024-04-16T16:17:13.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.900+0000 7f513c800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f513406fc50 con 0x7f5134147550 2024-04-16T16:17:13.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.900+0000 7f513c800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5134072b20 con 0x7f51340726c0 2024-04-16T16:17:13.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.900+0000 7f5139e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5134147550 0x7f5134072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.900+0000 7f5139e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5134147550 0x7f5134072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47174/0 (socket says 172.21.15.45:47174) 2024-04-16T16:17:13.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.900+0000 7f5139e00640 1 -- 172.21.15.45:0/3275142119 learned_addr learned my addr 172.21.15.45:0/3275142119 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:13.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.900+0000 7f5139400640 1 --2- 172.21.15.45:0/3275142119 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51340726c0 0x7f51340b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.901+0000 7f5139e00640 1 -- 172.21.15.45:0/3275142119 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51340726c0 msgr2=0x7f51340b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.901+0000 7f5139e00640 1 --2- 172.21.15.45:0/3275142119 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51340726c0 0x7f51340b53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.901+0000 7f5139e00640 1 -- 172.21.15.45:0/3275142119 >> v1:172.21.15.103:6789/0 conn(0x7f51340b58e0 legacy=0x7f5134162670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.901+0000 7f5139e00640 1 -- 172.21.15.45:0/3275142119 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513406df00 con 0x7f5134147550 2024-04-16T16:17:13.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.902+0000 7f5139e00640 1 --2- 172.21.15.45:0/3275142119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5134147550 0x7f5134072180 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f5128009870 tx=0x7f512802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaaa6f210c24beca server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:13.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.902+0000 7f5138a00640 1 -- 172.21.15.45:0/3275142119 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5128035020 con 0x7f5134147550 2024-04-16T16:17:13.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.902+0000 7f5138a00640 1 -- 172.21.15.45:0/3275142119 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5128031640 con 0x7f5134147550 2024-04-16T16:17:13.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.902+0000 7f5138a00640 1 -- 172.21.15.45:0/3275142119 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512802d040 con 0x7f5134147550 2024-04-16T16:17:13.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.903+0000 7f513c800640 1 -- 172.21.15.45:0/3275142119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5134147550 msgr2=0x7f5134072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.903+0000 7f513c800640 1 --2- 172.21.15.45:0/3275142119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5134147550 0x7f5134072180 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f5128009870 tx=0x7f512802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.903+0000 7f513c800640 1 -- 172.21.15.45:0/3275142119 shutdown_connections 2024-04-16T16:17:13.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.903+0000 7f513c800640 1 --2- 172.21.15.45:0/3275142119 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51340726c0 0x7f51340b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.903+0000 7f513c800640 1 --2- 172.21.15.45:0/3275142119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5134147550 0x7f5134072180 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.903+0000 7f513c800640 1 -- 172.21.15.45:0/3275142119 >> 172.21.15.45:0/3275142119 conn(0x7f513406b930 msgr2=0x7f513406bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:13.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.903+0000 7f513c800640 1 -- 172.21.15.45:0/3275142119 shutdown_connections 2024-04-16T16:17:13.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.903+0000 7f513c800640 1 -- 172.21.15.45:0/3275142119 wait complete. 2024-04-16T16:17:13.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.905+0000 7f513c800640 1 Processor -- start 2024-04-16T16:17:13.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.905+0000 7f513c800640 1 -- start start 2024-04-16T16:17:13.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.906+0000 7f513c800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51340726c0 0x7f51341f4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.906+0000 7f513c800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51340b58e0 0x7f51341f5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.906+0000 7f513c800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5134147550 0x7f51341f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.906+0000 7f513c800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f513406e7a0 con 0x7f51340726c0 2024-04-16T16:17:13.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.906+0000 7f513c800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f513406e4c0 con 0x7f51340b58e0 2024-04-16T16:17:13.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.906+0000 7f513c800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f513406ea80 con 0x7f5134147550 2024-04-16T16:17:13.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.906+0000 7f5139e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51340726c0 0x7f51341f4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.906+0000 7f5139e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51340726c0 0x7f51341f4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47186/0 (socket says 172.21.15.45:47186) 2024-04-16T16:17:13.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.906+0000 7f5139e00640 1 -- 172.21.15.45:0/2086996988 learned_addr learned my addr 172.21.15.45:0/2086996988 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.907+0000 7f5139e00640 1 -- 172.21.15.45:0/2086996988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5134147550 msgr2=0x7f51341f58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.907+0000 7f5139e00640 1 --2- 172.21.15.45:0/2086996988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5134147550 0x7f51341f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.907+0000 7f5139e00640 1 -- 172.21.15.45:0/2086996988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51340b58e0 msgr2=0x7f51341f5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.907+0000 7f5139e00640 1 --2- 172.21.15.45:0/2086996988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51340b58e0 0x7f51341f5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.907+0000 7f5139e00640 1 -- 172.21.15.45:0/2086996988 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51340cde00 con 0x7f51340726c0 2024-04-16T16:17:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.907+0000 7f5139e00640 1 --2- 172.21.15.45:0/2086996988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51340726c0 0x7f51341f4e20 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f512802b930 tx=0x7f51280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.907+0000 7f512e000640 1 -- 172.21.15.45:0/2086996988 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5128040070 con 0x7f51340726c0 2024-04-16T16:17:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.907+0000 7f512e000640 1 -- 172.21.15.45:0/2086996988 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51280399f0 con 0x7f51340726c0 2024-04-16T16:17:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.907+0000 7f512e000640 1 -- 172.21.15.45:0/2086996988 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5128035040 con 0x7f51340726c0 2024-04-16T16:17:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.907+0000 7f513c800640 1 -- 172.21.15.45:0/2086996988 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51340ce0e0 con 0x7f51340726c0 2024-04-16T16:17:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.908+0000 7f513c800640 1 -- 172.21.15.45:0/2086996988 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51340ce6a0 con 0x7f51340726c0 2024-04-16T16:17:13.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.910+0000 7f512e000640 1 -- 172.21.15.45:0/2086996988 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f512804b060 con 0x7f51340726c0 2024-04-16T16:17:13.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.911+0000 7f512e000640 1 --2- 172.21.15.45:0/2086996988 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f511806ca60 0x7f511806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.911+0000 7f512e000640 1 -- 172.21.15.45:0/2086996988 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f51280ad8f0 con 0x7f51340726c0 2024-04-16T16:17:13.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.911+0000 7f513f567740 1 Processor -- start 2024-04-16T16:17:13.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.912+0000 7f513f567740 1 -- start start 2024-04-16T16:17:13.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.912+0000 7f5139400640 1 --2- 172.21.15.45:0/2086996988 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f511806ca60 0x7f511806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.912+0000 7f513f567740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a60650 0x55d1f4a7f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.912+0000 7f513f567740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60a30 0x55d1f4a39330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.912+0000 7f513f567740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a39870 0x55d1f4a37200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.912+0000 7f513f567740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d1f499e9c0 con 0x55d1f4a60a30 2024-04-16T16:17:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.912+0000 7f513f567740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d1f4945560 con 0x55d1f4a39870 2024-04-16T16:17:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.912+0000 7f513f567740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d1f499a980 con 0x55d1f4a60650 2024-04-16T16:17:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.913+0000 7f5139400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60a30 0x55d1f4a39330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.913+0000 7f5139400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60a30 0x55d1f4a39330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47190/0 (socket says 172.21.15.45:47190) 2024-04-16T16:17:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.913+0000 7f5139400640 1 -- 172.21.15.45:0/3275836873 learned_addr learned my addr 172.21.15.45:0/3275836873 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.913+0000 7f5139400640 1 --2- 172.21.15.45:0/2086996988 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f511806ca60 0x7f511806ef20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f513415d940 tx=0x7f5124008040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.913+0000 7f513a800640 1 --2- 172.21.15.45:0/3275836873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a39870 0x55d1f4a37200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.913+0000 7f5139e00640 1 --2- 172.21.15.45:0/3275836873 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a60650 0x55d1f4a7f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.913+0000 7f5139400640 1 -- 172.21.15.45:0/3275836873 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a60650 msgr2=0x55d1f4a7f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.913+0000 7f5139400640 1 --2- 172.21.15.45:0/3275836873 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a60650 0x55d1f4a7f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.913+0000 7f5139400640 1 -- 172.21.15.45:0/3275836873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a39870 msgr2=0x55d1f4a37200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.913+0000 7f5139400640 1 --2- 172.21.15.45:0/3275836873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a39870 0x55d1f4a37200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.913+0000 7f5139400640 1 -- 172.21.15.45:0/3275836873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1f46b12d0 con 0x55d1f4a60a30 2024-04-16T16:17:13.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.913+0000 7f513a800640 1 --2- 172.21.15.45:0/3275836873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a39870 0x55d1f4a37200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:13.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.913+0000 7f5139400640 1 --2- 172.21.15.45:0/3275836873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60a30 0x55d1f4a39330 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f51340d0270 tx=0x7f5124010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96ce8d974a5fa79 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:13.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.914+0000 7f5122600640 1 -- 172.21.15.45:0/3275836873 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5124024070 con 0x55d1f4a60a30 2024-04-16T16:17:13.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.914+0000 7f5122600640 1 -- 172.21.15.45:0/3275836873 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51240149b0 con 0x55d1f4a60a30 2024-04-16T16:17:13.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.914+0000 7f5122600640 1 -- 172.21.15.45:0/3275836873 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5124014da0 con 0x55d1f4a60a30 2024-04-16T16:17:13.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.914+0000 7f513f567740 1 -- 172.21.15.45:0/3275836873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60a30 msgr2=0x55d1f4a39330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.914+0000 7f513f567740 1 --2- 172.21.15.45:0/3275836873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60a30 0x55d1f4a39330 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f51340d0270 tx=0x7f5124010000 comp rx=0 tx=0).stop 2024-04-16T16:17:13.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.914+0000 7f513f567740 1 -- 172.21.15.45:0/3275836873 shutdown_connections 2024-04-16T16:17:13.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.914+0000 7f513f567740 1 --2- 172.21.15.45:0/3275836873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a39870 0x55d1f4a37200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.914+0000 7f513f567740 1 --2- 172.21.15.45:0/3275836873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60a30 0x55d1f4a39330 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.914+0000 7f513f567740 1 --2- 172.21.15.45:0/3275836873 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a60650 0x55d1f4a7f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.915+0000 7f513f567740 1 -- 172.21.15.45:0/3275836873 >> 172.21.15.45:0/3275836873 conn(0x55d1f4a35910 msgr2=0x55d1f4a7e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:13.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.915+0000 7f513f567740 1 -- 172.21.15.45:0/3275836873 shutdown_connections 2024-04-16T16:17:13.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.915+0000 7f513f567740 1 -- 172.21.15.45:0/3275836873 wait complete. 2024-04-16T16:17:13.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.915+0000 7f513f567740 1 -- start start 2024-04-16T16:17:13.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.915+0000 7f513f567740 1 Processor -- start 2024-04-16T16:17:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.916+0000 7f513f567740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a60650 0x55d1f4a5b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.916+0000 7f513f567740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60a30 0x55d1f4a61470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.916+0000 7f513f567740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a619b0 0x55d1f4a7cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.916+0000 7f513f567740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d1f4a33740 con 0x55d1f4a60a30 2024-04-16T16:17:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.917+0000 7f513f567740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d1f4958920 con 0x55d1f4a619b0 2024-04-16T16:17:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.917+0000 7f513f567740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d1f4a39cf0 con 0x55d1f4a60650 2024-04-16T16:17:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.917+0000 7f513f567740 10 client.0 _ll_register_callbacks cb 0x55d1f4958310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:17:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.917+0000 7f5139400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60a30 0x55d1f4a61470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.917+0000 7f5139400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60a30 0x55d1f4a61470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47202/0 (socket says 172.21.15.45:47202) 2024-04-16T16:17:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.917+0000 7f5139400640 1 -- 172.21.15.45:0/3538885954 learned_addr learned my addr 172.21.15.45:0/3538885954 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.917+0000 7f513a800640 1 --2- 172.21.15.45:0/3538885954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a619b0 0x55d1f4a7cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.917+0000 7f5139e00640 1 --2- 172.21.15.45:0/3538885954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a60650 0x55d1f4a5b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.917+0000 7f5139400640 1 -- 172.21.15.45:0/3538885954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a60650 msgr2=0x55d1f4a5b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.917+0000 7f5139400640 1 --2- 172.21.15.45:0/3538885954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a60650 0x55d1f4a5b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.917+0000 7f5139400640 1 -- 172.21.15.45:0/3538885954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a619b0 msgr2=0x55d1f4a7cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.917+0000 7f5139400640 1 --2- 172.21.15.45:0/3538885954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a619b0 0x55d1f4a7cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.917+0000 7f5139400640 1 -- 172.21.15.45:0/3538885954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1f477b6b0 con 0x55d1f4a60a30 2024-04-16T16:17:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.918+0000 7f5139400640 1 --2- 172.21.15.45:0/3538885954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60a30 0x55d1f4a61470 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f5124022040 tx=0x7f5124013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c504f844fe7e6d87 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.918+0000 7f5116a00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.918+0000 7f5116a00640 1 -- 172.21.15.45:0/3538885954 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5124030060 con 0x55d1f4a60a30 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.918+0000 7f5116a00640 1 -- 172.21.15.45:0/3538885954 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f512401f950 con 0x55d1f4a60a30 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.918+0000 7f513f567740 1 -- 172.21.15.45:0/3538885954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51240149b0 con 0x55d1f4a60a30 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.918+0000 7f5116a00640 1 -- 172.21.15.45:0/3538885954 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f5124036050 con 0x55d1f4a60a30 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.918+0000 7f513f567740 10 client.6735 fetch_fsmap learned FSMap version 53 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.918+0000 7f513f567740 1 -- 172.21.15.45:0/3538885954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) -- 0x55d1f4a5baf0 con 0x55d1f4a60a30 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.918+0000 7f5116a00640 1 -- 172.21.15.45:0/3538885954 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51240149b0 con 0x55d1f4a60a30 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.919+0000 7f5116a00640 1 -- 172.21.15.45:0/3538885954 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 53) ==== 3514+0+0 (secure 0 0 0) 0x7f5124039020 con 0x55d1f4a60a30 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.919+0000 7f513f567740 10 client.6735 fetch_fsmap finished waiting for FSMap version 53 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.919+0000 7f513f567740 10 client.6735 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.919+0000 7f513f567740 20 client.6735 populate_metadata read hostname 'smithi045' 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.919+0000 7f513f567740 1 --2- 172.21.15.45:0/3538885954 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x55d1f4a9b000 0x55d1f4a9d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.919+0000 7f513f567740 4 client.6735 mds_command: new command op to 6402 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.919+0000 7f513f567740 1 -- 172.21.15.45:0/3538885954 --> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1f4944b70 con 0x55d1f4a9b000 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.919+0000 7f5139e00640 1 --2- 172.21.15.45:0/3538885954 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x55d1f4a9b000 0x55d1f4a9d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.920+0000 7f5139e00640 1 --2- 172.21.15.45:0/3538885954 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x55d1f4a9b000 0x55d1f4a9d3f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0f7789c2a436365 server_cookie=ed5b639860659930 in_seq=0 out_seq=0 2024-04-16T16:17:13.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.920+0000 7f5116a00640 10 client.6735 ms_handle_connect on v2:172.21.15.45:6833/268877398 2024-04-16T16:17:13.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.920+0000 7f5116a00640 1 -- 172.21.15.45:0/3538885954 <== mds.0 v2:172.21.15.45:6833/268877398 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d1f4944b70 con 0x55d1f4a9b000 2024-04-16T16:17:13.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.920+0000 7f5116a00640 10 client.6735 handle_command_reply: tid=0 2024-04-16T16:17:13.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.920+0000 7f513f567740 1 client.6735 shutdown 2024-04-16T16:17:13.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.921+0000 7f513f567740 1 -- 172.21.15.45:0/3538885954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60a30 msgr2=0x55d1f4a61470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.921+0000 7f513f567740 1 --2- 172.21.15.45:0/3538885954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60a30 0x55d1f4a61470 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f5124022040 tx=0x7f5124013000 comp rx=0 tx=0).stop 2024-04-16T16:17:13.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.921+0000 7f513f567740 1 -- 172.21.15.45:0/3538885954 shutdown_connections 2024-04-16T16:17:13.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.921+0000 7f513f567740 1 --2- 172.21.15.45:0/3538885954 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x55d1f4a9b000 0x55d1f4a9d3f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.921+0000 7f513f567740 1 --2- 172.21.15.45:0/3538885954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a619b0 0x55d1f4a7cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.921+0000 7f513f567740 1 --2- 172.21.15.45:0/3538885954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60a30 0x55d1f4a61470 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.921+0000 7f513f567740 1 --2- 172.21.15.45:0/3538885954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a60650 0x55d1f4a5b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.921+0000 7f5116a00640 0 client.6735 ms_handle_reset on v2:172.21.15.45:6833/268877398 2024-04-16T16:17:13.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.921+0000 7f513f567740 1 -- 172.21.15.45:0/3538885954 >> 172.21.15.45:0/3538885954 conn(0x55d1f4a39870 msgr2=0x55d1f4a32c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:13.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.921+0000 7f513f567740 1 -- 172.21.15.45:0/3538885954 shutdown_connections 2024-04-16T16:17:13.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.921+0000 7f513f567740 1 -- 172.21.15.45:0/3538885954 wait complete. 2024-04-16T16:17:13.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.921+0000 7f513f567740 20 client.6735 trim_cache size 0 max 16384 2024-04-16T16:17:13.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.940+0000 7f513f567740 1 Processor -- start 2024-04-16T16:17:13.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.940+0000 7f513f567740 1 -- start start 2024-04-16T16:17:13.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.941+0000 7f513f567740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a60650 0x55d1f4a37110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.941+0000 7f513f567740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a60a30 0x55d1f4a37650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.941+0000 7f513f567740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a619b0 0x55d1f4a32550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.941+0000 7f513f567740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5134200f40 con 0x55d1f4a619b0 2024-04-16T16:17:13.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.941+0000 7f513f567740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5134201610 con 0x55d1f4a60650 2024-04-16T16:17:13.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.941+0000 7f513f567740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5134140680 con 0x55d1f4a60a30 2024-04-16T16:17:13.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.942+0000 7f513a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a619b0 0x55d1f4a32550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.942+0000 7f513a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a619b0 0x55d1f4a32550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47204/0 (socket says 172.21.15.45:47204) 2024-04-16T16:17:13.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.942+0000 7f513a800640 1 -- 172.21.15.45:0/321194210 learned_addr learned my addr 172.21.15.45:0/321194210 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:13.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.942+0000 7f5139e00640 1 --2- 172.21.15.45:0/321194210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a60650 0x55d1f4a37110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.942+0000 7f5139400640 1 --2- 172.21.15.45:0/321194210 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a60a30 0x55d1f4a37650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.942+0000 7f513a800640 1 -- 172.21.15.45:0/321194210 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a60a30 msgr2=0x55d1f4a37650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.942+0000 7f513a800640 1 --2- 172.21.15.45:0/321194210 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a60a30 0x55d1f4a37650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.942+0000 7f513a800640 1 -- 172.21.15.45:0/321194210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a60650 msgr2=0x55d1f4a37110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.942+0000 7f513a800640 1 --2- 172.21.15.45:0/321194210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a60650 0x55d1f4a37110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.942+0000 7f513a800640 1 -- 172.21.15.45:0/321194210 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1f4a37fe0 con 0x55d1f4a619b0 2024-04-16T16:17:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.943+0000 7f513a800640 1 --2- 172.21.15.45:0/321194210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a619b0 0x55d1f4a32550 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f51300053f0 tx=0x7f51300058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60274b5fe6ef52fa server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.943+0000 7f5121c00640 1 -- 172.21.15.45:0/321194210 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5130017020 con 0x55d1f4a619b0 2024-04-16T16:17:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.943+0000 7f5121c00640 1 -- 172.21.15.45:0/321194210 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f513000fb40 con 0x55d1f4a619b0 2024-04-16T16:17:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.943+0000 7f5121c00640 1 -- 172.21.15.45:0/321194210 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51300133f0 con 0x55d1f4a619b0 2024-04-16T16:17:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.943+0000 7f513f567740 1 -- 172.21.15.45:0/321194210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a619b0 msgr2=0x55d1f4a32550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.943+0000 7f513f567740 1 --2- 172.21.15.45:0/321194210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a619b0 0x55d1f4a32550 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f51300053f0 tx=0x7f51300058b0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.943+0000 7f513f567740 1 -- 172.21.15.45:0/321194210 shutdown_connections 2024-04-16T16:17:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.943+0000 7f513f567740 1 --2- 172.21.15.45:0/321194210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a619b0 0x55d1f4a32550 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.943+0000 7f513f567740 1 --2- 172.21.15.45:0/321194210 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a60a30 0x55d1f4a37650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.943+0000 7f513f567740 1 --2- 172.21.15.45:0/321194210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a60650 0x55d1f4a37110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.943+0000 7f513f567740 1 -- 172.21.15.45:0/321194210 >> 172.21.15.45:0/321194210 conn(0x55d1f4a39870 msgr2=0x55d1f4a67440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:13.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.944+0000 7f513f567740 1 -- 172.21.15.45:0/321194210 shutdown_connections 2024-04-16T16:17:13.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.944+0000 7f513f567740 1 -- 172.21.15.45:0/321194210 wait complete. 2024-04-16T16:17:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.944+0000 7f513f567740 1 -- start start 2024-04-16T16:17:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.944+0000 7f513f567740 1 Processor -- start 2024-04-16T16:17:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.945+0000 7f513f567740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60650 0x55d1f4af8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.945+0000 7f513f567740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a60a30 0x55d1f4a33030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.945+0000 7f513f567740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a619b0 0x55d1f4a33570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.945+0000 7f513f567740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d1f4a7c540 con 0x55d1f4a60650 2024-04-16T16:17:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.946+0000 7f513f567740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5134200020 con 0x55d1f4a60a30 2024-04-16T16:17:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.946+0000 7f513f567740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d1f4a79f70 con 0x55d1f4a619b0 2024-04-16T16:17:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.946+0000 7f513f567740 10 client.0 _ll_register_callbacks cb 0x7f5124022070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:17:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.946+0000 7f5139e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60650 0x55d1f4af8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.946+0000 7f5139e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60650 0x55d1f4af8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47212/0 (socket says 172.21.15.45:47212) 2024-04-16T16:17:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.946+0000 7f5139e00640 1 -- 172.21.15.45:0/1527633927 learned_addr learned my addr 172.21.15.45:0/1527633927 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.946+0000 7f5139e00640 1 -- 172.21.15.45:0/1527633927 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a619b0 msgr2=0x55d1f4a33570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.946+0000 7f5139e00640 1 --2- 172.21.15.45:0/1527633927 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a619b0 0x55d1f4a33570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.946+0000 7f5139e00640 1 -- 172.21.15.45:0/1527633927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a60a30 msgr2=0x55d1f4a33030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.946+0000 7f5139e00640 1 --2- 172.21.15.45:0/1527633927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a60a30 0x55d1f4a33030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.946+0000 7f5139e00640 1 -- 172.21.15.45:0/1527633927 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1f4a611e0 con 0x55d1f4a60650 2024-04-16T16:17:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.946+0000 7f5139e00640 1 --2- 172.21.15.45:0/1527633927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60650 0x55d1f4af8d60 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f5128077720 tx=0x7f512803c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=750d3e1a9c9d4378 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.946+0000 7f5116a00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:17:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.947+0000 7f5116a00640 1 -- 172.21.15.45:0/1527633927 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512804bd80 con 0x55d1f4a60650 2024-04-16T16:17:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.947+0000 7f5116a00640 1 -- 172.21.15.45:0/1527633927 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f5128077930 con 0x55d1f4a60650 2024-04-16T16:17:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.947+0000 7f513f567740 1 -- 172.21.15.45:0/1527633927 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f513000fb40 con 0x55d1f4a60650 2024-04-16T16:17:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.947+0000 7f5116a00640 1 -- 172.21.15.45:0/1527633927 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f51280abbc0 con 0x55d1f4a60650 2024-04-16T16:17:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.947+0000 7f513f567740 10 client.6741 fetch_fsmap learned FSMap version 53 2024-04-16T16:17:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.947+0000 7f513f567740 1 -- 172.21.15.45:0/1527633927 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) -- 0x55d1f4a7b100 con 0x55d1f4a60650 2024-04-16T16:17:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.947+0000 7f5116a00640 1 -- 172.21.15.45:0/1527633927 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f513000fb40 con 0x55d1f4a60650 2024-04-16T16:17:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.947+0000 7f5116a00640 1 -- 172.21.15.45:0/1527633927 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 53) ==== 3514+0+0 (secure 0 0 0) 0x7f5128083a10 con 0x55d1f4a60650 2024-04-16T16:17:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.947+0000 7f513f567740 10 client.6741 fetch_fsmap finished waiting for FSMap version 53 2024-04-16T16:17:13.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.947+0000 7f513f567740 10 client.6741 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-16T16:17:13.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.947+0000 7f513f567740 20 client.6741 populate_metadata read hostname 'smithi045' 2024-04-16T16:17:13.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.947+0000 7f513f567740 1 --2- 172.21.15.45:0/1527633927 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x55d1f4a5bf40 0x55d1f4a5c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:13.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.947+0000 7f513f567740 4 client.6741 mds_command: new command op to 6402 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-16T16:17:13.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.947+0000 7f513f567740 1 -- 172.21.15.45:0/1527633927 --> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] -- command(tid 0: {"prefix": "status"}) -- 0x55d1f49ac3e0 con 0x55d1f4a5bf40 2024-04-16T16:17:13.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.947+0000 7f5139400640 1 --2- 172.21.15.45:0/1527633927 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x55d1f4a5bf40 0x55d1f4a5c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:13.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.948+0000 7f5139400640 1 --2- 172.21.15.45:0/1527633927 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x55d1f4a5bf40 0x55d1f4a5c320 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9cca0154b13c06e server_cookie=c2b0bccd37b6d273 in_seq=0 out_seq=0 2024-04-16T16:17:13.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.948+0000 7f5116a00640 10 client.6741 ms_handle_connect on v2:172.21.15.45:6833/268877398 2024-04-16T16:17:13.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.948+0000 7f5116a00640 1 -- 172.21.15.45:0/1527633927 <== mds.0 v2:172.21.15.45:6833/268877398 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55d1f49ac3e0 con 0x55d1f4a5bf40 2024-04-16T16:17:13.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.948+0000 7f5116a00640 10 client.6741 handle_command_reply: tid=0 2024-04-16T16:17:13.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.948+0000 7f513f567740 1 client.6741 shutdown 2024-04-16T16:17:13.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.948+0000 7f513f567740 1 -- 172.21.15.45:0/1527633927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60650 msgr2=0x55d1f4af8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.948+0000 7f513f567740 1 --2- 172.21.15.45:0/1527633927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60650 0x55d1f4af8d60 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f5128077720 tx=0x7f512803c040 comp rx=0 tx=0).stop 2024-04-16T16:17:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.948+0000 7f513f567740 1 -- 172.21.15.45:0/1527633927 shutdown_connections 2024-04-16T16:17:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.948+0000 7f513f567740 1 --2- 172.21.15.45:0/1527633927 >> [v2:172.21.15.45:6833/268877398,v1:172.21.15.45:6835/268877398] conn(0x55d1f4a5bf40 0x55d1f4a5c320 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513f567740 1 --2- 172.21.15.45:0/1527633927 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1f4a619b0 0x55d1f4a33570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513f567740 1 --2- 172.21.15.45:0/1527633927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1f4a60a30 0x55d1f4a33030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513f567740 1 --2- 172.21.15.45:0/1527633927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d1f4a60650 0x55d1f4af8d60 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f5116a00640 0 client.6741 ms_handle_reset on v2:172.21.15.45:6833/268877398 2024-04-16T16:17:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513f567740 1 -- 172.21.15.45:0/1527633927 >> 172.21.15.45:0/1527633927 conn(0x55d1f4a39870 msgr2=0x55d1f4a67440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513f567740 1 -- 172.21.15.45:0/1527633927 shutdown_connections 2024-04-16T16:17:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513f567740 1 -- 172.21.15.45:0/1527633927 wait complete. 2024-04-16T16:17:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513f567740 20 client.6741 trim_cache size 0 max 16384 2024-04-16T16:17:13.951 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:17:13.952 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "748c4b85-0190-4c59-80ad-098a6a5007c0", 2024-04-16T16:17:13.952 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-04-16T16:17:13.952 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 6402, 2024-04-16T16:17:13.952 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-04-16T16:17:13.952 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-04-16T16:17:13.952 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-04-16T16:17:13.952 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 33.898231727000002, 2024-04-16T16:17:13.952 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 53, 2024-04-16T16:17:13.952 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 73, 2024-04-16T16:17:13.952 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 73, 2024-04-16T16:17:13.952 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 40.265572128000002 2024-04-16T16:17:13.952 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:17:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513c800640 1 -- 172.21.15.45:0/2086996988 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f511806ca60 msgr2=0x7f511806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513c800640 1 --2- 172.21.15.45:0/2086996988 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f511806ca60 0x7f511806ef20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f513415d940 tx=0x7f5124008040 comp rx=0 tx=0).stop 2024-04-16T16:17:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513c800640 1 -- 172.21.15.45:0/2086996988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51340726c0 msgr2=0x7f51341f4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513c800640 1 --2- 172.21.15.45:0/2086996988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51340726c0 0x7f51341f4e20 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f512802b930 tx=0x7f51280046c0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513c800640 1 -- 172.21.15.45:0/2086996988 shutdown_connections 2024-04-16T16:17:13.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513c800640 1 --2- 172.21.15.45:0/2086996988 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f511806ca60 0x7f511806ef20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513c800640 1 --2- 172.21.15.45:0/2086996988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5134147550 0x7f51341f58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513c800640 1 --2- 172.21.15.45:0/2086996988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51340b58e0 0x7f51341f5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.949+0000 7f513c800640 1 --2- 172.21.15.45:0/2086996988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51340726c0 0x7f51341f4e20 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:13.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.950+0000 7f513c800640 1 -- 172.21.15.45:0/2086996988 >> 172.21.15.45:0/2086996988 conn(0x7f513406b930 msgr2=0x7f51340b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:13.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.950+0000 7f513c800640 1 -- 172.21.15.45:0/2086996988 shutdown_connections 2024-04-16T16:17:13.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:13.950+0000 7f513c800640 1 -- 172.21.15.45:0/2086996988 wait complete. 2024-04-16T16:17:13.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:17:14.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.185+0000 7fd752e00640 1 Processor -- start 2024-04-16T16:17:14.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.185+0000 7fd752e00640 1 -- start start 2024-04-16T16:17:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.187+0000 7fd752e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd74c06dfb0 0x7fd74c0c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.187+0000 7fd752e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd74c0c5bb0 0x7fd74c15d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.187+0000 7fd752e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd74c15c700 con 0x7fd74c144ff0 2024-04-16T16:17:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.187+0000 7fd752e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd74c15d2f0 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.187+0000 7fd752e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd74c15c9c0 con 0x7fd74c06dfb0 2024-04-16T16:17:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.187+0000 7fd752400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd74c0c5bb0 0x7fd74c15d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.187+0000 7fd752400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd74c0c5bb0 0x7fd74c15d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47220/0 (socket says 172.21.15.45:47220) 2024-04-16T16:17:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.187+0000 7fd752400640 1 -- 172.21.15.45:0/3185441300 learned_addr learned my addr 172.21.15.45:0/3185441300 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:14.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.187+0000 7fd751000640 1 --2- 172.21.15.45:0/3185441300 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd74c06dfb0 0x7fd74c0c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:14.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.187+0000 7fd752400640 1 -- 172.21.15.45:0/3185441300 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd74c06dfb0 msgr2=0x7fd74c0c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:14.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.188+0000 7fd752400640 1 --2- 172.21.15.45:0/3185441300 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd74c06dfb0 0x7fd74c0c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:14.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.188+0000 7fd752400640 1 -- 172.21.15.45:0/3185441300 >> v1:172.21.15.103:6789/0 conn(0x7fd74c144ff0 legacy=0x7fd74c0cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:14.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.188+0000 7fd752400640 1 -- 172.21.15.45:0/3185441300 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd74c15cca0 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.188+0000 7fd751000640 1 --2- 172.21.15.45:0/3185441300 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd74c06dfb0 0x7fd74c0c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.189+0000 7fd752400640 1 --2- 172.21.15.45:0/3185441300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c0c5bb0 0x7fd74c15d780 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd744009870 tx=0x7fd74402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c198c61323ae459 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.189+0000 7fd74be00640 1 -- 172.21.15.45:0/3185441300 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd744035020 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.189+0000 7fd74be00640 1 -- 172.21.15.45:0/3185441300 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd744031640 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.189+0000 7fd74be00640 1 -- 172.21.15.45:0/3185441300 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74402d040 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.190+0000 7fd752e00640 1 -- 172.21.15.45:0/3185441300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c0c5bb0 msgr2=0x7fd74c15d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.190+0000 7fd752e00640 1 --2- 172.21.15.45:0/3185441300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c0c5bb0 0x7fd74c15d780 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd744009870 tx=0x7fd74402b8f0 comp rx=0 tx=0).stop 2024-04-16T16:17:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.190+0000 7fd752e00640 1 -- 172.21.15.45:0/3185441300 shutdown_connections 2024-04-16T16:17:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.190+0000 7fd752e00640 1 --2- 172.21.15.45:0/3185441300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c0c5bb0 0x7fd74c15d780 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.190+0000 7fd752e00640 1 --2- 172.21.15.45:0/3185441300 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd74c06dfb0 0x7fd74c0c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.190+0000 7fd752e00640 1 -- 172.21.15.45:0/3185441300 >> 172.21.15.45:0/3185441300 conn(0x7fd74c06b930 msgr2=0x7fd74c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.190+0000 7fd752e00640 1 -- 172.21.15.45:0/3185441300 shutdown_connections 2024-04-16T16:17:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.190+0000 7fd752e00640 1 -- 172.21.15.45:0/3185441300 wait complete. 2024-04-16T16:17:14.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.192+0000 7fd752e00640 1 Processor -- start 2024-04-16T16:17:14.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.192+0000 7fd752e00640 1 -- start start 2024-04-16T16:17:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.193+0000 7fd752e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd74c06dfb0 0x7fd74c1f7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.193+0000 7fd752e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c0c5bb0 0x7fd74c0b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.193+0000 7fd752e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd74c144ff0 0x7fd74c06e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.193+0000 7fd752e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd74c15dcc0 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.193+0000 7fd752e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd74c1453d0 con 0x7fd74c144ff0 2024-04-16T16:17:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.193+0000 7fd752e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd74c15de40 con 0x7fd74c06dfb0 2024-04-16T16:17:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.193+0000 7fd751000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c0c5bb0 0x7fd74c0b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.193+0000 7fd751000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c0c5bb0 0x7fd74c0b4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47230/0 (socket says 172.21.15.45:47230) 2024-04-16T16:17:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.193+0000 7fd751000640 1 -- 172.21.15.45:0/3286999121 learned_addr learned my addr 172.21.15.45:0/3286999121 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:14.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.193+0000 7fd751a00640 1 --2- 172.21.15.45:0/3286999121 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd74c06dfb0 0x7fd74c1f7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:14.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.193+0000 7fd751000640 1 -- 172.21.15.45:0/3286999121 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd74c06dfb0 msgr2=0x7fd74c1f7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:14.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.194+0000 7fd751000640 1 --2- 172.21.15.45:0/3286999121 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd74c06dfb0 0x7fd74c1f7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:14.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.194+0000 7fd752400640 1 --2- 172.21.15.45:0/3286999121 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd74c144ff0 0x7fd74c06e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:14.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.194+0000 7fd751000640 1 -- 172.21.15.45:0/3286999121 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd74c144ff0 msgr2=0x7fd74c06e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:14.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.194+0000 7fd751000640 1 --2- 172.21.15.45:0/3286999121 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd74c144ff0 0x7fd74c06e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:14.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.194+0000 7fd751000640 1 -- 172.21.15.45:0/3286999121 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd74c1f7c70 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.194+0000 7fd751000640 1 --2- 172.21.15.45:0/3286999121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c0c5bb0 0x7fd74c0b4de0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fd73800c7f0 tx=0x7fd73800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:14.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.194+0000 7fd749600640 1 -- 172.21.15.45:0/3286999121 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd738004230 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.194+0000 7fd749600640 1 -- 172.21.15.45:0/3286999121 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7380043d0 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.194+0000 7fd749600640 1 -- 172.21.15.45:0/3286999121 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd738007a70 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.194+0000 7fd752e00640 1 -- 172.21.15.45:0/3286999121 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd74c0cc230 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.195+0000 7fd752e00640 1 -- 172.21.15.45:0/3286999121 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd74c0cc7f0 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.196+0000 7fd752e00640 1 -- 172.21.15.45:0/3286999121 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd74c0ccf00 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.197+0000 7fd749600640 1 -- 172.21.15.45:0/3286999121 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7fd738007c10 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.198+0000 7fd749600640 1 --2- 172.21.15.45:0/3286999121 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd72c06ca60 0x7fd72c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:14.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.198+0000 7fd749600640 1 -- 172.21.15.45:0/3286999121 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7fd73808c060 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.198+0000 7fd751a00640 1 --2- 172.21.15.45:0/3286999121 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd72c06ca60 0x7fd72c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:14.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.200+0000 7fd751a00640 1 --2- 172.21.15.45:0/3286999121 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd72c06ca60 0x7fd72c06ef20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd74c0cd4d0 tx=0x7fd7400072f0 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:14.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.202+0000 7fd749600640 1 -- 172.21.15.45:0/3286999121 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd738015020 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.445+0000 7fd752e00640 1 -- 172.21.15.45:0/3286999121 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd74c075780 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.446+0000 7fd749600640 1 -- 172.21.15.45:0/3286999121 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5200 (secure 0 0 0) 0x7fd73805a0c0 con 0x7fd74c0c5bb0 2024-04-16T16:17:14.445 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:14.446 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":53,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:17:11.564712+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6402,"mds_1":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"c","rank":1,"incarnation":52,"state":"up:active","state_seq":11,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6399,6402]},"id":5}]} 2024-04-16T16:17:14.446 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 53 2024-04-16T16:17:14.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.449+0000 7fd752e00640 1 -- 172.21.15.45:0/3286999121 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd72c06ca60 msgr2=0x7fd72c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:14.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.449+0000 7fd752e00640 1 --2- 172.21.15.45:0/3286999121 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd72c06ca60 0x7fd72c06ef20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd74c0cd4d0 tx=0x7fd7400072f0 comp rx=0 tx=0).stop 2024-04-16T16:17:14.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.449+0000 7fd752e00640 1 -- 172.21.15.45:0/3286999121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c0c5bb0 msgr2=0x7fd74c0b4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:14.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.449+0000 7fd752e00640 1 --2- 172.21.15.45:0/3286999121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c0c5bb0 0x7fd74c0b4de0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fd73800c7f0 tx=0x7fd73800ccb0 comp rx=0 tx=0).stop 2024-04-16T16:17:14.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.449+0000 7fd752e00640 1 -- 172.21.15.45:0/3286999121 shutdown_connections 2024-04-16T16:17:14.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.449+0000 7fd752e00640 1 --2- 172.21.15.45:0/3286999121 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd72c06ca60 0x7fd72c06ef20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:14.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.449+0000 7fd752e00640 1 --2- 172.21.15.45:0/3286999121 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd74c144ff0 0x7fd74c06e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:14.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.449+0000 7fd752e00640 1 --2- 172.21.15.45:0/3286999121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c0c5bb0 0x7fd74c0b4de0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:14.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.449+0000 7fd752e00640 1 --2- 172.21.15.45:0/3286999121 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd74c06dfb0 0x7fd74c1f7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:14.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.449+0000 7fd752e00640 1 -- 172.21.15.45:0/3286999121 >> 172.21.15.45:0/3286999121 conn(0x7fd74c06b930 msgr2=0x7fd74c0c64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:14.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.449+0000 7fd752e00640 1 -- 172.21.15.45:0/3286999121 shutdown_connections 2024-04-16T16:17:14.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:14.450+0000 7fd752e00640 1 -- 172.21.15.45:0/3286999121 wait complete. 2024-04-16T16:17:14.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:14.467 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir test_mds_metrics_0) 2024-04-16T16:17:14.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:14.529 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 test_mds_metrics_0) 2024-04-16T16:17:14.595 INFO:tasks.cephfs.test_mds_metrics:pinning directory test_mds_metrics_0 to rank 0 2024-04-16T16:17:14.595 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:14.595 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:14.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:14.652 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.0 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:14.747 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:14.747 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:14.747 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409987 s, 25.6 MB/s 2024-04-16T16:17:14.749 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:14.749 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:14.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:14.807 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.1 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:14.902 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:14.902 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:14.902 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359774 s, 29.1 MB/s 2024-04-16T16:17:14.903 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:14.904 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:14.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:14.960 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.2 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:15.036 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:15.037 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:15.037 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224553 s, 46.7 MB/s 2024-04-16T16:17:15.038 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:15.039 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:15.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:15.095 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.3 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:15.187 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:15.187 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:15.187 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338068 s, 31.0 MB/s 2024-04-16T16:17:15.189 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:15.189 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:15.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:15.248 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.4 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:15.324 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:15.324 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:15.324 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189359 s, 55.4 MB/s 2024-04-16T16:17:15.325 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:15.325 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:15.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:15.380 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.5 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300597 s, 34.9 MB/s 2024-04-16T16:17:15.458 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:15.458 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:15.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:15.511 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.6 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:15.596 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:15.597 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:15.597 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335585 s, 31.2 MB/s 2024-04-16T16:17:15.599 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:15.599 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:15.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:15.658 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.7 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322174 s, 32.5 MB/s 2024-04-16T16:17:15.747 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:15.748 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:15.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:15.806 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.8 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:15.894 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:15.894 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:15.894 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320549 s, 32.7 MB/s 2024-04-16T16:17:15.896 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:15.896 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:15.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:15.956 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.9 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:16.045 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:16.045 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:16.045 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309806 s, 33.8 MB/s 2024-04-16T16:17:16.047 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:16.048 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:16.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:16.108 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.10 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:16.193 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:16.194 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:16.194 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291827 s, 35.9 MB/s 2024-04-16T16:17:16.196 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:16.196 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:16.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:16.256 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.11 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:16.342 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:16.342 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:16.342 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312552 s, 33.5 MB/s 2024-04-16T16:17:16.344 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:16.344 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:16.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:16.400 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.12 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:16.490 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:16.490 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:16.490 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348934 s, 30.1 MB/s 2024-04-16T16:17:16.492 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:16.492 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:16.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:16.552 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.13 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:16.633 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:16.633 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:16.633 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238795 s, 43.9 MB/s 2024-04-16T16:17:16.635 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:16.635 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:16.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:16.694 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.14 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:16.777 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:16.778 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:16.778 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257395 s, 40.7 MB/s 2024-04-16T16:17:16.780 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:16.780 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:16.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:16.839 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.15 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:16.924 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:16.924 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:16.924 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029901 s, 35.1 MB/s 2024-04-16T16:17:16.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:16.926 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir test_mds_metrics_1) 2024-04-16T16:17:16.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:16.983 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 test_mds_metrics_1) 2024-04-16T16:17:17.055 INFO:tasks.cephfs.test_mds_metrics:pinning directory test_mds_metrics_1 to rank 1 2024-04-16T16:17:17.055 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:17.055 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:17.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:17.106 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.0 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:17.178 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:17.178 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:17.178 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203415 s, 51.5 MB/s 2024-04-16T16:17:17.180 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:17.180 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:17.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:17.228 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.1 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:17.354 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:17.354 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:17.354 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228974 s, 45.8 MB/s 2024-04-16T16:17:17.356 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:17.356 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:17.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:17.414 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.2 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241794 s, 43.4 MB/s 2024-04-16T16:17:17.501 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:17.501 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:17.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:17.559 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.3 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:17.650 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:17.650 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:17.650 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380918 s, 27.5 MB/s 2024-04-16T16:17:17.652 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:17.652 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:17.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:17.711 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.4 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:17.793 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:17.793 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:17.793 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278365 s, 37.7 MB/s 2024-04-16T16:17:17.795 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:17.795 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:17.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:17.853 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.5 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:17.936 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:17.936 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:17.936 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276085 s, 38.0 MB/s 2024-04-16T16:17:17.938 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:17.938 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:17.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:17.998 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.6 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255795 s, 41.0 MB/s 2024-04-16T16:17:18.083 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:18.083 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:18.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:18.139 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.7 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:18.222 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:18.222 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:18.223 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229655 s, 45.7 MB/s 2024-04-16T16:17:18.224 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:18.224 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:18.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:18.280 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.8 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:18.358 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:18.359 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:18.359 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250891 s, 41.8 MB/s 2024-04-16T16:17:18.360 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:18.360 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:18.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:18.417 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.9 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:18.500 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286907 s, 36.5 MB/s 2024-04-16T16:17:18.502 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:18.502 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:18.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:18.557 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.10 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:18.639 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:18.640 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:18.640 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024721 s, 42.4 MB/s 2024-04-16T16:17:18.641 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:18.641 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:18.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:18.701 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.11 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:18.779 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:18.779 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:18.779 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259189 s, 40.5 MB/s 2024-04-16T16:17:18.780 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:18.780 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:18.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:18.836 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.12 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:18.923 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:18.923 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:18.923 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027963 s, 37.5 MB/s 2024-04-16T16:17:18.924 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:18.924 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:18.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:18.983 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.13 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:19.069 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:19.069 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:19.069 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295031 s, 35.5 MB/s 2024-04-16T16:17:19.071 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:19.071 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:19.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:19.129 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.14 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:19.211 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:19.211 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:19.211 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284587 s, 36.8 MB/s 2024-04-16T16:17:19.212 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:19.212 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:19.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:19.270 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.15 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:17:19.353 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:17:19.353 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:17:19.354 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246965 s, 42.5 MB/s 2024-04-16T16:17:19.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:19.355 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec find .) 2024-04-16T16:17:19.437 INFO:teuthology.orchestra.run.smithi103.stdout:. 2024-04-16T16:17:19.437 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0 2024-04-16T16:17:19.437 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.5 2024-04-16T16:17:19.437 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.7 2024-04-16T16:17:19.437 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.6 2024-04-16T16:17:19.437 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.10 2024-04-16T16:17:19.437 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.4 2024-04-16T16:17:19.438 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.12 2024-04-16T16:17:19.438 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.1 2024-04-16T16:17:19.438 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.9 2024-04-16T16:17:19.438 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.13 2024-04-16T16:17:19.438 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.0 2024-04-16T16:17:19.438 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.11 2024-04-16T16:17:19.438 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.8 2024-04-16T16:17:19.438 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.3 2024-04-16T16:17:19.438 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.2 2024-04-16T16:17:19.438 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.14 2024-04-16T16:17:19.438 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.15 2024-04-16T16:17:19.438 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1 2024-04-16T16:17:19.439 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.5 2024-04-16T16:17:19.439 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.7 2024-04-16T16:17:19.439 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.6 2024-04-16T16:17:19.439 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.10 2024-04-16T16:17:19.439 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.4 2024-04-16T16:17:19.439 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.12 2024-04-16T16:17:19.439 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.1 2024-04-16T16:17:19.439 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.9 2024-04-16T16:17:19.439 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.13 2024-04-16T16:17:19.439 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.0 2024-04-16T16:17:19.439 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.11 2024-04-16T16:17:19.439 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.8 2024-04-16T16:17:19.439 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.3 2024-04-16T16:17:19.439 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.2 2024-04-16T16:17:19.440 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.14 2024-04-16T16:17:19.440 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.15 2024-04-16T16:17:24.111 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:17:24.119 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:17:24.441 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:24.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.686+0000 7ff596400640 1 Processor -- start 2024-04-16T16:17:24.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.687+0000 7ff596400640 1 -- start start 2024-04-16T16:17:24.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.688+0000 7ff596400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff590162090 0x7ff5900cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:24.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.688+0000 7ff596400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff5900d5520 0x7ff5900cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:24.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.688+0000 7ff596400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5900d2600 con 0x7ff5900ba140 2024-04-16T16:17:24.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.688+0000 7ff596400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff5900ba8b0 con 0x7ff5900d5520 2024-04-16T16:17:24.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.688+0000 7ff596400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff5900ba5d0 con 0x7ff590162090 2024-04-16T16:17:24.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.689+0000 7ff58fe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff5900ba140 0x7ff5901628a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47334/0 (socket says 172.21.15.45:47334) 2024-04-16T16:17:24.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.689+0000 7ff58fe00640 1 -- 172.21.15.45:0/1648170752 learned_addr learned my addr 172.21.15.45:0/1648170752 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:24.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.689+0000 7ff595000640 1 --2- 172.21.15.45:0/1648170752 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff590162090 0x7ff5900cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:24.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.689+0000 7ff58f400640 1 -- 172.21.15.45:0/1648170752 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 668810046 0 0) 0x7ff5900d2600 con 0x7ff5900ba140 2024-04-16T16:17:24.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.689+0000 7ff58f400640 1 -- 172.21.15.45:0/1648170752 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff570003520 con 0x7ff5900ba140 2024-04-16T16:17:24.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.689+0000 7ff595a00640 1 --2- 172.21.15.45:0/1648170752 >> v2:172.21.15.103:3300/0 conn(0x7ff5900d5520 0x7ff5900cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:24.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff58f400640 1 -- 172.21.15.45:0/1648170752 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3617069832 0 0) 0x7ff570003520 con 0x7ff5900ba140 2024-04-16T16:17:24.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff58f400640 1 -- 172.21.15.45:0/1648170752 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff590162090 msgr2=0x7ff5900cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:24.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff58f400640 1 --2- 172.21.15.45:0/1648170752 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff590162090 0x7ff5900cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff58f400640 1 -- 172.21.15.45:0/1648170752 >> v2:172.21.15.103:3300/0 conn(0x7ff5900d5520 msgr2=0x7ff5900cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff58f400640 1 --2- 172.21.15.45:0/1648170752 >> v2:172.21.15.103:3300/0 conn(0x7ff5900d5520 0x7ff5900cfa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff58f400640 1 -- 172.21.15.45:0/1648170752 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5900d28a0 con 0x7ff5900ba140 2024-04-16T16:17:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff58f400640 1 -- 172.21.15.45:0/1648170752 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7ff578001670 con 0x7ff5900ba140 2024-04-16T16:17:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff595a00640 1 --2- 172.21.15.45:0/1648170752 >> v2:172.21.15.103:3300/0 conn(0x7ff5900d5520 0x7ff5900cfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:17:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff58f400640 1 -- 172.21.15.45:0/1648170752 >> v1:172.21.15.45:6789/0 conn(0x7ff5900ba140 legacy=0x7ff5901628a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff58f400640 1 --2- 172.21.15.45:0/1648170752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5700039b0 0x7ff570003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff58f400640 1 --2- 172.21.15.45:0/1648170752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff570004800 0x7ff570004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff58f400640 1 --2- 172.21.15.45:0/1648170752 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff590162090 0x7ff5700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff58f400640 1 -- 172.21.15.45:0/1648170752 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff5900ba8b0 con 0x7ff570004800 2024-04-16T16:17:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff58f400640 1 -- 172.21.15.45:0/1648170752 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff5900ba5d0 con 0x7ff5700039b0 2024-04-16T16:17:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff58f400640 1 -- 172.21.15.45:0/1648170752 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff57000ba10 con 0x7ff590162090 2024-04-16T16:17:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff58fe00640 1 --2- 172.21.15.45:0/1648170752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff570004800 0x7ff570004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.690+0000 7ff595a00640 1 --2- 172.21.15.45:0/1648170752 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff590162090 0x7ff5700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:24.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.691+0000 7ff595000640 1 --2- 172.21.15.45:0/1648170752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5700039b0 0x7ff570003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:24.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.691+0000 7ff58fe00640 1 -- 172.21.15.45:0/1648170752 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff590162090 msgr2=0x7ff5700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:24.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.691+0000 7ff58fe00640 1 --2- 172.21.15.45:0/1648170752 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff590162090 0x7ff5700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:24.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.691+0000 7ff58fe00640 1 -- 172.21.15.45:0/1648170752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5700039b0 msgr2=0x7ff570003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:24.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.691+0000 7ff58fe00640 1 --2- 172.21.15.45:0/1648170752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5700039b0 0x7ff570003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:24.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.691+0000 7ff58fe00640 1 -- 172.21.15.45:0/1648170752 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff570003520 con 0x7ff570004800 2024-04-16T16:17:24.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.692+0000 7ff58fe00640 1 --2- 172.21.15.45:0/1648170752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff570004800 0x7ff570004c80 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c190 tx=0x7ff570005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e34f241d3483ae7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:24.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.692+0000 7ff58f400640 1 -- 172.21.15.45:0/1648170752 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff578007c80 con 0x7ff570004800 2024-04-16T16:17:24.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.692+0000 7ff58f400640 1 -- 172.21.15.45:0/1648170752 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff578030070 con 0x7ff570004800 2024-04-16T16:17:24.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.692+0000 7ff58f400640 1 -- 172.21.15.45:0/1648170752 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff578005d30 con 0x7ff570004800 2024-04-16T16:17:24.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.693+0000 7ff596400640 1 -- 172.21.15.45:0/1648170752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff570004800 msgr2=0x7ff570004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:24.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.693+0000 7ff596400640 1 --2- 172.21.15.45:0/1648170752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff570004800 0x7ff570004c80 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c190 tx=0x7ff570005830 comp rx=0 tx=0).stop 2024-04-16T16:17:24.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.693+0000 7ff595a00640 1 -- 172.21.15.45:0/1648170752 reap_dead start 2024-04-16T16:17:24.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.693+0000 7ff596400640 1 -- 172.21.15.45:0/1648170752 shutdown_connections 2024-04-16T16:17:24.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.693+0000 7ff596400640 1 -- 172.21.15.45:0/1648170752 >> 172.21.15.45:0/1648170752 conn(0x7ff5900d1940 msgr2=0x7ff5900d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:24.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.693+0000 7ff596400640 1 -- 172.21.15.45:0/1648170752 shutdown_connections 2024-04-16T16:17:24.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.693+0000 7ff596400640 1 -- 172.21.15.45:0/1648170752 wait complete. 2024-04-16T16:17:24.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.695+0000 7ff596400640 1 Processor -- start 2024-04-16T16:17:24.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.695+0000 7ff596400640 1 -- start start 2024-04-16T16:17:24.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.697+0000 7ff596400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff570004800 0x7ff5901f77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:24.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.697+0000 7ff596400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff59015de50 0x7ff59015e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:24.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.697+0000 7ff596400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5901558e0 0x7ff590155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:24.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.697+0000 7ff596400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff5901625f0 con 0x7ff570004800 2024-04-16T16:17:24.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.697+0000 7ff596400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff590162470 con 0x7ff5901558e0 2024-04-16T16:17:24.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.697+0000 7ff596400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff5900d40b0 con 0x7ff59015de50 2024-04-16T16:17:24.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.697+0000 7ff595000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff570004800 0x7ff5901f77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:24.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.697+0000 7ff595000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff570004800 0x7ff5901f77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41510/0 (socket says 172.21.15.45:41510) 2024-04-16T16:17:24.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.697+0000 7ff595000640 1 -- 172.21.15.45:0/1162512324 learned_addr learned my addr 172.21.15.45:0/1162512324 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.697+0000 7ff595000640 1 -- 172.21.15.45:0/1162512324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff59015de50 msgr2=0x7ff59015e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.697+0000 7ff595000640 1 --2- 172.21.15.45:0/1162512324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff59015de50 0x7ff59015e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.697+0000 7ff595000640 1 -- 172.21.15.45:0/1162512324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5901558e0 msgr2=0x7ff590155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.697+0000 7ff595000640 1 --2- 172.21.15.45:0/1162512324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5901558e0 0x7ff590155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.697+0000 7ff595000640 1 -- 172.21.15.45:0/1162512324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5901562a0 con 0x7ff570004800 2024-04-16T16:17:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.698+0000 7ff595000640 1 --2- 172.21.15.45:0/1162512324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff570004800 0x7ff5901f77f0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ff580011600 tx=0x7ff580011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.698+0000 7ff58cc00640 1 -- 172.21.15.45:0/1162512324 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580015870 con 0x7ff570004800 2024-04-16T16:17:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.698+0000 7ff58cc00640 1 -- 172.21.15.45:0/1162512324 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff580015a10 con 0x7ff570004800 2024-04-16T16:17:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.698+0000 7ff58cc00640 1 -- 172.21.15.45:0/1162512324 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580015e00 con 0x7ff570004800 2024-04-16T16:17:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.698+0000 7ff596400640 1 -- 172.21.15.45:0/1162512324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff590156560 con 0x7ff570004800 2024-04-16T16:17:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.698+0000 7ff596400640 1 -- 172.21.15.45:0/1162512324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff590156b20 con 0x7ff570004800 2024-04-16T16:17:24.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.699+0000 7ff596400640 1 -- 172.21.15.45:0/1162512324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff590075780 con 0x7ff570004800 2024-04-16T16:17:24.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.700+0000 7ff58cc00640 1 -- 172.21.15.45:0/1162512324 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7ff58000f450 con 0x7ff570004800 2024-04-16T16:17:24.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.701+0000 7ff58cc00640 1 --2- 172.21.15.45:0/1162512324 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff56c06ca90 0x7ff56c06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:24.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.702+0000 7ff58cc00640 1 -- 172.21.15.45:0/1162512324 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7ff58008c7a0 con 0x7ff570004800 2024-04-16T16:17:24.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.702+0000 7ff58fe00640 1 --2- 172.21.15.45:0/1162512324 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff56c06ca90 0x7ff56c06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:24.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.703+0000 7ff58fe00640 1 --2- 172.21.15.45:0/1162512324 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff56c06ca90 0x7ff56c06ef50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff578034000 tx=0x7ff578039040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:24.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.707+0000 7ff58cc00640 1 -- 172.21.15.45:0/1162512324 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff580019030 con 0x7ff570004800 2024-04-16T16:17:24.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.937+0000 7ff596400640 1 -- 172.21.15.45:0/1162512324 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7ff59006dab0 con 0x7ff56c06ca90 2024-04-16T16:17:24.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.940+0000 7ff58cc00640 1 -- 172.21.15.45:0/1162512324 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3009 (secure 0 0 0) 0x7ff59006dab0 con 0x7ff56c06ca90 2024-04-16T16:17:24.939 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[419, 22], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], "client.5285": [[2, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[2, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[52, 2], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}}} 2024-04-16T16:17:24.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.943+0000 7ff596400640 1 -- 172.21.15.45:0/1162512324 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff56c06ca90 msgr2=0x7ff56c06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:24.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.943+0000 7ff596400640 1 --2- 172.21.15.45:0/1162512324 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff56c06ca90 0x7ff56c06ef50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff578034000 tx=0x7ff578039040 comp rx=0 tx=0).stop 2024-04-16T16:17:24.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.943+0000 7ff596400640 1 -- 172.21.15.45:0/1162512324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff570004800 msgr2=0x7ff5901f77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:24.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.943+0000 7ff596400640 1 --2- 172.21.15.45:0/1162512324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff570004800 0x7ff5901f77f0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ff580011600 tx=0x7ff580011ac0 comp rx=0 tx=0).stop 2024-04-16T16:17:24.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.943+0000 7ff596400640 1 -- 172.21.15.45:0/1162512324 shutdown_connections 2024-04-16T16:17:24.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.943+0000 7ff596400640 1 --2- 172.21.15.45:0/1162512324 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff56c06ca90 0x7ff56c06ef50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:24.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.943+0000 7ff596400640 1 --2- 172.21.15.45:0/1162512324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5901558e0 0x7ff590155d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:24.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.943+0000 7ff596400640 1 --2- 172.21.15.45:0/1162512324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff59015de50 0x7ff59015e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:24.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.943+0000 7ff596400640 1 --2- 172.21.15.45:0/1162512324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff570004800 0x7ff5901f77f0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:24.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.943+0000 7ff596400640 1 -- 172.21.15.45:0/1162512324 >> 172.21.15.45:0/1162512324 conn(0x7ff5900d1940 msgr2=0x7ff5900b5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:24.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.943+0000 7ff596400640 1 -- 172.21.15.45:0/1162512324 shutdown_connections 2024-04-16T16:17:24.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:24.943+0000 7ff596400640 1 -- 172.21.15.45:0/1162512324 wait complete. 2024-04-16T16:17:25.963 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:26.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.196+0000 7f573cc00640 1 Processor -- start 2024-04-16T16:17:26.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.197+0000 7f573cc00640 1 -- start start 2024-04-16T16:17:26.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.198+0000 7f573cc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f57380cfc20 0x7f57380d37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.198+0000 7f573cc00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57380d2600 con 0x7f5738162090 2024-04-16T16:17:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.198+0000 7f573cc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57380ba5d0 con 0x7f57380ba140 2024-04-16T16:17:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.198+0000 7f573cc00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f57380ba8b0 con 0x7f57380cfc20 2024-04-16T16:17:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.198+0000 7f5737e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f57380cfc20 0x7f57380d37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.199+0000 7f5737400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5738162090 0x7f57380cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47344/0 (socket says 172.21.15.45:47344) 2024-04-16T16:17:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.199+0000 7f5737e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f57380cfc20 0x7f57380d37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41524/0 (socket says 172.21.15.45:41524) 2024-04-16T16:17:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.199+0000 7f5737400640 1 -- 172.21.15.45:0/1509415389 learned_addr learned my addr 172.21.15.45:0/1509415389 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.199+0000 7f5737e00640 1 -- 172.21.15.45:0/1509415389 >> v1:172.21.15.103:6789/0 conn(0x7f57380ba140 legacy=0x7f57380d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.199+0000 7f5737e00640 1 -- 172.21.15.45:0/1509415389 >> v1:172.21.15.45:6789/0 conn(0x7f5738162090 legacy=0x7f57380cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.199+0000 7f5737e00640 1 -- 172.21.15.45:0/1509415389 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57380d28a0 con 0x7f57380cfc20 2024-04-16T16:17:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.200+0000 7f5737e00640 1 --2- 172.21.15.45:0/1509415389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57380cfc20 0x7f57380d37b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5730009d90 tx=0x7f573002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c7f3cbe78e35fd server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.201+0000 7f5736a00640 1 -- 172.21.15.45:0/1509415389 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730039070 con 0x7f57380cfc20 2024-04-16T16:17:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.201+0000 7f5736a00640 1 -- 172.21.15.45:0/1509415389 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f573002cc60 con 0x7f57380cfc20 2024-04-16T16:17:26.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.201+0000 7f5736a00640 1 -- 172.21.15.45:0/1509415389 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730034660 con 0x7f57380cfc20 2024-04-16T16:17:26.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.201+0000 7f573cc00640 1 -- 172.21.15.45:0/1509415389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57380cfc20 msgr2=0x7f57380d37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:26.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.201+0000 7f573cc00640 1 --2- 172.21.15.45:0/1509415389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57380cfc20 0x7f57380d37b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5730009d90 tx=0x7f573002beb0 comp rx=0 tx=0).stop 2024-04-16T16:17:26.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.201+0000 7f573cc00640 1 -- 172.21.15.45:0/1509415389 shutdown_connections 2024-04-16T16:17:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.201+0000 7f573cc00640 1 --2- 172.21.15.45:0/1509415389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57380cfc20 0x7f57380d37b0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.201+0000 7f573cc00640 1 -- 172.21.15.45:0/1509415389 >> 172.21.15.45:0/1509415389 conn(0x7f57380d1940 msgr2=0x7f57380d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.201+0000 7f573cc00640 1 -- 172.21.15.45:0/1509415389 shutdown_connections 2024-04-16T16:17:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.201+0000 7f573cc00640 1 -- 172.21.15.45:0/1509415389 wait complete. 2024-04-16T16:17:26.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.203+0000 7f573cc00640 1 Processor -- start 2024-04-16T16:17:26.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.203+0000 7f573cc00640 1 -- start start 2024-04-16T16:17:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.204+0000 7f573cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57380cfc20 0x7f57381f4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.204+0000 7f573cc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5738162090 0x7f573815b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.204+0000 7f573cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f573815bde0 0x7f57380b5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.204+0000 7f573cc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f57381625f0 con 0x7f57380cfc20 2024-04-16T16:17:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.204+0000 7f573cc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5738162470 con 0x7f573815bde0 2024-04-16T16:17:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.204+0000 7f573cc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f57380d5520 con 0x7f5738162090 2024-04-16T16:17:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.205+0000 7f5737400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57380cfc20 0x7f57381f4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.205+0000 7f5737400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57380cfc20 0x7f57381f4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41526/0 (socket says 172.21.15.45:41526) 2024-04-16T16:17:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.205+0000 7f5737400640 1 -- 172.21.15.45:0/1617080746 learned_addr learned my addr 172.21.15.45:0/1617080746 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.205+0000 7f5737e00640 1 --2- 172.21.15.45:0/1617080746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f573815bde0 0x7f57380b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.205+0000 7f572fe00640 1 --2- 172.21.15.45:0/1617080746 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5738162090 0x7f573815b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.205+0000 7f5737400640 1 -- 172.21.15.45:0/1617080746 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5738162090 msgr2=0x7f573815b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.205+0000 7f5737400640 1 --2- 172.21.15.45:0/1617080746 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5738162090 0x7f573815b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.205+0000 7f5737400640 1 -- 172.21.15.45:0/1617080746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f573815bde0 msgr2=0x7f57380b5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.205+0000 7f5737400640 1 --2- 172.21.15.45:0/1617080746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f573815bde0 0x7f57380b5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.205+0000 7f5737400640 1 -- 172.21.15.45:0/1617080746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57381474e0 con 0x7f57380cfc20 2024-04-16T16:17:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.205+0000 7f5737e00640 1 --2- 172.21.15.45:0/1617080746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f573815bde0 0x7f57380b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.206+0000 7f5737400640 1 --2- 172.21.15.45:0/1617080746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57380cfc20 0x7f57381f4780 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f572800c3c0 tx=0x7f572800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.206+0000 7f572f400640 1 -- 172.21.15.45:0/1617080746 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5728017070 con 0x7f57380cfc20 2024-04-16T16:17:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.206+0000 7f572f400640 1 -- 172.21.15.45:0/1617080746 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5728005ce0 con 0x7f57380cfc20 2024-04-16T16:17:26.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.206+0000 7f572f400640 1 -- 172.21.15.45:0/1617080746 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5728007310 con 0x7f57380cfc20 2024-04-16T16:17:26.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.206+0000 7f573cc00640 1 -- 172.21.15.45:0/1617080746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57381477c0 con 0x7f57380cfc20 2024-04-16T16:17:26.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.206+0000 7f573cc00640 1 -- 172.21.15.45:0/1617080746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5738147d80 con 0x7f57380cfc20 2024-04-16T16:17:26.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.207+0000 7f573cc00640 1 -- 172.21.15.45:0/1617080746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5738075780 con 0x7f57380cfc20 2024-04-16T16:17:26.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.208+0000 7f572f400640 1 -- 172.21.15.45:0/1617080746 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f572801a070 con 0x7f57380cfc20 2024-04-16T16:17:26.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.209+0000 7f572f400640 1 --2- 172.21.15.45:0/1617080746 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f571406ca90 0x7f571406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:26.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.210+0000 7f572f400640 1 -- 172.21.15.45:0/1617080746 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f572808b250 con 0x7f57380cfc20 2024-04-16T16:17:26.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.210+0000 7f572fe00640 1 --2- 172.21.15.45:0/1617080746 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f571406ca90 0x7f571406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:26.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.211+0000 7f572fe00640 1 --2- 172.21.15.45:0/1617080746 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f571406ca90 0x7f571406ef50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f57380cedb0 tx=0x7f5720009040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:26.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.215+0000 7f572f400640 1 -- 172.21.15.45:0/1617080746 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5728059320 con 0x7f57380cfc20 2024-04-16T16:17:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.447+0000 7f573cc00640 1 -- 172.21.15.45:0/1617080746 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f573806dab0 con 0x7f571406ca90 2024-04-16T16:17:26.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.451+0000 7f572f400640 1 -- 172.21.15.45:0/1617080746 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3009 (secure 0 0 0) 0x7f573806dab0 con 0x7f571406ca90 2024-04-16T16:17:26.450 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[419, 22], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], "client.5285": [[2, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[2, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[52, 2], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}}} 2024-04-16T16:17:26.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.453+0000 7f573cc00640 1 -- 172.21.15.45:0/1617080746 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f571406ca90 msgr2=0x7f571406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:26.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.453+0000 7f573cc00640 1 --2- 172.21.15.45:0/1617080746 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f571406ca90 0x7f571406ef50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f57380cedb0 tx=0x7f5720009040 comp rx=0 tx=0).stop 2024-04-16T16:17:26.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.453+0000 7f573cc00640 1 -- 172.21.15.45:0/1617080746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57380cfc20 msgr2=0x7f57381f4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:26.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.453+0000 7f573cc00640 1 --2- 172.21.15.45:0/1617080746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57380cfc20 0x7f57381f4780 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f572800c3c0 tx=0x7f572800c880 comp rx=0 tx=0).stop 2024-04-16T16:17:26.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.453+0000 7f573cc00640 1 -- 172.21.15.45:0/1617080746 shutdown_connections 2024-04-16T16:17:26.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.453+0000 7f573cc00640 1 --2- 172.21.15.45:0/1617080746 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f571406ca90 0x7f571406ef50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:26.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.453+0000 7f573cc00640 1 --2- 172.21.15.45:0/1617080746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f573815bde0 0x7f57380b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:26.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.453+0000 7f573cc00640 1 --2- 172.21.15.45:0/1617080746 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5738162090 0x7f573815b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:26.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.453+0000 7f573cc00640 1 --2- 172.21.15.45:0/1617080746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57380cfc20 0x7f57381f4780 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:26.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.453+0000 7f573cc00640 1 -- 172.21.15.45:0/1617080746 >> 172.21.15.45:0/1617080746 conn(0x7f57380d1940 msgr2=0x7f573815f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:26.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.454+0000 7f573cc00640 1 -- 172.21.15.45:0/1617080746 shutdown_connections 2024-04-16T16:17:26.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:26.454+0000 7f573cc00640 1 -- 172.21.15.45:0/1617080746 wait complete. 2024-04-16T16:17:27.475 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:27.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.709+0000 7f4a3b200640 1 Processor -- start 2024-04-16T16:17:27.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.709+0000 7f4a3b200640 1 -- start start 2024-04-16T16:17:27.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.710+0000 7f4a3b200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4a34162090 0x7f4a340cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:27.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.710+0000 7f4a3b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a340ba140 0x7f4a340cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:27.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.710+0000 7f4a3b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4a340d5520 0x7f4a341628a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:27.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.710+0000 7f4a3b200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f4a340ba8b0 con 0x7f4a34162090 2024-04-16T16:17:27.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.710+0000 7f4a3b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4a340d2600 con 0x7f4a340d5520 2024-04-16T16:17:27.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.710+0000 7f4a3b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4a340ba5d0 con 0x7f4a340ba140 2024-04-16T16:17:27.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.711+0000 7f4a39e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4a34162090 0x7f4a340cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:27.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.711+0000 7f4a39e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4a34162090 0x7f4a340cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41534/0 (socket says 172.21.15.45:41534) 2024-04-16T16:17:27.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.711+0000 7f4a39e00640 1 -- 172.21.15.45:0/1153456391 learned_addr learned my addr 172.21.15.45:0/1153456391 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:27.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.711+0000 7f4a3a800640 1 --2- 172.21.15.45:0/1153456391 >> v2:172.21.15.103:3300/0 conn(0x7f4a340d5520 0x7f4a341628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:27.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.711+0000 7f4a39400640 1 --2- 172.21.15.45:0/1153456391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a340ba140 0x7f4a340cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:27.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.711+0000 7f4a39e00640 1 -- 172.21.15.45:0/1153456391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a340ba140 msgr2=0x7f4a340cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:27.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.711+0000 7f4a39e00640 1 --2- 172.21.15.45:0/1153456391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a340ba140 0x7f4a340cfa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:27.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.711+0000 7f4a39e00640 1 -- 172.21.15.45:0/1153456391 >> v2:172.21.15.103:3300/0 conn(0x7f4a340d5520 msgr2=0x7f4a341628a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:27.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.711+0000 7f4a39e00640 1 --2- 172.21.15.45:0/1153456391 >> v2:172.21.15.103:3300/0 conn(0x7f4a340d5520 0x7f4a341628a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:27.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.712+0000 7f4a39e00640 1 -- 172.21.15.45:0/1153456391 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a340d28a0 con 0x7f4a34162090 2024-04-16T16:17:27.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.713+0000 7f4a39e00640 1 --2- 172.21.15.45:0/1153456391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a34162090 0x7f4a340cf510 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f4a340b7430 tx=0x7f4a2802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560e14879494a489 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:27.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.713+0000 7f4a38a00640 1 -- 172.21.15.45:0/1153456391 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2802ca70 con 0x7f4a34162090 2024-04-16T16:17:27.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.714+0000 7f4a38a00640 1 -- 172.21.15.45:0/1153456391 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a2802cc10 con 0x7f4a34162090 2024-04-16T16:17:27.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.714+0000 7f4a38a00640 1 -- 172.21.15.45:0/1153456391 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a280343f0 con 0x7f4a34162090 2024-04-16T16:17:27.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.714+0000 7f4a3b200640 1 -- 172.21.15.45:0/1153456391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a34162090 msgr2=0x7f4a340cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:27.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.714+0000 7f4a3b200640 1 --2- 172.21.15.45:0/1153456391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a34162090 0x7f4a340cf510 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f4a340b7430 tx=0x7f4a2802bed0 comp rx=0 tx=0).stop 2024-04-16T16:17:27.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.714+0000 7f4a3b200640 1 -- 172.21.15.45:0/1153456391 shutdown_connections 2024-04-16T16:17:27.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.714+0000 7f4a3b200640 1 --2- 172.21.15.45:0/1153456391 >> v2:172.21.15.103:3300/0 conn(0x7f4a340d5520 0x7f4a341628a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:27.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.714+0000 7f4a3b200640 1 --2- 172.21.15.45:0/1153456391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a340ba140 0x7f4a340cfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:27.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.714+0000 7f4a3b200640 1 --2- 172.21.15.45:0/1153456391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a34162090 0x7f4a340cf510 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:27.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.714+0000 7f4a3b200640 1 -- 172.21.15.45:0/1153456391 >> 172.21.15.45:0/1153456391 conn(0x7f4a340d1940 msgr2=0x7f4a340d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:27.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.714+0000 7f4a3b200640 1 -- 172.21.15.45:0/1153456391 shutdown_connections 2024-04-16T16:17:27.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.714+0000 7f4a3b200640 1 -- 172.21.15.45:0/1153456391 wait complete. 2024-04-16T16:17:27.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.716+0000 7f4a3b200640 1 Processor -- start 2024-04-16T16:17:27.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.716+0000 7f4a3b200640 1 -- start start 2024-04-16T16:17:27.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.717+0000 7f4a3b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a340d5520 0x7f4a341f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:27.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.717+0000 7f4a3b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a34162090 0x7f4a34161250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:27.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.717+0000 7f4a3b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a3415cce0 0x7f4a340cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:27.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.717+0000 7f4a3b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4a341625f0 con 0x7f4a3415cce0 2024-04-16T16:17:27.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.717+0000 7f4a3b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4a34162470 con 0x7f4a34162090 2024-04-16T16:17:27.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.717+0000 7f4a3b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4a340b36c0 con 0x7f4a340d5520 2024-04-16T16:17:27.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.718+0000 7f4a3a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a3415cce0 0x7f4a340cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:27.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.718+0000 7f4a3a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a3415cce0 0x7f4a340cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41536/0 (socket says 172.21.15.45:41536) 2024-04-16T16:17:27.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.718+0000 7f4a3a800640 1 -- 172.21.15.45:0/3110124505 learned_addr learned my addr 172.21.15.45:0/3110124505 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:27.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.718+0000 7f4a39e00640 1 --2- 172.21.15.45:0/3110124505 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a340d5520 0x7f4a341f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:27.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.718+0000 7f4a39400640 1 --2- 172.21.15.45:0/3110124505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a34162090 0x7f4a34161250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:27.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.718+0000 7f4a3a800640 1 -- 172.21.15.45:0/3110124505 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a340d5520 msgr2=0x7f4a341f4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:27.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.718+0000 7f4a3a800640 1 --2- 172.21.15.45:0/3110124505 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a340d5520 0x7f4a341f4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.718+0000 7f4a3a800640 1 -- 172.21.15.45:0/3110124505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a34162090 msgr2=0x7f4a34161250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.718+0000 7f4a3a800640 1 --2- 172.21.15.45:0/3110124505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a34162090 0x7f4a34161250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.718+0000 7f4a3a800640 1 -- 172.21.15.45:0/3110124505 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a341f5660 con 0x7f4a3415cce0 2024-04-16T16:17:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.718+0000 7f4a39400640 1 --2- 172.21.15.45:0/3110124505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a34162090 0x7f4a34161250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.718+0000 7f4a3a800640 1 --2- 172.21.15.45:0/3110124505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a3415cce0 0x7f4a340cf590 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000d5d0 tx=0x7f4a3000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.719+0000 7f4a2e000640 1 -- 172.21.15.45:0/3110124505 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a300125f0 con 0x7f4a3415cce0 2024-04-16T16:17:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.719+0000 7f4a2e000640 1 -- 172.21.15.45:0/3110124505 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a30019070 con 0x7f4a3415cce0 2024-04-16T16:17:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.719+0000 7f4a2e000640 1 -- 172.21.15.45:0/3110124505 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a300129d0 con 0x7f4a3415cce0 2024-04-16T16:17:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.719+0000 7f4a3b200640 1 -- 172.21.15.45:0/3110124505 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a341f5940 con 0x7f4a3415cce0 2024-04-16T16:17:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.719+0000 7f4a3b200640 1 -- 172.21.15.45:0/3110124505 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a3415f350 con 0x7f4a3415cce0 2024-04-16T16:17:27.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.720+0000 7f4a3b200640 1 -- 172.21.15.45:0/3110124505 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a34075780 con 0x7f4a3415cce0 2024-04-16T16:17:27.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.721+0000 7f4a2e000640 1 -- 172.21.15.45:0/3110124505 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f4a30012b70 con 0x7f4a3415cce0 2024-04-16T16:17:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.722+0000 7f4a2e000640 1 --2- 172.21.15.45:0/3110124505 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4a1806ca60 0x7f4a1806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.722+0000 7f4a2e000640 1 -- 172.21.15.45:0/3110124505 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f4a3008c380 con 0x7f4a3415cce0 2024-04-16T16:17:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.723+0000 7f4a39e00640 1 --2- 172.21.15.45:0/3110124505 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4a1806ca60 0x7f4a1806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:27.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.724+0000 7f4a39e00640 1 --2- 172.21.15.45:0/3110124505 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4a1806ca60 0x7f4a1806ef20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4a28035000 tx=0x7f4a28030000 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:27.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.727+0000 7f4a2e000640 1 -- 172.21.15.45:0/3110124505 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a3005a4e0 con 0x7f4a3415cce0 2024-04-16T16:17:27.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.960+0000 7f4a3b200640 1 -- 172.21.15.45:0/3110124505 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f4a34161790 con 0x7f4a1806ca60 2024-04-16T16:17:27.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.963+0000 7f4a2e000640 1 -- 172.21.15.45:0/3110124505 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3039 (secure 0 0 0) 0x7f4a34161790 con 0x7f4a1806ca60 2024-04-16T16:17:27.963 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[419, 22], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], "client.5285": [[2, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[2, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[52, 2], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}, "mds.1": {"client.5243": []}}} 2024-04-16T16:17:27.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.966+0000 7f4a3b200640 1 -- 172.21.15.45:0/3110124505 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4a1806ca60 msgr2=0x7f4a1806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:27.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.966+0000 7f4a3b200640 1 --2- 172.21.15.45:0/3110124505 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4a1806ca60 0x7f4a1806ef20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4a28035000 tx=0x7f4a28030000 comp rx=0 tx=0).stop 2024-04-16T16:17:27.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.966+0000 7f4a3b200640 1 -- 172.21.15.45:0/3110124505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a3415cce0 msgr2=0x7f4a340cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:27.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.966+0000 7f4a3b200640 1 --2- 172.21.15.45:0/3110124505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a3415cce0 0x7f4a340cf590 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000d5d0 tx=0x7f4a3000da90 comp rx=0 tx=0).stop 2024-04-16T16:17:27.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.966+0000 7f4a3b200640 1 -- 172.21.15.45:0/3110124505 shutdown_connections 2024-04-16T16:17:27.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.966+0000 7f4a3b200640 1 --2- 172.21.15.45:0/3110124505 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4a1806ca60 0x7f4a1806ef20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:27.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.966+0000 7f4a3b200640 1 --2- 172.21.15.45:0/3110124505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a3415cce0 0x7f4a340cf590 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:27.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.966+0000 7f4a3b200640 1 --2- 172.21.15.45:0/3110124505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a34162090 0x7f4a34161250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:27.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.966+0000 7f4a3b200640 1 --2- 172.21.15.45:0/3110124505 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a340d5520 0x7f4a341f4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:27.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.966+0000 7f4a3b200640 1 -- 172.21.15.45:0/3110124505 >> 172.21.15.45:0/3110124505 conn(0x7f4a340d1940 msgr2=0x7f4a340ca3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:27.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.967+0000 7f4a3b200640 1 -- 172.21.15.45:0/3110124505 shutdown_connections 2024-04-16T16:17:27.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:27.967+0000 7f4a3b200640 1 -- 172.21.15.45:0/3110124505 wait complete. 2024-04-16T16:17:27.985 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:17:28.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.204+0000 7f5156400640 1 Processor -- start 2024-04-16T16:17:28.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.204+0000 7f5156400640 1 -- start start 2024-04-16T16:17:28.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.205+0000 7f5156400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f515009b210 0x7f51500720d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.205+0000 7f5156400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f515009be50 con 0x7f515009ae30 2024-04-16T16:17:28.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.205+0000 7f5156400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51500cd5b0 con 0x7f5150160470 2024-04-16T16:17:28.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.205+0000 7f5156400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f515009bc00 con 0x7f515009b210 2024-04-16T16:17:28.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.206+0000 7f514f400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f515009ae30 0x7f515015fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47352/0 (socket says 172.21.15.45:47352) 2024-04-16T16:17:28.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.206+0000 7f514f400640 1 -- 172.21.15.45:0/3805560185 learned_addr learned my addr 172.21.15.45:0/3805560185 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:28.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.206+0000 7f514fe00640 1 --2- 172.21.15.45:0/3805560185 >> v2:172.21.15.103:3300/0 conn(0x7f515009b210 0x7f51500720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.206+0000 7f514e000640 1 -- 172.21.15.45:0/3805560185 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3948242908 0 0) 0x7f515009be50 con 0x7f515009ae30 2024-04-16T16:17:28.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.206+0000 7f514e000640 1 -- 172.21.15.45:0/3805560185 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5130003520 con 0x7f515009ae30 2024-04-16T16:17:28.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.207+0000 7f514e000640 1 -- 172.21.15.45:0/3805560185 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2822681205 0 0) 0x7f5130003520 con 0x7f515009ae30 2024-04-16T16:17:28.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.207+0000 7f514e000640 1 -- 172.21.15.45:0/3805560185 >> v2:172.21.15.103:3300/0 conn(0x7f515009b210 msgr2=0x7f51500720d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:28.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.207+0000 7f514e000640 1 --2- 172.21.15.45:0/3805560185 >> v2:172.21.15.103:3300/0 conn(0x7f515009b210 0x7f51500720d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.207+0000 7f514e000640 1 -- 172.21.15.45:0/3805560185 >> v1:172.21.15.103:6789/0 conn(0x7f5150160470 legacy=0x7f51500ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:28.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.207+0000 7f514e000640 1 -- 172.21.15.45:0/3805560185 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51500cd890 con 0x7f515009ae30 2024-04-16T16:17:28.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.207+0000 7f514e000640 1 -- 172.21.15.45:0/3805560185 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f5140001670 con 0x7f515009ae30 2024-04-16T16:17:28.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.207+0000 7f514e000640 1 -- 172.21.15.45:0/3805560185 >> v1:172.21.15.45:6789/0 conn(0x7f515009ae30 legacy=0x7f515015fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:28.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.207+0000 7f514e000640 1 --2- 172.21.15.45:0/3805560185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5130003a00 0x7f5130003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.207+0000 7f514e000640 1 --2- 172.21.15.45:0/3805560185 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5130004800 0x7f5130004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.207+0000 7f514e000640 1 --2- 172.21.15.45:0/3805560185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51300051c0 0x7f513000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.207+0000 7f514e000640 1 -- 172.21.15.45:0/3805560185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f515009bc00 con 0x7f5130003a00 2024-04-16T16:17:28.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.207+0000 7f514e000640 1 -- 172.21.15.45:0/3805560185 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f513000c0f0 con 0x7f51300051c0 2024-04-16T16:17:28.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.207+0000 7f514e000640 1 -- 172.21.15.45:0/3805560185 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f513000c2f0 con 0x7f5130004800 2024-04-16T16:17:28.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.207+0000 7f514f400640 1 --2- 172.21.15.45:0/3805560185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5130003a00 0x7f5130003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.208+0000 7f514fe00640 1 --2- 172.21.15.45:0/3805560185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51300051c0 0x7f513000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.208+0000 7f514f400640 1 -- 172.21.15.45:0/3805560185 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5130004800 msgr2=0x7f5130004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:28.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.208+0000 7f514ea00640 1 --2- 172.21.15.45:0/3805560185 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5130004800 0x7f5130004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.208+0000 7f514f400640 1 --2- 172.21.15.45:0/3805560185 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5130004800 0x7f5130004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.208+0000 7f514f400640 1 -- 172.21.15.45:0/3805560185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51300051c0 msgr2=0x7f513000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:28.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.208+0000 7f514f400640 1 --2- 172.21.15.45:0/3805560185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51300051c0 0x7f513000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.208+0000 7f514f400640 1 -- 172.21.15.45:0/3805560185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5130003520 con 0x7f5130003a00 2024-04-16T16:17:28.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.208+0000 7f514ea00640 1 --2- 172.21.15.45:0/3805560185 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5130004800 0x7f5130004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:28.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.208+0000 7f514fe00640 1 --2- 172.21.15.45:0/3805560185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51300051c0 0x7f513000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:28.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.208+0000 7f514fe00640 1 -- 172.21.15.45:0/3805560185 reap_dead start 2024-04-16T16:17:28.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.209+0000 7f514f400640 1 --2- 172.21.15.45:0/3805560185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5130003a00 0x7f5130003e40 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f514000c110 tx=0x7f5130005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1210f01363bfb452 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:28.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.209+0000 7f514e000640 1 -- 172.21.15.45:0/3805560185 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5140007c40 con 0x7f5130003a00 2024-04-16T16:17:28.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.209+0000 7f514e000640 1 -- 172.21.15.45:0/3805560185 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5140030070 con 0x7f5130003a00 2024-04-16T16:17:28.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.209+0000 7f514e000640 1 -- 172.21.15.45:0/3805560185 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5140005d70 con 0x7f5130003a00 2024-04-16T16:17:28.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.210+0000 7f5156400640 1 -- 172.21.15.45:0/3805560185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5130003a00 msgr2=0x7f5130003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:28.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.210+0000 7f5156400640 1 --2- 172.21.15.45:0/3805560185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5130003a00 0x7f5130003e40 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f514000c110 tx=0x7f5130005830 comp rx=0 tx=0).stop 2024-04-16T16:17:28.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.210+0000 7f5156400640 1 -- 172.21.15.45:0/3805560185 shutdown_connections 2024-04-16T16:17:28.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.210+0000 7f5156400640 1 --2- 172.21.15.45:0/3805560185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5130003a00 0x7f5130003e40 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.210+0000 7f5156400640 1 -- 172.21.15.45:0/3805560185 >> 172.21.15.45:0/3805560185 conn(0x7f51500b4350 msgr2=0x7f51500b4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:28.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.210+0000 7f5156400640 1 -- 172.21.15.45:0/3805560185 shutdown_connections 2024-04-16T16:17:28.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.210+0000 7f5156400640 1 -- 172.21.15.45:0/3805560185 wait complete. 2024-04-16T16:17:28.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.212+0000 7f5156400640 1 Processor -- start 2024-04-16T16:17:28.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.212+0000 7f5156400640 1 -- start start 2024-04-16T16:17:28.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.213+0000 7f5156400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5130003a00 0x7f51500cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.213+0000 7f5156400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5150151080 0x7f51500cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.213+0000 7f5156400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51500b6560 0x7f51500b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.213+0000 7f5156400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f51500cd1d0 con 0x7f51500b6560 2024-04-16T16:17:28.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.213+0000 7f5156400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f51500cdc00 con 0x7f5130003a00 2024-04-16T16:17:28.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.213+0000 7f5156400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f51500cd350 con 0x7f5150151080 2024-04-16T16:17:28.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.214+0000 7f514fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51500b6560 0x7f51500b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.214+0000 7f514fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51500b6560 0x7f51500b69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41560/0 (socket says 172.21.15.45:41560) 2024-04-16T16:17:28.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.214+0000 7f514fe00640 1 -- 172.21.15.45:0/1801358589 learned_addr learned my addr 172.21.15.45:0/1801358589 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:28.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.214+0000 7f514f400640 1 --2- 172.21.15.45:0/1801358589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5130003a00 0x7f51500cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.214+0000 7f514ea00640 1 --2- 172.21.15.45:0/1801358589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5150151080 0x7f51500cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.214+0000 7f514fe00640 1 -- 172.21.15.45:0/1801358589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5150151080 msgr2=0x7f51500cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:28.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.214+0000 7f514fe00640 1 --2- 172.21.15.45:0/1801358589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5150151080 0x7f51500cefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.214+0000 7f514fe00640 1 -- 172.21.15.45:0/1801358589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5130003a00 msgr2=0x7f51500cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:28.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.214+0000 7f514fe00640 1 --2- 172.21.15.45:0/1801358589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5130003a00 0x7f51500cdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.214+0000 7f514fe00640 1 -- 172.21.15.45:0/1801358589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51501f7ed0 con 0x7f51500b6560 2024-04-16T16:17:28.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.214+0000 7f514fe00640 1 --2- 172.21.15.45:0/1801358589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51500b6560 0x7f51500b69e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f51500cce60 tx=0x7f5148010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:28.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.215+0000 7f5147400640 1 -- 172.21.15.45:0/1801358589 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5148018070 con 0x7f51500b6560 2024-04-16T16:17:28.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.215+0000 7f5147400640 1 -- 172.21.15.45:0/1801358589 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5148004230 con 0x7f51500b6560 2024-04-16T16:17:28.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.215+0000 7f5147400640 1 -- 172.21.15.45:0/1801358589 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514800f040 con 0x7f51500b6560 2024-04-16T16:17:28.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.215+0000 7f5156400640 1 -- 172.21.15.45:0/1801358589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51501f81b0 con 0x7f51500b6560 2024-04-16T16:17:28.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.215+0000 7f5156400640 1 -- 172.21.15.45:0/1801358589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51501f8770 con 0x7f51500b6560 2024-04-16T16:17:28.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.216+0000 7f5156400640 1 -- 172.21.15.45:0/1801358589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5150075780 con 0x7f51500b6560 2024-04-16T16:17:28.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.217+0000 7f5147400640 1 -- 172.21.15.45:0/1801358589 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f514801b070 con 0x7f51500b6560 2024-04-16T16:17:28.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.218+0000 7f5147400640 1 --2- 172.21.15.45:0/1801358589 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f512c06ca60 0x7f512c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.218+0000 7f5147400640 1 -- 172.21.15.45:0/1801358589 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f514808bc00 con 0x7f51500b6560 2024-04-16T16:17:28.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.219+0000 7f514f400640 1 --2- 172.21.15.45:0/1801358589 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f512c06ca60 0x7f512c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.220+0000 7f514f400640 1 --2- 172.21.15.45:0/1801358589 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f512c06ca60 0x7f512c06ef20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5140034000 tx=0x7f5140009030 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:28.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.223+0000 7f5147400640 1 -- 172.21.15.45:0/1801358589 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5148092080 con 0x7f51500b6560 2024-04-16T16:17:28.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.470+0000 7f5156400640 1 -- 172.21.15.45:0/1801358589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51500d11b0 con 0x7f51500b6560 2024-04-16T16:17:28.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.471+0000 7f5147400640 1 -- 172.21.15.45:0/1801358589 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5201 (secure 0 0 0) 0x7f5148059e00 con 0x7f51500b6560 2024-04-16T16:17:28.470 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:28.471 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":54,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:17:23.689033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6402,"mds_1":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"c","rank":1,"incarnation":52,"state":"up:active","state_seq":11,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402,6399]},"id":5}]} 2024-04-16T16:17:28.471 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 54 2024-04-16T16:17:28.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.474+0000 7f5156400640 1 -- 172.21.15.45:0/1801358589 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f512c06ca60 msgr2=0x7f512c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:28.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.474+0000 7f5156400640 1 --2- 172.21.15.45:0/1801358589 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f512c06ca60 0x7f512c06ef20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5140034000 tx=0x7f5140009030 comp rx=0 tx=0).stop 2024-04-16T16:17:28.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.474+0000 7f5156400640 1 -- 172.21.15.45:0/1801358589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51500b6560 msgr2=0x7f51500b69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:28.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.474+0000 7f5156400640 1 --2- 172.21.15.45:0/1801358589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51500b6560 0x7f51500b69e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f51500cce60 tx=0x7f5148010c20 comp rx=0 tx=0).stop 2024-04-16T16:17:28.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.474+0000 7f5156400640 1 -- 172.21.15.45:0/1801358589 shutdown_connections 2024-04-16T16:17:28.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.474+0000 7f5156400640 1 --2- 172.21.15.45:0/1801358589 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f512c06ca60 0x7f512c06ef20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.474+0000 7f5156400640 1 --2- 172.21.15.45:0/1801358589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51500b6560 0x7f51500b69e0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.474+0000 7f5156400640 1 --2- 172.21.15.45:0/1801358589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5150151080 0x7f51500cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.474+0000 7f5156400640 1 --2- 172.21.15.45:0/1801358589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5130003a00 0x7f51500cdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.474+0000 7f5156400640 1 -- 172.21.15.45:0/1801358589 >> 172.21.15.45:0/1801358589 conn(0x7f51500b4350 msgr2=0x7f515015e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:28.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.474+0000 7f5156400640 1 -- 172.21.15.45:0/1801358589 shutdown_connections 2024-04-16T16:17:28.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.475+0000 7f5156400640 1 -- 172.21.15.45:0/1801358589 wait complete. 2024-04-16T16:17:28.495 DEBUG:tasks.cephfs.test_mds_metrics:metrics={'version': 2, 'global_counters': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'counters': [], 'client_metadata': {'cephfs': {'client.5243': {'hostname': 'smithi045', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '172.21.15.45'}, 'client.5285': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'IP': '172.21.15.103'}, 'client.5288': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'IP': '172.21.15.103'}, 'client.6591': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '172.21.15.103'}}}, 'global_metrics': {'cephfs': {'client.5243': [[419, 22], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], 'client.5285': [[2, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], 'client.5288': [[2, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], 'client.6591': [[52, 2], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, 'metrics': {'delayed_ranks': [], 'mds.0': {'client.5243': [], 'client.5285': [], 'client.5288': [], 'client.6591': []}, 'mds.1': {'client.5243': []}}} 2024-04-16T16:17:28.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:17:28.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.721+0000 7f400e200640 1 Processor -- start 2024-04-16T16:17:28.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.722+0000 7f400e200640 1 -- start start 2024-04-16T16:17:28.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.723+0000 7f400e200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f400815ea70 0x7f40080cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.723+0000 7f400e200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40080cf6d0 con 0x7f4008072a90 2024-04-16T16:17:28.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.723+0000 7f400e200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40081466c0 con 0x7f40081462e0 2024-04-16T16:17:28.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.723+0000 7f400e200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f40080cf3f0 con 0x7f400815ea70 2024-04-16T16:17:28.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.724+0000 7f4006a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4008072a90 0x7f40080ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47360/0 (socket says 172.21.15.45:47360) 2024-04-16T16:17:28.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.724+0000 7f4006a00640 1 -- 172.21.15.45:0/1895728891 learned_addr learned my addr 172.21.15.45:0/1895728891 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:28.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.724+0000 7f4007e00640 1 --2- 172.21.15.45:0/1895728891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f400815ea70 0x7f40080cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.724+0000 7f4006000640 1 -- 172.21.15.45:0/1895728891 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1483222200 0 0) 0x7f40080cf6d0 con 0x7f4008072a90 2024-04-16T16:17:28.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.724+0000 7f4006000640 1 -- 172.21.15.45:0/1895728891 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fec003520 con 0x7f4008072a90 2024-04-16T16:17:28.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.725+0000 7f4006000640 1 -- 172.21.15.45:0/1895728891 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2057759488 0 0) 0x7f3fec003520 con 0x7f4008072a90 2024-04-16T16:17:28.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.725+0000 7f4006000640 1 -- 172.21.15.45:0/1895728891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f400815ea70 msgr2=0x7f40080cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:28.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.725+0000 7f4006000640 1 --2- 172.21.15.45:0/1895728891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f400815ea70 0x7f40080cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.725+0000 7f4006000640 1 -- 172.21.15.45:0/1895728891 >> v1:172.21.15.103:6789/0 conn(0x7f40081462e0 legacy=0x7f4008072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:28.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.725+0000 7f4006000640 1 -- 172.21.15.45:0/1895728891 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400815cf80 con 0x7f4008072a90 2024-04-16T16:17:28.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.725+0000 7f4006000640 1 -- 172.21.15.45:0/1895728891 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f3ff8001670 con 0x7f4008072a90 2024-04-16T16:17:28.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.725+0000 7f4006000640 1 -- 172.21.15.45:0/1895728891 >> v1:172.21.15.45:6789/0 conn(0x7f4008072a90 legacy=0x7f40080ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:28.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.725+0000 7f4006000640 1 --2- 172.21.15.45:0/1895728891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fec0039e0 0x7f3fec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.725+0000 7f4006000640 1 --2- 172.21.15.45:0/1895728891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f400815ea70 0x7f3fec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.725+0000 7f4006000640 1 --2- 172.21.15.45:0/1895728891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fec004d40 0x7f3fec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.725+0000 7f4006000640 1 -- 172.21.15.45:0/1895728891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f40080cf3f0 con 0x7f3fec0039e0 2024-04-16T16:17:28.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.725+0000 7f4006000640 1 -- 172.21.15.45:0/1895728891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec00bba0 con 0x7f3fec004d40 2024-04-16T16:17:28.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.725+0000 7f4006000640 1 -- 172.21.15.45:0/1895728891 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3fec00bda0 con 0x7f400815ea70 2024-04-16T16:17:28.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.725+0000 7f4007400640 1 --2- 172.21.15.45:0/1895728891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fec0039e0 0x7f3fec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.726+0000 7f4006a00640 1 --2- 172.21.15.45:0/1895728891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f400815ea70 0x7f3fec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.726+0000 7f4007e00640 1 --2- 172.21.15.45:0/1895728891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fec004d40 0x7f3fec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.726+0000 7f4007400640 1 -- 172.21.15.45:0/1895728891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f400815ea70 msgr2=0x7f3fec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:28.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.726+0000 7f4007400640 1 --2- 172.21.15.45:0/1895728891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f400815ea70 0x7f3fec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.726+0000 7f4007400640 1 -- 172.21.15.45:0/1895728891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fec004d40 msgr2=0x7f3fec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:28.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.726+0000 7f4007400640 1 --2- 172.21.15.45:0/1895728891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fec004d40 0x7f3fec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.726+0000 7f4007400640 1 -- 172.21.15.45:0/1895728891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fec003520 con 0x7f3fec0039e0 2024-04-16T16:17:28.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.726+0000 7f4007e00640 1 --2- 172.21.15.45:0/1895728891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fec004d40 0x7f3fec0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:28.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.727+0000 7f4007400640 1 --2- 172.21.15.45:0/1895728891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fec0039e0 0x7f3fec003e00 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0001750 tx=0x7f3fec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1046acd4dedf46eb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:28.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.727+0000 7f4006000640 1 -- 172.21.15.45:0/1895728891 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0001370 con 0x7f3fec0039e0 2024-04-16T16:17:28.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.727+0000 7f4006000640 1 -- 172.21.15.45:0/1895728891 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff00052d0 con 0x7f3fec0039e0 2024-04-16T16:17:28.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.727+0000 7f4006000640 1 -- 172.21.15.45:0/1895728891 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0003830 con 0x7f3fec0039e0 2024-04-16T16:17:28.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.728+0000 7f400e200640 1 -- 172.21.15.45:0/1895728891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fec0039e0 msgr2=0x7f3fec003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:28.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.728+0000 7f400e200640 1 --2- 172.21.15.45:0/1895728891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fec0039e0 0x7f3fec003e00 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0001750 tx=0x7f3fec005830 comp rx=0 tx=0).stop 2024-04-16T16:17:28.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.728+0000 7f4007e00640 1 -- 172.21.15.45:0/1895728891 reap_dead start 2024-04-16T16:17:28.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.728+0000 7f400e200640 1 -- 172.21.15.45:0/1895728891 shutdown_connections 2024-04-16T16:17:28.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.728+0000 7f400e200640 1 -- 172.21.15.45:0/1895728891 >> 172.21.15.45:0/1895728891 conn(0x7f40080b2750 msgr2=0x7f40080b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:28.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.728+0000 7f400e200640 1 -- 172.21.15.45:0/1895728891 shutdown_connections 2024-04-16T16:17:28.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.728+0000 7f400e200640 1 -- 172.21.15.45:0/1895728891 wait complete. 2024-04-16T16:17:28.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.730+0000 7f400e200640 1 Processor -- start 2024-04-16T16:17:28.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.730+0000 7f400e200640 1 -- start start 2024-04-16T16:17:28.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.731+0000 7f400e200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fec0039e0 0x7f400815c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.731+0000 7f400e200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40081f0390 0x7f40080af160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.731+0000 7f400e200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40081f07f0 0x7f400806e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.731+0000 7f400e200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f40080cedd0 con 0x7f40081f07f0 2024-04-16T16:17:28.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.731+0000 7f400e200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f400815d540 con 0x7f40081f0390 2024-04-16T16:17:28.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.731+0000 7f400e200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f40080cef50 con 0x7f3fec0039e0 2024-04-16T16:17:28.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.731+0000 7f4007e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40081f07f0 0x7f400806e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.732+0000 7f4007e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40081f07f0 0x7f400806e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41588/0 (socket says 172.21.15.45:41588) 2024-04-16T16:17:28.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.732+0000 7f4007e00640 1 -- 172.21.15.45:0/4224696013 learned_addr learned my addr 172.21.15.45:0/4224696013 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:28.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.732+0000 7f4007400640 1 --2- 172.21.15.45:0/4224696013 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fec0039e0 0x7f400815c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.732+0000 7f4006a00640 1 --2- 172.21.15.45:0/4224696013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40081f0390 0x7f40080af160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.732+0000 7f4007e00640 1 -- 172.21.15.45:0/4224696013 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fec0039e0 msgr2=0x7f400815c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:28.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.732+0000 7f4007e00640 1 --2- 172.21.15.45:0/4224696013 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fec0039e0 0x7f400815c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.732+0000 7f4007e00640 1 -- 172.21.15.45:0/4224696013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40081f0390 msgr2=0x7f40080af160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:28.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.732+0000 7f4007e00640 1 --2- 172.21.15.45:0/4224696013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40081f0390 0x7f40080af160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.732+0000 7f4007e00640 1 -- 172.21.15.45:0/4224696013 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4008140500 con 0x7f40081f07f0 2024-04-16T16:17:28.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.732+0000 7f4007e00640 1 --2- 172.21.15.45:0/4224696013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40081f07f0 0x7f400806e760 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f40080730d0 tx=0x7f4000009d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:28.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.733+0000 7f3fff400640 1 -- 172.21.15.45:0/4224696013 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4000019070 con 0x7f40081f07f0 2024-04-16T16:17:28.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.733+0000 7f3fff400640 1 -- 172.21.15.45:0/4224696013 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4000007ce0 con 0x7f40081f07f0 2024-04-16T16:17:28.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.733+0000 7f3fff400640 1 -- 172.21.15.45:0/4224696013 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4000014640 con 0x7f40081f07f0 2024-04-16T16:17:28.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.733+0000 7f400e200640 1 -- 172.21.15.45:0/4224696013 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40080aab90 con 0x7f40081f07f0 2024-04-16T16:17:28.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.733+0000 7f400e200640 1 -- 172.21.15.45:0/4224696013 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40080ab3d0 con 0x7f40081f07f0 2024-04-16T16:17:28.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.734+0000 7f400e200640 1 -- 172.21.15.45:0/4224696013 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4008075780 con 0x7f40081f07f0 2024-04-16T16:17:28.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.735+0000 7f3fff400640 1 -- 172.21.15.45:0/4224696013 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f4000005d10 con 0x7f40081f07f0 2024-04-16T16:17:28.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.736+0000 7f3fff400640 1 --2- 172.21.15.45:0/4224696013 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f3fe406ca60 0x7f3fe406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:28.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.736+0000 7f3fff400640 1 -- 172.21.15.45:0/4224696013 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f400008bab0 con 0x7f40081f07f0 2024-04-16T16:17:28.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.737+0000 7f4007400640 1 --2- 172.21.15.45:0/4224696013 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f3fe406ca60 0x7f3fe406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:28.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.738+0000 7f4007400640 1 --2- 172.21.15.45:0/4224696013 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f3fe406ca60 0x7f3fe406ef20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0033000 tx=0x7f3ff0009030 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:28.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.741+0000 7f3fff400640 1 -- 172.21.15.45:0/4224696013 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f40080aab90 con 0x7f40081f07f0 2024-04-16T16:17:28.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.986+0000 7f400e200640 1 -- 172.21.15.45:0/4224696013 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4008142b40 con 0x7f40081f07f0 2024-04-16T16:17:28.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.988+0000 7f3fff400640 1 -- 172.21.15.45:0/4224696013 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5201 (secure 0 0 0) 0x7f4000059b90 con 0x7f40081f07f0 2024-04-16T16:17:28.987 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:28.987 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":54,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:17:23.689033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6402,"mds_1":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"c","rank":1,"incarnation":52,"state":"up:active","state_seq":11,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402,6399]},"id":5}]} 2024-04-16T16:17:28.987 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 54 2024-04-16T16:17:28.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.990+0000 7f400e200640 1 -- 172.21.15.45:0/4224696013 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f3fe406ca60 msgr2=0x7f3fe406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:28.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.990+0000 7f400e200640 1 --2- 172.21.15.45:0/4224696013 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f3fe406ca60 0x7f3fe406ef20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0033000 tx=0x7f3ff0009030 comp rx=0 tx=0).stop 2024-04-16T16:17:28.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.990+0000 7f400e200640 1 -- 172.21.15.45:0/4224696013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40081f07f0 msgr2=0x7f400806e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:28.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.990+0000 7f400e200640 1 --2- 172.21.15.45:0/4224696013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40081f07f0 0x7f400806e760 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f40080730d0 tx=0x7f4000009d50 comp rx=0 tx=0).stop 2024-04-16T16:17:28.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.990+0000 7f400e200640 1 -- 172.21.15.45:0/4224696013 shutdown_connections 2024-04-16T16:17:28.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.990+0000 7f400e200640 1 --2- 172.21.15.45:0/4224696013 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f3fe406ca60 0x7f3fe406ef20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.990+0000 7f400e200640 1 --2- 172.21.15.45:0/4224696013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40081f07f0 0x7f400806e760 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.990+0000 7f400e200640 1 --2- 172.21.15.45:0/4224696013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40081f0390 0x7f40080af160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.990+0000 7f400e200640 1 --2- 172.21.15.45:0/4224696013 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3fec0039e0 0x7f400815c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:28.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.990+0000 7f400e200640 1 -- 172.21.15.45:0/4224696013 >> 172.21.15.45:0/4224696013 conn(0x7f40080b2750 msgr2=0x7f4008145610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:28.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.990+0000 7f400e200640 1 -- 172.21.15.45:0/4224696013 shutdown_connections 2024-04-16T16:17:28.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:28.991+0000 7f400e200640 1 -- 172.21.15.45:0/4224696013 wait complete. 2024-04-16T16:17:29.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:17:29.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.233+0000 7fc8d7a00640 1 Processor -- start 2024-04-16T16:17:29.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.234+0000 7fc8d7a00640 1 -- start start 2024-04-16T16:17:29.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.235+0000 7fc8d7a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d0147840 0x7fc8d015fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:29.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.235+0000 7fc8d7a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc8d00aed90 0x7fc8d0072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:29.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.235+0000 7fc8d7a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc8d00728c0 0x7fc8d0162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:29.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.235+0000 7fc8d7a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc8d00afe80 con 0x7fc8d00728c0 2024-04-16T16:17:29.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.235+0000 7fc8d7a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc8d0147c20 con 0x7fc8d00aed90 2024-04-16T16:17:29.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.235+0000 7fc8d7a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d00af270 con 0x7fc8d0147840 2024-04-16T16:17:29.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.236+0000 7fc8d5a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc8d00728c0 0x7fc8d0162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:29.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.236+0000 7fc8d5a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc8d00728c0 0x7fc8d0162610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41594/0 (socket says 172.21.15.45:41594) 2024-04-16T16:17:29.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.236+0000 7fc8d5a00640 1 -- 172.21.15.45:0/1475924708 learned_addr learned my addr 172.21.15.45:0/1475924708 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:29.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.236+0000 7fc8d5000640 1 --2- 172.21.15.45:0/1475924708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d0147840 0x7fc8d015fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:29.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.236+0000 7fc8cfe00640 1 --2- 172.21.15.45:0/1475924708 >> v2:172.21.15.103:3300/0 conn(0x7fc8d00aed90 0x7fc8d0072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:29.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.236+0000 7fc8d5a00640 1 -- 172.21.15.45:0/1475924708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d0147840 msgr2=0x7fc8d015fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:29.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.236+0000 7fc8d5a00640 1 --2- 172.21.15.45:0/1475924708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d0147840 0x7fc8d015fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.236+0000 7fc8d5a00640 1 -- 172.21.15.45:0/1475924708 >> v2:172.21.15.103:3300/0 conn(0x7fc8d00aed90 msgr2=0x7fc8d0072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:29.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.236+0000 7fc8d5a00640 1 --2- 172.21.15.45:0/1475924708 >> v2:172.21.15.103:3300/0 conn(0x7fc8d00aed90 0x7fc8d0072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.236+0000 7fc8d5a00640 1 -- 172.21.15.45:0/1475924708 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d00af550 con 0x7fc8d00728c0 2024-04-16T16:17:29.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.236+0000 7fc8cfe00640 1 --2- 172.21.15.45:0/1475924708 >> v2:172.21.15.103:3300/0 conn(0x7fc8d00aed90 0x7fc8d0072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:29.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.238+0000 7fc8d5a00640 1 --2- 172.21.15.45:0/1475924708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d00728c0 0x7fc8d0162610 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc8d00b3600 tx=0x7fc8c802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd9645a93318af28 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:29.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.238+0000 7fc8cf400640 1 -- 172.21.15.45:0/1475924708 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c802ca70 con 0x7fc8d00728c0 2024-04-16T16:17:29.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.238+0000 7fc8cf400640 1 -- 172.21.15.45:0/1475924708 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c802cc10 con 0x7fc8d00728c0 2024-04-16T16:17:29.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.238+0000 7fc8cf400640 1 -- 172.21.15.45:0/1475924708 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c80343f0 con 0x7fc8d00728c0 2024-04-16T16:17:29.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.239+0000 7fc8d7a00640 1 -- 172.21.15.45:0/1475924708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d00728c0 msgr2=0x7fc8d0162610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:29.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.239+0000 7fc8d7a00640 1 --2- 172.21.15.45:0/1475924708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d00728c0 0x7fc8d0162610 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc8d00b3600 tx=0x7fc8c802bed0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.239+0000 7fc8d7a00640 1 -- 172.21.15.45:0/1475924708 shutdown_connections 2024-04-16T16:17:29.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.239+0000 7fc8d7a00640 1 --2- 172.21.15.45:0/1475924708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d00728c0 0x7fc8d0162610 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.239+0000 7fc8d7a00640 1 --2- 172.21.15.45:0/1475924708 >> v2:172.21.15.103:3300/0 conn(0x7fc8d00aed90 0x7fc8d0072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.239+0000 7fc8d7a00640 1 --2- 172.21.15.45:0/1475924708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d0147840 0x7fc8d015fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.239+0000 7fc8d7a00640 1 -- 172.21.15.45:0/1475924708 >> 172.21.15.45:0/1475924708 conn(0x7fc8d00b2750 msgr2=0x7fc8d00b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:29.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.239+0000 7fc8d7a00640 1 -- 172.21.15.45:0/1475924708 shutdown_connections 2024-04-16T16:17:29.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.239+0000 7fc8d7a00640 1 -- 172.21.15.45:0/1475924708 wait complete. 2024-04-16T16:17:29.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.241+0000 7fc8d7a00640 1 Processor -- start 2024-04-16T16:17:29.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.241+0000 7fc8d7a00640 1 -- start start 2024-04-16T16:17:29.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.242+0000 7fc8d7a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d00728c0 0x7fc8d0162e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:29.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.242+0000 7fc8d7a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8d00aed90 0x7fc8d0134c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:29.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.242+0000 7fc8d7a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d0147840 0x7fc8d0144960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:29.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.242+0000 7fc8d7a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0160040 con 0x7fc8d00728c0 2024-04-16T16:17:29.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.242+0000 7fc8d7a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d00afb10 con 0x7fc8d00aed90 2024-04-16T16:17:29.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.242+0000 7fc8d7a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d01601c0 con 0x7fc8d0147840 2024-04-16T16:17:29.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.242+0000 7fc8d5000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d00728c0 0x7fc8d0162e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:29.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.242+0000 7fc8d5000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d00728c0 0x7fc8d0162e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41604/0 (socket says 172.21.15.45:41604) 2024-04-16T16:17:29.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.242+0000 7fc8d5000640 1 -- 172.21.15.45:0/2203180693 learned_addr learned my addr 172.21.15.45:0/2203180693 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.243+0000 7fc8d5a00640 1 --2- 172.21.15.45:0/2203180693 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d0147840 0x7fc8d0144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.243+0000 7fc8cfe00640 1 --2- 172.21.15.45:0/2203180693 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8d00aed90 0x7fc8d0134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.243+0000 7fc8d5000640 1 -- 172.21.15.45:0/2203180693 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d0147840 msgr2=0x7fc8d0144960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.243+0000 7fc8d5000640 1 --2- 172.21.15.45:0/2203180693 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d0147840 0x7fc8d0144960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.243+0000 7fc8d5000640 1 -- 172.21.15.45:0/2203180693 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8d00aed90 msgr2=0x7fc8d0134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.243+0000 7fc8d5000640 1 --2- 172.21.15.45:0/2203180693 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8d00aed90 0x7fc8d0134c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.243+0000 7fc8d5000640 1 -- 172.21.15.45:0/2203180693 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d01438c0 con 0x7fc8d00728c0 2024-04-16T16:17:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.243+0000 7fc8cfe00640 1 --2- 172.21.15.45:0/2203180693 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8d00aed90 0x7fc8d0134c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.243+0000 7fc8d5000640 1 --2- 172.21.15.45:0/2203180693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d00728c0 0x7fc8d0162e10 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000d800 tx=0x7fc8c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.243+0000 7fc8ccc00640 1 -- 172.21.15.45:0/2203180693 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c000b6c0 con 0x7fc8d00728c0 2024-04-16T16:17:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.244+0000 7fc8ccc00640 1 -- 172.21.15.45:0/2203180693 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c0019070 con 0x7fc8d00728c0 2024-04-16T16:17:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.244+0000 7fc8ccc00640 1 -- 172.21.15.45:0/2203180693 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c000ba80 con 0x7fc8d00728c0 2024-04-16T16:17:29.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.244+0000 7fc8d7a00640 1 -- 172.21.15.45:0/2203180693 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d0143ba0 con 0x7fc8d00728c0 2024-04-16T16:17:29.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.244+0000 7fc8d7a00640 1 -- 172.21.15.45:0/2203180693 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d0144160 con 0x7fc8d00728c0 2024-04-16T16:17:29.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.245+0000 7fc8d7a00640 1 -- 172.21.15.45:0/2203180693 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d0075780 con 0x7fc8d00728c0 2024-04-16T16:17:29.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.246+0000 7fc8ccc00640 1 -- 172.21.15.45:0/2203180693 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7fc8c000bc20 con 0x7fc8d00728c0 2024-04-16T16:17:29.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.247+0000 7fc8ccc00640 1 --2- 172.21.15.45:0/2203180693 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fc8b006ca60 0x7fc8b006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:29.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.247+0000 7fc8ccc00640 1 -- 172.21.15.45:0/2203180693 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7fc8c008c350 con 0x7fc8d00728c0 2024-04-16T16:17:29.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.248+0000 7fc8cfe00640 1 --2- 172.21.15.45:0/2203180693 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fc8b006ca60 0x7fc8b006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:29.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.249+0000 7fc8cfe00640 1 --2- 172.21.15.45:0/2203180693 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fc8b006ca60 0x7fc8b006ef20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc006fd0 tx=0x7fc8bc008040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:29.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.251+0000 7fc8ccc00640 1 -- 172.21.15.45:0/2203180693 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc8c005a4c0 con 0x7fc8d00728c0 2024-04-16T16:17:29.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.497+0000 7fc8d7a00640 1 -- 172.21.15.45:0/2203180693 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8d0070220 con 0x7fc8d00728c0 2024-04-16T16:17:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.499+0000 7fc8ccc00640 1 -- 172.21.15.45:0/2203180693 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5201 (secure 0 0 0) 0x7fc8c0059dc0 con 0x7fc8d00728c0 2024-04-16T16:17:29.498 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:29.499 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":54,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:17:23.689033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6402,"mds_1":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"c","rank":1,"incarnation":52,"state":"up:active","state_seq":11,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402,6399]},"id":5}]} 2024-04-16T16:17:29.499 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 54 2024-04-16T16:17:29.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.501+0000 7fc8d7a00640 1 -- 172.21.15.45:0/2203180693 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fc8b006ca60 msgr2=0x7fc8b006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:29.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.501+0000 7fc8d7a00640 1 --2- 172.21.15.45:0/2203180693 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fc8b006ca60 0x7fc8b006ef20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc006fd0 tx=0x7fc8bc008040 comp rx=0 tx=0).stop 2024-04-16T16:17:29.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.501+0000 7fc8d7a00640 1 -- 172.21.15.45:0/2203180693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d00728c0 msgr2=0x7fc8d0162e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:29.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.501+0000 7fc8d7a00640 1 --2- 172.21.15.45:0/2203180693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d00728c0 0x7fc8d0162e10 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000d800 tx=0x7fc8c000dcc0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.502+0000 7fc8d7a00640 1 -- 172.21.15.45:0/2203180693 shutdown_connections 2024-04-16T16:17:29.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.502+0000 7fc8d7a00640 1 --2- 172.21.15.45:0/2203180693 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fc8b006ca60 0x7fc8b006ef20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.502+0000 7fc8d7a00640 1 --2- 172.21.15.45:0/2203180693 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d0147840 0x7fc8d0144960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.502+0000 7fc8d7a00640 1 --2- 172.21.15.45:0/2203180693 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8d00aed90 0x7fc8d0134c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.502+0000 7fc8d7a00640 1 --2- 172.21.15.45:0/2203180693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8d00728c0 0x7fc8d0162e10 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.502+0000 7fc8d7a00640 1 -- 172.21.15.45:0/2203180693 >> 172.21.15.45:0/2203180693 conn(0x7fc8d00b2750 msgr2=0x7fc8d00b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:29.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.502+0000 7fc8d7a00640 1 -- 172.21.15.45:0/2203180693 shutdown_connections 2024-04-16T16:17:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.502+0000 7fc8d7a00640 1 -- 172.21.15.45:0/2203180693 wait complete. 2024-04-16T16:17:29.522 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:17:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.742+0000 7fece3200640 1 Processor -- start 2024-04-16T16:17:29.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.742+0000 7fece3200640 1 -- start start 2024-04-16T16:17:29.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.744+0000 7fece3200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fece406dfb0 0x7fece40c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:29.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.744+0000 7fece3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fece40c5bb0 0x7fece415d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:29.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.744+0000 7fece3200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fece415c700 con 0x7fece4144ff0 2024-04-16T16:17:29.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.744+0000 7fece3200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fece415d2f0 con 0x7fece406dfb0 2024-04-16T16:17:29.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.744+0000 7fece3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fece415c9c0 con 0x7fece40c5bb0 2024-04-16T16:17:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.745+0000 7fece2800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fece40c5bb0 0x7fece415d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.745+0000 7fece2800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fece40c5bb0 0x7fece415d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:43920/0 (socket says 172.21.15.45:43920) 2024-04-16T16:17:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.745+0000 7fece2800640 1 -- 172.21.15.45:0/205896579 learned_addr learned my addr 172.21.15.45:0/205896579 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.745+0000 7fecd9400640 1 --2- 172.21.15.45:0/205896579 >> v2:172.21.15.103:3300/0 conn(0x7fece406dfb0 0x7fece40c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.746+0000 7fecd9400640 1 -- 172.21.15.45:0/205896579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fece40c5bb0 msgr2=0x7fece415d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.746+0000 7fecd9400640 1 --2- 172.21.15.45:0/205896579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fece40c5bb0 0x7fece415d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.746+0000 7fecd9400640 1 -- 172.21.15.45:0/205896579 >> v1:172.21.15.103:6789/0 conn(0x7fece4144ff0 legacy=0x7fece40cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:29.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.746+0000 7fecd9400640 1 -- 172.21.15.45:0/205896579 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece415cca0 con 0x7fece406dfb0 2024-04-16T16:17:29.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.748+0000 7fecd9400640 1 --2- 172.21.15.45:0/205896579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fece406dfb0 0x7fece40c5670 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7feccc0097a0 tx=0x7feccc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=359a69f402db36d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:29.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.748+0000 7fece1400640 1 -- 172.21.15.45:0/205896579 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feccc035020 con 0x7fece406dfb0 2024-04-16T16:17:29.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.748+0000 7fece1400640 1 -- 172.21.15.45:0/205896579 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feccc031830 con 0x7fece406dfb0 2024-04-16T16:17:29.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.748+0000 7fece1400640 1 -- 172.21.15.45:0/205896579 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feccc031c00 con 0x7fece406dfb0 2024-04-16T16:17:29.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.749+0000 7fece3200640 1 -- 172.21.15.45:0/205896579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fece406dfb0 msgr2=0x7fece40c5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:29.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.749+0000 7fece3200640 1 --2- 172.21.15.45:0/205896579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fece406dfb0 0x7fece40c5670 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7feccc0097a0 tx=0x7feccc02b8d0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.749+0000 7fece3200640 1 -- 172.21.15.45:0/205896579 shutdown_connections 2024-04-16T16:17:29.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.749+0000 7fece3200640 1 --2- 172.21.15.45:0/205896579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fece40c5bb0 0x7fece415d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.749+0000 7fece3200640 1 --2- 172.21.15.45:0/205896579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fece406dfb0 0x7fece40c5670 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.749+0000 7fece3200640 1 -- 172.21.15.45:0/205896579 >> 172.21.15.45:0/205896579 conn(0x7fece406b930 msgr2=0x7fece406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:29.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.749+0000 7fece3200640 1 -- 172.21.15.45:0/205896579 shutdown_connections 2024-04-16T16:17:29.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.749+0000 7fece3200640 1 -- 172.21.15.45:0/205896579 wait complete. 2024-04-16T16:17:29.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.751+0000 7fece3200640 1 Processor -- start 2024-04-16T16:17:29.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.751+0000 7fece3200640 1 -- start start 2024-04-16T16:17:29.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.752+0000 7fece3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fece406dfb0 0x7fece4135830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:29.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.752+0000 7fece3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fece40c5bb0 0x7fece415af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:29.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.752+0000 7fece3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece4144ff0 0x7fece413b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:29.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.752+0000 7fece3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fece415dcc0 con 0x7fece4144ff0 2024-04-16T16:17:29.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.752+0000 7fece3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fece41453d0 con 0x7fece406dfb0 2024-04-16T16:17:29.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.752+0000 7fece3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fece415de40 con 0x7fece40c5bb0 2024-04-16T16:17:29.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.752+0000 7fece2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece4144ff0 0x7fece413b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:29.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.752+0000 7fece2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece4144ff0 0x7fece413b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41606/0 (socket says 172.21.15.45:41606) 2024-04-16T16:17:29.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.752+0000 7fece2800640 1 -- 172.21.15.45:0/2964640221 learned_addr learned my addr 172.21.15.45:0/2964640221 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:29.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.753+0000 7fecd9400640 1 --2- 172.21.15.45:0/2964640221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fece40c5bb0 0x7fece415af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:29.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.753+0000 7fece1e00640 1 --2- 172.21.15.45:0/2964640221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fece406dfb0 0x7fece4135830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:29.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.753+0000 7fece2800640 1 -- 172.21.15.45:0/2964640221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fece40c5bb0 msgr2=0x7fece415af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:29.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.753+0000 7fece2800640 1 --2- 172.21.15.45:0/2964640221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fece40c5bb0 0x7fece415af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.753+0000 7fece2800640 1 -- 172.21.15.45:0/2964640221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fece406dfb0 msgr2=0x7fece4135830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:29.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.753+0000 7fece2800640 1 --2- 172.21.15.45:0/2964640221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fece406dfb0 0x7fece4135830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:29.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.753+0000 7fece2800640 1 -- 172.21.15.45:0/2964640221 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece4136060 con 0x7fece4144ff0 2024-04-16T16:17:29.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.753+0000 7fece1e00640 1 --2- 172.21.15.45:0/2964640221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fece406dfb0 0x7fece4135830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:29.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.753+0000 7fece2800640 1 --2- 172.21.15.45:0/2964640221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece4144ff0 0x7fece413b760 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fecdc010f30 tx=0x7fecdc00e550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:29.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.754+0000 7fecdaa00640 1 -- 172.21.15.45:0/2964640221 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc00b310 con 0x7fece4144ff0 2024-04-16T16:17:29.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.754+0000 7fecdaa00640 1 -- 172.21.15.45:0/2964640221 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecdc003990 con 0x7fece4144ff0 2024-04-16T16:17:29.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.754+0000 7fecdaa00640 1 -- 172.21.15.45:0/2964640221 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc003d60 con 0x7fece4144ff0 2024-04-16T16:17:29.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.754+0000 7fece3200640 1 -- 172.21.15.45:0/2964640221 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fece4136340 con 0x7fece4144ff0 2024-04-16T16:17:29.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.754+0000 7fece3200640 1 -- 172.21.15.45:0/2964640221 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fece4136900 con 0x7fece4144ff0 2024-04-16T16:17:29.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.755+0000 7fece3200640 1 -- 172.21.15.45:0/2964640221 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fece4075780 con 0x7fece4144ff0 2024-04-16T16:17:29.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.756+0000 7fecdaa00640 1 -- 172.21.15.45:0/2964640221 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7fecdc014050 con 0x7fece4144ff0 2024-04-16T16:17:29.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.757+0000 7fecdaa00640 1 --2- 172.21.15.45:0/2964640221 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fecc006ca60 0x7fecc006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:29.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.757+0000 7fecdaa00640 1 -- 172.21.15.45:0/2964640221 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7fecdc08e800 con 0x7fece4144ff0 2024-04-16T16:17:29.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.758+0000 7fece1e00640 1 --2- 172.21.15.45:0/2964640221 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fecc006ca60 0x7fecc006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:29.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.759+0000 7fece1e00640 1 --2- 172.21.15.45:0/2964640221 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fecc006ca60 0x7fecc006ef20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fece41402a0 tx=0x7fecd4009040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:29.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:29.761+0000 7fecdaa00640 1 -- 172.21.15.45:0/2964640221 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fecdc012040 con 0x7fece4144ff0 2024-04-16T16:17:30.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.010+0000 7fece3200640 1 -- 172.21.15.45:0/2964640221 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fece415f580 con 0x7fece4144ff0 2024-04-16T16:17:30.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.012+0000 7fecdaa00640 1 -- 172.21.15.45:0/2964640221 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5201 (secure 0 0 0) 0x7fecdc05c7c0 con 0x7fece4144ff0 2024-04-16T16:17:30.011 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:30.011 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":54,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:17:23.689033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6402,"mds_1":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"c","rank":1,"incarnation":52,"state":"up:active","state_seq":11,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402,6399]},"id":5}]} 2024-04-16T16:17:30.011 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 54 2024-04-16T16:17:30.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.015+0000 7fece3200640 1 -- 172.21.15.45:0/2964640221 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fecc006ca60 msgr2=0x7fecc006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:30.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.015+0000 7fece3200640 1 --2- 172.21.15.45:0/2964640221 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fecc006ca60 0x7fecc006ef20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fece41402a0 tx=0x7fecd4009040 comp rx=0 tx=0).stop 2024-04-16T16:17:30.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.015+0000 7fece3200640 1 -- 172.21.15.45:0/2964640221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece4144ff0 msgr2=0x7fece413b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:30.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.015+0000 7fece3200640 1 --2- 172.21.15.45:0/2964640221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece4144ff0 0x7fece413b760 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fecdc010f30 tx=0x7fecdc00e550 comp rx=0 tx=0).stop 2024-04-16T16:17:30.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.015+0000 7fece3200640 1 -- 172.21.15.45:0/2964640221 shutdown_connections 2024-04-16T16:17:30.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.015+0000 7fece3200640 1 --2- 172.21.15.45:0/2964640221 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fecc006ca60 0x7fecc006ef20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:30.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.015+0000 7fece3200640 1 --2- 172.21.15.45:0/2964640221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece4144ff0 0x7fece413b760 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:30.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.015+0000 7fece3200640 1 --2- 172.21.15.45:0/2964640221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fece40c5bb0 0x7fece415af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:30.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.015+0000 7fece3200640 1 --2- 172.21.15.45:0/2964640221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fece406dfb0 0x7fece4135830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:30.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.015+0000 7fece3200640 1 -- 172.21.15.45:0/2964640221 >> 172.21.15.45:0/2964640221 conn(0x7fece406b930 msgr2=0x7fece40c64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:30.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.015+0000 7fece3200640 1 -- 172.21.15.45:0/2964640221 shutdown_connections 2024-04-16T16:17:30.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.015+0000 7fece3200640 1 -- 172.21.15.45:0/2964640221 wait complete. 2024-04-16T16:17:30.036 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A INPUT -p tcp --dport 6833 -j REJECT -m comment --comment teuthology 2024-04-16T16:17:30.174 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A INPUT -p tcp --dport 6835 -j REJECT -m comment --comment teuthology 2024-04-16T16:17:30.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A OUTPUT -p tcp --sport 6832 -j REJECT -m comment --comment teuthology 2024-04-16T16:17:30.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A INPUT -p tcp --dport 6832 -j REJECT -m comment --comment teuthology 2024-04-16T16:17:30.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A OUTPUT -p tcp --sport 6834 -j REJECT -m comment --comment teuthology 2024-04-16T16:17:30.590 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A INPUT -p tcp --dport 6834 -j REJECT -m comment --comment teuthology 2024-04-16T16:17:30.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:30.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.925+0000 7f1ceca00640 1 Processor -- start 2024-04-16T16:17:30.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.926+0000 7f1ceca00640 1 -- start start 2024-04-16T16:17:30.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.927+0000 7f1ceca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1ce815e830 0x7f1ce80ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:30.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.927+0000 7f1ceca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1ce80cb7d0 0x7f1ce80cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:30.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.927+0000 7f1ceca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ce815f1b0 con 0x7f1ce80cbc80 2024-04-16T16:17:30.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.927+0000 7f1ceca00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1ce80cf280 con 0x7f1ce80cb7d0 2024-04-16T16:17:30.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.927+0000 7f1ceca00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1ce80ce190 con 0x7f1ce815e830 2024-04-16T16:17:30.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.928+0000 7f1ce6a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1ce80cb7d0 0x7f1ce80cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:30.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.928+0000 7f1ce6a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1ce80cb7d0 0x7f1ce80cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41610/0 (socket says 172.21.15.45:41610) 2024-04-16T16:17:30.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.928+0000 7f1ce6a00640 1 -- 172.21.15.45:0/2693468095 learned_addr learned my addr 172.21.15.45:0/2693468095 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:30.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.928+0000 7f1ce6a00640 1 -- 172.21.15.45:0/2693468095 >> v2:172.21.15.103:3300/0 conn(0x7f1ce815e830 msgr2=0x7f1ce80ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:30.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.928+0000 7f1ce6a00640 1 --2- 172.21.15.45:0/2693468095 >> v2:172.21.15.103:3300/0 conn(0x7f1ce815e830 0x7f1ce80ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:30.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.928+0000 7f1ce6a00640 1 -- 172.21.15.45:0/2693468095 >> v1:172.21.15.103:6789/0 conn(0x7f1ce80cbc80 legacy=0x7f1ce80cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:30.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.928+0000 7f1ce6a00640 1 -- 172.21.15.45:0/2693468095 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce80ce470 con 0x7f1ce80cb7d0 2024-04-16T16:17:30.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.929+0000 7f1ce6a00640 1 --2- 172.21.15.45:0/2693468095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce80cb7d0 0x7f1ce80cd620 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8070350 tx=0x7f1cd402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e718064a982db037 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:30.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.930+0000 7f1ce6000640 1 -- 172.21.15.45:0/2693468095 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd402e070 con 0x7f1ce80cb7d0 2024-04-16T16:17:30.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.930+0000 7f1ce6000640 1 -- 172.21.15.45:0/2693468095 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cd4037070 con 0x7f1ce80cb7d0 2024-04-16T16:17:30.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.930+0000 7f1ce6000640 1 -- 172.21.15.45:0/2693468095 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4031a00 con 0x7f1ce80cb7d0 2024-04-16T16:17:30.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.930+0000 7f1ceca00640 1 -- 172.21.15.45:0/2693468095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce80cb7d0 msgr2=0x7f1ce80cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:30.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.930+0000 7f1ceca00640 1 --2- 172.21.15.45:0/2693468095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce80cb7d0 0x7f1ce80cd620 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8070350 tx=0x7f1cd402b880 comp rx=0 tx=0).stop 2024-04-16T16:17:30.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.930+0000 7f1ceca00640 1 -- 172.21.15.45:0/2693468095 shutdown_connections 2024-04-16T16:17:30.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.931+0000 7f1ceca00640 1 --2- 172.21.15.45:0/2693468095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce80cb7d0 0x7f1ce80cd620 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:30.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.931+0000 7f1ceca00640 1 --2- 172.21.15.45:0/2693468095 >> v2:172.21.15.103:3300/0 conn(0x7f1ce815e830 0x7f1ce80ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:30.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.931+0000 7f1ceca00640 1 -- 172.21.15.45:0/2693468095 >> 172.21.15.45:0/2693468095 conn(0x7f1ce806b930 msgr2=0x7f1ce806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:30.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.931+0000 7f1ceca00640 1 -- 172.21.15.45:0/2693468095 shutdown_connections 2024-04-16T16:17:30.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.931+0000 7f1ceca00640 1 -- 172.21.15.45:0/2693468095 wait complete. 2024-04-16T16:17:30.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.932+0000 7f1ceca00640 1 Processor -- start 2024-04-16T16:17:30.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.932+0000 7f1ceca00640 1 -- start start 2024-04-16T16:17:30.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.933+0000 7f1ceca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ce80cb7d0 0x7f1ce8158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:30.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.933+0000 7f1ceca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ce80cbc80 0x7f1ce81f4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:30.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.933+0000 7f1ceca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce815e830 0x7f1ce81f4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:30.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.933+0000 7f1ceca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce815ec10 con 0x7f1ce815e830 2024-04-16T16:17:30.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.933+0000 7f1ceca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce80cb2b0 con 0x7f1ce80cbc80 2024-04-16T16:17:30.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.933+0000 7f1ceca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1ce80cac30 con 0x7f1ce80cb7d0 2024-04-16T16:17:30.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.934+0000 7f1ce7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce815e830 0x7f1ce81f4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:30.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.934+0000 7f1ce7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce815e830 0x7f1ce81f4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41626/0 (socket says 172.21.15.45:41626) 2024-04-16T16:17:30.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.934+0000 7f1ce7e00640 1 -- 172.21.15.45:0/501022369 learned_addr learned my addr 172.21.15.45:0/501022369 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:30.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.934+0000 7f1ce7e00640 1 -- 172.21.15.45:0/501022369 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ce80cb7d0 msgr2=0x7f1ce8158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:30.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.934+0000 7f1ce7e00640 1 --2- 172.21.15.45:0/501022369 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ce80cb7d0 0x7f1ce8158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:30.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.934+0000 7f1ce7e00640 1 -- 172.21.15.45:0/501022369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ce80cbc80 msgr2=0x7f1ce81f4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:30.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.934+0000 7f1ce7e00640 1 --2- 172.21.15.45:0/501022369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ce80cbc80 0x7f1ce81f4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:30.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.934+0000 7f1ce7e00640 1 -- 172.21.15.45:0/501022369 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce815be50 con 0x7f1ce815e830 2024-04-16T16:17:30.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.935+0000 7f1ce7e00640 1 --2- 172.21.15.45:0/501022369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce815e830 0x7f1ce81f4d80 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000b990 tx=0x7f1ce000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:30.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.935+0000 7f1cdf400640 1 -- 172.21.15.45:0/501022369 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0019070 con 0x7f1ce815e830 2024-04-16T16:17:30.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.935+0000 7f1cdf400640 1 -- 172.21.15.45:0/501022369 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ce000cba0 con 0x7f1ce815e830 2024-04-16T16:17:30.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.935+0000 7f1cdf400640 1 -- 172.21.15.45:0/501022369 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0014440 con 0x7f1ce815e830 2024-04-16T16:17:30.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.935+0000 7f1ceca00640 1 -- 172.21.15.45:0/501022369 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ce815bb70 con 0x7f1ce815e830 2024-04-16T16:17:30.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.935+0000 7f1ceca00640 1 -- 172.21.15.45:0/501022369 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ce8143b00 con 0x7f1ce815e830 2024-04-16T16:17:30.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.936+0000 7f1ceca00640 1 -- 172.21.15.45:0/501022369 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ce8075780 con 0x7f1ce815e830 2024-04-16T16:17:30.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.937+0000 7f1cdf400640 1 -- 172.21.15.45:0/501022369 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f1ce001c070 con 0x7f1ce815e830 2024-04-16T16:17:30.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.938+0000 7f1cdf400640 1 --2- 172.21.15.45:0/501022369 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f1cc806ca60 0x7f1cc806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:30.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.938+0000 7f1cdf400640 1 -- 172.21.15.45:0/501022369 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f1ce008bc50 con 0x7f1ce815e830 2024-04-16T16:17:30.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.939+0000 7f1ce7400640 1 --2- 172.21.15.45:0/501022369 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f1cc806ca60 0x7f1cc806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:30.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.940+0000 7f1ce7400640 1 --2- 172.21.15.45:0/501022369 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f1cc806ca60 0x7f1cc806ef20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1ce815c9c0 tx=0x7f1cd8006c90 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:30.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:30.943+0000 7f1cdf400640 1 -- 172.21.15.45:0/501022369 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1ce0091020 con 0x7f1ce815e830 2024-04-16T16:17:31.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:31.174+0000 7f1ceca00640 1 -- 172.21.15.45:0/501022369 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f1ce8075480 con 0x7f1cc806ca60 2024-04-16T16:17:31.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:31.177+0000 7f1cdf400640 1 -- 172.21.15.45:0/501022369 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3039 (secure 0 0 0) 0x7f1ce8075480 con 0x7f1cc806ca60 2024-04-16T16:17:31.176 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[419, 22], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], "client.5285": [[2, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[2, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[52, 2], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}, "mds.1": {"client.5243": []}}} 2024-04-16T16:17:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:31.179+0000 7f1ceca00640 1 -- 172.21.15.45:0/501022369 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f1cc806ca60 msgr2=0x7f1cc806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:31.179+0000 7f1ceca00640 1 --2- 172.21.15.45:0/501022369 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f1cc806ca60 0x7f1cc806ef20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1ce815c9c0 tx=0x7f1cd8006c90 comp rx=0 tx=0).stop 2024-04-16T16:17:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:31.179+0000 7f1ceca00640 1 -- 172.21.15.45:0/501022369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce815e830 msgr2=0x7f1ce81f4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:31.179+0000 7f1ceca00640 1 --2- 172.21.15.45:0/501022369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce815e830 0x7f1ce81f4d80 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000b990 tx=0x7f1ce000be50 comp rx=0 tx=0).stop 2024-04-16T16:17:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:31.180+0000 7f1ceca00640 1 -- 172.21.15.45:0/501022369 shutdown_connections 2024-04-16T16:17:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:31.180+0000 7f1ceca00640 1 --2- 172.21.15.45:0/501022369 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f1cc806ca60 0x7f1cc806ef20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:31.180+0000 7f1ceca00640 1 --2- 172.21.15.45:0/501022369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce815e830 0x7f1ce81f4d80 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:31.180+0000 7f1ceca00640 1 --2- 172.21.15.45:0/501022369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ce80cbc80 0x7f1ce81f4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:31.180+0000 7f1ceca00640 1 --2- 172.21.15.45:0/501022369 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ce80cb7d0 0x7f1ce8158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:31.180+0000 7f1ceca00640 1 -- 172.21.15.45:0/501022369 >> 172.21.15.45:0/501022369 conn(0x7f1ce806b930 msgr2=0x7f1ce80cc410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:31.180+0000 7f1ceca00640 1 -- 172.21.15.45:0/501022369 shutdown_connections 2024-04-16T16:17:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:31.180+0000 7f1ceca00640 1 -- 172.21.15.45:0/501022369 wait complete. 2024-04-16T16:17:32.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:32.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.435+0000 7f4d76a00640 1 Processor -- start 2024-04-16T16:17:32.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.435+0000 7f4d76a00640 1 -- start start 2024-04-16T16:17:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.437+0000 7f4d76a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4d700cd930 0x7f4d700cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.437+0000 7f4d76a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d70146600 con 0x7f4d70146220 2024-04-16T16:17:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.437+0000 7f4d76a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d700cde00 con 0x7f4d70072890 2024-04-16T16:17:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.437+0000 7f4d76a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f4d7006f840 con 0x7f4d700cd930 2024-04-16T16:17:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.437+0000 7f4d6fe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4d70146220 0x7f4d70072180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47376/0 (socket says 172.21.15.45:47376) 2024-04-16T16:17:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.437+0000 7f4d6fe00640 1 -- 172.21.15.45:0/1229274597 learned_addr learned my addr 172.21.15.45:0/1229274597 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.437+0000 7f4d74a00640 1 --2- 172.21.15.45:0/1229274597 >> v2:172.21.15.45:3300/0 conn(0x7f4d700cd930 0x7f4d700cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:32.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.437+0000 7f4d6f400640 1 -- 172.21.15.45:0/1229274597 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3949756575 0 0) 0x7f4d70146600 con 0x7f4d70146220 2024-04-16T16:17:32.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.438+0000 7f4d6f400640 1 -- 172.21.15.45:0/1229274597 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d54003520 con 0x7f4d70146220 2024-04-16T16:17:32.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.438+0000 7f4d74a00640 1 -- 172.21.15.45:0/1229274597 >> v1:172.21.15.103:6789/0 conn(0x7f4d70072890 legacy=0x7f4d700cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:32.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.438+0000 7f4d74a00640 1 -- 172.21.15.45:0/1229274597 >> v1:172.21.15.45:6789/0 conn(0x7f4d70146220 legacy=0x7f4d70072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:32.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.438+0000 7f4d74a00640 1 -- 172.21.15.45:0/1229274597 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d700ce630 con 0x7f4d700cd930 2024-04-16T16:17:32.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.439+0000 7f4d74a00640 1 --2- 172.21.15.45:0/1229274597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700cd930 0x7f4d700cf110 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f4d68009d90 tx=0x7f4d6802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85a47b01cccb0d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:32.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.439+0000 7f4d6f400640 1 -- 172.21.15.45:0/1229274597 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d68039070 con 0x7f4d700cd930 2024-04-16T16:17:32.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.439+0000 7f4d6f400640 1 -- 172.21.15.45:0/1229274597 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d6802ccf0 con 0x7f4d700cd930 2024-04-16T16:17:32.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.439+0000 7f4d6f400640 1 -- 172.21.15.45:0/1229274597 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d68034640 con 0x7f4d700cd930 2024-04-16T16:17:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.440+0000 7f4d76a00640 1 -- 172.21.15.45:0/1229274597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700cd930 msgr2=0x7f4d700cf110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.440+0000 7f4d76a00640 1 --2- 172.21.15.45:0/1229274597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700cd930 0x7f4d700cf110 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f4d68009d90 tx=0x7f4d6802bf40 comp rx=0 tx=0).stop 2024-04-16T16:17:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.440+0000 7f4d76a00640 1 -- 172.21.15.45:0/1229274597 shutdown_connections 2024-04-16T16:17:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.440+0000 7f4d76a00640 1 --2- 172.21.15.45:0/1229274597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700cd930 0x7f4d700cf110 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.440+0000 7f4d76a00640 1 -- 172.21.15.45:0/1229274597 >> 172.21.15.45:0/1229274597 conn(0x7f4d7006b930 msgr2=0x7f4d7006bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.440+0000 7f4d76a00640 1 -- 172.21.15.45:0/1229274597 shutdown_connections 2024-04-16T16:17:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.440+0000 7f4d76a00640 1 -- 172.21.15.45:0/1229274597 wait complete. 2024-04-16T16:17:32.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.442+0000 7f4d76a00640 1 Processor -- start 2024-04-16T16:17:32.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.442+0000 7f4d76a00640 1 -- start start 2024-04-16T16:17:32.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.443+0000 7f4d76a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d70072890 0x7f4d701f6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:32.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.443+0000 7f4d76a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d700cd930 0x7f4d7015f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:32.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.443+0000 7f4d76a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d70146220 0x7f4d700c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:32.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.443+0000 7f4d76a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4d700cebf0 con 0x7f4d70072890 2024-04-16T16:17:32.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.443+0000 7f4d76a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4d700ce0e0 con 0x7f4d700cd930 2024-04-16T16:17:32.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.443+0000 7f4d76a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4d7006df00 con 0x7f4d70146220 2024-04-16T16:17:32.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.443+0000 7f4d6fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d70072890 0x7f4d701f6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:32.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.443+0000 7f4d6fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d70072890 0x7f4d701f6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41642/0 (socket says 172.21.15.45:41642) 2024-04-16T16:17:32.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.443+0000 7f4d6fe00640 1 -- 172.21.15.45:0/3051883475 learned_addr learned my addr 172.21.15.45:0/3051883475 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:32.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.444+0000 7f4d6fe00640 1 -- 172.21.15.45:0/3051883475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d70146220 msgr2=0x7f4d700c8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:32.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.444+0000 7f4d6fe00640 1 --2- 172.21.15.45:0/3051883475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d70146220 0x7f4d700c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:32.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.444+0000 7f4d6fe00640 1 -- 172.21.15.45:0/3051883475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d700cd930 msgr2=0x7f4d7015f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:32.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.444+0000 7f4d6fe00640 1 --2- 172.21.15.45:0/3051883475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d700cd930 0x7f4d7015f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:32.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.444+0000 7f4d6fe00640 1 -- 172.21.15.45:0/3051883475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d700cb550 con 0x7f4d70072890 2024-04-16T16:17:32.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.444+0000 7f4d6fe00640 1 --2- 172.21.15.45:0/3051883475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d70072890 0x7f4d701f6ff0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f4d6000c400 tx=0x7f4d6000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:32.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.444+0000 7f4d6cc00640 1 -- 172.21.15.45:0/3051883475 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d60017070 con 0x7f4d70072890 2024-04-16T16:17:32.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.444+0000 7f4d6cc00640 1 -- 172.21.15.45:0/3051883475 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d60005ce0 con 0x7f4d70072890 2024-04-16T16:17:32.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.445+0000 7f4d6cc00640 1 -- 172.21.15.45:0/3051883475 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d600072d0 con 0x7f4d70072890 2024-04-16T16:17:32.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.445+0000 7f4d76a00640 1 -- 172.21.15.45:0/3051883475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d700cb830 con 0x7f4d70072890 2024-04-16T16:17:32.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.445+0000 7f4d76a00640 1 -- 172.21.15.45:0/3051883475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d700cbdf0 con 0x7f4d70072890 2024-04-16T16:17:32.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.446+0000 7f4d76a00640 1 -- 172.21.15.45:0/3051883475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d70075780 con 0x7f4d70072890 2024-04-16T16:17:32.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.447+0000 7f4d6cc00640 1 -- 172.21.15.45:0/3051883475 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f4d6001a070 con 0x7f4d70072890 2024-04-16T16:17:32.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.448+0000 7f4d6cc00640 1 --2- 172.21.15.45:0/3051883475 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4d4c06ca60 0x7f4d4c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:32.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.448+0000 7f4d6cc00640 1 -- 172.21.15.45:0/3051883475 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f4d6008b200 con 0x7f4d70072890 2024-04-16T16:17:32.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.449+0000 7f4d67400640 1 --2- 172.21.15.45:0/3051883475 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4d4c06ca60 0x7f4d4c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:32.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.450+0000 7f4d67400640 1 --2- 172.21.15.45:0/3051883475 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4d4c06ca60 0x7f4d4c06ef20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4d58000f60 tx=0x7f4d580079b0 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:32.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.452+0000 7f4d6cc00640 1 -- 172.21.15.45:0/3051883475 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d60058d10 con 0x7f4d70072890 2024-04-16T16:17:32.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.690+0000 7f4d76a00640 1 -- 172.21.15.45:0/3051883475 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f4d7015c980 con 0x7f4d4c06ca60 2024-04-16T16:17:32.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.693+0000 7f4d6cc00640 1 -- 172.21.15.45:0/3051883475 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3039 (secure 0 0 0) 0x7f4d7015c980 con 0x7f4d4c06ca60 2024-04-16T16:17:32.692 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[419, 22], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], "client.5285": [[2, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[2, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[52, 2], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}, "mds.1": {"client.5243": []}}} 2024-04-16T16:17:32.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.696+0000 7f4d76a00640 1 -- 172.21.15.45:0/3051883475 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4d4c06ca60 msgr2=0x7f4d4c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:32.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.696+0000 7f4d76a00640 1 --2- 172.21.15.45:0/3051883475 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4d4c06ca60 0x7f4d4c06ef20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4d58000f60 tx=0x7f4d580079b0 comp rx=0 tx=0).stop 2024-04-16T16:17:32.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.696+0000 7f4d76a00640 1 -- 172.21.15.45:0/3051883475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d70072890 msgr2=0x7f4d701f6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.696+0000 7f4d76a00640 1 --2- 172.21.15.45:0/3051883475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d70072890 0x7f4d701f6ff0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f4d6000c400 tx=0x7f4d6000c8c0 comp rx=0 tx=0).stop 2024-04-16T16:17:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.696+0000 7f4d76a00640 1 -- 172.21.15.45:0/3051883475 shutdown_connections 2024-04-16T16:17:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.696+0000 7f4d76a00640 1 --2- 172.21.15.45:0/3051883475 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4d4c06ca60 0x7f4d4c06ef20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.696+0000 7f4d76a00640 1 --2- 172.21.15.45:0/3051883475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d70146220 0x7f4d700c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.696+0000 7f4d76a00640 1 --2- 172.21.15.45:0/3051883475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d700cd930 0x7f4d7015f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.696+0000 7f4d76a00640 1 --2- 172.21.15.45:0/3051883475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d70072890 0x7f4d701f6ff0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.696+0000 7f4d76a00640 1 -- 172.21.15.45:0/3051883475 >> 172.21.15.45:0/3051883475 conn(0x7f4d7006b930 msgr2=0x7f4d7006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.696+0000 7f4d76a00640 1 -- 172.21.15.45:0/3051883475 shutdown_connections 2024-04-16T16:17:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:32.697+0000 7f4d76a00640 1 -- 172.21.15.45:0/3051883475 wait complete. 2024-04-16T16:17:33.714 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:33.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.951+0000 7f4129e00640 1 Processor -- start 2024-04-16T16:17:33.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.952+0000 7f4129e00640 1 -- start start 2024-04-16T16:17:33.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.953+0000 7f4129e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4124162090 0x7f41240cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:33.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.953+0000 7f4129e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f41240d5520 0x7f41240cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:33.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.953+0000 7f4129e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41240d2600 con 0x7f41240ba140 2024-04-16T16:17:33.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.953+0000 7f4129e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f41240ba8b0 con 0x7f41240d5520 2024-04-16T16:17:33.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.953+0000 7f4129e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f41240ba5d0 con 0x7f4124162090 2024-04-16T16:17:33.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.954+0000 7f4129400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f41240d5520 0x7f41240cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:33.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.954+0000 7f4129400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f41240d5520 0x7f41240cfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50180/0 (socket says 172.21.15.45:50180) 2024-04-16T16:17:33.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.954+0000 7f4129400640 1 -- 172.21.15.45:0/3344505750 learned_addr learned my addr 172.21.15.45:0/3344505750 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:33.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.954+0000 7f4128a00640 1 --2- 172.21.15.45:0/3344505750 >> v2:172.21.15.103:3300/0 conn(0x7f4124162090 0x7f41240cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:33.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.954+0000 7f4129400640 1 -- 172.21.15.45:0/3344505750 >> v2:172.21.15.103:3300/0 conn(0x7f4124162090 msgr2=0x7f41240cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:33.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.955+0000 7f4129400640 1 --2- 172.21.15.45:0/3344505750 >> v2:172.21.15.103:3300/0 conn(0x7f4124162090 0x7f41240cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:33.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.955+0000 7f4129400640 1 -- 172.21.15.45:0/3344505750 >> v1:172.21.15.103:6789/0 conn(0x7f41240ba140 legacy=0x7f41241628a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:33.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.955+0000 7f4129400640 1 -- 172.21.15.45:0/3344505750 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41240d28a0 con 0x7f41240d5520 2024-04-16T16:17:33.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.956+0000 7f4129400640 1 --2- 172.21.15.45:0/3344505750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41240d5520 0x7f41240cfa50 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f412408d5c0 tx=0x7f411c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54082ef9fa68df0f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:33.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.956+0000 7f4123400640 1 -- 172.21.15.45:0/3344505750 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c034070 con 0x7f41240d5520 2024-04-16T16:17:33.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.957+0000 7f4123400640 1 -- 172.21.15.45:0/3344505750 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f411c030860 con 0x7f41240d5520 2024-04-16T16:17:33.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.957+0000 7f4123400640 1 -- 172.21.15.45:0/3344505750 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c030c30 con 0x7f41240d5520 2024-04-16T16:17:33.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.957+0000 7f4129e00640 1 -- 172.21.15.45:0/3344505750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41240d5520 msgr2=0x7f41240cfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:33.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.957+0000 7f4129e00640 1 --2- 172.21.15.45:0/3344505750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41240d5520 0x7f41240cfa50 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f412408d5c0 tx=0x7f411c02b9a0 comp rx=0 tx=0).stop 2024-04-16T16:17:33.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.957+0000 7f4129e00640 1 -- 172.21.15.45:0/3344505750 shutdown_connections 2024-04-16T16:17:33.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.957+0000 7f4129e00640 1 --2- 172.21.15.45:0/3344505750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41240d5520 0x7f41240cfa50 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:33.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.957+0000 7f4129e00640 1 --2- 172.21.15.45:0/3344505750 >> v2:172.21.15.103:3300/0 conn(0x7f4124162090 0x7f41240cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:33.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.957+0000 7f4129e00640 1 -- 172.21.15.45:0/3344505750 >> 172.21.15.45:0/3344505750 conn(0x7f41240d1940 msgr2=0x7f41240d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:33.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.957+0000 7f4129e00640 1 -- 172.21.15.45:0/3344505750 shutdown_connections 2024-04-16T16:17:33.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.957+0000 7f4129e00640 1 -- 172.21.15.45:0/3344505750 wait complete. 2024-04-16T16:17:33.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.959+0000 7f4129e00640 1 Processor -- start 2024-04-16T16:17:33.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.959+0000 7f4129e00640 1 -- start start 2024-04-16T16:17:33.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.960+0000 7f4129e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41240d5520 0x7f41240ce1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:33.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.960+0000 7f4129e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4124162090 0x7f41241538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:33.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.960+0000 7f4129e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41240cc370 0x7f41240c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:33.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.960+0000 7f4129e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f41241625f0 con 0x7f41240cc370 2024-04-16T16:17:33.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.960+0000 7f4129e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4124162470 con 0x7f41240d5520 2024-04-16T16:17:33.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.960+0000 7f4129e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f41240d40b0 con 0x7f4124162090 2024-04-16T16:17:33.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.961+0000 7f4129400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41240cc370 0x7f41240c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:33.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.961+0000 7f4129400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41240cc370 0x7f41240c9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50182/0 (socket says 172.21.15.45:50182) 2024-04-16T16:17:33.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.961+0000 7f4129400640 1 -- 172.21.15.45:0/1074241557 learned_addr learned my addr 172.21.15.45:0/1074241557 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:33.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.961+0000 7f4128a00640 1 --2- 172.21.15.45:0/1074241557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41240d5520 0x7f41240ce1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:33.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.961+0000 7f4123e00640 1 --2- 172.21.15.45:0/1074241557 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4124162090 0x7f41241538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:33.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.961+0000 7f4129400640 1 -- 172.21.15.45:0/1074241557 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4124162090 msgr2=0x7f41241538d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:33.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.962+0000 7f4129400640 1 --2- 172.21.15.45:0/1074241557 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4124162090 0x7f41241538d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:33.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.962+0000 7f4129400640 1 -- 172.21.15.45:0/1074241557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41240d5520 msgr2=0x7f41240ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:33.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.962+0000 7f4129400640 1 --2- 172.21.15.45:0/1074241557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41240d5520 0x7f41240ce1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:33.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.962+0000 7f4129400640 1 -- 172.21.15.45:0/1074241557 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4124157f10 con 0x7f41240cc370 2024-04-16T16:17:33.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.962+0000 7f4129400640 1 --2- 172.21.15.45:0/1074241557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41240cc370 0x7f41240c9f60 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f411c009960 tx=0x7f411c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:33.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.962+0000 7f4120c00640 1 -- 172.21.15.45:0/1074241557 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c048070 con 0x7f41240cc370 2024-04-16T16:17:33.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.962+0000 7f4120c00640 1 -- 172.21.15.45:0/1074241557 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f411c030660 con 0x7f41240cc370 2024-04-16T16:17:33.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.962+0000 7f4120c00640 1 -- 172.21.15.45:0/1074241557 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c034040 con 0x7f41240cc370 2024-04-16T16:17:33.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.962+0000 7f4129e00640 1 -- 172.21.15.45:0/1074241557 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41240ccba0 con 0x7f41240cc370 2024-04-16T16:17:33.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.963+0000 7f4129e00640 1 -- 172.21.15.45:0/1074241557 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4124158490 con 0x7f41240cc370 2024-04-16T16:17:33.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.964+0000 7f4129e00640 1 -- 172.21.15.45:0/1074241557 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4124075780 con 0x7f41240cc370 2024-04-16T16:17:33.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.965+0000 7f4120c00640 1 -- 172.21.15.45:0/1074241557 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f411c04b070 con 0x7f41240cc370 2024-04-16T16:17:33.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.966+0000 7f4120c00640 1 --2- 172.21.15.45:0/1074241557 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f410006ca60 0x7f410006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:33.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.966+0000 7f4120c00640 1 -- 172.21.15.45:0/1074241557 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f411c0adef0 con 0x7f41240cc370 2024-04-16T16:17:33.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.966+0000 7f4128a00640 1 --2- 172.21.15.45:0/1074241557 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f410006ca60 0x7f410006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:33.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.968+0000 7f4128a00640 1 --2- 172.21.15.45:0/1074241557 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f410006ca60 0x7f410006ef20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4114009aa0 tx=0x7f4114009190 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:33.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:33.970+0000 7f4120c00640 1 -- 172.21.15.45:0/1074241557 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f411c07b8f0 con 0x7f41240cc370 2024-04-16T16:17:34.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:34.200+0000 7f4129e00640 1 -- 172.21.15.45:0/1074241557 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f412406dab0 con 0x7f410006ca60 2024-04-16T16:17:34.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:34.204+0000 7f4120c00640 1 -- 172.21.15.45:0/1074241557 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3039 (secure 0 0 0) 0x7f412406dab0 con 0x7f410006ca60 2024-04-16T16:17:34.203 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[419, 22], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], "client.5285": [[2, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[2, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[52, 2], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}, "mds.1": {"client.5243": []}}} 2024-04-16T16:17:34.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:34.206+0000 7f4129e00640 1 -- 172.21.15.45:0/1074241557 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f410006ca60 msgr2=0x7f410006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:34.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:34.206+0000 7f4129e00640 1 --2- 172.21.15.45:0/1074241557 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f410006ca60 0x7f410006ef20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4114009aa0 tx=0x7f4114009190 comp rx=0 tx=0).stop 2024-04-16T16:17:34.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:34.206+0000 7f4129e00640 1 -- 172.21.15.45:0/1074241557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41240cc370 msgr2=0x7f41240c9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:34.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:34.206+0000 7f4129e00640 1 --2- 172.21.15.45:0/1074241557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41240cc370 0x7f41240c9f60 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f411c009960 tx=0x7f411c004730 comp rx=0 tx=0).stop 2024-04-16T16:17:34.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:34.206+0000 7f4129e00640 1 -- 172.21.15.45:0/1074241557 shutdown_connections 2024-04-16T16:17:34.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:34.206+0000 7f4129e00640 1 --2- 172.21.15.45:0/1074241557 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f410006ca60 0x7f410006ef20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:34.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:34.206+0000 7f4129e00640 1 --2- 172.21.15.45:0/1074241557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41240cc370 0x7f41240c9f60 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:34.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:34.206+0000 7f4129e00640 1 --2- 172.21.15.45:0/1074241557 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4124162090 0x7f41241538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:34.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:34.206+0000 7f4129e00640 1 --2- 172.21.15.45:0/1074241557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41240d5520 0x7f41240ce1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:34.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:34.206+0000 7f4129e00640 1 -- 172.21.15.45:0/1074241557 >> 172.21.15.45:0/1074241557 conn(0x7f41240d1940 msgr2=0x7f41240b8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:34.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:34.207+0000 7f4129e00640 1 -- 172.21.15.45:0/1074241557 shutdown_connections 2024-04-16T16:17:34.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:34.207+0000 7f4129e00640 1 -- 172.21.15.45:0/1074241557 wait complete. 2024-04-16T16:17:35.226 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:35.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.458+0000 7efda0c00640 1 Processor -- start 2024-04-16T16:17:35.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.459+0000 7efda0c00640 1 -- start start 2024-04-16T16:17:35.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.460+0000 7efda0c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efd9c15e1e0 0x7efd9c071470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:35.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.460+0000 7efda0c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd9c0d0e50 con 0x7efd9c070410 2024-04-16T16:17:35.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.460+0000 7efda0c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd9c071190 con 0x7efd9c09ab50 2024-04-16T16:17:35.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.460+0000 7efda0c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7efd9c0d10c0 con 0x7efd9c15e1e0 2024-04-16T16:17:35.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.460+0000 7efd99400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7efd9c070410 0x7efd9c0708c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36470/0 (socket says 172.21.15.45:36470) 2024-04-16T16:17:35.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.460+0000 7efd99400640 1 -- 172.21.15.45:0/3964561741 learned_addr learned my addr 172.21.15.45:0/3964561741 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:35.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.461+0000 7efd98a00640 1 -- 172.21.15.45:0/3964561741 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1344724021 0 0) 0x7efd9c0d0e50 con 0x7efd9c070410 2024-04-16T16:17:35.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.461+0000 7efd99e00640 1 --2- 172.21.15.45:0/3964561741 >> v2:172.21.15.103:3300/0 conn(0x7efd9c15e1e0 0x7efd9c071470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:35.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.461+0000 7efd98a00640 1 -- 172.21.15.45:0/3964561741 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd7c003520 con 0x7efd9c070410 2024-04-16T16:17:35.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.461+0000 7efd98a00640 1 -- 172.21.15.45:0/3964561741 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2762761313 0 0) 0x7efd7c003520 con 0x7efd9c070410 2024-04-16T16:17:35.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.461+0000 7efd98a00640 1 -- 172.21.15.45:0/3964561741 >> v2:172.21.15.103:3300/0 conn(0x7efd9c15e1e0 msgr2=0x7efd9c071470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:35.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.461+0000 7efd98a00640 1 --2- 172.21.15.45:0/3964561741 >> v2:172.21.15.103:3300/0 conn(0x7efd9c15e1e0 0x7efd9c071470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:35.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.461+0000 7efd98a00640 1 -- 172.21.15.45:0/3964561741 >> v1:172.21.15.103:6789/0 conn(0x7efd9c09ab50 legacy=0x7efd9c09af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:35.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.462+0000 7efd98a00640 1 -- 172.21.15.45:0/3964561741 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd9c0d15e0 con 0x7efd9c070410 2024-04-16T16:17:35.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.462+0000 7efd98a00640 1 -- 172.21.15.45:0/3964561741 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7efd84001670 con 0x7efd9c070410 2024-04-16T16:17:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.462+0000 7efd98a00640 1 -- 172.21.15.45:0/3964561741 >> v1:172.21.15.45:6789/0 conn(0x7efd9c070410 legacy=0x7efd9c0708c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.462+0000 7efd98a00640 1 --2- 172.21.15.45:0/3964561741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd7c0039b0 0x7efd7c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.462+0000 7efd98a00640 1 --2- 172.21.15.45:0/3964561741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd7c004800 0x7efd7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.462+0000 7efd98a00640 1 --2- 172.21.15.45:0/3964561741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7c0051c0 0x7efd7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.462+0000 7efd98a00640 1 -- 172.21.15.45:0/3964561741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efd9c0d10c0 con 0x7efd7c0051c0 2024-04-16T16:17:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.462+0000 7efd98a00640 1 -- 172.21.15.45:0/3964561741 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efd7c00c0c0 con 0x7efd7c004800 2024-04-16T16:17:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.462+0000 7efd98a00640 1 -- 172.21.15.45:0/3964561741 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efd7c00c2c0 con 0x7efd7c0039b0 2024-04-16T16:17:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.462+0000 7efd9a800640 1 --2- 172.21.15.45:0/3964561741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7c0051c0 0x7efd7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.462+0000 7efd99400640 1 --2- 172.21.15.45:0/3964561741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd7c004800 0x7efd7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.463+0000 7efd99e00640 1 --2- 172.21.15.45:0/3964561741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd7c0039b0 0x7efd7c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.463+0000 7efd9a800640 1 -- 172.21.15.45:0/3964561741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd7c0039b0 msgr2=0x7efd7c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.463+0000 7efd9a800640 1 --2- 172.21.15.45:0/3964561741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd7c0039b0 0x7efd7c003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.463+0000 7efd9a800640 1 -- 172.21.15.45:0/3964561741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd7c004800 msgr2=0x7efd7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.463+0000 7efd9a800640 1 --2- 172.21.15.45:0/3964561741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd7c004800 0x7efd7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:35.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.463+0000 7efd9a800640 1 -- 172.21.15.45:0/3964561741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd7c003520 con 0x7efd7c0051c0 2024-04-16T16:17:35.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.463+0000 7efd99400640 1 --2- 172.21.15.45:0/3964561741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd7c004800 0x7efd7c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:35.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.464+0000 7efd9a800640 1 -- 172.21.15.45:0/3964561741 reap_dead start 2024-04-16T16:17:35.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.464+0000 7efd9a800640 1 --2- 172.21.15.45:0/3964561741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7c0051c0 0x7efd7c00b9c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7efd94001f60 tx=0x7efd7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec9e277f541af94 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:35.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.464+0000 7efd98a00640 1 -- 172.21.15.45:0/3964561741 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd94001b30 con 0x7efd7c0051c0 2024-04-16T16:17:35.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.464+0000 7efd98a00640 1 -- 172.21.15.45:0/3964561741 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd94003b90 con 0x7efd7c0051c0 2024-04-16T16:17:35.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.464+0000 7efd98a00640 1 -- 172.21.15.45:0/3964561741 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd94005070 con 0x7efd7c0051c0 2024-04-16T16:17:35.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.465+0000 7efda0c00640 1 -- 172.21.15.45:0/3964561741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7c0051c0 msgr2=0x7efd7c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:35.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.465+0000 7efda0c00640 1 --2- 172.21.15.45:0/3964561741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7c0051c0 0x7efd7c00b9c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7efd94001f60 tx=0x7efd7c005830 comp rx=0 tx=0).stop 2024-04-16T16:17:35.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.465+0000 7efda0c00640 1 -- 172.21.15.45:0/3964561741 shutdown_connections 2024-04-16T16:17:35.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.465+0000 7efda0c00640 1 --2- 172.21.15.45:0/3964561741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7c0051c0 0x7efd7c00b9c0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:35.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.465+0000 7efda0c00640 1 -- 172.21.15.45:0/3964561741 >> 172.21.15.45:0/3964561741 conn(0x7efd9c0b42c0 msgr2=0x7efd9c0b46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:35.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.465+0000 7efda0c00640 1 -- 172.21.15.45:0/3964561741 shutdown_connections 2024-04-16T16:17:35.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.465+0000 7efda0c00640 1 -- 172.21.15.45:0/3964561741 wait complete. 2024-04-16T16:17:35.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.467+0000 7efda0c00640 1 Processor -- start 2024-04-16T16:17:35.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.467+0000 7efda0c00640 1 -- start start 2024-04-16T16:17:35.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.468+0000 7efda0c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7c0051c0 0x7efd9c15c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.468+0000 7efda0c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd9c0cc2e0 0x7efd9c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.468+0000 7efda0c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd9c0ca270 0x7efd9c0ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.468+0000 7efda0c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efd9c071470 con 0x7efd7c0051c0 2024-04-16T16:17:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.468+0000 7efda0c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efd9c072460 con 0x7efd9c0cc2e0 2024-04-16T16:17:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.468+0000 7efda0c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efd9c0715f0 con 0x7efd9c0ca270 2024-04-16T16:17:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.468+0000 7efd99e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7c0051c0 0x7efd9c15c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.468+0000 7efd99e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7c0051c0 0x7efd9c15c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50194/0 (socket says 172.21.15.45:50194) 2024-04-16T16:17:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.468+0000 7efd99e00640 1 -- 172.21.15.45:0/722256820 learned_addr learned my addr 172.21.15.45:0/722256820 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.469+0000 7efd99e00640 1 -- 172.21.15.45:0/722256820 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd9c0ca270 msgr2=0x7efd9c0ca720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.469+0000 7efd99e00640 1 --2- 172.21.15.45:0/722256820 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd9c0ca270 0x7efd9c0ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.469+0000 7efd99e00640 1 -- 172.21.15.45:0/722256820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd9c0cc2e0 msgr2=0x7efd9c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.469+0000 7efd99e00640 1 --2- 172.21.15.45:0/722256820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd9c0cc2e0 0x7efd9c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.469+0000 7efd99e00640 1 -- 172.21.15.45:0/722256820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd9c1f4ee0 con 0x7efd7c0051c0 2024-04-16T16:17:35.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.469+0000 7efd99e00640 1 --2- 172.21.15.45:0/722256820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7c0051c0 0x7efd9c15c950 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7efd8c010890 tx=0x7efd8c010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:35.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.469+0000 7efd92000640 1 -- 172.21.15.45:0/722256820 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8c00f040 con 0x7efd7c0051c0 2024-04-16T16:17:35.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.469+0000 7efd92000640 1 -- 172.21.15.45:0/722256820 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd8c00eb00 con 0x7efd7c0051c0 2024-04-16T16:17:35.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.469+0000 7efd92000640 1 -- 172.21.15.45:0/722256820 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8c01c400 con 0x7efd7c0051c0 2024-04-16T16:17:35.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.469+0000 7efda0c00640 1 -- 172.21.15.45:0/722256820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd9c1f51c0 con 0x7efd7c0051c0 2024-04-16T16:17:35.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.469+0000 7efda0c00640 1 -- 172.21.15.45:0/722256820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd9c1f5780 con 0x7efd7c0051c0 2024-04-16T16:17:35.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.471+0000 7efda0c00640 1 -- 172.21.15.45:0/722256820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd9c06e110 con 0x7efd7c0051c0 2024-04-16T16:17:35.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.472+0000 7efd92000640 1 -- 172.21.15.45:0/722256820 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7efd8c01c5a0 con 0x7efd7c0051c0 2024-04-16T16:17:35.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.473+0000 7efd92000640 1 --2- 172.21.15.45:0/722256820 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7efd7806ca90 0x7efd7806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:35.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.473+0000 7efd92000640 1 -- 172.21.15.45:0/722256820 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7efd8c08c4d0 con 0x7efd7c0051c0 2024-04-16T16:17:35.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.473+0000 7efd99400640 1 --2- 172.21.15.45:0/722256820 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7efd7806ca90 0x7efd7806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:35.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.474+0000 7efd99400640 1 --2- 172.21.15.45:0/722256820 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7efd7806ca90 0x7efd7806ef50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efd84001200 tx=0x7efd84009040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:35.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.477+0000 7efd92000640 1 -- 172.21.15.45:0/722256820 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efd8c018030 con 0x7efd7c0051c0 2024-04-16T16:17:35.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.711+0000 7efda0c00640 1 -- 172.21.15.45:0/722256820 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7efd9c15ce90 con 0x7efd7806ca90 2024-04-16T16:17:35.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.715+0000 7efd92000640 1 -- 172.21.15.45:0/722256820 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3039 (secure 0 0 0) 0x7efd840014d0 con 0x7efd7806ca90 2024-04-16T16:17:35.713 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[419, 22], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], "client.5285": [[2, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[2, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[52, 2], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}, "mds.1": {"client.5243": []}}} 2024-04-16T16:17:35.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.717+0000 7efda0c00640 1 -- 172.21.15.45:0/722256820 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7efd7806ca90 msgr2=0x7efd7806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:35.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.717+0000 7efda0c00640 1 --2- 172.21.15.45:0/722256820 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7efd7806ca90 0x7efd7806ef50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efd84001200 tx=0x7efd84009040 comp rx=0 tx=0).stop 2024-04-16T16:17:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.717+0000 7efda0c00640 1 -- 172.21.15.45:0/722256820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7c0051c0 msgr2=0x7efd9c15c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.717+0000 7efda0c00640 1 --2- 172.21.15.45:0/722256820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7c0051c0 0x7efd9c15c950 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7efd8c010890 tx=0x7efd8c010d50 comp rx=0 tx=0).stop 2024-04-16T16:17:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.717+0000 7efda0c00640 1 -- 172.21.15.45:0/722256820 shutdown_connections 2024-04-16T16:17:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.717+0000 7efda0c00640 1 --2- 172.21.15.45:0/722256820 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7efd7806ca90 0x7efd7806ef50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.717+0000 7efda0c00640 1 --2- 172.21.15.45:0/722256820 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd9c0ca270 0x7efd9c0ca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.717+0000 7efda0c00640 1 --2- 172.21.15.45:0/722256820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd9c0cc2e0 0x7efd9c0cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.717+0000 7efda0c00640 1 --2- 172.21.15.45:0/722256820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7c0051c0 0x7efd9c15c950 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.717+0000 7efda0c00640 1 -- 172.21.15.45:0/722256820 >> 172.21.15.45:0/722256820 conn(0x7efd9c0b42c0 msgr2=0x7efd9c0ce600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.717+0000 7efda0c00640 1 -- 172.21.15.45:0/722256820 shutdown_connections 2024-04-16T16:17:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:35.717+0000 7efda0c00640 1 -- 172.21.15.45:0/722256820 wait complete. 2024-04-16T16:17:36.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:36.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.971+0000 7fb161c00640 1 Processor -- start 2024-04-16T16:17:36.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.972+0000 7fb161c00640 1 -- start start 2024-04-16T16:17:36.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.973+0000 7fb161c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb15c0ba140 0x7fb15c0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:36.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.973+0000 7fb161c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb15c0cfc20 0x7fb15c0d35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:36.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.973+0000 7fb161c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb15c0d2600 con 0x7fb15c162090 2024-04-16T16:17:36.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.973+0000 7fb161c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb15c0ba8b0 con 0x7fb15c0cfc20 2024-04-16T16:17:36.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.973+0000 7fb161c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb15c0ba5d0 con 0x7fb15c0ba140 2024-04-16T16:17:36.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.974+0000 7fb161200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb15c0cfc20 0x7fb15c0d35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:36.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.974+0000 7fb161200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb15c0cfc20 0x7fb15c0d35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50198/0 (socket says 172.21.15.45:50198) 2024-04-16T16:17:36.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.974+0000 7fb161200640 1 -- 172.21.15.45:0/590985772 learned_addr learned my addr 172.21.15.45:0/590985772 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:36.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.974+0000 7fb15b400640 1 --2- 172.21.15.45:0/590985772 >> v2:172.21.15.103:3300/0 conn(0x7fb15c0ba140 0x7fb15c0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:36.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.974+0000 7fb161200640 1 -- 172.21.15.45:0/590985772 >> v2:172.21.15.103:3300/0 conn(0x7fb15c0ba140 msgr2=0x7fb15c0d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:36.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.974+0000 7fb161200640 1 --2- 172.21.15.45:0/590985772 >> v2:172.21.15.103:3300/0 conn(0x7fb15c0ba140 0x7fb15c0d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:36.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.974+0000 7fb161200640 1 -- 172.21.15.45:0/590985772 >> v1:172.21.15.103:6789/0 conn(0x7fb15c162090 legacy=0x7fb15c0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:36.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.974+0000 7fb161200640 1 -- 172.21.15.45:0/590985772 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb15c0d28a0 con 0x7fb15c0cfc20 2024-04-16T16:17:36.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.974+0000 7fb15b400640 1 --2- 172.21.15.45:0/590985772 >> v2:172.21.15.103:3300/0 conn(0x7fb15c0ba140 0x7fb15c0d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:36.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.976+0000 7fb161200640 1 --2- 172.21.15.45:0/590985772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb15c0cfc20 0x7fb15c0d35e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fb15c0cb8f0 tx=0x7fb15402b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b1398b33af1aef server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:36.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.976+0000 7fb15aa00640 1 -- 172.21.15.45:0/590985772 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb154035020 con 0x7fb15c0cfc20 2024-04-16T16:17:36.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.976+0000 7fb15aa00640 1 -- 172.21.15.45:0/590985772 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb154031640 con 0x7fb15c0cfc20 2024-04-16T16:17:36.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.976+0000 7fb15aa00640 1 -- 172.21.15.45:0/590985772 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb154031a10 con 0x7fb15c0cfc20 2024-04-16T16:17:36.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.977+0000 7fb161c00640 1 -- 172.21.15.45:0/590985772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb15c0cfc20 msgr2=0x7fb15c0d35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:36.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.977+0000 7fb161c00640 1 --2- 172.21.15.45:0/590985772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb15c0cfc20 0x7fb15c0d35e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fb15c0cb8f0 tx=0x7fb15402b8e0 comp rx=0 tx=0).stop 2024-04-16T16:17:36.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.977+0000 7fb161c00640 1 -- 172.21.15.45:0/590985772 shutdown_connections 2024-04-16T16:17:36.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.977+0000 7fb161c00640 1 --2- 172.21.15.45:0/590985772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb15c0cfc20 0x7fb15c0d35e0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:36.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.977+0000 7fb161c00640 1 --2- 172.21.15.45:0/590985772 >> v2:172.21.15.103:3300/0 conn(0x7fb15c0ba140 0x7fb15c0d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:36.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.977+0000 7fb161c00640 1 -- 172.21.15.45:0/590985772 >> 172.21.15.45:0/590985772 conn(0x7fb15c0d1940 msgr2=0x7fb15c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:36.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.977+0000 7fb161c00640 1 -- 172.21.15.45:0/590985772 shutdown_connections 2024-04-16T16:17:36.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.977+0000 7fb161c00640 1 -- 172.21.15.45:0/590985772 wait complete. 2024-04-16T16:17:36.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.979+0000 7fb161c00640 1 Processor -- start 2024-04-16T16:17:36.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.979+0000 7fb161c00640 1 -- start start 2024-04-16T16:17:36.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.980+0000 7fb161c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb15c0cfc20 0x7fb15c0ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:36.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.980+0000 7fb161c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb15c162090 0x7fb15c153db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:36.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.980+0000 7fb161c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb15c0c93a0 0x7fb15c0cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:36.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.980+0000 7fb161c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c1625f0 con 0x7fb15c162090 2024-04-16T16:17:36.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.980+0000 7fb161c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c162470 con 0x7fb15c0c93a0 2024-04-16T16:17:36.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.980+0000 7fb161c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb15c0d5520 con 0x7fb15c0cfc20 2024-04-16T16:17:36.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.980+0000 7fb15b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb15c162090 0x7fb15c153db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:36.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.980+0000 7fb15b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb15c162090 0x7fb15c153db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50212/0 (socket says 172.21.15.45:50212) 2024-04-16T16:17:36.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.980+0000 7fb15b400640 1 -- 172.21.15.45:0/3327942759 learned_addr learned my addr 172.21.15.45:0/3327942759 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:36.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.981+0000 7fb161200640 1 --2- 172.21.15.45:0/3327942759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb15c0c93a0 0x7fb15c0cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:36.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.981+0000 7fb15be00640 1 --2- 172.21.15.45:0/3327942759 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb15c0cfc20 0x7fb15c0ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:36.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.981+0000 7fb15b400640 1 -- 172.21.15.45:0/3327942759 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb15c0cfc20 msgr2=0x7fb15c0ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:36.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.981+0000 7fb15b400640 1 --2- 172.21.15.45:0/3327942759 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb15c0cfc20 0x7fb15c0ca940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:36.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.981+0000 7fb15b400640 1 -- 172.21.15.45:0/3327942759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb15c0c93a0 msgr2=0x7fb15c0cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:36.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.981+0000 7fb15b400640 1 --2- 172.21.15.45:0/3327942759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb15c0c93a0 0x7fb15c0cdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:36.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.981+0000 7fb15b400640 1 -- 172.21.15.45:0/3327942759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb15c0c98a0 con 0x7fb15c162090 2024-04-16T16:17:36.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.981+0000 7fb15be00640 1 --2- 172.21.15.45:0/3327942759 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb15c0cfc20 0x7fb15c0ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:36.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.981+0000 7fb15b400640 1 --2- 172.21.15.45:0/3327942759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb15c162090 0x7fb15c153db0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fb14400c820 tx=0x7fb14400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:36.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.981+0000 7fb153e00640 1 -- 172.21.15.45:0/3327942759 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb144015070 con 0x7fb15c162090 2024-04-16T16:17:36.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.982+0000 7fb153e00640 1 -- 172.21.15.45:0/3327942759 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb144007a30 con 0x7fb15c162090 2024-04-16T16:17:36.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.982+0000 7fb153e00640 1 -- 172.21.15.45:0/3327942759 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb144007e00 con 0x7fb15c162090 2024-04-16T16:17:36.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.982+0000 7fb161c00640 1 -- 172.21.15.45:0/3327942759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb15c0c9b80 con 0x7fb15c162090 2024-04-16T16:17:36.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.982+0000 7fb161c00640 1 -- 172.21.15.45:0/3327942759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb15c0ca140 con 0x7fb15c162090 2024-04-16T16:17:36.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.983+0000 7fb161c00640 1 -- 172.21.15.45:0/3327942759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb15c075780 con 0x7fb15c162090 2024-04-16T16:17:36.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.984+0000 7fb153e00640 1 -- 172.21.15.45:0/3327942759 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7fb144018070 con 0x7fb15c162090 2024-04-16T16:17:36.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.985+0000 7fb153e00640 1 --2- 172.21.15.45:0/3327942759 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fb13806ca90 0x7fb13806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:36.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.985+0000 7fb153e00640 1 -- 172.21.15.45:0/3327942759 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7fb14408bf10 con 0x7fb15c162090 2024-04-16T16:17:36.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.986+0000 7fb15be00640 1 --2- 172.21.15.45:0/3327942759 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fb13806ca90 0x7fb13806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:36.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.987+0000 7fb15be00640 1 --2- 172.21.15.45:0/3327942759 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fb13806ca90 0x7fb13806ef50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb14c007e90 tx=0x7fb14c0073a0 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:36.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:36.990+0000 7fb153e00640 1 -- 172.21.15.45:0/3327942759 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb14405a000 con 0x7fb15c162090 2024-04-16T16:17:37.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:37.225+0000 7fb161c00640 1 -- 172.21.15.45:0/3327942759 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7fb15c0cebc0 con 0x7fb13806ca90 2024-04-16T16:17:37.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:37.229+0000 7fb153e00640 1 -- 172.21.15.45:0/3327942759 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3039 (secure 0 0 0) 0x7fb15c0cebc0 con 0x7fb13806ca90 2024-04-16T16:17:37.228 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[419, 22], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], "client.5285": [[2, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[2, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[52, 2], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}, "mds.1": {"client.5243": []}}} 2024-04-16T16:17:37.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:37.231+0000 7fb161c00640 1 -- 172.21.15.45:0/3327942759 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fb13806ca90 msgr2=0x7fb13806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:37.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:37.231+0000 7fb161c00640 1 --2- 172.21.15.45:0/3327942759 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fb13806ca90 0x7fb13806ef50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb14c007e90 tx=0x7fb14c0073a0 comp rx=0 tx=0).stop 2024-04-16T16:17:37.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:37.231+0000 7fb161c00640 1 -- 172.21.15.45:0/3327942759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb15c162090 msgr2=0x7fb15c153db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:37.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:37.231+0000 7fb161c00640 1 --2- 172.21.15.45:0/3327942759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb15c162090 0x7fb15c153db0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fb14400c820 tx=0x7fb14400cc20 comp rx=0 tx=0).stop 2024-04-16T16:17:37.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:37.232+0000 7fb161c00640 1 -- 172.21.15.45:0/3327942759 shutdown_connections 2024-04-16T16:17:37.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:37.232+0000 7fb161c00640 1 --2- 172.21.15.45:0/3327942759 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fb13806ca90 0x7fb13806ef50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:37.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:37.232+0000 7fb161c00640 1 --2- 172.21.15.45:0/3327942759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb15c0c93a0 0x7fb15c0cdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:37.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:37.232+0000 7fb161c00640 1 --2- 172.21.15.45:0/3327942759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb15c162090 0x7fb15c153db0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:37.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:37.232+0000 7fb161c00640 1 --2- 172.21.15.45:0/3327942759 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb15c0cfc20 0x7fb15c0ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:37.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:37.232+0000 7fb161c00640 1 -- 172.21.15.45:0/3327942759 >> 172.21.15.45:0/3327942759 conn(0x7fb15c0d1940 msgr2=0x7fb15c0b7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:37.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:37.232+0000 7fb161c00640 1 -- 172.21.15.45:0/3327942759 shutdown_connections 2024-04-16T16:17:37.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:37.232+0000 7fb161c00640 1 -- 172.21.15.45:0/3327942759 wait complete. 2024-04-16T16:17:38.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.484+0000 7f5e5ce00640 1 Processor -- start 2024-04-16T16:17:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.485+0000 7f5e5ce00640 1 -- start start 2024-04-16T16:17:38.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.486+0000 7f5e5ce00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5e58147550 0x7f5e58072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:38.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.486+0000 7f5e5ce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e580b5ab0 0x7f5e580b3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:38.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.486+0000 7f5e5ce00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e5806df00 con 0x7f5e580726c0 2024-04-16T16:17:38.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.486+0000 7f5e5ce00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5e5806fc50 con 0x7f5e58147550 2024-04-16T16:17:38.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.486+0000 7f5e5ce00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5e5806e1e0 con 0x7f5e580b5ab0 2024-04-16T16:17:38.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.487+0000 7f5e55e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5e58147550 0x7f5e58072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:38.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.487+0000 7f5e55e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5e58147550 0x7f5e58072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50222/0 (socket says 172.21.15.45:50222) 2024-04-16T16:17:38.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.487+0000 7f5e55e00640 1 -- 172.21.15.45:0/672755658 learned_addr learned my addr 172.21.15.45:0/672755658 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:38.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.487+0000 7f5e56800640 1 --2- 172.21.15.45:0/672755658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e580b5ab0 0x7f5e580b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:38.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.487+0000 7f5e55e00640 1 -- 172.21.15.45:0/672755658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e580b5ab0 msgr2=0x7f5e580b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:38.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.487+0000 7f5e55e00640 1 --2- 172.21.15.45:0/672755658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e580b5ab0 0x7f5e580b3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:38.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.488+0000 7f5e55e00640 1 -- 172.21.15.45:0/672755658 >> v1:172.21.15.103:6789/0 conn(0x7f5e580726c0 legacy=0x7f5e580b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:38.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.488+0000 7f5e55e00640 1 -- 172.21.15.45:0/672755658 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e5806e4c0 con 0x7f5e58147550 2024-04-16T16:17:38.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.489+0000 7f5e55e00640 1 --2- 172.21.15.45:0/672755658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e58147550 0x7f5e58072180 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f5e48009870 tx=0x7f5e4802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f9d3583b6029f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:38.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.490+0000 7f5e54a00640 1 -- 172.21.15.45:0/672755658 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e48034070 con 0x7f5e58147550 2024-04-16T16:17:38.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.490+0000 7f5e54a00640 1 -- 172.21.15.45:0/672755658 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e48030840 con 0x7f5e58147550 2024-04-16T16:17:38.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.490+0000 7f5e54a00640 1 -- 172.21.15.45:0/672755658 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e48030c10 con 0x7f5e58147550 2024-04-16T16:17:38.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.490+0000 7f5e5ce00640 1 -- 172.21.15.45:0/672755658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e58147550 msgr2=0x7f5e58072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:38.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.490+0000 7f5e5ce00640 1 --2- 172.21.15.45:0/672755658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e58147550 0x7f5e58072180 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f5e48009870 tx=0x7f5e4802b980 comp rx=0 tx=0).stop 2024-04-16T16:17:38.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.490+0000 7f5e5ce00640 1 -- 172.21.15.45:0/672755658 shutdown_connections 2024-04-16T16:17:38.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.490+0000 7f5e5ce00640 1 --2- 172.21.15.45:0/672755658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e580b5ab0 0x7f5e580b3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:38.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.490+0000 7f5e5ce00640 1 --2- 172.21.15.45:0/672755658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e58147550 0x7f5e58072180 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:38.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.490+0000 7f5e5ce00640 1 -- 172.21.15.45:0/672755658 >> 172.21.15.45:0/672755658 conn(0x7f5e5806b930 msgr2=0x7f5e5806bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:38.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.490+0000 7f5e5ce00640 1 -- 172.21.15.45:0/672755658 shutdown_connections 2024-04-16T16:17:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.490+0000 7f5e5ce00640 1 -- 172.21.15.45:0/672755658 wait complete. 2024-04-16T16:17:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.492+0000 7f5e5ce00640 1 Processor -- start 2024-04-16T16:17:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.492+0000 7f5e5ce00640 1 -- start start 2024-04-16T16:17:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.493+0000 7f5e5ce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e580726c0 0x7f5e58160f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.493+0000 7f5e5ce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e580b5ab0 0x7f5e5815f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.493+0000 7f5e5ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e58147550 0x7f5e5815fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.493+0000 7f5e5ce00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5806efa0 con 0x7f5e58147550 2024-04-16T16:17:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.493+0000 7f5e5ce00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5806ea80 con 0x7f5e580b5ab0 2024-04-16T16:17:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.493+0000 7f5e5ce00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5e58147930 con 0x7f5e580726c0 2024-04-16T16:17:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.494+0000 7f5e56800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e58147550 0x7f5e5815fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:38.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.494+0000 7f5e56800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e58147550 0x7f5e5815fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50234/0 (socket says 172.21.15.45:50234) 2024-04-16T16:17:38.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.494+0000 7f5e56800640 1 -- 172.21.15.45:0/2229933605 learned_addr learned my addr 172.21.15.45:0/2229933605 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:38.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.494+0000 7f5e55400640 1 --2- 172.21.15.45:0/2229933605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e580b5ab0 0x7f5e5815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:38.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.494+0000 7f5e55e00640 1 --2- 172.21.15.45:0/2229933605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e580726c0 0x7f5e58160f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:38.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.494+0000 7f5e56800640 1 -- 172.21.15.45:0/2229933605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e580726c0 msgr2=0x7f5e58160f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:38.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.494+0000 7f5e56800640 1 --2- 172.21.15.45:0/2229933605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e580726c0 0x7f5e58160f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:38.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.494+0000 7f5e56800640 1 -- 172.21.15.45:0/2229933605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e580b5ab0 msgr2=0x7f5e5815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:38.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.494+0000 7f5e56800640 1 --2- 172.21.15.45:0/2229933605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e580b5ab0 0x7f5e5815f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:38.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.494+0000 7f5e56800640 1 -- 172.21.15.45:0/2229933605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e58160aa0 con 0x7f5e58147550 2024-04-16T16:17:38.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.495+0000 7f5e56800640 1 --2- 172.21.15.45:0/2229933605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e58147550 0x7f5e5815fe60 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000d830 tx=0x7f5e5000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:38.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.495+0000 7f5e4e000640 1 -- 172.21.15.45:0/2229933605 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5000b6c0 con 0x7f5e58147550 2024-04-16T16:17:38.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.495+0000 7f5e4e000640 1 -- 172.21.15.45:0/2229933605 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e50019070 con 0x7f5e58147550 2024-04-16T16:17:38.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.495+0000 7f5e4e000640 1 -- 172.21.15.45:0/2229933605 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5000bac0 con 0x7f5e58147550 2024-04-16T16:17:38.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.495+0000 7f5e5ce00640 1 -- 172.21.15.45:0/2229933605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e581607c0 con 0x7f5e58147550 2024-04-16T16:17:38.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.495+0000 7f5e5ce00640 1 -- 172.21.15.45:0/2229933605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e581e88a0 con 0x7f5e58147550 2024-04-16T16:17:38.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.496+0000 7f5e5ce00640 1 -- 172.21.15.45:0/2229933605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e58075780 con 0x7f5e58147550 2024-04-16T16:17:38.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.497+0000 7f5e4e000640 1 -- 172.21.15.45:0/2229933605 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f5e5001e460 con 0x7f5e58147550 2024-04-16T16:17:38.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.498+0000 7f5e4e000640 1 --2- 172.21.15.45:0/2229933605 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f5e3406ca60 0x7f5e3406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:38.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.498+0000 7f5e4e000640 1 -- 172.21.15.45:0/2229933605 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f5e5008c4a0 con 0x7f5e58147550 2024-04-16T16:17:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.499+0000 7f5e55e00640 1 --2- 172.21.15.45:0/2229933605 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f5e3406ca60 0x7f5e3406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.500+0000 7f5e55e00640 1 --2- 172.21.15.45:0/2229933605 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f5e3406ca60 0x7f5e3406ef20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5e48036000 tx=0x7f5e4803b040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.503+0000 7f5e4e000640 1 -- 172.21.15.45:0/2229933605 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5e5005a600 con 0x7f5e58147550 2024-04-16T16:17:38.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.736+0000 7f5e5ce00640 1 -- 172.21.15.45:0/2229933605 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f5e5806f4b0 con 0x7f5e3406ca60 2024-04-16T16:17:38.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.739+0000 7f5e4e000640 1 -- 172.21.15.45:0/2229933605 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3035 (secure 0 0 0) 0x7f5e5806f4b0 con 0x7f5e3406ca60 2024-04-16T16:17:38.738 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[0, 0], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], "client.5285": [[0, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[0, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[0, 0], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}, "mds.1": {"client.5243": []}}} 2024-04-16T16:17:38.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.742+0000 7f5e5ce00640 1 -- 172.21.15.45:0/2229933605 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f5e3406ca60 msgr2=0x7f5e3406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:38.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.742+0000 7f5e5ce00640 1 --2- 172.21.15.45:0/2229933605 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f5e3406ca60 0x7f5e3406ef20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5e48036000 tx=0x7f5e4803b040 comp rx=0 tx=0).stop 2024-04-16T16:17:38.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.742+0000 7f5e5ce00640 1 -- 172.21.15.45:0/2229933605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e58147550 msgr2=0x7f5e5815fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:38.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.742+0000 7f5e5ce00640 1 --2- 172.21.15.45:0/2229933605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e58147550 0x7f5e5815fe60 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000d830 tx=0x7f5e5000dcf0 comp rx=0 tx=0).stop 2024-04-16T16:17:38.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.742+0000 7f5e5ce00640 1 -- 172.21.15.45:0/2229933605 shutdown_connections 2024-04-16T16:17:38.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.742+0000 7f5e5ce00640 1 --2- 172.21.15.45:0/2229933605 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f5e3406ca60 0x7f5e3406ef20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:38.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.742+0000 7f5e5ce00640 1 --2- 172.21.15.45:0/2229933605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e58147550 0x7f5e5815fe60 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:38.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.742+0000 7f5e5ce00640 1 --2- 172.21.15.45:0/2229933605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e580b5ab0 0x7f5e5815f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:38.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.742+0000 7f5e5ce00640 1 --2- 172.21.15.45:0/2229933605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e580726c0 0x7f5e58160f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:38.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.742+0000 7f5e5ce00640 1 -- 172.21.15.45:0/2229933605 >> 172.21.15.45:0/2229933605 conn(0x7f5e5806b930 msgr2=0x7f5e580b2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:38.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.742+0000 7f5e5ce00640 1 -- 172.21.15.45:0/2229933605 shutdown_connections 2024-04-16T16:17:38.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:38.742+0000 7f5e5ce00640 1 -- 172.21.15.45:0/2229933605 wait complete. 2024-04-16T16:17:39.761 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:39.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.991+0000 7f11e0a00640 1 Processor -- start 2024-04-16T16:17:39.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.991+0000 7f11e0a00640 1 -- start start 2024-04-16T16:17:39.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.993+0000 7f11e0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11dc162090 0x7f11dc0cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:39.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.993+0000 7f11e0a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f11dc0d5520 0x7f11dc0cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:39.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.993+0000 7f11e0a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11dc0d2600 con 0x7f11dc0ba140 2024-04-16T16:17:39.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.993+0000 7f11e0a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f11dc0ba8b0 con 0x7f11dc0d5520 2024-04-16T16:17:39.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.993+0000 7f11e0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f11dc0ba5d0 con 0x7f11dc162090 2024-04-16T16:17:39.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.994+0000 7f11db400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11dc162090 0x7f11dc0cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:39.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.994+0000 7f11dbe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f11dc0d5520 0x7f11dc0cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:39.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.994+0000 7f11d2a00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f11dc0ba140 0x7f11dc1628a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:34248/0 (socket says 172.21.15.45:34248) 2024-04-16T16:17:39.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.994+0000 7f11d2a00640 1 -- 172.21.15.45:0/2950245643 learned_addr learned my addr 172.21.15.45:0/2950245643 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:39.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.995+0000 7f11dbe00640 1 -- 172.21.15.45:0/2950245643 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11dc162090 msgr2=0x7f11dc0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:39.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.995+0000 7f11dbe00640 1 --2- 172.21.15.45:0/2950245643 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11dc162090 0x7f11dc0cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:39.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.995+0000 7f11dbe00640 1 -- 172.21.15.45:0/2950245643 >> v1:172.21.15.103:6789/0 conn(0x7f11dc0ba140 legacy=0x7f11dc1628a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:39.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.995+0000 7f11dbe00640 1 -- 172.21.15.45:0/2950245643 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11dc0d28a0 con 0x7f11dc0d5520 2024-04-16T16:17:39.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.996+0000 7f11dbe00640 1 --2- 172.21.15.45:0/2950245643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11dc0d5520 0x7f11dc0cfa50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f11d40097a0 tx=0x7f11d402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96eab8393c211378 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:39.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.996+0000 7f11daa00640 1 -- 172.21.15.45:0/2950245643 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d4035020 con 0x7f11dc0d5520 2024-04-16T16:17:39.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.996+0000 7f11daa00640 1 -- 172.21.15.45:0/2950245643 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d4031810 con 0x7f11dc0d5520 2024-04-16T16:17:39.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.996+0000 7f11daa00640 1 -- 172.21.15.45:0/2950245643 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d4031be0 con 0x7f11dc0d5520 2024-04-16T16:17:39.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.997+0000 7f11e0a00640 1 -- 172.21.15.45:0/2950245643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11dc0d5520 msgr2=0x7f11dc0cfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:39.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.997+0000 7f11e0a00640 1 --2- 172.21.15.45:0/2950245643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11dc0d5520 0x7f11dc0cfa50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f11d40097a0 tx=0x7f11d402b8b0 comp rx=0 tx=0).stop 2024-04-16T16:17:39.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.997+0000 7f11e0a00640 1 -- 172.21.15.45:0/2950245643 shutdown_connections 2024-04-16T16:17:39.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.997+0000 7f11e0a00640 1 --2- 172.21.15.45:0/2950245643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11dc0d5520 0x7f11dc0cfa50 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:39.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.997+0000 7f11e0a00640 1 --2- 172.21.15.45:0/2950245643 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11dc162090 0x7f11dc0cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:39.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.997+0000 7f11e0a00640 1 -- 172.21.15.45:0/2950245643 >> 172.21.15.45:0/2950245643 conn(0x7f11dc0d1940 msgr2=0x7f11dc0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:39.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.997+0000 7f11e0a00640 1 -- 172.21.15.45:0/2950245643 shutdown_connections 2024-04-16T16:17:39.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.997+0000 7f11e0a00640 1 -- 172.21.15.45:0/2950245643 wait complete. 2024-04-16T16:17:39.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.999+0000 7f11e0a00640 1 Processor -- start 2024-04-16T16:17:39.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:39.999+0000 7f11e0a00640 1 -- start start 2024-04-16T16:17:40.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.000+0000 7f11e0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11dc0d5520 0x7f11dc0ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:40.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.000+0000 7f11e0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11dc162090 0x7f11dc1538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:40.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.000+0000 7f11e0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11dc0cad70 0x7f11dc153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:40.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.000+0000 7f11e0a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1625f0 con 0x7f11dc0cad70 2024-04-16T16:17:40.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.000+0000 7f11e0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc162470 con 0x7f11dc0d5520 2024-04-16T16:17:40.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.000+0000 7f11e0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f11dc0d40b0 con 0x7f11dc162090 2024-04-16T16:17:40.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.000+0000 7f11dbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11dc0cad70 0x7f11dc153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:40.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.001+0000 7f11dbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11dc0cad70 0x7f11dc153e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50236/0 (socket says 172.21.15.45:50236) 2024-04-16T16:17:40.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.001+0000 7f11dbe00640 1 -- 172.21.15.45:0/1387834801 learned_addr learned my addr 172.21.15.45:0/1387834801 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:40.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.001+0000 7f11d2a00640 1 --2- 172.21.15.45:0/1387834801 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11dc162090 0x7f11dc1538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:40.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.001+0000 7f11db400640 1 --2- 172.21.15.45:0/1387834801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11dc0d5520 0x7f11dc0ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:40.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.001+0000 7f11dbe00640 1 -- 172.21.15.45:0/1387834801 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11dc162090 msgr2=0x7f11dc1538d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:40.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.001+0000 7f11dbe00640 1 --2- 172.21.15.45:0/1387834801 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11dc162090 0x7f11dc1538d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:40.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.001+0000 7f11dbe00640 1 -- 172.21.15.45:0/1387834801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11dc0d5520 msgr2=0x7f11dc0ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:40.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.001+0000 7f11dbe00640 1 --2- 172.21.15.45:0/1387834801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11dc0d5520 0x7f11dc0ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:40.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.001+0000 7f11dbe00640 1 -- 172.21.15.45:0/1387834801 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11dc0c9790 con 0x7f11dc0cad70 2024-04-16T16:17:40.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.001+0000 7f11dbe00640 1 --2- 172.21.15.45:0/1387834801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11dc0cad70 0x7f11dc153e10 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f11d40098c0 tx=0x7f11d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:40.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.002+0000 7f11d3e00640 1 -- 172.21.15.45:0/1387834801 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d4041070 con 0x7f11dc0cad70 2024-04-16T16:17:40.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.002+0000 7f11d3e00640 1 -- 172.21.15.45:0/1387834801 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d403ac00 con 0x7f11dc0cad70 2024-04-16T16:17:40.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.002+0000 7f11d3e00640 1 -- 172.21.15.45:0/1387834801 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d4035040 con 0x7f11dc0cad70 2024-04-16T16:17:40.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.002+0000 7f11e0a00640 1 -- 172.21.15.45:0/1387834801 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11dc0c9a70 con 0x7f11dc0cad70 2024-04-16T16:17:40.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.002+0000 7f11e0a00640 1 -- 172.21.15.45:0/1387834801 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11dc0ca030 con 0x7f11dc0cad70 2024-04-16T16:17:40.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.003+0000 7f11e0a00640 1 -- 172.21.15.45:0/1387834801 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11dc075780 con 0x7f11dc0cad70 2024-04-16T16:17:40.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.004+0000 7f11d3e00640 1 -- 172.21.15.45:0/1387834801 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f11d4038020 con 0x7f11dc0cad70 2024-04-16T16:17:40.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.005+0000 7f11d3e00640 1 --2- 172.21.15.45:0/1387834801 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f11b806ca60 0x7f11b806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:40.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.005+0000 7f11d3e00640 1 -- 172.21.15.45:0/1387834801 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f11d40adbc0 con 0x7f11dc0cad70 2024-04-16T16:17:40.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.006+0000 7f11db400640 1 --2- 172.21.15.45:0/1387834801 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f11b806ca60 0x7f11b806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:40.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.007+0000 7f11db400640 1 --2- 172.21.15.45:0/1387834801 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f11b806ca60 0x7f11b806ef20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f11cc0073f0 tx=0x7f11cc00b040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:40.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.010+0000 7f11d3e00640 1 -- 172.21.15.45:0/1387834801 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f11d407bc00 con 0x7f11dc0cad70 2024-04-16T16:17:40.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.248+0000 7f11e0a00640 1 -- 172.21.15.45:0/1387834801 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f11dc0ca380 con 0x7f11b806ca60 2024-04-16T16:17:40.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.251+0000 7f11d3e00640 1 -- 172.21.15.45:0/1387834801 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3035 (secure 0 0 0) 0x7f11dc0ca380 con 0x7f11b806ca60 2024-04-16T16:17:40.250 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[0, 0], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], "client.5285": [[0, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[0, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[0, 0], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}, "mds.1": {"client.5243": []}}} 2024-04-16T16:17:40.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.254+0000 7f11e0a00640 1 -- 172.21.15.45:0/1387834801 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f11b806ca60 msgr2=0x7f11b806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:40.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.254+0000 7f11e0a00640 1 --2- 172.21.15.45:0/1387834801 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f11b806ca60 0x7f11b806ef20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f11cc0073f0 tx=0x7f11cc00b040 comp rx=0 tx=0).stop 2024-04-16T16:17:40.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.254+0000 7f11e0a00640 1 -- 172.21.15.45:0/1387834801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11dc0cad70 msgr2=0x7f11dc153e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:40.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.254+0000 7f11e0a00640 1 --2- 172.21.15.45:0/1387834801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11dc0cad70 0x7f11dc153e10 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f11d40098c0 tx=0x7f11d40046c0 comp rx=0 tx=0).stop 2024-04-16T16:17:40.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.254+0000 7f11e0a00640 1 -- 172.21.15.45:0/1387834801 shutdown_connections 2024-04-16T16:17:40.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.254+0000 7f11e0a00640 1 --2- 172.21.15.45:0/1387834801 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f11b806ca60 0x7f11b806ef20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:40.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.254+0000 7f11e0a00640 1 --2- 172.21.15.45:0/1387834801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11dc0cad70 0x7f11dc153e10 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:40.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.254+0000 7f11e0a00640 1 --2- 172.21.15.45:0/1387834801 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11dc162090 0x7f11dc1538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:40.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.254+0000 7f11e0a00640 1 --2- 172.21.15.45:0/1387834801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11dc0d5520 0x7f11dc0ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:40.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.254+0000 7f11e0a00640 1 -- 172.21.15.45:0/1387834801 >> 172.21.15.45:0/1387834801 conn(0x7f11dc0d1940 msgr2=0x7f11dc0b9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:40.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.254+0000 7f11e0a00640 1 -- 172.21.15.45:0/1387834801 shutdown_connections 2024-04-16T16:17:40.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:40.254+0000 7f11e0a00640 1 -- 172.21.15.45:0/1387834801 wait complete. 2024-04-16T16:17:41.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:41.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.505+0000 7f530fe00640 1 Processor -- start 2024-04-16T16:17:41.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.505+0000 7f530fe00640 1 -- start start 2024-04-16T16:17:41.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.507+0000 7f530fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f53100c6fc0 0x7f531006f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:41.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.507+0000 7f530fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f531015cdc0 0x7f53100c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:41.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.507+0000 7f530fe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f531015d600 con 0x7f53100c6320 2024-04-16T16:17:41.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.507+0000 7f530fe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f53100c5670 con 0x7f53100c6fc0 2024-04-16T16:17:41.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.507+0000 7f530fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f531015d8e0 con 0x7f531015cdc0 2024-04-16T16:17:41.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.507+0000 7f530f400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f53100c6320 0x7f531006df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36480/0 (socket says 172.21.15.45:36480) 2024-04-16T16:17:41.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.507+0000 7f530f400640 1 -- 172.21.15.45:0/4247270150 learned_addr learned my addr 172.21.15.45:0/4247270150 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:41.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.507+0000 7f530ea00640 1 --2- 172.21.15.45:0/4247270150 >> v2:172.21.15.45:3300/0 conn(0x7f53100c6fc0 0x7f531006f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:41.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.508+0000 7f530e000640 1 --2- 172.21.15.45:0/4247270150 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f531015cdc0 0x7f53100c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:41.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.508+0000 7f530d600640 1 -- 172.21.15.45:0/4247270150 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 794494365 0 0) 0x7f531015d600 con 0x7f53100c6320 2024-04-16T16:17:41.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.508+0000 7f530ea00640 1 -- 172.21.15.45:0/4247270150 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f531015cdc0 msgr2=0x7f53100c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:41.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.508+0000 7f530ea00640 1 --2- 172.21.15.45:0/4247270150 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f531015cdc0 0x7f53100c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:41.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.508+0000 7f530ea00640 1 -- 172.21.15.45:0/4247270150 >> v1:172.21.15.45:6789/0 conn(0x7f53100c6320 legacy=0x7f531006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:41.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.508+0000 7f530ea00640 1 -- 172.21.15.45:0/4247270150 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f531015dbc0 con 0x7f53100c6fc0 2024-04-16T16:17:41.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.509+0000 7f530ea00640 1 --2- 172.21.15.45:0/4247270150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53100c6fc0 0x7f531006f730 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f53000097a0 tx=0x7f530002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=589b8eca370a4786 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:41.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.509+0000 7f530d600640 1 -- 172.21.15.45:0/4247270150 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300035020 con 0x7f53100c6fc0 2024-04-16T16:17:41.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.509+0000 7f530d600640 1 -- 172.21.15.45:0/4247270150 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5300031830 con 0x7f53100c6fc0 2024-04-16T16:17:41.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.509+0000 7f530d600640 1 -- 172.21.15.45:0/4247270150 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300031c00 con 0x7f53100c6fc0 2024-04-16T16:17:41.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.510+0000 7f530fe00640 1 -- 172.21.15.45:0/4247270150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53100c6fc0 msgr2=0x7f531006f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:41.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.510+0000 7f530fe00640 1 --2- 172.21.15.45:0/4247270150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53100c6fc0 0x7f531006f730 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f53000097a0 tx=0x7f530002b8d0 comp rx=0 tx=0).stop 2024-04-16T16:17:41.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.510+0000 7f530fe00640 1 -- 172.21.15.45:0/4247270150 shutdown_connections 2024-04-16T16:17:41.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.510+0000 7f530fe00640 1 --2- 172.21.15.45:0/4247270150 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f531015cdc0 0x7f53100c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:41.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.510+0000 7f530fe00640 1 --2- 172.21.15.45:0/4247270150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53100c6fc0 0x7f531006f730 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:41.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.510+0000 7f530fe00640 1 -- 172.21.15.45:0/4247270150 >> 172.21.15.45:0/4247270150 conn(0x7f531006b930 msgr2=0x7f531006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:41.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.510+0000 7f530fe00640 1 -- 172.21.15.45:0/4247270150 shutdown_connections 2024-04-16T16:17:41.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.510+0000 7f530fe00640 1 -- 172.21.15.45:0/4247270150 wait complete. 2024-04-16T16:17:41.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.512+0000 7f530fe00640 1 Processor -- start 2024-04-16T16:17:41.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.512+0000 7f530fe00640 1 -- start start 2024-04-16T16:17:41.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.513+0000 7f530fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53100c6320 0x7f53100b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:41.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.513+0000 7f530fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53100c6fc0 0x7f5310146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:41.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.513+0000 7f530fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f531015cdc0 0x7f5310146fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:41.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.513+0000 7f530fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53100c73a0 con 0x7f53100c6fc0 2024-04-16T16:17:41.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.513+0000 7f530fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f531015d270 con 0x7f53100c6320 2024-04-16T16:17:41.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.513+0000 7f530fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f53100c7520 con 0x7f531015cdc0 2024-04-16T16:17:41.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.513+0000 7f530e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53100c6fc0 0x7f5310146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:41.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.513+0000 7f530e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53100c6fc0 0x7f5310146a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50250/0 (socket says 172.21.15.45:50250) 2024-04-16T16:17:41.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.513+0000 7f530e000640 1 -- 172.21.15.45:0/204557540 learned_addr learned my addr 172.21.15.45:0/204557540 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:41.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.514+0000 7f530f400640 1 --2- 172.21.15.45:0/204557540 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f531015cdc0 0x7f5310146fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:41.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.514+0000 7f530ea00640 1 --2- 172.21.15.45:0/204557540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53100c6320 0x7f53100b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:41.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.514+0000 7f530e000640 1 -- 172.21.15.45:0/204557540 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f531015cdc0 msgr2=0x7f5310146fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:41.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.514+0000 7f530e000640 1 --2- 172.21.15.45:0/204557540 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f531015cdc0 0x7f5310146fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:41.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.514+0000 7f530e000640 1 -- 172.21.15.45:0/204557540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53100c6320 msgr2=0x7f53100b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:41.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.514+0000 7f530e000640 1 --2- 172.21.15.45:0/204557540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53100c6320 0x7f53100b48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:41.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.514+0000 7f530e000640 1 -- 172.21.15.45:0/204557540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5310141e00 con 0x7f53100c6fc0 2024-04-16T16:17:41.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.514+0000 7f530e000640 1 --2- 172.21.15.45:0/204557540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53100c6fc0 0x7f5310146a70 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00d620 tx=0x7f52fc00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:41.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.515+0000 7f5306a00640 1 -- 172.21.15.45:0/204557540 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc00b620 con 0x7f53100c6fc0 2024-04-16T16:17:41.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.515+0000 7f5306a00640 1 -- 172.21.15.45:0/204557540 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52fc019070 con 0x7f53100c6fc0 2024-04-16T16:17:41.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.515+0000 7f5306a00640 1 -- 172.21.15.45:0/204557540 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc00ba00 con 0x7f53100c6fc0 2024-04-16T16:17:41.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.515+0000 7f530fe00640 1 -- 172.21.15.45:0/204557540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53101420e0 con 0x7f53100c6fc0 2024-04-16T16:17:41.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.515+0000 7f530fe00640 1 -- 172.21.15.45:0/204557540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53101426a0 con 0x7f53100c6fc0 2024-04-16T16:17:41.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.516+0000 7f530fe00640 1 -- 172.21.15.45:0/204557540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5310075780 con 0x7f53100c6fc0 2024-04-16T16:17:41.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.517+0000 7f5306a00640 1 -- 172.21.15.45:0/204557540 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f52fc00bba0 con 0x7f53100c6fc0 2024-04-16T16:17:41.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.518+0000 7f5306a00640 1 --2- 172.21.15.45:0/204557540 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f52f006ca90 0x7f52f006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:41.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.518+0000 7f5306a00640 1 -- 172.21.15.45:0/204557540 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f52fc08c370 con 0x7f53100c6fc0 2024-04-16T16:17:41.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.519+0000 7f530ea00640 1 --2- 172.21.15.45:0/204557540 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f52f006ca90 0x7f52f006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:41.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.520+0000 7f530ea00640 1 --2- 172.21.15.45:0/204557540 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f52f006ca90 0x7f52f006ef50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5300036000 tx=0x7f5300008030 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:41.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.523+0000 7f5306a00640 1 -- 172.21.15.45:0/204557540 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f52fc05a440 con 0x7f53100c6fc0 2024-04-16T16:17:41.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.757+0000 7f530fe00640 1 -- 172.21.15.45:0/204557540 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f531006cf70 con 0x7f52f006ca90 2024-04-16T16:17:41.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.761+0000 7f5306a00640 1 -- 172.21.15.45:0/204557540 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3035 (secure 0 0 0) 0x7f531006cf70 con 0x7f52f006ca90 2024-04-16T16:17:41.760 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[0, 0], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], "client.5285": [[0, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[0, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[0, 0], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}, "mds.1": {"client.5243": []}}} 2024-04-16T16:17:41.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.763+0000 7f530fe00640 1 -- 172.21.15.45:0/204557540 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f52f006ca90 msgr2=0x7f52f006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:41.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.763+0000 7f530fe00640 1 --2- 172.21.15.45:0/204557540 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f52f006ca90 0x7f52f006ef50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5300036000 tx=0x7f5300008030 comp rx=0 tx=0).stop 2024-04-16T16:17:41.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.763+0000 7f530fe00640 1 -- 172.21.15.45:0/204557540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53100c6fc0 msgr2=0x7f5310146a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:41.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.763+0000 7f530fe00640 1 --2- 172.21.15.45:0/204557540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53100c6fc0 0x7f5310146a70 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00d620 tx=0x7f52fc00dae0 comp rx=0 tx=0).stop 2024-04-16T16:17:41.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.764+0000 7f530fe00640 1 -- 172.21.15.45:0/204557540 shutdown_connections 2024-04-16T16:17:41.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.764+0000 7f530fe00640 1 --2- 172.21.15.45:0/204557540 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f52f006ca90 0x7f52f006ef50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:41.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.764+0000 7f530fe00640 1 --2- 172.21.15.45:0/204557540 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f531015cdc0 0x7f5310146fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:41.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.764+0000 7f530fe00640 1 --2- 172.21.15.45:0/204557540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53100c6fc0 0x7f5310146a70 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:41.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.764+0000 7f530fe00640 1 --2- 172.21.15.45:0/204557540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53100c6320 0x7f53100b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:41.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.764+0000 7f530fe00640 1 -- 172.21.15.45:0/204557540 >> 172.21.15.45:0/204557540 conn(0x7f531006b930 msgr2=0x7f53100cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:41.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.764+0000 7f530fe00640 1 -- 172.21.15.45:0/204557540 shutdown_connections 2024-04-16T16:17:41.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:41.764+0000 7f530fe00640 1 -- 172.21.15.45:0/204557540 wait complete. 2024-04-16T16:17:42.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:43.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.017+0000 7f28d8800640 1 Processor -- start 2024-04-16T16:17:43.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.018+0000 7f28d8800640 1 -- start start 2024-04-16T16:17:43.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.019+0000 7f28d8800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f28d00afe20 0x7f28d006f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:43.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.019+0000 7f28d8800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f28d00cdfe0 0x7f28d00b4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:43.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.019+0000 7f28d8800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28d006fab0 con 0x7f28d006f550 2024-04-16T16:17:43.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.019+0000 7f28d8800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f28d015bf20 con 0x7f28d00cdfe0 2024-04-16T16:17:43.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.019+0000 7f28d8800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f28d015c200 con 0x7f28d00afe20 2024-04-16T16:17:43.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.020+0000 7f28d6800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f28d00cdfe0 0x7f28d00b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:43.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.020+0000 7f28d6800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f28d00cdfe0 0x7f28d00b4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50252/0 (socket says 172.21.15.45:50252) 2024-04-16T16:17:43.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.020+0000 7f28d6800640 1 -- 172.21.15.45:0/4085869062 learned_addr learned my addr 172.21.15.45:0/4085869062 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:43.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.020+0000 7f28d6800640 1 -- 172.21.15.45:0/4085869062 >> v2:172.21.15.103:3300/0 conn(0x7f28d00afe20 msgr2=0x7f28d006f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:43.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.020+0000 7f28d6800640 1 --2- 172.21.15.45:0/4085869062 >> v2:172.21.15.103:3300/0 conn(0x7f28d00afe20 0x7f28d006f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.020+0000 7f28d6800640 1 -- 172.21.15.45:0/4085869062 >> v1:172.21.15.103:6789/0 conn(0x7f28d006f550 legacy=0x7f28d00cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:43.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.020+0000 7f28d6800640 1 -- 172.21.15.45:0/4085869062 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d015c4e0 con 0x7f28d00cdfe0 2024-04-16T16:17:43.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.021+0000 7f28d6800640 1 --2- 172.21.15.45:0/4085869062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28d00cdfe0 0x7f28d00b4160 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f28d0070350 tx=0x7f28cc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f557bcbad0f3cb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:43.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.022+0000 7f28d4a00640 1 -- 172.21.15.45:0/4085869062 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc02ca80 con 0x7f28d00cdfe0 2024-04-16T16:17:43.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.022+0000 7f28d4a00640 1 -- 172.21.15.45:0/4085869062 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28cc02cc20 con 0x7f28d00cdfe0 2024-04-16T16:17:43.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.022+0000 7f28d4a00640 1 -- 172.21.15.45:0/4085869062 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc0343f0 con 0x7f28d00cdfe0 2024-04-16T16:17:43.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.022+0000 7f28d8800640 1 -- 172.21.15.45:0/4085869062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28d00cdfe0 msgr2=0x7f28d00b4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:43.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.022+0000 7f28d8800640 1 --2- 172.21.15.45:0/4085869062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28d00cdfe0 0x7f28d00b4160 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f28d0070350 tx=0x7f28cc02bee0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.023+0000 7f28d8800640 1 -- 172.21.15.45:0/4085869062 shutdown_connections 2024-04-16T16:17:43.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.023+0000 7f28d8800640 1 --2- 172.21.15.45:0/4085869062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28d00cdfe0 0x7f28d00b4160 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.023+0000 7f28d8800640 1 --2- 172.21.15.45:0/4085869062 >> v2:172.21.15.103:3300/0 conn(0x7f28d00afe20 0x7f28d006f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.023+0000 7f28d8800640 1 -- 172.21.15.45:0/4085869062 >> 172.21.15.45:0/4085869062 conn(0x7f28d006b930 msgr2=0x7f28d006bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:43.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.023+0000 7f28d8800640 1 -- 172.21.15.45:0/4085869062 shutdown_connections 2024-04-16T16:17:43.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.023+0000 7f28d8800640 1 -- 172.21.15.45:0/4085869062 wait complete. 2024-04-16T16:17:43.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.024+0000 7f28d8800640 1 Processor -- start 2024-04-16T16:17:43.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.025+0000 7f28d8800640 1 -- start start 2024-04-16T16:17:43.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.025+0000 7f28d8800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28d006f550 0x7f28d014a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:43.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.025+0000 7f28d8800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28d00afe20 0x7f28d0154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:43.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.025+0000 7f28d8800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28d00cdfe0 0x7f28d00ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:43.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.025+0000 7f28d8800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f28d00b59d0 con 0x7f28d00cdfe0 2024-04-16T16:17:43.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.025+0000 7f28d8800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f28d00b5850 con 0x7f28d006f550 2024-04-16T16:17:43.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.025+0000 7f28d8800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f28d015ca00 con 0x7f28d00afe20 2024-04-16T16:17:43.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.026+0000 7f28d6800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28d00cdfe0 0x7f28d00ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:43.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.026+0000 7f28d6800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28d00cdfe0 0x7f28d00ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50258/0 (socket says 172.21.15.45:50258) 2024-04-16T16:17:43.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.026+0000 7f28d6800640 1 -- 172.21.15.45:0/2287747333 learned_addr learned my addr 172.21.15.45:0/2287747333 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:43.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.026+0000 7f28d6800640 1 -- 172.21.15.45:0/2287747333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28d00afe20 msgr2=0x7f28d0154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:43.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.026+0000 7f28d6800640 1 --2- 172.21.15.45:0/2287747333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28d00afe20 0x7f28d0154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.026+0000 7f28d6800640 1 -- 172.21.15.45:0/2287747333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28d006f550 msgr2=0x7f28d014a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:43.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.026+0000 7f28d5e00640 1 --2- 172.21.15.45:0/2287747333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28d006f550 0x7f28d014a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:43.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.026+0000 7f28d6800640 1 --2- 172.21.15.45:0/2287747333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28d006f550 0x7f28d014a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.026+0000 7f28d6800640 1 -- 172.21.15.45:0/2287747333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d0154e90 con 0x7f28d00cdfe0 2024-04-16T16:17:43.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.026+0000 7f28d5e00640 1 --2- 172.21.15.45:0/2287747333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28d006f550 0x7f28d014a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:43.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.027+0000 7f28d6800640 1 --2- 172.21.15.45:0/2287747333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28d00cdfe0 0x7f28d00ad8e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009f40 tx=0x7f28cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:43.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.027+0000 7f28ca000640 1 -- 172.21.15.45:0/2287747333 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc043020 con 0x7f28d00cdfe0 2024-04-16T16:17:43.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.027+0000 7f28ca000640 1 -- 172.21.15.45:0/2287747333 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28cc03e450 con 0x7f28d00cdfe0 2024-04-16T16:17:43.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.027+0000 7f28ca000640 1 -- 172.21.15.45:0/2287747333 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc03e840 con 0x7f28d00cdfe0 2024-04-16T16:17:43.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.027+0000 7f28d8800640 1 -- 172.21.15.45:0/2287747333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28d0155170 con 0x7f28d00cdfe0 2024-04-16T16:17:43.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.027+0000 7f28d8800640 1 -- 172.21.15.45:0/2287747333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28d00b2610 con 0x7f28d00cdfe0 2024-04-16T16:17:43.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.028+0000 7f28d8800640 1 -- 172.21.15.45:0/2287747333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28d0075780 con 0x7f28d00cdfe0 2024-04-16T16:17:43.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.029+0000 7f28ca000640 1 -- 172.21.15.45:0/2287747333 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f28cc046020 con 0x7f28d00cdfe0 2024-04-16T16:17:43.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.031+0000 7f28ca000640 1 --2- 172.21.15.45:0/2287747333 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f28b406ca60 0x7f28b406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:43.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.031+0000 7f28ca000640 1 -- 172.21.15.45:0/2287747333 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f28cc0ae870 con 0x7f28d00cdfe0 2024-04-16T16:17:43.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.031+0000 7f28d5e00640 1 --2- 172.21.15.45:0/2287747333 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f28b406ca60 0x7f28b406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:43.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.033+0000 7f28d5e00640 1 --2- 172.21.15.45:0/2287747333 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f28b406ca60 0x7f28b406ef20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f28c4006fd0 tx=0x7f28c4008040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:43.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.035+0000 7f28ca000640 1 -- 172.21.15.45:0/2287747333 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f28cc04d030 con 0x7f28d00cdfe0 2024-04-16T16:17:43.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.268+0000 7f28d8800640 1 -- 172.21.15.45:0/2287747333 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f28d014ad80 con 0x7f28b406ca60 2024-04-16T16:17:43.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.271+0000 7f28ca000640 1 -- 172.21.15.45:0/2287747333 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3036 (secure 0 0 0) 0x7f28d014ad80 con 0x7f28b406ca60 2024-04-16T16:17:43.270 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[0, 0], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], "client.5285": [[0, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[0, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[0, 0], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, "metrics": {"delayed_ranks": [1], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}, "mds.1": {"client.5243": []}}} 2024-04-16T16:17:43.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.274+0000 7f28d8800640 1 -- 172.21.15.45:0/2287747333 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f28b406ca60 msgr2=0x7f28b406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:43.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.274+0000 7f28d8800640 1 --2- 172.21.15.45:0/2287747333 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f28b406ca60 0x7f28b406ef20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f28c4006fd0 tx=0x7f28c4008040 comp rx=0 tx=0).stop 2024-04-16T16:17:43.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.274+0000 7f28d8800640 1 -- 172.21.15.45:0/2287747333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28d00cdfe0 msgr2=0x7f28d00ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:43.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.274+0000 7f28d8800640 1 --2- 172.21.15.45:0/2287747333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28d00cdfe0 0x7f28d00ad8e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009f40 tx=0x7f28cc004a40 comp rx=0 tx=0).stop 2024-04-16T16:17:43.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.274+0000 7f28d8800640 1 -- 172.21.15.45:0/2287747333 shutdown_connections 2024-04-16T16:17:43.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.274+0000 7f28d8800640 1 --2- 172.21.15.45:0/2287747333 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f28b406ca60 0x7f28b406ef20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.274+0000 7f28d8800640 1 --2- 172.21.15.45:0/2287747333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28d00cdfe0 0x7f28d00ad8e0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.274+0000 7f28d8800640 1 --2- 172.21.15.45:0/2287747333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28d00afe20 0x7f28d0154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.274+0000 7f28d8800640 1 --2- 172.21.15.45:0/2287747333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28d006f550 0x7f28d014a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.274+0000 7f28d8800640 1 -- 172.21.15.45:0/2287747333 >> 172.21.15.45:0/2287747333 conn(0x7f28d006b930 msgr2=0x7f28d00b4970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:43.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.274+0000 7f28d8800640 1 -- 172.21.15.45:0/2287747333 shutdown_connections 2024-04-16T16:17:43.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.274+0000 7f28d8800640 1 -- 172.21.15.45:0/2287747333 wait complete. 2024-04-16T16:17:43.295 DEBUG:tasks.cephfs.test_mds_metrics:metrics={'version': 2, 'global_counters': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'counters': [], 'client_metadata': {'cephfs': {'client.5243': {'hostname': 'smithi045', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '172.21.15.45'}, 'client.5285': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'IP': '172.21.15.103'}, 'client.5288': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'IP': '172.21.15.103'}, 'client.6591': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '172.21.15.103'}}}, 'global_metrics': {'cephfs': {'client.5243': [[0, 0], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], 'client.5285': [[0, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], 'client.5288': [[0, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], 'client.6591': [[0, 0], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, 'metrics': {'delayed_ranks': [1], 'mds.0': {'client.5243': [], 'client.5285': [], 'client.5288': [], 'client.6591': []}, 'mds.1': {'client.5243': []}}} 2024-04-16T16:17:43.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:17:43.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.522+0000 7fd669000640 1 Processor -- start 2024-04-16T16:17:43.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.523+0000 7fd669000640 1 -- start start 2024-04-16T16:17:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.524+0000 7fd669000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd664071a40 0x7fd6640b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.524+0000 7fd669000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6640b7750 0x7fd66415cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.524+0000 7fd669000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6641604d0 con 0x7fd664071e20 2024-04-16T16:17:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.524+0000 7fd669000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd6641607b0 con 0x7fd664071a40 2024-04-16T16:17:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.524+0000 7fd669000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd664160a90 con 0x7fd6640b7750 2024-04-16T16:17:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.525+0000 7fd663400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd664071a40 0x7fd6640b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.525+0000 7fd663400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd664071a40 0x7fd6640b56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48742/0 (socket says 172.21.15.45:48742) 2024-04-16T16:17:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.525+0000 7fd663400640 1 -- 172.21.15.45:0/1769176960 learned_addr learned my addr 172.21.15.45:0/1769176960 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:43.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.525+0000 7fd662a00640 1 --2- 172.21.15.45:0/1769176960 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6640b7750 0x7fd66415cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:43.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.525+0000 7fd663400640 1 -- 172.21.15.45:0/1769176960 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6640b7750 msgr2=0x7fd66415cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:43.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.525+0000 7fd663400640 1 --2- 172.21.15.45:0/1769176960 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6640b7750 0x7fd66415cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.525+0000 7fd663400640 1 -- 172.21.15.45:0/1769176960 >> v1:172.21.15.103:6789/0 conn(0x7fd664071e20 legacy=0x7fd6640cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:43.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.525+0000 7fd663400640 1 -- 172.21.15.45:0/1769176960 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd664160d70 con 0x7fd664071a40 2024-04-16T16:17:43.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.526+0000 7fd663400640 1 --2- 172.21.15.45:0/1769176960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd664071a40 0x7fd6640b56a0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd654009870 tx=0x7fd65402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=643b511153a2d0f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:43.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.527+0000 7fd662000640 1 -- 172.21.15.45:0/1769176960 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654035020 con 0x7fd664071a40 2024-04-16T16:17:43.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.527+0000 7fd662000640 1 -- 172.21.15.45:0/1769176960 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd654031640 con 0x7fd664071a40 2024-04-16T16:17:43.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.527+0000 7fd662000640 1 -- 172.21.15.45:0/1769176960 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65402d040 con 0x7fd664071a40 2024-04-16T16:17:43.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.527+0000 7fd669000640 1 -- 172.21.15.45:0/1769176960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd664071a40 msgr2=0x7fd6640b56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:43.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.527+0000 7fd669000640 1 --2- 172.21.15.45:0/1769176960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd664071a40 0x7fd6640b56a0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd654009870 tx=0x7fd65402b8f0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.528+0000 7fd669000640 1 -- 172.21.15.45:0/1769176960 shutdown_connections 2024-04-16T16:17:43.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.528+0000 7fd669000640 1 --2- 172.21.15.45:0/1769176960 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6640b7750 0x7fd66415cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.528+0000 7fd669000640 1 --2- 172.21.15.45:0/1769176960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd664071a40 0x7fd6640b56a0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.528+0000 7fd669000640 1 -- 172.21.15.45:0/1769176960 >> 172.21.15.45:0/1769176960 conn(0x7fd6640b43a0 msgr2=0x7fd6640b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:43.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.528+0000 7fd669000640 1 -- 172.21.15.45:0/1769176960 shutdown_connections 2024-04-16T16:17:43.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.528+0000 7fd669000640 1 -- 172.21.15.45:0/1769176960 wait complete. 2024-04-16T16:17:43.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.529+0000 7fd669000640 1 Processor -- start 2024-04-16T16:17:43.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.530+0000 7fd669000640 1 -- start start 2024-04-16T16:17:43.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.531+0000 7fd669000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd664071a40 0x7fd66414f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:43.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.531+0000 7fd669000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd664071e20 0x7fd66414f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:43.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.531+0000 7fd669000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6640b7750 0x7fd66415c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:43.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.531+0000 7fd669000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd664072b20 con 0x7fd664071e20 2024-04-16T16:17:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.531+0000 7fd669000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd66415a670 con 0x7fd664071a40 2024-04-16T16:17:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.531+0000 7fd669000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd6640722d0 con 0x7fd6640b7750 2024-04-16T16:17:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.531+0000 7fd662a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd664071e20 0x7fd66414f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.531+0000 7fd662a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd664071e20 0x7fd66414f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48756/0 (socket says 172.21.15.45:48756) 2024-04-16T16:17:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.531+0000 7fd662a00640 1 -- 172.21.15.45:0/2176943872 learned_addr learned my addr 172.21.15.45:0/2176943872 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.531+0000 7fd663400640 1 --2- 172.21.15.45:0/2176943872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd664071a40 0x7fd66414f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.531+0000 7fd663e00640 1 --2- 172.21.15.45:0/2176943872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6640b7750 0x7fd66415c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.531+0000 7fd662a00640 1 -- 172.21.15.45:0/2176943872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6640b7750 msgr2=0x7fd66415c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.531+0000 7fd662a00640 1 --2- 172.21.15.45:0/2176943872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6640b7750 0x7fd66415c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.531+0000 7fd662a00640 1 -- 172.21.15.45:0/2176943872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd664071a40 msgr2=0x7fd66414f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.531+0000 7fd662a00640 1 --2- 172.21.15.45:0/2176943872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd664071a40 0x7fd66414f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.532+0000 7fd662a00640 1 -- 172.21.15.45:0/2176943872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66414ff20 con 0x7fd664071e20 2024-04-16T16:17:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.532+0000 7fd663e00640 1 --2- 172.21.15.45:0/2176943872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6640b7750 0x7fd66415c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.532+0000 7fd662a00640 1 --2- 172.21.15.45:0/2176943872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd664071e20 0x7fd66414f9e0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00c7c0 tx=0x7fd64c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.532+0000 7fd65b400640 1 -- 172.21.15.45:0/2176943872 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c015070 con 0x7fd664071e20 2024-04-16T16:17:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.532+0000 7fd65b400640 1 -- 172.21.15.45:0/2176943872 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd64c007820 con 0x7fd664071e20 2024-04-16T16:17:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.532+0000 7fd65b400640 1 -- 172.21.15.45:0/2176943872 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c007bd0 con 0x7fd664071e20 2024-04-16T16:17:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.532+0000 7fd669000640 1 -- 172.21.15.45:0/2176943872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd66415bd30 con 0x7fd664071e20 2024-04-16T16:17:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.532+0000 7fd669000640 1 -- 172.21.15.45:0/2176943872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd66406cd20 con 0x7fd664071e20 2024-04-16T16:17:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.534+0000 7fd669000640 1 -- 172.21.15.45:0/2176943872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6640b4f50 con 0x7fd664071e20 2024-04-16T16:17:43.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.535+0000 7fd65b400640 1 -- 172.21.15.45:0/2176943872 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7fd64c018070 con 0x7fd664071e20 2024-04-16T16:17:43.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.536+0000 7fd65b400640 1 --2- 172.21.15.45:0/2176943872 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd64006ca60 0x7fd64006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:43.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.536+0000 7fd65b400640 1 -- 172.21.15.45:0/2176943872 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7fd64c08ba30 con 0x7fd664071e20 2024-04-16T16:17:43.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.537+0000 7fd663400640 1 --2- 172.21.15.45:0/2176943872 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd64006ca60 0x7fd64006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:43.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.538+0000 7fd663400640 1 --2- 172.21.15.45:0/2176943872 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd64006ca60 0x7fd64006ef20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd654036000 tx=0x7fd65403c040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:43.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.540+0000 7fd65b400640 1 -- 172.21.15.45:0/2176943872 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd64c059b20 con 0x7fd664071e20 2024-04-16T16:17:43.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.785+0000 7fd669000640 1 -- 172.21.15.45:0/2176943872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd664075350 con 0x7fd664071e20 2024-04-16T16:17:43.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.786+0000 7fd65b400640 1 -- 172.21.15.45:0/2176943872 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5201 (secure 0 0 0) 0x7fd64c059420 con 0x7fd664071e20 2024-04-16T16:17:43.786 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:43.786 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":54,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:17:23.689033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6402,"mds_1":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"c","rank":1,"incarnation":52,"state":"up:active","state_seq":11,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402,6399]},"id":5}]} 2024-04-16T16:17:43.786 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 54 2024-04-16T16:17:43.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.789+0000 7fd669000640 1 -- 172.21.15.45:0/2176943872 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd64006ca60 msgr2=0x7fd64006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:43.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.789+0000 7fd669000640 1 --2- 172.21.15.45:0/2176943872 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd64006ca60 0x7fd64006ef20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd654036000 tx=0x7fd65403c040 comp rx=0 tx=0).stop 2024-04-16T16:17:43.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.789+0000 7fd669000640 1 -- 172.21.15.45:0/2176943872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd664071e20 msgr2=0x7fd66414f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:43.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.789+0000 7fd669000640 1 --2- 172.21.15.45:0/2176943872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd664071e20 0x7fd66414f9e0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00c7c0 tx=0x7fd64c00cc80 comp rx=0 tx=0).stop 2024-04-16T16:17:43.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.789+0000 7fd669000640 1 -- 172.21.15.45:0/2176943872 shutdown_connections 2024-04-16T16:17:43.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.789+0000 7fd669000640 1 --2- 172.21.15.45:0/2176943872 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd64006ca60 0x7fd64006ef20 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd654036000 tx=0x7fd65403c040 comp rx=0 tx=0).stop 2024-04-16T16:17:43.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.789+0000 7fd669000640 1 --2- 172.21.15.45:0/2176943872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6640b7750 0x7fd66415c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.789+0000 7fd669000640 1 --2- 172.21.15.45:0/2176943872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd664071e20 0x7fd66414f9e0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.789+0000 7fd669000640 1 --2- 172.21.15.45:0/2176943872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd664071a40 0x7fd66414f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:43.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.789+0000 7fd669000640 1 -- 172.21.15.45:0/2176943872 >> 172.21.15.45:0/2176943872 conn(0x7fd6640b43a0 msgr2=0x7fd6640cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:43.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.790+0000 7fd669000640 1 -- 172.21.15.45:0/2176943872 shutdown_connections 2024-04-16T16:17:43.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:43.790+0000 7fd669000640 1 -- 172.21.15.45:0/2176943872 wait complete. 2024-04-16T16:17:43.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -D INPUT -p tcp --dport 6833 -j REJECT -m comment --comment teuthology 2024-04-16T16:17:43.932 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -D INPUT -p tcp --dport 6835 -j REJECT -m comment --comment teuthology 2024-04-16T16:17:44.052 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -D OUTPUT -p tcp --sport 6832 -j REJECT -m comment --comment teuthology 2024-04-16T16:17:44.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -D INPUT -p tcp --dport 6832 -j REJECT -m comment --comment teuthology 2024-04-16T16:17:44.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -D OUTPUT -p tcp --sport 6834 -j REJECT -m comment --comment teuthology 2024-04-16T16:17:44.391 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -D INPUT -p tcp --dport 6834 -j REJECT -m comment --comment teuthology 2024-04-16T16:17:44.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:44.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.736+0000 7fd265600640 1 Processor -- start 2024-04-16T16:17:44.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.737+0000 7fd265600640 1 -- start start 2024-04-16T16:17:44.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.738+0000 7fd265600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd260145230 0x7fd26015bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:44.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.738+0000 7fd265600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd2600c5670 0x7fd2600c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:44.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.738+0000 7fd265600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd26015d3b0 con 0x7fd26015cbe0 2024-04-16T16:17:44.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.738+0000 7fd265600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd2600c6020 con 0x7fd2600c5670 2024-04-16T16:17:44.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.738+0000 7fd265600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd26015dfc0 con 0x7fd260145230 2024-04-16T16:17:44.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.739+0000 7fd25f400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd26015cbe0 0x7fd2600cd0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59276/0 (socket says 172.21.15.45:59276) 2024-04-16T16:17:44.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.739+0000 7fd25f400640 1 -- 172.21.15.45:0/2392873001 learned_addr learned my addr 172.21.15.45:0/2392873001 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:44.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.739+0000 7fd264c00640 1 --2- 172.21.15.45:0/2392873001 >> v2:172.21.15.45:3300/0 conn(0x7fd2600c5670 0x7fd2600c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:44.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.739+0000 7fd25fe00640 1 --2- 172.21.15.45:0/2392873001 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd260145230 0x7fd26015bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:44.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.739+0000 7fd264c00640 1 -- 172.21.15.45:0/2392873001 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd260145230 msgr2=0x7fd26015bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:44.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.739+0000 7fd264c00640 1 --2- 172.21.15.45:0/2392873001 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd260145230 0x7fd26015bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:44.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.739+0000 7fd264c00640 1 -- 172.21.15.45:0/2392873001 >> v1:172.21.15.45:6789/0 conn(0x7fd26015cbe0 legacy=0x7fd2600cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:44.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.739+0000 7fd264c00640 1 -- 172.21.15.45:0/2392873001 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd26015d690 con 0x7fd2600c5670 2024-04-16T16:17:44.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.741+0000 7fd264c00640 1 --2- 172.21.15.45:0/2392873001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2600c5670 0x7fd2600c5ae0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fd258009d90 tx=0x7fd25802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54d2a7c1d0be21af server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:44.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.741+0000 7fd25ea00640 1 -- 172.21.15.45:0/2392873001 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd258039070 con 0x7fd2600c5670 2024-04-16T16:17:44.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.741+0000 7fd25ea00640 1 -- 172.21.15.45:0/2392873001 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd25802cbc0 con 0x7fd2600c5670 2024-04-16T16:17:44.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.741+0000 7fd25ea00640 1 -- 172.21.15.45:0/2392873001 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd258034440 con 0x7fd2600c5670 2024-04-16T16:17:44.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.742+0000 7fd265600640 1 -- 172.21.15.45:0/2392873001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2600c5670 msgr2=0x7fd2600c5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:44.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.742+0000 7fd265600640 1 --2- 172.21.15.45:0/2392873001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2600c5670 0x7fd2600c5ae0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fd258009d90 tx=0x7fd25802be10 comp rx=0 tx=0).stop 2024-04-16T16:17:44.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.742+0000 7fd265600640 1 -- 172.21.15.45:0/2392873001 shutdown_connections 2024-04-16T16:17:44.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.742+0000 7fd265600640 1 --2- 172.21.15.45:0/2392873001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2600c5670 0x7fd2600c5ae0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:44.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.742+0000 7fd265600640 1 --2- 172.21.15.45:0/2392873001 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd260145230 0x7fd26015bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:44.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.742+0000 7fd265600640 1 -- 172.21.15.45:0/2392873001 >> 172.21.15.45:0/2392873001 conn(0x7fd26006b930 msgr2=0x7fd26006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:44.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.742+0000 7fd265600640 1 -- 172.21.15.45:0/2392873001 shutdown_connections 2024-04-16T16:17:44.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.742+0000 7fd265600640 1 -- 172.21.15.45:0/2392873001 wait complete. 2024-04-16T16:17:44.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.744+0000 7fd265600640 1 Processor -- start 2024-04-16T16:17:44.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.745+0000 7fd265600640 1 -- start start 2024-04-16T16:17:44.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.746+0000 7fd265600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2600c5670 0x7fd260140610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:44.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.746+0000 7fd265600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd260145230 0x7fd26013fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:44.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.746+0000 7fd265600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd26015cbe0 0x7fd2600b3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:44.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.746+0000 7fd265600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd26015d040 con 0x7fd260145230 2024-04-16T16:17:44.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.746+0000 7fd265600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd26015dc50 con 0x7fd2600c5670 2024-04-16T16:17:44.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.746+0000 7fd265600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd26015e450 con 0x7fd26015cbe0 2024-04-16T16:17:44.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.746+0000 7fd25f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd260145230 0x7fd26013fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:44.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.746+0000 7fd25f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd260145230 0x7fd26013fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48768/0 (socket says 172.21.15.45:48768) 2024-04-16T16:17:44.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.746+0000 7fd25f400640 1 -- 172.21.15.45:0/1343433773 learned_addr learned my addr 172.21.15.45:0/1343433773 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:44.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.747+0000 7fd264c00640 1 --2- 172.21.15.45:0/1343433773 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd26015cbe0 0x7fd2600b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:44.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.747+0000 7fd25f400640 1 -- 172.21.15.45:0/1343433773 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd26015cbe0 msgr2=0x7fd2600b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:44.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.747+0000 7fd25fe00640 1 --2- 172.21.15.45:0/1343433773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2600c5670 0x7fd260140610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:44.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.747+0000 7fd25f400640 1 --2- 172.21.15.45:0/1343433773 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd26015cbe0 0x7fd2600b3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:44.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.747+0000 7fd25f400640 1 -- 172.21.15.45:0/1343433773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2600c5670 msgr2=0x7fd260140610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:44.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.747+0000 7fd25f400640 1 --2- 172.21.15.45:0/1343433773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2600c5670 0x7fd260140610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:44.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.747+0000 7fd25f400640 1 -- 172.21.15.45:0/1343433773 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2600b3fa0 con 0x7fd260145230 2024-04-16T16:17:44.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.747+0000 7fd264c00640 1 --2- 172.21.15.45:0/1343433773 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd26015cbe0 0x7fd2600b3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:44.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.747+0000 7fd25f400640 1 --2- 172.21.15.45:0/1343433773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd260145230 0x7fd26013fbe0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00c3f0 tx=0x7fd24c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:44.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.747+0000 7fd257e00640 1 -- 172.21.15.45:0/1343433773 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c017070 con 0x7fd260145230 2024-04-16T16:17:44.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.747+0000 7fd257e00640 1 -- 172.21.15.45:0/1343433773 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd24c005ce0 con 0x7fd260145230 2024-04-16T16:17:44.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.747+0000 7fd257e00640 1 -- 172.21.15.45:0/1343433773 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c0072f0 con 0x7fd260145230 2024-04-16T16:17:44.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.747+0000 7fd265600640 1 -- 172.21.15.45:0/1343433773 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2600b4280 con 0x7fd260145230 2024-04-16T16:17:44.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.748+0000 7fd265600640 1 -- 172.21.15.45:0/1343433773 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2600b3ca0 con 0x7fd260145230 2024-04-16T16:17:44.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.749+0000 7fd265600640 1 -- 172.21.15.45:0/1343433773 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2600b5070 con 0x7fd260145230 2024-04-16T16:17:44.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.750+0000 7fd257e00640 1 -- 172.21.15.45:0/1343433773 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7fd24c01a070 con 0x7fd260145230 2024-04-16T16:17:44.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.750+0000 7fd257e00640 1 --2- 172.21.15.45:0/1343433773 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd24006ca60 0x7fd24006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:44.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.751+0000 7fd257e00640 1 -- 172.21.15.45:0/1343433773 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7fd24c08b270 con 0x7fd260145230 2024-04-16T16:17:44.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.751+0000 7fd25fe00640 1 --2- 172.21.15.45:0/1343433773 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd24006ca60 0x7fd24006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:44.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.753+0000 7fd25fe00640 1 --2- 172.21.15.45:0/1343433773 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd24006ca60 0x7fd24006ef20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd250006fd0 tx=0x7fd250008040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:44.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.755+0000 7fd257e00640 1 -- 172.21.15.45:0/1343433773 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd2600b5070 con 0x7fd260145230 2024-04-16T16:17:44.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.986+0000 7fd265600640 1 -- 172.21.15.45:0/1343433773 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7fd260075780 con 0x7fd24006ca60 2024-04-16T16:17:44.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.989+0000 7fd257e00640 1 -- 172.21.15.45:0/1343433773 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3036 (secure 0 0 0) 0x7fd260075780 con 0x7fd24006ca60 2024-04-16T16:17:44.988 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[0, 0], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], "client.5285": [[0, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[0, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[0, 0], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, "metrics": {"delayed_ranks": [1], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}, "mds.1": {"client.5243": []}}} 2024-04-16T16:17:44.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.991+0000 7fd265600640 1 -- 172.21.15.45:0/1343433773 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd24006ca60 msgr2=0x7fd24006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:44.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.991+0000 7fd265600640 1 --2- 172.21.15.45:0/1343433773 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd24006ca60 0x7fd24006ef20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd250006fd0 tx=0x7fd250008040 comp rx=0 tx=0).stop 2024-04-16T16:17:44.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.991+0000 7fd265600640 1 -- 172.21.15.45:0/1343433773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd260145230 msgr2=0x7fd26013fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:44.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.991+0000 7fd265600640 1 --2- 172.21.15.45:0/1343433773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd260145230 0x7fd26013fbe0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00c3f0 tx=0x7fd24c00c8b0 comp rx=0 tx=0).stop 2024-04-16T16:17:44.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.992+0000 7fd265600640 1 -- 172.21.15.45:0/1343433773 shutdown_connections 2024-04-16T16:17:44.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.992+0000 7fd265600640 1 --2- 172.21.15.45:0/1343433773 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fd24006ca60 0x7fd24006ef20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:44.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.992+0000 7fd265600640 1 --2- 172.21.15.45:0/1343433773 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd26015cbe0 0x7fd2600b3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:44.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.992+0000 7fd265600640 1 --2- 172.21.15.45:0/1343433773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd260145230 0x7fd26013fbe0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:44.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.992+0000 7fd265600640 1 --2- 172.21.15.45:0/1343433773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2600c5670 0x7fd260140610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:44.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.992+0000 7fd265600640 1 -- 172.21.15.45:0/1343433773 >> 172.21.15.45:0/1343433773 conn(0x7fd26006b930 msgr2=0x7fd260142940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:44.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.992+0000 7fd265600640 1 -- 172.21.15.45:0/1343433773 shutdown_connections 2024-04-16T16:17:44.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:44.992+0000 7fd265600640 1 -- 172.21.15.45:0/1343433773 wait complete. 2024-04-16T16:17:45.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:17:45.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.235+0000 7f978b200640 1 Processor -- start 2024-04-16T16:17:45.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.236+0000 7f978b200640 1 -- start start 2024-04-16T16:17:45.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.237+0000 7f978b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f978c162270 0x7f978c0d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:45.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.237+0000 7f978b200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f978c0d52d0 0x7f978c06ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:45.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.237+0000 7f978b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f978c0d3080 con 0x7f978c161e90 2024-04-16T16:17:45.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.237+0000 7f978b200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f978c0d2e30 con 0x7f978c0d52d0 2024-04-16T16:17:45.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.237+0000 7f978b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f978c0d3360 con 0x7f978c162270 2024-04-16T16:17:45.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.238+0000 7f978a800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f978c0d52d0 0x7f978c06ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:45.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.238+0000 7f978a800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f978c0d52d0 0x7f978c06ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48780/0 (socket says 172.21.15.45:48780) 2024-04-16T16:17:45.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.238+0000 7f978a800640 1 -- 172.21.15.45:0/3007607985 learned_addr learned my addr 172.21.15.45:0/3007607985 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:45.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.238+0000 7f9789400640 1 --2- 172.21.15.45:0/3007607985 >> v2:172.21.15.103:3300/0 conn(0x7f978c162270 0x7f978c0d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:45.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.238+0000 7f978a800640 1 -- 172.21.15.45:0/3007607985 >> v2:172.21.15.103:3300/0 conn(0x7f978c162270 msgr2=0x7f978c0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:45.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.238+0000 7f978a800640 1 --2- 172.21.15.45:0/3007607985 >> v2:172.21.15.103:3300/0 conn(0x7f978c162270 0x7f978c0d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:45.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.238+0000 7f978a800640 1 -- 172.21.15.45:0/3007607985 >> v1:172.21.15.103:6789/0 conn(0x7f978c161e90 legacy=0x7f978c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:45.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.238+0000 7f978a800640 1 -- 172.21.15.45:0/3007607985 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978c0d3640 con 0x7f978c0d52d0 2024-04-16T16:17:45.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.239+0000 7f978a800640 1 --2- 172.21.15.45:0/3007607985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c0d52d0 0x7f978c06ce40 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f9784009870 tx=0x7f978402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e7214130d633a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:45.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.240+0000 7f9788a00640 1 -- 172.21.15.45:0/3007607985 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784035020 con 0x7f978c0d52d0 2024-04-16T16:17:45.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.240+0000 7f9788a00640 1 -- 172.21.15.45:0/3007607985 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9784031870 con 0x7f978c0d52d0 2024-04-16T16:17:45.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.240+0000 7f9788a00640 1 -- 172.21.15.45:0/3007607985 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784031c40 con 0x7f978c0d52d0 2024-04-16T16:17:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.240+0000 7f978b200640 1 -- 172.21.15.45:0/3007607985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c0d52d0 msgr2=0x7f978c06ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.240+0000 7f978b200640 1 --2- 172.21.15.45:0/3007607985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c0d52d0 0x7f978c06ce40 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f9784009870 tx=0x7f978402b910 comp rx=0 tx=0).stop 2024-04-16T16:17:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.241+0000 7f978b200640 1 -- 172.21.15.45:0/3007607985 shutdown_connections 2024-04-16T16:17:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.241+0000 7f978b200640 1 --2- 172.21.15.45:0/3007607985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c0d52d0 0x7f978c06ce40 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.241+0000 7f978b200640 1 --2- 172.21.15.45:0/3007607985 >> v2:172.21.15.103:3300/0 conn(0x7f978c162270 0x7f978c0d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.241+0000 7f978b200640 1 -- 172.21.15.45:0/3007607985 >> 172.21.15.45:0/3007607985 conn(0x7f978c0d1940 msgr2=0x7f978c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.241+0000 7f978b200640 1 -- 172.21.15.45:0/3007607985 shutdown_connections 2024-04-16T16:17:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.241+0000 7f978b200640 1 -- 172.21.15.45:0/3007607985 wait complete. 2024-04-16T16:17:45.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.242+0000 7f978b200640 1 Processor -- start 2024-04-16T16:17:45.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.243+0000 7f978b200640 1 -- start start 2024-04-16T16:17:45.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f978b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f978c0d52d0 0x7f978c156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:45.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f978b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f978c161e90 0x7f978c0ad520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:45.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f978b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c162270 0x7f978c0ada60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:45.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f978b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f978c0d4500 con 0x7f978c162270 2024-04-16T16:17:45.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f978b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f978c0d4380 con 0x7f978c0d52d0 2024-04-16T16:17:45.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f978b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f978c0d4680 con 0x7f978c161e90 2024-04-16T16:17:45.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f978a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c162270 0x7f978c0ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:45.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f978a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c162270 0x7f978c0ada60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48788/0 (socket says 172.21.15.45:48788) 2024-04-16T16:17:45.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f978a800640 1 -- 172.21.15.45:0/2069755810 learned_addr learned my addr 172.21.15.45:0/2069755810 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:45.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f9789400640 1 --2- 172.21.15.45:0/2069755810 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f978c161e90 0x7f978c0ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:45.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f9789e00640 1 --2- 172.21.15.45:0/2069755810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f978c0d52d0 0x7f978c156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:45.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f978a800640 1 -- 172.21.15.45:0/2069755810 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f978c161e90 msgr2=0x7f978c0ad520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:45.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f978a800640 1 --2- 172.21.15.45:0/2069755810 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f978c161e90 0x7f978c0ad520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:45.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f978a800640 1 -- 172.21.15.45:0/2069755810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f978c0d52d0 msgr2=0x7f978c156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:45.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f978a800640 1 --2- 172.21.15.45:0/2069755810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f978c0d52d0 0x7f978c156f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:45.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f978a800640 1 -- 172.21.15.45:0/2069755810 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978c0cd4f0 con 0x7f978c162270 2024-04-16T16:17:45.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.244+0000 7f9789e00640 1 --2- 172.21.15.45:0/2069755810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f978c0d52d0 0x7f978c156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:45.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.245+0000 7f978a800640 1 --2- 172.21.15.45:0/2069755810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c162270 0x7f978c0ada60 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f978402b950 tx=0x7f97840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:45.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.245+0000 7f9782000640 1 -- 172.21.15.45:0/2069755810 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784040070 con 0x7f978c162270 2024-04-16T16:17:45.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.245+0000 7f9782000640 1 -- 172.21.15.45:0/2069755810 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9784030d00 con 0x7f978c162270 2024-04-16T16:17:45.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.245+0000 7f9782000640 1 -- 172.21.15.45:0/2069755810 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784035040 con 0x7f978c162270 2024-04-16T16:17:45.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.245+0000 7f978b200640 1 -- 172.21.15.45:0/2069755810 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f978c0cd7d0 con 0x7f978c162270 2024-04-16T16:17:45.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.245+0000 7f978b200640 1 -- 172.21.15.45:0/2069755810 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f978c0cdd90 con 0x7f978c162270 2024-04-16T16:17:45.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.246+0000 7f978b200640 1 -- 172.21.15.45:0/2069755810 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f978c075780 con 0x7f978c162270 2024-04-16T16:17:45.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.247+0000 7f9782000640 1 -- 172.21.15.45:0/2069755810 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f978404b060 con 0x7f978c162270 2024-04-16T16:17:45.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.248+0000 7f9782000640 1 --2- 172.21.15.45:0/2069755810 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f976806ca60 0x7f976806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:45.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.248+0000 7f9782000640 1 -- 172.21.15.45:0/2069755810 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f97840adca0 con 0x7f978c162270 2024-04-16T16:17:45.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.249+0000 7f9789e00640 1 --2- 172.21.15.45:0/2069755810 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f976806ca60 0x7f976806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:45.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.250+0000 7f9789e00640 1 --2- 172.21.15.45:0/2069755810 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f976806ca60 0x7f976806ef20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f977c004520 tx=0x7f977c009040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:45.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.253+0000 7f9782000640 1 -- 172.21.15.45:0/2069755810 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f978407be10 con 0x7f978c162270 2024-04-16T16:17:45.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.501+0000 7f978b200640 1 -- 172.21.15.45:0/2069755810 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f978c06dae0 con 0x7f978c162270 2024-04-16T16:17:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.503+0000 7f9782000640 1 -- 172.21.15.45:0/2069755810 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5201 (secure 0 0 0) 0x7f978407b710 con 0x7f978c162270 2024-04-16T16:17:45.502 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:45.502 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":54,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:17:23.689033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6402,"mds_1":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"c","rank":1,"incarnation":52,"state":"up:active","state_seq":11,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402,6399]},"id":5}]} 2024-04-16T16:17:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 54 2024-04-16T16:17:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.505+0000 7f978b200640 1 -- 172.21.15.45:0/2069755810 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f976806ca60 msgr2=0x7f976806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.505+0000 7f978b200640 1 --2- 172.21.15.45:0/2069755810 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f976806ca60 0x7f976806ef20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f977c004520 tx=0x7f977c009040 comp rx=0 tx=0).stop 2024-04-16T16:17:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.505+0000 7f978b200640 1 -- 172.21.15.45:0/2069755810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c162270 msgr2=0x7f978c0ada60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.505+0000 7f978b200640 1 --2- 172.21.15.45:0/2069755810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c162270 0x7f978c0ada60 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f978402b950 tx=0x7f97840046c0 comp rx=0 tx=0).stop 2024-04-16T16:17:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.506+0000 7f978b200640 1 -- 172.21.15.45:0/2069755810 shutdown_connections 2024-04-16T16:17:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.506+0000 7f978b200640 1 --2- 172.21.15.45:0/2069755810 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f976806ca60 0x7f976806ef20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.506+0000 7f978b200640 1 --2- 172.21.15.45:0/2069755810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c162270 0x7f978c0ada60 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.506+0000 7f978b200640 1 --2- 172.21.15.45:0/2069755810 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f978c161e90 0x7f978c0ad520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.506+0000 7f978b200640 1 --2- 172.21.15.45:0/2069755810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f978c0d52d0 0x7f978c156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.506+0000 7f978b200640 1 -- 172.21.15.45:0/2069755810 >> 172.21.15.45:0/2069755810 conn(0x7f978c0d1940 msgr2=0x7f978c0b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.506+0000 7f978b200640 1 -- 172.21.15.45:0/2069755810 shutdown_connections 2024-04-16T16:17:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:45.506+0000 7f978b200640 1 -- 172.21.15.45:0/2069755810 wait complete. 2024-04-16T16:17:46.526 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:46.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.761+0000 7ff4a7e00640 1 Processor -- start 2024-04-16T16:17:46.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.761+0000 7ff4a7e00640 1 -- start start 2024-04-16T16:17:46.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.762+0000 7ff4a7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff4a815e830 0x7ff4a80ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:46.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.762+0000 7ff4a7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4a80cbc80 0x7ff4a80cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:46.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.763+0000 7ff4a7e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4a815f1b0 con 0x7ff4a80cb7d0 2024-04-16T16:17:46.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.763+0000 7ff4a7e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff4a80cf280 con 0x7ff4a815e830 2024-04-16T16:17:46.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.763+0000 7ff4a7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a80ce190 con 0x7ff4a80cbc80 2024-04-16T16:17:46.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.763+0000 7ff4a6000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff4a80cb7d0 0x7ff4a80cd620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59278/0 (socket says 172.21.15.45:59278) 2024-04-16T16:17:46.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.763+0000 7ff4a6000640 1 -- 172.21.15.45:0/2531753519 learned_addr learned my addr 172.21.15.45:0/2531753519 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:46.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.763+0000 7ff4a5600640 1 -- 172.21.15.45:0/2531753519 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1714527241 0 0) 0x7ff490001110 con 0x7ff4a80cb7d0 2024-04-16T16:17:46.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.763+0000 7ff4a5600640 1 -- 172.21.15.45:0/2531753519 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff48c003520 con 0x7ff4a80cb7d0 2024-04-16T16:17:46.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.763+0000 7ff4a7400640 1 --2- 172.21.15.45:0/2531753519 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4a80cbc80 0x7ff4a80cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:46.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a6a00640 1 --2- 172.21.15.45:0/2531753519 >> v2:172.21.15.103:3300/0 conn(0x7ff4a815e830 0x7ff4a80ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:46.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a5600640 1 -- 172.21.15.45:0/2531753519 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3946720863 0 0) 0x7ff48c003520 con 0x7ff4a80cb7d0 2024-04-16T16:17:46.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a5600640 1 -- 172.21.15.45:0/2531753519 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4a80cbc80 msgr2=0x7ff4a80cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:46.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a5600640 1 --2- 172.21.15.45:0/2531753519 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4a80cbc80 0x7ff4a80cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:46.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a5600640 1 -- 172.21.15.45:0/2531753519 >> v2:172.21.15.103:3300/0 conn(0x7ff4a815e830 msgr2=0x7ff4a80ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:46.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a5600640 1 --2- 172.21.15.45:0/2531753519 >> v2:172.21.15.103:3300/0 conn(0x7ff4a815e830 0x7ff4a80ce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:46.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a5600640 1 -- 172.21.15.45:0/2531753519 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a80ce470 con 0x7ff4a80cb7d0 2024-04-16T16:17:46.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a5600640 1 -- 172.21.15.45:0/2531753519 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7ff490001840 con 0x7ff4a80cb7d0 2024-04-16T16:17:46.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a5600640 1 -- 172.21.15.45:0/2531753519 >> v1:172.21.15.45:6789/0 conn(0x7ff4a80cb7d0 legacy=0x7ff4a80cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:46.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a5600640 1 --2- 172.21.15.45:0/2531753519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff48c0039b0 0x7ff48c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:46.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a5600640 1 --2- 172.21.15.45:0/2531753519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c004800 0x7ff48c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:46.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a5600640 1 --2- 172.21.15.45:0/2531753519 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4a80cbc80 0x7ff48c0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:46.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a5600640 1 -- 172.21.15.45:0/2531753519 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a80cf280 con 0x7ff48c004800 2024-04-16T16:17:46.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a5600640 1 -- 172.21.15.45:0/2531753519 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a80ce190 con 0x7ff48c0039b0 2024-04-16T16:17:46.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a5600640 1 -- 172.21.15.45:0/2531753519 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff48c00bb50 con 0x7ff4a80cbc80 2024-04-16T16:17:46.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.764+0000 7ff4a6000640 1 --2- 172.21.15.45:0/2531753519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c004800 0x7ff48c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:46.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.765+0000 7ff4a7400640 1 --2- 172.21.15.45:0/2531753519 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4a80cbc80 0x7ff48c0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:46.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.765+0000 7ff4a6a00640 1 --2- 172.21.15.45:0/2531753519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff48c0039b0 0x7ff48c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:46.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.765+0000 7ff4a6000640 1 -- 172.21.15.45:0/2531753519 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4a80cbc80 msgr2=0x7ff48c0051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:46.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.765+0000 7ff4a6000640 1 --2- 172.21.15.45:0/2531753519 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4a80cbc80 0x7ff48c0051e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:46.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.765+0000 7ff4a6000640 1 -- 172.21.15.45:0/2531753519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff48c0039b0 msgr2=0x7ff48c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:46.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.765+0000 7ff4a6000640 1 --2- 172.21.15.45:0/2531753519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff48c0039b0 0x7ff48c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:46.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.765+0000 7ff4a6000640 1 -- 172.21.15.45:0/2531753519 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48c003520 con 0x7ff48c004800 2024-04-16T16:17:46.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.765+0000 7ff4a6a00640 1 --2- 172.21.15.45:0/2531753519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff48c0039b0 0x7ff48c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:46.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.766+0000 7ff4a6000640 1 --2- 172.21.15.45:0/2531753519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c004800 0x7ff48c004c80 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7ff49000c320 tx=0x7ff48c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689d33c82ef2035d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:46.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.766+0000 7ff4a5600640 1 -- 172.21.15.45:0/2531753519 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49002cc40 con 0x7ff48c004800 2024-04-16T16:17:46.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.766+0000 7ff4a5600640 1 -- 172.21.15.45:0/2531753519 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff490030070 con 0x7ff48c004800 2024-04-16T16:17:46.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.766+0000 7ff4a5600640 1 -- 172.21.15.45:0/2531753519 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff490005b10 con 0x7ff48c004800 2024-04-16T16:17:46.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.767+0000 7ff4a7e00640 1 -- 172.21.15.45:0/2531753519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c004800 msgr2=0x7ff48c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:46.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.767+0000 7ff4a7e00640 1 --2- 172.21.15.45:0/2531753519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c004800 0x7ff48c004c80 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7ff49000c320 tx=0x7ff48c005830 comp rx=0 tx=0).stop 2024-04-16T16:17:46.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.767+0000 7ff4a7400640 1 -- 172.21.15.45:0/2531753519 reap_dead start 2024-04-16T16:17:46.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.767+0000 7ff4a7e00640 1 -- 172.21.15.45:0/2531753519 shutdown_connections 2024-04-16T16:17:46.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.767+0000 7ff4a7e00640 1 -- 172.21.15.45:0/2531753519 >> 172.21.15.45:0/2531753519 conn(0x7ff4a806b930 msgr2=0x7ff4a806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:46.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.767+0000 7ff4a7e00640 1 -- 172.21.15.45:0/2531753519 shutdown_connections 2024-04-16T16:17:46.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.767+0000 7ff4a7e00640 1 -- 172.21.15.45:0/2531753519 wait complete. 2024-04-16T16:17:46.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.769+0000 7ff4a7e00640 1 Processor -- start 2024-04-16T16:17:46.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.769+0000 7ff4a7e00640 1 -- start start 2024-04-16T16:17:46.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.770+0000 7ff4a7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c004800 0x7ff4a81538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:46.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.770+0000 7ff4a7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4a815d0b0 0x7ff4a8159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:46.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.770+0000 7ff4a7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4a8159fa0 0x7ff4a8140d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:46.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.770+0000 7ff4a7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a815ec10 con 0x7ff48c004800 2024-04-16T16:17:46.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.770+0000 7ff4a7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a80cb2b0 con 0x7ff4a8159fa0 2024-04-16T16:17:46.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.770+0000 7ff4a7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a80cac30 con 0x7ff4a815d0b0 2024-04-16T16:17:46.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.770+0000 7ff4a6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c004800 0x7ff4a81538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:46.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.770+0000 7ff4a6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c004800 0x7ff4a81538a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48810/0 (socket says 172.21.15.45:48810) 2024-04-16T16:17:46.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.770+0000 7ff4a6a00640 1 -- 172.21.15.45:0/1855274841 learned_addr learned my addr 172.21.15.45:0/1855274841 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:46.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.771+0000 7ff4a6000640 1 --2- 172.21.15.45:0/1855274841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4a815d0b0 0x7ff4a8159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:46.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.771+0000 7ff4a6a00640 1 -- 172.21.15.45:0/1855274841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4a815d0b0 msgr2=0x7ff4a8159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:46.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.771+0000 7ff4a7400640 1 --2- 172.21.15.45:0/1855274841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4a8159fa0 0x7ff4a8140d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:46.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.771+0000 7ff4a6a00640 1 --2- 172.21.15.45:0/1855274841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4a815d0b0 0x7ff4a8159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:46.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.771+0000 7ff4a6a00640 1 -- 172.21.15.45:0/1855274841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4a8159fa0 msgr2=0x7ff4a8140d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:46.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.771+0000 7ff4a6a00640 1 --2- 172.21.15.45:0/1855274841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4a8159fa0 0x7ff4a8140d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:46.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.771+0000 7ff4a6a00640 1 -- 172.21.15.45:0/1855274841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a8156a50 con 0x7ff48c004800 2024-04-16T16:17:46.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.771+0000 7ff4a7400640 1 --2- 172.21.15.45:0/1855274841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4a8159fa0 0x7ff4a8140d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:46.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.771+0000 7ff4a6a00640 1 --2- 172.21.15.45:0/1855274841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c004800 0x7ff4a81538a0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7ff49800e810 tx=0x7ff49800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:46.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.771+0000 7ff49ea00640 1 -- 172.21.15.45:0/1855274841 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498017070 con 0x7ff48c004800 2024-04-16T16:17:46.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.771+0000 7ff49ea00640 1 -- 172.21.15.45:0/1855274841 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff49800c8d0 con 0x7ff48c004800 2024-04-16T16:17:46.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.771+0000 7ff49ea00640 1 -- 172.21.15.45:0/1855274841 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49800f040 con 0x7ff48c004800 2024-04-16T16:17:46.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.772+0000 7ff4a7e00640 1 -- 172.21.15.45:0/1855274841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a815c5d0 con 0x7ff48c004800 2024-04-16T16:17:46.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.772+0000 7ff4a7e00640 1 -- 172.21.15.45:0/1855274841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a8153de0 con 0x7ff48c004800 2024-04-16T16:17:46.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.773+0000 7ff4a7e00640 1 -- 172.21.15.45:0/1855274841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4a8075780 con 0x7ff48c004800 2024-04-16T16:17:46.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.774+0000 7ff49ea00640 1 -- 172.21.15.45:0/1855274841 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7ff49801a070 con 0x7ff48c004800 2024-04-16T16:17:46.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.775+0000 7ff49ea00640 1 --2- 172.21.15.45:0/1855274841 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff48406ca60 0x7ff48406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:46.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.775+0000 7ff49ea00640 1 -- 172.21.15.45:0/1855274841 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7ff49808bd90 con 0x7ff48c004800 2024-04-16T16:17:46.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.775+0000 7ff4a6000640 1 --2- 172.21.15.45:0/1855274841 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff48406ca60 0x7ff48406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:46.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.776+0000 7ff4a6000640 1 --2- 172.21.15.45:0/1855274841 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff48406ca60 0x7ff48406ef20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff490034000 tx=0x7ff490007000 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:46.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:46.779+0000 7ff49ea00640 1 -- 172.21.15.45:0/1855274841 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff498059d50 con 0x7ff48c004800 2024-04-16T16:17:47.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.010+0000 7ff4a7e00640 1 -- 172.21.15.45:0/1855274841 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7ff4a815c3c0 con 0x7ff48406ca60 2024-04-16T16:17:47.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.013+0000 7ff49ea00640 1 -- 172.21.15.45:0/1855274841 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3036 (secure 0 0 0) 0x7ff4a815c3c0 con 0x7ff48406ca60 2024-04-16T16:17:47.014 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[0, 0], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], "client.5285": [[0, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[0, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[0, 0], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, "metrics": {"delayed_ranks": [1], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}, "mds.1": {"client.5243": []}}} 2024-04-16T16:17:47.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.016+0000 7ff4a7e00640 1 -- 172.21.15.45:0/1855274841 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff48406ca60 msgr2=0x7ff48406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:47.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.016+0000 7ff4a7e00640 1 --2- 172.21.15.45:0/1855274841 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff48406ca60 0x7ff48406ef20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff490034000 tx=0x7ff490007000 comp rx=0 tx=0).stop 2024-04-16T16:17:47.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.016+0000 7ff4a7e00640 1 -- 172.21.15.45:0/1855274841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c004800 msgr2=0x7ff4a81538a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:47.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.016+0000 7ff4a7e00640 1 --2- 172.21.15.45:0/1855274841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c004800 0x7ff4a81538a0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7ff49800e810 tx=0x7ff49800ecd0 comp rx=0 tx=0).stop 2024-04-16T16:17:47.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.016+0000 7ff4a7e00640 1 -- 172.21.15.45:0/1855274841 shutdown_connections 2024-04-16T16:17:47.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.016+0000 7ff4a7e00640 1 --2- 172.21.15.45:0/1855274841 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff48406ca60 0x7ff48406ef20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:47.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.016+0000 7ff4a7e00640 1 --2- 172.21.15.45:0/1855274841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4a8159fa0 0x7ff4a8140d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:47.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.016+0000 7ff4a7e00640 1 --2- 172.21.15.45:0/1855274841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4a815d0b0 0x7ff4a8159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:47.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.016+0000 7ff4a7e00640 1 --2- 172.21.15.45:0/1855274841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c004800 0x7ff4a81538a0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:47.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.016+0000 7ff4a7e00640 1 -- 172.21.15.45:0/1855274841 >> 172.21.15.45:0/1855274841 conn(0x7ff4a806b930 msgr2=0x7ff4a80b5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:47.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.016+0000 7ff4a7e00640 1 -- 172.21.15.45:0/1855274841 shutdown_connections 2024-04-16T16:17:47.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.016+0000 7ff4a7e00640 1 -- 172.21.15.45:0/1855274841 wait complete. 2024-04-16T16:17:47.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:17:47.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.255+0000 7f8fab200640 1 Processor -- start 2024-04-16T16:17:47.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.256+0000 7f8fab200640 1 -- start start 2024-04-16T16:17:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.257+0000 7f8fab200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8fac0d52d0 0x7f8fac06ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.257+0000 7f8fab200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fac0d3080 con 0x7f8fac161e90 2024-04-16T16:17:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.257+0000 7f8fab200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fac0d3360 con 0x7f8fac162270 2024-04-16T16:17:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.257+0000 7f8fab200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8fac0d2e30 con 0x7f8fac0d52d0 2024-04-16T16:17:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.257+0000 7f8fa9e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8fac161e90 0x7f8fac0cf7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59288/0 (socket says 172.21.15.45:59288) 2024-04-16T16:17:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.257+0000 7f8fa9e00640 1 -- 172.21.15.45:0/3795094829 learned_addr learned my addr 172.21.15.45:0/3795094829 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.257+0000 7f8faa800640 1 --2- 172.21.15.45:0/3795094829 >> v2:172.21.15.45:3300/0 conn(0x7f8fac0d52d0 0x7f8fac06ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:47.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.258+0000 7f8fa8a00640 1 -- 172.21.15.45:0/3795094829 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 481614020 0 0) 0x7f8fac0d3080 con 0x7f8fac161e90 2024-04-16T16:17:47.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.258+0000 7f8fa8a00640 1 -- 172.21.15.45:0/3795094829 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f90003520 con 0x7f8fac161e90 2024-04-16T16:17:47.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.258+0000 7f8faa800640 1 -- 172.21.15.45:0/3795094829 >> v1:172.21.15.103:6789/0 conn(0x7f8fac162270 legacy=0x7f8fac0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:47.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.258+0000 7f8faa800640 1 -- 172.21.15.45:0/3795094829 >> v1:172.21.15.45:6789/0 conn(0x7f8fac161e90 legacy=0x7f8fac0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:47.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.258+0000 7f8faa800640 1 -- 172.21.15.45:0/3795094829 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fac0d3640 con 0x7f8fac0d52d0 2024-04-16T16:17:47.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.259+0000 7f8faa800640 1 --2- 172.21.15.45:0/3795094829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac0d52d0 0x7f8fac06ce40 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4009dc0 tx=0x7f8fa402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e82389cd4d5b14 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:47.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.259+0000 7f8fa8a00640 1 -- 172.21.15.45:0/3795094829 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4039070 con 0x7f8fac0d52d0 2024-04-16T16:17:47.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.259+0000 7f8fa8a00640 1 -- 172.21.15.45:0/3795094829 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fa402cc40 con 0x7f8fac0d52d0 2024-04-16T16:17:47.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.259+0000 7f8fa8a00640 1 -- 172.21.15.45:0/3795094829 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4034440 con 0x7f8fac0d52d0 2024-04-16T16:17:47.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.260+0000 7f8fab200640 1 -- 172.21.15.45:0/3795094829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac0d52d0 msgr2=0x7f8fac06ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:47.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.260+0000 7f8fab200640 1 --2- 172.21.15.45:0/3795094829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac0d52d0 0x7f8fac06ce40 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4009dc0 tx=0x7f8fa402be90 comp rx=0 tx=0).stop 2024-04-16T16:17:47.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.260+0000 7f8fab200640 1 -- 172.21.15.45:0/3795094829 shutdown_connections 2024-04-16T16:17:47.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.260+0000 7f8fab200640 1 --2- 172.21.15.45:0/3795094829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac0d52d0 0x7f8fac06ce40 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:47.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.260+0000 7f8fab200640 1 -- 172.21.15.45:0/3795094829 >> 172.21.15.45:0/3795094829 conn(0x7f8fac0d1940 msgr2=0x7f8fac0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:47.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.260+0000 7f8fab200640 1 -- 172.21.15.45:0/3795094829 shutdown_connections 2024-04-16T16:17:47.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.260+0000 7f8fab200640 1 -- 172.21.15.45:0/3795094829 wait complete. 2024-04-16T16:17:47.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.262+0000 7f8fab200640 1 Processor -- start 2024-04-16T16:17:47.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.262+0000 7f8fab200640 1 -- start start 2024-04-16T16:17:47.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.263+0000 7f8fab200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac0d52d0 0x7f8fac1eae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:47.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.263+0000 7f8fab200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fac161e90 0x7f8fac0cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:47.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.263+0000 7f8fab200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8fac162270 0x7f8fac0b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:47.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.263+0000 7f8fab200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac0d46d0 con 0x7f8fac0d52d0 2024-04-16T16:17:47.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.263+0000 7f8fab200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac0d4550 con 0x7f8fac161e90 2024-04-16T16:17:47.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.263+0000 7f8fab200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8fac06d380 con 0x7f8fac162270 2024-04-16T16:17:47.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.263+0000 7f8fa9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac0d52d0 0x7f8fac1eae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:47.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.264+0000 7f8fa9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac0d52d0 0x7f8fac1eae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48832/0 (socket says 172.21.15.45:48832) 2024-04-16T16:17:47.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.264+0000 7f8fa9e00640 1 -- 172.21.15.45:0/3306496441 learned_addr learned my addr 172.21.15.45:0/3306496441 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:47.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.264+0000 7f8fa9400640 1 --2- 172.21.15.45:0/3306496441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fac161e90 0x7f8fac0cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:47.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.264+0000 7f8fa9e00640 1 -- 172.21.15.45:0/3306496441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8fac162270 msgr2=0x7f8fac0b2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:47.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.264+0000 7f8fa9e00640 1 --2- 172.21.15.45:0/3306496441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8fac162270 0x7f8fac0b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:47.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.264+0000 7f8fa9e00640 1 -- 172.21.15.45:0/3306496441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fac161e90 msgr2=0x7f8fac0cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:47.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.264+0000 7f8fa9e00640 1 --2- 172.21.15.45:0/3306496441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fac161e90 0x7f8fac0cc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:47.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.264+0000 7f8fa9e00640 1 -- 172.21.15.45:0/3306496441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fac0cd200 con 0x7f8fac0d52d0 2024-04-16T16:17:47.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.264+0000 7f8fa9400640 1 --2- 172.21.15.45:0/3306496441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fac161e90 0x7f8fac0cc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:47.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.264+0000 7f8fa9e00640 1 --2- 172.21.15.45:0/3306496441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac0d52d0 0x7f8fac1eae30 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c00c430 tx=0x7f8f9c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:47.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.265+0000 7f8fa2000640 1 -- 172.21.15.45:0/3306496441 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c017070 con 0x7f8fac0d52d0 2024-04-16T16:17:47.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.265+0000 7f8fa2000640 1 -- 172.21.15.45:0/3306496441 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f9c005ce0 con 0x7f8fac0d52d0 2024-04-16T16:17:47.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.265+0000 7f8fa2000640 1 -- 172.21.15.45:0/3306496441 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c0072f0 con 0x7f8fac0d52d0 2024-04-16T16:17:47.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.265+0000 7f8fab200640 1 -- 172.21.15.45:0/3306496441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fac0cd4e0 con 0x7f8fac0d52d0 2024-04-16T16:17:47.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.265+0000 7f8fab200640 1 -- 172.21.15.45:0/3306496441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fac0cdaa0 con 0x7f8fac0d52d0 2024-04-16T16:17:47.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.266+0000 7f8fab200640 1 -- 172.21.15.45:0/3306496441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fac0c7c00 con 0x7f8fac0d52d0 2024-04-16T16:17:47.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.267+0000 7f8fa2000640 1 -- 172.21.15.45:0/3306496441 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f8f9c01a070 con 0x7f8fac0d52d0 2024-04-16T16:17:47.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.268+0000 7f8fa2000640 1 --2- 172.21.15.45:0/3306496441 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f8f8806ca60 0x7f8f8806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:47.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.268+0000 7f8fa2000640 1 -- 172.21.15.45:0/3306496441 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f8f9c08b450 con 0x7f8fac0d52d0 2024-04-16T16:17:47.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.269+0000 7f8fa9400640 1 --2- 172.21.15.45:0/3306496441 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f8f8806ca60 0x7f8f8806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:47.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.270+0000 7f8fa9400640 1 --2- 172.21.15.45:0/3306496441 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f8f8806ca60 0x7f8f8806ef20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0d29d0 tx=0x7f8f94008040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:47.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.273+0000 7f8fa2000640 1 -- 172.21.15.45:0/3306496441 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8fac0c7c00 con 0x7f8fac0d52d0 2024-04-16T16:17:47.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.521+0000 7f8fab200640 1 -- 172.21.15.45:0/3306496441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fac06dae0 con 0x7f8fac0d52d0 2024-04-16T16:17:47.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.522+0000 7f8fa2000640 1 -- 172.21.15.45:0/3306496441 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5201 (secure 0 0 0) 0x7f8f9c059520 con 0x7f8fac0d52d0 2024-04-16T16:17:47.523 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:47.523 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":54,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:17:23.689033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6402,"mds_1":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"c","rank":1,"incarnation":52,"state":"up:active","state_seq":11,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402,6399]},"id":5}]} 2024-04-16T16:17:47.523 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 54 2024-04-16T16:17:47.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.525+0000 7f8fab200640 1 -- 172.21.15.45:0/3306496441 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f8f8806ca60 msgr2=0x7f8f8806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.525+0000 7f8fab200640 1 --2- 172.21.15.45:0/3306496441 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f8f8806ca60 0x7f8f8806ef20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0d29d0 tx=0x7f8f94008040 comp rx=0 tx=0).stop 2024-04-16T16:17:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.525+0000 7f8fab200640 1 -- 172.21.15.45:0/3306496441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac0d52d0 msgr2=0x7f8fac1eae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.525+0000 7f8fab200640 1 --2- 172.21.15.45:0/3306496441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac0d52d0 0x7f8fac1eae30 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c00c430 tx=0x7f8f9c00c8f0 comp rx=0 tx=0).stop 2024-04-16T16:17:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.525+0000 7f8fab200640 1 -- 172.21.15.45:0/3306496441 shutdown_connections 2024-04-16T16:17:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.525+0000 7f8fab200640 1 --2- 172.21.15.45:0/3306496441 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f8f8806ca60 0x7f8f8806ef20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.525+0000 7f8fab200640 1 --2- 172.21.15.45:0/3306496441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8fac162270 0x7f8fac0b2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.525+0000 7f8fab200640 1 --2- 172.21.15.45:0/3306496441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fac161e90 0x7f8fac0cc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.525+0000 7f8fab200640 1 --2- 172.21.15.45:0/3306496441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac0d52d0 0x7f8fac1eae30 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:47.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.525+0000 7f8fab200640 1 -- 172.21.15.45:0/3306496441 >> 172.21.15.45:0/3306496441 conn(0x7f8fac0d1940 msgr2=0x7f8fac0b8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:47.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.525+0000 7f8fab200640 1 -- 172.21.15.45:0/3306496441 shutdown_connections 2024-04-16T16:17:47.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:47.526+0000 7f8fab200640 1 -- 172.21.15.45:0/3306496441 wait complete. 2024-04-16T16:17:48.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:17:48.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.774+0000 7f9db9400640 1 Processor -- start 2024-04-16T16:17:48.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.774+0000 7f9db9400640 1 -- start start 2024-04-16T16:17:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.776+0000 7f9db9400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9db40ba140 0x7f9db40d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.776+0000 7f9db9400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9db40d2600 con 0x7f9db40cfc20 2024-04-16T16:17:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.776+0000 7f9db9400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9db40ba5d0 con 0x7f9db4162090 2024-04-16T16:17:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.776+0000 7f9db9400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9db40ba8b0 con 0x7f9db40ba140 2024-04-16T16:17:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.776+0000 7f9db8a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9db40cfc20 0x7f9db40d35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59300/0 (socket says 172.21.15.45:59300) 2024-04-16T16:17:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.776+0000 7f9db8a00640 1 -- 172.21.15.45:0/3023800538 learned_addr learned my addr 172.21.15.45:0/3023800538 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.777+0000 7f9dab400640 1 --2- 172.21.15.45:0/3023800538 >> v2:172.21.15.103:3300/0 conn(0x7f9db40ba140 0x7f9db40d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.777+0000 7f9db3400640 1 -- 172.21.15.45:0/3023800538 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3265576101 0 0) 0x7f9db40d2600 con 0x7f9db40cfc20 2024-04-16T16:17:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.777+0000 7f9db3400640 1 -- 172.21.15.45:0/3023800538 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d98003520 con 0x7f9db40cfc20 2024-04-16T16:17:48.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.777+0000 7f9db3400640 1 -- 172.21.15.45:0/3023800538 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3458403088 0 0) 0x7f9d98003520 con 0x7f9db40cfc20 2024-04-16T16:17:48.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.777+0000 7f9db3400640 1 -- 172.21.15.45:0/3023800538 >> v2:172.21.15.103:3300/0 conn(0x7f9db40ba140 msgr2=0x7f9db40d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:48.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.777+0000 7f9db3400640 1 --2- 172.21.15.45:0/3023800538 >> v2:172.21.15.103:3300/0 conn(0x7f9db40ba140 0x7f9db40d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:48.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.777+0000 7f9db3400640 1 -- 172.21.15.45:0/3023800538 >> v1:172.21.15.103:6789/0 conn(0x7f9db4162090 legacy=0x7f9db40cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:48.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.777+0000 7f9db3400640 1 -- 172.21.15.45:0/3023800538 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9db40d28a0 con 0x7f9db40cfc20 2024-04-16T16:17:48.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.777+0000 7f9db3400640 1 -- 172.21.15.45:0/3023800538 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f9dac001a10 con 0x7f9db40cfc20 2024-04-16T16:17:48.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.778+0000 7f9db3400640 1 -- 172.21.15.45:0/3023800538 >> v1:172.21.15.45:6789/0 conn(0x7f9db40cfc20 legacy=0x7f9db40d35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:48.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.778+0000 7f9db3400640 1 --2- 172.21.15.45:0/3023800538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d980039b0 0x7f9d98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:48.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.778+0000 7f9db3400640 1 --2- 172.21.15.45:0/3023800538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d98004800 0x7f9d98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:48.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.778+0000 7f9db3400640 1 --2- 172.21.15.45:0/3023800538 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d980051c0 0x7f9d9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:48.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.778+0000 7f9db3400640 1 -- 172.21.15.45:0/3023800538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9db40ba8b0 con 0x7f9d98004800 2024-04-16T16:17:48.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.778+0000 7f9db3400640 1 -- 172.21.15.45:0/3023800538 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9800c060 con 0x7f9d980039b0 2024-04-16T16:17:48.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.778+0000 7f9db3400640 1 -- 172.21.15.45:0/3023800538 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9d9800c260 con 0x7f9d980051c0 2024-04-16T16:17:48.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.778+0000 7f9dab400640 1 --2- 172.21.15.45:0/3023800538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d98004800 0x7f9d98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:48.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.778+0000 7f9db8a00640 1 --2- 172.21.15.45:0/3023800538 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d980051c0 0x7f9d9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:48.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.778+0000 7f9dab400640 1 -- 172.21.15.45:0/3023800538 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d980051c0 msgr2=0x7f9d9800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:48.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.778+0000 7f9db3e00640 1 --2- 172.21.15.45:0/3023800538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d980039b0 0x7f9d98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:48.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.778+0000 7f9dab400640 1 --2- 172.21.15.45:0/3023800538 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9d980051c0 0x7f9d9800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:48.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.778+0000 7f9dab400640 1 -- 172.21.15.45:0/3023800538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d980039b0 msgr2=0x7f9d98003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:48.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.779+0000 7f9dab400640 1 --2- 172.21.15.45:0/3023800538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d980039b0 0x7f9d98003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:48.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.779+0000 7f9dab400640 1 -- 172.21.15.45:0/3023800538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d98003520 con 0x7f9d98004800 2024-04-16T16:17:48.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.779+0000 7f9db8a00640 1 -- 172.21.15.45:0/3023800538 reap_dead start 2024-04-16T16:17:48.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.779+0000 7f9db3e00640 1 --2- 172.21.15.45:0/3023800538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d980039b0 0x7f9d98003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:48.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.780+0000 7f9dab400640 1 --2- 172.21.15.45:0/3023800538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d98004800 0x7f9d98004c80 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c009ac0 tx=0x7f9d98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74527869d376cc34 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:48.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.780+0000 7f9db3400640 1 -- 172.21.15.45:0/3023800538 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c039070 con 0x7f9d98004800 2024-04-16T16:17:48.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.780+0000 7f9db3400640 1 -- 172.21.15.45:0/3023800538 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d9c02c6b0 con 0x7f9d98004800 2024-04-16T16:17:48.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.780+0000 7f9db3400640 1 -- 172.21.15.45:0/3023800538 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c009690 con 0x7f9d98004800 2024-04-16T16:17:48.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.781+0000 7f9db9400640 1 -- 172.21.15.45:0/3023800538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d98004800 msgr2=0x7f9d98004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:48.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.781+0000 7f9db9400640 1 --2- 172.21.15.45:0/3023800538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d98004800 0x7f9d98004c80 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c009ac0 tx=0x7f9d98005830 comp rx=0 tx=0).stop 2024-04-16T16:17:48.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.781+0000 7f9db9400640 1 -- 172.21.15.45:0/3023800538 shutdown_connections 2024-04-16T16:17:48.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.781+0000 7f9db9400640 1 --2- 172.21.15.45:0/3023800538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d98004800 0x7f9d98004c80 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:48.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.781+0000 7f9db9400640 1 -- 172.21.15.45:0/3023800538 >> 172.21.15.45:0/3023800538 conn(0x7f9db40d1940 msgr2=0x7f9db40d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:48.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.781+0000 7f9db9400640 1 -- 172.21.15.45:0/3023800538 shutdown_connections 2024-04-16T16:17:48.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.781+0000 7f9db9400640 1 -- 172.21.15.45:0/3023800538 wait complete. 2024-04-16T16:17:48.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.783+0000 7f9db9400640 1 Processor -- start 2024-04-16T16:17:48.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.783+0000 7f9db9400640 1 -- start start 2024-04-16T16:17:48.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.784+0000 7f9db9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d98004800 0x7f9db40c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:48.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.784+0000 7f9db9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9db4153eb0 0x7f9db41612e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:48.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.784+0000 7f9db9400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9db4154990 0x7f9db4154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:48.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.784+0000 7f9db9400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9db41625f0 con 0x7f9d98004800 2024-04-16T16:17:48.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.784+0000 7f9db9400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9db4162470 con 0x7f9db4154990 2024-04-16T16:17:48.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.784+0000 7f9db9400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9db40d5520 con 0x7f9db4153eb0 2024-04-16T16:17:48.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.784+0000 7f9db3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d98004800 0x7f9db40c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:48.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.785+0000 7f9db3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d98004800 0x7f9db40c83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48842/0 (socket says 172.21.15.45:48842) 2024-04-16T16:17:48.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.785+0000 7f9db3e00640 1 -- 172.21.15.45:0/1832991364 learned_addr learned my addr 172.21.15.45:0/1832991364 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:48.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.785+0000 7f9dab400640 1 --2- 172.21.15.45:0/1832991364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9db4153eb0 0x7f9db41612e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:48.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.785+0000 7f9db8a00640 1 --2- 172.21.15.45:0/1832991364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9db4154990 0x7f9db4154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:48.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.785+0000 7f9db3e00640 1 -- 172.21.15.45:0/1832991364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9db4153eb0 msgr2=0x7f9db41612e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:48.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.785+0000 7f9db3e00640 1 --2- 172.21.15.45:0/1832991364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9db4153eb0 0x7f9db41612e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:48.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.785+0000 7f9db3e00640 1 -- 172.21.15.45:0/1832991364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9db4154990 msgr2=0x7f9db4154e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:48.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.785+0000 7f9db3e00640 1 --2- 172.21.15.45:0/1832991364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9db4154990 0x7f9db4154e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:48.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.785+0000 7f9db3e00640 1 -- 172.21.15.45:0/1832991364 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9db40c89a0 con 0x7f9d98004800 2024-04-16T16:17:48.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.785+0000 7f9db8a00640 1 --2- 172.21.15.45:0/1832991364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9db4154990 0x7f9db4154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:48.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.785+0000 7f9db3e00640 1 --2- 172.21.15.45:0/1832991364 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d98004800 0x7f9db40c83c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f9da400d400 tx=0x7f9da400d8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:48.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.786+0000 7f9db0c00640 1 -- 172.21.15.45:0/1832991364 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da4018070 con 0x7f9d98004800 2024-04-16T16:17:48.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.786+0000 7f9db0c00640 1 -- 172.21.15.45:0/1832991364 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9da4003610 con 0x7f9d98004800 2024-04-16T16:17:48.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.786+0000 7f9db0c00640 1 -- 172.21.15.45:0/1832991364 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da4003a40 con 0x7f9d98004800 2024-04-16T16:17:48.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.786+0000 7f9db9400640 1 -- 172.21.15.45:0/1832991364 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9db40c8c80 con 0x7f9d98004800 2024-04-16T16:17:48.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.786+0000 7f9db9400640 1 -- 172.21.15.45:0/1832991364 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9db40c9520 con 0x7f9d98004800 2024-04-16T16:17:48.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.787+0000 7f9db9400640 1 -- 172.21.15.45:0/1832991364 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9db4075780 con 0x7f9d98004800 2024-04-16T16:17:48.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.788+0000 7f9db0c00640 1 -- 172.21.15.45:0/1832991364 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f9da401b070 con 0x7f9d98004800 2024-04-16T16:17:48.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.789+0000 7f9db0c00640 1 --2- 172.21.15.45:0/1832991364 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f9d9006ca60 0x7f9d9006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:48.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.789+0000 7f9db0c00640 1 -- 172.21.15.45:0/1832991364 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f9da408b7d0 con 0x7f9d98004800 2024-04-16T16:17:48.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.790+0000 7f9dab400640 1 --2- 172.21.15.45:0/1832991364 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f9d9006ca60 0x7f9d9006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:48.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.791+0000 7f9dab400640 1 --2- 172.21.15.45:0/1832991364 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f9d9006ca60 0x7f9d9006ef20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c036000 tx=0x7f9d9c030000 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:48.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:48.794+0000 7f9db0c00640 1 -- 172.21.15.45:0/1832991364 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9da4059140 con 0x7f9d98004800 2024-04-16T16:17:49.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.028+0000 7f9db9400640 1 -- 172.21.15.45:0/1832991364 --> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f9db406dab0 con 0x7f9d9006ca60 2024-04-16T16:17:49.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.032+0000 7f9db0c00640 1 -- 172.21.15.45:0/1832991364 <== mgr.5620 v2:172.21.15.103:6832/541757982 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3039 (secure 0 0 0) 0x7f9db406dab0 con 0x7f9d9006ca60 2024-04-16T16:17:49.032 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.5243": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.45"}, "client.5285": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.5288": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.6591": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "172.21.15.103"}}}, "global_metrics": {"cephfs": {"client.5243": [[419, 22], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], "client.5285": [[2, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], "client.5288": [[2, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], "client.6591": [[52, 2], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.5243": [], "client.5285": [], "client.5288": [], "client.6591": []}, "mds.1": {"client.5243": []}}} 2024-04-16T16:17:49.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.034+0000 7f9db9400640 1 -- 172.21.15.45:0/1832991364 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f9d9006ca60 msgr2=0x7f9d9006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:49.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.035+0000 7f9db9400640 1 --2- 172.21.15.45:0/1832991364 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f9d9006ca60 0x7f9d9006ef20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c036000 tx=0x7f9d9c030000 comp rx=0 tx=0).stop 2024-04-16T16:17:49.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.035+0000 7f9db9400640 1 -- 172.21.15.45:0/1832991364 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d98004800 msgr2=0x7f9db40c83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:49.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.035+0000 7f9db9400640 1 --2- 172.21.15.45:0/1832991364 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d98004800 0x7f9db40c83c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f9da400d400 tx=0x7f9da400d8c0 comp rx=0 tx=0).stop 2024-04-16T16:17:49.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.035+0000 7f9db9400640 1 -- 172.21.15.45:0/1832991364 shutdown_connections 2024-04-16T16:17:49.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.035+0000 7f9db9400640 1 --2- 172.21.15.45:0/1832991364 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f9d9006ca60 0x7f9d9006ef20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:49.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.035+0000 7f9db9400640 1 --2- 172.21.15.45:0/1832991364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9db4154990 0x7f9db4154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:49.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.035+0000 7f9db9400640 1 --2- 172.21.15.45:0/1832991364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9db4153eb0 0x7f9db41612e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:49.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.035+0000 7f9db9400640 1 --2- 172.21.15.45:0/1832991364 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d98004800 0x7f9db40c83c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:49.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.035+0000 7f9db9400640 1 -- 172.21.15.45:0/1832991364 >> 172.21.15.45:0/1832991364 conn(0x7f9db40d1940 msgr2=0x7f9db40b4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:49.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.035+0000 7f9db9400640 1 -- 172.21.15.45:0/1832991364 shutdown_connections 2024-04-16T16:17:49.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.035+0000 7f9db9400640 1 -- 172.21.15.45:0/1832991364 wait complete. 2024-04-16T16:17:49.053 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:17:49.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.280+0000 7f6f0ce00640 1 Processor -- start 2024-04-16T16:17:49.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.280+0000 7f6f0ce00640 1 -- start start 2024-04-16T16:17:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.282+0000 7f6f0ce00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6f080ccb20 0x7f6f080cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.282+0000 7f6f0ce00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6f080ce9a0 0x7f6f080cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.282+0000 7f6f0ce00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0815ecd0 con 0x7f6f0815e8f0 2024-04-16T16:17:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.282+0000 7f6f0ce00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f6f080ce2e0 con 0x7f6f080ccb20 2024-04-16T16:17:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.282+0000 7f6f0ce00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6f080cd6d0 con 0x7f6f080ce9a0 2024-04-16T16:17:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.282+0000 7f6effe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6f080ccb20 0x7f6f080cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.282+0000 7f6effe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6f080ccb20 0x7f6f080cac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48850/0 (socket says 172.21.15.45:48850) 2024-04-16T16:17:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.282+0000 7f6effe00640 1 -- 172.21.15.45:0/4093205262 learned_addr learned my addr 172.21.15.45:0/4093205262 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.283+0000 7f6effe00640 1 -- 172.21.15.45:0/4093205262 >> v2:172.21.15.103:3300/0 conn(0x7f6f080ce9a0 msgr2=0x7f6f080cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.283+0000 7f6f07e00640 1 --2- 172.21.15.45:0/4093205262 >> v2:172.21.15.103:3300/0 conn(0x7f6f080ce9a0 0x7f6f080cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.283+0000 7f6effe00640 1 --2- 172.21.15.45:0/4093205262 >> v2:172.21.15.103:3300/0 conn(0x7f6f080ce9a0 0x7f6f080cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.283+0000 7f6effe00640 1 -- 172.21.15.45:0/4093205262 >> v1:172.21.15.103:6789/0 conn(0x7f6f0815e8f0 legacy=0x7f6f080cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.283+0000 7f6effe00640 1 -- 172.21.15.45:0/4093205262 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f080cd9b0 con 0x7f6f080ccb20 2024-04-16T16:17:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.283+0000 7f6f07e00640 1 --2- 172.21.15.45:0/4093205262 >> v2:172.21.15.103:3300/0 conn(0x7f6f080ce9a0 0x7f6f080cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.284+0000 7f6effe00640 1 --2- 172.21.15.45:0/4093205262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f080ccb20 0x7f6f080cac30 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f6f0806ff60 tx=0x7f6ef002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec949beceec31e51 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.284+0000 7f6f06a00640 1 -- 172.21.15.45:0/4093205262 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef002e070 con 0x7f6f080ccb20 2024-04-16T16:17:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.284+0000 7f6f06a00640 1 -- 172.21.15.45:0/4093205262 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ef0037070 con 0x7f6f080ccb20 2024-04-16T16:17:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.284+0000 7f6f06a00640 1 -- 172.21.15.45:0/4093205262 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0031a00 con 0x7f6f080ccb20 2024-04-16T16:17:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.285+0000 7f6f0ce00640 1 -- 172.21.15.45:0/4093205262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f080ccb20 msgr2=0x7f6f080cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.285+0000 7f6f0ce00640 1 --2- 172.21.15.45:0/4093205262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f080ccb20 0x7f6f080cac30 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f6f0806ff60 tx=0x7f6ef002b880 comp rx=0 tx=0).stop 2024-04-16T16:17:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.285+0000 7f6f0ce00640 1 -- 172.21.15.45:0/4093205262 shutdown_connections 2024-04-16T16:17:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.285+0000 7f6f0ce00640 1 --2- 172.21.15.45:0/4093205262 >> v2:172.21.15.103:3300/0 conn(0x7f6f080ce9a0 0x7f6f080cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.285+0000 7f6f0ce00640 1 --2- 172.21.15.45:0/4093205262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f080ccb20 0x7f6f080cac30 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.285+0000 7f6f0ce00640 1 -- 172.21.15.45:0/4093205262 >> 172.21.15.45:0/4093205262 conn(0x7f6f0806b930 msgr2=0x7f6f0806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.285+0000 7f6f0ce00640 1 -- 172.21.15.45:0/4093205262 shutdown_connections 2024-04-16T16:17:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.285+0000 7f6f0ce00640 1 -- 172.21.15.45:0/4093205262 wait complete. 2024-04-16T16:17:49.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.287+0000 7f6f0ce00640 1 Processor -- start 2024-04-16T16:17:49.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.287+0000 7f6f0ce00640 1 -- start start 2024-04-16T16:17:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.288+0000 7f6f0ce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f080ccb20 0x7f6f0815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.288+0000 7f6f0ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f080ce9a0 0x7f6f081623d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.288+0000 7f6f0ce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f0815e8f0 0x7f6f08159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.288+0000 7f6f0ce00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6f080cd360 con 0x7f6f080ce9a0 2024-04-16T16:17:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.288+0000 7f6f0ce00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6f080cdf70 con 0x7f6f0815e8f0 2024-04-16T16:17:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.288+0000 7f6f0ce00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6f080cf080 con 0x7f6f080ccb20 2024-04-16T16:17:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.288+0000 7f6effe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f080ce9a0 0x7f6f081623d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.288+0000 7f6effe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f080ce9a0 0x7f6f081623d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48854/0 (socket says 172.21.15.45:48854) 2024-04-16T16:17:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.288+0000 7f6effe00640 1 -- 172.21.15.45:0/1581947127 learned_addr learned my addr 172.21.15.45:0/1581947127 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.289+0000 7f6f07e00640 1 --2- 172.21.15.45:0/1581947127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f0815e8f0 0x7f6f08159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.289+0000 7f6f07400640 1 --2- 172.21.15.45:0/1581947127 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f080ccb20 0x7f6f0815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.289+0000 7f6effe00640 1 -- 172.21.15.45:0/1581947127 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f080ccb20 msgr2=0x7f6f0815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.289+0000 7f6effe00640 1 --2- 172.21.15.45:0/1581947127 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f080ccb20 0x7f6f0815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.289+0000 7f6effe00640 1 -- 172.21.15.45:0/1581947127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f0815e8f0 msgr2=0x7f6f08159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.289+0000 7f6effe00640 1 --2- 172.21.15.45:0/1581947127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f0815e8f0 0x7f6f08159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.289+0000 7f6effe00640 1 -- 172.21.15.45:0/1581947127 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f08158bb0 con 0x7f6f080ce9a0 2024-04-16T16:17:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.289+0000 7f6f07400640 1 --2- 172.21.15.45:0/1581947127 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f080ccb20 0x7f6f0815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.289+0000 7f6effe00640 1 --2- 172.21.15.45:0/1581947127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f080ce9a0 0x7f6f081623d0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f6ef002b8c0 tx=0x7f6ef00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.289+0000 7f6eff400640 1 -- 172.21.15.45:0/1581947127 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0032880 con 0x7f6f080ce9a0 2024-04-16T16:17:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.290+0000 7f6eff400640 1 -- 172.21.15.45:0/1581947127 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ef0032a20 con 0x7f6f080ce9a0 2024-04-16T16:17:49.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.290+0000 7f6eff400640 1 -- 172.21.15.45:0/1581947127 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0032e10 con 0x7f6f080ce9a0 2024-04-16T16:17:49.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.290+0000 7f6f0ce00640 1 -- 172.21.15.45:0/1581947127 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f08158e90 con 0x7f6f080ce9a0 2024-04-16T16:17:49.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.290+0000 7f6f0ce00640 1 -- 172.21.15.45:0/1581947127 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f08159450 con 0x7f6f080ce9a0 2024-04-16T16:17:49.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.291+0000 7f6f0ce00640 1 -- 172.21.15.45:0/1581947127 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f08075780 con 0x7f6f080ce9a0 2024-04-16T16:17:49.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.292+0000 7f6eff400640 1 -- 172.21.15.45:0/1581947127 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7f6ef004c060 con 0x7f6f080ce9a0 2024-04-16T16:17:49.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.293+0000 7f6eff400640 1 --2- 172.21.15.45:0/1581947127 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f6ee406ca60 0x7f6ee406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:49.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.293+0000 7f6eff400640 1 -- 172.21.15.45:0/1581947127 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f6ef00adc10 con 0x7f6f080ce9a0 2024-04-16T16:17:49.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.294+0000 7f6f07400640 1 --2- 172.21.15.45:0/1581947127 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f6ee406ca60 0x7f6ee406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:49.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.295+0000 7f6f07400640 1 --2- 172.21.15.45:0/1581947127 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f6ee406ca60 0x7f6ee406ef20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800a830 tx=0x7f6ef8008040 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:49.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.298+0000 7f6eff400640 1 -- 172.21.15.45:0/1581947127 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ef0040070 con 0x7f6f080ce9a0 2024-04-16T16:17:49.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.548+0000 7f6f0ce00640 1 -- 172.21.15.45:0/1581947127 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f081546e0 con 0x7f6f080ce9a0 2024-04-16T16:17:49.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.550+0000 7f6eff400640 1 -- 172.21.15.45:0/1581947127 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5201 (secure 0 0 0) 0x7f6ef004c350 con 0x7f6f080ce9a0 2024-04-16T16:17:49.550 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:49.551 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":54,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:17:23.689033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6402,"mds_1":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"c","rank":1,"incarnation":52,"state":"up:active","state_seq":11,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402,6399]},"id":5}]} 2024-04-16T16:17:49.551 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 54 2024-04-16T16:17:49.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.553+0000 7f6f0ce00640 1 -- 172.21.15.45:0/1581947127 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f6ee406ca60 msgr2=0x7f6ee406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:49.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.553+0000 7f6f0ce00640 1 --2- 172.21.15.45:0/1581947127 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f6ee406ca60 0x7f6ee406ef20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800a830 tx=0x7f6ef8008040 comp rx=0 tx=0).stop 2024-04-16T16:17:49.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.553+0000 7f6f0ce00640 1 -- 172.21.15.45:0/1581947127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f080ce9a0 msgr2=0x7f6f081623d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:49.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.553+0000 7f6f0ce00640 1 --2- 172.21.15.45:0/1581947127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f080ce9a0 0x7f6f081623d0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f6ef002b8c0 tx=0x7f6ef00046c0 comp rx=0 tx=0).stop 2024-04-16T16:17:49.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.553+0000 7f6f0ce00640 1 -- 172.21.15.45:0/1581947127 shutdown_connections 2024-04-16T16:17:49.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.553+0000 7f6f0ce00640 1 --2- 172.21.15.45:0/1581947127 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f6ee406ca60 0x7f6ee406ef20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:49.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.553+0000 7f6f0ce00640 1 --2- 172.21.15.45:0/1581947127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f0815e8f0 0x7f6f08159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:49.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.553+0000 7f6f0ce00640 1 --2- 172.21.15.45:0/1581947127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f080ce9a0 0x7f6f081623d0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:49.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.553+0000 7f6f0ce00640 1 --2- 172.21.15.45:0/1581947127 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f080ccb20 0x7f6f0815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:49.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.553+0000 7f6f0ce00640 1 -- 172.21.15.45:0/1581947127 >> 172.21.15.45:0/1581947127 conn(0x7f6f0806b930 msgr2=0x7f6f080cc410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:49.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.553+0000 7f6f0ce00640 1 -- 172.21.15.45:0/1581947127 shutdown_connections 2024-04-16T16:17:49.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:49.553+0000 7f6f0ce00640 1 -- 172.21.15.45:0/1581947127 wait complete. 2024-04-16T16:17:49.573 DEBUG:tasks.cephfs.test_mds_metrics:metrics={'version': 2, 'global_counters': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'counters': [], 'client_metadata': {'cephfs': {'client.5243': {'hostname': 'smithi045', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '172.21.15.45'}, 'client.5285': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'IP': '172.21.15.103'}, 'client.5288': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'IP': '172.21.15.103'}, 'client.6591': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '172.21.15.103'}}}, 'global_metrics': {'cephfs': {'client.5243': [[419, 22], [0, 0], [0, 569929271], [0, 99043063], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 17810295], [620376031323031, 32], [0, 1547550], [211216113446114, 64]], 'client.5285': [[2, 0], [0, 0], [0, 0], [0, 1610257], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1610257], [0, 1]], 'client.5288': [[2, 0], [0, 0], [0, 0], [0, 1085297], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1085297], [0, 1]], 'client.6591': [[52, 2], [0, 0], [0, 0], [0, 34703565], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 3855952], [102641700855632, 9]]}}, 'metrics': {'delayed_ranks': [], 'mds.0': {'client.5243': [], 'client.5285': [], 'client.5288': [], 'client.6591': []}, 'mds.1': {'client.5243': []}}} 2024-04-16T16:17:49.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:49.573 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls) 2024-04-16T16:17:49.627 INFO:teuthology.orchestra.run.smithi045.stdout:test_mds_metrics_0 2024-04-16T16:17:49.627 INFO:teuthology.orchestra.run.smithi045.stdout:test_mds_metrics_1 2024-04-16T16:17:49.629 INFO:tasks.cephfs.test_mds_metrics:cleaning directory test_mds_metrics_0 2024-04-16T16:17:49.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:49.629 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf test_mds_metrics_0) 2024-04-16T16:17:49.803 INFO:tasks.cephfs.test_mds_metrics:cleaning directory test_mds_metrics_1 2024-04-16T16:17:49.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:49.804 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf test_mds_metrics_1) 2024-04-16T16:17:50.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable stats 2024-04-16T16:17:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.277+0000 7fde17400640 1 Processor -- start 2024-04-16T16:17:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.277+0000 7fde17400640 1 -- start start 2024-04-16T16:17:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.279+0000 7fde17400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fde100afda0 0x7fde100b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.279+0000 7fde17400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde100ccb70 con 0x7fde100d0e80 2024-04-16T16:17:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.279+0000 7fde17400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde100cce50 con 0x7fde1015b920 2024-04-16T16:17:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.279+0000 7fde17400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fde1006e7a0 con 0x7fde100afda0 2024-04-16T16:17:50.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.279+0000 7fde0fe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fde100d0e80 0x7fde100ce5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59310/0 (socket says 172.21.15.45:59310) 2024-04-16T16:17:50.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.279+0000 7fde0fe00640 1 -- 172.21.15.45:0/708076780 learned_addr learned my addr 172.21.15.45:0/708076780 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:50.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.279+0000 7fde15400640 1 --2- 172.21.15.45:0/708076780 >> v2:172.21.15.45:3300/0 conn(0x7fde100afda0 0x7fde100b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:50.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.280+0000 7fde15400640 1 -- 172.21.15.45:0/708076780 >> v1:172.21.15.103:6789/0 conn(0x7fde1015b920 legacy=0x7fde100d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:50.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.280+0000 7fde15400640 1 -- 172.21.15.45:0/708076780 >> v1:172.21.15.45:6789/0 conn(0x7fde100d0e80 legacy=0x7fde100ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:50.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.280+0000 7fde15400640 1 -- 172.21.15.45:0/708076780 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1006e430 con 0x7fde100afda0 2024-04-16T16:17:50.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.281+0000 7fde15400640 1 --2- 172.21.15.45:0/708076780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde100afda0 0x7fde100b4e20 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fde1006e920 tx=0x7fde0802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d0c3dd4a94595 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.281+0000 7fde0f400640 1 -- 172.21.15.45:0/708076780 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde08035020 con 0x7fde100afda0 2024-04-16T16:17:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.281+0000 7fde0f400640 1 -- 172.21.15.45:0/708076780 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde08031830 con 0x7fde100afda0 2024-04-16T16:17:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.281+0000 7fde0f400640 1 -- 172.21.15.45:0/708076780 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde08031c00 con 0x7fde100afda0 2024-04-16T16:17:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.282+0000 7fde17400640 1 -- 172.21.15.45:0/708076780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde100afda0 msgr2=0x7fde100b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.282+0000 7fde17400640 1 --2- 172.21.15.45:0/708076780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde100afda0 0x7fde100b4e20 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fde1006e920 tx=0x7fde0802b8d0 comp rx=0 tx=0).stop 2024-04-16T16:17:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.282+0000 7fde17400640 1 -- 172.21.15.45:0/708076780 shutdown_connections 2024-04-16T16:17:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.282+0000 7fde17400640 1 --2- 172.21.15.45:0/708076780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde100afda0 0x7fde100b4e20 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.282+0000 7fde17400640 1 -- 172.21.15.45:0/708076780 >> 172.21.15.45:0/708076780 conn(0x7fde100b3970 msgr2=0x7fde100b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.282+0000 7fde17400640 1 -- 172.21.15.45:0/708076780 shutdown_connections 2024-04-16T16:17:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.283+0000 7fde17400640 1 -- 172.21.15.45:0/708076780 wait complete. 2024-04-16T16:17:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.284+0000 7fde17400640 1 Processor -- start 2024-04-16T16:17:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.285+0000 7fde17400640 1 -- start start 2024-04-16T16:17:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.285+0000 7fde17400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde100afda0 0x7fde100b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.285+0000 7fde17400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde100d0e80 0x7fde100aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.286+0000 7fde17400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde1015b920 0x7fde1015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.286+0000 7fde17400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fde1006f000 con 0x7fde1015b920 2024-04-16T16:17:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.286+0000 7fde17400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde1006ee80 con 0x7fde100d0e80 2024-04-16T16:17:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.286+0000 7fde17400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fde1006f180 con 0x7fde100afda0 2024-04-16T16:17:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.286+0000 7fde15400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde1015b920 0x7fde1015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.286+0000 7fde15400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde1015b920 0x7fde1015bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48872/0 (socket says 172.21.15.45:48872) 2024-04-16T16:17:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.286+0000 7fde15400640 1 -- 172.21.15.45:0/2055334550 learned_addr learned my addr 172.21.15.45:0/2055334550 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.286+0000 7fde0fe00640 1 --2- 172.21.15.45:0/2055334550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde100d0e80 0x7fde100aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.286+0000 7fde15400640 1 -- 172.21.15.45:0/2055334550 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde100afda0 msgr2=0x7fde100b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.286+0000 7fde15400640 1 --2- 172.21.15.45:0/2055334550 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde100afda0 0x7fde100b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.286+0000 7fde15400640 1 -- 172.21.15.45:0/2055334550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde100d0e80 msgr2=0x7fde100aeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.286+0000 7fde15400640 1 --2- 172.21.15.45:0/2055334550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde100d0e80 0x7fde100aeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.286+0000 7fde15400640 1 -- 172.21.15.45:0/2055334550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde100ada80 con 0x7fde1015b920 2024-04-16T16:17:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.286+0000 7fde0fe00640 1 --2- 172.21.15.45:0/2055334550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde100d0e80 0x7fde100aeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.287+0000 7fde15400640 1 --2- 172.21.15.45:0/2055334550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde1015b920 0x7fde1015bd00 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fde08009890 tx=0x7fde080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.287+0000 7fde0cc00640 1 -- 172.21.15.45:0/2055334550 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0803b9c0 con 0x7fde1015b920 2024-04-16T16:17:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.287+0000 7fde0cc00640 1 -- 172.21.15.45:0/2055334550 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde0803bb60 con 0x7fde1015b920 2024-04-16T16:17:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.287+0000 7fde17400640 1 -- 172.21.15.45:0/2055334550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde100add60 con 0x7fde1015b920 2024-04-16T16:17:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.287+0000 7fde0cc00640 1 -- 172.21.15.45:0/2055334550 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde080453f0 con 0x7fde1015b920 2024-04-16T16:17:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.287+0000 7fde17400640 1 -- 172.21.15.45:0/2055334550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde100ae320 con 0x7fde1015b920 2024-04-16T16:17:50.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.289+0000 7fde17400640 1 -- 172.21.15.45:0/2055334550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde10075780 con 0x7fde1015b920 2024-04-16T16:17:50.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.290+0000 7fde0cc00640 1 -- 172.21.15.45:0/2055334550 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) ==== 91151+0+0 (secure 0 0 0) 0x7fde08038020 con 0x7fde1015b920 2024-04-16T16:17:50.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.290+0000 7fde0cc00640 1 --2- 172.21.15.45:0/2055334550 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fddec06ca60 0x7fddec06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:50.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.291+0000 7fde0cc00640 1 -- 172.21.15.45:0/2055334550 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7fde080ade70 con 0x7fde1015b920 2024-04-16T16:17:50.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.291+0000 7fde14a00640 1 --2- 172.21.15.45:0/2055334550 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fddec06ca60 0x7fddec06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.293+0000 7fde14a00640 1 --2- 172.21.15.45:0/2055334550 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fddec06ca60 0x7fddec06ef20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fde00006fd0 tx=0x7fde00006bc0 comp rx=0 tx=0).ready entity=mgr.5620 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.296+0000 7fde0cc00640 1 -- 172.21.15.45:0/2055334550 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fde08040070 con 0x7fde1015b920 2024-04-16T16:17:50.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.558+0000 7fde17400640 1 -- 172.21.15.45:0/2055334550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "stats"} v 0) -- 0x7fde100d0310 con 0x7fde1015b920 2024-04-16T16:17:50.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.793+0000 7fde0cc00640 1 -- 172.21.15.45:0/2055334550 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "stats"}]=0 v21) ==== 85+0+0 (secure 0 0 0) 0x7fde0807bf50 con 0x7fde1015b920 2024-04-16T16:17:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.796+0000 7fde17400640 1 -- 172.21.15.45:0/2055334550 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fddec06ca60 msgr2=0x7fddec06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.796+0000 7fde17400640 1 --2- 172.21.15.45:0/2055334550 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fddec06ca60 0x7fddec06ef20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fde00006fd0 tx=0x7fde00006bc0 comp rx=0 tx=0).stop 2024-04-16T16:17:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.796+0000 7fde17400640 1 -- 172.21.15.45:0/2055334550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde1015b920 msgr2=0x7fde1015bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.796+0000 7fde17400640 1 --2- 172.21.15.45:0/2055334550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde1015b920 0x7fde1015bd00 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fde08009890 tx=0x7fde080046c0 comp rx=0 tx=0).stop 2024-04-16T16:17:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.797+0000 7fde17400640 1 -- 172.21.15.45:0/2055334550 shutdown_connections 2024-04-16T16:17:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.797+0000 7fde17400640 1 --2- 172.21.15.45:0/2055334550 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7fddec06ca60 0x7fddec06ef20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.797+0000 7fde17400640 1 --2- 172.21.15.45:0/2055334550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde1015b920 0x7fde1015bd00 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.797+0000 7fde17400640 1 --2- 172.21.15.45:0/2055334550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde100d0e80 0x7fde100aeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.797+0000 7fde17400640 1 --2- 172.21.15.45:0/2055334550 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde100afda0 0x7fde100b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.797+0000 7fde17400640 1 -- 172.21.15.45:0/2055334550 >> 172.21.15.45:0/2055334550 conn(0x7fde100b3970 msgr2=0x7fde100cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.797+0000 7fde17400640 1 -- 172.21.15.45:0/2055334550 shutdown_connections 2024-04-16T16:17:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:50.799+0000 7fde17400640 1 -- 172.21.15.45:0/2055334550 wait complete. 2024-04-16T16:17:50.821 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-16T16:17:50.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:17:50.934 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:17:51.025 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-16T16:17:51.025 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:17:51.025 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:51.025 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:51.082 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:51.082 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:51.130 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:17:51.339 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:51.339+0000 7f51fbe3e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:17:51.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.373+0000 7f576be00640 1 Processor -- start 2024-04-16T16:17:51.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.373+0000 7f576be00640 1 -- start start 2024-04-16T16:17:51.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.374+0000 7f576be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f57641462e0 0x7f5764072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:51.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.374+0000 7f576be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57640728c0 0x7f57640ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:51.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.374+0000 7f576be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57640cf6d0 con 0x7f57640cec00 2024-04-16T16:17:51.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.374+0000 7f576be00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5764072d20 con 0x7f57641462e0 2024-04-16T16:17:51.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.374+0000 7f576be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f57641466c0 con 0x7f57640728c0 2024-04-16T16:17:51.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.375+0000 7f5768a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57640728c0 0x7f57640ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:51.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.375+0000 7f5768a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57640728c0 0x7f57640ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:46780/0 (socket says 172.21.15.45:46780) 2024-04-16T16:17:51.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.375+0000 7f5768a00640 1 -- 172.21.15.45:0/3390154822 learned_addr learned my addr 172.21.15.45:0/3390154822 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:51.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.375+0000 7f5769400640 1 --2- 172.21.15.45:0/3390154822 >> v2:172.21.15.103:3300/0 conn(0x7f57641462e0 0x7f5764072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:51.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.376+0000 7f5769400640 1 -- 172.21.15.45:0/3390154822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57640728c0 msgr2=0x7f57640ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:51.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.376+0000 7f5769400640 1 --2- 172.21.15.45:0/3390154822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57640728c0 0x7f57640ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:51.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.376+0000 7f5769400640 1 -- 172.21.15.45:0/3390154822 >> v1:172.21.15.103:6789/0 conn(0x7f57640cec00 legacy=0x7f576415da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:51.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.376+0000 7f5769400640 1 -- 172.21.15.45:0/3390154822 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57640cf3f0 con 0x7f57641462e0 2024-04-16T16:17:51.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.377+0000 7f5769400640 1 --2- 172.21.15.45:0/3390154822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57641462e0 0x7f5764072380 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f57580097a0 tx=0x7f575802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81d9322fad7bb18 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:51.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.378+0000 7f5763e00640 1 -- 172.21.15.45:0/3390154822 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57580315a0 con 0x7f57641462e0 2024-04-16T16:17:51.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.378+0000 7f5763e00640 1 -- 172.21.15.45:0/3390154822 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5758037070 con 0x7f57641462e0 2024-04-16T16:17:51.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.378+0000 7f5763e00640 1 -- 172.21.15.45:0/3390154822 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758031960 con 0x7f57641462e0 2024-04-16T16:17:51.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.378+0000 7f576be00640 1 -- 172.21.15.45:0/3390154822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57641462e0 msgr2=0x7f5764072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:51.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.378+0000 7f576be00640 1 --2- 172.21.15.45:0/3390154822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57641462e0 0x7f5764072380 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f57580097a0 tx=0x7f575802b840 comp rx=0 tx=0).stop 2024-04-16T16:17:51.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.379+0000 7f576be00640 1 -- 172.21.15.45:0/3390154822 shutdown_connections 2024-04-16T16:17:51.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.379+0000 7f576be00640 1 --2- 172.21.15.45:0/3390154822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57640728c0 0x7f57640ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:51.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.379+0000 7f576be00640 1 --2- 172.21.15.45:0/3390154822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57641462e0 0x7f5764072380 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:51.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.379+0000 7f576be00640 1 -- 172.21.15.45:0/3390154822 >> 172.21.15.45:0/3390154822 conn(0x7f57640b2750 msgr2=0x7f57640b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:51.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.379+0000 7f576be00640 1 -- 172.21.15.45:0/3390154822 shutdown_connections 2024-04-16T16:17:51.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.379+0000 7f576be00640 1 -- 172.21.15.45:0/3390154822 wait complete. 2024-04-16T16:17:51.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.381+0000 7f576be00640 1 Processor -- start 2024-04-16T16:17:51.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.382+0000 7f576be00640 1 -- start start 2024-04-16T16:17:51.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.383+0000 7f576be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57640728c0 0x7f57641e5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:51.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.383+0000 7f576be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57640cec00 0x7f5764141b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:51.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.383+0000 7f576be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57641462e0 0x7f5764142070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:51.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.383+0000 7f576be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f576415d540 con 0x7f57640cec00 2024-04-16T16:17:51.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.383+0000 7f576be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f576415d260 con 0x7f57640728c0 2024-04-16T16:17:51.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.383+0000 7f576be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f57640cf090 con 0x7f57641462e0 2024-04-16T16:17:51.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.383+0000 7f5768a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57640cec00 0x7f5764141b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:51.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.384+0000 7f5768a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57640cec00 0x7f5764141b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48886/0 (socket says 172.21.15.45:48886) 2024-04-16T16:17:51.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.384+0000 7f5768a00640 1 -- 172.21.15.45:0/1123729750 learned_addr learned my addr 172.21.15.45:0/1123729750 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:51.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.384+0000 7f5769400640 1 --2- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57640728c0 0x7f57641e5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:51.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.384+0000 7f5769e00640 1 --2- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57641462e0 0x7f5764142070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:51.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.384+0000 7f5768a00640 1 -- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57641462e0 msgr2=0x7f5764142070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:51.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.384+0000 7f5768a00640 1 --2- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57641462e0 0x7f5764142070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:51.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.384+0000 7f5768a00640 1 -- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57640728c0 msgr2=0x7f57641e5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:51.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.384+0000 7f5768a00640 1 --2- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57640728c0 0x7f57641e5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:51.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.384+0000 7f5768a00640 1 -- 172.21.15.45:0/1123729750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57640adeb0 con 0x7f57640cec00 2024-04-16T16:17:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.384+0000 7f5768a00640 1 --2- 172.21.15.45:0/1123729750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57640cec00 0x7f5764141b30 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f575000e870 tx=0x7f575000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.385+0000 7f5761600640 1 -- 172.21.15.45:0/1123729750 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5750017070 con 0x7f57640cec00 2024-04-16T16:17:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.385+0000 7f5761600640 1 -- 172.21.15.45:0/1123729750 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f575000c940 con 0x7f57640cec00 2024-04-16T16:17:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.385+0000 7f5761600640 1 -- 172.21.15.45:0/1123729750 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575000cd30 con 0x7f57640cec00 2024-04-16T16:17:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.385+0000 7f576be00640 1 -- 172.21.15.45:0/1123729750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57640ae190 con 0x7f57640cec00 2024-04-16T16:17:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.385+0000 7f576be00640 1 -- 172.21.15.45:0/1123729750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57640ae750 con 0x7f57640cec00 2024-04-16T16:17:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.386+0000 7f576be00640 1 -- 172.21.15.45:0/1123729750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5764075780 con 0x7f57640cec00 2024-04-16T16:17:51.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.388+0000 7f5761600640 1 -- 172.21.15.45:0/1123729750 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 21) ==== 91142+0+0 (secure 0 0 0) 0x7f575001a070 con 0x7f57640cec00 2024-04-16T16:17:51.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.389+0000 7f5761600640 1 --2- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f574406ca10 0x7f574406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:51.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.389+0000 7f5761600640 1 -- 172.21.15.45:0/1123729750 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f575008c200 con 0x7f57640cec00 2024-04-16T16:17:51.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.389+0000 7f5769400640 1 -- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f574406ca10 msgr2=0x7f574406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/541757982 2024-04-16T16:17:51.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.389+0000 7f5769400640 1 --2- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f574406ca10 0x7f574406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:17:51.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.394+0000 7f5761600640 1 -- 172.21.15.45:0/1123729750 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f575005a2d0 con 0x7f57640cec00 2024-04-16T16:17:51.395 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:51.395+0000 7f51fbe3e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:17:51.399 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:51.399+0000 7f24c2a0d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:17:51.457 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:51.457+0000 7f24c2a0d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:17:51.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.590+0000 7f5769400640 1 -- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f574406ca10 msgr2=0x7f574406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/541757982 2024-04-16T16:17:51.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.590+0000 7f5769400640 1 --2- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f574406ca10 0x7f574406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:17:51.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.599+0000 7f576be00640 1 -- 172.21.15.45:0/1123729750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f576415db90 con 0x7f57640cec00 2024-04-16T16:17:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.601+0000 7f5761600640 1 -- 172.21.15.45:0/1123729750 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+17299 (secure 0 0 0) 0x7f5750059bd0 con 0x7f57640cec00 2024-04-16T16:17:51.601 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:51.602 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":73,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:17:01.408977+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:16:37.449973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-16T16:16:38.485027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6833/4155603915":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/951047877":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/3176524506":"2024-04-17T16:17:01.408806+0000","172.21.15.103:6832/4155603915":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/365521086":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/3345679536":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/3606536347":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/1223877773":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/252084277":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/650078727":"2024-04-17T16:17:01.408806+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:17:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.604+0000 7f576be00640 1 -- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f574406ca10 msgr2=0x7f574406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:17:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.604+0000 7f576be00640 1 --2- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f574406ca10 0x7f574406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.604+0000 7f576be00640 1 -- 172.21.15.45:0/1123729750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57640cec00 msgr2=0x7f5764141b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.604+0000 7f576be00640 1 --2- 172.21.15.45:0/1123729750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57640cec00 0x7f5764141b30 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f575000e870 tx=0x7f575000ed30 comp rx=0 tx=0).stop 2024-04-16T16:17:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.604+0000 7f576be00640 1 -- 172.21.15.45:0/1123729750 shutdown_connections 2024-04-16T16:17:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.604+0000 7f576be00640 1 --2- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f574406ca10 0x7f574406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.604+0000 7f576be00640 1 --2- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57641462e0 0x7f5764142070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.604+0000 7f576be00640 1 --2- 172.21.15.45:0/1123729750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57640cec00 0x7f5764141b30 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.604+0000 7f576be00640 1 --2- 172.21.15.45:0/1123729750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57640728c0 0x7f57641e5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.604+0000 7f576be00640 1 -- 172.21.15.45:0/1123729750 >> 172.21.15.45:0/1123729750 conn(0x7f57640b2750 msgr2=0x7f576406ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.604+0000 7f576be00640 1 -- 172.21.15.45:0/1123729750 shutdown_connections 2024-04-16T16:17:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.604+0000 7f576be00640 1 -- 172.21.15.45:0/1123729750 wait complete. 2024-04-16T16:17:51.622 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:16:37.449973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:16:38.485027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:17:51.623 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:17:51.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.828+0000 7f8d4c200640 1 Processor -- start 2024-04-16T16:17:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.828+0000 7f8d4c200640 1 -- start start 2024-04-16T16:17:51.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.829+0000 7f8d4c200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8d4415f060 0x7f8d440d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:51.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.829+0000 7f8d4c200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d440b7ca0 con 0x7f8d4415ec80 2024-04-16T16:17:51.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.829+0000 7f8d4c200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d440d3450 con 0x7f8d440b6c00 2024-04-16T16:17:51.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.829+0000 7f8d4c200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8d440d3220 con 0x7f8d4415f060 2024-04-16T16:17:51.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.830+0000 7f8d49800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8d4415ec80 0x7f8d440b64f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59330/0 (socket says 172.21.15.45:59330) 2024-04-16T16:17:51.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.830+0000 7f8d49800640 1 -- 172.21.15.45:0/1962168390 learned_addr learned my addr 172.21.15.45:0/1962168390 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:51.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.830+0000 7f8d48e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8d4415f060 0x7f8d440d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:51.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.830+0000 7f8d43e00640 1 -- 172.21.15.45:0/1962168390 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4200802286 0 0) 0x7f8d440b7ca0 con 0x7f8d4415ec80 2024-04-16T16:17:51.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.830+0000 7f8d43e00640 1 -- 172.21.15.45:0/1962168390 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d28003520 con 0x7f8d4415ec80 2024-04-16T16:17:51.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.830+0000 7f8d48e00640 1 -- 172.21.15.45:0/1962168390 >> v1:172.21.15.103:6789/0 conn(0x7f8d440b6c00 legacy=0x7f8d44168990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:51.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.830+0000 7f8d48e00640 1 -- 172.21.15.45:0/1962168390 >> v1:172.21.15.45:6789/0 conn(0x7f8d4415ec80 legacy=0x7f8d440b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:51.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.830+0000 7f8d48e00640 1 -- 172.21.15.45:0/1962168390 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d440d3730 con 0x7f8d4415f060 2024-04-16T16:17:51.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.831+0000 7f8d48e00640 1 --2- 172.21.15.45:0/1962168390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d4415f060 0x7f8d440d2ce0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f8d300097a0 tx=0x7f8d3002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2209f96c7185036e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:51.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.832+0000 7f8d43e00640 1 -- 172.21.15.45:0/1962168390 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3002f810 con 0x7f8d4415f060 2024-04-16T16:17:51.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.832+0000 7f8d43e00640 1 -- 172.21.15.45:0/1962168390 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d3002f9b0 con 0x7f8d4415f060 2024-04-16T16:17:51.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.832+0000 7f8d43e00640 1 -- 172.21.15.45:0/1962168390 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3002fd80 con 0x7f8d4415f060 2024-04-16T16:17:51.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.832+0000 7f8d4c200640 1 -- 172.21.15.45:0/1962168390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d4415f060 msgr2=0x7f8d440d2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:51.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.832+0000 7f8d4c200640 1 --2- 172.21.15.45:0/1962168390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d4415f060 0x7f8d440d2ce0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f8d300097a0 tx=0x7f8d3002b9f0 comp rx=0 tx=0).stop 2024-04-16T16:17:51.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.833+0000 7f8d4c200640 1 -- 172.21.15.45:0/1962168390 shutdown_connections 2024-04-16T16:17:51.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.833+0000 7f8d4c200640 1 --2- 172.21.15.45:0/1962168390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d4415f060 0x7f8d440d2ce0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:51.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.833+0000 7f8d4c200640 1 -- 172.21.15.45:0/1962168390 >> 172.21.15.45:0/1962168390 conn(0x7f8d440d16d0 msgr2=0x7f8d440d1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:51.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.833+0000 7f8d4c200640 1 -- 172.21.15.45:0/1962168390 shutdown_connections 2024-04-16T16:17:51.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.833+0000 7f8d4c200640 1 -- 172.21.15.45:0/1962168390 wait complete. 2024-04-16T16:17:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.834+0000 7f8d4c200640 1 Processor -- start 2024-04-16T16:17:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.835+0000 7f8d4c200640 1 -- start start 2024-04-16T16:17:51.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.836+0000 7f8d4c200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d440b6c00 0x7f8d44155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.836+0000 7f8d4c200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d4415ec80 0x7f8d440c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.836+0000 7f8d4c200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d440ca3d0 0x7f8d440c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.836+0000 7f8d4c200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8d440ba190 con 0x7f8d440ca3d0 2024-04-16T16:17:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.836+0000 7f8d4c200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8d440d3cf0 con 0x7f8d4415ec80 2024-04-16T16:17:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.836+0000 7f8d4c200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8d440ba310 con 0x7f8d440b6c00 2024-04-16T16:17:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.836+0000 7f8d4a200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d440ca3d0 0x7f8d440c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.836+0000 7f8d4a200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d440ca3d0 0x7f8d440c8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48896/0 (socket says 172.21.15.45:48896) 2024-04-16T16:17:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.836+0000 7f8d4a200640 1 -- 172.21.15.45:0/2561475080 learned_addr learned my addr 172.21.15.45:0/2561475080 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.836+0000 7f8d48e00640 1 --2- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d4415ec80 0x7f8d440c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.836+0000 7f8d49800640 1 --2- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d440b6c00 0x7f8d44155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.837+0000 7f8d4a200640 1 -- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d440b6c00 msgr2=0x7f8d44155c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.837+0000 7f8d4a200640 1 --2- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d440b6c00 0x7f8d44155c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.837+0000 7f8d4a200640 1 -- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d4415ec80 msgr2=0x7f8d440c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.837+0000 7f8d4a200640 1 --2- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d4415ec80 0x7f8d440c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.837+0000 7f8d4a200640 1 -- 172.21.15.45:0/2561475080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d441561c0 con 0x7f8d440ca3d0 2024-04-16T16:17:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.837+0000 7f8d4a200640 1 --2- 172.21.15.45:0/2561475080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d440ca3d0 0x7f8d440c8ae0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00c3a0 tx=0x7f8d3c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.837+0000 7f8d41600640 1 -- 172.21.15.45:0/2561475080 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c017070 con 0x7f8d440ca3d0 2024-04-16T16:17:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.837+0000 7f8d41600640 1 -- 172.21.15.45:0/2561475080 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d3c0052d0 con 0x7f8d440ca3d0 2024-04-16T16:17:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.837+0000 7f8d41600640 1 -- 172.21.15.45:0/2561475080 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c003830 con 0x7f8d440ca3d0 2024-04-16T16:17:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.837+0000 7f8d4c200640 1 -- 172.21.15.45:0/2561475080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d440c9730 con 0x7f8d440ca3d0 2024-04-16T16:17:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.837+0000 7f8d4c200640 1 -- 172.21.15.45:0/2561475080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d44160f30 con 0x7f8d440ca3d0 2024-04-16T16:17:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.839+0000 7f8d4c200640 1 -- 172.21.15.45:0/2561475080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d44075780 con 0x7f8d440ca3d0 2024-04-16T16:17:51.839 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:51.840+0000 7f51fbe3e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:17:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.840+0000 7f8d41600640 1 -- 172.21.15.45:0/2561475080 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 21) ==== 91142+0+0 (secure 0 0 0) 0x7f8d3c01a070 con 0x7f8d440ca3d0 2024-04-16T16:17:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.841+0000 7f8d41600640 1 --2- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f8d2406ca40 0x7f8d2406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.841+0000 7f8d41600640 1 -- 172.21.15.45:0/2561475080 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f8d3c08b1e0 con 0x7f8d440ca3d0 2024-04-16T16:17:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.841+0000 7f8d49800640 1 -- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f8d2406ca40 msgr2=0x7f8d2406ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/541757982 2024-04-16T16:17:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.841+0000 7f8d49800640 1 --2- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f8d2406ca40 0x7f8d2406ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:17:51.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:51.845+0000 7f8d41600640 1 -- 172.21.15.45:0/2561475080 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d3c059410 con 0x7f8d440ca3d0 2024-04-16T16:17:51.892 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:51.893+0000 7f51fbe3e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:17:51.931 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:51.931+0000 7f24c2a0d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:17:51.943 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:51.944+0000 7f51fbe3e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:17:51.987 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:51.986+0000 7f24c2a0d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:17:52.004 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:52.004+0000 7f51fbe3e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:17:52.040 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:52.040+0000 7f24c2a0d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:17:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.042+0000 7f8d49800640 1 -- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f8d2406ca40 msgr2=0x7f8d2406ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/541757982 2024-04-16T16:17:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.042+0000 7f8d49800640 1 --2- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f8d2406ca40 0x7f8d2406ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:17:52.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.047+0000 7f8d4c200640 1 -- 172.21.15.45:0/2561475080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8d44161100 con 0x7f8d440ca3d0 2024-04-16T16:17:52.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.047+0000 7f8d41600640 1 -- 172.21.15.45:0/2561475080 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7f8d3c058d10 con 0x7f8d440ca3d0 2024-04-16T16:17:52.047 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:17:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.050+0000 7f8d4c200640 1 -- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f8d2406ca40 msgr2=0x7f8d2406ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:17:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.050+0000 7f8d4c200640 1 --2- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f8d2406ca40 0x7f8d2406ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.050+0000 7f8d4c200640 1 -- 172.21.15.45:0/2561475080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d440ca3d0 msgr2=0x7f8d440c8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.050+0000 7f8d4c200640 1 --2- 172.21.15.45:0/2561475080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d440ca3d0 0x7f8d440c8ae0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00c3a0 tx=0x7f8d3c00c860 comp rx=0 tx=0).stop 2024-04-16T16:17:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.050+0000 7f8d4c200640 1 -- 172.21.15.45:0/2561475080 shutdown_connections 2024-04-16T16:17:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.050+0000 7f8d4c200640 1 --2- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f8d2406ca40 0x7f8d2406ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.050+0000 7f8d4c200640 1 --2- 172.21.15.45:0/2561475080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d440ca3d0 0x7f8d440c8ae0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.050+0000 7f8d4c200640 1 --2- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d4415ec80 0x7f8d440c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.050+0000 7f8d4c200640 1 --2- 172.21.15.45:0/2561475080 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d440b6c00 0x7f8d44155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.050+0000 7f8d4c200640 1 -- 172.21.15.45:0/2561475080 >> 172.21.15.45:0/2561475080 conn(0x7f8d440d16d0 msgr2=0x7f8d440b9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.050+0000 7f8d4c200640 1 -- 172.21.15.45:0/2561475080 shutdown_connections 2024-04-16T16:17:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.050+0000 7f8d4c200640 1 -- 172.21.15.45:0/2561475080 wait complete. 2024-04-16T16:17:52.055 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:52.056+0000 7f51fbe3e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:17:52.071 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:17:52.110 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:52.110+0000 7f24c2a0d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:17:52.164 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:52.164+0000 7f24c2a0d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:17:52.188 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:52.189+0000 7f51fbe3e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:17:52.250 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:52.251+0000 7f51fbe3e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:17:52.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.271+0000 7f4d95600640 1 Processor -- start 2024-04-16T16:17:52.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.272+0000 7f4d95600640 1 -- start start 2024-04-16T16:17:52.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.273+0000 7f4d95600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4d900d10e0 0x7f4d90072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:52.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.273+0000 7f4d95600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d900b4930 0x7f4d9006f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:52.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.273+0000 7f4d95600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d900b5530 con 0x7f4d9015c7c0 2024-04-16T16:17:52.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.273+0000 7f4d95600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4d900b52e0 con 0x7f4d900d10e0 2024-04-16T16:17:52.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.273+0000 7f4d95600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4d900d0260 con 0x7f4d900b4930 2024-04-16T16:17:52.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.274+0000 7f4d87e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4d900d10e0 0x7f4d90072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:52.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.274+0000 7f4d8f400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d900b4930 0x7f4d9006f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:52.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.274+0000 7f4d87e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4d900d10e0 0x7f4d90072e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:43174/0 (socket says 172.21.15.45:43174) 2024-04-16T16:17:52.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.274+0000 7f4d87e00640 1 -- 172.21.15.45:0/2705789492 learned_addr learned my addr 172.21.15.45:0/2705789492 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:52.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.275+0000 7f4d87e00640 1 -- 172.21.15.45:0/2705789492 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d900b4930 msgr2=0x7f4d9006f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:52.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.275+0000 7f4d87e00640 1 --2- 172.21.15.45:0/2705789492 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d900b4930 0x7f4d9006f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.275+0000 7f4d87e00640 1 -- 172.21.15.45:0/2705789492 >> v1:172.21.15.103:6789/0 conn(0x7f4d9015c7c0 legacy=0x7f4d900d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:52.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.275+0000 7f4d87e00640 1 -- 172.21.15.45:0/2705789492 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d900b5810 con 0x7f4d900d10e0 2024-04-16T16:17:52.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.277+0000 7f4d87e00640 1 --2- 172.21.15.45:0/2705789492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d900d10e0 0x7f4d90072e10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009870 tx=0x7f4d7802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83364100b0f45dc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:52.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.277+0000 7f4d8e000640 1 -- 172.21.15.45:0/2705789492 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d780317c0 con 0x7f4d900d10e0 2024-04-16T16:17:52.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.277+0000 7f4d8e000640 1 -- 172.21.15.45:0/2705789492 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d78037070 con 0x7f4d900d10e0 2024-04-16T16:17:52.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.277+0000 7f4d8e000640 1 -- 172.21.15.45:0/2705789492 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d78031b80 con 0x7f4d900d10e0 2024-04-16T16:17:52.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.278+0000 7f4d95600640 1 -- 172.21.15.45:0/2705789492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d900d10e0 msgr2=0x7f4d90072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:52.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.278+0000 7f4d95600640 1 --2- 172.21.15.45:0/2705789492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d900d10e0 0x7f4d90072e10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009870 tx=0x7f4d7802b860 comp rx=0 tx=0).stop 2024-04-16T16:17:52.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.278+0000 7f4d95600640 1 -- 172.21.15.45:0/2705789492 shutdown_connections 2024-04-16T16:17:52.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.278+0000 7f4d95600640 1 --2- 172.21.15.45:0/2705789492 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d900b4930 0x7f4d9006f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.278+0000 7f4d95600640 1 --2- 172.21.15.45:0/2705789492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d900d10e0 0x7f4d90072e10 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.278+0000 7f4d95600640 1 -- 172.21.15.45:0/2705789492 >> 172.21.15.45:0/2705789492 conn(0x7f4d900b3970 msgr2=0x7f4d900b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:52.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.278+0000 7f4d95600640 1 -- 172.21.15.45:0/2705789492 shutdown_connections 2024-04-16T16:17:52.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.278+0000 7f4d95600640 1 -- 172.21.15.45:0/2705789492 wait complete. 2024-04-16T16:17:52.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.280+0000 7f4d95600640 1 Processor -- start 2024-04-16T16:17:52.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.280+0000 7f4d95600640 1 -- start start 2024-04-16T16:17:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.281+0000 7f4d95600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d900b4930 0x7f4d901f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.281+0000 7f4d95600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d900d10e0 0x7f4d9014a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.281+0000 7f4d95600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d9015c7c0 0x7f4d901e5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.281+0000 7f4d95600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9006f6a0 con 0x7f4d900b4930 2024-04-16T16:17:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.281+0000 7f4d95600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4d90073350 con 0x7f4d9015c7c0 2024-04-16T16:17:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.281+0000 7f4d95600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4d9006f820 con 0x7f4d900d10e0 2024-04-16T16:17:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.282+0000 7f4d8ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d900b4930 0x7f4d901f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.282+0000 7f4d8ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d900b4930 0x7f4d901f8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48904/0 (socket says 172.21.15.45:48904) 2024-04-16T16:17:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.282+0000 7f4d8ea00640 1 -- 172.21.15.45:0/4102175409 learned_addr learned my addr 172.21.15.45:0/4102175409 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.282+0000 7f4d87e00640 1 --2- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d900d10e0 0x7f4d9014a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.282+0000 7f4d8f400640 1 --2- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d9015c7c0 0x7f4d901e5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.282+0000 7f4d8ea00640 1 -- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d900d10e0 msgr2=0x7f4d9014a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.282+0000 7f4d8ea00640 1 --2- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d900d10e0 0x7f4d9014a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.282+0000 7f4d8ea00640 1 -- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d9015c7c0 msgr2=0x7f4d901e5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.282+0000 7f4d8ea00640 1 --2- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d9015c7c0 0x7f4d901e5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.282+0000 7f4d8ea00640 1 -- 172.21.15.45:0/4102175409 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d9014aa50 con 0x7f4d900b4930 2024-04-16T16:17:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.283+0000 7f4d8ea00640 1 --2- 172.21.15.45:0/4102175409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d900b4930 0x7f4d901f8a30 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000c340 tx=0x7f4d8000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.283+0000 7f4d86a00640 1 -- 172.21.15.45:0/4102175409 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d80017070 con 0x7f4d900b4930 2024-04-16T16:17:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.283+0000 7f4d86a00640 1 -- 172.21.15.45:0/4102175409 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d800052d0 con 0x7f4d900b4930 2024-04-16T16:17:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.283+0000 7f4d95600640 1 -- 172.21.15.45:0/4102175409 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d90159890 con 0x7f4d900b4930 2024-04-16T16:17:52.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.283+0000 7f4d86a00640 1 -- 172.21.15.45:0/4102175409 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d80003850 con 0x7f4d900b4930 2024-04-16T16:17:52.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.283+0000 7f4d95600640 1 -- 172.21.15.45:0/4102175409 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d901e5200 con 0x7f4d900b4930 2024-04-16T16:17:52.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.284+0000 7f4d95600640 1 -- 172.21.15.45:0/4102175409 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d90075780 con 0x7f4d900b4930 2024-04-16T16:17:52.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.285+0000 7f4d86a00640 1 -- 172.21.15.45:0/4102175409 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 21) ==== 91142+0+0 (secure 0 0 0) 0x7f4d8001a070 con 0x7f4d900b4930 2024-04-16T16:17:52.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.286+0000 7f4d86a00640 1 --2- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4d6c06ca10 0x7f4d6c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:52.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.286+0000 7f4d86a00640 1 -- 172.21.15.45:0/4102175409 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f4d8008b160 con 0x7f4d900b4930 2024-04-16T16:17:52.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.286+0000 7f4d87e00640 1 -- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4d6c06ca10 msgr2=0x7f4d6c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/541757982 2024-04-16T16:17:52.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.286+0000 7f4d87e00640 1 --2- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4d6c06ca10 0x7f4d6c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:17:52.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.290+0000 7f4d86a00640 1 -- 172.21.15.45:0/4102175409 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d9014aa50 con 0x7f4d900b4930 2024-04-16T16:17:52.302 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:52.303+0000 7f51fbe3e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:17:52.305 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:52.305+0000 7f24c2a0d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:17:52.359 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:52.359+0000 7f51fbe3e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:17:52.371 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:52.372+0000 7f24c2a0d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:17:52.409 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:52.410+0000 7f51fbe3e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:17:52.424 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:52.424+0000 7f24c2a0d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:17:52.485 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:52.485+0000 7f24c2a0d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:17:52.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.487+0000 7f4d87e00640 1 -- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4d6c06ca10 msgr2=0x7f4d6c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/541757982 2024-04-16T16:17:52.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.487+0000 7f4d87e00640 1 --2- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4d6c06ca10 0x7f4d6c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:17:52.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.494+0000 7f4d95600640 1 -- 172.21.15.45:0/4102175409 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4d9006f1f0 con 0x7f4d900b4930 2024-04-16T16:17:52.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.495+0000 7f4d86a00640 1 -- 172.21.15.45:0/4102175409 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v73) ==== 104+0+11 (secure 0 0 0) 0x7f4d80058be0 con 0x7f4d900b4930 2024-04-16T16:17:52.495 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:17:52.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.497+0000 7f4d95600640 1 -- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4d6c06ca10 msgr2=0x7f4d6c06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:17:52.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.497+0000 7f4d95600640 1 --2- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4d6c06ca10 0x7f4d6c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.497+0000 7f4d95600640 1 -- 172.21.15.45:0/4102175409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d900b4930 msgr2=0x7f4d901f8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:52.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.497+0000 7f4d95600640 1 --2- 172.21.15.45:0/4102175409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d900b4930 0x7f4d901f8a30 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000c340 tx=0x7f4d8000c800 comp rx=0 tx=0).stop 2024-04-16T16:17:52.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.497+0000 7f4d95600640 1 -- 172.21.15.45:0/4102175409 shutdown_connections 2024-04-16T16:17:52.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.497+0000 7f4d95600640 1 --2- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f4d6c06ca10 0x7f4d6c06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.497+0000 7f4d95600640 1 --2- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d9015c7c0 0x7f4d901e5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.498+0000 7f4d95600640 1 --2- 172.21.15.45:0/4102175409 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d900d10e0 0x7f4d9014a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.498+0000 7f4d95600640 1 --2- 172.21.15.45:0/4102175409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d900b4930 0x7f4d901f8a30 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.498+0000 7f4d95600640 1 -- 172.21.15.45:0/4102175409 >> 172.21.15.45:0/4102175409 conn(0x7f4d900b3970 msgr2=0x7f4d900b2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:52.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.498+0000 7f4d95600640 1 -- 172.21.15.45:0/4102175409 shutdown_connections 2024-04-16T16:17:52.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.498+0000 7f4d95600640 1 -- 172.21.15.45:0/4102175409 wait complete. 2024-04-16T16:17:52.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:17:52.540 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:52.540+0000 7f24c2a0d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:17:52.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.754+0000 7f6a7d800640 1 Processor -- start 2024-04-16T16:17:52.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.755+0000 7f6a7d800640 1 -- start start 2024-04-16T16:17:52.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.756+0000 7f6a7d800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6a7815e980 0x7f6a780d4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:52.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.756+0000 7f6a7d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a780ba500 0x7f6a780d2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:52.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.756+0000 7f6a7d800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a7815ed60 con 0x7f6a780cf4d0 2024-04-16T16:17:52.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.756+0000 7f6a7d800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6a780d3170 con 0x7f6a7815e980 2024-04-16T16:17:52.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.756+0000 7f6a7d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6a780d3450 con 0x7f6a780ba500 2024-04-16T16:17:52.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.757+0000 7f6a7ce00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6a780cf4d0 0x7f6a780cf960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59342/0 (socket says 172.21.15.45:59342) 2024-04-16T16:17:52.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.757+0000 7f6a7ce00640 1 -- 172.21.15.45:0/4034696630 learned_addr learned my addr 172.21.15.45:0/4034696630 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:52.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.757+0000 7f6a77400640 1 --2- 172.21.15.45:0/4034696630 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a780ba500 0x7f6a780d2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:52.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.757+0000 7f6a77e00640 1 --2- 172.21.15.45:0/4034696630 >> v2:172.21.15.103:3300/0 conn(0x7f6a7815e980 0x7f6a780d4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:52.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.757+0000 7f6a76a00640 1 -- 172.21.15.45:0/4034696630 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 768528114 0 0) 0x7f6a7815ed60 con 0x7f6a780cf4d0 2024-04-16T16:17:52.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.757+0000 7f6a76a00640 1 -- 172.21.15.45:0/4034696630 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a5c003520 con 0x7f6a780cf4d0 2024-04-16T16:17:52.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.757+0000 7f6a77e00640 1 -- 172.21.15.45:0/4034696630 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a780ba500 msgr2=0x7f6a780d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:52.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.757+0000 7f6a77e00640 1 --2- 172.21.15.45:0/4034696630 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a780ba500 0x7f6a780d2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.757+0000 7f6a77e00640 1 -- 172.21.15.45:0/4034696630 >> v1:172.21.15.45:6789/0 conn(0x7f6a780cf4d0 legacy=0x7f6a780cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:52.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.757+0000 7f6a77e00640 1 -- 172.21.15.45:0/4034696630 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a780d3730 con 0x7f6a7815e980 2024-04-16T16:17:52.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.759+0000 7f6a77e00640 1 --2- 172.21.15.45:0/4034696630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7815e980 0x7f6a780d4210 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6a680097a0 tx=0x7f6a6802b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f393d7c6102a7d38 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:52.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.759+0000 7f6a76a00640 1 -- 172.21.15.45:0/4034696630 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68034070 con 0x7f6a7815e980 2024-04-16T16:17:52.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.759+0000 7f6a76a00640 1 -- 172.21.15.45:0/4034696630 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a68030800 con 0x7f6a7815e980 2024-04-16T16:17:52.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.759+0000 7f6a76a00640 1 -- 172.21.15.45:0/4034696630 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68030bb0 con 0x7f6a7815e980 2024-04-16T16:17:52.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.760+0000 7f6a7d800640 1 -- 172.21.15.45:0/4034696630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7815e980 msgr2=0x7f6a780d4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:52.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.760+0000 7f6a7d800640 1 --2- 172.21.15.45:0/4034696630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7815e980 0x7f6a780d4210 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6a680097a0 tx=0x7f6a6802b940 comp rx=0 tx=0).stop 2024-04-16T16:17:52.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.760+0000 7f6a7d800640 1 -- 172.21.15.45:0/4034696630 shutdown_connections 2024-04-16T16:17:52.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.760+0000 7f6a7d800640 1 --2- 172.21.15.45:0/4034696630 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a780ba500 0x7f6a780d2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.760+0000 7f6a7d800640 1 --2- 172.21.15.45:0/4034696630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7815e980 0x7f6a780d4210 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.760+0000 7f6a7d800640 1 -- 172.21.15.45:0/4034696630 >> 172.21.15.45:0/4034696630 conn(0x7f6a7806d2f0 msgr2=0x7f6a780d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:52.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.760+0000 7f6a7d800640 1 -- 172.21.15.45:0/4034696630 shutdown_connections 2024-04-16T16:17:52.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.760+0000 7f6a7d800640 1 -- 172.21.15.45:0/4034696630 wait complete. 2024-04-16T16:17:52.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.762+0000 7f6a7d800640 1 Processor -- start 2024-04-16T16:17:52.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.762+0000 7f6a7d800640 1 -- start start 2024-04-16T16:17:52.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.763+0000 7f6a7d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a780ba500 0x7f6a781581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:52.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.763+0000 7f6a7d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a780cf4d0 0x7f6a78159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:52.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.763+0000 7f6a7d800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7815e980 0x7f6a780732c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:52.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.764+0000 7f6a7d800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6a780d2d80 con 0x7f6a780ba500 2024-04-16T16:17:52.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.764+0000 7f6a7d800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6a780d3cf0 con 0x7f6a7815e980 2024-04-16T16:17:52.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.764+0000 7f6a7d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6a780d2f00 con 0x7f6a780cf4d0 2024-04-16T16:17:52.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.764+0000 7f6a77e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a780ba500 0x7f6a781581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:52.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.764+0000 7f6a77e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a780ba500 0x7f6a781581b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48920/0 (socket says 172.21.15.45:48920) 2024-04-16T16:17:52.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.764+0000 7f6a77e00640 1 -- 172.21.15.45:0/3776792856 learned_addr learned my addr 172.21.15.45:0/3776792856 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:52.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.764+0000 7f6a77400640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a780cf4d0 0x7f6a78159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:52.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.764+0000 7f6a7ce00640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7815e980 0x7f6a780732c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:52.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.764+0000 7f6a77e00640 1 -- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a780cf4d0 msgr2=0x7f6a78159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:52.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.764+0000 7f6a77e00640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a780cf4d0 0x7f6a78159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.764+0000 7f6a77e00640 1 -- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7815e980 msgr2=0x7f6a780732c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:52.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.764+0000 7f6a77e00640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7815e980 0x7f6a780732c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.764+0000 7f6a77e00640 1 -- 172.21.15.45:0/3776792856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a781586f0 con 0x7f6a780ba500 2024-04-16T16:17:52.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.764+0000 7f6a7ce00640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7815e980 0x7f6a780732c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:17:52.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.764+0000 7f6a77400640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a780cf4d0 0x7f6a78159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:17:52.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.765+0000 7f6a77e00640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a780ba500 0x7f6a781581b0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f6a6802bf50 tx=0x7f6a680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:52.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.765+0000 7f6a6fe00640 1 -- 172.21.15.45:0/3776792856 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68034040 con 0x7f6a780ba500 2024-04-16T16:17:52.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.765+0000 7f6a6fe00640 1 -- 172.21.15.45:0/3776792856 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a68030e30 con 0x7f6a780ba500 2024-04-16T16:17:52.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.765+0000 7f6a6fe00640 1 -- 172.21.15.45:0/3776792856 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a680443f0 con 0x7f6a780ba500 2024-04-16T16:17:52.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.765+0000 7f6a7d800640 1 -- 172.21.15.45:0/3776792856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a78158ac0 con 0x7f6a780ba500 2024-04-16T16:17:52.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.765+0000 7f6a7d800640 1 -- 172.21.15.45:0/3776792856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a78159080 con 0x7f6a780ba500 2024-04-16T16:17:52.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.767+0000 7f6a7d800640 1 -- 172.21.15.45:0/3776792856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a780b4930 con 0x7f6a780ba500 2024-04-16T16:17:52.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.768+0000 7f6a6fe00640 1 -- 172.21.15.45:0/3776792856 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 21) ==== 91142+0+0 (secure 0 0 0) 0x7f6a68037070 con 0x7f6a780ba500 2024-04-16T16:17:52.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.768+0000 7f6a6fe00640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f6a5406ca10 0x7f6a5406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:52.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.769+0000 7f6a6fe00640 1 -- 172.21.15.45:0/3776792856 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f6a680ade10 con 0x7f6a780ba500 2024-04-16T16:17:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.769+0000 7f6a77400640 1 -- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f6a5406ca10 msgr2=0x7f6a5406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/541757982 2024-04-16T16:17:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.769+0000 7f6a77400640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f6a5406ca10 0x7f6a5406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:17:52.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.773+0000 7f6a6fe00640 1 -- 172.21.15.45:0/3776792856 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6a6803f070 con 0x7f6a780ba500 2024-04-16T16:17:52.794 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:52.794+0000 7f51fbe3e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:17:52.952 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:52.953+0000 7f24c2a0d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:17:52.954 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:52.955+0000 7f51fbe3e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:17:52.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.970+0000 7f6a77400640 1 -- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f6a5406ca10 msgr2=0x7f6a5406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/541757982 2024-04-16T16:17:52.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.970+0000 7f6a77400640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f6a5406ca10 0x7f6a5406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:17:52.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.974+0000 7f6a7d800640 1 -- 172.21.15.45:0/3776792856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6a7815def0 con 0x7f6a780ba500 2024-04-16T16:17:52.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.975+0000 7f6a6fe00640 1 -- 172.21.15.45:0/3776792856 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v73) ==== 100+0+11 (secure 0 0 0) 0x7f6a6807c020 con 0x7f6a780ba500 2024-04-16T16:17:52.975 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:17:52.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.977+0000 7f6a7d800640 1 -- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f6a5406ca10 msgr2=0x7f6a5406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:17:52.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.977+0000 7f6a7d800640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f6a5406ca10 0x7f6a5406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.977+0000 7f6a7d800640 1 -- 172.21.15.45:0/3776792856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a780ba500 msgr2=0x7f6a781581b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:52.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.977+0000 7f6a7d800640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a780ba500 0x7f6a781581b0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f6a6802bf50 tx=0x7f6a680046c0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.977+0000 7f6a7d800640 1 -- 172.21.15.45:0/3776792856 shutdown_connections 2024-04-16T16:17:52.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.977+0000 7f6a7d800640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f6a5406ca10 0x7f6a5406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.977+0000 7f6a7d800640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7815e980 0x7f6a780732c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.977+0000 7f6a7d800640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a780cf4d0 0x7f6a78159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.977+0000 7f6a7d800640 1 --2- 172.21.15.45:0/3776792856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a780ba500 0x7f6a781581b0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:52.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.978+0000 7f6a7d800640 1 -- 172.21.15.45:0/3776792856 >> 172.21.15.45:0/3776792856 conn(0x7f6a7806d2f0 msgr2=0x7f6a780b5f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:52.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.978+0000 7f6a7d800640 1 -- 172.21.15.45:0/3776792856 shutdown_connections 2024-04-16T16:17:52.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:52.978+0000 7f6a7d800640 1 -- 172.21.15.45:0/3776792856 wait complete. 2024-04-16T16:17:52.995 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:17:53.064 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:53.064+0000 7f51fbe3e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:17:53.114 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:53.115+0000 7f51fbe3e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:17:53.125 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:53.125+0000 7f24c2a0d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:17:53.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.194+0000 7f7719a00640 1 Processor -- start 2024-04-16T16:17:53.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.194+0000 7f7719a00640 1 -- start start 2024-04-16T16:17:53.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.195+0000 7f7719a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7714162090 0x7f77140cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:53.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.195+0000 7f7719a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f77140ba140 0x7f77140cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:53.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.195+0000 7f7719a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77140d2600 con 0x7f77140d5520 2024-04-16T16:17:53.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.195+0000 7f7719a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f77140ba8b0 con 0x7f77140ba140 2024-04-16T16:17:53.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.195+0000 7f7719a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f77140ba5d0 con 0x7f7714162090 2024-04-16T16:17:53.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.196+0000 7f770be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f77140ba140 0x7f77140cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:53.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.196+0000 7f770be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f77140ba140 0x7f77140cfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48922/0 (socket says 172.21.15.45:48922) 2024-04-16T16:17:53.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.196+0000 7f770be00640 1 -- 172.21.15.45:0/1324672570 learned_addr learned my addr 172.21.15.45:0/1324672570 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:53.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.196+0000 7f7713e00640 1 --2- 172.21.15.45:0/1324672570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7714162090 0x7f77140cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:53.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.196+0000 7f770be00640 1 -- 172.21.15.45:0/1324672570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7714162090 msgr2=0x7f77140cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:53.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.196+0000 7f770be00640 1 --2- 172.21.15.45:0/1324672570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7714162090 0x7f77140cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:53.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.196+0000 7f770be00640 1 -- 172.21.15.45:0/1324672570 >> v1:172.21.15.103:6789/0 conn(0x7f77140d5520 legacy=0x7f77141628a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:53.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.196+0000 7f770be00640 1 -- 172.21.15.45:0/1324672570 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77140d28a0 con 0x7f77140ba140 2024-04-16T16:17:53.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.198+0000 7f770be00640 1 --2- 172.21.15.45:0/1324672570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77140ba140 0x7f77140cfa50 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009870 tx=0x7f76fc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f7e89621b16746 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:53.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.198+0000 7f7713400640 1 -- 172.21.15.45:0/1324672570 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc034070 con 0x7f77140ba140 2024-04-16T16:17:53.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.198+0000 7f7713400640 1 -- 172.21.15.45:0/1324672570 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76fc030840 con 0x7f77140ba140 2024-04-16T16:17:53.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.198+0000 7f7713400640 1 -- 172.21.15.45:0/1324672570 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc030c10 con 0x7f77140ba140 2024-04-16T16:17:53.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.199+0000 7f7719a00640 1 -- 172.21.15.45:0/1324672570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77140ba140 msgr2=0x7f77140cfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:53.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.199+0000 7f7719a00640 1 --2- 172.21.15.45:0/1324672570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77140ba140 0x7f77140cfa50 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009870 tx=0x7f76fc02b980 comp rx=0 tx=0).stop 2024-04-16T16:17:53.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.199+0000 7f7719a00640 1 -- 172.21.15.45:0/1324672570 shutdown_connections 2024-04-16T16:17:53.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.199+0000 7f7719a00640 1 --2- 172.21.15.45:0/1324672570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77140ba140 0x7f77140cfa50 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:53.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.199+0000 7f7719a00640 1 --2- 172.21.15.45:0/1324672570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7714162090 0x7f77140cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:53.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.199+0000 7f7719a00640 1 -- 172.21.15.45:0/1324672570 >> 172.21.15.45:0/1324672570 conn(0x7f77140d1940 msgr2=0x7f77140d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:53.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.199+0000 7f7719a00640 1 -- 172.21.15.45:0/1324672570 shutdown_connections 2024-04-16T16:17:53.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.200+0000 7f7719a00640 1 -- 172.21.15.45:0/1324672570 wait complete. 2024-04-16T16:17:53.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.201+0000 7f7719a00640 1 Processor -- start 2024-04-16T16:17:53.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.201+0000 7f7719a00640 1 -- start start 2024-04-16T16:17:53.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.202+0000 7f7719a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77140d5520 0x7f77140cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:53.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.202+0000 7f7719a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7714162090 0x7f77140b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:53.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.202+0000 7f7719a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7714153a10 0x7f77140ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:53.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.202+0000 7f7719a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f77141625f0 con 0x7f7714153a10 2024-04-16T16:17:53.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.202+0000 7f7719a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7714162470 con 0x7f77140d5520 2024-04-16T16:17:53.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.202+0000 7f7719a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f77140d40b0 con 0x7f7714162090 2024-04-16T16:17:53.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.203+0000 7f7719000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7714153a10 0x7f77140ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:53.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.203+0000 7f7719000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7714153a10 0x7f77140ce290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48928/0 (socket says 172.21.15.45:48928) 2024-04-16T16:17:53.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.203+0000 7f7719000640 1 -- 172.21.15.45:0/600750171 learned_addr learned my addr 172.21.15.45:0/600750171 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:53.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.203+0000 7f770be00640 1 --2- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7714162090 0x7f77140b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:53.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.203+0000 7f7713e00640 1 --2- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77140d5520 0x7f77140cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:53.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.203+0000 7f7719000640 1 -- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7714162090 msgr2=0x7f77140b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:53.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.203+0000 7f7719000640 1 --2- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7714162090 0x7f77140b4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:53.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.203+0000 7f7719000640 1 -- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77140d5520 msgr2=0x7f77140cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:53.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.203+0000 7f7719000640 1 --2- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77140d5520 0x7f77140cc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:53.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.203+0000 7f7719000640 1 -- 172.21.15.45:0/600750171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77140c9190 con 0x7f7714153a10 2024-04-16T16:17:53.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.204+0000 7f7719000640 1 --2- 172.21.15.45:0/600750171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7714153a10 0x7f77140ce290 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f770c00c730 tx=0x7f770c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:53.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.204+0000 7f7710c00640 1 -- 172.21.15.45:0/600750171 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c003b70 con 0x7f7714153a10 2024-04-16T16:17:53.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.204+0000 7f7710c00640 1 -- 172.21.15.45:0/600750171 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f770c003d10 con 0x7f7714153a10 2024-04-16T16:17:53.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.204+0000 7f7710c00640 1 -- 172.21.15.45:0/600750171 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c00aa90 con 0x7f7714153a10 2024-04-16T16:17:53.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.204+0000 7f7719a00640 1 -- 172.21.15.45:0/600750171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77140c9470 con 0x7f7714153a10 2024-04-16T16:17:53.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.204+0000 7f7719a00640 1 -- 172.21.15.45:0/600750171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77140c9a30 con 0x7f7714153a10 2024-04-16T16:17:53.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.205+0000 7f7719a00640 1 -- 172.21.15.45:0/600750171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77140d0ac0 con 0x7f7714153a10 2024-04-16T16:17:53.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.206+0000 7f7710c00640 1 -- 172.21.15.45:0/600750171 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 21) ==== 91142+0+0 (secure 0 0 0) 0x7f770c00ac30 con 0x7f7714153a10 2024-04-16T16:17:53.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.207+0000 7f7710c00640 1 --2- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f76f006ca10 0x7f76f006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:53.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.207+0000 7f7710c00640 1 -- 172.21.15.45:0/600750171 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f770c08bd80 con 0x7f7714153a10 2024-04-16T16:17:53.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.207+0000 7f7713e00640 1 -- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f76f006ca10 msgr2=0x7f76f006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/541757982 2024-04-16T16:17:53.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.207+0000 7f7713e00640 1 --2- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f76f006ca10 0x7f76f006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:17:53.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.212+0000 7f7710c00640 1 -- 172.21.15.45:0/600750171 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f77140d0ac0 con 0x7f7714153a10 2024-04-16T16:17:53.241 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:53.242+0000 7f51fbe3e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:17:53.245 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:53.245+0000 7f24c2a0d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:17:53.298 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:53.299+0000 7f24c2a0d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:17:53.348 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:53.348+0000 7f51fbe3e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:17:53.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.408+0000 7f7713e00640 1 -- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f76f006ca10 msgr2=0x7f76f006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/541757982 2024-04-16T16:17:53.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.408+0000 7f7713e00640 1 --2- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f76f006ca10 0x7f76f006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:17:53.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.430+0000 7f7719a00640 1 -- 172.21.15.45:0/600750171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7714075780 con 0x7f7714153a10 2024-04-16T16:17:53.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.431+0000 7f7710c00640 1 -- 172.21.15.45:0/600750171 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v73) ==== 81+0+592 (secure 0 0 0) 0x7f770c059ee0 con 0x7f7714153a10 2024-04-16T16:17:53.431 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/4155603915 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:53.431 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/951047877 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:53.431 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3176524506 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:53.431 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/4155603915 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:53.431 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/365521086 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:53.431 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3345679536 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:53.431 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3606536347 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:53.431 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1223877773 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:53.432 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/252084277 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:53.432 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/650078727 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:53.432 INFO:teuthology.orchestra.run.smithi045.stderr:listed 10 entries 2024-04-16T16:17:53.432 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:53.433+0000 7f24c2a0d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:17:53.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.433+0000 7f7719a00640 1 -- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f76f006ca10 msgr2=0x7f76f006eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:17:53.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.433+0000 7f7719a00640 1 --2- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f76f006ca10 0x7f76f006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:53.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.433+0000 7f7719a00640 1 -- 172.21.15.45:0/600750171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7714153a10 msgr2=0x7f77140ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:53.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.433+0000 7f7719a00640 1 --2- 172.21.15.45:0/600750171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7714153a10 0x7f77140ce290 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f770c00c730 tx=0x7f770c00cbf0 comp rx=0 tx=0).stop 2024-04-16T16:17:53.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.434+0000 7f7719a00640 1 -- 172.21.15.45:0/600750171 shutdown_connections 2024-04-16T16:17:53.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.434+0000 7f7719a00640 1 --2- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f76f006ca10 0x7f76f006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:53.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.434+0000 7f7719a00640 1 --2- 172.21.15.45:0/600750171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7714153a10 0x7f77140ce290 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:53.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.434+0000 7f7719a00640 1 --2- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7714162090 0x7f77140b4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:53.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.434+0000 7f7719a00640 1 --2- 172.21.15.45:0/600750171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77140d5520 0x7f77140cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:53.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.434+0000 7f7719a00640 1 -- 172.21.15.45:0/600750171 >> 172.21.15.45:0/600750171 conn(0x7f77140d1940 msgr2=0x7f77140cf750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:53.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.434+0000 7f7719a00640 1 -- 172.21.15.45:0/600750171 shutdown_connections 2024-04-16T16:17:53.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.434+0000 7f7719a00640 1 -- 172.21.15.45:0/600750171 wait complete. 2024-04-16T16:17:53.452 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-16T16:17:53.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:53.452 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-16T16:17:53.521 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:53.522+0000 7f51fbe3e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:17:53.557 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:53.557+0000 7f24c2a0d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:17:53.575 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:17:53.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:53.576 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-16T16:17:53.596 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:53.595+0000 7f51fbe3e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:17:53.630 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:17:53.630 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:17:53.630 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:53.671 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:17:53.671 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:53.752 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:53.752+0000 7f51fbe3e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:17:53.753 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:17:53.772 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:53.772+0000 7f24c2a0d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:17:53.829 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:53.829+0000 7f24c2a0d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:17:53.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.960+0000 7f2ebbe00640 1 Processor -- start 2024-04-16T16:17:53.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.960+0000 7f2ebbe00640 1 -- start start 2024-04-16T16:17:53.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.962+0000 7f2ebbe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2ebc160da0 0x7f2ebc15c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:53.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.962+0000 7f2ebbe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ebc161200 con 0x7f2ebc06ef00 2024-04-16T16:17:53.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.962+0000 7f2ebbe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ebc070ec0 con 0x7f2ebc15c600 2024-04-16T16:17:53.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.962+0000 7f2ebbe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2ebc15cf20 con 0x7f2ebc160da0 2024-04-16T16:17:53.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.962+0000 7f2eba000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2ebc160da0 0x7f2ebc15c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:53.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.962+0000 7f2ebb400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2ebc06ef00 0x7f2ebc06c260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52236/0 (socket says 172.21.15.45:52236) 2024-04-16T16:17:53.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.962+0000 7f2ebb400640 1 -- 172.21.15.45:0/3649309691 learned_addr learned my addr 172.21.15.45:0/3649309691 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:53.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.963+0000 7f2eba000640 1 -- 172.21.15.45:0/3649309691 >> v1:172.21.15.103:6789/0 conn(0x7f2ebc15c600 legacy=0x7f2ebc06e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:53.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.963+0000 7f2eba000640 1 -- 172.21.15.45:0/3649309691 >> v1:172.21.15.45:6789/0 conn(0x7f2ebc06ef00 legacy=0x7f2ebc06c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:53.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.963+0000 7f2eba000640 1 -- 172.21.15.45:0/3649309691 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ebc06ba60 con 0x7f2ebc160da0 2024-04-16T16:17:53.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.963+0000 7f2eb9600640 1 -- 172.21.15.45:0/3649309691 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1595008430 0 0) 0x7f2ebc161200 con 0x7f2ebc06ef00 2024-04-16T16:17:53.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.964+0000 7f2eba000640 1 --2- 172.21.15.45:0/3649309691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc160da0 0x7f2ebc15c9e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f2ea80097a0 tx=0x7f2ea802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38510443bc9ac4b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:53.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.965+0000 7f2eb9600640 1 -- 172.21.15.45:0/3649309691 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8035020 con 0x7f2ebc160da0 2024-04-16T16:17:53.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.965+0000 7f2eb9600640 1 -- 172.21.15.45:0/3649309691 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea8031830 con 0x7f2ebc160da0 2024-04-16T16:17:53.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.965+0000 7f2eb9600640 1 -- 172.21.15.45:0/3649309691 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8031c00 con 0x7f2ebc160da0 2024-04-16T16:17:53.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.966+0000 7f2ebbe00640 1 -- 172.21.15.45:0/3649309691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc160da0 msgr2=0x7f2ebc15c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:53.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.966+0000 7f2ebbe00640 1 --2- 172.21.15.45:0/3649309691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc160da0 0x7f2ebc15c9e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f2ea80097a0 tx=0x7f2ea802b8d0 comp rx=0 tx=0).stop 2024-04-16T16:17:53.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.966+0000 7f2ebbe00640 1 -- 172.21.15.45:0/3649309691 shutdown_connections 2024-04-16T16:17:53.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.966+0000 7f2ebbe00640 1 --2- 172.21.15.45:0/3649309691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc160da0 0x7f2ebc15c9e0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:53.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.966+0000 7f2ebbe00640 1 -- 172.21.15.45:0/3649309691 >> 172.21.15.45:0/3649309691 conn(0x7f2ebc0d0f10 msgr2=0x7f2ebc0d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:53.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.966+0000 7f2ebbe00640 1 -- 172.21.15.45:0/3649309691 shutdown_connections 2024-04-16T16:17:53.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.966+0000 7f2ebbe00640 1 -- 172.21.15.45:0/3649309691 wait complete. 2024-04-16T16:17:53.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.969+0000 7f2ebbe00640 1 Processor -- start 2024-04-16T16:17:53.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.969+0000 7f2ebbe00640 1 -- start start 2024-04-16T16:17:53.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.970+0000 7f2ebbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc06ef00 0x7f2ebc151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:53.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.970+0000 7f2ebbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ebc15c600 0x7f2ebc158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:53.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.970+0000 7f2ebbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ebc160da0 0x7f2ebc15f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:53.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.970+0000 7f2ebbe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2ebc0b0600 con 0x7f2ebc06ef00 2024-04-16T16:17:53.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.970+0000 7f2ebbe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2ebc0711a0 con 0x7f2ebc15c600 2024-04-16T16:17:53.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.970+0000 7f2ebbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2ebc070ae0 con 0x7f2ebc160da0 2024-04-16T16:17:53.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.971+0000 7f2ebaa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc06ef00 0x7f2ebc151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:53.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.971+0000 7f2ebaa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc06ef00 0x7f2ebc151cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57576/0 (socket says 172.21.15.45:57576) 2024-04-16T16:17:53.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.971+0000 7f2ebaa00640 1 -- 172.21.15.45:0/3016050487 learned_addr learned my addr 172.21.15.45:0/3016050487 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:53.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.971+0000 7f2ebb400640 1 --2- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ebc160da0 0x7f2ebc15f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:53.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.971+0000 7f2eba000640 1 --2- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ebc15c600 0x7f2ebc158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:53.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.971+0000 7f2ebaa00640 1 -- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ebc160da0 msgr2=0x7f2ebc15f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:53.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.971+0000 7f2ebaa00640 1 --2- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ebc160da0 0x7f2ebc15f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:53.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.971+0000 7f2ebaa00640 1 -- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ebc15c600 msgr2=0x7f2ebc158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:53.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.971+0000 7f2ebaa00640 1 --2- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ebc15c600 0x7f2ebc158ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:53.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.971+0000 7f2ebaa00640 1 -- 172.21.15.45:0/3016050487 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ebc158110 con 0x7f2ebc06ef00 2024-04-16T16:17:53.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.972+0000 7f2ebaa00640 1 --2- 172.21.15.45:0/3016050487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc06ef00 0x7f2ebc151cf0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f2eac00c870 tx=0x7f2eac00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:53.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.972+0000 7f2eb2a00640 1 -- 172.21.15.45:0/3016050487 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac003b70 con 0x7f2ebc06ef00 2024-04-16T16:17:53.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.972+0000 7f2eb2a00640 1 -- 172.21.15.45:0/3016050487 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2eac003d30 con 0x7f2ebc06ef00 2024-04-16T16:17:53.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.972+0000 7f2eb2a00640 1 -- 172.21.15.45:0/3016050487 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac00aa90 con 0x7f2ebc06ef00 2024-04-16T16:17:53.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.972+0000 7f2ebbe00640 1 -- 172.21.15.45:0/3016050487 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ebc1583f0 con 0x7f2ebc06ef00 2024-04-16T16:17:53.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.972+0000 7f2ebbe00640 1 -- 172.21.15.45:0/3016050487 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ebc1589b0 con 0x7f2ebc06ef00 2024-04-16T16:17:53.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.973+0000 7f2ebbe00640 1 -- 172.21.15.45:0/3016050487 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ebc075780 con 0x7f2ebc06ef00 2024-04-16T16:17:53.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.974+0000 7f2eb2a00640 1 -- 172.21.15.45:0/3016050487 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 21) ==== 91142+0+0 (secure 0 0 0) 0x7f2eac00ac30 con 0x7f2ebc06ef00 2024-04-16T16:17:53.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.976+0000 7f2eb2a00640 1 --2- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f2e9c06ca40 0x7f2e9c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:53.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.976+0000 7f2eb2a00640 1 -- 172.21.15.45:0/3016050487 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f2eac08be10 con 0x7f2ebc06ef00 2024-04-16T16:17:53.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.976+0000 7f2eba000640 1 -- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f2e9c06ca40 msgr2=0x7f2e9c06ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/541757982 2024-04-16T16:17:53.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.976+0000 7f2eba000640 1 --2- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f2e9c06ca40 0x7f2e9c06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:17:53.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:53.979+0000 7f2eb2a00640 1 -- 172.21.15.45:0/3016050487 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2eac08f030 con 0x7f2ebc06ef00 2024-04-16T16:17:53.983 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:53.984+0000 7f24c2a0d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:17:54.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.177+0000 7f2eba000640 1 -- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f2e9c06ca40 msgr2=0x7f2e9c06ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/541757982 2024-04-16T16:17:54.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.177+0000 7f2eba000640 1 --2- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f2e9c06ca40 0x7f2e9c06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:17:54.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.181+0000 7f2ebbe00640 1 -- 172.21.15.45:0/3016050487 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ebc158bf0 con 0x7f2ebc06ef00 2024-04-16T16:17:54.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.185+0000 7f2eb2a00640 1 -- 172.21.15.45:0/3016050487 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+17299 (secure 0 0 0) 0x7f2eac059f70 con 0x7f2ebc06ef00 2024-04-16T16:17:54.185 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:54.185 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":73,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:17:01.408977+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:16:37.449973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-16T16:16:38.485027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6833/4155603915":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/951047877":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/3176524506":"2024-04-17T16:17:01.408806+0000","172.21.15.103:6832/4155603915":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/365521086":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/3345679536":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/3606536347":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/1223877773":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/252084277":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/650078727":"2024-04-17T16:17:01.408806+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:17:54.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.187+0000 7f2ebbe00640 1 -- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f2e9c06ca40 msgr2=0x7f2e9c06ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:17:54.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.187+0000 7f2ebbe00640 1 --2- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f2e9c06ca40 0x7f2e9c06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.187+0000 7f2ebbe00640 1 -- 172.21.15.45:0/3016050487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc06ef00 msgr2=0x7f2ebc151cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:54.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.187+0000 7f2ebbe00640 1 --2- 172.21.15.45:0/3016050487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc06ef00 0x7f2ebc151cf0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f2eac00c870 tx=0x7f2eac00cd30 comp rx=0 tx=0).stop 2024-04-16T16:17:54.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.187+0000 7f2ebbe00640 1 -- 172.21.15.45:0/3016050487 shutdown_connections 2024-04-16T16:17:54.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.187+0000 7f2ebbe00640 1 --2- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f2e9c06ca40 0x7f2e9c06ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.187+0000 7f2ebbe00640 1 --2- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ebc160da0 0x7f2ebc15f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.187+0000 7f2ebbe00640 1 --2- 172.21.15.45:0/3016050487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ebc15c600 0x7f2ebc158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.187+0000 7f2ebbe00640 1 --2- 172.21.15.45:0/3016050487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc06ef00 0x7f2ebc151cf0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.187+0000 7f2ebbe00640 1 -- 172.21.15.45:0/3016050487 >> 172.21.15.45:0/3016050487 conn(0x7f2ebc0d0f10 msgr2=0x7f2ebc0d12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:54.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.188+0000 7f2ebbe00640 1 -- 172.21.15.45:0/3016050487 shutdown_connections 2024-04-16T16:17:54.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.188+0000 7f2ebbe00640 1 -- 172.21.15.45:0/3016050487 wait complete. 2024-04-16T16:17:54.207 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:16:37.449973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:16:38.485027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:17:54.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:17:54.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:17:54.298 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:17:54.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.442+0000 7ff767200640 1 Processor -- start 2024-04-16T16:17:54.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.442+0000 7ff767200640 1 -- start start 2024-04-16T16:17:54.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.443+0000 7ff767200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff7600d4370 0x7ff7600cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:54.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.443+0000 7ff767200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7600cfa70 0x7ff760162560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:54.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.443+0000 7ff767200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff76015ee50 con 0x7ff76015ea70 2024-04-16T16:17:54.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.443+0000 7ff767200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff7600d1a10 con 0x7ff7600d4370 2024-04-16T16:17:54.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.443+0000 7ff767200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff7600b6480 con 0x7ff7600cfa70 2024-04-16T16:17:54.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.444+0000 7ff765400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff7600d4370 0x7ff7600cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:54.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.444+0000 7ff766800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7600cfa70 0x7ff760162560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:54.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.444+0000 7ff765400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff7600d4370 0x7ff7600cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:54824/0 (socket says 172.21.15.45:54824) 2024-04-16T16:17:54.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.444+0000 7ff765400640 1 -- 172.21.15.45:0/4122129756 learned_addr learned my addr 172.21.15.45:0/4122129756 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:54.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.444+0000 7ff765400640 1 -- 172.21.15.45:0/4122129756 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7600cfa70 msgr2=0x7ff760162560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:54.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.444+0000 7ff765400640 1 --2- 172.21.15.45:0/4122129756 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7600cfa70 0x7ff760162560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.444+0000 7ff765400640 1 -- 172.21.15.45:0/4122129756 >> v1:172.21.15.103:6789/0 conn(0x7ff76015ea70 legacy=0x7ff7600d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:54.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.444+0000 7ff765400640 1 -- 172.21.15.45:0/4122129756 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7600b6760 con 0x7ff7600d4370 2024-04-16T16:17:54.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.446+0000 7ff765400640 1 --2- 172.21.15.45:0/4122129756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7600d4370 0x7ff7600cf530 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff754009870 tx=0x7ff75402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e64c546a21466301 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:54.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.446+0000 7ff764a00640 1 -- 172.21.15.45:0/4122129756 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff754034070 con 0x7ff7600d4370 2024-04-16T16:17:54.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.446+0000 7ff764a00640 1 -- 172.21.15.45:0/4122129756 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff754030840 con 0x7ff7600d4370 2024-04-16T16:17:54.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.446+0000 7ff764a00640 1 -- 172.21.15.45:0/4122129756 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff754030c10 con 0x7ff7600d4370 2024-04-16T16:17:54.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.447+0000 7ff767200640 1 -- 172.21.15.45:0/4122129756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7600d4370 msgr2=0x7ff7600cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:54.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.447+0000 7ff767200640 1 --2- 172.21.15.45:0/4122129756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7600d4370 0x7ff7600cf530 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff754009870 tx=0x7ff75402b980 comp rx=0 tx=0).stop 2024-04-16T16:17:54.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.447+0000 7ff767200640 1 -- 172.21.15.45:0/4122129756 shutdown_connections 2024-04-16T16:17:54.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.447+0000 7ff767200640 1 --2- 172.21.15.45:0/4122129756 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7600cfa70 0x7ff760162560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.447+0000 7ff767200640 1 --2- 172.21.15.45:0/4122129756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7600d4370 0x7ff7600cf530 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.447+0000 7ff767200640 1 -- 172.21.15.45:0/4122129756 >> 172.21.15.45:0/4122129756 conn(0x7ff76006d3b0 msgr2=0x7ff760073060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:54.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.447+0000 7ff767200640 1 -- 172.21.15.45:0/4122129756 shutdown_connections 2024-04-16T16:17:54.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.447+0000 7ff767200640 1 -- 172.21.15.45:0/4122129756 wait complete. 2024-04-16T16:17:54.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.449+0000 7ff767200640 1 Processor -- start 2024-04-16T16:17:54.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.449+0000 7ff767200640 1 -- start start 2024-04-16T16:17:54.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.450+0000 7ff767200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7600cfa70 0x7ff7601e8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:54.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.451+0000 7ff767200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600d4370 0x7ff7601616e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:54.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.451+0000 7ff767200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff76015ea70 0x7ff760161c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:54.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.451+0000 7ff767200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff7600b6f60 con 0x7ff7600d4370 2024-04-16T16:17:54.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.451+0000 7ff767200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff760162aa0 con 0x7ff76015ea70 2024-04-16T16:17:54.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.451+0000 7ff767200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff7601e9260 con 0x7ff7600cfa70 2024-04-16T16:17:54.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.451+0000 7ff765400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600d4370 0x7ff7601616e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:54.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.451+0000 7ff765400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600d4370 0x7ff7601616e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57580/0 (socket says 172.21.15.45:57580) 2024-04-16T16:17:54.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.451+0000 7ff765400640 1 -- 172.21.15.45:0/215725820 learned_addr learned my addr 172.21.15.45:0/215725820 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:54.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.451+0000 7ff765e00640 1 --2- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7600cfa70 0x7ff7601e8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:54.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.451+0000 7ff766800640 1 --2- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff76015ea70 0x7ff760161c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:54.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.451+0000 7ff765400640 1 -- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7600cfa70 msgr2=0x7ff7601e8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:54.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.451+0000 7ff765400640 1 --2- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7600cfa70 0x7ff7601e8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.451+0000 7ff765400640 1 -- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff76015ea70 msgr2=0x7ff760161c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:54.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.451+0000 7ff765400640 1 --2- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff76015ea70 0x7ff760161c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.451+0000 7ff765400640 1 -- 172.21.15.45:0/215725820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7600c89c0 con 0x7ff7600d4370 2024-04-16T16:17:54.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.452+0000 7ff765400640 1 --2- 172.21.15.45:0/215725820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600d4370 0x7ff7601616e0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7ff754000c00 tx=0x7ff754004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:54.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.452+0000 7ff75a000640 1 -- 172.21.15.45:0/215725820 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff754048070 con 0x7ff7600d4370 2024-04-16T16:17:54.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.452+0000 7ff75a000640 1 -- 172.21.15.45:0/215725820 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff754030640 con 0x7ff7600d4370 2024-04-16T16:17:54.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.452+0000 7ff75a000640 1 -- 172.21.15.45:0/215725820 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff754034040 con 0x7ff7600d4370 2024-04-16T16:17:54.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.452+0000 7ff767200640 1 -- 172.21.15.45:0/215725820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7600c8ca0 con 0x7ff7600d4370 2024-04-16T16:17:54.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.452+0000 7ff767200640 1 -- 172.21.15.45:0/215725820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7600c9260 con 0x7ff7600d4370 2024-04-16T16:17:54.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.454+0000 7ff767200640 1 -- 172.21.15.45:0/215725820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff760075780 con 0x7ff7600d4370 2024-04-16T16:17:54.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.454+0000 7ff75a000640 1 -- 172.21.15.45:0/215725820 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 21) ==== 91142+0+0 (secure 0 0 0) 0x7ff754037070 con 0x7ff7600d4370 2024-04-16T16:17:54.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.455+0000 7ff75a000640 1 --2- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff74006ca10 0x7ff74006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:54.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.455+0000 7ff75a000640 1 -- 172.21.15.45:0/215725820 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7ff7540ade50 con 0x7ff7600d4370 2024-04-16T16:17:54.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.456+0000 7ff765e00640 1 -- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff74006ca10 msgr2=0x7ff74006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/541757982 2024-04-16T16:17:54.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.456+0000 7ff765e00640 1 --2- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff74006ca10 0x7ff74006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:17:54.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.460+0000 7ff75a000640 1 -- 172.21.15.45:0/215725820 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff75407bf30 con 0x7ff7600d4370 2024-04-16T16:17:54.572 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:54.572+0000 7f51fbe3e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:17:54.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.657+0000 7ff765e00640 1 -- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff74006ca10 msgr2=0x7ff74006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/541757982 2024-04-16T16:17:54.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.657+0000 7ff765e00640 1 --2- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff74006ca10 0x7ff74006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:17:54.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.682+0000 7ff767200640 1 -- 172.21.15.45:0/215725820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff7600b6190 con 0x7ff7600d4370 2024-04-16T16:17:54.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.683+0000 7ff75a000640 1 -- 172.21.15.45:0/215725820 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7ff75407b830 con 0x7ff7600d4370 2024-04-16T16:17:54.683 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:17:54.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.686+0000 7ff767200640 1 -- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff74006ca10 msgr2=0x7ff74006eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:17:54.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.686+0000 7ff767200640 1 --2- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff74006ca10 0x7ff74006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.686+0000 7ff767200640 1 -- 172.21.15.45:0/215725820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600d4370 msgr2=0x7ff7601616e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:54.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.686+0000 7ff767200640 1 --2- 172.21.15.45:0/215725820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600d4370 0x7ff7601616e0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7ff754000c00 tx=0x7ff754004730 comp rx=0 tx=0).stop 2024-04-16T16:17:54.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.686+0000 7ff767200640 1 -- 172.21.15.45:0/215725820 shutdown_connections 2024-04-16T16:17:54.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.686+0000 7ff767200640 1 --2- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7ff74006ca10 0x7ff74006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.686+0000 7ff767200640 1 --2- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff76015ea70 0x7ff760161c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.686+0000 7ff767200640 1 --2- 172.21.15.45:0/215725820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600d4370 0x7ff7601616e0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.686+0000 7ff767200640 1 --2- 172.21.15.45:0/215725820 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7600cfa70 0x7ff7601e8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.686+0000 7ff767200640 1 -- 172.21.15.45:0/215725820 >> 172.21.15.45:0/215725820 conn(0x7ff76006d3b0 msgr2=0x7ff76015dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:54.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.686+0000 7ff767200640 1 -- 172.21.15.45:0/215725820 shutdown_connections 2024-04-16T16:17:54.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.686+0000 7ff767200640 1 -- 172.21.15.45:0/215725820 wait complete. 2024-04-16T16:17:54.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:17:54.879 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:54.879+0000 7f51fbe3e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:17:54.888 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:54.888+0000 7f24c2a0d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:17:54.937 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:54.937+0000 7f51fbe3e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:17:54.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.968+0000 7f7296000640 1 Processor -- start 2024-04-16T16:17:54.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.968+0000 7f7296000640 1 -- start start 2024-04-16T16:17:54.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.970+0000 7f7296000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72900b6310 0x7f7290161db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:54.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.970+0000 7f7296000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f72900d3e90 0x7f72901622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:54.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.970+0000 7f7296000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72900b6810 con 0x7f72900d18f0 2024-04-16T16:17:54.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.970+0000 7f7296000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f72900d29f0 con 0x7f72900d3e90 2024-04-16T16:17:54.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.970+0000 7f7296000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f72900b6af0 con 0x7f72900b6310 2024-04-16T16:17:54.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.970+0000 7f7295600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f72900d3e90 0x7f72901622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:54.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.970+0000 7f7294c00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f72900d18f0 0x7f72900ba220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52238/0 (socket says 172.21.15.45:52238) 2024-04-16T16:17:54.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.970+0000 7f7294c00640 1 -- 172.21.15.45:0/4202033385 learned_addr learned my addr 172.21.15.45:0/4202033385 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:54.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.970+0000 7f728fe00640 1 --2- 172.21.15.45:0/4202033385 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72900b6310 0x7f7290161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:54.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.971+0000 7f7295600640 1 -- 172.21.15.45:0/4202033385 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72900b6310 msgr2=0x7f7290161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:54.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.971+0000 7f7295600640 1 --2- 172.21.15.45:0/4202033385 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72900b6310 0x7f7290161db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.971+0000 7f7295600640 1 -- 172.21.15.45:0/4202033385 >> v1:172.21.15.45:6789/0 conn(0x7f72900d18f0 legacy=0x7f72900ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:54.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.971+0000 7f7295600640 1 -- 172.21.15.45:0/4202033385 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72900b6dd0 con 0x7f72900d3e90 2024-04-16T16:17:54.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.972+0000 7f7295600640 1 --2- 172.21.15.45:0/4202033385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72900d3e90 0x7f72901622f0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f7288009d90 tx=0x7f728802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b10758f46ea796f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:54.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.972+0000 7f728f400640 1 -- 172.21.15.45:0/4202033385 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7288039070 con 0x7f72900d3e90 2024-04-16T16:17:54.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.972+0000 7f728f400640 1 -- 172.21.15.45:0/4202033385 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f728802cbd0 con 0x7f72900d3e90 2024-04-16T16:17:54.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.972+0000 7f728f400640 1 -- 172.21.15.45:0/4202033385 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7288034440 con 0x7f72900d3e90 2024-04-16T16:17:54.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.973+0000 7f7296000640 1 -- 172.21.15.45:0/4202033385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72900d3e90 msgr2=0x7f72901622f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:54.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.973+0000 7f7296000640 1 --2- 172.21.15.45:0/4202033385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72900d3e90 0x7f72901622f0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f7288009d90 tx=0x7f728802be20 comp rx=0 tx=0).stop 2024-04-16T16:17:54.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.973+0000 7f7296000640 1 -- 172.21.15.45:0/4202033385 shutdown_connections 2024-04-16T16:17:54.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.973+0000 7f7296000640 1 --2- 172.21.15.45:0/4202033385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72900d3e90 0x7f72901622f0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.973+0000 7f7296000640 1 --2- 172.21.15.45:0/4202033385 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72900b6310 0x7f7290161db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.973+0000 7f7296000640 1 -- 172.21.15.45:0/4202033385 >> 172.21.15.45:0/4202033385 conn(0x7f729006cf90 msgr2=0x7f729006d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.973+0000 7f7296000640 1 -- 172.21.15.45:0/4202033385 shutdown_connections 2024-04-16T16:17:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.973+0000 7f7296000640 1 -- 172.21.15.45:0/4202033385 wait complete. 2024-04-16T16:17:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.975+0000 7f7296000640 1 Processor -- start 2024-04-16T16:17:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.975+0000 7f7296000640 1 -- start start 2024-04-16T16:17:54.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7296000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72900b6310 0x7f72900cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:54.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7296000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72900d18f0 0x7f72900ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:54.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7296000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72900d3e90 0x7f72900caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:54.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7296000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f72900d1e50 con 0x7f72900b6310 2024-04-16T16:17:54.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7296000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72900d1cd0 con 0x7f72900d3e90 2024-04-16T16:17:54.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7296000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f72900d1fd0 con 0x7f72900d18f0 2024-04-16T16:17:54.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7294c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72900b6310 0x7f72900cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:54.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7294c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72900b6310 0x7f72900cb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57588/0 (socket says 172.21.15.45:57588) 2024-04-16T16:17:54.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7294c00640 1 -- 172.21.15.45:0/1667841401 learned_addr learned my addr 172.21.15.45:0/1667841401 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:54.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7295600640 1 --2- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72900d3e90 0x7f72900caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:54.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f728fe00640 1 --2- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72900d18f0 0x7f72900ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:54.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7294c00640 1 -- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72900d18f0 msgr2=0x7f72900ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:54.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7294c00640 1 --2- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72900d18f0 0x7f72900ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7294c00640 1 -- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72900d3e90 msgr2=0x7f72900caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:54.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7294c00640 1 --2- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72900d3e90 0x7f72900caaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:54.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7294c00640 1 -- 172.21.15.45:0/1667841401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72900c9a50 con 0x7f72900b6310 2024-04-16T16:17:54.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f7295600640 1 --2- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72900d3e90 0x7f72900caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:54.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.976+0000 7f728fe00640 1 --2- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72900d18f0 0x7f72900ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:54.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.977+0000 7f7294c00640 1 --2- 172.21.15.45:0/1667841401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72900b6310 0x7f72900cb960 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f728400c400 tx=0x7f728400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:54.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.977+0000 7f728cc00640 1 -- 172.21.15.45:0/1667841401 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284017070 con 0x7f72900b6310 2024-04-16T16:17:54.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.977+0000 7f728cc00640 1 -- 172.21.15.45:0/1667841401 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7284005ce0 con 0x7f72900b6310 2024-04-16T16:17:54.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.977+0000 7f728cc00640 1 -- 172.21.15.45:0/1667841401 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72840072f0 con 0x7f72900b6310 2024-04-16T16:17:54.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.977+0000 7f7296000640 1 -- 172.21.15.45:0/1667841401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72900c9d30 con 0x7f72900b6310 2024-04-16T16:17:54.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.977+0000 7f7296000640 1 -- 172.21.15.45:0/1667841401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72900ca2f0 con 0x7f72900b6310 2024-04-16T16:17:54.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.979+0000 7f7296000640 1 -- 172.21.15.45:0/1667841401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7290075780 con 0x7f72900b6310 2024-04-16T16:17:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.980+0000 7f728cc00640 1 -- 172.21.15.45:0/1667841401 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 21) ==== 91142+0+0 (secure 0 0 0) 0x7f728401a070 con 0x7f72900b6310 2024-04-16T16:17:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.981+0000 7f728cc00640 1 --2- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f727006ca10 0x7f727006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.981+0000 7f728cc00640 1 -- 172.21.15.45:0/1667841401 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6139+0+0 (secure 0 0 0) 0x7f728408b2b0 con 0x7f72900b6310 2024-04-16T16:17:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.981+0000 7f728fe00640 1 -- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f727006ca10 msgr2=0x7f727006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/541757982 2024-04-16T16:17:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.981+0000 7f728fe00640 1 --2- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f727006ca10 0x7f727006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:17:54.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:54.985+0000 7f728cc00640 1 -- 172.21.15.45:0/1667841401 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7284059410 con 0x7f72900b6310 2024-04-16T16:17:55.070 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:55.070+0000 7f51fbe3e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:17:55.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.092+0000 7f728cc00640 1 -- 172.21.15.45:0/1667841401 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 22) ==== 90751+0+0 (secure 0 0 0) 0x7f7284055380 con 0x7f72900b6310 2024-04-16T16:17:55.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.092+0000 7f728cc00640 1 -- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f727006ca10 msgr2=0x7f727006eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:17:55.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.092+0000 7f728cc00640 1 --2- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f727006ca10 0x7f727006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.167 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:55.167+0000 7f5175c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:55.167 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:55.167+0000 7f5175c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:55.167 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:55.167+0000 7f5175c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:55.167 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:55.167+0000 7f5175c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:55.167 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:55.167+0000 7f5175c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:55.168 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:55.168+0000 7f5179800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:55.168 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:55.168+0000 7f5179800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:55.168 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:55.168+0000 7f5179800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:55.168 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:55.168+0000 7f5179800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:55.168 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:17:55.168+0000 7f5179800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:17:55.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.206+0000 7f7296000640 1 -- 172.21.15.45:0/1667841401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f72900d25a0 con 0x7f72900b6310 2024-04-16T16:17:55.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.207+0000 7f728cc00640 1 -- 172.21.15.45:0/1667841401 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v74) ==== 104+0+11 (secure 0 0 0) 0x7f7284079a20 con 0x7f72900b6310 2024-04-16T16:17:55.207 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:17:55.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.209+0000 7f7296000640 1 -- 172.21.15.45:0/1667841401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72900b6310 msgr2=0x7f72900cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:55.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.209+0000 7f7296000640 1 --2- 172.21.15.45:0/1667841401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72900b6310 0x7f72900cb960 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f728400c400 tx=0x7f728400c8c0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.209+0000 7f7296000640 1 -- 172.21.15.45:0/1667841401 shutdown_connections 2024-04-16T16:17:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.209+0000 7f7296000640 1 --2- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:6832/541757982,v1:172.21.15.103:6833/541757982] conn(0x7f727006ca10 0x7f727006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.209+0000 7f7296000640 1 --2- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72900d3e90 0x7f72900caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.209+0000 7f7296000640 1 --2- 172.21.15.45:0/1667841401 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72900d18f0 0x7f72900ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.209+0000 7f7296000640 1 --2- 172.21.15.45:0/1667841401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72900b6310 0x7f72900cb960 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.209+0000 7f7296000640 1 -- 172.21.15.45:0/1667841401 >> 172.21.15.45:0/1667841401 conn(0x7f729006cf90 msgr2=0x7f72900b42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.209+0000 7f7296000640 1 -- 172.21.15.45:0/1667841401 shutdown_connections 2024-04-16T16:17:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.209+0000 7f7296000640 1 -- 172.21.15.45:0/1667841401 wait complete. 2024-04-16T16:17:55.226 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:17:55.245 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:55.245+0000 7f24c2a0d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:17:55.307 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:55.307+0000 7f24c2a0d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:17:55.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.429+0000 7f218b800640 1 Processor -- start 2024-04-16T16:17:55.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.429+0000 7f218b800640 1 -- start start 2024-04-16T16:17:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.431+0000 7f218b800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f218415b920 0x7f21840b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.431+0000 7f218b800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21840d0e80 0x7f218406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.431+0000 7f218b800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21840ccb70 con 0x7f21840afda0 2024-04-16T16:17:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.431+0000 7f218b800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f218406e7a0 con 0x7f218415b920 2024-04-16T16:17:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.431+0000 7f218b800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f218406f170 con 0x7f21840d0e80 2024-04-16T16:17:55.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.432+0000 7f2188e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f218415b920 0x7f21840b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:55.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.432+0000 7f2189800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21840d0e80 0x7f218406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:55.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.432+0000 7f2188e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f218415b920 0x7f21840b4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:54854/0 (socket says 172.21.15.45:54854) 2024-04-16T16:17:55.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.432+0000 7f2189800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21840d0e80 0x7f218406ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:35232/0 (socket says 172.21.15.45:35232) 2024-04-16T16:17:55.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.432+0000 7f2188e00640 1 -- 172.21.15.45:0/452807646 learned_addr learned my addr 172.21.15.45:0/452807646 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.432+0000 7f2188e00640 1 -- 172.21.15.45:0/452807646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21840d0e80 msgr2=0x7f218406ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.432+0000 7f2188e00640 1 --2- 172.21.15.45:0/452807646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21840d0e80 0x7f218406ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.432+0000 7f2188e00640 1 -- 172.21.15.45:0/452807646 >> v1:172.21.15.103:6789/0 conn(0x7f21840afda0 legacy=0x7f21840d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.432+0000 7f2188e00640 1 -- 172.21.15.45:0/452807646 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21840cce50 con 0x7f218415b920 2024-04-16T16:17:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.434+0000 7f2188e00640 1 --2- 172.21.15.45:0/452807646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f218415b920 0x7f21840b4e20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2174009870 tx=0x7f217402db00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6e0dfc9ef88ddc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:55.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.434+0000 7f2183e00640 1 -- 172.21.15.45:0/452807646 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217402f8c0 con 0x7f218415b920 2024-04-16T16:17:55.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.434+0000 7f2183e00640 1 -- 172.21.15.45:0/452807646 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f217402fa60 con 0x7f218415b920 2024-04-16T16:17:55.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.434+0000 7f2183e00640 1 -- 172.21.15.45:0/452807646 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217402fe30 con 0x7f218415b920 2024-04-16T16:17:55.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.435+0000 7f218b800640 1 -- 172.21.15.45:0/452807646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f218415b920 msgr2=0x7f21840b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:55.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.435+0000 7f218b800640 1 --2- 172.21.15.45:0/452807646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f218415b920 0x7f21840b4e20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2174009870 tx=0x7f217402db00 comp rx=0 tx=0).stop 2024-04-16T16:17:55.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.435+0000 7f218b800640 1 -- 172.21.15.45:0/452807646 shutdown_connections 2024-04-16T16:17:55.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.435+0000 7f218b800640 1 --2- 172.21.15.45:0/452807646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21840d0e80 0x7f218406ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.435+0000 7f218b800640 1 --2- 172.21.15.45:0/452807646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f218415b920 0x7f21840b4e20 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.435+0000 7f218b800640 1 -- 172.21.15.45:0/452807646 >> 172.21.15.45:0/452807646 conn(0x7f21840b3970 msgr2=0x7f21840b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:55.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.435+0000 7f218b800640 1 -- 172.21.15.45:0/452807646 shutdown_connections 2024-04-16T16:17:55.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.435+0000 7f218b800640 1 -- 172.21.15.45:0/452807646 wait complete. 2024-04-16T16:17:55.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.437+0000 7f218b800640 1 Processor -- start 2024-04-16T16:17:55.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.437+0000 7f218b800640 1 -- start start 2024-04-16T16:17:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.438+0000 7f218b800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21840afda0 0x7f2184143390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.438+0000 7f218b800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21840d0e80 0x7f21840b0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.439+0000 7f218b800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f218415b920 0x7f21840b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.439+0000 7f218b800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f21840ce760 con 0x7f21840d0e80 2024-04-16T16:17:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.439+0000 7f218b800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f21840ce5e0 con 0x7f21840afda0 2024-04-16T16:17:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.439+0000 7f218b800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f21840ce8e0 con 0x7f218415b920 2024-04-16T16:17:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.439+0000 7f217be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21840d0e80 0x7f21840b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.439+0000 7f217be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21840d0e80 0x7f21840b0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57598/0 (socket says 172.21.15.45:57598) 2024-04-16T16:17:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.439+0000 7f217be00640 1 -- 172.21.15.45:0/3205332433 learned_addr learned my addr 172.21.15.45:0/3205332433 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.439+0000 7f2189800640 1 --2- 172.21.15.45:0/3205332433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f218415b920 0x7f21840b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.439+0000 7f2188e00640 1 --2- 172.21.15.45:0/3205332433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21840afda0 0x7f2184143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.439+0000 7f217be00640 1 -- 172.21.15.45:0/3205332433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f218415b920 msgr2=0x7f21840b0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.439+0000 7f217be00640 1 --2- 172.21.15.45:0/3205332433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f218415b920 0x7f21840b0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.439+0000 7f217be00640 1 -- 172.21.15.45:0/3205332433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21840afda0 msgr2=0x7f2184143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:55.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.439+0000 7f217be00640 1 --2- 172.21.15.45:0/3205332433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21840afda0 0x7f2184143390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.439+0000 7f217be00640 1 -- 172.21.15.45:0/3205332433 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218413d2f0 con 0x7f21840d0e80 2024-04-16T16:17:55.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.440+0000 7f217be00640 1 --2- 172.21.15.45:0/3205332433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21840d0e80 0x7f21840b0550 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f216c00c430 tx=0x7f216c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:55.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.440+0000 7f2181600640 1 -- 172.21.15.45:0/3205332433 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c0052d0 con 0x7f21840d0e80 2024-04-16T16:17:55.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.440+0000 7f2181600640 1 -- 172.21.15.45:0/3205332433 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f216c005490 con 0x7f21840d0e80 2024-04-16T16:17:55.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.440+0000 7f2181600640 1 -- 172.21.15.45:0/3205332433 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c007260 con 0x7f21840d0e80 2024-04-16T16:17:55.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.440+0000 7f218b800640 1 -- 172.21.15.45:0/3205332433 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f218415e780 con 0x7f21840d0e80 2024-04-16T16:17:55.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.440+0000 7f218b800640 1 -- 172.21.15.45:0/3205332433 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f218414fb80 con 0x7f21840d0e80 2024-04-16T16:17:55.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.441+0000 7f218b800640 1 -- 172.21.15.45:0/3205332433 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2184075780 con 0x7f21840d0e80 2024-04-16T16:17:55.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.442+0000 7f2181600640 1 -- 172.21.15.45:0/3205332433 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 22) ==== 90751+0+0 (secure 0 0 0) 0x7f216c007400 con 0x7f21840d0e80 2024-04-16T16:17:55.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.444+0000 7f2181600640 1 -- 172.21.15.45:0/3205332433 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6483+0+0 (secure 0 0 0) 0x7f216c08afd0 con 0x7f21840d0e80 2024-04-16T16:17:55.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.448+0000 7f2181600640 1 -- 172.21.15.45:0/3205332433 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f216c0593c0 con 0x7f21840d0e80 2024-04-16T16:17:55.451 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:17:55.451+0000 7f24c2a0d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:17:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.653+0000 7f218b800640 1 -- 172.21.15.45:0/3205332433 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f218414f950 con 0x7f21840d0e80 2024-04-16T16:17:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.653+0000 7f2181600640 1 -- 172.21.15.45:0/3205332433 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v74) ==== 100+0+11 (secure 0 0 0) 0x7f216c058cc0 con 0x7f21840d0e80 2024-04-16T16:17:55.653 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:17:55.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.656+0000 7f218b800640 1 -- 172.21.15.45:0/3205332433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21840d0e80 msgr2=0x7f21840b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:55.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.656+0000 7f218b800640 1 --2- 172.21.15.45:0/3205332433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21840d0e80 0x7f21840b0550 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f216c00c430 tx=0x7f216c00c8f0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.656+0000 7f218b800640 1 -- 172.21.15.45:0/3205332433 shutdown_connections 2024-04-16T16:17:55.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.656+0000 7f218b800640 1 --2- 172.21.15.45:0/3205332433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f218415b920 0x7f21840b0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.657+0000 7f218b800640 1 --2- 172.21.15.45:0/3205332433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21840d0e80 0x7f21840b0550 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.657+0000 7f218b800640 1 --2- 172.21.15.45:0/3205332433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21840afda0 0x7f2184143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.657+0000 7f218b800640 1 -- 172.21.15.45:0/3205332433 >> 172.21.15.45:0/3205332433 conn(0x7f21840b3970 msgr2=0x7f2184157020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:55.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.657+0000 7f218b800640 1 -- 172.21.15.45:0/3205332433 shutdown_connections 2024-04-16T16:17:55.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.657+0000 7f218b800640 1 -- 172.21.15.45:0/3205332433 wait complete. 2024-04-16T16:17:55.677 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:17:55.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.938+0000 7f4f55200640 1 Processor -- start 2024-04-16T16:17:55.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.939+0000 7f4f55200640 1 -- start start 2024-04-16T16:17:55.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.940+0000 7f4f55200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f500c6fc0 0x7f4f5006f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:55.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.940+0000 7f4f55200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4f5015cdc0 0x7f4f500c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:55.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.941+0000 7f4f55200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f5015d600 con 0x7f4f500c6320 2024-04-16T16:17:55.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.941+0000 7f4f55200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4f500c5670 con 0x7f4f5015cdc0 2024-04-16T16:17:55.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.941+0000 7f4f55200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4f5015d8e0 con 0x7f4f500c6fc0 2024-04-16T16:17:55.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.941+0000 7f4f4fe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4f500c6320 0x7f4f5006df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52246/0 (socket says 172.21.15.45:52246) 2024-04-16T16:17:55.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.941+0000 7f4f4fe00640 1 -- 172.21.15.45:0/4160249478 learned_addr learned my addr 172.21.15.45:0/4160249478 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:55.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.941+0000 7f4f4ea00640 1 --2- 172.21.15.45:0/4160249478 >> v2:172.21.15.103:3300/0 conn(0x7f4f5015cdc0 0x7f4f500c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:55.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.941+0000 7f4f4f400640 1 --2- 172.21.15.45:0/4160249478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f500c6fc0 0x7f4f5006f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:55.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.941+0000 7f4f4e000640 1 -- 172.21.15.45:0/4160249478 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2775522346 0 0) 0x7f4f5015d600 con 0x7f4f500c6320 2024-04-16T16:17:55.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.941+0000 7f4f4e000640 1 -- 172.21.15.45:0/4160249478 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f34003520 con 0x7f4f500c6320 2024-04-16T16:17:55.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.942+0000 7f4f4e000640 1 -- 172.21.15.45:0/4160249478 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2312887956 0 0) 0x7f4f34003520 con 0x7f4f500c6320 2024-04-16T16:17:55.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.942+0000 7f4f4e000640 1 -- 172.21.15.45:0/4160249478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f500c6fc0 msgr2=0x7f4f5006f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:55.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.942+0000 7f4f4e000640 1 --2- 172.21.15.45:0/4160249478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f500c6fc0 0x7f4f5006f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.942+0000 7f4f4e000640 1 -- 172.21.15.45:0/4160249478 >> v2:172.21.15.103:3300/0 conn(0x7f4f5015cdc0 msgr2=0x7f4f500c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:55.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.942+0000 7f4f4e000640 1 --2- 172.21.15.45:0/4160249478 >> v2:172.21.15.103:3300/0 conn(0x7f4f5015cdc0 0x7f4f500c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.942+0000 7f4f4e000640 1 -- 172.21.15.45:0/4160249478 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f5015dbc0 con 0x7f4f500c6320 2024-04-16T16:17:55.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.942+0000 7f4f4e000640 1 -- 172.21.15.45:0/4160249478 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f4f48001670 con 0x7f4f500c6320 2024-04-16T16:17:55.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.942+0000 7f4f4e000640 1 -- 172.21.15.45:0/4160249478 >> v1:172.21.15.45:6789/0 conn(0x7f4f500c6320 legacy=0x7f4f5006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:55.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.942+0000 7f4f4e000640 1 --2- 172.21.15.45:0/4160249478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f340039b0 0x7f4f34003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:55.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.942+0000 7f4f4e000640 1 --2- 172.21.15.45:0/4160249478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f500c6fc0 0x7f4f34004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:55.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.942+0000 7f4f4e000640 1 --2- 172.21.15.45:0/4160249478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f34004d40 0x7f4f340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:55.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.942+0000 7f4f4e000640 1 -- 172.21.15.45:0/4160249478 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4f500c5670 con 0x7f4f340039b0 2024-04-16T16:17:55.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.942+0000 7f4f4e000640 1 -- 172.21.15.45:0/4160249478 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5015d8e0 con 0x7f4f34004d40 2024-04-16T16:17:55.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.942+0000 7f4f4e000640 1 -- 172.21.15.45:0/4160249478 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4f3400b9c0 con 0x7f4f500c6fc0 2024-04-16T16:17:55.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.943+0000 7f4f4f400640 1 --2- 172.21.15.45:0/4160249478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f340039b0 0x7f4f34003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:55.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.943+0000 7f4f4f400640 1 -- 172.21.15.45:0/4160249478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f500c6fc0 msgr2=0x7f4f34004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:55.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.943+0000 7f4f4f400640 1 --2- 172.21.15.45:0/4160249478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f500c6fc0 0x7f4f34004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.943+0000 7f4f4f400640 1 -- 172.21.15.45:0/4160249478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f34004d40 msgr2=0x7f4f340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:55.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.943+0000 7f4f4f400640 1 --2- 172.21.15.45:0/4160249478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f34004d40 0x7f4f340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.943+0000 7f4f4f400640 1 -- 172.21.15.45:0/4160249478 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f5015dbc0 con 0x7f4f340039b0 2024-04-16T16:17:55.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.944+0000 7f4f4f400640 1 --2- 172.21.15.45:0/4160249478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f340039b0 0x7f4f34003dd0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f4f40009fc0 tx=0x7f4f34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfb9268dca6dc64 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:55.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.944+0000 7f4f4e000640 1 -- 172.21.15.45:0/4160249478 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f40039070 con 0x7f4f340039b0 2024-04-16T16:17:55.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.944+0000 7f4f4e000640 1 -- 172.21.15.45:0/4160249478 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f4002cc60 con 0x7f4f340039b0 2024-04-16T16:17:55.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.944+0000 7f4f4e000640 1 -- 172.21.15.45:0/4160249478 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f40009c10 con 0x7f4f340039b0 2024-04-16T16:17:55.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.945+0000 7f4f55200640 1 -- 172.21.15.45:0/4160249478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f340039b0 msgr2=0x7f4f34003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:55.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.945+0000 7f4f55200640 1 --2- 172.21.15.45:0/4160249478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f340039b0 0x7f4f34003dd0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f4f40009fc0 tx=0x7f4f34005830 comp rx=0 tx=0).stop 2024-04-16T16:17:55.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.945+0000 7f4f4fe00640 1 -- 172.21.15.45:0/4160249478 reap_dead start 2024-04-16T16:17:55.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.945+0000 7f4f55200640 1 -- 172.21.15.45:0/4160249478 shutdown_connections 2024-04-16T16:17:55.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.945+0000 7f4f55200640 1 -- 172.21.15.45:0/4160249478 >> 172.21.15.45:0/4160249478 conn(0x7f4f5006b930 msgr2=0x7f4f5006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:55.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.945+0000 7f4f55200640 1 -- 172.21.15.45:0/4160249478 shutdown_connections 2024-04-16T16:17:55.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.945+0000 7f4f55200640 1 -- 172.21.15.45:0/4160249478 wait complete. 2024-04-16T16:17:55.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.947+0000 7f4f55200640 1 Processor -- start 2024-04-16T16:17:55.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.947+0000 7f4f55200640 1 -- start start 2024-04-16T16:17:55.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.948+0000 7f4f55200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f340039b0 0x7f4f5014e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:55.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.948+0000 7f4f55200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f500cdd90 0x7f4f500b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:55.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.948+0000 7f4f55200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f500b55d0 0x7f4f501f4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.948+0000 7f4f55200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5015d3c0 con 0x7f4f500b55d0 2024-04-16T16:17:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.948+0000 7f4f55200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5015d240 con 0x7f4f500cdd90 2024-04-16T16:17:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.948+0000 7f4f55200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4f500c73a0 con 0x7f4f340039b0 2024-04-16T16:17:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.948+0000 7f4f4fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f500b55d0 0x7f4f501f4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.948+0000 7f4f4fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f500b55d0 0x7f4f501f4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57620/0 (socket says 172.21.15.45:57620) 2024-04-16T16:17:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.948+0000 7f4f4fe00640 1 -- 172.21.15.45:0/2032961671 learned_addr learned my addr 172.21.15.45:0/2032961671 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.949+0000 7f4f4f400640 1 --2- 172.21.15.45:0/2032961671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f340039b0 0x7f4f5014e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.949+0000 7f4f4ea00640 1 --2- 172.21.15.45:0/2032961671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f500cdd90 0x7f4f500b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.949+0000 7f4f4fe00640 1 -- 172.21.15.45:0/2032961671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f340039b0 msgr2=0x7f4f5014e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.949+0000 7f4f4fe00640 1 --2- 172.21.15.45:0/2032961671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f340039b0 0x7f4f5014e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.949+0000 7f4f4fe00640 1 -- 172.21.15.45:0/2032961671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f500cdd90 msgr2=0x7f4f500b5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.949+0000 7f4f4fe00640 1 --2- 172.21.15.45:0/2032961671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f500cdd90 0x7f4f500b5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.949+0000 7f4f4fe00640 1 -- 172.21.15.45:0/2032961671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f501f5830 con 0x7f4f500b55d0 2024-04-16T16:17:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.949+0000 7f4f4fe00640 1 --2- 172.21.15.45:0/2032961671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f500b55d0 0x7f4f501f4a90 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f4f500b5be0 tx=0x7f4f4800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:55.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.949+0000 7f4f47400640 1 -- 172.21.15.45:0/2032961671 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f480072d0 con 0x7f4f500b55d0 2024-04-16T16:17:55.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.949+0000 7f4f47400640 1 -- 172.21.15.45:0/2032961671 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f48007490 con 0x7f4f500b55d0 2024-04-16T16:17:55.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.949+0000 7f4f47400640 1 -- 172.21.15.45:0/2032961671 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f48004a20 con 0x7f4f500b55d0 2024-04-16T16:17:55.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.950+0000 7f4f55200640 1 -- 172.21.15.45:0/2032961671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f500cd2d0 con 0x7f4f500b55d0 2024-04-16T16:17:55.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.950+0000 7f4f55200640 1 -- 172.21.15.45:0/2032961671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f500cd890 con 0x7f4f500b55d0 2024-04-16T16:17:55.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.951+0000 7f4f55200640 1 -- 172.21.15.45:0/2032961671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f50075780 con 0x7f4f500b55d0 2024-04-16T16:17:55.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.952+0000 7f4f47400640 1 -- 172.21.15.45:0/2032961671 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 22) ==== 90751+0+0 (secure 0 0 0) 0x7f4f48011070 con 0x7f4f500b55d0 2024-04-16T16:17:55.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.953+0000 7f4f47400640 1 -- 172.21.15.45:0/2032961671 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6483+0+0 (secure 0 0 0) 0x7f4f4808ad10 con 0x7f4f500b55d0 2024-04-16T16:17:55.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:55.957+0000 7f4f47400640 1 -- 172.21.15.45:0/2032961671 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f48090020 con 0x7f4f500b55d0 2024-04-16T16:17:56.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.101+0000 7f4f47400640 1 -- 172.21.15.45:0/2032961671 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 23) ==== 90878+0+0 (secure 0 0 0) 0x7f4f480547e0 con 0x7f4f500b55d0 2024-04-16T16:17:56.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.102+0000 7f4f47400640 1 --2- 172.21.15.45:0/2032961671 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f4f300720e0 0x7f4f300744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:56.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.105+0000 7f4f4f400640 1 --2- 172.21.15.45:0/2032961671 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f4f300720e0 0x7f4f300744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:56.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.108+0000 7f4f4f400640 1 --2- 172.21.15.45:0/2032961671 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f4f300720e0 0x7f4f300744d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4f40036000 tx=0x7f4f4002e000 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.176+0000 7f4f55200640 1 -- 172.21.15.45:0/2032961671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4f500683c0 con 0x7f4f500b55d0 2024-04-16T16:17:56.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.177+0000 7f4f47400640 1 -- 172.21.15.45:0/2032961671 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 18 entries v74) ==== 81+0+1068 (secure 0 0 0) 0x7f4f48059190 con 0x7f4f500b55d0 2024-04-16T16:17:56.177 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1692905448 2024-04-17T16:17:55.075639+0000 2024-04-16T16:17:56.178 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1736628710 2024-04-17T16:17:55.075639+0000 2024-04-16T16:17:56.178 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1783855160 2024-04-17T16:17:55.075639+0000 2024-04-16T16:17:56.178 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/541757982 2024-04-17T16:17:55.075639+0000 2024-04-16T16:17:56.178 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2044864797 2024-04-17T16:17:55.075639+0000 2024-04-16T16:17:56.178 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/4155603915 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:56.178 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/951047877 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:56.178 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1131547374 2024-04-17T16:17:55.075639+0000 2024-04-16T16:17:56.178 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3176524506 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:56.178 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/4155603915 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:56.178 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/365521086 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:56.179 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/541757982 2024-04-17T16:17:55.075639+0000 2024-04-16T16:17:56.179 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3345679536 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:56.179 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2747644735 2024-04-17T16:17:55.075639+0000 2024-04-16T16:17:56.179 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3606536347 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:56.179 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1223877773 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:56.179 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/252084277 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:56.179 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/650078727 2024-04-17T16:17:01.408806+0000 2024-04-16T16:17:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:listed 18 entries 2024-04-16T16:17:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.179+0000 7f4f55200640 1 -- 172.21.15.45:0/2032961671 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f4f300720e0 msgr2=0x7f4f300744d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.179+0000 7f4f55200640 1 --2- 172.21.15.45:0/2032961671 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f4f300720e0 0x7f4f300744d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4f40036000 tx=0x7f4f4002e000 comp rx=0 tx=0).stop 2024-04-16T16:17:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.179+0000 7f4f55200640 1 -- 172.21.15.45:0/2032961671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f500b55d0 msgr2=0x7f4f501f4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.179+0000 7f4f55200640 1 --2- 172.21.15.45:0/2032961671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f500b55d0 0x7f4f501f4a90 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f4f500b5be0 tx=0x7f4f4800cb90 comp rx=0 tx=0).stop 2024-04-16T16:17:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.180+0000 7f4f55200640 1 -- 172.21.15.45:0/2032961671 shutdown_connections 2024-04-16T16:17:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.180+0000 7f4f55200640 1 --2- 172.21.15.45:0/2032961671 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f4f300720e0 0x7f4f300744d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.180+0000 7f4f55200640 1 --2- 172.21.15.45:0/2032961671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f500b55d0 0x7f4f501f4a90 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.180+0000 7f4f55200640 1 --2- 172.21.15.45:0/2032961671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f500cdd90 0x7f4f500b5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.180+0000 7f4f55200640 1 --2- 172.21.15.45:0/2032961671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f340039b0 0x7f4f5014e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.180+0000 7f4f55200640 1 -- 172.21.15.45:0/2032961671 >> 172.21.15.45:0/2032961671 conn(0x7f4f5006b930 msgr2=0x7f4f5006df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.180+0000 7f4f55200640 1 -- 172.21.15.45:0/2032961671 shutdown_connections 2024-04-16T16:17:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.180+0000 7f4f55200640 1 -- 172.21.15.45:0/2032961671 wait complete. 2024-04-16T16:17:56.197 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-16T16:17:56.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:56.197 DEBUG:teuthology.orchestra.run.smithi103:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-04-16T16:17:56.303 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:17:56.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:17:56.303 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-16T16:17:56.353 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:17:56.353 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:17:56.353 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:56.412 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:17:56.412 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:17:56.412 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:17:56.412 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:17:56.471 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:17:56.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:17:56.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.695+0000 7fc788000640 1 Processor -- start 2024-04-16T16:17:56.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.696+0000 7fc788000640 1 -- start start 2024-04-16T16:17:56.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.697+0000 7fc788000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7801462e0 0x7fc780072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:56.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.697+0000 7fc788000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc7800728c0 0x7fc7800ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:56.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.697+0000 7fc788000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7800cf6d0 con 0x7fc7800cec00 2024-04-16T16:17:56.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.697+0000 7fc788000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc780072d20 con 0x7fc7800728c0 2024-04-16T16:17:56.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.697+0000 7fc788000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc7801466c0 con 0x7fc7801462e0 2024-04-16T16:17:56.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.697+0000 7fc784c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc7800728c0 0x7fc7800ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:56.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.697+0000 7fc784c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc7800728c0 0x7fc7800ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57632/0 (socket says 172.21.15.45:57632) 2024-04-16T16:17:56.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.697+0000 7fc784c00640 1 -- 172.21.15.45:0/541807697 learned_addr learned my addr 172.21.15.45:0/541807697 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:56.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.698+0000 7fc785600640 1 --2- 172.21.15.45:0/541807697 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7801462e0 0x7fc780072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:56.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.698+0000 7fc784c00640 1 -- 172.21.15.45:0/541807697 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7801462e0 msgr2=0x7fc780072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:56.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.698+0000 7fc784c00640 1 --2- 172.21.15.45:0/541807697 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7801462e0 0x7fc780072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:56.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.698+0000 7fc784c00640 1 -- 172.21.15.45:0/541807697 >> v1:172.21.15.45:6789/0 conn(0x7fc7800cec00 legacy=0x7fc78015da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:56.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.698+0000 7fc784c00640 1 -- 172.21.15.45:0/541807697 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7800cf3f0 con 0x7fc7800728c0 2024-04-16T16:17:56.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.698+0000 7fc785600640 1 --2- 172.21.15.45:0/541807697 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7801462e0 0x7fc780072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:56.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.699+0000 7fc784c00640 1 --2- 172.21.15.45:0/541807697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7800728c0 0x7fc7800ce6c0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009870 tx=0x7fc76c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c24f2f04b349e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:56.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.699+0000 7fc77fe00640 1 -- 172.21.15.45:0/541807697 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c035020 con 0x7fc7800728c0 2024-04-16T16:17:56.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.699+0000 7fc77fe00640 1 -- 172.21.15.45:0/541807697 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc76c031640 con 0x7fc7800728c0 2024-04-16T16:17:56.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.699+0000 7fc77fe00640 1 -- 172.21.15.45:0/541807697 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c02d040 con 0x7fc7800728c0 2024-04-16T16:17:56.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.700+0000 7fc788000640 1 -- 172.21.15.45:0/541807697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7800728c0 msgr2=0x7fc7800ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:56.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.700+0000 7fc788000640 1 --2- 172.21.15.45:0/541807697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7800728c0 0x7fc7800ce6c0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009870 tx=0x7fc76c02b8f0 comp rx=0 tx=0).stop 2024-04-16T16:17:56.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.700+0000 7fc788000640 1 -- 172.21.15.45:0/541807697 shutdown_connections 2024-04-16T16:17:56.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.700+0000 7fc788000640 1 --2- 172.21.15.45:0/541807697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7800728c0 0x7fc7800ce6c0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:56.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.700+0000 7fc788000640 1 --2- 172.21.15.45:0/541807697 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7801462e0 0x7fc780072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:56.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.700+0000 7fc788000640 1 -- 172.21.15.45:0/541807697 >> 172.21.15.45:0/541807697 conn(0x7fc7800b2750 msgr2=0x7fc7800b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:56.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.700+0000 7fc788000640 1 -- 172.21.15.45:0/541807697 shutdown_connections 2024-04-16T16:17:56.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.700+0000 7fc788000640 1 -- 172.21.15.45:0/541807697 wait complete. 2024-04-16T16:17:56.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.702+0000 7fc788000640 1 Processor -- start 2024-04-16T16:17:56.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.702+0000 7fc788000640 1 -- start start 2024-04-16T16:17:56.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.703+0000 7fc788000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7800728c0 0x7fc7800af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:56.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.703+0000 7fc788000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7800cec00 0x7fc78013e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:56.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.703+0000 7fc788000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7801462e0 0x7fc78015eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:56.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.703+0000 7fc788000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc78015d540 con 0x7fc7800728c0 2024-04-16T16:17:56.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.703+0000 7fc788000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc78015d260 con 0x7fc7801462e0 2024-04-16T16:17:56.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.703+0000 7fc788000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc7800cf090 con 0x7fc7800cec00 2024-04-16T16:17:56.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.703+0000 7fc785600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7800728c0 0x7fc7800af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:56.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.703+0000 7fc785600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7800728c0 0x7fc7800af8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57646/0 (socket says 172.21.15.45:57646) 2024-04-16T16:17:56.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.703+0000 7fc785600640 1 -- 172.21.15.45:0/1891845977 learned_addr learned my addr 172.21.15.45:0/1891845977 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:56.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.703+0000 7fc785600640 1 -- 172.21.15.45:0/1891845977 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7800cec00 msgr2=0x7fc78013e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:56.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.704+0000 7fc785600640 1 --2- 172.21.15.45:0/1891845977 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7800cec00 0x7fc78013e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:56.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.704+0000 7fc785600640 1 -- 172.21.15.45:0/1891845977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7801462e0 msgr2=0x7fc78015eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:56.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.704+0000 7fc785600640 1 --2- 172.21.15.45:0/1891845977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7801462e0 0x7fc78015eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:56.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.704+0000 7fc785600640 1 -- 172.21.15.45:0/1891845977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc78013ea70 con 0x7fc7800728c0 2024-04-16T16:17:56.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.704+0000 7fc785600640 1 --2- 172.21.15.45:0/1891845977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7800728c0 0x7fc7800af8b0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fc77400cf60 tx=0x7fc77400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:56.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.704+0000 7fc77d600640 1 -- 172.21.15.45:0/1891845977 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77401a070 con 0x7fc7800728c0 2024-04-16T16:17:56.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.704+0000 7fc77d600640 1 -- 172.21.15.45:0/1891845977 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc77400d840 con 0x7fc7800728c0 2024-04-16T16:17:56.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.704+0000 7fc788000640 1 -- 172.21.15.45:0/1891845977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc78013ed30 con 0x7fc7800728c0 2024-04-16T16:17:56.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.704+0000 7fc77d600640 1 -- 172.21.15.45:0/1891845977 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77400dc10 con 0x7fc7800728c0 2024-04-16T16:17:56.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.704+0000 7fc788000640 1 -- 172.21.15.45:0/1891845977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7800add20 con 0x7fc7800728c0 2024-04-16T16:17:56.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.706+0000 7fc788000640 1 -- 172.21.15.45:0/1891845977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7800d02c0 con 0x7fc7800728c0 2024-04-16T16:17:56.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.707+0000 7fc77d600640 1 -- 172.21.15.45:0/1891845977 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 23) ==== 90878+0+0 (secure 0 0 0) 0x7fc77401d070 con 0x7fc7800728c0 2024-04-16T16:17:56.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.708+0000 7fc77d600640 1 --2- 172.21.15.45:0/1891845977 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc76006c740 0x7fc76006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:56.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.708+0000 7fc77d600640 1 -- 172.21.15.45:0/1891845977 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6483+0+0 (secure 0 0 0) 0x7fc77408c2a0 con 0x7fc7800728c0 2024-04-16T16:17:56.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.709+0000 7fc784c00640 1 --2- 172.21.15.45:0/1891845977 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc76006c740 0x7fc76006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:56.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.710+0000 7fc784c00640 1 --2- 172.21.15.45:0/1891845977 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc76006c740 0x7fc76006ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc76c036000 tx=0x7fc76c030000 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:56.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.713+0000 7fc77d600640 1 -- 172.21.15.45:0/1891845977 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc7800d02c0 con 0x7fc7800728c0 2024-04-16T16:17:56.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.956+0000 7fc788000640 1 -- 172.21.15.45:0/1891845977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc780075780 con 0x7fc7800728c0 2024-04-16T16:17:56.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.956+0000 7fc77d600640 1 -- 172.21.15.45:0/1891845977 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5202 (secure 0 0 0) 0x7fc77408c0c0 con 0x7fc7800728c0 2024-04-16T16:17:56.957 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:56.957 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":55,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:17:53.689159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6402,"mds_1":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"c","rank":1,"incarnation":52,"state":"up:active","state_seq":11,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6399,6402]},"id":5}]} 2024-04-16T16:17:56.957 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 55 2024-04-16T16:17:56.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.959+0000 7fc788000640 1 -- 172.21.15.45:0/1891845977 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc76006c740 msgr2=0x7fc76006ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:56.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.959+0000 7fc788000640 1 --2- 172.21.15.45:0/1891845977 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc76006c740 0x7fc76006ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc76c036000 tx=0x7fc76c030000 comp rx=0 tx=0).stop 2024-04-16T16:17:56.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.959+0000 7fc788000640 1 -- 172.21.15.45:0/1891845977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7800728c0 msgr2=0x7fc7800af8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:56.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.959+0000 7fc788000640 1 --2- 172.21.15.45:0/1891845977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7800728c0 0x7fc7800af8b0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fc77400cf60 tx=0x7fc77400cf90 comp rx=0 tx=0).stop 2024-04-16T16:17:56.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.959+0000 7fc788000640 1 -- 172.21.15.45:0/1891845977 shutdown_connections 2024-04-16T16:17:56.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.959+0000 7fc788000640 1 --2- 172.21.15.45:0/1891845977 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc76006c740 0x7fc76006ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:56.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.959+0000 7fc788000640 1 --2- 172.21.15.45:0/1891845977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7801462e0 0x7fc78015eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:56.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.959+0000 7fc788000640 1 --2- 172.21.15.45:0/1891845977 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7800cec00 0x7fc78013e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:56.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.959+0000 7fc788000640 1 --2- 172.21.15.45:0/1891845977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7800728c0 0x7fc7800af8b0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:56.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.959+0000 7fc788000640 1 -- 172.21.15.45:0/1891845977 >> 172.21.15.45:0/1891845977 conn(0x7fc7800b2750 msgr2=0x7fc7801447e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:56.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.959+0000 7fc788000640 1 -- 172.21.15.45:0/1891845977 shutdown_connections 2024-04-16T16:17:56.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:56.959+0000 7fc788000640 1 -- 172.21.15.45:0/1891845977 wait complete. 2024-04-16T16:17:56.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:17:57.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.200+0000 7f6131600640 1 Processor -- start 2024-04-16T16:17:57.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.200+0000 7f6131600640 1 -- start start 2024-04-16T16:17:57.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.202+0000 7f6131600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f612c15e8f0 0x7f612c0cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:57.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.202+0000 7f6131600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f612c1632a0 con 0x7f612c0cf080 2024-04-16T16:17:57.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.202+0000 7f6131600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f612c15ecd0 con 0x7f612c0ce9a0 2024-04-16T16:17:57.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.202+0000 7f6131600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f612c0ce2e0 con 0x7f612c15e8f0 2024-04-16T16:17:57.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.202+0000 7f612be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f612c15e8f0 0x7f612c0cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:57.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.202+0000 7f612be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f612c15e8f0 0x7f612c0cc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57652/0 (socket says 172.21.15.45:57652) 2024-04-16T16:17:57.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.202+0000 7f612be00640 1 -- 172.21.15.45:0/775392743 learned_addr learned my addr 172.21.15.45:0/775392743 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:57.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.202+0000 7f6130c00640 1 --1- 172.21.15.45:0/775392743 >> v1:172.21.15.45:6789/0 conn(0x7f612c0cf080 0x7f612c162b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52266/0 (socket says 172.21.15.45:52266) 2024-04-16T16:17:57.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.203+0000 7f612be00640 1 -- 172.21.15.45:0/775392743 >> v1:172.21.15.103:6789/0 conn(0x7f612c0ce9a0 legacy=0x7f612c162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:57.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.203+0000 7f612be00640 1 -- 172.21.15.45:0/775392743 >> v1:172.21.15.45:6789/0 conn(0x7f612c0cf080 legacy=0x7f612c162b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:57.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.203+0000 7f612be00640 1 -- 172.21.15.45:0/775392743 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612c0cd6d0 con 0x7f612c15e8f0 2024-04-16T16:17:57.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.204+0000 7f612be00640 1 --2- 172.21.15.45:0/775392743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f612c15e8f0 0x7f612c0cc410 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f611c009870 tx=0x7f611c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac1bcf327b1379d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:57.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.204+0000 7f612aa00640 1 -- 172.21.15.45:0/775392743 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c034070 con 0x7f612c15e8f0 2024-04-16T16:17:57.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.204+0000 7f612aa00640 1 -- 172.21.15.45:0/775392743 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f611c030860 con 0x7f612c15e8f0 2024-04-16T16:17:57.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.204+0000 7f612aa00640 1 -- 172.21.15.45:0/775392743 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c030c30 con 0x7f612c15e8f0 2024-04-16T16:17:57.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.205+0000 7f6131600640 1 -- 172.21.15.45:0/775392743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f612c15e8f0 msgr2=0x7f612c0cc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:57.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.205+0000 7f6131600640 1 --2- 172.21.15.45:0/775392743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f612c15e8f0 0x7f612c0cc410 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f611c009870 tx=0x7f611c02b9a0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.205+0000 7f6131600640 1 -- 172.21.15.45:0/775392743 shutdown_connections 2024-04-16T16:17:57.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.205+0000 7f6131600640 1 --2- 172.21.15.45:0/775392743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f612c15e8f0 0x7f612c0cc410 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.205+0000 7f6131600640 1 -- 172.21.15.45:0/775392743 >> 172.21.15.45:0/775392743 conn(0x7f612c06b930 msgr2=0x7f612c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:57.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.205+0000 7f6131600640 1 -- 172.21.15.45:0/775392743 shutdown_connections 2024-04-16T16:17:57.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.205+0000 7f6131600640 1 -- 172.21.15.45:0/775392743 wait complete. 2024-04-16T16:17:57.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.207+0000 7f6131600640 1 Processor -- start 2024-04-16T16:17:57.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.207+0000 7f6131600640 1 -- start start 2024-04-16T16:17:57.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.208+0000 7f6131600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f612c0ce9a0 0x7f612c15acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:57.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.208+0000 7f6131600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f612c0cf080 0x7f612c14e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:57.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.208+0000 7f6131600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f612c15e8f0 0x7f612c13f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:57.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.208+0000 7f6131600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f612c0cdf70 con 0x7f612c0ce9a0 2024-04-16T16:17:57.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.208+0000 7f6131600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f612c0cdc90 con 0x7f612c15e8f0 2024-04-16T16:17:57.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.208+0000 7f6131600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f612c0cd360 con 0x7f612c0cf080 2024-04-16T16:17:57.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.208+0000 7f612be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f612c0ce9a0 0x7f612c15acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.208+0000 7f612be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f612c0ce9a0 0x7f612c15acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57660/0 (socket says 172.21.15.45:57660) 2024-04-16T16:17:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.208+0000 7f612be00640 1 -- 172.21.15.45:0/3008518702 learned_addr learned my addr 172.21.15.45:0/3008518702 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.209+0000 7f612b400640 1 --2- 172.21.15.45:0/3008518702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f612c0cf080 0x7f612c14e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.209+0000 7f6130c00640 1 --2- 172.21.15.45:0/3008518702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f612c15e8f0 0x7f612c13f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.209+0000 7f612be00640 1 -- 172.21.15.45:0/3008518702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f612c0cf080 msgr2=0x7f612c14e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.209+0000 7f612be00640 1 --2- 172.21.15.45:0/3008518702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f612c0cf080 0x7f612c14e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.209+0000 7f612be00640 1 -- 172.21.15.45:0/3008518702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f612c15e8f0 msgr2=0x7f612c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.209+0000 7f612be00640 1 --2- 172.21.15.45:0/3008518702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f612c15e8f0 0x7f612c13f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.209+0000 7f612be00640 1 -- 172.21.15.45:0/3008518702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612c13ce20 con 0x7f612c0ce9a0 2024-04-16T16:17:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.209+0000 7f612be00640 1 --2- 172.21.15.45:0/3008518702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f612c0ce9a0 0x7f612c15acf0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f611c009870 tx=0x7f611c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.210+0000 7f6123e00640 1 -- 172.21.15.45:0/3008518702 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c034040 con 0x7f612c0ce9a0 2024-04-16T16:17:57.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.210+0000 7f6123e00640 1 -- 172.21.15.45:0/3008518702 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f611c0433f0 con 0x7f612c0ce9a0 2024-04-16T16:17:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.210+0000 7f6123e00640 1 -- 172.21.15.45:0/3008518702 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c0437e0 con 0x7f612c0ce9a0 2024-04-16T16:17:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.210+0000 7f6131600640 1 -- 172.21.15.45:0/3008518702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f612c14e800 con 0x7f612c0ce9a0 2024-04-16T16:17:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.210+0000 7f6131600640 1 -- 172.21.15.45:0/3008518702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f612c14ed80 con 0x7f612c0ce9a0 2024-04-16T16:17:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.211+0000 7f6131600640 1 -- 172.21.15.45:0/3008518702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f612c075780 con 0x7f612c0ce9a0 2024-04-16T16:17:57.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.212+0000 7f6123e00640 1 -- 172.21.15.45:0/3008518702 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 24) ==== 91086+0+0 (secure 0 0 0) 0x7f611c04a060 con 0x7f612c0ce9a0 2024-04-16T16:17:57.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.213+0000 7f6123e00640 1 --2- 172.21.15.45:0/3008518702 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f610c06c9b0 0x7f610c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:57.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.214+0000 7f6123e00640 1 -- 172.21.15.45:0/3008518702 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6483+0+0 (secure 0 0 0) 0x7f611c0ae200 con 0x7f612c0ce9a0 2024-04-16T16:17:57.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.214+0000 7f612b400640 1 --2- 172.21.15.45:0/3008518702 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f610c06c9b0 0x7f610c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:57.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.216+0000 7f612b400640 1 --2- 172.21.15.45:0/3008518702 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f610c06c9b0 0x7f610c06ee70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f612c0b3ba0 tx=0x7f6118009040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:57.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.219+0000 7f6123e00640 1 -- 172.21.15.45:0/3008518702 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f611c03e070 con 0x7f612c0ce9a0 2024-04-16T16:17:57.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.425+0000 7f6131600640 1 -- 172.21.15.45:0/3008518702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f612c0b5b90 con 0x7f612c0ce9a0 2024-04-16T16:17:57.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.427+0000 7f6123e00640 1 -- 172.21.15.45:0/3008518702 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+17807 (secure 0 0 0) 0x7f611c04be90 con 0x7f612c0ce9a0 2024-04-16T16:17:57.428 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:57.428 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":74,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:17:55.075736+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:16:37.449973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-16T16:16:38.485027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/1692905448":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/1736628710":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/1783855160":"2024-04-17T16:17:55.075639+0000","172.21.15.103:6833/541757982":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/2044864797":"2024-04-17T16:17:55.075639+0000","172.21.15.103:6833/4155603915":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/951047877":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/1131547374":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/3176524506":"2024-04-17T16:17:01.408806+0000","172.21.15.103:6832/4155603915":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/365521086":"2024-04-17T16:17:01.408806+0000","172.21.15.103:6832/541757982":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/3345679536":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/2747644735":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/3606536347":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/1223877773":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/252084277":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/650078727":"2024-04-17T16:17:01.408806+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:17:57.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.430+0000 7f6131600640 1 -- 172.21.15.45:0/3008518702 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f610c06c9b0 msgr2=0x7f610c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:57.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.430+0000 7f6131600640 1 --2- 172.21.15.45:0/3008518702 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f610c06c9b0 0x7f610c06ee70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f612c0b3ba0 tx=0x7f6118009040 comp rx=0 tx=0).stop 2024-04-16T16:17:57.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.430+0000 7f6131600640 1 -- 172.21.15.45:0/3008518702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f612c0ce9a0 msgr2=0x7f612c15acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:57.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.430+0000 7f6131600640 1 --2- 172.21.15.45:0/3008518702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f612c0ce9a0 0x7f612c15acf0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f611c009870 tx=0x7f611c0046c0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.430+0000 7f6131600640 1 -- 172.21.15.45:0/3008518702 shutdown_connections 2024-04-16T16:17:57.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.430+0000 7f6131600640 1 --2- 172.21.15.45:0/3008518702 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f610c06c9b0 0x7f610c06ee70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.430+0000 7f6131600640 1 --2- 172.21.15.45:0/3008518702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f612c15e8f0 0x7f612c13f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.430+0000 7f6131600640 1 --2- 172.21.15.45:0/3008518702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f612c0cf080 0x7f612c14e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.430+0000 7f6131600640 1 --2- 172.21.15.45:0/3008518702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f612c0ce9a0 0x7f612c15acf0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.430+0000 7f6131600640 1 -- 172.21.15.45:0/3008518702 >> 172.21.15.45:0/3008518702 conn(0x7f612c06b930 msgr2=0x7f612c06eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:57.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.431+0000 7f6131600640 1 -- 172.21.15.45:0/3008518702 shutdown_connections 2024-04-16T16:17:57.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.431+0000 7f6131600640 1 -- 172.21.15.45:0/3008518702 wait complete. 2024-04-16T16:17:57.451 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:16:37.449973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:16:38.485027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:17:57.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:17:57.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.682+0000 7fe7b1c00640 1 Processor -- start 2024-04-16T16:17:57.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.682+0000 7fe7b1c00640 1 -- start start 2024-04-16T16:17:57.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.683+0000 7fe7b1c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe7ac15b550 0x7fe7ac15f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:57.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.683+0000 7fe7b1c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe7ac06f0d0 0x7fe7ac15f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:57.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.683+0000 7fe7b1c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7ac0ce7b0 con 0x7fe7ac15b930 2024-04-16T16:17:57.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.683+0000 7fe7b1c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe7ac0ced90 con 0x7fe7ac06f0d0 2024-04-16T16:17:57.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.683+0000 7fe7b1c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe7ac0d0230 con 0x7fe7ac15b550 2024-04-16T16:17:57.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.684+0000 7fe7ab400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe7ac15b930 0x7fe7ac06e9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52280/0 (socket says 172.21.15.45:52280) 2024-04-16T16:17:57.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.684+0000 7fe7ab400640 1 -- 172.21.15.45:0/560041424 learned_addr learned my addr 172.21.15.45:0/560041424 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:57.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.684+0000 7fe7b1200640 1 --2- 172.21.15.45:0/560041424 >> v2:172.21.15.45:3300/0 conn(0x7fe7ac06f0d0 0x7fe7ac15f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:57.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.685+0000 7fe7abe00640 1 --2- 172.21.15.45:0/560041424 >> v2:172.21.15.103:3300/0 conn(0x7fe7ac15b550 0x7fe7ac15f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:57.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.685+0000 7fe7aaa00640 1 -- 172.21.15.45:0/560041424 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3355089121 0 0) 0x7fe7ac0ce7b0 con 0x7fe7ac15b930 2024-04-16T16:17:57.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.685+0000 7fe7b1200640 1 -- 172.21.15.45:0/560041424 >> v2:172.21.15.103:3300/0 conn(0x7fe7ac15b550 msgr2=0x7fe7ac15f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:57.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.685+0000 7fe7b1200640 1 --2- 172.21.15.45:0/560041424 >> v2:172.21.15.103:3300/0 conn(0x7fe7ac15b550 0x7fe7ac15f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.685+0000 7fe7b1200640 1 -- 172.21.15.45:0/560041424 >> v1:172.21.15.45:6789/0 conn(0x7fe7ac15b930 legacy=0x7fe7ac06e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:57.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.685+0000 7fe7b1200640 1 -- 172.21.15.45:0/560041424 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7ac06f6b0 con 0x7fe7ac06f0d0 2024-04-16T16:17:57.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.685+0000 7fe7abe00640 1 --2- 172.21.15.45:0/560041424 >> v2:172.21.15.103:3300/0 conn(0x7fe7ac15b550 0x7fe7ac15f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:57.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.686+0000 7fe7b1200640 1 --2- 172.21.15.45:0/560041424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ac06f0d0 0x7fe7ac15f660 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0733c0 tx=0x7fe7a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b63bea7856503873 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:57.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.686+0000 7fe7aaa00640 1 -- 172.21.15.45:0/560041424 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4035020 con 0x7fe7ac06f0d0 2024-04-16T16:17:57.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.686+0000 7fe7aaa00640 1 -- 172.21.15.45:0/560041424 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a4031830 con 0x7fe7ac06f0d0 2024-04-16T16:17:57.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.686+0000 7fe7aaa00640 1 -- 172.21.15.45:0/560041424 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4031c00 con 0x7fe7ac06f0d0 2024-04-16T16:17:57.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.687+0000 7fe7b1c00640 1 -- 172.21.15.45:0/560041424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ac06f0d0 msgr2=0x7fe7ac15f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:57.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.687+0000 7fe7b1c00640 1 --2- 172.21.15.45:0/560041424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ac06f0d0 0x7fe7ac15f660 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0733c0 tx=0x7fe7a402b8d0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.687+0000 7fe7b1c00640 1 -- 172.21.15.45:0/560041424 shutdown_connections 2024-04-16T16:17:57.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.687+0000 7fe7b1c00640 1 --2- 172.21.15.45:0/560041424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ac06f0d0 0x7fe7ac15f660 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.687+0000 7fe7b1c00640 1 --2- 172.21.15.45:0/560041424 >> v2:172.21.15.103:3300/0 conn(0x7fe7ac15b550 0x7fe7ac15f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.687+0000 7fe7b1c00640 1 -- 172.21.15.45:0/560041424 >> 172.21.15.45:0/560041424 conn(0x7fe7ac067e00 msgr2=0x7fe7ac06b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:57.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.687+0000 7fe7b1c00640 1 -- 172.21.15.45:0/560041424 shutdown_connections 2024-04-16T16:17:57.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.687+0000 7fe7b1c00640 1 -- 172.21.15.45:0/560041424 wait complete. 2024-04-16T16:17:57.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.689+0000 7fe7b1c00640 1 Processor -- start 2024-04-16T16:17:57.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.689+0000 7fe7b1c00640 1 -- start start 2024-04-16T16:17:57.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.690+0000 7fe7b1c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7ac06f0d0 0x7fe7ac1f7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:57.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.690+0000 7fe7b1c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7ac15b550 0x7fe7ac151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:57.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.690+0000 7fe7b1c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ac15b930 0x7fe7ac14d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:57.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.690+0000 7fe7b1c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac0cf200 con 0x7fe7ac15b930 2024-04-16T16:17:57.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.690+0000 7fe7b1c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac06f990 con 0x7fe7ac06f0d0 2024-04-16T16:17:57.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.690+0000 7fe7b1c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe7ac0cf380 con 0x7fe7ac15b550 2024-04-16T16:17:57.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.690+0000 7fe7b1200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ac15b930 0x7fe7ac14d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:57.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.690+0000 7fe7b1200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ac15b930 0x7fe7ac14d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57672/0 (socket says 172.21.15.45:57672) 2024-04-16T16:17:57.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.690+0000 7fe7b1200640 1 -- 172.21.15.45:0/2245431889 learned_addr learned my addr 172.21.15.45:0/2245431889 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:57.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.691+0000 7fe7abe00640 1 --2- 172.21.15.45:0/2245431889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7ac06f0d0 0x7fe7ac1f7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:57.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.691+0000 7fe7ab400640 1 --2- 172.21.15.45:0/2245431889 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7ac15b550 0x7fe7ac151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:57.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.691+0000 7fe7b1200640 1 -- 172.21.15.45:0/2245431889 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7ac15b550 msgr2=0x7fe7ac151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:57.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.691+0000 7fe7b1200640 1 --2- 172.21.15.45:0/2245431889 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7ac15b550 0x7fe7ac151630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.691+0000 7fe7b1200640 1 -- 172.21.15.45:0/2245431889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7ac06f0d0 msgr2=0x7fe7ac1f7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:57.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.691+0000 7fe7b1200640 1 --2- 172.21.15.45:0/2245431889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7ac06f0d0 0x7fe7ac1f7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.691+0000 7fe7b1200640 1 -- 172.21.15.45:0/2245431889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7ac154360 con 0x7fe7ac15b930 2024-04-16T16:17:57.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.691+0000 7fe7b1200640 1 --2- 172.21.15.45:0/2245431889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ac15b930 0x7fe7ac14d990 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009890 tx=0x7fe7a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:57.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.692+0000 7fe7a3e00640 1 -- 172.21.15.45:0/2245431889 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4035050 con 0x7fe7ac15b930 2024-04-16T16:17:57.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.692+0000 7fe7a3e00640 1 -- 172.21.15.45:0/2245431889 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a4004230 con 0x7fe7ac15b930 2024-04-16T16:17:57.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.692+0000 7fe7a3e00640 1 -- 172.21.15.45:0/2245431889 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a403b780 con 0x7fe7ac15b930 2024-04-16T16:17:57.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.692+0000 7fe7b1c00640 1 -- 172.21.15.45:0/2245431889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7ac154640 con 0x7fe7ac15b930 2024-04-16T16:17:57.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.692+0000 7fe7b1c00640 1 -- 172.21.15.45:0/2245431889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7ac154c00 con 0x7fe7ac15b930 2024-04-16T16:17:57.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.693+0000 7fe7b1c00640 1 -- 172.21.15.45:0/2245431889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7ac075780 con 0x7fe7ac15b930 2024-04-16T16:17:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.694+0000 7fe7a3e00640 1 -- 172.21.15.45:0/2245431889 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 24) ==== 91086+0+0 (secure 0 0 0) 0x7fe7a4038020 con 0x7fe7ac15b930 2024-04-16T16:17:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.695+0000 7fe7a3e00640 1 --2- 172.21.15.45:0/2245431889 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fe78c06c980 0x7fe78c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.695+0000 7fe7a3e00640 1 -- 172.21.15.45:0/2245431889 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6483+0+0 (secure 0 0 0) 0x7fe7a40ade90 con 0x7fe7ac15b930 2024-04-16T16:17:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.696+0000 7fe7abe00640 1 --2- 172.21.15.45:0/2245431889 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fe78c06c980 0x7fe78c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:57.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.697+0000 7fe7abe00640 1 --2- 172.21.15.45:0/2245431889 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fe78c06c980 0x7fe78c06ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00a990 tx=0x7fe79c008040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:57.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.700+0000 7fe7a3e00640 1 -- 172.21.15.45:0/2245431889 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe7a4040070 con 0x7fe7ac15b930 2024-04-16T16:17:57.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.901+0000 7fe7b1c00640 1 -- 172.21.15.45:0/2245431889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe7ac06c550 con 0x7fe7ac15b930 2024-04-16T16:17:57.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.902+0000 7fe7a3e00640 1 -- 172.21.15.45:0/2245431889 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) ==== 93+0+10 (secure 0 0 0) 0x7fe7a4031800 con 0x7fe7ac15b930 2024-04-16T16:17:57.902 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:17:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.904+0000 7fe7b1c00640 1 -- 172.21.15.45:0/2245431889 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fe78c06c980 msgr2=0x7fe78c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.904+0000 7fe7b1c00640 1 --2- 172.21.15.45:0/2245431889 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fe78c06c980 0x7fe78c06ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00a990 tx=0x7fe79c008040 comp rx=0 tx=0).stop 2024-04-16T16:17:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.904+0000 7fe7b1c00640 1 -- 172.21.15.45:0/2245431889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ac15b930 msgr2=0x7fe7ac14d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.904+0000 7fe7b1c00640 1 --2- 172.21.15.45:0/2245431889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ac15b930 0x7fe7ac14d990 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009890 tx=0x7fe7a40046c0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.904+0000 7fe7b1c00640 1 -- 172.21.15.45:0/2245431889 shutdown_connections 2024-04-16T16:17:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.904+0000 7fe7b1c00640 1 --2- 172.21.15.45:0/2245431889 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fe78c06c980 0x7fe78c06ee40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.904+0000 7fe7b1c00640 1 --2- 172.21.15.45:0/2245431889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ac15b930 0x7fe7ac14d990 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.904+0000 7fe7b1c00640 1 --2- 172.21.15.45:0/2245431889 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7ac15b550 0x7fe7ac151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.904+0000 7fe7b1c00640 1 --2- 172.21.15.45:0/2245431889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7ac06f0d0 0x7fe7ac1f7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.904+0000 7fe7b1c00640 1 -- 172.21.15.45:0/2245431889 >> 172.21.15.45:0/2245431889 conn(0x7fe7ac067e00 msgr2=0x7fe7ac0cb020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.904+0000 7fe7b1c00640 1 -- 172.21.15.45:0/2245431889 shutdown_connections 2024-04-16T16:17:57.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:57.904+0000 7fe7b1c00640 1 -- 172.21.15.45:0/2245431889 wait complete. 2024-04-16T16:17:57.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:17:58.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.152+0000 7f4ffa600640 1 Processor -- start 2024-04-16T16:17:58.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.152+0000 7f4ffa600640 1 -- start start 2024-04-16T16:17:58.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.154+0000 7f4ffa600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4ff40d18f0 0x7f4ff40ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:58.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.154+0000 7f4ffa600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4ff40d1cd0 0x7f4ff4162dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:58.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.154+0000 7f4ffa600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ff40b6810 con 0x7f4ff40b6310 2024-04-16T16:17:58.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.154+0000 7f4ffa600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f4ff40d29f0 con 0x7f4ff40d1cd0 2024-04-16T16:17:58.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.154+0000 7f4ffa600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4ff40b6af0 con 0x7f4ff40d18f0 2024-04-16T16:17:58.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.154+0000 7f4ff3e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4ff40b6310 0x7f4ff40d2e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52294/0 (socket says 172.21.15.45:52294) 2024-04-16T16:17:58.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.154+0000 7f4ff3e00640 1 -- 172.21.15.45:0/1484415339 learned_addr learned my addr 172.21.15.45:0/1484415339 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:58.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.154+0000 7f4ff9c00640 1 --2- 172.21.15.45:0/1484415339 >> v2:172.21.15.45:3300/0 conn(0x7f4ff40d1cd0 0x7f4ff4162dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:58.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.155+0000 7f4ff9200640 1 --2- 172.21.15.45:0/1484415339 >> v2:172.21.15.103:3300/0 conn(0x7f4ff40d18f0 0x7f4ff40ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:58.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.155+0000 7f4ff3400640 1 -- 172.21.15.45:0/1484415339 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3243273095 0 0) 0x7f4ff40b6810 con 0x7f4ff40b6310 2024-04-16T16:17:58.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.155+0000 7f4ff3400640 1 -- 172.21.15.45:0/1484415339 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fdc003520 con 0x7f4ff40b6310 2024-04-16T16:17:58.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.155+0000 7f4ff9c00640 1 -- 172.21.15.45:0/1484415339 >> v2:172.21.15.103:3300/0 conn(0x7f4ff40d18f0 msgr2=0x7f4ff40ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:58.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.155+0000 7f4ff9c00640 1 --2- 172.21.15.45:0/1484415339 >> v2:172.21.15.103:3300/0 conn(0x7f4ff40d18f0 0x7f4ff40ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.155+0000 7f4ff9c00640 1 -- 172.21.15.45:0/1484415339 >> v1:172.21.15.45:6789/0 conn(0x7f4ff40b6310 legacy=0x7f4ff40d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:58.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.155+0000 7f4ff9c00640 1 -- 172.21.15.45:0/1484415339 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff40b6dd0 con 0x7f4ff40d1cd0 2024-04-16T16:17:58.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.156+0000 7f4ff9c00640 1 --2- 172.21.15.45:0/1484415339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ff40d1cd0 0x7f4ff4162dc0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4073f70 tx=0x7f4fec02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9a0e0997bab327 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:58.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.156+0000 7f4ff3400640 1 -- 172.21.15.45:0/1484415339 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec039070 con 0x7f4ff40d1cd0 2024-04-16T16:17:58.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.156+0000 7f4ff3400640 1 -- 172.21.15.45:0/1484415339 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fec02cb60 con 0x7f4ff40d1cd0 2024-04-16T16:17:58.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.157+0000 7f4ff3400640 1 -- 172.21.15.45:0/1484415339 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec034440 con 0x7f4ff40d1cd0 2024-04-16T16:17:58.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.157+0000 7f4ffa600640 1 -- 172.21.15.45:0/1484415339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ff40d1cd0 msgr2=0x7f4ff4162dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:58.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.157+0000 7f4ffa600640 1 --2- 172.21.15.45:0/1484415339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ff40d1cd0 0x7f4ff4162dc0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4073f70 tx=0x7f4fec02bdb0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.157+0000 7f4ffa600640 1 -- 172.21.15.45:0/1484415339 shutdown_connections 2024-04-16T16:17:58.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.157+0000 7f4ffa600640 1 --2- 172.21.15.45:0/1484415339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ff40d1cd0 0x7f4ff4162dc0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.157+0000 7f4ffa600640 1 --2- 172.21.15.45:0/1484415339 >> v2:172.21.15.103:3300/0 conn(0x7f4ff40d18f0 0x7f4ff40ba220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.157+0000 7f4ffa600640 1 -- 172.21.15.45:0/1484415339 >> 172.21.15.45:0/1484415339 conn(0x7f4ff406cf90 msgr2=0x7f4ff406d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:58.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.157+0000 7f4ffa600640 1 -- 172.21.15.45:0/1484415339 shutdown_connections 2024-04-16T16:17:58.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.157+0000 7f4ffa600640 1 -- 172.21.15.45:0/1484415339 wait complete. 2024-04-16T16:17:58.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.159+0000 7f4ffa600640 1 Processor -- start 2024-04-16T16:17:58.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.159+0000 7f4ffa600640 1 -- start start 2024-04-16T16:17:58.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.160+0000 7f4ffa600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ff40b6310 0x7f4ff415f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:58.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.160+0000 7f4ffa600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ff40d18f0 0x7f4ff415fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:58.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.160+0000 7f4ffa600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ff40d1cd0 0x7f4ff40cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:58.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.160+0000 7f4ffa600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff4163300 con 0x7f4ff40b6310 2024-04-16T16:17:58.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.160+0000 7f4ffa600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff40ba760 con 0x7f4ff40d18f0 2024-04-16T16:17:58.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.160+0000 7f4ffa600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff40cd100 con 0x7f4ff40d1cd0 2024-04-16T16:17:58.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.160+0000 7f4ff9200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ff40b6310 0x7f4ff415f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:58.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.161+0000 7f4ff9200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ff40b6310 0x7f4ff415f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57678/0 (socket says 172.21.15.45:57678) 2024-04-16T16:17:58.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.161+0000 7f4ff9200640 1 -- 172.21.15.45:0/1971110738 learned_addr learned my addr 172.21.15.45:0/1971110738 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:58.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.161+0000 7f4ff3e00640 1 --2- 172.21.15.45:0/1971110738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ff40d18f0 0x7f4ff415fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:58.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.161+0000 7f4ff9200640 1 -- 172.21.15.45:0/1971110738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ff40d1cd0 msgr2=0x7f4ff40cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:58.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.161+0000 7f4ff9200640 1 --2- 172.21.15.45:0/1971110738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ff40d1cd0 0x7f4ff40cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.161+0000 7f4ff9200640 1 -- 172.21.15.45:0/1971110738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ff40d18f0 msgr2=0x7f4ff415fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:58.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.161+0000 7f4ff9200640 1 --2- 172.21.15.45:0/1971110738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ff40d18f0 0x7f4ff415fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.161+0000 7f4ff9200640 1 -- 172.21.15.45:0/1971110738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff40c97b0 con 0x7f4ff40b6310 2024-04-16T16:17:58.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.161+0000 7f4ff9200640 1 --2- 172.21.15.45:0/1971110738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ff40b6310 0x7f4ff415f610 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800d800 tx=0x7f4fe800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:58.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.162+0000 7f4ff0c00640 1 -- 172.21.15.45:0/1971110738 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe800b660 con 0x7f4ff40b6310 2024-04-16T16:17:58.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.162+0000 7f4ff0c00640 1 -- 172.21.15.45:0/1971110738 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fe8019070 con 0x7f4ff40b6310 2024-04-16T16:17:58.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.162+0000 7f4ff0c00640 1 -- 172.21.15.45:0/1971110738 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe800ba20 con 0x7f4ff40b6310 2024-04-16T16:17:58.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.162+0000 7f4ffa600640 1 -- 172.21.15.45:0/1971110738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ff40c9a90 con 0x7f4ff40b6310 2024-04-16T16:17:58.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.162+0000 7f4ffa600640 1 -- 172.21.15.45:0/1971110738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ff40cc9f0 con 0x7f4ff40b6310 2024-04-16T16:17:58.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.163+0000 7f4ffa600640 1 -- 172.21.15.45:0/1971110738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ff4075780 con 0x7f4ff40b6310 2024-04-16T16:17:58.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.164+0000 7f4ff0c00640 1 -- 172.21.15.45:0/1971110738 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 24) ==== 91086+0+0 (secure 0 0 0) 0x7f4fe800bbc0 con 0x7f4ff40b6310 2024-04-16T16:17:58.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.164+0000 7f4ff0c00640 1 --2- 172.21.15.45:0/1971110738 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f4fd406c980 0x7f4fd406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:58.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.165+0000 7f4ff0c00640 1 -- 172.21.15.45:0/1971110738 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6483+0+0 (secure 0 0 0) 0x7f4fe808c380 con 0x7f4ff40b6310 2024-04-16T16:17:58.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.165+0000 7f4ff3e00640 1 --2- 172.21.15.45:0/1971110738 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f4fd406c980 0x7f4fd406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:58.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.166+0000 7f4ff3e00640 1 --2- 172.21.15.45:0/1971110738 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f4fd406c980 0x7f4fd406ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40d3b20 tx=0x7f4fe0009040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:58.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.169+0000 7f4ff0c00640 1 -- 172.21.15.45:0/1971110738 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4fe801c360 con 0x7f4ff40b6310 2024-04-16T16:17:58.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.369+0000 7f4ffa600640 1 -- 172.21.15.45:0/1971110738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4ff40d25a0 con 0x7f4ff40b6310 2024-04-16T16:17:58.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.370+0000 7f4ff0c00640 1 -- 172.21.15.45:0/1971110738 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v74) ==== 104+0+11 (secure 0 0 0) 0x7f4fe805a440 con 0x7f4ff40b6310 2024-04-16T16:17:58.370 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:17:58.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.372+0000 7f4ffa600640 1 -- 172.21.15.45:0/1971110738 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f4fd406c980 msgr2=0x7f4fd406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:58.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.372+0000 7f4ffa600640 1 --2- 172.21.15.45:0/1971110738 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f4fd406c980 0x7f4fd406ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40d3b20 tx=0x7f4fe0009040 comp rx=0 tx=0).stop 2024-04-16T16:17:58.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.372+0000 7f4ffa600640 1 -- 172.21.15.45:0/1971110738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ff40b6310 msgr2=0x7f4ff415f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:58.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.372+0000 7f4ffa600640 1 --2- 172.21.15.45:0/1971110738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ff40b6310 0x7f4ff415f610 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800d800 tx=0x7f4fe800dcc0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.373+0000 7f4ffa600640 1 -- 172.21.15.45:0/1971110738 shutdown_connections 2024-04-16T16:17:58.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.373+0000 7f4ffa600640 1 --2- 172.21.15.45:0/1971110738 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f4fd406c980 0x7f4fd406ee40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.373+0000 7f4ffa600640 1 --2- 172.21.15.45:0/1971110738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ff40d1cd0 0x7f4ff40cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.373+0000 7f4ffa600640 1 --2- 172.21.15.45:0/1971110738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ff40d18f0 0x7f4ff415fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.373+0000 7f4ffa600640 1 --2- 172.21.15.45:0/1971110738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ff40b6310 0x7f4ff415f610 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.373+0000 7f4ffa600640 1 -- 172.21.15.45:0/1971110738 >> 172.21.15.45:0/1971110738 conn(0x7f4ff406cf90 msgr2=0x7f4ff41612e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:58.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.373+0000 7f4ffa600640 1 -- 172.21.15.45:0/1971110738 shutdown_connections 2024-04-16T16:17:58.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.373+0000 7f4ffa600640 1 -- 172.21.15.45:0/1971110738 wait complete. 2024-04-16T16:17:58.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:17:58.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.619+0000 7f528a000640 1 Processor -- start 2024-04-16T16:17:58.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.620+0000 7f528a000640 1 -- start start 2024-04-16T16:17:58.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.621+0000 7f528a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52841461b0 0x7f5284160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:58.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.621+0000 7f528a000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f52840b58f0 0x7f52840b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:58.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.621+0000 7f528a000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5284146590 con 0x7f52840cf440 2024-04-16T16:17:58.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.621+0000 7f528a000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f528406dfd0 con 0x7f52840b58f0 2024-04-16T16:17:58.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.621+0000 7f528a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f52840b6290 con 0x7f52841461b0 2024-04-16T16:17:58.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.622+0000 7f5289600640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f52840cf440 0x7f52840cf8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52304/0 (socket says 172.21.15.45:52304) 2024-04-16T16:17:58.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.622+0000 7f5289600640 1 -- 172.21.15.45:0/856149648 learned_addr learned my addr 172.21.15.45:0/856149648 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:17:58.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.622+0000 7f5288c00640 1 --2- 172.21.15.45:0/856149648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52841461b0 0x7f5284160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:58.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.622+0000 7f5283400640 1 -- 172.21.15.45:0/856149648 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 130394303 0 0) 0x7f5284146590 con 0x7f52840cf440 2024-04-16T16:17:58.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.622+0000 7f5283400640 1 -- 172.21.15.45:0/856149648 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f526c003520 con 0x7f52840cf440 2024-04-16T16:17:58.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.622+0000 7f5283e00640 1 --2- 172.21.15.45:0/856149648 >> v2:172.21.15.103:3300/0 conn(0x7f52840b58f0 0x7f52840b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:58.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.623+0000 7f5283400640 1 -- 172.21.15.45:0/856149648 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2507893827 0 0) 0x7f526c003520 con 0x7f52840cf440 2024-04-16T16:17:58.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.623+0000 7f5283400640 1 -- 172.21.15.45:0/856149648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52841461b0 msgr2=0x7f5284160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:58.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.623+0000 7f5283400640 1 --2- 172.21.15.45:0/856149648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52841461b0 0x7f5284160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.623+0000 7f5283400640 1 -- 172.21.15.45:0/856149648 >> v2:172.21.15.103:3300/0 conn(0x7f52840b58f0 msgr2=0x7f52840b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:58.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.623+0000 7f5283400640 1 --2- 172.21.15.45:0/856149648 >> v2:172.21.15.103:3300/0 conn(0x7f52840b58f0 0x7f52840b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.623+0000 7f5283400640 1 -- 172.21.15.45:0/856149648 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52840c5750 con 0x7f52840cf440 2024-04-16T16:17:58.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.623+0000 7f5283400640 1 -- 172.21.15.45:0/856149648 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f527c0019f0 con 0x7f52840cf440 2024-04-16T16:17:58.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.623+0000 7f5283400640 1 -- 172.21.15.45:0/856149648 >> v1:172.21.15.45:6789/0 conn(0x7f52840cf440 legacy=0x7f52840cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:58.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.623+0000 7f5283400640 1 --2- 172.21.15.45:0/856149648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52841461b0 0x7f526c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:58.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.623+0000 7f5283400640 1 --2- 172.21.15.45:0/856149648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f526c003f80 0x7f526c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:58.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.623+0000 7f5283400640 1 --2- 172.21.15.45:0/856149648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526c004d40 0x7f526c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:58.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.623+0000 7f5283400640 1 -- 172.21.15.45:0/856149648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f526c00b9c0 con 0x7f526c004d40 2024-04-16T16:17:58.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.623+0000 7f5283400640 1 -- 172.21.15.45:0/856149648 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f526c00bb70 con 0x7f526c003f80 2024-04-16T16:17:58.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.623+0000 7f5283400640 1 -- 172.21.15.45:0/856149648 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f526c00bd70 con 0x7f52841461b0 2024-04-16T16:17:58.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.623+0000 7f5289600640 1 --2- 172.21.15.45:0/856149648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526c004d40 0x7f526c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:58.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.624+0000 7f5283e00640 1 --2- 172.21.15.45:0/856149648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f526c003f80 0x7f526c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:58.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.624+0000 7f5289600640 1 -- 172.21.15.45:0/856149648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52841461b0 msgr2=0x7f526c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:58.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.624+0000 7f5289600640 1 --2- 172.21.15.45:0/856149648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52841461b0 0x7f526c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.624+0000 7f5289600640 1 -- 172.21.15.45:0/856149648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f526c003f80 msgr2=0x7f526c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:58.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.624+0000 7f5289600640 1 --2- 172.21.15.45:0/856149648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f526c003f80 0x7f526c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.624+0000 7f5289600640 1 -- 172.21.15.45:0/856149648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526c003520 con 0x7f526c004d40 2024-04-16T16:17:58.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.624+0000 7f5283e00640 1 --2- 172.21.15.45:0/856149648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f526c003f80 0x7f526c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:58.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.625+0000 7f5289600640 1 --2- 172.21.15.45:0/856149648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526c004d40 0x7f526c0051c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f527c00a7a0 tx=0x7f526c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b169caa7eb4b4fbc server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:58.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.625+0000 7f5283400640 1 -- 172.21.15.45:0/856149648 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c00a3f0 con 0x7f526c004d40 2024-04-16T16:17:58.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.625+0000 7f5283400640 1 -- 172.21.15.45:0/856149648 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f527c034630 con 0x7f526c004d40 2024-04-16T16:17:58.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.625+0000 7f5283400640 1 -- 172.21.15.45:0/856149648 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c034a80 con 0x7f526c004d40 2024-04-16T16:17:58.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.626+0000 7f528a000640 1 -- 172.21.15.45:0/856149648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526c004d40 msgr2=0x7f526c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:58.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.626+0000 7f528a000640 1 --2- 172.21.15.45:0/856149648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526c004d40 0x7f526c0051c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f527c00a7a0 tx=0x7f526c005830 comp rx=0 tx=0).stop 2024-04-16T16:17:58.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.626+0000 7f5289600640 1 -- 172.21.15.45:0/856149648 reap_dead start 2024-04-16T16:17:58.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.626+0000 7f528a000640 1 -- 172.21.15.45:0/856149648 shutdown_connections 2024-04-16T16:17:58.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.626+0000 7f528a000640 1 -- 172.21.15.45:0/856149648 >> 172.21.15.45:0/856149648 conn(0x7f528406b930 msgr2=0x7f528406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:58.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.626+0000 7f528a000640 1 -- 172.21.15.45:0/856149648 shutdown_connections 2024-04-16T16:17:58.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.626+0000 7f528a000640 1 -- 172.21.15.45:0/856149648 wait complete. 2024-04-16T16:17:58.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.628+0000 7f528a000640 1 Processor -- start 2024-04-16T16:17:58.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.628+0000 7f528a000640 1 -- start start 2024-04-16T16:17:58.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.629+0000 7f528a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526c004d40 0x7f52841442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:58.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.629+0000 7f528a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f528413c010 0x7f528414cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:58.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.629+0000 7f528a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f528413c470 0x7f528414d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:58.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.629+0000 7f528a000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f52840c5cd0 con 0x7f526c004d40 2024-04-16T16:17:58.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.629+0000 7f528a000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f52840c59f0 con 0x7f528413c010 2024-04-16T16:17:58.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.629+0000 7f528a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f52840c5fb0 con 0x7f528413c470 2024-04-16T16:17:58.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.630+0000 7f5288c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526c004d40 0x7f52841442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:58.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.630+0000 7f5288c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526c004d40 0x7f52841442b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57700/0 (socket says 172.21.15.45:57700) 2024-04-16T16:17:58.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.630+0000 7f5288c00640 1 -- 172.21.15.45:0/2341986299 learned_addr learned my addr 172.21.15.45:0/2341986299 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:58.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.630+0000 7f5289600640 1 --2- 172.21.15.45:0/2341986299 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f528413c470 0x7f528414d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:58.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.630+0000 7f5283e00640 1 --2- 172.21.15.45:0/2341986299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f528413c010 0x7f528414cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:58.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.630+0000 7f5288c00640 1 -- 172.21.15.45:0/2341986299 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f528413c470 msgr2=0x7f528414d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:58.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.630+0000 7f5288c00640 1 --2- 172.21.15.45:0/2341986299 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f528413c470 0x7f528414d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.630+0000 7f5288c00640 1 -- 172.21.15.45:0/2341986299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f528413c010 msgr2=0x7f528414cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:58.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.630+0000 7f5288c00640 1 --2- 172.21.15.45:0/2341986299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f528413c010 0x7f528414cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.630+0000 7f5288c00640 1 -- 172.21.15.45:0/2341986299 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5284143210 con 0x7f526c004d40 2024-04-16T16:17:58.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.630+0000 7f5283e00640 1 --2- 172.21.15.45:0/2341986299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f528413c010 0x7f528414cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:58.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.631+0000 7f5288c00640 1 --2- 172.21.15.45:0/2341986299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526c004d40 0x7f52841442b0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f527800e7f0 tx=0x7f527800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:58.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.631+0000 7f5280c00640 1 -- 172.21.15.45:0/2341986299 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5278017070 con 0x7f526c004d40 2024-04-16T16:17:58.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.631+0000 7f5280c00640 1 -- 172.21.15.45:0/2341986299 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f527800c940 con 0x7f526c004d40 2024-04-16T16:17:58.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.631+0000 7f5280c00640 1 -- 172.21.15.45:0/2341986299 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527800eea0 con 0x7f526c004d40 2024-04-16T16:17:58.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.631+0000 7f528a000640 1 -- 172.21.15.45:0/2341986299 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52841434f0 con 0x7f526c004d40 2024-04-16T16:17:58.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.631+0000 7f528a000640 1 -- 172.21.15.45:0/2341986299 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5284143ab0 con 0x7f526c004d40 2024-04-16T16:17:58.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.632+0000 7f528a000640 1 -- 172.21.15.45:0/2341986299 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5284075780 con 0x7f526c004d40 2024-04-16T16:17:58.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.633+0000 7f5280c00640 1 -- 172.21.15.45:0/2341986299 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 24) ==== 91086+0+0 (secure 0 0 0) 0x7f527801a070 con 0x7f526c004d40 2024-04-16T16:17:58.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.634+0000 7f5280c00640 1 --2- 172.21.15.45:0/2341986299 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f526406c980 0x7f526406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:58.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.635+0000 7f5280c00640 1 -- 172.21.15.45:0/2341986299 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6483+0+0 (secure 0 0 0) 0x7f527808c1a0 con 0x7f526c004d40 2024-04-16T16:17:58.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.635+0000 7f5283e00640 1 --2- 172.21.15.45:0/2341986299 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f526406c980 0x7f526406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:58.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.636+0000 7f5283e00640 1 --2- 172.21.15.45:0/2341986299 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f526406c980 0x7f526406ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f52840b1850 tx=0x7f5270008040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:58.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.641+0000 7f5280c00640 1 -- 172.21.15.45:0/2341986299 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f527805a1d0 con 0x7f526c004d40 2024-04-16T16:17:58.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.844+0000 7f528a000640 1 -- 172.21.15.45:0/2341986299 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5284072500 con 0x7f526c004d40 2024-04-16T16:17:58.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.845+0000 7f5280c00640 1 -- 172.21.15.45:0/2341986299 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v74) ==== 100+0+11 (secure 0 0 0) 0x7f5278059ad0 con 0x7f526c004d40 2024-04-16T16:17:58.846 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:17:58.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.848+0000 7f528a000640 1 -- 172.21.15.45:0/2341986299 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f526406c980 msgr2=0x7f526406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:58.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.848+0000 7f528a000640 1 --2- 172.21.15.45:0/2341986299 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f526406c980 0x7f526406ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f52840b1850 tx=0x7f5270008040 comp rx=0 tx=0).stop 2024-04-16T16:17:58.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.848+0000 7f528a000640 1 -- 172.21.15.45:0/2341986299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526c004d40 msgr2=0x7f52841442b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:58.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.848+0000 7f528a000640 1 --2- 172.21.15.45:0/2341986299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526c004d40 0x7f52841442b0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f527800e7f0 tx=0x7f527800ecb0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.848+0000 7f528a000640 1 -- 172.21.15.45:0/2341986299 shutdown_connections 2024-04-16T16:17:58.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.848+0000 7f528a000640 1 --2- 172.21.15.45:0/2341986299 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f526406c980 0x7f526406ee40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.848+0000 7f528a000640 1 --2- 172.21.15.45:0/2341986299 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f528413c470 0x7f528414d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.848+0000 7f528a000640 1 --2- 172.21.15.45:0/2341986299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f528413c010 0x7f528414cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.848+0000 7f528a000640 1 --2- 172.21.15.45:0/2341986299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526c004d40 0x7f52841442b0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:58.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.848+0000 7f528a000640 1 -- 172.21.15.45:0/2341986299 >> 172.21.15.45:0/2341986299 conn(0x7f528406b930 msgr2=0x7f5284072c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:58.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.848+0000 7f528a000640 1 -- 172.21.15.45:0/2341986299 shutdown_connections 2024-04-16T16:17:58.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:58.849+0000 7f528a000640 1 -- 172.21.15.45:0/2341986299 wait complete. 2024-04-16T16:17:58.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:17:59.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.097+0000 7fef4fa00640 1 Processor -- start 2024-04-16T16:17:59.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.097+0000 7fef4fa00640 1 -- start start 2024-04-16T16:17:59.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.098+0000 7fef4fa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fef4815b730 0x7fef480cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:59.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.098+0000 7fef4fa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fef480b5840 0x7fef480cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:59.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.098+0000 7fef4fa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef480d0060 0x7fef4815f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.098+0000 7fef4fa00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fef480ce5b0 con 0x7fef4815b730 2024-04-16T16:17:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.098+0000 7fef4fa00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fef48072420 con 0x7fef480b5840 2024-04-16T16:17:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.098+0000 7fef4fa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fef48072700 con 0x7fef480d0060 2024-04-16T16:17:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.099+0000 7fef4d000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fef4815b730 0x7fef480cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.099+0000 7fef4d000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fef4815b730 0x7fef480cdeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57714/0 (socket says 172.21.15.45:57714) 2024-04-16T16:17:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.099+0000 7fef4d000640 1 -- 172.21.15.45:0/384606946 learned_addr learned my addr 172.21.15.45:0/384606946 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.099+0000 7fef47e00640 1 --2- 172.21.15.45:0/384606946 >> v2:172.21.15.103:3300/0 conn(0x7fef480b5840 0x7fef480cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.099+0000 7fef4d000640 1 -- 172.21.15.45:0/384606946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef480d0060 msgr2=0x7fef4815f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.099+0000 7fef4d000640 1 --2- 172.21.15.45:0/384606946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef480d0060 0x7fef4815f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.099+0000 7fef4d000640 1 -- 172.21.15.45:0/384606946 >> v2:172.21.15.103:3300/0 conn(0x7fef480b5840 msgr2=0x7fef480cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.099+0000 7fef4d000640 1 --2- 172.21.15.45:0/384606946 >> v2:172.21.15.103:3300/0 conn(0x7fef480b5840 0x7fef480cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.099+0000 7fef4d000640 1 -- 172.21.15.45:0/384606946 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef480729e0 con 0x7fef4815b730 2024-04-16T16:17:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.100+0000 7fef4d000640 1 --2- 172.21.15.45:0/384606946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef4815b730 0x7fef480cdeb0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fef48070410 tx=0x7fef3002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a4a867c1c902b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.101+0000 7fef47400640 1 -- 172.21.15.45:0/384606946 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef30035020 con 0x7fef4815b730 2024-04-16T16:17:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.101+0000 7fef47400640 1 -- 172.21.15.45:0/384606946 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef30031870 con 0x7fef4815b730 2024-04-16T16:17:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.101+0000 7fef47400640 1 -- 172.21.15.45:0/384606946 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef30031c40 con 0x7fef4815b730 2024-04-16T16:17:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.101+0000 7fef4fa00640 1 -- 172.21.15.45:0/384606946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef4815b730 msgr2=0x7fef480cdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.101+0000 7fef4fa00640 1 --2- 172.21.15.45:0/384606946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef4815b730 0x7fef480cdeb0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fef48070410 tx=0x7fef3002b910 comp rx=0 tx=0).stop 2024-04-16T16:17:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.101+0000 7fef4fa00640 1 -- 172.21.15.45:0/384606946 shutdown_connections 2024-04-16T16:17:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.101+0000 7fef4fa00640 1 --2- 172.21.15.45:0/384606946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef480d0060 0x7fef4815f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.101+0000 7fef4fa00640 1 --2- 172.21.15.45:0/384606946 >> v2:172.21.15.103:3300/0 conn(0x7fef480b5840 0x7fef480cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.101+0000 7fef4fa00640 1 --2- 172.21.15.45:0/384606946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef4815b730 0x7fef480cdeb0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.101+0000 7fef4fa00640 1 -- 172.21.15.45:0/384606946 >> 172.21.15.45:0/384606946 conn(0x7fef480b2750 msgr2=0x7fef480b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.101+0000 7fef4fa00640 1 -- 172.21.15.45:0/384606946 shutdown_connections 2024-04-16T16:17:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.102+0000 7fef4fa00640 1 -- 172.21.15.45:0/384606946 wait complete. 2024-04-16T16:17:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.103+0000 7fef4fa00640 1 Processor -- start 2024-04-16T16:17:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.103+0000 7fef4fa00640 1 -- start start 2024-04-16T16:17:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.104+0000 7fef4fa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef480b5840 0x7fef4815a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.104+0000 7fef4fa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef480d0060 0x7fef4814c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.104+0000 7fef4fa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef4815b730 0x7fef4815d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.104+0000 7fef4fa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fef480b5ca0 con 0x7fef480d0060 2024-04-16T16:17:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.104+0000 7fef4fa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fef480d04f0 con 0x7fef4815b730 2024-04-16T16:17:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.104+0000 7fef4fa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fef4815bb10 con 0x7fef480b5840 2024-04-16T16:17:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.105+0000 7fef47e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef480d0060 0x7fef4814c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.105+0000 7fef47e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef480d0060 0x7fef4814c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57730/0 (socket says 172.21.15.45:57730) 2024-04-16T16:17:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.105+0000 7fef47e00640 1 -- 172.21.15.45:0/2824040243 learned_addr learned my addr 172.21.15.45:0/2824040243 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.105+0000 7fef4da00640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef4815b730 0x7fef4815d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.105+0000 7fef47e00640 1 -- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef480b5840 msgr2=0x7fef4815a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.105+0000 7fef4d000640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef480b5840 0x7fef4815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.105+0000 7fef47e00640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef480b5840 0x7fef4815a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.105+0000 7fef47e00640 1 -- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef4815b730 msgr2=0x7fef4815d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.105+0000 7fef47e00640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef4815b730 0x7fef4815d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.105+0000 7fef47e00640 1 -- 172.21.15.45:0/2824040243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4815bd80 con 0x7fef480d0060 2024-04-16T16:17:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.105+0000 7fef4da00640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef4815b730 0x7fef4815d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.105+0000 7fef4d000640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef480b5840 0x7fef4815a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.105+0000 7fef47e00640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef480d0060 0x7fef4814c5f0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fef3800c820 tx=0x7fef3800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.106+0000 7fef44c00640 1 -- 172.21.15.45:0/2824040243 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef38015070 con 0x7fef480d0060 2024-04-16T16:17:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.106+0000 7fef44c00640 1 -- 172.21.15.45:0/2824040243 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef38004020 con 0x7fef480d0060 2024-04-16T16:17:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.106+0000 7fef44c00640 1 -- 172.21.15.45:0/2824040243 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef380043f0 con 0x7fef480d0060 2024-04-16T16:17:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.106+0000 7fef4fa00640 1 -- 172.21.15.45:0/2824040243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef4815c020 con 0x7fef480d0060 2024-04-16T16:17:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.106+0000 7fef4fa00640 1 -- 172.21.15.45:0/2824040243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef4815c5e0 con 0x7fef480d0060 2024-04-16T16:17:59.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.107+0000 7fef4fa00640 1 -- 172.21.15.45:0/2824040243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef48068300 con 0x7fef480d0060 2024-04-16T16:17:59.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.108+0000 7fef44c00640 1 -- 172.21.15.45:0/2824040243 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 24) ==== 91086+0+0 (secure 0 0 0) 0x7fef38007830 con 0x7fef480d0060 2024-04-16T16:17:59.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.109+0000 7fef44c00640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fef2406c980 0x7fef2406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:59.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.110+0000 7fef44c00640 1 -- 172.21.15.45:0/2824040243 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6483+0+0 (secure 0 0 0) 0x7fef3808bde0 con 0x7fef480d0060 2024-04-16T16:17:59.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.110+0000 7fef4d000640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fef2406c980 0x7fef2406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:59.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.111+0000 7fef4d000640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fef2406c980 0x7fef2406ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fef30036000 tx=0x7fef30008030 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:59.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.114+0000 7fef44c00640 1 -- 172.21.15.45:0/2824040243 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fef48068300 con 0x7fef480d0060 2024-04-16T16:17:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.133+0000 7fef44c00640 1 -- 172.21.15.45:0/2824040243 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7fef38055d00 con 0x7fef480d0060 2024-04-16T16:17:59.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.362+0000 7fef4fa00640 1 -- 172.21.15.45:0/2824040243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef48075780 con 0x7fef480d0060 2024-04-16T16:17:59.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.364+0000 7fef44c00640 1 -- 172.21.15.45:0/2824040243 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5202 (secure 0 0 0) 0x7fef38059d90 con 0x7fef480d0060 2024-04-16T16:17:59.364 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:59.364 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":55,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:17:53.689159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6402,"mds_1":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"c","rank":1,"incarnation":52,"state":"up:active","state_seq":11,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6399,6402]},"id":5}]} 2024-04-16T16:17:59.364 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 55 2024-04-16T16:17:59.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.366+0000 7fef4fa00640 1 -- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fef2406c980 msgr2=0x7fef2406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:59.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.366+0000 7fef4fa00640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fef2406c980 0x7fef2406ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fef30036000 tx=0x7fef30008030 comp rx=0 tx=0).stop 2024-04-16T16:17:59.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.366+0000 7fef4fa00640 1 -- 172.21.15.45:0/2824040243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef480d0060 msgr2=0x7fef4814c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:59.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.366+0000 7fef4fa00640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef480d0060 0x7fef4814c5f0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fef3800c820 tx=0x7fef3800cce0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.366+0000 7fef4fa00640 1 -- 172.21.15.45:0/2824040243 shutdown_connections 2024-04-16T16:17:59.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.366+0000 7fef4fa00640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fef2406c980 0x7fef2406ee40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.366+0000 7fef4fa00640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef4815b730 0x7fef4815d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.366+0000 7fef4fa00640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef480d0060 0x7fef4814c5f0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.366+0000 7fef4fa00640 1 --2- 172.21.15.45:0/2824040243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef480b5840 0x7fef4815a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.366+0000 7fef4fa00640 1 -- 172.21.15.45:0/2824040243 >> 172.21.15.45:0/2824040243 conn(0x7fef480b2750 msgr2=0x7fef480b0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:59.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.366+0000 7fef4fa00640 1 -- 172.21.15.45:0/2824040243 shutdown_connections 2024-04-16T16:17:59.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.366+0000 7fef4fa00640 1 -- 172.21.15.45:0/2824040243 wait complete. 2024-04-16T16:17:59.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:17:59.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.592+0000 7fad72200640 1 Processor -- start 2024-04-16T16:17:59.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.592+0000 7fad72200640 1 -- start start 2024-04-16T16:17:59.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.594+0000 7fad72200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fad6c146220 0x7fad6c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:59.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.594+0000 7fad72200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad6c06f7a0 0x7fad6c06df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:59.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.594+0000 7fad72200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad6c15d360 con 0x7fad6c06e480 2024-04-16T16:17:59.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.594+0000 7fad72200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fad6c160410 con 0x7fad6c146220 2024-04-16T16:17:59.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.594+0000 7fad72200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fad6c15d640 con 0x7fad6c06f7a0 2024-04-16T16:17:59.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.594+0000 7fad70e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fad6c146220 0x7fad6c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:59.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.594+0000 7fad70e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fad6c146220 0x7fad6c15ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57746/0 (socket says 172.21.15.45:57746) 2024-04-16T16:17:59.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.594+0000 7fad70e00640 1 -- 172.21.15.45:0/1172719495 learned_addr learned my addr 172.21.15.45:0/1172719495 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:59.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.595+0000 7fad6be00640 1 --2- 172.21.15.45:0/1172719495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad6c06f7a0 0x7fad6c06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:59.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.595+0000 7fad70e00640 1 -- 172.21.15.45:0/1172719495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad6c06f7a0 msgr2=0x7fad6c06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:59.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.595+0000 7fad70e00640 1 --2- 172.21.15.45:0/1172719495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad6c06f7a0 0x7fad6c06df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.595+0000 7fad70e00640 1 -- 172.21.15.45:0/1172719495 >> v1:172.21.15.103:6789/0 conn(0x7fad6c06e480 legacy=0x7fad6c1608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:17:59.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.595+0000 7fad70e00640 1 -- 172.21.15.45:0/1172719495 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6c15d920 con 0x7fad6c146220 2024-04-16T16:17:59.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.596+0000 7fad70e00640 1 --2- 172.21.15.45:0/1172719495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6c146220 0x7fad6c15ccb0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fad60009870 tx=0x7fad6002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d32fe678929651 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:59.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.597+0000 7fad6b400640 1 -- 172.21.15.45:0/1172719495 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60035020 con 0x7fad6c146220 2024-04-16T16:17:59.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.597+0000 7fad6b400640 1 -- 172.21.15.45:0/1172719495 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad60031640 con 0x7fad6c146220 2024-04-16T16:17:59.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.597+0000 7fad6b400640 1 -- 172.21.15.45:0/1172719495 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6002d040 con 0x7fad6c146220 2024-04-16T16:17:59.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.597+0000 7fad72200640 1 -- 172.21.15.45:0/1172719495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6c146220 msgr2=0x7fad6c15ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:59.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.597+0000 7fad72200640 1 --2- 172.21.15.45:0/1172719495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6c146220 0x7fad6c15ccb0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fad60009870 tx=0x7fad6002b8f0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.597+0000 7fad72200640 1 -- 172.21.15.45:0/1172719495 shutdown_connections 2024-04-16T16:17:59.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.597+0000 7fad72200640 1 --2- 172.21.15.45:0/1172719495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad6c06f7a0 0x7fad6c06df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.597+0000 7fad72200640 1 --2- 172.21.15.45:0/1172719495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6c146220 0x7fad6c15ccb0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.597+0000 7fad72200640 1 -- 172.21.15.45:0/1172719495 >> 172.21.15.45:0/1172719495 conn(0x7fad6c06b930 msgr2=0x7fad6c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:59.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.598+0000 7fad72200640 1 -- 172.21.15.45:0/1172719495 shutdown_connections 2024-04-16T16:17:59.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.598+0000 7fad72200640 1 -- 172.21.15.45:0/1172719495 wait complete. 2024-04-16T16:17:59.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.599+0000 7fad72200640 1 Processor -- start 2024-04-16T16:17:59.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.599+0000 7fad72200640 1 -- start start 2024-04-16T16:17:59.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.600+0000 7fad72200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad6c06e480 0x7fad6c145470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:59.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.600+0000 7fad72200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6c06f7a0 0x7fad6c136c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:59.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.600+0000 7fad72200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c146220 0x7fad6c0ca920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:59.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.600+0000 7fad72200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c06fda0 con 0x7fad6c06f7a0 2024-04-16T16:17:59.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.600+0000 7fad72200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c06fc20 con 0x7fad6c146220 2024-04-16T16:17:59.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.600+0000 7fad72200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fad6c160fb0 con 0x7fad6c06e480 2024-04-16T16:17:59.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.601+0000 7fad6be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6c06f7a0 0x7fad6c136c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:59.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.601+0000 7fad6be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6c06f7a0 0x7fad6c136c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57752/0 (socket says 172.21.15.45:57752) 2024-04-16T16:17:59.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.601+0000 7fad6be00640 1 -- 172.21.15.45:0/757333766 learned_addr learned my addr 172.21.15.45:0/757333766 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:17:59.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.601+0000 7fad71800640 1 --2- 172.21.15.45:0/757333766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c146220 0x7fad6c0ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.601+0000 7fad6be00640 1 -- 172.21.15.45:0/757333766 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad6c06e480 msgr2=0x7fad6c145470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.601+0000 7fad6be00640 1 --2- 172.21.15.45:0/757333766 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad6c06e480 0x7fad6c145470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.601+0000 7fad6be00640 1 -- 172.21.15.45:0/757333766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c146220 msgr2=0x7fad6c0ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.601+0000 7fad6be00640 1 --2- 172.21.15.45:0/757333766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c146220 0x7fad6c0ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.601+0000 7fad6be00640 1 -- 172.21.15.45:0/757333766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6c0c9950 con 0x7fad6c06f7a0 2024-04-16T16:17:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.601+0000 7fad71800640 1 --2- 172.21.15.45:0/757333766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c146220 0x7fad6c0ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:17:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.602+0000 7fad6be00640 1 --2- 172.21.15.45:0/757333766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6c06f7a0 0x7fad6c136c80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fad5800d800 tx=0x7fad5800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.602+0000 7fad68c00640 1 -- 172.21.15.45:0/757333766 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5800b6c0 con 0x7fad6c06f7a0 2024-04-16T16:17:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.602+0000 7fad68c00640 1 -- 172.21.15.45:0/757333766 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad58019070 con 0x7fad6c06f7a0 2024-04-16T16:17:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.602+0000 7fad68c00640 1 -- 172.21.15.45:0/757333766 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5800ba80 con 0x7fad6c06f7a0 2024-04-16T16:17:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.602+0000 7fad72200640 1 -- 172.21.15.45:0/757333766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad6c0c9c30 con 0x7fad6c06f7a0 2024-04-16T16:17:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.602+0000 7fad72200640 1 -- 172.21.15.45:0/757333766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad6c1459b0 con 0x7fad6c06f7a0 2024-04-16T16:17:59.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.603+0000 7fad72200640 1 -- 172.21.15.45:0/757333766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad6c0ca150 con 0x7fad6c06f7a0 2024-04-16T16:17:59.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.604+0000 7fad68c00640 1 -- 172.21.15.45:0/757333766 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7fad5800bc20 con 0x7fad6c06f7a0 2024-04-16T16:17:59.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.605+0000 7fad68c00640 1 --2- 172.21.15.45:0/757333766 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fad4c06ca10 0x7fad4c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:17:59.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.605+0000 7fad68c00640 1 -- 172.21.15.45:0/757333766 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6483+0+0 (secure 0 0 0) 0x7fad5808c580 con 0x7fad6c06f7a0 2024-04-16T16:17:59.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.606+0000 7fad70e00640 1 --2- 172.21.15.45:0/757333766 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fad4c06ca10 0x7fad4c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:17:59.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.607+0000 7fad70e00640 1 --2- 172.21.15.45:0/757333766 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fad4c06ca10 0x7fad4c06eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fad60036000 tx=0x7fad60030000 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:17:59.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.610+0000 7fad68c00640 1 -- 172.21.15.45:0/757333766 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fad6c0ca150 con 0x7fad6c06f7a0 2024-04-16T16:17:59.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.815+0000 7fad72200640 1 -- 172.21.15.45:0/757333766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad6c1427f0 con 0x7fad6c06f7a0 2024-04-16T16:17:59.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.817+0000 7fad68c00640 1 -- 172.21.15.45:0/757333766 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+17807 (secure 0 0 0) 0x7fad6c1427f0 con 0x7fad6c06f7a0 2024-04-16T16:17:59.818 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:17:59.818 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":74,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:17:55.075736+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:16:37.449973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-16T16:16:38.485027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/1692905448":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/1736628710":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/1783855160":"2024-04-17T16:17:55.075639+0000","172.21.15.103:6833/541757982":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/2044864797":"2024-04-17T16:17:55.075639+0000","172.21.15.103:6833/4155603915":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/951047877":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/1131547374":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/3176524506":"2024-04-17T16:17:01.408806+0000","172.21.15.103:6832/4155603915":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/365521086":"2024-04-17T16:17:01.408806+0000","172.21.15.103:6832/541757982":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/3345679536":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/2747644735":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/3606536347":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/1223877773":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/252084277":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/650078727":"2024-04-17T16:17:01.408806+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:17:59.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.820+0000 7fad72200640 1 -- 172.21.15.45:0/757333766 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fad4c06ca10 msgr2=0x7fad4c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:59.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.820+0000 7fad72200640 1 --2- 172.21.15.45:0/757333766 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fad4c06ca10 0x7fad4c06eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fad60036000 tx=0x7fad60030000 comp rx=0 tx=0).stop 2024-04-16T16:17:59.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.820+0000 7fad72200640 1 -- 172.21.15.45:0/757333766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6c06f7a0 msgr2=0x7fad6c136c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:17:59.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.820+0000 7fad72200640 1 --2- 172.21.15.45:0/757333766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6c06f7a0 0x7fad6c136c80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fad5800d800 tx=0x7fad5800dcc0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.820+0000 7fad72200640 1 -- 172.21.15.45:0/757333766 shutdown_connections 2024-04-16T16:17:59.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.820+0000 7fad72200640 1 --2- 172.21.15.45:0/757333766 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fad4c06ca10 0x7fad4c06eed0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.820+0000 7fad72200640 1 --2- 172.21.15.45:0/757333766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c146220 0x7fad6c0ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.820+0000 7fad72200640 1 --2- 172.21.15.45:0/757333766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6c06f7a0 0x7fad6c136c80 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.820+0000 7fad72200640 1 --2- 172.21.15.45:0/757333766 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad6c06e480 0x7fad6c145470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:17:59.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.820+0000 7fad72200640 1 -- 172.21.15.45:0/757333766 >> 172.21.15.45:0/757333766 conn(0x7fad6c06b930 msgr2=0x7fad6c15cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:17:59.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.820+0000 7fad72200640 1 -- 172.21.15.45:0/757333766 shutdown_connections 2024-04-16T16:17:59.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:17:59.820+0000 7fad72200640 1 -- 172.21.15.45:0/757333766 wait complete. 2024-04-16T16:17:59.839 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-16T16:17:59.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:18:00.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.066+0000 7f7994a00640 1 Processor -- start 2024-04-16T16:18:00.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.066+0000 7f7994a00640 1 -- start start 2024-04-16T16:18:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.068+0000 7f7994a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f799015ec60 0x7f79900d2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.068+0000 7f7994a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f79900d1ac0 0x7f79900b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.068+0000 7f7994a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79900d34c0 con 0x7f79900d4560 2024-04-16T16:18:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.068+0000 7f7994a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f79900d37a0 con 0x7f79900d1ac0 2024-04-16T16:18:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.068+0000 7f7994a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f79900d3a80 con 0x7f799015ec60 2024-04-16T16:18:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.068+0000 7f798d400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f79900d4560 0x7f79900d13b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52306/0 (socket says 172.21.15.45:52306) 2024-04-16T16:18:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.068+0000 7f798d400640 1 -- 172.21.15.45:0/1261778891 learned_addr learned my addr 172.21.15.45:0/1261778891 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:00.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.068+0000 7f798e800640 1 --2- 172.21.15.45:0/1261778891 >> v2:172.21.15.103:3300/0 conn(0x7f79900d1ac0 0x7f79900b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:00.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.068+0000 7f798de00640 1 --2- 172.21.15.45:0/1261778891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f799015ec60 0x7f79900d2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:00.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 -- 172.21.15.45:0/1261778891 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1073324687 0 0) 0x7f79900d34c0 con 0x7f79900d4560 2024-04-16T16:18:00.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 -- 172.21.15.45:0/1261778891 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7974003520 con 0x7f79900d4560 2024-04-16T16:18:00.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 -- 172.21.15.45:0/1261778891 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 219111478 0 0) 0x7f7974003520 con 0x7f79900d4560 2024-04-16T16:18:00.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 -- 172.21.15.45:0/1261778891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f799015ec60 msgr2=0x7f79900d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:00.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 --2- 172.21.15.45:0/1261778891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f799015ec60 0x7f79900d2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 -- 172.21.15.45:0/1261778891 >> v2:172.21.15.103:3300/0 conn(0x7f79900d1ac0 msgr2=0x7f79900b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:00.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 --2- 172.21.15.45:0/1261778891 >> v2:172.21.15.103:3300/0 conn(0x7f79900d1ac0 0x7f79900b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 -- 172.21.15.45:0/1261778891 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79900d3d60 con 0x7f79900d4560 2024-04-16T16:18:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 -- 172.21.15.45:0/1261778891 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f7978001670 con 0x7f79900d4560 2024-04-16T16:18:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 -- 172.21.15.45:0/1261778891 >> v1:172.21.15.45:6789/0 conn(0x7f79900d4560 legacy=0x7f79900d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 --2- 172.21.15.45:0/1261778891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79740039b0 0x7f7974003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 --2- 172.21.15.45:0/1261778891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7974004800 0x7f7974004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 --2- 172.21.15.45:0/1261778891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f799015ec60 0x7f79740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 -- 172.21.15.45:0/1261778891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f79900d37a0 con 0x7f79740039b0 2024-04-16T16:18:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 -- 172.21.15.45:0/1261778891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79900d3a80 con 0x7f7974004800 2024-04-16T16:18:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798ca00640 1 -- 172.21.15.45:0/1261778891 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f797400b9c0 con 0x7f799015ec60 2024-04-16T16:18:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.069+0000 7f798de00640 1 --2- 172.21.15.45:0/1261778891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79740039b0 0x7f7974003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.070+0000 7f798d400640 1 --2- 172.21.15.45:0/1261778891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7974004800 0x7f7974004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.070+0000 7f798de00640 1 -- 172.21.15.45:0/1261778891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f799015ec60 msgr2=0x7f79740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.070+0000 7f798de00640 1 --2- 172.21.15.45:0/1261778891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f799015ec60 0x7f79740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.070+0000 7f798de00640 1 -- 172.21.15.45:0/1261778891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7974004800 msgr2=0x7f7974004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:00.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.070+0000 7f798de00640 1 --2- 172.21.15.45:0/1261778891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7974004800 0x7f7974004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.070+0000 7f798de00640 1 -- 172.21.15.45:0/1261778891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7974003520 con 0x7f79740039b0 2024-04-16T16:18:00.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.070+0000 7f798d400640 1 --2- 172.21.15.45:0/1261778891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7974004800 0x7f7974004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:00.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.071+0000 7f798de00640 1 --2- 172.21.15.45:0/1261778891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79740039b0 0x7f7974003dd0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f7980009ac0 tx=0x7f7974005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28f8c5252502fd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:00.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.071+0000 7f798ca00640 1 -- 172.21.15.45:0/1261778891 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7980039070 con 0x7f79740039b0 2024-04-16T16:18:00.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.071+0000 7f798ca00640 1 -- 172.21.15.45:0/1261778891 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f798002c6d0 con 0x7f79740039b0 2024-04-16T16:18:00.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.071+0000 7f798ca00640 1 -- 172.21.15.45:0/1261778891 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7980009690 con 0x7f79740039b0 2024-04-16T16:18:00.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.072+0000 7f7994a00640 1 -- 172.21.15.45:0/1261778891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79740039b0 msgr2=0x7f7974003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:00.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.072+0000 7f7994a00640 1 --2- 172.21.15.45:0/1261778891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79740039b0 0x7f7974003dd0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f7980009ac0 tx=0x7f7974005830 comp rx=0 tx=0).stop 2024-04-16T16:18:00.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.072+0000 7f798e800640 1 -- 172.21.15.45:0/1261778891 reap_dead start 2024-04-16T16:18:00.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.072+0000 7f7994a00640 1 -- 172.21.15.45:0/1261778891 shutdown_connections 2024-04-16T16:18:00.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.072+0000 7f7994a00640 1 -- 172.21.15.45:0/1261778891 >> 172.21.15.45:0/1261778891 conn(0x7f7990073690 msgr2=0x7f799006cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:00.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.072+0000 7f7994a00640 1 -- 172.21.15.45:0/1261778891 shutdown_connections 2024-04-16T16:18:00.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.072+0000 7f7994a00640 1 -- 172.21.15.45:0/1261778891 wait complete. 2024-04-16T16:18:00.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.074+0000 7f7994a00640 1 Processor -- start 2024-04-16T16:18:00.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.074+0000 7f7994a00640 1 -- start start 2024-04-16T16:18:00.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.075+0000 7f7994a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79740039b0 0x7f799015ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:00.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.075+0000 7f7994a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79901e57b0 0x7f79901ea260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:00.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.075+0000 7f7994a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79901e5c30 0x7f79901ea7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:00.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.075+0000 7f7994a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f79900d2a00 con 0x7f79740039b0 2024-04-16T16:18:00.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.075+0000 7f7994a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79900ba460 con 0x7f79901e5c30 2024-04-16T16:18:00.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.075+0000 7f7994a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f79900d2b80 con 0x7f79901e57b0 2024-04-16T16:18:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.075+0000 7f798de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79740039b0 0x7f799015ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.075+0000 7f798de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79740039b0 0x7f799015ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57764/0 (socket says 172.21.15.45:57764) 2024-04-16T16:18:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.075+0000 7f798de00640 1 -- 172.21.15.45:0/1969013776 learned_addr learned my addr 172.21.15.45:0/1969013776 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.076+0000 7f798de00640 1 -- 172.21.15.45:0/1969013776 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79901e57b0 msgr2=0x7f79901ea260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.076+0000 7f798de00640 1 --2- 172.21.15.45:0/1969013776 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79901e57b0 0x7f79901ea260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.076+0000 7f798de00640 1 -- 172.21.15.45:0/1969013776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79901e5c30 msgr2=0x7f79901ea7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:00.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.076+0000 7f798de00640 1 --2- 172.21.15.45:0/1969013776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79901e5c30 0x7f79901ea7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.076+0000 7f798de00640 1 -- 172.21.15.45:0/1969013776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79901e6190 con 0x7f79740039b0 2024-04-16T16:18:00.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.076+0000 7f798de00640 1 --2- 172.21.15.45:0/1969013776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79740039b0 0x7f799015ae50 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f7980009be0 tx=0x7f7980004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:00.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.076+0000 7f7986000640 1 -- 172.21.15.45:0/1969013776 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7980042020 con 0x7f79740039b0 2024-04-16T16:18:00.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.076+0000 7f7986000640 1 -- 172.21.15.45:0/1969013776 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f798003a040 con 0x7f79740039b0 2024-04-16T16:18:00.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.076+0000 7f7986000640 1 -- 172.21.15.45:0/1969013776 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7980039070 con 0x7f79740039b0 2024-04-16T16:18:00.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.076+0000 7f7994a00640 1 -- 172.21.15.45:0/1969013776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79901e6470 con 0x7f79740039b0 2024-04-16T16:18:00.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.077+0000 7f7994a00640 1 -- 172.21.15.45:0/1969013776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79901e7e30 con 0x7f79740039b0 2024-04-16T16:18:00.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.078+0000 7f7994a00640 1 -- 172.21.15.45:0/1969013776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7990075780 con 0x7f79740039b0 2024-04-16T16:18:00.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.079+0000 7f7986000640 1 -- 172.21.15.45:0/1969013776 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f7980045020 con 0x7f79740039b0 2024-04-16T16:18:00.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.079+0000 7f7986000640 1 --2- 172.21.15.45:0/1969013776 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f796c06ca10 0x7f796c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:00.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.080+0000 7f7986000640 1 -- 172.21.15.45:0/1969013776 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6483+0+0 (secure 0 0 0) 0x7f79800ad460 con 0x7f79740039b0 2024-04-16T16:18:00.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.080+0000 7f798d400640 1 --2- 172.21.15.45:0/1969013776 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f796c06ca10 0x7f796c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:00.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.081+0000 7f798d400640 1 --2- 172.21.15.45:0/1969013776 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f796c06ca10 0x7f796c06eed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f79901f7d80 tx=0x7f7978009040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:00.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.084+0000 7f7986000640 1 -- 172.21.15.45:0/1969013776 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f798004c070 con 0x7f79740039b0 2024-04-16T16:18:00.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.331+0000 7f7994a00640 1 -- 172.21.15.45:0/1969013776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79900ba960 con 0x7f79740039b0 2024-04-16T16:18:00.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.332+0000 7f7986000640 1 -- 172.21.15.45:0/1969013776 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5202 (secure 0 0 0) 0x7f798007b460 con 0x7f79740039b0 2024-04-16T16:18:00.332 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:00.332 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":55,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:17:53.689159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6402,"mds_1":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"c","rank":1,"incarnation":52,"state":"up:active","state_seq":11,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6399,6402]},"id":5}]} 2024-04-16T16:18:00.333 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 55 2024-04-16T16:18:00.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.335+0000 7f7994a00640 1 -- 172.21.15.45:0/1969013776 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f796c06ca10 msgr2=0x7f796c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:00.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.335+0000 7f7994a00640 1 --2- 172.21.15.45:0/1969013776 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f796c06ca10 0x7f796c06eed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f79901f7d80 tx=0x7f7978009040 comp rx=0 tx=0).stop 2024-04-16T16:18:00.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.335+0000 7f7994a00640 1 -- 172.21.15.45:0/1969013776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79740039b0 msgr2=0x7f799015ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:00.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.335+0000 7f7994a00640 1 --2- 172.21.15.45:0/1969013776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79740039b0 0x7f799015ae50 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f7980009be0 tx=0x7f7980004730 comp rx=0 tx=0).stop 2024-04-16T16:18:00.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.335+0000 7f7994a00640 1 -- 172.21.15.45:0/1969013776 shutdown_connections 2024-04-16T16:18:00.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.335+0000 7f7994a00640 1 --2- 172.21.15.45:0/1969013776 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f796c06ca10 0x7f796c06eed0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.335+0000 7f7994a00640 1 --2- 172.21.15.45:0/1969013776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79901e5c30 0x7f79901ea7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.335+0000 7f7994a00640 1 --2- 172.21.15.45:0/1969013776 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79901e57b0 0x7f79901ea260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.335+0000 7f7994a00640 1 --2- 172.21.15.45:0/1969013776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79740039b0 0x7f799015ae50 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.335+0000 7f7994a00640 1 -- 172.21.15.45:0/1969013776 >> 172.21.15.45:0/1969013776 conn(0x7f7990073690 msgr2=0x7f79900cd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:00.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.335+0000 7f7994a00640 1 -- 172.21.15.45:0/1969013776 shutdown_connections 2024-04-16T16:18:00.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.335+0000 7f7994a00640 1 -- 172.21.15.45:0/1969013776 wait complete. 2024-04-16T16:18:00.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:18:00.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.579+0000 7ff37be00640 1 Processor -- start 2024-04-16T16:18:00.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.579+0000 7ff37be00640 1 -- start start 2024-04-16T16:18:00.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.581+0000 7ff37be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff37c158a40 0x7ff37c15be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:00.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.581+0000 7ff37be00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff37c0cfff0 con 0x7ff37c06eee0 2024-04-16T16:18:00.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.581+0000 7ff37be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff37c0d02d0 con 0x7ff37c0b2de0 2024-04-16T16:18:00.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.581+0000 7ff37be00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff37c15c890 con 0x7ff37c158a40 2024-04-16T16:18:00.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.581+0000 7ff37aa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff37c158a40 0x7ff37c15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:00.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.581+0000 7ff37aa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff37c158a40 0x7ff37c15be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57768/0 (socket says 172.21.15.45:57768) 2024-04-16T16:18:00.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.581+0000 7ff37aa00640 1 -- 172.21.15.45:0/461059678 learned_addr learned my addr 172.21.15.45:0/461059678 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:00.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.582+0000 7ff37aa00640 1 -- 172.21.15.45:0/461059678 >> v1:172.21.15.103:6789/0 conn(0x7ff37c0b2de0 legacy=0x7ff37c06e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:00.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.582+0000 7ff37aa00640 1 -- 172.21.15.45:0/461059678 >> v1:172.21.15.45:6789/0 conn(0x7ff37c06eee0 legacy=0x7ff37c15cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:00.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.582+0000 7ff37aa00640 1 -- 172.21.15.45:0/461059678 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37c15c3b0 con 0x7ff37c158a40 2024-04-16T16:18:00.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.583+0000 7ff37aa00640 1 --2- 172.21.15.45:0/461059678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff37c158a40 0x7ff37c15be70 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff36c009870 tx=0x7ff36c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7345d52e8e48296d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:00.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.583+0000 7ff379600640 1 -- 172.21.15.45:0/461059678 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c035020 con 0x7ff37c158a40 2024-04-16T16:18:00.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.584+0000 7ff379600640 1 -- 172.21.15.45:0/461059678 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff36c031870 con 0x7ff37c158a40 2024-04-16T16:18:00.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.584+0000 7ff379600640 1 -- 172.21.15.45:0/461059678 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c031c40 con 0x7ff37c158a40 2024-04-16T16:18:00.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.584+0000 7ff37be00640 1 -- 172.21.15.45:0/461059678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff37c158a40 msgr2=0x7ff37c15be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:00.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.584+0000 7ff37be00640 1 --2- 172.21.15.45:0/461059678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff37c158a40 0x7ff37c15be70 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff36c009870 tx=0x7ff36c02b910 comp rx=0 tx=0).stop 2024-04-16T16:18:00.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.584+0000 7ff37be00640 1 -- 172.21.15.45:0/461059678 shutdown_connections 2024-04-16T16:18:00.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.584+0000 7ff37be00640 1 --2- 172.21.15.45:0/461059678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff37c158a40 0x7ff37c15be70 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.584+0000 7ff37be00640 1 -- 172.21.15.45:0/461059678 >> 172.21.15.45:0/461059678 conn(0x7ff37c06b930 msgr2=0x7ff37c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:00.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.584+0000 7ff37be00640 1 -- 172.21.15.45:0/461059678 shutdown_connections 2024-04-16T16:18:00.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.584+0000 7ff37be00640 1 -- 172.21.15.45:0/461059678 wait complete. 2024-04-16T16:18:00.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.587+0000 7ff37be00640 1 Processor -- start 2024-04-16T16:18:00.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.587+0000 7ff37be00640 1 -- start start 2024-04-16T16:18:00.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.588+0000 7ff37be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff37c06eee0 0x7ff37c157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:00.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.588+0000 7ff37be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff37c0b2de0 0x7ff37c0ab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:00.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.589+0000 7ff37be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff37c158a40 0x7ff37c0ab9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.589+0000 7ff37be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c06e1d0 con 0x7ff37c0b2de0 2024-04-16T16:18:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.589+0000 7ff37be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c15c5e0 con 0x7ff37c06eee0 2024-04-16T16:18:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.589+0000 7ff37be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff37c06e350 con 0x7ff37c158a40 2024-04-16T16:18:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.589+0000 7ff37a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff37c0b2de0 0x7ff37c0ab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.589+0000 7ff37a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff37c0b2de0 0x7ff37c0ab4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57774/0 (socket says 172.21.15.45:57774) 2024-04-16T16:18:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.589+0000 7ff37a000640 1 -- 172.21.15.45:0/1553616559 learned_addr learned my addr 172.21.15.45:0/1553616559 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.589+0000 7ff37aa00640 1 --2- 172.21.15.45:0/1553616559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff37c06eee0 0x7ff37c157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.589+0000 7ff37a000640 1 -- 172.21.15.45:0/1553616559 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff37c158a40 msgr2=0x7ff37c0ab9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.589+0000 7ff37a000640 1 --2- 172.21.15.45:0/1553616559 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff37c158a40 0x7ff37c0ab9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.589+0000 7ff37a000640 1 -- 172.21.15.45:0/1553616559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff37c06eee0 msgr2=0x7ff37c157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.589+0000 7ff37a000640 1 --2- 172.21.15.45:0/1553616559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff37c06eee0 0x7ff37c157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.589+0000 7ff37a000640 1 -- 172.21.15.45:0/1553616559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37c156a40 con 0x7ff37c0b2de0 2024-04-16T16:18:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.590+0000 7ff37a000640 1 --2- 172.21.15.45:0/1553616559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff37c0b2de0 0x7ff37c0ab4a0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ff36800c430 tx=0x7ff36800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:00.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.590+0000 7ff372a00640 1 -- 172.21.15.45:0/1553616559 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff368011790 con 0x7ff37c0b2de0 2024-04-16T16:18:00.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.590+0000 7ff372a00640 1 -- 172.21.15.45:0/1553616559 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff368017070 con 0x7ff37c0b2de0 2024-04-16T16:18:00.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.590+0000 7ff372a00640 1 -- 172.21.15.45:0/1553616559 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff368011bb0 con 0x7ff37c0b2de0 2024-04-16T16:18:00.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.590+0000 7ff37be00640 1 -- 172.21.15.45:0/1553616559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff37c157970 con 0x7ff37c0b2de0 2024-04-16T16:18:00.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.590+0000 7ff37be00640 1 -- 172.21.15.45:0/1553616559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff37c155b20 con 0x7ff37c0b2de0 2024-04-16T16:18:00.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.592+0000 7ff37be00640 1 -- 172.21.15.45:0/1553616559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff37c075780 con 0x7ff37c0b2de0 2024-04-16T16:18:00.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.593+0000 7ff372a00640 1 -- 172.21.15.45:0/1553616559 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7ff36801c460 con 0x7ff37c0b2de0 2024-04-16T16:18:00.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.594+0000 7ff372a00640 1 --2- 172.21.15.45:0/1553616559 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff35c06ca10 0x7ff35c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:00.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.594+0000 7ff372a00640 1 -- 172.21.15.45:0/1553616559 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6483+0+0 (secure 0 0 0) 0x7ff36808bf10 con 0x7ff37c0b2de0 2024-04-16T16:18:00.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.595+0000 7ff37aa00640 1 --2- 172.21.15.45:0/1553616559 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff35c06ca10 0x7ff35c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:00.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.596+0000 7ff37aa00640 1 --2- 172.21.15.45:0/1553616559 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff35c06ca10 0x7ff35c06eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff36c036000 tx=0x7ff36c03c040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:00.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.599+0000 7ff372a00640 1 -- 172.21.15.45:0/1553616559 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff368059f10 con 0x7ff37c0b2de0 2024-04-16T16:18:00.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.837+0000 7ff37be00640 1 -- 172.21.15.45:0/1553616559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff37c15d880 con 0x7ff37c0b2de0 2024-04-16T16:18:00.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.838+0000 7ff372a00640 1 -- 172.21.15.45:0/1553616559 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5202 (secure 0 0 0) 0x7ff368059810 con 0x7ff37c0b2de0 2024-04-16T16:18:00.838 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:00.838 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":55,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:16:39.513364+0000","modified":"2024-04-16T16:17:53.689159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6402,"mds_1":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"c","rank":1,"incarnation":52,"state":"up:active","state_seq":11,"addr":"172.21.15.45:6834/883721507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":883721507},{"type":"v1","addr":"172.21.15.45:6834","nonce":883721507}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/268877398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":268877398},{"type":"v1","addr":"172.21.15.45:6835","nonce":268877398}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6399,6402]},"id":5}]} 2024-04-16T16:18:00.839 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 55 2024-04-16T16:18:00.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.840+0000 7ff37be00640 1 -- 172.21.15.45:0/1553616559 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff35c06ca10 msgr2=0x7ff35c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:00.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.840+0000 7ff37be00640 1 --2- 172.21.15.45:0/1553616559 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff35c06ca10 0x7ff35c06eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff36c036000 tx=0x7ff36c03c040 comp rx=0 tx=0).stop 2024-04-16T16:18:00.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.840+0000 7ff37be00640 1 -- 172.21.15.45:0/1553616559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff37c0b2de0 msgr2=0x7ff37c0ab4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:00.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.840+0000 7ff37be00640 1 --2- 172.21.15.45:0/1553616559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff37c0b2de0 0x7ff37c0ab4a0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ff36800c430 tx=0x7ff36800c8f0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.841+0000 7ff37be00640 1 -- 172.21.15.45:0/1553616559 shutdown_connections 2024-04-16T16:18:00.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.841+0000 7ff37be00640 1 --2- 172.21.15.45:0/1553616559 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff35c06ca10 0x7ff35c06eed0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.841+0000 7ff37be00640 1 --2- 172.21.15.45:0/1553616559 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff37c158a40 0x7ff37c0ab9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.841+0000 7ff37be00640 1 --2- 172.21.15.45:0/1553616559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff37c0b2de0 0x7ff37c0ab4a0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.841+0000 7ff37be00640 1 --2- 172.21.15.45:0/1553616559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff37c06eee0 0x7ff37c157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:00.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.841+0000 7ff37be00640 1 -- 172.21.15.45:0/1553616559 >> 172.21.15.45:0/1553616559 conn(0x7ff37c06b930 msgr2=0x7ff37c0b53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:00.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.841+0000 7ff37be00640 1 -- 172.21.15.45:0/1553616559 shutdown_connections 2024-04-16T16:18:00.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:00.841+0000 7ff37be00640 1 -- 172.21.15.45:0/1553616559 wait complete. 2024-04-16T16:18:00.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:18:01.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.057+0000 7f793c200640 1 Processor -- start 2024-04-16T16:18:01.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.058+0000 7f793c200640 1 -- start start 2024-04-16T16:18:01.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.059+0000 7f793c200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f793409af60 0x7f793406b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:01.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.059+0000 7f793c200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793406f490 con 0x7f793406be70 2024-04-16T16:18:01.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.059+0000 7f793c200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793406f730 con 0x7f793415ecc0 2024-04-16T16:18:01.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.059+0000 7f793c200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f79340703f0 con 0x7f793409af60 2024-04-16T16:18:01.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.060+0000 7f7938e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f793409af60 0x7f793406b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:01.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.060+0000 7f793a200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f793406be70 0x7f79340cf580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52322/0 (socket says 172.21.15.45:52322) 2024-04-16T16:18:01.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.060+0000 7f793a200640 1 -- 172.21.15.45:0/2950210985 learned_addr learned my addr 172.21.15.45:0/2950210985 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:01.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.060+0000 7f7938e00640 1 -- 172.21.15.45:0/2950210985 >> v1:172.21.15.103:6789/0 conn(0x7f793415ecc0 legacy=0x7f793406fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:01.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.060+0000 7f7938e00640 1 -- 172.21.15.45:0/2950210985 >> v1:172.21.15.45:6789/0 conn(0x7f793406be70 legacy=0x7f79340cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:01.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.060+0000 7f7938e00640 1 -- 172.21.15.45:0/2950210985 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79340b3bf0 con 0x7f793409af60 2024-04-16T16:18:01.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.060+0000 7f7933e00640 1 -- 172.21.15.45:0/2950210985 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1101635674 0 0) 0x7f793406f490 con 0x7f793406be70 2024-04-16T16:18:01.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.061+0000 7f7938e00640 1 --2- 172.21.15.45:0/2950210985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f793409af60 0x7f793406b930 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f79200097a0 tx=0x7f792002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5caf9bdaa7bd6b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:01.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.061+0000 7f7933e00640 1 -- 172.21.15.45:0/2950210985 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7920035020 con 0x7f793409af60 2024-04-16T16:18:01.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.062+0000 7f7933e00640 1 -- 172.21.15.45:0/2950210985 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7920031830 con 0x7f793409af60 2024-04-16T16:18:01.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.062+0000 7f7933e00640 1 -- 172.21.15.45:0/2950210985 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7920031c00 con 0x7f793409af60 2024-04-16T16:18:01.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.062+0000 7f793c200640 1 -- 172.21.15.45:0/2950210985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f793409af60 msgr2=0x7f793406b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:01.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.062+0000 7f793c200640 1 --2- 172.21.15.45:0/2950210985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f793409af60 0x7f793406b930 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f79200097a0 tx=0x7f792002b8d0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.062+0000 7f793c200640 1 -- 172.21.15.45:0/2950210985 shutdown_connections 2024-04-16T16:18:01.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.062+0000 7f793c200640 1 --2- 172.21.15.45:0/2950210985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f793409af60 0x7f793406b930 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.062+0000 7f793c200640 1 -- 172.21.15.45:0/2950210985 >> 172.21.15.45:0/2950210985 conn(0x7f793409aa70 msgr2=0x7f79340b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:01.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.062+0000 7f793c200640 1 -- 172.21.15.45:0/2950210985 shutdown_connections 2024-04-16T16:18:01.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.062+0000 7f793c200640 1 -- 172.21.15.45:0/2950210985 wait complete. 2024-04-16T16:18:01.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.064+0000 7f793c200640 1 Processor -- start 2024-04-16T16:18:01.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.064+0000 7f793c200640 1 -- start start 2024-04-16T16:18:01.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.065+0000 7f793c200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f793406be70 0x7f793413e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:01.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.065+0000 7f793c200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f793409af60 0x7f79341f45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.065+0000 7f793c200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f793415ecc0 0x7f7934160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.065+0000 7f793c200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f79340cfc90 con 0x7f793406be70 2024-04-16T16:18:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.065+0000 7f793c200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79340b3f60 con 0x7f793409af60 2024-04-16T16:18:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.065+0000 7f793c200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f79340cfe10 con 0x7f793415ecc0 2024-04-16T16:18:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.065+0000 7f7939800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f793406be70 0x7f793413e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.065+0000 7f7939800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f793406be70 0x7f793413e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57800/0 (socket says 172.21.15.45:57800) 2024-04-16T16:18:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.066+0000 7f7939800640 1 -- 172.21.15.45:0/709999781 learned_addr learned my addr 172.21.15.45:0/709999781 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.066+0000 7f7938e00640 1 --2- 172.21.15.45:0/709999781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f793409af60 0x7f79341f45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.066+0000 7f793a200640 1 --2- 172.21.15.45:0/709999781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f793415ecc0 0x7f7934160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.066+0000 7f7939800640 1 -- 172.21.15.45:0/709999781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f793415ecc0 msgr2=0x7f7934160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.066+0000 7f7939800640 1 --2- 172.21.15.45:0/709999781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f793415ecc0 0x7f7934160360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.066+0000 7f7939800640 1 -- 172.21.15.45:0/709999781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f793409af60 msgr2=0x7f79341f45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:01.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.066+0000 7f7939800640 1 --2- 172.21.15.45:0/709999781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f793409af60 0x7f79341f45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.066+0000 7f7939800640 1 -- 172.21.15.45:0/709999781 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79341f5540 con 0x7f793406be70 2024-04-16T16:18:01.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.066+0000 7f7938e00640 1 --2- 172.21.15.45:0/709999781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f793409af60 0x7f79341f45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:01.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.066+0000 7f7939800640 1 --2- 172.21.15.45:0/709999781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f793406be70 0x7f793413e4d0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f792800ba50 tx=0x7f792800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:01.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.066+0000 7f7931600640 1 -- 172.21.15.45:0/709999781 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7928019070 con 0x7f793406be70 2024-04-16T16:18:01.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.067+0000 7f7931600640 1 -- 172.21.15.45:0/709999781 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f792800cc20 con 0x7f793406be70 2024-04-16T16:18:01.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.067+0000 7f7931600640 1 -- 172.21.15.45:0/709999781 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7928014680 con 0x7f793406be70 2024-04-16T16:18:01.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.067+0000 7f793c200640 1 -- 172.21.15.45:0/709999781 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f793415be40 con 0x7f793406be70 2024-04-16T16:18:01.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.067+0000 7f793c200640 1 -- 172.21.15.45:0/709999781 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79341f5260 con 0x7f793406be70 2024-04-16T16:18:01.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.068+0000 7f793c200640 1 -- 172.21.15.45:0/709999781 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7934075780 con 0x7f793406be70 2024-04-16T16:18:01.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.069+0000 7f7931600640 1 -- 172.21.15.45:0/709999781 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f792801c070 con 0x7f793406be70 2024-04-16T16:18:01.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.070+0000 7f7931600640 1 --2- 172.21.15.45:0/709999781 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f791406ca10 0x7f791406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:01.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.071+0000 7f7931600640 1 -- 172.21.15.45:0/709999781 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6483+0+0 (secure 0 0 0) 0x7f792808bc40 con 0x7f793406be70 2024-04-16T16:18:01.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.071+0000 7f7938e00640 1 --2- 172.21.15.45:0/709999781 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f791406ca10 0x7f791406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:01.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.072+0000 7f7938e00640 1 --2- 172.21.15.45:0/709999781 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f791406ca10 0x7f791406eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7920036000 tx=0x7f7920008030 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:01.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.076+0000 7f7931600640 1 -- 172.21.15.45:0/709999781 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f79280595f0 con 0x7f793406be70 2024-04-16T16:18:01.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.283+0000 7f793c200640 1 -- 172.21.15.45:0/709999781 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f793406e210 con 0x7f793406be70 2024-04-16T16:18:01.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.285+0000 7f7931600640 1 -- 172.21.15.45:0/709999781 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+17807 (secure 0 0 0) 0x7f792805d8f0 con 0x7f793406be70 2024-04-16T16:18:01.285 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:01.285 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":74,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:17:55.075736+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:16:37.449973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-16T16:16:38.485027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/1692905448":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/1736628710":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/1783855160":"2024-04-17T16:17:55.075639+0000","172.21.15.103:6833/541757982":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/2044864797":"2024-04-17T16:17:55.075639+0000","172.21.15.103:6833/4155603915":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/951047877":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/1131547374":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/3176524506":"2024-04-17T16:17:01.408806+0000","172.21.15.103:6832/4155603915":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/365521086":"2024-04-17T16:17:01.408806+0000","172.21.15.103:6832/541757982":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/3345679536":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/2747644735":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/3606536347":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/1223877773":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/252084277":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/650078727":"2024-04-17T16:17:01.408806+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:18:01.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.287+0000 7f793c200640 1 -- 172.21.15.45:0/709999781 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f791406ca10 msgr2=0x7f791406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:01.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.287+0000 7f793c200640 1 --2- 172.21.15.45:0/709999781 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f791406ca10 0x7f791406eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7920036000 tx=0x7f7920008030 comp rx=0 tx=0).stop 2024-04-16T16:18:01.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.287+0000 7f793c200640 1 -- 172.21.15.45:0/709999781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f793406be70 msgr2=0x7f793413e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:01.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.287+0000 7f793c200640 1 --2- 172.21.15.45:0/709999781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f793406be70 0x7f793413e4d0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f792800ba50 tx=0x7f792800bf10 comp rx=0 tx=0).stop 2024-04-16T16:18:01.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.288+0000 7f793c200640 1 -- 172.21.15.45:0/709999781 shutdown_connections 2024-04-16T16:18:01.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.288+0000 7f793c200640 1 --2- 172.21.15.45:0/709999781 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f791406ca10 0x7f791406eed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.288+0000 7f793c200640 1 --2- 172.21.15.45:0/709999781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f793415ecc0 0x7f7934160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.288+0000 7f793c200640 1 --2- 172.21.15.45:0/709999781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f793409af60 0x7f79341f45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.288+0000 7f793c200640 1 --2- 172.21.15.45:0/709999781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f793406be70 0x7f793413e4d0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.288+0000 7f793c200640 1 -- 172.21.15.45:0/709999781 >> 172.21.15.45:0/709999781 conn(0x7f793409aa70 msgr2=0x7f79340cd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.288+0000 7f793c200640 1 -- 172.21.15.45:0/709999781 shutdown_connections 2024-04-16T16:18:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.288+0000 7f793c200640 1 -- 172.21.15.45:0/709999781 wait complete. 2024-04-16T16:18:01.307 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-16T16:18:01.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.521+0000 7faf23e00640 1 Processor -- start 2024-04-16T16:18:01.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.522+0000 7faf23e00640 1 -- start start 2024-04-16T16:18:01.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.523+0000 7faf23e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faf2406f730 0x7faf240c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:01.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.523+0000 7faf23e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7faf240c6320 0x7faf2406df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:01.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.523+0000 7faf23e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf2415d600 con 0x7faf240c6fc0 2024-04-16T16:18:01.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.523+0000 7faf23e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7faf240c5670 con 0x7faf2406f730 2024-04-16T16:18:01.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.523+0000 7faf23e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7faf2415d8e0 con 0x7faf240c6320 2024-04-16T16:18:01.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.524+0000 7faf22000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faf2406f730 0x7faf240c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:01.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.524+0000 7faf22000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faf2406f730 0x7faf240c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57808/0 (socket says 172.21.15.45:57808) 2024-04-16T16:18:01.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.524+0000 7faf22000640 1 -- 172.21.15.45:0/45207125 learned_addr learned my addr 172.21.15.45:0/45207125 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:01.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.524+0000 7faf22000640 1 -- 172.21.15.45:0/45207125 >> v2:172.21.15.103:3300/0 conn(0x7faf240c6320 msgr2=0x7faf2406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:01.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.524+0000 7faf22000640 1 --2- 172.21.15.45:0/45207125 >> v2:172.21.15.103:3300/0 conn(0x7faf240c6320 0x7faf2406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.524+0000 7faf22000640 1 -- 172.21.15.45:0/45207125 >> v1:172.21.15.45:6789/0 conn(0x7faf240c6fc0 legacy=0x7faf2415cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:01.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.524+0000 7faf22000640 1 -- 172.21.15.45:0/45207125 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf2415dbc0 con 0x7faf2406f730 2024-04-16T16:18:01.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.525+0000 7faf22000640 1 --2- 172.21.15.45:0/45207125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf2406f730 0x7faf240c5de0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7faf240c84e0 tx=0x7faf0c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92bf3df5c7ff2d2e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:01.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.526+0000 7faf21600640 1 -- 172.21.15.45:0/45207125 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0c035020 con 0x7faf2406f730 2024-04-16T16:18:01.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.526+0000 7faf21600640 1 -- 172.21.15.45:0/45207125 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf0c031870 con 0x7faf2406f730 2024-04-16T16:18:01.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.526+0000 7faf21600640 1 -- 172.21.15.45:0/45207125 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0c031c40 con 0x7faf2406f730 2024-04-16T16:18:01.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.526+0000 7faf23e00640 1 -- 172.21.15.45:0/45207125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf2406f730 msgr2=0x7faf240c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:01.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.526+0000 7faf23e00640 1 --2- 172.21.15.45:0/45207125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf2406f730 0x7faf240c5de0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7faf240c84e0 tx=0x7faf0c02b910 comp rx=0 tx=0).stop 2024-04-16T16:18:01.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.527+0000 7faf23e00640 1 -- 172.21.15.45:0/45207125 shutdown_connections 2024-04-16T16:18:01.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.527+0000 7faf23e00640 1 --2- 172.21.15.45:0/45207125 >> v2:172.21.15.103:3300/0 conn(0x7faf240c6320 0x7faf2406df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.527+0000 7faf23e00640 1 --2- 172.21.15.45:0/45207125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf2406f730 0x7faf240c5de0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.527+0000 7faf23e00640 1 -- 172.21.15.45:0/45207125 >> 172.21.15.45:0/45207125 conn(0x7faf2406b930 msgr2=0x7faf2406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:01.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.527+0000 7faf23e00640 1 -- 172.21.15.45:0/45207125 shutdown_connections 2024-04-16T16:18:01.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.527+0000 7faf23e00640 1 -- 172.21.15.45:0/45207125 wait complete. 2024-04-16T16:18:01.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.528+0000 7faf23e00640 1 Processor -- start 2024-04-16T16:18:01.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.528+0000 7faf23e00640 1 -- start start 2024-04-16T16:18:01.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.529+0000 7faf23e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf2406f730 0x7faf2415a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:01.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.529+0000 7faf23e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faf240c6320 0x7faf24158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:01.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.529+0000 7faf23e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf240c6fc0 0x7faf241f4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:01.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.529+0000 7faf23e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7faf240c7520 con 0x7faf240c6fc0 2024-04-16T16:18:01.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.529+0000 7faf23e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faf240c73a0 con 0x7faf2406f730 2024-04-16T16:18:01.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.529+0000 7faf23e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faf2406e480 con 0x7faf240c6320 2024-04-16T16:18:01.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.530+0000 7faf23400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf240c6fc0 0x7faf241f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:01.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.530+0000 7faf23400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf240c6fc0 0x7faf241f4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57820/0 (socket says 172.21.15.45:57820) 2024-04-16T16:18:01.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.530+0000 7faf23400640 1 -- 172.21.15.45:0/1064144332 learned_addr learned my addr 172.21.15.45:0/1064144332 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:01.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.530+0000 7faf22a00640 1 --2- 172.21.15.45:0/1064144332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf2406f730 0x7faf2415a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:01.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.530+0000 7faf22000640 1 --2- 172.21.15.45:0/1064144332 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faf240c6320 0x7faf24158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:01.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.530+0000 7faf23400640 1 -- 172.21.15.45:0/1064144332 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faf240c6320 msgr2=0x7faf24158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:01.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.530+0000 7faf23400640 1 --2- 172.21.15.45:0/1064144332 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faf240c6320 0x7faf24158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.530+0000 7faf23400640 1 -- 172.21.15.45:0/1064144332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf2406f730 msgr2=0x7faf2415a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:01.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.530+0000 7faf23400640 1 --2- 172.21.15.45:0/1064144332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf2406f730 0x7faf2415a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.530+0000 7faf23400640 1 -- 172.21.15.45:0/1064144332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf24158580 con 0x7faf240c6fc0 2024-04-16T16:18:01.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.530+0000 7faf23400640 1 --2- 172.21.15.45:0/1064144332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf240c6fc0 0x7faf241f4720 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00b450 tx=0x7faf1c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:01.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.531+0000 7faf1aa00640 1 -- 172.21.15.45:0/1064144332 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c015020 con 0x7faf240c6fc0 2024-04-16T16:18:01.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.531+0000 7faf1aa00640 1 -- 172.21.15.45:0/1064144332 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf1c011830 con 0x7faf240c6fc0 2024-04-16T16:18:01.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.531+0000 7faf1aa00640 1 -- 172.21.15.45:0/1064144332 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c011c00 con 0x7faf240c6fc0 2024-04-16T16:18:01.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.531+0000 7faf23e00640 1 -- 172.21.15.45:0/1064144332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf2415aba0 con 0x7faf240c6fc0 2024-04-16T16:18:01.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.531+0000 7faf23e00640 1 -- 172.21.15.45:0/1064144332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf2415b440 con 0x7faf240c6fc0 2024-04-16T16:18:01.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.532+0000 7faf23e00640 1 -- 172.21.15.45:0/1064144332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf24075780 con 0x7faf240c6fc0 2024-04-16T16:18:01.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.533+0000 7faf1aa00640 1 -- 172.21.15.45:0/1064144332 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7faf1c018020 con 0x7faf240c6fc0 2024-04-16T16:18:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.534+0000 7faf1aa00640 1 --2- 172.21.15.45:0/1064144332 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7faf0006ca10 0x7faf0006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.534+0000 7faf1aa00640 1 -- 172.21.15.45:0/1064144332 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6483+0+0 (secure 0 0 0) 0x7faf1c08bf90 con 0x7faf240c6fc0 2024-04-16T16:18:01.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.535+0000 7faf22a00640 1 --2- 172.21.15.45:0/1064144332 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7faf0006ca10 0x7faf0006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.536+0000 7faf22a00640 1 --2- 172.21.15.45:0/1064144332 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7faf0006ca10 0x7faf0006eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faf14001ca0 tx=0x7faf14009040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.539+0000 7faf1aa00640 1 -- 172.21.15.45:0/1064144332 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faf1c01f020 con 0x7faf240c6fc0 2024-04-16T16:18:01.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.774+0000 7faf23e00640 1 -- 172.21.15.45:0/1064144332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7faf24068ce0 con 0x7faf240c6fc0 2024-04-16T16:18:01.775 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-04-16T16:18:01.776+0000 7fe7a4800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T16:18:01.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.779+0000 7faf1aa00640 1 -- 172.21.15.45:0/1064144332 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v56) ==== 161+0+0 (secure 0 0 0) 0x7faf1c0172c0 con 0x7faf240c6fc0 2024-04-16T16:18:01.780 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-16T16:18:01.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.783+0000 7faf23e00640 1 -- 172.21.15.45:0/1064144332 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7faf0006ca10 msgr2=0x7faf0006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:01.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.783+0000 7faf23e00640 1 --2- 172.21.15.45:0/1064144332 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7faf0006ca10 0x7faf0006eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faf14001ca0 tx=0x7faf14009040 comp rx=0 tx=0).stop 2024-04-16T16:18:01.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.783+0000 7faf23e00640 1 -- 172.21.15.45:0/1064144332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf240c6fc0 msgr2=0x7faf241f4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:01.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.783+0000 7faf23e00640 1 --2- 172.21.15.45:0/1064144332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf240c6fc0 0x7faf241f4720 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00b450 tx=0x7faf1c00b910 comp rx=0 tx=0).stop 2024-04-16T16:18:01.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.783+0000 7faf23e00640 1 -- 172.21.15.45:0/1064144332 shutdown_connections 2024-04-16T16:18:01.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.783+0000 7faf23e00640 1 --2- 172.21.15.45:0/1064144332 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7faf0006ca10 0x7faf0006eed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.783+0000 7faf23e00640 1 --2- 172.21.15.45:0/1064144332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf240c6fc0 0x7faf241f4720 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.783+0000 7faf23e00640 1 --2- 172.21.15.45:0/1064144332 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faf240c6320 0x7faf24158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.783+0000 7faf23e00640 1 --2- 172.21.15.45:0/1064144332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf2406f730 0x7faf2415a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.783+0000 7faf23e00640 1 -- 172.21.15.45:0/1064144332 >> 172.21.15.45:0/1064144332 conn(0x7faf2406b930 msgr2=0x7faf24160b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:01.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.785+0000 7faf23e00640 1 -- 172.21.15.45:0/1064144332 shutdown_connections 2024-04-16T16:18:01.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.785+0000 7faf23e00640 1 -- 172.21.15.45:0/1064144332 wait complete. 2024-04-16T16:18:01.803 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-16T16:18:01.866 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2024-04-16T16:18:01.896 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2024-04-16T16:18:01.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.993+0000 7f46a3e00640 1 Processor -- start 2024-04-16T16:18:01.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.993+0000 7f46a3e00640 1 -- start start 2024-04-16T16:18:01.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.995+0000 7f46a3e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f469c0d0e80 0x7f469c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:01.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.995+0000 7f46a3e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f469c0ccb70 con 0x7f469c0afda0 2024-04-16T16:18:01.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.995+0000 7f46a3e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f469c0cce50 con 0x7f469c15b920 2024-04-16T16:18:01.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.995+0000 7f46a3e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f469c06e7a0 con 0x7f469c0d0e80 2024-04-16T16:18:01.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.995+0000 7f46a0a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f469c0d0e80 0x7f469c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:01.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.995+0000 7f46a1e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f469c0afda0 0x7f469c0ce5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52348/0 (socket says 172.21.15.45:52348) 2024-04-16T16:18:01.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.995+0000 7f46a0a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f469c0d0e80 0x7f469c0b4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57850/0 (socket says 172.21.15.45:57850) 2024-04-16T16:18:01.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.995+0000 7f46a1e00640 1 -- 172.21.15.45:0/2035397149 learned_addr learned my addr 172.21.15.45:0/2035397149 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:01.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.996+0000 7f46a0a00640 1 -- 172.21.15.45:0/2035397149 >> v1:172.21.15.103:6789/0 conn(0x7f469c15b920 legacy=0x7f469c0d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:01.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.996+0000 7f46a0a00640 1 -- 172.21.15.45:0/2035397149 >> v1:172.21.15.45:6789/0 conn(0x7f469c0afda0 legacy=0x7f469c0ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:01.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.996+0000 7f46a0a00640 1 -- 172.21.15.45:0/2035397149 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469c06e430 con 0x7f469c0d0e80 2024-04-16T16:18:01.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.996+0000 7f469be00640 1 -- 172.21.15.45:0/2035397149 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1315401580 0 0) 0x7f469c0ccb70 con 0x7f469c0afda0 2024-04-16T16:18:01.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.997+0000 7f46a0a00640 1 --2- 172.21.15.45:0/2035397149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f469c0d0e80 0x7f469c0b4e20 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f469c06e920 tx=0x7f468802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298ee85c2436ec85 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:01.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.997+0000 7f469be00640 1 -- 172.21.15.45:0/2035397149 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4688035020 con 0x7f469c0d0e80 2024-04-16T16:18:01.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.997+0000 7f469be00640 1 -- 172.21.15.45:0/2035397149 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4688031810 con 0x7f469c0d0e80 2024-04-16T16:18:01.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.997+0000 7f469be00640 1 -- 172.21.15.45:0/2035397149 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4688031be0 con 0x7f469c0d0e80 2024-04-16T16:18:01.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.998+0000 7f46a3e00640 1 -- 172.21.15.45:0/2035397149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f469c0d0e80 msgr2=0x7f469c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:01.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.998+0000 7f46a3e00640 1 --2- 172.21.15.45:0/2035397149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f469c0d0e80 0x7f469c0b4e20 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f469c06e920 tx=0x7f468802b8b0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.998+0000 7f46a3e00640 1 -- 172.21.15.45:0/2035397149 shutdown_connections 2024-04-16T16:18:01.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.998+0000 7f46a3e00640 1 --2- 172.21.15.45:0/2035397149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f469c0d0e80 0x7f469c0b4e20 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:01.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.998+0000 7f46a3e00640 1 -- 172.21.15.45:0/2035397149 >> 172.21.15.45:0/2035397149 conn(0x7f469c0b3970 msgr2=0x7f469c0b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:01.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.998+0000 7f46a3e00640 1 -- 172.21.15.45:0/2035397149 shutdown_connections 2024-04-16T16:18:01.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:01.998+0000 7f46a3e00640 1 -- 172.21.15.45:0/2035397149 wait complete. 2024-04-16T16:18:02.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.000+0000 7f46a3e00640 1 Processor -- start 2024-04-16T16:18:02.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.000+0000 7f46a3e00640 1 -- start start 2024-04-16T16:18:02.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.001+0000 7f46a3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f469c0afda0 0x7f469c149490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:02.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.001+0000 7f46a3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f469c0d0e80 0x7f469c149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:02.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.001+0000 7f46a3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f469c15b920 0x7f469c152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:02.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.001+0000 7f46a3e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f469c06f000 con 0x7f469c0d0e80 2024-04-16T16:18:02.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.001+0000 7f46a3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f469c06ee80 con 0x7f469c0afda0 2024-04-16T16:18:02.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.001+0000 7f46a3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f469c06f180 con 0x7f469c15b920 2024-04-16T16:18:02.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.001+0000 7f46a0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f469c0d0e80 0x7f469c149dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:02.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.001+0000 7f46a0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f469c0d0e80 0x7f469c149dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57852/0 (socket says 172.21.15.45:57852) 2024-04-16T16:18:02.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.001+0000 7f46a0a00640 1 -- 172.21.15.45:0/2739673314 learned_addr learned my addr 172.21.15.45:0/2739673314 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:02.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.001+0000 7f46a0a00640 1 -- 172.21.15.45:0/2739673314 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f469c15b920 msgr2=0x7f469c152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:02.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.001+0000 7f46a0a00640 1 --2- 172.21.15.45:0/2739673314 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f469c15b920 0x7f469c152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:02.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.001+0000 7f46a0a00640 1 -- 172.21.15.45:0/2739673314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f469c0afda0 msgr2=0x7f469c149490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:02.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.001+0000 7f46a0a00640 1 --2- 172.21.15.45:0/2739673314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f469c0afda0 0x7f469c149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:02.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.001+0000 7f46a0a00640 1 -- 172.21.15.45:0/2739673314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469c1499d0 con 0x7f469c0d0e80 2024-04-16T16:18:02.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.002+0000 7f46a0a00640 1 --2- 172.21.15.45:0/2739673314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f469c0d0e80 0x7f469c149dc0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f4688009890 tx=0x7f46880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:02.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.002+0000 7f4699600640 1 -- 172.21.15.45:0/2739673314 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46880453f0 con 0x7f469c0d0e80 2024-04-16T16:18:02.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.002+0000 7f4699600640 1 -- 172.21.15.45:0/2739673314 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46880455b0 con 0x7f469c0d0e80 2024-04-16T16:18:02.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.002+0000 7f4699600640 1 -- 172.21.15.45:0/2739673314 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46880459a0 con 0x7f469c0d0e80 2024-04-16T16:18:02.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.002+0000 7f46a3e00640 1 -- 172.21.15.45:0/2739673314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f469c14a810 con 0x7f469c0d0e80 2024-04-16T16:18:02.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.002+0000 7f46a3e00640 1 -- 172.21.15.45:0/2739673314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f469c14add0 con 0x7f469c0d0e80 2024-04-16T16:18:02.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.004+0000 7f46a3e00640 1 -- 172.21.15.45:0/2739673314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f469c075780 con 0x7f469c0d0e80 2024-04-16T16:18:02.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.004+0000 7f4699600640 1 -- 172.21.15.45:0/2739673314 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f4688038020 con 0x7f469c0d0e80 2024-04-16T16:18:02.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.006+0000 7f4699600640 1 --2- 172.21.15.45:0/2739673314 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f467c06ca40 0x7f467c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:02.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.006+0000 7f4699600640 1 -- 172.21.15.45:0/2739673314 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6655+0+0 (secure 0 0 0) 0x7f46880ae2e0 con 0x7f469c0d0e80 2024-04-16T16:18:02.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.007+0000 7f46a1400640 1 --2- 172.21.15.45:0/2739673314 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f467c06ca40 0x7f467c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:02.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.008+0000 7f46a1400640 1 --2- 172.21.15.45:0/2739673314 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f467c06ca40 0x7f467c06ef00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4690006fd0 tx=0x7f4690006bc0 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:02.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.011+0000 7f4699600640 1 -- 172.21.15.45:0/2739673314 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f469c1499d0 con 0x7f469c0d0e80 2024-04-16T16:18:02.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.256+0000 7f46a3e00640 1 -- 172.21.15.45:0/2739673314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f469c0af8e0 con 0x7f469c0d0e80 2024-04-16T16:18:02.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.783+0000 7f4699600640 1 -- 172.21.15.45:0/2739673314 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v57) ==== 104+0+0 (secure 0 0 0) 0x7f4688040070 con 0x7f469c0d0e80 2024-04-16T16:18:02.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.787+0000 7f46a3e00640 1 -- 172.21.15.45:0/2739673314 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f467c06ca40 msgr2=0x7f467c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:02.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.787+0000 7f46a3e00640 1 --2- 172.21.15.45:0/2739673314 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f467c06ca40 0x7f467c06ef00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4690006fd0 tx=0x7f4690006bc0 comp rx=0 tx=0).stop 2024-04-16T16:18:02.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.787+0000 7f46a3e00640 1 -- 172.21.15.45:0/2739673314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f469c0d0e80 msgr2=0x7f469c149dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:02.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.787+0000 7f46a3e00640 1 --2- 172.21.15.45:0/2739673314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f469c0d0e80 0x7f469c149dc0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f4688009890 tx=0x7f46880046c0 comp rx=0 tx=0).stop 2024-04-16T16:18:02.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.787+0000 7f46a3e00640 1 -- 172.21.15.45:0/2739673314 shutdown_connections 2024-04-16T16:18:02.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.787+0000 7f46a3e00640 1 --2- 172.21.15.45:0/2739673314 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f467c06ca40 0x7f467c06ef00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:02.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.787+0000 7f46a3e00640 1 --2- 172.21.15.45:0/2739673314 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f469c15b920 0x7f469c152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:02.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.787+0000 7f46a3e00640 1 --2- 172.21.15.45:0/2739673314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f469c0d0e80 0x7f469c149dc0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:02.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.787+0000 7f46a3e00640 1 --2- 172.21.15.45:0/2739673314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f469c0afda0 0x7f469c149490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:02.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.787+0000 7f46a3e00640 1 -- 172.21.15.45:0/2739673314 >> 172.21.15.45:0/2739673314 conn(0x7f469c0b3970 msgr2=0x7f469c0721c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:02.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.787+0000 7f46a3e00640 1 -- 172.21.15.45:0/2739673314 shutdown_connections 2024-04-16T16:18:02.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:02.787+0000 7f46a3e00640 1 -- 172.21.15.45:0/2739673314 wait complete. 2024-04-16T16:18:02.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-16T16:18:03.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.023+0000 7f0a5a000640 1 Processor -- start 2024-04-16T16:18:03.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.023+0000 7f0a5a000640 1 -- start start 2024-04-16T16:18:03.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.025+0000 7f0a5a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a54160950 0x7f0a54167370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:03.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.025+0000 7f0a5a000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0a540730e0 0x7f0a540b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:03.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.025+0000 7f0a5a000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a541678b0 con 0x7f0a540d07c0 2024-04-16T16:18:03.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.025+0000 7f0a5a000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0a540b5260 con 0x7f0a540730e0 2024-04-16T16:18:03.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.025+0000 7f0a5a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0a540b5540 con 0x7f0a54160950 2024-04-16T16:18:03.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.025+0000 7f0a53e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0a540d07c0 0x7f0a540d0be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52362/0 (socket says 172.21.15.45:52362) 2024-04-16T16:18:03.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.025+0000 7f0a53e00640 1 -- 172.21.15.45:0/2735113834 learned_addr learned my addr 172.21.15.45:0/2735113834 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:03.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.025+0000 7f0a52a00640 1 --2- 172.21.15.45:0/2735113834 >> v2:172.21.15.103:3300/0 conn(0x7f0a540730e0 0x7f0a540b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:03.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a53400640 1 --2- 172.21.15.45:0/2735113834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a54160950 0x7f0a54167370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:03.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 -- 172.21.15.45:0/2735113834 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1113241703 0 0) 0x7f0a541678b0 con 0x7f0a540d07c0 2024-04-16T16:18:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 -- 172.21.15.45:0/2735113834 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a38003520 con 0x7f0a540d07c0 2024-04-16T16:18:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 -- 172.21.15.45:0/2735113834 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4193561620 0 0) 0x7f0a38003520 con 0x7f0a540d07c0 2024-04-16T16:18:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 -- 172.21.15.45:0/2735113834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a54160950 msgr2=0x7f0a54167370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 --2- 172.21.15.45:0/2735113834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a54160950 0x7f0a54167370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 -- 172.21.15.45:0/2735113834 >> v2:172.21.15.103:3300/0 conn(0x7f0a540730e0 msgr2=0x7f0a540b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 --2- 172.21.15.45:0/2735113834 >> v2:172.21.15.103:3300/0 conn(0x7f0a540730e0 0x7f0a540b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 -- 172.21.15.45:0/2735113834 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a54166b70 con 0x7f0a540d07c0 2024-04-16T16:18:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 -- 172.21.15.45:0/2735113834 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f0a4c001670 con 0x7f0a540d07c0 2024-04-16T16:18:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 -- 172.21.15.45:0/2735113834 >> v1:172.21.15.45:6789/0 conn(0x7f0a540d07c0 legacy=0x7f0a540d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 --2- 172.21.15.45:0/2735113834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a380039b0 0x7f0a38003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 --2- 172.21.15.45:0/2735113834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a38004800 0x7f0a38004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 --2- 172.21.15.45:0/2735113834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a54160950 0x7f0a380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 -- 172.21.15.45:0/2735113834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0a540b5260 con 0x7f0a38004800 2024-04-16T16:18:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 -- 172.21.15.45:0/2735113834 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0a540b5540 con 0x7f0a380039b0 2024-04-16T16:18:03.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.026+0000 7f0a52000640 1 -- 172.21.15.45:0/2735113834 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0a3800b9c0 con 0x7f0a54160950 2024-04-16T16:18:03.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.027+0000 7f0a52a00640 1 --2- 172.21.15.45:0/2735113834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a38004800 0x7f0a38004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:03.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.027+0000 7f0a53400640 1 --2- 172.21.15.45:0/2735113834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a380039b0 0x7f0a38003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:03.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.027+0000 7f0a53e00640 1 --2- 172.21.15.45:0/2735113834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a54160950 0x7f0a380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:03.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.027+0000 7f0a52a00640 1 -- 172.21.15.45:0/2735113834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a54160950 msgr2=0x7f0a380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:03.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.027+0000 7f0a52a00640 1 --2- 172.21.15.45:0/2735113834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a54160950 0x7f0a380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:03.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.027+0000 7f0a52a00640 1 -- 172.21.15.45:0/2735113834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a380039b0 msgr2=0x7f0a38003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:03.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.027+0000 7f0a52a00640 1 --2- 172.21.15.45:0/2735113834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a380039b0 0x7f0a38003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:03.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.027+0000 7f0a52a00640 1 -- 172.21.15.45:0/2735113834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a38003520 con 0x7f0a38004800 2024-04-16T16:18:03.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.028+0000 7f0a52a00640 1 --2- 172.21.15.45:0/2735113834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a38004800 0x7f0a38004c80 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f0a40009a90 tx=0x7f0a38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d166f888b8f5183d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:03.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.029+0000 7f0a52000640 1 -- 172.21.15.45:0/2735113834 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a40039070 con 0x7f0a38004800 2024-04-16T16:18:03.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.029+0000 7f0a52000640 1 -- 172.21.15.45:0/2735113834 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a4002c6b0 con 0x7f0a38004800 2024-04-16T16:18:03.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.029+0000 7f0a52000640 1 -- 172.21.15.45:0/2735113834 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a40009690 con 0x7f0a38004800 2024-04-16T16:18:03.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.029+0000 7f0a5a000640 1 -- 172.21.15.45:0/2735113834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a38004800 msgr2=0x7f0a38004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.029+0000 7f0a5a000640 1 --2- 172.21.15.45:0/2735113834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a38004800 0x7f0a38004c80 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f0a40009a90 tx=0x7f0a38005830 comp rx=0 tx=0).stop 2024-04-16T16:18:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.029+0000 7f0a53e00640 1 -- 172.21.15.45:0/2735113834 reap_dead start 2024-04-16T16:18:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.029+0000 7f0a5a000640 1 -- 172.21.15.45:0/2735113834 shutdown_connections 2024-04-16T16:18:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.029+0000 7f0a5a000640 1 -- 172.21.15.45:0/2735113834 >> 172.21.15.45:0/2735113834 conn(0x7f0a540b3970 msgr2=0x7f0a540b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.029+0000 7f0a5a000640 1 -- 172.21.15.45:0/2735113834 shutdown_connections 2024-04-16T16:18:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.029+0000 7f0a5a000640 1 -- 172.21.15.45:0/2735113834 wait complete. 2024-04-16T16:18:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.031+0000 7f0a5a000640 1 Processor -- start 2024-04-16T16:18:03.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.031+0000 7f0a5a000640 1 -- start start 2024-04-16T16:18:03.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.032+0000 7f0a5a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a38004800 0x7f0a540c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:03.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.032+0000 7f0a5a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a540cdf70 0x7f0a540ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:03.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.032+0000 7f0a5a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a540c6460 0x7f0a540c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:03.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.032+0000 7f0a5a000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0a540b4e40 con 0x7f0a540cdf70 2024-04-16T16:18:03.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.032+0000 7f0a5a000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0a54160d30 con 0x7f0a38004800 2024-04-16T16:18:03.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.032+0000 7f0a5a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0a540b4fc0 con 0x7f0a540c6460 2024-04-16T16:18:03.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.032+0000 7f0a52a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a540cdf70 0x7f0a540ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:03.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.032+0000 7f0a52a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a540cdf70 0x7f0a540ce3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57866/0 (socket says 172.21.15.45:57866) 2024-04-16T16:18:03.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.032+0000 7f0a52a00640 1 -- 172.21.15.45:0/1328467181 learned_addr learned my addr 172.21.15.45:0/1328467181 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:03.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.033+0000 7f0a53e00640 1 --2- 172.21.15.45:0/1328467181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a540c6460 0x7f0a540c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:03.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.033+0000 7f0a53400640 1 --2- 172.21.15.45:0/1328467181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a38004800 0x7f0a540c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:03.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.033+0000 7f0a52a00640 1 -- 172.21.15.45:0/1328467181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a540c6460 msgr2=0x7f0a540c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:03.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.033+0000 7f0a52a00640 1 --2- 172.21.15.45:0/1328467181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a540c6460 0x7f0a540c68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:03.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.033+0000 7f0a52a00640 1 -- 172.21.15.45:0/1328467181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a38004800 msgr2=0x7f0a540c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:03.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.033+0000 7f0a52a00640 1 --2- 172.21.15.45:0/1328467181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a38004800 0x7f0a540c7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:03.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.033+0000 7f0a52a00640 1 -- 172.21.15.45:0/1328467181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a540cecf0 con 0x7f0a540cdf70 2024-04-16T16:18:03.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.033+0000 7f0a53400640 1 --2- 172.21.15.45:0/1328467181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a38004800 0x7f0a540c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:03.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.033+0000 7f0a52a00640 1 --2- 172.21.15.45:0/1328467181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a540cdf70 0x7f0a540ce3f0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f0a40000c00 tx=0x7f0a40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:03.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.034+0000 7f0a4b400640 1 -- 172.21.15.45:0/1328467181 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a40039050 con 0x7f0a540cdf70 2024-04-16T16:18:03.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.034+0000 7f0a4b400640 1 -- 172.21.15.45:0/1328467181 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a40045070 con 0x7f0a540cdf70 2024-04-16T16:18:03.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.034+0000 7f0a4b400640 1 -- 172.21.15.45:0/1328467181 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a400044e0 con 0x7f0a540cdf70 2024-04-16T16:18:03.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.034+0000 7f0a5a000640 1 -- 172.21.15.45:0/1328467181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a540c5980 con 0x7f0a540cdf70 2024-04-16T16:18:03.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.034+0000 7f0a5a000640 1 -- 172.21.15.45:0/1328467181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a540c5f40 con 0x7f0a540cdf70 2024-04-16T16:18:03.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.035+0000 7f0a5a000640 1 -- 172.21.15.45:0/1328467181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a5406e540 con 0x7f0a540cdf70 2024-04-16T16:18:03.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.036+0000 7f0a4b400640 1 -- 172.21.15.45:0/1328467181 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f0a40003a50 con 0x7f0a540cdf70 2024-04-16T16:18:03.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.037+0000 7f0a4b400640 1 --2- 172.21.15.45:0/1328467181 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f0a3406ca10 0x7f0a3406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:03.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.037+0000 7f0a4b400640 1 -- 172.21.15.45:0/1328467181 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6655+0+0 (secure 0 0 0) 0x7f0a400adce0 con 0x7f0a540cdf70 2024-04-16T16:18:03.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.038+0000 7f0a53400640 1 --2- 172.21.15.45:0/1328467181 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f0a3406ca10 0x7f0a3406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:03.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.039+0000 7f0a53400640 1 --2- 172.21.15.45:0/1328467181 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f0a3406ca10 0x7f0a3406eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0a44012430 tx=0x7f0a4400b040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:03.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.041+0000 7f0a4b400640 1 -- 172.21.15.45:0/1328467181 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a4004e020 con 0x7f0a540cdf70 2024-04-16T16:18:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:03.250+0000 7f0a5a000640 1 -- 172.21.15.45:0/1328467181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0a54165450 con 0x7f0a540cdf70 2024-04-16T16:18:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.157+0000 7f0a4b400640 1 -- 172.21.15.45:0/1328467181 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v76) ==== 181+0+0 (secure 0 0 0) 0x7f0a4007bc40 con 0x7f0a540cdf70 2024-04-16T16:18:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2024-04-16T16:18:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.162+0000 7f0a5a000640 1 -- 172.21.15.45:0/1328467181 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f0a3406ca10 msgr2=0x7f0a3406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.162+0000 7f0a5a000640 1 --2- 172.21.15.45:0/1328467181 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f0a3406ca10 0x7f0a3406eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0a44012430 tx=0x7f0a4400b040 comp rx=0 tx=0).stop 2024-04-16T16:18:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.162+0000 7f0a5a000640 1 -- 172.21.15.45:0/1328467181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a540cdf70 msgr2=0x7f0a540ce3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.162+0000 7f0a5a000640 1 --2- 172.21.15.45:0/1328467181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a540cdf70 0x7f0a540ce3f0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f0a40000c00 tx=0x7f0a40004730 comp rx=0 tx=0).stop 2024-04-16T16:18:04.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.164+0000 7f0a5a000640 1 -- 172.21.15.45:0/1328467181 shutdown_connections 2024-04-16T16:18:04.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.164+0000 7f0a5a000640 1 --2- 172.21.15.45:0/1328467181 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f0a3406ca10 0x7f0a3406eed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:04.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.164+0000 7f0a5a000640 1 --2- 172.21.15.45:0/1328467181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a540c6460 0x7f0a540c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:04.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.164+0000 7f0a5a000640 1 --2- 172.21.15.45:0/1328467181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a540cdf70 0x7f0a540ce3f0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:04.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.164+0000 7f0a5a000640 1 --2- 172.21.15.45:0/1328467181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a38004800 0x7f0a540c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:04.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.164+0000 7f0a5a000640 1 -- 172.21.15.45:0/1328467181 >> 172.21.15.45:0/1328467181 conn(0x7f0a540b3970 msgr2=0x7f0a540d0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:04.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.164+0000 7f0a5a000640 1 -- 172.21.15.45:0/1328467181 shutdown_connections 2024-04-16T16:18:04.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.164+0000 7f0a5a000640 1 -- 172.21.15.45:0/1328467181 wait complete. 2024-04-16T16:18:04.188 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-16T16:18:04.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.399+0000 7f131e600640 1 Processor -- start 2024-04-16T16:18:04.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.400+0000 7f131e600640 1 -- start start 2024-04-16T16:18:04.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.401+0000 7f131e600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1318155ab0 0x7f1318162ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:04.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.401+0000 7f131e600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f13180ba610 0x7f131815efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.401+0000 7f131e600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1318155e90 con 0x7f13180d18c0 2024-04-16T16:18:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.401+0000 7f131e600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f131815ff00 con 0x7f13180ba610 2024-04-16T16:18:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.401+0000 7f131e600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f131815f940 con 0x7f1318155ab0 2024-04-16T16:18:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.401+0000 7f1317e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f13180ba610 0x7f131815efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.402+0000 7f131dc00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f13180d18c0 0x7f13180d2a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46750/0 (socket says 172.21.15.45:46750) 2024-04-16T16:18:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.402+0000 7f131dc00640 1 -- 172.21.15.45:0/2173415422 learned_addr learned my addr 172.21.15.45:0/2173415422 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.402+0000 7f1317e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f13180ba610 0x7f131815efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39256/0 (socket says 172.21.15.45:39256) 2024-04-16T16:18:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.402+0000 7f131d200640 1 --2- 172.21.15.45:0/2173415422 >> v2:172.21.15.103:3300/0 conn(0x7f1318155ab0 0x7f1318162ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.402+0000 7f1317e00640 1 -- 172.21.15.45:0/2173415422 >> v2:172.21.15.103:3300/0 conn(0x7f1318155ab0 msgr2=0x7f1318162ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.402+0000 7f1317e00640 1 --2- 172.21.15.45:0/2173415422 >> v2:172.21.15.103:3300/0 conn(0x7f1318155ab0 0x7f1318162ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:04.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.402+0000 7f1317e00640 1 -- 172.21.15.45:0/2173415422 >> v1:172.21.15.45:6789/0 conn(0x7f13180d18c0 legacy=0x7f13180d2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:04.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.402+0000 7f1317e00640 1 -- 172.21.15.45:0/2173415422 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131815fc20 con 0x7f13180ba610 2024-04-16T16:18:04.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.402+0000 7f1317400640 1 -- 172.21.15.45:0/2173415422 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 249130849 0 0) 0x7f1318155e90 con 0x7f13180d18c0 2024-04-16T16:18:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.403+0000 7f1317e00640 1 --2- 172.21.15.45:0/2173415422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13180ba610 0x7f131815efd0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f13180d5990 tx=0x7f130402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdebb683848be31b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.404+0000 7f1317400640 1 -- 172.21.15.45:0/2173415422 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13040315a0 con 0x7f13180ba610 2024-04-16T16:18:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.404+0000 7f1317400640 1 -- 172.21.15.45:0/2173415422 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1304037070 con 0x7f13180ba610 2024-04-16T16:18:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.404+0000 7f1317400640 1 -- 172.21.15.45:0/2173415422 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1304031960 con 0x7f13180ba610 2024-04-16T16:18:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.404+0000 7f131e600640 1 -- 172.21.15.45:0/2173415422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13180ba610 msgr2=0x7f131815efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:04.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.404+0000 7f131e600640 1 --2- 172.21.15.45:0/2173415422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13180ba610 0x7f131815efd0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f13180d5990 tx=0x7f130402b840 comp rx=0 tx=0).stop 2024-04-16T16:18:04.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.404+0000 7f131e600640 1 -- 172.21.15.45:0/2173415422 shutdown_connections 2024-04-16T16:18:04.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.404+0000 7f131e600640 1 --2- 172.21.15.45:0/2173415422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13180ba610 0x7f131815efd0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:04.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.404+0000 7f131e600640 1 --2- 172.21.15.45:0/2173415422 >> v2:172.21.15.103:3300/0 conn(0x7f1318155ab0 0x7f1318162ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:04.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.404+0000 7f131e600640 1 -- 172.21.15.45:0/2173415422 >> 172.21.15.45:0/2173415422 conn(0x7f131806d1a0 msgr2=0x7f131806d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:04.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.404+0000 7f131e600640 1 -- 172.21.15.45:0/2173415422 shutdown_connections 2024-04-16T16:18:04.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.405+0000 7f131e600640 1 -- 172.21.15.45:0/2173415422 wait complete. 2024-04-16T16:18:04.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.406+0000 7f131e600640 1 Processor -- start 2024-04-16T16:18:04.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.406+0000 7f131e600640 1 -- start start 2024-04-16T16:18:04.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.407+0000 7f131e600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13180ba610 0x7f13180b7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:04.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.407+0000 7f131e600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13180d18c0 0x7f13180b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:04.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.407+0000 7f131e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1318155ab0 0x7f13180b84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:04.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.407+0000 7f131e600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1318163430 con 0x7f1318155ab0 2024-04-16T16:18:04.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.407+0000 7f131e600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f13181629d0 con 0x7f13180d18c0 2024-04-16T16:18:04.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.407+0000 7f131e600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f131815f510 con 0x7f13180ba610 2024-04-16T16:18:04.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.408+0000 7f131dc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1318155ab0 0x7f13180b84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.408+0000 7f131dc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1318155ab0 0x7f13180b84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39260/0 (socket says 172.21.15.45:39260) 2024-04-16T16:18:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.408+0000 7f131dc00640 1 -- 172.21.15.45:0/1343228687 learned_addr learned my addr 172.21.15.45:0/1343228687 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.408+0000 7f131d200640 1 --2- 172.21.15.45:0/1343228687 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13180ba610 0x7f13180b7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.408+0000 7f1317e00640 1 --2- 172.21.15.45:0/1343228687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13180d18c0 0x7f13180b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.408+0000 7f131dc00640 1 -- 172.21.15.45:0/1343228687 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13180ba610 msgr2=0x7f13180b7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.408+0000 7f131dc00640 1 --2- 172.21.15.45:0/1343228687 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13180ba610 0x7f13180b7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.408+0000 7f131dc00640 1 -- 172.21.15.45:0/1343228687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13180d18c0 msgr2=0x7f13180b7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.408+0000 7f131dc00640 1 --2- 172.21.15.45:0/1343228687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13180d18c0 0x7f13180b7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.408+0000 7f131dc00640 1 -- 172.21.15.45:0/1343228687 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13180b8ad0 con 0x7f1318155ab0 2024-04-16T16:18:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.409+0000 7f131dc00640 1 --2- 172.21.15.45:0/1343228687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1318155ab0 0x7f13180b84a0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f131000c7a0 tx=0x7f131000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.409+0000 7f1314c00640 1 -- 172.21.15.45:0/1343228687 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1310018070 con 0x7f1318155ab0 2024-04-16T16:18:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.409+0000 7f1314c00640 1 -- 172.21.15.45:0/1343228687 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f131000a640 con 0x7f1318155ab0 2024-04-16T16:18:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.409+0000 7f1314c00640 1 -- 172.21.15.45:0/1343228687 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131000aa70 con 0x7f1318155ab0 2024-04-16T16:18:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.409+0000 7f131e600640 1 -- 172.21.15.45:0/1343228687 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13180b8db0 con 0x7f1318155ab0 2024-04-16T16:18:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.409+0000 7f131e600640 1 -- 172.21.15.45:0/1343228687 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13181e89d0 con 0x7f1318155ab0 2024-04-16T16:18:04.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.411+0000 7f131e600640 1 -- 172.21.15.45:0/1343228687 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1318075780 con 0x7f1318155ab0 2024-04-16T16:18:04.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.412+0000 7f1314c00640 1 -- 172.21.15.45:0/1343228687 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f131001b070 con 0x7f1318155ab0 2024-04-16T16:18:04.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.412+0000 7f1314c00640 1 --2- 172.21.15.45:0/1343228687 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f12f806ca10 0x7f12f806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:04.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.413+0000 7f1314c00640 1 -- 172.21.15.45:0/1343228687 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6217+0+0 (secure 0 0 0) 0x7f131008b490 con 0x7f1318155ab0 2024-04-16T16:18:04.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.413+0000 7f131d200640 1 --2- 172.21.15.45:0/1343228687 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f12f806ca10 0x7f12f806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:04.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.414+0000 7f131d200640 1 --2- 172.21.15.45:0/1343228687 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f12f806ca10 0x7f12f806eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f130c006fd0 tx=0x7f130c008040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:04.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.417+0000 7f1314c00640 1 -- 172.21.15.45:0/1343228687 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1310059640 con 0x7f1318155ab0 2024-04-16T16:18:04.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:04.625+0000 7f131e600640 1 -- 172.21.15.45:0/1343228687 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1318155f20 con 0x7f1318155ab0 2024-04-16T16:18:05.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.170+0000 7f1314c00640 1 -- 172.21.15.45:0/1343228687 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v77) ==== 169+0+0 (secure 0 0 0) 0x7f1310058f40 con 0x7f1318155ab0 2024-04-16T16:18:05.171 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2024-04-16T16:18:05.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.174+0000 7f131e600640 1 -- 172.21.15.45:0/1343228687 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f12f806ca10 msgr2=0x7f12f806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:05.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.174+0000 7f131e600640 1 --2- 172.21.15.45:0/1343228687 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f12f806ca10 0x7f12f806eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f130c006fd0 tx=0x7f130c008040 comp rx=0 tx=0).stop 2024-04-16T16:18:05.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.174+0000 7f131e600640 1 -- 172.21.15.45:0/1343228687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1318155ab0 msgr2=0x7f13180b84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:05.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.174+0000 7f131e600640 1 --2- 172.21.15.45:0/1343228687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1318155ab0 0x7f13180b84a0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f131000c7a0 tx=0x7f131000cc60 comp rx=0 tx=0).stop 2024-04-16T16:18:05.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.174+0000 7f131e600640 1 -- 172.21.15.45:0/1343228687 shutdown_connections 2024-04-16T16:18:05.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.174+0000 7f131e600640 1 --2- 172.21.15.45:0/1343228687 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f12f806ca10 0x7f12f806eed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.174+0000 7f131e600640 1 --2- 172.21.15.45:0/1343228687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1318155ab0 0x7f13180b84a0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.174+0000 7f131e600640 1 --2- 172.21.15.45:0/1343228687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13180d18c0 0x7f13180b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.174+0000 7f131e600640 1 --2- 172.21.15.45:0/1343228687 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13180ba610 0x7f13180b7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.174+0000 7f131e600640 1 -- 172.21.15.45:0/1343228687 >> 172.21.15.45:0/1343228687 conn(0x7f131806d1a0 msgr2=0x7f13180d1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:05.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.174+0000 7f131e600640 1 -- 172.21.15.45:0/1343228687 shutdown_connections 2024-04-16T16:18:05.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.174+0000 7f131e600640 1 -- 172.21.15.45:0/1343228687 wait complete. 2024-04-16T16:18:05.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-16T16:18:05.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.414+0000 7f6796c00640 1 Processor -- start 2024-04-16T16:18:05.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.414+0000 7f6796c00640 1 -- start start 2024-04-16T16:18:05.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.416+0000 7f6796c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f67900b4930 0x7f6790072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:05.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.416+0000 7f6796c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67900b5530 con 0x7f679015c7c0 2024-04-16T16:18:05.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.416+0000 7f6796c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67900b5810 con 0x7f67900d10e0 2024-04-16T16:18:05.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.416+0000 7f6796c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f67900b52e0 con 0x7f67900b4930 2024-04-16T16:18:05.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.416+0000 7f678fe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f679015c7c0 0x7f67900d09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46764/0 (socket says 172.21.15.45:46764) 2024-04-16T16:18:05.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.416+0000 7f678fe00640 1 -- 172.21.15.45:0/2531764880 learned_addr learned my addr 172.21.15.45:0/2531764880 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:05.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.416+0000 7f678ea00640 1 -- 172.21.15.45:0/2531764880 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2092966719 0 0) 0x7f67900b5530 con 0x7f679015c7c0 2024-04-16T16:18:05.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.416+0000 7f6794c00640 1 --2- 172.21.15.45:0/2531764880 >> v2:172.21.15.103:3300/0 conn(0x7f67900b4930 0x7f6790072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:05.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.416+0000 7f678ea00640 1 -- 172.21.15.45:0/2531764880 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6774003520 con 0x7f679015c7c0 2024-04-16T16:18:05.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.417+0000 7f678ea00640 1 -- 172.21.15.45:0/2531764880 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1747020376 0 0) 0x7f6774003520 con 0x7f679015c7c0 2024-04-16T16:18:05.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.417+0000 7f678ea00640 1 -- 172.21.15.45:0/2531764880 >> v2:172.21.15.103:3300/0 conn(0x7f67900b4930 msgr2=0x7f6790072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:05.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.417+0000 7f678ea00640 1 --2- 172.21.15.45:0/2531764880 >> v2:172.21.15.103:3300/0 conn(0x7f67900b4930 0x7f6790072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.417+0000 7f678ea00640 1 -- 172.21.15.45:0/2531764880 >> v1:172.21.15.103:6789/0 conn(0x7f67900d10e0 legacy=0x7f679006f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:05.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.417+0000 7f678ea00640 1 -- 172.21.15.45:0/2531764880 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67900d04b0 con 0x7f679015c7c0 2024-04-16T16:18:05.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.417+0000 7f678ea00640 1 -- 172.21.15.45:0/2531764880 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f6780001670 con 0x7f679015c7c0 2024-04-16T16:18:05.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.417+0000 7f678ea00640 1 -- 172.21.15.45:0/2531764880 >> v1:172.21.15.45:6789/0 conn(0x7f679015c7c0 legacy=0x7f67900d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:05.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.417+0000 7f678ea00640 1 --2- 172.21.15.45:0/2531764880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6774003a00 0x7f6774003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:05.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.417+0000 7f678ea00640 1 --2- 172.21.15.45:0/2531764880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6774004800 0x7f6774004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:05.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.417+0000 7f678ea00640 1 --2- 172.21.15.45:0/2531764880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67740053c0 0x7f677400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:05.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.417+0000 7f678ea00640 1 -- 172.21.15.45:0/2531764880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f67900b52e0 con 0x7f67740053c0 2024-04-16T16:18:05.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.417+0000 7f678ea00640 1 -- 172.21.15.45:0/2531764880 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f677400c280 con 0x7f6774004800 2024-04-16T16:18:05.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.417+0000 7f678ea00640 1 -- 172.21.15.45:0/2531764880 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f677400c480 con 0x7f6774003a00 2024-04-16T16:18:05.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.417+0000 7f6794c00640 1 --2- 172.21.15.45:0/2531764880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67740053c0 0x7f677400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:05.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.418+0000 7f678f400640 1 --2- 172.21.15.45:0/2531764880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6774004800 0x7f6774004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:05.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.418+0000 7f678fe00640 1 --2- 172.21.15.45:0/2531764880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6774003a00 0x7f6774003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:05.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.418+0000 7f6794c00640 1 -- 172.21.15.45:0/2531764880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6774003a00 msgr2=0x7f6774003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:05.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.418+0000 7f6794c00640 1 --2- 172.21.15.45:0/2531764880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6774003a00 0x7f6774003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.418+0000 7f6794c00640 1 -- 172.21.15.45:0/2531764880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6774004800 msgr2=0x7f6774004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:05.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.418+0000 7f6794c00640 1 --2- 172.21.15.45:0/2531764880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6774004800 0x7f6774004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.418+0000 7f6794c00640 1 -- 172.21.15.45:0/2531764880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6774003520 con 0x7f67740053c0 2024-04-16T16:18:05.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.418+0000 7f678f400640 1 --2- 172.21.15.45:0/2531764880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6774004800 0x7f6774004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:05.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.418+0000 7f678fe00640 1 --2- 172.21.15.45:0/2531764880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6774003a00 0x7f6774003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:05.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.419+0000 7f6794c00640 1 -- 172.21.15.45:0/2531764880 reap_dead start 2024-04-16T16:18:05.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.419+0000 7f6794c00640 1 --2- 172.21.15.45:0/2531764880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67740053c0 0x7f677400b9c0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f6788007360 tx=0x7f6774005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b370ba7f0e07aaf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:05.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.419+0000 7f678ea00640 1 -- 172.21.15.45:0/2531764880 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6788009be0 con 0x7f67740053c0 2024-04-16T16:18:05.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.419+0000 7f678ea00640 1 -- 172.21.15.45:0/2531764880 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f678802cd30 con 0x7f67740053c0 2024-04-16T16:18:05.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.419+0000 7f678ea00640 1 -- 172.21.15.45:0/2531764880 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6788003bd0 con 0x7f67740053c0 2024-04-16T16:18:05.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.420+0000 7f6796c00640 1 -- 172.21.15.45:0/2531764880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67740053c0 msgr2=0x7f677400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:05.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.420+0000 7f6796c00640 1 --2- 172.21.15.45:0/2531764880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67740053c0 0x7f677400b9c0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f6788007360 tx=0x7f6774005830 comp rx=0 tx=0).stop 2024-04-16T16:18:05.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.420+0000 7f6796c00640 1 -- 172.21.15.45:0/2531764880 shutdown_connections 2024-04-16T16:18:05.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.420+0000 7f6796c00640 1 --2- 172.21.15.45:0/2531764880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67740053c0 0x7f677400b9c0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.420+0000 7f6796c00640 1 -- 172.21.15.45:0/2531764880 >> 172.21.15.45:0/2531764880 conn(0x7f67900b3970 msgr2=0x7f67900b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:05.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.420+0000 7f6796c00640 1 -- 172.21.15.45:0/2531764880 shutdown_connections 2024-04-16T16:18:05.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.420+0000 7f6796c00640 1 -- 172.21.15.45:0/2531764880 wait complete. 2024-04-16T16:18:05.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.422+0000 7f6796c00640 1 Processor -- start 2024-04-16T16:18:05.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.422+0000 7f6796c00640 1 -- start start 2024-04-16T16:18:05.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.423+0000 7f6796c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67740053c0 0x7f67901573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:05.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.423+0000 7f6796c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790149360 0x7f679006b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:05.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.423+0000 7f6796c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f679006be70 0x7f67901f7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:05.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.423+0000 7f6796c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f67900731a0 con 0x7f6790149360 2024-04-16T16:18:05.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.423+0000 7f6796c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6790073020 con 0x7f679006be70 2024-04-16T16:18:05.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.423+0000 7f6796c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6790073320 con 0x7f67740053c0 2024-04-16T16:18:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.424+0000 7f678f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790149360 0x7f679006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.424+0000 7f678f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790149360 0x7f679006b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39282/0 (socket says 172.21.15.45:39282) 2024-04-16T16:18:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.424+0000 7f678f400640 1 -- 172.21.15.45:0/1780731334 learned_addr learned my addr 172.21.15.45:0/1780731334 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.424+0000 7f6794c00640 1 --2- 172.21.15.45:0/1780731334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f679006be70 0x7f67901f7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.424+0000 7f678fe00640 1 --2- 172.21.15.45:0/1780731334 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67740053c0 0x7f67901573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.424+0000 7f678f400640 1 -- 172.21.15.45:0/1780731334 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67740053c0 msgr2=0x7f67901573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.424+0000 7f678f400640 1 --2- 172.21.15.45:0/1780731334 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67740053c0 0x7f67901573c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.424+0000 7f678f400640 1 -- 172.21.15.45:0/1780731334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f679006be70 msgr2=0x7f67901f7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:05.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.424+0000 7f678f400640 1 --2- 172.21.15.45:0/1780731334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f679006be70 0x7f67901f7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.424+0000 7f678f400640 1 -- 172.21.15.45:0/1780731334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67901f8c80 con 0x7f6790149360 2024-04-16T16:18:05.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.424+0000 7f678f400640 1 --2- 172.21.15.45:0/1780731334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790149360 0x7f679006b930 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f677c00d5c0 tx=0x7f677c00da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:05.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.425+0000 7f6787e00640 1 -- 172.21.15.45:0/1780731334 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c016070 con 0x7f6790149360 2024-04-16T16:18:05.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.425+0000 7f6787e00640 1 -- 172.21.15.45:0/1780731334 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f677c007240 con 0x7f6790149360 2024-04-16T16:18:05.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.425+0000 7f6787e00640 1 -- 172.21.15.45:0/1780731334 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c007630 con 0x7f6790149360 2024-04-16T16:18:05.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.425+0000 7f6796c00640 1 -- 172.21.15.45:0/1780731334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67901f89a0 con 0x7f6790149360 2024-04-16T16:18:05.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.425+0000 7f6796c00640 1 -- 172.21.15.45:0/1780731334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6790156bc0 con 0x7f6790149360 2024-04-16T16:18:05.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.426+0000 7f6796c00640 1 -- 172.21.15.45:0/1780731334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67901566d0 con 0x7f6790149360 2024-04-16T16:18:05.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.427+0000 7f6787e00640 1 -- 172.21.15.45:0/1780731334 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f677c0067d0 con 0x7f6790149360 2024-04-16T16:18:05.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.428+0000 7f6787e00640 1 --2- 172.21.15.45:0/1780731334 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f677006ca10 0x7f677006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:05.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.428+0000 7f6787e00640 1 -- 172.21.15.45:0/1780731334 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5803+0+0 (secure 0 0 0) 0x7f677c08bc70 con 0x7f6790149360 2024-04-16T16:18:05.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.429+0000 7f678fe00640 1 --2- 172.21.15.45:0/1780731334 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f677006ca10 0x7f677006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:05.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.430+0000 7f678fe00640 1 --2- 172.21.15.45:0/1780731334 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f677006ca10 0x7f677006eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6780001eb0 tx=0x7f6780009040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:05.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.432+0000 7f6787e00640 1 -- 172.21.15.45:0/1780731334 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f677c0597c0 con 0x7f6790149360 2024-04-16T16:18:05.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.636+0000 7f6796c00640 1 -- 172.21.15.45:0/1780731334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f6790075780 con 0x7f6790149360 2024-04-16T16:18:05.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.644+0000 7f6787e00640 1 -- 172.21.15.45:0/1780731334 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7f677c05dac0 con 0x7f6790149360 2024-04-16T16:18:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.646+0000 7f6796c00640 1 -- 172.21.15.45:0/1780731334 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f677006ca10 msgr2=0x7f677006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.646+0000 7f6796c00640 1 --2- 172.21.15.45:0/1780731334 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f677006ca10 0x7f677006eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6780001eb0 tx=0x7f6780009040 comp rx=0 tx=0).stop 2024-04-16T16:18:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.646+0000 7f6796c00640 1 -- 172.21.15.45:0/1780731334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790149360 msgr2=0x7f679006b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.646+0000 7f6796c00640 1 --2- 172.21.15.45:0/1780731334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790149360 0x7f679006b930 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f677c00d5c0 tx=0x7f677c00da80 comp rx=0 tx=0).stop 2024-04-16T16:18:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.647+0000 7f6796c00640 1 -- 172.21.15.45:0/1780731334 shutdown_connections 2024-04-16T16:18:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.647+0000 7f6796c00640 1 --2- 172.21.15.45:0/1780731334 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f677006ca10 0x7f677006eed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.647+0000 7f6796c00640 1 --2- 172.21.15.45:0/1780731334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f679006be70 0x7f67901f7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.647+0000 7f6796c00640 1 --2- 172.21.15.45:0/1780731334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790149360 0x7f679006b930 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.647+0000 7f6796c00640 1 --2- 172.21.15.45:0/1780731334 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67740053c0 0x7f67901573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.647+0000 7f6796c00640 1 -- 172.21.15.45:0/1780731334 >> 172.21.15.45:0/1780731334 conn(0x7f67900b3970 msgr2=0x7f679015a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.647+0000 7f6796c00640 1 -- 172.21.15.45:0/1780731334 shutdown_connections 2024-04-16T16:18:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.647+0000 7f6796c00640 1 -- 172.21.15.45:0/1780731334 wait complete. 2024-04-16T16:18:05.667 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_delayed_metrics' 2024-04-16T16:18:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.883+0000 7f616ac00640 1 Processor -- start 2024-04-16T16:18:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.883+0000 7f616ac00640 1 -- start start 2024-04-16T16:18:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.885+0000 7f616ac00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f616415ccb0 0x7f616406df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.885+0000 7f616ac00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f616415d360 con 0x7f616406e480 2024-04-16T16:18:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.885+0000 7f616ac00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f616415d640 con 0x7f6164146220 2024-04-16T16:18:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.885+0000 7f616ac00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f616415d130 con 0x7f616415ccb0 2024-04-16T16:18:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.885+0000 7f616a200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f616406e480 0x7f61641608a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46776/0 (socket says 172.21.15.45:46776) 2024-04-16T16:18:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.885+0000 7f616a200640 1 -- 172.21.15.45:0/349383584 learned_addr learned my addr 172.21.15.45:0/349383584 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:05.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.885+0000 7f6168e00640 1 --2- 172.21.15.45:0/349383584 >> v2:172.21.15.45:3300/0 conn(0x7f616415ccb0 0x7f616406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.885+0000 7f6163e00640 1 -- 172.21.15.45:0/349383584 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3189048926 0 0) 0x7f616415d360 con 0x7f616406e480 2024-04-16T16:18:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.886+0000 7f6163e00640 1 -- 172.21.15.45:0/349383584 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f614c003520 con 0x7f616406e480 2024-04-16T16:18:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.886+0000 7f6168e00640 1 -- 172.21.15.45:0/349383584 >> v1:172.21.15.103:6789/0 conn(0x7f6164146220 legacy=0x7f616406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.886+0000 7f6168e00640 1 -- 172.21.15.45:0/349383584 >> v1:172.21.15.45:6789/0 conn(0x7f616406e480 legacy=0x7f61641608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.886+0000 7f6168e00640 1 -- 172.21.15.45:0/349383584 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f616415d920 con 0x7f616415ccb0 2024-04-16T16:18:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.887+0000 7f6168e00640 1 --2- 172.21.15.45:0/349383584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616415ccb0 0x7f616406df40 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f61500097a0 tx=0x7f615002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20a9514415c0dc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.887+0000 7f6163e00640 1 -- 172.21.15.45:0/349383584 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150035020 con 0x7f616415ccb0 2024-04-16T16:18:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.887+0000 7f6163e00640 1 -- 172.21.15.45:0/349383584 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6150031830 con 0x7f616415ccb0 2024-04-16T16:18:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.887+0000 7f6163e00640 1 -- 172.21.15.45:0/349383584 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150031c00 con 0x7f616415ccb0 2024-04-16T16:18:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.888+0000 7f616ac00640 1 -- 172.21.15.45:0/349383584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616415ccb0 msgr2=0x7f616406df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.888+0000 7f616ac00640 1 --2- 172.21.15.45:0/349383584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616415ccb0 0x7f616406df40 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f61500097a0 tx=0x7f615002b8d0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.888+0000 7f616ac00640 1 -- 172.21.15.45:0/349383584 shutdown_connections 2024-04-16T16:18:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.888+0000 7f616ac00640 1 --2- 172.21.15.45:0/349383584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616415ccb0 0x7f616406df40 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.888+0000 7f616ac00640 1 -- 172.21.15.45:0/349383584 >> 172.21.15.45:0/349383584 conn(0x7f616406b930 msgr2=0x7f616406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.888+0000 7f616ac00640 1 -- 172.21.15.45:0/349383584 shutdown_connections 2024-04-16T16:18:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.888+0000 7f616ac00640 1 -- 172.21.15.45:0/349383584 wait complete. 2024-04-16T16:18:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.890+0000 7f616ac00640 1 Processor -- start 2024-04-16T16:18:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.890+0000 7f616ac00640 1 -- start start 2024-04-16T16:18:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.891+0000 7f616ac00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616406e480 0x7f6164141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.891+0000 7f616ac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6164146220 0x7f616413e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.891+0000 7f616ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f616415ccb0 0x7f6164144880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.891+0000 7f616ac00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6164160fb0 con 0x7f616406e480 2024-04-16T16:18:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.891+0000 7f616ac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6164160410 con 0x7f6164146220 2024-04-16T16:18:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.891+0000 7f616ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6164161130 con 0x7f616415ccb0 2024-04-16T16:18:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.891+0000 7f6169800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616406e480 0x7f6164141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.891+0000 7f6169800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616406e480 0x7f6164141c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39292/0 (socket says 172.21.15.45:39292) 2024-04-16T16:18:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.891+0000 7f6169800640 1 -- 172.21.15.45:0/2628295850 learned_addr learned my addr 172.21.15.45:0/2628295850 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.892+0000 7f6168e00640 1 --2- 172.21.15.45:0/2628295850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6164146220 0x7f616413e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.892+0000 7f6169800640 1 -- 172.21.15.45:0/2628295850 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f616415ccb0 msgr2=0x7f6164144880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.892+0000 7f6169800640 1 --2- 172.21.15.45:0/2628295850 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f616415ccb0 0x7f6164144880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.892+0000 7f6169800640 1 -- 172.21.15.45:0/2628295850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6164146220 msgr2=0x7f616413e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.892+0000 7f6169800640 1 --2- 172.21.15.45:0/2628295850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6164146220 0x7f616413e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.892+0000 7f6169800640 1 -- 172.21.15.45:0/2628295850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6164145160 con 0x7f616406e480 2024-04-16T16:18:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.892+0000 7f6169800640 1 --2- 172.21.15.45:0/2628295850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616406e480 0x7f6164141c30 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f615800c460 tx=0x7f615800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.892+0000 7f6161600640 1 -- 172.21.15.45:0/2628295850 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6158016020 con 0x7f616406e480 2024-04-16T16:18:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.892+0000 7f6161600640 1 -- 172.21.15.45:0/2628295850 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6158012870 con 0x7f616406e480 2024-04-16T16:18:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.892+0000 7f6161600640 1 -- 172.21.15.45:0/2628295850 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6158012c60 con 0x7f616406e480 2024-04-16T16:18:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.892+0000 7f616ac00640 1 -- 172.21.15.45:0/2628295850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6164145440 con 0x7f616406e480 2024-04-16T16:18:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.892+0000 7f616ac00640 1 -- 172.21.15.45:0/2628295850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6164140210 con 0x7f616406e480 2024-04-16T16:18:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.894+0000 7f616ac00640 1 -- 172.21.15.45:0/2628295850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6164140410 con 0x7f616406e480 2024-04-16T16:18:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.894+0000 7f6161600640 1 -- 172.21.15.45:0/2628295850 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f6158019020 con 0x7f616406e480 2024-04-16T16:18:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.895+0000 7f6161600640 1 --2- 172.21.15.45:0/2628295850 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f614406ca10 0x7f614406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.895+0000 7f6161600640 1 -- 172.21.15.45:0/2628295850 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5803+0+0 (secure 0 0 0) 0x7f615808bee0 con 0x7f616406e480 2024-04-16T16:18:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.896+0000 7f6168e00640 1 --2- 172.21.15.45:0/2628295850 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f614406ca10 0x7f614406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.897+0000 7f6168e00640 1 --2- 172.21.15.45:0/2628295850 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f614406ca10 0x7f614406eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6150036000 tx=0x7f6150008030 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:05.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:05.901+0000 7f6161600640 1 -- 172.21.15.45:0/2628295850 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6164140410 con 0x7f616406e480 2024-04-16T16:18:06.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:06.177+0000 7f616ac00640 1 -- 172.21.15.45:0/2628295850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_delayed_metrics"]} v 0) -- 0x7f616415c9a0 con 0x7f616406e480 2024-04-16T16:18:07.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.184+0000 7f6161600640 1 -- 172.21.15.45:0/2628295850 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_delayed_metrics"]}]=0 v276) ==== 144+0+0 (secure 0 0 0) 0x7f6158020020 con 0x7f616406e480 2024-04-16T16:18:07.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.188+0000 7f616ac00640 1 -- 172.21.15.45:0/2628295850 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f614406ca10 msgr2=0x7f614406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:07.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.188+0000 7f616ac00640 1 --2- 172.21.15.45:0/2628295850 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f614406ca10 0x7f614406eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6150036000 tx=0x7f6150008030 comp rx=0 tx=0).stop 2024-04-16T16:18:07.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.188+0000 7f616ac00640 1 -- 172.21.15.45:0/2628295850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616406e480 msgr2=0x7f6164141c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:07.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.188+0000 7f616ac00640 1 --2- 172.21.15.45:0/2628295850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616406e480 0x7f6164141c30 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f615800c460 tx=0x7f615800c920 comp rx=0 tx=0).stop 2024-04-16T16:18:07.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.188+0000 7f616ac00640 1 -- 172.21.15.45:0/2628295850 shutdown_connections 2024-04-16T16:18:07.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.188+0000 7f616ac00640 1 --2- 172.21.15.45:0/2628295850 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f614406ca10 0x7f614406eed0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.188+0000 7f616ac00640 1 --2- 172.21.15.45:0/2628295850 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f616415ccb0 0x7f6164144880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.188+0000 7f616ac00640 1 --2- 172.21.15.45:0/2628295850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6164146220 0x7f616413e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.188+0000 7f616ac00640 1 --2- 172.21.15.45:0/2628295850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616406e480 0x7f6164141c30 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.188+0000 7f616ac00640 1 -- 172.21.15.45:0/2628295850 >> 172.21.15.45:0/2628295850 conn(0x7f616406b930 msgr2=0x7f616415f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:07.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.188+0000 7f616ac00640 1 -- 172.21.15.45:0/2628295850 shutdown_connections 2024-04-16T16:18:07.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.188+0000 7f616ac00640 1 -- 172.21.15.45:0/2628295850 wait complete. 2024-04-16T16:18:07.211 INFO:tasks.cephfs_test_runner:test_delayed_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics) ... ok 2024-04-16T16:18:07.212 INFO:tasks.cephfs_test_runner:Starting test: test_for_invalid_client_id (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2024-04-16T16:18:07.212 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:18:07.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.442+0000 7f7e8ca00640 1 Processor -- start 2024-04-16T16:18:07.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.442+0000 7f7e8ca00640 1 -- start start 2024-04-16T16:18:07.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.443+0000 7f7e8ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7e88162270 0x7f7e880d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:07.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.443+0000 7f7e8ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e880d52d0 0x7f7e8806ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:07.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.443+0000 7f7e8ca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e880d3080 con 0x7f7e88161e90 2024-04-16T16:18:07.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.444+0000 7f7e8ca00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f7e880d2e30 con 0x7f7e88162270 2024-04-16T16:18:07.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.444+0000 7f7e8ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7e880d3360 con 0x7f7e880d52d0 2024-04-16T16:18:07.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.444+0000 7f7e86a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7e88162270 0x7f7e880d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:07.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.444+0000 7f7e86a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7e88162270 0x7f7e880d3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39308/0 (socket says 172.21.15.45:39308) 2024-04-16T16:18:07.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.444+0000 7f7e86a00640 1 -- 172.21.15.45:0/3189620228 learned_addr learned my addr 172.21.15.45:0/3189620228 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:07.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.444+0000 7f7e87e00640 1 --2- 172.21.15.45:0/3189620228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e880d52d0 0x7f7e8806ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:07.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.444+0000 7f7e86a00640 1 -- 172.21.15.45:0/3189620228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e880d52d0 msgr2=0x7f7e8806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:07.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.444+0000 7f7e86a00640 1 --2- 172.21.15.45:0/3189620228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e880d52d0 0x7f7e8806ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.444+0000 7f7e86a00640 1 -- 172.21.15.45:0/3189620228 >> v1:172.21.15.103:6789/0 conn(0x7f7e88161e90 legacy=0x7f7e880cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:07.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.444+0000 7f7e86a00640 1 -- 172.21.15.45:0/3189620228 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e880d3640 con 0x7f7e88162270 2024-04-16T16:18:07.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.446+0000 7f7e86a00640 1 --2- 172.21.15.45:0/3189620228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e88162270 0x7f7e880d3e40 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f7e78009e80 tx=0x7f7e7802bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=252123a530d26614 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:07.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.446+0000 7f7e86000640 1 -- 172.21.15.45:0/3189620228 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7802cb70 con 0x7f7e88162270 2024-04-16T16:18:07.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.446+0000 7f7e86000640 1 -- 172.21.15.45:0/3189620228 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e7802cd10 con 0x7f7e88162270 2024-04-16T16:18:07.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.446+0000 7f7e86000640 1 -- 172.21.15.45:0/3189620228 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e78034610 con 0x7f7e88162270 2024-04-16T16:18:07.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.447+0000 7f7e8ca00640 1 -- 172.21.15.45:0/3189620228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e88162270 msgr2=0x7f7e880d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:07.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.447+0000 7f7e8ca00640 1 --2- 172.21.15.45:0/3189620228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e88162270 0x7f7e880d3e40 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f7e78009e80 tx=0x7f7e7802bfd0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.447+0000 7f7e8ca00640 1 -- 172.21.15.45:0/3189620228 shutdown_connections 2024-04-16T16:18:07.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.447+0000 7f7e8ca00640 1 --2- 172.21.15.45:0/3189620228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e880d52d0 0x7f7e8806ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.447+0000 7f7e8ca00640 1 --2- 172.21.15.45:0/3189620228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e88162270 0x7f7e880d3e40 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.447+0000 7f7e8ca00640 1 -- 172.21.15.45:0/3189620228 >> 172.21.15.45:0/3189620228 conn(0x7f7e880d1940 msgr2=0x7f7e880d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:07.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.447+0000 7f7e8ca00640 1 -- 172.21.15.45:0/3189620228 shutdown_connections 2024-04-16T16:18:07.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.447+0000 7f7e8ca00640 1 -- 172.21.15.45:0/3189620228 wait complete. 2024-04-16T16:18:07.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.448+0000 7f7e8ca00640 1 Processor -- start 2024-04-16T16:18:07.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.449+0000 7f7e8ca00640 1 -- start start 2024-04-16T16:18:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e8ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e880d52d0 0x7f7e880c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e8ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e88161e90 0x7f7e881ea640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e8ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e88162270 0x7f7e8814fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e8ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7e880d4500 con 0x7f7e88161e90 2024-04-16T16:18:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e8ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7e880d4380 con 0x7f7e88162270 2024-04-16T16:18:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e8ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7e880d4680 con 0x7f7e880d52d0 2024-04-16T16:18:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e86a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e88161e90 0x7f7e881ea640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e86a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e88161e90 0x7f7e881ea640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39316/0 (socket says 172.21.15.45:39316) 2024-04-16T16:18:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e86a00640 1 -- 172.21.15.45:0/4161524705 learned_addr learned my addr 172.21.15.45:0/4161524705 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:07.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e87400640 1 --2- 172.21.15.45:0/4161524705 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e880d52d0 0x7f7e880c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:07.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e86a00640 1 -- 172.21.15.45:0/4161524705 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e880d52d0 msgr2=0x7f7e880c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:07.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e86a00640 1 --2- 172.21.15.45:0/4161524705 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e880d52d0 0x7f7e880c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e86a00640 1 -- 172.21.15.45:0/4161524705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e88162270 msgr2=0x7f7e8814fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:07.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e86a00640 1 --2- 172.21.15.45:0/4161524705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e88162270 0x7f7e8814fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e86a00640 1 -- 172.21.15.45:0/4161524705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e880ca470 con 0x7f7e88161e90 2024-04-16T16:18:07.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.450+0000 7f7e87400640 1 --2- 172.21.15.45:0/4161524705 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e880d52d0 0x7f7e880c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:07.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.451+0000 7f7e86a00640 1 --2- 172.21.15.45:0/4161524705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e88161e90 0x7f7e881ea640 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f7e78036040 tx=0x7f7e78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:07.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.451+0000 7f7e7f400640 1 -- 172.21.15.45:0/4161524705 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e78043020 con 0x7f7e88161e90 2024-04-16T16:18:07.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.451+0000 7f7e7f400640 1 -- 172.21.15.45:0/4161524705 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e7803e450 con 0x7f7e88161e90 2024-04-16T16:18:07.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.451+0000 7f7e7f400640 1 -- 172.21.15.45:0/4161524705 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7803e840 con 0x7f7e88161e90 2024-04-16T16:18:07.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.451+0000 7f7e8ca00640 1 -- 172.21.15.45:0/4161524705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e880ca750 con 0x7f7e88161e90 2024-04-16T16:18:07.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.452+0000 7f7e8ca00640 1 -- 172.21.15.45:0/4161524705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e881500f0 con 0x7f7e88161e90 2024-04-16T16:18:07.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.453+0000 7f7e8ca00640 1 -- 172.21.15.45:0/4161524705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e88075780 con 0x7f7e88161e90 2024-04-16T16:18:07.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.454+0000 7f7e7f400640 1 -- 172.21.15.45:0/4161524705 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f7e78046020 con 0x7f7e88161e90 2024-04-16T16:18:07.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.455+0000 7f7e7f400640 1 --2- 172.21.15.45:0/4161524705 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7e6406ca10 0x7f7e6406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:07.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.455+0000 7f7e7f400640 1 -- 172.21.15.45:0/4161524705 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5803+0+0 (secure 0 0 0) 0x7f7e780ae500 con 0x7f7e88161e90 2024-04-16T16:18:07.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.456+0000 7f7e87400640 1 --2- 172.21.15.45:0/4161524705 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7e6406ca10 0x7f7e6406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:07.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.457+0000 7f7e87400640 1 --2- 172.21.15.45:0/4161524705 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7e6406ca10 0x7f7e6406eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7e70000f60 tx=0x7f7e70009040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:07.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.460+0000 7f7e7f400640 1 -- 172.21.15.45:0/4161524705 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7e7804d030 con 0x7f7e88161e90 2024-04-16T16:18:07.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.670+0000 7f7e8ca00640 1 -- 172.21.15.45:0/4161524705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7e880cb700 con 0x7f7e88161e90 2024-04-16T16:18:07.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.673+0000 7f7e7f400640 1 -- 172.21.15.45:0/4161524705 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+14287 (secure 0 0 0) 0x7f7e7807c850 con 0x7f7e88161e90 2024-04-16T16:18:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:07.673 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":77,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:18:05.159486+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/268877398":"2024-04-17T16:18:01.776989+0000","172.21.15.45:6834/883721507":"2024-04-17T16:18:01.776961+0000","172.21.15.103:0/1692905448":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/1736628710":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/1783855160":"2024-04-17T16:17:55.075639+0000","172.21.15.103:6833/541757982":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/2044864797":"2024-04-17T16:17:55.075639+0000","172.21.15.103:6833/4155603915":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/951047877":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/1131547374":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/3176524506":"2024-04-17T16:17:01.408806+0000","172.21.15.103:6832/4155603915":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/365521086":"2024-04-17T16:17:01.408806+0000","172.21.15.45:6833/268877398":"2024-04-17T16:18:01.776989+0000","172.21.15.103:6832/541757982":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/3345679536":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/2747644735":"2024-04-17T16:17:55.075639+0000","172.21.15.103:0/3606536347":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/1223877773":"2024-04-17T16:17:01.408806+0000","172.21.15.45:6832/883721507":"2024-04-17T16:18:01.776961+0000","172.21.15.103:0/252084277":"2024-04-17T16:17:01.408806+0000","172.21.15.103:0/650078727":"2024-04-17T16:17:01.408806+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:18:07.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.675+0000 7f7e8ca00640 1 -- 172.21.15.45:0/4161524705 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7e6406ca10 msgr2=0x7f7e6406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:07.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.675+0000 7f7e8ca00640 1 --2- 172.21.15.45:0/4161524705 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7e6406ca10 0x7f7e6406eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7e70000f60 tx=0x7f7e70009040 comp rx=0 tx=0).stop 2024-04-16T16:18:07.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.675+0000 7f7e8ca00640 1 -- 172.21.15.45:0/4161524705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e88161e90 msgr2=0x7f7e881ea640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:07.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.675+0000 7f7e8ca00640 1 --2- 172.21.15.45:0/4161524705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e88161e90 0x7f7e881ea640 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f7e78036040 tx=0x7f7e78004a40 comp rx=0 tx=0).stop 2024-04-16T16:18:07.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.675+0000 7f7e8ca00640 1 -- 172.21.15.45:0/4161524705 shutdown_connections 2024-04-16T16:18:07.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.675+0000 7f7e8ca00640 1 --2- 172.21.15.45:0/4161524705 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7e6406ca10 0x7f7e6406eed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.675+0000 7f7e8ca00640 1 --2- 172.21.15.45:0/4161524705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e88162270 0x7f7e8814fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.675+0000 7f7e8ca00640 1 --2- 172.21.15.45:0/4161524705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e88161e90 0x7f7e881ea640 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.675+0000 7f7e8ca00640 1 --2- 172.21.15.45:0/4161524705 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e880d52d0 0x7f7e880c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.675+0000 7f7e8ca00640 1 -- 172.21.15.45:0/4161524705 >> 172.21.15.45:0/4161524705 conn(0x7f7e880d1940 msgr2=0x7f7e880b8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:07.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.675+0000 7f7e8ca00640 1 -- 172.21.15.45:0/4161524705 shutdown_connections 2024-04-16T16:18:07.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.676+0000 7f7e8ca00640 1 -- 172.21.15.45:0/4161524705 wait complete. 2024-04-16T16:18:07.695 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:18:07.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:18:07.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.926+0000 7ffa07e00640 1 Processor -- start 2024-04-16T16:18:07.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.926+0000 7ffa07e00640 1 -- start start 2024-04-16T16:18:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.928+0000 7ffa07e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffa0815b550 0x7ffa0815f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.928+0000 7ffa07e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa0815b930 0x7ffa0815f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.928+0000 7ffa07e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ffa0806e9c0 0x7ffa0806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.928+0000 7ffa07e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ffa080ced90 con 0x7ffa0815b550 2024-04-16T16:18:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.928+0000 7ffa07e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ffa080ce7b0 con 0x7ffa0806e9c0 2024-04-16T16:18:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.928+0000 7ffa07e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffa0806f6b0 con 0x7ffa0815b930 2024-04-16T16:18:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.928+0000 7ffa06a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffa0815b550 0x7ffa0815f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.928+0000 7ffa06a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffa0815b550 0x7ffa0815f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39324/0 (socket says 172.21.15.45:39324) 2024-04-16T16:18:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.928+0000 7ffa06a00640 1 -- 172.21.15.45:0/541961372 learned_addr learned my addr 172.21.15.45:0/541961372 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.929+0000 7ffa06000640 1 --2- 172.21.15.45:0/541961372 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa0815b930 0x7ffa0815f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.929+0000 7ffa07400640 1 --2- 172.21.15.45:0/541961372 >> v2:172.21.15.103:3300/0 conn(0x7ffa0806e9c0 0x7ffa0806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.929+0000 7ffa06a00640 1 -- 172.21.15.45:0/541961372 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa0815b930 msgr2=0x7ffa0815f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:07.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.929+0000 7ffa06a00640 1 --2- 172.21.15.45:0/541961372 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa0815b930 0x7ffa0815f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.929+0000 7ffa06a00640 1 -- 172.21.15.45:0/541961372 >> v2:172.21.15.103:3300/0 conn(0x7ffa0806e9c0 msgr2=0x7ffa0806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:07.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.929+0000 7ffa06a00640 1 --2- 172.21.15.45:0/541961372 >> v2:172.21.15.103:3300/0 conn(0x7ffa0806e9c0 0x7ffa0806ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.929+0000 7ffa06a00640 1 -- 172.21.15.45:0/541961372 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa080cea20 con 0x7ffa0815b550 2024-04-16T16:18:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.930+0000 7ffa06a00640 1 --2- 172.21.15.45:0/541961372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa0815b550 0x7ffa0815f120 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7ffa080b2150 tx=0x7ff9f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acfa56539a8c6d42 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.930+0000 7ffa05600640 1 -- 172.21.15.45:0/541961372 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8035020 con 0x7ffa0815b550 2024-04-16T16:18:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.930+0000 7ffa05600640 1 -- 172.21.15.45:0/541961372 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9f8031870 con 0x7ffa0815b550 2024-04-16T16:18:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.930+0000 7ffa05600640 1 -- 172.21.15.45:0/541961372 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8031c40 con 0x7ffa0815b550 2024-04-16T16:18:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.931+0000 7ffa07e00640 1 -- 172.21.15.45:0/541961372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa0815b550 msgr2=0x7ffa0815f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.931+0000 7ffa07e00640 1 --2- 172.21.15.45:0/541961372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa0815b550 0x7ffa0815f120 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7ffa080b2150 tx=0x7ff9f802b910 comp rx=0 tx=0).stop 2024-04-16T16:18:07.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.931+0000 7ffa07e00640 1 -- 172.21.15.45:0/541961372 shutdown_connections 2024-04-16T16:18:07.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.931+0000 7ffa07e00640 1 --2- 172.21.15.45:0/541961372 >> v2:172.21.15.103:3300/0 conn(0x7ffa0806e9c0 0x7ffa0806ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.931+0000 7ffa07e00640 1 --2- 172.21.15.45:0/541961372 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa0815b930 0x7ffa0815f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.931+0000 7ffa07e00640 1 --2- 172.21.15.45:0/541961372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa0815b550 0x7ffa0815f120 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.931+0000 7ffa07e00640 1 -- 172.21.15.45:0/541961372 >> 172.21.15.45:0/541961372 conn(0x7ffa08067e00 msgr2=0x7ffa0806b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:07.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.931+0000 7ffa07e00640 1 -- 172.21.15.45:0/541961372 shutdown_connections 2024-04-16T16:18:07.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.931+0000 7ffa07e00640 1 -- 172.21.15.45:0/541961372 wait complete. 2024-04-16T16:18:07.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.933+0000 7ffa07e00640 1 Processor -- start 2024-04-16T16:18:07.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.933+0000 7ffa07e00640 1 -- start start 2024-04-16T16:18:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.934+0000 7ffa07e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa0806e9c0 0x7ffa0815e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.934+0000 7ffa07e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa0815b550 0x7ffa0814caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.934+0000 7ffa07e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa0815b930 0x7ffa0814d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.934+0000 7ffa07e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffa080cd700 con 0x7ffa0815b930 2024-04-16T16:18:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.934+0000 7ffa07e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0806f990 con 0x7ffa0815b550 2024-04-16T16:18:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.934+0000 7ffa07e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffa080cd880 con 0x7ffa0806e9c0 2024-04-16T16:18:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.934+0000 7ffa07400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa0815b930 0x7ffa0814d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.935+0000 7ffa07400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa0815b930 0x7ffa0814d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39338/0 (socket says 172.21.15.45:39338) 2024-04-16T16:18:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.935+0000 7ffa07400640 1 -- 172.21.15.45:0/610858332 learned_addr learned my addr 172.21.15.45:0/610858332 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.935+0000 7ffa06a00640 1 --2- 172.21.15.45:0/610858332 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa0806e9c0 0x7ffa0815e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.935+0000 7ffa06000640 1 --2- 172.21.15.45:0/610858332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa0815b550 0x7ffa0814caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.935+0000 7ffa07400640 1 -- 172.21.15.45:0/610858332 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa0806e9c0 msgr2=0x7ffa0815e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.935+0000 7ffa07400640 1 --2- 172.21.15.45:0/610858332 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa0806e9c0 0x7ffa0815e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.935+0000 7ffa07400640 1 -- 172.21.15.45:0/610858332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa0815b550 msgr2=0x7ffa0814caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.935+0000 7ffa07400640 1 --2- 172.21.15.45:0/610858332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa0815b550 0x7ffa0814caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.935+0000 7ffa07400640 1 -- 172.21.15.45:0/610858332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa081493e0 con 0x7ffa0815b930 2024-04-16T16:18:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.935+0000 7ffa07400640 1 --2- 172.21.15.45:0/610858332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa0815b930 0x7ffa0814d030 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7ffa00007e90 tx=0x7ffa0000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.936+0000 7ff9fea00640 1 -- 172.21.15.45:0/610858332 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa00014420 con 0x7ffa0815b930 2024-04-16T16:18:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.936+0000 7ff9fea00640 1 -- 172.21.15.45:0/610858332 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa00019070 con 0x7ffa0815b930 2024-04-16T16:18:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.936+0000 7ff9fea00640 1 -- 172.21.15.45:0/610858332 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa00014820 con 0x7ffa0815b930 2024-04-16T16:18:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.936+0000 7ffa07e00640 1 -- 172.21.15.45:0/610858332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa08149680 con 0x7ffa0815b930 2024-04-16T16:18:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.936+0000 7ffa07e00640 1 -- 172.21.15.45:0/610858332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa08149c40 con 0x7ffa0815b930 2024-04-16T16:18:07.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.937+0000 7ffa07e00640 1 -- 172.21.15.45:0/610858332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa08075780 con 0x7ffa0815b930 2024-04-16T16:18:07.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.938+0000 7ff9fea00640 1 -- 172.21.15.45:0/610858332 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7ffa000149c0 con 0x7ffa0815b930 2024-04-16T16:18:07.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.939+0000 7ff9fea00640 1 --2- 172.21.15.45:0/610858332 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff9e806ca40 0x7ff9e806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:07.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.939+0000 7ff9fea00640 1 -- 172.21.15.45:0/610858332 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5803+0+0 (secure 0 0 0) 0x7ffa0008c570 con 0x7ffa0815b930 2024-04-16T16:18:07.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.940+0000 7ffa06a00640 1 --2- 172.21.15.45:0/610858332 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff9e806ca40 0x7ff9e806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:07.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.941+0000 7ffa06a00640 1 --2- 172.21.15.45:0/610858332 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff9e806ca40 0x7ff9e806ef00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8036000 tx=0x7ff9f803c040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:07.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:07.944+0000 7ff9fea00640 1 -- 172.21.15.45:0/610858332 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffa0005a830 con 0x7ffa0815b930 2024-04-16T16:18:08.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.150+0000 7ffa07e00640 1 -- 172.21.15.45:0/610858332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffa0806c550 con 0x7ffa0815b930 2024-04-16T16:18:08.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.151+0000 7ff9fea00640 1 -- 172.21.15.45:0/610858332 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) ==== 93+0+10 (secure 0 0 0) 0x7ffa0005a130 con 0x7ffa0815b930 2024-04-16T16:18:08.151 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:18:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.153+0000 7ffa07e00640 1 -- 172.21.15.45:0/610858332 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff9e806ca40 msgr2=0x7ff9e806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.153+0000 7ffa07e00640 1 --2- 172.21.15.45:0/610858332 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff9e806ca40 0x7ff9e806ef00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8036000 tx=0x7ff9f803c040 comp rx=0 tx=0).stop 2024-04-16T16:18:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.153+0000 7ffa07e00640 1 -- 172.21.15.45:0/610858332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa0815b930 msgr2=0x7ffa0814d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.154+0000 7ffa07e00640 1 --2- 172.21.15.45:0/610858332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa0815b930 0x7ffa0814d030 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7ffa00007e90 tx=0x7ffa0000e4b0 comp rx=0 tx=0).stop 2024-04-16T16:18:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.154+0000 7ffa07e00640 1 -- 172.21.15.45:0/610858332 shutdown_connections 2024-04-16T16:18:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.154+0000 7ffa07e00640 1 --2- 172.21.15.45:0/610858332 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff9e806ca40 0x7ff9e806ef00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.154+0000 7ffa07e00640 1 --2- 172.21.15.45:0/610858332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa0815b930 0x7ffa0814d030 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.154+0000 7ffa07e00640 1 --2- 172.21.15.45:0/610858332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa0815b550 0x7ffa0814caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.154+0000 7ffa07e00640 1 --2- 172.21.15.45:0/610858332 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa0806e9c0 0x7ffa0815e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.154+0000 7ffa07e00640 1 -- 172.21.15.45:0/610858332 >> 172.21.15.45:0/610858332 conn(0x7ffa08067e00 msgr2=0x7ffa080caf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.154+0000 7ffa07e00640 1 -- 172.21.15.45:0/610858332 shutdown_connections 2024-04-16T16:18:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.154+0000 7ffa07e00640 1 -- 172.21.15.45:0/610858332 wait complete. 2024-04-16T16:18:08.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_client_id' 2024-04-16T16:18:08.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.397+0000 7fc9c2600640 1 Processor -- start 2024-04-16T16:18:08.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.397+0000 7fc9c2600640 1 -- start start 2024-04-16T16:18:08.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.399+0000 7fc9c2600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc9bc072a90 0x7fc9bc0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:08.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.399+0000 7fc9c2600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc9bc15ea70 0x7fc9bc0cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.399+0000 7fc9c2600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9bc0cf140 con 0x7fc9bc1462e0 2024-04-16T16:18:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.399+0000 7fc9c2600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc9bc0cf6d0 con 0x7fc9bc15ea70 2024-04-16T16:18:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.399+0000 7fc9c2600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc9bc1466c0 con 0x7fc9bc072a90 2024-04-16T16:18:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.399+0000 7fc9bbe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc9bc15ea70 0x7fc9bc0cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.399+0000 7fc9bbe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc9bc15ea70 0x7fc9bc0cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39354/0 (socket says 172.21.15.45:39354) 2024-04-16T16:18:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.399+0000 7fc9bbe00640 1 -- 172.21.15.45:0/1705239435 learned_addr learned my addr 172.21.15.45:0/1705239435 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.399+0000 7fc9b3e00640 1 --2- 172.21.15.45:0/1705239435 >> v2:172.21.15.103:3300/0 conn(0x7fc9bc072a90 0x7fc9bc0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.400+0000 7fc9bbe00640 1 -- 172.21.15.45:0/1705239435 >> v2:172.21.15.103:3300/0 conn(0x7fc9bc072a90 msgr2=0x7fc9bc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.400+0000 7fc9bbe00640 1 --2- 172.21.15.45:0/1705239435 >> v2:172.21.15.103:3300/0 conn(0x7fc9bc072a90 0x7fc9bc0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.400+0000 7fc9bbe00640 1 -- 172.21.15.45:0/1705239435 >> v1:172.21.15.103:6789/0 conn(0x7fc9bc1462e0 legacy=0x7fc9bc072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:08.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.400+0000 7fc9bbe00640 1 -- 172.21.15.45:0/1705239435 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9bc0cf3f0 con 0x7fc9bc15ea70 2024-04-16T16:18:08.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.401+0000 7fc9bbe00640 1 --2- 172.21.15.45:0/1705239435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9bc15ea70 0x7fc9bc0cec00 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc06f9b0 tx=0x7fc9b402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61b6b044bb356da server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:08.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.401+0000 7fc9baa00640 1 -- 172.21.15.45:0/1705239435 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b402c9f0 con 0x7fc9bc15ea70 2024-04-16T16:18:08.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.401+0000 7fc9baa00640 1 -- 172.21.15.45:0/1705239435 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9b402cb90 con 0x7fc9bc15ea70 2024-04-16T16:18:08.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.401+0000 7fc9baa00640 1 -- 172.21.15.45:0/1705239435 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b40343f0 con 0x7fc9bc15ea70 2024-04-16T16:18:08.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.402+0000 7fc9c2600640 1 -- 172.21.15.45:0/1705239435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9bc15ea70 msgr2=0x7fc9bc0cec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:08.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.402+0000 7fc9c2600640 1 --2- 172.21.15.45:0/1705239435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9bc15ea70 0x7fc9bc0cec00 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc06f9b0 tx=0x7fc9b402be50 comp rx=0 tx=0).stop 2024-04-16T16:18:08.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.402+0000 7fc9c2600640 1 -- 172.21.15.45:0/1705239435 shutdown_connections 2024-04-16T16:18:08.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.402+0000 7fc9c2600640 1 --2- 172.21.15.45:0/1705239435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9bc15ea70 0x7fc9bc0cec00 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:08.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.402+0000 7fc9c2600640 1 --2- 172.21.15.45:0/1705239435 >> v2:172.21.15.103:3300/0 conn(0x7fc9bc072a90 0x7fc9bc0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:08.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.402+0000 7fc9c2600640 1 -- 172.21.15.45:0/1705239435 >> 172.21.15.45:0/1705239435 conn(0x7fc9bc0b2750 msgr2=0x7fc9bc0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:08.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.403+0000 7fc9c2600640 1 -- 172.21.15.45:0/1705239435 shutdown_connections 2024-04-16T16:18:08.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.403+0000 7fc9c2600640 1 -- 172.21.15.45:0/1705239435 wait complete. 2024-04-16T16:18:08.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.405+0000 7fc9c2600640 1 Processor -- start 2024-04-16T16:18:08.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.405+0000 7fc9c2600640 1 -- start start 2024-04-16T16:18:08.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.406+0000 7fc9c2600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9bc072a90 0x7fc9bc0b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:08.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.406+0000 7fc9c2600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9bc1462e0 0x7fc9bc13c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:08.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.406+0000 7fc9c2600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9bc15ea70 0x7fc9bc06bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.406+0000 7fc9c2600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc9bc15d540 con 0x7fc9bc1462e0 2024-04-16T16:18:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.406+0000 7fc9c2600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc9bc15d260 con 0x7fc9bc15ea70 2024-04-16T16:18:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.406+0000 7fc9c2600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc9bc166490 con 0x7fc9bc072a90 2024-04-16T16:18:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.407+0000 7fc9b3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9bc1462e0 0x7fc9bc13c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.407+0000 7fc9b3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9bc1462e0 0x7fc9bc13c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39358/0 (socket says 172.21.15.45:39358) 2024-04-16T16:18:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.407+0000 7fc9b3e00640 1 -- 172.21.15.45:0/2414581037 learned_addr learned my addr 172.21.15.45:0/2414581037 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.407+0000 7fc9bb400640 1 --2- 172.21.15.45:0/2414581037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9bc072a90 0x7fc9bc0b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.407+0000 7fc9bbe00640 1 --2- 172.21.15.45:0/2414581037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9bc15ea70 0x7fc9bc06bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.407+0000 7fc9b3e00640 1 -- 172.21.15.45:0/2414581037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9bc072a90 msgr2=0x7fc9bc0b5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.407+0000 7fc9b3e00640 1 --2- 172.21.15.45:0/2414581037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9bc072a90 0x7fc9bc0b5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.407+0000 7fc9b3e00640 1 -- 172.21.15.45:0/2414581037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9bc15ea70 msgr2=0x7fc9bc06bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.407+0000 7fc9b3e00640 1 --2- 172.21.15.45:0/2414581037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9bc15ea70 0x7fc9bc06bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.407+0000 7fc9b3e00640 1 -- 172.21.15.45:0/2414581037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9bc0b4760 con 0x7fc9bc1462e0 2024-04-16T16:18:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.408+0000 7fc9b3e00640 1 --2- 172.21.15.45:0/2414581037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9bc1462e0 0x7fc9bc13c3d0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400d5f0 tx=0x7fc9a400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:08.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.408+0000 7fc9b3400640 1 -- 172.21.15.45:0/2414581037 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a40125f0 con 0x7fc9bc1462e0 2024-04-16T16:18:08.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.408+0000 7fc9b3400640 1 -- 172.21.15.45:0/2414581037 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9a4019070 con 0x7fc9bc1462e0 2024-04-16T16:18:08.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.408+0000 7fc9b3400640 1 -- 172.21.15.45:0/2414581037 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a40129d0 con 0x7fc9bc1462e0 2024-04-16T16:18:08.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.408+0000 7fc9c2600640 1 -- 172.21.15.45:0/2414581037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9bc0b4a40 con 0x7fc9bc1462e0 2024-04-16T16:18:08.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.408+0000 7fc9c2600640 1 -- 172.21.15.45:0/2414581037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9bc0b5000 con 0x7fc9bc1462e0 2024-04-16T16:18:08.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.410+0000 7fc9c2600640 1 -- 172.21.15.45:0/2414581037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9bc0afe10 con 0x7fc9bc1462e0 2024-04-16T16:18:08.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.411+0000 7fc9b3400640 1 -- 172.21.15.45:0/2414581037 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7fc9a4012b70 con 0x7fc9bc1462e0 2024-04-16T16:18:08.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.411+0000 7fc9b3400640 1 --2- 172.21.15.45:0/2414581037 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc99806ca10 0x7fc99806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:08.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.412+0000 7fc9b3400640 1 -- 172.21.15.45:0/2414581037 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5803+0+0 (secure 0 0 0) 0x7fc9a408c190 con 0x7fc9bc1462e0 2024-04-16T16:18:08.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.412+0000 7fc9bb400640 1 --2- 172.21.15.45:0/2414581037 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc99806ca10 0x7fc99806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:08.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.414+0000 7fc9bb400640 1 --2- 172.21.15.45:0/2414581037 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc99806ca10 0x7fc99806eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac006fd0 tx=0x7fc9ac008040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:08.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.416+0000 7fc9b3400640 1 -- 172.21.15.45:0/2414581037 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc9bc0afe10 con 0x7fc9bc1462e0 2024-04-16T16:18:08.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:08.690+0000 7fc9c2600640 1 -- 172.21.15.45:0/2414581037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_client_id"]} v 0) -- 0x7fc9bc1433e0 con 0x7fc9bc1462e0 2024-04-16T16:18:09.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.197+0000 7fc9b3400640 1 -- 172.21.15.45:0/2414581037 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_client_id"]}]=0 v278) ==== 153+0+0 (secure 0 0 0) 0x7fc9a405a450 con 0x7fc9bc1462e0 2024-04-16T16:18:09.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.200+0000 7fc9c2600640 1 -- 172.21.15.45:0/2414581037 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc99806ca10 msgr2=0x7fc99806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:09.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.200+0000 7fc9c2600640 1 --2- 172.21.15.45:0/2414581037 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc99806ca10 0x7fc99806eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac006fd0 tx=0x7fc9ac008040 comp rx=0 tx=0).stop 2024-04-16T16:18:09.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.200+0000 7fc9c2600640 1 -- 172.21.15.45:0/2414581037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9bc1462e0 msgr2=0x7fc9bc13c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:09.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.200+0000 7fc9c2600640 1 --2- 172.21.15.45:0/2414581037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9bc1462e0 0x7fc9bc13c3d0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400d5f0 tx=0x7fc9a400dab0 comp rx=0 tx=0).stop 2024-04-16T16:18:09.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.200+0000 7fc9c2600640 1 -- 172.21.15.45:0/2414581037 shutdown_connections 2024-04-16T16:18:09.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.200+0000 7fc9c2600640 1 --2- 172.21.15.45:0/2414581037 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc99806ca10 0x7fc99806eed0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:09.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.200+0000 7fc9c2600640 1 --2- 172.21.15.45:0/2414581037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9bc15ea70 0x7fc9bc06bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:09.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.200+0000 7fc9c2600640 1 --2- 172.21.15.45:0/2414581037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9bc1462e0 0x7fc9bc13c3d0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:09.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.200+0000 7fc9c2600640 1 --2- 172.21.15.45:0/2414581037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9bc072a90 0x7fc9bc0b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:09.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.200+0000 7fc9c2600640 1 -- 172.21.15.45:0/2414581037 >> 172.21.15.45:0/2414581037 conn(0x7fc9bc0b2750 msgr2=0x7fc9bc1424d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:09.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.201+0000 7fc9c2600640 1 -- 172.21.15.45:0/2414581037 shutdown_connections 2024-04-16T16:18:09.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.201+0000 7fc9c2600640 1 -- 172.21.15.45:0/2414581037 wait complete. 2024-04-16T16:18:09.224 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-16T16:18:09.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.456+0000 7fca12400640 1 Processor -- start 2024-04-16T16:18:09.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.457+0000 7fca12400640 1 -- start start 2024-04-16T16:18:09.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.458+0000 7fca12400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fca0c0b5d50 0x7fca0c06eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:09.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.458+0000 7fca12400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca0c146590 con 0x7fca0c1461b0 2024-04-16T16:18:09.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.458+0000 7fca12400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca0c0c5750 con 0x7fca0c0b58f0 2024-04-16T16:18:09.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.458+0000 7fca12400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fca0c06dfd0 con 0x7fca0c0b5d50 2024-04-16T16:18:09.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.458+0000 7fca11a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fca0c0b5d50 0x7fca0c06eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:09.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.458+0000 7fca11000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fca0c1461b0 0x7fca0c160cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46790/0 (socket says 172.21.15.45:46790) 2024-04-16T16:18:09.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.458+0000 7fca11000640 1 -- 172.21.15.45:0/2903734952 learned_addr learned my addr 172.21.15.45:0/2903734952 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:09.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.459+0000 7fca11a00640 1 -- 172.21.15.45:0/2903734952 >> v1:172.21.15.103:6789/0 conn(0x7fca0c0b58f0 legacy=0x7fca0c06e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:09.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.459+0000 7fca11a00640 1 -- 172.21.15.45:0/2903734952 >> v1:172.21.15.45:6789/0 conn(0x7fca0c1461b0 legacy=0x7fca0c160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:09.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.459+0000 7fca11a00640 1 -- 172.21.15.45:0/2903734952 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0c15ccb0 con 0x7fca0c0b5d50 2024-04-16T16:18:09.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.460+0000 7fca11a00640 1 --2- 172.21.15.45:0/2903734952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca0c0b5d50 0x7fca0c06eb70 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fca04009d90 tx=0x7fca0402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f23452fec9ae7a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:09.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.460+0000 7fca0b400640 1 -- 172.21.15.45:0/2903734952 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04039070 con 0x7fca0c0b5d50 2024-04-16T16:18:09.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.460+0000 7fca0b400640 1 -- 172.21.15.45:0/2903734952 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca0402cc60 con 0x7fca0c0b5d50 2024-04-16T16:18:09.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.460+0000 7fca0b400640 1 -- 172.21.15.45:0/2903734952 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04034660 con 0x7fca0c0b5d50 2024-04-16T16:18:09.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.461+0000 7fca12400640 1 -- 172.21.15.45:0/2903734952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca0c0b5d50 msgr2=0x7fca0c06eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:09.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.461+0000 7fca12400640 1 --2- 172.21.15.45:0/2903734952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca0c0b5d50 0x7fca0c06eb70 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fca04009d90 tx=0x7fca0402beb0 comp rx=0 tx=0).stop 2024-04-16T16:18:09.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.461+0000 7fca12400640 1 -- 172.21.15.45:0/2903734952 shutdown_connections 2024-04-16T16:18:09.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.461+0000 7fca12400640 1 --2- 172.21.15.45:0/2903734952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca0c0b5d50 0x7fca0c06eb70 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:09.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.461+0000 7fca12400640 1 -- 172.21.15.45:0/2903734952 >> 172.21.15.45:0/2903734952 conn(0x7fca0c06b930 msgr2=0x7fca0c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:09.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.461+0000 7fca12400640 1 -- 172.21.15.45:0/2903734952 shutdown_connections 2024-04-16T16:18:09.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.461+0000 7fca12400640 1 -- 172.21.15.45:0/2903734952 wait complete. 2024-04-16T16:18:09.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.463+0000 7fca12400640 1 Processor -- start 2024-04-16T16:18:09.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.463+0000 7fca12400640 1 -- start start 2024-04-16T16:18:09.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.464+0000 7fca12400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca0c0b58f0 0x7fca0c136f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:09.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.464+0000 7fca12400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca0c0b5d50 0x7fca0c144810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:09.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.464+0000 7fca12400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca0c1461b0 0x7fca0c0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:09.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.464+0000 7fca12400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c0c5fb0 con 0x7fca0c0b58f0 2024-04-16T16:18:09.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.464+0000 7fca12400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c0c5cd0 con 0x7fca0c1461b0 2024-04-16T16:18:09.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.464+0000 7fca12400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fca0c0c6290 con 0x7fca0c0b5d50 2024-04-16T16:18:09.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.464+0000 7fca11000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca0c0b58f0 0x7fca0c136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:09.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.464+0000 7fca11000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca0c0b58f0 0x7fca0c136f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39376/0 (socket says 172.21.15.45:39376) 2024-04-16T16:18:09.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.464+0000 7fca11000640 1 -- 172.21.15.45:0/851401757 learned_addr learned my addr 172.21.15.45:0/851401757 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:09.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.465+0000 7fca0be00640 1 --2- 172.21.15.45:0/851401757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca0c0b5d50 0x7fca0c144810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:09.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.465+0000 7fca11a00640 1 --2- 172.21.15.45:0/851401757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca0c1461b0 0x7fca0c0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:09.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.465+0000 7fca11000640 1 -- 172.21.15.45:0/851401757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca0c0b5d50 msgr2=0x7fca0c144810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:09.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.465+0000 7fca11000640 1 --2- 172.21.15.45:0/851401757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca0c0b5d50 0x7fca0c144810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:09.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.465+0000 7fca11000640 1 -- 172.21.15.45:0/851401757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca0c1461b0 msgr2=0x7fca0c0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:09.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.465+0000 7fca11000640 1 --2- 172.21.15.45:0/851401757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca0c1461b0 0x7fca0c0cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:09.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.465+0000 7fca11000640 1 -- 172.21.15.45:0/851401757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0c1374c0 con 0x7fca0c0b58f0 2024-04-16T16:18:09.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.465+0000 7fca11a00640 1 --2- 172.21.15.45:0/851401757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca0c1461b0 0x7fca0c0cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:09.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.465+0000 7fca11000640 1 --2- 172.21.15.45:0/851401757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca0c0b58f0 0x7fca0c136f80 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fca0000c3c0 tx=0x7fca0000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:09.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.465+0000 7fca08c00640 1 -- 172.21.15.45:0/851401757 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca00017070 con 0x7fca0c0b58f0 2024-04-16T16:18:09.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.465+0000 7fca08c00640 1 -- 172.21.15.45:0/851401757 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca00005ce0 con 0x7fca0c0b58f0 2024-04-16T16:18:09.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.465+0000 7fca08c00640 1 -- 172.21.15.45:0/851401757 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca000072f0 con 0x7fca0c0b58f0 2024-04-16T16:18:09.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.465+0000 7fca12400640 1 -- 172.21.15.45:0/851401757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca0c0cd6c0 con 0x7fca0c0b58f0 2024-04-16T16:18:09.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.466+0000 7fca12400640 1 -- 172.21.15.45:0/851401757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca0c1f8040 con 0x7fca0c0b58f0 2024-04-16T16:18:09.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.467+0000 7fca12400640 1 -- 172.21.15.45:0/851401757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca0c1f8750 con 0x7fca0c0b58f0 2024-04-16T16:18:09.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.468+0000 7fca08c00640 1 -- 172.21.15.45:0/851401757 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7fca0001a070 con 0x7fca0c0b58f0 2024-04-16T16:18:09.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.469+0000 7fca08c00640 1 --2- 172.21.15.45:0/851401757 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc9ec06ca40 0x7fc9ec06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:09.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.469+0000 7fca08c00640 1 -- 172.21.15.45:0/851401757 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5803+0+0 (secure 0 0 0) 0x7fca0008b0e0 con 0x7fca0c0b58f0 2024-04-16T16:18:09.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.470+0000 7fca0be00640 1 --2- 172.21.15.45:0/851401757 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc9ec06ca40 0x7fc9ec06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:09.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.471+0000 7fca0be00640 1 --2- 172.21.15.45:0/851401757 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc9ec06ca40 0x7fc9ec06ef00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fca0c142190 tx=0x7fc9f8009040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:09.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.474+0000 7fca08c00640 1 -- 172.21.15.45:0/851401757 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fca0c1374c0 con 0x7fca0c0b58f0 2024-04-16T16:18:09.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.669+0000 7fca12400640 1 -- 172.21.15.45:0/851401757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fca0c075780 con 0x7fca0c0b58f0 2024-04-16T16:18:09.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.677+0000 7fca08c00640 1 -- 172.21.15.45:0/851401757 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) ==== 122+0+0 (secure 0 0 0) 0x7fca00059390 con 0x7fca0c0b58f0 2024-04-16T16:18:09.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.679+0000 7fca12400640 1 -- 172.21.15.45:0/851401757 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc9ec06ca40 msgr2=0x7fc9ec06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:09.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.679+0000 7fca12400640 1 --2- 172.21.15.45:0/851401757 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc9ec06ca40 0x7fc9ec06ef00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fca0c142190 tx=0x7fc9f8009040 comp rx=0 tx=0).stop 2024-04-16T16:18:09.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.679+0000 7fca12400640 1 -- 172.21.15.45:0/851401757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca0c0b58f0 msgr2=0x7fca0c136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:09.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.679+0000 7fca12400640 1 --2- 172.21.15.45:0/851401757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca0c0b58f0 0x7fca0c136f80 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fca0000c3c0 tx=0x7fca0000c880 comp rx=0 tx=0).stop 2024-04-16T16:18:09.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.679+0000 7fca12400640 1 -- 172.21.15.45:0/851401757 shutdown_connections 2024-04-16T16:18:09.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.679+0000 7fca12400640 1 --2- 172.21.15.45:0/851401757 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc9ec06ca40 0x7fc9ec06ef00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:09.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.679+0000 7fca12400640 1 --2- 172.21.15.45:0/851401757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca0c1461b0 0x7fca0c0cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:09.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.679+0000 7fca12400640 1 --2- 172.21.15.45:0/851401757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca0c0b5d50 0x7fca0c144810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:09.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.679+0000 7fca12400640 1 --2- 172.21.15.45:0/851401757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca0c0b58f0 0x7fca0c136f80 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:09.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.679+0000 7fca12400640 1 -- 172.21.15.45:0/851401757 >> 172.21.15.45:0/851401757 conn(0x7fca0c06b930 msgr2=0x7fca0c145170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:09.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.680+0000 7fca12400640 1 -- 172.21.15.45:0/851401757 shutdown_connections 2024-04-16T16:18:09.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:09.680+0000 7fca12400640 1 -- 172.21.15.45:0/851401757 wait complete. 2024-04-16T16:18:09.697 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-16T16:18:09.697 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:18:09.816 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:18:09.938 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-16T16:18:09.938 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:18:09.938 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:09.999 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:18:09.999 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:18:09.999 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:10.056 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:18:10.056 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:18:10.057 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:10.117 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:18:10.118 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:18:10.118 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:10.175 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:18:10.175 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-16T16:18:10.176 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:18:10.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.408+0000 7f4731400640 1 Processor -- start 2024-04-16T16:18:10.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.408+0000 7f4731400640 1 -- start start 2024-04-16T16:18:10.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.410+0000 7f4731400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f472c146220 0x7f472c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:10.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.410+0000 7f4731400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f472c15d360 con 0x7f472c06f7a0 2024-04-16T16:18:10.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.410+0000 7f4731400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f472c15d640 con 0x7f472c1608a0 2024-04-16T16:18:10.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.410+0000 7f4731400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f472c160410 con 0x7f472c146220 2024-04-16T16:18:10.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.410+0000 7f472b400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f472c06f7a0 0x7f472c06df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46796/0 (socket says 172.21.15.45:46796) 2024-04-16T16:18:10.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.410+0000 7f472b400640 1 -- 172.21.15.45:0/991144775 learned_addr learned my addr 172.21.15.45:0/991144775 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:10.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472be00640 1 --2- 172.21.15.45:0/991144775 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f472c146220 0x7f472c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:10.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472aa00640 1 -- 172.21.15.45:0/991144775 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 548191959 0 0) 0x7f472c15d360 con 0x7f472c06f7a0 2024-04-16T16:18:10.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472aa00640 1 -- 172.21.15.45:0/991144775 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4710003520 con 0x7f472c06f7a0 2024-04-16T16:18:10.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472aa00640 1 -- 172.21.15.45:0/991144775 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1810437120 0 0) 0x7f4710003520 con 0x7f472c06f7a0 2024-04-16T16:18:10.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472aa00640 1 -- 172.21.15.45:0/991144775 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f472c146220 msgr2=0x7f472c15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:10.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472aa00640 1 --2- 172.21.15.45:0/991144775 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f472c146220 0x7f472c15ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:10.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472aa00640 1 -- 172.21.15.45:0/991144775 >> v1:172.21.15.103:6789/0 conn(0x7f472c1608a0 legacy=0x7f472c0cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:10.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472aa00640 1 -- 172.21.15.45:0/991144775 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f472c15d920 con 0x7f472c06f7a0 2024-04-16T16:18:10.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472aa00640 1 -- 172.21.15.45:0/991144775 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f4718001670 con 0x7f472c06f7a0 2024-04-16T16:18:10.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472aa00640 1 -- 172.21.15.45:0/991144775 >> v1:172.21.15.45:6789/0 conn(0x7f472c06f7a0 legacy=0x7f472c06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:10.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472aa00640 1 --2- 172.21.15.45:0/991144775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47100039e0 0x7f4710003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:10.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472aa00640 1 --2- 172.21.15.45:0/991144775 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f472c146220 0x7f4710004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:10.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472aa00640 1 --2- 172.21.15.45:0/991144775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4710004d40 0x7f47100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:10.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472aa00640 1 -- 172.21.15.45:0/991144775 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f472c160410 con 0x7f4710004d40 2024-04-16T16:18:10.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472aa00640 1 -- 172.21.15.45:0/991144775 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f471000baa0 con 0x7f47100039e0 2024-04-16T16:18:10.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.411+0000 7f472aa00640 1 -- 172.21.15.45:0/991144775 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f471000bca0 con 0x7f472c146220 2024-04-16T16:18:10.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.412+0000 7f4730a00640 1 --2- 172.21.15.45:0/991144775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4710004d40 0x7f47100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:10.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.412+0000 7f472be00640 1 --2- 172.21.15.45:0/991144775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47100039e0 0x7f4710003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:10.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.412+0000 7f472b400640 1 --2- 172.21.15.45:0/991144775 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f472c146220 0x7f4710004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:10.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.412+0000 7f4730a00640 1 -- 172.21.15.45:0/991144775 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f472c146220 msgr2=0x7f4710004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:10.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.412+0000 7f4730a00640 1 --2- 172.21.15.45:0/991144775 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f472c146220 0x7f4710004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:10.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.412+0000 7f4730a00640 1 -- 172.21.15.45:0/991144775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47100039e0 msgr2=0x7f4710003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:10.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.412+0000 7f4730a00640 1 --2- 172.21.15.45:0/991144775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47100039e0 0x7f4710003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:10.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.412+0000 7f4730a00640 1 -- 172.21.15.45:0/991144775 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4710003520 con 0x7f4710004d40 2024-04-16T16:18:10.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.412+0000 7f472be00640 1 --2- 172.21.15.45:0/991144775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47100039e0 0x7f4710003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:18:10.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.413+0000 7f4730a00640 1 --2- 172.21.15.45:0/991144775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4710004d40 0x7f47100051c0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f4724001e20 tx=0x7f4710005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb4aafff5bacc12 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:10.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.414+0000 7f472aa00640 1 -- 172.21.15.45:0/991144775 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4724001a70 con 0x7f4710004d40 2024-04-16T16:18:10.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.414+0000 7f472aa00640 1 -- 172.21.15.45:0/991144775 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4724003b70 con 0x7f4710004d40 2024-04-16T16:18:10.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.414+0000 7f472aa00640 1 -- 172.21.15.45:0/991144775 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47240050d0 con 0x7f4710004d40 2024-04-16T16:18:10.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.414+0000 7f4731400640 1 -- 172.21.15.45:0/991144775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4710004d40 msgr2=0x7f47100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:10.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.414+0000 7f4731400640 1 --2- 172.21.15.45:0/991144775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4710004d40 0x7f47100051c0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f4724001e20 tx=0x7f4710005830 comp rx=0 tx=0).stop 2024-04-16T16:18:10.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.414+0000 7f4730a00640 1 -- 172.21.15.45:0/991144775 reap_dead start 2024-04-16T16:18:10.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.414+0000 7f4731400640 1 -- 172.21.15.45:0/991144775 shutdown_connections 2024-04-16T16:18:10.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.414+0000 7f4731400640 1 -- 172.21.15.45:0/991144775 >> 172.21.15.45:0/991144775 conn(0x7f472c06b930 msgr2=0x7f472c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:10.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.414+0000 7f4731400640 1 -- 172.21.15.45:0/991144775 shutdown_connections 2024-04-16T16:18:10.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.414+0000 7f4731400640 1 -- 172.21.15.45:0/991144775 wait complete. 2024-04-16T16:18:10.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.416+0000 7f4731400640 1 Processor -- start 2024-04-16T16:18:10.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.416+0000 7f4731400640 1 -- start start 2024-04-16T16:18:10.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.417+0000 7f4731400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4710004d40 0x7f472c142cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:10.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.417+0000 7f4731400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f472c137930 0x7f472c1f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:10.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.417+0000 7f4731400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f472c1f84c0 0x7f472c0cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:10.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.417+0000 7f4731400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f472c06fc20 con 0x7f4710004d40 2024-04-16T16:18:10.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.417+0000 7f4731400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f472c15dc00 con 0x7f472c1f84c0 2024-04-16T16:18:10.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.417+0000 7f4731400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f472c06fda0 con 0x7f472c137930 2024-04-16T16:18:10.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.418+0000 7f472be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4710004d40 0x7f472c142cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:10.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.418+0000 7f472be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4710004d40 0x7f472c142cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39382/0 (socket says 172.21.15.45:39382) 2024-04-16T16:18:10.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.418+0000 7f472be00640 1 -- 172.21.15.45:0/3172529572 learned_addr learned my addr 172.21.15.45:0/3172529572 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:10.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.418+0000 7f472b400640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f472c137930 0x7f472c1f7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:10.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.418+0000 7f4730a00640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f472c1f84c0 0x7f472c0cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:10.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.418+0000 7f472be00640 1 -- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f472c137930 msgr2=0x7f472c1f7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:10.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.418+0000 7f472be00640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f472c137930 0x7f472c1f7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:10.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.418+0000 7f472be00640 1 -- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f472c1f84c0 msgr2=0x7f472c0cdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:10.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.418+0000 7f472be00640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f472c1f84c0 0x7f472c0cdc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:10.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.418+0000 7f472be00640 1 -- 172.21.15.45:0/3172529572 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f472c13f5e0 con 0x7f4710004d40 2024-04-16T16:18:10.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.418+0000 7f472b400640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f472c137930 0x7f472c1f7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:10.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.418+0000 7f4730a00640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f472c1f84c0 0x7f472c0cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:10.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.418+0000 7f472be00640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4710004d40 0x7f472c142cf0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f471c011600 tx=0x7f471c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:10.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.419+0000 7f4723e00640 1 -- 172.21.15.45:0/3172529572 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471c00f920 con 0x7f4710004d40 2024-04-16T16:18:10.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.419+0000 7f4723e00640 1 -- 172.21.15.45:0/3172529572 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f471c00fac0 con 0x7f4710004d40 2024-04-16T16:18:10.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.419+0000 7f4723e00640 1 -- 172.21.15.45:0/3172529572 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471c00fe90 con 0x7f4710004d40 2024-04-16T16:18:10.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.419+0000 7f4731400640 1 -- 172.21.15.45:0/3172529572 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f472c0ce990 con 0x7f4710004d40 2024-04-16T16:18:10.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.419+0000 7f4731400640 1 -- 172.21.15.45:0/3172529572 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f472c13fb80 con 0x7f4710004d40 2024-04-16T16:18:10.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.420+0000 7f4731400640 1 -- 172.21.15.45:0/3172529572 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f472c075780 con 0x7f4710004d40 2024-04-16T16:18:10.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.421+0000 7f4723e00640 1 -- 172.21.15.45:0/3172529572 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f471c015460 con 0x7f4710004d40 2024-04-16T16:18:10.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.422+0000 7f4723e00640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f470c06cbb0 0x7f470c06f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:10.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.423+0000 7f4723e00640 1 -- 172.21.15.45:0/3172529572 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5803+0+0 (secure 0 0 0) 0x7f471c08c730 con 0x7f4710004d40 2024-04-16T16:18:10.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.423+0000 7f472b400640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f470c06cbb0 0x7f470c06f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:10.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.424+0000 7f472b400640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f470c06cbb0 0x7f470c06f070 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4718001c00 tx=0x7f4718000fb0 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:10.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.427+0000 7f4723e00640 1 -- 172.21.15.45:0/3172529572 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f471c019030 con 0x7f4710004d40 2024-04-16T16:18:10.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.675+0000 7f4731400640 1 -- 172.21.15.45:0/3172529572 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f472c0cff60 con 0x7f4710004d40 2024-04-16T16:18:10.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.677+0000 7f4723e00640 1 -- 172.21.15.45:0/3172529572 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+3872 (secure 0 0 0) 0x7f471c05a950 con 0x7f4710004d40 2024-04-16T16:18:10.677 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:10.677 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":57,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/275131331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":275131331},{"type":"v1","addr":"172.21.15.103:6835","nonce":275131331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4074316043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":4074316043},{"type":"v1","addr":"172.21.15.103:6837","nonce":4074316043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":7083,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3337952826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3337952826},{"type":"v1","addr":"172.21.15.45:6833","nonce":3337952826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57},{"gid":7089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2807301325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2807301325},{"type":"v1","addr":"172.21.15.45:6835","nonce":2807301325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57}],"filesystems":[]} 2024-04-16T16:18:10.678 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 57 2024-04-16T16:18:10.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.680+0000 7f4731400640 1 -- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f470c06cbb0 msgr2=0x7f470c06f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:10.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.680+0000 7f4731400640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f470c06cbb0 0x7f470c06f070 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4718001c00 tx=0x7f4718000fb0 comp rx=0 tx=0).stop 2024-04-16T16:18:10.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.680+0000 7f4731400640 1 -- 172.21.15.45:0/3172529572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4710004d40 msgr2=0x7f472c142cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:10.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.680+0000 7f4731400640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4710004d40 0x7f472c142cf0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f471c011600 tx=0x7f471c011ac0 comp rx=0 tx=0).stop 2024-04-16T16:18:10.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.680+0000 7f4731400640 1 -- 172.21.15.45:0/3172529572 shutdown_connections 2024-04-16T16:18:10.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.680+0000 7f4731400640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f470c06cbb0 0x7f470c06f070 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:10.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.680+0000 7f4731400640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f472c1f84c0 0x7f472c0cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:10.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.680+0000 7f4731400640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f472c137930 0x7f472c1f7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:10.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.680+0000 7f4731400640 1 --2- 172.21.15.45:0/3172529572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4710004d40 0x7f472c142cf0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:10.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.680+0000 7f4731400640 1 -- 172.21.15.45:0/3172529572 >> 172.21.15.45:0/3172529572 conn(0x7f472c06b930 msgr2=0x7f472c0c5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:10.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.680+0000 7f4731400640 1 -- 172.21.15.45:0/3172529572 shutdown_connections 2024-04-16T16:18:10.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.680+0000 7f4731400640 1 -- 172.21.15.45:0/3172529572 wait complete. 2024-04-16T16:18:10.700 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-16T16:18:10.700 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-16T16:18:10.700 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-16T16:18:10.700 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:18:10.700 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-16T16:18:10.700 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-16T16:18:10.701 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-16T16:18:10.701 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:18:10.701 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-16T16:18:10.701 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-16T16:18:10.701 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-16T16:18:10.701 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:18:10.702 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-16T16:18:10.702 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-16T16:18:10.702 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-16T16:18:10.702 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:18:10.727 INFO:tasks.ceph.mds.d:Stopped 2024-04-16T16:18:10.727 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-16T16:18:10.728 INFO:tasks.ceph.mds.c:Stopped 2024-04-16T16:18:10.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-16T16:18:10.729 INFO:tasks.ceph.mds.a:Stopped 2024-04-16T16:18:10.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-16T16:18:10.730 INFO:tasks.ceph.mds.b:Stopped 2024-04-16T16:18:10.731 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-16T16:18:10.776 INFO:tasks.ceph.mds.c:Started 2024-04-16T16:18:10.777 DEBUG:teuthology.parallel:result is None 2024-04-16T16:18:10.777 INFO:tasks.ceph.mds.d:Started 2024-04-16T16:18:10.778 DEBUG:teuthology.parallel:result is None 2024-04-16T16:18:10.782 INFO:tasks.ceph.mds.a:Started 2024-04-16T16:18:10.782 DEBUG:teuthology.parallel:result is None 2024-04-16T16:18:10.784 INFO:tasks.ceph.mds.b:Started 2024-04-16T16:18:10.784 DEBUG:teuthology.parallel:result is None 2024-04-16T16:18:10.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-16T16:18:10.939 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2024-04-16T16:18:10.940 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2024-04-16T16:18:10.941 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-16T16:18:10.941 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-16T16:18:10.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.977+0000 7fb10a200640 1 Processor -- start 2024-04-16T16:18:10.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.977+0000 7fb10a200640 1 -- start start 2024-04-16T16:18:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.978+0000 7fb10a200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb1040ba140 0x7fb1040d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.978+0000 7fb10a200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1040d2600 con 0x7fb1040cfc20 2024-04-16T16:18:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.978+0000 7fb10a200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1040ba5d0 con 0x7fb104162090 2024-04-16T16:18:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.978+0000 7fb10a200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb1040ba8b0 con 0x7fb1040ba140 2024-04-16T16:18:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.978+0000 7fb103e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb1040ba140 0x7fb1040d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.979+0000 7fb103e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb1040ba140 0x7fb1040d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39424/0 (socket says 172.21.15.45:39424) 2024-04-16T16:18:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.979+0000 7fb103e00640 1 -- 172.21.15.45:0/882464428 learned_addr learned my addr 172.21.15.45:0/882464428 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.979+0000 7fb108e00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb104162090 0x7fb1040cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:55748/0 (socket says 172.21.15.45:55748) 2024-04-16T16:18:10.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.979+0000 7fb103400640 1 -- 172.21.15.45:0/882464428 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3182299844 0 0) 0x7fb1040ba5d0 con 0x7fb104162090 2024-04-16T16:18:10.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.979+0000 7fb103400640 1 -- 172.21.15.45:0/882464428 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0ec003520 con 0x7fb104162090 2024-04-16T16:18:10.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.979+0000 7fb103e00640 1 -- 172.21.15.45:0/882464428 >> v1:172.21.15.103:6789/0 conn(0x7fb104162090 legacy=0x7fb1040cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:10.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.979+0000 7fb103e00640 1 -- 172.21.15.45:0/882464428 >> v1:172.21.15.45:6789/0 conn(0x7fb1040cfc20 legacy=0x7fb1040d35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:10.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.979+0000 7fb103e00640 1 -- 172.21.15.45:0/882464428 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1040d28a0 con 0x7fb1040ba140 2024-04-16T16:18:10.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.980+0000 7fb103e00640 1 --2- 172.21.15.45:0/882464428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1040ba140 0x7fb1040d30a0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009870 tx=0x7fb0f002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2ee8c326ab574e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:10.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.980+0000 7fb103400640 1 -- 172.21.15.45:0/882464428 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f002f850 con 0x7fb1040ba140 2024-04-16T16:18:10.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.980+0000 7fb103400640 1 -- 172.21.15.45:0/882464428 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f002f9f0 con 0x7fb1040ba140 2024-04-16T16:18:10.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.980+0000 7fb103400640 1 -- 172.21.15.45:0/882464428 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f002fdc0 con 0x7fb1040ba140 2024-04-16T16:18:10.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.981+0000 7fb10a200640 1 -- 172.21.15.45:0/882464428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1040ba140 msgr2=0x7fb1040d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:10.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.981+0000 7fb10a200640 1 --2- 172.21.15.45:0/882464428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1040ba140 0x7fb1040d30a0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009870 tx=0x7fb0f002ba30 comp rx=0 tx=0).stop 2024-04-16T16:18:10.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.981+0000 7fb10a200640 1 -- 172.21.15.45:0/882464428 shutdown_connections 2024-04-16T16:18:10.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.981+0000 7fb10a200640 1 --2- 172.21.15.45:0/882464428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1040ba140 0x7fb1040d30a0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:10.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.981+0000 7fb10a200640 1 -- 172.21.15.45:0/882464428 >> 172.21.15.45:0/882464428 conn(0x7fb1040d1940 msgr2=0x7fb1040d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:10.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.981+0000 7fb10a200640 1 -- 172.21.15.45:0/882464428 shutdown_connections 2024-04-16T16:18:10.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.981+0000 7fb10a200640 1 -- 172.21.15.45:0/882464428 wait complete. 2024-04-16T16:18:10.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.983+0000 7fb10a200640 1 Processor -- start 2024-04-16T16:18:10.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.983+0000 7fb10a200640 1 -- start start 2024-04-16T16:18:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.984+0000 7fb10a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1040cfc20 0x7fb1040ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.984+0000 7fb10a200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb104162090 0x7fb104153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.984+0000 7fb10a200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1040c98c0 0x7fb1040cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.984+0000 7fb10a200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb1041625f0 con 0x7fb104162090 2024-04-16T16:18:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.984+0000 7fb10a200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb104162470 con 0x7fb1040c98c0 2024-04-16T16:18:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.984+0000 7fb10a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb1040d5520 con 0x7fb1040cfc20 2024-04-16T16:18:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.984+0000 7fb103e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb104162090 0x7fb104153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.984+0000 7fb103e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb104162090 0x7fb104153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39430/0 (socket says 172.21.15.45:39430) 2024-04-16T16:18:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.984+0000 7fb103e00640 1 -- 172.21.15.45:0/1908130450 learned_addr learned my addr 172.21.15.45:0/1908130450 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:10.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.984+0000 7fb108e00640 1 --2- 172.21.15.45:0/1908130450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1040cfc20 0x7fb1040ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:10.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.984+0000 7fb109800640 1 --2- 172.21.15.45:0/1908130450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1040c98c0 0x7fb1040cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:10.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.985+0000 7fb103e00640 1 -- 172.21.15.45:0/1908130450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1040cfc20 msgr2=0x7fb1040ca9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:10.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.985+0000 7fb103e00640 1 --2- 172.21.15.45:0/1908130450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1040cfc20 0x7fb1040ca9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:10.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.985+0000 7fb103e00640 1 -- 172.21.15.45:0/1908130450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1040c98c0 msgr2=0x7fb1040cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:10.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.985+0000 7fb103e00640 1 --2- 172.21.15.45:0/1908130450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1040c98c0 0x7fb1040cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:10.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.985+0000 7fb103e00640 1 -- 172.21.15.45:0/1908130450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1040cb090 con 0x7fb104162090 2024-04-16T16:18:10.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.985+0000 7fb103e00640 1 --2- 172.21.15.45:0/1908130450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb104162090 0x7fb104153e20 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009990 tx=0x7fb0f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:10.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.986+0000 7fb100c00640 1 -- 172.21.15.45:0/1908130450 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0047070 con 0x7fb104162090 2024-04-16T16:18:10.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.986+0000 7fb100c00640 1 -- 172.21.15.45:0/1908130450 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f0037c40 con 0x7fb104162090 2024-04-16T16:18:10.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.986+0000 7fb100c00640 1 -- 172.21.15.45:0/1908130450 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0042460 con 0x7fb104162090 2024-04-16T16:18:10.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.986+0000 7fb10a200640 1 -- 172.21.15.45:0/1908130450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1040cbdc0 con 0x7fb104162090 2024-04-16T16:18:10.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.986+0000 7fb10a200640 1 -- 172.21.15.45:0/1908130450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1040cc380 con 0x7fb104162090 2024-04-16T16:18:10.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.987+0000 7fb10a200640 1 -- 172.21.15.45:0/1908130450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb104075780 con 0x7fb104162090 2024-04-16T16:18:10.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.989+0000 7fb100c00640 1 -- 172.21.15.45:0/1908130450 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7fb0f004a070 con 0x7fb104162090 2024-04-16T16:18:10.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.989+0000 7fb100c00640 1 --2- 172.21.15.45:0/1908130450 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fb0e406ca10 0x7fb0e406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:10.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.990+0000 7fb100c00640 1 -- 172.21.15.45:0/1908130450 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5803+0+0 (secure 0 0 0) 0x7fb0f00ae530 con 0x7fb104162090 2024-04-16T16:18:10.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.990+0000 7fb108e00640 1 --2- 172.21.15.45:0/1908130450 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fb0e406ca10 0x7fb0e406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:10.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.991+0000 7fb108e00640 1 --2- 172.21.15.45:0/1908130450 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fb0e406ca10 0x7fb0e406eed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb1040cec00 tx=0x7fb0f8007cf0 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:10.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:10.993+0000 7fb100c00640 1 -- 172.21.15.45:0/1908130450 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb0f0033030 con 0x7fb104162090 2024-04-16T16:18:11.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.212+0000 7fb10a200640 1 -- 172.21.15.45:0/1908130450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fb1040cecd0 con 0x7fb104162090 2024-04-16T16:18:11.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.225+0000 7fb100c00640 1 -- 172.21.15.45:0/1908130450 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v79) ==== 97+0+0 (secure 0 0 0) 0x7fb0f007c070 con 0x7fb104162090 2024-04-16T16:18:11.225 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2024-04-16T16:18:11.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.228+0000 7fb10a200640 1 -- 172.21.15.45:0/1908130450 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fb0e406ca10 msgr2=0x7fb0e406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:11.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.228+0000 7fb10a200640 1 --2- 172.21.15.45:0/1908130450 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fb0e406ca10 0x7fb0e406eed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb1040cec00 tx=0x7fb0f8007cf0 comp rx=0 tx=0).stop 2024-04-16T16:18:11.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.228+0000 7fb10a200640 1 -- 172.21.15.45:0/1908130450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb104162090 msgr2=0x7fb104153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:11.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.228+0000 7fb10a200640 1 --2- 172.21.15.45:0/1908130450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb104162090 0x7fb104153e20 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009990 tx=0x7fb0f00046c0 comp rx=0 tx=0).stop 2024-04-16T16:18:11.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.228+0000 7fb10a200640 1 -- 172.21.15.45:0/1908130450 shutdown_connections 2024-04-16T16:18:11.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.228+0000 7fb10a200640 1 --2- 172.21.15.45:0/1908130450 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fb0e406ca10 0x7fb0e406eed0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:11.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.228+0000 7fb10a200640 1 --2- 172.21.15.45:0/1908130450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1040c98c0 0x7fb1040cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:11.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.228+0000 7fb10a200640 1 --2- 172.21.15.45:0/1908130450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb104162090 0x7fb104153e20 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:11.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.228+0000 7fb10a200640 1 --2- 172.21.15.45:0/1908130450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1040cfc20 0x7fb1040ca9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:11.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.228+0000 7fb10a200640 1 -- 172.21.15.45:0/1908130450 >> 172.21.15.45:0/1908130450 conn(0x7fb1040d1940 msgr2=0x7fb1040ceec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:11.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.229+0000 7fb10a200640 1 -- 172.21.15.45:0/1908130450 shutdown_connections 2024-04-16T16:18:11.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.229+0000 7fb10a200640 1 -- 172.21.15.45:0/1908130450 wait complete. 2024-04-16T16:18:11.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-16T16:18:11.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.463+0000 7f7057200640 1 Processor -- start 2024-04-16T16:18:11.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.463+0000 7f7057200640 1 -- start start 2024-04-16T16:18:11.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.464+0000 7f7057200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f705806dfb0 0x7f70580c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.465+0000 7f7057200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f70580c5bb0 0x7f705815d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.465+0000 7f7057200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f705815c700 con 0x7f7058144ff0 2024-04-16T16:18:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.465+0000 7f7057200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f705815d2f0 con 0x7f70580c5bb0 2024-04-16T16:18:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.465+0000 7f7057200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f705815c9c0 con 0x7f705806dfb0 2024-04-16T16:18:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.465+0000 7f7056800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f70580c5bb0 0x7f705815d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.465+0000 7f7056800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f70580c5bb0 0x7f705815d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39440/0 (socket says 172.21.15.45:39440) 2024-04-16T16:18:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.465+0000 7f7056800640 1 -- 172.21.15.45:0/2299063446 learned_addr learned my addr 172.21.15.45:0/2299063446 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.465+0000 7f704fe00640 1 --2- 172.21.15.45:0/2299063446 >> v2:172.21.15.103:3300/0 conn(0x7f705806dfb0 0x7f70580c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.465+0000 7f7056800640 1 -- 172.21.15.45:0/2299063446 >> v2:172.21.15.103:3300/0 conn(0x7f705806dfb0 msgr2=0x7f70580c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.465+0000 7f7056800640 1 --2- 172.21.15.45:0/2299063446 >> v2:172.21.15.103:3300/0 conn(0x7f705806dfb0 0x7f70580c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:11.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.465+0000 7f7056800640 1 -- 172.21.15.45:0/2299063446 >> v1:172.21.15.45:6789/0 conn(0x7f7058144ff0 legacy=0x7f70580cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:11.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.466+0000 7f7056800640 1 -- 172.21.15.45:0/2299063446 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705815cca0 con 0x7f70580c5bb0 2024-04-16T16:18:11.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.466+0000 7f704fe00640 1 --2- 172.21.15.45:0/2299063446 >> v2:172.21.15.103:3300/0 conn(0x7f705806dfb0 0x7f70580c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:11.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.467+0000 7f7056800640 1 --2- 172.21.15.45:0/2299063446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70580c5bb0 0x7f705815d780 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f70580701c0 tx=0x7f705002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a7541fec6ac0482 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:11.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.467+0000 7f7055400640 1 -- 172.21.15.45:0/2299063446 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705002cb10 con 0x7f70580c5bb0 2024-04-16T16:18:11.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.467+0000 7f7055400640 1 -- 172.21.15.45:0/2299063446 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f705002ccb0 con 0x7f70580c5bb0 2024-04-16T16:18:11.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.467+0000 7f7055400640 1 -- 172.21.15.45:0/2299063446 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705002ce60 con 0x7f70580c5bb0 2024-04-16T16:18:11.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.467+0000 7f7057200640 1 -- 172.21.15.45:0/2299063446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70580c5bb0 msgr2=0x7f705815d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:11.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.468+0000 7f7057200640 1 --2- 172.21.15.45:0/2299063446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70580c5bb0 0x7f705815d780 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f70580701c0 tx=0x7f705002bf70 comp rx=0 tx=0).stop 2024-04-16T16:18:11.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.468+0000 7f7057200640 1 -- 172.21.15.45:0/2299063446 shutdown_connections 2024-04-16T16:18:11.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.468+0000 7f7057200640 1 --2- 172.21.15.45:0/2299063446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70580c5bb0 0x7f705815d780 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:11.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.468+0000 7f7057200640 1 --2- 172.21.15.45:0/2299063446 >> v2:172.21.15.103:3300/0 conn(0x7f705806dfb0 0x7f70580c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:11.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.468+0000 7f7057200640 1 -- 172.21.15.45:0/2299063446 >> 172.21.15.45:0/2299063446 conn(0x7f705806b930 msgr2=0x7f705806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:11.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.468+0000 7f7057200640 1 -- 172.21.15.45:0/2299063446 shutdown_connections 2024-04-16T16:18:11.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.468+0000 7f7057200640 1 -- 172.21.15.45:0/2299063446 wait complete. 2024-04-16T16:18:11.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.469+0000 7f7057200640 1 Processor -- start 2024-04-16T16:18:11.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.470+0000 7f7057200640 1 -- start start 2024-04-16T16:18:11.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.470+0000 7f7057200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f705806dfb0 0x7f7058158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:11.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.471+0000 7f7057200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70580c5bb0 0x7f70580c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:11.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.471+0000 7f7057200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7058144ff0 0x7f70580cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:11.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.471+0000 7f7057200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f705815dcc0 con 0x7f705806dfb0 2024-04-16T16:18:11.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.471+0000 7f7057200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f70581453d0 con 0x7f7058144ff0 2024-04-16T16:18:11.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.471+0000 7f7057200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f705815de40 con 0x7f70580c5bb0 2024-04-16T16:18:11.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.471+0000 7f7055e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f705806dfb0 0x7f7058158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:11.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.471+0000 7f7055e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f705806dfb0 0x7f7058158b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39452/0 (socket says 172.21.15.45:39452) 2024-04-16T16:18:11.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.471+0000 7f7055e00640 1 -- 172.21.15.45:0/2755605642 learned_addr learned my addr 172.21.15.45:0/2755605642 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:11.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.471+0000 7f7055e00640 1 -- 172.21.15.45:0/2755605642 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70580c5bb0 msgr2=0x7f70580c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:11.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.471+0000 7f7055e00640 1 --2- 172.21.15.45:0/2755605642 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70580c5bb0 0x7f70580c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:11.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.471+0000 7f7055e00640 1 -- 172.21.15.45:0/2755605642 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7058144ff0 msgr2=0x7f70580cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:11.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.471+0000 7f7055e00640 1 --2- 172.21.15.45:0/2755605642 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7058144ff0 0x7f70580cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:11.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.471+0000 7f7055e00640 1 -- 172.21.15.45:0/2755605642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70580c9e10 con 0x7f705806dfb0 2024-04-16T16:18:11.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.472+0000 7f7055e00640 1 --2- 172.21.15.45:0/2755605642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f705806dfb0 0x7f7058158b20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f704800c3c0 tx=0x7f704800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:11.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.472+0000 7f704e000640 1 -- 172.21.15.45:0/2755605642 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70480052d0 con 0x7f705806dfb0 2024-04-16T16:18:11.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.472+0000 7f704e000640 1 -- 172.21.15.45:0/2755605642 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7048005490 con 0x7f705806dfb0 2024-04-16T16:18:11.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.472+0000 7f704e000640 1 -- 172.21.15.45:0/2755605642 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7048007260 con 0x7f705806dfb0 2024-04-16T16:18:11.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.472+0000 7f7057200640 1 -- 172.21.15.45:0/2755605642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70580ca0f0 con 0x7f705806dfb0 2024-04-16T16:18:11.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.472+0000 7f7057200640 1 -- 172.21.15.45:0/2755605642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70580ca6b0 con 0x7f705806dfb0 2024-04-16T16:18:11.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.473+0000 7f7057200640 1 -- 172.21.15.45:0/2755605642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7058075780 con 0x7f705806dfb0 2024-04-16T16:18:11.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.474+0000 7f704e000640 1 -- 172.21.15.45:0/2755605642 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f7048007400 con 0x7f705806dfb0 2024-04-16T16:18:11.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.475+0000 7f704e000640 1 --2- 172.21.15.45:0/2755605642 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f703406ca10 0x7f703406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:11.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.475+0000 7f704e000640 1 -- 172.21.15.45:0/2755605642 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4857+0+0 (secure 0 0 0) 0x7f704808ade0 con 0x7f705806dfb0 2024-04-16T16:18:11.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.476+0000 7f704fe00640 1 --2- 172.21.15.45:0/2755605642 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f703406ca10 0x7f703406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:11.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.477+0000 7f704fe00640 1 --2- 172.21.15.45:0/2755605642 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f703406ca10 0x7f703406eed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7040006fd0 tx=0x7f7040008040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:11.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.480+0000 7f704e000640 1 -- 172.21.15.45:0/2755605642 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f70480594e0 con 0x7f705806dfb0 2024-04-16T16:18:11.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.761+0000 7f7057200640 1 -- 172.21.15.45:0/2755605642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f705815e9e0 con 0x7f705806dfb0 2024-04-16T16:18:11.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.762+0000 7f704e000640 1 -- 172.21.15.45:0/2755605642 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) ==== 80+0+5238 (secure 0 0 0) 0x7f7048058de0 con 0x7f705806dfb0 2024-04-16T16:18:11.762 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:11.762 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:18:11.762 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2024-04-16T16:18:11.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.762 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2024-04-16T16:18:11.763 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mpjReORAAJMoNf4qvfuxPSiV9+VlKfg==", 2024-04-16T16:18:11.763 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:18:11.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:18:11.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:18:11.763 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:18:11.763 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.763 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2024-04-16T16:18:11.763 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mxHeBIBAA3G8maaK65zaOaqXpAOA6xA==", 2024-04-16T16:18:11.763 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:18:11.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:18:11.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:18:11.764 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:18:11.764 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.764 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2024-04-16T16:18:11.764 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5m+AaODxAA08XSjc44/i2KYExYQSukaQ==", 2024-04-16T16:18:11.764 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:18:11.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:18:11.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:18:11.765 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:18:11.765 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.765 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2024-04-16T16:18:11.765 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mEQ4UMRAAOslSsxQ17bOxPrI6QMlRRw==", 2024-04-16T16:18:11.765 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.765 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:18:11.765 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:18:11.765 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:18:11.765 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:18:11.765 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.766 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2024-04-16T16:18:11.766 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBuox5m3l3OHxAA+UjwZx8Mn6tCu95TTYIG3w==", 2024-04-16T16:18:11.766 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.766 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:18:11.766 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:18:11.766 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:18:11.766 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.766 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2024-04-16T16:18:11.766 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxox5mozt1LhAAJL5NtMQdPVBtKOaVlMQBtw==", 2024-04-16T16:18:11.766 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:18:11.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:18:11.767 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:18:11.767 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.767 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2024-04-16T16:18:11.767 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB1ox5mfE7kABAAIvgZ5Wx+eIJ6QNKj2GVdpw==", 2024-04-16T16:18:11.767 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:18:11.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:18:11.767 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:18:11.768 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.768 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2024-04-16T16:18:11.768 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB4ox5mEvxcDRAAO0yXBKvsC7hZ6bpOAJyuag==", 2024-04-16T16:18:11.768 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.768 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:18:11.768 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:18:11.768 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:18:11.768 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.768 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2024-04-16T16:18:11.768 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+ox5mZubrORAAcGJLyRN5Ob9aBnopumnEUg==", 2024-04-16T16:18:11.769 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.769 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:18:11.769 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:18:11.769 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:18:11.769 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.769 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2024-04-16T16:18:11.769 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCCox5mcRqsChAACktmuUHvqWkp/mHT3UcjRw==", 2024-04-16T16:18:11.769 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.769 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:18:11.769 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:18:11.769 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:18:11.769 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.770 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2024-04-16T16:18:11.770 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFox5mT52FFhAAChhDDfNZmIMgxE1j90LZpQ==", 2024-04-16T16:18:11.770 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.770 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:18:11.770 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:18:11.770 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:18:11.770 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.770 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2024-04-16T16:18:11.770 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCIox5mC+EDIxAAxtvM0on00lxTLaoRjN0Gfw==", 2024-04-16T16:18:11.770 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.771 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:18:11.771 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:18:11.771 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:18:11.771 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.771 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2024-04-16T16:18:11.771 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ==", 2024-04-16T16:18:11.771 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.771 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:18:11.771 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:18:11.771 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-16T16:18:11.772 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.772 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2024-04-16T16:18:11.772 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mYpVUGhAAYdR+40O6zbWqgbI55F0FYA==", 2024-04-16T16:18:11.772 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.772 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:18:11.772 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:18:11.772 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:18:11.772 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.772 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2024-04-16T16:18:11.772 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mWdlUDxAAuO2FsGH1/E+qP1xcyUEe3A==", 2024-04-16T16:18:11.773 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.773 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:18:11.773 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:18:11.773 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:18:11.773 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.773 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2024-04-16T16:18:11.773 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5miPdMJBAAqjKpctNYu9ot1lff3rouxg==", 2024-04-16T16:18:11.773 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.773 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:18:11.773 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:18:11.773 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:18:11.773 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.774 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2024-04-16T16:18:11.774 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBpox5mO4PMLBAA/0YbQCfHujgdEzpXs0nOLw==", 2024-04-16T16:18:11.774 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.774 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:18:11.774 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:18:11.774 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:18:11.774 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:18:11.774 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.774 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2024-04-16T16:18:11.775 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mGevaLBAAPpA6g+Fdi/XHqAqG0QZNag==", 2024-04-16T16:18:11.775 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.775 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:18:11.775 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:18:11.775 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:18:11.775 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:18:11.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:11.775 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2024-04-16T16:18:11.775 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5moiNRIRAAz5JzOrjKNLkcphT+iMo+zw==", 2024-04-16T16:18:11.775 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:18:11.775 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:18:11.775 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:18:11.775 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:18:11.776 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.776 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:11.776 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-16T16:18:11.776 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:18:11.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.764+0000 7f7057200640 1 -- 172.21.15.45:0/2755605642 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f703406ca10 msgr2=0x7f703406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:11.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.764+0000 7f7057200640 1 --2- 172.21.15.45:0/2755605642 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f703406ca10 0x7f703406eed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7040006fd0 tx=0x7f7040008040 comp rx=0 tx=0).stop 2024-04-16T16:18:11.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.764+0000 7f7057200640 1 -- 172.21.15.45:0/2755605642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f705806dfb0 msgr2=0x7f7058158b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:11.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.764+0000 7f7057200640 1 --2- 172.21.15.45:0/2755605642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f705806dfb0 0x7f7058158b20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f704800c3c0 tx=0x7f704800c880 comp rx=0 tx=0).stop 2024-04-16T16:18:11.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.764+0000 7f7057200640 1 -- 172.21.15.45:0/2755605642 shutdown_connections 2024-04-16T16:18:11.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.764+0000 7f7057200640 1 --2- 172.21.15.45:0/2755605642 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f703406ca10 0x7f703406eed0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:11.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.764+0000 7f7057200640 1 --2- 172.21.15.45:0/2755605642 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7058144ff0 0x7f70580cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:11.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.764+0000 7f7057200640 1 --2- 172.21.15.45:0/2755605642 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70580c5bb0 0x7f70580c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:11.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.764+0000 7f7057200640 1 --2- 172.21.15.45:0/2755605642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f705806dfb0 0x7f7058158b20 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:11.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.764+0000 7f7057200640 1 -- 172.21.15.45:0/2755605642 >> 172.21.15.45:0/2755605642 conn(0x7f705806b930 msgr2=0x7f705815bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:11.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.765+0000 7f7057200640 1 -- 172.21.15.45:0/2755605642 shutdown_connections 2024-04-16T16:18:11.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:11.765+0000 7f7057200640 1 -- 172.21.15.45:0/2755605642 wait complete. 2024-04-16T16:18:11.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:18:12.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.008+0000 7f7fe6000640 1 Processor -- start 2024-04-16T16:18:12.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.008+0000 7f7fe6000640 1 -- start start 2024-04-16T16:18:12.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fe6000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe0146220 0x7f7fe015ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fe6000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7fe01608a0 0x7f7fe0160d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fe6000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fe015d360 con 0x7f7fe006f7a0 2024-04-16T16:18:12.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fe6000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7fe0160410 con 0x7f7fe01608a0 2024-04-16T16:18:12.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fe6000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fe015d640 con 0x7f7fe0146220 2024-04-16T16:18:12.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdfe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7fe006f7a0 0x7f7fe006df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46828/0 (socket says 172.21.15.45:46828) 2024-04-16T16:18:12.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdfe00640 1 -- 172.21.15.45:0/3489085413 learned_addr learned my addr 172.21.15.45:0/3489085413 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:12.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fe4c00640 1 --2- 172.21.15.45:0/3489085413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe0146220 0x7f7fe015ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fe5600640 1 --2- 172.21.15.45:0/3489085413 >> v2:172.21.15.103:3300/0 conn(0x7f7fe01608a0 0x7f7fe0160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdf400640 1 -- 172.21.15.45:0/3489085413 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4062114374 0 0) 0x7f7fe015d360 con 0x7f7fe006f7a0 2024-04-16T16:18:12.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdf400640 1 -- 172.21.15.45:0/3489085413 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fc8003520 con 0x7f7fe006f7a0 2024-04-16T16:18:12.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdf400640 1 -- 172.21.15.45:0/3489085413 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2569871668 0 0) 0x7f7fc8003520 con 0x7f7fe006f7a0 2024-04-16T16:18:12.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdf400640 1 -- 172.21.15.45:0/3489085413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe0146220 msgr2=0x7f7fe015ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:12.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdf400640 1 --2- 172.21.15.45:0/3489085413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe0146220 0x7f7fe015ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdf400640 1 -- 172.21.15.45:0/3489085413 >> v2:172.21.15.103:3300/0 conn(0x7f7fe01608a0 msgr2=0x7f7fe0160d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:12.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdf400640 1 --2- 172.21.15.45:0/3489085413 >> v2:172.21.15.103:3300/0 conn(0x7f7fe01608a0 0x7f7fe0160d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdf400640 1 -- 172.21.15.45:0/3489085413 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe015d920 con 0x7f7fe006f7a0 2024-04-16T16:18:12.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdf400640 1 -- 172.21.15.45:0/3489085413 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f7fcc001670 con 0x7f7fe006f7a0 2024-04-16T16:18:12.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdf400640 1 -- 172.21.15.45:0/3489085413 >> v1:172.21.15.45:6789/0 conn(0x7f7fe006f7a0 legacy=0x7f7fe006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdf400640 1 --2- 172.21.15.45:0/3489085413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe0146220 0x7f7fc80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdf400640 1 --2- 172.21.15.45:0/3489085413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc8003f30 0x7f7fc8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdf400640 1 --2- 172.21.15.45:0/3489085413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fc8004d40 0x7f7fc80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.010+0000 7f7fdf400640 1 -- 172.21.15.45:0/3489085413 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe0160410 con 0x7f7fc8003f30 2024-04-16T16:18:12.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.011+0000 7f7fdf400640 1 -- 172.21.15.45:0/3489085413 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe015d640 con 0x7f7fc8004d40 2024-04-16T16:18:12.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.011+0000 7f7fdf400640 1 -- 172.21.15.45:0/3489085413 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fc800b9c0 con 0x7f7fe0146220 2024-04-16T16:18:12.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.011+0000 7f7fdfe00640 1 --2- 172.21.15.45:0/3489085413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc8003f30 0x7f7fc8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.011+0000 7f7fe4c00640 1 --2- 172.21.15.45:0/3489085413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe0146220 0x7f7fc80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.011+0000 7f7fdfe00640 1 -- 172.21.15.45:0/3489085413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe0146220 msgr2=0x7f7fc80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:12.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.011+0000 7f7fdfe00640 1 --2- 172.21.15.45:0/3489085413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe0146220 0x7f7fc80039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.011+0000 7f7fdfe00640 1 -- 172.21.15.45:0/3489085413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fc8004d40 msgr2=0x7f7fc80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:12.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.011+0000 7f7fdfe00640 1 --2- 172.21.15.45:0/3489085413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fc8004d40 0x7f7fc80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.011+0000 7f7fdfe00640 1 -- 172.21.15.45:0/3489085413 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc8003520 con 0x7f7fc8003f30 2024-04-16T16:18:12.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.012+0000 7f7fdfe00640 1 --2- 172.21.15.45:0/3489085413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc8003f30 0x7f7fc8004800 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00c210 tx=0x7f7fc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8342217ccab052 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:12.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.013+0000 7f7fdf400640 1 -- 172.21.15.45:0/3489085413 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc007d50 con 0x7f7fc8003f30 2024-04-16T16:18:12.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.013+0000 7f7fdf400640 1 -- 172.21.15.45:0/3489085413 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fcc030070 con 0x7f7fc8003f30 2024-04-16T16:18:12.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.013+0000 7f7fdf400640 1 -- 172.21.15.45:0/3489085413 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc02cdb0 con 0x7f7fc8003f30 2024-04-16T16:18:12.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.013+0000 7f7fe6000640 1 -- 172.21.15.45:0/3489085413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc8003f30 msgr2=0x7f7fc8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.013+0000 7f7fe6000640 1 --2- 172.21.15.45:0/3489085413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc8003f30 0x7f7fc8004800 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00c210 tx=0x7f7fc8005830 comp rx=0 tx=0).stop 2024-04-16T16:18:12.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.013+0000 7f7fe5600640 1 -- 172.21.15.45:0/3489085413 reap_dead start 2024-04-16T16:18:12.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.013+0000 7f7fe6000640 1 -- 172.21.15.45:0/3489085413 shutdown_connections 2024-04-16T16:18:12.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.013+0000 7f7fe6000640 1 -- 172.21.15.45:0/3489085413 >> 172.21.15.45:0/3489085413 conn(0x7f7fe006b930 msgr2=0x7f7fe006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:12.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.014+0000 7f7fe6000640 1 -- 172.21.15.45:0/3489085413 shutdown_connections 2024-04-16T16:18:12.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.014+0000 7f7fe6000640 1 -- 172.21.15.45:0/3489085413 wait complete. 2024-04-16T16:18:12.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.015+0000 7f7fe6000640 1 Processor -- start 2024-04-16T16:18:12.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.015+0000 7f7fe6000640 1 -- start start 2024-04-16T16:18:12.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.016+0000 7f7fe6000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fc8003f30 0x7f7fe013c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.016+0000 7f7fe6000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fe01397e0 0x7f7fe013e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.016+0000 7f7fe6000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fe0139c40 0x7f7fe013e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.016+0000 7f7fe6000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe006fc20 con 0x7f7fe01397e0 2024-04-16T16:18:12.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.016+0000 7f7fe6000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe015dc00 con 0x7f7fe0139c40 2024-04-16T16:18:12.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.016+0000 7f7fe6000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fe006fda0 con 0x7f7fc8003f30 2024-04-16T16:18:12.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.017+0000 7f7fdfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fe01397e0 0x7f7fe013e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.017+0000 7f7fdfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fe01397e0 0x7f7fe013e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39480/0 (socket says 172.21.15.45:39480) 2024-04-16T16:18:12.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.017+0000 7f7fdfe00640 1 -- 172.21.15.45:0/262473279 learned_addr learned my addr 172.21.15.45:0/262473279 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:12.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.017+0000 7f7fe4c00640 1 --2- 172.21.15.45:0/262473279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fc8003f30 0x7f7fe013c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.017+0000 7f7fe5600640 1 --2- 172.21.15.45:0/262473279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fe0139c40 0x7f7fe013e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.017+0000 7f7fdfe00640 1 -- 172.21.15.45:0/262473279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fc8003f30 msgr2=0x7f7fe013c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.017+0000 7f7fdfe00640 1 --2- 172.21.15.45:0/262473279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fc8003f30 0x7f7fe013c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.017+0000 7f7fdfe00640 1 -- 172.21.15.45:0/262473279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fe0139c40 msgr2=0x7f7fe013e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.017+0000 7f7fdfe00640 1 --2- 172.21.15.45:0/262473279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fe0139c40 0x7f7fe013e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.017+0000 7f7fdfe00640 1 -- 172.21.15.45:0/262473279 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe00cdb60 con 0x7f7fe01397e0 2024-04-16T16:18:12.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.018+0000 7f7fdfe00640 1 --2- 172.21.15.45:0/262473279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fe01397e0 0x7f7fe013e270 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc001970 tx=0x7f7fcc001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:12.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.018+0000 7f7fdcc00640 1 -- 172.21.15.45:0/262473279 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc001c90 con 0x7f7fe01397e0 2024-04-16T16:18:12.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.018+0000 7f7fdcc00640 1 -- 172.21.15.45:0/262473279 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fcc004bf0 con 0x7f7fe01397e0 2024-04-16T16:18:12.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.018+0000 7f7fdcc00640 1 -- 172.21.15.45:0/262473279 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc007d50 con 0x7f7fe01397e0 2024-04-16T16:18:12.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.018+0000 7f7fe6000640 1 -- 172.21.15.45:0/262473279 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe00b2d80 con 0x7f7fe01397e0 2024-04-16T16:18:12.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.018+0000 7f7fe6000640 1 -- 172.21.15.45:0/262473279 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe00b3340 con 0x7f7fe01397e0 2024-04-16T16:18:12.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.019+0000 7f7fe6000640 1 -- 172.21.15.45:0/262473279 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fe0075780 con 0x7f7fe01397e0 2024-04-16T16:18:12.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.020+0000 7f7fdcc00640 1 -- 172.21.15.45:0/262473279 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f7fcc007070 con 0x7f7fe01397e0 2024-04-16T16:18:12.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.021+0000 7f7fdcc00640 1 --2- 172.21.15.45:0/262473279 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7fc006ca10 0x7f7fc006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.022+0000 7f7fdcc00640 1 -- 172.21.15.45:0/262473279 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4857+0+0 (secure 0 0 0) 0x7f7fcc0ad020 con 0x7f7fe01397e0 2024-04-16T16:18:12.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.022+0000 7f7fe4c00640 1 --2- 172.21.15.45:0/262473279 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7fc006ca10 0x7f7fc006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.023+0000 7f7fe4c00640 1 --2- 172.21.15.45:0/262473279 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7fc006ca10 0x7f7fc006eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4009930 tx=0x7f7fd4008040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:12.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.026+0000 7f7fdcc00640 1 -- 172.21.15.45:0/262473279 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7fcc07b720 con 0x7f7fe01397e0 2024-04-16T16:18:12.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.237+0000 7f7fe6000640 1 -- 172.21.15.45:0/262473279 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7fe00cdf40 con 0x7f7fe01397e0 2024-04-16T16:18:12.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.239+0000 7f7fdcc00640 1 -- 172.21.15.45:0/262473279 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+12892 (secure 0 0 0) 0x7f7fcc07b020 con 0x7f7fe01397e0 2024-04-16T16:18:12.239 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:12.239 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":81,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:18:11.242678+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:18:12.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.241+0000 7f7fe6000640 1 -- 172.21.15.45:0/262473279 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7fc006ca10 msgr2=0x7f7fc006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.241+0000 7f7fe6000640 1 --2- 172.21.15.45:0/262473279 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7fc006ca10 0x7f7fc006eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4009930 tx=0x7f7fd4008040 comp rx=0 tx=0).stop 2024-04-16T16:18:12.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.241+0000 7f7fe6000640 1 -- 172.21.15.45:0/262473279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fe01397e0 msgr2=0x7f7fe013e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.241+0000 7f7fe6000640 1 --2- 172.21.15.45:0/262473279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fe01397e0 0x7f7fe013e270 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc001970 tx=0x7f7fcc001a40 comp rx=0 tx=0).stop 2024-04-16T16:18:12.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.241+0000 7f7fe6000640 1 -- 172.21.15.45:0/262473279 shutdown_connections 2024-04-16T16:18:12.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.241+0000 7f7fe6000640 1 --2- 172.21.15.45:0/262473279 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7fc006ca10 0x7f7fc006eed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.241+0000 7f7fe6000640 1 --2- 172.21.15.45:0/262473279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fe0139c40 0x7f7fe013e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.241+0000 7f7fe6000640 1 --2- 172.21.15.45:0/262473279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fe01397e0 0x7f7fe013e270 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.241+0000 7f7fe6000640 1 --2- 172.21.15.45:0/262473279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fc8003f30 0x7f7fe013c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.241+0000 7f7fe6000640 1 -- 172.21.15.45:0/262473279 >> 172.21.15.45:0/262473279 conn(0x7f7fe006b930 msgr2=0x7f7fe015f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:12.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.242+0000 7f7fe6000640 1 -- 172.21.15.45:0/262473279 shutdown_connections 2024-04-16T16:18:12.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.242+0000 7f7fe6000640 1 -- 172.21.15.45:0/262473279 wait complete. 2024-04-16T16:18:12.261 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:18:12.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:18:12.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.479+0000 7fa463200640 1 Processor -- start 2024-04-16T16:18:12.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.479+0000 7fa463200640 1 -- start start 2024-04-16T16:18:12.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.480+0000 7fa463200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa45c06bd90 0x7fa45c143f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.480+0000 7fa463200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa45c071a00 con 0x7fa45c142ed0 2024-04-16T16:18:12.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.480+0000 7fa463200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa45c0cf660 con 0x7fa45c06b930 2024-04-16T16:18:12.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.480+0000 7fa463200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa45c0cf940 con 0x7fa45c06bd90 2024-04-16T16:18:12.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.481+0000 7fa461e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa45c142ed0 0x7fa45c0712f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46842/0 (socket says 172.21.15.45:46842) 2024-04-16T16:18:12.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.481+0000 7fa461e00640 1 -- 172.21.15.45:0/1126762711 learned_addr learned my addr 172.21.15.45:0/1126762711 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:12.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.481+0000 7fa462800640 1 --2- 172.21.15.45:0/1126762711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa45c06bd90 0x7fa45c143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.481+0000 7fa460a00640 1 -- 172.21.15.45:0/1126762711 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4150135601 0 0) 0x7fa45c071a00 con 0x7fa45c142ed0 2024-04-16T16:18:12.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.481+0000 7fa460a00640 1 -- 172.21.15.45:0/1126762711 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa448003520 con 0x7fa45c142ed0 2024-04-16T16:18:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.481+0000 7fa460a00640 1 -- 172.21.15.45:0/1126762711 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 237287803 0 0) 0x7fa448003520 con 0x7fa45c142ed0 2024-04-16T16:18:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.481+0000 7fa460a00640 1 -- 172.21.15.45:0/1126762711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa45c06bd90 msgr2=0x7fa45c143f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.481+0000 7fa460a00640 1 --2- 172.21.15.45:0/1126762711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa45c06bd90 0x7fa45c143f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.481+0000 7fa460a00640 1 -- 172.21.15.45:0/1126762711 >> v1:172.21.15.103:6789/0 conn(0x7fa45c06b930 legacy=0x7fa45c1437f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.481+0000 7fa460a00640 1 -- 172.21.15.45:0/1126762711 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa45c0cfc20 con 0x7fa45c142ed0 2024-04-16T16:18:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.482+0000 7fa460a00640 1 -- 172.21.15.45:0/1126762711 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fa454001a10 con 0x7fa45c142ed0 2024-04-16T16:18:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.482+0000 7fa460a00640 1 -- 172.21.15.45:0/1126762711 >> v1:172.21.15.45:6789/0 conn(0x7fa45c142ed0 legacy=0x7fa45c0712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.482+0000 7fa460a00640 1 --2- 172.21.15.45:0/1126762711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa45c06bd90 0x7fa4480039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.482+0000 7fa460a00640 1 --2- 172.21.15.45:0/1126762711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448004060 0x7fa448004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.482+0000 7fa460a00640 1 --2- 172.21.15.45:0/1126762711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa448004d40 0x7fa4480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.482+0000 7fa460a00640 1 -- 172.21.15.45:0/1126762711 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa45c0cf940 con 0x7fa448004d40 2024-04-16T16:18:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.482+0000 7fa460a00640 1 -- 172.21.15.45:0/1126762711 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa44800bc00 con 0x7fa448004060 2024-04-16T16:18:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.482+0000 7fa460a00640 1 -- 172.21.15.45:0/1126762711 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa44800be00 con 0x7fa45c06bd90 2024-04-16T16:18:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.482+0000 7fa462800640 1 --2- 172.21.15.45:0/1126762711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa448004d40 0x7fa4480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.482+0000 7fa461e00640 1 --2- 172.21.15.45:0/1126762711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa45c06bd90 0x7fa4480039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.482+0000 7fa462800640 1 -- 172.21.15.45:0/1126762711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa45c06bd90 msgr2=0x7fa4480039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:12.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.482+0000 7fa462800640 1 --2- 172.21.15.45:0/1126762711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa45c06bd90 0x7fa4480039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.482+0000 7fa462800640 1 -- 172.21.15.45:0/1126762711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448004060 msgr2=0x7fa448004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:12.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.482+0000 7fa462800640 1 --2- 172.21.15.45:0/1126762711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448004060 0x7fa448004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.482+0000 7fa462800640 1 -- 172.21.15.45:0/1126762711 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa448003520 con 0x7fa448004d40 2024-04-16T16:18:12.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.485+0000 7fa462800640 1 --2- 172.21.15.45:0/1126762711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa448004d40 0x7fa4480051c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fa45800ba30 tx=0x7fa448005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6731a77a3dd126 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:12.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.485+0000 7fa460a00640 1 -- 172.21.15.45:0/1126762711 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458036020 con 0x7fa448004d40 2024-04-16T16:18:12.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.485+0000 7fa460a00640 1 -- 172.21.15.45:0/1126762711 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa458032640 con 0x7fa448004d40 2024-04-16T16:18:12.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.485+0000 7fa460a00640 1 -- 172.21.15.45:0/1126762711 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45800b600 con 0x7fa448004d40 2024-04-16T16:18:12.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.485+0000 7fa463200640 1 -- 172.21.15.45:0/1126762711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa448004d40 msgr2=0x7fa4480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.485+0000 7fa463200640 1 --2- 172.21.15.45:0/1126762711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa448004d40 0x7fa4480051c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fa45800ba30 tx=0x7fa448005830 comp rx=0 tx=0).stop 2024-04-16T16:18:12.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.485+0000 7fa462800640 1 -- 172.21.15.45:0/1126762711 reap_dead start 2024-04-16T16:18:12.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.485+0000 7fa463200640 1 -- 172.21.15.45:0/1126762711 shutdown_connections 2024-04-16T16:18:12.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.485+0000 7fa463200640 1 -- 172.21.15.45:0/1126762711 >> 172.21.15.45:0/1126762711 conn(0x7fa45c06f0e0 msgr2=0x7fa45c06f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:12.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.485+0000 7fa463200640 1 -- 172.21.15.45:0/1126762711 shutdown_connections 2024-04-16T16:18:12.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.485+0000 7fa463200640 1 -- 172.21.15.45:0/1126762711 wait complete. 2024-04-16T16:18:12.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.486+0000 7fa463200640 1 Processor -- start 2024-04-16T16:18:12.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.487+0000 7fa463200640 1 -- start start 2024-04-16T16:18:12.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.487+0000 7fa463200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448004d40 0x7fa45c138e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.487+0000 7fa463200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa45c1f49f0 0x7fa45c15cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.487+0000 7fa463200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa45c15d490 0x7fa45c13e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.487+0000 7fa463200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa45c06c200 con 0x7fa45c15d490 2024-04-16T16:18:12.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.487+0000 7fa463200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa45c0d01e0 con 0x7fa448004d40 2024-04-16T16:18:12.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.487+0000 7fa463200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa45c0cf310 con 0x7fa45c1f49f0 2024-04-16T16:18:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.488+0000 7fa462800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa45c15d490 0x7fa45c13e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.488+0000 7fa462800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa45c15d490 0x7fa45c13e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39484/0 (socket says 172.21.15.45:39484) 2024-04-16T16:18:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.488+0000 7fa462800640 1 -- 172.21.15.45:0/2084874979 learned_addr learned my addr 172.21.15.45:0/2084874979 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.488+0000 7fa461e00640 1 --2- 172.21.15.45:0/2084874979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448004d40 0x7fa45c138e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.488+0000 7fa462800640 1 -- 172.21.15.45:0/2084874979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa45c1f49f0 msgr2=0x7fa45c15cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.488+0000 7fa461400640 1 --2- 172.21.15.45:0/2084874979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa45c1f49f0 0x7fa45c15cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.488+0000 7fa462800640 1 --2- 172.21.15.45:0/2084874979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa45c1f49f0 0x7fa45c15cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.488+0000 7fa462800640 1 -- 172.21.15.45:0/2084874979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448004d40 msgr2=0x7fa45c138e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.488+0000 7fa462800640 1 --2- 172.21.15.45:0/2084874979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448004d40 0x7fa45c138e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.488+0000 7fa462800640 1 -- 172.21.15.45:0/2084874979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa45c15d980 con 0x7fa45c15d490 2024-04-16T16:18:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.488+0000 7fa461e00640 1 --2- 172.21.15.45:0/2084874979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448004d40 0x7fa45c138e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.488+0000 7fa462800640 1 --2- 172.21.15.45:0/2084874979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa45c15d490 0x7fa45c13e590 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fa45800bb50 tx=0x7fa458004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.489+0000 7fa452000640 1 -- 172.21.15.45:0/2084874979 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458042070 con 0x7fa45c15d490 2024-04-16T16:18:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.489+0000 7fa452000640 1 -- 172.21.15.45:0/2084874979 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa458004020 con 0x7fa45c15d490 2024-04-16T16:18:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.489+0000 7fa463200640 1 -- 172.21.15.45:0/2084874979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa45c15dc60 con 0x7fa45c15d490 2024-04-16T16:18:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.489+0000 7fa452000640 1 -- 172.21.15.45:0/2084874979 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458036040 con 0x7fa45c15d490 2024-04-16T16:18:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.489+0000 7fa463200640 1 -- 172.21.15.45:0/2084874979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa45c15e220 con 0x7fa45c15d490 2024-04-16T16:18:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.490+0000 7fa463200640 1 -- 172.21.15.45:0/2084874979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa45c075780 con 0x7fa45c15d490 2024-04-16T16:18:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.491+0000 7fa452000640 1 -- 172.21.15.45:0/2084874979 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7fa458039020 con 0x7fa45c15d490 2024-04-16T16:18:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.493+0000 7fa452000640 1 --2- 172.21.15.45:0/2084874979 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fa44006ca40 0x7fa44006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.493+0000 7fa452000640 1 -- 172.21.15.45:0/2084874979 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4857+0+0 (secure 0 0 0) 0x7fa4580ad920 con 0x7fa45c15d490 2024-04-16T16:18:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.493+0000 7fa461e00640 1 --2- 172.21.15.45:0/2084874979 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fa44006ca40 0x7fa44006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.494+0000 7fa461e00640 1 --2- 172.21.15.45:0/2084874979 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fa44006ca40 0x7fa44006ef00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa45c0b08f0 tx=0x7fa4540012d0 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.498+0000 7fa452000640 1 -- 172.21.15.45:0/2084874979 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa45807be60 con 0x7fa45c15d490 2024-04-16T16:18:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.694+0000 7fa463200640 1 -- 172.21.15.45:0/2084874979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa45c06f690 con 0x7fa45c15d490 2024-04-16T16:18:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.695+0000 7fa452000640 1 -- 172.21.15.45:0/2084874979 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fa45807b760 con 0x7fa45c15d490 2024-04-16T16:18:12.695 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:18:12.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.698+0000 7fa463200640 1 -- 172.21.15.45:0/2084874979 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fa44006ca40 msgr2=0x7fa44006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.698+0000 7fa463200640 1 --2- 172.21.15.45:0/2084874979 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fa44006ca40 0x7fa44006ef00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa45c0b08f0 tx=0x7fa4540012d0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.698+0000 7fa463200640 1 -- 172.21.15.45:0/2084874979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa45c15d490 msgr2=0x7fa45c13e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.698+0000 7fa463200640 1 --2- 172.21.15.45:0/2084874979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa45c15d490 0x7fa45c13e590 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fa45800bb50 tx=0x7fa458004730 comp rx=0 tx=0).stop 2024-04-16T16:18:12.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.698+0000 7fa463200640 1 -- 172.21.15.45:0/2084874979 shutdown_connections 2024-04-16T16:18:12.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.698+0000 7fa463200640 1 --2- 172.21.15.45:0/2084874979 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fa44006ca40 0x7fa44006ef00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.698+0000 7fa463200640 1 --2- 172.21.15.45:0/2084874979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa45c15d490 0x7fa45c13e590 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.698+0000 7fa463200640 1 --2- 172.21.15.45:0/2084874979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa45c1f49f0 0x7fa45c15cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.698+0000 7fa463200640 1 --2- 172.21.15.45:0/2084874979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448004d40 0x7fa45c138e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.698+0000 7fa463200640 1 -- 172.21.15.45:0/2084874979 >> 172.21.15.45:0/2084874979 conn(0x7fa45c06f0e0 msgr2=0x7fa45c0b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:12.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.698+0000 7fa463200640 1 -- 172.21.15.45:0/2084874979 shutdown_connections 2024-04-16T16:18:12.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.698+0000 7fa463200640 1 -- 172.21.15.45:0/2084874979 wait complete. 2024-04-16T16:18:12.716 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-16T16:18:12.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.926+0000 7f7fa8600640 1 Processor -- start 2024-04-16T16:18:12.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.926+0000 7f7fa8600640 1 -- start start 2024-04-16T16:18:12.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.926+0000 7f7fa8600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa00d0230 0x7f7fa01666f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.926+0000 7f7fa8600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa00ce310 con 0x7f7fa00cdeb0 2024-04-16T16:18:12.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.926+0000 7f7fa8600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa00ce5b0 con 0x7f7fa015b730 2024-04-16T16:18:12.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.926+0000 7f7fa8600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fa00b5410 con 0x7f7fa00d0230 2024-04-16T16:18:12.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.927+0000 7f7fa5200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7fa00cdeb0 0x7f7fa00cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46846/0 (socket says 172.21.15.45:46846) 2024-04-16T16:18:12.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.927+0000 7f7fa5200640 1 -- 172.21.15.45:0/4010019161 learned_addr learned my addr 172.21.15.45:0/4010019161 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:12.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.927+0000 7f7fa6600640 1 --2- 172.21.15.45:0/4010019161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa00d0230 0x7f7fa01666f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.927+0000 7f7f9fe00640 1 -- 172.21.15.45:0/4010019161 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2427932082 0 0) 0x7f7fa00ce310 con 0x7f7fa00cdeb0 2024-04-16T16:18:12.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.927+0000 7f7f9fe00640 1 -- 172.21.15.45:0/4010019161 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f88005730 con 0x7f7fa00cdeb0 2024-04-16T16:18:12.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.927+0000 7f7f9fe00640 1 -- 172.21.15.45:0/4010019161 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1171285493 0 0) 0x7f7f88005730 con 0x7f7fa00cdeb0 2024-04-16T16:18:12.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7f9fe00640 1 -- 172.21.15.45:0/4010019161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa00d0230 msgr2=0x7f7fa01666f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:12.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7f9fe00640 1 --2- 172.21.15.45:0/4010019161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa00d0230 0x7f7fa01666f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7f9fe00640 1 -- 172.21.15.45:0/4010019161 >> v1:172.21.15.103:6789/0 conn(0x7f7fa015b730 legacy=0x7f7fa00b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7f9fe00640 1 -- 172.21.15.45:0/4010019161 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa0072420 con 0x7f7fa00cdeb0 2024-04-16T16:18:12.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7f9fe00640 1 -- 172.21.15.45:0/4010019161 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f7f8c001670 con 0x7f7fa00cdeb0 2024-04-16T16:18:12.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7f9fe00640 1 -- 172.21.15.45:0/4010019161 >> v1:172.21.15.45:6789/0 conn(0x7f7fa00cdeb0 legacy=0x7f7fa00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7f9fe00640 1 --2- 172.21.15.45:0/4010019161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f88005bf0 0x7f7f88006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7f9fe00640 1 --2- 172.21.15.45:0/4010019161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f88006a10 0x7f7f88006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7f9fe00640 1 --2- 172.21.15.45:0/4010019161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa00d0230 0x7f7f8800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7f9fe00640 1 -- 172.21.15.45:0/4010019161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa00b5410 con 0x7f7f88005bf0 2024-04-16T16:18:12.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7f9fe00640 1 -- 172.21.15.45:0/4010019161 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f88007730 con 0x7f7f88006a10 2024-04-16T16:18:12.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7f9fe00640 1 -- 172.21.15.45:0/4010019161 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7f8800e110 con 0x7f7fa00d0230 2024-04-16T16:18:12.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7fa5c00640 1 --2- 172.21.15.45:0/4010019161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f88005bf0 0x7f7f88006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7fa5c00640 1 -- 172.21.15.45:0/4010019161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa00d0230 msgr2=0x7f7f8800dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:12.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7fa5c00640 1 --2- 172.21.15.45:0/4010019161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa00d0230 0x7f7f8800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7fa5200640 1 --2- 172.21.15.45:0/4010019161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f88006a10 0x7f7f88006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7fa5c00640 1 -- 172.21.15.45:0/4010019161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f88006a10 msgr2=0x7f7f88006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:12.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7fa5c00640 1 --2- 172.21.15.45:0/4010019161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f88006a10 0x7f7f88006e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.928+0000 7f7fa5c00640 1 -- 172.21.15.45:0/4010019161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f88005730 con 0x7f7f88005bf0 2024-04-16T16:18:12.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.930+0000 7f7fa5c00640 1 --2- 172.21.15.45:0/4010019161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f88005bf0 0x7f7f88006030 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f7f94001750 tx=0x7f7f88007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d790d9b7ef4b84cb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:12.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.930+0000 7f7f9fe00640 1 -- 172.21.15.45:0/4010019161 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94001370 con 0x7f7f88005bf0 2024-04-16T16:18:12.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.930+0000 7f7f9fe00640 1 -- 172.21.15.45:0/4010019161 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f940052d0 con 0x7f7f88005bf0 2024-04-16T16:18:12.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.930+0000 7f7f9fe00640 1 -- 172.21.15.45:0/4010019161 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94003830 con 0x7f7f88005bf0 2024-04-16T16:18:12.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.930+0000 7f7fa8600640 1 -- 172.21.15.45:0/4010019161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f88005bf0 msgr2=0x7f7f88006030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.930+0000 7f7fa8600640 1 --2- 172.21.15.45:0/4010019161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f88005bf0 0x7f7f88006030 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f7f94001750 tx=0x7f7f88007a40 comp rx=0 tx=0).stop 2024-04-16T16:18:12.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.930+0000 7f7fa6600640 1 -- 172.21.15.45:0/4010019161 reap_dead start 2024-04-16T16:18:12.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.930+0000 7f7fa8600640 1 -- 172.21.15.45:0/4010019161 shutdown_connections 2024-04-16T16:18:12.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.930+0000 7f7fa8600640 1 -- 172.21.15.45:0/4010019161 >> 172.21.15.45:0/4010019161 conn(0x7f7fa00b2750 msgr2=0x7f7fa00b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:12.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.930+0000 7f7fa8600640 1 -- 172.21.15.45:0/4010019161 shutdown_connections 2024-04-16T16:18:12.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.931+0000 7f7fa8600640 1 -- 172.21.15.45:0/4010019161 wait complete. 2024-04-16T16:18:12.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.932+0000 7f7fa8600640 1 Processor -- start 2024-04-16T16:18:12.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.932+0000 7f7fa8600640 1 -- start start 2024-04-16T16:18:12.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa8600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f88005bf0 0x7f7fa0149720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa8600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fa014bdd0 0x7f7fa014f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa8600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa014c250 0x7f7fa014fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa8600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa0072cc0 con 0x7f7f88005bf0 2024-04-16T16:18:12.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa8600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa00729e0 con 0x7f7fa014bdd0 2024-04-16T16:18:12.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa8600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fa015bb10 con 0x7f7fa014c250 2024-04-16T16:18:12.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa5c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f88005bf0 0x7f7fa0149720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa5c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f88005bf0 0x7f7fa0149720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39506/0 (socket says 172.21.15.45:39506) 2024-04-16T16:18:12.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa5c00640 1 -- 172.21.15.45:0/3386983970 learned_addr learned my addr 172.21.15.45:0/3386983970 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:12.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa5200640 1 --2- 172.21.15.45:0/3386983970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fa014bdd0 0x7f7fa014f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa5c00640 1 -- 172.21.15.45:0/3386983970 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa014c250 msgr2=0x7f7fa014fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa5c00640 1 --2- 172.21.15.45:0/3386983970 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa014c250 0x7f7fa014fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa5c00640 1 -- 172.21.15.45:0/3386983970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fa014bdd0 msgr2=0x7f7fa014f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:12.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa5c00640 1 --2- 172.21.15.45:0/3386983970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fa014bdd0 0x7f7fa014f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:12.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa5c00640 1 -- 172.21.15.45:0/3386983970 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa014c7b0 con 0x7f7f88005bf0 2024-04-16T16:18:12.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7fa5c00640 1 --2- 172.21.15.45:0/3386983970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f88005bf0 0x7f7fa0149720 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f7f94005040 tx=0x7f7f94001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:12.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.934+0000 7f7f9d600640 1 -- 172.21.15.45:0/3386983970 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94041070 con 0x7f7f88005bf0 2024-04-16T16:18:12.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.936+0000 7f7f9d600640 1 -- 172.21.15.45:0/3386983970 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f94005480 con 0x7f7f88005bf0 2024-04-16T16:18:12.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.936+0000 7f7fa8600640 1 -- 172.21.15.45:0/3386983970 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fa014ca90 con 0x7f7f88005bf0 2024-04-16T16:18:12.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.936+0000 7f7f9d600640 1 -- 172.21.15.45:0/3386983970 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94007a30 con 0x7f7f88005bf0 2024-04-16T16:18:12.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.936+0000 7f7fa8600640 1 -- 172.21.15.45:0/3386983970 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fa0157d20 con 0x7f7f88005bf0 2024-04-16T16:18:12.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.936+0000 7f7fa8600640 1 -- 172.21.15.45:0/3386983970 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fa014cfb0 con 0x7f7f88005bf0 2024-04-16T16:18:12.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.937+0000 7f7f9d600640 1 -- 172.21.15.45:0/3386983970 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f7f94044070 con 0x7f7f88005bf0 2024-04-16T16:18:12.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.937+0000 7f7f9d600640 1 --2- 172.21.15.45:0/3386983970 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7f8006ca10 0x7f7f8006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:12.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.939+0000 7f7f9d600640 1 -- 172.21.15.45:0/3386983970 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4857+0+0 (secure 0 0 0) 0x7f7f940ad5f0 con 0x7f7f88005bf0 2024-04-16T16:18:12.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.939+0000 7f7fa5200640 1 --2- 172.21.15.45:0/3386983970 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7f8006ca10 0x7f7f8006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:12.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.940+0000 7f7fa5200640 1 --2- 172.21.15.45:0/3386983970 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7f8006ca10 0x7f7f8006eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c001200 tx=0x7f7f8c009040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:12.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:12.942+0000 7f7f9d600640 1 -- 172.21.15.45:0/3386983970 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7fa014cfb0 con 0x7f7f88005bf0 2024-04-16T16:18:13.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.148+0000 7f7fa8600640 1 -- 172.21.15.45:0/3386983970 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f7fa0075780 con 0x7f7f88005bf0 2024-04-16T16:18:13.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.149+0000 7f7f9d600640 1 -- 172.21.15.45:0/3386983970 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f7f9407bbd0 con 0x7f7f88005bf0 2024-04-16T16:18:13.148 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:13.148 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2024-04-16T16:18:13.148 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:18:13.148 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2024-04-16T16:18:13.148 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2024-04-16T16:18:13.148 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:18:13.148 INFO:teuthology.orchestra.run.smithi045.stdout:] 2024-04-16T16:18:13.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.151+0000 7f7fa8600640 1 -- 172.21.15.45:0/3386983970 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7f8006ca10 msgr2=0x7f7f8006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:13.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.151+0000 7f7fa8600640 1 --2- 172.21.15.45:0/3386983970 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7f8006ca10 0x7f7f8006eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c001200 tx=0x7f7f8c009040 comp rx=0 tx=0).stop 2024-04-16T16:18:13.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.151+0000 7f7fa8600640 1 -- 172.21.15.45:0/3386983970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f88005bf0 msgr2=0x7f7fa0149720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:13.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.151+0000 7f7fa8600640 1 --2- 172.21.15.45:0/3386983970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f88005bf0 0x7f7fa0149720 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f7f94005040 tx=0x7f7f94001120 comp rx=0 tx=0).stop 2024-04-16T16:18:13.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.151+0000 7f7fa8600640 1 -- 172.21.15.45:0/3386983970 shutdown_connections 2024-04-16T16:18:13.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.151+0000 7f7fa8600640 1 --2- 172.21.15.45:0/3386983970 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7f8006ca10 0x7f7f8006eed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:13.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.151+0000 7f7fa8600640 1 --2- 172.21.15.45:0/3386983970 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa014c250 0x7f7fa014fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:13.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.151+0000 7f7fa8600640 1 --2- 172.21.15.45:0/3386983970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fa014bdd0 0x7f7fa014f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:13.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.151+0000 7f7fa8600640 1 --2- 172.21.15.45:0/3386983970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f88005bf0 0x7f7fa0149720 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:13.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.151+0000 7f7fa8600640 1 -- 172.21.15.45:0/3386983970 >> 172.21.15.45:0/3386983970 conn(0x7f7fa00b2750 msgr2=0x7f7fa006b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:13.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.152+0000 7f7fa8600640 1 -- 172.21.15.45:0/3386983970 shutdown_connections 2024-04-16T16:18:13.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.152+0000 7f7fa8600640 1 -- 172.21.15.45:0/3386983970 wait complete. 2024-04-16T16:18:13.168 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-16T16:18:13.168 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-16T16:18:13.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.394+0000 7ff6bbe00640 1 Processor -- start 2024-04-16T16:18:13.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.394+0000 7ff6bbe00640 1 -- start start 2024-04-16T16:18:13.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.396+0000 7ff6bbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6bc0b1500 0x7ff6bc06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:13.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.396+0000 7ff6bbe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6bc0d0430 con 0x7ff6bc0d0050 2024-04-16T16:18:13.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.396+0000 7ff6bbe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6bc15bbd0 con 0x7ff6bc06e920 2024-04-16T16:18:13.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.396+0000 7ff6bbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff6bc0b0fd0 con 0x7ff6bc0b1500 2024-04-16T16:18:13.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.396+0000 7ff6baa00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff6bc0d0050 0x7ff6bc0cd8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59304/0 (socket says 172.21.15.45:59304) 2024-04-16T16:18:13.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.396+0000 7ff6baa00640 1 -- 172.21.15.45:0/1791852161 learned_addr learned my addr 172.21.15.45:0/1791852161 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:13.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.396+0000 7ff6b3e00640 1 --2- 172.21.15.45:0/1791852161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6bc0b1500 0x7ff6bc06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:13.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.397+0000 7ff6ba000640 1 -- 172.21.15.45:0/1791852161 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2607713389 0 0) 0x7ff6bc0d0430 con 0x7ff6bc0d0050 2024-04-16T16:18:13.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.397+0000 7ff6ba000640 1 -- 172.21.15.45:0/1791852161 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6a0005730 con 0x7ff6bc0d0050 2024-04-16T16:18:13.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.397+0000 7ff6ba000640 1 -- 172.21.15.45:0/1791852161 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1589057000 0 0) 0x7ff6a0005730 con 0x7ff6bc0d0050 2024-04-16T16:18:13.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.397+0000 7ff6ba000640 1 -- 172.21.15.45:0/1791852161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6bc0b1500 msgr2=0x7ff6bc06e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:13.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.397+0000 7ff6ba000640 1 --2- 172.21.15.45:0/1791852161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6bc0b1500 0x7ff6bc06e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:13.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.397+0000 7ff6ba000640 1 -- 172.21.15.45:0/1791852161 >> v1:172.21.15.103:6789/0 conn(0x7ff6bc06e920 legacy=0x7ff6bc15f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:13.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.397+0000 7ff6ba000640 1 -- 172.21.15.45:0/1791852161 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6bc15be90 con 0x7ff6bc0d0050 2024-04-16T16:18:13.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.397+0000 7ff6ba000640 1 -- 172.21.15.45:0/1791852161 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7ff6ac001670 con 0x7ff6bc0d0050 2024-04-16T16:18:13.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.397+0000 7ff6ba000640 1 -- 172.21.15.45:0/1791852161 >> v1:172.21.15.45:6789/0 conn(0x7ff6bc0d0050 legacy=0x7ff6bc0cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:13.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.397+0000 7ff6ba000640 1 --2- 172.21.15.45:0/1791852161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a0005bf0 0x7ff6a0006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:13.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.397+0000 7ff6ba000640 1 --2- 172.21.15.45:0/1791852161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6bc0b1500 0x7ff6a0006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:13.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.397+0000 7ff6ba000640 1 --2- 172.21.15.45:0/1791852161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6a0006f50 0x7ff6a00073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:13.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.397+0000 7ff6ba000640 1 -- 172.21.15.45:0/1791852161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc0b0fd0 con 0x7ff6a0005bf0 2024-04-16T16:18:13.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.397+0000 7ff6ba000640 1 -- 172.21.15.45:0/1791852161 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a000ffd0 con 0x7ff6a0006f50 2024-04-16T16:18:13.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.397+0000 7ff6ba000640 1 -- 172.21.15.45:0/1791852161 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a0010150 con 0x7ff6bc0b1500 2024-04-16T16:18:13.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.398+0000 7ff6baa00640 1 --2- 172.21.15.45:0/1791852161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a0005bf0 0x7ff6a0006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:13.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.398+0000 7ff6baa00640 1 -- 172.21.15.45:0/1791852161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6bc0b1500 msgr2=0x7ff6a0006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:13.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.398+0000 7ff6b3e00640 1 --2- 172.21.15.45:0/1791852161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6bc0b1500 0x7ff6a0006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:13.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.398+0000 7ff6bb400640 1 --2- 172.21.15.45:0/1791852161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6a0006f50 0x7ff6a00073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:13.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.398+0000 7ff6baa00640 1 --2- 172.21.15.45:0/1791852161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6bc0b1500 0x7ff6a0006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:13.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.398+0000 7ff6baa00640 1 -- 172.21.15.45:0/1791852161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6a0006f50 msgr2=0x7ff6a00073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:13.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.398+0000 7ff6baa00640 1 --2- 172.21.15.45:0/1791852161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6a0006f50 0x7ff6a00073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:13.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.398+0000 7ff6baa00640 1 -- 172.21.15.45:0/1791852161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a0005730 con 0x7ff6a0005bf0 2024-04-16T16:18:13.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.398+0000 7ff6b3e00640 1 --2- 172.21.15.45:0/1791852161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6bc0b1500 0x7ff6a0006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:13.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.399+0000 7ff6baa00640 1 --2- 172.21.15.45:0/1791852161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a0005bf0 0x7ff6a0006010 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00c240 tx=0x7ff6a0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385dd049ccb4d082 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:13.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.400+0000 7ff6ba000640 1 -- 172.21.15.45:0/1791852161 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac007cf0 con 0x7ff6a0005bf0 2024-04-16T16:18:13.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.400+0000 7ff6ba000640 1 -- 172.21.15.45:0/1791852161 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6ac030070 con 0x7ff6a0005bf0 2024-04-16T16:18:13.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.400+0000 7ff6ba000640 1 -- 172.21.15.45:0/1791852161 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac005d30 con 0x7ff6a0005bf0 2024-04-16T16:18:13.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.400+0000 7ff6bbe00640 1 -- 172.21.15.45:0/1791852161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a0005bf0 msgr2=0x7ff6a0006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:13.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.400+0000 7ff6bbe00640 1 --2- 172.21.15.45:0/1791852161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a0005bf0 0x7ff6a0006010 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00c240 tx=0x7ff6a0007a40 comp rx=0 tx=0).stop 2024-04-16T16:18:13.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.400+0000 7ff6bb400640 1 -- 172.21.15.45:0/1791852161 reap_dead start 2024-04-16T16:18:13.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.400+0000 7ff6bbe00640 1 -- 172.21.15.45:0/1791852161 shutdown_connections 2024-04-16T16:18:13.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.400+0000 7ff6bbe00640 1 -- 172.21.15.45:0/1791852161 >> 172.21.15.45:0/1791852161 conn(0x7ff6bc0b3bb0 msgr2=0x7ff6bc0b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:13.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.400+0000 7ff6bbe00640 1 -- 172.21.15.45:0/1791852161 shutdown_connections 2024-04-16T16:18:13.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.400+0000 7ff6bbe00640 1 -- 172.21.15.45:0/1791852161 wait complete. 2024-04-16T16:18:13.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.402+0000 7ff6bbe00640 1 Processor -- start 2024-04-16T16:18:13.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.402+0000 7ff6bbe00640 1 -- start start 2024-04-16T16:18:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.403+0000 7ff6bbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6a0005bf0 0x7ff6bc146ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.403+0000 7ff6bbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6bc153340 0x7ff6bc151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.403+0000 7ff6bbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6bc153780 0x7ff6bc0b1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.403+0000 7ff6bbe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc15c730 con 0x7ff6bc153780 2024-04-16T16:18:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.403+0000 7ff6bbe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc15c450 con 0x7ff6bc153340 2024-04-16T16:18:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.403+0000 7ff6bbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff6bc0cdfb0 con 0x7ff6a0005bf0 2024-04-16T16:18:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.404+0000 7ff6bb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6bc153780 0x7ff6bc0b1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.404+0000 7ff6bb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6bc153780 0x7ff6bc0b1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57162/0 (socket says 172.21.15.45:57162) 2024-04-16T16:18:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.404+0000 7ff6bb400640 1 -- 172.21.15.45:0/368198079 learned_addr learned my addr 172.21.15.45:0/368198079 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.404+0000 7ff6b3e00640 1 --2- 172.21.15.45:0/368198079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6bc153340 0x7ff6bc151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.404+0000 7ff6baa00640 1 --2- 172.21.15.45:0/368198079 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6a0005bf0 0x7ff6bc146ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:13.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.404+0000 7ff6bb400640 1 -- 172.21.15.45:0/368198079 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6a0005bf0 msgr2=0x7ff6bc146ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:13.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.404+0000 7ff6bb400640 1 --2- 172.21.15.45:0/368198079 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6a0005bf0 0x7ff6bc146ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:13.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.404+0000 7ff6bb400640 1 -- 172.21.15.45:0/368198079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6bc153340 msgr2=0x7ff6bc151850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:13.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.404+0000 7ff6bb400640 1 --2- 172.21.15.45:0/368198079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6bc153340 0x7ff6bc151850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:13.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.404+0000 7ff6bb400640 1 -- 172.21.15.45:0/368198079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6bc1522a0 con 0x7ff6bc153780 2024-04-16T16:18:13.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.404+0000 7ff6bb400640 1 --2- 172.21.15.45:0/368198079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6bc153780 0x7ff6bc0b1980 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc06efa0 tx=0x7ff6b400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:13.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.405+0000 7ff6b2a00640 1 -- 172.21.15.45:0/368198079 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4018070 con 0x7ff6bc153780 2024-04-16T16:18:13.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.405+0000 7ff6b2a00640 1 -- 172.21.15.45:0/368198079 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6b4003b70 con 0x7ff6bc153780 2024-04-16T16:18:13.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.405+0000 7ff6b2a00640 1 -- 172.21.15.45:0/368198079 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4005070 con 0x7ff6bc153780 2024-04-16T16:18:13.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.405+0000 7ff6bbe00640 1 -- 172.21.15.45:0/368198079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6bc152580 con 0x7ff6bc153780 2024-04-16T16:18:13.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.405+0000 7ff6bbe00640 1 -- 172.21.15.45:0/368198079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6bc152b40 con 0x7ff6bc153780 2024-04-16T16:18:13.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.406+0000 7ff6bbe00640 1 -- 172.21.15.45:0/368198079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6bc075780 con 0x7ff6bc153780 2024-04-16T16:18:13.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.407+0000 7ff6b2a00640 1 -- 172.21.15.45:0/368198079 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7ff6b401b070 con 0x7ff6bc153780 2024-04-16T16:18:13.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.408+0000 7ff6b2a00640 1 --2- 172.21.15.45:0/368198079 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff69806ca10 0x7ff69806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:13.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.408+0000 7ff6b2a00640 1 -- 172.21.15.45:0/368198079 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4857+0+0 (secure 0 0 0) 0x7ff6b408b250 con 0x7ff6bc153780 2024-04-16T16:18:13.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.409+0000 7ff6baa00640 1 --2- 172.21.15.45:0/368198079 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff69806ca10 0x7ff69806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:13.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.410+0000 7ff6baa00640 1 --2- 172.21.15.45:0/368198079 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff69806ca10 0x7ff69806eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac034000 tx=0x7ff6ac007000 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:13.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.413+0000 7ff6b2a00640 1 -- 172.21.15.45:0/368198079 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff6b4059710 con 0x7ff6bc153780 2024-04-16T16:18:13.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:13.618+0000 7ff6bbe00640 1 -- 172.21.15.45:0/368198079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ff6bc0b2d90 con 0x7ff6bc153780 2024-04-16T16:18:14.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.254+0000 7ff6b2a00640 1 -- 172.21.15.45:0/368198079 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7ff6b4059010 con 0x7ff6bc153780 2024-04-16T16:18:14.254 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2024-04-16T16:18:14.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.258+0000 7ff6bbe00640 1 -- 172.21.15.45:0/368198079 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff69806ca10 msgr2=0x7ff69806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:14.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.258+0000 7ff6bbe00640 1 --2- 172.21.15.45:0/368198079 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff69806ca10 0x7ff69806eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac034000 tx=0x7ff6ac007000 comp rx=0 tx=0).stop 2024-04-16T16:18:14.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.258+0000 7ff6bbe00640 1 -- 172.21.15.45:0/368198079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6bc153780 msgr2=0x7ff6bc0b1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:14.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.258+0000 7ff6bbe00640 1 --2- 172.21.15.45:0/368198079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6bc153780 0x7ff6bc0b1980 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc06efa0 tx=0x7ff6b400dc00 comp rx=0 tx=0).stop 2024-04-16T16:18:14.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.259+0000 7ff6bbe00640 1 -- 172.21.15.45:0/368198079 shutdown_connections 2024-04-16T16:18:14.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.259+0000 7ff6bbe00640 1 --2- 172.21.15.45:0/368198079 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff69806ca10 0x7ff69806eed0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:14.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.259+0000 7ff6bbe00640 1 --2- 172.21.15.45:0/368198079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6bc153780 0x7ff6bc0b1980 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:14.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.259+0000 7ff6bbe00640 1 --2- 172.21.15.45:0/368198079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6bc153340 0x7ff6bc151850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:14.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.259+0000 7ff6bbe00640 1 --2- 172.21.15.45:0/368198079 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6a0005bf0 0x7ff6bc146ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:14.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.259+0000 7ff6bbe00640 1 -- 172.21.15.45:0/368198079 >> 172.21.15.45:0/368198079 conn(0x7ff6bc0b3bb0 msgr2=0x7ff6bc072a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:14.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.259+0000 7ff6bbe00640 1 -- 172.21.15.45:0/368198079 shutdown_connections 2024-04-16T16:18:14.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.259+0000 7ff6bbe00640 1 -- 172.21.15.45:0/368198079 wait complete. 2024-04-16T16:18:14.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-16T16:18:14.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.515+0000 7f728f000640 1 Processor -- start 2024-04-16T16:18:14.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.515+0000 7f728f000640 1 -- start start 2024-04-16T16:18:14.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.517+0000 7f728f000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f72881452e0 0x7f728806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:14.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.517+0000 7f728f000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72880cecc0 0x7f728815cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:14.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.517+0000 7f728f000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7288071e60 con 0x7f728806e920 2024-04-16T16:18:14.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.517+0000 7f728f000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f7288072f00 con 0x7f72881452e0 2024-04-16T16:18:14.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.517+0000 7f728f000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f728806ed90 con 0x7f72880cecc0 2024-04-16T16:18:14.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.517+0000 7f728dc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f72881452e0 0x7f728806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:14.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.518+0000 7f728dc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f72881452e0 0x7f728806e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57174/0 (socket says 172.21.15.45:57174) 2024-04-16T16:18:14.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.518+0000 7f728dc00640 1 -- 172.21.15.45:0/2553514722 learned_addr learned my addr 172.21.15.45:0/2553514722 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:14.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.518+0000 7f728e600640 1 --2- 172.21.15.45:0/2553514722 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72880cecc0 0x7f728815cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:14.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.518+0000 7f728dc00640 1 -- 172.21.15.45:0/2553514722 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72880cecc0 msgr2=0x7f728815cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:14.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.518+0000 7f728dc00640 1 --2- 172.21.15.45:0/2553514722 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72880cecc0 0x7f728815cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:14.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.518+0000 7f728dc00640 1 -- 172.21.15.45:0/2553514722 >> v1:172.21.15.45:6789/0 conn(0x7f728806e920 legacy=0x7f72880ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:14.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.518+0000 7f728dc00640 1 -- 172.21.15.45:0/2553514722 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7288072140 con 0x7f72881452e0 2024-04-16T16:18:14.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.519+0000 7f728dc00640 1 --2- 172.21.15.45:0/2553514722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72881452e0 0x7f728806e3e0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f727c0098a0 tx=0x7f727c02ba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d489cf13f4d08248 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:14.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.520+0000 7f7287e00640 1 -- 172.21.15.45:0/2553514722 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c02f820 con 0x7f72881452e0 2024-04-16T16:18:14.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.520+0000 7f7287e00640 1 -- 172.21.15.45:0/2553514722 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f727c02f9c0 con 0x7f72881452e0 2024-04-16T16:18:14.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.520+0000 7f7287e00640 1 -- 172.21.15.45:0/2553514722 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c02fd90 con 0x7f72881452e0 2024-04-16T16:18:14.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.520+0000 7f728f000640 1 -- 172.21.15.45:0/2553514722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72881452e0 msgr2=0x7f728806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:14.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.520+0000 7f728f000640 1 --2- 172.21.15.45:0/2553514722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72881452e0 0x7f728806e3e0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f727c0098a0 tx=0x7f727c02ba00 comp rx=0 tx=0).stop 2024-04-16T16:18:14.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.520+0000 7f728f000640 1 -- 172.21.15.45:0/2553514722 shutdown_connections 2024-04-16T16:18:14.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.520+0000 7f728f000640 1 --2- 172.21.15.45:0/2553514722 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72880cecc0 0x7f728815cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:14.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.520+0000 7f728f000640 1 --2- 172.21.15.45:0/2553514722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72881452e0 0x7f728806e3e0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:14.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.520+0000 7f728f000640 1 -- 172.21.15.45:0/2553514722 >> 172.21.15.45:0/2553514722 conn(0x7f72880b3bb0 msgr2=0x7f72880b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:14.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.521+0000 7f728f000640 1 -- 172.21.15.45:0/2553514722 shutdown_connections 2024-04-16T16:18:14.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.521+0000 7f728f000640 1 -- 172.21.15.45:0/2553514722 wait complete. 2024-04-16T16:18:14.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.522+0000 7f728f000640 1 Processor -- start 2024-04-16T16:18:14.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.522+0000 7f728f000640 1 -- start start 2024-04-16T16:18:14.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.523+0000 7f728f000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f728806e920 0x7f7288136770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:14.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.523+0000 7f728f000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72880cecc0 0x7f72880ac1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:14.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.523+0000 7f728f000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72881452e0 0x7f72880b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:14.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.523+0000 7f728f000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f72880729e0 con 0x7f72881452e0 2024-04-16T16:18:14.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.523+0000 7f728f000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7288072700 con 0x7f72880cecc0 2024-04-16T16:18:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.523+0000 7f728f000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f72881456c0 con 0x7f728806e920 2024-04-16T16:18:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.524+0000 7f728e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72881452e0 0x7f72880b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.524+0000 7f728e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72881452e0 0x7f72880b13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57178/0 (socket says 172.21.15.45:57178) 2024-04-16T16:18:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.524+0000 7f728e600640 1 -- 172.21.15.45:0/4203341573 learned_addr learned my addr 172.21.15.45:0/4203341573 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.524+0000 7f728dc00640 1 --2- 172.21.15.45:0/4203341573 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f728806e920 0x7f7288136770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.524+0000 7f728d200640 1 --2- 172.21.15.45:0/4203341573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72880cecc0 0x7f72880ac1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.524+0000 7f728e600640 1 -- 172.21.15.45:0/4203341573 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f728806e920 msgr2=0x7f7288136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.524+0000 7f728e600640 1 --2- 172.21.15.45:0/4203341573 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f728806e920 0x7f7288136770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.524+0000 7f728e600640 1 -- 172.21.15.45:0/4203341573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72880cecc0 msgr2=0x7f72880ac1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.524+0000 7f728e600640 1 --2- 172.21.15.45:0/4203341573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72880cecc0 0x7f72880ac1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.524+0000 7f728e600640 1 -- 172.21.15.45:0/4203341573 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72880acc40 con 0x7f72881452e0 2024-04-16T16:18:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.524+0000 7f728dc00640 1 --2- 172.21.15.45:0/4203341573 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f728806e920 0x7f7288136770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.525+0000 7f728e600640 1 --2- 172.21.15.45:0/4203341573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72881452e0 0x7f72880b13f0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f7280007e40 tx=0x7f728000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:14.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.525+0000 7f7285600640 1 -- 172.21.15.45:0/4203341573 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7280014620 con 0x7f72881452e0 2024-04-16T16:18:14.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.525+0000 7f7285600640 1 -- 172.21.15.45:0/4203341573 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7280019070 con 0x7f72881452e0 2024-04-16T16:18:14.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.525+0000 7f7285600640 1 -- 172.21.15.45:0/4203341573 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7280014a20 con 0x7f72881452e0 2024-04-16T16:18:14.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.525+0000 7f728f000640 1 -- 172.21.15.45:0/4203341573 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72880acf20 con 0x7f72881452e0 2024-04-16T16:18:14.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.525+0000 7f728f000640 1 -- 172.21.15.45:0/4203341573 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72880ad4e0 con 0x7f72881452e0 2024-04-16T16:18:14.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.526+0000 7f728f000640 1 -- 172.21.15.45:0/4203341573 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7288075780 con 0x7f72881452e0 2024-04-16T16:18:14.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.528+0000 7f7285600640 1 -- 172.21.15.45:0/4203341573 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f7280014bc0 con 0x7f72881452e0 2024-04-16T16:18:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.529+0000 7f7285600640 1 --2- 172.21.15.45:0/4203341573 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f726806ca10 0x7f726806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.529+0000 7f7285600640 1 -- 172.21.15.45:0/4203341573 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5227+0+0 (secure 0 0 0) 0x7f728008d020 con 0x7f72881452e0 2024-04-16T16:18:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.530+0000 7f728dc00640 1 --2- 172.21.15.45:0/4203341573 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f726806ca10 0x7f726806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:14.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.531+0000 7f728dc00640 1 --2- 172.21.15.45:0/4203341573 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f726806ca10 0x7f726806eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f727c036000 tx=0x7f727c039040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:14.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.533+0000 7f7285600640 1 -- 172.21.15.45:0/4203341573 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f728005b500 con 0x7f72881452e0 2024-04-16T16:18:14.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:14.741+0000 7f728f000640 1 -- 172.21.15.45:0/4203341573 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f72880b2370 con 0x7f72881452e0 2024-04-16T16:18:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.271+0000 7f7285600640 1 -- 172.21.15.45:0/4203341573 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) ==== 170+0+0 (secure 0 0 0) 0x7f728005ae00 con 0x7f72881452e0 2024-04-16T16:18:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2024-04-16T16:18:15.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.275+0000 7f728f000640 1 -- 172.21.15.45:0/4203341573 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f726806ca10 msgr2=0x7f726806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:15.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.275+0000 7f728f000640 1 --2- 172.21.15.45:0/4203341573 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f726806ca10 0x7f726806eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f727c036000 tx=0x7f727c039040 comp rx=0 tx=0).stop 2024-04-16T16:18:15.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.275+0000 7f728f000640 1 -- 172.21.15.45:0/4203341573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72881452e0 msgr2=0x7f72880b13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:15.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.275+0000 7f728f000640 1 --2- 172.21.15.45:0/4203341573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72881452e0 0x7f72880b13f0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f7280007e40 tx=0x7f728000e4b0 comp rx=0 tx=0).stop 2024-04-16T16:18:15.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.276+0000 7f728f000640 1 -- 172.21.15.45:0/4203341573 shutdown_connections 2024-04-16T16:18:15.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.276+0000 7f728f000640 1 --2- 172.21.15.45:0/4203341573 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f726806ca10 0x7f726806eed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:15.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.276+0000 7f728f000640 1 --2- 172.21.15.45:0/4203341573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72881452e0 0x7f72880b13f0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:15.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.276+0000 7f728f000640 1 --2- 172.21.15.45:0/4203341573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72880cecc0 0x7f72880ac1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:15.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.276+0000 7f728f000640 1 --2- 172.21.15.45:0/4203341573 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f728806e920 0x7f7288136770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:15.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.276+0000 7f728f000640 1 -- 172.21.15.45:0/4203341573 >> 172.21.15.45:0/4203341573 conn(0x7f72880b3bb0 msgr2=0x7f72881459f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:15.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.277+0000 7f728f000640 1 -- 172.21.15.45:0/4203341573 shutdown_connections 2024-04-16T16:18:15.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.277+0000 7f728f000640 1 -- 172.21.15.45:0/4203341573 wait complete. 2024-04-16T16:18:15.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-16T16:18:15.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.499+0000 7f46b2600640 1 Processor -- start 2024-04-16T16:18:15.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.499+0000 7f46b2600640 1 -- start start 2024-04-16T16:18:15.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.500+0000 7f46b2600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f46ac0b5a30 0x7f46ac1625e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:15.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.501+0000 7f46b2600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46ac0b5e10 con 0x7f46ac06e290 2024-04-16T16:18:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.501+0000 7f46b2600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46ac15ff70 con 0x7f46ac0d1030 2024-04-16T16:18:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.501+0000 7f46b2600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f46ac15f3d0 con 0x7f46ac0b5a30 2024-04-16T16:18:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.501+0000 7f46abe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f46ac06e290 0x7f46ac0b4920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59316/0 (socket says 172.21.15.45:59316) 2024-04-16T16:18:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.501+0000 7f46abe00640 1 -- 172.21.15.45:0/3753209775 learned_addr learned my addr 172.21.15.45:0/3753209775 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.501+0000 7f46ab400640 1 --2- 172.21.15.45:0/3753209775 >> v2:172.21.15.103:3300/0 conn(0x7f46ac0b5a30 0x7f46ac1625e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.501+0000 7f46aaa00640 1 -- 172.21.15.45:0/3753209775 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1777650001 0 0) 0x7f46ac0b5e10 con 0x7f46ac06e290 2024-04-16T16:18:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.502+0000 7f46aaa00640 1 -- 172.21.15.45:0/3753209775 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4690003520 con 0x7f46ac06e290 2024-04-16T16:18:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.502+0000 7f46aaa00640 1 -- 172.21.15.45:0/3753209775 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 658300463 0 0) 0x7f4690003520 con 0x7f46ac06e290 2024-04-16T16:18:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.502+0000 7f46aaa00640 1 -- 172.21.15.45:0/3753209775 >> v2:172.21.15.103:3300/0 conn(0x7f46ac0b5a30 msgr2=0x7f46ac1625e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.502+0000 7f46aaa00640 1 --2- 172.21.15.45:0/3753209775 >> v2:172.21.15.103:3300/0 conn(0x7f46ac0b5a30 0x7f46ac1625e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.502+0000 7f46aaa00640 1 -- 172.21.15.45:0/3753209775 >> v1:172.21.15.103:6789/0 conn(0x7f46ac0d1030 legacy=0x7f46ac15f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.502+0000 7f46aaa00640 1 -- 172.21.15.45:0/3753209775 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46ac162e70 con 0x7f46ac06e290 2024-04-16T16:18:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.502+0000 7f46aaa00640 1 -- 172.21.15.45:0/3753209775 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f46a40019f0 con 0x7f46ac06e290 2024-04-16T16:18:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.502+0000 7f46aaa00640 1 -- 172.21.15.45:0/3753209775 >> v1:172.21.15.45:6789/0 conn(0x7f46ac06e290 legacy=0x7f46ac0b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.502+0000 7f46aaa00640 1 --2- 172.21.15.45:0/3753209775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46900039b0 0x7f4690003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.502+0000 7f46aaa00640 1 --2- 172.21.15.45:0/3753209775 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4690004800 0x7f4690004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.502+0000 7f46aaa00640 1 --2- 172.21.15.45:0/3753209775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46900051c0 0x7f469000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.502+0000 7f46aaa00640 1 -- 172.21.15.45:0/3753209775 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f46ac15f3d0 con 0x7f46900039b0 2024-04-16T16:18:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.502+0000 7f46aaa00640 1 -- 172.21.15.45:0/3753209775 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f469000c060 con 0x7f46900051c0 2024-04-16T16:18:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.502+0000 7f46aaa00640 1 -- 172.21.15.45:0/3753209775 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f469000c260 con 0x7f4690004800 2024-04-16T16:18:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.502+0000 7f46ab400640 1 --2- 172.21.15.45:0/3753209775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46900039b0 0x7f4690003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.503+0000 7f46a3e00640 1 --2- 172.21.15.45:0/3753209775 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4690004800 0x7f4690004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.503+0000 7f46abe00640 1 --2- 172.21.15.45:0/3753209775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46900051c0 0x7f469000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.503+0000 7f46ab400640 1 -- 172.21.15.45:0/3753209775 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4690004800 msgr2=0x7f4690004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.503+0000 7f46ab400640 1 --2- 172.21.15.45:0/3753209775 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4690004800 0x7f4690004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.503+0000 7f46ab400640 1 -- 172.21.15.45:0/3753209775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46900051c0 msgr2=0x7f469000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.503+0000 7f46ab400640 1 --2- 172.21.15.45:0/3753209775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46900051c0 0x7f469000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.503+0000 7f46ab400640 1 -- 172.21.15.45:0/3753209775 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4690003520 con 0x7f46900039b0 2024-04-16T16:18:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.503+0000 7f46abe00640 1 -- 172.21.15.45:0/3753209775 reap_dead start 2024-04-16T16:18:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.504+0000 7f46ab400640 1 --2- 172.21.15.45:0/3753209775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46900039b0 0x7f4690003dd0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f469c009ac0 tx=0x7f4690005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea94be31ec565336 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.504+0000 7f46aaa00640 1 -- 172.21.15.45:0/3753209775 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c039070 con 0x7f46900039b0 2024-04-16T16:18:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.504+0000 7f46aaa00640 1 -- 172.21.15.45:0/3753209775 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f469c02c6b0 con 0x7f46900039b0 2024-04-16T16:18:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.504+0000 7f46aaa00640 1 -- 172.21.15.45:0/3753209775 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c009690 con 0x7f46900039b0 2024-04-16T16:18:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.505+0000 7f46b2600640 1 -- 172.21.15.45:0/3753209775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46900039b0 msgr2=0x7f4690003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.505+0000 7f46b2600640 1 --2- 172.21.15.45:0/3753209775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46900039b0 0x7f4690003dd0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f469c009ac0 tx=0x7f4690005830 comp rx=0 tx=0).stop 2024-04-16T16:18:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.505+0000 7f46b2600640 1 -- 172.21.15.45:0/3753209775 shutdown_connections 2024-04-16T16:18:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.505+0000 7f46b2600640 1 --2- 172.21.15.45:0/3753209775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46900039b0 0x7f4690003dd0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.505+0000 7f46b2600640 1 -- 172.21.15.45:0/3753209775 >> 172.21.15.45:0/3753209775 conn(0x7f46ac0b3970 msgr2=0x7f46ac0b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.505+0000 7f46b2600640 1 -- 172.21.15.45:0/3753209775 shutdown_connections 2024-04-16T16:18:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.505+0000 7f46b2600640 1 -- 172.21.15.45:0/3753209775 wait complete. 2024-04-16T16:18:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.507+0000 7f46b2600640 1 Processor -- start 2024-04-16T16:18:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.507+0000 7f46b2600640 1 -- start start 2024-04-16T16:18:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.508+0000 7f46b2600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46900039b0 0x7f46ac152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.508+0000 7f46b2600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac153270 0x7f46ac1e9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.508+0000 7f46b2600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ac0ceef0 0x7f46ac0aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.508+0000 7f46b2600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f46ac15eff0 con 0x7f46ac0ceef0 2024-04-16T16:18:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.508+0000 7f46b2600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f46ac163430 con 0x7f46900039b0 2024-04-16T16:18:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.508+0000 7f46b2600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f46ac06e6e0 con 0x7f46ac153270 2024-04-16T16:18:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.509+0000 7f46abe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ac0ceef0 0x7f46ac0aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.509+0000 7f46ab400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46900039b0 0x7f46ac152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.509+0000 7f46abe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ac0ceef0 0x7f46ac0aeae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57196/0 (socket says 172.21.15.45:57196) 2024-04-16T16:18:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.509+0000 7f46abe00640 1 -- 172.21.15.45:0/1571235583 learned_addr learned my addr 172.21.15.45:0/1571235583 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.509+0000 7f46a3e00640 1 --2- 172.21.15.45:0/1571235583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac153270 0x7f46ac1e9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.509+0000 7f46abe00640 1 -- 172.21.15.45:0/1571235583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac153270 msgr2=0x7f46ac1e9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.509+0000 7f46abe00640 1 --2- 172.21.15.45:0/1571235583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac153270 0x7f46ac1e9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.509+0000 7f46abe00640 1 -- 172.21.15.45:0/1571235583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46900039b0 msgr2=0x7f46ac152a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.509+0000 7f46abe00640 1 --2- 172.21.15.45:0/1571235583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46900039b0 0x7f46ac152a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.509+0000 7f46abe00640 1 -- 172.21.15.45:0/1571235583 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46ac152f90 con 0x7f46ac0ceef0 2024-04-16T16:18:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.510+0000 7f46abe00640 1 --2- 172.21.15.45:0/1571235583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ac0ceef0 0x7f46ac0aeae0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f46a4000f80 tx=0x7f46a400b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.510+0000 7f46a3400640 1 -- 172.21.15.45:0/1571235583 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a400b870 con 0x7f46ac0ceef0 2024-04-16T16:18:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.510+0000 7f46a3400640 1 -- 172.21.15.45:0/1571235583 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46a400ba50 con 0x7f46ac0ceef0 2024-04-16T16:18:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.510+0000 7f46a3400640 1 -- 172.21.15.45:0/1571235583 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a400be40 con 0x7f46ac0ceef0 2024-04-16T16:18:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.510+0000 7f46b2600640 1 -- 172.21.15.45:0/1571235583 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46ac1e8e30 con 0x7f46ac0ceef0 2024-04-16T16:18:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.510+0000 7f46b2600640 1 -- 172.21.15.45:0/1571235583 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46ac1e93f0 con 0x7f46ac0ceef0 2024-04-16T16:18:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.511+0000 7f46b2600640 1 -- 172.21.15.45:0/1571235583 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46ac075780 con 0x7f46ac0ceef0 2024-04-16T16:18:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.513+0000 7f46a3400640 1 -- 172.21.15.45:0/1571235583 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f46a401b020 con 0x7f46ac0ceef0 2024-04-16T16:18:15.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.514+0000 7f46a3400640 1 --2- 172.21.15.45:0/1571235583 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f468806ca10 0x7f468806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:15.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.514+0000 7f46a3400640 1 -- 172.21.15.45:0/1571235583 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5609+0+0 (secure 0 0 0) 0x7f46a408c5f0 con 0x7f46ac0ceef0 2024-04-16T16:18:15.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.515+0000 7f46ab400640 1 --2- 172.21.15.45:0/1571235583 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f468806ca10 0x7f468806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:15.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.516+0000 7f46ab400640 1 --2- 172.21.15.45:0/1571235583 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f468806ca10 0x7f468806eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f469c036000 tx=0x7f469c030000 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:15.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.518+0000 7f46a3400640 1 -- 172.21.15.45:0/1571235583 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f46a4022070 con 0x7f46ac0ceef0 2024-04-16T16:18:15.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:15.761+0000 7f46b2600640 1 -- 172.21.15.45:0/1571235583 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f46ac159470 con 0x7f46ac0ceef0 2024-04-16T16:18:16.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.275+0000 7f46a3400640 1 -- 172.21.15.45:0/1571235583 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v60) ==== 174+0+0 (secure 0 0 0) 0x7f46a405a7b0 con 0x7f46ac0ceef0 2024-04-16T16:18:16.275 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 12 and data pool 13 2024-04-16T16:18:16.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.280+0000 7f46b2600640 1 -- 172.21.15.45:0/1571235583 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f468806ca10 msgr2=0x7f468806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.280+0000 7f46b2600640 1 --2- 172.21.15.45:0/1571235583 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f468806ca10 0x7f468806eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f469c036000 tx=0x7f469c030000 comp rx=0 tx=0).stop 2024-04-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.280+0000 7f46b2600640 1 -- 172.21.15.45:0/1571235583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ac0ceef0 msgr2=0x7f46ac0aeae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.280+0000 7f46b2600640 1 --2- 172.21.15.45:0/1571235583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ac0ceef0 0x7f46ac0aeae0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f46a4000f80 tx=0x7f46a400b0d0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.280+0000 7f46b2600640 1 -- 172.21.15.45:0/1571235583 shutdown_connections 2024-04-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.280+0000 7f46b2600640 1 --2- 172.21.15.45:0/1571235583 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f468806ca10 0x7f468806eed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.280+0000 7f46b2600640 1 --2- 172.21.15.45:0/1571235583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ac0ceef0 0x7f46ac0aeae0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.280+0000 7f46b2600640 1 --2- 172.21.15.45:0/1571235583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac153270 0x7f46ac1e9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.280+0000 7f46b2600640 1 --2- 172.21.15.45:0/1571235583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46900039b0 0x7f46ac152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.280+0000 7f46b2600640 1 -- 172.21.15.45:0/1571235583 >> 172.21.15.45:0/1571235583 conn(0x7f46ac0b3970 msgr2=0x7f46ac0adac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.281+0000 7f46b2600640 1 -- 172.21.15.45:0/1571235583 shutdown_connections 2024-04-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.281+0000 7f46b2600640 1 -- 172.21.15.45:0/1571235583 wait complete. 2024-04-16T16:18:16.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:18:16.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.505+0000 7f0419400640 1 Processor -- start 2024-04-16T16:18:16.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.505+0000 7f0419400640 1 -- start start 2024-04-16T16:18:16.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.506+0000 7f0419400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0414146220 0x7f041415ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:16.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.506+0000 7f0419400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04141608a0 0x7f0414160d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:16.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.506+0000 7f0419400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f041415d360 con 0x7f041406f7a0 2024-04-16T16:18:16.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.507+0000 7f0419400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0414160410 con 0x7f0414146220 2024-04-16T16:18:16.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.507+0000 7f0419400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f041415d640 con 0x7f04141608a0 2024-04-16T16:18:16.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.507+0000 7f0413e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0414146220 0x7f041415ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:16.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.507+0000 7f0413e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0414146220 0x7f041415ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57204/0 (socket says 172.21.15.45:57204) 2024-04-16T16:18:16.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.507+0000 7f0413e00640 1 -- 172.21.15.45:0/1378602194 learned_addr learned my addr 172.21.15.45:0/1378602194 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:16.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.507+0000 7f0418a00640 1 --2- 172.21.15.45:0/1378602194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04141608a0 0x7f0414160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:16.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.507+0000 7f0413e00640 1 -- 172.21.15.45:0/1378602194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04141608a0 msgr2=0x7f0414160d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:16.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.507+0000 7f0413e00640 1 --2- 172.21.15.45:0/1378602194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04141608a0 0x7f0414160d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.507+0000 7f0413e00640 1 -- 172.21.15.45:0/1378602194 >> v1:172.21.15.103:6789/0 conn(0x7f041406f7a0 legacy=0x7f041406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:16.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.507+0000 7f0413e00640 1 -- 172.21.15.45:0/1378602194 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f041415d920 con 0x7f0414146220 2024-04-16T16:18:16.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.507+0000 7f0418a00640 1 --2- 172.21.15.45:0/1378602194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04141608a0 0x7f0414160d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:16.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.509+0000 7f0413e00640 1 --2- 172.21.15.45:0/1378602194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0414146220 0x7f041415ccb0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f0404009870 tx=0x7f040402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54547d4b1c8af5c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:16.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.509+0000 7f0413400640 1 -- 172.21.15.45:0/1378602194 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0404035020 con 0x7f0414146220 2024-04-16T16:18:16.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.509+0000 7f0413400640 1 -- 172.21.15.45:0/1378602194 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0404031640 con 0x7f0414146220 2024-04-16T16:18:16.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.509+0000 7f0413400640 1 -- 172.21.15.45:0/1378602194 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040402d040 con 0x7f0414146220 2024-04-16T16:18:16.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.510+0000 7f0419400640 1 -- 172.21.15.45:0/1378602194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0414146220 msgr2=0x7f041415ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:16.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.510+0000 7f0419400640 1 --2- 172.21.15.45:0/1378602194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0414146220 0x7f041415ccb0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f0404009870 tx=0x7f040402b8f0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.510+0000 7f0419400640 1 -- 172.21.15.45:0/1378602194 shutdown_connections 2024-04-16T16:18:16.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.510+0000 7f0419400640 1 --2- 172.21.15.45:0/1378602194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04141608a0 0x7f0414160d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.510+0000 7f0419400640 1 --2- 172.21.15.45:0/1378602194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0414146220 0x7f041415ccb0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.510+0000 7f0419400640 1 -- 172.21.15.45:0/1378602194 >> 172.21.15.45:0/1378602194 conn(0x7f041406b930 msgr2=0x7f041406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:16.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.510+0000 7f0419400640 1 -- 172.21.15.45:0/1378602194 shutdown_connections 2024-04-16T16:18:16.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.510+0000 7f0419400640 1 -- 172.21.15.45:0/1378602194 wait complete. 2024-04-16T16:18:16.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.512+0000 7f0419400640 1 Processor -- start 2024-04-16T16:18:16.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.512+0000 7f0419400640 1 -- start start 2024-04-16T16:18:16.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.513+0000 7f0419400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f041406f7a0 0x7f0414144dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:16.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.513+0000 7f0419400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0414146220 0x7f041415a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:16.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.513+0000 7f0419400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04141608a0 0x7f041415bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:16.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.513+0000 7f0419400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f041406fc20 con 0x7f04141608a0 2024-04-16T16:18:16.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.513+0000 7f0419400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f041415dc00 con 0x7f041406f7a0 2024-04-16T16:18:16.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.513+0000 7f0419400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f041406fda0 con 0x7f0414146220 2024-04-16T16:18:16.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.513+0000 7f0418a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04141608a0 0x7f041415bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:16.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.513+0000 7f0418a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04141608a0 0x7f041415bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57218/0 (socket says 172.21.15.45:57218) 2024-04-16T16:18:16.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.513+0000 7f0418a00640 1 -- 172.21.15.45:0/3145932679 learned_addr learned my addr 172.21.15.45:0/3145932679 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.513+0000 7f0413e00640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f041406f7a0 0x7f0414144dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.513+0000 7f040be00640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0414146220 0x7f041415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.514+0000 7f0418a00640 1 -- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0414146220 msgr2=0x7f041415a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.514+0000 7f0418a00640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0414146220 0x7f041415a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.514+0000 7f0418a00640 1 -- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f041406f7a0 msgr2=0x7f0414144dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.514+0000 7f0418a00640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f041406f7a0 0x7f0414144dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.514+0000 7f0418a00640 1 -- 172.21.15.45:0/3145932679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f041413ec60 con 0x7f04141608a0 2024-04-16T16:18:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.514+0000 7f0413e00640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f041406f7a0 0x7f0414144dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.514+0000 7f040be00640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0414146220 0x7f041415a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.514+0000 7f0418a00640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04141608a0 0x7f041415bd20 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f040c00cd50 tx=0x7f040c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.514+0000 7f0410c00640 1 -- 172.21.15.45:0/3145932679 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c00f680 con 0x7f04141608a0 2024-04-16T16:18:16.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.514+0000 7f0410c00640 1 -- 172.21.15.45:0/3145932679 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f040c00f840 con 0x7f04141608a0 2024-04-16T16:18:16.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.514+0000 7f0419400640 1 -- 172.21.15.45:0/3145932679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f041413ef40 con 0x7f04141608a0 2024-04-16T16:18:16.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.514+0000 7f0410c00640 1 -- 172.21.15.45:0/3145932679 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c00fc30 con 0x7f04141608a0 2024-04-16T16:18:16.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.515+0000 7f0419400640 1 -- 172.21.15.45:0/3145932679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f041413f500 con 0x7f04141608a0 2024-04-16T16:18:16.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.516+0000 7f0419400640 1 -- 172.21.15.45:0/3145932679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f041413f700 con 0x7f04141608a0 2024-04-16T16:18:16.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.517+0000 7f0410c00640 1 -- 172.21.15.45:0/3145932679 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f040c0213f0 con 0x7f04141608a0 2024-04-16T16:18:16.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.518+0000 7f0410c00640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f03f006ca10 0x7f03f006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:16.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.518+0000 7f0410c00640 1 -- 172.21.15.45:0/3145932679 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f040c08cb90 con 0x7f04141608a0 2024-04-16T16:18:16.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.519+0000 7f0413e00640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f03f006ca10 0x7f03f006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.520+0000 7f0413e00640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f03f006ca10 0x7f03f006eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0404036000 tx=0x7f0404008030 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:16.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.523+0000 7f0410c00640 1 -- 172.21.15.45:0/3145932679 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f041413f700 con 0x7f04141608a0 2024-04-16T16:18:16.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.725+0000 7f0419400640 1 -- 172.21.15.45:0/3145932679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04140cff60 con 0x7f04141608a0 2024-04-16T16:18:16.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.729+0000 7f0410c00640 1 -- 172.21.15.45:0/3145932679 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+16673 (secure 0 0 0) 0x7f040c014030 con 0x7f04141608a0 2024-04-16T16:18:16.728 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:16.728 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":84,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:18:15.762337+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:18:13.633261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-16T16:18:14.757055+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:18:16.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.731+0000 7f0419400640 1 -- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f03f006ca10 msgr2=0x7f03f006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:16.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.731+0000 7f0419400640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f03f006ca10 0x7f03f006eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0404036000 tx=0x7f0404008030 comp rx=0 tx=0).stop 2024-04-16T16:18:16.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.731+0000 7f0419400640 1 -- 172.21.15.45:0/3145932679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04141608a0 msgr2=0x7f041415bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:16.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.731+0000 7f0419400640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04141608a0 0x7f041415bd20 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f040c00cd50 tx=0x7f040c00a650 comp rx=0 tx=0).stop 2024-04-16T16:18:16.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.731+0000 7f0419400640 1 -- 172.21.15.45:0/3145932679 shutdown_connections 2024-04-16T16:18:16.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.731+0000 7f0419400640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f03f006ca10 0x7f03f006eed0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.731+0000 7f0419400640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04141608a0 0x7f041415bd20 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.731+0000 7f0419400640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0414146220 0x7f041415a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.731+0000 7f0419400640 1 --2- 172.21.15.45:0/3145932679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f041406f7a0 0x7f0414144dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.731+0000 7f0419400640 1 -- 172.21.15.45:0/3145932679 >> 172.21.15.45:0/3145932679 conn(0x7f041406b930 msgr2=0x7f04140c64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:16.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.732+0000 7f0419400640 1 -- 172.21.15.45:0/3145932679 shutdown_connections 2024-04-16T16:18:16.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.732+0000 7f0419400640 1 -- 172.21.15.45:0/3145932679 wait complete. 2024-04-16T16:18:16.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:18:16.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.969+0000 7f34e1400640 1 Processor -- start 2024-04-16T16:18:16.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.969+0000 7f34e1400640 1 -- start start 2024-04-16T16:18:16.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.971+0000 7f34e1400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f34dc146220 0x7f34dc15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.971+0000 7f34e1400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f34dc1608a0 0x7f34dc160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.971+0000 7f34e1400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34dc15d360 con 0x7f34dc06f7a0 2024-04-16T16:18:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.971+0000 7f34e1400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f34dc161230 con 0x7f34dc1608a0 2024-04-16T16:18:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.971+0000 7f34e1400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f34dc160410 con 0x7f34dc146220 2024-04-16T16:18:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.971+0000 7f34e0a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f34dc1608a0 0x7f34dc160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.971+0000 7f34e0a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f34dc1608a0 0x7f34dc160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57220/0 (socket says 172.21.15.45:57220) 2024-04-16T16:18:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.971+0000 7f34e0a00640 1 -- 172.21.15.45:0/2253296643 learned_addr learned my addr 172.21.15.45:0/2253296643 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.971+0000 7f34e0a00640 1 -- 172.21.15.45:0/2253296643 >> v2:172.21.15.103:3300/0 conn(0x7f34dc146220 msgr2=0x7f34dc15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.971+0000 7f34dbe00640 1 --2- 172.21.15.45:0/2253296643 >> v2:172.21.15.103:3300/0 conn(0x7f34dc146220 0x7f34dc15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.971+0000 7f34e0a00640 1 --2- 172.21.15.45:0/2253296643 >> v2:172.21.15.103:3300/0 conn(0x7f34dc146220 0x7f34dc15ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.971+0000 7f34e0a00640 1 -- 172.21.15.45:0/2253296643 >> v1:172.21.15.103:6789/0 conn(0x7f34dc06f7a0 legacy=0x7f34dc06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.972+0000 7f34e0a00640 1 -- 172.21.15.45:0/2253296643 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34dc15d640 con 0x7f34dc1608a0 2024-04-16T16:18:16.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.973+0000 7f34e0a00640 1 --2- 172.21.15.45:0/2253296643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34dc1608a0 0x7f34dc160cf0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f34dc070210 tx=0x7f34d402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65abe551da0aae97 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.973+0000 7f34daa00640 1 -- 172.21.15.45:0/2253296643 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d402ca80 con 0x7f34dc1608a0 2024-04-16T16:18:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.973+0000 7f34daa00640 1 -- 172.21.15.45:0/2253296643 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34d402cc20 con 0x7f34dc1608a0 2024-04-16T16:18:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.973+0000 7f34daa00640 1 -- 172.21.15.45:0/2253296643 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d40343f0 con 0x7f34dc1608a0 2024-04-16T16:18:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.973+0000 7f34e1400640 1 -- 172.21.15.45:0/2253296643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34dc1608a0 msgr2=0x7f34dc160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.973+0000 7f34e1400640 1 --2- 172.21.15.45:0/2253296643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34dc1608a0 0x7f34dc160cf0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f34dc070210 tx=0x7f34d402bee0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.974+0000 7f34e1400640 1 -- 172.21.15.45:0/2253296643 shutdown_connections 2024-04-16T16:18:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.974+0000 7f34e1400640 1 --2- 172.21.15.45:0/2253296643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34dc1608a0 0x7f34dc160cf0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.974+0000 7f34e1400640 1 --2- 172.21.15.45:0/2253296643 >> v2:172.21.15.103:3300/0 conn(0x7f34dc146220 0x7f34dc15ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.974+0000 7f34e1400640 1 -- 172.21.15.45:0/2253296643 >> 172.21.15.45:0/2253296643 conn(0x7f34dc06b930 msgr2=0x7f34dc06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.974+0000 7f34e1400640 1 -- 172.21.15.45:0/2253296643 shutdown_connections 2024-04-16T16:18:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.974+0000 7f34e1400640 1 -- 172.21.15.45:0/2253296643 wait complete. 2024-04-16T16:18:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.975+0000 7f34e1400640 1 Processor -- start 2024-04-16T16:18:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.976+0000 7f34e1400640 1 -- start start 2024-04-16T16:18:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.976+0000 7f34e1400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34dc06f7a0 0x7f34dc1448c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.976+0000 7f34e1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34dc146220 0x7f34dc150c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.977+0000 7f34e1400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34dc1608a0 0x7f34dc1511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.977+0000 7f34e1400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f34dc15dc00 con 0x7f34dc146220 2024-04-16T16:18:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.977+0000 7f34e1400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34dc06e650 con 0x7f34dc1608a0 2024-04-16T16:18:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.977+0000 7f34e1400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f34dc06fc20 con 0x7f34dc06f7a0 2024-04-16T16:18:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.977+0000 7f34db400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34dc146220 0x7f34dc150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.977+0000 7f34db400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34dc146220 0x7f34dc150c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57224/0 (socket says 172.21.15.45:57224) 2024-04-16T16:18:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.977+0000 7f34db400640 1 -- 172.21.15.45:0/2067861835 learned_addr learned my addr 172.21.15.45:0/2067861835 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.977+0000 7f34dbe00640 1 --2- 172.21.15.45:0/2067861835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34dc06f7a0 0x7f34dc1448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.977+0000 7f34e0a00640 1 --2- 172.21.15.45:0/2067861835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34dc1608a0 0x7f34dc1511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.977+0000 7f34db400640 1 -- 172.21.15.45:0/2067861835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34dc06f7a0 msgr2=0x7f34dc1448c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.977+0000 7f34db400640 1 --2- 172.21.15.45:0/2067861835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34dc06f7a0 0x7f34dc1448c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.977+0000 7f34db400640 1 -- 172.21.15.45:0/2067861835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34dc1608a0 msgr2=0x7f34dc1511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.977+0000 7f34db400640 1 --2- 172.21.15.45:0/2067861835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34dc1608a0 0x7f34dc1511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.977+0000 7f34db400640 1 -- 172.21.15.45:0/2067861835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34dc144e00 con 0x7f34dc146220 2024-04-16T16:18:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.978+0000 7f34db400640 1 --2- 172.21.15.45:0/2067861835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34dc146220 0x7f34dc150c80 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f34c800c410 tx=0x7f34c800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:16.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.978+0000 7f34d3e00640 1 -- 172.21.15.45:0/2067861835 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c8017070 con 0x7f34dc146220 2024-04-16T16:18:16.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.978+0000 7f34d3e00640 1 -- 172.21.15.45:0/2067861835 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c80035f0 con 0x7f34dc146220 2024-04-16T16:18:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.978+0000 7f34d3e00640 1 -- 172.21.15.45:0/2067861835 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c8003a20 con 0x7f34dc146220 2024-04-16T16:18:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.978+0000 7f34e1400640 1 -- 172.21.15.45:0/2067861835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34dc14f840 con 0x7f34dc146220 2024-04-16T16:18:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.978+0000 7f34e1400640 1 -- 172.21.15.45:0/2067861835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34dc14fe00 con 0x7f34dc146220 2024-04-16T16:18:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.979+0000 7f34e1400640 1 -- 172.21.15.45:0/2067861835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34dc075780 con 0x7f34dc146220 2024-04-16T16:18:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.980+0000 7f34d3e00640 1 -- 172.21.15.45:0/2067861835 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f34c801a070 con 0x7f34dc146220 2024-04-16T16:18:16.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.981+0000 7f34d3e00640 1 --2- 172.21.15.45:0/2067861835 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f34bc06ca10 0x7f34bc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:16.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.981+0000 7f34d3e00640 1 -- 172.21.15.45:0/2067861835 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f34c808b240 con 0x7f34dc146220 2024-04-16T16:18:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.982+0000 7f34dbe00640 1 --2- 172.21.15.45:0/2067861835 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f34bc06ca10 0x7f34bc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:16.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.983+0000 7f34dbe00640 1 --2- 172.21.15.45:0/2067861835 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f34bc06ca10 0x7f34bc06eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f34cc006fd0 tx=0x7f34cc008040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:16.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:16.986+0000 7f34d3e00640 1 -- 172.21.15.45:0/2067861835 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f34dc144e00 con 0x7f34dc146220 2024-04-16T16:18:17.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.193+0000 7f34e1400640 1 -- 172.21.15.45:0/2067861835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f34dc0cff60 con 0x7f34dc146220 2024-04-16T16:18:17.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.196+0000 7f34d3e00640 1 -- 172.21.15.45:0/2067861835 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+16673 (secure 0 0 0) 0x7f34c80594c0 con 0x7f34dc146220 2024-04-16T16:18:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:17.195 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":84,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:18:15.762337+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:18:13.633261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-16T16:18:14.757055+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:18:17.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.198+0000 7f34e1400640 1 -- 172.21.15.45:0/2067861835 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f34bc06ca10 msgr2=0x7f34bc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:17.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.198+0000 7f34e1400640 1 --2- 172.21.15.45:0/2067861835 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f34bc06ca10 0x7f34bc06eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f34cc006fd0 tx=0x7f34cc008040 comp rx=0 tx=0).stop 2024-04-16T16:18:17.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.198+0000 7f34e1400640 1 -- 172.21.15.45:0/2067861835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34dc146220 msgr2=0x7f34dc150c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:17.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.198+0000 7f34e1400640 1 --2- 172.21.15.45:0/2067861835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34dc146220 0x7f34dc150c80 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f34c800c410 tx=0x7f34c800c8d0 comp rx=0 tx=0).stop 2024-04-16T16:18:17.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.199+0000 7f34e1400640 1 -- 172.21.15.45:0/2067861835 shutdown_connections 2024-04-16T16:18:17.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.199+0000 7f34e1400640 1 --2- 172.21.15.45:0/2067861835 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f34bc06ca10 0x7f34bc06eed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:17.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.199+0000 7f34e1400640 1 --2- 172.21.15.45:0/2067861835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34dc1608a0 0x7f34dc1511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:17.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.199+0000 7f34e1400640 1 --2- 172.21.15.45:0/2067861835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34dc146220 0x7f34dc150c80 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:17.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.199+0000 7f34e1400640 1 --2- 172.21.15.45:0/2067861835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34dc06f7a0 0x7f34dc1448c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:17.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.199+0000 7f34e1400640 1 -- 172.21.15.45:0/2067861835 >> 172.21.15.45:0/2067861835 conn(0x7f34dc06b930 msgr2=0x7f34dc0cdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:17.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.199+0000 7f34e1400640 1 -- 172.21.15.45:0/2067861835 shutdown_connections 2024-04-16T16:18:17.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.199+0000 7f34e1400640 1 -- 172.21.15.45:0/2067861835 wait complete. 2024-04-16T16:18:17.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-16T16:18:17.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.445+0000 7f2ecca00640 1 Processor -- start 2024-04-16T16:18:17.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.446+0000 7f2ecca00640 1 -- start start 2024-04-16T16:18:17.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.447+0000 7f2ecca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2ec80d3e90 0x7f2ec8169930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:17.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.447+0000 7f2ecca00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ec80b6810 con 0x7f2ec80d18f0 2024-04-16T16:18:17.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.447+0000 7f2ecca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ec80b6af0 con 0x7f2ec80b6310 2024-04-16T16:18:17.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.447+0000 7f2ecca00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2ec80d29f0 con 0x7f2ec80d3e90 2024-04-16T16:18:17.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.448+0000 7f2ec7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2ec80d3e90 0x7f2ec8169930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:17.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.448+0000 7f2ec7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2ec80d3e90 0x7f2ec8169930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57236/0 (socket says 172.21.15.45:57236) 2024-04-16T16:18:17.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.448+0000 7f2ec7e00640 1 -- 172.21.15.45:0/1718372790 learned_addr learned my addr 172.21.15.45:0/1718372790 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:17.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.448+0000 7f2ec7e00640 1 -- 172.21.15.45:0/1718372790 >> v1:172.21.15.103:6789/0 conn(0x7f2ec80b6310 legacy=0x7f2ec8161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:17.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.448+0000 7f2ec7e00640 1 -- 172.21.15.45:0/1718372790 >> v1:172.21.15.45:6789/0 conn(0x7f2ec80d18f0 legacy=0x7f2ec80ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:17.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.448+0000 7f2ec7e00640 1 -- 172.21.15.45:0/1718372790 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec80b6dd0 con 0x7f2ec80d3e90 2024-04-16T16:18:17.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.449+0000 7f2ec7e00640 1 --2- 172.21.15.45:0/1718372790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec80d3e90 0x7f2ec8169930 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80d2b70 tx=0x7f2ec002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d826267603835f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:17.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.450+0000 7f2ec6a00640 1 -- 172.21.15.45:0/1718372790 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec002ca70 con 0x7f2ec80d3e90 2024-04-16T16:18:17.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.450+0000 7f2ec6a00640 1 -- 172.21.15.45:0/1718372790 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ec002cc10 con 0x7f2ec80d3e90 2024-04-16T16:18:17.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.450+0000 7f2ec6a00640 1 -- 172.21.15.45:0/1718372790 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec00343f0 con 0x7f2ec80d3e90 2024-04-16T16:18:17.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.450+0000 7f2ecca00640 1 -- 172.21.15.45:0/1718372790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec80d3e90 msgr2=0x7f2ec8169930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:17.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.450+0000 7f2ecca00640 1 --2- 172.21.15.45:0/1718372790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec80d3e90 0x7f2ec8169930 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80d2b70 tx=0x7f2ec002bed0 comp rx=0 tx=0).stop 2024-04-16T16:18:17.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.451+0000 7f2ecca00640 1 -- 172.21.15.45:0/1718372790 shutdown_connections 2024-04-16T16:18:17.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.451+0000 7f2ecca00640 1 --2- 172.21.15.45:0/1718372790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec80d3e90 0x7f2ec8169930 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:17.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.451+0000 7f2ecca00640 1 -- 172.21.15.45:0/1718372790 >> 172.21.15.45:0/1718372790 conn(0x7f2ec806cf90 msgr2=0x7f2ec806d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:17.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.451+0000 7f2ecca00640 1 -- 172.21.15.45:0/1718372790 shutdown_connections 2024-04-16T16:18:17.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.451+0000 7f2ecca00640 1 -- 172.21.15.45:0/1718372790 wait complete. 2024-04-16T16:18:17.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.452+0000 7f2ecca00640 1 Processor -- start 2024-04-16T16:18:17.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.453+0000 7f2ecca00640 1 -- start start 2024-04-16T16:18:17.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.453+0000 7f2ecca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec80b6310 0x7f2ec81f4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:17.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.453+0000 7f2ecca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ec80d18f0 0x7f2ec81f4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:17.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.453+0000 7f2ecca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ec80d3e90 0x7f2ec815aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:17.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.453+0000 7f2ecca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec80d1e50 con 0x7f2ec80b6310 2024-04-16T16:18:17.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.453+0000 7f2ecca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec80d1cd0 con 0x7f2ec80d18f0 2024-04-16T16:18:17.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.453+0000 7f2ecca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2ec80d1fd0 con 0x7f2ec80d3e90 2024-04-16T16:18:17.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.454+0000 7f2ec7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec80b6310 0x7f2ec81f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:17.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.454+0000 7f2ec7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec80b6310 0x7f2ec81f4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57252/0 (socket says 172.21.15.45:57252) 2024-04-16T16:18:17.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.454+0000 7f2ec7400640 1 -- 172.21.15.45:0/226551459 learned_addr learned my addr 172.21.15.45:0/226551459 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:17.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.454+0000 7f2ec7e00640 1 --2- 172.21.15.45:0/226551459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ec80d3e90 0x7f2ec815aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:17.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.454+0000 7f2ec7400640 1 -- 172.21.15.45:0/226551459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ec80d3e90 msgr2=0x7f2ec815aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:17.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.454+0000 7f2ec7400640 1 --2- 172.21.15.45:0/226551459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ec80d3e90 0x7f2ec815aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:17.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.454+0000 7f2ec7400640 1 -- 172.21.15.45:0/226551459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ec80d18f0 msgr2=0x7f2ec81f4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:17.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.454+0000 7f2ebfe00640 1 --2- 172.21.15.45:0/226551459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ec80d18f0 0x7f2ec81f4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:17.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.454+0000 7f2ec7400640 1 --2- 172.21.15.45:0/226551459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ec80d18f0 0x7f2ec81f4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:17.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.454+0000 7f2ec7400640 1 -- 172.21.15.45:0/226551459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec815bba0 con 0x7f2ec80b6310 2024-04-16T16:18:17.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.455+0000 7f2ec7400640 1 --2- 172.21.15.45:0/226551459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec80b6310 0x7f2ec81f4000 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800c400 tx=0x7f2eb800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:17.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.455+0000 7f2ebf400640 1 -- 172.21.15.45:0/226551459 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb80052d0 con 0x7f2ec80b6310 2024-04-16T16:18:17.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.455+0000 7f2ebf400640 1 -- 172.21.15.45:0/226551459 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2eb8005490 con 0x7f2ec80b6310 2024-04-16T16:18:17.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.455+0000 7f2ebf400640 1 -- 172.21.15.45:0/226551459 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8007260 con 0x7f2ec80b6310 2024-04-16T16:18:17.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.455+0000 7f2ecca00640 1 -- 172.21.15.45:0/226551459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ec815be80 con 0x7f2ec80b6310 2024-04-16T16:18:17.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.455+0000 7f2ecca00640 1 -- 172.21.15.45:0/226551459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ec81f49b0 con 0x7f2ec80b6310 2024-04-16T16:18:17.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.456+0000 7f2ecca00640 1 -- 172.21.15.45:0/226551459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ec8075780 con 0x7f2ec80b6310 2024-04-16T16:18:17.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.457+0000 7f2ebf400640 1 -- 172.21.15.45:0/226551459 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f2eb8007400 con 0x7f2ec80b6310 2024-04-16T16:18:17.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.458+0000 7f2ebf400640 1 --2- 172.21.15.45:0/226551459 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2ea406ca10 0x7f2ea406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:17.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.459+0000 7f2ebf400640 1 -- 172.21.15.45:0/226551459 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f2eb808b290 con 0x7f2ec80b6310 2024-04-16T16:18:17.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.459+0000 7f2ebfe00640 1 --2- 172.21.15.45:0/226551459 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2ea406ca10 0x7f2ea406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:17.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.460+0000 7f2ebfe00640 1 --2- 172.21.15.45:0/226551459 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2ea406ca10 0x7f2ea406eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80cf970 tx=0x7f2eb0009040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:17.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.464+0000 7f2ebf400640 1 -- 172.21.15.45:0/226551459 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2eb80595a0 con 0x7f2ec80b6310 2024-04-16T16:18:17.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:17.710+0000 7f2ecca00640 1 -- 172.21.15.45:0/226551459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f2ec80d25a0 con 0x7f2ec80b6310 2024-04-16T16:18:18.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.326+0000 7f2ebf400640 1 -- 172.21.15.45:0/226551459 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v63) ==== 118+0+0 (secure 0 0 0) 0x7f2eb8058ea0 con 0x7f2ec80b6310 2024-04-16T16:18:18.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.330+0000 7f2ecca00640 1 -- 172.21.15.45:0/226551459 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2ea406ca10 msgr2=0x7f2ea406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:18.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.330+0000 7f2ecca00640 1 --2- 172.21.15.45:0/226551459 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2ea406ca10 0x7f2ea406eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80cf970 tx=0x7f2eb0009040 comp rx=0 tx=0).stop 2024-04-16T16:18:18.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.330+0000 7f2ecca00640 1 -- 172.21.15.45:0/226551459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec80b6310 msgr2=0x7f2ec81f4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:18.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.330+0000 7f2ecca00640 1 --2- 172.21.15.45:0/226551459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec80b6310 0x7f2ec81f4000 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800c400 tx=0x7f2eb800c8c0 comp rx=0 tx=0).stop 2024-04-16T16:18:18.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.330+0000 7f2ecca00640 1 -- 172.21.15.45:0/226551459 shutdown_connections 2024-04-16T16:18:18.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.330+0000 7f2ecca00640 1 --2- 172.21.15.45:0/226551459 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2ea406ca10 0x7f2ea406eed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:18.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.330+0000 7f2ecca00640 1 --2- 172.21.15.45:0/226551459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ec80d3e90 0x7f2ec815aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:18.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.330+0000 7f2ecca00640 1 --2- 172.21.15.45:0/226551459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ec80d18f0 0x7f2ec81f4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:18.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.330+0000 7f2ecca00640 1 --2- 172.21.15.45:0/226551459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec80b6310 0x7f2ec81f4000 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:18.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.330+0000 7f2ecca00640 1 -- 172.21.15.45:0/226551459 >> 172.21.15.45:0/226551459 conn(0x7f2ec806cf90 msgr2=0x7f2ec806d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:18.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.330+0000 7f2ecca00640 1 -- 172.21.15.45:0/226551459 shutdown_connections 2024-04-16T16:18:18.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.330+0000 7f2ecca00640 1 -- 172.21.15.45:0/226551459 wait complete. 2024-04-16T16:18:18.354 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-16T16:18:18.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-16T16:18:18.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.583+0000 7ff948e00640 1 Processor -- start 2024-04-16T16:18:18.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.583+0000 7ff948e00640 1 -- start start 2024-04-16T16:18:18.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.584+0000 7ff948e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9440d18f0 0x7ff9440ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:18.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.584+0000 7ff948e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff9440b6310 0x7ff9440d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:18.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.584+0000 7ff948e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9440b6810 con 0x7ff9440d1cd0 2024-04-16T16:18:18.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.584+0000 7ff948e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff9440d29f0 con 0x7ff9440b6310 2024-04-16T16:18:18.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.584+0000 7ff948e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff9440b6af0 con 0x7ff9440d18f0 2024-04-16T16:18:18.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.585+0000 7ff943400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9440d18f0 0x7ff9440ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:18.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.585+0000 7ff93aa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff9440b6310 0x7ff9440d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:18.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.585+0000 7ff943400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9440d18f0 0x7ff9440ba220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:55516/0 (socket says 172.21.15.45:55516) 2024-04-16T16:18:18.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.585+0000 7ff943400640 1 -- 172.21.15.45:0/298639390 learned_addr learned my addr 172.21.15.45:0/298639390 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:18.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.585+0000 7ff93aa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff9440b6310 0x7ff9440d2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:41530/0 (socket says 172.21.15.45:41530) 2024-04-16T16:18:18.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.586+0000 7ff943400640 1 -- 172.21.15.45:0/298639390 >> v2:172.21.15.103:3300/0 conn(0x7ff9440b6310 msgr2=0x7ff9440d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:18.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.586+0000 7ff943400640 1 --2- 172.21.15.45:0/298639390 >> v2:172.21.15.103:3300/0 conn(0x7ff9440b6310 0x7ff9440d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:18.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.586+0000 7ff943400640 1 -- 172.21.15.45:0/298639390 >> v1:172.21.15.103:6789/0 conn(0x7ff9440d1cd0 legacy=0x7ff9440d33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:18.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.586+0000 7ff943400640 1 -- 172.21.15.45:0/298639390 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9440b6dd0 con 0x7ff9440d18f0 2024-04-16T16:18:18.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.587+0000 7ff943400640 1 --2- 172.21.15.45:0/298639390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9440d18f0 0x7ff9440ba220 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff934009e50 tx=0x7ff93402bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67b08541cd38bc09 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:18.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.588+0000 7ff942a00640 1 -- 172.21.15.45:0/298639390 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93402ca80 con 0x7ff9440d18f0 2024-04-16T16:18:18.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.588+0000 7ff942a00640 1 -- 172.21.15.45:0/298639390 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff93402cc20 con 0x7ff9440d18f0 2024-04-16T16:18:18.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.588+0000 7ff942a00640 1 -- 172.21.15.45:0/298639390 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9340343f0 con 0x7ff9440d18f0 2024-04-16T16:18:18.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.588+0000 7ff948e00640 1 -- 172.21.15.45:0/298639390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9440d18f0 msgr2=0x7ff9440ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:18.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.588+0000 7ff948e00640 1 --2- 172.21.15.45:0/298639390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9440d18f0 0x7ff9440ba220 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff934009e50 tx=0x7ff93402bee0 comp rx=0 tx=0).stop 2024-04-16T16:18:18.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.588+0000 7ff948e00640 1 -- 172.21.15.45:0/298639390 shutdown_connections 2024-04-16T16:18:18.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.588+0000 7ff948e00640 1 --2- 172.21.15.45:0/298639390 >> v2:172.21.15.103:3300/0 conn(0x7ff9440b6310 0x7ff9440d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:18.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.588+0000 7ff948e00640 1 --2- 172.21.15.45:0/298639390 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9440d18f0 0x7ff9440ba220 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:18.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.588+0000 7ff948e00640 1 -- 172.21.15.45:0/298639390 >> 172.21.15.45:0/298639390 conn(0x7ff94406cf90 msgr2=0x7ff94406d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:18.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.588+0000 7ff948e00640 1 -- 172.21.15.45:0/298639390 shutdown_connections 2024-04-16T16:18:18.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.589+0000 7ff948e00640 1 -- 172.21.15.45:0/298639390 wait complete. 2024-04-16T16:18:18.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.590+0000 7ff948e00640 1 Processor -- start 2024-04-16T16:18:18.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.591+0000 7ff948e00640 1 -- start start 2024-04-16T16:18:18.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.591+0000 7ff948e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9440b6310 0x7ff9440ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:18.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.591+0000 7ff948e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9440d18f0 0x7ff9440c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:18.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.591+0000 7ff948e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9440d1cd0 0x7ff9440b8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:18.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.591+0000 7ff948e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff944162dc0 con 0x7ff9440b6310 2024-04-16T16:18:18.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.591+0000 7ff948e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff9440ba760 con 0x7ff9440d1cd0 2024-04-16T16:18:18.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.591+0000 7ff948e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff944162f40 con 0x7ff9440d18f0 2024-04-16T16:18:18.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.592+0000 7ff943400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9440b6310 0x7ff9440ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:18.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.592+0000 7ff943400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9440b6310 0x7ff9440ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57256/0 (socket says 172.21.15.45:57256) 2024-04-16T16:18:18.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.592+0000 7ff943400640 1 -- 172.21.15.45:0/4176909314 learned_addr learned my addr 172.21.15.45:0/4176909314 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:18.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.592+0000 7ff943400640 1 -- 172.21.15.45:0/4176909314 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9440d18f0 msgr2=0x7ff9440c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:18.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.592+0000 7ff943400640 1 --2- 172.21.15.45:0/4176909314 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9440d18f0 0x7ff9440c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:18.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.592+0000 7ff943e00640 1 --2- 172.21.15.45:0/4176909314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9440d1cd0 0x7ff9440b8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:18.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.592+0000 7ff943400640 1 -- 172.21.15.45:0/4176909314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9440d1cd0 msgr2=0x7ff9440b8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:18.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.592+0000 7ff943400640 1 --2- 172.21.15.45:0/4176909314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9440d1cd0 0x7ff9440b8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:18.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.592+0000 7ff943400640 1 -- 172.21.15.45:0/4176909314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9441544d0 con 0x7ff9440b6310 2024-04-16T16:18:18.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.593+0000 7ff943400640 1 --2- 172.21.15.45:0/4176909314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9440b6310 0x7ff9440ce300 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7ff934000f80 tx=0x7ff934004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:18.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.593+0000 7ff93be00640 1 -- 172.21.15.45:0/4176909314 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff934043020 con 0x7ff9440b6310 2024-04-16T16:18:18.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.593+0000 7ff93be00640 1 -- 172.21.15.45:0/4176909314 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff93403e450 con 0x7ff9440b6310 2024-04-16T16:18:18.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.593+0000 7ff93be00640 1 -- 172.21.15.45:0/4176909314 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff934009510 con 0x7ff9440b6310 2024-04-16T16:18:18.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.593+0000 7ff948e00640 1 -- 172.21.15.45:0/4176909314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9441547b0 con 0x7ff9440b6310 2024-04-16T16:18:18.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.593+0000 7ff948e00640 1 -- 172.21.15.45:0/4176909314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff944154d70 con 0x7ff9440b6310 2024-04-16T16:18:18.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.594+0000 7ff948e00640 1 -- 172.21.15.45:0/4176909314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff944075780 con 0x7ff9440b6310 2024-04-16T16:18:18.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.595+0000 7ff93be00640 1 -- 172.21.15.45:0/4176909314 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7ff934046020 con 0x7ff9440b6310 2024-04-16T16:18:18.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.596+0000 7ff93be00640 1 --2- 172.21.15.45:0/4176909314 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff92006ca10 0x7ff92006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.596+0000 7ff93be00640 1 -- 172.21.15.45:0/4176909314 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7ff9340ae590 con 0x7ff9440b6310 2024-04-16T16:18:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.597+0000 7ff93aa00640 1 --2- 172.21.15.45:0/4176909314 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff92006ca10 0x7ff92006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:18.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.598+0000 7ff93aa00640 1 --2- 172.21.15.45:0/4176909314 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff92006ca10 0x7ff92006eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff92c006a90 tx=0x7ff92c006a20 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:18.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.601+0000 7ff93be00640 1 -- 172.21.15.45:0/4176909314 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff93404d030 con 0x7ff9440b6310 2024-04-16T16:18:18.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:18.856+0000 7ff948e00640 1 -- 172.21.15.45:0/4176909314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7ff94406d370 con 0x7ff9440b6310 2024-04-16T16:18:19.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.335+0000 7ff93be00640 1 -- 172.21.15.45:0/4176909314 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v64) ==== 122+0+0 (secure 0 0 0) 0x7ff93407c820 con 0x7ff9440b6310 2024-04-16T16:18:19.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.339+0000 7ff948e00640 1 -- 172.21.15.45:0/4176909314 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff92006ca10 msgr2=0x7ff92006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:19.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.339+0000 7ff948e00640 1 --2- 172.21.15.45:0/4176909314 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff92006ca10 0x7ff92006eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff92c006a90 tx=0x7ff92c006a20 comp rx=0 tx=0).stop 2024-04-16T16:18:19.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.339+0000 7ff948e00640 1 -- 172.21.15.45:0/4176909314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9440b6310 msgr2=0x7ff9440ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:19.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.339+0000 7ff948e00640 1 --2- 172.21.15.45:0/4176909314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9440b6310 0x7ff9440ce300 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7ff934000f80 tx=0x7ff934004a40 comp rx=0 tx=0).stop 2024-04-16T16:18:19.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.339+0000 7ff948e00640 1 -- 172.21.15.45:0/4176909314 shutdown_connections 2024-04-16T16:18:19.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.339+0000 7ff948e00640 1 --2- 172.21.15.45:0/4176909314 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff92006ca10 0x7ff92006eed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:19.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.339+0000 7ff948e00640 1 --2- 172.21.15.45:0/4176909314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9440d1cd0 0x7ff9440b8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:19.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.339+0000 7ff948e00640 1 --2- 172.21.15.45:0/4176909314 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9440d18f0 0x7ff9440c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:19.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.339+0000 7ff948e00640 1 --2- 172.21.15.45:0/4176909314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9440b6310 0x7ff9440ce300 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:19.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.339+0000 7ff948e00640 1 -- 172.21.15.45:0/4176909314 >> 172.21.15.45:0/4176909314 conn(0x7ff94406cf90 msgr2=0x7ff9440b4cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:19.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.340+0000 7ff948e00640 1 -- 172.21.15.45:0/4176909314 shutdown_connections 2024-04-16T16:18:19.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.340+0000 7ff948e00640 1 -- 172.21.15.45:0/4176909314 wait complete. 2024-04-16T16:18:19.363 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-16T16:18:19.363 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:18:19.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.585+0000 7ff239e00640 1 Processor -- start 2024-04-16T16:18:19.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.585+0000 7ff239e00640 1 -- start start 2024-04-16T16:18:19.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.587+0000 7ff239e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff2340cff30 0x7ff23406f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:19.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.587+0000 7ff239e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff234163510 con 0x7ff234163010 2024-04-16T16:18:19.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.587+0000 7ff239e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2341637f0 con 0x7ff2340cde00 2024-04-16T16:18:19.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.587+0000 7ff239e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff2340d0360 con 0x7ff2340cff30 2024-04-16T16:18:19.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.588+0000 7ff239400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff2340cff30 0x7ff23406f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:19.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.588+0000 7ff239400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff2340cff30 0x7ff23406f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57258/0 (socket says 172.21.15.45:57258) 2024-04-16T16:18:19.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.588+0000 7ff239400640 1 -- 172.21.15.45:0/4232651044 learned_addr learned my addr 172.21.15.45:0/4232651044 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:19.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.588+0000 7ff239400640 1 -- 172.21.15.45:0/4232651044 >> v1:172.21.15.103:6789/0 conn(0x7ff2340cde00 legacy=0x7ff2340ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:19.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.588+0000 7ff239400640 1 -- 172.21.15.45:0/4232651044 >> v1:172.21.15.45:6789/0 conn(0x7ff234163010 legacy=0x7ff2341645b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:19.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.588+0000 7ff239400640 1 -- 172.21.15.45:0/4232651044 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff234163ad0 con 0x7ff2340cff30 2024-04-16T16:18:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.590+0000 7ff239400640 1 --2- 172.21.15.45:0/4232651044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2340cff30 0x7ff23406f540 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7ff2340d04e0 tx=0x7ff22c02bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40fbeeab33f5c7af server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.590+0000 7ff233400640 1 -- 172.21.15.45:0/4232651044 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c02cb90 con 0x7ff2340cff30 2024-04-16T16:18:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.590+0000 7ff233400640 1 -- 172.21.15.45:0/4232651044 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff22c02cd30 con 0x7ff2340cff30 2024-04-16T16:18:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.590+0000 7ff233400640 1 -- 172.21.15.45:0/4232651044 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c034610 con 0x7ff2340cff30 2024-04-16T16:18:19.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.591+0000 7ff239e00640 1 -- 172.21.15.45:0/4232651044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2340cff30 msgr2=0x7ff23406f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:19.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.591+0000 7ff239e00640 1 --2- 172.21.15.45:0/4232651044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2340cff30 0x7ff23406f540 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7ff2340d04e0 tx=0x7ff22c02bff0 comp rx=0 tx=0).stop 2024-04-16T16:18:19.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.591+0000 7ff239e00640 1 -- 172.21.15.45:0/4232651044 shutdown_connections 2024-04-16T16:18:19.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.591+0000 7ff239e00640 1 --2- 172.21.15.45:0/4232651044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2340cff30 0x7ff23406f540 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:19.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.591+0000 7ff239e00640 1 -- 172.21.15.45:0/4232651044 >> 172.21.15.45:0/4232651044 conn(0x7ff23406b930 msgr2=0x7ff23406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:19.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.591+0000 7ff239e00640 1 -- 172.21.15.45:0/4232651044 shutdown_connections 2024-04-16T16:18:19.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.591+0000 7ff239e00640 1 -- 172.21.15.45:0/4232651044 wait complete. 2024-04-16T16:18:19.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.593+0000 7ff239e00640 1 Processor -- start 2024-04-16T16:18:19.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.593+0000 7ff239e00640 1 -- start start 2024-04-16T16:18:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.594+0000 7ff239e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2340cde00 0x7ff2341463b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.594+0000 7ff239e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2340cff30 0x7ff2340b1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.594+0000 7ff239e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff234163010 0x7ff2340b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.594+0000 7ff239e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff234164cc0 con 0x7ff234163010 2024-04-16T16:18:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.594+0000 7ff239e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff234164090 con 0x7ff2340cff30 2024-04-16T16:18:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.594+0000 7ff239e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff234164e40 con 0x7ff2340cde00 2024-04-16T16:18:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.594+0000 7ff239400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff234163010 0x7ff2340b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.594+0000 7ff239400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff234163010 0x7ff2340b2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57266/0 (socket says 172.21.15.45:57266) 2024-04-16T16:18:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.594+0000 7ff239400640 1 -- 172.21.15.45:0/3856711375 learned_addr learned my addr 172.21.15.45:0/3856711375 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.595+0000 7ff233e00640 1 --2- 172.21.15.45:0/3856711375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2340cff30 0x7ff2340b1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.595+0000 7ff238a00640 1 --2- 172.21.15.45:0/3856711375 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2340cde00 0x7ff2341463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.595+0000 7ff239400640 1 -- 172.21.15.45:0/3856711375 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2340cde00 msgr2=0x7ff2341463b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.595+0000 7ff239400640 1 --2- 172.21.15.45:0/3856711375 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2340cde00 0x7ff2341463b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.595+0000 7ff239400640 1 -- 172.21.15.45:0/3856711375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2340cff30 msgr2=0x7ff2340b1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.595+0000 7ff239400640 1 --2- 172.21.15.45:0/3856711375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2340cff30 0x7ff2340b1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.595+0000 7ff239400640 1 -- 172.21.15.45:0/3856711375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff234146d00 con 0x7ff234163010 2024-04-16T16:18:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.595+0000 7ff233e00640 1 --2- 172.21.15.45:0/3856711375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2340cff30 0x7ff2340b1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.595+0000 7ff239400640 1 --2- 172.21.15.45:0/3856711375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff234163010 0x7ff2340b2220 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7ff22c036040 tx=0x7ff22c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.596+0000 7ff230c00640 1 -- 172.21.15.45:0/3856711375 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c043020 con 0x7ff234163010 2024-04-16T16:18:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.596+0000 7ff230c00640 1 -- 172.21.15.45:0/3856711375 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff22c03e450 con 0x7ff234163010 2024-04-16T16:18:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.596+0000 7ff230c00640 1 -- 172.21.15.45:0/3856711375 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c009510 con 0x7ff234163010 2024-04-16T16:18:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.596+0000 7ff239e00640 1 -- 172.21.15.45:0/3856711375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff234149ba0 con 0x7ff234163010 2024-04-16T16:18:19.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.596+0000 7ff239e00640 1 -- 172.21.15.45:0/3856711375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff234147280 con 0x7ff234163010 2024-04-16T16:18:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.597+0000 7ff239e00640 1 -- 172.21.15.45:0/3856711375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff234075780 con 0x7ff234163010 2024-04-16T16:18:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.598+0000 7ff230c00640 1 -- 172.21.15.45:0/3856711375 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7ff22c046020 con 0x7ff234163010 2024-04-16T16:18:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.599+0000 7ff230c00640 1 --2- 172.21.15.45:0/3856711375 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff21406ca10 0x7ff21406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.599+0000 7ff230c00640 1 -- 172.21.15.45:0/3856711375 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7ff22c0ae5c0 con 0x7ff234163010 2024-04-16T16:18:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.600+0000 7ff238a00640 1 --2- 172.21.15.45:0/3856711375 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff21406ca10 0x7ff21406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:19.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.601+0000 7ff238a00640 1 --2- 172.21.15.45:0/3856711375 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff21406ca10 0x7ff21406eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff228000f60 tx=0x7ff228009040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:19.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.604+0000 7ff230c00640 1 -- 172.21.15.45:0/3856711375 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff22c04d030 con 0x7ff234163010 2024-04-16T16:18:19.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.853+0000 7ff239e00640 1 -- 172.21.15.45:0/3856711375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2340cf510 con 0x7ff234163010 2024-04-16T16:18:19.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.854+0000 7ff230c00640 1 -- 172.21.15.45:0/3856711375 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5186 (secure 0 0 0) 0x7ff22c07c840 con 0x7ff234163010 2024-04-16T16:18:19.854 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:19.854 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2514711815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2514711815},{"type":"v1","addr":"172.21.15.103:6836","nonce":2514711815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2189404408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2189404408},{"type":"v1","addr":"172.21.15.103:6837","nonce":2189404408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7179,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/991116998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":991116998},{"type":"v1","addr":"172.21.15.45:6834","nonce":991116998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:18:15.762294+0000","modified":"2024-04-16T16:18:18.858040+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7188":{"gid":7188,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3813781309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3813781309},{"type":"v1","addr":"172.21.15.45:6835","nonce":3813781309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7188,"qdb_cluster":[7188]},"id":6}]} 2024-04-16T16:18:19.854 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 64 2024-04-16T16:18:19.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.857+0000 7ff239e00640 1 -- 172.21.15.45:0/3856711375 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff21406ca10 msgr2=0x7ff21406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:19.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.857+0000 7ff239e00640 1 --2- 172.21.15.45:0/3856711375 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff21406ca10 0x7ff21406eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff228000f60 tx=0x7ff228009040 comp rx=0 tx=0).stop 2024-04-16T16:18:19.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.857+0000 7ff239e00640 1 -- 172.21.15.45:0/3856711375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff234163010 msgr2=0x7ff2340b2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:19.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.857+0000 7ff239e00640 1 --2- 172.21.15.45:0/3856711375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff234163010 0x7ff2340b2220 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7ff22c036040 tx=0x7ff22c004a40 comp rx=0 tx=0).stop 2024-04-16T16:18:19.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.858+0000 7ff239e00640 1 -- 172.21.15.45:0/3856711375 shutdown_connections 2024-04-16T16:18:19.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.858+0000 7ff239e00640 1 --2- 172.21.15.45:0/3856711375 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7ff21406ca10 0x7ff21406eed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:19.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.858+0000 7ff239e00640 1 --2- 172.21.15.45:0/3856711375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff234163010 0x7ff2340b2220 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:19.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.858+0000 7ff239e00640 1 --2- 172.21.15.45:0/3856711375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2340cff30 0x7ff2340b1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:19.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.858+0000 7ff239e00640 1 --2- 172.21.15.45:0/3856711375 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2340cde00 0x7ff2341463b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:19.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.858+0000 7ff239e00640 1 -- 172.21.15.45:0/3856711375 >> 172.21.15.45:0/3856711375 conn(0x7ff23406b930 msgr2=0x7ff23406eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:19.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.858+0000 7ff239e00640 1 -- 172.21.15.45:0/3856711375 shutdown_connections 2024-04-16T16:18:19.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:19.858+0000 7ff239e00640 1 -- 172.21.15.45:0/3856711375 wait complete. 2024-04-16T16:18:19.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:18:20.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.094+0000 7f7556c00640 1 Processor -- start 2024-04-16T16:18:20.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.095+0000 7f7556c00640 1 -- start start 2024-04-16T16:18:20.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.096+0000 7f7556c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7550161e90 0x7f75500cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:20.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.096+0000 7f7556c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75500d52d0 0x7f755006ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:20.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.096+0000 7f7556c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75500d3080 con 0x7f7550162270 2024-04-16T16:18:20.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.096+0000 7f7556c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f75500d2e30 con 0x7f7550161e90 2024-04-16T16:18:20.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.096+0000 7f7556c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f75500d3360 con 0x7f75500d52d0 2024-04-16T16:18:20.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.097+0000 7f7554e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7550162270 0x7f75500d3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59332/0 (socket says 172.21.15.45:59332) 2024-04-16T16:18:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.097+0000 7f7554e00640 1 -- 172.21.15.45:0/2756133799 learned_addr learned my addr 172.21.15.45:0/2756133799 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.097+0000 7f7556200640 1 --2- 172.21.15.45:0/2756133799 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75500d52d0 0x7f755006ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.097+0000 7f7555800640 1 --2- 172.21.15.45:0/2756133799 >> v2:172.21.15.103:3300/0 conn(0x7f7550161e90 0x7f75500cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.097+0000 7f754fe00640 1 -- 172.21.15.45:0/2756133799 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4105220639 0 0) 0x7f75500d3080 con 0x7f7550162270 2024-04-16T16:18:20.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.097+0000 7f754fe00640 1 -- 172.21.15.45:0/2756133799 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7538003520 con 0x7f7550162270 2024-04-16T16:18:20.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.097+0000 7f754fe00640 1 -- 172.21.15.45:0/2756133799 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 305553840 0 0) 0x7f7538003520 con 0x7f7550162270 2024-04-16T16:18:20.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f754fe00640 1 -- 172.21.15.45:0/2756133799 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75500d52d0 msgr2=0x7f755006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:20.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f754fe00640 1 --2- 172.21.15.45:0/2756133799 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75500d52d0 0x7f755006ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f754fe00640 1 -- 172.21.15.45:0/2756133799 >> v2:172.21.15.103:3300/0 conn(0x7f7550161e90 msgr2=0x7f75500cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:20.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f754fe00640 1 --2- 172.21.15.45:0/2756133799 >> v2:172.21.15.103:3300/0 conn(0x7f7550161e90 0x7f75500cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f754fe00640 1 -- 172.21.15.45:0/2756133799 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75500d3640 con 0x7f7550162270 2024-04-16T16:18:20.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f754fe00640 1 -- 172.21.15.45:0/2756133799 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f753c001670 con 0x7f7550162270 2024-04-16T16:18:20.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f754fe00640 1 -- 172.21.15.45:0/2756133799 >> v1:172.21.15.45:6789/0 conn(0x7f7550162270 legacy=0x7f75500d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:20.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f754fe00640 1 --2- 172.21.15.45:0/2756133799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75380039b0 0x7f7538003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:20.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f754fe00640 1 --2- 172.21.15.45:0/2756133799 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75500d52d0 0x7f7538004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:20.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f754fe00640 1 --2- 172.21.15.45:0/2756133799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7538004d40 0x7f75380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:20.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f754fe00640 1 -- 172.21.15.45:0/2756133799 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f75500d2e30 con 0x7f7538004d40 2024-04-16T16:18:20.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f754fe00640 1 -- 172.21.15.45:0/2756133799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f75500d3360 con 0x7f75380039b0 2024-04-16T16:18:20.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f754fe00640 1 -- 172.21.15.45:0/2756133799 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f753800b9c0 con 0x7f75500d52d0 2024-04-16T16:18:20.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f7556200640 1 --2- 172.21.15.45:0/2756133799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7538004d40 0x7f75380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:20.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f7555800640 1 --2- 172.21.15.45:0/2756133799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75380039b0 0x7f7538003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:20.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.098+0000 7f7554e00640 1 --2- 172.21.15.45:0/2756133799 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75500d52d0 0x7f7538004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:20.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.099+0000 7f7556200640 1 -- 172.21.15.45:0/2756133799 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75500d52d0 msgr2=0x7f7538004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:20.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.099+0000 7f7556200640 1 --2- 172.21.15.45:0/2756133799 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75500d52d0 0x7f7538004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.099+0000 7f7556200640 1 -- 172.21.15.45:0/2756133799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75380039b0 msgr2=0x7f7538003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:20.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.099+0000 7f7556200640 1 --2- 172.21.15.45:0/2756133799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75380039b0 0x7f7538003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.099+0000 7f7556200640 1 -- 172.21.15.45:0/2756133799 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7538003520 con 0x7f7538004d40 2024-04-16T16:18:20.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.100+0000 7f7556200640 1 --2- 172.21.15.45:0/2756133799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7538004d40 0x7f75380051c0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f7548009f70 tx=0x7f7538005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d36623985fc0bcea server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:20.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.100+0000 7f754fe00640 1 -- 172.21.15.45:0/2756133799 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7548039070 con 0x7f7538004d40 2024-04-16T16:18:20.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.100+0000 7f754fe00640 1 -- 172.21.15.45:0/2756133799 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f754802cc90 con 0x7f7538004d40 2024-04-16T16:18:20.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.100+0000 7f754fe00640 1 -- 172.21.15.45:0/2756133799 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7548009bf0 con 0x7f7538004d40 2024-04-16T16:18:20.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.101+0000 7f7556c00640 1 -- 172.21.15.45:0/2756133799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7538004d40 msgr2=0x7f75380051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:20.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.101+0000 7f7556c00640 1 --2- 172.21.15.45:0/2756133799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7538004d40 0x7f75380051c0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f7548009f70 tx=0x7f7538005830 comp rx=0 tx=0).stop 2024-04-16T16:18:20.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.101+0000 7f7556200640 1 -- 172.21.15.45:0/2756133799 reap_dead start 2024-04-16T16:18:20.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.101+0000 7f7556c00640 1 -- 172.21.15.45:0/2756133799 shutdown_connections 2024-04-16T16:18:20.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.101+0000 7f7556c00640 1 -- 172.21.15.45:0/2756133799 >> 172.21.15.45:0/2756133799 conn(0x7f75500d1940 msgr2=0x7f75500d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:20.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.101+0000 7f7556c00640 1 -- 172.21.15.45:0/2756133799 shutdown_connections 2024-04-16T16:18:20.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.101+0000 7f7556c00640 1 -- 172.21.15.45:0/2756133799 wait complete. 2024-04-16T16:18:20.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.103+0000 7f7556c00640 1 Processor -- start 2024-04-16T16:18:20.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.103+0000 7f7556c00640 1 -- start start 2024-04-16T16:18:20.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.104+0000 7f7556c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75501ea720 0x7f75501508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.104+0000 7f7556c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75501eab00 0x7f75501e7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.104+0000 7f7556c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f755014d1b0 0x7f755014d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.104+0000 7f7556c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f75500d4550 con 0x7f755014d1b0 2024-04-16T16:18:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.104+0000 7f7556c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f75500cfd10 con 0x7f75501eab00 2024-04-16T16:18:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.104+0000 7f7556c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f75500d46d0 con 0x7f75501ea720 2024-04-16T16:18:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.104+0000 7f7556200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f755014d1b0 0x7f755014d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.104+0000 7f7556200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f755014d1b0 0x7f755014d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57276/0 (socket says 172.21.15.45:57276) 2024-04-16T16:18:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.104+0000 7f7556200640 1 -- 172.21.15.45:0/3639566677 learned_addr learned my addr 172.21.15.45:0/3639566677 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.104+0000 7f7555800640 1 --2- 172.21.15.45:0/3639566677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75501ea720 0x7f75501508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.105+0000 7f7554e00640 1 --2- 172.21.15.45:0/3639566677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75501eab00 0x7f75501e7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.105+0000 7f7556200640 1 -- 172.21.15.45:0/3639566677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75501ea720 msgr2=0x7f75501508a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:20.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.105+0000 7f7556200640 1 --2- 172.21.15.45:0/3639566677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75501ea720 0x7f75501508a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.105+0000 7f7556200640 1 -- 172.21.15.45:0/3639566677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75501eab00 msgr2=0x7f75501e7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:20.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.105+0000 7f7556200640 1 --2- 172.21.15.45:0/3639566677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75501eab00 0x7f75501e7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.105+0000 7f7556200640 1 -- 172.21.15.45:0/3639566677 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7550150de0 con 0x7f755014d1b0 2024-04-16T16:18:20.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.105+0000 7f7556200640 1 --2- 172.21.15.45:0/3639566677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f755014d1b0 0x7f755014d630 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f7548037040 tx=0x7f7548009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:20.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.105+0000 7f754d600640 1 -- 172.21.15.45:0/3639566677 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7548039040 con 0x7f755014d1b0 2024-04-16T16:18:20.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.105+0000 7f754d600640 1 -- 172.21.15.45:0/3639566677 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7548045070 con 0x7f755014d1b0 2024-04-16T16:18:20.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.105+0000 7f754d600640 1 -- 172.21.15.45:0/3639566677 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7548009bf0 con 0x7f755014d1b0 2024-04-16T16:18:20.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.105+0000 7f7556c00640 1 -- 172.21.15.45:0/3639566677 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f755014e1a0 con 0x7f755014d1b0 2024-04-16T16:18:20.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.105+0000 7f7556c00640 1 -- 172.21.15.45:0/3639566677 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f755014e760 con 0x7f755014d1b0 2024-04-16T16:18:20.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.107+0000 7f7556c00640 1 -- 172.21.15.45:0/3639566677 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7550075780 con 0x7f755014d1b0 2024-04-16T16:18:20.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.108+0000 7f754d600640 1 -- 172.21.15.45:0/3639566677 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f7548005d10 con 0x7f755014d1b0 2024-04-16T16:18:20.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.108+0000 7f754d600640 1 --2- 172.21.15.45:0/3639566677 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f753006ca40 0x7f753006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:20.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.109+0000 7f754d600640 1 -- 172.21.15.45:0/3639566677 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f75480adde0 con 0x7f755014d1b0 2024-04-16T16:18:20.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.109+0000 7f7555800640 1 --2- 172.21.15.45:0/3639566677 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f753006ca40 0x7f753006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:20.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.110+0000 7f7555800640 1 --2- 172.21.15.45:0/3639566677 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f753006ca40 0x7f753006ef00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7544009950 tx=0x7f7544008040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:20.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.113+0000 7f754d600640 1 -- 172.21.15.45:0/3639566677 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7550150de0 con 0x7f755014d1b0 2024-04-16T16:18:20.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.318+0000 7f7556c00640 1 -- 172.21.15.45:0/3639566677 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75500cb720 con 0x7f755014d1b0 2024-04-16T16:18:20.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.320+0000 7f754d600640 1 -- 172.21.15.45:0/3639566677 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+16660 (secure 0 0 0) 0x7f754807c070 con 0x7f755014d1b0 2024-04-16T16:18:20.320 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:20.320 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":85,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:18:17.297849+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:18:13.633261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-16T16:18:14.757055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:18:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.323+0000 7f7556c00640 1 -- 172.21.15.45:0/3639566677 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f753006ca40 msgr2=0x7f753006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.323+0000 7f7556c00640 1 --2- 172.21.15.45:0/3639566677 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f753006ca40 0x7f753006ef00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7544009950 tx=0x7f7544008040 comp rx=0 tx=0).stop 2024-04-16T16:18:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.323+0000 7f7556c00640 1 -- 172.21.15.45:0/3639566677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f755014d1b0 msgr2=0x7f755014d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.323+0000 7f7556c00640 1 --2- 172.21.15.45:0/3639566677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f755014d1b0 0x7f755014d630 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f7548037040 tx=0x7f7548009750 comp rx=0 tx=0).stop 2024-04-16T16:18:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.323+0000 7f7556c00640 1 -- 172.21.15.45:0/3639566677 shutdown_connections 2024-04-16T16:18:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.323+0000 7f7556c00640 1 --2- 172.21.15.45:0/3639566677 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f753006ca40 0x7f753006ef00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.323+0000 7f7556c00640 1 --2- 172.21.15.45:0/3639566677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f755014d1b0 0x7f755014d630 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.323+0000 7f7556c00640 1 --2- 172.21.15.45:0/3639566677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75501eab00 0x7f75501e7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.323+0000 7f7556c00640 1 --2- 172.21.15.45:0/3639566677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75501ea720 0x7f75501508a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.323+0000 7f7556c00640 1 -- 172.21.15.45:0/3639566677 >> 172.21.15.45:0/3639566677 conn(0x7f75500d1940 msgr2=0x7f75500b7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:20.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.323+0000 7f7556c00640 1 -- 172.21.15.45:0/3639566677 shutdown_connections 2024-04-16T16:18:20.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.323+0000 7f7556c00640 1 -- 172.21.15.45:0/3639566677 wait complete. 2024-04-16T16:18:20.342 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-16T16:18:20.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:18:20.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.566+0000 7f5af8a00640 1 Processor -- start 2024-04-16T16:18:20.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.566+0000 7f5af8a00640 1 -- start start 2024-04-16T16:18:20.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.567+0000 7f5af8a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5af40ccb20 0x7f5af40cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:20.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.567+0000 7f5af8a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5af40ce9a0 0x7f5af40cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:20.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.568+0000 7f5af8a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5af415ecd0 con 0x7f5af415e8f0 2024-04-16T16:18:20.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.568+0000 7f5af8a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5af40ce2e0 con 0x7f5af40ce9a0 2024-04-16T16:18:20.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.568+0000 7f5af8a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5af40cd6d0 con 0x7f5af40ccb20 2024-04-16T16:18:20.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.568+0000 7f5af3400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5af415e8f0 0x7f5af40cc410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59336/0 (socket says 172.21.15.45:59336) 2024-04-16T16:18:20.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.568+0000 7f5af3400640 1 -- 172.21.15.45:0/13761133 learned_addr learned my addr 172.21.15.45:0/13761133 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:20.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.568+0000 7f5af3e00640 1 --2- 172.21.15.45:0/13761133 >> v2:172.21.15.45:3300/0 conn(0x7f5af40ce9a0 0x7f5af40cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:20.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.568+0000 7f5aebe00640 1 --2- 172.21.15.45:0/13761133 >> v2:172.21.15.103:3300/0 conn(0x7f5af40ccb20 0x7f5af40cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:20.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.569+0000 7f5af3e00640 1 -- 172.21.15.45:0/13761133 >> v2:172.21.15.103:3300/0 conn(0x7f5af40ccb20 msgr2=0x7f5af40cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:20.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.569+0000 7f5af3e00640 1 --2- 172.21.15.45:0/13761133 >> v2:172.21.15.103:3300/0 conn(0x7f5af40ccb20 0x7f5af40cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.569+0000 7f5af3e00640 1 -- 172.21.15.45:0/13761133 >> v1:172.21.15.45:6789/0 conn(0x7f5af415e8f0 legacy=0x7f5af40cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:20.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.569+0000 7f5af3e00640 1 -- 172.21.15.45:0/13761133 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af40cd9b0 con 0x7f5af40ce9a0 2024-04-16T16:18:20.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.570+0000 7f5af3e00640 1 --2- 172.21.15.45:0/13761133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af40ce9a0 0x7f5af40cb170 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f5af4070330 tx=0x7f5aec02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea9de8de9ffbf66 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:20.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.570+0000 7f5af2a00640 1 -- 172.21.15.45:0/13761133 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aec039070 con 0x7f5af40ce9a0 2024-04-16T16:18:20.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.570+0000 7f5af2a00640 1 -- 172.21.15.45:0/13761133 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aec02cbe0 con 0x7f5af40ce9a0 2024-04-16T16:18:20.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.570+0000 7f5af2a00640 1 -- 172.21.15.45:0/13761133 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aec034440 con 0x7f5af40ce9a0 2024-04-16T16:18:20.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.571+0000 7f5af8a00640 1 -- 172.21.15.45:0/13761133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af40ce9a0 msgr2=0x7f5af40cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:20.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.571+0000 7f5af8a00640 1 --2- 172.21.15.45:0/13761133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af40ce9a0 0x7f5af40cb170 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f5af4070330 tx=0x7f5aec02be30 comp rx=0 tx=0).stop 2024-04-16T16:18:20.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.571+0000 7f5af8a00640 1 -- 172.21.15.45:0/13761133 shutdown_connections 2024-04-16T16:18:20.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.571+0000 7f5af8a00640 1 --2- 172.21.15.45:0/13761133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af40ce9a0 0x7f5af40cb170 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.571+0000 7f5af8a00640 1 --2- 172.21.15.45:0/13761133 >> v2:172.21.15.103:3300/0 conn(0x7f5af40ccb20 0x7f5af40cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.571+0000 7f5af8a00640 1 -- 172.21.15.45:0/13761133 >> 172.21.15.45:0/13761133 conn(0x7f5af406b930 msgr2=0x7f5af406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:20.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.571+0000 7f5af8a00640 1 -- 172.21.15.45:0/13761133 shutdown_connections 2024-04-16T16:18:20.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.571+0000 7f5af8a00640 1 -- 172.21.15.45:0/13761133 wait complete. 2024-04-16T16:18:20.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.573+0000 7f5af8a00640 1 Processor -- start 2024-04-16T16:18:20.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.573+0000 7f5af8a00640 1 -- start start 2024-04-16T16:18:20.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.574+0000 7f5af8a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af40ccb20 0x7f5af40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:20.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.574+0000 7f5af8a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5af40ce9a0 0x7f5af4142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:20.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.574+0000 7f5af8a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5af415e8f0 0x7f5af4156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:20.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.574+0000 7f5af8a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5af40cd360 con 0x7f5af40ccb20 2024-04-16T16:18:20.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.574+0000 7f5af8a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5af40cdf70 con 0x7f5af40ce9a0 2024-04-16T16:18:20.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.574+0000 7f5af8a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5af40cf080 con 0x7f5af415e8f0 2024-04-16T16:18:20.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.574+0000 7f5af3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af40ccb20 0x7f5af40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:20.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.574+0000 7f5af3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af40ccb20 0x7f5af40b4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57286/0 (socket says 172.21.15.45:57286) 2024-04-16T16:18:20.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.574+0000 7f5af3400640 1 -- 172.21.15.45:0/3341929292 learned_addr learned my addr 172.21.15.45:0/3341929292 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:20.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.574+0000 7f5af3400640 1 -- 172.21.15.45:0/3341929292 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5af415e8f0 msgr2=0x7f5af4156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:20.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.574+0000 7f5af3400640 1 --2- 172.21.15.45:0/3341929292 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5af415e8f0 0x7f5af4156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.574+0000 7f5af3400640 1 -- 172.21.15.45:0/3341929292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5af40ce9a0 msgr2=0x7f5af4142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:20.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.574+0000 7f5af3400640 1 --2- 172.21.15.45:0/3341929292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5af40ce9a0 0x7f5af4142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.574+0000 7f5af3400640 1 -- 172.21.15.45:0/3341929292 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af40b5860 con 0x7f5af40ccb20 2024-04-16T16:18:20.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.575+0000 7f5af3400640 1 --2- 172.21.15.45:0/3341929292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af40ccb20 0x7f5af40b4f30 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400c3f0 tx=0x7f5ae400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:20.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.575+0000 7f5aeb400640 1 -- 172.21.15.45:0/3341929292 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4017070 con 0x7f5af40ccb20 2024-04-16T16:18:20.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.575+0000 7f5aeb400640 1 -- 172.21.15.45:0/3341929292 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae4005ce0 con 0x7f5af40ccb20 2024-04-16T16:18:20.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.575+0000 7f5af8a00640 1 -- 172.21.15.45:0/3341929292 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5af40b5580 con 0x7f5af40ccb20 2024-04-16T16:18:20.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.575+0000 7f5aeb400640 1 -- 172.21.15.45:0/3341929292 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40072f0 con 0x7f5af40ccb20 2024-04-16T16:18:20.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.575+0000 7f5af8a00640 1 -- 172.21.15.45:0/3341929292 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5af4160ee0 con 0x7f5af40ccb20 2024-04-16T16:18:20.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.576+0000 7f5af8a00640 1 -- 172.21.15.45:0/3341929292 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5af4075780 con 0x7f5af40ccb20 2024-04-16T16:18:20.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.577+0000 7f5aeb400640 1 -- 172.21.15.45:0/3341929292 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f5ae401a070 con 0x7f5af40ccb20 2024-04-16T16:18:20.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.578+0000 7f5aeb400640 1 --2- 172.21.15.45:0/3341929292 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f5ad006ca40 0x7f5ad006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:20.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.578+0000 7f5aeb400640 1 -- 172.21.15.45:0/3341929292 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f5ae408b100 con 0x7f5af40ccb20 2024-04-16T16:18:20.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.579+0000 7f5aebe00640 1 --2- 172.21.15.45:0/3341929292 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f5ad006ca40 0x7f5ad006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:20.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.580+0000 7f5aebe00640 1 --2- 172.21.15.45:0/3341929292 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f5ad006ca40 0x7f5ad006ef00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5af40cf5c0 tx=0x7f5adc008040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:20.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.583+0000 7f5aeb400640 1 -- 172.21.15.45:0/3341929292 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5ae4059380 con 0x7f5af40ccb20 2024-04-16T16:18:20.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.785+0000 7f5af8a00640 1 -- 172.21.15.45:0/3341929292 --> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5af40b2b80 con 0x7f5ad006ca40 2024-04-16T16:18:20.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.807+0000 7f5aeb400640 1 -- 172.21.15.45:0/3341929292 <== mgr.5791 v2:172.21.15.103:6832/3728775368 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246761 (secure 0 0 0) 0x7f5af40b2b80 con 0x7f5ad006ca40 2024-04-16T16:18:20.809 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:20.811 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:18:20.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.812+0000 7f5af8a00640 1 -- 172.21.15.45:0/3341929292 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f5ad006ca40 msgr2=0x7f5ad006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:20.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.813+0000 7f5af8a00640 1 --2- 172.21.15.45:0/3341929292 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f5ad006ca40 0x7f5ad006ef00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5af40cf5c0 tx=0x7f5adc008040 comp rx=0 tx=0).stop 2024-04-16T16:18:20.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.813+0000 7f5af8a00640 1 -- 172.21.15.45:0/3341929292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af40ccb20 msgr2=0x7f5af40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:20.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.813+0000 7f5af8a00640 1 --2- 172.21.15.45:0/3341929292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af40ccb20 0x7f5af40b4f30 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400c3f0 tx=0x7f5ae400c8b0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.813+0000 7f5af8a00640 1 -- 172.21.15.45:0/3341929292 shutdown_connections 2024-04-16T16:18:20.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.813+0000 7f5af8a00640 1 --2- 172.21.15.45:0/3341929292 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f5ad006ca40 0x7f5ad006ef00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.813+0000 7f5af8a00640 1 --2- 172.21.15.45:0/3341929292 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5af415e8f0 0x7f5af4156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.813+0000 7f5af8a00640 1 --2- 172.21.15.45:0/3341929292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5af40ce9a0 0x7f5af4142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.813+0000 7f5af8a00640 1 --2- 172.21.15.45:0/3341929292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af40ccb20 0x7f5af40b4f30 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:20.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.813+0000 7f5af8a00640 1 -- 172.21.15.45:0/3341929292 >> 172.21.15.45:0/3341929292 conn(0x7f5af406b930 msgr2=0x7f5af40cb620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:20.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.813+0000 7f5af8a00640 1 -- 172.21.15.45:0/3341929292 shutdown_connections 2024-04-16T16:18:20.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:20.813+0000 7f5af8a00640 1 -- 172.21.15.45:0/3341929292 wait complete. 2024-04-16T16:18:20.831 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-16T16:18:19.118703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258552,"kb_used_data":9092,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716168,"statfs":{"total":773094113280,"available":772829356032,"internally_reserved":0,"allocated":9310208,"data_stored":4988559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.861340"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.298782+0000","last_change":"2024-04-16T16:18:15.298782+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.298782+0000","last_undegraded":"2024-04-16T16:18:15.298782+0000","last_fullsized":"2024-04-16T16:18:15.298782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293043+0000","last_change":"2024-04-16T16:18:15.293043+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293043+0000","last_undegraded":"2024-04-16T16:18:15.293043+0000","last_fullsized":"2024-04-16T16:18:15.293043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.284139+0000","last_change":"2024-04-16T16:18:15.284139+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.284139+0000","last_undegraded":"2024-04-16T16:18:15.284139+0000","last_fullsized":"2024-04-16T16:18:15.284139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.285012+0000","last_change":"2024-04-16T16:18:15.285012+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.285012+0000","last_undegraded":"2024-04-16T16:18:15.285012+0000","last_fullsized":"2024-04-16T16:18:15.285012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289497+0000","last_change":"2024-04-16T16:18:15.289497+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289497+0000","last_undegraded":"2024-04-16T16:18:15.289497+0000","last_fullsized":"2024-04-16T16:18:15.289497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.292510+0000","last_change":"2024-04-16T16:18:15.292510+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.292510+0000","last_undegraded":"2024-04-16T16:18:15.292510+0000","last_fullsized":"2024-04-16T16:18:15.292510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.284152+0000","last_change":"2024-04-16T16:18:15.284152+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.284152+0000","last_undegraded":"2024-04-16T16:18:15.284152+0000","last_fullsized":"2024-04-16T16:18:15.284152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.291320+0000","last_change":"2024-04-16T16:18:15.291320+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.291320+0000","last_undegraded":"2024-04-16T16:18:15.291320+0000","last_fullsized":"2024-04-16T16:18:15.291320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.297496+0000","last_change":"2024-04-16T16:18:15.297496+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.297496+0000","last_undegraded":"2024-04-16T16:18:15.297496+0000","last_fullsized":"2024-04-16T16:18:15.297496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.295832+0000","last_change":"2024-04-16T16:18:15.295832+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.295832+0000","last_undegraded":"2024-04-16T16:18:15.295832+0000","last_fullsized":"2024-04-16T16:18:15.295832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.291161+0000","last_change":"2024-04-16T16:18:15.291161+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.291161+0000","last_undegraded":"2024-04-16T16:18:15.291161+0000","last_fullsized":"2024-04-16T16:18:15.291161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.304631+0000","last_change":"2024-04-16T16:18:15.304631+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.304631+0000","last_undegraded":"2024-04-16T16:18:15.304631+0000","last_fullsized":"2024-04-16T16:18:15.304631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.281906+0000","last_change":"2024-04-16T16:18:15.281906+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.281906+0000","last_undegraded":"2024-04-16T16:18:15.281906+0000","last_fullsized":"2024-04-16T16:18:15.281906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.285574+0000","last_change":"2024-04-16T16:18:15.285574+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.285574+0000","last_undegraded":"2024-04-16T16:18:15.285574+0000","last_fullsized":"2024-04-16T16:18:15.285574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.282359+0000","last_change":"2024-04-16T16:18:15.282359+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.282359+0000","last_undegraded":"2024-04-16T16:18:15.282359+0000","last_fullsized":"2024-04-16T16:18:15.282359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.287619+0000","last_change":"2024-04-16T16:18:15.287619+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.287619+0000","last_undegraded":"2024-04-16T16:18:15.287619+0000","last_fullsized":"2024-04-16T16:18:15.287619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.290484+0000","last_change":"2024-04-16T16:18:15.290484+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.290484+0000","last_undegraded":"2024-04-16T16:18:15.290484+0000","last_fullsized":"2024-04-16T16:18:15.290484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.284930+0000","last_change":"2024-04-16T16:18:15.284930+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.284930+0000","last_undegraded":"2024-04-16T16:18:15.284930+0000","last_fullsized":"2024-04-16T16:18:15.284930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.288391+0000","last_change":"2024-04-16T16:18:15.288391+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.288391+0000","last_undegraded":"2024-04-16T16:18:15.288391+0000","last_fullsized":"2024-04-16T16:18:15.288391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.283891+0000","last_change":"2024-04-16T16:18:15.283891+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.283891+0000","last_undegraded":"2024-04-16T16:18:15.283891+0000","last_fullsized":"2024-04-16T16:18:15.283891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.286476+0000","last_change":"2024-04-16T16:18:15.286476+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.286476+0000","last_undegraded":"2024-04-16T16:18:15.286476+0000","last_fullsized":"2024-04-16T16:18:15.286476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296503+0000","last_change":"2024-04-16T16:18:15.296503+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296503+0000","last_undegraded":"2024-04-16T16:18:15.296503+0000","last_fullsized":"2024-04-16T16:18:15.296503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296468+0000","last_change":"2024-04-16T16:18:15.296468+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296468+0000","last_undegraded":"2024-04-16T16:18:15.296468+0000","last_fullsized":"2024-04-16T16:18:15.296468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.286848+0000","last_change":"2024-04-16T16:18:15.286848+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.286848+0000","last_undegraded":"2024-04-16T16:18:15.286848+0000","last_fullsized":"2024-04-16T16:18:15.286848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293145+0000","last_change":"2024-04-16T16:18:15.293145+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293145+0000","last_undegraded":"2024-04-16T16:18:15.293145+0000","last_fullsized":"2024-04-16T16:18:15.293145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.282088+0000","last_change":"2024-04-16T16:18:15.282088+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.282088+0000","last_undegraded":"2024-04-16T16:18:15.282088+0000","last_fullsized":"2024-04-16T16:18:15.282088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.292185+0000","last_change":"2024-04-16T16:18:15.292185+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.292185+0000","last_undegraded":"2024-04-16T16:18:15.292185+0000","last_fullsized":"2024-04-16T16:18:15.292185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.304185+0000","last_change":"2024-04-16T16:18:15.304185+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.304185+0000","last_undegraded":"2024-04-16T16:18:15.304185+0000","last_fullsized":"2024-04-16T16:18:15.304185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289755+0000","last_change":"2024-04-16T16:18:15.289755+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289755+0000","last_undegraded":"2024-04-16T16:18:15.289755+0000","last_fullsized":"2024-04-16T16:18:15.289755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296039+0000","last_change":"2024-04-16T16:18:15.296039+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296039+0000","last_undegraded":"2024-04-16T16:18:15.296039+0000","last_fullsized":"2024-04-16T16:18:15.296039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.287029+0000","last_change":"2024-04-16T16:18:15.287029+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.287029+0000","last_undegraded":"2024-04-16T16:18:15.287029+0000","last_fullsized":"2024-04-16T16:18:15.287029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.302832+0000","last_change":"2024-04-16T16:18:15.302832+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.302832+0000","last_undegraded":"2024-04-16T16:18:15.302832+0000","last_fullsized":"2024-04-16T16:18:15.302832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.291173+0000","last_change":"2024-04-16T16:18:15.291173+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.291173+0000","last_undegraded":"2024-04-16T16:18:15.291173+0000","last_fullsized":"2024-04-16T16:18:15.291173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311763+0000","last_change":"2024-04-16T16:18:15.307894+0000","last_active":"2024-04-16T16:18:15.311763+0000","last_peered":"2024-04-16T16:18:15.311763+0000","last_clean":"2024-04-16T16:18:15.311763+0000","last_became_active":"2024-04-16T16:18:15.307584+0000","last_became_peered":"2024-04-16T16:18:15.307584+0000","last_unstale":"2024-04-16T16:18:15.311763+0000","last_undegraded":"2024-04-16T16:18:15.311763+0000","last_fullsized":"2024-04-16T16:18:15.311763+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:30:36.737096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.297298+0000","last_change":"2024-04-16T16:18:15.297298+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.297298+0000","last_undegraded":"2024-04-16T16:18:15.297298+0000","last_fullsized":"2024-04-16T16:18:15.297298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.303705+0000","last_change":"2024-04-16T16:18:15.299901+0000","last_active":"2024-04-16T16:18:15.303705+0000","last_peered":"2024-04-16T16:18:15.303705+0000","last_clean":"2024-04-16T16:18:15.303705+0000","last_became_active":"2024-04-16T16:18:15.299622+0000","last_became_peered":"2024-04-16T16:18:15.299622+0000","last_unstale":"2024-04-16T16:18:15.303705+0000","last_undegraded":"2024-04-16T16:18:15.303705+0000","last_fullsized":"2024-04-16T16:18:15.303705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:51.934340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.303193+0000","last_change":"2024-04-16T16:18:15.303193+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.303193+0000","last_undegraded":"2024-04-16T16:18:15.303193+0000","last_fullsized":"2024-04-16T16:18:15.303193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.306427+0000","last_change":"2024-04-16T16:18:15.297918+0000","last_active":"2024-04-16T16:18:15.306427+0000","last_peered":"2024-04-16T16:18:15.306427+0000","last_clean":"2024-04-16T16:18:15.306427+0000","last_became_active":"2024-04-16T16:18:15.297217+0000","last_became_peered":"2024-04-16T16:18:15.297217+0000","last_unstale":"2024-04-16T16:18:15.306427+0000","last_undegraded":"2024-04-16T16:18:15.306427+0000","last_fullsized":"2024-04-16T16:18:15.306427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:05:41.836526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.283014+0000","last_change":"2024-04-16T16:18:15.283014+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.283014+0000","last_undegraded":"2024-04-16T16:18:15.283014+0000","last_fullsized":"2024-04-16T16:18:15.283014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.307736+0000","last_change":"2024-04-16T16:18:15.300207+0000","last_active":"2024-04-16T16:18:15.307736+0000","last_peered":"2024-04-16T16:18:15.307736+0000","last_clean":"2024-04-16T16:18:15.307736+0000","last_became_active":"2024-04-16T16:18:15.297220+0000","last_became_peered":"2024-04-16T16:18:15.297220+0000","last_unstale":"2024-04-16T16:18:15.307736+0000","last_undegraded":"2024-04-16T16:18:15.307736+0000","last_fullsized":"2024-04-16T16:18:15.307736+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:30:55.552396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.295251+0000","last_change":"2024-04-16T16:18:15.295251+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.295251+0000","last_undegraded":"2024-04-16T16:18:15.295251+0000","last_fullsized":"2024-04-16T16:18:15.295251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311825+0000","last_change":"2024-04-16T16:18:15.309872+0000","last_active":"2024-04-16T16:18:15.311825+0000","last_peered":"2024-04-16T16:18:15.311825+0000","last_clean":"2024-04-16T16:18:15.311825+0000","last_became_active":"2024-04-16T16:18:15.309546+0000","last_became_peered":"2024-04-16T16:18:15.309546+0000","last_unstale":"2024-04-16T16:18:15.311825+0000","last_undegraded":"2024-04-16T16:18:15.311825+0000","last_fullsized":"2024-04-16T16:18:15.311825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:27:40.938359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.288554+0000","last_change":"2024-04-16T16:18:15.288554+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.288554+0000","last_undegraded":"2024-04-16T16:18:15.288554+0000","last_fullsized":"2024-04-16T16:18:15.288554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.303669+0000","last_change":"2024-04-16T16:18:15.298912+0000","last_active":"2024-04-16T16:18:15.303669+0000","last_peered":"2024-04-16T16:18:15.303669+0000","last_clean":"2024-04-16T16:18:15.303669+0000","last_became_active":"2024-04-16T16:18:15.298638+0000","last_became_peered":"2024-04-16T16:18:15.298638+0000","last_unstale":"2024-04-16T16:18:15.303669+0000","last_undegraded":"2024-04-16T16:18:15.303669+0000","last_fullsized":"2024-04-16T16:18:15.303669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:30:36.737096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.282766+0000","last_change":"2024-04-16T16:18:15.282766+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.282766+0000","last_undegraded":"2024-04-16T16:18:15.282766+0000","last_fullsized":"2024-04-16T16:18:15.282766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.307254+0000","last_change":"2024-04-16T16:18:15.301897+0000","last_active":"2024-04-16T16:18:15.307254+0000","last_peered":"2024-04-16T16:18:15.307254+0000","last_clean":"2024-04-16T16:18:15.307254+0000","last_became_active":"2024-04-16T16:18:15.301549+0000","last_became_peered":"2024-04-16T16:18:15.301549+0000","last_unstale":"2024-04-16T16:18:15.307254+0000","last_undegraded":"2024-04-16T16:18:15.307254+0000","last_fullsized":"2024-04-16T16:18:15.307254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:57.659561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.281621+0000","last_change":"2024-04-16T16:18:15.281621+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.281621+0000","last_undegraded":"2024-04-16T16:18:15.281621+0000","last_fullsized":"2024-04-16T16:18:15.281621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.307302+0000","last_change":"2024-04-16T16:18:15.303661+0000","last_active":"2024-04-16T16:18:15.307302+0000","last_peered":"2024-04-16T16:18:15.307302+0000","last_clean":"2024-04-16T16:18:15.307302+0000","last_became_active":"2024-04-16T16:18:15.300455+0000","last_became_peered":"2024-04-16T16:18:15.300455+0000","last_unstale":"2024-04-16T16:18:15.307302+0000","last_undegraded":"2024-04-16T16:18:15.307302+0000","last_fullsized":"2024-04-16T16:18:15.307302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:21:04.639131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.281257+0000","last_change":"2024-04-16T16:18:15.281257+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.281257+0000","last_undegraded":"2024-04-16T16:18:15.281257+0000","last_fullsized":"2024-04-16T16:18:15.281257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311275+0000","last_change":"2024-04-16T16:18:15.305839+0000","last_active":"2024-04-16T16:18:15.311275+0000","last_peered":"2024-04-16T16:18:15.311275+0000","last_clean":"2024-04-16T16:18:15.311275+0000","last_became_active":"2024-04-16T16:18:15.305453+0000","last_became_peered":"2024-04-16T16:18:15.305453+0000","last_unstale":"2024-04-16T16:18:15.311275+0000","last_undegraded":"2024-04-16T16:18:15.311275+0000","last_fullsized":"2024-04-16T16:18:15.311275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:51:24.645998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293391+0000","last_change":"2024-04-16T16:18:15.293391+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293391+0000","last_undegraded":"2024-04-16T16:18:15.293391+0000","last_fullsized":"2024-04-16T16:18:15.293391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311136+0000","last_change":"2024-04-16T16:18:15.304936+0000","last_active":"2024-04-16T16:18:15.311136+0000","last_peered":"2024-04-16T16:18:15.311136+0000","last_clean":"2024-04-16T16:18:15.311136+0000","last_became_active":"2024-04-16T16:18:15.304590+0000","last_became_peered":"2024-04-16T16:18:15.304590+0000","last_unstale":"2024-04-16T16:18:15.311136+0000","last_undegraded":"2024-04-16T16:18:15.311136+0000","last_fullsized":"2024-04-16T16:18:15.311136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:33.215073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.288806+0000","last_change":"2024-04-16T16:18:15.288806+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.288806+0000","last_undegraded":"2024-04-16T16:18:15.288806+0000","last_fullsized":"2024-04-16T16:18:15.288806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311896+0000","last_change":"2024-04-16T16:18:15.308173+0000","last_active":"2024-04-16T16:18:15.311896+0000","last_peered":"2024-04-16T16:18:15.311896+0000","last_clean":"2024-04-16T16:18:15.311896+0000","last_became_active":"2024-04-16T16:18:15.307837+0000","last_became_peered":"2024-04-16T16:18:15.307837+0000","last_unstale":"2024-04-16T16:18:15.311896+0000","last_undegraded":"2024-04-16T16:18:15.311896+0000","last_fullsized":"2024-04-16T16:18:15.311896+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:45:40.148360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.290639+0000","last_change":"2024-04-16T16:18:15.290639+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.290639+0000","last_undegraded":"2024-04-16T16:18:15.290639+0000","last_fullsized":"2024-04-16T16:18:15.290639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.309798+0000","last_change":"2024-04-16T16:18:15.306390+0000","last_active":"2024-04-16T16:18:15.309798+0000","last_peered":"2024-04-16T16:18:15.309798+0000","last_clean":"2024-04-16T16:18:15.309798+0000","last_became_active":"2024-04-16T16:18:15.306063+0000","last_became_peered":"2024-04-16T16:18:15.306063+0000","last_unstale":"2024-04-16T16:18:15.309798+0000","last_undegraded":"2024-04-16T16:18:15.309798+0000","last_fullsized":"2024-04-16T16:18:15.309798+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:57.184445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.290280+0000","last_change":"2024-04-16T16:18:15.290280+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.290280+0000","last_undegraded":"2024-04-16T16:18:15.290280+0000","last_fullsized":"2024-04-16T16:18:15.290280+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.310987+0000","last_change":"2024-04-16T16:18:15.308221+0000","last_active":"2024-04-16T16:18:15.310987+0000","last_peered":"2024-04-16T16:18:15.310987+0000","last_clean":"2024-04-16T16:18:15.310987+0000","last_became_active":"2024-04-16T16:18:15.307923+0000","last_became_peered":"2024-04-16T16:18:15.307923+0000","last_unstale":"2024-04-16T16:18:15.310987+0000","last_undegraded":"2024-04-16T16:18:15.310987+0000","last_fullsized":"2024-04-16T16:18:15.310987+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:42:36.176215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.280567+0000","last_change":"2024-04-16T16:18:15.280567+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.280567+0000","last_undegraded":"2024-04-16T16:18:15.280567+0000","last_fullsized":"2024-04-16T16:18:15.280567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311047+0000","last_change":"2024-04-16T16:18:15.309398+0000","last_active":"2024-04-16T16:18:15.311047+0000","last_peered":"2024-04-16T16:18:15.311047+0000","last_clean":"2024-04-16T16:18:15.311047+0000","last_became_active":"2024-04-16T16:18:15.308959+0000","last_became_peered":"2024-04-16T16:18:15.308959+0000","last_unstale":"2024-04-16T16:18:15.311047+0000","last_undegraded":"2024-04-16T16:18:15.311047+0000","last_fullsized":"2024-04-16T16:18:15.311047+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:47:27.576872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.295248+0000","last_change":"2024-04-16T16:18:15.295248+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.295248+0000","last_undegraded":"2024-04-16T16:18:15.295248+0000","last_fullsized":"2024-04-16T16:18:15.295248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311722+0000","last_change":"2024-04-16T16:18:15.309922+0000","last_active":"2024-04-16T16:18:15.311722+0000","last_peered":"2024-04-16T16:18:15.311722+0000","last_clean":"2024-04-16T16:18:15.311722+0000","last_became_active":"2024-04-16T16:18:15.308635+0000","last_became_peered":"2024-04-16T16:18:15.308635+0000","last_unstale":"2024-04-16T16:18:15.311722+0000","last_undegraded":"2024-04-16T16:18:15.311722+0000","last_fullsized":"2024-04-16T16:18:15.311722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:05:41.836526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.287330+0000","last_change":"2024-04-16T16:18:15.287330+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.287330+0000","last_undegraded":"2024-04-16T16:18:15.287330+0000","last_fullsized":"2024-04-16T16:18:15.287330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.306762+0000","last_change":"2024-04-16T16:18:15.299371+0000","last_active":"2024-04-16T16:18:15.306762+0000","last_peered":"2024-04-16T16:18:15.306762+0000","last_clean":"2024-04-16T16:18:15.306762+0000","last_became_active":"2024-04-16T16:18:15.299047+0000","last_became_peered":"2024-04-16T16:18:15.299047+0000","last_unstale":"2024-04-16T16:18:15.306762+0000","last_undegraded":"2024-04-16T16:18:15.306762+0000","last_fullsized":"2024-04-16T16:18:15.306762+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:40.620507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.280762+0000","last_change":"2024-04-16T16:18:15.280762+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.280762+0000","last_undegraded":"2024-04-16T16:18:15.280762+0000","last_fullsized":"2024-04-16T16:18:15.280762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311071+0000","last_change":"2024-04-16T16:18:15.305063+0000","last_active":"2024-04-16T16:18:15.311071+0000","last_peered":"2024-04-16T16:18:15.311071+0000","last_clean":"2024-04-16T16:18:15.311071+0000","last_became_active":"2024-04-16T16:18:15.304669+0000","last_became_peered":"2024-04-16T16:18:15.304669+0000","last_unstale":"2024-04-16T16:18:15.311071+0000","last_undegraded":"2024-04-16T16:18:15.311071+0000","last_fullsized":"2024-04-16T16:18:15.311071+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:59.303188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296848+0000","last_change":"2024-04-16T16:18:15.296848+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296848+0000","last_undegraded":"2024-04-16T16:18:15.296848+0000","last_fullsized":"2024-04-16T16:18:15.296848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.307283+0000","last_change":"2024-04-16T16:18:15.302798+0000","last_active":"2024-04-16T16:18:15.307283+0000","last_peered":"2024-04-16T16:18:15.307283+0000","last_clean":"2024-04-16T16:18:15.307283+0000","last_became_active":"2024-04-16T16:18:15.301509+0000","last_became_peered":"2024-04-16T16:18:15.301509+0000","last_unstale":"2024-04-16T16:18:15.307283+0000","last_undegraded":"2024-04-16T16:18:15.307283+0000","last_fullsized":"2024-04-16T16:18:15.307283+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:52:23.158135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289641+0000","last_change":"2024-04-16T16:18:15.289641+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289641+0000","last_undegraded":"2024-04-16T16:18:15.289641+0000","last_fullsized":"2024-04-16T16:18:15.289641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.307758+0000","last_change":"2024-04-16T16:18:15.302329+0000","last_active":"2024-04-16T16:18:15.307758+0000","last_peered":"2024-04-16T16:18:15.307758+0000","last_clean":"2024-04-16T16:18:15.307758+0000","last_became_active":"2024-04-16T16:18:15.301993+0000","last_became_peered":"2024-04-16T16:18:15.301993+0000","last_unstale":"2024-04-16T16:18:15.307758+0000","last_undegraded":"2024-04-16T16:18:15.307758+0000","last_fullsized":"2024-04-16T16:18:15.307758+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:59:22.965812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.283736+0000","last_change":"2024-04-16T16:18:15.283736+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.283736+0000","last_undegraded":"2024-04-16T16:18:15.283736+0000","last_fullsized":"2024-04-16T16:18:15.283736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.295819+0000","last_change":"2024-04-16T16:18:15.291033+0000","last_active":"2024-04-16T16:18:15.295819+0000","last_peered":"2024-04-16T16:18:15.295819+0000","last_clean":"2024-04-16T16:18:15.295819+0000","last_became_active":"2024-04-16T16:18:15.290067+0000","last_became_peered":"2024-04-16T16:18:15.290067+0000","last_unstale":"2024-04-16T16:18:15.295819+0000","last_undegraded":"2024-04-16T16:18:15.295819+0000","last_fullsized":"2024-04-16T16:18:15.295819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:01:14.407993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.298204+0000","last_change":"2024-04-16T16:18:15.298204+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.298204+0000","last_undegraded":"2024-04-16T16:18:15.298204+0000","last_fullsized":"2024-04-16T16:18:15.298204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.304458+0000","last_change":"2024-04-16T16:18:15.301900+0000","last_active":"2024-04-16T16:18:15.304458+0000","last_peered":"2024-04-16T16:18:15.304458+0000","last_clean":"2024-04-16T16:18:15.304458+0000","last_became_active":"2024-04-16T16:18:15.301268+0000","last_became_peered":"2024-04-16T16:18:15.301268+0000","last_unstale":"2024-04-16T16:18:15.304458+0000","last_undegraded":"2024-04-16T16:18:15.304458+0000","last_fullsized":"2024-04-16T16:18:15.304458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:27:47.476140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.284020+0000","last_change":"2024-04-16T16:18:15.284020+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.284020+0000","last_undegraded":"2024-04-16T16:18:15.284020+0000","last_fullsized":"2024-04-16T16:18:15.284020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311381+0000","last_change":"2024-04-16T16:18:15.308430+0000","last_active":"2024-04-16T16:18:15.311381+0000","last_peered":"2024-04-16T16:18:15.311381+0000","last_clean":"2024-04-16T16:18:15.311381+0000","last_became_active":"2024-04-16T16:18:15.308128+0000","last_became_peered":"2024-04-16T16:18:15.308128+0000","last_unstale":"2024-04-16T16:18:15.311381+0000","last_undegraded":"2024-04-16T16:18:15.311381+0000","last_fullsized":"2024-04-16T16:18:15.311381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:42.174300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293789+0000","last_change":"2024-04-16T16:18:15.293789+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293789+0000","last_undegraded":"2024-04-16T16:18:15.293789+0000","last_fullsized":"2024-04-16T16:18:15.293789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.295501+0000","last_change":"2024-04-16T16:18:15.292003+0000","last_active":"2024-04-16T16:18:15.295501+0000","last_peered":"2024-04-16T16:18:15.295501+0000","last_clean":"2024-04-16T16:18:15.295501+0000","last_became_active":"2024-04-16T16:18:15.291691+0000","last_became_peered":"2024-04-16T16:18:15.291691+0000","last_unstale":"2024-04-16T16:18:15.295501+0000","last_undegraded":"2024-04-16T16:18:15.295501+0000","last_fullsized":"2024-04-16T16:18:15.295501+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:58:51.896704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289712+0000","last_change":"2024-04-16T16:18:15.289712+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289712+0000","last_undegraded":"2024-04-16T16:18:15.289712+0000","last_fullsized":"2024-04-16T16:18:15.289712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.308009+0000","last_change":"2024-04-16T16:18:15.305818+0000","last_active":"2024-04-16T16:18:15.308009+0000","last_peered":"2024-04-16T16:18:15.308009+0000","last_clean":"2024-04-16T16:18:15.308009+0000","last_became_active":"2024-04-16T16:18:15.304636+0000","last_became_peered":"2024-04-16T16:18:15.304636+0000","last_unstale":"2024-04-16T16:18:15.308009+0000","last_undegraded":"2024-04-16T16:18:15.308009+0000","last_fullsized":"2024-04-16T16:18:15.308009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:21.912428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.283292+0000","last_change":"2024-04-16T16:18:15.283292+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.283292+0000","last_undegraded":"2024-04-16T16:18:15.283292+0000","last_fullsized":"2024-04-16T16:18:15.283292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.308128+0000","last_change":"2024-04-16T16:18:15.305858+0000","last_active":"2024-04-16T16:18:15.308128+0000","last_peered":"2024-04-16T16:18:15.308128+0000","last_clean":"2024-04-16T16:18:15.308128+0000","last_became_active":"2024-04-16T16:18:15.305250+0000","last_became_peered":"2024-04-16T16:18:15.305250+0000","last_unstale":"2024-04-16T16:18:15.308128+0000","last_undegraded":"2024-04-16T16:18:15.308128+0000","last_fullsized":"2024-04-16T16:18:15.308128+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:46:45.674751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.302671+0000","last_change":"2024-04-16T16:18:15.302671+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.302671+0000","last_undegraded":"2024-04-16T16:18:15.302671+0000","last_fullsized":"2024-04-16T16:18:15.302671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311926+0000","last_change":"2024-04-16T16:18:15.309023+0000","last_active":"2024-04-16T16:18:15.311926+0000","last_peered":"2024-04-16T16:18:15.311926+0000","last_clean":"2024-04-16T16:18:15.311926+0000","last_became_active":"2024-04-16T16:18:15.308399+0000","last_became_peered":"2024-04-16T16:18:15.308399+0000","last_unstale":"2024-04-16T16:18:15.311926+0000","last_undegraded":"2024-04-16T16:18:15.311926+0000","last_fullsized":"2024-04-16T16:18:15.311926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:27:40.938359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.288574+0000","last_change":"2024-04-16T16:18:15.288574+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.288574+0000","last_undegraded":"2024-04-16T16:18:15.288574+0000","last_fullsized":"2024-04-16T16:18:15.288574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.314507+0000","last_change":"2024-04-16T16:18:15.311086+0000","last_active":"2024-04-16T16:18:15.314507+0000","last_peered":"2024-04-16T16:18:15.314507+0000","last_clean":"2024-04-16T16:18:15.314507+0000","last_became_active":"2024-04-16T16:18:15.310594+0000","last_became_peered":"2024-04-16T16:18:15.310594+0000","last_unstale":"2024-04-16T16:18:15.314507+0000","last_undegraded":"2024-04-16T16:18:15.314507+0000","last_fullsized":"2024-04-16T16:18:15.314507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:02:52.308842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.292047+0000","last_change":"2024-04-16T16:18:15.292047+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.292047+0000","last_undegraded":"2024-04-16T16:18:15.292047+0000","last_fullsized":"2024-04-16T16:18:15.292047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.307790+0000","last_change":"2024-04-16T16:18:15.299375+0000","last_active":"2024-04-16T16:18:15.307790+0000","last_peered":"2024-04-16T16:18:15.307790+0000","last_clean":"2024-04-16T16:18:15.307790+0000","last_became_active":"2024-04-16T16:18:15.298710+0000","last_became_peered":"2024-04-16T16:18:15.298710+0000","last_unstale":"2024-04-16T16:18:15.307790+0000","last_undegraded":"2024-04-16T16:18:15.307790+0000","last_fullsized":"2024-04-16T16:18:15.307790+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:13.308144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289079+0000","last_change":"2024-04-16T16:18:15.289079+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289079+0000","last_undegraded":"2024-04-16T16:18:15.289079+0000","last_fullsized":"2024-04-16T16:18:15.289079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.303620+0000","last_change":"2024-04-16T16:18:15.298300+0000","last_active":"2024-04-16T16:18:15.303620+0000","last_peered":"2024-04-16T16:18:15.303620+0000","last_clean":"2024-04-16T16:18:15.303620+0000","last_became_active":"2024-04-16T16:18:15.297872+0000","last_became_peered":"2024-04-16T16:18:15.297872+0000","last_unstale":"2024-04-16T16:18:15.303620+0000","last_undegraded":"2024-04-16T16:18:15.303620+0000","last_fullsized":"2024-04-16T16:18:15.303620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:51:24.645998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.281611+0000","last_change":"2024-04-16T16:18:15.281611+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.281611+0000","last_undegraded":"2024-04-16T16:18:15.281611+0000","last_fullsized":"2024-04-16T16:18:15.281611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311874+0000","last_change":"2024-04-16T16:18:15.308009+0000","last_active":"2024-04-16T16:18:15.311874+0000","last_peered":"2024-04-16T16:18:15.311874+0000","last_clean":"2024-04-16T16:18:15.311874+0000","last_became_active":"2024-04-16T16:18:15.307561+0000","last_became_peered":"2024-04-16T16:18:15.307561+0000","last_unstale":"2024-04-16T16:18:15.311874+0000","last_undegraded":"2024-04-16T16:18:15.311874+0000","last_fullsized":"2024-04-16T16:18:15.311874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:51.934340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296023+0000","last_change":"2024-04-16T16:18:15.296023+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296023+0000","last_undegraded":"2024-04-16T16:18:15.296023+0000","last_fullsized":"2024-04-16T16:18:15.296023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311093+0000","last_change":"2024-04-16T16:18:15.309487+0000","last_active":"2024-04-16T16:18:15.311093+0000","last_peered":"2024-04-16T16:18:15.311093+0000","last_clean":"2024-04-16T16:18:15.311093+0000","last_became_active":"2024-04-16T16:18:15.308448+0000","last_became_peered":"2024-04-16T16:18:15.308448+0000","last_unstale":"2024-04-16T16:18:15.311093+0000","last_undegraded":"2024-04-16T16:18:15.311093+0000","last_fullsized":"2024-04-16T16:18:15.311093+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:45:40.148360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296266+0000","last_change":"2024-04-16T16:18:15.296266+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296266+0000","last_undegraded":"2024-04-16T16:18:15.296266+0000","last_fullsized":"2024-04-16T16:18:15.296266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.307177+0000","last_change":"2024-04-16T16:18:15.301975+0000","last_active":"2024-04-16T16:18:15.307177+0000","last_peered":"2024-04-16T16:18:15.307177+0000","last_clean":"2024-04-16T16:18:15.307177+0000","last_became_active":"2024-04-16T16:18:15.300433+0000","last_became_peered":"2024-04-16T16:18:15.300433+0000","last_unstale":"2024-04-16T16:18:15.307177+0000","last_undegraded":"2024-04-16T16:18:15.307177+0000","last_fullsized":"2024-04-16T16:18:15.307177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:57.659561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"74'161","reported_seq":452,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.286818+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:18:15.286818+0000","last_peered":"2024-04-16T16:18:15.286818+0000","last_clean":"2024-04-16T16:18:15.286818+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:18:15.286818+0000","last_undegraded":"2024-04-16T16:18:15.286818+0000","last_fullsized":"2024-04-16T16:18:15.286818+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":1601536,"data_stored":1062424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":478331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":478331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574909,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":478331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":478331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1007616,"data_stored":472056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1007616,"data_stored":472056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32760,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1613824,"data_stored":1068699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:18:20.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:18:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.103+0000 7fc067200640 1 Processor -- start 2024-04-16T16:18:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.103+0000 7fc067200640 1 -- start start 2024-04-16T16:18:21.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.105+0000 7fc067200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc06815e8f0 0x7fc0680cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.105+0000 7fc067200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc0680ce9a0 0x7fc0680cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.105+0000 7fc067200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc0680cf080 0x7fc068162480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.105+0000 7fc067200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc0680ce2e0 con 0x7fc0680cf080 2024-04-16T16:18:21.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.105+0000 7fc067200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc06815ecd0 con 0x7fc0680ce9a0 2024-04-16T16:18:21.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.105+0000 7fc067200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc0680cd6d0 con 0x7fc06815e8f0 2024-04-16T16:18:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.106+0000 7fc066800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc0680cf080 0x7fc068162480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.106+0000 7fc066800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc0680cf080 0x7fc068162480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57300/0 (socket says 172.21.15.45:57300) 2024-04-16T16:18:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.106+0000 7fc066800640 1 -- 172.21.15.45:0/160634762 learned_addr learned my addr 172.21.15.45:0/160634762 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.106+0000 7fc065400640 1 --2- 172.21.15.45:0/160634762 >> v2:172.21.15.103:3300/0 conn(0x7fc0680ce9a0 0x7fc0680cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.106+0000 7fc065e00640 1 --2- 172.21.15.45:0/160634762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc06815e8f0 0x7fc0680cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.106+0000 7fc066800640 1 -- 172.21.15.45:0/160634762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc06815e8f0 msgr2=0x7fc0680cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.106+0000 7fc066800640 1 --2- 172.21.15.45:0/160634762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc06815e8f0 0x7fc0680cc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.106+0000 7fc066800640 1 -- 172.21.15.45:0/160634762 >> v2:172.21.15.103:3300/0 conn(0x7fc0680ce9a0 msgr2=0x7fc0680cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.106+0000 7fc066800640 1 --2- 172.21.15.45:0/160634762 >> v2:172.21.15.103:3300/0 conn(0x7fc0680ce9a0 0x7fc0680cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.106+0000 7fc066800640 1 -- 172.21.15.45:0/160634762 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0680cd9b0 con 0x7fc0680cf080 2024-04-16T16:18:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.107+0000 7fc066800640 1 --2- 172.21.15.45:0/160634762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0680cf080 0x7fc068162480 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fc06806fe10 tx=0x7fc06002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d49292ac737fba8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:21.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.108+0000 7fc064a00640 1 -- 172.21.15.45:0/160634762 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc060035020 con 0x7fc0680cf080 2024-04-16T16:18:21.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.108+0000 7fc064a00640 1 -- 172.21.15.45:0/160634762 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc060031870 con 0x7fc0680cf080 2024-04-16T16:18:21.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.108+0000 7fc064a00640 1 -- 172.21.15.45:0/160634762 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc060031c40 con 0x7fc0680cf080 2024-04-16T16:18:21.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.108+0000 7fc067200640 1 -- 172.21.15.45:0/160634762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0680cf080 msgr2=0x7fc068162480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:21.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.108+0000 7fc067200640 1 --2- 172.21.15.45:0/160634762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0680cf080 0x7fc068162480 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fc06806fe10 tx=0x7fc06002b910 comp rx=0 tx=0).stop 2024-04-16T16:18:21.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.109+0000 7fc067200640 1 -- 172.21.15.45:0/160634762 shutdown_connections 2024-04-16T16:18:21.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.109+0000 7fc067200640 1 --2- 172.21.15.45:0/160634762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0680cf080 0x7fc068162480 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.109+0000 7fc067200640 1 --2- 172.21.15.45:0/160634762 >> v2:172.21.15.103:3300/0 conn(0x7fc0680ce9a0 0x7fc0680cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.109+0000 7fc067200640 1 --2- 172.21.15.45:0/160634762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc06815e8f0 0x7fc0680cc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.109+0000 7fc067200640 1 -- 172.21.15.45:0/160634762 >> 172.21.15.45:0/160634762 conn(0x7fc06806b930 msgr2=0x7fc06806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:21.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.109+0000 7fc067200640 1 -- 172.21.15.45:0/160634762 shutdown_connections 2024-04-16T16:18:21.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.109+0000 7fc067200640 1 -- 172.21.15.45:0/160634762 wait complete. 2024-04-16T16:18:21.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.110+0000 7fc067200640 1 Processor -- start 2024-04-16T16:18:21.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.111+0000 7fc067200640 1 -- start start 2024-04-16T16:18:21.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.111+0000 7fc067200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0680ce9a0 0x7fc06815ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.112+0000 7fc067200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0680cf080 0x7fc06813bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.112+0000 7fc067200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc06815e8f0 0x7fc06813c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.112+0000 7fc067200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc0680cd360 con 0x7fc0680cf080 2024-04-16T16:18:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.112+0000 7fc067200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc0680cdf70 con 0x7fc06815e8f0 2024-04-16T16:18:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.112+0000 7fc067200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc0681629c0 con 0x7fc0680ce9a0 2024-04-16T16:18:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.112+0000 7fc065400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0680cf080 0x7fc06813bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.112+0000 7fc065400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0680cf080 0x7fc06813bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57312/0 (socket says 172.21.15.45:57312) 2024-04-16T16:18:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.112+0000 7fc065400640 1 -- 172.21.15.45:0/3260434177 learned_addr learned my addr 172.21.15.45:0/3260434177 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.112+0000 7fc066800640 1 --2- 172.21.15.45:0/3260434177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc06815e8f0 0x7fc06813c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.112+0000 7fc065e00640 1 --2- 172.21.15.45:0/3260434177 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0680ce9a0 0x7fc06815ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.112+0000 7fc065400640 1 -- 172.21.15.45:0/3260434177 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0680ce9a0 msgr2=0x7fc06815ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.112+0000 7fc065400640 1 --2- 172.21.15.45:0/3260434177 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0680ce9a0 0x7fc06815ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.112+0000 7fc065400640 1 -- 172.21.15.45:0/3260434177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc06815e8f0 msgr2=0x7fc06813c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.112+0000 7fc065400640 1 --2- 172.21.15.45:0/3260434177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc06815e8f0 0x7fc06813c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.112+0000 7fc065400640 1 -- 172.21.15.45:0/3260434177 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06814e250 con 0x7fc0680cf080 2024-04-16T16:18:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.113+0000 7fc065e00640 1 --2- 172.21.15.45:0/3260434177 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0680ce9a0 0x7fc06815ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.113+0000 7fc065400640 1 --2- 172.21.15.45:0/3260434177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0680cf080 0x7fc06813bcf0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fc05000c7f0 tx=0x7fc05000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.113+0000 7fc05e000640 1 -- 172.21.15.45:0/3260434177 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc050015070 con 0x7fc0680cf080 2024-04-16T16:18:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.113+0000 7fc05e000640 1 -- 172.21.15.45:0/3260434177 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc050004240 con 0x7fc0680cf080 2024-04-16T16:18:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.113+0000 7fc05e000640 1 -- 172.21.15.45:0/3260434177 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc050007620 con 0x7fc0680cf080 2024-04-16T16:18:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.113+0000 7fc067200640 1 -- 172.21.15.45:0/3260434177 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc06814e530 con 0x7fc0680cf080 2024-04-16T16:18:21.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.113+0000 7fc067200640 1 -- 172.21.15.45:0/3260434177 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc06814eaf0 con 0x7fc0680cf080 2024-04-16T16:18:21.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.114+0000 7fc067200640 1 -- 172.21.15.45:0/3260434177 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc068075780 con 0x7fc0680cf080 2024-04-16T16:18:21.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.116+0000 7fc05e000640 1 -- 172.21.15.45:0/3260434177 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7fc0500079d0 con 0x7fc0680cf080 2024-04-16T16:18:21.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.116+0000 7fc05e000640 1 --2- 172.21.15.45:0/3260434177 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc04406ca10 0x7fc04406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.117+0000 7fc05e000640 1 -- 172.21.15.45:0/3260434177 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7fc05008c330 con 0x7fc0680cf080 2024-04-16T16:18:21.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.117+0000 7fc065e00640 1 --2- 172.21.15.45:0/3260434177 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc04406ca10 0x7fc04406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:21.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.118+0000 7fc065e00640 1 --2- 172.21.15.45:0/3260434177 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc04406ca10 0x7fc04406eed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc06814d7f0 tx=0x7fc0580073a0 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:21.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.121+0000 7fc05e000640 1 -- 172.21.15.45:0/3260434177 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc05005a4a0 con 0x7fc0680cf080 2024-04-16T16:18:21.231 INFO:tasks.ceph.mds.a.smithi045.stderr:2024-04-16T16:18:21.232+0000 7f6a7d400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:18:21.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.323+0000 7fc067200640 1 -- 172.21.15.45:0/3260434177 --> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc06806e330 con 0x7fc04406ca10 2024-04-16T16:18:21.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.344+0000 7fc05e000640 1 -- 172.21.15.45:0/3260434177 <== mgr.5791 v2:172.21.15.103:6832/3728775368 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+248113 (secure 0 0 0) 0x7fc06806e330 con 0x7fc04406ca10 2024-04-16T16:18:21.346 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:21.347 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:18:21.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.349+0000 7fc067200640 1 -- 172.21.15.45:0/3260434177 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc04406ca10 msgr2=0x7fc04406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:21.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.349+0000 7fc067200640 1 --2- 172.21.15.45:0/3260434177 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc04406ca10 0x7fc04406eed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc06814d7f0 tx=0x7fc0580073a0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.349+0000 7fc067200640 1 -- 172.21.15.45:0/3260434177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0680cf080 msgr2=0x7fc06813bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:21.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.349+0000 7fc067200640 1 --2- 172.21.15.45:0/3260434177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0680cf080 0x7fc06813bcf0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fc05000c7f0 tx=0x7fc05000ccb0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.349+0000 7fc067200640 1 -- 172.21.15.45:0/3260434177 shutdown_connections 2024-04-16T16:18:21.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.349+0000 7fc067200640 1 --2- 172.21.15.45:0/3260434177 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fc04406ca10 0x7fc04406eed0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.349+0000 7fc067200640 1 --2- 172.21.15.45:0/3260434177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc06815e8f0 0x7fc06813c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.349+0000 7fc067200640 1 --2- 172.21.15.45:0/3260434177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0680cf080 0x7fc06813bcf0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.349+0000 7fc067200640 1 --2- 172.21.15.45:0/3260434177 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0680ce9a0 0x7fc06815ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.349+0000 7fc067200640 1 -- 172.21.15.45:0/3260434177 >> 172.21.15.45:0/3260434177 conn(0x7fc06806b930 msgr2=0x7fc06806e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:21.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.349+0000 7fc067200640 1 -- 172.21.15.45:0/3260434177 shutdown_connections 2024-04-16T16:18:21.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.349+0000 7fc067200640 1 -- 172.21.15.45:0/3260434177 wait complete. 2024-04-16T16:18:21.367 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-16T16:18:21.119366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":261,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258652,"kb_used_data":9196,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716068,"statfs":{"total":773094113280,"available":772829253632,"internally_reserved":0,"allocated":9416704,"data_stored":5036287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002211"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.298782+0000","last_change":"2024-04-16T16:18:15.298782+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.298782+0000","last_undegraded":"2024-04-16T16:18:15.298782+0000","last_fullsized":"2024-04-16T16:18:15.298782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293043+0000","last_change":"2024-04-16T16:18:15.293043+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293043+0000","last_undegraded":"2024-04-16T16:18:15.293043+0000","last_fullsized":"2024-04-16T16:18:15.293043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.284139+0000","last_change":"2024-04-16T16:18:15.284139+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.284139+0000","last_undegraded":"2024-04-16T16:18:15.284139+0000","last_fullsized":"2024-04-16T16:18:15.284139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312806+0000","last_change":"2024-04-16T16:18:16.316835+0000","last_active":"2024-04-16T16:18:17.312806+0000","last_peered":"2024-04-16T16:18:17.312806+0000","last_clean":"2024-04-16T16:18:17.312806+0000","last_became_active":"2024-04-16T16:18:16.309649+0000","last_became_peered":"2024-04-16T16:18:16.309649+0000","last_unstale":"2024-04-16T16:18:17.312806+0000","last_undegraded":"2024-04-16T16:18:17.312806+0000","last_fullsized":"2024-04-16T16:18:17.312806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:21:05.656211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289497+0000","last_change":"2024-04-16T16:18:15.289497+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289497+0000","last_undegraded":"2024-04-16T16:18:15.289497+0000","last_fullsized":"2024-04-16T16:18:15.289497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.292510+0000","last_change":"2024-04-16T16:18:15.292510+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.292510+0000","last_undegraded":"2024-04-16T16:18:15.292510+0000","last_fullsized":"2024-04-16T16:18:15.292510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312314+0000","last_change":"2024-04-16T16:18:16.308558+0000","last_active":"2024-04-16T16:18:17.312314+0000","last_peered":"2024-04-16T16:18:17.312314+0000","last_clean":"2024-04-16T16:18:17.312314+0000","last_became_active":"2024-04-16T16:18:16.308268+0000","last_became_peered":"2024-04-16T16:18:16.308268+0000","last_unstale":"2024-04-16T16:18:17.312314+0000","last_undegraded":"2024-04-16T16:18:17.312314+0000","last_fullsized":"2024-04-16T16:18:17.312314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:13:57.589683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.291320+0000","last_change":"2024-04-16T16:18:15.291320+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.291320+0000","last_undegraded":"2024-04-16T16:18:15.291320+0000","last_fullsized":"2024-04-16T16:18:15.291320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.297496+0000","last_change":"2024-04-16T16:18:15.297496+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.297496+0000","last_undegraded":"2024-04-16T16:18:15.297496+0000","last_fullsized":"2024-04-16T16:18:15.297496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.295832+0000","last_change":"2024-04-16T16:18:15.295832+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.295832+0000","last_undegraded":"2024-04-16T16:18:15.295832+0000","last_fullsized":"2024-04-16T16:18:15.295832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.291161+0000","last_change":"2024-04-16T16:18:15.291161+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.291161+0000","last_undegraded":"2024-04-16T16:18:15.291161+0000","last_fullsized":"2024-04-16T16:18:15.291161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.304631+0000","last_change":"2024-04-16T16:18:15.304631+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.304631+0000","last_undegraded":"2024-04-16T16:18:15.304631+0000","last_fullsized":"2024-04-16T16:18:15.304631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.281906+0000","last_change":"2024-04-16T16:18:15.281906+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.281906+0000","last_undegraded":"2024-04-16T16:18:15.281906+0000","last_fullsized":"2024-04-16T16:18:15.281906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.313189+0000","last_change":"2024-04-16T16:18:16.321879+0000","last_active":"2024-04-16T16:18:17.313189+0000","last_peered":"2024-04-16T16:18:17.313189+0000","last_clean":"2024-04-16T16:18:17.313189+0000","last_became_active":"2024-04-16T16:18:16.310779+0000","last_became_peered":"2024-04-16T16:18:16.310779+0000","last_unstale":"2024-04-16T16:18:17.313189+0000","last_undegraded":"2024-04-16T16:18:17.313189+0000","last_fullsized":"2024-04-16T16:18:17.313189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:46:46.691832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394765+0000","last_change":"2024-04-16T16:18:16.314327+0000","last_active":"2024-04-16T16:18:17.394765+0000","last_peered":"2024-04-16T16:18:17.394765+0000","last_clean":"2024-04-16T16:18:17.394765+0000","last_became_active":"2024-04-16T16:18:16.313983+0000","last_became_peered":"2024-04-16T16:18:16.313983+0000","last_unstale":"2024-04-16T16:18:17.394765+0000","last_undegraded":"2024-04-16T16:18:17.394765+0000","last_fullsized":"2024-04-16T16:18:17.394765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:06:33.666684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.287619+0000","last_change":"2024-04-16T16:18:15.287619+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.287619+0000","last_undegraded":"2024-04-16T16:18:15.287619+0000","last_fullsized":"2024-04-16T16:18:15.287619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.290484+0000","last_change":"2024-04-16T16:18:15.290484+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.290484+0000","last_undegraded":"2024-04-16T16:18:15.290484+0000","last_fullsized":"2024-04-16T16:18:15.290484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312691+0000","last_change":"2024-04-16T16:18:16.314027+0000","last_active":"2024-04-16T16:18:17.312691+0000","last_peered":"2024-04-16T16:18:17.312691+0000","last_clean":"2024-04-16T16:18:17.312691+0000","last_became_active":"2024-04-16T16:18:16.313426+0000","last_became_peered":"2024-04-16T16:18:16.313426+0000","last_unstale":"2024-04-16T16:18:17.312691+0000","last_undegraded":"2024-04-16T16:18:17.312691+0000","last_fullsized":"2024-04-16T16:18:17.312691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:30:17.865648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444884+0000","last_change":"2024-04-16T16:18:16.305406+0000","last_active":"2024-04-16T16:18:17.444884+0000","last_peered":"2024-04-16T16:18:17.444884+0000","last_clean":"2024-04-16T16:18:17.444884+0000","last_became_active":"2024-04-16T16:18:16.305023+0000","last_became_peered":"2024-04-16T16:18:16.305023+0000","last_unstale":"2024-04-16T16:18:17.444884+0000","last_undegraded":"2024-04-16T16:18:17.444884+0000","last_fullsized":"2024-04-16T16:18:17.444884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:20.251382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312975+0000","last_change":"2024-04-16T16:18:16.317943+0000","last_active":"2024-04-16T16:18:17.312975+0000","last_peered":"2024-04-16T16:18:17.312975+0000","last_clean":"2024-04-16T16:18:17.312975+0000","last_became_active":"2024-04-16T16:18:16.317474+0000","last_became_peered":"2024-04-16T16:18:16.317474+0000","last_unstale":"2024-04-16T16:18:17.312975+0000","last_undegraded":"2024-04-16T16:18:17.312975+0000","last_fullsized":"2024-04-16T16:18:17.312975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:52:31.379753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312792+0000","last_change":"2024-04-16T16:18:16.300164+0000","last_active":"2024-04-16T16:18:17.312792+0000","last_peered":"2024-04-16T16:18:17.312792+0000","last_clean":"2024-04-16T16:18:17.312792+0000","last_became_active":"2024-04-16T16:18:16.299838+0000","last_became_peered":"2024-04-16T16:18:16.299838+0000","last_unstale":"2024-04-16T16:18:17.312792+0000","last_undegraded":"2024-04-16T16:18:17.312792+0000","last_fullsized":"2024-04-16T16:18:17.312792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:22.876310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296503+0000","last_change":"2024-04-16T16:18:15.296503+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296503+0000","last_undegraded":"2024-04-16T16:18:15.296503+0000","last_fullsized":"2024-04-16T16:18:15.296503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296468+0000","last_change":"2024-04-16T16:18:15.296468+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296468+0000","last_undegraded":"2024-04-16T16:18:15.296468+0000","last_fullsized":"2024-04-16T16:18:15.296468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312382+0000","last_change":"2024-04-16T16:18:16.315372+0000","last_active":"2024-04-16T16:18:17.312382+0000","last_peered":"2024-04-16T16:18:17.312382+0000","last_clean":"2024-04-16T16:18:17.312382+0000","last_became_active":"2024-04-16T16:18:16.315098+0000","last_became_peered":"2024-04-16T16:18:16.315098+0000","last_unstale":"2024-04-16T16:18:17.312382+0000","last_undegraded":"2024-04-16T16:18:17.312382+0000","last_fullsized":"2024-04-16T16:18:17.312382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:22.929509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293145+0000","last_change":"2024-04-16T16:18:15.293145+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293145+0000","last_undegraded":"2024-04-16T16:18:15.293145+0000","last_fullsized":"2024-04-16T16:18:15.293145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.282088+0000","last_change":"2024-04-16T16:18:15.282088+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.282088+0000","last_undegraded":"2024-04-16T16:18:15.282088+0000","last_fullsized":"2024-04-16T16:18:15.282088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.292185+0000","last_change":"2024-04-16T16:18:15.292185+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.292185+0000","last_undegraded":"2024-04-16T16:18:15.292185+0000","last_fullsized":"2024-04-16T16:18:15.292185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.304185+0000","last_change":"2024-04-16T16:18:15.304185+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.304185+0000","last_undegraded":"2024-04-16T16:18:15.304185+0000","last_fullsized":"2024-04-16T16:18:15.304185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289755+0000","last_change":"2024-04-16T16:18:15.289755+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289755+0000","last_undegraded":"2024-04-16T16:18:15.289755+0000","last_fullsized":"2024-04-16T16:18:15.289755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296039+0000","last_change":"2024-04-16T16:18:15.296039+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296039+0000","last_undegraded":"2024-04-16T16:18:15.296039+0000","last_fullsized":"2024-04-16T16:18:15.296039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.287029+0000","last_change":"2024-04-16T16:18:15.287029+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.287029+0000","last_undegraded":"2024-04-16T16:18:15.287029+0000","last_fullsized":"2024-04-16T16:18:15.287029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.302832+0000","last_change":"2024-04-16T16:18:15.302832+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.302832+0000","last_undegraded":"2024-04-16T16:18:15.302832+0000","last_fullsized":"2024-04-16T16:18:15.302832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.291173+0000","last_change":"2024-04-16T16:18:15.291173+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.291173+0000","last_undegraded":"2024-04-16T16:18:15.291173+0000","last_fullsized":"2024-04-16T16:18:15.291173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311763+0000","last_change":"2024-04-16T16:18:15.307894+0000","last_active":"2024-04-16T16:18:15.311763+0000","last_peered":"2024-04-16T16:18:15.311763+0000","last_clean":"2024-04-16T16:18:15.311763+0000","last_became_active":"2024-04-16T16:18:15.307584+0000","last_became_peered":"2024-04-16T16:18:15.307584+0000","last_unstale":"2024-04-16T16:18:15.311763+0000","last_undegraded":"2024-04-16T16:18:15.311763+0000","last_fullsized":"2024-04-16T16:18:15.311763+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:30:36.737096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.297298+0000","last_change":"2024-04-16T16:18:15.297298+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.297298+0000","last_undegraded":"2024-04-16T16:18:15.297298+0000","last_fullsized":"2024-04-16T16:18:15.297298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.303705+0000","last_change":"2024-04-16T16:18:15.299901+0000","last_active":"2024-04-16T16:18:15.303705+0000","last_peered":"2024-04-16T16:18:15.303705+0000","last_clean":"2024-04-16T16:18:15.303705+0000","last_became_active":"2024-04-16T16:18:15.299622+0000","last_became_peered":"2024-04-16T16:18:15.299622+0000","last_unstale":"2024-04-16T16:18:15.303705+0000","last_undegraded":"2024-04-16T16:18:15.303705+0000","last_fullsized":"2024-04-16T16:18:15.303705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:51.934340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.303193+0000","last_change":"2024-04-16T16:18:15.303193+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.303193+0000","last_undegraded":"2024-04-16T16:18:15.303193+0000","last_fullsized":"2024-04-16T16:18:15.303193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444710+0000","last_change":"2024-04-16T16:18:15.297918+0000","last_active":"2024-04-16T16:18:17.444710+0000","last_peered":"2024-04-16T16:18:17.444710+0000","last_clean":"2024-04-16T16:18:17.444710+0000","last_became_active":"2024-04-16T16:18:15.297217+0000","last_became_peered":"2024-04-16T16:18:15.297217+0000","last_unstale":"2024-04-16T16:18:17.444710+0000","last_undegraded":"2024-04-16T16:18:17.444710+0000","last_fullsized":"2024-04-16T16:18:17.444710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:03:48.538913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.283014+0000","last_change":"2024-04-16T16:18:15.283014+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.283014+0000","last_undegraded":"2024-04-16T16:18:15.283014+0000","last_fullsized":"2024-04-16T16:18:15.283014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443826+0000","last_change":"2024-04-16T16:18:15.300207+0000","last_active":"2024-04-16T16:18:17.443826+0000","last_peered":"2024-04-16T16:18:17.443826+0000","last_clean":"2024-04-16T16:18:17.443826+0000","last_became_active":"2024-04-16T16:18:15.297220+0000","last_became_peered":"2024-04-16T16:18:15.297220+0000","last_unstale":"2024-04-16T16:18:17.443826+0000","last_undegraded":"2024-04-16T16:18:17.443826+0000","last_fullsized":"2024-04-16T16:18:17.443826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:58:34.162900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.295251+0000","last_change":"2024-04-16T16:18:15.295251+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.295251+0000","last_undegraded":"2024-04-16T16:18:15.295251+0000","last_fullsized":"2024-04-16T16:18:15.295251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311825+0000","last_change":"2024-04-16T16:18:15.309872+0000","last_active":"2024-04-16T16:18:15.311825+0000","last_peered":"2024-04-16T16:18:15.311825+0000","last_clean":"2024-04-16T16:18:15.311825+0000","last_became_active":"2024-04-16T16:18:15.309546+0000","last_became_peered":"2024-04-16T16:18:15.309546+0000","last_unstale":"2024-04-16T16:18:15.311825+0000","last_undegraded":"2024-04-16T16:18:15.311825+0000","last_fullsized":"2024-04-16T16:18:15.311825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:27:40.938359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.288554+0000","last_change":"2024-04-16T16:18:15.288554+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.288554+0000","last_undegraded":"2024-04-16T16:18:15.288554+0000","last_fullsized":"2024-04-16T16:18:15.288554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.303669+0000","last_change":"2024-04-16T16:18:15.298912+0000","last_active":"2024-04-16T16:18:15.303669+0000","last_peered":"2024-04-16T16:18:15.303669+0000","last_clean":"2024-04-16T16:18:15.303669+0000","last_became_active":"2024-04-16T16:18:15.298638+0000","last_became_peered":"2024-04-16T16:18:15.298638+0000","last_unstale":"2024-04-16T16:18:15.303669+0000","last_undegraded":"2024-04-16T16:18:15.303669+0000","last_fullsized":"2024-04-16T16:18:15.303669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:30:36.737096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.282766+0000","last_change":"2024-04-16T16:18:15.282766+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.282766+0000","last_undegraded":"2024-04-16T16:18:15.282766+0000","last_fullsized":"2024-04-16T16:18:15.282766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444198+0000","last_change":"2024-04-16T16:18:15.301897+0000","last_active":"2024-04-16T16:18:17.444198+0000","last_peered":"2024-04-16T16:18:17.444198+0000","last_clean":"2024-04-16T16:18:17.444198+0000","last_became_active":"2024-04-16T16:18:15.301549+0000","last_became_peered":"2024-04-16T16:18:15.301549+0000","last_unstale":"2024-04-16T16:18:17.444198+0000","last_undegraded":"2024-04-16T16:18:17.444198+0000","last_fullsized":"2024-04-16T16:18:17.444198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:29:16.985875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394783+0000","last_change":"2024-04-16T16:18:16.308198+0000","last_active":"2024-04-16T16:18:17.394783+0000","last_peered":"2024-04-16T16:18:17.394783+0000","last_clean":"2024-04-16T16:18:17.394783+0000","last_became_active":"2024-04-16T16:18:16.307924+0000","last_became_peered":"2024-04-16T16:18:16.307924+0000","last_unstale":"2024-04-16T16:18:17.394783+0000","last_undegraded":"2024-04-16T16:18:17.394783+0000","last_fullsized":"2024-04-16T16:18:17.394783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:34:10.200444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.307302+0000","last_change":"2024-04-16T16:18:15.303661+0000","last_active":"2024-04-16T16:18:15.307302+0000","last_peered":"2024-04-16T16:18:15.307302+0000","last_clean":"2024-04-16T16:18:15.307302+0000","last_became_active":"2024-04-16T16:18:15.300455+0000","last_became_peered":"2024-04-16T16:18:15.300455+0000","last_unstale":"2024-04-16T16:18:15.307302+0000","last_undegraded":"2024-04-16T16:18:15.307302+0000","last_fullsized":"2024-04-16T16:18:15.307302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:21:04.639131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394997+0000","last_change":"2024-04-16T16:18:16.324165+0000","last_active":"2024-04-16T16:18:17.394997+0000","last_peered":"2024-04-16T16:18:17.394997+0000","last_clean":"2024-04-16T16:18:17.394997+0000","last_became_active":"2024-04-16T16:18:16.323865+0000","last_became_peered":"2024-04-16T16:18:16.323865+0000","last_unstale":"2024-04-16T16:18:17.394997+0000","last_undegraded":"2024-04-16T16:18:17.394997+0000","last_fullsized":"2024-04-16T16:18:17.394997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:51:43.715542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311275+0000","last_change":"2024-04-16T16:18:15.305839+0000","last_active":"2024-04-16T16:18:15.311275+0000","last_peered":"2024-04-16T16:18:15.311275+0000","last_clean":"2024-04-16T16:18:15.311275+0000","last_became_active":"2024-04-16T16:18:15.305453+0000","last_became_peered":"2024-04-16T16:18:15.305453+0000","last_unstale":"2024-04-16T16:18:15.311275+0000","last_undegraded":"2024-04-16T16:18:15.311275+0000","last_fullsized":"2024-04-16T16:18:15.311275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:51:24.645998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293391+0000","last_change":"2024-04-16T16:18:15.293391+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293391+0000","last_undegraded":"2024-04-16T16:18:15.293391+0000","last_fullsized":"2024-04-16T16:18:15.293391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311136+0000","last_change":"2024-04-16T16:18:15.304936+0000","last_active":"2024-04-16T16:18:15.311136+0000","last_peered":"2024-04-16T16:18:15.311136+0000","last_clean":"2024-04-16T16:18:15.311136+0000","last_became_active":"2024-04-16T16:18:15.304590+0000","last_became_peered":"2024-04-16T16:18:15.304590+0000","last_unstale":"2024-04-16T16:18:15.311136+0000","last_undegraded":"2024-04-16T16:18:15.311136+0000","last_fullsized":"2024-04-16T16:18:15.311136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:33.215073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443924+0000","last_change":"2024-04-16T16:18:16.316337+0000","last_active":"2024-04-16T16:18:17.443924+0000","last_peered":"2024-04-16T16:18:17.443924+0000","last_clean":"2024-04-16T16:18:17.443924+0000","last_became_active":"2024-04-16T16:18:16.316019+0000","last_became_peered":"2024-04-16T16:18:16.316019+0000","last_unstale":"2024-04-16T16:18:17.443924+0000","last_undegraded":"2024-04-16T16:18:17.443924+0000","last_fullsized":"2024-04-16T16:18:17.443924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:21:33.844543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311896+0000","last_change":"2024-04-16T16:18:15.308173+0000","last_active":"2024-04-16T16:18:15.311896+0000","last_peered":"2024-04-16T16:18:15.311896+0000","last_clean":"2024-04-16T16:18:15.311896+0000","last_became_active":"2024-04-16T16:18:15.307837+0000","last_became_peered":"2024-04-16T16:18:15.307837+0000","last_unstale":"2024-04-16T16:18:15.311896+0000","last_undegraded":"2024-04-16T16:18:15.311896+0000","last_fullsized":"2024-04-16T16:18:15.311896+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:45:40.148360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.290639+0000","last_change":"2024-04-16T16:18:15.290639+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.290639+0000","last_undegraded":"2024-04-16T16:18:15.290639+0000","last_fullsized":"2024-04-16T16:18:15.290639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.309798+0000","last_change":"2024-04-16T16:18:15.306390+0000","last_active":"2024-04-16T16:18:15.309798+0000","last_peered":"2024-04-16T16:18:15.309798+0000","last_clean":"2024-04-16T16:18:15.309798+0000","last_became_active":"2024-04-16T16:18:15.306063+0000","last_became_peered":"2024-04-16T16:18:15.306063+0000","last_unstale":"2024-04-16T16:18:15.309798+0000","last_undegraded":"2024-04-16T16:18:15.309798+0000","last_fullsized":"2024-04-16T16:18:15.309798+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:57.184445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443652+0000","last_change":"2024-04-16T16:18:16.315534+0000","last_active":"2024-04-16T16:18:17.443652+0000","last_peered":"2024-04-16T16:18:17.443652+0000","last_clean":"2024-04-16T16:18:17.443652+0000","last_became_active":"2024-04-16T16:18:16.315301+0000","last_became_peered":"2024-04-16T16:18:16.315301+0000","last_unstale":"2024-04-16T16:18:17.443652+0000","last_undegraded":"2024-04-16T16:18:17.443652+0000","last_fullsized":"2024-04-16T16:18:17.443652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:42.181514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.310987+0000","last_change":"2024-04-16T16:18:15.308221+0000","last_active":"2024-04-16T16:18:15.310987+0000","last_peered":"2024-04-16T16:18:15.310987+0000","last_clean":"2024-04-16T16:18:15.310987+0000","last_became_active":"2024-04-16T16:18:15.307923+0000","last_became_peered":"2024-04-16T16:18:15.307923+0000","last_unstale":"2024-04-16T16:18:15.310987+0000","last_undegraded":"2024-04-16T16:18:15.310987+0000","last_fullsized":"2024-04-16T16:18:15.310987+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:42:36.176215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.280567+0000","last_change":"2024-04-16T16:18:15.280567+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.280567+0000","last_undegraded":"2024-04-16T16:18:15.280567+0000","last_fullsized":"2024-04-16T16:18:15.280567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.395004+0000","last_change":"2024-04-16T16:18:15.309398+0000","last_active":"2024-04-16T16:18:17.395004+0000","last_peered":"2024-04-16T16:18:17.395004+0000","last_clean":"2024-04-16T16:18:17.395004+0000","last_became_active":"2024-04-16T16:18:15.308959+0000","last_became_peered":"2024-04-16T16:18:15.308959+0000","last_unstale":"2024-04-16T16:18:17.395004+0000","last_undegraded":"2024-04-16T16:18:17.395004+0000","last_fullsized":"2024-04-16T16:18:17.395004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:10:27.168721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.295248+0000","last_change":"2024-04-16T16:18:15.295248+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.295248+0000","last_undegraded":"2024-04-16T16:18:15.295248+0000","last_fullsized":"2024-04-16T16:18:15.295248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311722+0000","last_change":"2024-04-16T16:18:15.309922+0000","last_active":"2024-04-16T16:18:15.311722+0000","last_peered":"2024-04-16T16:18:15.311722+0000","last_clean":"2024-04-16T16:18:15.311722+0000","last_became_active":"2024-04-16T16:18:15.308635+0000","last_became_peered":"2024-04-16T16:18:15.308635+0000","last_unstale":"2024-04-16T16:18:15.311722+0000","last_undegraded":"2024-04-16T16:18:15.311722+0000","last_fullsized":"2024-04-16T16:18:15.311722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:05:41.836526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.287330+0000","last_change":"2024-04-16T16:18:15.287330+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.287330+0000","last_undegraded":"2024-04-16T16:18:15.287330+0000","last_fullsized":"2024-04-16T16:18:15.287330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"84'8","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443410+0000","last_change":"2024-04-16T16:18:15.299371+0000","last_active":"2024-04-16T16:18:17.443410+0000","last_peered":"2024-04-16T16:18:17.443410+0000","last_clean":"2024-04-16T16:18:17.443410+0000","last_became_active":"2024-04-16T16:18:15.299047+0000","last_became_peered":"2024-04-16T16:18:15.299047+0000","last_unstale":"2024-04-16T16:18:17.443410+0000","last_undegraded":"2024-04-16T16:18:17.443410+0000","last_fullsized":"2024-04-16T16:18:17.443410+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:14:18.404852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.280762+0000","last_change":"2024-04-16T16:18:15.280762+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.280762+0000","last_undegraded":"2024-04-16T16:18:15.280762+0000","last_fullsized":"2024-04-16T16:18:15.280762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311071+0000","last_change":"2024-04-16T16:18:15.305063+0000","last_active":"2024-04-16T16:18:15.311071+0000","last_peered":"2024-04-16T16:18:15.311071+0000","last_clean":"2024-04-16T16:18:15.311071+0000","last_became_active":"2024-04-16T16:18:15.304669+0000","last_became_peered":"2024-04-16T16:18:15.304669+0000","last_unstale":"2024-04-16T16:18:15.311071+0000","last_undegraded":"2024-04-16T16:18:15.311071+0000","last_fullsized":"2024-04-16T16:18:15.311071+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:59.303188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296848+0000","last_change":"2024-04-16T16:18:15.296848+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296848+0000","last_undegraded":"2024-04-16T16:18:15.296848+0000","last_fullsized":"2024-04-16T16:18:15.296848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443949+0000","last_change":"2024-04-16T16:18:15.302798+0000","last_active":"2024-04-16T16:18:17.443949+0000","last_peered":"2024-04-16T16:18:17.443949+0000","last_clean":"2024-04-16T16:18:17.443949+0000","last_became_active":"2024-04-16T16:18:15.301509+0000","last_became_peered":"2024-04-16T16:18:15.301509+0000","last_unstale":"2024-04-16T16:18:17.443949+0000","last_undegraded":"2024-04-16T16:18:17.443949+0000","last_fullsized":"2024-04-16T16:18:17.443949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:20:56.511072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289641+0000","last_change":"2024-04-16T16:18:15.289641+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289641+0000","last_undegraded":"2024-04-16T16:18:15.289641+0000","last_fullsized":"2024-04-16T16:18:15.289641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443959+0000","last_change":"2024-04-16T16:18:15.302329+0000","last_active":"2024-04-16T16:18:17.443959+0000","last_peered":"2024-04-16T16:18:17.443959+0000","last_clean":"2024-04-16T16:18:17.443959+0000","last_became_active":"2024-04-16T16:18:15.301993+0000","last_became_peered":"2024-04-16T16:18:15.301993+0000","last_unstale":"2024-04-16T16:18:17.443959+0000","last_undegraded":"2024-04-16T16:18:17.443959+0000","last_fullsized":"2024-04-16T16:18:17.443959+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:00:22.780800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.313248+0000","last_change":"2024-04-16T16:18:16.318257+0000","last_active":"2024-04-16T16:18:17.313248+0000","last_peered":"2024-04-16T16:18:17.313248+0000","last_clean":"2024-04-16T16:18:17.313248+0000","last_became_active":"2024-04-16T16:18:16.311533+0000","last_became_peered":"2024-04-16T16:18:16.311533+0000","last_unstale":"2024-04-16T16:18:17.313248+0000","last_undegraded":"2024-04-16T16:18:17.313248+0000","last_fullsized":"2024-04-16T16:18:17.313248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:42:37.193295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394822+0000","last_change":"2024-04-16T16:18:15.291033+0000","last_active":"2024-04-16T16:18:17.394822+0000","last_peered":"2024-04-16T16:18:17.394822+0000","last_clean":"2024-04-16T16:18:17.394822+0000","last_became_active":"2024-04-16T16:18:15.290067+0000","last_became_peered":"2024-04-16T16:18:15.290067+0000","last_unstale":"2024-04-16T16:18:17.394822+0000","last_undegraded":"2024-04-16T16:18:17.394822+0000","last_fullsized":"2024-04-16T16:18:17.394822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:06:23.738648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.298204+0000","last_change":"2024-04-16T16:18:15.298204+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.298204+0000","last_undegraded":"2024-04-16T16:18:15.298204+0000","last_fullsized":"2024-04-16T16:18:15.298204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.313304+0000","last_change":"2024-04-16T16:18:15.301900+0000","last_active":"2024-04-16T16:18:17.313304+0000","last_peered":"2024-04-16T16:18:17.313304+0000","last_clean":"2024-04-16T16:18:17.313304+0000","last_became_active":"2024-04-16T16:18:15.301268+0000","last_became_peered":"2024-04-16T16:18:15.301268+0000","last_unstale":"2024-04-16T16:18:17.313304+0000","last_undegraded":"2024-04-16T16:18:17.313304+0000","last_fullsized":"2024-04-16T16:18:17.313304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:15.061257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312362+0000","last_change":"2024-04-16T16:18:16.314030+0000","last_active":"2024-04-16T16:18:17.312362+0000","last_peered":"2024-04-16T16:18:17.312362+0000","last_clean":"2024-04-16T16:18:17.312362+0000","last_became_active":"2024-04-16T16:18:16.313572+0000","last_became_peered":"2024-04-16T16:18:16.313572+0000","last_unstale":"2024-04-16T16:18:17.312362+0000","last_undegraded":"2024-04-16T16:18:17.312362+0000","last_fullsized":"2024-04-16T16:18:17.312362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:50:11.604716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312426+0000","last_change":"2024-04-16T16:18:15.308430+0000","last_active":"2024-04-16T16:18:17.312426+0000","last_peered":"2024-04-16T16:18:17.312426+0000","last_clean":"2024-04-16T16:18:17.312426+0000","last_became_active":"2024-04-16T16:18:15.308128+0000","last_became_peered":"2024-04-16T16:18:15.308128+0000","last_unstale":"2024-04-16T16:18:17.312426+0000","last_undegraded":"2024-04-16T16:18:17.312426+0000","last_fullsized":"2024-04-16T16:18:17.312426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:23.975082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293789+0000","last_change":"2024-04-16T16:18:15.293789+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293789+0000","last_undegraded":"2024-04-16T16:18:15.293789+0000","last_fullsized":"2024-04-16T16:18:15.293789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394296+0000","last_change":"2024-04-16T16:18:15.292003+0000","last_active":"2024-04-16T16:18:17.394296+0000","last_peered":"2024-04-16T16:18:17.394296+0000","last_clean":"2024-04-16T16:18:17.394296+0000","last_became_active":"2024-04-16T16:18:15.291691+0000","last_became_peered":"2024-04-16T16:18:15.291691+0000","last_unstale":"2024-04-16T16:18:17.394296+0000","last_undegraded":"2024-04-16T16:18:17.394296+0000","last_fullsized":"2024-04-16T16:18:17.394296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:28.452108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289712+0000","last_change":"2024-04-16T16:18:15.289712+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289712+0000","last_undegraded":"2024-04-16T16:18:15.289712+0000","last_fullsized":"2024-04-16T16:18:15.289712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.308009+0000","last_change":"2024-04-16T16:18:15.305818+0000","last_active":"2024-04-16T16:18:15.308009+0000","last_peered":"2024-04-16T16:18:15.308009+0000","last_clean":"2024-04-16T16:18:15.308009+0000","last_became_active":"2024-04-16T16:18:15.304636+0000","last_became_peered":"2024-04-16T16:18:15.304636+0000","last_unstale":"2024-04-16T16:18:15.308009+0000","last_undegraded":"2024-04-16T16:18:15.308009+0000","last_fullsized":"2024-04-16T16:18:15.308009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:21.912428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.283292+0000","last_change":"2024-04-16T16:18:15.283292+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.283292+0000","last_undegraded":"2024-04-16T16:18:15.283292+0000","last_fullsized":"2024-04-16T16:18:15.283292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.308128+0000","last_change":"2024-04-16T16:18:15.305858+0000","last_active":"2024-04-16T16:18:15.308128+0000","last_peered":"2024-04-16T16:18:15.308128+0000","last_clean":"2024-04-16T16:18:15.308128+0000","last_became_active":"2024-04-16T16:18:15.305250+0000","last_became_peered":"2024-04-16T16:18:15.305250+0000","last_unstale":"2024-04-16T16:18:15.308128+0000","last_undegraded":"2024-04-16T16:18:15.308128+0000","last_fullsized":"2024-04-16T16:18:15.308128+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:46:45.674751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.302671+0000","last_change":"2024-04-16T16:18:15.302671+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.302671+0000","last_undegraded":"2024-04-16T16:18:15.302671+0000","last_fullsized":"2024-04-16T16:18:15.302671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311926+0000","last_change":"2024-04-16T16:18:15.309023+0000","last_active":"2024-04-16T16:18:15.311926+0000","last_peered":"2024-04-16T16:18:15.311926+0000","last_clean":"2024-04-16T16:18:15.311926+0000","last_became_active":"2024-04-16T16:18:15.308399+0000","last_became_peered":"2024-04-16T16:18:15.308399+0000","last_unstale":"2024-04-16T16:18:15.311926+0000","last_undegraded":"2024-04-16T16:18:15.311926+0000","last_fullsized":"2024-04-16T16:18:15.311926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:27:40.938359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.288574+0000","last_change":"2024-04-16T16:18:15.288574+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.288574+0000","last_undegraded":"2024-04-16T16:18:15.288574+0000","last_fullsized":"2024-04-16T16:18:15.288574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.314507+0000","last_change":"2024-04-16T16:18:15.311086+0000","last_active":"2024-04-16T16:18:15.314507+0000","last_peered":"2024-04-16T16:18:15.314507+0000","last_clean":"2024-04-16T16:18:15.314507+0000","last_became_active":"2024-04-16T16:18:15.310594+0000","last_became_peered":"2024-04-16T16:18:15.310594+0000","last_unstale":"2024-04-16T16:18:15.314507+0000","last_undegraded":"2024-04-16T16:18:15.314507+0000","last_fullsized":"2024-04-16T16:18:15.314507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:02:52.308842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.292047+0000","last_change":"2024-04-16T16:18:15.292047+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.292047+0000","last_undegraded":"2024-04-16T16:18:15.292047+0000","last_fullsized":"2024-04-16T16:18:15.292047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444049+0000","last_change":"2024-04-16T16:18:15.299375+0000","last_active":"2024-04-16T16:18:17.444049+0000","last_peered":"2024-04-16T16:18:17.444049+0000","last_clean":"2024-04-16T16:18:17.444049+0000","last_became_active":"2024-04-16T16:18:15.298710+0000","last_became_peered":"2024-04-16T16:18:15.298710+0000","last_unstale":"2024-04-16T16:18:17.444049+0000","last_undegraded":"2024-04-16T16:18:17.444049+0000","last_fullsized":"2024-04-16T16:18:17.444049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:12:39.298882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444137+0000","last_change":"2024-04-16T16:18:16.322248+0000","last_active":"2024-04-16T16:18:17.444137+0000","last_peered":"2024-04-16T16:18:17.444137+0000","last_clean":"2024-04-16T16:18:17.444137+0000","last_became_active":"2024-04-16T16:18:16.318615+0000","last_became_peered":"2024-04-16T16:18:16.318615+0000","last_unstale":"2024-04-16T16:18:17.444137+0000","last_undegraded":"2024-04-16T16:18:17.444137+0000","last_fullsized":"2024-04-16T16:18:17.444137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:16:57.729600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.303620+0000","last_change":"2024-04-16T16:18:15.298300+0000","last_active":"2024-04-16T16:18:15.303620+0000","last_peered":"2024-04-16T16:18:15.303620+0000","last_clean":"2024-04-16T16:18:15.303620+0000","last_became_active":"2024-04-16T16:18:15.297872+0000","last_became_peered":"2024-04-16T16:18:15.297872+0000","last_unstale":"2024-04-16T16:18:15.303620+0000","last_undegraded":"2024-04-16T16:18:15.303620+0000","last_fullsized":"2024-04-16T16:18:15.303620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:51:24.645998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.281611+0000","last_change":"2024-04-16T16:18:15.281611+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.281611+0000","last_undegraded":"2024-04-16T16:18:15.281611+0000","last_fullsized":"2024-04-16T16:18:15.281611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311874+0000","last_change":"2024-04-16T16:18:15.308009+0000","last_active":"2024-04-16T16:18:15.311874+0000","last_peered":"2024-04-16T16:18:15.311874+0000","last_clean":"2024-04-16T16:18:15.311874+0000","last_became_active":"2024-04-16T16:18:15.307561+0000","last_became_peered":"2024-04-16T16:18:15.307561+0000","last_unstale":"2024-04-16T16:18:15.311874+0000","last_undegraded":"2024-04-16T16:18:15.311874+0000","last_fullsized":"2024-04-16T16:18:15.311874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:51.934340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296023+0000","last_change":"2024-04-16T16:18:15.296023+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296023+0000","last_undegraded":"2024-04-16T16:18:15.296023+0000","last_fullsized":"2024-04-16T16:18:15.296023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311093+0000","last_change":"2024-04-16T16:18:15.309487+0000","last_active":"2024-04-16T16:18:15.311093+0000","last_peered":"2024-04-16T16:18:15.311093+0000","last_clean":"2024-04-16T16:18:15.311093+0000","last_became_active":"2024-04-16T16:18:15.308448+0000","last_became_peered":"2024-04-16T16:18:15.308448+0000","last_unstale":"2024-04-16T16:18:15.311093+0000","last_undegraded":"2024-04-16T16:18:15.311093+0000","last_fullsized":"2024-04-16T16:18:15.311093+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:45:40.148360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296266+0000","last_change":"2024-04-16T16:18:15.296266+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296266+0000","last_undegraded":"2024-04-16T16:18:15.296266+0000","last_fullsized":"2024-04-16T16:18:15.296266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.307177+0000","last_change":"2024-04-16T16:18:15.301975+0000","last_active":"2024-04-16T16:18:15.307177+0000","last_peered":"2024-04-16T16:18:15.307177+0000","last_clean":"2024-04-16T16:18:15.307177+0000","last_became_active":"2024-04-16T16:18:15.300433+0000","last_became_peered":"2024-04-16T16:18:15.300433+0000","last_unstale":"2024-04-16T16:18:15.307177+0000","last_undegraded":"2024-04-16T16:18:15.307177+0000","last_fullsized":"2024-04-16T16:18:15.307177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:57.659561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"74'161","reported_seq":452,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.286818+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:18:15.286818+0000","last_peered":"2024-04-16T16:18:15.286818+0000","last_clean":"2024-04-16T16:18:15.286818+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:18:15.286818+0000","last_undegraded":"2024-04-16T16:18:15.286818+0000","last_fullsized":"2024-04-16T16:18:15.286818+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16,"ondisk_log_size":16,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":1601536,"data_stored":1062424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":478331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":478331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574909,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":478331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339628,"statfs":{"total":96636764160,"available":96603779072,"internally_reserved":0,"allocated":1056768,"data_stored":493018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574908,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1048576,"data_stored":491988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1007616,"data_stored":472056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574909,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32788,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":1642496,"data_stored":1081808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:18:21.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:18:21.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.637+0000 7f1eae600640 1 Processor -- start 2024-04-16T16:18:21.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.638+0000 7f1eae600640 1 -- start start 2024-04-16T16:18:21.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.639+0000 7f1eae600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea815ccb0 0x7f1ea806df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.639+0000 7f1eae600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1ea806e480 0x7f1ea81608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.639+0000 7f1eae600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ea815d360 con 0x7f1ea8146220 2024-04-16T16:18:21.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.639+0000 7f1eae600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1ea815d130 con 0x7f1ea806e480 2024-04-16T16:18:21.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.639+0000 7f1eae600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea8160410 con 0x7f1ea815ccb0 2024-04-16T16:18:21.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.640+0000 7f1ead200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1ea8146220 0x7f1ea806f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59348/0 (socket says 172.21.15.45:59348) 2024-04-16T16:18:21.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.640+0000 7f1ead200640 1 -- 172.21.15.45:0/3793520497 learned_addr learned my addr 172.21.15.45:0/3793520497 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.640+0000 7f1eadc00640 1 --2- 172.21.15.45:0/3793520497 >> v2:172.21.15.103:3300/0 conn(0x7f1ea806e480 0x7f1ea81608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.640+0000 7f1e9fe00640 1 --2- 172.21.15.45:0/3793520497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea815ccb0 0x7f1ea806df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.640+0000 7f1ea7e00640 1 -- 172.21.15.45:0/3793520497 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 966122756 0 0) 0x7f1ea815d360 con 0x7f1ea8146220 2024-04-16T16:18:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.640+0000 7f1ea7e00640 1 -- 172.21.15.45:0/3793520497 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e8c003520 con 0x7f1ea8146220 2024-04-16T16:18:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ea7e00640 1 -- 172.21.15.45:0/3793520497 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2523653380 0 0) 0x7f1e8c003520 con 0x7f1ea8146220 2024-04-16T16:18:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ea7e00640 1 -- 172.21.15.45:0/3793520497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea815ccb0 msgr2=0x7f1ea806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ea7e00640 1 --2- 172.21.15.45:0/3793520497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea815ccb0 0x7f1ea806df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ea7e00640 1 -- 172.21.15.45:0/3793520497 >> v2:172.21.15.103:3300/0 conn(0x7f1ea806e480 msgr2=0x7f1ea81608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ea7e00640 1 --2- 172.21.15.45:0/3793520497 >> v2:172.21.15.103:3300/0 conn(0x7f1ea806e480 0x7f1ea81608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ea7e00640 1 -- 172.21.15.45:0/3793520497 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea815d640 con 0x7f1ea8146220 2024-04-16T16:18:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ea7e00640 1 -- 172.21.15.45:0/3793520497 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f1e980019f0 con 0x7f1ea8146220 2024-04-16T16:18:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ea7e00640 1 -- 172.21.15.45:0/3793520497 >> v1:172.21.15.45:6789/0 conn(0x7f1ea8146220 legacy=0x7f1ea806f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ea7e00640 1 --2- 172.21.15.45:0/3793520497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e8c0039b0 0x7f1e8c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ea7e00640 1 --2- 172.21.15.45:0/3793520497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea815ccb0 0x7f1e8c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ea7e00640 1 --2- 172.21.15.45:0/3793520497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8c004d40 0x7f1e8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ea7e00640 1 -- 172.21.15.45:0/3793520497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea8160410 con 0x7f1e8c0039b0 2024-04-16T16:18:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ea7e00640 1 -- 172.21.15.45:0/3793520497 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8c00b9c0 con 0x7f1e8c004d40 2024-04-16T16:18:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ea7e00640 1 -- 172.21.15.45:0/3793520497 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1e8c00bb40 con 0x7f1ea815ccb0 2024-04-16T16:18:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ead200640 1 --2- 172.21.15.45:0/3793520497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e8c0039b0 0x7f1e8c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1eadc00640 1 --2- 172.21.15.45:0/3793520497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8c004d40 0x7f1e8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ead200640 1 -- 172.21.15.45:0/3793520497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea815ccb0 msgr2=0x7f1e8c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ead200640 1 --2- 172.21.15.45:0/3793520497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea815ccb0 0x7f1e8c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.641+0000 7f1ead200640 1 -- 172.21.15.45:0/3793520497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8c004d40 msgr2=0x7f1e8c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.642+0000 7f1ead200640 1 --2- 172.21.15.45:0/3793520497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8c004d40 0x7f1e8c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.642+0000 7f1ead200640 1 -- 172.21.15.45:0/3793520497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e8c003520 con 0x7f1e8c0039b0 2024-04-16T16:18:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.642+0000 7f1eadc00640 1 --2- 172.21.15.45:0/3793520497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8c004d40 0x7f1e8c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:21.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.643+0000 7f1ead200640 1 --2- 172.21.15.45:0/3793520497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e8c0039b0 0x7f1e8c003dd0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800a870 tx=0x7f1e8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b95ef6cb2dd610d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:21.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.643+0000 7f1ea7e00640 1 -- 172.21.15.45:0/3793520497 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9800a4c0 con 0x7f1e8c0039b0 2024-04-16T16:18:21.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.643+0000 7f1ea7e00640 1 -- 172.21.15.45:0/3793520497 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e98034630 con 0x7f1e8c0039b0 2024-04-16T16:18:21.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.643+0000 7f1ea7e00640 1 -- 172.21.15.45:0/3793520497 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e98034a20 con 0x7f1e8c0039b0 2024-04-16T16:18:21.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.644+0000 7f1eae600640 1 -- 172.21.15.45:0/3793520497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e8c0039b0 msgr2=0x7f1e8c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:21.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.644+0000 7f1eae600640 1 --2- 172.21.15.45:0/3793520497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e8c0039b0 0x7f1e8c003dd0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800a870 tx=0x7f1e8c005830 comp rx=0 tx=0).stop 2024-04-16T16:18:21.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.644+0000 7f1eadc00640 1 -- 172.21.15.45:0/3793520497 reap_dead start 2024-04-16T16:18:21.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.644+0000 7f1eae600640 1 -- 172.21.15.45:0/3793520497 shutdown_connections 2024-04-16T16:18:21.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.644+0000 7f1eae600640 1 -- 172.21.15.45:0/3793520497 >> 172.21.15.45:0/3793520497 conn(0x7f1ea806b930 msgr2=0x7f1ea806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:21.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.644+0000 7f1eae600640 1 -- 172.21.15.45:0/3793520497 shutdown_connections 2024-04-16T16:18:21.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.644+0000 7f1eae600640 1 -- 172.21.15.45:0/3793520497 wait complete. 2024-04-16T16:18:21.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.646+0000 7f1eae600640 1 Processor -- start 2024-04-16T16:18:21.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.646+0000 7f1eae600640 1 -- start start 2024-04-16T16:18:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.647+0000 7f1eae600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8c0039b0 0x7f1ea80ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.647+0000 7f1eae600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea80b1fe0 0x7f1ea81f5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.647+0000 7f1eae600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea815e910 0x7f1ea815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.647+0000 7f1eae600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea806df40 con 0x7f1ea80b1fe0 2024-04-16T16:18:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.647+0000 7f1eae600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea815dc00 con 0x7f1e8c0039b0 2024-04-16T16:18:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.647+0000 7f1eae600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea806e0c0 con 0x7f1ea815e910 2024-04-16T16:18:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.647+0000 7f1e9fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea80b1fe0 0x7f1ea81f5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.647+0000 7f1e9fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea80b1fe0 0x7f1ea81f5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57336/0 (socket says 172.21.15.45:57336) 2024-04-16T16:18:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.647+0000 7f1e9fe00640 1 -- 172.21.15.45:0/118257098 learned_addr learned my addr 172.21.15.45:0/118257098 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.647+0000 7f1ead200640 1 --2- 172.21.15.45:0/118257098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8c0039b0 0x7f1ea80ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.648+0000 7f1eadc00640 1 --2- 172.21.15.45:0/118257098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea815e910 0x7f1ea815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.648+0000 7f1e9fe00640 1 -- 172.21.15.45:0/118257098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea815e910 msgr2=0x7f1ea815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.648+0000 7f1e9fe00640 1 --2- 172.21.15.45:0/118257098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea815e910 0x7f1ea815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.648+0000 7f1e9fe00640 1 -- 172.21.15.45:0/118257098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8c0039b0 msgr2=0x7f1ea80ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.648+0000 7f1e9fe00640 1 --2- 172.21.15.45:0/118257098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8c0039b0 0x7f1ea80ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.648+0000 7f1e9fe00640 1 -- 172.21.15.45:0/118257098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea813d440 con 0x7f1ea80b1fe0 2024-04-16T16:18:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.648+0000 7f1e9fe00640 1 --2- 172.21.15.45:0/118257098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea80b1fe0 0x7f1ea81f5020 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f1e9000e7f0 tx=0x7f1e9000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.648+0000 7f1ea5600640 1 -- 172.21.15.45:0/118257098 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e90017070 con 0x7f1ea80b1fe0 2024-04-16T16:18:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.648+0000 7f1ea5600640 1 -- 172.21.15.45:0/118257098 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e9000c940 con 0x7f1ea80b1fe0 2024-04-16T16:18:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.648+0000 7f1ea5600640 1 -- 172.21.15.45:0/118257098 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9000cd10 con 0x7f1ea80b1fe0 2024-04-16T16:18:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.648+0000 7f1eae600640 1 -- 172.21.15.45:0/118257098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea81f5560 con 0x7f1ea80b1fe0 2024-04-16T16:18:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.649+0000 7f1eae600640 1 -- 172.21.15.45:0/118257098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea80cd4d0 con 0x7f1ea80b1fe0 2024-04-16T16:18:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.650+0000 7f1eae600640 1 -- 172.21.15.45:0/118257098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ea8075780 con 0x7f1ea80b1fe0 2024-04-16T16:18:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.651+0000 7f1ea5600640 1 -- 172.21.15.45:0/118257098 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f1e9001a070 con 0x7f1ea80b1fe0 2024-04-16T16:18:21.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.651+0000 7f1ea5600640 1 --2- 172.21.15.45:0/118257098 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f1e8406ca40 0x7f1e8406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:21.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.652+0000 7f1ea5600640 1 -- 172.21.15.45:0/118257098 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f1e9008bb10 con 0x7f1ea80b1fe0 2024-04-16T16:18:21.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.652+0000 7f1ead200640 1 --2- 172.21.15.45:0/118257098 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f1e8406ca40 0x7f1e8406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:21.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.653+0000 7f1ead200640 1 --2- 172.21.15.45:0/118257098 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f1e8406ca40 0x7f1e8406ef00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1e98034000 tx=0x7f1e98009030 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:21.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.656+0000 7f1ea5600640 1 -- 172.21.15.45:0/118257098 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1ea81f5560 con 0x7f1ea80b1fe0 2024-04-16T16:18:21.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.862+0000 7f1eae600640 1 -- 172.21.15.45:0/118257098 --> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1ea806f580 con 0x7f1e8406ca40 2024-04-16T16:18:21.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.881+0000 7f1ea5600640 1 -- 172.21.15.45:0/118257098 <== mgr.5791 v2:172.21.15.103:6832/3728775368 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+248113 (secure 0 0 0) 0x7f1ea806f580 con 0x7f1e8406ca40 2024-04-16T16:18:21.883 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:21.884 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:18:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.886+0000 7f1eae600640 1 -- 172.21.15.45:0/118257098 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f1e8406ca40 msgr2=0x7f1e8406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.886+0000 7f1eae600640 1 --2- 172.21.15.45:0/118257098 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f1e8406ca40 0x7f1e8406ef00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1e98034000 tx=0x7f1e98009030 comp rx=0 tx=0).stop 2024-04-16T16:18:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.886+0000 7f1eae600640 1 -- 172.21.15.45:0/118257098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea80b1fe0 msgr2=0x7f1ea81f5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.886+0000 7f1eae600640 1 --2- 172.21.15.45:0/118257098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea80b1fe0 0x7f1ea81f5020 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f1e9000e7f0 tx=0x7f1e9000ecb0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.887+0000 7f1eae600640 1 -- 172.21.15.45:0/118257098 shutdown_connections 2024-04-16T16:18:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.887+0000 7f1eae600640 1 --2- 172.21.15.45:0/118257098 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f1e8406ca40 0x7f1e8406ef00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.887+0000 7f1eae600640 1 --2- 172.21.15.45:0/118257098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea815e910 0x7f1ea815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.887+0000 7f1eae600640 1 --2- 172.21.15.45:0/118257098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea80b1fe0 0x7f1ea81f5020 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.887+0000 7f1eae600640 1 --2- 172.21.15.45:0/118257098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8c0039b0 0x7f1ea80ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.887+0000 7f1eae600640 1 -- 172.21.15.45:0/118257098 >> 172.21.15.45:0/118257098 conn(0x7f1ea806b930 msgr2=0x7f1ea80c5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.887+0000 7f1eae600640 1 -- 172.21.15.45:0/118257098 shutdown_connections 2024-04-16T16:18:21.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:21.887+0000 7f1eae600640 1 -- 172.21.15.45:0/118257098 wait complete. 2024-04-16T16:18:21.906 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-16T16:18:21.119366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":261,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258652,"kb_used_data":9196,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716068,"statfs":{"total":773094113280,"available":772829253632,"internally_reserved":0,"allocated":9416704,"data_stored":5036287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002211"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.298782+0000","last_change":"2024-04-16T16:18:15.298782+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.298782+0000","last_undegraded":"2024-04-16T16:18:15.298782+0000","last_fullsized":"2024-04-16T16:18:15.298782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293043+0000","last_change":"2024-04-16T16:18:15.293043+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293043+0000","last_undegraded":"2024-04-16T16:18:15.293043+0000","last_fullsized":"2024-04-16T16:18:15.293043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.284139+0000","last_change":"2024-04-16T16:18:15.284139+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.284139+0000","last_undegraded":"2024-04-16T16:18:15.284139+0000","last_fullsized":"2024-04-16T16:18:15.284139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312806+0000","last_change":"2024-04-16T16:18:16.316835+0000","last_active":"2024-04-16T16:18:17.312806+0000","last_peered":"2024-04-16T16:18:17.312806+0000","last_clean":"2024-04-16T16:18:17.312806+0000","last_became_active":"2024-04-16T16:18:16.309649+0000","last_became_peered":"2024-04-16T16:18:16.309649+0000","last_unstale":"2024-04-16T16:18:17.312806+0000","last_undegraded":"2024-04-16T16:18:17.312806+0000","last_fullsized":"2024-04-16T16:18:17.312806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:21:05.656211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289497+0000","last_change":"2024-04-16T16:18:15.289497+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289497+0000","last_undegraded":"2024-04-16T16:18:15.289497+0000","last_fullsized":"2024-04-16T16:18:15.289497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.292510+0000","last_change":"2024-04-16T16:18:15.292510+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.292510+0000","last_undegraded":"2024-04-16T16:18:15.292510+0000","last_fullsized":"2024-04-16T16:18:15.292510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312314+0000","last_change":"2024-04-16T16:18:16.308558+0000","last_active":"2024-04-16T16:18:17.312314+0000","last_peered":"2024-04-16T16:18:17.312314+0000","last_clean":"2024-04-16T16:18:17.312314+0000","last_became_active":"2024-04-16T16:18:16.308268+0000","last_became_peered":"2024-04-16T16:18:16.308268+0000","last_unstale":"2024-04-16T16:18:17.312314+0000","last_undegraded":"2024-04-16T16:18:17.312314+0000","last_fullsized":"2024-04-16T16:18:17.312314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:13:57.589683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.291320+0000","last_change":"2024-04-16T16:18:15.291320+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.291320+0000","last_undegraded":"2024-04-16T16:18:15.291320+0000","last_fullsized":"2024-04-16T16:18:15.291320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.297496+0000","last_change":"2024-04-16T16:18:15.297496+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.297496+0000","last_undegraded":"2024-04-16T16:18:15.297496+0000","last_fullsized":"2024-04-16T16:18:15.297496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.295832+0000","last_change":"2024-04-16T16:18:15.295832+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.295832+0000","last_undegraded":"2024-04-16T16:18:15.295832+0000","last_fullsized":"2024-04-16T16:18:15.295832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.291161+0000","last_change":"2024-04-16T16:18:15.291161+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.291161+0000","last_undegraded":"2024-04-16T16:18:15.291161+0000","last_fullsized":"2024-04-16T16:18:15.291161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.304631+0000","last_change":"2024-04-16T16:18:15.304631+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.304631+0000","last_undegraded":"2024-04-16T16:18:15.304631+0000","last_fullsized":"2024-04-16T16:18:15.304631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.281906+0000","last_change":"2024-04-16T16:18:15.281906+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.281906+0000","last_undegraded":"2024-04-16T16:18:15.281906+0000","last_fullsized":"2024-04-16T16:18:15.281906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.313189+0000","last_change":"2024-04-16T16:18:16.321879+0000","last_active":"2024-04-16T16:18:17.313189+0000","last_peered":"2024-04-16T16:18:17.313189+0000","last_clean":"2024-04-16T16:18:17.313189+0000","last_became_active":"2024-04-16T16:18:16.310779+0000","last_became_peered":"2024-04-16T16:18:16.310779+0000","last_unstale":"2024-04-16T16:18:17.313189+0000","last_undegraded":"2024-04-16T16:18:17.313189+0000","last_fullsized":"2024-04-16T16:18:17.313189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:46:46.691832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394765+0000","last_change":"2024-04-16T16:18:16.314327+0000","last_active":"2024-04-16T16:18:17.394765+0000","last_peered":"2024-04-16T16:18:17.394765+0000","last_clean":"2024-04-16T16:18:17.394765+0000","last_became_active":"2024-04-16T16:18:16.313983+0000","last_became_peered":"2024-04-16T16:18:16.313983+0000","last_unstale":"2024-04-16T16:18:17.394765+0000","last_undegraded":"2024-04-16T16:18:17.394765+0000","last_fullsized":"2024-04-16T16:18:17.394765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:06:33.666684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.287619+0000","last_change":"2024-04-16T16:18:15.287619+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.287619+0000","last_undegraded":"2024-04-16T16:18:15.287619+0000","last_fullsized":"2024-04-16T16:18:15.287619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.290484+0000","last_change":"2024-04-16T16:18:15.290484+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.290484+0000","last_undegraded":"2024-04-16T16:18:15.290484+0000","last_fullsized":"2024-04-16T16:18:15.290484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312691+0000","last_change":"2024-04-16T16:18:16.314027+0000","last_active":"2024-04-16T16:18:17.312691+0000","last_peered":"2024-04-16T16:18:17.312691+0000","last_clean":"2024-04-16T16:18:17.312691+0000","last_became_active":"2024-04-16T16:18:16.313426+0000","last_became_peered":"2024-04-16T16:18:16.313426+0000","last_unstale":"2024-04-16T16:18:17.312691+0000","last_undegraded":"2024-04-16T16:18:17.312691+0000","last_fullsized":"2024-04-16T16:18:17.312691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:30:17.865648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444884+0000","last_change":"2024-04-16T16:18:16.305406+0000","last_active":"2024-04-16T16:18:17.444884+0000","last_peered":"2024-04-16T16:18:17.444884+0000","last_clean":"2024-04-16T16:18:17.444884+0000","last_became_active":"2024-04-16T16:18:16.305023+0000","last_became_peered":"2024-04-16T16:18:16.305023+0000","last_unstale":"2024-04-16T16:18:17.444884+0000","last_undegraded":"2024-04-16T16:18:17.444884+0000","last_fullsized":"2024-04-16T16:18:17.444884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:20.251382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312975+0000","last_change":"2024-04-16T16:18:16.317943+0000","last_active":"2024-04-16T16:18:17.312975+0000","last_peered":"2024-04-16T16:18:17.312975+0000","last_clean":"2024-04-16T16:18:17.312975+0000","last_became_active":"2024-04-16T16:18:16.317474+0000","last_became_peered":"2024-04-16T16:18:16.317474+0000","last_unstale":"2024-04-16T16:18:17.312975+0000","last_undegraded":"2024-04-16T16:18:17.312975+0000","last_fullsized":"2024-04-16T16:18:17.312975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:52:31.379753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312792+0000","last_change":"2024-04-16T16:18:16.300164+0000","last_active":"2024-04-16T16:18:17.312792+0000","last_peered":"2024-04-16T16:18:17.312792+0000","last_clean":"2024-04-16T16:18:17.312792+0000","last_became_active":"2024-04-16T16:18:16.299838+0000","last_became_peered":"2024-04-16T16:18:16.299838+0000","last_unstale":"2024-04-16T16:18:17.312792+0000","last_undegraded":"2024-04-16T16:18:17.312792+0000","last_fullsized":"2024-04-16T16:18:17.312792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:22.876310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296503+0000","last_change":"2024-04-16T16:18:15.296503+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296503+0000","last_undegraded":"2024-04-16T16:18:15.296503+0000","last_fullsized":"2024-04-16T16:18:15.296503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296468+0000","last_change":"2024-04-16T16:18:15.296468+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296468+0000","last_undegraded":"2024-04-16T16:18:15.296468+0000","last_fullsized":"2024-04-16T16:18:15.296468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312382+0000","last_change":"2024-04-16T16:18:16.315372+0000","last_active":"2024-04-16T16:18:17.312382+0000","last_peered":"2024-04-16T16:18:17.312382+0000","last_clean":"2024-04-16T16:18:17.312382+0000","last_became_active":"2024-04-16T16:18:16.315098+0000","last_became_peered":"2024-04-16T16:18:16.315098+0000","last_unstale":"2024-04-16T16:18:17.312382+0000","last_undegraded":"2024-04-16T16:18:17.312382+0000","last_fullsized":"2024-04-16T16:18:17.312382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:22.929509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293145+0000","last_change":"2024-04-16T16:18:15.293145+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293145+0000","last_undegraded":"2024-04-16T16:18:15.293145+0000","last_fullsized":"2024-04-16T16:18:15.293145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.282088+0000","last_change":"2024-04-16T16:18:15.282088+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.282088+0000","last_undegraded":"2024-04-16T16:18:15.282088+0000","last_fullsized":"2024-04-16T16:18:15.282088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.292185+0000","last_change":"2024-04-16T16:18:15.292185+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.292185+0000","last_undegraded":"2024-04-16T16:18:15.292185+0000","last_fullsized":"2024-04-16T16:18:15.292185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.304185+0000","last_change":"2024-04-16T16:18:15.304185+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.304185+0000","last_undegraded":"2024-04-16T16:18:15.304185+0000","last_fullsized":"2024-04-16T16:18:15.304185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289755+0000","last_change":"2024-04-16T16:18:15.289755+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289755+0000","last_undegraded":"2024-04-16T16:18:15.289755+0000","last_fullsized":"2024-04-16T16:18:15.289755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296039+0000","last_change":"2024-04-16T16:18:15.296039+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296039+0000","last_undegraded":"2024-04-16T16:18:15.296039+0000","last_fullsized":"2024-04-16T16:18:15.296039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.287029+0000","last_change":"2024-04-16T16:18:15.287029+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.287029+0000","last_undegraded":"2024-04-16T16:18:15.287029+0000","last_fullsized":"2024-04-16T16:18:15.287029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.302832+0000","last_change":"2024-04-16T16:18:15.302832+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.302832+0000","last_undegraded":"2024-04-16T16:18:15.302832+0000","last_fullsized":"2024-04-16T16:18:15.302832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.291173+0000","last_change":"2024-04-16T16:18:15.291173+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.291173+0000","last_undegraded":"2024-04-16T16:18:15.291173+0000","last_fullsized":"2024-04-16T16:18:15.291173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311763+0000","last_change":"2024-04-16T16:18:15.307894+0000","last_active":"2024-04-16T16:18:15.311763+0000","last_peered":"2024-04-16T16:18:15.311763+0000","last_clean":"2024-04-16T16:18:15.311763+0000","last_became_active":"2024-04-16T16:18:15.307584+0000","last_became_peered":"2024-04-16T16:18:15.307584+0000","last_unstale":"2024-04-16T16:18:15.311763+0000","last_undegraded":"2024-04-16T16:18:15.311763+0000","last_fullsized":"2024-04-16T16:18:15.311763+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:30:36.737096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.297298+0000","last_change":"2024-04-16T16:18:15.297298+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.297298+0000","last_undegraded":"2024-04-16T16:18:15.297298+0000","last_fullsized":"2024-04-16T16:18:15.297298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.303705+0000","last_change":"2024-04-16T16:18:15.299901+0000","last_active":"2024-04-16T16:18:15.303705+0000","last_peered":"2024-04-16T16:18:15.303705+0000","last_clean":"2024-04-16T16:18:15.303705+0000","last_became_active":"2024-04-16T16:18:15.299622+0000","last_became_peered":"2024-04-16T16:18:15.299622+0000","last_unstale":"2024-04-16T16:18:15.303705+0000","last_undegraded":"2024-04-16T16:18:15.303705+0000","last_fullsized":"2024-04-16T16:18:15.303705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:51.934340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.303193+0000","last_change":"2024-04-16T16:18:15.303193+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.303193+0000","last_undegraded":"2024-04-16T16:18:15.303193+0000","last_fullsized":"2024-04-16T16:18:15.303193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444710+0000","last_change":"2024-04-16T16:18:15.297918+0000","last_active":"2024-04-16T16:18:17.444710+0000","last_peered":"2024-04-16T16:18:17.444710+0000","last_clean":"2024-04-16T16:18:17.444710+0000","last_became_active":"2024-04-16T16:18:15.297217+0000","last_became_peered":"2024-04-16T16:18:15.297217+0000","last_unstale":"2024-04-16T16:18:17.444710+0000","last_undegraded":"2024-04-16T16:18:17.444710+0000","last_fullsized":"2024-04-16T16:18:17.444710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:03:48.538913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.283014+0000","last_change":"2024-04-16T16:18:15.283014+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.283014+0000","last_undegraded":"2024-04-16T16:18:15.283014+0000","last_fullsized":"2024-04-16T16:18:15.283014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443826+0000","last_change":"2024-04-16T16:18:15.300207+0000","last_active":"2024-04-16T16:18:17.443826+0000","last_peered":"2024-04-16T16:18:17.443826+0000","last_clean":"2024-04-16T16:18:17.443826+0000","last_became_active":"2024-04-16T16:18:15.297220+0000","last_became_peered":"2024-04-16T16:18:15.297220+0000","last_unstale":"2024-04-16T16:18:17.443826+0000","last_undegraded":"2024-04-16T16:18:17.443826+0000","last_fullsized":"2024-04-16T16:18:17.443826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:58:34.162900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.295251+0000","last_change":"2024-04-16T16:18:15.295251+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.295251+0000","last_undegraded":"2024-04-16T16:18:15.295251+0000","last_fullsized":"2024-04-16T16:18:15.295251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311825+0000","last_change":"2024-04-16T16:18:15.309872+0000","last_active":"2024-04-16T16:18:15.311825+0000","last_peered":"2024-04-16T16:18:15.311825+0000","last_clean":"2024-04-16T16:18:15.311825+0000","last_became_active":"2024-04-16T16:18:15.309546+0000","last_became_peered":"2024-04-16T16:18:15.309546+0000","last_unstale":"2024-04-16T16:18:15.311825+0000","last_undegraded":"2024-04-16T16:18:15.311825+0000","last_fullsized":"2024-04-16T16:18:15.311825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:27:40.938359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.288554+0000","last_change":"2024-04-16T16:18:15.288554+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.288554+0000","last_undegraded":"2024-04-16T16:18:15.288554+0000","last_fullsized":"2024-04-16T16:18:15.288554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.303669+0000","last_change":"2024-04-16T16:18:15.298912+0000","last_active":"2024-04-16T16:18:15.303669+0000","last_peered":"2024-04-16T16:18:15.303669+0000","last_clean":"2024-04-16T16:18:15.303669+0000","last_became_active":"2024-04-16T16:18:15.298638+0000","last_became_peered":"2024-04-16T16:18:15.298638+0000","last_unstale":"2024-04-16T16:18:15.303669+0000","last_undegraded":"2024-04-16T16:18:15.303669+0000","last_fullsized":"2024-04-16T16:18:15.303669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:30:36.737096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.282766+0000","last_change":"2024-04-16T16:18:15.282766+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.282766+0000","last_undegraded":"2024-04-16T16:18:15.282766+0000","last_fullsized":"2024-04-16T16:18:15.282766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444198+0000","last_change":"2024-04-16T16:18:15.301897+0000","last_active":"2024-04-16T16:18:17.444198+0000","last_peered":"2024-04-16T16:18:17.444198+0000","last_clean":"2024-04-16T16:18:17.444198+0000","last_became_active":"2024-04-16T16:18:15.301549+0000","last_became_peered":"2024-04-16T16:18:15.301549+0000","last_unstale":"2024-04-16T16:18:17.444198+0000","last_undegraded":"2024-04-16T16:18:17.444198+0000","last_fullsized":"2024-04-16T16:18:17.444198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:29:16.985875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394783+0000","last_change":"2024-04-16T16:18:16.308198+0000","last_active":"2024-04-16T16:18:17.394783+0000","last_peered":"2024-04-16T16:18:17.394783+0000","last_clean":"2024-04-16T16:18:17.394783+0000","last_became_active":"2024-04-16T16:18:16.307924+0000","last_became_peered":"2024-04-16T16:18:16.307924+0000","last_unstale":"2024-04-16T16:18:17.394783+0000","last_undegraded":"2024-04-16T16:18:17.394783+0000","last_fullsized":"2024-04-16T16:18:17.394783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:34:10.200444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.307302+0000","last_change":"2024-04-16T16:18:15.303661+0000","last_active":"2024-04-16T16:18:15.307302+0000","last_peered":"2024-04-16T16:18:15.307302+0000","last_clean":"2024-04-16T16:18:15.307302+0000","last_became_active":"2024-04-16T16:18:15.300455+0000","last_became_peered":"2024-04-16T16:18:15.300455+0000","last_unstale":"2024-04-16T16:18:15.307302+0000","last_undegraded":"2024-04-16T16:18:15.307302+0000","last_fullsized":"2024-04-16T16:18:15.307302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:21:04.639131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394997+0000","last_change":"2024-04-16T16:18:16.324165+0000","last_active":"2024-04-16T16:18:17.394997+0000","last_peered":"2024-04-16T16:18:17.394997+0000","last_clean":"2024-04-16T16:18:17.394997+0000","last_became_active":"2024-04-16T16:18:16.323865+0000","last_became_peered":"2024-04-16T16:18:16.323865+0000","last_unstale":"2024-04-16T16:18:17.394997+0000","last_undegraded":"2024-04-16T16:18:17.394997+0000","last_fullsized":"2024-04-16T16:18:17.394997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:51:43.715542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311275+0000","last_change":"2024-04-16T16:18:15.305839+0000","last_active":"2024-04-16T16:18:15.311275+0000","last_peered":"2024-04-16T16:18:15.311275+0000","last_clean":"2024-04-16T16:18:15.311275+0000","last_became_active":"2024-04-16T16:18:15.305453+0000","last_became_peered":"2024-04-16T16:18:15.305453+0000","last_unstale":"2024-04-16T16:18:15.311275+0000","last_undegraded":"2024-04-16T16:18:15.311275+0000","last_fullsized":"2024-04-16T16:18:15.311275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:51:24.645998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293391+0000","last_change":"2024-04-16T16:18:15.293391+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293391+0000","last_undegraded":"2024-04-16T16:18:15.293391+0000","last_fullsized":"2024-04-16T16:18:15.293391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311136+0000","last_change":"2024-04-16T16:18:15.304936+0000","last_active":"2024-04-16T16:18:15.311136+0000","last_peered":"2024-04-16T16:18:15.311136+0000","last_clean":"2024-04-16T16:18:15.311136+0000","last_became_active":"2024-04-16T16:18:15.304590+0000","last_became_peered":"2024-04-16T16:18:15.304590+0000","last_unstale":"2024-04-16T16:18:15.311136+0000","last_undegraded":"2024-04-16T16:18:15.311136+0000","last_fullsized":"2024-04-16T16:18:15.311136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:33.215073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443924+0000","last_change":"2024-04-16T16:18:16.316337+0000","last_active":"2024-04-16T16:18:17.443924+0000","last_peered":"2024-04-16T16:18:17.443924+0000","last_clean":"2024-04-16T16:18:17.443924+0000","last_became_active":"2024-04-16T16:18:16.316019+0000","last_became_peered":"2024-04-16T16:18:16.316019+0000","last_unstale":"2024-04-16T16:18:17.443924+0000","last_undegraded":"2024-04-16T16:18:17.443924+0000","last_fullsized":"2024-04-16T16:18:17.443924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:21:33.844543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311896+0000","last_change":"2024-04-16T16:18:15.308173+0000","last_active":"2024-04-16T16:18:15.311896+0000","last_peered":"2024-04-16T16:18:15.311896+0000","last_clean":"2024-04-16T16:18:15.311896+0000","last_became_active":"2024-04-16T16:18:15.307837+0000","last_became_peered":"2024-04-16T16:18:15.307837+0000","last_unstale":"2024-04-16T16:18:15.311896+0000","last_undegraded":"2024-04-16T16:18:15.311896+0000","last_fullsized":"2024-04-16T16:18:15.311896+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:45:40.148360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.290639+0000","last_change":"2024-04-16T16:18:15.290639+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.290639+0000","last_undegraded":"2024-04-16T16:18:15.290639+0000","last_fullsized":"2024-04-16T16:18:15.290639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.309798+0000","last_change":"2024-04-16T16:18:15.306390+0000","last_active":"2024-04-16T16:18:15.309798+0000","last_peered":"2024-04-16T16:18:15.309798+0000","last_clean":"2024-04-16T16:18:15.309798+0000","last_became_active":"2024-04-16T16:18:15.306063+0000","last_became_peered":"2024-04-16T16:18:15.306063+0000","last_unstale":"2024-04-16T16:18:15.309798+0000","last_undegraded":"2024-04-16T16:18:15.309798+0000","last_fullsized":"2024-04-16T16:18:15.309798+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:57.184445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443652+0000","last_change":"2024-04-16T16:18:16.315534+0000","last_active":"2024-04-16T16:18:17.443652+0000","last_peered":"2024-04-16T16:18:17.443652+0000","last_clean":"2024-04-16T16:18:17.443652+0000","last_became_active":"2024-04-16T16:18:16.315301+0000","last_became_peered":"2024-04-16T16:18:16.315301+0000","last_unstale":"2024-04-16T16:18:17.443652+0000","last_undegraded":"2024-04-16T16:18:17.443652+0000","last_fullsized":"2024-04-16T16:18:17.443652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:42.181514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.310987+0000","last_change":"2024-04-16T16:18:15.308221+0000","last_active":"2024-04-16T16:18:15.310987+0000","last_peered":"2024-04-16T16:18:15.310987+0000","last_clean":"2024-04-16T16:18:15.310987+0000","last_became_active":"2024-04-16T16:18:15.307923+0000","last_became_peered":"2024-04-16T16:18:15.307923+0000","last_unstale":"2024-04-16T16:18:15.310987+0000","last_undegraded":"2024-04-16T16:18:15.310987+0000","last_fullsized":"2024-04-16T16:18:15.310987+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:42:36.176215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.280567+0000","last_change":"2024-04-16T16:18:15.280567+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.280567+0000","last_undegraded":"2024-04-16T16:18:15.280567+0000","last_fullsized":"2024-04-16T16:18:15.280567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.395004+0000","last_change":"2024-04-16T16:18:15.309398+0000","last_active":"2024-04-16T16:18:17.395004+0000","last_peered":"2024-04-16T16:18:17.395004+0000","last_clean":"2024-04-16T16:18:17.395004+0000","last_became_active":"2024-04-16T16:18:15.308959+0000","last_became_peered":"2024-04-16T16:18:15.308959+0000","last_unstale":"2024-04-16T16:18:17.395004+0000","last_undegraded":"2024-04-16T16:18:17.395004+0000","last_fullsized":"2024-04-16T16:18:17.395004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:10:27.168721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.295248+0000","last_change":"2024-04-16T16:18:15.295248+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.295248+0000","last_undegraded":"2024-04-16T16:18:15.295248+0000","last_fullsized":"2024-04-16T16:18:15.295248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311722+0000","last_change":"2024-04-16T16:18:15.309922+0000","last_active":"2024-04-16T16:18:15.311722+0000","last_peered":"2024-04-16T16:18:15.311722+0000","last_clean":"2024-04-16T16:18:15.311722+0000","last_became_active":"2024-04-16T16:18:15.308635+0000","last_became_peered":"2024-04-16T16:18:15.308635+0000","last_unstale":"2024-04-16T16:18:15.311722+0000","last_undegraded":"2024-04-16T16:18:15.311722+0000","last_fullsized":"2024-04-16T16:18:15.311722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:05:41.836526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.287330+0000","last_change":"2024-04-16T16:18:15.287330+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.287330+0000","last_undegraded":"2024-04-16T16:18:15.287330+0000","last_fullsized":"2024-04-16T16:18:15.287330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"84'8","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443410+0000","last_change":"2024-04-16T16:18:15.299371+0000","last_active":"2024-04-16T16:18:17.443410+0000","last_peered":"2024-04-16T16:18:17.443410+0000","last_clean":"2024-04-16T16:18:17.443410+0000","last_became_active":"2024-04-16T16:18:15.299047+0000","last_became_peered":"2024-04-16T16:18:15.299047+0000","last_unstale":"2024-04-16T16:18:17.443410+0000","last_undegraded":"2024-04-16T16:18:17.443410+0000","last_fullsized":"2024-04-16T16:18:17.443410+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:14:18.404852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.280762+0000","last_change":"2024-04-16T16:18:15.280762+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.280762+0000","last_undegraded":"2024-04-16T16:18:15.280762+0000","last_fullsized":"2024-04-16T16:18:15.280762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311071+0000","last_change":"2024-04-16T16:18:15.305063+0000","last_active":"2024-04-16T16:18:15.311071+0000","last_peered":"2024-04-16T16:18:15.311071+0000","last_clean":"2024-04-16T16:18:15.311071+0000","last_became_active":"2024-04-16T16:18:15.304669+0000","last_became_peered":"2024-04-16T16:18:15.304669+0000","last_unstale":"2024-04-16T16:18:15.311071+0000","last_undegraded":"2024-04-16T16:18:15.311071+0000","last_fullsized":"2024-04-16T16:18:15.311071+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:59.303188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296848+0000","last_change":"2024-04-16T16:18:15.296848+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296848+0000","last_undegraded":"2024-04-16T16:18:15.296848+0000","last_fullsized":"2024-04-16T16:18:15.296848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443949+0000","last_change":"2024-04-16T16:18:15.302798+0000","last_active":"2024-04-16T16:18:17.443949+0000","last_peered":"2024-04-16T16:18:17.443949+0000","last_clean":"2024-04-16T16:18:17.443949+0000","last_became_active":"2024-04-16T16:18:15.301509+0000","last_became_peered":"2024-04-16T16:18:15.301509+0000","last_unstale":"2024-04-16T16:18:17.443949+0000","last_undegraded":"2024-04-16T16:18:17.443949+0000","last_fullsized":"2024-04-16T16:18:17.443949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:20:56.511072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289641+0000","last_change":"2024-04-16T16:18:15.289641+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289641+0000","last_undegraded":"2024-04-16T16:18:15.289641+0000","last_fullsized":"2024-04-16T16:18:15.289641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443959+0000","last_change":"2024-04-16T16:18:15.302329+0000","last_active":"2024-04-16T16:18:17.443959+0000","last_peered":"2024-04-16T16:18:17.443959+0000","last_clean":"2024-04-16T16:18:17.443959+0000","last_became_active":"2024-04-16T16:18:15.301993+0000","last_became_peered":"2024-04-16T16:18:15.301993+0000","last_unstale":"2024-04-16T16:18:17.443959+0000","last_undegraded":"2024-04-16T16:18:17.443959+0000","last_fullsized":"2024-04-16T16:18:17.443959+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:00:22.780800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.313248+0000","last_change":"2024-04-16T16:18:16.318257+0000","last_active":"2024-04-16T16:18:17.313248+0000","last_peered":"2024-04-16T16:18:17.313248+0000","last_clean":"2024-04-16T16:18:17.313248+0000","last_became_active":"2024-04-16T16:18:16.311533+0000","last_became_peered":"2024-04-16T16:18:16.311533+0000","last_unstale":"2024-04-16T16:18:17.313248+0000","last_undegraded":"2024-04-16T16:18:17.313248+0000","last_fullsized":"2024-04-16T16:18:17.313248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:42:37.193295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394822+0000","last_change":"2024-04-16T16:18:15.291033+0000","last_active":"2024-04-16T16:18:17.394822+0000","last_peered":"2024-04-16T16:18:17.394822+0000","last_clean":"2024-04-16T16:18:17.394822+0000","last_became_active":"2024-04-16T16:18:15.290067+0000","last_became_peered":"2024-04-16T16:18:15.290067+0000","last_unstale":"2024-04-16T16:18:17.394822+0000","last_undegraded":"2024-04-16T16:18:17.394822+0000","last_fullsized":"2024-04-16T16:18:17.394822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:06:23.738648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.298204+0000","last_change":"2024-04-16T16:18:15.298204+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.298204+0000","last_undegraded":"2024-04-16T16:18:15.298204+0000","last_fullsized":"2024-04-16T16:18:15.298204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.313304+0000","last_change":"2024-04-16T16:18:15.301900+0000","last_active":"2024-04-16T16:18:17.313304+0000","last_peered":"2024-04-16T16:18:17.313304+0000","last_clean":"2024-04-16T16:18:17.313304+0000","last_became_active":"2024-04-16T16:18:15.301268+0000","last_became_peered":"2024-04-16T16:18:15.301268+0000","last_unstale":"2024-04-16T16:18:17.313304+0000","last_undegraded":"2024-04-16T16:18:17.313304+0000","last_fullsized":"2024-04-16T16:18:17.313304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:15.061257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312362+0000","last_change":"2024-04-16T16:18:16.314030+0000","last_active":"2024-04-16T16:18:17.312362+0000","last_peered":"2024-04-16T16:18:17.312362+0000","last_clean":"2024-04-16T16:18:17.312362+0000","last_became_active":"2024-04-16T16:18:16.313572+0000","last_became_peered":"2024-04-16T16:18:16.313572+0000","last_unstale":"2024-04-16T16:18:17.312362+0000","last_undegraded":"2024-04-16T16:18:17.312362+0000","last_fullsized":"2024-04-16T16:18:17.312362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:50:11.604716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312426+0000","last_change":"2024-04-16T16:18:15.308430+0000","last_active":"2024-04-16T16:18:17.312426+0000","last_peered":"2024-04-16T16:18:17.312426+0000","last_clean":"2024-04-16T16:18:17.312426+0000","last_became_active":"2024-04-16T16:18:15.308128+0000","last_became_peered":"2024-04-16T16:18:15.308128+0000","last_unstale":"2024-04-16T16:18:17.312426+0000","last_undegraded":"2024-04-16T16:18:17.312426+0000","last_fullsized":"2024-04-16T16:18:17.312426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:23.975082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293789+0000","last_change":"2024-04-16T16:18:15.293789+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293789+0000","last_undegraded":"2024-04-16T16:18:15.293789+0000","last_fullsized":"2024-04-16T16:18:15.293789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394296+0000","last_change":"2024-04-16T16:18:15.292003+0000","last_active":"2024-04-16T16:18:17.394296+0000","last_peered":"2024-04-16T16:18:17.394296+0000","last_clean":"2024-04-16T16:18:17.394296+0000","last_became_active":"2024-04-16T16:18:15.291691+0000","last_became_peered":"2024-04-16T16:18:15.291691+0000","last_unstale":"2024-04-16T16:18:17.394296+0000","last_undegraded":"2024-04-16T16:18:17.394296+0000","last_fullsized":"2024-04-16T16:18:17.394296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:28.452108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289712+0000","last_change":"2024-04-16T16:18:15.289712+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289712+0000","last_undegraded":"2024-04-16T16:18:15.289712+0000","last_fullsized":"2024-04-16T16:18:15.289712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.308009+0000","last_change":"2024-04-16T16:18:15.305818+0000","last_active":"2024-04-16T16:18:15.308009+0000","last_peered":"2024-04-16T16:18:15.308009+0000","last_clean":"2024-04-16T16:18:15.308009+0000","last_became_active":"2024-04-16T16:18:15.304636+0000","last_became_peered":"2024-04-16T16:18:15.304636+0000","last_unstale":"2024-04-16T16:18:15.308009+0000","last_undegraded":"2024-04-16T16:18:15.308009+0000","last_fullsized":"2024-04-16T16:18:15.308009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:21.912428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.283292+0000","last_change":"2024-04-16T16:18:15.283292+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.283292+0000","last_undegraded":"2024-04-16T16:18:15.283292+0000","last_fullsized":"2024-04-16T16:18:15.283292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.308128+0000","last_change":"2024-04-16T16:18:15.305858+0000","last_active":"2024-04-16T16:18:15.308128+0000","last_peered":"2024-04-16T16:18:15.308128+0000","last_clean":"2024-04-16T16:18:15.308128+0000","last_became_active":"2024-04-16T16:18:15.305250+0000","last_became_peered":"2024-04-16T16:18:15.305250+0000","last_unstale":"2024-04-16T16:18:15.308128+0000","last_undegraded":"2024-04-16T16:18:15.308128+0000","last_fullsized":"2024-04-16T16:18:15.308128+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:46:45.674751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.302671+0000","last_change":"2024-04-16T16:18:15.302671+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.302671+0000","last_undegraded":"2024-04-16T16:18:15.302671+0000","last_fullsized":"2024-04-16T16:18:15.302671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311926+0000","last_change":"2024-04-16T16:18:15.309023+0000","last_active":"2024-04-16T16:18:15.311926+0000","last_peered":"2024-04-16T16:18:15.311926+0000","last_clean":"2024-04-16T16:18:15.311926+0000","last_became_active":"2024-04-16T16:18:15.308399+0000","last_became_peered":"2024-04-16T16:18:15.308399+0000","last_unstale":"2024-04-16T16:18:15.311926+0000","last_undegraded":"2024-04-16T16:18:15.311926+0000","last_fullsized":"2024-04-16T16:18:15.311926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:27:40.938359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.288574+0000","last_change":"2024-04-16T16:18:15.288574+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.288574+0000","last_undegraded":"2024-04-16T16:18:15.288574+0000","last_fullsized":"2024-04-16T16:18:15.288574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.314507+0000","last_change":"2024-04-16T16:18:15.311086+0000","last_active":"2024-04-16T16:18:15.314507+0000","last_peered":"2024-04-16T16:18:15.314507+0000","last_clean":"2024-04-16T16:18:15.314507+0000","last_became_active":"2024-04-16T16:18:15.310594+0000","last_became_peered":"2024-04-16T16:18:15.310594+0000","last_unstale":"2024-04-16T16:18:15.314507+0000","last_undegraded":"2024-04-16T16:18:15.314507+0000","last_fullsized":"2024-04-16T16:18:15.314507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:02:52.308842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.292047+0000","last_change":"2024-04-16T16:18:15.292047+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.292047+0000","last_undegraded":"2024-04-16T16:18:15.292047+0000","last_fullsized":"2024-04-16T16:18:15.292047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444049+0000","last_change":"2024-04-16T16:18:15.299375+0000","last_active":"2024-04-16T16:18:17.444049+0000","last_peered":"2024-04-16T16:18:17.444049+0000","last_clean":"2024-04-16T16:18:17.444049+0000","last_became_active":"2024-04-16T16:18:15.298710+0000","last_became_peered":"2024-04-16T16:18:15.298710+0000","last_unstale":"2024-04-16T16:18:17.444049+0000","last_undegraded":"2024-04-16T16:18:17.444049+0000","last_fullsized":"2024-04-16T16:18:17.444049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:12:39.298882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444137+0000","last_change":"2024-04-16T16:18:16.322248+0000","last_active":"2024-04-16T16:18:17.444137+0000","last_peered":"2024-04-16T16:18:17.444137+0000","last_clean":"2024-04-16T16:18:17.444137+0000","last_became_active":"2024-04-16T16:18:16.318615+0000","last_became_peered":"2024-04-16T16:18:16.318615+0000","last_unstale":"2024-04-16T16:18:17.444137+0000","last_undegraded":"2024-04-16T16:18:17.444137+0000","last_fullsized":"2024-04-16T16:18:17.444137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:16:57.729600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.303620+0000","last_change":"2024-04-16T16:18:15.298300+0000","last_active":"2024-04-16T16:18:15.303620+0000","last_peered":"2024-04-16T16:18:15.303620+0000","last_clean":"2024-04-16T16:18:15.303620+0000","last_became_active":"2024-04-16T16:18:15.297872+0000","last_became_peered":"2024-04-16T16:18:15.297872+0000","last_unstale":"2024-04-16T16:18:15.303620+0000","last_undegraded":"2024-04-16T16:18:15.303620+0000","last_fullsized":"2024-04-16T16:18:15.303620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:51:24.645998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.281611+0000","last_change":"2024-04-16T16:18:15.281611+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.281611+0000","last_undegraded":"2024-04-16T16:18:15.281611+0000","last_fullsized":"2024-04-16T16:18:15.281611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311874+0000","last_change":"2024-04-16T16:18:15.308009+0000","last_active":"2024-04-16T16:18:15.311874+0000","last_peered":"2024-04-16T16:18:15.311874+0000","last_clean":"2024-04-16T16:18:15.311874+0000","last_became_active":"2024-04-16T16:18:15.307561+0000","last_became_peered":"2024-04-16T16:18:15.307561+0000","last_unstale":"2024-04-16T16:18:15.311874+0000","last_undegraded":"2024-04-16T16:18:15.311874+0000","last_fullsized":"2024-04-16T16:18:15.311874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:51.934340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296023+0000","last_change":"2024-04-16T16:18:15.296023+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296023+0000","last_undegraded":"2024-04-16T16:18:15.296023+0000","last_fullsized":"2024-04-16T16:18:15.296023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311093+0000","last_change":"2024-04-16T16:18:15.309487+0000","last_active":"2024-04-16T16:18:15.311093+0000","last_peered":"2024-04-16T16:18:15.311093+0000","last_clean":"2024-04-16T16:18:15.311093+0000","last_became_active":"2024-04-16T16:18:15.308448+0000","last_became_peered":"2024-04-16T16:18:15.308448+0000","last_unstale":"2024-04-16T16:18:15.311093+0000","last_undegraded":"2024-04-16T16:18:15.311093+0000","last_fullsized":"2024-04-16T16:18:15.311093+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:45:40.148360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296266+0000","last_change":"2024-04-16T16:18:15.296266+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296266+0000","last_undegraded":"2024-04-16T16:18:15.296266+0000","last_fullsized":"2024-04-16T16:18:15.296266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.307177+0000","last_change":"2024-04-16T16:18:15.301975+0000","last_active":"2024-04-16T16:18:15.307177+0000","last_peered":"2024-04-16T16:18:15.307177+0000","last_clean":"2024-04-16T16:18:15.307177+0000","last_became_active":"2024-04-16T16:18:15.300433+0000","last_became_peered":"2024-04-16T16:18:15.300433+0000","last_unstale":"2024-04-16T16:18:15.307177+0000","last_undegraded":"2024-04-16T16:18:15.307177+0000","last_fullsized":"2024-04-16T16:18:15.307177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:57.659561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"74'161","reported_seq":452,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.286818+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:18:15.286818+0000","last_peered":"2024-04-16T16:18:15.286818+0000","last_clean":"2024-04-16T16:18:15.286818+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:18:15.286818+0000","last_undegraded":"2024-04-16T16:18:15.286818+0000","last_fullsized":"2024-04-16T16:18:15.286818+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16,"ondisk_log_size":16,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":1601536,"data_stored":1062424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":478331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":478331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574909,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":478331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339628,"statfs":{"total":96636764160,"available":96603779072,"internally_reserved":0,"allocated":1056768,"data_stored":493018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574908,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1048576,"data_stored":491988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1007616,"data_stored":472056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574909,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32788,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":1642496,"data_stored":1081808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:18:21.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3f is not active+clean 2024-04-16T16:18:21.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.298782+0000', 'last_change': '2024-04-16T16:18:15.298782+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.298782+0000', 'last_undegraded': '2024-04-16T16:18:15.298782+0000', 'last_fullsized': '2024-04-16T16:18:15.298782+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:18:21.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3e is not active+clean 2024-04-16T16:18:21.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.293043+0000', 'last_change': '2024-04-16T16:18:15.293043+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.293043+0000', 'last_undegraded': '2024-04-16T16:18:15.293043+0000', 'last_fullsized': '2024-04-16T16:18:15.293043+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-16T16:18:21.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3d is not active+clean 2024-04-16T16:18:21.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.284139+0000', 'last_change': '2024-04-16T16:18:15.284139+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.284139+0000', 'last_undegraded': '2024-04-16T16:18:15.284139+0000', 'last_fullsized': '2024-04-16T16:18:15.284139+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-16T16:18:21.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3b is not active+clean 2024-04-16T16:18:21.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.289497+0000', 'last_change': '2024-04-16T16:18:15.289497+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.289497+0000', 'last_undegraded': '2024-04-16T16:18:15.289497+0000', 'last_fullsized': '2024-04-16T16:18:15.289497+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-16T16:18:21.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3a is not active+clean 2024-04-16T16:18:21.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.292510+0000', 'last_change': '2024-04-16T16:18:15.292510+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.292510+0000', 'last_undegraded': '2024-04-16T16:18:15.292510+0000', 'last_fullsized': '2024-04-16T16:18:15.292510+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-16T16:18:21.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.38 is not active+clean 2024-04-16T16:18:21.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.291320+0000', 'last_change': '2024-04-16T16:18:15.291320+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.291320+0000', 'last_undegraded': '2024-04-16T16:18:15.291320+0000', 'last_fullsized': '2024-04-16T16:18:15.291320+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-16T16:18:21.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.37 is not active+clean 2024-04-16T16:18:21.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.297496+0000', 'last_change': '2024-04-16T16:18:15.297496+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.297496+0000', 'last_undegraded': '2024-04-16T16:18:15.297496+0000', 'last_fullsized': '2024-04-16T16:18:15.297496+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-16T16:18:21.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.36 is not active+clean 2024-04-16T16:18:21.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.295832+0000', 'last_change': '2024-04-16T16:18:15.295832+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.295832+0000', 'last_undegraded': '2024-04-16T16:18:15.295832+0000', 'last_fullsized': '2024-04-16T16:18:15.295832+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:18:21.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.35 is not active+clean 2024-04-16T16:18:21.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.291161+0000', 'last_change': '2024-04-16T16:18:15.291161+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.291161+0000', 'last_undegraded': '2024-04-16T16:18:15.291161+0000', 'last_fullsized': '2024-04-16T16:18:15.291161+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-16T16:18:21.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.34 is not active+clean 2024-04-16T16:18:21.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.304631+0000', 'last_change': '2024-04-16T16:18:15.304631+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.304631+0000', 'last_undegraded': '2024-04-16T16:18:15.304631+0000', 'last_fullsized': '2024-04-16T16:18:15.304631+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:18:21.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.33 is not active+clean 2024-04-16T16:18:21.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.281906+0000', 'last_change': '2024-04-16T16:18:15.281906+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.281906+0000', 'last_undegraded': '2024-04-16T16:18:15.281906+0000', 'last_fullsized': '2024-04-16T16:18:15.281906+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-16T16:18:21.912 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.30 is not active+clean 2024-04-16T16:18:21.912 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.287619+0000', 'last_change': '2024-04-16T16:18:15.287619+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.287619+0000', 'last_undegraded': '2024-04-16T16:18:15.287619+0000', 'last_fullsized': '2024-04-16T16:18:15.287619+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-16T16:18:21.912 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2f is not active+clean 2024-04-16T16:18:21.912 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.290484+0000', 'last_change': '2024-04-16T16:18:15.290484+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.290484+0000', 'last_undegraded': '2024-04-16T16:18:15.290484+0000', 'last_fullsized': '2024-04-16T16:18:15.290484+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-16T16:18:21.912 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2a is not active+clean 2024-04-16T16:18:21.912 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.296503+0000', 'last_change': '2024-04-16T16:18:15.296503+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.296503+0000', 'last_undegraded': '2024-04-16T16:18:15.296503+0000', 'last_fullsized': '2024-04-16T16:18:15.296503+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-16T16:18:21.912 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.29 is not active+clean 2024-04-16T16:18:21.912 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.296468+0000', 'last_change': '2024-04-16T16:18:15.296468+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.296468+0000', 'last_undegraded': '2024-04-16T16:18:15.296468+0000', 'last_fullsized': '2024-04-16T16:18:15.296468+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:18:21.912 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.27 is not active+clean 2024-04-16T16:18:21.912 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.293145+0000', 'last_change': '2024-04-16T16:18:15.293145+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.293145+0000', 'last_undegraded': '2024-04-16T16:18:15.293145+0000', 'last_fullsized': '2024-04-16T16:18:15.293145+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-16T16:18:21.912 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.26 is not active+clean 2024-04-16T16:18:21.912 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.282088+0000', 'last_change': '2024-04-16T16:18:15.282088+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.282088+0000', 'last_undegraded': '2024-04-16T16:18:15.282088+0000', 'last_fullsized': '2024-04-16T16:18:15.282088+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-16T16:18:21.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.25 is not active+clean 2024-04-16T16:18:21.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.292185+0000', 'last_change': '2024-04-16T16:18:15.292185+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.292185+0000', 'last_undegraded': '2024-04-16T16:18:15.292185+0000', 'last_fullsized': '2024-04-16T16:18:15.292185+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-16T16:18:21.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.24 is not active+clean 2024-04-16T16:18:21.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.304185+0000', 'last_change': '2024-04-16T16:18:15.304185+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.304185+0000', 'last_undegraded': '2024-04-16T16:18:15.304185+0000', 'last_fullsized': '2024-04-16T16:18:15.304185+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:18:21.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.23 is not active+clean 2024-04-16T16:18:21.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.289755+0000', 'last_change': '2024-04-16T16:18:15.289755+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.289755+0000', 'last_undegraded': '2024-04-16T16:18:15.289755+0000', 'last_fullsized': '2024-04-16T16:18:15.289755+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-16T16:18:21.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.22 is not active+clean 2024-04-16T16:18:21.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.296039+0000', 'last_change': '2024-04-16T16:18:15.296039+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.296039+0000', 'last_undegraded': '2024-04-16T16:18:15.296039+0000', 'last_fullsized': '2024-04-16T16:18:15.296039+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-16T16:18:21.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.21 is not active+clean 2024-04-16T16:18:21.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.287029+0000', 'last_change': '2024-04-16T16:18:15.287029+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.287029+0000', 'last_undegraded': '2024-04-16T16:18:15.287029+0000', 'last_fullsized': '2024-04-16T16:18:15.287029+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-16T16:18:21.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.20 is not active+clean 2024-04-16T16:18:21.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.302832+0000', 'last_change': '2024-04-16T16:18:15.302832+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.302832+0000', 'last_undegraded': '2024-04-16T16:18:15.302832+0000', 'last_fullsized': '2024-04-16T16:18:15.302832+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:18:21.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1e is not active+clean 2024-04-16T16:18:21.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.291173+0000', 'last_change': '2024-04-16T16:18:15.291173+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.291173+0000', 'last_undegraded': '2024-04-16T16:18:15.291173+0000', 'last_fullsized': '2024-04-16T16:18:15.291173+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-16T16:18:21.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1f is not active+clean 2024-04-16T16:18:21.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.297298+0000', 'last_change': '2024-04-16T16:18:15.297298+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.297298+0000', 'last_undegraded': '2024-04-16T16:18:15.297298+0000', 'last_fullsized': '2024-04-16T16:18:15.297298+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:18:21.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1c is not active+clean 2024-04-16T16:18:21.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.303193+0000', 'last_change': '2024-04-16T16:18:15.303193+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.303193+0000', 'last_undegraded': '2024-04-16T16:18:15.303193+0000', 'last_fullsized': '2024-04-16T16:18:15.303193+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:18:21.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1d is not active+clean 2024-04-16T16:18:21.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.283014+0000', 'last_change': '2024-04-16T16:18:15.283014+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.283014+0000', 'last_undegraded': '2024-04-16T16:18:15.283014+0000', 'last_fullsized': '2024-04-16T16:18:15.283014+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-16T16:18:21.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1a is not active+clean 2024-04-16T16:18:21.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.295251+0000', 'last_change': '2024-04-16T16:18:15.295251+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.295251+0000', 'last_undegraded': '2024-04-16T16:18:15.295251+0000', 'last_fullsized': '2024-04-16T16:18:15.295251+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-16T16:18:21.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1b is not active+clean 2024-04-16T16:18:21.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.288554+0000', 'last_change': '2024-04-16T16:18:15.288554+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.288554+0000', 'last_undegraded': '2024-04-16T16:18:15.288554+0000', 'last_fullsized': '2024-04-16T16:18:15.288554+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-16T16:18:21.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.18 is not active+clean 2024-04-16T16:18:21.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.282766+0000', 'last_change': '2024-04-16T16:18:15.282766+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.282766+0000', 'last_undegraded': '2024-04-16T16:18:15.282766+0000', 'last_fullsized': '2024-04-16T16:18:15.282766+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-16T16:18:21.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.17 is not active+clean 2024-04-16T16:18:21.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.293391+0000', 'last_change': '2024-04-16T16:18:15.293391+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.293391+0000', 'last_undegraded': '2024-04-16T16:18:15.293391+0000', 'last_fullsized': '2024-04-16T16:18:15.293391+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-16T16:18:21.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.15 is not active+clean 2024-04-16T16:18:21.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.290639+0000', 'last_change': '2024-04-16T16:18:15.290639+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.290639+0000', 'last_undegraded': '2024-04-16T16:18:15.290639+0000', 'last_fullsized': '2024-04-16T16:18:15.290639+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-16T16:18:21.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.13 is not active+clean 2024-04-16T16:18:21.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.280567+0000', 'last_change': '2024-04-16T16:18:15.280567+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.280567+0000', 'last_undegraded': '2024-04-16T16:18:15.280567+0000', 'last_fullsized': '2024-04-16T16:18:15.280567+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-16T16:18:21.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.10 is not active+clean 2024-04-16T16:18:21.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.295248+0000', 'last_change': '2024-04-16T16:18:15.295248+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.295248+0000', 'last_undegraded': '2024-04-16T16:18:15.295248+0000', 'last_fullsized': '2024-04-16T16:18:15.295248+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-16T16:18:21.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.11 is not active+clean 2024-04-16T16:18:21.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.287330+0000', 'last_change': '2024-04-16T16:18:15.287330+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.287330+0000', 'last_undegraded': '2024-04-16T16:18:15.287330+0000', 'last_fullsized': '2024-04-16T16:18:15.287330+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-16T16:18:21.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.e is not active+clean 2024-04-16T16:18:21.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.280762+0000', 'last_change': '2024-04-16T16:18:15.280762+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.280762+0000', 'last_undegraded': '2024-04-16T16:18:15.280762+0000', 'last_fullsized': '2024-04-16T16:18:15.280762+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-16T16:18:21.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.f is not active+clean 2024-04-16T16:18:21.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.296848+0000', 'last_change': '2024-04-16T16:18:15.296848+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.296848+0000', 'last_undegraded': '2024-04-16T16:18:15.296848+0000', 'last_fullsized': '2024-04-16T16:18:15.296848+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:18:21.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.d is not active+clean 2024-04-16T16:18:21.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.289641+0000', 'last_change': '2024-04-16T16:18:15.289641+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.289641+0000', 'last_undegraded': '2024-04-16T16:18:15.289641+0000', 'last_fullsized': '2024-04-16T16:18:15.289641+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-16T16:18:21.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.b is not active+clean 2024-04-16T16:18:21.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.298204+0000', 'last_change': '2024-04-16T16:18:15.298204+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.298204+0000', 'last_undegraded': '2024-04-16T16:18:15.298204+0000', 'last_fullsized': '2024-04-16T16:18:15.298204+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-16T16:18:21.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.9 is not active+clean 2024-04-16T16:18:21.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.293789+0000', 'last_change': '2024-04-16T16:18:15.293789+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.293789+0000', 'last_undegraded': '2024-04-16T16:18:15.293789+0000', 'last_fullsized': '2024-04-16T16:18:15.293789+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-16T16:18:21.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.6 is not active+clean 2024-04-16T16:18:21.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.289712+0000', 'last_change': '2024-04-16T16:18:15.289712+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.289712+0000', 'last_undegraded': '2024-04-16T16:18:15.289712+0000', 'last_fullsized': '2024-04-16T16:18:15.289712+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-16T16:18:21.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.7 is not active+clean 2024-04-16T16:18:21.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.283292+0000', 'last_change': '2024-04-16T16:18:15.283292+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.283292+0000', 'last_undegraded': '2024-04-16T16:18:15.283292+0000', 'last_fullsized': '2024-04-16T16:18:15.283292+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-16T16:18:21.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.4 is not active+clean 2024-04-16T16:18:21.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.302671+0000', 'last_change': '2024-04-16T16:18:15.302671+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.302671+0000', 'last_undegraded': '2024-04-16T16:18:15.302671+0000', 'last_fullsized': '2024-04-16T16:18:15.302671+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:18:21.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.5 is not active+clean 2024-04-16T16:18:21.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.288574+0000', 'last_change': '2024-04-16T16:18:15.288574+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.288574+0000', 'last_undegraded': '2024-04-16T16:18:15.288574+0000', 'last_fullsized': '2024-04-16T16:18:15.288574+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-16T16:18:21.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2 is not active+clean 2024-04-16T16:18:21.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.292047+0000', 'last_change': '2024-04-16T16:18:15.292047+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.292047+0000', 'last_undegraded': '2024-04-16T16:18:15.292047+0000', 'last_fullsized': '2024-04-16T16:18:15.292047+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-16T16:18:21.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.0 is not active+clean 2024-04-16T16:18:21.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.281611+0000', 'last_change': '2024-04-16T16:18:15.281611+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.281611+0000', 'last_undegraded': '2024-04-16T16:18:15.281611+0000', 'last_fullsized': '2024-04-16T16:18:15.281611+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-16T16:18:21.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1 is not active+clean 2024-04-16T16:18:21.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.296023+0000', 'last_change': '2024-04-16T16:18:15.296023+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.296023+0000', 'last_undegraded': '2024-04-16T16:18:15.296023+0000', 'last_fullsized': '2024-04-16T16:18:15.296023+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:18:21.918 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.c is not active+clean 2024-04-16T16:18:21.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 83, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:18:15.296266+0000', 'last_change': '2024-04-16T16:18:15.296266+0000', 'last_active': '2024-04-16T16:18:15.256054+0000', 'last_peered': '2024-04-16T16:18:15.256054+0000', 'last_clean': '2024-04-16T16:18:15.256054+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:18:15.296266+0000', 'last_undegraded': '2024-04-16T16:18:15.296266+0000', 'last_fullsized': '2024-04-16T16:18:15.296266+0000', 'mapping_epoch': 83, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'last_clean_scrub_stamp': '2024-04-16T16:18:15.256054+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-16T16:18:21.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T16:18:22.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.169+0000 7fe001e00640 1 Processor -- start 2024-04-16T16:18:22.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.170+0000 7fe001e00640 1 -- start start 2024-04-16T16:18:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.171+0000 7fe001e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdffc0cde00 0x7fdffc0cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.171+0000 7fe001e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdffc163010 0x7fdffc06f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.171+0000 7fe001e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdffc0ce1e0 0x7fdffc06df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.171+0000 7fe001e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fdffc163510 con 0x7fdffc0ce1e0 2024-04-16T16:18:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.171+0000 7fe001e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fdffc1637f0 con 0x7fdffc163010 2024-04-16T16:18:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.171+0000 7fe001e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdffc163ad0 con 0x7fdffc0cde00 2024-04-16T16:18:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.171+0000 7fe001400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdffc0ce1e0 0x7fdffc06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.172+0000 7fe001400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdffc0ce1e0 0x7fdffc06df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57338/0 (socket says 172.21.15.45:57338) 2024-04-16T16:18:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.172+0000 7fe001400640 1 -- 172.21.15.45:0/2900980447 learned_addr learned my addr 172.21.15.45:0/2900980447 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.172+0000 7fdffbe00640 1 --2- 172.21.15.45:0/2900980447 >> v2:172.21.15.103:3300/0 conn(0x7fdffc163010 0x7fdffc06f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.172+0000 7fe000a00640 1 --2- 172.21.15.45:0/2900980447 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdffc0cde00 0x7fdffc0cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.172+0000 7fe001400640 1 -- 172.21.15.45:0/2900980447 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdffc0cde00 msgr2=0x7fdffc0cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.172+0000 7fe001400640 1 --2- 172.21.15.45:0/2900980447 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdffc0cde00 0x7fdffc0cff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.172+0000 7fe001400640 1 -- 172.21.15.45:0/2900980447 >> v2:172.21.15.103:3300/0 conn(0x7fdffc163010 msgr2=0x7fdffc06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.172+0000 7fe001400640 1 --2- 172.21.15.45:0/2900980447 >> v2:172.21.15.103:3300/0 conn(0x7fdffc163010 0x7fdffc06f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.172+0000 7fe001400640 1 -- 172.21.15.45:0/2900980447 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdffc163db0 con 0x7fdffc0ce1e0 2024-04-16T16:18:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.172+0000 7fdffbe00640 1 --2- 172.21.15.45:0/2900980447 >> v2:172.21.15.103:3300/0 conn(0x7fdffc163010 0x7fdffc06f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:18:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.174+0000 7fe001400640 1 --2- 172.21.15.45:0/2900980447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdffc0ce1e0 0x7fdffc06df40 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fdffc070240 tx=0x7fdff402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9050a8cefdef4ea server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.174+0000 7fdffb400640 1 -- 172.21.15.45:0/2900980447 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff402ca70 con 0x7fdffc0ce1e0 2024-04-16T16:18:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.174+0000 7fdffb400640 1 -- 172.21.15.45:0/2900980447 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdff402cc10 con 0x7fdffc0ce1e0 2024-04-16T16:18:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.174+0000 7fdffb400640 1 -- 172.21.15.45:0/2900980447 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff40343f0 con 0x7fdffc0ce1e0 2024-04-16T16:18:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.175+0000 7fe001e00640 1 -- 172.21.15.45:0/2900980447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdffc0ce1e0 msgr2=0x7fdffc06df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.175+0000 7fe001e00640 1 --2- 172.21.15.45:0/2900980447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdffc0ce1e0 0x7fdffc06df40 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fdffc070240 tx=0x7fdff402bed0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.175+0000 7fe001e00640 1 -- 172.21.15.45:0/2900980447 shutdown_connections 2024-04-16T16:18:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.175+0000 7fe001e00640 1 --2- 172.21.15.45:0/2900980447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdffc0ce1e0 0x7fdffc06df40 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.175+0000 7fe001e00640 1 --2- 172.21.15.45:0/2900980447 >> v2:172.21.15.103:3300/0 conn(0x7fdffc163010 0x7fdffc06f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.175+0000 7fe001e00640 1 --2- 172.21.15.45:0/2900980447 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdffc0cde00 0x7fdffc0cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.175+0000 7fe001e00640 1 -- 172.21.15.45:0/2900980447 >> 172.21.15.45:0/2900980447 conn(0x7fdffc06b930 msgr2=0x7fdffc06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.175+0000 7fe001e00640 1 -- 172.21.15.45:0/2900980447 shutdown_connections 2024-04-16T16:18:22.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.175+0000 7fe001e00640 1 -- 172.21.15.45:0/2900980447 wait complete. 2024-04-16T16:18:22.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.177+0000 7fe001e00640 1 Processor -- start 2024-04-16T16:18:22.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.177+0000 7fe001e00640 1 -- start start 2024-04-16T16:18:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.178+0000 7fe001e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdffc0cde00 0x7fdffc156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.178+0000 7fe001e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdffc0ce1e0 0x7fdffc159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.178+0000 7fe001e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdffc163010 0x7fdffc1480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.178+0000 7fe001e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc0ce7b0 con 0x7fdffc163010 2024-04-16T16:18:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.178+0000 7fe001e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc0ce630 con 0x7fdffc0cde00 2024-04-16T16:18:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.178+0000 7fe001e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdffc06e480 con 0x7fdffc0ce1e0 2024-04-16T16:18:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.178+0000 7fe001400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdffc163010 0x7fdffc1480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.178+0000 7fe001400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdffc163010 0x7fdffc1480b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57340/0 (socket says 172.21.15.45:57340) 2024-04-16T16:18:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.178+0000 7fe001400640 1 -- 172.21.15.45:0/2153596852 learned_addr learned my addr 172.21.15.45:0/2153596852 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.178+0000 7fe000a00640 1 --2- 172.21.15.45:0/2153596852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdffc0cde00 0x7fdffc156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.178+0000 7fdffbe00640 1 --2- 172.21.15.45:0/2153596852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdffc0ce1e0 0x7fdffc159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.179+0000 7fe001400640 1 -- 172.21.15.45:0/2153596852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdffc0ce1e0 msgr2=0x7fdffc159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.179+0000 7fe001400640 1 --2- 172.21.15.45:0/2153596852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdffc0ce1e0 0x7fdffc159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.179+0000 7fe001400640 1 -- 172.21.15.45:0/2153596852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdffc0cde00 msgr2=0x7fdffc156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.179+0000 7fe001400640 1 --2- 172.21.15.45:0/2153596852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdffc0cde00 0x7fdffc156b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.179+0000 7fe001400640 1 -- 172.21.15.45:0/2153596852 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdffc1f8ca0 con 0x7fdffc163010 2024-04-16T16:18:22.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.179+0000 7fe001400640 1 --2- 172.21.15.45:0/2153596852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdffc163010 0x7fdffc1480b0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fdff4036040 tx=0x7fdff4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:22.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.179+0000 7fdff8c00640 1 -- 172.21.15.45:0/2153596852 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4043020 con 0x7fdffc163010 2024-04-16T16:18:22.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.179+0000 7fdff8c00640 1 -- 172.21.15.45:0/2153596852 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdff403e450 con 0x7fdffc163010 2024-04-16T16:18:22.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.179+0000 7fdff8c00640 1 -- 172.21.15.45:0/2153596852 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4009510 con 0x7fdffc163010 2024-04-16T16:18:22.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.179+0000 7fe001e00640 1 -- 172.21.15.45:0/2153596852 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdffc1485f0 con 0x7fdffc163010 2024-04-16T16:18:22.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.180+0000 7fe001e00640 1 -- 172.21.15.45:0/2153596852 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdffc15a8d0 con 0x7fdffc163010 2024-04-16T16:18:22.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.181+0000 7fe001e00640 1 -- 172.21.15.45:0/2153596852 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdffc075780 con 0x7fdffc163010 2024-04-16T16:18:22.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.182+0000 7fdff8c00640 1 -- 172.21.15.45:0/2153596852 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7fdff4046020 con 0x7fdffc163010 2024-04-16T16:18:22.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.183+0000 7fdff8c00640 1 --2- 172.21.15.45:0/2153596852 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fdfdc06ca10 0x7fdfdc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:22.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.183+0000 7fdff8c00640 1 -- 172.21.15.45:0/2153596852 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7fdff40ae560 con 0x7fdffc163010 2024-04-16T16:18:22.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.183+0000 7fe000a00640 1 --2- 172.21.15.45:0/2153596852 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fdfdc06ca10 0x7fdfdc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.185+0000 7fe000a00640 1 --2- 172.21.15.45:0/2153596852 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fdfdc06ca10 0x7fdfdc06eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdff0006fd0 tx=0x7fdff0008040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:22.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.188+0000 7fdff8c00640 1 -- 172.21.15.45:0/2153596852 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdff404d030 con 0x7fdffc163010 2024-04-16T16:18:22.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.474+0000 7fe001e00640 1 -- 172.21.15.45:0/2153596852 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fdffc06fc20 con 0x7fdffc163010 2024-04-16T16:18:22.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.474+0000 7fdff8c00640 1 -- 172.21.15.45:0/2153596852 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+3291 (secure 0 0 0) 0x7fdff407c870 con 0x7fdffc163010 2024-04-16T16:18:22.473 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:22.474 INFO:teuthology.orchestra.run.smithi045.stdout:{"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":316,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":85,"num_osds":8,"num_up_osds":8,"osd_up_since":1713283997,"num_in_osds":8,"osd_in_since":1713283989,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":49},{"state_name":"creating+peering","count":48}],"num_pgs":97,"num_pools":3,"num_objects":10,"data_bytes":590368,"bytes_used":264859648,"bytes_avail":772829253632,"bytes_total":773094113280,"inactive_pgs_ratio":0.49484536051750183,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":3},"fsmap":{"epoch":64,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"a","status":"up:active","gid":7188}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-16T16:16:25.060027+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"y":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"6":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"7":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-16T16:18:22.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.477+0000 7fe001e00640 1 -- 172.21.15.45:0/2153596852 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fdfdc06ca10 msgr2=0x7fdfdc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:22.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.477+0000 7fe001e00640 1 --2- 172.21.15.45:0/2153596852 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fdfdc06ca10 0x7fdfdc06eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdff0006fd0 tx=0x7fdff0008040 comp rx=0 tx=0).stop 2024-04-16T16:18:22.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.477+0000 7fe001e00640 1 -- 172.21.15.45:0/2153596852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdffc163010 msgr2=0x7fdffc1480b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:22.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.477+0000 7fe001e00640 1 --2- 172.21.15.45:0/2153596852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdffc163010 0x7fdffc1480b0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fdff4036040 tx=0x7fdff4004a40 comp rx=0 tx=0).stop 2024-04-16T16:18:22.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.478+0000 7fe001e00640 1 -- 172.21.15.45:0/2153596852 shutdown_connections 2024-04-16T16:18:22.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.478+0000 7fe001e00640 1 --2- 172.21.15.45:0/2153596852 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fdfdc06ca10 0x7fdfdc06eed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.478+0000 7fe001e00640 1 --2- 172.21.15.45:0/2153596852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdffc163010 0x7fdffc1480b0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.478+0000 7fe001e00640 1 --2- 172.21.15.45:0/2153596852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdffc0ce1e0 0x7fdffc159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.478+0000 7fe001e00640 1 --2- 172.21.15.45:0/2153596852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdffc0cde00 0x7fdffc156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.478+0000 7fe001e00640 1 -- 172.21.15.45:0/2153596852 >> 172.21.15.45:0/2153596852 conn(0x7fdffc06b930 msgr2=0x7fdffc1653f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:22.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.478+0000 7fe001e00640 1 -- 172.21.15.45:0/2153596852 shutdown_connections 2024-04-16T16:18:22.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.478+0000 7fe001e00640 1 -- 172.21.15.45:0/2153596852 wait complete. 2024-04-16T16:18:22.495 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-16T16:18:22.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:18:22.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.716+0000 7f0c46a00640 1 Processor -- start 2024-04-16T16:18:22.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.716+0000 7f0c46a00640 1 -- start start 2024-04-16T16:18:22.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.718+0000 7f0c46a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0c40144ff0 0x7f0c400cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:22.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.718+0000 7f0c46a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c4015c700 con 0x7f0c4006dfb0 2024-04-16T16:18:22.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.718+0000 7f0c46a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c4015c9c0 con 0x7f0c400c5d80 2024-04-16T16:18:22.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.718+0000 7f0c46a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0c4015d2f0 con 0x7f0c40144ff0 2024-04-16T16:18:22.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.719+0000 7f0c44c00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0c4006dfb0 0x7f0c400c5670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59352/0 (socket says 172.21.15.45:59352) 2024-04-16T16:18:22.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.719+0000 7f0c44c00640 1 -- 172.21.15.45:0/3534755111 learned_addr learned my addr 172.21.15.45:0/3534755111 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:22.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.719+0000 7f0c45600640 1 --2- 172.21.15.45:0/3534755111 >> v2:172.21.15.103:3300/0 conn(0x7f0c40144ff0 0x7f0c400cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:22.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.719+0000 7f0c3fe00640 1 -- 172.21.15.45:0/3534755111 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 768273172 0 0) 0x7f0c4015c700 con 0x7f0c4006dfb0 2024-04-16T16:18:22.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.719+0000 7f0c3fe00640 1 -- 172.21.15.45:0/3534755111 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c28003520 con 0x7f0c4006dfb0 2024-04-16T16:18:22.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.720+0000 7f0c45600640 1 -- 172.21.15.45:0/3534755111 >> v1:172.21.15.103:6789/0 conn(0x7f0c400c5d80 legacy=0x7f0c4015d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:22.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.720+0000 7f0c3fe00640 1 -- 172.21.15.45:0/3534755111 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 444569655 0 0) 0x7f0c28003520 con 0x7f0c4006dfb0 2024-04-16T16:18:22.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.720+0000 7f0c45600640 1 -- 172.21.15.45:0/3534755111 >> v1:172.21.15.45:6789/0 conn(0x7f0c4006dfb0 legacy=0x7f0c400c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:22.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.720+0000 7f0c45600640 1 -- 172.21.15.45:0/3534755111 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c4015cca0 con 0x7f0c40144ff0 2024-04-16T16:18:22.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.721+0000 7f0c45600640 1 --2- 172.21.15.45:0/3534755111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c40144ff0 0x7f0c400cfd30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0c4015d470 tx=0x7f0c3402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c766324c6f4b0ca3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:22.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.722+0000 7f0c3fe00640 1 -- 172.21.15.45:0/3534755111 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34038070 con 0x7f0c40144ff0 2024-04-16T16:18:22.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.722+0000 7f0c3fe00640 1 -- 172.21.15.45:0/3534755111 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c34031500 con 0x7f0c40144ff0 2024-04-16T16:18:22.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.722+0000 7f0c3fe00640 1 -- 172.21.15.45:0/3534755111 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c001670 con 0x7f0c40144ff0 2024-04-16T16:18:22.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.723+0000 7f0c46a00640 1 -- 172.21.15.45:0/3534755111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c40144ff0 msgr2=0x7f0c400cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:22.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.723+0000 7f0c46a00640 1 --2- 172.21.15.45:0/3534755111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c40144ff0 0x7f0c400cfd30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0c4015d470 tx=0x7f0c3402b820 comp rx=0 tx=0).stop 2024-04-16T16:18:22.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.723+0000 7f0c46a00640 1 -- 172.21.15.45:0/3534755111 shutdown_connections 2024-04-16T16:18:22.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.723+0000 7f0c46a00640 1 --2- 172.21.15.45:0/3534755111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c40144ff0 0x7f0c400cfd30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.723+0000 7f0c46a00640 1 -- 172.21.15.45:0/3534755111 >> 172.21.15.45:0/3534755111 conn(0x7f0c4006b930 msgr2=0x7f0c4006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:22.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.723+0000 7f0c46a00640 1 -- 172.21.15.45:0/3534755111 shutdown_connections 2024-04-16T16:18:22.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.723+0000 7f0c46a00640 1 -- 172.21.15.45:0/3534755111 wait complete. 2024-04-16T16:18:22.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.725+0000 7f0c46a00640 1 Processor -- start 2024-04-16T16:18:22.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.725+0000 7f0c46a00640 1 -- start start 2024-04-16T16:18:22.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.726+0000 7f0c46a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c4006dfb0 0x7f0c4015b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:22.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.726+0000 7f0c46a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c400c5d80 0x7f0c400cde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:22.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.726+0000 7f0c46a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c40144ff0 0x7f0c400ce350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:22.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.726+0000 7f0c46a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0c400d0270 con 0x7f0c4006dfb0 2024-04-16T16:18:22.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.726+0000 7f0c46a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0c401453d0 con 0x7f0c40144ff0 2024-04-16T16:18:22.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.726+0000 7f0c46a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0c400d03f0 con 0x7f0c400c5d80 2024-04-16T16:18:22.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.726+0000 7f0c45600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c4006dfb0 0x7f0c4015b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:22.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.726+0000 7f0c45600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c4006dfb0 0x7f0c4015b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57344/0 (socket says 172.21.15.45:57344) 2024-04-16T16:18:22.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.726+0000 7f0c45600640 1 -- 172.21.15.45:0/2563270005 learned_addr learned my addr 172.21.15.45:0/2563270005 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:22.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.726+0000 7f0c45600640 1 -- 172.21.15.45:0/2563270005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c400c5d80 msgr2=0x7f0c400cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:22.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.726+0000 7f0c44c00640 1 --2- 172.21.15.45:0/2563270005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c400c5d80 0x7f0c400cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:22.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.726+0000 7f0c46000640 1 --2- 172.21.15.45:0/2563270005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c40144ff0 0x7f0c400ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:22.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.726+0000 7f0c45600640 1 --2- 172.21.15.45:0/2563270005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c400c5d80 0x7f0c400cde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.726+0000 7f0c45600640 1 -- 172.21.15.45:0/2563270005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c40144ff0 msgr2=0x7f0c400ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:22.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.727+0000 7f0c45600640 1 --2- 172.21.15.45:0/2563270005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c40144ff0 0x7f0c400ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.727+0000 7f0c45600640 1 -- 172.21.15.45:0/2563270005 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c4015b560 con 0x7f0c4006dfb0 2024-04-16T16:18:22.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.727+0000 7f0c46000640 1 --2- 172.21.15.45:0/2563270005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c40144ff0 0x7f0c400ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:22.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.727+0000 7f0c45600640 1 --2- 172.21.15.45:0/2563270005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c4006dfb0 0x7f0c4015b020 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f0c34036040 tx=0x7f0c340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:22.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.727+0000 7f0c3d600640 1 -- 172.21.15.45:0/2563270005 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34043070 con 0x7f0c4006dfb0 2024-04-16T16:18:22.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.727+0000 7f0c3d600640 1 -- 172.21.15.45:0/2563270005 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c3403f870 con 0x7f0c4006dfb0 2024-04-16T16:18:22.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.727+0000 7f0c3d600640 1 -- 172.21.15.45:0/2563270005 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34038040 con 0x7f0c4006dfb0 2024-04-16T16:18:22.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.727+0000 7f0c46a00640 1 -- 172.21.15.45:0/2563270005 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c40139c50 con 0x7f0c4006dfb0 2024-04-16T16:18:22.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.728+0000 7f0c46a00640 1 -- 172.21.15.45:0/2563270005 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c4013a210 con 0x7f0c4006dfb0 2024-04-16T16:18:22.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.729+0000 7f0c46a00640 1 -- 172.21.15.45:0/2563270005 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c40075780 con 0x7f0c4006dfb0 2024-04-16T16:18:22.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.730+0000 7f0c3d600640 1 -- 172.21.15.45:0/2563270005 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f0c34046070 con 0x7f0c4006dfb0 2024-04-16T16:18:22.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.731+0000 7f0c3d600640 1 --2- 172.21.15.45:0/2563270005 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f0c2006ca10 0x7f0c2006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:22.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.731+0000 7f0c3d600640 1 -- 172.21.15.45:0/2563270005 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f0c340adad0 con 0x7f0c4006dfb0 2024-04-16T16:18:22.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.732+0000 7f0c44c00640 1 --2- 172.21.15.45:0/2563270005 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f0c2006ca10 0x7f0c2006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:22.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.733+0000 7f0c44c00640 1 --2- 172.21.15.45:0/2563270005 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f0c2006ca10 0x7f0c2006eed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0017e0 tx=0x7f0c2c009040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:22.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.736+0000 7f0c3d600640 1 -- 172.21.15.45:0/2563270005 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0c4015b560 con 0x7f0c4006dfb0 2024-04-16T16:18:22.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.939+0000 7f0c46a00640 1 -- 172.21.15.45:0/2563270005 --> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0c400b59c0 con 0x7f0c2006ca10 2024-04-16T16:18:22.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.959+0000 7f0c3d600640 1 -- 172.21.15.45:0/2563270005 <== mgr.5791 v2:172.21.15.103:6832/3728775368 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+248113 (secure 0 0 0) 0x7f0c400b59c0 con 0x7f0c2006ca10 2024-04-16T16:18:22.960 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:22.962 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:18:22.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.965+0000 7f0c46a00640 1 -- 172.21.15.45:0/2563270005 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f0c2006ca10 msgr2=0x7f0c2006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:22.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.965+0000 7f0c46a00640 1 --2- 172.21.15.45:0/2563270005 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f0c2006ca10 0x7f0c2006eed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0017e0 tx=0x7f0c2c009040 comp rx=0 tx=0).stop 2024-04-16T16:18:22.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.965+0000 7f0c46a00640 1 -- 172.21.15.45:0/2563270005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c4006dfb0 msgr2=0x7f0c4015b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:22.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.965+0000 7f0c46a00640 1 --2- 172.21.15.45:0/2563270005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c4006dfb0 0x7f0c4015b020 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f0c34036040 tx=0x7f0c340046c0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.965+0000 7f0c46a00640 1 -- 172.21.15.45:0/2563270005 shutdown_connections 2024-04-16T16:18:22.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.965+0000 7f0c46a00640 1 --2- 172.21.15.45:0/2563270005 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f0c2006ca10 0x7f0c2006eed0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.965+0000 7f0c46a00640 1 --2- 172.21.15.45:0/2563270005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c40144ff0 0x7f0c400ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.965+0000 7f0c46a00640 1 --2- 172.21.15.45:0/2563270005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c400c5d80 0x7f0c400cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.965+0000 7f0c46a00640 1 --2- 172.21.15.45:0/2563270005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c4006dfb0 0x7f0c4015b020 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:22.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.965+0000 7f0c46a00640 1 -- 172.21.15.45:0/2563270005 >> 172.21.15.45:0/2563270005 conn(0x7f0c4006b930 msgr2=0x7f0c4006e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:22.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.965+0000 7f0c46a00640 1 -- 172.21.15.45:0/2563270005 shutdown_connections 2024-04-16T16:18:22.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:22.965+0000 7f0c46a00640 1 -- 172.21.15.45:0/2563270005 wait complete. 2024-04-16T16:18:22.983 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-16T16:18:21.119366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":261,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258652,"kb_used_data":9196,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716068,"statfs":{"total":773094113280,"available":772829253632,"internally_reserved":0,"allocated":9416704,"data_stored":5036287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002211"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.298782+0000","last_change":"2024-04-16T16:18:15.298782+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.298782+0000","last_undegraded":"2024-04-16T16:18:15.298782+0000","last_fullsized":"2024-04-16T16:18:15.298782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293043+0000","last_change":"2024-04-16T16:18:15.293043+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293043+0000","last_undegraded":"2024-04-16T16:18:15.293043+0000","last_fullsized":"2024-04-16T16:18:15.293043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.284139+0000","last_change":"2024-04-16T16:18:15.284139+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.284139+0000","last_undegraded":"2024-04-16T16:18:15.284139+0000","last_fullsized":"2024-04-16T16:18:15.284139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312806+0000","last_change":"2024-04-16T16:18:16.316835+0000","last_active":"2024-04-16T16:18:17.312806+0000","last_peered":"2024-04-16T16:18:17.312806+0000","last_clean":"2024-04-16T16:18:17.312806+0000","last_became_active":"2024-04-16T16:18:16.309649+0000","last_became_peered":"2024-04-16T16:18:16.309649+0000","last_unstale":"2024-04-16T16:18:17.312806+0000","last_undegraded":"2024-04-16T16:18:17.312806+0000","last_fullsized":"2024-04-16T16:18:17.312806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:21:05.656211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289497+0000","last_change":"2024-04-16T16:18:15.289497+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289497+0000","last_undegraded":"2024-04-16T16:18:15.289497+0000","last_fullsized":"2024-04-16T16:18:15.289497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.292510+0000","last_change":"2024-04-16T16:18:15.292510+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.292510+0000","last_undegraded":"2024-04-16T16:18:15.292510+0000","last_fullsized":"2024-04-16T16:18:15.292510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312314+0000","last_change":"2024-04-16T16:18:16.308558+0000","last_active":"2024-04-16T16:18:17.312314+0000","last_peered":"2024-04-16T16:18:17.312314+0000","last_clean":"2024-04-16T16:18:17.312314+0000","last_became_active":"2024-04-16T16:18:16.308268+0000","last_became_peered":"2024-04-16T16:18:16.308268+0000","last_unstale":"2024-04-16T16:18:17.312314+0000","last_undegraded":"2024-04-16T16:18:17.312314+0000","last_fullsized":"2024-04-16T16:18:17.312314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:13:57.589683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.291320+0000","last_change":"2024-04-16T16:18:15.291320+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.291320+0000","last_undegraded":"2024-04-16T16:18:15.291320+0000","last_fullsized":"2024-04-16T16:18:15.291320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.297496+0000","last_change":"2024-04-16T16:18:15.297496+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.297496+0000","last_undegraded":"2024-04-16T16:18:15.297496+0000","last_fullsized":"2024-04-16T16:18:15.297496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.295832+0000","last_change":"2024-04-16T16:18:15.295832+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.295832+0000","last_undegraded":"2024-04-16T16:18:15.295832+0000","last_fullsized":"2024-04-16T16:18:15.295832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.291161+0000","last_change":"2024-04-16T16:18:15.291161+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.291161+0000","last_undegraded":"2024-04-16T16:18:15.291161+0000","last_fullsized":"2024-04-16T16:18:15.291161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.304631+0000","last_change":"2024-04-16T16:18:15.304631+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.304631+0000","last_undegraded":"2024-04-16T16:18:15.304631+0000","last_fullsized":"2024-04-16T16:18:15.304631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.281906+0000","last_change":"2024-04-16T16:18:15.281906+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.281906+0000","last_undegraded":"2024-04-16T16:18:15.281906+0000","last_fullsized":"2024-04-16T16:18:15.281906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.313189+0000","last_change":"2024-04-16T16:18:16.321879+0000","last_active":"2024-04-16T16:18:17.313189+0000","last_peered":"2024-04-16T16:18:17.313189+0000","last_clean":"2024-04-16T16:18:17.313189+0000","last_became_active":"2024-04-16T16:18:16.310779+0000","last_became_peered":"2024-04-16T16:18:16.310779+0000","last_unstale":"2024-04-16T16:18:17.313189+0000","last_undegraded":"2024-04-16T16:18:17.313189+0000","last_fullsized":"2024-04-16T16:18:17.313189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:46:46.691832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394765+0000","last_change":"2024-04-16T16:18:16.314327+0000","last_active":"2024-04-16T16:18:17.394765+0000","last_peered":"2024-04-16T16:18:17.394765+0000","last_clean":"2024-04-16T16:18:17.394765+0000","last_became_active":"2024-04-16T16:18:16.313983+0000","last_became_peered":"2024-04-16T16:18:16.313983+0000","last_unstale":"2024-04-16T16:18:17.394765+0000","last_undegraded":"2024-04-16T16:18:17.394765+0000","last_fullsized":"2024-04-16T16:18:17.394765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:06:33.666684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.287619+0000","last_change":"2024-04-16T16:18:15.287619+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.287619+0000","last_undegraded":"2024-04-16T16:18:15.287619+0000","last_fullsized":"2024-04-16T16:18:15.287619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.290484+0000","last_change":"2024-04-16T16:18:15.290484+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.290484+0000","last_undegraded":"2024-04-16T16:18:15.290484+0000","last_fullsized":"2024-04-16T16:18:15.290484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312691+0000","last_change":"2024-04-16T16:18:16.314027+0000","last_active":"2024-04-16T16:18:17.312691+0000","last_peered":"2024-04-16T16:18:17.312691+0000","last_clean":"2024-04-16T16:18:17.312691+0000","last_became_active":"2024-04-16T16:18:16.313426+0000","last_became_peered":"2024-04-16T16:18:16.313426+0000","last_unstale":"2024-04-16T16:18:17.312691+0000","last_undegraded":"2024-04-16T16:18:17.312691+0000","last_fullsized":"2024-04-16T16:18:17.312691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:30:17.865648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444884+0000","last_change":"2024-04-16T16:18:16.305406+0000","last_active":"2024-04-16T16:18:17.444884+0000","last_peered":"2024-04-16T16:18:17.444884+0000","last_clean":"2024-04-16T16:18:17.444884+0000","last_became_active":"2024-04-16T16:18:16.305023+0000","last_became_peered":"2024-04-16T16:18:16.305023+0000","last_unstale":"2024-04-16T16:18:17.444884+0000","last_undegraded":"2024-04-16T16:18:17.444884+0000","last_fullsized":"2024-04-16T16:18:17.444884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:20.251382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312975+0000","last_change":"2024-04-16T16:18:16.317943+0000","last_active":"2024-04-16T16:18:17.312975+0000","last_peered":"2024-04-16T16:18:17.312975+0000","last_clean":"2024-04-16T16:18:17.312975+0000","last_became_active":"2024-04-16T16:18:16.317474+0000","last_became_peered":"2024-04-16T16:18:16.317474+0000","last_unstale":"2024-04-16T16:18:17.312975+0000","last_undegraded":"2024-04-16T16:18:17.312975+0000","last_fullsized":"2024-04-16T16:18:17.312975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:52:31.379753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312792+0000","last_change":"2024-04-16T16:18:16.300164+0000","last_active":"2024-04-16T16:18:17.312792+0000","last_peered":"2024-04-16T16:18:17.312792+0000","last_clean":"2024-04-16T16:18:17.312792+0000","last_became_active":"2024-04-16T16:18:16.299838+0000","last_became_peered":"2024-04-16T16:18:16.299838+0000","last_unstale":"2024-04-16T16:18:17.312792+0000","last_undegraded":"2024-04-16T16:18:17.312792+0000","last_fullsized":"2024-04-16T16:18:17.312792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:22.876310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296503+0000","last_change":"2024-04-16T16:18:15.296503+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296503+0000","last_undegraded":"2024-04-16T16:18:15.296503+0000","last_fullsized":"2024-04-16T16:18:15.296503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296468+0000","last_change":"2024-04-16T16:18:15.296468+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296468+0000","last_undegraded":"2024-04-16T16:18:15.296468+0000","last_fullsized":"2024-04-16T16:18:15.296468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312382+0000","last_change":"2024-04-16T16:18:16.315372+0000","last_active":"2024-04-16T16:18:17.312382+0000","last_peered":"2024-04-16T16:18:17.312382+0000","last_clean":"2024-04-16T16:18:17.312382+0000","last_became_active":"2024-04-16T16:18:16.315098+0000","last_became_peered":"2024-04-16T16:18:16.315098+0000","last_unstale":"2024-04-16T16:18:17.312382+0000","last_undegraded":"2024-04-16T16:18:17.312382+0000","last_fullsized":"2024-04-16T16:18:17.312382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:22.929509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293145+0000","last_change":"2024-04-16T16:18:15.293145+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293145+0000","last_undegraded":"2024-04-16T16:18:15.293145+0000","last_fullsized":"2024-04-16T16:18:15.293145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.282088+0000","last_change":"2024-04-16T16:18:15.282088+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.282088+0000","last_undegraded":"2024-04-16T16:18:15.282088+0000","last_fullsized":"2024-04-16T16:18:15.282088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.292185+0000","last_change":"2024-04-16T16:18:15.292185+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.292185+0000","last_undegraded":"2024-04-16T16:18:15.292185+0000","last_fullsized":"2024-04-16T16:18:15.292185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.304185+0000","last_change":"2024-04-16T16:18:15.304185+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.304185+0000","last_undegraded":"2024-04-16T16:18:15.304185+0000","last_fullsized":"2024-04-16T16:18:15.304185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289755+0000","last_change":"2024-04-16T16:18:15.289755+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289755+0000","last_undegraded":"2024-04-16T16:18:15.289755+0000","last_fullsized":"2024-04-16T16:18:15.289755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296039+0000","last_change":"2024-04-16T16:18:15.296039+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296039+0000","last_undegraded":"2024-04-16T16:18:15.296039+0000","last_fullsized":"2024-04-16T16:18:15.296039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.287029+0000","last_change":"2024-04-16T16:18:15.287029+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.287029+0000","last_undegraded":"2024-04-16T16:18:15.287029+0000","last_fullsized":"2024-04-16T16:18:15.287029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.302832+0000","last_change":"2024-04-16T16:18:15.302832+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.302832+0000","last_undegraded":"2024-04-16T16:18:15.302832+0000","last_fullsized":"2024-04-16T16:18:15.302832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.291173+0000","last_change":"2024-04-16T16:18:15.291173+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.291173+0000","last_undegraded":"2024-04-16T16:18:15.291173+0000","last_fullsized":"2024-04-16T16:18:15.291173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311763+0000","last_change":"2024-04-16T16:18:15.307894+0000","last_active":"2024-04-16T16:18:15.311763+0000","last_peered":"2024-04-16T16:18:15.311763+0000","last_clean":"2024-04-16T16:18:15.311763+0000","last_became_active":"2024-04-16T16:18:15.307584+0000","last_became_peered":"2024-04-16T16:18:15.307584+0000","last_unstale":"2024-04-16T16:18:15.311763+0000","last_undegraded":"2024-04-16T16:18:15.311763+0000","last_fullsized":"2024-04-16T16:18:15.311763+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:30:36.737096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.297298+0000","last_change":"2024-04-16T16:18:15.297298+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.297298+0000","last_undegraded":"2024-04-16T16:18:15.297298+0000","last_fullsized":"2024-04-16T16:18:15.297298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.303705+0000","last_change":"2024-04-16T16:18:15.299901+0000","last_active":"2024-04-16T16:18:15.303705+0000","last_peered":"2024-04-16T16:18:15.303705+0000","last_clean":"2024-04-16T16:18:15.303705+0000","last_became_active":"2024-04-16T16:18:15.299622+0000","last_became_peered":"2024-04-16T16:18:15.299622+0000","last_unstale":"2024-04-16T16:18:15.303705+0000","last_undegraded":"2024-04-16T16:18:15.303705+0000","last_fullsized":"2024-04-16T16:18:15.303705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:51.934340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.303193+0000","last_change":"2024-04-16T16:18:15.303193+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.303193+0000","last_undegraded":"2024-04-16T16:18:15.303193+0000","last_fullsized":"2024-04-16T16:18:15.303193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444710+0000","last_change":"2024-04-16T16:18:15.297918+0000","last_active":"2024-04-16T16:18:17.444710+0000","last_peered":"2024-04-16T16:18:17.444710+0000","last_clean":"2024-04-16T16:18:17.444710+0000","last_became_active":"2024-04-16T16:18:15.297217+0000","last_became_peered":"2024-04-16T16:18:15.297217+0000","last_unstale":"2024-04-16T16:18:17.444710+0000","last_undegraded":"2024-04-16T16:18:17.444710+0000","last_fullsized":"2024-04-16T16:18:17.444710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:03:48.538913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.283014+0000","last_change":"2024-04-16T16:18:15.283014+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.283014+0000","last_undegraded":"2024-04-16T16:18:15.283014+0000","last_fullsized":"2024-04-16T16:18:15.283014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443826+0000","last_change":"2024-04-16T16:18:15.300207+0000","last_active":"2024-04-16T16:18:17.443826+0000","last_peered":"2024-04-16T16:18:17.443826+0000","last_clean":"2024-04-16T16:18:17.443826+0000","last_became_active":"2024-04-16T16:18:15.297220+0000","last_became_peered":"2024-04-16T16:18:15.297220+0000","last_unstale":"2024-04-16T16:18:17.443826+0000","last_undegraded":"2024-04-16T16:18:17.443826+0000","last_fullsized":"2024-04-16T16:18:17.443826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:58:34.162900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.295251+0000","last_change":"2024-04-16T16:18:15.295251+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.295251+0000","last_undegraded":"2024-04-16T16:18:15.295251+0000","last_fullsized":"2024-04-16T16:18:15.295251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311825+0000","last_change":"2024-04-16T16:18:15.309872+0000","last_active":"2024-04-16T16:18:15.311825+0000","last_peered":"2024-04-16T16:18:15.311825+0000","last_clean":"2024-04-16T16:18:15.311825+0000","last_became_active":"2024-04-16T16:18:15.309546+0000","last_became_peered":"2024-04-16T16:18:15.309546+0000","last_unstale":"2024-04-16T16:18:15.311825+0000","last_undegraded":"2024-04-16T16:18:15.311825+0000","last_fullsized":"2024-04-16T16:18:15.311825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:27:40.938359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.288554+0000","last_change":"2024-04-16T16:18:15.288554+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.288554+0000","last_undegraded":"2024-04-16T16:18:15.288554+0000","last_fullsized":"2024-04-16T16:18:15.288554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.303669+0000","last_change":"2024-04-16T16:18:15.298912+0000","last_active":"2024-04-16T16:18:15.303669+0000","last_peered":"2024-04-16T16:18:15.303669+0000","last_clean":"2024-04-16T16:18:15.303669+0000","last_became_active":"2024-04-16T16:18:15.298638+0000","last_became_peered":"2024-04-16T16:18:15.298638+0000","last_unstale":"2024-04-16T16:18:15.303669+0000","last_undegraded":"2024-04-16T16:18:15.303669+0000","last_fullsized":"2024-04-16T16:18:15.303669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:30:36.737096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.282766+0000","last_change":"2024-04-16T16:18:15.282766+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.282766+0000","last_undegraded":"2024-04-16T16:18:15.282766+0000","last_fullsized":"2024-04-16T16:18:15.282766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444198+0000","last_change":"2024-04-16T16:18:15.301897+0000","last_active":"2024-04-16T16:18:17.444198+0000","last_peered":"2024-04-16T16:18:17.444198+0000","last_clean":"2024-04-16T16:18:17.444198+0000","last_became_active":"2024-04-16T16:18:15.301549+0000","last_became_peered":"2024-04-16T16:18:15.301549+0000","last_unstale":"2024-04-16T16:18:17.444198+0000","last_undegraded":"2024-04-16T16:18:17.444198+0000","last_fullsized":"2024-04-16T16:18:17.444198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:29:16.985875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394783+0000","last_change":"2024-04-16T16:18:16.308198+0000","last_active":"2024-04-16T16:18:17.394783+0000","last_peered":"2024-04-16T16:18:17.394783+0000","last_clean":"2024-04-16T16:18:17.394783+0000","last_became_active":"2024-04-16T16:18:16.307924+0000","last_became_peered":"2024-04-16T16:18:16.307924+0000","last_unstale":"2024-04-16T16:18:17.394783+0000","last_undegraded":"2024-04-16T16:18:17.394783+0000","last_fullsized":"2024-04-16T16:18:17.394783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:34:10.200444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.307302+0000","last_change":"2024-04-16T16:18:15.303661+0000","last_active":"2024-04-16T16:18:15.307302+0000","last_peered":"2024-04-16T16:18:15.307302+0000","last_clean":"2024-04-16T16:18:15.307302+0000","last_became_active":"2024-04-16T16:18:15.300455+0000","last_became_peered":"2024-04-16T16:18:15.300455+0000","last_unstale":"2024-04-16T16:18:15.307302+0000","last_undegraded":"2024-04-16T16:18:15.307302+0000","last_fullsized":"2024-04-16T16:18:15.307302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:21:04.639131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394997+0000","last_change":"2024-04-16T16:18:16.324165+0000","last_active":"2024-04-16T16:18:17.394997+0000","last_peered":"2024-04-16T16:18:17.394997+0000","last_clean":"2024-04-16T16:18:17.394997+0000","last_became_active":"2024-04-16T16:18:16.323865+0000","last_became_peered":"2024-04-16T16:18:16.323865+0000","last_unstale":"2024-04-16T16:18:17.394997+0000","last_undegraded":"2024-04-16T16:18:17.394997+0000","last_fullsized":"2024-04-16T16:18:17.394997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:51:43.715542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311275+0000","last_change":"2024-04-16T16:18:15.305839+0000","last_active":"2024-04-16T16:18:15.311275+0000","last_peered":"2024-04-16T16:18:15.311275+0000","last_clean":"2024-04-16T16:18:15.311275+0000","last_became_active":"2024-04-16T16:18:15.305453+0000","last_became_peered":"2024-04-16T16:18:15.305453+0000","last_unstale":"2024-04-16T16:18:15.311275+0000","last_undegraded":"2024-04-16T16:18:15.311275+0000","last_fullsized":"2024-04-16T16:18:15.311275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:51:24.645998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293391+0000","last_change":"2024-04-16T16:18:15.293391+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293391+0000","last_undegraded":"2024-04-16T16:18:15.293391+0000","last_fullsized":"2024-04-16T16:18:15.293391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311136+0000","last_change":"2024-04-16T16:18:15.304936+0000","last_active":"2024-04-16T16:18:15.311136+0000","last_peered":"2024-04-16T16:18:15.311136+0000","last_clean":"2024-04-16T16:18:15.311136+0000","last_became_active":"2024-04-16T16:18:15.304590+0000","last_became_peered":"2024-04-16T16:18:15.304590+0000","last_unstale":"2024-04-16T16:18:15.311136+0000","last_undegraded":"2024-04-16T16:18:15.311136+0000","last_fullsized":"2024-04-16T16:18:15.311136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:33.215073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443924+0000","last_change":"2024-04-16T16:18:16.316337+0000","last_active":"2024-04-16T16:18:17.443924+0000","last_peered":"2024-04-16T16:18:17.443924+0000","last_clean":"2024-04-16T16:18:17.443924+0000","last_became_active":"2024-04-16T16:18:16.316019+0000","last_became_peered":"2024-04-16T16:18:16.316019+0000","last_unstale":"2024-04-16T16:18:17.443924+0000","last_undegraded":"2024-04-16T16:18:17.443924+0000","last_fullsized":"2024-04-16T16:18:17.443924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:21:33.844543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311896+0000","last_change":"2024-04-16T16:18:15.308173+0000","last_active":"2024-04-16T16:18:15.311896+0000","last_peered":"2024-04-16T16:18:15.311896+0000","last_clean":"2024-04-16T16:18:15.311896+0000","last_became_active":"2024-04-16T16:18:15.307837+0000","last_became_peered":"2024-04-16T16:18:15.307837+0000","last_unstale":"2024-04-16T16:18:15.311896+0000","last_undegraded":"2024-04-16T16:18:15.311896+0000","last_fullsized":"2024-04-16T16:18:15.311896+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:45:40.148360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.290639+0000","last_change":"2024-04-16T16:18:15.290639+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.290639+0000","last_undegraded":"2024-04-16T16:18:15.290639+0000","last_fullsized":"2024-04-16T16:18:15.290639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.309798+0000","last_change":"2024-04-16T16:18:15.306390+0000","last_active":"2024-04-16T16:18:15.309798+0000","last_peered":"2024-04-16T16:18:15.309798+0000","last_clean":"2024-04-16T16:18:15.309798+0000","last_became_active":"2024-04-16T16:18:15.306063+0000","last_became_peered":"2024-04-16T16:18:15.306063+0000","last_unstale":"2024-04-16T16:18:15.309798+0000","last_undegraded":"2024-04-16T16:18:15.309798+0000","last_fullsized":"2024-04-16T16:18:15.309798+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:57.184445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443652+0000","last_change":"2024-04-16T16:18:16.315534+0000","last_active":"2024-04-16T16:18:17.443652+0000","last_peered":"2024-04-16T16:18:17.443652+0000","last_clean":"2024-04-16T16:18:17.443652+0000","last_became_active":"2024-04-16T16:18:16.315301+0000","last_became_peered":"2024-04-16T16:18:16.315301+0000","last_unstale":"2024-04-16T16:18:17.443652+0000","last_undegraded":"2024-04-16T16:18:17.443652+0000","last_fullsized":"2024-04-16T16:18:17.443652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:42.181514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.310987+0000","last_change":"2024-04-16T16:18:15.308221+0000","last_active":"2024-04-16T16:18:15.310987+0000","last_peered":"2024-04-16T16:18:15.310987+0000","last_clean":"2024-04-16T16:18:15.310987+0000","last_became_active":"2024-04-16T16:18:15.307923+0000","last_became_peered":"2024-04-16T16:18:15.307923+0000","last_unstale":"2024-04-16T16:18:15.310987+0000","last_undegraded":"2024-04-16T16:18:15.310987+0000","last_fullsized":"2024-04-16T16:18:15.310987+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:42:36.176215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.280567+0000","last_change":"2024-04-16T16:18:15.280567+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.280567+0000","last_undegraded":"2024-04-16T16:18:15.280567+0000","last_fullsized":"2024-04-16T16:18:15.280567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.395004+0000","last_change":"2024-04-16T16:18:15.309398+0000","last_active":"2024-04-16T16:18:17.395004+0000","last_peered":"2024-04-16T16:18:17.395004+0000","last_clean":"2024-04-16T16:18:17.395004+0000","last_became_active":"2024-04-16T16:18:15.308959+0000","last_became_peered":"2024-04-16T16:18:15.308959+0000","last_unstale":"2024-04-16T16:18:17.395004+0000","last_undegraded":"2024-04-16T16:18:17.395004+0000","last_fullsized":"2024-04-16T16:18:17.395004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:10:27.168721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.295248+0000","last_change":"2024-04-16T16:18:15.295248+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.295248+0000","last_undegraded":"2024-04-16T16:18:15.295248+0000","last_fullsized":"2024-04-16T16:18:15.295248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311722+0000","last_change":"2024-04-16T16:18:15.309922+0000","last_active":"2024-04-16T16:18:15.311722+0000","last_peered":"2024-04-16T16:18:15.311722+0000","last_clean":"2024-04-16T16:18:15.311722+0000","last_became_active":"2024-04-16T16:18:15.308635+0000","last_became_peered":"2024-04-16T16:18:15.308635+0000","last_unstale":"2024-04-16T16:18:15.311722+0000","last_undegraded":"2024-04-16T16:18:15.311722+0000","last_fullsized":"2024-04-16T16:18:15.311722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:05:41.836526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.287330+0000","last_change":"2024-04-16T16:18:15.287330+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.287330+0000","last_undegraded":"2024-04-16T16:18:15.287330+0000","last_fullsized":"2024-04-16T16:18:15.287330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"84'8","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443410+0000","last_change":"2024-04-16T16:18:15.299371+0000","last_active":"2024-04-16T16:18:17.443410+0000","last_peered":"2024-04-16T16:18:17.443410+0000","last_clean":"2024-04-16T16:18:17.443410+0000","last_became_active":"2024-04-16T16:18:15.299047+0000","last_became_peered":"2024-04-16T16:18:15.299047+0000","last_unstale":"2024-04-16T16:18:17.443410+0000","last_undegraded":"2024-04-16T16:18:17.443410+0000","last_fullsized":"2024-04-16T16:18:17.443410+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:14:18.404852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.280762+0000","last_change":"2024-04-16T16:18:15.280762+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.280762+0000","last_undegraded":"2024-04-16T16:18:15.280762+0000","last_fullsized":"2024-04-16T16:18:15.280762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311071+0000","last_change":"2024-04-16T16:18:15.305063+0000","last_active":"2024-04-16T16:18:15.311071+0000","last_peered":"2024-04-16T16:18:15.311071+0000","last_clean":"2024-04-16T16:18:15.311071+0000","last_became_active":"2024-04-16T16:18:15.304669+0000","last_became_peered":"2024-04-16T16:18:15.304669+0000","last_unstale":"2024-04-16T16:18:15.311071+0000","last_undegraded":"2024-04-16T16:18:15.311071+0000","last_fullsized":"2024-04-16T16:18:15.311071+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:59.303188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296848+0000","last_change":"2024-04-16T16:18:15.296848+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296848+0000","last_undegraded":"2024-04-16T16:18:15.296848+0000","last_fullsized":"2024-04-16T16:18:15.296848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443949+0000","last_change":"2024-04-16T16:18:15.302798+0000","last_active":"2024-04-16T16:18:17.443949+0000","last_peered":"2024-04-16T16:18:17.443949+0000","last_clean":"2024-04-16T16:18:17.443949+0000","last_became_active":"2024-04-16T16:18:15.301509+0000","last_became_peered":"2024-04-16T16:18:15.301509+0000","last_unstale":"2024-04-16T16:18:17.443949+0000","last_undegraded":"2024-04-16T16:18:17.443949+0000","last_fullsized":"2024-04-16T16:18:17.443949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:20:56.511072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289641+0000","last_change":"2024-04-16T16:18:15.289641+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289641+0000","last_undegraded":"2024-04-16T16:18:15.289641+0000","last_fullsized":"2024-04-16T16:18:15.289641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443959+0000","last_change":"2024-04-16T16:18:15.302329+0000","last_active":"2024-04-16T16:18:17.443959+0000","last_peered":"2024-04-16T16:18:17.443959+0000","last_clean":"2024-04-16T16:18:17.443959+0000","last_became_active":"2024-04-16T16:18:15.301993+0000","last_became_peered":"2024-04-16T16:18:15.301993+0000","last_unstale":"2024-04-16T16:18:17.443959+0000","last_undegraded":"2024-04-16T16:18:17.443959+0000","last_fullsized":"2024-04-16T16:18:17.443959+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:00:22.780800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.313248+0000","last_change":"2024-04-16T16:18:16.318257+0000","last_active":"2024-04-16T16:18:17.313248+0000","last_peered":"2024-04-16T16:18:17.313248+0000","last_clean":"2024-04-16T16:18:17.313248+0000","last_became_active":"2024-04-16T16:18:16.311533+0000","last_became_peered":"2024-04-16T16:18:16.311533+0000","last_unstale":"2024-04-16T16:18:17.313248+0000","last_undegraded":"2024-04-16T16:18:17.313248+0000","last_fullsized":"2024-04-16T16:18:17.313248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:42:37.193295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394822+0000","last_change":"2024-04-16T16:18:15.291033+0000","last_active":"2024-04-16T16:18:17.394822+0000","last_peered":"2024-04-16T16:18:17.394822+0000","last_clean":"2024-04-16T16:18:17.394822+0000","last_became_active":"2024-04-16T16:18:15.290067+0000","last_became_peered":"2024-04-16T16:18:15.290067+0000","last_unstale":"2024-04-16T16:18:17.394822+0000","last_undegraded":"2024-04-16T16:18:17.394822+0000","last_fullsized":"2024-04-16T16:18:17.394822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:06:23.738648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.298204+0000","last_change":"2024-04-16T16:18:15.298204+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.298204+0000","last_undegraded":"2024-04-16T16:18:15.298204+0000","last_fullsized":"2024-04-16T16:18:15.298204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.313304+0000","last_change":"2024-04-16T16:18:15.301900+0000","last_active":"2024-04-16T16:18:17.313304+0000","last_peered":"2024-04-16T16:18:17.313304+0000","last_clean":"2024-04-16T16:18:17.313304+0000","last_became_active":"2024-04-16T16:18:15.301268+0000","last_became_peered":"2024-04-16T16:18:15.301268+0000","last_unstale":"2024-04-16T16:18:17.313304+0000","last_undegraded":"2024-04-16T16:18:17.313304+0000","last_fullsized":"2024-04-16T16:18:17.313304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:15.061257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312362+0000","last_change":"2024-04-16T16:18:16.314030+0000","last_active":"2024-04-16T16:18:17.312362+0000","last_peered":"2024-04-16T16:18:17.312362+0000","last_clean":"2024-04-16T16:18:17.312362+0000","last_became_active":"2024-04-16T16:18:16.313572+0000","last_became_peered":"2024-04-16T16:18:16.313572+0000","last_unstale":"2024-04-16T16:18:17.312362+0000","last_undegraded":"2024-04-16T16:18:17.312362+0000","last_fullsized":"2024-04-16T16:18:17.312362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:50:11.604716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312426+0000","last_change":"2024-04-16T16:18:15.308430+0000","last_active":"2024-04-16T16:18:17.312426+0000","last_peered":"2024-04-16T16:18:17.312426+0000","last_clean":"2024-04-16T16:18:17.312426+0000","last_became_active":"2024-04-16T16:18:15.308128+0000","last_became_peered":"2024-04-16T16:18:15.308128+0000","last_unstale":"2024-04-16T16:18:17.312426+0000","last_undegraded":"2024-04-16T16:18:17.312426+0000","last_fullsized":"2024-04-16T16:18:17.312426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:23.975082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.293789+0000","last_change":"2024-04-16T16:18:15.293789+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.293789+0000","last_undegraded":"2024-04-16T16:18:15.293789+0000","last_fullsized":"2024-04-16T16:18:15.293789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394296+0000","last_change":"2024-04-16T16:18:15.292003+0000","last_active":"2024-04-16T16:18:17.394296+0000","last_peered":"2024-04-16T16:18:17.394296+0000","last_clean":"2024-04-16T16:18:17.394296+0000","last_became_active":"2024-04-16T16:18:15.291691+0000","last_became_peered":"2024-04-16T16:18:15.291691+0000","last_unstale":"2024-04-16T16:18:17.394296+0000","last_undegraded":"2024-04-16T16:18:17.394296+0000","last_fullsized":"2024-04-16T16:18:17.394296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:28.452108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.289712+0000","last_change":"2024-04-16T16:18:15.289712+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.289712+0000","last_undegraded":"2024-04-16T16:18:15.289712+0000","last_fullsized":"2024-04-16T16:18:15.289712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.308009+0000","last_change":"2024-04-16T16:18:15.305818+0000","last_active":"2024-04-16T16:18:15.308009+0000","last_peered":"2024-04-16T16:18:15.308009+0000","last_clean":"2024-04-16T16:18:15.308009+0000","last_became_active":"2024-04-16T16:18:15.304636+0000","last_became_peered":"2024-04-16T16:18:15.304636+0000","last_unstale":"2024-04-16T16:18:15.308009+0000","last_undegraded":"2024-04-16T16:18:15.308009+0000","last_fullsized":"2024-04-16T16:18:15.308009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:21.912428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.283292+0000","last_change":"2024-04-16T16:18:15.283292+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.283292+0000","last_undegraded":"2024-04-16T16:18:15.283292+0000","last_fullsized":"2024-04-16T16:18:15.283292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.308128+0000","last_change":"2024-04-16T16:18:15.305858+0000","last_active":"2024-04-16T16:18:15.308128+0000","last_peered":"2024-04-16T16:18:15.308128+0000","last_clean":"2024-04-16T16:18:15.308128+0000","last_became_active":"2024-04-16T16:18:15.305250+0000","last_became_peered":"2024-04-16T16:18:15.305250+0000","last_unstale":"2024-04-16T16:18:15.308128+0000","last_undegraded":"2024-04-16T16:18:15.308128+0000","last_fullsized":"2024-04-16T16:18:15.308128+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:46:45.674751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.302671+0000","last_change":"2024-04-16T16:18:15.302671+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.302671+0000","last_undegraded":"2024-04-16T16:18:15.302671+0000","last_fullsized":"2024-04-16T16:18:15.302671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311926+0000","last_change":"2024-04-16T16:18:15.309023+0000","last_active":"2024-04-16T16:18:15.311926+0000","last_peered":"2024-04-16T16:18:15.311926+0000","last_clean":"2024-04-16T16:18:15.311926+0000","last_became_active":"2024-04-16T16:18:15.308399+0000","last_became_peered":"2024-04-16T16:18:15.308399+0000","last_unstale":"2024-04-16T16:18:15.311926+0000","last_undegraded":"2024-04-16T16:18:15.311926+0000","last_fullsized":"2024-04-16T16:18:15.311926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:27:40.938359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.288574+0000","last_change":"2024-04-16T16:18:15.288574+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.288574+0000","last_undegraded":"2024-04-16T16:18:15.288574+0000","last_fullsized":"2024-04-16T16:18:15.288574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.314507+0000","last_change":"2024-04-16T16:18:15.311086+0000","last_active":"2024-04-16T16:18:15.314507+0000","last_peered":"2024-04-16T16:18:15.314507+0000","last_clean":"2024-04-16T16:18:15.314507+0000","last_became_active":"2024-04-16T16:18:15.310594+0000","last_became_peered":"2024-04-16T16:18:15.310594+0000","last_unstale":"2024-04-16T16:18:15.314507+0000","last_undegraded":"2024-04-16T16:18:15.314507+0000","last_fullsized":"2024-04-16T16:18:15.314507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:02:52.308842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.292047+0000","last_change":"2024-04-16T16:18:15.292047+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.292047+0000","last_undegraded":"2024-04-16T16:18:15.292047+0000","last_fullsized":"2024-04-16T16:18:15.292047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444049+0000","last_change":"2024-04-16T16:18:15.299375+0000","last_active":"2024-04-16T16:18:17.444049+0000","last_peered":"2024-04-16T16:18:17.444049+0000","last_clean":"2024-04-16T16:18:17.444049+0000","last_became_active":"2024-04-16T16:18:15.298710+0000","last_became_peered":"2024-04-16T16:18:15.298710+0000","last_unstale":"2024-04-16T16:18:17.444049+0000","last_undegraded":"2024-04-16T16:18:17.444049+0000","last_fullsized":"2024-04-16T16:18:17.444049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:12:39.298882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444137+0000","last_change":"2024-04-16T16:18:16.322248+0000","last_active":"2024-04-16T16:18:17.444137+0000","last_peered":"2024-04-16T16:18:17.444137+0000","last_clean":"2024-04-16T16:18:17.444137+0000","last_became_active":"2024-04-16T16:18:16.318615+0000","last_became_peered":"2024-04-16T16:18:16.318615+0000","last_unstale":"2024-04-16T16:18:17.444137+0000","last_undegraded":"2024-04-16T16:18:17.444137+0000","last_fullsized":"2024-04-16T16:18:17.444137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:16:57.729600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.303620+0000","last_change":"2024-04-16T16:18:15.298300+0000","last_active":"2024-04-16T16:18:15.303620+0000","last_peered":"2024-04-16T16:18:15.303620+0000","last_clean":"2024-04-16T16:18:15.303620+0000","last_became_active":"2024-04-16T16:18:15.297872+0000","last_became_peered":"2024-04-16T16:18:15.297872+0000","last_unstale":"2024-04-16T16:18:15.303620+0000","last_undegraded":"2024-04-16T16:18:15.303620+0000","last_fullsized":"2024-04-16T16:18:15.303620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:51:24.645998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.281611+0000","last_change":"2024-04-16T16:18:15.281611+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.281611+0000","last_undegraded":"2024-04-16T16:18:15.281611+0000","last_fullsized":"2024-04-16T16:18:15.281611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311874+0000","last_change":"2024-04-16T16:18:15.308009+0000","last_active":"2024-04-16T16:18:15.311874+0000","last_peered":"2024-04-16T16:18:15.311874+0000","last_clean":"2024-04-16T16:18:15.311874+0000","last_became_active":"2024-04-16T16:18:15.307561+0000","last_became_peered":"2024-04-16T16:18:15.307561+0000","last_unstale":"2024-04-16T16:18:15.311874+0000","last_undegraded":"2024-04-16T16:18:15.311874+0000","last_fullsized":"2024-04-16T16:18:15.311874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:51.934340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296023+0000","last_change":"2024-04-16T16:18:15.296023+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296023+0000","last_undegraded":"2024-04-16T16:18:15.296023+0000","last_fullsized":"2024-04-16T16:18:15.296023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.311093+0000","last_change":"2024-04-16T16:18:15.309487+0000","last_active":"2024-04-16T16:18:15.311093+0000","last_peered":"2024-04-16T16:18:15.311093+0000","last_clean":"2024-04-16T16:18:15.311093+0000","last_became_active":"2024-04-16T16:18:15.308448+0000","last_became_peered":"2024-04-16T16:18:15.308448+0000","last_unstale":"2024-04-16T16:18:15.311093+0000","last_undegraded":"2024-04-16T16:18:15.311093+0000","last_fullsized":"2024-04-16T16:18:15.311093+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:45:40.148360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":83,"state":"creating+peering","last_fresh":"2024-04-16T16:18:15.296266+0000","last_change":"2024-04-16T16:18:15.296266+0000","last_active":"2024-04-16T16:18:15.256054+0000","last_peered":"2024-04-16T16:18:15.256054+0000","last_clean":"2024-04-16T16:18:15.256054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:18:15.296266+0000","last_undegraded":"2024-04-16T16:18:15.296266+0000","last_fullsized":"2024-04-16T16:18:15.296266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.307177+0000","last_change":"2024-04-16T16:18:15.301975+0000","last_active":"2024-04-16T16:18:15.307177+0000","last_peered":"2024-04-16T16:18:15.307177+0000","last_clean":"2024-04-16T16:18:15.307177+0000","last_became_active":"2024-04-16T16:18:15.300433+0000","last_became_peered":"2024-04-16T16:18:15.300433+0000","last_unstale":"2024-04-16T16:18:15.307177+0000","last_undegraded":"2024-04-16T16:18:15.307177+0000","last_fullsized":"2024-04-16T16:18:15.307177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:57.659561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"74'161","reported_seq":452,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-16T16:18:15.286818+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:18:15.286818+0000","last_peered":"2024-04-16T16:18:15.286818+0000","last_clean":"2024-04-16T16:18:15.286818+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:18:15.286818+0000","last_undegraded":"2024-04-16T16:18:15.286818+0000","last_fullsized":"2024-04-16T16:18:15.286818+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16,"ondisk_log_size":16,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":1601536,"data_stored":1062424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":478331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":478331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574909,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":478331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339628,"statfs":{"total":96636764160,"available":96603779072,"internally_reserved":0,"allocated":1056768,"data_stored":493018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574908,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1048576,"data_stored":491988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1007616,"data_stored":472056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574909,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32788,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":1642496,"data_stored":1081808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:18:24.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:18:24.433 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:18:25.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:18:26.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.218+0000 7f62fde00640 1 Processor -- start 2024-04-16T16:18:26.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.219+0000 7f62fde00640 1 -- start start 2024-04-16T16:18:26.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.220+0000 7f62fde00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62f8161e90 0x7f62f80cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.220+0000 7f62fde00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f62f8162270 0x7f62f80d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.220+0000 7f62fde00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62f80d3080 con 0x7f62f80d52d0 2024-04-16T16:18:26.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.220+0000 7f62fde00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f62f80d2e30 con 0x7f62f8162270 2024-04-16T16:18:26.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.221+0000 7f62fde00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f62f80d3360 con 0x7f62f8161e90 2024-04-16T16:18:26.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.221+0000 7f62fd400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f62f80d52d0 0x7f62f806ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41130/0 (socket says 172.21.15.45:41130) 2024-04-16T16:18:26.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.221+0000 7f62fd400640 1 -- 172.21.15.45:0/2503764724 learned_addr learned my addr 172.21.15.45:0/2503764724 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:26.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.221+0000 7f62fca00640 1 --2- 172.21.15.45:0/2503764724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62f8161e90 0x7f62f80cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.221+0000 7f62f7e00640 1 --2- 172.21.15.45:0/2503764724 >> v2:172.21.15.103:3300/0 conn(0x7f62f8162270 0x7f62f80d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 -- 172.21.15.45:0/2503764724 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1210626572 0 0) 0x7f62f80d3080 con 0x7f62f80d52d0 2024-04-16T16:18:26.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 -- 172.21.15.45:0/2503764724 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62dc003520 con 0x7f62f80d52d0 2024-04-16T16:18:26.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 -- 172.21.15.45:0/2503764724 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 927702858 0 0) 0x7f62dc003520 con 0x7f62f80d52d0 2024-04-16T16:18:26.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 -- 172.21.15.45:0/2503764724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62f8161e90 msgr2=0x7f62f80cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:26.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 --2- 172.21.15.45:0/2503764724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62f8161e90 0x7f62f80cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 -- 172.21.15.45:0/2503764724 >> v2:172.21.15.103:3300/0 conn(0x7f62f8162270 msgr2=0x7f62f80d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 --2- 172.21.15.45:0/2503764724 >> v2:172.21.15.103:3300/0 conn(0x7f62f8162270 0x7f62f80d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 -- 172.21.15.45:0/2503764724 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f80d3640 con 0x7f62f80d52d0 2024-04-16T16:18:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 -- 172.21.15.45:0/2503764724 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f62f00019f0 con 0x7f62f80d52d0 2024-04-16T16:18:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 -- 172.21.15.45:0/2503764724 >> v1:172.21.15.45:6789/0 conn(0x7f62f80d52d0 legacy=0x7f62f806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 --2- 172.21.15.45:0/2503764724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62dc0039b0 0x7f62dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 --2- 172.21.15.45:0/2503764724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62dc004800 0x7f62dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 --2- 172.21.15.45:0/2503764724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62f8161e90 0x7f62dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 -- 172.21.15.45:0/2503764724 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f62f80d2e30 con 0x7f62dc0039b0 2024-04-16T16:18:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 -- 172.21.15.45:0/2503764724 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f62f80d3360 con 0x7f62dc004800 2024-04-16T16:18:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62f7400640 1 -- 172.21.15.45:0/2503764724 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f62dc00b9c0 con 0x7f62f8161e90 2024-04-16T16:18:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.222+0000 7f62fca00640 1 --2- 172.21.15.45:0/2503764724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62dc0039b0 0x7f62dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.223+0000 7f62f7e00640 1 --2- 172.21.15.45:0/2503764724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62dc004800 0x7f62dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.223+0000 7f62fca00640 1 -- 172.21.15.45:0/2503764724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62f8161e90 msgr2=0x7f62dc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:26.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.223+0000 7f62fd400640 1 --2- 172.21.15.45:0/2503764724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62f8161e90 0x7f62dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.223+0000 7f62fca00640 1 --2- 172.21.15.45:0/2503764724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62f8161e90 0x7f62dc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.223+0000 7f62fca00640 1 -- 172.21.15.45:0/2503764724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62dc004800 msgr2=0x7f62dc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:26.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.223+0000 7f62fca00640 1 --2- 172.21.15.45:0/2503764724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62dc004800 0x7f62dc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.223+0000 7f62fca00640 1 -- 172.21.15.45:0/2503764724 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62dc003520 con 0x7f62dc0039b0 2024-04-16T16:18:26.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.223+0000 7f62f7e00640 1 --2- 172.21.15.45:0/2503764724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62dc004800 0x7f62dc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:26.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.224+0000 7f62fca00640 1 --2- 172.21.15.45:0/2503764724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62dc0039b0 0x7f62dc003dd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f62e8009ac0 tx=0x7f62dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bccd0032fa5ef698 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:26.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.224+0000 7f62f7400640 1 -- 172.21.15.45:0/2503764724 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e8039070 con 0x7f62dc0039b0 2024-04-16T16:18:26.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.224+0000 7f62f7400640 1 -- 172.21.15.45:0/2503764724 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e802c6b0 con 0x7f62dc0039b0 2024-04-16T16:18:26.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.225+0000 7f62f7400640 1 -- 172.21.15.45:0/2503764724 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e8009690 con 0x7f62dc0039b0 2024-04-16T16:18:26.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.225+0000 7f62fde00640 1 -- 172.21.15.45:0/2503764724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62dc0039b0 msgr2=0x7f62dc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:26.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.225+0000 7f62fde00640 1 --2- 172.21.15.45:0/2503764724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62dc0039b0 0x7f62dc003dd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f62e8009ac0 tx=0x7f62dc005830 comp rx=0 tx=0).stop 2024-04-16T16:18:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.225+0000 7f62fd400640 1 -- 172.21.15.45:0/2503764724 reap_dead start 2024-04-16T16:18:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.225+0000 7f62fde00640 1 -- 172.21.15.45:0/2503764724 shutdown_connections 2024-04-16T16:18:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.225+0000 7f62fde00640 1 -- 172.21.15.45:0/2503764724 >> 172.21.15.45:0/2503764724 conn(0x7f62f80d1940 msgr2=0x7f62f80d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.225+0000 7f62fde00640 1 -- 172.21.15.45:0/2503764724 shutdown_connections 2024-04-16T16:18:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.225+0000 7f62fde00640 1 -- 172.21.15.45:0/2503764724 wait complete. 2024-04-16T16:18:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.227+0000 7f62fde00640 1 Processor -- start 2024-04-16T16:18:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.227+0000 7f62fde00640 1 -- start start 2024-04-16T16:18:26.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.228+0000 7f62fde00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62dc0039b0 0x7f62f815a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.228+0000 7f62fde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62f815b720 0x7f62f815c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.228+0000 7f62fde00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62f80c9800 0x7f62f80c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.228+0000 7f62fde00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f62f806d6d0 con 0x7f62f815b720 2024-04-16T16:18:26.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.228+0000 7f62fde00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f62f806d550 con 0x7f62dc0039b0 2024-04-16T16:18:26.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.228+0000 7f62fde00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f62f80d4380 con 0x7f62f80c9800 2024-04-16T16:18:26.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.229+0000 7f62f7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62f815b720 0x7f62f815c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.229+0000 7f62f7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62f815b720 0x7f62f815c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57584/0 (socket says 172.21.15.45:57584) 2024-04-16T16:18:26.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.229+0000 7f62f7e00640 1 -- 172.21.15.45:0/2016313025 learned_addr learned my addr 172.21.15.45:0/2016313025 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:26.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.229+0000 7f62fca00640 1 --2- 172.21.15.45:0/2016313025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62dc0039b0 0x7f62f815a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.229+0000 7f62fd400640 1 --2- 172.21.15.45:0/2016313025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62f80c9800 0x7f62f80c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.229+0000 7f62f7e00640 1 -- 172.21.15.45:0/2016313025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62f80c9800 msgr2=0x7f62f80c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:26.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.229+0000 7f62f7e00640 1 --2- 172.21.15.45:0/2016313025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62f80c9800 0x7f62f80c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.229+0000 7f62f7e00640 1 -- 172.21.15.45:0/2016313025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62dc0039b0 msgr2=0x7f62f815a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:26.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.229+0000 7f62f7e00640 1 --2- 172.21.15.45:0/2016313025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62dc0039b0 0x7f62f815a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.229+0000 7f62f7e00640 1 -- 172.21.15.45:0/2016313025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f815c9b0 con 0x7f62f815b720 2024-04-16T16:18:26.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.229+0000 7f62fca00640 1 --2- 172.21.15.45:0/2016313025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62dc0039b0 0x7f62f815a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:26.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.229+0000 7f62f7e00640 1 --2- 172.21.15.45:0/2016313025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62f815b720 0x7f62f815c470 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f62e40109d0 tx=0x7f62e4010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:26.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.230+0000 7f62f4c00640 1 -- 172.21.15.45:0/2016313025 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e400f040 con 0x7f62f815b720 2024-04-16T16:18:26.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.230+0000 7f62f4c00640 1 -- 172.21.15.45:0/2016313025 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e400ec20 con 0x7f62f815b720 2024-04-16T16:18:26.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.230+0000 7f62f4c00640 1 -- 172.21.15.45:0/2016313025 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e401c400 con 0x7f62f815b720 2024-04-16T16:18:26.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.230+0000 7f62fde00640 1 -- 172.21.15.45:0/2016313025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62f815a680 con 0x7f62f815b720 2024-04-16T16:18:26.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.230+0000 7f62fde00640 1 -- 172.21.15.45:0/2016313025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62f815ac40 con 0x7f62f815b720 2024-04-16T16:18:26.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.231+0000 7f62fde00640 1 -- 172.21.15.45:0/2016313025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62f8075780 con 0x7f62f815b720 2024-04-16T16:18:26.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.232+0000 7f62f4c00640 1 -- 172.21.15.45:0/2016313025 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f62e401c5a0 con 0x7f62f815b720 2024-04-16T16:18:26.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.234+0000 7f62f4c00640 1 --2- 172.21.15.45:0/2016313025 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f62d806ca40 0x7f62d806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.234+0000 7f62f4c00640 1 -- 172.21.15.45:0/2016313025 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f62e408c420 con 0x7f62f815b720 2024-04-16T16:18:26.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.234+0000 7f62fca00640 1 --2- 172.21.15.45:0/2016313025 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f62d806ca40 0x7f62d806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.235+0000 7f62fca00640 1 --2- 172.21.15.45:0/2016313025 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f62d806ca40 0x7f62d806ef00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f62e8036000 tx=0x7f62e8030000 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:26.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.239+0000 7f62f4c00640 1 -- 172.21.15.45:0/2016313025 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f62e4018030 con 0x7f62f815b720 2024-04-16T16:18:26.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.439+0000 7f62fde00640 1 -- 172.21.15.45:0/2016313025 --> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f62f806dae0 con 0x7f62d806ca40 2024-04-16T16:18:26.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.459+0000 7f62f4c00640 1 -- 172.21.15.45:0/2016313025 <== mgr.5791 v2:172.21.15.103:6832/3728775368 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252085 (secure 0 0 0) 0x7f62f806dae0 con 0x7f62d806ca40 2024-04-16T16:18:26.462 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:26.463 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.465+0000 7f62fde00640 1 -- 172.21.15.45:0/2016313025 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f62d806ca40 msgr2=0x7f62d806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.465+0000 7f62fde00640 1 --2- 172.21.15.45:0/2016313025 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f62d806ca40 0x7f62d806ef00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f62e8036000 tx=0x7f62e8030000 comp rx=0 tx=0).stop 2024-04-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.465+0000 7f62fde00640 1 -- 172.21.15.45:0/2016313025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62f815b720 msgr2=0x7f62f815c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.465+0000 7f62fde00640 1 --2- 172.21.15.45:0/2016313025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62f815b720 0x7f62f815c470 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f62e40109d0 tx=0x7f62e4010e90 comp rx=0 tx=0).stop 2024-04-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.465+0000 7f62fde00640 1 -- 172.21.15.45:0/2016313025 shutdown_connections 2024-04-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.465+0000 7f62fde00640 1 --2- 172.21.15.45:0/2016313025 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f62d806ca40 0x7f62d806ef00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.465+0000 7f62fde00640 1 --2- 172.21.15.45:0/2016313025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62f80c9800 0x7f62f80c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.465+0000 7f62fde00640 1 --2- 172.21.15.45:0/2016313025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62f815b720 0x7f62f815c470 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.465+0000 7f62fde00640 1 --2- 172.21.15.45:0/2016313025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62dc0039b0 0x7f62f815a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.465+0000 7f62fde00640 1 -- 172.21.15.45:0/2016313025 >> 172.21.15.45:0/2016313025 conn(0x7f62f80d1940 msgr2=0x7f62f80b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:26.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.465+0000 7f62fde00640 1 -- 172.21.15.45:0/2016313025 shutdown_connections 2024-04-16T16:18:26.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.466+0000 7f62fde00640 1 -- 172.21.15.45:0/2016313025 wait complete. 2024-04-16T16:18:26.486 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-04-16T16:18:25.121114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":284,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":200,"ondisk_log_size":200,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258832,"kb_used_data":9384,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715888,"statfs":{"total":773094113280,"available":772829069312,"internally_reserved":0,"allocated":9609216,"data_stored":5116804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.841006"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.447012+0000","last_change":"2024-04-16T16:18:16.323389+0000","last_active":"2024-04-16T16:18:17.447012+0000","last_peered":"2024-04-16T16:18:17.447012+0000","last_clean":"2024-04-16T16:18:17.447012+0000","last_became_active":"2024-04-16T16:18:16.322870+0000","last_became_peered":"2024-04-16T16:18:16.322870+0000","last_unstale":"2024-04-16T16:18:17.447012+0000","last_undegraded":"2024-04-16T16:18:17.447012+0000","last_fullsized":"2024-04-16T16:18:17.447012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:21:33.844543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443171+0000","last_change":"2024-04-16T16:18:16.329075+0000","last_active":"2024-04-16T16:18:17.443171+0000","last_peered":"2024-04-16T16:18:17.443171+0000","last_clean":"2024-04-16T16:18:17.443171+0000","last_became_active":"2024-04-16T16:18:16.328741+0000","last_became_peered":"2024-04-16T16:18:16.328741+0000","last_unstale":"2024-04-16T16:18:17.443171+0000","last_undegraded":"2024-04-16T16:18:17.443171+0000","last_fullsized":"2024-04-16T16:18:17.443171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:59:03.797574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.319674+0000","last_change":"2024-04-16T16:18:16.328219+0000","last_active":"2024-04-16T16:18:17.319674+0000","last_peered":"2024-04-16T16:18:17.319674+0000","last_clean":"2024-04-16T16:18:17.319674+0000","last_became_active":"2024-04-16T16:18:16.327456+0000","last_became_peered":"2024-04-16T16:18:16.327456+0000","last_unstale":"2024-04-16T16:18:17.319674+0000","last_undegraded":"2024-04-16T16:18:17.319674+0000","last_fullsized":"2024-04-16T16:18:17.319674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:21:33.844543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312806+0000","last_change":"2024-04-16T16:18:16.316835+0000","last_active":"2024-04-16T16:18:17.312806+0000","last_peered":"2024-04-16T16:18:17.312806+0000","last_clean":"2024-04-16T16:18:17.312806+0000","last_became_active":"2024-04-16T16:18:16.309649+0000","last_became_peered":"2024-04-16T16:18:16.309649+0000","last_unstale":"2024-04-16T16:18:17.312806+0000","last_undegraded":"2024-04-16T16:18:17.312806+0000","last_fullsized":"2024-04-16T16:18:17.312806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:21:05.656211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.322114+0000","last_change":"2024-04-16T16:18:16.311850+0000","last_active":"2024-04-16T16:18:17.322114+0000","last_peered":"2024-04-16T16:18:17.322114+0000","last_clean":"2024-04-16T16:18:17.322114+0000","last_became_active":"2024-04-16T16:18:16.311518+0000","last_became_peered":"2024-04-16T16:18:16.311518+0000","last_unstale":"2024-04-16T16:18:17.322114+0000","last_undegraded":"2024-04-16T16:18:17.322114+0000","last_fullsized":"2024-04-16T16:18:17.322114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:00:23.797881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.445185+0000","last_change":"2024-04-16T16:18:16.313121+0000","last_active":"2024-04-16T16:18:17.445185+0000","last_peered":"2024-04-16T16:18:17.445185+0000","last_clean":"2024-04-16T16:18:17.445185+0000","last_became_active":"2024-04-16T16:18:16.311686+0000","last_became_peered":"2024-04-16T16:18:16.311686+0000","last_unstale":"2024-04-16T16:18:17.445185+0000","last_undegraded":"2024-04-16T16:18:17.445185+0000","last_fullsized":"2024-04-16T16:18:17.445185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:14:19.421933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312314+0000","last_change":"2024-04-16T16:18:16.308558+0000","last_active":"2024-04-16T16:18:17.312314+0000","last_peered":"2024-04-16T16:18:17.312314+0000","last_clean":"2024-04-16T16:18:17.312314+0000","last_became_active":"2024-04-16T16:18:16.308268+0000","last_became_peered":"2024-04-16T16:18:16.308268+0000","last_unstale":"2024-04-16T16:18:17.312314+0000","last_undegraded":"2024-04-16T16:18:17.312314+0000","last_fullsized":"2024-04-16T16:18:17.312314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:13:57.589683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444695+0000","last_change":"2024-04-16T16:18:16.313130+0000","last_active":"2024-04-16T16:18:17.444695+0000","last_peered":"2024-04-16T16:18:17.444695+0000","last_clean":"2024-04-16T16:18:17.444695+0000","last_became_active":"2024-04-16T16:18:16.312774+0000","last_became_peered":"2024-04-16T16:18:16.312774+0000","last_unstale":"2024-04-16T16:18:17.444695+0000","last_undegraded":"2024-04-16T16:18:17.444695+0000","last_fullsized":"2024-04-16T16:18:17.444695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:00:23.797881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.445911+0000","last_change":"2024-04-16T16:18:16.314839+0000","last_active":"2024-04-16T16:18:17.445911+0000","last_peered":"2024-04-16T16:18:17.445911+0000","last_clean":"2024-04-16T16:18:17.445911+0000","last_became_active":"2024-04-16T16:18:16.314573+0000","last_became_peered":"2024-04-16T16:18:16.314573+0000","last_unstale":"2024-04-16T16:18:17.445911+0000","last_undegraded":"2024-04-16T16:18:17.445911+0000","last_fullsized":"2024-04-16T16:18:17.445911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:22:04.486147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.446652+0000","last_change":"2024-04-16T16:18:16.318475+0000","last_active":"2024-04-16T16:18:17.446652+0000","last_peered":"2024-04-16T16:18:17.446652+0000","last_clean":"2024-04-16T16:18:17.446652+0000","last_became_active":"2024-04-16T16:18:16.315798+0000","last_became_peered":"2024-04-16T16:18:16.315798+0000","last_unstale":"2024-04-16T16:18:17.446652+0000","last_undegraded":"2024-04-16T16:18:17.446652+0000","last_fullsized":"2024-04-16T16:18:17.446652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:54:07.935177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443352+0000","last_change":"2024-04-16T16:18:16.311426+0000","last_active":"2024-04-16T16:18:17.443352+0000","last_peered":"2024-04-16T16:18:17.443352+0000","last_clean":"2024-04-16T16:18:17.443352+0000","last_became_active":"2024-04-16T16:18:16.311163+0000","last_became_peered":"2024-04-16T16:18:16.311163+0000","last_unstale":"2024-04-16T16:18:17.443352+0000","last_undegraded":"2024-04-16T16:18:17.443352+0000","last_fullsized":"2024-04-16T16:18:17.443352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:34:29.300189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.446108+0000","last_change":"2024-04-16T16:18:16.319073+0000","last_active":"2024-04-16T16:18:17.446108+0000","last_peered":"2024-04-16T16:18:17.446108+0000","last_clean":"2024-04-16T16:18:17.446108+0000","last_became_active":"2024-04-16T16:18:16.318733+0000","last_became_peered":"2024-04-16T16:18:16.318733+0000","last_unstale":"2024-04-16T16:18:17.446108+0000","last_undegraded":"2024-04-16T16:18:17.446108+0000","last_fullsized":"2024-04-16T16:18:17.446108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:20:57.528152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.319390+0000","last_change":"2024-04-16T16:18:16.320689+0000","last_active":"2024-04-16T16:18:17.319390+0000","last_peered":"2024-04-16T16:18:17.319390+0000","last_clean":"2024-04-16T16:18:17.319390+0000","last_became_active":"2024-04-16T16:18:16.320388+0000","last_became_peered":"2024-04-16T16:18:16.320388+0000","last_unstale":"2024-04-16T16:18:17.319390+0000","last_undegraded":"2024-04-16T16:18:17.319390+0000","last_fullsized":"2024-04-16T16:18:17.319390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:42.181514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.313189+0000","last_change":"2024-04-16T16:18:16.321879+0000","last_active":"2024-04-16T16:18:17.313189+0000","last_peered":"2024-04-16T16:18:17.313189+0000","last_clean":"2024-04-16T16:18:17.313189+0000","last_became_active":"2024-04-16T16:18:16.310779+0000","last_became_peered":"2024-04-16T16:18:16.310779+0000","last_unstale":"2024-04-16T16:18:17.313189+0000","last_undegraded":"2024-04-16T16:18:17.313189+0000","last_fullsized":"2024-04-16T16:18:17.313189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:46:46.691832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394765+0000","last_change":"2024-04-16T16:18:16.314327+0000","last_active":"2024-04-16T16:18:17.394765+0000","last_peered":"2024-04-16T16:18:17.394765+0000","last_clean":"2024-04-16T16:18:17.394765+0000","last_became_active":"2024-04-16T16:18:16.313983+0000","last_became_peered":"2024-04-16T16:18:16.313983+0000","last_unstale":"2024-04-16T16:18:17.394765+0000","last_undegraded":"2024-04-16T16:18:17.394765+0000","last_fullsized":"2024-04-16T16:18:17.394765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:06:33.666684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.321416+0000","last_change":"2024-04-16T16:18:16.315413+0000","last_active":"2024-04-16T16:18:17.321416+0000","last_peered":"2024-04-16T16:18:17.321416+0000","last_clean":"2024-04-16T16:18:17.321416+0000","last_became_active":"2024-04-16T16:18:16.314926+0000","last_became_peered":"2024-04-16T16:18:16.314926+0000","last_unstale":"2024-04-16T16:18:17.321416+0000","last_undegraded":"2024-04-16T16:18:17.321416+0000","last_fullsized":"2024-04-16T16:18:17.321416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:12:40.315962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.323079+0000","last_change":"2024-04-16T16:18:16.303180+0000","last_active":"2024-04-16T16:18:17.323079+0000","last_peered":"2024-04-16T16:18:17.323079+0000","last_clean":"2024-04-16T16:18:17.323079+0000","last_became_active":"2024-04-16T16:18:16.302897+0000","last_became_peered":"2024-04-16T16:18:16.302897+0000","last_unstale":"2024-04-16T16:18:17.323079+0000","last_undegraded":"2024-04-16T16:18:17.323079+0000","last_fullsized":"2024-04-16T16:18:17.323079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:42.181514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312691+0000","last_change":"2024-04-16T16:18:16.314027+0000","last_active":"2024-04-16T16:18:17.312691+0000","last_peered":"2024-04-16T16:18:17.312691+0000","last_clean":"2024-04-16T16:18:17.312691+0000","last_became_active":"2024-04-16T16:18:16.313426+0000","last_became_peered":"2024-04-16T16:18:16.313426+0000","last_unstale":"2024-04-16T16:18:17.312691+0000","last_undegraded":"2024-04-16T16:18:17.312691+0000","last_fullsized":"2024-04-16T16:18:17.312691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:30:17.865648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444884+0000","last_change":"2024-04-16T16:18:16.305406+0000","last_active":"2024-04-16T16:18:17.444884+0000","last_peered":"2024-04-16T16:18:17.444884+0000","last_clean":"2024-04-16T16:18:17.444884+0000","last_became_active":"2024-04-16T16:18:16.305023+0000","last_became_peered":"2024-04-16T16:18:16.305023+0000","last_unstale":"2024-04-16T16:18:17.444884+0000","last_undegraded":"2024-04-16T16:18:17.444884+0000","last_fullsized":"2024-04-16T16:18:17.444884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:20.251382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312975+0000","last_change":"2024-04-16T16:18:16.317943+0000","last_active":"2024-04-16T16:18:17.312975+0000","last_peered":"2024-04-16T16:18:17.312975+0000","last_clean":"2024-04-16T16:18:17.312975+0000","last_became_active":"2024-04-16T16:18:16.317474+0000","last_became_peered":"2024-04-16T16:18:16.317474+0000","last_unstale":"2024-04-16T16:18:17.312975+0000","last_undegraded":"2024-04-16T16:18:17.312975+0000","last_fullsized":"2024-04-16T16:18:17.312975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:52:31.379753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312792+0000","last_change":"2024-04-16T16:18:16.300164+0000","last_active":"2024-04-16T16:18:17.312792+0000","last_peered":"2024-04-16T16:18:17.312792+0000","last_clean":"2024-04-16T16:18:17.312792+0000","last_became_active":"2024-04-16T16:18:16.299838+0000","last_became_peered":"2024-04-16T16:18:16.299838+0000","last_unstale":"2024-04-16T16:18:17.312792+0000","last_undegraded":"2024-04-16T16:18:17.312792+0000","last_fullsized":"2024-04-16T16:18:17.312792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:22.876310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.442354+0000","last_change":"2024-04-16T16:18:16.320126+0000","last_active":"2024-04-16T16:18:17.442354+0000","last_peered":"2024-04-16T16:18:17.442354+0000","last_clean":"2024-04-16T16:18:17.442354+0000","last_became_active":"2024-04-16T16:18:16.319868+0000","last_became_peered":"2024-04-16T16:18:16.319868+0000","last_unstale":"2024-04-16T16:18:17.442354+0000","last_undegraded":"2024-04-16T16:18:17.442354+0000","last_fullsized":"2024-04-16T16:18:17.442354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:27:41.955440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.445466+0000","last_change":"2024-04-16T16:18:16.318565+0000","last_active":"2024-04-16T16:18:17.445466+0000","last_peered":"2024-04-16T16:18:17.445466+0000","last_clean":"2024-04-16T16:18:17.445466+0000","last_became_active":"2024-04-16T16:18:16.315191+0000","last_became_peered":"2024-04-16T16:18:16.315191+0000","last_unstale":"2024-04-16T16:18:17.445466+0000","last_undegraded":"2024-04-16T16:18:17.445466+0000","last_fullsized":"2024-04-16T16:18:17.445466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:03:49.555993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312382+0000","last_change":"2024-04-16T16:18:16.315372+0000","last_active":"2024-04-16T16:18:17.312382+0000","last_peered":"2024-04-16T16:18:17.312382+0000","last_clean":"2024-04-16T16:18:17.312382+0000","last_became_active":"2024-04-16T16:18:16.315098+0000","last_became_peered":"2024-04-16T16:18:16.315098+0000","last_unstale":"2024-04-16T16:18:17.312382+0000","last_undegraded":"2024-04-16T16:18:17.312382+0000","last_fullsized":"2024-04-16T16:18:17.312382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:22.929509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.442801+0000","last_change":"2024-04-16T16:18:16.314845+0000","last_active":"2024-04-16T16:18:17.442801+0000","last_peered":"2024-04-16T16:18:17.442801+0000","last_clean":"2024-04-16T16:18:17.442801+0000","last_became_active":"2024-04-16T16:18:16.314566+0000","last_became_peered":"2024-04-16T16:18:16.314566+0000","last_unstale":"2024-04-16T16:18:17.442801+0000","last_undegraded":"2024-04-16T16:18:17.442801+0000","last_fullsized":"2024-04-16T16:18:17.442801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:32.951527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.320161+0000","last_change":"2024-04-16T16:18:16.328678+0000","last_active":"2024-04-16T16:18:17.320161+0000","last_peered":"2024-04-16T16:18:17.320161+0000","last_clean":"2024-04-16T16:18:17.320161+0000","last_became_active":"2024-04-16T16:18:16.328404+0000","last_became_peered":"2024-04-16T16:18:16.328404+0000","last_unstale":"2024-04-16T16:18:17.320161+0000","last_undegraded":"2024-04-16T16:18:17.320161+0000","last_fullsized":"2024-04-16T16:18:17.320161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:58:52.913785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.322104+0000","last_change":"2024-04-16T16:18:16.302162+0000","last_active":"2024-04-16T16:18:17.322104+0000","last_peered":"2024-04-16T16:18:17.322104+0000","last_clean":"2024-04-16T16:18:17.322104+0000","last_became_active":"2024-04-16T16:18:16.301818+0000","last_became_peered":"2024-04-16T16:18:16.301818+0000","last_unstale":"2024-04-16T16:18:17.322104+0000","last_undegraded":"2024-04-16T16:18:17.322104+0000","last_fullsized":"2024-04-16T16:18:17.322104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:03:49.555993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.446556+0000","last_change":"2024-04-16T16:18:16.322217+0000","last_active":"2024-04-16T16:18:17.446556+0000","last_peered":"2024-04-16T16:18:17.446556+0000","last_clean":"2024-04-16T16:18:17.446556+0000","last_became_active":"2024-04-16T16:18:16.321887+0000","last_became_peered":"2024-04-16T16:18:16.321887+0000","last_unstale":"2024-04-16T16:18:17.446556+0000","last_undegraded":"2024-04-16T16:18:17.446556+0000","last_fullsized":"2024-04-16T16:18:17.446556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:22:04.486147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.442722+0000","last_change":"2024-04-16T16:18:16.306369+0000","last_active":"2024-04-16T16:18:17.442722+0000","last_peered":"2024-04-16T16:18:17.442722+0000","last_clean":"2024-04-16T16:18:17.442722+0000","last_became_active":"2024-04-16T16:18:16.299215+0000","last_became_peered":"2024-04-16T16:18:16.299215+0000","last_unstale":"2024-04-16T16:18:17.442722+0000","last_undegraded":"2024-04-16T16:18:17.442722+0000","last_fullsized":"2024-04-16T16:18:17.442722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:02:53.325922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.442402+0000","last_change":"2024-04-16T16:18:16.307123+0000","last_active":"2024-04-16T16:18:17.442402+0000","last_peered":"2024-04-16T16:18:17.442402+0000","last_clean":"2024-04-16T16:18:17.442402+0000","last_became_active":"2024-04-16T16:18:16.306494+0000","last_became_peered":"2024-04-16T16:18:16.306494+0000","last_unstale":"2024-04-16T16:18:17.442402+0000","last_undegraded":"2024-04-16T16:18:17.442402+0000","last_fullsized":"2024-04-16T16:18:17.442402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:52.951420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.321487+0000","last_change":"2024-04-16T16:18:16.297447+0000","last_active":"2024-04-16T16:18:17.321487+0000","last_peered":"2024-04-16T16:18:17.321487+0000","last_clean":"2024-04-16T16:18:17.321487+0000","last_became_active":"2024-04-16T16:18:16.296986+0000","last_became_peered":"2024-04-16T16:18:16.296986+0000","last_unstale":"2024-04-16T16:18:17.321487+0000","last_undegraded":"2024-04-16T16:18:17.321487+0000","last_fullsized":"2024-04-16T16:18:17.321487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:58:35.179981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.446443+0000","last_change":"2024-04-16T16:18:16.309768+0000","last_active":"2024-04-16T16:18:17.446443+0000","last_peered":"2024-04-16T16:18:17.446443+0000","last_clean":"2024-04-16T16:18:17.446443+0000","last_became_active":"2024-04-16T16:18:16.309391+0000","last_became_peered":"2024-04-16T16:18:16.309391+0000","last_unstale":"2024-04-16T16:18:17.446443+0000","last_undegraded":"2024-04-16T16:18:17.446443+0000","last_fullsized":"2024-04-16T16:18:17.446443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:08.532092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.322845+0000","last_change":"2024-04-16T16:18:16.322906+0000","last_active":"2024-04-16T16:18:17.322845+0000","last_peered":"2024-04-16T16:18:17.322845+0000","last_clean":"2024-04-16T16:18:17.322845+0000","last_became_active":"2024-04-16T16:18:16.322588+0000","last_became_peered":"2024-04-16T16:18:16.322588+0000","last_unstale":"2024-04-16T16:18:17.322845+0000","last_undegraded":"2024-04-16T16:18:17.322845+0000","last_fullsized":"2024-04-16T16:18:17.322845+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:22:04.486147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"84'4","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.447533+0000","last_change":"2024-04-16T16:18:15.307894+0000","last_active":"2024-04-16T16:18:17.447533+0000","last_peered":"2024-04-16T16:18:17.447533+0000","last_clean":"2024-04-16T16:18:17.447533+0000","last_became_active":"2024-04-16T16:18:15.307584+0000","last_became_peered":"2024-04-16T16:18:15.307584+0000","last_unstale":"2024-04-16T16:18:17.447533+0000","last_undegraded":"2024-04-16T16:18:17.447533+0000","last_fullsized":"2024-04-16T16:18:17.447533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:59:22.965812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.447291+0000","last_change":"2024-04-16T16:18:16.331597+0000","last_active":"2024-04-16T16:18:17.447291+0000","last_peered":"2024-04-16T16:18:17.447291+0000","last_clean":"2024-04-16T16:18:17.447291+0000","last_became_active":"2024-04-16T16:18:16.331324+0000","last_became_peered":"2024-04-16T16:18:16.331324+0000","last_unstale":"2024-04-16T16:18:17.447291+0000","last_undegraded":"2024-04-16T16:18:17.447291+0000","last_fullsized":"2024-04-16T16:18:17.447291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:16:57.729600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"84'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.323212+0000","last_change":"2024-04-16T16:18:15.299901+0000","last_active":"2024-04-16T16:18:17.323212+0000","last_peered":"2024-04-16T16:18:17.323212+0000","last_clean":"2024-04-16T16:18:17.323212+0000","last_became_active":"2024-04-16T16:18:15.299622+0000","last_became_peered":"2024-04-16T16:18:15.299622+0000","last_unstale":"2024-04-16T16:18:17.323212+0000","last_undegraded":"2024-04-16T16:18:17.323212+0000","last_fullsized":"2024-04-16T16:18:17.323212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:59:02.780493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.446916+0000","last_change":"2024-04-16T16:18:16.310001+0000","last_active":"2024-04-16T16:18:17.446916+0000","last_peered":"2024-04-16T16:18:17.446916+0000","last_clean":"2024-04-16T16:18:17.446916+0000","last_became_active":"2024-04-16T16:18:16.309437+0000","last_became_peered":"2024-04-16T16:18:16.309437+0000","last_unstale":"2024-04-16T16:18:17.446916+0000","last_undegraded":"2024-04-16T16:18:17.446916+0000","last_fullsized":"2024-04-16T16:18:17.446916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:07:31.030194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444710+0000","last_change":"2024-04-16T16:18:15.297918+0000","last_active":"2024-04-16T16:18:17.444710+0000","last_peered":"2024-04-16T16:18:17.444710+0000","last_clean":"2024-04-16T16:18:17.444710+0000","last_became_active":"2024-04-16T16:18:15.297217+0000","last_became_peered":"2024-04-16T16:18:15.297217+0000","last_unstale":"2024-04-16T16:18:17.444710+0000","last_undegraded":"2024-04-16T16:18:17.444710+0000","last_fullsized":"2024-04-16T16:18:17.444710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:03:48.538913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.320023+0000","last_change":"2024-04-16T16:18:16.326814+0000","last_active":"2024-04-16T16:18:17.320023+0000","last_peered":"2024-04-16T16:18:17.320023+0000","last_clean":"2024-04-16T16:18:17.320023+0000","last_became_active":"2024-04-16T16:18:16.325080+0000","last_became_peered":"2024-04-16T16:18:16.325080+0000","last_unstale":"2024-04-16T16:18:17.320023+0000","last_undegraded":"2024-04-16T16:18:17.320023+0000","last_fullsized":"2024-04-16T16:18:17.320023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:06:40.052453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443826+0000","last_change":"2024-04-16T16:18:15.300207+0000","last_active":"2024-04-16T16:18:17.443826+0000","last_peered":"2024-04-16T16:18:17.443826+0000","last_clean":"2024-04-16T16:18:17.443826+0000","last_became_active":"2024-04-16T16:18:15.297220+0000","last_became_peered":"2024-04-16T16:18:15.297220+0000","last_unstale":"2024-04-16T16:18:17.443826+0000","last_undegraded":"2024-04-16T16:18:17.443826+0000","last_fullsized":"2024-04-16T16:18:17.443826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:58:34.162900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.442754+0000","last_change":"2024-04-16T16:18:16.300235+0000","last_active":"2024-04-16T16:18:17.442754+0000","last_peered":"2024-04-16T16:18:17.442754+0000","last_clean":"2024-04-16T16:18:17.442754+0000","last_became_active":"2024-04-16T16:18:16.299958+0000","last_became_peered":"2024-04-16T16:18:16.299958+0000","last_unstale":"2024-04-16T16:18:17.442754+0000","last_undegraded":"2024-04-16T16:18:17.442754+0000","last_fullsized":"2024-04-16T16:18:17.442754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:21.463944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"84'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.322628+0000","last_change":"2024-04-16T16:18:15.309872+0000","last_active":"2024-04-16T16:18:17.322628+0000","last_peered":"2024-04-16T16:18:17.322628+0000","last_clean":"2024-04-16T16:18:17.322628+0000","last_became_active":"2024-04-16T16:18:15.309546+0000","last_became_peered":"2024-04-16T16:18:15.309546+0000","last_unstale":"2024-04-16T16:18:17.322628+0000","last_undegraded":"2024-04-16T16:18:17.322628+0000","last_fullsized":"2024-04-16T16:18:17.322628+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:34:28.283109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.322572+0000","last_change":"2024-04-16T16:18:16.310722+0000","last_active":"2024-04-16T16:18:17.322572+0000","last_peered":"2024-04-16T16:18:17.322572+0000","last_clean":"2024-04-16T16:18:17.322572+0000","last_became_active":"2024-04-16T16:18:16.310414+0000","last_became_peered":"2024-04-16T16:18:16.310414+0000","last_unstale":"2024-04-16T16:18:17.322572+0000","last_undegraded":"2024-04-16T16:18:17.322572+0000","last_fullsized":"2024-04-16T16:18:17.322572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:20:57.528152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"84'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.321717+0000","last_change":"2024-04-16T16:18:15.298912+0000","last_active":"2024-04-16T16:18:17.321717+0000","last_peered":"2024-04-16T16:18:17.321717+0000","last_clean":"2024-04-16T16:18:17.321717+0000","last_became_active":"2024-04-16T16:18:15.298638+0000","last_became_peered":"2024-04-16T16:18:15.298638+0000","last_unstale":"2024-04-16T16:18:17.321717+0000","last_undegraded":"2024-04-16T16:18:17.321717+0000","last_fullsized":"2024-04-16T16:18:17.321717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:31.934447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.318385+0000","last_change":"2024-04-16T16:18:16.310420+0000","last_active":"2024-04-16T16:18:17.318385+0000","last_peered":"2024-04-16T16:18:17.318385+0000","last_clean":"2024-04-16T16:18:17.318385+0000","last_became_active":"2024-04-16T16:18:16.310112+0000","last_became_peered":"2024-04-16T16:18:16.310112+0000","last_unstale":"2024-04-16T16:18:17.318385+0000","last_undegraded":"2024-04-16T16:18:17.318385+0000","last_fullsized":"2024-04-16T16:18:17.318385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:54:07.935177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444198+0000","last_change":"2024-04-16T16:18:15.301897+0000","last_active":"2024-04-16T16:18:17.444198+0000","last_peered":"2024-04-16T16:18:17.444198+0000","last_clean":"2024-04-16T16:18:17.444198+0000","last_became_active":"2024-04-16T16:18:15.301549+0000","last_became_peered":"2024-04-16T16:18:15.301549+0000","last_unstale":"2024-04-16T16:18:17.444198+0000","last_undegraded":"2024-04-16T16:18:17.444198+0000","last_fullsized":"2024-04-16T16:18:17.444198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:29:16.985875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394783+0000","last_change":"2024-04-16T16:18:16.308198+0000","last_active":"2024-04-16T16:18:17.394783+0000","last_peered":"2024-04-16T16:18:17.394783+0000","last_clean":"2024-04-16T16:18:17.394783+0000","last_became_active":"2024-04-16T16:18:16.307924+0000","last_became_peered":"2024-04-16T16:18:16.307924+0000","last_unstale":"2024-04-16T16:18:17.394783+0000","last_undegraded":"2024-04-16T16:18:17.394783+0000","last_fullsized":"2024-04-16T16:18:17.394783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:34:10.200444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443259+0000","last_change":"2024-04-16T16:18:15.303661+0000","last_active":"2024-04-16T16:18:17.443259+0000","last_peered":"2024-04-16T16:18:17.443259+0000","last_clean":"2024-04-16T16:18:17.443259+0000","last_became_active":"2024-04-16T16:18:15.300455+0000","last_became_peered":"2024-04-16T16:18:15.300455+0000","last_unstale":"2024-04-16T16:18:17.443259+0000","last_undegraded":"2024-04-16T16:18:17.443259+0000","last_fullsized":"2024-04-16T16:18:17.443259+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:30:23.896642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394997+0000","last_change":"2024-04-16T16:18:16.324165+0000","last_active":"2024-04-16T16:18:17.394997+0000","last_peered":"2024-04-16T16:18:17.394997+0000","last_clean":"2024-04-16T16:18:17.394997+0000","last_became_active":"2024-04-16T16:18:16.323865+0000","last_became_peered":"2024-04-16T16:18:16.323865+0000","last_unstale":"2024-04-16T16:18:17.394997+0000","last_undegraded":"2024-04-16T16:18:17.394997+0000","last_fullsized":"2024-04-16T16:18:17.394997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:51:43.715542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.447667+0000","last_change":"2024-04-16T16:18:15.305839+0000","last_active":"2024-04-16T16:18:17.447667+0000","last_peered":"2024-04-16T16:18:17.447667+0000","last_clean":"2024-04-16T16:18:17.447667+0000","last_became_active":"2024-04-16T16:18:15.305453+0000","last_became_peered":"2024-04-16T16:18:15.305453+0000","last_unstale":"2024-04-16T16:18:17.447667+0000","last_undegraded":"2024-04-16T16:18:17.447667+0000","last_fullsized":"2024-04-16T16:18:17.447667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:57.184445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.446001+0000","last_change":"2024-04-16T16:18:16.310838+0000","last_active":"2024-04-16T16:18:17.446001+0000","last_peered":"2024-04-16T16:18:17.446001+0000","last_clean":"2024-04-16T16:18:17.446001+0000","last_became_active":"2024-04-16T16:18:16.307754+0000","last_became_peered":"2024-04-16T16:18:16.307754+0000","last_unstale":"2024-04-16T16:18:17.446001+0000","last_undegraded":"2024-04-16T16:18:17.446001+0000","last_fullsized":"2024-04-16T16:18:17.446001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:54:07.935177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.446633+0000","last_change":"2024-04-16T16:18:15.304936+0000","last_active":"2024-04-16T16:18:17.446633+0000","last_peered":"2024-04-16T16:18:17.446633+0000","last_clean":"2024-04-16T16:18:17.446633+0000","last_became_active":"2024-04-16T16:18:15.304590+0000","last_became_peered":"2024-04-16T16:18:15.304590+0000","last_unstale":"2024-04-16T16:18:17.446633+0000","last_undegraded":"2024-04-16T16:18:17.446633+0000","last_fullsized":"2024-04-16T16:18:17.446633+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:29:49.510991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443924+0000","last_change":"2024-04-16T16:18:16.316337+0000","last_active":"2024-04-16T16:18:17.443924+0000","last_peered":"2024-04-16T16:18:17.443924+0000","last_clean":"2024-04-16T16:18:17.443924+0000","last_became_active":"2024-04-16T16:18:16.316019+0000","last_became_peered":"2024-04-16T16:18:16.316019+0000","last_unstale":"2024-04-16T16:18:17.443924+0000","last_undegraded":"2024-04-16T16:18:17.443924+0000","last_fullsized":"2024-04-16T16:18:17.443924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:21:33.844543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"84'3","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.446000+0000","last_change":"2024-04-16T16:18:15.308173+0000","last_active":"2024-04-16T16:18:17.446000+0000","last_peered":"2024-04-16T16:18:17.446000+0000","last_clean":"2024-04-16T16:18:17.446000+0000","last_became_active":"2024-04-16T16:18:15.307837+0000","last_became_peered":"2024-04-16T16:18:15.307837+0000","last_unstale":"2024-04-16T16:18:17.446000+0000","last_undegraded":"2024-04-16T16:18:17.446000+0000","last_fullsized":"2024-04-16T16:18:17.446000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:37:18.683266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.322133+0000","last_change":"2024-04-16T16:18:16.313403+0000","last_active":"2024-04-16T16:18:17.322133+0000","last_peered":"2024-04-16T16:18:17.322133+0000","last_clean":"2024-04-16T16:18:17.322133+0000","last_became_active":"2024-04-16T16:18:16.311966+0000","last_became_peered":"2024-04-16T16:18:16.311966+0000","last_unstale":"2024-04-16T16:18:17.322133+0000","last_undegraded":"2024-04-16T16:18:17.322133+0000","last_fullsized":"2024-04-16T16:18:17.322133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:14:19.421933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"84'6","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.445267+0000","last_change":"2024-04-16T16:18:15.306390+0000","last_active":"2024-04-16T16:18:17.445267+0000","last_peered":"2024-04-16T16:18:17.445267+0000","last_clean":"2024-04-16T16:18:17.445267+0000","last_became_active":"2024-04-16T16:18:15.306063+0000","last_became_peered":"2024-04-16T16:18:15.306063+0000","last_unstale":"2024-04-16T16:18:17.445267+0000","last_undegraded":"2024-04-16T16:18:17.445267+0000","last_fullsized":"2024-04-16T16:18:17.445267+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:40.620507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443652+0000","last_change":"2024-04-16T16:18:16.315534+0000","last_active":"2024-04-16T16:18:17.443652+0000","last_peered":"2024-04-16T16:18:17.443652+0000","last_clean":"2024-04-16T16:18:17.443652+0000","last_became_active":"2024-04-16T16:18:16.315301+0000","last_became_peered":"2024-04-16T16:18:16.315301+0000","last_unstale":"2024-04-16T16:18:17.443652+0000","last_undegraded":"2024-04-16T16:18:17.443652+0000","last_fullsized":"2024-04-16T16:18:17.443652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:42.181514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.442734+0000","last_change":"2024-04-16T16:18:15.308221+0000","last_active":"2024-04-16T16:18:17.442734+0000","last_peered":"2024-04-16T16:18:17.442734+0000","last_clean":"2024-04-16T16:18:17.442734+0000","last_became_active":"2024-04-16T16:18:15.307923+0000","last_became_peered":"2024-04-16T16:18:15.307923+0000","last_unstale":"2024-04-16T16:18:17.442734+0000","last_undegraded":"2024-04-16T16:18:17.442734+0000","last_fullsized":"2024-04-16T16:18:17.442734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:06:31.278027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.320002+0000","last_change":"2024-04-16T16:18:16.314940+0000","last_active":"2024-04-16T16:18:17.320002+0000","last_peered":"2024-04-16T16:18:17.320002+0000","last_clean":"2024-04-16T16:18:17.320002+0000","last_became_active":"2024-04-16T16:18:16.314654+0000","last_became_peered":"2024-04-16T16:18:16.314654+0000","last_unstale":"2024-04-16T16:18:17.320002+0000","last_undegraded":"2024-04-16T16:18:17.320002+0000","last_fullsized":"2024-04-16T16:18:17.320002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:20.251382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.395004+0000","last_change":"2024-04-16T16:18:15.309398+0000","last_active":"2024-04-16T16:18:17.395004+0000","last_peered":"2024-04-16T16:18:17.395004+0000","last_clean":"2024-04-16T16:18:17.395004+0000","last_became_active":"2024-04-16T16:18:15.308959+0000","last_became_peered":"2024-04-16T16:18:15.308959+0000","last_unstale":"2024-04-16T16:18:17.395004+0000","last_undegraded":"2024-04-16T16:18:17.395004+0000","last_fullsized":"2024-04-16T16:18:17.395004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:10:27.168721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.442460+0000","last_change":"2024-04-16T16:18:16.308089+0000","last_active":"2024-04-16T16:18:17.442460+0000","last_peered":"2024-04-16T16:18:17.442460+0000","last_clean":"2024-04-16T16:18:17.442460+0000","last_became_active":"2024-04-16T16:18:16.307014+0000","last_became_peered":"2024-04-16T16:18:16.307014+0000","last_unstale":"2024-04-16T16:18:17.442460+0000","last_undegraded":"2024-04-16T16:18:17.442460+0000","last_fullsized":"2024-04-16T16:18:17.442460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:45:41.165440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444755+0000","last_change":"2024-04-16T16:18:15.309922+0000","last_active":"2024-04-16T16:18:17.444755+0000","last_peered":"2024-04-16T16:18:17.444755+0000","last_clean":"2024-04-16T16:18:17.444755+0000","last_became_active":"2024-04-16T16:18:15.308635+0000","last_became_peered":"2024-04-16T16:18:15.308635+0000","last_unstale":"2024-04-16T16:18:17.444755+0000","last_undegraded":"2024-04-16T16:18:17.444755+0000","last_fullsized":"2024-04-16T16:18:17.444755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:13.308144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.442226+0000","last_change":"2024-04-16T16:18:16.309209+0000","last_active":"2024-04-16T16:18:17.442226+0000","last_peered":"2024-04-16T16:18:17.442226+0000","last_clean":"2024-04-16T16:18:17.442226+0000","last_became_active":"2024-04-16T16:18:16.308806+0000","last_became_peered":"2024-04-16T16:18:16.308806+0000","last_unstale":"2024-04-16T16:18:17.442226+0000","last_undegraded":"2024-04-16T16:18:17.442226+0000","last_fullsized":"2024-04-16T16:18:17.442226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:30:37.754176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"84'8","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443410+0000","last_change":"2024-04-16T16:18:15.299371+0000","last_active":"2024-04-16T16:18:17.443410+0000","last_peered":"2024-04-16T16:18:17.443410+0000","last_clean":"2024-04-16T16:18:17.443410+0000","last_became_active":"2024-04-16T16:18:15.299047+0000","last_became_peered":"2024-04-16T16:18:15.299047+0000","last_unstale":"2024-04-16T16:18:17.443410+0000","last_undegraded":"2024-04-16T16:18:17.443410+0000","last_fullsized":"2024-04-16T16:18:17.443410+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:14:18.404852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.320038+0000","last_change":"2024-04-16T16:18:16.324372+0000","last_active":"2024-04-16T16:18:17.320038+0000","last_peered":"2024-04-16T16:18:17.320038+0000","last_clean":"2024-04-16T16:18:17.320038+0000","last_became_active":"2024-04-16T16:18:16.320780+0000","last_became_peered":"2024-04-16T16:18:16.320780+0000","last_unstale":"2024-04-16T16:18:17.320038+0000","last_undegraded":"2024-04-16T16:18:17.320038+0000","last_fullsized":"2024-04-16T16:18:17.320038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:01:15.425073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"84'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.447837+0000","last_change":"2024-04-16T16:18:15.305063+0000","last_active":"2024-04-16T16:18:17.447837+0000","last_peered":"2024-04-16T16:18:17.447837+0000","last_clean":"2024-04-16T16:18:17.447837+0000","last_became_active":"2024-04-16T16:18:15.304669+0000","last_became_peered":"2024-04-16T16:18:15.304669+0000","last_unstale":"2024-04-16T16:18:17.447837+0000","last_undegraded":"2024-04-16T16:18:17.447837+0000","last_fullsized":"2024-04-16T16:18:17.447837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:05:41.836526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.447429+0000","last_change":"2024-04-16T16:18:16.321538+0000","last_active":"2024-04-16T16:18:17.447429+0000","last_peered":"2024-04-16T16:18:17.447429+0000","last_clean":"2024-04-16T16:18:17.447429+0000","last_became_active":"2024-04-16T16:18:16.321194+0000","last_became_peered":"2024-04-16T16:18:16.321194+0000","last_unstale":"2024-04-16T16:18:17.447429+0000","last_undegraded":"2024-04-16T16:18:17.447429+0000","last_fullsized":"2024-04-16T16:18:17.447429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:20.251382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443949+0000","last_change":"2024-04-16T16:18:15.302798+0000","last_active":"2024-04-16T16:18:17.443949+0000","last_peered":"2024-04-16T16:18:17.443949+0000","last_clean":"2024-04-16T16:18:17.443949+0000","last_became_active":"2024-04-16T16:18:15.301509+0000","last_became_peered":"2024-04-16T16:18:15.301509+0000","last_unstale":"2024-04-16T16:18:17.443949+0000","last_undegraded":"2024-04-16T16:18:17.443949+0000","last_fullsized":"2024-04-16T16:18:17.443949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:20:56.511072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.322587+0000","last_change":"2024-04-16T16:18:16.308321+0000","last_active":"2024-04-16T16:18:17.322587+0000","last_peered":"2024-04-16T16:18:17.322587+0000","last_clean":"2024-04-16T16:18:17.322587+0000","last_became_active":"2024-04-16T16:18:16.308004+0000","last_became_peered":"2024-04-16T16:18:16.308004+0000","last_unstale":"2024-04-16T16:18:17.322587+0000","last_undegraded":"2024-04-16T16:18:17.322587+0000","last_fullsized":"2024-04-16T16:18:17.322587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:54:07.935177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443959+0000","last_change":"2024-04-16T16:18:15.302329+0000","last_active":"2024-04-16T16:18:17.443959+0000","last_peered":"2024-04-16T16:18:17.443959+0000","last_clean":"2024-04-16T16:18:17.443959+0000","last_became_active":"2024-04-16T16:18:15.301993+0000","last_became_peered":"2024-04-16T16:18:15.301993+0000","last_unstale":"2024-04-16T16:18:17.443959+0000","last_undegraded":"2024-04-16T16:18:17.443959+0000","last_fullsized":"2024-04-16T16:18:17.443959+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:00:22.780800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.313248+0000","last_change":"2024-04-16T16:18:16.318257+0000","last_active":"2024-04-16T16:18:17.313248+0000","last_peered":"2024-04-16T16:18:17.313248+0000","last_clean":"2024-04-16T16:18:17.313248+0000","last_became_active":"2024-04-16T16:18:16.311533+0000","last_became_peered":"2024-04-16T16:18:16.311533+0000","last_unstale":"2024-04-16T16:18:17.313248+0000","last_undegraded":"2024-04-16T16:18:17.313248+0000","last_fullsized":"2024-04-16T16:18:17.313248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:42:37.193295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394822+0000","last_change":"2024-04-16T16:18:15.291033+0000","last_active":"2024-04-16T16:18:17.394822+0000","last_peered":"2024-04-16T16:18:17.394822+0000","last_clean":"2024-04-16T16:18:17.394822+0000","last_became_active":"2024-04-16T16:18:15.290067+0000","last_became_peered":"2024-04-16T16:18:15.290067+0000","last_unstale":"2024-04-16T16:18:17.394822+0000","last_undegraded":"2024-04-16T16:18:17.394822+0000","last_fullsized":"2024-04-16T16:18:17.394822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:06:23.738648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.445183+0000","last_change":"2024-04-16T16:18:16.310040+0000","last_active":"2024-04-16T16:18:17.445183+0000","last_peered":"2024-04-16T16:18:17.445183+0000","last_clean":"2024-04-16T16:18:17.445183+0000","last_became_active":"2024-04-16T16:18:16.309712+0000","last_became_peered":"2024-04-16T16:18:16.309712+0000","last_unstale":"2024-04-16T16:18:17.445183+0000","last_undegraded":"2024-04-16T16:18:17.445183+0000","last_fullsized":"2024-04-16T16:18:17.445183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:20:57.528152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.313304+0000","last_change":"2024-04-16T16:18:15.301900+0000","last_active":"2024-04-16T16:18:17.313304+0000","last_peered":"2024-04-16T16:18:17.313304+0000","last_clean":"2024-04-16T16:18:17.313304+0000","last_became_active":"2024-04-16T16:18:15.301268+0000","last_became_peered":"2024-04-16T16:18:15.301268+0000","last_unstale":"2024-04-16T16:18:17.313304+0000","last_undegraded":"2024-04-16T16:18:17.313304+0000","last_fullsized":"2024-04-16T16:18:17.313304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:15.061257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312362+0000","last_change":"2024-04-16T16:18:16.314030+0000","last_active":"2024-04-16T16:18:17.312362+0000","last_peered":"2024-04-16T16:18:17.312362+0000","last_clean":"2024-04-16T16:18:17.312362+0000","last_became_active":"2024-04-16T16:18:16.313572+0000","last_became_peered":"2024-04-16T16:18:16.313572+0000","last_unstale":"2024-04-16T16:18:17.312362+0000","last_undegraded":"2024-04-16T16:18:17.312362+0000","last_fullsized":"2024-04-16T16:18:17.312362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:50:11.604716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.312426+0000","last_change":"2024-04-16T16:18:15.308430+0000","last_active":"2024-04-16T16:18:17.312426+0000","last_peered":"2024-04-16T16:18:17.312426+0000","last_clean":"2024-04-16T16:18:17.312426+0000","last_became_active":"2024-04-16T16:18:15.308128+0000","last_became_peered":"2024-04-16T16:18:15.308128+0000","last_unstale":"2024-04-16T16:18:17.312426+0000","last_undegraded":"2024-04-16T16:18:17.312426+0000","last_fullsized":"2024-04-16T16:18:17.312426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:23.975082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444708+0000","last_change":"2024-04-16T16:18:16.313950+0000","last_active":"2024-04-16T16:18:17.444708+0000","last_peered":"2024-04-16T16:18:17.444708+0000","last_clean":"2024-04-16T16:18:17.444708+0000","last_became_active":"2024-04-16T16:18:16.313690+0000","last_became_peered":"2024-04-16T16:18:16.313690+0000","last_unstale":"2024-04-16T16:18:17.444708+0000","last_undegraded":"2024-04-16T16:18:17.444708+0000","last_fullsized":"2024-04-16T16:18:17.444708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:03:49.555993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.394296+0000","last_change":"2024-04-16T16:18:15.292003+0000","last_active":"2024-04-16T16:18:17.394296+0000","last_peered":"2024-04-16T16:18:17.394296+0000","last_clean":"2024-04-16T16:18:17.394296+0000","last_became_active":"2024-04-16T16:18:15.291691+0000","last_became_peered":"2024-04-16T16:18:15.291691+0000","last_unstale":"2024-04-16T16:18:17.394296+0000","last_undegraded":"2024-04-16T16:18:17.394296+0000","last_fullsized":"2024-04-16T16:18:17.394296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:28.452108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.322949+0000","last_change":"2024-04-16T16:18:16.317183+0000","last_active":"2024-04-16T16:18:17.322949+0000","last_peered":"2024-04-16T16:18:17.322949+0000","last_clean":"2024-04-16T16:18:17.322949+0000","last_became_active":"2024-04-16T16:18:16.315296+0000","last_became_peered":"2024-04-16T16:18:16.315296+0000","last_unstale":"2024-04-16T16:18:17.322949+0000","last_undegraded":"2024-04-16T16:18:17.322949+0000","last_fullsized":"2024-04-16T16:18:17.322949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:07:31.030194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.442875+0000","last_change":"2024-04-16T16:18:15.305818+0000","last_active":"2024-04-16T16:18:17.442875+0000","last_peered":"2024-04-16T16:18:17.442875+0000","last_clean":"2024-04-16T16:18:17.442875+0000","last_became_active":"2024-04-16T16:18:15.304636+0000","last_became_peered":"2024-04-16T16:18:15.304636+0000","last_unstale":"2024-04-16T16:18:17.442875+0000","last_undegraded":"2024-04-16T16:18:17.442875+0000","last_fullsized":"2024-04-16T16:18:17.442875+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:39:41.844712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.319760+0000","last_change":"2024-04-16T16:18:16.327566+0000","last_active":"2024-04-16T16:18:17.319760+0000","last_peered":"2024-04-16T16:18:17.319760+0000","last_clean":"2024-04-16T16:18:17.319760+0000","last_became_active":"2024-04-16T16:18:16.327141+0000","last_became_peered":"2024-04-16T16:18:16.327141+0000","last_unstale":"2024-04-16T16:18:17.319760+0000","last_undegraded":"2024-04-16T16:18:17.319760+0000","last_fullsized":"2024-04-16T16:18:17.319760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:16:57.729600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"85'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.443008+0000","last_change":"2024-04-16T16:18:15.305858+0000","last_active":"2024-04-16T16:18:17.443008+0000","last_peered":"2024-04-16T16:18:17.443008+0000","last_clean":"2024-04-16T16:18:17.443008+0000","last_became_active":"2024-04-16T16:18:15.305250+0000","last_became_peered":"2024-04-16T16:18:15.305250+0000","last_unstale":"2024-04-16T16:18:17.443008+0000","last_undegraded":"2024-04-16T16:18:17.443008+0000","last_fullsized":"2024-04-16T16:18:17.443008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:43:27.954843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.446948+0000","last_change":"2024-04-16T16:18:16.309749+0000","last_active":"2024-04-16T16:18:17.446948+0000","last_peered":"2024-04-16T16:18:17.446948+0000","last_clean":"2024-04-16T16:18:17.446948+0000","last_became_active":"2024-04-16T16:18:16.308813+0000","last_became_peered":"2024-04-16T16:18:16.308813+0000","last_unstale":"2024-04-16T16:18:17.446948+0000","last_undegraded":"2024-04-16T16:18:17.446948+0000","last_fullsized":"2024-04-16T16:18:17.446948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:42.181514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.446332+0000","last_change":"2024-04-16T16:18:15.309023+0000","last_active":"2024-04-16T16:18:17.446332+0000","last_peered":"2024-04-16T16:18:17.446332+0000","last_clean":"2024-04-16T16:18:17.446332+0000","last_became_active":"2024-04-16T16:18:15.308399+0000","last_became_peered":"2024-04-16T16:18:15.308399+0000","last_unstale":"2024-04-16T16:18:17.446332+0000","last_undegraded":"2024-04-16T16:18:17.446332+0000","last_fullsized":"2024-04-16T16:18:17.446332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:30:55.552396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.322680+0000","last_change":"2024-04-16T16:18:16.313627+0000","last_active":"2024-04-16T16:18:17.322680+0000","last_peered":"2024-04-16T16:18:17.322680+0000","last_clean":"2024-04-16T16:18:17.322680+0000","last_became_active":"2024-04-16T16:18:16.313148+0000","last_became_peered":"2024-04-16T16:18:16.313148+0000","last_unstale":"2024-04-16T16:18:17.322680+0000","last_undegraded":"2024-04-16T16:18:17.322680+0000","last_fullsized":"2024-04-16T16:18:17.322680+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:08.532092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.447072+0000","last_change":"2024-04-16T16:18:15.311086+0000","last_active":"2024-04-16T16:18:17.447072+0000","last_peered":"2024-04-16T16:18:17.447072+0000","last_clean":"2024-04-16T16:18:17.447072+0000","last_became_active":"2024-04-16T16:18:15.310594+0000","last_became_peered":"2024-04-16T16:18:15.310594+0000","last_unstale":"2024-04-16T16:18:17.447072+0000","last_undegraded":"2024-04-16T16:18:17.447072+0000","last_fullsized":"2024-04-16T16:18:17.447072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:34:28.283109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.445738+0000","last_change":"2024-04-16T16:18:16.308872+0000","last_active":"2024-04-16T16:18:17.445738+0000","last_peered":"2024-04-16T16:18:17.445738+0000","last_clean":"2024-04-16T16:18:17.445738+0000","last_became_active":"2024-04-16T16:18:16.308461+0000","last_became_peered":"2024-04-16T16:18:16.308461+0000","last_unstale":"2024-04-16T16:18:17.445738+0000","last_undegraded":"2024-04-16T16:18:17.445738+0000","last_fullsized":"2024-04-16T16:18:17.445738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:08.532092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444049+0000","last_change":"2024-04-16T16:18:15.299375+0000","last_active":"2024-04-16T16:18:17.444049+0000","last_peered":"2024-04-16T16:18:17.444049+0000","last_clean":"2024-04-16T16:18:17.444049+0000","last_became_active":"2024-04-16T16:18:15.298710+0000","last_became_peered":"2024-04-16T16:18:15.298710+0000","last_unstale":"2024-04-16T16:18:17.444049+0000","last_undegraded":"2024-04-16T16:18:17.444049+0000","last_fullsized":"2024-04-16T16:18:17.444049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:12:39.298882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.444137+0000","last_change":"2024-04-16T16:18:16.322248+0000","last_active":"2024-04-16T16:18:17.444137+0000","last_peered":"2024-04-16T16:18:17.444137+0000","last_clean":"2024-04-16T16:18:17.444137+0000","last_became_active":"2024-04-16T16:18:16.318615+0000","last_became_peered":"2024-04-16T16:18:16.318615+0000","last_unstale":"2024-04-16T16:18:17.444137+0000","last_undegraded":"2024-04-16T16:18:17.444137+0000","last_fullsized":"2024-04-16T16:18:17.444137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:16:57.729600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.321697+0000","last_change":"2024-04-16T16:18:15.298300+0000","last_active":"2024-04-16T16:18:17.321697+0000","last_peered":"2024-04-16T16:18:17.321697+0000","last_clean":"2024-04-16T16:18:17.321697+0000","last_became_active":"2024-04-16T16:18:15.297872+0000","last_became_peered":"2024-04-16T16:18:15.297872+0000","last_unstale":"2024-04-16T16:18:17.321697+0000","last_undegraded":"2024-04-16T16:18:17.321697+0000","last_fullsized":"2024-04-16T16:18:17.321697+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:17:46.901396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.318928+0000","last_change":"2024-04-16T16:18:16.310865+0000","last_active":"2024-04-16T16:18:17.318928+0000","last_peered":"2024-04-16T16:18:17.318928+0000","last_clean":"2024-04-16T16:18:17.318928+0000","last_became_active":"2024-04-16T16:18:16.310557+0000","last_became_peered":"2024-04-16T16:18:16.310557+0000","last_unstale":"2024-04-16T16:18:17.318928+0000","last_undegraded":"2024-04-16T16:18:17.318928+0000","last_fullsized":"2024-04-16T16:18:17.318928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:07:31.030194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.445913+0000","last_change":"2024-04-16T16:18:15.308009+0000","last_active":"2024-04-16T16:18:17.445913+0000","last_peered":"2024-04-16T16:18:17.445913+0000","last_clean":"2024-04-16T16:18:17.445913+0000","last_became_active":"2024-04-16T16:18:15.307561+0000","last_became_peered":"2024-04-16T16:18:15.307561+0000","last_unstale":"2024-04-16T16:18:17.445913+0000","last_undegraded":"2024-04-16T16:18:17.445913+0000","last_fullsized":"2024-04-16T16:18:17.445913+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:37:10.400769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.445688+0000","last_change":"2024-04-16T16:18:16.312174+0000","last_active":"2024-04-16T16:18:17.445688+0000","last_peered":"2024-04-16T16:18:17.445688+0000","last_clean":"2024-04-16T16:18:17.445688+0000","last_became_active":"2024-04-16T16:18:16.305979+0000","last_became_peered":"2024-04-16T16:18:16.305979+0000","last_unstale":"2024-04-16T16:18:17.445688+0000","last_undegraded":"2024-04-16T16:18:17.445688+0000","last_fullsized":"2024-04-16T16:18:17.445688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:14:19.421933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.321258+0000","last_change":"2024-04-16T16:18:15.309487+0000","last_active":"2024-04-16T16:18:17.321258+0000","last_peered":"2024-04-16T16:18:17.321258+0000","last_clean":"2024-04-16T16:18:17.321258+0000","last_became_active":"2024-04-16T16:18:15.308448+0000","last_became_peered":"2024-04-16T16:18:15.308448+0000","last_unstale":"2024-04-16T16:18:17.321258+0000","last_undegraded":"2024-04-16T16:18:17.321258+0000","last_fullsized":"2024-04-16T16:18:17.321258+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:20.446864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.442709+0000","last_change":"2024-04-16T16:18:16.321228+0000","last_active":"2024-04-16T16:18:17.442709+0000","last_peered":"2024-04-16T16:18:17.442709+0000","last_clean":"2024-04-16T16:18:17.442709+0000","last_became_active":"2024-04-16T16:18:16.320916+0000","last_became_peered":"2024-04-16T16:18:16.320916+0000","last_unstale":"2024-04-16T16:18:17.442709+0000","last_undegraded":"2024-04-16T16:18:17.442709+0000","last_fullsized":"2024-04-16T16:18:17.442709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:15.256054+0000","last_clean_scrub_stamp":"2024-04-16T16:18:15.256054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:17:47.918477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"84'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.319938+0000","last_change":"2024-04-16T16:18:15.301975+0000","last_active":"2024-04-16T16:18:17.319938+0000","last_peered":"2024-04-16T16:18:17.319938+0000","last_clean":"2024-04-16T16:18:17.319938+0000","last_became_active":"2024-04-16T16:18:15.300433+0000","last_became_peered":"2024-04-16T16:18:15.300433+0000","last_unstale":"2024-04-16T16:18:17.319938+0000","last_undegraded":"2024-04-16T16:18:17.319938+0000","last_fullsized":"2024-04-16T16:18:17.319938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:18:14.238974+0000","last_clean_scrub_stamp":"2024-04-16T16:18:14.238974+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:52:23.158135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"74'161","reported_seq":456,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-16T16:18:17.318787+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:18:17.318787+0000","last_peered":"2024-04-16T16:18:17.318787+0000","last_clean":"2024-04-16T16:18:17.318787+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:18:17.318787+0000","last_undegraded":"2024-04-16T16:18:17.318787+0000","last_fullsized":"2024-04-16T16:18:17.318787+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574908,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32788,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":1642496,"data_stored":1081844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574909,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339628,"statfs":{"total":96636764160,"available":96603779072,"internally_reserved":0,"allocated":1056768,"data_stored":492100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":493040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574910,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339640,"statfs":{"total":96636764160,"available":96603791360,"internally_reserved":0,"allocated":1044480,"data_stored":491406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339628,"statfs":{"total":96636764160,"available":96603779072,"internally_reserved":0,"allocated":1056768,"data_stored":493018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574908,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1048576,"data_stored":491988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339628,"statfs":{"total":96636764160,"available":96603779072,"internally_reserved":0,"allocated":1056768,"data_stored":491600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574909,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32788,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":1642496,"data_stored":1081808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:18:26.522 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-16T16:18:26.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-16T16:18:26.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.767+0000 7f7c5f200640 1 Processor -- start 2024-04-16T16:18:26.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.767+0000 7f7c5f200640 1 -- start start 2024-04-16T16:18:26.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.769+0000 7f7c5f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7c601452e0 0x7f7c6006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.769+0000 7f7c5f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c600cecc0 0x7f7c6015cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.769+0000 7f7c5f200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c60071e60 con 0x7f7c6006e920 2024-04-16T16:18:26.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.769+0000 7f7c5f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7c60072f00 con 0x7f7c601452e0 2024-04-16T16:18:26.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.769+0000 7f7c5f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7c6006ed90 con 0x7f7c600cecc0 2024-04-16T16:18:26.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.770+0000 7f7c55400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7c6006e920 0x7f7c600ce5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41134/0 (socket says 172.21.15.45:41134) 2024-04-16T16:18:26.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.770+0000 7f7c55400640 1 -- 172.21.15.45:0/1343231272 learned_addr learned my addr 172.21.15.45:0/1343231272 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:26.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.770+0000 7f7c5e800640 1 --2- 172.21.15.45:0/1343231272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c600cecc0 0x7f7c6015cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.770+0000 7f7c5de00640 1 --2- 172.21.15.45:0/1343231272 >> v2:172.21.15.103:3300/0 conn(0x7f7c601452e0 0x7f7c6006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.770+0000 7f7c5d400640 1 -- 172.21.15.45:0/1343231272 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3720921191 0 0) 0x7f7c60071e60 con 0x7f7c6006e920 2024-04-16T16:18:26.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.770+0000 7f7c5d400640 1 -- 172.21.15.45:0/1343231272 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c44003520 con 0x7f7c6006e920 2024-04-16T16:18:26.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c5d400640 1 -- 172.21.15.45:0/1343231272 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 87727619 0 0) 0x7f7c44003520 con 0x7f7c6006e920 2024-04-16T16:18:26.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c5d400640 1 -- 172.21.15.45:0/1343231272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c600cecc0 msgr2=0x7f7c6015cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:26.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c5d400640 1 --2- 172.21.15.45:0/1343231272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c600cecc0 0x7f7c6015cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c5d400640 1 -- 172.21.15.45:0/1343231272 >> v2:172.21.15.103:3300/0 conn(0x7f7c601452e0 msgr2=0x7f7c6006e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:26.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c5d400640 1 --2- 172.21.15.45:0/1343231272 >> v2:172.21.15.103:3300/0 conn(0x7f7c601452e0 0x7f7c6006e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c5d400640 1 -- 172.21.15.45:0/1343231272 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c60072140 con 0x7f7c6006e920 2024-04-16T16:18:26.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c5d400640 1 -- 172.21.15.45:0/1343231272 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f7c48001690 con 0x7f7c6006e920 2024-04-16T16:18:26.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c5d400640 1 -- 172.21.15.45:0/1343231272 >> v1:172.21.15.45:6789/0 conn(0x7f7c6006e920 legacy=0x7f7c600ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:26.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c5d400640 1 --2- 172.21.15.45:0/1343231272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c440039e0 0x7f7c44003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c5d400640 1 --2- 172.21.15.45:0/1343231272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c44004800 0x7f7c44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c5d400640 1 --2- 172.21.15.45:0/1343231272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c440051c0 0x7f7c4400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c5d400640 1 -- 172.21.15.45:0/1343231272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7c60072f00 con 0x7f7c44004800 2024-04-16T16:18:26.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c5d400640 1 -- 172.21.15.45:0/1343231272 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6006ed90 con 0x7f7c440039e0 2024-04-16T16:18:26.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c5d400640 1 -- 172.21.15.45:0/1343231272 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4400bf00 con 0x7f7c440051c0 2024-04-16T16:18:26.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c5e800640 1 --2- 172.21.15.45:0/1343231272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c600cecc0 0x7f7c6015cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:18:26.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.771+0000 7f7c55400640 1 --2- 172.21.15.45:0/1343231272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c44004800 0x7f7c44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.772+0000 7f7c5de00640 1 --2- 172.21.15.45:0/1343231272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c440039e0 0x7f7c44003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.772+0000 7f7c5e800640 1 --2- 172.21.15.45:0/1343231272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c440051c0 0x7f7c4400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.772+0000 7f7c55400640 1 -- 172.21.15.45:0/1343231272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c440051c0 msgr2=0x7f7c4400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:26.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.772+0000 7f7c55400640 1 --2- 172.21.15.45:0/1343231272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c440051c0 0x7f7c4400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.772+0000 7f7c55400640 1 -- 172.21.15.45:0/1343231272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c440039e0 msgr2=0x7f7c44003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:26.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.772+0000 7f7c55400640 1 --2- 172.21.15.45:0/1343231272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c440039e0 0x7f7c44003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.772+0000 7f7c55400640 1 -- 172.21.15.45:0/1343231272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c44003520 con 0x7f7c44004800 2024-04-16T16:18:26.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.772+0000 7f7c5e800640 1 --2- 172.21.15.45:0/1343231272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c440051c0 0x7f7c4400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:26.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.772+0000 7f7c5de00640 1 --2- 172.21.15.45:0/1343231272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c440039e0 0x7f7c44003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:26.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.773+0000 7f7c55400640 1 --2- 172.21.15.45:0/1343231272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c44004800 0x7f7c44004c80 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800c210 tx=0x7f7c44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d97c1cdb60cdcbe server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:26.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.773+0000 7f7c5d400640 1 -- 172.21.15.45:0/1343231272 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48007d10 con 0x7f7c44004800 2024-04-16T16:18:26.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.773+0000 7f7c5d400640 1 -- 172.21.15.45:0/1343231272 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c48030070 con 0x7f7c44004800 2024-04-16T16:18:26.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.773+0000 7f7c5d400640 1 -- 172.21.15.45:0/1343231272 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48005d10 con 0x7f7c44004800 2024-04-16T16:18:26.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.774+0000 7f7c5f200640 1 -- 172.21.15.45:0/1343231272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c44004800 msgr2=0x7f7c44004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:26.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.774+0000 7f7c5f200640 1 --2- 172.21.15.45:0/1343231272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c44004800 0x7f7c44004c80 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800c210 tx=0x7f7c44005830 comp rx=0 tx=0).stop 2024-04-16T16:18:26.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.774+0000 7f7c5e800640 1 -- 172.21.15.45:0/1343231272 reap_dead start 2024-04-16T16:18:26.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.774+0000 7f7c5f200640 1 -- 172.21.15.45:0/1343231272 shutdown_connections 2024-04-16T16:18:26.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.774+0000 7f7c5f200640 1 -- 172.21.15.45:0/1343231272 >> 172.21.15.45:0/1343231272 conn(0x7f7c600b3bb0 msgr2=0x7f7c600b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:26.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.774+0000 7f7c5f200640 1 -- 172.21.15.45:0/1343231272 shutdown_connections 2024-04-16T16:18:26.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.774+0000 7f7c5f200640 1 -- 172.21.15.45:0/1343231272 wait complete. 2024-04-16T16:18:26.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.776+0000 7f7c5f200640 1 Processor -- start 2024-04-16T16:18:26.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.776+0000 7f7c5f200640 1 -- start start 2024-04-16T16:18:26.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.777+0000 7f7c5f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c44004800 0x7f7c600ae730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.777+0000 7f7c5f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c600b1000 0x7f7c600afe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.777+0000 7f7c5f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c601f3ea0 0x7f7c601f4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.777+0000 7f7c5f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7c600729e0 con 0x7f7c44004800 2024-04-16T16:18:26.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.777+0000 7f7c5f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7c60072700 con 0x7f7c600b1000 2024-04-16T16:18:26.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.777+0000 7f7c5f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7c601456c0 con 0x7f7c601f3ea0 2024-04-16T16:18:26.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.777+0000 7f7c5de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c44004800 0x7f7c600ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.778+0000 7f7c5de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c44004800 0x7f7c600ae730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57600/0 (socket says 172.21.15.45:57600) 2024-04-16T16:18:26.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.778+0000 7f7c5de00640 1 -- 172.21.15.45:0/2705192434 learned_addr learned my addr 172.21.15.45:0/2705192434 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:26.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.778+0000 7f7c5e800640 1 --2- 172.21.15.45:0/2705192434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c601f3ea0 0x7f7c601f4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.778+0000 7f7c55400640 1 --2- 172.21.15.45:0/2705192434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c600b1000 0x7f7c600afe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.778+0000 7f7c5de00640 1 -- 172.21.15.45:0/2705192434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c601f3ea0 msgr2=0x7f7c601f4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:26.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.778+0000 7f7c5de00640 1 --2- 172.21.15.45:0/2705192434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c601f3ea0 0x7f7c601f4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.778+0000 7f7c5de00640 1 -- 172.21.15.45:0/2705192434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c600b1000 msgr2=0x7f7c600afe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:26.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.778+0000 7f7c5de00640 1 --2- 172.21.15.45:0/2705192434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c600b1000 0x7f7c600afe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:26.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.778+0000 7f7c5de00640 1 -- 172.21.15.45:0/2705192434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c600aedc0 con 0x7f7c44004800 2024-04-16T16:18:26.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.778+0000 7f7c55400640 1 --2- 172.21.15.45:0/2705192434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c600b1000 0x7f7c600afe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:26.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.778+0000 7f7c5de00640 1 --2- 172.21.15.45:0/2705192434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c44004800 0x7f7c600ae730 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f7c5000f3d0 tx=0x7f7c5000f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:26.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.779+0000 7f7c56a00640 1 -- 172.21.15.45:0/2705192434 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50003990 con 0x7f7c44004800 2024-04-16T16:18:26.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.779+0000 7f7c56a00640 1 -- 172.21.15.45:0/2705192434 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c50003b30 con 0x7f7c44004800 2024-04-16T16:18:26.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.779+0000 7f7c56a00640 1 -- 172.21.15.45:0/2705192434 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50004020 con 0x7f7c44004800 2024-04-16T16:18:26.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.779+0000 7f7c5f200640 1 -- 172.21.15.45:0/2705192434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c600af0a0 con 0x7f7c44004800 2024-04-16T16:18:26.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.779+0000 7f7c5f200640 1 -- 172.21.15.45:0/2705192434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c600af660 con 0x7f7c44004800 2024-04-16T16:18:26.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.780+0000 7f7c5f200640 1 -- 172.21.15.45:0/2705192434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c60075780 con 0x7f7c44004800 2024-04-16T16:18:26.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.781+0000 7f7c56a00640 1 -- 172.21.15.45:0/2705192434 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f7c50005d10 con 0x7f7c44004800 2024-04-16T16:18:26.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.782+0000 7f7c56a00640 1 --2- 172.21.15.45:0/2705192434 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7c3c06ca10 0x7f7c3c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:26.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.782+0000 7f7c56a00640 1 -- 172.21.15.45:0/2705192434 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f7c5008c970 con 0x7f7c44004800 2024-04-16T16:18:26.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.783+0000 7f7c55400640 1 --2- 172.21.15.45:0/2705192434 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7c3c06ca10 0x7f7c3c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:26.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.784+0000 7f7c55400640 1 --2- 172.21.15.45:0/2705192434 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7c3c06ca10 0x7f7c3c06eed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7c48034000 tx=0x7f7c48007000 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:26.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:26.786+0000 7f7c56a00640 1 -- 172.21.15.45:0/2705192434 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7c50024020 con 0x7f7c44004800 2024-04-16T16:18:27.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.074+0000 7f7c5f200640 1 -- 172.21.15.45:0/2705192434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f7c6006cf70 con 0x7f7c44004800 2024-04-16T16:18:27.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.078+0000 7f7c56a00640 1 -- 172.21.15.45:0/2705192434 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) ==== 190+0+147 (secure 0 0 0) 0x7f7c5005aae0 con 0x7f7c44004800 2024-04-16T16:18:27.077 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-04-16T16:18:27.078 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ== 2024-04-16T16:18:27.078 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:18:27.078 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:18:27.078 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-16T16:18:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2024-04-16T16:18:27.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.081+0000 7f7c5f200640 1 -- 172.21.15.45:0/2705192434 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7c3c06ca10 msgr2=0x7f7c3c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:27.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.081+0000 7f7c5f200640 1 --2- 172.21.15.45:0/2705192434 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7c3c06ca10 0x7f7c3c06eed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7c48034000 tx=0x7f7c48007000 comp rx=0 tx=0).stop 2024-04-16T16:18:27.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.081+0000 7f7c5f200640 1 -- 172.21.15.45:0/2705192434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c44004800 msgr2=0x7f7c600ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.081+0000 7f7c5f200640 1 --2- 172.21.15.45:0/2705192434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c44004800 0x7f7c600ae730 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f7c5000f3d0 tx=0x7f7c5000f890 comp rx=0 tx=0).stop 2024-04-16T16:18:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.081+0000 7f7c5f200640 1 -- 172.21.15.45:0/2705192434 shutdown_connections 2024-04-16T16:18:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.081+0000 7f7c5f200640 1 --2- 172.21.15.45:0/2705192434 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7c3c06ca10 0x7f7c3c06eed0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.081+0000 7f7c5f200640 1 --2- 172.21.15.45:0/2705192434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c601f3ea0 0x7f7c601f4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.081+0000 7f7c5f200640 1 --2- 172.21.15.45:0/2705192434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c600b1000 0x7f7c600afe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.081+0000 7f7c5f200640 1 --2- 172.21.15.45:0/2705192434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c44004800 0x7f7c600ae730 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.081+0000 7f7c5f200640 1 -- 172.21.15.45:0/2705192434 >> 172.21.15.45:0/2705192434 conn(0x7f7c600b3bb0 msgr2=0x7f7c60142f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.081+0000 7f7c5f200640 1 -- 172.21.15.45:0/2705192434 shutdown_connections 2024-04-16T16:18:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.081+0000 7f7c5f200640 1 -- 172.21.15.45:0/2705192434 wait complete. 2024-04-16T16:18:27.101 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:18:27.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.325+0000 7f658f200640 1 Processor -- start 2024-04-16T16:18:27.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.325+0000 7f658f200640 1 -- start start 2024-04-16T16:18:27.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.327+0000 7f658f200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6590146220 0x7f659015ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:27.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.327+0000 7f658f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f659006f7a0 0x7f659006df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:27.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.327+0000 7f658f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f659015d360 con 0x7f659006e480 2024-04-16T16:18:27.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.327+0000 7f658f200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f6590160410 con 0x7f6590146220 2024-04-16T16:18:27.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.327+0000 7f658f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f659015d640 con 0x7f659006f7a0 2024-04-16T16:18:27.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.327+0000 7f658de00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6590146220 0x7f659015ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:27.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.327+0000 7f658de00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6590146220 0x7f659015ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57614/0 (socket says 172.21.15.45:57614) 2024-04-16T16:18:27.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.327+0000 7f658de00640 1 -- 172.21.15.45:0/2700864464 learned_addr learned my addr 172.21.15.45:0/2700864464 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:27.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.327+0000 7f658de00640 1 -- 172.21.15.45:0/2700864464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f659006f7a0 msgr2=0x7f659006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:27.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.328+0000 7f658de00640 1 --2- 172.21.15.45:0/2700864464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f659006f7a0 0x7f659006df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:27.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.328+0000 7f658de00640 1 -- 172.21.15.45:0/2700864464 >> v1:172.21.15.103:6789/0 conn(0x7f659006e480 legacy=0x7f65901608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:27.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.328+0000 7f658de00640 1 -- 172.21.15.45:0/2700864464 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f659015d920 con 0x7f6590146220 2024-04-16T16:18:27.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.329+0000 7f658de00640 1 --2- 172.21.15.45:0/2700864464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6590146220 0x7f659015ccb0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f6580009870 tx=0x7f658002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2093e8f11af54d61 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.329+0000 7f658d400640 1 -- 172.21.15.45:0/2700864464 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6580035020 con 0x7f6590146220 2024-04-16T16:18:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.329+0000 7f658d400640 1 -- 172.21.15.45:0/2700864464 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6580031640 con 0x7f6590146220 2024-04-16T16:18:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.329+0000 7f658d400640 1 -- 172.21.15.45:0/2700864464 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658002d040 con 0x7f6590146220 2024-04-16T16:18:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.330+0000 7f658f200640 1 -- 172.21.15.45:0/2700864464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6590146220 msgr2=0x7f659015ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.330+0000 7f658f200640 1 --2- 172.21.15.45:0/2700864464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6590146220 0x7f659015ccb0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f6580009870 tx=0x7f658002b8f0 comp rx=0 tx=0).stop 2024-04-16T16:18:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.330+0000 7f658f200640 1 -- 172.21.15.45:0/2700864464 shutdown_connections 2024-04-16T16:18:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.330+0000 7f658f200640 1 --2- 172.21.15.45:0/2700864464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f659006f7a0 0x7f659006df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.330+0000 7f658f200640 1 --2- 172.21.15.45:0/2700864464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6590146220 0x7f659015ccb0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.330+0000 7f658f200640 1 -- 172.21.15.45:0/2700864464 >> 172.21.15.45:0/2700864464 conn(0x7f659006b930 msgr2=0x7f659006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.330+0000 7f658f200640 1 -- 172.21.15.45:0/2700864464 shutdown_connections 2024-04-16T16:18:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.330+0000 7f658f200640 1 -- 172.21.15.45:0/2700864464 wait complete. 2024-04-16T16:18:27.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.332+0000 7f658f200640 1 Processor -- start 2024-04-16T16:18:27.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.332+0000 7f658f200640 1 -- start start 2024-04-16T16:18:27.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.333+0000 7f658f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f659006e480 0x7f659013dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:27.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.333+0000 7f658f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f659006f7a0 0x7f65900ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:27.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.333+0000 7f658f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6590146220 0x7f65901382d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:27.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.333+0000 7f658f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f659006fda0 con 0x7f659006e480 2024-04-16T16:18:27.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.333+0000 7f658f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f659006fc20 con 0x7f6590146220 2024-04-16T16:18:27.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.333+0000 7f658f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6590160fb0 con 0x7f659006f7a0 2024-04-16T16:18:27.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.333+0000 7f658de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f659006e480 0x7f659013dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:27.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.333+0000 7f658de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f659006e480 0x7f659013dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57624/0 (socket says 172.21.15.45:57624) 2024-04-16T16:18:27.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.333+0000 7f658de00640 1 -- 172.21.15.45:0/936626217 learned_addr learned my addr 172.21.15.45:0/936626217 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:27.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.334+0000 7f658de00640 1 -- 172.21.15.45:0/936626217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f659006f7a0 msgr2=0x7f65900ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:27.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.334+0000 7f6587e00640 1 --2- 172.21.15.45:0/936626217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f659006f7a0 0x7f65900ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:27.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.334+0000 7f658de00640 1 --2- 172.21.15.45:0/936626217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f659006f7a0 0x7f65900ceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:27.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.334+0000 7f658de00640 1 -- 172.21.15.45:0/936626217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6590146220 msgr2=0x7f65901382d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:27.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.334+0000 7f658de00640 1 --2- 172.21.15.45:0/936626217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6590146220 0x7f65901382d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:27.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.334+0000 7f658de00640 1 -- 172.21.15.45:0/936626217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f659013ccd0 con 0x7f659006e480 2024-04-16T16:18:27.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.334+0000 7f6587e00640 1 --2- 172.21.15.45:0/936626217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f659006f7a0 0x7f65900ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:27.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.334+0000 7f658de00640 1 --2- 172.21.15.45:0/936626217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f659006e480 0x7f659013dd70 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f6580000c00 tx=0x7f65800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:27.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.334+0000 7f6586000640 1 -- 172.21.15.45:0/936626217 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658003ac60 con 0x7f659006e480 2024-04-16T16:18:27.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.334+0000 7f6586000640 1 -- 172.21.15.45:0/936626217 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f658003ae20 con 0x7f659006e480 2024-04-16T16:18:27.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.334+0000 7f658f200640 1 -- 172.21.15.45:0/936626217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f659013cfb0 con 0x7f659006e480 2024-04-16T16:18:27.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.334+0000 7f6586000640 1 -- 172.21.15.45:0/936626217 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6580044630 con 0x7f659006e480 2024-04-16T16:18:27.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.335+0000 7f658f200640 1 -- 172.21.15.45:0/936626217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f659013d850 con 0x7f659006e480 2024-04-16T16:18:27.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.336+0000 7f658f200640 1 -- 172.21.15.45:0/936626217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f659013d3a0 con 0x7f659006e480 2024-04-16T16:18:27.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.337+0000 7f6586000640 1 -- 172.21.15.45:0/936626217 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f658004b060 con 0x7f659006e480 2024-04-16T16:18:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.338+0000 7f6586000640 1 --2- 172.21.15.45:0/936626217 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f656c06ca10 0x7f656c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.338+0000 7f6586000640 1 -- 172.21.15.45:0/936626217 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f65800adae0 con 0x7f659006e480 2024-04-16T16:18:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.338+0000 7f6587e00640 1 --2- 172.21.15.45:0/936626217 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f656c06ca10 0x7f656c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:27.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.340+0000 7f6587e00640 1 --2- 172.21.15.45:0/936626217 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f656c06ca10 0x7f656c06eed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6578006fd0 tx=0x7f6578008040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.344+0000 7f6586000640 1 -- 172.21.15.45:0/936626217 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f658003f070 con 0x7f659006e480 2024-04-16T16:18:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.593+0000 7f658f200640 1 -- 172.21.15.45:0/936626217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6590075780 con 0x7f659006e480 2024-04-16T16:18:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.595+0000 7f6586000640 1 -- 172.21.15.45:0/936626217 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5186 (secure 0 0 0) 0x7f658007bcf0 con 0x7f659006e480 2024-04-16T16:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2514711815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2514711815},{"type":"v1","addr":"172.21.15.103:6836","nonce":2514711815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2189404408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2189404408},{"type":"v1","addr":"172.21.15.103:6837","nonce":2189404408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7179,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/991116998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":991116998},{"type":"v1","addr":"172.21.15.45:6834","nonce":991116998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:18:15.762294+0000","modified":"2024-04-16T16:18:18.858040+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7188":{"gid":7188,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3813781309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3813781309},{"type":"v1","addr":"172.21.15.45:6835","nonce":3813781309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7188,"qdb_cluster":[7188]},"id":6}]} 2024-04-16T16:18:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 64 2024-04-16T16:18:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.598+0000 7f658f200640 1 -- 172.21.15.45:0/936626217 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f656c06ca10 msgr2=0x7f656c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.598+0000 7f658f200640 1 --2- 172.21.15.45:0/936626217 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f656c06ca10 0x7f656c06eed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6578006fd0 tx=0x7f6578008040 comp rx=0 tx=0).stop 2024-04-16T16:18:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.598+0000 7f658f200640 1 -- 172.21.15.45:0/936626217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f659006e480 msgr2=0x7f659013dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.598+0000 7f658f200640 1 --2- 172.21.15.45:0/936626217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f659006e480 0x7f659013dd70 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f6580000c00 tx=0x7f65800046c0 comp rx=0 tx=0).stop 2024-04-16T16:18:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.598+0000 7f658f200640 1 -- 172.21.15.45:0/936626217 shutdown_connections 2024-04-16T16:18:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.598+0000 7f658f200640 1 --2- 172.21.15.45:0/936626217 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f656c06ca10 0x7f656c06eed0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.598+0000 7f658f200640 1 --2- 172.21.15.45:0/936626217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6590146220 0x7f65901382d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.598+0000 7f658f200640 1 --2- 172.21.15.45:0/936626217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f659006f7a0 0x7f65900ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.598+0000 7f658f200640 1 --2- 172.21.15.45:0/936626217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f659006e480 0x7f659013dd70 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.598+0000 7f658f200640 1 -- 172.21.15.45:0/936626217 >> 172.21.15.45:0/936626217 conn(0x7f659006b930 msgr2=0x7f659015c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.598+0000 7f658f200640 1 -- 172.21.15.45:0/936626217 shutdown_connections 2024-04-16T16:18:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:27.598+0000 7f658f200640 1 -- 172.21.15.45:0/936626217 wait complete. 2024-04-16T16:18:28.022 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:18:15.762294+0000', 'modified': '2024-04-16T16:18:18.858040+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7188}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7188': {'gid': 7188, 'name': 'a', 'rank': 0, 'incarnation': 60, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6835/3813781309', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6833', 'nonce': 3813781309}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 3813781309}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7188, 'qdb_cluster': [7188]} 2024-04-16T16:18:28.022 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-16T16:18:28.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-16T16:18:28.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.259+0000 7f3f47200640 1 Processor -- start 2024-04-16T16:18:28.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.260+0000 7f3f47200640 1 -- start start 2024-04-16T16:18:28.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.261+0000 7f3f47200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f400ba140 0x7f3f400d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.261+0000 7f3f47200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3f400cfc20 0x7f3f400d35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.261+0000 7f3f47200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f400d2600 con 0x7f3f40162090 2024-04-16T16:18:28.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.261+0000 7f3f47200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3f400ba8b0 con 0x7f3f400cfc20 2024-04-16T16:18:28.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.261+0000 7f3f47200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3f400ba5d0 con 0x7f3f400ba140 2024-04-16T16:18:28.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.261+0000 7f3f46800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3f400cfc20 0x7f3f400d35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.262+0000 7f3f46800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3f400cfc20 0x7f3f400d35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57634/0 (socket says 172.21.15.45:57634) 2024-04-16T16:18:28.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.262+0000 7f3f46800640 1 -- 172.21.15.45:0/3393354407 learned_addr learned my addr 172.21.15.45:0/3393354407 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:28.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.262+0000 7f3f45400640 1 --2- 172.21.15.45:0/3393354407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f400ba140 0x7f3f400d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.262+0000 7f3f46800640 1 -- 172.21.15.45:0/3393354407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f400ba140 msgr2=0x7f3f400d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:28.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.262+0000 7f3f46800640 1 --2- 172.21.15.45:0/3393354407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f400ba140 0x7f3f400d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.262+0000 7f3f46800640 1 -- 172.21.15.45:0/3393354407 >> v1:172.21.15.45:6789/0 conn(0x7f3f40162090 legacy=0x7f3f400cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:28.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.262+0000 7f3f46800640 1 -- 172.21.15.45:0/3393354407 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f400d28a0 con 0x7f3f400cfc20 2024-04-16T16:18:28.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.263+0000 7f3f46800640 1 --2- 172.21.15.45:0/3393354407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f400cfc20 0x7f3f400d35e0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009870 tx=0x7f3f3c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf605a149ff59a91 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:28.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.263+0000 7f3f44a00640 1 -- 172.21.15.45:0/3393354407 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c034070 con 0x7f3f400cfc20 2024-04-16T16:18:28.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.264+0000 7f3f44a00640 1 -- 172.21.15.45:0/3393354407 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f3c030840 con 0x7f3f400cfc20 2024-04-16T16:18:28.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.264+0000 7f3f44a00640 1 -- 172.21.15.45:0/3393354407 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c030c10 con 0x7f3f400cfc20 2024-04-16T16:18:28.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.264+0000 7f3f47200640 1 -- 172.21.15.45:0/3393354407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f400cfc20 msgr2=0x7f3f400d35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:28.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.264+0000 7f3f47200640 1 --2- 172.21.15.45:0/3393354407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f400cfc20 0x7f3f400d35e0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009870 tx=0x7f3f3c02b980 comp rx=0 tx=0).stop 2024-04-16T16:18:28.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.264+0000 7f3f47200640 1 -- 172.21.15.45:0/3393354407 shutdown_connections 2024-04-16T16:18:28.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.264+0000 7f3f47200640 1 --2- 172.21.15.45:0/3393354407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f400cfc20 0x7f3f400d35e0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.264+0000 7f3f47200640 1 --2- 172.21.15.45:0/3393354407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f400ba140 0x7f3f400d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.264+0000 7f3f47200640 1 -- 172.21.15.45:0/3393354407 >> 172.21.15.45:0/3393354407 conn(0x7f3f400d1940 msgr2=0x7f3f400d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:28.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.264+0000 7f3f47200640 1 -- 172.21.15.45:0/3393354407 shutdown_connections 2024-04-16T16:18:28.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.264+0000 7f3f47200640 1 -- 172.21.15.45:0/3393354407 wait complete. 2024-04-16T16:18:28.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.266+0000 7f3f47200640 1 Processor -- start 2024-04-16T16:18:28.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.266+0000 7f3f47200640 1 -- start start 2024-04-16T16:18:28.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.267+0000 7f3f47200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f400cfc20 0x7f3f400c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.267+0000 7f3f47200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40162090 0x7f3f400ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.267+0000 7f3f47200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f400cd2f0 0x7f3f400b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.267+0000 7f3f47200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3f401625f0 con 0x7f3f400cfc20 2024-04-16T16:18:28.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.267+0000 7f3f47200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3f40162470 con 0x7f3f40162090 2024-04-16T16:18:28.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.267+0000 7f3f47200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3f400d5520 con 0x7f3f400cd2f0 2024-04-16T16:18:28.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.267+0000 7f3f45e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f400cfc20 0x7f3f400c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.268+0000 7f3f45e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f400cfc20 0x7f3f400c9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57648/0 (socket says 172.21.15.45:57648) 2024-04-16T16:18:28.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.268+0000 7f3f45e00640 1 -- 172.21.15.45:0/3675444834 learned_addr learned my addr 172.21.15.45:0/3675444834 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:28.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.268+0000 7f3f46800640 1 --2- 172.21.15.45:0/3675444834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f400cd2f0 0x7f3f400b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.268+0000 7f3f45400640 1 --2- 172.21.15.45:0/3675444834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40162090 0x7f3f400ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.268+0000 7f3f45e00640 1 -- 172.21.15.45:0/3675444834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f400cd2f0 msgr2=0x7f3f400b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:28.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.268+0000 7f3f45e00640 1 --2- 172.21.15.45:0/3675444834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f400cd2f0 0x7f3f400b57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.268+0000 7f3f45e00640 1 -- 172.21.15.45:0/3675444834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40162090 msgr2=0x7f3f400ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:28.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.268+0000 7f3f45e00640 1 --2- 172.21.15.45:0/3675444834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40162090 0x7f3f400ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.268+0000 7f3f45e00640 1 -- 172.21.15.45:0/3675444834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f400ca310 con 0x7f3f400cfc20 2024-04-16T16:18:28.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.268+0000 7f3f45400640 1 --2- 172.21.15.45:0/3675444834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40162090 0x7f3f400ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:28.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.269+0000 7f3f45e00640 1 --2- 172.21.15.45:0/3675444834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f400cfc20 0x7f3f400c9a30 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800c730 tx=0x7f3f3800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:28.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.269+0000 7f3f36000640 1 -- 172.21.15.45:0/3675444834 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f38016020 con 0x7f3f400cfc20 2024-04-16T16:18:28.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.269+0000 7f3f36000640 1 -- 172.21.15.45:0/3675444834 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f38006d50 con 0x7f3f400cfc20 2024-04-16T16:18:28.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.269+0000 7f3f36000640 1 -- 172.21.15.45:0/3675444834 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f38003bb0 con 0x7f3f400cfc20 2024-04-16T16:18:28.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.269+0000 7f3f47200640 1 -- 172.21.15.45:0/3675444834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f401f87d0 con 0x7f3f400cfc20 2024-04-16T16:18:28.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.269+0000 7f3f47200640 1 -- 172.21.15.45:0/3675444834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f401f8d90 con 0x7f3f400cfc20 2024-04-16T16:18:28.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.271+0000 7f3f36000640 1 -- 172.21.15.45:0/3675444834 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f3f3800a850 con 0x7f3f400cfc20 2024-04-16T16:18:28.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.272+0000 7f3f36000640 1 --2- 172.21.15.45:0/3675444834 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f3f2406ca10 0x7f3f2406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.272+0000 7f3f36000640 1 -- 172.21.15.45:0/3675444834 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f3f3808b9b0 con 0x7f3f400cfc20 2024-04-16T16:18:28.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.273+0000 7f3f45400640 1 --2- 172.21.15.45:0/3675444834 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f3f2406ca10 0x7f3f2406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.273+0000 7f3f4b5be740 1 Processor -- start 2024-04-16T16:18:28.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.273+0000 7f3f4b5be740 1 -- start start 2024-04-16T16:18:28.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.274+0000 7f3f4b5be740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0d890 0x55bf17d2c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.274+0000 7f3f4b5be740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d0dc70 0x55bf17ce6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.274+0000 7f3f4b5be740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17ce6ab0 0x55bf17ce4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.274+0000 7f3f4b5be740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55bf17c4b9c0 con 0x55bf17d0dc70 2024-04-16T16:18:28.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.274+0000 7f3f4b5be740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bf17bf2560 con 0x55bf17ce6ab0 2024-04-16T16:18:28.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.274+0000 7f3f4b5be740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55bf17c47980 con 0x55bf17d0d890 2024-04-16T16:18:28.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.275+0000 7f3f45400640 1 --2- 172.21.15.45:0/3675444834 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f3f2406ca10 0x7f3f2406eed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3f4015b4d0 tx=0x7f3f30008040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:28.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.275+0000 7f3f45400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d0dc70 0x55bf17ce6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.275+0000 7f3f45400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d0dc70 0x55bf17ce6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57650/0 (socket says 172.21.15.45:57650) 2024-04-16T16:18:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.275+0000 7f3f45400640 1 -- 172.21.15.45:0/1936137674 learned_addr learned my addr 172.21.15.45:0/1936137674 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.275+0000 7f3f45e00640 1 --2- 172.21.15.45:0/1936137674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0d890 0x55bf17d2c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.275+0000 7f3f46800640 1 --2- 172.21.15.45:0/1936137674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17ce6ab0 0x55bf17ce4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.275+0000 7f3f45400640 1 -- 172.21.15.45:0/1936137674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0d890 msgr2=0x55bf17d2c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.275+0000 7f3f45400640 1 --2- 172.21.15.45:0/1936137674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0d890 0x55bf17d2c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.275+0000 7f3f45400640 1 -- 172.21.15.45:0/1936137674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17ce6ab0 msgr2=0x55bf17ce4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.275+0000 7f3f45400640 1 --2- 172.21.15.45:0/1936137674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17ce6ab0 0x55bf17ce4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.275+0000 7f3f45400640 1 -- 172.21.15.45:0/1936137674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf1795e2d0 con 0x55bf17d0dc70 2024-04-16T16:18:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.275+0000 7f3f45400640 1 --2- 172.21.15.45:0/1936137674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d0dc70 0x55bf17ce6570 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f3f30004350 tx=0x7f3f30011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520c00525ba3aa1b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:28.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.276+0000 7f3f2a600640 1 -- 172.21.15.45:0/1936137674 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3000f040 con 0x55bf17d0dc70 2024-04-16T16:18:28.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.276+0000 7f3f2a600640 1 -- 172.21.15.45:0/1936137674 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f30003610 con 0x55bf17d0dc70 2024-04-16T16:18:28.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.276+0000 7f3f2a600640 1 -- 172.21.15.45:0/1936137674 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f300039c0 con 0x55bf17d0dc70 2024-04-16T16:18:28.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.276+0000 7f3f4b5be740 1 -- 172.21.15.45:0/1936137674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d0dc70 msgr2=0x55bf17ce6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:28.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.276+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/1936137674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d0dc70 0x55bf17ce6570 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f3f30004350 tx=0x7f3f30011000 comp rx=0 tx=0).stop 2024-04-16T16:18:28.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.277+0000 7f3f4b5be740 1 -- 172.21.15.45:0/1936137674 shutdown_connections 2024-04-16T16:18:28.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.277+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/1936137674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17ce6ab0 0x55bf17ce4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.277+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/1936137674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d0dc70 0x55bf17ce6570 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.277+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/1936137674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0d890 0x55bf17d2c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.277+0000 7f3f4b5be740 1 -- 172.21.15.45:0/1936137674 >> 172.21.15.45:0/1936137674 conn(0x55bf17ce2b50 msgr2=0x55bf17d2b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:28.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.277+0000 7f3f4b5be740 1 -- 172.21.15.45:0/1936137674 shutdown_connections 2024-04-16T16:18:28.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.277+0000 7f3f4b5be740 1 -- 172.21.15.45:0/1936137674 wait complete. 2024-04-16T16:18:28.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.277+0000 7f3f4b5be740 1 -- start start 2024-04-16T16:18:28.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.278+0000 7f3f4b5be740 1 Processor -- start 2024-04-16T16:18:28.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.279+0000 7f3f4b5be740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0d890 0x55bf17d09db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.279+0000 7f3f4b5be740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17d0dc70 0x55bf17d25f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.279+0000 7f3f4b5be740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d26470 0x55bf17d0eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.279+0000 7f3f4b5be740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55bf17ce0980 con 0x55bf17d26470 2024-04-16T16:18:28.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.279+0000 7f3f4b5be740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bf17c05920 con 0x55bf17d0dc70 2024-04-16T16:18:28.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.279+0000 7f3f4b5be740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55bf17ce6f30 con 0x55bf17d0d890 2024-04-16T16:18:28.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.279+0000 7f3f4b5be740 10 client.0 _ll_register_callbacks cb 0x55bf17c05310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:18:28.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.279+0000 7f3f46800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d26470 0x55bf17d0eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.279+0000 7f3f46800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d26470 0x55bf17d0eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57666/0 (socket says 172.21.15.45:57666) 2024-04-16T16:18:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.279+0000 7f3f46800640 1 -- 172.21.15.45:0/3575675523 learned_addr learned my addr 172.21.15.45:0/3575675523 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.279+0000 7f3f45e00640 1 --2- 172.21.15.45:0/3575675523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0d890 0x55bf17d09db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.280+0000 7f3f46800640 1 -- 172.21.15.45:0/3575675523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0d890 msgr2=0x55bf17d09db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.280+0000 7f3f45400640 1 --2- 172.21.15.45:0/3575675523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17d0dc70 0x55bf17d25f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.280+0000 7f3f46800640 1 --2- 172.21.15.45:0/3575675523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0d890 0x55bf17d09db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.280+0000 7f3f46800640 1 -- 172.21.15.45:0/3575675523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17d0dc70 msgr2=0x55bf17d25f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.280+0000 7f3f46800640 1 --2- 172.21.15.45:0/3575675523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17d0dc70 0x55bf17d25f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.280+0000 7f3f46800640 1 -- 172.21.15.45:0/3575675523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf17a286b0 con 0x55bf17d26470 2024-04-16T16:18:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.280+0000 7f3f46800640 1 --2- 172.21.15.45:0/3575675523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d26470 0x55bf17d0eae0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009990 tx=0x7f3f3c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c73a954c930ae server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.280+0000 7f3f1ea00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:18:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.280+0000 7f3f1ea00640 1 -- 172.21.15.45:0/3575675523 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c034040 con 0x55bf17d26470 2024-04-16T16:18:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.280+0000 7f3f1ea00640 1 -- 172.21.15.45:0/3575675523 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f3f3c039860 con 0x55bf17d26470 2024-04-16T16:18:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.280+0000 7f3f4b5be740 1 -- 172.21.15.45:0/3575675523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f30003610 con 0x55bf17d26470 2024-04-16T16:18:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.281+0000 7f3f1ea00640 1 -- 172.21.15.45:0/3575675523 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f3f3c051050 con 0x55bf17d26470 2024-04-16T16:18:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.281+0000 7f3f4b5be740 10 client.7341 fetch_fsmap learned FSMap version 64 2024-04-16T16:18:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.281+0000 7f3f4b5be740 1 -- 172.21.15.45:0/3575675523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) -- 0x55bf17d27260 con 0x55bf17d26470 2024-04-16T16:18:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.281+0000 7f3f1ea00640 1 -- 172.21.15.45:0/3575675523 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f30003610 con 0x55bf17d26470 2024-04-16T16:18:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.281+0000 7f3f1ea00640 1 -- 172.21.15.45:0/3575675523 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 64) ==== 3494+0+0 (secure 0 0 0) 0x7f3f3c054020 con 0x55bf17d26470 2024-04-16T16:18:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.281+0000 7f3f4b5be740 10 client.7341 fetch_fsmap finished waiting for FSMap version 64 2024-04-16T16:18:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.281+0000 7f3f4b5be740 10 client.7341 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-16T16:18:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.281+0000 7f3f4b5be740 20 client.7341 populate_metadata read hostname 'smithi045' 2024-04-16T16:18:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.281+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/3575675523 >> [v2:172.21.15.45:6833/3813781309,v1:172.21.15.45:6835/3813781309] conn(0x55bf17d3f8e0 0x55bf17d3d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.281+0000 7f3f4b5be740 4 client.7341 mds_command: new command op to 7188 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-16T16:18:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.281+0000 7f3f4b5be740 1 -- 172.21.15.45:0/3575675523 --> [v2:172.21.15.45:6833/3813781309,v1:172.21.15.45:6835/3813781309] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf17bf1b70 con 0x55bf17d3f8e0 2024-04-16T16:18:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.281+0000 7f3f45e00640 1 --2- 172.21.15.45:0/3575675523 >> [v2:172.21.15.45:6833/3813781309,v1:172.21.15.45:6835/3813781309] conn(0x55bf17d3f8e0 0x55bf17d3d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.282+0000 7f3f45e00640 1 --2- 172.21.15.45:0/3575675523 >> [v2:172.21.15.45:6833/3813781309,v1:172.21.15.45:6835/3813781309] conn(0x55bf17d3f8e0 0x55bf17d3d980 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f348d3eb17824e9d server_cookie=423edc7f51020442 in_seq=0 out_seq=0 2024-04-16T16:18:28.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.282+0000 7f3f1ea00640 10 client.7341 ms_handle_connect on v2:172.21.15.45:6833/3813781309 2024-04-16T16:18:28.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.283+0000 7f3f1ea00640 1 -- 172.21.15.45:0/3575675523 <== mds.0 v2:172.21.15.45:6833/3813781309 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bf17bf1b70 con 0x55bf17d3f8e0 2024-04-16T16:18:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.283+0000 7f3f1ea00640 10 client.7341 handle_command_reply: tid=0 2024-04-16T16:18:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.283+0000 7f3f4b5be740 1 client.7341 shutdown 2024-04-16T16:18:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.283+0000 7f3f4b5be740 1 -- 172.21.15.45:0/3575675523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d26470 msgr2=0x55bf17d0eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.283+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/3575675523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d26470 0x55bf17d0eae0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009990 tx=0x7f3f3c004730 comp rx=0 tx=0).stop 2024-04-16T16:18:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.283+0000 7f3f4b5be740 1 -- 172.21.15.45:0/3575675523 shutdown_connections 2024-04-16T16:18:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.283+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/3575675523 >> [v2:172.21.15.45:6833/3813781309,v1:172.21.15.45:6835/3813781309] conn(0x55bf17d3f8e0 0x55bf17d3d980 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.283+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/3575675523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d26470 0x55bf17d0eae0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.283+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/3575675523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17d0dc70 0x55bf17d25f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.283+0000 7f3f1ea00640 0 client.7341 ms_handle_reset on v2:172.21.15.45:6833/3813781309 2024-04-16T16:18:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.283+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/3575675523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0d890 0x55bf17d09db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.283+0000 7f3f4b5be740 1 -- 172.21.15.45:0/3575675523 >> 172.21.15.45:0/3575675523 conn(0x55bf17ce6ab0 msgr2=0x55bf17ce57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.283+0000 7f3f4b5be740 1 -- 172.21.15.45:0/3575675523 shutdown_connections 2024-04-16T16:18:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.284+0000 7f3f4b5be740 1 -- 172.21.15.45:0/3575675523 wait complete. 2024-04-16T16:18:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.284+0000 7f3f4b5be740 20 client.7341 trim_cache size 0 max 16384 2024-04-16T16:18:28.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.304+0000 7f3f4b5be740 1 Processor -- start 2024-04-16T16:18:28.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.304+0000 7f3f4b5be740 1 -- start start 2024-04-16T16:18:28.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.305+0000 7f3f4b5be740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17ce6ab0 0x55bf17cdbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.305+0000 7f3f4b5be740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d0d890 0x55bf17d3ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.305+0000 7f3f4b5be740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0dc70 0x55bf17d08c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.305+0000 7f3f4b5be740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3f400cbf70 con 0x55bf17d0d890 2024-04-16T16:18:28.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.305+0000 7f3f4b5be740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3f400c8d70 con 0x55bf17ce6ab0 2024-04-16T16:18:28.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.305+0000 7f3f4b5be740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3f400cb920 con 0x55bf17d0dc70 2024-04-16T16:18:28.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.305+0000 7f3f45400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d0d890 0x55bf17d3ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.306+0000 7f3f45400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d0d890 0x55bf17d3ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57680/0 (socket says 172.21.15.45:57680) 2024-04-16T16:18:28.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.306+0000 7f3f45400640 1 -- 172.21.15.45:0/664969176 learned_addr learned my addr 172.21.15.45:0/664969176 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:28.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.306+0000 7f3f45e00640 1 --2- 172.21.15.45:0/664969176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17ce6ab0 0x55bf17cdbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.306+0000 7f3f45400640 1 -- 172.21.15.45:0/664969176 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0dc70 msgr2=0x55bf17d08c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:28.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.306+0000 7f3f45400640 1 --2- 172.21.15.45:0/664969176 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0dc70 0x55bf17d08c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.306+0000 7f3f45400640 1 -- 172.21.15.45:0/664969176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17ce6ab0 msgr2=0x55bf17cdbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:28.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.306+0000 7f3f45400640 1 --2- 172.21.15.45:0/664969176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17ce6ab0 0x55bf17cdbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.306+0000 7f3f45400640 1 -- 172.21.15.45:0/664969176 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf17d0f930 con 0x55bf17d0d890 2024-04-16T16:18:28.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.306+0000 7f3f45e00640 1 --2- 172.21.15.45:0/664969176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17ce6ab0 0x55bf17cdbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:28.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.306+0000 7f3f45400640 1 --2- 172.21.15.45:0/664969176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d0d890 0x55bf17d3ded0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f3f30009270 tx=0x7f3f30012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39a3bdc373a9d31 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:28.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.307+0000 7f3f29c00640 1 -- 172.21.15.45:0/664969176 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f30005990 con 0x55bf17d0d890 2024-04-16T16:18:28.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.307+0000 7f3f29c00640 1 -- 172.21.15.45:0/664969176 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f30005b30 con 0x55bf17d0d890 2024-04-16T16:18:28.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.307+0000 7f3f29c00640 1 -- 172.21.15.45:0/664969176 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f30003610 con 0x55bf17d0d890 2024-04-16T16:18:28.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.307+0000 7f3f4b5be740 1 -- 172.21.15.45:0/664969176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d0d890 msgr2=0x55bf17d3ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:28.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.307+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/664969176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d0d890 0x55bf17d3ded0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f3f30009270 tx=0x7f3f30012000 comp rx=0 tx=0).stop 2024-04-16T16:18:28.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.307+0000 7f3f4b5be740 1 -- 172.21.15.45:0/664969176 shutdown_connections 2024-04-16T16:18:28.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.307+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/664969176 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0dc70 0x55bf17d08c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.307+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/664969176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17d0d890 0x55bf17d3ded0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.307+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/664969176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17ce6ab0 0x55bf17cdbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.307+0000 7f3f4b5be740 1 -- 172.21.15.45:0/664969176 >> 172.21.15.45:0/664969176 conn(0x55bf17d28330 msgr2=0x55bf17ce7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:28.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.307+0000 7f3f4b5be740 1 -- 172.21.15.45:0/664969176 shutdown_connections 2024-04-16T16:18:28.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.308+0000 7f3f4b5be740 1 -- 172.21.15.45:0/664969176 wait complete. 2024-04-16T16:18:28.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.308+0000 7f3f4b5be740 1 -- start start 2024-04-16T16:18:28.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.308+0000 7f3f4b5be740 1 Processor -- start 2024-04-16T16:18:28.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.309+0000 7f3f4b5be740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17ce6ab0 0x55bf17da05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.309+0000 7f3f4b5be740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17d0d890 0x55bf17d8b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.309+0000 7f3f4b5be740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0dc70 0x55bf17d4e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.309+0000 7f3f4b5be740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55bf17d3f760 con 0x55bf17ce6ab0 2024-04-16T16:18:28.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.309+0000 7f3f4b5be740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bf17d27590 con 0x55bf17d0d890 2024-04-16T16:18:28.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.309+0000 7f3f4b5be740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55bf17d3fdb0 con 0x55bf17d0dc70 2024-04-16T16:18:28.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.310+0000 7f3f4b5be740 10 client.0 _ll_register_callbacks cb 0x7f3f3c0096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:18:28.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.310+0000 7f3f45e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17ce6ab0 0x55bf17da05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.310+0000 7f3f45e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17ce6ab0 0x55bf17da05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57690/0 (socket says 172.21.15.45:57690) 2024-04-16T16:18:28.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.310+0000 7f3f45e00640 1 -- 172.21.15.45:0/143583774 learned_addr learned my addr 172.21.15.45:0/143583774 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:28.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.310+0000 7f3f46800640 1 --2- 172.21.15.45:0/143583774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0dc70 0x55bf17d4e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.310+0000 7f3f45400640 1 --2- 172.21.15.45:0/143583774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17d0d890 0x55bf17d8b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.310+0000 7f3f45e00640 1 -- 172.21.15.45:0/143583774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0dc70 msgr2=0x55bf17d4e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:28.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.310+0000 7f3f45e00640 1 --2- 172.21.15.45:0/143583774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0dc70 0x55bf17d4e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.310+0000 7f3f45e00640 1 -- 172.21.15.45:0/143583774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17d0d890 msgr2=0x55bf17d8b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:28.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.310+0000 7f3f45e00640 1 --2- 172.21.15.45:0/143583774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17d0d890 0x55bf17d8b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.310+0000 7f3f45e00640 1 -- 172.21.15.45:0/143583774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf17d9d1e0 con 0x55bf17ce6ab0 2024-04-16T16:18:28.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.311+0000 7f3f45e00640 1 --2- 172.21.15.45:0/143583774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17ce6ab0 0x55bf17da05c0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f3f38055a20 tx=0x7f3f38011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5048567cb92c0e6f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:28.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.311+0000 7f3f1ea00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:18:28.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.311+0000 7f3f1ea00640 1 -- 172.21.15.45:0/143583774 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f38055c30 con 0x55bf17ce6ab0 2024-04-16T16:18:28.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.311+0000 7f3f1ea00640 1 -- 172.21.15.45:0/143583774 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f3f38089e30 con 0x55bf17ce6ab0 2024-04-16T16:18:28.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.311+0000 7f3f4b5be740 1 -- 172.21.15.45:0/143583774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f30005b30 con 0x55bf17ce6ab0 2024-04-16T16:18:28.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.311+0000 7f3f1ea00640 1 -- 172.21.15.45:0/143583774 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f3f38089050 con 0x55bf17ce6ab0 2024-04-16T16:18:28.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.311+0000 7f3f4b5be740 10 client.7347 fetch_fsmap learned FSMap version 64 2024-04-16T16:18:28.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.311+0000 7f3f4b5be740 1 -- 172.21.15.45:0/143583774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) -- 0x55bf17d9fdc0 con 0x55bf17ce6ab0 2024-04-16T16:18:28.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.312+0000 7f3f1ea00640 1 -- 172.21.15.45:0/143583774 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f30005b30 con 0x55bf17ce6ab0 2024-04-16T16:18:28.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.312+0000 7f3f1ea00640 1 -- 172.21.15.45:0/143583774 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 64) ==== 3494+0+0 (secure 0 0 0) 0x7f3f3800ab40 con 0x55bf17ce6ab0 2024-04-16T16:18:28.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.312+0000 7f3f4b5be740 10 client.7347 fetch_fsmap finished waiting for FSMap version 64 2024-04-16T16:18:28.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.312+0000 7f3f4b5be740 10 client.7347 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-16T16:18:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.312+0000 7f3f4b5be740 20 client.7347 populate_metadata read hostname 'smithi045' 2024-04-16T16:18:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.312+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/143583774 >> [v2:172.21.15.45:6833/3813781309,v1:172.21.15.45:6835/3813781309] conn(0x55bf17d26470 0x55bf17d08f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.312+0000 7f3f4b5be740 4 client.7347 mds_command: new command op to 7188 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-16T16:18:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.312+0000 7f3f4b5be740 1 -- 172.21.15.45:0/143583774 --> [v2:172.21.15.45:6833/3813781309,v1:172.21.15.45:6835/3813781309] -- command(tid 0: {"prefix": "status"}) -- 0x55bf17c593e0 con 0x55bf17d26470 2024-04-16T16:18:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.312+0000 7f3f45400640 1 --2- 172.21.15.45:0/143583774 >> [v2:172.21.15.45:6833/3813781309,v1:172.21.15.45:6835/3813781309] conn(0x55bf17d26470 0x55bf17d08f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.313+0000 7f3f45400640 1 --2- 172.21.15.45:0/143583774 >> [v2:172.21.15.45:6833/3813781309,v1:172.21.15.45:6835/3813781309] conn(0x55bf17d26470 0x55bf17d08f50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea8c72252b05b06b server_cookie=15b4ca0d49a7a852 in_seq=0 out_seq=0 2024-04-16T16:18:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.313+0000 7f3f1ea00640 10 client.7347 ms_handle_connect on v2:172.21.15.45:6833/3813781309 2024-04-16T16:18:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.313+0000 7f3f1ea00640 1 -- 172.21.15.45:0/143583774 <== mds.0 v2:172.21.15.45:6833/3813781309 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55bf17c593e0 con 0x55bf17d26470 2024-04-16T16:18:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.313+0000 7f3f1ea00640 10 client.7347 handle_command_reply: tid=0 2024-04-16T16:18:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.313+0000 7f3f4b5be740 1 client.7347 shutdown 2024-04-16T16:18:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.313+0000 7f3f4b5be740 1 -- 172.21.15.45:0/143583774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17ce6ab0 msgr2=0x55bf17da05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.313+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/143583774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17ce6ab0 0x55bf17da05c0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f3f38055a20 tx=0x7f3f38011000 comp rx=0 tx=0).stop 2024-04-16T16:18:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.313+0000 7f3f4b5be740 1 -- 172.21.15.45:0/143583774 shutdown_connections 2024-04-16T16:18:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.313+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/143583774 >> [v2:172.21.15.45:6833/3813781309,v1:172.21.15.45:6835/3813781309] conn(0x55bf17d26470 0x55bf17d08f50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.313+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/143583774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bf17d0dc70 0x55bf17d4e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.313+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/143583774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bf17d0d890 0x55bf17d8b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.313+0000 7f3f1ea00640 0 client.7347 ms_handle_reset on v2:172.21.15.45:6833/3813781309 2024-04-16T16:18:28.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.313+0000 7f3f4b5be740 1 --2- 172.21.15.45:0/143583774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bf17ce6ab0 0x55bf17da05c0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.313+0000 7f3f4b5be740 1 -- 172.21.15.45:0/143583774 >> 172.21.15.45:0/143583774 conn(0x55bf17d28330 msgr2=0x55bf17d09bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:28.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.314+0000 7f3f4b5be740 1 -- 172.21.15.45:0/143583774 shutdown_connections 2024-04-16T16:18:28.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.314+0000 7f3f4b5be740 1 -- 172.21.15.45:0/143583774 wait complete. 2024-04-16T16:18:28.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.314+0000 7f3f4b5be740 20 client.7347 trim_cache size 0 max 16384 2024-04-16T16:18:28.315 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:18:28.315 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "748c4b85-0190-4c59-80ad-098a6a5007c0", 2024-04-16T16:18:28.315 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-04-16T16:18:28.315 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 7188, 2024-04-16T16:18:28.316 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-04-16T16:18:28.316 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-04-16T16:18:28.316 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-04-16T16:18:28.316 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 12.036356485000001, 2024-04-16T16:18:28.316 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 64, 2024-04-16T16:18:28.316 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 85, 2024-04-16T16:18:28.316 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-04-16T16:18:28.316 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 17.366553872000001 2024-04-16T16:18:28.316 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:18:28.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.315+0000 7f3f47200640 1 -- 172.21.15.45:0/3675444834 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f3f2406ca10 msgr2=0x7f3f2406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:28.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.315+0000 7f3f47200640 1 --2- 172.21.15.45:0/3675444834 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f3f2406ca10 0x7f3f2406eed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3f4015b4d0 tx=0x7f3f30008040 comp rx=0 tx=0).stop 2024-04-16T16:18:28.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.315+0000 7f3f47200640 1 -- 172.21.15.45:0/3675444834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f400cfc20 msgr2=0x7f3f400c9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:28.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.315+0000 7f3f47200640 1 --2- 172.21.15.45:0/3675444834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f400cfc20 0x7f3f400c9a30 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800c730 tx=0x7f3f3800cbf0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.315+0000 7f3f47200640 1 -- 172.21.15.45:0/3675444834 shutdown_connections 2024-04-16T16:18:28.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.315+0000 7f3f47200640 1 --2- 172.21.15.45:0/3675444834 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f3f2406ca10 0x7f3f2406eed0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.315+0000 7f3f47200640 1 --2- 172.21.15.45:0/3675444834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f400cd2f0 0x7f3f400b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.315+0000 7f3f47200640 1 --2- 172.21.15.45:0/3675444834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40162090 0x7f3f400ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.315+0000 7f3f47200640 1 --2- 172.21.15.45:0/3675444834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f400cfc20 0x7f3f400c9a30 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.315+0000 7f3f47200640 1 -- 172.21.15.45:0/3675444834 >> 172.21.15.45:0/3675444834 conn(0x7f3f400d1940 msgr2=0x7f3f400b8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:28.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.315+0000 7f3f47200640 1 -- 172.21.15.45:0/3675444834 shutdown_connections 2024-04-16T16:18:28.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.315+0000 7f3f47200640 1 -- 172.21.15.45:0/3675444834 wait complete. 2024-04-16T16:18:28.332 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:18:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.552+0000 7fd167200640 1 Processor -- start 2024-04-16T16:18:28.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.553+0000 7fd167200640 1 -- start start 2024-04-16T16:18:28.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.554+0000 7fd167200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd160161e90 0x7fd1600cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.554+0000 7fd167200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd1600d52d0 0x7fd16006ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.554+0000 7fd167200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1600d3080 con 0x7fd160162270 2024-04-16T16:18:28.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.554+0000 7fd167200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd1600d2e30 con 0x7fd160161e90 2024-04-16T16:18:28.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.554+0000 7fd167200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd1600d3360 con 0x7fd1600d52d0 2024-04-16T16:18:28.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.555+0000 7fd165e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd160161e90 0x7fd1600cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.555+0000 7fd165e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd160161e90 0x7fd1600cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57704/0 (socket says 172.21.15.45:57704) 2024-04-16T16:18:28.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.555+0000 7fd165e00640 1 -- 172.21.15.45:0/1856209 learned_addr learned my addr 172.21.15.45:0/1856209 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:28.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.555+0000 7fd166800640 1 --2- 172.21.15.45:0/1856209 >> v2:172.21.15.103:3300/0 conn(0x7fd1600d52d0 0x7fd16006ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.555+0000 7fd165e00640 1 -- 172.21.15.45:0/1856209 >> v2:172.21.15.103:3300/0 conn(0x7fd1600d52d0 msgr2=0x7fd16006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:28.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.555+0000 7fd165e00640 1 --2- 172.21.15.45:0/1856209 >> v2:172.21.15.103:3300/0 conn(0x7fd1600d52d0 0x7fd16006ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.555+0000 7fd165e00640 1 -- 172.21.15.45:0/1856209 >> v1:172.21.15.103:6789/0 conn(0x7fd160162270 legacy=0x7fd1600d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:28.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.555+0000 7fd165e00640 1 -- 172.21.15.45:0/1856209 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1600d3640 con 0x7fd160161e90 2024-04-16T16:18:28.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.556+0000 7fd165e00640 1 --2- 172.21.15.45:0/1856209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd160161e90 0x7fd1600cf7d0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fd154009870 tx=0x7fd15402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16223549e7289dbf server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.557+0000 7fd164a00640 1 -- 172.21.15.45:0/1856209 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd154035020 con 0x7fd160161e90 2024-04-16T16:18:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.557+0000 7fd164a00640 1 -- 172.21.15.45:0/1856209 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd154031870 con 0x7fd160161e90 2024-04-16T16:18:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.557+0000 7fd164a00640 1 -- 172.21.15.45:0/1856209 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd154031c40 con 0x7fd160161e90 2024-04-16T16:18:28.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.557+0000 7fd167200640 1 -- 172.21.15.45:0/1856209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd160161e90 msgr2=0x7fd1600cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:28.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.557+0000 7fd167200640 1 --2- 172.21.15.45:0/1856209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd160161e90 0x7fd1600cf7d0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fd154009870 tx=0x7fd15402b910 comp rx=0 tx=0).stop 2024-04-16T16:18:28.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.558+0000 7fd167200640 1 -- 172.21.15.45:0/1856209 shutdown_connections 2024-04-16T16:18:28.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.558+0000 7fd167200640 1 --2- 172.21.15.45:0/1856209 >> v2:172.21.15.103:3300/0 conn(0x7fd1600d52d0 0x7fd16006ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.558+0000 7fd167200640 1 --2- 172.21.15.45:0/1856209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd160161e90 0x7fd1600cf7d0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.558+0000 7fd167200640 1 -- 172.21.15.45:0/1856209 >> 172.21.15.45:0/1856209 conn(0x7fd1600d1940 msgr2=0x7fd1600d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:28.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.558+0000 7fd167200640 1 -- 172.21.15.45:0/1856209 shutdown_connections 2024-04-16T16:18:28.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.558+0000 7fd167200640 1 -- 172.21.15.45:0/1856209 wait complete. 2024-04-16T16:18:28.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.559+0000 7fd167200640 1 Processor -- start 2024-04-16T16:18:28.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.560+0000 7fd167200640 1 -- start start 2024-04-16T16:18:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.561+0000 7fd167200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1600d52d0 0x7fd1601558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.561+0000 7fd167200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd160161e90 0x7fd1601f80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.561+0000 7fd167200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd160162270 0x7fd1601f8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.561+0000 7fd167200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd1600d4550 con 0x7fd160162270 2024-04-16T16:18:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.561+0000 7fd167200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1600cfd10 con 0x7fd1600d52d0 2024-04-16T16:18:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.561+0000 7fd167200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd1600d46d0 con 0x7fd160161e90 2024-04-16T16:18:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.561+0000 7fd166800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd160162270 0x7fd1601f8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.561+0000 7fd166800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd160162270 0x7fd1601f8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57708/0 (socket says 172.21.15.45:57708) 2024-04-16T16:18:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.561+0000 7fd166800640 1 -- 172.21.15.45:0/3385802460 learned_addr learned my addr 172.21.15.45:0/3385802460 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:28.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.561+0000 7fd165e00640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1600d52d0 0x7fd1601558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.561+0000 7fd165400640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd160161e90 0x7fd1601f80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.561+0000 7fd166800640 1 -- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd160161e90 msgr2=0x7fd1601f80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:28.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.561+0000 7fd166800640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd160161e90 0x7fd1601f80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.561+0000 7fd166800640 1 -- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1600d52d0 msgr2=0x7fd1601558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:28.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.562+0000 7fd166800640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1600d52d0 0x7fd1601558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.562+0000 7fd166800640 1 -- 172.21.15.45:0/3385802460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16015c180 con 0x7fd160162270 2024-04-16T16:18:28.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.562+0000 7fd165e00640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1600d52d0 0x7fd1601558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:28.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.562+0000 7fd165400640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd160161e90 0x7fd1601f80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:28.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.562+0000 7fd166800640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd160162270 0x7fd1601f8600 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fd15c007e60 tx=0x7fd15c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:28.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.562+0000 7fd15a000640 1 -- 172.21.15.45:0/3385802460 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c014620 con 0x7fd160162270 2024-04-16T16:18:28.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.562+0000 7fd15a000640 1 -- 172.21.15.45:0/3385802460 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd15c019070 con 0x7fd160162270 2024-04-16T16:18:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.562+0000 7fd15a000640 1 -- 172.21.15.45:0/3385802460 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c014a00 con 0x7fd160162270 2024-04-16T16:18:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.562+0000 7fd167200640 1 -- 172.21.15.45:0/3385802460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd16015c460 con 0x7fd160162270 2024-04-16T16:18:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.562+0000 7fd167200640 1 -- 172.21.15.45:0/3385802460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd16015ca20 con 0x7fd160162270 2024-04-16T16:18:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.564+0000 7fd167200640 1 -- 172.21.15.45:0/3385802460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1600ba7a0 con 0x7fd160162270 2024-04-16T16:18:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.565+0000 7fd15a000640 1 -- 172.21.15.45:0/3385802460 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7fd15c014ba0 con 0x7fd160162270 2024-04-16T16:18:28.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.566+0000 7fd15a000640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fd14406ca10 0x7fd14406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:28.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.566+0000 7fd15a000640 1 -- 172.21.15.45:0/3385802460 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7fd15c08c5e0 con 0x7fd160162270 2024-04-16T16:18:28.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.566+0000 7fd165e00640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fd14406ca10 0x7fd14406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:28.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.568+0000 7fd165e00640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fd14406ca10 0x7fd14406eed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd154036000 tx=0x7fd154008030 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:28.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.570+0000 7fd15a000640 1 -- 172.21.15.45:0/3385802460 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd1600ba7a0 con 0x7fd160162270 2024-04-16T16:18:28.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.818+0000 7fd167200640 1 -- 172.21.15.45:0/3385802460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd1601593e0 con 0x7fd160162270 2024-04-16T16:18:28.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.819+0000 7fd15a000640 1 -- 172.21.15.45:0/3385802460 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v64) ==== 53+0+73 (secure 0 0 0) 0x7fd1601593e0 con 0x7fd160162270 2024-04-16T16:18:28.818 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:18:28.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.821+0000 7fd167200640 1 -- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fd14406ca10 msgr2=0x7fd14406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:28.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.821+0000 7fd167200640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fd14406ca10 0x7fd14406eed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd154036000 tx=0x7fd154008030 comp rx=0 tx=0).stop 2024-04-16T16:18:28.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.821+0000 7fd167200640 1 -- 172.21.15.45:0/3385802460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd160162270 msgr2=0x7fd1601f8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:28.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.821+0000 7fd167200640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd160162270 0x7fd1601f8600 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fd15c007e60 tx=0x7fd15c00e4b0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.821+0000 7fd167200640 1 -- 172.21.15.45:0/3385802460 shutdown_connections 2024-04-16T16:18:28.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.821+0000 7fd167200640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fd14406ca10 0x7fd14406eed0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.821+0000 7fd167200640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd160162270 0x7fd1601f8600 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.821+0000 7fd167200640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd160161e90 0x7fd1601f80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.821+0000 7fd167200640 1 --2- 172.21.15.45:0/3385802460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1600d52d0 0x7fd1601558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:28.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.821+0000 7fd167200640 1 -- 172.21.15.45:0/3385802460 >> 172.21.15.45:0/3385802460 conn(0x7fd1600d1940 msgr2=0x7fd1600b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:28.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.821+0000 7fd167200640 1 -- 172.21.15.45:0/3385802460 shutdown_connections 2024-04-16T16:18:28.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:28.822+0000 7fd167200640 1 -- 172.21.15.45:0/3385802460 wait complete. 2024-04-16T16:18:28.839 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-16T16:18:28.839 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:18:28.839 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-04-16T16:18:28.839 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-16T16:18:28.839 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T16:18:28.840 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-16T16:18:28.840 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:18:28.840 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:18:28.840 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:18:28.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:28.840 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-04-16T16:18:28.890 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:18:28.891 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:18:28.891 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:18:28.891 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:18:28.891 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-04-16T16:18:28.891 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:18:28.891 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:18:28.891 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:18:28.891 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-04-16T16:18:28.892 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:18:28.892 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:18:28.892 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-04-16T16:18:28.892 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:18:28.892 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-04-16T16:18:28.892 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-04-16T16:18:28.892 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-16T16:18:28.892 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42292sec preferred_lft 42292sec 2024-04-16T16:18:28.892 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-04-16T16:18:28.892 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:18:28.892 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:18:28.892 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-04-16T16:18:28.892 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-04-16T16:18:28.892 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:18:28.892 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff 2024-04-16T16:18:28.893 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:18:28.893 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:18:28.893 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::50f1:99ff:fe2b:225b/64 scope link 2024-04-16T16:18:28.893 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:18:28.893 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:18:28.893 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T16:18:28.893 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::7007:c3ff:fe13:7bcd/64 scope link 2024-04-16T16:18:28.893 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:18:28.893 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:18:28.893 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether ae:10:ea:fc:0a:13 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-16T16:18:28.893 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ac10:eaff:fefc:a13/64 scope link 2024-04-16T16:18:28.893 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:18:28.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:28.894 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:18:28.978 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-04-16T16:18:28.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:28.982 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-04-16T16:18:29.037 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-04-16T16:18:29.037 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:18:29.037 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-16T16:18:29.037 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-16T16:18:29.038 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:18:29.038 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:18:29.038 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:18:29.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-16T16:18:29.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-16T16:18:29.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-16T16:18:29.038 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:18:29.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:18:29.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:18:29.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:29.301 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T16:18:29.352 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-16T16:18:29.352 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-16T16:18:29.353 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:18:29.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:29.353 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-04-16T16:18:29.457 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:18:29.457 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-16T16:18:29.511 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:18:29.511 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:18:29.511 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:18:29.563 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-16T16:18:29.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:29.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:18:29.706 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:18:29.706 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:18:29.742 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:18:29.742 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:18:29.742 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:18:29.742 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 0@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:18:29.742 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:18:29.745 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-16T16:18:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:18:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:18:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:18:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:18:29.750 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:18:29.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:29.750 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:18:29.845 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:18:29.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:18:29.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:18:29.988 INFO:teuthology.orchestra.run.smithi045.stdout:client5612 2024-04-16T16:18:29.988 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:18:29.988 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:30.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:18:30.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.257+0000 7fba86e00640 1 Processor -- start 2024-04-16T16:18:30.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.257+0000 7fba86e00640 1 -- start start 2024-04-16T16:18:30.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.259+0000 7fba86e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fba8015ccb0 0x7fba8006df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:30.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.259+0000 7fba86e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba8015d360 con 0x7fba8006e480 2024-04-16T16:18:30.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.259+0000 7fba86e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba8015d640 con 0x7fba80146220 2024-04-16T16:18:30.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.259+0000 7fba86e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fba8015d130 con 0x7fba8015ccb0 2024-04-16T16:18:30.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.259+0000 7fba85000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fba8015ccb0 0x7fba8006df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:30.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.259+0000 7fba85000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fba8015ccb0 0x7fba8006df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57718/0 (socket says 172.21.15.45:57718) 2024-04-16T16:18:30.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.259+0000 7fba86400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fba8006e480 0x7fba801608a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41158/0 (socket says 172.21.15.45:41158) 2024-04-16T16:18:30.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.259+0000 7fba85000640 1 -- 172.21.15.45:0/3113568057 learned_addr learned my addr 172.21.15.45:0/3113568057 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:30.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.260+0000 7fba85000640 1 -- 172.21.15.45:0/3113568057 >> v1:172.21.15.103:6789/0 conn(0x7fba80146220 legacy=0x7fba8006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:30.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.260+0000 7fba85000640 1 -- 172.21.15.45:0/3113568057 >> v1:172.21.15.45:6789/0 conn(0x7fba8006e480 legacy=0x7fba801608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:30.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.260+0000 7fba85000640 1 -- 172.21.15.45:0/3113568057 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba8015d920 con 0x7fba8015ccb0 2024-04-16T16:18:30.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.261+0000 7fba85000640 1 --2- 172.21.15.45:0/3113568057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba8015ccb0 0x7fba8006df40 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fba6c009870 tx=0x7fba6c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7cf8753b440988d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:30.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.261+0000 7fba7fe00640 1 -- 172.21.15.45:0/3113568057 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6c034070 con 0x7fba8015ccb0 2024-04-16T16:18:30.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.261+0000 7fba7fe00640 1 -- 172.21.15.45:0/3113568057 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba6c030860 con 0x7fba8015ccb0 2024-04-16T16:18:30.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.261+0000 7fba7fe00640 1 -- 172.21.15.45:0/3113568057 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6c030c30 con 0x7fba8015ccb0 2024-04-16T16:18:30.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.262+0000 7fba86e00640 1 -- 172.21.15.45:0/3113568057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba8015ccb0 msgr2=0x7fba8006df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:30.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.262+0000 7fba86e00640 1 --2- 172.21.15.45:0/3113568057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba8015ccb0 0x7fba8006df40 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fba6c009870 tx=0x7fba6c02b9a0 comp rx=0 tx=0).stop 2024-04-16T16:18:30.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.262+0000 7fba86e00640 1 -- 172.21.15.45:0/3113568057 shutdown_connections 2024-04-16T16:18:30.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.262+0000 7fba86e00640 1 --2- 172.21.15.45:0/3113568057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba8015ccb0 0x7fba8006df40 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:30.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.262+0000 7fba86e00640 1 -- 172.21.15.45:0/3113568057 >> 172.21.15.45:0/3113568057 conn(0x7fba8006b930 msgr2=0x7fba8006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:30.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.262+0000 7fba86e00640 1 -- 172.21.15.45:0/3113568057 shutdown_connections 2024-04-16T16:18:30.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.262+0000 7fba86e00640 1 -- 172.21.15.45:0/3113568057 wait complete. 2024-04-16T16:18:30.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.264+0000 7fba86e00640 1 Processor -- start 2024-04-16T16:18:30.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.264+0000 7fba86e00640 1 -- start start 2024-04-16T16:18:30.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.265+0000 7fba86e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba8006e480 0x7fba80135110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:30.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.265+0000 7fba86e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba80146220 0x7fba800ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:30.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.265+0000 7fba86e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba8015ccb0 0x7fba800cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:30.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.265+0000 7fba86e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fba80160fb0 con 0x7fba8015ccb0 2024-04-16T16:18:30.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.265+0000 7fba86e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fba80160410 con 0x7fba8006e480 2024-04-16T16:18:30.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.265+0000 7fba86e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fba80161130 con 0x7fba80146220 2024-04-16T16:18:30.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.265+0000 7fba86400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba8015ccb0 0x7fba800cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:30.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.265+0000 7fba86400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba8015ccb0 0x7fba800cbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57734/0 (socket says 172.21.15.45:57734) 2024-04-16T16:18:30.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.265+0000 7fba86400640 1 -- 172.21.15.45:0/3856802094 learned_addr learned my addr 172.21.15.45:0/3856802094 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:30.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.265+0000 7fba86400640 1 -- 172.21.15.45:0/3856802094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba80146220 msgr2=0x7fba800ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:30.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.265+0000 7fba86400640 1 --2- 172.21.15.45:0/3856802094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba80146220 0x7fba800ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:30.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.265+0000 7fba85a00640 1 --2- 172.21.15.45:0/3856802094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba8006e480 0x7fba80135110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:30.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.265+0000 7fba86400640 1 -- 172.21.15.45:0/3856802094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba8006e480 msgr2=0x7fba80135110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:30.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.266+0000 7fba86400640 1 --2- 172.21.15.45:0/3856802094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba8006e480 0x7fba80135110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:30.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.266+0000 7fba86400640 1 -- 172.21.15.45:0/3856802094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba801346a0 con 0x7fba8015ccb0 2024-04-16T16:18:30.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.266+0000 7fba85a00640 1 --2- 172.21.15.45:0/3856802094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba8006e480 0x7fba80135110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:30.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.266+0000 7fba86400640 1 --2- 172.21.15.45:0/3856802094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba8015ccb0 0x7fba800cbbc0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fba7800c780 tx=0x7fba7800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:30.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.266+0000 7fba7d600640 1 -- 172.21.15.45:0/3856802094 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba78017070 con 0x7fba8015ccb0 2024-04-16T16:18:30.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.266+0000 7fba7d600640 1 -- 172.21.15.45:0/3856802094 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba7800a640 con 0x7fba8015ccb0 2024-04-16T16:18:30.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.266+0000 7fba7d600640 1 -- 172.21.15.45:0/3856802094 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7800a9f0 con 0x7fba8015ccb0 2024-04-16T16:18:30.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.266+0000 7fba86e00640 1 -- 172.21.15.45:0/3856802094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba800cc5c0 con 0x7fba8015ccb0 2024-04-16T16:18:30.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.266+0000 7fba86e00640 1 -- 172.21.15.45:0/3856802094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba800cc8a0 con 0x7fba8015ccb0 2024-04-16T16:18:30.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.268+0000 7fba86e00640 1 -- 172.21.15.45:0/3856802094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba80075780 con 0x7fba8015ccb0 2024-04-16T16:18:30.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.269+0000 7fba7d600640 1 -- 172.21.15.45:0/3856802094 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7fba7801a070 con 0x7fba8015ccb0 2024-04-16T16:18:30.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.269+0000 7fba7d600640 1 --2- 172.21.15.45:0/3856802094 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fba6006ca10 0x7fba6006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:30.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.269+0000 7fba7d600640 1 -- 172.21.15.45:0/3856802094 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7fba7808b250 con 0x7fba8015ccb0 2024-04-16T16:18:30.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.270+0000 7fba85a00640 1 --2- 172.21.15.45:0/3856802094 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fba6006ca10 0x7fba6006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:30.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.271+0000 7fba85a00640 1 --2- 172.21.15.45:0/3856802094 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fba6006ca10 0x7fba6006eed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fba74003eb0 tx=0x7fba74009040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:30.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.274+0000 7fba7d600640 1 -- 172.21.15.45:0/3856802094 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fba78058f20 con 0x7fba8015ccb0 2024-04-16T16:18:30.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.519+0000 7fba86e00640 1 -- 172.21.15.45:0/3856802094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fba80160c20 con 0x7fba8015ccb0 2024-04-16T16:18:30.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.520+0000 7fba7d600640 1 -- 172.21.15.45:0/3856802094 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v64) ==== 53+0+73 (secure 0 0 0) 0x7fba7805d220 con 0x7fba8015ccb0 2024-04-16T16:18:30.519 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:18:30.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.523+0000 7fba86e00640 1 -- 172.21.15.45:0/3856802094 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fba6006ca10 msgr2=0x7fba6006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:30.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.523+0000 7fba86e00640 1 --2- 172.21.15.45:0/3856802094 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fba6006ca10 0x7fba6006eed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fba74003eb0 tx=0x7fba74009040 comp rx=0 tx=0).stop 2024-04-16T16:18:30.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.523+0000 7fba86e00640 1 -- 172.21.15.45:0/3856802094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba8015ccb0 msgr2=0x7fba800cbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:30.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.523+0000 7fba86e00640 1 --2- 172.21.15.45:0/3856802094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba8015ccb0 0x7fba800cbbc0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fba7800c780 tx=0x7fba7800cc40 comp rx=0 tx=0).stop 2024-04-16T16:18:30.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.523+0000 7fba86e00640 1 -- 172.21.15.45:0/3856802094 shutdown_connections 2024-04-16T16:18:30.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.523+0000 7fba86e00640 1 --2- 172.21.15.45:0/3856802094 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7fba6006ca10 0x7fba6006eed0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:30.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.523+0000 7fba86e00640 1 --2- 172.21.15.45:0/3856802094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba8015ccb0 0x7fba800cbbc0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:30.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.523+0000 7fba86e00640 1 --2- 172.21.15.45:0/3856802094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba80146220 0x7fba800ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:30.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.523+0000 7fba86e00640 1 --2- 172.21.15.45:0/3856802094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba8006e480 0x7fba80135110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:30.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.523+0000 7fba86e00640 1 -- 172.21.15.45:0/3856802094 >> 172.21.15.45:0/3856802094 conn(0x7fba8006b930 msgr2=0x7fba800b5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:30.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.523+0000 7fba86e00640 1 -- 172.21.15.45:0/3856802094 shutdown_connections 2024-04-16T16:18:30.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:30.523+0000 7fba86e00640 1 -- 172.21.15.45:0/3856802094 wait complete. 2024-04-16T16:18:30.541 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-16T16:18:30.541 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:18:30.541 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-04-16T16:18:30.542 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-16T16:18:30.542 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T16:18:30.542 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-16T16:18:30.542 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:18:30.542 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:18:30.542 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:18:30.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:30.542 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-04-16T16:18:30.602 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:18:30.602 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:18:30.602 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:18:30.602 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:18:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-16T16:18:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:18:30.603 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:18:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:18:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-04-16T16:18:30.603 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:18:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:18:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-04-16T16:18:30.603 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:18:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-04-16T16:18:30.603 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:18:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-04-16T16:18:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-16T16:18:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42314sec preferred_lft 42314sec 2024-04-16T16:18:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-04-16T16:18:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:18:30.604 INFO:teuthology.orchestra.run.smithi103.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:18:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:18:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:18:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:18:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::74ce:ceff:fec6:2a00/64 scope link 2024-04-16T16:18:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:18:30.604 INFO:teuthology.orchestra.run.smithi103.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:18:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T16:18:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::3cf7:ffff:feca:f879/64 scope link 2024-04-16T16:18:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:18:30.605 INFO:teuthology.orchestra.run.smithi103.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:18:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 86:4c:6e:ce:c3:07 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-16T16:18:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::844c:6eff:fece:c307/64 scope link 2024-04-16T16:18:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:18:30.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:30.606 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:18:30.694 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-16T16:18:30.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:30.698 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-04-16T16:18:30.753 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-04-16T16:18:30.753 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:18:30.753 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-16T16:18:30.753 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-16T16:18:30.753 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:18:30.753 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:18:30.753 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:18:30.754 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-16T16:18:30.754 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-16T16:18:30.754 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-16T16:18:30.754 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:18:30.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:18:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:18:30.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:18:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:18:30.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:30.999 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-16T16:18:31.049 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-16T16:18:31.049 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-16T16:18:31.049 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:18:31.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:31.049 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.0 up 2024-04-16T16:18:31.149 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:18:31.149 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-16T16:18:31.203 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:18:31.203 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:18:31.203 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:18:31.257 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-16T16:18:31.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:31.257 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:18:31.402 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:18:31.402 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:18:31.440 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:18:31.440 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:18:31.440 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:18:31.440 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with new device syntax: 1@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:18:31.440 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:18:31.443 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-16T16:18:31.443 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:18:31.443 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:18:31.444 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:18:31.444 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:18:31.447 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:18:31.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:31.448 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:18:31.530 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:18:31.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:18:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:18:31.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:18:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:18:31.646 INFO:teuthology.orchestra.run.smithi103.stdout:client7365 2024-04-16T16:18:31.647 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:18:31.647 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:31.700 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:18:31.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.931+0000 7f6727c00640 1 Processor -- start 2024-04-16T16:18:31.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.931+0000 7f6727c00640 1 -- start start 2024-04-16T16:18:31.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.932+0000 7f6727c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6720072c50 0x7f672006e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:31.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.932+0000 7f6727c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f672006f880 0x7f672006fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:31.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.932+0000 7f6727c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f672015a440 con 0x7f672015a060 2024-04-16T16:18:31.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.932+0000 7f6727c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f67200b3a50 con 0x7f6720072c50 2024-04-16T16:18:31.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.932+0000 7f6727c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f67200b3c80 con 0x7f672006f880 2024-04-16T16:18:31.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.933+0000 7f671fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6720072c50 0x7f672006e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:31.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.933+0000 7f671fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6720072c50 0x7f672006e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57750/0 (socket says 172.21.15.45:57750) 2024-04-16T16:18:31.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.933+0000 7f671fe00640 1 -- 172.21.15.45:0/1351973377 learned_addr learned my addr 172.21.15.45:0/1351973377 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:31.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.933+0000 7f671fe00640 1 -- 172.21.15.45:0/1351973377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f672006f880 msgr2=0x7f672006fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:31.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.933+0000 7f671fe00640 1 --2- 172.21.15.45:0/1351973377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f672006f880 0x7f672006fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:31.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.933+0000 7f671fe00640 1 -- 172.21.15.45:0/1351973377 >> v1:172.21.15.103:6789/0 conn(0x7f672015a060 legacy=0x7f6720072540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:31.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.933+0000 7f671fe00640 1 -- 172.21.15.45:0/1351973377 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67200b3f60 con 0x7f6720072c50 2024-04-16T16:18:31.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.934+0000 7f671fe00640 1 --2- 172.21.15.45:0/1351973377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6720072c50 0x7f672006e2b0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f6708009870 tx=0x7f670802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=791fa673b5553e05 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:31.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.935+0000 7f671f400640 1 -- 172.21.15.45:0/1351973377 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67080317c0 con 0x7f6720072c50 2024-04-16T16:18:31.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.935+0000 7f671f400640 1 -- 172.21.15.45:0/1351973377 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6708037070 con 0x7f6720072c50 2024-04-16T16:18:31.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.935+0000 7f671f400640 1 -- 172.21.15.45:0/1351973377 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708031b80 con 0x7f6720072c50 2024-04-16T16:18:31.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.935+0000 7f6727c00640 1 -- 172.21.15.45:0/1351973377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6720072c50 msgr2=0x7f672006e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:31.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.935+0000 7f6727c00640 1 --2- 172.21.15.45:0/1351973377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6720072c50 0x7f672006e2b0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f6708009870 tx=0x7f670802b860 comp rx=0 tx=0).stop 2024-04-16T16:18:31.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.935+0000 7f6727c00640 1 -- 172.21.15.45:0/1351973377 shutdown_connections 2024-04-16T16:18:31.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.935+0000 7f6727c00640 1 --2- 172.21.15.45:0/1351973377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f672006f880 0x7f672006fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:31.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.935+0000 7f6727c00640 1 --2- 172.21.15.45:0/1351973377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6720072c50 0x7f672006e2b0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:31.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.935+0000 7f6727c00640 1 -- 172.21.15.45:0/1351973377 >> 172.21.15.45:0/1351973377 conn(0x7f67200b2750 msgr2=0x7f67200b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:31.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.935+0000 7f6727c00640 1 -- 172.21.15.45:0/1351973377 shutdown_connections 2024-04-16T16:18:31.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.935+0000 7f6727c00640 1 -- 172.21.15.45:0/1351973377 wait complete. 2024-04-16T16:18:31.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.937+0000 7f6727c00640 1 Processor -- start 2024-04-16T16:18:31.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.937+0000 7f6727c00640 1 -- start start 2024-04-16T16:18:31.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.938+0000 7f6727c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f672006f880 0x7f67200b1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:31.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.938+0000 7f6727c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f672015a060 0x7f672015a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:31.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.938+0000 7f6727c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f672015ab50 0x7f672015fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:31.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.938+0000 7f6727c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f67200b4f40 con 0x7f672015a060 2024-04-16T16:18:31.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.938+0000 7f6727c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f67200b4520 con 0x7f672015ab50 2024-04-16T16:18:31.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.938+0000 7f6727c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f67200b50c0 con 0x7f672006f880 2024-04-16T16:18:31.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.939+0000 7f671fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f672015a060 0x7f672015a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:31.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.939+0000 7f671fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f672015a060 0x7f672015a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57766/0 (socket says 172.21.15.45:57766) 2024-04-16T16:18:31.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.939+0000 7f671fe00640 1 -- 172.21.15.45:0/749827242 learned_addr learned my addr 172.21.15.45:0/749827242 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:31.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.939+0000 7f6725200640 1 --2- 172.21.15.45:0/749827242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f672006f880 0x7f67200b1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:31.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.939+0000 7f6725c00640 1 --2- 172.21.15.45:0/749827242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f672015ab50 0x7f672015fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:31.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.939+0000 7f671fe00640 1 -- 172.21.15.45:0/749827242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f672006f880 msgr2=0x7f67200b1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:31.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.939+0000 7f671fe00640 1 --2- 172.21.15.45:0/749827242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f672006f880 0x7f67200b1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:31.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.939+0000 7f671fe00640 1 -- 172.21.15.45:0/749827242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f672015ab50 msgr2=0x7f672015fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:31.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.939+0000 7f671fe00640 1 --2- 172.21.15.45:0/749827242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f672015ab50 0x7f672015fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:31.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.939+0000 7f671fe00640 1 -- 172.21.15.45:0/749827242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6720160540 con 0x7f672015a060 2024-04-16T16:18:31.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.939+0000 7f6725200640 1 --2- 172.21.15.45:0/749827242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f672006f880 0x7f67200b1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:31.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.940+0000 7f671fe00640 1 --2- 172.21.15.45:0/749827242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f672015a060 0x7f672015a610 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f6708009870 tx=0x7f67080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:31.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.940+0000 7f671cc00640 1 -- 172.21.15.45:0/749827242 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708032860 con 0x7f672015a060 2024-04-16T16:18:31.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.940+0000 7f671cc00640 1 -- 172.21.15.45:0/749827242 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6708032a00 con 0x7f672015a060 2024-04-16T16:18:31.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.940+0000 7f671cc00640 1 -- 172.21.15.45:0/749827242 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708032df0 con 0x7f672015a060 2024-04-16T16:18:31.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.940+0000 7f6727c00640 1 -- 172.21.15.45:0/749827242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6720160820 con 0x7f672015a060 2024-04-16T16:18:31.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.940+0000 7f6727c00640 1 -- 172.21.15.45:0/749827242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6720157e90 con 0x7f672015a060 2024-04-16T16:18:31.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.941+0000 7f6727c00640 1 -- 172.21.15.45:0/749827242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67201579a0 con 0x7f672015a060 2024-04-16T16:18:31.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.942+0000 7f671cc00640 1 -- 172.21.15.45:0/749827242 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f670804c060 con 0x7f672015a060 2024-04-16T16:18:31.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.944+0000 7f671cc00640 1 --2- 172.21.15.45:0/749827242 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f66fc06ca10 0x7f66fc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:31.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.944+0000 7f671cc00640 1 -- 172.21.15.45:0/749827242 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f67080ad660 con 0x7f672015a060 2024-04-16T16:18:31.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.944+0000 7f6725200640 1 --2- 172.21.15.45:0/749827242 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f66fc06ca10 0x7f66fc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:31.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.946+0000 7f6725200640 1 --2- 172.21.15.45:0/749827242 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f66fc06ca10 0x7f66fc06eed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6710007eb0 tx=0x7f67100073c0 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:31.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:31.949+0000 7f671cc00640 1 -- 172.21.15.45:0/749827242 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6708040070 con 0x7f672015a060 2024-04-16T16:18:32.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.190+0000 7f6727c00640 1 -- 172.21.15.45:0/749827242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6720075780 con 0x7f672015a060 2024-04-16T16:18:32.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.192+0000 7f671cc00640 1 -- 172.21.15.45:0/749827242 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5186 (secure 0 0 0) 0x7f670807b8e0 con 0x7f672015a060 2024-04-16T16:18:32.191 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:32.191 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2514711815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2514711815},{"type":"v1","addr":"172.21.15.103:6836","nonce":2514711815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2189404408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2189404408},{"type":"v1","addr":"172.21.15.103:6837","nonce":2189404408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7179,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/991116998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":991116998},{"type":"v1","addr":"172.21.15.45:6834","nonce":991116998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:18:15.762294+0000","modified":"2024-04-16T16:18:18.858040+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7188":{"gid":7188,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3813781309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3813781309},{"type":"v1","addr":"172.21.15.45:6835","nonce":3813781309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7188,"qdb_cluster":[7188]},"id":6}]} 2024-04-16T16:18:32.192 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 64 2024-04-16T16:18:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.194+0000 7f6727c00640 1 -- 172.21.15.45:0/749827242 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f66fc06ca10 msgr2=0x7f66fc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.194+0000 7f6727c00640 1 --2- 172.21.15.45:0/749827242 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f66fc06ca10 0x7f66fc06eed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6710007eb0 tx=0x7f67100073c0 comp rx=0 tx=0).stop 2024-04-16T16:18:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.194+0000 7f6727c00640 1 -- 172.21.15.45:0/749827242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f672015a060 msgr2=0x7f672015a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.194+0000 7f6727c00640 1 --2- 172.21.15.45:0/749827242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f672015a060 0x7f672015a610 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f6708009870 tx=0x7f67080046c0 comp rx=0 tx=0).stop 2024-04-16T16:18:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.195+0000 7f6727c00640 1 -- 172.21.15.45:0/749827242 shutdown_connections 2024-04-16T16:18:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.195+0000 7f6727c00640 1 --2- 172.21.15.45:0/749827242 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f66fc06ca10 0x7f66fc06eed0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.195+0000 7f6727c00640 1 --2- 172.21.15.45:0/749827242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f672015ab50 0x7f672015fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.195+0000 7f6727c00640 1 --2- 172.21.15.45:0/749827242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f672015a060 0x7f672015a610 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.195+0000 7f6727c00640 1 --2- 172.21.15.45:0/749827242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f672006f880 0x7f67200b1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.195+0000 7f6727c00640 1 -- 172.21.15.45:0/749827242 >> 172.21.15.45:0/749827242 conn(0x7f67200b2750 msgr2=0x7f67200cf520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.195+0000 7f6727c00640 1 -- 172.21.15.45:0/749827242 shutdown_connections 2024-04-16T16:18:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.195+0000 7f6727c00640 1 -- 172.21.15.45:0/749827242 wait complete. 2024-04-16T16:18:32.214 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable stats 2024-04-16T16:18:32.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.451+0000 7f7ef6c00640 1 Processor -- start 2024-04-16T16:18:32.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.451+0000 7f7ef6c00640 1 -- start start 2024-04-16T16:18:32.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.452+0000 7f7ef6c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef015c7c0 0x7f7ef0072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:32.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.452+0000 7f7ef6c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7ef00d10e0 0x7f7ef006f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:32.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.453+0000 7f7ef6c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ef00b5530 con 0x7f7ef00b4930 2024-04-16T16:18:32.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.453+0000 7f7ef6c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7ef00b52e0 con 0x7f7ef00d10e0 2024-04-16T16:18:32.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.453+0000 7f7ef6c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef00d0260 con 0x7f7ef015c7c0 2024-04-16T16:18:32.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.453+0000 7f7eef400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7ef00b4930 0x7f7ef00d09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41170/0 (socket says 172.21.15.45:41170) 2024-04-16T16:18:32.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.453+0000 7f7eef400640 1 -- 172.21.15.45:0/502646499 learned_addr learned my addr 172.21.15.45:0/502646499 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:32.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.453+0000 7f7ef4c00640 1 --2- 172.21.15.45:0/502646499 >> v2:172.21.15.103:3300/0 conn(0x7f7ef00d10e0 0x7f7ef006f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:32.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.453+0000 7f7eefe00640 1 --2- 172.21.15.45:0/502646499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef015c7c0 0x7f7ef0072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:32.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.454+0000 7f7eeea00640 1 -- 172.21.15.45:0/502646499 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2468907808 0 0) 0x7f7ef00b5530 con 0x7f7ef00b4930 2024-04-16T16:18:32.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.454+0000 7f7eeea00640 1 -- 172.21.15.45:0/502646499 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ed4003520 con 0x7f7ef00b4930 2024-04-16T16:18:32.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.454+0000 7f7eeea00640 1 -- 172.21.15.45:0/502646499 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3890623114 0 0) 0x7f7ed4003520 con 0x7f7ef00b4930 2024-04-16T16:18:32.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.454+0000 7f7ef4c00640 1 -- 172.21.15.45:0/502646499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef015c7c0 msgr2=0x7f7ef0072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:32.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.454+0000 7f7ef4c00640 1 --2- 172.21.15.45:0/502646499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef015c7c0 0x7f7ef0072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:32.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.454+0000 7f7ef4c00640 1 -- 172.21.15.45:0/502646499 >> v1:172.21.15.45:6789/0 conn(0x7f7ef00b4930 legacy=0x7f7ef00d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:32.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.454+0000 7f7ef4c00640 1 -- 172.21.15.45:0/502646499 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef00b5810 con 0x7f7ef00d10e0 2024-04-16T16:18:32.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.456+0000 7f7ef4c00640 1 --2- 172.21.15.45:0/502646499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef00d10e0 0x7f7ef006f160 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009d90 tx=0x7f7ee802bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b859d1e9407086 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:32.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.456+0000 7f7eeea00640 1 -- 172.21.15.45:0/502646499 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8039070 con 0x7f7ef00d10e0 2024-04-16T16:18:32.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.456+0000 7f7eeea00640 1 -- 172.21.15.45:0/502646499 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee802c780 con 0x7f7ef00d10e0 2024-04-16T16:18:32.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.456+0000 7f7eeea00640 1 -- 172.21.15.45:0/502646499 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc001670 con 0x7f7ef00d10e0 2024-04-16T16:18:32.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.457+0000 7f7ef6c00640 1 -- 172.21.15.45:0/502646499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef00d10e0 msgr2=0x7f7ef006f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:32.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.457+0000 7f7ef6c00640 1 --2- 172.21.15.45:0/502646499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef00d10e0 0x7f7ef006f160 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009d90 tx=0x7f7ee802bc50 comp rx=0 tx=0).stop 2024-04-16T16:18:32.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.457+0000 7f7ef6c00640 1 -- 172.21.15.45:0/502646499 shutdown_connections 2024-04-16T16:18:32.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.457+0000 7f7ef6c00640 1 --2- 172.21.15.45:0/502646499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef00d10e0 0x7f7ef006f160 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:32.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.457+0000 7f7ef6c00640 1 --2- 172.21.15.45:0/502646499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef015c7c0 0x7f7ef0072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:32.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.457+0000 7f7ef6c00640 1 -- 172.21.15.45:0/502646499 >> 172.21.15.45:0/502646499 conn(0x7f7ef00b3970 msgr2=0x7f7ef00b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:32.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.457+0000 7f7ef6c00640 1 -- 172.21.15.45:0/502646499 shutdown_connections 2024-04-16T16:18:32.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.457+0000 7f7ef6c00640 1 -- 172.21.15.45:0/502646499 wait complete. 2024-04-16T16:18:32.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.459+0000 7f7ef6c00640 1 Processor -- start 2024-04-16T16:18:32.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.459+0000 7f7ef6c00640 1 -- start start 2024-04-16T16:18:32.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.460+0000 7f7ef6c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef00b4930 0x7f7ef00cee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:32.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.460+0000 7f7ef6c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef00d10e0 0x7f7ef00c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:32.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.460+0000 7f7ef6c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef015c7c0 0x7f7ef00cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:32.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.460+0000 7f7ef6c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef006f6a0 con 0x7f7ef00d10e0 2024-04-16T16:18:32.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.460+0000 7f7ef6c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef0073350 con 0x7f7ef015c7c0 2024-04-16T16:18:32.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.460+0000 7f7ef6c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef006f820 con 0x7f7ef00b4930 2024-04-16T16:18:32.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.460+0000 7f7eef400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef00d10e0 0x7f7ef00c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:32.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.460+0000 7f7eef400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef00d10e0 0x7f7ef00c68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57770/0 (socket says 172.21.15.45:57770) 2024-04-16T16:18:32.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.460+0000 7f7eef400640 1 -- 172.21.15.45:0/2404149404 learned_addr learned my addr 172.21.15.45:0/2404149404 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:32.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.460+0000 7f7ef4c00640 1 --2- 172.21.15.45:0/2404149404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef015c7c0 0x7f7ef00cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:32.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.460+0000 7f7eef400640 1 -- 172.21.15.45:0/2404149404 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef00b4930 msgr2=0x7f7ef00cee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:32.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.460+0000 7f7eef400640 1 --2- 172.21.15.45:0/2404149404 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef00b4930 0x7f7ef00cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:32.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.460+0000 7f7eef400640 1 -- 172.21.15.45:0/2404149404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef015c7c0 msgr2=0x7f7ef00cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:32.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.460+0000 7f7eef400640 1 --2- 172.21.15.45:0/2404149404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef015c7c0 0x7f7ef00cbf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:32.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.460+0000 7f7eef400640 1 -- 172.21.15.45:0/2404149404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef00c7010 con 0x7f7ef00d10e0 2024-04-16T16:18:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.461+0000 7f7eef400640 1 --2- 172.21.15.45:0/2404149404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef00d10e0 0x7f7ef00c68a0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00c760 tx=0x7f7edc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.461+0000 7f7ee7e00640 1 -- 172.21.15.45:0/2404149404 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc017070 con 0x7f7ef00d10e0 2024-04-16T16:18:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.461+0000 7f7ee7e00640 1 -- 172.21.15.45:0/2404149404 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7edc005af0 con 0x7f7ef00d10e0 2024-04-16T16:18:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.461+0000 7f7ee7e00640 1 -- 172.21.15.45:0/2404149404 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc007070 con 0x7f7ef00d10e0 2024-04-16T16:18:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.461+0000 7f7ef6c00640 1 -- 172.21.15.45:0/2404149404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef00c72f0 con 0x7f7ef00d10e0 2024-04-16T16:18:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.461+0000 7f7ef6c00640 1 -- 172.21.15.45:0/2404149404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef00c78b0 con 0x7f7ef00d10e0 2024-04-16T16:18:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.462+0000 7f7ef6c00640 1 -- 172.21.15.45:0/2404149404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ef0075780 con 0x7f7ef00d10e0 2024-04-16T16:18:32.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.463+0000 7f7ee7e00640 1 -- 172.21.15.45:0/2404149404 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) ==== 91142+0+0 (secure 0 0 0) 0x7f7edc01a070 con 0x7f7ef00d10e0 2024-04-16T16:18:32.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.464+0000 7f7ee7e00640 1 --2- 172.21.15.45:0/2404149404 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7ed006ca10 0x7f7ed006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:32.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.464+0000 7f7ee7e00640 1 -- 172.21.15.45:0/2404149404 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f7edc08b250 con 0x7f7ef00d10e0 2024-04-16T16:18:32.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.465+0000 7f7eefe00640 1 --2- 172.21.15.45:0/2404149404 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7ed006ca10 0x7f7ed006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:32.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.466+0000 7f7eefe00640 1 --2- 172.21.15.45:0/2404149404 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7ed006ca10 0x7f7ed006eed0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0006a90 tx=0x7f7ee0006a20 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:32.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.469+0000 7f7ee7e00640 1 -- 172.21.15.45:0/2404149404 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7edc059560 con 0x7f7ef00d10e0 2024-04-16T16:18:32.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:32.729+0000 7f7ef6c00640 1 -- 172.21.15.45:0/2404149404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "stats"} v 0) -- 0x7f7ef006d210 con 0x7f7ef00d10e0 2024-04-16T16:18:33.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.419+0000 7f7ee7e00640 1 -- 172.21.15.45:0/2404149404 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "stats"}]=0 v26) ==== 84+0+0 (secure 0 0 0) 0x7f7edc058e60 con 0x7f7ef00d10e0 2024-04-16T16:18:33.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.422+0000 7f7ef6c00640 1 -- 172.21.15.45:0/2404149404 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7ed006ca10 msgr2=0x7f7ed006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:33.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.422+0000 7f7ef6c00640 1 --2- 172.21.15.45:0/2404149404 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7ed006ca10 0x7f7ed006eed0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0006a90 tx=0x7f7ee0006a20 comp rx=0 tx=0).stop 2024-04-16T16:18:33.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.422+0000 7f7ef6c00640 1 -- 172.21.15.45:0/2404149404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef00d10e0 msgr2=0x7f7ef00c68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:33.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.422+0000 7f7ef6c00640 1 --2- 172.21.15.45:0/2404149404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef00d10e0 0x7f7ef00c68a0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00c760 tx=0x7f7edc00cc20 comp rx=0 tx=0).stop 2024-04-16T16:18:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.422+0000 7f7ef6c00640 1 -- 172.21.15.45:0/2404149404 shutdown_connections 2024-04-16T16:18:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.422+0000 7f7ef6c00640 1 --2- 172.21.15.45:0/2404149404 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f7ed006ca10 0x7f7ed006eed0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0006a90 tx=0x7f7ee0006a20 comp rx=0 tx=0).stop 2024-04-16T16:18:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.422+0000 7f7ef6c00640 1 --2- 172.21.15.45:0/2404149404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef015c7c0 0x7f7ef00cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.422+0000 7f7ef6c00640 1 --2- 172.21.15.45:0/2404149404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef00d10e0 0x7f7ef00c68a0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.422+0000 7f7ef6c00640 1 --2- 172.21.15.45:0/2404149404 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef00b4930 0x7f7ef00cee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.422+0000 7f7ef6c00640 1 -- 172.21.15.45:0/2404149404 >> 172.21.15.45:0/2404149404 conn(0x7f7ef00b3970 msgr2=0x7f7ef0159960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:33.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.423+0000 7f7ef6c00640 1 -- 172.21.15.45:0/2404149404 shutdown_connections 2024-04-16T16:18:33.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.423+0000 7f7ef6c00640 1 -- 172.21.15.45:0/2404149404 wait complete. 2024-04-16T16:18:33.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats --client_id abcd 2024-04-16T16:18:33.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.652+0000 7f2df1a00640 1 Processor -- start 2024-04-16T16:18:33.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.652+0000 7f2df1a00640 1 -- start start 2024-04-16T16:18:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.654+0000 7f2df1a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2dec0cdf30 0x7f2dec073a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.654+0000 7f2df1a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2dec06f040 0x7f2dec06f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.654+0000 7f2df1a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dec0b6470 con 0x7f2dec0d0f90 2024-04-16T16:18:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.654+0000 7f2df1a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2dec0b6230 con 0x7f2dec0cdf30 2024-04-16T16:18:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.654+0000 7f2df1a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2dec06f9d0 con 0x7f2dec06f040 2024-04-16T16:18:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.654+0000 7f2deb400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2dec0cdf30 0x7f2dec073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.654+0000 7f2deb400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2dec0cdf30 0x7f2dec073a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57278/0 (socket says 172.21.15.45:57278) 2024-04-16T16:18:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.654+0000 7f2deb400640 1 -- 172.21.15.45:0/3580452841 learned_addr learned my addr 172.21.15.45:0/3580452841 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:33.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.655+0000 7f2df1000640 1 --2- 172.21.15.45:0/3580452841 >> v2:172.21.15.103:3300/0 conn(0x7f2dec06f040 0x7f2dec06f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:33.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.655+0000 7f2deb400640 1 -- 172.21.15.45:0/3580452841 >> v2:172.21.15.103:3300/0 conn(0x7f2dec06f040 msgr2=0x7f2dec06f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:33.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.655+0000 7f2deb400640 1 --2- 172.21.15.45:0/3580452841 >> v2:172.21.15.103:3300/0 conn(0x7f2dec06f040 0x7f2dec06f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:33.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.655+0000 7f2deb400640 1 -- 172.21.15.45:0/3580452841 >> v1:172.21.15.103:6789/0 conn(0x7f2dec0d0f90 legacy=0x7f2dec0b5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:33.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.655+0000 7f2deb400640 1 -- 172.21.15.45:0/3580452841 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dec0b6750 con 0x7f2dec0cdf30 2024-04-16T16:18:33.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.656+0000 7f2deb400640 1 --2- 172.21.15.45:0/3580452841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dec0cdf30 0x7f2dec073a10 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0b7820 tx=0x7f2dd802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b911f6c9307d3dc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:33.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.656+0000 7f2deaa00640 1 -- 172.21.15.45:0/3580452841 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd802e070 con 0x7f2dec0cdf30 2024-04-16T16:18:33.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.656+0000 7f2deaa00640 1 -- 172.21.15.45:0/3580452841 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dd8037070 con 0x7f2dec0cdf30 2024-04-16T16:18:33.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.656+0000 7f2deaa00640 1 -- 172.21.15.45:0/3580452841 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8031a00 con 0x7f2dec0cdf30 2024-04-16T16:18:33.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.657+0000 7f2df1a00640 1 -- 172.21.15.45:0/3580452841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dec0cdf30 msgr2=0x7f2dec073a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:33.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.657+0000 7f2df1a00640 1 --2- 172.21.15.45:0/3580452841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dec0cdf30 0x7f2dec073a10 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0b7820 tx=0x7f2dd802b880 comp rx=0 tx=0).stop 2024-04-16T16:18:33.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.657+0000 7f2df1a00640 1 -- 172.21.15.45:0/3580452841 shutdown_connections 2024-04-16T16:18:33.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.657+0000 7f2df1a00640 1 --2- 172.21.15.45:0/3580452841 >> v2:172.21.15.103:3300/0 conn(0x7f2dec06f040 0x7f2dec06f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.657+0000 7f2df1a00640 1 --2- 172.21.15.45:0/3580452841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dec0cdf30 0x7f2dec073a10 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.657+0000 7f2df1a00640 1 -- 172.21.15.45:0/3580452841 >> 172.21.15.45:0/3580452841 conn(0x7f2dec0b6ae0 msgr2=0x7f2dec0b6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.657+0000 7f2df1a00640 1 -- 172.21.15.45:0/3580452841 shutdown_connections 2024-04-16T16:18:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.657+0000 7f2df1a00640 1 -- 172.21.15.45:0/3580452841 wait complete. 2024-04-16T16:18:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.659+0000 7f2df1a00640 1 Processor -- start 2024-04-16T16:18:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.659+0000 7f2df1a00640 1 -- start start 2024-04-16T16:18:33.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.660+0000 7f2df1a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dec06f040 0x7f2dec0ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:33.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.660+0000 7f2df1a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dec0cdf30 0x7f2dec0b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:33.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.660+0000 7f2df1a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dec0d0f90 0x7f2dec0b2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:33.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.660+0000 7f2df1a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2dec073580 con 0x7f2dec0cdf30 2024-04-16T16:18:33.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.660+0000 7f2df1a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2dec072fc0 con 0x7f2dec06f040 2024-04-16T16:18:33.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.660+0000 7f2df1a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2dec15bd00 con 0x7f2dec0d0f90 2024-04-16T16:18:33.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.661+0000 7f2deb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dec0cdf30 0x7f2dec0b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:33.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.661+0000 7f2deb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dec0cdf30 0x7f2dec0b21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57282/0 (socket says 172.21.15.45:57282) 2024-04-16T16:18:33.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.661+0000 7f2deb400640 1 -- 172.21.15.45:0/35898273 learned_addr learned my addr 172.21.15.45:0/35898273 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:33.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.661+0000 7f2debe00640 1 --2- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dec06f040 0x7f2dec0ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:33.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.661+0000 7f2df1000640 1 --2- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dec0d0f90 0x7f2dec0b2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:33.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.661+0000 7f2deb400640 1 -- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dec0d0f90 msgr2=0x7f2dec0b2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:33.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.661+0000 7f2deb400640 1 --2- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dec0d0f90 0x7f2dec0b2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:33.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.661+0000 7f2deb400640 1 -- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dec06f040 msgr2=0x7f2dec0ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:33.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.661+0000 7f2deb400640 1 --2- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dec06f040 0x7f2dec0ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:33.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.661+0000 7f2deb400640 1 -- 172.21.15.45:0/35898273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dec143500 con 0x7f2dec0cdf30 2024-04-16T16:18:33.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.662+0000 7f2deb400640 1 --2- 172.21.15.45:0/35898273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dec0cdf30 0x7f2dec0b21f0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8000c00 tx=0x7f2dd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:33.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.662+0000 7f2de3e00640 1 -- 172.21.15.45:0/35898273 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8032670 con 0x7f2dec0cdf30 2024-04-16T16:18:33.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.662+0000 7f2de3e00640 1 -- 172.21.15.45:0/35898273 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dd8032830 con 0x7f2dec0cdf30 2024-04-16T16:18:33.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.662+0000 7f2de3e00640 1 -- 172.21.15.45:0/35898273 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8032c20 con 0x7f2dec0cdf30 2024-04-16T16:18:33.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.662+0000 7f2df1a00640 1 -- 172.21.15.45:0/35898273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dec1437e0 con 0x7f2dec0cdf30 2024-04-16T16:18:33.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.663+0000 7f2df1a00640 1 -- 172.21.15.45:0/35898273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dec15d270 con 0x7f2dec0cdf30 2024-04-16T16:18:33.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.664+0000 7f2df1a00640 1 -- 172.21.15.45:0/35898273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dec075780 con 0x7f2dec0cdf30 2024-04-16T16:18:33.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.666+0000 7f2de3e00640 1 -- 172.21.15.45:0/35898273 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) ==== 91151+0+0 (secure 0 0 0) 0x7f2dd80453f0 con 0x7f2dec0cdf30 2024-04-16T16:18:33.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.667+0000 7f2de3e00640 1 --2- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2dcc06ca60 0x7f2dcc06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:33.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.667+0000 7f2de3e00640 1 -- 172.21.15.45:0/35898273 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f2dd80adb60 con 0x7f2dec0cdf30 2024-04-16T16:18:33.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.667+0000 7f2debe00640 1 -- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2dcc06ca60 msgr2=0x7f2dcc06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3728775368 2024-04-16T16:18:33.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.667+0000 7f2debe00640 1 --2- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2dcc06ca60 0x7f2dcc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:18:33.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.672+0000 7f2de3e00640 1 -- 172.21.15.45:0/35898273 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2dd8040070 con 0x7f2dec0cdf30 2024-04-16T16:18:33.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.868+0000 7f2debe00640 1 -- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2dcc06ca60 msgr2=0x7f2dcc06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3728775368 2024-04-16T16:18:33.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.868+0000 7f2debe00640 1 --2- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2dcc06ca60 0x7f2dcc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:18:33.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:33.906+0000 7f2df1a00640 1 -- 172.21.15.45:0/35898273 --> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] -- mgr_command(tid 0: {"prefix": "fs perf stats", "client_id": "abcd", "target": ["mon-mgr", ""]}) -- 0x7f2dec0cd120 con 0x7f2dcc06ca60 2024-04-16T16:18:33.967 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:33.967+0000 7f7937cf8040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:18:33.989 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:33.990+0000 7fde4ec2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:18:34.021 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:34.021+0000 7f7937cf8040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:18:34.046 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:34.047+0000 7fde4ec2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:18:34.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:34.269+0000 7f2debe00640 1 -- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2dcc06ca60 msgr2=0x7f2dcc06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3728775368 2024-04-16T16:18:34.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:34.269+0000 7f2debe00640 1 --2- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2dcc06ca60 0x7f2dcc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T16:18:34.467 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:34.467+0000 7f7937cf8040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:18:34.511 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:34.512+0000 7fde4ec2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:18:34.518 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:34.519+0000 7f7937cf8040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:18:34.566 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:34.567+0000 7fde4ec2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:18:34.568 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:34.569+0000 7f7937cf8040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:18:34.618 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:34.620+0000 7fde4ec2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:18:34.629 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:34.630+0000 7f7937cf8040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:18:34.678 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:34.679+0000 7f7937cf8040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:18:34.682 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:34.684+0000 7fde4ec2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:18:34.734 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:34.735+0000 7fde4ec2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:18:34.808 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:34.808+0000 7f7937cf8040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:18:34.870 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:34.871+0000 7fde4ec2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:18:34.871 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:34.872+0000 7f7937cf8040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:18:34.923 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:34.923+0000 7f7937cf8040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:18:34.937 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:34.938+0000 7fde4ec2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:18:34.980 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:34.980+0000 7f7937cf8040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:18:34.989 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:34.990+0000 7fde4ec2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:18:35.031 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:35.031+0000 7f7937cf8040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:18:35.049 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:35.050+0000 7fde4ec2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:18:35.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:35.070+0000 7f2debe00640 1 -- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2dcc06ca60 msgr2=0x7f2dcc06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3728775368 2024-04-16T16:18:35.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:35.070+0000 7f2debe00640 1 --2- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2dcc06ca60 0x7f2dcc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T16:18:35.102 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:35.103+0000 7fde4ec2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:18:35.415 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:35.416+0000 7f7937cf8040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:18:35.503 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:35.504+0000 7fde4ec2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:18:35.574 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:35.574+0000 7f7937cf8040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:18:35.671 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:35.672+0000 7fde4ec2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:18:35.682 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:35.683+0000 7f7937cf8040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:18:35.733 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:35.733+0000 7f7937cf8040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:18:35.784 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:35.786+0000 7fde4ec2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:18:35.837 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:35.838+0000 7fde4ec2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:18:35.859 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:35.859+0000 7f7937cf8040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:18:35.965 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:35.966+0000 7f7937cf8040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:18:35.970 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:35.971+0000 7fde4ec2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:18:36.082 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:36.083+0000 7fde4ec2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:18:36.140 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:36.140+0000 7f7937cf8040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:18:36.188 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:36.189+0000 7f7937cf8040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:18:36.265 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:36.266+0000 7fde4ec2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:18:36.317 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:36.318+0000 7fde4ec2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:18:36.324 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:36.325+0000 7f7937cf8040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:18:36.460 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:36.461+0000 7fde4ec2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:18:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:36.672+0000 7f2debe00640 1 -- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2dcc06ca60 msgr2=0x7f2dcc06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3728775368 2024-04-16T16:18:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:36.672+0000 7f2debe00640 1 --2- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2dcc06ca60 0x7f2dcc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T16:18:37.160 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:37.160+0000 7f7937cf8040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:18:37.343 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:37.344+0000 7fde4ec2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:18:37.463 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:37.463+0000 7f7937cf8040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:18:37.520 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:37.520+0000 7f7937cf8040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:18:37.652 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:37.652+0000 7f7937cf8040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:18:37.662 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:37.663+0000 7fde4ec2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:18:37.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:37.673+0000 7f2de3e00640 1 -- 172.21.15.45:0/35898273 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 27) ==== 90760+0+0 (secure 0 0 0) 0x7f2dd8077d80 con 0x7f2dec0cdf30 2024-04-16T16:18:37.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:37.674+0000 7f2de3e00640 1 -- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2dcc06ca60 msgr2=0x7f2dcc06ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:18:37.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:37.674+0000 7f2de3e00640 1 --2- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/3728775368,v1:172.21.15.103:6833/3728775368] conn(0x7f2dcc06ca60 0x7f2dcc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:37.727 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:37.728+0000 7fde4ec2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:18:37.760 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:37.761+0000 7f78b4000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:37.760 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:37.761+0000 7f78b4000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:37.760 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:37.761+0000 7f78b4000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:37.760 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:37.761+0000 7f78b4000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:37.760 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:37.761+0000 7f78b4000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:37.761 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:37.761+0000 7f78b0400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:37.761 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:37.761+0000 7f78b0400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:37.761 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:37.761+0000 7f78b0400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:37.761 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:37.761+0000 7f78b0400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:37.761 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:37.761+0000 7f78b0400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:37.871 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:37.872+0000 7fde4ec2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:18:38.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.689+0000 7f2de3e00640 1 -- 172.21.15.45:0/35898273 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 28) ==== 90887+0+0 (secure 0 0 0) 0x7f2dd80abfe0 con 0x7f2dec0cdf30 2024-04-16T16:18:38.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.690+0000 7f2de3e00640 1 --2- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f2dcc074910 0x7f2dcc076d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:38.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.690+0000 7f2de3e00640 1 -- 172.21.15.45:0/35898273 --> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] -- mgr_command(tid 0: {"prefix": "fs perf stats", "client_id": "abcd", "target": ["mon-mgr", ""]}) -- 0x7f2dd8037e70 con 0x7f2dcc074910 2024-04-16T16:18:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.698+0000 7f2debe00640 1 --2- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f2dcc074910 0x7f2dcc076d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.709+0000 7f2debe00640 1 --2- 172.21.15.45:0/35898273 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f2dcc074910 0x7f2dcc076d00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2dec136a10 tx=0x7f2ddc009060 comp rx=0 tx=0).ready entity=mgr.5627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:38.712 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:38.713+0000 7f78bf400640 -1 mgr.server reply reply (22) Invalid argument invalid client_id filter spec: abcd 2024-04-16T16:18:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.714+0000 7f2de3e00640 1 -- 172.21.15.45:0/35898273 <== mgr.5627 v2:172.21.15.103:6832/1293636998 1 ==== mgr_command_reply(tid 0: -22 invalid client_id filter spec: abcd) ==== 43+0+0 (secure 0 0 0) 0x7f2dd8037e70 con 0x7f2dcc074910 2024-04-16T16:18:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:Error EINVAL: invalid client_id filter spec: abcd 2024-04-16T16:18:38.730 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T16:18:38.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable stats 2024-04-16T16:18:38.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.957+0000 7f807da00640 1 Processor -- start 2024-04-16T16:18:38.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.958+0000 7f807da00640 1 -- start start 2024-04-16T16:18:38.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.959+0000 7f807da00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8078147ff0 0x7f807806df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:38.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.959+0000 7f807da00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f807806e5b0 con 0x7f807815f960 2024-04-16T16:18:38.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.959+0000 7f807da00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f807806e890 con 0x7f8078072630 2024-04-16T16:18:38.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.959+0000 7f807da00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f807815fde0 con 0x7f8078147ff0 2024-04-16T16:18:38.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.960+0000 7f8077400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f807815f960 0x7f8078071f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60684/0 (socket says 172.21.15.45:60684) 2024-04-16T16:18:38.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.960+0000 7f8077400640 1 -- 172.21.15.45:0/2403244613 learned_addr learned my addr 172.21.15.45:0/2403244613 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.960+0000 7f8077e00640 1 --2- 172.21.15.45:0/2403244613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8078147ff0 0x7f807806df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.960+0000 7f8076a00640 1 -- 172.21.15.45:0/2403244613 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3554767297 0 0) 0x7f807806e5b0 con 0x7f807815f960 2024-04-16T16:18:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.960+0000 7f8076a00640 1 -- 172.21.15.45:0/2403244613 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f805c003520 con 0x7f807815f960 2024-04-16T16:18:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f8076a00640 1 -- 172.21.15.45:0/2403244613 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2443036967 0 0) 0x7f805c003520 con 0x7f807815f960 2024-04-16T16:18:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f8076a00640 1 -- 172.21.15.45:0/2403244613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8078147ff0 msgr2=0x7f807806df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f8076a00640 1 --2- 172.21.15.45:0/2403244613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8078147ff0 0x7f807806df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f8076a00640 1 -- 172.21.15.45:0/2403244613 >> v1:172.21.15.103:6789/0 conn(0x7f8078072630 legacy=0x7f80780af150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f8076a00640 1 -- 172.21.15.45:0/2403244613 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807806eb70 con 0x7f807815f960 2024-04-16T16:18:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f8076a00640 1 -- 172.21.15.45:0/2403244613 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f80640017f0 con 0x7f807815f960 2024-04-16T16:18:38.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f8076a00640 1 -- 172.21.15.45:0/2403244613 >> v1:172.21.15.45:6789/0 conn(0x7f807815f960 legacy=0x7f8078071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:38.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f8076a00640 1 --2- 172.21.15.45:0/2403244613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f805c0039e0 0x7f805c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:38.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f8076a00640 1 --2- 172.21.15.45:0/2403244613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8078147ff0 0x7f805c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:38.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f8076a00640 1 --2- 172.21.15.45:0/2403244613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f805c004d40 0x7f805c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:38.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f8076a00640 1 -- 172.21.15.45:0/2403244613 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f807815fde0 con 0x7f805c004d40 2024-04-16T16:18:38.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f8076a00640 1 -- 172.21.15.45:0/2403244613 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f805c00ba70 con 0x7f805c0039e0 2024-04-16T16:18:38.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f8076a00640 1 -- 172.21.15.45:0/2403244613 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f805c00bc20 con 0x7f8078147ff0 2024-04-16T16:18:38.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f807d000640 1 --2- 172.21.15.45:0/2403244613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f805c004d40 0x7f805c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:38.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f8077e00640 1 --2- 172.21.15.45:0/2403244613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f805c0039e0 0x7f805c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:38.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.961+0000 7f8077400640 1 --2- 172.21.15.45:0/2403244613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8078147ff0 0x7f805c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:38.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.962+0000 7f807d000640 1 -- 172.21.15.45:0/2403244613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8078147ff0 msgr2=0x7f805c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:38.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.962+0000 7f807d000640 1 --2- 172.21.15.45:0/2403244613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8078147ff0 0x7f805c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:38.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.962+0000 7f807d000640 1 -- 172.21.15.45:0/2403244613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f805c0039e0 msgr2=0x7f805c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:38.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.962+0000 7f807d000640 1 --2- 172.21.15.45:0/2403244613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f805c0039e0 0x7f805c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:38.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.962+0000 7f807d000640 1 -- 172.21.15.45:0/2403244613 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805c003520 con 0x7f805c004d40 2024-04-16T16:18:38.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.962+0000 7f8077e00640 1 --2- 172.21.15.45:0/2403244613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f805c0039e0 0x7f805c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:38.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.963+0000 7f807d000640 1 --2- 172.21.15.45:0/2403244613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f805c004d40 0x7f805c0051c0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f80700044b0 tx=0x7f805c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d99187265c50f34 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:38.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.963+0000 7f8076a00640 1 -- 172.21.15.45:0/2403244613 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070034070 con 0x7f805c004d40 2024-04-16T16:18:38.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.963+0000 7f8076a00640 1 -- 172.21.15.45:0/2403244613 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8070030620 con 0x7f805c004d40 2024-04-16T16:18:38.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.963+0000 7f8076a00640 1 -- 172.21.15.45:0/2403244613 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070004080 con 0x7f805c004d40 2024-04-16T16:18:38.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.963+0000 7f807da00640 1 -- 172.21.15.45:0/2403244613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f805c004d40 msgr2=0x7f805c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:38.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.963+0000 7f807da00640 1 --2- 172.21.15.45:0/2403244613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f805c004d40 0x7f805c0051c0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f80700044b0 tx=0x7f805c005830 comp rx=0 tx=0).stop 2024-04-16T16:18:38.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.964+0000 7f807d000640 1 -- 172.21.15.45:0/2403244613 reap_dead start 2024-04-16T16:18:38.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.964+0000 7f807da00640 1 -- 172.21.15.45:0/2403244613 shutdown_connections 2024-04-16T16:18:38.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.964+0000 7f807da00640 1 -- 172.21.15.45:0/2403244613 >> 172.21.15.45:0/2403244613 conn(0x7f8078067e00 msgr2=0x7f807806b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:38.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.964+0000 7f807da00640 1 -- 172.21.15.45:0/2403244613 shutdown_connections 2024-04-16T16:18:38.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.964+0000 7f807da00640 1 -- 172.21.15.45:0/2403244613 wait complete. 2024-04-16T16:18:38.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.966+0000 7f807da00640 1 Processor -- start 2024-04-16T16:18:38.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.966+0000 7f807da00640 1 -- start start 2024-04-16T16:18:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.967+0000 7f807da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80781f4330 0x7f8078160340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.967+0000 7f807da00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80781f4710 0x7f8078160880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.967+0000 7f807da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80780b0300 0x7f80780b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.967+0000 7f807da00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f80781483d0 con 0x7f80780b0300 2024-04-16T16:18:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.967+0000 7f807da00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f80780aebc0 con 0x7f80781f4330 2024-04-16T16:18:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.967+0000 7f807da00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8078160030 con 0x7f80781f4710 2024-04-16T16:18:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.967+0000 7f807d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80780b0300 0x7f80780b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.967+0000 7f807d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80780b0300 0x7f80780b0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57298/0 (socket says 172.21.15.45:57298) 2024-04-16T16:18:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.967+0000 7f807d000640 1 -- 172.21.15.45:0/3833823328 learned_addr learned my addr 172.21.15.45:0/3833823328 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.968+0000 7f8077e00640 1 --2- 172.21.15.45:0/3833823328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80781f4330 0x7f8078160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.968+0000 7f8077400640 1 --2- 172.21.15.45:0/3833823328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80781f4710 0x7f8078160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.968+0000 7f807d000640 1 -- 172.21.15.45:0/3833823328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80781f4710 msgr2=0x7f8078160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.968+0000 7f807d000640 1 --2- 172.21.15.45:0/3833823328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80781f4710 0x7f8078160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:38.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.968+0000 7f807d000640 1 -- 172.21.15.45:0/3833823328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80781f4330 msgr2=0x7f8078160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:38.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.968+0000 7f807d000640 1 --2- 172.21.15.45:0/3833823328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80781f4330 0x7f8078160340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:38.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.968+0000 7f807d000640 1 -- 172.21.15.45:0/3833823328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8078141580 con 0x7f80780b0300 2024-04-16T16:18:38.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.968+0000 7f807d000640 1 --2- 172.21.15.45:0/3833823328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80780b0300 0x7f80780b0780 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f807002bf80 tx=0x7f8070003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:38.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.968+0000 7f806fe00640 1 -- 172.21.15.45:0/3833823328 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070034050 con 0x7f80780b0300 2024-04-16T16:18:38.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.968+0000 7f806fe00640 1 -- 172.21.15.45:0/3833823328 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8070030420 con 0x7f80780b0300 2024-04-16T16:18:38.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.968+0000 7f806fe00640 1 -- 172.21.15.45:0/3833823328 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070007df0 con 0x7f80780b0300 2024-04-16T16:18:38.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.968+0000 7f807da00640 1 -- 172.21.15.45:0/3833823328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80780b0fe0 con 0x7f80780b0300 2024-04-16T16:18:38.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.968+0000 7f807da00640 1 -- 172.21.15.45:0/3833823328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80780b15a0 con 0x7f80780b0300 2024-04-16T16:18:38.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.970+0000 7f807da00640 1 -- 172.21.15.45:0/3833823328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8078075780 con 0x7f80780b0300 2024-04-16T16:18:38.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.970+0000 7f806fe00640 1 -- 172.21.15.45:0/3833823328 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 28) ==== 90887+0+0 (secure 0 0 0) 0x7f8070049020 con 0x7f80780b0300 2024-04-16T16:18:38.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.971+0000 7f806fe00640 1 --2- 172.21.15.45:0/3833823328 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f805806c790 0x7f805806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:38.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.971+0000 7f806fe00640 1 -- 172.21.15.45:0/3833823328 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6053+0+0 (secure 0 0 0) 0x7f80700ace30 con 0x7f80780b0300 2024-04-16T16:18:38.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.972+0000 7f8077e00640 1 --2- 172.21.15.45:0/3833823328 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f805806c790 0x7f805806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:38.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.973+0000 7f8077e00640 1 --2- 172.21.15.45:0/3833823328 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f805806c790 0x7f805806ec50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f80680099c0 tx=0x7f8068008040 comp rx=0 tx=0).ready entity=mgr.5627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:38.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:38.976+0000 7f806fe00640 1 -- 172.21.15.45:0/3833823328 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8078141580 con 0x7f80780b0300 2024-04-16T16:18:39.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:39.239+0000 7f807da00640 1 -- 172.21.15.45:0/3833823328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "stats"} v 0) -- 0x7f807806f4b0 con 0x7f80780b0300 2024-04-16T16:18:39.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:39.734+0000 7f806fe00640 1 -- 172.21.15.45:0/3833823328 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "stats"}]=0 v29) ==== 85+0+0 (secure 0 0 0) 0x7f8070081030 con 0x7f80780b0300 2024-04-16T16:18:39.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:39.738+0000 7f807da00640 1 -- 172.21.15.45:0/3833823328 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f805806c790 msgr2=0x7f805806ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:39.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:39.738+0000 7f807da00640 1 --2- 172.21.15.45:0/3833823328 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f805806c790 0x7f805806ec50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f80680099c0 tx=0x7f8068008040 comp rx=0 tx=0).stop 2024-04-16T16:18:39.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:39.738+0000 7f807da00640 1 -- 172.21.15.45:0/3833823328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80780b0300 msgr2=0x7f80780b0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:39.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:39.738+0000 7f807da00640 1 --2- 172.21.15.45:0/3833823328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80780b0300 0x7f80780b0780 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f807002bf80 tx=0x7f8070003a70 comp rx=0 tx=0).stop 2024-04-16T16:18:39.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:39.738+0000 7f807da00640 1 -- 172.21.15.45:0/3833823328 shutdown_connections 2024-04-16T16:18:39.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:39.738+0000 7f807da00640 1 --2- 172.21.15.45:0/3833823328 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f805806c790 0x7f805806ec50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:39.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:39.738+0000 7f807da00640 1 --2- 172.21.15.45:0/3833823328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80780b0300 0x7f80780b0780 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:39.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:39.738+0000 7f807da00640 1 --2- 172.21.15.45:0/3833823328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80781f4710 0x7f8078160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:39.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:39.738+0000 7f807da00640 1 --2- 172.21.15.45:0/3833823328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80781f4330 0x7f8078160340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:39.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:39.738+0000 7f807da00640 1 -- 172.21.15.45:0/3833823328 >> 172.21.15.45:0/3833823328 conn(0x7f8078067e00 msgr2=0x7f80780b4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:39.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:39.738+0000 7f807da00640 1 -- 172.21.15.45:0/3833823328 shutdown_connections 2024-04-16T16:18:39.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:39.738+0000 7f807da00640 1 -- 172.21.15.45:0/3833823328 wait complete. 2024-04-16T16:18:39.760 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-16T16:18:39.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:18:39.887 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:18:39.995 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-16T16:18:39.995 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:18:39.996 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:18:39.996 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:40.053 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:18:40.053 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:40.104 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:18:40.299 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:40.299+0000 7fc5514fa040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:18:40.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.320+0000 7f76bfe00640 1 Processor -- start 2024-04-16T16:18:40.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.321+0000 7f76bfe00640 1 -- start start 2024-04-16T16:18:40.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.322+0000 7f76bfe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f76c006ef00 0x7f76c015d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:40.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.322+0000 7f76bfe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76c0141ec0 0x7f76c006bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:40.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.322+0000 7f76bfe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76c01425a0 con 0x7f76c0159fe0 2024-04-16T16:18:40.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.322+0000 7f76bfe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f76c0142350 con 0x7f76c006ef00 2024-04-16T16:18:40.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.322+0000 7f76bfe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f76c0142880 con 0x7f76c0141ec0 2024-04-16T16:18:40.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.323+0000 7f76bf400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76c0141ec0 0x7f76c006bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:40.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.323+0000 7f76be000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f76c006ef00 0x7f76c015d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:40.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.323+0000 7f76be000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f76c006ef00 0x7f76c015d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:60252/0 (socket says 172.21.15.45:60252) 2024-04-16T16:18:40.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.323+0000 7f76be000640 1 -- 172.21.15.45:0/3894414750 learned_addr learned my addr 172.21.15.45:0/3894414750 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:40.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.324+0000 7f76bf400640 1 -- 172.21.15.45:0/3894414750 >> v2:172.21.15.103:3300/0 conn(0x7f76c006ef00 msgr2=0x7f76c015d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:40.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.324+0000 7f76bf400640 1 --2- 172.21.15.45:0/3894414750 >> v2:172.21.15.103:3300/0 conn(0x7f76c006ef00 0x7f76c015d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.324+0000 7f76bf400640 1 -- 172.21.15.45:0/3894414750 >> v1:172.21.15.103:6789/0 conn(0x7f76c0159fe0 legacy=0x7f76c006e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:40.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.324+0000 7f76bf400640 1 -- 172.21.15.45:0/3894414750 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c0142b60 con 0x7f76c0141ec0 2024-04-16T16:18:40.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.326+0000 7f76bf400640 1 --2- 172.21.15.45:0/3894414750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76c0141ec0 0x7f76c006bff0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009d90 tx=0x7f76b802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d4e83e886347732 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:40.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.326+0000 7f76bd600640 1 -- 172.21.15.45:0/3894414750 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b8039070 con 0x7f76c0141ec0 2024-04-16T16:18:40.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.326+0000 7f76bd600640 1 -- 172.21.15.45:0/3894414750 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76b802cbc0 con 0x7f76c0141ec0 2024-04-16T16:18:40.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.326+0000 7f76bd600640 1 -- 172.21.15.45:0/3894414750 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b8034440 con 0x7f76c0141ec0 2024-04-16T16:18:40.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.327+0000 7f76bfe00640 1 -- 172.21.15.45:0/3894414750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76c0141ec0 msgr2=0x7f76c006bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:40.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.327+0000 7f76bfe00640 1 --2- 172.21.15.45:0/3894414750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76c0141ec0 0x7f76c006bff0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009d90 tx=0x7f76b802be10 comp rx=0 tx=0).stop 2024-04-16T16:18:40.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.327+0000 7f76bfe00640 1 -- 172.21.15.45:0/3894414750 shutdown_connections 2024-04-16T16:18:40.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.327+0000 7f76bfe00640 1 --2- 172.21.15.45:0/3894414750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76c0141ec0 0x7f76c006bff0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.327+0000 7f76bfe00640 1 --2- 172.21.15.45:0/3894414750 >> v2:172.21.15.103:3300/0 conn(0x7f76c006ef00 0x7f76c015d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.327+0000 7f76bfe00640 1 -- 172.21.15.45:0/3894414750 >> 172.21.15.45:0/3894414750 conn(0x7f76c00d0f10 msgr2=0x7f76c00d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:40.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.327+0000 7f76bfe00640 1 -- 172.21.15.45:0/3894414750 shutdown_connections 2024-04-16T16:18:40.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.327+0000 7f76bfe00640 1 -- 172.21.15.45:0/3894414750 wait complete. 2024-04-16T16:18:40.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.329+0000 7f76bfe00640 1 Processor -- start 2024-04-16T16:18:40.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.329+0000 7f76bfe00640 1 -- start start 2024-04-16T16:18:40.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76bfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76c006ef00 0x7f76c00cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:40.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76bfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76c0141ec0 0x7f76c015f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76bfe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76c0159fe0 0x7f76c0139590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76bfe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f76c015a540 con 0x7f76c006ef00 2024-04-16T16:18:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76bfe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f76c015a3c0 con 0x7f76c0141ec0 2024-04-16T16:18:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76bfe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f76c006c530 con 0x7f76c0159fe0 2024-04-16T16:18:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76bea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76c006ef00 0x7f76c00cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76bea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76c006ef00 0x7f76c00cd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57316/0 (socket says 172.21.15.45:57316) 2024-04-16T16:18:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76bea00640 1 -- 172.21.15.45:0/1911254441 learned_addr learned my addr 172.21.15.45:0/1911254441 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76bf400640 1 --2- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76c0159fe0 0x7f76c0139590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76be000640 1 --2- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76c0141ec0 0x7f76c015f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76bea00640 1 -- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76c0159fe0 msgr2=0x7f76c0139590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76bea00640 1 --2- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76c0159fe0 0x7f76c0139590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76bea00640 1 -- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76c0141ec0 msgr2=0x7f76c015f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76bea00640 1 --2- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76c0141ec0 0x7f76c015f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.330+0000 7f76bea00640 1 -- 172.21.15.45:0/1911254441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c015a760 con 0x7f76c006ef00 2024-04-16T16:18:40.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.331+0000 7f76be000640 1 --2- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76c0141ec0 0x7f76c015f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:40.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.331+0000 7f76bea00640 1 --2- 172.21.15.45:0/1911254441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76c006ef00 0x7f76c00cd3b0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f76b000c3f0 tx=0x7f76b000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:40.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.331+0000 7f76b6a00640 1 -- 172.21.15.45:0/1911254441 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b00052d0 con 0x7f76c006ef00 2024-04-16T16:18:40.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.331+0000 7f76b6a00640 1 -- 172.21.15.45:0/1911254441 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76b0005490 con 0x7f76c006ef00 2024-04-16T16:18:40.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.331+0000 7f76bfe00640 1 -- 172.21.15.45:0/1911254441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76c015f830 con 0x7f76c006ef00 2024-04-16T16:18:40.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.331+0000 7f76b6a00640 1 -- 172.21.15.45:0/1911254441 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0007260 con 0x7f76c006ef00 2024-04-16T16:18:40.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.331+0000 7f76bfe00640 1 -- 172.21.15.45:0/1911254441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76c015ace0 con 0x7f76c006ef00 2024-04-16T16:18:40.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.332+0000 7f76bfe00640 1 -- 172.21.15.45:0/1911254441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76c0075780 con 0x7f76c006ef00 2024-04-16T16:18:40.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.333+0000 7f76b6a00640 1 -- 172.21.15.45:0/1911254441 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 29) ==== 91086+0+0 (secure 0 0 0) 0x7f76b0007400 con 0x7f76c006ef00 2024-04-16T16:18:40.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.334+0000 7f76b6a00640 1 --2- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f76a006c980 0x7f76a006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:40.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.334+0000 7f76b6a00640 1 -- 172.21.15.45:0/1911254441 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6053+0+0 (secure 0 0 0) 0x7f76b008b260 con 0x7f76c006ef00 2024-04-16T16:18:40.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.334+0000 7f76be000640 1 -- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f76a006c980 msgr2=0x7f76a006ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1293636998 2024-04-16T16:18:40.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.335+0000 7f76be000640 1 --2- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f76a006c980 0x7f76a006ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:18:40.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.343+0000 7f76b6a00640 1 -- 172.21.15.45:0/1911254441 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f76b00594d0 con 0x7f76c006ef00 2024-04-16T16:18:40.356 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:40.357+0000 7f3e69032040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:18:40.362 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:40.363+0000 7fc5514fa040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:18:40.414 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:40.415+0000 7f3e69032040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:18:40.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.536+0000 7f76be000640 1 -- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f76a006c980 msgr2=0x7f76a006ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1293636998 2024-04-16T16:18:40.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.536+0000 7f76be000640 1 --2- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f76a006c980 0x7f76a006ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:18:40.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.548+0000 7f76bfe00640 1 -- 172.21.15.45:0/1911254441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f76c015f0d0 con 0x7f76c006ef00 2024-04-16T16:18:40.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.550+0000 7f76b6a00640 1 -- 172.21.15.45:0/1911254441 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+17167 (secure 0 0 0) 0x7f76b0058dd0 con 0x7f76c006ef00 2024-04-16T16:18:40.549 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:40.549 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":86,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:18:37.658709+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:18:13.633261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-16T16:18:14.757055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3489855481":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/674186192":"2024-04-17T16:18:37.658582+0000","172.21.15.103:6833/3728775368":"2024-04-17T16:18:37.658582+0000","172.21.15.103:6832/3728775368":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/4160076462":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/1325022358":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/981309315":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/3673976854":"2024-04-17T16:18:37.658582+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:18:40.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.552+0000 7f76bfe00640 1 -- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f76a006c980 msgr2=0x7f76a006ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:18:40.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.552+0000 7f76bfe00640 1 --2- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f76a006c980 0x7f76a006ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.552+0000 7f76bfe00640 1 -- 172.21.15.45:0/1911254441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76c006ef00 msgr2=0x7f76c00cd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:40.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.552+0000 7f76bfe00640 1 --2- 172.21.15.45:0/1911254441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76c006ef00 0x7f76c00cd3b0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f76b000c3f0 tx=0x7f76b000c8b0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.552+0000 7f76bfe00640 1 -- 172.21.15.45:0/1911254441 shutdown_connections 2024-04-16T16:18:40.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.552+0000 7f76bfe00640 1 --2- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f76a006c980 0x7f76a006ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.552+0000 7f76bfe00640 1 --2- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76c0159fe0 0x7f76c0139590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.552+0000 7f76bfe00640 1 --2- 172.21.15.45:0/1911254441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76c0141ec0 0x7f76c015f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.552+0000 7f76bfe00640 1 --2- 172.21.15.45:0/1911254441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76c006ef00 0x7f76c00cd3b0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.553+0000 7f76bfe00640 1 -- 172.21.15.45:0/1911254441 >> 172.21.15.45:0/1911254441 conn(0x7f76c00d0f10 msgr2=0x7f76c00d12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:40.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.553+0000 7f76bfe00640 1 -- 172.21.15.45:0/1911254441 shutdown_connections 2024-04-16T16:18:40.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.553+0000 7f76bfe00640 1 -- 172.21.15.45:0/1911254441 wait complete. 2024-04-16T16:18:40.570 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:18:13.633261+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:18:14.757055+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:18:40.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:18:40.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.788+0000 7fde6a800640 1 Processor -- start 2024-04-16T16:18:40.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.788+0000 7fde6a800640 1 -- start start 2024-04-16T16:18:40.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.790+0000 7fde6a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fde6406cf10 0x7fde6406d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:40.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.790+0000 7fde6a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde640d35d0 0x7fde640d3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:40.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.790+0000 7fde6a800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde6406d8a0 con 0x7fde6415ecc0 2024-04-16T16:18:40.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.790+0000 7fde6a800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fde640d2b10 con 0x7fde6406cf10 2024-04-16T16:18:40.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.790+0000 7fde6a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fde64163210 con 0x7fde640d35d0 2024-04-16T16:18:40.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.790+0000 7fde63400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fde6415ecc0 0x7fde641626a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60688/0 (socket says 172.21.15.45:60688) 2024-04-16T16:18:40.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.790+0000 7fde63400640 1 -- 172.21.15.45:0/1979423025 learned_addr learned my addr 172.21.15.45:0/1979423025 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:40.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde63e00640 1 --2- 172.21.15.45:0/1979423025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde640d35d0 0x7fde640d3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 -- 172.21.15.45:0/1979423025 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1810033667 0 0) 0x7fde6406d8a0 con 0x7fde6415ecc0 2024-04-16T16:18:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62a00640 1 --2- 172.21.15.45:0/1979423025 >> v2:172.21.15.103:3300/0 conn(0x7fde6406cf10 0x7fde6406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 -- 172.21.15.45:0/1979423025 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde48003520 con 0x7fde6415ecc0 2024-04-16T16:18:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 -- 172.21.15.45:0/1979423025 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 342439008 0 0) 0x7fde48003520 con 0x7fde6415ecc0 2024-04-16T16:18:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 -- 172.21.15.45:0/1979423025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde640d35d0 msgr2=0x7fde640d3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 --2- 172.21.15.45:0/1979423025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde640d35d0 0x7fde640d3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 -- 172.21.15.45:0/1979423025 >> v2:172.21.15.103:3300/0 conn(0x7fde6406cf10 msgr2=0x7fde6406d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 --2- 172.21.15.45:0/1979423025 >> v2:172.21.15.103:3300/0 conn(0x7fde6406cf10 0x7fde6406d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 -- 172.21.15.45:0/1979423025 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde641634f0 con 0x7fde6415ecc0 2024-04-16T16:18:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 -- 172.21.15.45:0/1979423025 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fde54001670 con 0x7fde6415ecc0 2024-04-16T16:18:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62a00640 1 --2- 172.21.15.45:0/1979423025 >> v2:172.21.15.103:3300/0 conn(0x7fde6406cf10 0x7fde6406d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:18:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 -- 172.21.15.45:0/1979423025 >> v1:172.21.15.45:6789/0 conn(0x7fde6415ecc0 legacy=0x7fde641626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 --2- 172.21.15.45:0/1979423025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde640d35d0 0x7fde480039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:40.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 --2- 172.21.15.45:0/1979423025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde48003f80 0x7fde48004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 --2- 172.21.15.45:0/1979423025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde48004d40 0x7fde480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 -- 172.21.15.45:0/1979423025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fde640d2b10 con 0x7fde48004d40 2024-04-16T16:18:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 -- 172.21.15.45:0/1979423025 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde64163210 con 0x7fde48003f80 2024-04-16T16:18:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.791+0000 7fde62000640 1 -- 172.21.15.45:0/1979423025 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fde4800bb00 con 0x7fde640d35d0 2024-04-16T16:18:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.792+0000 7fde63e00640 1 --2- 172.21.15.45:0/1979423025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde48004d40 0x7fde480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.792+0000 7fde63400640 1 --2- 172.21.15.45:0/1979423025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde640d35d0 0x7fde480039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.792+0000 7fde62a00640 1 --2- 172.21.15.45:0/1979423025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde48003f80 0x7fde48004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.792+0000 7fde63e00640 1 -- 172.21.15.45:0/1979423025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde640d35d0 msgr2=0x7fde480039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.792+0000 7fde63e00640 1 --2- 172.21.15.45:0/1979423025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde640d35d0 0x7fde480039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.792+0000 7fde63e00640 1 -- 172.21.15.45:0/1979423025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde48003f80 msgr2=0x7fde48004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.792+0000 7fde63e00640 1 --2- 172.21.15.45:0/1979423025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde48003f80 0x7fde48004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.792+0000 7fde63e00640 1 -- 172.21.15.45:0/1979423025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde48003520 con 0x7fde48004d40 2024-04-16T16:18:40.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.793+0000 7fde63e00640 1 --2- 172.21.15.45:0/1979423025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde48004d40 0x7fde480051c0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fde5c009fc0 tx=0x7fde48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74a40c5429bc4c2b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:40.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.793+0000 7fde62000640 1 -- 172.21.15.45:0/1979423025 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c039070 con 0x7fde48004d40 2024-04-16T16:18:40.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.793+0000 7fde62000640 1 -- 172.21.15.45:0/1979423025 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde5c02cc20 con 0x7fde48004d40 2024-04-16T16:18:40.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.793+0000 7fde62000640 1 -- 172.21.15.45:0/1979423025 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c009c10 con 0x7fde48004d40 2024-04-16T16:18:40.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.794+0000 7fde6a800640 1 -- 172.21.15.45:0/1979423025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde48004d40 msgr2=0x7fde480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:40.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.794+0000 7fde6a800640 1 --2- 172.21.15.45:0/1979423025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde48004d40 0x7fde480051c0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fde5c009fc0 tx=0x7fde48005830 comp rx=0 tx=0).stop 2024-04-16T16:18:40.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.794+0000 7fde63e00640 1 -- 172.21.15.45:0/1979423025 reap_dead start 2024-04-16T16:18:40.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.794+0000 7fde6a800640 1 -- 172.21.15.45:0/1979423025 shutdown_connections 2024-04-16T16:18:40.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.794+0000 7fde6a800640 1 -- 172.21.15.45:0/1979423025 >> 172.21.15.45:0/1979423025 conn(0x7fde64073690 msgr2=0x7fde640d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:40.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.794+0000 7fde6a800640 1 -- 172.21.15.45:0/1979423025 shutdown_connections 2024-04-16T16:18:40.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.794+0000 7fde6a800640 1 -- 172.21.15.45:0/1979423025 wait complete. 2024-04-16T16:18:40.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.796+0000 7fde6a800640 1 Processor -- start 2024-04-16T16:18:40.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.796+0000 7fde6a800640 1 -- start start 2024-04-16T16:18:40.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.797+0000 7fde6a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde48004d40 0x7fde640c8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:40.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.797+0000 7fde6a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde641f8950 0x7fde640c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:40.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.797+0000 7fde6a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde6415c2b0 0x7fde6415c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:40.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.797+0000 7fde6a800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fde640ba540 con 0x7fde48004d40 2024-04-16T16:18:40.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.797+0000 7fde6a800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde640d30b0 con 0x7fde641f8950 2024-04-16T16:18:40.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.797+0000 7fde6a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fde64162db0 con 0x7fde6415c2b0 2024-04-16T16:18:40.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.797+0000 7fde63400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde48004d40 0x7fde640c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:40.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.797+0000 7fde63400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde48004d40 0x7fde640c8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57346/0 (socket says 172.21.15.45:57346) 2024-04-16T16:18:40.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.797+0000 7fde63400640 1 -- 172.21.15.45:0/547553530 learned_addr learned my addr 172.21.15.45:0/547553530 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:40.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.798+0000 7fde63e00640 1 --2- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde6415c2b0 0x7fde6415c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:40.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.798+0000 7fde62a00640 1 --2- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde641f8950 0x7fde640c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.798+0000 7fde63400640 1 -- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde6415c2b0 msgr2=0x7fde6415c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.798+0000 7fde63400640 1 --2- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde6415c2b0 0x7fde6415c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.798+0000 7fde63400640 1 -- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde641f8950 msgr2=0x7fde640c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.798+0000 7fde63400640 1 --2- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde641f8950 0x7fde640c77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.798+0000 7fde63400640 1 -- 172.21.15.45:0/547553530 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde640c8d80 con 0x7fde48004d40 2024-04-16T16:18:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.798+0000 7fde63400640 1 --2- 172.21.15.45:0/547553530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde48004d40 0x7fde640c8480 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fde54001b20 tx=0x7fde5400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.798+0000 7fde5b400640 1 -- 172.21.15.45:0/547553530 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5400c8d0 con 0x7fde48004d40 2024-04-16T16:18:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.798+0000 7fde5b400640 1 -- 172.21.15.45:0/547553530 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde54018070 con 0x7fde48004d40 2024-04-16T16:18:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.798+0000 7fde5b400640 1 -- 172.21.15.45:0/547553530 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5400cc90 con 0x7fde48004d40 2024-04-16T16:18:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.799+0000 7fde6a800640 1 -- 172.21.15.45:0/547553530 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde640c9060 con 0x7fde48004d40 2024-04-16T16:18:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.799+0000 7fde6a800640 1 -- 172.21.15.45:0/547553530 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde640c9b20 con 0x7fde48004d40 2024-04-16T16:18:40.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.800+0000 7fde6a800640 1 -- 172.21.15.45:0/547553530 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde6415cc70 con 0x7fde48004d40 2024-04-16T16:18:40.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.801+0000 7fde5b400640 1 -- 172.21.15.45:0/547553530 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 29) ==== 91086+0+0 (secure 0 0 0) 0x7fde54013420 con 0x7fde48004d40 2024-04-16T16:18:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.802+0000 7fde5b400640 1 --2- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fde4406c9b0 0x7fde4406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.802+0000 7fde5b400640 1 -- 172.21.15.45:0/547553530 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6053+0+0 (secure 0 0 0) 0x7fde5408ba70 con 0x7fde48004d40 2024-04-16T16:18:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.802+0000 7fde62a00640 1 -- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fde4406c9b0 msgr2=0x7fde4406ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1293636998 2024-04-16T16:18:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.802+0000 7fde62a00640 1 --2- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fde4406c9b0 0x7fde4406ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:18:40.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:40.806+0000 7fde5b400640 1 -- 172.21.15.45:0/547553530 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fde54059ce0 con 0x7fde48004d40 2024-04-16T16:18:40.806 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:40.807+0000 7fc5514fa040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:18:40.859 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:40.859+0000 7fc5514fa040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:18:40.893 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:40.893+0000 7f3e69032040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:18:40.909 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:40.910+0000 7fc5514fa040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:18:40.948 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:40.949+0000 7f3e69032040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:18:40.970 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:40.971+0000 7fc5514fa040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:18:41.001 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:41.002+0000 7f3e69032040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:18:41.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.003+0000 7fde62a00640 1 -- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fde4406c9b0 msgr2=0x7fde4406ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1293636998 2024-04-16T16:18:41.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.003+0000 7fde62a00640 1 --2- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fde4406c9b0 0x7fde4406ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:18:41.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.009+0000 7fde6a800640 1 -- 172.21.15.45:0/547553530 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fde640d26c0 con 0x7fde48004d40 2024-04-16T16:18:41.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.009+0000 7fde5b400640 1 -- 172.21.15.45:0/547553530 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) ==== 93+0+10 (secure 0 0 0) 0x7fde540595e0 con 0x7fde48004d40 2024-04-16T16:18:41.009 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:18:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.012+0000 7fde6a800640 1 -- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fde4406c9b0 msgr2=0x7fde4406ee70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:18:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.012+0000 7fde6a800640 1 --2- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fde4406c9b0 0x7fde4406ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.012+0000 7fde6a800640 1 -- 172.21.15.45:0/547553530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde48004d40 msgr2=0x7fde640c8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.012+0000 7fde6a800640 1 --2- 172.21.15.45:0/547553530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde48004d40 0x7fde640c8480 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fde54001b20 tx=0x7fde5400ecd0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.012+0000 7fde6a800640 1 -- 172.21.15.45:0/547553530 shutdown_connections 2024-04-16T16:18:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.012+0000 7fde6a800640 1 --2- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fde4406c9b0 0x7fde4406ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.012+0000 7fde6a800640 1 --2- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde6415c2b0 0x7fde6415c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.012+0000 7fde6a800640 1 --2- 172.21.15.45:0/547553530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde641f8950 0x7fde640c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.012+0000 7fde6a800640 1 --2- 172.21.15.45:0/547553530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde48004d40 0x7fde640c8480 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.012+0000 7fde6a800640 1 -- 172.21.15.45:0/547553530 >> 172.21.15.45:0/547553530 conn(0x7fde64073690 msgr2=0x7fde6415b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.012+0000 7fde6a800640 1 -- 172.21.15.45:0/547553530 shutdown_connections 2024-04-16T16:18:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.012+0000 7fde6a800640 1 -- 172.21.15.45:0/547553530 wait complete. 2024-04-16T16:18:41.020 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:41.021+0000 7fc5514fa040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:18:41.030 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:18:41.067 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:41.068+0000 7f3e69032040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:18:41.120 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:41.121+0000 7f3e69032040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:18:41.147 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:41.147+0000 7fc5514fa040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:18:41.210 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:41.211+0000 7fc5514fa040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:18:41.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.246+0000 7f75e1800640 1 Processor -- start 2024-04-16T16:18:41.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.246+0000 7f75e1800640 1 -- start start 2024-04-16T16:18:41.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.248+0000 7f75e1800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75dc06c040 0x7f75dc0cfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.248+0000 7f75e1800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75dc160390 con 0x7f75dc146120 2024-04-16T16:18:41.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.248+0000 7f75e1800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75dc160630 con 0x7f75dc0cdf70 2024-04-16T16:18:41.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.248+0000 7f75e1800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f75dc160f60 con 0x7f75dc06c040 2024-04-16T16:18:41.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.248+0000 7f75dbe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f75dc146120 0x7f75dc06b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60702/0 (socket says 172.21.15.45:60702) 2024-04-16T16:18:41.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.248+0000 7f75dbe00640 1 -- 172.21.15.45:0/2348651837 learned_addr learned my addr 172.21.15.45:0/2348651837 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:41.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75e0e00640 1 --2- 172.21.15.45:0/2348651837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75dc06c040 0x7f75dc0cfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:41.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75daa00640 1 -- 172.21.15.45:0/2348651837 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 720893134 0 0) 0x7f75dc160390 con 0x7f75dc146120 2024-04-16T16:18:41.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75daa00640 1 -- 172.21.15.45:0/2348651837 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75c0003520 con 0x7f75dc146120 2024-04-16T16:18:41.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75daa00640 1 -- 172.21.15.45:0/2348651837 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1248389143 0 0) 0x7f75c0003520 con 0x7f75dc146120 2024-04-16T16:18:41.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75daa00640 1 -- 172.21.15.45:0/2348651837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75dc06c040 msgr2=0x7f75dc0cfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:41.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75daa00640 1 --2- 172.21.15.45:0/2348651837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75dc06c040 0x7f75dc0cfcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75daa00640 1 -- 172.21.15.45:0/2348651837 >> v1:172.21.15.103:6789/0 conn(0x7f75dc0cdf70 legacy=0x7f75dc0cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75daa00640 1 -- 172.21.15.45:0/2348651837 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75dc160910 con 0x7f75dc146120 2024-04-16T16:18:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75daa00640 1 -- 172.21.15.45:0/2348651837 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f75cc001690 con 0x7f75dc146120 2024-04-16T16:18:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75daa00640 1 -- 172.21.15.45:0/2348651837 >> v1:172.21.15.45:6789/0 conn(0x7f75dc146120 legacy=0x7f75dc06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75daa00640 1 --2- 172.21.15.45:0/2348651837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75dc06c040 0x7f75c0003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75daa00640 1 --2- 172.21.15.45:0/2348651837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c0003fd0 0x7f75c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75daa00640 1 --2- 172.21.15.45:0/2348651837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75c0004d40 0x7f75c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75daa00640 1 -- 172.21.15.45:0/2348651837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc160f60 con 0x7f75c0003fd0 2024-04-16T16:18:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75daa00640 1 -- 172.21.15.45:0/2348651837 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f75c000bb20 con 0x7f75c0004d40 2024-04-16T16:18:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.249+0000 7f75daa00640 1 -- 172.21.15.45:0/2348651837 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f75c000bd20 con 0x7f75dc06c040 2024-04-16T16:18:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.250+0000 7f75db400640 1 --2- 172.21.15.45:0/2348651837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c0003fd0 0x7f75c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.250+0000 7f75dbe00640 1 --2- 172.21.15.45:0/2348651837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75dc06c040 0x7f75c0003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:41.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.250+0000 7f75e0e00640 1 --2- 172.21.15.45:0/2348651837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75c0004d40 0x7f75c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:41.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.250+0000 7f75db400640 1 -- 172.21.15.45:0/2348651837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75dc06c040 msgr2=0x7f75c0003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:41.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.250+0000 7f75db400640 1 --2- 172.21.15.45:0/2348651837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75dc06c040 0x7f75c0003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.250+0000 7f75db400640 1 -- 172.21.15.45:0/2348651837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75c0004d40 msgr2=0x7f75c00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:41.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.250+0000 7f75db400640 1 --2- 172.21.15.45:0/2348651837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75c0004d40 0x7f75c00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.250+0000 7f75db400640 1 -- 172.21.15.45:0/2348651837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c0003520 con 0x7f75c0003fd0 2024-04-16T16:18:41.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.251+0000 7f75db400640 1 --2- 172.21.15.45:0/2348651837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c0003fd0 0x7f75c0004800 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f75c8001750 tx=0x7f75c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2160df1cb82cb946 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:41.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.252+0000 7f75daa00640 1 -- 172.21.15.45:0/2348651837 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c8001370 con 0x7f75c0003fd0 2024-04-16T16:18:41.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.252+0000 7f75daa00640 1 -- 172.21.15.45:0/2348651837 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c80052d0 con 0x7f75c0003fd0 2024-04-16T16:18:41.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.252+0000 7f75daa00640 1 -- 172.21.15.45:0/2348651837 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c8003830 con 0x7f75c0003fd0 2024-04-16T16:18:41.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.252+0000 7f75e1800640 1 -- 172.21.15.45:0/2348651837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c0003fd0 msgr2=0x7f75c0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:41.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.252+0000 7f75e1800640 1 --2- 172.21.15.45:0/2348651837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c0003fd0 0x7f75c0004800 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f75c8001750 tx=0x7f75c0005830 comp rx=0 tx=0).stop 2024-04-16T16:18:41.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.252+0000 7f75e0e00640 1 -- 172.21.15.45:0/2348651837 reap_dead start 2024-04-16T16:18:41.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.253+0000 7f75e1800640 1 -- 172.21.15.45:0/2348651837 shutdown_connections 2024-04-16T16:18:41.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.253+0000 7f75e1800640 1 -- 172.21.15.45:0/2348651837 >> 172.21.15.45:0/2348651837 conn(0x7f75dc06f120 msgr2=0x7f75dc06f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:41.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.253+0000 7f75e1800640 1 -- 172.21.15.45:0/2348651837 shutdown_connections 2024-04-16T16:18:41.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.253+0000 7f75e1800640 1 -- 172.21.15.45:0/2348651837 wait complete. 2024-04-16T16:18:41.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.255+0000 7f75e1800640 1 Processor -- start 2024-04-16T16:18:41.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.256+0000 7f75e1800640 1 -- start start 2024-04-16T16:18:41.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.257+0000 7f75e1800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c0003fd0 0x7f75dc13dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.257+0000 7f75e1800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75dc1e7a50 0x7f75dc1e6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.257+0000 7f75e1800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75dc13fd00 0x7f75dc140180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.257+0000 7f75e1800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc0cfe30 con 0x7f75c0003fd0 2024-04-16T16:18:41.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.257+0000 7f75e1800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc0cfcb0 con 0x7f75dc13fd00 2024-04-16T16:18:41.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.257+0000 7f75e1800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f75dc0cffb0 con 0x7f75dc1e7a50 2024-04-16T16:18:41.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.257+0000 7f75dbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c0003fd0 0x7f75dc13dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:41.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.258+0000 7f75dbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c0003fd0 0x7f75dc13dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57378/0 (socket says 172.21.15.45:57378) 2024-04-16T16:18:41.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.258+0000 7f75dbe00640 1 -- 172.21.15.45:0/3621219703 learned_addr learned my addr 172.21.15.45:0/3621219703 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:41.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.258+0000 7f75e0e00640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75dc13fd00 0x7f75dc140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:41.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.258+0000 7f75db400640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75dc1e7a50 0x7f75dc1e6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:41.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.258+0000 7f75dbe00640 1 -- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75dc1e7a50 msgr2=0x7f75dc1e6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:41.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.258+0000 7f75dbe00640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75dc1e7a50 0x7f75dc1e6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.258+0000 7f75dbe00640 1 -- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75dc13fd00 msgr2=0x7f75dc140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:41.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.258+0000 7f75dbe00640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75dc13fd00 0x7f75dc140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.258+0000 7f75dbe00640 1 -- 172.21.15.45:0/3621219703 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75dc1e6a30 con 0x7f75c0003fd0 2024-04-16T16:18:41.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.258+0000 7f75db400640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75dc1e7a50 0x7f75dc1e6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:41.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.258+0000 7f75e0e00640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75dc13fd00 0x7f75dc140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:41.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.258+0000 7f75dbe00640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c0003fd0 0x7f75dc13dda0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0072c0 tx=0x7f75cc007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:41.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.259+0000 7f75d3e00640 1 -- 172.21.15.45:0/3621219703 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc00f9c0 con 0x7f75c0003fd0 2024-04-16T16:18:41.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.259+0000 7f75d3e00640 1 -- 172.21.15.45:0/3621219703 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75cc018070 con 0x7f75c0003fd0 2024-04-16T16:18:41.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.259+0000 7f75d3e00640 1 -- 172.21.15.45:0/3621219703 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc00fda0 con 0x7f75c0003fd0 2024-04-16T16:18:41.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.259+0000 7f75e1800640 1 -- 172.21.15.45:0/3621219703 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75dc1e6d10 con 0x7f75c0003fd0 2024-04-16T16:18:41.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.259+0000 7f75e1800640 1 -- 172.21.15.45:0/3621219703 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75dc1e72d0 con 0x7f75c0003fd0 2024-04-16T16:18:41.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.260+0000 7f75e1800640 1 -- 172.21.15.45:0/3621219703 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75dc075780 con 0x7f75c0003fd0 2024-04-16T16:18:41.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.262+0000 7f75d3e00640 1 -- 172.21.15.45:0/3621219703 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 29) ==== 91086+0+0 (secure 0 0 0) 0x7f75cc01d460 con 0x7f75c0003fd0 2024-04-16T16:18:41.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.263+0000 7f75d3e00640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f75bc06c980 0x7f75bc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.263+0000 7f75d3e00640 1 -- 172.21.15.45:0/3621219703 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6053+0+0 (secure 0 0 0) 0x7f75cc08be70 con 0x7f75c0003fd0 2024-04-16T16:18:41.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.263+0000 7f75db400640 1 -- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f75bc06c980 msgr2=0x7f75bc06ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1293636998 2024-04-16T16:18:41.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.263+0000 7f75db400640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f75bc06c980 0x7f75bc06ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:18:41.263 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:41.263+0000 7fc5514fa040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:18:41.264 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:41.266+0000 7f3e69032040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:18:41.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.268+0000 7f75d3e00640 1 -- 172.21.15.45:0/3621219703 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f75cc05a050 con 0x7f75c0003fd0 2024-04-16T16:18:41.319 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:41.320+0000 7fc5514fa040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:18:41.332 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:41.333+0000 7f3e69032040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:18:41.370 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:41.370+0000 7fc5514fa040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:18:41.385 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:41.386+0000 7f3e69032040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:18:41.445 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:41.446+0000 7f3e69032040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:18:41.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.464+0000 7f75db400640 1 -- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f75bc06c980 msgr2=0x7f75bc06ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1293636998 2024-04-16T16:18:41.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.464+0000 7f75db400640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f75bc06c980 0x7f75bc06ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:18:41.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.468+0000 7f75e1800640 1 -- 172.21.15.45:0/3621219703 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f75dc070850 con 0x7f75c0003fd0 2024-04-16T16:18:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.469+0000 7f75d3e00640 1 -- 172.21.15.45:0/3621219703 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) ==== 104+0+11 (secure 0 0 0) 0x7f75cc059950 con 0x7f75c0003fd0 2024-04-16T16:18:41.468 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:18:41.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.472+0000 7f75e1800640 1 -- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f75bc06c980 msgr2=0x7f75bc06ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:18:41.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.472+0000 7f75e1800640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f75bc06c980 0x7f75bc06ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.472+0000 7f75e1800640 1 -- 172.21.15.45:0/3621219703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c0003fd0 msgr2=0x7f75dc13dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:41.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.472+0000 7f75e1800640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c0003fd0 0x7f75dc13dda0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0072c0 tx=0x7f75cc007780 comp rx=0 tx=0).stop 2024-04-16T16:18:41.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.472+0000 7f75e1800640 1 -- 172.21.15.45:0/3621219703 shutdown_connections 2024-04-16T16:18:41.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.472+0000 7f75e1800640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f75bc06c980 0x7f75bc06ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.472+0000 7f75e1800640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75dc13fd00 0x7f75dc140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.472+0000 7f75e1800640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75dc1e7a50 0x7f75dc1e6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.472+0000 7f75e1800640 1 --2- 172.21.15.45:0/3621219703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c0003fd0 0x7f75dc13dda0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.472+0000 7f75e1800640 1 -- 172.21.15.45:0/3621219703 >> 172.21.15.45:0/3621219703 conn(0x7f75dc06f120 msgr2=0x7f75dc15ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:41.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.472+0000 7f75e1800640 1 -- 172.21.15.45:0/3621219703 shutdown_connections 2024-04-16T16:18:41.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.472+0000 7f75e1800640 1 -- 172.21.15.45:0/3621219703 wait complete. 2024-04-16T16:18:41.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:18:41.500 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:41.501+0000 7f3e69032040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:18:41.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.720+0000 7fb226a00640 1 Processor -- start 2024-04-16T16:18:41.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.720+0000 7fb226a00640 1 -- start start 2024-04-16T16:18:41.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.722+0000 7fb226a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb2200d0e80 0x7fb2200b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.722+0000 7fb226a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2200afda0 0x7fb22006ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.722+0000 7fb226a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2200ccb70 con 0x7fb22015b920 2024-04-16T16:18:41.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.722+0000 7fb226a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb22006e7a0 con 0x7fb2200d0e80 2024-04-16T16:18:41.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.722+0000 7fb226a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb22006f170 con 0x7fb2200afda0 2024-04-16T16:18:41.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.723+0000 7fb21fe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb22015b920 0x7fb2200d0770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60716/0 (socket says 172.21.15.45:60716) 2024-04-16T16:18:41.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.723+0000 7fb21fe00640 1 -- 172.21.15.45:0/3175450341 learned_addr learned my addr 172.21.15.45:0/3175450341 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:41.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.723+0000 7fb224a00640 1 --2- 172.21.15.45:0/3175450341 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2200afda0 0x7fb22006ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:41.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.723+0000 7fb21f400640 1 --2- 172.21.15.45:0/3175450341 >> v2:172.21.15.103:3300/0 conn(0x7fb2200d0e80 0x7fb2200b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:41.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.723+0000 7fb21ea00640 1 -- 172.21.15.45:0/3175450341 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3506131903 0 0) 0x7fb2200ccb70 con 0x7fb22015b920 2024-04-16T16:18:41.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.723+0000 7fb21ea00640 1 -- 172.21.15.45:0/3175450341 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb204005730 con 0x7fb22015b920 2024-04-16T16:18:41.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.724+0000 7fb21ea00640 1 -- 172.21.15.45:0/3175450341 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3298588354 0 0) 0x7fb204005730 con 0x7fb22015b920 2024-04-16T16:18:41.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.724+0000 7fb21ea00640 1 -- 172.21.15.45:0/3175450341 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2200afda0 msgr2=0x7fb22006ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:41.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.724+0000 7fb21ea00640 1 --2- 172.21.15.45:0/3175450341 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2200afda0 0x7fb22006ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.724+0000 7fb21ea00640 1 -- 172.21.15.45:0/3175450341 >> v2:172.21.15.103:3300/0 conn(0x7fb2200d0e80 msgr2=0x7fb2200b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:41.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.724+0000 7fb21ea00640 1 --2- 172.21.15.45:0/3175450341 >> v2:172.21.15.103:3300/0 conn(0x7fb2200d0e80 0x7fb2200b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.724+0000 7fb21ea00640 1 -- 172.21.15.45:0/3175450341 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2200cce50 con 0x7fb22015b920 2024-04-16T16:18:41.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.724+0000 7fb21ea00640 1 -- 172.21.15.45:0/3175450341 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fb210001690 con 0x7fb22015b920 2024-04-16T16:18:41.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.724+0000 7fb21ea00640 1 -- 172.21.15.45:0/3175450341 >> v1:172.21.15.45:6789/0 conn(0x7fb22015b920 legacy=0x7fb2200d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:41.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.724+0000 7fb21ea00640 1 --2- 172.21.15.45:0/3175450341 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2200afda0 0x7fb204005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.724+0000 7fb21ea00640 1 --2- 172.21.15.45:0/3175450341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb204006190 0x7fb204006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.724+0000 7fb21ea00640 1 --2- 172.21.15.45:0/3175450341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb204006f50 0x7fb2040073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.724+0000 7fb21ea00640 1 -- 172.21.15.45:0/3175450341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb22006e7a0 con 0x7fb204006190 2024-04-16T16:18:41.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.724+0000 7fb21ea00640 1 -- 172.21.15.45:0/3175450341 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb22006f170 con 0x7fb204006f50 2024-04-16T16:18:41.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.724+0000 7fb21ea00640 1 -- 172.21.15.45:0/3175450341 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb20400dbd0 con 0x7fb2200afda0 2024-04-16T16:18:41.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.724+0000 7fb21f400640 1 --2- 172.21.15.45:0/3175450341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb204006190 0x7fb204006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:41.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.725+0000 7fb21fe00640 1 --2- 172.21.15.45:0/3175450341 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2200afda0 0x7fb204005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:41.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.725+0000 7fb224a00640 1 --2- 172.21.15.45:0/3175450341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb204006f50 0x7fb2040073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:41.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.725+0000 7fb21f400640 1 -- 172.21.15.45:0/3175450341 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2200afda0 msgr2=0x7fb204005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:41.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.725+0000 7fb21f400640 1 --2- 172.21.15.45:0/3175450341 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2200afda0 0x7fb204005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.725+0000 7fb21f400640 1 -- 172.21.15.45:0/3175450341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb204006f50 msgr2=0x7fb2040073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:41.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.725+0000 7fb21f400640 1 --2- 172.21.15.45:0/3175450341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb204006f50 0x7fb2040073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.725+0000 7fb21f400640 1 -- 172.21.15.45:0/3175450341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb204005730 con 0x7fb204006190 2024-04-16T16:18:41.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.727+0000 7fb21f400640 1 --2- 172.21.15.45:0/3175450341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb204006190 0x7fb204006a10 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fb20c009ac0 tx=0x7fb204007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac1c02e71597c981 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:41.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.727+0000 7fb21ea00640 1 -- 172.21.15.45:0/3175450341 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c039070 con 0x7fb204006190 2024-04-16T16:18:41.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.727+0000 7fb21ea00640 1 -- 172.21.15.45:0/3175450341 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb20c02c6b0 con 0x7fb204006190 2024-04-16T16:18:41.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.727+0000 7fb21ea00640 1 -- 172.21.15.45:0/3175450341 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c009690 con 0x7fb204006190 2024-04-16T16:18:41.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.727+0000 7fb226a00640 1 -- 172.21.15.45:0/3175450341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb204006190 msgr2=0x7fb204006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:41.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.727+0000 7fb226a00640 1 --2- 172.21.15.45:0/3175450341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb204006190 0x7fb204006a10 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fb20c009ac0 tx=0x7fb204007a40 comp rx=0 tx=0).stop 2024-04-16T16:18:41.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.727+0000 7fb224a00640 1 -- 172.21.15.45:0/3175450341 reap_dead start 2024-04-16T16:18:41.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.728+0000 7fb226a00640 1 -- 172.21.15.45:0/3175450341 shutdown_connections 2024-04-16T16:18:41.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.728+0000 7fb226a00640 1 -- 172.21.15.45:0/3175450341 >> 172.21.15.45:0/3175450341 conn(0x7fb2200b3970 msgr2=0x7fb2200b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:41.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.728+0000 7fb226a00640 1 -- 172.21.15.45:0/3175450341 shutdown_connections 2024-04-16T16:18:41.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.728+0000 7fb226a00640 1 -- 172.21.15.45:0/3175450341 wait complete. 2024-04-16T16:18:41.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.729+0000 7fb226a00640 1 Processor -- start 2024-04-16T16:18:41.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.730+0000 7fb226a00640 1 -- start start 2024-04-16T16:18:41.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.730+0000 7fb226a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb204006190 0x7fb220141960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.730+0000 7fb226a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2201420d0 0x7fb220142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.730+0000 7fb226a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2200b0c90 0x7fb2200b1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.730+0000 7fb226a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb22006edb0 con 0x7fb2200b0c90 2024-04-16T16:18:41.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.731+0000 7fb226a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb22006ec30 con 0x7fb2201420d0 2024-04-16T16:18:41.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.731+0000 7fb226a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb22006ef30 con 0x7fb204006190 2024-04-16T16:18:41.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.731+0000 7fb224a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2200b0c90 0x7fb2200b1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:41.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.731+0000 7fb224a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2200b0c90 0x7fb2200b1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57394/0 (socket says 172.21.15.45:57394) 2024-04-16T16:18:41.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.731+0000 7fb224a00640 1 -- 172.21.15.45:0/3777031111 learned_addr learned my addr 172.21.15.45:0/3777031111 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:41.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.731+0000 7fb21fe00640 1 --2- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb204006190 0x7fb220141960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:41.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.731+0000 7fb21f400640 1 --2- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2201420d0 0x7fb220142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:41.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.731+0000 7fb224a00640 1 -- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb204006190 msgr2=0x7fb220141960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:41.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.731+0000 7fb224a00640 1 --2- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb204006190 0x7fb220141960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.731+0000 7fb224a00640 1 -- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2201420d0 msgr2=0x7fb220142550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:41.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.731+0000 7fb224a00640 1 --2- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2201420d0 0x7fb220142550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.731+0000 7fb224a00640 1 -- 172.21.15.45:0/3777031111 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb220141ea0 con 0x7fb2200b0c90 2024-04-16T16:18:41.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.732+0000 7fb224a00640 1 --2- 172.21.15.45:0/3777031111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2200b0c90 0x7fb2200b1110 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fb22015f2a0 tx=0x7fb2180115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:41.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.732+0000 7fb217e00640 1 -- 172.21.15.45:0/3777031111 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb218011e40 con 0x7fb2200b0c90 2024-04-16T16:18:41.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.732+0000 7fb217e00640 1 -- 172.21.15.45:0/3777031111 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb21801b070 con 0x7fb2200b0c90 2024-04-16T16:18:41.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.732+0000 7fb217e00640 1 -- 172.21.15.45:0/3777031111 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb218003990 con 0x7fb2200b0c90 2024-04-16T16:18:41.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.732+0000 7fb226a00640 1 -- 172.21.15.45:0/3777031111 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb220153110 con 0x7fb2200b0c90 2024-04-16T16:18:41.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.732+0000 7fb226a00640 1 -- 172.21.15.45:0/3777031111 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2200b0490 con 0x7fb2200b0c90 2024-04-16T16:18:41.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.734+0000 7fb226a00640 1 -- 172.21.15.45:0/3777031111 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb220154400 con 0x7fb2200b0c90 2024-04-16T16:18:41.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.734+0000 7fb217e00640 1 -- 172.21.15.45:0/3777031111 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 29) ==== 91086+0+0 (secure 0 0 0) 0x7fb218005d10 con 0x7fb2200b0c90 2024-04-16T16:18:41.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.736+0000 7fb217e00640 1 --2- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fb20006c9b0 0x7fb20006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:41.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.736+0000 7fb217e00640 1 -- 172.21.15.45:0/3777031111 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6053+0+0 (secure 0 0 0) 0x7fb21808c240 con 0x7fb2200b0c90 2024-04-16T16:18:41.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.736+0000 7fb21fe00640 1 -- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fb20006c9b0 msgr2=0x7fb20006ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1293636998 2024-04-16T16:18:41.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.736+0000 7fb21fe00640 1 --2- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fb20006c9b0 0x7fb20006ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:18:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.740+0000 7fb217e00640 1 -- 172.21.15.45:0/3777031111 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb218025020 con 0x7fb2200b0c90 2024-04-16T16:18:41.753 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:41.754+0000 7fc5514fa040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:18:41.912 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:41.913+0000 7f3e69032040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:18:41.913 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:41.913+0000 7fc5514fa040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:18:41.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.937+0000 7fb21fe00640 1 -- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fb20006c9b0 msgr2=0x7fb20006ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1293636998 2024-04-16T16:18:41.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.937+0000 7fb21fe00640 1 --2- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fb20006c9b0 0x7fb20006ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:18:41.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.942+0000 7fb226a00640 1 -- 172.21.15.45:0/3777031111 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb220075780 con 0x7fb2200b0c90 2024-04-16T16:18:41.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.943+0000 7fb217e00640 1 -- 172.21.15.45:0/3777031111 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) ==== 100+0+11 (secure 0 0 0) 0x7fb21805a410 con 0x7fb2200b0c90 2024-04-16T16:18:41.942 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:18:41.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.945+0000 7fb226a00640 1 -- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fb20006c9b0 msgr2=0x7fb20006ee70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:18:41.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.945+0000 7fb226a00640 1 --2- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fb20006c9b0 0x7fb20006ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.945+0000 7fb226a00640 1 -- 172.21.15.45:0/3777031111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2200b0c90 msgr2=0x7fb2200b1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:41.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.945+0000 7fb226a00640 1 --2- 172.21.15.45:0/3777031111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2200b0c90 0x7fb2200b1110 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fb22015f2a0 tx=0x7fb2180115a0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.946+0000 7fb226a00640 1 -- 172.21.15.45:0/3777031111 shutdown_connections 2024-04-16T16:18:41.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.946+0000 7fb226a00640 1 --2- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7fb20006c9b0 0x7fb20006ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.946+0000 7fb226a00640 1 --2- 172.21.15.45:0/3777031111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2200b0c90 0x7fb2200b1110 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.946+0000 7fb226a00640 1 --2- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2201420d0 0x7fb220142550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.946+0000 7fb226a00640 1 --2- 172.21.15.45:0/3777031111 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb204006190 0x7fb220141960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:41.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.946+0000 7fb226a00640 1 -- 172.21.15.45:0/3777031111 >> 172.21.15.45:0/3777031111 conn(0x7fb2200b3970 msgr2=0x7fb220159500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:41.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.946+0000 7fb226a00640 1 -- 172.21.15.45:0/3777031111 shutdown_connections 2024-04-16T16:18:41.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:41.946+0000 7fb226a00640 1 -- 172.21.15.45:0/3777031111 wait complete. 2024-04-16T16:18:41.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:18:42.021 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:42.021+0000 7fc5514fa040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:18:42.071 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:42.071+0000 7fc5514fa040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:18:42.086 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:42.087+0000 7f3e69032040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:18:42.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.165+0000 7f2826c00640 1 Processor -- start 2024-04-16T16:18:42.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.165+0000 7f2826c00640 1 -- start start 2024-04-16T16:18:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.166+0000 7f2826c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2820162090 0x7f28200cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.166+0000 7f2826c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f28200ba140 0x7f28200cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.167+0000 7f2826c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f28200d5520 0x7f28201628a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.167+0000 7f2826c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f28200ba8b0 con 0x7f28200d5520 2024-04-16T16:18:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.167+0000 7f2826c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f28200d2600 con 0x7f28200ba140 2024-04-16T16:18:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.167+0000 7f2826c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f28200ba5d0 con 0x7f2820162090 2024-04-16T16:18:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.167+0000 7f2826200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f28200d5520 0x7f28201628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.167+0000 7f2826200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f28200d5520 0x7f28201628a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57396/0 (socket says 172.21.15.45:57396) 2024-04-16T16:18:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.167+0000 7f2826200640 1 -- 172.21.15.45:0/2976418125 learned_addr learned my addr 172.21.15.45:0/2976418125 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:42.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.167+0000 7f2826200640 1 -- 172.21.15.45:0/2976418125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2820162090 msgr2=0x7f28200cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:42.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.167+0000 7f2826200640 1 --2- 172.21.15.45:0/2976418125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2820162090 0x7f28200cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.167+0000 7f2826200640 1 -- 172.21.15.45:0/2976418125 >> v2:172.21.15.103:3300/0 conn(0x7f28200ba140 msgr2=0x7f28200cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:42.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.167+0000 7f2826200640 1 --2- 172.21.15.45:0/2976418125 >> v2:172.21.15.103:3300/0 conn(0x7f28200ba140 0x7f28200cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.167+0000 7f2826200640 1 -- 172.21.15.45:0/2976418125 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28200d28a0 con 0x7f28200d5520 2024-04-16T16:18:42.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.168+0000 7f2826200640 1 --2- 172.21.15.45:0/2976418125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28200d5520 0x7f28201628a0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f282008cea0 tx=0x7f281802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac1218a57fda8e08 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:42.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.169+0000 7f281fe00640 1 -- 172.21.15.45:0/2976418125 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281802ca80 con 0x7f28200d5520 2024-04-16T16:18:42.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.169+0000 7f281fe00640 1 -- 172.21.15.45:0/2976418125 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f281802cc20 con 0x7f28200d5520 2024-04-16T16:18:42.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.169+0000 7f281fe00640 1 -- 172.21.15.45:0/2976418125 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28180343f0 con 0x7f28200d5520 2024-04-16T16:18:42.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.169+0000 7f2826c00640 1 -- 172.21.15.45:0/2976418125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28200d5520 msgr2=0x7f28201628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:42.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.169+0000 7f2826c00640 1 --2- 172.21.15.45:0/2976418125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28200d5520 0x7f28201628a0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f282008cea0 tx=0x7f281802bee0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.169+0000 7f2826c00640 1 -- 172.21.15.45:0/2976418125 shutdown_connections 2024-04-16T16:18:42.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.169+0000 7f2826c00640 1 --2- 172.21.15.45:0/2976418125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28200d5520 0x7f28201628a0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.169+0000 7f2826c00640 1 --2- 172.21.15.45:0/2976418125 >> v2:172.21.15.103:3300/0 conn(0x7f28200ba140 0x7f28200cfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.169+0000 7f2826c00640 1 --2- 172.21.15.45:0/2976418125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2820162090 0x7f28200cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.169+0000 7f2826c00640 1 -- 172.21.15.45:0/2976418125 >> 172.21.15.45:0/2976418125 conn(0x7f28200d1940 msgr2=0x7f28200d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:42.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.170+0000 7f2826c00640 1 -- 172.21.15.45:0/2976418125 shutdown_connections 2024-04-16T16:18:42.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.170+0000 7f2826c00640 1 -- 172.21.15.45:0/2976418125 wait complete. 2024-04-16T16:18:42.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.171+0000 7f2826c00640 1 Processor -- start 2024-04-16T16:18:42.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.171+0000 7f2826c00640 1 -- start start 2024-04-16T16:18:42.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.172+0000 7f2826c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28200d5520 0x7f28200c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:42.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.172+0000 7f2826c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2820162090 0x7f28200cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:42.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.172+0000 7f2826c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28200b6000 0x7f28200ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:42.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.172+0000 7f2826c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f28201625f0 con 0x7f28200b6000 2024-04-16T16:18:42.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.172+0000 7f2826c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2820162470 con 0x7f2820162090 2024-04-16T16:18:42.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.172+0000 7f2826c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f28200b36c0 con 0x7f28200d5520 2024-04-16T16:18:42.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.172+0000 7f2826200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28200b6000 0x7f28200ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:42.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.173+0000 7f2826200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28200b6000 0x7f28200ce3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57402/0 (socket says 172.21.15.45:57402) 2024-04-16T16:18:42.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.173+0000 7f2826200640 1 -- 172.21.15.45:0/82730590 learned_addr learned my addr 172.21.15.45:0/82730590 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:42.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.173+0000 7f2824e00640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2820162090 0x7f28200cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:42.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.173+0000 7f2825800640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28200d5520 0x7f28200c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:42.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.173+0000 7f2826200640 1 -- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28200d5520 msgr2=0x7f28200c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:42.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.173+0000 7f2826200640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28200d5520 0x7f28200c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.173+0000 7f2826200640 1 -- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2820162090 msgr2=0x7f28200cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:42.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.173+0000 7f2826200640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2820162090 0x7f28200cad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.173+0000 7f2826200640 1 -- 172.21.15.45:0/82730590 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28200b67c0 con 0x7f28200b6000 2024-04-16T16:18:42.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.173+0000 7f2825800640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28200d5520 0x7f28200c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:42.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.173+0000 7f2824e00640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2820162090 0x7f28200cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:42.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.173+0000 7f2826200640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28200b6000 0x7f28200ce3e0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f2818036040 tx=0x7f2818004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:42.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.174+0000 7f281d600640 1 -- 172.21.15.45:0/82730590 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2818043020 con 0x7f28200b6000 2024-04-16T16:18:42.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.174+0000 7f281d600640 1 -- 172.21.15.45:0/82730590 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f281803e450 con 0x7f28200b6000 2024-04-16T16:18:42.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.174+0000 7f281d600640 1 -- 172.21.15.45:0/82730590 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281803e820 con 0x7f28200b6000 2024-04-16T16:18:42.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.174+0000 7f2826c00640 1 -- 172.21.15.45:0/82730590 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28200b6aa0 con 0x7f28200b6000 2024-04-16T16:18:42.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.174+0000 7f2826c00640 1 -- 172.21.15.45:0/82730590 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28200b4570 con 0x7f28200b6000 2024-04-16T16:18:42.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.175+0000 7f2826c00640 1 -- 172.21.15.45:0/82730590 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2820075780 con 0x7f28200b6000 2024-04-16T16:18:42.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.176+0000 7f281d600640 1 -- 172.21.15.45:0/82730590 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 29) ==== 91086+0+0 (secure 0 0 0) 0x7f2818046020 con 0x7f28200b6000 2024-04-16T16:18:42.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.177+0000 7f281d600640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f280006c980 0x7f280006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:42.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.177+0000 7f281d600640 1 -- 172.21.15.45:0/82730590 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6053+0+0 (secure 0 0 0) 0x7f28180ae7c0 con 0x7f28200b6000 2024-04-16T16:18:42.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.177+0000 7f2825800640 1 -- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f280006c980 msgr2=0x7f280006ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1293636998 2024-04-16T16:18:42.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.177+0000 7f2825800640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f280006c980 0x7f280006ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:18:42.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.182+0000 7f281d600640 1 -- 172.21.15.45:0/82730590 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f281804d030 con 0x7f28200b6000 2024-04-16T16:18:42.196 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:42.197+0000 7fc5514fa040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:18:42.204 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:42.206+0000 7f3e69032040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:18:42.258 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:42.259+0000 7f3e69032040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:18:42.302 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:42.302+0000 7fc5514fa040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:18:42.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.378+0000 7f2825800640 1 -- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f280006c980 msgr2=0x7f280006ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1293636998 2024-04-16T16:18:42.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.378+0000 7f2825800640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f280006c980 0x7f280006ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:18:42.391 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:42.392+0000 7f3e69032040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:18:42.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.394+0000 7f2826c00640 1 -- 172.21.15.45:0/82730590 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f28200b4740 con 0x7f28200b6000 2024-04-16T16:18:42.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.395+0000 7f281d600640 1 -- 172.21.15.45:0/82730590 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v86) ==== 80+0+476 (secure 0 0 0) 0x7f281807c9a0 con 0x7f28200b6000 2024-04-16T16:18:42.394 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3489855481 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:42.394 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/674186192 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:42.394 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/3728775368 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:42.394 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/3728775368 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:42.394 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/4160076462 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:42.394 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1325022358 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:42.395 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/981309315 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:42.395 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3673976854 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:42.395 INFO:teuthology.orchestra.run.smithi045.stderr:listed 8 entries 2024-04-16T16:18:42.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.398+0000 7f2826c00640 1 -- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f280006c980 msgr2=0x7f280006ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:18:42.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.398+0000 7f2826c00640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f280006c980 0x7f280006ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.398+0000 7f2826c00640 1 -- 172.21.15.45:0/82730590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28200b6000 msgr2=0x7f28200ce3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:42.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.398+0000 7f2826c00640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28200b6000 0x7f28200ce3e0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f2818036040 tx=0x7f2818004a40 comp rx=0 tx=0).stop 2024-04-16T16:18:42.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.398+0000 7f2826c00640 1 -- 172.21.15.45:0/82730590 shutdown_connections 2024-04-16T16:18:42.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.398+0000 7f2826c00640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f280006c980 0x7f280006ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.398+0000 7f2826c00640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28200b6000 0x7f28200ce3e0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.398+0000 7f2826c00640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2820162090 0x7f28200cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.398+0000 7f2826c00640 1 --2- 172.21.15.45:0/82730590 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28200d5520 0x7f28200c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.398+0000 7f2826c00640 1 -- 172.21.15.45:0/82730590 >> 172.21.15.45:0/82730590 conn(0x7f28200d1940 msgr2=0x7f28200b8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:42.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.398+0000 7f2826c00640 1 -- 172.21.15.45:0/82730590 shutdown_connections 2024-04-16T16:18:42.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.398+0000 7f2826c00640 1 -- 172.21.15.45:0/82730590 wait complete. 2024-04-16T16:18:42.416 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-16T16:18:42.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:42.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-16T16:18:42.475 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:42.475+0000 7fc5514fa040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:18:42.505 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:18:42.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:42.505 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-16T16:18:42.512 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:42.512+0000 7f3e69032040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:18:42.525 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:42.526+0000 7fc5514fa040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:18:42.554 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:18:42.554 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:18:42.554 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:42.608 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:18:42.608 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:42.655 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:18:42.668 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:42.668+0000 7fc5514fa040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:18:42.697 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:42.698+0000 7f3e69032040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:18:42.753 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:42.754+0000 7f3e69032040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:18:42.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.861+0000 7f8269200640 1 Processor -- start 2024-04-16T16:18:42.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.862+0000 7f8269200640 1 -- start start 2024-04-16T16:18:42.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.863+0000 7f8269200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f826415cd40 0x7f82640d0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:42.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.863+0000 7f8269200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f82640cd5a0 0x7f82640cd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.863+0000 7f8269200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f826415d1e0 con 0x7f82640d0e70 2024-04-16T16:18:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.863+0000 7f8269200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f826415d4c0 con 0x7f826415cd40 2024-04-16T16:18:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.863+0000 7f8269200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f826415d7a0 con 0x7f82640cd5a0 2024-04-16T16:18:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.863+0000 7f825be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f826415cd40 0x7f82640d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.864+0000 7f8261e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f82640d0e70 0x7f82640d1250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60726/0 (socket says 172.21.15.45:60726) 2024-04-16T16:18:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.864+0000 7f8261e00640 1 -- 172.21.15.45:0/3180779828 learned_addr learned my addr 172.21.15.45:0/3180779828 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.864+0000 7f8262800640 1 --2- 172.21.15.45:0/3180779828 >> v2:172.21.15.103:3300/0 conn(0x7f82640cd5a0 0x7f82640cd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:42.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.864+0000 7f825be00640 1 -- 172.21.15.45:0/3180779828 >> v2:172.21.15.103:3300/0 conn(0x7f82640cd5a0 msgr2=0x7f82640cd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:42.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.864+0000 7f825be00640 1 --2- 172.21.15.45:0/3180779828 >> v2:172.21.15.103:3300/0 conn(0x7f82640cd5a0 0x7f82640cd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.864+0000 7f825be00640 1 -- 172.21.15.45:0/3180779828 >> v1:172.21.15.45:6789/0 conn(0x7f82640d0e70 legacy=0x7f82640d1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:42.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.864+0000 7f825be00640 1 -- 172.21.15.45:0/3180779828 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826415da80 con 0x7f826415cd40 2024-04-16T16:18:42.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.864+0000 7f8261400640 1 -- 172.21.15.45:0/3180779828 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3757415254 0 0) 0x7f826415d1e0 con 0x7f82640d0e70 2024-04-16T16:18:42.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.865+0000 7f825be00640 1 --2- 172.21.15.45:0/3180779828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f826415cd40 0x7f82640d0460 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f82640c98c0 tx=0x7f824c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8a926c68fc9a43 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:42.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.866+0000 7f8261400640 1 -- 172.21.15.45:0/3180779828 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c035020 con 0x7f826415cd40 2024-04-16T16:18:42.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.866+0000 7f8261400640 1 -- 172.21.15.45:0/3180779828 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f824c031830 con 0x7f826415cd40 2024-04-16T16:18:42.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.866+0000 7f8261400640 1 -- 172.21.15.45:0/3180779828 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c031c00 con 0x7f826415cd40 2024-04-16T16:18:42.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.866+0000 7f8269200640 1 -- 172.21.15.45:0/3180779828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f826415cd40 msgr2=0x7f82640d0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:42.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.866+0000 7f8269200640 1 --2- 172.21.15.45:0/3180779828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f826415cd40 0x7f82640d0460 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f82640c98c0 tx=0x7f824c02b8d0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.867+0000 7f8269200640 1 -- 172.21.15.45:0/3180779828 shutdown_connections 2024-04-16T16:18:42.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.867+0000 7f8269200640 1 --2- 172.21.15.45:0/3180779828 >> v2:172.21.15.103:3300/0 conn(0x7f82640cd5a0 0x7f82640cd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.867+0000 7f8269200640 1 --2- 172.21.15.45:0/3180779828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f826415cd40 0x7f82640d0460 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.867+0000 7f8269200640 1 -- 172.21.15.45:0/3180779828 >> 172.21.15.45:0/3180779828 conn(0x7f82640d2890 msgr2=0x7f82640d2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:42.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.867+0000 7f8269200640 1 -- 172.21.15.45:0/3180779828 shutdown_connections 2024-04-16T16:18:42.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.867+0000 7f8269200640 1 -- 172.21.15.45:0/3180779828 wait complete. 2024-04-16T16:18:42.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.868+0000 7f8269200640 1 Processor -- start 2024-04-16T16:18:42.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.868+0000 7f8269200640 1 -- start start 2024-04-16T16:18:42.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.869+0000 7f8269200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82640cd5a0 0x7f826414dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:42.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.869+0000 7f8269200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82640d0e70 0x7f82640c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:42.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.869+0000 7f8269200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f826415cd40 0x7f82641f4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:42.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.869+0000 7f8269200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f82640d0b20 con 0x7f82640d0e70 2024-04-16T16:18:42.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.869+0000 7f8269200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f82640d09a0 con 0x7f82640cd5a0 2024-04-16T16:18:42.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.869+0000 7f8269200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f82640cdee0 con 0x7f826415cd40 2024-04-16T16:18:42.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.870+0000 7f825be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82640d0e70 0x7f82640c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:42.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.870+0000 7f825be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82640d0e70 0x7f82640c9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57426/0 (socket says 172.21.15.45:57426) 2024-04-16T16:18:42.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.870+0000 7f825be00640 1 -- 172.21.15.45:0/3651019648 learned_addr learned my addr 172.21.15.45:0/3651019648 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:42.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.870+0000 7f8262800640 1 --2- 172.21.15.45:0/3651019648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f826415cd40 0x7f82641f4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:42.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.870+0000 7f825be00640 1 -- 172.21.15.45:0/3651019648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f826415cd40 msgr2=0x7f82641f4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:42.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.870+0000 7f825be00640 1 --2- 172.21.15.45:0/3651019648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f826415cd40 0x7f82641f4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.870+0000 7f825be00640 1 -- 172.21.15.45:0/3651019648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82640cd5a0 msgr2=0x7f826414dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:42.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.870+0000 7f825be00640 1 --2- 172.21.15.45:0/3651019648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82640cd5a0 0x7f826414dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:42.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.870+0000 7f825be00640 1 -- 172.21.15.45:0/3651019648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82641f5520 con 0x7f82640d0e70 2024-04-16T16:18:42.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.870+0000 7f825be00640 1 --2- 172.21.15.45:0/3651019648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82640d0e70 0x7f82640c9960 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f824c034040 tx=0x7f824c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:42.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.871+0000 7f825a000640 1 -- 172.21.15.45:0/3651019648 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c035040 con 0x7f82640d0e70 2024-04-16T16:18:42.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.871+0000 7f825a000640 1 -- 172.21.15.45:0/3651019648 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f824c004230 con 0x7f82640d0e70 2024-04-16T16:18:42.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.871+0000 7f8269200640 1 -- 172.21.15.45:0/3651019648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82641f5800 con 0x7f82640d0e70 2024-04-16T16:18:42.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.871+0000 7f825a000640 1 -- 172.21.15.45:0/3651019648 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c03a700 con 0x7f82640d0e70 2024-04-16T16:18:42.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.871+0000 7f8269200640 1 -- 172.21.15.45:0/3651019648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f826414e190 con 0x7f82640d0e70 2024-04-16T16:18:42.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.872+0000 7f8269200640 1 -- 172.21.15.45:0/3651019648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8264075780 con 0x7f82640d0e70 2024-04-16T16:18:42.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.873+0000 7f825a000640 1 -- 172.21.15.45:0/3651019648 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 29) ==== 91086+0+0 (secure 0 0 0) 0x7f824c038020 con 0x7f82640d0e70 2024-04-16T16:18:42.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.874+0000 7f825a000640 1 --2- 172.21.15.45:0/3651019648 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f824006c980 0x7f824006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:42.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.874+0000 7f825a000640 1 -- 172.21.15.45:0/3651019648 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6053+0+0 (secure 0 0 0) 0x7f824c0adbb0 con 0x7f82640d0e70 2024-04-16T16:18:42.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.874+0000 7f8261e00640 1 -- 172.21.15.45:0/3651019648 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f824006c980 msgr2=0x7f824006ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1293636998 2024-04-16T16:18:42.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.874+0000 7f8261e00640 1 --2- 172.21.15.45:0/3651019648 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f824006c980 0x7f824006ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:18:42.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:42.878+0000 7f825a000640 1 -- 172.21.15.45:0/3651019648 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f824c07be20 con 0x7f82640d0e70 2024-04-16T16:18:42.902 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:42.903+0000 7f3e69032040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:18:43.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.075+0000 7f8261e00640 1 -- 172.21.15.45:0/3651019648 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f824006c980 msgr2=0x7f824006ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1293636998 2024-04-16T16:18:43.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.075+0000 7f8261e00640 1 --2- 172.21.15.45:0/3651019648 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f824006c980 0x7f824006ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:18:43.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.084+0000 7f8269200640 1 -- 172.21.15.45:0/3651019648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82640d3710 con 0x7f82640d0e70 2024-04-16T16:18:43.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.086+0000 7f825a000640 1 -- 172.21.15.45:0/3651019648 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+17167 (secure 0 0 0) 0x7f824c07b720 con 0x7f82640d0e70 2024-04-16T16:18:43.086 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:43.086 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":86,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:18:37.658709+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:18:13.633261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-16T16:18:14.757055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3489855481":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/674186192":"2024-04-17T16:18:37.658582+0000","172.21.15.103:6833/3728775368":"2024-04-17T16:18:37.658582+0000","172.21.15.103:6832/3728775368":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/4160076462":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/1325022358":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/981309315":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/3673976854":"2024-04-17T16:18:37.658582+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:18:43.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.089+0000 7f8269200640 1 -- 172.21.15.45:0/3651019648 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f824006c980 msgr2=0x7f824006ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:18:43.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.089+0000 7f8269200640 1 --2- 172.21.15.45:0/3651019648 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f824006c980 0x7f824006ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.089+0000 7f8269200640 1 -- 172.21.15.45:0/3651019648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82640d0e70 msgr2=0x7f82640c9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:43.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.089+0000 7f8269200640 1 --2- 172.21.15.45:0/3651019648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82640d0e70 0x7f82640c9960 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f824c034040 tx=0x7f824c0046c0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.089+0000 7f8269200640 1 -- 172.21.15.45:0/3651019648 shutdown_connections 2024-04-16T16:18:43.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.089+0000 7f8269200640 1 --2- 172.21.15.45:0/3651019648 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f824006c980 0x7f824006ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.089+0000 7f8269200640 1 --2- 172.21.15.45:0/3651019648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f826415cd40 0x7f82641f4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.089+0000 7f8269200640 1 --2- 172.21.15.45:0/3651019648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82640d0e70 0x7f82640c9960 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.089+0000 7f8269200640 1 --2- 172.21.15.45:0/3651019648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82640cd5a0 0x7f826414dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.089+0000 7f8269200640 1 -- 172.21.15.45:0/3651019648 >> 172.21.15.45:0/3651019648 conn(0x7f82640d2890 msgr2=0x7f826406cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:43.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.089+0000 7f8269200640 1 -- 172.21.15.45:0/3651019648 shutdown_connections 2024-04-16T16:18:43.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.089+0000 7f8269200640 1 -- 172.21.15.45:0/3651019648 wait complete. 2024-04-16T16:18:43.105 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:18:13.633261+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:18:14.757055+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:18:43.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:18:43.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.363+0000 7f1b29400640 1 Processor -- start 2024-04-16T16:18:43.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.363+0000 7f1b29400640 1 -- start start 2024-04-16T16:18:43.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.365+0000 7f1b29400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1b240b6270 0x7f1b240d2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:43.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.365+0000 7f1b29400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1b240b7920 0x7f1b240b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:43.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.365+0000 7f1b29400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b240d32c0 con 0x7f1b240ba2a0 2024-04-16T16:18:43.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.365+0000 7f1b29400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1b240d35a0 con 0x7f1b240b7920 2024-04-16T16:18:43.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.365+0000 7f1b29400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1b240d3880 con 0x7f1b240b6270 2024-04-16T16:18:43.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.365+0000 7f1b28a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1b240b7920 0x7f1b240b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:43.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.365+0000 7f1b28a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1b240b7920 0x7f1b240b7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51700/0 (socket says 172.21.15.45:51700) 2024-04-16T16:18:43.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.365+0000 7f1b28a00640 1 -- 172.21.15.45:0/3147428711 learned_addr learned my addr 172.21.15.45:0/3147428711 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:43.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.365+0000 7f1b23e00640 1 --2- 172.21.15.45:0/3147428711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1b240b6270 0x7f1b240d2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:43.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.366+0000 7f1b28a00640 1 -- 172.21.15.45:0/3147428711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1b240b6270 msgr2=0x7f1b240d2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:43.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.366+0000 7f1b28a00640 1 --2- 172.21.15.45:0/3147428711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1b240b6270 0x7f1b240d2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.366+0000 7f1b28a00640 1 -- 172.21.15.45:0/3147428711 >> v1:172.21.15.103:6789/0 conn(0x7f1b240ba2a0 legacy=0x7f1b240b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:43.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.366+0000 7f1b28a00640 1 -- 172.21.15.45:0/3147428711 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b240d3b60 con 0x7f1b240b7920 2024-04-16T16:18:43.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.367+0000 7f1b28a00640 1 --2- 172.21.15.45:0/3147428711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b240b7920 0x7f1b240b7db0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009e50 tx=0x7f1b1c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=266334d5dcb730e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:43.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.367+0000 7f1b22a00640 1 -- 172.21.15.45:0/3147428711 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c02cb00 con 0x7f1b240b7920 2024-04-16T16:18:43.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.367+0000 7f1b22a00640 1 -- 172.21.15.45:0/3147428711 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b1c02cca0 con 0x7f1b240b7920 2024-04-16T16:18:43.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.367+0000 7f1b22a00640 1 -- 172.21.15.45:0/3147428711 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c034610 con 0x7f1b240b7920 2024-04-16T16:18:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.368+0000 7f1b29400640 1 -- 172.21.15.45:0/3147428711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b240b7920 msgr2=0x7f1b240b7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.368+0000 7f1b29400640 1 --2- 172.21.15.45:0/3147428711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b240b7920 0x7f1b240b7db0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009e50 tx=0x7f1b1c02bf60 comp rx=0 tx=0).stop 2024-04-16T16:18:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.368+0000 7f1b29400640 1 -- 172.21.15.45:0/3147428711 shutdown_connections 2024-04-16T16:18:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.368+0000 7f1b29400640 1 --2- 172.21.15.45:0/3147428711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b240b7920 0x7f1b240b7db0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.368+0000 7f1b29400640 1 --2- 172.21.15.45:0/3147428711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1b240b6270 0x7f1b240d2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.368+0000 7f1b29400640 1 -- 172.21.15.45:0/3147428711 >> 172.21.15.45:0/3147428711 conn(0x7f1b2406d300 msgr2=0x7f1b240d1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.368+0000 7f1b29400640 1 -- 172.21.15.45:0/3147428711 shutdown_connections 2024-04-16T16:18:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.368+0000 7f1b29400640 1 -- 172.21.15.45:0/3147428711 wait complete. 2024-04-16T16:18:43.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.370+0000 7f1b29400640 1 Processor -- start 2024-04-16T16:18:43.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.371+0000 7f1b29400640 1 -- start start 2024-04-16T16:18:43.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.372+0000 7f1b29400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b240b6270 0x7f1b2415b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:43.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.372+0000 7f1b29400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1b240b7920 0x7f1b241f7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:43.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.372+0000 7f1b29400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b240ba2a0 0x7f1b241f8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:43.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.372+0000 7f1b29400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1b240d4360 con 0x7f1b240ba2a0 2024-04-16T16:18:43.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.372+0000 7f1b29400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1b240ba700 con 0x7f1b240b6270 2024-04-16T16:18:43.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.372+0000 7f1b29400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1b240ce600 con 0x7f1b240b7920 2024-04-16T16:18:43.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.372+0000 7f1b28a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b240ba2a0 0x7f1b241f8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:43.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.372+0000 7f1b28a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b240ba2a0 0x7f1b241f8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51712/0 (socket says 172.21.15.45:51712) 2024-04-16T16:18:43.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.372+0000 7f1b28a00640 1 -- 172.21.15.45:0/2563642917 learned_addr learned my addr 172.21.15.45:0/2563642917 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:43.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.372+0000 7f1b23400640 1 --2- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1b240b7920 0x7f1b241f7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.372+0000 7f1b23e00640 1 --2- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b240b6270 0x7f1b2415b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.373+0000 7f1b28a00640 1 -- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1b240b7920 msgr2=0x7f1b241f7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.373+0000 7f1b28a00640 1 --2- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1b240b7920 0x7f1b241f7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.373+0000 7f1b28a00640 1 -- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b240b6270 msgr2=0x7f1b2415b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.373+0000 7f1b28a00640 1 --2- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b240b6270 0x7f1b2415b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.373+0000 7f1b28a00640 1 -- 172.21.15.45:0/2563642917 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b241e8420 con 0x7f1b240ba2a0 2024-04-16T16:18:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.373+0000 7f1b28a00640 1 --2- 172.21.15.45:0/2563642917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b240ba2a0 0x7f1b241f8360 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c036040 tx=0x7f1b1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:43.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.373+0000 7f1b1be00640 1 -- 172.21.15.45:0/2563642917 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c043020 con 0x7f1b240ba2a0 2024-04-16T16:18:43.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.373+0000 7f1b1be00640 1 -- 172.21.15.45:0/2563642917 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b1c02ce50 con 0x7f1b240ba2a0 2024-04-16T16:18:43.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.373+0000 7f1b1be00640 1 -- 172.21.15.45:0/2563642917 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c009510 con 0x7f1b240ba2a0 2024-04-16T16:18:43.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.373+0000 7f1b29400640 1 -- 172.21.15.45:0/2563642917 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b241e8700 con 0x7f1b240ba2a0 2024-04-16T16:18:43.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.374+0000 7f1b29400640 1 -- 172.21.15.45:0/2563642917 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b2415b900 con 0x7f1b240ba2a0 2024-04-16T16:18:43.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.375+0000 7f1b29400640 1 -- 172.21.15.45:0/2563642917 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b24075780 con 0x7f1b240ba2a0 2024-04-16T16:18:43.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.376+0000 7f1b1be00640 1 -- 172.21.15.45:0/2563642917 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 29) ==== 91086+0+0 (secure 0 0 0) 0x7f1b1c046020 con 0x7f1b240ba2a0 2024-04-16T16:18:43.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.377+0000 7f1b1be00640 1 --2- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f1b0406c980 0x7f1b0406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:43.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.377+0000 7f1b1be00640 1 -- 172.21.15.45:0/2563642917 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6053+0+0 (secure 0 0 0) 0x7f1b1c0ae3e0 con 0x7f1b240ba2a0 2024-04-16T16:18:43.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.377+0000 7f1b23e00640 1 -- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f1b0406c980 msgr2=0x7f1b0406ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1293636998 2024-04-16T16:18:43.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.377+0000 7f1b23e00640 1 --2- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f1b0406c980 0x7f1b0406ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:18:43.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.381+0000 7f1b1be00640 1 -- 172.21.15.45:0/2563642917 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1b1c04d030 con 0x7f1b240ba2a0 2024-04-16T16:18:43.489 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:43.489+0000 7fc5514fa040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:18:43.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.578+0000 7f1b23e00640 1 -- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f1b0406c980 msgr2=0x7f1b0406ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1293636998 2024-04-16T16:18:43.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.578+0000 7f1b23e00640 1 --2- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f1b0406c980 0x7f1b0406ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:18:43.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.588+0000 7f1b29400640 1 -- 172.21.15.45:0/2563642917 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1b240b74c0 con 0x7f1b240ba2a0 2024-04-16T16:18:43.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.589+0000 7f1b1be00640 1 -- 172.21.15.45:0/2563642917 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) ==== 93+0+10 (secure 0 0 0) 0x7f1b1c07c650 con 0x7f1b240ba2a0 2024-04-16T16:18:43.588 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:18:43.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.591+0000 7f1b29400640 1 -- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f1b0406c980 msgr2=0x7f1b0406ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:18:43.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.591+0000 7f1b29400640 1 --2- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f1b0406c980 0x7f1b0406ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.591+0000 7f1b29400640 1 -- 172.21.15.45:0/2563642917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b240ba2a0 msgr2=0x7f1b241f8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:43.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.591+0000 7f1b29400640 1 --2- 172.21.15.45:0/2563642917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b240ba2a0 0x7f1b241f8360 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c036040 tx=0x7f1b1c004a40 comp rx=0 tx=0).stop 2024-04-16T16:18:43.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.592+0000 7f1b29400640 1 -- 172.21.15.45:0/2563642917 shutdown_connections 2024-04-16T16:18:43.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.592+0000 7f1b29400640 1 --2- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f1b0406c980 0x7f1b0406ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.592+0000 7f1b29400640 1 --2- 172.21.15.45:0/2563642917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b240ba2a0 0x7f1b241f8360 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.592+0000 7f1b29400640 1 --2- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1b240b7920 0x7f1b241f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.592+0000 7f1b29400640 1 --2- 172.21.15.45:0/2563642917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b240b6270 0x7f1b2415b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.592+0000 7f1b29400640 1 -- 172.21.15.45:0/2563642917 >> 172.21.15.45:0/2563642917 conn(0x7f1b2406d300 msgr2=0x7f1b240b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:43.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.592+0000 7f1b29400640 1 -- 172.21.15.45:0/2563642917 shutdown_connections 2024-04-16T16:18:43.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.592+0000 7f1b29400640 1 -- 172.21.15.45:0/2563642917 wait complete. 2024-04-16T16:18:43.610 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:18:43.766 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:43.767+0000 7f3e69032040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:18:43.790 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:43.791+0000 7fc5514fa040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:18:43.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.806+0000 7f23ef000640 1 Processor -- start 2024-04-16T16:18:43.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.806+0000 7f23ef000640 1 -- start start 2024-04-16T16:18:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.808+0000 7f23ef000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23e80d18f0 0x7f23e80ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.808+0000 7f23ef000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f23e80b6310 0x7f23e80d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.808+0000 7f23ef000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f23e80d1cd0 0x7f23e80d33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.808+0000 7f23ef000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f23e80d29f0 con 0x7f23e80b6310 2024-04-16T16:18:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.808+0000 7f23ef000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f23e80b6810 con 0x7f23e80d1cd0 2024-04-16T16:18:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.808+0000 7f23ef000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f23e80b6af0 con 0x7f23e80d18f0 2024-04-16T16:18:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.808+0000 7f23ed200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f23e80b6310 0x7f23e80d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.808+0000 7f23ed200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f23e80b6310 0x7f23e80d2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51720/0 (socket says 172.21.15.45:51720) 2024-04-16T16:18:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.808+0000 7f23ed200640 1 -- 172.21.15.45:0/1477990298 learned_addr learned my addr 172.21.15.45:0/1477990298 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.809+0000 7f23ee600640 1 --2- 172.21.15.45:0/1477990298 >> v2:172.21.15.103:3300/0 conn(0x7f23e80d1cd0 0x7f23e80d33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.809+0000 7f23edc00640 1 --2- 172.21.15.45:0/1477990298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23e80d18f0 0x7f23e80ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.809+0000 7f23ed200640 1 -- 172.21.15.45:0/1477990298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23e80d18f0 msgr2=0x7f23e80ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.809+0000 7f23ed200640 1 --2- 172.21.15.45:0/1477990298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23e80d18f0 0x7f23e80ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.809+0000 7f23ed200640 1 -- 172.21.15.45:0/1477990298 >> v2:172.21.15.103:3300/0 conn(0x7f23e80d1cd0 msgr2=0x7f23e80d33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.809+0000 7f23ed200640 1 --2- 172.21.15.45:0/1477990298 >> v2:172.21.15.103:3300/0 conn(0x7f23e80d1cd0 0x7f23e80d33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.809+0000 7f23ed200640 1 -- 172.21.15.45:0/1477990298 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e80b6dd0 con 0x7f23e80b6310 2024-04-16T16:18:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.810+0000 7f23ed200640 1 --2- 172.21.15.45:0/1477990298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23e80b6310 0x7f23e80d2e80 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f23e80b4140 tx=0x7f23d402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37cf5e813f4e527b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:43.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.810+0000 7f23e7e00640 1 -- 172.21.15.45:0/1477990298 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d40317d0 con 0x7f23e80b6310 2024-04-16T16:18:43.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.810+0000 7f23e7e00640 1 -- 172.21.15.45:0/1477990298 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23d4037070 con 0x7f23e80b6310 2024-04-16T16:18:43.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.810+0000 7f23e7e00640 1 -- 172.21.15.45:0/1477990298 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d4031b90 con 0x7f23e80b6310 2024-04-16T16:18:43.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.811+0000 7f23ef000640 1 -- 172.21.15.45:0/1477990298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23e80b6310 msgr2=0x7f23e80d2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.811+0000 7f23ef000640 1 --2- 172.21.15.45:0/1477990298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23e80b6310 0x7f23e80d2e80 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f23e80b4140 tx=0x7f23d402b870 comp rx=0 tx=0).stop 2024-04-16T16:18:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.811+0000 7f23ef000640 1 -- 172.21.15.45:0/1477990298 shutdown_connections 2024-04-16T16:18:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.811+0000 7f23ef000640 1 --2- 172.21.15.45:0/1477990298 >> v2:172.21.15.103:3300/0 conn(0x7f23e80d1cd0 0x7f23e80d33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.811+0000 7f23ef000640 1 --2- 172.21.15.45:0/1477990298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23e80b6310 0x7f23e80d2e80 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.811+0000 7f23ef000640 1 --2- 172.21.15.45:0/1477990298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23e80d18f0 0x7f23e80ba220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.811+0000 7f23ef000640 1 -- 172.21.15.45:0/1477990298 >> 172.21.15.45:0/1477990298 conn(0x7f23e806cf90 msgr2=0x7f23e806d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.811+0000 7f23ef000640 1 -- 172.21.15.45:0/1477990298 shutdown_connections 2024-04-16T16:18:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.811+0000 7f23ef000640 1 -- 172.21.15.45:0/1477990298 wait complete. 2024-04-16T16:18:43.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.813+0000 7f23ef000640 1 Processor -- start 2024-04-16T16:18:43.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.813+0000 7f23ef000640 1 -- start start 2024-04-16T16:18:43.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.814+0000 7f23ef000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e80b6310 0x7f23e81558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:43.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.814+0000 7f23ef000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23e80d18f0 0x7f23e80cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:43.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.814+0000 7f23ef000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23e80d1cd0 0x7f23e80cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:43.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.814+0000 7f23ef000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f23e80d3900 con 0x7f23e80d1cd0 2024-04-16T16:18:43.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.814+0000 7f23ef000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f23e80ba760 con 0x7f23e80b6310 2024-04-16T16:18:43.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.814+0000 7f23ef000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f23e80d3a80 con 0x7f23e80d18f0 2024-04-16T16:18:43.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.815+0000 7f23ee600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23e80d1cd0 0x7f23e80cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:43.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.815+0000 7f23ee600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23e80d1cd0 0x7f23e80cc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51730/0 (socket says 172.21.15.45:51730) 2024-04-16T16:18:43.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.815+0000 7f23ee600640 1 -- 172.21.15.45:0/1691768812 learned_addr learned my addr 172.21.15.45:0/1691768812 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:43.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.815+0000 7f23edc00640 1 --2- 172.21.15.45:0/1691768812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e80b6310 0x7f23e81558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.815+0000 7f23ed200640 1 --2- 172.21.15.45:0/1691768812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23e80d18f0 0x7f23e80cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.815+0000 7f23ee600640 1 -- 172.21.15.45:0/1691768812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23e80d18f0 msgr2=0x7f23e80cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.815+0000 7f23ee600640 1 --2- 172.21.15.45:0/1691768812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23e80d18f0 0x7f23e80cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.815+0000 7f23ee600640 1 -- 172.21.15.45:0/1691768812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e80b6310 msgr2=0x7f23e81558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.815+0000 7f23ee600640 1 --2- 172.21.15.45:0/1691768812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e80b6310 0x7f23e81558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.815+0000 7f23ee600640 1 -- 172.21.15.45:0/1691768812 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e8156a10 con 0x7f23e80d1cd0 2024-04-16T16:18:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.815+0000 7f23ed200640 1 --2- 172.21.15.45:0/1691768812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23e80d18f0 0x7f23e80cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.815+0000 7f23ee600640 1 --2- 172.21.15.45:0/1691768812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23e80d1cd0 0x7f23e80cc370 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f23e0007e90 tx=0x7f23e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.816+0000 7f23e5600640 1 -- 172.21.15.45:0/1691768812 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e0014420 con 0x7f23e80d1cd0 2024-04-16T16:18:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.816+0000 7f23e5600640 1 -- 172.21.15.45:0/1691768812 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23e0019070 con 0x7f23e80d1cd0 2024-04-16T16:18:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.816+0000 7f23e5600640 1 -- 172.21.15.45:0/1691768812 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e0014800 con 0x7f23e80d1cd0 2024-04-16T16:18:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.816+0000 7f23ef000640 1 -- 172.21.15.45:0/1691768812 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23e8156cf0 con 0x7f23e80d1cd0 2024-04-16T16:18:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.816+0000 7f23ef000640 1 -- 172.21.15.45:0/1691768812 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23e815d320 con 0x7f23e80d1cd0 2024-04-16T16:18:43.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.817+0000 7f23ef000640 1 -- 172.21.15.45:0/1691768812 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23e8075780 con 0x7f23e80d1cd0 2024-04-16T16:18:43.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.818+0000 7f23e5600640 1 -- 172.21.15.45:0/1691768812 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 29) ==== 91086+0+0 (secure 0 0 0) 0x7f23e00149a0 con 0x7f23e80d1cd0 2024-04-16T16:18:43.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.819+0000 7f23e5600640 1 --2- 172.21.15.45:0/1691768812 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f23c806c9b0 0x7f23c806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:43.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.819+0000 7f23e5600640 1 -- 172.21.15.45:0/1691768812 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6053+0+0 (secure 0 0 0) 0x7f23e008c5f0 con 0x7f23e80d1cd0 2024-04-16T16:18:43.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.819+0000 7f23edc00640 1 -- 172.21.15.45:0/1691768812 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f23c806c9b0 msgr2=0x7f23c806ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1293636998 2024-04-16T16:18:43.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.819+0000 7f23edc00640 1 --2- 172.21.15.45:0/1691768812 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f23c806c9b0 0x7f23c806ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:18:43.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:43.823+0000 7f23e5600640 1 -- 172.21.15.45:0/1691768812 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f23e005a7d0 con 0x7f23e80d1cd0 2024-04-16T16:18:43.849 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:43.850+0000 7fc5514fa040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:18:43.986 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:43.986+0000 7fc5514fa040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:18:44.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.011+0000 7f23e5600640 1 -- 172.21.15.45:0/1691768812 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 30) ==== 90751+0+0 (secure 0 0 0) 0x7f23e0056740 con 0x7f23e80d1cd0 2024-04-16T16:18:44.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.011+0000 7f23e5600640 1 -- 172.21.15.45:0/1691768812 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f23c806c9b0 msgr2=0x7f23c806ee70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:18:44.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.012+0000 7f23e5600640 1 --2- 172.21.15.45:0/1691768812 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f23c806c9b0 0x7f23c806ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.033+0000 7f23ef000640 1 -- 172.21.15.45:0/1691768812 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f23e815d4f0 con 0x7f23e80d1cd0 2024-04-16T16:18:44.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.033+0000 7f23e5600640 1 -- 172.21.15.45:0/1691768812 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f23e00765b0 con 0x7f23e80d1cd0 2024-04-16T16:18:44.032 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:18:44.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.036+0000 7f23ef000640 1 -- 172.21.15.45:0/1691768812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23e80d1cd0 msgr2=0x7f23e80cc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:44.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.036+0000 7f23ef000640 1 --2- 172.21.15.45:0/1691768812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23e80d1cd0 0x7f23e80cc370 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f23e0007e90 tx=0x7f23e000e4b0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.036+0000 7f23ef000640 1 -- 172.21.15.45:0/1691768812 shutdown_connections 2024-04-16T16:18:44.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.036+0000 7f23ef000640 1 --2- 172.21.15.45:0/1691768812 >> [v2:172.21.15.103:6832/1293636998,v1:172.21.15.103:6833/1293636998] conn(0x7f23c806c9b0 0x7f23c806ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.036+0000 7f23ef000640 1 --2- 172.21.15.45:0/1691768812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23e80d1cd0 0x7f23e80cc370 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.036+0000 7f23ef000640 1 --2- 172.21.15.45:0/1691768812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23e80d18f0 0x7f23e80cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.036+0000 7f23ef000640 1 --2- 172.21.15.45:0/1691768812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e80b6310 0x7f23e81558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.036+0000 7f23ef000640 1 -- 172.21.15.45:0/1691768812 >> 172.21.15.45:0/1691768812 conn(0x7f23e806cf90 msgr2=0x7f23e80b9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:44.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.036+0000 7f23ef000640 1 -- 172.21.15.45:0/1691768812 shutdown_connections 2024-04-16T16:18:44.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.036+0000 7f23ef000640 1 -- 172.21.15.45:0/1691768812 wait complete. 2024-04-16T16:18:44.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:18:44.087 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:44.087+0000 7fc4cba00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:44.087 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:44.087+0000 7fc4cba00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:44.087 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:44.087+0000 7fc4cba00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:44.087 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:44.087+0000 7fc4cba00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:44.087 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:44.087+0000 7fc4cba00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:44.088 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:44.089+0000 7fc4cf600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:44.088 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:44.089+0000 7fc4cf600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:44.088 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:44.089+0000 7fc4cf600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:44.088 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:44.089+0000 7fc4cf600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:44.088 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:44.089+0000 7fc4cf600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:18:44.103 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:44.104+0000 7f3e69032040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:18:44.165 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:44.166+0000 7f3e69032040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:18:44.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.253+0000 7f73cf200640 1 Processor -- start 2024-04-16T16:18:44.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.253+0000 7f73cf200640 1 -- start start 2024-04-16T16:18:44.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.255+0000 7f73cf200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f73c815e980 0x7f73c80d4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:44.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.255+0000 7f73cf200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73c815ed60 con 0x7f73c80ba500 2024-04-16T16:18:44.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.255+0000 7f73cf200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73c80d3170 con 0x7f73c80cf4d0 2024-04-16T16:18:44.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.255+0000 7f73cf200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f73c80d2f50 con 0x7f73c815e980 2024-04-16T16:18:44.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.255+0000 7f73cde00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f73c815e980 0x7f73c80d4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:44.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.255+0000 7f73cde00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f73c815e980 0x7f73c80d4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51746/0 (socket says 172.21.15.45:51746) 2024-04-16T16:18:44.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.255+0000 7f73cd400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f73c80ba500 0x7f73c80d2840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54410/0 (socket says 172.21.15.45:54410) 2024-04-16T16:18:44.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.255+0000 7f73cde00640 1 -- 172.21.15.45:0/3119598097 learned_addr learned my addr 172.21.15.45:0/3119598097 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:44.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.256+0000 7f73cde00640 1 -- 172.21.15.45:0/3119598097 >> v1:172.21.15.103:6789/0 conn(0x7f73c80cf4d0 legacy=0x7f73c80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:44.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.256+0000 7f73cde00640 1 -- 172.21.15.45:0/3119598097 >> v1:172.21.15.45:6789/0 conn(0x7f73c80ba500 legacy=0x7f73c80d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:44.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.256+0000 7f73cde00640 1 -- 172.21.15.45:0/3119598097 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c80d3450 con 0x7f73c815e980 2024-04-16T16:18:44.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.257+0000 7f73cde00640 1 --2- 172.21.15.45:0/3119598097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73c815e980 0x7f73c80d4210 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f73bc009870 tx=0x7f73bc02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921e913261253186 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:44.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.257+0000 7f73cca00640 1 -- 172.21.15.45:0/3119598097 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73bc02f850 con 0x7f73c815e980 2024-04-16T16:18:44.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.257+0000 7f73cca00640 1 -- 172.21.15.45:0/3119598097 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73bc02f9f0 con 0x7f73c815e980 2024-04-16T16:18:44.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.257+0000 7f73cca00640 1 -- 172.21.15.45:0/3119598097 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73bc02fdc0 con 0x7f73c815e980 2024-04-16T16:18:44.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.258+0000 7f73cf200640 1 -- 172.21.15.45:0/3119598097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73c815e980 msgr2=0x7f73c80d4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:44.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.258+0000 7f73cf200640 1 --2- 172.21.15.45:0/3119598097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73c815e980 0x7f73c80d4210 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f73bc009870 tx=0x7f73bc02ba30 comp rx=0 tx=0).stop 2024-04-16T16:18:44.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.258+0000 7f73cf200640 1 -- 172.21.15.45:0/3119598097 shutdown_connections 2024-04-16T16:18:44.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.258+0000 7f73cf200640 1 --2- 172.21.15.45:0/3119598097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73c815e980 0x7f73c80d4210 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.258+0000 7f73cf200640 1 -- 172.21.15.45:0/3119598097 >> 172.21.15.45:0/3119598097 conn(0x7f73c806d2f0 msgr2=0x7f73c80d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:44.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.258+0000 7f73cf200640 1 -- 172.21.15.45:0/3119598097 shutdown_connections 2024-04-16T16:18:44.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.258+0000 7f73cf200640 1 -- 172.21.15.45:0/3119598097 wait complete. 2024-04-16T16:18:44.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.260+0000 7f73cf200640 1 Processor -- start 2024-04-16T16:18:44.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.260+0000 7f73cf200640 1 -- start start 2024-04-16T16:18:44.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.261+0000 7f73cf200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73c80ba500 0x7f73c815f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:44.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.261+0000 7f73cf200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73c80cf4d0 0x7f73c81538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:44.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.261+0000 7f73cf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73c815e980 0x7f73c8161340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:44.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.261+0000 7f73cf200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f73c80d3cf0 con 0x7f73c80ba500 2024-04-16T16:18:44.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.261+0000 7f73cf200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f73c80d3a50 con 0x7f73c80cf4d0 2024-04-16T16:18:44.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.261+0000 7f73cf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f73c80b7870 con 0x7f73c815e980 2024-04-16T16:18:44.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.261+0000 7f73cde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73c80ba500 0x7f73c815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:44.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.261+0000 7f73cde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73c80ba500 0x7f73c815f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51760/0 (socket says 172.21.15.45:51760) 2024-04-16T16:18:44.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.261+0000 7f73cde00640 1 -- 172.21.15.45:0/3632440409 learned_addr learned my addr 172.21.15.45:0/3632440409 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:44.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.262+0000 7f73cde00640 1 -- 172.21.15.45:0/3632440409 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73c815e980 msgr2=0x7f73c8161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:44.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.262+0000 7f73cde00640 1 --2- 172.21.15.45:0/3632440409 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73c815e980 0x7f73c8161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.262+0000 7f73cde00640 1 -- 172.21.15.45:0/3632440409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73c80cf4d0 msgr2=0x7f73c81538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:44.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.262+0000 7f73cde00640 1 --2- 172.21.15.45:0/3632440409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73c80cf4d0 0x7f73c81538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.262+0000 7f73cde00640 1 -- 172.21.15.45:0/3632440409 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c80caa30 con 0x7f73c80ba500 2024-04-16T16:18:44.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.262+0000 7f73cde00640 1 --2- 172.21.15.45:0/3632440409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73c80ba500 0x7f73c815f920 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f73bc009870 tx=0x7f73bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:44.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.262+0000 7f73c2000640 1 -- 172.21.15.45:0/3632440409 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73bc02f820 con 0x7f73c80ba500 2024-04-16T16:18:44.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.263+0000 7f73c2000640 1 -- 172.21.15.45:0/3632440409 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73bc046070 con 0x7f73c80ba500 2024-04-16T16:18:44.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.263+0000 7f73c2000640 1 -- 172.21.15.45:0/3632440409 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73bc041640 con 0x7f73c80ba500 2024-04-16T16:18:44.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.263+0000 7f73cf200640 1 -- 172.21.15.45:0/3632440409 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73c80cc500 con 0x7f73c80ba500 2024-04-16T16:18:44.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.263+0000 7f73cf200640 1 -- 172.21.15.45:0/3632440409 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73c80ccac0 con 0x7f73c80ba500 2024-04-16T16:18:44.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.264+0000 7f73cf200640 1 -- 172.21.15.45:0/3632440409 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73c80b5f30 con 0x7f73c80ba500 2024-04-16T16:18:44.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.265+0000 7f73c2000640 1 -- 172.21.15.45:0/3632440409 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) ==== 90751+0+0 (secure 0 0 0) 0x7f73bc040070 con 0x7f73c80ba500 2024-04-16T16:18:44.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.266+0000 7f73c2000640 1 -- 172.21.15.45:0/3632440409 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6354+0+0 (secure 0 0 0) 0x7f73bc0ae280 con 0x7f73c80ba500 2024-04-16T16:18:44.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.274+0000 7f73c2000640 1 -- 172.21.15.45:0/3632440409 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f73bc033030 con 0x7f73c80ba500 2024-04-16T16:18:44.309 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:18:44.310+0000 7f3e69032040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:18:44.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.483+0000 7f73cf200640 1 -- 172.21.15.45:0/3632440409 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f73c815e280 con 0x7f73c80ba500 2024-04-16T16:18:44.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.484+0000 7f73c2000640 1 -- 172.21.15.45:0/3632440409 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f73bc02f9c0 con 0x7f73c80ba500 2024-04-16T16:18:44.483 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:18:44.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.487+0000 7f73cf200640 1 -- 172.21.15.45:0/3632440409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73c80ba500 msgr2=0x7f73c815f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:44.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.487+0000 7f73cf200640 1 --2- 172.21.15.45:0/3632440409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73c80ba500 0x7f73c815f920 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f73bc009870 tx=0x7f73bc004730 comp rx=0 tx=0).stop 2024-04-16T16:18:44.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.487+0000 7f73cf200640 1 -- 172.21.15.45:0/3632440409 shutdown_connections 2024-04-16T16:18:44.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.487+0000 7f73cf200640 1 --2- 172.21.15.45:0/3632440409 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73c815e980 0x7f73c8161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.487+0000 7f73cf200640 1 --2- 172.21.15.45:0/3632440409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73c80cf4d0 0x7f73c81538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.487+0000 7f73cf200640 1 --2- 172.21.15.45:0/3632440409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73c80ba500 0x7f73c815f920 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.487+0000 7f73cf200640 1 -- 172.21.15.45:0/3632440409 >> 172.21.15.45:0/3632440409 conn(0x7f73c806d2f0 msgr2=0x7f73c80b50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:44.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.488+0000 7f73cf200640 1 -- 172.21.15.45:0/3632440409 shutdown_connections 2024-04-16T16:18:44.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.488+0000 7f73cf200640 1 -- 172.21.15.45:0/3632440409 wait complete. 2024-04-16T16:18:44.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:18:44.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.724+0000 7f283a400640 1 Processor -- start 2024-04-16T16:18:44.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.725+0000 7f283a400640 1 -- start start 2024-04-16T16:18:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.726+0000 7f283a400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f28340ba140 0x7f28340d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.726+0000 7f283a400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28340cfc20 0x7f28340d35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.726+0000 7f283a400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28340d2600 con 0x7f2834162090 2024-04-16T16:18:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.726+0000 7f283a400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f28340ba8b0 con 0x7f28340ba140 2024-04-16T16:18:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.726+0000 7f283a400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f28340ba5d0 con 0x7f28340cfc20 2024-04-16T16:18:44.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.727+0000 7f2833e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f28340ba140 0x7f28340d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:44.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.727+0000 7f2833e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f28340ba140 0x7f28340d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:48478/0 (socket says 172.21.15.45:48478) 2024-04-16T16:18:44.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.727+0000 7f2833e00640 1 -- 172.21.15.45:0/1867738679 learned_addr learned my addr 172.21.15.45:0/1867738679 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:44.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.727+0000 7f2839a00640 1 --2- 172.21.15.45:0/1867738679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28340cfc20 0x7f28340d35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:44.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.728+0000 7f2833e00640 1 -- 172.21.15.45:0/1867738679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28340cfc20 msgr2=0x7f28340d35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:44.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.728+0000 7f2833e00640 1 --2- 172.21.15.45:0/1867738679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28340cfc20 0x7f28340d35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.728+0000 7f2833e00640 1 -- 172.21.15.45:0/1867738679 >> v1:172.21.15.103:6789/0 conn(0x7f2834162090 legacy=0x7f28340cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:44.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.728+0000 7f2833e00640 1 -- 172.21.15.45:0/1867738679 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28340d28a0 con 0x7f28340ba140 2024-04-16T16:18:44.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.729+0000 7f2833e00640 1 --2- 172.21.15.45:0/1867738679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28340ba140 0x7f28340d30a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2820009870 tx=0x7f282002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dbdc6e655a42459 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:44.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.730+0000 7f2833400640 1 -- 172.21.15.45:0/1867738679 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2820034070 con 0x7f28340ba140 2024-04-16T16:18:44.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.730+0000 7f2833400640 1 -- 172.21.15.45:0/1867738679 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2820030840 con 0x7f28340ba140 2024-04-16T16:18:44.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.730+0000 7f2833400640 1 -- 172.21.15.45:0/1867738679 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2820030c10 con 0x7f28340ba140 2024-04-16T16:18:44.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.731+0000 7f283a400640 1 -- 172.21.15.45:0/1867738679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28340ba140 msgr2=0x7f28340d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:44.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.731+0000 7f283a400640 1 --2- 172.21.15.45:0/1867738679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28340ba140 0x7f28340d30a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2820009870 tx=0x7f282002b980 comp rx=0 tx=0).stop 2024-04-16T16:18:44.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.731+0000 7f283a400640 1 -- 172.21.15.45:0/1867738679 shutdown_connections 2024-04-16T16:18:44.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.731+0000 7f283a400640 1 --2- 172.21.15.45:0/1867738679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28340cfc20 0x7f28340d35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.731+0000 7f283a400640 1 --2- 172.21.15.45:0/1867738679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28340ba140 0x7f28340d30a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.731+0000 7f283a400640 1 -- 172.21.15.45:0/1867738679 >> 172.21.15.45:0/1867738679 conn(0x7f28340d1940 msgr2=0x7f28340d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:44.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.731+0000 7f283a400640 1 -- 172.21.15.45:0/1867738679 shutdown_connections 2024-04-16T16:18:44.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.731+0000 7f283a400640 1 -- 172.21.15.45:0/1867738679 wait complete. 2024-04-16T16:18:44.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.733+0000 7f283a400640 1 Processor -- start 2024-04-16T16:18:44.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.733+0000 7f283a400640 1 -- start start 2024-04-16T16:18:44.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.734+0000 7f283a400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28340cfc20 0x7f28340ce630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:44.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.734+0000 7f283a400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2834162090 0x7f28340cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:44.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.734+0000 7f283a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28341f84a0 0x7f2834154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:44.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.734+0000 7f283a400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f28341625f0 con 0x7f28341f84a0 2024-04-16T16:18:44.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.734+0000 7f283a400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2834162470 con 0x7f2834162090 2024-04-16T16:18:44.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.734+0000 7f283a400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f28340d5520 con 0x7f28340cfc20 2024-04-16T16:18:44.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.735+0000 7f2839a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28341f84a0 0x7f2834154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:44.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.735+0000 7f2839a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28341f84a0 0x7f2834154b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51766/0 (socket says 172.21.15.45:51766) 2024-04-16T16:18:44.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.735+0000 7f2839a00640 1 -- 172.21.15.45:0/1644687976 learned_addr learned my addr 172.21.15.45:0/1644687976 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:44.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.735+0000 7f2833e00640 1 --2- 172.21.15.45:0/1644687976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2834162090 0x7f28340cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:44.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.735+0000 7f2839000640 1 --2- 172.21.15.45:0/1644687976 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28340cfc20 0x7f28340ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:44.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.735+0000 7f2839a00640 1 -- 172.21.15.45:0/1644687976 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28340cfc20 msgr2=0x7f28340ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:44.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.735+0000 7f2839a00640 1 --2- 172.21.15.45:0/1644687976 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28340cfc20 0x7f28340ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.735+0000 7f2839a00640 1 -- 172.21.15.45:0/1644687976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2834162090 msgr2=0x7f28340cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:44.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.735+0000 7f2839a00640 1 --2- 172.21.15.45:0/1644687976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2834162090 0x7f28340cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.735+0000 7f2839a00640 1 -- 172.21.15.45:0/1644687976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28341f89a0 con 0x7f28341f84a0 2024-04-16T16:18:44.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.735+0000 7f2839a00640 1 --2- 172.21.15.45:0/1644687976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28341f84a0 0x7f2834154b00 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f282c00efc0 tx=0x7f282c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:44.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.736+0000 7f2830c00640 1 -- 172.21.15.45:0/1644687976 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282c018020 con 0x7f28341f84a0 2024-04-16T16:18:44.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.736+0000 7f2830c00640 1 -- 172.21.15.45:0/1644687976 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f282c014480 con 0x7f28341f84a0 2024-04-16T16:18:44.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.736+0000 7f2830c00640 1 -- 172.21.15.45:0/1644687976 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282c014870 con 0x7f28341f84a0 2024-04-16T16:18:44.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.736+0000 7f283a400640 1 -- 172.21.15.45:0/1644687976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28341f8c80 con 0x7f28341f84a0 2024-04-16T16:18:44.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.736+0000 7f283a400640 1 -- 172.21.15.45:0/1644687976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28341f9240 con 0x7f28341f84a0 2024-04-16T16:18:44.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.737+0000 7f283a400640 1 -- 172.21.15.45:0/1644687976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2834075780 con 0x7f28341f84a0 2024-04-16T16:18:44.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.738+0000 7f2830c00640 1 -- 172.21.15.45:0/1644687976 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) ==== 90751+0+0 (secure 0 0 0) 0x7f282c005d10 con 0x7f28341f84a0 2024-04-16T16:18:44.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.739+0000 7f2830c00640 1 -- 172.21.15.45:0/1644687976 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6354+0+0 (secure 0 0 0) 0x7f282c08c8e0 con 0x7f28341f84a0 2024-04-16T16:18:44.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.743+0000 7f2830c00640 1 -- 172.21.15.45:0/1644687976 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f282c022030 con 0x7f28341f84a0 2024-04-16T16:18:44.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.959+0000 7f283a400640 1 -- 172.21.15.45:0/1644687976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f283406dab0 con 0x7f28341f84a0 2024-04-16T16:18:44.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.960+0000 7f2830c00640 1 -- 172.21.15.45:0/1644687976 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v87) ==== 81+0+894 (secure 0 0 0) 0x7f282c010040 con 0x7f28341f84a0 2024-04-16T16:18:44.959 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/1293636998 2024-04-17T16:18:43.992878+0000 2024-04-16T16:18:44.960 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2879899995 2024-04-17T16:18:43.992878+0000 2024-04-16T16:18:44.960 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/766691889 2024-04-17T16:18:43.992878+0000 2024-04-16T16:18:44.960 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1187061250 2024-04-17T16:18:43.992878+0000 2024-04-16T16:18:44.960 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/1293636998 2024-04-17T16:18:43.992878+0000 2024-04-16T16:18:44.960 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/4007272017 2024-04-17T16:18:43.992878+0000 2024-04-16T16:18:44.960 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3489855481 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:44.960 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/674186192 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:44.960 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/3728775368 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:44.960 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/3728775368 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:44.960 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/4160076462 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:44.960 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1325022358 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:44.960 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/981309315 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:44.960 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1207000852 2024-04-17T16:18:43.992878+0000 2024-04-16T16:18:44.960 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3673976854 2024-04-17T16:18:37.658582+0000 2024-04-16T16:18:44.961 INFO:teuthology.orchestra.run.smithi045.stderr:listed 15 entries 2024-04-16T16:18:44.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.963+0000 7f283a400640 1 -- 172.21.15.45:0/1644687976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28341f84a0 msgr2=0x7f2834154b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:44.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.963+0000 7f283a400640 1 --2- 172.21.15.45:0/1644687976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28341f84a0 0x7f2834154b00 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f282c00efc0 tx=0x7f282c00c490 comp rx=0 tx=0).stop 2024-04-16T16:18:44.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.963+0000 7f283a400640 1 -- 172.21.15.45:0/1644687976 shutdown_connections 2024-04-16T16:18:44.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.963+0000 7f283a400640 1 --2- 172.21.15.45:0/1644687976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28341f84a0 0x7f2834154b00 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.963+0000 7f283a400640 1 --2- 172.21.15.45:0/1644687976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2834162090 0x7f28340cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.963+0000 7f283a400640 1 --2- 172.21.15.45:0/1644687976 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28340cfc20 0x7f28340ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:44.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.963+0000 7f283a400640 1 -- 172.21.15.45:0/1644687976 >> 172.21.15.45:0/1644687976 conn(0x7f28340d1940 msgr2=0x7f28340b5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:44.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.963+0000 7f283a400640 1 -- 172.21.15.45:0/1644687976 shutdown_connections 2024-04-16T16:18:44.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:44.963+0000 7f283a400640 1 -- 172.21.15.45:0/1644687976 wait complete. 2024-04-16T16:18:44.980 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-16T16:18:44.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:44.980 DEBUG:teuthology.orchestra.run.smithi103:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-04-16T16:18:45.085 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:18:45.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:18:45.086 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-16T16:18:45.130 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:18:45.130 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:18:45.130 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:45.190 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:18:45.212 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:18:45.212 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:18:45.212 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:45.266 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:18:45.267 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:18:45.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.496+0000 7f0055600640 1 Processor -- start 2024-04-16T16:18:45.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.496+0000 7f0055600640 1 -- start start 2024-04-16T16:18:45.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.497+0000 7f0055600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f005015ec60 0x7f00500d2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:45.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.497+0000 7f0055600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00500d34c0 con 0x7f00500d4560 2024-04-16T16:18:45.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.497+0000 7f0055600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00500d37a0 con 0x7f00500d1ac0 2024-04-16T16:18:45.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.497+0000 7f0055600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f00500d3a80 con 0x7f005015ec60 2024-04-16T16:18:45.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.498+0000 7f004e000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f00500d4560 0x7f00500d13b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54418/0 (socket says 172.21.15.45:54418) 2024-04-16T16:18:45.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.498+0000 7f004e000640 1 -- 172.21.15.45:0/3972424932 learned_addr learned my addr 172.21.15.45:0/3972424932 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:45.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.498+0000 7f004ea00640 1 --2- 172.21.15.45:0/3972424932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f005015ec60 0x7f00500d2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:45.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.498+0000 7f004d600640 1 -- 172.21.15.45:0/3972424932 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3261795467 0 0) 0x7f00500d34c0 con 0x7f00500d4560 2024-04-16T16:18:45.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.498+0000 7f004d600640 1 -- 172.21.15.45:0/3972424932 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0034003520 con 0x7f00500d4560 2024-04-16T16:18:45.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.499+0000 7f004d600640 1 -- 172.21.15.45:0/3972424932 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2353017620 0 0) 0x7f0034003520 con 0x7f00500d4560 2024-04-16T16:18:45.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.499+0000 7f004d600640 1 -- 172.21.15.45:0/3972424932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f005015ec60 msgr2=0x7f00500d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:45.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.499+0000 7f004d600640 1 --2- 172.21.15.45:0/3972424932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f005015ec60 0x7f00500d2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:45.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.499+0000 7f004d600640 1 -- 172.21.15.45:0/3972424932 >> v1:172.21.15.103:6789/0 conn(0x7f00500d1ac0 legacy=0x7f00500b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:45.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.499+0000 7f004d600640 1 -- 172.21.15.45:0/3972424932 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00500d3d60 con 0x7f00500d4560 2024-04-16T16:18:45.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.499+0000 7f004d600640 1 -- 172.21.15.45:0/3972424932 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f003c001670 con 0x7f00500d4560 2024-04-16T16:18:45.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.499+0000 7f004d600640 1 -- 172.21.15.45:0/3972424932 >> v1:172.21.15.45:6789/0 conn(0x7f00500d4560 legacy=0x7f00500d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:45.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.499+0000 7f004d600640 1 --2- 172.21.15.45:0/3972424932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f005015ec60 0x7f00340039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:45.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.499+0000 7f004d600640 1 --2- 172.21.15.45:0/3972424932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0034003fb0 0x7f0034004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:45.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.499+0000 7f004d600640 1 --2- 172.21.15.45:0/3972424932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0034004d40 0x7f00340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:45.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.499+0000 7f004d600640 1 -- 172.21.15.45:0/3972424932 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f00500d3a80 con 0x7f0034004d40 2024-04-16T16:18:45.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.499+0000 7f004d600640 1 -- 172.21.15.45:0/3972424932 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f003400bb00 con 0x7f0034003fb0 2024-04-16T16:18:45.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.499+0000 7f004d600640 1 -- 172.21.15.45:0/3972424932 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f003400bd00 con 0x7f005015ec60 2024-04-16T16:18:45.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.499+0000 7f004f400640 1 --2- 172.21.15.45:0/3972424932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0034004d40 0x7f00340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:45.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.499+0000 7f004e000640 1 --2- 172.21.15.45:0/3972424932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0034003fb0 0x7f0034004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:45.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.500+0000 7f004f400640 1 -- 172.21.15.45:0/3972424932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f005015ec60 msgr2=0x7f00340039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:45.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.500+0000 7f004f400640 1 --2- 172.21.15.45:0/3972424932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f005015ec60 0x7f00340039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:45.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.500+0000 7f004f400640 1 -- 172.21.15.45:0/3972424932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0034003fb0 msgr2=0x7f0034004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:45.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.500+0000 7f004f400640 1 --2- 172.21.15.45:0/3972424932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0034003fb0 0x7f0034004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:45.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.500+0000 7f004f400640 1 -- 172.21.15.45:0/3972424932 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0034003520 con 0x7f0034004d40 2024-04-16T16:18:45.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.501+0000 7f004f400640 1 --2- 172.21.15.45:0/3972424932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0034004d40 0x7f00340051c0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f00500cefe0 tx=0x7f0034005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc86a1d6a3d8e4b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:45.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.501+0000 7f004d600640 1 -- 172.21.15.45:0/3972424932 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0048001a70 con 0x7f0034004d40 2024-04-16T16:18:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.501+0000 7f004d600640 1 -- 172.21.15.45:0/3972424932 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0048003b70 con 0x7f0034004d40 2024-04-16T16:18:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.501+0000 7f004d600640 1 -- 172.21.15.45:0/3972424932 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00480050d0 con 0x7f0034004d40 2024-04-16T16:18:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.501+0000 7f0055600640 1 -- 172.21.15.45:0/3972424932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0034004d40 msgr2=0x7f00340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.501+0000 7f0055600640 1 --2- 172.21.15.45:0/3972424932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0034004d40 0x7f00340051c0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f00500cefe0 tx=0x7f0034005830 comp rx=0 tx=0).stop 2024-04-16T16:18:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.501+0000 7f004f400640 1 -- 172.21.15.45:0/3972424932 reap_dead start 2024-04-16T16:18:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.502+0000 7f0055600640 1 -- 172.21.15.45:0/3972424932 shutdown_connections 2024-04-16T16:18:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.502+0000 7f0055600640 1 -- 172.21.15.45:0/3972424932 >> 172.21.15.45:0/3972424932 conn(0x7f0050073690 msgr2=0x7f005006cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.502+0000 7f0055600640 1 -- 172.21.15.45:0/3972424932 shutdown_connections 2024-04-16T16:18:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.502+0000 7f0055600640 1 -- 172.21.15.45:0/3972424932 wait complete. 2024-04-16T16:18:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.503+0000 7f0055600640 1 Processor -- start 2024-04-16T16:18:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.504+0000 7f0055600640 1 -- start start 2024-04-16T16:18:45.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.504+0000 7f0055600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0034004d40 0x7f00501f43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:45.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.504+0000 7f0055600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f005015a7e0 0x7f005015ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:45.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.504+0000 7f0055600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f005014ac70 0x7f005014b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:45.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.504+0000 7f0055600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f00500d2a00 con 0x7f005015a7e0 2024-04-16T16:18:45.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.504+0000 7f0055600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f00500ba460 con 0x7f005014ac70 2024-04-16T16:18:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.504+0000 7f0055600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f00500d2b80 con 0x7f0034004d40 2024-04-16T16:18:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.505+0000 7f004e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f005015a7e0 0x7f005015ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.505+0000 7f004e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f005015a7e0 0x7f005015ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51778/0 (socket says 172.21.15.45:51778) 2024-04-16T16:18:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.505+0000 7f004e000640 1 -- 172.21.15.45:0/2685562444 learned_addr learned my addr 172.21.15.45:0/2685562444 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.505+0000 7f004ea00640 1 --2- 172.21.15.45:0/2685562444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0034004d40 0x7f00501f43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.505+0000 7f004f400640 1 --2- 172.21.15.45:0/2685562444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f005014ac70 0x7f005014b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.505+0000 7f004e000640 1 -- 172.21.15.45:0/2685562444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0034004d40 msgr2=0x7f00501f43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.505+0000 7f004e000640 1 --2- 172.21.15.45:0/2685562444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0034004d40 0x7f00501f43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.505+0000 7f004e000640 1 -- 172.21.15.45:0/2685562444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f005014ac70 msgr2=0x7f005014b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.505+0000 7f004e000640 1 --2- 172.21.15.45:0/2685562444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f005014ac70 0x7f005014b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.505+0000 7f004e000640 1 -- 172.21.15.45:0/2685562444 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005015b260 con 0x7f005015a7e0 2024-04-16T16:18:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.506+0000 7f004e000640 1 --2- 172.21.15.45:0/2685562444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f005015a7e0 0x7f005015ac20 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f003c00d7f0 tx=0x7f003c00dcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:45.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.506+0000 7f0046a00640 1 -- 172.21.15.45:0/2685562444 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c019070 con 0x7f005015a7e0 2024-04-16T16:18:45.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.506+0000 7f0046a00640 1 -- 172.21.15.45:0/2685562444 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f003c0074e0 con 0x7f005015a7e0 2024-04-16T16:18:45.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.506+0000 7f0046a00640 1 -- 172.21.15.45:0/2685562444 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c004a00 con 0x7f005015a7e0 2024-04-16T16:18:45.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.506+0000 7f0055600640 1 -- 172.21.15.45:0/2685562444 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f005015b540 con 0x7f005015a7e0 2024-04-16T16:18:45.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.506+0000 7f0055600640 1 -- 172.21.15.45:0/2685562444 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f005015bb00 con 0x7f005015a7e0 2024-04-16T16:18:45.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.507+0000 7f0055600640 1 -- 172.21.15.45:0/2685562444 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0050075780 con 0x7f005015a7e0 2024-04-16T16:18:45.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.508+0000 7f0046a00640 1 -- 172.21.15.45:0/2685562444 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 31) ==== 90878+0+0 (secure 0 0 0) 0x7f003c01c070 con 0x7f005015a7e0 2024-04-16T16:18:45.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.509+0000 7f0046a00640 1 --2- 172.21.15.45:0/2685562444 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f003006c740 0x7f003006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:45.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.509+0000 7f0046a00640 1 -- 172.21.15.45:0/2685562444 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6354+0+0 (secure 0 0 0) 0x7f003c08b5a0 con 0x7f005015a7e0 2024-04-16T16:18:45.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.510+0000 7f004ea00640 1 --2- 172.21.15.45:0/2685562444 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f003006c740 0x7f003006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:45.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.511+0000 7f004ea00640 1 --2- 172.21.15.45:0/2685562444 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f003006c740 0x7f003006ec00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0040004770 tx=0x7f0040008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:45.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.514+0000 7f0046a00640 1 -- 172.21.15.45:0/2685562444 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f003c059920 con 0x7f005015a7e0 2024-04-16T16:18:45.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.759+0000 7f0055600640 1 -- 172.21.15.45:0/2685562444 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00500ba960 con 0x7f005015a7e0 2024-04-16T16:18:45.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.760+0000 7f0046a00640 1 -- 172.21.15.45:0/2685562444 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5186 (secure 0 0 0) 0x7f003c059220 con 0x7f005015a7e0 2024-04-16T16:18:45.760 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:45.760 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2514711815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2514711815},{"type":"v1","addr":"172.21.15.103:6836","nonce":2514711815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2189404408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2189404408},{"type":"v1","addr":"172.21.15.103:6837","nonce":2189404408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7179,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/991116998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":991116998},{"type":"v1","addr":"172.21.15.45:6834","nonce":991116998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:18:15.762294+0000","modified":"2024-04-16T16:18:18.858040+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7188":{"gid":7188,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3813781309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3813781309},{"type":"v1","addr":"172.21.15.45:6835","nonce":3813781309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7188,"qdb_cluster":[7188]},"id":6}]} 2024-04-16T16:18:45.760 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 64 2024-04-16T16:18:45.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.763+0000 7f0055600640 1 -- 172.21.15.45:0/2685562444 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f003006c740 msgr2=0x7f003006ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:45.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.763+0000 7f0055600640 1 --2- 172.21.15.45:0/2685562444 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f003006c740 0x7f003006ec00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0040004770 tx=0x7f0040008040 comp rx=0 tx=0).stop 2024-04-16T16:18:45.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.763+0000 7f0055600640 1 -- 172.21.15.45:0/2685562444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f005015a7e0 msgr2=0x7f005015ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:45.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.763+0000 7f0055600640 1 --2- 172.21.15.45:0/2685562444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f005015a7e0 0x7f005015ac20 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f003c00d7f0 tx=0x7f003c00dcb0 comp rx=0 tx=0).stop 2024-04-16T16:18:45.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.763+0000 7f0055600640 1 -- 172.21.15.45:0/2685562444 shutdown_connections 2024-04-16T16:18:45.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.763+0000 7f0055600640 1 --2- 172.21.15.45:0/2685562444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f005014ac70 0x7f005014b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:45.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.763+0000 7f0055600640 1 --2- 172.21.15.45:0/2685562444 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f003006c740 0x7f003006ec00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:45.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.763+0000 7f0055600640 1 --2- 172.21.15.45:0/2685562444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f005015a7e0 0x7f005015ac20 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:45.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.763+0000 7f0055600640 1 --2- 172.21.15.45:0/2685562444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0034004d40 0x7f00501f43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:45.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.763+0000 7f0055600640 1 -- 172.21.15.45:0/2685562444 >> 172.21.15.45:0/2685562444 conn(0x7f0050073690 msgr2=0x7f00500ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:45.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.764+0000 7f0055600640 1 -- 172.21.15.45:0/2685562444 shutdown_connections 2024-04-16T16:18:45.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:45.764+0000 7f0055600640 1 -- 172.21.15.45:0/2685562444 wait complete. 2024-04-16T16:18:45.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:18:46.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.009+0000 7fc6f7a00640 1 Processor -- start 2024-04-16T16:18:46.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.009+0000 7fc6f7a00640 1 -- start start 2024-04-16T16:18:46.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.010+0000 7fc6f7a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6f015ecc0 0x7fc6f006fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.011+0000 7fc6f7a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc6f009af60 0x7fc6f006b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.011+0000 7fc6f7a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6f006f490 con 0x7fc6f006be70 2024-04-16T16:18:46.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.011+0000 7fc6f7a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc6f00b3f60 con 0x7fc6f009af60 2024-04-16T16:18:46.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.011+0000 7fc6f7a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc6f006f730 con 0x7fc6f015ecc0 2024-04-16T16:18:46.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.011+0000 7fc6f5a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc6f006be70 0x7fc6f00ce070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54424/0 (socket says 172.21.15.45:54424) 2024-04-16T16:18:46.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.011+0000 7fc6f5a00640 1 -- 172.21.15.45:0/728897670 learned_addr learned my addr 172.21.15.45:0/728897670 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:46.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.011+0000 7fc6f5000640 1 --2- 172.21.15.45:0/728897670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6f015ecc0 0x7fc6f006fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.011+0000 7fc6efe00640 1 --2- 172.21.15.45:0/728897670 >> v2:172.21.15.103:3300/0 conn(0x7fc6f009af60 0x7fc6f006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.012+0000 7fc6f5000640 1 -- 172.21.15.45:0/728897670 >> v2:172.21.15.103:3300/0 conn(0x7fc6f009af60 msgr2=0x7fc6f006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:46.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.012+0000 7fc6f5000640 1 --2- 172.21.15.45:0/728897670 >> v2:172.21.15.103:3300/0 conn(0x7fc6f009af60 0x7fc6f006b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.012+0000 7fc6f5000640 1 -- 172.21.15.45:0/728897670 >> v1:172.21.15.45:6789/0 conn(0x7fc6f006be70 legacy=0x7fc6f00ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.012+0000 7fc6f5000640 1 -- 172.21.15.45:0/728897670 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f00b3bf0 con 0x7fc6f015ecc0 2024-04-16T16:18:46.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.014+0000 7fc6f5000640 1 --2- 172.21.15.45:0/728897670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6f015ecc0 0x7fc6f006fce0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009d90 tx=0x7fc6e002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d53163f9e6733f70 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:46.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.017+0000 7fc6ef400640 1 -- 172.21.15.45:0/728897670 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0039070 con 0x7fc6f015ecc0 2024-04-16T16:18:46.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.017+0000 7fc6ef400640 1 -- 172.21.15.45:0/728897670 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6e002cc50 con 0x7fc6f015ecc0 2024-04-16T16:18:46.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.017+0000 7fc6ef400640 1 -- 172.21.15.45:0/728897670 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0034440 con 0x7fc6f015ecc0 2024-04-16T16:18:46.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.018+0000 7fc6f7a00640 1 -- 172.21.15.45:0/728897670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6f015ecc0 msgr2=0x7fc6f006fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.018+0000 7fc6f7a00640 1 --2- 172.21.15.45:0/728897670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6f015ecc0 0x7fc6f006fce0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009d90 tx=0x7fc6e002bea0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.018+0000 7fc6f7a00640 1 -- 172.21.15.45:0/728897670 shutdown_connections 2024-04-16T16:18:46.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.018+0000 7fc6f7a00640 1 --2- 172.21.15.45:0/728897670 >> v2:172.21.15.103:3300/0 conn(0x7fc6f009af60 0x7fc6f006b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.018+0000 7fc6f7a00640 1 --2- 172.21.15.45:0/728897670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6f015ecc0 0x7fc6f006fce0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.018+0000 7fc6f7a00640 1 -- 172.21.15.45:0/728897670 >> 172.21.15.45:0/728897670 conn(0x7fc6f009aa70 msgr2=0x7fc6f00b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:46.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.018+0000 7fc6f7a00640 1 -- 172.21.15.45:0/728897670 shutdown_connections 2024-04-16T16:18:46.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.018+0000 7fc6f7a00640 1 -- 172.21.15.45:0/728897670 wait complete. 2024-04-16T16:18:46.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.020+0000 7fc6f7a00640 1 Processor -- start 2024-04-16T16:18:46.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.021+0000 7fc6f7a00640 1 -- start start 2024-04-16T16:18:46.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.022+0000 7fc6f7a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6f006be70 0x7fc6f0161200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.022+0000 7fc6f7a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6f009af60 0x7fc6f0154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.022+0000 7fc6f7a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6f015ecc0 0x7fc6f00b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.022+0000 7fc6f7a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f00703a0 con 0x7fc6f009af60 2024-04-16T16:18:46.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.022+0000 7fc6f7a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f0070220 con 0x7fc6f006be70 2024-04-16T16:18:46.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.022+0000 7fc6f7a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc6f00d0590 con 0x7fc6f015ecc0 2024-04-16T16:18:46.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.022+0000 7fc6efe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6f009af60 0x7fc6f0154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.022+0000 7fc6efe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6f009af60 0x7fc6f0154cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51788/0 (socket says 172.21.15.45:51788) 2024-04-16T16:18:46.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.022+0000 7fc6efe00640 1 -- 172.21.15.45:0/1007813520 learned_addr learned my addr 172.21.15.45:0/1007813520 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:46.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.023+0000 7fc6f5000640 1 --2- 172.21.15.45:0/1007813520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6f006be70 0x7fc6f0161200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.023+0000 7fc6f5a00640 1 --2- 172.21.15.45:0/1007813520 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6f015ecc0 0x7fc6f00b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.023+0000 7fc6efe00640 1 -- 172.21.15.45:0/1007813520 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6f015ecc0 msgr2=0x7fc6f00b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.023+0000 7fc6efe00640 1 --2- 172.21.15.45:0/1007813520 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6f015ecc0 0x7fc6f00b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.023+0000 7fc6efe00640 1 -- 172.21.15.45:0/1007813520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6f006be70 msgr2=0x7fc6f0161200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.023+0000 7fc6efe00640 1 --2- 172.21.15.45:0/1007813520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6f006be70 0x7fc6f0161200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.023+0000 7fc6efe00640 1 -- 172.21.15.45:0/1007813520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f00b0950 con 0x7fc6f009af60 2024-04-16T16:18:46.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.023+0000 7fc6efe00640 1 --2- 172.21.15.45:0/1007813520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6f009af60 0x7fc6f0154cb0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fc6d800e9b0 tx=0x7fc6d800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:46.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.024+0000 7fc6ecc00640 1 -- 172.21.15.45:0/1007813520 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8017070 con 0x7fc6f009af60 2024-04-16T16:18:46.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.024+0000 7fc6ecc00640 1 -- 172.21.15.45:0/1007813520 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d800c9a0 con 0x7fc6f009af60 2024-04-16T16:18:46.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.024+0000 7fc6ecc00640 1 -- 172.21.15.45:0/1007813520 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d800cd90 con 0x7fc6f009af60 2024-04-16T16:18:46.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.024+0000 7fc6f7a00640 1 -- 172.21.15.45:0/1007813520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6f0161890 con 0x7fc6f009af60 2024-04-16T16:18:46.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.024+0000 7fc6f7a00640 1 -- 172.21.15.45:0/1007813520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6f0161e50 con 0x7fc6f009af60 2024-04-16T16:18:46.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.025+0000 7fc6f7a00640 1 -- 172.21.15.45:0/1007813520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6f0075780 con 0x7fc6f009af60 2024-04-16T16:18:46.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.026+0000 7fc6ecc00640 1 -- 172.21.15.45:0/1007813520 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 31) ==== 90878+0+0 (secure 0 0 0) 0x7fc6d801a070 con 0x7fc6f009af60 2024-04-16T16:18:46.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.027+0000 7fc6ecc00640 1 --2- 172.21.15.45:0/1007813520 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fc6cc06c740 0x7fc6cc06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.027+0000 7fc6ecc00640 1 -- 172.21.15.45:0/1007813520 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6354+0+0 (secure 0 0 0) 0x7fc6d808bb00 con 0x7fc6f009af60 2024-04-16T16:18:46.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.027+0000 7fc6f5000640 1 --2- 172.21.15.45:0/1007813520 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fc6cc06c740 0x7fc6cc06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.029+0000 7fc6f5000640 1 --2- 172.21.15.45:0/1007813520 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fc6cc06c740 0x7fc6cc06ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0036000 tx=0x7fc6e0030000 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:46.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.032+0000 7fc6ecc00640 1 -- 172.21.15.45:0/1007813520 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc6d8059f10 con 0x7fc6f009af60 2024-04-16T16:18:46.065 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:46.065+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:18:46.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.241+0000 7fc6f7a00640 1 -- 172.21.15.45:0/1007813520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6f00d1dc0 con 0x7fc6f009af60 2024-04-16T16:18:46.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.244+0000 7fc6ecc00640 1 -- 172.21.15.45:0/1007813520 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+17613 (secure 0 0 0) 0x7fc6d8059810 con 0x7fc6f009af60 2024-04-16T16:18:46.244 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:46.244 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:18:43.993003+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:18:13.633261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-16T16:18:14.757055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6833/1293636998":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/2879899995":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/766691889":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/1187061250":"2024-04-17T16:18:43.992878+0000","172.21.15.103:6832/1293636998":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/4007272017":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/3489855481":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/674186192":"2024-04-17T16:18:37.658582+0000","172.21.15.103:6833/3728775368":"2024-04-17T16:18:37.658582+0000","172.21.15.103:6832/3728775368":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/4160076462":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/1325022358":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/981309315":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/1207000852":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/3673976854":"2024-04-17T16:18:37.658582+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:18:46.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.247+0000 7fc6f7a00640 1 -- 172.21.15.45:0/1007813520 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fc6cc06c740 msgr2=0x7fc6cc06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.247+0000 7fc6f7a00640 1 --2- 172.21.15.45:0/1007813520 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fc6cc06c740 0x7fc6cc06ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0036000 tx=0x7fc6e0030000 comp rx=0 tx=0).stop 2024-04-16T16:18:46.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.247+0000 7fc6f7a00640 1 -- 172.21.15.45:0/1007813520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6f009af60 msgr2=0x7fc6f0154cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.247+0000 7fc6f7a00640 1 --2- 172.21.15.45:0/1007813520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6f009af60 0x7fc6f0154cb0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fc6d800e9b0 tx=0x7fc6d800ee70 comp rx=0 tx=0).stop 2024-04-16T16:18:46.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.247+0000 7fc6f7a00640 1 -- 172.21.15.45:0/1007813520 shutdown_connections 2024-04-16T16:18:46.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.247+0000 7fc6f7a00640 1 --2- 172.21.15.45:0/1007813520 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6f015ecc0 0x7fc6f00b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.247+0000 7fc6f7a00640 1 --2- 172.21.15.45:0/1007813520 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fc6cc06c740 0x7fc6cc06ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.247+0000 7fc6f7a00640 1 --2- 172.21.15.45:0/1007813520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6f009af60 0x7fc6f0154cb0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.247+0000 7fc6f7a00640 1 --2- 172.21.15.45:0/1007813520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6f006be70 0x7fc6f0161200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.247+0000 7fc6f7a00640 1 -- 172.21.15.45:0/1007813520 >> 172.21.15.45:0/1007813520 conn(0x7fc6f009aa70 msgr2=0x7fc6f00cd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:46.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.247+0000 7fc6f7a00640 1 -- 172.21.15.45:0/1007813520 shutdown_connections 2024-04-16T16:18:46.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.247+0000 7fc6f7a00640 1 -- 172.21.15.45:0/1007813520 wait complete. 2024-04-16T16:18:46.267 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:18:13.633261+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:18:14.757055+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:18:46.268 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:18:46.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.501+0000 7f40d2800640 1 Processor -- start 2024-04-16T16:18:46.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.501+0000 7f40d2800640 1 -- start start 2024-04-16T16:18:46.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.502+0000 7f40d2800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40cc0d4370 0x7f40cc0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.502+0000 7f40d2800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f40cc0cfa70 0x7f40cc162560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.502+0000 7f40d2800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40cc15ee50 con 0x7f40cc15ea70 2024-04-16T16:18:46.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.502+0000 7f40d2800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f40cc0d1a10 con 0x7f40cc0cfa70 2024-04-16T16:18:46.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.502+0000 7f40d2800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f40cc0b6480 con 0x7f40cc0d4370 2024-04-16T16:18:46.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.503+0000 7f40d1400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f40cc15ea70 0x7f40cc0d11a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54438/0 (socket says 172.21.15.45:54438) 2024-04-16T16:18:46.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.503+0000 7f40d1400640 1 -- 172.21.15.45:0/2271529610 learned_addr learned my addr 172.21.15.45:0/2271529610 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:46.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.503+0000 7f40cbe00640 1 -- 172.21.15.45:0/2271529610 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1903550422 0 0) 0x7f40cc15ee50 con 0x7f40cc15ea70 2024-04-16T16:18:46.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.503+0000 7f40cbe00640 1 -- 172.21.15.45:0/2271529610 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40b4003520 con 0x7f40cc15ea70 2024-04-16T16:18:46.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.503+0000 7f40d1e00640 1 --2- 172.21.15.45:0/2271529610 >> v2:172.21.15.103:3300/0 conn(0x7f40cc0cfa70 0x7f40cc162560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40cbe00640 1 -- 172.21.15.45:0/2271529610 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3212356762 0 0) 0x7f40b4003520 con 0x7f40cc15ea70 2024-04-16T16:18:46.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40d0a00640 1 --2- 172.21.15.45:0/2271529610 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40cc0d4370 0x7f40cc0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40cbe00640 1 -- 172.21.15.45:0/2271529610 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40cc0d4370 msgr2=0x7f40cc0cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:46.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40cbe00640 1 --2- 172.21.15.45:0/2271529610 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40cc0d4370 0x7f40cc0cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40cbe00640 1 -- 172.21.15.45:0/2271529610 >> v2:172.21.15.103:3300/0 conn(0x7f40cc0cfa70 msgr2=0x7f40cc162560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:46.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40cbe00640 1 --2- 172.21.15.45:0/2271529610 >> v2:172.21.15.103:3300/0 conn(0x7f40cc0cfa70 0x7f40cc162560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40cbe00640 1 -- 172.21.15.45:0/2271529610 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40cc0b6760 con 0x7f40cc15ea70 2024-04-16T16:18:46.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40cbe00640 1 -- 172.21.15.45:0/2271529610 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f40c0001670 con 0x7f40cc15ea70 2024-04-16T16:18:46.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40cbe00640 1 -- 172.21.15.45:0/2271529610 >> v1:172.21.15.45:6789/0 conn(0x7f40cc15ea70 legacy=0x7f40cc0d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40cbe00640 1 --2- 172.21.15.45:0/2271529610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b40039b0 0x7f40b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40cbe00640 1 --2- 172.21.15.45:0/2271529610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40b4004800 0x7f40b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40cbe00640 1 --2- 172.21.15.45:0/2271529610 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40cc0d4370 0x7f40b4005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40cbe00640 1 -- 172.21.15.45:0/2271529610 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc0d1a10 con 0x7f40b40039b0 2024-04-16T16:18:46.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40cbe00640 1 -- 172.21.15.45:0/2271529610 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc0b6480 con 0x7f40b4004800 2024-04-16T16:18:46.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40cbe00640 1 -- 172.21.15.45:0/2271529610 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f40b400bba0 con 0x7f40cc0d4370 2024-04-16T16:18:46.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.504+0000 7f40d1400640 1 --2- 172.21.15.45:0/2271529610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b40039b0 0x7f40b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.505+0000 7f40d1400640 1 -- 172.21.15.45:0/2271529610 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40cc0d4370 msgr2=0x7f40b4005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:46.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.505+0000 7f40d1400640 1 --2- 172.21.15.45:0/2271529610 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40cc0d4370 0x7f40b4005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.505+0000 7f40d1400640 1 -- 172.21.15.45:0/2271529610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40b4004800 msgr2=0x7f40b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:46.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.505+0000 7f40d1400640 1 --2- 172.21.15.45:0/2271529610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40b4004800 0x7f40b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.505+0000 7f40d1400640 1 -- 172.21.15.45:0/2271529610 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b4003520 con 0x7f40b40039b0 2024-04-16T16:18:46.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.506+0000 7f40d1400640 1 --2- 172.21.15.45:0/2271529610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b40039b0 0x7f40b4003dd0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f40c000c1c0 tx=0x7f40b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04518ad4f9caca7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:46.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.506+0000 7f40cbe00640 1 -- 172.21.15.45:0/2271529610 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0007c80 con 0x7f40b40039b0 2024-04-16T16:18:46.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.506+0000 7f40cbe00640 1 -- 172.21.15.45:0/2271529610 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c0030070 con 0x7f40b40039b0 2024-04-16T16:18:46.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.506+0000 7f40cbe00640 1 -- 172.21.15.45:0/2271529610 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0005d30 con 0x7f40b40039b0 2024-04-16T16:18:46.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.507+0000 7f40d2800640 1 -- 172.21.15.45:0/2271529610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b40039b0 msgr2=0x7f40b4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.507+0000 7f40d2800640 1 --2- 172.21.15.45:0/2271529610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b40039b0 0x7f40b4003dd0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f40c000c1c0 tx=0x7f40b4005830 comp rx=0 tx=0).stop 2024-04-16T16:18:46.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.507+0000 7f40d1e00640 1 -- 172.21.15.45:0/2271529610 reap_dead start 2024-04-16T16:18:46.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.507+0000 7f40d2800640 1 -- 172.21.15.45:0/2271529610 shutdown_connections 2024-04-16T16:18:46.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.507+0000 7f40d2800640 1 -- 172.21.15.45:0/2271529610 >> 172.21.15.45:0/2271529610 conn(0x7f40cc06d3b0 msgr2=0x7f40cc073060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:46.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.507+0000 7f40d2800640 1 -- 172.21.15.45:0/2271529610 shutdown_connections 2024-04-16T16:18:46.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.507+0000 7f40d2800640 1 -- 172.21.15.45:0/2271529610 wait complete. 2024-04-16T16:18:46.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.509+0000 7f40d2800640 1 Processor -- start 2024-04-16T16:18:46.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.509+0000 7f40d2800640 1 -- start start 2024-04-16T16:18:46.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.510+0000 7f40d2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b40039b0 0x7f40cc156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.510+0000 7f40d2800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40cc1f85a0 0x7f40cc1f4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.510+0000 7f40d2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40cc15b8b0 0x7f40cc15bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.510+0000 7f40d2800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc0b6f60 con 0x7f40b40039b0 2024-04-16T16:18:46.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.510+0000 7f40d2800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc0cf530 con 0x7f40cc15b8b0 2024-04-16T16:18:46.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.510+0000 7f40d2800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f40cc1620b0 con 0x7f40cc1f85a0 2024-04-16T16:18:46.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.510+0000 7f40d1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b40039b0 0x7f40cc156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.510+0000 7f40d1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b40039b0 0x7f40cc156580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51804/0 (socket says 172.21.15.45:51804) 2024-04-16T16:18:46.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.510+0000 7f40d1400640 1 -- 172.21.15.45:0/1181912542 learned_addr learned my addr 172.21.15.45:0/1181912542 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:46.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.510+0000 7f40d0a00640 1 --2- 172.21.15.45:0/1181912542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40cc1f85a0 0x7f40cc1f4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.510+0000 7f40d1400640 1 -- 172.21.15.45:0/1181912542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40cc1f85a0 msgr2=0x7f40cc1f4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.510+0000 7f40d1400640 1 --2- 172.21.15.45:0/1181912542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40cc1f85a0 0x7f40cc1f4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.511+0000 7f40d1e00640 1 --2- 172.21.15.45:0/1181912542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40cc15b8b0 0x7f40cc15bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.511+0000 7f40d1400640 1 -- 172.21.15.45:0/1181912542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40cc15b8b0 msgr2=0x7f40cc15bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.511+0000 7f40d1400640 1 --2- 172.21.15.45:0/1181912542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40cc15b8b0 0x7f40cc15bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.511+0000 7f40d1400640 1 -- 172.21.15.45:0/1181912542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40cc156bf0 con 0x7f40b40039b0 2024-04-16T16:18:46.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.511+0000 7f40d1400640 1 --2- 172.21.15.45:0/1181912542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b40039b0 0x7f40cc156580 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f40c000c1c0 tx=0x7f40c0001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:46.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.511+0000 7f40c9600640 1 -- 172.21.15.45:0/1181912542 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0001bc0 con 0x7f40b40039b0 2024-04-16T16:18:46.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.511+0000 7f40c9600640 1 -- 172.21.15.45:0/1181912542 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c0004bf0 con 0x7f40b40039b0 2024-04-16T16:18:46.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.511+0000 7f40c9600640 1 -- 172.21.15.45:0/1181912542 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0007070 con 0x7f40b40039b0 2024-04-16T16:18:46.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.511+0000 7f40d2800640 1 -- 172.21.15.45:0/1181912542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40cc156ed0 con 0x7f40b40039b0 2024-04-16T16:18:46.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.512+0000 7f40d2800640 1 -- 172.21.15.45:0/1181912542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40cc157490 con 0x7f40b40039b0 2024-04-16T16:18:46.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.513+0000 7f40d2800640 1 -- 172.21.15.45:0/1181912542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40cc075780 con 0x7f40b40039b0 2024-04-16T16:18:46.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.514+0000 7f40c9600640 1 -- 172.21.15.45:0/1181912542 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 31) ==== 90878+0+0 (secure 0 0 0) 0x7f40c0048020 con 0x7f40b40039b0 2024-04-16T16:18:46.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.515+0000 7f40c9600640 1 --2- 172.21.15.45:0/1181912542 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f40ac06c740 0x7f40ac06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.515+0000 7f40c9600640 1 -- 172.21.15.45:0/1181912542 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6354+0+0 (secure 0 0 0) 0x7f40c00ad0d0 con 0x7f40b40039b0 2024-04-16T16:18:46.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.515+0000 7f40d0a00640 1 --2- 172.21.15.45:0/1181912542 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f40ac06c740 0x7f40ac06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.517+0000 7f40d0a00640 1 --2- 172.21.15.45:0/1181912542 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f40ac06c740 0x7f40ac06ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f40b8006b10 tx=0x7f40b8008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:46.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.519+0000 7f40c9600640 1 -- 172.21.15.45:0/1181912542 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f40c007b560 con 0x7f40b40039b0 2024-04-16T16:18:46.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.724+0000 7f40d2800640 1 -- 172.21.15.45:0/1181912542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f40cc0cf850 con 0x7f40b40039b0 2024-04-16T16:18:46.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.725+0000 7f40c9600640 1 -- 172.21.15.45:0/1181912542 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f40c007ae60 con 0x7f40b40039b0 2024-04-16T16:18:46.724 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:18:46.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.727+0000 7f40d2800640 1 -- 172.21.15.45:0/1181912542 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f40ac06c740 msgr2=0x7f40ac06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.727+0000 7f40d2800640 1 --2- 172.21.15.45:0/1181912542 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f40ac06c740 0x7f40ac06ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f40b8006b10 tx=0x7f40b8008040 comp rx=0 tx=0).stop 2024-04-16T16:18:46.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.727+0000 7f40d2800640 1 -- 172.21.15.45:0/1181912542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b40039b0 msgr2=0x7f40cc156580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.727+0000 7f40d2800640 1 --2- 172.21.15.45:0/1181912542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b40039b0 0x7f40cc156580 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f40c000c1c0 tx=0x7f40c0001970 comp rx=0 tx=0).stop 2024-04-16T16:18:46.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.728+0000 7f40d2800640 1 -- 172.21.15.45:0/1181912542 shutdown_connections 2024-04-16T16:18:46.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.728+0000 7f40d2800640 1 --2- 172.21.15.45:0/1181912542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40cc15b8b0 0x7f40cc15bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.728+0000 7f40d2800640 1 --2- 172.21.15.45:0/1181912542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40cc1f85a0 0x7f40cc1f4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.728+0000 7f40d2800640 1 --2- 172.21.15.45:0/1181912542 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f40ac06c740 0x7f40ac06ec00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.728+0000 7f40d2800640 1 --2- 172.21.15.45:0/1181912542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b40039b0 0x7f40cc156580 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.728+0000 7f40d2800640 1 -- 172.21.15.45:0/1181912542 >> 172.21.15.45:0/1181912542 conn(0x7f40cc06d3b0 msgr2=0x7f40cc0b7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:46.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.728+0000 7f40d2800640 1 -- 172.21.15.45:0/1181912542 shutdown_connections 2024-04-16T16:18:46.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.728+0000 7f40d2800640 1 -- 172.21.15.45:0/1181912542 wait complete. 2024-04-16T16:18:46.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:18:46.979 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:46.979+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:18:46.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.982+0000 7f4090e00640 1 Processor -- start 2024-04-16T16:18:46.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.982+0000 7f4090e00640 1 -- start start 2024-04-16T16:18:46.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.983+0000 7f4090e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f408c0d18f0 0x7f408c0ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.983+0000 7f4090e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f408c0d1cd0 0x7f408c162dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.983+0000 7f4090e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f408c0b6810 con 0x7f408c0b6310 2024-04-16T16:18:46.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.983+0000 7f4090e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f408c0d29f0 con 0x7f408c0d18f0 2024-04-16T16:18:46.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.983+0000 7f4090e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f408c0b6af0 con 0x7f408c0d1cd0 2024-04-16T16:18:46.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.984+0000 7f408be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f408c0d1cd0 0x7f408c162dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.984+0000 7f408be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f408c0d1cd0 0x7f408c162dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:39188/0 (socket says 172.21.15.45:39188) 2024-04-16T16:18:46.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.984+0000 7f408be00640 1 -- 172.21.15.45:0/2326503948 learned_addr learned my addr 172.21.15.45:0/2326503948 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:46.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.984+0000 7f408b400640 1 --2- 172.21.15.45:0/2326503948 >> v2:172.21.15.103:3300/0 conn(0x7f408c0d18f0 0x7f408c0ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.985+0000 7f408be00640 1 -- 172.21.15.45:0/2326503948 >> v2:172.21.15.103:3300/0 conn(0x7f408c0d18f0 msgr2=0x7f408c0ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:46.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.985+0000 7f408be00640 1 --2- 172.21.15.45:0/2326503948 >> v2:172.21.15.103:3300/0 conn(0x7f408c0d18f0 0x7f408c0ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.985+0000 7f408be00640 1 -- 172.21.15.45:0/2326503948 >> v1:172.21.15.103:6789/0 conn(0x7f408c0b6310 legacy=0x7f408c0d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.985+0000 7f408be00640 1 -- 172.21.15.45:0/2326503948 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408c0b6dd0 con 0x7f408c0d1cd0 2024-04-16T16:18:46.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.987+0000 7f408be00640 1 --2- 172.21.15.45:0/2326503948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f408c0d1cd0 0x7f408c162dc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4084009e50 tx=0x7f408402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=782988f6fd19bfe6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:46.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.987+0000 7f408a000640 1 -- 172.21.15.45:0/2326503948 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408402ca70 con 0x7f408c0d1cd0 2024-04-16T16:18:46.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.987+0000 7f408a000640 1 -- 172.21.15.45:0/2326503948 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408402cc10 con 0x7f408c0d1cd0 2024-04-16T16:18:46.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.987+0000 7f408a000640 1 -- 172.21.15.45:0/2326503948 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40840343f0 con 0x7f408c0d1cd0 2024-04-16T16:18:46.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.988+0000 7f4090e00640 1 -- 172.21.15.45:0/2326503948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f408c0d1cd0 msgr2=0x7f408c162dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.988+0000 7f4090e00640 1 --2- 172.21.15.45:0/2326503948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f408c0d1cd0 0x7f408c162dc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4084009e50 tx=0x7f408402bed0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.988+0000 7f4090e00640 1 -- 172.21.15.45:0/2326503948 shutdown_connections 2024-04-16T16:18:46.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.988+0000 7f4090e00640 1 --2- 172.21.15.45:0/2326503948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f408c0d1cd0 0x7f408c162dc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.988+0000 7f4090e00640 1 --2- 172.21.15.45:0/2326503948 >> v2:172.21.15.103:3300/0 conn(0x7f408c0d18f0 0x7f408c0ba220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.988+0000 7f4090e00640 1 -- 172.21.15.45:0/2326503948 >> 172.21.15.45:0/2326503948 conn(0x7f408c06cf90 msgr2=0x7f408c06d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:46.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.988+0000 7f4090e00640 1 -- 172.21.15.45:0/2326503948 shutdown_connections 2024-04-16T16:18:46.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.988+0000 7f4090e00640 1 -- 172.21.15.45:0/2326503948 wait complete. 2024-04-16T16:18:46.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.990+0000 7f4090e00640 1 Processor -- start 2024-04-16T16:18:46.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.990+0000 7f4090e00640 1 -- start start 2024-04-16T16:18:46.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.991+0000 7f4090e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f408c0b6310 0x7f408c150cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.991+0000 7f4090e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f408c0d18f0 0x7f408c0cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.991+0000 7f4090e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f408c0d1cd0 0x7f408c0cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.991+0000 7f4090e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f408c163300 con 0x7f408c0d1cd0 2024-04-16T16:18:46.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.991+0000 7f4090e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f408c0ba760 con 0x7f408c0d18f0 2024-04-16T16:18:46.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.991+0000 7f4090e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f408c151210 con 0x7f408c0b6310 2024-04-16T16:18:46.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.991+0000 7f408be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f408c0d1cd0 0x7f408c0cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.991+0000 7f408be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f408c0d1cd0 0x7f408c0cd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51808/0 (socket says 172.21.15.45:51808) 2024-04-16T16:18:46.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.991+0000 7f408be00640 1 -- 172.21.15.45:0/1370271354 learned_addr learned my addr 172.21.15.45:0/1370271354 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:46.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.992+0000 7f408b400640 1 --2- 172.21.15.45:0/1370271354 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f408c0b6310 0x7f408c150cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.992+0000 7f408be00640 1 -- 172.21.15.45:0/1370271354 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f408c0b6310 msgr2=0x7f408c150cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.992+0000 7f408be00640 1 --2- 172.21.15.45:0/1370271354 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f408c0b6310 0x7f408c150cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.992+0000 7f408be00640 1 -- 172.21.15.45:0/1370271354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f408c0d18f0 msgr2=0x7f408c0cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:46.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.992+0000 7f408aa00640 1 --2- 172.21.15.45:0/1370271354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f408c0d18f0 0x7f408c0cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.992+0000 7f408be00640 1 --2- 172.21.15.45:0/1370271354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f408c0d18f0 0x7f408c0cb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:46.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.992+0000 7f408be00640 1 -- 172.21.15.45:0/1370271354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408c0cc190 con 0x7f408c0d1cd0 2024-04-16T16:18:46.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.992+0000 7f408be00640 1 --2- 172.21.15.45:0/1370271354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f408c0d1cd0 0x7f408c0cd230 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f4084036040 tx=0x7f4084004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:46.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.992+0000 7f4083400640 1 -- 172.21.15.45:0/1370271354 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084044020 con 0x7f408c0d1cd0 2024-04-16T16:18:46.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.993+0000 7f4083400640 1 -- 172.21.15.45:0/1370271354 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40840406b0 con 0x7f408c0d1cd0 2024-04-16T16:18:46.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.993+0000 7f4083400640 1 -- 172.21.15.45:0/1370271354 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084009510 con 0x7f408c0d1cd0 2024-04-16T16:18:46.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.993+0000 7f4090e00640 1 -- 172.21.15.45:0/1370271354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f408c0cc470 con 0x7f408c0d1cd0 2024-04-16T16:18:46.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.993+0000 7f4090e00640 1 -- 172.21.15.45:0/1370271354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f408c0cca30 con 0x7f408c0d1cd0 2024-04-16T16:18:46.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.994+0000 7f4090e00640 1 -- 172.21.15.45:0/1370271354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f408c075780 con 0x7f408c0d1cd0 2024-04-16T16:18:46.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.995+0000 7f4083400640 1 -- 172.21.15.45:0/1370271354 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f4084047020 con 0x7f408c0d1cd0 2024-04-16T16:18:46.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.996+0000 7f4083400640 1 --2- 172.21.15.45:0/1370271354 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f406c06c980 0x7f406c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:46.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.996+0000 7f4083400640 1 -- 172.21.15.45:0/1370271354 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6354+0+0 (secure 0 0 0) 0x7f40840ae970 con 0x7f408c0d1cd0 2024-04-16T16:18:46.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.997+0000 7f408b400640 1 --2- 172.21.15.45:0/1370271354 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f406c06c980 0x7f406c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:46.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:46.998+0000 7f408b400640 1 --2- 172.21.15.45:0/1370271354 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f406c06c980 0x7f406c06ee40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f407c009a10 tx=0x7f407c008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:46.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.000+0000 7f4083400640 1 -- 172.21.15.45:0/1370271354 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f408404e030 con 0x7f408c0d1cd0 2024-04-16T16:18:47.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.211+0000 7f4090e00640 1 -- 172.21.15.45:0/1370271354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f408c0d25a0 con 0x7f408c0d1cd0 2024-04-16T16:18:47.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.212+0000 7f4083400640 1 -- 172.21.15.45:0/1370271354 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f408407ca20 con 0x7f408c0d1cd0 2024-04-16T16:18:47.211 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:18:47.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.215+0000 7f4090e00640 1 -- 172.21.15.45:0/1370271354 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f406c06c980 msgr2=0x7f406c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:47.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.215+0000 7f4090e00640 1 --2- 172.21.15.45:0/1370271354 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f406c06c980 0x7f406c06ee40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f407c009a10 tx=0x7f407c008040 comp rx=0 tx=0).stop 2024-04-16T16:18:47.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.215+0000 7f4090e00640 1 -- 172.21.15.45:0/1370271354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f408c0d1cd0 msgr2=0x7f408c0cd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:47.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.215+0000 7f4090e00640 1 --2- 172.21.15.45:0/1370271354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f408c0d1cd0 0x7f408c0cd230 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f4084036040 tx=0x7f4084004a40 comp rx=0 tx=0).stop 2024-04-16T16:18:47.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.215+0000 7f4090e00640 1 -- 172.21.15.45:0/1370271354 shutdown_connections 2024-04-16T16:18:47.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.215+0000 7f4090e00640 1 --2- 172.21.15.45:0/1370271354 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f406c06c980 0x7f406c06ee40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.215+0000 7f4090e00640 1 --2- 172.21.15.45:0/1370271354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f408c0d1cd0 0x7f408c0cd230 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.215+0000 7f4090e00640 1 --2- 172.21.15.45:0/1370271354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f408c0d18f0 0x7f408c0cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.215+0000 7f4090e00640 1 --2- 172.21.15.45:0/1370271354 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f408c0b6310 0x7f408c150cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.215+0000 7f4090e00640 1 -- 172.21.15.45:0/1370271354 >> 172.21.15.45:0/1370271354 conn(0x7f408c06cf90 msgr2=0x7f408c0b9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.215+0000 7f4090e00640 1 -- 172.21.15.45:0/1370271354 shutdown_connections 2024-04-16T16:18:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.215+0000 7f4090e00640 1 -- 172.21.15.45:0/1370271354 wait complete. 2024-04-16T16:18:47.232 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:18:47.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.452+0000 7fae31200640 1 Processor -- start 2024-04-16T16:18:47.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.452+0000 7fae31200640 1 -- start start 2024-04-16T16:18:47.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.454+0000 7fae31200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae2c0d10e0 0x7fae2c072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.454+0000 7fae31200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fae2c0b4930 0x7fae2c06f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.454+0000 7fae31200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae2c0b5530 con 0x7fae2c15c7c0 2024-04-16T16:18:47.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.454+0000 7fae31200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fae2c0b52e0 con 0x7fae2c0b4930 2024-04-16T16:18:47.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.454+0000 7fae31200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fae2c0d0260 con 0x7fae2c0d10e0 2024-04-16T16:18:47.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.454+0000 7fae29e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fae2c15c7c0 0x7fae2c0d09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54440/0 (socket says 172.21.15.45:54440) 2024-04-16T16:18:47.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.454+0000 7fae29e00640 1 -- 172.21.15.45:0/3977508101 learned_addr learned my addr 172.21.15.45:0/3977508101 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:47.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.455+0000 7fae29400640 1 --2- 172.21.15.45:0/3977508101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae2c0d10e0 0x7fae2c072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.455+0000 7fae2a800640 1 --2- 172.21.15.45:0/3977508101 >> v2:172.21.15.103:3300/0 conn(0x7fae2c0b4930 0x7fae2c06f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.455+0000 7fae28a00640 1 -- 172.21.15.45:0/3977508101 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2738875260 0 0) 0x7fae2c0b5530 con 0x7fae2c15c7c0 2024-04-16T16:18:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.455+0000 7fae28a00640 1 -- 172.21.15.45:0/3977508101 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae10003520 con 0x7fae2c15c7c0 2024-04-16T16:18:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.455+0000 7fae28a00640 1 -- 172.21.15.45:0/3977508101 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2470555760 0 0) 0x7fae10003520 con 0x7fae2c15c7c0 2024-04-16T16:18:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.456+0000 7fae28a00640 1 -- 172.21.15.45:0/3977508101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae2c0d10e0 msgr2=0x7fae2c072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.456+0000 7fae28a00640 1 --2- 172.21.15.45:0/3977508101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae2c0d10e0 0x7fae2c072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.456+0000 7fae28a00640 1 -- 172.21.15.45:0/3977508101 >> v2:172.21.15.103:3300/0 conn(0x7fae2c0b4930 msgr2=0x7fae2c06f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.456+0000 7fae28a00640 1 --2- 172.21.15.45:0/3977508101 >> v2:172.21.15.103:3300/0 conn(0x7fae2c0b4930 0x7fae2c06f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.456+0000 7fae28a00640 1 -- 172.21.15.45:0/3977508101 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2c0b5810 con 0x7fae2c15c7c0 2024-04-16T16:18:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.456+0000 7fae28a00640 1 -- 172.21.15.45:0/3977508101 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fae1c001670 con 0x7fae2c15c7c0 2024-04-16T16:18:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.456+0000 7fae28a00640 1 -- 172.21.15.45:0/3977508101 >> v1:172.21.15.45:6789/0 conn(0x7fae2c15c7c0 legacy=0x7fae2c0d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.456+0000 7fae28a00640 1 --2- 172.21.15.45:0/3977508101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae100039b0 0x7fae10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.456+0000 7fae28a00640 1 --2- 172.21.15.45:0/3977508101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae2c0d10e0 0x7fae10004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.456+0000 7fae28a00640 1 --2- 172.21.15.45:0/3977508101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae10004d40 0x7fae100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.456+0000 7fae28a00640 1 -- 172.21.15.45:0/3977508101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c0b52e0 con 0x7fae100039b0 2024-04-16T16:18:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.456+0000 7fae28a00640 1 -- 172.21.15.45:0/3977508101 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c0d0260 con 0x7fae10004d40 2024-04-16T16:18:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.456+0000 7fae28a00640 1 -- 172.21.15.45:0/3977508101 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fae1000ba70 con 0x7fae2c0d10e0 2024-04-16T16:18:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.456+0000 7fae29e00640 1 --2- 172.21.15.45:0/3977508101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae100039b0 0x7fae10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.456+0000 7fae29400640 1 --2- 172.21.15.45:0/3977508101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae2c0d10e0 0x7fae10004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.457+0000 7fae29e00640 1 -- 172.21.15.45:0/3977508101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae2c0d10e0 msgr2=0x7fae10004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.457+0000 7fae29e00640 1 --2- 172.21.15.45:0/3977508101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae2c0d10e0 0x7fae10004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.457+0000 7fae29e00640 1 -- 172.21.15.45:0/3977508101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae10004d40 msgr2=0x7fae100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.457+0000 7fae29e00640 1 --2- 172.21.15.45:0/3977508101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae10004d40 0x7fae100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.457+0000 7fae29e00640 1 -- 172.21.15.45:0/3977508101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae10003520 con 0x7fae100039b0 2024-04-16T16:18:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.457+0000 7fae29400640 1 --2- 172.21.15.45:0/3977508101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae2c0d10e0 0x7fae10004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:47.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.458+0000 7fae29e00640 1 --2- 172.21.15.45:0/3977508101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae100039b0 0x7fae10003dd0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fae1c007fd0 tx=0x7fae10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc226100f9965b75 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:47.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.458+0000 7fae28a00640 1 -- 172.21.15.45:0/3977508101 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c007d50 con 0x7fae100039b0 2024-04-16T16:18:47.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.458+0000 7fae28a00640 1 -- 172.21.15.45:0/3977508101 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae1c030070 con 0x7fae100039b0 2024-04-16T16:18:47.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.458+0000 7fae28a00640 1 -- 172.21.15.45:0/3977508101 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c02cd40 con 0x7fae100039b0 2024-04-16T16:18:47.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.459+0000 7fae31200640 1 -- 172.21.15.45:0/3977508101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae100039b0 msgr2=0x7fae10003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:47.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.459+0000 7fae31200640 1 --2- 172.21.15.45:0/3977508101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae100039b0 0x7fae10003dd0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fae1c007fd0 tx=0x7fae10005830 comp rx=0 tx=0).stop 2024-04-16T16:18:47.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.459+0000 7fae2a800640 1 -- 172.21.15.45:0/3977508101 reap_dead start 2024-04-16T16:18:47.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.459+0000 7fae31200640 1 -- 172.21.15.45:0/3977508101 shutdown_connections 2024-04-16T16:18:47.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.459+0000 7fae31200640 1 -- 172.21.15.45:0/3977508101 >> 172.21.15.45:0/3977508101 conn(0x7fae2c0b3970 msgr2=0x7fae2c0b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:47.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.459+0000 7fae31200640 1 -- 172.21.15.45:0/3977508101 shutdown_connections 2024-04-16T16:18:47.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.459+0000 7fae31200640 1 -- 172.21.15.45:0/3977508101 wait complete. 2024-04-16T16:18:47.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.461+0000 7fae31200640 1 Processor -- start 2024-04-16T16:18:47.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.461+0000 7fae31200640 1 -- start start 2024-04-16T16:18:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.462+0000 7fae31200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae100039b0 0x7fae2c0c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.462+0000 7fae31200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c0ac0b0 0x7fae2c159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.462+0000 7fae31200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae2c0b2700 0x7fae2c0b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.462+0000 7fae31200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c06f820 con 0x7fae2c0ac0b0 2024-04-16T16:18:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.462+0000 7fae31200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c06f6a0 con 0x7fae2c0b2700 2024-04-16T16:18:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.462+0000 7fae31200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fae2c06f9a0 con 0x7fae100039b0 2024-04-16T16:18:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.462+0000 7fae29400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c0ac0b0 0x7fae2c159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.462+0000 7fae29400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c0ac0b0 0x7fae2c159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51822/0 (socket says 172.21.15.45:51822) 2024-04-16T16:18:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.462+0000 7fae29400640 1 -- 172.21.15.45:0/262413704 learned_addr learned my addr 172.21.15.45:0/262413704 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:47.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.462+0000 7fae29e00640 1 --2- 172.21.15.45:0/262413704 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae100039b0 0x7fae2c0c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:47.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.463+0000 7fae29400640 1 -- 172.21.15.45:0/262413704 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae100039b0 msgr2=0x7fae2c0c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.463+0000 7fae29400640 1 --2- 172.21.15.45:0/262413704 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae100039b0 0x7fae2c0c9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.463+0000 7fae29400640 1 -- 172.21.15.45:0/262413704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae2c0b2700 msgr2=0x7fae2c0b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.463+0000 7fae29400640 1 --2- 172.21.15.45:0/262413704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae2c0b2700 0x7fae2c0b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.463+0000 7fae29400640 1 -- 172.21.15.45:0/262413704 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2c0ac890 con 0x7fae2c0ac0b0 2024-04-16T16:18:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.463+0000 7fae29400640 1 --2- 172.21.15.45:0/262413704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c0ac0b0 0x7fae2c159120 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fae18010880 tx=0x7fae18010d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.463+0000 7fae22000640 1 -- 172.21.15.45:0/262413704 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1800f040 con 0x7fae2c0ac0b0 2024-04-16T16:18:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.463+0000 7fae22000640 1 -- 172.21.15.45:0/262413704 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae1800ea80 con 0x7fae2c0ac0b0 2024-04-16T16:18:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.463+0000 7fae31200640 1 -- 172.21.15.45:0/262413704 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae2c0ad050 con 0x7fae2c0ac0b0 2024-04-16T16:18:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.463+0000 7fae22000640 1 -- 172.21.15.45:0/262413704 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1800ee50 con 0x7fae2c0ac0b0 2024-04-16T16:18:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.464+0000 7fae31200640 1 -- 172.21.15.45:0/262413704 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae2c0ad610 con 0x7fae2c0ac0b0 2024-04-16T16:18:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.465+0000 7fae31200640 1 -- 172.21.15.45:0/262413704 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae2c0b30c0 con 0x7fae2c0ac0b0 2024-04-16T16:18:47.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.466+0000 7fae22000640 1 -- 172.21.15.45:0/262413704 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7fae1801c400 con 0x7fae2c0ac0b0 2024-04-16T16:18:47.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.467+0000 7fae22000640 1 --2- 172.21.15.45:0/262413704 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fae0c06c9b0 0x7fae0c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.467+0000 7fae22000640 1 -- 172.21.15.45:0/262413704 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6354+0+0 (secure 0 0 0) 0x7fae1808c420 con 0x7fae2c0ac0b0 2024-04-16T16:18:47.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.468+0000 7fae29e00640 1 --2- 172.21.15.45:0/262413704 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fae0c06c9b0 0x7fae0c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:47.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.469+0000 7fae29e00640 1 --2- 172.21.15.45:0/262413704 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fae0c06c9b0 0x7fae0c06ee70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fae1c034000 tx=0x7fae1c007000 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:47.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.472+0000 7fae22000640 1 -- 172.21.15.45:0/262413704 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae18018030 con 0x7fae2c0ac0b0 2024-04-16T16:18:47.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.681+0000 7fae31200640 1 -- 172.21.15.45:0/262413704 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fae2c075780 con 0x7fae2c0ac0b0 2024-04-16T16:18:47.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.682+0000 7fae22000640 1 -- 172.21.15.45:0/262413704 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fae1805a5d0 con 0x7fae2c0ac0b0 2024-04-16T16:18:47.681 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:18:47.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.685+0000 7fae31200640 1 -- 172.21.15.45:0/262413704 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fae0c06c9b0 msgr2=0x7fae0c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:47.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.685+0000 7fae31200640 1 --2- 172.21.15.45:0/262413704 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fae0c06c9b0 0x7fae0c06ee70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fae1c034000 tx=0x7fae1c007000 comp rx=0 tx=0).stop 2024-04-16T16:18:47.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.685+0000 7fae31200640 1 -- 172.21.15.45:0/262413704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c0ac0b0 msgr2=0x7fae2c159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:47.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.685+0000 7fae31200640 1 --2- 172.21.15.45:0/262413704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c0ac0b0 0x7fae2c159120 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fae18010880 tx=0x7fae18010d40 comp rx=0 tx=0).stop 2024-04-16T16:18:47.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.685+0000 7fae31200640 1 -- 172.21.15.45:0/262413704 shutdown_connections 2024-04-16T16:18:47.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.685+0000 7fae31200640 1 --2- 172.21.15.45:0/262413704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae2c0b2700 0x7fae2c0b2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.685+0000 7fae31200640 1 --2- 172.21.15.45:0/262413704 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fae0c06c9b0 0x7fae0c06ee70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.685+0000 7fae31200640 1 --2- 172.21.15.45:0/262413704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c0ac0b0 0x7fae2c159120 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.685+0000 7fae31200640 1 --2- 172.21.15.45:0/262413704 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae100039b0 0x7fae2c0c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.685+0000 7fae31200640 1 -- 172.21.15.45:0/262413704 >> 172.21.15.45:0/262413704 conn(0x7fae2c0b3970 msgr2=0x7fae2c0721c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:47.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.685+0000 7fae31200640 1 -- 172.21.15.45:0/262413704 shutdown_connections 2024-04-16T16:18:47.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.685+0000 7fae31200640 1 -- 172.21.15.45:0/262413704 wait complete. 2024-04-16T16:18:47.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:18:47.894 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:47.893+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:18:47.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.934+0000 7f6a7c200640 1 Processor -- start 2024-04-16T16:18:47.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.934+0000 7f6a7c200640 1 -- start start 2024-04-16T16:18:47.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.936+0000 7f6a7c200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a74072a90 0x7f6a741699a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.936+0000 7f6a7c200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a74147c20 con 0x7f6a740aed90 2024-04-16T16:18:47.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.936+0000 7f6a7c200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a74160210 con 0x7f6a74147840 2024-04-16T16:18:47.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.936+0000 7f6a7c200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6a740afe80 con 0x7f6a74072a90 2024-04-16T16:18:47.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.936+0000 7f6a78e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6a740aed90 0x7f6a74072380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54444/0 (socket says 172.21.15.45:54444) 2024-04-16T16:18:47.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.936+0000 7f6a78e00640 1 -- 172.21.15.45:0/3190790228 learned_addr learned my addr 172.21.15.45:0/3190790228 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:47.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a7a200640 1 --2- 172.21.15.45:0/3190790228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a74072a90 0x7f6a741699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:47.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a73e00640 1 -- 172.21.15.45:0/3190790228 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4041682862 0 0) 0x7f6a74147c20 con 0x7f6a740aed90 2024-04-16T16:18:47.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a73e00640 1 -- 172.21.15.45:0/3190790228 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a5c003520 con 0x7f6a740aed90 2024-04-16T16:18:47.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a73e00640 1 -- 172.21.15.45:0/3190790228 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3525833402 0 0) 0x7f6a5c003520 con 0x7f6a740aed90 2024-04-16T16:18:47.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a73e00640 1 -- 172.21.15.45:0/3190790228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a74072a90 msgr2=0x7f6a741699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:47.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a73e00640 1 --2- 172.21.15.45:0/3190790228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a74072a90 0x7f6a741699a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a73e00640 1 -- 172.21.15.45:0/3190790228 >> v1:172.21.15.103:6789/0 conn(0x7f6a74147840 legacy=0x7f6a7415fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:47.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a73e00640 1 -- 172.21.15.45:0/3190790228 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a740af270 con 0x7f6a740aed90 2024-04-16T16:18:47.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a73e00640 1 -- 172.21.15.45:0/3190790228 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f6a60001670 con 0x7f6a740aed90 2024-04-16T16:18:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a73e00640 1 -- 172.21.15.45:0/3190790228 >> v1:172.21.15.45:6789/0 conn(0x7f6a740aed90 legacy=0x7f6a74072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a73e00640 1 --2- 172.21.15.45:0/3190790228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a74072a90 0x7f6a5c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a73e00640 1 --2- 172.21.15.45:0/3190790228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a5c003fd0 0x7f6a5c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a73e00640 1 --2- 172.21.15.45:0/3190790228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a5c004d40 0x7f6a5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a73e00640 1 -- 172.21.15.45:0/3190790228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6a740afe80 con 0x7f6a5c004d40 2024-04-16T16:18:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a73e00640 1 -- 172.21.15.45:0/3190790228 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c00bb20 con 0x7f6a5c003fd0 2024-04-16T16:18:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.937+0000 7f6a73e00640 1 -- 172.21.15.45:0/3190790228 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6a5c00bd20 con 0x7f6a74072a90 2024-04-16T16:18:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.938+0000 7f6a7a200640 1 --2- 172.21.15.45:0/3190790228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a5c004d40 0x7f6a5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.938+0000 7f6a7a200640 1 -- 172.21.15.45:0/3190790228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a74072a90 msgr2=0x7f6a5c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.938+0000 7f6a7a200640 1 --2- 172.21.15.45:0/3190790228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a74072a90 0x7f6a5c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.938+0000 7f6a7a200640 1 -- 172.21.15.45:0/3190790228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a5c003fd0 msgr2=0x7f6a5c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.938+0000 7f6a7a200640 1 --2- 172.21.15.45:0/3190790228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a5c003fd0 0x7f6a5c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.938+0000 7f6a7a200640 1 -- 172.21.15.45:0/3190790228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a5c003520 con 0x7f6a5c004d40 2024-04-16T16:18:47.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.939+0000 7f6a7a200640 1 --2- 172.21.15.45:0/3190790228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a5c004d40 0x7f6a5c0051c0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f6a740b0690 tx=0x7f6a5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14fc38ef220b1943 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:47.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.940+0000 7f6a73e00640 1 -- 172.21.15.45:0/3190790228 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c039070 con 0x7f6a5c004d40 2024-04-16T16:18:47.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.940+0000 7f6a73e00640 1 -- 172.21.15.45:0/3190790228 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a6c02cbd0 con 0x7f6a5c004d40 2024-04-16T16:18:47.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.940+0000 7f6a73e00640 1 -- 172.21.15.45:0/3190790228 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c009c10 con 0x7f6a5c004d40 2024-04-16T16:18:47.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.940+0000 7f6a7c200640 1 -- 172.21.15.45:0/3190790228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a5c004d40 msgr2=0x7f6a5c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:47.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.940+0000 7f6a7c200640 1 --2- 172.21.15.45:0/3190790228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a5c004d40 0x7f6a5c0051c0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f6a740b0690 tx=0x7f6a5c005830 comp rx=0 tx=0).stop 2024-04-16T16:18:47.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.940+0000 7f6a7a200640 1 -- 172.21.15.45:0/3190790228 reap_dead start 2024-04-16T16:18:47.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.940+0000 7f6a7c200640 1 -- 172.21.15.45:0/3190790228 shutdown_connections 2024-04-16T16:18:47.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.940+0000 7f6a7c200640 1 -- 172.21.15.45:0/3190790228 >> 172.21.15.45:0/3190790228 conn(0x7f6a740b2750 msgr2=0x7f6a740b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:47.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.940+0000 7f6a7c200640 1 -- 172.21.15.45:0/3190790228 shutdown_connections 2024-04-16T16:18:47.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.940+0000 7f6a7c200640 1 -- 172.21.15.45:0/3190790228 wait complete. 2024-04-16T16:18:47.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.942+0000 7f6a7c200640 1 Processor -- start 2024-04-16T16:18:47.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.942+0000 7f6a7c200640 1 -- start start 2024-04-16T16:18:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.943+0000 7f6a7c200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7415dc70 0x7f6a740cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.943+0000 7f6a7c200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a740b5220 0x7f6a740cf690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.943+0000 7f6a7c200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a741e8450 0x7f6a741e88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.943+0000 7f6a7c200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6a740afb10 con 0x7f6a740b5220 2024-04-16T16:18:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.943+0000 7f6a7c200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6a740af830 con 0x7f6a7415dc70 2024-04-16T16:18:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.943+0000 7f6a7c200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6a7406f2c0 con 0x7f6a741e8450 2024-04-16T16:18:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.944+0000 7f6a78e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a740b5220 0x7f6a740cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.944+0000 7f6a78e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a740b5220 0x7f6a740cf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51850/0 (socket says 172.21.15.45:51850) 2024-04-16T16:18:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.944+0000 7f6a78e00640 1 -- 172.21.15.45:0/2137747556 learned_addr learned my addr 172.21.15.45:0/2137747556 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.944+0000 7f6a79800640 1 --2- 172.21.15.45:0/2137747556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7415dc70 0x7f6a740cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.944+0000 7f6a78e00640 1 -- 172.21.15.45:0/2137747556 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a741e8450 msgr2=0x7f6a741e88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.944+0000 7f6a7a200640 1 --2- 172.21.15.45:0/2137747556 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a741e8450 0x7f6a741e88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.944+0000 7f6a78e00640 1 --2- 172.21.15.45:0/2137747556 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a741e8450 0x7f6a741e88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.944+0000 7f6a78e00640 1 -- 172.21.15.45:0/2137747556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7415dc70 msgr2=0x7f6a740cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.944+0000 7f6a78e00640 1 --2- 172.21.15.45:0/2137747556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7415dc70 0x7f6a740cdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.944+0000 7f6a78e00640 1 -- 172.21.15.45:0/2137747556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a740ce5f0 con 0x7f6a740b5220 2024-04-16T16:18:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.944+0000 7f6a79800640 1 --2- 172.21.15.45:0/2137747556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7415dc70 0x7f6a740cdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:18:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.944+0000 7f6a78e00640 1 --2- 172.21.15.45:0/2137747556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a740b5220 0x7f6a740cf690 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f6a6000c700 tx=0x7f6a6000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.945+0000 7f6a71600640 1 -- 172.21.15.45:0/2137747556 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60018070 con 0x7f6a740b5220 2024-04-16T16:18:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.945+0000 7f6a71600640 1 -- 172.21.15.45:0/2137747556 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a600074e0 con 0x7f6a740b5220 2024-04-16T16:18:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.945+0000 7f6a71600640 1 -- 172.21.15.45:0/2137747556 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60004a20 con 0x7f6a740b5220 2024-04-16T16:18:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.945+0000 7f6a7c200640 1 -- 172.21.15.45:0/2137747556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a740ce8d0 con 0x7f6a740b5220 2024-04-16T16:18:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.945+0000 7f6a7c200640 1 -- 172.21.15.45:0/2137747556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a740cee90 con 0x7f6a740b5220 2024-04-16T16:18:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.946+0000 7f6a7c200640 1 -- 172.21.15.45:0/2137747556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a74075780 con 0x7f6a740b5220 2024-04-16T16:18:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.947+0000 7f6a71600640 1 -- 172.21.15.45:0/2137747556 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f6a6001b070 con 0x7f6a740b5220 2024-04-16T16:18:47.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.948+0000 7f6a71600640 1 --2- 172.21.15.45:0/2137747556 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6a5406c980 0x7f6a5406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:47.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.948+0000 7f6a71600640 1 -- 172.21.15.45:0/2137747556 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6354+0+0 (secure 0 0 0) 0x7f6a6008b4a0 con 0x7f6a740b5220 2024-04-16T16:18:47.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.949+0000 7f6a79800640 1 --2- 172.21.15.45:0/2137747556 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6a5406c980 0x7f6a5406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:47.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.950+0000 7f6a79800640 1 --2- 172.21.15.45:0/2137747556 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6a5406c980 0x7f6a5406ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6a740b5b80 tx=0x7f6a68009040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:47.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:47.953+0000 7f6a71600640 1 -- 172.21.15.45:0/2137747556 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6a60059550 con 0x7f6a740b5220 2024-04-16T16:18:48.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.213+0000 7f6a7c200640 1 -- 172.21.15.45:0/2137747556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a74070220 con 0x7f6a740b5220 2024-04-16T16:18:48.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.214+0000 7f6a71600640 1 -- 172.21.15.45:0/2137747556 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5186 (secure 0 0 0) 0x7f6a60058e50 con 0x7f6a740b5220 2024-04-16T16:18:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:48.214 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2514711815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2514711815},{"type":"v1","addr":"172.21.15.103:6836","nonce":2514711815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2189404408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2189404408},{"type":"v1","addr":"172.21.15.103:6837","nonce":2189404408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7179,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/991116998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":991116998},{"type":"v1","addr":"172.21.15.45:6834","nonce":991116998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:18:15.762294+0000","modified":"2024-04-16T16:18:18.858040+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7188":{"gid":7188,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3813781309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3813781309},{"type":"v1","addr":"172.21.15.45:6835","nonce":3813781309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7188,"qdb_cluster":[7188]},"id":6}]} 2024-04-16T16:18:48.214 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 64 2024-04-16T16:18:48.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.217+0000 7f6a7c200640 1 -- 172.21.15.45:0/2137747556 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6a5406c980 msgr2=0x7f6a5406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:48.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.217+0000 7f6a7c200640 1 --2- 172.21.15.45:0/2137747556 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6a5406c980 0x7f6a5406ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6a740b5b80 tx=0x7f6a68009040 comp rx=0 tx=0).stop 2024-04-16T16:18:48.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.217+0000 7f6a7c200640 1 -- 172.21.15.45:0/2137747556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a740b5220 msgr2=0x7f6a740cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:48.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.217+0000 7f6a7c200640 1 --2- 172.21.15.45:0/2137747556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a740b5220 0x7f6a740cf690 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f6a6000c700 tx=0x7f6a6000cbc0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.217+0000 7f6a7c200640 1 -- 172.21.15.45:0/2137747556 shutdown_connections 2024-04-16T16:18:48.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.217+0000 7f6a7c200640 1 --2- 172.21.15.45:0/2137747556 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a741e8450 0x7f6a741e88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.217+0000 7f6a7c200640 1 --2- 172.21.15.45:0/2137747556 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6a5406c980 0x7f6a5406ee40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.217+0000 7f6a7c200640 1 --2- 172.21.15.45:0/2137747556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a740b5220 0x7f6a740cf690 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.217+0000 7f6a7c200640 1 --2- 172.21.15.45:0/2137747556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a7415dc70 0x7f6a740cdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.217+0000 7f6a7c200640 1 -- 172.21.15.45:0/2137747556 >> 172.21.15.45:0/2137747556 conn(0x7f6a740b2750 msgr2=0x7f6a7415fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:48.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.218+0000 7f6a7c200640 1 -- 172.21.15.45:0/2137747556 shutdown_connections 2024-04-16T16:18:48.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.218+0000 7f6a7c200640 1 -- 172.21.15.45:0/2137747556 wait complete. 2024-04-16T16:18:48.236 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:18:48.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.456+0000 7fa48fe00640 1 Processor -- start 2024-04-16T16:18:48.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.456+0000 7fa48fe00640 1 -- start start 2024-04-16T16:18:48.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.458+0000 7fa48fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa490161e90 0x7fa4900cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:48.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.458+0000 7fa48fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa490162270 0x7fa4900d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:48.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.458+0000 7fa48fe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4900d3080 con 0x7fa4900d52d0 2024-04-16T16:18:48.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.458+0000 7fa48fe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa4900d2e30 con 0x7fa490161e90 2024-04-16T16:18:48.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.458+0000 7fa48fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa4900d3360 con 0x7fa490162270 2024-04-16T16:18:48.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.458+0000 7fa48ea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa490161e90 0x7fa4900cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:48.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.458+0000 7fa48f400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa4900d52d0 0x7fa49006ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54450/0 (socket says 172.21.15.45:54450) 2024-04-16T16:18:48.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.458+0000 7fa48f400640 1 -- 172.21.15.45:0/312306887 learned_addr learned my addr 172.21.15.45:0/312306887 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:48.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.458+0000 7fa48ea00640 1 --2- 172.21.15.45:0/312306887 >> v2:172.21.15.45:3300/0 conn(0x7fa490161e90 0x7fa4900cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51860/0 (socket says 172.21.15.45:51860) 2024-04-16T16:18:48.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.459+0000 7fa486000640 1 --2- 172.21.15.45:0/312306887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa490162270 0x7fa4900d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:48.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.459+0000 7fa48ea00640 1 -- 172.21.15.45:0/312306887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa490162270 msgr2=0x7fa4900d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:48.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.459+0000 7fa48ea00640 1 --2- 172.21.15.45:0/312306887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa490162270 0x7fa4900d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.459+0000 7fa48ea00640 1 -- 172.21.15.45:0/312306887 >> v1:172.21.15.45:6789/0 conn(0x7fa4900d52d0 legacy=0x7fa49006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:48.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.459+0000 7fa48e000640 1 -- 172.21.15.45:0/312306887 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1510168164 0 0) 0x7fa4900d3080 con 0x7fa4900d52d0 2024-04-16T16:18:48.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.459+0000 7fa48ea00640 1 -- 172.21.15.45:0/312306887 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4900d3640 con 0x7fa490161e90 2024-04-16T16:18:48.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.460+0000 7fa48ea00640 1 --2- 172.21.15.45:0/312306887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa490161e90 0x7fa4900cf7d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fa4800097a0 tx=0x7fa48002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49997436cd05c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:48.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.461+0000 7fa48e000640 1 -- 172.21.15.45:0/312306887 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480034070 con 0x7fa490161e90 2024-04-16T16:18:48.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.461+0000 7fa48e000640 1 -- 172.21.15.45:0/312306887 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa480030610 con 0x7fa490161e90 2024-04-16T16:18:48.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.461+0000 7fa48e000640 1 -- 172.21.15.45:0/312306887 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4800309e0 con 0x7fa490161e90 2024-04-16T16:18:48.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.461+0000 7fa48fe00640 1 -- 172.21.15.45:0/312306887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa490161e90 msgr2=0x7fa4900cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:48.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.461+0000 7fa48fe00640 1 --2- 172.21.15.45:0/312306887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa490161e90 0x7fa4900cf7d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fa4800097a0 tx=0x7fa48002b960 comp rx=0 tx=0).stop 2024-04-16T16:18:48.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.461+0000 7fa48fe00640 1 -- 172.21.15.45:0/312306887 shutdown_connections 2024-04-16T16:18:48.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.461+0000 7fa48fe00640 1 --2- 172.21.15.45:0/312306887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa490162270 0x7fa4900d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.461+0000 7fa48fe00640 1 --2- 172.21.15.45:0/312306887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa490161e90 0x7fa4900cf7d0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.461+0000 7fa48fe00640 1 -- 172.21.15.45:0/312306887 >> 172.21.15.45:0/312306887 conn(0x7fa4900d1940 msgr2=0x7fa4900d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:48.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.461+0000 7fa48fe00640 1 -- 172.21.15.45:0/312306887 shutdown_connections 2024-04-16T16:18:48.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.461+0000 7fa48fe00640 1 -- 172.21.15.45:0/312306887 wait complete. 2024-04-16T16:18:48.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.463+0000 7fa48fe00640 1 Processor -- start 2024-04-16T16:18:48.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.463+0000 7fa48fe00640 1 -- start start 2024-04-16T16:18:48.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.464+0000 7fa48fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4900d52d0 0x7fa49014cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:48.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.464+0000 7fa48fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa490161e90 0x7fa4900cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:48.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.464+0000 7fa48fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa490162270 0x7fa4900ce500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:48.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.464+0000 7fa48fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa49006d550 con 0x7fa490162270 2024-04-16T16:18:48.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.464+0000 7fa48fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa4900cfd10 con 0x7fa4900d52d0 2024-04-16T16:18:48.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.464+0000 7fa48fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa49006d6d0 con 0x7fa490161e90 2024-04-16T16:18:48.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.464+0000 7fa48f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa490162270 0x7fa4900ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:48.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.465+0000 7fa48f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa490162270 0x7fa4900ce500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51862/0 (socket says 172.21.15.45:51862) 2024-04-16T16:18:48.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.465+0000 7fa48f400640 1 -- 172.21.15.45:0/2609581507 learned_addr learned my addr 172.21.15.45:0/2609581507 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:48.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.465+0000 7fa486000640 1 --2- 172.21.15.45:0/2609581507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa490161e90 0x7fa4900cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:48.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.465+0000 7fa48ea00640 1 --2- 172.21.15.45:0/2609581507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4900d52d0 0x7fa49014cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:48.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.465+0000 7fa48f400640 1 -- 172.21.15.45:0/2609581507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa490161e90 msgr2=0x7fa4900cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:48.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.465+0000 7fa48f400640 1 --2- 172.21.15.45:0/2609581507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa490161e90 0x7fa4900cdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.465+0000 7fa48f400640 1 -- 172.21.15.45:0/2609581507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4900d52d0 msgr2=0x7fa49014cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:48.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.465+0000 7fa48f400640 1 --2- 172.21.15.45:0/2609581507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4900d52d0 0x7fa49014cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.465+0000 7fa48f400640 1 -- 172.21.15.45:0/2609581507 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4900cc620 con 0x7fa490162270 2024-04-16T16:18:48.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.465+0000 7fa48f400640 1 --2- 172.21.15.45:0/2609581507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa490162270 0x7fa4900ce500 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fa48800c7d0 tx=0x7fa48800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:48.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.466+0000 7fa487400640 1 -- 172.21.15.45:0/2609581507 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa488018070 con 0x7fa490162270 2024-04-16T16:18:48.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.466+0000 7fa487400640 1 -- 172.21.15.45:0/2609581507 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa48800a640 con 0x7fa490162270 2024-04-16T16:18:48.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.466+0000 7fa487400640 1 -- 172.21.15.45:0/2609581507 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48800aa70 con 0x7fa490162270 2024-04-16T16:18:48.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.466+0000 7fa48fe00640 1 -- 172.21.15.45:0/2609581507 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4900cc900 con 0x7fa490162270 2024-04-16T16:18:48.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.466+0000 7fa48fe00640 1 -- 172.21.15.45:0/2609581507 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4900ccec0 con 0x7fa490162270 2024-04-16T16:18:48.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.467+0000 7fa48fe00640 1 -- 172.21.15.45:0/2609581507 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa490075780 con 0x7fa490162270 2024-04-16T16:18:48.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.468+0000 7fa487400640 1 -- 172.21.15.45:0/2609581507 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7fa48801b070 con 0x7fa490162270 2024-04-16T16:18:48.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.469+0000 7fa487400640 1 --2- 172.21.15.45:0/2609581507 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fa46c06c980 0x7fa46c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:48.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.469+0000 7fa487400640 1 -- 172.21.15.45:0/2609581507 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6354+0+0 (secure 0 0 0) 0x7fa48808b620 con 0x7fa490162270 2024-04-16T16:18:48.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.470+0000 7fa48ea00640 1 --2- 172.21.15.45:0/2609581507 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fa46c06c980 0x7fa46c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:48.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.471+0000 7fa48ea00640 1 --2- 172.21.15.45:0/2609581507 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fa46c06c980 0x7fa46c06ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa480036000 tx=0x7fa48003b040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:48.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.474+0000 7fa487400640 1 -- 172.21.15.45:0/2609581507 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa488059640 con 0x7fa490162270 2024-04-16T16:18:48.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.685+0000 7fa48fe00640 1 -- 172.21.15.45:0/2609581507 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa4900d4bf0 con 0x7fa490162270 2024-04-16T16:18:48.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.687+0000 7fa487400640 1 -- 172.21.15.45:0/2609581507 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+17613 (secure 0 0 0) 0x7fa488058f40 con 0x7fa490162270 2024-04-16T16:18:48.687 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:48.687 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:18:43.993003+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:18:13.633261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-16T16:18:14.757055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6833/1293636998":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/2879899995":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/766691889":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/1187061250":"2024-04-17T16:18:43.992878+0000","172.21.15.103:6832/1293636998":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/4007272017":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/3489855481":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/674186192":"2024-04-17T16:18:37.658582+0000","172.21.15.103:6833/3728775368":"2024-04-17T16:18:37.658582+0000","172.21.15.103:6832/3728775368":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/4160076462":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/1325022358":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/981309315":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/1207000852":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/3673976854":"2024-04-17T16:18:37.658582+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:18:48.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.690+0000 7fa48fe00640 1 -- 172.21.15.45:0/2609581507 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fa46c06c980 msgr2=0x7fa46c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:48.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.690+0000 7fa48fe00640 1 --2- 172.21.15.45:0/2609581507 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fa46c06c980 0x7fa46c06ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa480036000 tx=0x7fa48003b040 comp rx=0 tx=0).stop 2024-04-16T16:18:48.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.690+0000 7fa48fe00640 1 -- 172.21.15.45:0/2609581507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa490162270 msgr2=0x7fa4900ce500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:48.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.690+0000 7fa48fe00640 1 --2- 172.21.15.45:0/2609581507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa490162270 0x7fa4900ce500 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fa48800c7d0 tx=0x7fa48800cc90 comp rx=0 tx=0).stop 2024-04-16T16:18:48.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.690+0000 7fa48fe00640 1 -- 172.21.15.45:0/2609581507 shutdown_connections 2024-04-16T16:18:48.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.690+0000 7fa48fe00640 1 --2- 172.21.15.45:0/2609581507 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fa46c06c980 0x7fa46c06ee40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.690+0000 7fa48fe00640 1 --2- 172.21.15.45:0/2609581507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa490162270 0x7fa4900ce500 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.690+0000 7fa48fe00640 1 --2- 172.21.15.45:0/2609581507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa490161e90 0x7fa4900cdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.690+0000 7fa48fe00640 1 --2- 172.21.15.45:0/2609581507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4900d52d0 0x7fa49014cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.690+0000 7fa48fe00640 1 -- 172.21.15.45:0/2609581507 >> 172.21.15.45:0/2609581507 conn(0x7fa4900d1940 msgr2=0x7fa490161130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:48.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.690+0000 7fa48fe00640 1 -- 172.21.15.45:0/2609581507 shutdown_connections 2024-04-16T16:18:48.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.690+0000 7fa48fe00640 1 -- 172.21.15.45:0/2609581507 wait complete. 2024-04-16T16:18:48.708 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-16T16:18:48.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:18:48.825 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:48.825+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:18:48.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.949+0000 7f4c7d000640 1 Processor -- start 2024-04-16T16:18:48.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.949+0000 7f4c7d000640 1 -- start start 2024-04-16T16:18:48.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.950+0000 7f4c7d000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4c780ccb20 0x7f4c780cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.950+0000 7f4c7d000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4c780ce9a0 0x7f4c780cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.950+0000 7f4c7d000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c7815ecd0 con 0x7f4c7815e8f0 2024-04-16T16:18:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.950+0000 7f4c7d000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f4c780ce2e0 con 0x7f4c780ce9a0 2024-04-16T16:18:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.950+0000 7f4c7d000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4c780cd6d0 con 0x7f4c780ccb20 2024-04-16T16:18:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.951+0000 7f4c77e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4c780ce9a0 0x7f4c780cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.951+0000 7f4c77400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4c7815e8f0 0x7f4c780cc410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54456/0 (socket says 172.21.15.45:54456) 2024-04-16T16:18:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.951+0000 7f4c77400640 1 -- 172.21.15.45:0/3333452306 learned_addr learned my addr 172.21.15.45:0/3333452306 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.951+0000 7f4c6fe00640 1 --2- 172.21.15.45:0/3333452306 >> v2:172.21.15.103:3300/0 conn(0x7f4c780ccb20 0x7f4c780cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.951+0000 7f4c77e00640 1 -- 172.21.15.45:0/3333452306 >> v2:172.21.15.103:3300/0 conn(0x7f4c780ccb20 msgr2=0x7f4c780cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.951+0000 7f4c77e00640 1 --2- 172.21.15.45:0/3333452306 >> v2:172.21.15.103:3300/0 conn(0x7f4c780ccb20 0x7f4c780cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.951+0000 7f4c77e00640 1 -- 172.21.15.45:0/3333452306 >> v1:172.21.15.45:6789/0 conn(0x7f4c7815e8f0 legacy=0x7f4c780cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:48.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.951+0000 7f4c77e00640 1 -- 172.21.15.45:0/3333452306 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c780cd9b0 con 0x7f4c780ce9a0 2024-04-16T16:18:48.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.953+0000 7f4c77e00640 1 --2- 172.21.15.45:0/3333452306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c780ce9a0 0x7f4c780cb170 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f4c78070330 tx=0x7f4c7002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df5626cda4cecf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:48.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.953+0000 7f4c76a00640 1 -- 172.21.15.45:0/3333452306 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c70039070 con 0x7f4c780ce9a0 2024-04-16T16:18:48.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.953+0000 7f4c76a00640 1 -- 172.21.15.45:0/3333452306 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c7002cbe0 con 0x7f4c780ce9a0 2024-04-16T16:18:48.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.953+0000 7f4c76a00640 1 -- 172.21.15.45:0/3333452306 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c70034440 con 0x7f4c780ce9a0 2024-04-16T16:18:48.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.953+0000 7f4c7d000640 1 -- 172.21.15.45:0/3333452306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c780ce9a0 msgr2=0x7f4c780cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:48.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.953+0000 7f4c7d000640 1 --2- 172.21.15.45:0/3333452306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c780ce9a0 0x7f4c780cb170 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f4c78070330 tx=0x7f4c7002be30 comp rx=0 tx=0).stop 2024-04-16T16:18:48.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.954+0000 7f4c7d000640 1 -- 172.21.15.45:0/3333452306 shutdown_connections 2024-04-16T16:18:48.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.954+0000 7f4c7d000640 1 --2- 172.21.15.45:0/3333452306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c780ce9a0 0x7f4c780cb170 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.954+0000 7f4c7d000640 1 --2- 172.21.15.45:0/3333452306 >> v2:172.21.15.103:3300/0 conn(0x7f4c780ccb20 0x7f4c780cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.954+0000 7f4c7d000640 1 -- 172.21.15.45:0/3333452306 >> 172.21.15.45:0/3333452306 conn(0x7f4c7806b930 msgr2=0x7f4c7806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:48.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.954+0000 7f4c7d000640 1 -- 172.21.15.45:0/3333452306 shutdown_connections 2024-04-16T16:18:48.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.954+0000 7f4c7d000640 1 -- 172.21.15.45:0/3333452306 wait complete. 2024-04-16T16:18:48.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.956+0000 7f4c7d000640 1 Processor -- start 2024-04-16T16:18:48.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.956+0000 7f4c7d000640 1 -- start start 2024-04-16T16:18:48.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.957+0000 7f4c7d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c780ccb20 0x7f4c780b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:48.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.957+0000 7f4c7d000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c780ce9a0 0x7f4c78142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:48.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.957+0000 7f4c7d000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c7815e8f0 0x7f4c78156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:48.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.957+0000 7f4c7d000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4c780cd360 con 0x7f4c780ccb20 2024-04-16T16:18:48.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.957+0000 7f4c7d000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4c780cdf70 con 0x7f4c7815e8f0 2024-04-16T16:18:48.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.957+0000 7f4c7d000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4c780cf080 con 0x7f4c780ce9a0 2024-04-16T16:18:48.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.957+0000 7f4c77400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c780ccb20 0x7f4c780b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:48.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.957+0000 7f4c77400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c780ccb20 0x7f4c780b4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51890/0 (socket says 172.21.15.45:51890) 2024-04-16T16:18:48.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.957+0000 7f4c77400640 1 -- 172.21.15.45:0/350629449 learned_addr learned my addr 172.21.15.45:0/350629449 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:48.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.958+0000 7f4c77e00640 1 --2- 172.21.15.45:0/350629449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c7815e8f0 0x7f4c78156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:48.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.958+0000 7f4c77400640 1 -- 172.21.15.45:0/350629449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c780ce9a0 msgr2=0x7f4c78142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:48.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.958+0000 7f4c77400640 1 --2- 172.21.15.45:0/350629449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c780ce9a0 0x7f4c78142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.958+0000 7f4c77400640 1 -- 172.21.15.45:0/350629449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c7815e8f0 msgr2=0x7f4c78156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:48.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.958+0000 7f4c77400640 1 --2- 172.21.15.45:0/350629449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c7815e8f0 0x7f4c78156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:48.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.958+0000 7f4c77400640 1 -- 172.21.15.45:0/350629449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c780b5860 con 0x7f4c780ccb20 2024-04-16T16:18:48.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.958+0000 7f4c77e00640 1 --2- 172.21.15.45:0/350629449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c7815e8f0 0x7f4c78156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:48.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.958+0000 7f4c77400640 1 --2- 172.21.15.45:0/350629449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c780ccb20 0x7f4c780b4f30 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f4c6800c3f0 tx=0x7f4c6800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:48.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.958+0000 7f4c6f400640 1 -- 172.21.15.45:0/350629449 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c68017070 con 0x7f4c780ccb20 2024-04-16T16:18:48.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.958+0000 7f4c6f400640 1 -- 172.21.15.45:0/350629449 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c68005ce0 con 0x7f4c780ccb20 2024-04-16T16:18:48.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.958+0000 7f4c6f400640 1 -- 172.21.15.45:0/350629449 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c680072f0 con 0x7f4c780ccb20 2024-04-16T16:18:48.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.958+0000 7f4c7d000640 1 -- 172.21.15.45:0/350629449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c780b5580 con 0x7f4c780ccb20 2024-04-16T16:18:48.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.959+0000 7f4c7d000640 1 -- 172.21.15.45:0/350629449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c78160ee0 con 0x7f4c780ccb20 2024-04-16T16:18:48.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.960+0000 7f4c7d000640 1 -- 172.21.15.45:0/350629449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c78075780 con 0x7f4c780ccb20 2024-04-16T16:18:48.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.961+0000 7f4c6f400640 1 -- 172.21.15.45:0/350629449 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f4c6801a070 con 0x7f4c780ccb20 2024-04-16T16:18:48.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.962+0000 7f4c6f400640 1 --2- 172.21.15.45:0/350629449 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f4c5406c9b0 0x7f4c5406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:48.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.962+0000 7f4c6f400640 1 -- 172.21.15.45:0/350629449 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6354+0+0 (secure 0 0 0) 0x7f4c6808b290 con 0x7f4c780ccb20 2024-04-16T16:18:48.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.963+0000 7f4c6fe00640 1 --2- 172.21.15.45:0/350629449 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f4c5406c9b0 0x7f4c5406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:48.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.964+0000 7f4c6fe00640 1 --2- 172.21.15.45:0/350629449 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f4c5406c9b0 0x7f4c5406ee70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4c780cf5c0 tx=0x7f4c60008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:48.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:48.967+0000 7f4c6f400640 1 -- 172.21.15.45:0/350629449 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4c68059340 con 0x7f4c780ccb20 2024-04-16T16:18:49.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.225+0000 7f4c7d000640 1 -- 172.21.15.45:0/350629449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c780b2b80 con 0x7f4c780ccb20 2024-04-16T16:18:49.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.226+0000 7f4c6f400640 1 -- 172.21.15.45:0/350629449 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5186 (secure 0 0 0) 0x7f4c68058c40 con 0x7f4c780ccb20 2024-04-16T16:18:49.226 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:49.226 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2514711815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2514711815},{"type":"v1","addr":"172.21.15.103:6836","nonce":2514711815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2189404408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2189404408},{"type":"v1","addr":"172.21.15.103:6837","nonce":2189404408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7179,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/991116998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":991116998},{"type":"v1","addr":"172.21.15.45:6834","nonce":991116998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:18:15.762294+0000","modified":"2024-04-16T16:18:18.858040+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7188":{"gid":7188,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3813781309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3813781309},{"type":"v1","addr":"172.21.15.45:6835","nonce":3813781309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7188,"qdb_cluster":[7188]},"id":6}]} 2024-04-16T16:18:49.226 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 64 2024-04-16T16:18:49.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.229+0000 7f4c7d000640 1 -- 172.21.15.45:0/350629449 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f4c5406c9b0 msgr2=0x7f4c5406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:49.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.229+0000 7f4c7d000640 1 --2- 172.21.15.45:0/350629449 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f4c5406c9b0 0x7f4c5406ee70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4c780cf5c0 tx=0x7f4c60008040 comp rx=0 tx=0).stop 2024-04-16T16:18:49.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.229+0000 7f4c7d000640 1 -- 172.21.15.45:0/350629449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c780ccb20 msgr2=0x7f4c780b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:49.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.229+0000 7f4c7d000640 1 --2- 172.21.15.45:0/350629449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c780ccb20 0x7f4c780b4f30 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f4c6800c3f0 tx=0x7f4c6800c8b0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.229+0000 7f4c7d000640 1 -- 172.21.15.45:0/350629449 shutdown_connections 2024-04-16T16:18:49.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.229+0000 7f4c7d000640 1 --2- 172.21.15.45:0/350629449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c7815e8f0 0x7f4c78156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.229+0000 7f4c7d000640 1 --2- 172.21.15.45:0/350629449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c780ce9a0 0x7f4c78142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.229+0000 7f4c7d000640 1 --2- 172.21.15.45:0/350629449 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f4c5406c9b0 0x7f4c5406ee70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.229+0000 7f4c7d000640 1 --2- 172.21.15.45:0/350629449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c780ccb20 0x7f4c780b4f30 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.230+0000 7f4c7d000640 1 -- 172.21.15.45:0/350629449 >> 172.21.15.45:0/350629449 conn(0x7f4c7806b930 msgr2=0x7f4c780cb620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:49.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.230+0000 7f4c7d000640 1 -- 172.21.15.45:0/350629449 shutdown_connections 2024-04-16T16:18:49.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.230+0000 7f4c7d000640 1 -- 172.21.15.45:0/350629449 wait complete. 2024-04-16T16:18:49.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:18:49.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.476+0000 7fed5ca00640 1 Processor -- start 2024-04-16T16:18:49.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.476+0000 7fed5ca00640 1 -- start start 2024-04-16T16:18:49.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.477+0000 7fed5ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed58161e90 0x7fed580cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:49.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.477+0000 7fed5ca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fed58162270 0x7fed580d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.477+0000 7fed5ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fed580d52d0 0x7fed5806ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.477+0000 7fed5ca00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fed580d2e30 con 0x7fed580d52d0 2024-04-16T16:18:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.477+0000 7fed5ca00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fed580d3080 con 0x7fed58162270 2024-04-16T16:18:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.477+0000 7fed5ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fed580d3360 con 0x7fed58161e90 2024-04-16T16:18:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.478+0000 7fed57200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fed580d52d0 0x7fed5806ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.478+0000 7fed57200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fed580d52d0 0x7fed5806ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51892/0 (socket says 172.21.15.45:51892) 2024-04-16T16:18:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.478+0000 7fed57200640 1 -- 172.21.15.45:0/905191205 learned_addr learned my addr 172.21.15.45:0/905191205 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.478+0000 7fed56800640 1 --2- 172.21.15.45:0/905191205 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed58161e90 0x7fed580cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.478+0000 7fed55e00640 1 --2- 172.21.15.45:0/905191205 >> v2:172.21.15.103:3300/0 conn(0x7fed58162270 0x7fed580d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:49.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.478+0000 7fed57200640 1 -- 172.21.15.45:0/905191205 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed58161e90 msgr2=0x7fed580cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:49.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.478+0000 7fed57200640 1 --2- 172.21.15.45:0/905191205 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed58161e90 0x7fed580cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.478+0000 7fed57200640 1 -- 172.21.15.45:0/905191205 >> v2:172.21.15.103:3300/0 conn(0x7fed58162270 msgr2=0x7fed580d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:49.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.478+0000 7fed57200640 1 --2- 172.21.15.45:0/905191205 >> v2:172.21.15.103:3300/0 conn(0x7fed58162270 0x7fed580d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.478+0000 7fed57200640 1 -- 172.21.15.45:0/905191205 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed580d3640 con 0x7fed580d52d0 2024-04-16T16:18:49.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.479+0000 7fed57200640 1 --2- 172.21.15.45:0/905191205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed580d52d0 0x7fed5806ce40 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fed5808d360 tx=0x7fed5002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a78be3fd042cfd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:49.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.480+0000 7fed55400640 1 -- 172.21.15.45:0/905191205 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5002cb00 con 0x7fed580d52d0 2024-04-16T16:18:49.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.480+0000 7fed55400640 1 -- 172.21.15.45:0/905191205 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed5002cca0 con 0x7fed580d52d0 2024-04-16T16:18:49.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.480+0000 7fed55400640 1 -- 172.21.15.45:0/905191205 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50034610 con 0x7fed580d52d0 2024-04-16T16:18:49.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.480+0000 7fed5ca00640 1 -- 172.21.15.45:0/905191205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed580d52d0 msgr2=0x7fed5806ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:49.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.480+0000 7fed5ca00640 1 --2- 172.21.15.45:0/905191205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed580d52d0 0x7fed5806ce40 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fed5808d360 tx=0x7fed5002bf60 comp rx=0 tx=0).stop 2024-04-16T16:18:49.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.480+0000 7fed5ca00640 1 -- 172.21.15.45:0/905191205 shutdown_connections 2024-04-16T16:18:49.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.481+0000 7fed5ca00640 1 --2- 172.21.15.45:0/905191205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed580d52d0 0x7fed5806ce40 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.481+0000 7fed5ca00640 1 --2- 172.21.15.45:0/905191205 >> v2:172.21.15.103:3300/0 conn(0x7fed58162270 0x7fed580d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.481+0000 7fed5ca00640 1 --2- 172.21.15.45:0/905191205 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed58161e90 0x7fed580cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.481+0000 7fed5ca00640 1 -- 172.21.15.45:0/905191205 >> 172.21.15.45:0/905191205 conn(0x7fed580d1940 msgr2=0x7fed580d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:49.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.481+0000 7fed5ca00640 1 -- 172.21.15.45:0/905191205 shutdown_connections 2024-04-16T16:18:49.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.481+0000 7fed5ca00640 1 -- 172.21.15.45:0/905191205 wait complete. 2024-04-16T16:18:49.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.482+0000 7fed5ca00640 1 Processor -- start 2024-04-16T16:18:49.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.483+0000 7fed5ca00640 1 -- start start 2024-04-16T16:18:49.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.483+0000 7fed5ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed580d52d0 0x7fed5815bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:49.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.483+0000 7fed5ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed58161e90 0x7fed5814ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:49.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed5ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed58162270 0x7fed5814d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:49.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed5ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fed580d4380 con 0x7fed58161e90 2024-04-16T16:18:49.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed5ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fed580cfd10 con 0x7fed580d52d0 2024-04-16T16:18:49.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed5ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fed580d4500 con 0x7fed58162270 2024-04-16T16:18:49.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed55e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed58161e90 0x7fed5814ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:49.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed55e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed58161e90 0x7fed5814ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51904/0 (socket says 172.21.15.45:51904) 2024-04-16T16:18:49.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed55e00640 1 -- 172.21.15.45:0/3845019724 learned_addr learned my addr 172.21.15.45:0/3845019724 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:49.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed57200640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed58162270 0x7fed5814d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:49.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed56800640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed580d52d0 0x7fed5815bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:49.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed55e00640 1 -- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed58162270 msgr2=0x7fed5814d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:49.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed55e00640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed58162270 0x7fed5814d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed55e00640 1 -- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed580d52d0 msgr2=0x7fed5815bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:49.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed55e00640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed580d52d0 0x7fed5815bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed55e00640 1 -- 172.21.15.45:0/3845019724 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5814b9b0 con 0x7fed58161e90 2024-04-16T16:18:49.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed56800640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed580d52d0 0x7fed5815bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:49.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.484+0000 7fed57200640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed58162270 0x7fed5814d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:49.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.485+0000 7fed55e00640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed58161e90 0x7fed5814ce60 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fed4400d5f0 tx=0x7fed4400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:49.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.485+0000 7fed4ea00640 1 -- 172.21.15.45:0/3845019724 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed440125f0 con 0x7fed58161e90 2024-04-16T16:18:49.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.485+0000 7fed4ea00640 1 -- 172.21.15.45:0/3845019724 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed44019070 con 0x7fed58161e90 2024-04-16T16:18:49.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.485+0000 7fed4ea00640 1 -- 172.21.15.45:0/3845019724 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed440129d0 con 0x7fed58161e90 2024-04-16T16:18:49.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.485+0000 7fed5ca00640 1 -- 172.21.15.45:0/3845019724 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed5814bc90 con 0x7fed58161e90 2024-04-16T16:18:49.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.485+0000 7fed5ca00640 1 -- 172.21.15.45:0/3845019724 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed5814c250 con 0x7fed58161e90 2024-04-16T16:18:49.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.486+0000 7fed5ca00640 1 -- 172.21.15.45:0/3845019724 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed580b3650 con 0x7fed58161e90 2024-04-16T16:18:49.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.487+0000 7fed4ea00640 1 -- 172.21.15.45:0/3845019724 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7fed44012b70 con 0x7fed58161e90 2024-04-16T16:18:49.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.488+0000 7fed4ea00640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fed3806c980 0x7fed3806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:49.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.488+0000 7fed4ea00640 1 -- 172.21.15.45:0/3845019724 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6354+0+0 (secure 0 0 0) 0x7fed4408c220 con 0x7fed58161e90 2024-04-16T16:18:49.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.489+0000 7fed56800640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fed3806c980 0x7fed3806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:49.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.490+0000 7fed56800640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fed3806c980 0x7fed3806ee40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fed48006fd0 tx=0x7fed48008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:49.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.493+0000 7fed4ea00640 1 -- 172.21.15.45:0/3845019724 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fed580b3650 con 0x7fed58161e90 2024-04-16T16:18:49.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.743+0000 7fed5ca00640 1 -- 172.21.15.45:0/3845019724 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed58075780 con 0x7fed58161e90 2024-04-16T16:18:49.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.744+0000 7fed4ea00640 1 -- 172.21.15.45:0/3845019724 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5186 (secure 0 0 0) 0x7fed4405a360 con 0x7fed58161e90 2024-04-16T16:18:49.744 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:49.744 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2514711815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2514711815},{"type":"v1","addr":"172.21.15.103:6836","nonce":2514711815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2189404408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2189404408},{"type":"v1","addr":"172.21.15.103:6837","nonce":2189404408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7179,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/991116998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":991116998},{"type":"v1","addr":"172.21.15.45:6834","nonce":991116998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:18:15.762294+0000","modified":"2024-04-16T16:18:18.858040+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7188":{"gid":7188,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3813781309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3813781309},{"type":"v1","addr":"172.21.15.45:6835","nonce":3813781309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7188,"qdb_cluster":[7188]},"id":6}]} 2024-04-16T16:18:49.744 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 64 2024-04-16T16:18:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.747+0000 7fed5ca00640 1 -- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fed3806c980 msgr2=0x7fed3806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.747+0000 7fed5ca00640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fed3806c980 0x7fed3806ee40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fed48006fd0 tx=0x7fed48008040 comp rx=0 tx=0).stop 2024-04-16T16:18:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.747+0000 7fed5ca00640 1 -- 172.21.15.45:0/3845019724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed58161e90 msgr2=0x7fed5814ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.747+0000 7fed5ca00640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed58161e90 0x7fed5814ce60 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fed4400d5f0 tx=0x7fed4400dab0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.747+0000 7fed5ca00640 1 -- 172.21.15.45:0/3845019724 shutdown_connections 2024-04-16T16:18:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.747+0000 7fed5ca00640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed58162270 0x7fed5814d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.747+0000 7fed5ca00640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fed3806c980 0x7fed3806ee40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.747+0000 7fed5ca00640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed58161e90 0x7fed5814ce60 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.747+0000 7fed5ca00640 1 --2- 172.21.15.45:0/3845019724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed580d52d0 0x7fed5815bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.747+0000 7fed5ca00640 1 -- 172.21.15.45:0/3845019724 >> 172.21.15.45:0/3845019724 conn(0x7fed580d1940 msgr2=0x7fed580b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.747+0000 7fed5ca00640 1 -- 172.21.15.45:0/3845019724 shutdown_connections 2024-04-16T16:18:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.747+0000 7fed5ca00640 1 -- 172.21.15.45:0/3845019724 wait complete. 2024-04-16T16:18:49.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:18:49.781 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:49.780+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:18:49.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.992+0000 7f61c9a00640 1 Processor -- start 2024-04-16T16:18:49.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.992+0000 7f61c9a00640 1 -- start start 2024-04-16T16:18:49.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.993+0000 7f61c9a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c40d0e10 0x7f61c4167cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:49.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.993+0000 7f61c9a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61c40d12a0 con 0x7f61c40d0980 2024-04-16T16:18:49.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.993+0000 7f61c9a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61c406c040 con 0x7f61c4146350 2024-04-16T16:18:49.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.993+0000 7f61c9a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f61c40cf2a0 con 0x7f61c40d0e10 2024-04-16T16:18:49.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.994+0000 7f61c2000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f61c40d0980 0x7f61c4164590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54464/0 (socket says 172.21.15.45:54464) 2024-04-16T16:18:49.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.994+0000 7f61c2000640 1 -- 172.21.15.45:0/1142281450 learned_addr learned my addr 172.21.15.45:0/1142281450 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:49.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.994+0000 7f61c1600640 1 -- 172.21.15.45:0/1142281450 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2368458171 0 0) 0x7f61c40d12a0 con 0x7f61c40d0980 2024-04-16T16:18:49.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.994+0000 7f61c1600640 1 -- 172.21.15.45:0/1142281450 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61a8003520 con 0x7f61c40d0980 2024-04-16T16:18:49.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.994+0000 7f61c3400640 1 --2- 172.21.15.45:0/1142281450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c40d0e10 0x7f61c4167cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:49.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.995+0000 7f61c1600640 1 -- 172.21.15.45:0/1142281450 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1750739133 0 0) 0x7f61a8003520 con 0x7f61c40d0980 2024-04-16T16:18:49.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.995+0000 7f61c1600640 1 -- 172.21.15.45:0/1142281450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c40d0e10 msgr2=0x7f61c4167cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:49.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.995+0000 7f61c1600640 1 --2- 172.21.15.45:0/1142281450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c40d0e10 0x7f61c4167cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.995+0000 7f61c1600640 1 -- 172.21.15.45:0/1142281450 >> v1:172.21.15.103:6789/0 conn(0x7f61c4146350 legacy=0x7f61c406b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:49.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.995+0000 7f61c1600640 1 -- 172.21.15.45:0/1142281450 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c406c310 con 0x7f61c40d0980 2024-04-16T16:18:49.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.995+0000 7f61c1600640 1 -- 172.21.15.45:0/1142281450 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f61b0001690 con 0x7f61c40d0980 2024-04-16T16:18:49.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.995+0000 7f61c1600640 1 -- 172.21.15.45:0/1142281450 >> v1:172.21.15.45:6789/0 conn(0x7f61c40d0980 legacy=0x7f61c4164590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:49.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.995+0000 7f61c1600640 1 --2- 172.21.15.45:0/1142281450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61a80039e0 0x7f61a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:49.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.995+0000 7f61c1600640 1 --2- 172.21.15.45:0/1142281450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c40d0e10 0x7f61a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:49.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.995+0000 7f61c1600640 1 --2- 172.21.15.45:0/1142281450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a8004d40 0x7f61a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:49.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.995+0000 7f61c1600640 1 -- 172.21.15.45:0/1142281450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f61a800bb50 con 0x7f61a8004d40 2024-04-16T16:18:49.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.995+0000 7f61c1600640 1 -- 172.21.15.45:0/1142281450 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f61a800bcd0 con 0x7f61a80039e0 2024-04-16T16:18:49.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.995+0000 7f61c1600640 1 -- 172.21.15.45:0/1142281450 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f61a800bea0 con 0x7f61c40d0e10 2024-04-16T16:18:49.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.995+0000 7f61c3400640 1 --2- 172.21.15.45:0/1142281450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a8004d40 0x7f61a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:49.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.996+0000 7f61c2000640 1 --2- 172.21.15.45:0/1142281450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c40d0e10 0x7f61a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:49.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.996+0000 7f61c2a00640 1 --2- 172.21.15.45:0/1142281450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61a80039e0 0x7f61a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:49.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.996+0000 7f61c3400640 1 -- 172.21.15.45:0/1142281450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c40d0e10 msgr2=0x7f61a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:49.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.996+0000 7f61c3400640 1 --2- 172.21.15.45:0/1142281450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c40d0e10 0x7f61a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.996+0000 7f61c3400640 1 -- 172.21.15.45:0/1142281450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61a80039e0 msgr2=0x7f61a8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:49.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.996+0000 7f61c3400640 1 --2- 172.21.15.45:0/1142281450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61a80039e0 0x7f61a8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:49.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.996+0000 7f61c3400640 1 -- 172.21.15.45:0/1142281450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a8003520 con 0x7f61a8004d40 2024-04-16T16:18:49.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.997+0000 7f61c3400640 1 --2- 172.21.15.45:0/1142281450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a8004d40 0x7f61a80051c0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f61bc007390 tx=0x7f61a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307535c17c644370 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:49.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.997+0000 7f61c1600640 1 -- 172.21.15.45:0/1142281450 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61bc039070 con 0x7f61a8004d40 2024-04-16T16:18:49.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.997+0000 7f61c1600640 1 -- 172.21.15.45:0/1142281450 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61bc02ccb0 con 0x7f61a8004d40 2024-04-16T16:18:49.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.997+0000 7f61c1600640 1 -- 172.21.15.45:0/1142281450 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61bc009be0 con 0x7f61a8004d40 2024-04-16T16:18:50.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.998+0000 7f61c9a00640 1 -- 172.21.15.45:0/1142281450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a8004d40 msgr2=0x7f61a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:50.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.998+0000 7f61c9a00640 1 --2- 172.21.15.45:0/1142281450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a8004d40 0x7f61a80051c0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f61bc007390 tx=0x7f61a8005830 comp rx=0 tx=0).stop 2024-04-16T16:18:50.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.998+0000 7f61c3400640 1 -- 172.21.15.45:0/1142281450 reap_dead start 2024-04-16T16:18:50.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.998+0000 7f61c9a00640 1 -- 172.21.15.45:0/1142281450 shutdown_connections 2024-04-16T16:18:50.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.998+0000 7f61c9a00640 1 -- 172.21.15.45:0/1142281450 >> 172.21.15.45:0/1142281450 conn(0x7f61c409aa70 msgr2=0x7f61c40b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:50.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.998+0000 7f61c9a00640 1 -- 172.21.15.45:0/1142281450 shutdown_connections 2024-04-16T16:18:50.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:49.998+0000 7f61c9a00640 1 -- 172.21.15.45:0/1142281450 wait complete. 2024-04-16T16:18:50.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.000+0000 7f61c9a00640 1 Processor -- start 2024-04-16T16:18:50.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.000+0000 7f61c9a00640 1 -- start start 2024-04-16T16:18:50.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.001+0000 7f61c9a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a8004d40 0x7f61c40b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:50.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.001+0000 7f61c9a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c4134df0 0x7f61c40b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:50.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.001+0000 7f61c9a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61c40b6f90 0x7f61c40ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:50.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.001+0000 7f61c9a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f61c415d5b0 con 0x7f61a8004d40 2024-04-16T16:18:50.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.001+0000 7f61c9a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f61c415d920 con 0x7f61c40b6f90 2024-04-16T16:18:50.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.001+0000 7f61c9a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f61c415cf80 con 0x7f61c4134df0 2024-04-16T16:18:50.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.001+0000 7f61c2a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a8004d40 0x7f61c40b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:50.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.001+0000 7f61c2a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a8004d40 0x7f61c40b4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51926/0 (socket says 172.21.15.45:51926) 2024-04-16T16:18:50.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.001+0000 7f61c2a00640 1 -- 172.21.15.45:0/3764541399 learned_addr learned my addr 172.21.15.45:0/3764541399 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:50.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.002+0000 7f61c3400640 1 --2- 172.21.15.45:0/3764541399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61c40b6f90 0x7f61c40ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:50.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.002+0000 7f61c2000640 1 --2- 172.21.15.45:0/3764541399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c4134df0 0x7f61c40b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:50.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.002+0000 7f61c2a00640 1 -- 172.21.15.45:0/3764541399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c4134df0 msgr2=0x7f61c40b6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:50.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.002+0000 7f61c2a00640 1 --2- 172.21.15.45:0/3764541399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c4134df0 0x7f61c40b6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.002+0000 7f61c2a00640 1 -- 172.21.15.45:0/3764541399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61c40b6f90 msgr2=0x7f61c40ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:50.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.002+0000 7f61c2a00640 1 --2- 172.21.15.45:0/3764541399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61c40b6f90 0x7f61c40ca9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.002+0000 7f61c2a00640 1 -- 172.21.15.45:0/3764541399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c41f45b0 con 0x7f61a8004d40 2024-04-16T16:18:50.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.002+0000 7f61c2000640 1 --2- 172.21.15.45:0/3764541399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c4134df0 0x7f61c40b6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:50.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.002+0000 7f61c2a00640 1 --2- 172.21.15.45:0/3764541399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a8004d40 0x7f61c40b4ef0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f61b400e550 tx=0x7f61b400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:50.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.003+0000 7f61baa00640 1 -- 172.21.15.45:0/3764541399 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b40052e0 con 0x7f61a8004d40 2024-04-16T16:18:50.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.003+0000 7f61baa00640 1 -- 172.21.15.45:0/3764541399 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61b4018070 con 0x7f61a8004d40 2024-04-16T16:18:50.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.003+0000 7f61baa00640 1 -- 172.21.15.45:0/3764541399 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4003810 con 0x7f61a8004d40 2024-04-16T16:18:50.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.003+0000 7f61c9a00640 1 -- 172.21.15.45:0/3764541399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61c41f4890 con 0x7f61a8004d40 2024-04-16T16:18:50.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.003+0000 7f61c9a00640 1 -- 172.21.15.45:0/3764541399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61c41f4e50 con 0x7f61a8004d40 2024-04-16T16:18:50.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.004+0000 7f61c9a00640 1 -- 172.21.15.45:0/3764541399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61c4075780 con 0x7f61a8004d40 2024-04-16T16:18:50.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.005+0000 7f61baa00640 1 -- 172.21.15.45:0/3764541399 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f61b40039b0 con 0x7f61a8004d40 2024-04-16T16:18:50.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.006+0000 7f61baa00640 1 --2- 172.21.15.45:0/3764541399 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f61a406c9b0 0x7f61a406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:50.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.006+0000 7f61baa00640 1 -- 172.21.15.45:0/3764541399 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6354+0+0 (secure 0 0 0) 0x7f61b408ba70 con 0x7f61a8004d40 2024-04-16T16:18:50.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.007+0000 7f61c2000640 1 --2- 172.21.15.45:0/3764541399 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f61a406c9b0 0x7f61a406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:50.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.008+0000 7f61c2000640 1 --2- 172.21.15.45:0/3764541399 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f61a406c9b0 0x7f61a406ee70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f61b0001220 tx=0x7f61b0001150 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:50.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.010+0000 7f61baa00640 1 -- 172.21.15.45:0/3764541399 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f61b4059bb0 con 0x7f61a8004d40 2024-04-16T16:18:50.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.224+0000 7f61c9a00640 1 -- 172.21.15.45:0/3764541399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f61c40b7370 con 0x7f61a8004d40 2024-04-16T16:18:50.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.228+0000 7f61baa00640 1 -- 172.21.15.45:0/3764541399 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+17613 (secure 0 0 0) 0x7f61b40594b0 con 0x7f61a8004d40 2024-04-16T16:18:50.228 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:50.228 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:18:43.993003+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:18:13.633261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-16T16:18:14.757055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6833/1293636998":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/2879899995":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/766691889":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/1187061250":"2024-04-17T16:18:43.992878+0000","172.21.15.103:6832/1293636998":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/4007272017":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/3489855481":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/674186192":"2024-04-17T16:18:37.658582+0000","172.21.15.103:6833/3728775368":"2024-04-17T16:18:37.658582+0000","172.21.15.103:6832/3728775368":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/4160076462":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/1325022358":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/981309315":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/1207000852":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/3673976854":"2024-04-17T16:18:37.658582+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:18:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.231+0000 7f61c9a00640 1 -- 172.21.15.45:0/3764541399 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f61a406c9b0 msgr2=0x7f61a406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.231+0000 7f61c9a00640 1 --2- 172.21.15.45:0/3764541399 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f61a406c9b0 0x7f61a406ee70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f61b0001220 tx=0x7f61b0001150 comp rx=0 tx=0).stop 2024-04-16T16:18:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.231+0000 7f61c9a00640 1 -- 172.21.15.45:0/3764541399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a8004d40 msgr2=0x7f61c40b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.231+0000 7f61c9a00640 1 --2- 172.21.15.45:0/3764541399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a8004d40 0x7f61c40b4ef0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f61b400e550 tx=0x7f61b400ea10 comp rx=0 tx=0).stop 2024-04-16T16:18:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.231+0000 7f61c9a00640 1 -- 172.21.15.45:0/3764541399 shutdown_connections 2024-04-16T16:18:50.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.231+0000 7f61c9a00640 1 --2- 172.21.15.45:0/3764541399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61c40b6f90 0x7f61c40ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.231+0000 7f61c9a00640 1 --2- 172.21.15.45:0/3764541399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c4134df0 0x7f61c40b6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.231+0000 7f61c9a00640 1 --2- 172.21.15.45:0/3764541399 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f61a406c9b0 0x7f61a406ee70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.231+0000 7f61c9a00640 1 --2- 172.21.15.45:0/3764541399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a8004d40 0x7f61c40b4ef0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.231+0000 7f61c9a00640 1 -- 172.21.15.45:0/3764541399 >> 172.21.15.45:0/3764541399 conn(0x7f61c409aa70 msgr2=0x7f61c40b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:50.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.232+0000 7f61c9a00640 1 -- 172.21.15.45:0/3764541399 shutdown_connections 2024-04-16T16:18:50.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.232+0000 7f61c9a00640 1 -- 172.21.15.45:0/3764541399 wait complete. 2024-04-16T16:18:50.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-16T16:18:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.481+0000 7f4e45e00640 1 Processor -- start 2024-04-16T16:18:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.482+0000 7f4e45e00640 1 -- start start 2024-04-16T16:18:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.483+0000 7f4e45e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4e40145230 0x7f4e4015bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.483+0000 7f4e45e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4e400c5670 0x7f4e400c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.483+0000 7f4e45e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e4015d3b0 con 0x7f4e4015cbe0 2024-04-16T16:18:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.483+0000 7f4e45e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f4e4015dfc0 con 0x7f4e400c5670 2024-04-16T16:18:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.483+0000 7f4e45e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4e400c5fe0 con 0x7f4e40145230 2024-04-16T16:18:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.484+0000 7f4e45400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4e400c5670 0x7f4e400c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.484+0000 7f4e45400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4e400c5670 0x7f4e400c5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51932/0 (socket says 172.21.15.45:51932) 2024-04-16T16:18:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.484+0000 7f4e45400640 1 -- 172.21.15.45:0/1250832894 learned_addr learned my addr 172.21.15.45:0/1250832894 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.484+0000 7f4e44a00640 1 --2- 172.21.15.45:0/1250832894 >> v2:172.21.15.103:3300/0 conn(0x7f4e40145230 0x7f4e4015bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.485+0000 7f4e45400640 1 -- 172.21.15.45:0/1250832894 >> v2:172.21.15.103:3300/0 conn(0x7f4e40145230 msgr2=0x7f4e4015bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.485+0000 7f4e45400640 1 --2- 172.21.15.45:0/1250832894 >> v2:172.21.15.103:3300/0 conn(0x7f4e40145230 0x7f4e4015bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.485+0000 7f4e45400640 1 -- 172.21.15.45:0/1250832894 >> v1:172.21.15.103:6789/0 conn(0x7f4e4015cbe0 legacy=0x7f4e400cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.485+0000 7f4e45400640 1 -- 172.21.15.45:0/1250832894 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e4015d690 con 0x7f4e400c5670 2024-04-16T16:18:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.486+0000 7f4e45400640 1 --2- 172.21.15.45:0/1250832894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e400c5670 0x7f4e400c5aa0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f4e40073180 tx=0x7f4e3802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e11c8c5aaf9430 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.487+0000 7f4e3f400640 1 -- 172.21.15.45:0/1250832894 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3802ca80 con 0x7f4e400c5670 2024-04-16T16:18:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.487+0000 7f4e3f400640 1 -- 172.21.15.45:0/1250832894 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e3802cc20 con 0x7f4e400c5670 2024-04-16T16:18:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.487+0000 7f4e3f400640 1 -- 172.21.15.45:0/1250832894 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e380343f0 con 0x7f4e400c5670 2024-04-16T16:18:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.488+0000 7f4e45e00640 1 -- 172.21.15.45:0/1250832894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e400c5670 msgr2=0x7f4e400c5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.488+0000 7f4e45e00640 1 --2- 172.21.15.45:0/1250832894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e400c5670 0x7f4e400c5aa0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f4e40073180 tx=0x7f4e3802bee0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.488+0000 7f4e45e00640 1 -- 172.21.15.45:0/1250832894 shutdown_connections 2024-04-16T16:18:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.488+0000 7f4e45e00640 1 --2- 172.21.15.45:0/1250832894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e400c5670 0x7f4e400c5aa0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.488+0000 7f4e45e00640 1 --2- 172.21.15.45:0/1250832894 >> v2:172.21.15.103:3300/0 conn(0x7f4e40145230 0x7f4e4015bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.488+0000 7f4e45e00640 1 -- 172.21.15.45:0/1250832894 >> 172.21.15.45:0/1250832894 conn(0x7f4e4006b930 msgr2=0x7f4e4006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.488+0000 7f4e45e00640 1 -- 172.21.15.45:0/1250832894 shutdown_connections 2024-04-16T16:18:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.488+0000 7f4e45e00640 1 -- 172.21.15.45:0/1250832894 wait complete. 2024-04-16T16:18:50.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.490+0000 7f4e45e00640 1 Processor -- start 2024-04-16T16:18:50.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.490+0000 7f4e45e00640 1 -- start start 2024-04-16T16:18:50.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.491+0000 7f4e45e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e400c5670 0x7f4e40132f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.491+0000 7f4e45e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e40145230 0x7f4e400cf830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.491+0000 7f4e45e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e4015cbe0 0x7f4e40133f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.491+0000 7f4e45e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4015d070 con 0x7f4e4015cbe0 2024-04-16T16:18:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.491+0000 7f4e45e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4015dc50 con 0x7f4e400c5670 2024-04-16T16:18:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.491+0000 7f4e45e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4e4015e450 con 0x7f4e40145230 2024-04-16T16:18:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.491+0000 7f4e45400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e4015cbe0 0x7f4e40133f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.491+0000 7f4e45400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e4015cbe0 0x7f4e40133f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51938/0 (socket says 172.21.15.45:51938) 2024-04-16T16:18:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.491+0000 7f4e45400640 1 -- 172.21.15.45:0/177313323 learned_addr learned my addr 172.21.15.45:0/177313323 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.491+0000 7f4e44a00640 1 --2- 172.21.15.45:0/177313323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e400c5670 0x7f4e40132f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.492+0000 7f4e3fe00640 1 --2- 172.21.15.45:0/177313323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e40145230 0x7f4e400cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.492+0000 7f4e45400640 1 -- 172.21.15.45:0/177313323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e40145230 msgr2=0x7f4e400cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.492+0000 7f4e45400640 1 --2- 172.21.15.45:0/177313323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e40145230 0x7f4e400cf830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.492+0000 7f4e45400640 1 -- 172.21.15.45:0/177313323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e400c5670 msgr2=0x7f4e40132f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.492+0000 7f4e45400640 1 --2- 172.21.15.45:0/177313323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e400c5670 0x7f4e40132f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.492+0000 7f4e45400640 1 -- 172.21.15.45:0/177313323 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e401361b0 con 0x7f4e4015cbe0 2024-04-16T16:18:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.492+0000 7f4e45400640 1 --2- 172.21.15.45:0/177313323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e4015cbe0 0x7f4e40133f20 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f4e38009f40 tx=0x7f4e38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.492+0000 7f4e3cc00640 1 -- 172.21.15.45:0/177313323 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e38043020 con 0x7f4e4015cbe0 2024-04-16T16:18:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.492+0000 7f4e3cc00640 1 -- 172.21.15.45:0/177313323 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e3803e450 con 0x7f4e4015cbe0 2024-04-16T16:18:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.492+0000 7f4e3cc00640 1 -- 172.21.15.45:0/177313323 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e38009510 con 0x7f4e4015cbe0 2024-04-16T16:18:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.492+0000 7f4e45e00640 1 -- 172.21.15.45:0/177313323 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e40136450 con 0x7f4e4015cbe0 2024-04-16T16:18:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.493+0000 7f4e45e00640 1 -- 172.21.15.45:0/177313323 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e40136a10 con 0x7f4e4015cbe0 2024-04-16T16:18:50.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.494+0000 7f4e45e00640 1 -- 172.21.15.45:0/177313323 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e40075780 con 0x7f4e4015cbe0 2024-04-16T16:18:50.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.495+0000 7f4e3cc00640 1 -- 172.21.15.45:0/177313323 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f4e38046020 con 0x7f4e4015cbe0 2024-04-16T16:18:50.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.496+0000 7f4e3cc00640 1 --2- 172.21.15.45:0/177313323 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f4e2006c980 0x7f4e2006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:50.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.496+0000 7f4e3cc00640 1 -- 172.21.15.45:0/177313323 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6354+0+0 (secure 0 0 0) 0x7f4e380ae5a0 con 0x7f4e4015cbe0 2024-04-16T16:18:50.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.497+0000 7f4e44a00640 1 --2- 172.21.15.45:0/177313323 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f4e2006c980 0x7f4e2006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:50.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.498+0000 7f4e44a00640 1 --2- 172.21.15.45:0/177313323 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f4e2006c980 0x7f4e2006ee40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4e400cdab0 tx=0x7f4e34008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:50.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.502+0000 7f4e3cc00640 1 -- 172.21.15.45:0/177313323 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e3804d030 con 0x7f4e4015cbe0 2024-04-16T16:18:50.710 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:50.709+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:18:50.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.753+0000 7f4e45e00640 1 -- 172.21.15.45:0/177313323 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4e400b5950 con 0x7f4e4015cbe0 2024-04-16T16:18:50.754 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-04-16T16:18:50.754+0000 7fe7a4800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T16:18:50.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.757+0000 7f4e3cc00640 1 -- 172.21.15.45:0/177313323 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v65) ==== 161+0+0 (secure 0 0 0) 0x7f4e3807c7e0 con 0x7f4e4015cbe0 2024-04-16T16:18:50.757 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-16T16:18:50.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.760+0000 7f4e45e00640 1 -- 172.21.15.45:0/177313323 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f4e2006c980 msgr2=0x7f4e2006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:50.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.760+0000 7f4e45e00640 1 --2- 172.21.15.45:0/177313323 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f4e2006c980 0x7f4e2006ee40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4e400cdab0 tx=0x7f4e34008040 comp rx=0 tx=0).stop 2024-04-16T16:18:50.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.761+0000 7f4e45e00640 1 -- 172.21.15.45:0/177313323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e4015cbe0 msgr2=0x7f4e40133f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:50.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.761+0000 7f4e45e00640 1 --2- 172.21.15.45:0/177313323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e4015cbe0 0x7f4e40133f20 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f4e38009f40 tx=0x7f4e38004a40 comp rx=0 tx=0).stop 2024-04-16T16:18:50.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.761+0000 7f4e45e00640 1 -- 172.21.15.45:0/177313323 shutdown_connections 2024-04-16T16:18:50.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.761+0000 7f4e45e00640 1 --2- 172.21.15.45:0/177313323 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f4e2006c980 0x7f4e2006ee40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.761+0000 7f4e45e00640 1 --2- 172.21.15.45:0/177313323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e4015cbe0 0x7f4e40133f20 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.761+0000 7f4e45e00640 1 --2- 172.21.15.45:0/177313323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e40145230 0x7f4e400cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.761+0000 7f4e45e00640 1 --2- 172.21.15.45:0/177313323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e400c5670 0x7f4e40132f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.761+0000 7f4e45e00640 1 -- 172.21.15.45:0/177313323 >> 172.21.15.45:0/177313323 conn(0x7f4e4006b930 msgr2=0x7f4e400b35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:50.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.761+0000 7f4e45e00640 1 -- 172.21.15.45:0/177313323 shutdown_connections 2024-04-16T16:18:50.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.761+0000 7f4e45e00640 1 -- 172.21.15.45:0/177313323 wait complete. 2024-04-16T16:18:50.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-16T16:18:50.793 INFO:tasks.ceph.mds.a.smithi045.stderr: -3716> 2024-04-16T16:18:21.232+0000 7f6a7d400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:18:50.868 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2024-04-16T16:18:50.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.991+0000 7fc5b3600640 1 Processor -- start 2024-04-16T16:18:50.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.992+0000 7fc5b3600640 1 -- start start 2024-04-16T16:18:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.993+0000 7fc5b3600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc5ac06d450 0x7fc5ac06d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.993+0000 7fc5b3600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5ac1607f0 0x7fc5ac160c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.993+0000 7fc5b3600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5ac0d0a00 con 0x7fc5ac09ac30 2024-04-16T16:18:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.993+0000 7fc5b3600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc5ac0d15d0 con 0x7fc5ac06d450 2024-04-16T16:18:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.993+0000 7fc5b3600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ac06df10 con 0x7fc5ac1607f0 2024-04-16T16:18:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.993+0000 7fc5b0c00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc5ac09ac30 0x7fc5ac09b010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54482/0 (socket says 172.21.15.45:54482) 2024-04-16T16:18:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.993+0000 7fc5b0c00640 1 -- 172.21.15.45:0/3640482658 learned_addr learned my addr 172.21.15.45:0/3640482658 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.994+0000 7fc5abe00640 1 --2- 172.21.15.45:0/3640482658 >> v2:172.21.15.103:3300/0 conn(0x7fc5ac06d450 0x7fc5ac06d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.994+0000 7fc5b1600640 1 --2- 172.21.15.45:0/3640482658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5ac1607f0 0x7fc5ac160c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.994+0000 7fc5ab400640 1 -- 172.21.15.45:0/3640482658 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2474391405 0 0) 0x7fc5ac0d0a00 con 0x7fc5ac09ac30 2024-04-16T16:18:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.994+0000 7fc5ab400640 1 -- 172.21.15.45:0/3640482658 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc594003520 con 0x7fc5ac09ac30 2024-04-16T16:18:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.994+0000 7fc5ab400640 1 -- 172.21.15.45:0/3640482658 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 543457027 0 0) 0x7fc594003520 con 0x7fc5ac09ac30 2024-04-16T16:18:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.994+0000 7fc5ab400640 1 -- 172.21.15.45:0/3640482658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5ac1607f0 msgr2=0x7fc5ac160c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.994+0000 7fc5ab400640 1 --2- 172.21.15.45:0/3640482658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5ac1607f0 0x7fc5ac160c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.994+0000 7fc5ab400640 1 -- 172.21.15.45:0/3640482658 >> v2:172.21.15.103:3300/0 conn(0x7fc5ac06d450 msgr2=0x7fc5ac06d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.994+0000 7fc5ab400640 1 --2- 172.21.15.45:0/3640482658 >> v2:172.21.15.103:3300/0 conn(0x7fc5ac06d450 0x7fc5ac06d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.994+0000 7fc5ab400640 1 -- 172.21.15.45:0/3640482658 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ac0d0ca0 con 0x7fc5ac09ac30 2024-04-16T16:18:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.994+0000 7fc5ab400640 1 -- 172.21.15.45:0/3640482658 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fc5a0001670 con 0x7fc5ac09ac30 2024-04-16T16:18:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.994+0000 7fc5ab400640 1 -- 172.21.15.45:0/3640482658 >> v1:172.21.15.45:6789/0 conn(0x7fc5ac09ac30 legacy=0x7fc5ac09b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:50.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.994+0000 7fc5ab400640 1 --2- 172.21.15.45:0/3640482658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5940039b0 0x7fc594003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:50.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.994+0000 7fc5ab400640 1 --2- 172.21.15.45:0/3640482658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc594004800 0x7fc594004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:50.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.995+0000 7fc5ab400640 1 --2- 172.21.15.45:0/3640482658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5ac1607f0 0x7fc5940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:50.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.995+0000 7fc5ab400640 1 -- 172.21.15.45:0/3640482658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac0d15d0 con 0x7fc5940039b0 2024-04-16T16:18:50.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.995+0000 7fc5ab400640 1 -- 172.21.15.45:0/3640482658 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac06df10 con 0x7fc594004800 2024-04-16T16:18:50.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.995+0000 7fc5ab400640 1 -- 172.21.15.45:0/3640482658 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc59400b9f0 con 0x7fc5ac1607f0 2024-04-16T16:18:50.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.995+0000 7fc5b0c00640 1 --2- 172.21.15.45:0/3640482658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5940039b0 0x7fc594003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:50.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.995+0000 7fc5abe00640 1 --2- 172.21.15.45:0/3640482658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc594004800 0x7fc594004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:50.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.995+0000 7fc5b0c00640 1 -- 172.21.15.45:0/3640482658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5ac1607f0 msgr2=0x7fc5940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:50.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.995+0000 7fc5b0c00640 1 --2- 172.21.15.45:0/3640482658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5ac1607f0 0x7fc5940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.995+0000 7fc5b0c00640 1 -- 172.21.15.45:0/3640482658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc594004800 msgr2=0x7fc594004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:50.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.995+0000 7fc5b0c00640 1 --2- 172.21.15.45:0/3640482658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc594004800 0x7fc594004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:50.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.995+0000 7fc5b0c00640 1 -- 172.21.15.45:0/3640482658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc594003520 con 0x7fc5940039b0 2024-04-16T16:18:50.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.995+0000 7fc5abe00640 1 --2- 172.21.15.45:0/3640482658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc594004800 0x7fc594004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:51.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.996+0000 7fc5b0c00640 1 --2- 172.21.15.45:0/3640482658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5940039b0 0x7fc594003dd0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000c210 tx=0x7fc594005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9912e431ea3df503 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:51.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.996+0000 7fc5ab400640 1 -- 172.21.15.45:0/3640482658 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0007d50 con 0x7fc5940039b0 2024-04-16T16:18:51.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.997+0000 7fc5ab400640 1 -- 172.21.15.45:0/3640482658 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5a0030070 con 0x7fc5940039b0 2024-04-16T16:18:51.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.997+0000 7fc5ab400640 1 -- 172.21.15.45:0/3640482658 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a002cd70 con 0x7fc5940039b0 2024-04-16T16:18:51.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.997+0000 7fc5b3600640 1 -- 172.21.15.45:0/3640482658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5940039b0 msgr2=0x7fc594003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:51.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.997+0000 7fc5b3600640 1 --2- 172.21.15.45:0/3640482658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5940039b0 0x7fc594003dd0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000c210 tx=0x7fc594005830 comp rx=0 tx=0).stop 2024-04-16T16:18:51.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.997+0000 7fc5b1600640 1 -- 172.21.15.45:0/3640482658 reap_dead start 2024-04-16T16:18:51.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.997+0000 7fc5b3600640 1 -- 172.21.15.45:0/3640482658 shutdown_connections 2024-04-16T16:18:51.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.997+0000 7fc5b3600640 1 -- 172.21.15.45:0/3640482658 >> 172.21.15.45:0/3640482658 conn(0x7fc5ac0d2870 msgr2=0x7fc5ac0d2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:51.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.997+0000 7fc5b3600640 1 -- 172.21.15.45:0/3640482658 shutdown_connections 2024-04-16T16:18:51.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.997+0000 7fc5b3600640 1 -- 172.21.15.45:0/3640482658 wait complete. 2024-04-16T16:18:51.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.999+0000 7fc5b3600640 1 Processor -- start 2024-04-16T16:18:51.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:50.999+0000 7fc5b3600640 1 -- start start 2024-04-16T16:18:51.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.000+0000 7fc5b3600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5940039b0 0x7fc5ac14cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:51.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.000+0000 7fc5b3600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5ac0b5ea0 0x7fc5ac0b6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:51.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.000+0000 7fc5b3600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5ac15e4c0 0x7fc5ac15e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:51.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.000+0000 7fc5b3600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac160330 con 0x7fc5940039b0 2024-04-16T16:18:51.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.000+0000 7fc5b3600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac0d1260 con 0x7fc5ac0b5ea0 2024-04-16T16:18:51.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.000+0000 7fc5b3600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ac0b3b00 con 0x7fc5ac15e4c0 2024-04-16T16:18:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.001+0000 7fc5b0c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5940039b0 0x7fc5ac14cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.001+0000 7fc5b0c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5940039b0 0x7fc5ac14cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51984/0 (socket says 172.21.15.45:51984) 2024-04-16T16:18:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.001+0000 7fc5b0c00640 1 -- 172.21.15.45:0/927417472 learned_addr learned my addr 172.21.15.45:0/927417472 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.001+0000 7fc5abe00640 1 --2- 172.21.15.45:0/927417472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5ac0b5ea0 0x7fc5ac0b6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.001+0000 7fc5b1600640 1 --2- 172.21.15.45:0/927417472 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5ac15e4c0 0x7fc5ac15e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.001+0000 7fc5b0c00640 1 -- 172.21.15.45:0/927417472 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5ac15e4c0 msgr2=0x7fc5ac15e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.001+0000 7fc5b0c00640 1 --2- 172.21.15.45:0/927417472 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5ac15e4c0 0x7fc5ac15e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.001+0000 7fc5b0c00640 1 -- 172.21.15.45:0/927417472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5ac0b5ea0 msgr2=0x7fc5ac0b6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.001+0000 7fc5b0c00640 1 --2- 172.21.15.45:0/927417472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5ac0b5ea0 0x7fc5ac0b6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.001+0000 7fc5b0c00640 1 -- 172.21.15.45:0/927417472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ac0b6920 con 0x7fc5940039b0 2024-04-16T16:18:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.001+0000 7fc5b0c00640 1 --2- 172.21.15.45:0/927417472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5940039b0 0x7fc5ac14cd40 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0001970 tx=0x7fc5a0001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:51.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.002+0000 7fc5a8c00640 1 -- 172.21.15.45:0/927417472 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0001c90 con 0x7fc5940039b0 2024-04-16T16:18:51.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.002+0000 7fc5a8c00640 1 -- 172.21.15.45:0/927417472 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5a0004bf0 con 0x7fc5940039b0 2024-04-16T16:18:51.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.002+0000 7fc5a8c00640 1 -- 172.21.15.45:0/927417472 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0007070 con 0x7fc5940039b0 2024-04-16T16:18:51.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.002+0000 7fc5b3600640 1 -- 172.21.15.45:0/927417472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5ac0b6c00 con 0x7fc5940039b0 2024-04-16T16:18:51.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.002+0000 7fc5b3600640 1 -- 172.21.15.45:0/927417472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5ac15c7a0 con 0x7fc5940039b0 2024-04-16T16:18:51.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.003+0000 7fc5b3600640 1 -- 172.21.15.45:0/927417472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5ac0b6dd0 con 0x7fc5940039b0 2024-04-16T16:18:51.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.004+0000 7fc5a8c00640 1 -- 172.21.15.45:0/927417472 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7fc5a0007210 con 0x7fc5940039b0 2024-04-16T16:18:51.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.005+0000 7fc5a8c00640 1 --2- 172.21.15.45:0/927417472 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fc58c06c9b0 0x7fc58c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:51.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.005+0000 7fc5a8c00640 1 -- 172.21.15.45:0/927417472 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6440+0+0 (secure 0 0 0) 0x7fc5a00acf30 con 0x7fc5940039b0 2024-04-16T16:18:51.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.006+0000 7fc5abe00640 1 --2- 172.21.15.45:0/927417472 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fc58c06c9b0 0x7fc58c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:51.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.007+0000 7fc5abe00640 1 --2- 172.21.15.45:0/927417472 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fc58c06c9b0 0x7fc58c06ee70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc5980099c0 tx=0x7fc598008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:51.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.010+0000 7fc5a8c00640 1 -- 172.21.15.45:0/927417472 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc5a007b020 con 0x7fc5940039b0 2024-04-16T16:18:51.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.264+0000 7fc5b3600640 1 -- 172.21.15.45:0/927417472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fc5ac075780 con 0x7fc5940039b0 2024-04-16T16:18:51.646 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:51.645+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:18:51.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.795+0000 7fc5a8c00640 1 -- 172.21.15.45:0/927417472 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v66) ==== 104+0+0 (secure 0 0 0) 0x7fc5a007a920 con 0x7fc5940039b0 2024-04-16T16:18:51.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.799+0000 7fc5b3600640 1 -- 172.21.15.45:0/927417472 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fc58c06c9b0 msgr2=0x7fc58c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:51.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.799+0000 7fc5b3600640 1 --2- 172.21.15.45:0/927417472 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fc58c06c9b0 0x7fc58c06ee70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc5980099c0 tx=0x7fc598008040 comp rx=0 tx=0).stop 2024-04-16T16:18:51.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.799+0000 7fc5b3600640 1 -- 172.21.15.45:0/927417472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5940039b0 msgr2=0x7fc5ac14cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:51.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.799+0000 7fc5b3600640 1 --2- 172.21.15.45:0/927417472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5940039b0 0x7fc5ac14cd40 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0001970 tx=0x7fc5a0001a40 comp rx=0 tx=0).stop 2024-04-16T16:18:51.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.799+0000 7fc5b3600640 1 -- 172.21.15.45:0/927417472 shutdown_connections 2024-04-16T16:18:51.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.799+0000 7fc5b3600640 1 --2- 172.21.15.45:0/927417472 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5ac15e4c0 0x7fc5ac15e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:51.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.799+0000 7fc5b3600640 1 --2- 172.21.15.45:0/927417472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5ac0b5ea0 0x7fc5ac0b6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:51.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.799+0000 7fc5b3600640 1 --2- 172.21.15.45:0/927417472 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fc58c06c9b0 0x7fc58c06ee70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:51.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.799+0000 7fc5b3600640 1 --2- 172.21.15.45:0/927417472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5940039b0 0x7fc5ac14cd40 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.799+0000 7fc5b3600640 1 -- 172.21.15.45:0/927417472 >> 172.21.15.45:0/927417472 conn(0x7fc5ac0d2870 msgr2=0x7fc5ac0d1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.799+0000 7fc5b3600640 1 -- 172.21.15.45:0/927417472 shutdown_connections 2024-04-16T16:18:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:51.799+0000 7fc5b3600640 1 -- 172.21.15.45:0/927417472 wait complete. 2024-04-16T16:18:51.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-16T16:18:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.042+0000 7f5f59c00640 1 Processor -- start 2024-04-16T16:18:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.042+0000 7f5f59c00640 1 -- start start 2024-04-16T16:18:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.044+0000 7f5f59c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5f5415e850 0x7f5f5415fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.044+0000 7f5f59c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f5415ec30 0x7f5f5406b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.044+0000 7f5f59c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f5415ff70 con 0x7f5f5406be70 2024-04-16T16:18:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.044+0000 7f5f59c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5f540716f0 con 0x7f5f5415e850 2024-04-16T16:18:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.044+0000 7f5f59c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5f54070ae0 con 0x7f5f5415ec30 2024-04-16T16:18:52.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.045+0000 7f5f53400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f5415ec30 0x7f5f5406b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:52.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.045+0000 7f5f53400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f5415ec30 0x7f5f5406b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:39360/0 (socket says 172.21.15.45:39360) 2024-04-16T16:18:52.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.045+0000 7f5f53400640 1 -- 172.21.15.45:0/333045416 learned_addr learned my addr 172.21.15.45:0/333045416 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:52.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.045+0000 7f5f53e00640 1 --2- 172.21.15.45:0/333045416 >> v2:172.21.15.103:3300/0 conn(0x7f5f5415e850 0x7f5f5415fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:52.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.046+0000 7f5f53400640 1 -- 172.21.15.45:0/333045416 >> v2:172.21.15.103:3300/0 conn(0x7f5f5415e850 msgr2=0x7f5f5415fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:52.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.046+0000 7f5f53400640 1 --2- 172.21.15.45:0/333045416 >> v2:172.21.15.103:3300/0 conn(0x7f5f5415e850 0x7f5f5415fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:52.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.046+0000 7f5f53400640 1 -- 172.21.15.45:0/333045416 >> v1:172.21.15.103:6789/0 conn(0x7f5f5406be70 legacy=0x7f5f541625a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:52.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.046+0000 7f5f53400640 1 -- 172.21.15.45:0/333045416 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f54070dc0 con 0x7f5f5415ec30 2024-04-16T16:18:52.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.047+0000 7f5f53400640 1 --2- 172.21.15.45:0/333045416 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f5415ec30 0x7f5f5406b930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5f40009870 tx=0x7f5f4002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eab6175c069a977c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:52.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.048+0000 7f5f52a00640 1 -- 172.21.15.45:0/333045416 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f40034070 con 0x7f5f5415ec30 2024-04-16T16:18:52.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.048+0000 7f5f52a00640 1 -- 172.21.15.45:0/333045416 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f40030840 con 0x7f5f5415ec30 2024-04-16T16:18:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.048+0000 7f5f52a00640 1 -- 172.21.15.45:0/333045416 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f40030c10 con 0x7f5f5415ec30 2024-04-16T16:18:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.048+0000 7f5f59c00640 1 -- 172.21.15.45:0/333045416 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f5415ec30 msgr2=0x7f5f5406b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.048+0000 7f5f59c00640 1 --2- 172.21.15.45:0/333045416 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f5415ec30 0x7f5f5406b930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5f40009870 tx=0x7f5f4002b980 comp rx=0 tx=0).stop 2024-04-16T16:18:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.048+0000 7f5f59c00640 1 -- 172.21.15.45:0/333045416 shutdown_connections 2024-04-16T16:18:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.048+0000 7f5f59c00640 1 --2- 172.21.15.45:0/333045416 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f5415ec30 0x7f5f5406b930 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.048+0000 7f5f59c00640 1 --2- 172.21.15.45:0/333045416 >> v2:172.21.15.103:3300/0 conn(0x7f5f5415e850 0x7f5f5415fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.048+0000 7f5f59c00640 1 -- 172.21.15.45:0/333045416 >> 172.21.15.45:0/333045416 conn(0x7f5f540b3330 msgr2=0x7f5f540b3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.049+0000 7f5f59c00640 1 -- 172.21.15.45:0/333045416 shutdown_connections 2024-04-16T16:18:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.049+0000 7f5f59c00640 1 -- 172.21.15.45:0/333045416 wait complete. 2024-04-16T16:18:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.051+0000 7f5f59c00640 1 Processor -- start 2024-04-16T16:18:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.051+0000 7f5f59c00640 1 -- start start 2024-04-16T16:18:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.053+0000 7f5f59c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f5406be70 0x7f5f54153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.053+0000 7f5f59c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f5415e850 0x7f5f5415a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.053+0000 7f5f59c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f54163030 0x7f5f5415ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.053+0000 7f5f59c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5f540707f0 con 0x7f5f5406be70 2024-04-16T16:18:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.053+0000 7f5f59c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5f54071380 con 0x7f5f54163030 2024-04-16T16:18:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.053+0000 7f5f59c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5f540b08e0 con 0x7f5f5415e850 2024-04-16T16:18:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.053+0000 7f5f53e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f5406be70 0x7f5f54153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.053+0000 7f5f53e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f5406be70 0x7f5f54153250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51988/0 (socket says 172.21.15.45:51988) 2024-04-16T16:18:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.053+0000 7f5f53e00640 1 -- 172.21.15.45:0/84588708 learned_addr learned my addr 172.21.15.45:0/84588708 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.053+0000 7f5f53400640 1 --2- 172.21.15.45:0/84588708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f5415e850 0x7f5f5415a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.053+0000 7f5f53e00640 1 -- 172.21.15.45:0/84588708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f5415e850 msgr2=0x7f5f5415a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.053+0000 7f5f53e00640 1 --2- 172.21.15.45:0/84588708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f5415e850 0x7f5f5415a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.053+0000 7f5f59200640 1 --2- 172.21.15.45:0/84588708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f54163030 0x7f5f5415ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.054+0000 7f5f53e00640 1 -- 172.21.15.45:0/84588708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f54163030 msgr2=0x7f5f5415ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.054+0000 7f5f53e00640 1 --2- 172.21.15.45:0/84588708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f54163030 0x7f5f5415ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.054+0000 7f5f53e00640 1 -- 172.21.15.45:0/84588708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f540cc910 con 0x7f5f5406be70 2024-04-16T16:18:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.054+0000 7f5f59200640 1 --2- 172.21.15.45:0/84588708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f54163030 0x7f5f5415ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.054+0000 7f5f53e00640 1 --2- 172.21.15.45:0/84588708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f5406be70 0x7f5f54153250 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400e9b0 tx=0x7f5f4400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.054+0000 7f5f4be00640 1 -- 172.21.15.45:0/84588708 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f44017070 con 0x7f5f5406be70 2024-04-16T16:18:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.054+0000 7f5f4be00640 1 -- 172.21.15.45:0/84588708 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f4400c9a0 con 0x7f5f5406be70 2024-04-16T16:18:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.054+0000 7f5f4be00640 1 -- 172.21.15.45:0/84588708 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4400cd90 con 0x7f5f5406be70 2024-04-16T16:18:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.054+0000 7f5f59c00640 1 -- 172.21.15.45:0/84588708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f540ccbb0 con 0x7f5f5406be70 2024-04-16T16:18:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.055+0000 7f5f59c00640 1 -- 172.21.15.45:0/84588708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f540cd170 con 0x7f5f5406be70 2024-04-16T16:18:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.056+0000 7f5f59c00640 1 -- 172.21.15.45:0/84588708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f54075780 con 0x7f5f5406be70 2024-04-16T16:18:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.057+0000 7f5f4be00640 1 -- 172.21.15.45:0/84588708 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f5f4401a070 con 0x7f5f5406be70 2024-04-16T16:18:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.057+0000 7f5f4be00640 1 --2- 172.21.15.45:0/84588708 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5f3406c9b0 0x7f5f3406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.058+0000 7f5f4be00640 1 -- 172.21.15.45:0/84588708 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6440+0+0 (secure 0 0 0) 0x7f5f4408c250 con 0x7f5f5406be70 2024-04-16T16:18:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.058+0000 7f5f53400640 1 --2- 172.21.15.45:0/84588708 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5f3406c9b0 0x7f5f3406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.059+0000 7f5f53400640 1 --2- 172.21.15.45:0/84588708 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5f3406c9b0 0x7f5f3406ee70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5f40036000 tx=0x7f5f4003b040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:52.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.062+0000 7f5f4be00640 1 -- 172.21.15.45:0/84588708 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5f4405a340 con 0x7f5f5406be70 2024-04-16T16:18:52.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.274+0000 7f5f59c00640 1 -- 172.21.15.45:0/84588708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5f541e9430 con 0x7f5f5406be70 2024-04-16T16:18:52.587 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:52.587+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:18:52.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.818+0000 7f5f4be00640 1 -- 172.21.15.45:0/84588708 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v89) ==== 181+0+0 (secure 0 0 0) 0x7f5f44059c40 con 0x7f5f5406be70 2024-04-16T16:18:52.819 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2024-04-16T16:18:52.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.824+0000 7f5f59c00640 1 -- 172.21.15.45:0/84588708 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5f3406c9b0 msgr2=0x7f5f3406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:52.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.824+0000 7f5f59c00640 1 --2- 172.21.15.45:0/84588708 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5f3406c9b0 0x7f5f3406ee70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5f40036000 tx=0x7f5f4003b040 comp rx=0 tx=0).stop 2024-04-16T16:18:52.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.824+0000 7f5f59c00640 1 -- 172.21.15.45:0/84588708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f5406be70 msgr2=0x7f5f54153250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:52.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.824+0000 7f5f59c00640 1 --2- 172.21.15.45:0/84588708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f5406be70 0x7f5f54153250 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400e9b0 tx=0x7f5f4400ee70 comp rx=0 tx=0).stop 2024-04-16T16:18:52.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.824+0000 7f5f59c00640 1 -- 172.21.15.45:0/84588708 shutdown_connections 2024-04-16T16:18:52.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.824+0000 7f5f59c00640 1 --2- 172.21.15.45:0/84588708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f54163030 0x7f5f5415ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:52.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.824+0000 7f5f59c00640 1 --2- 172.21.15.45:0/84588708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f5415e850 0x7f5f5415a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:52.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.824+0000 7f5f59c00640 1 --2- 172.21.15.45:0/84588708 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5f3406c9b0 0x7f5f3406ee70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:52.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.825+0000 7f5f59c00640 1 --2- 172.21.15.45:0/84588708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f5406be70 0x7f5f54153250 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:52.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.825+0000 7f5f59c00640 1 -- 172.21.15.45:0/84588708 >> 172.21.15.45:0/84588708 conn(0x7f5f540b3330 msgr2=0x7f5f540b4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:52.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.825+0000 7f5f59c00640 1 -- 172.21.15.45:0/84588708 shutdown_connections 2024-04-16T16:18:52.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:52.825+0000 7f5f59c00640 1 -- 172.21.15.45:0/84588708 wait complete. 2024-04-16T16:18:52.852 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-16T16:18:53.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.066+0000 7f6aa3200640 1 Processor -- start 2024-04-16T16:18:53.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.066+0000 7f6aa3200640 1 -- start start 2024-04-16T16:18:53.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.068+0000 7f6aa3200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6aa40b4930 0x7f6aa40b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:53.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.068+0000 7f6aa3200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aa40b66d0 con 0x7f6aa40b6260 2024-04-16T16:18:53.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.068+0000 7f6aa3200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aa40b7a10 con 0x7f6aa41460f0 2024-04-16T16:18:53.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.068+0000 7f6aa3200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f6aa40b6970 con 0x7f6aa40b4930 2024-04-16T16:18:53.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.068+0000 7f6aa1400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6aa40b4930 0x7f6aa40b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:53.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.068+0000 7f6aa2800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6aa40b6260 0x7f6aa40d1580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54484/0 (socket says 172.21.15.45:54484) 2024-04-16T16:18:53.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.068+0000 7f6aa2800640 1 -- 172.21.15.45:0/2371582576 learned_addr learned my addr 172.21.15.45:0/2371582576 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:53.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.069+0000 7f6aa1400640 1 -- 172.21.15.45:0/2371582576 >> v1:172.21.15.103:6789/0 conn(0x7f6aa41460f0 legacy=0x7f6aa40b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:53.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.069+0000 7f6aa1400640 1 -- 172.21.15.45:0/2371582576 >> v1:172.21.15.45:6789/0 conn(0x7f6aa40b6260 legacy=0x7f6aa40d1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:53.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.069+0000 7f6aa1400640 1 -- 172.21.15.45:0/2371582576 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa40b6c50 con 0x7f6aa40b4930 2024-04-16T16:18:53.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.069+0000 7f6aa0a00640 1 -- 172.21.15.45:0/2371582576 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3979228494 0 0) 0x7f6aa40b66d0 con 0x7f6aa40b6260 2024-04-16T16:18:53.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.070+0000 7f6aa1400640 1 --2- 172.21.15.45:0/2371582576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa40b4930 0x7f6aa40b5d20 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c0097a0 tx=0x7f6a8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90995bfc46a0907 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:53.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.070+0000 7f6aa0a00640 1 -- 172.21.15.45:0/2371582576 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c035020 con 0x7f6aa40b4930 2024-04-16T16:18:53.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.070+0000 7f6aa0a00640 1 -- 172.21.15.45:0/2371582576 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a8c031830 con 0x7f6aa40b4930 2024-04-16T16:18:53.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.070+0000 7f6aa0a00640 1 -- 172.21.15.45:0/2371582576 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c031c00 con 0x7f6aa40b4930 2024-04-16T16:18:53.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.071+0000 7f6aa3200640 1 -- 172.21.15.45:0/2371582576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa40b4930 msgr2=0x7f6aa40b5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:53.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.071+0000 7f6aa3200640 1 --2- 172.21.15.45:0/2371582576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa40b4930 0x7f6aa40b5d20 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c0097a0 tx=0x7f6a8c02b8d0 comp rx=0 tx=0).stop 2024-04-16T16:18:53.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.071+0000 7f6aa3200640 1 -- 172.21.15.45:0/2371582576 shutdown_connections 2024-04-16T16:18:53.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.071+0000 7f6aa3200640 1 --2- 172.21.15.45:0/2371582576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa40b4930 0x7f6aa40b5d20 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:53.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.071+0000 7f6aa3200640 1 -- 172.21.15.45:0/2371582576 >> 172.21.15.45:0/2371582576 conn(0x7f6aa406b930 msgr2=0x7f6aa406bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:53.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.071+0000 7f6aa3200640 1 -- 172.21.15.45:0/2371582576 shutdown_connections 2024-04-16T16:18:53.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.071+0000 7f6aa3200640 1 -- 172.21.15.45:0/2371582576 wait complete. 2024-04-16T16:18:53.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.073+0000 7f6aa3200640 1 Processor -- start 2024-04-16T16:18:53.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.073+0000 7f6aa3200640 1 -- start start 2024-04-16T16:18:53.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.074+0000 7f6aa3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa40b4930 0x7f6aa40cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:53.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.074+0000 7f6aa3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aa40b6260 0x7f6aa406ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:53.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.074+0000 7f6aa3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa41460f0 0x7f6aa41e4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:53.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.074+0000 7f6aa3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa40b74f0 con 0x7f6aa41460f0 2024-04-16T16:18:53.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.074+0000 7f6aa3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa40b7210 con 0x7f6aa40b6260 2024-04-16T16:18:53.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.074+0000 7f6aa3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa41464d0 con 0x7f6aa40b4930 2024-04-16T16:18:53.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.074+0000 7f6aa2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa41460f0 0x7f6aa41e4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:53.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.074+0000 7f6aa2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa41460f0 0x7f6aa41e4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52010/0 (socket says 172.21.15.45:52010) 2024-04-16T16:18:53.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.074+0000 7f6aa2800640 1 -- 172.21.15.45:0/525749910 learned_addr learned my addr 172.21.15.45:0/525749910 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:53.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.074+0000 7f6aa1e00640 1 --2- 172.21.15.45:0/525749910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa40b4930 0x7f6aa40cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:53.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.075+0000 7f6aa1400640 1 --2- 172.21.15.45:0/525749910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aa40b6260 0x7f6aa406ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:53.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.075+0000 7f6aa2800640 1 -- 172.21.15.45:0/525749910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa40b4930 msgr2=0x7f6aa40cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:53.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.075+0000 7f6aa2800640 1 --2- 172.21.15.45:0/525749910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa40b4930 0x7f6aa40cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:53.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.075+0000 7f6aa2800640 1 -- 172.21.15.45:0/525749910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aa40b6260 msgr2=0x7f6aa406ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:53.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.075+0000 7f6aa2800640 1 --2- 172.21.15.45:0/525749910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aa40b6260 0x7f6aa406ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:53.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.075+0000 7f6aa2800640 1 -- 172.21.15.45:0/525749910 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa40ce4f0 con 0x7f6aa41460f0 2024-04-16T16:18:53.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.075+0000 7f6aa1e00640 1 --2- 172.21.15.45:0/525749910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa40b4930 0x7f6aa40cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:18:53.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.075+0000 7f6aa2800640 1 --2- 172.21.15.45:0/525749910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa41460f0 0x7f6aa41e4290 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c00c810 tx=0x7f6a9c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:53.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.075+0000 7f6a9a000640 1 -- 172.21.15.45:0/525749910 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c018070 con 0x7f6aa41460f0 2024-04-16T16:18:53.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.075+0000 7f6a9a000640 1 -- 172.21.15.45:0/525749910 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a9c00a640 con 0x7f6aa41460f0 2024-04-16T16:18:53.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.075+0000 7f6a9a000640 1 -- 172.21.15.45:0/525749910 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c00aa30 con 0x7f6aa41460f0 2024-04-16T16:18:53.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.075+0000 7f6aa3200640 1 -- 172.21.15.45:0/525749910 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aa406d340 con 0x7f6aa41460f0 2024-04-16T16:18:53.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.076+0000 7f6aa3200640 1 -- 172.21.15.45:0/525749910 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aa406d900 con 0x7f6aa41460f0 2024-04-16T16:18:53.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.077+0000 7f6aa3200640 1 -- 172.21.15.45:0/525749910 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aa4075780 con 0x7f6aa41460f0 2024-04-16T16:18:53.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.078+0000 7f6a9a000640 1 -- 172.21.15.45:0/525749910 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f6a9c00abd0 con 0x7f6aa41460f0 2024-04-16T16:18:53.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.079+0000 7f6a9a000640 1 --2- 172.21.15.45:0/525749910 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6a8006c980 0x7f6a8006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:53.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.079+0000 7f6a9a000640 1 -- 172.21.15.45:0/525749910 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6002+0+0 (secure 0 0 0) 0x7f6a9c08b2e0 con 0x7f6aa41460f0 2024-04-16T16:18:53.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.080+0000 7f6aa1e00640 1 --2- 172.21.15.45:0/525749910 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6a8006c980 0x7f6a8006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:53.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.081+0000 7f6aa1e00640 1 --2- 172.21.15.45:0/525749910 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6a8006c980 0x7f6a8006ee40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6a94006fd0 tx=0x7f6a94008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:53.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.084+0000 7f6a9a000640 1 -- 172.21.15.45:0/525749910 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6a9c059580 con 0x7f6aa41460f0 2024-04-16T16:18:53.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.296+0000 7f6aa3200640 1 -- 172.21.15.45:0/525749910 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6aa40d2060 con 0x7f6aa41460f0 2024-04-16T16:18:53.535 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:53.534+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:18:53.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.822+0000 7f6a9a000640 1 -- 172.21.15.45:0/525749910 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v90) ==== 169+0+0 (secure 0 0 0) 0x7f6a9c08b0c0 con 0x7f6aa41460f0 2024-04-16T16:18:53.823 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2024-04-16T16:18:53.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.826+0000 7f6aa3200640 1 -- 172.21.15.45:0/525749910 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6a8006c980 msgr2=0x7f6a8006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:53.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.826+0000 7f6aa3200640 1 --2- 172.21.15.45:0/525749910 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6a8006c980 0x7f6a8006ee40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6a94006fd0 tx=0x7f6a94008040 comp rx=0 tx=0).stop 2024-04-16T16:18:53.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.827+0000 7f6aa3200640 1 -- 172.21.15.45:0/525749910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa41460f0 msgr2=0x7f6aa41e4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:53.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.827+0000 7f6aa3200640 1 --2- 172.21.15.45:0/525749910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa41460f0 0x7f6aa41e4290 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c00c810 tx=0x7f6a9c00ccd0 comp rx=0 tx=0).stop 2024-04-16T16:18:53.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.827+0000 7f6aa3200640 1 -- 172.21.15.45:0/525749910 shutdown_connections 2024-04-16T16:18:53.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.827+0000 7f6aa3200640 1 --2- 172.21.15.45:0/525749910 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6a8006c980 0x7f6a8006ee40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:53.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.827+0000 7f6aa3200640 1 --2- 172.21.15.45:0/525749910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa41460f0 0x7f6aa41e4290 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:53.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.827+0000 7f6aa3200640 1 --2- 172.21.15.45:0/525749910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aa40b6260 0x7f6aa406ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:53.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.827+0000 7f6aa3200640 1 --2- 172.21.15.45:0/525749910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa40b4930 0x7f6aa40cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:53.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.827+0000 7f6aa3200640 1 -- 172.21.15.45:0/525749910 >> 172.21.15.45:0/525749910 conn(0x7f6aa406b930 msgr2=0x7f6aa41e3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:53.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.828+0000 7f6aa3200640 1 -- 172.21.15.45:0/525749910 shutdown_connections 2024-04-16T16:18:53.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:53.828+0000 7f6aa3200640 1 -- 172.21.15.45:0/525749910 wait complete. 2024-04-16T16:18:53.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-16T16:18:54.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.072+0000 7fea61600640 1 Processor -- start 2024-04-16T16:18:54.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.072+0000 7fea61600640 1 -- start start 2024-04-16T16:18:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.074+0000 7fea61600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fea5c15c9f0 0x7fea5c0ce000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.074+0000 7fea61600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5c0d0980 0x7fea5c0d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.074+0000 7fea61600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea5c0d12f0 con 0x7fea5c0d01f0 2024-04-16T16:18:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.074+0000 7fea61600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fea5c15cdd0 con 0x7fea5c15c9f0 2024-04-16T16:18:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.074+0000 7fea61600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fea5c09bcb0 con 0x7fea5c0d0980 2024-04-16T16:18:54.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.074+0000 7fea60c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5c0d0980 0x7fea5c0d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:54.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.074+0000 7fea60c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5c0d0980 0x7fea5c0d0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:43672/0 (socket says 172.21.15.45:43672) 2024-04-16T16:18:54.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.074+0000 7fea60c00640 1 -- 172.21.15.45:0/1373558631 learned_addr learned my addr 172.21.15.45:0/1373558631 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:54.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.075+0000 7fea5be00640 1 --2- 172.21.15.45:0/1373558631 >> v2:172.21.15.103:3300/0 conn(0x7fea5c15c9f0 0x7fea5c0ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:54.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.075+0000 7fea60c00640 1 -- 172.21.15.45:0/1373558631 >> v2:172.21.15.103:3300/0 conn(0x7fea5c15c9f0 msgr2=0x7fea5c0ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:54.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.075+0000 7fea60c00640 1 --2- 172.21.15.45:0/1373558631 >> v2:172.21.15.103:3300/0 conn(0x7fea5c15c9f0 0x7fea5c0ce000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:54.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.075+0000 7fea60c00640 1 -- 172.21.15.45:0/1373558631 >> v1:172.21.15.103:6789/0 conn(0x7fea5c0d01f0 legacy=0x7fea5c09b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:54.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.075+0000 7fea60c00640 1 -- 172.21.15.45:0/1373558631 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea5c0cd260 con 0x7fea5c0d0980 2024-04-16T16:18:54.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.077+0000 7fea60c00640 1 --2- 172.21.15.45:0/1373558631 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5c0d0980 0x7fea5c0d0db0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fea54009870 tx=0x7fea5402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a60c6726587ba5f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:54.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.077+0000 7fea5aa00640 1 -- 172.21.15.45:0/1373558631 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea54035020 con 0x7fea5c0d0980 2024-04-16T16:18:54.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.077+0000 7fea5aa00640 1 -- 172.21.15.45:0/1373558631 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea54031640 con 0x7fea5c0d0980 2024-04-16T16:18:54.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.077+0000 7fea5aa00640 1 -- 172.21.15.45:0/1373558631 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5402d040 con 0x7fea5c0d0980 2024-04-16T16:18:54.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.078+0000 7fea61600640 1 -- 172.21.15.45:0/1373558631 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5c0d0980 msgr2=0x7fea5c0d0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:54.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.078+0000 7fea61600640 1 --2- 172.21.15.45:0/1373558631 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5c0d0980 0x7fea5c0d0db0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fea54009870 tx=0x7fea5402b8f0 comp rx=0 tx=0).stop 2024-04-16T16:18:54.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.078+0000 7fea61600640 1 -- 172.21.15.45:0/1373558631 shutdown_connections 2024-04-16T16:18:54.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.078+0000 7fea61600640 1 --2- 172.21.15.45:0/1373558631 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5c0d0980 0x7fea5c0d0db0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:54.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.078+0000 7fea61600640 1 --2- 172.21.15.45:0/1373558631 >> v2:172.21.15.103:3300/0 conn(0x7fea5c15c9f0 0x7fea5c0ce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:54.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.078+0000 7fea61600640 1 -- 172.21.15.45:0/1373558631 >> 172.21.15.45:0/1373558631 conn(0x7fea5c0b4110 msgr2=0x7fea5c0b4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:54.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.078+0000 7fea61600640 1 -- 172.21.15.45:0/1373558631 shutdown_connections 2024-04-16T16:18:54.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.078+0000 7fea61600640 1 -- 172.21.15.45:0/1373558631 wait complete. 2024-04-16T16:18:54.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.080+0000 7fea61600640 1 Processor -- start 2024-04-16T16:18:54.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.080+0000 7fea61600640 1 -- start start 2024-04-16T16:18:54.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.081+0000 7fea61600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5c0d01f0 0x7fea5c0c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:54.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.081+0000 7fea61600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea5c0d0980 0x7fea5c06dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:54.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.082+0000 7fea61600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea5c15c9f0 0x7fea5c06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:54.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.082+0000 7fea61600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c0cdb00 con 0x7fea5c15c9f0 2024-04-16T16:18:54.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.082+0000 7fea61600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c0cd820 con 0x7fea5c0d0980 2024-04-16T16:18:54.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.082+0000 7fea61600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fea5c0c6170 con 0x7fea5c0d01f0 2024-04-16T16:18:54.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.082+0000 7fea60c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea5c15c9f0 0x7fea5c06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:54.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.082+0000 7fea60c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea5c15c9f0 0x7fea5c06e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41134/0 (socket says 172.21.15.45:41134) 2024-04-16T16:18:54.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.082+0000 7fea60c00640 1 -- 172.21.15.45:0/3428337226 learned_addr learned my addr 172.21.15.45:0/3428337226 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:54.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.082+0000 7fea5be00640 1 --2- 172.21.15.45:0/3428337226 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5c0d01f0 0x7fea5c0c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:54.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.082+0000 7fea5b400640 1 --2- 172.21.15.45:0/3428337226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea5c0d0980 0x7fea5c06dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:54.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.082+0000 7fea60c00640 1 -- 172.21.15.45:0/3428337226 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5c0d01f0 msgr2=0x7fea5c0c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.082+0000 7fea60c00640 1 --2- 172.21.15.45:0/3428337226 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5c0d01f0 0x7fea5c0c5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.082+0000 7fea60c00640 1 -- 172.21.15.45:0/3428337226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea5c0d0980 msgr2=0x7fea5c06dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.082+0000 7fea60c00640 1 --2- 172.21.15.45:0/3428337226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea5c0d0980 0x7fea5c06dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.082+0000 7fea60c00640 1 -- 172.21.15.45:0/3428337226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea5c0cbe80 con 0x7fea5c15c9f0 2024-04-16T16:18:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.082+0000 7fea5b400640 1 --2- 172.21.15.45:0/3428337226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea5c0d0980 0x7fea5c06dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.083+0000 7fea60c00640 1 --2- 172.21.15.45:0/3428337226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea5c15c9f0 0x7fea5c06e4f0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fea54009870 tx=0x7fea540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.083+0000 7fea53e00640 1 -- 172.21.15.45:0/3428337226 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea54040070 con 0x7fea5c15c9f0 2024-04-16T16:18:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.083+0000 7fea53e00640 1 -- 172.21.15.45:0/3428337226 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea540399f0 con 0x7fea5c15c9f0 2024-04-16T16:18:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.083+0000 7fea53e00640 1 -- 172.21.15.45:0/3428337226 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea54035040 con 0x7fea5c15c9f0 2024-04-16T16:18:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.083+0000 7fea61600640 1 -- 172.21.15.45:0/3428337226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea5c0b53e0 con 0x7fea5c15c9f0 2024-04-16T16:18:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.084+0000 7fea61600640 1 -- 172.21.15.45:0/3428337226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea5c0b59a0 con 0x7fea5c15c9f0 2024-04-16T16:18:54.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.085+0000 7fea61600640 1 -- 172.21.15.45:0/3428337226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea5c075780 con 0x7fea5c15c9f0 2024-04-16T16:18:54.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.086+0000 7fea53e00640 1 -- 172.21.15.45:0/3428337226 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7fea54038020 con 0x7fea5c15c9f0 2024-04-16T16:18:54.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.087+0000 7fea53e00640 1 --2- 172.21.15.45:0/3428337226 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fea3c06c980 0x7fea3c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:54.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.087+0000 7fea53e00640 1 -- 172.21.15.45:0/3428337226 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5588+0+0 (secure 0 0 0) 0x7fea540ad470 con 0x7fea5c15c9f0 2024-04-16T16:18:54.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.087+0000 7fea5be00640 1 --2- 172.21.15.45:0/3428337226 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fea3c06c980 0x7fea3c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.089+0000 7fea5be00640 1 --2- 172.21.15.45:0/3428337226 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fea3c06c980 0x7fea3c06ee40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fea4c0073f0 tx=0x7fea4c00b040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:54.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.093+0000 7fea53e00640 1 -- 172.21.15.45:0/3428337226 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fea5407b8b0 con 0x7fea5c15c9f0 2024-04-16T16:18:54.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.304+0000 7fea61600640 1 -- 172.21.15.45:0/3428337226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fea5c0b4d00 con 0x7fea5c15c9f0 2024-04-16T16:18:54.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.311+0000 7fea53e00640 1 -- 172.21.15.45:0/3428337226 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) ==== 104+0+0 (secure 0 0 0) 0x7fea5407b1b0 con 0x7fea5c15c9f0 2024-04-16T16:18:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.314+0000 7fea61600640 1 -- 172.21.15.45:0/3428337226 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fea3c06c980 msgr2=0x7fea3c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.314+0000 7fea61600640 1 --2- 172.21.15.45:0/3428337226 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fea3c06c980 0x7fea3c06ee40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fea4c0073f0 tx=0x7fea4c00b040 comp rx=0 tx=0).stop 2024-04-16T16:18:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.314+0000 7fea61600640 1 -- 172.21.15.45:0/3428337226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea5c15c9f0 msgr2=0x7fea5c06e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.314+0000 7fea61600640 1 --2- 172.21.15.45:0/3428337226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea5c15c9f0 0x7fea5c06e4f0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fea54009870 tx=0x7fea540046c0 comp rx=0 tx=0).stop 2024-04-16T16:18:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.315+0000 7fea61600640 1 -- 172.21.15.45:0/3428337226 shutdown_connections 2024-04-16T16:18:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.315+0000 7fea61600640 1 --2- 172.21.15.45:0/3428337226 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fea3c06c980 0x7fea3c06ee40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:54.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.315+0000 7fea61600640 1 --2- 172.21.15.45:0/3428337226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea5c15c9f0 0x7fea5c06e4f0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:54.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.315+0000 7fea61600640 1 --2- 172.21.15.45:0/3428337226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea5c0d0980 0x7fea5c06dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:54.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.315+0000 7fea61600640 1 --2- 172.21.15.45:0/3428337226 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5c0d01f0 0x7fea5c0c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:54.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.315+0000 7fea61600640 1 -- 172.21.15.45:0/3428337226 >> 172.21.15.45:0/3428337226 conn(0x7fea5c0b4110 msgr2=0x7fea5c0c83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:54.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.315+0000 7fea61600640 1 -- 172.21.15.45:0/3428337226 shutdown_connections 2024-04-16T16:18:54.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.315+0000 7fea61600640 1 -- 172.21.15.45:0/3428337226 wait complete. 2024-04-16T16:18:54.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_client_id' 2024-04-16T16:18:54.497 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:54.496+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:18:54.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.547+0000 7f7b7f200640 1 Processor -- start 2024-04-16T16:18:54.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.548+0000 7f7b7f200640 1 -- start start 2024-04-16T16:18:54.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.549+0000 7f7b7f200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7b800ce9a0 0x7f7b800cb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:54.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.549+0000 7f7b7f200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b8015ecd0 con 0x7f7b800ccb20 2024-04-16T16:18:54.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.549+0000 7f7b7f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b800cd6d0 con 0x7f7b8015e8f0 2024-04-16T16:18:54.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.549+0000 7f7b7f200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f7b800ce2e0 con 0x7f7b800ce9a0 2024-04-16T16:18:54.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.550+0000 7f7b7e800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7b800ce9a0 0x7f7b800cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:54.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.550+0000 7f7b7e800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7b800ce9a0 0x7f7b800cb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41150/0 (socket says 172.21.15.45:41150) 2024-04-16T16:18:54.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.550+0000 7f7b7e800640 1 -- 172.21.15.45:0/1373149919 learned_addr learned my addr 172.21.15.45:0/1373149919 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:54.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.550+0000 7f7b7e800640 1 -- 172.21.15.45:0/1373149919 >> v1:172.21.15.103:6789/0 conn(0x7f7b8015e8f0 legacy=0x7f7b800cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:54.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.550+0000 7f7b7e800640 1 -- 172.21.15.45:0/1373149919 >> v1:172.21.15.45:6789/0 conn(0x7f7b800ccb20 legacy=0x7f7b800cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:54.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.550+0000 7f7b7e800640 1 -- 172.21.15.45:0/1373149919 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b800cd9b0 con 0x7f7b800ce9a0 2024-04-16T16:18:54.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.550+0000 7f7b7d400640 1 -- 172.21.15.45:0/1373149919 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2847406445 0 0) 0x7f7b8015ecd0 con 0x7f7b800ccb20 2024-04-16T16:18:54.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.552+0000 7f7b7e800640 1 --2- 172.21.15.45:0/1373149919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b800ce9a0 0x7f7b800cb340 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f7b78009e50 tx=0x7f7b7802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced3e2c3b64cfba7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:54.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.552+0000 7f7b7d400640 1 -- 172.21.15.45:0/1373149919 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7802cb20 con 0x7f7b800ce9a0 2024-04-16T16:18:54.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.552+0000 7f7b7d400640 1 -- 172.21.15.45:0/1373149919 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b7802ccc0 con 0x7f7b800ce9a0 2024-04-16T16:18:54.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.552+0000 7f7b7d400640 1 -- 172.21.15.45:0/1373149919 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78034610 con 0x7f7b800ce9a0 2024-04-16T16:18:54.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.553+0000 7f7b7f200640 1 -- 172.21.15.45:0/1373149919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b800ce9a0 msgr2=0x7f7b800cb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:54.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.553+0000 7f7b7f200640 1 --2- 172.21.15.45:0/1373149919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b800ce9a0 0x7f7b800cb340 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f7b78009e50 tx=0x7f7b7802bf80 comp rx=0 tx=0).stop 2024-04-16T16:18:54.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.553+0000 7f7b7f200640 1 -- 172.21.15.45:0/1373149919 shutdown_connections 2024-04-16T16:18:54.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.553+0000 7f7b7f200640 1 --2- 172.21.15.45:0/1373149919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b800ce9a0 0x7f7b800cb340 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:54.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.553+0000 7f7b7f200640 1 -- 172.21.15.45:0/1373149919 >> 172.21.15.45:0/1373149919 conn(0x7f7b8006b930 msgr2=0x7f7b8006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:54.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.553+0000 7f7b7f200640 1 -- 172.21.15.45:0/1373149919 shutdown_connections 2024-04-16T16:18:54.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.553+0000 7f7b7f200640 1 -- 172.21.15.45:0/1373149919 wait complete. 2024-04-16T16:18:54.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.555+0000 7f7b7f200640 1 Processor -- start 2024-04-16T16:18:54.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.555+0000 7f7b7f200640 1 -- start start 2024-04-16T16:18:54.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.556+0000 7f7b7f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b800ccb20 0x7f7b8013fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:54.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.556+0000 7f7b7f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b800ce9a0 0x7f7b80160d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:54.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.556+0000 7f7b7f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8015e8f0 0x7f7b800cb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:54.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.556+0000 7f7b7f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7b800cd360 con 0x7f7b8015e8f0 2024-04-16T16:18:54.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.556+0000 7f7b7f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7b800cdf70 con 0x7f7b800ce9a0 2024-04-16T16:18:54.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.556+0000 7f7b7f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7b800cf080 con 0x7f7b800ccb20 2024-04-16T16:18:54.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.556+0000 7f7b7e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8015e8f0 0x7f7b800cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:54.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.556+0000 7f7b7e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8015e8f0 0x7f7b800cb480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41152/0 (socket says 172.21.15.45:41152) 2024-04-16T16:18:54.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.556+0000 7f7b7e800640 1 -- 172.21.15.45:0/981116859 learned_addr learned my addr 172.21.15.45:0/981116859 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:54.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.557+0000 7f7b7de00640 1 --2- 172.21.15.45:0/981116859 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b800ccb20 0x7f7b8013fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:54.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.557+0000 7f7b75400640 1 --2- 172.21.15.45:0/981116859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b800ce9a0 0x7f7b80160d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:54.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.557+0000 7f7b7e800640 1 -- 172.21.15.45:0/981116859 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b800ccb20 msgr2=0x7f7b8013fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:54.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.557+0000 7f7b7e800640 1 --2- 172.21.15.45:0/981116859 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b800ccb20 0x7f7b8013fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:54.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.557+0000 7f7b7e800640 1 -- 172.21.15.45:0/981116859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b800ce9a0 msgr2=0x7f7b80160d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:54.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.557+0000 7f7b7e800640 1 --2- 172.21.15.45:0/981116859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b800ce9a0 0x7f7b80160d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:54.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.557+0000 7f7b7e800640 1 -- 172.21.15.45:0/981116859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b80161810 con 0x7f7b8015e8f0 2024-04-16T16:18:54.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.557+0000 7f7b7e800640 1 --2- 172.21.15.45:0/981116859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8015e8f0 0x7f7b800cb480 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f7b78036040 tx=0x7f7b78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:54.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.557+0000 7f7b76a00640 1 -- 172.21.15.45:0/981116859 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78043030 con 0x7f7b8015e8f0 2024-04-16T16:18:54.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.558+0000 7f7b76a00640 1 -- 172.21.15.45:0/981116859 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b78003990 con 0x7f7b8015e8f0 2024-04-16T16:18:54.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.558+0000 7f7b7f200640 1 -- 172.21.15.45:0/981116859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b80161af0 con 0x7f7b8015e8f0 2024-04-16T16:18:54.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.558+0000 7f7b76a00640 1 -- 172.21.15.45:0/981116859 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78003b60 con 0x7f7b8015e8f0 2024-04-16T16:18:54.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.558+0000 7f7b7f200640 1 -- 172.21.15.45:0/981116859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b8015c340 con 0x7f7b8015e8f0 2024-04-16T16:18:54.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.559+0000 7f7b7f200640 1 -- 172.21.15.45:0/981116859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b80075780 con 0x7f7b8015e8f0 2024-04-16T16:18:54.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.560+0000 7f7b76a00640 1 -- 172.21.15.45:0/981116859 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f7b7802cb30 con 0x7f7b8015e8f0 2024-04-16T16:18:54.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.561+0000 7f7b76a00640 1 --2- 172.21.15.45:0/981116859 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f7b5c06c9d0 0x7f7b5c06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:54.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.561+0000 7f7b76a00640 1 -- 172.21.15.45:0/981116859 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5588+0+0 (secure 0 0 0) 0x7f7b780adf20 con 0x7f7b8015e8f0 2024-04-16T16:18:54.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.562+0000 7f7b7de00640 1 --2- 172.21.15.45:0/981116859 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f7b5c06c9d0 0x7f7b5c06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:54.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.563+0000 7f7b7de00640 1 --2- 172.21.15.45:0/981116859 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f7b5c06c9d0 0x7f7b5c06ee90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7b70000f60 tx=0x7f7b70009040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:54.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.566+0000 7f7b76a00640 1 -- 172.21.15.45:0/981116859 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7b7804d020 con 0x7f7b8015e8f0 2024-04-16T16:18:54.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:18:54.580 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:18:54.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:54.860+0000 7f7b7f200640 1 -- 172.21.15.45:0/981116859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_client_id"]} v 0) -- 0x7f7b800b5b90 con 0x7f7b8015e8f0 2024-04-16T16:18:55.453 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:55.452+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:18:55.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:55.852+0000 7f7b76a00640 1 -- 172.21.15.45:0/981116859 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_client_id"]}]=0 v321) ==== 150+0+0 (secure 0 0 0) 0x7f7b7807c450 con 0x7f7b8015e8f0 2024-04-16T16:18:55.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:55.855+0000 7f7b7f200640 1 -- 172.21.15.45:0/981116859 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f7b5c06c9d0 msgr2=0x7f7b5c06ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:55.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:55.855+0000 7f7b7f200640 1 --2- 172.21.15.45:0/981116859 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f7b5c06c9d0 0x7f7b5c06ee90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7b70000f60 tx=0x7f7b70009040 comp rx=0 tx=0).stop 2024-04-16T16:18:55.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:55.855+0000 7f7b7f200640 1 -- 172.21.15.45:0/981116859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8015e8f0 msgr2=0x7f7b800cb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:55.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:55.855+0000 7f7b7f200640 1 --2- 172.21.15.45:0/981116859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8015e8f0 0x7f7b800cb480 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f7b78036040 tx=0x7f7b78004a40 comp rx=0 tx=0).stop 2024-04-16T16:18:55.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:55.856+0000 7f7b7f200640 1 -- 172.21.15.45:0/981116859 shutdown_connections 2024-04-16T16:18:55.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:55.856+0000 7f7b7f200640 1 --2- 172.21.15.45:0/981116859 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f7b5c06c9d0 0x7f7b5c06ee90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:55.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:55.856+0000 7f7b7f200640 1 --2- 172.21.15.45:0/981116859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8015e8f0 0x7f7b800cb480 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:55.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:55.856+0000 7f7b7f200640 1 --2- 172.21.15.45:0/981116859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b800ce9a0 0x7f7b80160d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:55.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:55.856+0000 7f7b7f200640 1 --2- 172.21.15.45:0/981116859 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b800ccb20 0x7f7b8013fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:55.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:55.856+0000 7f7b7f200640 1 -- 172.21.15.45:0/981116859 >> 172.21.15.45:0/981116859 conn(0x7f7b8006b930 msgr2=0x7f7b8006f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:55.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:55.856+0000 7f7b7f200640 1 -- 172.21.15.45:0/981116859 shutdown_connections 2024-04-16T16:18:55.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:55.856+0000 7f7b7f200640 1 -- 172.21.15.45:0/981116859 wait complete. 2024-04-16T16:18:55.878 INFO:tasks.cephfs_test_runner:test_for_invalid_client_id (tasks.cephfs.test_mds_metrics.TestMDSMetrics) ... ok 2024-04-16T16:18:55.878 INFO:tasks.cephfs_test_runner:Starting test: test_for_invalid_client_ip (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2024-04-16T16:18:55.879 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:18:56.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.112+0000 7ff625000640 1 Processor -- start 2024-04-16T16:18:56.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.112+0000 7ff625000640 1 -- start start 2024-04-16T16:18:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.114+0000 7ff625000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff620162270 0x7ff6200d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.114+0000 7ff625000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6200d3080 con 0x7ff620161e90 2024-04-16T16:18:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.114+0000 7ff625000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6200d3360 con 0x7ff6200d52d0 2024-04-16T16:18:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.114+0000 7ff625000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff6200d2e30 con 0x7ff620162270 2024-04-16T16:18:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.114+0000 7ff61f400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff620161e90 0x7ff6200cf7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47830/0 (socket says 172.21.15.45:47830) 2024-04-16T16:18:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.114+0000 7ff61f400640 1 -- 172.21.15.45:0/873725596 learned_addr learned my addr 172.21.15.45:0/873725596 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:56.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff61ea00640 1 -- 172.21.15.45:0/873725596 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2345091921 0 0) 0x7ff6200d3080 con 0x7ff620161e90 2024-04-16T16:18:56.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff61ea00640 1 -- 172.21.15.45:0/873725596 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff604003520 con 0x7ff620161e90 2024-04-16T16:18:56.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff616a00640 1 --2- 172.21.15.45:0/873725596 >> v2:172.21.15.103:3300/0 conn(0x7ff620162270 0x7ff6200d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff61ea00640 1 -- 172.21.15.45:0/873725596 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3366657830 0 0) 0x7ff604003520 con 0x7ff620161e90 2024-04-16T16:18:56.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff61ea00640 1 -- 172.21.15.45:0/873725596 >> v2:172.21.15.103:3300/0 conn(0x7ff620162270 msgr2=0x7ff6200d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:56.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff61ea00640 1 --2- 172.21.15.45:0/873725596 >> v2:172.21.15.103:3300/0 conn(0x7ff620162270 0x7ff6200d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff61ea00640 1 -- 172.21.15.45:0/873725596 >> v1:172.21.15.103:6789/0 conn(0x7ff6200d52d0 legacy=0x7ff62006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:56.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff61ea00640 1 -- 172.21.15.45:0/873725596 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6200d3640 con 0x7ff620161e90 2024-04-16T16:18:56.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff61ea00640 1 -- 172.21.15.45:0/873725596 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7ff610001670 con 0x7ff620161e90 2024-04-16T16:18:56.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff61ea00640 1 -- 172.21.15.45:0/873725596 >> v1:172.21.15.45:6789/0 conn(0x7ff620161e90 legacy=0x7ff6200cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:56.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff61ea00640 1 --2- 172.21.15.45:0/873725596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff604003a30 0x7ff604003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff61ea00640 1 --2- 172.21.15.45:0/873725596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff604004800 0x7ff604004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff61ea00640 1 --2- 172.21.15.45:0/873725596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6040052e0 0x7ff60400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff61ea00640 1 -- 172.21.15.45:0/873725596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6200d2e30 con 0x7ff604003a30 2024-04-16T16:18:56.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff61ea00640 1 -- 172.21.15.45:0/873725596 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff60400c190 con 0x7ff604004800 2024-04-16T16:18:56.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.115+0000 7ff61ea00640 1 -- 172.21.15.45:0/873725596 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff60400c390 con 0x7ff6040052e0 2024-04-16T16:18:56.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.116+0000 7ff61f400640 1 --2- 172.21.15.45:0/873725596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff604003a30 0x7ff604003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.116+0000 7ff61f400640 1 -- 172.21.15.45:0/873725596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6040052e0 msgr2=0x7ff60400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:56.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.116+0000 7ff61fe00640 1 --2- 172.21.15.45:0/873725596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6040052e0 0x7ff60400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.116+0000 7ff616a00640 1 --2- 172.21.15.45:0/873725596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff604004800 0x7ff604004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.116+0000 7ff61f400640 1 --2- 172.21.15.45:0/873725596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6040052e0 0x7ff60400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.116+0000 7ff61f400640 1 -- 172.21.15.45:0/873725596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff604004800 msgr2=0x7ff604004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:56.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.116+0000 7ff61f400640 1 --2- 172.21.15.45:0/873725596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff604004800 0x7ff604004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.116+0000 7ff61f400640 1 -- 172.21.15.45:0/873725596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff604003520 con 0x7ff604003a30 2024-04-16T16:18:56.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.116+0000 7ff61fe00640 1 -- 172.21.15.45:0/873725596 reap_dead start 2024-04-16T16:18:56.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.117+0000 7ff61f400640 1 --2- 172.21.15.45:0/873725596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff604003a30 0x7ff604003e50 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7ff61000c160 tx=0x7ff604005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7477675ac4693104 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:56.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.117+0000 7ff61ea00640 1 -- 172.21.15.45:0/873725596 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610007c60 con 0x7ff604003a30 2024-04-16T16:18:56.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.117+0000 7ff61ea00640 1 -- 172.21.15.45:0/873725596 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff610030070 con 0x7ff604003a30 2024-04-16T16:18:56.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.118+0000 7ff61ea00640 1 -- 172.21.15.45:0/873725596 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610005d90 con 0x7ff604003a30 2024-04-16T16:18:56.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.118+0000 7ff625000640 1 -- 172.21.15.45:0/873725596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff604003a30 msgr2=0x7ff604003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:56.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.118+0000 7ff625000640 1 --2- 172.21.15.45:0/873725596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff604003a30 0x7ff604003e50 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7ff61000c160 tx=0x7ff604005830 comp rx=0 tx=0).stop 2024-04-16T16:18:56.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.118+0000 7ff625000640 1 -- 172.21.15.45:0/873725596 shutdown_connections 2024-04-16T16:18:56.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.118+0000 7ff625000640 1 --2- 172.21.15.45:0/873725596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff604003a30 0x7ff604003e50 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.118+0000 7ff625000640 1 -- 172.21.15.45:0/873725596 >> 172.21.15.45:0/873725596 conn(0x7ff6200d1940 msgr2=0x7ff6200d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:56.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.118+0000 7ff625000640 1 -- 172.21.15.45:0/873725596 shutdown_connections 2024-04-16T16:18:56.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.118+0000 7ff625000640 1 -- 172.21.15.45:0/873725596 wait complete. 2024-04-16T16:18:56.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.120+0000 7ff625000640 1 Processor -- start 2024-04-16T16:18:56.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.120+0000 7ff625000640 1 -- start start 2024-04-16T16:18:56.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.121+0000 7ff625000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff604003a30 0x7ff62014fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.121+0000 7ff625000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff62015ba30 0x7ff62015be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.121+0000 7ff625000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6200c8380 0x7ff6200c8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.121+0000 7ff625000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6200cf950 con 0x7ff604003a30 2024-04-16T16:18:56.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.121+0000 7ff625000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff6200cf7d0 con 0x7ff62015ba30 2024-04-16T16:18:56.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.121+0000 7ff625000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff6200cfad0 con 0x7ff6200c8380 2024-04-16T16:18:56.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.121+0000 7ff61f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff604003a30 0x7ff62014fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.122+0000 7ff61f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff604003a30 0x7ff62014fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41174/0 (socket says 172.21.15.45:41174) 2024-04-16T16:18:56.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.122+0000 7ff61f400640 1 -- 172.21.15.45:0/4074104887 learned_addr learned my addr 172.21.15.45:0/4074104887 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:56.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.122+0000 7ff616a00640 1 --2- 172.21.15.45:0/4074104887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff62015ba30 0x7ff62015be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.122+0000 7ff61fe00640 1 --2- 172.21.15.45:0/4074104887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6200c8380 0x7ff6200c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.122+0000 7ff61f400640 1 -- 172.21.15.45:0/4074104887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6200c8380 msgr2=0x7ff6200c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:56.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.122+0000 7ff61f400640 1 --2- 172.21.15.45:0/4074104887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6200c8380 0x7ff6200c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.122+0000 7ff61f400640 1 -- 172.21.15.45:0/4074104887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff62015ba30 msgr2=0x7ff62015be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:56.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.122+0000 7ff61f400640 1 --2- 172.21.15.45:0/4074104887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff62015ba30 0x7ff62015be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.122+0000 7ff61f400640 1 -- 172.21.15.45:0/4074104887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6200cb160 con 0x7ff604003a30 2024-04-16T16:18:56.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.122+0000 7ff61f400640 1 --2- 172.21.15.45:0/4074104887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff604003a30 0x7ff62014fd30 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7ff61000c280 tx=0x7ff610001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:56.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.123+0000 7ff617e00640 1 -- 172.21.15.45:0/4074104887 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610045070 con 0x7ff604003a30 2024-04-16T16:18:56.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.123+0000 7ff617e00640 1 -- 172.21.15.45:0/4074104887 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6100047e0 con 0x7ff604003a30 2024-04-16T16:18:56.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.123+0000 7ff617e00640 1 -- 172.21.15.45:0/4074104887 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610004bd0 con 0x7ff604003a30 2024-04-16T16:18:56.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.123+0000 7ff625000640 1 -- 172.21.15.45:0/4074104887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6200c8f60 con 0x7ff604003a30 2024-04-16T16:18:56.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.123+0000 7ff625000640 1 -- 172.21.15.45:0/4074104887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6200c9520 con 0x7ff604003a30 2024-04-16T16:18:56.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.124+0000 7ff625000640 1 -- 172.21.15.45:0/4074104887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff620075780 con 0x7ff604003a30 2024-04-16T16:18:56.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.125+0000 7ff617e00640 1 -- 172.21.15.45:0/4074104887 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7ff610007070 con 0x7ff604003a30 2024-04-16T16:18:56.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.126+0000 7ff617e00640 1 --2- 172.21.15.45:0/4074104887 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff5fc06c980 0x7ff5fc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.126+0000 7ff617e00640 1 -- 172.21.15.45:0/4074104887 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5588+0+0 (secure 0 0 0) 0x7ff6100ac970 con 0x7ff604003a30 2024-04-16T16:18:56.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.127+0000 7ff616a00640 1 --2- 172.21.15.45:0/4074104887 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff5fc06c980 0x7ff5fc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.128+0000 7ff616a00640 1 --2- 172.21.15.45:0/4074104887 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff5fc06c980 0x7ff5fc06ee40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff6200caba0 tx=0x7ff608008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:56.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.132+0000 7ff617e00640 1 -- 172.21.15.45:0/4074104887 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff61007adb0 con 0x7ff604003a30 2024-04-16T16:18:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.344+0000 7ff625000640 1 -- 172.21.15.45:0/4074104887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff62006dae0 con 0x7ff604003a30 2024-04-16T16:18:56.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.346+0000 7ff617e00640 1 -- 172.21.15.45:0/4074104887 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+13975 (secure 0 0 0) 0x7ff61007a6b0 con 0x7ff604003a30 2024-04-16T16:18:56.346 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:56.347 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":90,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:18:53.819672+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/3813781309":"2024-04-17T16:18:50.755068+0000","172.21.15.103:6833/1293636998":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/2879899995":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/766691889":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/1187061250":"2024-04-17T16:18:43.992878+0000","172.21.15.103:6832/1293636998":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/4007272017":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/3489855481":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/674186192":"2024-04-17T16:18:37.658582+0000","172.21.15.103:6833/3728775368":"2024-04-17T16:18:37.658582+0000","172.21.15.103:6832/3728775368":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/4160076462":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/1325022358":"2024-04-17T16:18:37.658582+0000","172.21.15.45:6833/3813781309":"2024-04-17T16:18:50.755068+0000","172.21.15.103:0/981309315":"2024-04-17T16:18:37.658582+0000","172.21.15.103:0/1207000852":"2024-04-17T16:18:43.992878+0000","172.21.15.103:0/3673976854":"2024-04-17T16:18:37.658582+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:18:56.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.349+0000 7ff625000640 1 -- 172.21.15.45:0/4074104887 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff5fc06c980 msgr2=0x7ff5fc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:56.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.349+0000 7ff625000640 1 --2- 172.21.15.45:0/4074104887 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff5fc06c980 0x7ff5fc06ee40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff6200caba0 tx=0x7ff608008040 comp rx=0 tx=0).stop 2024-04-16T16:18:56.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.349+0000 7ff625000640 1 -- 172.21.15.45:0/4074104887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff604003a30 msgr2=0x7ff62014fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:56.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.349+0000 7ff625000640 1 --2- 172.21.15.45:0/4074104887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff604003a30 0x7ff62014fd30 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7ff61000c280 tx=0x7ff610001970 comp rx=0 tx=0).stop 2024-04-16T16:18:56.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.349+0000 7ff625000640 1 -- 172.21.15.45:0/4074104887 shutdown_connections 2024-04-16T16:18:56.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.349+0000 7ff625000640 1 --2- 172.21.15.45:0/4074104887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6200c8380 0x7ff6200c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.349+0000 7ff625000640 1 --2- 172.21.15.45:0/4074104887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff62015ba30 0x7ff62015be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.349+0000 7ff625000640 1 --2- 172.21.15.45:0/4074104887 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff5fc06c980 0x7ff5fc06ee40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.349+0000 7ff625000640 1 --2- 172.21.15.45:0/4074104887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff604003a30 0x7ff62014fd30 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.349+0000 7ff625000640 1 -- 172.21.15.45:0/4074104887 >> 172.21.15.45:0/4074104887 conn(0x7ff6200d1940 msgr2=0x7ff6200b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:56.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.350+0000 7ff625000640 1 -- 172.21.15.45:0/4074104887 shutdown_connections 2024-04-16T16:18:56.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.350+0000 7ff625000640 1 -- 172.21.15.45:0/4074104887 wait complete. 2024-04-16T16:18:56.368 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:18:56.369 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:18:56.417 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:56.417+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:18:56.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.590+0000 7f78f3200640 1 Processor -- start 2024-04-16T16:18:56.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.590+0000 7f78f3200640 1 -- start start 2024-04-16T16:18:56.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.592+0000 7f78f3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f415c560 0x7f78f406f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.592+0000 7f78f3200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f78f40ce570 0x7f78f415f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.592+0000 7f78f3200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f40cf2a0 con 0x7f78f406fc60 2024-04-16T16:18:56.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.592+0000 7f78f3200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f78f415f0b0 con 0x7f78f40ce570 2024-04-16T16:18:56.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.592+0000 7f78f3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f78f415edc0 con 0x7f78f415c560 2024-04-16T16:18:56.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.592+0000 7f78f2800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f78f406fc60 0x7f78f40cd160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47840/0 (socket says 172.21.15.45:47840) 2024-04-16T16:18:56.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.592+0000 7f78f2800640 1 -- 172.21.15.45:0/2343210438 learned_addr learned my addr 172.21.15.45:0/2343210438 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:56.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.593+0000 7f78f1400640 1 --2- 172.21.15.45:0/2343210438 >> v2:172.21.15.103:3300/0 conn(0x7f78f40ce570 0x7f78f415f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.593+0000 7f78f1e00640 1 --2- 172.21.15.45:0/2343210438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f415c560 0x7f78f406f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.593+0000 7f78f0a00640 1 -- 172.21.15.45:0/2343210438 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2679710149 0 0) 0x7f78f40cf2a0 con 0x7f78f406fc60 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.593+0000 7f78f0a00640 1 -- 172.21.15.45:0/2343210438 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78d8003520 con 0x7f78f406fc60 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.593+0000 7f78f0a00640 1 -- 172.21.15.45:0/2343210438 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 337220480 0 0) 0x7f78d8003520 con 0x7f78f406fc60 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.593+0000 7f78f0a00640 1 -- 172.21.15.45:0/2343210438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f415c560 msgr2=0x7f78f406f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.593+0000 7f78f0a00640 1 --2- 172.21.15.45:0/2343210438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f415c560 0x7f78f406f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.593+0000 7f78f0a00640 1 -- 172.21.15.45:0/2343210438 >> v2:172.21.15.103:3300/0 conn(0x7f78f40ce570 msgr2=0x7f78f415f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.593+0000 7f78f0a00640 1 --2- 172.21.15.45:0/2343210438 >> v2:172.21.15.103:3300/0 conn(0x7f78f40ce570 0x7f78f415f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.593+0000 7f78f0a00640 1 -- 172.21.15.45:0/2343210438 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f40cea00 con 0x7f78f406fc60 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.593+0000 7f78f0a00640 1 -- 172.21.15.45:0/2343210438 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f78ec0019f0 con 0x7f78f406fc60 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.593+0000 7f78f0a00640 1 -- 172.21.15.45:0/2343210438 >> v1:172.21.15.45:6789/0 conn(0x7f78f406fc60 legacy=0x7f78f40cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.593+0000 7f78f0a00640 1 --2- 172.21.15.45:0/2343210438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f415c560 0x7f78d80039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.593+0000 7f78f0a00640 1 --2- 172.21.15.45:0/2343210438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d8003f50 0x7f78d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.593+0000 7f78f0a00640 1 --2- 172.21.15.45:0/2343210438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78d8004d40 0x7f78d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.594+0000 7f78f0a00640 1 -- 172.21.15.45:0/2343210438 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f78f415f0b0 con 0x7f78d8003f50 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.594+0000 7f78f0a00640 1 -- 172.21.15.45:0/2343210438 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f78f415edc0 con 0x7f78d8004d40 2024-04-16T16:18:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.594+0000 7f78f0a00640 1 -- 172.21.15.45:0/2343210438 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f78d800bad0 con 0x7f78f415c560 2024-04-16T16:18:56.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.594+0000 7f78f1400640 1 --2- 172.21.15.45:0/2343210438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d8003f50 0x7f78d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.594+0000 7f78f1e00640 1 --2- 172.21.15.45:0/2343210438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f415c560 0x7f78d80039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.594+0000 7f78f1400640 1 -- 172.21.15.45:0/2343210438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f415c560 msgr2=0x7f78d80039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:56.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.594+0000 7f78f1400640 1 --2- 172.21.15.45:0/2343210438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f415c560 0x7f78d80039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.594+0000 7f78f2800640 1 --2- 172.21.15.45:0/2343210438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78d8004d40 0x7f78d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.594+0000 7f78f1400640 1 -- 172.21.15.45:0/2343210438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78d8004d40 msgr2=0x7f78d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:56.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.594+0000 7f78f1400640 1 --2- 172.21.15.45:0/2343210438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78d8004d40 0x7f78d80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.594+0000 7f78f1400640 1 -- 172.21.15.45:0/2343210438 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d8003520 con 0x7f78d8003f50 2024-04-16T16:18:56.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.596+0000 7f78f1400640 1 --2- 172.21.15.45:0/2343210438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d8003f50 0x7f78d8004800 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f78e000ba30 tx=0x7f78d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc58aecf131a28fe server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:56.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.596+0000 7f78f0a00640 1 -- 172.21.15.45:0/2343210438 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e0035070 con 0x7f78d8003f50 2024-04-16T16:18:56.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.596+0000 7f78f0a00640 1 -- 172.21.15.45:0/2343210438 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e0030660 con 0x7f78d8003f50 2024-04-16T16:18:56.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.596+0000 7f78f0a00640 1 -- 172.21.15.45:0/2343210438 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e000b600 con 0x7f78d8003f50 2024-04-16T16:18:56.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.597+0000 7f78f3200640 1 -- 172.21.15.45:0/2343210438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d8003f50 msgr2=0x7f78d8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:56.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.597+0000 7f78f3200640 1 --2- 172.21.15.45:0/2343210438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d8003f50 0x7f78d8004800 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f78e000ba30 tx=0x7f78d8005830 comp rx=0 tx=0).stop 2024-04-16T16:18:56.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.597+0000 7f78f2800640 1 -- 172.21.15.45:0/2343210438 reap_dead start 2024-04-16T16:18:56.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.597+0000 7f78f3200640 1 -- 172.21.15.45:0/2343210438 shutdown_connections 2024-04-16T16:18:56.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.597+0000 7f78f3200640 1 -- 172.21.15.45:0/2343210438 >> 172.21.15.45:0/2343210438 conn(0x7f78f4067e00 msgr2=0x7f78f406b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:56.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.597+0000 7f78f3200640 1 -- 172.21.15.45:0/2343210438 shutdown_connections 2024-04-16T16:18:56.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.597+0000 7f78f3200640 1 -- 172.21.15.45:0/2343210438 wait complete. 2024-04-16T16:18:56.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.599+0000 7f78f3200640 1 Processor -- start 2024-04-16T16:18:56.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.599+0000 7f78f3200640 1 -- start start 2024-04-16T16:18:56.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.600+0000 7f78f3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d8003f50 0x7f78f4148100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.600+0000 7f78f3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f4158940 0x7f78f4158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.600+0000 7f78f3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78f4156eb0 0x7f78f4157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.600+0000 7f78f3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f78f40cd870 con 0x7f78d8003f50 2024-04-16T16:18:56.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.600+0000 7f78f3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f78f40cefc0 con 0x7f78f4156eb0 2024-04-16T16:18:56.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.600+0000 7f78f3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f78f40cd9f0 con 0x7f78f4158940 2024-04-16T16:18:56.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.600+0000 7f78f1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d8003f50 0x7f78f4148100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.600+0000 7f78f1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d8003f50 0x7f78f4148100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41180/0 (socket says 172.21.15.45:41180) 2024-04-16T16:18:56.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.600+0000 7f78f1e00640 1 -- 172.21.15.45:0/1522434844 learned_addr learned my addr 172.21.15.45:0/1522434844 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:56.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.601+0000 7f78f2800640 1 --2- 172.21.15.45:0/1522434844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78f4156eb0 0x7f78f4157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.601+0000 7f78f1e00640 1 -- 172.21.15.45:0/1522434844 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f4158940 msgr2=0x7f78f4158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:56.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.601+0000 7f78f1e00640 1 --2- 172.21.15.45:0/1522434844 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f4158940 0x7f78f4158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.601+0000 7f78f1e00640 1 -- 172.21.15.45:0/1522434844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78f4156eb0 msgr2=0x7f78f4157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:56.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.601+0000 7f78f1e00640 1 --2- 172.21.15.45:0/1522434844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78f4156eb0 0x7f78f4157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.601+0000 7f78f1e00640 1 -- 172.21.15.45:0/1522434844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f41593c0 con 0x7f78d8003f50 2024-04-16T16:18:56.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.601+0000 7f78f2800640 1 --2- 172.21.15.45:0/1522434844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78f4156eb0 0x7f78f4157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:56.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.601+0000 7f78f1e00640 1 --2- 172.21.15.45:0/1522434844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d8003f50 0x7f78f4148100 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f78e4011600 tx=0x7f78e4011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:56.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.601+0000 7f78ea000640 1 -- 172.21.15.45:0/1522434844 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e400f920 con 0x7f78d8003f50 2024-04-16T16:18:56.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.601+0000 7f78ea000640 1 -- 172.21.15.45:0/1522434844 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e400fac0 con 0x7f78d8003f50 2024-04-16T16:18:56.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.601+0000 7f78f3200640 1 -- 172.21.15.45:0/1522434844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f41596a0 con 0x7f78d8003f50 2024-04-16T16:18:56.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.601+0000 7f78ea000640 1 -- 172.21.15.45:0/1522434844 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e400fe70 con 0x7f78d8003f50 2024-04-16T16:18:56.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.602+0000 7f78f3200640 1 -- 172.21.15.45:0/1522434844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f4152d60 con 0x7f78d8003f50 2024-04-16T16:18:56.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.603+0000 7f78f3200640 1 -- 172.21.15.45:0/1522434844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f4157870 con 0x7f78d8003f50 2024-04-16T16:18:56.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.604+0000 7f78ea000640 1 -- 172.21.15.45:0/1522434844 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f78e4015460 con 0x7f78d8003f50 2024-04-16T16:18:56.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.605+0000 7f78ea000640 1 --2- 172.21.15.45:0/1522434844 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f78d406c980 0x7f78d406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:56.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.605+0000 7f78ea000640 1 -- 172.21.15.45:0/1522434844 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5588+0+0 (secure 0 0 0) 0x7f78e408c4b0 con 0x7f78d8003f50 2024-04-16T16:18:56.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.606+0000 7f78f1400640 1 --2- 172.21.15.45:0/1522434844 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f78d406c980 0x7f78d406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:56.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.607+0000 7f78f1400640 1 --2- 172.21.15.45:0/1522434844 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f78d406c980 0x7f78d406ee40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f78e0037000 tx=0x7f78e0032000 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:56.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.609+0000 7f78ea000640 1 -- 172.21.15.45:0/1522434844 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78e4019030 con 0x7f78d8003f50 2024-04-16T16:18:56.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.820+0000 7f78f3200640 1 -- 172.21.15.45:0/1522434844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f78f4075780 con 0x7f78d8003f50 2024-04-16T16:18:56.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.821+0000 7f78ea000640 1 -- 172.21.15.45:0/1522434844 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f78e405a860 con 0x7f78d8003f50 2024-04-16T16:18:56.821 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:18:56.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.824+0000 7f78f3200640 1 -- 172.21.15.45:0/1522434844 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f78d406c980 msgr2=0x7f78d406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:56.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.824+0000 7f78f3200640 1 --2- 172.21.15.45:0/1522434844 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f78d406c980 0x7f78d406ee40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f78e0037000 tx=0x7f78e0032000 comp rx=0 tx=0).stop 2024-04-16T16:18:56.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.824+0000 7f78f3200640 1 -- 172.21.15.45:0/1522434844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d8003f50 msgr2=0x7f78f4148100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:56.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.824+0000 7f78f3200640 1 --2- 172.21.15.45:0/1522434844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d8003f50 0x7f78f4148100 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f78e4011600 tx=0x7f78e4011ac0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.825+0000 7f78f3200640 1 -- 172.21.15.45:0/1522434844 shutdown_connections 2024-04-16T16:18:56.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.825+0000 7f78f3200640 1 --2- 172.21.15.45:0/1522434844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78f4156eb0 0x7f78f4157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.825+0000 7f78f3200640 1 --2- 172.21.15.45:0/1522434844 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f78f4158940 0x7f78f4158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.825+0000 7f78f3200640 1 --2- 172.21.15.45:0/1522434844 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f78d406c980 0x7f78d406ee40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.825+0000 7f78f3200640 1 --2- 172.21.15.45:0/1522434844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d8003f50 0x7f78f4148100 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:56.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.825+0000 7f78f3200640 1 -- 172.21.15.45:0/1522434844 >> 172.21.15.45:0/1522434844 conn(0x7f78f4067e00 msgr2=0x7f78f415ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:56.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.825+0000 7f78f3200640 1 -- 172.21.15.45:0/1522434844 shutdown_connections 2024-04-16T16:18:56.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:56.825+0000 7f78f3200640 1 -- 172.21.15.45:0/1522434844 wait complete. 2024-04-16T16:18:56.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_client_ip' 2024-04-16T16:18:57.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.071+0000 7febaf200640 1 Processor -- start 2024-04-16T16:18:57.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.072+0000 7febaf200640 1 -- start start 2024-04-16T16:18:57.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.073+0000 7febaf200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7febb0161e90 0x7febb00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:57.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.073+0000 7febaf200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7febb0162270 0x7febb00d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:57.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.073+0000 7febaf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb00d52d0 0x7febb006ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:57.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.073+0000 7febaf200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7febb00d2e30 con 0x7febb0162270 2024-04-16T16:18:57.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.073+0000 7febaf200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7febb00d3080 con 0x7febb0161e90 2024-04-16T16:18:57.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.073+0000 7febaf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7febb00d3360 con 0x7febb00d52d0 2024-04-16T16:18:57.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.073+0000 7feba5400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7febb0162270 0x7febb00d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:57.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.073+0000 7feba5400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7febb0162270 0x7febb00d3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41182/0 (socket says 172.21.15.45:41182) 2024-04-16T16:18:57.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.073+0000 7feba5400640 1 -- 172.21.15.45:0/2185886282 learned_addr learned my addr 172.21.15.45:0/2185886282 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:57.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.074+0000 7febae800640 1 --2- 172.21.15.45:0/2185886282 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb00d52d0 0x7febb006ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:57.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.074+0000 7febade00640 1 --2- 172.21.15.45:0/2185886282 >> v2:172.21.15.103:3300/0 conn(0x7febb0161e90 0x7febb00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:57.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.074+0000 7feba5400640 1 -- 172.21.15.45:0/2185886282 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb00d52d0 msgr2=0x7febb006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:57.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.074+0000 7feba5400640 1 --2- 172.21.15.45:0/2185886282 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb00d52d0 0x7febb006ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:57.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.074+0000 7feba5400640 1 -- 172.21.15.45:0/2185886282 >> v2:172.21.15.103:3300/0 conn(0x7febb0161e90 msgr2=0x7febb00cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:57.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.074+0000 7feba5400640 1 --2- 172.21.15.45:0/2185886282 >> v2:172.21.15.103:3300/0 conn(0x7febb0161e90 0x7febb00cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:57.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.074+0000 7feba5400640 1 -- 172.21.15.45:0/2185886282 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb00d3640 con 0x7febb0162270 2024-04-16T16:18:57.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.074+0000 7febade00640 1 --2- 172.21.15.45:0/2185886282 >> v2:172.21.15.103:3300/0 conn(0x7febb0161e90 0x7febb00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:57.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.075+0000 7feba5400640 1 --2- 172.21.15.45:0/2185886282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb0162270 0x7febb00d3e40 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7feb98009870 tx=0x7feb9802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=275502355eb46df6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:57.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.075+0000 7febad400640 1 -- 172.21.15.45:0/2185886282 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb98034070 con 0x7febb0162270 2024-04-16T16:18:57.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.075+0000 7febad400640 1 -- 172.21.15.45:0/2185886282 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb98030840 con 0x7febb0162270 2024-04-16T16:18:57.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.075+0000 7febad400640 1 -- 172.21.15.45:0/2185886282 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb98030c10 con 0x7febb0162270 2024-04-16T16:18:57.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.076+0000 7febaf200640 1 -- 172.21.15.45:0/2185886282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb0162270 msgr2=0x7febb00d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:57.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.076+0000 7febaf200640 1 --2- 172.21.15.45:0/2185886282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb0162270 0x7febb00d3e40 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7feb98009870 tx=0x7feb9802b980 comp rx=0 tx=0).stop 2024-04-16T16:18:57.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.076+0000 7febaf200640 1 -- 172.21.15.45:0/2185886282 shutdown_connections 2024-04-16T16:18:57.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.076+0000 7febaf200640 1 --2- 172.21.15.45:0/2185886282 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb00d52d0 0x7febb006ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:57.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.076+0000 7febaf200640 1 --2- 172.21.15.45:0/2185886282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb0162270 0x7febb00d3e40 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:57.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.076+0000 7febaf200640 1 --2- 172.21.15.45:0/2185886282 >> v2:172.21.15.103:3300/0 conn(0x7febb0161e90 0x7febb00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:57.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.076+0000 7febaf200640 1 -- 172.21.15.45:0/2185886282 >> 172.21.15.45:0/2185886282 conn(0x7febb00d1940 msgr2=0x7febb00d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:57.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.076+0000 7febaf200640 1 -- 172.21.15.45:0/2185886282 shutdown_connections 2024-04-16T16:18:57.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.076+0000 7febaf200640 1 -- 172.21.15.45:0/2185886282 wait complete. 2024-04-16T16:18:57.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.078+0000 7febaf200640 1 Processor -- start 2024-04-16T16:18:57.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.078+0000 7febaf200640 1 -- start start 2024-04-16T16:18:57.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.079+0000 7febaf200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb00d52d0 0x7febb0151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:57.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.079+0000 7febaf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb0161e90 0x7febb00cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:57.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.079+0000 7febaf200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb0162270 0x7febb00b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:57.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.079+0000 7febaf200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7febb00d4380 con 0x7febb00d52d0 2024-04-16T16:18:57.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.079+0000 7febaf200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7febb00cfd10 con 0x7febb0162270 2024-04-16T16:18:57.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.079+0000 7febaf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7febb00d4500 con 0x7febb0161e90 2024-04-16T16:18:57.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.080+0000 7febade00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb00d52d0 0x7febb0151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:57.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.080+0000 7febade00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb00d52d0 0x7febb0151c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41190/0 (socket says 172.21.15.45:41190) 2024-04-16T16:18:57.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.080+0000 7febade00640 1 -- 172.21.15.45:0/3353713310 learned_addr learned my addr 172.21.15.45:0/3353713310 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:57.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.080+0000 7feba5400640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb0161e90 0x7febb00cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:57.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.080+0000 7febae800640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb0162270 0x7febb00b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.080+0000 7febade00640 1 -- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb0161e90 msgr2=0x7febb00cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.080+0000 7febade00640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb0161e90 0x7febb00cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.080+0000 7febade00640 1 -- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb0162270 msgr2=0x7febb00b53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.080+0000 7febade00640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb0162270 0x7febb00b53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.080+0000 7febade00640 1 -- 172.21.15.45:0/3353713310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb00cda60 con 0x7febb00d52d0 2024-04-16T16:18:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.080+0000 7feba5400640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb0161e90 0x7febb00cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.080+0000 7febae800640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb0162270 0x7febb00b53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.080+0000 7febade00640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb00d52d0 0x7febb0151c90 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7feba000c820 tx=0x7feba000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.081+0000 7feba6a00640 1 -- 172.21.15.45:0/3353713310 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba0015070 con 0x7febb00d52d0 2024-04-16T16:18:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.081+0000 7feba6a00640 1 -- 172.21.15.45:0/3353713310 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feba0007a30 con 0x7febb00d52d0 2024-04-16T16:18:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.081+0000 7febaf200640 1 -- 172.21.15.45:0/3353713310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febb00cdd40 con 0x7febb00d52d0 2024-04-16T16:18:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.081+0000 7feba6a00640 1 -- 172.21.15.45:0/3353713310 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba0007e00 con 0x7febb00d52d0 2024-04-16T16:18:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.081+0000 7febaf200640 1 -- 172.21.15.45:0/3353713310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febb00ce300 con 0x7febb00d52d0 2024-04-16T16:18:57.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.082+0000 7febaf200640 1 -- 172.21.15.45:0/3353713310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febb01600d0 con 0x7febb00d52d0 2024-04-16T16:18:57.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.083+0000 7feba6a00640 1 -- 172.21.15.45:0/3353713310 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7feba0018070 con 0x7febb00d52d0 2024-04-16T16:18:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.084+0000 7feba6a00640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7feb8c06c980 0x7feb8c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.084+0000 7feba6a00640 1 -- 172.21.15.45:0/3353713310 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5588+0+0 (secure 0 0 0) 0x7feba008bd30 con 0x7febb00d52d0 2024-04-16T16:18:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.085+0000 7feba5400640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7feb8c06c980 0x7feb8c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:57.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.086+0000 7feba5400640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7feb8c06c980 0x7feb8c06ee40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feb98036000 tx=0x7feb9803b040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:57.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.089+0000 7feba6a00640 1 -- 172.21.15.45:0/3353713310 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7febb01600d0 con 0x7febb00d52d0 2024-04-16T16:18:57.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.375+0000 7febaf200640 1 -- 172.21.15.45:0/3353713310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_client_ip"]} v 0) -- 0x7febb006dae0 con 0x7febb00d52d0 2024-04-16T16:18:57.386 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:57.386+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:18:57.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.866+0000 7feba6a00640 1 -- 172.21.15.45:0/3353713310 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_client_ip"]}]=0 v323) ==== 153+0+0 (secure 0 0 0) 0x7feba0059fd0 con 0x7febb00d52d0 2024-04-16T16:18:57.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.869+0000 7febaf200640 1 -- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7feb8c06c980 msgr2=0x7feb8c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:57.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.869+0000 7febaf200640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7feb8c06c980 0x7feb8c06ee40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feb98036000 tx=0x7feb9803b040 comp rx=0 tx=0).stop 2024-04-16T16:18:57.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.869+0000 7febaf200640 1 -- 172.21.15.45:0/3353713310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb00d52d0 msgr2=0x7febb0151c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:57.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.870+0000 7febaf200640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb00d52d0 0x7febb0151c90 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7feba000c820 tx=0x7feba000cc20 comp rx=0 tx=0).stop 2024-04-16T16:18:57.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.870+0000 7febaf200640 1 -- 172.21.15.45:0/3353713310 shutdown_connections 2024-04-16T16:18:57.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.870+0000 7febaf200640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb0162270 0x7febb00b53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:57.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.870+0000 7febaf200640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb0161e90 0x7febb00cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:57.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.870+0000 7febaf200640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7feb8c06c980 0x7feb8c06ee40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:57.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.870+0000 7febaf200640 1 --2- 172.21.15.45:0/3353713310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb00d52d0 0x7febb0151c90 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:57.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.870+0000 7febaf200640 1 -- 172.21.15.45:0/3353713310 >> 172.21.15.45:0/3353713310 conn(0x7febb00d1940 msgr2=0x7febb00b79f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:57.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.870+0000 7febaf200640 1 -- 172.21.15.45:0/3353713310 shutdown_connections 2024-04-16T16:18:57.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:57.870+0000 7febaf200640 1 -- 172.21.15.45:0/3353713310 wait complete. 2024-04-16T16:18:57.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-16T16:18:58.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.135+0000 7fba09600640 1 Processor -- start 2024-04-16T16:18:58.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.135+0000 7fba09600640 1 -- start start 2024-04-16T16:18:58.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.136+0000 7fba09600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fba040b6310 0x7fba04161db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:58.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.136+0000 7fba09600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba040d3e90 0x7fba041622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:58.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.136+0000 7fba09600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba040b6810 con 0x7fba040d18f0 2024-04-16T16:18:58.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.136+0000 7fba09600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fba040d29f0 con 0x7fba040b6310 2024-04-16T16:18:58.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.136+0000 7fba09600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fba040b6af0 con 0x7fba040d3e90 2024-04-16T16:18:58.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.137+0000 7fba03e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fba040d18f0 0x7fba040ba220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47842/0 (socket says 172.21.15.45:47842) 2024-04-16T16:18:58.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.137+0000 7fba03e00640 1 -- 172.21.15.45:0/1168718823 learned_addr learned my addr 172.21.15.45:0/1168718823 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:58.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.137+0000 7fba08c00640 1 --2- 172.21.15.45:0/1168718823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba040d3e90 0x7fba041622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:58.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.137+0000 7fba03400640 1 --2- 172.21.15.45:0/1168718823 >> v2:172.21.15.103:3300/0 conn(0x7fba040b6310 0x7fba04161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:58.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.137+0000 7fba02a00640 1 -- 172.21.15.45:0/1168718823 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 54894037 0 0) 0x7fba040b6810 con 0x7fba040d18f0 2024-04-16T16:18:58.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.137+0000 7fba02a00640 1 -- 172.21.15.45:0/1168718823 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9e8003520 con 0x7fba040d18f0 2024-04-16T16:18:58.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba02a00640 1 -- 172.21.15.45:0/1168718823 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1391087639 0 0) 0x7fb9e8003520 con 0x7fba040d18f0 2024-04-16T16:18:58.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba02a00640 1 -- 172.21.15.45:0/1168718823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba040d3e90 msgr2=0x7fba041622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:58.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba02a00640 1 --2- 172.21.15.45:0/1168718823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba040d3e90 0x7fba041622f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:58.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba02a00640 1 -- 172.21.15.45:0/1168718823 >> v2:172.21.15.103:3300/0 conn(0x7fba040b6310 msgr2=0x7fba04161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:58.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba02a00640 1 --2- 172.21.15.45:0/1168718823 >> v2:172.21.15.103:3300/0 conn(0x7fba040b6310 0x7fba04161db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:58.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba02a00640 1 -- 172.21.15.45:0/1168718823 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba040b6dd0 con 0x7fba040d18f0 2024-04-16T16:18:58.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba02a00640 1 -- 172.21.15.45:0/1168718823 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fb9f40019f0 con 0x7fba040d18f0 2024-04-16T16:18:58.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba03400640 1 --2- 172.21.15.45:0/1168718823 >> v2:172.21.15.103:3300/0 conn(0x7fba040b6310 0x7fba04161db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:18:58.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba02a00640 1 -- 172.21.15.45:0/1168718823 >> v1:172.21.15.45:6789/0 conn(0x7fba040d18f0 legacy=0x7fba040ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:58.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba02a00640 1 --2- 172.21.15.45:0/1168718823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e80039b0 0x7fb9e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:58.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba02a00640 1 --2- 172.21.15.45:0/1168718823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e8004800 0x7fb9e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:58.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba02a00640 1 --2- 172.21.15.45:0/1168718823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba040d3e90 0x7fb9e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:58.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba02a00640 1 -- 172.21.15.45:0/1168718823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fba040d29f0 con 0x7fb9e8004800 2024-04-16T16:18:58.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba02a00640 1 -- 172.21.15.45:0/1168718823 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fba040b6af0 con 0x7fb9e80039b0 2024-04-16T16:18:58.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba02a00640 1 -- 172.21.15.45:0/1168718823 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e800bac0 con 0x7fba040d3e90 2024-04-16T16:18:58.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba03400640 1 --2- 172.21.15.45:0/1168718823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e8004800 0x7fb9e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:58.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.138+0000 7fba03e00640 1 --2- 172.21.15.45:0/1168718823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e80039b0 0x7fb9e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:58.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.139+0000 7fba08c00640 1 --2- 172.21.15.45:0/1168718823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba040d3e90 0x7fb9e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:58.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.139+0000 7fba03400640 1 -- 172.21.15.45:0/1168718823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba040d3e90 msgr2=0x7fb9e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:58.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.139+0000 7fba03400640 1 --2- 172.21.15.45:0/1168718823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba040d3e90 0x7fb9e80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:58.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.139+0000 7fba03400640 1 -- 172.21.15.45:0/1168718823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e80039b0 msgr2=0x7fb9e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:58.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.139+0000 7fba03400640 1 --2- 172.21.15.45:0/1168718823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e80039b0 0x7fb9e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:58.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.139+0000 7fba03400640 1 -- 172.21.15.45:0/1168718823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e8003520 con 0x7fb9e8004800 2024-04-16T16:18:58.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.140+0000 7fba03400640 1 --2- 172.21.15.45:0/1168718823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e8004800 0x7fb9e8004c80 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fba040b6740 tx=0x7fb9e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d66364b8d565ed8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:58.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.140+0000 7fba02a00640 1 -- 172.21.15.45:0/1168718823 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec00f040 con 0x7fb9e8004800 2024-04-16T16:18:58.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.140+0000 7fba02a00640 1 -- 172.21.15.45:0/1168718823 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9ec004030 con 0x7fb9e8004800 2024-04-16T16:18:58.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.140+0000 7fba02a00640 1 -- 172.21.15.45:0/1168718823 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec004400 con 0x7fb9e8004800 2024-04-16T16:18:58.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.141+0000 7fba09600640 1 -- 172.21.15.45:0/1168718823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e8004800 msgr2=0x7fb9e8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:58.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.141+0000 7fba09600640 1 --2- 172.21.15.45:0/1168718823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e8004800 0x7fb9e8004c80 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fba040b6740 tx=0x7fb9e8005830 comp rx=0 tx=0).stop 2024-04-16T16:18:58.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.141+0000 7fba08c00640 1 -- 172.21.15.45:0/1168718823 reap_dead start 2024-04-16T16:18:58.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.141+0000 7fba09600640 1 -- 172.21.15.45:0/1168718823 shutdown_connections 2024-04-16T16:18:58.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.141+0000 7fba09600640 1 -- 172.21.15.45:0/1168718823 >> 172.21.15.45:0/1168718823 conn(0x7fba0406cf90 msgr2=0x7fba0406d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:58.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.141+0000 7fba09600640 1 -- 172.21.15.45:0/1168718823 shutdown_connections 2024-04-16T16:18:58.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.141+0000 7fba09600640 1 -- 172.21.15.45:0/1168718823 wait complete. 2024-04-16T16:18:58.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.143+0000 7fba09600640 1 Processor -- start 2024-04-16T16:18:58.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.143+0000 7fba09600640 1 -- start start 2024-04-16T16:18:58.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.144+0000 7fba09600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e8004800 0x7fba040ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:58.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.144+0000 7fba09600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba040cc5c0 0x7fba040b8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:58.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.144+0000 7fba09600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba040c9fe0 0x7fba040ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:58.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.144+0000 7fba09600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fba040d1e50 con 0x7fba040cc5c0 2024-04-16T16:18:58.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.144+0000 7fba09600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fba040d1cd0 con 0x7fb9e8004800 2024-04-16T16:18:58.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.144+0000 7fba09600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fba040d1fd0 con 0x7fba040c9fe0 2024-04-16T16:18:58.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.144+0000 7fba03400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba040cc5c0 0x7fba040b8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:58.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.144+0000 7fba03400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba040cc5c0 0x7fba040b8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41218/0 (socket says 172.21.15.45:41218) 2024-04-16T16:18:58.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.144+0000 7fba03400640 1 -- 172.21.15.45:0/493578719 learned_addr learned my addr 172.21.15.45:0/493578719 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:58.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.145+0000 7fba03e00640 1 --2- 172.21.15.45:0/493578719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e8004800 0x7fba040ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:58.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.145+0000 7fba08c00640 1 --2- 172.21.15.45:0/493578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba040c9fe0 0x7fba040ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:58.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.145+0000 7fba03400640 1 -- 172.21.15.45:0/493578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba040c9fe0 msgr2=0x7fba040ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:58.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.145+0000 7fba03400640 1 --2- 172.21.15.45:0/493578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba040c9fe0 0x7fba040ca470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:58.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.145+0000 7fba03400640 1 -- 172.21.15.45:0/493578719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e8004800 msgr2=0x7fba040ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:58.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.145+0000 7fba03400640 1 --2- 172.21.15.45:0/493578719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e8004800 0x7fba040ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:58.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.145+0000 7fba03400640 1 -- 172.21.15.45:0/493578719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba040cd6d0 con 0x7fba040cc5c0 2024-04-16T16:18:58.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.145+0000 7fba08c00640 1 --2- 172.21.15.45:0/493578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba040c9fe0 0x7fba040ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:18:58.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.145+0000 7fba03400640 1 --2- 172.21.15.45:0/493578719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba040cc5c0 0x7fba040b8f20 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec009960 tx=0x7fb9ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:58.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.146+0000 7fb9fbe00640 1 -- 172.21.15.45:0/493578719 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec030600 con 0x7fba040cc5c0 2024-04-16T16:18:58.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.146+0000 7fb9fbe00640 1 -- 172.21.15.45:0/493578719 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9ec047070 con 0x7fba040cc5c0 2024-04-16T16:18:58.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.146+0000 7fb9fbe00640 1 -- 172.21.15.45:0/493578719 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec0309e0 con 0x7fba040cc5c0 2024-04-16T16:18:58.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.146+0000 7fba09600640 1 -- 172.21.15.45:0/493578719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba040cd9b0 con 0x7fba040cc5c0 2024-04-16T16:18:58.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.146+0000 7fba09600640 1 -- 172.21.15.45:0/493578719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba040cdf70 con 0x7fba040cc5c0 2024-04-16T16:18:58.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.147+0000 7fba09600640 1 -- 172.21.15.45:0/493578719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba04156380 con 0x7fba040cc5c0 2024-04-16T16:18:58.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.148+0000 7fb9fbe00640 1 -- 172.21.15.45:0/493578719 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7fb9ec030b80 con 0x7fba040cc5c0 2024-04-16T16:18:58.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.149+0000 7fb9fbe00640 1 --2- 172.21.15.45:0/493578719 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb9e006c980 0x7fb9e006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:58.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.150+0000 7fb9fbe00640 1 -- 172.21.15.45:0/493578719 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5588+0+0 (secure 0 0 0) 0x7fb9ec0addd0 con 0x7fba040cc5c0 2024-04-16T16:18:58.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.150+0000 7fba03e00640 1 --2- 172.21.15.45:0/493578719 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb9e006c980 0x7fb9e006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:58.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.151+0000 7fba03e00640 1 --2- 172.21.15.45:0/493578719 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb9e006c980 0x7fb9e006ee40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4001330 tx=0x7fb9f4009040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:58.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.154+0000 7fb9fbe00640 1 -- 172.21.15.45:0/493578719 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fba04156380 con 0x7fba040cc5c0 2024-04-16T16:18:58.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.358+0000 7fba09600640 1 -- 172.21.15.45:0/493578719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fba040d0db0 con 0x7fba040cc5c0 2024-04-16T16:18:58.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.365+0000 7fb9fbe00640 1 -- 172.21.15.45:0/493578719 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7fb9ec03f070 con 0x7fba040cc5c0 2024-04-16T16:18:58.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.368+0000 7fba09600640 1 -- 172.21.15.45:0/493578719 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb9e006c980 msgr2=0x7fb9e006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:58.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.368+0000 7fba09600640 1 --2- 172.21.15.45:0/493578719 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb9e006c980 0x7fb9e006ee40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4001330 tx=0x7fb9f4009040 comp rx=0 tx=0).stop 2024-04-16T16:18:58.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.368+0000 7fba09600640 1 -- 172.21.15.45:0/493578719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba040cc5c0 msgr2=0x7fba040b8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:58.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.368+0000 7fba09600640 1 --2- 172.21.15.45:0/493578719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba040cc5c0 0x7fba040b8f20 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec009960 tx=0x7fb9ec004730 comp rx=0 tx=0).stop 2024-04-16T16:18:58.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.368+0000 7fba09600640 1 -- 172.21.15.45:0/493578719 shutdown_connections 2024-04-16T16:18:58.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.368+0000 7fba09600640 1 --2- 172.21.15.45:0/493578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba040c9fe0 0x7fba040ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:58.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.369+0000 7fba09600640 1 --2- 172.21.15.45:0/493578719 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb9e006c980 0x7fb9e006ee40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:58.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.369+0000 7fba09600640 1 --2- 172.21.15.45:0/493578719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba040cc5c0 0x7fba040b8f20 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:58.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.369+0000 7fba09600640 1 --2- 172.21.15.45:0/493578719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e8004800 0x7fba040ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:58.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.369+0000 7fba09600640 1 -- 172.21.15.45:0/493578719 >> 172.21.15.45:0/493578719 conn(0x7fba0406cf90 msgr2=0x7fba04160150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:58.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.369+0000 7fba09600640 1 -- 172.21.15.45:0/493578719 shutdown_connections 2024-04-16T16:18:58.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:58.369+0000 7fba09600640 1 -- 172.21.15.45:0/493578719 wait complete. 2024-04-16T16:18:58.389 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-16T16:18:58.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:18:58.395 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:58.395+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:18:58.517 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:18:58.634 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-16T16:18:58.634 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:18:58.634 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:58.697 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:18:58.698 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:18:58.698 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:58.756 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:18:58.756 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:18:58.756 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:58.817 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:18:58.817 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:18:58.817 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:18:58.876 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:18:58.877 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-16T16:18:58.877 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:18:59.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.109+0000 7eff46a00640 1 Processor -- start 2024-04-16T16:18:59.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.109+0000 7eff46a00640 1 -- start start 2024-04-16T16:18:59.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.111+0000 7eff46a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7eff400d52d0 0x7eff4006ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:59.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.111+0000 7eff46a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff400d3080 con 0x7eff40161e90 2024-04-16T16:18:59.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.111+0000 7eff46a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff400d3360 con 0x7eff40162270 2024-04-16T16:18:59.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.111+0000 7eff46a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7eff400d2e30 con 0x7eff400d52d0 2024-04-16T16:18:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.111+0000 7eff45600640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7eff40161e90 0x7eff400cf7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47856/0 (socket says 172.21.15.45:47856) 2024-04-16T16:18:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.111+0000 7eff45600640 1 -- 172.21.15.45:0/2507235164 learned_addr learned my addr 172.21.15.45:0/2507235164 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:18:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.111+0000 7eff46000640 1 --2- 172.21.15.45:0/2507235164 >> v2:172.21.15.45:3300/0 conn(0x7eff400d52d0 0x7eff4006ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:59.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.112+0000 7eff46000640 1 -- 172.21.15.45:0/2507235164 >> v1:172.21.15.103:6789/0 conn(0x7eff40162270 legacy=0x7eff400d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.112+0000 7eff46000640 1 -- 172.21.15.45:0/2507235164 >> v1:172.21.15.45:6789/0 conn(0x7eff40161e90 legacy=0x7eff400cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.112+0000 7eff46000640 1 -- 172.21.15.45:0/2507235164 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff400d3640 con 0x7eff400d52d0 2024-04-16T16:18:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.112+0000 7eff3fe00640 1 -- 172.21.15.45:0/2507235164 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4095279300 0 0) 0x7eff400d3080 con 0x7eff40161e90 2024-04-16T16:18:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.113+0000 7eff46000640 1 --2- 172.21.15.45:0/2507235164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff400d52d0 0x7eff4006ce40 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7eff38009dc0 tx=0x7eff3802be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99226d80e200de8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.113+0000 7eff3fe00640 1 -- 172.21.15.45:0/2507235164 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff38039070 con 0x7eff400d52d0 2024-04-16T16:18:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.113+0000 7eff3fe00640 1 -- 172.21.15.45:0/2507235164 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff3802cc40 con 0x7eff400d52d0 2024-04-16T16:18:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.113+0000 7eff3fe00640 1 -- 172.21.15.45:0/2507235164 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff38034440 con 0x7eff400d52d0 2024-04-16T16:18:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.114+0000 7eff46a00640 1 -- 172.21.15.45:0/2507235164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff400d52d0 msgr2=0x7eff4006ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.114+0000 7eff46a00640 1 --2- 172.21.15.45:0/2507235164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff400d52d0 0x7eff4006ce40 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7eff38009dc0 tx=0x7eff3802be90 comp rx=0 tx=0).stop 2024-04-16T16:18:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.114+0000 7eff46a00640 1 -- 172.21.15.45:0/2507235164 shutdown_connections 2024-04-16T16:18:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.114+0000 7eff46a00640 1 --2- 172.21.15.45:0/2507235164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff400d52d0 0x7eff4006ce40 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.114+0000 7eff46a00640 1 -- 172.21.15.45:0/2507235164 >> 172.21.15.45:0/2507235164 conn(0x7eff400d1940 msgr2=0x7eff400d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.114+0000 7eff46a00640 1 -- 172.21.15.45:0/2507235164 shutdown_connections 2024-04-16T16:18:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.114+0000 7eff46a00640 1 -- 172.21.15.45:0/2507235164 wait complete. 2024-04-16T16:18:59.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.116+0000 7eff46a00640 1 Processor -- start 2024-04-16T16:18:59.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.116+0000 7eff46a00640 1 -- start start 2024-04-16T16:18:59.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.117+0000 7eff46a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff400d52d0 0x7eff4015f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:59.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.117+0000 7eff46a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff40161e90 0x7eff400cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:59.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.117+0000 7eff46a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff40162270 0x7eff401f4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:59.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.117+0000 7eff46a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7eff400d46d0 con 0x7eff400d52d0 2024-04-16T16:18:59.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.117+0000 7eff46a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7eff400d4550 con 0x7eff40162270 2024-04-16T16:18:59.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.117+0000 7eff46a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7eff4006d380 con 0x7eff40161e90 2024-04-16T16:18:59.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.117+0000 7eff45600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff400d52d0 0x7eff4015f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:59.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.117+0000 7eff45600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff400d52d0 0x7eff4015f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41226/0 (socket says 172.21.15.45:41226) 2024-04-16T16:18:59.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.117+0000 7eff45600640 1 -- 172.21.15.45:0/3486868600 learned_addr learned my addr 172.21.15.45:0/3486868600 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:59.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.118+0000 7eff45600640 1 -- 172.21.15.45:0/3486868600 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff40161e90 msgr2=0x7eff400cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:59.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.118+0000 7eff45600640 1 --2- 172.21.15.45:0/3486868600 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff40161e90 0x7eff400cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:59.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.118+0000 7eff45600640 1 -- 172.21.15.45:0/3486868600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff40162270 msgr2=0x7eff401f4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:59.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.118+0000 7eff45600640 1 --2- 172.21.15.45:0/3486868600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff40162270 0x7eff401f4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:59.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.118+0000 7eff45600640 1 -- 172.21.15.45:0/3486868600 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff400cc0b0 con 0x7eff400d52d0 2024-04-16T16:18:59.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.118+0000 7eff45600640 1 --2- 172.21.15.45:0/3486868600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff400d52d0 0x7eff4015f530 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7eff3400c4a0 tx=0x7eff3400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.118+0000 7eff3d600640 1 -- 172.21.15.45:0/3486868600 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff340074e0 con 0x7eff400d52d0 2024-04-16T16:18:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.118+0000 7eff3d600640 1 -- 172.21.15.45:0/3486868600 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff340047e0 con 0x7eff400d52d0 2024-04-16T16:18:59.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.118+0000 7eff3d600640 1 -- 172.21.15.45:0/3486868600 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff34004c10 con 0x7eff400d52d0 2024-04-16T16:18:59.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.118+0000 7eff46a00640 1 -- 172.21.15.45:0/3486868600 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff400cc390 con 0x7eff400d52d0 2024-04-16T16:18:59.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.119+0000 7eff46a00640 1 -- 172.21.15.45:0/3486868600 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff400cc950 con 0x7eff400d52d0 2024-04-16T16:18:59.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.120+0000 7eff46a00640 1 -- 172.21.15.45:0/3486868600 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff40075780 con 0x7eff400d52d0 2024-04-16T16:18:59.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.121+0000 7eff3d600640 1 -- 172.21.15.45:0/3486868600 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7eff34011070 con 0x7eff400d52d0 2024-04-16T16:18:59.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.122+0000 7eff3d600640 1 --2- 172.21.15.45:0/3486868600 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7eff2006c980 0x7eff2006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:59.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.122+0000 7eff3d600640 1 -- 172.21.15.45:0/3486868600 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5588+0+0 (secure 0 0 0) 0x7eff3408b7b0 con 0x7eff400d52d0 2024-04-16T16:18:59.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.123+0000 7eff44c00640 1 --2- 172.21.15.45:0/3486868600 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7eff2006c980 0x7eff2006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:59.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.124+0000 7eff44c00640 1 --2- 172.21.15.45:0/3486868600 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7eff2006c980 0x7eff2006ee40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7eff2c006fd0 tx=0x7eff2c006bc0 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:59.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.128+0000 7eff3d600640 1 -- 172.21.15.45:0/3486868600 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7eff34059c60 con 0x7eff400d52d0 2024-04-16T16:18:59.361 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:18:59.361+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:18:59.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.390+0000 7eff46a00640 1 -- 172.21.15.45:0/3486868600 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff4006dae0 con 0x7eff400d52d0 2024-04-16T16:18:59.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.391+0000 7eff3d600640 1 -- 172.21.15.45:0/3486868600 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+3872 (secure 0 0 0) 0x7eff34059560 con 0x7eff400d52d0 2024-04-16T16:18:59.392 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:18:59.392 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":66,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2514711815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2514711815},{"type":"v1","addr":"172.21.15.103:6836","nonce":2514711815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2189404408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2189404408},{"type":"v1","addr":"172.21.15.103:6837","nonce":2189404408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7179,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/991116998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":991116998},{"type":"v1","addr":"172.21.15.45:6834","nonce":991116998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7587,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4091660845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":4091660845},{"type":"v1","addr":"172.21.15.45:6835","nonce":4091660845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[]} 2024-04-16T16:18:59.392 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 66 2024-04-16T16:18:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.394+0000 7eff46a00640 1 -- 172.21.15.45:0/3486868600 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7eff2006c980 msgr2=0x7eff2006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.394+0000 7eff46a00640 1 --2- 172.21.15.45:0/3486868600 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7eff2006c980 0x7eff2006ee40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7eff2c006fd0 tx=0x7eff2c006bc0 comp rx=0 tx=0).stop 2024-04-16T16:18:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.394+0000 7eff46a00640 1 -- 172.21.15.45:0/3486868600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff400d52d0 msgr2=0x7eff4015f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.394+0000 7eff46a00640 1 --2- 172.21.15.45:0/3486868600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff400d52d0 0x7eff4015f530 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7eff3400c4a0 tx=0x7eff3400c960 comp rx=0 tx=0).stop 2024-04-16T16:18:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.394+0000 7eff46a00640 1 -- 172.21.15.45:0/3486868600 shutdown_connections 2024-04-16T16:18:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.394+0000 7eff46a00640 1 --2- 172.21.15.45:0/3486868600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff40162270 0x7eff401f4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.394+0000 7eff46a00640 1 --2- 172.21.15.45:0/3486868600 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff40161e90 0x7eff400cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.394+0000 7eff46a00640 1 --2- 172.21.15.45:0/3486868600 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7eff2006c980 0x7eff2006ee40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:59.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.394+0000 7eff46a00640 1 --2- 172.21.15.45:0/3486868600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff400d52d0 0x7eff4015f530 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:59.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.394+0000 7eff46a00640 1 -- 172.21.15.45:0/3486868600 >> 172.21.15.45:0/3486868600 conn(0x7eff400d1940 msgr2=0x7eff4015fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:59.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.395+0000 7eff46a00640 1 -- 172.21.15.45:0/3486868600 shutdown_connections 2024-04-16T16:18:59.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.395+0000 7eff46a00640 1 -- 172.21.15.45:0/3486868600 wait complete. 2024-04-16T16:18:59.417 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-16T16:18:59.418 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-16T16:18:59.418 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-16T16:18:59.418 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:18:59.418 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-16T16:18:59.419 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-16T16:18:59.419 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-16T16:18:59.419 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:18:59.419 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-16T16:18:59.419 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-16T16:18:59.420 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-16T16:18:59.420 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:18:59.420 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-16T16:18:59.420 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-16T16:18:59.420 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-16T16:18:59.420 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:18:59.439 INFO:tasks.ceph.mds.a:Stopped 2024-04-16T16:18:59.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-16T16:18:59.441 INFO:tasks.ceph.mds.d:Stopped 2024-04-16T16:18:59.441 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-16T16:18:59.443 INFO:tasks.ceph.mds.b:Stopped 2024-04-16T16:18:59.443 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-16T16:18:59.445 INFO:tasks.ceph.mds.c:Stopped 2024-04-16T16:18:59.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-16T16:18:59.489 INFO:tasks.ceph.mds.d:Started 2024-04-16T16:18:59.489 DEBUG:teuthology.parallel:result is None 2024-04-16T16:18:59.492 INFO:tasks.ceph.mds.a:Started 2024-04-16T16:18:59.492 DEBUG:teuthology.parallel:result is None 2024-04-16T16:18:59.493 INFO:tasks.ceph.mds.b:Started 2024-04-16T16:18:59.494 DEBUG:teuthology.parallel:result is None 2024-04-16T16:18:59.499 INFO:tasks.ceph.mds.c:Started 2024-04-16T16:18:59.499 DEBUG:teuthology.parallel:result is None 2024-04-16T16:18:59.500 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-16T16:18:59.660 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-16T16:18:59.662 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-16T16:18:59.663 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2024-04-16T16:18:59.663 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2024-04-16T16:18:59.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.698+0000 7f480d800640 1 Processor -- start 2024-04-16T16:18:59.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.698+0000 7f480d800640 1 -- start start 2024-04-16T16:18:59.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.699+0000 7f480d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f48080afe20 0x7f480806f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:59.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.699+0000 7f480d800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f480806f550 0x7f48080cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:59.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.699+0000 7f480d800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f480806fab0 con 0x7f480815ca00 2024-04-16T16:18:59.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.699+0000 7f480d800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f480815bf20 con 0x7f480806f550 2024-04-16T16:18:59.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.699+0000 7f480d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f480815c200 con 0x7f48080afe20 2024-04-16T16:18:59.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.699+0000 7f4806000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f480806f550 0x7f48080cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:59.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.699+0000 7f4806000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f480806f550 0x7f48080cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41264/0 (socket says 172.21.15.45:41264) 2024-04-16T16:18:59.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.699+0000 7f4806000640 1 -- 172.21.15.45:0/724316211 learned_addr learned my addr 172.21.15.45:0/724316211 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:59.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.700+0000 7f4806a00640 1 --2- 172.21.15.45:0/724316211 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f48080afe20 0x7f480806f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:59.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.700+0000 7f4806000640 1 -- 172.21.15.45:0/724316211 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f48080afe20 msgr2=0x7f480806f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:59.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.700+0000 7f4806000640 1 --2- 172.21.15.45:0/724316211 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f48080afe20 0x7f480806f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:59.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.700+0000 7f4806000640 1 -- 172.21.15.45:0/724316211 >> v1:172.21.15.45:6789/0 conn(0x7f480815ca00 legacy=0x7f48080b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:18:59.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.700+0000 7f4806000640 1 -- 172.21.15.45:0/724316211 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480815c4e0 con 0x7f480806f550 2024-04-16T16:18:59.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.701+0000 7f4806000640 1 --2- 172.21.15.45:0/724316211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f480806f550 0x7f48080cd8d0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009870 tx=0x7f47f402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97a27f25fc4132a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:59.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.702+0000 7f4805600640 1 -- 172.21.15.45:0/724316211 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4035020 con 0x7f480806f550 2024-04-16T16:18:59.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.702+0000 7f4805600640 1 -- 172.21.15.45:0/724316211 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47f4031640 con 0x7f480806f550 2024-04-16T16:18:59.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.702+0000 7f4805600640 1 -- 172.21.15.45:0/724316211 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f402d040 con 0x7f480806f550 2024-04-16T16:18:59.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.702+0000 7f480d800640 1 -- 172.21.15.45:0/724316211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f480806f550 msgr2=0x7f48080cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:59.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.702+0000 7f480d800640 1 --2- 172.21.15.45:0/724316211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f480806f550 0x7f48080cd8d0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009870 tx=0x7f47f402b8f0 comp rx=0 tx=0).stop 2024-04-16T16:18:59.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.702+0000 7f480d800640 1 -- 172.21.15.45:0/724316211 shutdown_connections 2024-04-16T16:18:59.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.702+0000 7f480d800640 1 --2- 172.21.15.45:0/724316211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f480806f550 0x7f48080cd8d0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:59.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.702+0000 7f480d800640 1 --2- 172.21.15.45:0/724316211 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f48080afe20 0x7f480806f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:59.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.702+0000 7f480d800640 1 -- 172.21.15.45:0/724316211 >> 172.21.15.45:0/724316211 conn(0x7f480806b930 msgr2=0x7f480806bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:18:59.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.702+0000 7f480d800640 1 -- 172.21.15.45:0/724316211 shutdown_connections 2024-04-16T16:18:59.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.703+0000 7f480d800640 1 -- 172.21.15.45:0/724316211 wait complete. 2024-04-16T16:18:59.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.704+0000 7f480d800640 1 Processor -- start 2024-04-16T16:18:59.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.704+0000 7f480d800640 1 -- start start 2024-04-16T16:18:59.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.705+0000 7f480d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f480806f550 0x7f480814bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:59.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.705+0000 7f480d800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48080afe20 0x7f4808153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:59.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.705+0000 7f480d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f480815ca00 0x7f480806df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:59.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.706+0000 7f480d800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f48080b59d0 con 0x7f480815ca00 2024-04-16T16:18:59.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.706+0000 7f480d800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f48080b5850 con 0x7f48080afe20 2024-04-16T16:18:59.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.706+0000 7f480d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f48080cde10 con 0x7f480806f550 2024-04-16T16:18:59.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.706+0000 7f4806a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f480806f550 0x7f480814bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:59.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.706+0000 7f4806a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f480806f550 0x7f480814bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:43846/0 (socket says 172.21.15.45:43846) 2024-04-16T16:18:59.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.706+0000 7f4806a00640 1 -- 172.21.15.45:0/441767591 learned_addr learned my addr 172.21.15.45:0/441767591 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:18:59.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.706+0000 7f4807400640 1 --2- 172.21.15.45:0/441767591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f480815ca00 0x7f480806df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:59.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.706+0000 7f4806000640 1 --2- 172.21.15.45:0/441767591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48080afe20 0x7f4808153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:59.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.706+0000 7f4806a00640 1 -- 172.21.15.45:0/441767591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48080afe20 msgr2=0x7f4808153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:59.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.706+0000 7f4806a00640 1 --2- 172.21.15.45:0/441767591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48080afe20 0x7f4808153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:59.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.706+0000 7f4806a00640 1 -- 172.21.15.45:0/441767591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f480815ca00 msgr2=0x7f480806df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:18:59.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.706+0000 7f4806a00640 1 --2- 172.21.15.45:0/441767591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f480815ca00 0x7f480806df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:18:59.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.706+0000 7f4806a00640 1 -- 172.21.15.45:0/441767591 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480815d040 con 0x7f480806f550 2024-04-16T16:18:59.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.706+0000 7f4806000640 1 --2- 172.21.15.45:0/441767591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48080afe20 0x7f4808153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:18:59.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.707+0000 7f4806a00640 1 --2- 172.21.15.45:0/441767591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f480806f550 0x7f480814bd30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f47f800d800 tx=0x7f47f800dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:59.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.707+0000 7f47fea00640 1 -- 172.21.15.45:0/441767591 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f800b6c0 con 0x7f480806f550 2024-04-16T16:18:59.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.707+0000 7f47fea00640 1 -- 172.21.15.45:0/441767591 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47f8019070 con 0x7f480806f550 2024-04-16T16:18:59.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.707+0000 7f47fea00640 1 -- 172.21.15.45:0/441767591 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f800ba80 con 0x7f480806f550 2024-04-16T16:18:59.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.708+0000 7f480d800640 1 -- 172.21.15.45:0/441767591 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4808155e80 con 0x7f480806f550 2024-04-16T16:18:59.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.708+0000 7f480d800640 1 -- 172.21.15.45:0/441767591 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48081555e0 con 0x7f480806f550 2024-04-16T16:18:59.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.709+0000 7f480d800640 1 -- 172.21.15.45:0/441767591 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4808075780 con 0x7f480806f550 2024-04-16T16:18:59.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.710+0000 7f47fea00640 1 -- 172.21.15.45:0/441767591 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f47f800bc20 con 0x7f480806f550 2024-04-16T16:18:59.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.711+0000 7f47fea00640 1 --2- 172.21.15.45:0/441767591 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f47e806c9b0 0x7f47e806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:18:59.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.711+0000 7f47fea00640 1 -- 172.21.15.45:0/441767591 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(90..90 src has 1..90) ==== 5588+0+0 (secure 0 0 0) 0x7f47f808bf90 con 0x7f480806f550 2024-04-16T16:18:59.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.711+0000 7f4806000640 1 --2- 172.21.15.45:0/441767591 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f47e806c9b0 0x7f47e806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:18:59.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.712+0000 7f4806000640 1 --2- 172.21.15.45:0/441767591 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f47e806c9b0 0x7f47e806ee70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f47f4036000 tx=0x7f47f403c040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:18:59.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.719+0000 7f47fea00640 1 -- 172.21.15.45:0/441767591 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f47f805a3d0 con 0x7f480806f550 2024-04-16T16:18:59.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:18:59.956+0000 7f480d800640 1 -- 172.21.15.45:0/441767591 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f48080cf1b0 con 0x7f480806f550 2024-04-16T16:19:00.327 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:00.327+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:19:00.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:00.930+0000 7f47fea00640 1 -- 172.21.15.45:0/441767591 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v95) ==== 97+0+0 (secure 0 0 0) 0x7f47f8059cd0 con 0x7f480806f550 2024-04-16T16:19:00.930 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2024-04-16T16:19:00.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:00.934+0000 7f480d800640 1 -- 172.21.15.45:0/441767591 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f47e806c9b0 msgr2=0x7f47e806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:00.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:00.934+0000 7f480d800640 1 --2- 172.21.15.45:0/441767591 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f47e806c9b0 0x7f47e806ee70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f47f4036000 tx=0x7f47f403c040 comp rx=0 tx=0).stop 2024-04-16T16:19:00.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:00.934+0000 7f480d800640 1 -- 172.21.15.45:0/441767591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f480806f550 msgr2=0x7f480814bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:00.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:00.934+0000 7f480d800640 1 --2- 172.21.15.45:0/441767591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f480806f550 0x7f480814bd30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f47f800d800 tx=0x7f47f800dcc0 comp rx=0 tx=0).stop 2024-04-16T16:19:00.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:00.934+0000 7f480d800640 1 -- 172.21.15.45:0/441767591 shutdown_connections 2024-04-16T16:19:00.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:00.934+0000 7f480d800640 1 --2- 172.21.15.45:0/441767591 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f47e806c9b0 0x7f47e806ee70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:00.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:00.934+0000 7f480d800640 1 --2- 172.21.15.45:0/441767591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f480815ca00 0x7f480806df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:00.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:00.934+0000 7f480d800640 1 --2- 172.21.15.45:0/441767591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f48080afe20 0x7f4808153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:00.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:00.934+0000 7f480d800640 1 --2- 172.21.15.45:0/441767591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f480806f550 0x7f480814bd30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:00.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:00.934+0000 7f480d800640 1 -- 172.21.15.45:0/441767591 >> 172.21.15.45:0/441767591 conn(0x7f480806b930 msgr2=0x7f48080af120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:00.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:00.934+0000 7f480d800640 1 -- 172.21.15.45:0/441767591 shutdown_connections 2024-04-16T16:19:00.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:00.934+0000 7f480d800640 1 -- 172.21.15.45:0/441767591 wait complete. 2024-04-16T16:19:00.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-16T16:19:01.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.180+0000 7fe9efe00640 1 Processor -- start 2024-04-16T16:19:01.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.180+0000 7fe9efe00640 1 -- start start 2024-04-16T16:19:01.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.182+0000 7fe9efe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe9f00c5d80 0x7fe9f0166600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:01.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.182+0000 7fe9efe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9f015c700 con 0x7fe9f0144ff0 2024-04-16T16:19:01.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.182+0000 7fe9efe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9f015c9c0 con 0x7fe9f006dfb0 2024-04-16T16:19:01.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.182+0000 7fe9efe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe9f015d2f0 con 0x7fe9f00c5d80 2024-04-16T16:19:01.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.182+0000 7fe9eea00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe9f0144ff0 0x7fe9f00cfd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47890/0 (socket says 172.21.15.45:47890) 2024-04-16T16:19:01.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.182+0000 7fe9eea00640 1 -- 172.21.15.45:0/2740061838 learned_addr learned my addr 172.21.15.45:0/2740061838 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:01.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.182+0000 7fe9ef400640 1 --2- 172.21.15.45:0/2740061838 >> v2:172.21.15.103:3300/0 conn(0x7fe9f00c5d80 0x7fe9f0166600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:01.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.182+0000 7fe9ee000640 1 -- 172.21.15.45:0/2740061838 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2832013469 0 0) 0x7fe9f015c700 con 0x7fe9f0144ff0 2024-04-16T16:19:01.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.183+0000 7fe9ee000640 1 -- 172.21.15.45:0/2740061838 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9d4003520 con 0x7fe9f0144ff0 2024-04-16T16:19:01.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.183+0000 7fe9ee000640 1 -- 172.21.15.45:0/2740061838 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2264612105 0 0) 0x7fe9d4003520 con 0x7fe9f0144ff0 2024-04-16T16:19:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.183+0000 7fe9ee000640 1 -- 172.21.15.45:0/2740061838 >> v2:172.21.15.103:3300/0 conn(0x7fe9f00c5d80 msgr2=0x7fe9f0166600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.183+0000 7fe9ee000640 1 --2- 172.21.15.45:0/2740061838 >> v2:172.21.15.103:3300/0 conn(0x7fe9f00c5d80 0x7fe9f0166600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.183+0000 7fe9ee000640 1 -- 172.21.15.45:0/2740061838 >> v1:172.21.15.103:6789/0 conn(0x7fe9f006dfb0 legacy=0x7fe9f00c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.183+0000 7fe9ee000640 1 -- 172.21.15.45:0/2740061838 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f015cca0 con 0x7fe9f0144ff0 2024-04-16T16:19:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.183+0000 7fe9ee000640 1 -- 172.21.15.45:0/2740061838 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fe9e0001670 con 0x7fe9f0144ff0 2024-04-16T16:19:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.183+0000 7fe9ee000640 1 -- 172.21.15.45:0/2740061838 >> v1:172.21.15.45:6789/0 conn(0x7fe9f0144ff0 legacy=0x7fe9f00cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.183+0000 7fe9ee000640 1 --2- 172.21.15.45:0/2740061838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9d4003a00 0x7fe9d4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.183+0000 7fe9ee000640 1 --2- 172.21.15.45:0/2740061838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9d4004800 0x7fe9d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.183+0000 7fe9ee000640 1 --2- 172.21.15.45:0/2740061838 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9d40051c0 0x7fe9d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.183+0000 7fe9ee000640 1 -- 172.21.15.45:0/2740061838 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f015d2f0 con 0x7fe9d4003a00 2024-04-16T16:19:01.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.183+0000 7fe9ee000640 1 -- 172.21.15.45:0/2740061838 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d400c0f0 con 0x7fe9d4004800 2024-04-16T16:19:01.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.183+0000 7fe9ee000640 1 -- 172.21.15.45:0/2740061838 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe9d400c2f0 con 0x7fe9d40051c0 2024-04-16T16:19:01.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.183+0000 7fe9eea00640 1 --2- 172.21.15.45:0/2740061838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9d4003a00 0x7fe9d4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:01.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.184+0000 7fe9e7e00640 1 --2- 172.21.15.45:0/2740061838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9d4004800 0x7fe9d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:01.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.184+0000 7fe9eea00640 1 -- 172.21.15.45:0/2740061838 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9d40051c0 msgr2=0x7fe9d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:01.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.184+0000 7fe9eea00640 1 --2- 172.21.15.45:0/2740061838 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9d40051c0 0x7fe9d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.184+0000 7fe9eea00640 1 -- 172.21.15.45:0/2740061838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9d4004800 msgr2=0x7fe9d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:01.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.184+0000 7fe9eea00640 1 --2- 172.21.15.45:0/2740061838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9d4004800 0x7fe9d4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.184+0000 7fe9eea00640 1 -- 172.21.15.45:0/2740061838 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d4003520 con 0x7fe9d4003a00 2024-04-16T16:19:01.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.184+0000 7fe9ef400640 1 -- 172.21.15.45:0/2740061838 reap_dead start 2024-04-16T16:19:01.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.185+0000 7fe9eea00640 1 --2- 172.21.15.45:0/2740061838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9d4003a00 0x7fe9d4003e40 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fe9e000c110 tx=0x7fe9d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a6c54ab92de4bc server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:01.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.185+0000 7fe9ee000640 1 -- 172.21.15.45:0/2740061838 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0007c40 con 0x7fe9d4003a00 2024-04-16T16:19:01.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.185+0000 7fe9ee000640 1 -- 172.21.15.45:0/2740061838 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9e0030070 con 0x7fe9d4003a00 2024-04-16T16:19:01.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.185+0000 7fe9ee000640 1 -- 172.21.15.45:0/2740061838 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0005d70 con 0x7fe9d4003a00 2024-04-16T16:19:01.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.186+0000 7fe9efe00640 1 -- 172.21.15.45:0/2740061838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9d4003a00 msgr2=0x7fe9d4003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:01.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.186+0000 7fe9efe00640 1 --2- 172.21.15.45:0/2740061838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9d4003a00 0x7fe9d4003e40 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fe9e000c110 tx=0x7fe9d4005830 comp rx=0 tx=0).stop 2024-04-16T16:19:01.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.186+0000 7fe9efe00640 1 -- 172.21.15.45:0/2740061838 shutdown_connections 2024-04-16T16:19:01.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.186+0000 7fe9efe00640 1 --2- 172.21.15.45:0/2740061838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9d4003a00 0x7fe9d4003e40 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.186+0000 7fe9efe00640 1 -- 172.21.15.45:0/2740061838 >> 172.21.15.45:0/2740061838 conn(0x7fe9f006b930 msgr2=0x7fe9f006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:01.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.186+0000 7fe9efe00640 1 -- 172.21.15.45:0/2740061838 shutdown_connections 2024-04-16T16:19:01.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.186+0000 7fe9efe00640 1 -- 172.21.15.45:0/2740061838 wait complete. 2024-04-16T16:19:01.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.188+0000 7fe9efe00640 1 Processor -- start 2024-04-16T16:19:01.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.188+0000 7fe9efe00640 1 -- start start 2024-04-16T16:19:01.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.189+0000 7fe9efe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9d4003a00 0x7fe9f013e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:01.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.189+0000 7fe9efe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9f013c030 0x7fe9f01529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:01.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.189+0000 7fe9efe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9f013d620 0x7fe9f0152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:01.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.189+0000 7fe9efe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f00c5670 con 0x7fe9f013d620 2024-04-16T16:19:01.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.189+0000 7fe9efe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f01453d0 con 0x7fe9f013c030 2024-04-16T16:19:01.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.189+0000 7fe9efe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe9f00c57f0 con 0x7fe9d4003a00 2024-04-16T16:19:01.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.190+0000 7fe9ef400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9f013d620 0x7fe9f0152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:01.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.190+0000 7fe9ef400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9f013d620 0x7fe9f0152f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41284/0 (socket says 172.21.15.45:41284) 2024-04-16T16:19:01.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.190+0000 7fe9ef400640 1 -- 172.21.15.45:0/1389567272 learned_addr learned my addr 172.21.15.45:0/1389567272 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:01.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.190+0000 7fe9eea00640 1 --2- 172.21.15.45:0/1389567272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9d4003a00 0x7fe9f013e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:01.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.190+0000 7fe9e7e00640 1 --2- 172.21.15.45:0/1389567272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9f013c030 0x7fe9f01529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:01.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.190+0000 7fe9ef400640 1 -- 172.21.15.45:0/1389567272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9d4003a00 msgr2=0x7fe9f013e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:01.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.190+0000 7fe9ef400640 1 --2- 172.21.15.45:0/1389567272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9d4003a00 0x7fe9f013e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.190+0000 7fe9ef400640 1 -- 172.21.15.45:0/1389567272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9f013c030 msgr2=0x7fe9f01529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:01.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.190+0000 7fe9ef400640 1 --2- 172.21.15.45:0/1389567272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9f013c030 0x7fe9f01529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.190+0000 7fe9ef400640 1 -- 172.21.15.45:0/1389567272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f015b280 con 0x7fe9f013d620 2024-04-16T16:19:01.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.190+0000 7fe9ef400640 1 --2- 172.21.15.45:0/1389567272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9f013d620 0x7fe9f0152f20 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0071ef0 tx=0x7fe9e800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:01.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.191+0000 7fe9e6a00640 1 -- 172.21.15.45:0/1389567272 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e800ba80 con 0x7fe9f013d620 2024-04-16T16:19:01.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.191+0000 7fe9e6a00640 1 -- 172.21.15.45:0/1389567272 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9e801a070 con 0x7fe9f013d620 2024-04-16T16:19:01.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.191+0000 7fe9efe00640 1 -- 172.21.15.45:0/1389567272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9f013c580 con 0x7fe9f013d620 2024-04-16T16:19:01.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.191+0000 7fe9e6a00640 1 -- 172.21.15.45:0/1389567272 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e800be40 con 0x7fe9f013d620 2024-04-16T16:19:01.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.191+0000 7fe9efe00640 1 -- 172.21.15.45:0/1389567272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9f013cb40 con 0x7fe9f013d620 2024-04-16T16:19:01.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.192+0000 7fe9efe00640 1 -- 172.21.15.45:0/1389567272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9f0075780 con 0x7fe9f013d620 2024-04-16T16:19:01.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.193+0000 7fe9e6a00640 1 -- 172.21.15.45:0/1389567272 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7fe9e8013070 con 0x7fe9f013d620 2024-04-16T16:19:01.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.194+0000 7fe9e6a00640 1 --2- 172.21.15.45:0/1389567272 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fe9cc06c9b0 0x7fe9cc06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:01.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.195+0000 7fe9e6a00640 1 -- 172.21.15.45:0/1389567272 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4857+0+0 (secure 0 0 0) 0x7fe9e808bb40 con 0x7fe9f013d620 2024-04-16T16:19:01.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.195+0000 7fe9eea00640 1 --2- 172.21.15.45:0/1389567272 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fe9cc06c9b0 0x7fe9cc06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:01.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.196+0000 7fe9eea00640 1 --2- 172.21.15.45:0/1389567272 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fe9cc06c9b0 0x7fe9cc06ee70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0034000 tx=0x7fe9e0009030 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:01.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.200+0000 7fe9e6a00640 1 -- 172.21.15.45:0/1389567272 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe9e805a0c0 con 0x7fe9f013d620 2024-04-16T16:19:01.348 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:01.348+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:19:01.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.486+0000 7fe9efe00640 1 -- 172.21.15.45:0/1389567272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fe9f00cfac0 con 0x7fe9f013d620 2024-04-16T16:19:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.487+0000 7fe9e6a00640 1 -- 172.21.15.45:0/1389567272 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) ==== 80+0+5238 (secure 0 0 0) 0x7fe9e80599c0 con 0x7fe9f013d620 2024-04-16T16:19:01.487 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:01.487 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:19:01.487 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2024-04-16T16:19:01.487 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.487 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2024-04-16T16:19:01.488 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mpjReORAAJMoNf4qvfuxPSiV9+VlKfg==", 2024-04-16T16:19:01.488 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.488 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:01.488 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:19:01.488 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:19:01.488 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:01.488 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.488 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2024-04-16T16:19:01.489 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mxHeBIBAA3G8maaK65zaOaqXpAOA6xA==", 2024-04-16T16:19:01.489 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.489 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:01.489 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:19:01.489 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:19:01.489 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:01.489 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.489 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.489 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2024-04-16T16:19:01.490 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5m+AaODxAA08XSjc44/i2KYExYQSukaQ==", 2024-04-16T16:19:01.490 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.490 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:01.490 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:19:01.490 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:19:01.490 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:01.490 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.490 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2024-04-16T16:19:01.491 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mEQ4UMRAAOslSsxQ17bOxPrI6QMlRRw==", 2024-04-16T16:19:01.491 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.491 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:01.491 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:19:01.491 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:19:01.491 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:01.491 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.491 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.491 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2024-04-16T16:19:01.492 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBuox5m3l3OHxAA+UjwZx8Mn6tCu95TTYIG3w==", 2024-04-16T16:19:01.492 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.492 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:01.492 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:01.492 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:01.492 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.492 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.492 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.492 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2024-04-16T16:19:01.492 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxox5mozt1LhAAJL5NtMQdPVBtKOaVlMQBtw==", 2024-04-16T16:19:01.492 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.493 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:01.493 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:01.493 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:01.493 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.493 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2024-04-16T16:19:01.493 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB1ox5mfE7kABAAIvgZ5Wx+eIJ6QNKj2GVdpw==", 2024-04-16T16:19:01.493 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.493 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:01.493 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:01.494 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:01.494 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.494 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2024-04-16T16:19:01.494 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB4ox5mEvxcDRAAO0yXBKvsC7hZ6bpOAJyuag==", 2024-04-16T16:19:01.494 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.494 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:01.494 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:01.494 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:01.495 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.495 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2024-04-16T16:19:01.495 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+ox5mZubrORAAcGJLyRN5Ob9aBnopumnEUg==", 2024-04-16T16:19:01.495 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.495 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:01.495 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:01.495 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:01.495 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.496 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.496 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2024-04-16T16:19:01.496 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCCox5mcRqsChAACktmuUHvqWkp/mHT3UcjRw==", 2024-04-16T16:19:01.496 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.496 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:01.496 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:01.496 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:01.496 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.496 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.496 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2024-04-16T16:19:01.497 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFox5mT52FFhAAChhDDfNZmIMgxE1j90LZpQ==", 2024-04-16T16:19:01.497 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.497 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:01.497 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:01.497 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:01.497 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.497 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.497 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2024-04-16T16:19:01.497 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCIox5mC+EDIxAAxtvM0on00lxTLaoRjN0Gfw==", 2024-04-16T16:19:01.497 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.498 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:01.498 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:01.498 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:01.498 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.498 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.498 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2024-04-16T16:19:01.498 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ==", 2024-04-16T16:19:01.498 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.499 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:01.499 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:19:01.499 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-16T16:19:01.499 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.499 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2024-04-16T16:19:01.499 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mYpVUGhAAYdR+40O6zbWqgbI55F0FYA==", 2024-04-16T16:19:01.499 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.499 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:01.500 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:19:01.500 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:19:01.500 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.500 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.500 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2024-04-16T16:19:01.500 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mWdlUDxAAuO2FsGH1/E+qP1xcyUEe3A==", 2024-04-16T16:19:01.500 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.500 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:01.500 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:19:01.500 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:19:01.501 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.501 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2024-04-16T16:19:01.501 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5miPdMJBAAqjKpctNYu9ot1lff3rouxg==", 2024-04-16T16:19:01.501 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.501 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:01.501 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:19:01.501 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:19:01.501 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.502 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2024-04-16T16:19:01.502 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBpox5mO4PMLBAA/0YbQCfHujgdEzpXs0nOLw==", 2024-04-16T16:19:01.502 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.502 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:19:01.502 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:19:01.502 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:19:01.502 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:01.502 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.503 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2024-04-16T16:19:01.503 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mGevaLBAAPpA6g+Fdi/XHqAqG0QZNag==", 2024-04-16T16:19:01.503 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.503 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:19:01.503 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:19:01.503 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:01.503 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.503 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:01.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:01.503 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2024-04-16T16:19:01.504 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5moiNRIRAAz5JzOrjKNLkcphT+iMo+zw==", 2024-04-16T16:19:01.504 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:01.504 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:19:01.504 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:19:01.504 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:01.504 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.504 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:01.504 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-16T16:19:01.504 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:19:01.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.489+0000 7fe9efe00640 1 -- 172.21.15.45:0/1389567272 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fe9cc06c9b0 msgr2=0x7fe9cc06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:01.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.489+0000 7fe9efe00640 1 --2- 172.21.15.45:0/1389567272 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fe9cc06c9b0 0x7fe9cc06ee70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0034000 tx=0x7fe9e0009030 comp rx=0 tx=0).stop 2024-04-16T16:19:01.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.489+0000 7fe9efe00640 1 -- 172.21.15.45:0/1389567272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9f013d620 msgr2=0x7fe9f0152f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:01.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.489+0000 7fe9efe00640 1 --2- 172.21.15.45:0/1389567272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9f013d620 0x7fe9f0152f20 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0071ef0 tx=0x7fe9e800b480 comp rx=0 tx=0).stop 2024-04-16T16:19:01.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.489+0000 7fe9efe00640 1 -- 172.21.15.45:0/1389567272 shutdown_connections 2024-04-16T16:19:01.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.489+0000 7fe9efe00640 1 --2- 172.21.15.45:0/1389567272 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fe9cc06c9b0 0x7fe9cc06ee70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.489+0000 7fe9efe00640 1 --2- 172.21.15.45:0/1389567272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9f013d620 0x7fe9f0152f20 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.489+0000 7fe9efe00640 1 --2- 172.21.15.45:0/1389567272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9f013c030 0x7fe9f01529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.489+0000 7fe9efe00640 1 --2- 172.21.15.45:0/1389567272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9d4003a00 0x7fe9f013e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.489+0000 7fe9efe00640 1 -- 172.21.15.45:0/1389567272 >> 172.21.15.45:0/1389567272 conn(0x7fe9f006b930 msgr2=0x7fe9f0141ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:01.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.489+0000 7fe9efe00640 1 -- 172.21.15.45:0/1389567272 shutdown_connections 2024-04-16T16:19:01.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.489+0000 7fe9efe00640 1 -- 172.21.15.45:0/1389567272 wait complete. 2024-04-16T16:19:01.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:19:01.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.737+0000 7fb8e7e00640 1 Processor -- start 2024-04-16T16:19:01.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.737+0000 7fb8e7e00640 1 -- start start 2024-04-16T16:19:01.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.738+0000 7fb8e7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb8e8146220 0x7fb8e815ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:01.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.738+0000 7fb8e7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb8e806f7a0 0x7fb8e806df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:01.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.738+0000 7fb8e7e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8e815d360 con 0x7fb8e806e480 2024-04-16T16:19:01.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.738+0000 7fb8e7e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb8e8160410 con 0x7fb8e806f7a0 2024-04-16T16:19:01.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.738+0000 7fb8e7e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb8e815d640 con 0x7fb8e8146220 2024-04-16T16:19:01.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.739+0000 7fb8dfe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb8e806f7a0 0x7fb8e806df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:01.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.739+0000 7fb8e7400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb8e806e480 0x7fb8e81608a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47896/0 (socket says 172.21.15.45:47896) 2024-04-16T16:19:01.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.739+0000 7fb8e7400640 1 -- 172.21.15.45:0/1054018951 learned_addr learned my addr 172.21.15.45:0/1054018951 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:01.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.739+0000 7fb8dfe00640 1 -- 172.21.15.45:0/1054018951 >> v2:172.21.15.103:3300/0 conn(0x7fb8e8146220 msgr2=0x7fb8e815ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:01.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.739+0000 7fb8dfe00640 1 --2- 172.21.15.45:0/1054018951 >> v2:172.21.15.103:3300/0 conn(0x7fb8e8146220 0x7fb8e815ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.739+0000 7fb8dfe00640 1 -- 172.21.15.45:0/1054018951 >> v1:172.21.15.45:6789/0 conn(0x7fb8e806e480 legacy=0x7fb8e81608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:01.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.739+0000 7fb8dfe00640 1 -- 172.21.15.45:0/1054018951 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e815d920 con 0x7fb8e806f7a0 2024-04-16T16:19:01.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.740+0000 7fb8dfe00640 1 --2- 172.21.15.45:0/1054018951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e806f7a0 0x7fb8e806df40 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8160590 tx=0x7fb8d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8532a33fea2f0306 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:01.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.741+0000 7fb8e6000640 1 -- 172.21.15.45:0/1054018951 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0035020 con 0x7fb8e806f7a0 2024-04-16T16:19:01.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.741+0000 7fb8e6000640 1 -- 172.21.15.45:0/1054018951 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d0031830 con 0x7fb8e806f7a0 2024-04-16T16:19:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.741+0000 7fb8e6000640 1 -- 172.21.15.45:0/1054018951 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0031c00 con 0x7fb8e806f7a0 2024-04-16T16:19:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.741+0000 7fb8e7e00640 1 -- 172.21.15.45:0/1054018951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e806f7a0 msgr2=0x7fb8e806df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.741+0000 7fb8e7e00640 1 --2- 172.21.15.45:0/1054018951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e806f7a0 0x7fb8e806df40 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8160590 tx=0x7fb8d002b8d0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.741+0000 7fb8e7e00640 1 -- 172.21.15.45:0/1054018951 shutdown_connections 2024-04-16T16:19:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.741+0000 7fb8e7e00640 1 --2- 172.21.15.45:0/1054018951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e806f7a0 0x7fb8e806df40 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.741+0000 7fb8e7e00640 1 --2- 172.21.15.45:0/1054018951 >> v2:172.21.15.103:3300/0 conn(0x7fb8e8146220 0x7fb8e815ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.741+0000 7fb8e7e00640 1 -- 172.21.15.45:0/1054018951 >> 172.21.15.45:0/1054018951 conn(0x7fb8e806b930 msgr2=0x7fb8e806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.741+0000 7fb8e7e00640 1 -- 172.21.15.45:0/1054018951 shutdown_connections 2024-04-16T16:19:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.741+0000 7fb8e7e00640 1 -- 172.21.15.45:0/1054018951 wait complete. 2024-04-16T16:19:01.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.743+0000 7fb8e7e00640 1 Processor -- start 2024-04-16T16:19:01.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.743+0000 7fb8e7e00640 1 -- start start 2024-04-16T16:19:01.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.744+0000 7fb8e7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb8e806e480 0x7fb8e8142a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:01.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.744+0000 7fb8e7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e806f7a0 0x7fb8e815bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:01.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.744+0000 7fb8e7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8e8146220 0x7fb8e815c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:01.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.744+0000 7fb8e7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e806fda0 con 0x7fb8e806f7a0 2024-04-16T16:19:01.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.744+0000 7fb8e7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e806fc20 con 0x7fb8e8146220 2024-04-16T16:19:01.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.744+0000 7fb8e7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e8160fb0 con 0x7fb8e806e480 2024-04-16T16:19:01.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.745+0000 7fb8dfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e806f7a0 0x7fb8e815bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:01.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.745+0000 7fb8dfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e806f7a0 0x7fb8e815bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41308/0 (socket says 172.21.15.45:41308) 2024-04-16T16:19:01.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.745+0000 7fb8dfe00640 1 -- 172.21.15.45:0/3519313571 learned_addr learned my addr 172.21.15.45:0/3519313571 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:01.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.745+0000 7fb8e6a00640 1 --2- 172.21.15.45:0/3519313571 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb8e806e480 0x7fb8e8142a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:01.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.745+0000 7fb8e7400640 1 --2- 172.21.15.45:0/3519313571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8e8146220 0x7fb8e815c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:01.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.745+0000 7fb8dfe00640 1 -- 172.21.15.45:0/3519313571 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb8e806e480 msgr2=0x7fb8e8142a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:01.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.745+0000 7fb8dfe00640 1 --2- 172.21.15.45:0/3519313571 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb8e806e480 0x7fb8e8142a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.745+0000 7fb8dfe00640 1 -- 172.21.15.45:0/3519313571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8e8146220 msgr2=0x7fb8e815c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:01.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.745+0000 7fb8dfe00640 1 --2- 172.21.15.45:0/3519313571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8e8146220 0x7fb8e815c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.745+0000 7fb8dfe00640 1 -- 172.21.15.45:0/3519313571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e80cdc00 con 0x7fb8e806f7a0 2024-04-16T16:19:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.745+0000 7fb8dfe00640 1 --2- 172.21.15.45:0/3519313571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e806f7a0 0x7fb8e815bca0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0009860 tx=0x7fb8d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.746+0000 7fb8dea00640 1 -- 172.21.15.45:0/3519313571 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0041070 con 0x7fb8e806f7a0 2024-04-16T16:19:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.746+0000 7fb8dea00640 1 -- 172.21.15.45:0/3519313571 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d003a950 con 0x7fb8e806f7a0 2024-04-16T16:19:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.746+0000 7fb8e7e00640 1 -- 172.21.15.45:0/3519313571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8e80cdee0 con 0x7fb8e806f7a0 2024-04-16T16:19:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.746+0000 7fb8dea00640 1 -- 172.21.15.45:0/3519313571 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0035040 con 0x7fb8e806f7a0 2024-04-16T16:19:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.746+0000 7fb8e7e00640 1 -- 172.21.15.45:0/3519313571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8e80ce4a0 con 0x7fb8e806f7a0 2024-04-16T16:19:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.747+0000 7fb8e7e00640 1 -- 172.21.15.45:0/3519313571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8e8075780 con 0x7fb8e806f7a0 2024-04-16T16:19:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.748+0000 7fb8dea00640 1 -- 172.21.15.45:0/3519313571 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7fb8d0038020 con 0x7fb8e806f7a0 2024-04-16T16:19:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.749+0000 7fb8dea00640 1 --2- 172.21.15.45:0/3519313571 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb8c406c9b0 0x7fb8c406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.749+0000 7fb8dea00640 1 -- 172.21.15.45:0/3519313571 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4857+0+0 (secure 0 0 0) 0x7fb8d00ad490 con 0x7fb8e806f7a0 2024-04-16T16:19:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.750+0000 7fb8e6a00640 1 --2- 172.21.15.45:0/3519313571 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb8c406c9b0 0x7fb8c406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:01.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.751+0000 7fb8e6a00640 1 --2- 172.21.15.45:0/3519313571 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb8c406c9b0 0x7fb8c406ee70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800a770 tx=0x7fb8d8008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:01.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.754+0000 7fb8dea00640 1 -- 172.21.15.45:0/3519313571 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb8d007ba90 con 0x7fb8e806f7a0 2024-04-16T16:19:01.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.964+0000 7fb8e7e00640 1 -- 172.21.15.45:0/3519313571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb8e80cff60 con 0x7fb8e806f7a0 2024-04-16T16:19:01.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.966+0000 7fb8dea00640 1 -- 172.21.15.45:0/3519313571 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+12892 (secure 0 0 0) 0x7fb8d007b390 con 0x7fb8e806f7a0 2024-04-16T16:19:01.966 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:01.966 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":95,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:19:00.921572+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:19:01.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.968+0000 7fb8e7e00640 1 -- 172.21.15.45:0/3519313571 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb8c406c9b0 msgr2=0x7fb8c406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:01.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.968+0000 7fb8e7e00640 1 --2- 172.21.15.45:0/3519313571 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb8c406c9b0 0x7fb8c406ee70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800a770 tx=0x7fb8d8008040 comp rx=0 tx=0).stop 2024-04-16T16:19:01.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.968+0000 7fb8e7e00640 1 -- 172.21.15.45:0/3519313571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e806f7a0 msgr2=0x7fb8e815bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:01.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.968+0000 7fb8e7e00640 1 --2- 172.21.15.45:0/3519313571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e806f7a0 0x7fb8e815bca0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0009860 tx=0x7fb8d00046c0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.968+0000 7fb8e7e00640 1 -- 172.21.15.45:0/3519313571 shutdown_connections 2024-04-16T16:19:01.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.968+0000 7fb8e7e00640 1 --2- 172.21.15.45:0/3519313571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8e8146220 0x7fb8e815c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.968+0000 7fb8e7e00640 1 --2- 172.21.15.45:0/3519313571 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb8c406c9b0 0x7fb8c406ee70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.968+0000 7fb8e7e00640 1 --2- 172.21.15.45:0/3519313571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e806f7a0 0x7fb8e815bca0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.968+0000 7fb8e7e00640 1 --2- 172.21.15.45:0/3519313571 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb8e806e480 0x7fb8e8142a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:01.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.968+0000 7fb8e7e00640 1 -- 172.21.15.45:0/3519313571 >> 172.21.15.45:0/3519313571 conn(0x7fb8e806b930 msgr2=0x7fb8e80b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:01.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.968+0000 7fb8e7e00640 1 -- 172.21.15.45:0/3519313571 shutdown_connections 2024-04-16T16:19:01.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:01.968+0000 7fb8e7e00640 1 -- 172.21.15.45:0/3519313571 wait complete. 2024-04-16T16:19:01.987 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:19:01.987 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:19:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.209+0000 7fd0a8600640 1 Processor -- start 2024-04-16T16:19:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.209+0000 7fd0a8600640 1 -- start start 2024-04-16T16:19:02.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.211+0000 7fd0a8600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd0a006e290 0x7fd0a006e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:02.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.211+0000 7fd0a8600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0a00b5e10 con 0x7fd0a00d1030 2024-04-16T16:19:02.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.211+0000 7fd0a8600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0a015ff70 con 0x7fd0a00b5a30 2024-04-16T16:19:02.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.211+0000 7fd0a8600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd0a006ec20 con 0x7fd0a006e290 2024-04-16T16:19:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.211+0000 7fd0a5200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd0a00d1030 0x7fd0a015f860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47898/0 (socket says 172.21.15.45:47898) 2024-04-16T16:19:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.211+0000 7fd0a5200640 1 -- 172.21.15.45:0/1818632005 learned_addr learned my addr 172.21.15.45:0/1818632005 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.211+0000 7fd0a6600640 1 --2- 172.21.15.45:0/1818632005 >> v2:172.21.15.103:3300/0 conn(0x7fd0a006e290 0x7fd0a006e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.211+0000 7fd09fe00640 1 -- 172.21.15.45:0/1818632005 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 497255678 0 0) 0x7fd0a00b5e10 con 0x7fd0a00d1030 2024-04-16T16:19:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.211+0000 7fd09fe00640 1 -- 172.21.15.45:0/1818632005 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd088003520 con 0x7fd0a00d1030 2024-04-16T16:19:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.212+0000 7fd09fe00640 1 -- 172.21.15.45:0/1818632005 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2843298644 0 0) 0x7fd088003520 con 0x7fd0a00d1030 2024-04-16T16:19:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.212+0000 7fd09fe00640 1 -- 172.21.15.45:0/1818632005 >> v2:172.21.15.103:3300/0 conn(0x7fd0a006e290 msgr2=0x7fd0a006e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.212+0000 7fd09fe00640 1 --2- 172.21.15.45:0/1818632005 >> v2:172.21.15.103:3300/0 conn(0x7fd0a006e290 0x7fd0a006e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.212+0000 7fd09fe00640 1 -- 172.21.15.45:0/1818632005 >> v1:172.21.15.103:6789/0 conn(0x7fd0a00b5a30 legacy=0x7fd0a01625e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.212+0000 7fd09fe00640 1 -- 172.21.15.45:0/1818632005 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a0162e70 con 0x7fd0a00d1030 2024-04-16T16:19:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.212+0000 7fd09fe00640 1 -- 172.21.15.45:0/1818632005 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fd08c001670 con 0x7fd0a00d1030 2024-04-16T16:19:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.212+0000 7fd09fe00640 1 -- 172.21.15.45:0/1818632005 >> v1:172.21.15.45:6789/0 conn(0x7fd0a00d1030 legacy=0x7fd0a015f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.212+0000 7fd09fe00640 1 --2- 172.21.15.45:0/1818632005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd088003a00 0x7fd088003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.212+0000 7fd09fe00640 1 --2- 172.21.15.45:0/1818632005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd088004800 0x7fd088004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.212+0000 7fd09fe00640 1 --2- 172.21.15.45:0/1818632005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0880051c0 0x7fd08800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.212+0000 7fd09fe00640 1 -- 172.21.15.45:0/1818632005 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a006ec20 con 0x7fd0880051c0 2024-04-16T16:19:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.212+0000 7fd09fe00640 1 -- 172.21.15.45:0/1818632005 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd08800c090 con 0x7fd088004800 2024-04-16T16:19:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.212+0000 7fd09fe00640 1 -- 172.21.15.45:0/1818632005 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd08800c290 con 0x7fd088003a00 2024-04-16T16:19:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.212+0000 7fd0a6600640 1 --2- 172.21.15.45:0/1818632005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0880051c0 0x7fd08800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.213+0000 7fd0a5200640 1 --2- 172.21.15.45:0/1818632005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd088004800 0x7fd088004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.213+0000 7fd0a6600640 1 -- 172.21.15.45:0/1818632005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd088003a00 msgr2=0x7fd088003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.213+0000 7fd0a6600640 1 --2- 172.21.15.45:0/1818632005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd088003a00 0x7fd088003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.213+0000 7fd0a6600640 1 -- 172.21.15.45:0/1818632005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd088004800 msgr2=0x7fd088004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.213+0000 7fd0a6600640 1 --2- 172.21.15.45:0/1818632005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd088004800 0x7fd088004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.213+0000 7fd0a6600640 1 -- 172.21.15.45:0/1818632005 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd088003520 con 0x7fd0880051c0 2024-04-16T16:19:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.213+0000 7fd0a5200640 1 --2- 172.21.15.45:0/1818632005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd088004800 0x7fd088004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:02.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.214+0000 7fd0a6600640 1 -- 172.21.15.45:0/1818632005 reap_dead start 2024-04-16T16:19:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.214+0000 7fd0a6600640 1 --2- 172.21.15.45:0/1818632005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0880051c0 0x7fd08800b9c0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fd098009fc0 tx=0x7fd088005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc9484b422d9da5e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.214+0000 7fd09fe00640 1 -- 172.21.15.45:0/1818632005 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd098009c10 con 0x7fd0880051c0 2024-04-16T16:19:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.214+0000 7fd09fe00640 1 -- 172.21.15.45:0/1818632005 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd09802cca0 con 0x7fd0880051c0 2024-04-16T16:19:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.214+0000 7fd09fe00640 1 -- 172.21.15.45:0/1818632005 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd098003bd0 con 0x7fd0880051c0 2024-04-16T16:19:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.215+0000 7fd0a8600640 1 -- 172.21.15.45:0/1818632005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0880051c0 msgr2=0x7fd08800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.215+0000 7fd0a8600640 1 --2- 172.21.15.45:0/1818632005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0880051c0 0x7fd08800b9c0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fd098009fc0 tx=0x7fd088005830 comp rx=0 tx=0).stop 2024-04-16T16:19:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.215+0000 7fd0a8600640 1 -- 172.21.15.45:0/1818632005 shutdown_connections 2024-04-16T16:19:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.215+0000 7fd0a8600640 1 --2- 172.21.15.45:0/1818632005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0880051c0 0x7fd08800b9c0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.215+0000 7fd0a8600640 1 -- 172.21.15.45:0/1818632005 >> 172.21.15.45:0/1818632005 conn(0x7fd0a00b3970 msgr2=0x7fd0a00b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.215+0000 7fd0a8600640 1 -- 172.21.15.45:0/1818632005 shutdown_connections 2024-04-16T16:19:02.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.215+0000 7fd0a8600640 1 -- 172.21.15.45:0/1818632005 wait complete. 2024-04-16T16:19:02.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.218+0000 7fd0a8600640 1 Processor -- start 2024-04-16T16:19:02.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.218+0000 7fd0a8600640 1 -- start start 2024-04-16T16:19:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.219+0000 7fd0a8600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0880051c0 0x7fd0a0141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.219+0000 7fd0a8600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0a0140c50 0x7fd0a015fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.219+0000 7fd0a8600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0a0160260 0x7fd0a00b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.219+0000 7fd0a8600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a0163430 con 0x7fd0a0140c50 2024-04-16T16:19:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.219+0000 7fd0a8600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a015f3d0 con 0x7fd0a0160260 2024-04-16T16:19:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.219+0000 7fd0a8600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd0a015eff0 con 0x7fd0880051c0 2024-04-16T16:19:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.219+0000 7fd0a5200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0a0140c50 0x7fd0a015fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.219+0000 7fd0a5200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0a0140c50 0x7fd0a015fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41332/0 (socket says 172.21.15.45:41332) 2024-04-16T16:19:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.219+0000 7fd0a5200640 1 -- 172.21.15.45:0/1199567931 learned_addr learned my addr 172.21.15.45:0/1199567931 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.220+0000 7fd0a6600640 1 --2- 172.21.15.45:0/1199567931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0a0160260 0x7fd0a00b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.220+0000 7fd0a5200640 1 -- 172.21.15.45:0/1199567931 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0880051c0 msgr2=0x7fd0a0141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.220+0000 7fd0a5200640 1 --2- 172.21.15.45:0/1199567931 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0880051c0 0x7fd0a0141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.220+0000 7fd0a5200640 1 -- 172.21.15.45:0/1199567931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0a0160260 msgr2=0x7fd0a00b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.220+0000 7fd0a5200640 1 --2- 172.21.15.45:0/1199567931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0a0160260 0x7fd0a00b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.220+0000 7fd0a5200640 1 -- 172.21.15.45:0/1199567931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a00b22d0 con 0x7fd0a0140c50 2024-04-16T16:19:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.220+0000 7fd0a6600640 1 --2- 172.21.15.45:0/1199567931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0a0160260 0x7fd0a00b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.220+0000 7fd0a5200640 1 --2- 172.21.15.45:0/1199567931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0a0140c50 0x7fd0a015fd20 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00d840 tx=0x7fd08c00dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.220+0000 7fd09d600640 1 -- 172.21.15.45:0/1199567931 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c019070 con 0x7fd0a0140c50 2024-04-16T16:19:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.221+0000 7fd09d600640 1 -- 172.21.15.45:0/1199567931 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd08c0074e0 con 0x7fd0a0140c50 2024-04-16T16:19:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.221+0000 7fd09d600640 1 -- 172.21.15.45:0/1199567931 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c004a00 con 0x7fd0a0140c50 2024-04-16T16:19:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.221+0000 7fd0a8600640 1 -- 172.21.15.45:0/1199567931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a00ce9c0 con 0x7fd0a0140c50 2024-04-16T16:19:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.221+0000 7fd0a8600640 1 -- 172.21.15.45:0/1199567931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a00cef80 con 0x7fd0a0140c50 2024-04-16T16:19:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.222+0000 7fd0a8600640 1 -- 172.21.15.45:0/1199567931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0a0075780 con 0x7fd0a0140c50 2024-04-16T16:19:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.223+0000 7fd09d600640 1 -- 172.21.15.45:0/1199567931 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7fd08c01c070 con 0x7fd0a0140c50 2024-04-16T16:19:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.224+0000 7fd09d600640 1 --2- 172.21.15.45:0/1199567931 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd08006c9b0 0x7fd08006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.224+0000 7fd09d600640 1 -- 172.21.15.45:0/1199567931 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4857+0+0 (secure 0 0 0) 0x7fd08c08b350 con 0x7fd0a0140c50 2024-04-16T16:19:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.225+0000 7fd0a5c00640 1 --2- 172.21.15.45:0/1199567931 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd08006c9b0 0x7fd08006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.226+0000 7fd0a5c00640 1 --2- 172.21.15.45:0/1199567931 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd08006c9b0 0x7fd08006ee70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd094003c40 tx=0x7fd0940079b0 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:02.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.229+0000 7fd09d600640 1 -- 172.21.15.45:0/1199567931 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd08c059a60 con 0x7fd0a0140c50 2024-04-16T16:19:02.335 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:02.335+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:19:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.436+0000 7fd0a8600640 1 -- 172.21.15.45:0/1199567931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd0a00728a0 con 0x7fd0a0140c50 2024-04-16T16:19:02.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.437+0000 7fd09d600640 1 -- 172.21.15.45:0/1199567931 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7fd08c059360 con 0x7fd0a0140c50 2024-04-16T16:19:02.438 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:19:02.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.440+0000 7fd0a8600640 1 -- 172.21.15.45:0/1199567931 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd08006c9b0 msgr2=0x7fd08006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:02.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.440+0000 7fd0a8600640 1 --2- 172.21.15.45:0/1199567931 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd08006c9b0 0x7fd08006ee70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd094003c40 tx=0x7fd0940079b0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.441+0000 7fd0a8600640 1 -- 172.21.15.45:0/1199567931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0a0140c50 msgr2=0x7fd0a015fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:02.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.441+0000 7fd0a8600640 1 --2- 172.21.15.45:0/1199567931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0a0140c50 0x7fd0a015fd20 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00d840 tx=0x7fd08c00dd00 comp rx=0 tx=0).stop 2024-04-16T16:19:02.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.441+0000 7fd0a8600640 1 -- 172.21.15.45:0/1199567931 shutdown_connections 2024-04-16T16:19:02.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.441+0000 7fd0a8600640 1 --2- 172.21.15.45:0/1199567931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0a0160260 0x7fd0a00b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.441+0000 7fd0a8600640 1 --2- 172.21.15.45:0/1199567931 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd08006c9b0 0x7fd08006ee70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.441+0000 7fd0a8600640 1 --2- 172.21.15.45:0/1199567931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0a0140c50 0x7fd0a015fd20 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.441+0000 7fd0a8600640 1 --2- 172.21.15.45:0/1199567931 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0880051c0 0x7fd0a0141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.441+0000 7fd0a8600640 1 -- 172.21.15.45:0/1199567931 >> 172.21.15.45:0/1199567931 conn(0x7fd0a00b3970 msgr2=0x7fd0a006b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:02.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.441+0000 7fd0a8600640 1 -- 172.21.15.45:0/1199567931 shutdown_connections 2024-04-16T16:19:02.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.441+0000 7fd0a8600640 1 -- 172.21.15.45:0/1199567931 wait complete. 2024-04-16T16:19:02.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-16T16:19:02.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.686+0000 7ff247e00640 1 Processor -- start 2024-04-16T16:19:02.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.686+0000 7ff247e00640 1 -- start start 2024-04-16T16:19:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.688+0000 7ff247e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff24015b730 0x7ff2400cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.688+0000 7ff247e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff2400b5840 0x7ff2400cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.688+0000 7ff247e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff2400d0060 0x7ff24015f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.688+0000 7ff247e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff2400ce5b0 con 0x7ff2400d0060 2024-04-16T16:19:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.688+0000 7ff247e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff240072420 con 0x7ff2400b5840 2024-04-16T16:19:02.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.688+0000 7ff247e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff240072700 con 0x7ff24015b730 2024-04-16T16:19:02.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.688+0000 7ff245e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff2400d0060 0x7ff24015f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:02.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.688+0000 7ff245e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff2400d0060 0x7ff24015f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41342/0 (socket says 172.21.15.45:41342) 2024-04-16T16:19:02.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.688+0000 7ff245e00640 1 -- 172.21.15.45:0/1804208545 learned_addr learned my addr 172.21.15.45:0/1804208545 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:02.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.689+0000 7ff244a00640 1 --2- 172.21.15.45:0/1804208545 >> v2:172.21.15.103:3300/0 conn(0x7ff2400b5840 0x7ff2400cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.689+0000 7ff245e00640 1 -- 172.21.15.45:0/1804208545 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff24015b730 msgr2=0x7ff2400cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.689+0000 7ff245400640 1 --2- 172.21.15.45:0/1804208545 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff24015b730 0x7ff2400cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.689+0000 7ff245e00640 1 --2- 172.21.15.45:0/1804208545 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff24015b730 0x7ff2400cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.689+0000 7ff245e00640 1 -- 172.21.15.45:0/1804208545 >> v2:172.21.15.103:3300/0 conn(0x7ff2400b5840 msgr2=0x7ff2400cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.689+0000 7ff245e00640 1 --2- 172.21.15.45:0/1804208545 >> v2:172.21.15.103:3300/0 conn(0x7ff2400b5840 0x7ff2400cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.689+0000 7ff245e00640 1 -- 172.21.15.45:0/1804208545 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2400729e0 con 0x7ff2400d0060 2024-04-16T16:19:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.689+0000 7ff244a00640 1 --2- 172.21.15.45:0/1804208545 >> v2:172.21.15.103:3300/0 conn(0x7ff2400b5840 0x7ff2400cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.689+0000 7ff245400640 1 --2- 172.21.15.45:0/1804208545 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff24015b730 0x7ff2400cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.690+0000 7ff245e00640 1 --2- 172.21.15.45:0/1804208545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2400d0060 0x7ff24015f360 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7ff2400b3510 tx=0x7ff23802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f61e650ab281a4c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.691+0000 7ff23fe00640 1 -- 172.21.15.45:0/1804208545 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23802ea80 con 0x7ff2400d0060 2024-04-16T16:19:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.691+0000 7ff23fe00640 1 -- 172.21.15.45:0/1804208545 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff23802ec20 con 0x7ff2400d0060 2024-04-16T16:19:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.691+0000 7ff23fe00640 1 -- 172.21.15.45:0/1804208545 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff238009600 con 0x7ff2400d0060 2024-04-16T16:19:02.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.691+0000 7ff247e00640 1 -- 172.21.15.45:0/1804208545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2400d0060 msgr2=0x7ff24015f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.691+0000 7ff247e00640 1 --2- 172.21.15.45:0/1804208545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2400d0060 0x7ff24015f360 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7ff2400b3510 tx=0x7ff23802e0e0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.691+0000 7ff247e00640 1 -- 172.21.15.45:0/1804208545 shutdown_connections 2024-04-16T16:19:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.691+0000 7ff247e00640 1 --2- 172.21.15.45:0/1804208545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2400d0060 0x7ff24015f360 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.691+0000 7ff247e00640 1 --2- 172.21.15.45:0/1804208545 >> v2:172.21.15.103:3300/0 conn(0x7ff2400b5840 0x7ff2400cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.691+0000 7ff247e00640 1 --2- 172.21.15.45:0/1804208545 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff24015b730 0x7ff2400cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.691+0000 7ff247e00640 1 -- 172.21.15.45:0/1804208545 >> 172.21.15.45:0/1804208545 conn(0x7ff2400b2750 msgr2=0x7ff2400b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.691+0000 7ff247e00640 1 -- 172.21.15.45:0/1804208545 shutdown_connections 2024-04-16T16:19:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.692+0000 7ff247e00640 1 -- 172.21.15.45:0/1804208545 wait complete. 2024-04-16T16:19:02.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.693+0000 7ff247e00640 1 Processor -- start 2024-04-16T16:19:02.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.693+0000 7ff247e00640 1 -- start start 2024-04-16T16:19:02.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.694+0000 7ff247e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2400b5840 0x7ff24014a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:02.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.694+0000 7ff247e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2400d0060 0x7ff24015a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:02.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.694+0000 7ff247e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff24015b730 0x7ff2400ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:02.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.694+0000 7ff247e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff2400b5ca0 con 0x7ff24015b730 2024-04-16T16:19:02.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.694+0000 7ff247e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff2400d04f0 con 0x7ff2400b5840 2024-04-16T16:19:02.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.694+0000 7ff247e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff24015bb10 con 0x7ff2400d0060 2024-04-16T16:19:02.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.695+0000 7ff245e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff24015b730 0x7ff2400ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:02.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.695+0000 7ff245e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff24015b730 0x7ff2400ae930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41350/0 (socket says 172.21.15.45:41350) 2024-04-16T16:19:02.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.695+0000 7ff245e00640 1 -- 172.21.15.45:0/324934050 learned_addr learned my addr 172.21.15.45:0/324934050 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:02.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.695+0000 7ff244a00640 1 --2- 172.21.15.45:0/324934050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2400d0060 0x7ff24015a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:02.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.695+0000 7ff245400640 1 --2- 172.21.15.45:0/324934050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2400b5840 0x7ff24014a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:02.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.695+0000 7ff245e00640 1 -- 172.21.15.45:0/324934050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2400d0060 msgr2=0x7ff24015a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:02.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.695+0000 7ff245e00640 1 --2- 172.21.15.45:0/324934050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2400d0060 0x7ff24015a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.695+0000 7ff245e00640 1 -- 172.21.15.45:0/324934050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2400b5840 msgr2=0x7ff24014a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:02.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.695+0000 7ff245e00640 1 --2- 172.21.15.45:0/324934050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2400b5840 0x7ff24014a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.695+0000 7ff245e00640 1 -- 172.21.15.45:0/324934050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24014a8d0 con 0x7ff24015b730 2024-04-16T16:19:02.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.695+0000 7ff245400640 1 --2- 172.21.15.45:0/324934050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2400b5840 0x7ff24014a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:02.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.696+0000 7ff245e00640 1 --2- 172.21.15.45:0/324934050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff24015b730 0x7ff2400ae930 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7ff238038040 tx=0x7ff238004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:02.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.696+0000 7ff23d600640 1 -- 172.21.15.45:0/324934050 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23800b720 con 0x7ff24015b730 2024-04-16T16:19:02.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.696+0000 7ff23d600640 1 -- 172.21.15.45:0/324934050 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff238045070 con 0x7ff24015b730 2024-04-16T16:19:02.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.696+0000 7ff23d600640 1 -- 172.21.15.45:0/324934050 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23802ea80 con 0x7ff24015b730 2024-04-16T16:19:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.696+0000 7ff247e00640 1 -- 172.21.15.45:0/324934050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff240159090 con 0x7ff24015b730 2024-04-16T16:19:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.696+0000 7ff247e00640 1 -- 172.21.15.45:0/324934050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff240159650 con 0x7ff24015b730 2024-04-16T16:19:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.697+0000 7ff247e00640 1 -- 172.21.15.45:0/324934050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff24006ff30 con 0x7ff24015b730 2024-04-16T16:19:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.699+0000 7ff23d600640 1 -- 172.21.15.45:0/324934050 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7ff238036a40 con 0x7ff24015b730 2024-04-16T16:19:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.699+0000 7ff23d600640 1 --2- 172.21.15.45:0/324934050 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff22006c980 0x7ff22006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:02.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.699+0000 7ff23d600640 1 -- 172.21.15.45:0/324934050 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4857+0+0 (secure 0 0 0) 0x7ff2380ae870 con 0x7ff24015b730 2024-04-16T16:19:02.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.700+0000 7ff245400640 1 --2- 172.21.15.45:0/324934050 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff22006c980 0x7ff22006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:02.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.701+0000 7ff245400640 1 --2- 172.21.15.45:0/324934050 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff22006c980 0x7ff22006ee40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff234006fd0 tx=0x7ff234008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:02.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.705+0000 7ff23d600640 1 -- 172.21.15.45:0/324934050 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff23807cf10 con 0x7ff24015b730 2024-04-16T16:19:02.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.916+0000 7ff247e00640 1 -- 172.21.15.45:0/324934050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff240075780 con 0x7ff24015b730 2024-04-16T16:19:02.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.918+0000 7ff23d600640 1 -- 172.21.15.45:0/324934050 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v95) ==== 84+0+65 (secure 0 0 0) 0x7ff23807c810 con 0x7ff24015b730 2024-04-16T16:19:02.918 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:02.918 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2024-04-16T16:19:02.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:02.918 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2024-04-16T16:19:02.918 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2024-04-16T16:19:02.918 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:02.918 INFO:teuthology.orchestra.run.smithi045.stdout:] 2024-04-16T16:19:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.920+0000 7ff247e00640 1 -- 172.21.15.45:0/324934050 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff22006c980 msgr2=0x7ff22006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.920+0000 7ff247e00640 1 --2- 172.21.15.45:0/324934050 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff22006c980 0x7ff22006ee40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff234006fd0 tx=0x7ff234008040 comp rx=0 tx=0).stop 2024-04-16T16:19:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.920+0000 7ff247e00640 1 -- 172.21.15.45:0/324934050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff24015b730 msgr2=0x7ff2400ae930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.920+0000 7ff247e00640 1 --2- 172.21.15.45:0/324934050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff24015b730 0x7ff2400ae930 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7ff238038040 tx=0x7ff238004a40 comp rx=0 tx=0).stop 2024-04-16T16:19:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.921+0000 7ff247e00640 1 -- 172.21.15.45:0/324934050 shutdown_connections 2024-04-16T16:19:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.921+0000 7ff247e00640 1 --2- 172.21.15.45:0/324934050 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff22006c980 0x7ff22006ee40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.921+0000 7ff247e00640 1 --2- 172.21.15.45:0/324934050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff24015b730 0x7ff2400ae930 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.921+0000 7ff247e00640 1 --2- 172.21.15.45:0/324934050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2400d0060 0x7ff24015a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.921+0000 7ff247e00640 1 --2- 172.21.15.45:0/324934050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2400b5840 0x7ff24014a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:02.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.921+0000 7ff247e00640 1 -- 172.21.15.45:0/324934050 >> 172.21.15.45:0/324934050 conn(0x7ff2400b2750 msgr2=0x7ff240157b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:02.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.921+0000 7ff247e00640 1 -- 172.21.15.45:0/324934050 shutdown_connections 2024-04-16T16:19:02.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:02.921+0000 7ff247e00640 1 -- 172.21.15.45:0/324934050 wait complete. 2024-04-16T16:19:02.942 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-16T16:19:02.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-16T16:19:03.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.175+0000 7fe582800640 1 Processor -- start 2024-04-16T16:19:03.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.175+0000 7fe582800640 1 -- start start 2024-04-16T16:19:03.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.177+0000 7fe582800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe57c0b1500 0x7fe57c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:03.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.177+0000 7fe582800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe57c06e920 0x7fe57c15f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:03.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.177+0000 7fe582800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe57c0d0430 con 0x7fe57c0d0050 2024-04-16T16:19:03.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.177+0000 7fe582800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe57c15bbd0 con 0x7fe57c0b1500 2024-04-16T16:19:03.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.177+0000 7fe582800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe57c0b0fd0 con 0x7fe57c06e920 2024-04-16T16:19:03.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.177+0000 7fe580a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe57c0b1500 0x7fe57c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:03.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.177+0000 7fe580a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe57c0b1500 0x7fe57c06e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41356/0 (socket says 172.21.15.45:41356) 2024-04-16T16:19:03.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.177+0000 7fe581400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe57c0d0050 0x7fe57c0cd8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47902/0 (socket says 172.21.15.45:47902) 2024-04-16T16:19:03.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.177+0000 7fe580a00640 1 -- 172.21.15.45:0/2377133079 learned_addr learned my addr 172.21.15.45:0/2377133079 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:03.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.178+0000 7fe581e00640 1 --2- 172.21.15.45:0/2377133079 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe57c06e920 0x7fe57c15f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:03.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.178+0000 7fe580a00640 1 -- 172.21.15.45:0/2377133079 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe57c06e920 msgr2=0x7fe57c15f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:03.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.178+0000 7fe580a00640 1 --2- 172.21.15.45:0/2377133079 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe57c06e920 0x7fe57c15f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:03.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.178+0000 7fe580a00640 1 -- 172.21.15.45:0/2377133079 >> v1:172.21.15.45:6789/0 conn(0x7fe57c0d0050 legacy=0x7fe57c0cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:03.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.178+0000 7fe580a00640 1 -- 172.21.15.45:0/2377133079 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe57c15be90 con 0x7fe57c0b1500 2024-04-16T16:19:03.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.178+0000 7fe57be00640 1 -- 172.21.15.45:0/2377133079 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3371151473 0 0) 0x7fe57c0d0430 con 0x7fe57c0d0050 2024-04-16T16:19:03.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.178+0000 7fe581e00640 1 --2- 172.21.15.45:0/2377133079 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe57c06e920 0x7fe57c15f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:03.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.179+0000 7fe580a00640 1 --2- 172.21.15.45:0/2377133079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe57c0b1500 0x7fe57c06e3e0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe568009870 tx=0x7fe56802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61363ef0dbe4eefa server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:03.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.180+0000 7fe57be00640 1 -- 172.21.15.45:0/2377133079 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe568034070 con 0x7fe57c0b1500 2024-04-16T16:19:03.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.180+0000 7fe57be00640 1 -- 172.21.15.45:0/2377133079 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe568030840 con 0x7fe57c0b1500 2024-04-16T16:19:03.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.180+0000 7fe57be00640 1 -- 172.21.15.45:0/2377133079 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe568030c10 con 0x7fe57c0b1500 2024-04-16T16:19:03.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.180+0000 7fe582800640 1 -- 172.21.15.45:0/2377133079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe57c0b1500 msgr2=0x7fe57c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:03.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.180+0000 7fe582800640 1 --2- 172.21.15.45:0/2377133079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe57c0b1500 0x7fe57c06e3e0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe568009870 tx=0x7fe56802b980 comp rx=0 tx=0).stop 2024-04-16T16:19:03.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.181+0000 7fe582800640 1 -- 172.21.15.45:0/2377133079 shutdown_connections 2024-04-16T16:19:03.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.181+0000 7fe582800640 1 --2- 172.21.15.45:0/2377133079 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe57c06e920 0x7fe57c15f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:03.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.181+0000 7fe582800640 1 --2- 172.21.15.45:0/2377133079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe57c0b1500 0x7fe57c06e3e0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:03.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.181+0000 7fe582800640 1 -- 172.21.15.45:0/2377133079 >> 172.21.15.45:0/2377133079 conn(0x7fe57c0b3bb0 msgr2=0x7fe57c0b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:03.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.181+0000 7fe582800640 1 -- 172.21.15.45:0/2377133079 shutdown_connections 2024-04-16T16:19:03.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.181+0000 7fe582800640 1 -- 172.21.15.45:0/2377133079 wait complete. 2024-04-16T16:19:03.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.183+0000 7fe582800640 1 Processor -- start 2024-04-16T16:19:03.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.183+0000 7fe582800640 1 -- start start 2024-04-16T16:19:03.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.184+0000 7fe582800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe57c06e920 0x7fe57c0c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:03.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.184+0000 7fe582800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe57c0b1500 0x7fe57c0cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:03.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.184+0000 7fe582800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe57c0d0050 0x7fe57c0732e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:03.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.184+0000 7fe582800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c15c730 con 0x7fe57c0d0050 2024-04-16T16:19:03.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.184+0000 7fe582800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c15c450 con 0x7fe57c06e920 2024-04-16T16:19:03.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.184+0000 7fe582800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe57c0cdfb0 con 0x7fe57c0b1500 2024-04-16T16:19:03.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.184+0000 7fe581e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe57c0d0050 0x7fe57c0732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:03.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.184+0000 7fe581e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe57c0d0050 0x7fe57c0732e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41358/0 (socket says 172.21.15.45:41358) 2024-04-16T16:19:03.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.184+0000 7fe581e00640 1 -- 172.21.15.45:0/2712938400 learned_addr learned my addr 172.21.15.45:0/2712938400 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:03.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.185+0000 7fe581400640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe57c06e920 0x7fe57c0c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:03.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.185+0000 7fe580a00640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe57c0b1500 0x7fe57c0cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:03.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.185+0000 7fe581e00640 1 -- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe57c0b1500 msgr2=0x7fe57c0cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:03.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.185+0000 7fe581e00640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe57c0b1500 0x7fe57c0cb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:03.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.185+0000 7fe581e00640 1 -- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe57c06e920 msgr2=0x7fe57c0c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:03.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.185+0000 7fe581e00640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe57c06e920 0x7fe57c0c9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:03.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.185+0000 7fe581e00640 1 -- 172.21.15.45:0/2712938400 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe57c0cbc60 con 0x7fe57c0d0050 2024-04-16T16:19:03.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.185+0000 7fe581400640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe57c06e920 0x7fe57c0c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:03.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.185+0000 7fe580a00640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe57c0b1500 0x7fe57c0cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:03.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.185+0000 7fe581e00640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe57c0d0050 0x7fe57c0732e0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe57400cf20 tx=0x7fe57400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:03.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.185+0000 7fe579600640 1 -- 172.21.15.45:0/2712938400 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57401a070 con 0x7fe57c0d0050 2024-04-16T16:19:03.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.185+0000 7fe579600640 1 -- 172.21.15.45:0/2712938400 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe57400d840 con 0x7fe57c0d0050 2024-04-16T16:19:03.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.185+0000 7fe579600640 1 -- 172.21.15.45:0/2712938400 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57400dc30 con 0x7fe57c0d0050 2024-04-16T16:19:03.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.185+0000 7fe582800640 1 -- 172.21.15.45:0/2712938400 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe57c0cbf30 con 0x7fe57c0d0050 2024-04-16T16:19:03.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.186+0000 7fe582800640 1 -- 172.21.15.45:0/2712938400 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe57c1f5a30 con 0x7fe57c0d0050 2024-04-16T16:19:03.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.187+0000 7fe582800640 1 -- 172.21.15.45:0/2712938400 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe57c075780 con 0x7fe57c0d0050 2024-04-16T16:19:03.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.188+0000 7fe579600640 1 -- 172.21.15.45:0/2712938400 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7fe57401d070 con 0x7fe57c0d0050 2024-04-16T16:19:03.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.189+0000 7fe579600640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fe55c06c980 0x7fe55c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:03.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.189+0000 7fe579600640 1 -- 172.21.15.45:0/2712938400 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4857+0+0 (secure 0 0 0) 0x7fe57408c180 con 0x7fe57c0d0050 2024-04-16T16:19:03.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.189+0000 7fe581400640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fe55c06c980 0x7fe55c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:03.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.191+0000 7fe581400640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fe55c06c980 0x7fe55c06ee40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe570000f60 tx=0x7fe570009040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:03.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.193+0000 7fe579600640 1 -- 172.21.15.45:0/2712938400 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe57405a8a0 con 0x7fe57c0d0050 2024-04-16T16:19:03.348 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:03.348+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:19:03.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.411+0000 7fe582800640 1 -- 172.21.15.45:0/2712938400 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fe57c0af900 con 0x7fe57c0d0050 2024-04-16T16:19:03.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.932+0000 7fe579600640 1 -- 172.21.15.45:0/2712938400 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v96) ==== 138+0+0 (secure 0 0 0) 0x7fe57405a1a0 con 0x7fe57c0d0050 2024-04-16T16:19:03.933 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2024-04-16T16:19:03.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.936+0000 7fe582800640 1 -- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fe55c06c980 msgr2=0x7fe55c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:03.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.936+0000 7fe582800640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fe55c06c980 0x7fe55c06ee40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe570000f60 tx=0x7fe570009040 comp rx=0 tx=0).stop 2024-04-16T16:19:03.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.936+0000 7fe582800640 1 -- 172.21.15.45:0/2712938400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe57c0d0050 msgr2=0x7fe57c0732e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:03.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.936+0000 7fe582800640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe57c0d0050 0x7fe57c0732e0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe57400cf20 tx=0x7fe57400a7d0 comp rx=0 tx=0).stop 2024-04-16T16:19:03.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.936+0000 7fe582800640 1 -- 172.21.15.45:0/2712938400 shutdown_connections 2024-04-16T16:19:03.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.936+0000 7fe582800640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fe55c06c980 0x7fe55c06ee40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:03.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.936+0000 7fe582800640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe57c0d0050 0x7fe57c0732e0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:03.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.936+0000 7fe582800640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe57c0b1500 0x7fe57c0cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:03.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.936+0000 7fe582800640 1 --2- 172.21.15.45:0/2712938400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe57c06e920 0x7fe57c0c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:03.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.936+0000 7fe582800640 1 -- 172.21.15.45:0/2712938400 >> 172.21.15.45:0/2712938400 conn(0x7fe57c0b3bb0 msgr2=0x7fe57c0b3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:03.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.937+0000 7fe582800640 1 -- 172.21.15.45:0/2712938400 shutdown_connections 2024-04-16T16:19:03.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:03.937+0000 7fe582800640 1 -- 172.21.15.45:0/2712938400 wait complete. 2024-04-16T16:19:03.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-16T16:19:04.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.182+0000 7f07f3200640 1 Processor -- start 2024-04-16T16:19:04.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.182+0000 7f07f3200640 1 -- start start 2024-04-16T16:19:04.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.183+0000 7f07f3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07f4155ab0 0x7f07f4162ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:04.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.183+0000 7f07f3200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f07f40d18c0 0x7f07f40d1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:04.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.183+0000 7f07f3200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f4155e90 con 0x7f07f40ba610 2024-04-16T16:19:04.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.183+0000 7f07f3200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f07f415ff00 con 0x7f07f40d18c0 2024-04-16T16:19:04.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.183+0000 7f07f3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f07f415f6e0 con 0x7f07f4155ab0 2024-04-16T16:19:04.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.184+0000 7f07f2800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f07f40d18c0 0x7f07f40d1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:04.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.184+0000 7f07f2800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f07f40d18c0 0x7f07f40d1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44548/0 (socket says 172.21.15.45:44548) 2024-04-16T16:19:04.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.184+0000 7f07f2800640 1 -- 172.21.15.45:0/3505990426 learned_addr learned my addr 172.21.15.45:0/3505990426 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:04.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.184+0000 7f07f1e00640 1 --2- 172.21.15.45:0/3505990426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07f4155ab0 0x7f07f4162ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:04.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.184+0000 7f07f2800640 1 -- 172.21.15.45:0/3505990426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07f4155ab0 msgr2=0x7f07f4162ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:04.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.184+0000 7f07f2800640 1 --2- 172.21.15.45:0/3505990426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07f4155ab0 0x7f07f4162ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:04.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.184+0000 7f07f2800640 1 -- 172.21.15.45:0/3505990426 >> v1:172.21.15.103:6789/0 conn(0x7f07f40ba610 legacy=0x7f07f415efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:04.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.184+0000 7f07f2800640 1 -- 172.21.15.45:0/3505990426 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f415f940 con 0x7f07f40d18c0 2024-04-16T16:19:04.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.186+0000 7f07f2800640 1 --2- 172.21.15.45:0/3505990426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07f40d18c0 0x7f07f40d1d50 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f07ec009e50 tx=0x7f07ec02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d2a1d18fb12a0fb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:04.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.186+0000 7f07f0a00640 1 -- 172.21.15.45:0/3505990426 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec02c9e0 con 0x7f07f40d18c0 2024-04-16T16:19:04.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.186+0000 7f07f0a00640 1 -- 172.21.15.45:0/3505990426 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07ec02cb80 con 0x7f07f40d18c0 2024-04-16T16:19:04.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.186+0000 7f07f0a00640 1 -- 172.21.15.45:0/3505990426 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec0343f0 con 0x7f07f40d18c0 2024-04-16T16:19:04.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.187+0000 7f07f3200640 1 -- 172.21.15.45:0/3505990426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07f40d18c0 msgr2=0x7f07f40d1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:04.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.187+0000 7f07f3200640 1 --2- 172.21.15.45:0/3505990426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07f40d18c0 0x7f07f40d1d50 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f07ec009e50 tx=0x7f07ec02be40 comp rx=0 tx=0).stop 2024-04-16T16:19:04.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.187+0000 7f07f3200640 1 -- 172.21.15.45:0/3505990426 shutdown_connections 2024-04-16T16:19:04.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.187+0000 7f07f3200640 1 --2- 172.21.15.45:0/3505990426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07f40d18c0 0x7f07f40d1d50 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:04.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.187+0000 7f07f3200640 1 --2- 172.21.15.45:0/3505990426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07f4155ab0 0x7f07f4162ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:04.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.187+0000 7f07f3200640 1 -- 172.21.15.45:0/3505990426 >> 172.21.15.45:0/3505990426 conn(0x7f07f406d1a0 msgr2=0x7f07f406d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:04.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.187+0000 7f07f3200640 1 -- 172.21.15.45:0/3505990426 shutdown_connections 2024-04-16T16:19:04.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.187+0000 7f07f3200640 1 -- 172.21.15.45:0/3505990426 wait complete. 2024-04-16T16:19:04.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.189+0000 7f07f3200640 1 Processor -- start 2024-04-16T16:19:04.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.189+0000 7f07f3200640 1 -- start start 2024-04-16T16:19:04.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.190+0000 7f07f3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07f40ba610 0x7f07f40ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.190+0000 7f07f3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07f40d18c0 0x7f07f40cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.190+0000 7f07f3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07f4155ab0 0x7f07f41e83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.190+0000 7f07f3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f07f41629d0 con 0x7f07f4155ab0 2024-04-16T16:19:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.190+0000 7f07f3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f07f41626f0 con 0x7f07f40d18c0 2024-04-16T16:19:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.190+0000 7f07f3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f07f4163430 con 0x7f07f40ba610 2024-04-16T16:19:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.191+0000 7f07f2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07f4155ab0 0x7f07f41e83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.191+0000 7f07f2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07f4155ab0 0x7f07f41e83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44558/0 (socket says 172.21.15.45:44558) 2024-04-16T16:19:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.191+0000 7f07f2800640 1 -- 172.21.15.45:0/394312003 learned_addr learned my addr 172.21.15.45:0/394312003 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:04.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.191+0000 7f07f1400640 1 --2- 172.21.15.45:0/394312003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07f40d18c0 0x7f07f40cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:04.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.191+0000 7f07f1e00640 1 --2- 172.21.15.45:0/394312003 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07f40ba610 0x7f07f40ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:04.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.191+0000 7f07f2800640 1 -- 172.21.15.45:0/394312003 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07f40ba610 msgr2=0x7f07f40ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:04.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.191+0000 7f07f2800640 1 --2- 172.21.15.45:0/394312003 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07f40ba610 0x7f07f40ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:04.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.191+0000 7f07f2800640 1 -- 172.21.15.45:0/394312003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07f40d18c0 msgr2=0x7f07f40cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:04.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.191+0000 7f07f2800640 1 --2- 172.21.15.45:0/394312003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07f40d18c0 0x7f07f40cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:04.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.191+0000 7f07f2800640 1 -- 172.21.15.45:0/394312003 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f40cc960 con 0x7f07f4155ab0 2024-04-16T16:19:04.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.192+0000 7f07f2800640 1 --2- 172.21.15.45:0/394312003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07f4155ab0 0x7f07f41e83e0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f07ec036040 tx=0x7f07ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:04.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.192+0000 7f07ea000640 1 -- 172.21.15.45:0/394312003 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec043020 con 0x7f07f4155ab0 2024-04-16T16:19:04.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.192+0000 7f07ea000640 1 -- 172.21.15.45:0/394312003 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07ec03e450 con 0x7f07f4155ab0 2024-04-16T16:19:04.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.192+0000 7f07ea000640 1 -- 172.21.15.45:0/394312003 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec009510 con 0x7f07f4155ab0 2024-04-16T16:19:04.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.192+0000 7f07f3200640 1 -- 172.21.15.45:0/394312003 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07f40ccc40 con 0x7f07f4155ab0 2024-04-16T16:19:04.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.192+0000 7f07f3200640 1 -- 172.21.15.45:0/394312003 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07f40cd200 con 0x7f07f4155ab0 2024-04-16T16:19:04.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.194+0000 7f07f3200640 1 -- 172.21.15.45:0/394312003 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07f4075780 con 0x7f07f4155ab0 2024-04-16T16:19:04.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.195+0000 7f07ea000640 1 -- 172.21.15.45:0/394312003 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f07ec046020 con 0x7f07f4155ab0 2024-04-16T16:19:04.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.197+0000 7f07ea000640 1 --2- 172.21.15.45:0/394312003 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f07d406c9b0 0x7f07d406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:04.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.197+0000 7f07ea000640 1 -- 172.21.15.45:0/394312003 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5227+0+0 (secure 0 0 0) 0x7f07ec0ae6a0 con 0x7f07f4155ab0 2024-04-16T16:19:04.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.198+0000 7f07f1e00640 1 --2- 172.21.15.45:0/394312003 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f07d406c9b0 0x7f07d406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:04.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.199+0000 7f07f1e00640 1 --2- 172.21.15.45:0/394312003 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f07d406c9b0 0x7f07d406ee70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f07e4006fd0 tx=0x7f07e4008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:04.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.202+0000 7f07ea000640 1 -- 172.21.15.45:0/394312003 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f07ec04d070 con 0x7f07f4155ab0 2024-04-16T16:19:04.398 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:04.398+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:19:04.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.418+0000 7f07f3200640 1 -- 172.21.15.45:0/394312003 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f07f40caf50 con 0x7f07f4155ab0 2024-04-16T16:19:04.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.951+0000 7f07ea000640 1 -- 172.21.15.45:0/394312003 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v97) ==== 170+0+0 (secure 0 0 0) 0x7f07ec07cd40 con 0x7f07f4155ab0 2024-04-16T16:19:04.952 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2024-04-16T16:19:04.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.956+0000 7f07f3200640 1 -- 172.21.15.45:0/394312003 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f07d406c9b0 msgr2=0x7f07d406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:04.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.956+0000 7f07f3200640 1 --2- 172.21.15.45:0/394312003 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f07d406c9b0 0x7f07d406ee70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f07e4006fd0 tx=0x7f07e4008040 comp rx=0 tx=0).stop 2024-04-16T16:19:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.956+0000 7f07f3200640 1 -- 172.21.15.45:0/394312003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07f4155ab0 msgr2=0x7f07f41e83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.956+0000 7f07f3200640 1 --2- 172.21.15.45:0/394312003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07f4155ab0 0x7f07f41e83e0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f07ec036040 tx=0x7f07ec004a40 comp rx=0 tx=0).stop 2024-04-16T16:19:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.956+0000 7f07f3200640 1 -- 172.21.15.45:0/394312003 shutdown_connections 2024-04-16T16:19:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.956+0000 7f07f3200640 1 --2- 172.21.15.45:0/394312003 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f07d406c9b0 0x7f07d406ee70 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f07e4006fd0 tx=0x7f07e4008040 comp rx=0 tx=0).stop 2024-04-16T16:19:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.956+0000 7f07f3200640 1 --2- 172.21.15.45:0/394312003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07f4155ab0 0x7f07f41e83e0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.956+0000 7f07f3200640 1 --2- 172.21.15.45:0/394312003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07f40d18c0 0x7f07f40cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.956+0000 7f07f3200640 1 --2- 172.21.15.45:0/394312003 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07f40ba610 0x7f07f40ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.956+0000 7f07f3200640 1 -- 172.21.15.45:0/394312003 >> 172.21.15.45:0/394312003 conn(0x7f07f406d1a0 msgr2=0x7f07f41615a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:04.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.957+0000 7f07f3200640 1 -- 172.21.15.45:0/394312003 shutdown_connections 2024-04-16T16:19:04.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:04.957+0000 7f07f3200640 1 -- 172.21.15.45:0/394312003 wait complete. 2024-04-16T16:19:04.992 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-16T16:19:05.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.185+0000 7f4577600640 1 Processor -- start 2024-04-16T16:19:05.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.185+0000 7f4577600640 1 -- start start 2024-04-16T16:19:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.187+0000 7f4577600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f457015c7c0 0x7f4570072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.187+0000 7f4577600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f45700d10e0 0x7f457006f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.187+0000 7f4577600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45700b5530 con 0x7f45700b4930 2024-04-16T16:19:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.187+0000 7f4577600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f45700b52e0 con 0x7f45700d10e0 2024-04-16T16:19:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.187+0000 7f4577600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f45700d0260 con 0x7f457015c7c0 2024-04-16T16:19:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.187+0000 7f4575600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f45700d10e0 0x7f457006f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:05.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.187+0000 7f4575600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f45700d10e0 0x7f457006f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44564/0 (socket says 172.21.15.45:44564) 2024-04-16T16:19:05.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.187+0000 7f4575600640 1 -- 172.21.15.45:0/1266918586 learned_addr learned my addr 172.21.15.45:0/1266918586 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:05.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.187+0000 7f4574c00640 1 --2- 172.21.15.45:0/1266918586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f457015c7c0 0x7f4570072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.188+0000 7f4575600640 1 -- 172.21.15.45:0/1266918586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f457015c7c0 msgr2=0x7f4570072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.188+0000 7f4575600640 1 --2- 172.21.15.45:0/1266918586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f457015c7c0 0x7f4570072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.188+0000 7f4575600640 1 -- 172.21.15.45:0/1266918586 >> v1:172.21.15.103:6789/0 conn(0x7f45700b4930 legacy=0x7f45700d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.188+0000 7f4575600640 1 -- 172.21.15.45:0/1266918586 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45700b5810 con 0x7f45700d10e0 2024-04-16T16:19:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.188+0000 7f4574c00640 1 --2- 172.21.15.45:0/1266918586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f457015c7c0 0x7f4570072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.189+0000 7f4575600640 1 --2- 172.21.15.45:0/1266918586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45700d10e0 0x7f457006f160 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f4568009e50 tx=0x7f456802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa728e0703d76fc server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:05.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.189+0000 7f456f400640 1 -- 172.21.15.45:0/1266918586 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456802c950 con 0x7f45700d10e0 2024-04-16T16:19:05.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.189+0000 7f456f400640 1 -- 172.21.15.45:0/1266918586 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f456802caf0 con 0x7f45700d10e0 2024-04-16T16:19:05.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.189+0000 7f456f400640 1 -- 172.21.15.45:0/1266918586 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45680343f0 con 0x7f45700d10e0 2024-04-16T16:19:05.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.190+0000 7f4577600640 1 -- 172.21.15.45:0/1266918586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45700d10e0 msgr2=0x7f457006f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:05.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.190+0000 7f4577600640 1 --2- 172.21.15.45:0/1266918586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45700d10e0 0x7f457006f160 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f4568009e50 tx=0x7f456802bdb0 comp rx=0 tx=0).stop 2024-04-16T16:19:05.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.190+0000 7f4577600640 1 -- 172.21.15.45:0/1266918586 shutdown_connections 2024-04-16T16:19:05.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.190+0000 7f4577600640 1 --2- 172.21.15.45:0/1266918586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45700d10e0 0x7f457006f160 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:05.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.190+0000 7f4577600640 1 --2- 172.21.15.45:0/1266918586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f457015c7c0 0x7f4570072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:05.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.190+0000 7f4577600640 1 -- 172.21.15.45:0/1266918586 >> 172.21.15.45:0/1266918586 conn(0x7f45700b3970 msgr2=0x7f45700b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:05.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.190+0000 7f4577600640 1 -- 172.21.15.45:0/1266918586 shutdown_connections 2024-04-16T16:19:05.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.190+0000 7f4577600640 1 -- 172.21.15.45:0/1266918586 wait complete. 2024-04-16T16:19:05.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.192+0000 7f4577600640 1 Processor -- start 2024-04-16T16:19:05.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.192+0000 7f4577600640 1 -- start start 2024-04-16T16:19:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.193+0000 7f4577600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45700b4930 0x7f4570146a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.193+0000 7f4577600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45700d10e0 0x7f4570146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.193+0000 7f4577600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f457015c7c0 0x7f45701f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.193+0000 7f4577600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f457006f6a0 con 0x7f457015c7c0 2024-04-16T16:19:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.193+0000 7f4577600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4570073350 con 0x7f45700b4930 2024-04-16T16:19:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.193+0000 7f4577600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f457006f820 con 0x7f45700d10e0 2024-04-16T16:19:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.194+0000 7f4575600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f457015c7c0 0x7f45701f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.194+0000 7f4575600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f457015c7c0 0x7f45701f26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44580/0 (socket says 172.21.15.45:44580) 2024-04-16T16:19:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.194+0000 7f4575600640 1 -- 172.21.15.45:0/3894145265 learned_addr learned my addr 172.21.15.45:0/3894145265 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.194+0000 7f4574c00640 1 --2- 172.21.15.45:0/3894145265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45700b4930 0x7f4570146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.194+0000 7f4575600640 1 -- 172.21.15.45:0/3894145265 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45700d10e0 msgr2=0x7f4570146340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.194+0000 7f4575600640 1 --2- 172.21.15.45:0/3894145265 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45700d10e0 0x7f4570146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.194+0000 7f4575600640 1 -- 172.21.15.45:0/3894145265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45700b4930 msgr2=0x7f4570146a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.194+0000 7f4575600640 1 --2- 172.21.15.45:0/3894145265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45700b4930 0x7f4570146a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.194+0000 7f4575600640 1 -- 172.21.15.45:0/3894145265 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4570153250 con 0x7f457015c7c0 2024-04-16T16:19:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.194+0000 7f4575600640 1 --2- 172.21.15.45:0/3894145265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f457015c7c0 0x7f45701f26c0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f4568009e50 tx=0x7f4568004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.195+0000 7f456cc00640 1 -- 172.21.15.45:0/3894145265 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4568043020 con 0x7f457015c7c0 2024-04-16T16:19:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.195+0000 7f456cc00640 1 -- 172.21.15.45:0/3894145265 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f456803e450 con 0x7f457015c7c0 2024-04-16T16:19:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.195+0000 7f4577600640 1 -- 172.21.15.45:0/3894145265 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4570150380 con 0x7f457015c7c0 2024-04-16T16:19:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.195+0000 7f456cc00640 1 -- 172.21.15.45:0/3894145265 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456803e860 con 0x7f457015c7c0 2024-04-16T16:19:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.195+0000 7f4577600640 1 -- 172.21.15.45:0/3894145265 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4570152a50 con 0x7f457015c7c0 2024-04-16T16:19:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.196+0000 7f4577600640 1 -- 172.21.15.45:0/3894145265 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4570075780 con 0x7f457015c7c0 2024-04-16T16:19:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.198+0000 7f456cc00640 1 -- 172.21.15.45:0/3894145265 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f4568046020 con 0x7f457015c7c0 2024-04-16T16:19:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.199+0000 7f456cc00640 1 --2- 172.21.15.45:0/3894145265 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f455006c980 0x7f455006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.199+0000 7f456cc00640 1 -- 172.21.15.45:0/3894145265 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5609+0+0 (secure 0 0 0) 0x7f45680ae880 con 0x7f457015c7c0 2024-04-16T16:19:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.200+0000 7f4574c00640 1 --2- 172.21.15.45:0/3894145265 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f455006c980 0x7f455006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:05.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.201+0000 7f4574c00640 1 --2- 172.21.15.45:0/3894145265 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f455006c980 0x7f455006ee40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f456400a950 tx=0x7f4564008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.203+0000 7f456cc00640 1 -- 172.21.15.45:0/3894145265 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f456804d030 con 0x7f457015c7c0 2024-04-16T16:19:05.446 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:05.446+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:19:05.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.455+0000 7f4577600640 1 -- 172.21.15.45:0/3894145265 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f45700b32d0 con 0x7f457015c7c0 2024-04-16T16:19:05.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.968+0000 7f456cc00640 1 -- 172.21.15.45:0/3894145265 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v69) ==== 174+0+0 (secure 0 0 0) 0x7f456807cec0 con 0x7f457015c7c0 2024-04-16T16:19:05.972 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 14 and data pool 15 2024-04-16T16:19:05.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.980+0000 7f4577600640 1 -- 172.21.15.45:0/3894145265 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f455006c980 msgr2=0x7f455006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:05.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.980+0000 7f4577600640 1 --2- 172.21.15.45:0/3894145265 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f455006c980 0x7f455006ee40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f456400a950 tx=0x7f4564008040 comp rx=0 tx=0).stop 2024-04-16T16:19:05.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.980+0000 7f4577600640 1 -- 172.21.15.45:0/3894145265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f457015c7c0 msgr2=0x7f45701f26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:05.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.980+0000 7f4577600640 1 --2- 172.21.15.45:0/3894145265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f457015c7c0 0x7f45701f26c0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f4568009e50 tx=0x7f4568004a40 comp rx=0 tx=0).stop 2024-04-16T16:19:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.980+0000 7f4577600640 1 -- 172.21.15.45:0/3894145265 shutdown_connections 2024-04-16T16:19:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.980+0000 7f4577600640 1 --2- 172.21.15.45:0/3894145265 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f455006c980 0x7f455006ee40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.980+0000 7f4577600640 1 --2- 172.21.15.45:0/3894145265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f457015c7c0 0x7f45701f26c0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.980+0000 7f4577600640 1 --2- 172.21.15.45:0/3894145265 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45700d10e0 0x7f4570146340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.980+0000 7f4577600640 1 --2- 172.21.15.45:0/3894145265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45700b4930 0x7f4570146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.980+0000 7f4577600640 1 -- 172.21.15.45:0/3894145265 >> 172.21.15.45:0/3894145265 conn(0x7f45700b3970 msgr2=0x7f457015f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.983+0000 7f4577600640 1 -- 172.21.15.45:0/3894145265 shutdown_connections 2024-04-16T16:19:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:05.983+0000 7f4577600640 1 -- 172.21.15.45:0/3894145265 wait complete. 2024-04-16T16:19:06.019 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:19:06.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.209+0000 7ff782800640 1 Processor -- start 2024-04-16T16:19:06.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.210+0000 7ff782800640 1 -- start start 2024-04-16T16:19:06.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.211+0000 7ff782800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff77c161e90 0x7ff77c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:06.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.211+0000 7ff782800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff77c0d52d0 0x7ff77c06ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:06.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.211+0000 7ff782800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff77c0d3080 con 0x7ff77c162270 2024-04-16T16:19:06.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.211+0000 7ff782800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff77c0d2e30 con 0x7ff77c0d52d0 2024-04-16T16:19:06.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.211+0000 7ff782800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff77c0d3360 con 0x7ff77c161e90 2024-04-16T16:19:06.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.212+0000 7ff781e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff77c0d52d0 0x7ff77c06ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:06.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.212+0000 7ff781e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff77c0d52d0 0x7ff77c06ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44596/0 (socket says 172.21.15.45:44596) 2024-04-16T16:19:06.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.212+0000 7ff781e00640 1 -- 172.21.15.45:0/2279911306 learned_addr learned my addr 172.21.15.45:0/2279911306 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:06.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.212+0000 7ff781400640 1 --2- 172.21.15.45:0/2279911306 >> v2:172.21.15.103:3300/0 conn(0x7ff77c161e90 0x7ff77c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.212+0000 7ff781e00640 1 -- 172.21.15.45:0/2279911306 >> v2:172.21.15.103:3300/0 conn(0x7ff77c161e90 msgr2=0x7ff77c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.212+0000 7ff781e00640 1 --2- 172.21.15.45:0/2279911306 >> v2:172.21.15.103:3300/0 conn(0x7ff77c161e90 0x7ff77c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.212+0000 7ff781e00640 1 -- 172.21.15.45:0/2279911306 >> v1:172.21.15.45:6789/0 conn(0x7ff77c162270 legacy=0x7ff77c0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.212+0000 7ff781e00640 1 -- 172.21.15.45:0/2279911306 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77c0d3640 con 0x7ff77c0d52d0 2024-04-16T16:19:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.212+0000 7ff77be00640 1 -- 172.21.15.45:0/2279911306 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3710995542 0 0) 0x7ff77c0d3080 con 0x7ff77c162270 2024-04-16T16:19:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.213+0000 7ff781e00640 1 --2- 172.21.15.45:0/2279911306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c0d52d0 0x7ff77c06ce40 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7ff774009e50 tx=0x7ff77402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a9df7cae77fc7f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.214+0000 7ff77be00640 1 -- 172.21.15.45:0/2279911306 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77402cb20 con 0x7ff77c0d52d0 2024-04-16T16:19:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.214+0000 7ff77be00640 1 -- 172.21.15.45:0/2279911306 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff77402ccc0 con 0x7ff77c0d52d0 2024-04-16T16:19:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.214+0000 7ff77be00640 1 -- 172.21.15.45:0/2279911306 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774034610 con 0x7ff77c0d52d0 2024-04-16T16:19:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.214+0000 7ff782800640 1 -- 172.21.15.45:0/2279911306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c0d52d0 msgr2=0x7ff77c06ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.214+0000 7ff782800640 1 --2- 172.21.15.45:0/2279911306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c0d52d0 0x7ff77c06ce40 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7ff774009e50 tx=0x7ff77402bf80 comp rx=0 tx=0).stop 2024-04-16T16:19:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.214+0000 7ff782800640 1 -- 172.21.15.45:0/2279911306 shutdown_connections 2024-04-16T16:19:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.214+0000 7ff782800640 1 --2- 172.21.15.45:0/2279911306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c0d52d0 0x7ff77c06ce40 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.214+0000 7ff782800640 1 --2- 172.21.15.45:0/2279911306 >> v2:172.21.15.103:3300/0 conn(0x7ff77c161e90 0x7ff77c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.214+0000 7ff782800640 1 -- 172.21.15.45:0/2279911306 >> 172.21.15.45:0/2279911306 conn(0x7ff77c0d1940 msgr2=0x7ff77c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.215+0000 7ff782800640 1 -- 172.21.15.45:0/2279911306 shutdown_connections 2024-04-16T16:19:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.215+0000 7ff782800640 1 -- 172.21.15.45:0/2279911306 wait complete. 2024-04-16T16:19:06.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.216+0000 7ff782800640 1 Processor -- start 2024-04-16T16:19:06.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.217+0000 7ff782800640 1 -- start start 2024-04-16T16:19:06.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.217+0000 7ff782800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff77c0d52d0 0x7ff77c0ab210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.217+0000 7ff782800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c161e90 0x7ff77c154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.217+0000 7ff782800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff77c162270 0x7ff77c1f3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.218+0000 7ff782800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff77c0d4550 con 0x7ff77c161e90 2024-04-16T16:19:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.218+0000 7ff782800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff77c0cfd10 con 0x7ff77c0d52d0 2024-04-16T16:19:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.218+0000 7ff782800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff77c0d46d0 con 0x7ff77c162270 2024-04-16T16:19:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.218+0000 7ff780a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c161e90 0x7ff77c154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.218+0000 7ff780a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c161e90 0x7ff77c154840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44600/0 (socket says 172.21.15.45:44600) 2024-04-16T16:19:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.218+0000 7ff780a00640 1 -- 172.21.15.45:0/3081678449 learned_addr learned my addr 172.21.15.45:0/3081678449 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:06.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.218+0000 7ff781400640 1 --2- 172.21.15.45:0/3081678449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff77c0d52d0 0x7ff77c0ab210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:06.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.218+0000 7ff781e00640 1 --2- 172.21.15.45:0/3081678449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff77c162270 0x7ff77c1f3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.218+0000 7ff780a00640 1 -- 172.21.15.45:0/3081678449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff77c162270 msgr2=0x7ff77c1f3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.218+0000 7ff780a00640 1 --2- 172.21.15.45:0/3081678449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff77c162270 0x7ff77c1f3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.218+0000 7ff780a00640 1 -- 172.21.15.45:0/3081678449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff77c0d52d0 msgr2=0x7ff77c0ab210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.218+0000 7ff780a00640 1 --2- 172.21.15.45:0/3081678449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff77c0d52d0 0x7ff77c0ab210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.218+0000 7ff780a00640 1 -- 172.21.15.45:0/3081678449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77c0ab8a0 con 0x7ff77c161e90 2024-04-16T16:19:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.218+0000 7ff781400640 1 --2- 172.21.15.45:0/3081678449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff77c0d52d0 0x7ff77c0ab210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.219+0000 7ff780a00640 1 --2- 172.21.15.45:0/3081678449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c161e90 0x7ff77c154840 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7ff76800c3e0 tx=0x7ff76800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.219+0000 7ff779600640 1 -- 172.21.15.45:0/3081678449 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768017070 con 0x7ff77c161e90 2024-04-16T16:19:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.219+0000 7ff779600640 1 -- 172.21.15.45:0/3081678449 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7680035f0 con 0x7ff77c161e90 2024-04-16T16:19:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.219+0000 7ff779600640 1 -- 172.21.15.45:0/3081678449 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768003a00 con 0x7ff77c161e90 2024-04-16T16:19:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.219+0000 7ff782800640 1 -- 172.21.15.45:0/3081678449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff77c14f400 con 0x7ff77c161e90 2024-04-16T16:19:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.219+0000 7ff782800640 1 -- 172.21.15.45:0/3081678449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff77c14fee0 con 0x7ff77c161e90 2024-04-16T16:19:06.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.220+0000 7ff782800640 1 -- 172.21.15.45:0/3081678449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff77c075780 con 0x7ff77c161e90 2024-04-16T16:19:06.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.222+0000 7ff779600640 1 -- 172.21.15.45:0/3081678449 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7ff76801a070 con 0x7ff77c161e90 2024-04-16T16:19:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.222+0000 7ff779600640 1 --2- 172.21.15.45:0/3081678449 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff75c06c9b0 0x7ff75c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.223+0000 7ff779600640 1 -- 172.21.15.45:0/3081678449 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7ff76808b090 con 0x7ff77c161e90 2024-04-16T16:19:06.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.223+0000 7ff781400640 1 --2- 172.21.15.45:0/3081678449 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff75c06c9b0 0x7ff75c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.224+0000 7ff781400640 1 --2- 172.21.15.45:0/3081678449 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff75c06c9b0 0x7ff75c06ee70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff770006fd0 tx=0x7ff770008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:06.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.227+0000 7ff779600640 1 -- 172.21.15.45:0/3081678449 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff7680593d0 con 0x7ff77c161e90 2024-04-16T16:19:06.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.436+0000 7ff782800640 1 -- 172.21.15.45:0/3081678449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff77c06dae0 con 0x7ff77c161e90 2024-04-16T16:19:06.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.440+0000 7ff779600640 1 -- 172.21.15.45:0/3081678449 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+16621 (secure 0 0 0) 0x7ff768058cd0 con 0x7ff77c161e90 2024-04-16T16:19:06.440 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:06.440 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":98,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:19:05.457441+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:03.429683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:04.434699+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:19:06.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.443+0000 7ff782800640 1 -- 172.21.15.45:0/3081678449 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff75c06c9b0 msgr2=0x7ff75c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:06.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.443+0000 7ff782800640 1 --2- 172.21.15.45:0/3081678449 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff75c06c9b0 0x7ff75c06ee70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff770006fd0 tx=0x7ff770008040 comp rx=0 tx=0).stop 2024-04-16T16:19:06.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.443+0000 7ff782800640 1 -- 172.21.15.45:0/3081678449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c161e90 msgr2=0x7ff77c154840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:06.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.443+0000 7ff782800640 1 --2- 172.21.15.45:0/3081678449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c161e90 0x7ff77c154840 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7ff76800c3e0 tx=0x7ff76800c8a0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.443+0000 7ff782800640 1 -- 172.21.15.45:0/3081678449 shutdown_connections 2024-04-16T16:19:06.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.443+0000 7ff782800640 1 --2- 172.21.15.45:0/3081678449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff77c162270 0x7ff77c1f3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.443+0000 7ff782800640 1 --2- 172.21.15.45:0/3081678449 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7ff75c06c9b0 0x7ff75c06ee70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.443+0000 7ff782800640 1 --2- 172.21.15.45:0/3081678449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c161e90 0x7ff77c154840 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.443+0000 7ff782800640 1 --2- 172.21.15.45:0/3081678449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff77c0d52d0 0x7ff77c0ab210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.443+0000 7ff782800640 1 -- 172.21.15.45:0/3081678449 >> 172.21.15.45:0/3081678449 conn(0x7ff77c0d1940 msgr2=0x7ff77c0b88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:06.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.443+0000 7ff782800640 1 -- 172.21.15.45:0/3081678449 shutdown_connections 2024-04-16T16:19:06.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.443+0000 7ff782800640 1 -- 172.21.15.45:0/3081678449 wait complete. 2024-04-16T16:19:06.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:19:06.520 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:06.520+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:19:06.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.683+0000 7f14cb800640 1 Processor -- start 2024-04-16T16:19:06.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.683+0000 7f14cb800640 1 -- start start 2024-04-16T16:19:06.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.685+0000 7f14cb800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c415fe20 0x7f14c406bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:06.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.685+0000 7f14cb800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f14c4160cd0 0x7f14c4161160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:06.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.685+0000 7f14cb800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14c41607f0 con 0x7f14c406b930 2024-04-16T16:19:06.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.685+0000 7f14cb800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f14c40b7290 con 0x7f14c4160cd0 2024-04-16T16:19:06.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.685+0000 7f14cb800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f14c406c250 con 0x7f14c415fe20 2024-04-16T16:19:06.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.686+0000 7f14c9800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f14c4160cd0 0x7f14c4161160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:06.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.686+0000 7f14c9800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f14c4160cd0 0x7f14c4161160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:33046/0 (socket says 172.21.15.45:33046) 2024-04-16T16:19:06.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.686+0000 7f14c9800640 1 -- 172.21.15.45:0/3603807373 learned_addr learned my addr 172.21.15.45:0/3603807373 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:06.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.686+0000 7f14c3e00640 1 --2- 172.21.15.45:0/3603807373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c415fe20 0x7f14c406bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:06.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.687+0000 7f14c9800640 1 -- 172.21.15.45:0/3603807373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c415fe20 msgr2=0x7f14c406bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:06.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.687+0000 7f14c9800640 1 --2- 172.21.15.45:0/3603807373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c415fe20 0x7f14c406bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.687+0000 7f14c9800640 1 -- 172.21.15.45:0/3603807373 >> v1:172.21.15.103:6789/0 conn(0x7f14c406b930 legacy=0x7f14c40b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:06.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.687+0000 7f14c9800640 1 -- 172.21.15.45:0/3603807373 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c40b3060 con 0x7f14c4160cd0 2024-04-16T16:19:06.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.687+0000 7f14c3e00640 1 --2- 172.21.15.45:0/3603807373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c415fe20 0x7f14c406bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:19:06.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.688+0000 7f14c9800640 1 --2- 172.21.15.45:0/3603807373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c4160cd0 0x7f14c4161160 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f14bc009e50 tx=0x7f14bc02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86df15132447eb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:06.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.689+0000 7f14c3400640 1 -- 172.21.15.45:0/3603807373 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc02cb00 con 0x7f14c4160cd0 2024-04-16T16:19:06.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.689+0000 7f14c3400640 1 -- 172.21.15.45:0/3603807373 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14bc02cca0 con 0x7f14c4160cd0 2024-04-16T16:19:06.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.689+0000 7f14c3400640 1 -- 172.21.15.45:0/3603807373 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc034610 con 0x7f14c4160cd0 2024-04-16T16:19:06.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.690+0000 7f14cb800640 1 -- 172.21.15.45:0/3603807373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c4160cd0 msgr2=0x7f14c4161160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:06.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.690+0000 7f14cb800640 1 --2- 172.21.15.45:0/3603807373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c4160cd0 0x7f14c4161160 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f14bc009e50 tx=0x7f14bc02bf60 comp rx=0 tx=0).stop 2024-04-16T16:19:06.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.690+0000 7f14cb800640 1 -- 172.21.15.45:0/3603807373 shutdown_connections 2024-04-16T16:19:06.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.690+0000 7f14cb800640 1 --2- 172.21.15.45:0/3603807373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c4160cd0 0x7f14c4161160 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.690+0000 7f14cb800640 1 --2- 172.21.15.45:0/3603807373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c415fe20 0x7f14c406bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.690+0000 7f14cb800640 1 -- 172.21.15.45:0/3603807373 >> 172.21.15.45:0/3603807373 conn(0x7f14c4073810 msgr2=0x7f14c409aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:06.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.690+0000 7f14cb800640 1 -- 172.21.15.45:0/3603807373 shutdown_connections 2024-04-16T16:19:06.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.690+0000 7f14cb800640 1 -- 172.21.15.45:0/3603807373 wait complete. 2024-04-16T16:19:06.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.692+0000 7f14cb800640 1 Processor -- start 2024-04-16T16:19:06.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.692+0000 7f14cb800640 1 -- start start 2024-04-16T16:19:06.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.693+0000 7f14cb800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c406b930 0x7f14c4159830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:06.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.693+0000 7f14cb800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c415fe20 0x7f14c4150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:06.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.693+0000 7f14cb800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c4160cd0 0x7f14c4150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:06.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.693+0000 7f14cb800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f14c40b2d80 con 0x7f14c406b930 2024-04-16T16:19:06.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.693+0000 7f14cb800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f14c40b2aa0 con 0x7f14c4160cd0 2024-04-16T16:19:06.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.693+0000 7f14cb800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f14c415ef20 con 0x7f14c415fe20 2024-04-16T16:19:06.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.693+0000 7f14c8e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c406b930 0x7f14c4159830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:06.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.693+0000 7f14c8e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c406b930 0x7f14c4159830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44610/0 (socket says 172.21.15.45:44610) 2024-04-16T16:19:06.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.693+0000 7f14c8e00640 1 -- 172.21.15.45:0/1461395198 learned_addr learned my addr 172.21.15.45:0/1461395198 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:06.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.693+0000 7f14c3e00640 1 --2- 172.21.15.45:0/1461395198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c415fe20 0x7f14c4150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:06.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.694+0000 7f14c8e00640 1 -- 172.21.15.45:0/1461395198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c415fe20 msgr2=0x7f14c4150e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:06.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.694+0000 7f14c8e00640 1 --2- 172.21.15.45:0/1461395198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c415fe20 0x7f14c4150e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.694+0000 7f14c8e00640 1 -- 172.21.15.45:0/1461395198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c4160cd0 msgr2=0x7f14c4150860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:06.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.694+0000 7f14c9800640 1 --2- 172.21.15.45:0/1461395198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c4160cd0 0x7f14c4150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:06.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.694+0000 7f14c8e00640 1 --2- 172.21.15.45:0/1461395198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c4160cd0 0x7f14c4150860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.694+0000 7f14c8e00640 1 -- 172.21.15.45:0/1461395198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c415c1f0 con 0x7f14c406b930 2024-04-16T16:19:06.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.694+0000 7f14c8e00640 1 --2- 172.21.15.45:0/1461395198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c406b930 0x7f14c4159830 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f14b800c430 tx=0x7f14b800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:06.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.694+0000 7f14c0c00640 1 -- 172.21.15.45:0/1461395198 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b8017070 con 0x7f14c406b930 2024-04-16T16:19:06.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.694+0000 7f14c0c00640 1 -- 172.21.15.45:0/1461395198 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b80052d0 con 0x7f14c406b930 2024-04-16T16:19:06.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.694+0000 7f14c0c00640 1 -- 172.21.15.45:0/1461395198 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b8003830 con 0x7f14c406b930 2024-04-16T16:19:06.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.694+0000 7f14cb800640 1 -- 172.21.15.45:0/1461395198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14c415c4d0 con 0x7f14c406b930 2024-04-16T16:19:06.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.694+0000 7f14cb800640 1 -- 172.21.15.45:0/1461395198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14c40d0580 con 0x7f14c406b930 2024-04-16T16:19:06.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.696+0000 7f14cb800640 1 -- 172.21.15.45:0/1461395198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14c415c990 con 0x7f14c406b930 2024-04-16T16:19:06.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.697+0000 7f14c0c00640 1 -- 172.21.15.45:0/1461395198 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f14b801a070 con 0x7f14c406b930 2024-04-16T16:19:06.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.698+0000 7f14c0c00640 1 --2- 172.21.15.45:0/1461395198 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f14a406c980 0x7f14a406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:06.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.698+0000 7f14c0c00640 1 -- 172.21.15.45:0/1461395198 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7f14b808afb0 con 0x7f14c406b930 2024-04-16T16:19:06.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.698+0000 7f14c3e00640 1 --2- 172.21.15.45:0/1461395198 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f14a406c980 0x7f14a406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:06.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.700+0000 7f14c3e00640 1 --2- 172.21.15.45:0/1461395198 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f14a406c980 0x7f14a406ee40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f14c415a400 tx=0x7f14b0008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:06.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.702+0000 7f14c0c00640 1 -- 172.21.15.45:0/1461395198 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f14b80592f0 con 0x7f14c406b930 2024-04-16T16:19:06.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.911+0000 7f14cb800640 1 -- 172.21.15.45:0/1461395198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14c4075780 con 0x7f14c406b930 2024-04-16T16:19:06.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.914+0000 7f14c0c00640 1 -- 172.21.15.45:0/1461395198 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+16621 (secure 0 0 0) 0x7f14b8058bf0 con 0x7f14c406b930 2024-04-16T16:19:06.915 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:06.915 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":98,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:19:05.457441+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:03.429683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:04.434699+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:19:06.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.917+0000 7f14cb800640 1 -- 172.21.15.45:0/1461395198 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f14a406c980 msgr2=0x7f14a406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:06.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.917+0000 7f14cb800640 1 --2- 172.21.15.45:0/1461395198 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f14a406c980 0x7f14a406ee40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f14c415a400 tx=0x7f14b0008040 comp rx=0 tx=0).stop 2024-04-16T16:19:06.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.917+0000 7f14cb800640 1 -- 172.21.15.45:0/1461395198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c406b930 msgr2=0x7f14c4159830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:06.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.917+0000 7f14cb800640 1 --2- 172.21.15.45:0/1461395198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c406b930 0x7f14c4159830 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f14b800c430 tx=0x7f14b800c8f0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.917+0000 7f14cb800640 1 -- 172.21.15.45:0/1461395198 shutdown_connections 2024-04-16T16:19:06.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.917+0000 7f14cb800640 1 --2- 172.21.15.45:0/1461395198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c4160cd0 0x7f14c4150860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.917+0000 7f14cb800640 1 --2- 172.21.15.45:0/1461395198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c415fe20 0x7f14c4150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.917+0000 7f14cb800640 1 --2- 172.21.15.45:0/1461395198 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f14a406c980 0x7f14a406ee40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.917+0000 7f14cb800640 1 --2- 172.21.15.45:0/1461395198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c406b930 0x7f14c4159830 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:06.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.917+0000 7f14cb800640 1 -- 172.21.15.45:0/1461395198 >> 172.21.15.45:0/1461395198 conn(0x7f14c4073810 msgr2=0x7f14c409aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:06.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.917+0000 7f14cb800640 1 -- 172.21.15.45:0/1461395198 shutdown_connections 2024-04-16T16:19:06.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:06.917+0000 7f14cb800640 1 -- 172.21.15.45:0/1461395198 wait complete. 2024-04-16T16:19:06.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-16T16:19:07.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.154+0000 7fbca1400640 1 Processor -- start 2024-04-16T16:19:07.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.155+0000 7fbca1400640 1 -- start start 2024-04-16T16:19:07.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.157+0000 7fbca1400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbc9c0d18f0 0x7fbc9c0ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:07.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.157+0000 7fbca1400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc9c0b6810 con 0x7fbc9c0b6310 2024-04-16T16:19:07.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.157+0000 7fbca1400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc9c0b6af0 con 0x7fbc9c0d1cd0 2024-04-16T16:19:07.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.157+0000 7fbca1400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbc9c0d29f0 con 0x7fbc9c0d18f0 2024-04-16T16:19:07.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.157+0000 7fbc93e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbc9c0b6310 0x7fbc9c0d2e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37858/0 (socket says 172.21.15.45:37858) 2024-04-16T16:19:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.157+0000 7fbc93e00640 1 -- 172.21.15.45:0/4044143290 learned_addr learned my addr 172.21.15.45:0/4044143290 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.157+0000 7fbc9be00640 1 --2- 172.21.15.45:0/4044143290 >> v2:172.21.15.103:3300/0 conn(0x7fbc9c0d18f0 0x7fbc9c0ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.158+0000 7fbc9b400640 1 -- 172.21.15.45:0/4044143290 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2288881479 0 0) 0x7fbc9c0b6810 con 0x7fbc9c0b6310 2024-04-16T16:19:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.158+0000 7fbc9b400640 1 -- 172.21.15.45:0/4044143290 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc80003520 con 0x7fbc9c0b6310 2024-04-16T16:19:07.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.158+0000 7fbc9b400640 1 -- 172.21.15.45:0/4044143290 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2861846275 0 0) 0x7fbc80003520 con 0x7fbc9c0b6310 2024-04-16T16:19:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.158+0000 7fbc9b400640 1 -- 172.21.15.45:0/4044143290 >> v2:172.21.15.103:3300/0 conn(0x7fbc9c0d18f0 msgr2=0x7fbc9c0ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.158+0000 7fbc9b400640 1 --2- 172.21.15.45:0/4044143290 >> v2:172.21.15.103:3300/0 conn(0x7fbc9c0d18f0 0x7fbc9c0ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.158+0000 7fbc9b400640 1 -- 172.21.15.45:0/4044143290 >> v1:172.21.15.103:6789/0 conn(0x7fbc9c0d1cd0 legacy=0x7fbc9c162dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.158+0000 7fbc9b400640 1 -- 172.21.15.45:0/4044143290 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9c0b6dd0 con 0x7fbc9c0b6310 2024-04-16T16:19:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.158+0000 7fbc9b400640 1 -- 172.21.15.45:0/4044143290 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fbc84001670 con 0x7fbc9c0b6310 2024-04-16T16:19:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.158+0000 7fbc9b400640 1 -- 172.21.15.45:0/4044143290 >> v1:172.21.15.45:6789/0 conn(0x7fbc9c0b6310 legacy=0x7fbc9c0d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.158+0000 7fbc9b400640 1 --2- 172.21.15.45:0/4044143290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc800039b0 0x7fbc80003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.158+0000 7fbc9b400640 1 --2- 172.21.15.45:0/4044143290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc80004800 0x7fbc80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.158+0000 7fbc9b400640 1 --2- 172.21.15.45:0/4044143290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc800052e0 0x7fbc8000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.158+0000 7fbc9b400640 1 -- 172.21.15.45:0/4044143290 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c0d29f0 con 0x7fbc80004800 2024-04-16T16:19:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.158+0000 7fbc9b400640 1 -- 172.21.15.45:0/4044143290 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8000c1b0 con 0x7fbc800052e0 2024-04-16T16:19:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.158+0000 7fbc9b400640 1 -- 172.21.15.45:0/4044143290 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbc8000c3b0 con 0x7fbc800039b0 2024-04-16T16:19:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.159+0000 7fbc93e00640 1 --2- 172.21.15.45:0/4044143290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc80004800 0x7fbc80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.159+0000 7fbca0a00640 1 --2- 172.21.15.45:0/4044143290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc800052e0 0x7fbc8000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:07.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.159+0000 7fbc9be00640 1 --2- 172.21.15.45:0/4044143290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc800039b0 0x7fbc80003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:07.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.159+0000 7fbc93e00640 1 -- 172.21.15.45:0/4044143290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc800039b0 msgr2=0x7fbc80003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:07.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.159+0000 7fbc93e00640 1 --2- 172.21.15.45:0/4044143290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc800039b0 0x7fbc80003df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:07.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.159+0000 7fbc93e00640 1 -- 172.21.15.45:0/4044143290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc800052e0 msgr2=0x7fbc8000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:07.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.159+0000 7fbc93e00640 1 --2- 172.21.15.45:0/4044143290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc800052e0 0x7fbc8000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:07.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.159+0000 7fbc93e00640 1 -- 172.21.15.45:0/4044143290 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc80003520 con 0x7fbc80004800 2024-04-16T16:19:07.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.159+0000 7fbca0a00640 1 -- 172.21.15.45:0/4044143290 reap_dead start 2024-04-16T16:19:07.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.160+0000 7fbc93e00640 1 --2- 172.21.15.45:0/4044143290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc80004800 0x7fbc80004c80 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400c210 tx=0x7fbc80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f64b43f9358e2165 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:07.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.160+0000 7fbc9b400640 1 -- 172.21.15.45:0/4044143290 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc84007d10 con 0x7fbc80004800 2024-04-16T16:19:07.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.161+0000 7fbc9b400640 1 -- 172.21.15.45:0/4044143290 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc84030070 con 0x7fbc80004800 2024-04-16T16:19:07.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.161+0000 7fbc9b400640 1 -- 172.21.15.45:0/4044143290 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc84005d90 con 0x7fbc80004800 2024-04-16T16:19:07.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.161+0000 7fbca1400640 1 -- 172.21.15.45:0/4044143290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc80004800 msgr2=0x7fbc80004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:07.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.161+0000 7fbca1400640 1 --2- 172.21.15.45:0/4044143290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc80004800 0x7fbc80004c80 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400c210 tx=0x7fbc80005830 comp rx=0 tx=0).stop 2024-04-16T16:19:07.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.161+0000 7fbca1400640 1 -- 172.21.15.45:0/4044143290 shutdown_connections 2024-04-16T16:19:07.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.161+0000 7fbca1400640 1 --2- 172.21.15.45:0/4044143290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc80004800 0x7fbc80004c80 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:07.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.161+0000 7fbca1400640 1 -- 172.21.15.45:0/4044143290 >> 172.21.15.45:0/4044143290 conn(0x7fbc9c06cf90 msgr2=0x7fbc9c06d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:07.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.161+0000 7fbca1400640 1 -- 172.21.15.45:0/4044143290 shutdown_connections 2024-04-16T16:19:07.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.161+0000 7fbca1400640 1 -- 172.21.15.45:0/4044143290 wait complete. 2024-04-16T16:19:07.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.163+0000 7fbca1400640 1 Processor -- start 2024-04-16T16:19:07.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.164+0000 7fbca1400640 1 -- start start 2024-04-16T16:19:07.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.164+0000 7fbca1400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc80004800 0x7fbc9c0ab260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:07.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.164+0000 7fbca1400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc9c151f00 0x7fbc9c1e78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:07.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.164+0000 7fbca1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0cdc90 0x7fbc9c0ce110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:07.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.165+0000 7fbca1400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c0ba3a0 con 0x7fbc9c0cdc90 2024-04-16T16:19:07.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.165+0000 7fbca1400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c0ba220 con 0x7fbc9c151f00 2024-04-16T16:19:07.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.165+0000 7fbca1400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbc9c0ba520 con 0x7fbc80004800 2024-04-16T16:19:07.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.165+0000 7fbca0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0cdc90 0x7fbc9c0ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:07.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.165+0000 7fbca0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0cdc90 0x7fbc9c0ce110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44634/0 (socket says 172.21.15.45:44634) 2024-04-16T16:19:07.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.165+0000 7fbca0a00640 1 -- 172.21.15.45:0/1188946909 learned_addr learned my addr 172.21.15.45:0/1188946909 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:07.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.165+0000 7fbc9be00640 1 --2- 172.21.15.45:0/1188946909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc80004800 0x7fbc9c0ab260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:07.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.165+0000 7fbc93e00640 1 --2- 172.21.15.45:0/1188946909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc9c151f00 0x7fbc9c1e78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:07.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.165+0000 7fbca0a00640 1 -- 172.21.15.45:0/1188946909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc80004800 msgr2=0x7fbc9c0ab260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:07.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.165+0000 7fbca0a00640 1 --2- 172.21.15.45:0/1188946909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc80004800 0x7fbc9c0ab260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:07.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.165+0000 7fbca0a00640 1 -- 172.21.15.45:0/1188946909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc9c151f00 msgr2=0x7fbc9c1e78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:07.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.165+0000 7fbca0a00640 1 --2- 172.21.15.45:0/1188946909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc9c151f00 0x7fbc9c1e78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:07.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.165+0000 7fbca0a00640 1 -- 172.21.15.45:0/1188946909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9c0cc7d0 con 0x7fbc9c0cdc90 2024-04-16T16:19:07.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.165+0000 7fbc93e00640 1 --2- 172.21.15.45:0/1188946909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc9c151f00 0x7fbc9c1e78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:07.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.166+0000 7fbca0a00640 1 --2- 172.21.15.45:0/1188946909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0cdc90 0x7fbc9c0ce110 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fbc94007fd0 tx=0x7fbc9400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:07.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.166+0000 7fbc98c00640 1 -- 172.21.15.45:0/1188946909 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc94019070 con 0x7fbc9c0cdc90 2024-04-16T16:19:07.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.166+0000 7fbc98c00640 1 -- 172.21.15.45:0/1188946909 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc94006b50 con 0x7fbc9c0cdc90 2024-04-16T16:19:07.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.166+0000 7fbc98c00640 1 -- 172.21.15.45:0/1188946909 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc94003970 con 0x7fbc9c0cdc90 2024-04-16T16:19:07.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.166+0000 7fbca1400640 1 -- 172.21.15.45:0/1188946909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc9c0ccab0 con 0x7fbc9c0cdc90 2024-04-16T16:19:07.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.166+0000 7fbca1400640 1 -- 172.21.15.45:0/1188946909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc9c0abf50 con 0x7fbc9c0cdc90 2024-04-16T16:19:07.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.167+0000 7fbca1400640 1 -- 172.21.15.45:0/1188946909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc9c075780 con 0x7fbc9c0cdc90 2024-04-16T16:19:07.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.168+0000 7fbc98c00640 1 -- 172.21.15.45:0/1188946909 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7fbc9401c070 con 0x7fbc9c0cdc90 2024-04-16T16:19:07.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.169+0000 7fbc98c00640 1 --2- 172.21.15.45:0/1188946909 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fbc7806c980 0x7fbc7806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:07.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.169+0000 7fbc98c00640 1 -- 172.21.15.45:0/1188946909 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5709+0+0 (secure 0 0 0) 0x7fbc9408b300 con 0x7fbc9c0cdc90 2024-04-16T16:19:07.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.170+0000 7fbc9be00640 1 --2- 172.21.15.45:0/1188946909 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fbc7806c980 0x7fbc7806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:07.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.171+0000 7fbc9be00640 1 --2- 172.21.15.45:0/1188946909 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fbc7806c980 0x7fbc7806ee40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c160de0 tx=0x7fbc8c008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:07.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.174+0000 7fbc98c00640 1 -- 172.21.15.45:0/1188946909 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbc94059640 con 0x7fbc9c0cdc90 2024-04-16T16:19:07.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.434+0000 7fbca1400640 1 -- 172.21.15.45:0/1188946909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fbc9c0cd9f0 con 0x7fbc9c0cdc90 2024-04-16T16:19:07.573 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:07.573+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:19:07.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.988+0000 7fbc98c00640 1 -- 172.21.15.45:0/1188946909 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v72) ==== 118+0+0 (secure 0 0 0) 0x7fbc94058f40 con 0x7fbc9c0cdc90 2024-04-16T16:19:07.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.992+0000 7fbca1400640 1 -- 172.21.15.45:0/1188946909 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fbc7806c980 msgr2=0x7fbc7806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:07.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.992+0000 7fbca1400640 1 --2- 172.21.15.45:0/1188946909 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fbc7806c980 0x7fbc7806ee40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c160de0 tx=0x7fbc8c008040 comp rx=0 tx=0).stop 2024-04-16T16:19:07.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.992+0000 7fbca1400640 1 -- 172.21.15.45:0/1188946909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0cdc90 msgr2=0x7fbc9c0ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:07.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.992+0000 7fbca1400640 1 --2- 172.21.15.45:0/1188946909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0cdc90 0x7fbc9c0ce110 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fbc94007fd0 tx=0x7fbc9400ec10 comp rx=0 tx=0).stop 2024-04-16T16:19:07.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.992+0000 7fbca1400640 1 -- 172.21.15.45:0/1188946909 shutdown_connections 2024-04-16T16:19:07.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.992+0000 7fbca1400640 1 --2- 172.21.15.45:0/1188946909 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fbc7806c980 0x7fbc7806ee40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:07.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.992+0000 7fbca1400640 1 --2- 172.21.15.45:0/1188946909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0cdc90 0x7fbc9c0ce110 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:07.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.992+0000 7fbca1400640 1 --2- 172.21.15.45:0/1188946909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc9c151f00 0x7fbc9c1e78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:07.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.992+0000 7fbca1400640 1 --2- 172.21.15.45:0/1188946909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc80004800 0x7fbc9c0ab260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:07.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.992+0000 7fbca1400640 1 -- 172.21.15.45:0/1188946909 >> 172.21.15.45:0/1188946909 conn(0x7fbc9c06cf90 msgr2=0x7fbc9c06d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:07.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.992+0000 7fbca1400640 1 -- 172.21.15.45:0/1188946909 shutdown_connections 2024-04-16T16:19:07.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:07.992+0000 7fbca1400640 1 -- 172.21.15.45:0/1188946909 wait complete. 2024-04-16T16:19:08.016 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-16T16:19:08.016 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-16T16:19:08.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.247+0000 7fd9a8000640 1 Processor -- start 2024-04-16T16:19:08.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.247+0000 7fd9a8000640 1 -- start start 2024-04-16T16:19:08.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.249+0000 7fd9a8000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9a00b4930 0x7fd9a0072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:08.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.249+0000 7fd9a8000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9a00b5530 con 0x7fd9a015c7c0 2024-04-16T16:19:08.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.249+0000 7fd9a8000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9a00b5810 con 0x7fd9a00d10e0 2024-04-16T16:19:08.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.249+0000 7fd9a8000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd9a00b52e0 con 0x7fd9a00b4930 2024-04-16T16:19:08.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.249+0000 7fd9a5600640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd9a015c7c0 0x7fd9a00d09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37870/0 (socket says 172.21.15.45:37870) 2024-04-16T16:19:08.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.249+0000 7fd9a5600640 1 -- 172.21.15.45:0/1374490181 learned_addr learned my addr 172.21.15.45:0/1374490181 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:08.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.250+0000 7fd9a6000640 1 --2- 172.21.15.45:0/1374490181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9a00b4930 0x7fd9a0072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:08.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.250+0000 7fd99fe00640 1 -- 172.21.15.45:0/1374490181 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2659739523 0 0) 0x7fd9a00b5530 con 0x7fd9a015c7c0 2024-04-16T16:19:08.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.250+0000 7fd99fe00640 1 -- 172.21.15.45:0/1374490181 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd988003520 con 0x7fd9a015c7c0 2024-04-16T16:19:08.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.250+0000 7fd99fe00640 1 -- 172.21.15.45:0/1374490181 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1878413385 0 0) 0x7fd988003520 con 0x7fd9a015c7c0 2024-04-16T16:19:08.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.250+0000 7fd99fe00640 1 -- 172.21.15.45:0/1374490181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9a00b4930 msgr2=0x7fd9a0072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:08.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.250+0000 7fd99fe00640 1 --2- 172.21.15.45:0/1374490181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9a00b4930 0x7fd9a0072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.250+0000 7fd99fe00640 1 -- 172.21.15.45:0/1374490181 >> v1:172.21.15.103:6789/0 conn(0x7fd9a00d10e0 legacy=0x7fd9a006f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.250+0000 7fd99fe00640 1 -- 172.21.15.45:0/1374490181 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a00d04b0 con 0x7fd9a015c7c0 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.250+0000 7fd99fe00640 1 -- 172.21.15.45:0/1374490181 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fd994001690 con 0x7fd9a015c7c0 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.250+0000 7fd99fe00640 1 -- 172.21.15.45:0/1374490181 >> v1:172.21.15.45:6789/0 conn(0x7fd9a015c7c0 legacy=0x7fd9a00d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.250+0000 7fd99fe00640 1 --2- 172.21.15.45:0/1374490181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9880039e0 0x7fd988003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.251+0000 7fd99fe00640 1 --2- 172.21.15.45:0/1374490181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9a00b4930 0x7fd988004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.251+0000 7fd99fe00640 1 --2- 172.21.15.45:0/1374490181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd988004e60 0x7fd98800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.251+0000 7fd99fe00640 1 -- 172.21.15.45:0/1374490181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a00b52e0 con 0x7fd988004e60 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.251+0000 7fd99fe00640 1 -- 172.21.15.45:0/1374490181 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd988005620 con 0x7fd9880039e0 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.251+0000 7fd99fe00640 1 -- 172.21.15.45:0/1374490181 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd98800bf00 con 0x7fd9a00b4930 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.251+0000 7fd9a6000640 1 --2- 172.21.15.45:0/1374490181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd988004e60 0x7fd98800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.251+0000 7fd9a5600640 1 --2- 172.21.15.45:0/1374490181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9880039e0 0x7fd988003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.251+0000 7fd9a6000640 1 -- 172.21.15.45:0/1374490181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9a00b4930 msgr2=0x7fd988004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.251+0000 7fd9a6000640 1 --2- 172.21.15.45:0/1374490181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9a00b4930 0x7fd988004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.251+0000 7fd9a6000640 1 -- 172.21.15.45:0/1374490181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9880039e0 msgr2=0x7fd988003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:08.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.251+0000 7fd9a6000640 1 --2- 172.21.15.45:0/1374490181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9880039e0 0x7fd988003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:08.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.251+0000 7fd9a6000640 1 -- 172.21.15.45:0/1374490181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd988003520 con 0x7fd988004e60 2024-04-16T16:19:08.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.251+0000 7fd9a5600640 1 --2- 172.21.15.45:0/1374490181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9880039e0 0x7fd988003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:08.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.252+0000 7fd9a6000640 1 --2- 172.21.15.45:0/1374490181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd988004e60 0x7fd98800b9c0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fd998007360 tx=0x7fd988005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d170d80a492f39dc server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:08.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.253+0000 7fd99fe00640 1 -- 172.21.15.45:0/1374490181 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd998039070 con 0x7fd988004e60 2024-04-16T16:19:08.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.253+0000 7fd99fe00640 1 -- 172.21.15.45:0/1374490181 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd99802cce0 con 0x7fd988004e60 2024-04-16T16:19:08.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.253+0000 7fd99fe00640 1 -- 172.21.15.45:0/1374490181 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd998009be0 con 0x7fd988004e60 2024-04-16T16:19:08.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.253+0000 7fd9a8000640 1 -- 172.21.15.45:0/1374490181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd988004e60 msgr2=0x7fd98800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:08.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.253+0000 7fd9a8000640 1 --2- 172.21.15.45:0/1374490181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd988004e60 0x7fd98800b9c0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fd998007360 tx=0x7fd988005830 comp rx=0 tx=0).stop 2024-04-16T16:19:08.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.253+0000 7fd9a6000640 1 -- 172.21.15.45:0/1374490181 reap_dead start 2024-04-16T16:19:08.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.253+0000 7fd9a8000640 1 -- 172.21.15.45:0/1374490181 shutdown_connections 2024-04-16T16:19:08.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.253+0000 7fd9a8000640 1 -- 172.21.15.45:0/1374490181 >> 172.21.15.45:0/1374490181 conn(0x7fd9a00b3970 msgr2=0x7fd9a00b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:08.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.253+0000 7fd9a8000640 1 -- 172.21.15.45:0/1374490181 shutdown_connections 2024-04-16T16:19:08.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.254+0000 7fd9a8000640 1 -- 172.21.15.45:0/1374490181 wait complete. 2024-04-16T16:19:08.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.255+0000 7fd9a8000640 1 Processor -- start 2024-04-16T16:19:08.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.255+0000 7fd9a8000640 1 -- start start 2024-04-16T16:19:08.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.256+0000 7fd9a8000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd988004e60 0x7fd9a0146010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:08.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.256+0000 7fd9a8000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9a01e58e0 0x7fd9a00ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:08.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.256+0000 7fd9a8000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9a01e5d60 0x7fd9a0147cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:08.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.256+0000 7fd9a8000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a00d0260 con 0x7fd9a01e58e0 2024-04-16T16:19:08.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.256+0000 7fd9a8000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a006fac0 con 0x7fd9a01e5d60 2024-04-16T16:19:08.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.256+0000 7fd9a8000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd9a0072e10 con 0x7fd988004e60 2024-04-16T16:19:08.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.257+0000 7fd9a4c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9a01e58e0 0x7fd9a00ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:08.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.257+0000 7fd9a4c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9a01e58e0 0x7fd9a00ae9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44658/0 (socket says 172.21.15.45:44658) 2024-04-16T16:19:08.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.257+0000 7fd9a4c00640 1 -- 172.21.15.45:0/3199621846 learned_addr learned my addr 172.21.15.45:0/3199621846 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:08.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.257+0000 7fd9a5600640 1 --2- 172.21.15.45:0/3199621846 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd988004e60 0x7fd9a0146010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:08.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.257+0000 7fd9a4c00640 1 -- 172.21.15.45:0/3199621846 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd988004e60 msgr2=0x7fd9a0146010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:08.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.257+0000 7fd9a4c00640 1 --2- 172.21.15.45:0/3199621846 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd988004e60 0x7fd9a0146010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:08.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.257+0000 7fd9a4c00640 1 -- 172.21.15.45:0/3199621846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9a01e5d60 msgr2=0x7fd9a0147cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:08.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.257+0000 7fd9a4c00640 1 --2- 172.21.15.45:0/3199621846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9a01e5d60 0x7fd9a0147cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:08.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.257+0000 7fd9a4c00640 1 -- 172.21.15.45:0/3199621846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a0146550 con 0x7fd9a01e58e0 2024-04-16T16:19:08.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.258+0000 7fd9a4c00640 1 --2- 172.21.15.45:0/3199621846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9a01e58e0 0x7fd9a00ae9b0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00c4b0 tx=0x7fd98c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:08.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.258+0000 7fd99d600640 1 -- 172.21.15.45:0/3199621846 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c0052d0 con 0x7fd9a01e58e0 2024-04-16T16:19:08.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.258+0000 7fd99d600640 1 -- 172.21.15.45:0/3199621846 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd98c005470 con 0x7fd9a01e58e0 2024-04-16T16:19:08.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.258+0000 7fd99d600640 1 -- 172.21.15.45:0/3199621846 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c012870 con 0x7fd9a01e58e0 2024-04-16T16:19:08.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.258+0000 7fd9a8000640 1 -- 172.21.15.45:0/3199621846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a0146c10 con 0x7fd9a01e58e0 2024-04-16T16:19:08.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.258+0000 7fd9a8000640 1 -- 172.21.15.45:0/3199621846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a01471d0 con 0x7fd9a01e58e0 2024-04-16T16:19:08.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.260+0000 7fd9a8000640 1 -- 172.21.15.45:0/3199621846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9a00d1740 con 0x7fd9a01e58e0 2024-04-16T16:19:08.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.260+0000 7fd99d600640 1 -- 172.21.15.45:0/3199621846 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7fd98c012a10 con 0x7fd9a01e58e0 2024-04-16T16:19:08.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.261+0000 7fd99d600640 1 --2- 172.21.15.45:0/3199621846 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd98006c9b0 0x7fd98006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:08.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.261+0000 7fd99d600640 1 -- 172.21.15.45:0/3199621846 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5709+0+0 (secure 0 0 0) 0x7fd98c08b730 con 0x7fd9a01e58e0 2024-04-16T16:19:08.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.262+0000 7fd9a5600640 1 --2- 172.21.15.45:0/3199621846 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd98006c9b0 0x7fd98006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:08.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.263+0000 7fd9a5600640 1 --2- 172.21.15.45:0/3199621846 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd98006c9b0 0x7fd98006ee70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd994001eb0 tx=0x7fd994009040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:08.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.266+0000 7fd99d600640 1 -- 172.21.15.45:0/3199621846 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9a0146550 con 0x7fd9a01e58e0 2024-04-16T16:19:08.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.524+0000 7fd9a8000640 1 -- 172.21.15.45:0/3199621846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd9a0075700 con 0x7fd9a01e58e0 2024-04-16T16:19:08.640 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:08.639+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:19:08.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:08.998+0000 7fd99d600640 1 -- 172.21.15.45:0/3199621846 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v73) ==== 122+0+0 (secure 0 0 0) 0x7fd98c059b00 con 0x7fd9a01e58e0 2024-04-16T16:19:09.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.003+0000 7fd9a8000640 1 -- 172.21.15.45:0/3199621846 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd98006c9b0 msgr2=0x7fd98006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:09.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.003+0000 7fd9a8000640 1 --2- 172.21.15.45:0/3199621846 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd98006c9b0 0x7fd98006ee70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd994001eb0 tx=0x7fd994009040 comp rx=0 tx=0).stop 2024-04-16T16:19:09.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.003+0000 7fd9a8000640 1 -- 172.21.15.45:0/3199621846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9a01e58e0 msgr2=0x7fd9a00ae9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:09.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.003+0000 7fd9a8000640 1 --2- 172.21.15.45:0/3199621846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9a01e58e0 0x7fd9a00ae9b0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00c4b0 tx=0x7fd98c00c970 comp rx=0 tx=0).stop 2024-04-16T16:19:09.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.003+0000 7fd9a8000640 1 -- 172.21.15.45:0/3199621846 shutdown_connections 2024-04-16T16:19:09.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.004+0000 7fd9a8000640 1 --2- 172.21.15.45:0/3199621846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9a01e5d60 0x7fd9a0147cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.004+0000 7fd9a8000640 1 --2- 172.21.15.45:0/3199621846 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd98006c9b0 0x7fd98006ee70 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.004+0000 7fd9a8000640 1 --2- 172.21.15.45:0/3199621846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9a01e58e0 0x7fd9a00ae9b0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.004+0000 7fd9a8000640 1 --2- 172.21.15.45:0/3199621846 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd988004e60 0x7fd9a0146010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.004+0000 7fd9a8000640 1 -- 172.21.15.45:0/3199621846 >> 172.21.15.45:0/3199621846 conn(0x7fd9a00b3970 msgr2=0x7fd9a015e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:09.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.004+0000 7fd9a8000640 1 -- 172.21.15.45:0/3199621846 shutdown_connections 2024-04-16T16:19:09.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.004+0000 7fd9a8000640 1 -- 172.21.15.45:0/3199621846 wait complete. 2024-04-16T16:19:09.028 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-16T16:19:09.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:19:09.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.251+0000 7fad1ae00640 1 Processor -- start 2024-04-16T16:19:09.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.252+0000 7fad1ae00640 1 -- start start 2024-04-16T16:19:09.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.253+0000 7fad1ae00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fad14162270 0x7fad140d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:09.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.253+0000 7fad1ae00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fad140d52d0 0x7fad1406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:09.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.253+0000 7fad1ae00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad140d3080 con 0x7fad14161e90 2024-04-16T16:19:09.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.253+0000 7fad1ae00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fad140d2e30 con 0x7fad140d52d0 2024-04-16T16:19:09.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.253+0000 7fad1ae00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fad140d3360 con 0x7fad14162270 2024-04-16T16:19:09.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.254+0000 7fad1a400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fad140d52d0 0x7fad1406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:09.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.254+0000 7fad1a400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fad140d52d0 0x7fad1406ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44664/0 (socket says 172.21.15.45:44664) 2024-04-16T16:19:09.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.254+0000 7fad1a400640 1 -- 172.21.15.45:0/140486941 learned_addr learned my addr 172.21.15.45:0/140486941 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:09.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.254+0000 7fad19000640 1 --2- 172.21.15.45:0/140486941 >> v2:172.21.15.103:3300/0 conn(0x7fad14162270 0x7fad140d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:09.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.254+0000 7fad1a400640 1 -- 172.21.15.45:0/140486941 >> v2:172.21.15.103:3300/0 conn(0x7fad14162270 msgr2=0x7fad140d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:09.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.254+0000 7fad1a400640 1 --2- 172.21.15.45:0/140486941 >> v2:172.21.15.103:3300/0 conn(0x7fad14162270 0x7fad140d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.254+0000 7fad1a400640 1 -- 172.21.15.45:0/140486941 >> v1:172.21.15.103:6789/0 conn(0x7fad14161e90 legacy=0x7fad140cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:09.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.254+0000 7fad1a400640 1 -- 172.21.15.45:0/140486941 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad140d3640 con 0x7fad140d52d0 2024-04-16T16:19:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.256+0000 7fad1a400640 1 --2- 172.21.15.45:0/140486941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad140d52d0 0x7fad1406ce40 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fad0c009e50 tx=0x7fad0c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db89dd1201fd8ee2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.256+0000 7fad13e00640 1 -- 172.21.15.45:0/140486941 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c02ca80 con 0x7fad140d52d0 2024-04-16T16:19:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.256+0000 7fad13e00640 1 -- 172.21.15.45:0/140486941 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad0c02cc20 con 0x7fad140d52d0 2024-04-16T16:19:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.256+0000 7fad13e00640 1 -- 172.21.15.45:0/140486941 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c0343f0 con 0x7fad140d52d0 2024-04-16T16:19:09.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.257+0000 7fad1ae00640 1 -- 172.21.15.45:0/140486941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad140d52d0 msgr2=0x7fad1406ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:09.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.257+0000 7fad1ae00640 1 --2- 172.21.15.45:0/140486941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad140d52d0 0x7fad1406ce40 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fad0c009e50 tx=0x7fad0c02bee0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.257+0000 7fad1ae00640 1 -- 172.21.15.45:0/140486941 shutdown_connections 2024-04-16T16:19:09.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.257+0000 7fad1ae00640 1 --2- 172.21.15.45:0/140486941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad140d52d0 0x7fad1406ce40 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.257+0000 7fad1ae00640 1 --2- 172.21.15.45:0/140486941 >> v2:172.21.15.103:3300/0 conn(0x7fad14162270 0x7fad140d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.257+0000 7fad1ae00640 1 -- 172.21.15.45:0/140486941 >> 172.21.15.45:0/140486941 conn(0x7fad140d1940 msgr2=0x7fad140d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:09.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.257+0000 7fad1ae00640 1 -- 172.21.15.45:0/140486941 shutdown_connections 2024-04-16T16:19:09.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.257+0000 7fad1ae00640 1 -- 172.21.15.45:0/140486941 wait complete. 2024-04-16T16:19:09.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.260+0000 7fad1ae00640 1 Processor -- start 2024-04-16T16:19:09.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.260+0000 7fad1ae00640 1 -- start start 2024-04-16T16:19:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.261+0000 7fad1ae00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad140d52d0 0x7fad140b8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.262+0000 7fad1ae00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad14161e90 0x7fad141f7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.262+0000 7fad1ae00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad14162270 0x7fad14152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.262+0000 7fad1ae00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fad140d4500 con 0x7fad140d52d0 2024-04-16T16:19:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.262+0000 7fad1ae00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fad140d4380 con 0x7fad14161e90 2024-04-16T16:19:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.262+0000 7fad1ae00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fad140d4680 con 0x7fad14162270 2024-04-16T16:19:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.262+0000 7fad19a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad140d52d0 0x7fad140b8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.262+0000 7fad19a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad140d52d0 0x7fad140b8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44680/0 (socket says 172.21.15.45:44680) 2024-04-16T16:19:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.262+0000 7fad19a00640 1 -- 172.21.15.45:0/3259991260 learned_addr learned my addr 172.21.15.45:0/3259991260 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.262+0000 7fad19000640 1 --2- 172.21.15.45:0/3259991260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad14161e90 0x7fad141f7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.262+0000 7fad19a00640 1 -- 172.21.15.45:0/3259991260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad14162270 msgr2=0x7fad14152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.262+0000 7fad19a00640 1 --2- 172.21.15.45:0/3259991260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad14162270 0x7fad14152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.263+0000 7fad19a00640 1 -- 172.21.15.45:0/3259991260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad14161e90 msgr2=0x7fad141f7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.263+0000 7fad19a00640 1 --2- 172.21.15.45:0/3259991260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad14161e90 0x7fad141f7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.263+0000 7fad19a00640 1 -- 172.21.15.45:0/3259991260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad141f89b0 con 0x7fad140d52d0 2024-04-16T16:19:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.263+0000 7fad19a00640 1 --2- 172.21.15.45:0/3259991260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad140d52d0 0x7fad140b8d00 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fad0800c430 tx=0x7fad0800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.263+0000 7fad11600640 1 -- 172.21.15.45:0/3259991260 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad08011790 con 0x7fad140d52d0 2024-04-16T16:19:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.263+0000 7fad11600640 1 -- 172.21.15.45:0/3259991260 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad08017070 con 0x7fad140d52d0 2024-04-16T16:19:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.263+0000 7fad11600640 1 -- 172.21.15.45:0/3259991260 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad08011bb0 con 0x7fad140d52d0 2024-04-16T16:19:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.263+0000 7fad1ae00640 1 -- 172.21.15.45:0/3259991260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad141f8c90 con 0x7fad140d52d0 2024-04-16T16:19:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.263+0000 7fad1ae00640 1 -- 172.21.15.45:0/3259991260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad14160040 con 0x7fad140d52d0 2024-04-16T16:19:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.265+0000 7fad1ae00640 1 -- 172.21.15.45:0/3259991260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad14075780 con 0x7fad140d52d0 2024-04-16T16:19:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.266+0000 7fad11600640 1 -- 172.21.15.45:0/3259991260 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7fad0801c460 con 0x7fad140d52d0 2024-04-16T16:19:09.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.267+0000 7fad11600640 1 --2- 172.21.15.45:0/3259991260 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7facf406c980 0x7facf406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.267+0000 7fad11600640 1 -- 172.21.15.45:0/3259991260 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5709+0+0 (secure 0 0 0) 0x7fad0808b840 con 0x7fad140d52d0 2024-04-16T16:19:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.267+0000 7fad19000640 1 --2- 172.21.15.45:0/3259991260 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7facf406c980 0x7facf406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.268+0000 7fad19000640 1 --2- 172.21.15.45:0/3259991260 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7facf406c980 0x7facf406ee40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fad00006fd0 tx=0x7fad00008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:09.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.271+0000 7fad11600640 1 -- 172.21.15.45:0/3259991260 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fad08021020 con 0x7fad140d52d0 2024-04-16T16:19:09.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.526+0000 7fad1ae00640 1 -- 172.21.15.45:0/3259991260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad141e74f0 con 0x7fad140d52d0 2024-04-16T16:19:09.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.527+0000 7fad11600640 1 -- 172.21.15.45:0/3259991260 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5186 (secure 0 0 0) 0x7fad08059c80 con 0x7fad140d52d0 2024-04-16T16:19:09.527 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:09.527 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5804,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2862612740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2862612740},{"type":"v1","addr":"172.21.15.103:6836","nonce":2862612740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6304,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2934367463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2934367463},{"type":"v1","addr":"172.21.15.103:6837","nonce":2934367463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1425499906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1425499906},{"type":"v1","addr":"172.21.15.45:6835","nonce":1425499906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:19:05.457389+0000","modified":"2024-04-16T16:19:08.527316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7677},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7677":{"gid":7677,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/882883484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":882883484},{"type":"v1","addr":"172.21.15.45:6834","nonce":882883484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7677,"qdb_cluster":[7677]},"id":7}]} 2024-04-16T16:19:09.528 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 73 2024-04-16T16:19:09.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.530+0000 7fad1ae00640 1 -- 172.21.15.45:0/3259991260 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7facf406c980 msgr2=0x7facf406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:09.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.530+0000 7fad1ae00640 1 --2- 172.21.15.45:0/3259991260 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7facf406c980 0x7facf406ee40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fad00006fd0 tx=0x7fad00008040 comp rx=0 tx=0).stop 2024-04-16T16:19:09.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.530+0000 7fad1ae00640 1 -- 172.21.15.45:0/3259991260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad140d52d0 msgr2=0x7fad140b8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:09.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.530+0000 7fad1ae00640 1 --2- 172.21.15.45:0/3259991260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad140d52d0 0x7fad140b8d00 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fad0800c430 tx=0x7fad0800c8f0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.530+0000 7fad1ae00640 1 -- 172.21.15.45:0/3259991260 shutdown_connections 2024-04-16T16:19:09.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.530+0000 7fad1ae00640 1 --2- 172.21.15.45:0/3259991260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad14162270 0x7fad14152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.530+0000 7fad1ae00640 1 --2- 172.21.15.45:0/3259991260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad14161e90 0x7fad141f7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.530+0000 7fad1ae00640 1 --2- 172.21.15.45:0/3259991260 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7facf406c980 0x7facf406ee40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.530+0000 7fad1ae00640 1 --2- 172.21.15.45:0/3259991260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad140d52d0 0x7fad140b8d00 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.530+0000 7fad1ae00640 1 -- 172.21.15.45:0/3259991260 >> 172.21.15.45:0/3259991260 conn(0x7fad140d1940 msgr2=0x7fad1406ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:09.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.530+0000 7fad1ae00640 1 -- 172.21.15.45:0/3259991260 shutdown_connections 2024-04-16T16:19:09.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.530+0000 7fad1ae00640 1 -- 172.21.15.45:0/3259991260 wait complete. 2024-04-16T16:19:09.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:19:09.720 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:09.719+0000 7fc4e9000640 -1 client.6157: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5636:6fd81961-4957-4e05-85de-333740754282:v1:172.21.15.103:0/2879899995] 2024-04-16T16:19:09.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.772+0000 7f6e87e00640 1 Processor -- start 2024-04-16T16:19:09.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.772+0000 7f6e87e00640 1 -- start start 2024-04-16T16:19:09.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.774+0000 7f6e87e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6e880ccb20 0x7f6e880cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:09.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.774+0000 7f6e87e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6e880ce9a0 0x7f6e880cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.774+0000 7f6e87e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e8815ecd0 con 0x7f6e8815e8f0 2024-04-16T16:19:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.774+0000 7f6e87e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f6e880ce2e0 con 0x7f6e880ccb20 2024-04-16T16:19:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.774+0000 7f6e87e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6e880cd6d0 con 0x7f6e880ce9a0 2024-04-16T16:19:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.774+0000 7f6e86a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6e8815e8f0 0x7f6e880cc410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37880/0 (socket says 172.21.15.45:37880) 2024-04-16T16:19:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.774+0000 7f6e86a00640 1 -- 172.21.15.45:0/905467796 learned_addr learned my addr 172.21.15.45:0/905467796 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.775+0000 7f6e7e000640 1 --2- 172.21.15.45:0/905467796 >> v2:172.21.15.45:3300/0 conn(0x7f6e880ccb20 0x7f6e880cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.775+0000 7f6e87400640 1 --2- 172.21.15.45:0/905467796 >> v2:172.21.15.103:3300/0 conn(0x7f6e880ce9a0 0x7f6e880cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.775+0000 7f6e86000640 1 -- 172.21.15.45:0/905467796 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1006761867 0 0) 0x7f6e8815ecd0 con 0x7f6e8815e8f0 2024-04-16T16:19:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.775+0000 7f6e86000640 1 -- 172.21.15.45:0/905467796 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e6c003520 con 0x7f6e8815e8f0 2024-04-16T16:19:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.775+0000 7f6e7e000640 1 -- 172.21.15.45:0/905467796 >> v2:172.21.15.103:3300/0 conn(0x7f6e880ce9a0 msgr2=0x7f6e880cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.775+0000 7f6e7e000640 1 --2- 172.21.15.45:0/905467796 >> v2:172.21.15.103:3300/0 conn(0x7f6e880ce9a0 0x7f6e880cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.775+0000 7f6e7e000640 1 -- 172.21.15.45:0/905467796 >> v1:172.21.15.45:6789/0 conn(0x7f6e8815e8f0 legacy=0x7f6e880cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.775+0000 7f6e7e000640 1 -- 172.21.15.45:0/905467796 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e880cd9b0 con 0x7f6e880ccb20 2024-04-16T16:19:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.776+0000 7f6e7e000640 1 --2- 172.21.15.45:0/905467796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e880ccb20 0x7f6e880cac30 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f6e8806ff60 tx=0x7f6e7002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4864533443cd649 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.777+0000 7f6e86000640 1 -- 172.21.15.45:0/905467796 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e700315a0 con 0x7f6e880ccb20 2024-04-16T16:19:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.777+0000 7f6e86000640 1 -- 172.21.15.45:0/905467796 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e70037070 con 0x7f6e880ccb20 2024-04-16T16:19:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.777+0000 7f6e86000640 1 -- 172.21.15.45:0/905467796 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e70031960 con 0x7f6e880ccb20 2024-04-16T16:19:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.777+0000 7f6e87e00640 1 -- 172.21.15.45:0/905467796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e880ccb20 msgr2=0x7f6e880cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.777+0000 7f6e87e00640 1 --2- 172.21.15.45:0/905467796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e880ccb20 0x7f6e880cac30 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f6e8806ff60 tx=0x7f6e7002b840 comp rx=0 tx=0).stop 2024-04-16T16:19:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.777+0000 7f6e87e00640 1 -- 172.21.15.45:0/905467796 shutdown_connections 2024-04-16T16:19:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.777+0000 7f6e87e00640 1 --2- 172.21.15.45:0/905467796 >> v2:172.21.15.103:3300/0 conn(0x7f6e880ce9a0 0x7f6e880cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.777+0000 7f6e87e00640 1 --2- 172.21.15.45:0/905467796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e880ccb20 0x7f6e880cac30 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.777+0000 7f6e87e00640 1 -- 172.21.15.45:0/905467796 >> 172.21.15.45:0/905467796 conn(0x7f6e8806b930 msgr2=0x7f6e8806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.778+0000 7f6e87e00640 1 -- 172.21.15.45:0/905467796 shutdown_connections 2024-04-16T16:19:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.778+0000 7f6e87e00640 1 -- 172.21.15.45:0/905467796 wait complete. 2024-04-16T16:19:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.779+0000 7f6e87e00640 1 Processor -- start 2024-04-16T16:19:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.779+0000 7f6e87e00640 1 -- start start 2024-04-16T16:19:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.780+0000 7f6e87e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e880ccb20 0x7f6e8813f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.780+0000 7f6e87e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e880ce9a0 0x7f6e88151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.780+0000 7f6e87e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e8815e8f0 0x7f6e88162160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.780+0000 7f6e87e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6e880cd360 con 0x7f6e8815e8f0 2024-04-16T16:19:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.780+0000 7f6e87e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6e880cdf70 con 0x7f6e880ce9a0 2024-04-16T16:19:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.780+0000 7f6e87e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6e880cf080 con 0x7f6e880ccb20 2024-04-16T16:19:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.781+0000 7f6e87400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e8815e8f0 0x7f6e88162160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.781+0000 7f6e87400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e8815e8f0 0x7f6e88162160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44698/0 (socket says 172.21.15.45:44698) 2024-04-16T16:19:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.781+0000 7f6e87400640 1 -- 172.21.15.45:0/3150338437 learned_addr learned my addr 172.21.15.45:0/3150338437 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.781+0000 7f6e86a00640 1 --2- 172.21.15.45:0/3150338437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e880ccb20 0x7f6e8813f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.781+0000 7f6e7e000640 1 --2- 172.21.15.45:0/3150338437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e880ce9a0 0x7f6e88151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.781+0000 7f6e87400640 1 -- 172.21.15.45:0/3150338437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e880ccb20 msgr2=0x7f6e8813f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.781+0000 7f6e87400640 1 --2- 172.21.15.45:0/3150338437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e880ccb20 0x7f6e8813f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.781+0000 7f6e87400640 1 -- 172.21.15.45:0/3150338437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e880ce9a0 msgr2=0x7f6e88151080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.781+0000 7f6e87400640 1 --2- 172.21.15.45:0/3150338437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e880ce9a0 0x7f6e88151080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.781+0000 7f6e87400640 1 -- 172.21.15.45:0/3150338437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e88152df0 con 0x7f6e8815e8f0 2024-04-16T16:19:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.781+0000 7f6e7e000640 1 --2- 172.21.15.45:0/3150338437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e880ce9a0 0x7f6e88151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.782+0000 7f6e87400640 1 --2- 172.21.15.45:0/3150338437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e8815e8f0 0x7f6e88162160 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f6e80007e40 tx=0x7f6e8000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.782+0000 7f6e7f400640 1 -- 172.21.15.45:0/3150338437 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e80014620 con 0x7f6e8815e8f0 2024-04-16T16:19:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.782+0000 7f6e7f400640 1 -- 172.21.15.45:0/3150338437 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e80019070 con 0x7f6e8815e8f0 2024-04-16T16:19:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.782+0000 7f6e7f400640 1 -- 172.21.15.45:0/3150338437 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e80014a20 con 0x7f6e8815e8f0 2024-04-16T16:19:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.782+0000 7f6e87e00640 1 -- 172.21.15.45:0/3150338437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e881530d0 con 0x7f6e8815e8f0 2024-04-16T16:19:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.782+0000 7f6e87e00640 1 -- 172.21.15.45:0/3150338437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e88153690 con 0x7f6e8815e8f0 2024-04-16T16:19:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.783+0000 7f6e87e00640 1 -- 172.21.15.45:0/3150338437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e88075780 con 0x7f6e8815e8f0 2024-04-16T16:19:09.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.784+0000 7f6e7f400640 1 -- 172.21.15.45:0/3150338437 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f6e80014bc0 con 0x7f6e8815e8f0 2024-04-16T16:19:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.786+0000 7f6e7f400640 1 --2- 172.21.15.45:0/3150338437 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6e6406c980 0x7f6e6406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.786+0000 7f6e7f400640 1 -- 172.21.15.45:0/3150338437 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5709+0+0 (secure 0 0 0) 0x7f6e8008c5b0 con 0x7f6e8815e8f0 2024-04-16T16:19:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.787+0000 7f6e86a00640 1 --2- 172.21.15.45:0/3150338437 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6e6406c980 0x7f6e6406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.788+0000 7f6e86a00640 1 --2- 172.21.15.45:0/3150338437 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6e6406c980 0x7f6e6406ee40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6e78005880 tx=0x7f6e78009040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:09.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:09.791+0000 7f6e7f400640 1 -- 172.21.15.45:0/3150338437 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6e8005a8f0 con 0x7f6e8815e8f0 2024-04-16T16:19:10.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.000+0000 7f6e87e00640 1 -- 172.21.15.45:0/3150338437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e880b5b90 con 0x7f6e8815e8f0 2024-04-16T16:19:10.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.003+0000 7f6e7f400640 1 -- 172.21.15.45:0/3150338437 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+16608 (secure 0 0 0) 0x7f6e8005a1f0 con 0x7f6e8815e8f0 2024-04-16T16:19:10.004 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:10.004 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":99,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:19:06.966205+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:03.429683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:04.434699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:19:10.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.006+0000 7f6e87e00640 1 -- 172.21.15.45:0/3150338437 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6e6406c980 msgr2=0x7f6e6406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:10.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.006+0000 7f6e87e00640 1 --2- 172.21.15.45:0/3150338437 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6e6406c980 0x7f6e6406ee40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6e78005880 tx=0x7f6e78009040 comp rx=0 tx=0).stop 2024-04-16T16:19:10.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.006+0000 7f6e87e00640 1 -- 172.21.15.45:0/3150338437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e8815e8f0 msgr2=0x7f6e88162160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:10.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.006+0000 7f6e87e00640 1 --2- 172.21.15.45:0/3150338437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e8815e8f0 0x7f6e88162160 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f6e80007e40 tx=0x7f6e8000e4b0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.006+0000 7f6e87e00640 1 -- 172.21.15.45:0/3150338437 shutdown_connections 2024-04-16T16:19:10.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.006+0000 7f6e87e00640 1 --2- 172.21.15.45:0/3150338437 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f6e6406c980 0x7f6e6406ee40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.006+0000 7f6e87e00640 1 --2- 172.21.15.45:0/3150338437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e8815e8f0 0x7f6e88162160 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.006+0000 7f6e87e00640 1 --2- 172.21.15.45:0/3150338437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e880ce9a0 0x7f6e88151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.006+0000 7f6e87e00640 1 --2- 172.21.15.45:0/3150338437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e880ccb20 0x7f6e8813f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.006+0000 7f6e87e00640 1 -- 172.21.15.45:0/3150338437 >> 172.21.15.45:0/3150338437 conn(0x7f6e8806b930 msgr2=0x7f6e880cabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:10.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.007+0000 7f6e87e00640 1 -- 172.21.15.45:0/3150338437 shutdown_connections 2024-04-16T16:19:10.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.007+0000 7f6e87e00640 1 -- 172.21.15.45:0/3150338437 wait complete. 2024-04-16T16:19:10.032 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-16T16:19:10.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:19:10.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.244+0000 7fdc37e00640 1 Processor -- start 2024-04-16T16:19:10.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.244+0000 7fdc37e00640 1 -- start start 2024-04-16T16:19:10.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.246+0000 7fdc37e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdc381608a0 0x7fdc38160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.246+0000 7fdc37e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc3815d360 con 0x7fdc38146220 2024-04-16T16:19:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.246+0000 7fdc37e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc3815d640 con 0x7fdc3815ccb0 2024-04-16T16:19:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.246+0000 7fdc37e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fdc3815d130 con 0x7fdc381608a0 2024-04-16T16:19:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.246+0000 7fdc36a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdc38146220 0x7fdc3806f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37886/0 (socket says 172.21.15.45:37886) 2024-04-16T16:19:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.246+0000 7fdc36a00640 1 -- 172.21.15.45:0/920974771 learned_addr learned my addr 172.21.15.45:0/920974771 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.247+0000 7fdc37400640 1 --2- 172.21.15.45:0/920974771 >> v2:172.21.15.103:3300/0 conn(0x7fdc381608a0 0x7fdc38160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.247+0000 7fdc35600640 1 -- 172.21.15.45:0/920974771 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3304895682 0 0) 0x7fdc3815d360 con 0x7fdc38146220 2024-04-16T16:19:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.247+0000 7fdc35600640 1 -- 172.21.15.45:0/920974771 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc1c003520 con 0x7fdc38146220 2024-04-16T16:19:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.247+0000 7fdc35600640 1 -- 172.21.15.45:0/920974771 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 948969421 0 0) 0x7fdc1c003520 con 0x7fdc38146220 2024-04-16T16:19:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.247+0000 7fdc35600640 1 -- 172.21.15.45:0/920974771 >> v2:172.21.15.103:3300/0 conn(0x7fdc381608a0 msgr2=0x7fdc38160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:10.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.247+0000 7fdc35600640 1 --2- 172.21.15.45:0/920974771 >> v2:172.21.15.103:3300/0 conn(0x7fdc381608a0 0x7fdc38160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.247+0000 7fdc35600640 1 -- 172.21.15.45:0/920974771 >> v1:172.21.15.103:6789/0 conn(0x7fdc3815ccb0 legacy=0x7fdc3806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:10.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.247+0000 7fdc35600640 1 -- 172.21.15.45:0/920974771 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3815d920 con 0x7fdc38146220 2024-04-16T16:19:10.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.247+0000 7fdc35600640 1 -- 172.21.15.45:0/920974771 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fdc280019f0 con 0x7fdc38146220 2024-04-16T16:19:10.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.247+0000 7fdc35600640 1 -- 172.21.15.45:0/920974771 >> v1:172.21.15.45:6789/0 conn(0x7fdc38146220 legacy=0x7fdc3806f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:10.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.247+0000 7fdc35600640 1 --2- 172.21.15.45:0/920974771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc1c003a00 0x7fdc1c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:10.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.247+0000 7fdc35600640 1 --2- 172.21.15.45:0/920974771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc1c004800 0x7fdc1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:10.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.248+0000 7fdc35600640 1 --2- 172.21.15.45:0/920974771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc1c005210 0x7fdc1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:10.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.248+0000 7fdc35600640 1 -- 172.21.15.45:0/920974771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c00c110 con 0x7fdc1c004800 2024-04-16T16:19:10.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.248+0000 7fdc35600640 1 -- 172.21.15.45:0/920974771 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c00c2c0 con 0x7fdc1c005210 2024-04-16T16:19:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.248+0000 7fdc35600640 1 -- 172.21.15.45:0/920974771 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdc1c00c4c0 con 0x7fdc1c003a00 2024-04-16T16:19:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.248+0000 7fdc37400640 1 --2- 172.21.15.45:0/920974771 >> v2:172.21.15.103:3300/0 conn(0x7fdc381608a0 0x7fdc38160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:19:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.248+0000 7fdc36000640 1 --2- 172.21.15.45:0/920974771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc1c004800 0x7fdc1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.248+0000 7fdc36a00640 1 --2- 172.21.15.45:0/920974771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc1c003a00 0x7fdc1c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.248+0000 7fdc36000640 1 -- 172.21.15.45:0/920974771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc1c003a00 msgr2=0x7fdc1c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.248+0000 7fdc36000640 1 --2- 172.21.15.45:0/920974771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc1c003a00 0x7fdc1c003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.248+0000 7fdc37400640 1 --2- 172.21.15.45:0/920974771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc1c005210 0x7fdc1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.248+0000 7fdc36000640 1 -- 172.21.15.45:0/920974771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc1c005210 msgr2=0x7fdc1c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.248+0000 7fdc36000640 1 --2- 172.21.15.45:0/920974771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc1c005210 0x7fdc1c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.248+0000 7fdc36000640 1 -- 172.21.15.45:0/920974771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c003520 con 0x7fdc1c004800 2024-04-16T16:19:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.248+0000 7fdc37400640 1 -- 172.21.15.45:0/920974771 reap_dead start 2024-04-16T16:19:10.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.249+0000 7fdc36000640 1 --2- 172.21.15.45:0/920974771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc1c004800 0x7fdc1c004c80 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fdc20003d80 tx=0x7fdc1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9546fae2f8e91348 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:10.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.250+0000 7fdc35600640 1 -- 172.21.15.45:0/920974771 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20037070 con 0x7fdc1c004800 2024-04-16T16:19:10.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.250+0000 7fdc35600640 1 -- 172.21.15.45:0/920974771 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc20032600 con 0x7fdc1c004800 2024-04-16T16:19:10.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.250+0000 7fdc35600640 1 -- 172.21.15.45:0/920974771 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20003980 con 0x7fdc1c004800 2024-04-16T16:19:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.250+0000 7fdc37e00640 1 -- 172.21.15.45:0/920974771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc1c004800 msgr2=0x7fdc1c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.250+0000 7fdc37e00640 1 --2- 172.21.15.45:0/920974771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc1c004800 0x7fdc1c004c80 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fdc20003d80 tx=0x7fdc1c005830 comp rx=0 tx=0).stop 2024-04-16T16:19:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.250+0000 7fdc37e00640 1 -- 172.21.15.45:0/920974771 shutdown_connections 2024-04-16T16:19:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.250+0000 7fdc37e00640 1 --2- 172.21.15.45:0/920974771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc1c004800 0x7fdc1c004c80 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.250+0000 7fdc37e00640 1 -- 172.21.15.45:0/920974771 >> 172.21.15.45:0/920974771 conn(0x7fdc3806b930 msgr2=0x7fdc3806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.251+0000 7fdc37e00640 1 -- 172.21.15.45:0/920974771 shutdown_connections 2024-04-16T16:19:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.251+0000 7fdc37e00640 1 -- 172.21.15.45:0/920974771 wait complete. 2024-04-16T16:19:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.252+0000 7fdc37e00640 1 Processor -- start 2024-04-16T16:19:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.252+0000 7fdc37e00640 1 -- start start 2024-04-16T16:19:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.253+0000 7fdc37e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc1c004800 0x7fdc3815c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:10.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.253+0000 7fdc37e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc380aeab0 0x7fdc380b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:10.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.253+0000 7fdc37e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc380aef30 0x7fdc380b1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:10.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.253+0000 7fdc37e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdc38160cf0 con 0x7fdc380aeab0 2024-04-16T16:19:10.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.253+0000 7fdc37e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdc38160410 con 0x7fdc380aef30 2024-04-16T16:19:10.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.253+0000 7fdc37e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdc38160e70 con 0x7fdc1c004800 2024-04-16T16:19:10.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.254+0000 7fdc36000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc380aeab0 0x7fdc380b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:10.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.254+0000 7fdc36000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc380aeab0 0x7fdc380b0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44718/0 (socket says 172.21.15.45:44718) 2024-04-16T16:19:10.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.254+0000 7fdc36000640 1 -- 172.21.15.45:0/517268928 learned_addr learned my addr 172.21.15.45:0/517268928 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:10.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.254+0000 7fdc36a00640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc1c004800 0x7fdc3815c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:10.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.254+0000 7fdc37400640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc380aef30 0x7fdc380b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:10.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.254+0000 7fdc36000640 1 -- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc1c004800 msgr2=0x7fdc3815c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:10.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.254+0000 7fdc36000640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc1c004800 0x7fdc3815c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.254+0000 7fdc36000640 1 -- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc380aef30 msgr2=0x7fdc380b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:10.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.254+0000 7fdc36000640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc380aef30 0x7fdc380b1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.254+0000 7fdc36000640 1 -- 172.21.15.45:0/517268928 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc380ada10 con 0x7fdc380aeab0 2024-04-16T16:19:10.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.254+0000 7fdc37400640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc380aef30 0x7fdc380b1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:10.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.254+0000 7fdc36a00640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc1c004800 0x7fdc3815c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:10.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.255+0000 7fdc36000640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc380aeab0 0x7fdc380b0e00 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fdc2002bbc0 tx=0x7fdc200036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:10.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.255+0000 7fdc2ea00640 1 -- 172.21.15.45:0/517268928 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20042030 con 0x7fdc380aeab0 2024-04-16T16:19:10.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.255+0000 7fdc2ea00640 1 -- 172.21.15.45:0/517268928 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc20032400 con 0x7fdc380aeab0 2024-04-16T16:19:10.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.255+0000 7fdc2ea00640 1 -- 172.21.15.45:0/517268928 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20037040 con 0x7fdc380aeab0 2024-04-16T16:19:10.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.255+0000 7fdc37e00640 1 -- 172.21.15.45:0/517268928 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc380adcf0 con 0x7fdc380aeab0 2024-04-16T16:19:10.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.255+0000 7fdc37e00640 1 -- 172.21.15.45:0/517268928 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc380ae2b0 con 0x7fdc380aeab0 2024-04-16T16:19:10.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.256+0000 7fdc37e00640 1 -- 172.21.15.45:0/517268928 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc38075780 con 0x7fdc380aeab0 2024-04-16T16:19:10.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.257+0000 7fdc2ea00640 1 -- 172.21.15.45:0/517268928 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7fdc20045070 con 0x7fdc380aeab0 2024-04-16T16:19:10.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.258+0000 7fdc2ea00640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fdc1406c980 0x7fdc1406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:10.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.258+0000 7fdc2ea00640 1 -- 172.21.15.45:0/517268928 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5752+0+0 (secure 0 0 0) 0x7fdc200ad6c0 con 0x7fdc380aeab0 2024-04-16T16:19:10.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.259+0000 7fdc36a00640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fdc1406c980 0x7fdc1406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:10.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.260+0000 7fdc36a00640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fdc1406c980 0x7fdc1406ee40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800e280 tx=0x7fdc28009040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:10.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.263+0000 7fdc2ea00640 1 -- 172.21.15.45:0/517268928 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdc2007ba50 con 0x7fdc380aeab0 2024-04-16T16:19:10.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.461+0000 7fdc37e00640 1 -- 172.21.15.45:0/517268928 --> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdc380cff60 con 0x7fdc1406c980 2024-04-16T16:19:10.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.484+0000 7fdc2ea00640 1 -- 172.21.15.45:0/517268928 <== mgr.6115 v2:172.21.15.103:6832/839990480 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252151 (secure 0 0 0) 0x7fdc380cff60 con 0x7fdc1406c980 2024-04-16T16:19:10.489 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:10.491 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:19:10.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.491+0000 7fdc37e00640 1 -- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fdc1406c980 msgr2=0x7fdc1406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:10.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.491+0000 7fdc37e00640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fdc1406c980 0x7fdc1406ee40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800e280 tx=0x7fdc28009040 comp rx=0 tx=0).stop 2024-04-16T16:19:10.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.491+0000 7fdc37e00640 1 -- 172.21.15.45:0/517268928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc380aeab0 msgr2=0x7fdc380b0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:10.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.491+0000 7fdc37e00640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc380aeab0 0x7fdc380b0e00 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fdc2002bbc0 tx=0x7fdc200036f0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.491+0000 7fdc37e00640 1 -- 172.21.15.45:0/517268928 shutdown_connections 2024-04-16T16:19:10.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.491+0000 7fdc37e00640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc380aef30 0x7fdc380b1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.491+0000 7fdc37e00640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fdc1406c980 0x7fdc1406ee40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.491+0000 7fdc37e00640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc380aeab0 0x7fdc380b0e00 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.491+0000 7fdc37e00640 1 --2- 172.21.15.45:0/517268928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc1c004800 0x7fdc3815c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.491+0000 7fdc37e00640 1 -- 172.21.15.45:0/517268928 >> 172.21.15.45:0/517268928 conn(0x7fdc3806b930 msgr2=0x7fdc380b37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:10.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.491+0000 7fdc37e00640 1 -- 172.21.15.45:0/517268928 shutdown_connections 2024-04-16T16:19:10.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.491+0000 7fdc37e00640 1 -- 172.21.15.45:0/517268928 wait complete. 2024-04-16T16:19:10.512 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-04-16T16:19:10.040078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":551,"num_read_kb":972,"num_write":288,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2608,"ondisk_log_size":2608,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":191,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264428,"kb_used_data":10792,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754710292,"statfs":{"total":773094113280,"available":772823339008,"internally_reserved":0,"allocated":11051008,"data_stored":5812860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":47,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.083436"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012374+0000","last_change":"2024-04-16T16:19:05.980861+0000","last_active":"2024-04-16T16:19:10.012374+0000","last_peered":"2024-04-16T16:19:10.012374+0000","last_clean":"2024-04-16T16:19:10.012374+0000","last_became_active":"2024-04-16T16:19:05.980280+0000","last_became_peered":"2024-04-16T16:19:05.980280+0000","last_unstale":"2024-04-16T16:19:10.012374+0000","last_undegraded":"2024-04-16T16:19:10.012374+0000","last_fullsized":"2024-04-16T16:19:10.012374+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:54:42.032276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985931+0000","last_change":"2024-04-16T16:19:06.004931+0000","last_active":"2024-04-16T16:19:06.985931+0000","last_peered":"2024-04-16T16:19:06.985931+0000","last_clean":"2024-04-16T16:19:06.985931+0000","last_became_active":"2024-04-16T16:19:05.999878+0000","last_became_peered":"2024-04-16T16:19:05.999878+0000","last_unstale":"2024-04-16T16:19:06.985931+0000","last_undegraded":"2024-04-16T16:19:06.985931+0000","last_fullsized":"2024-04-16T16:19:06.985931+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:17.234151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012237+0000","last_change":"2024-04-16T16:19:05.993873+0000","last_active":"2024-04-16T16:19:10.012237+0000","last_peered":"2024-04-16T16:19:10.012237+0000","last_clean":"2024-04-16T16:19:10.012237+0000","last_became_active":"2024-04-16T16:19:05.993512+0000","last_became_peered":"2024-04-16T16:19:05.993512+0000","last_unstale":"2024-04-16T16:19:10.012237+0000","last_undegraded":"2024-04-16T16:19:10.012237+0000","last_fullsized":"2024-04-16T16:19:10.012237+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:19.148184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.978542+0000","last_change":"2024-04-16T16:19:05.995781+0000","last_active":"2024-04-16T16:19:06.978542+0000","last_peered":"2024-04-16T16:19:06.978542+0000","last_clean":"2024-04-16T16:19:06.978542+0000","last_became_active":"2024-04-16T16:19:05.995469+0000","last_became_peered":"2024-04-16T16:19:05.995469+0000","last_unstale":"2024-04-16T16:19:06.978542+0000","last_undegraded":"2024-04-16T16:19:06.978542+0000","last_fullsized":"2024-04-16T16:19:06.978542+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:23:31.314921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.188821+0000","last_change":"2024-04-16T16:19:05.986907+0000","last_active":"2024-04-16T16:19:07.188821+0000","last_peered":"2024-04-16T16:19:07.188821+0000","last_clean":"2024-04-16T16:19:07.188821+0000","last_became_active":"2024-04-16T16:19:05.986607+0000","last_became_peered":"2024-04-16T16:19:05.986607+0000","last_unstale":"2024-04-16T16:19:07.188821+0000","last_undegraded":"2024-04-16T16:19:07.188821+0000","last_fullsized":"2024-04-16T16:19:07.188821+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:45:16.357214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.014299+0000","last_change":"2024-04-16T16:19:05.979722+0000","last_active":"2024-04-16T16:19:10.014299+0000","last_peered":"2024-04-16T16:19:10.014299+0000","last_clean":"2024-04-16T16:19:10.014299+0000","last_became_active":"2024-04-16T16:19:05.979418+0000","last_became_peered":"2024-04-16T16:19:05.979418+0000","last_unstale":"2024-04-16T16:19:10.014299+0000","last_undegraded":"2024-04-16T16:19:10.014299+0000","last_fullsized":"2024-04-16T16:19:10.014299+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:02.954116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.013948+0000","last_change":"2024-04-16T16:19:05.979260+0000","last_active":"2024-04-16T16:19:10.013948+0000","last_peered":"2024-04-16T16:19:10.013948+0000","last_clean":"2024-04-16T16:19:10.013948+0000","last_became_active":"2024-04-16T16:19:05.978961+0000","last_became_peered":"2024-04-16T16:19:05.978961+0000","last_unstale":"2024-04-16T16:19:10.013948+0000","last_undegraded":"2024-04-16T16:19:10.013948+0000","last_fullsized":"2024-04-16T16:19:10.013948+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:13.871924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985611+0000","last_change":"2024-04-16T16:19:05.981860+0000","last_active":"2024-04-16T16:19:06.985611+0000","last_peered":"2024-04-16T16:19:06.985611+0000","last_clean":"2024-04-16T16:19:06.985611+0000","last_became_active":"2024-04-16T16:19:05.981428+0000","last_became_peered":"2024-04-16T16:19:05.981428+0000","last_unstale":"2024-04-16T16:19:06.985611+0000","last_undegraded":"2024-04-16T16:19:06.985611+0000","last_fullsized":"2024-04-16T16:19:06.985611+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:32:02.583804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986006+0000","last_change":"2024-04-16T16:19:05.987627+0000","last_active":"2024-04-16T16:19:06.986006+0000","last_peered":"2024-04-16T16:19:06.986006+0000","last_clean":"2024-04-16T16:19:06.986006+0000","last_became_active":"2024-04-16T16:19:05.987145+0000","last_became_peered":"2024-04-16T16:19:05.987145+0000","last_unstale":"2024-04-16T16:19:06.986006+0000","last_undegraded":"2024-04-16T16:19:06.986006+0000","last_fullsized":"2024-04-16T16:19:06.986006+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:39:55.163258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.190391+0000","last_change":"2024-04-16T16:19:05.979043+0000","last_active":"2024-04-16T16:19:07.190391+0000","last_peered":"2024-04-16T16:19:07.190391+0000","last_clean":"2024-04-16T16:19:07.190391+0000","last_became_active":"2024-04-16T16:19:05.978722+0000","last_became_peered":"2024-04-16T16:19:05.978722+0000","last_unstale":"2024-04-16T16:19:07.190391+0000","last_undegraded":"2024-04-16T16:19:07.190391+0000","last_fullsized":"2024-04-16T16:19:07.190391+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:03.968225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.014757+0000","last_change":"2024-04-16T16:19:05.976920+0000","last_active":"2024-04-16T16:19:10.014757+0000","last_peered":"2024-04-16T16:19:10.014757+0000","last_clean":"2024-04-16T16:19:10.014757+0000","last_became_active":"2024-04-16T16:19:05.976381+0000","last_became_peered":"2024-04-16T16:19:05.976381+0000","last_unstale":"2024-04-16T16:19:10.014757+0000","last_undegraded":"2024-04-16T16:19:10.014757+0000","last_fullsized":"2024-04-16T16:19:10.014757+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:03.968225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.984876+0000","last_change":"2024-04-16T16:19:05.977966+0000","last_active":"2024-04-16T16:19:06.984876+0000","last_peered":"2024-04-16T16:19:06.984876+0000","last_clean":"2024-04-16T16:19:06.984876+0000","last_became_active":"2024-04-16T16:19:05.975530+0000","last_became_peered":"2024-04-16T16:19:05.975530+0000","last_unstale":"2024-04-16T16:19:06.984876+0000","last_undegraded":"2024-04-16T16:19:06.984876+0000","last_fullsized":"2024-04-16T16:19:06.984876+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:59.489811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.978144+0000","last_change":"2024-04-16T16:19:05.978675+0000","last_active":"2024-04-16T16:19:06.978144+0000","last_peered":"2024-04-16T16:19:06.978144+0000","last_clean":"2024-04-16T16:19:06.978144+0000","last_became_active":"2024-04-16T16:19:05.976724+0000","last_became_peered":"2024-04-16T16:19:05.976724+0000","last_unstale":"2024-04-16T16:19:06.978144+0000","last_undegraded":"2024-04-16T16:19:06.978144+0000","last_fullsized":"2024-04-16T16:19:06.978144+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:48.694556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010032+0000","last_change":"2024-04-16T16:19:05.991399+0000","last_active":"2024-04-16T16:19:10.010032+0000","last_peered":"2024-04-16T16:19:10.010032+0000","last_clean":"2024-04-16T16:19:10.010032+0000","last_became_active":"2024-04-16T16:19:05.991105+0000","last_became_peered":"2024-04-16T16:19:05.991105+0000","last_unstale":"2024-04-16T16:19:10.010032+0000","last_undegraded":"2024-04-16T16:19:10.010032+0000","last_fullsized":"2024-04-16T16:19:10.010032+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:59:24.858976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.977751+0000","last_change":"2024-04-16T16:19:05.973800+0000","last_active":"2024-04-16T16:19:06.977751+0000","last_peered":"2024-04-16T16:19:06.977751+0000","last_clean":"2024-04-16T16:19:06.977751+0000","last_became_active":"2024-04-16T16:19:05.972807+0000","last_became_peered":"2024-04-16T16:19:05.972807+0000","last_unstale":"2024-04-16T16:19:06.977751+0000","last_undegraded":"2024-04-16T16:19:06.977751+0000","last_fullsized":"2024-04-16T16:19:06.977751+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:57:21.364765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.013948+0000","last_change":"2024-04-16T16:19:05.972005+0000","last_active":"2024-04-16T16:19:10.013948+0000","last_peered":"2024-04-16T16:19:10.013948+0000","last_clean":"2024-04-16T16:19:10.013948+0000","last_became_active":"2024-04-16T16:19:05.971388+0000","last_became_peered":"2024-04-16T16:19:05.971388+0000","last_unstale":"2024-04-16T16:19:10.013948+0000","last_undegraded":"2024-04-16T16:19:10.013948+0000","last_fullsized":"2024-04-16T16:19:10.013948+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:32:08.093006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986319+0000","last_change":"2024-04-16T16:19:06.001464+0000","last_active":"2024-04-16T16:19:06.986319+0000","last_peered":"2024-04-16T16:19:06.986319+0000","last_clean":"2024-04-16T16:19:06.986319+0000","last_became_active":"2024-04-16T16:19:05.998584+0000","last_became_peered":"2024-04-16T16:19:05.998584+0000","last_unstale":"2024-04-16T16:19:06.986319+0000","last_undegraded":"2024-04-16T16:19:06.986319+0000","last_fullsized":"2024-04-16T16:19:06.986319+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:21:27.367871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.978468+0000","last_change":"2024-04-16T16:19:05.995193+0000","last_active":"2024-04-16T16:19:06.978468+0000","last_peered":"2024-04-16T16:19:06.978468+0000","last_clean":"2024-04-16T16:19:06.978468+0000","last_became_active":"2024-04-16T16:19:05.994843+0000","last_became_peered":"2024-04-16T16:19:05.994843+0000","last_unstale":"2024-04-16T16:19:06.978468+0000","last_undegraded":"2024-04-16T16:19:06.978468+0000","last_fullsized":"2024-04-16T16:19:06.978468+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:17.234151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010279+0000","last_change":"2024-04-16T16:19:05.990057+0000","last_active":"2024-04-16T16:19:10.010279+0000","last_peered":"2024-04-16T16:19:10.010279+0000","last_clean":"2024-04-16T16:19:10.010279+0000","last_became_active":"2024-04-16T16:19:05.989306+0000","last_became_peered":"2024-04-16T16:19:05.989306+0000","last_unstale":"2024-04-16T16:19:10.010279+0000","last_undegraded":"2024-04-16T16:19:10.010279+0000","last_fullsized":"2024-04-16T16:19:10.010279+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:15:09.100928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985000+0000","last_change":"2024-04-16T16:19:05.977562+0000","last_active":"2024-04-16T16:19:06.985000+0000","last_peered":"2024-04-16T16:19:06.985000+0000","last_clean":"2024-04-16T16:19:06.985000+0000","last_became_active":"2024-04-16T16:19:05.976423+0000","last_became_peered":"2024-04-16T16:19:05.976423+0000","last_unstale":"2024-04-16T16:19:06.985000+0000","last_undegraded":"2024-04-16T16:19:06.985000+0000","last_fullsized":"2024-04-16T16:19:06.985000+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:16:03.019315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010120+0000","last_change":"2024-04-16T16:19:05.988445+0000","last_active":"2024-04-16T16:19:10.010120+0000","last_peered":"2024-04-16T16:19:10.010120+0000","last_clean":"2024-04-16T16:19:10.010120+0000","last_became_active":"2024-04-16T16:19:05.988154+0000","last_became_peered":"2024-04-16T16:19:05.988154+0000","last_unstale":"2024-04-16T16:19:10.010120+0000","last_undegraded":"2024-04-16T16:19:10.010120+0000","last_fullsized":"2024-04-16T16:19:10.010120+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:01:13.476876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985634+0000","last_change":"2024-04-16T16:19:05.982959+0000","last_active":"2024-04-16T16:19:06.985634+0000","last_peered":"2024-04-16T16:19:06.985634+0000","last_clean":"2024-04-16T16:19:06.985634+0000","last_became_active":"2024-04-16T16:19:05.980754+0000","last_became_peered":"2024-04-16T16:19:05.980754+0000","last_unstale":"2024-04-16T16:19:06.985634+0000","last_undegraded":"2024-04-16T16:19:06.985634+0000","last_fullsized":"2024-04-16T16:19:06.985634+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:48.694556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986235+0000","last_change":"2024-04-16T16:19:05.982625+0000","last_active":"2024-04-16T16:19:06.986235+0000","last_peered":"2024-04-16T16:19:06.986235+0000","last_clean":"2024-04-16T16:19:06.986235+0000","last_became_active":"2024-04-16T16:19:05.982182+0000","last_became_peered":"2024-04-16T16:19:05.982182+0000","last_unstale":"2024-04-16T16:19:06.986235+0000","last_undegraded":"2024-04-16T16:19:06.986235+0000","last_fullsized":"2024-04-16T16:19:06.986235+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:07:56.982348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.014025+0000","last_change":"2024-04-16T16:19:05.969908+0000","last_active":"2024-04-16T16:19:10.014025+0000","last_peered":"2024-04-16T16:19:10.014025+0000","last_clean":"2024-04-16T16:19:10.014025+0000","last_became_active":"2024-04-16T16:19:05.969572+0000","last_became_peered":"2024-04-16T16:19:05.969572+0000","last_unstale":"2024-04-16T16:19:10.014025+0000","last_undegraded":"2024-04-16T16:19:10.014025+0000","last_fullsized":"2024-04-16T16:19:10.014025+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:33:25.856909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.190682+0000","last_change":"2024-04-16T16:19:06.008841+0000","last_active":"2024-04-16T16:19:07.190682+0000","last_peered":"2024-04-16T16:19:07.190682+0000","last_clean":"2024-04-16T16:19:07.190682+0000","last_became_active":"2024-04-16T16:19:06.008601+0000","last_became_peered":"2024-04-16T16:19:06.008601+0000","last_unstale":"2024-04-16T16:19:07.190682+0000","last_undegraded":"2024-04-16T16:19:07.190682+0000","last_fullsized":"2024-04-16T16:19:07.190682+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:34:58.912595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985960+0000","last_change":"2024-04-16T16:19:05.997077+0000","last_active":"2024-04-16T16:19:06.985960+0000","last_peered":"2024-04-16T16:19:06.985960+0000","last_clean":"2024-04-16T16:19:06.985960+0000","last_became_active":"2024-04-16T16:19:05.996308+0000","last_became_peered":"2024-04-16T16:19:05.996308+0000","last_unstale":"2024-04-16T16:19:06.985960+0000","last_undegraded":"2024-04-16T16:19:06.985960+0000","last_fullsized":"2024-04-16T16:19:06.985960+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:23:31.314921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.014897+0000","last_change":"2024-04-16T16:19:05.989978+0000","last_active":"2024-04-16T16:19:10.014897+0000","last_peered":"2024-04-16T16:19:10.014897+0000","last_clean":"2024-04-16T16:19:10.014897+0000","last_became_active":"2024-04-16T16:19:05.989686+0000","last_became_peered":"2024-04-16T16:19:05.989686+0000","last_unstale":"2024-04-16T16:19:10.014897+0000","last_undegraded":"2024-04-16T16:19:10.014897+0000","last_fullsized":"2024-04-16T16:19:10.014897+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:21:32.529352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.188994+0000","last_change":"2024-04-16T16:19:05.984266+0000","last_active":"2024-04-16T16:19:07.188994+0000","last_peered":"2024-04-16T16:19:07.188994+0000","last_clean":"2024-04-16T16:19:07.188994+0000","last_became_active":"2024-04-16T16:19:05.983968+0000","last_became_peered":"2024-04-16T16:19:05.983968+0000","last_unstale":"2024-04-16T16:19:07.188994+0000","last_undegraded":"2024-04-16T16:19:07.188994+0000","last_fullsized":"2024-04-16T16:19:07.188994+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:13.871924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985985+0000","last_change":"2024-04-16T16:19:05.975292+0000","last_active":"2024-04-16T16:19:06.985985+0000","last_peered":"2024-04-16T16:19:06.985985+0000","last_clean":"2024-04-16T16:19:06.985985+0000","last_became_active":"2024-04-16T16:19:05.975013+0000","last_became_peered":"2024-04-16T16:19:05.975013+0000","last_unstale":"2024-04-16T16:19:06.985985+0000","last_undegraded":"2024-04-16T16:19:06.985985+0000","last_fullsized":"2024-04-16T16:19:06.985985+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:22:35.948275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010180+0000","last_change":"2024-04-16T16:19:05.991370+0000","last_active":"2024-04-16T16:19:10.010180+0000","last_peered":"2024-04-16T16:19:10.010180+0000","last_clean":"2024-04-16T16:19:10.010180+0000","last_became_active":"2024-04-16T16:19:05.991037+0000","last_became_peered":"2024-04-16T16:19:05.991037+0000","last_unstale":"2024-04-16T16:19:10.010180+0000","last_undegraded":"2024-04-16T16:19:10.010180+0000","last_fullsized":"2024-04-16T16:19:10.010180+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:21:47.207147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.977898+0000","last_change":"2024-04-16T16:19:05.985659+0000","last_active":"2024-04-16T16:19:06.977898+0000","last_peered":"2024-04-16T16:19:06.977898+0000","last_clean":"2024-04-16T16:19:06.977898+0000","last_became_active":"2024-04-16T16:19:05.977965+0000","last_became_peered":"2024-04-16T16:19:05.977965+0000","last_unstale":"2024-04-16T16:19:06.977898+0000","last_undegraded":"2024-04-16T16:19:06.977898+0000","last_fullsized":"2024-04-16T16:19:06.977898+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:56:46.027761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012123+0000","last_change":"2024-04-16T16:19:06.003286+0000","last_active":"2024-04-16T16:19:10.012123+0000","last_peered":"2024-04-16T16:19:10.012123+0000","last_clean":"2024-04-16T16:19:10.012123+0000","last_became_active":"2024-04-16T16:19:06.003003+0000","last_became_peered":"2024-04-16T16:19:06.003003+0000","last_unstale":"2024-04-16T16:19:10.012123+0000","last_undegraded":"2024-04-16T16:19:10.012123+0000","last_fullsized":"2024-04-16T16:19:10.012123+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:11:17.864797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.011010+0000","last_change":"2024-04-16T16:19:05.988162+0000","last_active":"2024-04-16T16:19:10.011010+0000","last_peered":"2024-04-16T16:19:10.011010+0000","last_clean":"2024-04-16T16:19:10.011010+0000","last_became_active":"2024-04-16T16:19:05.987809+0000","last_became_peered":"2024-04-16T16:19:05.987809+0000","last_unstale":"2024-04-16T16:19:10.011010+0000","last_undegraded":"2024-04-16T16:19:10.011010+0000","last_fullsized":"2024-04-16T16:19:10.011010+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:22:54.165142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"98'4","reported_seq":18,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986377+0000","last_change":"2024-04-16T16:19:04.984438+0000","last_active":"2024-04-16T16:19:06.986377+0000","last_peered":"2024-04-16T16:19:06.986377+0000","last_clean":"2024-04-16T16:19:06.986377+0000","last_became_active":"2024-04-16T16:19:04.983642+0000","last_became_peered":"2024-04-16T16:19:04.983642+0000","last_unstale":"2024-04-16T16:19:06.986377+0000","last_undegraded":"2024-04-16T16:19:06.986377+0000","last_fullsized":"2024-04-16T16:19:06.986377+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:12.853741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.978816+0000","last_change":"2024-04-16T16:19:05.982797+0000","last_active":"2024-04-16T16:19:06.978816+0000","last_peered":"2024-04-16T16:19:06.978816+0000","last_clean":"2024-04-16T16:19:06.978816+0000","last_became_active":"2024-04-16T16:19:05.982481+0000","last_became_peered":"2024-04-16T16:19:05.982481+0000","last_unstale":"2024-04-16T16:19:06.978816+0000","last_undegraded":"2024-04-16T16:19:06.978816+0000","last_fullsized":"2024-04-16T16:19:06.978816+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:22:35.948275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"98'1","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.015417+0000","last_change":"2024-04-16T16:19:04.986411+0000","last_active":"2024-04-16T16:19:10.015417+0000","last_peered":"2024-04-16T16:19:10.015417+0000","last_clean":"2024-04-16T16:19:10.015417+0000","last_became_active":"2024-04-16T16:19:04.986104+0000","last_became_peered":"2024-04-16T16:19:04.986104+0000","last_unstale":"2024-04-16T16:19:10.015417+0000","last_undegraded":"2024-04-16T16:19:10.015417+0000","last_fullsized":"2024-04-16T16:19:10.015417+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:00.223941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012197+0000","last_change":"2024-04-16T16:19:05.986159+0000","last_active":"2024-04-16T16:19:10.012197+0000","last_peered":"2024-04-16T16:19:10.012197+0000","last_clean":"2024-04-16T16:19:10.012197+0000","last_became_active":"2024-04-16T16:19:05.985815+0000","last_became_peered":"2024-04-16T16:19:05.985815+0000","last_unstale":"2024-04-16T16:19:10.012197+0000","last_undegraded":"2024-04-16T16:19:10.012197+0000","last_fullsized":"2024-04-16T16:19:10.012197+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:07:14.434723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"98'2","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986453+0000","last_change":"2024-04-16T16:19:04.979795+0000","last_active":"2024-04-16T16:19:06.986453+0000","last_peered":"2024-04-16T16:19:06.986453+0000","last_clean":"2024-04-16T16:19:06.986453+0000","last_became_active":"2024-04-16T16:19:04.979405+0000","last_became_peered":"2024-04-16T16:19:04.979405+0000","last_unstale":"2024-04-16T16:19:06.986453+0000","last_undegraded":"2024-04-16T16:19:06.986453+0000","last_fullsized":"2024-04-16T16:19:06.986453+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:11.492016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985536+0000","last_change":"2024-04-16T16:19:05.986605+0000","last_active":"2024-04-16T16:19:06.985536+0000","last_peered":"2024-04-16T16:19:06.985536+0000","last_clean":"2024-04-16T16:19:06.985536+0000","last_became_active":"2024-04-16T16:19:05.985635+0000","last_became_peered":"2024-04-16T16:19:05.985635+0000","last_unstale":"2024-04-16T16:19:06.985536+0000","last_undegraded":"2024-04-16T16:19:06.985536+0000","last_fullsized":"2024-04-16T16:19:06.985536+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:03:59.658810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"98'2","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985956+0000","last_change":"2024-04-16T16:19:04.986153+0000","last_active":"2024-04-16T16:19:06.985956+0000","last_peered":"2024-04-16T16:19:06.985956+0000","last_clean":"2024-04-16T16:19:06.985956+0000","last_became_active":"2024-04-16T16:19:04.985884+0000","last_became_peered":"2024-04-16T16:19:04.985884+0000","last_unstale":"2024-04-16T16:19:06.985956+0000","last_undegraded":"2024-04-16T16:19:06.985956+0000","last_fullsized":"2024-04-16T16:19:06.985956+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:22:15.998081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.188185+0000","last_change":"2024-04-16T16:19:05.986424+0000","last_active":"2024-04-16T16:19:07.188185+0000","last_peered":"2024-04-16T16:19:07.188185+0000","last_clean":"2024-04-16T16:19:07.188185+0000","last_became_active":"2024-04-16T16:19:05.985986+0000","last_became_peered":"2024-04-16T16:19:05.985986+0000","last_unstale":"2024-04-16T16:19:07.188185+0000","last_undegraded":"2024-04-16T16:19:07.188185+0000","last_fullsized":"2024-04-16T16:19:07.188185+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:07.966106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"98'1","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012160+0000","last_change":"2024-04-16T16:19:04.988776+0000","last_active":"2024-04-16T16:19:10.012160+0000","last_peered":"2024-04-16T16:19:10.012160+0000","last_clean":"2024-04-16T16:19:10.012160+0000","last_became_active":"2024-04-16T16:19:04.988460+0000","last_became_peered":"2024-04-16T16:19:04.988460+0000","last_unstale":"2024-04-16T16:19:10.012160+0000","last_undegraded":"2024-04-16T16:19:10.012160+0000","last_fullsized":"2024-04-16T16:19:10.012160+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:15:30.842326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.014976+0000","last_change":"2024-04-16T16:19:05.978166+0000","last_active":"2024-04-16T16:19:10.014976+0000","last_peered":"2024-04-16T16:19:10.014976+0000","last_clean":"2024-04-16T16:19:10.014976+0000","last_became_active":"2024-04-16T16:19:05.977736+0000","last_became_peered":"2024-04-16T16:19:05.977736+0000","last_unstale":"2024-04-16T16:19:10.014976+0000","last_undegraded":"2024-04-16T16:19:10.014976+0000","last_fullsized":"2024-04-16T16:19:10.014976+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:23:17.568253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"98'1","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985700+0000","last_change":"2024-04-16T16:19:04.987702+0000","last_active":"2024-04-16T16:19:06.985700+0000","last_peered":"2024-04-16T16:19:06.985700+0000","last_clean":"2024-04-16T16:19:06.985700+0000","last_became_active":"2024-04-16T16:19:04.987401+0000","last_became_peered":"2024-04-16T16:19:04.987401+0000","last_unstale":"2024-04-16T16:19:06.985700+0000","last_undegraded":"2024-04-16T16:19:06.985700+0000","last_fullsized":"2024-04-16T16:19:06.985700+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:39:54.145075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.977605+0000","last_change":"2024-04-16T16:19:05.975027+0000","last_active":"2024-04-16T16:19:06.977605+0000","last_peered":"2024-04-16T16:19:06.977605+0000","last_clean":"2024-04-16T16:19:06.977605+0000","last_became_active":"2024-04-16T16:19:05.974652+0000","last_became_peered":"2024-04-16T16:19:05.974652+0000","last_unstale":"2024-04-16T16:19:06.977605+0000","last_undegraded":"2024-04-16T16:19:06.977605+0000","last_fullsized":"2024-04-16T16:19:06.977605+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:34:58.912595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.190472+0000","last_change":"2024-04-16T16:19:04.983292+0000","last_active":"2024-04-16T16:19:07.190472+0000","last_peered":"2024-04-16T16:19:07.190472+0000","last_clean":"2024-04-16T16:19:07.190472+0000","last_became_active":"2024-04-16T16:19:04.982893+0000","last_became_peered":"2024-04-16T16:19:04.982893+0000","last_unstale":"2024-04-16T16:19:07.190472+0000","last_undegraded":"2024-04-16T16:19:07.190472+0000","last_fullsized":"2024-04-16T16:19:07.190472+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:06.947923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.984475+0000","last_change":"2024-04-16T16:19:05.986431+0000","last_active":"2024-04-16T16:19:06.984475+0000","last_peered":"2024-04-16T16:19:06.984475+0000","last_clean":"2024-04-16T16:19:06.984475+0000","last_became_active":"2024-04-16T16:19:05.986090+0000","last_became_peered":"2024-04-16T16:19:05.986090+0000","last_unstale":"2024-04-16T16:19:06.984475+0000","last_undegraded":"2024-04-16T16:19:06.984475+0000","last_fullsized":"2024-04-16T16:19:06.984475+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:27:38.222936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012522+0000","last_change":"2024-04-16T16:19:04.992846+0000","last_active":"2024-04-16T16:19:10.012522+0000","last_peered":"2024-04-16T16:19:10.012522+0000","last_clean":"2024-04-16T16:19:10.012522+0000","last_became_active":"2024-04-16T16:19:04.992484+0000","last_became_peered":"2024-04-16T16:19:04.992484+0000","last_unstale":"2024-04-16T16:19:10.012522+0000","last_undegraded":"2024-04-16T16:19:10.012522+0000","last_fullsized":"2024-04-16T16:19:10.012522+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:22:53.146959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986430+0000","last_change":"2024-04-16T16:19:05.996863+0000","last_active":"2024-04-16T16:19:06.986430+0000","last_peered":"2024-04-16T16:19:06.986430+0000","last_clean":"2024-04-16T16:19:06.986430+0000","last_became_active":"2024-04-16T16:19:05.985107+0000","last_became_peered":"2024-04-16T16:19:05.985107+0000","last_unstale":"2024-04-16T16:19:06.986430+0000","last_undegraded":"2024-04-16T16:19:06.986430+0000","last_fullsized":"2024-04-16T16:19:06.986430+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:19.002643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012627+0000","last_change":"2024-04-16T16:19:04.987235+0000","last_active":"2024-04-16T16:19:10.012627+0000","last_peered":"2024-04-16T16:19:10.012627+0000","last_clean":"2024-04-16T16:19:10.012627+0000","last_became_active":"2024-04-16T16:19:04.986897+0000","last_became_peered":"2024-04-16T16:19:04.986897+0000","last_unstale":"2024-04-16T16:19:10.012627+0000","last_undegraded":"2024-04-16T16:19:10.012627+0000","last_fullsized":"2024-04-16T16:19:10.012627+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:08:19.691006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986187+0000","last_change":"2024-04-16T16:19:05.982487+0000","last_active":"2024-04-16T16:19:06.986187+0000","last_peered":"2024-04-16T16:19:06.986187+0000","last_clean":"2024-04-16T16:19:06.986187+0000","last_became_active":"2024-04-16T16:19:05.982138+0000","last_became_peered":"2024-04-16T16:19:05.982138+0000","last_unstale":"2024-04-16T16:19:06.986187+0000","last_undegraded":"2024-04-16T16:19:06.986187+0000","last_fullsized":"2024-04-16T16:19:06.986187+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:45:16.357214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.978453+0000","last_change":"2024-04-16T16:19:04.982523+0000","last_active":"2024-04-16T16:19:06.978453+0000","last_peered":"2024-04-16T16:19:06.978453+0000","last_clean":"2024-04-16T16:19:06.978453+0000","last_became_active":"2024-04-16T16:19:04.982268+0000","last_became_peered":"2024-04-16T16:19:04.982268+0000","last_unstale":"2024-04-16T16:19:06.978453+0000","last_undegraded":"2024-04-16T16:19:06.978453+0000","last_fullsized":"2024-04-16T16:19:06.978453+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:03:58.640627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.189417+0000","last_change":"2024-04-16T16:19:06.005295+0000","last_active":"2024-04-16T16:19:07.189417+0000","last_peered":"2024-04-16T16:19:07.189417+0000","last_clean":"2024-04-16T16:19:07.189417+0000","last_became_active":"2024-04-16T16:19:06.004923+0000","last_became_peered":"2024-04-16T16:19:06.004923+0000","last_unstale":"2024-04-16T16:19:07.189417+0000","last_undegraded":"2024-04-16T16:19:07.189417+0000","last_fullsized":"2024-04-16T16:19:07.189417+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:33:25.856909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"98'3","reported_seq":19,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.015415+0000","last_change":"2024-04-16T16:19:04.992164+0000","last_active":"2024-04-16T16:19:10.015415+0000","last_peered":"2024-04-16T16:19:10.015415+0000","last_clean":"2024-04-16T16:19:10.015415+0000","last_became_active":"2024-04-16T16:19:04.991847+0000","last_became_peered":"2024-04-16T16:19:04.991847+0000","last_unstale":"2024-04-16T16:19:10.015415+0000","last_undegraded":"2024-04-16T16:19:10.015415+0000","last_fullsized":"2024-04-16T16:19:10.015415+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:24:18.616264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.015035+0000","last_change":"2024-04-16T16:19:05.974727+0000","last_active":"2024-04-16T16:19:10.015035+0000","last_peered":"2024-04-16T16:19:10.015035+0000","last_clean":"2024-04-16T16:19:10.015035+0000","last_became_active":"2024-04-16T16:19:05.974040+0000","last_became_peered":"2024-04-16T16:19:05.974040+0000","last_unstale":"2024-04-16T16:19:10.015035+0000","last_undegraded":"2024-04-16T16:19:10.015035+0000","last_fullsized":"2024-04-16T16:19:10.015035+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:57:21.364765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"98'6","reported_seq":20,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985321+0000","last_change":"2024-04-16T16:19:04.995202+0000","last_active":"2024-04-16T16:19:06.985321+0000","last_peered":"2024-04-16T16:19:06.985321+0000","last_clean":"2024-04-16T16:19:06.985321+0000","last_became_active":"2024-04-16T16:19:04.994801+0000","last_became_peered":"2024-04-16T16:19:04.994801+0000","last_unstale":"2024-04-16T16:19:06.985321+0000","last_undegraded":"2024-04-16T16:19:06.985321+0000","last_fullsized":"2024-04-16T16:19:06.985321+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:10:32.648003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.978075+0000","last_change":"2024-04-16T16:19:05.961526+0000","last_active":"2024-04-16T16:19:06.978075+0000","last_peered":"2024-04-16T16:19:06.978075+0000","last_clean":"2024-04-16T16:19:06.978075+0000","last_became_active":"2024-04-16T16:19:05.960960+0000","last_became_peered":"2024-04-16T16:19:05.960960+0000","last_unstale":"2024-04-16T16:19:06.978075+0000","last_undegraded":"2024-04-16T16:19:06.978075+0000","last_fullsized":"2024-04-16T16:19:06.978075+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:32:02.583804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"98'2","reported_seq":18,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010403+0000","last_change":"2024-04-16T16:19:04.991336+0000","last_active":"2024-04-16T16:19:10.010403+0000","last_peered":"2024-04-16T16:19:10.010403+0000","last_clean":"2024-04-16T16:19:10.010403+0000","last_became_active":"2024-04-16T16:19:04.990932+0000","last_became_peered":"2024-04-16T16:19:04.990932+0000","last_unstale":"2024-04-16T16:19:10.010403+0000","last_undegraded":"2024-04-16T16:19:10.010403+0000","last_fullsized":"2024-04-16T16:19:10.010403+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:21.612339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012249+0000","last_change":"2024-04-16T16:19:05.989803+0000","last_active":"2024-04-16T16:19:10.012249+0000","last_peered":"2024-04-16T16:19:10.012249+0000","last_clean":"2024-04-16T16:19:10.012249+0000","last_became_active":"2024-04-16T16:19:05.989395+0000","last_became_peered":"2024-04-16T16:19:05.989395+0000","last_unstale":"2024-04-16T16:19:10.012249+0000","last_undegraded":"2024-04-16T16:19:10.012249+0000","last_fullsized":"2024-04-16T16:19:10.012249+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:06:00.810268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010432+0000","last_change":"2024-04-16T16:19:04.989089+0000","last_active":"2024-04-16T16:19:10.010432+0000","last_peered":"2024-04-16T16:19:10.010432+0000","last_clean":"2024-04-16T16:19:10.010432+0000","last_became_active":"2024-04-16T16:19:04.988850+0000","last_became_peered":"2024-04-16T16:19:04.988850+0000","last_unstale":"2024-04-16T16:19:10.010432+0000","last_undegraded":"2024-04-16T16:19:10.010432+0000","last_fullsized":"2024-04-16T16:19:10.010432+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:18:36.579289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.983974+0000","last_change":"2024-04-16T16:19:05.980698+0000","last_active":"2024-04-16T16:19:06.983974+0000","last_peered":"2024-04-16T16:19:06.983974+0000","last_clean":"2024-04-16T16:19:06.983974+0000","last_became_active":"2024-04-16T16:19:05.980421+0000","last_became_peered":"2024-04-16T16:19:05.980421+0000","last_unstale":"2024-04-16T16:19:06.983974+0000","last_undegraded":"2024-04-16T16:19:06.983974+0000","last_fullsized":"2024-04-16T16:19:06.983974+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:12.510199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.984674+0000","last_change":"2024-04-16T16:19:04.989484+0000","last_active":"2024-04-16T16:19:06.984674+0000","last_peered":"2024-04-16T16:19:06.984674+0000","last_clean":"2024-04-16T16:19:06.984674+0000","last_became_active":"2024-04-16T16:19:04.988809+0000","last_became_peered":"2024-04-16T16:19:04.988809+0000","last_unstale":"2024-04-16T16:19:06.984674+0000","last_undegraded":"2024-04-16T16:19:06.984674+0000","last_fullsized":"2024-04-16T16:19:06.984674+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:41:31.246677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010123+0000","last_change":"2024-04-16T16:19:05.994123+0000","last_active":"2024-04-16T16:19:10.010123+0000","last_peered":"2024-04-16T16:19:10.010123+0000","last_clean":"2024-04-16T16:19:10.010123+0000","last_became_active":"2024-04-16T16:19:05.993792+0000","last_became_peered":"2024-04-16T16:19:05.993792+0000","last_unstale":"2024-04-16T16:19:10.010123+0000","last_undegraded":"2024-04-16T16:19:10.010123+0000","last_fullsized":"2024-04-16T16:19:10.010123+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:13:29.994957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"98'8","reported_seq":24,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012679+0000","last_change":"2024-04-16T16:19:04.986489+0000","last_active":"2024-04-16T16:19:10.012679+0000","last_peered":"2024-04-16T16:19:10.012679+0000","last_clean":"2024-04-16T16:19:10.012679+0000","last_became_active":"2024-04-16T16:19:04.986243+0000","last_became_peered":"2024-04-16T16:19:04.986243+0000","last_unstale":"2024-04-16T16:19:10.012679+0000","last_undegraded":"2024-04-16T16:19:10.012679+0000","last_fullsized":"2024-04-16T16:19:10.012679+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:54:56.595989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.190558+0000","last_change":"2024-04-16T16:19:05.980661+0000","last_active":"2024-04-16T16:19:07.190558+0000","last_peered":"2024-04-16T16:19:07.190558+0000","last_clean":"2024-04-16T16:19:07.190558+0000","last_became_active":"2024-04-16T16:19:05.980320+0000","last_became_peered":"2024-04-16T16:19:05.980320+0000","last_unstale":"2024-04-16T16:19:07.190558+0000","last_undegraded":"2024-04-16T16:19:07.190558+0000","last_fullsized":"2024-04-16T16:19:07.190558+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:02.954116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"98'2","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985626+0000","last_change":"2024-04-16T16:19:04.982453+0000","last_active":"2024-04-16T16:19:06.985626+0000","last_peered":"2024-04-16T16:19:06.985626+0000","last_clean":"2024-04-16T16:19:06.985626+0000","last_became_active":"2024-04-16T16:19:04.982007+0000","last_became_peered":"2024-04-16T16:19:04.982007+0000","last_unstale":"2024-04-16T16:19:06.985626+0000","last_undegraded":"2024-04-16T16:19:06.985626+0000","last_fullsized":"2024-04-16T16:19:06.985626+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:35.123256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.188682+0000","last_change":"2024-04-16T16:19:05.979712+0000","last_active":"2024-04-16T16:19:07.188682+0000","last_peered":"2024-04-16T16:19:07.188682+0000","last_clean":"2024-04-16T16:19:07.188682+0000","last_became_active":"2024-04-16T16:19:05.979378+0000","last_became_peered":"2024-04-16T16:19:05.979378+0000","last_unstale":"2024-04-16T16:19:07.188682+0000","last_undegraded":"2024-04-16T16:19:07.188682+0000","last_fullsized":"2024-04-16T16:19:07.188682+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:17:56.354646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"98'1","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.189565+0000","last_change":"2024-04-16T16:19:04.987633+0000","last_active":"2024-04-16T16:19:07.189565+0000","last_peered":"2024-04-16T16:19:07.189565+0000","last_clean":"2024-04-16T16:19:07.189565+0000","last_became_active":"2024-04-16T16:19:04.985775+0000","last_became_peered":"2024-04-16T16:19:04.985775+0000","last_unstale":"2024-04-16T16:19:07.189565+0000","last_undegraded":"2024-04-16T16:19:07.189565+0000","last_fullsized":"2024-04-16T16:19:07.189565+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:17:55.336463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.015344+0000","last_change":"2024-04-16T16:19:05.980511+0000","last_active":"2024-04-16T16:19:10.015344+0000","last_peered":"2024-04-16T16:19:10.015344+0000","last_clean":"2024-04-16T16:19:10.015344+0000","last_became_active":"2024-04-16T16:19:05.980198+0000","last_became_peered":"2024-04-16T16:19:05.980198+0000","last_unstale":"2024-04-16T16:19:10.015344+0000","last_undegraded":"2024-04-16T16:19:10.015344+0000","last_fullsized":"2024-04-16T16:19:10.015344+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:34:58.912595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986074+0000","last_change":"2024-04-16T16:19:04.986720+0000","last_active":"2024-04-16T16:19:06.986074+0000","last_peered":"2024-04-16T16:19:06.986074+0000","last_clean":"2024-04-16T16:19:06.986074+0000","last_became_active":"2024-04-16T16:19:04.985494+0000","last_became_peered":"2024-04-16T16:19:04.985494+0000","last_unstale":"2024-04-16T16:19:06.986074+0000","last_undegraded":"2024-04-16T16:19:06.986074+0000","last_fullsized":"2024-04-16T16:19:06.986074+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:27:37.204753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010203+0000","last_change":"2024-04-16T16:19:05.994829+0000","last_active":"2024-04-16T16:19:10.010203+0000","last_peered":"2024-04-16T16:19:10.010203+0000","last_clean":"2024-04-16T16:19:10.010203+0000","last_became_active":"2024-04-16T16:19:05.994505+0000","last_became_peered":"2024-04-16T16:19:05.994505+0000","last_unstale":"2024-04-16T16:19:10.010203+0000","last_undegraded":"2024-04-16T16:19:10.010203+0000","last_fullsized":"2024-04-16T16:19:10.010203+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:04:39.234988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"98'2","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986093+0000","last_change":"2024-04-16T16:19:04.979940+0000","last_active":"2024-04-16T16:19:06.986093+0000","last_peered":"2024-04-16T16:19:06.986093+0000","last_clean":"2024-04-16T16:19:06.986093+0000","last_became_active":"2024-04-16T16:19:04.978496+0000","last_became_peered":"2024-04-16T16:19:04.978496+0000","last_unstale":"2024-04-16T16:19:06.986093+0000","last_undegraded":"2024-04-16T16:19:06.986093+0000","last_fullsized":"2024-04-16T16:19:06.986093+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:58.471628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010221+0000","last_change":"2024-04-16T16:19:06.001148+0000","last_active":"2024-04-16T16:19:10.010221+0000","last_peered":"2024-04-16T16:19:10.010221+0000","last_clean":"2024-04-16T16:19:10.010221+0000","last_became_active":"2024-04-16T16:19:06.000880+0000","last_became_peered":"2024-04-16T16:19:06.000880+0000","last_unstale":"2024-04-16T16:19:10.010221+0000","last_undegraded":"2024-04-16T16:19:10.010221+0000","last_fullsized":"2024-04-16T16:19:10.010221+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:54:57.614172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.188125+0000","last_change":"2024-04-16T16:19:04.988799+0000","last_active":"2024-04-16T16:19:07.188125+0000","last_peered":"2024-04-16T16:19:07.188125+0000","last_clean":"2024-04-16T16:19:07.188125+0000","last_became_active":"2024-04-16T16:19:04.988477+0000","last_became_peered":"2024-04-16T16:19:04.988477+0000","last_unstale":"2024-04-16T16:19:07.188125+0000","last_undegraded":"2024-04-16T16:19:07.188125+0000","last_fullsized":"2024-04-16T16:19:07.188125+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:10:32.648003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.009921+0000","last_change":"2024-04-16T16:19:05.984181+0000","last_active":"2024-04-16T16:19:10.009921+0000","last_peered":"2024-04-16T16:19:10.009921+0000","last_clean":"2024-04-16T16:19:10.009921+0000","last_became_active":"2024-04-16T16:19:05.983618+0000","last_became_peered":"2024-04-16T16:19:05.983618+0000","last_unstale":"2024-04-16T16:19:10.009921+0000","last_undegraded":"2024-04-16T16:19:10.009921+0000","last_fullsized":"2024-04-16T16:19:10.009921+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:53:13.854210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.188084+0000","last_change":"2024-04-16T16:19:04.996624+0000","last_active":"2024-04-16T16:19:07.188084+0000","last_peered":"2024-04-16T16:19:07.188084+0000","last_clean":"2024-04-16T16:19:07.188084+0000","last_became_active":"2024-04-16T16:19:04.996192+0000","last_became_peered":"2024-04-16T16:19:04.996192+0000","last_unstale":"2024-04-16T16:19:07.188084+0000","last_undegraded":"2024-04-16T16:19:07.188084+0000","last_fullsized":"2024-04-16T16:19:07.188084+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:35.123256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.190170+0000","last_change":"2024-04-16T16:19:05.981506+0000","last_active":"2024-04-16T16:19:07.190170+0000","last_peered":"2024-04-16T16:19:07.190170+0000","last_clean":"2024-04-16T16:19:07.190170+0000","last_became_active":"2024-04-16T16:19:05.981215+0000","last_became_peered":"2024-04-16T16:19:05.981215+0000","last_unstale":"2024-04-16T16:19:07.190170+0000","last_undegraded":"2024-04-16T16:19:07.190170+0000","last_fullsized":"2024-04-16T16:19:07.190170+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:21:32.529352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985627+0000","last_change":"2024-04-16T16:19:04.981948+0000","last_active":"2024-04-16T16:19:06.985627+0000","last_peered":"2024-04-16T16:19:06.985627+0000","last_clean":"2024-04-16T16:19:06.985627+0000","last_became_active":"2024-04-16T16:19:04.981687+0000","last_became_peered":"2024-04-16T16:19:04.981687+0000","last_unstale":"2024-04-16T16:19:06.985627+0000","last_undegraded":"2024-04-16T16:19:06.985627+0000","last_fullsized":"2024-04-16T16:19:06.985627+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:16.487324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986264+0000","last_change":"2024-04-16T16:19:05.996631+0000","last_active":"2024-04-16T16:19:06.986264+0000","last_peered":"2024-04-16T16:19:06.986264+0000","last_clean":"2024-04-16T16:19:06.986264+0000","last_became_active":"2024-04-16T16:19:05.984675+0000","last_became_peered":"2024-04-16T16:19:05.984675+0000","last_unstale":"2024-04-16T16:19:06.986264+0000","last_undegraded":"2024-04-16T16:19:06.986264+0000","last_fullsized":"2024-04-16T16:19:06.986264+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:00:34.345957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"98'2","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.189935+0000","last_change":"2024-04-16T16:19:04.991640+0000","last_active":"2024-04-16T16:19:07.189935+0000","last_peered":"2024-04-16T16:19:07.189935+0000","last_clean":"2024-04-16T16:19:07.189935+0000","last_became_active":"2024-04-16T16:19:04.991350+0000","last_became_peered":"2024-04-16T16:19:04.991350+0000","last_unstale":"2024-04-16T16:19:07.189935+0000","last_undegraded":"2024-04-16T16:19:07.189935+0000","last_fullsized":"2024-04-16T16:19:07.189935+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:21:51.170584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012522+0000","last_change":"2024-04-16T16:19:05.994408+0000","last_active":"2024-04-16T16:19:10.012522+0000","last_peered":"2024-04-16T16:19:10.012522+0000","last_clean":"2024-04-16T16:19:10.012522+0000","last_became_active":"2024-04-16T16:19:05.994001+0000","last_became_peered":"2024-04-16T16:19:05.994001+0000","last_unstale":"2024-04-16T16:19:10.012522+0000","last_undegraded":"2024-04-16T16:19:10.012522+0000","last_fullsized":"2024-04-16T16:19:10.012522+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:49.973970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"99'1","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986407+0000","last_change":"2024-04-16T16:19:04.984755+0000","last_active":"2024-04-16T16:19:06.986407+0000","last_peered":"2024-04-16T16:19:06.986407+0000","last_clean":"2024-04-16T16:19:06.986407+0000","last_became_active":"2024-04-16T16:19:04.984478+0000","last_became_peered":"2024-04-16T16:19:04.984478+0000","last_unstale":"2024-04-16T16:19:06.986407+0000","last_undegraded":"2024-04-16T16:19:06.986407+0000","last_fullsized":"2024-04-16T16:19:06.986407+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:45:15.339031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012753+0000","last_change":"2024-04-16T16:19:05.978246+0000","last_active":"2024-04-16T16:19:10.012753+0000","last_peered":"2024-04-16T16:19:10.012753+0000","last_clean":"2024-04-16T16:19:10.012753+0000","last_became_active":"2024-04-16T16:19:05.977965+0000","last_became_peered":"2024-04-16T16:19:05.977965+0000","last_unstale":"2024-04-16T16:19:10.012753+0000","last_undegraded":"2024-04-16T16:19:10.012753+0000","last_fullsized":"2024-04-16T16:19:10.012753+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:37:23.592456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986504+0000","last_change":"2024-04-16T16:19:04.976741+0000","last_active":"2024-04-16T16:19:06.986504+0000","last_peered":"2024-04-16T16:19:06.986504+0000","last_clean":"2024-04-16T16:19:06.986504+0000","last_became_active":"2024-04-16T16:19:04.976404+0000","last_became_peered":"2024-04-16T16:19:04.976404+0000","last_unstale":"2024-04-16T16:19:06.986504+0000","last_undegraded":"2024-04-16T16:19:06.986504+0000","last_fullsized":"2024-04-16T16:19:06.986504+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:16:02.001132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.189312+0000","last_change":"2024-04-16T16:19:05.975482+0000","last_active":"2024-04-16T16:19:07.189312+0000","last_peered":"2024-04-16T16:19:07.189312+0000","last_clean":"2024-04-16T16:19:07.189312+0000","last_became_active":"2024-04-16T16:19:05.975144+0000","last_became_peered":"2024-04-16T16:19:05.975144+0000","last_unstale":"2024-04-16T16:19:07.189312+0000","last_undegraded":"2024-04-16T16:19:07.189312+0000","last_fullsized":"2024-04-16T16:19:07.189312+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:32:08.093006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.189646+0000","last_change":"2024-04-16T16:19:04.988355+0000","last_active":"2024-04-16T16:19:07.189646+0000","last_peered":"2024-04-16T16:19:07.189646+0000","last_clean":"2024-04-16T16:19:07.189646+0000","last_became_active":"2024-04-16T16:19:04.987739+0000","last_became_peered":"2024-04-16T16:19:04.987739+0000","last_unstale":"2024-04-16T16:19:07.189646+0000","last_undegraded":"2024-04-16T16:19:07.189646+0000","last_fullsized":"2024-04-16T16:19:07.189646+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:00.223941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010330+0000","last_change":"2024-04-16T16:19:05.988169+0000","last_active":"2024-04-16T16:19:10.010330+0000","last_peered":"2024-04-16T16:19:10.010330+0000","last_clean":"2024-04-16T16:19:10.010330+0000","last_became_active":"2024-04-16T16:19:05.987863+0000","last_became_peered":"2024-04-16T16:19:05.987863+0000","last_unstale":"2024-04-16T16:19:10.010330+0000","last_undegraded":"2024-04-16T16:19:10.010330+0000","last_fullsized":"2024-04-16T16:19:10.010330+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:58.211087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986204+0000","last_change":"2024-04-16T16:19:04.994251+0000","last_active":"2024-04-16T16:19:06.986204+0000","last_peered":"2024-04-16T16:19:06.986204+0000","last_clean":"2024-04-16T16:19:06.986204+0000","last_became_active":"2024-04-16T16:19:04.993943+0000","last_became_peered":"2024-04-16T16:19:04.993943+0000","last_unstale":"2024-04-16T16:19:06.986204+0000","last_undegraded":"2024-04-16T16:19:06.986204+0000","last_fullsized":"2024-04-16T16:19:06.986204+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:03:58.640627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.984919+0000","last_change":"2024-04-16T16:19:05.988368+0000","last_active":"2024-04-16T16:19:06.984919+0000","last_peered":"2024-04-16T16:19:06.984919+0000","last_clean":"2024-04-16T16:19:06.984919+0000","last_became_active":"2024-04-16T16:19:05.987624+0000","last_became_peered":"2024-04-16T16:19:05.987624+0000","last_unstale":"2024-04-16T16:19:06.984919+0000","last_undegraded":"2024-04-16T16:19:06.984919+0000","last_fullsized":"2024-04-16T16:19:06.984919+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:17.505507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.188145+0000","last_change":"2024-04-16T16:19:04.989722+0000","last_active":"2024-04-16T16:19:07.188145+0000","last_peered":"2024-04-16T16:19:07.188145+0000","last_clean":"2024-04-16T16:19:07.188145+0000","last_became_active":"2024-04-16T16:19:04.989282+0000","last_became_peered":"2024-04-16T16:19:04.989282+0000","last_unstale":"2024-04-16T16:19:07.188145+0000","last_undegraded":"2024-04-16T16:19:07.188145+0000","last_fullsized":"2024-04-16T16:19:07.188145+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:48:25.937161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.009925+0000","last_change":"2024-04-16T16:19:05.983908+0000","last_active":"2024-04-16T16:19:10.009925+0000","last_peered":"2024-04-16T16:19:10.009925+0000","last_clean":"2024-04-16T16:19:10.009925+0000","last_became_active":"2024-04-16T16:19:05.983276+0000","last_became_peered":"2024-04-16T16:19:05.983276+0000","last_unstale":"2024-04-16T16:19:10.009925+0000","last_undegraded":"2024-04-16T16:19:10.009925+0000","last_fullsized":"2024-04-16T16:19:10.009925+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:30:07.681951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.977848+0000","last_change":"2024-04-16T16:19:04.979267+0000","last_active":"2024-04-16T16:19:06.977848+0000","last_peered":"2024-04-16T16:19:06.977848+0000","last_clean":"2024-04-16T16:19:06.977848+0000","last_became_active":"2024-04-16T16:19:04.978995+0000","last_became_peered":"2024-04-16T16:19:04.978995+0000","last_unstale":"2024-04-16T16:19:06.977848+0000","last_undegraded":"2024-04-16T16:19:06.977848+0000","last_fullsized":"2024-04-16T16:19:06.977848+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:16:02.001132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.190528+0000","last_change":"2024-04-16T16:19:05.974759+0000","last_active":"2024-04-16T16:19:07.190528+0000","last_peered":"2024-04-16T16:19:07.190528+0000","last_clean":"2024-04-16T16:19:07.190528+0000","last_became_active":"2024-04-16T16:19:05.974440+0000","last_became_peered":"2024-04-16T16:19:05.974440+0000","last_unstale":"2024-04-16T16:19:07.190528+0000","last_undegraded":"2024-04-16T16:19:07.190528+0000","last_fullsized":"2024-04-16T16:19:07.190528+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:23:17.568253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010110+0000","last_change":"2024-04-16T16:19:04.988628+0000","last_active":"2024-04-16T16:19:10.010110+0000","last_peered":"2024-04-16T16:19:10.010110+0000","last_clean":"2024-04-16T16:19:10.010110+0000","last_became_active":"2024-04-16T16:19:04.988315+0000","last_became_peered":"2024-04-16T16:19:04.988315+0000","last_unstale":"2024-04-16T16:19:10.010110+0000","last_undegraded":"2024-04-16T16:19:10.010110+0000","last_fullsized":"2024-04-16T16:19:10.010110+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:04:10.124756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.011155+0000","last_change":"2024-04-16T16:19:05.999564+0000","last_active":"2024-04-16T16:19:10.011155+0000","last_peered":"2024-04-16T16:19:10.011155+0000","last_clean":"2024-04-16T16:19:10.011155+0000","last_became_active":"2024-04-16T16:19:05.994184+0000","last_became_peered":"2024-04-16T16:19:05.994184+0000","last_unstale":"2024-04-16T16:19:10.011155+0000","last_undegraded":"2024-04-16T16:19:10.011155+0000","last_fullsized":"2024-04-16T16:19:10.011155+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:08:20.709189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"98'1","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.014896+0000","last_change":"2024-04-16T16:19:04.995570+0000","last_active":"2024-04-16T16:19:10.014896+0000","last_peered":"2024-04-16T16:19:10.014896+0000","last_clean":"2024-04-16T16:19:10.014896+0000","last_became_active":"2024-04-16T16:19:04.995171+0000","last_became_peered":"2024-04-16T16:19:04.995171+0000","last_unstale":"2024-04-16T16:19:10.014896+0000","last_undegraded":"2024-04-16T16:19:10.014896+0000","last_fullsized":"2024-04-16T16:19:10.014896+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:21:44.566556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"100'2569","reported_seq":2927,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.029884+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:19:10.029884+0000","last_peered":"2024-04-16T16:19:10.029884+0000","last_clean":"2024-04-16T16:19:10.029884+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:19:10.029884+0000","last_undegraded":"2024-04-16T16:19:10.029884+0000","last_fullsized":"2024-04-16T16:19:10.029884+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":2569,"log_dups_size":0,"ondisk_log_size":2569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":551,"num_read_kb":972,"num_write":243,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":551,"num_read_kb":972,"num_write":243,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2569,"ondisk_log_size":2569,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574917,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41280,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330560,"statfs":{"total":96636764160,"available":96594493440,"internally_reserved":0,"allocated":1826816,"data_stored":1168893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574918,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":1232896,"data_stored":579065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574918,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1228800,"data_stored":578483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574919,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32400,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339440,"statfs":{"total":96636764160,"available":96603586560,"internally_reserved":0,"allocated":1245184,"data_stored":580149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1241088,"data_stored":580047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1228800,"data_stored":578447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32384,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339456,"statfs":{"total":96636764160,"available":96603602944,"internally_reserved":0,"allocated":1224704,"data_stored":578413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574918,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28800,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343040,"statfs":{"total":96636764160,"available":96607272960,"internally_reserved":0,"allocated":1822720,"data_stored":1169363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:19:10.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:19:10.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.787+0000 7f7422e00640 1 Processor -- start 2024-04-16T16:19:10.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.788+0000 7f7422e00640 1 -- start start 2024-04-16T16:19:10.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.789+0000 7f7422e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f741c15fea0 0x7f741c071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:10.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.789+0000 7f7422e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f741c0d1070 0x7f741c160d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:10.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.789+0000 7f7422e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f741c071f10 con 0x7f741c0b1b00 2024-04-16T16:19:10.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.789+0000 7f7422e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f741c160280 con 0x7f741c0d1070 2024-04-16T16:19:10.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.789+0000 7f7422e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f741c071ce0 con 0x7f741c15fea0 2024-04-16T16:19:10.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.789+0000 7f7421000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f741c0d1070 0x7f741c160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:10.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.789+0000 7f7421000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f741c0d1070 0x7f741c160d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44728/0 (socket says 172.21.15.45:44728) 2024-04-16T16:19:10.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.789+0000 7f7421000640 1 -- 172.21.15.45:0/3488276840 learned_addr learned my addr 172.21.15.45:0/3488276840 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:10.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.790+0000 7f7421000640 1 -- 172.21.15.45:0/3488276840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f741c15fea0 msgr2=0x7f741c071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:10.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.790+0000 7f7421000640 1 --2- 172.21.15.45:0/3488276840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f741c15fea0 0x7f741c071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.790+0000 7f7421000640 1 -- 172.21.15.45:0/3488276840 >> v1:172.21.15.103:6789/0 conn(0x7f741c0b1b00 legacy=0x7f741c09aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:10.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.790+0000 7f7421000640 1 -- 172.21.15.45:0/3488276840 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741c160520 con 0x7f741c0d1070 2024-04-16T16:19:10.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.791+0000 7f7421000640 1 --2- 172.21.15.45:0/3488276840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f741c0d1070 0x7f741c160d20 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f7408009870 tx=0x7f740802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769840c231298b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:10.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.791+0000 7f741be00640 1 -- 172.21.15.45:0/3488276840 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74080317c0 con 0x7f741c0d1070 2024-04-16T16:19:10.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.791+0000 7f741be00640 1 -- 172.21.15.45:0/3488276840 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7408037070 con 0x7f741c0d1070 2024-04-16T16:19:10.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.791+0000 7f741be00640 1 -- 172.21.15.45:0/3488276840 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7408031b80 con 0x7f741c0d1070 2024-04-16T16:19:10.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.792+0000 7f7422e00640 1 -- 172.21.15.45:0/3488276840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f741c0d1070 msgr2=0x7f741c160d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:10.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.792+0000 7f7422e00640 1 --2- 172.21.15.45:0/3488276840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f741c0d1070 0x7f741c160d20 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f7408009870 tx=0x7f740802b860 comp rx=0 tx=0).stop 2024-04-16T16:19:10.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.792+0000 7f7422e00640 1 -- 172.21.15.45:0/3488276840 shutdown_connections 2024-04-16T16:19:10.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.792+0000 7f7422e00640 1 --2- 172.21.15.45:0/3488276840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f741c0d1070 0x7f741c160d20 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.792+0000 7f7422e00640 1 --2- 172.21.15.45:0/3488276840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f741c15fea0 0x7f741c071240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.792+0000 7f7422e00640 1 -- 172.21.15.45:0/3488276840 >> 172.21.15.45:0/3488276840 conn(0x7f741c0d2610 msgr2=0x7f741c0d2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:10.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.792+0000 7f7422e00640 1 -- 172.21.15.45:0/3488276840 shutdown_connections 2024-04-16T16:19:10.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.792+0000 7f7422e00640 1 -- 172.21.15.45:0/3488276840 wait complete. 2024-04-16T16:19:10.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.794+0000 7f7422e00640 1 Processor -- start 2024-04-16T16:19:10.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.794+0000 7f7422e00640 1 -- start start 2024-04-16T16:19:10.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.795+0000 7f7422e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f741c0b1b00 0x7f741c0c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:10.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.795+0000 7f7422e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f741c15fea0 0x7f741c14e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:10.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.795+0000 7f7422e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f741c14f4c0 0x7f741c070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:10.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.795+0000 7f7422e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f741c0721f0 con 0x7f741c14f4c0 2024-04-16T16:19:10.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.795+0000 7f7422e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f741c161260 con 0x7f741c15fea0 2024-04-16T16:19:10.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.795+0000 7f7422e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f741c09b180 con 0x7f741c0b1b00 2024-04-16T16:19:10.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.795+0000 7f7422400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f741c14f4c0 0x7f741c070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:10.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.795+0000 7f7422400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f741c14f4c0 0x7f741c070aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44736/0 (socket says 172.21.15.45:44736) 2024-04-16T16:19:10.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.795+0000 7f7422400640 1 -- 172.21.15.45:0/3374236007 learned_addr learned my addr 172.21.15.45:0/3374236007 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:10.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.796+0000 7f7421000640 1 --2- 172.21.15.45:0/3374236007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f741c15fea0 0x7f741c14e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:10.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.796+0000 7f7422400640 1 -- 172.21.15.45:0/3374236007 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f741c0b1b00 msgr2=0x7f741c0c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:10.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.796+0000 7f7421a00640 1 --2- 172.21.15.45:0/3374236007 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f741c0b1b00 0x7f741c0c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:10.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.796+0000 7f7422400640 1 --2- 172.21.15.45:0/3374236007 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f741c0b1b00 0x7f741c0c6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.796+0000 7f7422400640 1 -- 172.21.15.45:0/3374236007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f741c15fea0 msgr2=0x7f741c14e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:10.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.796+0000 7f7422400640 1 --2- 172.21.15.45:0/3374236007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f741c15fea0 0x7f741c14e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:10.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.796+0000 7f7422400640 1 -- 172.21.15.45:0/3374236007 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741c0c6680 con 0x7f741c14f4c0 2024-04-16T16:19:10.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.796+0000 7f7421000640 1 --2- 172.21.15.45:0/3374236007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f741c15fea0 0x7f741c14e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:10.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.796+0000 7f7422400640 1 --2- 172.21.15.45:0/3374236007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f741c14f4c0 0x7f741c070aa0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f741400c7e0 tx=0x7f741400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:10.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.796+0000 7f7419600640 1 -- 172.21.15.45:0/3374236007 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7414004b60 con 0x7f741c14f4c0 2024-04-16T16:19:10.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.797+0000 7f7419600640 1 -- 172.21.15.45:0/3374236007 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7414004d00 con 0x7f741c14f4c0 2024-04-16T16:19:10.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.797+0000 7f7419600640 1 -- 172.21.15.45:0/3374236007 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741400aa90 con 0x7f741c14f4c0 2024-04-16T16:19:10.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.797+0000 7f7422e00640 1 -- 172.21.15.45:0/3374236007 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f741c0b3a30 con 0x7f741c14f4c0 2024-04-16T16:19:10.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.797+0000 7f7422e00640 1 -- 172.21.15.45:0/3374236007 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f741c0b05d0 con 0x7f741c14f4c0 2024-04-16T16:19:10.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.798+0000 7f7422e00640 1 -- 172.21.15.45:0/3374236007 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f741c0cca20 con 0x7f741c14f4c0 2024-04-16T16:19:10.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.799+0000 7f7419600640 1 -- 172.21.15.45:0/3374236007 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f741400ac30 con 0x7f741c14f4c0 2024-04-16T16:19:10.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.800+0000 7f7419600640 1 --2- 172.21.15.45:0/3374236007 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f73fc06c980 0x7f73fc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:10.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.800+0000 7f7419600640 1 -- 172.21.15.45:0/3374236007 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5752+0+0 (secure 0 0 0) 0x7f741408bee0 con 0x7f741c14f4c0 2024-04-16T16:19:10.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.801+0000 7f7421a00640 1 --2- 172.21.15.45:0/3374236007 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f73fc06c980 0x7f73fc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:10.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.802+0000 7f7421a00640 1 --2- 172.21.15.45:0/3374236007 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f73fc06c980 0x7f73fc06ee40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f741000a770 tx=0x7f7410008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:10.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:10.805+0000 7f7419600640 1 -- 172.21.15.45:0/3374236007 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7414020070 con 0x7f741c14f4c0 2024-04-16T16:19:10.910 INFO:tasks.ceph.mds.c.smithi045.stderr:2024-04-16T16:19:10.910+0000 7f0e99000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:19:11.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.009+0000 7f7422e00640 1 -- 172.21.15.45:0/3374236007 --> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f741c0b37e0 con 0x7f73fc06c980 2024-04-16T16:19:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.032+0000 7f7419600640 1 -- 172.21.15.45:0/3374236007 <== mgr.6115 v2:172.21.15.103:6832/839990480 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252151 (secure 0 0 0) 0x7f741c0b37e0 con 0x7f73fc06c980 2024-04-16T16:19:11.035 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:11.037 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:19:11.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.039+0000 7f7422e00640 1 -- 172.21.15.45:0/3374236007 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f73fc06c980 msgr2=0x7f73fc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:11.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.039+0000 7f7422e00640 1 --2- 172.21.15.45:0/3374236007 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f73fc06c980 0x7f73fc06ee40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f741000a770 tx=0x7f7410008040 comp rx=0 tx=0).stop 2024-04-16T16:19:11.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.039+0000 7f7422e00640 1 -- 172.21.15.45:0/3374236007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f741c14f4c0 msgr2=0x7f741c070aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:11.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.039+0000 7f7422e00640 1 --2- 172.21.15.45:0/3374236007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f741c14f4c0 0x7f741c070aa0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f741400c7e0 tx=0x7f741400cca0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.039+0000 7f7422e00640 1 -- 172.21.15.45:0/3374236007 shutdown_connections 2024-04-16T16:19:11.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.039+0000 7f7422e00640 1 --2- 172.21.15.45:0/3374236007 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f73fc06c980 0x7f73fc06ee40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.039+0000 7f7422e00640 1 --2- 172.21.15.45:0/3374236007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f741c14f4c0 0x7f741c070aa0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.039+0000 7f7422e00640 1 --2- 172.21.15.45:0/3374236007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f741c15fea0 0x7f741c14e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.039+0000 7f7422e00640 1 --2- 172.21.15.45:0/3374236007 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f741c0b1b00 0x7f741c0c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.039+0000 7f7422e00640 1 -- 172.21.15.45:0/3374236007 >> 172.21.15.45:0/3374236007 conn(0x7f741c0d2610 msgr2=0x7f741c0ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:11.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.039+0000 7f7422e00640 1 -- 172.21.15.45:0/3374236007 shutdown_connections 2024-04-16T16:19:11.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.040+0000 7f7422e00640 1 -- 172.21.15.45:0/3374236007 wait complete. 2024-04-16T16:19:11.060 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-04-16T16:19:10.040078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":551,"num_read_kb":972,"num_write":288,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2608,"ondisk_log_size":2608,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":191,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264428,"kb_used_data":10792,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754710292,"statfs":{"total":773094113280,"available":772823339008,"internally_reserved":0,"allocated":11051008,"data_stored":5812860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":47,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.083436"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012374+0000","last_change":"2024-04-16T16:19:05.980861+0000","last_active":"2024-04-16T16:19:10.012374+0000","last_peered":"2024-04-16T16:19:10.012374+0000","last_clean":"2024-04-16T16:19:10.012374+0000","last_became_active":"2024-04-16T16:19:05.980280+0000","last_became_peered":"2024-04-16T16:19:05.980280+0000","last_unstale":"2024-04-16T16:19:10.012374+0000","last_undegraded":"2024-04-16T16:19:10.012374+0000","last_fullsized":"2024-04-16T16:19:10.012374+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:54:42.032276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985931+0000","last_change":"2024-04-16T16:19:06.004931+0000","last_active":"2024-04-16T16:19:06.985931+0000","last_peered":"2024-04-16T16:19:06.985931+0000","last_clean":"2024-04-16T16:19:06.985931+0000","last_became_active":"2024-04-16T16:19:05.999878+0000","last_became_peered":"2024-04-16T16:19:05.999878+0000","last_unstale":"2024-04-16T16:19:06.985931+0000","last_undegraded":"2024-04-16T16:19:06.985931+0000","last_fullsized":"2024-04-16T16:19:06.985931+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:17.234151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012237+0000","last_change":"2024-04-16T16:19:05.993873+0000","last_active":"2024-04-16T16:19:10.012237+0000","last_peered":"2024-04-16T16:19:10.012237+0000","last_clean":"2024-04-16T16:19:10.012237+0000","last_became_active":"2024-04-16T16:19:05.993512+0000","last_became_peered":"2024-04-16T16:19:05.993512+0000","last_unstale":"2024-04-16T16:19:10.012237+0000","last_undegraded":"2024-04-16T16:19:10.012237+0000","last_fullsized":"2024-04-16T16:19:10.012237+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:19.148184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.978542+0000","last_change":"2024-04-16T16:19:05.995781+0000","last_active":"2024-04-16T16:19:06.978542+0000","last_peered":"2024-04-16T16:19:06.978542+0000","last_clean":"2024-04-16T16:19:06.978542+0000","last_became_active":"2024-04-16T16:19:05.995469+0000","last_became_peered":"2024-04-16T16:19:05.995469+0000","last_unstale":"2024-04-16T16:19:06.978542+0000","last_undegraded":"2024-04-16T16:19:06.978542+0000","last_fullsized":"2024-04-16T16:19:06.978542+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:23:31.314921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.188821+0000","last_change":"2024-04-16T16:19:05.986907+0000","last_active":"2024-04-16T16:19:07.188821+0000","last_peered":"2024-04-16T16:19:07.188821+0000","last_clean":"2024-04-16T16:19:07.188821+0000","last_became_active":"2024-04-16T16:19:05.986607+0000","last_became_peered":"2024-04-16T16:19:05.986607+0000","last_unstale":"2024-04-16T16:19:07.188821+0000","last_undegraded":"2024-04-16T16:19:07.188821+0000","last_fullsized":"2024-04-16T16:19:07.188821+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:45:16.357214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.014299+0000","last_change":"2024-04-16T16:19:05.979722+0000","last_active":"2024-04-16T16:19:10.014299+0000","last_peered":"2024-04-16T16:19:10.014299+0000","last_clean":"2024-04-16T16:19:10.014299+0000","last_became_active":"2024-04-16T16:19:05.979418+0000","last_became_peered":"2024-04-16T16:19:05.979418+0000","last_unstale":"2024-04-16T16:19:10.014299+0000","last_undegraded":"2024-04-16T16:19:10.014299+0000","last_fullsized":"2024-04-16T16:19:10.014299+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:02.954116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.013948+0000","last_change":"2024-04-16T16:19:05.979260+0000","last_active":"2024-04-16T16:19:10.013948+0000","last_peered":"2024-04-16T16:19:10.013948+0000","last_clean":"2024-04-16T16:19:10.013948+0000","last_became_active":"2024-04-16T16:19:05.978961+0000","last_became_peered":"2024-04-16T16:19:05.978961+0000","last_unstale":"2024-04-16T16:19:10.013948+0000","last_undegraded":"2024-04-16T16:19:10.013948+0000","last_fullsized":"2024-04-16T16:19:10.013948+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:13.871924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985611+0000","last_change":"2024-04-16T16:19:05.981860+0000","last_active":"2024-04-16T16:19:06.985611+0000","last_peered":"2024-04-16T16:19:06.985611+0000","last_clean":"2024-04-16T16:19:06.985611+0000","last_became_active":"2024-04-16T16:19:05.981428+0000","last_became_peered":"2024-04-16T16:19:05.981428+0000","last_unstale":"2024-04-16T16:19:06.985611+0000","last_undegraded":"2024-04-16T16:19:06.985611+0000","last_fullsized":"2024-04-16T16:19:06.985611+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:32:02.583804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986006+0000","last_change":"2024-04-16T16:19:05.987627+0000","last_active":"2024-04-16T16:19:06.986006+0000","last_peered":"2024-04-16T16:19:06.986006+0000","last_clean":"2024-04-16T16:19:06.986006+0000","last_became_active":"2024-04-16T16:19:05.987145+0000","last_became_peered":"2024-04-16T16:19:05.987145+0000","last_unstale":"2024-04-16T16:19:06.986006+0000","last_undegraded":"2024-04-16T16:19:06.986006+0000","last_fullsized":"2024-04-16T16:19:06.986006+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:39:55.163258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.190391+0000","last_change":"2024-04-16T16:19:05.979043+0000","last_active":"2024-04-16T16:19:07.190391+0000","last_peered":"2024-04-16T16:19:07.190391+0000","last_clean":"2024-04-16T16:19:07.190391+0000","last_became_active":"2024-04-16T16:19:05.978722+0000","last_became_peered":"2024-04-16T16:19:05.978722+0000","last_unstale":"2024-04-16T16:19:07.190391+0000","last_undegraded":"2024-04-16T16:19:07.190391+0000","last_fullsized":"2024-04-16T16:19:07.190391+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:03.968225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.014757+0000","last_change":"2024-04-16T16:19:05.976920+0000","last_active":"2024-04-16T16:19:10.014757+0000","last_peered":"2024-04-16T16:19:10.014757+0000","last_clean":"2024-04-16T16:19:10.014757+0000","last_became_active":"2024-04-16T16:19:05.976381+0000","last_became_peered":"2024-04-16T16:19:05.976381+0000","last_unstale":"2024-04-16T16:19:10.014757+0000","last_undegraded":"2024-04-16T16:19:10.014757+0000","last_fullsized":"2024-04-16T16:19:10.014757+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:03.968225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.984876+0000","last_change":"2024-04-16T16:19:05.977966+0000","last_active":"2024-04-16T16:19:06.984876+0000","last_peered":"2024-04-16T16:19:06.984876+0000","last_clean":"2024-04-16T16:19:06.984876+0000","last_became_active":"2024-04-16T16:19:05.975530+0000","last_became_peered":"2024-04-16T16:19:05.975530+0000","last_unstale":"2024-04-16T16:19:06.984876+0000","last_undegraded":"2024-04-16T16:19:06.984876+0000","last_fullsized":"2024-04-16T16:19:06.984876+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:59.489811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.978144+0000","last_change":"2024-04-16T16:19:05.978675+0000","last_active":"2024-04-16T16:19:06.978144+0000","last_peered":"2024-04-16T16:19:06.978144+0000","last_clean":"2024-04-16T16:19:06.978144+0000","last_became_active":"2024-04-16T16:19:05.976724+0000","last_became_peered":"2024-04-16T16:19:05.976724+0000","last_unstale":"2024-04-16T16:19:06.978144+0000","last_undegraded":"2024-04-16T16:19:06.978144+0000","last_fullsized":"2024-04-16T16:19:06.978144+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:48.694556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010032+0000","last_change":"2024-04-16T16:19:05.991399+0000","last_active":"2024-04-16T16:19:10.010032+0000","last_peered":"2024-04-16T16:19:10.010032+0000","last_clean":"2024-04-16T16:19:10.010032+0000","last_became_active":"2024-04-16T16:19:05.991105+0000","last_became_peered":"2024-04-16T16:19:05.991105+0000","last_unstale":"2024-04-16T16:19:10.010032+0000","last_undegraded":"2024-04-16T16:19:10.010032+0000","last_fullsized":"2024-04-16T16:19:10.010032+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:59:24.858976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.977751+0000","last_change":"2024-04-16T16:19:05.973800+0000","last_active":"2024-04-16T16:19:06.977751+0000","last_peered":"2024-04-16T16:19:06.977751+0000","last_clean":"2024-04-16T16:19:06.977751+0000","last_became_active":"2024-04-16T16:19:05.972807+0000","last_became_peered":"2024-04-16T16:19:05.972807+0000","last_unstale":"2024-04-16T16:19:06.977751+0000","last_undegraded":"2024-04-16T16:19:06.977751+0000","last_fullsized":"2024-04-16T16:19:06.977751+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:57:21.364765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.013948+0000","last_change":"2024-04-16T16:19:05.972005+0000","last_active":"2024-04-16T16:19:10.013948+0000","last_peered":"2024-04-16T16:19:10.013948+0000","last_clean":"2024-04-16T16:19:10.013948+0000","last_became_active":"2024-04-16T16:19:05.971388+0000","last_became_peered":"2024-04-16T16:19:05.971388+0000","last_unstale":"2024-04-16T16:19:10.013948+0000","last_undegraded":"2024-04-16T16:19:10.013948+0000","last_fullsized":"2024-04-16T16:19:10.013948+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:32:08.093006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986319+0000","last_change":"2024-04-16T16:19:06.001464+0000","last_active":"2024-04-16T16:19:06.986319+0000","last_peered":"2024-04-16T16:19:06.986319+0000","last_clean":"2024-04-16T16:19:06.986319+0000","last_became_active":"2024-04-16T16:19:05.998584+0000","last_became_peered":"2024-04-16T16:19:05.998584+0000","last_unstale":"2024-04-16T16:19:06.986319+0000","last_undegraded":"2024-04-16T16:19:06.986319+0000","last_fullsized":"2024-04-16T16:19:06.986319+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:21:27.367871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.978468+0000","last_change":"2024-04-16T16:19:05.995193+0000","last_active":"2024-04-16T16:19:06.978468+0000","last_peered":"2024-04-16T16:19:06.978468+0000","last_clean":"2024-04-16T16:19:06.978468+0000","last_became_active":"2024-04-16T16:19:05.994843+0000","last_became_peered":"2024-04-16T16:19:05.994843+0000","last_unstale":"2024-04-16T16:19:06.978468+0000","last_undegraded":"2024-04-16T16:19:06.978468+0000","last_fullsized":"2024-04-16T16:19:06.978468+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:17.234151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010279+0000","last_change":"2024-04-16T16:19:05.990057+0000","last_active":"2024-04-16T16:19:10.010279+0000","last_peered":"2024-04-16T16:19:10.010279+0000","last_clean":"2024-04-16T16:19:10.010279+0000","last_became_active":"2024-04-16T16:19:05.989306+0000","last_became_peered":"2024-04-16T16:19:05.989306+0000","last_unstale":"2024-04-16T16:19:10.010279+0000","last_undegraded":"2024-04-16T16:19:10.010279+0000","last_fullsized":"2024-04-16T16:19:10.010279+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:15:09.100928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985000+0000","last_change":"2024-04-16T16:19:05.977562+0000","last_active":"2024-04-16T16:19:06.985000+0000","last_peered":"2024-04-16T16:19:06.985000+0000","last_clean":"2024-04-16T16:19:06.985000+0000","last_became_active":"2024-04-16T16:19:05.976423+0000","last_became_peered":"2024-04-16T16:19:05.976423+0000","last_unstale":"2024-04-16T16:19:06.985000+0000","last_undegraded":"2024-04-16T16:19:06.985000+0000","last_fullsized":"2024-04-16T16:19:06.985000+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:16:03.019315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010120+0000","last_change":"2024-04-16T16:19:05.988445+0000","last_active":"2024-04-16T16:19:10.010120+0000","last_peered":"2024-04-16T16:19:10.010120+0000","last_clean":"2024-04-16T16:19:10.010120+0000","last_became_active":"2024-04-16T16:19:05.988154+0000","last_became_peered":"2024-04-16T16:19:05.988154+0000","last_unstale":"2024-04-16T16:19:10.010120+0000","last_undegraded":"2024-04-16T16:19:10.010120+0000","last_fullsized":"2024-04-16T16:19:10.010120+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:01:13.476876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985634+0000","last_change":"2024-04-16T16:19:05.982959+0000","last_active":"2024-04-16T16:19:06.985634+0000","last_peered":"2024-04-16T16:19:06.985634+0000","last_clean":"2024-04-16T16:19:06.985634+0000","last_became_active":"2024-04-16T16:19:05.980754+0000","last_became_peered":"2024-04-16T16:19:05.980754+0000","last_unstale":"2024-04-16T16:19:06.985634+0000","last_undegraded":"2024-04-16T16:19:06.985634+0000","last_fullsized":"2024-04-16T16:19:06.985634+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:48.694556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986235+0000","last_change":"2024-04-16T16:19:05.982625+0000","last_active":"2024-04-16T16:19:06.986235+0000","last_peered":"2024-04-16T16:19:06.986235+0000","last_clean":"2024-04-16T16:19:06.986235+0000","last_became_active":"2024-04-16T16:19:05.982182+0000","last_became_peered":"2024-04-16T16:19:05.982182+0000","last_unstale":"2024-04-16T16:19:06.986235+0000","last_undegraded":"2024-04-16T16:19:06.986235+0000","last_fullsized":"2024-04-16T16:19:06.986235+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:07:56.982348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.014025+0000","last_change":"2024-04-16T16:19:05.969908+0000","last_active":"2024-04-16T16:19:10.014025+0000","last_peered":"2024-04-16T16:19:10.014025+0000","last_clean":"2024-04-16T16:19:10.014025+0000","last_became_active":"2024-04-16T16:19:05.969572+0000","last_became_peered":"2024-04-16T16:19:05.969572+0000","last_unstale":"2024-04-16T16:19:10.014025+0000","last_undegraded":"2024-04-16T16:19:10.014025+0000","last_fullsized":"2024-04-16T16:19:10.014025+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:33:25.856909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.190682+0000","last_change":"2024-04-16T16:19:06.008841+0000","last_active":"2024-04-16T16:19:07.190682+0000","last_peered":"2024-04-16T16:19:07.190682+0000","last_clean":"2024-04-16T16:19:07.190682+0000","last_became_active":"2024-04-16T16:19:06.008601+0000","last_became_peered":"2024-04-16T16:19:06.008601+0000","last_unstale":"2024-04-16T16:19:07.190682+0000","last_undegraded":"2024-04-16T16:19:07.190682+0000","last_fullsized":"2024-04-16T16:19:07.190682+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:34:58.912595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985960+0000","last_change":"2024-04-16T16:19:05.997077+0000","last_active":"2024-04-16T16:19:06.985960+0000","last_peered":"2024-04-16T16:19:06.985960+0000","last_clean":"2024-04-16T16:19:06.985960+0000","last_became_active":"2024-04-16T16:19:05.996308+0000","last_became_peered":"2024-04-16T16:19:05.996308+0000","last_unstale":"2024-04-16T16:19:06.985960+0000","last_undegraded":"2024-04-16T16:19:06.985960+0000","last_fullsized":"2024-04-16T16:19:06.985960+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:23:31.314921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.014897+0000","last_change":"2024-04-16T16:19:05.989978+0000","last_active":"2024-04-16T16:19:10.014897+0000","last_peered":"2024-04-16T16:19:10.014897+0000","last_clean":"2024-04-16T16:19:10.014897+0000","last_became_active":"2024-04-16T16:19:05.989686+0000","last_became_peered":"2024-04-16T16:19:05.989686+0000","last_unstale":"2024-04-16T16:19:10.014897+0000","last_undegraded":"2024-04-16T16:19:10.014897+0000","last_fullsized":"2024-04-16T16:19:10.014897+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:21:32.529352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.188994+0000","last_change":"2024-04-16T16:19:05.984266+0000","last_active":"2024-04-16T16:19:07.188994+0000","last_peered":"2024-04-16T16:19:07.188994+0000","last_clean":"2024-04-16T16:19:07.188994+0000","last_became_active":"2024-04-16T16:19:05.983968+0000","last_became_peered":"2024-04-16T16:19:05.983968+0000","last_unstale":"2024-04-16T16:19:07.188994+0000","last_undegraded":"2024-04-16T16:19:07.188994+0000","last_fullsized":"2024-04-16T16:19:07.188994+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:13.871924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985985+0000","last_change":"2024-04-16T16:19:05.975292+0000","last_active":"2024-04-16T16:19:06.985985+0000","last_peered":"2024-04-16T16:19:06.985985+0000","last_clean":"2024-04-16T16:19:06.985985+0000","last_became_active":"2024-04-16T16:19:05.975013+0000","last_became_peered":"2024-04-16T16:19:05.975013+0000","last_unstale":"2024-04-16T16:19:06.985985+0000","last_undegraded":"2024-04-16T16:19:06.985985+0000","last_fullsized":"2024-04-16T16:19:06.985985+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:22:35.948275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010180+0000","last_change":"2024-04-16T16:19:05.991370+0000","last_active":"2024-04-16T16:19:10.010180+0000","last_peered":"2024-04-16T16:19:10.010180+0000","last_clean":"2024-04-16T16:19:10.010180+0000","last_became_active":"2024-04-16T16:19:05.991037+0000","last_became_peered":"2024-04-16T16:19:05.991037+0000","last_unstale":"2024-04-16T16:19:10.010180+0000","last_undegraded":"2024-04-16T16:19:10.010180+0000","last_fullsized":"2024-04-16T16:19:10.010180+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:21:47.207147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.977898+0000","last_change":"2024-04-16T16:19:05.985659+0000","last_active":"2024-04-16T16:19:06.977898+0000","last_peered":"2024-04-16T16:19:06.977898+0000","last_clean":"2024-04-16T16:19:06.977898+0000","last_became_active":"2024-04-16T16:19:05.977965+0000","last_became_peered":"2024-04-16T16:19:05.977965+0000","last_unstale":"2024-04-16T16:19:06.977898+0000","last_undegraded":"2024-04-16T16:19:06.977898+0000","last_fullsized":"2024-04-16T16:19:06.977898+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:56:46.027761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012123+0000","last_change":"2024-04-16T16:19:06.003286+0000","last_active":"2024-04-16T16:19:10.012123+0000","last_peered":"2024-04-16T16:19:10.012123+0000","last_clean":"2024-04-16T16:19:10.012123+0000","last_became_active":"2024-04-16T16:19:06.003003+0000","last_became_peered":"2024-04-16T16:19:06.003003+0000","last_unstale":"2024-04-16T16:19:10.012123+0000","last_undegraded":"2024-04-16T16:19:10.012123+0000","last_fullsized":"2024-04-16T16:19:10.012123+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:11:17.864797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.011010+0000","last_change":"2024-04-16T16:19:05.988162+0000","last_active":"2024-04-16T16:19:10.011010+0000","last_peered":"2024-04-16T16:19:10.011010+0000","last_clean":"2024-04-16T16:19:10.011010+0000","last_became_active":"2024-04-16T16:19:05.987809+0000","last_became_peered":"2024-04-16T16:19:05.987809+0000","last_unstale":"2024-04-16T16:19:10.011010+0000","last_undegraded":"2024-04-16T16:19:10.011010+0000","last_fullsized":"2024-04-16T16:19:10.011010+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:22:54.165142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"98'4","reported_seq":18,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986377+0000","last_change":"2024-04-16T16:19:04.984438+0000","last_active":"2024-04-16T16:19:06.986377+0000","last_peered":"2024-04-16T16:19:06.986377+0000","last_clean":"2024-04-16T16:19:06.986377+0000","last_became_active":"2024-04-16T16:19:04.983642+0000","last_became_peered":"2024-04-16T16:19:04.983642+0000","last_unstale":"2024-04-16T16:19:06.986377+0000","last_undegraded":"2024-04-16T16:19:06.986377+0000","last_fullsized":"2024-04-16T16:19:06.986377+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:12.853741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.978816+0000","last_change":"2024-04-16T16:19:05.982797+0000","last_active":"2024-04-16T16:19:06.978816+0000","last_peered":"2024-04-16T16:19:06.978816+0000","last_clean":"2024-04-16T16:19:06.978816+0000","last_became_active":"2024-04-16T16:19:05.982481+0000","last_became_peered":"2024-04-16T16:19:05.982481+0000","last_unstale":"2024-04-16T16:19:06.978816+0000","last_undegraded":"2024-04-16T16:19:06.978816+0000","last_fullsized":"2024-04-16T16:19:06.978816+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:22:35.948275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"98'1","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.015417+0000","last_change":"2024-04-16T16:19:04.986411+0000","last_active":"2024-04-16T16:19:10.015417+0000","last_peered":"2024-04-16T16:19:10.015417+0000","last_clean":"2024-04-16T16:19:10.015417+0000","last_became_active":"2024-04-16T16:19:04.986104+0000","last_became_peered":"2024-04-16T16:19:04.986104+0000","last_unstale":"2024-04-16T16:19:10.015417+0000","last_undegraded":"2024-04-16T16:19:10.015417+0000","last_fullsized":"2024-04-16T16:19:10.015417+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:00.223941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012197+0000","last_change":"2024-04-16T16:19:05.986159+0000","last_active":"2024-04-16T16:19:10.012197+0000","last_peered":"2024-04-16T16:19:10.012197+0000","last_clean":"2024-04-16T16:19:10.012197+0000","last_became_active":"2024-04-16T16:19:05.985815+0000","last_became_peered":"2024-04-16T16:19:05.985815+0000","last_unstale":"2024-04-16T16:19:10.012197+0000","last_undegraded":"2024-04-16T16:19:10.012197+0000","last_fullsized":"2024-04-16T16:19:10.012197+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:07:14.434723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"98'2","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986453+0000","last_change":"2024-04-16T16:19:04.979795+0000","last_active":"2024-04-16T16:19:06.986453+0000","last_peered":"2024-04-16T16:19:06.986453+0000","last_clean":"2024-04-16T16:19:06.986453+0000","last_became_active":"2024-04-16T16:19:04.979405+0000","last_became_peered":"2024-04-16T16:19:04.979405+0000","last_unstale":"2024-04-16T16:19:06.986453+0000","last_undegraded":"2024-04-16T16:19:06.986453+0000","last_fullsized":"2024-04-16T16:19:06.986453+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:11.492016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985536+0000","last_change":"2024-04-16T16:19:05.986605+0000","last_active":"2024-04-16T16:19:06.985536+0000","last_peered":"2024-04-16T16:19:06.985536+0000","last_clean":"2024-04-16T16:19:06.985536+0000","last_became_active":"2024-04-16T16:19:05.985635+0000","last_became_peered":"2024-04-16T16:19:05.985635+0000","last_unstale":"2024-04-16T16:19:06.985536+0000","last_undegraded":"2024-04-16T16:19:06.985536+0000","last_fullsized":"2024-04-16T16:19:06.985536+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:03:59.658810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"98'2","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985956+0000","last_change":"2024-04-16T16:19:04.986153+0000","last_active":"2024-04-16T16:19:06.985956+0000","last_peered":"2024-04-16T16:19:06.985956+0000","last_clean":"2024-04-16T16:19:06.985956+0000","last_became_active":"2024-04-16T16:19:04.985884+0000","last_became_peered":"2024-04-16T16:19:04.985884+0000","last_unstale":"2024-04-16T16:19:06.985956+0000","last_undegraded":"2024-04-16T16:19:06.985956+0000","last_fullsized":"2024-04-16T16:19:06.985956+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:22:15.998081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.188185+0000","last_change":"2024-04-16T16:19:05.986424+0000","last_active":"2024-04-16T16:19:07.188185+0000","last_peered":"2024-04-16T16:19:07.188185+0000","last_clean":"2024-04-16T16:19:07.188185+0000","last_became_active":"2024-04-16T16:19:05.985986+0000","last_became_peered":"2024-04-16T16:19:05.985986+0000","last_unstale":"2024-04-16T16:19:07.188185+0000","last_undegraded":"2024-04-16T16:19:07.188185+0000","last_fullsized":"2024-04-16T16:19:07.188185+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:07.966106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"98'1","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012160+0000","last_change":"2024-04-16T16:19:04.988776+0000","last_active":"2024-04-16T16:19:10.012160+0000","last_peered":"2024-04-16T16:19:10.012160+0000","last_clean":"2024-04-16T16:19:10.012160+0000","last_became_active":"2024-04-16T16:19:04.988460+0000","last_became_peered":"2024-04-16T16:19:04.988460+0000","last_unstale":"2024-04-16T16:19:10.012160+0000","last_undegraded":"2024-04-16T16:19:10.012160+0000","last_fullsized":"2024-04-16T16:19:10.012160+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:15:30.842326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.014976+0000","last_change":"2024-04-16T16:19:05.978166+0000","last_active":"2024-04-16T16:19:10.014976+0000","last_peered":"2024-04-16T16:19:10.014976+0000","last_clean":"2024-04-16T16:19:10.014976+0000","last_became_active":"2024-04-16T16:19:05.977736+0000","last_became_peered":"2024-04-16T16:19:05.977736+0000","last_unstale":"2024-04-16T16:19:10.014976+0000","last_undegraded":"2024-04-16T16:19:10.014976+0000","last_fullsized":"2024-04-16T16:19:10.014976+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:23:17.568253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"98'1","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985700+0000","last_change":"2024-04-16T16:19:04.987702+0000","last_active":"2024-04-16T16:19:06.985700+0000","last_peered":"2024-04-16T16:19:06.985700+0000","last_clean":"2024-04-16T16:19:06.985700+0000","last_became_active":"2024-04-16T16:19:04.987401+0000","last_became_peered":"2024-04-16T16:19:04.987401+0000","last_unstale":"2024-04-16T16:19:06.985700+0000","last_undegraded":"2024-04-16T16:19:06.985700+0000","last_fullsized":"2024-04-16T16:19:06.985700+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:39:54.145075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.977605+0000","last_change":"2024-04-16T16:19:05.975027+0000","last_active":"2024-04-16T16:19:06.977605+0000","last_peered":"2024-04-16T16:19:06.977605+0000","last_clean":"2024-04-16T16:19:06.977605+0000","last_became_active":"2024-04-16T16:19:05.974652+0000","last_became_peered":"2024-04-16T16:19:05.974652+0000","last_unstale":"2024-04-16T16:19:06.977605+0000","last_undegraded":"2024-04-16T16:19:06.977605+0000","last_fullsized":"2024-04-16T16:19:06.977605+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:34:58.912595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.190472+0000","last_change":"2024-04-16T16:19:04.983292+0000","last_active":"2024-04-16T16:19:07.190472+0000","last_peered":"2024-04-16T16:19:07.190472+0000","last_clean":"2024-04-16T16:19:07.190472+0000","last_became_active":"2024-04-16T16:19:04.982893+0000","last_became_peered":"2024-04-16T16:19:04.982893+0000","last_unstale":"2024-04-16T16:19:07.190472+0000","last_undegraded":"2024-04-16T16:19:07.190472+0000","last_fullsized":"2024-04-16T16:19:07.190472+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:06.947923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.984475+0000","last_change":"2024-04-16T16:19:05.986431+0000","last_active":"2024-04-16T16:19:06.984475+0000","last_peered":"2024-04-16T16:19:06.984475+0000","last_clean":"2024-04-16T16:19:06.984475+0000","last_became_active":"2024-04-16T16:19:05.986090+0000","last_became_peered":"2024-04-16T16:19:05.986090+0000","last_unstale":"2024-04-16T16:19:06.984475+0000","last_undegraded":"2024-04-16T16:19:06.984475+0000","last_fullsized":"2024-04-16T16:19:06.984475+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:27:38.222936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012522+0000","last_change":"2024-04-16T16:19:04.992846+0000","last_active":"2024-04-16T16:19:10.012522+0000","last_peered":"2024-04-16T16:19:10.012522+0000","last_clean":"2024-04-16T16:19:10.012522+0000","last_became_active":"2024-04-16T16:19:04.992484+0000","last_became_peered":"2024-04-16T16:19:04.992484+0000","last_unstale":"2024-04-16T16:19:10.012522+0000","last_undegraded":"2024-04-16T16:19:10.012522+0000","last_fullsized":"2024-04-16T16:19:10.012522+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:22:53.146959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986430+0000","last_change":"2024-04-16T16:19:05.996863+0000","last_active":"2024-04-16T16:19:06.986430+0000","last_peered":"2024-04-16T16:19:06.986430+0000","last_clean":"2024-04-16T16:19:06.986430+0000","last_became_active":"2024-04-16T16:19:05.985107+0000","last_became_peered":"2024-04-16T16:19:05.985107+0000","last_unstale":"2024-04-16T16:19:06.986430+0000","last_undegraded":"2024-04-16T16:19:06.986430+0000","last_fullsized":"2024-04-16T16:19:06.986430+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:19.002643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012627+0000","last_change":"2024-04-16T16:19:04.987235+0000","last_active":"2024-04-16T16:19:10.012627+0000","last_peered":"2024-04-16T16:19:10.012627+0000","last_clean":"2024-04-16T16:19:10.012627+0000","last_became_active":"2024-04-16T16:19:04.986897+0000","last_became_peered":"2024-04-16T16:19:04.986897+0000","last_unstale":"2024-04-16T16:19:10.012627+0000","last_undegraded":"2024-04-16T16:19:10.012627+0000","last_fullsized":"2024-04-16T16:19:10.012627+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:08:19.691006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986187+0000","last_change":"2024-04-16T16:19:05.982487+0000","last_active":"2024-04-16T16:19:06.986187+0000","last_peered":"2024-04-16T16:19:06.986187+0000","last_clean":"2024-04-16T16:19:06.986187+0000","last_became_active":"2024-04-16T16:19:05.982138+0000","last_became_peered":"2024-04-16T16:19:05.982138+0000","last_unstale":"2024-04-16T16:19:06.986187+0000","last_undegraded":"2024-04-16T16:19:06.986187+0000","last_fullsized":"2024-04-16T16:19:06.986187+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:45:16.357214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.978453+0000","last_change":"2024-04-16T16:19:04.982523+0000","last_active":"2024-04-16T16:19:06.978453+0000","last_peered":"2024-04-16T16:19:06.978453+0000","last_clean":"2024-04-16T16:19:06.978453+0000","last_became_active":"2024-04-16T16:19:04.982268+0000","last_became_peered":"2024-04-16T16:19:04.982268+0000","last_unstale":"2024-04-16T16:19:06.978453+0000","last_undegraded":"2024-04-16T16:19:06.978453+0000","last_fullsized":"2024-04-16T16:19:06.978453+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:03:58.640627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.189417+0000","last_change":"2024-04-16T16:19:06.005295+0000","last_active":"2024-04-16T16:19:07.189417+0000","last_peered":"2024-04-16T16:19:07.189417+0000","last_clean":"2024-04-16T16:19:07.189417+0000","last_became_active":"2024-04-16T16:19:06.004923+0000","last_became_peered":"2024-04-16T16:19:06.004923+0000","last_unstale":"2024-04-16T16:19:07.189417+0000","last_undegraded":"2024-04-16T16:19:07.189417+0000","last_fullsized":"2024-04-16T16:19:07.189417+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:33:25.856909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"98'3","reported_seq":19,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.015415+0000","last_change":"2024-04-16T16:19:04.992164+0000","last_active":"2024-04-16T16:19:10.015415+0000","last_peered":"2024-04-16T16:19:10.015415+0000","last_clean":"2024-04-16T16:19:10.015415+0000","last_became_active":"2024-04-16T16:19:04.991847+0000","last_became_peered":"2024-04-16T16:19:04.991847+0000","last_unstale":"2024-04-16T16:19:10.015415+0000","last_undegraded":"2024-04-16T16:19:10.015415+0000","last_fullsized":"2024-04-16T16:19:10.015415+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:24:18.616264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.015035+0000","last_change":"2024-04-16T16:19:05.974727+0000","last_active":"2024-04-16T16:19:10.015035+0000","last_peered":"2024-04-16T16:19:10.015035+0000","last_clean":"2024-04-16T16:19:10.015035+0000","last_became_active":"2024-04-16T16:19:05.974040+0000","last_became_peered":"2024-04-16T16:19:05.974040+0000","last_unstale":"2024-04-16T16:19:10.015035+0000","last_undegraded":"2024-04-16T16:19:10.015035+0000","last_fullsized":"2024-04-16T16:19:10.015035+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:57:21.364765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"98'6","reported_seq":20,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985321+0000","last_change":"2024-04-16T16:19:04.995202+0000","last_active":"2024-04-16T16:19:06.985321+0000","last_peered":"2024-04-16T16:19:06.985321+0000","last_clean":"2024-04-16T16:19:06.985321+0000","last_became_active":"2024-04-16T16:19:04.994801+0000","last_became_peered":"2024-04-16T16:19:04.994801+0000","last_unstale":"2024-04-16T16:19:06.985321+0000","last_undegraded":"2024-04-16T16:19:06.985321+0000","last_fullsized":"2024-04-16T16:19:06.985321+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:10:32.648003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.978075+0000","last_change":"2024-04-16T16:19:05.961526+0000","last_active":"2024-04-16T16:19:06.978075+0000","last_peered":"2024-04-16T16:19:06.978075+0000","last_clean":"2024-04-16T16:19:06.978075+0000","last_became_active":"2024-04-16T16:19:05.960960+0000","last_became_peered":"2024-04-16T16:19:05.960960+0000","last_unstale":"2024-04-16T16:19:06.978075+0000","last_undegraded":"2024-04-16T16:19:06.978075+0000","last_fullsized":"2024-04-16T16:19:06.978075+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:32:02.583804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"98'2","reported_seq":18,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010403+0000","last_change":"2024-04-16T16:19:04.991336+0000","last_active":"2024-04-16T16:19:10.010403+0000","last_peered":"2024-04-16T16:19:10.010403+0000","last_clean":"2024-04-16T16:19:10.010403+0000","last_became_active":"2024-04-16T16:19:04.990932+0000","last_became_peered":"2024-04-16T16:19:04.990932+0000","last_unstale":"2024-04-16T16:19:10.010403+0000","last_undegraded":"2024-04-16T16:19:10.010403+0000","last_fullsized":"2024-04-16T16:19:10.010403+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:21.612339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012249+0000","last_change":"2024-04-16T16:19:05.989803+0000","last_active":"2024-04-16T16:19:10.012249+0000","last_peered":"2024-04-16T16:19:10.012249+0000","last_clean":"2024-04-16T16:19:10.012249+0000","last_became_active":"2024-04-16T16:19:05.989395+0000","last_became_peered":"2024-04-16T16:19:05.989395+0000","last_unstale":"2024-04-16T16:19:10.012249+0000","last_undegraded":"2024-04-16T16:19:10.012249+0000","last_fullsized":"2024-04-16T16:19:10.012249+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:06:00.810268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010432+0000","last_change":"2024-04-16T16:19:04.989089+0000","last_active":"2024-04-16T16:19:10.010432+0000","last_peered":"2024-04-16T16:19:10.010432+0000","last_clean":"2024-04-16T16:19:10.010432+0000","last_became_active":"2024-04-16T16:19:04.988850+0000","last_became_peered":"2024-04-16T16:19:04.988850+0000","last_unstale":"2024-04-16T16:19:10.010432+0000","last_undegraded":"2024-04-16T16:19:10.010432+0000","last_fullsized":"2024-04-16T16:19:10.010432+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:18:36.579289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.983974+0000","last_change":"2024-04-16T16:19:05.980698+0000","last_active":"2024-04-16T16:19:06.983974+0000","last_peered":"2024-04-16T16:19:06.983974+0000","last_clean":"2024-04-16T16:19:06.983974+0000","last_became_active":"2024-04-16T16:19:05.980421+0000","last_became_peered":"2024-04-16T16:19:05.980421+0000","last_unstale":"2024-04-16T16:19:06.983974+0000","last_undegraded":"2024-04-16T16:19:06.983974+0000","last_fullsized":"2024-04-16T16:19:06.983974+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:12.510199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.984674+0000","last_change":"2024-04-16T16:19:04.989484+0000","last_active":"2024-04-16T16:19:06.984674+0000","last_peered":"2024-04-16T16:19:06.984674+0000","last_clean":"2024-04-16T16:19:06.984674+0000","last_became_active":"2024-04-16T16:19:04.988809+0000","last_became_peered":"2024-04-16T16:19:04.988809+0000","last_unstale":"2024-04-16T16:19:06.984674+0000","last_undegraded":"2024-04-16T16:19:06.984674+0000","last_fullsized":"2024-04-16T16:19:06.984674+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:41:31.246677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010123+0000","last_change":"2024-04-16T16:19:05.994123+0000","last_active":"2024-04-16T16:19:10.010123+0000","last_peered":"2024-04-16T16:19:10.010123+0000","last_clean":"2024-04-16T16:19:10.010123+0000","last_became_active":"2024-04-16T16:19:05.993792+0000","last_became_peered":"2024-04-16T16:19:05.993792+0000","last_unstale":"2024-04-16T16:19:10.010123+0000","last_undegraded":"2024-04-16T16:19:10.010123+0000","last_fullsized":"2024-04-16T16:19:10.010123+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:13:29.994957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"98'8","reported_seq":24,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012679+0000","last_change":"2024-04-16T16:19:04.986489+0000","last_active":"2024-04-16T16:19:10.012679+0000","last_peered":"2024-04-16T16:19:10.012679+0000","last_clean":"2024-04-16T16:19:10.012679+0000","last_became_active":"2024-04-16T16:19:04.986243+0000","last_became_peered":"2024-04-16T16:19:04.986243+0000","last_unstale":"2024-04-16T16:19:10.012679+0000","last_undegraded":"2024-04-16T16:19:10.012679+0000","last_fullsized":"2024-04-16T16:19:10.012679+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:54:56.595989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.190558+0000","last_change":"2024-04-16T16:19:05.980661+0000","last_active":"2024-04-16T16:19:07.190558+0000","last_peered":"2024-04-16T16:19:07.190558+0000","last_clean":"2024-04-16T16:19:07.190558+0000","last_became_active":"2024-04-16T16:19:05.980320+0000","last_became_peered":"2024-04-16T16:19:05.980320+0000","last_unstale":"2024-04-16T16:19:07.190558+0000","last_undegraded":"2024-04-16T16:19:07.190558+0000","last_fullsized":"2024-04-16T16:19:07.190558+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:02.954116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"98'2","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985626+0000","last_change":"2024-04-16T16:19:04.982453+0000","last_active":"2024-04-16T16:19:06.985626+0000","last_peered":"2024-04-16T16:19:06.985626+0000","last_clean":"2024-04-16T16:19:06.985626+0000","last_became_active":"2024-04-16T16:19:04.982007+0000","last_became_peered":"2024-04-16T16:19:04.982007+0000","last_unstale":"2024-04-16T16:19:06.985626+0000","last_undegraded":"2024-04-16T16:19:06.985626+0000","last_fullsized":"2024-04-16T16:19:06.985626+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:35.123256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.188682+0000","last_change":"2024-04-16T16:19:05.979712+0000","last_active":"2024-04-16T16:19:07.188682+0000","last_peered":"2024-04-16T16:19:07.188682+0000","last_clean":"2024-04-16T16:19:07.188682+0000","last_became_active":"2024-04-16T16:19:05.979378+0000","last_became_peered":"2024-04-16T16:19:05.979378+0000","last_unstale":"2024-04-16T16:19:07.188682+0000","last_undegraded":"2024-04-16T16:19:07.188682+0000","last_fullsized":"2024-04-16T16:19:07.188682+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:17:56.354646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"98'1","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.189565+0000","last_change":"2024-04-16T16:19:04.987633+0000","last_active":"2024-04-16T16:19:07.189565+0000","last_peered":"2024-04-16T16:19:07.189565+0000","last_clean":"2024-04-16T16:19:07.189565+0000","last_became_active":"2024-04-16T16:19:04.985775+0000","last_became_peered":"2024-04-16T16:19:04.985775+0000","last_unstale":"2024-04-16T16:19:07.189565+0000","last_undegraded":"2024-04-16T16:19:07.189565+0000","last_fullsized":"2024-04-16T16:19:07.189565+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:17:55.336463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.015344+0000","last_change":"2024-04-16T16:19:05.980511+0000","last_active":"2024-04-16T16:19:10.015344+0000","last_peered":"2024-04-16T16:19:10.015344+0000","last_clean":"2024-04-16T16:19:10.015344+0000","last_became_active":"2024-04-16T16:19:05.980198+0000","last_became_peered":"2024-04-16T16:19:05.980198+0000","last_unstale":"2024-04-16T16:19:10.015344+0000","last_undegraded":"2024-04-16T16:19:10.015344+0000","last_fullsized":"2024-04-16T16:19:10.015344+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:34:58.912595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986074+0000","last_change":"2024-04-16T16:19:04.986720+0000","last_active":"2024-04-16T16:19:06.986074+0000","last_peered":"2024-04-16T16:19:06.986074+0000","last_clean":"2024-04-16T16:19:06.986074+0000","last_became_active":"2024-04-16T16:19:04.985494+0000","last_became_peered":"2024-04-16T16:19:04.985494+0000","last_unstale":"2024-04-16T16:19:06.986074+0000","last_undegraded":"2024-04-16T16:19:06.986074+0000","last_fullsized":"2024-04-16T16:19:06.986074+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:27:37.204753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010203+0000","last_change":"2024-04-16T16:19:05.994829+0000","last_active":"2024-04-16T16:19:10.010203+0000","last_peered":"2024-04-16T16:19:10.010203+0000","last_clean":"2024-04-16T16:19:10.010203+0000","last_became_active":"2024-04-16T16:19:05.994505+0000","last_became_peered":"2024-04-16T16:19:05.994505+0000","last_unstale":"2024-04-16T16:19:10.010203+0000","last_undegraded":"2024-04-16T16:19:10.010203+0000","last_fullsized":"2024-04-16T16:19:10.010203+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:04:39.234988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"98'2","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986093+0000","last_change":"2024-04-16T16:19:04.979940+0000","last_active":"2024-04-16T16:19:06.986093+0000","last_peered":"2024-04-16T16:19:06.986093+0000","last_clean":"2024-04-16T16:19:06.986093+0000","last_became_active":"2024-04-16T16:19:04.978496+0000","last_became_peered":"2024-04-16T16:19:04.978496+0000","last_unstale":"2024-04-16T16:19:06.986093+0000","last_undegraded":"2024-04-16T16:19:06.986093+0000","last_fullsized":"2024-04-16T16:19:06.986093+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:58.471628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010221+0000","last_change":"2024-04-16T16:19:06.001148+0000","last_active":"2024-04-16T16:19:10.010221+0000","last_peered":"2024-04-16T16:19:10.010221+0000","last_clean":"2024-04-16T16:19:10.010221+0000","last_became_active":"2024-04-16T16:19:06.000880+0000","last_became_peered":"2024-04-16T16:19:06.000880+0000","last_unstale":"2024-04-16T16:19:10.010221+0000","last_undegraded":"2024-04-16T16:19:10.010221+0000","last_fullsized":"2024-04-16T16:19:10.010221+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:54:57.614172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.188125+0000","last_change":"2024-04-16T16:19:04.988799+0000","last_active":"2024-04-16T16:19:07.188125+0000","last_peered":"2024-04-16T16:19:07.188125+0000","last_clean":"2024-04-16T16:19:07.188125+0000","last_became_active":"2024-04-16T16:19:04.988477+0000","last_became_peered":"2024-04-16T16:19:04.988477+0000","last_unstale":"2024-04-16T16:19:07.188125+0000","last_undegraded":"2024-04-16T16:19:07.188125+0000","last_fullsized":"2024-04-16T16:19:07.188125+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:10:32.648003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.009921+0000","last_change":"2024-04-16T16:19:05.984181+0000","last_active":"2024-04-16T16:19:10.009921+0000","last_peered":"2024-04-16T16:19:10.009921+0000","last_clean":"2024-04-16T16:19:10.009921+0000","last_became_active":"2024-04-16T16:19:05.983618+0000","last_became_peered":"2024-04-16T16:19:05.983618+0000","last_unstale":"2024-04-16T16:19:10.009921+0000","last_undegraded":"2024-04-16T16:19:10.009921+0000","last_fullsized":"2024-04-16T16:19:10.009921+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:53:13.854210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.188084+0000","last_change":"2024-04-16T16:19:04.996624+0000","last_active":"2024-04-16T16:19:07.188084+0000","last_peered":"2024-04-16T16:19:07.188084+0000","last_clean":"2024-04-16T16:19:07.188084+0000","last_became_active":"2024-04-16T16:19:04.996192+0000","last_became_peered":"2024-04-16T16:19:04.996192+0000","last_unstale":"2024-04-16T16:19:07.188084+0000","last_undegraded":"2024-04-16T16:19:07.188084+0000","last_fullsized":"2024-04-16T16:19:07.188084+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:35.123256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.190170+0000","last_change":"2024-04-16T16:19:05.981506+0000","last_active":"2024-04-16T16:19:07.190170+0000","last_peered":"2024-04-16T16:19:07.190170+0000","last_clean":"2024-04-16T16:19:07.190170+0000","last_became_active":"2024-04-16T16:19:05.981215+0000","last_became_peered":"2024-04-16T16:19:05.981215+0000","last_unstale":"2024-04-16T16:19:07.190170+0000","last_undegraded":"2024-04-16T16:19:07.190170+0000","last_fullsized":"2024-04-16T16:19:07.190170+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:21:32.529352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.985627+0000","last_change":"2024-04-16T16:19:04.981948+0000","last_active":"2024-04-16T16:19:06.985627+0000","last_peered":"2024-04-16T16:19:06.985627+0000","last_clean":"2024-04-16T16:19:06.985627+0000","last_became_active":"2024-04-16T16:19:04.981687+0000","last_became_peered":"2024-04-16T16:19:04.981687+0000","last_unstale":"2024-04-16T16:19:06.985627+0000","last_undegraded":"2024-04-16T16:19:06.985627+0000","last_fullsized":"2024-04-16T16:19:06.985627+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:16.487324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986264+0000","last_change":"2024-04-16T16:19:05.996631+0000","last_active":"2024-04-16T16:19:06.986264+0000","last_peered":"2024-04-16T16:19:06.986264+0000","last_clean":"2024-04-16T16:19:06.986264+0000","last_became_active":"2024-04-16T16:19:05.984675+0000","last_became_peered":"2024-04-16T16:19:05.984675+0000","last_unstale":"2024-04-16T16:19:06.986264+0000","last_undegraded":"2024-04-16T16:19:06.986264+0000","last_fullsized":"2024-04-16T16:19:06.986264+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:00:34.345957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"98'2","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.189935+0000","last_change":"2024-04-16T16:19:04.991640+0000","last_active":"2024-04-16T16:19:07.189935+0000","last_peered":"2024-04-16T16:19:07.189935+0000","last_clean":"2024-04-16T16:19:07.189935+0000","last_became_active":"2024-04-16T16:19:04.991350+0000","last_became_peered":"2024-04-16T16:19:04.991350+0000","last_unstale":"2024-04-16T16:19:07.189935+0000","last_undegraded":"2024-04-16T16:19:07.189935+0000","last_fullsized":"2024-04-16T16:19:07.189935+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:21:51.170584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012522+0000","last_change":"2024-04-16T16:19:05.994408+0000","last_active":"2024-04-16T16:19:10.012522+0000","last_peered":"2024-04-16T16:19:10.012522+0000","last_clean":"2024-04-16T16:19:10.012522+0000","last_became_active":"2024-04-16T16:19:05.994001+0000","last_became_peered":"2024-04-16T16:19:05.994001+0000","last_unstale":"2024-04-16T16:19:10.012522+0000","last_undegraded":"2024-04-16T16:19:10.012522+0000","last_fullsized":"2024-04-16T16:19:10.012522+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:49.973970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"99'1","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986407+0000","last_change":"2024-04-16T16:19:04.984755+0000","last_active":"2024-04-16T16:19:06.986407+0000","last_peered":"2024-04-16T16:19:06.986407+0000","last_clean":"2024-04-16T16:19:06.986407+0000","last_became_active":"2024-04-16T16:19:04.984478+0000","last_became_peered":"2024-04-16T16:19:04.984478+0000","last_unstale":"2024-04-16T16:19:06.986407+0000","last_undegraded":"2024-04-16T16:19:06.986407+0000","last_fullsized":"2024-04-16T16:19:06.986407+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:45:15.339031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.012753+0000","last_change":"2024-04-16T16:19:05.978246+0000","last_active":"2024-04-16T16:19:10.012753+0000","last_peered":"2024-04-16T16:19:10.012753+0000","last_clean":"2024-04-16T16:19:10.012753+0000","last_became_active":"2024-04-16T16:19:05.977965+0000","last_became_peered":"2024-04-16T16:19:05.977965+0000","last_unstale":"2024-04-16T16:19:10.012753+0000","last_undegraded":"2024-04-16T16:19:10.012753+0000","last_fullsized":"2024-04-16T16:19:10.012753+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:37:23.592456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986504+0000","last_change":"2024-04-16T16:19:04.976741+0000","last_active":"2024-04-16T16:19:06.986504+0000","last_peered":"2024-04-16T16:19:06.986504+0000","last_clean":"2024-04-16T16:19:06.986504+0000","last_became_active":"2024-04-16T16:19:04.976404+0000","last_became_peered":"2024-04-16T16:19:04.976404+0000","last_unstale":"2024-04-16T16:19:06.986504+0000","last_undegraded":"2024-04-16T16:19:06.986504+0000","last_fullsized":"2024-04-16T16:19:06.986504+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:16:02.001132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.189312+0000","last_change":"2024-04-16T16:19:05.975482+0000","last_active":"2024-04-16T16:19:07.189312+0000","last_peered":"2024-04-16T16:19:07.189312+0000","last_clean":"2024-04-16T16:19:07.189312+0000","last_became_active":"2024-04-16T16:19:05.975144+0000","last_became_peered":"2024-04-16T16:19:05.975144+0000","last_unstale":"2024-04-16T16:19:07.189312+0000","last_undegraded":"2024-04-16T16:19:07.189312+0000","last_fullsized":"2024-04-16T16:19:07.189312+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:32:08.093006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.189646+0000","last_change":"2024-04-16T16:19:04.988355+0000","last_active":"2024-04-16T16:19:07.189646+0000","last_peered":"2024-04-16T16:19:07.189646+0000","last_clean":"2024-04-16T16:19:07.189646+0000","last_became_active":"2024-04-16T16:19:04.987739+0000","last_became_peered":"2024-04-16T16:19:04.987739+0000","last_unstale":"2024-04-16T16:19:07.189646+0000","last_undegraded":"2024-04-16T16:19:07.189646+0000","last_fullsized":"2024-04-16T16:19:07.189646+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:00.223941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010330+0000","last_change":"2024-04-16T16:19:05.988169+0000","last_active":"2024-04-16T16:19:10.010330+0000","last_peered":"2024-04-16T16:19:10.010330+0000","last_clean":"2024-04-16T16:19:10.010330+0000","last_became_active":"2024-04-16T16:19:05.987863+0000","last_became_peered":"2024-04-16T16:19:05.987863+0000","last_unstale":"2024-04-16T16:19:10.010330+0000","last_undegraded":"2024-04-16T16:19:10.010330+0000","last_fullsized":"2024-04-16T16:19:10.010330+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:58.211087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.986204+0000","last_change":"2024-04-16T16:19:04.994251+0000","last_active":"2024-04-16T16:19:06.986204+0000","last_peered":"2024-04-16T16:19:06.986204+0000","last_clean":"2024-04-16T16:19:06.986204+0000","last_became_active":"2024-04-16T16:19:04.993943+0000","last_became_peered":"2024-04-16T16:19:04.993943+0000","last_unstale":"2024-04-16T16:19:06.986204+0000","last_undegraded":"2024-04-16T16:19:06.986204+0000","last_fullsized":"2024-04-16T16:19:06.986204+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:03:58.640627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.984919+0000","last_change":"2024-04-16T16:19:05.988368+0000","last_active":"2024-04-16T16:19:06.984919+0000","last_peered":"2024-04-16T16:19:06.984919+0000","last_clean":"2024-04-16T16:19:06.984919+0000","last_became_active":"2024-04-16T16:19:05.987624+0000","last_became_peered":"2024-04-16T16:19:05.987624+0000","last_unstale":"2024-04-16T16:19:06.984919+0000","last_undegraded":"2024-04-16T16:19:06.984919+0000","last_fullsized":"2024-04-16T16:19:06.984919+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:17.505507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.188145+0000","last_change":"2024-04-16T16:19:04.989722+0000","last_active":"2024-04-16T16:19:07.188145+0000","last_peered":"2024-04-16T16:19:07.188145+0000","last_clean":"2024-04-16T16:19:07.188145+0000","last_became_active":"2024-04-16T16:19:04.989282+0000","last_became_peered":"2024-04-16T16:19:04.989282+0000","last_unstale":"2024-04-16T16:19:07.188145+0000","last_undegraded":"2024-04-16T16:19:07.188145+0000","last_fullsized":"2024-04-16T16:19:07.188145+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:48:25.937161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.009925+0000","last_change":"2024-04-16T16:19:05.983908+0000","last_active":"2024-04-16T16:19:10.009925+0000","last_peered":"2024-04-16T16:19:10.009925+0000","last_clean":"2024-04-16T16:19:10.009925+0000","last_became_active":"2024-04-16T16:19:05.983276+0000","last_became_peered":"2024-04-16T16:19:05.983276+0000","last_unstale":"2024-04-16T16:19:10.009925+0000","last_undegraded":"2024-04-16T16:19:10.009925+0000","last_fullsized":"2024-04-16T16:19:10.009925+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:30:07.681951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:06.977848+0000","last_change":"2024-04-16T16:19:04.979267+0000","last_active":"2024-04-16T16:19:06.977848+0000","last_peered":"2024-04-16T16:19:06.977848+0000","last_clean":"2024-04-16T16:19:06.977848+0000","last_became_active":"2024-04-16T16:19:04.978995+0000","last_became_peered":"2024-04-16T16:19:04.978995+0000","last_unstale":"2024-04-16T16:19:06.977848+0000","last_undegraded":"2024-04-16T16:19:06.977848+0000","last_fullsized":"2024-04-16T16:19:06.977848+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:16:02.001132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-16T16:19:07.190528+0000","last_change":"2024-04-16T16:19:05.974759+0000","last_active":"2024-04-16T16:19:07.190528+0000","last_peered":"2024-04-16T16:19:07.190528+0000","last_clean":"2024-04-16T16:19:07.190528+0000","last_became_active":"2024-04-16T16:19:05.974440+0000","last_became_peered":"2024-04-16T16:19:05.974440+0000","last_unstale":"2024-04-16T16:19:07.190528+0000","last_undegraded":"2024-04-16T16:19:07.190528+0000","last_fullsized":"2024-04-16T16:19:07.190528+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:23:17.568253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.010110+0000","last_change":"2024-04-16T16:19:04.988628+0000","last_active":"2024-04-16T16:19:10.010110+0000","last_peered":"2024-04-16T16:19:10.010110+0000","last_clean":"2024-04-16T16:19:10.010110+0000","last_became_active":"2024-04-16T16:19:04.988315+0000","last_became_peered":"2024-04-16T16:19:04.988315+0000","last_unstale":"2024-04-16T16:19:10.010110+0000","last_undegraded":"2024-04-16T16:19:10.010110+0000","last_fullsized":"2024-04-16T16:19:10.010110+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:04:10.124756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.011155+0000","last_change":"2024-04-16T16:19:05.999564+0000","last_active":"2024-04-16T16:19:10.011155+0000","last_peered":"2024-04-16T16:19:10.011155+0000","last_clean":"2024-04-16T16:19:10.011155+0000","last_became_active":"2024-04-16T16:19:05.994184+0000","last_became_peered":"2024-04-16T16:19:05.994184+0000","last_unstale":"2024-04-16T16:19:10.011155+0000","last_undegraded":"2024-04-16T16:19:10.011155+0000","last_fullsized":"2024-04-16T16:19:10.011155+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:04.935049+0000","last_clean_scrub_stamp":"2024-04-16T16:19:04.935049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:08:20.709189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"98'1","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.014896+0000","last_change":"2024-04-16T16:19:04.995570+0000","last_active":"2024-04-16T16:19:10.014896+0000","last_peered":"2024-04-16T16:19:10.014896+0000","last_clean":"2024-04-16T16:19:10.014896+0000","last_became_active":"2024-04-16T16:19:04.995171+0000","last_became_peered":"2024-04-16T16:19:04.995171+0000","last_unstale":"2024-04-16T16:19:10.014896+0000","last_undegraded":"2024-04-16T16:19:10.014896+0000","last_fullsized":"2024-04-16T16:19:10.014896+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:03.916866+0000","last_clean_scrub_stamp":"2024-04-16T16:19:03.916866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:21:44.566556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"100'2569","reported_seq":2927,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-16T16:19:10.029884+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:19:10.029884+0000","last_peered":"2024-04-16T16:19:10.029884+0000","last_clean":"2024-04-16T16:19:10.029884+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:19:10.029884+0000","last_undegraded":"2024-04-16T16:19:10.029884+0000","last_fullsized":"2024-04-16T16:19:10.029884+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":2569,"log_dups_size":0,"ondisk_log_size":2569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":551,"num_read_kb":972,"num_write":243,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":551,"num_read_kb":972,"num_write":243,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2569,"ondisk_log_size":2569,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574917,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41280,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330560,"statfs":{"total":96636764160,"available":96594493440,"internally_reserved":0,"allocated":1826816,"data_stored":1168893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574918,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":1232896,"data_stored":579065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574918,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1228800,"data_stored":578483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574919,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32400,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339440,"statfs":{"total":96636764160,"available":96603586560,"internally_reserved":0,"allocated":1245184,"data_stored":580149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1241088,"data_stored":580047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1228800,"data_stored":578447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32384,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339456,"statfs":{"total":96636764160,"available":96603602944,"internally_reserved":0,"allocated":1224704,"data_stored":578413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574918,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28800,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343040,"statfs":{"total":96636764160,"available":96607272960,"internally_reserved":0,"allocated":1822720,"data_stored":1169363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:19:11.064 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-16T16:19:11.064 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-16T16:19:11.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.338+0000 7f5c9ac00640 1 Processor -- start 2024-04-16T16:19:11.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.338+0000 7f5c9ac00640 1 -- start start 2024-04-16T16:19:11.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.340+0000 7f5c9ac00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5c941452e0 0x7f5c9406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:11.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.340+0000 7f5c9ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c940cecc0 0x7f5c9415cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:11.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.340+0000 7f5c9ac00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c94071e60 con 0x7f5c9406e920 2024-04-16T16:19:11.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.340+0000 7f5c9ac00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5c94072f00 con 0x7f5c941452e0 2024-04-16T16:19:11.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.340+0000 7f5c9ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5c9406ed90 con 0x7f5c940cecc0 2024-04-16T16:19:11.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.340+0000 7f5c99800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5c941452e0 0x7f5c9406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:11.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.340+0000 7f5c99800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5c941452e0 0x7f5c9406e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44742/0 (socket says 172.21.15.45:44742) 2024-04-16T16:19:11.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.340+0000 7f5c99800640 1 -- 172.21.15.45:0/2050980327 learned_addr learned my addr 172.21.15.45:0/2050980327 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:11.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.340+0000 7f5c9a200640 1 --2- 172.21.15.45:0/2050980327 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c940cecc0 0x7f5c9415cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:11.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.341+0000 7f5c99800640 1 -- 172.21.15.45:0/2050980327 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c940cecc0 msgr2=0x7f5c9415cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:11.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.341+0000 7f5c99800640 1 --2- 172.21.15.45:0/2050980327 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c940cecc0 0x7f5c9415cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.341+0000 7f5c99800640 1 -- 172.21.15.45:0/2050980327 >> v1:172.21.15.103:6789/0 conn(0x7f5c9406e920 legacy=0x7f5c940ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:11.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.341+0000 7f5c99800640 1 -- 172.21.15.45:0/2050980327 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c94072140 con 0x7f5c941452e0 2024-04-16T16:19:11.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.342+0000 7f5c99800640 1 --2- 172.21.15.45:0/2050980327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c941452e0 0x7f5c9406e3e0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f5c940702a0 tx=0x7f5c8c030330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34fce7c03bb338cc server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:11.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.342+0000 7f5c93e00640 1 -- 172.21.15.45:0/2050980327 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c0383f0 con 0x7f5c941452e0 2024-04-16T16:19:11.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.343+0000 7f5c93e00640 1 -- 172.21.15.45:0/2050980327 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c8c038590 con 0x7f5c941452e0 2024-04-16T16:19:11.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.343+0000 7f5c93e00640 1 -- 172.21.15.45:0/2050980327 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c038960 con 0x7f5c941452e0 2024-04-16T16:19:11.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.343+0000 7f5c9ac00640 1 -- 172.21.15.45:0/2050980327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c941452e0 msgr2=0x7f5c9406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:11.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.343+0000 7f5c9ac00640 1 --2- 172.21.15.45:0/2050980327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c941452e0 0x7f5c9406e3e0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f5c940702a0 tx=0x7f5c8c030330 comp rx=0 tx=0).stop 2024-04-16T16:19:11.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.343+0000 7f5c9ac00640 1 -- 172.21.15.45:0/2050980327 shutdown_connections 2024-04-16T16:19:11.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.343+0000 7f5c9ac00640 1 --2- 172.21.15.45:0/2050980327 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c940cecc0 0x7f5c9415cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.343+0000 7f5c9ac00640 1 --2- 172.21.15.45:0/2050980327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c941452e0 0x7f5c9406e3e0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.343+0000 7f5c9ac00640 1 -- 172.21.15.45:0/2050980327 >> 172.21.15.45:0/2050980327 conn(0x7f5c940b3bb0 msgr2=0x7f5c940b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:11.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.344+0000 7f5c9ac00640 1 -- 172.21.15.45:0/2050980327 shutdown_connections 2024-04-16T16:19:11.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.344+0000 7f5c9ac00640 1 -- 172.21.15.45:0/2050980327 wait complete. 2024-04-16T16:19:11.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.345+0000 7f5c9ac00640 1 Processor -- start 2024-04-16T16:19:11.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.345+0000 7f5c9ac00640 1 -- start start 2024-04-16T16:19:11.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.346+0000 7f5c9ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c9406e920 0x7f5c94160cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:11.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.346+0000 7f5c9ac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c940cecc0 0x7f5c941605b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:11.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.346+0000 7f5c9ac00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c941452e0 0x7f5c9413ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:11.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.346+0000 7f5c9ac00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5c940729e0 con 0x7f5c941452e0 2024-04-16T16:19:11.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.346+0000 7f5c9ac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5c94072700 con 0x7f5c940cecc0 2024-04-16T16:19:11.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.346+0000 7f5c9ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5c941456c0 con 0x7f5c9406e920 2024-04-16T16:19:11.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.347+0000 7f5c9a200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c941452e0 0x7f5c9413ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:11.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.347+0000 7f5c9a200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c941452e0 0x7f5c9413ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44752/0 (socket says 172.21.15.45:44752) 2024-04-16T16:19:11.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.347+0000 7f5c9a200640 1 -- 172.21.15.45:0/2668154770 learned_addr learned my addr 172.21.15.45:0/2668154770 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:11.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.347+0000 7f5c99800640 1 --2- 172.21.15.45:0/2668154770 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c9406e920 0x7f5c94160cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:11.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.347+0000 7f5c98e00640 1 --2- 172.21.15.45:0/2668154770 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c940cecc0 0x7f5c941605b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:11.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.347+0000 7f5c9a200640 1 -- 172.21.15.45:0/2668154770 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c9406e920 msgr2=0x7f5c94160cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:11.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.347+0000 7f5c9a200640 1 --2- 172.21.15.45:0/2668154770 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c9406e920 0x7f5c94160cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.347+0000 7f5c9a200640 1 -- 172.21.15.45:0/2668154770 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c940cecc0 msgr2=0x7f5c941605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:11.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.347+0000 7f5c9a200640 1 --2- 172.21.15.45:0/2668154770 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c940cecc0 0x7f5c941605b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.347+0000 7f5c9a200640 1 -- 172.21.15.45:0/2668154770 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c9415c740 con 0x7f5c941452e0 2024-04-16T16:19:11.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.347+0000 7f5c98e00640 1 --2- 172.21.15.45:0/2668154770 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c940cecc0 0x7f5c941605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:11.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.347+0000 7f5c9a200640 1 --2- 172.21.15.45:0/2668154770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c941452e0 0x7f5c9413ddb0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f5c8400d830 tx=0x7f5c8400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:11.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.348+0000 7f5c91600640 1 -- 172.21.15.45:0/2668154770 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8400b6c0 con 0x7f5c941452e0 2024-04-16T16:19:11.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.348+0000 7f5c91600640 1 -- 172.21.15.45:0/2668154770 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c84019070 con 0x7f5c941452e0 2024-04-16T16:19:11.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.348+0000 7f5c91600640 1 -- 172.21.15.45:0/2668154770 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8400bac0 con 0x7f5c941452e0 2024-04-16T16:19:11.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.348+0000 7f5c9ac00640 1 -- 172.21.15.45:0/2668154770 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c940aa230 con 0x7f5c941452e0 2024-04-16T16:19:11.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.348+0000 7f5c9ac00640 1 -- 172.21.15.45:0/2668154770 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c9415fdb0 con 0x7f5c941452e0 2024-04-16T16:19:11.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.349+0000 7f5c9ac00640 1 -- 172.21.15.45:0/2668154770 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c94075780 con 0x7f5c941452e0 2024-04-16T16:19:11.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.350+0000 7f5c91600640 1 -- 172.21.15.45:0/2668154770 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f5c8401e460 con 0x7f5c941452e0 2024-04-16T16:19:11.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.351+0000 7f5c91600640 1 --2- 172.21.15.45:0/2668154770 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5c7406c980 0x7f5c7406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:11.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.351+0000 7f5c91600640 1 -- 172.21.15.45:0/2668154770 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5752+0+0 (secure 0 0 0) 0x7f5c8408c1f0 con 0x7f5c941452e0 2024-04-16T16:19:11.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.352+0000 7f5c99800640 1 --2- 172.21.15.45:0/2668154770 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5c7406c980 0x7f5c7406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:11.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.353+0000 7f5c99800640 1 --2- 172.21.15.45:0/2668154770 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5c7406c980 0x7f5c7406ee40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c039000 tx=0x7f5c8c034000 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:11.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.356+0000 7f5c91600640 1 -- 172.21.15.45:0/2668154770 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c8405a690 con 0x7f5c941452e0 2024-04-16T16:19:11.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.634+0000 7f5c9ac00640 1 -- 172.21.15.45:0/2668154770 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f5c9413ed60 con 0x7f5c941452e0 2024-04-16T16:19:11.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.639+0000 7f5c91600640 1 -- 172.21.15.45:0/2668154770 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) ==== 190+0+147 (secure 0 0 0) 0x7f5c84059f90 con 0x7f5c941452e0 2024-04-16T16:19:11.639 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-04-16T16:19:11.639 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ== 2024-04-16T16:19:11.639 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:19:11.639 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:19:11.639 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-16T16:19:11.639 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2024-04-16T16:19:11.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.641+0000 7f5c9ac00640 1 -- 172.21.15.45:0/2668154770 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5c7406c980 msgr2=0x7f5c7406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:11.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.641+0000 7f5c9ac00640 1 --2- 172.21.15.45:0/2668154770 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5c7406c980 0x7f5c7406ee40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c039000 tx=0x7f5c8c034000 comp rx=0 tx=0).stop 2024-04-16T16:19:11.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.641+0000 7f5c9ac00640 1 -- 172.21.15.45:0/2668154770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c941452e0 msgr2=0x7f5c9413ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:11.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.641+0000 7f5c9ac00640 1 --2- 172.21.15.45:0/2668154770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c941452e0 0x7f5c9413ddb0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f5c8400d830 tx=0x7f5c8400dcf0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.641+0000 7f5c9ac00640 1 -- 172.21.15.45:0/2668154770 shutdown_connections 2024-04-16T16:19:11.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.641+0000 7f5c9ac00640 1 --2- 172.21.15.45:0/2668154770 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5c7406c980 0x7f5c7406ee40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.642+0000 7f5c9ac00640 1 --2- 172.21.15.45:0/2668154770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c941452e0 0x7f5c9413ddb0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.642+0000 7f5c9ac00640 1 --2- 172.21.15.45:0/2668154770 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c940cecc0 0x7f5c941605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.642+0000 7f5c9ac00640 1 --2- 172.21.15.45:0/2668154770 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c9406e920 0x7f5c94160cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.642+0000 7f5c9ac00640 1 -- 172.21.15.45:0/2668154770 >> 172.21.15.45:0/2668154770 conn(0x7f5c940b3bb0 msgr2=0x7f5c940b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:11.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.642+0000 7f5c9ac00640 1 -- 172.21.15.45:0/2668154770 shutdown_connections 2024-04-16T16:19:11.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.642+0000 7f5c9ac00640 1 -- 172.21.15.45:0/2668154770 wait complete. 2024-04-16T16:19:11.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:19:11.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.881+0000 7f601b200640 1 Processor -- start 2024-04-16T16:19:11.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.882+0000 7f601b200640 1 -- start start 2024-04-16T16:19:11.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.883+0000 7f601b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6014161e90 0x7f60140cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:11.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.883+0000 7f601b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6014162270 0x7f60140d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:11.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.883+0000 7f601b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60140d3080 con 0x7f60140d52d0 2024-04-16T16:19:11.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.883+0000 7f601b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f60140d2e30 con 0x7f6014162270 2024-04-16T16:19:11.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.883+0000 7f601b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f60140d3360 con 0x7f6014161e90 2024-04-16T16:19:11.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.884+0000 7f6019400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6014162270 0x7f60140d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:11.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.884+0000 7f6019e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6014161e90 0x7f60140cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:11.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.884+0000 7f6019400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6014162270 0x7f60140d3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:33176/0 (socket says 172.21.15.45:33176) 2024-04-16T16:19:11.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.884+0000 7f6019400640 1 -- 172.21.15.45:0/631733824 learned_addr learned my addr 172.21.15.45:0/631733824 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:11.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.885+0000 7f6019400640 1 -- 172.21.15.45:0/631733824 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6014161e90 msgr2=0x7f60140cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:11.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.885+0000 7f6019400640 1 --2- 172.21.15.45:0/631733824 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6014161e90 0x7f60140cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.885+0000 7f6019400640 1 -- 172.21.15.45:0/631733824 >> v1:172.21.15.103:6789/0 conn(0x7f60140d52d0 legacy=0x7f601406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:11.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.885+0000 7f6019400640 1 -- 172.21.15.45:0/631733824 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60140d3640 con 0x7f6014162270 2024-04-16T16:19:11.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.886+0000 7f6019400640 1 --2- 172.21.15.45:0/631733824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6014162270 0x7f60140d3e40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6004009870 tx=0x7f600402ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20dd203a2769a3c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:11.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.887+0000 7f6018a00640 1 -- 172.21.15.45:0/631733824 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600402f830 con 0x7f6014162270 2024-04-16T16:19:11.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.887+0000 7f6018a00640 1 -- 172.21.15.45:0/631733824 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f600402f9d0 con 0x7f6014162270 2024-04-16T16:19:11.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.887+0000 7f6018a00640 1 -- 172.21.15.45:0/631733824 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600402fda0 con 0x7f6014162270 2024-04-16T16:19:11.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.887+0000 7f601b200640 1 -- 172.21.15.45:0/631733824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6014162270 msgr2=0x7f60140d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:11.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.887+0000 7f601b200640 1 --2- 172.21.15.45:0/631733824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6014162270 0x7f60140d3e40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6004009870 tx=0x7f600402ba10 comp rx=0 tx=0).stop 2024-04-16T16:19:11.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.887+0000 7f601b200640 1 -- 172.21.15.45:0/631733824 shutdown_connections 2024-04-16T16:19:11.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.887+0000 7f601b200640 1 --2- 172.21.15.45:0/631733824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6014162270 0x7f60140d3e40 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.887+0000 7f601b200640 1 --2- 172.21.15.45:0/631733824 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6014161e90 0x7f60140cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.887+0000 7f601b200640 1 -- 172.21.15.45:0/631733824 >> 172.21.15.45:0/631733824 conn(0x7f60140d1940 msgr2=0x7f60140d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:11.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.888+0000 7f601b200640 1 -- 172.21.15.45:0/631733824 shutdown_connections 2024-04-16T16:19:11.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.888+0000 7f601b200640 1 -- 172.21.15.45:0/631733824 wait complete. 2024-04-16T16:19:11.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.889+0000 7f601b200640 1 Processor -- start 2024-04-16T16:19:11.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.889+0000 7f601b200640 1 -- start start 2024-04-16T16:19:11.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.890+0000 7f601b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60140d52d0 0x7f601415c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:11.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.890+0000 7f601b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6014161e90 0x7f601414da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:11.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.890+0000 7f601b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6014162270 0x7f60140cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:11.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.890+0000 7f601b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f601406d550 con 0x7f6014161e90 2024-04-16T16:19:11.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.890+0000 7f601b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60140cfd10 con 0x7f6014162270 2024-04-16T16:19:11.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.890+0000 7f601b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f601406d6d0 con 0x7f60140d52d0 2024-04-16T16:19:11.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.891+0000 7f6019400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6014161e90 0x7f601414da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:11.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.891+0000 7f6019400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6014161e90 0x7f601414da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44756/0 (socket says 172.21.15.45:44756) 2024-04-16T16:19:11.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.891+0000 7f6019400640 1 -- 172.21.15.45:0/2295589579 learned_addr learned my addr 172.21.15.45:0/2295589579 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:11.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.891+0000 7f6019e00640 1 --2- 172.21.15.45:0/2295589579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60140d52d0 0x7f601415c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:11.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.891+0000 7f601a800640 1 --2- 172.21.15.45:0/2295589579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6014162270 0x7f60140cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:11.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.891+0000 7f6019400640 1 -- 172.21.15.45:0/2295589579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60140d52d0 msgr2=0x7f601415c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:11.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.891+0000 7f6019400640 1 --2- 172.21.15.45:0/2295589579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60140d52d0 0x7f601415c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.891+0000 7f6019400640 1 -- 172.21.15.45:0/2295589579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6014162270 msgr2=0x7f60140cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:11.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.891+0000 7f6019400640 1 --2- 172.21.15.45:0/2295589579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6014162270 0x7f60140cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:11.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.891+0000 7f6019400640 1 -- 172.21.15.45:0/2295589579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601415a4a0 con 0x7f6014161e90 2024-04-16T16:19:11.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.891+0000 7f6019e00640 1 --2- 172.21.15.45:0/2295589579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60140d52d0 0x7f601415c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:11.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.892+0000 7f6019400640 1 --2- 172.21.15.45:0/2295589579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6014161e90 0x7f601414da90 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f600402ba50 tx=0x7f6004004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:11.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.892+0000 7f600e000640 1 -- 172.21.15.45:0/2295589579 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6004047070 con 0x7f6014161e90 2024-04-16T16:19:11.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.892+0000 7f600e000640 1 -- 172.21.15.45:0/2295589579 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6004037c60 con 0x7f6014161e90 2024-04-16T16:19:11.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.892+0000 7f600e000640 1 -- 172.21.15.45:0/2295589579 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6004042630 con 0x7f6014161e90 2024-04-16T16:19:11.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.892+0000 7f601b200640 1 -- 172.21.15.45:0/2295589579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f601415a740 con 0x7f6014161e90 2024-04-16T16:19:11.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.892+0000 7f601b200640 1 -- 172.21.15.45:0/2295589579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f601415ad00 con 0x7f6014161e90 2024-04-16T16:19:11.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.894+0000 7f601b200640 1 -- 172.21.15.45:0/2295589579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60140b9150 con 0x7f6014161e90 2024-04-16T16:19:11.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.894+0000 7f600e000640 1 -- 172.21.15.45:0/2295589579 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 32) ==== 91086+0+0 (secure 0 0 0) 0x7f600402d3c0 con 0x7f6014161e90 2024-04-16T16:19:11.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.895+0000 7f600e000640 1 --2- 172.21.15.45:0/2295589579 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5ff806c980 0x7f5ff806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:11.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.895+0000 7f600e000640 1 -- 172.21.15.45:0/2295589579 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5752+0+0 (secure 0 0 0) 0x7f60040adcd0 con 0x7f6014161e90 2024-04-16T16:19:11.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.896+0000 7f6019e00640 1 --2- 172.21.15.45:0/2295589579 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5ff806c980 0x7f5ff806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:11.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.897+0000 7f6019e00640 1 --2- 172.21.15.45:0/2295589579 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5ff806c980 0x7f5ff806ee40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6008009ac0 tx=0x7f6008008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:11.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:11.900+0000 7f600e000640 1 -- 172.21.15.45:0/2295589579 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f60140b9150 con 0x7f6014161e90 2024-04-16T16:19:12.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.153+0000 7f601b200640 1 -- 172.21.15.45:0/2295589579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6014075780 con 0x7f6014161e90 2024-04-16T16:19:12.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.154+0000 7f600e000640 1 -- 172.21.15.45:0/2295589579 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5186 (secure 0 0 0) 0x7f6004033030 con 0x7f6014161e90 2024-04-16T16:19:12.155 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:12.155 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5804,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2862612740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2862612740},{"type":"v1","addr":"172.21.15.103:6836","nonce":2862612740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6304,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2934367463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2934367463},{"type":"v1","addr":"172.21.15.103:6837","nonce":2934367463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1425499906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1425499906},{"type":"v1","addr":"172.21.15.45:6835","nonce":1425499906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:19:05.457389+0000","modified":"2024-04-16T16:19:08.527316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7677},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7677":{"gid":7677,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/882883484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":882883484},{"type":"v1","addr":"172.21.15.45:6834","nonce":882883484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7677,"qdb_cluster":[7677]},"id":7}]} 2024-04-16T16:19:12.155 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 73 2024-04-16T16:19:12.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.157+0000 7f601b200640 1 -- 172.21.15.45:0/2295589579 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5ff806c980 msgr2=0x7f5ff806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:12.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.157+0000 7f601b200640 1 --2- 172.21.15.45:0/2295589579 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5ff806c980 0x7f5ff806ee40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6008009ac0 tx=0x7f6008008040 comp rx=0 tx=0).stop 2024-04-16T16:19:12.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.157+0000 7f601b200640 1 -- 172.21.15.45:0/2295589579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6014161e90 msgr2=0x7f601414da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:12.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.157+0000 7f601b200640 1 --2- 172.21.15.45:0/2295589579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6014161e90 0x7f601414da90 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f600402ba50 tx=0x7f6004004730 comp rx=0 tx=0).stop 2024-04-16T16:19:12.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.158+0000 7f601b200640 1 -- 172.21.15.45:0/2295589579 shutdown_connections 2024-04-16T16:19:12.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.158+0000 7f601b200640 1 --2- 172.21.15.45:0/2295589579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6014162270 0x7f60140cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.158+0000 7f601b200640 1 --2- 172.21.15.45:0/2295589579 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f5ff806c980 0x7f5ff806ee40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.158+0000 7f601b200640 1 --2- 172.21.15.45:0/2295589579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6014161e90 0x7f601414da90 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.158+0000 7f601b200640 1 --2- 172.21.15.45:0/2295589579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60140d52d0 0x7f601415c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.158+0000 7f601b200640 1 -- 172.21.15.45:0/2295589579 >> 172.21.15.45:0/2295589579 conn(0x7f60140d1940 msgr2=0x7f60140b5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:12.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.158+0000 7f601b200640 1 -- 172.21.15.45:0/2295589579 shutdown_connections 2024-04-16T16:19:12.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.158+0000 7f601b200640 1 -- 172.21.15.45:0/2295589579 wait complete. 2024-04-16T16:19:12.579 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:19:05.457389+0000', 'modified': '2024-04-16T16:19:08.527316+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7677}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7677': {'gid': 7677, 'name': 'c', 'rank': 0, 'incarnation': 69, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6834/882883484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6833', 'nonce': 882883484}, {'type': 'v1', 'addr': '172.21.15.45:6834', 'nonce': 882883484}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7677, 'qdb_cluster': [7677]} 2024-04-16T16:19:12.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-16T16:19:12.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-16T16:19:12.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.811+0000 7f7f97200640 1 Processor -- start 2024-04-16T16:19:12.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.811+0000 7f7f97200640 1 -- start start 2024-04-16T16:19:12.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.813+0000 7f7f97200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7f980c5670 0x7f7f980c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.813+0000 7f7f97200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f9815d3b0 con 0x7f7f9815bc80 2024-04-16T16:19:12.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.813+0000 7f7f97200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f9815d690 con 0x7f7f98145230 2024-04-16T16:19:12.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.813+0000 7f7f97200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7f9815dfc0 con 0x7f7f980c5670 2024-04-16T16:19:12.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.813+0000 7f7f8fe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7f9815bc80 0x7f7f980cd0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37902/0 (socket says 172.21.15.45:37902) 2024-04-16T16:19:12.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.813+0000 7f7f8fe00640 1 -- 172.21.15.45:0/569432931 learned_addr learned my addr 172.21.15.45:0/569432931 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:12.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.813+0000 7f7f96800640 1 --2- 172.21.15.45:0/569432931 >> v2:172.21.15.103:3300/0 conn(0x7f7f980c5670 0x7f7f980c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.813+0000 7f7f95400640 1 -- 172.21.15.45:0/569432931 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1376532122 0 0) 0x7f7f9815d3b0 con 0x7f7f9815bc80 2024-04-16T16:19:12.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.814+0000 7f7f95400640 1 -- 172.21.15.45:0/569432931 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f7c003520 con 0x7f7f9815bc80 2024-04-16T16:19:12.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.814+0000 7f7f95400640 1 -- 172.21.15.45:0/569432931 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3325142056 0 0) 0x7f7f7c003520 con 0x7f7f9815bc80 2024-04-16T16:19:12.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.814+0000 7f7f95400640 1 -- 172.21.15.45:0/569432931 >> v2:172.21.15.103:3300/0 conn(0x7f7f980c5670 msgr2=0x7f7f980c5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:12.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.814+0000 7f7f95400640 1 --2- 172.21.15.45:0/569432931 >> v2:172.21.15.103:3300/0 conn(0x7f7f980c5670 0x7f7f980c5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.814+0000 7f7f95400640 1 -- 172.21.15.45:0/569432931 >> v1:172.21.15.103:6789/0 conn(0x7f7f98145230 legacy=0x7f7f9815cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:12.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.814+0000 7f7f95400640 1 -- 172.21.15.45:0/569432931 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9815d970 con 0x7f7f9815bc80 2024-04-16T16:19:12.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.814+0000 7f7f95400640 1 -- 172.21.15.45:0/569432931 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f7f80001670 con 0x7f7f9815bc80 2024-04-16T16:19:12.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.814+0000 7f7f95400640 1 -- 172.21.15.45:0/569432931 >> v1:172.21.15.45:6789/0 conn(0x7f7f9815bc80 legacy=0x7f7f980cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:12.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.814+0000 7f7f95400640 1 --2- 172.21.15.45:0/569432931 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f7c003a00 0x7f7f7c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.814+0000 7f7f95400640 1 --2- 172.21.15.45:0/569432931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f7c004800 0x7f7f7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.814+0000 7f7f95400640 1 --2- 172.21.15.45:0/569432931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f7c005230 0x7f7f7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.814+0000 7f7f95400640 1 -- 172.21.15.45:0/569432931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9815dfc0 con 0x7f7f7c004800 2024-04-16T16:19:12.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.814+0000 7f7f95400640 1 -- 172.21.15.45:0/569432931 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c00c190 con 0x7f7f7c005230 2024-04-16T16:19:12.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.814+0000 7f7f95400640 1 -- 172.21.15.45:0/569432931 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7f7c00c390 con 0x7f7f7c003a00 2024-04-16T16:19:12.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.815+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/569432931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f7c004800 0x7f7f7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.815+0000 7f7f96800640 1 --2- 172.21.15.45:0/569432931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f7c005230 0x7f7f7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.815+0000 7f7f95e00640 1 --2- 172.21.15.45:0/569432931 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f7c003a00 0x7f7f7c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.815+0000 7f7f8fe00640 1 -- 172.21.15.45:0/569432931 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f7c003a00 msgr2=0x7f7f7c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:12.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.815+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/569432931 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f7c003a00 0x7f7f7c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.815+0000 7f7f8fe00640 1 -- 172.21.15.45:0/569432931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f7c005230 msgr2=0x7f7f7c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:12.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.815+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/569432931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f7c005230 0x7f7f7c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.815+0000 7f7f8fe00640 1 -- 172.21.15.45:0/569432931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7c003520 con 0x7f7f7c004800 2024-04-16T16:19:12.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.815+0000 7f7f96800640 1 --2- 172.21.15.45:0/569432931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f7c005230 0x7f7f7c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:19:12.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.815+0000 7f7f96800640 1 -- 172.21.15.45:0/569432931 reap_dead start 2024-04-16T16:19:12.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.816+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/569432931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f7c004800 0x7f7f7c004c80 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000c0e0 tx=0x7f7f7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62682ac8400262ff server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:12.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.817+0000 7f7f95400640 1 -- 172.21.15.45:0/569432931 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f80007c20 con 0x7f7f7c004800 2024-04-16T16:19:12.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.817+0000 7f7f95400640 1 -- 172.21.15.45:0/569432931 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f80030070 con 0x7f7f7c004800 2024-04-16T16:19:12.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.817+0000 7f7f95400640 1 -- 172.21.15.45:0/569432931 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f80005d70 con 0x7f7f7c004800 2024-04-16T16:19:12.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.817+0000 7f7f97200640 1 -- 172.21.15.45:0/569432931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f7c004800 msgr2=0x7f7f7c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:12.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.817+0000 7f7f97200640 1 --2- 172.21.15.45:0/569432931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f7c004800 0x7f7f7c004c80 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000c0e0 tx=0x7f7f7c005830 comp rx=0 tx=0).stop 2024-04-16T16:19:12.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.818+0000 7f7f97200640 1 -- 172.21.15.45:0/569432931 shutdown_connections 2024-04-16T16:19:12.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.818+0000 7f7f97200640 1 --2- 172.21.15.45:0/569432931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f7c004800 0x7f7f7c004c80 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.818+0000 7f7f97200640 1 -- 172.21.15.45:0/569432931 >> 172.21.15.45:0/569432931 conn(0x7f7f9806b930 msgr2=0x7f7f9806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:12.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.818+0000 7f7f97200640 1 -- 172.21.15.45:0/569432931 shutdown_connections 2024-04-16T16:19:12.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.818+0000 7f7f97200640 1 -- 172.21.15.45:0/569432931 wait complete. 2024-04-16T16:19:12.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.819+0000 7f7f97200640 1 Processor -- start 2024-04-16T16:19:12.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.820+0000 7f7f97200640 1 -- start start 2024-04-16T16:19:12.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.821+0000 7f7f97200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f7c004800 0x7f7f981f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.821+0000 7f7f97200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f98138300 0x7f7f98139f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.821+0000 7f7f97200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f980cbf10 0x7f7f9815acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.821+0000 7f7f97200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7f980c5c20 con 0x7f7f980cbf10 2024-04-16T16:19:12.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.821+0000 7f7f97200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f980c5aa0 con 0x7f7f7c004800 2024-04-16T16:19:12.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.821+0000 7f7f97200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7f980c5da0 con 0x7f7f98138300 2024-04-16T16:19:12.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.821+0000 7f7f96800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f980cbf10 0x7f7f9815acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.821+0000 7f7f96800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f980cbf10 0x7f7f9815acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44768/0 (socket says 172.21.15.45:44768) 2024-04-16T16:19:12.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.821+0000 7f7f96800640 1 -- 172.21.15.45:0/1638712984 learned_addr learned my addr 172.21.15.45:0/1638712984 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:12.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.821+0000 7f7f96800640 1 -- 172.21.15.45:0/1638712984 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f98138300 msgr2=0x7f7f98139f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:12.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.821+0000 7f7f95e00640 1 --2- 172.21.15.45:0/1638712984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f7c004800 0x7f7f981f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.821+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/1638712984 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f98138300 0x7f7f98139f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.821+0000 7f7f96800640 1 --2- 172.21.15.45:0/1638712984 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f98138300 0x7f7f98139f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.821+0000 7f7f96800640 1 -- 172.21.15.45:0/1638712984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f7c004800 msgr2=0x7f7f981f51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:12.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.821+0000 7f7f96800640 1 --2- 172.21.15.45:0/1638712984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f7c004800 0x7f7f981f51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.822+0000 7f7f96800640 1 -- 172.21.15.45:0/1638712984 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9813a840 con 0x7f7f980cbf10 2024-04-16T16:19:12.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.822+0000 7f7f95e00640 1 --2- 172.21.15.45:0/1638712984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f7c004800 0x7f7f981f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:12.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.822+0000 7f7f96800640 1 --2- 172.21.15.45:0/1638712984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f980cbf10 0x7f7f9815acd0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f7f98073730 tx=0x7f7f90010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:12.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.822+0000 7f7f8e000640 1 -- 172.21.15.45:0/1638712984 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90010e00 con 0x7f7f980cbf10 2024-04-16T16:19:12.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.822+0000 7f7f8e000640 1 -- 172.21.15.45:0/1638712984 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f90003990 con 0x7f7f980cbf10 2024-04-16T16:19:12.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.822+0000 7f7f8e000640 1 -- 172.21.15.45:0/1638712984 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90003d80 con 0x7f7f980cbf10 2024-04-16T16:19:12.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.822+0000 7f7f97200640 1 -- 172.21.15.45:0/1638712984 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f9813ab20 con 0x7f7f980cbf10 2024-04-16T16:19:12.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.822+0000 7f7f97200640 1 -- 172.21.15.45:0/1638712984 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f981f4c80 con 0x7f7f980cbf10 2024-04-16T16:19:12.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.825+0000 7f7f8e000640 1 -- 172.21.15.45:0/1638712984 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 33) ==== 91142+0+0 (secure 0 0 0) 0x7f7f9001e020 con 0x7f7f980cbf10 2024-04-16T16:19:12.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.826+0000 7f7f8e000640 1 --2- 172.21.15.45:0/1638712984 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f7f7406ca40 0x7f7f7406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.826+0000 7f7f8e000640 1 -- 172.21.15.45:0/1638712984 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5752+0+0 (secure 0 0 0) 0x7f7f9008c600 con 0x7f7f980cbf10 2024-04-16T16:19:12.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.826+0000 7f7f9fffe740 1 Processor -- start 2024-04-16T16:19:12.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.826+0000 7f7f9fffe740 1 -- start start 2024-04-16T16:19:12.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.827+0000 7f7f95e00640 1 --2- 172.21.15.45:0/1638712984 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f7f7406ca40 0x7f7f7406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.827+0000 7f7f9fffe740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097a0890 0x5566097bf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.827+0000 7f7f9fffe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0c70 0x556609779570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.827+0000 7f7f9fffe740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556609779ab0 0x556609777440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.827+0000 7f7f9fffe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5566096de9c0 con 0x5566097a0c70 2024-04-16T16:19:12.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.827+0000 7f7f9fffe740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556609685560 con 0x556609779ab0 2024-04-16T16:19:12.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.827+0000 7f7f9fffe740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5566096da980 con 0x5566097a0890 2024-04-16T16:19:12.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.828+0000 7f7f8fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0c70 0x556609779570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.828+0000 7f7f8fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0c70 0x556609779570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44782/0 (socket says 172.21.15.45:44782) 2024-04-16T16:19:12.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.828+0000 7f7f8fe00640 1 -- 172.21.15.45:0/1218967594 learned_addr learned my addr 172.21.15.45:0/1218967594 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:12.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.828+0000 7f7f96800640 1 --2- 172.21.15.45:0/1218967594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556609779ab0 0x556609777440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.828+0000 7f7f95e00640 1 --2- 172.21.15.45:0/1218967594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097a0890 0x5566097bf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.828+0000 7f7f95e00640 1 --2- 172.21.15.45:0/1638712984 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f7f7406ca40 0x7f7f7406ef00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7f88001700 tx=0x7f7f88009040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:12.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.828+0000 7f7f8fe00640 1 -- 172.21.15.45:0/1218967594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097a0890 msgr2=0x5566097bf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:12.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.828+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/1218967594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097a0890 0x5566097bf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.828+0000 7f7f8fe00640 1 -- 172.21.15.45:0/1218967594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556609779ab0 msgr2=0x556609777440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:12.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.828+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/1218967594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556609779ab0 0x556609777440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.828+0000 7f7f8fe00640 1 -- 172.21.15.45:0/1218967594 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566093f12d0 con 0x5566097a0c70 2024-04-16T16:19:12.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.828+0000 7f7f95e00640 1 --2- 172.21.15.45:0/1218967594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097a0890 0x5566097bf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:12.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.828+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/1218967594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0c70 0x556609779570 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000c200 tx=0x7f7f80007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddce262f9b250de2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.829+0000 7f7f86a00640 1 -- 172.21.15.45:0/1218967594 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f800047e0 con 0x5566097a0c70 2024-04-16T16:19:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.829+0000 7f7f86a00640 1 -- 172.21.15.45:0/1218967594 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f80030040 con 0x5566097a0c70 2024-04-16T16:19:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.829+0000 7f7f86a00640 1 -- 172.21.15.45:0/1218967594 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f80004b80 con 0x5566097a0c70 2024-04-16T16:19:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.829+0000 7f7f9fffe740 1 -- 172.21.15.45:0/1218967594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0c70 msgr2=0x556609779570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.829+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/1218967594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0c70 0x556609779570 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000c200 tx=0x7f7f80007000 comp rx=0 tx=0).stop 2024-04-16T16:19:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.829+0000 7f7f9fffe740 1 -- 172.21.15.45:0/1218967594 shutdown_connections 2024-04-16T16:19:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.829+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/1218967594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556609779ab0 0x556609777440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.829+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/1218967594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0c70 0x556609779570 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.829+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/1218967594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097a0890 0x5566097bf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.829+0000 7f7f9fffe740 1 -- 172.21.15.45:0/1218967594 >> 172.21.15.45:0/1218967594 conn(0x556609775b50 msgr2=0x5566097be6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.830+0000 7f7f9fffe740 1 -- 172.21.15.45:0/1218967594 shutdown_connections 2024-04-16T16:19:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.830+0000 7f7f9fffe740 1 -- 172.21.15.45:0/1218967594 wait complete. 2024-04-16T16:19:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.830+0000 7f7f9fffe740 1 -- start start 2024-04-16T16:19:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.831+0000 7f7f9fffe740 1 Processor -- start 2024-04-16T16:19:12.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.831+0000 7f7f9fffe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0890 0x5566097d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.831+0000 7f7f9fffe740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566097a0c70 0x5566097bbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.831+0000 7f7f9fffe740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097bc210 0x5566097ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.832+0000 7f7f9fffe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x556609773980 con 0x5566097a0890 2024-04-16T16:19:12.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.832+0000 7f7f9fffe740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556609698920 con 0x5566097a0c70 2024-04-16T16:19:12.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.832+0000 7f7f9fffe740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x556609779f30 con 0x5566097bc210 2024-04-16T16:19:12.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.832+0000 7f7f9fffe740 10 client.0 _ll_register_callbacks cb 0x556609698310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:19:12.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.832+0000 7f7f95e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0890 0x5566097d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.832+0000 7f7f95e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0890 0x5566097d1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44786/0 (socket says 172.21.15.45:44786) 2024-04-16T16:19:12.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.832+0000 7f7f95e00640 1 -- 172.21.15.45:0/2787467942 learned_addr learned my addr 172.21.15.45:0/2787467942 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:12.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.832+0000 7f7f96800640 1 --2- 172.21.15.45:0/2787467942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097bc210 0x5566097ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.832+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/2787467942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566097a0c70 0x5566097bbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.832+0000 7f7f95e00640 1 -- 172.21.15.45:0/2787467942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097bc210 msgr2=0x5566097ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:12.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.832+0000 7f7f95e00640 1 --2- 172.21.15.45:0/2787467942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097bc210 0x5566097ce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.832+0000 7f7f95e00640 1 -- 172.21.15.45:0/2787467942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566097a0c70 msgr2=0x5566097bbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:12.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.832+0000 7f7f95e00640 1 --2- 172.21.15.45:0/2787467942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566097a0c70 0x5566097bbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.832+0000 7f7f95e00640 1 -- 172.21.15.45:0/2787467942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566094bb6b0 con 0x5566097a0890 2024-04-16T16:19:12.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.833+0000 7f7f95e00640 1 --2- 172.21.15.45:0/2787467942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0890 0x5566097d1650 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f7f880156f0 tx=0x7f7f88001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e9e602325289f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:12.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.833+0000 7f7f7b400640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:19:12.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.833+0000 7f7f7b400640 1 -- 172.21.15.45:0/2787467942 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f88026060 con 0x5566097a0890 2024-04-16T16:19:12.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.833+0000 7f7f7b400640 1 -- 172.21.15.45:0/2787467942 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 5752+0+0 (secure 0 0 0) 0x7f7f88014790 con 0x5566097a0890 2024-04-16T16:19:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.833+0000 7f7f9fffe740 1 -- 172.21.15.45:0/2787467942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f80030040 con 0x5566097a0890 2024-04-16T16:19:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.833+0000 7f7f7b400640 1 -- 172.21.15.45:0/2787467942 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f7f88014990 con 0x5566097a0890 2024-04-16T16:19:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.833+0000 7f7f9fffe740 10 client.7812 fetch_fsmap learned FSMap version 73 2024-04-16T16:19:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.833+0000 7f7f9fffe740 1 -- 172.21.15.45:0/2787467942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) -- 0x5566097cf870 con 0x5566097a0890 2024-04-16T16:19:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.834+0000 7f7f7b400640 1 -- 172.21.15.45:0/2787467942 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f80030040 con 0x5566097a0890 2024-04-16T16:19:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.834+0000 7f7f7b400640 1 -- 172.21.15.45:0/2787467942 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 73) ==== 3494+0+0 (secure 0 0 0) 0x7f7f88014b30 con 0x5566097a0890 2024-04-16T16:19:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.834+0000 7f7f9fffe740 10 client.7812 fetch_fsmap finished waiting for FSMap version 73 2024-04-16T16:19:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.834+0000 7f7f9fffe740 10 client.7812 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-16T16:19:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.834+0000 7f7f9fffe740 20 client.7812 populate_metadata read hostname 'smithi045' 2024-04-16T16:19:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.834+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/2787467942 >> [v2:172.21.15.45:6833/882883484,v1:172.21.15.45:6834/882883484] conn(0x5566097da800 0x5566097dcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.834+0000 7f7f9fffe740 4 client.7812 mds_command: new command op to 7677 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-16T16:19:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.834+0000 7f7f9fffe740 1 -- 172.21.15.45:0/2787467942 --> [v2:172.21.15.45:6833/882883484,v1:172.21.15.45:6834/882883484] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556609684b70 con 0x5566097da800 2024-04-16T16:19:12.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.834+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/2787467942 >> [v2:172.21.15.45:6833/882883484,v1:172.21.15.45:6834/882883484] conn(0x5566097da800 0x5566097dcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.835+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/2787467942 >> [v2:172.21.15.45:6833/882883484,v1:172.21.15.45:6834/882883484] conn(0x5566097da800 0x5566097dcbf0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ce5e3d0fe0a39e5 server_cookie=d122bc413a6314cc in_seq=0 out_seq=0 2024-04-16T16:19:12.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.835+0000 7f7f7b400640 10 client.7812 ms_handle_connect on v2:172.21.15.45:6833/882883484 2024-04-16T16:19:12.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.836+0000 7f7f7b400640 1 -- 172.21.15.45:0/2787467942 <== mds.0 v2:172.21.15.45:6833/882883484 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556609684b70 con 0x5566097da800 2024-04-16T16:19:12.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.836+0000 7f7f7b400640 10 client.7812 handle_command_reply: tid=0 2024-04-16T16:19:12.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.836+0000 7f7f9fffe740 1 client.7812 shutdown 2024-04-16T16:19:12.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.836+0000 7f7f9fffe740 1 -- 172.21.15.45:0/2787467942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0890 msgr2=0x5566097d1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:12.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.836+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/2787467942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0890 0x5566097d1650 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f7f880156f0 tx=0x7f7f88001190 comp rx=0 tx=0).stop 2024-04-16T16:19:12.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.836+0000 7f7f9fffe740 1 -- 172.21.15.45:0/2787467942 shutdown_connections 2024-04-16T16:19:12.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.836+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/2787467942 >> [v2:172.21.15.45:6833/882883484,v1:172.21.15.45:6834/882883484] conn(0x5566097da800 0x5566097dcbf0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.836+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/2787467942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097bc210 0x5566097ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.836+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/2787467942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566097a0c70 0x5566097bbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.836+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/2787467942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0890 0x5566097d1650 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.836+0000 7f7f7b400640 0 client.7812 ms_handle_reset on v2:172.21.15.45:6833/882883484 2024-04-16T16:19:12.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.836+0000 7f7f9fffe740 1 -- 172.21.15.45:0/2787467942 >> 172.21.15.45:0/2787467942 conn(0x556609779ab0 msgr2=0x55660979e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:12.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.836+0000 7f7f9fffe740 1 -- 172.21.15.45:0/2787467942 shutdown_connections 2024-04-16T16:19:12.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.836+0000 7f7f9fffe740 1 -- 172.21.15.45:0/2787467942 wait complete. 2024-04-16T16:19:12.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.837+0000 7f7f9fffe740 20 client.7812 trim_cache size 0 max 16384 2024-04-16T16:19:12.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.857+0000 7f7f9fffe740 1 Processor -- start 2024-04-16T16:19:12.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.857+0000 7f7f9fffe740 1 -- start start 2024-04-16T16:19:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.858+0000 7f7f9fffe740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566097a0890 0x556609777630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.858+0000 7f7f9fffe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0c70 0x5566097e1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.858+0000 7f7f9fffe740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097bc210 0x5566097e2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.858+0000 7f7f9fffe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7f982006a0 con 0x5566097a0c70 2024-04-16T16:19:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.858+0000 7f7f9fffe740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f981f8750 con 0x5566097a0890 2024-04-16T16:19:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.858+0000 7f7f9fffe740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7f98200160 con 0x5566097bc210 2024-04-16T16:19:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.858+0000 7f7f8fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0c70 0x5566097e1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.858+0000 7f7f8fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0c70 0x5566097e1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44802/0 (socket says 172.21.15.45:44802) 2024-04-16T16:19:12.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.858+0000 7f7f8fe00640 1 -- 172.21.15.45:0/455827381 learned_addr learned my addr 172.21.15.45:0/455827381 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:12.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.859+0000 7f7f96800640 1 --2- 172.21.15.45:0/455827381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097bc210 0x5566097e2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.859+0000 7f7f95e00640 1 --2- 172.21.15.45:0/455827381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566097a0890 0x556609777630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.859+0000 7f7f8fe00640 1 -- 172.21.15.45:0/455827381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097bc210 msgr2=0x5566097e2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:12.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.859+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/455827381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097bc210 0x5566097e2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.859+0000 7f7f8fe00640 1 -- 172.21.15.45:0/455827381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566097a0890 msgr2=0x556609777630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:12.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.859+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/455827381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566097a0890 0x556609777630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.859+0000 7f7f8fe00640 1 -- 172.21.15.45:0/455827381 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566097c26e0 con 0x5566097a0c70 2024-04-16T16:19:12.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.859+0000 7f7f96800640 1 --2- 172.21.15.45:0/455827381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097bc210 0x5566097e2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:12.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.859+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/455827381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0c70 0x5566097e1e50 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f7f80043040 tx=0x7f7f8002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef2a3c97bf5d2eb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:12.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.859+0000 7f7f86000640 1 -- 172.21.15.45:0/455827381 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f800047e0 con 0x5566097a0c70 2024-04-16T16:19:12.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.859+0000 7f7f86000640 1 -- 172.21.15.45:0/455827381 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f80030070 con 0x5566097a0c70 2024-04-16T16:19:12.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.859+0000 7f7f86000640 1 -- 172.21.15.45:0/455827381 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f80004b80 con 0x5566097a0c70 2024-04-16T16:19:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.860+0000 7f7f9fffe740 1 -- 172.21.15.45:0/455827381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0c70 msgr2=0x5566097e1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.860+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/455827381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0c70 0x5566097e1e50 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f7f80043040 tx=0x7f7f8002e000 comp rx=0 tx=0).stop 2024-04-16T16:19:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.860+0000 7f7f9fffe740 1 -- 172.21.15.45:0/455827381 shutdown_connections 2024-04-16T16:19:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.860+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/455827381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097bc210 0x5566097e2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.860+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/455827381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0c70 0x5566097e1e50 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.860+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/455827381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566097a0890 0x556609777630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.860+0000 7f7f9fffe740 1 -- 172.21.15.45:0/455827381 >> 172.21.15.45:0/455827381 conn(0x556609779ab0 msgr2=0x55660977a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.860+0000 7f7f9fffe740 1 -- 172.21.15.45:0/455827381 shutdown_connections 2024-04-16T16:19:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.860+0000 7f7f9fffe740 1 -- 172.21.15.45:0/455827381 wait complete. 2024-04-16T16:19:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.861+0000 7f7f9fffe740 1 -- start start 2024-04-16T16:19:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.861+0000 7f7f9fffe740 1 Processor -- start 2024-04-16T16:19:12.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.862+0000 7f7f9fffe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0890 0x5566097ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.862+0000 7f7f9fffe740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097a0c70 0x5566097bf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.862+0000 7f7f9fffe740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566097bc210 0x5566097bfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.862+0000 7f7f9fffe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5566097bac90 con 0x5566097a0890 2024-04-16T16:19:12.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.862+0000 7f7f9fffe740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5566097ba890 con 0x5566097bc210 2024-04-16T16:19:12.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.862+0000 7f7f9fffe740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5566097cf590 con 0x5566097a0c70 2024-04-16T16:19:12.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.862+0000 7f7f9fffe740 10 client.0 _ll_register_callbacks cb 0x7f7f880013d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:19:12.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.862+0000 7f7f95e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0890 0x5566097ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.862+0000 7f7f95e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0890 0x5566097ce2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44812/0 (socket says 172.21.15.45:44812) 2024-04-16T16:19:12.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.862+0000 7f7f95e00640 1 -- 172.21.15.45:0/3237006594 learned_addr learned my addr 172.21.15.45:0/3237006594 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:12.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.863+0000 7f7f95e00640 1 -- 172.21.15.45:0/3237006594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097a0c70 msgr2=0x5566097bf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:12.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.863+0000 7f7f95e00640 1 --2- 172.21.15.45:0/3237006594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097a0c70 0x5566097bf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.863+0000 7f7f95e00640 1 -- 172.21.15.45:0/3237006594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566097bc210 msgr2=0x5566097bfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:12.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.863+0000 7f7f95e00640 1 --2- 172.21.15.45:0/3237006594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566097bc210 0x5566097bfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.863+0000 7f7f95e00640 1 -- 172.21.15.45:0/3237006594 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566097beba0 con 0x5566097a0890 2024-04-16T16:19:12.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.863+0000 7f7f95e00640 1 --2- 172.21.15.45:0/3237006594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0890 0x5566097ce2e0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f7f88000c00 tx=0x7f7f88016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe290ad8786dc2ec server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:12.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.863+0000 7f7f7b400640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:19:12.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.863+0000 7f7f7b400640 1 -- 172.21.15.45:0/3237006594 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f88036060 con 0x5566097a0890 2024-04-16T16:19:12.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.863+0000 7f7f7b400640 1 -- 172.21.15.45:0/3237006594 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 5752+0+0 (secure 0 0 0) 0x7f7f88007240 con 0x5566097a0890 2024-04-16T16:19:12.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.863+0000 7f7f9fffe740 1 -- 172.21.15.45:0/3237006594 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f80030070 con 0x5566097a0890 2024-04-16T16:19:12.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.864+0000 7f7f7b400640 1 -- 172.21.15.45:0/3237006594 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f7f8803c050 con 0x5566097a0890 2024-04-16T16:19:12.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.864+0000 7f7f9fffe740 10 client.7818 fetch_fsmap learned FSMap version 73 2024-04-16T16:19:12.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.864+0000 7f7f9fffe740 1 -- 172.21.15.45:0/3237006594 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) -- 0x5566097ce8b0 con 0x5566097a0890 2024-04-16T16:19:12.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.864+0000 7f7f7b400640 1 -- 172.21.15.45:0/3237006594 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f80030070 con 0x5566097a0890 2024-04-16T16:19:12.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.864+0000 7f7f7b400640 1 -- 172.21.15.45:0/3237006594 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 73) ==== 3494+0+0 (secure 0 0 0) 0x7f7f8803f020 con 0x5566097a0890 2024-04-16T16:19:12.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.864+0000 7f7f9fffe740 10 client.7818 fetch_fsmap finished waiting for FSMap version 73 2024-04-16T16:19:12.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.864+0000 7f7f9fffe740 10 client.7818 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-16T16:19:12.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.864+0000 7f7f9fffe740 20 client.7818 populate_metadata read hostname 'smithi045' 2024-04-16T16:19:12.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.864+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/3237006594 >> [v2:172.21.15.45:6833/882883484,v1:172.21.15.45:6834/882883484] conn(0x5566097baf20 0x5566097bb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:12.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.864+0000 7f7f9fffe740 4 client.7818 mds_command: new command op to 7677 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-16T16:19:12.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.864+0000 7f7f9fffe740 1 -- 172.21.15.45:0/3237006594 --> [v2:172.21.15.45:6833/882883484,v1:172.21.15.45:6834/882883484] -- command(tid 0: {"prefix": "status"}) -- 0x5566096ec3e0 con 0x5566097baf20 2024-04-16T16:19:12.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.864+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/3237006594 >> [v2:172.21.15.45:6833/882883484,v1:172.21.15.45:6834/882883484] conn(0x5566097baf20 0x5566097bb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:12.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.865+0000 7f7f8fe00640 1 --2- 172.21.15.45:0/3237006594 >> [v2:172.21.15.45:6833/882883484,v1:172.21.15.45:6834/882883484] conn(0x5566097baf20 0x5566097bb300 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9548af8958d87eeb server_cookie=9100c6994fdc5527 in_seq=0 out_seq=0 2024-04-16T16:19:12.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.865+0000 7f7f7b400640 10 client.7818 ms_handle_connect on v2:172.21.15.45:6833/882883484 2024-04-16T16:19:12.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.865+0000 7f7f7b400640 1 -- 172.21.15.45:0/3237006594 <== mds.0 v2:172.21.15.45:6833/882883484 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5566096ec3e0 con 0x5566097baf20 2024-04-16T16:19:12.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.865+0000 7f7f7b400640 10 client.7818 handle_command_reply: tid=0 2024-04-16T16:19:12.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.865+0000 7f7f9fffe740 1 client.7818 shutdown 2024-04-16T16:19:12.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.865+0000 7f7f9fffe740 1 -- 172.21.15.45:0/3237006594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0890 msgr2=0x5566097ce2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:12.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.865+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/3237006594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0890 0x5566097ce2e0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f7f88000c00 tx=0x7f7f88016000 comp rx=0 tx=0).stop 2024-04-16T16:19:12.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.865+0000 7f7f9fffe740 1 -- 172.21.15.45:0/3237006594 shutdown_connections 2024-04-16T16:19:12.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.865+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/3237006594 >> [v2:172.21.15.45:6833/882883484,v1:172.21.15.45:6834/882883484] conn(0x5566097baf20 0x5566097bb300 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.865+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/3237006594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566097bc210 0x5566097bfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.865+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/3237006594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5566097a0c70 0x5566097bf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.865+0000 7f7f9fffe740 1 --2- 172.21.15.45:0/3237006594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566097a0890 0x5566097ce2e0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.865+0000 7f7f7b400640 0 client.7818 ms_handle_reset on v2:172.21.15.45:6833/882883484 2024-04-16T16:19:12.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.865+0000 7f7f9fffe740 1 -- 172.21.15.45:0/3237006594 >> 172.21.15.45:0/3237006594 conn(0x556609779ab0 msgr2=0x5566097bdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:12.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.866+0000 7f7f9fffe740 1 -- 172.21.15.45:0/3237006594 shutdown_connections 2024-04-16T16:19:12.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.866+0000 7f7f9fffe740 1 -- 172.21.15.45:0/3237006594 wait complete. 2024-04-16T16:19:12.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.866+0000 7f7f9fffe740 20 client.7818 trim_cache size 0 max 16384 2024-04-16T16:19:12.871 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:19:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "748c4b85-0190-4c59-80ad-098a6a5007c0", 2024-04-16T16:19:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-04-16T16:19:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 7677, 2024-04-16T16:19:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-04-16T16:19:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-04-16T16:19:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-04-16T16:19:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 6.8950232580000002, 2024-04-16T16:19:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 73, 2024-04-16T16:19:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 100, 2024-04-16T16:19:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 99, 2024-04-16T16:19:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 13.197297033 2024-04-16T16:19:12.872 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:19:12.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.866+0000 7f7f97200640 1 -- 172.21.15.45:0/1638712984 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f7f7406ca40 msgr2=0x7f7f7406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:12.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.866+0000 7f7f97200640 1 --2- 172.21.15.45:0/1638712984 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f7f7406ca40 0x7f7f7406ef00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7f88001700 tx=0x7f7f88009040 comp rx=0 tx=0).stop 2024-04-16T16:19:12.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.866+0000 7f7f97200640 1 -- 172.21.15.45:0/1638712984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f980cbf10 msgr2=0x7f7f9815acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.866+0000 7f7f97200640 1 --2- 172.21.15.45:0/1638712984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f980cbf10 0x7f7f9815acd0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f7f98073730 tx=0x7f7f90010590 comp rx=0 tx=0).stop 2024-04-16T16:19:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.866+0000 7f7f97200640 1 -- 172.21.15.45:0/1638712984 shutdown_connections 2024-04-16T16:19:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.866+0000 7f7f97200640 1 --2- 172.21.15.45:0/1638712984 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f7f7406ca40 0x7f7f7406ef00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.866+0000 7f7f97200640 1 --2- 172.21.15.45:0/1638712984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f980cbf10 0x7f7f9815acd0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.866+0000 7f7f97200640 1 --2- 172.21.15.45:0/1638712984 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f98138300 0x7f7f98139f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.866+0000 7f7f97200640 1 --2- 172.21.15.45:0/1638712984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f7c004800 0x7f7f981f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.866+0000 7f7f97200640 1 -- 172.21.15.45:0/1638712984 >> 172.21.15.45:0/1638712984 conn(0x7f7f9806b930 msgr2=0x7f7f980c6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.867+0000 7f7f97200640 1 -- 172.21.15.45:0/1638712984 shutdown_connections 2024-04-16T16:19:12.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:12.867+0000 7f7f97200640 1 -- 172.21.15.45:0/1638712984 wait complete. 2024-04-16T16:19:12.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:19:13.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.101+0000 7fb24f200640 1 Processor -- start 2024-04-16T16:19:13.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.101+0000 7fb24f200640 1 -- start start 2024-04-16T16:19:13.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.103+0000 7fb24f200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb2481608a0 0x7fb248160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:13.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.103+0000 7fb24f200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb24815d360 con 0x7fb248146220 2024-04-16T16:19:13.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.103+0000 7fb24f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb24815d640 con 0x7fb24815ccb0 2024-04-16T16:19:13.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.103+0000 7fb24f200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb24815d130 con 0x7fb2481608a0 2024-04-16T16:19:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.103+0000 7fb24e800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb2481608a0 0x7fb248160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.103+0000 7fb24e800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb2481608a0 0x7fb248160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44824/0 (socket says 172.21.15.45:44824) 2024-04-16T16:19:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.103+0000 7fb24de00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb248146220 0x7fb24806f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37906/0 (socket says 172.21.15.45:37906) 2024-04-16T16:19:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.103+0000 7fb24e800640 1 -- 172.21.15.45:0/1397925824 learned_addr learned my addr 172.21.15.45:0/1397925824 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.104+0000 7fb24e800640 1 -- 172.21.15.45:0/1397925824 >> v1:172.21.15.103:6789/0 conn(0x7fb24815ccb0 legacy=0x7fb24806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.104+0000 7fb24e800640 1 -- 172.21.15.45:0/1397925824 >> v1:172.21.15.45:6789/0 conn(0x7fb248146220 legacy=0x7fb24806f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.104+0000 7fb24e800640 1 -- 172.21.15.45:0/1397925824 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24815d920 con 0x7fb2481608a0 2024-04-16T16:19:13.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.105+0000 7fb24e800640 1 --2- 172.21.15.45:0/1397925824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2481608a0 0x7fb248160cf0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fb244009e50 tx=0x7fb24402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad058222d53245c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:13.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.105+0000 7fb24ca00640 1 -- 172.21.15.45:0/1397925824 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24402cb10 con 0x7fb2481608a0 2024-04-16T16:19:13.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.105+0000 7fb24ca00640 1 -- 172.21.15.45:0/1397925824 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb24402ccb0 con 0x7fb2481608a0 2024-04-16T16:19:13.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.105+0000 7fb24ca00640 1 -- 172.21.15.45:0/1397925824 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24402ce60 con 0x7fb2481608a0 2024-04-16T16:19:13.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.106+0000 7fb24f200640 1 -- 172.21.15.45:0/1397925824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2481608a0 msgr2=0x7fb248160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:13.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.106+0000 7fb24f200640 1 --2- 172.21.15.45:0/1397925824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2481608a0 0x7fb248160cf0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fb244009e50 tx=0x7fb24402bf70 comp rx=0 tx=0).stop 2024-04-16T16:19:13.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.106+0000 7fb24f200640 1 -- 172.21.15.45:0/1397925824 shutdown_connections 2024-04-16T16:19:13.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.106+0000 7fb24f200640 1 --2- 172.21.15.45:0/1397925824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2481608a0 0x7fb248160cf0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:13.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.106+0000 7fb24f200640 1 -- 172.21.15.45:0/1397925824 >> 172.21.15.45:0/1397925824 conn(0x7fb24806b930 msgr2=0x7fb24806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:13.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.106+0000 7fb24f200640 1 -- 172.21.15.45:0/1397925824 shutdown_connections 2024-04-16T16:19:13.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.106+0000 7fb24f200640 1 -- 172.21.15.45:0/1397925824 wait complete. 2024-04-16T16:19:13.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.108+0000 7fb24f200640 1 Processor -- start 2024-04-16T16:19:13.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.108+0000 7fb24f200640 1 -- start start 2024-04-16T16:19:13.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.109+0000 7fb24f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb248146220 0x7fb24815bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:13.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.109+0000 7fb24f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb24815ccb0 0x7fb248144210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:13.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.109+0000 7fb24f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2481608a0 0x7fb24813fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:13.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.109+0000 7fb24f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb248160410 con 0x7fb248146220 2024-04-16T16:19:13.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.109+0000 7fb24f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb24815dc00 con 0x7fb24815ccb0 2024-04-16T16:19:13.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.109+0000 7fb24f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb2480cf360 con 0x7fb2481608a0 2024-04-16T16:19:13.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.109+0000 7fb24de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb248146220 0x7fb24815bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:13.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.109+0000 7fb24de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb248146220 0x7fb24815bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44830/0 (socket says 172.21.15.45:44830) 2024-04-16T16:19:13.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.109+0000 7fb24de00640 1 -- 172.21.15.45:0/2297384037 learned_addr learned my addr 172.21.15.45:0/2297384037 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:13.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.110+0000 7fb24de00640 1 -- 172.21.15.45:0/2297384037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2481608a0 msgr2=0x7fb24813fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:13.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.110+0000 7fb24de00640 1 --2- 172.21.15.45:0/2297384037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2481608a0 0x7fb24813fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:13.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.110+0000 7fb24de00640 1 -- 172.21.15.45:0/2297384037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb24815ccb0 msgr2=0x7fb248144210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:13.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.110+0000 7fb24de00640 1 --2- 172.21.15.45:0/2297384037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb24815ccb0 0x7fb248144210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:13.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.110+0000 7fb24de00640 1 -- 172.21.15.45:0/2297384037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb248144c40 con 0x7fb248146220 2024-04-16T16:19:13.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.110+0000 7fb24de00640 1 --2- 172.21.15.45:0/2297384037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb248146220 0x7fb24815bf10 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00c400 tx=0x7fb23c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:13.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.110+0000 7fb242000640 1 -- 172.21.15.45:0/2297384037 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c0074e0 con 0x7fb248146220 2024-04-16T16:19:13.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.110+0000 7fb242000640 1 -- 172.21.15.45:0/2297384037 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb23c007680 con 0x7fb248146220 2024-04-16T16:19:13.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.110+0000 7fb242000640 1 -- 172.21.15.45:0/2297384037 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c004a00 con 0x7fb248146220 2024-04-16T16:19:13.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.110+0000 7fb24f200640 1 -- 172.21.15.45:0/2297384037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb248144f20 con 0x7fb248146220 2024-04-16T16:19:13.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.111+0000 7fb24f200640 1 -- 172.21.15.45:0/2297384037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb248138170 con 0x7fb248146220 2024-04-16T16:19:13.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.112+0000 7fb24f200640 1 -- 172.21.15.45:0/2297384037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb248137c80 con 0x7fb248146220 2024-04-16T16:19:13.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.113+0000 7fb242000640 1 -- 172.21.15.45:0/2297384037 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 33) ==== 91142+0+0 (secure 0 0 0) 0x7fb23c01a020 con 0x7fb248146220 2024-04-16T16:19:13.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.114+0000 7fb242000640 1 --2- 172.21.15.45:0/2297384037 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb22c06ca10 0x7fb22c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:13.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.114+0000 7fb242000640 1 -- 172.21.15.45:0/2297384037 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5752+0+0 (secure 0 0 0) 0x7fb23c08b480 con 0x7fb248146220 2024-04-16T16:19:13.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.114+0000 7fb24d400640 1 --2- 172.21.15.45:0/2297384037 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb22c06ca10 0x7fb22c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:13.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.116+0000 7fb24d400640 1 --2- 172.21.15.45:0/2297384037 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb22c06ca10 0x7fb22c06eed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb24806c100 tx=0x7fb2380079b0 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:13.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.119+0000 7fb242000640 1 -- 172.21.15.45:0/2297384037 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb23c059800 con 0x7fb248146220 2024-04-16T16:19:13.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.373+0000 7fb24f200640 1 -- 172.21.15.45:0/2297384037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb248075780 con 0x7fb248146220 2024-04-16T16:19:13.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.374+0000 7fb242000640 1 -- 172.21.15.45:0/2297384037 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v73) ==== 53+0+73 (secure 0 0 0) 0x7fb23c059100 con 0x7fb248146220 2024-04-16T16:19:13.374 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:19:13.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.376+0000 7fb24f200640 1 -- 172.21.15.45:0/2297384037 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb22c06ca10 msgr2=0x7fb22c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:13.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.376+0000 7fb24f200640 1 --2- 172.21.15.45:0/2297384037 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb22c06ca10 0x7fb22c06eed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb24806c100 tx=0x7fb2380079b0 comp rx=0 tx=0).stop 2024-04-16T16:19:13.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.376+0000 7fb24f200640 1 -- 172.21.15.45:0/2297384037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb248146220 msgr2=0x7fb24815bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:13.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.377+0000 7fb24f200640 1 --2- 172.21.15.45:0/2297384037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb248146220 0x7fb24815bf10 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00c400 tx=0x7fb23c00c8c0 comp rx=0 tx=0).stop 2024-04-16T16:19:13.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.377+0000 7fb24f200640 1 -- 172.21.15.45:0/2297384037 shutdown_connections 2024-04-16T16:19:13.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.377+0000 7fb24f200640 1 --2- 172.21.15.45:0/2297384037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2481608a0 0x7fb24813fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:13.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.377+0000 7fb24f200640 1 --2- 172.21.15.45:0/2297384037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb24815ccb0 0x7fb248144210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:13.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.377+0000 7fb24f200640 1 --2- 172.21.15.45:0/2297384037 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb22c06ca10 0x7fb22c06eed0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:13.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.377+0000 7fb24f200640 1 --2- 172.21.15.45:0/2297384037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb248146220 0x7fb24815bf10 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:13.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.377+0000 7fb24f200640 1 -- 172.21.15.45:0/2297384037 >> 172.21.15.45:0/2297384037 conn(0x7fb24806b930 msgr2=0x7fb2480c6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:13.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.377+0000 7fb24f200640 1 -- 172.21.15.45:0/2297384037 shutdown_connections 2024-04-16T16:19:13.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13.377+0000 7fb24f200640 1 -- 172.21.15.45:0/2297384037 wait complete. 2024-04-16T16:19:13.397 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-16T16:19:13.397 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:19:13.397 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-04-16T16:19:13.397 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-16T16:19:13.397 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T16:19:13.397 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-16T16:19:13.397 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:19:13.397 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:19:13.397 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:19:13.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:13.398 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-04-16T16:19:13.452 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:19:13.452 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:19:13.452 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:19:13.452 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:13.452 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-04-16T16:19:13.452 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:13.452 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:19:13.452 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:13.453 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-04-16T16:19:13.453 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:19:13.453 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:13.453 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-04-16T16:19:13.453 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:19:13.453 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:13.453 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-04-16T16:19:13.453 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-16T16:19:13.453 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42248sec preferred_lft 42248sec 2024-04-16T16:19:13.453 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-04-16T16:19:13.453 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:13.453 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:19:13.453 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:13.453 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-04-16T16:19:13.453 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:19:13.454 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:13.454 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:19:13.454 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:13.454 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::50f1:99ff:fe2b:225b/64 scope link 2024-04-16T16:19:13.454 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:13.454 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:19:13.454 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T16:19:13.454 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::7007:c3ff:fe13:7bcd/64 scope link 2024-04-16T16:19:13.454 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:13.454 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:19:13.454 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether ae:10:ea:fc:0a:13 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-16T16:19:13.454 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ac10:eaff:fefc:a13/64 scope link 2024-04-16T16:19:13.454 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:13.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:13.455 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:19:13.548 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-04-16T16:19:13.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:13.553 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-04-16T16:19:13.610 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-04-16T16:19:13.610 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:19:13.610 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-16T16:19:13.610 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-16T16:19:13.610 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:19:13.611 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:19:13.611 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:19:13.611 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-16T16:19:13.611 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-16T16:19:13.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-16T16:19:13.612 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:19:13.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:19:13.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:19:13.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:13.871 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T16:19:13.922 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-16T16:19:13.922 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-16T16:19:13.923 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:19:13.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:13.923 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-04-16T16:19:14.028 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:19:14.028 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-16T16:19:14.083 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:19:14.083 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:19:14.083 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:19:14.136 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-16T16:19:14.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:14.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:19:14.279 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:19:14.279 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:19:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:19:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:19:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:19:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 0@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:19:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:19:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-16T16:19:14.319 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:19:14.319 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:19:14.319 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:19:14.319 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:19:14.323 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:19:14.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:14.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:19:14.418 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:19:14.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:19:14.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:19:14.557 INFO:teuthology.orchestra.run.smithi045.stdout:client6382 2024-04-16T16:19:14.557 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:19:14.557 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:19:14.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:19:14.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.831+0000 7fa15a400640 1 Processor -- start 2024-04-16T16:19:14.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.831+0000 7fa15a400640 1 -- start start 2024-04-16T16:19:14.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.832+0000 7fa15a400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa1541462e0 0x7fa154072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:14.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.832+0000 7fa15a400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1540728c0 0x7fa1540ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:14.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.832+0000 7fa15a400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa1540cec00 0x7fa15415da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:14.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.832+0000 7fa15a400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa154072d20 con 0x7fa1540cec00 2024-04-16T16:19:14.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.832+0000 7fa15a400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa1540cf6d0 con 0x7fa1541462e0 2024-04-16T16:19:14.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.832+0000 7fa15a400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa1541466c0 con 0x7fa1540728c0 2024-04-16T16:19:14.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.833+0000 7fa153e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa1540cec00 0x7fa15415da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:14.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.833+0000 7fa153e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa1540cec00 0x7fa15415da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33322/0 (socket says 172.21.15.45:33322) 2024-04-16T16:19:14.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.833+0000 7fa153e00640 1 -- 172.21.15.45:0/4060232304 learned_addr learned my addr 172.21.15.45:0/4060232304 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:14.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.833+0000 7fa153400640 1 --2- 172.21.15.45:0/4060232304 >> v2:172.21.15.103:3300/0 conn(0x7fa1541462e0 0x7fa154072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:14.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.833+0000 7fa152a00640 1 --2- 172.21.15.45:0/4060232304 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1540728c0 0x7fa1540ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:14.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.833+0000 7fa153e00640 1 -- 172.21.15.45:0/4060232304 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1540728c0 msgr2=0x7fa1540ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:14.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.833+0000 7fa153e00640 1 --2- 172.21.15.45:0/4060232304 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1540728c0 0x7fa1540ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:14.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.833+0000 7fa153e00640 1 -- 172.21.15.45:0/4060232304 >> v2:172.21.15.103:3300/0 conn(0x7fa1541462e0 msgr2=0x7fa154072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:14.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.833+0000 7fa153e00640 1 --2- 172.21.15.45:0/4060232304 >> v2:172.21.15.103:3300/0 conn(0x7fa1541462e0 0x7fa154072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:14.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.833+0000 7fa153e00640 1 -- 172.21.15.45:0/4060232304 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1540cf3f0 con 0x7fa1540cec00 2024-04-16T16:19:14.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.835+0000 7fa153e00640 1 --2- 172.21.15.45:0/4060232304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1540cec00 0x7fa15415da60 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fa1540b5ec0 tx=0x7fa14c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f8b577e6c291a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:14.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.835+0000 7fa152000640 1 -- 172.21.15.45:0/4060232304 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14c02c9e0 con 0x7fa1540cec00 2024-04-16T16:19:14.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.835+0000 7fa152000640 1 -- 172.21.15.45:0/4060232304 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa14c02cb80 con 0x7fa1540cec00 2024-04-16T16:19:14.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.835+0000 7fa152000640 1 -- 172.21.15.45:0/4060232304 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14c0343f0 con 0x7fa1540cec00 2024-04-16T16:19:14.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.836+0000 7fa15a400640 1 -- 172.21.15.45:0/4060232304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1540cec00 msgr2=0x7fa15415da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:14.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.836+0000 7fa15a400640 1 --2- 172.21.15.45:0/4060232304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1540cec00 0x7fa15415da60 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fa1540b5ec0 tx=0x7fa14c02be40 comp rx=0 tx=0).stop 2024-04-16T16:19:14.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.836+0000 7fa15a400640 1 -- 172.21.15.45:0/4060232304 shutdown_connections 2024-04-16T16:19:14.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.836+0000 7fa15a400640 1 --2- 172.21.15.45:0/4060232304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1540cec00 0x7fa15415da60 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:14.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.836+0000 7fa15a400640 1 --2- 172.21.15.45:0/4060232304 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1540728c0 0x7fa1540ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:14.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.836+0000 7fa15a400640 1 --2- 172.21.15.45:0/4060232304 >> v2:172.21.15.103:3300/0 conn(0x7fa1541462e0 0x7fa154072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:14.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.836+0000 7fa15a400640 1 -- 172.21.15.45:0/4060232304 >> 172.21.15.45:0/4060232304 conn(0x7fa1540b2750 msgr2=0x7fa1540b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:14.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.836+0000 7fa15a400640 1 -- 172.21.15.45:0/4060232304 shutdown_connections 2024-04-16T16:19:14.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.836+0000 7fa15a400640 1 -- 172.21.15.45:0/4060232304 wait complete. 2024-04-16T16:19:14.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.838+0000 7fa15a400640 1 Processor -- start 2024-04-16T16:19:14.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.838+0000 7fa15a400640 1 -- start start 2024-04-16T16:19:14.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.839+0000 7fa15a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1540728c0 0x7fa15415f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:14.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.839+0000 7fa15a400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1540cec00 0x7fa15413da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:14.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.839+0000 7fa15a400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1541462e0 0x7fa15415c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:14.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.839+0000 7fa15a400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa15415d540 con 0x7fa1540728c0 2024-04-16T16:19:14.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.839+0000 7fa15a400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa15415d260 con 0x7fa1541462e0 2024-04-16T16:19:14.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.839+0000 7fa15a400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa1540cf090 con 0x7fa1540cec00 2024-04-16T16:19:14.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.839+0000 7fa153400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1540728c0 0x7fa15415f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:14.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.839+0000 7fa153400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1540728c0 0x7fa15415f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33334/0 (socket says 172.21.15.45:33334) 2024-04-16T16:19:14.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.839+0000 7fa153400640 1 -- 172.21.15.45:0/4163849581 learned_addr learned my addr 172.21.15.45:0/4163849581 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:14.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.839+0000 7fa152a00640 1 --2- 172.21.15.45:0/4163849581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1540cec00 0x7fa15413da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:14.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.839+0000 7fa153400640 1 -- 172.21.15.45:0/4163849581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1540cec00 msgr2=0x7fa15413da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:14.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.839+0000 7fa153400640 1 --2- 172.21.15.45:0/4163849581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1540cec00 0x7fa15413da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:14.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.839+0000 7fa153400640 1 -- 172.21.15.45:0/4163849581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1541462e0 msgr2=0x7fa15415c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:14.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.839+0000 7fa153400640 1 --2- 172.21.15.45:0/4163849581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1541462e0 0x7fa15415c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:14.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.839+0000 7fa153400640 1 -- 172.21.15.45:0/4163849581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa15413e1f0 con 0x7fa1540728c0 2024-04-16T16:19:14.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.840+0000 7fa153400640 1 --2- 172.21.15.45:0/4163849581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1540728c0 0x7fa15415f060 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fa14400d620 tx=0x7fa14400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:14.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.840+0000 7fa14b400640 1 -- 172.21.15.45:0/4163849581 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14400b620 con 0x7fa1540728c0 2024-04-16T16:19:14.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.840+0000 7fa14b400640 1 -- 172.21.15.45:0/4163849581 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa144019070 con 0x7fa1540728c0 2024-04-16T16:19:14.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.840+0000 7fa15a400640 1 -- 172.21.15.45:0/4163849581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa15413e4d0 con 0x7fa1540728c0 2024-04-16T16:19:14.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.840+0000 7fa14b400640 1 -- 172.21.15.45:0/4163849581 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14400ba00 con 0x7fa1540728c0 2024-04-16T16:19:14.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.840+0000 7fa15a400640 1 -- 172.21.15.45:0/4163849581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa15413ea90 con 0x7fa1540728c0 2024-04-16T16:19:14.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.842+0000 7fa15a400640 1 -- 172.21.15.45:0/4163849581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa154075780 con 0x7fa1540728c0 2024-04-16T16:19:14.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.842+0000 7fa14b400640 1 -- 172.21.15.45:0/4163849581 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 33) ==== 91142+0+0 (secure 0 0 0) 0x7fa14400bba0 con 0x7fa1540728c0 2024-04-16T16:19:14.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.843+0000 7fa14b400640 1 --2- 172.21.15.45:0/4163849581 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fa13406ca10 0x7fa13406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:14.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.843+0000 7fa14b400640 1 -- 172.21.15.45:0/4163849581 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5752+0+0 (secure 0 0 0) 0x7fa14408c170 con 0x7fa1540728c0 2024-04-16T16:19:14.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.844+0000 7fa152a00640 1 --2- 172.21.15.45:0/4163849581 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fa13406ca10 0x7fa13406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:14.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.845+0000 7fa152a00640 1 --2- 172.21.15.45:0/4163849581 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fa13406ca10 0x7fa13406eed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa1540b5b70 tx=0x7fa140008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:14.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:14.848+0000 7fa14b400640 1 -- 172.21.15.45:0/4163849581 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa14405a330 con 0x7fa1540728c0 2024-04-16T16:19:15.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:15.093+0000 7fa15a400640 1 -- 172.21.15.45:0/4163849581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa1541456b0 con 0x7fa1540728c0 2024-04-16T16:19:15.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:15.094+0000 7fa14b400640 1 -- 172.21.15.45:0/4163849581 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v73) ==== 53+0+73 (secure 0 0 0) 0x7fa144059c30 con 0x7fa1540728c0 2024-04-16T16:19:15.094 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:19:15.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:15.096+0000 7fa15a400640 1 -- 172.21.15.45:0/4163849581 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fa13406ca10 msgr2=0x7fa13406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:15.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:15.096+0000 7fa15a400640 1 --2- 172.21.15.45:0/4163849581 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fa13406ca10 0x7fa13406eed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa1540b5b70 tx=0x7fa140008040 comp rx=0 tx=0).stop 2024-04-16T16:19:15.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:15.097+0000 7fa15a400640 1 -- 172.21.15.45:0/4163849581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1540728c0 msgr2=0x7fa15415f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:15.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:15.097+0000 7fa15a400640 1 --2- 172.21.15.45:0/4163849581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1540728c0 0x7fa15415f060 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fa14400d620 tx=0x7fa14400dae0 comp rx=0 tx=0).stop 2024-04-16T16:19:15.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:15.097+0000 7fa15a400640 1 -- 172.21.15.45:0/4163849581 shutdown_connections 2024-04-16T16:19:15.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:15.097+0000 7fa15a400640 1 --2- 172.21.15.45:0/4163849581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1541462e0 0x7fa15415c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:15.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:15.097+0000 7fa15a400640 1 --2- 172.21.15.45:0/4163849581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1540cec00 0x7fa15413da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:15.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:15.097+0000 7fa15a400640 1 --2- 172.21.15.45:0/4163849581 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fa13406ca10 0x7fa13406eed0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:15.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:15.097+0000 7fa15a400640 1 --2- 172.21.15.45:0/4163849581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1540728c0 0x7fa15415f060 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:15.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:15.097+0000 7fa15a400640 1 -- 172.21.15.45:0/4163849581 >> 172.21.15.45:0/4163849581 conn(0x7fa1540b2750 msgr2=0x7fa15406fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:15.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:15.097+0000 7fa15a400640 1 -- 172.21.15.45:0/4163849581 shutdown_connections 2024-04-16T16:19:15.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:15.097+0000 7fa15a400640 1 -- 172.21.15.45:0/4163849581 wait complete. 2024-04-16T16:19:15.116 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-16T16:19:15.116 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:19:15.116 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-04-16T16:19:15.116 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-16T16:19:15.116 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T16:19:15.116 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-16T16:19:15.116 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:19:15.116 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:19:15.116 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:19:15.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:15.116 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-04-16T16:19:15.175 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:19:15.175 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:19:15.176 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:19:15.176 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:15.176 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-16T16:19:15.176 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:15.176 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:19:15.176 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:15.176 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-04-16T16:19:15.176 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:19:15.176 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:15.176 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-04-16T16:19:15.176 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:19:15.176 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:15.176 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:19:15.177 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:15.177 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-16T16:19:15.177 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42270sec preferred_lft 42270sec 2024-04-16T16:19:15.177 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-04-16T16:19:15.177 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:15.177 INFO:teuthology.orchestra.run.smithi103.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:19:15.177 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:15.177 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:19:15.177 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:15.177 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::74ce:ceff:fec6:2a00/64 scope link 2024-04-16T16:19:15.177 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:15.177 INFO:teuthology.orchestra.run.smithi103.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:19:15.177 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T16:19:15.177 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::3cf7:ffff:feca:f879/64 scope link 2024-04-16T16:19:15.178 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:15.178 INFO:teuthology.orchestra.run.smithi103.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:19:15.178 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 86:4c:6e:ce:c3:07 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-16T16:19:15.178 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::844c:6eff:fece:c307/64 scope link 2024-04-16T16:19:15.178 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:15.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:15.179 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:19:15.263 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-16T16:19:15.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:15.268 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-04-16T16:19:15.320 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-04-16T16:19:15.320 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:19:15.321 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-16T16:19:15.321 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-16T16:19:15.321 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:19:15.321 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:19:15.321 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:19:15.321 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-16T16:19:15.322 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-16T16:19:15.322 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-16T16:19:15.322 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:19:15.422 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:19:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:19:15.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:19:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:19:15.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:15.569 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-16T16:19:15.616 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-16T16:19:15.617 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-16T16:19:15.617 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:19:15.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:15.618 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.0 up 2024-04-16T16:19:15.702 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:19:15.703 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-16T16:19:15.756 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:19:15.756 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:19:15.756 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:19:15.807 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-16T16:19:15.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:15.807 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:19:15.951 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:19:15.951 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:19:15.986 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:19:15.986 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:19:15.987 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:19:15.987 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with new device syntax: 1@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:19:15.987 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:19:15.989 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-16T16:19:15.989 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:19:15.989 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:19:15.989 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:19:15.989 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:19:15.992 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:19:15.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:15.993 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:19:16.075 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:19:16.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:19:16 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:19:16.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:19:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:19:16.194 INFO:teuthology.orchestra.run.smithi103.stdout:client6388 2024-04-16T16:19:16.194 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:19:16.194 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:19:16.245 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:19:16.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.473+0000 7fd7e5e00640 1 Processor -- start 2024-04-16T16:19:16.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.473+0000 7fd7e5e00640 1 -- start start 2024-04-16T16:19:16.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.474+0000 7fd7e5e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e01462e0 0x7fd7e0072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:16.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.474+0000 7fd7e5e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd7e00728c0 0x7fd7e00ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:16.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.474+0000 7fd7e5e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7e00cf6d0 con 0x7fd7e00cec00 2024-04-16T16:19:16.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.474+0000 7fd7e5e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd7e0072d20 con 0x7fd7e00728c0 2024-04-16T16:19:16.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.474+0000 7fd7e5e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd7e01466c0 con 0x7fd7e01462e0 2024-04-16T16:19:16.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.475+0000 7fd7de000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd7e00728c0 0x7fd7e00ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:16.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.475+0000 7fd7df400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd7e00cec00 0x7fd7e015da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45016/0 (socket says 172.21.15.45:45016) 2024-04-16T16:19:16.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.475+0000 7fd7df400640 1 -- 172.21.15.45:0/3490789216 learned_addr learned my addr 172.21.15.45:0/3490789216 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:16.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.475+0000 7fd7dea00640 1 --2- 172.21.15.45:0/3490789216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e01462e0 0x7fd7e0072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:16.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.475+0000 7fd7de000640 1 -- 172.21.15.45:0/3490789216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e01462e0 msgr2=0x7fd7e0072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:16.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.475+0000 7fd7de000640 1 --2- 172.21.15.45:0/3490789216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e01462e0 0x7fd7e0072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.475+0000 7fd7de000640 1 -- 172.21.15.45:0/3490789216 >> v1:172.21.15.45:6789/0 conn(0x7fd7e00cec00 legacy=0x7fd7e015da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:16.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.476+0000 7fd7de000640 1 -- 172.21.15.45:0/3490789216 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e00cf3f0 con 0x7fd7e00728c0 2024-04-16T16:19:16.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.477+0000 7fd7de000640 1 --2- 172.21.15.45:0/3490789216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e00728c0 0x7fd7e00ce6c0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8009d90 tx=0x7fd7c802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2203e10a5147d114 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:16.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.477+0000 7fd7dd600640 1 -- 172.21.15.45:0/3490789216 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8039070 con 0x7fd7e00728c0 2024-04-16T16:19:16.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.477+0000 7fd7dd600640 1 -- 172.21.15.45:0/3490789216 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7c802cbd0 con 0x7fd7e00728c0 2024-04-16T16:19:16.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.477+0000 7fd7dd600640 1 -- 172.21.15.45:0/3490789216 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8034440 con 0x7fd7e00728c0 2024-04-16T16:19:16.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.478+0000 7fd7e5e00640 1 -- 172.21.15.45:0/3490789216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e00728c0 msgr2=0x7fd7e00ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:16.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.478+0000 7fd7e5e00640 1 --2- 172.21.15.45:0/3490789216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e00728c0 0x7fd7e00ce6c0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8009d90 tx=0x7fd7c802be20 comp rx=0 tx=0).stop 2024-04-16T16:19:16.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.478+0000 7fd7e5e00640 1 -- 172.21.15.45:0/3490789216 shutdown_connections 2024-04-16T16:19:16.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.478+0000 7fd7e5e00640 1 --2- 172.21.15.45:0/3490789216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e00728c0 0x7fd7e00ce6c0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.478+0000 7fd7e5e00640 1 --2- 172.21.15.45:0/3490789216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e01462e0 0x7fd7e0072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.478+0000 7fd7e5e00640 1 -- 172.21.15.45:0/3490789216 >> 172.21.15.45:0/3490789216 conn(0x7fd7e00b2750 msgr2=0x7fd7e00b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:16.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.478+0000 7fd7e5e00640 1 -- 172.21.15.45:0/3490789216 shutdown_connections 2024-04-16T16:19:16.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.478+0000 7fd7e5e00640 1 -- 172.21.15.45:0/3490789216 wait complete. 2024-04-16T16:19:16.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.480+0000 7fd7e5e00640 1 Processor -- start 2024-04-16T16:19:16.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.480+0000 7fd7e5e00640 1 -- start start 2024-04-16T16:19:16.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.481+0000 7fd7e5e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7e00728c0 0x7fd7e01f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:16.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.481+0000 7fd7e5e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e00cec00 0x7fd7e015c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:16.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.481+0000 7fd7e5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e01462e0 0x7fd7e01e5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:16.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.481+0000 7fd7e5e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e015d540 con 0x7fd7e01462e0 2024-04-16T16:19:16.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.481+0000 7fd7e5e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e015d260 con 0x7fd7e00728c0 2024-04-16T16:19:16.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.481+0000 7fd7e5e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd7e00cf090 con 0x7fd7e00cec00 2024-04-16T16:19:16.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.481+0000 7fd7df400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e01462e0 0x7fd7e01e5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:16.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.481+0000 7fd7df400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e01462e0 0x7fd7e01e5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33348/0 (socket says 172.21.15.45:33348) 2024-04-16T16:19:16.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.481+0000 7fd7df400640 1 -- 172.21.15.45:0/3443071741 learned_addr learned my addr 172.21.15.45:0/3443071741 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:16.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.482+0000 7fd7de000640 1 --2- 172.21.15.45:0/3443071741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e00cec00 0x7fd7e015c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:16.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.482+0000 7fd7dea00640 1 --2- 172.21.15.45:0/3443071741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7e00728c0 0x7fd7e01f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:16.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.482+0000 7fd7df400640 1 -- 172.21.15.45:0/3443071741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e00cec00 msgr2=0x7fd7e015c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:16.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.482+0000 7fd7df400640 1 --2- 172.21.15.45:0/3443071741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e00cec00 0x7fd7e015c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.482+0000 7fd7df400640 1 -- 172.21.15.45:0/3443071741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7e00728c0 msgr2=0x7fd7e01f23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:16.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.482+0000 7fd7df400640 1 --2- 172.21.15.45:0/3443071741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7e00728c0 0x7fd7e01f23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.482+0000 7fd7df400640 1 -- 172.21.15.45:0/3443071741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e015c9c0 con 0x7fd7e01462e0 2024-04-16T16:19:16.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.482+0000 7fd7dea00640 1 --2- 172.21.15.45:0/3443071741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7e00728c0 0x7fd7e01f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:16.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.482+0000 7fd7df400640 1 --2- 172.21.15.45:0/3443071741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e01462e0 0x7fd7e01e5870 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fd7d800c420 tx=0x7fd7d800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:16.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.482+0000 7fd7d7400640 1 -- 172.21.15.45:0/3443071741 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d8017070 con 0x7fd7e01462e0 2024-04-16T16:19:16.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.482+0000 7fd7d7400640 1 -- 172.21.15.45:0/3443071741 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7d8005ce0 con 0x7fd7e01462e0 2024-04-16T16:19:16.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.482+0000 7fd7d7400640 1 -- 172.21.15.45:0/3443071741 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d80072d0 con 0x7fd7e01462e0 2024-04-16T16:19:16.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.483+0000 7fd7e5e00640 1 -- 172.21.15.45:0/3443071741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e01e6250 con 0x7fd7e01462e0 2024-04-16T16:19:16.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.483+0000 7fd7e5e00640 1 -- 172.21.15.45:0/3443071741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e013d4e0 con 0x7fd7e01462e0 2024-04-16T16:19:16.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.484+0000 7fd7e5e00640 1 -- 172.21.15.45:0/3443071741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7e00cfca0 con 0x7fd7e01462e0 2024-04-16T16:19:16.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.485+0000 7fd7d7400640 1 -- 172.21.15.45:0/3443071741 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 33) ==== 91142+0+0 (secure 0 0 0) 0x7fd7d801a070 con 0x7fd7e01462e0 2024-04-16T16:19:16.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.486+0000 7fd7d7400640 1 --2- 172.21.15.45:0/3443071741 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd7bc06ca10 0x7fd7bc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:16.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.486+0000 7fd7d7400640 1 -- 172.21.15.45:0/3443071741 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5752+0+0 (secure 0 0 0) 0x7fd7d808afb0 con 0x7fd7e01462e0 2024-04-16T16:19:16.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.486+0000 7fd7dea00640 1 --2- 172.21.15.45:0/3443071741 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd7bc06ca10 0x7fd7bc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:16.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.488+0000 7fd7dea00640 1 --2- 172.21.15.45:0/3443071741 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd7bc06ca10 0x7fd7bc06eed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0006fd0 tx=0x7fd7d0008040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:16.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.491+0000 7fd7d7400640 1 -- 172.21.15.45:0/3443071741 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd7e00cfca0 con 0x7fd7e01462e0 2024-04-16T16:19:16.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.736+0000 7fd7e5e00640 1 -- 172.21.15.45:0/3443071741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7e0075780 con 0x7fd7e01462e0 2024-04-16T16:19:16.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.737+0000 7fd7d7400640 1 -- 172.21.15.45:0/3443071741 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5186 (secure 0 0 0) 0x7fd7d8058c40 con 0x7fd7e01462e0 2024-04-16T16:19:16.737 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:16.737 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5804,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2862612740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2862612740},{"type":"v1","addr":"172.21.15.103:6836","nonce":2862612740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6304,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2934367463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2934367463},{"type":"v1","addr":"172.21.15.103:6837","nonce":2934367463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1425499906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1425499906},{"type":"v1","addr":"172.21.15.45:6835","nonce":1425499906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:19:05.457389+0000","modified":"2024-04-16T16:19:08.527316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7677},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7677":{"gid":7677,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/882883484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":882883484},{"type":"v1","addr":"172.21.15.45:6834","nonce":882883484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7677,"qdb_cluster":[7677]},"id":7}]} 2024-04-16T16:19:16.737 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 73 2024-04-16T16:19:16.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.739+0000 7fd7e5e00640 1 -- 172.21.15.45:0/3443071741 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd7bc06ca10 msgr2=0x7fd7bc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:16.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.739+0000 7fd7e5e00640 1 --2- 172.21.15.45:0/3443071741 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd7bc06ca10 0x7fd7bc06eed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0006fd0 tx=0x7fd7d0008040 comp rx=0 tx=0).stop 2024-04-16T16:19:16.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.739+0000 7fd7e5e00640 1 -- 172.21.15.45:0/3443071741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e01462e0 msgr2=0x7fd7e01e5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:16.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.739+0000 7fd7e5e00640 1 --2- 172.21.15.45:0/3443071741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e01462e0 0x7fd7e01e5870 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fd7d800c420 tx=0x7fd7d800c8e0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.740+0000 7fd7e5e00640 1 -- 172.21.15.45:0/3443071741 shutdown_connections 2024-04-16T16:19:16.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.740+0000 7fd7e5e00640 1 --2- 172.21.15.45:0/3443071741 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fd7bc06ca10 0x7fd7bc06eed0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.740+0000 7fd7e5e00640 1 --2- 172.21.15.45:0/3443071741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e01462e0 0x7fd7e01e5870 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.740+0000 7fd7e5e00640 1 --2- 172.21.15.45:0/3443071741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e00cec00 0x7fd7e015c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.740+0000 7fd7e5e00640 1 --2- 172.21.15.45:0/3443071741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7e00728c0 0x7fd7e01f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.740+0000 7fd7e5e00640 1 -- 172.21.15.45:0/3443071741 >> 172.21.15.45:0/3443071741 conn(0x7fd7e00b2750 msgr2=0x7fd7e00b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:16.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.740+0000 7fd7e5e00640 1 -- 172.21.15.45:0/3443071741 shutdown_connections 2024-04-16T16:19:16.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.740+0000 7fd7e5e00640 1 -- 172.21.15.45:0/3443071741 wait complete. 2024-04-16T16:19:16.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable stats 2024-04-16T16:19:16.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.984+0000 7f070aa00640 1 Processor -- start 2024-04-16T16:19:16.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.985+0000 7f070aa00640 1 -- start start 2024-04-16T16:19:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.986+0000 7f070aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07040b5a30 0x7f07041625e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.986+0000 7f070aa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f070406e290 0x7f070406e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.986+0000 7f070aa00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07040b5e10 con 0x7f07040d1030 2024-04-16T16:19:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.986+0000 7f070aa00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f070415f3d0 con 0x7f070406e290 2024-04-16T16:19:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.986+0000 7f070aa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f070415ff70 con 0x7f07040b5a30 2024-04-16T16:19:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.987+0000 7f0703400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f07040d1030 0x7f070415f860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45018/0 (socket says 172.21.15.45:45018) 2024-04-16T16:19:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.987+0000 7f0703400640 1 -- 172.21.15.45:0/3938129420 learned_addr learned my addr 172.21.15.45:0/3938129420 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.987+0000 7f0708a00640 1 --2- 172.21.15.45:0/3938129420 >> v2:172.21.15.45:3300/0 conn(0x7f070406e290 0x7f070406e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.987+0000 7f0703e00640 1 --2- 172.21.15.45:0/3938129420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07040b5a30 0x7f07041625e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:16.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.987+0000 7f0702a00640 1 -- 172.21.15.45:0/3938129420 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4251766311 0 0) 0x7f07040b5e10 con 0x7f07040d1030 2024-04-16T16:19:16.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.987+0000 7f0702a00640 1 -- 172.21.15.45:0/3938129420 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06e4003520 con 0x7f07040d1030 2024-04-16T16:19:16.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.988+0000 7f0708a00640 1 -- 172.21.15.45:0/3938129420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07040b5a30 msgr2=0x7f07041625e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:16.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.988+0000 7f0708a00640 1 --2- 172.21.15.45:0/3938129420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07040b5a30 0x7f07041625e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.988+0000 7f0708a00640 1 -- 172.21.15.45:0/3938129420 >> v1:172.21.15.45:6789/0 conn(0x7f07040d1030 legacy=0x7f070415f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:16.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.988+0000 7f0708a00640 1 -- 172.21.15.45:0/3938129420 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0704162e70 con 0x7f070406e290 2024-04-16T16:19:16.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.988+0000 7f0703e00640 1 --2- 172.21.15.45:0/3938129420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07040b5a30 0x7f07041625e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:19:16.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.989+0000 7f0708a00640 1 --2- 172.21.15.45:0/3938129420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f070406e290 0x7f070406e720 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0097d0 tx=0x7f06fc02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c1ed0c285bc9047 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:16.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.990+0000 7f0702a00640 1 -- 172.21.15.45:0/3938129420 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06fc035020 con 0x7f070406e290 2024-04-16T16:19:16.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.990+0000 7f0702a00640 1 -- 172.21.15.45:0/3938129420 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06fc031800 con 0x7f070406e290 2024-04-16T16:19:16.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.990+0000 7f0702a00640 1 -- 172.21.15.45:0/3938129420 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06fc031bd0 con 0x7f070406e290 2024-04-16T16:19:16.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.990+0000 7f070aa00640 1 -- 172.21.15.45:0/3938129420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f070406e290 msgr2=0x7f070406e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:16.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.990+0000 7f070aa00640 1 --2- 172.21.15.45:0/3938129420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f070406e290 0x7f070406e720 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0097d0 tx=0x7f06fc02b8a0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.990+0000 7f070aa00640 1 -- 172.21.15.45:0/3938129420 shutdown_connections 2024-04-16T16:19:16.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.990+0000 7f070aa00640 1 --2- 172.21.15.45:0/3938129420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f070406e290 0x7f070406e720 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.990+0000 7f070aa00640 1 --2- 172.21.15.45:0/3938129420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07040b5a30 0x7f07041625e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.990+0000 7f070aa00640 1 -- 172.21.15.45:0/3938129420 >> 172.21.15.45:0/3938129420 conn(0x7f07040b3970 msgr2=0x7f07040b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:16.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.990+0000 7f070aa00640 1 -- 172.21.15.45:0/3938129420 shutdown_connections 2024-04-16T16:19:16.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.990+0000 7f070aa00640 1 -- 172.21.15.45:0/3938129420 wait complete. 2024-04-16T16:19:16.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.992+0000 7f070aa00640 1 Processor -- start 2024-04-16T16:19:16.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.992+0000 7f070aa00640 1 -- start start 2024-04-16T16:19:16.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.993+0000 7f070aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f070406e290 0x7f0704154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:16.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.993+0000 7f070aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07040b5a30 0x7f07041596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:16.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.993+0000 7f070aa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07040d1030 0x7f0704159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:16.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.993+0000 7f070aa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f070415eff0 con 0x7f070406e290 2024-04-16T16:19:16.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.993+0000 7f070aa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0704163430 con 0x7f07040d1030 2024-04-16T16:19:16.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.993+0000 7f070aa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0704162b20 con 0x7f07040b5a30 2024-04-16T16:19:16.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.994+0000 7f0703e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f070406e290 0x7f0704154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:16.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.994+0000 7f0703e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f070406e290 0x7f0704154d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33366/0 (socket says 172.21.15.45:33366) 2024-04-16T16:19:16.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.994+0000 7f0703e00640 1 -- 172.21.15.45:0/429489757 learned_addr learned my addr 172.21.15.45:0/429489757 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:16.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.994+0000 7f0703400640 1 --2- 172.21.15.45:0/429489757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07040b5a30 0x7f07041596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:16.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.994+0000 7f0708a00640 1 --2- 172.21.15.45:0/429489757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07040d1030 0x7f0704159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:16.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.994+0000 7f0703e00640 1 -- 172.21.15.45:0/429489757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07040b5a30 msgr2=0x7f07041596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:16.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.994+0000 7f0703e00640 1 --2- 172.21.15.45:0/429489757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07040b5a30 0x7f07041596b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.994+0000 7f0703e00640 1 -- 172.21.15.45:0/429489757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07040d1030 msgr2=0x7f0704159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:16.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.994+0000 7f0703e00640 1 --2- 172.21.15.45:0/429489757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07040d1030 0x7f0704159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:16.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.994+0000 7f0703e00640 1 -- 172.21.15.45:0/429489757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0704155280 con 0x7f070406e290 2024-04-16T16:19:16.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.995+0000 7f0703e00640 1 --2- 172.21.15.45:0/429489757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f070406e290 0x7f0704154d40 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f06f400e730 tx=0x7f06f400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:16.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.995+0000 7f06fbe00640 1 -- 172.21.15.45:0/429489757 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f400f040 con 0x7f070406e290 2024-04-16T16:19:16.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.995+0000 7f06fbe00640 1 -- 172.21.15.45:0/429489757 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06f4004230 con 0x7f070406e290 2024-04-16T16:19:16.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.995+0000 7f06fbe00640 1 -- 172.21.15.45:0/429489757 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f400ee40 con 0x7f070406e290 2024-04-16T16:19:16.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.995+0000 7f070aa00640 1 -- 172.21.15.45:0/429489757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07041580c0 con 0x7f070406e290 2024-04-16T16:19:16.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.995+0000 7f070aa00640 1 -- 172.21.15.45:0/429489757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0704158680 con 0x7f070406e290 2024-04-16T16:19:16.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.996+0000 7f070aa00640 1 -- 172.21.15.45:0/429489757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0704075780 con 0x7f070406e290 2024-04-16T16:19:16.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.997+0000 7f06fbe00640 1 -- 172.21.15.45:0/429489757 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 33) ==== 91142+0+0 (secure 0 0 0) 0x7f06f400c670 con 0x7f070406e290 2024-04-16T16:19:16.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.998+0000 7f06fbe00640 1 --2- 172.21.15.45:0/429489757 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f06e006ca10 0x7f06e006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:16.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.998+0000 7f06fbe00640 1 -- 172.21.15.45:0/429489757 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5752+0+0 (secure 0 0 0) 0x7f06f408bb60 con 0x7f070406e290 2024-04-16T16:19:16.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:16.999+0000 7f0703400640 1 --2- 172.21.15.45:0/429489757 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f06e006ca10 0x7f06e006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:17.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:17.001+0000 7f0703400640 1 --2- 172.21.15.45:0/429489757 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f06e006ca10 0x7f06e006eed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f07040cfee0 tx=0x7f06ec009040 comp rx=0 tx=0).ready entity=mgr.6115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:17.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:17.003+0000 7f06fbe00640 1 -- 172.21.15.45:0/429489757 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06f4017020 con 0x7f070406e290 2024-04-16T16:19:17.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:17.262+0000 7f070aa00640 1 -- 172.21.15.45:0/429489757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "stats"} v 0) -- 0x7f07040ade00 con 0x7f070406e290 2024-04-16T16:19:18.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.120+0000 7f06fbe00640 1 -- 172.21.15.45:0/429489757 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "stats"}]=0 v34) ==== 84+0+0 (secure 0 0 0) 0x7f06f4059d30 con 0x7f070406e290 2024-04-16T16:19:18.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.124+0000 7f070aa00640 1 -- 172.21.15.45:0/429489757 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f06e006ca10 msgr2=0x7f06e006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:18.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.124+0000 7f070aa00640 1 --2- 172.21.15.45:0/429489757 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f06e006ca10 0x7f06e006eed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f07040cfee0 tx=0x7f06ec009040 comp rx=0 tx=0).stop 2024-04-16T16:19:18.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.124+0000 7f070aa00640 1 -- 172.21.15.45:0/429489757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f070406e290 msgr2=0x7f0704154d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:18.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.124+0000 7f070aa00640 1 --2- 172.21.15.45:0/429489757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f070406e290 0x7f0704154d40 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f06f400e730 tx=0x7f06f400ebf0 comp rx=0 tx=0).stop 2024-04-16T16:19:18.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.125+0000 7f070aa00640 1 -- 172.21.15.45:0/429489757 shutdown_connections 2024-04-16T16:19:18.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.125+0000 7f070aa00640 1 --2- 172.21.15.45:0/429489757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07040d1030 0x7f0704159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:18.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.125+0000 7f070aa00640 1 --2- 172.21.15.45:0/429489757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07040b5a30 0x7f07041596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:18.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.125+0000 7f070aa00640 1 --2- 172.21.15.45:0/429489757 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7f06e006ca10 0x7f06e006eed0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:18.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.125+0000 7f070aa00640 1 --2- 172.21.15.45:0/429489757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f070406e290 0x7f0704154d40 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:18.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.125+0000 7f070aa00640 1 -- 172.21.15.45:0/429489757 >> 172.21.15.45:0/429489757 conn(0x7f07040b3970 msgr2=0x7f0704161c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:18.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.125+0000 7f070aa00640 1 -- 172.21.15.45:0/429489757 shutdown_connections 2024-04-16T16:19:18.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.125+0000 7f070aa00640 1 -- 172.21.15.45:0/429489757 wait complete. 2024-04-16T16:19:18.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats --client_ip 1.2.3 2024-04-16T16:19:18.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.361+0000 7fb43a400640 1 Processor -- start 2024-04-16T16:19:18.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.361+0000 7fb43a400640 1 -- start start 2024-04-16T16:19:18.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.362+0000 7fb43a400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb4340b6310 0x7fb434161db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:18.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.362+0000 7fb43a400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4340b6810 con 0x7fb4340d18f0 2024-04-16T16:19:18.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.363+0000 7fb43a400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4340b6af0 con 0x7fb4340d3e90 2024-04-16T16:19:18.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.363+0000 7fb43a400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb4340d29f0 con 0x7fb4340b6310 2024-04-16T16:19:18.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.363+0000 7fb439000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb4340d18f0 0x7fb4340ba220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45038/0 (socket says 172.21.15.45:45038) 2024-04-16T16:19:18.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.363+0000 7fb439000640 1 -- 172.21.15.45:0/2883877691 learned_addr learned my addr 172.21.15.45:0/2883877691 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:18.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.363+0000 7fb42be00640 1 --2- 172.21.15.45:0/2883877691 >> v2:172.21.15.103:3300/0 conn(0x7fb4340b6310 0x7fb434161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:18.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb433e00640 1 -- 172.21.15.45:0/2883877691 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4125978579 0 0) 0x7fb4340b6810 con 0x7fb4340d18f0 2024-04-16T16:19:18.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb433e00640 1 -- 172.21.15.45:0/2883877691 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb418003520 con 0x7fb4340d18f0 2024-04-16T16:19:18.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb433e00640 1 -- 172.21.15.45:0/2883877691 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4047911637 0 0) 0x7fb418003520 con 0x7fb4340d18f0 2024-04-16T16:19:18.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb433e00640 1 -- 172.21.15.45:0/2883877691 >> v2:172.21.15.103:3300/0 conn(0x7fb4340b6310 msgr2=0x7fb434161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:18.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb433e00640 1 --2- 172.21.15.45:0/2883877691 >> v2:172.21.15.103:3300/0 conn(0x7fb4340b6310 0x7fb434161db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:18.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb433e00640 1 -- 172.21.15.45:0/2883877691 >> v1:172.21.15.103:6789/0 conn(0x7fb4340d3e90 legacy=0x7fb4341622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb433e00640 1 -- 172.21.15.45:0/2883877691 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4340b6dd0 con 0x7fb4340d18f0 2024-04-16T16:19:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb433e00640 1 -- 172.21.15.45:0/2883877691 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fb424001670 con 0x7fb4340d18f0 2024-04-16T16:19:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb433e00640 1 -- 172.21.15.45:0/2883877691 >> v1:172.21.15.45:6789/0 conn(0x7fb4340d18f0 legacy=0x7fb4340ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb433e00640 1 --2- 172.21.15.45:0/2883877691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4180039b0 0x7fb418003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb433e00640 1 --2- 172.21.15.45:0/2883877691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb418004800 0x7fb418004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb433e00640 1 --2- 172.21.15.45:0/2883877691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb418005210 0x7fb41800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb433e00640 1 -- 172.21.15.45:0/2883877691 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb4340d29f0 con 0x7fb4180039b0 2024-04-16T16:19:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb433e00640 1 -- 172.21.15.45:0/2883877691 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb41800c140 con 0x7fb418005210 2024-04-16T16:19:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb433e00640 1 -- 172.21.15.45:0/2883877691 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb41800c340 con 0x7fb418004800 2024-04-16T16:19:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.364+0000 7fb439000640 1 --2- 172.21.15.45:0/2883877691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4180039b0 0x7fb418003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.365+0000 7fb439000640 1 -- 172.21.15.45:0/2883877691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb418004800 msgr2=0x7fb418004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.365+0000 7fb439000640 1 --2- 172.21.15.45:0/2883877691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb418004800 0x7fb418004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.365+0000 7fb439000640 1 -- 172.21.15.45:0/2883877691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb418005210 msgr2=0x7fb41800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.365+0000 7fb439a00640 1 --2- 172.21.15.45:0/2883877691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb418005210 0x7fb41800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.365+0000 7fb439000640 1 --2- 172.21.15.45:0/2883877691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb418005210 0x7fb41800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:18.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.365+0000 7fb439000640 1 -- 172.21.15.45:0/2883877691 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb418003520 con 0x7fb4180039b0 2024-04-16T16:19:18.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.365+0000 7fb439a00640 1 --2- 172.21.15.45:0/2883877691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb418005210 0x7fb41800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:18.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.365+0000 7fb439a00640 1 -- 172.21.15.45:0/2883877691 reap_dead start 2024-04-16T16:19:18.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.366+0000 7fb439000640 1 --2- 172.21.15.45:0/2883877691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4180039b0 0x7fb418003df0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fb42400c240 tx=0x7fb418005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a414c2230d5abc9d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:18.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.366+0000 7fb433e00640 1 -- 172.21.15.45:0/2883877691 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb424007d10 con 0x7fb4180039b0 2024-04-16T16:19:18.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.366+0000 7fb433e00640 1 -- 172.21.15.45:0/2883877691 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb424030070 con 0x7fb4180039b0 2024-04-16T16:19:18.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.366+0000 7fb433e00640 1 -- 172.21.15.45:0/2883877691 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42402cdc0 con 0x7fb4180039b0 2024-04-16T16:19:18.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.367+0000 7fb43a400640 1 -- 172.21.15.45:0/2883877691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4180039b0 msgr2=0x7fb418003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:18.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.367+0000 7fb43a400640 1 --2- 172.21.15.45:0/2883877691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4180039b0 0x7fb418003df0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fb42400c240 tx=0x7fb418005830 comp rx=0 tx=0).stop 2024-04-16T16:19:18.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.367+0000 7fb43a400640 1 -- 172.21.15.45:0/2883877691 shutdown_connections 2024-04-16T16:19:18.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.367+0000 7fb43a400640 1 --2- 172.21.15.45:0/2883877691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4180039b0 0x7fb418003df0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:18.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.367+0000 7fb43a400640 1 -- 172.21.15.45:0/2883877691 >> 172.21.15.45:0/2883877691 conn(0x7fb43406cf90 msgr2=0x7fb43406d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:18.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.367+0000 7fb43a400640 1 -- 172.21.15.45:0/2883877691 shutdown_connections 2024-04-16T16:19:18.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.367+0000 7fb43a400640 1 -- 172.21.15.45:0/2883877691 wait complete. 2024-04-16T16:19:18.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.369+0000 7fb43a400640 1 Processor -- start 2024-04-16T16:19:18.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.369+0000 7fb43a400640 1 -- start start 2024-04-16T16:19:18.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.370+0000 7fb43a400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4180039b0 0x7fb4340c8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:18.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.370+0000 7fb43a400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb434155bb0 0x7fb43414dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:18.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.370+0000 7fb43a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4340c8a40 0x7fb4340c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:18.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.370+0000 7fb43a400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb4340d1e50 con 0x7fb4340c8a40 2024-04-16T16:19:18.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.370+0000 7fb43a400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb4340d1cd0 con 0x7fb434155bb0 2024-04-16T16:19:18.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.370+0000 7fb43a400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb4340d1fd0 con 0x7fb4180039b0 2024-04-16T16:19:18.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.371+0000 7fb439a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4340c8a40 0x7fb4340c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:18.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.371+0000 7fb439a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4340c8a40 0x7fb4340c8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33386/0 (socket says 172.21.15.45:33386) 2024-04-16T16:19:18.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.371+0000 7fb439a00640 1 -- 172.21.15.45:0/3890938891 learned_addr learned my addr 172.21.15.45:0/3890938891 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:18.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.371+0000 7fb42be00640 1 --2- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb434155bb0 0x7fb43414dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:18.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.371+0000 7fb439000640 1 --2- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4180039b0 0x7fb4340c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:18.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.371+0000 7fb439a00640 1 -- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4180039b0 msgr2=0x7fb4340c8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:18.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.371+0000 7fb439a00640 1 --2- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4180039b0 0x7fb4340c8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:18.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.371+0000 7fb439a00640 1 -- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb434155bb0 msgr2=0x7fb43414dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:18.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.371+0000 7fb439a00640 1 --2- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb434155bb0 0x7fb43414dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:18.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.371+0000 7fb439a00640 1 -- 172.21.15.45:0/3890938891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4340ce380 con 0x7fb4340c8a40 2024-04-16T16:19:18.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.371+0000 7fb439a00640 1 --2- 172.21.15.45:0/3890938891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4340c8a40 0x7fb4340c8ed0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00d4c0 tx=0x7fb42c00d8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:18.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.372+0000 7fb431600640 1 -- 172.21.15.45:0/3890938891 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42c018070 con 0x7fb4340c8a40 2024-04-16T16:19:18.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.372+0000 7fb431600640 1 -- 172.21.15.45:0/3890938891 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb42c003da0 con 0x7fb4340c8a40 2024-04-16T16:19:18.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.372+0000 7fb431600640 1 -- 172.21.15.45:0/3890938891 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42c005310 con 0x7fb4340c8a40 2024-04-16T16:19:18.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.372+0000 7fb43a400640 1 -- 172.21.15.45:0/3890938891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4340ce660 con 0x7fb4340c8a40 2024-04-16T16:19:18.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.372+0000 7fb43a400640 1 -- 172.21.15.45:0/3890938891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4340caea0 con 0x7fb4340c8a40 2024-04-16T16:19:18.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.374+0000 7fb43a400640 1 -- 172.21.15.45:0/3890938891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4340cb0a0 con 0x7fb4340c8a40 2024-04-16T16:19:18.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.374+0000 7fb431600640 1 -- 172.21.15.45:0/3890938891 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 34) ==== 91151+0+0 (secure 0 0 0) 0x7fb42c0054b0 con 0x7fb4340c8a40 2024-04-16T16:19:18.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.375+0000 7fb431600640 1 --2- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb41006ca90 0x7fb41006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:18.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.375+0000 7fb431600640 1 -- 172.21.15.45:0/3890938891 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5752+0+0 (secure 0 0 0) 0x7fb42c08b5d0 con 0x7fb4340c8a40 2024-04-16T16:19:18.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.375+0000 7fb439000640 1 -- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb41006ca90 msgr2=0x7fb41006ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/839990480 2024-04-16T16:19:18.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.375+0000 7fb439000640 1 --2- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb41006ca90 0x7fb41006ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:19:18.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.379+0000 7fb431600640 1 -- 172.21.15.45:0/3890938891 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb4340cb0a0 con 0x7fb4340c8a40 2024-04-16T16:19:18.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.576+0000 7fb439000640 1 -- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb41006ca90 msgr2=0x7fb41006ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/839990480 2024-04-16T16:19:18.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.576+0000 7fb439000640 1 --2- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb41006ca90 0x7fb41006ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:19:18.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.611+0000 7fb43a400640 1 -- 172.21.15.45:0/3890938891 --> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] -- mgr_command(tid 0: {"prefix": "fs perf stats", "client_ip": "1.2.3", "target": ["mon-mgr", ""]}) -- 0x7fb434075780 con 0x7fb41006ca90 2024-04-16T16:19:18.682 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:18.682+0000 7f2f251b6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:19:18.720 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:18.721+0000 7f52a1c16040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:19:18.737 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:18.737+0000 7f2f251b6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:19:18.779 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:18.781+0000 7f52a1c16040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:19:18.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.977+0000 7fb439000640 1 -- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb41006ca90 msgr2=0x7fb41006ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/839990480 2024-04-16T16:19:18.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:18.977+0000 7fb439000640 1 --2- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb41006ca90 0x7fb41006ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T16:19:19.183 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:19.184+0000 7f2f251b6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:19:19.236 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:19.236+0000 7f2f251b6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:19:19.249 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:19.250+0000 7f52a1c16040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:19:19.286 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:19.287+0000 7f2f251b6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:19:19.304 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:19.305+0000 7f52a1c16040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:19:19.347 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:19.348+0000 7f2f251b6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:19:19.357 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:19.359+0000 7f52a1c16040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:19:19.396 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:19.397+0000 7f2f251b6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:19:19.422 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:19.423+0000 7f52a1c16040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:19:19.474 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:19.475+0000 7f52a1c16040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:19:19.524 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:19.525+0000 7f2f251b6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:19:19.586 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:19.587+0000 7f2f251b6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:19:19.610 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:19.611+0000 7f52a1c16040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:19:19.638 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:19.638+0000 7f2f251b6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:19:19.676 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:19.677+0000 7f52a1c16040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:19:19.696 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:19.696+0000 7f2f251b6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:19:19.729 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:19.730+0000 7f52a1c16040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:19:19.747 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:19.748+0000 7f2f251b6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:19:19.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:19.778+0000 7fb439000640 1 -- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb41006ca90 msgr2=0x7fb41006ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/839990480 2024-04-16T16:19:19.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:19.778+0000 7fb439000640 1 --2- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb41006ca90 0x7fb41006ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T16:19:19.789 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:19.791+0000 7f52a1c16040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:19:19.843 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:19.844+0000 7f52a1c16040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:19:20.134 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:20.135+0000 7f2f251b6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:19:20.249 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:20.250+0000 7f52a1c16040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:19:20.295 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:20.295+0000 7f2f251b6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:19:20.403 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:20.404+0000 7f2f251b6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:19:20.417 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:20.419+0000 7f52a1c16040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:19:20.454 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:20.454+0000 7f2f251b6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:19:20.533 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:20.534+0000 7f52a1c16040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:19:20.579 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:20.579+0000 7f2f251b6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:19:20.587 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:20.588+0000 7f52a1c16040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:19:20.688 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:20.688+0000 7f2f251b6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:19:20.720 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:20.721+0000 7f52a1c16040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:19:20.833 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:20.834+0000 7f52a1c16040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:19:20.864 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:20.864+0000 7f2f251b6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:19:20.913 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:20.913+0000 7f2f251b6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:19:21.022 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:21.022+0000 7f52a1c16040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:19:21.054 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:21.054+0000 7f2f251b6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:19:21.073 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:21.075+0000 7f52a1c16040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:19:21.218 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:21.219+0000 7f52a1c16040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:19:21.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:21.380+0000 7fb439000640 1 -- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb41006ca90 msgr2=0x7fb41006ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/839990480 2024-04-16T16:19:21.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:21.381+0000 7fb439000640 1 --2- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb41006ca90 0x7fb41006ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T16:19:21.868 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:21.868+0000 7f2f251b6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:19:22.135 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:22.136+0000 7f52a1c16040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:19:22.179 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:22.179+0000 7f2f251b6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:19:22.236 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:22.236+0000 7f2f251b6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:19:22.367 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:22.367+0000 7f2f251b6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:19:22.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:22.388+0000 7fb431600640 1 -- 172.21.15.45:0/3890938891 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 35) ==== 90760+0+0 (secure 0 0 0) 0x7fb42c055680 con 0x7fb4340c8a40 2024-04-16T16:19:22.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:22.388+0000 7fb431600640 1 -- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb41006ca90 msgr2=0x7fb41006ef50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:19:22.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:22.388+0000 7fb431600640 1 --2- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/839990480,v1:172.21.15.103:6833/839990480] conn(0x7fb41006ca90 0x7fb41006ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:22.465 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:22.466+0000 7f52a1c16040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:19:22.466 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:22.466+0000 7f2ea0a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:22.466 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:22.466+0000 7f2ea0a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:22.466 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:22.466+0000 7f2ea0a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:22.466 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:22.466+0000 7f2ea0a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:22.466 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:22.466+0000 7f2ea0a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:22.467 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:22.467+0000 7f2e9ce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:22.467 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:22.467+0000 7f2e9ce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:22.467 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:22.467+0000 7f2e9ce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:22.467 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:22.467+0000 7f2e9ce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:22.467 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:22.467+0000 7f2e9ce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:22.527 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:22.528+0000 7f52a1c16040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:19:22.666 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:22.667+0000 7f52a1c16040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:19:23.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.399+0000 7fb431600640 1 -- 172.21.15.45:0/3890938891 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 36) ==== 90887+0+0 (secure 0 0 0) 0x7fb42c089a20 con 0x7fb4340c8a40 2024-04-16T16:19:23.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.400+0000 7fb431600640 1 --2- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb410074a00 0x7fb410076df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:23.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.400+0000 7fb431600640 1 -- 172.21.15.45:0/3890938891 --> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] -- mgr_command(tid 0: {"prefix": "fs perf stats", "client_ip": "1.2.3", "target": ["mon-mgr", ""]}) -- 0x7fb42c018e80 con 0x7fb410074a00 2024-04-16T16:19:23.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.403+0000 7fb439000640 1 --2- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb410074a00 0x7fb410076df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:23.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.411+0000 7fb439000640 1 --2- 172.21.15.45:0/3890938891 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb410074a00 0x7fb410076df0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb424034000 tx=0x7fb424039040 comp rx=0 tx=0).ready entity=mgr.6397 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:23.415 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:23.415+0000 7f2eac800640 -1 mgr.server reply reply (22) Invalid argument invalid client_ip filter spec: 1.2.3 2024-04-16T16:19:23.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.417+0000 7fb431600640 1 -- 172.21.15.45:0/3890938891 <== mgr.6397 v2:172.21.15.103:6832/3948990093 1 ==== mgr_command_reply(tid 0: -22 invalid client_ip filter spec: 1.2.3) ==== 44+0+0 (secure 0 0 0) 0x7fb42c018e80 con 0x7fb410074a00 2024-04-16T16:19:23.417 INFO:teuthology.orchestra.run.smithi045.stderr:Error EINVAL: invalid client_ip filter spec: 1.2.3 2024-04-16T16:19:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T16:19:23.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable stats 2024-04-16T16:19:23.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.656+0000 7fb9c0a00640 1 Processor -- start 2024-04-16T16:19:23.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.657+0000 7fb9c0a00640 1 -- start start 2024-04-16T16:19:23.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.658+0000 7fb9c0a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb9bc1460f0 0x7fb9bc06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.658+0000 7fb9c0a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9bc0cece0 con 0x7fb9bc0ce520 2024-04-16T16:19:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.658+0000 7fb9c0a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9bc0cefc0 con 0x7fb9bc072630 2024-04-16T16:19:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.658+0000 7fb9c0a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb9bc1464d0 con 0x7fb9bc1460f0 2024-04-16T16:19:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.659+0000 7fb9bb400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb9bc1460f0 0x7fb9bc06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.659+0000 7fb9b3e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb9bc0ce520 0x7fb9bc071f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43398/0 (socket says 172.21.15.45:43398) 2024-04-16T16:19:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.659+0000 7fb9b3e00640 1 -- 172.21.15.45:0/2766948776 learned_addr learned my addr 172.21.15.45:0/2766948776 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:23.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.659+0000 7fb9bb400640 1 -- 172.21.15.45:0/2766948776 >> v1:172.21.15.103:6789/0 conn(0x7fb9bc072630 legacy=0x7fb9bc15d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:23.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.659+0000 7fb9bb400640 1 -- 172.21.15.45:0/2766948776 >> v1:172.21.15.45:6789/0 conn(0x7fb9bc0ce520 legacy=0x7fb9bc071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:23.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.659+0000 7fb9bb400640 1 -- 172.21.15.45:0/2766948776 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9bc15cf20 con 0x7fb9bc1460f0 2024-04-16T16:19:23.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.660+0000 7fb9bb400640 1 --2- 172.21.15.45:0/2766948776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc1460f0 0x7fb9bc06ed60 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc146650 tx=0x7fb9ac02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e11654657dc2370 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.661+0000 7fb9baa00640 1 -- 172.21.15.45:0/2766948776 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac035020 con 0x7fb9bc1460f0 2024-04-16T16:19:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.661+0000 7fb9baa00640 1 -- 172.21.15.45:0/2766948776 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9ac031810 con 0x7fb9bc1460f0 2024-04-16T16:19:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.661+0000 7fb9baa00640 1 -- 172.21.15.45:0/2766948776 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac031be0 con 0x7fb9bc1460f0 2024-04-16T16:19:23.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.661+0000 7fb9c0a00640 1 -- 172.21.15.45:0/2766948776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc1460f0 msgr2=0x7fb9bc06ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:23.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.661+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/2766948776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc1460f0 0x7fb9bc06ed60 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc146650 tx=0x7fb9ac02b8b0 comp rx=0 tx=0).stop 2024-04-16T16:19:23.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.661+0000 7fb9c0a00640 1 -- 172.21.15.45:0/2766948776 shutdown_connections 2024-04-16T16:19:23.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.661+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/2766948776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc1460f0 0x7fb9bc06ed60 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:23.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.661+0000 7fb9c0a00640 1 -- 172.21.15.45:0/2766948776 >> 172.21.15.45:0/2766948776 conn(0x7fb9bc067e00 msgr2=0x7fb9bc06b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:23.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.661+0000 7fb9c0a00640 1 -- 172.21.15.45:0/2766948776 shutdown_connections 2024-04-16T16:19:23.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.661+0000 7fb9c0a00640 1 -- 172.21.15.45:0/2766948776 wait complete. 2024-04-16T16:19:23.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.663+0000 7fb9c0a00640 1 Processor -- start 2024-04-16T16:19:23.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.663+0000 7fb9c0a00640 1 -- start start 2024-04-16T16:19:23.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.664+0000 7fb9c0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9bc072630 0x7fb9bc15c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:23.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.664+0000 7fb9c0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc0ce520 0x7fb9bc1f22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:23.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.664+0000 7fb9c0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9bc1460f0 0x7fb9bc140020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:23.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.664+0000 7fb9c0a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc0ce990 con 0x7fb9bc0ce520 2024-04-16T16:19:23.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.664+0000 7fb9c0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc15cc20 con 0x7fb9bc1460f0 2024-04-16T16:19:23.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.664+0000 7fb9c0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb9bc15de30 con 0x7fb9bc072630 2024-04-16T16:19:23.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.664+0000 7fb9b3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc0ce520 0x7fb9bc1f22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:23.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.664+0000 7fb9b3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc0ce520 0x7fb9bc1f22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34662/0 (socket says 172.21.15.45:34662) 2024-04-16T16:19:23.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.664+0000 7fb9b3e00640 1 -- 172.21.15.45:0/263776835 learned_addr learned my addr 172.21.15.45:0/263776835 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:23.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.665+0000 7fb9bbe00640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9bc1460f0 0x7fb9bc140020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:23.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.665+0000 7fb9b3e00640 1 -- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9bc072630 msgr2=0x7fb9bc15c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:23.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.665+0000 7fb9bb400640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9bc072630 0x7fb9bc15c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:23.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.665+0000 7fb9b3e00640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9bc072630 0x7fb9bc15c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:23.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.665+0000 7fb9b3e00640 1 -- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9bc1460f0 msgr2=0x7fb9bc140020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:23.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.665+0000 7fb9b3e00640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9bc1460f0 0x7fb9bc140020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:23.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.665+0000 7fb9b3e00640 1 -- 172.21.15.45:0/263776835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9bc13ac60 con 0x7fb9bc0ce520 2024-04-16T16:19:23.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.665+0000 7fb9bbe00640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9bc1460f0 0x7fb9bc140020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:23.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.665+0000 7fb9bb400640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9bc072630 0x7fb9bc15c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:23.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.665+0000 7fb9b3e00640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc0ce520 0x7fb9bc1f22e0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fb9a400c400 tx=0x7fb9a400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:23.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.665+0000 7fb9b3400640 1 -- 172.21.15.45:0/263776835 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a4017070 con 0x7fb9bc0ce520 2024-04-16T16:19:23.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.666+0000 7fb9b3400640 1 -- 172.21.15.45:0/263776835 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9a4005ce0 con 0x7fb9bc0ce520 2024-04-16T16:19:23.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.666+0000 7fb9b3400640 1 -- 172.21.15.45:0/263776835 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a40072f0 con 0x7fb9bc0ce520 2024-04-16T16:19:23.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.666+0000 7fb9c0a00640 1 -- 172.21.15.45:0/263776835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9bc13af40 con 0x7fb9bc0ce520 2024-04-16T16:19:23.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.666+0000 7fb9c0a00640 1 -- 172.21.15.45:0/263776835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9bc13b500 con 0x7fb9bc0ce520 2024-04-16T16:19:23.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.667+0000 7fb9c0a00640 1 -- 172.21.15.45:0/263776835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9bc075780 con 0x7fb9bc0ce520 2024-04-16T16:19:23.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.668+0000 7fb9b3400640 1 -- 172.21.15.45:0/263776835 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 36) ==== 90887+0+0 (secure 0 0 0) 0x7fb9a401a070 con 0x7fb9bc0ce520 2024-04-16T16:19:23.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.669+0000 7fb9b3400640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb99806c7c0 0x7fb99806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:23.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.669+0000 7fb9b3400640 1 -- 172.21.15.45:0/263776835 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6096+0+0 (secure 0 0 0) 0x7fb9a408ae00 con 0x7fb9bc0ce520 2024-04-16T16:19:23.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.669+0000 7fb9bb400640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb99806c7c0 0x7fb99806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:23.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.671+0000 7fb9bb400640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb99806c7c0 0x7fb99806ec80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac036000 tx=0x7fb9ac008030 comp rx=0 tx=0).ready entity=mgr.6397 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:23.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.674+0000 7fb9b3400640 1 -- 172.21.15.45:0/263776835 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb9a4059340 con 0x7fb9bc0ce520 2024-04-16T16:19:23.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:23.931+0000 7fb9c0a00640 1 -- 172.21.15.45:0/263776835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "stats"} v 0) -- 0x7fb9bc0ce140 con 0x7fb9bc0ce520 2024-04-16T16:19:24.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:24.445+0000 7fb9b3400640 1 -- 172.21.15.45:0/263776835 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "stats"}]=0 v37) ==== 85+0+0 (secure 0 0 0) 0x7fb9a4058c40 con 0x7fb9bc0ce520 2024-04-16T16:19:24.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:24.449+0000 7fb9c0a00640 1 -- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb99806c7c0 msgr2=0x7fb99806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:24.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:24.449+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb99806c7c0 0x7fb99806ec80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac036000 tx=0x7fb9ac008030 comp rx=0 tx=0).stop 2024-04-16T16:19:24.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:24.449+0000 7fb9c0a00640 1 -- 172.21.15.45:0/263776835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc0ce520 msgr2=0x7fb9bc1f22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:24.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:24.449+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc0ce520 0x7fb9bc1f22e0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fb9a400c400 tx=0x7fb9a400c8c0 comp rx=0 tx=0).stop 2024-04-16T16:19:24.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:24.450+0000 7fb9c0a00640 1 -- 172.21.15.45:0/263776835 shutdown_connections 2024-04-16T16:19:24.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:24.450+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb99806c7c0 0x7fb99806ec80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:24.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:24.450+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9bc1460f0 0x7fb9bc140020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:24.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:24.450+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9bc0ce520 0x7fb9bc1f22e0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:24.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:24.450+0000 7fb9c0a00640 1 --2- 172.21.15.45:0/263776835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb9bc072630 0x7fb9bc15c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:24.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:24.450+0000 7fb9c0a00640 1 -- 172.21.15.45:0/263776835 >> 172.21.15.45:0/263776835 conn(0x7fb9bc067e00 msgr2=0x7fb9bc15d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:24.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:24.450+0000 7fb9c0a00640 1 -- 172.21.15.45:0/263776835 shutdown_connections 2024-04-16T16:19:24.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:24.451+0000 7fb9c0a00640 1 -- 172.21.15.45:0/263776835 wait complete. 2024-04-16T16:19:24.471 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-16T16:19:24.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:19:24.582 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:19:24.678 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-16T16:19:24.678 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:19:24.678 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:19:24.678 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:19:24.714 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:19:24.732 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:19:24.734 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:19:24.734 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:19:24.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:19:25.003 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:25.004+0000 7f8a1ba4b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:19:25.008 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:25.009+0000 7f4fde20f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:19:25.062 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:25.063+0000 7f8a1ba4b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:19:25.063 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:25.064+0000 7f4fde20f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:19:25.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.089+0000 7f706d400640 1 Processor -- start 2024-04-16T16:19:25.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.089+0000 7f706d400640 1 -- start start 2024-04-16T16:19:25.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.091+0000 7f706d400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f70680b35d0 0x7f7068163220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:25.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.091+0000 7f706d400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7068163920 con 0x7f70680d24d0 2024-04-16T16:19:25.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.091+0000 7f706d400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f706806cc50 con 0x7f706815eea0 2024-04-16T16:19:25.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.091+0000 7f706d400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f70680d0940 con 0x7f70680b35d0 2024-04-16T16:19:25.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.091+0000 7f705de00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f70680d24d0 0x7f70680d0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43418/0 (socket says 172.21.15.45:43418) 2024-04-16T16:19:25.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.091+0000 7f705de00640 1 -- 172.21.15.45:0/1372522565 learned_addr learned my addr 172.21.15.45:0/1372522565 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:25.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.091+0000 7f7067200640 1 --2- 172.21.15.45:0/1372522565 >> v2:172.21.15.103:3300/0 conn(0x7f70680b35d0 0x7f7068163220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:25.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.091+0000 7f7065e00640 1 -- 172.21.15.45:0/1372522565 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3507058304 0 0) 0x7f7068163920 con 0x7f70680d24d0 2024-04-16T16:19:25.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.092+0000 7f7065e00640 1 -- 172.21.15.45:0/1372522565 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f704c003520 con 0x7f70680d24d0 2024-04-16T16:19:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.092+0000 7f7065e00640 1 -- 172.21.15.45:0/1372522565 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 466618894 0 0) 0x7f704c003520 con 0x7f70680d24d0 2024-04-16T16:19:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.092+0000 7f7065e00640 1 -- 172.21.15.45:0/1372522565 >> v2:172.21.15.103:3300/0 conn(0x7f70680b35d0 msgr2=0x7f7068163220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.092+0000 7f7065e00640 1 --2- 172.21.15.45:0/1372522565 >> v2:172.21.15.103:3300/0 conn(0x7f70680b35d0 0x7f7068163220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.092+0000 7f7065e00640 1 -- 172.21.15.45:0/1372522565 >> v1:172.21.15.103:6789/0 conn(0x7f706815eea0 legacy=0x7f7068162b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.092+0000 7f7065e00640 1 -- 172.21.15.45:0/1372522565 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f706806cf30 con 0x7f70680d24d0 2024-04-16T16:19:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.092+0000 7f7065e00640 1 -- 172.21.15.45:0/1372522565 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f7050001670 con 0x7f70680d24d0 2024-04-16T16:19:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.092+0000 7f7065e00640 1 -- 172.21.15.45:0/1372522565 >> v1:172.21.15.45:6789/0 conn(0x7f70680d24d0 legacy=0x7f70680d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.092+0000 7f7065e00640 1 --2- 172.21.15.45:0/1372522565 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f704c003a00 0x7f704c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.092+0000 7f7065e00640 1 --2- 172.21.15.45:0/1372522565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f704c004800 0x7f704c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.092+0000 7f7065e00640 1 --2- 172.21.15.45:0/1372522565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f704c0051c0 0x7f704c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.092+0000 7f7065e00640 1 -- 172.21.15.45:0/1372522565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f704c00c090 con 0x7f704c004800 2024-04-16T16:19:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.092+0000 7f7065e00640 1 -- 172.21.15.45:0/1372522565 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f704c00c240 con 0x7f704c0051c0 2024-04-16T16:19:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.092+0000 7f7065e00640 1 -- 172.21.15.45:0/1372522565 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f704c00c440 con 0x7f704c003a00 2024-04-16T16:19:25.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.092+0000 7f705de00640 1 --2- 172.21.15.45:0/1372522565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f704c004800 0x7f704c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:25.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.093+0000 7f7067200640 1 --2- 172.21.15.45:0/1372522565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f704c0051c0 0x7f704c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:25.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.093+0000 7f7066800640 1 --2- 172.21.15.45:0/1372522565 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f704c003a00 0x7f704c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:25.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.093+0000 7f705de00640 1 -- 172.21.15.45:0/1372522565 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f704c003a00 msgr2=0x7f704c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:25.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.093+0000 7f705de00640 1 --2- 172.21.15.45:0/1372522565 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f704c003a00 0x7f704c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.093+0000 7f705de00640 1 -- 172.21.15.45:0/1372522565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f704c0051c0 msgr2=0x7f704c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:25.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.093+0000 7f705de00640 1 --2- 172.21.15.45:0/1372522565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f704c0051c0 0x7f704c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.093+0000 7f705de00640 1 -- 172.21.15.45:0/1372522565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f704c003520 con 0x7f704c004800 2024-04-16T16:19:25.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.093+0000 7f7067200640 1 -- 172.21.15.45:0/1372522565 reap_dead start 2024-04-16T16:19:25.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.094+0000 7f705de00640 1 --2- 172.21.15.45:0/1372522565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f704c004800 0x7f704c004c80 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f705000c190 tx=0x7f704c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d43916f50f436bff server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.094+0000 7f7065e00640 1 -- 172.21.15.45:0/1372522565 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7050007c60 con 0x7f704c004800 2024-04-16T16:19:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.094+0000 7f7065e00640 1 -- 172.21.15.45:0/1372522565 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7050030070 con 0x7f704c004800 2024-04-16T16:19:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.094+0000 7f7065e00640 1 -- 172.21.15.45:0/1372522565 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7050005d90 con 0x7f704c004800 2024-04-16T16:19:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.095+0000 7f706d400640 1 -- 172.21.15.45:0/1372522565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f704c004800 msgr2=0x7f704c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.095+0000 7f706d400640 1 --2- 172.21.15.45:0/1372522565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f704c004800 0x7f704c004c80 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f705000c190 tx=0x7f704c005830 comp rx=0 tx=0).stop 2024-04-16T16:19:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.095+0000 7f706d400640 1 -- 172.21.15.45:0/1372522565 shutdown_connections 2024-04-16T16:19:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.095+0000 7f706d400640 1 --2- 172.21.15.45:0/1372522565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f704c004800 0x7f704c004c80 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.095+0000 7f706d400640 1 -- 172.21.15.45:0/1372522565 >> 172.21.15.45:0/1372522565 conn(0x7f70680d28b0 msgr2=0x7f70680d2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.095+0000 7f706d400640 1 -- 172.21.15.45:0/1372522565 shutdown_connections 2024-04-16T16:19:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.095+0000 7f706d400640 1 -- 172.21.15.45:0/1372522565 wait complete. 2024-04-16T16:19:25.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.097+0000 7f706d400640 1 Processor -- start 2024-04-16T16:19:25.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.097+0000 7f706d400640 1 -- start start 2024-04-16T16:19:25.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.098+0000 7f706d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f704c004800 0x7f706815b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:25.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.098+0000 7f706d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70681f8070 0x7f70680b6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:25.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.098+0000 7f706d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7068162f80 0x7f70680c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:25.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.098+0000 7f706d400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f706809ac30 con 0x7f7068162f80 2024-04-16T16:19:25.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.098+0000 7f706d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f706815f490 con 0x7f70681f8070 2024-04-16T16:19:25.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.098+0000 7f706d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f706809adb0 con 0x7f704c004800 2024-04-16T16:19:25.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.098+0000 7f7067200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7068162f80 0x7f70680c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:25.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.098+0000 7f7067200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7068162f80 0x7f70680c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34698/0 (socket says 172.21.15.45:34698) 2024-04-16T16:19:25.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.098+0000 7f7067200640 1 -- 172.21.15.45:0/1679704035 learned_addr learned my addr 172.21.15.45:0/1679704035 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.099+0000 7f705de00640 1 --2- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70681f8070 0x7f70680b6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.099+0000 7f7066800640 1 --2- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f704c004800 0x7f706815b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.099+0000 7f7067200640 1 -- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f704c004800 msgr2=0x7f706815b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.099+0000 7f7067200640 1 --2- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f704c004800 0x7f706815b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.099+0000 7f7067200640 1 -- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70681f8070 msgr2=0x7f70680b6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.099+0000 7f7067200640 1 --2- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70681f8070 0x7f70680b6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.099+0000 7f7067200640 1 -- 172.21.15.45:0/1679704035 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70681f85f0 con 0x7f7068162f80 2024-04-16T16:19:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.099+0000 7f7067200640 1 --2- 172.21.15.45:0/1679704035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7068162f80 0x7f70680c83f0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f7068073c30 tx=0x7f7060011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.099+0000 7f705f400640 1 -- 172.21.15.45:0/1679704035 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7060004240 con 0x7f7068162f80 2024-04-16T16:19:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.100+0000 7f705f400640 1 -- 172.21.15.45:0/1679704035 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f706001c070 con 0x7f7068162f80 2024-04-16T16:19:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.100+0000 7f706d400640 1 -- 172.21.15.45:0/1679704035 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70681f88d0 con 0x7f7068162f80 2024-04-16T16:19:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.100+0000 7f705f400640 1 -- 172.21.15.45:0/1679704035 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706000f740 con 0x7f7068162f80 2024-04-16T16:19:25.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.100+0000 7f706d400640 1 -- 172.21.15.45:0/1679704035 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70681f8e90 con 0x7f7068162f80 2024-04-16T16:19:25.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.101+0000 7f706d400640 1 -- 172.21.15.45:0/1679704035 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7068075780 con 0x7f7068162f80 2024-04-16T16:19:25.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.102+0000 7f705f400640 1 -- 172.21.15.45:0/1679704035 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) ==== 91086+0+0 (secure 0 0 0) 0x7f7060012040 con 0x7f7068162f80 2024-04-16T16:19:25.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.103+0000 7f705f400640 1 --2- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f704406c9b0 0x7f704406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:25.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.103+0000 7f705f400640 1 -- 172.21.15.45:0/1679704035 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6096+0+0 (secure 0 0 0) 0x7f706008bd30 con 0x7f7068162f80 2024-04-16T16:19:25.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.103+0000 7f7066800640 1 -- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f704406c9b0 msgr2=0x7f704406ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:25.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.103+0000 7f7066800640 1 --2- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f704406c9b0 0x7f704406ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:19:25.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.107+0000 7f705f400640 1 -- 172.21.15.45:0/1679704035 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7060059e90 con 0x7f7068162f80 2024-04-16T16:19:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.304+0000 7f7066800640 1 -- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f704406c9b0 msgr2=0x7f704406ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.304+0000 7f7066800640 1 --2- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f704406c9b0 0x7f704406ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:19:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.311+0000 7f706d400640 1 -- 172.21.15.45:0/1679704035 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70680d37c0 con 0x7f7068162f80 2024-04-16T16:19:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.314+0000 7f705f400640 1 -- 172.21.15.45:0/1679704035 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) ==== 74+0+17178 (secure 0 0 0) 0x7f7060059790 con 0x7f7068162f80 2024-04-16T16:19:25.313 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:25.313 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":101,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:19:22.373208+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:03.429683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:04.434699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6833/839990480":"2024-04-17T16:19:22.373133+0000","172.21.15.103:6832/839990480":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2646915013":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/1161670790":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/287602360":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2666528075":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/4022617615":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2967808514":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2879899995":"2024-04-16T17:19:09.753356+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:19:25.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.316+0000 7f706d400640 1 -- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f704406c9b0 msgr2=0x7f704406ee70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:19:25.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.316+0000 7f706d400640 1 --2- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f704406c9b0 0x7f704406ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.316+0000 7f706d400640 1 -- 172.21.15.45:0/1679704035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7068162f80 msgr2=0x7f70680c83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:25.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.317+0000 7f706d400640 1 --2- 172.21.15.45:0/1679704035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7068162f80 0x7f70680c83f0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f7068073c30 tx=0x7f7060011cf0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.317+0000 7f706d400640 1 -- 172.21.15.45:0/1679704035 shutdown_connections 2024-04-16T16:19:25.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.317+0000 7f706d400640 1 --2- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f704406c9b0 0x7f704406ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.317+0000 7f706d400640 1 --2- 172.21.15.45:0/1679704035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7068162f80 0x7f70680c83f0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.317+0000 7f706d400640 1 --2- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70681f8070 0x7f70680b6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.317+0000 7f706d400640 1 --2- 172.21.15.45:0/1679704035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f704c004800 0x7f706815b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.317+0000 7f706d400640 1 -- 172.21.15.45:0/1679704035 >> 172.21.15.45:0/1679704035 conn(0x7f70680d28b0 msgr2=0x7f70680af7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:25.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.317+0000 7f706d400640 1 -- 172.21.15.45:0/1679704035 shutdown_connections 2024-04-16T16:19:25.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.317+0000 7f706d400640 1 -- 172.21.15.45:0/1679704035 wait complete. 2024-04-16T16:19:25.335 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:19:03.429683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:19:04.434699+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:19:25.336 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:19:25.512 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:25.513+0000 7f4fde20f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:19:25.540 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:25.541+0000 7f8a1ba4b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:19:25.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.547+0000 7f9dffe00640 1 Processor -- start 2024-04-16T16:19:25.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.547+0000 7f9dffe00640 1 -- start start 2024-04-16T16:19:25.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.549+0000 7f9dffe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9e000d2bc0 0x7f9e000ba2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:25.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.549+0000 7f9dffe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e000d3020 con 0x7f9e000b6270 2024-04-16T16:19:25.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.549+0000 7f9dffe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e000d32c0 con 0x7f9e000b7920 2024-04-16T16:19:25.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.549+0000 7f9dffe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9e000ba7e0 con 0x7f9e000d2bc0 2024-04-16T16:19:25.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.549+0000 7f9dfe000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9e000d2bc0 0x7f9e000ba2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:25.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.549+0000 7f9dfea00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9e000b6270 0x7f9e000b6650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43432/0 (socket says 172.21.15.45:43432) 2024-04-16T16:19:25.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.549+0000 7f9dfea00640 1 -- 172.21.15.45:0/3164483448 learned_addr learned my addr 172.21.15.45:0/3164483448 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:25.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.549+0000 7f9dfe000640 1 -- 172.21.15.45:0/3164483448 >> v1:172.21.15.103:6789/0 conn(0x7f9e000b7920 legacy=0x7f9e000b7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:25.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.549+0000 7f9dfe000640 1 -- 172.21.15.45:0/3164483448 >> v1:172.21.15.45:6789/0 conn(0x7f9e000b6270 legacy=0x7f9e000b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:25.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.549+0000 7f9dfe000640 1 -- 172.21.15.45:0/3164483448 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e000d35a0 con 0x7f9e000d2bc0 2024-04-16T16:19:25.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.551+0000 7f9dfe000640 1 --2- 172.21.15.45:0/3164483448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e000d2bc0 0x7f9e000ba2a0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f9dec0097a0 tx=0x7f9dec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a42e1f8b0f9d4b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:25.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.551+0000 7f9dfd600640 1 -- 172.21.15.45:0/3164483448 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec035020 con 0x7f9e000d2bc0 2024-04-16T16:19:25.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.551+0000 7f9dfd600640 1 -- 172.21.15.45:0/3164483448 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dec031830 con 0x7f9e000d2bc0 2024-04-16T16:19:25.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.551+0000 7f9dfd600640 1 -- 172.21.15.45:0/3164483448 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec031c00 con 0x7f9e000d2bc0 2024-04-16T16:19:25.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.551+0000 7f9dffe00640 1 -- 172.21.15.45:0/3164483448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e000d2bc0 msgr2=0x7f9e000ba2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:25.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.551+0000 7f9dffe00640 1 --2- 172.21.15.45:0/3164483448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e000d2bc0 0x7f9e000ba2a0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f9dec0097a0 tx=0x7f9dec02b8d0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.552+0000 7f9dffe00640 1 -- 172.21.15.45:0/3164483448 shutdown_connections 2024-04-16T16:19:25.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.552+0000 7f9dffe00640 1 --2- 172.21.15.45:0/3164483448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e000d2bc0 0x7f9e000ba2a0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.552+0000 7f9dffe00640 1 -- 172.21.15.45:0/3164483448 >> 172.21.15.45:0/3164483448 conn(0x7f9e0006d300 msgr2=0x7f9e000d1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:25.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.552+0000 7f9dffe00640 1 -- 172.21.15.45:0/3164483448 shutdown_connections 2024-04-16T16:19:25.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.552+0000 7f9dffe00640 1 -- 172.21.15.45:0/3164483448 wait complete. 2024-04-16T16:19:25.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.553+0000 7f9dffe00640 1 Processor -- start 2024-04-16T16:19:25.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.553+0000 7f9dffe00640 1 -- start start 2024-04-16T16:19:25.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.554+0000 7f9dffe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e000b6270 0x7f9e00150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:25.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.554+0000 7f9dffe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e000b7920 0x7f9e000cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:25.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.554+0000 7f9dffe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e000d2bc0 0x7f9e000cf070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:25.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.554+0000 7f9dffe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9e000d3e40 con 0x7f9e000b6270 2024-04-16T16:19:25.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.554+0000 7f9dffe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e000d3b60 con 0x7f9e000b7920 2024-04-16T16:19:25.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.554+0000 7f9dffe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9e000d4360 con 0x7f9e000d2bc0 2024-04-16T16:19:25.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.555+0000 7f9dfea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e000b6270 0x7f9e00150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:25.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.555+0000 7f9dfea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e000b6270 0x7f9e00150ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34726/0 (socket says 172.21.15.45:34726) 2024-04-16T16:19:25.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.555+0000 7f9dfea00640 1 -- 172.21.15.45:0/2758374859 learned_addr learned my addr 172.21.15.45:0/2758374859 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:25.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.555+0000 7f9dfe000640 1 --2- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e000b7920 0x7f9e000cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:25.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.555+0000 7f9dfea00640 1 -- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e000d2bc0 msgr2=0x7f9e000cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:25.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.555+0000 7f9dff400640 1 --2- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e000d2bc0 0x7f9e000cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:25.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.555+0000 7f9dfea00640 1 --2- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e000d2bc0 0x7f9e000cf070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.555+0000 7f9dfea00640 1 -- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e000b7920 msgr2=0x7f9e000cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:25.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.555+0000 7f9dfea00640 1 --2- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e000b7920 0x7f9e000cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.555+0000 7f9dfea00640 1 -- 172.21.15.45:0/2758374859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e001f80a0 con 0x7f9e000b6270 2024-04-16T16:19:25.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.555+0000 7f9dfea00640 1 --2- 172.21.15.45:0/2758374859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e000b6270 0x7f9e00150ef0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f9df000c400 tx=0x7f9df000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:25.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.556+0000 7f9df6a00640 1 -- 172.21.15.45:0/2758374859 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0017070 con 0x7f9e000b6270 2024-04-16T16:19:25.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.556+0000 7f9df6a00640 1 -- 172.21.15.45:0/2758374859 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df0005ce0 con 0x7f9e000b6270 2024-04-16T16:19:25.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.556+0000 7f9df6a00640 1 -- 172.21.15.45:0/2758374859 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df00072f0 con 0x7f9e000b6270 2024-04-16T16:19:25.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.556+0000 7f9dffe00640 1 -- 172.21.15.45:0/2758374859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e001f8380 con 0x7f9e000b6270 2024-04-16T16:19:25.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.556+0000 7f9dffe00640 1 -- 172.21.15.45:0/2758374859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e001f8940 con 0x7f9e000b6270 2024-04-16T16:19:25.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.557+0000 7f9dffe00640 1 -- 172.21.15.45:0/2758374859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e00075780 con 0x7f9e000b6270 2024-04-16T16:19:25.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.558+0000 7f9df6a00640 1 -- 172.21.15.45:0/2758374859 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) ==== 91086+0+0 (secure 0 0 0) 0x7f9df001a070 con 0x7f9e000b6270 2024-04-16T16:19:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.559+0000 7f9df6a00640 1 --2- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f9de006c980 0x7f9de006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:25.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.560+0000 7f9df6a00640 1 -- 172.21.15.45:0/2758374859 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6096+0+0 (secure 0 0 0) 0x7f9df008b1e0 con 0x7f9e000b6270 2024-04-16T16:19:25.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.560+0000 7f9dfe000640 1 -- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f9de006c980 msgr2=0x7f9de006ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:25.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.560+0000 7f9dfe000640 1 --2- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f9de006c980 0x7f9de006ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:19:25.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.564+0000 7f9df6a00640 1 -- 172.21.15.45:0/2758374859 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9df00593c0 con 0x7f9e000b6270 2024-04-16T16:19:25.565 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:25.566+0000 7f4fde20f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:19:25.598 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:25.600+0000 7f8a1ba4b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:19:25.616 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:25.617+0000 7f4fde20f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:19:25.652 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:25.653+0000 7f8a1ba4b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:19:25.677 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:25.678+0000 7f4fde20f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:19:25.717 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:25.718+0000 7f8a1ba4b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:19:25.726 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:25.727+0000 7f4fde20f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:19:25.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.761+0000 7f9dfe000640 1 -- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f9de006c980 msgr2=0x7f9de006ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:25.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.761+0000 7f9dfe000640 1 --2- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f9de006c980 0x7f9de006ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:19:25.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.764+0000 7f9dffe00640 1 -- 172.21.15.45:0/2758374859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9e000b6860 con 0x7f9e000b6270 2024-04-16T16:19:25.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.764+0000 7f9df6a00640 1 -- 172.21.15.45:0/2758374859 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) ==== 93+0+10 (secure 0 0 0) 0x7f9df0058cc0 con 0x7f9e000b6270 2024-04-16T16:19:25.763 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:19:25.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.768+0000 7f9dffe00640 1 -- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f9de006c980 msgr2=0x7f9de006ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:19:25.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.768+0000 7f9dffe00640 1 --2- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f9de006c980 0x7f9de006ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.768+0000 7f9dffe00640 1 -- 172.21.15.45:0/2758374859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e000b6270 msgr2=0x7f9e00150ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:25.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.768+0000 7f9dffe00640 1 --2- 172.21.15.45:0/2758374859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e000b6270 0x7f9e00150ef0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f9df000c400 tx=0x7f9df000c8c0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.768+0000 7f9dffe00640 1 -- 172.21.15.45:0/2758374859 shutdown_connections 2024-04-16T16:19:25.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.768+0000 7f9dffe00640 1 --2- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f9de006c980 0x7f9de006ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.768+0000 7f9dffe00640 1 --2- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e000d2bc0 0x7f9e000cf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.768+0000 7f9dffe00640 1 --2- 172.21.15.45:0/2758374859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e000b7920 0x7f9e000cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.768+0000 7f9dffe00640 1 --2- 172.21.15.45:0/2758374859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e000b6270 0x7f9e00150ef0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:25.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.768+0000 7f9dffe00640 1 -- 172.21.15.45:0/2758374859 >> 172.21.15.45:0/2758374859 conn(0x7f9e0006d300 msgr2=0x7f9e000b42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:25.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.768+0000 7f9dffe00640 1 -- 172.21.15.45:0/2758374859 shutdown_connections 2024-04-16T16:19:25.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:25.768+0000 7f9dffe00640 1 -- 172.21.15.45:0/2758374859 wait complete. 2024-04-16T16:19:25.771 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:25.772+0000 7f8a1ba4b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:19:25.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:19:25.855 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:25.855+0000 7f4fde20f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:19:25.912 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:25.913+0000 7f8a1ba4b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:19:25.919 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:25.919+0000 7f4fde20f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:19:25.970 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:25.971+0000 7f4fde20f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:19:25.980 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:25.981+0000 7f8a1ba4b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:19:26.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.003+0000 7fb29fa00640 1 Processor -- start 2024-04-16T16:19:26.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.003+0000 7fb29fa00640 1 -- start start 2024-04-16T16:19:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.005+0000 7fb29fa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb2980d0e70 0x7fb2980d1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.005+0000 7fb29fa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2980cd5a0 0x7fb2980cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.005+0000 7fb29fa00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2980d0b70 con 0x7fb29815cd40 2024-04-16T16:19:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.005+0000 7fb29fa00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb2980d1790 con 0x7fb2980d0e70 2024-04-16T16:19:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.005+0000 7fb29fa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb29815d1e0 con 0x7fb2980cd5a0 2024-04-16T16:19:26.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.006+0000 7fb29d000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb2980d0e70 0x7fb2980d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:26.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.006+0000 7fb29d000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb2980d0e70 0x7fb2980d1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:56684/0 (socket says 172.21.15.45:56684) 2024-04-16T16:19:26.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.006+0000 7fb29d000640 1 -- 172.21.15.45:0/2608493784 learned_addr learned my addr 172.21.15.45:0/2608493784 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:26.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.006+0000 7fb29da00640 1 --2- 172.21.15.45:0/2608493784 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2980cd5a0 0x7fb2980cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:26.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.006+0000 7fb29d000640 1 -- 172.21.15.45:0/2608493784 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2980cd5a0 msgr2=0x7fb2980cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:26.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.006+0000 7fb29d000640 1 --2- 172.21.15.45:0/2608493784 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2980cd5a0 0x7fb2980cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.006+0000 7fb29d000640 1 -- 172.21.15.45:0/2608493784 >> v1:172.21.15.103:6789/0 conn(0x7fb29815cd40 legacy=0x7fb2980d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:26.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.006+0000 7fb29d000640 1 -- 172.21.15.45:0/2608493784 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29815d4c0 con 0x7fb2980d0e70 2024-04-16T16:19:26.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.008+0000 7fb29d000640 1 --2- 172.21.15.45:0/2608493784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2980d0e70 0x7fb2980d1250 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009870 tx=0x7fb28c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=351a5a5755d99d90 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:26.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.008+0000 7fb297400640 1 -- 172.21.15.45:0/2608493784 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c034070 con 0x7fb2980d0e70 2024-04-16T16:19:26.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.008+0000 7fb297400640 1 -- 172.21.15.45:0/2608493784 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb28c030840 con 0x7fb2980d0e70 2024-04-16T16:19:26.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.008+0000 7fb297400640 1 -- 172.21.15.45:0/2608493784 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c030c10 con 0x7fb2980d0e70 2024-04-16T16:19:26.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.009+0000 7fb29fa00640 1 -- 172.21.15.45:0/2608493784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2980d0e70 msgr2=0x7fb2980d1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:26.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.009+0000 7fb29fa00640 1 --2- 172.21.15.45:0/2608493784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2980d0e70 0x7fb2980d1250 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009870 tx=0x7fb28c02b980 comp rx=0 tx=0).stop 2024-04-16T16:19:26.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.009+0000 7fb29fa00640 1 -- 172.21.15.45:0/2608493784 shutdown_connections 2024-04-16T16:19:26.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.009+0000 7fb29fa00640 1 --2- 172.21.15.45:0/2608493784 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2980cd5a0 0x7fb2980cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.009+0000 7fb29fa00640 1 --2- 172.21.15.45:0/2608493784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2980d0e70 0x7fb2980d1250 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.009+0000 7fb29fa00640 1 -- 172.21.15.45:0/2608493784 >> 172.21.15.45:0/2608493784 conn(0x7fb2980d2890 msgr2=0x7fb2980d2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:26.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.009+0000 7fb29fa00640 1 -- 172.21.15.45:0/2608493784 shutdown_connections 2024-04-16T16:19:26.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.009+0000 7fb29fa00640 1 -- 172.21.15.45:0/2608493784 wait complete. 2024-04-16T16:19:26.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.011+0000 7fb29fa00640 1 Processor -- start 2024-04-16T16:19:26.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.012+0000 7fb29fa00640 1 -- start start 2024-04-16T16:19:26.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb29fa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2980cd5a0 0x7fb2981f44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb29fa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2980d0e70 0x7fb29806db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb29fa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb29815cd40 0x7fb29806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb29fa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb29809b7f0 con 0x7fb2980d0e70 2024-04-16T16:19:26.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb29fa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb29815da80 con 0x7fb2980cd5a0 2024-04-16T16:19:26.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb29fa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb29809b970 con 0x7fb29815cd40 2024-04-16T16:19:26.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb297e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2980d0e70 0x7fb29806db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:26.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb297e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2980d0e70 0x7fb29806db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34738/0 (socket says 172.21.15.45:34738) 2024-04-16T16:19:26.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb297e00640 1 -- 172.21.15.45:0/28725291 learned_addr learned my addr 172.21.15.45:0/28725291 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb29d000640 1 --2- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2980cd5a0 0x7fb2981f44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb29da00640 1 --2- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb29815cd40 0x7fb29806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb297e00640 1 -- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb29815cd40 msgr2=0x7fb29806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb297e00640 1 --2- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb29815cd40 0x7fb29806e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb297e00640 1 -- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2980cd5a0 msgr2=0x7fb2981f44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb297e00640 1 --2- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2980cd5a0 0x7fb2981f44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.013+0000 7fb297e00640 1 -- 172.21.15.45:0/28725291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2981491b0 con 0x7fb2980d0e70 2024-04-16T16:19:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.014+0000 7fb297e00640 1 --2- 172.21.15.45:0/28725291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2980d0e70 0x7fb29806db80 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fb28400c3c0 tx=0x7fb28400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.014+0000 7fb294c00640 1 -- 172.21.15.45:0/28725291 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2840052d0 con 0x7fb2980d0e70 2024-04-16T16:19:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.014+0000 7fb294c00640 1 -- 172.21.15.45:0/28725291 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb284005490 con 0x7fb2980d0e70 2024-04-16T16:19:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.014+0000 7fb294c00640 1 -- 172.21.15.45:0/28725291 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb284007260 con 0x7fb2980d0e70 2024-04-16T16:19:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.014+0000 7fb29fa00640 1 -- 172.21.15.45:0/28725291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2981f8300 con 0x7fb2980d0e70 2024-04-16T16:19:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.014+0000 7fb29fa00640 1 -- 172.21.15.45:0/28725291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2981f88c0 con 0x7fb2980d0e70 2024-04-16T16:19:26.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.015+0000 7fb29fa00640 1 -- 172.21.15.45:0/28725291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb298075780 con 0x7fb2980d0e70 2024-04-16T16:19:26.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.017+0000 7fb294c00640 1 -- 172.21.15.45:0/28725291 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) ==== 91086+0+0 (secure 0 0 0) 0x7fb284007400 con 0x7fb2980d0e70 2024-04-16T16:19:26.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.017+0000 7fb294c00640 1 --2- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb27806c9b0 0x7fb27806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.018+0000 7fb294c00640 1 -- 172.21.15.45:0/28725291 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6096+0+0 (secure 0 0 0) 0x7fb28408b1f0 con 0x7fb2980d0e70 2024-04-16T16:19:26.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.018+0000 7fb29d000640 1 -- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb27806c9b0 msgr2=0x7fb27806ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:26.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.018+0000 7fb29d000640 1 --2- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb27806c9b0 0x7fb27806ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:19:26.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.022+0000 7fb294c00640 1 -- 172.21.15.45:0/28725291 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb284059460 con 0x7fb2980d0e70 2024-04-16T16:19:26.029 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:26.029+0000 7f4fde20f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:19:26.035 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:26.037+0000 7f8a1ba4b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:19:26.079 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:26.080+0000 7f4fde20f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:19:26.096 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:26.097+0000 7f8a1ba4b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:19:26.150 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:26.151+0000 7f8a1ba4b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:19:26.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.219+0000 7fb29d000640 1 -- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb27806c9b0 msgr2=0x7fb27806ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:26.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.219+0000 7fb29d000640 1 --2- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb27806c9b0 0x7fb27806ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:19:26.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.224+0000 7fb29fa00640 1 -- 172.21.15.45:0/28725291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb2980d3710 con 0x7fb2980d0e70 2024-04-16T16:19:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.224+0000 7fb294c00640 1 -- 172.21.15.45:0/28725291 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) ==== 104+0+11 (secure 0 0 0) 0x7fb284058d60 con 0x7fb2980d0e70 2024-04-16T16:19:26.223 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:19:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.226+0000 7fb29fa00640 1 -- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb27806c9b0 msgr2=0x7fb27806ee70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:19:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.226+0000 7fb29fa00640 1 --2- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb27806c9b0 0x7fb27806ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.227+0000 7fb29fa00640 1 -- 172.21.15.45:0/28725291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2980d0e70 msgr2=0x7fb29806db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.227+0000 7fb29fa00640 1 --2- 172.21.15.45:0/28725291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2980d0e70 0x7fb29806db80 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fb28400c3c0 tx=0x7fb28400c880 comp rx=0 tx=0).stop 2024-04-16T16:19:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.227+0000 7fb29fa00640 1 -- 172.21.15.45:0/28725291 shutdown_connections 2024-04-16T16:19:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.227+0000 7fb29fa00640 1 --2- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7fb27806c9b0 0x7fb27806ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.227+0000 7fb29fa00640 1 --2- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb29815cd40 0x7fb29806e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.227+0000 7fb29fa00640 1 --2- 172.21.15.45:0/28725291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2980d0e70 0x7fb29806db80 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.227+0000 7fb29fa00640 1 --2- 172.21.15.45:0/28725291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2980cd5a0 0x7fb2981f44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.227+0000 7fb29fa00640 1 -- 172.21.15.45:0/28725291 >> 172.21.15.45:0/28725291 conn(0x7fb2980d2890 msgr2=0x7fb2980d0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:26.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.227+0000 7fb29fa00640 1 -- 172.21.15.45:0/28725291 shutdown_connections 2024-04-16T16:19:26.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.227+0000 7fb29fa00640 1 -- 172.21.15.45:0/28725291 wait complete. 2024-04-16T16:19:26.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:19:26.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.454+0000 7f2283200640 1 Processor -- start 2024-04-16T16:19:26.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.455+0000 7f2283200640 1 -- start start 2024-04-16T16:19:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.456+0000 7f2283200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f228415cbd0 0x7f228415d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.456+0000 7f2283200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f22840cd0a0 0x7f228409aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.456+0000 7f2283200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22840cd580 con 0x7f22840d0790 2024-04-16T16:19:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.456+0000 7f2283200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f228415df30 con 0x7f22840cd0a0 2024-04-16T16:19:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.456+0000 7f2283200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f22840cd860 con 0x7f228415cbd0 2024-04-16T16:19:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.457+0000 7f2282800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f22840cd0a0 0x7f228409aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.457+0000 7f2282800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f22840cd0a0 0x7f228409aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34754/0 (socket says 172.21.15.45:34754) 2024-04-16T16:19:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.457+0000 7f2282800640 1 -- 172.21.15.45:0/1920224525 learned_addr learned my addr 172.21.15.45:0/1920224525 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.457+0000 7f2281e00640 1 --1- 172.21.15.45:0/1920224525 >> v1:172.21.15.103:6789/0 conn(0x7f22840d0790 0x7f22840d0b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:47314/0 (socket says 172.21.15.45:47314) 2024-04-16T16:19:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.457+0000 7f2281400640 1 --2- 172.21.15.45:0/1920224525 >> v2:172.21.15.103:3300/0 conn(0x7f228415cbd0 0x7f228415d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:26.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.457+0000 7f2282800640 1 -- 172.21.15.45:0/1920224525 >> v2:172.21.15.103:3300/0 conn(0x7f228415cbd0 msgr2=0x7f228415d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:26.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.457+0000 7f2282800640 1 --2- 172.21.15.45:0/1920224525 >> v2:172.21.15.103:3300/0 conn(0x7f228415cbd0 0x7f228415d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.457+0000 7f2282800640 1 -- 172.21.15.45:0/1920224525 >> v1:172.21.15.103:6789/0 conn(0x7f22840d0790 legacy=0x7f22840d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:26.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.457+0000 7f2282800640 1 -- 172.21.15.45:0/1920224525 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22840cdb40 con 0x7f22840cd0a0 2024-04-16T16:19:26.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.458+0000 7f2282800640 1 --2- 172.21.15.45:0/1920224525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840cd0a0 0x7f228409aa70 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f22840733d0 tx=0x7f227c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce92cabad4518a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:26.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.459+0000 7f2280a00640 1 -- 172.21.15.45:0/1920224525 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c02ca80 con 0x7f22840cd0a0 2024-04-16T16:19:26.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.459+0000 7f2280a00640 1 -- 172.21.15.45:0/1920224525 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f227c02cc20 con 0x7f22840cd0a0 2024-04-16T16:19:26.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.459+0000 7f2280a00640 1 -- 172.21.15.45:0/1920224525 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c0343f0 con 0x7f22840cd0a0 2024-04-16T16:19:26.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.460+0000 7f2283200640 1 -- 172.21.15.45:0/1920224525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840cd0a0 msgr2=0x7f228409aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:26.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.460+0000 7f2283200640 1 --2- 172.21.15.45:0/1920224525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840cd0a0 0x7f228409aa70 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f22840733d0 tx=0x7f227c02bee0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.460+0000 7f2283200640 1 -- 172.21.15.45:0/1920224525 shutdown_connections 2024-04-16T16:19:26.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.460+0000 7f2283200640 1 --2- 172.21.15.45:0/1920224525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840cd0a0 0x7f228409aa70 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.460+0000 7f2283200640 1 --2- 172.21.15.45:0/1920224525 >> v2:172.21.15.103:3300/0 conn(0x7f228415cbd0 0x7f228415d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.460+0000 7f2283200640 1 -- 172.21.15.45:0/1920224525 >> 172.21.15.45:0/1920224525 conn(0x7f22840b40f0 msgr2=0x7f22840b4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:26.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.460+0000 7f2283200640 1 -- 172.21.15.45:0/1920224525 shutdown_connections 2024-04-16T16:19:26.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.460+0000 7f2283200640 1 -- 172.21.15.45:0/1920224525 wait complete. 2024-04-16T16:19:26.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.462+0000 7f2283200640 1 Processor -- start 2024-04-16T16:19:26.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.462+0000 7f2283200640 1 -- start start 2024-04-16T16:19:26.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.463+0000 7f2283200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840cd0a0 0x7f22840c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.463+0000 7f2283200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22840d0790 0x7f22840b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.463+0000 7f2283200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f228415cbd0 0x7f22841f5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.463+0000 7f2283200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f22840d0460 con 0x7f22840cd0a0 2024-04-16T16:19:26.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.463+0000 7f2283200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f22840ce100 con 0x7f22840d0790 2024-04-16T16:19:26.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.463+0000 7f2283200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f228409afb0 con 0x7f228415cbd0 2024-04-16T16:19:26.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.463+0000 7f2281e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840cd0a0 0x7f22840c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:26.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.463+0000 7f2281e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840cd0a0 0x7f22840c6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34758/0 (socket says 172.21.15.45:34758) 2024-04-16T16:19:26.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.463+0000 7f2281e00640 1 -- 172.21.15.45:0/689245889 learned_addr learned my addr 172.21.15.45:0/689245889 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:26.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.463+0000 7f2281400640 1 --2- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22840d0790 0x7f22840b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:26.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.463+0000 7f2282800640 1 --2- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f228415cbd0 0x7f22841f5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:26.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.463+0000 7f2281e00640 1 -- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f228415cbd0 msgr2=0x7f22841f5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:26.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.463+0000 7f2281e00640 1 --2- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f228415cbd0 0x7f22841f5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.464+0000 7f2281e00640 1 -- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22840d0790 msgr2=0x7f22840b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:26.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.464+0000 7f2281e00640 1 --2- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22840d0790 0x7f22840b6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.464+0000 7f2281e00640 1 -- 172.21.15.45:0/689245889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22840cb2d0 con 0x7f22840cd0a0 2024-04-16T16:19:26.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.464+0000 7f2281e00640 1 --2- 172.21.15.45:0/689245889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840cd0a0 0x7f22840c6130 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f227400c500 tx=0x7f227400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:26.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.464+0000 7f227a000640 1 -- 172.21.15.45:0/689245889 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274018070 con 0x7f22840cd0a0 2024-04-16T16:19:26.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.464+0000 7f227a000640 1 -- 172.21.15.45:0/689245889 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2274007500 con 0x7f22840cd0a0 2024-04-16T16:19:26.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.464+0000 7f227a000640 1 -- 172.21.15.45:0/689245889 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22740049e0 con 0x7f22840cd0a0 2024-04-16T16:19:26.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.464+0000 7f2283200640 1 -- 172.21.15.45:0/689245889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22840cb5b0 con 0x7f22840cd0a0 2024-04-16T16:19:26.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.464+0000 7f2283200640 1 -- 172.21.15.45:0/689245889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22840b7490 con 0x7f22840cd0a0 2024-04-16T16:19:26.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.466+0000 7f2283200640 1 -- 172.21.15.45:0/689245889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2284075780 con 0x7f22840cd0a0 2024-04-16T16:19:26.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.467+0000 7f227a000640 1 -- 172.21.15.45:0/689245889 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) ==== 91086+0+0 (secure 0 0 0) 0x7f2274004b80 con 0x7f22840cd0a0 2024-04-16T16:19:26.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.468+0000 7f227a000640 1 --2- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f226406c980 0x7f226406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.468+0000 7f227a000640 1 -- 172.21.15.45:0/689245889 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6096+0+0 (secure 0 0 0) 0x7f227408af90 con 0x7f22840cd0a0 2024-04-16T16:19:26.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.468+0000 7f2281400640 1 -- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f226406c980 msgr2=0x7f226406ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:26.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.468+0000 7f2281400640 1 --2- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f226406c980 0x7f226406ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:19:26.470 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:26.470+0000 7f4fde20f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:19:26.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.477+0000 7f227a000640 1 -- 172.21.15.45:0/689245889 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2274058bb0 con 0x7f22840cd0a0 2024-04-16T16:19:26.568 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:26.569+0000 7f8a1ba4b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:19:26.633 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:26.634+0000 7f4fde20f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:19:26.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.669+0000 7f2281400640 1 -- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f226406c980 msgr2=0x7f226406ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:26.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.669+0000 7f2281400640 1 --2- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f226406c980 0x7f226406ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:19:26.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.678+0000 7f2283200640 1 -- 172.21.15.45:0/689245889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f22840b5000 con 0x7f22840cd0a0 2024-04-16T16:19:26.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.679+0000 7f227a000640 1 -- 172.21.15.45:0/689245889 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) ==== 100+0+11 (secure 0 0 0) 0x7f227405ceb0 con 0x7f22840cd0a0 2024-04-16T16:19:26.678 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:19:26.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.682+0000 7f2283200640 1 -- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f226406c980 msgr2=0x7f226406ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:19:26.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.682+0000 7f2283200640 1 --2- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f226406c980 0x7f226406ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.682+0000 7f2283200640 1 -- 172.21.15.45:0/689245889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840cd0a0 msgr2=0x7f22840c6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:26.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.682+0000 7f2283200640 1 --2- 172.21.15.45:0/689245889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840cd0a0 0x7f22840c6130 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f227400c500 tx=0x7f227400c9c0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.682+0000 7f2283200640 1 -- 172.21.15.45:0/689245889 shutdown_connections 2024-04-16T16:19:26.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.682+0000 7f2283200640 1 --2- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f226406c980 0x7f226406ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.682+0000 7f2283200640 1 --2- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f228415cbd0 0x7f22841f5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.682+0000 7f2283200640 1 --2- 172.21.15.45:0/689245889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22840d0790 0x7f22840b6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.682+0000 7f2283200640 1 --2- 172.21.15.45:0/689245889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840cd0a0 0x7f22840c6130 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.682+0000 7f2283200640 1 -- 172.21.15.45:0/689245889 >> 172.21.15.45:0/689245889 conn(0x7f22840b40f0 msgr2=0x7f22840b2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:26.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.682+0000 7f2283200640 1 -- 172.21.15.45:0/689245889 shutdown_connections 2024-04-16T16:19:26.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.682+0000 7f2283200640 1 -- 172.21.15.45:0/689245889 wait complete. 2024-04-16T16:19:26.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:19:26.738 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:26.739+0000 7f8a1ba4b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:19:26.742 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:26.743+0000 7f4fde20f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:19:26.793 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:26.793+0000 7f4fde20f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:19:26.854 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:26.856+0000 7f8a1ba4b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:19:26.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.900+0000 7f5542400640 1 Processor -- start 2024-04-16T16:19:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.900+0000 7f5542400640 1 -- start start 2024-04-16T16:19:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.901+0000 7f5542400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f553c15ea70 0x7f553c1626f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.901+0000 7f5542400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f553c162f40 con 0x7f553c0ba5a0 2024-04-16T16:19:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.901+0000 7f5542400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f553c163220 con 0x7f553c0cfdf0 2024-04-16T16:19:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.901+0000 7f5542400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f553c163500 con 0x7f553c15ea70 2024-04-16T16:19:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.902+0000 7f553be00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f553c0ba5a0 0x7f553c0cf6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43448/0 (socket says 172.21.15.45:43448) 2024-04-16T16:19:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.902+0000 7f553be00640 1 -- 172.21.15.45:0/2191372217 learned_addr learned my addr 172.21.15.45:0/2191372217 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.902+0000 7f5541000640 1 --2- 172.21.15.45:0/2191372217 >> v2:172.21.15.103:3300/0 conn(0x7f553c15ea70 0x7f553c1626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.902+0000 7f553b400640 1 -- 172.21.15.45:0/2191372217 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2244169972 0 0) 0x7f553c162f40 con 0x7f553c0ba5a0 2024-04-16T16:19:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.902+0000 7f553b400640 1 -- 172.21.15.45:0/2191372217 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5520003520 con 0x7f553c0ba5a0 2024-04-16T16:19:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f553b400640 1 -- 172.21.15.45:0/2191372217 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 946531071 0 0) 0x7f5520003520 con 0x7f553c0ba5a0 2024-04-16T16:19:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f553b400640 1 -- 172.21.15.45:0/2191372217 >> v2:172.21.15.103:3300/0 conn(0x7f553c15ea70 msgr2=0x7f553c1626f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f553b400640 1 --2- 172.21.15.45:0/2191372217 >> v2:172.21.15.103:3300/0 conn(0x7f553c15ea70 0x7f553c1626f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f553b400640 1 -- 172.21.15.45:0/2191372217 >> v1:172.21.15.103:6789/0 conn(0x7f553c0cfdf0 legacy=0x7f553c0d3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f553b400640 1 -- 172.21.15.45:0/2191372217 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f553c0d2d50 con 0x7f553c0ba5a0 2024-04-16T16:19:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f553b400640 1 -- 172.21.15.45:0/2191372217 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f5524001670 con 0x7f553c0ba5a0 2024-04-16T16:19:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f553b400640 1 -- 172.21.15.45:0/2191372217 >> v1:172.21.15.45:6789/0 conn(0x7f553c0ba5a0 legacy=0x7f553c0cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f553b400640 1 --2- 172.21.15.45:0/2191372217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5520003a00 0x7f5520003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f553b400640 1 --2- 172.21.15.45:0/2191372217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5520004800 0x7f5520004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f553b400640 1 --2- 172.21.15.45:0/2191372217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f55200051c0 0x7f552000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f553b400640 1 -- 172.21.15.45:0/2191372217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f553c163500 con 0x7f5520003a00 2024-04-16T16:19:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f553b400640 1 -- 172.21.15.45:0/2191372217 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f552000c0a0 con 0x7f5520004800 2024-04-16T16:19:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f553b400640 1 -- 172.21.15.45:0/2191372217 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f552000c2a0 con 0x7f55200051c0 2024-04-16T16:19:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f5541000640 1 --2- 172.21.15.45:0/2191372217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5520003a00 0x7f5520003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f5541000640 1 -- 172.21.15.45:0/2191372217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f55200051c0 msgr2=0x7f552000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f5541000640 1 --2- 172.21.15.45:0/2191372217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f55200051c0 0x7f552000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f5541000640 1 -- 172.21.15.45:0/2191372217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5520004800 msgr2=0x7f5520004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f5541000640 1 --2- 172.21.15.45:0/2191372217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5520004800 0x7f5520004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.903+0000 7f5541000640 1 -- 172.21.15.45:0/2191372217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5520003520 con 0x7f5520003a00 2024-04-16T16:19:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.904+0000 7f5541a00640 1 -- 172.21.15.45:0/2191372217 reap_dead start 2024-04-16T16:19:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.905+0000 7f5541000640 1 --2- 172.21.15.45:0/2191372217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5520003a00 0x7f5520003e20 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f552c009fc0 tx=0x7f5520005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e4b57854fc9688 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.905+0000 7f553b400640 1 -- 172.21.15.45:0/2191372217 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552c039070 con 0x7f5520003a00 2024-04-16T16:19:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.905+0000 7f553b400640 1 -- 172.21.15.45:0/2191372217 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f552c02cc60 con 0x7f5520003a00 2024-04-16T16:19:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.905+0000 7f553b400640 1 -- 172.21.15.45:0/2191372217 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552c009c10 con 0x7f5520003a00 2024-04-16T16:19:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.905+0000 7f5542400640 1 -- 172.21.15.45:0/2191372217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5520003a00 msgr2=0x7f5520003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.905+0000 7f5542400640 1 --2- 172.21.15.45:0/2191372217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5520003a00 0x7f5520003e20 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f552c009fc0 tx=0x7f5520005830 comp rx=0 tx=0).stop 2024-04-16T16:19:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.906+0000 7f5542400640 1 -- 172.21.15.45:0/2191372217 shutdown_connections 2024-04-16T16:19:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.906+0000 7f5542400640 1 --2- 172.21.15.45:0/2191372217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5520003a00 0x7f5520003e20 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.906+0000 7f5542400640 1 -- 172.21.15.45:0/2191372217 >> 172.21.15.45:0/2191372217 conn(0x7f553c0d1780 msgr2=0x7f553c0d1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.906+0000 7f5542400640 1 -- 172.21.15.45:0/2191372217 shutdown_connections 2024-04-16T16:19:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.906+0000 7f5542400640 1 -- 172.21.15.45:0/2191372217 wait complete. 2024-04-16T16:19:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.907+0000 7f5542400640 1 Processor -- start 2024-04-16T16:19:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.908+0000 7f5542400640 1 -- start start 2024-04-16T16:19:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.908+0000 7f5542400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5520003a00 0x7f553c1f5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f5542400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f553c1f6020 0x7f553c0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f5542400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f553c0cd160 0x7f553c1f7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f5542400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f553c0b7c30 con 0x7f553c0cd160 2024-04-16T16:19:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f5542400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f553c0b7ab0 con 0x7f5520003a00 2024-04-16T16:19:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f5542400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f553c15ee50 con 0x7f553c1f6020 2024-04-16T16:19:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f5541a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f553c0cd160 0x7f553c1f7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f5541a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f553c0cd160 0x7f553c1f7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34770/0 (socket says 172.21.15.45:34770) 2024-04-16T16:19:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f5541a00640 1 -- 172.21.15.45:0/1640399905 learned_addr learned my addr 172.21.15.45:0/1640399905 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f5541000640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5520003a00 0x7f553c1f5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f553be00640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f553c1f6020 0x7f553c0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f5541a00640 1 -- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f553c1f6020 msgr2=0x7f553c0ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f5541a00640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f553c1f6020 0x7f553c0ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f5541a00640 1 -- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5520003a00 msgr2=0x7f553c1f5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f5541a00640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5520003a00 0x7f553c1f5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f5541a00640 1 -- 172.21.15.45:0/1640399905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f553c0cbb80 con 0x7f553c0cd160 2024-04-16T16:19:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f553be00640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f553c1f6020 0x7f553c0ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.909+0000 7f5541000640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5520003a00 0x7f553c1f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.910+0000 7f5541a00640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f553c0cd160 0x7f553c1f7f10 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f553c0cf160 tx=0x7f553400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.910+0000 7f5538c00640 1 -- 172.21.15.45:0/1640399905 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55340074e0 con 0x7f553c0cd160 2024-04-16T16:19:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.910+0000 7f5538c00640 1 -- 172.21.15.45:0/1640399905 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5534004020 con 0x7f553c0cd160 2024-04-16T16:19:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.910+0000 7f5538c00640 1 -- 172.21.15.45:0/1640399905 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5534004410 con 0x7f553c0cd160 2024-04-16T16:19:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.910+0000 7f5542400640 1 -- 172.21.15.45:0/1640399905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f553c0cbe60 con 0x7f553c0cd160 2024-04-16T16:19:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.910+0000 7f5542400640 1 -- 172.21.15.45:0/1640399905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f553c0cc420 con 0x7f553c0cd160 2024-04-16T16:19:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.911+0000 7f5542400640 1 -- 172.21.15.45:0/1640399905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f553c075780 con 0x7f553c0cd160 2024-04-16T16:19:26.911 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:26.912+0000 7f8a1ba4b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:19:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.912+0000 7f5538c00640 1 -- 172.21.15.45:0/1640399905 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) ==== 91086+0+0 (secure 0 0 0) 0x7f55340057c0 con 0x7f553c0cd160 2024-04-16T16:19:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.913+0000 7f5538c00640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f551806c980 0x7f551806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.913+0000 7f5538c00640 1 -- 172.21.15.45:0/1640399905 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6096+0+0 (secure 0 0 0) 0x7f553408aeb0 con 0x7f553c0cd160 2024-04-16T16:19:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.913+0000 7f5541000640 1 -- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f551806c980 msgr2=0x7f551806ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.913+0000 7f5541000640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f551806c980 0x7f551806ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:19:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:26.917+0000 7f5538c00640 1 -- 172.21.15.45:0/1640399905 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f55340591a0 con 0x7f553c0cd160 2024-04-16T16:19:26.920 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:26.921+0000 7f4fde20f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:19:27.028 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:27.028+0000 7f4fde20f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:19:27.046 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:27.047+0000 7f8a1ba4b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:19:27.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.114+0000 7f5541000640 1 -- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f551806c980 msgr2=0x7f551806ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:27.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.114+0000 7f5541000640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f551806c980 0x7f551806ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:19:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.130+0000 7f5542400640 1 -- 172.21.15.45:0/1640399905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f553c162bd0 con 0x7f553c0cd160 2024-04-16T16:19:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.131+0000 7f5538c00640 1 -- 172.21.15.45:0/1640399905 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 9 entries v101) ==== 80+0+534 (secure 0 0 0) 0x7f5534058aa0 con 0x7f553c0cd160 2024-04-16T16:19:27.130 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/839990480 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:27.130 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/839990480 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:27.130 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2646915013 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:27.130 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1161670790 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:27.130 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/287602360 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:27.130 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2666528075 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:27.130 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/4022617615 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:27.130 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2967808514 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:27.130 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2879899995 2024-04-16T17:19:09.753356+0000 2024-04-16T16:19:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:listed 9 entries 2024-04-16T16:19:27.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.134+0000 7f5542400640 1 -- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f551806c980 msgr2=0x7f551806ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:19:27.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.134+0000 7f5542400640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f551806c980 0x7f551806ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.134+0000 7f5542400640 1 -- 172.21.15.45:0/1640399905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f553c0cd160 msgr2=0x7f553c1f7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:27.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.134+0000 7f5542400640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f553c0cd160 0x7f553c1f7f10 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f553c0cf160 tx=0x7f553400c820 comp rx=0 tx=0).stop 2024-04-16T16:19:27.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.134+0000 7f5542400640 1 -- 172.21.15.45:0/1640399905 shutdown_connections 2024-04-16T16:19:27.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.134+0000 7f5542400640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f551806c980 0x7f551806ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.134+0000 7f5542400640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f553c0cd160 0x7f553c1f7f10 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.134+0000 7f5542400640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f553c1f6020 0x7f553c0ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.134+0000 7f5542400640 1 --2- 172.21.15.45:0/1640399905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5520003a00 0x7f553c1f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.134+0000 7f5542400640 1 -- 172.21.15.45:0/1640399905 >> 172.21.15.45:0/1640399905 conn(0x7f553c0d1780 msgr2=0x7f553c1624c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:27.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.134+0000 7f5542400640 1 -- 172.21.15.45:0/1640399905 shutdown_connections 2024-04-16T16:19:27.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.134+0000 7f5542400640 1 -- 172.21.15.45:0/1640399905 wait complete. 2024-04-16T16:19:27.150 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-16T16:19:27.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:27.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-16T16:19:27.160 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:27.161+0000 7f8a1ba4b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:19:27.202 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:27.202+0000 7f4fde20f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:19:27.251 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:27.252+0000 7f4fde20f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:19:27.263 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:19:27.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:27.263 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-16T16:19:27.310 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:19:27.310 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:19:27.310 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:19:27.353 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:27.354+0000 7f8a1ba4b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:19:27.361 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:19:27.361 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:19:27.394 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:27.395+0000 7f4fde20f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:19:27.405 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:27.406+0000 7f8a1ba4b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:19:27.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:19:27.553 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:27.554+0000 7f8a1ba4b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:19:27.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.651+0000 7f6377e00640 1 Processor -- start 2024-04-16T16:19:27.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.651+0000 7f6377e00640 1 -- start start 2024-04-16T16:19:27.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.652+0000 7f6377e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6378161ff0 0x7f63780cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:27.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.653+0000 7f6377e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f63780cfd20 0x7f63780b6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:27.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.653+0000 7f6377e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63780d46e0 con 0x7f63780b6c70 2024-04-16T16:19:27.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.653+0000 7f6377e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f63780b70a0 con 0x7f63780cfd20 2024-04-16T16:19:27.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.653+0000 7f6377e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f63780ba560 con 0x7f6378161ff0 2024-04-16T16:19:27.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.653+0000 7f6376000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f63780cfd20 0x7f63780b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:27.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.653+0000 7f6377400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f63780b6c70 0x7f637806cd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43450/0 (socket says 172.21.15.45:43450) 2024-04-16T16:19:27.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.653+0000 7f6377400640 1 -- 172.21.15.45:0/3182612238 learned_addr learned my addr 172.21.15.45:0/3182612238 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:27.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.653+0000 7f6376a00640 1 --2- 172.21.15.45:0/3182612238 >> v2:172.21.15.103:3300/0 conn(0x7f6378161ff0 0x7f63780cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:27.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.654+0000 7f6376000640 1 -- 172.21.15.45:0/3182612238 >> v2:172.21.15.103:3300/0 conn(0x7f6378161ff0 msgr2=0x7f63780cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:27.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.654+0000 7f6376000640 1 --2- 172.21.15.45:0/3182612238 >> v2:172.21.15.103:3300/0 conn(0x7f6378161ff0 0x7f63780cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.654+0000 7f6376000640 1 -- 172.21.15.45:0/3182612238 >> v1:172.21.15.45:6789/0 conn(0x7f63780b6c70 legacy=0x7f637806cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:27.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.654+0000 7f6376000640 1 -- 172.21.15.45:0/3182612238 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63780d3640 con 0x7f63780cfd20 2024-04-16T16:19:27.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.655+0000 7f6376000640 1 --2- 172.21.15.45:0/3182612238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63780cfd20 0x7f63780b6730 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f637808d360 tx=0x7f636002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c99ed60a8f7b071 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:27.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.655+0000 7f6375600640 1 -- 172.21.15.45:0/3182612238 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6360035020 con 0x7f63780cfd20 2024-04-16T16:19:27.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.655+0000 7f6375600640 1 -- 172.21.15.45:0/3182612238 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6360031830 con 0x7f63780cfd20 2024-04-16T16:19:27.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.655+0000 7f6375600640 1 -- 172.21.15.45:0/3182612238 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6360031c00 con 0x7f63780cfd20 2024-04-16T16:19:27.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.655+0000 7f6377e00640 1 -- 172.21.15.45:0/3182612238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63780cfd20 msgr2=0x7f63780b6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:27.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.656+0000 7f6377e00640 1 --2- 172.21.15.45:0/3182612238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63780cfd20 0x7f63780b6730 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f637808d360 tx=0x7f636002b8d0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.656+0000 7f6377e00640 1 -- 172.21.15.45:0/3182612238 shutdown_connections 2024-04-16T16:19:27.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.656+0000 7f6377e00640 1 --2- 172.21.15.45:0/3182612238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63780cfd20 0x7f63780b6730 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.656+0000 7f6377e00640 1 --2- 172.21.15.45:0/3182612238 >> v2:172.21.15.103:3300/0 conn(0x7f6378161ff0 0x7f63780cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.656+0000 7f6377e00640 1 -- 172.21.15.45:0/3182612238 >> 172.21.15.45:0/3182612238 conn(0x7f63780d17b0 msgr2=0x7f63780d1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:27.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.656+0000 7f6377e00640 1 -- 172.21.15.45:0/3182612238 shutdown_connections 2024-04-16T16:19:27.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.656+0000 7f6377e00640 1 -- 172.21.15.45:0/3182612238 wait complete. 2024-04-16T16:19:27.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.657+0000 7f6377e00640 1 Processor -- start 2024-04-16T16:19:27.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.658+0000 7f6377e00640 1 -- start start 2024-04-16T16:19:27.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.658+0000 7f6377e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63780b6c70 0x7f63780ade70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:27.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.658+0000 7f6377e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63780cfd20 0x7f6378150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:27.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.658+0000 7f6377e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6378161ff0 0x7f63780cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:27.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.658+0000 7f6377e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f63780d3ee0 con 0x7f63780b6c70 2024-04-16T16:19:27.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.658+0000 7f6377e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63780d3c00 con 0x7f63780cfd20 2024-04-16T16:19:27.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.658+0000 7f6377e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f63780d41c0 con 0x7f6378161ff0 2024-04-16T16:19:27.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.659+0000 7f6376a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63780b6c70 0x7f63780ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:27.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.659+0000 7f6376a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63780b6c70 0x7f63780ade70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34802/0 (socket says 172.21.15.45:34802) 2024-04-16T16:19:27.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.659+0000 7f6376a00640 1 -- 172.21.15.45:0/2061138815 learned_addr learned my addr 172.21.15.45:0/2061138815 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:27.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.659+0000 7f6376000640 1 --2- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63780cfd20 0x7f6378150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:27.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.659+0000 7f6377400640 1 --2- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6378161ff0 0x7f63780cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:27.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.659+0000 7f6376a00640 1 -- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6378161ff0 msgr2=0x7f63780cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:27.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.659+0000 7f6376a00640 1 --2- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6378161ff0 0x7f63780cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.659+0000 7f6376a00640 1 -- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63780cfd20 msgr2=0x7f6378150bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:27.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.659+0000 7f6376a00640 1 --2- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63780cfd20 0x7f6378150bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.659+0000 7f6376a00640 1 -- 172.21.15.45:0/2061138815 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63780cca30 con 0x7f63780b6c70 2024-04-16T16:19:27.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.660+0000 7f6376a00640 1 --2- 172.21.15.45:0/2061138815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63780b6c70 0x7f63780ade70 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f6368007ef0 tx=0x7f636800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:27.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.660+0000 7f636ea00640 1 -- 172.21.15.45:0/2061138815 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6368014620 con 0x7f63780b6c70 2024-04-16T16:19:27.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.660+0000 7f636ea00640 1 -- 172.21.15.45:0/2061138815 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6368019070 con 0x7f63780b6c70 2024-04-16T16:19:27.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.660+0000 7f636ea00640 1 -- 172.21.15.45:0/2061138815 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6368014a00 con 0x7f63780b6c70 2024-04-16T16:19:27.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.660+0000 7f6377e00640 1 -- 172.21.15.45:0/2061138815 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63780cccd0 con 0x7f63780b6c70 2024-04-16T16:19:27.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.660+0000 7f6377e00640 1 -- 172.21.15.45:0/2061138815 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63780cd570 con 0x7f63780b6c70 2024-04-16T16:19:27.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.661+0000 7f6377e00640 1 -- 172.21.15.45:0/2061138815 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63780ccea0 con 0x7f63780b6c70 2024-04-16T16:19:27.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.662+0000 7f636ea00640 1 -- 172.21.15.45:0/2061138815 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) ==== 91086+0+0 (secure 0 0 0) 0x7f6368014ba0 con 0x7f63780b6c70 2024-04-16T16:19:27.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.663+0000 7f636ea00640 1 --2- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f635406c980 0x7f635406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:27.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.663+0000 7f636ea00640 1 -- 172.21.15.45:0/2061138815 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6096+0+0 (secure 0 0 0) 0x7f636808c720 con 0x7f63780b6c70 2024-04-16T16:19:27.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.663+0000 7f6376000640 1 -- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f635406c980 msgr2=0x7f635406ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:27.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.663+0000 7f6376000640 1 --2- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f635406c980 0x7f635406ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:19:27.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.668+0000 7f636ea00640 1 -- 172.21.15.45:0/2061138815 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f636805a900 con 0x7f63780b6c70 2024-04-16T16:19:27.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.864+0000 7f6376000640 1 -- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f635406c980 msgr2=0x7f635406ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:27.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.864+0000 7f6376000640 1 --2- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f635406c980 0x7f635406ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:19:27.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.871+0000 7f6377e00640 1 -- 172.21.15.45:0/2061138815 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6378075780 con 0x7f63780b6c70 2024-04-16T16:19:27.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.873+0000 7f636ea00640 1 -- 172.21.15.45:0/2061138815 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) ==== 74+0+17178 (secure 0 0 0) 0x7f636805a200 con 0x7f63780b6c70 2024-04-16T16:19:27.872 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:27.872 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":101,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:19:22.373208+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:03.429683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:04.434699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6833/839990480":"2024-04-17T16:19:22.373133+0000","172.21.15.103:6832/839990480":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2646915013":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/1161670790":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/287602360":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2666528075":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/4022617615":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2967808514":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2879899995":"2024-04-16T17:19:09.753356+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:19:27.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.875+0000 7f6377e00640 1 -- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f635406c980 msgr2=0x7f635406ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:19:27.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.875+0000 7f6377e00640 1 --2- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f635406c980 0x7f635406ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.875+0000 7f6377e00640 1 -- 172.21.15.45:0/2061138815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63780b6c70 msgr2=0x7f63780ade70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:27.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.875+0000 7f6377e00640 1 --2- 172.21.15.45:0/2061138815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63780b6c70 0x7f63780ade70 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f6368007ef0 tx=0x7f636800e4b0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.876+0000 7f6377e00640 1 -- 172.21.15.45:0/2061138815 shutdown_connections 2024-04-16T16:19:27.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.876+0000 7f6377e00640 1 --2- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f635406c980 0x7f635406ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.876+0000 7f6377e00640 1 --2- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6378161ff0 0x7f63780cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.876+0000 7f6377e00640 1 --2- 172.21.15.45:0/2061138815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63780cfd20 0x7f6378150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.876+0000 7f6377e00640 1 --2- 172.21.15.45:0/2061138815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63780b6c70 0x7f63780ade70 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:27.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.876+0000 7f6377e00640 1 -- 172.21.15.45:0/2061138815 >> 172.21.15.45:0/2061138815 conn(0x7f63780d17b0 msgr2=0x7f63780b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:27.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.876+0000 7f6377e00640 1 -- 172.21.15.45:0/2061138815 shutdown_connections 2024-04-16T16:19:27.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:27.876+0000 7f6377e00640 1 -- 172.21.15.45:0/2061138815 wait complete. 2024-04-16T16:19:27.892 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:19:03.429683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:19:04.434699+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:19:27.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:19:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.098+0000 7f166b400640 1 Processor -- start 2024-04-16T16:19:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.098+0000 7f166b400640 1 -- start start 2024-04-16T16:19:28.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.100+0000 7f166b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f166406cf10 0x7f166406d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:28.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.100+0000 7f166b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f16640d35d0 0x7f16640d3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:28.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.100+0000 7f166b400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16640d2b10 con 0x7f166415ecc0 2024-04-16T16:19:28.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.100+0000 7f166b400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f166406d870 con 0x7f16640d35d0 2024-04-16T16:19:28.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.100+0000 7f166b400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1664163210 con 0x7f166406cf10 2024-04-16T16:19:28.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.100+0000 7f1669400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f16640d35d0 0x7f16640d3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:28.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.100+0000 7f1669400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f16640d35d0 0x7f16640d3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34810/0 (socket says 172.21.15.45:34810) 2024-04-16T16:19:28.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.100+0000 7f1669400640 1 -- 172.21.15.45:0/3016265663 learned_addr learned my addr 172.21.15.45:0/3016265663 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:28.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.100+0000 7f1663e00640 1 --2- 172.21.15.45:0/3016265663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f166406cf10 0x7f166406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:28.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.100+0000 7f1669400640 1 -- 172.21.15.45:0/3016265663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f166406cf10 msgr2=0x7f166406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:28.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.100+0000 7f1669400640 1 --2- 172.21.15.45:0/3016265663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f166406cf10 0x7f166406d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.100+0000 7f1669400640 1 -- 172.21.15.45:0/3016265663 >> v1:172.21.15.103:6789/0 conn(0x7f166415ecc0 legacy=0x7f16641626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:28.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.100+0000 7f1669400640 1 -- 172.21.15.45:0/3016265663 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16641634f0 con 0x7f16640d35d0 2024-04-16T16:19:28.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.102+0000 7f1669400640 1 --2- 172.21.15.45:0/3016265663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16640d35d0 0x7f16640d3a40 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f165c009e50 tx=0x7f165c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68234a90295cf0b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:28.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.102+0000 7f1663400640 1 -- 172.21.15.45:0/3016265663 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c02cb00 con 0x7f16640d35d0 2024-04-16T16:19:28.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.102+0000 7f1663400640 1 -- 172.21.15.45:0/3016265663 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f165c02cca0 con 0x7f16640d35d0 2024-04-16T16:19:28.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.102+0000 7f1663400640 1 -- 172.21.15.45:0/3016265663 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c034610 con 0x7f16640d35d0 2024-04-16T16:19:28.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.103+0000 7f166b400640 1 -- 172.21.15.45:0/3016265663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16640d35d0 msgr2=0x7f16640d3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:28.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.103+0000 7f166b400640 1 --2- 172.21.15.45:0/3016265663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16640d35d0 0x7f16640d3a40 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f165c009e50 tx=0x7f165c02bf60 comp rx=0 tx=0).stop 2024-04-16T16:19:28.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.103+0000 7f166b400640 1 -- 172.21.15.45:0/3016265663 shutdown_connections 2024-04-16T16:19:28.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.103+0000 7f166b400640 1 --2- 172.21.15.45:0/3016265663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16640d35d0 0x7f16640d3a40 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.103+0000 7f166b400640 1 --2- 172.21.15.45:0/3016265663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f166406cf10 0x7f166406d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.103+0000 7f166b400640 1 -- 172.21.15.45:0/3016265663 >> 172.21.15.45:0/3016265663 conn(0x7f1664073690 msgr2=0x7f16640d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:28.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.103+0000 7f166b400640 1 -- 172.21.15.45:0/3016265663 shutdown_connections 2024-04-16T16:19:28.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.103+0000 7f166b400640 1 -- 172.21.15.45:0/3016265663 wait complete. 2024-04-16T16:19:28.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.105+0000 7f166b400640 1 Processor -- start 2024-04-16T16:19:28.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.105+0000 7f166b400640 1 -- start start 2024-04-16T16:19:28.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.106+0000 7f166b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166406cf10 0x7f16641581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:28.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.106+0000 7f166b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16640d35d0 0x7f16640c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:28.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.106+0000 7f166b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f166415ecc0 0x7f16640c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:28.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.106+0000 7f166b400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f16640ba540 con 0x7f166415ecc0 2024-04-16T16:19:28.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.106+0000 7f166b400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f16640d30b0 con 0x7f166406cf10 2024-04-16T16:19:28.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.106+0000 7f166b400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1664162db0 con 0x7f16640d35d0 2024-04-16T16:19:28.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.107+0000 7f1669400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f166415ecc0 0x7f16640c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:28.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.107+0000 7f1669400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f166415ecc0 0x7f16640c9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34826/0 (socket says 172.21.15.45:34826) 2024-04-16T16:19:28.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.107+0000 7f1669400640 1 -- 172.21.15.45:0/2205992617 learned_addr learned my addr 172.21.15.45:0/2205992617 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:28.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.107+0000 7f1668a00640 1 --2- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166406cf10 0x7f16641581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:28.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.107+0000 7f1663e00640 1 --2- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16640d35d0 0x7f16640c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:28.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.107+0000 7f1669400640 1 -- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16640d35d0 msgr2=0x7f16640c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:28.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.107+0000 7f1669400640 1 --2- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16640d35d0 0x7f16640c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.107+0000 7f1669400640 1 -- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166406cf10 msgr2=0x7f16641581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:28.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.107+0000 7f1669400640 1 --2- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166406cf10 0x7f16641581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.107+0000 7f1669400640 1 -- 172.21.15.45:0/2205992617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16641586e0 con 0x7f166415ecc0 2024-04-16T16:19:28.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.107+0000 7f1663e00640 1 --2- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16640d35d0 0x7f16640c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:19:28.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.107+0000 7f1669400640 1 --2- 172.21.15.45:0/2205992617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f166415ecc0 0x7f16640c9f60 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f165c009e50 tx=0x7f165c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:28.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.108+0000 7f1660c00640 1 -- 172.21.15.45:0/2205992617 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c043020 con 0x7f166415ecc0 2024-04-16T16:19:28.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.108+0000 7f1660c00640 1 -- 172.21.15.45:0/2205992617 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f165c02ce50 con 0x7f166415ecc0 2024-04-16T16:19:28.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.108+0000 7f1660c00640 1 -- 172.21.15.45:0/2205992617 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c009510 con 0x7f166415ecc0 2024-04-16T16:19:28.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.108+0000 7f166b400640 1 -- 172.21.15.45:0/2205992617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16640cad30 con 0x7f166415ecc0 2024-04-16T16:19:28.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.108+0000 7f166b400640 1 -- 172.21.15.45:0/2205992617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16640cb2f0 con 0x7f166415ecc0 2024-04-16T16:19:28.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.109+0000 7f166b400640 1 -- 172.21.15.45:0/2205992617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f166415e520 con 0x7f166415ecc0 2024-04-16T16:19:28.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.110+0000 7f1660c00640 1 -- 172.21.15.45:0/2205992617 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) ==== 91086+0+0 (secure 0 0 0) 0x7f165c046020 con 0x7f166415ecc0 2024-04-16T16:19:28.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.112+0000 7f1660c00640 1 --2- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f164406c980 0x7f164406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:28.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.112+0000 7f1660c00640 1 -- 172.21.15.45:0/2205992617 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6096+0+0 (secure 0 0 0) 0x7f165c0ae400 con 0x7f166415ecc0 2024-04-16T16:19:28.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.112+0000 7f1668a00640 1 -- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f164406c980 msgr2=0x7f164406ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:28.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.112+0000 7f1668a00640 1 --2- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f164406c980 0x7f164406ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:19:28.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.116+0000 7f1660c00640 1 -- 172.21.15.45:0/2205992617 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f166415e520 con 0x7f166415ecc0 2024-04-16T16:19:28.253 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:28.253+0000 7f4fde20f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:19:28.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.313+0000 7f1668a00640 1 -- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f164406c980 msgr2=0x7f164406ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:28.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.313+0000 7f1668a00640 1 --2- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f164406c980 0x7f164406ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:19:28.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.319+0000 7f166b400640 1 -- 172.21.15.45:0/2205992617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1664075780 con 0x7f166415ecc0 2024-04-16T16:19:28.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.319+0000 7f1660c00640 1 -- 172.21.15.45:0/2205992617 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) ==== 93+0+10 (secure 0 0 0) 0x7f165c04d030 con 0x7f166415ecc0 2024-04-16T16:19:28.318 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:19:28.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.322+0000 7f166b400640 1 -- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f164406c980 msgr2=0x7f164406ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:19:28.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.322+0000 7f166b400640 1 --2- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f164406c980 0x7f164406ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.322+0000 7f166b400640 1 -- 172.21.15.45:0/2205992617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f166415ecc0 msgr2=0x7f16640c9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:28.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.322+0000 7f166b400640 1 --2- 172.21.15.45:0/2205992617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f166415ecc0 0x7f16640c9f60 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f165c009e50 tx=0x7f165c004a40 comp rx=0 tx=0).stop 2024-04-16T16:19:28.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.322+0000 7f166b400640 1 -- 172.21.15.45:0/2205992617 shutdown_connections 2024-04-16T16:19:28.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.322+0000 7f166b400640 1 --2- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7f164406c980 0x7f164406ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.322+0000 7f166b400640 1 --2- 172.21.15.45:0/2205992617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f166415ecc0 0x7f16640c9f60 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.322+0000 7f166b400640 1 --2- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16640d35d0 0x7f16640c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.322+0000 7f166b400640 1 --2- 172.21.15.45:0/2205992617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166406cf10 0x7f16641581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.322+0000 7f166b400640 1 -- 172.21.15.45:0/2205992617 >> 172.21.15.45:0/2205992617 conn(0x7f1664073690 msgr2=0x7f16640d3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:28.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.323+0000 7f166b400640 1 -- 172.21.15.45:0/2205992617 shutdown_connections 2024-04-16T16:19:28.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.323+0000 7f166b400640 1 -- 172.21.15.45:0/2205992617 wait complete. 2024-04-16T16:19:28.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:19:28.430 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:28.431+0000 7f8a1ba4b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:19:28.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.547+0000 7ff112400640 1 Processor -- start 2024-04-16T16:19:28.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.548+0000 7ff112400640 1 -- start start 2024-04-16T16:19:28.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.549+0000 7ff112400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff10c0d18f0 0x7ff10c0ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:28.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.549+0000 7ff112400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff10c0b6810 con 0x7ff10c0b6310 2024-04-16T16:19:28.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.549+0000 7ff112400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff10c0b6af0 con 0x7ff10c0d1cd0 2024-04-16T16:19:28.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.549+0000 7ff112400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff10c0d29f0 con 0x7ff10c0d18f0 2024-04-16T16:19:28.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.550+0000 7ff10be00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff10c0b6310 0x7ff10c0d2e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43454/0 (socket says 172.21.15.45:43454) 2024-04-16T16:19:28.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.550+0000 7ff10be00640 1 -- 172.21.15.45:0/551411632 learned_addr learned my addr 172.21.15.45:0/551411632 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:28.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.550+0000 7ff111000640 1 --2- 172.21.15.45:0/551411632 >> v2:172.21.15.103:3300/0 conn(0x7ff10c0d18f0 0x7ff10c0ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:28.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.550+0000 7ff10b400640 1 -- 172.21.15.45:0/551411632 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2949221798 0 0) 0x7ff10c0b6810 con 0x7ff10c0b6310 2024-04-16T16:19:28.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.550+0000 7ff10b400640 1 -- 172.21.15.45:0/551411632 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0f4003520 con 0x7ff10c0b6310 2024-04-16T16:19:28.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.550+0000 7ff10b400640 1 -- 172.21.15.45:0/551411632 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2435921752 0 0) 0x7ff0f4003520 con 0x7ff10c0b6310 2024-04-16T16:19:28.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.550+0000 7ff10b400640 1 -- 172.21.15.45:0/551411632 >> v2:172.21.15.103:3300/0 conn(0x7ff10c0d18f0 msgr2=0x7ff10c0ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.550+0000 7ff10b400640 1 --2- 172.21.15.45:0/551411632 >> v2:172.21.15.103:3300/0 conn(0x7ff10c0d18f0 0x7ff10c0ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.550+0000 7ff10b400640 1 -- 172.21.15.45:0/551411632 >> v1:172.21.15.103:6789/0 conn(0x7ff10c0d1cd0 legacy=0x7ff10c162dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.550+0000 7ff10b400640 1 -- 172.21.15.45:0/551411632 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10c0b6dd0 con 0x7ff10c0b6310 2024-04-16T16:19:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.550+0000 7ff10b400640 1 -- 172.21.15.45:0/551411632 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7ff0f8001670 con 0x7ff10c0b6310 2024-04-16T16:19:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.551+0000 7ff10b400640 1 -- 172.21.15.45:0/551411632 >> v1:172.21.15.45:6789/0 conn(0x7ff10c0b6310 legacy=0x7ff10c0d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.551+0000 7ff10b400640 1 --2- 172.21.15.45:0/551411632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f40039b0 0x7ff0f4003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.551+0000 7ff10b400640 1 --2- 172.21.15.45:0/551411632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0f4004800 0x7ff0f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.551+0000 7ff10b400640 1 --2- 172.21.15.45:0/551411632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f40053c0 0x7ff0f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.551+0000 7ff10b400640 1 -- 172.21.15.45:0/551411632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c0d29f0 con 0x7ff0f40053c0 2024-04-16T16:19:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.551+0000 7ff10b400640 1 -- 172.21.15.45:0/551411632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f400c260 con 0x7ff0f40039b0 2024-04-16T16:19:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.551+0000 7ff10b400640 1 -- 172.21.15.45:0/551411632 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff0f400c460 con 0x7ff0f4004800 2024-04-16T16:19:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.551+0000 7ff111a00640 1 --2- 172.21.15.45:0/551411632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f40053c0 0x7ff0f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.551+0000 7ff111000640 1 --2- 172.21.15.45:0/551411632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f40039b0 0x7ff0f4003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.551+0000 7ff10be00640 1 --2- 172.21.15.45:0/551411632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0f4004800 0x7ff0f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:28.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.551+0000 7ff111a00640 1 -- 172.21.15.45:0/551411632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0f4004800 msgr2=0x7ff0f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:28.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.551+0000 7ff111a00640 1 --2- 172.21.15.45:0/551411632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0f4004800 0x7ff0f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.551+0000 7ff111a00640 1 -- 172.21.15.45:0/551411632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f40039b0 msgr2=0x7ff0f4003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:28.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.551+0000 7ff111a00640 1 --2- 172.21.15.45:0/551411632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f40039b0 0x7ff0f4003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.551+0000 7ff111a00640 1 -- 172.21.15.45:0/551411632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f4003520 con 0x7ff0f40053c0 2024-04-16T16:19:28.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.552+0000 7ff111a00640 1 -- 172.21.15.45:0/551411632 reap_dead start 2024-04-16T16:19:28.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.553+0000 7ff111a00640 1 --2- 172.21.15.45:0/551411632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f40053c0 0x7ff0f400b9c0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7ff104001e50 tx=0x7ff0f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6213b62c6fecd990 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:28.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.553+0000 7ff10b400640 1 -- 172.21.15.45:0/551411632 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff104001a70 con 0x7ff0f40053c0 2024-04-16T16:19:28.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.553+0000 7ff10b400640 1 -- 172.21.15.45:0/551411632 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff104003b90 con 0x7ff0f40053c0 2024-04-16T16:19:28.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.553+0000 7ff10b400640 1 -- 172.21.15.45:0/551411632 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1040050d0 con 0x7ff0f40053c0 2024-04-16T16:19:28.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.553+0000 7ff112400640 1 -- 172.21.15.45:0/551411632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f40053c0 msgr2=0x7ff0f400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:28.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.553+0000 7ff112400640 1 --2- 172.21.15.45:0/551411632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f40053c0 0x7ff0f400b9c0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7ff104001e50 tx=0x7ff0f4005830 comp rx=0 tx=0).stop 2024-04-16T16:19:28.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.553+0000 7ff112400640 1 -- 172.21.15.45:0/551411632 shutdown_connections 2024-04-16T16:19:28.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.553+0000 7ff112400640 1 --2- 172.21.15.45:0/551411632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f40053c0 0x7ff0f400b9c0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.553+0000 7ff112400640 1 -- 172.21.15.45:0/551411632 >> 172.21.15.45:0/551411632 conn(0x7ff10c06cf90 msgr2=0x7ff10c06d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:28.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.554+0000 7ff112400640 1 -- 172.21.15.45:0/551411632 shutdown_connections 2024-04-16T16:19:28.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.554+0000 7ff112400640 1 -- 172.21.15.45:0/551411632 wait complete. 2024-04-16T16:19:28.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.555+0000 7ff112400640 1 Processor -- start 2024-04-16T16:19:28.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.555+0000 7ff112400640 1 -- start start 2024-04-16T16:19:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.556+0000 7ff112400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f40053c0 0x7ff10c15cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.556+0000 7ff112400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff10c0cacc0 0x7ff10c0c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.556+0000 7ff112400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff10c155c10 0x7ff10c0b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.556+0000 7ff112400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c0ba3a0 con 0x7ff0f40053c0 2024-04-16T16:19:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.556+0000 7ff112400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c0ba220 con 0x7ff10c0cacc0 2024-04-16T16:19:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.556+0000 7ff112400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff10c0ba520 con 0x7ff10c155c10 2024-04-16T16:19:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.556+0000 7ff111000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f40053c0 0x7ff10c15cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.556+0000 7ff111000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f40053c0 0x7ff10c15cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34842/0 (socket says 172.21.15.45:34842) 2024-04-16T16:19:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.556+0000 7ff111000640 1 -- 172.21.15.45:0/3566910778 learned_addr learned my addr 172.21.15.45:0/3566910778 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.557+0000 7ff111a00640 1 --2- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff10c155c10 0x7ff10c0b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.557+0000 7ff10be00640 1 --2- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff10c0cacc0 0x7ff10c0c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.557+0000 7ff111000640 1 -- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff10c155c10 msgr2=0x7ff10c0b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.557+0000 7ff111000640 1 --2- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff10c155c10 0x7ff10c0b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.557+0000 7ff111000640 1 -- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff10c0cacc0 msgr2=0x7ff10c0c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:28.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.557+0000 7ff111000640 1 --2- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff10c0cacc0 0x7ff10c0c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.557+0000 7ff111000640 1 -- 172.21.15.45:0/3566910778 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10c0c9cd0 con 0x7ff0f40053c0 2024-04-16T16:19:28.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.557+0000 7ff10be00640 1 --2- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff10c0cacc0 0x7ff10c0c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:28.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.557+0000 7ff111000640 1 --2- 172.21.15.45:0/3566910778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f40053c0 0x7ff10c15cff0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7ff10000f580 tx=0x7ff10000fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:28.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.558+0000 7ff108c00640 1 -- 172.21.15.45:0/3566910778 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff100013810 con 0x7ff0f40053c0 2024-04-16T16:19:28.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.558+0000 7ff108c00640 1 -- 172.21.15.45:0/3566910778 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff10001a070 con 0x7ff0f40053c0 2024-04-16T16:19:28.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.558+0000 7ff108c00640 1 -- 172.21.15.45:0/3566910778 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff100013bb0 con 0x7ff0f40053c0 2024-04-16T16:19:28.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.558+0000 7ff112400640 1 -- 172.21.15.45:0/3566910778 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff10c0c9fb0 con 0x7ff0f40053c0 2024-04-16T16:19:28.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.558+0000 7ff112400640 1 -- 172.21.15.45:0/3566910778 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff10c158880 con 0x7ff0f40053c0 2024-04-16T16:19:28.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.559+0000 7ff112400640 1 -- 172.21.15.45:0/3566910778 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff10c075780 con 0x7ff0f40053c0 2024-04-16T16:19:28.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.560+0000 7ff108c00640 1 -- 172.21.15.45:0/3566910778 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) ==== 91086+0+0 (secure 0 0 0) 0x7ff100012070 con 0x7ff0f40053c0 2024-04-16T16:19:28.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.562+0000 7ff108c00640 1 --2- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7ff0ec06c980 0x7ff0ec06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:28.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.562+0000 7ff108c00640 1 -- 172.21.15.45:0/3566910778 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6096+0+0 (secure 0 0 0) 0x7ff10008be00 con 0x7ff0f40053c0 2024-04-16T16:19:28.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.562+0000 7ff10be00640 1 -- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7ff0ec06c980 msgr2=0x7ff0ec06ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:28.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.562+0000 7ff10be00640 1 --2- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7ff0ec06c980 0x7ff0ec06ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:19:28.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.567+0000 7ff108c00640 1 -- 172.21.15.45:0/3566910778 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff10001c310 con 0x7ff0f40053c0 2024-04-16T16:19:28.582 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:28.583+0000 7f4fde20f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:19:28.641 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:28.641+0000 7f4fde20f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:19:28.759 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:28.760+0000 7f8a1ba4b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:19:28.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.763+0000 7ff10be00640 1 -- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7ff0ec06c980 msgr2=0x7ff0ec06ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/3948990093 2024-04-16T16:19:28.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.763+0000 7ff10be00640 1 --2- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7ff0ec06c980 0x7ff0ec06ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:19:28.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.766+0000 7ff112400640 1 -- 172.21.15.45:0/3566910778 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff10c0b55a0 con 0x7ff0f40053c0 2024-04-16T16:19:28.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.767+0000 7ff108c00640 1 -- 172.21.15.45:0/3566910778 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) ==== 104+0+11 (secure 0 0 0) 0x7ff100059ff0 con 0x7ff0f40053c0 2024-04-16T16:19:28.766 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:19:28.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.770+0000 7ff112400640 1 -- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7ff0ec06c980 msgr2=0x7ff0ec06ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:19:28.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.770+0000 7ff112400640 1 --2- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7ff0ec06c980 0x7ff0ec06ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.770+0000 7ff112400640 1 -- 172.21.15.45:0/3566910778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f40053c0 msgr2=0x7ff10c15cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:28.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.770+0000 7ff112400640 1 --2- 172.21.15.45:0/3566910778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f40053c0 0x7ff10c15cff0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7ff10000f580 tx=0x7ff10000fa40 comp rx=0 tx=0).stop 2024-04-16T16:19:28.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.770+0000 7ff112400640 1 -- 172.21.15.45:0/3566910778 shutdown_connections 2024-04-16T16:19:28.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.770+0000 7ff112400640 1 --2- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:6832/3948990093,v1:172.21.15.103:6833/3948990093] conn(0x7ff0ec06c980 0x7ff0ec06ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.770+0000 7ff112400640 1 --2- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff10c155c10 0x7ff10c0b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.770+0000 7ff112400640 1 --2- 172.21.15.45:0/3566910778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff10c0cacc0 0x7ff10c0c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.770+0000 7ff112400640 1 --2- 172.21.15.45:0/3566910778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f40053c0 0x7ff10c15cff0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:28.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.770+0000 7ff112400640 1 -- 172.21.15.45:0/3566910778 >> 172.21.15.45:0/3566910778 conn(0x7ff10c06cf90 msgr2=0x7ff10c0b7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:28.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.770+0000 7ff112400640 1 -- 172.21.15.45:0/3566910778 shutdown_connections 2024-04-16T16:19:28.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:28.770+0000 7ff112400640 1 -- 172.21.15.45:0/3566910778 wait complete. 2024-04-16T16:19:28.771 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:28.771+0000 7f4fde20f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:19:28.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:19:28.828 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:28.829+0000 7f8a1ba4b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:19:28.864 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:28.865+0000 7f4f5a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:28.864 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:28.865+0000 7f4f5a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:28.864 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:28.865+0000 7f4f5a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:28.864 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:28.865+0000 7f4f5a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:28.865 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:28.865+0000 7f4f5a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:28.865 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:28.866+0000 7f4f57a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:28.865 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:28.866+0000 7f4f57a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:28.865 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:28.866+0000 7f4f57a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:28.865 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:28.866+0000 7f4f57a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:28.865 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:28.866+0000 7f4f57a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:19:28.977 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:19:28.978+0000 7f8a1ba4b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:19:29.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.002+0000 7f22af200640 1 Processor -- start 2024-04-16T16:19:29.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.002+0000 7f22af200640 1 -- start start 2024-04-16T16:19:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.003+0000 7f22af200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f22b00d18f0 0x7f22b00ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.003+0000 7f22af200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22b00b6310 0x7f22b00d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.003+0000 7f22af200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22b00b6810 con 0x7f22b00d1cd0 2024-04-16T16:19:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.003+0000 7f22af200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f22b00d29f0 con 0x7f22b00d18f0 2024-04-16T16:19:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.003+0000 7f22af200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f22b00b6af0 con 0x7f22b00b6310 2024-04-16T16:19:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.004+0000 7f22ade00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f22b00d18f0 0x7f22b00ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.004+0000 7f22ade00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f22b00d18f0 0x7f22b00ba220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34854/0 (socket says 172.21.15.45:34854) 2024-04-16T16:19:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.004+0000 7f22ade00640 1 -- 172.21.15.45:0/517757394 learned_addr learned my addr 172.21.15.45:0/517757394 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.004+0000 7f22ad400640 1 --2- 172.21.15.45:0/517757394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22b00b6310 0x7f22b00d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.004+0000 7f22ade00640 1 -- 172.21.15.45:0/517757394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22b00b6310 msgr2=0x7f22b00d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.004+0000 7f22ade00640 1 --2- 172.21.15.45:0/517757394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22b00b6310 0x7f22b00d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.004+0000 7f22ade00640 1 -- 172.21.15.45:0/517757394 >> v1:172.21.15.103:6789/0 conn(0x7f22b00d1cd0 legacy=0x7f22b00d33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.004+0000 7f22ade00640 1 -- 172.21.15.45:0/517757394 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b00b6dd0 con 0x7f22b00d18f0 2024-04-16T16:19:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.005+0000 7f22ade00640 1 --2- 172.21.15.45:0/517757394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22b00d18f0 0x7f22b00ba220 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f22a0009870 tx=0x7f22a002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f86fd5610a99508 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.006+0000 7f22aca00640 1 -- 172.21.15.45:0/517757394 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a00317c0 con 0x7f22b00d18f0 2024-04-16T16:19:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.006+0000 7f22aca00640 1 -- 172.21.15.45:0/517757394 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22a0037070 con 0x7f22b00d18f0 2024-04-16T16:19:29.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.006+0000 7f22aca00640 1 -- 172.21.15.45:0/517757394 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a0031b80 con 0x7f22b00d18f0 2024-04-16T16:19:29.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.006+0000 7f22af200640 1 -- 172.21.15.45:0/517757394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22b00d18f0 msgr2=0x7f22b00ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:29.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.006+0000 7f22af200640 1 --2- 172.21.15.45:0/517757394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22b00d18f0 0x7f22b00ba220 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f22a0009870 tx=0x7f22a002b860 comp rx=0 tx=0).stop 2024-04-16T16:19:29.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.006+0000 7f22af200640 1 -- 172.21.15.45:0/517757394 shutdown_connections 2024-04-16T16:19:29.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.006+0000 7f22af200640 1 --2- 172.21.15.45:0/517757394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22b00b6310 0x7f22b00d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.006+0000 7f22af200640 1 --2- 172.21.15.45:0/517757394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22b00d18f0 0x7f22b00ba220 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.006+0000 7f22af200640 1 -- 172.21.15.45:0/517757394 >> 172.21.15.45:0/517757394 conn(0x7f22b006cf90 msgr2=0x7f22b006d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:29.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.006+0000 7f22af200640 1 -- 172.21.15.45:0/517757394 shutdown_connections 2024-04-16T16:19:29.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.006+0000 7f22af200640 1 -- 172.21.15.45:0/517757394 wait complete. 2024-04-16T16:19:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.008+0000 7f22af200640 1 Processor -- start 2024-04-16T16:19:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.008+0000 7f22af200640 1 -- start start 2024-04-16T16:19:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.009+0000 7f22af200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22b00b6310 0x7f22b014e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.009+0000 7f22af200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22b00d18f0 0x7f22b014ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.009+0000 7f22af200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22b00d1cd0 0x7f22b01f8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.009+0000 7f22af200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f22b0162dc0 con 0x7f22b00d18f0 2024-04-16T16:19:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.009+0000 7f22af200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f22b00ba760 con 0x7f22b00b6310 2024-04-16T16:19:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.009+0000 7f22af200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f22b0162f40 con 0x7f22b00d1cd0 2024-04-16T16:19:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.009+0000 7f22ad400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22b00d18f0 0x7f22b014ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.009+0000 7f22ad400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22b00d18f0 0x7f22b014ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34860/0 (socket says 172.21.15.45:34860) 2024-04-16T16:19:29.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.009+0000 7f22ad400640 1 -- 172.21.15.45:0/430725570 learned_addr learned my addr 172.21.15.45:0/430725570 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.010+0000 7f22ae800640 1 --2- 172.21.15.45:0/430725570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22b00d1cd0 0x7f22b01f8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.010+0000 7f22ade00640 1 --2- 172.21.15.45:0/430725570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22b00b6310 0x7f22b014e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.010+0000 7f22ad400640 1 -- 172.21.15.45:0/430725570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22b00d1cd0 msgr2=0x7f22b01f8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.010+0000 7f22ad400640 1 --2- 172.21.15.45:0/430725570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22b00d1cd0 0x7f22b01f8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.010+0000 7f22ad400640 1 -- 172.21.15.45:0/430725570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22b00b6310 msgr2=0x7f22b014e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.010+0000 7f22ad400640 1 --2- 172.21.15.45:0/430725570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22b00b6310 0x7f22b014e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.010+0000 7f22ad400640 1 -- 172.21.15.45:0/430725570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b00ca820 con 0x7f22b00d18f0 2024-04-16T16:19:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.010+0000 7f22ad400640 1 --2- 172.21.15.45:0/430725570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22b00d18f0 0x7f22b014ede0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f229c00d830 tx=0x7f229c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.010+0000 7f22a6000640 1 -- 172.21.15.45:0/430725570 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c00b6c0 con 0x7f22b00d18f0 2024-04-16T16:19:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.010+0000 7f22a6000640 1 -- 172.21.15.45:0/430725570 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f229c019070 con 0x7f22b00d18f0 2024-04-16T16:19:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.010+0000 7f22a6000640 1 -- 172.21.15.45:0/430725570 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c00ba80 con 0x7f22b00d18f0 2024-04-16T16:19:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.010+0000 7f22af200640 1 -- 172.21.15.45:0/430725570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b00cab00 con 0x7f22b00d18f0 2024-04-16T16:19:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.011+0000 7f22af200640 1 -- 172.21.15.45:0/430725570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b00cc840 con 0x7f22b00d18f0 2024-04-16T16:19:29.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.012+0000 7f22af200640 1 -- 172.21.15.45:0/430725570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22b0075780 con 0x7f22b00d18f0 2024-04-16T16:19:29.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.013+0000 7f22a6000640 1 -- 172.21.15.45:0/430725570 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 38) ==== 90751+0+0 (secure 0 0 0) 0x7f229c01e460 con 0x7f22b00d18f0 2024-04-16T16:19:29.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.014+0000 7f22a6000640 1 -- 172.21.15.45:0/430725570 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6397+0+0 (secure 0 0 0) 0x7f229c08bfb0 con 0x7f22b00d18f0 2024-04-16T16:19:29.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.019+0000 7f22a6000640 1 -- 172.21.15.45:0/430725570 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f229c05a580 con 0x7f22b00d18f0 2024-04-16T16:19:29.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.212+0000 7f22af200640 1 -- 172.21.15.45:0/430725570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f22b00d25a0 con 0x7f22b00d18f0 2024-04-16T16:19:29.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.213+0000 7f22a6000640 1 -- 172.21.15.45:0/430725570 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) ==== 100+0+11 (secure 0 0 0) 0x7f229c059e80 con 0x7f22b00d18f0 2024-04-16T16:19:29.212 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:19:29.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.215+0000 7f22af200640 1 -- 172.21.15.45:0/430725570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22b00d18f0 msgr2=0x7f22b014ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:29.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.215+0000 7f22af200640 1 --2- 172.21.15.45:0/430725570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22b00d18f0 0x7f22b014ede0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f229c00d830 tx=0x7f229c00dcf0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.216+0000 7f22af200640 1 -- 172.21.15.45:0/430725570 shutdown_connections 2024-04-16T16:19:29.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.216+0000 7f22af200640 1 --2- 172.21.15.45:0/430725570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22b00d1cd0 0x7f22b01f8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.216+0000 7f22af200640 1 --2- 172.21.15.45:0/430725570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22b00d18f0 0x7f22b014ede0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.216+0000 7f22af200640 1 --2- 172.21.15.45:0/430725570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22b00b6310 0x7f22b014e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.216+0000 7f22af200640 1 -- 172.21.15.45:0/430725570 >> 172.21.15.45:0/430725570 conn(0x7f22b006cf90 msgr2=0x7f22b015fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:29.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.216+0000 7f22af200640 1 -- 172.21.15.45:0/430725570 shutdown_connections 2024-04-16T16:19:29.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.216+0000 7f22af200640 1 -- 172.21.15.45:0/430725570 wait complete. 2024-04-16T16:19:29.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:19:29.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.498+0000 7f1691200640 1 Processor -- start 2024-04-16T16:19:29.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.498+0000 7f1691200640 1 -- start start 2024-04-16T16:19:29.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.500+0000 7f1691200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f168c06f410 0x7f168c163010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:29.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.500+0000 7f1691200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f168c0ce710 0x7f168c163ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:29.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.500+0000 7f1691200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f168c0cd210 con 0x7f168c0b1ce0 2024-04-16T16:19:29.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.500+0000 7f1691200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f168c0d02f0 con 0x7f168c0ce710 2024-04-16T16:19:29.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.500+0000 7f1691200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f168c0cd4f0 con 0x7f168c06f410 2024-04-16T16:19:29.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.501+0000 7f168b400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f168c0b1ce0 0x7f168c0b20c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43456/0 (socket says 172.21.15.45:43456) 2024-04-16T16:19:29.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.501+0000 7f168b400640 1 -- 172.21.15.45:0/2741681440 learned_addr learned my addr 172.21.15.45:0/2741681440 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:29.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.501+0000 7f168be00640 1 --2- 172.21.15.45:0/2741681440 >> v2:172.21.15.45:3300/0 conn(0x7f168c0ce710 0x7f168c163ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:29.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.501+0000 7f168aa00640 1 --2- 172.21.15.45:0/2741681440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f168c06f410 0x7f168c163010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:29.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.501+0000 7f168be00640 1 -- 172.21.15.45:0/2741681440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f168c06f410 msgr2=0x7f168c163010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:29.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.501+0000 7f168be00640 1 --2- 172.21.15.45:0/2741681440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f168c06f410 0x7f168c163010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.501+0000 7f168a000640 1 -- 172.21.15.45:0/2741681440 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1600490607 0 0) 0x7f168c0cd210 con 0x7f168c0b1ce0 2024-04-16T16:19:29.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.501+0000 7f168be00640 1 -- 172.21.15.45:0/2741681440 >> v1:172.21.15.45:6789/0 conn(0x7f168c0b1ce0 legacy=0x7f168c0b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:29.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.501+0000 7f168be00640 1 -- 172.21.15.45:0/2741681440 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f168c163680 con 0x7f168c0ce710 2024-04-16T16:19:29.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.501+0000 7f168aa00640 1 --2- 172.21.15.45:0/2741681440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f168c06f410 0x7f168c163010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:19:29.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.502+0000 7f168be00640 1 --2- 172.21.15.45:0/2741681440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f168c0ce710 0x7f168c163ba0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f1684009d90 tx=0x7f168402bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f537ca9c7074f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:29.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.503+0000 7f168a000640 1 -- 172.21.15.45:0/2741681440 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1684039070 con 0x7f168c0ce710 2024-04-16T16:19:29.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.503+0000 7f168a000640 1 -- 172.21.15.45:0/2741681440 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f168402cb40 con 0x7f168c0ce710 2024-04-16T16:19:29.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.503+0000 7f168a000640 1 -- 172.21.15.45:0/2741681440 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1684034440 con 0x7f168c0ce710 2024-04-16T16:19:29.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.503+0000 7f1691200640 1 -- 172.21.15.45:0/2741681440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f168c0ce710 msgr2=0x7f168c163ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.503+0000 7f1691200640 1 --2- 172.21.15.45:0/2741681440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f168c0ce710 0x7f168c163ba0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f1684009d90 tx=0x7f168402bd90 comp rx=0 tx=0).stop 2024-04-16T16:19:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.503+0000 7f1691200640 1 -- 172.21.15.45:0/2741681440 shutdown_connections 2024-04-16T16:19:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.503+0000 7f1691200640 1 --2- 172.21.15.45:0/2741681440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f168c0ce710 0x7f168c163ba0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.503+0000 7f1691200640 1 --2- 172.21.15.45:0/2741681440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f168c06f410 0x7f168c163010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.503+0000 7f1691200640 1 -- 172.21.15.45:0/2741681440 >> 172.21.15.45:0/2741681440 conn(0x7f168c06b930 msgr2=0x7f168c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.503+0000 7f1691200640 1 -- 172.21.15.45:0/2741681440 shutdown_connections 2024-04-16T16:19:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.503+0000 7f1691200640 1 -- 172.21.15.45:0/2741681440 wait complete. 2024-04-16T16:19:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.505+0000 7f1691200640 1 Processor -- start 2024-04-16T16:19:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.505+0000 7f1691200640 1 -- start start 2024-04-16T16:19:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.506+0000 7f1691200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f168c06f410 0x7f168c1578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.506+0000 7f1691200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f168c0b1ce0 0x7f168c15ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.506+0000 7f1691200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f168c0ce710 0x7f168c15b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.506+0000 7f1691200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f168c1640e0 con 0x7f168c06f410 2024-04-16T16:19:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.506+0000 7f1691200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f168c0d00d0 con 0x7f168c0b1ce0 2024-04-16T16:19:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.506+0000 7f1691200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f168c164260 con 0x7f168c0ce710 2024-04-16T16:19:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.506+0000 7f168b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f168c06f410 0x7f168c1578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.506+0000 7f168b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f168c06f410 0x7f168c1578c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34874/0 (socket says 172.21.15.45:34874) 2024-04-16T16:19:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.506+0000 7f168b400640 1 -- 172.21.15.45:0/2681923940 learned_addr learned my addr 172.21.15.45:0/2681923940 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.507+0000 7f168b400640 1 -- 172.21.15.45:0/2681923940 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f168c0ce710 msgr2=0x7f168c15b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.507+0000 7f168be00640 1 --2- 172.21.15.45:0/2681923940 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f168c0ce710 0x7f168c15b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.507+0000 7f168aa00640 1 --2- 172.21.15.45:0/2681923940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f168c0b1ce0 0x7f168c15ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.507+0000 7f168b400640 1 --2- 172.21.15.45:0/2681923940 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f168c0ce710 0x7f168c15b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.507+0000 7f168b400640 1 -- 172.21.15.45:0/2681923940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f168c0b1ce0 msgr2=0x7f168c15ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.507+0000 7f168b400640 1 --2- 172.21.15.45:0/2681923940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f168c0b1ce0 0x7f168c15ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.507+0000 7f168b400640 1 -- 172.21.15.45:0/2681923940 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f168c154b80 con 0x7f168c06f410 2024-04-16T16:19:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.507+0000 7f168b400640 1 --2- 172.21.15.45:0/2681923940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f168c06f410 0x7f168c1578c0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f167c00c4d0 tx=0x7f167c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.507+0000 7f1683400640 1 -- 172.21.15.45:0/2681923940 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167c018070 con 0x7f168c06f410 2024-04-16T16:19:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.507+0000 7f1683400640 1 -- 172.21.15.45:0/2681923940 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f167c007500 con 0x7f168c06f410 2024-04-16T16:19:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.507+0000 7f1683400640 1 -- 172.21.15.45:0/2681923940 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167c004a20 con 0x7f168c06f410 2024-04-16T16:19:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.507+0000 7f1691200640 1 -- 172.21.15.45:0/2681923940 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f168c15f260 con 0x7f168c06f410 2024-04-16T16:19:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.508+0000 7f1691200640 1 -- 172.21.15.45:0/2681923940 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f168c154e20 con 0x7f168c06f410 2024-04-16T16:19:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.509+0000 7f1691200640 1 -- 172.21.15.45:0/2681923940 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f168c063660 con 0x7f168c06f410 2024-04-16T16:19:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.510+0000 7f1683400640 1 -- 172.21.15.45:0/2681923940 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 38) ==== 90751+0+0 (secure 0 0 0) 0x7f167c004bc0 con 0x7f168c06f410 2024-04-16T16:19:29.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.510+0000 7f1683400640 1 -- 172.21.15.45:0/2681923940 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6397+0+0 (secure 0 0 0) 0x7f167c08ab50 con 0x7f168c06f410 2024-04-16T16:19:29.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.515+0000 7f1683400640 1 -- 172.21.15.45:0/2681923940 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f168c063660 con 0x7f168c06f410 2024-04-16T16:19:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.722+0000 7f1691200640 1 -- 172.21.15.45:0/2681923940 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f168c075780 con 0x7f168c06f410 2024-04-16T16:19:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.723+0000 7f1683400640 1 -- 172.21.15.45:0/2681923940 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v102) ==== 81+0+952 (secure 0 0 0) 0x7f167c059120 con 0x7f168c06f410 2024-04-16T16:19:29.722 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3000963222 2024-04-17T16:19:28.777448+0000 2024-04-16T16:19:29.722 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/963070403 2024-04-17T16:19:28.777448+0000 2024-04-16T16:19:29.722 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/3948990093 2024-04-17T16:19:28.777448+0000 2024-04-16T16:19:29.722 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2784388736 2024-04-17T16:19:28.777448+0000 2024-04-16T16:19:29.723 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2130767640 2024-04-17T16:19:28.777448+0000 2024-04-16T16:19:29.723 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/839990480 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:29.723 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/839990480 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:29.723 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2646915013 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:29.723 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1161670790 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:29.723 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/287602360 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:29.723 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2666528075 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:29.723 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/3948990093 2024-04-17T16:19:28.777448+0000 2024-04-16T16:19:29.723 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/4022617615 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:29.723 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2967808514 2024-04-17T16:19:22.373133+0000 2024-04-16T16:19:29.723 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1615159946 2024-04-17T16:19:28.777448+0000 2024-04-16T16:19:29.723 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2879899995 2024-04-16T17:19:09.753356+0000 2024-04-16T16:19:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:listed 16 entries 2024-04-16T16:19:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.726+0000 7f1691200640 1 -- 172.21.15.45:0/2681923940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f168c06f410 msgr2=0x7f168c1578c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.726+0000 7f1691200640 1 --2- 172.21.15.45:0/2681923940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f168c06f410 0x7f168c1578c0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f167c00c4d0 tx=0x7f167c00c990 comp rx=0 tx=0).stop 2024-04-16T16:19:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.726+0000 7f1691200640 1 -- 172.21.15.45:0/2681923940 shutdown_connections 2024-04-16T16:19:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.726+0000 7f1691200640 1 --2- 172.21.15.45:0/2681923940 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f168c0ce710 0x7f168c15b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.726+0000 7f1691200640 1 --2- 172.21.15.45:0/2681923940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f168c0b1ce0 0x7f168c15ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.726+0000 7f1691200640 1 --2- 172.21.15.45:0/2681923940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f168c06f410 0x7f168c1578c0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.726+0000 7f1691200640 1 -- 172.21.15.45:0/2681923940 >> 172.21.15.45:0/2681923940 conn(0x7f168c06b930 msgr2=0x7f168c0b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.726+0000 7f1691200640 1 -- 172.21.15.45:0/2681923940 shutdown_connections 2024-04-16T16:19:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:29.726+0000 7f1691200640 1 -- 172.21.15.45:0/2681923940 wait complete. 2024-04-16T16:19:29.744 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-16T16:19:29.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:29.744 DEBUG:teuthology.orchestra.run.smithi103:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-04-16T16:19:29.851 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:19:29.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:29.852 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-16T16:19:29.895 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:19:29.895 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:19:29.895 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:19:29.955 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:19:29.955 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:19:29.955 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:19:29.955 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:19:30.010 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:19:30.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:19:30.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.242+0000 7fa934e00640 1 Processor -- start 2024-04-16T16:19:30.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.242+0000 7fa934e00640 1 -- start start 2024-04-16T16:19:30.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.244+0000 7fa934e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa93015ea70 0x7fa9300cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:30.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.244+0000 7fa934e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9300cf6d0 con 0x7fa930072a90 2024-04-16T16:19:30.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.244+0000 7fa934e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9301466c0 con 0x7fa9301462e0 2024-04-16T16:19:30.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.244+0000 7fa934e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa9300cf3f0 con 0x7fa93015ea70 2024-04-16T16:19:30.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.244+0000 7fa927e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa930072a90 0x7fa9300ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43466/0 (socket says 172.21.15.45:43466) 2024-04-16T16:19:30.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.244+0000 7fa927e00640 1 -- 172.21.15.45:0/1757085553 learned_addr learned my addr 172.21.15.45:0/1757085553 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:30.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92e800640 1 --2- 172.21.15.45:0/1757085553 >> v2:172.21.15.103:3300/0 conn(0x7fa93015ea70 0x7fa9300cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92d400640 1 -- 172.21.15.45:0/1757085553 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3502784359 0 0) 0x7fa9300cf6d0 con 0x7fa930072a90 2024-04-16T16:19:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92d400640 1 -- 172.21.15.45:0/1757085553 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa914003520 con 0x7fa930072a90 2024-04-16T16:19:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92d400640 1 -- 172.21.15.45:0/1757085553 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3809092298 0 0) 0x7fa914003520 con 0x7fa930072a90 2024-04-16T16:19:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92d400640 1 -- 172.21.15.45:0/1757085553 >> v2:172.21.15.103:3300/0 conn(0x7fa93015ea70 msgr2=0x7fa9300cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92d400640 1 --2- 172.21.15.45:0/1757085553 >> v2:172.21.15.103:3300/0 conn(0x7fa93015ea70 0x7fa9300cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92d400640 1 -- 172.21.15.45:0/1757085553 >> v1:172.21.15.103:6789/0 conn(0x7fa9301462e0 legacy=0x7fa930072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92d400640 1 -- 172.21.15.45:0/1757085553 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa93015cf80 con 0x7fa930072a90 2024-04-16T16:19:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92d400640 1 -- 172.21.15.45:0/1757085553 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fa918001670 con 0x7fa930072a90 2024-04-16T16:19:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92d400640 1 -- 172.21.15.45:0/1757085553 >> v1:172.21.15.45:6789/0 conn(0x7fa930072a90 legacy=0x7fa9300ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92d400640 1 --2- 172.21.15.45:0/1757085553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa914003a00 0x7fa914003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92d400640 1 --2- 172.21.15.45:0/1757085553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa914004800 0x7fa914004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92d400640 1 --2- 172.21.15.45:0/1757085553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa914005210 0x7fa91400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92d400640 1 -- 172.21.15.45:0/1757085553 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa9300cf3f0 con 0x7fa914005210 2024-04-16T16:19:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92d400640 1 -- 172.21.15.45:0/1757085553 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa91400c160 con 0x7fa914004800 2024-04-16T16:19:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.245+0000 7fa92d400640 1 -- 172.21.15.45:0/1757085553 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa91400c360 con 0x7fa914003a00 2024-04-16T16:19:30.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.246+0000 7fa92e800640 1 --2- 172.21.15.45:0/1757085553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa914005210 0x7fa91400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:30.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.246+0000 7fa92de00640 1 --2- 172.21.15.45:0/1757085553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa914003a00 0x7fa914003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:30.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.246+0000 7fa92e800640 1 -- 172.21.15.45:0/1757085553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa914003a00 msgr2=0x7fa914003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.246+0000 7fa92e800640 1 --2- 172.21.15.45:0/1757085553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa914003a00 0x7fa914003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.246+0000 7fa92e800640 1 -- 172.21.15.45:0/1757085553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa914004800 msgr2=0x7fa914004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.246+0000 7fa92e800640 1 --2- 172.21.15.45:0/1757085553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa914004800 0x7fa914004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.246+0000 7fa92e800640 1 -- 172.21.15.45:0/1757085553 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa914003520 con 0x7fa914005210 2024-04-16T16:19:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.247+0000 7fa92e800640 1 -- 172.21.15.45:0/1757085553 reap_dead start 2024-04-16T16:19:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.247+0000 7fa92e800640 1 --2- 172.21.15.45:0/1757085553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa914005210 0x7fa91400b9c0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fa928009f90 tx=0x7fa914005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9bc430906bccde7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.248+0000 7fa92d400640 1 -- 172.21.15.45:0/1757085553 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa928009c10 con 0x7fa914005210 2024-04-16T16:19:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.248+0000 7fa92d400640 1 -- 172.21.15.45:0/1757085553 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa92802cc20 con 0x7fa914005210 2024-04-16T16:19:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.248+0000 7fa92d400640 1 -- 172.21.15.45:0/1757085553 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa928003990 con 0x7fa914005210 2024-04-16T16:19:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.248+0000 7fa934e00640 1 -- 172.21.15.45:0/1757085553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa914005210 msgr2=0x7fa91400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.248+0000 7fa934e00640 1 --2- 172.21.15.45:0/1757085553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa914005210 0x7fa91400b9c0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fa928009f90 tx=0x7fa914005830 comp rx=0 tx=0).stop 2024-04-16T16:19:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.248+0000 7fa934e00640 1 -- 172.21.15.45:0/1757085553 shutdown_connections 2024-04-16T16:19:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.248+0000 7fa934e00640 1 --2- 172.21.15.45:0/1757085553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa914005210 0x7fa91400b9c0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.248+0000 7fa934e00640 1 -- 172.21.15.45:0/1757085553 >> 172.21.15.45:0/1757085553 conn(0x7fa9300b2750 msgr2=0x7fa9300b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.248+0000 7fa934e00640 1 -- 172.21.15.45:0/1757085553 shutdown_connections 2024-04-16T16:19:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.249+0000 7fa934e00640 1 -- 172.21.15.45:0/1757085553 wait complete. 2024-04-16T16:19:30.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.250+0000 7fa934e00640 1 Processor -- start 2024-04-16T16:19:30.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.250+0000 7fa934e00640 1 -- start start 2024-04-16T16:19:30.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.251+0000 7fa934e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa914005210 0x7fa93013f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:30.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.251+0000 7fa934e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa93013ea70 0x7fa930141b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:30.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.251+0000 7fa934e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa930140c30 0x7fa930157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:30.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.251+0000 7fa934e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa930072380 con 0x7fa930140c30 2024-04-16T16:19:30.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.252+0000 7fa934e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa93015d540 con 0x7fa93013ea70 2024-04-16T16:19:30.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.252+0000 7fa934e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa930072500 con 0x7fa914005210 2024-04-16T16:19:30.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.252+0000 7fa92e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa930140c30 0x7fa930157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:30.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.252+0000 7fa92e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa930140c30 0x7fa930157210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34900/0 (socket says 172.21.15.45:34900) 2024-04-16T16:19:30.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.252+0000 7fa92e800640 1 -- 172.21.15.45:0/1699161817 learned_addr learned my addr 172.21.15.45:0/1699161817 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:30.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.252+0000 7fa92de00640 1 --2- 172.21.15.45:0/1699161817 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa914005210 0x7fa93013f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:30.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.252+0000 7fa927e00640 1 --2- 172.21.15.45:0/1699161817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa93013ea70 0x7fa930141b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:30.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.252+0000 7fa92e800640 1 -- 172.21.15.45:0/1699161817 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa914005210 msgr2=0x7fa93013f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:30.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.252+0000 7fa92e800640 1 --2- 172.21.15.45:0/1699161817 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa914005210 0x7fa93013f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.252+0000 7fa92e800640 1 -- 172.21.15.45:0/1699161817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa93013ea70 msgr2=0x7fa930141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:30.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.252+0000 7fa92e800640 1 --2- 172.21.15.45:0/1699161817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa93013ea70 0x7fa930141b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.252+0000 7fa92e800640 1 -- 172.21.15.45:0/1699161817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa93013b930 con 0x7fa930140c30 2024-04-16T16:19:30.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.253+0000 7fa92e800640 1 --2- 172.21.15.45:0/1699161817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa930140c30 0x7fa930157210 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fa92802c690 tx=0x7fa928009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:30.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.253+0000 7fa926000640 1 -- 172.21.15.45:0/1699161817 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa928035ed0 con 0x7fa930140c30 2024-04-16T16:19:30.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.253+0000 7fa926000640 1 -- 172.21.15.45:0/1699161817 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa928044070 con 0x7fa930140c30 2024-04-16T16:19:30.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.253+0000 7fa926000640 1 -- 172.21.15.45:0/1699161817 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa928003dc0 con 0x7fa930140c30 2024-04-16T16:19:30.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.253+0000 7fa934e00640 1 -- 172.21.15.45:0/1699161817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa93013fb90 con 0x7fa930140c30 2024-04-16T16:19:30.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.253+0000 7fa934e00640 1 -- 172.21.15.45:0/1699161817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa930140150 con 0x7fa930140c30 2024-04-16T16:19:30.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.255+0000 7fa934e00640 1 -- 172.21.15.45:0/1699161817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa93013ee50 con 0x7fa930140c30 2024-04-16T16:19:30.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.256+0000 7fa926000640 1 -- 172.21.15.45:0/1699161817 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 39) ==== 90878+0+0 (secure 0 0 0) 0x7fa928005d10 con 0x7fa930140c30 2024-04-16T16:19:30.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.256+0000 7fa926000640 1 --2- 172.21.15.45:0/1699161817 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa90c06c740 0x7fa90c06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:30.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.257+0000 7fa926000640 1 -- 172.21.15.45:0/1699161817 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6397+0+0 (secure 0 0 0) 0x7fa9280ae120 con 0x7fa930140c30 2024-04-16T16:19:30.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.257+0000 7fa92de00640 1 --2- 172.21.15.45:0/1699161817 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa90c06c740 0x7fa90c06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:30.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.258+0000 7fa92de00640 1 --2- 172.21.15.45:0/1699161817 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa90c06c740 0x7fa90c06ec00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa920005490 tx=0x7fa920009040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:30.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.261+0000 7fa926000640 1 -- 172.21.15.45:0/1699161817 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa93013ee50 con 0x7fa930140c30 2024-04-16T16:19:30.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.505+0000 7fa934e00640 1 -- 172.21.15.45:0/1699161817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa930075780 con 0x7fa930140c30 2024-04-16T16:19:30.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.506+0000 7fa926000640 1 -- 172.21.15.45:0/1699161817 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5186 (secure 0 0 0) 0x7fa93013b930 con 0x7fa930140c30 2024-04-16T16:19:30.505 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:30.505 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5804,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2862612740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2862612740},{"type":"v1","addr":"172.21.15.103:6836","nonce":2862612740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6304,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2934367463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2934367463},{"type":"v1","addr":"172.21.15.103:6837","nonce":2934367463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1425499906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1425499906},{"type":"v1","addr":"172.21.15.45:6835","nonce":1425499906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:19:05.457389+0000","modified":"2024-04-16T16:19:08.527316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7677},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7677":{"gid":7677,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/882883484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":882883484},{"type":"v1","addr":"172.21.15.45:6834","nonce":882883484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7677,"qdb_cluster":[7677]},"id":7}]} 2024-04-16T16:19:30.506 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 73 2024-04-16T16:19:30.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.509+0000 7fa934e00640 1 -- 172.21.15.45:0/1699161817 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa90c06c740 msgr2=0x7fa90c06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:30.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.509+0000 7fa934e00640 1 --2- 172.21.15.45:0/1699161817 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa90c06c740 0x7fa90c06ec00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa920005490 tx=0x7fa920009040 comp rx=0 tx=0).stop 2024-04-16T16:19:30.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.509+0000 7fa934e00640 1 -- 172.21.15.45:0/1699161817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa930140c30 msgr2=0x7fa930157210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:30.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.509+0000 7fa934e00640 1 --2- 172.21.15.45:0/1699161817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa930140c30 0x7fa930157210 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fa92802c690 tx=0x7fa928009750 comp rx=0 tx=0).stop 2024-04-16T16:19:30.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.509+0000 7fa934e00640 1 -- 172.21.15.45:0/1699161817 shutdown_connections 2024-04-16T16:19:30.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.509+0000 7fa934e00640 1 --2- 172.21.15.45:0/1699161817 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa90c06c740 0x7fa90c06ec00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.509+0000 7fa934e00640 1 --2- 172.21.15.45:0/1699161817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa930140c30 0x7fa930157210 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.509+0000 7fa934e00640 1 --2- 172.21.15.45:0/1699161817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa93013ea70 0x7fa930141b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.509+0000 7fa934e00640 1 --2- 172.21.15.45:0/1699161817 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa914005210 0x7fa93013f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.509+0000 7fa934e00640 1 -- 172.21.15.45:0/1699161817 >> 172.21.15.45:0/1699161817 conn(0x7fa9300b2750 msgr2=0x7fa9300b1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:30.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.509+0000 7fa934e00640 1 -- 172.21.15.45:0/1699161817 shutdown_connections 2024-04-16T16:19:30.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.509+0000 7fa934e00640 1 -- 172.21.15.45:0/1699161817 wait complete. 2024-04-16T16:19:30.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:19:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.755+0000 7fc3a9e00640 1 Processor -- start 2024-04-16T16:19:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.756+0000 7fc3a9e00640 1 -- start start 2024-04-16T16:19:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.757+0000 7fc3a9e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc3a415ccb0 0x7fc3a406df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.757+0000 7fc3a9e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3a406e480 0x7fc3a41608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.757+0000 7fc3a9e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a415d360 con 0x7fc3a4146220 2024-04-16T16:19:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.757+0000 7fc3a9e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc3a415d130 con 0x7fc3a415ccb0 2024-04-16T16:19:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.757+0000 7fc3a9e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc3a4160410 con 0x7fc3a406e480 2024-04-16T16:19:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.757+0000 7fc39be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc3a415ccb0 0x7fc3a406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.757+0000 7fc39be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc3a415ccb0 0x7fc3a406df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34908/0 (socket says 172.21.15.45:34908) 2024-04-16T16:19:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.757+0000 7fc39be00640 1 -- 172.21.15.45:0/3040941795 learned_addr learned my addr 172.21.15.45:0/3040941795 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.758+0000 7fc3a9400640 1 --2- 172.21.15.45:0/3040941795 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3a406e480 0x7fc3a41608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.758+0000 7fc39be00640 1 -- 172.21.15.45:0/3040941795 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3a406e480 msgr2=0x7fc3a41608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.758+0000 7fc39be00640 1 --2- 172.21.15.45:0/3040941795 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3a406e480 0x7fc3a41608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.758+0000 7fc39be00640 1 -- 172.21.15.45:0/3040941795 >> v1:172.21.15.103:6789/0 conn(0x7fc3a4146220 legacy=0x7fc3a406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.758+0000 7fc39be00640 1 -- 172.21.15.45:0/3040941795 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a415d640 con 0x7fc3a415ccb0 2024-04-16T16:19:30.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.758+0000 7fc3a9400640 1 --2- 172.21.15.45:0/3040941795 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3a406e480 0x7fc3a41608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:30.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.759+0000 7fc39be00640 1 --2- 172.21.15.45:0/3040941795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a415ccb0 0x7fc3a406df40 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fc394009870 tx=0x7fc39402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d9eaf3203a9608 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:30.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.759+0000 7fc3a3e00640 1 -- 172.21.15.45:0/3040941795 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc394035020 con 0x7fc3a415ccb0 2024-04-16T16:19:30.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.759+0000 7fc3a3e00640 1 -- 172.21.15.45:0/3040941795 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc394031640 con 0x7fc3a415ccb0 2024-04-16T16:19:30.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.759+0000 7fc3a3e00640 1 -- 172.21.15.45:0/3040941795 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39402d040 con 0x7fc3a415ccb0 2024-04-16T16:19:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.760+0000 7fc3a9e00640 1 -- 172.21.15.45:0/3040941795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a415ccb0 msgr2=0x7fc3a406df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.760+0000 7fc3a9e00640 1 --2- 172.21.15.45:0/3040941795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a415ccb0 0x7fc3a406df40 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fc394009870 tx=0x7fc39402b8f0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.760+0000 7fc3a9e00640 1 -- 172.21.15.45:0/3040941795 shutdown_connections 2024-04-16T16:19:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.760+0000 7fc3a9e00640 1 --2- 172.21.15.45:0/3040941795 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3a406e480 0x7fc3a41608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.760+0000 7fc3a9e00640 1 --2- 172.21.15.45:0/3040941795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a415ccb0 0x7fc3a406df40 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.760+0000 7fc3a9e00640 1 -- 172.21.15.45:0/3040941795 >> 172.21.15.45:0/3040941795 conn(0x7fc3a406b930 msgr2=0x7fc3a406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:30.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.760+0000 7fc3a9e00640 1 -- 172.21.15.45:0/3040941795 shutdown_connections 2024-04-16T16:19:30.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.760+0000 7fc3a9e00640 1 -- 172.21.15.45:0/3040941795 wait complete. 2024-04-16T16:19:30.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.762+0000 7fc3a9e00640 1 Processor -- start 2024-04-16T16:19:30.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.762+0000 7fc3a9e00640 1 -- start start 2024-04-16T16:19:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.763+0000 7fc3a9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3a406e480 0x7fc3a40c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.763+0000 7fc3a9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a4146220 0x7fc3a415c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.763+0000 7fc3a9e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3a415ccb0 0x7fc3a4138840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.763+0000 7fc3a9e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a4160de0 con 0x7fc3a4146220 2024-04-16T16:19:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.763+0000 7fc3a9e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a415dc00 con 0x7fc3a406e480 2024-04-16T16:19:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.763+0000 7fc3a9e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc3a4160f60 con 0x7fc3a415ccb0 2024-04-16T16:19:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.763+0000 7fc39be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a4146220 0x7fc3a415c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.764+0000 7fc39be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a4146220 0x7fc3a415c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34910/0 (socket says 172.21.15.45:34910) 2024-04-16T16:19:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.764+0000 7fc39be00640 1 -- 172.21.15.45:0/544786010 learned_addr learned my addr 172.21.15.45:0/544786010 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.764+0000 7fc3a9400640 1 --2- 172.21.15.45:0/544786010 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3a415ccb0 0x7fc3a4138840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.764+0000 7fc3a8a00640 1 --2- 172.21.15.45:0/544786010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3a406e480 0x7fc3a40c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.764+0000 7fc39be00640 1 -- 172.21.15.45:0/544786010 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3a415ccb0 msgr2=0x7fc3a4138840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.764+0000 7fc39be00640 1 --2- 172.21.15.45:0/544786010 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3a415ccb0 0x7fc3a4138840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.764+0000 7fc39be00640 1 -- 172.21.15.45:0/544786010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3a406e480 msgr2=0x7fc3a40c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.764+0000 7fc39be00640 1 --2- 172.21.15.45:0/544786010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3a406e480 0x7fc3a40c9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.764+0000 7fc39be00640 1 -- 172.21.15.45:0/544786010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a4139310 con 0x7fc3a4146220 2024-04-16T16:19:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.764+0000 7fc39be00640 1 --2- 172.21.15.45:0/544786010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a4146220 0x7fc3a415c160 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fc39402b930 tx=0x7fc3940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.765+0000 7fc3a1600640 1 -- 172.21.15.45:0/544786010 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc394040070 con 0x7fc3a4146220 2024-04-16T16:19:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.765+0000 7fc3a1600640 1 -- 172.21.15.45:0/544786010 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc394039bf0 con 0x7fc3a4146220 2024-04-16T16:19:30.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.765+0000 7fc3a1600640 1 -- 172.21.15.45:0/544786010 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc394035040 con 0x7fc3a4146220 2024-04-16T16:19:30.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.765+0000 7fc3a9e00640 1 -- 172.21.15.45:0/544786010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3a41395f0 con 0x7fc3a4146220 2024-04-16T16:19:30.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.765+0000 7fc3a9e00640 1 -- 172.21.15.45:0/544786010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3a40addd0 con 0x7fc3a4146220 2024-04-16T16:19:30.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.766+0000 7fc3a9e00640 1 -- 172.21.15.45:0/544786010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3a4075780 con 0x7fc3a4146220 2024-04-16T16:19:30.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.767+0000 7fc3a1600640 1 -- 172.21.15.45:0/544786010 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 39) ==== 90878+0+0 (secure 0 0 0) 0x7fc39404b060 con 0x7fc3a4146220 2024-04-16T16:19:30.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.768+0000 7fc3a1600640 1 --2- 172.21.15.45:0/544786010 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc38006c7c0 0x7fc38006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:30.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.768+0000 7fc3a1600640 1 -- 172.21.15.45:0/544786010 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6397+0+0 (secure 0 0 0) 0x7fc3940ad560 con 0x7fc3a4146220 2024-04-16T16:19:30.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.769+0000 7fc3a8a00640 1 --2- 172.21.15.45:0/544786010 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc38006c7c0 0x7fc38006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:30.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.770+0000 7fc3a8a00640 1 --2- 172.21.15.45:0/544786010 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc38006c7c0 0x7fc38006ec80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc38c004ea0 tx=0x7fc38c009040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:30.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.773+0000 7fc3a1600640 1 -- 172.21.15.45:0/544786010 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc39407b9d0 con 0x7fc3a4146220 2024-04-16T16:19:30.850 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:30.849+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:30.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.987+0000 7fc3a9e00640 1 -- 172.21.15.45:0/544786010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3a40cc930 con 0x7fc3a4146220 2024-04-16T16:19:30.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.990+0000 7fc3a1600640 1 -- 172.21.15.45:0/544786010 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+17624 (secure 0 0 0) 0x7fc39407b2d0 con 0x7fc3a4146220 2024-04-16T16:19:30.990 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:30.990 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":102,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:19:28.777496+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:03.429683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:04.434699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3000963222":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/963070403":"2024-04-17T16:19:28.777448+0000","172.21.15.103:6832/3948990093":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/2784388736":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/2130767640":"2024-04-17T16:19:28.777448+0000","172.21.15.103:6833/839990480":"2024-04-17T16:19:22.373133+0000","172.21.15.103:6832/839990480":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2646915013":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/1161670790":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/287602360":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2666528075":"2024-04-17T16:19:22.373133+0000","172.21.15.103:6833/3948990093":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/4022617615":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2967808514":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/1615159946":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/2879899995":"2024-04-16T17:19:09.753356+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:19:30.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.993+0000 7fc3a9e00640 1 -- 172.21.15.45:0/544786010 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc38006c7c0 msgr2=0x7fc38006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:30.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.993+0000 7fc3a9e00640 1 --2- 172.21.15.45:0/544786010 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc38006c7c0 0x7fc38006ec80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc38c004ea0 tx=0x7fc38c009040 comp rx=0 tx=0).stop 2024-04-16T16:19:30.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.993+0000 7fc3a9e00640 1 -- 172.21.15.45:0/544786010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a4146220 msgr2=0x7fc3a415c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:30.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.993+0000 7fc3a9e00640 1 --2- 172.21.15.45:0/544786010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a4146220 0x7fc3a415c160 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fc39402b930 tx=0x7fc3940046c0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.993+0000 7fc3a9e00640 1 -- 172.21.15.45:0/544786010 shutdown_connections 2024-04-16T16:19:30.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.993+0000 7fc3a9e00640 1 --2- 172.21.15.45:0/544786010 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3a415ccb0 0x7fc3a4138840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.993+0000 7fc3a9e00640 1 --2- 172.21.15.45:0/544786010 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc38006c7c0 0x7fc38006ec80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.993+0000 7fc3a9e00640 1 --2- 172.21.15.45:0/544786010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a4146220 0x7fc3a415c160 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.993+0000 7fc3a9e00640 1 --2- 172.21.15.45:0/544786010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3a406e480 0x7fc3a40c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:30.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.993+0000 7fc3a9e00640 1 -- 172.21.15.45:0/544786010 >> 172.21.15.45:0/544786010 conn(0x7fc3a406b930 msgr2=0x7fc3a40c5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:30.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.994+0000 7fc3a9e00640 1 -- 172.21.15.45:0/544786010 shutdown_connections 2024-04-16T16:19:30.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:30.994+0000 7fc3a9e00640 1 -- 172.21.15.45:0/544786010 wait complete. 2024-04-16T16:19:31.014 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:19:03.429683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:19:04.434699+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:19:31.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:19:31.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.247+0000 7f29dd200640 1 Processor -- start 2024-04-16T16:19:31.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.247+0000 7f29dd200640 1 -- start start 2024-04-16T16:19:31.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.249+0000 7f29dd200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f29d8145fe0 0x7f29d80c5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:31.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.249+0000 7f29dd200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29d8160e90 0x7f29d80cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:31.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.249+0000 7f29dd200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f29d80c5c90 0x7f29d80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:31.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.249+0000 7f29dd200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f29d80c6210 con 0x7f29d80c5c90 2024-04-16T16:19:31.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.249+0000 7f29dd200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f29d81603b0 con 0x7f29d8145fe0 2024-04-16T16:19:31.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.249+0000 7f29dd200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f29d8160690 con 0x7f29d8160e90 2024-04-16T16:19:31.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.249+0000 7f29d7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f29d80c5c90 0x7f29d80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:31.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.249+0000 7f29d7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f29d80c5c90 0x7f29d80cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34912/0 (socket says 172.21.15.45:34912) 2024-04-16T16:19:31.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.249+0000 7f29d7e00640 1 -- 172.21.15.45:0/2937528506 learned_addr learned my addr 172.21.15.45:0/2937528506 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.250+0000 7f29d6a00640 1 --2- 172.21.15.45:0/2937528506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29d8160e90 0x7f29d80cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.250+0000 7f29d7400640 1 --2- 172.21.15.45:0/2937528506 >> v2:172.21.15.103:3300/0 conn(0x7f29d8145fe0 0x7f29d80c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.250+0000 7f29d7e00640 1 -- 172.21.15.45:0/2937528506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29d8160e90 msgr2=0x7f29d80cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.250+0000 7f29d7e00640 1 --2- 172.21.15.45:0/2937528506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29d8160e90 0x7f29d80cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.250+0000 7f29d7e00640 1 -- 172.21.15.45:0/2937528506 >> v2:172.21.15.103:3300/0 conn(0x7f29d8145fe0 msgr2=0x7f29d80c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.250+0000 7f29d7e00640 1 --2- 172.21.15.45:0/2937528506 >> v2:172.21.15.103:3300/0 conn(0x7f29d8145fe0 0x7f29d80c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.250+0000 7f29d7e00640 1 -- 172.21.15.45:0/2937528506 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d8160970 con 0x7f29d80c5c90 2024-04-16T16:19:31.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.251+0000 7f29d7e00640 1 --2- 172.21.15.45:0/2937528506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d80c5c90 0x7f29d80cf980 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f29d80d0490 tx=0x7f29d002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d0fa68d48d28e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:31.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.251+0000 7f29d6000640 1 -- 172.21.15.45:0/2937528506 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d002c9e0 con 0x7f29d80c5c90 2024-04-16T16:19:31.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.251+0000 7f29d6000640 1 -- 172.21.15.45:0/2937528506 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29d002cb80 con 0x7f29d80c5c90 2024-04-16T16:19:31.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.251+0000 7f29d6000640 1 -- 172.21.15.45:0/2937528506 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d00343f0 con 0x7f29d80c5c90 2024-04-16T16:19:31.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.252+0000 7f29dd200640 1 -- 172.21.15.45:0/2937528506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d80c5c90 msgr2=0x7f29d80cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:31.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.252+0000 7f29dd200640 1 --2- 172.21.15.45:0/2937528506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d80c5c90 0x7f29d80cf980 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f29d80d0490 tx=0x7f29d002be40 comp rx=0 tx=0).stop 2024-04-16T16:19:31.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.252+0000 7f29dd200640 1 -- 172.21.15.45:0/2937528506 shutdown_connections 2024-04-16T16:19:31.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.252+0000 7f29dd200640 1 --2- 172.21.15.45:0/2937528506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d80c5c90 0x7f29d80cf980 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.252+0000 7f29dd200640 1 --2- 172.21.15.45:0/2937528506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29d8160e90 0x7f29d80cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.252+0000 7f29dd200640 1 --2- 172.21.15.45:0/2937528506 >> v2:172.21.15.103:3300/0 conn(0x7f29d8145fe0 0x7f29d80c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.252+0000 7f29dd200640 1 -- 172.21.15.45:0/2937528506 >> 172.21.15.45:0/2937528506 conn(0x7f29d8067e00 msgr2=0x7f29d806b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:31.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.252+0000 7f29dd200640 1 -- 172.21.15.45:0/2937528506 shutdown_connections 2024-04-16T16:19:31.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.252+0000 7f29dd200640 1 -- 172.21.15.45:0/2937528506 wait complete. 2024-04-16T16:19:31.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.255+0000 7f29dd200640 1 Processor -- start 2024-04-16T16:19:31.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.255+0000 7f29dd200640 1 -- start start 2024-04-16T16:19:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.256+0000 7f29dd200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d80c5c90 0x7f29d8156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.256+0000 7f29dd200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29d8145fe0 0x7f29d81e88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.256+0000 7f29dd200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29d8160e90 0x7f29d815de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.256+0000 7f29dd200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f29d8146540 con 0x7f29d80c5c90 2024-04-16T16:19:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.256+0000 7f29dd200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f29d81463c0 con 0x7f29d8160e90 2024-04-16T16:19:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.256+0000 7f29dd200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f29d806dfb0 con 0x7f29d8145fe0 2024-04-16T16:19:31.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.256+0000 7f29d7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d80c5c90 0x7f29d8156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:31.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.256+0000 7f29d7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d80c5c90 0x7f29d8156230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34918/0 (socket says 172.21.15.45:34918) 2024-04-16T16:19:31.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.256+0000 7f29d7400640 1 -- 172.21.15.45:0/404547312 learned_addr learned my addr 172.21.15.45:0/404547312 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:31.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.257+0000 7f29d6a00640 1 --2- 172.21.15.45:0/404547312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29d8145fe0 0x7f29d81e88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:31.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.257+0000 7f29d7400640 1 -- 172.21.15.45:0/404547312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29d8145fe0 msgr2=0x7f29d81e88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:31.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.257+0000 7f29d7400640 1 --2- 172.21.15.45:0/404547312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29d8145fe0 0x7f29d81e88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.257+0000 7f29d7400640 1 -- 172.21.15.45:0/404547312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29d8160e90 msgr2=0x7f29d815de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:31.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.257+0000 7f29d7e00640 1 --2- 172.21.15.45:0/404547312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29d8160e90 0x7f29d815de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:31.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.257+0000 7f29d7400640 1 --2- 172.21.15.45:0/404547312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29d8160e90 0x7f29d815de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.257+0000 7f29d7400640 1 -- 172.21.15.45:0/404547312 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d8156770 con 0x7f29d80c5c90 2024-04-16T16:19:31.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.257+0000 7f29d7400640 1 --2- 172.21.15.45:0/404547312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d80c5c90 0x7f29d8156230 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f29c800d800 tx=0x7f29c800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:31.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.257+0000 7f29cf400640 1 -- 172.21.15.45:0/404547312 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c800b6c0 con 0x7f29d80c5c90 2024-04-16T16:19:31.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.257+0000 7f29cf400640 1 -- 172.21.15.45:0/404547312 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29c8019070 con 0x7f29d80c5c90 2024-04-16T16:19:31.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.257+0000 7f29cf400640 1 -- 172.21.15.45:0/404547312 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c800ba80 con 0x7f29d80c5c90 2024-04-16T16:19:31.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.258+0000 7f29dd200640 1 -- 172.21.15.45:0/404547312 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29d81e9e10 con 0x7f29d80c5c90 2024-04-16T16:19:31.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.258+0000 7f29dd200640 1 -- 172.21.15.45:0/404547312 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29d81e9570 con 0x7f29d80c5c90 2024-04-16T16:19:31.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.259+0000 7f29dd200640 1 -- 172.21.15.45:0/404547312 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29d8075780 con 0x7f29d80c5c90 2024-04-16T16:19:31.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.260+0000 7f29cf400640 1 -- 172.21.15.45:0/404547312 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 39) ==== 90878+0+0 (secure 0 0 0) 0x7f29c800bc20 con 0x7f29d80c5c90 2024-04-16T16:19:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.261+0000 7f29cf400640 1 --2- 172.21.15.45:0/404547312 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f29b806c740 0x7f29b806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.261+0000 7f29cf400640 1 -- 172.21.15.45:0/404547312 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6397+0+0 (secure 0 0 0) 0x7f29c808bf00 con 0x7f29d80c5c90 2024-04-16T16:19:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.261+0000 7f29d6a00640 1 --2- 172.21.15.45:0/404547312 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f29b806c740 0x7f29b806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.263+0000 7f29d6a00640 1 --2- 172.21.15.45:0/404547312 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f29b806c740 0x7f29b806ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f29c4006fd0 tx=0x7f29c4008040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:31.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.266+0000 7f29cf400640 1 -- 172.21.15.45:0/404547312 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f29d8156770 con 0x7f29d80c5c90 2024-04-16T16:19:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.472+0000 7f29dd200640 1 -- 172.21.15.45:0/404547312 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f29d806c110 con 0x7f29d80c5c90 2024-04-16T16:19:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.474+0000 7f29cf400640 1 -- 172.21.15.45:0/404547312 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) ==== 93+0+10 (secure 0 0 0) 0x7f29c805a3c0 con 0x7f29d80c5c90 2024-04-16T16:19:31.473 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:19:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.476+0000 7f29dd200640 1 -- 172.21.15.45:0/404547312 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f29b806c740 msgr2=0x7f29b806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.476+0000 7f29dd200640 1 --2- 172.21.15.45:0/404547312 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f29b806c740 0x7f29b806ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f29c4006fd0 tx=0x7f29c4008040 comp rx=0 tx=0).stop 2024-04-16T16:19:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.476+0000 7f29dd200640 1 -- 172.21.15.45:0/404547312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d80c5c90 msgr2=0x7f29d8156230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.476+0000 7f29dd200640 1 --2- 172.21.15.45:0/404547312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d80c5c90 0x7f29d8156230 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f29c800d800 tx=0x7f29c800dcc0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.477+0000 7f29dd200640 1 -- 172.21.15.45:0/404547312 shutdown_connections 2024-04-16T16:19:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.477+0000 7f29dd200640 1 --2- 172.21.15.45:0/404547312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29d8160e90 0x7f29d815de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.477+0000 7f29dd200640 1 --2- 172.21.15.45:0/404547312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29d8145fe0 0x7f29d81e88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.477+0000 7f29dd200640 1 --2- 172.21.15.45:0/404547312 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f29b806c740 0x7f29b806ec00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.477+0000 7f29dd200640 1 --2- 172.21.15.45:0/404547312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d80c5c90 0x7f29d8156230 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.477+0000 7f29dd200640 1 -- 172.21.15.45:0/404547312 >> 172.21.15.45:0/404547312 conn(0x7f29d8067e00 msgr2=0x7f29d80cbd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:31.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.477+0000 7f29dd200640 1 -- 172.21.15.45:0/404547312 shutdown_connections 2024-04-16T16:19:31.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.477+0000 7f29dd200640 1 -- 172.21.15.45:0/404547312 wait complete. 2024-04-16T16:19:31.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:19:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.726+0000 7f3bb9e00640 1 Processor -- start 2024-04-16T16:19:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.726+0000 7f3bb9e00640 1 -- start start 2024-04-16T16:19:31.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.728+0000 7f3bb9e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bb415c7c0 0x7f3bb4072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.728+0000 7f3bb9e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3bb40b4930 0x7f3bb40d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.728+0000 7f3bb9e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3bb40d0f10 0x7f3bb406f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.728+0000 7f3bb9e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3bb40b52e0 con 0x7f3bb40b4930 2024-04-16T16:19:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.728+0000 7f3bb9e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3bb40d0260 con 0x7f3bb40d0f10 2024-04-16T16:19:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.728+0000 7f3bb9e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3bb40d1380 con 0x7f3bb415c7c0 2024-04-16T16:19:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.728+0000 7f3bb2000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3bb40b4930 0x7f3bb40d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.728+0000 7f3bb2000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3bb40b4930 0x7f3bb40d09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34926/0 (socket says 172.21.15.45:34926) 2024-04-16T16:19:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.728+0000 7f3bb2000640 1 -- 172.21.15.45:0/2614511995 learned_addr learned my addr 172.21.15.45:0/2614511995 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.729+0000 7f3bb3400640 1 --2- 172.21.15.45:0/2614511995 >> v2:172.21.15.103:3300/0 conn(0x7f3bb40d0f10 0x7f3bb406f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.729+0000 7f3bb2a00640 1 --2- 172.21.15.45:0/2614511995 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bb415c7c0 0x7f3bb4072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.729+0000 7f3bb2000640 1 -- 172.21.15.45:0/2614511995 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bb415c7c0 msgr2=0x7f3bb4072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.729+0000 7f3bb2000640 1 --2- 172.21.15.45:0/2614511995 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bb415c7c0 0x7f3bb4072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.729+0000 7f3bb2000640 1 -- 172.21.15.45:0/2614511995 >> v2:172.21.15.103:3300/0 conn(0x7f3bb40d0f10 msgr2=0x7f3bb406f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:31.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.729+0000 7f3bb2000640 1 --2- 172.21.15.45:0/2614511995 >> v2:172.21.15.103:3300/0 conn(0x7f3bb40d0f10 0x7f3bb406f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.729+0000 7f3bb2000640 1 -- 172.21.15.45:0/2614511995 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb40b5530 con 0x7f3bb40b4930 2024-04-16T16:19:31.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.731+0000 7f3bb2000640 1 --2- 172.21.15.45:0/2614511995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb40b4930 0x7f3bb40d09d0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40afd50 tx=0x7f3b9c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb10759ebf30460 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.731+0000 7f3bb1600640 1 -- 172.21.15.45:0/2614511995 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c038070 con 0x7f3bb40b4930 2024-04-16T16:19:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.731+0000 7f3bb1600640 1 -- 172.21.15.45:0/2614511995 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b9c02f050 con 0x7f3bb40b4930 2024-04-16T16:19:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.731+0000 7f3bb1600640 1 -- 172.21.15.45:0/2614511995 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c031970 con 0x7f3bb40b4930 2024-04-16T16:19:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.732+0000 7f3bb9e00640 1 -- 172.21.15.45:0/2614511995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb40b4930 msgr2=0x7f3bb40d09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.732+0000 7f3bb9e00640 1 --2- 172.21.15.45:0/2614511995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb40b4930 0x7f3bb40d09d0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40afd50 tx=0x7f3b9c02b7f0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.732+0000 7f3bb9e00640 1 -- 172.21.15.45:0/2614511995 shutdown_connections 2024-04-16T16:19:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.732+0000 7f3bb9e00640 1 --2- 172.21.15.45:0/2614511995 >> v2:172.21.15.103:3300/0 conn(0x7f3bb40d0f10 0x7f3bb406f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.732+0000 7f3bb9e00640 1 --2- 172.21.15.45:0/2614511995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb40b4930 0x7f3bb40d09d0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.732+0000 7f3bb9e00640 1 --2- 172.21.15.45:0/2614511995 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bb415c7c0 0x7f3bb4072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.732+0000 7f3bb9e00640 1 -- 172.21.15.45:0/2614511995 >> 172.21.15.45:0/2614511995 conn(0x7f3bb40b3970 msgr2=0x7f3bb40b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.732+0000 7f3bb9e00640 1 -- 172.21.15.45:0/2614511995 shutdown_connections 2024-04-16T16:19:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.732+0000 7f3bb9e00640 1 -- 172.21.15.45:0/2614511995 wait complete. 2024-04-16T16:19:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.734+0000 7f3bb9e00640 1 Processor -- start 2024-04-16T16:19:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.734+0000 7f3bb9e00640 1 -- start start 2024-04-16T16:19:31.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.735+0000 7f3bb9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb40b4930 0x7f3bb41495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:31.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.735+0000 7f3bb9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bb40d0f10 0x7f3bb4154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:31.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.735+0000 7f3bb9e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bb415c7c0 0x7f3bb40b2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:31.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.735+0000 7f3bb9e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb40d04b0 con 0x7f3bb40b4930 2024-04-16T16:19:31.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.735+0000 7f3bb9e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb4073350 con 0x7f3bb40d0f10 2024-04-16T16:19:31.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.735+0000 7f3bb9e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3bb406f6a0 con 0x7f3bb415c7c0 2024-04-16T16:19:31.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.735+0000 7f3bb2a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb40b4930 0x7f3bb41495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:31.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.735+0000 7f3bb2a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb40b4930 0x7f3bb41495b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34938/0 (socket says 172.21.15.45:34938) 2024-04-16T16:19:31.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.735+0000 7f3bb2a00640 1 -- 172.21.15.45:0/1784821280 learned_addr learned my addr 172.21.15.45:0/1784821280 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:31.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.736+0000 7f3bb2000640 1 --2- 172.21.15.45:0/1784821280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bb40d0f10 0x7f3bb4154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:31.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.736+0000 7f3bb3400640 1 --2- 172.21.15.45:0/1784821280 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bb415c7c0 0x7f3bb40b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:31.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.736+0000 7f3bb2a00640 1 -- 172.21.15.45:0/1784821280 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bb415c7c0 msgr2=0x7f3bb40b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:31.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.736+0000 7f3bb2a00640 1 --2- 172.21.15.45:0/1784821280 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bb415c7c0 0x7f3bb40b2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.736+0000 7f3bb2a00640 1 -- 172.21.15.45:0/1784821280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bb40d0f10 msgr2=0x7f3bb4154010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:31.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.736+0000 7f3bb2a00640 1 --2- 172.21.15.45:0/1784821280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bb40d0f10 0x7f3bb4154010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.736+0000 7f3bb2a00640 1 -- 172.21.15.45:0/1784821280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb40cdf90 con 0x7f3bb40b4930 2024-04-16T16:19:31.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.736+0000 7f3bb2a00640 1 --2- 172.21.15.45:0/1784821280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb40b4930 0x7f3bb41495b0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400d620 tx=0x7f3ba400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:31.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.736+0000 7f3bab400640 1 -- 172.21.15.45:0/1784821280 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba400b620 con 0x7f3bb40b4930 2024-04-16T16:19:31.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.736+0000 7f3bab400640 1 -- 172.21.15.45:0/1784821280 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ba4019070 con 0x7f3bb40b4930 2024-04-16T16:19:31.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.736+0000 7f3bab400640 1 -- 172.21.15.45:0/1784821280 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba400ba00 con 0x7f3bb40b4930 2024-04-16T16:19:31.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.736+0000 7f3bb9e00640 1 -- 172.21.15.45:0/1784821280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bb40ce270 con 0x7f3bb40b4930 2024-04-16T16:19:31.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.737+0000 7f3bb9e00640 1 -- 172.21.15.45:0/1784821280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bb40ce830 con 0x7f3bb40b4930 2024-04-16T16:19:31.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.738+0000 7f3bb9e00640 1 -- 172.21.15.45:0/1784821280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bb4075780 con 0x7f3bb40b4930 2024-04-16T16:19:31.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.739+0000 7f3bab400640 1 -- 172.21.15.45:0/1784821280 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f3ba400bba0 con 0x7f3bb40b4930 2024-04-16T16:19:31.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.740+0000 7f3bab400640 1 --2- 172.21.15.45:0/1784821280 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3b9006c980 0x7f3b9006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:31.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.740+0000 7f3bab400640 1 -- 172.21.15.45:0/1784821280 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6397+0+0 (secure 0 0 0) 0x7f3ba408c2e0 con 0x7f3bb40b4930 2024-04-16T16:19:31.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.741+0000 7f3bb2000640 1 --2- 172.21.15.45:0/1784821280 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3b9006c980 0x7f3b9006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:31.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.742+0000 7f3bb2000640 1 --2- 172.21.15.45:0/1784821280 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3b9006c980 0x7f3b9006ee40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c035000 tx=0x7f3b9c008030 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:31.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.745+0000 7f3bab400640 1 -- 172.21.15.45:0/1784821280 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3ba405a370 con 0x7f3bb40b4930 2024-04-16T16:19:31.921 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:31.921+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:31.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.954+0000 7f3bb9e00640 1 -- 172.21.15.45:0/1784821280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3bb40d0820 con 0x7f3bb40b4930 2024-04-16T16:19:31.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.955+0000 7f3bab400640 1 -- 172.21.15.45:0/1784821280 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) ==== 104+0+11 (secure 0 0 0) 0x7f3ba4060070 con 0x7f3bb40b4930 2024-04-16T16:19:31.954 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:19:31.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.958+0000 7f3bb9e00640 1 -- 172.21.15.45:0/1784821280 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3b9006c980 msgr2=0x7f3b9006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:31.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.958+0000 7f3bb9e00640 1 --2- 172.21.15.45:0/1784821280 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3b9006c980 0x7f3b9006ee40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c035000 tx=0x7f3b9c008030 comp rx=0 tx=0).stop 2024-04-16T16:19:31.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.958+0000 7f3bb9e00640 1 -- 172.21.15.45:0/1784821280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb40b4930 msgr2=0x7f3bb41495b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:31.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.958+0000 7f3bb9e00640 1 --2- 172.21.15.45:0/1784821280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb40b4930 0x7f3bb41495b0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400d620 tx=0x7f3ba400dae0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.958+0000 7f3bb9e00640 1 -- 172.21.15.45:0/1784821280 shutdown_connections 2024-04-16T16:19:31.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.958+0000 7f3bb9e00640 1 --2- 172.21.15.45:0/1784821280 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bb415c7c0 0x7f3bb40b2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.958+0000 7f3bb9e00640 1 --2- 172.21.15.45:0/1784821280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bb40d0f10 0x7f3bb4154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.958+0000 7f3bb9e00640 1 --2- 172.21.15.45:0/1784821280 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3b9006c980 0x7f3b9006ee40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.958+0000 7f3bb9e00640 1 --2- 172.21.15.45:0/1784821280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb40b4930 0x7f3bb41495b0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:31.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.958+0000 7f3bb9e00640 1 -- 172.21.15.45:0/1784821280 >> 172.21.15.45:0/1784821280 conn(0x7f3bb40b3970 msgr2=0x7f3bb4157590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:31.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.958+0000 7f3bb9e00640 1 -- 172.21.15.45:0/1784821280 shutdown_connections 2024-04-16T16:19:31.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:31.958+0000 7f3bb9e00640 1 -- 172.21.15.45:0/1784821280 wait complete. 2024-04-16T16:19:31.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:19:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.208+0000 7fd33b200640 1 Processor -- start 2024-04-16T16:19:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.208+0000 7fd33b200640 1 -- start start 2024-04-16T16:19:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.210+0000 7fd33b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3341461b0 0x7fd334160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.210+0000 7fd33b200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd3340b58f0 0x7fd3340b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.210+0000 7fd33b200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd334146590 con 0x7fd3340cf440 2024-04-16T16:19:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.210+0000 7fd33b200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd33406dfd0 con 0x7fd3340b58f0 2024-04-16T16:19:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.210+0000 7fd33b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd3340b6290 con 0x7fd3341461b0 2024-04-16T16:19:32.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.210+0000 7fd339400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd3340b58f0 0x7fd3340b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:32.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.210+0000 7fd339400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd3340b58f0 0x7fd3340b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34944/0 (socket says 172.21.15.45:34944) 2024-04-16T16:19:32.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.210+0000 7fd33a800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd3340cf440 0x7fd3340cf8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43480/0 (socket says 172.21.15.45:43480) 2024-04-16T16:19:32.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.210+0000 7fd339400640 1 -- 172.21.15.45:0/3302694068 learned_addr learned my addr 172.21.15.45:0/3302694068 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:32.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.211+0000 7fd339e00640 1 --2- 172.21.15.45:0/3302694068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3341461b0 0x7fd334160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:32.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.211+0000 7fd339400640 1 -- 172.21.15.45:0/3302694068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3341461b0 msgr2=0x7fd334160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:32.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.211+0000 7fd339400640 1 --2- 172.21.15.45:0/3302694068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3341461b0 0x7fd334160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.211+0000 7fd339400640 1 -- 172.21.15.45:0/3302694068 >> v1:172.21.15.45:6789/0 conn(0x7fd3340cf440 legacy=0x7fd3340cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:32.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.211+0000 7fd339400640 1 -- 172.21.15.45:0/3302694068 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3340c5750 con 0x7fd3340b58f0 2024-04-16T16:19:32.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.211+0000 7fd338a00640 1 -- 172.21.15.45:0/3302694068 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4160398240 0 0) 0x7fd334146590 con 0x7fd3340cf440 2024-04-16T16:19:32.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.212+0000 7fd339400640 1 --2- 172.21.15.45:0/3302694068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3340b58f0 0x7fd3340b5d50 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fd324009870 tx=0x7fd32402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a373b3bd77f58f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:32.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.213+0000 7fd338a00640 1 -- 172.21.15.45:0/3302694068 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd324034070 con 0x7fd3340b58f0 2024-04-16T16:19:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.213+0000 7fd338a00640 1 -- 172.21.15.45:0/3302694068 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd324030840 con 0x7fd3340b58f0 2024-04-16T16:19:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.213+0000 7fd338a00640 1 -- 172.21.15.45:0/3302694068 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd324030c10 con 0x7fd3340b58f0 2024-04-16T16:19:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.213+0000 7fd33b200640 1 -- 172.21.15.45:0/3302694068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3340b58f0 msgr2=0x7fd3340b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.213+0000 7fd33b200640 1 --2- 172.21.15.45:0/3302694068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3340b58f0 0x7fd3340b5d50 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fd324009870 tx=0x7fd32402b980 comp rx=0 tx=0).stop 2024-04-16T16:19:32.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.213+0000 7fd33b200640 1 -- 172.21.15.45:0/3302694068 shutdown_connections 2024-04-16T16:19:32.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.213+0000 7fd33b200640 1 --2- 172.21.15.45:0/3302694068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3340b58f0 0x7fd3340b5d50 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.213+0000 7fd33b200640 1 --2- 172.21.15.45:0/3302694068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3341461b0 0x7fd334160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.213+0000 7fd33b200640 1 -- 172.21.15.45:0/3302694068 >> 172.21.15.45:0/3302694068 conn(0x7fd33406b930 msgr2=0x7fd33406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:32.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.214+0000 7fd33b200640 1 -- 172.21.15.45:0/3302694068 shutdown_connections 2024-04-16T16:19:32.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.214+0000 7fd33b200640 1 -- 172.21.15.45:0/3302694068 wait complete. 2024-04-16T16:19:32.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.215+0000 7fd33b200640 1 Processor -- start 2024-04-16T16:19:32.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.216+0000 7fd33b200640 1 -- start start 2024-04-16T16:19:32.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.216+0000 7fd33b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3340b58f0 0x7fd33415a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:32.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.217+0000 7fd33b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3340cf440 0x7fd33413bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:32.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.217+0000 7fd33b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3341461b0 0x7fd33413c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:32.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.217+0000 7fd33b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd3340c59f0 con 0x7fd3340cf440 2024-04-16T16:19:32.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.217+0000 7fd33b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd334161230 con 0x7fd3340b58f0 2024-04-16T16:19:32.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.217+0000 7fd33b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd3340c5cd0 con 0x7fd3341461b0 2024-04-16T16:19:32.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.217+0000 7fd339400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3340cf440 0x7fd33413bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:32.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.217+0000 7fd339400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3340cf440 0x7fd33413bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34960/0 (socket says 172.21.15.45:34960) 2024-04-16T16:19:32.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.217+0000 7fd339400640 1 -- 172.21.15.45:0/682733160 learned_addr learned my addr 172.21.15.45:0/682733160 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:32.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.217+0000 7fd339e00640 1 --2- 172.21.15.45:0/682733160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3340b58f0 0x7fd33415a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:32.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.217+0000 7fd339400640 1 -- 172.21.15.45:0/682733160 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3341461b0 msgr2=0x7fd33413c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:32.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.217+0000 7fd33a800640 1 --2- 172.21.15.45:0/682733160 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3341461b0 0x7fd33413c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.217+0000 7fd339400640 1 --2- 172.21.15.45:0/682733160 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3341461b0 0x7fd33413c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.217+0000 7fd339400640 1 -- 172.21.15.45:0/682733160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3340b58f0 msgr2=0x7fd33415a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.217+0000 7fd339400640 1 --2- 172.21.15.45:0/682733160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3340b58f0 0x7fd33415a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.217+0000 7fd339400640 1 -- 172.21.15.45:0/682733160 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd33415af00 con 0x7fd3340cf440 2024-04-16T16:19:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.218+0000 7fd339e00640 1 --2- 172.21.15.45:0/682733160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3340b58f0 0x7fd33415a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.218+0000 7fd339400640 1 --2- 172.21.15.45:0/682733160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3340cf440 0x7fd33413bbe0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fd324009870 tx=0x7fd324004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.218+0000 7fd32e000640 1 -- 172.21.15.45:0/682733160 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd324048070 con 0x7fd3340cf440 2024-04-16T16:19:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.218+0000 7fd32e000640 1 -- 172.21.15.45:0/682733160 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd324030640 con 0x7fd3340cf440 2024-04-16T16:19:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.218+0000 7fd32e000640 1 -- 172.21.15.45:0/682733160 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd324034040 con 0x7fd3340cf440 2024-04-16T16:19:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.218+0000 7fd33b200640 1 -- 172.21.15.45:0/682733160 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3341f45d0 con 0x7fd3340cf440 2024-04-16T16:19:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.218+0000 7fd33b200640 1 -- 172.21.15.45:0/682733160 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3341f4b90 con 0x7fd3340cf440 2024-04-16T16:19:32.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.220+0000 7fd33b200640 1 -- 172.21.15.45:0/682733160 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd334075780 con 0x7fd3340cf440 2024-04-16T16:19:32.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.221+0000 7fd32e000640 1 -- 172.21.15.45:0/682733160 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7fd324037070 con 0x7fd3340cf440 2024-04-16T16:19:32.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.222+0000 7fd32e000640 1 --2- 172.21.15.45:0/682733160 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fd31806c980 0x7fd31806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:32.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.222+0000 7fd32e000640 1 -- 172.21.15.45:0/682733160 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6397+0+0 (secure 0 0 0) 0x7fd3240add10 con 0x7fd3340cf440 2024-04-16T16:19:32.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.222+0000 7fd339e00640 1 --2- 172.21.15.45:0/682733160 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fd31806c980 0x7fd31806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:32.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.224+0000 7fd339e00640 1 --2- 172.21.15.45:0/682733160 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fd31806c980 0x7fd31806ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd328006fd0 tx=0x7fd328008040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:32.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.226+0000 7fd32e000640 1 -- 172.21.15.45:0/682733160 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd32407c000 con 0x7fd3340cf440 2024-04-16T16:19:32.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.436+0000 7fd33b200640 1 -- 172.21.15.45:0/682733160 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd334072500 con 0x7fd3340cf440 2024-04-16T16:19:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.437+0000 7fd32e000640 1 -- 172.21.15.45:0/682733160 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) ==== 100+0+11 (secure 0 0 0) 0x7fd32407b900 con 0x7fd3340cf440 2024-04-16T16:19:32.436 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:19:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.440+0000 7fd33b200640 1 -- 172.21.15.45:0/682733160 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fd31806c980 msgr2=0x7fd31806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.440+0000 7fd33b200640 1 --2- 172.21.15.45:0/682733160 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fd31806c980 0x7fd31806ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd328006fd0 tx=0x7fd328008040 comp rx=0 tx=0).stop 2024-04-16T16:19:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.440+0000 7fd33b200640 1 -- 172.21.15.45:0/682733160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3340cf440 msgr2=0x7fd33413bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.440+0000 7fd33b200640 1 --2- 172.21.15.45:0/682733160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3340cf440 0x7fd33413bbe0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fd324009870 tx=0x7fd324004730 comp rx=0 tx=0).stop 2024-04-16T16:19:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.440+0000 7fd33b200640 1 -- 172.21.15.45:0/682733160 shutdown_connections 2024-04-16T16:19:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.440+0000 7fd33b200640 1 --2- 172.21.15.45:0/682733160 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3341461b0 0x7fd33413c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.440+0000 7fd33b200640 1 --2- 172.21.15.45:0/682733160 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fd31806c980 0x7fd31806ee40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.440+0000 7fd33b200640 1 --2- 172.21.15.45:0/682733160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3340cf440 0x7fd33413bbe0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.440+0000 7fd33b200640 1 --2- 172.21.15.45:0/682733160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3340b58f0 0x7fd33415a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.440+0000 7fd33b200640 1 -- 172.21.15.45:0/682733160 >> 172.21.15.45:0/682733160 conn(0x7fd33406b930 msgr2=0x7fd3340cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.441+0000 7fd33b200640 1 -- 172.21.15.45:0/682733160 shutdown_connections 2024-04-16T16:19:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.441+0000 7fd33b200640 1 -- 172.21.15.45:0/682733160 wait complete. 2024-04-16T16:19:32.460 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:19:32.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.672+0000 7f4aea400640 1 Processor -- start 2024-04-16T16:19:32.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.673+0000 7f4aea400640 1 -- start start 2024-04-16T16:19:32.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.674+0000 7f4aea400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ae409ab50 0x7f4ae409af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:32.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.674+0000 7f4aea400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4ae40d1cc0 0x7f4ae415f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:32.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.674+0000 7f4aea400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ae409b470 con 0x7f4ae4162720 2024-04-16T16:19:32.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.674+0000 7f4aea400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f4ae406dbf0 con 0x7f4ae40d1cc0 2024-04-16T16:19:32.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.674+0000 7f4aea400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ae406cfe0 con 0x7f4ae409ab50 2024-04-16T16:19:32.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.675+0000 7f4ae2a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4ae4162720 0x7f4ae415eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43482/0 (socket says 172.21.15.45:43482) 2024-04-16T16:19:32.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.675+0000 7f4ae2a00640 1 -- 172.21.15.45:0/3798478363 learned_addr learned my addr 172.21.15.45:0/3798478363 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:32.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.675+0000 7f4ae3e00640 1 --2- 172.21.15.45:0/3798478363 >> v2:172.21.15.45:3300/0 conn(0x7f4ae40d1cc0 0x7f4ae415f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:32.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.675+0000 7f4ae3400640 1 --2- 172.21.15.45:0/3798478363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ae409ab50 0x7f4ae409af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:32.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.675+0000 7f4ae2000640 1 -- 172.21.15.45:0/3798478363 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3832188540 0 0) 0x7f4ae409b470 con 0x7f4ae4162720 2024-04-16T16:19:32.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.675+0000 7f4ae2000640 1 -- 172.21.15.45:0/3798478363 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ac8003520 con 0x7f4ae4162720 2024-04-16T16:19:32.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.675+0000 7f4ae3e00640 1 -- 172.21.15.45:0/3798478363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ae409ab50 msgr2=0x7f4ae409af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:32.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.675+0000 7f4ae3e00640 1 --2- 172.21.15.45:0/3798478363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ae409ab50 0x7f4ae409af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.675+0000 7f4ae3e00640 1 -- 172.21.15.45:0/3798478363 >> v1:172.21.15.45:6789/0 conn(0x7f4ae4162720 legacy=0x7f4ae415eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:32.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.675+0000 7f4ae3e00640 1 -- 172.21.15.45:0/3798478363 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae406d2c0 con 0x7f4ae40d1cc0 2024-04-16T16:19:32.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.677+0000 7f4ae3e00640 1 --2- 172.21.15.45:0/3798478363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae40d1cc0 0x7f4ae415f600 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f4adc009d90 tx=0x7f4adc02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92119a1f3b32082 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:32.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.677+0000 7f4ae2000640 1 -- 172.21.15.45:0/3798478363 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4adc039070 con 0x7f4ae40d1cc0 2024-04-16T16:19:32.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.677+0000 7f4ae2000640 1 -- 172.21.15.45:0/3798478363 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4adc02cbc0 con 0x7f4ae40d1cc0 2024-04-16T16:19:32.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.677+0000 7f4ae2000640 1 -- 172.21.15.45:0/3798478363 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4adc034440 con 0x7f4ae40d1cc0 2024-04-16T16:19:32.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.677+0000 7f4aea400640 1 -- 172.21.15.45:0/3798478363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae40d1cc0 msgr2=0x7f4ae415f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:32.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.677+0000 7f4aea400640 1 --2- 172.21.15.45:0/3798478363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae40d1cc0 0x7f4ae415f600 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f4adc009d90 tx=0x7f4adc02be10 comp rx=0 tx=0).stop 2024-04-16T16:19:32.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.678+0000 7f4aea400640 1 -- 172.21.15.45:0/3798478363 shutdown_connections 2024-04-16T16:19:32.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.678+0000 7f4aea400640 1 --2- 172.21.15.45:0/3798478363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae40d1cc0 0x7f4ae415f600 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.678+0000 7f4aea400640 1 --2- 172.21.15.45:0/3798478363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ae409ab50 0x7f4ae409af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.678+0000 7f4aea400640 1 -- 172.21.15.45:0/3798478363 >> 172.21.15.45:0/3798478363 conn(0x7f4ae40d2890 msgr2=0x7f4ae40d2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:32.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.678+0000 7f4aea400640 1 -- 172.21.15.45:0/3798478363 shutdown_connections 2024-04-16T16:19:32.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.678+0000 7f4aea400640 1 -- 172.21.15.45:0/3798478363 wait complete. 2024-04-16T16:19:32.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.680+0000 7f4aea400640 1 Processor -- start 2024-04-16T16:19:32.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.680+0000 7f4aea400640 1 -- start start 2024-04-16T16:19:32.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.681+0000 7f4aea400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae409ab50 0x7f4ae40b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:32.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.681+0000 7f4aea400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ae40d1cc0 0x7f4ae40c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:32.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.681+0000 7f4aea400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ae4162720 0x7f4ae40c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:32.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.681+0000 7f4aea400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae409b8b0 con 0x7f4ae409ab50 2024-04-16T16:19:32.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.681+0000 7f4aea400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae406d880 con 0x7f4ae4162720 2024-04-16T16:19:32.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.681+0000 7f4aea400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ae406ca40 con 0x7f4ae40d1cc0 2024-04-16T16:19:32.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.681+0000 7f4ae3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae409ab50 0x7f4ae40b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:32.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.681+0000 7f4ae3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae409ab50 0x7f4ae40b8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34978/0 (socket says 172.21.15.45:34978) 2024-04-16T16:19:32.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.681+0000 7f4ae3400640 1 -- 172.21.15.45:0/3751660095 learned_addr learned my addr 172.21.15.45:0/3751660095 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:32.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.681+0000 7f4ae2a00640 1 --2- 172.21.15.45:0/3751660095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ae40d1cc0 0x7f4ae40c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:32.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.682+0000 7f4ae3e00640 1 --2- 172.21.15.45:0/3751660095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ae4162720 0x7f4ae40c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:32.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.682+0000 7f4ae3400640 1 -- 172.21.15.45:0/3751660095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ae40d1cc0 msgr2=0x7f4ae40c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:32.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.682+0000 7f4ae3400640 1 --2- 172.21.15.45:0/3751660095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ae40d1cc0 0x7f4ae40c7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.682+0000 7f4ae3400640 1 -- 172.21.15.45:0/3751660095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ae4162720 msgr2=0x7f4ae40c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:32.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.682+0000 7f4ae3400640 1 --2- 172.21.15.45:0/3751660095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ae4162720 0x7f4ae40c8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.682+0000 7f4ae3400640 1 -- 172.21.15.45:0/3751660095 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae40c89c0 con 0x7f4ae409ab50 2024-04-16T16:19:32.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.682+0000 7f4ae2a00640 1 --2- 172.21.15.45:0/3751660095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ae40d1cc0 0x7f4ae40c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:32.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.682+0000 7f4ae3400640 1 --2- 172.21.15.45:0/3751660095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae409ab50 0x7f4ae40b8220 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400d830 tx=0x7f4ad400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:32.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.682+0000 7f4adb400640 1 -- 172.21.15.45:0/3751660095 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad400b6c0 con 0x7f4ae409ab50 2024-04-16T16:19:32.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.682+0000 7f4adb400640 1 -- 172.21.15.45:0/3751660095 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ad4019070 con 0x7f4ae409ab50 2024-04-16T16:19:32.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.682+0000 7f4adb400640 1 -- 172.21.15.45:0/3751660095 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad400ba80 con 0x7f4ae409ab50 2024-04-16T16:19:32.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.682+0000 7f4aea400640 1 -- 172.21.15.45:0/3751660095 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae4144fc0 con 0x7f4ae409ab50 2024-04-16T16:19:32.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.682+0000 7f4aea400640 1 -- 172.21.15.45:0/3751660095 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ae4145580 con 0x7f4ae409ab50 2024-04-16T16:19:32.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.684+0000 7f4aea400640 1 -- 172.21.15.45:0/3751660095 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ae4075780 con 0x7f4ae409ab50 2024-04-16T16:19:32.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.685+0000 7f4adb400640 1 -- 172.21.15.45:0/3751660095 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f4ad401e460 con 0x7f4ae409ab50 2024-04-16T16:19:32.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.686+0000 7f4adb400640 1 --2- 172.21.15.45:0/3751660095 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f4ac406c980 0x7f4ac406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:32.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.686+0000 7f4adb400640 1 -- 172.21.15.45:0/3751660095 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6397+0+0 (secure 0 0 0) 0x7f4ad408c570 con 0x7f4ae409ab50 2024-04-16T16:19:32.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.686+0000 7f4ae2a00640 1 --2- 172.21.15.45:0/3751660095 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f4ac406c980 0x7f4ac406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:32.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.688+0000 7f4ae2a00640 1 --2- 172.21.15.45:0/3751660095 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f4ac406c980 0x7f4ac406ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4ae409b1c0 tx=0x7f4ad0009040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:32.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.691+0000 7f4adb400640 1 -- 172.21.15.45:0/3751660095 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ad405a690 con 0x7f4ae409ab50 2024-04-16T16:19:32.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.942+0000 7f4aea400640 1 -- 172.21.15.45:0/3751660095 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ae415e350 con 0x7f4ae409ab50 2024-04-16T16:19:32.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.943+0000 7f4adb400640 1 -- 172.21.15.45:0/3751660095 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5186 (secure 0 0 0) 0x7f4ad4059f90 con 0x7f4ae409ab50 2024-04-16T16:19:32.943 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:32.943 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5804,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2862612740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2862612740},{"type":"v1","addr":"172.21.15.103:6836","nonce":2862612740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6304,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2934367463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2934367463},{"type":"v1","addr":"172.21.15.103:6837","nonce":2934367463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1425499906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1425499906},{"type":"v1","addr":"172.21.15.45:6835","nonce":1425499906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:19:05.457389+0000","modified":"2024-04-16T16:19:08.527316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7677},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7677":{"gid":7677,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/882883484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":882883484},{"type":"v1","addr":"172.21.15.45:6834","nonce":882883484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7677,"qdb_cluster":[7677]},"id":7}]} 2024-04-16T16:19:32.943 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 73 2024-04-16T16:19:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.946+0000 7f4aea400640 1 -- 172.21.15.45:0/3751660095 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f4ac406c980 msgr2=0x7f4ac406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.946+0000 7f4aea400640 1 --2- 172.21.15.45:0/3751660095 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f4ac406c980 0x7f4ac406ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4ae409b1c0 tx=0x7f4ad0009040 comp rx=0 tx=0).stop 2024-04-16T16:19:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.946+0000 7f4aea400640 1 -- 172.21.15.45:0/3751660095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae409ab50 msgr2=0x7f4ae40b8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.946+0000 7f4aea400640 1 --2- 172.21.15.45:0/3751660095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae409ab50 0x7f4ae40b8220 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400d830 tx=0x7f4ad400dcf0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.946+0000 7f4aea400640 1 -- 172.21.15.45:0/3751660095 shutdown_connections 2024-04-16T16:19:32.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.946+0000 7f4aea400640 1 --2- 172.21.15.45:0/3751660095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ae4162720 0x7f4ae40c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.946+0000 7f4aea400640 1 --2- 172.21.15.45:0/3751660095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ae40d1cc0 0x7f4ae40c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.946+0000 7f4aea400640 1 --2- 172.21.15.45:0/3751660095 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f4ac406c980 0x7f4ac406ee40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.946+0000 7f4aea400640 1 --2- 172.21.15.45:0/3751660095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae409ab50 0x7f4ae40b8220 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:32.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.946+0000 7f4aea400640 1 -- 172.21.15.45:0/3751660095 >> 172.21.15.45:0/3751660095 conn(0x7f4ae40d2890 msgr2=0x7f4ae4162fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:32.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.946+0000 7f4aea400640 1 -- 172.21.15.45:0/3751660095 shutdown_connections 2024-04-16T16:19:32.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:32.947+0000 7f4aea400640 1 -- 172.21.15.45:0/3751660095 wait complete. 2024-04-16T16:19:32.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:19:32.988 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:32.987+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:33.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.174+0000 7f0be2e00640 1 Processor -- start 2024-04-16T16:19:33.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.174+0000 7f0be2e00640 1 -- start start 2024-04-16T16:19:33.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.176+0000 7f0be2e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0bdc06e7f0 0x7f0bdc06ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:33.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.176+0000 7f0be2e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0bdc0b5830 0x7f0bdc0b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:33.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.176+0000 7f0be2e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bdc06d210 con 0x7f0bdc0ae2e0 2024-04-16T16:19:33.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.176+0000 7f0be2e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0bdc06c670 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.176+0000 7f0be2e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0bdc06ba60 con 0x7f0bdc0b5830 2024-04-16T16:19:33.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.176+0000 7f0be1000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0bdc06e7f0 0x7f0bdc06ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:33.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.176+0000 7f0be1a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0bdc0ae2e0 0x7f0bdc06cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43488/0 (socket says 172.21.15.45:43488) 2024-04-16T16:19:33.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.176+0000 7f0be1a00640 1 -- 172.21.15.45:0/1465405723 learned_addr learned my addr 172.21.15.45:0/1465405723 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:33.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.177+0000 7f0be1000640 1 -- 172.21.15.45:0/1465405723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0bdc0b5830 msgr2=0x7f0bdc0b5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:33.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.177+0000 7f0be2400640 1 --2- 172.21.15.45:0/1465405723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0bdc0b5830 0x7f0bdc0b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:33.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.177+0000 7f0bdbe00640 1 -- 172.21.15.45:0/1465405723 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3211893747 0 0) 0x7f0bdc06d210 con 0x7f0bdc0ae2e0 2024-04-16T16:19:33.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.177+0000 7f0be1000640 1 --2- 172.21.15.45:0/1465405723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0bdc0b5830 0x7f0bdc0b5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.177+0000 7f0be1000640 1 -- 172.21.15.45:0/1465405723 >> v1:172.21.15.45:6789/0 conn(0x7f0bdc0ae2e0 legacy=0x7f0bdc06cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:33.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.177+0000 7f0be1000640 1 -- 172.21.15.45:0/1465405723 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bdc06bd40 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.177+0000 7f0be2400640 1 --2- 172.21.15.45:0/1465405723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0bdc0b5830 0x7f0bdc0b5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:33.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.178+0000 7f0be1000640 1 --2- 172.21.15.45:0/1465405723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bdc06e7f0 0x7f0bdc06ec50 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f0bc80097a0 tx=0x7f0bc802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27ea31846142766 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:33.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.178+0000 7f0bdbe00640 1 -- 172.21.15.45:0/1465405723 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8039070 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.178+0000 7f0bdbe00640 1 -- 172.21.15.45:0/1465405723 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bc80316f0 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.178+0000 7f0bdbe00640 1 -- 172.21.15.45:0/1465405723 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8031ac0 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.179+0000 7f0be2e00640 1 -- 172.21.15.45:0/1465405723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bdc06e7f0 msgr2=0x7f0bdc06ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:33.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.179+0000 7f0be2e00640 1 --2- 172.21.15.45:0/1465405723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bdc06e7f0 0x7f0bdc06ec50 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f0bc80097a0 tx=0x7f0bc802b790 comp rx=0 tx=0).stop 2024-04-16T16:19:33.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.179+0000 7f0be2e00640 1 -- 172.21.15.45:0/1465405723 shutdown_connections 2024-04-16T16:19:33.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.179+0000 7f0be2e00640 1 --2- 172.21.15.45:0/1465405723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0bdc0b5830 0x7f0bdc0b5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.179+0000 7f0be2e00640 1 --2- 172.21.15.45:0/1465405723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bdc06e7f0 0x7f0bdc06ec50 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.179+0000 7f0be2e00640 1 -- 172.21.15.45:0/1465405723 >> 172.21.15.45:0/1465405723 conn(0x7f0bdc0d0f10 msgr2=0x7f0bdc0d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:33.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.179+0000 7f0be2e00640 1 -- 172.21.15.45:0/1465405723 shutdown_connections 2024-04-16T16:19:33.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.179+0000 7f0be2e00640 1 -- 172.21.15.45:0/1465405723 wait complete. 2024-04-16T16:19:33.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.181+0000 7f0be2e00640 1 Processor -- start 2024-04-16T16:19:33.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.181+0000 7f0be2e00640 1 -- start start 2024-04-16T16:19:33.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.182+0000 7f0be2e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bdc06e7f0 0x7f0bdc1f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:33.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.182+0000 7f0be2e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0bdc0ae2e0 0x7f0bdc151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:33.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.182+0000 7f0be2e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0bdc0b5830 0x7f0bdc153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:33.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.182+0000 7f0be2e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc0b61c0 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.182+0000 7f0be2e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc06c300 con 0x7f0bdc0b5830 2024-04-16T16:19:33.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.182+0000 7f0be2e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0bdc0ae6c0 con 0x7f0bdc0ae2e0 2024-04-16T16:19:33.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.182+0000 7f0be1a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bdc06e7f0 0x7f0bdc1f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:33.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.182+0000 7f0be1a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bdc06e7f0 0x7f0bdc1f3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34988/0 (socket says 172.21.15.45:34988) 2024-04-16T16:19:33.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.182+0000 7f0be1a00640 1 -- 172.21.15.45:0/1922827027 learned_addr learned my addr 172.21.15.45:0/1922827027 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:33.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.183+0000 7f0be1a00640 1 -- 172.21.15.45:0/1922827027 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0bdc0ae2e0 msgr2=0x7f0bdc151950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:33.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.183+0000 7f0be1a00640 1 --2- 172.21.15.45:0/1922827027 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0bdc0ae2e0 0x7f0bdc151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.183+0000 7f0be1a00640 1 -- 172.21.15.45:0/1922827027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0bdc0b5830 msgr2=0x7f0bdc153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:33.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.183+0000 7f0be2400640 1 --2- 172.21.15.45:0/1922827027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0bdc0b5830 0x7f0bdc153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:33.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.183+0000 7f0be1a00640 1 --2- 172.21.15.45:0/1922827027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0bdc0b5830 0x7f0bdc153220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.183+0000 7f0be1a00640 1 -- 172.21.15.45:0/1922827027 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bdc152180 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.183+0000 7f0be1a00640 1 --2- 172.21.15.45:0/1922827027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bdc06e7f0 0x7f0bdc1f3b20 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000c4d0 tx=0x7f0bd000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:33.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.183+0000 7f0bd9600640 1 -- 172.21.15.45:0/1922827027 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0018070 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.183+0000 7f0bd9600640 1 -- 172.21.15.45:0/1922827027 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd00074e0 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.183+0000 7f0bd9600640 1 -- 172.21.15.45:0/1922827027 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0004a20 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.183+0000 7f0be2e00640 1 -- 172.21.15.45:0/1922827027 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bdc152460 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.183+0000 7f0be2e00640 1 -- 172.21.15.45:0/1922827027 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bdc152a20 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.185+0000 7f0be2e00640 1 -- 172.21.15.45:0/1922827027 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bdc075780 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.186+0000 7f0bd9600640 1 -- 172.21.15.45:0/1922827027 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f0bd0004bc0 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.186+0000 7f0bd9600640 1 --2- 172.21.15.45:0/1922827027 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f0bbc06c980 0x7f0bbc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:33.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.187+0000 7f0bd9600640 1 -- 172.21.15.45:0/1922827027 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6397+0+0 (secure 0 0 0) 0x7f0bd008b230 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.187+0000 7f0be1000640 1 --2- 172.21.15.45:0/1922827027 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f0bbc06c980 0x7f0bbc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:33.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.188+0000 7f0be1000640 1 --2- 172.21.15.45:0/1922827027 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f0bbc06c980 0x7f0bbc06ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8036000 tx=0x7f0bc8030000 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:33.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.191+0000 7f0bd9600640 1 -- 172.21.15.45:0/1922827027 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0bd00592c0 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.402+0000 7f0be2e00640 1 -- 172.21.15.45:0/1922827027 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0bdc0b4650 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.406+0000 7f0bd9600640 1 -- 172.21.15.45:0/1922827027 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+17624 (secure 0 0 0) 0x7f0bd0058bc0 con 0x7f0bdc06e7f0 2024-04-16T16:19:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:33.406 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":102,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:19:28.777496+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:03.429683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:04.434699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3000963222":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/963070403":"2024-04-17T16:19:28.777448+0000","172.21.15.103:6832/3948990093":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/2784388736":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/2130767640":"2024-04-17T16:19:28.777448+0000","172.21.15.103:6833/839990480":"2024-04-17T16:19:22.373133+0000","172.21.15.103:6832/839990480":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2646915013":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/1161670790":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/287602360":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2666528075":"2024-04-17T16:19:22.373133+0000","172.21.15.103:6833/3948990093":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/4022617615":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2967808514":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/1615159946":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/2879899995":"2024-04-16T17:19:09.753356+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:19:33.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.409+0000 7f0be2e00640 1 -- 172.21.15.45:0/1922827027 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f0bbc06c980 msgr2=0x7f0bbc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:33.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.409+0000 7f0be2e00640 1 --2- 172.21.15.45:0/1922827027 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f0bbc06c980 0x7f0bbc06ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8036000 tx=0x7f0bc8030000 comp rx=0 tx=0).stop 2024-04-16T16:19:33.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.409+0000 7f0be2e00640 1 -- 172.21.15.45:0/1922827027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bdc06e7f0 msgr2=0x7f0bdc1f3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:33.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.409+0000 7f0be2e00640 1 --2- 172.21.15.45:0/1922827027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bdc06e7f0 0x7f0bdc1f3b20 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000c4d0 tx=0x7f0bd000c990 comp rx=0 tx=0).stop 2024-04-16T16:19:33.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.409+0000 7f0be2e00640 1 -- 172.21.15.45:0/1922827027 shutdown_connections 2024-04-16T16:19:33.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.409+0000 7f0be2e00640 1 --2- 172.21.15.45:0/1922827027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0bdc0b5830 0x7f0bdc153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.409+0000 7f0be2e00640 1 --2- 172.21.15.45:0/1922827027 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0bdc0ae2e0 0x7f0bdc151950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.409+0000 7f0be2e00640 1 --2- 172.21.15.45:0/1922827027 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f0bbc06c980 0x7f0bbc06ee40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.409+0000 7f0be2e00640 1 --2- 172.21.15.45:0/1922827027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bdc06e7f0 0x7f0bdc1f3b20 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.409+0000 7f0be2e00640 1 -- 172.21.15.45:0/1922827027 >> 172.21.15.45:0/1922827027 conn(0x7f0bdc0d0f10 msgr2=0x7f0bdc0b4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:33.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.409+0000 7f0be2e00640 1 -- 172.21.15.45:0/1922827027 shutdown_connections 2024-04-16T16:19:33.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.409+0000 7f0be2e00640 1 -- 172.21.15.45:0/1922827027 wait complete. 2024-04-16T16:19:33.428 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-16T16:19:33.428 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:19:33.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.653+0000 7f13e9a00640 1 Processor -- start 2024-04-16T16:19:33.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.653+0000 7f13e9a00640 1 -- start start 2024-04-16T16:19:33.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.654+0000 7f13e9a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f13e415edf0 0x7f13e40b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.655+0000 7f13e9a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13e40cfa50 0x7f13e40b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.655+0000 7f13e9a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e40d2e20 con 0x7f13e40b6a30 2024-04-16T16:19:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.655+0000 7f13e9a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f13e40b6ea0 con 0x7f13e415edf0 2024-04-16T16:19:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.655+0000 7f13e9a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f13e40d3be0 con 0x7f13e40cfa50 2024-04-16T16:19:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.655+0000 7f13e2000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f13e40b6a30 0x7f13e40cf340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57472/0 (socket says 172.21.15.45:57472) 2024-04-16T16:19:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.655+0000 7f13e2000640 1 -- 172.21.15.45:0/1905944439 learned_addr learned my addr 172.21.15.45:0/1905944439 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.655+0000 7f13e3400640 1 --2- 172.21.15.45:0/1905944439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13e40cfa50 0x7f13e40b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.655+0000 7f13e2a00640 1 --2- 172.21.15.45:0/1905944439 >> v2:172.21.15.103:3300/0 conn(0x7f13e415edf0 0x7f13e40b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 -- 172.21.15.45:0/1905944439 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1955930620 0 0) 0x7f13e40d2e20 con 0x7f13e40b6a30 2024-04-16T16:19:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 -- 172.21.15.45:0/1905944439 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13c8003520 con 0x7f13e40b6a30 2024-04-16T16:19:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 -- 172.21.15.45:0/1905944439 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2637054279 0 0) 0x7f13c8003520 con 0x7f13e40b6a30 2024-04-16T16:19:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 -- 172.21.15.45:0/1905944439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13e40cfa50 msgr2=0x7f13e40b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 --2- 172.21.15.45:0/1905944439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13e40cfa50 0x7f13e40b7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 -- 172.21.15.45:0/1905944439 >> v2:172.21.15.103:3300/0 conn(0x7f13e415edf0 msgr2=0x7f13e40b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 --2- 172.21.15.45:0/1905944439 >> v2:172.21.15.103:3300/0 conn(0x7f13e415edf0 0x7f13e40b64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 -- 172.21.15.45:0/1905944439 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e40d3100 con 0x7f13e40b6a30 2024-04-16T16:19:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 -- 172.21.15.45:0/1905944439 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f13d40019f0 con 0x7f13e40b6a30 2024-04-16T16:19:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 -- 172.21.15.45:0/1905944439 >> v1:172.21.15.45:6789/0 conn(0x7f13e40b6a30 legacy=0x7f13e40cf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 --2- 172.21.15.45:0/1905944439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13e40cfa50 0x7f13c80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 --2- 172.21.15.45:0/1905944439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13c8003f80 0x7f13c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 --2- 172.21.15.45:0/1905944439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13c8004d40 0x7f13c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 -- 172.21.15.45:0/1905944439 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f13e40b6ea0 con 0x7f13c8003f80 2024-04-16T16:19:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 -- 172.21.15.45:0/1905944439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f13e40d3be0 con 0x7f13c8004d40 2024-04-16T16:19:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.656+0000 7f13e1600640 1 -- 172.21.15.45:0/1905944439 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f13c800b9c0 con 0x7f13e40cfa50 2024-04-16T16:19:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.657+0000 7f13e2000640 1 --2- 172.21.15.45:0/1905944439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13c8003f80 0x7f13c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.657+0000 7f13e3400640 1 --2- 172.21.15.45:0/1905944439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13c8004d40 0x7f13c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.657+0000 7f13e2000640 1 -- 172.21.15.45:0/1905944439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13e40cfa50 msgr2=0x7f13c80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.657+0000 7f13e2000640 1 --2- 172.21.15.45:0/1905944439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13e40cfa50 0x7f13c80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.657+0000 7f13e2000640 1 -- 172.21.15.45:0/1905944439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13c8004d40 msgr2=0x7f13c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.657+0000 7f13e2000640 1 --2- 172.21.15.45:0/1905944439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13c8004d40 0x7f13c80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.657+0000 7f13e2000640 1 -- 172.21.15.45:0/1905944439 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c8003520 con 0x7f13c8003f80 2024-04-16T16:19:33.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.658+0000 7f13e2000640 1 --2- 172.21.15.45:0/1905944439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13c8003f80 0x7f13c8004800 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f13d400a7c0 tx=0x7f13c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7027400a811f9dd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:33.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.659+0000 7f13e1600640 1 -- 172.21.15.45:0/1905944439 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d400a410 con 0x7f13c8003f80 2024-04-16T16:19:33.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.659+0000 7f13e1600640 1 -- 172.21.15.45:0/1905944439 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d4034630 con 0x7f13c8003f80 2024-04-16T16:19:33.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.659+0000 7f13e1600640 1 -- 172.21.15.45:0/1905944439 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d4034a20 con 0x7f13c8003f80 2024-04-16T16:19:33.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.659+0000 7f13e9a00640 1 -- 172.21.15.45:0/1905944439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13c8003f80 msgr2=0x7f13c8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:33.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.659+0000 7f13e9a00640 1 --2- 172.21.15.45:0/1905944439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13c8003f80 0x7f13c8004800 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f13d400a7c0 tx=0x7f13c8005830 comp rx=0 tx=0).stop 2024-04-16T16:19:33.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.659+0000 7f13e3400640 1 -- 172.21.15.45:0/1905944439 reap_dead start 2024-04-16T16:19:33.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.659+0000 7f13e9a00640 1 -- 172.21.15.45:0/1905944439 shutdown_connections 2024-04-16T16:19:33.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.659+0000 7f13e9a00640 1 -- 172.21.15.45:0/1905944439 >> 172.21.15.45:0/1905944439 conn(0x7f13e4073690 msgr2=0x7f13e40d14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:33.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.659+0000 7f13e9a00640 1 -- 172.21.15.45:0/1905944439 shutdown_connections 2024-04-16T16:19:33.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.659+0000 7f13e9a00640 1 -- 172.21.15.45:0/1905944439 wait complete. 2024-04-16T16:19:33.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.661+0000 7f13e9a00640 1 Processor -- start 2024-04-16T16:19:33.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.661+0000 7f13e9a00640 1 -- start start 2024-04-16T16:19:33.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.662+0000 7f13e9a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13c8003f80 0x7f13e41566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:33.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.662+0000 7f13e9a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13e4157330 0x7f13e4153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:33.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.662+0000 7f13e9a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13e41577b0 0x7f13e40ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:33.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.662+0000 7f13e9a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f13e40b7a70 con 0x7f13e41577b0 2024-04-16T16:19:33.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.662+0000 7f13e9a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f13e40d3750 con 0x7f13c8003f80 2024-04-16T16:19:33.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.662+0000 7f13e9a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f13e40b7bf0 con 0x7f13e4157330 2024-04-16T16:19:33.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.663+0000 7f13e3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13e41577b0 0x7f13e40ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:33.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.663+0000 7f13e3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13e41577b0 0x7f13e40ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33228/0 (socket says 172.21.15.45:33228) 2024-04-16T16:19:33.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.663+0000 7f13e3400640 1 -- 172.21.15.45:0/1298730068 learned_addr learned my addr 172.21.15.45:0/1298730068 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:33.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.663+0000 7f13e2a00640 1 --2- 172.21.15.45:0/1298730068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13c8003f80 0x7f13e41566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:33.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.663+0000 7f13e3400640 1 -- 172.21.15.45:0/1298730068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13e4157330 msgr2=0x7f13e4153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:33.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.663+0000 7f13e3400640 1 --2- 172.21.15.45:0/1298730068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13e4157330 0x7f13e4153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.663+0000 7f13e3400640 1 -- 172.21.15.45:0/1298730068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13c8003f80 msgr2=0x7f13e41566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:33.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.663+0000 7f13e3400640 1 --2- 172.21.15.45:0/1298730068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13c8003f80 0x7f13e41566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.663+0000 7f13e3400640 1 -- 172.21.15.45:0/1298730068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e4155630 con 0x7f13e41577b0 2024-04-16T16:19:33.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.663+0000 7f13e2a00640 1 --2- 172.21.15.45:0/1298730068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13c8003f80 0x7f13e41566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:33.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.664+0000 7f13e3400640 1 --2- 172.21.15.45:0/1298730068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13e41577b0 0x7f13e40ca830 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f13e406df60 tx=0x7f13dc0109f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:33.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.664+0000 7f13daa00640 1 -- 172.21.15.45:0/1298730068 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc00f040 con 0x7f13e41577b0 2024-04-16T16:19:33.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.664+0000 7f13daa00640 1 -- 172.21.15.45:0/1298730068 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13dc01c070 con 0x7f13e41577b0 2024-04-16T16:19:33.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.664+0000 7f13daa00640 1 -- 172.21.15.45:0/1298730068 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc014b40 con 0x7f13e41577b0 2024-04-16T16:19:33.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.664+0000 7f13e9a00640 1 -- 172.21.15.45:0/1298730068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e4155910 con 0x7f13e41577b0 2024-04-16T16:19:33.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.664+0000 7f13e9a00640 1 -- 172.21.15.45:0/1298730068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e4155ed0 con 0x7f13e41577b0 2024-04-16T16:19:33.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.665+0000 7f13e9a00640 1 -- 172.21.15.45:0/1298730068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13e4075780 con 0x7f13e41577b0 2024-04-16T16:19:33.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.667+0000 7f13daa00640 1 -- 172.21.15.45:0/1298730068 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f13dc014ce0 con 0x7f13e41577b0 2024-04-16T16:19:33.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.667+0000 7f13daa00640 1 --2- 172.21.15.45:0/1298730068 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f13c006c980 0x7f13c006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:33.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.668+0000 7f13daa00640 1 -- 172.21.15.45:0/1298730068 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6397+0+0 (secure 0 0 0) 0x7f13dc08b910 con 0x7f13e41577b0 2024-04-16T16:19:33.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.668+0000 7f13e2a00640 1 --2- 172.21.15.45:0/1298730068 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f13c006c980 0x7f13c006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:33.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.670+0000 7f13e2a00640 1 --2- 172.21.15.45:0/1298730068 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f13c006c980 0x7f13c006ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f13e41553e0 tx=0x7f13cc008040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:33.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.672+0000 7f13daa00640 1 -- 172.21.15.45:0/1298730068 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f13dc026020 con 0x7f13e41577b0 2024-04-16T16:19:33.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.920+0000 7f13e9a00640 1 -- 172.21.15.45:0/1298730068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13e40d4c40 con 0x7f13e41577b0 2024-04-16T16:19:33.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.922+0000 7f13daa00640 1 -- 172.21.15.45:0/1298730068 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5186 (secure 0 0 0) 0x7f13dc059a30 con 0x7f13e41577b0 2024-04-16T16:19:33.921 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:33.921 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5804,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2862612740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2862612740},{"type":"v1","addr":"172.21.15.103:6836","nonce":2862612740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6304,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2934367463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2934367463},{"type":"v1","addr":"172.21.15.103:6837","nonce":2934367463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1425499906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1425499906},{"type":"v1","addr":"172.21.15.45:6835","nonce":1425499906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:19:05.457389+0000","modified":"2024-04-16T16:19:08.527316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7677},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7677":{"gid":7677,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/882883484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":882883484},{"type":"v1","addr":"172.21.15.45:6834","nonce":882883484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7677,"qdb_cluster":[7677]},"id":7}]} 2024-04-16T16:19:33.921 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 73 2024-04-16T16:19:33.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.924+0000 7f13e9a00640 1 -- 172.21.15.45:0/1298730068 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f13c006c980 msgr2=0x7f13c006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:33.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.925+0000 7f13e9a00640 1 --2- 172.21.15.45:0/1298730068 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f13c006c980 0x7f13c006ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f13e41553e0 tx=0x7f13cc008040 comp rx=0 tx=0).stop 2024-04-16T16:19:33.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.925+0000 7f13e9a00640 1 -- 172.21.15.45:0/1298730068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13e41577b0 msgr2=0x7f13e40ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:33.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.925+0000 7f13e9a00640 1 --2- 172.21.15.45:0/1298730068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13e41577b0 0x7f13e40ca830 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f13e406df60 tx=0x7f13dc0109f0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.925+0000 7f13e9a00640 1 -- 172.21.15.45:0/1298730068 shutdown_connections 2024-04-16T16:19:33.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.925+0000 7f13e9a00640 1 --2- 172.21.15.45:0/1298730068 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f13c006c980 0x7f13c006ee40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.925+0000 7f13e9a00640 1 --2- 172.21.15.45:0/1298730068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13e41577b0 0x7f13e40ca830 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.925+0000 7f13e9a00640 1 --2- 172.21.15.45:0/1298730068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13e4157330 0x7f13e4153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.925+0000 7f13e9a00640 1 --2- 172.21.15.45:0/1298730068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13c8003f80 0x7f13e41566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:33.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.925+0000 7f13e9a00640 1 -- 172.21.15.45:0/1298730068 >> 172.21.15.45:0/1298730068 conn(0x7f13e4073690 msgr2=0x7f13e4074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:33.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.925+0000 7f13e9a00640 1 -- 172.21.15.45:0/1298730068 shutdown_connections 2024-04-16T16:19:33.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:33.925+0000 7f13e9a00640 1 -- 172.21.15.45:0/1298730068 wait complete. 2024-04-16T16:19:33.945 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:19:34.065 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:34.064+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:34.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.168+0000 7fa317e00640 1 Processor -- start 2024-04-16T16:19:34.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.168+0000 7fa317e00640 1 -- start start 2024-04-16T16:19:34.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.169+0000 7fa317e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa31815e8f0 0x7fa3180cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.169+0000 7fa317e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3180cf080 0x7fa3180cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.169+0000 7fa317e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa31815ecd0 con 0x7fa3180ce9a0 2024-04-16T16:19:34.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.169+0000 7fa317e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa3180ce2e0 con 0x7fa31815e8f0 2024-04-16T16:19:34.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.169+0000 7fa317e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa3180cd6d0 con 0x7fa3180cf080 2024-04-16T16:19:34.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.170+0000 7fa316000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa3180ce9a0 0x7fa318162480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57480/0 (socket says 172.21.15.45:57480) 2024-04-16T16:19:34.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.170+0000 7fa316000640 1 -- 172.21.15.45:0/799249872 learned_addr learned my addr 172.21.15.45:0/799249872 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.170+0000 7fa316a00640 1 --2- 172.21.15.45:0/799249872 >> v2:172.21.15.103:3300/0 conn(0x7fa31815e8f0 0x7fa3180cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.170+0000 7fa317400640 1 --2- 172.21.15.45:0/799249872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3180cf080 0x7fa3180cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.170+0000 7fa315600640 1 -- 172.21.15.45:0/799249872 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 385989526 0 0) 0x7fa31815ecd0 con 0x7fa3180ce9a0 2024-04-16T16:19:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.170+0000 7fa315600640 1 -- 172.21.15.45:0/799249872 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2fc003520 con 0x7fa3180ce9a0 2024-04-16T16:19:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.170+0000 7fa315600640 1 -- 172.21.15.45:0/799249872 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1473065511 0 0) 0x7fa2fc003520 con 0x7fa3180ce9a0 2024-04-16T16:19:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa315600640 1 -- 172.21.15.45:0/799249872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3180cf080 msgr2=0x7fa3180cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa315600640 1 --2- 172.21.15.45:0/799249872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3180cf080 0x7fa3180cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa315600640 1 -- 172.21.15.45:0/799249872 >> v2:172.21.15.103:3300/0 conn(0x7fa31815e8f0 msgr2=0x7fa3180cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa315600640 1 --2- 172.21.15.45:0/799249872 >> v2:172.21.15.103:3300/0 conn(0x7fa31815e8f0 0x7fa3180cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa315600640 1 -- 172.21.15.45:0/799249872 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3180cd9b0 con 0x7fa3180ce9a0 2024-04-16T16:19:34.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa315600640 1 -- 172.21.15.45:0/799249872 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fa3080019f0 con 0x7fa3180ce9a0 2024-04-16T16:19:34.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa315600640 1 -- 172.21.15.45:0/799249872 >> v1:172.21.15.45:6789/0 conn(0x7fa3180ce9a0 legacy=0x7fa318162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:34.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa315600640 1 --2- 172.21.15.45:0/799249872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2fc0039b0 0x7fa2fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa315600640 1 --2- 172.21.15.45:0/799249872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3180cf080 0x7fa2fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa315600640 1 --2- 172.21.15.45:0/799249872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc004d40 0x7fa2fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa315600640 1 -- 172.21.15.45:0/799249872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa3180ce2e0 con 0x7fa2fc004d40 2024-04-16T16:19:34.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa315600640 1 -- 172.21.15.45:0/799249872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa3180cd6d0 con 0x7fa2fc0039b0 2024-04-16T16:19:34.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa315600640 1 -- 172.21.15.45:0/799249872 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa2fc00b9c0 con 0x7fa3180cf080 2024-04-16T16:19:34.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa317400640 1 --2- 172.21.15.45:0/799249872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc004d40 0x7fa2fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa316000640 1 --2- 172.21.15.45:0/799249872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3180cf080 0x7fa2fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.171+0000 7fa316a00640 1 --2- 172.21.15.45:0/799249872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2fc0039b0 0x7fa2fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.172+0000 7fa317400640 1 -- 172.21.15.45:0/799249872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3180cf080 msgr2=0x7fa2fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:34.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.172+0000 7fa317400640 1 --2- 172.21.15.45:0/799249872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3180cf080 0x7fa2fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.172+0000 7fa317400640 1 -- 172.21.15.45:0/799249872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2fc0039b0 msgr2=0x7fa2fc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:34.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.172+0000 7fa317400640 1 --2- 172.21.15.45:0/799249872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2fc0039b0 0x7fa2fc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.172+0000 7fa317400640 1 -- 172.21.15.45:0/799249872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2fc003520 con 0x7fa2fc004d40 2024-04-16T16:19:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.173+0000 7fa317400640 1 --2- 172.21.15.45:0/799249872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc004d40 0x7fa2fc0051c0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fa310009ac0 tx=0x7fa2fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b2f62221b24dae server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.173+0000 7fa315600640 1 -- 172.21.15.45:0/799249872 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa310039070 con 0x7fa2fc004d40 2024-04-16T16:19:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.173+0000 7fa315600640 1 -- 172.21.15.45:0/799249872 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa31002c6b0 con 0x7fa2fc004d40 2024-04-16T16:19:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.173+0000 7fa315600640 1 -- 172.21.15.45:0/799249872 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa310009690 con 0x7fa2fc004d40 2024-04-16T16:19:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.174+0000 7fa317e00640 1 -- 172.21.15.45:0/799249872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc004d40 msgr2=0x7fa2fc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.174+0000 7fa317e00640 1 --2- 172.21.15.45:0/799249872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc004d40 0x7fa2fc0051c0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fa310009ac0 tx=0x7fa2fc005830 comp rx=0 tx=0).stop 2024-04-16T16:19:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.174+0000 7fa317400640 1 -- 172.21.15.45:0/799249872 reap_dead start 2024-04-16T16:19:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.174+0000 7fa317e00640 1 -- 172.21.15.45:0/799249872 shutdown_connections 2024-04-16T16:19:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.174+0000 7fa317e00640 1 -- 172.21.15.45:0/799249872 >> 172.21.15.45:0/799249872 conn(0x7fa31806b930 msgr2=0x7fa31806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.174+0000 7fa317e00640 1 -- 172.21.15.45:0/799249872 shutdown_connections 2024-04-16T16:19:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.174+0000 7fa317e00640 1 -- 172.21.15.45:0/799249872 wait complete. 2024-04-16T16:19:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.176+0000 7fa317e00640 1 Processor -- start 2024-04-16T16:19:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.176+0000 7fa317e00640 1 -- start start 2024-04-16T16:19:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.177+0000 7fa317e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc004d40 0x7fa3181549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.177+0000 7fa317e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa31814f3b0 0x7fa3180cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.177+0000 7fa317e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3180cfa00 0x7fa318152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.177+0000 7fa317e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa3180cd360 con 0x7fa2fc004d40 2024-04-16T16:19:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.177+0000 7fa317e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa3180cdf70 con 0x7fa3180cfa00 2024-04-16T16:19:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.177+0000 7fa317e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa3180cbc40 con 0x7fa31814f3b0 2024-04-16T16:19:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.177+0000 7fa316a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc004d40 0x7fa3181549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.177+0000 7fa316a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc004d40 0x7fa3181549d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33240/0 (socket says 172.21.15.45:33240) 2024-04-16T16:19:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.177+0000 7fa316a00640 1 -- 172.21.15.45:0/2676733511 learned_addr learned my addr 172.21.15.45:0/2676733511 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.178+0000 7fa317400640 1 --2- 172.21.15.45:0/2676733511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3180cfa00 0x7fa318152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.178+0000 7fa316a00640 1 -- 172.21.15.45:0/2676733511 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa31814f3b0 msgr2=0x7fa3180cf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.178+0000 7fa316a00640 1 --2- 172.21.15.45:0/2676733511 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa31814f3b0 0x7fa3180cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.178+0000 7fa316a00640 1 -- 172.21.15.45:0/2676733511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3180cfa00 msgr2=0x7fa318152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.178+0000 7fa316a00640 1 --2- 172.21.15.45:0/2676733511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3180cfa00 0x7fa318152150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.178+0000 7fa316a00640 1 -- 172.21.15.45:0/2676733511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa318152ea0 con 0x7fa2fc004d40 2024-04-16T16:19:34.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.178+0000 7fa316a00640 1 --2- 172.21.15.45:0/2676733511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc004d40 0x7fa3181549d0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fa300011630 tx=0x7fa300011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:34.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.178+0000 7fa30ea00640 1 -- 172.21.15.45:0/2676733511 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30000f8c0 con 0x7fa2fc004d40 2024-04-16T16:19:34.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.179+0000 7fa30ea00640 1 -- 172.21.15.45:0/2676733511 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa30000fa60 con 0x7fa2fc004d40 2024-04-16T16:19:34.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.179+0000 7fa30ea00640 1 -- 172.21.15.45:0/2676733511 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30000fe10 con 0x7fa2fc004d40 2024-04-16T16:19:34.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.179+0000 7fa317e00640 1 -- 172.21.15.45:0/2676733511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa318153180 con 0x7fa2fc004d40 2024-04-16T16:19:34.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.179+0000 7fa317e00640 1 -- 172.21.15.45:0/2676733511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3181504f0 con 0x7fa2fc004d40 2024-04-16T16:19:34.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.180+0000 7fa317e00640 1 -- 172.21.15.45:0/2676733511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa31815e450 con 0x7fa2fc004d40 2024-04-16T16:19:34.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.181+0000 7fa30ea00640 1 -- 172.21.15.45:0/2676733511 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7fa300015460 con 0x7fa2fc004d40 2024-04-16T16:19:34.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.182+0000 7fa30ea00640 1 --2- 172.21.15.45:0/2676733511 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa2f406c9b0 0x7fa2f406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.182+0000 7fa30ea00640 1 -- 172.21.15.45:0/2676733511 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6397+0+0 (secure 0 0 0) 0x7fa30008c6a0 con 0x7fa2fc004d40 2024-04-16T16:19:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.183+0000 7fa316000640 1 --2- 172.21.15.45:0/2676733511 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa2f406c9b0 0x7fa2f406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.184+0000 7fa316000640 1 --2- 172.21.15.45:0/2676733511 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa2f406c9b0 0x7fa2f406ee70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa3080013d0 tx=0x7fa308001360 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:34.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.186+0000 7fa30ea00640 1 -- 172.21.15.45:0/2676733511 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa31815e450 con 0x7fa2fc004d40 2024-04-16T16:19:34.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.434+0000 7fa317e00640 1 -- 172.21.15.45:0/2676733511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa318075780 con 0x7fa2fc004d40 2024-04-16T16:19:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.436+0000 7fa30ea00640 1 -- 172.21.15.45:0/2676733511 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5186 (secure 0 0 0) 0x7fa300019030 con 0x7fa2fc004d40 2024-04-16T16:19:34.435 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:34.435 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5804,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2862612740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2862612740},{"type":"v1","addr":"172.21.15.103:6836","nonce":2862612740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6304,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2934367463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2934367463},{"type":"v1","addr":"172.21.15.103:6837","nonce":2934367463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1425499906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1425499906},{"type":"v1","addr":"172.21.15.45:6835","nonce":1425499906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:19:05.457389+0000","modified":"2024-04-16T16:19:08.527316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7677},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7677":{"gid":7677,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/882883484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":882883484},{"type":"v1","addr":"172.21.15.45:6834","nonce":882883484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7677,"qdb_cluster":[7677]},"id":7}]} 2024-04-16T16:19:34.436 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 73 2024-04-16T16:19:34.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.438+0000 7fa317e00640 1 -- 172.21.15.45:0/2676733511 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa2f406c9b0 msgr2=0x7fa2f406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:34.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.438+0000 7fa317e00640 1 --2- 172.21.15.45:0/2676733511 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa2f406c9b0 0x7fa2f406ee70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa3080013d0 tx=0x7fa308001360 comp rx=0 tx=0).stop 2024-04-16T16:19:34.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.438+0000 7fa317e00640 1 -- 172.21.15.45:0/2676733511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc004d40 msgr2=0x7fa3181549d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:34.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.438+0000 7fa317e00640 1 --2- 172.21.15.45:0/2676733511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc004d40 0x7fa3181549d0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fa300011630 tx=0x7fa300011af0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.439+0000 7fa317e00640 1 -- 172.21.15.45:0/2676733511 shutdown_connections 2024-04-16T16:19:34.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.439+0000 7fa317e00640 1 --2- 172.21.15.45:0/2676733511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3180cfa00 0x7fa318152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.439+0000 7fa317e00640 1 --2- 172.21.15.45:0/2676733511 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa31814f3b0 0x7fa3180cf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.439+0000 7fa317e00640 1 --2- 172.21.15.45:0/2676733511 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa2f406c9b0 0x7fa2f406ee70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.439+0000 7fa317e00640 1 --2- 172.21.15.45:0/2676733511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc004d40 0x7fa3181549d0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.439+0000 7fa317e00640 1 -- 172.21.15.45:0/2676733511 >> 172.21.15.45:0/2676733511 conn(0x7fa31806b930 msgr2=0x7fa3180b4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:34.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.439+0000 7fa317e00640 1 -- 172.21.15.45:0/2676733511 shutdown_connections 2024-04-16T16:19:34.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.439+0000 7fa317e00640 1 -- 172.21.15.45:0/2676733511 wait complete. 2024-04-16T16:19:34.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:19:34.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.664+0000 7f6a9f200640 1 Processor -- start 2024-04-16T16:19:34.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.664+0000 7f6a9f200640 1 -- start start 2024-04-16T16:19:34.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.666+0000 7f6a9f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa0146220 0x7f6aa015ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.666+0000 7f6a9f200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aa015d360 con 0x7f6aa01608a0 2024-04-16T16:19:34.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.666+0000 7f6a9f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aa015d640 con 0x7f6aa006f7a0 2024-04-16T16:19:34.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.666+0000 7f6a9f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa0160410 con 0x7f6aa0146220 2024-04-16T16:19:34.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.666+0000 7f6a9e800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6aa01608a0 0x7f6aa00cf360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57496/0 (socket says 172.21.15.45:57496) 2024-04-16T16:19:34.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.666+0000 7f6a9e800640 1 -- 172.21.15.45:0/3798554460 learned_addr learned my addr 172.21.15.45:0/3798554460 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:34.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9de00640 1 --2- 172.21.15.45:0/3798554460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa0146220 0x7f6aa015ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9ca00640 1 -- 172.21.15.45:0/3798554460 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1818881780 0 0) 0x7f6aa015d360 con 0x7f6aa01608a0 2024-04-16T16:19:34.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9ca00640 1 -- 172.21.15.45:0/3798554460 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a84003520 con 0x7f6aa01608a0 2024-04-16T16:19:34.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9ca00640 1 -- 172.21.15.45:0/3798554460 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3245653237 0 0) 0x7f6a84003520 con 0x7f6aa01608a0 2024-04-16T16:19:34.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9ca00640 1 -- 172.21.15.45:0/3798554460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa0146220 msgr2=0x7f6aa015ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:34.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9ca00640 1 --2- 172.21.15.45:0/3798554460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa0146220 0x7f6aa015ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9ca00640 1 -- 172.21.15.45:0/3798554460 >> v1:172.21.15.103:6789/0 conn(0x7f6aa006f7a0 legacy=0x7f6aa006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:34.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9ca00640 1 -- 172.21.15.45:0/3798554460 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa015d920 con 0x7f6aa01608a0 2024-04-16T16:19:34.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9ca00640 1 -- 172.21.15.45:0/3798554460 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f6a980019f0 con 0x7f6aa01608a0 2024-04-16T16:19:34.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9ca00640 1 -- 172.21.15.45:0/3798554460 >> v1:172.21.15.45:6789/0 conn(0x7f6aa01608a0 legacy=0x7f6aa00cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:34.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9ca00640 1 --2- 172.21.15.45:0/3798554460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a840039e0 0x7f6a84003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9ca00640 1 --2- 172.21.15.45:0/3798554460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa0146220 0x7f6a84004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9ca00640 1 --2- 172.21.15.45:0/3798554460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a84004d40 0x7f6a840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9ca00640 1 -- 172.21.15.45:0/3798554460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa0160410 con 0x7f6a84004d40 2024-04-16T16:19:34.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9ca00640 1 -- 172.21.15.45:0/3798554460 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8400baa0 con 0x7f6a840039e0 2024-04-16T16:19:34.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.667+0000 7f6a9ca00640 1 -- 172.21.15.45:0/3798554460 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6a8400bca0 con 0x7f6aa0146220 2024-04-16T16:19:34.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.668+0000 7f6a9e800640 1 --2- 172.21.15.45:0/3798554460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a84004d40 0x7f6a840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.668+0000 7f6a9de00640 1 --2- 172.21.15.45:0/3798554460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a840039e0 0x7f6a84003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.668+0000 7f6a9d400640 1 --2- 172.21.15.45:0/3798554460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa0146220 0x7f6a84004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.668+0000 7f6a9e800640 1 -- 172.21.15.45:0/3798554460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa0146220 msgr2=0x7f6a84004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:34.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.668+0000 7f6a9e800640 1 --2- 172.21.15.45:0/3798554460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa0146220 0x7f6a84004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.668+0000 7f6a9e800640 1 -- 172.21.15.45:0/3798554460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a840039e0 msgr2=0x7f6a84003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:34.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.668+0000 7f6a9e800640 1 --2- 172.21.15.45:0/3798554460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a840039e0 0x7f6a84003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.668+0000 7f6a9e800640 1 -- 172.21.15.45:0/3798554460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a84003520 con 0x7f6a84004d40 2024-04-16T16:19:34.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.669+0000 7f6a9e800640 1 --2- 172.21.15.45:0/3798554460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a84004d40 0x7f6a840051c0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f6a9800a780 tx=0x7f6a84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93aae55b22c64051 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:34.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.670+0000 7f6a9ca00640 1 -- 172.21.15.45:0/3798554460 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9800a3d0 con 0x7f6a84004d40 2024-04-16T16:19:34.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.670+0000 7f6a9ca00640 1 -- 172.21.15.45:0/3798554460 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a98034630 con 0x7f6a84004d40 2024-04-16T16:19:34.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.670+0000 7f6a9ca00640 1 -- 172.21.15.45:0/3798554460 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a98034a60 con 0x7f6a84004d40 2024-04-16T16:19:34.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.670+0000 7f6a9f200640 1 -- 172.21.15.45:0/3798554460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a84004d40 msgr2=0x7f6a840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:34.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.670+0000 7f6a9f200640 1 --2- 172.21.15.45:0/3798554460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a84004d40 0x7f6a840051c0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f6a9800a780 tx=0x7f6a84005830 comp rx=0 tx=0).stop 2024-04-16T16:19:34.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.670+0000 7f6a9e800640 1 -- 172.21.15.45:0/3798554460 reap_dead start 2024-04-16T16:19:34.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.670+0000 7f6a9f200640 1 -- 172.21.15.45:0/3798554460 shutdown_connections 2024-04-16T16:19:34.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.670+0000 7f6a9f200640 1 -- 172.21.15.45:0/3798554460 >> 172.21.15.45:0/3798554460 conn(0x7f6aa006b930 msgr2=0x7f6aa006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:34.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.671+0000 7f6a9f200640 1 -- 172.21.15.45:0/3798554460 shutdown_connections 2024-04-16T16:19:34.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.671+0000 7f6a9f200640 1 -- 172.21.15.45:0/3798554460 wait complete. 2024-04-16T16:19:34.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.672+0000 7f6a9f200640 1 Processor -- start 2024-04-16T16:19:34.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.673+0000 7f6a9f200640 1 -- start start 2024-04-16T16:19:34.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.673+0000 7f6a9f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a84004d40 0x7f6aa00b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.674+0000 7f6a9f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa013e770 0x7f6aa015c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.674+0000 7f6a9f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa013ad90 0x7f6aa015e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.674+0000 7f6a9f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa006fc20 con 0x7f6aa013e770 2024-04-16T16:19:34.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.674+0000 7f6a9f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa015dc00 con 0x7f6a84004d40 2024-04-16T16:19:34.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.674+0000 7f6a9f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa006fda0 con 0x7f6aa013ad90 2024-04-16T16:19:34.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.674+0000 7f6a9d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa013e770 0x7f6aa015c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.674+0000 7f6a9d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa013e770 0x7f6aa015c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33254/0 (socket says 172.21.15.45:33254) 2024-04-16T16:19:34.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.674+0000 7f6a9d400640 1 -- 172.21.15.45:0/2817545601 learned_addr learned my addr 172.21.15.45:0/2817545601 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:34.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.674+0000 7f6a9de00640 1 --2- 172.21.15.45:0/2817545601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a84004d40 0x7f6aa00b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.674+0000 7f6a9e800640 1 --2- 172.21.15.45:0/2817545601 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa013ad90 0x7f6aa015e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.674+0000 7f6a9d400640 1 -- 172.21.15.45:0/2817545601 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa013ad90 msgr2=0x7f6aa015e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:34.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.674+0000 7f6a9d400640 1 --2- 172.21.15.45:0/2817545601 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa013ad90 0x7f6aa015e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.674+0000 7f6a9d400640 1 -- 172.21.15.45:0/2817545601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a84004d40 msgr2=0x7f6aa00b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:34.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.674+0000 7f6a9d400640 1 --2- 172.21.15.45:0/2817545601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a84004d40 0x7f6aa00b30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.674+0000 7f6a9d400640 1 -- 172.21.15.45:0/2817545601 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa00b2bd0 con 0x7f6aa013e770 2024-04-16T16:19:34.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.675+0000 7f6a9d400640 1 --2- 172.21.15.45:0/2817545601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa013e770 0x7f6aa015c040 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00e580 tx=0x7f6a8c00ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:34.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.675+0000 7f6a96000640 1 -- 172.21.15.45:0/2817545601 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c0052e0 con 0x7f6aa013e770 2024-04-16T16:19:34.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.675+0000 7f6a96000640 1 -- 172.21.15.45:0/2817545601 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a8c018070 con 0x7f6aa013e770 2024-04-16T16:19:34.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.675+0000 7f6a96000640 1 -- 172.21.15.45:0/2817545601 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c003810 con 0x7f6aa013e770 2024-04-16T16:19:34.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.675+0000 7f6a9f200640 1 -- 172.21.15.45:0/2817545601 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aa0142890 con 0x7f6aa013e770 2024-04-16T16:19:34.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.675+0000 7f6a9f200640 1 -- 172.21.15.45:0/2817545601 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aa0142e50 con 0x7f6aa013e770 2024-04-16T16:19:34.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.676+0000 7f6a9f200640 1 -- 172.21.15.45:0/2817545601 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aa0075780 con 0x7f6aa013e770 2024-04-16T16:19:34.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.677+0000 7f6a96000640 1 -- 172.21.15.45:0/2817545601 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f6a8c0039b0 con 0x7f6aa013e770 2024-04-16T16:19:34.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.678+0000 7f6a96000640 1 --2- 172.21.15.45:0/2817545601 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f6a8006c980 0x7f6a8006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:34.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.678+0000 7f6a96000640 1 -- 172.21.15.45:0/2817545601 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6397+0+0 (secure 0 0 0) 0x7f6a8c08bc10 con 0x7f6aa013e770 2024-04-16T16:19:34.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.679+0000 7f6a9de00640 1 --2- 172.21.15.45:0/2817545601 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f6a8006c980 0x7f6a8006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:34.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.680+0000 7f6a9de00640 1 --2- 172.21.15.45:0/2817545601 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f6a8006c980 0x7f6a8006ee40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009a10 tx=0x7f6a90008040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:34.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.683+0000 7f6a96000640 1 -- 172.21.15.45:0/2817545601 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6a8c059cb0 con 0x7f6aa013e770 2024-04-16T16:19:34.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.894+0000 7f6a9f200640 1 -- 172.21.15.45:0/2817545601 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6aa00cff60 con 0x7f6aa013e770 2024-04-16T16:19:34.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.897+0000 7f6a96000640 1 -- 172.21.15.45:0/2817545601 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+17624 (secure 0 0 0) 0x7f6a8c0595b0 con 0x7f6aa013e770 2024-04-16T16:19:34.897 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:34.897 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":102,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:19:28.777496+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:03.429683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:04.434699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3000963222":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/963070403":"2024-04-17T16:19:28.777448+0000","172.21.15.103:6832/3948990093":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/2784388736":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/2130767640":"2024-04-17T16:19:28.777448+0000","172.21.15.103:6833/839990480":"2024-04-17T16:19:22.373133+0000","172.21.15.103:6832/839990480":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2646915013":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/1161670790":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/287602360":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2666528075":"2024-04-17T16:19:22.373133+0000","172.21.15.103:6833/3948990093":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/4022617615":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2967808514":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/1615159946":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/2879899995":"2024-04-16T17:19:09.753356+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:19:34.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.900+0000 7f6a9f200640 1 -- 172.21.15.45:0/2817545601 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f6a8006c980 msgr2=0x7f6a8006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:34.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.900+0000 7f6a9f200640 1 --2- 172.21.15.45:0/2817545601 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f6a8006c980 0x7f6a8006ee40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009a10 tx=0x7f6a90008040 comp rx=0 tx=0).stop 2024-04-16T16:19:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.900+0000 7f6a9f200640 1 -- 172.21.15.45:0/2817545601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa013e770 msgr2=0x7f6aa015c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.900+0000 7f6a9f200640 1 --2- 172.21.15.45:0/2817545601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa013e770 0x7f6aa015c040 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00e580 tx=0x7f6a8c00ea40 comp rx=0 tx=0).stop 2024-04-16T16:19:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.900+0000 7f6a9f200640 1 -- 172.21.15.45:0/2817545601 shutdown_connections 2024-04-16T16:19:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.900+0000 7f6a9f200640 1 --2- 172.21.15.45:0/2817545601 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6aa013ad90 0x7f6aa015e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.900+0000 7f6a9f200640 1 --2- 172.21.15.45:0/2817545601 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f6a8006c980 0x7f6a8006ee40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.900+0000 7f6a9f200640 1 --2- 172.21.15.45:0/2817545601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aa013e770 0x7f6aa015c040 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.900+0000 7f6a9f200640 1 --2- 172.21.15.45:0/2817545601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a84004d40 0x7f6aa00b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.900+0000 7f6a9f200640 1 -- 172.21.15.45:0/2817545601 >> 172.21.15.45:0/2817545601 conn(0x7f6aa006b930 msgr2=0x7f6aa015dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.900+0000 7f6a9f200640 1 -- 172.21.15.45:0/2817545601 shutdown_connections 2024-04-16T16:19:34.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:34.900+0000 7f6a9f200640 1 -- 172.21.15.45:0/2817545601 wait complete. 2024-04-16T16:19:34.920 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-16T16:19:35.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.145+0000 7fb236e00640 1 Processor -- start 2024-04-16T16:19:35.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.145+0000 7fb236e00640 1 -- start start 2024-04-16T16:19:35.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.147+0000 7fb236e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb23015e830 0x7fb2300ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:35.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.147+0000 7fb236e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb2300cb7d0 0x7fb2300cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:35.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.147+0000 7fb236e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb23015f1b0 con 0x7fb2300cbc80 2024-04-16T16:19:35.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.147+0000 7fb236e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb2300cf280 con 0x7fb2300cb7d0 2024-04-16T16:19:35.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.147+0000 7fb236e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb2300ce190 con 0x7fb23015e830 2024-04-16T16:19:35.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.148+0000 7fb235000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb2300cb7d0 0x7fb2300cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:35.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.148+0000 7fb235000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb2300cb7d0 0x7fb2300cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33260/0 (socket says 172.21.15.45:33260) 2024-04-16T16:19:35.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.148+0000 7fb235000640 1 -- 172.21.15.45:0/331793841 learned_addr learned my addr 172.21.15.45:0/331793841 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:35.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.148+0000 7fb235a00640 1 --2- 172.21.15.45:0/331793841 >> v2:172.21.15.103:3300/0 conn(0x7fb23015e830 0x7fb2300ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:35.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.148+0000 7fb235000640 1 -- 172.21.15.45:0/331793841 >> v2:172.21.15.103:3300/0 conn(0x7fb23015e830 msgr2=0x7fb2300ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:35.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.148+0000 7fb235000640 1 --2- 172.21.15.45:0/331793841 >> v2:172.21.15.103:3300/0 conn(0x7fb23015e830 0x7fb2300ce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:35.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.149+0000 7fb235000640 1 -- 172.21.15.45:0/331793841 >> v1:172.21.15.103:6789/0 conn(0x7fb2300cbc80 legacy=0x7fb2300cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:35.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.149+0000 7fb235000640 1 -- 172.21.15.45:0/331793841 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2300ce470 con 0x7fb2300cb7d0 2024-04-16T16:19:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.150+0000 7fb235000640 1 --2- 172.21.15.45:0/331793841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2300cb7d0 0x7fb2300cd620 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fb230070350 tx=0x7fb22002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0f3bd2e806af9d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.150+0000 7fb22fe00640 1 -- 172.21.15.45:0/331793841 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb220035020 con 0x7fb2300cb7d0 2024-04-16T16:19:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.150+0000 7fb22fe00640 1 -- 172.21.15.45:0/331793841 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb220031870 con 0x7fb2300cb7d0 2024-04-16T16:19:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.151+0000 7fb22fe00640 1 -- 172.21.15.45:0/331793841 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb220031c40 con 0x7fb2300cb7d0 2024-04-16T16:19:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.151+0000 7fb236e00640 1 -- 172.21.15.45:0/331793841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2300cb7d0 msgr2=0x7fb2300cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.151+0000 7fb236e00640 1 --2- 172.21.15.45:0/331793841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2300cb7d0 0x7fb2300cd620 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fb230070350 tx=0x7fb22002b910 comp rx=0 tx=0).stop 2024-04-16T16:19:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.151+0000 7fb236e00640 1 -- 172.21.15.45:0/331793841 shutdown_connections 2024-04-16T16:19:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.151+0000 7fb236e00640 1 --2- 172.21.15.45:0/331793841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2300cb7d0 0x7fb2300cd620 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.151+0000 7fb236e00640 1 --2- 172.21.15.45:0/331793841 >> v2:172.21.15.103:3300/0 conn(0x7fb23015e830 0x7fb2300ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.151+0000 7fb236e00640 1 -- 172.21.15.45:0/331793841 >> 172.21.15.45:0/331793841 conn(0x7fb23006b930 msgr2=0x7fb23006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.151+0000 7fb236e00640 1 -- 172.21.15.45:0/331793841 shutdown_connections 2024-04-16T16:19:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.151+0000 7fb236e00640 1 -- 172.21.15.45:0/331793841 wait complete. 2024-04-16T16:19:35.152 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:35.149+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:35.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.153+0000 7fb236e00640 1 Processor -- start 2024-04-16T16:19:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.153+0000 7fb236e00640 1 -- start start 2024-04-16T16:19:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.154+0000 7fb236e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2300cb7d0 0x7fb230160920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.154+0000 7fb236e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2300cbc80 0x7fb2301f4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.154+0000 7fb236e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb23015e830 0x7fb2301f4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.154+0000 7fb236e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb23015ec10 con 0x7fb2300cb7d0 2024-04-16T16:19:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.154+0000 7fb236e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2300cb2b0 con 0x7fb2300cbc80 2024-04-16T16:19:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.154+0000 7fb236e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb2300cac30 con 0x7fb23015e830 2024-04-16T16:19:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.154+0000 7fb235a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2300cb7d0 0x7fb230160920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.154+0000 7fb235a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2300cb7d0 0x7fb230160920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33264/0 (socket says 172.21.15.45:33264) 2024-04-16T16:19:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.154+0000 7fb235a00640 1 -- 172.21.15.45:0/4282958686 learned_addr learned my addr 172.21.15.45:0/4282958686 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.155+0000 7fb235000640 1 --2- 172.21.15.45:0/4282958686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2300cbc80 0x7fb2301f4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.155+0000 7fb235a00640 1 -- 172.21.15.45:0/4282958686 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb23015e830 msgr2=0x7fb2301f4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.155+0000 7fb235a00640 1 --2- 172.21.15.45:0/4282958686 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb23015e830 0x7fb2301f4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.155+0000 7fb235a00640 1 -- 172.21.15.45:0/4282958686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2300cbc80 msgr2=0x7fb2301f4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.155+0000 7fb235a00640 1 --2- 172.21.15.45:0/4282958686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2300cbc80 0x7fb2301f4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.155+0000 7fb235a00640 1 -- 172.21.15.45:0/4282958686 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2301f52d0 con 0x7fb2300cb7d0 2024-04-16T16:19:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.155+0000 7fb235a00640 1 --2- 172.21.15.45:0/4282958686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2300cb7d0 0x7fb230160920 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fb22400e930 tx=0x7fb22400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.155+0000 7fb22d600640 1 -- 172.21.15.45:0/4282958686 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb224017070 con 0x7fb2300cb7d0 2024-04-16T16:19:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.155+0000 7fb22d600640 1 -- 172.21.15.45:0/4282958686 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb22400ca00 con 0x7fb2300cb7d0 2024-04-16T16:19:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.155+0000 7fb236e00640 1 -- 172.21.15.45:0/4282958686 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2301458b0 con 0x7fb2300cb7d0 2024-04-16T16:19:35.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.155+0000 7fb22d600640 1 -- 172.21.15.45:0/4282958686 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22400cdf0 con 0x7fb2300cb7d0 2024-04-16T16:19:35.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.156+0000 7fb236e00640 1 -- 172.21.15.45:0/4282958686 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb230145e70 con 0x7fb2300cb7d0 2024-04-16T16:19:35.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.157+0000 7fb236e00640 1 -- 172.21.15.45:0/4282958686 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb230075780 con 0x7fb2300cb7d0 2024-04-16T16:19:35.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.158+0000 7fb22d600640 1 -- 172.21.15.45:0/4282958686 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7fb22401a070 con 0x7fb2300cb7d0 2024-04-16T16:19:35.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.159+0000 7fb22d600640 1 --2- 172.21.15.45:0/4282958686 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fb21406c980 0x7fb21406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:35.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.159+0000 7fb22d600640 1 -- 172.21.15.45:0/4282958686 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6397+0+0 (secure 0 0 0) 0x7fb22408bf20 con 0x7fb2300cb7d0 2024-04-16T16:19:35.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.160+0000 7fb235000640 1 --2- 172.21.15.45:0/4282958686 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fb21406c980 0x7fb21406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:35.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.161+0000 7fb235000640 1 --2- 172.21.15.45:0/4282958686 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fb21406c980 0x7fb21406ee40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb220036000 tx=0x7fb220008030 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:35.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.163+0000 7fb22d600640 1 -- 172.21.15.45:0/4282958686 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb22405a100 con 0x7fb2300cb7d0 2024-04-16T16:19:35.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.406+0000 7fb236e00640 1 -- 172.21.15.45:0/4282958686 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb2300b5b90 con 0x7fb2300cb7d0 2024-04-16T16:19:35.405 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-04-16T16:19:35.407+0000 7fe7a4800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T16:19:35.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.409+0000 7fb22d600640 1 -- 172.21.15.45:0/4282958686 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v74) ==== 161+0+0 (secure 0 0 0) 0x7fb224059a00 con 0x7fb2300cb7d0 2024-04-16T16:19:35.409 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-16T16:19:35.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.413+0000 7fb236e00640 1 -- 172.21.15.45:0/4282958686 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fb21406c980 msgr2=0x7fb21406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:35.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.413+0000 7fb236e00640 1 --2- 172.21.15.45:0/4282958686 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fb21406c980 0x7fb21406ee40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb220036000 tx=0x7fb220008030 comp rx=0 tx=0).stop 2024-04-16T16:19:35.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.413+0000 7fb236e00640 1 -- 172.21.15.45:0/4282958686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2300cb7d0 msgr2=0x7fb230160920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:35.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.413+0000 7fb236e00640 1 --2- 172.21.15.45:0/4282958686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2300cb7d0 0x7fb230160920 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fb22400e930 tx=0x7fb22400edf0 comp rx=0 tx=0).stop 2024-04-16T16:19:35.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.414+0000 7fb236e00640 1 -- 172.21.15.45:0/4282958686 shutdown_connections 2024-04-16T16:19:35.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.414+0000 7fb236e00640 1 --2- 172.21.15.45:0/4282958686 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb23015e830 0x7fb2301f4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:35.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.414+0000 7fb236e00640 1 --2- 172.21.15.45:0/4282958686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2300cbc80 0x7fb2301f4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:35.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.414+0000 7fb236e00640 1 --2- 172.21.15.45:0/4282958686 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fb21406c980 0x7fb21406ee40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:35.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.414+0000 7fb236e00640 1 --2- 172.21.15.45:0/4282958686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2300cb7d0 0x7fb230160920 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:35.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.414+0000 7fb236e00640 1 -- 172.21.15.45:0/4282958686 >> 172.21.15.45:0/4282958686 conn(0x7fb23006b930 msgr2=0x7fb23006f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:35.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.414+0000 7fb236e00640 1 -- 172.21.15.45:0/4282958686 shutdown_connections 2024-04-16T16:19:35.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.414+0000 7fb236e00640 1 -- 172.21.15.45:0/4282958686 wait complete. 2024-04-16T16:19:35.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-16T16:19:35.433 INFO:tasks.ceph.mds.c.smithi045.stderr: -3597> 2024-04-16T16:19:10.910+0000 7f0e99000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:19:35.508 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2024-04-16T16:19:35.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.630+0000 7f3ddd800640 1 Processor -- start 2024-04-16T16:19:35.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.631+0000 7f3ddd800640 1 -- start start 2024-04-16T16:19:35.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.632+0000 7f3ddd800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3dd815b920 0x7f3dd80b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:35.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.632+0000 7f3ddd800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3dd80d0e80 0x7f3dd806ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:35.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.632+0000 7f3ddd800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dd80ccb70 con 0x7f3dd80afda0 2024-04-16T16:19:35.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.632+0000 7f3ddd800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3dd806e7a0 con 0x7f3dd80d0e80 2024-04-16T16:19:35.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.632+0000 7f3ddd800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3dd806f170 con 0x7f3dd815b920 2024-04-16T16:19:35.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.632+0000 7f3dd7400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3dd80d0e80 0x7f3dd806ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:35.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.632+0000 7f3dd6000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3dd80afda0 0x7f3dd80d0770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57498/0 (socket says 172.21.15.45:57498) 2024-04-16T16:19:35.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.633+0000 7f3dd6000640 1 -- 172.21.15.45:0/584046668 learned_addr learned my addr 172.21.15.45:0/584046668 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:35.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.633+0000 7f3dd6a00640 1 --2- 172.21.15.45:0/584046668 >> v2:172.21.15.103:3300/0 conn(0x7f3dd815b920 0x7f3dd80b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:35.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.633+0000 7f3dd5600640 1 -- 172.21.15.45:0/584046668 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2085490542 0 0) 0x7f3dd80ccb70 con 0x7f3dd80afda0 2024-04-16T16:19:35.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.633+0000 7f3dd5600640 1 -- 172.21.15.45:0/584046668 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dbc005730 con 0x7f3dd80afda0 2024-04-16T16:19:35.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.633+0000 7f3dd7400640 1 -- 172.21.15.45:0/584046668 >> v2:172.21.15.103:3300/0 conn(0x7f3dd815b920 msgr2=0x7f3dd80b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:35.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.633+0000 7f3dd7400640 1 --2- 172.21.15.45:0/584046668 >> v2:172.21.15.103:3300/0 conn(0x7f3dd815b920 0x7f3dd80b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:35.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.633+0000 7f3dd7400640 1 -- 172.21.15.45:0/584046668 >> v1:172.21.15.45:6789/0 conn(0x7f3dd80afda0 legacy=0x7f3dd80d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:35.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.633+0000 7f3dd7400640 1 -- 172.21.15.45:0/584046668 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd80cce50 con 0x7f3dd80d0e80 2024-04-16T16:19:35.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.635+0000 7f3dd7400640 1 --2- 172.21.15.45:0/584046668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd80d0e80 0x7f3dd806ec30 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80d1480 tx=0x7f3dd002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e74962e36c9fa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.635+0000 7f3dd5600640 1 -- 172.21.15.45:0/584046668 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd00315a0 con 0x7f3dd80d0e80 2024-04-16T16:19:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.635+0000 7f3dd5600640 1 -- 172.21.15.45:0/584046668 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dd0037070 con 0x7f3dd80d0e80 2024-04-16T16:19:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.635+0000 7f3dd5600640 1 -- 172.21.15.45:0/584046668 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0031940 con 0x7f3dd80d0e80 2024-04-16T16:19:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.635+0000 7f3ddd800640 1 -- 172.21.15.45:0/584046668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd80d0e80 msgr2=0x7f3dd806ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.635+0000 7f3ddd800640 1 --2- 172.21.15.45:0/584046668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd80d0e80 0x7f3dd806ec30 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80d1480 tx=0x7f3dd002b840 comp rx=0 tx=0).stop 2024-04-16T16:19:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.635+0000 7f3ddd800640 1 -- 172.21.15.45:0/584046668 shutdown_connections 2024-04-16T16:19:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.635+0000 7f3ddd800640 1 --2- 172.21.15.45:0/584046668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd80d0e80 0x7f3dd806ec30 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.635+0000 7f3ddd800640 1 --2- 172.21.15.45:0/584046668 >> v2:172.21.15.103:3300/0 conn(0x7f3dd815b920 0x7f3dd80b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.636+0000 7f3ddd800640 1 -- 172.21.15.45:0/584046668 >> 172.21.15.45:0/584046668 conn(0x7f3dd80b3970 msgr2=0x7f3dd80b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:35.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.636+0000 7f3ddd800640 1 -- 172.21.15.45:0/584046668 shutdown_connections 2024-04-16T16:19:35.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.636+0000 7f3ddd800640 1 -- 172.21.15.45:0/584046668 wait complete. 2024-04-16T16:19:35.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.637+0000 7f3ddd800640 1 Processor -- start 2024-04-16T16:19:35.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.637+0000 7f3ddd800640 1 -- start start 2024-04-16T16:19:35.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.638+0000 7f3ddd800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd80afda0 0x7f3dd815d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:35.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.638+0000 7f3ddd800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dd80d0e80 0x7f3dd815e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:35.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.638+0000 7f3ddd800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd815b920 0x7f3dd814e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:35.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.638+0000 7f3ddd800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd80ce760 con 0x7f3dd80afda0 2024-04-16T16:19:35.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.638+0000 7f3ddd800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd80ce5e0 con 0x7f3dd80d0e80 2024-04-16T16:19:35.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.638+0000 7f3ddd800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3dd80ce8e0 con 0x7f3dd815b920 2024-04-16T16:19:35.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.639+0000 7f3dd6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd80afda0 0x7f3dd815d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:35.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.639+0000 7f3dd6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd80afda0 0x7f3dd815d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33294/0 (socket says 172.21.15.45:33294) 2024-04-16T16:19:35.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.639+0000 7f3dd6a00640 1 -- 172.21.15.45:0/418807383 learned_addr learned my addr 172.21.15.45:0/418807383 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:35.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.639+0000 7f3dd7400640 1 --2- 172.21.15.45:0/418807383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd815b920 0x7f3dd814e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:35.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.639+0000 7f3dd6a00640 1 -- 172.21.15.45:0/418807383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd815b920 msgr2=0x7f3dd814e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:35.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.639+0000 7f3dd6a00640 1 --2- 172.21.15.45:0/418807383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd815b920 0x7f3dd814e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:35.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.639+0000 7f3dd6a00640 1 -- 172.21.15.45:0/418807383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dd80d0e80 msgr2=0x7f3dd815e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:35.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.639+0000 7f3dd6000640 1 --2- 172.21.15.45:0/418807383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dd80d0e80 0x7f3dd815e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:35.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.639+0000 7f3dd6a00640 1 --2- 172.21.15.45:0/418807383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dd80d0e80 0x7f3dd815e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:35.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.639+0000 7f3dd6a00640 1 -- 172.21.15.45:0/418807383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd815c190 con 0x7f3dd80afda0 2024-04-16T16:19:35.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.639+0000 7f3dd6a00640 1 --2- 172.21.15.45:0/418807383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd80afda0 0x7f3dd815d230 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8007e40 tx=0x7f3dc800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:35.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.640+0000 7f3dcea00640 1 -- 172.21.15.45:0/418807383 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8014620 con 0x7f3dd80afda0 2024-04-16T16:19:35.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.640+0000 7f3dcea00640 1 -- 172.21.15.45:0/418807383 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc8019070 con 0x7f3dd80afda0 2024-04-16T16:19:35.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.640+0000 7f3dcea00640 1 -- 172.21.15.45:0/418807383 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8014a00 con 0x7f3dd80afda0 2024-04-16T16:19:35.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.640+0000 7f3ddd800640 1 -- 172.21.15.45:0/418807383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dd815c470 con 0x7f3dd80afda0 2024-04-16T16:19:35.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.640+0000 7f3ddd800640 1 -- 172.21.15.45:0/418807383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dd815ca30 con 0x7f3dd80afda0 2024-04-16T16:19:35.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.641+0000 7f3ddd800640 1 -- 172.21.15.45:0/418807383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dd8075780 con 0x7f3dd80afda0 2024-04-16T16:19:35.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.642+0000 7f3dcea00640 1 -- 172.21.15.45:0/418807383 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f3dc8014ba0 con 0x7f3dd80afda0 2024-04-16T16:19:35.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.643+0000 7f3dcea00640 1 --2- 172.21.15.45:0/418807383 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3db806c980 0x7f3db806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:35.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.643+0000 7f3dcea00640 1 -- 172.21.15.45:0/418807383 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6483+0+0 (secure 0 0 0) 0x7f3dc808c840 con 0x7f3dd80afda0 2024-04-16T16:19:35.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.643+0000 7f3dd6000640 1 --2- 172.21.15.45:0/418807383 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3db806c980 0x7f3db806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:35.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.645+0000 7f3dd6000640 1 --2- 172.21.15.45:0/418807383 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3db806c980 0x7f3db806ee40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3dd81e7a90 tx=0x7f3dc4009040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:35.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.647+0000 7f3dcea00640 1 -- 172.21.15.45:0/418807383 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3dc805a870 con 0x7f3dd80afda0 2024-04-16T16:19:35.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:35.900+0000 7f3ddd800640 1 -- 172.21.15.45:0/418807383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f3dd80d0440 con 0x7f3dd80afda0 2024-04-16T16:19:36.261 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:36.260+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:36.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.491+0000 7f3dcea00640 1 -- 172.21.15.45:0/418807383 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v75) ==== 104+0+0 (secure 0 0 0) 0x7f3dc805a170 con 0x7f3dd80afda0 2024-04-16T16:19:36.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.495+0000 7f3ddd800640 1 -- 172.21.15.45:0/418807383 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3db806c980 msgr2=0x7f3db806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:36.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.495+0000 7f3ddd800640 1 --2- 172.21.15.45:0/418807383 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3db806c980 0x7f3db806ee40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3dd81e7a90 tx=0x7f3dc4009040 comp rx=0 tx=0).stop 2024-04-16T16:19:36.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.496+0000 7f3ddd800640 1 -- 172.21.15.45:0/418807383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd80afda0 msgr2=0x7f3dd815d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:36.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.496+0000 7f3ddd800640 1 --2- 172.21.15.45:0/418807383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd80afda0 0x7f3dd815d230 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8007e40 tx=0x7f3dc800e4b0 comp rx=0 tx=0).stop 2024-04-16T16:19:36.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.496+0000 7f3ddd800640 1 -- 172.21.15.45:0/418807383 shutdown_connections 2024-04-16T16:19:36.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.496+0000 7f3ddd800640 1 --2- 172.21.15.45:0/418807383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dd815b920 0x7f3dd814e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:36.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.496+0000 7f3ddd800640 1 --2- 172.21.15.45:0/418807383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dd80d0e80 0x7f3dd815e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:36.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.496+0000 7f3ddd800640 1 --2- 172.21.15.45:0/418807383 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3db806c980 0x7f3db806ee40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:36.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.496+0000 7f3ddd800640 1 --2- 172.21.15.45:0/418807383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd80afda0 0x7f3dd815d230 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:36.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.496+0000 7f3ddd800640 1 -- 172.21.15.45:0/418807383 >> 172.21.15.45:0/418807383 conn(0x7f3dd80b3970 msgr2=0x7f3dd80aeb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:36.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.496+0000 7f3ddd800640 1 -- 172.21.15.45:0/418807383 shutdown_connections 2024-04-16T16:19:36.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.496+0000 7f3ddd800640 1 -- 172.21.15.45:0/418807383 wait complete. 2024-04-16T16:19:36.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-16T16:19:36.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.743+0000 7f726f800640 1 Processor -- start 2024-04-16T16:19:36.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.744+0000 7f726f800640 1 -- start start 2024-04-16T16:19:36.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.745+0000 7f726f800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72680cfe00 0x7f72680d2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:36.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.745+0000 7f726f800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f72680d1f10 0x7f72680d2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:36.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.745+0000 7f726f800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72680d4640 con 0x7f72681622d0 2024-04-16T16:19:36.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.745+0000 7f726f800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f72680d43f0 con 0x7f72680d1f10 2024-04-16T16:19:36.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.745+0000 7f726f800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f72680ba4c0 con 0x7f72680cfe00 2024-04-16T16:19:36.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.746+0000 7f726ce00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f72681622d0 0x7f72680cf6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57500/0 (socket says 172.21.15.45:57500) 2024-04-16T16:19:36.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.746+0000 7f726ce00640 1 -- 172.21.15.45:0/390950116 learned_addr learned my addr 172.21.15.45:0/390950116 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:36.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.746+0000 7f726d800640 1 --2- 172.21.15.45:0/390950116 >> v2:172.21.15.103:3300/0 conn(0x7f72680d1f10 0x7f72680d2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:36.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.746+0000 7f7267e00640 1 --2- 172.21.15.45:0/390950116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72680cfe00 0x7f72680d2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:36.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.746+0000 7f7267400640 1 -- 172.21.15.45:0/390950116 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1589142089 0 0) 0x7f72680d4640 con 0x7f72681622d0 2024-04-16T16:19:36.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.746+0000 7f7267400640 1 -- 172.21.15.45:0/390950116 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7250003520 con 0x7f72681622d0 2024-04-16T16:19:36.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.747+0000 7f7267400640 1 -- 172.21.15.45:0/390950116 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 95905593 0 0) 0x7f7250003520 con 0x7f72681622d0 2024-04-16T16:19:36.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.747+0000 7f7267400640 1 -- 172.21.15.45:0/390950116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72680cfe00 msgr2=0x7f72680d2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:36.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.747+0000 7f7267400640 1 --2- 172.21.15.45:0/390950116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72680cfe00 0x7f72680d2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:36.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.747+0000 7f7267400640 1 -- 172.21.15.45:0/390950116 >> v2:172.21.15.103:3300/0 conn(0x7f72680d1f10 msgr2=0x7f72680d2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:36.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.747+0000 7f7267400640 1 --2- 172.21.15.45:0/390950116 >> v2:172.21.15.103:3300/0 conn(0x7f72680d1f10 0x7f72680d2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:36.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.747+0000 7f7267400640 1 -- 172.21.15.45:0/390950116 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72680b7b50 con 0x7f72681622d0 2024-04-16T16:19:36.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.747+0000 7f7267400640 1 -- 172.21.15.45:0/390950116 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f725c001670 con 0x7f72681622d0 2024-04-16T16:19:36.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.747+0000 7f7267400640 1 -- 172.21.15.45:0/390950116 >> v1:172.21.15.45:6789/0 conn(0x7f72681622d0 legacy=0x7f72680cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:36.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.747+0000 7f7267400640 1 --2- 172.21.15.45:0/390950116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72500039b0 0x7f7250003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:36.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.747+0000 7f7267400640 1 --2- 172.21.15.45:0/390950116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72680cfe00 0x7f7250004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:36.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.747+0000 7f7267400640 1 --2- 172.21.15.45:0/390950116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7250004d40 0x7f72500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:36.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.747+0000 7f7267400640 1 -- 172.21.15.45:0/390950116 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f72680d43f0 con 0x7f7250004d40 2024-04-16T16:19:36.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.747+0000 7f7267400640 1 -- 172.21.15.45:0/390950116 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72680ba4c0 con 0x7f72500039b0 2024-04-16T16:19:36.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.747+0000 7f7267400640 1 -- 172.21.15.45:0/390950116 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f725000b9c0 con 0x7f72680cfe00 2024-04-16T16:19:36.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.747+0000 7f726d800640 1 --2- 172.21.15.45:0/390950116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7250004d40 0x7f72500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:36.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.748+0000 7f726ce00640 1 --2- 172.21.15.45:0/390950116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72500039b0 0x7f7250003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:36.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.748+0000 7f7267e00640 1 --2- 172.21.15.45:0/390950116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72680cfe00 0x7f7250004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:36.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.748+0000 7f726d800640 1 -- 172.21.15.45:0/390950116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72680cfe00 msgr2=0x7f7250004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:36.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.748+0000 7f726d800640 1 --2- 172.21.15.45:0/390950116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72680cfe00 0x7f7250004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:36.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.748+0000 7f726d800640 1 -- 172.21.15.45:0/390950116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72500039b0 msgr2=0x7f7250003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:36.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.748+0000 7f726d800640 1 --2- 172.21.15.45:0/390950116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72500039b0 0x7f7250003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:36.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.748+0000 7f726d800640 1 -- 172.21.15.45:0/390950116 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7250003520 con 0x7f7250004d40 2024-04-16T16:19:36.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.749+0000 7f726d800640 1 --2- 172.21.15.45:0/390950116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7250004d40 0x7f72500051c0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f7260009fc0 tx=0x7f7250005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84fa3a86fc4a53f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:36.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.749+0000 7f7267400640 1 -- 172.21.15.45:0/390950116 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260039070 con 0x7f7250004d40 2024-04-16T16:19:36.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.749+0000 7f7267400640 1 -- 172.21.15.45:0/390950116 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f726002cc60 con 0x7f7250004d40 2024-04-16T16:19:36.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.749+0000 7f7267400640 1 -- 172.21.15.45:0/390950116 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260009c10 con 0x7f7250004d40 2024-04-16T16:19:36.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.750+0000 7f726f800640 1 -- 172.21.15.45:0/390950116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7250004d40 msgr2=0x7f72500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:36.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.750+0000 7f726f800640 1 --2- 172.21.15.45:0/390950116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7250004d40 0x7f72500051c0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f7260009fc0 tx=0x7f7250005830 comp rx=0 tx=0).stop 2024-04-16T16:19:36.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.750+0000 7f726d800640 1 -- 172.21.15.45:0/390950116 reap_dead start 2024-04-16T16:19:36.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.750+0000 7f726f800640 1 -- 172.21.15.45:0/390950116 shutdown_connections 2024-04-16T16:19:36.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.750+0000 7f726f800640 1 -- 172.21.15.45:0/390950116 >> 172.21.15.45:0/390950116 conn(0x7f726806d400 msgr2=0x7f726806d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:36.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.750+0000 7f726f800640 1 -- 172.21.15.45:0/390950116 shutdown_connections 2024-04-16T16:19:36.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.750+0000 7f726f800640 1 -- 172.21.15.45:0/390950116 wait complete. 2024-04-16T16:19:36.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.752+0000 7f726f800640 1 Processor -- start 2024-04-16T16:19:36.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.752+0000 7f726f800640 1 -- start start 2024-04-16T16:19:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.753+0000 7f726f800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7250004d40 0x7f72680b2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.753+0000 7f726f800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72680cbc40 0x7f72680cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.753+0000 7f726f800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72681ea260 0x7f72681ea6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.753+0000 7f726f800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f72680b70e0 con 0x7f72681ea260 2024-04-16T16:19:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.753+0000 7f726f800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72680b6f60 con 0x7f7250004d40 2024-04-16T16:19:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.753+0000 7f726f800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f72680d2680 con 0x7f72680cbc40 2024-04-16T16:19:36.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.753+0000 7f726d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72681ea260 0x7f72681ea6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:36.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.754+0000 7f726d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72681ea260 0x7f72681ea6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33300/0 (socket says 172.21.15.45:33300) 2024-04-16T16:19:36.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.754+0000 7f726d800640 1 -- 172.21.15.45:0/60743256 learned_addr learned my addr 172.21.15.45:0/60743256 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:36.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.754+0000 7f726ce00640 1 --2- 172.21.15.45:0/60743256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7250004d40 0x7f72680b2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:36.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.754+0000 7f7267e00640 1 --2- 172.21.15.45:0/60743256 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72680cbc40 0x7f72680cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:36.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.754+0000 7f726d800640 1 -- 172.21.15.45:0/60743256 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72680cbc40 msgr2=0x7f72680cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:36.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.754+0000 7f726d800640 1 --2- 172.21.15.45:0/60743256 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72680cbc40 0x7f72680cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:36.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.754+0000 7f726d800640 1 -- 172.21.15.45:0/60743256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7250004d40 msgr2=0x7f72680b2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:36.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.754+0000 7f726d800640 1 --2- 172.21.15.45:0/60743256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7250004d40 0x7f72680b2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:36.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.754+0000 7f726d800640 1 -- 172.21.15.45:0/60743256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72681eb020 con 0x7f72681ea260 2024-04-16T16:19:36.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.754+0000 7f726d800640 1 --2- 172.21.15.45:0/60743256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72681ea260 0x7f72681ea6e0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f7260002230 tx=0x7f7260009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:36.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.755+0000 7f7264c00640 1 -- 172.21.15.45:0/60743256 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260043030 con 0x7f72681ea260 2024-04-16T16:19:36.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.755+0000 7f7264c00640 1 -- 172.21.15.45:0/60743256 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f726002ca60 con 0x7f72681ea260 2024-04-16T16:19:36.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.755+0000 7f7264c00640 1 -- 172.21.15.45:0/60743256 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260039040 con 0x7f72681ea260 2024-04-16T16:19:36.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.755+0000 7f726f800640 1 -- 172.21.15.45:0/60743256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72680c8ea0 con 0x7f72681ea260 2024-04-16T16:19:36.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.755+0000 7f726f800640 1 -- 172.21.15.45:0/60743256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72681eac20 con 0x7f72681ea260 2024-04-16T16:19:36.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.756+0000 7f726f800640 1 -- 172.21.15.45:0/60743256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7268163910 con 0x7f72681ea260 2024-04-16T16:19:36.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.757+0000 7f7264c00640 1 -- 172.21.15.45:0/60743256 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f7260046070 con 0x7f72681ea260 2024-04-16T16:19:36.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.758+0000 7f7264c00640 1 --2- 172.21.15.45:0/60743256 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f724806c980 0x7f724806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:36.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.758+0000 7f7264c00640 1 -- 172.21.15.45:0/60743256 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6483+0+0 (secure 0 0 0) 0x7f72600addd0 con 0x7f72681ea260 2024-04-16T16:19:36.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.759+0000 7f726ce00640 1 --2- 172.21.15.45:0/60743256 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f724806c980 0x7f724806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:36.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.760+0000 7f726ce00640 1 --2- 172.21.15.45:0/60743256 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f724806c980 0x7f724806ee40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f725c000c00 tx=0x7f725c009040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:36.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.763+0000 7f7264c00640 1 -- 172.21.15.45:0/60743256 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7268163910 con 0x7f72681ea260 2024-04-16T16:19:36.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:36.978+0000 7f726f800640 1 -- 172.21.15.45:0/60743256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7268075780 con 0x7f72681ea260 2024-04-16T16:19:37.392 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:37.391+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:37.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.511+0000 7f7264c00640 1 -- 172.21.15.45:0/60743256 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v104) ==== 181+0+0 (secure 0 0 0) 0x7f726004e020 con 0x7f72681ea260 2024-04-16T16:19:37.510 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2024-04-16T16:19:37.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.514+0000 7f726f800640 1 -- 172.21.15.45:0/60743256 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f724806c980 msgr2=0x7f724806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:37.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.514+0000 7f726f800640 1 --2- 172.21.15.45:0/60743256 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f724806c980 0x7f724806ee40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f725c000c00 tx=0x7f725c009040 comp rx=0 tx=0).stop 2024-04-16T16:19:37.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.514+0000 7f726f800640 1 -- 172.21.15.45:0/60743256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72681ea260 msgr2=0x7f72681ea6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:37.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.514+0000 7f726f800640 1 --2- 172.21.15.45:0/60743256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72681ea260 0x7f72681ea6e0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f7260002230 tx=0x7f7260009750 comp rx=0 tx=0).stop 2024-04-16T16:19:37.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.515+0000 7f726f800640 1 -- 172.21.15.45:0/60743256 shutdown_connections 2024-04-16T16:19:37.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.515+0000 7f726f800640 1 --2- 172.21.15.45:0/60743256 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f724806c980 0x7f724806ee40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:37.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.515+0000 7f726f800640 1 --2- 172.21.15.45:0/60743256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72681ea260 0x7f72681ea6e0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:37.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.515+0000 7f726f800640 1 --2- 172.21.15.45:0/60743256 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72680cbc40 0x7f72680cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:37.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.515+0000 7f726f800640 1 --2- 172.21.15.45:0/60743256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7250004d40 0x7f72680b2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:37.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.515+0000 7f726f800640 1 -- 172.21.15.45:0/60743256 >> 172.21.15.45:0/60743256 conn(0x7f726806d400 msgr2=0x7f72680b64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:37.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.515+0000 7f726f800640 1 -- 172.21.15.45:0/60743256 shutdown_connections 2024-04-16T16:19:37.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.515+0000 7f726f800640 1 -- 172.21.15.45:0/60743256 wait complete. 2024-04-16T16:19:37.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-16T16:19:37.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.765+0000 7fd2ee200640 1 Processor -- start 2024-04-16T16:19:37.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.765+0000 7fd2ee200640 1 -- start start 2024-04-16T16:19:37.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.767+0000 7fd2ee200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd2e815ecc0 0x7fd2e806fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:37.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.767+0000 7fd2ee200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd2e809af60 0x7fd2e806b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:37.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.767+0000 7fd2ee200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2e806f490 con 0x7fd2e806be70 2024-04-16T16:19:37.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.767+0000 7fd2ee200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd2e80b3f60 con 0x7fd2e815ecc0 2024-04-16T16:19:37.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.767+0000 7fd2ee200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd2e806f730 con 0x7fd2e809af60 2024-04-16T16:19:37.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.768+0000 7fd2e7400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd2e815ecc0 0x7fd2e806fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:37.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.768+0000 7fd2e7400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd2e815ecc0 0x7fd2e806fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33308/0 (socket says 172.21.15.45:33308) 2024-04-16T16:19:37.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.768+0000 7fd2e7400640 1 -- 172.21.15.45:0/2842240627 learned_addr learned my addr 172.21.15.45:0/2842240627 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:37.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.768+0000 7fd2e6a00640 1 --2- 172.21.15.45:0/2842240627 >> v2:172.21.15.103:3300/0 conn(0x7fd2e809af60 0x7fd2e806b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.768+0000 7fd2e7400640 1 -- 172.21.15.45:0/2842240627 >> v2:172.21.15.103:3300/0 conn(0x7fd2e809af60 msgr2=0x7fd2e806b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.768+0000 7fd2e7400640 1 --2- 172.21.15.45:0/2842240627 >> v2:172.21.15.103:3300/0 conn(0x7fd2e809af60 0x7fd2e806b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.768+0000 7fd2e7400640 1 -- 172.21.15.45:0/2842240627 >> v1:172.21.15.103:6789/0 conn(0x7fd2e806be70 legacy=0x7fd2e80ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.768+0000 7fd2e7400640 1 -- 172.21.15.45:0/2842240627 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e80b3bf0 con 0x7fd2e815ecc0 2024-04-16T16:19:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.768+0000 7fd2e6a00640 1 --2- 172.21.15.45:0/2842240627 >> v2:172.21.15.103:3300/0 conn(0x7fd2e809af60 0x7fd2e806b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.769+0000 7fd2e7400640 1 --2- 172.21.15.45:0/2842240627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2e815ecc0 0x7fd2e806fce0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fd2e806c7f0 tx=0x7fd2d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba0dc534d3ec8a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.770+0000 7fd2e6000640 1 -- 172.21.15.45:0/2842240627 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8035020 con 0x7fd2e815ecc0 2024-04-16T16:19:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.770+0000 7fd2e6000640 1 -- 172.21.15.45:0/2842240627 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2d8031870 con 0x7fd2e815ecc0 2024-04-16T16:19:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.770+0000 7fd2e6000640 1 -- 172.21.15.45:0/2842240627 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8031c40 con 0x7fd2e815ecc0 2024-04-16T16:19:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.770+0000 7fd2ee200640 1 -- 172.21.15.45:0/2842240627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2e815ecc0 msgr2=0x7fd2e806fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.770+0000 7fd2ee200640 1 --2- 172.21.15.45:0/2842240627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2e815ecc0 0x7fd2e806fce0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fd2e806c7f0 tx=0x7fd2d802b910 comp rx=0 tx=0).stop 2024-04-16T16:19:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.770+0000 7fd2ee200640 1 -- 172.21.15.45:0/2842240627 shutdown_connections 2024-04-16T16:19:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.770+0000 7fd2ee200640 1 --2- 172.21.15.45:0/2842240627 >> v2:172.21.15.103:3300/0 conn(0x7fd2e809af60 0x7fd2e806b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.771+0000 7fd2ee200640 1 --2- 172.21.15.45:0/2842240627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2e815ecc0 0x7fd2e806fce0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.771+0000 7fd2ee200640 1 -- 172.21.15.45:0/2842240627 >> 172.21.15.45:0/2842240627 conn(0x7fd2e809aa70 msgr2=0x7fd2e80b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.771+0000 7fd2ee200640 1 -- 172.21.15.45:0/2842240627 shutdown_connections 2024-04-16T16:19:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.771+0000 7fd2ee200640 1 -- 172.21.15.45:0/2842240627 wait complete. 2024-04-16T16:19:37.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.773+0000 7fd2ee200640 1 Processor -- start 2024-04-16T16:19:37.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.773+0000 7fd2ee200640 1 -- start start 2024-04-16T16:19:37.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.775+0000 7fd2ee200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2e806be70 0x7fd2e8155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.775+0000 7fd2ee200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2e809af60 0x7fd2e81610b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.775+0000 7fd2ee200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2e815ecc0 0x7fd2e815ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.775+0000 7fd2ee200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e80703a0 con 0x7fd2e809af60 2024-04-16T16:19:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.775+0000 7fd2ee200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e8070220 con 0x7fd2e806be70 2024-04-16T16:19:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.775+0000 7fd2ee200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd2e80d0590 con 0x7fd2e815ecc0 2024-04-16T16:19:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.775+0000 7fd2e6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2e809af60 0x7fd2e81610b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.775+0000 7fd2e6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2e809af60 0x7fd2e81610b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33324/0 (socket says 172.21.15.45:33324) 2024-04-16T16:19:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.775+0000 7fd2e6a00640 1 -- 172.21.15.45:0/3251751644 learned_addr learned my addr 172.21.15.45:0/3251751644 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.775+0000 7fd2e7400640 1 --2- 172.21.15.45:0/3251751644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2e806be70 0x7fd2e8155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.776+0000 7fd2e7e00640 1 --2- 172.21.15.45:0/3251751644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2e815ecc0 0x7fd2e815ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.776+0000 7fd2e6a00640 1 -- 172.21.15.45:0/3251751644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2e815ecc0 msgr2=0x7fd2e815ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.776+0000 7fd2e6a00640 1 --2- 172.21.15.45:0/3251751644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2e815ecc0 0x7fd2e815ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.776+0000 7fd2e6a00640 1 -- 172.21.15.45:0/3251751644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2e806be70 msgr2=0x7fd2e8155440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.776+0000 7fd2e6a00640 1 --2- 172.21.15.45:0/3251751644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2e806be70 0x7fd2e8155440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:37.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.776+0000 7fd2e6a00640 1 -- 172.21.15.45:0/3251751644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e81602f0 con 0x7fd2e809af60 2024-04-16T16:19:37.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.776+0000 7fd2e6a00640 1 --2- 172.21.15.45:0/3251751644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2e809af60 0x7fd2e81610b0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400c7f0 tx=0x7fd2d400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:37.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.776+0000 7fd2df400640 1 -- 172.21.15.45:0/3251751644 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4015070 con 0x7fd2e809af60 2024-04-16T16:19:37.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.776+0000 7fd2df400640 1 -- 172.21.15.45:0/3251751644 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2d4007a30 con 0x7fd2e809af60 2024-04-16T16:19:37.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.776+0000 7fd2df400640 1 -- 172.21.15.45:0/3251751644 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4007e00 con 0x7fd2e809af60 2024-04-16T16:19:37.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.776+0000 7fd2ee200640 1 -- 172.21.15.45:0/3251751644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2e81605d0 con 0x7fd2e809af60 2024-04-16T16:19:37.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.777+0000 7fd2ee200640 1 -- 172.21.15.45:0/3251751644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2e815ffa0 con 0x7fd2e809af60 2024-04-16T16:19:37.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.778+0000 7fd2ee200640 1 -- 172.21.15.45:0/3251751644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2e8075780 con 0x7fd2e809af60 2024-04-16T16:19:37.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.779+0000 7fd2df400640 1 -- 172.21.15.45:0/3251751644 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7fd2d4018070 con 0x7fd2e809af60 2024-04-16T16:19:37.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.780+0000 7fd2df400640 1 --2- 172.21.15.45:0/3251751644 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fd2c806c980 0x7fd2c806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:37.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.780+0000 7fd2df400640 1 -- 172.21.15.45:0/3251751644 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 6045+0+0 (secure 0 0 0) 0x7fd2d408be50 con 0x7fd2e809af60 2024-04-16T16:19:37.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.781+0000 7fd2e7400640 1 --2- 172.21.15.45:0/3251751644 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fd2c806c980 0x7fd2c806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:37.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.782+0000 7fd2e7400640 1 --2- 172.21.15.45:0/3251751644 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fd2c806c980 0x7fd2c806ee40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8036000 tx=0x7fd2d803c040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:37.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.784+0000 7fd2df400640 1 -- 172.21.15.45:0/3251751644 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd2d4059fc0 con 0x7fd2e809af60 2024-04-16T16:19:37.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:37.997+0000 7fd2ee200640 1 -- 172.21.15.45:0/3251751644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd2e80cd760 con 0x7fd2e809af60 2024-04-16T16:19:38.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.514+0000 7fd2df400640 1 -- 172.21.15.45:0/3251751644 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v105) ==== 169+0+0 (secure 0 0 0) 0x7fd2d40598c0 con 0x7fd2e809af60 2024-04-16T16:19:38.516 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2024-04-16T16:19:38.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.519+0000 7fd2ee200640 1 -- 172.21.15.45:0/3251751644 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fd2c806c980 msgr2=0x7fd2c806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:38.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.519+0000 7fd2ee200640 1 --2- 172.21.15.45:0/3251751644 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fd2c806c980 0x7fd2c806ee40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8036000 tx=0x7fd2d803c040 comp rx=0 tx=0).stop 2024-04-16T16:19:38.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.519+0000 7fd2ee200640 1 -- 172.21.15.45:0/3251751644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2e809af60 msgr2=0x7fd2e81610b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:38.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.519+0000 7fd2ee200640 1 --2- 172.21.15.45:0/3251751644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2e809af60 0x7fd2e81610b0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400c7f0 tx=0x7fd2d400cbf0 comp rx=0 tx=0).stop 2024-04-16T16:19:38.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.519+0000 7fd2ee200640 1 -- 172.21.15.45:0/3251751644 shutdown_connections 2024-04-16T16:19:38.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.519+0000 7fd2ee200640 1 --2- 172.21.15.45:0/3251751644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2e815ecc0 0x7fd2e815ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:38.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.519+0000 7fd2ee200640 1 --2- 172.21.15.45:0/3251751644 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fd2c806c980 0x7fd2c806ee40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:38.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.519+0000 7fd2ee200640 1 --2- 172.21.15.45:0/3251751644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2e809af60 0x7fd2e81610b0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:38.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.519+0000 7fd2ee200640 1 --2- 172.21.15.45:0/3251751644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2e806be70 0x7fd2e8155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:38.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.519+0000 7fd2ee200640 1 -- 172.21.15.45:0/3251751644 >> 172.21.15.45:0/3251751644 conn(0x7fd2e809aa70 msgr2=0x7fd2e80ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:38.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.520+0000 7fd2ee200640 1 -- 172.21.15.45:0/3251751644 shutdown_connections 2024-04-16T16:19:38.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.521+0000 7fd2ee200640 1 -- 172.21.15.45:0/3251751644 wait complete. 2024-04-16T16:19:38.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-16T16:19:38.555 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:38.553+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:38.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.764+0000 7f64b0e00640 1 Processor -- start 2024-04-16T16:19:38.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.764+0000 7f64b0e00640 1 -- start start 2024-04-16T16:19:38.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.766+0000 7f64b0e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f64ac1460f0 0x7f64ac06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:38.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.766+0000 7f64b0e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f64ac072630 0x7f64ac15d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:38.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.766+0000 7f64b0e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64ac0cece0 con 0x7f64ac0ce520 2024-04-16T16:19:38.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.766+0000 7f64b0e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f64ac1464d0 con 0x7f64ac1460f0 2024-04-16T16:19:38.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.766+0000 7f64b0e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f64ac0cefc0 con 0x7f64ac072630 2024-04-16T16:19:38.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.767+0000 7f64ab400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f64ac1460f0 0x7f64ac06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:38.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.767+0000 7f64aaa00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f64ac0ce520 0x7f64ac071f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57516/0 (socket says 172.21.15.45:57516) 2024-04-16T16:19:38.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.767+0000 7f64aaa00640 1 -- 172.21.15.45:0/2969274698 learned_addr learned my addr 172.21.15.45:0/2969274698 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:38.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.767+0000 7f64abe00640 1 --2- 172.21.15.45:0/2969274698 >> v2:172.21.15.103:3300/0 conn(0x7f64ac072630 0x7f64ac15d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:38.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.767+0000 7f64ab400640 1 -- 172.21.15.45:0/2969274698 >> v2:172.21.15.103:3300/0 conn(0x7f64ac072630 msgr2=0x7f64ac15d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:38.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.767+0000 7f64ab400640 1 --2- 172.21.15.45:0/2969274698 >> v2:172.21.15.103:3300/0 conn(0x7f64ac072630 0x7f64ac15d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:38.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.767+0000 7f64ab400640 1 -- 172.21.15.45:0/2969274698 >> v1:172.21.15.45:6789/0 conn(0x7f64ac0ce520 legacy=0x7f64ac071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:38.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.767+0000 7f64ab400640 1 -- 172.21.15.45:0/2969274698 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64ac15cf20 con 0x7f64ac1460f0 2024-04-16T16:19:38.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.767+0000 7f64aa000640 1 -- 172.21.15.45:0/2969274698 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3592021369 0 0) 0x7f64ac0cece0 con 0x7f64ac0ce520 2024-04-16T16:19:38.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.768+0000 7f64ab400640 1 --2- 172.21.15.45:0/2969274698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ac1460f0 0x7f64ac06ed60 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f64ac0a94d0 tx=0x7f649c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ff70bf3e8a5516 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:38.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.769+0000 7f64aa000640 1 -- 172.21.15.45:0/2969274698 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c035020 con 0x7f64ac1460f0 2024-04-16T16:19:38.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.769+0000 7f64aa000640 1 -- 172.21.15.45:0/2969274698 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f649c031830 con 0x7f64ac1460f0 2024-04-16T16:19:38.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.769+0000 7f64aa000640 1 -- 172.21.15.45:0/2969274698 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c031c00 con 0x7f64ac1460f0 2024-04-16T16:19:38.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.769+0000 7f64b0e00640 1 -- 172.21.15.45:0/2969274698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ac1460f0 msgr2=0x7f64ac06ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:38.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.769+0000 7f64b0e00640 1 --2- 172.21.15.45:0/2969274698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ac1460f0 0x7f64ac06ed60 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f64ac0a94d0 tx=0x7f649c02b8d0 comp rx=0 tx=0).stop 2024-04-16T16:19:38.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.769+0000 7f64b0e00640 1 -- 172.21.15.45:0/2969274698 shutdown_connections 2024-04-16T16:19:38.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.769+0000 7f64b0e00640 1 --2- 172.21.15.45:0/2969274698 >> v2:172.21.15.103:3300/0 conn(0x7f64ac072630 0x7f64ac15d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:38.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.769+0000 7f64b0e00640 1 --2- 172.21.15.45:0/2969274698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ac1460f0 0x7f64ac06ed60 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:38.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.769+0000 7f64b0e00640 1 -- 172.21.15.45:0/2969274698 >> 172.21.15.45:0/2969274698 conn(0x7f64ac067e00 msgr2=0x7f64ac06b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:38.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.770+0000 7f64b0e00640 1 -- 172.21.15.45:0/2969274698 shutdown_connections 2024-04-16T16:19:38.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.770+0000 7f64b0e00640 1 -- 172.21.15.45:0/2969274698 wait complete. 2024-04-16T16:19:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.771+0000 7f64b0e00640 1 Processor -- start 2024-04-16T16:19:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.771+0000 7f64b0e00640 1 -- start start 2024-04-16T16:19:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.772+0000 7f64b0e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ac072630 0x7f64ac0aa410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.772+0000 7f64b0e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f64ac0ce520 0x7f64ac15c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.772+0000 7f64b0e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64ac1460f0 0x7f64ac15e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.772+0000 7f64b0e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f64ac0ce990 con 0x7f64ac072630 2024-04-16T16:19:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.772+0000 7f64b0e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f64ac15cc20 con 0x7f64ac1460f0 2024-04-16T16:19:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.772+0000 7f64b0e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f64ac15dc60 con 0x7f64ac0ce520 2024-04-16T16:19:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.773+0000 7f64ab400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ac072630 0x7f64ac0aa410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:38.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.773+0000 7f64ab400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ac072630 0x7f64ac0aa410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33336/0 (socket says 172.21.15.45:33336) 2024-04-16T16:19:38.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.773+0000 7f64ab400640 1 -- 172.21.15.45:0/2768912756 learned_addr learned my addr 172.21.15.45:0/2768912756 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:38.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.773+0000 7f64ab400640 1 -- 172.21.15.45:0/2768912756 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f64ac0ce520 msgr2=0x7f64ac15c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.773+0000 7f64ab400640 1 --2- 172.21.15.45:0/2768912756 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f64ac0ce520 0x7f64ac15c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.773+0000 7f64ab400640 1 -- 172.21.15.45:0/2768912756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64ac1460f0 msgr2=0x7f64ac15e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.773+0000 7f64ab400640 1 --2- 172.21.15.45:0/2768912756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64ac1460f0 0x7f64ac15e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.773+0000 7f64ab400640 1 -- 172.21.15.45:0/2768912756 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64ac0a9650 con 0x7f64ac072630 2024-04-16T16:19:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.774+0000 7f64ab400640 1 --2- 172.21.15.45:0/2768912756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ac072630 0x7f64ac0aa410 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f649c009890 tx=0x7f649c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.774+0000 7f64a3400640 1 -- 172.21.15.45:0/2768912756 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c040070 con 0x7f64ac072630 2024-04-16T16:19:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.774+0000 7f64a3400640 1 -- 172.21.15.45:0/2768912756 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f649c03e070 con 0x7f64ac072630 2024-04-16T16:19:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.774+0000 7f64b0e00640 1 -- 172.21.15.45:0/2768912756 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64ac0a9930 con 0x7f64ac072630 2024-04-16T16:19:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.774+0000 7f64a3400640 1 -- 172.21.15.45:0/2768912756 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c035040 con 0x7f64ac072630 2024-04-16T16:19:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.774+0000 7f64b0e00640 1 -- 172.21.15.45:0/2768912756 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64ac0a9ef0 con 0x7f64ac072630 2024-04-16T16:19:38.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.775+0000 7f64b0e00640 1 -- 172.21.15.45:0/2768912756 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64ac075780 con 0x7f64ac072630 2024-04-16T16:19:38.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.777+0000 7f64a3400640 1 -- 172.21.15.45:0/2768912756 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f649c038020 con 0x7f64ac072630 2024-04-16T16:19:38.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.777+0000 7f64a3400640 1 --2- 172.21.15.45:0/2768912756 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f648c06c980 0x7f648c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:38.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.778+0000 7f64a3400640 1 -- 172.21.15.45:0/2768912756 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5631+0+0 (secure 0 0 0) 0x7f649c0ad5a0 con 0x7f64ac072630 2024-04-16T16:19:38.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.778+0000 7f64aaa00640 1 --2- 172.21.15.45:0/2768912756 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f648c06c980 0x7f648c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:38.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.779+0000 7f64aaa00640 1 --2- 172.21.15.45:0/2768912756 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f648c06c980 0x7f648c06ee40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f64980010e0 tx=0x7f64980077c0 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:38.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:38.782+0000 7f64a3400640 1 -- 172.21.15.45:0/2768912756 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f649c07b9c0 con 0x7f64ac072630 2024-04-16T16:19:39.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.003+0000 7f64b0e00640 1 -- 172.21.15.45:0/2768912756 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f64ac15f910 con 0x7f64ac072630 2024-04-16T16:19:39.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.008+0000 7f64a3400640 1 -- 172.21.15.45:0/2768912756 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) ==== 104+0+0 (secure 0 0 0) 0x7f649c07b2c0 con 0x7f64ac072630 2024-04-16T16:19:39.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.010+0000 7f64b0e00640 1 -- 172.21.15.45:0/2768912756 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f648c06c980 msgr2=0x7f648c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:39.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.010+0000 7f64b0e00640 1 --2- 172.21.15.45:0/2768912756 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f648c06c980 0x7f648c06ee40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f64980010e0 tx=0x7f64980077c0 comp rx=0 tx=0).stop 2024-04-16T16:19:39.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.010+0000 7f64b0e00640 1 -- 172.21.15.45:0/2768912756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ac072630 msgr2=0x7f64ac0aa410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:39.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.010+0000 7f64b0e00640 1 --2- 172.21.15.45:0/2768912756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ac072630 0x7f64ac0aa410 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f649c009890 tx=0x7f649c0046c0 comp rx=0 tx=0).stop 2024-04-16T16:19:39.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.010+0000 7f64b0e00640 1 -- 172.21.15.45:0/2768912756 shutdown_connections 2024-04-16T16:19:39.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.010+0000 7f64b0e00640 1 --2- 172.21.15.45:0/2768912756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64ac1460f0 0x7f64ac15e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:39.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.010+0000 7f64b0e00640 1 --2- 172.21.15.45:0/2768912756 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f64ac0ce520 0x7f64ac15c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:39.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.010+0000 7f64b0e00640 1 --2- 172.21.15.45:0/2768912756 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f648c06c980 0x7f648c06ee40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:39.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.010+0000 7f64b0e00640 1 --2- 172.21.15.45:0/2768912756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ac072630 0x7f64ac0aa410 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:39.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.010+0000 7f64b0e00640 1 -- 172.21.15.45:0/2768912756 >> 172.21.15.45:0/2768912756 conn(0x7f64ac067e00 msgr2=0x7f64ac071f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:39.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.011+0000 7f64b0e00640 1 -- 172.21.15.45:0/2768912756 shutdown_connections 2024-04-16T16:19:39.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.011+0000 7f64b0e00640 1 -- 172.21.15.45:0/2768912756 wait complete. 2024-04-16T16:19:39.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_client_ip' 2024-04-16T16:19:39.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.242+0000 7f206e400640 1 Processor -- start 2024-04-16T16:19:39.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.242+0000 7f206e400640 1 -- start start 2024-04-16T16:19:39.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.244+0000 7f206e400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f206815b730 0x7f20680cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:39.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.244+0000 7f206e400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f20680b5840 0x7f20680cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:39.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.244+0000 7f206e400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20680ce5b0 con 0x7f20680d0060 2024-04-16T16:19:39.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.244+0000 7f206e400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f20680b5410 con 0x7f20680b5840 2024-04-16T16:19:39.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.244+0000 7f206e400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2068072420 con 0x7f206815b730 2024-04-16T16:19:39.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.244+0000 7f2067e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f20680d0060 0x7f206815f360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57520/0 (socket says 172.21.15.45:57520) 2024-04-16T16:19:39.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.244+0000 7f2067e00640 1 -- 172.21.15.45:0/400489837 learned_addr learned my addr 172.21.15.45:0/400489837 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:39.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2067400640 1 --2- 172.21.15.45:0/400489837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f206815b730 0x7f20680cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:39.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066a00640 1 --2- 172.21.15.45:0/400489837 >> v2:172.21.15.103:3300/0 conn(0x7f20680b5840 0x7f20680cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:39.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 -- 172.21.15.45:0/400489837 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4156117171 0 0) 0x7f20680ce5b0 con 0x7f20680d0060 2024-04-16T16:19:39.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 -- 172.21.15.45:0/400489837 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f204c003520 con 0x7f20680d0060 2024-04-16T16:19:39.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 -- 172.21.15.45:0/400489837 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4063417185 0 0) 0x7f204c003520 con 0x7f20680d0060 2024-04-16T16:19:39.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 -- 172.21.15.45:0/400489837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f206815b730 msgr2=0x7f20680cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:39.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 --2- 172.21.15.45:0/400489837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f206815b730 0x7f20680cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:39.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 -- 172.21.15.45:0/400489837 >> v2:172.21.15.103:3300/0 conn(0x7f20680b5840 msgr2=0x7f20680cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:39.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 --2- 172.21.15.45:0/400489837 >> v2:172.21.15.103:3300/0 conn(0x7f20680b5840 0x7f20680cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:39.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 -- 172.21.15.45:0/400489837 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2068072700 con 0x7f20680d0060 2024-04-16T16:19:39.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 -- 172.21.15.45:0/400489837 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f2060001670 con 0x7f20680d0060 2024-04-16T16:19:39.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 -- 172.21.15.45:0/400489837 >> v1:172.21.15.45:6789/0 conn(0x7f20680d0060 legacy=0x7f206815f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:39.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 --2- 172.21.15.45:0/400489837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f206815b730 0x7f204c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:39.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 --2- 172.21.15.45:0/400489837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f204c003f30 0x7f204c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:39.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 --2- 172.21.15.45:0/400489837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204c004d40 0x7f204c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:39.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 -- 172.21.15.45:0/400489837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f20680b5410 con 0x7f204c003f30 2024-04-16T16:19:39.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 -- 172.21.15.45:0/400489837 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2068072420 con 0x7f204c004d40 2024-04-16T16:19:39.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.245+0000 7f2066000640 1 -- 172.21.15.45:0/400489837 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f204c00b9c0 con 0x7f206815b730 2024-04-16T16:19:39.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.246+0000 7f2066a00640 1 --2- 172.21.15.45:0/400489837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f204c003f30 0x7f204c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:39.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.246+0000 7f2067e00640 1 --2- 172.21.15.45:0/400489837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204c004d40 0x7f204c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:39.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.246+0000 7f2066a00640 1 -- 172.21.15.45:0/400489837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f206815b730 msgr2=0x7f204c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:39.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.246+0000 7f2066a00640 1 --2- 172.21.15.45:0/400489837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f206815b730 0x7f204c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:39.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.246+0000 7f2066a00640 1 -- 172.21.15.45:0/400489837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204c004d40 msgr2=0x7f204c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:39.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.246+0000 7f2066a00640 1 --2- 172.21.15.45:0/400489837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204c004d40 0x7f204c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:39.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.246+0000 7f2066a00640 1 -- 172.21.15.45:0/400489837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f204c003520 con 0x7f204c003f30 2024-04-16T16:19:39.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.247+0000 7f2066a00640 1 --2- 172.21.15.45:0/400489837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f204c003f30 0x7f204c004800 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f2054009fc0 tx=0x7f204c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a281cf93690a20 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:39.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.247+0000 7f2066000640 1 -- 172.21.15.45:0/400489837 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2054039070 con 0x7f204c003f30 2024-04-16T16:19:39.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.247+0000 7f2066000640 1 -- 172.21.15.45:0/400489837 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f205402cc40 con 0x7f204c003f30 2024-04-16T16:19:39.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.247+0000 7f2066000640 1 -- 172.21.15.45:0/400489837 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2054009c10 con 0x7f204c003f30 2024-04-16T16:19:39.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.248+0000 7f206e400640 1 -- 172.21.15.45:0/400489837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f204c003f30 msgr2=0x7f204c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:39.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.248+0000 7f206e400640 1 --2- 172.21.15.45:0/400489837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f204c003f30 0x7f204c004800 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f2054009fc0 tx=0x7f204c005830 comp rx=0 tx=0).stop 2024-04-16T16:19:39.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.248+0000 7f2067e00640 1 -- 172.21.15.45:0/400489837 reap_dead start 2024-04-16T16:19:39.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.248+0000 7f206e400640 1 -- 172.21.15.45:0/400489837 shutdown_connections 2024-04-16T16:19:39.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.248+0000 7f206e400640 1 -- 172.21.15.45:0/400489837 >> 172.21.15.45:0/400489837 conn(0x7f20680b2750 msgr2=0x7f20680b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:39.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.248+0000 7f206e400640 1 -- 172.21.15.45:0/400489837 shutdown_connections 2024-04-16T16:19:39.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.248+0000 7f206e400640 1 -- 172.21.15.45:0/400489837 wait complete. 2024-04-16T16:19:39.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.250+0000 7f206e400640 1 Processor -- start 2024-04-16T16:19:39.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.251+0000 7f206e400640 1 -- start start 2024-04-16T16:19:39.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.252+0000 7f206e400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f204c003f30 0x7f2068148c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:39.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.252+0000 7f206e400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2068156750 0x7f206815d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:39.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.252+0000 7f206e400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2068156bd0 0x7f206815db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:39.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.252+0000 7f206e400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f20680d04f0 con 0x7f204c003f30 2024-04-16T16:19:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.252+0000 7f206e400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2068072cc0 con 0x7f2068156bd0 2024-04-16T16:19:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.252+0000 7f206e400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f20680b5ca0 con 0x7f2068156750 2024-04-16T16:19:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.252+0000 7f2067400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f204c003f30 0x7f2068148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.252+0000 7f2067400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f204c003f30 0x7f2068148c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33360/0 (socket says 172.21.15.45:33360) 2024-04-16T16:19:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.252+0000 7f2067400640 1 -- 172.21.15.45:0/3744667803 learned_addr learned my addr 172.21.15.45:0/3744667803 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.252+0000 7f2066a00640 1 --2- 172.21.15.45:0/3744667803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2068156750 0x7f206815d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.252+0000 7f2067e00640 1 --2- 172.21.15.45:0/3744667803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2068156bd0 0x7f206815db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.253+0000 7f2067400640 1 -- 172.21.15.45:0/3744667803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2068156750 msgr2=0x7f206815d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.253+0000 7f2067400640 1 --2- 172.21.15.45:0/3744667803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2068156750 0x7f206815d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:39.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.253+0000 7f2067400640 1 -- 172.21.15.45:0/3744667803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2068156bd0 msgr2=0x7f206815db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:39.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.253+0000 7f2067400640 1 --2- 172.21.15.45:0/3744667803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2068156bd0 0x7f206815db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:39.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.253+0000 7f2067400640 1 -- 172.21.15.45:0/3744667803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20681491d0 con 0x7f204c003f30 2024-04-16T16:19:39.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.253+0000 7f2067e00640 1 --2- 172.21.15.45:0/3744667803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2068156bd0 0x7f206815db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:19:39.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.253+0000 7f2067400640 1 --2- 172.21.15.45:0/3744667803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f204c003f30 0x7f2068148c90 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f205800e7c0 tx=0x7f205800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:39.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.254+0000 7f205f400640 1 -- 172.21.15.45:0/3744667803 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2058017070 con 0x7f204c003f30 2024-04-16T16:19:39.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.254+0000 7f205f400640 1 -- 172.21.15.45:0/3744667803 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f205800c940 con 0x7f204c003f30 2024-04-16T16:19:39.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.254+0000 7f205f400640 1 -- 172.21.15.45:0/3744667803 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205800cd10 con 0x7f204c003f30 2024-04-16T16:19:39.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.254+0000 7f206e400640 1 -- 172.21.15.45:0/3744667803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2068150530 con 0x7f204c003f30 2024-04-16T16:19:39.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.254+0000 7f206e400640 1 -- 172.21.15.45:0/3744667803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2068150af0 con 0x7f204c003f30 2024-04-16T16:19:39.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.255+0000 7f206e400640 1 -- 172.21.15.45:0/3744667803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2068075780 con 0x7f204c003f30 2024-04-16T16:19:39.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.256+0000 7f205f400640 1 -- 172.21.15.45:0/3744667803 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f205801a070 con 0x7f204c003f30 2024-04-16T16:19:39.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.257+0000 7f205f400640 1 --2- 172.21.15.45:0/3744667803 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f204806c980 0x7f204806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:39.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.257+0000 7f205f400640 1 -- 172.21.15.45:0/3744667803 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5631+0+0 (secure 0 0 0) 0x7f205808bbb0 con 0x7f204c003f30 2024-04-16T16:19:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.258+0000 7f2066a00640 1 --2- 172.21.15.45:0/3744667803 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f204806c980 0x7f204806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:39.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.259+0000 7f2066a00640 1 --2- 172.21.15.45:0/3744667803 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f204806c980 0x7f204806ee40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2054036000 tx=0x7f205402e000 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:39.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.262+0000 7f205f400640 1 -- 172.21.15.45:0/3744667803 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2058092080 con 0x7f204c003f30 2024-04-16T16:19:39.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:39.561+0000 7f206e400640 1 -- 172.21.15.45:0/3744667803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_client_ip"]} v 0) -- 0x7f20680b3d00 con 0x7f204c003f30 2024-04-16T16:19:39.686 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:39.685+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:40.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.540+0000 7f205f400640 1 -- 172.21.15.45:0/3744667803 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_client_ip"]}]=0 v362) ==== 150+0+0 (secure 0 0 0) 0x7f2058059a80 con 0x7f204c003f30 2024-04-16T16:19:40.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.544+0000 7f206e400640 1 -- 172.21.15.45:0/3744667803 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f204806c980 msgr2=0x7f204806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:40.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.544+0000 7f206e400640 1 --2- 172.21.15.45:0/3744667803 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f204806c980 0x7f204806ee40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2054036000 tx=0x7f205402e000 comp rx=0 tx=0).stop 2024-04-16T16:19:40.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.544+0000 7f206e400640 1 -- 172.21.15.45:0/3744667803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f204c003f30 msgr2=0x7f2068148c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:40.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.544+0000 7f206e400640 1 --2- 172.21.15.45:0/3744667803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f204c003f30 0x7f2068148c90 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f205800e7c0 tx=0x7f205800ec80 comp rx=0 tx=0).stop 2024-04-16T16:19:40.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.544+0000 7f206e400640 1 -- 172.21.15.45:0/3744667803 shutdown_connections 2024-04-16T16:19:40.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.544+0000 7f206e400640 1 --2- 172.21.15.45:0/3744667803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2068156bd0 0x7f206815db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:40.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.544+0000 7f206e400640 1 --2- 172.21.15.45:0/3744667803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2068156750 0x7f206815d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:40.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.544+0000 7f206e400640 1 --2- 172.21.15.45:0/3744667803 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f204806c980 0x7f204806ee40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:40.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.544+0000 7f206e400640 1 --2- 172.21.15.45:0/3744667803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f204c003f30 0x7f2068148c90 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:40.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.544+0000 7f206e400640 1 -- 172.21.15.45:0/3744667803 >> 172.21.15.45:0/3744667803 conn(0x7f20680b2750 msgr2=0x7f20680b1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:40.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.544+0000 7f206e400640 1 -- 172.21.15.45:0/3744667803 shutdown_connections 2024-04-16T16:19:40.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.544+0000 7f206e400640 1 -- 172.21.15.45:0/3744667803 wait complete. 2024-04-16T16:19:40.569 INFO:tasks.cephfs_test_runner:test_for_invalid_client_ip (tasks.cephfs.test_mds_metrics.TestMDSMetrics) ... ok 2024-04-16T16:19:40.569 INFO:tasks.cephfs_test_runner:Starting test: test_for_invalid_mds_rank (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2024-04-16T16:19:40.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:19:40.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.803+0000 7fc363200640 1 Processor -- start 2024-04-16T16:19:40.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.804+0000 7fc363200640 1 -- start start 2024-04-16T16:19:40.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.805+0000 7fc363200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc35c0c5d80 0x7fc35c166600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:40.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.805+0000 7fc363200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc35c15c700 con 0x7fc35c06dfb0 2024-04-16T16:19:40.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.805+0000 7fc363200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc35c15c9c0 con 0x7fc35c144ff0 2024-04-16T16:19:40.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.805+0000 7fc363200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc35c15d2f0 con 0x7fc35c0c5d80 2024-04-16T16:19:40.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.806+0000 7fc361400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc35c06dfb0 0x7fc35c0c5670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57534/0 (socket says 172.21.15.45:57534) 2024-04-16T16:19:40.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.806+0000 7fc361400640 1 -- 172.21.15.45:0/3168133712 learned_addr learned my addr 172.21.15.45:0/3168133712 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:40.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.806+0000 7fc362800640 1 --2- 172.21.15.45:0/3168133712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc35c0c5d80 0x7fc35c166600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:40.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.806+0000 7fc360a00640 1 -- 172.21.15.45:0/3168133712 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 983419180 0 0) 0x7fc35c15c700 con 0x7fc35c06dfb0 2024-04-16T16:19:40.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.806+0000 7fc360a00640 1 -- 172.21.15.45:0/3168133712 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc348003520 con 0x7fc35c06dfb0 2024-04-16T16:19:40.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.806+0000 7fc360a00640 1 -- 172.21.15.45:0/3168133712 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3202226834 0 0) 0x7fc348003520 con 0x7fc35c06dfb0 2024-04-16T16:19:40.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc360a00640 1 -- 172.21.15.45:0/3168133712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc35c0c5d80 msgr2=0x7fc35c166600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:40.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc360a00640 1 --2- 172.21.15.45:0/3168133712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc35c0c5d80 0x7fc35c166600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:40.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc360a00640 1 -- 172.21.15.45:0/3168133712 >> v1:172.21.15.103:6789/0 conn(0x7fc35c144ff0 legacy=0x7fc35c0cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:40.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc360a00640 1 -- 172.21.15.45:0/3168133712 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35c15cca0 con 0x7fc35c06dfb0 2024-04-16T16:19:40.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc360a00640 1 -- 172.21.15.45:0/3168133712 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fc34c001670 con 0x7fc35c06dfb0 2024-04-16T16:19:40.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc360a00640 1 -- 172.21.15.45:0/3168133712 >> v1:172.21.15.45:6789/0 conn(0x7fc35c06dfb0 legacy=0x7fc35c0c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:40.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc360a00640 1 --2- 172.21.15.45:0/3168133712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc35c0c5d80 0x7fc3480039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:40.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc360a00640 1 --2- 172.21.15.45:0/3168133712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc348003f60 0x7fc348004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:40.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc360a00640 1 --2- 172.21.15.45:0/3168133712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc348004d40 0x7fc3480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:40.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc360a00640 1 -- 172.21.15.45:0/3168133712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c15d2f0 con 0x7fc348003f60 2024-04-16T16:19:40.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc360a00640 1 -- 172.21.15.45:0/3168133712 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc34800baa0 con 0x7fc348004d40 2024-04-16T16:19:40.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc360a00640 1 -- 172.21.15.45:0/3168133712 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc34800bca0 con 0x7fc35c0c5d80 2024-04-16T16:19:40.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc361400640 1 --2- 172.21.15.45:0/3168133712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc348003f60 0x7fc348004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:40.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc361e00640 1 --2- 172.21.15.45:0/3168133712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc35c0c5d80 0x7fc3480039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:40.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc362800640 1 --2- 172.21.15.45:0/3168133712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc348004d40 0x7fc3480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:40.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc361400640 1 -- 172.21.15.45:0/3168133712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc35c0c5d80 msgr2=0x7fc3480039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:40.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.807+0000 7fc361400640 1 --2- 172.21.15.45:0/3168133712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc35c0c5d80 0x7fc3480039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:40.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.808+0000 7fc361400640 1 -- 172.21.15.45:0/3168133712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc348004d40 msgr2=0x7fc3480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:40.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.808+0000 7fc361400640 1 --2- 172.21.15.45:0/3168133712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc348004d40 0x7fc3480051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:40.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.808+0000 7fc361400640 1 -- 172.21.15.45:0/3168133712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc348003520 con 0x7fc348003f60 2024-04-16T16:19:40.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.809+0000 7fc361400640 1 --2- 172.21.15.45:0/3168133712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc348003f60 0x7fc348004800 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fc34c007cd0 tx=0x7fc348005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3761ecaf1e9df04 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:40.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.809+0000 7fc360a00640 1 -- 172.21.15.45:0/3168133712 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c007d10 con 0x7fc348003f60 2024-04-16T16:19:40.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.809+0000 7fc360a00640 1 -- 172.21.15.45:0/3168133712 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc34c032070 con 0x7fc348003f60 2024-04-16T16:19:40.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.809+0000 7fc360a00640 1 -- 172.21.15.45:0/3168133712 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c0058d0 con 0x7fc348003f60 2024-04-16T16:19:40.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.809+0000 7fc363200640 1 -- 172.21.15.45:0/3168133712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc348003f60 msgr2=0x7fc348004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:40.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.809+0000 7fc363200640 1 --2- 172.21.15.45:0/3168133712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc348003f60 0x7fc348004800 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fc34c007cd0 tx=0x7fc348005830 comp rx=0 tx=0).stop 2024-04-16T16:19:40.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.810+0000 7fc362800640 1 -- 172.21.15.45:0/3168133712 reap_dead start 2024-04-16T16:19:40.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.810+0000 7fc363200640 1 -- 172.21.15.45:0/3168133712 shutdown_connections 2024-04-16T16:19:40.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.810+0000 7fc363200640 1 -- 172.21.15.45:0/3168133712 >> 172.21.15.45:0/3168133712 conn(0x7fc35c06b930 msgr2=0x7fc35c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:40.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.810+0000 7fc363200640 1 -- 172.21.15.45:0/3168133712 shutdown_connections 2024-04-16T16:19:40.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.810+0000 7fc363200640 1 -- 172.21.15.45:0/3168133712 wait complete. 2024-04-16T16:19:40.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.811+0000 7fc363200640 1 Processor -- start 2024-04-16T16:19:40.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.812+0000 7fc363200640 1 -- start start 2024-04-16T16:19:40.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.812+0000 7fc363200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc348003f60 0x7fc35c15b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:40.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.812+0000 7fc363200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc35c15f3b0 0x7fc35c15f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:40.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.813+0000 7fc363200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc35c12f5d0 0x7fc35c12fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:40.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.813+0000 7fc363200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c15e790 con 0x7fc348003f60 2024-04-16T16:19:40.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.813+0000 7fc363200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c1453d0 con 0x7fc35c15f3b0 2024-04-16T16:19:40.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.813+0000 7fc363200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc35c15e910 con 0x7fc35c12f5d0 2024-04-16T16:19:40.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.813+0000 7fc361e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc348003f60 0x7fc35c15b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:40.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.813+0000 7fc361e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc348003f60 0x7fc35c15b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33380/0 (socket says 172.21.15.45:33380) 2024-04-16T16:19:40.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.813+0000 7fc361e00640 1 -- 172.21.15.45:0/1934985993 learned_addr learned my addr 172.21.15.45:0/1934985993 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:40.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.813+0000 7fc362800640 1 --2- 172.21.15.45:0/1934985993 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc35c12f5d0 0x7fc35c12fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:40.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.813+0000 7fc361e00640 1 -- 172.21.15.45:0/1934985993 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc35c12f5d0 msgr2=0x7fc35c12fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:40.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.813+0000 7fc361e00640 1 --2- 172.21.15.45:0/1934985993 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc35c12f5d0 0x7fc35c12fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:40.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.813+0000 7fc361400640 1 --2- 172.21.15.45:0/1934985993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc35c15f3b0 0x7fc35c15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:40.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.813+0000 7fc361e00640 1 -- 172.21.15.45:0/1934985993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc35c15f3b0 msgr2=0x7fc35c15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:40.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.813+0000 7fc361e00640 1 --2- 172.21.15.45:0/1934985993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc35c15f3b0 0x7fc35c15f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:40.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.813+0000 7fc361e00640 1 -- 172.21.15.45:0/1934985993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35c15b5b0 con 0x7fc348003f60 2024-04-16T16:19:40.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.814+0000 7fc361e00640 1 --2- 172.21.15.45:0/1934985993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc348003f60 0x7fc35c15b070 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fc35400e8c0 tx=0x7fc35400ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:40.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.814+0000 7fc352000640 1 -- 172.21.15.45:0/1934985993 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc354003b80 con 0x7fc348003f60 2024-04-16T16:19:40.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.814+0000 7fc363200640 1 -- 172.21.15.45:0/1934985993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc35c130350 con 0x7fc348003f60 2024-04-16T16:19:40.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.814+0000 7fc352000640 1 -- 172.21.15.45:0/1934985993 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc354018070 con 0x7fc348003f60 2024-04-16T16:19:40.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.814+0000 7fc352000640 1 -- 172.21.15.45:0/1934985993 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3540050d0 con 0x7fc348003f60 2024-04-16T16:19:40.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.814+0000 7fc363200640 1 -- 172.21.15.45:0/1934985993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc35c1f5ea0 con 0x7fc348003f60 2024-04-16T16:19:40.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.816+0000 7fc363200640 1 -- 172.21.15.45:0/1934985993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc35c15fd70 con 0x7fc348003f60 2024-04-16T16:19:40.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.817+0000 7fc352000640 1 -- 172.21.15.45:0/1934985993 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7fc354005270 con 0x7fc348003f60 2024-04-16T16:19:40.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.818+0000 7fc352000640 1 --2- 172.21.15.45:0/1934985993 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc34006c9b0 0x7fc34006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:40.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.818+0000 7fc352000640 1 -- 172.21.15.45:0/1934985993 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5631+0+0 (secure 0 0 0) 0x7fc35408bc10 con 0x7fc348003f60 2024-04-16T16:19:40.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.819+0000 7fc361400640 1 --2- 172.21.15.45:0/1934985993 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc34006c9b0 0x7fc34006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:40.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.820+0000 7fc361400640 1 --2- 172.21.15.45:0/1934985993 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc34006c9b0 0x7fc34006ee70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc34c036000 tx=0x7fc34c009030 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:40.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:40.823+0000 7fc352000640 1 -- 172.21.15.45:0/1934985993 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc35401a310 con 0x7fc348003f60 2024-04-16T16:19:40.884 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:40.883+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:41.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.041+0000 7fc363200640 1 -- 172.21.15.45:0/1934985993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc35c1f59b0 con 0x7fc348003f60 2024-04-16T16:19:41.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.043+0000 7fc352000640 1 -- 172.21.15.45:0/1934985993 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) ==== 74+0+14036 (secure 0 0 0) 0x7fc35405a020 con 0x7fc348003f60 2024-04-16T16:19:41.043 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:41.043 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":105,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:19:38.511175+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6834/882883484":"2024-04-17T16:19:35.407385+0000","172.21.15.103:0/3000963222":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/963070403":"2024-04-17T16:19:28.777448+0000","172.21.15.103:6832/3948990093":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/2784388736":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/2130767640":"2024-04-17T16:19:28.777448+0000","172.21.15.103:6833/839990480":"2024-04-17T16:19:22.373133+0000","172.21.15.103:6832/839990480":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2646915013":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/1161670790":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/287602360":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2666528075":"2024-04-17T16:19:22.373133+0000","172.21.15.103:6833/3948990093":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/4022617615":"2024-04-17T16:19:22.373133+0000","172.21.15.103:0/2967808514":"2024-04-17T16:19:22.373133+0000","172.21.15.45:6833/882883484":"2024-04-17T16:19:35.407385+0000","172.21.15.103:0/1615159946":"2024-04-17T16:19:28.777448+0000","172.21.15.103:0/2879899995":"2024-04-16T17:19:09.753356+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:19:41.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.046+0000 7fc363200640 1 -- 172.21.15.45:0/1934985993 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc34006c9b0 msgr2=0x7fc34006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:41.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.046+0000 7fc363200640 1 --2- 172.21.15.45:0/1934985993 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc34006c9b0 0x7fc34006ee70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc34c036000 tx=0x7fc34c009030 comp rx=0 tx=0).stop 2024-04-16T16:19:41.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.046+0000 7fc363200640 1 -- 172.21.15.45:0/1934985993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc348003f60 msgr2=0x7fc35c15b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:41.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.047+0000 7fc363200640 1 --2- 172.21.15.45:0/1934985993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc348003f60 0x7fc35c15b070 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fc35400e8c0 tx=0x7fc35400ed80 comp rx=0 tx=0).stop 2024-04-16T16:19:41.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.047+0000 7fc363200640 1 -- 172.21.15.45:0/1934985993 shutdown_connections 2024-04-16T16:19:41.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.047+0000 7fc363200640 1 --2- 172.21.15.45:0/1934985993 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc35c12f5d0 0x7fc35c12fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.047+0000 7fc363200640 1 --2- 172.21.15.45:0/1934985993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc35c15f3b0 0x7fc35c15f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.047+0000 7fc363200640 1 --2- 172.21.15.45:0/1934985993 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc34006c9b0 0x7fc34006ee70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.047+0000 7fc363200640 1 --2- 172.21.15.45:0/1934985993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc348003f60 0x7fc35c15b070 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.047+0000 7fc363200640 1 -- 172.21.15.45:0/1934985993 >> 172.21.15.45:0/1934985993 conn(0x7fc35c06b930 msgr2=0x7fc35c15bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:41.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.047+0000 7fc363200640 1 -- 172.21.15.45:0/1934985993 shutdown_connections 2024-04-16T16:19:41.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.047+0000 7fc363200640 1 -- 172.21.15.45:0/1934985993 wait complete. 2024-04-16T16:19:41.067 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:19:41.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:19:41.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.300+0000 7f4431600640 1 Processor -- start 2024-04-16T16:19:41.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.300+0000 7f4431600640 1 -- start start 2024-04-16T16:19:41.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.302+0000 7f4431600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f442c15ec80 0x7f442c0d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:41.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.302+0000 7f4431600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f442c0b6c00 0x7f442c168990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:41.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.302+0000 7f4431600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f442c0b7ca0 con 0x7f442c15f060 2024-04-16T16:19:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.302+0000 7f4431600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f442c0d3220 con 0x7f442c15ec80 2024-04-16T16:19:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.302+0000 7f4431600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f442c0d3450 con 0x7f442c0b6c00 2024-04-16T16:19:41.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.303+0000 7f442aa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f442c15ec80 0x7f442c0d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:41.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.303+0000 7f442b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f442c0b6c00 0x7f442c168990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:41.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.303+0000 7f442aa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f442c15ec80 0x7f442c0d2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:56742/0 (socket says 172.21.15.45:56742) 2024-04-16T16:19:41.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.303+0000 7f442aa00640 1 -- 172.21.15.45:0/335265045 learned_addr learned my addr 172.21.15.45:0/335265045 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:41.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.304+0000 7f442b400640 1 -- 172.21.15.45:0/335265045 >> v2:172.21.15.103:3300/0 conn(0x7f442c15ec80 msgr2=0x7f442c0d2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:41.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.304+0000 7f442b400640 1 --2- 172.21.15.45:0/335265045 >> v2:172.21.15.103:3300/0 conn(0x7f442c15ec80 0x7f442c0d2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.304+0000 7f442b400640 1 -- 172.21.15.45:0/335265045 >> v1:172.21.15.103:6789/0 conn(0x7f442c15f060 legacy=0x7f442c0b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:41.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.304+0000 7f442b400640 1 -- 172.21.15.45:0/335265045 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442c0d3730 con 0x7f442c0b6c00 2024-04-16T16:19:41.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.305+0000 7f442b400640 1 --2- 172.21.15.45:0/335265045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f442c0b6c00 0x7f442c168990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4424009d90 tx=0x7f442402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fd5dc9f83b8bd18 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:41.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.306+0000 7f442a000640 1 -- 172.21.15.45:0/335265045 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424039070 con 0x7f442c0b6c00 2024-04-16T16:19:41.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.306+0000 7f442a000640 1 -- 172.21.15.45:0/335265045 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f442402cce0 con 0x7f442c0b6c00 2024-04-16T16:19:41.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.306+0000 7f442a000640 1 -- 172.21.15.45:0/335265045 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424034660 con 0x7f442c0b6c00 2024-04-16T16:19:41.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.307+0000 7f4431600640 1 -- 172.21.15.45:0/335265045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f442c0b6c00 msgr2=0x7f442c168990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:41.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.307+0000 7f4431600640 1 --2- 172.21.15.45:0/335265045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f442c0b6c00 0x7f442c168990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4424009d90 tx=0x7f442402bf30 comp rx=0 tx=0).stop 2024-04-16T16:19:41.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.307+0000 7f4431600640 1 -- 172.21.15.45:0/335265045 shutdown_connections 2024-04-16T16:19:41.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.307+0000 7f4431600640 1 --2- 172.21.15.45:0/335265045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f442c0b6c00 0x7f442c168990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.307+0000 7f4431600640 1 --2- 172.21.15.45:0/335265045 >> v2:172.21.15.103:3300/0 conn(0x7f442c15ec80 0x7f442c0d2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.307+0000 7f4431600640 1 -- 172.21.15.45:0/335265045 >> 172.21.15.45:0/335265045 conn(0x7f442c0d16d0 msgr2=0x7f442c0d1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:41.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.307+0000 7f4431600640 1 -- 172.21.15.45:0/335265045 shutdown_connections 2024-04-16T16:19:41.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.307+0000 7f4431600640 1 -- 172.21.15.45:0/335265045 wait complete. 2024-04-16T16:19:41.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.309+0000 7f4431600640 1 Processor -- start 2024-04-16T16:19:41.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.310+0000 7f4431600640 1 -- start start 2024-04-16T16:19:41.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.311+0000 7f4431600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f442c0b6c00 0x7f442c15f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:41.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.311+0000 7f4431600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f442c15ec80 0x7f442c1621d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:41.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.311+0000 7f4431600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f442c14ecf0 0x7f442c1f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.311+0000 7f4431600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f442c0ba190 con 0x7f442c14ecf0 2024-04-16T16:19:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.311+0000 7f4431600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f442c0d3cf0 con 0x7f442c15ec80 2024-04-16T16:19:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.311+0000 7f4431600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f442c0ba310 con 0x7f442c0b6c00 2024-04-16T16:19:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.311+0000 7f442b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f442c14ecf0 0x7f442c1f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.311+0000 7f442b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f442c14ecf0 0x7f442c1f4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33392/0 (socket says 172.21.15.45:33392) 2024-04-16T16:19:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.311+0000 7f442b400640 1 -- 172.21.15.45:0/1652451739 learned_addr learned my addr 172.21.15.45:0/1652451739 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.312+0000 7f442aa00640 1 --2- 172.21.15.45:0/1652451739 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f442c0b6c00 0x7f442c15f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.312+0000 7f4423e00640 1 --2- 172.21.15.45:0/1652451739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f442c15ec80 0x7f442c1621d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.312+0000 7f442b400640 1 -- 172.21.15.45:0/1652451739 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f442c0b6c00 msgr2=0x7f442c15f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.312+0000 7f442b400640 1 --2- 172.21.15.45:0/1652451739 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f442c0b6c00 0x7f442c15f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.312+0000 7f442b400640 1 -- 172.21.15.45:0/1652451739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f442c15ec80 msgr2=0x7f442c1621d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:41.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.312+0000 7f442b400640 1 --2- 172.21.15.45:0/1652451739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f442c15ec80 0x7f442c1621d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.312+0000 7f442b400640 1 -- 172.21.15.45:0/1652451739 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442c158310 con 0x7f442c14ecf0 2024-04-16T16:19:41.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.312+0000 7f442b400640 1 --2- 172.21.15.45:0/1652451739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f442c14ecf0 0x7f442c1f4ca0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f4424009d90 tx=0x7f4424004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:41.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.313+0000 7f4422a00640 1 -- 172.21.15.45:0/1652451739 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424039050 con 0x7f442c14ecf0 2024-04-16T16:19:41.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.313+0000 7f4422a00640 1 -- 172.21.15.45:0/1652451739 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4424045070 con 0x7f442c14ecf0 2024-04-16T16:19:41.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.313+0000 7f4422a00640 1 -- 172.21.15.45:0/1652451739 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424009450 con 0x7f442c14ecf0 2024-04-16T16:19:41.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.313+0000 7f4431600640 1 -- 172.21.15.45:0/1652451739 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f442c0c9b60 con 0x7f442c14ecf0 2024-04-16T16:19:41.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.313+0000 7f4431600640 1 -- 172.21.15.45:0/1652451739 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f442c0ca120 con 0x7f442c14ecf0 2024-04-16T16:19:41.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.315+0000 7f4431600640 1 -- 172.21.15.45:0/1652451739 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f442c075780 con 0x7f442c14ecf0 2024-04-16T16:19:41.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.315+0000 7f4422a00640 1 -- 172.21.15.45:0/1652451739 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f442403e850 con 0x7f442c14ecf0 2024-04-16T16:19:41.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.316+0000 7f4422a00640 1 --2- 172.21.15.45:0/1652451739 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f440806c980 0x7f440806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:41.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.316+0000 7f4422a00640 1 -- 172.21.15.45:0/1652451739 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5631+0+0 (secure 0 0 0) 0x7f44240ae7b0 con 0x7f442c14ecf0 2024-04-16T16:19:41.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.317+0000 7f442aa00640 1 --2- 172.21.15.45:0/1652451739 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f440806c980 0x7f440806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:41.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.318+0000 7f442aa00640 1 --2- 172.21.15.45:0/1652451739 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f440806c980 0x7f440806ee40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f441c009bd0 tx=0x7f441c009240 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:41.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.321+0000 7f4422a00640 1 -- 172.21.15.45:0/1652451739 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f442c158310 con 0x7f442c14ecf0 2024-04-16T16:19:41.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.534+0000 7f4431600640 1 -- 172.21.15.45:0/1652451739 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f442c06da40 con 0x7f442c14ecf0 2024-04-16T16:19:41.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.535+0000 7f4422a00640 1 -- 172.21.15.45:0/1652451739 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) ==== 93+0+10 (secure 0 0 0) 0x7f442407cdf0 con 0x7f442c14ecf0 2024-04-16T16:19:41.535 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:19:41.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.538+0000 7f4431600640 1 -- 172.21.15.45:0/1652451739 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f440806c980 msgr2=0x7f440806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:41.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.538+0000 7f4431600640 1 --2- 172.21.15.45:0/1652451739 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f440806c980 0x7f440806ee40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f441c009bd0 tx=0x7f441c009240 comp rx=0 tx=0).stop 2024-04-16T16:19:41.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.538+0000 7f4431600640 1 -- 172.21.15.45:0/1652451739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f442c14ecf0 msgr2=0x7f442c1f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:41.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.538+0000 7f4431600640 1 --2- 172.21.15.45:0/1652451739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f442c14ecf0 0x7f442c1f4ca0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f4424009d90 tx=0x7f4424004a40 comp rx=0 tx=0).stop 2024-04-16T16:19:41.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.538+0000 7f4431600640 1 -- 172.21.15.45:0/1652451739 shutdown_connections 2024-04-16T16:19:41.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.538+0000 7f4431600640 1 --2- 172.21.15.45:0/1652451739 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f440806c980 0x7f440806ee40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.538+0000 7f4431600640 1 --2- 172.21.15.45:0/1652451739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f442c14ecf0 0x7f442c1f4ca0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.538+0000 7f4431600640 1 --2- 172.21.15.45:0/1652451739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f442c15ec80 0x7f442c1621d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.538+0000 7f4431600640 1 --2- 172.21.15.45:0/1652451739 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f442c0b6c00 0x7f442c15f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.538+0000 7f4431600640 1 -- 172.21.15.45:0/1652451739 >> 172.21.15.45:0/1652451739 conn(0x7f442c0d16d0 msgr2=0x7f442c0cf1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:41.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.538+0000 7f4431600640 1 -- 172.21.15.45:0/1652451739 shutdown_connections 2024-04-16T16:19:41.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.538+0000 7f4431600640 1 -- 172.21.15.45:0/1652451739 wait complete. 2024-04-16T16:19:41.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_mds_rank' 2024-04-16T16:19:41.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.784+0000 7f1f2b200640 1 Processor -- start 2024-04-16T16:19:41.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.784+0000 7f1f2b200640 1 -- start start 2024-04-16T16:19:41.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.786+0000 7f1f2b200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1f2c161e90 0x7f1f2c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:41.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.786+0000 7f1f2b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1f2c162270 0x7f1f2c0d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:41.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.786+0000 7f1f2b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f2c0d52d0 0x7f1f2c06ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:41.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.786+0000 7f1f2b200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1f2c0d2e30 con 0x7f1f2c161e90 2024-04-16T16:19:41.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.786+0000 7f1f2b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1f2c0d3080 con 0x7f1f2c162270 2024-04-16T16:19:41.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.786+0000 7f1f2b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1f2c0d3360 con 0x7f1f2c0d52d0 2024-04-16T16:19:41.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.786+0000 7f1f29e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1f2c161e90 0x7f1f2c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:41.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.786+0000 7f1f29e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1f2c161e90 0x7f1f2c0cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33396/0 (socket says 172.21.15.45:33396) 2024-04-16T16:19:41.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.786+0000 7f1f29e00640 1 -- 172.21.15.45:0/2825257084 learned_addr learned my addr 172.21.15.45:0/2825257084 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:41.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.787+0000 7f1f29400640 1 --2- 172.21.15.45:0/2825257084 >> v2:172.21.15.103:3300/0 conn(0x7f1f2c162270 0x7f1f2c0d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:41.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.787+0000 7f1f2a800640 1 --2- 172.21.15.45:0/2825257084 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f2c0d52d0 0x7f1f2c06ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:41.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.787+0000 7f1f29e00640 1 -- 172.21.15.45:0/2825257084 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f2c0d52d0 msgr2=0x7f1f2c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:41.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.787+0000 7f1f29e00640 1 --2- 172.21.15.45:0/2825257084 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f2c0d52d0 0x7f1f2c06ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.787+0000 7f1f29e00640 1 -- 172.21.15.45:0/2825257084 >> v2:172.21.15.103:3300/0 conn(0x7f1f2c162270 msgr2=0x7f1f2c0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:41.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.787+0000 7f1f29e00640 1 --2- 172.21.15.45:0/2825257084 >> v2:172.21.15.103:3300/0 conn(0x7f1f2c162270 0x7f1f2c0d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.787+0000 7f1f29e00640 1 -- 172.21.15.45:0/2825257084 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f2c0d3640 con 0x7f1f2c161e90 2024-04-16T16:19:41.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.787+0000 7f1f29400640 1 --2- 172.21.15.45:0/2825257084 >> v2:172.21.15.103:3300/0 conn(0x7f1f2c162270 0x7f1f2c0d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:19:41.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.789+0000 7f1f29e00640 1 --2- 172.21.15.45:0/2825257084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c161e90 0x7f1f2c0cf7d0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c009870 tx=0x7f1f1c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35847c2943d5dab1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:41.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.789+0000 7f1f28a00640 1 -- 172.21.15.45:0/2825257084 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c034070 con 0x7f1f2c161e90 2024-04-16T16:19:41.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.789+0000 7f1f28a00640 1 -- 172.21.15.45:0/2825257084 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f1c030840 con 0x7f1f2c161e90 2024-04-16T16:19:41.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.789+0000 7f1f28a00640 1 -- 172.21.15.45:0/2825257084 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c030c10 con 0x7f1f2c161e90 2024-04-16T16:19:41.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.790+0000 7f1f2b200640 1 -- 172.21.15.45:0/2825257084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c161e90 msgr2=0x7f1f2c0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:41.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.790+0000 7f1f2b200640 1 --2- 172.21.15.45:0/2825257084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c161e90 0x7f1f2c0cf7d0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c009870 tx=0x7f1f1c02b980 comp rx=0 tx=0).stop 2024-04-16T16:19:41.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.790+0000 7f1f2b200640 1 -- 172.21.15.45:0/2825257084 shutdown_connections 2024-04-16T16:19:41.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.790+0000 7f1f2b200640 1 --2- 172.21.15.45:0/2825257084 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f2c0d52d0 0x7f1f2c06ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.790+0000 7f1f2b200640 1 --2- 172.21.15.45:0/2825257084 >> v2:172.21.15.103:3300/0 conn(0x7f1f2c162270 0x7f1f2c0d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.790+0000 7f1f2b200640 1 --2- 172.21.15.45:0/2825257084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c161e90 0x7f1f2c0cf7d0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.790+0000 7f1f2b200640 1 -- 172.21.15.45:0/2825257084 >> 172.21.15.45:0/2825257084 conn(0x7f1f2c0d1940 msgr2=0x7f1f2c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:41.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.790+0000 7f1f2b200640 1 -- 172.21.15.45:0/2825257084 shutdown_connections 2024-04-16T16:19:41.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.790+0000 7f1f2b200640 1 -- 172.21.15.45:0/2825257084 wait complete. 2024-04-16T16:19:41.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.792+0000 7f1f2b200640 1 Processor -- start 2024-04-16T16:19:41.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.792+0000 7f1f2b200640 1 -- start start 2024-04-16T16:19:41.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.793+0000 7f1f2b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f2c0d52d0 0x7f1f2c0cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:41.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.793+0000 7f1f2b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c161e90 0x7f1f2c1f5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:41.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.793+0000 7f1f2b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f2c162270 0x7f1f2c0c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:41.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.793+0000 7f1f2b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1f2c0d4380 con 0x7f1f2c161e90 2024-04-16T16:19:41.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.793+0000 7f1f2b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1f2c0cfd10 con 0x7f1f2c0d52d0 2024-04-16T16:19:41.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.793+0000 7f1f2b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1f2c0d4500 con 0x7f1f2c162270 2024-04-16T16:19:41.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.793+0000 7f1f29400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c161e90 0x7f1f2c1f5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:41.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.793+0000 7f1f29400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c161e90 0x7f1f2c1f5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33402/0 (socket says 172.21.15.45:33402) 2024-04-16T16:19:41.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.793+0000 7f1f29400640 1 -- 172.21.15.45:0/4188415179 learned_addr learned my addr 172.21.15.45:0/4188415179 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:41.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.793+0000 7f1f29e00640 1 --2- 172.21.15.45:0/4188415179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f2c0d52d0 0x7f1f2c0cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:41.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.793+0000 7f1f2a800640 1 --2- 172.21.15.45:0/4188415179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f2c162270 0x7f1f2c0c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:41.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.794+0000 7f1f29400640 1 -- 172.21.15.45:0/4188415179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f2c162270 msgr2=0x7f1f2c0c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:41.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.794+0000 7f1f29400640 1 --2- 172.21.15.45:0/4188415179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f2c162270 0x7f1f2c0c9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.794+0000 7f1f29400640 1 -- 172.21.15.45:0/4188415179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f2c0d52d0 msgr2=0x7f1f2c0cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:41.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.794+0000 7f1f29400640 1 --2- 172.21.15.45:0/4188415179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f2c0d52d0 0x7f1f2c0cc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:41.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.794+0000 7f1f29400640 1 -- 172.21.15.45:0/4188415179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f2c1f5890 con 0x7f1f2c161e90 2024-04-16T16:19:41.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.794+0000 7f1f29e00640 1 --2- 172.21.15.45:0/4188415179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f2c0d52d0 0x7f1f2c0cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:41.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.794+0000 7f1f29400640 1 --2- 172.21.15.45:0/4188415179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c161e90 0x7f1f2c1f5350 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f1f1400d7d0 tx=0x7f1f1400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:41.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.794+0000 7f1f22000640 1 -- 172.21.15.45:0/4188415179 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f140096a0 con 0x7f1f2c161e90 2024-04-16T16:19:41.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.794+0000 7f1f22000640 1 -- 172.21.15.45:0/4188415179 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f14019070 con 0x7f1f2c161e90 2024-04-16T16:19:41.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.794+0000 7f1f22000640 1 -- 172.21.15.45:0/4188415179 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f14009a80 con 0x7f1f2c161e90 2024-04-16T16:19:41.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.794+0000 7f1f2b200640 1 -- 172.21.15.45:0/4188415179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f2c0cee00 con 0x7f1f2c161e90 2024-04-16T16:19:41.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.794+0000 7f1f2b200640 1 -- 172.21.15.45:0/4188415179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f2c0cbb00 con 0x7f1f2c161e90 2024-04-16T16:19:41.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.796+0000 7f1f2b200640 1 -- 172.21.15.45:0/4188415179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f2c075780 con 0x7f1f2c161e90 2024-04-16T16:19:41.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.797+0000 7f1f22000640 1 -- 172.21.15.45:0/4188415179 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f1f14009c20 con 0x7f1f2c161e90 2024-04-16T16:19:41.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.798+0000 7f1f22000640 1 --2- 172.21.15.45:0/4188415179 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f1f0806c980 0x7f1f0806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:41.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.798+0000 7f1f22000640 1 -- 172.21.15.45:0/4188415179 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5631+0+0 (secure 0 0 0) 0x7f1f1408c060 con 0x7f1f2c161e90 2024-04-16T16:19:41.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.799+0000 7f1f29e00640 1 --2- 172.21.15.45:0/4188415179 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f1f0806c980 0x7f1f0806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:41.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.800+0000 7f1f29e00640 1 --2- 172.21.15.45:0/4188415179 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f1f0806c980 0x7f1f0806ee40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c036000 tx=0x7f1f1c03b040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:41.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:41.803+0000 7f1f22000640 1 -- 172.21.15.45:0/4188415179 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f2c1f5890 con 0x7f1f2c161e90 2024-04-16T16:19:42.009 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:42.007+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:42.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.109+0000 7f1f2b200640 1 -- 172.21.15.45:0/4188415179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_mds_rank"]} v 0) -- 0x7f1f2c06dae0 con 0x7f1f2c161e90 2024-04-16T16:19:42.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.552+0000 7f1f22000640 1 -- 172.21.15.45:0/4188415179 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_mds_rank"]}]=0 v364) ==== 152+0+0 (secure 0 0 0) 0x7f1f14011340 con 0x7f1f2c161e90 2024-04-16T16:19:42.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.556+0000 7f1f2b200640 1 -- 172.21.15.45:0/4188415179 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f1f0806c980 msgr2=0x7f1f0806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:42.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.556+0000 7f1f2b200640 1 --2- 172.21.15.45:0/4188415179 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f1f0806c980 0x7f1f0806ee40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c036000 tx=0x7f1f1c03b040 comp rx=0 tx=0).stop 2024-04-16T16:19:42.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.556+0000 7f1f2b200640 1 -- 172.21.15.45:0/4188415179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c161e90 msgr2=0x7f1f2c1f5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:42.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.556+0000 7f1f2b200640 1 --2- 172.21.15.45:0/4188415179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c161e90 0x7f1f2c1f5350 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f1f1400d7d0 tx=0x7f1f1400dc90 comp rx=0 tx=0).stop 2024-04-16T16:19:42.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.557+0000 7f1f2b200640 1 -- 172.21.15.45:0/4188415179 shutdown_connections 2024-04-16T16:19:42.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.557+0000 7f1f2b200640 1 --2- 172.21.15.45:0/4188415179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f2c162270 0x7f1f2c0c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:42.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.557+0000 7f1f2b200640 1 --2- 172.21.15.45:0/4188415179 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f1f0806c980 0x7f1f0806ee40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:42.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.557+0000 7f1f2b200640 1 --2- 172.21.15.45:0/4188415179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c161e90 0x7f1f2c1f5350 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:42.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.557+0000 7f1f2b200640 1 --2- 172.21.15.45:0/4188415179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f2c0d52d0 0x7f1f2c0cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:42.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.557+0000 7f1f2b200640 1 -- 172.21.15.45:0/4188415179 >> 172.21.15.45:0/4188415179 conn(0x7f1f2c0d1940 msgr2=0x7f1f2c0d25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:42.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.557+0000 7f1f2b200640 1 -- 172.21.15.45:0/4188415179 shutdown_connections 2024-04-16T16:19:42.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.557+0000 7f1f2b200640 1 -- 172.21.15.45:0/4188415179 wait complete. 2024-04-16T16:19:42.582 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-16T16:19:42.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.821+0000 7f05d5400640 1 Processor -- start 2024-04-16T16:19:42.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.822+0000 7f05d5400640 1 -- start start 2024-04-16T16:19:42.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.823+0000 7f05d5400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f05d00b6c60 0x7f05d015ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:42.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.823+0000 7f05d5400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f05d00d0310 0x7f05d00b7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:42.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.823+0000 7f05d5400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05d00ced60 con 0x7f05d00b4eb0 2024-04-16T16:19:42.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.823+0000 7f05d5400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f05d00b7580 con 0x7f05d00d0310 2024-04-16T16:19:42.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.823+0000 7f05d5400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f05d015d440 con 0x7f05d00b6c60 2024-04-16T16:19:42.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.824+0000 7f05cde00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f05d00d0310 0x7f05d00b7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:42.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.824+0000 7f05cde00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f05d00d0310 0x7f05d00b7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33410/0 (socket says 172.21.15.45:33410) 2024-04-16T16:19:42.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.824+0000 7f05cde00640 1 -- 172.21.15.45:0/3027897504 learned_addr learned my addr 172.21.15.45:0/3027897504 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:42.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.824+0000 7f05cde00640 1 -- 172.21.15.45:0/3027897504 >> v2:172.21.15.103:3300/0 conn(0x7f05d00b6c60 msgr2=0x7f05d015ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:42.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.824+0000 7f05cde00640 1 --2- 172.21.15.45:0/3027897504 >> v2:172.21.15.103:3300/0 conn(0x7f05d00b6c60 0x7f05d015ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:42.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.824+0000 7f05cde00640 1 -- 172.21.15.45:0/3027897504 >> v1:172.21.15.103:6789/0 conn(0x7f05d00b4eb0 legacy=0x7f05d00d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:42.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.824+0000 7f05cde00640 1 -- 172.21.15.45:0/3027897504 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d015d720 con 0x7f05d00d0310 2024-04-16T16:19:42.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.825+0000 7f05cde00640 1 --2- 172.21.15.45:0/3027897504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d00d0310 0x7f05d00b7040 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f05d0071c40 tx=0x7f05b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d502bf0dfff3f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:42.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.826+0000 7f05cd400640 1 -- 172.21.15.45:0/3027897504 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b8035020 con 0x7f05d00d0310 2024-04-16T16:19:42.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.826+0000 7f05cd400640 1 -- 172.21.15.45:0/3027897504 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05b8031870 con 0x7f05d00d0310 2024-04-16T16:19:42.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.826+0000 7f05cd400640 1 -- 172.21.15.45:0/3027897504 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b8031c40 con 0x7f05d00d0310 2024-04-16T16:19:42.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.826+0000 7f05d5400640 1 -- 172.21.15.45:0/3027897504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d00d0310 msgr2=0x7f05d00b7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:42.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.826+0000 7f05d5400640 1 --2- 172.21.15.45:0/3027897504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d00d0310 0x7f05d00b7040 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f05d0071c40 tx=0x7f05b802b910 comp rx=0 tx=0).stop 2024-04-16T16:19:42.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.826+0000 7f05d5400640 1 -- 172.21.15.45:0/3027897504 shutdown_connections 2024-04-16T16:19:42.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.826+0000 7f05d5400640 1 --2- 172.21.15.45:0/3027897504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d00d0310 0x7f05d00b7040 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:42.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.826+0000 7f05d5400640 1 --2- 172.21.15.45:0/3027897504 >> v2:172.21.15.103:3300/0 conn(0x7f05d00b6c60 0x7f05d015ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:42.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.826+0000 7f05d5400640 1 -- 172.21.15.45:0/3027897504 >> 172.21.15.45:0/3027897504 conn(0x7f05d009aa70 msgr2=0x7f05d009ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:42.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.826+0000 7f05d5400640 1 -- 172.21.15.45:0/3027897504 shutdown_connections 2024-04-16T16:19:42.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.826+0000 7f05d5400640 1 -- 172.21.15.45:0/3027897504 wait complete. 2024-04-16T16:19:42.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.828+0000 7f05d5400640 1 Processor -- start 2024-04-16T16:19:42.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.828+0000 7f05d5400640 1 -- start start 2024-04-16T16:19:42.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.829+0000 7f05d5400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05d00b4eb0 0x7f05d01f5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:42.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.829+0000 7f05d5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d00b6c60 0x7f05d01e36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:42.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.829+0000 7f05d5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05d00d0310 0x7f05d015bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:42.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.829+0000 7f05d5400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f05d015a870 con 0x7f05d00b6c60 2024-04-16T16:19:42.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.829+0000 7f05d5400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f05d015dce0 con 0x7f05d00d0310 2024-04-16T16:19:42.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.829+0000 7f05d5400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f05d00d1520 con 0x7f05d00b4eb0 2024-04-16T16:19:42.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.830+0000 7f05cde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d00b6c60 0x7f05d01e36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:42.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.830+0000 7f05cde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d00b6c60 0x7f05d01e36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33420/0 (socket says 172.21.15.45:33420) 2024-04-16T16:19:42.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.830+0000 7f05cde00640 1 -- 172.21.15.45:0/3280074445 learned_addr learned my addr 172.21.15.45:0/3280074445 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:42.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.830+0000 7f05cf200640 1 --2- 172.21.15.45:0/3280074445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05d00d0310 0x7f05d015bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:42.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.830+0000 7f05ce800640 1 --2- 172.21.15.45:0/3280074445 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05d00b4eb0 0x7f05d01f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:42.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.830+0000 7f05cde00640 1 -- 172.21.15.45:0/3280074445 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05d00b4eb0 msgr2=0x7f05d01f5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:42.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.830+0000 7f05cde00640 1 --2- 172.21.15.45:0/3280074445 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05d00b4eb0 0x7f05d01f5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:42.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.830+0000 7f05cde00640 1 -- 172.21.15.45:0/3280074445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05d00d0310 msgr2=0x7f05d015bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:42.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.830+0000 7f05cde00640 1 --2- 172.21.15.45:0/3280074445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05d00d0310 0x7f05d015bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:42.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.830+0000 7f05cde00640 1 -- 172.21.15.45:0/3280074445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d01f5ca0 con 0x7f05d00b6c60 2024-04-16T16:19:42.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.831+0000 7f05cde00640 1 --2- 172.21.15.45:0/3280074445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d00b6c60 0x7f05d01e36d0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f05b8000c00 tx=0x7f05b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:42.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.831+0000 7f05c6a00640 1 -- 172.21.15.45:0/3280074445 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b80443f0 con 0x7f05d00b6c60 2024-04-16T16:19:42.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.831+0000 7f05c6a00640 1 -- 172.21.15.45:0/3280074445 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05b80445b0 con 0x7f05d00b6c60 2024-04-16T16:19:42.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.831+0000 7f05c6a00640 1 -- 172.21.15.45:0/3280074445 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b80449a0 con 0x7f05d00b6c60 2024-04-16T16:19:42.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.831+0000 7f05d5400640 1 -- 172.21.15.45:0/3280074445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d01f5f80 con 0x7f05d00b6c60 2024-04-16T16:19:42.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.831+0000 7f05d5400640 1 -- 172.21.15.45:0/3280074445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d01f6540 con 0x7f05d00b6c60 2024-04-16T16:19:42.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.832+0000 7f05d5400640 1 -- 172.21.15.45:0/3280074445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05d0075780 con 0x7f05d00b6c60 2024-04-16T16:19:42.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.833+0000 7f05c6a00640 1 -- 172.21.15.45:0/3280074445 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f05b804b060 con 0x7f05d00b6c60 2024-04-16T16:19:42.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.834+0000 7f05c6a00640 1 --2- 172.21.15.45:0/3280074445 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f05ac06c980 0x7f05ac06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:42.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.834+0000 7f05c6a00640 1 -- 172.21.15.45:0/3280074445 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5631+0+0 (secure 0 0 0) 0x7f05b80adfc0 con 0x7f05d00b6c60 2024-04-16T16:19:42.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.835+0000 7f05ce800640 1 --2- 172.21.15.45:0/3280074445 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f05ac06c980 0x7f05ac06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:42.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.836+0000 7f05ce800640 1 --2- 172.21.15.45:0/3280074445 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f05ac06c980 0x7f05ac06ee40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f05c0009890 tx=0x7f05c0008040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:42.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:42.839+0000 7f05c6a00640 1 -- 172.21.15.45:0/3280074445 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f05b803f070 con 0x7f05d00b6c60 2024-04-16T16:19:43.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.051+0000 7f05d5400640 1 -- 172.21.15.45:0/3280074445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f05d00d25f0 con 0x7f05d00b6c60 2024-04-16T16:19:43.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.058+0000 7f05c6a00640 1 -- 172.21.15.45:0/3280074445 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7f05d00ced60 con 0x7f05d00b6c60 2024-04-16T16:19:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.061+0000 7f05d5400640 1 -- 172.21.15.45:0/3280074445 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f05ac06c980 msgr2=0x7f05ac06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.061+0000 7f05d5400640 1 --2- 172.21.15.45:0/3280074445 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f05ac06c980 0x7f05ac06ee40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f05c0009890 tx=0x7f05c0008040 comp rx=0 tx=0).stop 2024-04-16T16:19:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.061+0000 7f05d5400640 1 -- 172.21.15.45:0/3280074445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d00b6c60 msgr2=0x7f05d01e36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.061+0000 7f05d5400640 1 --2- 172.21.15.45:0/3280074445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d00b6c60 0x7f05d01e36d0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f05b8000c00 tx=0x7f05b80046c0 comp rx=0 tx=0).stop 2024-04-16T16:19:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.061+0000 7f05d5400640 1 -- 172.21.15.45:0/3280074445 shutdown_connections 2024-04-16T16:19:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.061+0000 7f05d5400640 1 --2- 172.21.15.45:0/3280074445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05d00d0310 0x7f05d015bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.061+0000 7f05d5400640 1 --2- 172.21.15.45:0/3280074445 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f05ac06c980 0x7f05ac06ee40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.061+0000 7f05d5400640 1 --2- 172.21.15.45:0/3280074445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d00b6c60 0x7f05d01e36d0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.061+0000 7f05d5400640 1 --2- 172.21.15.45:0/3280074445 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05d00b4eb0 0x7f05d01f5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.061+0000 7f05d5400640 1 -- 172.21.15.45:0/3280074445 >> 172.21.15.45:0/3280074445 conn(0x7f05d009aa70 msgr2=0x7f05d0157c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.062+0000 7f05d5400640 1 -- 172.21.15.45:0/3280074445 shutdown_connections 2024-04-16T16:19:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.062+0000 7f05d5400640 1 -- 172.21.15.45:0/3280074445 wait complete. 2024-04-16T16:19:43.080 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-16T16:19:43.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:19:43.157 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:43.156+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:43.204 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:19:43.327 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-16T16:19:43.327 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:19:43.327 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:19:43.389 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:19:43.389 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:19:43.389 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:19:43.447 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:19:43.447 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:19:43.447 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:19:43.509 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:19:43.509 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:19:43.509 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:19:43.569 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:19:43.570 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-16T16:19:43.570 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:19:43.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.800+0000 7f3f4c800640 1 Processor -- start 2024-04-16T16:19:43.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.801+0000 7f3f4c800640 1 -- start start 2024-04-16T16:19:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.802+0000 7f3f4c800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3f441462e0 0x7f3f44072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.802+0000 7f3f4c800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f440cf6d0 con 0x7f3f4415ea70 2024-04-16T16:19:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.802+0000 7f3f4c800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f441466c0 con 0x7f3f440728c0 2024-04-16T16:19:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.802+0000 7f3f4c800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3f440cf3f0 con 0x7f3f441462e0 2024-04-16T16:19:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.803+0000 7f3f4a800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3f4415ea70 0x7f3f44166490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60106/0 (socket says 172.21.15.45:60106) 2024-04-16T16:19:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.803+0000 7f3f4a800640 1 -- 172.21.15.45:0/1355853648 learned_addr learned my addr 172.21.15.45:0/1355853648 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:43.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.803+0000 7f3f49e00640 1 --2- 172.21.15.45:0/1355853648 >> v2:172.21.15.103:3300/0 conn(0x7f3f441462e0 0x7f3f44072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:43.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.803+0000 7f3f48a00640 1 -- 172.21.15.45:0/1355853648 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 760795432 0 0) 0x7f3f440cf6d0 con 0x7f3f4415ea70 2024-04-16T16:19:43.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.803+0000 7f3f48a00640 1 -- 172.21.15.45:0/1355853648 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f2c003520 con 0x7f3f4415ea70 2024-04-16T16:19:43.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.804+0000 7f3f48a00640 1 -- 172.21.15.45:0/1355853648 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 997079570 0 0) 0x7f3f2c003520 con 0x7f3f4415ea70 2024-04-16T16:19:43.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.804+0000 7f3f48a00640 1 -- 172.21.15.45:0/1355853648 >> v2:172.21.15.103:3300/0 conn(0x7f3f441462e0 msgr2=0x7f3f44072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:43.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.804+0000 7f3f48a00640 1 --2- 172.21.15.45:0/1355853648 >> v2:172.21.15.103:3300/0 conn(0x7f3f441462e0 0x7f3f44072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:43.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.804+0000 7f3f48a00640 1 -- 172.21.15.45:0/1355853648 >> v1:172.21.15.103:6789/0 conn(0x7f3f440728c0 legacy=0x7f3f440ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:43.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.804+0000 7f3f48a00640 1 -- 172.21.15.45:0/1355853648 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4415cf80 con 0x7f3f4415ea70 2024-04-16T16:19:43.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.804+0000 7f3f48a00640 1 -- 172.21.15.45:0/1355853648 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f3f400019f0 con 0x7f3f4415ea70 2024-04-16T16:19:43.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.804+0000 7f3f48a00640 1 -- 172.21.15.45:0/1355853648 >> v1:172.21.15.45:6789/0 conn(0x7f3f4415ea70 legacy=0x7f3f44166490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:43.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.804+0000 7f3f48a00640 1 --2- 172.21.15.45:0/1355853648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f2c003a00 0x7f3f2c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:43.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.804+0000 7f3f48a00640 1 --2- 172.21.15.45:0/1355853648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f2c004800 0x7f3f2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:43.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.804+0000 7f3f48a00640 1 --2- 172.21.15.45:0/1355853648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f2c005210 0x7f3f2c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:43.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.804+0000 7f3f48a00640 1 -- 172.21.15.45:0/1355853648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3f440cf3f0 con 0x7f3f2c005210 2024-04-16T16:19:43.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.804+0000 7f3f48a00640 1 -- 172.21.15.45:0/1355853648 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2c00c160 con 0x7f3f2c003a00 2024-04-16T16:19:43.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.804+0000 7f3f48a00640 1 -- 172.21.15.45:0/1355853648 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3f2c00c360 con 0x7f3f2c004800 2024-04-16T16:19:43.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.804+0000 7f3f4a800640 1 --2- 172.21.15.45:0/1355853648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f2c005210 0x7f3f2c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.805+0000 7f3f49e00640 1 --2- 172.21.15.45:0/1355853648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f2c003a00 0x7f3f2c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.805+0000 7f3f49400640 1 --2- 172.21.15.45:0/1355853648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f2c004800 0x7f3f2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.805+0000 7f3f4a800640 1 -- 172.21.15.45:0/1355853648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f2c004800 msgr2=0x7f3f2c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.805+0000 7f3f4a800640 1 --2- 172.21.15.45:0/1355853648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f2c004800 0x7f3f2c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.805+0000 7f3f4a800640 1 -- 172.21.15.45:0/1355853648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f2c003a00 msgr2=0x7f3f2c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.805+0000 7f3f4a800640 1 --2- 172.21.15.45:0/1355853648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f2c003a00 0x7f3f2c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.805+0000 7f3f4a800640 1 -- 172.21.15.45:0/1355853648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f2c003520 con 0x7f3f2c005210 2024-04-16T16:19:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.805+0000 7f3f49e00640 1 --2- 172.21.15.45:0/1355853648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f2c003a00 0x7f3f2c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.805+0000 7f3f49400640 1 --2- 172.21.15.45:0/1355853648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f2c004800 0x7f3f2c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.806+0000 7f3f4a800640 1 -- 172.21.15.45:0/1355853648 reap_dead start 2024-04-16T16:19:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.806+0000 7f3f4a800640 1 --2- 172.21.15.45:0/1355853648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f2c005210 0x7f3f2c00b9c0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000a740 tx=0x7f3f2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae70439ddf1339a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.806+0000 7f3f48a00640 1 -- 172.21.15.45:0/1355853648 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4000a390 con 0x7f3f2c005210 2024-04-16T16:19:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.806+0000 7f3f48a00640 1 -- 172.21.15.45:0/1355853648 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f40034650 con 0x7f3f2c005210 2024-04-16T16:19:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.806+0000 7f3f48a00640 1 -- 172.21.15.45:0/1355853648 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40034a40 con 0x7f3f2c005210 2024-04-16T16:19:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.807+0000 7f3f4c800640 1 -- 172.21.15.45:0/1355853648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f2c005210 msgr2=0x7f3f2c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.807+0000 7f3f4c800640 1 --2- 172.21.15.45:0/1355853648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f2c005210 0x7f3f2c00b9c0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000a740 tx=0x7f3f2c005830 comp rx=0 tx=0).stop 2024-04-16T16:19:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.807+0000 7f3f4c800640 1 -- 172.21.15.45:0/1355853648 shutdown_connections 2024-04-16T16:19:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.807+0000 7f3f4c800640 1 --2- 172.21.15.45:0/1355853648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f2c005210 0x7f3f2c00b9c0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.807+0000 7f3f4c800640 1 -- 172.21.15.45:0/1355853648 >> 172.21.15.45:0/1355853648 conn(0x7f3f440b2750 msgr2=0x7f3f440b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.807+0000 7f3f4c800640 1 -- 172.21.15.45:0/1355853648 shutdown_connections 2024-04-16T16:19:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.807+0000 7f3f4c800640 1 -- 172.21.15.45:0/1355853648 wait complete. 2024-04-16T16:19:43.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.809+0000 7f3f4c800640 1 Processor -- start 2024-04-16T16:19:43.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.809+0000 7f3f4c800640 1 -- start start 2024-04-16T16:19:43.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.810+0000 7f3f4c800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f2c005210 0x7f3f44140970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:43.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.810+0000 7f3f4c800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f4413e2f0 0x7f3f4413e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:43.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.810+0000 7f3f4c800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f44143550 0x7f3f441439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:43.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.810+0000 7f3f4c800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3f44072380 con 0x7f3f44143550 2024-04-16T16:19:43.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.810+0000 7f3f4c800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4415d540 con 0x7f3f4413e2f0 2024-04-16T16:19:43.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.810+0000 7f3f4c800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3f44072500 con 0x7f3f2c005210 2024-04-16T16:19:43.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.810+0000 7f3f4a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f44143550 0x7f3f441439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:43.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.810+0000 7f3f4a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f44143550 0x7f3f441439b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51936/0 (socket says 172.21.15.45:51936) 2024-04-16T16:19:43.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.810+0000 7f3f4a800640 1 -- 172.21.15.45:0/1601815078 learned_addr learned my addr 172.21.15.45:0/1601815078 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:43.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.811+0000 7f3f49400640 1 --2- 172.21.15.45:0/1601815078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f4413e2f0 0x7f3f4413e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:43.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.811+0000 7f3f49e00640 1 --2- 172.21.15.45:0/1601815078 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f2c005210 0x7f3f44140970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:43.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.811+0000 7f3f4a800640 1 -- 172.21.15.45:0/1601815078 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f2c005210 msgr2=0x7f3f44140970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:43.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.811+0000 7f3f4a800640 1 --2- 172.21.15.45:0/1601815078 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f2c005210 0x7f3f44140970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:43.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.811+0000 7f3f4a800640 1 -- 172.21.15.45:0/1601815078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f4413e2f0 msgr2=0x7f3f4413e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:43.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.811+0000 7f3f4a800640 1 --2- 172.21.15.45:0/1601815078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f4413e2f0 0x7f3f4413e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:43.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.811+0000 7f3f4a800640 1 -- 172.21.15.45:0/1601815078 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4413f060 con 0x7f3f44143550 2024-04-16T16:19:43.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.811+0000 7f3f4a800640 1 --2- 172.21.15.45:0/1601815078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f44143550 0x7f3f441439b0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000a860 tx=0x7f3f4415d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:43.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.811+0000 7f3f3e000640 1 -- 172.21.15.45:0/1601815078 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40039040 con 0x7f3f44143550 2024-04-16T16:19:43.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.811+0000 7f3f3e000640 1 -- 172.21.15.45:0/1601815078 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f40043070 con 0x7f3f44143550 2024-04-16T16:19:43.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.811+0000 7f3f3e000640 1 -- 172.21.15.45:0/1601815078 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40004d60 con 0x7f3f44143550 2024-04-16T16:19:43.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.812+0000 7f3f4c800640 1 -- 172.21.15.45:0/1601815078 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f4413f3a0 con 0x7f3f44143550 2024-04-16T16:19:43.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.812+0000 7f3f4c800640 1 -- 172.21.15.45:0/1601815078 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f4413fbe0 con 0x7f3f44143550 2024-04-16T16:19:43.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.813+0000 7f3f4c800640 1 -- 172.21.15.45:0/1601815078 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f44075780 con 0x7f3f44143550 2024-04-16T16:19:43.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.814+0000 7f3f3e000640 1 -- 172.21.15.45:0/1601815078 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f3f40034650 con 0x7f3f44143550 2024-04-16T16:19:43.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.815+0000 7f3f3e000640 1 --2- 172.21.15.45:0/1601815078 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3f2806c980 0x7f3f2806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.815+0000 7f3f3e000640 1 -- 172.21.15.45:0/1601815078 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5631+0+0 (secure 0 0 0) 0x7f3f400adae0 con 0x7f3f44143550 2024-04-16T16:19:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.815+0000 7f3f49e00640 1 --2- 172.21.15.45:0/1601815078 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3f2806c980 0x7f3f2806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:43.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.817+0000 7f3f49e00640 1 --2- 172.21.15.45:0/1601815078 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3f2806c980 0x7f3f2806ee40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f380099c0 tx=0x7f3f38008040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:43.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:43.819+0000 7f3f3e000640 1 -- 172.21.15.45:0/1601815078 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3f4413f3a0 con 0x7f3f44143550 2024-04-16T16:19:44.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.074+0000 7f3f4c800640 1 -- 172.21.15.45:0/1601815078 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f4413fdb0 con 0x7f3f44143550 2024-04-16T16:19:44.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.075+0000 7f3f3e000640 1 -- 172.21.15.45:0/1601815078 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) ==== 94+0+3875 (secure 0 0 0) 0x7f3f4007bd60 con 0x7f3f44143550 2024-04-16T16:19:44.074 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:44.074 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":75,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5804,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2862612740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2862612740},{"type":"v1","addr":"172.21.15.103:6836","nonce":2862612740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6304,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2934367463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2934367463},{"type":"v1","addr":"172.21.15.103:6837","nonce":2934367463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1425499906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1425499906},{"type":"v1","addr":"172.21.15.45:6835","nonce":1425499906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":8085,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/1016169285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":1016169285},{"type":"v1","addr":"172.21.15.45:6834","nonce":1016169285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75}],"filesystems":[]} 2024-04-16T16:19:44.075 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 75 2024-04-16T16:19:44.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.077+0000 7f3f4c800640 1 -- 172.21.15.45:0/1601815078 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3f2806c980 msgr2=0x7f3f2806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:44.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.077+0000 7f3f4c800640 1 --2- 172.21.15.45:0/1601815078 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3f2806c980 0x7f3f2806ee40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f380099c0 tx=0x7f3f38008040 comp rx=0 tx=0).stop 2024-04-16T16:19:44.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.077+0000 7f3f4c800640 1 -- 172.21.15.45:0/1601815078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f44143550 msgr2=0x7f3f441439b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:44.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.077+0000 7f3f4c800640 1 --2- 172.21.15.45:0/1601815078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f44143550 0x7f3f441439b0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000a860 tx=0x7f3f4415d150 comp rx=0 tx=0).stop 2024-04-16T16:19:44.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.078+0000 7f3f4c800640 1 -- 172.21.15.45:0/1601815078 shutdown_connections 2024-04-16T16:19:44.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.078+0000 7f3f4c800640 1 --2- 172.21.15.45:0/1601815078 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3f2806c980 0x7f3f2806ee40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:44.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.078+0000 7f3f4c800640 1 --2- 172.21.15.45:0/1601815078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f44143550 0x7f3f441439b0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.078+0000 7f3f4c800640 1 --2- 172.21.15.45:0/1601815078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f4413e2f0 0x7f3f4413e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.078+0000 7f3f4c800640 1 --2- 172.21.15.45:0/1601815078 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3f2c005210 0x7f3f44140970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.078+0000 7f3f4c800640 1 -- 172.21.15.45:0/1601815078 >> 172.21.15.45:0/1601815078 conn(0x7f3f440b2750 msgr2=0x7f3f4406e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.078+0000 7f3f4c800640 1 -- 172.21.15.45:0/1601815078 shutdown_connections 2024-04-16T16:19:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.078+0000 7f3f4c800640 1 -- 172.21.15.45:0/1601815078 wait complete. 2024-04-16T16:19:44.097 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-16T16:19:44.097 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-16T16:19:44.097 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-16T16:19:44.097 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:19:44.097 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-16T16:19:44.097 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-16T16:19:44.098 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-16T16:19:44.098 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:19:44.098 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-16T16:19:44.098 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-16T16:19:44.098 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-16T16:19:44.099 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:19:44.099 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-16T16:19:44.099 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-16T16:19:44.099 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-16T16:19:44.099 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:19:44.114 INFO:tasks.ceph.mds.c:Stopped 2024-04-16T16:19:44.114 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-16T16:19:44.115 INFO:tasks.ceph.mds.a:Stopped 2024-04-16T16:19:44.116 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-16T16:19:44.121 INFO:tasks.ceph.mds.b:Stopped 2024-04-16T16:19:44.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-16T16:19:44.122 INFO:tasks.ceph.mds.d:Stopped 2024-04-16T16:19:44.122 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-16T16:19:44.163 INFO:tasks.ceph.mds.c:Started 2024-04-16T16:19:44.164 DEBUG:teuthology.parallel:result is None 2024-04-16T16:19:44.169 INFO:tasks.ceph.mds.b:Started 2024-04-16T16:19:44.169 DEBUG:teuthology.parallel:result is None 2024-04-16T16:19:44.170 INFO:tasks.ceph.mds.a:Started 2024-04-16T16:19:44.170 DEBUG:teuthology.parallel:result is None 2024-04-16T16:19:44.176 INFO:tasks.ceph.mds.d:Started 2024-04-16T16:19:44.176 DEBUG:teuthology.parallel:result is None 2024-04-16T16:19:44.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-16T16:19:44.279 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:44.279+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:44.330 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2024-04-16T16:19:44.332 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2024-04-16T16:19:44.345 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-16T16:19:44.345 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-16T16:19:44.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.383+0000 7ff120e00640 1 Processor -- start 2024-04-16T16:19:44.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.384+0000 7ff120e00640 1 -- start start 2024-04-16T16:19:44.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.385+0000 7ff120e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff11c15e9a0 0x7ff11c0b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:44.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.385+0000 7ff120e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff11c15ed80 0x7ff11c0b6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:44.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.385+0000 7ff120e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff11c162820 con 0x7ff11c06b930 2024-04-16T16:19:44.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.385+0000 7ff120e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff11c0b56b0 con 0x7ff11c15e9a0 2024-04-16T16:19:44.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.385+0000 7ff120e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff11c071a80 con 0x7ff11c15ed80 2024-04-16T16:19:44.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.385+0000 7ff11be00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff11c06b930 0x7ff11c073050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60122/0 (socket says 172.21.15.45:60122) 2024-04-16T16:19:44.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.385+0000 7ff11be00640 1 -- 172.21.15.45:0/3567618353 learned_addr learned my addr 172.21.15.45:0/3567618353 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:44.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.386+0000 7ff11b400640 1 --2- 172.21.15.45:0/3567618353 >> v2:172.21.15.103:3300/0 conn(0x7ff11c15e9a0 0x7ff11c0b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:44.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.386+0000 7ff113e00640 1 --2- 172.21.15.45:0/3567618353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff11c15ed80 0x7ff11c0b6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:44.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.386+0000 7ff11aa00640 1 -- 172.21.15.45:0/3567618353 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 972735260 0 0) 0x7ff11c162820 con 0x7ff11c06b930 2024-04-16T16:19:44.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.386+0000 7ff11aa00640 1 -- 172.21.15.45:0/3567618353 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff100003520 con 0x7ff11c06b930 2024-04-16T16:19:44.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.386+0000 7ff11b400640 1 -- 172.21.15.45:0/3567618353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff11c15ed80 msgr2=0x7ff11c0b6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:44.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.386+0000 7ff11b400640 1 --2- 172.21.15.45:0/3567618353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff11c15ed80 0x7ff11c0b6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:44.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.386+0000 7ff11b400640 1 -- 172.21.15.45:0/3567618353 >> v1:172.21.15.45:6789/0 conn(0x7ff11c06b930 legacy=0x7ff11c073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:44.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.386+0000 7ff11b400640 1 -- 172.21.15.45:0/3567618353 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11c162ac0 con 0x7ff11c15e9a0 2024-04-16T16:19:44.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.387+0000 7ff11b400640 1 --2- 172.21.15.45:0/3567618353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff11c15e9a0 0x7ff11c0b6200 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff11c071c00 tx=0x7ff10c02b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa7e9778b01e870f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:44.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.388+0000 7ff11aa00640 1 -- 172.21.15.45:0/3567618353 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c031770 con 0x7ff11c15e9a0 2024-04-16T16:19:44.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.388+0000 7ff11aa00640 1 -- 172.21.15.45:0/3567618353 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff10c037070 con 0x7ff11c15e9a0 2024-04-16T16:19:44.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.388+0000 7ff11aa00640 1 -- 172.21.15.45:0/3567618353 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c031b30 con 0x7ff11c15e9a0 2024-04-16T16:19:44.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.388+0000 7ff120e00640 1 -- 172.21.15.45:0/3567618353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff11c15e9a0 msgr2=0x7ff11c0b6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:44.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.388+0000 7ff120e00640 1 --2- 172.21.15.45:0/3567618353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff11c15e9a0 0x7ff11c0b6200 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff11c071c00 tx=0x7ff10c02b810 comp rx=0 tx=0).stop 2024-04-16T16:19:44.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.389+0000 7ff120e00640 1 -- 172.21.15.45:0/3567618353 shutdown_connections 2024-04-16T16:19:44.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.389+0000 7ff120e00640 1 --2- 172.21.15.45:0/3567618353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff11c15ed80 0x7ff11c0b6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:44.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.389+0000 7ff120e00640 1 --2- 172.21.15.45:0/3567618353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff11c15e9a0 0x7ff11c0b6200 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:44.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.389+0000 7ff120e00640 1 -- 172.21.15.45:0/3567618353 >> 172.21.15.45:0/3567618353 conn(0x7ff11c06f0e0 msgr2=0x7ff11c06f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:44.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.389+0000 7ff120e00640 1 -- 172.21.15.45:0/3567618353 shutdown_connections 2024-04-16T16:19:44.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.389+0000 7ff120e00640 1 -- 172.21.15.45:0/3567618353 wait complete. 2024-04-16T16:19:44.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.391+0000 7ff120e00640 1 Processor -- start 2024-04-16T16:19:44.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.391+0000 7ff120e00640 1 -- start start 2024-04-16T16:19:44.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.392+0000 7ff120e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff11c06b930 0x7ff11c15bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:44.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.392+0000 7ff120e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff11c15e9a0 0x7ff11c1fbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:44.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.392+0000 7ff120e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff11c15ed80 0x7ff11c1fc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:44.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.392+0000 7ff120e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c06c560 con 0x7ff11c15e9a0 2024-04-16T16:19:44.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.392+0000 7ff120e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c163080 con 0x7ff11c06b930 2024-04-16T16:19:44.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.392+0000 7ff120e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff11c1635a0 con 0x7ff11c15ed80 2024-04-16T16:19:44.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.392+0000 7ff113e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff11c15e9a0 0x7ff11c1fbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:44.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.392+0000 7ff113e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff11c15e9a0 0x7ff11c1fbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51972/0 (socket says 172.21.15.45:51972) 2024-04-16T16:19:44.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.392+0000 7ff113e00640 1 -- 172.21.15.45:0/1189377071 learned_addr learned my addr 172.21.15.45:0/1189377071 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:44.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.392+0000 7ff11be00640 1 --2- 172.21.15.45:0/1189377071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff11c15ed80 0x7ff11c1fc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:44.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.392+0000 7ff11b400640 1 --2- 172.21.15.45:0/1189377071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff11c06b930 0x7ff11c15bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:44.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.393+0000 7ff113e00640 1 -- 172.21.15.45:0/1189377071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff11c15ed80 msgr2=0x7ff11c1fc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:44.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.393+0000 7ff113e00640 1 --2- 172.21.15.45:0/1189377071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff11c15ed80 0x7ff11c1fc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:44.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.393+0000 7ff113e00640 1 -- 172.21.15.45:0/1189377071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff11c06b930 msgr2=0x7ff11c15bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:44.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.393+0000 7ff113e00640 1 --2- 172.21.15.45:0/1189377071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff11c06b930 0x7ff11c15bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.393+0000 7ff113e00640 1 -- 172.21.15.45:0/1189377071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11c1585a0 con 0x7ff11c15e9a0 2024-04-16T16:19:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.393+0000 7ff11b400640 1 --2- 172.21.15.45:0/1189377071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff11c06b930 0x7ff11c15bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:19:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.393+0000 7ff113e00640 1 --2- 172.21.15.45:0/1189377071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff11c15e9a0 0x7ff11c1fbf20 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7ff10400e870 tx=0x7ff10400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.393+0000 7ff113400640 1 -- 172.21.15.45:0/1189377071 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff104017070 con 0x7ff11c15e9a0 2024-04-16T16:19:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.393+0000 7ff113400640 1 -- 172.21.15.45:0/1189377071 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff10400c940 con 0x7ff11c15e9a0 2024-04-16T16:19:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.393+0000 7ff113400640 1 -- 172.21.15.45:0/1189377071 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10400cd30 con 0x7ff11c15e9a0 2024-04-16T16:19:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.393+0000 7ff120e00640 1 -- 172.21.15.45:0/1189377071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff11c15cd20 con 0x7ff11c15e9a0 2024-04-16T16:19:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.393+0000 7ff120e00640 1 -- 172.21.15.45:0/1189377071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff11c15d2e0 con 0x7ff11c15e9a0 2024-04-16T16:19:44.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.395+0000 7ff120e00640 1 -- 172.21.15.45:0/1189377071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff11c075780 con 0x7ff11c15e9a0 2024-04-16T16:19:44.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.396+0000 7ff113400640 1 -- 172.21.15.45:0/1189377071 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7ff10401a070 con 0x7ff11c15e9a0 2024-04-16T16:19:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.396+0000 7ff113400640 1 --2- 172.21.15.45:0/1189377071 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7ff0f806c980 0x7ff0f806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.397+0000 7ff113400640 1 -- 172.21.15.45:0/1189377071 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5631+0+0 (secure 0 0 0) 0x7ff10408ba40 con 0x7ff11c15e9a0 2024-04-16T16:19:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.397+0000 7ff11b400640 1 --2- 172.21.15.45:0/1189377071 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7ff0f806c980 0x7ff0f806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.398+0000 7ff11b400640 1 --2- 172.21.15.45:0/1189377071 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7ff0f806c980 0x7ff0f806ee40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff10c036000 tx=0x7ff10c03d040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.401+0000 7ff113400640 1 -- 172.21.15.45:0/1189377071 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff104059780 con 0x7ff11c15e9a0 2024-04-16T16:19:44.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:44.640+0000 7ff120e00640 1 -- 172.21.15.45:0/1189377071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7ff11c0b0320 con 0x7ff11c15e9a0 2024-04-16T16:19:45.455 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:45.455+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:45.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.616+0000 7ff113400640 1 -- 172.21.15.45:0/1189377071 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v110) ==== 97+0+0 (secure 0 0 0) 0x7ff10405da80 con 0x7ff11c15e9a0 2024-04-16T16:19:45.615 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2024-04-16T16:19:45.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.620+0000 7ff120e00640 1 -- 172.21.15.45:0/1189377071 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7ff0f806c980 msgr2=0x7ff0f806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:45.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.620+0000 7ff120e00640 1 --2- 172.21.15.45:0/1189377071 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7ff0f806c980 0x7ff0f806ee40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff10c036000 tx=0x7ff10c03d040 comp rx=0 tx=0).stop 2024-04-16T16:19:45.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.620+0000 7ff120e00640 1 -- 172.21.15.45:0/1189377071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff11c15e9a0 msgr2=0x7ff11c1fbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:45.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.620+0000 7ff120e00640 1 --2- 172.21.15.45:0/1189377071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff11c15e9a0 0x7ff11c1fbf20 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7ff10400e870 tx=0x7ff10400ed30 comp rx=0 tx=0).stop 2024-04-16T16:19:45.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.620+0000 7ff120e00640 1 -- 172.21.15.45:0/1189377071 shutdown_connections 2024-04-16T16:19:45.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.620+0000 7ff120e00640 1 --2- 172.21.15.45:0/1189377071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff11c15ed80 0x7ff11c1fc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:45.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.620+0000 7ff120e00640 1 --2- 172.21.15.45:0/1189377071 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7ff0f806c980 0x7ff0f806ee40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:45.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.620+0000 7ff120e00640 1 --2- 172.21.15.45:0/1189377071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff11c15e9a0 0x7ff11c1fbf20 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:45.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.620+0000 7ff120e00640 1 --2- 172.21.15.45:0/1189377071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff11c06b930 0x7ff11c15bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:45.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.620+0000 7ff120e00640 1 -- 172.21.15.45:0/1189377071 >> 172.21.15.45:0/1189377071 conn(0x7ff11c06f0e0 msgr2=0x7ff11c15f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:45.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.620+0000 7ff120e00640 1 -- 172.21.15.45:0/1189377071 shutdown_connections 2024-04-16T16:19:45.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.620+0000 7ff120e00640 1 -- 172.21.15.45:0/1189377071 wait complete. 2024-04-16T16:19:45.642 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-16T16:19:45.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.868+0000 7f1db6c00640 1 Processor -- start 2024-04-16T16:19:45.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.868+0000 7f1db6c00640 1 -- start start 2024-04-16T16:19:45.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.870+0000 7f1db6c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1db00d4560 0x7f1db00d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:45.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.870+0000 7f1db6c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1db00d18f0 0x7f1db00b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:45.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.870+0000 7f1db6c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1db00d34c0 con 0x7f1db015ec60 2024-04-16T16:19:45.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.870+0000 7f1db6c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1db00d1d60 con 0x7f1db00d4560 2024-04-16T16:19:45.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.870+0000 7f1db6c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1db00d37a0 con 0x7f1db00d18f0 2024-04-16T16:19:45.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.870+0000 7f1dafe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1db015ec60 0x7f1db00d2a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60138/0 (socket says 172.21.15.45:60138) 2024-04-16T16:19:45.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.870+0000 7f1dafe00640 1 -- 172.21.15.45:0/2305717584 learned_addr learned my addr 172.21.15.45:0/2305717584 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:45.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.870+0000 7f1da7e00640 1 --2- 172.21.15.45:0/2305717584 >> v2:172.21.15.103:3300/0 conn(0x7f1db00d4560 0x7f1db00d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:45.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.871+0000 7f1db4c00640 1 --2- 172.21.15.45:0/2305717584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1db00d18f0 0x7f1db00b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:45.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.871+0000 7f1daf400640 1 -- 172.21.15.45:0/2305717584 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 812161275 0 0) 0x7f1db00d34c0 con 0x7f1db015ec60 2024-04-16T16:19:45.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.871+0000 7f1daf400640 1 -- 172.21.15.45:0/2305717584 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d94003520 con 0x7f1db015ec60 2024-04-16T16:19:45.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.871+0000 7f1daf400640 1 -- 172.21.15.45:0/2305717584 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2861910147 0 0) 0x7f1d94003520 con 0x7f1db015ec60 2024-04-16T16:19:45.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.871+0000 7f1daf400640 1 -- 172.21.15.45:0/2305717584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1db00d18f0 msgr2=0x7f1db00b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:45.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.871+0000 7f1daf400640 1 --2- 172.21.15.45:0/2305717584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1db00d18f0 0x7f1db00b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:45.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.871+0000 7f1daf400640 1 -- 172.21.15.45:0/2305717584 >> v2:172.21.15.103:3300/0 conn(0x7f1db00d4560 msgr2=0x7f1db00d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:45.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.871+0000 7f1daf400640 1 --2- 172.21.15.45:0/2305717584 >> v2:172.21.15.103:3300/0 conn(0x7f1db00d4560 0x7f1db00d13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:45.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.871+0000 7f1daf400640 1 -- 172.21.15.45:0/2305717584 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db00d3a80 con 0x7f1db015ec60 2024-04-16T16:19:45.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.871+0000 7f1daf400640 1 -- 172.21.15.45:0/2305717584 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f1da0001670 con 0x7f1db015ec60 2024-04-16T16:19:45.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.871+0000 7f1daf400640 1 -- 172.21.15.45:0/2305717584 >> v1:172.21.15.45:6789/0 conn(0x7f1db015ec60 legacy=0x7f1db00d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:45.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.871+0000 7f1daf400640 1 --2- 172.21.15.45:0/2305717584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d940039b0 0x7f1d94003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:45.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.871+0000 7f1daf400640 1 --2- 172.21.15.45:0/2305717584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1db00d18f0 0x7f1d94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:45.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.871+0000 7f1daf400640 1 --2- 172.21.15.45:0/2305717584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d94004d40 0x7f1d940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:45.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.872+0000 7f1daf400640 1 -- 172.21.15.45:0/2305717584 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1db00d1d60 con 0x7f1d94004d40 2024-04-16T16:19:45.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.872+0000 7f1daf400640 1 -- 172.21.15.45:0/2305717584 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1db00d37a0 con 0x7f1d940039b0 2024-04-16T16:19:45.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.872+0000 7f1daf400640 1 -- 172.21.15.45:0/2305717584 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1d9400b9c0 con 0x7f1db00d18f0 2024-04-16T16:19:45.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.872+0000 7f1db4c00640 1 --2- 172.21.15.45:0/2305717584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d94004d40 0x7f1d940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:45.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.872+0000 7f1dafe00640 1 --2- 172.21.15.45:0/2305717584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d940039b0 0x7f1d94003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:45.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.872+0000 7f1da7e00640 1 --2- 172.21.15.45:0/2305717584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1db00d18f0 0x7f1d94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:45.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.872+0000 7f1db4c00640 1 -- 172.21.15.45:0/2305717584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1db00d18f0 msgr2=0x7f1d94004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:45.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.872+0000 7f1db4c00640 1 --2- 172.21.15.45:0/2305717584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1db00d18f0 0x7f1d94004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:45.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.872+0000 7f1db4c00640 1 -- 172.21.15.45:0/2305717584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d940039b0 msgr2=0x7f1d94003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:45.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.872+0000 7f1db4c00640 1 --2- 172.21.15.45:0/2305717584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d940039b0 0x7f1d94003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:45.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.872+0000 7f1db4c00640 1 -- 172.21.15.45:0/2305717584 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d94003520 con 0x7f1d94004d40 2024-04-16T16:19:45.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.873+0000 7f1db4c00640 1 --2- 172.21.15.45:0/2305717584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d94004d40 0x7f1d940051c0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1da800e170 tx=0x7f1d94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a9fb8da0dcd6ab server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:45.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.874+0000 7f1daf400640 1 -- 172.21.15.45:0/2305717584 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da8039070 con 0x7f1d94004d40 2024-04-16T16:19:45.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.874+0000 7f1daf400640 1 -- 172.21.15.45:0/2305717584 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1da802cbf0 con 0x7f1d94004d40 2024-04-16T16:19:45.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.874+0000 7f1daf400640 1 -- 172.21.15.45:0/2305717584 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da8009c10 con 0x7f1d94004d40 2024-04-16T16:19:45.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.874+0000 7f1db6c00640 1 -- 172.21.15.45:0/2305717584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d94004d40 msgr2=0x7f1d940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:45.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.874+0000 7f1db6c00640 1 --2- 172.21.15.45:0/2305717584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d94004d40 0x7f1d940051c0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1da800e170 tx=0x7f1d94005830 comp rx=0 tx=0).stop 2024-04-16T16:19:45.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.874+0000 7f1db4c00640 1 -- 172.21.15.45:0/2305717584 reap_dead start 2024-04-16T16:19:45.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.875+0000 7f1db6c00640 1 -- 172.21.15.45:0/2305717584 shutdown_connections 2024-04-16T16:19:45.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.875+0000 7f1db6c00640 1 -- 172.21.15.45:0/2305717584 >> 172.21.15.45:0/2305717584 conn(0x7f1db0073690 msgr2=0x7f1db006cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:45.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.875+0000 7f1db6c00640 1 -- 172.21.15.45:0/2305717584 shutdown_connections 2024-04-16T16:19:45.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.875+0000 7f1db6c00640 1 -- 172.21.15.45:0/2305717584 wait complete. 2024-04-16T16:19:45.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.877+0000 7f1db6c00640 1 Processor -- start 2024-04-16T16:19:45.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.877+0000 7f1db6c00640 1 -- start start 2024-04-16T16:19:45.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.878+0000 7f1db6c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d94004d40 0x7f1db00ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:45.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.878+0000 7f1db6c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1db01f4f00 0x7f1db00b91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:45.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.878+0000 7f1db6c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1db014de00 0x7f1db014e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:45.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.878+0000 7f1db6c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1db00ba460 con 0x7f1db014de00 2024-04-16T16:19:45.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.878+0000 7f1db6c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1db00d4040 con 0x7f1db01f4f00 2024-04-16T16:19:45.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.878+0000 7f1db6c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1db00d3110 con 0x7f1d94004d40 2024-04-16T16:19:45.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.878+0000 7f1db4c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1db014de00 0x7f1db014e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:45.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.878+0000 7f1db4c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1db014de00 0x7f1db014e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51996/0 (socket says 172.21.15.45:51996) 2024-04-16T16:19:45.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.878+0000 7f1db4c00640 1 -- 172.21.15.45:0/3516322037 learned_addr learned my addr 172.21.15.45:0/3516322037 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:45.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.878+0000 7f1db4c00640 1 -- 172.21.15.45:0/3516322037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d94004d40 msgr2=0x7f1db00ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:45.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.878+0000 7f1db4c00640 1 --2- 172.21.15.45:0/3516322037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d94004d40 0x7f1db00ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:45.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.878+0000 7f1db4c00640 1 -- 172.21.15.45:0/3516322037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1db01f4f00 msgr2=0x7f1db00b91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:45.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.878+0000 7f1db4c00640 1 --2- 172.21.15.45:0/3516322037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1db01f4f00 0x7f1db00b91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:45.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.878+0000 7f1db4c00640 1 -- 172.21.15.45:0/3516322037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db01f5380 con 0x7f1db014de00 2024-04-16T16:19:45.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.879+0000 7f1db4c00640 1 --2- 172.21.15.45:0/3516322037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1db014de00 0x7f1db014e260 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f1da800e170 tx=0x7f1da8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:45.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.879+0000 7f1dacc00640 1 -- 172.21.15.45:0/3516322037 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da8039030 con 0x7f1db014de00 2024-04-16T16:19:45.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.879+0000 7f1dacc00640 1 -- 172.21.15.45:0/3516322037 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1da8043070 con 0x7f1db014de00 2024-04-16T16:19:45.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.879+0000 7f1db6c00640 1 -- 172.21.15.45:0/3516322037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1db00cd230 con 0x7f1db014de00 2024-04-16T16:19:45.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.879+0000 7f1dacc00640 1 -- 172.21.15.45:0/3516322037 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da803d7d0 con 0x7f1db014de00 2024-04-16T16:19:45.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.879+0000 7f1db6c00640 1 -- 172.21.15.45:0/3516322037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1db00cd7f0 con 0x7f1db014de00 2024-04-16T16:19:45.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.881+0000 7f1db6c00640 1 -- 172.21.15.45:0/3516322037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1db00cda30 con 0x7f1db014de00 2024-04-16T16:19:45.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.882+0000 7f1dacc00640 1 -- 172.21.15.45:0/3516322037 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f1da8005d10 con 0x7f1db014de00 2024-04-16T16:19:45.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.882+0000 7f1dacc00640 1 --2- 172.21.15.45:0/3516322037 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f1d8c06c980 0x7f1d8c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:45.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.883+0000 7f1dacc00640 1 -- 172.21.15.45:0/3516322037 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 4857+0+0 (secure 0 0 0) 0x7f1da80adad0 con 0x7f1db014de00 2024-04-16T16:19:45.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.883+0000 7f1dafe00640 1 --2- 172.21.15.45:0/3516322037 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f1d8c06c980 0x7f1d8c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:45.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.885+0000 7f1dafe00640 1 --2- 172.21.15.45:0/3516322037 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f1d8c06c980 0x7f1d8c06ee40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1db00b7d80 tx=0x7f1da0001e70 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:45.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:45.887+0000 7f1dacc00640 1 -- 172.21.15.45:0/3516322037 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1da807c260 con 0x7f1db014de00 2024-04-16T16:19:46.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.177+0000 7f1db6c00640 1 -- 172.21.15.45:0/3516322037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f1db0075780 con 0x7f1db014de00 2024-04-16T16:19:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.178+0000 7f1dacc00640 1 -- 172.21.15.45:0/3516322037 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) ==== 80+0+5238 (secure 0 0 0) 0x7f1da807bb60 con 0x7f1db014de00 2024-04-16T16:19:46.177 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:46.177 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:19:46.177 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2024-04-16T16:19:46.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.177 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2024-04-16T16:19:46.177 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mpjReORAAJMoNf4qvfuxPSiV9+VlKfg==", 2024-04-16T16:19:46.177 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.177 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:46.177 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:19:46.178 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:19:46.178 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:46.178 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.178 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2024-04-16T16:19:46.178 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mxHeBIBAA3G8maaK65zaOaqXpAOA6xA==", 2024-04-16T16:19:46.178 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.178 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:46.178 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:19:46.178 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:19:46.178 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:46.178 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.179 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2024-04-16T16:19:46.179 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5m+AaODxAA08XSjc44/i2KYExYQSukaQ==", 2024-04-16T16:19:46.179 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.179 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:46.179 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:19:46.179 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:19:46.179 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:46.179 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.179 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2024-04-16T16:19:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mEQ4UMRAAOslSsxQ17bOxPrI6QMlRRw==", 2024-04-16T16:19:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:19:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:19:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2024-04-16T16:19:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBuox5m3l3OHxAA+UjwZx8Mn6tCu95TTYIG3w==", 2024-04-16T16:19:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2024-04-16T16:19:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxox5mozt1LhAAJL5NtMQdPVBtKOaVlMQBtw==", 2024-04-16T16:19:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2024-04-16T16:19:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB1ox5mfE7kABAAIvgZ5Wx+eIJ6QNKj2GVdpw==", 2024-04-16T16:19:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2024-04-16T16:19:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB4ox5mEvxcDRAAO0yXBKvsC7hZ6bpOAJyuag==", 2024-04-16T16:19:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+ox5mZubrORAAcGJLyRN5Ob9aBnopumnEUg==", 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCCox5mcRqsChAACktmuUHvqWkp/mHT3UcjRw==", 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFox5mT52FFhAAChhDDfNZmIMgxE1j90LZpQ==", 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCIox5mC+EDIxAAxtvM0on00lxTLaoRjN0Gfw==", 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:19:46.184 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ==", 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mYpVUGhAAYdR+40O6zbWqgbI55F0FYA==", 2024-04-16T16:19:46.185 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.186 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:46.186 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:19:46.186 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:19:46.186 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.186 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2024-04-16T16:19:46.186 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mWdlUDxAAuO2FsGH1/E+qP1xcyUEe3A==", 2024-04-16T16:19:46.186 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.186 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:46.186 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:19:46.186 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:19:46.186 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.187 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2024-04-16T16:19:46.187 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5miPdMJBAAqjKpctNYu9ot1lff3rouxg==", 2024-04-16T16:19:46.187 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.187 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:19:46.187 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:19:46.187 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:19:46.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.187 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2024-04-16T16:19:46.187 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBpox5mO4PMLBAA/0YbQCfHujgdEzpXs0nOLw==", 2024-04-16T16:19:46.187 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.187 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mGevaLBAAPpA6g+Fdi/XHqAqG0QZNag==", 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:46.188 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2024-04-16T16:19:46.189 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5moiNRIRAAz5JzOrjKNLkcphT+iMo+zw==", 2024-04-16T16:19:46.189 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:19:46.189 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:19:46.189 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:19:46.189 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:19:46.189 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.189 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:46.189 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-16T16:19:46.189 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:19:46.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.181+0000 7f1db6c00640 1 -- 172.21.15.45:0/3516322037 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f1d8c06c980 msgr2=0x7f1d8c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:46.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.181+0000 7f1db6c00640 1 --2- 172.21.15.45:0/3516322037 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f1d8c06c980 0x7f1d8c06ee40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1db00b7d80 tx=0x7f1da0001e70 comp rx=0 tx=0).stop 2024-04-16T16:19:46.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.181+0000 7f1db6c00640 1 -- 172.21.15.45:0/3516322037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1db014de00 msgr2=0x7f1db014e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:46.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.181+0000 7f1db6c00640 1 --2- 172.21.15.45:0/3516322037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1db014de00 0x7f1db014e260 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f1da800e170 tx=0x7f1da8009750 comp rx=0 tx=0).stop 2024-04-16T16:19:46.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.181+0000 7f1db6c00640 1 -- 172.21.15.45:0/3516322037 shutdown_connections 2024-04-16T16:19:46.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.181+0000 7f1db6c00640 1 --2- 172.21.15.45:0/3516322037 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f1d8c06c980 0x7f1d8c06ee40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.181+0000 7f1db6c00640 1 --2- 172.21.15.45:0/3516322037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1db014de00 0x7f1db014e260 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.181+0000 7f1db6c00640 1 --2- 172.21.15.45:0/3516322037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1db01f4f00 0x7f1db00b91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.181+0000 7f1db6c00640 1 --2- 172.21.15.45:0/3516322037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1d94004d40 0x7f1db00ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.181+0000 7f1db6c00640 1 -- 172.21.15.45:0/3516322037 >> 172.21.15.45:0/3516322037 conn(0x7f1db0073690 msgr2=0x7f1db00c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.181+0000 7f1db6c00640 1 -- 172.21.15.45:0/3516322037 shutdown_connections 2024-04-16T16:19:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.181+0000 7f1db6c00640 1 -- 172.21.15.45:0/3516322037 wait complete. 2024-04-16T16:19:46.200 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:19:46.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.410+0000 7f7382800640 1 Processor -- start 2024-04-16T16:19:46.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.411+0000 7f7382800640 1 -- start start 2024-04-16T16:19:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.412+0000 7f7382800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f737c162270 0x7f737c0d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.412+0000 7f7382800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f737c0d52d0 0x7f737c06ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.412+0000 7f7382800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f737c0d3080 con 0x7f737c161e90 2024-04-16T16:19:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.412+0000 7f7382800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f737c0d2e30 con 0x7f737c0d52d0 2024-04-16T16:19:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.412+0000 7f7382800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f737c0d3360 con 0x7f737c162270 2024-04-16T16:19:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.413+0000 7f7381400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f737c161e90 0x7f737c0cf7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60154/0 (socket says 172.21.15.45:60154) 2024-04-16T16:19:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.413+0000 7f7381400640 1 -- 172.21.15.45:0/4163561652 learned_addr learned my addr 172.21.15.45:0/4163561652 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.413+0000 7f7381e00640 1 --2- 172.21.15.45:0/4163561652 >> v2:172.21.15.45:3300/0 conn(0x7f737c0d52d0 0x7f737c06ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:46.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.413+0000 7f7380a00640 1 --2- 172.21.15.45:0/4163561652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f737c162270 0x7f737c0d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:46.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.413+0000 7f737be00640 1 -- 172.21.15.45:0/4163561652 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4181366840 0 0) 0x7f737c0d3080 con 0x7f737c161e90 2024-04-16T16:19:46.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.413+0000 7f7381e00640 1 -- 172.21.15.45:0/4163561652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f737c162270 msgr2=0x7f737c0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:46.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.413+0000 7f7381e00640 1 --2- 172.21.15.45:0/4163561652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f737c162270 0x7f737c0d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.413+0000 7f7381e00640 1 -- 172.21.15.45:0/4163561652 >> v1:172.21.15.45:6789/0 conn(0x7f737c161e90 legacy=0x7f737c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:46.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.413+0000 7f7381e00640 1 -- 172.21.15.45:0/4163561652 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f737c0d3640 con 0x7f737c0d52d0 2024-04-16T16:19:46.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.414+0000 7f7381e00640 1 --2- 172.21.15.45:0/4163561652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f737c0d52d0 0x7f737c06ce40 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f7374009d90 tx=0x7f737402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0861722249cdeae server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:46.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.415+0000 7f737be00640 1 -- 172.21.15.45:0/4163561652 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7374039070 con 0x7f737c0d52d0 2024-04-16T16:19:46.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.415+0000 7f737be00640 1 -- 172.21.15.45:0/4163561652 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f737402cce0 con 0x7f737c0d52d0 2024-04-16T16:19:46.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.415+0000 7f737be00640 1 -- 172.21.15.45:0/4163561652 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7374034660 con 0x7f737c0d52d0 2024-04-16T16:19:46.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.415+0000 7f7382800640 1 -- 172.21.15.45:0/4163561652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f737c0d52d0 msgr2=0x7f737c06ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:46.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.415+0000 7f7382800640 1 --2- 172.21.15.45:0/4163561652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f737c0d52d0 0x7f737c06ce40 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f7374009d90 tx=0x7f737402bf30 comp rx=0 tx=0).stop 2024-04-16T16:19:46.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.415+0000 7f7382800640 1 -- 172.21.15.45:0/4163561652 shutdown_connections 2024-04-16T16:19:46.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.415+0000 7f7382800640 1 --2- 172.21.15.45:0/4163561652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f737c0d52d0 0x7f737c06ce40 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.415+0000 7f7382800640 1 --2- 172.21.15.45:0/4163561652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f737c162270 0x7f737c0d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.415+0000 7f7382800640 1 -- 172.21.15.45:0/4163561652 >> 172.21.15.45:0/4163561652 conn(0x7f737c0d1940 msgr2=0x7f737c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:46.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.416+0000 7f7382800640 1 -- 172.21.15.45:0/4163561652 shutdown_connections 2024-04-16T16:19:46.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.416+0000 7f7382800640 1 -- 172.21.15.45:0/4163561652 wait complete. 2024-04-16T16:19:46.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.417+0000 7f7382800640 1 Processor -- start 2024-04-16T16:19:46.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.418+0000 7f7382800640 1 -- start start 2024-04-16T16:19:46.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.419+0000 7f7382800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f737c0d52d0 0x7f737c158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:46.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.419+0000 7f7382800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f737c161e90 0x7f737c15a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:46.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.419+0000 7f7382800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f737c162270 0x7f737c0abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:46.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.419+0000 7f7382800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f737c0d4500 con 0x7f737c161e90 2024-04-16T16:19:46.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.419+0000 7f7382800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f737c0d4380 con 0x7f737c162270 2024-04-16T16:19:46.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.419+0000 7f7382800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f737c0d4680 con 0x7f737c0d52d0 2024-04-16T16:19:46.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.419+0000 7f7380a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f737c161e90 0x7f737c15a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:46.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.419+0000 7f7380a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f737c161e90 0x7f737c15a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52002/0 (socket says 172.21.15.45:52002) 2024-04-16T16:19:46.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.419+0000 7f7380a00640 1 -- 172.21.15.45:0/943081298 learned_addr learned my addr 172.21.15.45:0/943081298 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:46.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.419+0000 7f7381400640 1 --2- 172.21.15.45:0/943081298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f737c0d52d0 0x7f737c158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:46.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.419+0000 7f7381e00640 1 --2- 172.21.15.45:0/943081298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f737c162270 0x7f737c0abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:46.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.420+0000 7f7380a00640 1 -- 172.21.15.45:0/943081298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f737c0d52d0 msgr2=0x7f737c158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:46.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.420+0000 7f7380a00640 1 --2- 172.21.15.45:0/943081298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f737c0d52d0 0x7f737c158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.420+0000 7f7380a00640 1 -- 172.21.15.45:0/943081298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f737c162270 msgr2=0x7f737c0abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:46.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.420+0000 7f7380a00640 1 --2- 172.21.15.45:0/943081298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f737c162270 0x7f737c0abd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.420+0000 7f7380a00640 1 -- 172.21.15.45:0/943081298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f737c0cb960 con 0x7f737c161e90 2024-04-16T16:19:46.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.420+0000 7f7380a00640 1 --2- 172.21.15.45:0/943081298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f737c161e90 0x7f737c15a150 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f736800d620 tx=0x7f736800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:46.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.420+0000 7f7379600640 1 -- 172.21.15.45:0/943081298 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736800b620 con 0x7f737c161e90 2024-04-16T16:19:46.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.420+0000 7f7379600640 1 -- 172.21.15.45:0/943081298 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7368019070 con 0x7f737c161e90 2024-04-16T16:19:46.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.420+0000 7f7379600640 1 -- 172.21.15.45:0/943081298 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736800ba00 con 0x7f737c161e90 2024-04-16T16:19:46.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.421+0000 7f7382800640 1 -- 172.21.15.45:0/943081298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f737c0cbc20 con 0x7f737c161e90 2024-04-16T16:19:46.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.421+0000 7f7382800640 1 -- 172.21.15.45:0/943081298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f737c0cc1e0 con 0x7f737c161e90 2024-04-16T16:19:46.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.423+0000 7f7382800640 1 -- 172.21.15.45:0/943081298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f737c0b8680 con 0x7f737c161e90 2024-04-16T16:19:46.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.424+0000 7f7379600640 1 -- 172.21.15.45:0/943081298 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f736800bba0 con 0x7f737c161e90 2024-04-16T16:19:46.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.425+0000 7f7379600640 1 --2- 172.21.15.45:0/943081298 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f735c06c9b0 0x7f735c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:46.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.425+0000 7f7379600640 1 -- 172.21.15.45:0/943081298 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 4857+0+0 (secure 0 0 0) 0x7f736808bcd0 con 0x7f737c161e90 2024-04-16T16:19:46.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.426+0000 7f7381400640 1 --2- 172.21.15.45:0/943081298 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f735c06c9b0 0x7f735c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:46.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.427+0000 7f7381400640 1 --2- 172.21.15.45:0/943081298 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f735c06c9b0 0x7f735c06ee70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7370000f60 tx=0x7f7370007b50 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:46.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.430+0000 7f7379600640 1 -- 172.21.15.45:0/943081298 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f737c0b8680 con 0x7f737c161e90 2024-04-16T16:19:46.634 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:46.634+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:46.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.645+0000 7f7382800640 1 -- 172.21.15.45:0/943081298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f737c06dae0 con 0x7f737c161e90 2024-04-16T16:19:46.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.646+0000 7f7379600640 1 -- 172.21.15.45:0/943081298 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+12893 (secure 0 0 0) 0x7f736805a360 con 0x7f737c161e90 2024-04-16T16:19:46.645 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:46.645 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":110,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:19:45.605546+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:19:46.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.649+0000 7f7382800640 1 -- 172.21.15.45:0/943081298 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f735c06c9b0 msgr2=0x7f735c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:46.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.649+0000 7f7382800640 1 --2- 172.21.15.45:0/943081298 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f735c06c9b0 0x7f735c06ee70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7370000f60 tx=0x7f7370007b50 comp rx=0 tx=0).stop 2024-04-16T16:19:46.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.649+0000 7f7382800640 1 -- 172.21.15.45:0/943081298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f737c161e90 msgr2=0x7f737c15a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:46.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.649+0000 7f7382800640 1 --2- 172.21.15.45:0/943081298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f737c161e90 0x7f737c15a150 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f736800d620 tx=0x7f736800dae0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.649+0000 7f7382800640 1 -- 172.21.15.45:0/943081298 shutdown_connections 2024-04-16T16:19:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.649+0000 7f7382800640 1 --2- 172.21.15.45:0/943081298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f737c162270 0x7f737c0abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.649+0000 7f7382800640 1 --2- 172.21.15.45:0/943081298 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f735c06c9b0 0x7f735c06ee70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.649+0000 7f7382800640 1 --2- 172.21.15.45:0/943081298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f737c161e90 0x7f737c15a150 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.649+0000 7f7382800640 1 --2- 172.21.15.45:0/943081298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f737c0d52d0 0x7f737c158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.649+0000 7f7382800640 1 -- 172.21.15.45:0/943081298 >> 172.21.15.45:0/943081298 conn(0x7f737c0d1940 msgr2=0x7f737c06ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.649+0000 7f7382800640 1 -- 172.21.15.45:0/943081298 shutdown_connections 2024-04-16T16:19:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.649+0000 7f7382800640 1 -- 172.21.15.45:0/943081298 wait complete. 2024-04-16T16:19:46.669 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:19:46.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:19:46.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.897+0000 7f2a4f200640 1 Processor -- start 2024-04-16T16:19:46.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.898+0000 7f2a4f200640 1 -- start start 2024-04-16T16:19:46.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.899+0000 7f2a4f200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2a500d4370 0x7f2a500cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:46.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.899+0000 7f2a4f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2a500cfa70 0x7f2a50162560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:46.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.899+0000 7f2a4f200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a5015ee50 con 0x7f2a5015ea70 2024-04-16T16:19:46.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.899+0000 7f2a4f200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2a500d1a10 con 0x7f2a500d4370 2024-04-16T16:19:46.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.899+0000 7f2a4f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2a500b6480 con 0x7f2a500cfa70 2024-04-16T16:19:46.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.900+0000 7f2a4d400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2a500d4370 0x7f2a500cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:46.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.900+0000 7f2a4de00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2a5015ea70 0x7f2a500d11a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60168/0 (socket says 172.21.15.45:60168) 2024-04-16T16:19:46.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.900+0000 7f2a4d400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2a500d4370 0x7f2a500cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52012/0 (socket says 172.21.15.45:52012) 2024-04-16T16:19:46.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.900+0000 7f2a4de00640 1 -- 172.21.15.45:0/2831911325 learned_addr learned my addr 172.21.15.45:0/2831911325 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:46.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.900+0000 7f2a4d400640 1 -- 172.21.15.45:0/2831911325 >> v2:172.21.15.103:3300/0 conn(0x7f2a500cfa70 msgr2=0x7f2a50162560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:46.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.900+0000 7f2a4d400640 1 --2- 172.21.15.45:0/2831911325 >> v2:172.21.15.103:3300/0 conn(0x7f2a500cfa70 0x7f2a50162560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.900+0000 7f2a4d400640 1 -- 172.21.15.45:0/2831911325 >> v1:172.21.15.45:6789/0 conn(0x7f2a5015ea70 legacy=0x7f2a500d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:46.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.900+0000 7f2a4d400640 1 -- 172.21.15.45:0/2831911325 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a500b6760 con 0x7f2a500d4370 2024-04-16T16:19:46.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.900+0000 7f2a4ca00640 1 -- 172.21.15.45:0/2831911325 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 940740453 0 0) 0x7f2a5015ee50 con 0x7f2a5015ea70 2024-04-16T16:19:46.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.902+0000 7f2a4d400640 1 --2- 172.21.15.45:0/2831911325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a500d4370 0x7f2a500cf530 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f2a5008d5d0 tx=0x7f2a3802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce1671380e6fc612 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:46.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.902+0000 7f2a4ca00640 1 -- 172.21.15.45:0/2831911325 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a38039070 con 0x7f2a500d4370 2024-04-16T16:19:46.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.902+0000 7f2a4ca00640 1 -- 172.21.15.45:0/2831911325 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a3802cc80 con 0x7f2a500d4370 2024-04-16T16:19:46.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.902+0000 7f2a4ca00640 1 -- 172.21.15.45:0/2831911325 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a38034660 con 0x7f2a500d4370 2024-04-16T16:19:46.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.903+0000 7f2a4f200640 1 -- 172.21.15.45:0/2831911325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a500d4370 msgr2=0x7f2a500cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:46.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.903+0000 7f2a4f200640 1 --2- 172.21.15.45:0/2831911325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a500d4370 0x7f2a500cf530 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f2a5008d5d0 tx=0x7f2a3802bed0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.903+0000 7f2a4f200640 1 -- 172.21.15.45:0/2831911325 shutdown_connections 2024-04-16T16:19:46.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.903+0000 7f2a4f200640 1 --2- 172.21.15.45:0/2831911325 >> v2:172.21.15.103:3300/0 conn(0x7f2a500cfa70 0x7f2a50162560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.903+0000 7f2a4f200640 1 --2- 172.21.15.45:0/2831911325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a500d4370 0x7f2a500cf530 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.903+0000 7f2a4f200640 1 -- 172.21.15.45:0/2831911325 >> 172.21.15.45:0/2831911325 conn(0x7f2a5006d3b0 msgr2=0x7f2a50073060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:46.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.903+0000 7f2a4f200640 1 -- 172.21.15.45:0/2831911325 shutdown_connections 2024-04-16T16:19:46.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.903+0000 7f2a4f200640 1 -- 172.21.15.45:0/2831911325 wait complete. 2024-04-16T16:19:46.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.905+0000 7f2a4f200640 1 Processor -- start 2024-04-16T16:19:46.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.905+0000 7f2a4f200640 1 -- start start 2024-04-16T16:19:46.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.906+0000 7f2a4f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a500cfa70 0x7f2a500cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:46.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.906+0000 7f2a4f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a500d4370 0x7f2a501f4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:46.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.906+0000 7f2a4f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a5015ea70 0x7f2a50160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:46.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.906+0000 7f2a4f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2a500b6f60 con 0x7f2a5015ea70 2024-04-16T16:19:46.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.906+0000 7f2a4f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2a50162aa0 con 0x7f2a500cfa70 2024-04-16T16:19:46.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.906+0000 7f2a4f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2a500cb8c0 con 0x7f2a500d4370 2024-04-16T16:19:46.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.906+0000 7f2a4e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a5015ea70 0x7f2a50160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:46.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.906+0000 7f2a4e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a5015ea70 0x7f2a50160d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52020/0 (socket says 172.21.15.45:52020) 2024-04-16T16:19:46.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.906+0000 7f2a4e800640 1 -- 172.21.15.45:0/1906047407 learned_addr learned my addr 172.21.15.45:0/1906047407 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:46.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.907+0000 7f2a4de00640 1 --2- 172.21.15.45:0/1906047407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a500cfa70 0x7f2a500cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:46.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.907+0000 7f2a4d400640 1 --2- 172.21.15.45:0/1906047407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a500d4370 0x7f2a501f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.907+0000 7f2a4e800640 1 -- 172.21.15.45:0/1906047407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a500d4370 msgr2=0x7f2a501f4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.907+0000 7f2a4e800640 1 --2- 172.21.15.45:0/1906047407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a500d4370 0x7f2a501f4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.907+0000 7f2a4e800640 1 -- 172.21.15.45:0/1906047407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a500cfa70 msgr2=0x7f2a500cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.907+0000 7f2a4e800640 1 --2- 172.21.15.45:0/1906047407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a500cfa70 0x7f2a500cdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.907+0000 7f2a4e800640 1 -- 172.21.15.45:0/1906047407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a501f4f40 con 0x7f2a5015ea70 2024-04-16T16:19:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.907+0000 7f2a4d400640 1 --2- 172.21.15.45:0/1906047407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a500d4370 0x7f2a501f4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:46.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.907+0000 7f2a4e800640 1 --2- 172.21.15.45:0/1906047407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a5015ea70 0x7f2a50160d20 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009fd0 tx=0x7f2a4800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:46.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.907+0000 7f2a46000640 1 -- 172.21.15.45:0/1906047407 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a48011830 con 0x7f2a5015ea70 2024-04-16T16:19:46.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.908+0000 7f2a46000640 1 -- 172.21.15.45:0/1906047407 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a48016070 con 0x7f2a5015ea70 2024-04-16T16:19:46.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.908+0000 7f2a46000640 1 -- 172.21.15.45:0/1906047407 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a48011bf0 con 0x7f2a5015ea70 2024-04-16T16:19:46.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.908+0000 7f2a4f200640 1 -- 172.21.15.45:0/1906047407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a501f5220 con 0x7f2a5015ea70 2024-04-16T16:19:46.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.908+0000 7f2a4f200640 1 -- 172.21.15.45:0/1906047407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a500cbb60 con 0x7f2a5015ea70 2024-04-16T16:19:46.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.909+0000 7f2a4f200640 1 -- 172.21.15.45:0/1906047407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a50075780 con 0x7f2a5015ea70 2024-04-16T16:19:46.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.910+0000 7f2a46000640 1 -- 172.21.15.45:0/1906047407 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f2a4801b460 con 0x7f2a5015ea70 2024-04-16T16:19:46.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.911+0000 7f2a46000640 1 --2- 172.21.15.45:0/1906047407 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f2a2c06c980 0x7f2a2c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:46.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.912+0000 7f2a46000640 1 -- 172.21.15.45:0/1906047407 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 4857+0+0 (secure 0 0 0) 0x7f2a4808b6d0 con 0x7f2a5015ea70 2024-04-16T16:19:46.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.912+0000 7f2a4de00640 1 --2- 172.21.15.45:0/1906047407 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f2a2c06c980 0x7f2a2c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:46.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.913+0000 7f2a4de00640 1 --2- 172.21.15.45:0/1906047407 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f2a2c06c980 0x7f2a2c06ee40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2a40000f60 tx=0x7f2a40007c00 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:46.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:46.916+0000 7f2a46000640 1 -- 172.21.15.45:0/1906047407 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2a48059d60 con 0x7f2a5015ea70 2024-04-16T16:19:47.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.130+0000 7f2a4f200640 1 -- 172.21.15.45:0/1906047407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2a500b61e0 con 0x7f2a5015ea70 2024-04-16T16:19:47.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.131+0000 7f2a46000640 1 -- 172.21.15.45:0/1906047407 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) ==== 93+0+10 (secure 0 0 0) 0x7f2a48059660 con 0x7f2a5015ea70 2024-04-16T16:19:47.130 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:19:47.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.134+0000 7f2a4f200640 1 -- 172.21.15.45:0/1906047407 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f2a2c06c980 msgr2=0x7f2a2c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:47.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.134+0000 7f2a4f200640 1 --2- 172.21.15.45:0/1906047407 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f2a2c06c980 0x7f2a2c06ee40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2a40000f60 tx=0x7f2a40007c00 comp rx=0 tx=0).stop 2024-04-16T16:19:47.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.134+0000 7f2a4f200640 1 -- 172.21.15.45:0/1906047407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a5015ea70 msgr2=0x7f2a50160d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:47.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.134+0000 7f2a4f200640 1 --2- 172.21.15.45:0/1906047407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a5015ea70 0x7f2a50160d20 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009fd0 tx=0x7f2a4800b8e0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.134+0000 7f2a4f200640 1 -- 172.21.15.45:0/1906047407 shutdown_connections 2024-04-16T16:19:47.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.134+0000 7f2a4f200640 1 --2- 172.21.15.45:0/1906047407 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f2a2c06c980 0x7f2a2c06ee40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.134+0000 7f2a4f200640 1 --2- 172.21.15.45:0/1906047407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a5015ea70 0x7f2a50160d20 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.134+0000 7f2a4f200640 1 --2- 172.21.15.45:0/1906047407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a500d4370 0x7f2a501f4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.134+0000 7f2a4f200640 1 --2- 172.21.15.45:0/1906047407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a500cfa70 0x7f2a500cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.134+0000 7f2a4f200640 1 -- 172.21.15.45:0/1906047407 >> 172.21.15.45:0/1906047407 conn(0x7f2a5006d3b0 msgr2=0x7f2a5015da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:47.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.135+0000 7f2a4f200640 1 -- 172.21.15.45:0/1906047407 shutdown_connections 2024-04-16T16:19:47.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.135+0000 7f2a4f200640 1 -- 172.21.15.45:0/1906047407 wait complete. 2024-04-16T16:19:47.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-16T16:19:47.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.372+0000 7f7b8c000640 1 Processor -- start 2024-04-16T16:19:47.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.372+0000 7f7b8c000640 1 -- start start 2024-04-16T16:19:47.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.374+0000 7f7b8c000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b84141540 0x7f7b840b4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:47.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.374+0000 7f7b8c000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7b84073320 0x7f7b8415bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:47.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.374+0000 7f7b8c000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b8406e870 con 0x7f7b8406e2b0 2024-04-16T16:19:47.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.374+0000 7f7b8c000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7b84070240 con 0x7f7b84073320 2024-04-16T16:19:47.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.374+0000 7f7b8c000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7b8406eb50 con 0x7f7b84141540 2024-04-16T16:19:47.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.374+0000 7f7b88c00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7b8406e2b0 0x7f7b84072c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:46540/0 (socket says 172.21.15.45:46540) 2024-04-16T16:19:47.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.374+0000 7f7b88c00640 1 -- 172.21.15.45:0/2423015499 learned_addr learned my addr 172.21.15.45:0/2423015499 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:47.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.375+0000 7f7b8a000640 1 --2- 172.21.15.45:0/2423015499 >> v2:172.21.15.103:3300/0 conn(0x7f7b84073320 0x7f7b8415bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:47.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.375+0000 7f7b89600640 1 --2- 172.21.15.45:0/2423015499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b84141540 0x7f7b840b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:47.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.375+0000 7f7b83e00640 1 -- 172.21.15.45:0/2423015499 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 442096491 0 0) 0x7f7b8406e870 con 0x7f7b8406e2b0 2024-04-16T16:19:47.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.375+0000 7f7b83e00640 1 -- 172.21.15.45:0/2423015499 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b6c003520 con 0x7f7b8406e2b0 2024-04-16T16:19:47.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.375+0000 7f7b89600640 1 -- 172.21.15.45:0/2423015499 >> v2:172.21.15.103:3300/0 conn(0x7f7b84073320 msgr2=0x7f7b8415bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:47.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.375+0000 7f7b89600640 1 --2- 172.21.15.45:0/2423015499 >> v2:172.21.15.103:3300/0 conn(0x7f7b84073320 0x7f7b8415bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.375+0000 7f7b89600640 1 -- 172.21.15.45:0/2423015499 >> v1:172.21.15.103:6789/0 conn(0x7f7b8406e2b0 legacy=0x7f7b84072c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:47.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.376+0000 7f7b89600640 1 -- 172.21.15.45:0/2423015499 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8406ee30 con 0x7f7b84141540 2024-04-16T16:19:47.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.376+0000 7f7b8a000640 1 --2- 172.21.15.45:0/2423015499 >> v2:172.21.15.103:3300/0 conn(0x7f7b84073320 0x7f7b8415bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:19:47.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.377+0000 7f7b89600640 1 --2- 172.21.15.45:0/2423015499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b84141540 0x7f7b840b4020 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7b780097a0 tx=0x7f7b7802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcb74118534cf4f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:47.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.378+0000 7f7b83e00640 1 -- 172.21.15.45:0/2423015499 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b780315a0 con 0x7f7b84141540 2024-04-16T16:19:47.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.378+0000 7f7b83e00640 1 -- 172.21.15.45:0/2423015499 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b78037070 con 0x7f7b84141540 2024-04-16T16:19:47.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.378+0000 7f7b83e00640 1 -- 172.21.15.45:0/2423015499 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78031940 con 0x7f7b84141540 2024-04-16T16:19:47.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.379+0000 7f7b8c000640 1 -- 172.21.15.45:0/2423015499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b84141540 msgr2=0x7f7b840b4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:47.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.379+0000 7f7b8c000640 1 --2- 172.21.15.45:0/2423015499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b84141540 0x7f7b840b4020 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7b780097a0 tx=0x7f7b7802b840 comp rx=0 tx=0).stop 2024-04-16T16:19:47.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.380+0000 7f7b8c000640 1 -- 172.21.15.45:0/2423015499 shutdown_connections 2024-04-16T16:19:47.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.380+0000 7f7b8c000640 1 --2- 172.21.15.45:0/2423015499 >> v2:172.21.15.103:3300/0 conn(0x7f7b84073320 0x7f7b8415bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.380+0000 7f7b8c000640 1 --2- 172.21.15.45:0/2423015499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b84141540 0x7f7b840b4020 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.380+0000 7f7b8c000640 1 -- 172.21.15.45:0/2423015499 >> 172.21.15.45:0/2423015499 conn(0x7f7b840b2750 msgr2=0x7f7b840b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:47.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.380+0000 7f7b8c000640 1 -- 172.21.15.45:0/2423015499 shutdown_connections 2024-04-16T16:19:47.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.380+0000 7f7b8c000640 1 -- 172.21.15.45:0/2423015499 wait complete. 2024-04-16T16:19:47.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.382+0000 7f7b8c000640 1 Processor -- start 2024-04-16T16:19:47.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.382+0000 7f7b8c000640 1 -- start start 2024-04-16T16:19:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.384+0000 7f7b8c000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8406e2b0 0x7f7b8412ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.384+0000 7f7b8c000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b84073320 0x7f7b841532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.384+0000 7f7b8c000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b84141540 0x7f7b84166e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.384+0000 7f7b8c000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7b84072380 con 0x7f7b8406e2b0 2024-04-16T16:19:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.384+0000 7f7b8c000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7b840cf3b0 con 0x7f7b84141540 2024-04-16T16:19:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.384+0000 7f7b8c000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7b84072500 con 0x7f7b84073320 2024-04-16T16:19:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.384+0000 7f7b89600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8406e2b0 0x7f7b8412ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.384+0000 7f7b89600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8406e2b0 0x7f7b8412ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52022/0 (socket says 172.21.15.45:52022) 2024-04-16T16:19:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.384+0000 7f7b89600640 1 -- 172.21.15.45:0/23186301 learned_addr learned my addr 172.21.15.45:0/23186301 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:47.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.384+0000 7f7b8a000640 1 --2- 172.21.15.45:0/23186301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b84141540 0x7f7b84166e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:47.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.384+0000 7f7b88c00640 1 --2- 172.21.15.45:0/23186301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b84073320 0x7f7b841532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:47.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.385+0000 7f7b89600640 1 -- 172.21.15.45:0/23186301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b84073320 msgr2=0x7f7b841532b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:47.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.385+0000 7f7b89600640 1 --2- 172.21.15.45:0/23186301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b84073320 0x7f7b841532b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.385+0000 7f7b89600640 1 -- 172.21.15.45:0/23186301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b84141540 msgr2=0x7f7b84166e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:47.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.385+0000 7f7b89600640 1 --2- 172.21.15.45:0/23186301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b84141540 0x7f7b84166e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.385+0000 7f7b89600640 1 -- 172.21.15.45:0/23186301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b84138990 con 0x7f7b8406e2b0 2024-04-16T16:19:47.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.385+0000 7f7b88c00640 1 --2- 172.21.15.45:0/23186301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b84073320 0x7f7b841532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:47.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.385+0000 7f7b89600640 1 --2- 172.21.15.45:0/23186301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8406e2b0 0x7f7b8412ff90 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f7b78000c00 tx=0x7f7b780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:47.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.385+0000 7f7b81600640 1 -- 172.21.15.45:0/23186301 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78032630 con 0x7f7b8406e2b0 2024-04-16T16:19:47.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.385+0000 7f7b81600640 1 -- 172.21.15.45:0/23186301 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b780327d0 con 0x7f7b8406e2b0 2024-04-16T16:19:47.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.385+0000 7f7b81600640 1 -- 172.21.15.45:0/23186301 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78032b80 con 0x7f7b8406e2b0 2024-04-16T16:19:47.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.385+0000 7f7b8c000640 1 -- 172.21.15.45:0/23186301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b841547e0 con 0x7f7b8406e2b0 2024-04-16T16:19:47.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.385+0000 7f7b8c000640 1 -- 172.21.15.45:0/23186301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b841304d0 con 0x7f7b8406e2b0 2024-04-16T16:19:47.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.387+0000 7f7b8c000640 1 -- 172.21.15.45:0/23186301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b84075780 con 0x7f7b8406e2b0 2024-04-16T16:19:47.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.388+0000 7f7b81600640 1 -- 172.21.15.45:0/23186301 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f7b7803b3f0 con 0x7f7b8406e2b0 2024-04-16T16:19:47.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.388+0000 7f7b81600640 1 --2- 172.21.15.45:0/23186301 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f7b6406c980 0x7f7b6406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:47.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.389+0000 7f7b81600640 1 -- 172.21.15.45:0/23186301 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 4857+0+0 (secure 0 0 0) 0x7f7b7802f030 con 0x7f7b8406e2b0 2024-04-16T16:19:47.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.389+0000 7f7b88c00640 1 --2- 172.21.15.45:0/23186301 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f7b6406c980 0x7f7b6406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:47.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.390+0000 7f7b88c00640 1 --2- 172.21.15.45:0/23186301 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f7b6406c980 0x7f7b6406ee40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7b70005880 tx=0x7f7b70009040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:47.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.393+0000 7f7b81600640 1 -- 172.21.15.45:0/23186301 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7b78040070 con 0x7f7b8406e2b0 2024-04-16T16:19:47.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.608+0000 7f7b8c000640 1 -- 172.21.15.45:0/23186301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f7b8406fcf0 con 0x7f7b8406e2b0 2024-04-16T16:19:47.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.609+0000 7f7b81600640 1 -- 172.21.15.45:0/23186301 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v110) ==== 84+0+65 (secure 0 0 0) 0x7f7b7807bcd0 con 0x7f7b8406e2b0 2024-04-16T16:19:47.608 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:47.608 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2024-04-16T16:19:47.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:19:47.609 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2024-04-16T16:19:47.609 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2024-04-16T16:19:47.609 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:19:47.609 INFO:teuthology.orchestra.run.smithi045.stdout:] 2024-04-16T16:19:47.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.612+0000 7f7b8c000640 1 -- 172.21.15.45:0/23186301 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f7b6406c980 msgr2=0x7f7b6406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:47.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.612+0000 7f7b8c000640 1 --2- 172.21.15.45:0/23186301 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f7b6406c980 0x7f7b6406ee40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7b70005880 tx=0x7f7b70009040 comp rx=0 tx=0).stop 2024-04-16T16:19:47.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.612+0000 7f7b8c000640 1 -- 172.21.15.45:0/23186301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8406e2b0 msgr2=0x7f7b8412ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:47.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.612+0000 7f7b8c000640 1 --2- 172.21.15.45:0/23186301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8406e2b0 0x7f7b8412ff90 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f7b78000c00 tx=0x7f7b780046c0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.612+0000 7f7b8c000640 1 -- 172.21.15.45:0/23186301 shutdown_connections 2024-04-16T16:19:47.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.612+0000 7f7b8c000640 1 --2- 172.21.15.45:0/23186301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b84141540 0x7f7b84166e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.612+0000 7f7b8c000640 1 --2- 172.21.15.45:0/23186301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b84073320 0x7f7b841532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.612+0000 7f7b8c000640 1 --2- 172.21.15.45:0/23186301 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f7b6406c980 0x7f7b6406ee40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.612+0000 7f7b8c000640 1 --2- 172.21.15.45:0/23186301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8406e2b0 0x7f7b8412ff90 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.612+0000 7f7b8c000640 1 -- 172.21.15.45:0/23186301 >> 172.21.15.45:0/23186301 conn(0x7f7b840b2750 msgr2=0x7f7b840b1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:47.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.612+0000 7f7b8c000640 1 -- 172.21.15.45:0/23186301 shutdown_connections 2024-04-16T16:19:47.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.612+0000 7f7b8c000640 1 -- 172.21.15.45:0/23186301 wait complete. 2024-04-16T16:19:47.631 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-16T16:19:47.631 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-16T16:19:47.816 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:47.815+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:47.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.865+0000 7fa681a00640 1 Processor -- start 2024-04-16T16:19:47.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.865+0000 7fa681a00640 1 -- start start 2024-04-16T16:19:47.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.867+0000 7fa681a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa67c155a40 0x7fa67c06b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:47.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.867+0000 7fa681a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa67c070cd0 0x7fa67c06be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:47.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.867+0000 7fa681a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa67c0b0cd0 0x7fa67c15f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:47.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.867+0000 7fa681a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa67c0d23e0 con 0x7fa67c0b0cd0 2024-04-16T16:19:47.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.867+0000 7fa681a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa67c0b4550 con 0x7fa67c155a40 2024-04-16T16:19:47.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.867+0000 7fa681a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa67c155e20 con 0x7fa67c070cd0 2024-04-16T16:19:47.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.867+0000 7fa681000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa67c0b0cd0 0x7fa67c15f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:47.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.868+0000 7fa681000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa67c0b0cd0 0x7fa67c15f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52024/0 (socket says 172.21.15.45:52024) 2024-04-16T16:19:47.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.868+0000 7fa681000640 1 -- 172.21.15.45:0/3570746558 learned_addr learned my addr 172.21.15.45:0/3570746558 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:47.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.868+0000 7fa673e00640 1 --2- 172.21.15.45:0/3570746558 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa67c070cd0 0x7fa67c06be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:47.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.868+0000 7fa67be00640 1 --2- 172.21.15.45:0/3570746558 >> v2:172.21.15.103:3300/0 conn(0x7fa67c155a40 0x7fa67c06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:47.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.868+0000 7fa681000640 1 -- 172.21.15.45:0/3570746558 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa67c070cd0 msgr2=0x7fa67c06be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:47.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.868+0000 7fa681000640 1 --2- 172.21.15.45:0/3570746558 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa67c070cd0 0x7fa67c06be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.868+0000 7fa681000640 1 -- 172.21.15.45:0/3570746558 >> v2:172.21.15.103:3300/0 conn(0x7fa67c155a40 msgr2=0x7fa67c06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:47.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.868+0000 7fa681000640 1 --2- 172.21.15.45:0/3570746558 >> v2:172.21.15.103:3300/0 conn(0x7fa67c155a40 0x7fa67c06b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.868+0000 7fa681000640 1 -- 172.21.15.45:0/3570746558 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa67c0b7650 con 0x7fa67c0b0cd0 2024-04-16T16:19:47.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.869+0000 7fa681000640 1 --2- 172.21.15.45:0/3570746558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa67c0b0cd0 0x7fa67c15f190 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fa67c073f60 tx=0x7fa67402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baee9b3322670453 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:47.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.870+0000 7fa67b400640 1 -- 172.21.15.45:0/3570746558 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67402ca70 con 0x7fa67c0b0cd0 2024-04-16T16:19:47.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.870+0000 7fa67b400640 1 -- 172.21.15.45:0/3570746558 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa67402cc10 con 0x7fa67c0b0cd0 2024-04-16T16:19:47.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.870+0000 7fa67b400640 1 -- 172.21.15.45:0/3570746558 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6740343f0 con 0x7fa67c0b0cd0 2024-04-16T16:19:47.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.871+0000 7fa681a00640 1 -- 172.21.15.45:0/3570746558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa67c0b0cd0 msgr2=0x7fa67c15f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:47.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.871+0000 7fa681a00640 1 --2- 172.21.15.45:0/3570746558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa67c0b0cd0 0x7fa67c15f190 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fa67c073f60 tx=0x7fa67402bed0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.871+0000 7fa681a00640 1 -- 172.21.15.45:0/3570746558 shutdown_connections 2024-04-16T16:19:47.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.871+0000 7fa681a00640 1 --2- 172.21.15.45:0/3570746558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa67c0b0cd0 0x7fa67c15f190 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.871+0000 7fa681a00640 1 --2- 172.21.15.45:0/3570746558 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa67c070cd0 0x7fa67c06be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.871+0000 7fa681a00640 1 --2- 172.21.15.45:0/3570746558 >> v2:172.21.15.103:3300/0 conn(0x7fa67c155a40 0x7fa67c06b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.871+0000 7fa681a00640 1 -- 172.21.15.45:0/3570746558 >> 172.21.15.45:0/3570746558 conn(0x7fa67c09aa70 msgr2=0x7fa67c0b6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:47.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.871+0000 7fa681a00640 1 -- 172.21.15.45:0/3570746558 shutdown_connections 2024-04-16T16:19:47.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.871+0000 7fa681a00640 1 -- 172.21.15.45:0/3570746558 wait complete. 2024-04-16T16:19:47.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.873+0000 7fa681a00640 1 Processor -- start 2024-04-16T16:19:47.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.873+0000 7fa681a00640 1 -- start start 2024-04-16T16:19:47.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.874+0000 7fa681a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa67c070cd0 0x7fa67c0d1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:47.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.874+0000 7fa681a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa67c0b0cd0 0x7fa67c0cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:47.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.874+0000 7fa681a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa67c155a40 0x7fa67c09e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:47.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.874+0000 7fa681a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c0b6440 con 0x7fa67c0b0cd0 2024-04-16T16:19:47.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.874+0000 7fa681a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c0b61a0 con 0x7fa67c070cd0 2024-04-16T16:19:47.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.874+0000 7fa681a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa67c0b6720 con 0x7fa67c155a40 2024-04-16T16:19:47.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.874+0000 7fa673e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa67c0b0cd0 0x7fa67c0cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:47.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.874+0000 7fa673e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa67c0b0cd0 0x7fa67c0cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52040/0 (socket says 172.21.15.45:52040) 2024-04-16T16:19:47.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.874+0000 7fa673e00640 1 -- 172.21.15.45:0/2362910769 learned_addr learned my addr 172.21.15.45:0/2362910769 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:47.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.875+0000 7fa67be00640 1 --2- 172.21.15.45:0/2362910769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa67c070cd0 0x7fa67c0d1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:47.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.875+0000 7fa673e00640 1 -- 172.21.15.45:0/2362910769 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa67c155a40 msgr2=0x7fa67c09e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:47.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.875+0000 7fa673e00640 1 --2- 172.21.15.45:0/2362910769 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa67c155a40 0x7fa67c09e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.875+0000 7fa673e00640 1 -- 172.21.15.45:0/2362910769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa67c070cd0 msgr2=0x7fa67c0d1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:47.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.875+0000 7fa673e00640 1 --2- 172.21.15.45:0/2362910769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa67c070cd0 0x7fa67c0d1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:47.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.875+0000 7fa673e00640 1 -- 172.21.15.45:0/2362910769 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa67c0cffb0 con 0x7fa67c0b0cd0 2024-04-16T16:19:47.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.875+0000 7fa673e00640 1 --2- 172.21.15.45:0/2362910769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa67c0b0cd0 0x7fa67c0cdc50 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fa66400d830 tx=0x7fa66400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:47.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.875+0000 7fa678c00640 1 -- 172.21.15.45:0/2362910769 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa664018020 con 0x7fa67c0b0cd0 2024-04-16T16:19:47.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.875+0000 7fa678c00640 1 -- 172.21.15.45:0/2362910769 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6640096a0 con 0x7fa67c0b0cd0 2024-04-16T16:19:47.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.875+0000 7fa678c00640 1 -- 172.21.15.45:0/2362910769 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa664009a70 con 0x7fa67c0b0cd0 2024-04-16T16:19:47.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.876+0000 7fa681a00640 1 -- 172.21.15.45:0/2362910769 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa67c0d0290 con 0x7fa67c0b0cd0 2024-04-16T16:19:47.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.876+0000 7fa681a00640 1 -- 172.21.15.45:0/2362910769 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa67c0d0b30 con 0x7fa67c0b0cd0 2024-04-16T16:19:47.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.877+0000 7fa681a00640 1 -- 172.21.15.45:0/2362910769 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa67c0d0460 con 0x7fa67c0b0cd0 2024-04-16T16:19:47.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.878+0000 7fa678c00640 1 -- 172.21.15.45:0/2362910769 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7fa66401b020 con 0x7fa67c0b0cd0 2024-04-16T16:19:47.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.879+0000 7fa678c00640 1 --2- 172.21.15.45:0/2362910769 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa65806c980 0x7fa65806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:47.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.879+0000 7fa678c00640 1 -- 172.21.15.45:0/2362910769 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 4857+0+0 (secure 0 0 0) 0x7fa66408bf70 con 0x7fa67c0b0cd0 2024-04-16T16:19:47.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.879+0000 7fa67be00640 1 --2- 172.21.15.45:0/2362910769 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa65806c980 0x7fa65806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:47.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.881+0000 7fa67be00640 1 --2- 172.21.15.45:0/2362910769 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa65806c980 0x7fa65806ee40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa66c006fd0 tx=0x7fa66c008040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:47.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:47.883+0000 7fa678c00640 1 -- 172.21.15.45:0/2362910769 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa664022020 con 0x7fa67c0b0cd0 2024-04-16T16:19:48.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.098+0000 7fa681a00640 1 -- 172.21.15.45:0/2362910769 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa67c075780 con 0x7fa67c0b0cd0 2024-04-16T16:19:48.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.621+0000 7fa678c00640 1 -- 172.21.15.45:0/2362910769 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v111) ==== 138+0+0 (secure 0 0 0) 0x7fa664009e20 con 0x7fa67c0b0cd0 2024-04-16T16:19:48.620 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2024-04-16T16:19:48.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.625+0000 7fa681a00640 1 -- 172.21.15.45:0/2362910769 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa65806c980 msgr2=0x7fa65806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:48.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.625+0000 7fa681a00640 1 --2- 172.21.15.45:0/2362910769 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa65806c980 0x7fa65806ee40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa66c006fd0 tx=0x7fa66c008040 comp rx=0 tx=0).stop 2024-04-16T16:19:48.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.625+0000 7fa681a00640 1 -- 172.21.15.45:0/2362910769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa67c0b0cd0 msgr2=0x7fa67c0cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:48.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.625+0000 7fa681a00640 1 --2- 172.21.15.45:0/2362910769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa67c0b0cd0 0x7fa67c0cdc50 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fa66400d830 tx=0x7fa66400dcf0 comp rx=0 tx=0).stop 2024-04-16T16:19:48.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.625+0000 7fa681a00640 1 -- 172.21.15.45:0/2362910769 shutdown_connections 2024-04-16T16:19:48.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.625+0000 7fa681a00640 1 --2- 172.21.15.45:0/2362910769 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa67c155a40 0x7fa67c09e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:48.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.625+0000 7fa681a00640 1 --2- 172.21.15.45:0/2362910769 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fa65806c980 0x7fa65806ee40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:48.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.625+0000 7fa681a00640 1 --2- 172.21.15.45:0/2362910769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa67c0b0cd0 0x7fa67c0cdc50 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:48.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.625+0000 7fa681a00640 1 --2- 172.21.15.45:0/2362910769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa67c070cd0 0x7fa67c0d1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:48.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.625+0000 7fa681a00640 1 -- 172.21.15.45:0/2362910769 >> 172.21.15.45:0/2362910769 conn(0x7fa67c09aa70 msgr2=0x7fa67c0b01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:48.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.626+0000 7fa681a00640 1 -- 172.21.15.45:0/2362910769 shutdown_connections 2024-04-16T16:19:48.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.626+0000 7fa681a00640 1 -- 172.21.15.45:0/2362910769 wait complete. 2024-04-16T16:19:48.654 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-16T16:19:48.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.877+0000 7f13d8400640 1 Processor -- start 2024-04-16T16:19:48.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.877+0000 7f13d8400640 1 -- start start 2024-04-16T16:19:48.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.879+0000 7f13d8400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f13d006cf10 0x7f13d006d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:48.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.879+0000 7f13d8400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f13d00d35d0 0x7f13d00d3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:48.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.879+0000 7f13d8400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d006d8a0 con 0x7f13d015ecc0 2024-04-16T16:19:48.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.879+0000 7f13d8400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f13d00d2b10 con 0x7f13d00d35d0 2024-04-16T16:19:48.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.879+0000 7f13d8400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f13d0163210 con 0x7f13d006cf10 2024-04-16T16:19:48.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.879+0000 7f13d6400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f13d00d35d0 0x7f13d00d3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:48.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.879+0000 7f13d6400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f13d00d35d0 0x7f13d00d3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52050/0 (socket says 172.21.15.45:52050) 2024-04-16T16:19:48.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.879+0000 7f13d6400640 1 -- 172.21.15.45:0/1061991895 learned_addr learned my addr 172.21.15.45:0/1061991895 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:48.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.879+0000 7f13d5000640 1 --2- 172.21.15.45:0/1061991895 >> v2:172.21.15.103:3300/0 conn(0x7f13d006cf10 0x7f13d006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:48.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.880+0000 7f13d6400640 1 -- 172.21.15.45:0/1061991895 >> v2:172.21.15.103:3300/0 conn(0x7f13d006cf10 msgr2=0x7f13d006d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:48.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.880+0000 7f13d6400640 1 --2- 172.21.15.45:0/1061991895 >> v2:172.21.15.103:3300/0 conn(0x7f13d006cf10 0x7f13d006d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:48.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.880+0000 7f13d6400640 1 -- 172.21.15.45:0/1061991895 >> v1:172.21.15.103:6789/0 conn(0x7f13d015ecc0 legacy=0x7f13d01626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:48.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.880+0000 7f13d6400640 1 -- 172.21.15.45:0/1061991895 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d01634f0 con 0x7f13d00d35d0 2024-04-16T16:19:48.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.881+0000 7f13d6400640 1 --2- 172.21.15.45:0/1061991895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13d00d35d0 0x7f13d00d3a00 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f13d006e000 tx=0x7f13c802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e2a90c4808b4b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:48.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.881+0000 7f13cfe00640 1 -- 172.21.15.45:0/1061991895 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c802cb20 con 0x7f13d00d35d0 2024-04-16T16:19:48.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.881+0000 7f13cfe00640 1 -- 172.21.15.45:0/1061991895 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13c802ccc0 con 0x7f13d00d35d0 2024-04-16T16:19:48.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.881+0000 7f13cfe00640 1 -- 172.21.15.45:0/1061991895 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c8034610 con 0x7f13d00d35d0 2024-04-16T16:19:48.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.882+0000 7f13d8400640 1 -- 172.21.15.45:0/1061991895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13d00d35d0 msgr2=0x7f13d00d3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:48.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.882+0000 7f13d8400640 1 --2- 172.21.15.45:0/1061991895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13d00d35d0 0x7f13d00d3a00 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f13d006e000 tx=0x7f13c802bf80 comp rx=0 tx=0).stop 2024-04-16T16:19:48.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.882+0000 7f13d8400640 1 -- 172.21.15.45:0/1061991895 shutdown_connections 2024-04-16T16:19:48.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.882+0000 7f13d8400640 1 --2- 172.21.15.45:0/1061991895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13d00d35d0 0x7f13d00d3a00 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:48.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.882+0000 7f13d8400640 1 --2- 172.21.15.45:0/1061991895 >> v2:172.21.15.103:3300/0 conn(0x7f13d006cf10 0x7f13d006d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:48.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.882+0000 7f13d8400640 1 -- 172.21.15.45:0/1061991895 >> 172.21.15.45:0/1061991895 conn(0x7f13d0073690 msgr2=0x7f13d00d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:48.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.882+0000 7f13d8400640 1 -- 172.21.15.45:0/1061991895 shutdown_connections 2024-04-16T16:19:48.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.882+0000 7f13d8400640 1 -- 172.21.15.45:0/1061991895 wait complete. 2024-04-16T16:19:48.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.884+0000 7f13d8400640 1 Processor -- start 2024-04-16T16:19:48.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.884+0000 7f13d8400640 1 -- start start 2024-04-16T16:19:48.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.885+0000 7f13d8400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13d006cf10 0x7f13d0150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:48.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.885+0000 7f13d8400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d00d35d0 0x7f13d015f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:48.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.885+0000 7f13d8400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13d015ecc0 0x7f13d015f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:48.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.885+0000 7f13d8400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f13d00ba540 con 0x7f13d006cf10 2024-04-16T16:19:48.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.885+0000 7f13d8400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f13d00d30b0 con 0x7f13d00d35d0 2024-04-16T16:19:48.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.885+0000 7f13d8400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f13d0162db0 con 0x7f13d015ecc0 2024-04-16T16:19:48.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.885+0000 7f13d5a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13d006cf10 0x7f13d0150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:48.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.885+0000 7f13d5a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13d006cf10 0x7f13d0150b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52058/0 (socket says 172.21.15.45:52058) 2024-04-16T16:19:48.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.885+0000 7f13d5a00640 1 -- 172.21.15.45:0/3650224634 learned_addr learned my addr 172.21.15.45:0/3650224634 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:48.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.886+0000 7f13d5000640 1 --2- 172.21.15.45:0/3650224634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d00d35d0 0x7f13d015f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:48.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.886+0000 7f13d6400640 1 --2- 172.21.15.45:0/3650224634 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13d015ecc0 0x7f13d015f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:48.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.886+0000 7f13d5a00640 1 -- 172.21.15.45:0/3650224634 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13d015ecc0 msgr2=0x7f13d015f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:48.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.886+0000 7f13d5a00640 1 --2- 172.21.15.45:0/3650224634 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13d015ecc0 0x7f13d015f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:48.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.886+0000 7f13d5a00640 1 -- 172.21.15.45:0/3650224634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d00d35d0 msgr2=0x7f13d015f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:48.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.886+0000 7f13d5a00640 1 --2- 172.21.15.45:0/3650224634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d00d35d0 0x7f13d015f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:48.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.886+0000 7f13d5a00640 1 -- 172.21.15.45:0/3650224634 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d0162090 con 0x7f13d006cf10 2024-04-16T16:19:48.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.886+0000 7f13d5a00640 1 --2- 172.21.15.45:0/3650224634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13d006cf10 0x7f13d0150b70 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f13c400b3f0 tx=0x7f13c400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:48.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.887+0000 7f13cd600640 1 -- 172.21.15.45:0/3650224634 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c4015020 con 0x7f13d006cf10 2024-04-16T16:19:48.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.887+0000 7f13cd600640 1 -- 172.21.15.45:0/3650224634 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13c4011830 con 0x7f13d006cf10 2024-04-16T16:19:48.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.887+0000 7f13cd600640 1 -- 172.21.15.45:0/3650224634 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c4011c20 con 0x7f13d006cf10 2024-04-16T16:19:48.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.887+0000 7f13d8400640 1 -- 172.21.15.45:0/3650224634 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d0161310 con 0x7f13d006cf10 2024-04-16T16:19:48.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.887+0000 7f13d8400640 1 -- 172.21.15.45:0/3650224634 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d0161890 con 0x7f13d006cf10 2024-04-16T16:19:48.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.888+0000 7f13d8400640 1 -- 172.21.15.45:0/3650224634 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13d0075780 con 0x7f13d006cf10 2024-04-16T16:19:48.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.889+0000 7f13cd600640 1 -- 172.21.15.45:0/3650224634 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f13c4018020 con 0x7f13d006cf10 2024-04-16T16:19:48.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.890+0000 7f13cd600640 1 --2- 172.21.15.45:0/3650224634 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f13b006c980 0x7f13b006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:48.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.890+0000 7f13cd600640 1 -- 172.21.15.45:0/3650224634 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5227+0+0 (secure 0 0 0) 0x7f13c408b5d0 con 0x7f13d006cf10 2024-04-16T16:19:48.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.891+0000 7f13d5000640 1 --2- 172.21.15.45:0/3650224634 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f13b006c980 0x7f13b006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:48.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.892+0000 7f13d5000640 1 --2- 172.21.15.45:0/3650224634 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f13b006c980 0x7f13b006ee40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f13d0160e90 tx=0x7f13bc008040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:48.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:48.896+0000 7f13cd600640 1 -- 172.21.15.45:0/3650224634 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f13c401f030 con 0x7f13d006cf10 2024-04-16T16:19:49.037 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:49.037+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:49.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.116+0000 7f13d8400640 1 -- 172.21.15.45:0/3650224634 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f13d00ba320 con 0x7f13d006cf10 2024-04-16T16:19:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.640+0000 7f13cd600640 1 -- 172.21.15.45:0/3650224634 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v112) ==== 170+0+0 (secure 0 0 0) 0x7f13c40172c0 con 0x7f13d006cf10 2024-04-16T16:19:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2024-04-16T16:19:49.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.645+0000 7f13d8400640 1 -- 172.21.15.45:0/3650224634 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f13b006c980 msgr2=0x7f13b006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:49.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.645+0000 7f13d8400640 1 --2- 172.21.15.45:0/3650224634 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f13b006c980 0x7f13b006ee40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f13d0160e90 tx=0x7f13bc008040 comp rx=0 tx=0).stop 2024-04-16T16:19:49.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.645+0000 7f13d8400640 1 -- 172.21.15.45:0/3650224634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13d006cf10 msgr2=0x7f13d0150b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:49.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.645+0000 7f13d8400640 1 --2- 172.21.15.45:0/3650224634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13d006cf10 0x7f13d0150b70 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f13c400b3f0 tx=0x7f13c400b8b0 comp rx=0 tx=0).stop 2024-04-16T16:19:49.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.646+0000 7f13d8400640 1 -- 172.21.15.45:0/3650224634 shutdown_connections 2024-04-16T16:19:49.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.646+0000 7f13d8400640 1 --2- 172.21.15.45:0/3650224634 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13d015ecc0 0x7f13d015f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:49.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.646+0000 7f13d8400640 1 --2- 172.21.15.45:0/3650224634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d00d35d0 0x7f13d015f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:49.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.646+0000 7f13d8400640 1 --2- 172.21.15.45:0/3650224634 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f13b006c980 0x7f13b006ee40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:49.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.646+0000 7f13d8400640 1 --2- 172.21.15.45:0/3650224634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13d006cf10 0x7f13d0150b70 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:49.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.646+0000 7f13d8400640 1 -- 172.21.15.45:0/3650224634 >> 172.21.15.45:0/3650224634 conn(0x7f13d0073690 msgr2=0x7f13d00d3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:49.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.647+0000 7f13d8400640 1 -- 172.21.15.45:0/3650224634 shutdown_connections 2024-04-16T16:19:49.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.647+0000 7f13d8400640 1 -- 172.21.15.45:0/3650224634 wait complete. 2024-04-16T16:19:49.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-16T16:19:49.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.875+0000 7f13a7e00640 1 Processor -- start 2024-04-16T16:19:49.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.875+0000 7f13a7e00640 1 -- start start 2024-04-16T16:19:49.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.876+0000 7f13a7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f13a80d2bc0 0x7f13a80ba2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:49.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.876+0000 7f13a7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13a80b7920 0x7f13a80b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:49.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.876+0000 7f13a7e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13a80d3020 con 0x7f13a80b6270 2024-04-16T16:19:49.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.876+0000 7f13a7e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f13a80ba7e0 con 0x7f13a80d2bc0 2024-04-16T16:19:49.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.876+0000 7f13a7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f13a80d32c0 con 0x7f13a80b7920 2024-04-16T16:19:49.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.876+0000 7f139e000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f13a80d2bc0 0x7f13a80ba2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:49.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.877+0000 7f139e000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f13a80d2bc0 0x7f13a80ba2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52060/0 (socket says 172.21.15.45:52060) 2024-04-16T16:19:49.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.877+0000 7f139e000640 1 -- 172.21.15.45:0/2436734172 learned_addr learned my addr 172.21.15.45:0/2436734172 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:49.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.877+0000 7f13a6a00640 1 --1- 172.21.15.45:0/2436734172 >> v1:172.21.15.45:6789/0 conn(0x7f13a80b6270 0x7f13a80b6650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60176/0 (socket says 172.21.15.45:60176) 2024-04-16T16:19:49.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.877+0000 7f13a7400640 1 --2- 172.21.15.45:0/2436734172 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13a80b7920 0x7f13a80b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:49.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.877+0000 7f139e000640 1 -- 172.21.15.45:0/2436734172 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13a80b7920 msgr2=0x7f13a80b7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:49.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.877+0000 7f139e000640 1 --2- 172.21.15.45:0/2436734172 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13a80b7920 0x7f13a80b7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:49.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.877+0000 7f139e000640 1 -- 172.21.15.45:0/2436734172 >> v1:172.21.15.45:6789/0 conn(0x7f13a80b6270 legacy=0x7f13a80b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:49.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.877+0000 7f139e000640 1 -- 172.21.15.45:0/2436734172 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a80d35a0 con 0x7f13a80d2bc0 2024-04-16T16:19:49.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.877+0000 7f13a6000640 1 -- 172.21.15.45:0/2436734172 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1707731844 0 0) 0x7f13a80d3020 con 0x7f13a80b6270 2024-04-16T16:19:49.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.877+0000 7f13a7400640 1 --2- 172.21.15.45:0/2436734172 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13a80b7920 0x7f13a80b7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:49.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.878+0000 7f139e000640 1 --2- 172.21.15.45:0/2436734172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a80d2bc0 0x7f13a80ba2a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f1390009870 tx=0x7f139002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac25c5d7972b3490 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:49.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.878+0000 7f13a6000640 1 -- 172.21.15.45:0/2436734172 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1390034070 con 0x7f13a80d2bc0 2024-04-16T16:19:49.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.878+0000 7f13a6000640 1 -- 172.21.15.45:0/2436734172 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1390030840 con 0x7f13a80d2bc0 2024-04-16T16:19:49.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.879+0000 7f13a6000640 1 -- 172.21.15.45:0/2436734172 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1390030c10 con 0x7f13a80d2bc0 2024-04-16T16:19:49.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.879+0000 7f13a7e00640 1 -- 172.21.15.45:0/2436734172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a80d2bc0 msgr2=0x7f13a80ba2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:49.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.879+0000 7f13a7e00640 1 --2- 172.21.15.45:0/2436734172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a80d2bc0 0x7f13a80ba2a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f1390009870 tx=0x7f139002b980 comp rx=0 tx=0).stop 2024-04-16T16:19:49.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.879+0000 7f13a7e00640 1 -- 172.21.15.45:0/2436734172 shutdown_connections 2024-04-16T16:19:49.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.879+0000 7f13a7e00640 1 --2- 172.21.15.45:0/2436734172 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13a80b7920 0x7f13a80b7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:49.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.879+0000 7f13a7e00640 1 --2- 172.21.15.45:0/2436734172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a80d2bc0 0x7f13a80ba2a0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:49.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.879+0000 7f13a7e00640 1 -- 172.21.15.45:0/2436734172 >> 172.21.15.45:0/2436734172 conn(0x7f13a806d300 msgr2=0x7f13a80d1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:49.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.879+0000 7f13a7e00640 1 -- 172.21.15.45:0/2436734172 shutdown_connections 2024-04-16T16:19:49.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.879+0000 7f13a7e00640 1 -- 172.21.15.45:0/2436734172 wait complete. 2024-04-16T16:19:49.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.881+0000 7f13a7e00640 1 Processor -- start 2024-04-16T16:19:49.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.881+0000 7f13a7e00640 1 -- start start 2024-04-16T16:19:49.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.882+0000 7f13a7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a80b6270 0x7f13a8150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:49.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.882+0000 7f13a7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13a80b7920 0x7f13a80c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:49.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.882+0000 7f13a7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13a80d2bc0 0x7f13a80c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:49.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.882+0000 7f13a7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f13a80d3e40 con 0x7f13a80b6270 2024-04-16T16:19:49.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.882+0000 7f13a7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f13a80d3b60 con 0x7f13a80d2bc0 2024-04-16T16:19:49.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.882+0000 7f13a7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f13a80d4360 con 0x7f13a80b7920 2024-04-16T16:19:49.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.882+0000 7f13a6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a80b6270 0x7f13a8150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:49.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.882+0000 7f13a6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a80b6270 0x7f13a8150cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52072/0 (socket says 172.21.15.45:52072) 2024-04-16T16:19:49.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.882+0000 7f13a6a00640 1 -- 172.21.15.45:0/1129507422 learned_addr learned my addr 172.21.15.45:0/1129507422 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:49.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.883+0000 7f13a7400640 1 --2- 172.21.15.45:0/1129507422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13a80d2bc0 0x7f13a80c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:49.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.883+0000 7f139e000640 1 --2- 172.21.15.45:0/1129507422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13a80b7920 0x7f13a80c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:49.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.883+0000 7f13a6a00640 1 -- 172.21.15.45:0/1129507422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13a80b7920 msgr2=0x7f13a80c9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:49.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.883+0000 7f13a6a00640 1 --2- 172.21.15.45:0/1129507422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13a80b7920 0x7f13a80c9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:49.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.883+0000 7f13a6a00640 1 -- 172.21.15.45:0/1129507422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13a80d2bc0 msgr2=0x7f13a80c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:49.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.883+0000 7f13a6a00640 1 --2- 172.21.15.45:0/1129507422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13a80d2bc0 0x7f13a80c8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:49.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.883+0000 7f13a6a00640 1 -- 172.21.15.45:0/1129507422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a80c8320 con 0x7f13a80b6270 2024-04-16T16:19:49.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.883+0000 7f13a6a00640 1 --2- 172.21.15.45:0/1129507422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a80b6270 0x7f13a8150cc0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f139800c430 tx=0x7f139800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:49.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.883+0000 7f139f400640 1 -- 172.21.15.45:0/1129507422 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13980074e0 con 0x7f13a80b6270 2024-04-16T16:19:49.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.884+0000 7f139f400640 1 -- 172.21.15.45:0/1129507422 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13980047e0 con 0x7f13a80b6270 2024-04-16T16:19:49.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.884+0000 7f139f400640 1 -- 172.21.15.45:0/1129507422 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1398004bd0 con 0x7f13a80b6270 2024-04-16T16:19:49.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.884+0000 7f13a7e00640 1 -- 172.21.15.45:0/1129507422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13a80ca300 con 0x7f13a80b6270 2024-04-16T16:19:49.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.884+0000 7f13a7e00640 1 -- 172.21.15.45:0/1129507422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a80ca8c0 con 0x7f13a80b6270 2024-04-16T16:19:49.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.885+0000 7f13a7e00640 1 -- 172.21.15.45:0/1129507422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13a8075780 con 0x7f13a80b6270 2024-04-16T16:19:49.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.887+0000 7f139f400640 1 -- 172.21.15.45:0/1129507422 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f1398011070 con 0x7f13a80b6270 2024-04-16T16:19:49.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.888+0000 7f139f400640 1 --2- 172.21.15.45:0/1129507422 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f138406c980 0x7f138406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:49.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.888+0000 7f139f400640 1 -- 172.21.15.45:0/1129507422 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5609+0+0 (secure 0 0 0) 0x7f139808bea0 con 0x7f13a80b6270 2024-04-16T16:19:49.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.888+0000 7f139e000640 1 --2- 172.21.15.45:0/1129507422 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f138406c980 0x7f138406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:49.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.890+0000 7f139e000640 1 --2- 172.21.15.45:0/1129507422 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f138406c980 0x7f138406ee40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1390036000 tx=0x7f139003b040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:49.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:49.891+0000 7f139f400640 1 -- 172.21.15.45:0/1129507422 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f139800e280 con 0x7f13a80b6270 2024-04-16T16:19:50.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.153+0000 7f13a7e00640 1 -- 172.21.15.45:0/1129507422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f13a80d01f0 con 0x7f13a80b6270 2024-04-16T16:19:50.243 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:50.244+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:50.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.657+0000 7f139f400640 1 -- 172.21.15.45:0/1129507422 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v78) ==== 174+0+0 (secure 0 0 0) 0x7f139805a2c0 con 0x7f13a80b6270 2024-04-16T16:19:50.660 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 16 and data pool 17 2024-04-16T16:19:50.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.667+0000 7f13a7e00640 1 -- 172.21.15.45:0/1129507422 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f138406c980 msgr2=0x7f138406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:50.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.667+0000 7f13a7e00640 1 --2- 172.21.15.45:0/1129507422 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f138406c980 0x7f138406ee40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1390036000 tx=0x7f139003b040 comp rx=0 tx=0).stop 2024-04-16T16:19:50.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.667+0000 7f13a7e00640 1 -- 172.21.15.45:0/1129507422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a80b6270 msgr2=0x7f13a8150cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:50.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.667+0000 7f13a7e00640 1 --2- 172.21.15.45:0/1129507422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a80b6270 0x7f13a8150cc0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f139800c430 tx=0x7f139800c8f0 comp rx=0 tx=0).stop 2024-04-16T16:19:50.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.667+0000 7f13a7e00640 1 -- 172.21.15.45:0/1129507422 shutdown_connections 2024-04-16T16:19:50.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.667+0000 7f13a7e00640 1 --2- 172.21.15.45:0/1129507422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13a80d2bc0 0x7f13a80c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:50.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.667+0000 7f13a7e00640 1 --2- 172.21.15.45:0/1129507422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13a80b7920 0x7f13a80c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:50.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.667+0000 7f13a7e00640 1 --2- 172.21.15.45:0/1129507422 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f138406c980 0x7f138406ee40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:50.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.667+0000 7f13a7e00640 1 --2- 172.21.15.45:0/1129507422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a80b6270 0x7f13a8150cc0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:50.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.668+0000 7f13a7e00640 1 -- 172.21.15.45:0/1129507422 >> 172.21.15.45:0/1129507422 conn(0x7f13a806d300 msgr2=0x7f13a80b8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:50.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.668+0000 7f13a7e00640 1 -- 172.21.15.45:0/1129507422 shutdown_connections 2024-04-16T16:19:50.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.671+0000 7f13a7e00640 1 -- 172.21.15.45:0/1129507422 wait complete. 2024-04-16T16:19:50.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:19:50.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.885+0000 7fef67a00640 1 Processor -- start 2024-04-16T16:19:50.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.885+0000 7fef67a00640 1 -- start start 2024-04-16T16:19:50.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.886+0000 7fef67a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fef601462e0 0x7fef60072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:50.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.886+0000 7fef67a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef600728c0 0x7fef600ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:50.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.886+0000 7fef67a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fef600cec00 0x7fef6015da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:50.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.886+0000 7fef67a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fef60072d20 con 0x7fef600cec00 2024-04-16T16:19:50.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.886+0000 7fef67a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fef600cf6d0 con 0x7fef601462e0 2024-04-16T16:19:50.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.886+0000 7fef67a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fef601466c0 con 0x7fef600728c0 2024-04-16T16:19:50.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.887+0000 7fef65a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fef600cec00 0x7fef6015da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:50.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.887+0000 7fef65a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fef600cec00 0x7fef6015da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52080/0 (socket says 172.21.15.45:52080) 2024-04-16T16:19:50.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.887+0000 7fef65a00640 1 -- 172.21.15.45:0/3017954042 learned_addr learned my addr 172.21.15.45:0/3017954042 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:50.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.887+0000 7fef65a00640 1 -- 172.21.15.45:0/3017954042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef600728c0 msgr2=0x7fef600ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:50.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.887+0000 7fef65a00640 1 --2- 172.21.15.45:0/3017954042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef600728c0 0x7fef600ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:50.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.887+0000 7fef65a00640 1 -- 172.21.15.45:0/3017954042 >> v2:172.21.15.103:3300/0 conn(0x7fef601462e0 msgr2=0x7fef60072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:50.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.887+0000 7fef65a00640 1 --2- 172.21.15.45:0/3017954042 >> v2:172.21.15.103:3300/0 conn(0x7fef601462e0 0x7fef60072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:50.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.887+0000 7fef65a00640 1 -- 172.21.15.45:0/3017954042 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef600cf3f0 con 0x7fef600cec00 2024-04-16T16:19:50.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.888+0000 7fef65a00640 1 --2- 172.21.15.45:0/3017954042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef600cec00 0x7fef6015da60 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fef600b5ec0 tx=0x7fef5802b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d6f9088793acfc server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:50.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.889+0000 7fef5f400640 1 -- 172.21.15.45:0/3017954042 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef580317f0 con 0x7fef600cec00 2024-04-16T16:19:50.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.889+0000 7fef5f400640 1 -- 172.21.15.45:0/3017954042 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef58037070 con 0x7fef600cec00 2024-04-16T16:19:50.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.889+0000 7fef5f400640 1 -- 172.21.15.45:0/3017954042 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef58031bb0 con 0x7fef600cec00 2024-04-16T16:19:50.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.889+0000 7fef67a00640 1 -- 172.21.15.45:0/3017954042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef600cec00 msgr2=0x7fef6015da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:50.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.889+0000 7fef67a00640 1 --2- 172.21.15.45:0/3017954042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef600cec00 0x7fef6015da60 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fef600b5ec0 tx=0x7fef5802b890 comp rx=0 tx=0).stop 2024-04-16T16:19:50.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.890+0000 7fef67a00640 1 -- 172.21.15.45:0/3017954042 shutdown_connections 2024-04-16T16:19:50.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.890+0000 7fef67a00640 1 --2- 172.21.15.45:0/3017954042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef600cec00 0x7fef6015da60 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:50.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.890+0000 7fef67a00640 1 --2- 172.21.15.45:0/3017954042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef600728c0 0x7fef600ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:50.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.890+0000 7fef67a00640 1 --2- 172.21.15.45:0/3017954042 >> v2:172.21.15.103:3300/0 conn(0x7fef601462e0 0x7fef60072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:50.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.890+0000 7fef67a00640 1 -- 172.21.15.45:0/3017954042 >> 172.21.15.45:0/3017954042 conn(0x7fef600b2750 msgr2=0x7fef600b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:50.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.890+0000 7fef67a00640 1 -- 172.21.15.45:0/3017954042 shutdown_connections 2024-04-16T16:19:50.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.890+0000 7fef67a00640 1 -- 172.21.15.45:0/3017954042 wait complete. 2024-04-16T16:19:50.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.892+0000 7fef67a00640 1 Processor -- start 2024-04-16T16:19:50.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.892+0000 7fef67a00640 1 -- start start 2024-04-16T16:19:50.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.893+0000 7fef67a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef600728c0 0x7fef600cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:50.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.893+0000 7fef67a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef600cec00 0x7fef6015b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:50.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.893+0000 7fef67a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef601462e0 0x7fef600b3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:50.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.893+0000 7fef67a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fef6015d540 con 0x7fef600728c0 2024-04-16T16:19:50.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.893+0000 7fef67a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fef6015d260 con 0x7fef601462e0 2024-04-16T16:19:50.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.893+0000 7fef67a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fef600cf090 con 0x7fef600cec00 2024-04-16T16:19:50.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.893+0000 7fef65000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef600728c0 0x7fef600cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:50.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.893+0000 7fef65000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef600728c0 0x7fef600cdce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52086/0 (socket says 172.21.15.45:52086) 2024-04-16T16:19:50.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.893+0000 7fef65000640 1 -- 172.21.15.45:0/4216700748 learned_addr learned my addr 172.21.15.45:0/4216700748 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:50.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.894+0000 7fef65a00640 1 --2- 172.21.15.45:0/4216700748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef601462e0 0x7fef600b3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:50.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.894+0000 7fef5fe00640 1 --2- 172.21.15.45:0/4216700748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef600cec00 0x7fef6015b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:50.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.894+0000 7fef65000640 1 -- 172.21.15.45:0/4216700748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef600cec00 msgr2=0x7fef6015b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:50.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.894+0000 7fef65000640 1 --2- 172.21.15.45:0/4216700748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef600cec00 0x7fef6015b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:50.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.894+0000 7fef65000640 1 -- 172.21.15.45:0/4216700748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef601462e0 msgr2=0x7fef600b3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:50.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.894+0000 7fef65000640 1 --2- 172.21.15.45:0/4216700748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef601462e0 0x7fef600b3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:50.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.894+0000 7fef65000640 1 -- 172.21.15.45:0/4216700748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef600ce2c0 con 0x7fef600728c0 2024-04-16T16:19:50.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.894+0000 7fef65000640 1 --2- 172.21.15.45:0/4216700748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef600728c0 0x7fef600cdce0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fef5000ba50 tx=0x7fef5000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:50.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.894+0000 7fef5cc00640 1 -- 172.21.15.45:0/4216700748 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef50019070 con 0x7fef600728c0 2024-04-16T16:19:50.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.894+0000 7fef5cc00640 1 -- 172.21.15.45:0/4216700748 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef5000cc00 con 0x7fef600728c0 2024-04-16T16:19:50.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.894+0000 7fef5cc00640 1 -- 172.21.15.45:0/4216700748 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef50014440 con 0x7fef600728c0 2024-04-16T16:19:50.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.895+0000 7fef67a00640 1 -- 172.21.15.45:0/4216700748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef6013e0d0 con 0x7fef600728c0 2024-04-16T16:19:50.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.895+0000 7fef67a00640 1 -- 172.21.15.45:0/4216700748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef6013e650 con 0x7fef600728c0 2024-04-16T16:19:50.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.896+0000 7fef67a00640 1 -- 172.21.15.45:0/4216700748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef60075780 con 0x7fef600728c0 2024-04-16T16:19:50.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.897+0000 7fef5cc00640 1 -- 172.21.15.45:0/4216700748 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7fef5001c070 con 0x7fef600728c0 2024-04-16T16:19:50.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.898+0000 7fef5cc00640 1 --2- 172.21.15.45:0/4216700748 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fef3c06c980 0x7fef3c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:50.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.898+0000 7fef5cc00640 1 -- 172.21.15.45:0/4216700748 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5709+0+0 (secure 0 0 0) 0x7fef5008b880 con 0x7fef600728c0 2024-04-16T16:19:50.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.899+0000 7fef5fe00640 1 --2- 172.21.15.45:0/4216700748 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fef3c06c980 0x7fef3c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:50.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.900+0000 7fef5fe00640 1 --2- 172.21.15.45:0/4216700748 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fef3c06c980 0x7fef3c06ee40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fef601f3550 tx=0x7fef48008040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:50.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:50.903+0000 7fef5cc00640 1 -- 172.21.15.45:0/4216700748 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fef50059d50 con 0x7fef600728c0 2024-04-16T16:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.108+0000 7fef67a00640 1 -- 172.21.15.45:0/4216700748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fef6015bb00 con 0x7fef600728c0 2024-04-16T16:19:51.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.111+0000 7fef5cc00640 1 -- 172.21.15.45:0/4216700748 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+16640 (secure 0 0 0) 0x7fef50059650 con 0x7fef600728c0 2024-04-16T16:19:51.111 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:51.111 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":113,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:19:50.154982+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:48.114834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:49.135741+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.114+0000 7fef67a00640 1 -- 172.21.15.45:0/4216700748 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fef3c06c980 msgr2=0x7fef3c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.114+0000 7fef67a00640 1 --2- 172.21.15.45:0/4216700748 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fef3c06c980 0x7fef3c06ee40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fef601f3550 tx=0x7fef48008040 comp rx=0 tx=0).stop 2024-04-16T16:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.114+0000 7fef67a00640 1 -- 172.21.15.45:0/4216700748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef600728c0 msgr2=0x7fef600cdce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.114+0000 7fef67a00640 1 --2- 172.21.15.45:0/4216700748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef600728c0 0x7fef600cdce0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fef5000ba50 tx=0x7fef5000bf10 comp rx=0 tx=0).stop 2024-04-16T16:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.114+0000 7fef67a00640 1 -- 172.21.15.45:0/4216700748 shutdown_connections 2024-04-16T16:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.114+0000 7fef67a00640 1 --2- 172.21.15.45:0/4216700748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef601462e0 0x7fef600b3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.114+0000 7fef67a00640 1 --2- 172.21.15.45:0/4216700748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef600cec00 0x7fef6015b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.114+0000 7fef67a00640 1 --2- 172.21.15.45:0/4216700748 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fef3c06c980 0x7fef3c06ee40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.114+0000 7fef67a00640 1 --2- 172.21.15.45:0/4216700748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef600728c0 0x7fef600cdce0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.114+0000 7fef67a00640 1 -- 172.21.15.45:0/4216700748 >> 172.21.15.45:0/4216700748 conn(0x7fef600b2750 msgr2=0x7fef6015da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.114+0000 7fef67a00640 1 -- 172.21.15.45:0/4216700748 shutdown_connections 2024-04-16T16:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.114+0000 7fef67a00640 1 -- 172.21.15.45:0/4216700748 wait complete. 2024-04-16T16:19:51.132 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:19:51.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.361+0000 7f6992600640 1 Processor -- start 2024-04-16T16:19:51.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.362+0000 7f6992600640 1 -- start start 2024-04-16T16:19:51.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.363+0000 7f6992600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f698c0cde00 0x7f698c0cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:51.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.363+0000 7f6992600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f698c06f540 0x7f698c1645b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:51.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.363+0000 7f6992600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f698c163510 con 0x7f698c163010 2024-04-16T16:19:51.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.363+0000 7f6992600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f698c1637f0 con 0x7f698c0cde00 2024-04-16T16:19:51.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.363+0000 7f6992600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f698c163ad0 con 0x7f698c06f540 2024-04-16T16:19:51.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.364+0000 7f6991200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f698c0cde00 0x7f698c0cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:51.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.364+0000 7f6991200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f698c0cde00 0x7f698c0cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52098/0 (socket says 172.21.15.45:52098) 2024-04-16T16:19:51.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.364+0000 7f6991200640 1 -- 172.21.15.45:0/2414941705 learned_addr learned my addr 172.21.15.45:0/2414941705 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:51.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.364+0000 7f6991c00640 1 --2- 172.21.15.45:0/2414941705 >> v2:172.21.15.103:3300/0 conn(0x7f698c06f540 0x7f698c1645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:51.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.364+0000 7f6991200640 1 -- 172.21.15.45:0/2414941705 >> v2:172.21.15.103:3300/0 conn(0x7f698c06f540 msgr2=0x7f698c1645b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:51.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.364+0000 7f6991200640 1 --2- 172.21.15.45:0/2414941705 >> v2:172.21.15.103:3300/0 conn(0x7f698c06f540 0x7f698c1645b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.364+0000 7f6991200640 1 -- 172.21.15.45:0/2414941705 >> v1:172.21.15.103:6789/0 conn(0x7f698c163010 legacy=0x7f698c0ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:51.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.364+0000 7f6991200640 1 -- 172.21.15.45:0/2414941705 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c163db0 con 0x7f698c0cde00 2024-04-16T16:19:51.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.366+0000 7f6991200640 1 --2- 172.21.15.45:0/2414941705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f698c0cde00 0x7f698c0cff30 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f698c0733e0 tx=0x7f697c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa646752a1e7d151 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:51.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.366+0000 7f698b400640 1 -- 172.21.15.45:0/2414941705 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c035020 con 0x7f698c0cde00 2024-04-16T16:19:51.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.366+0000 7f698b400640 1 -- 172.21.15.45:0/2414941705 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f697c031870 con 0x7f698c0cde00 2024-04-16T16:19:51.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.366+0000 7f698b400640 1 -- 172.21.15.45:0/2414941705 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c031c40 con 0x7f698c0cde00 2024-04-16T16:19:51.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.367+0000 7f6992600640 1 -- 172.21.15.45:0/2414941705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f698c0cde00 msgr2=0x7f698c0cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:51.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.367+0000 7f6992600640 1 --2- 172.21.15.45:0/2414941705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f698c0cde00 0x7f698c0cff30 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f698c0733e0 tx=0x7f697c02b910 comp rx=0 tx=0).stop 2024-04-16T16:19:51.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.367+0000 7f6992600640 1 -- 172.21.15.45:0/2414941705 shutdown_connections 2024-04-16T16:19:51.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.367+0000 7f6992600640 1 --2- 172.21.15.45:0/2414941705 >> v2:172.21.15.103:3300/0 conn(0x7f698c06f540 0x7f698c1645b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.367+0000 7f6992600640 1 --2- 172.21.15.45:0/2414941705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f698c0cde00 0x7f698c0cff30 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.367+0000 7f6992600640 1 -- 172.21.15.45:0/2414941705 >> 172.21.15.45:0/2414941705 conn(0x7f698c06b930 msgr2=0x7f698c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:51.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.367+0000 7f6992600640 1 -- 172.21.15.45:0/2414941705 shutdown_connections 2024-04-16T16:19:51.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.367+0000 7f6992600640 1 -- 172.21.15.45:0/2414941705 wait complete. 2024-04-16T16:19:51.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.369+0000 7f6992600640 1 Processor -- start 2024-04-16T16:19:51.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.369+0000 7f6992600640 1 -- start start 2024-04-16T16:19:51.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.370+0000 7f6992600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f698c06f540 0x7f698c152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:51.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.370+0000 7f6992600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f698c0cde00 0x7f698c13efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:51.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.370+0000 7f6992600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f698c163010 0x7f698c1eb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:51.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.370+0000 7f6992600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f698c164af0 con 0x7f698c163010 2024-04-16T16:19:51.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.370+0000 7f6992600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f698c06f970 con 0x7f698c0cde00 2024-04-16T16:19:51.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.370+0000 7f6992600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f698c164c70 con 0x7f698c06f540 2024-04-16T16:19:51.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.370+0000 7f6991c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f698c163010 0x7f698c1eb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:51.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.370+0000 7f6991c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f698c163010 0x7f698c1eb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52106/0 (socket says 172.21.15.45:52106) 2024-04-16T16:19:51.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.370+0000 7f6991c00640 1 -- 172.21.15.45:0/46136587 learned_addr learned my addr 172.21.15.45:0/46136587 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:51.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.371+0000 7f6991200640 1 --2- 172.21.15.45:0/46136587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f698c06f540 0x7f698c152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:51.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.371+0000 7f698be00640 1 --2- 172.21.15.45:0/46136587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f698c0cde00 0x7f698c13efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:51.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.371+0000 7f6991c00640 1 -- 172.21.15.45:0/46136587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f698c06f540 msgr2=0x7f698c152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:51.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.371+0000 7f6991c00640 1 --2- 172.21.15.45:0/46136587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f698c06f540 0x7f698c152a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.371+0000 7f6991c00640 1 -- 172.21.15.45:0/46136587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f698c0cde00 msgr2=0x7f698c13efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:51.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.371+0000 7f6991c00640 1 --2- 172.21.15.45:0/46136587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f698c0cde00 0x7f698c13efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.371+0000 7f6991c00640 1 -- 172.21.15.45:0/46136587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c13f9a0 con 0x7f698c163010 2024-04-16T16:19:51.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.371+0000 7f6991c00640 1 --2- 172.21.15.45:0/46136587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f698c163010 0x7f698c1eb590 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f698400cd70 tx=0x7f698400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:51.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.372+0000 7f6988c00640 1 -- 172.21.15.45:0/46136587 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984003b90 con 0x7f698c163010 2024-04-16T16:19:51.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.372+0000 7f6988c00640 1 -- 172.21.15.45:0/46136587 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6984003d30 con 0x7f698c163010 2024-04-16T16:19:51.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.372+0000 7f6988c00640 1 -- 172.21.15.45:0/46136587 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984004240 con 0x7f698c163010 2024-04-16T16:19:51.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.372+0000 7f6992600640 1 -- 172.21.15.45:0/46136587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698c13fc10 con 0x7f698c163010 2024-04-16T16:19:51.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.372+0000 7f6992600640 1 -- 172.21.15.45:0/46136587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f698c1401d0 con 0x7f698c163010 2024-04-16T16:19:51.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.373+0000 7f6992600640 1 -- 172.21.15.45:0/46136587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f698c075780 con 0x7f698c163010 2024-04-16T16:19:51.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.374+0000 7f6988c00640 1 -- 172.21.15.45:0/46136587 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f69840043e0 con 0x7f698c163010 2024-04-16T16:19:51.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.375+0000 7f6988c00640 1 --2- 172.21.15.45:0/46136587 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f696c06c980 0x7f696c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:51.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.375+0000 7f6988c00640 1 -- 172.21.15.45:0/46136587 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5709+0+0 (secure 0 0 0) 0x7f698408c7c0 con 0x7f698c163010 2024-04-16T16:19:51.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.376+0000 7f6991200640 1 --2- 172.21.15.45:0/46136587 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f696c06c980 0x7f696c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:51.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.377+0000 7f6991200640 1 --2- 172.21.15.45:0/46136587 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f696c06c980 0x7f696c06ee40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f697c036000 tx=0x7f697c03c040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:51.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.379+0000 7f6988c00640 1 -- 172.21.15.45:0/46136587 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f698405aa60 con 0x7f698c163010 2024-04-16T16:19:51.452 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:51.452+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:51.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.593+0000 7f6992600640 1 -- 172.21.15.45:0/46136587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f698c06fc20 con 0x7f698c163010 2024-04-16T16:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.595+0000 7f6988c00640 1 -- 172.21.15.45:0/46136587 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+16640 (secure 0 0 0) 0x7f698405a360 con 0x7f698c163010 2024-04-16T16:19:51.595 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:51.595 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":113,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:19:50.154982+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:48.114834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:49.135741+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:19:51.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.598+0000 7f6992600640 1 -- 172.21.15.45:0/46136587 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f696c06c980 msgr2=0x7f696c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.598+0000 7f6992600640 1 --2- 172.21.15.45:0/46136587 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f696c06c980 0x7f696c06ee40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f697c036000 tx=0x7f697c03c040 comp rx=0 tx=0).stop 2024-04-16T16:19:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.598+0000 7f6992600640 1 -- 172.21.15.45:0/46136587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f698c163010 msgr2=0x7f698c1eb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.598+0000 7f6992600640 1 --2- 172.21.15.45:0/46136587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f698c163010 0x7f698c1eb590 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f698400cd70 tx=0x7f698400a660 comp rx=0 tx=0).stop 2024-04-16T16:19:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.598+0000 7f6992600640 1 -- 172.21.15.45:0/46136587 shutdown_connections 2024-04-16T16:19:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.598+0000 7f6992600640 1 --2- 172.21.15.45:0/46136587 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f696c06c980 0x7f696c06ee40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.598+0000 7f6992600640 1 --2- 172.21.15.45:0/46136587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f698c163010 0x7f698c1eb590 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.598+0000 7f6992600640 1 --2- 172.21.15.45:0/46136587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f698c0cde00 0x7f698c13efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.598+0000 7f6992600640 1 --2- 172.21.15.45:0/46136587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f698c06f540 0x7f698c152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.598+0000 7f6992600640 1 -- 172.21.15.45:0/46136587 >> 172.21.15.45:0/46136587 conn(0x7f698c06b930 msgr2=0x7f698c0b2380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.599+0000 7f6992600640 1 -- 172.21.15.45:0/46136587 shutdown_connections 2024-04-16T16:19:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.599+0000 7f6992600640 1 -- 172.21.15.45:0/46136587 wait complete. 2024-04-16T16:19:51.618 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-16T16:19:51.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.844+0000 7f930a400640 1 Processor -- start 2024-04-16T16:19:51.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.844+0000 7f930a400640 1 -- start start 2024-04-16T16:19:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.846+0000 7f930a400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f93040d18f0 0x7f93040ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.846+0000 7f930a400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93040b6310 0x7f93040d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.846+0000 7f930a400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93040d1cd0 0x7f93040d33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.846+0000 7f930a400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f93040d29f0 con 0x7f93040d1cd0 2024-04-16T16:19:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.846+0000 7f930a400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f93040b6810 con 0x7f93040d18f0 2024-04-16T16:19:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.846+0000 7f930a400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f93040b6af0 con 0x7f93040b6310 2024-04-16T16:19:51.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.846+0000 7f9309a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93040d1cd0 0x7f93040d33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:51.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.847+0000 7f9309a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93040d1cd0 0x7f93040d33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52112/0 (socket says 172.21.15.45:52112) 2024-04-16T16:19:51.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.847+0000 7f9309a00640 1 -- 172.21.15.45:0/2096485350 learned_addr learned my addr 172.21.15.45:0/2096485350 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:51.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.847+0000 7f9303e00640 1 --2- 172.21.15.45:0/2096485350 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93040b6310 0x7f93040d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:51.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.847+0000 7f9309000640 1 --2- 172.21.15.45:0/2096485350 >> v2:172.21.15.103:3300/0 conn(0x7f93040d18f0 0x7f93040ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:51.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.847+0000 7f9309a00640 1 -- 172.21.15.45:0/2096485350 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93040b6310 msgr2=0x7f93040d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.847+0000 7f9309a00640 1 --2- 172.21.15.45:0/2096485350 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93040b6310 0x7f93040d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.847+0000 7f9309a00640 1 -- 172.21.15.45:0/2096485350 >> v2:172.21.15.103:3300/0 conn(0x7f93040d18f0 msgr2=0x7f93040ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.847+0000 7f9309a00640 1 --2- 172.21.15.45:0/2096485350 >> v2:172.21.15.103:3300/0 conn(0x7f93040d18f0 0x7f93040ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.847+0000 7f9309a00640 1 -- 172.21.15.45:0/2096485350 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93040b6dd0 con 0x7f93040d1cd0 2024-04-16T16:19:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.847+0000 7f9309000640 1 --2- 172.21.15.45:0/2096485350 >> v2:172.21.15.103:3300/0 conn(0x7f93040d18f0 0x7f93040ba220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:51.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.849+0000 7f9309a00640 1 --2- 172.21.15.45:0/2096485350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93040d1cd0 0x7f93040d33c0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f9304073f70 tx=0x7f92fc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bcca8d4e38e7ccc server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.849+0000 7f9303400640 1 -- 172.21.15.45:0/2096485350 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc02c9e0 con 0x7f93040d1cd0 2024-04-16T16:19:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.849+0000 7f9303400640 1 -- 172.21.15.45:0/2096485350 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92fc02cb80 con 0x7f93040d1cd0 2024-04-16T16:19:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.849+0000 7f9303400640 1 -- 172.21.15.45:0/2096485350 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc0343f0 con 0x7f93040d1cd0 2024-04-16T16:19:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.850+0000 7f930a400640 1 -- 172.21.15.45:0/2096485350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93040d1cd0 msgr2=0x7f93040d33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.850+0000 7f930a400640 1 --2- 172.21.15.45:0/2096485350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93040d1cd0 0x7f93040d33c0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f9304073f70 tx=0x7f92fc02be40 comp rx=0 tx=0).stop 2024-04-16T16:19:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.850+0000 7f930a400640 1 -- 172.21.15.45:0/2096485350 shutdown_connections 2024-04-16T16:19:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.850+0000 7f930a400640 1 --2- 172.21.15.45:0/2096485350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93040d1cd0 0x7f93040d33c0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.850+0000 7f930a400640 1 --2- 172.21.15.45:0/2096485350 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93040b6310 0x7f93040d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.850+0000 7f930a400640 1 --2- 172.21.15.45:0/2096485350 >> v2:172.21.15.103:3300/0 conn(0x7f93040d18f0 0x7f93040ba220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.850+0000 7f930a400640 1 -- 172.21.15.45:0/2096485350 >> 172.21.15.45:0/2096485350 conn(0x7f930406cf90 msgr2=0x7f930406d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:51.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.850+0000 7f930a400640 1 -- 172.21.15.45:0/2096485350 shutdown_connections 2024-04-16T16:19:51.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.850+0000 7f930a400640 1 -- 172.21.15.45:0/2096485350 wait complete. 2024-04-16T16:19:51.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.852+0000 7f930a400640 1 Processor -- start 2024-04-16T16:19:51.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.852+0000 7f930a400640 1 -- start start 2024-04-16T16:19:51.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.853+0000 7f930a400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93040b6310 0x7f930415aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:51.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.853+0000 7f930a400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93040d18f0 0x7f9304158780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:51.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.853+0000 7f930a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93040d1cd0 0x7f930415dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:51.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.853+0000 7f930a400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f93040d3900 con 0x7f93040d1cd0 2024-04-16T16:19:51.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.853+0000 7f930a400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f93040ba760 con 0x7f93040d18f0 2024-04-16T16:19:51.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.853+0000 7f930a400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f93040d3a80 con 0x7f93040b6310 2024-04-16T16:19:51.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.853+0000 7f9309a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93040d1cd0 0x7f930415dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:51.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.853+0000 7f9309a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93040d1cd0 0x7f930415dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52118/0 (socket says 172.21.15.45:52118) 2024-04-16T16:19:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.853+0000 7f9309a00640 1 -- 172.21.15.45:0/2424616868 learned_addr learned my addr 172.21.15.45:0/2424616868 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.853+0000 7f9309000640 1 --2- 172.21.15.45:0/2424616868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93040b6310 0x7f930415aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.854+0000 7f9303e00640 1 --2- 172.21.15.45:0/2424616868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93040d18f0 0x7f9304158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.854+0000 7f9309a00640 1 -- 172.21.15.45:0/2424616868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93040b6310 msgr2=0x7f930415aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.854+0000 7f9309a00640 1 --2- 172.21.15.45:0/2424616868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93040b6310 0x7f930415aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.854+0000 7f9309a00640 1 -- 172.21.15.45:0/2424616868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93040d18f0 msgr2=0x7f9304158780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.854+0000 7f9309a00640 1 --2- 172.21.15.45:0/2424616868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93040d18f0 0x7f9304158780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.854+0000 7f9309a00640 1 -- 172.21.15.45:0/2424616868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f930415ca30 con 0x7f93040d1cd0 2024-04-16T16:19:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.854+0000 7f9303e00640 1 --2- 172.21.15.45:0/2424616868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93040d18f0 0x7f9304158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.854+0000 7f9309a00640 1 --2- 172.21.15.45:0/2424616868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93040d1cd0 0x7f930415dd30 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f92fc036040 tx=0x7f92fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.854+0000 7f9300c00640 1 -- 172.21.15.45:0/2424616868 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc043020 con 0x7f93040d1cd0 2024-04-16T16:19:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.854+0000 7f9300c00640 1 -- 172.21.15.45:0/2424616868 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92fc03e450 con 0x7f93040d1cd0 2024-04-16T16:19:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.854+0000 7f9300c00640 1 -- 172.21.15.45:0/2424616868 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc009510 con 0x7f93040d1cd0 2024-04-16T16:19:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.854+0000 7f930a400640 1 -- 172.21.15.45:0/2424616868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f930415d250 con 0x7f93040d1cd0 2024-04-16T16:19:51.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.855+0000 7f930a400640 1 -- 172.21.15.45:0/2424616868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f930415d810 con 0x7f93040d1cd0 2024-04-16T16:19:51.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.856+0000 7f930a400640 1 -- 172.21.15.45:0/2424616868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9304075780 con 0x7f93040d1cd0 2024-04-16T16:19:51.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.857+0000 7f9300c00640 1 -- 172.21.15.45:0/2424616868 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f92fc046020 con 0x7f93040d1cd0 2024-04-16T16:19:51.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.857+0000 7f9300c00640 1 --2- 172.21.15.45:0/2424616868 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f92e406c980 0x7f92e406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.858+0000 7f9300c00640 1 -- 172.21.15.45:0/2424616868 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 5709+0+0 (secure 0 0 0) 0x7f92fc0ae540 con 0x7f93040d1cd0 2024-04-16T16:19:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.858+0000 7f9309000640 1 --2- 172.21.15.45:0/2424616868 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f92e406c980 0x7f92e406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.859+0000 7f9309000640 1 --2- 172.21.15.45:0/2424616868 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f92e406c980 0x7f92e406ee40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f92f800a990 tx=0x7f92f8008040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:51.862+0000 7f9300c00640 1 -- 172.21.15.45:0/2424616868 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f92fc04d030 con 0x7f93040d1cd0 2024-04-16T16:19:52.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.115+0000 7f930a400640 1 -- 172.21.15.45:0/2424616868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f930415d9e0 con 0x7f93040d1cd0 2024-04-16T16:19:52.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.679+0000 7f9300c00640 1 -- 172.21.15.45:0/2424616868 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v81) ==== 118+0+0 (secure 0 0 0) 0x7f92fc07c880 con 0x7f93040d1cd0 2024-04-16T16:19:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.683+0000 7f930a400640 1 -- 172.21.15.45:0/2424616868 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f92e406c980 msgr2=0x7f92e406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.683+0000 7f930a400640 1 --2- 172.21.15.45:0/2424616868 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f92e406c980 0x7f92e406ee40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f92f800a990 tx=0x7f92f8008040 comp rx=0 tx=0).stop 2024-04-16T16:19:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.683+0000 7f930a400640 1 -- 172.21.15.45:0/2424616868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93040d1cd0 msgr2=0x7f930415dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.683+0000 7f930a400640 1 --2- 172.21.15.45:0/2424616868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93040d1cd0 0x7f930415dd30 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f92fc036040 tx=0x7f92fc004a40 comp rx=0 tx=0).stop 2024-04-16T16:19:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.684+0000 7f930a400640 1 -- 172.21.15.45:0/2424616868 shutdown_connections 2024-04-16T16:19:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.684+0000 7f930a400640 1 --2- 172.21.15.45:0/2424616868 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f92e406c980 0x7f92e406ee40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.684+0000 7f930a400640 1 --2- 172.21.15.45:0/2424616868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93040d1cd0 0x7f930415dd30 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.684+0000 7f930a400640 1 --2- 172.21.15.45:0/2424616868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93040d18f0 0x7f9304158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.684+0000 7f930a400640 1 --2- 172.21.15.45:0/2424616868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93040b6310 0x7f930415aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:52.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.684+0000 7f930a400640 1 -- 172.21.15.45:0/2424616868 >> 172.21.15.45:0/2424616868 conn(0x7f930406cf90 msgr2=0x7f93040cded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:52.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.684+0000 7f930a400640 1 -- 172.21.15.45:0/2424616868 shutdown_connections 2024-04-16T16:19:52.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.684+0000 7f930a400640 1 -- 172.21.15.45:0/2424616868 wait complete. 2024-04-16T16:19:52.693 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:52.693+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:52.705 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-16T16:19:52.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-16T16:19:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.939+0000 7f4f62c00640 1 Processor -- start 2024-04-16T16:19:52.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.939+0000 7f4f62c00640 1 -- start start 2024-04-16T16:19:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.941+0000 7f4f62c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f5c0d10e0 0x7f4f5c072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.941+0000 7f4f62c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4f5c0b4930 0x7f4f5c06f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.941+0000 7f4f62c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f5c0b5530 con 0x7f4f5c15c7c0 2024-04-16T16:19:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.941+0000 7f4f62c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f4f5c0b52e0 con 0x7f4f5c0b4930 2024-04-16T16:19:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.941+0000 7f4f62c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4f5c0d0260 con 0x7f4f5c0d10e0 2024-04-16T16:19:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.941+0000 7f4f60c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4f5c0b4930 0x7f4f5c06f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.941+0000 7f4f5be00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4f5c15c7c0 0x7f4f5c0d09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60182/0 (socket says 172.21.15.45:60182) 2024-04-16T16:19:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.941+0000 7f4f60c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4f5c0b4930 0x7f4f5c06f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52122/0 (socket says 172.21.15.45:52122) 2024-04-16T16:19:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.941+0000 7f4f5be00640 1 -- 172.21.15.45:0/573073276 learned_addr learned my addr 172.21.15.45:0/573073276 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.941+0000 7f4f5b400640 1 --2- 172.21.15.45:0/573073276 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f5c0d10e0 0x7f4f5c072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.941+0000 7f4f60c00640 1 -- 172.21.15.45:0/573073276 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f5c0d10e0 msgr2=0x7f4f5c072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.942+0000 7f4f60c00640 1 --2- 172.21.15.45:0/573073276 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f5c0d10e0 0x7f4f5c072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.942+0000 7f4f60c00640 1 -- 172.21.15.45:0/573073276 >> v1:172.21.15.45:6789/0 conn(0x7f4f5c15c7c0 legacy=0x7f4f5c0d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.942+0000 7f4f60c00640 1 -- 172.21.15.45:0/573073276 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f5c0b5810 con 0x7f4f5c0b4930 2024-04-16T16:19:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.942+0000 7f4f5aa00640 1 -- 172.21.15.45:0/573073276 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3686476383 0 0) 0x7f4f5c0b5530 con 0x7f4f5c15c7c0 2024-04-16T16:19:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.943+0000 7f4f60c00640 1 --2- 172.21.15.45:0/573073276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f5c0b4930 0x7f4f5c06f160 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f4f540097a0 tx=0x7f4f5402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede58ff93d0c942 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.943+0000 7f4f5aa00640 1 -- 172.21.15.45:0/573073276 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f54038070 con 0x7f4f5c0b4930 2024-04-16T16:19:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.943+0000 7f4f5aa00640 1 -- 172.21.15.45:0/573073276 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f54031700 con 0x7f4f5c0b4930 2024-04-16T16:19:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.943+0000 7f4f5aa00640 1 -- 172.21.15.45:0/573073276 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f54031ad0 con 0x7f4f5c0b4930 2024-04-16T16:19:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.944+0000 7f4f62c00640 1 -- 172.21.15.45:0/573073276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f5c0b4930 msgr2=0x7f4f5c06f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.944+0000 7f4f62c00640 1 --2- 172.21.15.45:0/573073276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f5c0b4930 0x7f4f5c06f160 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f4f540097a0 tx=0x7f4f5402b7a0 comp rx=0 tx=0).stop 2024-04-16T16:19:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.944+0000 7f4f62c00640 1 -- 172.21.15.45:0/573073276 shutdown_connections 2024-04-16T16:19:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.944+0000 7f4f62c00640 1 --2- 172.21.15.45:0/573073276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f5c0b4930 0x7f4f5c06f160 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.944+0000 7f4f62c00640 1 --2- 172.21.15.45:0/573073276 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f5c0d10e0 0x7f4f5c072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.944+0000 7f4f62c00640 1 -- 172.21.15.45:0/573073276 >> 172.21.15.45:0/573073276 conn(0x7f4f5c0b3970 msgr2=0x7f4f5c0b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.944+0000 7f4f62c00640 1 -- 172.21.15.45:0/573073276 shutdown_connections 2024-04-16T16:19:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.944+0000 7f4f62c00640 1 -- 172.21.15.45:0/573073276 wait complete. 2024-04-16T16:19:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.946+0000 7f4f62c00640 1 Processor -- start 2024-04-16T16:19:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.946+0000 7f4f62c00640 1 -- start start 2024-04-16T16:19:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.947+0000 7f4f62c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f5c0b4930 0x7f4f5c157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.947+0000 7f4f62c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f5c0d10e0 0x7f4f5c1f5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.947+0000 7f4f62c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f5c15c7c0 0x7f4f5c0ce380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.947+0000 7f4f62c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5c06f6a0 con 0x7f4f5c0d10e0 2024-04-16T16:19:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.947+0000 7f4f62c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5c073350 con 0x7f4f5c0b4930 2024-04-16T16:19:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.947+0000 7f4f62c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4f5c06f820 con 0x7f4f5c15c7c0 2024-04-16T16:19:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.947+0000 7f4f5b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f5c0d10e0 0x7f4f5c1f5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.947+0000 7f4f5b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f5c0d10e0 0x7f4f5c1f5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52128/0 (socket says 172.21.15.45:52128) 2024-04-16T16:19:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.947+0000 7f4f5b400640 1 -- 172.21.15.45:0/1664017851 learned_addr learned my addr 172.21.15.45:0/1664017851 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.947+0000 7f4f5be00640 1 --2- 172.21.15.45:0/1664017851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f5c0b4930 0x7f4f5c157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.947+0000 7f4f5b400640 1 -- 172.21.15.45:0/1664017851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f5c15c7c0 msgr2=0x7f4f5c0ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.947+0000 7f4f5b400640 1 --2- 172.21.15.45:0/1664017851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f5c15c7c0 0x7f4f5c0ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.948+0000 7f4f5b400640 1 -- 172.21.15.45:0/1664017851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f5c0b4930 msgr2=0x7f4f5c157450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.948+0000 7f4f5b400640 1 --2- 172.21.15.45:0/1664017851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f5c0b4930 0x7f4f5c157450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.948+0000 7f4f5b400640 1 -- 172.21.15.45:0/1664017851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f5c0ceb20 con 0x7f4f5c0d10e0 2024-04-16T16:19:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.948+0000 7f4f5b400640 1 --2- 172.21.15.45:0/1664017851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f5c0d10e0 0x7f4f5c1f5360 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f4f4400d800 tx=0x7f4f4400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.948+0000 7f4f53e00640 1 -- 172.21.15.45:0/1664017851 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f4400b6c0 con 0x7f4f5c0d10e0 2024-04-16T16:19:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.948+0000 7f4f53e00640 1 -- 172.21.15.45:0/1664017851 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f44019070 con 0x7f4f5c0d10e0 2024-04-16T16:19:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.948+0000 7f4f62c00640 1 -- 172.21.15.45:0/1664017851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f5c0cee00 con 0x7f4f5c0d10e0 2024-04-16T16:19:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.948+0000 7f4f53e00640 1 -- 172.21.15.45:0/1664017851 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f4400ba80 con 0x7f4f5c0d10e0 2024-04-16T16:19:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.948+0000 7f4f62c00640 1 -- 172.21.15.45:0/1664017851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f5c1f4e40 con 0x7f4f5c0d10e0 2024-04-16T16:19:52.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.950+0000 7f4f62c00640 1 -- 172.21.15.45:0/1664017851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f5c075780 con 0x7f4f5c0d10e0 2024-04-16T16:19:52.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.951+0000 7f4f53e00640 1 -- 172.21.15.45:0/1664017851 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f4f4400bc20 con 0x7f4f5c0d10e0 2024-04-16T16:19:52.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.952+0000 7f4f53e00640 1 --2- 172.21.15.45:0/1664017851 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f4f3806c980 0x7f4f3806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:52.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.952+0000 7f4f53e00640 1 -- 172.21.15.45:0/1664017851 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 5709+0+0 (secure 0 0 0) 0x7f4f4408c060 con 0x7f4f5c0d10e0 2024-04-16T16:19:52.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.953+0000 7f4f5be00640 1 --2- 172.21.15.45:0/1664017851 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f4f3806c980 0x7f4f3806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:52.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.954+0000 7f4f5be00640 1 --2- 172.21.15.45:0/1664017851 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f4f3806c980 0x7f4f3806ee40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c0012e0 tx=0x7f4f4c009040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:52.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:52.957+0000 7f4f53e00640 1 -- 172.21.15.45:0/1664017851 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f4401c360 con 0x7f4f5c0d10e0 2024-04-16T16:19:53.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.219+0000 7f4f62c00640 1 -- 172.21.15.45:0/1664017851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f4f5c0b1c40 con 0x7f4f5c0d10e0 2024-04-16T16:19:53.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.695+0000 7f4f53e00640 1 -- 172.21.15.45:0/1664017851 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v82) ==== 122+0+0 (secure 0 0 0) 0x7f4f4405a430 con 0x7f4f5c0d10e0 2024-04-16T16:19:53.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.698+0000 7f4f62c00640 1 -- 172.21.15.45:0/1664017851 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f4f3806c980 msgr2=0x7f4f3806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:53.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.698+0000 7f4f62c00640 1 --2- 172.21.15.45:0/1664017851 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f4f3806c980 0x7f4f3806ee40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c0012e0 tx=0x7f4f4c009040 comp rx=0 tx=0).stop 2024-04-16T16:19:53.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.698+0000 7f4f62c00640 1 -- 172.21.15.45:0/1664017851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f5c0d10e0 msgr2=0x7f4f5c1f5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:53.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.698+0000 7f4f62c00640 1 --2- 172.21.15.45:0/1664017851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f5c0d10e0 0x7f4f5c1f5360 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f4f4400d800 tx=0x7f4f4400dcc0 comp rx=0 tx=0).stop 2024-04-16T16:19:53.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.698+0000 7f4f62c00640 1 -- 172.21.15.45:0/1664017851 shutdown_connections 2024-04-16T16:19:53.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.698+0000 7f4f62c00640 1 --2- 172.21.15.45:0/1664017851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f5c15c7c0 0x7f4f5c0ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:53.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.698+0000 7f4f62c00640 1 --2- 172.21.15.45:0/1664017851 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f4f3806c980 0x7f4f3806ee40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:53.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.698+0000 7f4f62c00640 1 --2- 172.21.15.45:0/1664017851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f5c0d10e0 0x7f4f5c1f5360 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:53.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.698+0000 7f4f62c00640 1 --2- 172.21.15.45:0/1664017851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f5c0b4930 0x7f4f5c157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:53.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.698+0000 7f4f62c00640 1 -- 172.21.15.45:0/1664017851 >> 172.21.15.45:0/1664017851 conn(0x7f4f5c0b3970 msgr2=0x7f4f5c074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:53.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.699+0000 7f4f62c00640 1 -- 172.21.15.45:0/1664017851 shutdown_connections 2024-04-16T16:19:53.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.699+0000 7f4f62c00640 1 -- 172.21.15.45:0/1664017851 wait complete. 2024-04-16T16:19:53.724 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-16T16:19:53.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:19:53.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.951+0000 7f310f400640 1 Processor -- start 2024-04-16T16:19:53.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.952+0000 7f310f400640 1 -- start start 2024-04-16T16:19:53.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.953+0000 7f310f400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f310815b730 0x7f31080cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:53.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.953+0000 7f310f400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31080d0230 0x7f31080b4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:53.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.953+0000 7f310f400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31080ce5b0 con 0x7f31080b5840 2024-04-16T16:19:53.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.953+0000 7f310f400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3108072420 con 0x7f310815b730 2024-04-16T16:19:53.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.953+0000 7f310f400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3108072700 con 0x7f31080d0230 2024-04-16T16:19:53.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.954+0000 7f3107e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f31080b5840 0x7f31080cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33772/0 (socket says 172.21.15.45:33772) 2024-04-16T16:19:53.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.954+0000 7f3107e00640 1 -- 172.21.15.45:0/3774229069 learned_addr learned my addr 172.21.15.45:0/3774229069 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:53.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.954+0000 7f310ca00640 1 --2- 172.21.15.45:0/3774229069 >> v2:172.21.15.103:3300/0 conn(0x7f310815b730 0x7f31080cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:53.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.954+0000 7f310d400640 1 --2- 172.21.15.45:0/3774229069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31080d0230 0x7f31080b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:53.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.954+0000 7f3107400640 1 -- 172.21.15.45:0/3774229069 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3302491335 0 0) 0x7f31080ce5b0 con 0x7f31080b5840 2024-04-16T16:19:53.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.954+0000 7f3107400640 1 -- 172.21.15.45:0/3774229069 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30f0005730 con 0x7f31080b5840 2024-04-16T16:19:53.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.955+0000 7f3107400640 1 -- 172.21.15.45:0/3774229069 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3469470023 0 0) 0x7f30f0005730 con 0x7f31080b5840 2024-04-16T16:19:53.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.955+0000 7f3107400640 1 -- 172.21.15.45:0/3774229069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31080d0230 msgr2=0x7f31080b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:53.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.955+0000 7f3107400640 1 --2- 172.21.15.45:0/3774229069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31080d0230 0x7f31080b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:53.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.955+0000 7f3107400640 1 -- 172.21.15.45:0/3774229069 >> v2:172.21.15.103:3300/0 conn(0x7f310815b730 msgr2=0x7f31080cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:53.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.955+0000 7f3107400640 1 --2- 172.21.15.45:0/3774229069 >> v2:172.21.15.103:3300/0 conn(0x7f310815b730 0x7f31080cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:53.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.955+0000 7f3107400640 1 -- 172.21.15.45:0/3774229069 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31080729e0 con 0x7f31080b5840 2024-04-16T16:19:53.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.955+0000 7f3107400640 1 -- 172.21.15.45:0/3774229069 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f30f4001670 con 0x7f31080b5840 2024-04-16T16:19:53.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.955+0000 7f3107400640 1 -- 172.21.15.45:0/3774229069 >> v1:172.21.15.45:6789/0 conn(0x7f31080b5840 legacy=0x7f31080cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:53.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.955+0000 7f3107400640 1 --2- 172.21.15.45:0/3774229069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30f0005bc0 0x7f30f0005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:53.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.955+0000 7f3107400640 1 --2- 172.21.15.45:0/3774229069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30f0006a10 0x7f30f0006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:53.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.955+0000 7f3107400640 1 --2- 172.21.15.45:0/3774229069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31080d0230 0x7f30f00073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:53.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.955+0000 7f3107400640 1 -- 172.21.15.45:0/3774229069 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3108072420 con 0x7f30f0005bc0 2024-04-16T16:19:53.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.955+0000 7f3107400640 1 -- 172.21.15.45:0/3774229069 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3108072700 con 0x7f30f0006a10 2024-04-16T16:19:53.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.955+0000 7f3107400640 1 -- 172.21.15.45:0/3774229069 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f30f000dcd0 con 0x7f31080d0230 2024-04-16T16:19:53.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.955+0000 7f310ca00640 1 --2- 172.21.15.45:0/3774229069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30f0005bc0 0x7f30f0005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:53.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.956+0000 7f310d400640 1 --2- 172.21.15.45:0/3774229069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31080d0230 0x7f30f00073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:53.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.956+0000 7f3107e00640 1 --2- 172.21.15.45:0/3774229069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30f0006a10 0x7f30f0006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:53.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.956+0000 7f310ca00640 1 -- 172.21.15.45:0/3774229069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31080d0230 msgr2=0x7f30f00073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:53.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.956+0000 7f310ca00640 1 --2- 172.21.15.45:0/3774229069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31080d0230 0x7f30f00073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:53.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.956+0000 7f310ca00640 1 -- 172.21.15.45:0/3774229069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30f0006a10 msgr2=0x7f30f0006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:53.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.956+0000 7f310ca00640 1 --2- 172.21.15.45:0/3774229069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30f0006a10 0x7f30f0006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:53.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.956+0000 7f310ca00640 1 -- 172.21.15.45:0/3774229069 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f0005730 con 0x7f30f0005bc0 2024-04-16T16:19:53.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.956+0000 7f3107e00640 1 --2- 172.21.15.45:0/3774229069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30f0006a10 0x7f30f0006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:53.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.957+0000 7f310ca00640 1 --2- 172.21.15.45:0/3774229069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30f0005bc0 0x7f30f0005fe0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009ac0 tx=0x7f30f0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a17f18aeb8e118 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:53.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.957+0000 7f3107400640 1 -- 172.21.15.45:0/3774229069 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc039070 con 0x7f30f0005bc0 2024-04-16T16:19:53.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.957+0000 7f3107400640 1 -- 172.21.15.45:0/3774229069 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30fc02c680 con 0x7f30f0005bc0 2024-04-16T16:19:53.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.957+0000 7f3107400640 1 -- 172.21.15.45:0/3774229069 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc009690 con 0x7f30f0005bc0 2024-04-16T16:19:53.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.958+0000 7f310f400640 1 -- 172.21.15.45:0/3774229069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30f0005bc0 msgr2=0x7f30f0005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:53.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.958+0000 7f310f400640 1 --2- 172.21.15.45:0/3774229069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30f0005bc0 0x7f30f0005fe0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009ac0 tx=0x7f30f0007a40 comp rx=0 tx=0).stop 2024-04-16T16:19:53.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.958+0000 7f310d400640 1 -- 172.21.15.45:0/3774229069 reap_dead start 2024-04-16T16:19:53.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.958+0000 7f310f400640 1 -- 172.21.15.45:0/3774229069 shutdown_connections 2024-04-16T16:19:53.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.958+0000 7f310f400640 1 -- 172.21.15.45:0/3774229069 >> 172.21.15.45:0/3774229069 conn(0x7f31080b2750 msgr2=0x7f31080b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:53.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.958+0000 7f310f400640 1 -- 172.21.15.45:0/3774229069 shutdown_connections 2024-04-16T16:19:53.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.958+0000 7f310f400640 1 -- 172.21.15.45:0/3774229069 wait complete. 2024-04-16T16:19:53.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.960+0000 7f310f400640 1 Processor -- start 2024-04-16T16:19:53.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.960+0000 7f310f400640 1 -- start start 2024-04-16T16:19:53.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.961+0000 7f310f400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f30f0005bc0 0x7f3108138250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:53.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.961+0000 7f310f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f310814c260 0x7f310814c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:53.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.961+0000 7f310f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31081f3970 0x7f31081f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:53.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.961+0000 7f310f400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f310815bb10 con 0x7f31081f3970 2024-04-16T16:19:53.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.961+0000 7f310f400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f31080b5ca0 con 0x7f310814c260 2024-04-16T16:19:53.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.961+0000 7f310f400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f31080731e0 con 0x7f30f0005bc0 2024-04-16T16:19:53.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.962+0000 7f310d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31081f3970 0x7f31081f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:53.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.962+0000 7f310d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31081f3970 0x7f31081f3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57508/0 (socket says 172.21.15.45:57508) 2024-04-16T16:19:53.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.962+0000 7f310d400640 1 -- 172.21.15.45:0/2836320087 learned_addr learned my addr 172.21.15.45:0/2836320087 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:53.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.962+0000 7f3107e00640 1 --2- 172.21.15.45:0/2836320087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f310814c260 0x7f310814c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:53.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.962+0000 7f310ca00640 1 --2- 172.21.15.45:0/2836320087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f30f0005bc0 0x7f3108138250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:53.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.962+0000 7f310d400640 1 -- 172.21.15.45:0/2836320087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f30f0005bc0 msgr2=0x7f3108138250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:53.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.962+0000 7f310d400640 1 --2- 172.21.15.45:0/2836320087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f30f0005bc0 0x7f3108138250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:53.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.962+0000 7f310d400640 1 -- 172.21.15.45:0/2836320087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f310814c260 msgr2=0x7f310814c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:53.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.962+0000 7f310d400640 1 --2- 172.21.15.45:0/2836320087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f310814c260 0x7f310814c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:53.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.962+0000 7f310d400640 1 -- 172.21.15.45:0/2836320087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3108138de0 con 0x7f31081f3970 2024-04-16T16:19:53.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.962+0000 7f310d400640 1 --2- 172.21.15.45:0/2836320087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31081f3970 0x7f31081f3df0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f31080b34e0 tx=0x7f3100009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:53.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.962+0000 7f3104c00640 1 -- 172.21.15.45:0/2836320087 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3100019070 con 0x7f31081f3970 2024-04-16T16:19:53.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.963+0000 7f3104c00640 1 -- 172.21.15.45:0/2836320087 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3100007cb0 con 0x7f31081f3970 2024-04-16T16:19:53.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.963+0000 7f310f400640 1 -- 172.21.15.45:0/2836320087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31081f4630 con 0x7f31081f3970 2024-04-16T16:19:53.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.963+0000 7f3104c00640 1 -- 172.21.15.45:0/2836320087 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3100007e60 con 0x7f31081f3970 2024-04-16T16:19:53.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.963+0000 7f310f400640 1 -- 172.21.15.45:0/2836320087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f310815c110 con 0x7f31081f3970 2024-04-16T16:19:53.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.964+0000 7f310f400640 1 -- 172.21.15.45:0/2836320087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3108075780 con 0x7f31081f3970 2024-04-16T16:19:53.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.965+0000 7f3104c00640 1 -- 172.21.15.45:0/2836320087 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f310001c070 con 0x7f31081f3970 2024-04-16T16:19:53.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.966+0000 7f3104c00640 1 --2- 172.21.15.45:0/2836320087 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f30e806c950 0x7f30e806ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:53.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.966+0000 7f3104c00640 1 -- 172.21.15.45:0/2836320087 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 5709+0+0 (secure 0 0 0) 0x7f310008b740 con 0x7f31081f3970 2024-04-16T16:19:53.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.967+0000 7f310ca00640 1 --2- 172.21.15.45:0/2836320087 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f30e806c950 0x7f30e806ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:53.966 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:19:53.967+0000 7f4f75000640 -1 client.5948: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6421:8b6b9527-b5d7-4e56-90eb-a8840eafe696:v1:172.21.15.103:0/2784388736] 2024-04-16T16:19:53.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.968+0000 7f310ca00640 1 --2- 172.21.15.45:0/2836320087 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f30e806c950 0x7f30e806ee10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f30fc036000 tx=0x7f30fc030000 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:53.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:53.970+0000 7f3104c00640 1 -- 172.21.15.45:0/2836320087 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f31000599f0 con 0x7f31081f3970 2024-04-16T16:19:54.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.234+0000 7f310f400640 1 -- 172.21.15.45:0/2836320087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31080b3d00 con 0x7f31081f3970 2024-04-16T16:19:54.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.235+0000 7f3104c00640 1 -- 172.21.15.45:0/2836320087 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5186 (secure 0 0 0) 0x7f31000592f0 con 0x7f31081f3970 2024-04-16T16:19:54.234 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:54.234 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6056,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1032791151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":1032791151},{"type":"v1","addr":"172.21.15.45:6835","nonce":1032791151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6062,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/498835619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":498835619},{"type":"v1","addr":"172.21.15.103:6837","nonce":498835619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/3611628422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3611628422},{"type":"v1","addr":"172.21.15.103:6836","nonce":3611628422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:19:50.154928+0000","modified":"2024-04-16T16:19:53.221144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8160},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8160":{"gid":8160,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/3429980864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3429980864},{"type":"v1","addr":"172.21.15.45:6834","nonce":3429980864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8160,"qdb_cluster":[8160]},"id":8}]} 2024-04-16T16:19:54.234 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 82 2024-04-16T16:19:54.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.238+0000 7f310f400640 1 -- 172.21.15.45:0/2836320087 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f30e806c950 msgr2=0x7f30e806ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:54.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.238+0000 7f310f400640 1 --2- 172.21.15.45:0/2836320087 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f30e806c950 0x7f30e806ee10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f30fc036000 tx=0x7f30fc030000 comp rx=0 tx=0).stop 2024-04-16T16:19:54.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.238+0000 7f310f400640 1 -- 172.21.15.45:0/2836320087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31081f3970 msgr2=0x7f31081f3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:54.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.238+0000 7f310f400640 1 --2- 172.21.15.45:0/2836320087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31081f3970 0x7f31081f3df0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f31080b34e0 tx=0x7f3100009cd0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.238+0000 7f310f400640 1 -- 172.21.15.45:0/2836320087 shutdown_connections 2024-04-16T16:19:54.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.238+0000 7f310f400640 1 --2- 172.21.15.45:0/2836320087 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f30e806c950 0x7f30e806ee10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.238+0000 7f310f400640 1 --2- 172.21.15.45:0/2836320087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31081f3970 0x7f31081f3df0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.238+0000 7f310f400640 1 --2- 172.21.15.45:0/2836320087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f310814c260 0x7f310814c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.238+0000 7f310f400640 1 --2- 172.21.15.45:0/2836320087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f30f0005bc0 0x7f3108138250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.238+0000 7f310f400640 1 -- 172.21.15.45:0/2836320087 >> 172.21.15.45:0/2836320087 conn(0x7f31080b2750 msgr2=0x7f310814d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:54.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.238+0000 7f310f400640 1 -- 172.21.15.45:0/2836320087 shutdown_connections 2024-04-16T16:19:54.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.238+0000 7f310f400640 1 -- 172.21.15.45:0/2836320087 wait complete. 2024-04-16T16:19:54.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:19:54.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.479+0000 7fda2b600640 1 Processor -- start 2024-04-16T16:19:54.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.480+0000 7fda2b600640 1 -- start start 2024-04-16T16:19:54.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.481+0000 7fda2b600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fda2415a060 0x7fda2406e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:54.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.481+0000 7fda2b600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda24072540 0x7fda240729a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:54.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.481+0000 7fda2b600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda2415a440 con 0x7fda2406f880 2024-04-16T16:19:54.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.481+0000 7fda2b600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fda240b3a50 con 0x7fda2415a060 2024-04-16T16:19:54.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.481+0000 7fda2b600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda240b3c80 con 0x7fda24072540 2024-04-16T16:19:54.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.482+0000 7fda28c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fda2415a060 0x7fda2406e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:54.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.482+0000 7fda28c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fda2415a060 0x7fda2406e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57510/0 (socket says 172.21.15.45:57510) 2024-04-16T16:19:54.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.482+0000 7fda28c00640 1 -- 172.21.15.45:0/3408787696 learned_addr learned my addr 172.21.15.45:0/3408787696 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:54.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.482+0000 7fda28c00640 1 -- 172.21.15.45:0/3408787696 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda24072540 msgr2=0x7fda240729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:54.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.482+0000 7fda28c00640 1 --2- 172.21.15.45:0/3408787696 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda24072540 0x7fda240729a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.482+0000 7fda28c00640 1 -- 172.21.15.45:0/3408787696 >> v1:172.21.15.103:6789/0 conn(0x7fda2406f880 legacy=0x7fda240b4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:54.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.482+0000 7fda28c00640 1 -- 172.21.15.45:0/3408787696 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda240b3f60 con 0x7fda2415a060 2024-04-16T16:19:54.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.483+0000 7fda28c00640 1 --2- 172.21.15.45:0/3408787696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda2415a060 0x7fda2406e2b0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009870 tx=0x7fda0c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ddabe6ef0e5ceb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:54.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.484+0000 7fda23400640 1 -- 172.21.15.45:0/3408787696 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0c0317c0 con 0x7fda2415a060 2024-04-16T16:19:54.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.484+0000 7fda23400640 1 -- 172.21.15.45:0/3408787696 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda0c037070 con 0x7fda2415a060 2024-04-16T16:19:54.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.484+0000 7fda23400640 1 -- 172.21.15.45:0/3408787696 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0c031b80 con 0x7fda2415a060 2024-04-16T16:19:54.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.484+0000 7fda2b600640 1 -- 172.21.15.45:0/3408787696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda2415a060 msgr2=0x7fda2406e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:54.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.484+0000 7fda2b600640 1 --2- 172.21.15.45:0/3408787696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda2415a060 0x7fda2406e2b0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009870 tx=0x7fda0c02b860 comp rx=0 tx=0).stop 2024-04-16T16:19:54.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.485+0000 7fda2b600640 1 -- 172.21.15.45:0/3408787696 shutdown_connections 2024-04-16T16:19:54.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.485+0000 7fda2b600640 1 --2- 172.21.15.45:0/3408787696 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda24072540 0x7fda240729a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.485+0000 7fda2b600640 1 --2- 172.21.15.45:0/3408787696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda2415a060 0x7fda2406e2b0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.485+0000 7fda2b600640 1 -- 172.21.15.45:0/3408787696 >> 172.21.15.45:0/3408787696 conn(0x7fda240b2750 msgr2=0x7fda240b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:54.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.485+0000 7fda2b600640 1 -- 172.21.15.45:0/3408787696 shutdown_connections 2024-04-16T16:19:54.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.485+0000 7fda2b600640 1 -- 172.21.15.45:0/3408787696 wait complete. 2024-04-16T16:19:54.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.486+0000 7fda2b600640 1 Processor -- start 2024-04-16T16:19:54.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.487+0000 7fda2b600640 1 -- start start 2024-04-16T16:19:54.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.487+0000 7fda2b600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda2406f880 0x7fda240b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:54.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.488+0000 7fda2b600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda24072540 0x7fda24151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:54.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.488+0000 7fda2b600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda2415a060 0x7fda24160350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:54.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.488+0000 7fda2b600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fda2406fce0 con 0x7fda24072540 2024-04-16T16:19:54.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.488+0000 7fda2b600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda240b4520 con 0x7fda2406f880 2024-04-16T16:19:54.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.488+0000 7fda2b600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda2406fe60 con 0x7fda2415a060 2024-04-16T16:19:54.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.488+0000 7fda23e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda24072540 0x7fda24151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:54.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.488+0000 7fda23e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda24072540 0x7fda24151b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57518/0 (socket says 172.21.15.45:57518) 2024-04-16T16:19:54.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.488+0000 7fda23e00640 1 -- 172.21.15.45:0/136857010 learned_addr learned my addr 172.21.15.45:0/136857010 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:54.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.488+0000 7fda23e00640 1 -- 172.21.15.45:0/136857010 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda2415a060 msgr2=0x7fda24160350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:54.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.488+0000 7fda23e00640 1 --2- 172.21.15.45:0/136857010 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda2415a060 0x7fda24160350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.488+0000 7fda23e00640 1 -- 172.21.15.45:0/136857010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda2406f880 msgr2=0x7fda240b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:54.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.488+0000 7fda23e00640 1 --2- 172.21.15.45:0/136857010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda2406f880 0x7fda240b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.488+0000 7fda23e00640 1 -- 172.21.15.45:0/136857010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda240b0a50 con 0x7fda24072540 2024-04-16T16:19:54.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.489+0000 7fda23e00640 1 --2- 172.21.15.45:0/136857010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda24072540 0x7fda24151b90 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fda1400b4d0 tx=0x7fda1400b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:54.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.489+0000 7fda20c00640 1 -- 172.21.15.45:0/136857010 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda14014070 con 0x7fda24072540 2024-04-16T16:19:54.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.489+0000 7fda20c00640 1 -- 172.21.15.45:0/136857010 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda14010870 con 0x7fda24072540 2024-04-16T16:19:54.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.489+0000 7fda2b600640 1 -- 172.21.15.45:0/136857010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda240b0d30 con 0x7fda24072540 2024-04-16T16:19:54.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.489+0000 7fda20c00640 1 -- 172.21.15.45:0/136857010 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda14010c60 con 0x7fda24072540 2024-04-16T16:19:54.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.489+0000 7fda2b600640 1 -- 172.21.15.45:0/136857010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda240b12f0 con 0x7fda24072540 2024-04-16T16:19:54.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.490+0000 7fda2b600640 1 -- 172.21.15.45:0/136857010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda24075780 con 0x7fda24072540 2024-04-16T16:19:54.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.491+0000 7fda20c00640 1 -- 172.21.15.45:0/136857010 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7fda14017070 con 0x7fda24072540 2024-04-16T16:19:54.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.492+0000 7fda20c00640 1 --2- 172.21.15.45:0/136857010 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fda0006c980 0x7fda0006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:54.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.492+0000 7fda20c00640 1 -- 172.21.15.45:0/136857010 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 5709+0+0 (secure 0 0 0) 0x7fda1408ba00 con 0x7fda24072540 2024-04-16T16:19:54.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.493+0000 7fda28c00640 1 --2- 172.21.15.45:0/136857010 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fda0006c980 0x7fda0006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:54.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.494+0000 7fda28c00640 1 --2- 172.21.15.45:0/136857010 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fda0006c980 0x7fda0006ee40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fda0c036000 tx=0x7fda0c030000 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:54.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.497+0000 7fda20c00640 1 -- 172.21.15.45:0/136857010 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda14059dd0 con 0x7fda24072540 2024-04-16T16:19:54.584 INFO:tasks.ceph.mds.a.smithi045.stderr:2024-04-16T16:19:54.585+0000 7efc7c200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:19:54.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.707+0000 7fda2b600640 1 -- 172.21.15.45:0/136857010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda24155510 con 0x7fda24072540 2024-04-16T16:19:54.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.708+0000 7fda20c00640 1 -- 172.21.15.45:0/136857010 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) ==== 74+0+16689 (secure 0 0 0) 0x7fda140596d0 con 0x7fda24072540 2024-04-16T16:19:54.707 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:54.707 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":115,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:19:54.695071+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:48.114834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:49.135741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/2784388736":"2024-04-16T17:19:54.469649+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:19:54.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.711+0000 7fda2b600640 1 -- 172.21.15.45:0/136857010 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fda0006c980 msgr2=0x7fda0006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:54.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.711+0000 7fda2b600640 1 --2- 172.21.15.45:0/136857010 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fda0006c980 0x7fda0006ee40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fda0c036000 tx=0x7fda0c030000 comp rx=0 tx=0).stop 2024-04-16T16:19:54.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.711+0000 7fda2b600640 1 -- 172.21.15.45:0/136857010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda24072540 msgr2=0x7fda24151b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:54.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.711+0000 7fda2b600640 1 --2- 172.21.15.45:0/136857010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda24072540 0x7fda24151b90 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fda1400b4d0 tx=0x7fda1400b990 comp rx=0 tx=0).stop 2024-04-16T16:19:54.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.711+0000 7fda2b600640 1 -- 172.21.15.45:0/136857010 shutdown_connections 2024-04-16T16:19:54.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.711+0000 7fda2b600640 1 --2- 172.21.15.45:0/136857010 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda2415a060 0x7fda24160350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.711+0000 7fda2b600640 1 --2- 172.21.15.45:0/136857010 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fda0006c980 0x7fda0006ee40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.711+0000 7fda2b600640 1 --2- 172.21.15.45:0/136857010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda24072540 0x7fda24151b90 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.711+0000 7fda2b600640 1 --2- 172.21.15.45:0/136857010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda2406f880 0x7fda240b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.711+0000 7fda2b600640 1 -- 172.21.15.45:0/136857010 >> 172.21.15.45:0/136857010 conn(0x7fda240b2750 msgr2=0x7fda241548b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:54.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.711+0000 7fda2b600640 1 -- 172.21.15.45:0/136857010 shutdown_connections 2024-04-16T16:19:54.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.711+0000 7fda2b600640 1 -- 172.21.15.45:0/136857010 wait complete. 2024-04-16T16:19:54.732 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-16T16:19:54.732 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:19:54.845 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:19:54.850 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:19:54.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.948+0000 7fe2da800640 1 Processor -- start 2024-04-16T16:19:54.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.948+0000 7fe2da800640 1 -- start start 2024-04-16T16:19:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.950+0000 7fe2da800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe2d4161e90 0x7fe2d40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.950+0000 7fe2da800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d40d52d0 0x7fe2d406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.950+0000 7fe2da800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2d40d3080 con 0x7fe2d4162270 2024-04-16T16:19:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.950+0000 7fe2da800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe2d40d2e30 con 0x7fe2d4161e90 2024-04-16T16:19:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.950+0000 7fe2da800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d40d3360 con 0x7fe2d40d52d0 2024-04-16T16:19:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.950+0000 7fe2d9400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe2d4161e90 0x7fe2d40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.950+0000 7fe2d9400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe2d4161e90 0x7fe2d40cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57526/0 (socket says 172.21.15.45:57526) 2024-04-16T16:19:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.950+0000 7fe2d8a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe2d4162270 0x7fe2d40d3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33774/0 (socket says 172.21.15.45:33774) 2024-04-16T16:19:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.950+0000 7fe2d9400640 1 -- 172.21.15.45:0/3131432825 learned_addr learned my addr 172.21.15.45:0/3131432825 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.951+0000 7fe2d9400640 1 -- 172.21.15.45:0/3131432825 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d40d52d0 msgr2=0x7fe2d406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.951+0000 7fe2d9400640 1 --2- 172.21.15.45:0/3131432825 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d40d52d0 0x7fe2d406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.951+0000 7fe2d9400640 1 -- 172.21.15.45:0/3131432825 >> v1:172.21.15.45:6789/0 conn(0x7fe2d4162270 legacy=0x7fe2d40d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.951+0000 7fe2d9400640 1 -- 172.21.15.45:0/3131432825 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d40d3640 con 0x7fe2d4161e90 2024-04-16T16:19:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.951+0000 7fe2d3e00640 1 -- 172.21.15.45:0/3131432825 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2148100056 0 0) 0x7fe2d40d3080 con 0x7fe2d4162270 2024-04-16T16:19:54.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.952+0000 7fe2d9400640 1 --2- 172.21.15.45:0/3131432825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4161e90 0x7fe2d40cf7d0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fe2c80098a0 tx=0x7fe2c802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42767196d25e23c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:54.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.952+0000 7fe2d3e00640 1 -- 172.21.15.45:0/3131432825 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c802f830 con 0x7fe2d4161e90 2024-04-16T16:19:54.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.952+0000 7fe2d3e00640 1 -- 172.21.15.45:0/3131432825 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2c802f9d0 con 0x7fe2d4161e90 2024-04-16T16:19:54.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.952+0000 7fe2d3e00640 1 -- 172.21.15.45:0/3131432825 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c802fda0 con 0x7fe2d4161e90 2024-04-16T16:19:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.953+0000 7fe2da800640 1 -- 172.21.15.45:0/3131432825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4161e90 msgr2=0x7fe2d40cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.953+0000 7fe2da800640 1 --2- 172.21.15.45:0/3131432825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4161e90 0x7fe2d40cf7d0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fe2c80098a0 tx=0x7fe2c802ba10 comp rx=0 tx=0).stop 2024-04-16T16:19:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.953+0000 7fe2da800640 1 -- 172.21.15.45:0/3131432825 shutdown_connections 2024-04-16T16:19:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.953+0000 7fe2da800640 1 --2- 172.21.15.45:0/3131432825 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d40d52d0 0x7fe2d406ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.953+0000 7fe2da800640 1 --2- 172.21.15.45:0/3131432825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4161e90 0x7fe2d40cf7d0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.953+0000 7fe2da800640 1 -- 172.21.15.45:0/3131432825 >> 172.21.15.45:0/3131432825 conn(0x7fe2d40d1940 msgr2=0x7fe2d40d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.953+0000 7fe2da800640 1 -- 172.21.15.45:0/3131432825 shutdown_connections 2024-04-16T16:19:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.953+0000 7fe2da800640 1 -- 172.21.15.45:0/3131432825 wait complete. 2024-04-16T16:19:54.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.955+0000 7fe2da800640 1 Processor -- start 2024-04-16T16:19:54.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.955+0000 7fe2da800640 1 -- start start 2024-04-16T16:19:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.956+0000 7fe2da800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d40d52d0 0x7fe2d40b0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.956+0000 7fe2da800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4161e90 0x7fe2d4160430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.956+0000 7fe2da800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d4162270 0x7fe2d40ab210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.956+0000 7fe2da800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d40d4550 con 0x7fe2d4161e90 2024-04-16T16:19:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.956+0000 7fe2da800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d40cfd10 con 0x7fe2d4162270 2024-04-16T16:19:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.956+0000 7fe2da800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d40d46d0 con 0x7fe2d40d52d0 2024-04-16T16:19:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.956+0000 7fe2d8a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4161e90 0x7fe2d4160430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.956+0000 7fe2d8a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4161e90 0x7fe2d4160430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57542/0 (socket says 172.21.15.45:57542) 2024-04-16T16:19:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.956+0000 7fe2d8a00640 1 -- 172.21.15.45:0/2432231752 learned_addr learned my addr 172.21.15.45:0/2432231752 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:54.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.957+0000 7fe2d9e00640 1 --2- 172.21.15.45:0/2432231752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d4162270 0x7fe2d40ab210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:54.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.957+0000 7fe2d9400640 1 --2- 172.21.15.45:0/2432231752 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d40d52d0 0x7fe2d40b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:54.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.957+0000 7fe2d8a00640 1 -- 172.21.15.45:0/2432231752 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d40d52d0 msgr2=0x7fe2d40b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:54.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.957+0000 7fe2d8a00640 1 --2- 172.21.15.45:0/2432231752 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d40d52d0 0x7fe2d40b0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.957+0000 7fe2d8a00640 1 -- 172.21.15.45:0/2432231752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d4162270 msgr2=0x7fe2d40ab210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:54.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.957+0000 7fe2d8a00640 1 --2- 172.21.15.45:0/2432231752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d4162270 0x7fe2d40ab210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:54.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.957+0000 7fe2d8a00640 1 -- 172.21.15.45:0/2432231752 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d40abb80 con 0x7fe2d4161e90 2024-04-16T16:19:54.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.957+0000 7fe2d9e00640 1 --2- 172.21.15.45:0/2432231752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d4162270 0x7fe2d40ab210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:54.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.957+0000 7fe2d8a00640 1 --2- 172.21.15.45:0/2432231752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4161e90 0x7fe2d4160430 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000c410 tx=0x7fe2c000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:54.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.957+0000 7fe2d1600640 1 -- 172.21.15.45:0/2432231752 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c00074e0 con 0x7fe2d4161e90 2024-04-16T16:19:54.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.957+0000 7fe2d1600640 1 -- 172.21.15.45:0/2432231752 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2c00047e0 con 0x7fe2d4161e90 2024-04-16T16:19:54.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.957+0000 7fe2d1600640 1 -- 172.21.15.45:0/2432231752 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0004bd0 con 0x7fe2d4161e90 2024-04-16T16:19:54.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.957+0000 7fe2da800640 1 -- 172.21.15.45:0/2432231752 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2d40abe60 con 0x7fe2d4161e90 2024-04-16T16:19:54.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.958+0000 7fe2da800640 1 -- 172.21.15.45:0/2432231752 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2d40ac420 con 0x7fe2d4161e90 2024-04-16T16:19:54.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.958+0000 7fe2da800640 1 -- 172.21.15.45:0/2432231752 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2d4075780 con 0x7fe2d4161e90 2024-04-16T16:19:54.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.960+0000 7fe2d1600640 1 -- 172.21.15.45:0/2432231752 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7fe2c0011070 con 0x7fe2d4161e90 2024-04-16T16:19:54.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.961+0000 7fe2d1600640 1 --2- 172.21.15.45:0/2432231752 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fe2b406c980 0x7fe2b406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:54.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.961+0000 7fe2d1600640 1 -- 172.21.15.45:0/2432231752 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5752+0+0 (secure 0 0 0) 0x7fe2c008b8c0 con 0x7fe2d4161e90 2024-04-16T16:19:54.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.961+0000 7fe2d9400640 1 --2- 172.21.15.45:0/2432231752 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fe2b406c980 0x7fe2b406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:54.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.963+0000 7fe2d9400640 1 --2- 172.21.15.45:0/2432231752 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fe2b406c980 0x7fe2b406ee40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8036000 tx=0x7fe2c8039040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:54.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:54.965+0000 7fe2d1600640 1 -- 172.21.15.45:0/2432231752 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe2c000e280 con 0x7fe2d4161e90 2024-04-16T16:19:55.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.168+0000 7fe2da800640 1 -- 172.21.15.45:0/2432231752 --> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe2d406dae0 con 0x7fe2b406c980 2024-04-16T16:19:55.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.191+0000 7fe2d1600640 1 -- 172.21.15.45:0/2432231752 <== mgr.5921 v2:172.21.15.103:6832/2293932214 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252525 (secure 0 0 0) 0x7fe2d406dae0 con 0x7fe2b406c980 2024-04-16T16:19:55.192 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:19:55.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.197+0000 7fe2da800640 1 -- 172.21.15.45:0/2432231752 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fe2b406c980 msgr2=0x7fe2b406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.197+0000 7fe2da800640 1 --2- 172.21.15.45:0/2432231752 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fe2b406c980 0x7fe2b406ee40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8036000 tx=0x7fe2c8039040 comp rx=0 tx=0).stop 2024-04-16T16:19:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.197+0000 7fe2da800640 1 -- 172.21.15.45:0/2432231752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4161e90 msgr2=0x7fe2d4160430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.197+0000 7fe2da800640 1 --2- 172.21.15.45:0/2432231752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4161e90 0x7fe2d4160430 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000c410 tx=0x7fe2c000c8d0 comp rx=0 tx=0).stop 2024-04-16T16:19:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.198+0000 7fe2da800640 1 -- 172.21.15.45:0/2432231752 shutdown_connections 2024-04-16T16:19:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.198+0000 7fe2da800640 1 --2- 172.21.15.45:0/2432231752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d4162270 0x7fe2d40ab210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.198+0000 7fe2da800640 1 --2- 172.21.15.45:0/2432231752 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fe2b406c980 0x7fe2b406ee40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.198+0000 7fe2da800640 1 --2- 172.21.15.45:0/2432231752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4161e90 0x7fe2d4160430 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.198+0000 7fe2da800640 1 --2- 172.21.15.45:0/2432231752 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d40d52d0 0x7fe2d40b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.198+0000 7fe2da800640 1 -- 172.21.15.45:0/2432231752 >> 172.21.15.45:0/2432231752 conn(0x7fe2d40d1940 msgr2=0x7fe2d40b5ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.198+0000 7fe2da800640 1 -- 172.21.15.45:0/2432231752 shutdown_connections 2024-04-16T16:19:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.198+0000 7fe2da800640 1 -- 172.21.15.45:0/2432231752 wait complete. 2024-04-16T16:19:55.217 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-04-16T16:19:54.818975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":772,"num_read_kb":1413,"num_write":328,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4672,"ondisk_log_size":4672,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":190,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":265928,"kb_used_data":12308,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754708792,"statfs":{"total":773094113280,"available":772821803008,"internally_reserved":0,"allocated":12603392,"data_stored":6566247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":99,"num_read_kb":211,"num_write":65,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.174157"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725914+0000","last_change":"2024-04-16T16:19:50.682531+0000","last_active":"2024-04-16T16:19:54.725914+0000","last_peered":"2024-04-16T16:19:54.725914+0000","last_clean":"2024-04-16T16:19:54.725914+0000","last_became_active":"2024-04-16T16:19:50.682255+0000","last_became_peered":"2024-04-16T16:19:50.682255+0000","last_unstale":"2024-04-16T16:19:54.725914+0000","last_undegraded":"2024-04-16T16:19:54.725914+0000","last_fullsized":"2024-04-16T16:19:54.725914+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:37.810935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.677179+0000","last_change":"2024-04-16T16:19:50.674321+0000","last_active":"2024-04-16T16:19:51.677179+0000","last_peered":"2024-04-16T16:19:51.677179+0000","last_clean":"2024-04-16T16:19:51.677179+0000","last_became_active":"2024-04-16T16:19:50.673878+0000","last_became_peered":"2024-04-16T16:19:50.673878+0000","last_unstale":"2024-04-16T16:19:51.677179+0000","last_undegraded":"2024-04-16T16:19:51.677179+0000","last_fullsized":"2024-04-16T16:19:51.677179+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:51:53.231555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.709285+0000","last_change":"2024-04-16T16:19:50.674306+0000","last_active":"2024-04-16T16:19:54.709285+0000","last_peered":"2024-04-16T16:19:54.709285+0000","last_clean":"2024-04-16T16:19:54.709285+0000","last_became_active":"2024-04-16T16:19:50.672644+0000","last_became_peered":"2024-04-16T16:19:50.672644+0000","last_unstale":"2024-04-16T16:19:54.709285+0000","last_undegraded":"2024-04-16T16:19:54.709285+0000","last_fullsized":"2024-04-16T16:19:54.709285+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:06:56.896419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725512+0000","last_change":"2024-04-16T16:19:50.682471+0000","last_active":"2024-04-16T16:19:54.725512+0000","last_peered":"2024-04-16T16:19:54.725512+0000","last_clean":"2024-04-16T16:19:54.725512+0000","last_became_active":"2024-04-16T16:19:50.675263+0000","last_became_peered":"2024-04-16T16:19:50.675263+0000","last_unstale":"2024-04-16T16:19:54.725512+0000","last_undegraded":"2024-04-16T16:19:54.725512+0000","last_fullsized":"2024-04-16T16:19:54.725512+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:34:35.259518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.704473+0000","last_change":"2024-04-16T16:19:50.670413+0000","last_active":"2024-04-16T16:19:54.704473+0000","last_peered":"2024-04-16T16:19:54.704473+0000","last_clean":"2024-04-16T16:19:54.704473+0000","last_became_active":"2024-04-16T16:19:50.670140+0000","last_became_peered":"2024-04-16T16:19:50.670140+0000","last_unstale":"2024-04-16T16:19:54.704473+0000","last_undegraded":"2024-04-16T16:19:54.704473+0000","last_fullsized":"2024-04-16T16:19:54.704473+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:57.197846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.706577+0000","last_change":"2024-04-16T16:19:50.689165+0000","last_active":"2024-04-16T16:19:54.706577+0000","last_peered":"2024-04-16T16:19:54.706577+0000","last_clean":"2024-04-16T16:19:54.706577+0000","last_became_active":"2024-04-16T16:19:50.688845+0000","last_became_peered":"2024-04-16T16:19:50.688845+0000","last_unstale":"2024-04-16T16:19:54.706577+0000","last_undegraded":"2024-04-16T16:19:54.706577+0000","last_fullsized":"2024-04-16T16:19:54.706577+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:48.655872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.687537+0000","last_change":"2024-04-16T16:19:50.668931+0000","last_active":"2024-04-16T16:19:51.687537+0000","last_peered":"2024-04-16T16:19:51.687537+0000","last_clean":"2024-04-16T16:19:51.687537+0000","last_became_active":"2024-04-16T16:19:50.667238+0000","last_became_peered":"2024-04-16T16:19:50.667238+0000","last_unstale":"2024-04-16T16:19:51.687537+0000","last_undegraded":"2024-04-16T16:19:51.687537+0000","last_fullsized":"2024-04-16T16:19:51.687537+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:15:15.340700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.707774+0000","last_change":"2024-04-16T16:19:50.657825+0000","last_active":"2024-04-16T16:19:54.707774+0000","last_peered":"2024-04-16T16:19:54.707774+0000","last_clean":"2024-04-16T16:19:54.707774+0000","last_became_active":"2024-04-16T16:19:50.657459+0000","last_became_peered":"2024-04-16T16:19:50.657459+0000","last_unstale":"2024-04-16T16:19:54.707774+0000","last_undegraded":"2024-04-16T16:19:54.707774+0000","last_fullsized":"2024-04-16T16:19:54.707774+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:05.270118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.726030+0000","last_change":"2024-04-16T16:19:50.681575+0000","last_active":"2024-04-16T16:19:54.726030+0000","last_peered":"2024-04-16T16:19:54.726030+0000","last_clean":"2024-04-16T16:19:54.726030+0000","last_became_active":"2024-04-16T16:19:50.681301+0000","last_became_peered":"2024-04-16T16:19:50.681301+0000","last_unstale":"2024-04-16T16:19:54.726030+0000","last_undegraded":"2024-04-16T16:19:54.726030+0000","last_fullsized":"2024-04-16T16:19:54.726030+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:00:53.345070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725583+0000","last_change":"2024-04-16T16:19:50.674573+0000","last_active":"2024-04-16T16:19:54.725583+0000","last_peered":"2024-04-16T16:19:54.725583+0000","last_clean":"2024-04-16T16:19:54.725583+0000","last_became_active":"2024-04-16T16:19:50.670996+0000","last_became_peered":"2024-04-16T16:19:50.670996+0000","last_unstale":"2024-04-16T16:19:54.725583+0000","last_undegraded":"2024-04-16T16:19:54.725583+0000","last_fullsized":"2024-04-16T16:19:54.725583+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:24:48.836355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.674143+0000","last_change":"2024-04-16T16:19:50.681365+0000","last_active":"2024-04-16T16:19:51.674143+0000","last_peered":"2024-04-16T16:19:51.674143+0000","last_clean":"2024-04-16T16:19:51.674143+0000","last_became_active":"2024-04-16T16:19:50.681037+0000","last_became_peered":"2024-04-16T16:19:50.681037+0000","last_unstale":"2024-04-16T16:19:51.674143+0000","last_undegraded":"2024-04-16T16:19:51.674143+0000","last_fullsized":"2024-04-16T16:19:51.674143+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:04:12.850817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725425+0000","last_change":"2024-04-16T16:19:50.670303+0000","last_active":"2024-04-16T16:19:54.725425+0000","last_peered":"2024-04-16T16:19:54.725425+0000","last_clean":"2024-04-16T16:19:54.725425+0000","last_became_active":"2024-04-16T16:19:50.667968+0000","last_became_peered":"2024-04-16T16:19:50.667968+0000","last_unstale":"2024-04-16T16:19:54.725425+0000","last_undegraded":"2024-04-16T16:19:54.725425+0000","last_fullsized":"2024-04-16T16:19:54.725425+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:39:36.446379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.708838+0000","last_change":"2024-04-16T16:19:50.650659+0000","last_active":"2024-04-16T16:19:54.708838+0000","last_peered":"2024-04-16T16:19:54.708838+0000","last_clean":"2024-04-16T16:19:54.708838+0000","last_became_active":"2024-04-16T16:19:50.649720+0000","last_became_peered":"2024-04-16T16:19:50.649720+0000","last_unstale":"2024-04-16T16:19:54.708838+0000","last_undegraded":"2024-04-16T16:19:54.708838+0000","last_fullsized":"2024-04-16T16:19:54.708838+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:58:20.314097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.708434+0000","last_change":"2024-04-16T16:19:50.672923+0000","last_active":"2024-04-16T16:19:54.708434+0000","last_peered":"2024-04-16T16:19:54.708434+0000","last_clean":"2024-04-16T16:19:54.708434+0000","last_became_active":"2024-04-16T16:19:50.672333+0000","last_became_peered":"2024-04-16T16:19:50.672333+0000","last_unstale":"2024-04-16T16:19:54.708434+0000","last_undegraded":"2024-04-16T16:19:54.708434+0000","last_fullsized":"2024-04-16T16:19:54.708434+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:51:53.231555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.706287+0000","last_change":"2024-04-16T16:19:50.687724+0000","last_active":"2024-04-16T16:19:54.706287+0000","last_peered":"2024-04-16T16:19:54.706287+0000","last_clean":"2024-04-16T16:19:54.706287+0000","last_became_active":"2024-04-16T16:19:50.687428+0000","last_became_peered":"2024-04-16T16:19:50.687428+0000","last_unstale":"2024-04-16T16:19:54.706287+0000","last_undegraded":"2024-04-16T16:19:54.706287+0000","last_fullsized":"2024-04-16T16:19:54.706287+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:47.641763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725082+0000","last_change":"2024-04-16T16:19:50.658370+0000","last_active":"2024-04-16T16:19:54.725082+0000","last_peered":"2024-04-16T16:19:54.725082+0000","last_clean":"2024-04-16T16:19:54.725082+0000","last_became_active":"2024-04-16T16:19:50.657952+0000","last_became_peered":"2024-04-16T16:19:50.657952+0000","last_unstale":"2024-04-16T16:19:54.725082+0000","last_undegraded":"2024-04-16T16:19:54.725082+0000","last_fullsized":"2024-04-16T16:19:54.725082+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:22:47.893695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.677373+0000","last_change":"2024-04-16T16:19:50.684069+0000","last_active":"2024-04-16T16:19:51.677373+0000","last_peered":"2024-04-16T16:19:51.677373+0000","last_clean":"2024-04-16T16:19:51.677373+0000","last_became_active":"2024-04-16T16:19:50.683703+0000","last_became_peered":"2024-04-16T16:19:50.683703+0000","last_unstale":"2024-04-16T16:19:51.677373+0000","last_undegraded":"2024-04-16T16:19:51.677373+0000","last_fullsized":"2024-04-16T16:19:51.677373+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:05.270118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.680614+0000","last_change":"2024-04-16T16:19:50.676624+0000","last_active":"2024-04-16T16:19:51.680614+0000","last_peered":"2024-04-16T16:19:51.680614+0000","last_clean":"2024-04-16T16:19:51.680614+0000","last_became_active":"2024-04-16T16:19:50.672411+0000","last_became_peered":"2024-04-16T16:19:50.672411+0000","last_unstale":"2024-04-16T16:19:51.680614+0000","last_undegraded":"2024-04-16T16:19:51.680614+0000","last_fullsized":"2024-04-16T16:19:51.680614+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:49:25.637273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.709397+0000","last_change":"2024-04-16T16:19:50.663595+0000","last_active":"2024-04-16T16:19:54.709397+0000","last_peered":"2024-04-16T16:19:54.709397+0000","last_clean":"2024-04-16T16:19:54.709397+0000","last_became_active":"2024-04-16T16:19:50.661986+0000","last_became_peered":"2024-04-16T16:19:50.661986+0000","last_unstale":"2024-04-16T16:19:54.709397+0000","last_undegraded":"2024-04-16T16:19:54.709397+0000","last_fullsized":"2024-04-16T16:19:54.709397+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:46.957471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.676034+0000","last_change":"2024-04-16T16:19:50.676734+0000","last_active":"2024-04-16T16:19:51.676034+0000","last_peered":"2024-04-16T16:19:51.676034+0000","last_clean":"2024-04-16T16:19:51.676034+0000","last_became_active":"2024-04-16T16:19:50.676483+0000","last_became_peered":"2024-04-16T16:19:50.676483+0000","last_unstale":"2024-04-16T16:19:51.676034+0000","last_undegraded":"2024-04-16T16:19:51.676034+0000","last_fullsized":"2024-04-16T16:19:51.676034+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:49:25.637273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.676627+0000","last_change":"2024-04-16T16:19:50.678456+0000","last_active":"2024-04-16T16:19:51.676627+0000","last_peered":"2024-04-16T16:19:51.676627+0000","last_clean":"2024-04-16T16:19:51.676627+0000","last_became_active":"2024-04-16T16:19:50.678029+0000","last_became_peered":"2024-04-16T16:19:50.678029+0000","last_unstale":"2024-04-16T16:19:51.676627+0000","last_undegraded":"2024-04-16T16:19:51.676627+0000","last_fullsized":"2024-04-16T16:19:51.676627+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:29:22.105847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.382890+0000","last_change":"2024-04-16T16:19:50.679734+0000","last_active":"2024-04-16T16:19:52.382890+0000","last_peered":"2024-04-16T16:19:52.382890+0000","last_clean":"2024-04-16T16:19:52.382890+0000","last_became_active":"2024-04-16T16:19:50.679239+0000","last_became_peered":"2024-04-16T16:19:50.679239+0000","last_unstale":"2024-04-16T16:19:52.382890+0000","last_undegraded":"2024-04-16T16:19:52.382890+0000","last_fullsized":"2024-04-16T16:19:52.382890+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:01:51.327322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.674681+0000","last_change":"2024-04-16T16:19:50.661154+0000","last_active":"2024-04-16T16:19:51.674681+0000","last_peered":"2024-04-16T16:19:51.674681+0000","last_clean":"2024-04-16T16:19:51.674681+0000","last_became_active":"2024-04-16T16:19:50.660876+0000","last_became_peered":"2024-04-16T16:19:50.660876+0000","last_unstale":"2024-04-16T16:19:51.674681+0000","last_undegraded":"2024-04-16T16:19:51.674681+0000","last_fullsized":"2024-04-16T16:19:51.674681+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:05:16.700417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.382730+0000","last_change":"2024-04-16T16:19:50.674144+0000","last_active":"2024-04-16T16:19:52.382730+0000","last_peered":"2024-04-16T16:19:52.382730+0000","last_clean":"2024-04-16T16:19:52.382730+0000","last_became_active":"2024-04-16T16:19:50.668800+0000","last_became_peered":"2024-04-16T16:19:50.668800+0000","last_unstale":"2024-04-16T16:19:52.382730+0000","last_undegraded":"2024-04-16T16:19:52.382730+0000","last_fullsized":"2024-04-16T16:19:52.382730+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:58:20.314097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.677630+0000","last_change":"2024-04-16T16:19:50.683501+0000","last_active":"2024-04-16T16:19:51.677630+0000","last_peered":"2024-04-16T16:19:51.677630+0000","last_clean":"2024-04-16T16:19:51.677630+0000","last_became_active":"2024-04-16T16:19:50.683204+0000","last_became_peered":"2024-04-16T16:19:50.683204+0000","last_unstale":"2024-04-16T16:19:51.677630+0000","last_undegraded":"2024-04-16T16:19:51.677630+0000","last_fullsized":"2024-04-16T16:19:51.677630+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:19:20.124893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725718+0000","last_change":"2024-04-16T16:19:50.678764+0000","last_active":"2024-04-16T16:19:54.725718+0000","last_peered":"2024-04-16T16:19:54.725718+0000","last_clean":"2024-04-16T16:19:54.725718+0000","last_became_active":"2024-04-16T16:19:50.667826+0000","last_became_peered":"2024-04-16T16:19:50.667826+0000","last_unstale":"2024-04-16T16:19:54.725718+0000","last_undegraded":"2024-04-16T16:19:54.725718+0000","last_fullsized":"2024-04-16T16:19:54.725718+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:48:08.502336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.707335+0000","last_change":"2024-04-16T16:19:50.663783+0000","last_active":"2024-04-16T16:19:54.707335+0000","last_peered":"2024-04-16T16:19:54.707335+0000","last_clean":"2024-04-16T16:19:54.707335+0000","last_became_active":"2024-04-16T16:19:50.663502+0000","last_became_peered":"2024-04-16T16:19:50.663502+0000","last_unstale":"2024-04-16T16:19:54.707335+0000","last_undegraded":"2024-04-16T16:19:54.707335+0000","last_fullsized":"2024-04-16T16:19:54.707335+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:35:43.600242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.384820+0000","last_change":"2024-04-16T16:19:50.682940+0000","last_active":"2024-04-16T16:19:52.384820+0000","last_peered":"2024-04-16T16:19:52.384820+0000","last_clean":"2024-04-16T16:19:52.384820+0000","last_became_active":"2024-04-16T16:19:50.682640+0000","last_became_peered":"2024-04-16T16:19:50.682640+0000","last_unstale":"2024-04-16T16:19:52.384820+0000","last_undegraded":"2024-04-16T16:19:52.384820+0000","last_fullsized":"2024-04-16T16:19:52.384820+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:18.405203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.676466+0000","last_change":"2024-04-16T16:19:50.677665+0000","last_active":"2024-04-16T16:19:51.676466+0000","last_peered":"2024-04-16T16:19:51.676466+0000","last_clean":"2024-04-16T16:19:51.676466+0000","last_became_active":"2024-04-16T16:19:50.676049+0000","last_became_peered":"2024-04-16T16:19:50.676049+0000","last_unstale":"2024-04-16T16:19:51.676466+0000","last_undegraded":"2024-04-16T16:19:51.676466+0000","last_fullsized":"2024-04-16T16:19:51.676466+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:38:06.236942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.687354+0000","last_change":"2024-04-16T16:19:50.681099+0000","last_active":"2024-04-16T16:19:51.687354+0000","last_peered":"2024-04-16T16:19:51.687354+0000","last_clean":"2024-04-16T16:19:51.687354+0000","last_became_active":"2024-04-16T16:19:50.680798+0000","last_became_peered":"2024-04-16T16:19:50.680798+0000","last_unstale":"2024-04-16T16:19:51.687354+0000","last_undegraded":"2024-04-16T16:19:51.687354+0000","last_fullsized":"2024-04-16T16:19:51.687354+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:38:03.690291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725201+0000","last_change":"2024-04-16T16:19:50.667246+0000","last_active":"2024-04-16T16:19:54.725201+0000","last_peered":"2024-04-16T16:19:54.725201+0000","last_clean":"2024-04-16T16:19:54.725201+0000","last_became_active":"2024-04-16T16:19:50.666936+0000","last_became_peered":"2024-04-16T16:19:50.666936+0000","last_unstale":"2024-04-16T16:19:54.725201+0000","last_undegraded":"2024-04-16T16:19:54.725201+0000","last_fullsized":"2024-04-16T16:19:54.725201+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:21:20.798977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.706398+0000","last_change":"2024-04-16T16:19:50.673087+0000","last_active":"2024-04-16T16:19:54.706398+0000","last_peered":"2024-04-16T16:19:54.706398+0000","last_clean":"2024-04-16T16:19:54.706398+0000","last_became_active":"2024-04-16T16:19:50.672606+0000","last_became_peered":"2024-04-16T16:19:50.672606+0000","last_unstale":"2024-04-16T16:19:54.706398+0000","last_undegraded":"2024-04-16T16:19:54.706398+0000","last_fullsized":"2024-04-16T16:19:54.706398+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:22:17.216999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.707346+0000","last_change":"2024-04-16T16:19:50.662706+0000","last_active":"2024-04-16T16:19:54.707346+0000","last_peered":"2024-04-16T16:19:54.707346+0000","last_clean":"2024-04-16T16:19:54.707346+0000","last_became_active":"2024-04-16T16:19:50.661701+0000","last_became_peered":"2024-04-16T16:19:50.661701+0000","last_unstale":"2024-04-16T16:19:54.707346+0000","last_undegraded":"2024-04-16T16:19:54.707346+0000","last_fullsized":"2024-04-16T16:19:54.707346+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:58:06.052413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'4","reported_seq":18,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.687868+0000","last_change":"2024-04-16T16:19:49.673870+0000","last_active":"2024-04-16T16:19:51.687868+0000","last_peered":"2024-04-16T16:19:51.687868+0000","last_clean":"2024-04-16T16:19:51.687868+0000","last_became_active":"2024-04-16T16:19:49.672964+0000","last_became_peered":"2024-04-16T16:19:49.672964+0000","last_unstale":"2024-04-16T16:19:51.687868+0000","last_undegraded":"2024-04-16T16:19:51.687868+0000","last_fullsized":"2024-04-16T16:19:51.687868+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:32:46.255059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.681721+0000","last_change":"2024-04-16T16:19:50.681269+0000","last_active":"2024-04-16T16:19:51.681721+0000","last_peered":"2024-04-16T16:19:51.681721+0000","last_clean":"2024-04-16T16:19:51.681721+0000","last_became_active":"2024-04-16T16:19:50.680918+0000","last_became_peered":"2024-04-16T16:19:50.680918+0000","last_unstale":"2024-04-16T16:19:51.681721+0000","last_undegraded":"2024-04-16T16:19:51.681721+0000","last_fullsized":"2024-04-16T16:19:51.681721+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:51:53.231555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"113'1","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.709683+0000","last_change":"2024-04-16T16:19:49.677994+0000","last_active":"2024-04-16T16:19:54.709683+0000","last_peered":"2024-04-16T16:19:54.709683+0000","last_clean":"2024-04-16T16:19:54.709683+0000","last_became_active":"2024-04-16T16:19:49.677585+0000","last_became_peered":"2024-04-16T16:19:49.677585+0000","last_unstale":"2024-04-16T16:19:54.709683+0000","last_undegraded":"2024-04-16T16:19:54.709683+0000","last_fullsized":"2024-04-16T16:19:54.709683+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:43:17.373277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.706945+0000","last_change":"2024-04-16T16:19:50.680511+0000","last_active":"2024-04-16T16:19:54.706945+0000","last_peered":"2024-04-16T16:19:54.706945+0000","last_clean":"2024-04-16T16:19:54.706945+0000","last_became_active":"2024-04-16T16:19:50.680299+0000","last_became_peered":"2024-04-16T16:19:50.680299+0000","last_unstale":"2024-04-16T16:19:54.706945+0000","last_undegraded":"2024-04-16T16:19:54.706945+0000","last_fullsized":"2024-04-16T16:19:54.706945+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:24:02.255900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'2","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.704699+0000","last_change":"2024-04-16T16:19:49.672172+0000","last_active":"2024-04-16T16:19:54.704699+0000","last_peered":"2024-04-16T16:19:54.704699+0000","last_clean":"2024-04-16T16:19:54.704699+0000","last_became_active":"2024-04-16T16:19:49.671899+0000","last_became_peered":"2024-04-16T16:19:49.671899+0000","last_unstale":"2024-04-16T16:19:54.704699+0000","last_undegraded":"2024-04-16T16:19:54.704699+0000","last_fullsized":"2024-04-16T16:19:54.704699+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:18:40.025902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.709572+0000","last_change":"2024-04-16T16:19:50.663885+0000","last_active":"2024-04-16T16:19:54.709572+0000","last_peered":"2024-04-16T16:19:54.709572+0000","last_clean":"2024-04-16T16:19:54.709572+0000","last_became_active":"2024-04-16T16:19:50.662482+0000","last_became_peered":"2024-04-16T16:19:50.662482+0000","last_unstale":"2024-04-16T16:19:54.709572+0000","last_undegraded":"2024-04-16T16:19:54.709572+0000","last_fullsized":"2024-04-16T16:19:54.709572+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:55:59.256554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"113'2","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.704100+0000","last_change":"2024-04-16T16:19:49.677951+0000","last_active":"2024-04-16T16:19:54.704100+0000","last_peered":"2024-04-16T16:19:54.704100+0000","last_clean":"2024-04-16T16:19:54.704100+0000","last_became_active":"2024-04-16T16:19:49.677677+0000","last_became_peered":"2024-04-16T16:19:49.677677+0000","last_unstale":"2024-04-16T16:19:54.704100+0000","last_undegraded":"2024-04-16T16:19:54.704100+0000","last_fullsized":"2024-04-16T16:19:54.704100+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:44.913380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725083+0000","last_change":"2024-04-16T16:19:50.683119+0000","last_active":"2024-04-16T16:19:54.725083+0000","last_peered":"2024-04-16T16:19:54.725083+0000","last_clean":"2024-04-16T16:19:54.725083+0000","last_became_active":"2024-04-16T16:19:50.681542+0000","last_became_peered":"2024-04-16T16:19:50.681542+0000","last_unstale":"2024-04-16T16:19:54.725083+0000","last_undegraded":"2024-04-16T16:19:54.725083+0000","last_fullsized":"2024-04-16T16:19:54.725083+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:02:31.650639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.686982+0000","last_change":"2024-04-16T16:19:49.675753+0000","last_active":"2024-04-16T16:19:51.686982+0000","last_peered":"2024-04-16T16:19:51.686982+0000","last_clean":"2024-04-16T16:19:51.686982+0000","last_became_active":"2024-04-16T16:19:49.675136+0000","last_became_peered":"2024-04-16T16:19:49.675136+0000","last_unstale":"2024-04-16T16:19:51.686982+0000","last_undegraded":"2024-04-16T16:19:51.686982+0000","last_fullsized":"2024-04-16T16:19:51.686982+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:58:05.036021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.675548+0000","last_change":"2024-04-16T16:19:50.673753+0000","last_active":"2024-04-16T16:19:51.675548+0000","last_peered":"2024-04-16T16:19:51.675548+0000","last_clean":"2024-04-16T16:19:51.675548+0000","last_became_active":"2024-04-16T16:19:50.673347+0000","last_became_peered":"2024-04-16T16:19:50.673347+0000","last_unstale":"2024-04-16T16:19:51.675548+0000","last_undegraded":"2024-04-16T16:19:51.675548+0000","last_fullsized":"2024-04-16T16:19:51.675548+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:03.079006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.383282+0000","last_change":"2024-04-16T16:19:49.673794+0000","last_active":"2024-04-16T16:19:52.383282+0000","last_peered":"2024-04-16T16:19:52.383282+0000","last_clean":"2024-04-16T16:19:52.383282+0000","last_became_active":"2024-04-16T16:19:49.673425+0000","last_became_peered":"2024-04-16T16:19:49.673425+0000","last_unstale":"2024-04-16T16:19:52.383282+0000","last_undegraded":"2024-04-16T16:19:52.383282+0000","last_fullsized":"2024-04-16T16:19:52.383282+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:19:19.108502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725248+0000","last_change":"2024-04-16T16:19:50.659694+0000","last_active":"2024-04-16T16:19:54.725248+0000","last_peered":"2024-04-16T16:19:54.725248+0000","last_clean":"2024-04-16T16:19:54.725248+0000","last_became_active":"2024-04-16T16:19:50.658672+0000","last_became_peered":"2024-04-16T16:19:50.658672+0000","last_unstale":"2024-04-16T16:19:54.725248+0000","last_undegraded":"2024-04-16T16:19:54.725248+0000","last_fullsized":"2024-04-16T16:19:54.725248+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:10:51.501580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.674923+0000","last_change":"2024-04-16T16:19:49.671749+0000","last_active":"2024-04-16T16:19:51.674923+0000","last_peered":"2024-04-16T16:19:51.674923+0000","last_clean":"2024-04-16T16:19:51.674923+0000","last_became_active":"2024-04-16T16:19:49.671439+0000","last_became_peered":"2024-04-16T16:19:49.671439+0000","last_unstale":"2024-04-16T16:19:51.674923+0000","last_undegraded":"2024-04-16T16:19:51.674923+0000","last_fullsized":"2024-04-16T16:19:51.674923+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:00.905407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.686980+0000","last_change":"2024-04-16T16:19:50.669417+0000","last_active":"2024-04-16T16:19:51.686980+0000","last_peered":"2024-04-16T16:19:51.686980+0000","last_clean":"2024-04-16T16:19:51.686980+0000","last_became_active":"2024-04-16T16:19:50.669075+0000","last_became_peered":"2024-04-16T16:19:50.669075+0000","last_unstale":"2024-04-16T16:19:51.686980+0000","last_undegraded":"2024-04-16T16:19:51.686980+0000","last_fullsized":"2024-04-16T16:19:51.686980+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:22:12.055519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725303+0000","last_change":"2024-04-16T16:19:49.676039+0000","last_active":"2024-04-16T16:19:54.725303+0000","last_peered":"2024-04-16T16:19:54.725303+0000","last_clean":"2024-04-16T16:19:54.725303+0000","last_became_active":"2024-04-16T16:19:49.675780+0000","last_became_peered":"2024-04-16T16:19:49.675780+0000","last_unstale":"2024-04-16T16:19:54.725303+0000","last_undegraded":"2024-04-16T16:19:54.725303+0000","last_fullsized":"2024-04-16T16:19:54.725303+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:57:35.538218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.688188+0000","last_change":"2024-04-16T16:19:50.674409+0000","last_active":"2024-04-16T16:19:51.688188+0000","last_peered":"2024-04-16T16:19:51.688188+0000","last_clean":"2024-04-16T16:19:51.688188+0000","last_became_active":"2024-04-16T16:19:50.674041+0000","last_became_peered":"2024-04-16T16:19:50.674041+0000","last_unstale":"2024-04-16T16:19:51.688188+0000","last_undegraded":"2024-04-16T16:19:51.688188+0000","last_fullsized":"2024-04-16T16:19:51.688188+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:43:18.389668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.704686+0000","last_change":"2024-04-16T16:19:49.676409+0000","last_active":"2024-04-16T16:19:54.704686+0000","last_peered":"2024-04-16T16:19:54.704686+0000","last_clean":"2024-04-16T16:19:54.704686+0000","last_became_active":"2024-04-16T16:19:49.676121+0000","last_became_peered":"2024-04-16T16:19:49.676121+0000","last_unstale":"2024-04-16T16:19:54.704686+0000","last_undegraded":"2024-04-16T16:19:54.704686+0000","last_fullsized":"2024-04-16T16:19:54.704686+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:49:10.626600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.681885+0000","last_change":"2024-04-16T16:19:50.676935+0000","last_active":"2024-04-16T16:19:51.681885+0000","last_peered":"2024-04-16T16:19:51.681885+0000","last_clean":"2024-04-16T16:19:51.681885+0000","last_became_active":"2024-04-16T16:19:50.666863+0000","last_became_peered":"2024-04-16T16:19:50.666863+0000","last_unstale":"2024-04-16T16:19:51.681885+0000","last_undegraded":"2024-04-16T16:19:51.681885+0000","last_fullsized":"2024-04-16T16:19:51.681885+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:05.270118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.384016+0000","last_change":"2024-04-16T16:19:49.674690+0000","last_active":"2024-04-16T16:19:52.384016+0000","last_peered":"2024-04-16T16:19:52.384016+0000","last_clean":"2024-04-16T16:19:52.384016+0000","last_became_active":"2024-04-16T16:19:49.674363+0000","last_became_peered":"2024-04-16T16:19:49.674363+0000","last_unstale":"2024-04-16T16:19:52.384016+0000","last_undegraded":"2024-04-16T16:19:52.384016+0000","last_fullsized":"2024-04-16T16:19:52.384016+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:29:21.089455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.383933+0000","last_change":"2024-04-16T16:19:50.678373+0000","last_active":"2024-04-16T16:19:52.383933+0000","last_peered":"2024-04-16T16:19:52.383933+0000","last_clean":"2024-04-16T16:19:52.383933+0000","last_became_active":"2024-04-16T16:19:50.677344+0000","last_became_peered":"2024-04-16T16:19:50.677344+0000","last_unstale":"2024-04-16T16:19:52.383933+0000","last_undegraded":"2024-04-16T16:19:52.383933+0000","last_fullsized":"2024-04-16T16:19:52.383933+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:05.371427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'3","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725537+0000","last_change":"2024-04-16T16:19:49.675003+0000","last_active":"2024-04-16T16:19:54.725537+0000","last_peered":"2024-04-16T16:19:54.725537+0000","last_clean":"2024-04-16T16:19:54.725537+0000","last_became_active":"2024-04-16T16:19:49.674682+0000","last_became_peered":"2024-04-16T16:19:49.674682+0000","last_unstale":"2024-04-16T16:19:54.725537+0000","last_undegraded":"2024-04-16T16:19:54.725537+0000","last_fullsized":"2024-04-16T16:19:54.725537+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:45.941080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.687656+0000","last_change":"2024-04-16T16:19:50.665057+0000","last_active":"2024-04-16T16:19:51.687656+0000","last_peered":"2024-04-16T16:19:51.687656+0000","last_clean":"2024-04-16T16:19:51.687656+0000","last_became_active":"2024-04-16T16:19:50.664748+0000","last_became_peered":"2024-04-16T16:19:50.664748+0000","last_unstale":"2024-04-16T16:19:51.687656+0000","last_undegraded":"2024-04-16T16:19:51.687656+0000","last_fullsized":"2024-04-16T16:19:51.687656+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:24:52.705209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'6","reported_seq":22,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725736+0000","last_change":"2024-04-16T16:19:49.673684+0000","last_active":"2024-04-16T16:19:54.725736+0000","last_peered":"2024-04-16T16:19:54.725736+0000","last_clean":"2024-04-16T16:19:54.725736+0000","last_became_active":"2024-04-16T16:19:49.673005+0000","last_became_peered":"2024-04-16T16:19:49.673005+0000","last_unstale":"2024-04-16T16:19:54.725736+0000","last_undegraded":"2024-04-16T16:19:54.725736+0000","last_fullsized":"2024-04-16T16:19:54.725736+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:50.287462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.383804+0000","last_change":"2024-04-16T16:19:50.679625+0000","last_active":"2024-04-16T16:19:52.383804+0000","last_peered":"2024-04-16T16:19:52.383804+0000","last_clean":"2024-04-16T16:19:52.383804+0000","last_became_active":"2024-04-16T16:19:50.678843+0000","last_became_peered":"2024-04-16T16:19:50.678843+0000","last_unstale":"2024-04-16T16:19:52.383804+0000","last_undegraded":"2024-04-16T16:19:52.383804+0000","last_fullsized":"2024-04-16T16:19:52.383804+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:04:13.760145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'2","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.706701+0000","last_change":"2024-04-16T16:19:49.674159+0000","last_active":"2024-04-16T16:19:54.706701+0000","last_peered":"2024-04-16T16:19:54.706701+0000","last_clean":"2024-04-16T16:19:54.706701+0000","last_became_active":"2024-04-16T16:19:49.671967+0000","last_became_peered":"2024-04-16T16:19:49.671967+0000","last_unstale":"2024-04-16T16:19:54.706701+0000","last_undegraded":"2024-04-16T16:19:54.706701+0000","last_fullsized":"2024-04-16T16:19:54.706701+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:56.181455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.383467+0000","last_change":"2024-04-16T16:19:50.679016+0000","last_active":"2024-04-16T16:19:52.383467+0000","last_peered":"2024-04-16T16:19:52.383467+0000","last_clean":"2024-04-16T16:19:52.383467+0000","last_became_active":"2024-04-16T16:19:50.678711+0000","last_became_peered":"2024-04-16T16:19:50.678711+0000","last_unstale":"2024-04-16T16:19:52.383467+0000","last_undegraded":"2024-04-16T16:19:52.383467+0000","last_fullsized":"2024-04-16T16:19:52.383467+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:52:52.687219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.680879+0000","last_change":"2024-04-16T16:19:49.671378+0000","last_active":"2024-04-16T16:19:51.680879+0000","last_peered":"2024-04-16T16:19:51.680879+0000","last_clean":"2024-04-16T16:19:51.680879+0000","last_became_active":"2024-04-16T16:19:49.671067+0000","last_became_peered":"2024-04-16T16:19:49.671067+0000","last_unstale":"2024-04-16T16:19:51.680879+0000","last_undegraded":"2024-04-16T16:19:51.680879+0000","last_fullsized":"2024-04-16T16:19:51.680879+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:18:26.405524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.382798+0000","last_change":"2024-04-16T16:19:50.678011+0000","last_active":"2024-04-16T16:19:52.382798+0000","last_peered":"2024-04-16T16:19:52.382798+0000","last_clean":"2024-04-16T16:19:52.382798+0000","last_became_active":"2024-04-16T16:19:50.677636+0000","last_became_peered":"2024-04-16T16:19:50.677636+0000","last_unstale":"2024-04-16T16:19:52.382798+0000","last_undegraded":"2024-04-16T16:19:52.382798+0000","last_fullsized":"2024-04-16T16:19:52.382798+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:11:23.276291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'8","reported_seq":24,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.703637+0000","last_change":"2024-04-16T16:19:49.667874+0000","last_active":"2024-04-16T16:19:54.703637+0000","last_peered":"2024-04-16T16:19:54.703637+0000","last_clean":"2024-04-16T16:19:54.703637+0000","last_became_active":"2024-04-16T16:19:49.665858+0000","last_became_peered":"2024-04-16T16:19:49.665858+0000","last_unstale":"2024-04-16T16:19:54.703637+0000","last_undegraded":"2024-04-16T16:19:54.703637+0000","last_fullsized":"2024-04-16T16:19:54.703637+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:22:35.860023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.705245+0000","last_change":"2024-04-16T16:19:50.660808+0000","last_active":"2024-04-16T16:19:54.705245+0000","last_peered":"2024-04-16T16:19:54.705245+0000","last_clean":"2024-04-16T16:19:54.705245+0000","last_became_active":"2024-04-16T16:19:50.660437+0000","last_became_peered":"2024-04-16T16:19:50.660437+0000","last_unstale":"2024-04-16T16:19:54.705245+0000","last_undegraded":"2024-04-16T16:19:54.705245+0000","last_fullsized":"2024-04-16T16:19:54.705245+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:28:22.910583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'1","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.726289+0000","last_change":"2024-04-16T16:19:49.675020+0000","last_active":"2024-04-16T16:19:54.726289+0000","last_peered":"2024-04-16T16:19:54.726289+0000","last_clean":"2024-04-16T16:19:54.726289+0000","last_became_active":"2024-04-16T16:19:49.672278+0000","last_became_peered":"2024-04-16T16:19:49.672278+0000","last_unstale":"2024-04-16T16:19:54.726289+0000","last_undegraded":"2024-04-16T16:19:54.726289+0000","last_fullsized":"2024-04-16T16:19:54.726289+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:11:22.259899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.707341+0000","last_change":"2024-04-16T16:19:50.665818+0000","last_active":"2024-04-16T16:19:54.707341+0000","last_peered":"2024-04-16T16:19:54.707341+0000","last_clean":"2024-04-16T16:19:54.707341+0000","last_became_active":"2024-04-16T16:19:50.665532+0000","last_became_peered":"2024-04-16T16:19:50.665532+0000","last_unstale":"2024-04-16T16:19:54.707341+0000","last_undegraded":"2024-04-16T16:19:54.707341+0000","last_fullsized":"2024-04-16T16:19:54.707341+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:57:30.715409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'2","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.705477+0000","last_change":"2024-04-16T16:19:49.672793+0000","last_active":"2024-04-16T16:19:54.705477+0000","last_peered":"2024-04-16T16:19:54.705477+0000","last_clean":"2024-04-16T16:19:54.705477+0000","last_became_active":"2024-04-16T16:19:49.668074+0000","last_became_peered":"2024-04-16T16:19:49.668074+0000","last_unstale":"2024-04-16T16:19:54.705477+0000","last_undegraded":"2024-04-16T16:19:54.705477+0000","last_fullsized":"2024-04-16T16:19:54.705477+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:22:29.255995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.709132+0000","last_change":"2024-04-16T16:19:50.669891+0000","last_active":"2024-04-16T16:19:54.709132+0000","last_peered":"2024-04-16T16:19:54.709132+0000","last_clean":"2024-04-16T16:19:54.709132+0000","last_became_active":"2024-04-16T16:19:50.669607+0000","last_became_peered":"2024-04-16T16:19:50.669607+0000","last_unstale":"2024-04-16T16:19:54.709132+0000","last_undegraded":"2024-04-16T16:19:54.709132+0000","last_fullsized":"2024-04-16T16:19:54.709132+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:53:59.247664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.681341+0000","last_change":"2024-04-16T16:19:49.674399+0000","last_active":"2024-04-16T16:19:51.681341+0000","last_peered":"2024-04-16T16:19:51.681341+0000","last_clean":"2024-04-16T16:19:51.681341+0000","last_became_active":"2024-04-16T16:19:49.673597+0000","last_became_peered":"2024-04-16T16:19:49.673597+0000","last_unstale":"2024-04-16T16:19:51.681341+0000","last_undegraded":"2024-04-16T16:19:51.681341+0000","last_fullsized":"2024-04-16T16:19:51.681341+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:51:50.661164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.709646+0000","last_change":"2024-04-16T16:19:50.667992+0000","last_active":"2024-04-16T16:19:54.709646+0000","last_peered":"2024-04-16T16:19:54.709646+0000","last_clean":"2024-04-16T16:19:54.709646+0000","last_became_active":"2024-04-16T16:19:50.667398+0000","last_became_peered":"2024-04-16T16:19:50.667398+0000","last_unstale":"2024-04-16T16:19:54.709646+0000","last_undegraded":"2024-04-16T16:19:54.709646+0000","last_fullsized":"2024-04-16T16:19:54.709646+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:11:23.276291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.687646+0000","last_change":"2024-04-16T16:19:49.669902+0000","last_active":"2024-04-16T16:19:51.687646+0000","last_peered":"2024-04-16T16:19:51.687646+0000","last_clean":"2024-04-16T16:19:51.687646+0000","last_became_active":"2024-04-16T16:19:49.669591+0000","last_became_peered":"2024-04-16T16:19:49.669591+0000","last_unstale":"2024-04-16T16:19:51.687646+0000","last_undegraded":"2024-04-16T16:19:51.687646+0000","last_fullsized":"2024-04-16T16:19:51.687646+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:24:01.239509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.708899+0000","last_change":"2024-04-16T16:19:50.668112+0000","last_active":"2024-04-16T16:19:54.708899+0000","last_peered":"2024-04-16T16:19:54.708899+0000","last_clean":"2024-04-16T16:19:54.708899+0000","last_became_active":"2024-04-16T16:19:50.667682+0000","last_became_peered":"2024-04-16T16:19:50.667682+0000","last_unstale":"2024-04-16T16:19:54.708899+0000","last_undegraded":"2024-04-16T16:19:54.708899+0000","last_fullsized":"2024-04-16T16:19:54.708899+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:19:20.124893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"113'2","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.706842+0000","last_change":"2024-04-16T16:19:49.668597+0000","last_active":"2024-04-16T16:19:54.706842+0000","last_peered":"2024-04-16T16:19:54.706842+0000","last_clean":"2024-04-16T16:19:54.706842+0000","last_became_active":"2024-04-16T16:19:49.668242+0000","last_became_peered":"2024-04-16T16:19:49.668242+0000","last_unstale":"2024-04-16T16:19:54.706842+0000","last_undegraded":"2024-04-16T16:19:54.706842+0000","last_fullsized":"2024-04-16T16:19:54.706842+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:28:21.894192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725373+0000","last_change":"2024-04-16T16:19:50.660829+0000","last_active":"2024-04-16T16:19:54.725373+0000","last_peered":"2024-04-16T16:19:54.725373+0000","last_clean":"2024-04-16T16:19:54.725373+0000","last_became_active":"2024-04-16T16:19:50.660526+0000","last_became_peered":"2024-04-16T16:19:50.660526+0000","last_unstale":"2024-04-16T16:19:54.725373+0000","last_undegraded":"2024-04-16T16:19:54.725373+0000","last_fullsized":"2024-04-16T16:19:54.725373+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:31:50.879022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.703730+0000","last_change":"2024-04-16T16:19:49.676367+0000","last_active":"2024-04-16T16:19:54.703730+0000","last_peered":"2024-04-16T16:19:54.703730+0000","last_clean":"2024-04-16T16:19:54.703730+0000","last_became_active":"2024-04-16T16:19:49.676059+0000","last_became_peered":"2024-04-16T16:19:49.676059+0000","last_unstale":"2024-04-16T16:19:54.703730+0000","last_undegraded":"2024-04-16T16:19:54.703730+0000","last_fullsized":"2024-04-16T16:19:54.703730+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:51.637361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.707946+0000","last_change":"2024-04-16T16:19:50.661144+0000","last_active":"2024-04-16T16:19:54.707946+0000","last_peered":"2024-04-16T16:19:54.707946+0000","last_clean":"2024-04-16T16:19:54.707946+0000","last_became_active":"2024-04-16T16:19:50.660839+0000","last_became_peered":"2024-04-16T16:19:50.660839+0000","last_unstale":"2024-04-16T16:19:54.707946+0000","last_undegraded":"2024-04-16T16:19:54.707946+0000","last_fullsized":"2024-04-16T16:19:54.707946+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:05.864779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.706512+0000","last_change":"2024-04-16T16:19:49.668882+0000","last_active":"2024-04-16T16:19:54.706512+0000","last_peered":"2024-04-16T16:19:54.706512+0000","last_clean":"2024-04-16T16:19:54.706512+0000","last_became_active":"2024-04-16T16:19:49.668608+0000","last_became_peered":"2024-04-16T16:19:49.668608+0000","last_unstale":"2024-04-16T16:19:54.706512+0000","last_undegraded":"2024-04-16T16:19:54.706512+0000","last_fullsized":"2024-04-16T16:19:54.706512+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:21:43.161066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.382719+0000","last_change":"2024-04-16T16:19:50.684964+0000","last_active":"2024-04-16T16:19:52.382719+0000","last_peered":"2024-04-16T16:19:52.382719+0000","last_clean":"2024-04-16T16:19:52.382719+0000","last_became_active":"2024-04-16T16:19:50.684707+0000","last_became_peered":"2024-04-16T16:19:50.684707+0000","last_unstale":"2024-04-16T16:19:52.382719+0000","last_undegraded":"2024-04-16T16:19:52.382719+0000","last_fullsized":"2024-04-16T16:19:52.382719+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:55:59.256554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725779+0000","last_change":"2024-04-16T16:19:49.679453+0000","last_active":"2024-04-16T16:19:54.725779+0000","last_peered":"2024-04-16T16:19:54.725779+0000","last_clean":"2024-04-16T16:19:54.725779+0000","last_became_active":"2024-04-16T16:19:49.678672+0000","last_became_peered":"2024-04-16T16:19:49.678672+0000","last_unstale":"2024-04-16T16:19:54.725779+0000","last_undegraded":"2024-04-16T16:19:54.725779+0000","last_fullsized":"2024-04-16T16:19:54.725779+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:06:55.880027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.681502+0000","last_change":"2024-04-16T16:19:50.681955+0000","last_active":"2024-04-16T16:19:51.681502+0000","last_peered":"2024-04-16T16:19:51.681502+0000","last_clean":"2024-04-16T16:19:51.681502+0000","last_became_active":"2024-04-16T16:19:50.681618+0000","last_became_peered":"2024-04-16T16:19:50.681618+0000","last_unstale":"2024-04-16T16:19:51.681502+0000","last_undegraded":"2024-04-16T16:19:51.681502+0000","last_fullsized":"2024-04-16T16:19:51.681502+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:29:22.105847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.384355+0000","last_change":"2024-04-16T16:19:49.665024+0000","last_active":"2024-04-16T16:19:52.384355+0000","last_peered":"2024-04-16T16:19:52.384355+0000","last_clean":"2024-04-16T16:19:52.384355+0000","last_became_active":"2024-04-16T16:19:49.661483+0000","last_became_peered":"2024-04-16T16:19:49.661483+0000","last_unstale":"2024-04-16T16:19:52.384355+0000","last_undegraded":"2024-04-16T16:19:52.384355+0000","last_fullsized":"2024-04-16T16:19:52.384355+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:38:05.220551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.726192+0000","last_change":"2024-04-16T16:19:50.678108+0000","last_active":"2024-04-16T16:19:54.726192+0000","last_peered":"2024-04-16T16:19:54.726192+0000","last_clean":"2024-04-16T16:19:54.726192+0000","last_became_active":"2024-04-16T16:19:50.677726+0000","last_became_peered":"2024-04-16T16:19:50.677726+0000","last_unstale":"2024-04-16T16:19:54.726192+0000","last_undegraded":"2024-04-16T16:19:54.726192+0000","last_fullsized":"2024-04-16T16:19:54.726192+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:00:58.470213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"114'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.681635+0000","last_change":"2024-04-16T16:19:49.676883+0000","last_active":"2024-04-16T16:19:51.681635+0000","last_peered":"2024-04-16T16:19:51.681635+0000","last_clean":"2024-04-16T16:19:51.681635+0000","last_became_active":"2024-04-16T16:19:49.676018+0000","last_became_peered":"2024-04-16T16:19:49.676018+0000","last_unstale":"2024-04-16T16:19:51.681635+0000","last_undegraded":"2024-04-16T16:19:51.681635+0000","last_fullsized":"2024-04-16T16:19:51.681635+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:23:59.263948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725624+0000","last_change":"2024-04-16T16:19:50.665123+0000","last_active":"2024-04-16T16:19:54.725624+0000","last_peered":"2024-04-16T16:19:54.725624+0000","last_clean":"2024-04-16T16:19:54.725624+0000","last_became_active":"2024-04-16T16:19:50.664736+0000","last_became_peered":"2024-04-16T16:19:50.664736+0000","last_unstale":"2024-04-16T16:19:54.725624+0000","last_undegraded":"2024-04-16T16:19:54.725624+0000","last_fullsized":"2024-04-16T16:19:54.725624+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:46.071347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725584+0000","last_change":"2024-04-16T16:19:49.680209+0000","last_active":"2024-04-16T16:19:54.725584+0000","last_peered":"2024-04-16T16:19:54.725584+0000","last_clean":"2024-04-16T16:19:54.725584+0000","last_became_active":"2024-04-16T16:19:49.679884+0000","last_became_peered":"2024-04-16T16:19:49.679884+0000","last_unstale":"2024-04-16T16:19:54.725584+0000","last_undegraded":"2024-04-16T16:19:54.725584+0000","last_fullsized":"2024-04-16T16:19:54.725584+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:58:19.297705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.681343+0000","last_change":"2024-04-16T16:19:50.671141+0000","last_active":"2024-04-16T16:19:51.681343+0000","last_peered":"2024-04-16T16:19:51.681343+0000","last_clean":"2024-04-16T16:19:51.681343+0000","last_became_active":"2024-04-16T16:19:50.664773+0000","last_became_peered":"2024-04-16T16:19:50.664773+0000","last_unstale":"2024-04-16T16:19:51.681343+0000","last_undegraded":"2024-04-16T16:19:51.681343+0000","last_fullsized":"2024-04-16T16:19:51.681343+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:38:06.236942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.384175+0000","last_change":"2024-04-16T16:19:49.675695+0000","last_active":"2024-04-16T16:19:52.384175+0000","last_peered":"2024-04-16T16:19:52.384175+0000","last_clean":"2024-04-16T16:19:52.384175+0000","last_became_active":"2024-04-16T16:19:49.675414+0000","last_became_peered":"2024-04-16T16:19:49.675414+0000","last_unstale":"2024-04-16T16:19:52.384175+0000","last_undegraded":"2024-04-16T16:19:52.384175+0000","last_fullsized":"2024-04-16T16:19:52.384175+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:49:24.620881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.674652+0000","last_change":"2024-04-16T16:19:50.669255+0000","last_active":"2024-04-16T16:19:51.674652+0000","last_peered":"2024-04-16T16:19:51.674652+0000","last_clean":"2024-04-16T16:19:51.674652+0000","last_became_active":"2024-04-16T16:19:50.668848+0000","last_became_peered":"2024-04-16T16:19:50.668848+0000","last_unstale":"2024-04-16T16:19:51.674652+0000","last_undegraded":"2024-04-16T16:19:51.674652+0000","last_fullsized":"2024-04-16T16:19:51.674652+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:49.761822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.687375+0000","last_change":"2024-04-16T16:19:49.673063+0000","last_active":"2024-04-16T16:19:51.687375+0000","last_peered":"2024-04-16T16:19:51.687375+0000","last_clean":"2024-04-16T16:19:51.687375+0000","last_became_active":"2024-04-16T16:19:49.672631+0000","last_became_peered":"2024-04-16T16:19:49.672631+0000","last_unstale":"2024-04-16T16:19:51.687375+0000","last_undegraded":"2024-04-16T16:19:51.687375+0000","last_fullsized":"2024-04-16T16:19:51.687375+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:57:29.699017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.676030+0000","last_change":"2024-04-16T16:19:50.670272+0000","last_active":"2024-04-16T16:19:51.676030+0000","last_peered":"2024-04-16T16:19:51.676030+0000","last_clean":"2024-04-16T16:19:51.676030+0000","last_became_active":"2024-04-16T16:19:50.669978+0000","last_became_peered":"2024-04-16T16:19:50.669978+0000","last_unstale":"2024-04-16T16:19:51.676030+0000","last_undegraded":"2024-04-16T16:19:51.676030+0000","last_fullsized":"2024-04-16T16:19:51.676030+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:15:01.184007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.674918+0000","last_change":"2024-04-16T16:19:49.676172+0000","last_active":"2024-04-16T16:19:51.674918+0000","last_peered":"2024-04-16T16:19:51.674918+0000","last_clean":"2024-04-16T16:19:51.674918+0000","last_became_active":"2024-04-16T16:19:49.675716+0000","last_became_peered":"2024-04-16T16:19:49.675716+0000","last_unstale":"2024-04-16T16:19:51.674918+0000","last_undegraded":"2024-04-16T16:19:51.674918+0000","last_fullsized":"2024-04-16T16:19:51.674918+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:24:14.986177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.708033+0000","last_change":"2024-04-16T16:19:50.670935+0000","last_active":"2024-04-16T16:19:54.708033+0000","last_peered":"2024-04-16T16:19:54.708033+0000","last_clean":"2024-04-16T16:19:54.708033+0000","last_became_active":"2024-04-16T16:19:50.670647+0000","last_became_peered":"2024-04-16T16:19:50.670647+0000","last_unstale":"2024-04-16T16:19:54.708033+0000","last_undegraded":"2024-04-16T16:19:54.708033+0000","last_fullsized":"2024-04-16T16:19:54.708033+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:57:36.554609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.704041+0000","last_change":"2024-04-16T16:19:49.671536+0000","last_active":"2024-04-16T16:19:54.704041+0000","last_peered":"2024-04-16T16:19:54.704041+0000","last_clean":"2024-04-16T16:19:54.704041+0000","last_became_active":"2024-04-16T16:19:49.671195+0000","last_became_peered":"2024-04-16T16:19:49.671195+0000","last_unstale":"2024-04-16T16:19:54.704041+0000","last_undegraded":"2024-04-16T16:19:54.704041+0000","last_fullsized":"2024-04-16T16:19:54.704041+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:46:28.354522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725337+0000","last_change":"2024-04-16T16:19:50.659536+0000","last_active":"2024-04-16T16:19:54.725337+0000","last_peered":"2024-04-16T16:19:54.725337+0000","last_clean":"2024-04-16T16:19:54.725337+0000","last_became_active":"2024-04-16T16:19:50.659031+0000","last_became_peered":"2024-04-16T16:19:50.659031+0000","last_unstale":"2024-04-16T16:19:54.725337+0000","last_undegraded":"2024-04-16T16:19:54.725337+0000","last_fullsized":"2024-04-16T16:19:54.725337+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:32:20.376846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725907+0000","last_change":"2024-04-16T16:19:49.678362+0000","last_active":"2024-04-16T16:19:54.725907+0000","last_peered":"2024-04-16T16:19:54.725907+0000","last_clean":"2024-04-16T16:19:54.725907+0000","last_became_active":"2024-04-16T16:19:49.678009+0000","last_became_peered":"2024-04-16T16:19:49.678009+0000","last_unstale":"2024-04-16T16:19:54.725907+0000","last_undegraded":"2024-04-16T16:19:54.725907+0000","last_fullsized":"2024-04-16T16:19:54.725907+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:55:58.240162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.382996+0000","last_change":"2024-04-16T16:19:50.672405+0000","last_active":"2024-04-16T16:19:52.382996+0000","last_peered":"2024-04-16T16:19:52.382996+0000","last_clean":"2024-04-16T16:19:52.382996+0000","last_became_active":"2024-04-16T16:19:50.668999+0000","last_became_peered":"2024-04-16T16:19:50.668999+0000","last_unstale":"2024-04-16T16:19:52.382996+0000","last_undegraded":"2024-04-16T16:19:52.382996+0000","last_fullsized":"2024-04-16T16:19:52.382996+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:51.303853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.687343+0000","last_change":"2024-04-16T16:19:49.670869+0000","last_active":"2024-04-16T16:19:51.687343+0000","last_peered":"2024-04-16T16:19:51.687343+0000","last_clean":"2024-04-16T16:19:51.687343+0000","last_became_active":"2024-04-16T16:19:49.670496+0000","last_became_peered":"2024-04-16T16:19:49.670496+0000","last_unstale":"2024-04-16T16:19:51.687343+0000","last_undegraded":"2024-04-16T16:19:51.687343+0000","last_fullsized":"2024-04-16T16:19:51.687343+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:35:42.583851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"115'4633","reported_seq":5102,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.809440+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:19:54.809440+0000","last_peered":"2024-04-16T16:19:54.809440+0000","last_clean":"2024-04-16T16:19:54.809440+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:19:54.809440+0000","last_undegraded":"2024-04-16T16:19:54.809440+0000","last_fullsized":"2024-04-16T16:19:54.809440+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":4633,"log_dups_size":0,"ondisk_log_size":4633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":772,"num_read_kb":1413,"num_write":283,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":772,"num_read_kb":1413,"num_write":283,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4633,"ondisk_log_size":4633,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574925,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33148,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338692,"statfs":{"total":96636764160,"available":96602820608,"internally_reserved":0,"allocated":2019328,"data_stored":1262821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574926,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1421312,"data_stored":671907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574926,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1425408,"data_stored":672441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574927,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1417216,"data_stored":671837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574925,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1433600,"data_stored":673519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1417216,"data_stored":671837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1425408,"data_stored":671961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37336,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334504,"statfs":{"total":96636764160,"available":96598532096,"internally_reserved":0,"allocated":2043904,"data_stored":1269924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:19:55.221 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:19:55.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.479+0000 7f247da00640 1 Processor -- start 2024-04-16T16:19:55.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.480+0000 7f247da00640 1 -- start start 2024-04-16T16:19:55.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.481+0000 7f247da00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2478162270 0x7f24780d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:55.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.481+0000 7f247da00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24780d52d0 0x7f247806ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:55.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.481+0000 7f247da00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24780d3080 con 0x7f2478161e90 2024-04-16T16:19:55.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.481+0000 7f247da00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f24780d2e30 con 0x7f2478162270 2024-04-16T16:19:55.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.481+0000 7f247da00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f24780d3360 con 0x7f24780d52d0 2024-04-16T16:19:55.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.481+0000 7f2477400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2478162270 0x7f24780d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:55.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.481+0000 7f2477400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2478162270 0x7f24780d3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57554/0 (socket says 172.21.15.45:57554) 2024-04-16T16:19:55.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.481+0000 7f2477400640 1 -- 172.21.15.45:0/2956783440 learned_addr learned my addr 172.21.15.45:0/2956783440 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:55.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.482+0000 7f2477400640 1 -- 172.21.15.45:0/2956783440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24780d52d0 msgr2=0x7f247806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:55.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.482+0000 7f2477400640 1 --2- 172.21.15.45:0/2956783440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24780d52d0 0x7f247806ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:55.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.482+0000 7f2477400640 1 -- 172.21.15.45:0/2956783440 >> v1:172.21.15.103:6789/0 conn(0x7f2478161e90 legacy=0x7f24780cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:55.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.482+0000 7f2477400640 1 -- 172.21.15.45:0/2956783440 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24780d3640 con 0x7f2478162270 2024-04-16T16:19:55.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.483+0000 7f2477400640 1 --2- 172.21.15.45:0/2956783440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2478162270 0x7f24780d3e40 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f2460009e80 tx=0x7f246002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd47fba5ccccf71 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:55.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.483+0000 7f2476a00640 1 -- 172.21.15.45:0/2956783440 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246002cae0 con 0x7f2478162270 2024-04-16T16:19:55.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.483+0000 7f2476a00640 1 -- 172.21.15.45:0/2956783440 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f246002cc80 con 0x7f2478162270 2024-04-16T16:19:55.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.483+0000 7f2476a00640 1 -- 172.21.15.45:0/2956783440 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2460034610 con 0x7f2478162270 2024-04-16T16:19:55.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.484+0000 7f247da00640 1 -- 172.21.15.45:0/2956783440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2478162270 msgr2=0x7f24780d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:55.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.484+0000 7f247da00640 1 --2- 172.21.15.45:0/2956783440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2478162270 0x7f24780d3e40 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f2460009e80 tx=0x7f246002bf40 comp rx=0 tx=0).stop 2024-04-16T16:19:55.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.484+0000 7f247da00640 1 -- 172.21.15.45:0/2956783440 shutdown_connections 2024-04-16T16:19:55.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.484+0000 7f247da00640 1 --2- 172.21.15.45:0/2956783440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24780d52d0 0x7f247806ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:55.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.484+0000 7f247da00640 1 --2- 172.21.15.45:0/2956783440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2478162270 0x7f24780d3e40 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:55.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.484+0000 7f247da00640 1 -- 172.21.15.45:0/2956783440 >> 172.21.15.45:0/2956783440 conn(0x7f24780d1940 msgr2=0x7f24780d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:55.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.484+0000 7f247da00640 1 -- 172.21.15.45:0/2956783440 shutdown_connections 2024-04-16T16:19:55.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.484+0000 7f247da00640 1 -- 172.21.15.45:0/2956783440 wait complete. 2024-04-16T16:19:55.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.486+0000 7f247da00640 1 Processor -- start 2024-04-16T16:19:55.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.486+0000 7f247da00640 1 -- start start 2024-04-16T16:19:55.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.487+0000 7f247da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24780d52d0 0x7f24780cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:55.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.487+0000 7f247da00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2478161e90 0x7f24780cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:55.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.487+0000 7f247da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2478162270 0x7f2478154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:55.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.487+0000 7f247da00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f24780d4500 con 0x7f2478162270 2024-04-16T16:19:55.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.487+0000 7f247da00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24780d4380 con 0x7f24780d52d0 2024-04-16T16:19:55.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.487+0000 7f247da00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f24780d4680 con 0x7f2478161e90 2024-04-16T16:19:55.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.487+0000 7f247d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2478162270 0x7f2478154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:55.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.487+0000 7f247d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2478162270 0x7f2478154e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57558/0 (socket says 172.21.15.45:57558) 2024-04-16T16:19:55.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.487+0000 7f247d000640 1 -- 172.21.15.45:0/3238531759 learned_addr learned my addr 172.21.15.45:0/3238531759 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:55.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.488+0000 7f2477e00640 1 --2- 172.21.15.45:0/3238531759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24780d52d0 0x7f24780cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:55.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.488+0000 7f2477400640 1 --2- 172.21.15.45:0/3238531759 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2478161e90 0x7f24780cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:55.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.488+0000 7f247d000640 1 -- 172.21.15.45:0/3238531759 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2478161e90 msgr2=0x7f24780cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:55.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.488+0000 7f247d000640 1 --2- 172.21.15.45:0/3238531759 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2478161e90 0x7f24780cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:55.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.488+0000 7f247d000640 1 -- 172.21.15.45:0/3238531759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24780d52d0 msgr2=0x7f24780cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:55.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.488+0000 7f247d000640 1 --2- 172.21.15.45:0/3238531759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24780d52d0 0x7f24780cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:55.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.488+0000 7f247d000640 1 -- 172.21.15.45:0/3238531759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24780cc020 con 0x7f2478162270 2024-04-16T16:19:55.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.488+0000 7f247d000640 1 --2- 172.21.15.45:0/3238531759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2478162270 0x7f2478154e30 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f247000b3f0 tx=0x7f247000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:55.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.488+0000 7f246fe00640 1 -- 172.21.15.45:0/3238531759 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2470015020 con 0x7f2478162270 2024-04-16T16:19:55.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.488+0000 7f246fe00640 1 -- 172.21.15.45:0/3238531759 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2470011850 con 0x7f2478162270 2024-04-16T16:19:55.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.488+0000 7f246fe00640 1 -- 172.21.15.45:0/3238531759 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2470011c40 con 0x7f2478162270 2024-04-16T16:19:55.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.488+0000 7f247da00640 1 -- 172.21.15.45:0/3238531759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24780cc300 con 0x7f2478162270 2024-04-16T16:19:55.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.489+0000 7f247da00640 1 -- 172.21.15.45:0/3238531759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24780cc8c0 con 0x7f2478162270 2024-04-16T16:19:55.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.490+0000 7f247da00640 1 -- 172.21.15.45:0/3238531759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2478075780 con 0x7f2478162270 2024-04-16T16:19:55.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.490+0000 7f246fe00640 1 -- 172.21.15.45:0/3238531759 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7f2470018020 con 0x7f2478162270 2024-04-16T16:19:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.492+0000 7f246fe00640 1 --2- 172.21.15.45:0/3238531759 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f245406c980 0x7f245406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.492+0000 7f246fe00640 1 -- 172.21.15.45:0/3238531759 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5752+0+0 (secure 0 0 0) 0x7f247008bd10 con 0x7f2478162270 2024-04-16T16:19:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.493+0000 7f2477e00640 1 --2- 172.21.15.45:0/3238531759 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f245406c980 0x7f245406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:55.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.494+0000 7f2477e00640 1 --2- 172.21.15.45:0/3238531759 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f245406c980 0x7f245406ee40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2468007fd0 tx=0x7f2468009040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:55.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.497+0000 7f246fe00640 1 -- 172.21.15.45:0/3238531759 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2470091020 con 0x7f2478162270 2024-04-16T16:19:55.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.701+0000 7f247da00640 1 -- 172.21.15.45:0/3238531759 --> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f24780cb700 con 0x7f245406c980 2024-04-16T16:19:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.722+0000 7f246fe00640 1 -- 172.21.15.45:0/3238531759 <== mgr.5921 v2:172.21.15.103:6832/2293932214 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252525 (secure 0 0 0) 0x7f24780cb700 con 0x7f245406c980 2024-04-16T16:19:55.725 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:55.726 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:19:55.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.728+0000 7f247da00640 1 -- 172.21.15.45:0/3238531759 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f245406c980 msgr2=0x7f245406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.728+0000 7f247da00640 1 --2- 172.21.15.45:0/3238531759 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f245406c980 0x7f245406ee40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2468007fd0 tx=0x7f2468009040 comp rx=0 tx=0).stop 2024-04-16T16:19:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.728+0000 7f247da00640 1 -- 172.21.15.45:0/3238531759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2478162270 msgr2=0x7f2478154e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.728+0000 7f247da00640 1 --2- 172.21.15.45:0/3238531759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2478162270 0x7f2478154e30 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f247000b3f0 tx=0x7f247000b8b0 comp rx=0 tx=0).stop 2024-04-16T16:19:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.728+0000 7f247da00640 1 -- 172.21.15.45:0/3238531759 shutdown_connections 2024-04-16T16:19:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.728+0000 7f247da00640 1 --2- 172.21.15.45:0/3238531759 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f245406c980 0x7f245406ee40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.728+0000 7f247da00640 1 --2- 172.21.15.45:0/3238531759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2478162270 0x7f2478154e30 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.728+0000 7f247da00640 1 --2- 172.21.15.45:0/3238531759 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2478161e90 0x7f24780cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.728+0000 7f247da00640 1 --2- 172.21.15.45:0/3238531759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24780d52d0 0x7f24780cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.728+0000 7f247da00640 1 -- 172.21.15.45:0/3238531759 >> 172.21.15.45:0/3238531759 conn(0x7f24780d1940 msgr2=0x7f24780b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.729+0000 7f247da00640 1 -- 172.21.15.45:0/3238531759 shutdown_connections 2024-04-16T16:19:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:55.729+0000 7f247da00640 1 -- 172.21.15.45:0/3238531759 wait complete. 2024-04-16T16:19:55.750 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-04-16T16:19:54.818975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":772,"num_read_kb":1413,"num_write":328,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4672,"ondisk_log_size":4672,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":190,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":265928,"kb_used_data":12308,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754708792,"statfs":{"total":773094113280,"available":772821803008,"internally_reserved":0,"allocated":12603392,"data_stored":6566247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":99,"num_read_kb":211,"num_write":65,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.174157"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725914+0000","last_change":"2024-04-16T16:19:50.682531+0000","last_active":"2024-04-16T16:19:54.725914+0000","last_peered":"2024-04-16T16:19:54.725914+0000","last_clean":"2024-04-16T16:19:54.725914+0000","last_became_active":"2024-04-16T16:19:50.682255+0000","last_became_peered":"2024-04-16T16:19:50.682255+0000","last_unstale":"2024-04-16T16:19:54.725914+0000","last_undegraded":"2024-04-16T16:19:54.725914+0000","last_fullsized":"2024-04-16T16:19:54.725914+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:09:37.810935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.677179+0000","last_change":"2024-04-16T16:19:50.674321+0000","last_active":"2024-04-16T16:19:51.677179+0000","last_peered":"2024-04-16T16:19:51.677179+0000","last_clean":"2024-04-16T16:19:51.677179+0000","last_became_active":"2024-04-16T16:19:50.673878+0000","last_became_peered":"2024-04-16T16:19:50.673878+0000","last_unstale":"2024-04-16T16:19:51.677179+0000","last_undegraded":"2024-04-16T16:19:51.677179+0000","last_fullsized":"2024-04-16T16:19:51.677179+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:51:53.231555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.709285+0000","last_change":"2024-04-16T16:19:50.674306+0000","last_active":"2024-04-16T16:19:54.709285+0000","last_peered":"2024-04-16T16:19:54.709285+0000","last_clean":"2024-04-16T16:19:54.709285+0000","last_became_active":"2024-04-16T16:19:50.672644+0000","last_became_peered":"2024-04-16T16:19:50.672644+0000","last_unstale":"2024-04-16T16:19:54.709285+0000","last_undegraded":"2024-04-16T16:19:54.709285+0000","last_fullsized":"2024-04-16T16:19:54.709285+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:06:56.896419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725512+0000","last_change":"2024-04-16T16:19:50.682471+0000","last_active":"2024-04-16T16:19:54.725512+0000","last_peered":"2024-04-16T16:19:54.725512+0000","last_clean":"2024-04-16T16:19:54.725512+0000","last_became_active":"2024-04-16T16:19:50.675263+0000","last_became_peered":"2024-04-16T16:19:50.675263+0000","last_unstale":"2024-04-16T16:19:54.725512+0000","last_undegraded":"2024-04-16T16:19:54.725512+0000","last_fullsized":"2024-04-16T16:19:54.725512+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:34:35.259518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.704473+0000","last_change":"2024-04-16T16:19:50.670413+0000","last_active":"2024-04-16T16:19:54.704473+0000","last_peered":"2024-04-16T16:19:54.704473+0000","last_clean":"2024-04-16T16:19:54.704473+0000","last_became_active":"2024-04-16T16:19:50.670140+0000","last_became_peered":"2024-04-16T16:19:50.670140+0000","last_unstale":"2024-04-16T16:19:54.704473+0000","last_undegraded":"2024-04-16T16:19:54.704473+0000","last_fullsized":"2024-04-16T16:19:54.704473+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:57.197846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.706577+0000","last_change":"2024-04-16T16:19:50.689165+0000","last_active":"2024-04-16T16:19:54.706577+0000","last_peered":"2024-04-16T16:19:54.706577+0000","last_clean":"2024-04-16T16:19:54.706577+0000","last_became_active":"2024-04-16T16:19:50.688845+0000","last_became_peered":"2024-04-16T16:19:50.688845+0000","last_unstale":"2024-04-16T16:19:54.706577+0000","last_undegraded":"2024-04-16T16:19:54.706577+0000","last_fullsized":"2024-04-16T16:19:54.706577+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:48.655872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.687537+0000","last_change":"2024-04-16T16:19:50.668931+0000","last_active":"2024-04-16T16:19:51.687537+0000","last_peered":"2024-04-16T16:19:51.687537+0000","last_clean":"2024-04-16T16:19:51.687537+0000","last_became_active":"2024-04-16T16:19:50.667238+0000","last_became_peered":"2024-04-16T16:19:50.667238+0000","last_unstale":"2024-04-16T16:19:51.687537+0000","last_undegraded":"2024-04-16T16:19:51.687537+0000","last_fullsized":"2024-04-16T16:19:51.687537+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:15:15.340700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.707774+0000","last_change":"2024-04-16T16:19:50.657825+0000","last_active":"2024-04-16T16:19:54.707774+0000","last_peered":"2024-04-16T16:19:54.707774+0000","last_clean":"2024-04-16T16:19:54.707774+0000","last_became_active":"2024-04-16T16:19:50.657459+0000","last_became_peered":"2024-04-16T16:19:50.657459+0000","last_unstale":"2024-04-16T16:19:54.707774+0000","last_undegraded":"2024-04-16T16:19:54.707774+0000","last_fullsized":"2024-04-16T16:19:54.707774+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:05.270118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.726030+0000","last_change":"2024-04-16T16:19:50.681575+0000","last_active":"2024-04-16T16:19:54.726030+0000","last_peered":"2024-04-16T16:19:54.726030+0000","last_clean":"2024-04-16T16:19:54.726030+0000","last_became_active":"2024-04-16T16:19:50.681301+0000","last_became_peered":"2024-04-16T16:19:50.681301+0000","last_unstale":"2024-04-16T16:19:54.726030+0000","last_undegraded":"2024-04-16T16:19:54.726030+0000","last_fullsized":"2024-04-16T16:19:54.726030+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:00:53.345070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725583+0000","last_change":"2024-04-16T16:19:50.674573+0000","last_active":"2024-04-16T16:19:54.725583+0000","last_peered":"2024-04-16T16:19:54.725583+0000","last_clean":"2024-04-16T16:19:54.725583+0000","last_became_active":"2024-04-16T16:19:50.670996+0000","last_became_peered":"2024-04-16T16:19:50.670996+0000","last_unstale":"2024-04-16T16:19:54.725583+0000","last_undegraded":"2024-04-16T16:19:54.725583+0000","last_fullsized":"2024-04-16T16:19:54.725583+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:24:48.836355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.674143+0000","last_change":"2024-04-16T16:19:50.681365+0000","last_active":"2024-04-16T16:19:51.674143+0000","last_peered":"2024-04-16T16:19:51.674143+0000","last_clean":"2024-04-16T16:19:51.674143+0000","last_became_active":"2024-04-16T16:19:50.681037+0000","last_became_peered":"2024-04-16T16:19:50.681037+0000","last_unstale":"2024-04-16T16:19:51.674143+0000","last_undegraded":"2024-04-16T16:19:51.674143+0000","last_fullsized":"2024-04-16T16:19:51.674143+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:04:12.850817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725425+0000","last_change":"2024-04-16T16:19:50.670303+0000","last_active":"2024-04-16T16:19:54.725425+0000","last_peered":"2024-04-16T16:19:54.725425+0000","last_clean":"2024-04-16T16:19:54.725425+0000","last_became_active":"2024-04-16T16:19:50.667968+0000","last_became_peered":"2024-04-16T16:19:50.667968+0000","last_unstale":"2024-04-16T16:19:54.725425+0000","last_undegraded":"2024-04-16T16:19:54.725425+0000","last_fullsized":"2024-04-16T16:19:54.725425+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:39:36.446379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.708838+0000","last_change":"2024-04-16T16:19:50.650659+0000","last_active":"2024-04-16T16:19:54.708838+0000","last_peered":"2024-04-16T16:19:54.708838+0000","last_clean":"2024-04-16T16:19:54.708838+0000","last_became_active":"2024-04-16T16:19:50.649720+0000","last_became_peered":"2024-04-16T16:19:50.649720+0000","last_unstale":"2024-04-16T16:19:54.708838+0000","last_undegraded":"2024-04-16T16:19:54.708838+0000","last_fullsized":"2024-04-16T16:19:54.708838+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:58:20.314097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.708434+0000","last_change":"2024-04-16T16:19:50.672923+0000","last_active":"2024-04-16T16:19:54.708434+0000","last_peered":"2024-04-16T16:19:54.708434+0000","last_clean":"2024-04-16T16:19:54.708434+0000","last_became_active":"2024-04-16T16:19:50.672333+0000","last_became_peered":"2024-04-16T16:19:50.672333+0000","last_unstale":"2024-04-16T16:19:54.708434+0000","last_undegraded":"2024-04-16T16:19:54.708434+0000","last_fullsized":"2024-04-16T16:19:54.708434+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:51:53.231555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.706287+0000","last_change":"2024-04-16T16:19:50.687724+0000","last_active":"2024-04-16T16:19:54.706287+0000","last_peered":"2024-04-16T16:19:54.706287+0000","last_clean":"2024-04-16T16:19:54.706287+0000","last_became_active":"2024-04-16T16:19:50.687428+0000","last_became_peered":"2024-04-16T16:19:50.687428+0000","last_unstale":"2024-04-16T16:19:54.706287+0000","last_undegraded":"2024-04-16T16:19:54.706287+0000","last_fullsized":"2024-04-16T16:19:54.706287+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:47.641763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725082+0000","last_change":"2024-04-16T16:19:50.658370+0000","last_active":"2024-04-16T16:19:54.725082+0000","last_peered":"2024-04-16T16:19:54.725082+0000","last_clean":"2024-04-16T16:19:54.725082+0000","last_became_active":"2024-04-16T16:19:50.657952+0000","last_became_peered":"2024-04-16T16:19:50.657952+0000","last_unstale":"2024-04-16T16:19:54.725082+0000","last_undegraded":"2024-04-16T16:19:54.725082+0000","last_fullsized":"2024-04-16T16:19:54.725082+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:22:47.893695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.677373+0000","last_change":"2024-04-16T16:19:50.684069+0000","last_active":"2024-04-16T16:19:51.677373+0000","last_peered":"2024-04-16T16:19:51.677373+0000","last_clean":"2024-04-16T16:19:51.677373+0000","last_became_active":"2024-04-16T16:19:50.683703+0000","last_became_peered":"2024-04-16T16:19:50.683703+0000","last_unstale":"2024-04-16T16:19:51.677373+0000","last_undegraded":"2024-04-16T16:19:51.677373+0000","last_fullsized":"2024-04-16T16:19:51.677373+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:05.270118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.680614+0000","last_change":"2024-04-16T16:19:50.676624+0000","last_active":"2024-04-16T16:19:51.680614+0000","last_peered":"2024-04-16T16:19:51.680614+0000","last_clean":"2024-04-16T16:19:51.680614+0000","last_became_active":"2024-04-16T16:19:50.672411+0000","last_became_peered":"2024-04-16T16:19:50.672411+0000","last_unstale":"2024-04-16T16:19:51.680614+0000","last_undegraded":"2024-04-16T16:19:51.680614+0000","last_fullsized":"2024-04-16T16:19:51.680614+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:49:25.637273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.709397+0000","last_change":"2024-04-16T16:19:50.663595+0000","last_active":"2024-04-16T16:19:54.709397+0000","last_peered":"2024-04-16T16:19:54.709397+0000","last_clean":"2024-04-16T16:19:54.709397+0000","last_became_active":"2024-04-16T16:19:50.661986+0000","last_became_peered":"2024-04-16T16:19:50.661986+0000","last_unstale":"2024-04-16T16:19:54.709397+0000","last_undegraded":"2024-04-16T16:19:54.709397+0000","last_fullsized":"2024-04-16T16:19:54.709397+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:46.957471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.676034+0000","last_change":"2024-04-16T16:19:50.676734+0000","last_active":"2024-04-16T16:19:51.676034+0000","last_peered":"2024-04-16T16:19:51.676034+0000","last_clean":"2024-04-16T16:19:51.676034+0000","last_became_active":"2024-04-16T16:19:50.676483+0000","last_became_peered":"2024-04-16T16:19:50.676483+0000","last_unstale":"2024-04-16T16:19:51.676034+0000","last_undegraded":"2024-04-16T16:19:51.676034+0000","last_fullsized":"2024-04-16T16:19:51.676034+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:49:25.637273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.676627+0000","last_change":"2024-04-16T16:19:50.678456+0000","last_active":"2024-04-16T16:19:51.676627+0000","last_peered":"2024-04-16T16:19:51.676627+0000","last_clean":"2024-04-16T16:19:51.676627+0000","last_became_active":"2024-04-16T16:19:50.678029+0000","last_became_peered":"2024-04-16T16:19:50.678029+0000","last_unstale":"2024-04-16T16:19:51.676627+0000","last_undegraded":"2024-04-16T16:19:51.676627+0000","last_fullsized":"2024-04-16T16:19:51.676627+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:29:22.105847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.382890+0000","last_change":"2024-04-16T16:19:50.679734+0000","last_active":"2024-04-16T16:19:52.382890+0000","last_peered":"2024-04-16T16:19:52.382890+0000","last_clean":"2024-04-16T16:19:52.382890+0000","last_became_active":"2024-04-16T16:19:50.679239+0000","last_became_peered":"2024-04-16T16:19:50.679239+0000","last_unstale":"2024-04-16T16:19:52.382890+0000","last_undegraded":"2024-04-16T16:19:52.382890+0000","last_fullsized":"2024-04-16T16:19:52.382890+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:01:51.327322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.674681+0000","last_change":"2024-04-16T16:19:50.661154+0000","last_active":"2024-04-16T16:19:51.674681+0000","last_peered":"2024-04-16T16:19:51.674681+0000","last_clean":"2024-04-16T16:19:51.674681+0000","last_became_active":"2024-04-16T16:19:50.660876+0000","last_became_peered":"2024-04-16T16:19:50.660876+0000","last_unstale":"2024-04-16T16:19:51.674681+0000","last_undegraded":"2024-04-16T16:19:51.674681+0000","last_fullsized":"2024-04-16T16:19:51.674681+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:05:16.700417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.382730+0000","last_change":"2024-04-16T16:19:50.674144+0000","last_active":"2024-04-16T16:19:52.382730+0000","last_peered":"2024-04-16T16:19:52.382730+0000","last_clean":"2024-04-16T16:19:52.382730+0000","last_became_active":"2024-04-16T16:19:50.668800+0000","last_became_peered":"2024-04-16T16:19:50.668800+0000","last_unstale":"2024-04-16T16:19:52.382730+0000","last_undegraded":"2024-04-16T16:19:52.382730+0000","last_fullsized":"2024-04-16T16:19:52.382730+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:58:20.314097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.677630+0000","last_change":"2024-04-16T16:19:50.683501+0000","last_active":"2024-04-16T16:19:51.677630+0000","last_peered":"2024-04-16T16:19:51.677630+0000","last_clean":"2024-04-16T16:19:51.677630+0000","last_became_active":"2024-04-16T16:19:50.683204+0000","last_became_peered":"2024-04-16T16:19:50.683204+0000","last_unstale":"2024-04-16T16:19:51.677630+0000","last_undegraded":"2024-04-16T16:19:51.677630+0000","last_fullsized":"2024-04-16T16:19:51.677630+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:19:20.124893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725718+0000","last_change":"2024-04-16T16:19:50.678764+0000","last_active":"2024-04-16T16:19:54.725718+0000","last_peered":"2024-04-16T16:19:54.725718+0000","last_clean":"2024-04-16T16:19:54.725718+0000","last_became_active":"2024-04-16T16:19:50.667826+0000","last_became_peered":"2024-04-16T16:19:50.667826+0000","last_unstale":"2024-04-16T16:19:54.725718+0000","last_undegraded":"2024-04-16T16:19:54.725718+0000","last_fullsized":"2024-04-16T16:19:54.725718+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:48:08.502336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.707335+0000","last_change":"2024-04-16T16:19:50.663783+0000","last_active":"2024-04-16T16:19:54.707335+0000","last_peered":"2024-04-16T16:19:54.707335+0000","last_clean":"2024-04-16T16:19:54.707335+0000","last_became_active":"2024-04-16T16:19:50.663502+0000","last_became_peered":"2024-04-16T16:19:50.663502+0000","last_unstale":"2024-04-16T16:19:54.707335+0000","last_undegraded":"2024-04-16T16:19:54.707335+0000","last_fullsized":"2024-04-16T16:19:54.707335+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:35:43.600242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.384820+0000","last_change":"2024-04-16T16:19:50.682940+0000","last_active":"2024-04-16T16:19:52.384820+0000","last_peered":"2024-04-16T16:19:52.384820+0000","last_clean":"2024-04-16T16:19:52.384820+0000","last_became_active":"2024-04-16T16:19:50.682640+0000","last_became_peered":"2024-04-16T16:19:50.682640+0000","last_unstale":"2024-04-16T16:19:52.384820+0000","last_undegraded":"2024-04-16T16:19:52.384820+0000","last_fullsized":"2024-04-16T16:19:52.384820+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:18.405203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.676466+0000","last_change":"2024-04-16T16:19:50.677665+0000","last_active":"2024-04-16T16:19:51.676466+0000","last_peered":"2024-04-16T16:19:51.676466+0000","last_clean":"2024-04-16T16:19:51.676466+0000","last_became_active":"2024-04-16T16:19:50.676049+0000","last_became_peered":"2024-04-16T16:19:50.676049+0000","last_unstale":"2024-04-16T16:19:51.676466+0000","last_undegraded":"2024-04-16T16:19:51.676466+0000","last_fullsized":"2024-04-16T16:19:51.676466+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:38:06.236942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.687354+0000","last_change":"2024-04-16T16:19:50.681099+0000","last_active":"2024-04-16T16:19:51.687354+0000","last_peered":"2024-04-16T16:19:51.687354+0000","last_clean":"2024-04-16T16:19:51.687354+0000","last_became_active":"2024-04-16T16:19:50.680798+0000","last_became_peered":"2024-04-16T16:19:50.680798+0000","last_unstale":"2024-04-16T16:19:51.687354+0000","last_undegraded":"2024-04-16T16:19:51.687354+0000","last_fullsized":"2024-04-16T16:19:51.687354+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:38:03.690291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725201+0000","last_change":"2024-04-16T16:19:50.667246+0000","last_active":"2024-04-16T16:19:54.725201+0000","last_peered":"2024-04-16T16:19:54.725201+0000","last_clean":"2024-04-16T16:19:54.725201+0000","last_became_active":"2024-04-16T16:19:50.666936+0000","last_became_peered":"2024-04-16T16:19:50.666936+0000","last_unstale":"2024-04-16T16:19:54.725201+0000","last_undegraded":"2024-04-16T16:19:54.725201+0000","last_fullsized":"2024-04-16T16:19:54.725201+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:21:20.798977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.706398+0000","last_change":"2024-04-16T16:19:50.673087+0000","last_active":"2024-04-16T16:19:54.706398+0000","last_peered":"2024-04-16T16:19:54.706398+0000","last_clean":"2024-04-16T16:19:54.706398+0000","last_became_active":"2024-04-16T16:19:50.672606+0000","last_became_peered":"2024-04-16T16:19:50.672606+0000","last_unstale":"2024-04-16T16:19:54.706398+0000","last_undegraded":"2024-04-16T16:19:54.706398+0000","last_fullsized":"2024-04-16T16:19:54.706398+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:22:17.216999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.707346+0000","last_change":"2024-04-16T16:19:50.662706+0000","last_active":"2024-04-16T16:19:54.707346+0000","last_peered":"2024-04-16T16:19:54.707346+0000","last_clean":"2024-04-16T16:19:54.707346+0000","last_became_active":"2024-04-16T16:19:50.661701+0000","last_became_peered":"2024-04-16T16:19:50.661701+0000","last_unstale":"2024-04-16T16:19:54.707346+0000","last_undegraded":"2024-04-16T16:19:54.707346+0000","last_fullsized":"2024-04-16T16:19:54.707346+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:58:06.052413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'4","reported_seq":18,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.687868+0000","last_change":"2024-04-16T16:19:49.673870+0000","last_active":"2024-04-16T16:19:51.687868+0000","last_peered":"2024-04-16T16:19:51.687868+0000","last_clean":"2024-04-16T16:19:51.687868+0000","last_became_active":"2024-04-16T16:19:49.672964+0000","last_became_peered":"2024-04-16T16:19:49.672964+0000","last_unstale":"2024-04-16T16:19:51.687868+0000","last_undegraded":"2024-04-16T16:19:51.687868+0000","last_fullsized":"2024-04-16T16:19:51.687868+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:32:46.255059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.681721+0000","last_change":"2024-04-16T16:19:50.681269+0000","last_active":"2024-04-16T16:19:51.681721+0000","last_peered":"2024-04-16T16:19:51.681721+0000","last_clean":"2024-04-16T16:19:51.681721+0000","last_became_active":"2024-04-16T16:19:50.680918+0000","last_became_peered":"2024-04-16T16:19:50.680918+0000","last_unstale":"2024-04-16T16:19:51.681721+0000","last_undegraded":"2024-04-16T16:19:51.681721+0000","last_fullsized":"2024-04-16T16:19:51.681721+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:51:53.231555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"113'1","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.709683+0000","last_change":"2024-04-16T16:19:49.677994+0000","last_active":"2024-04-16T16:19:54.709683+0000","last_peered":"2024-04-16T16:19:54.709683+0000","last_clean":"2024-04-16T16:19:54.709683+0000","last_became_active":"2024-04-16T16:19:49.677585+0000","last_became_peered":"2024-04-16T16:19:49.677585+0000","last_unstale":"2024-04-16T16:19:54.709683+0000","last_undegraded":"2024-04-16T16:19:54.709683+0000","last_fullsized":"2024-04-16T16:19:54.709683+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:43:17.373277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.706945+0000","last_change":"2024-04-16T16:19:50.680511+0000","last_active":"2024-04-16T16:19:54.706945+0000","last_peered":"2024-04-16T16:19:54.706945+0000","last_clean":"2024-04-16T16:19:54.706945+0000","last_became_active":"2024-04-16T16:19:50.680299+0000","last_became_peered":"2024-04-16T16:19:50.680299+0000","last_unstale":"2024-04-16T16:19:54.706945+0000","last_undegraded":"2024-04-16T16:19:54.706945+0000","last_fullsized":"2024-04-16T16:19:54.706945+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:24:02.255900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'2","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.704699+0000","last_change":"2024-04-16T16:19:49.672172+0000","last_active":"2024-04-16T16:19:54.704699+0000","last_peered":"2024-04-16T16:19:54.704699+0000","last_clean":"2024-04-16T16:19:54.704699+0000","last_became_active":"2024-04-16T16:19:49.671899+0000","last_became_peered":"2024-04-16T16:19:49.671899+0000","last_unstale":"2024-04-16T16:19:54.704699+0000","last_undegraded":"2024-04-16T16:19:54.704699+0000","last_fullsized":"2024-04-16T16:19:54.704699+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:18:40.025902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.709572+0000","last_change":"2024-04-16T16:19:50.663885+0000","last_active":"2024-04-16T16:19:54.709572+0000","last_peered":"2024-04-16T16:19:54.709572+0000","last_clean":"2024-04-16T16:19:54.709572+0000","last_became_active":"2024-04-16T16:19:50.662482+0000","last_became_peered":"2024-04-16T16:19:50.662482+0000","last_unstale":"2024-04-16T16:19:54.709572+0000","last_undegraded":"2024-04-16T16:19:54.709572+0000","last_fullsized":"2024-04-16T16:19:54.709572+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:55:59.256554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"113'2","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.704100+0000","last_change":"2024-04-16T16:19:49.677951+0000","last_active":"2024-04-16T16:19:54.704100+0000","last_peered":"2024-04-16T16:19:54.704100+0000","last_clean":"2024-04-16T16:19:54.704100+0000","last_became_active":"2024-04-16T16:19:49.677677+0000","last_became_peered":"2024-04-16T16:19:49.677677+0000","last_unstale":"2024-04-16T16:19:54.704100+0000","last_undegraded":"2024-04-16T16:19:54.704100+0000","last_fullsized":"2024-04-16T16:19:54.704100+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:44.913380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725083+0000","last_change":"2024-04-16T16:19:50.683119+0000","last_active":"2024-04-16T16:19:54.725083+0000","last_peered":"2024-04-16T16:19:54.725083+0000","last_clean":"2024-04-16T16:19:54.725083+0000","last_became_active":"2024-04-16T16:19:50.681542+0000","last_became_peered":"2024-04-16T16:19:50.681542+0000","last_unstale":"2024-04-16T16:19:54.725083+0000","last_undegraded":"2024-04-16T16:19:54.725083+0000","last_fullsized":"2024-04-16T16:19:54.725083+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:02:31.650639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.686982+0000","last_change":"2024-04-16T16:19:49.675753+0000","last_active":"2024-04-16T16:19:51.686982+0000","last_peered":"2024-04-16T16:19:51.686982+0000","last_clean":"2024-04-16T16:19:51.686982+0000","last_became_active":"2024-04-16T16:19:49.675136+0000","last_became_peered":"2024-04-16T16:19:49.675136+0000","last_unstale":"2024-04-16T16:19:51.686982+0000","last_undegraded":"2024-04-16T16:19:51.686982+0000","last_fullsized":"2024-04-16T16:19:51.686982+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:58:05.036021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.675548+0000","last_change":"2024-04-16T16:19:50.673753+0000","last_active":"2024-04-16T16:19:51.675548+0000","last_peered":"2024-04-16T16:19:51.675548+0000","last_clean":"2024-04-16T16:19:51.675548+0000","last_became_active":"2024-04-16T16:19:50.673347+0000","last_became_peered":"2024-04-16T16:19:50.673347+0000","last_unstale":"2024-04-16T16:19:51.675548+0000","last_undegraded":"2024-04-16T16:19:51.675548+0000","last_fullsized":"2024-04-16T16:19:51.675548+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:03.079006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.383282+0000","last_change":"2024-04-16T16:19:49.673794+0000","last_active":"2024-04-16T16:19:52.383282+0000","last_peered":"2024-04-16T16:19:52.383282+0000","last_clean":"2024-04-16T16:19:52.383282+0000","last_became_active":"2024-04-16T16:19:49.673425+0000","last_became_peered":"2024-04-16T16:19:49.673425+0000","last_unstale":"2024-04-16T16:19:52.383282+0000","last_undegraded":"2024-04-16T16:19:52.383282+0000","last_fullsized":"2024-04-16T16:19:52.383282+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:19:19.108502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725248+0000","last_change":"2024-04-16T16:19:50.659694+0000","last_active":"2024-04-16T16:19:54.725248+0000","last_peered":"2024-04-16T16:19:54.725248+0000","last_clean":"2024-04-16T16:19:54.725248+0000","last_became_active":"2024-04-16T16:19:50.658672+0000","last_became_peered":"2024-04-16T16:19:50.658672+0000","last_unstale":"2024-04-16T16:19:54.725248+0000","last_undegraded":"2024-04-16T16:19:54.725248+0000","last_fullsized":"2024-04-16T16:19:54.725248+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:10:51.501580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.674923+0000","last_change":"2024-04-16T16:19:49.671749+0000","last_active":"2024-04-16T16:19:51.674923+0000","last_peered":"2024-04-16T16:19:51.674923+0000","last_clean":"2024-04-16T16:19:51.674923+0000","last_became_active":"2024-04-16T16:19:49.671439+0000","last_became_peered":"2024-04-16T16:19:49.671439+0000","last_unstale":"2024-04-16T16:19:51.674923+0000","last_undegraded":"2024-04-16T16:19:51.674923+0000","last_fullsized":"2024-04-16T16:19:51.674923+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:00.905407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.686980+0000","last_change":"2024-04-16T16:19:50.669417+0000","last_active":"2024-04-16T16:19:51.686980+0000","last_peered":"2024-04-16T16:19:51.686980+0000","last_clean":"2024-04-16T16:19:51.686980+0000","last_became_active":"2024-04-16T16:19:50.669075+0000","last_became_peered":"2024-04-16T16:19:50.669075+0000","last_unstale":"2024-04-16T16:19:51.686980+0000","last_undegraded":"2024-04-16T16:19:51.686980+0000","last_fullsized":"2024-04-16T16:19:51.686980+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:22:12.055519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725303+0000","last_change":"2024-04-16T16:19:49.676039+0000","last_active":"2024-04-16T16:19:54.725303+0000","last_peered":"2024-04-16T16:19:54.725303+0000","last_clean":"2024-04-16T16:19:54.725303+0000","last_became_active":"2024-04-16T16:19:49.675780+0000","last_became_peered":"2024-04-16T16:19:49.675780+0000","last_unstale":"2024-04-16T16:19:54.725303+0000","last_undegraded":"2024-04-16T16:19:54.725303+0000","last_fullsized":"2024-04-16T16:19:54.725303+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:57:35.538218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.688188+0000","last_change":"2024-04-16T16:19:50.674409+0000","last_active":"2024-04-16T16:19:51.688188+0000","last_peered":"2024-04-16T16:19:51.688188+0000","last_clean":"2024-04-16T16:19:51.688188+0000","last_became_active":"2024-04-16T16:19:50.674041+0000","last_became_peered":"2024-04-16T16:19:50.674041+0000","last_unstale":"2024-04-16T16:19:51.688188+0000","last_undegraded":"2024-04-16T16:19:51.688188+0000","last_fullsized":"2024-04-16T16:19:51.688188+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:43:18.389668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.704686+0000","last_change":"2024-04-16T16:19:49.676409+0000","last_active":"2024-04-16T16:19:54.704686+0000","last_peered":"2024-04-16T16:19:54.704686+0000","last_clean":"2024-04-16T16:19:54.704686+0000","last_became_active":"2024-04-16T16:19:49.676121+0000","last_became_peered":"2024-04-16T16:19:49.676121+0000","last_unstale":"2024-04-16T16:19:54.704686+0000","last_undegraded":"2024-04-16T16:19:54.704686+0000","last_fullsized":"2024-04-16T16:19:54.704686+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:49:10.626600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.681885+0000","last_change":"2024-04-16T16:19:50.676935+0000","last_active":"2024-04-16T16:19:51.681885+0000","last_peered":"2024-04-16T16:19:51.681885+0000","last_clean":"2024-04-16T16:19:51.681885+0000","last_became_active":"2024-04-16T16:19:50.666863+0000","last_became_peered":"2024-04-16T16:19:50.666863+0000","last_unstale":"2024-04-16T16:19:51.681885+0000","last_undegraded":"2024-04-16T16:19:51.681885+0000","last_fullsized":"2024-04-16T16:19:51.681885+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:05.270118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.384016+0000","last_change":"2024-04-16T16:19:49.674690+0000","last_active":"2024-04-16T16:19:52.384016+0000","last_peered":"2024-04-16T16:19:52.384016+0000","last_clean":"2024-04-16T16:19:52.384016+0000","last_became_active":"2024-04-16T16:19:49.674363+0000","last_became_peered":"2024-04-16T16:19:49.674363+0000","last_unstale":"2024-04-16T16:19:52.384016+0000","last_undegraded":"2024-04-16T16:19:52.384016+0000","last_fullsized":"2024-04-16T16:19:52.384016+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:29:21.089455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.383933+0000","last_change":"2024-04-16T16:19:50.678373+0000","last_active":"2024-04-16T16:19:52.383933+0000","last_peered":"2024-04-16T16:19:52.383933+0000","last_clean":"2024-04-16T16:19:52.383933+0000","last_became_active":"2024-04-16T16:19:50.677344+0000","last_became_peered":"2024-04-16T16:19:50.677344+0000","last_unstale":"2024-04-16T16:19:52.383933+0000","last_undegraded":"2024-04-16T16:19:52.383933+0000","last_fullsized":"2024-04-16T16:19:52.383933+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:05.371427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'3","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725537+0000","last_change":"2024-04-16T16:19:49.675003+0000","last_active":"2024-04-16T16:19:54.725537+0000","last_peered":"2024-04-16T16:19:54.725537+0000","last_clean":"2024-04-16T16:19:54.725537+0000","last_became_active":"2024-04-16T16:19:49.674682+0000","last_became_peered":"2024-04-16T16:19:49.674682+0000","last_unstale":"2024-04-16T16:19:54.725537+0000","last_undegraded":"2024-04-16T16:19:54.725537+0000","last_fullsized":"2024-04-16T16:19:54.725537+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:45.941080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.687656+0000","last_change":"2024-04-16T16:19:50.665057+0000","last_active":"2024-04-16T16:19:51.687656+0000","last_peered":"2024-04-16T16:19:51.687656+0000","last_clean":"2024-04-16T16:19:51.687656+0000","last_became_active":"2024-04-16T16:19:50.664748+0000","last_became_peered":"2024-04-16T16:19:50.664748+0000","last_unstale":"2024-04-16T16:19:51.687656+0000","last_undegraded":"2024-04-16T16:19:51.687656+0000","last_fullsized":"2024-04-16T16:19:51.687656+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:24:52.705209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'6","reported_seq":22,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725736+0000","last_change":"2024-04-16T16:19:49.673684+0000","last_active":"2024-04-16T16:19:54.725736+0000","last_peered":"2024-04-16T16:19:54.725736+0000","last_clean":"2024-04-16T16:19:54.725736+0000","last_became_active":"2024-04-16T16:19:49.673005+0000","last_became_peered":"2024-04-16T16:19:49.673005+0000","last_unstale":"2024-04-16T16:19:54.725736+0000","last_undegraded":"2024-04-16T16:19:54.725736+0000","last_fullsized":"2024-04-16T16:19:54.725736+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:50.287462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.383804+0000","last_change":"2024-04-16T16:19:50.679625+0000","last_active":"2024-04-16T16:19:52.383804+0000","last_peered":"2024-04-16T16:19:52.383804+0000","last_clean":"2024-04-16T16:19:52.383804+0000","last_became_active":"2024-04-16T16:19:50.678843+0000","last_became_peered":"2024-04-16T16:19:50.678843+0000","last_unstale":"2024-04-16T16:19:52.383804+0000","last_undegraded":"2024-04-16T16:19:52.383804+0000","last_fullsized":"2024-04-16T16:19:52.383804+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:04:13.760145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'2","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.706701+0000","last_change":"2024-04-16T16:19:49.674159+0000","last_active":"2024-04-16T16:19:54.706701+0000","last_peered":"2024-04-16T16:19:54.706701+0000","last_clean":"2024-04-16T16:19:54.706701+0000","last_became_active":"2024-04-16T16:19:49.671967+0000","last_became_peered":"2024-04-16T16:19:49.671967+0000","last_unstale":"2024-04-16T16:19:54.706701+0000","last_undegraded":"2024-04-16T16:19:54.706701+0000","last_fullsized":"2024-04-16T16:19:54.706701+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:18:56.181455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.383467+0000","last_change":"2024-04-16T16:19:50.679016+0000","last_active":"2024-04-16T16:19:52.383467+0000","last_peered":"2024-04-16T16:19:52.383467+0000","last_clean":"2024-04-16T16:19:52.383467+0000","last_became_active":"2024-04-16T16:19:50.678711+0000","last_became_peered":"2024-04-16T16:19:50.678711+0000","last_unstale":"2024-04-16T16:19:52.383467+0000","last_undegraded":"2024-04-16T16:19:52.383467+0000","last_fullsized":"2024-04-16T16:19:52.383467+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:52:52.687219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.680879+0000","last_change":"2024-04-16T16:19:49.671378+0000","last_active":"2024-04-16T16:19:51.680879+0000","last_peered":"2024-04-16T16:19:51.680879+0000","last_clean":"2024-04-16T16:19:51.680879+0000","last_became_active":"2024-04-16T16:19:49.671067+0000","last_became_peered":"2024-04-16T16:19:49.671067+0000","last_unstale":"2024-04-16T16:19:51.680879+0000","last_undegraded":"2024-04-16T16:19:51.680879+0000","last_fullsized":"2024-04-16T16:19:51.680879+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:18:26.405524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.382798+0000","last_change":"2024-04-16T16:19:50.678011+0000","last_active":"2024-04-16T16:19:52.382798+0000","last_peered":"2024-04-16T16:19:52.382798+0000","last_clean":"2024-04-16T16:19:52.382798+0000","last_became_active":"2024-04-16T16:19:50.677636+0000","last_became_peered":"2024-04-16T16:19:50.677636+0000","last_unstale":"2024-04-16T16:19:52.382798+0000","last_undegraded":"2024-04-16T16:19:52.382798+0000","last_fullsized":"2024-04-16T16:19:52.382798+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:11:23.276291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'8","reported_seq":24,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.703637+0000","last_change":"2024-04-16T16:19:49.667874+0000","last_active":"2024-04-16T16:19:54.703637+0000","last_peered":"2024-04-16T16:19:54.703637+0000","last_clean":"2024-04-16T16:19:54.703637+0000","last_became_active":"2024-04-16T16:19:49.665858+0000","last_became_peered":"2024-04-16T16:19:49.665858+0000","last_unstale":"2024-04-16T16:19:54.703637+0000","last_undegraded":"2024-04-16T16:19:54.703637+0000","last_fullsized":"2024-04-16T16:19:54.703637+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:22:35.860023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.705245+0000","last_change":"2024-04-16T16:19:50.660808+0000","last_active":"2024-04-16T16:19:54.705245+0000","last_peered":"2024-04-16T16:19:54.705245+0000","last_clean":"2024-04-16T16:19:54.705245+0000","last_became_active":"2024-04-16T16:19:50.660437+0000","last_became_peered":"2024-04-16T16:19:50.660437+0000","last_unstale":"2024-04-16T16:19:54.705245+0000","last_undegraded":"2024-04-16T16:19:54.705245+0000","last_fullsized":"2024-04-16T16:19:54.705245+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:28:22.910583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'1","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.726289+0000","last_change":"2024-04-16T16:19:49.675020+0000","last_active":"2024-04-16T16:19:54.726289+0000","last_peered":"2024-04-16T16:19:54.726289+0000","last_clean":"2024-04-16T16:19:54.726289+0000","last_became_active":"2024-04-16T16:19:49.672278+0000","last_became_peered":"2024-04-16T16:19:49.672278+0000","last_unstale":"2024-04-16T16:19:54.726289+0000","last_undegraded":"2024-04-16T16:19:54.726289+0000","last_fullsized":"2024-04-16T16:19:54.726289+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:11:22.259899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.707341+0000","last_change":"2024-04-16T16:19:50.665818+0000","last_active":"2024-04-16T16:19:54.707341+0000","last_peered":"2024-04-16T16:19:54.707341+0000","last_clean":"2024-04-16T16:19:54.707341+0000","last_became_active":"2024-04-16T16:19:50.665532+0000","last_became_peered":"2024-04-16T16:19:50.665532+0000","last_unstale":"2024-04-16T16:19:54.707341+0000","last_undegraded":"2024-04-16T16:19:54.707341+0000","last_fullsized":"2024-04-16T16:19:54.707341+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:57:30.715409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'2","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.705477+0000","last_change":"2024-04-16T16:19:49.672793+0000","last_active":"2024-04-16T16:19:54.705477+0000","last_peered":"2024-04-16T16:19:54.705477+0000","last_clean":"2024-04-16T16:19:54.705477+0000","last_became_active":"2024-04-16T16:19:49.668074+0000","last_became_peered":"2024-04-16T16:19:49.668074+0000","last_unstale":"2024-04-16T16:19:54.705477+0000","last_undegraded":"2024-04-16T16:19:54.705477+0000","last_fullsized":"2024-04-16T16:19:54.705477+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:22:29.255995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.709132+0000","last_change":"2024-04-16T16:19:50.669891+0000","last_active":"2024-04-16T16:19:54.709132+0000","last_peered":"2024-04-16T16:19:54.709132+0000","last_clean":"2024-04-16T16:19:54.709132+0000","last_became_active":"2024-04-16T16:19:50.669607+0000","last_became_peered":"2024-04-16T16:19:50.669607+0000","last_unstale":"2024-04-16T16:19:54.709132+0000","last_undegraded":"2024-04-16T16:19:54.709132+0000","last_fullsized":"2024-04-16T16:19:54.709132+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:53:59.247664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.681341+0000","last_change":"2024-04-16T16:19:49.674399+0000","last_active":"2024-04-16T16:19:51.681341+0000","last_peered":"2024-04-16T16:19:51.681341+0000","last_clean":"2024-04-16T16:19:51.681341+0000","last_became_active":"2024-04-16T16:19:49.673597+0000","last_became_peered":"2024-04-16T16:19:49.673597+0000","last_unstale":"2024-04-16T16:19:51.681341+0000","last_undegraded":"2024-04-16T16:19:51.681341+0000","last_fullsized":"2024-04-16T16:19:51.681341+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:51:50.661164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.709646+0000","last_change":"2024-04-16T16:19:50.667992+0000","last_active":"2024-04-16T16:19:54.709646+0000","last_peered":"2024-04-16T16:19:54.709646+0000","last_clean":"2024-04-16T16:19:54.709646+0000","last_became_active":"2024-04-16T16:19:50.667398+0000","last_became_peered":"2024-04-16T16:19:50.667398+0000","last_unstale":"2024-04-16T16:19:54.709646+0000","last_undegraded":"2024-04-16T16:19:54.709646+0000","last_fullsized":"2024-04-16T16:19:54.709646+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:11:23.276291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.687646+0000","last_change":"2024-04-16T16:19:49.669902+0000","last_active":"2024-04-16T16:19:51.687646+0000","last_peered":"2024-04-16T16:19:51.687646+0000","last_clean":"2024-04-16T16:19:51.687646+0000","last_became_active":"2024-04-16T16:19:49.669591+0000","last_became_peered":"2024-04-16T16:19:49.669591+0000","last_unstale":"2024-04-16T16:19:51.687646+0000","last_undegraded":"2024-04-16T16:19:51.687646+0000","last_fullsized":"2024-04-16T16:19:51.687646+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:24:01.239509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.708899+0000","last_change":"2024-04-16T16:19:50.668112+0000","last_active":"2024-04-16T16:19:54.708899+0000","last_peered":"2024-04-16T16:19:54.708899+0000","last_clean":"2024-04-16T16:19:54.708899+0000","last_became_active":"2024-04-16T16:19:50.667682+0000","last_became_peered":"2024-04-16T16:19:50.667682+0000","last_unstale":"2024-04-16T16:19:54.708899+0000","last_undegraded":"2024-04-16T16:19:54.708899+0000","last_fullsized":"2024-04-16T16:19:54.708899+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:19:20.124893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"113'2","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.706842+0000","last_change":"2024-04-16T16:19:49.668597+0000","last_active":"2024-04-16T16:19:54.706842+0000","last_peered":"2024-04-16T16:19:54.706842+0000","last_clean":"2024-04-16T16:19:54.706842+0000","last_became_active":"2024-04-16T16:19:49.668242+0000","last_became_peered":"2024-04-16T16:19:49.668242+0000","last_unstale":"2024-04-16T16:19:54.706842+0000","last_undegraded":"2024-04-16T16:19:54.706842+0000","last_fullsized":"2024-04-16T16:19:54.706842+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:28:21.894192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725373+0000","last_change":"2024-04-16T16:19:50.660829+0000","last_active":"2024-04-16T16:19:54.725373+0000","last_peered":"2024-04-16T16:19:54.725373+0000","last_clean":"2024-04-16T16:19:54.725373+0000","last_became_active":"2024-04-16T16:19:50.660526+0000","last_became_peered":"2024-04-16T16:19:50.660526+0000","last_unstale":"2024-04-16T16:19:54.725373+0000","last_undegraded":"2024-04-16T16:19:54.725373+0000","last_fullsized":"2024-04-16T16:19:54.725373+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:31:50.879022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.703730+0000","last_change":"2024-04-16T16:19:49.676367+0000","last_active":"2024-04-16T16:19:54.703730+0000","last_peered":"2024-04-16T16:19:54.703730+0000","last_clean":"2024-04-16T16:19:54.703730+0000","last_became_active":"2024-04-16T16:19:49.676059+0000","last_became_peered":"2024-04-16T16:19:49.676059+0000","last_unstale":"2024-04-16T16:19:54.703730+0000","last_undegraded":"2024-04-16T16:19:54.703730+0000","last_fullsized":"2024-04-16T16:19:54.703730+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:51.637361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.707946+0000","last_change":"2024-04-16T16:19:50.661144+0000","last_active":"2024-04-16T16:19:54.707946+0000","last_peered":"2024-04-16T16:19:54.707946+0000","last_clean":"2024-04-16T16:19:54.707946+0000","last_became_active":"2024-04-16T16:19:50.660839+0000","last_became_peered":"2024-04-16T16:19:50.660839+0000","last_unstale":"2024-04-16T16:19:54.707946+0000","last_undegraded":"2024-04-16T16:19:54.707946+0000","last_fullsized":"2024-04-16T16:19:54.707946+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:05.864779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.706512+0000","last_change":"2024-04-16T16:19:49.668882+0000","last_active":"2024-04-16T16:19:54.706512+0000","last_peered":"2024-04-16T16:19:54.706512+0000","last_clean":"2024-04-16T16:19:54.706512+0000","last_became_active":"2024-04-16T16:19:49.668608+0000","last_became_peered":"2024-04-16T16:19:49.668608+0000","last_unstale":"2024-04-16T16:19:54.706512+0000","last_undegraded":"2024-04-16T16:19:54.706512+0000","last_fullsized":"2024-04-16T16:19:54.706512+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:21:43.161066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.382719+0000","last_change":"2024-04-16T16:19:50.684964+0000","last_active":"2024-04-16T16:19:52.382719+0000","last_peered":"2024-04-16T16:19:52.382719+0000","last_clean":"2024-04-16T16:19:52.382719+0000","last_became_active":"2024-04-16T16:19:50.684707+0000","last_became_peered":"2024-04-16T16:19:50.684707+0000","last_unstale":"2024-04-16T16:19:52.382719+0000","last_undegraded":"2024-04-16T16:19:52.382719+0000","last_fullsized":"2024-04-16T16:19:52.382719+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:55:59.256554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725779+0000","last_change":"2024-04-16T16:19:49.679453+0000","last_active":"2024-04-16T16:19:54.725779+0000","last_peered":"2024-04-16T16:19:54.725779+0000","last_clean":"2024-04-16T16:19:54.725779+0000","last_became_active":"2024-04-16T16:19:49.678672+0000","last_became_peered":"2024-04-16T16:19:49.678672+0000","last_unstale":"2024-04-16T16:19:54.725779+0000","last_undegraded":"2024-04-16T16:19:54.725779+0000","last_fullsized":"2024-04-16T16:19:54.725779+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:06:55.880027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.681502+0000","last_change":"2024-04-16T16:19:50.681955+0000","last_active":"2024-04-16T16:19:51.681502+0000","last_peered":"2024-04-16T16:19:51.681502+0000","last_clean":"2024-04-16T16:19:51.681502+0000","last_became_active":"2024-04-16T16:19:50.681618+0000","last_became_peered":"2024-04-16T16:19:50.681618+0000","last_unstale":"2024-04-16T16:19:51.681502+0000","last_undegraded":"2024-04-16T16:19:51.681502+0000","last_fullsized":"2024-04-16T16:19:51.681502+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:29:22.105847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.384355+0000","last_change":"2024-04-16T16:19:49.665024+0000","last_active":"2024-04-16T16:19:52.384355+0000","last_peered":"2024-04-16T16:19:52.384355+0000","last_clean":"2024-04-16T16:19:52.384355+0000","last_became_active":"2024-04-16T16:19:49.661483+0000","last_became_peered":"2024-04-16T16:19:49.661483+0000","last_unstale":"2024-04-16T16:19:52.384355+0000","last_undegraded":"2024-04-16T16:19:52.384355+0000","last_fullsized":"2024-04-16T16:19:52.384355+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:38:05.220551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.726192+0000","last_change":"2024-04-16T16:19:50.678108+0000","last_active":"2024-04-16T16:19:54.726192+0000","last_peered":"2024-04-16T16:19:54.726192+0000","last_clean":"2024-04-16T16:19:54.726192+0000","last_became_active":"2024-04-16T16:19:50.677726+0000","last_became_peered":"2024-04-16T16:19:50.677726+0000","last_unstale":"2024-04-16T16:19:54.726192+0000","last_undegraded":"2024-04-16T16:19:54.726192+0000","last_fullsized":"2024-04-16T16:19:54.726192+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:00:58.470213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"114'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.681635+0000","last_change":"2024-04-16T16:19:49.676883+0000","last_active":"2024-04-16T16:19:51.681635+0000","last_peered":"2024-04-16T16:19:51.681635+0000","last_clean":"2024-04-16T16:19:51.681635+0000","last_became_active":"2024-04-16T16:19:49.676018+0000","last_became_peered":"2024-04-16T16:19:49.676018+0000","last_unstale":"2024-04-16T16:19:51.681635+0000","last_undegraded":"2024-04-16T16:19:51.681635+0000","last_fullsized":"2024-04-16T16:19:51.681635+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:23:59.263948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725624+0000","last_change":"2024-04-16T16:19:50.665123+0000","last_active":"2024-04-16T16:19:54.725624+0000","last_peered":"2024-04-16T16:19:54.725624+0000","last_clean":"2024-04-16T16:19:54.725624+0000","last_became_active":"2024-04-16T16:19:50.664736+0000","last_became_peered":"2024-04-16T16:19:50.664736+0000","last_unstale":"2024-04-16T16:19:54.725624+0000","last_undegraded":"2024-04-16T16:19:54.725624+0000","last_fullsized":"2024-04-16T16:19:54.725624+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:13:46.071347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725584+0000","last_change":"2024-04-16T16:19:49.680209+0000","last_active":"2024-04-16T16:19:54.725584+0000","last_peered":"2024-04-16T16:19:54.725584+0000","last_clean":"2024-04-16T16:19:54.725584+0000","last_became_active":"2024-04-16T16:19:49.679884+0000","last_became_peered":"2024-04-16T16:19:49.679884+0000","last_unstale":"2024-04-16T16:19:54.725584+0000","last_undegraded":"2024-04-16T16:19:54.725584+0000","last_fullsized":"2024-04-16T16:19:54.725584+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:58:19.297705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.681343+0000","last_change":"2024-04-16T16:19:50.671141+0000","last_active":"2024-04-16T16:19:51.681343+0000","last_peered":"2024-04-16T16:19:51.681343+0000","last_clean":"2024-04-16T16:19:51.681343+0000","last_became_active":"2024-04-16T16:19:50.664773+0000","last_became_peered":"2024-04-16T16:19:50.664773+0000","last_unstale":"2024-04-16T16:19:51.681343+0000","last_undegraded":"2024-04-16T16:19:51.681343+0000","last_fullsized":"2024-04-16T16:19:51.681343+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:38:06.236942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.384175+0000","last_change":"2024-04-16T16:19:49.675695+0000","last_active":"2024-04-16T16:19:52.384175+0000","last_peered":"2024-04-16T16:19:52.384175+0000","last_clean":"2024-04-16T16:19:52.384175+0000","last_became_active":"2024-04-16T16:19:49.675414+0000","last_became_peered":"2024-04-16T16:19:49.675414+0000","last_unstale":"2024-04-16T16:19:52.384175+0000","last_undegraded":"2024-04-16T16:19:52.384175+0000","last_fullsized":"2024-04-16T16:19:52.384175+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:49:24.620881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.674652+0000","last_change":"2024-04-16T16:19:50.669255+0000","last_active":"2024-04-16T16:19:51.674652+0000","last_peered":"2024-04-16T16:19:51.674652+0000","last_clean":"2024-04-16T16:19:51.674652+0000","last_became_active":"2024-04-16T16:19:50.668848+0000","last_became_peered":"2024-04-16T16:19:50.668848+0000","last_unstale":"2024-04-16T16:19:51.674652+0000","last_undegraded":"2024-04-16T16:19:51.674652+0000","last_fullsized":"2024-04-16T16:19:51.674652+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:49.761822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.687375+0000","last_change":"2024-04-16T16:19:49.673063+0000","last_active":"2024-04-16T16:19:51.687375+0000","last_peered":"2024-04-16T16:19:51.687375+0000","last_clean":"2024-04-16T16:19:51.687375+0000","last_became_active":"2024-04-16T16:19:49.672631+0000","last_became_peered":"2024-04-16T16:19:49.672631+0000","last_unstale":"2024-04-16T16:19:51.687375+0000","last_undegraded":"2024-04-16T16:19:51.687375+0000","last_fullsized":"2024-04-16T16:19:51.687375+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:57:29.699017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.676030+0000","last_change":"2024-04-16T16:19:50.670272+0000","last_active":"2024-04-16T16:19:51.676030+0000","last_peered":"2024-04-16T16:19:51.676030+0000","last_clean":"2024-04-16T16:19:51.676030+0000","last_became_active":"2024-04-16T16:19:50.669978+0000","last_became_peered":"2024-04-16T16:19:50.669978+0000","last_unstale":"2024-04-16T16:19:51.676030+0000","last_undegraded":"2024-04-16T16:19:51.676030+0000","last_fullsized":"2024-04-16T16:19:51.676030+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:15:01.184007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.674918+0000","last_change":"2024-04-16T16:19:49.676172+0000","last_active":"2024-04-16T16:19:51.674918+0000","last_peered":"2024-04-16T16:19:51.674918+0000","last_clean":"2024-04-16T16:19:51.674918+0000","last_became_active":"2024-04-16T16:19:49.675716+0000","last_became_peered":"2024-04-16T16:19:49.675716+0000","last_unstale":"2024-04-16T16:19:51.674918+0000","last_undegraded":"2024-04-16T16:19:51.674918+0000","last_fullsized":"2024-04-16T16:19:51.674918+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:24:14.986177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.708033+0000","last_change":"2024-04-16T16:19:50.670935+0000","last_active":"2024-04-16T16:19:54.708033+0000","last_peered":"2024-04-16T16:19:54.708033+0000","last_clean":"2024-04-16T16:19:54.708033+0000","last_became_active":"2024-04-16T16:19:50.670647+0000","last_became_peered":"2024-04-16T16:19:50.670647+0000","last_unstale":"2024-04-16T16:19:54.708033+0000","last_undegraded":"2024-04-16T16:19:54.708033+0000","last_fullsized":"2024-04-16T16:19:54.708033+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:57:36.554609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.704041+0000","last_change":"2024-04-16T16:19:49.671536+0000","last_active":"2024-04-16T16:19:54.704041+0000","last_peered":"2024-04-16T16:19:54.704041+0000","last_clean":"2024-04-16T16:19:54.704041+0000","last_became_active":"2024-04-16T16:19:49.671195+0000","last_became_peered":"2024-04-16T16:19:49.671195+0000","last_unstale":"2024-04-16T16:19:54.704041+0000","last_undegraded":"2024-04-16T16:19:54.704041+0000","last_fullsized":"2024-04-16T16:19:54.704041+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:46:28.354522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725337+0000","last_change":"2024-04-16T16:19:50.659536+0000","last_active":"2024-04-16T16:19:54.725337+0000","last_peered":"2024-04-16T16:19:54.725337+0000","last_clean":"2024-04-16T16:19:54.725337+0000","last_became_active":"2024-04-16T16:19:50.659031+0000","last_became_peered":"2024-04-16T16:19:50.659031+0000","last_unstale":"2024-04-16T16:19:54.725337+0000","last_undegraded":"2024-04-16T16:19:54.725337+0000","last_fullsized":"2024-04-16T16:19:54.725337+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:32:20.376846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.725907+0000","last_change":"2024-04-16T16:19:49.678362+0000","last_active":"2024-04-16T16:19:54.725907+0000","last_peered":"2024-04-16T16:19:54.725907+0000","last_clean":"2024-04-16T16:19:54.725907+0000","last_became_active":"2024-04-16T16:19:49.678009+0000","last_became_peered":"2024-04-16T16:19:49.678009+0000","last_unstale":"2024-04-16T16:19:54.725907+0000","last_undegraded":"2024-04-16T16:19:54.725907+0000","last_fullsized":"2024-04-16T16:19:54.725907+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:55:58.240162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:52.382996+0000","last_change":"2024-04-16T16:19:50.672405+0000","last_active":"2024-04-16T16:19:52.382996+0000","last_peered":"2024-04-16T16:19:52.382996+0000","last_clean":"2024-04-16T16:19:52.382996+0000","last_became_active":"2024-04-16T16:19:50.668999+0000","last_became_peered":"2024-04-16T16:19:50.668999+0000","last_unstale":"2024-04-16T16:19:52.382996+0000","last_undegraded":"2024-04-16T16:19:52.382996+0000","last_fullsized":"2024-04-16T16:19:52.382996+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:49.622696+0000","last_clean_scrub_stamp":"2024-04-16T16:19:49.622696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:51.303853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-16T16:19:51.687343+0000","last_change":"2024-04-16T16:19:49.670869+0000","last_active":"2024-04-16T16:19:51.687343+0000","last_peered":"2024-04-16T16:19:51.687343+0000","last_clean":"2024-04-16T16:19:51.687343+0000","last_became_active":"2024-04-16T16:19:49.670496+0000","last_became_peered":"2024-04-16T16:19:49.670496+0000","last_unstale":"2024-04-16T16:19:51.687343+0000","last_undegraded":"2024-04-16T16:19:51.687343+0000","last_fullsized":"2024-04-16T16:19:51.687343+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:19:48.606305+0000","last_clean_scrub_stamp":"2024-04-16T16:19:48.606305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:35:42.583851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"115'4633","reported_seq":5102,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-16T16:19:54.809440+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:19:54.809440+0000","last_peered":"2024-04-16T16:19:54.809440+0000","last_clean":"2024-04-16T16:19:54.809440+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:19:54.809440+0000","last_undegraded":"2024-04-16T16:19:54.809440+0000","last_fullsized":"2024-04-16T16:19:54.809440+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":4633,"log_dups_size":0,"ondisk_log_size":4633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":772,"num_read_kb":1413,"num_write":283,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":772,"num_read_kb":1413,"num_write":283,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4633,"ondisk_log_size":4633,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574925,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33148,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338692,"statfs":{"total":96636764160,"available":96602820608,"internally_reserved":0,"allocated":2019328,"data_stored":1262821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574926,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1421312,"data_stored":671907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574926,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1425408,"data_stored":672441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574927,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1417216,"data_stored":671837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574925,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1433600,"data_stored":673519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1417216,"data_stored":671837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1425408,"data_stored":671961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37336,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334504,"statfs":{"total":96636764160,"available":96598532096,"internally_reserved":0,"allocated":2043904,"data_stored":1269924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:19:55.754 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-16T16:19:55.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-16T16:19:56.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.024+0000 7fc980c00640 1 Processor -- start 2024-04-16T16:19:56.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.024+0000 7fc980c00640 1 -- start start 2024-04-16T16:19:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.025+0000 7fc980c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc97c0b3370 0x7fc97c169c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.025+0000 7fc980c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc97c06d450 con 0x7fc97c0ceab0 2024-04-16T16:19:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.025+0000 7fc980c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc97c0d0980 con 0x7fc97c0ce6d0 2024-04-16T16:19:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.025+0000 7fc980c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc97c0cceb0 con 0x7fc97c0b3370 2024-04-16T16:19:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.026+0000 7fc97a800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc97c0b3370 0x7fc97c169c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.026+0000 7fc979400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc97c0ceab0 0x7fc97c09b2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33786/0 (socket says 172.21.15.45:33786) 2024-04-16T16:19:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.026+0000 7fc979400640 1 -- 172.21.15.45:0/512635910 learned_addr learned my addr 172.21.15.45:0/512635910 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.026+0000 7fc978a00640 1 -- 172.21.15.45:0/512635910 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2610766205 0 0) 0x7fc97c06d450 con 0x7fc97c0ceab0 2024-04-16T16:19:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.027+0000 7fc97a800640 1 -- 172.21.15.45:0/512635910 >> v1:172.21.15.103:6789/0 conn(0x7fc97c0ce6d0 legacy=0x7fc97c09abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.027+0000 7fc97a800640 1 -- 172.21.15.45:0/512635910 >> v1:172.21.15.45:6789/0 conn(0x7fc97c0ceab0 legacy=0x7fc97c09b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.027+0000 7fc97a800640 1 -- 172.21.15.45:0/512635910 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97c09bf10 con 0x7fc97c0b3370 2024-04-16T16:19:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.028+0000 7fc97a800640 1 --2- 172.21.15.45:0/512635910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc97c0b3370 0x7fc97c169c30 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fc9740097a0 tx=0x7fc97402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e049d3a01ca5c473 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.028+0000 7fc978a00640 1 -- 172.21.15.45:0/512635910 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974035020 con 0x7fc97c0b3370 2024-04-16T16:19:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.028+0000 7fc978a00640 1 -- 172.21.15.45:0/512635910 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc974031830 con 0x7fc97c0b3370 2024-04-16T16:19:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.028+0000 7fc978a00640 1 -- 172.21.15.45:0/512635910 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974031c00 con 0x7fc97c0b3370 2024-04-16T16:19:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.029+0000 7fc980c00640 1 -- 172.21.15.45:0/512635910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc97c0b3370 msgr2=0x7fc97c169c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.029+0000 7fc980c00640 1 --2- 172.21.15.45:0/512635910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc97c0b3370 0x7fc97c169c30 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fc9740097a0 tx=0x7fc97402b8d0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.029+0000 7fc980c00640 1 -- 172.21.15.45:0/512635910 shutdown_connections 2024-04-16T16:19:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.029+0000 7fc980c00640 1 --2- 172.21.15.45:0/512635910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc97c0b3370 0x7fc97c169c30 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.029+0000 7fc980c00640 1 -- 172.21.15.45:0/512635910 >> 172.21.15.45:0/512635910 conn(0x7fc97c0b4350 msgr2=0x7fc97c0b4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.029+0000 7fc980c00640 1 -- 172.21.15.45:0/512635910 shutdown_connections 2024-04-16T16:19:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.029+0000 7fc980c00640 1 -- 172.21.15.45:0/512635910 wait complete. 2024-04-16T16:19:56.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.031+0000 7fc980c00640 1 Processor -- start 2024-04-16T16:19:56.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.031+0000 7fc980c00640 1 -- start start 2024-04-16T16:19:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.032+0000 7fc980c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc97c0b3370 0x7fc97c0c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.032+0000 7fc980c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc97c0ce6d0 0x7fc97c160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.032+0000 7fc980c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc97c0ceab0 0x7fc97c0af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.032+0000 7fc980c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc97c06dd40 con 0x7fc97c0ceab0 2024-04-16T16:19:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.032+0000 7fc980c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc97c0b40b0 con 0x7fc97c0ce6d0 2024-04-16T16:19:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.032+0000 7fc980c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc97c06d6f0 con 0x7fc97c0b3370 2024-04-16T16:19:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.032+0000 7fc97a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc97c0ceab0 0x7fc97c0af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.032+0000 7fc97a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc97c0ceab0 0x7fc97c0af610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57580/0 (socket says 172.21.15.45:57580) 2024-04-16T16:19:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.032+0000 7fc97a800640 1 -- 172.21.15.45:0/357056625 learned_addr learned my addr 172.21.15.45:0/357056625 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.033+0000 7fc979400640 1 --2- 172.21.15.45:0/357056625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc97c0ce6d0 0x7fc97c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.033+0000 7fc979e00640 1 --2- 172.21.15.45:0/357056625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc97c0b3370 0x7fc97c0c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.033+0000 7fc97a800640 1 -- 172.21.15.45:0/357056625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc97c0b3370 msgr2=0x7fc97c0c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.033+0000 7fc97a800640 1 --2- 172.21.15.45:0/357056625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc97c0b3370 0x7fc97c0c8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.033+0000 7fc97a800640 1 -- 172.21.15.45:0/357056625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc97c0ce6d0 msgr2=0x7fc97c160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.033+0000 7fc97a800640 1 --2- 172.21.15.45:0/357056625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc97c0ce6d0 0x7fc97c160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.033+0000 7fc97a800640 1 -- 172.21.15.45:0/357056625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97c0c6650 con 0x7fc97c0ceab0 2024-04-16T16:19:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.033+0000 7fc97a800640 1 --2- 172.21.15.45:0/357056625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc97c0ceab0 0x7fc97c0af610 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fc974034040 tx=0x7fc9740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.034+0000 7fc972000640 1 -- 172.21.15.45:0/357056625 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974004230 con 0x7fc97c0ceab0 2024-04-16T16:19:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.034+0000 7fc972000640 1 -- 172.21.15.45:0/357056625 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9740043d0 con 0x7fc97c0ceab0 2024-04-16T16:19:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.034+0000 7fc980c00640 1 -- 172.21.15.45:0/357056625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc97c0c6930 con 0x7fc97c0ceab0 2024-04-16T16:19:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.034+0000 7fc972000640 1 -- 172.21.15.45:0/357056625 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974003850 con 0x7fc97c0ceab0 2024-04-16T16:19:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.034+0000 7fc980c00640 1 -- 172.21.15.45:0/357056625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc97c0c80d0 con 0x7fc97c0ceab0 2024-04-16T16:19:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.035+0000 7fc980c00640 1 -- 172.21.15.45:0/357056625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc97c075780 con 0x7fc97c0ceab0 2024-04-16T16:19:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.036+0000 7fc972000640 1 -- 172.21.15.45:0/357056625 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7fc974038020 con 0x7fc97c0ceab0 2024-04-16T16:19:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.037+0000 7fc972000640 1 --2- 172.21.15.45:0/357056625 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc95c06c980 0x7fc95c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.037+0000 7fc972000640 1 -- 172.21.15.45:0/357056625 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5752+0+0 (secure 0 0 0) 0x7fc9740ad910 con 0x7fc97c0ceab0 2024-04-16T16:19:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.038+0000 7fc979e00640 1 --2- 172.21.15.45:0/357056625 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc95c06c980 0x7fc95c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:56.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.039+0000 7fc979e00640 1 --2- 172.21.15.45:0/357056625 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc95c06c980 0x7fc95c06ee40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0719c0 tx=0x7fc96c009040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:56.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.042+0000 7fc972000640 1 -- 172.21.15.45:0/357056625 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc974040070 con 0x7fc97c0ceab0 2024-04-16T16:19:56.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.320+0000 7fc980c00640 1 -- 172.21.15.45:0/357056625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fc97c0ce260 con 0x7fc97c0ceab0 2024-04-16T16:19:56.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.325+0000 7fc972000640 1 -- 172.21.15.45:0/357056625 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v23) ==== 190+0+147 (secure 0 0 0) 0x7fc97407bb20 con 0x7fc97c0ceab0 2024-04-16T16:19:56.326 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-04-16T16:19:56.326 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ== 2024-04-16T16:19:56.326 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:19:56.326 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:19:56.326 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-16T16:19:56.326 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2024-04-16T16:19:56.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.328+0000 7fc980c00640 1 -- 172.21.15.45:0/357056625 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc95c06c980 msgr2=0x7fc95c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:56.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.328+0000 7fc980c00640 1 --2- 172.21.15.45:0/357056625 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc95c06c980 0x7fc95c06ee40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0719c0 tx=0x7fc96c009040 comp rx=0 tx=0).stop 2024-04-16T16:19:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.328+0000 7fc980c00640 1 -- 172.21.15.45:0/357056625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc97c0ceab0 msgr2=0x7fc97c0af610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.328+0000 7fc980c00640 1 --2- 172.21.15.45:0/357056625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc97c0ceab0 0x7fc97c0af610 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fc974034040 tx=0x7fc9740046c0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.329+0000 7fc980c00640 1 -- 172.21.15.45:0/357056625 shutdown_connections 2024-04-16T16:19:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.329+0000 7fc980c00640 1 --2- 172.21.15.45:0/357056625 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fc95c06c980 0x7fc95c06ee40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.329+0000 7fc980c00640 1 --2- 172.21.15.45:0/357056625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc97c0ceab0 0x7fc97c0af610 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.329+0000 7fc980c00640 1 --2- 172.21.15.45:0/357056625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc97c0ce6d0 0x7fc97c160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.329+0000 7fc980c00640 1 --2- 172.21.15.45:0/357056625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc97c0b3370 0x7fc97c0c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.329+0000 7fc980c00640 1 -- 172.21.15.45:0/357056625 >> 172.21.15.45:0/357056625 conn(0x7fc97c0b4350 msgr2=0x7fc97c0d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.329+0000 7fc980c00640 1 -- 172.21.15.45:0/357056625 shutdown_connections 2024-04-16T16:19:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.329+0000 7fc980c00640 1 -- 172.21.15.45:0/357056625 wait complete. 2024-04-16T16:19:56.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:19:56.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.568+0000 7febbc400640 1 Processor -- start 2024-04-16T16:19:56.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.568+0000 7febbc400640 1 -- start start 2024-04-16T16:19:56.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.569+0000 7febbc400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7febb41462e0 0x7febb4072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:56.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.570+0000 7febbc400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb415ea70 0x7febb40cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:56.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.570+0000 7febbc400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febb40cf6d0 con 0x7febb40728c0 2024-04-16T16:19:56.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.570+0000 7febbc400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7febb41466c0 con 0x7febb41462e0 2024-04-16T16:19:56.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.570+0000 7febbc400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7febb40cf3f0 con 0x7febb415ea70 2024-04-16T16:19:56.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.570+0000 7febb9a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7febb41462e0 0x7febb4072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:56.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.570+0000 7febb9a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7febb41462e0 0x7febb4072380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:43818/0 (socket says 172.21.15.45:43818) 2024-04-16T16:19:56.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.570+0000 7febb9a00640 1 -- 172.21.15.45:0/2414539975 learned_addr learned my addr 172.21.15.45:0/2414539975 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:56.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.570+0000 7febba400640 1 --2- 172.21.15.45:0/2414539975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb415ea70 0x7febb40cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:56.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.571+0000 7febb9a00640 1 -- 172.21.15.45:0/2414539975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb415ea70 msgr2=0x7febb40cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:56.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.571+0000 7febb9a00640 1 --2- 172.21.15.45:0/2414539975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb415ea70 0x7febb40cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.571+0000 7febb9a00640 1 -- 172.21.15.45:0/2414539975 >> v1:172.21.15.103:6789/0 conn(0x7febb40728c0 legacy=0x7febb40ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:56.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.571+0000 7febb9a00640 1 -- 172.21.15.45:0/2414539975 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb415cf80 con 0x7febb41462e0 2024-04-16T16:19:56.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.573+0000 7febb9a00640 1 --2- 172.21.15.45:0/2414539975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb41462e0 0x7febb4072380 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7feba8009870 tx=0x7feba802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4aa01b4f8042e6a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:56.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.574+0000 7febb3e00640 1 -- 172.21.15.45:0/2414539975 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba8035020 con 0x7febb41462e0 2024-04-16T16:19:56.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.574+0000 7febb3e00640 1 -- 172.21.15.45:0/2414539975 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feba8031640 con 0x7febb41462e0 2024-04-16T16:19:56.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.574+0000 7febb3e00640 1 -- 172.21.15.45:0/2414539975 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba802d040 con 0x7febb41462e0 2024-04-16T16:19:56.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.575+0000 7febbc400640 1 -- 172.21.15.45:0/2414539975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb41462e0 msgr2=0x7febb4072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:56.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.575+0000 7febbc400640 1 --2- 172.21.15.45:0/2414539975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb41462e0 0x7febb4072380 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7feba8009870 tx=0x7feba802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.575+0000 7febbc400640 1 -- 172.21.15.45:0/2414539975 shutdown_connections 2024-04-16T16:19:56.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.575+0000 7febbc400640 1 --2- 172.21.15.45:0/2414539975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb415ea70 0x7febb40cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.575+0000 7febbc400640 1 --2- 172.21.15.45:0/2414539975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb41462e0 0x7febb4072380 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.575+0000 7febbc400640 1 -- 172.21.15.45:0/2414539975 >> 172.21.15.45:0/2414539975 conn(0x7febb40b2750 msgr2=0x7febb40b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:56.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.575+0000 7febbc400640 1 -- 172.21.15.45:0/2414539975 shutdown_connections 2024-04-16T16:19:56.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.575+0000 7febbc400640 1 -- 172.21.15.45:0/2414539975 wait complete. 2024-04-16T16:19:56.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.577+0000 7febbc400640 1 Processor -- start 2024-04-16T16:19:56.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.577+0000 7febbc400640 1 -- start start 2024-04-16T16:19:56.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.578+0000 7febbc400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb40728c0 0x7febb41f5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:56.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.578+0000 7febbc400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb41462e0 0x7febb41f7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:56.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.578+0000 7febbc400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb415ea70 0x7febb413a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:56.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.578+0000 7febbc400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7febb4166490 con 0x7febb415ea70 2024-04-16T16:19:56.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.578+0000 7febbc400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7febb415d540 con 0x7febb41462e0 2024-04-16T16:19:56.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.578+0000 7febbc400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7febb4166610 con 0x7febb40728c0 2024-04-16T16:19:56.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.578+0000 7febba400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb415ea70 0x7febb413a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:56.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.578+0000 7febba400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb415ea70 0x7febb413a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57584/0 (socket says 172.21.15.45:57584) 2024-04-16T16:19:56.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.578+0000 7febba400640 1 -- 172.21.15.45:0/2639322999 learned_addr learned my addr 172.21.15.45:0/2639322999 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:56.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.578+0000 7febb9000640 1 --2- 172.21.15.45:0/2639322999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb41462e0 0x7febb41f7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:56.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.578+0000 7febb9a00640 1 --2- 172.21.15.45:0/2639322999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb40728c0 0x7febb41f5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:56.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.579+0000 7febba400640 1 -- 172.21.15.45:0/2639322999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb40728c0 msgr2=0x7febb41f5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:56.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.579+0000 7febba400640 1 --2- 172.21.15.45:0/2639322999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb40728c0 0x7febb41f5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.579+0000 7febba400640 1 -- 172.21.15.45:0/2639322999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb41462e0 msgr2=0x7febb41f7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:56.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.579+0000 7febba400640 1 --2- 172.21.15.45:0/2639322999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb41462e0 0x7febb41f7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.579+0000 7febba400640 1 -- 172.21.15.45:0/2639322999 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb413acd0 con 0x7febb415ea70 2024-04-16T16:19:56.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.579+0000 7febba400640 1 --2- 172.21.15.45:0/2639322999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb415ea70 0x7febb413a180 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7febac00efc0 tx=0x7febac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:56.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.579+0000 7febb1600640 1 -- 172.21.15.45:0/2639322999 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac018020 con 0x7febb415ea70 2024-04-16T16:19:56.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.579+0000 7febb1600640 1 -- 172.21.15.45:0/2639322999 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febac014480 con 0x7febb415ea70 2024-04-16T16:19:56.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.579+0000 7febb1600640 1 -- 172.21.15.45:0/2639322999 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac014870 con 0x7febb415ea70 2024-04-16T16:19:56.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.579+0000 7febbc400640 1 -- 172.21.15.45:0/2639322999 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febb413afb0 con 0x7febb415ea70 2024-04-16T16:19:56.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.579+0000 7febbc400640 1 -- 172.21.15.45:0/2639322999 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febb40b3970 con 0x7febb415ea70 2024-04-16T16:19:56.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.581+0000 7febbc400640 1 -- 172.21.15.45:0/2639322999 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febb4075780 con 0x7febb415ea70 2024-04-16T16:19:56.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.582+0000 7febb1600640 1 -- 172.21.15.45:0/2639322999 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7febac005d10 con 0x7febb415ea70 2024-04-16T16:19:56.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.583+0000 7febb1600640 1 --2- 172.21.15.45:0/2639322999 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7feb9406c980 0x7feb9406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:56.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.583+0000 7febb1600640 1 -- 172.21.15.45:0/2639322999 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5752+0+0 (secure 0 0 0) 0x7febac08c860 con 0x7febb415ea70 2024-04-16T16:19:56.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.584+0000 7febb9a00640 1 --2- 172.21.15.45:0/2639322999 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7feb9406c980 0x7feb9406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:56.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.585+0000 7febb9a00640 1 --2- 172.21.15.45:0/2639322999 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7feb9406c980 0x7feb9406ee40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feba8036000 tx=0x7feba8008030 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:56.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.587+0000 7febb1600640 1 -- 172.21.15.45:0/2639322999 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7febac022070 con 0x7febb415ea70 2024-04-16T16:19:56.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.833+0000 7febbc400640 1 -- 172.21.15.45:0/2639322999 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7febb40aec60 con 0x7febb415ea70 2024-04-16T16:19:56.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.835+0000 7febb1600640 1 -- 172.21.15.45:0/2639322999 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5186 (secure 0 0 0) 0x7febac010040 con 0x7febb415ea70 2024-04-16T16:19:56.891 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:19:56.891 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6056,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1032791151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":1032791151},{"type":"v1","addr":"172.21.15.45:6835","nonce":1032791151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6062,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/498835619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":498835619},{"type":"v1","addr":"172.21.15.103:6837","nonce":498835619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/3611628422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3611628422},{"type":"v1","addr":"172.21.15.103:6836","nonce":3611628422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:19:50.154928+0000","modified":"2024-04-16T16:19:53.221144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8160},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8160":{"gid":8160,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/3429980864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3429980864},{"type":"v1","addr":"172.21.15.45:6834","nonce":3429980864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8160,"qdb_cluster":[8160]},"id":8}]} 2024-04-16T16:19:56.892 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 82 2024-04-16T16:19:56.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.837+0000 7febbc400640 1 -- 172.21.15.45:0/2639322999 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7feb9406c980 msgr2=0x7feb9406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:56.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.837+0000 7febbc400640 1 --2- 172.21.15.45:0/2639322999 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7feb9406c980 0x7feb9406ee40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feba8036000 tx=0x7feba8008030 comp rx=0 tx=0).stop 2024-04-16T16:19:56.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.837+0000 7febbc400640 1 -- 172.21.15.45:0/2639322999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb415ea70 msgr2=0x7febb413a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.837+0000 7febbc400640 1 --2- 172.21.15.45:0/2639322999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb415ea70 0x7febb413a180 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7febac00efc0 tx=0x7febac00c490 comp rx=0 tx=0).stop 2024-04-16T16:19:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.837+0000 7febbc400640 1 -- 172.21.15.45:0/2639322999 shutdown_connections 2024-04-16T16:19:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.837+0000 7febbc400640 1 --2- 172.21.15.45:0/2639322999 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7feb9406c980 0x7feb9406ee40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.838+0000 7febbc400640 1 --2- 172.21.15.45:0/2639322999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb415ea70 0x7febb413a180 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.838+0000 7febbc400640 1 --2- 172.21.15.45:0/2639322999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb41462e0 0x7febb41f7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.838+0000 7febbc400640 1 --2- 172.21.15.45:0/2639322999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febb40728c0 0x7febb41f5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.838+0000 7febbc400640 1 -- 172.21.15.45:0/2639322999 >> 172.21.15.45:0/2639322999 conn(0x7febb40b2750 msgr2=0x7febb40ae080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.838+0000 7febbc400640 1 -- 172.21.15.45:0/2639322999 shutdown_connections 2024-04-16T16:19:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:56.838+0000 7febbc400640 1 -- 172.21.15.45:0/2639322999 wait complete. 2024-04-16T16:19:57.297 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:19:50.154928+0000', 'modified': '2024-04-16T16:19:53.221144+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8160}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8160': {'gid': 8160, 'name': 'a', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6834/3429980864', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6832', 'nonce': 3429980864}, {'type': 'v1', 'addr': '172.21.15.45:6834', 'nonce': 3429980864}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8160, 'qdb_cluster': [8160]} 2024-04-16T16:19:57.297 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-16T16:19:57.297 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-16T16:19:57.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.533+0000 7fecd0c00640 1 Processor -- start 2024-04-16T16:19:57.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.534+0000 7fecd0c00640 1 -- start start 2024-04-16T16:19:57.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.535+0000 7fecd0c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7feccc146220 0x7feccc0cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.535+0000 7fecd0c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feccc072180 0x7feccc0725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.535+0000 7fecd0c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feccc146600 con 0x7feccc0cd220 2024-04-16T16:19:57.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.535+0000 7fecd0c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7feccc06f840 con 0x7feccc146220 2024-04-16T16:19:57.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.535+0000 7fecd0c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feccc0ce0e0 con 0x7feccc072180 2024-04-16T16:19:57.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.536+0000 7fecca800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7feccc0cd220 0x7feccc06df00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33792/0 (socket says 172.21.15.45:33792) 2024-04-16T16:19:57.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.536+0000 7fecca800640 1 -- 172.21.15.45:0/454901182 learned_addr learned my addr 172.21.15.45:0/454901182 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:57.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.536+0000 7fecc9e00640 1 --2- 172.21.15.45:0/454901182 >> v2:172.21.15.103:3300/0 conn(0x7feccc146220 0x7feccc0cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.536+0000 7fecc3e00640 1 --2- 172.21.15.45:0/454901182 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feccc072180 0x7feccc0725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.536+0000 7fecc9400640 1 -- 172.21.15.45:0/454901182 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 302779504 0 0) 0x7feccc146600 con 0x7feccc0cd220 2024-04-16T16:19:57.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc9400640 1 -- 172.21.15.45:0/454901182 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecb0003520 con 0x7feccc0cd220 2024-04-16T16:19:57.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc9400640 1 -- 172.21.15.45:0/454901182 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4119779174 0 0) 0x7fecb0003520 con 0x7feccc0cd220 2024-04-16T16:19:57.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc9400640 1 -- 172.21.15.45:0/454901182 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feccc072180 msgr2=0x7feccc0725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc9400640 1 --2- 172.21.15.45:0/454901182 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feccc072180 0x7feccc0725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc9400640 1 -- 172.21.15.45:0/454901182 >> v2:172.21.15.103:3300/0 conn(0x7feccc146220 msgr2=0x7feccc0cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc9400640 1 --2- 172.21.15.45:0/454901182 >> v2:172.21.15.103:3300/0 conn(0x7feccc146220 0x7feccc0cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc9400640 1 -- 172.21.15.45:0/454901182 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feccc0cde00 con 0x7feccc0cd220 2024-04-16T16:19:57.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc9400640 1 -- 172.21.15.45:0/454901182 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fecc40019f0 con 0x7feccc0cd220 2024-04-16T16:19:57.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc9400640 1 -- 172.21.15.45:0/454901182 >> v1:172.21.15.45:6789/0 conn(0x7feccc0cd220 legacy=0x7feccc06df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:57.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc9400640 1 --2- 172.21.15.45:0/454901182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecb00039b0 0x7fecb0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc9400640 1 --2- 172.21.15.45:0/454901182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fecb0004800 0x7fecb0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc9400640 1 --2- 172.21.15.45:0/454901182 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fecb00051c0 0x7fecb000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc9400640 1 -- 172.21.15.45:0/454901182 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feccc0ce0e0 con 0x7fecb0004800 2024-04-16T16:19:57.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc9400640 1 -- 172.21.15.45:0/454901182 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fecb000bf00 con 0x7fecb00039b0 2024-04-16T16:19:57.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc9400640 1 -- 172.21.15.45:0/454901182 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fecb000c0b0 con 0x7fecb00051c0 2024-04-16T16:19:57.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.537+0000 7fecc3e00640 1 --2- 172.21.15.45:0/454901182 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feccc072180 0x7feccc0725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:19:57.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.538+0000 7fecc3e00640 1 --2- 172.21.15.45:0/454901182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fecb0004800 0x7fecb0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.538+0000 7fecc9e00640 1 --2- 172.21.15.45:0/454901182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecb00039b0 0x7fecb0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.538+0000 7fecc3e00640 1 -- 172.21.15.45:0/454901182 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fecb00051c0 msgr2=0x7fecb000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.538+0000 7fecc3e00640 1 --2- 172.21.15.45:0/454901182 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fecb00051c0 0x7fecb000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.538+0000 7fecc3e00640 1 -- 172.21.15.45:0/454901182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecb00039b0 msgr2=0x7fecb0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.538+0000 7fecc3e00640 1 --2- 172.21.15.45:0/454901182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecb00039b0 0x7fecb0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.538+0000 7fecc3e00640 1 -- 172.21.15.45:0/454901182 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb0003520 con 0x7fecb0004800 2024-04-16T16:19:57.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.539+0000 7fecc3e00640 1 --2- 172.21.15.45:0/454901182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fecb0004800 0x7fecb0004c80 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fecb4006fd0 tx=0x7fecb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf748537f7df090 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:57.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.540+0000 7fecc9400640 1 -- 172.21.15.45:0/454901182 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4039070 con 0x7fecb0004800 2024-04-16T16:19:57.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.540+0000 7fecc9400640 1 -- 172.21.15.45:0/454901182 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecb4034680 con 0x7fecb0004800 2024-04-16T16:19:57.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.540+0000 7fecc9400640 1 -- 172.21.15.45:0/454901182 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb400f040 con 0x7fecb0004800 2024-04-16T16:19:57.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.540+0000 7fecd0c00640 1 -- 172.21.15.45:0/454901182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fecb0004800 msgr2=0x7fecb0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:57.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.540+0000 7fecd0c00640 1 --2- 172.21.15.45:0/454901182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fecb0004800 0x7fecb0004c80 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fecb4006fd0 tx=0x7fecb0005830 comp rx=0 tx=0).stop 2024-04-16T16:19:57.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.540+0000 7fecca800640 1 -- 172.21.15.45:0/454901182 reap_dead start 2024-04-16T16:19:57.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.540+0000 7fecd0c00640 1 -- 172.21.15.45:0/454901182 shutdown_connections 2024-04-16T16:19:57.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.540+0000 7fecd0c00640 1 -- 172.21.15.45:0/454901182 >> 172.21.15.45:0/454901182 conn(0x7feccc06b930 msgr2=0x7feccc06bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:57.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.540+0000 7fecd0c00640 1 -- 172.21.15.45:0/454901182 shutdown_connections 2024-04-16T16:19:57.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.540+0000 7fecd0c00640 1 -- 172.21.15.45:0/454901182 wait complete. 2024-04-16T16:19:57.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.542+0000 7fecd0c00640 1 Processor -- start 2024-04-16T16:19:57.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.542+0000 7fecd0c00640 1 -- start start 2024-04-16T16:19:57.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.543+0000 7fecd0c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fecb0004800 0x7feccc0cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.543+0000 7fecd0c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feccc0b0c90 0x7feccc0b1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.543+0000 7fecd0c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feccc0aa730 0x7feccc0aabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.543+0000 7fecd0c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feccc0cebf0 con 0x7feccc0b0c90 2024-04-16T16:19:57.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.543+0000 7fecd0c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feccc0ce910 con 0x7feccc0aa730 2024-04-16T16:19:57.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.543+0000 7fecd0c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feccc0725c0 con 0x7fecb0004800 2024-04-16T16:19:57.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.544+0000 7fecc3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feccc0b0c90 0x7feccc0b1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.544+0000 7fecc3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feccc0b0c90 0x7feccc0b1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57602/0 (socket says 172.21.15.45:57602) 2024-04-16T16:19:57.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.544+0000 7fecc3e00640 1 -- 172.21.15.45:0/1413859260 learned_addr learned my addr 172.21.15.45:0/1413859260 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:57.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.544+0000 7fecc9e00640 1 --2- 172.21.15.45:0/1413859260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fecb0004800 0x7feccc0cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.544+0000 7fecc3e00640 1 -- 172.21.15.45:0/1413859260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fecb0004800 msgr2=0x7feccc0cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:57.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.544+0000 7fecca800640 1 --2- 172.21.15.45:0/1413859260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feccc0aa730 0x7feccc0aabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.544+0000 7fecc3e00640 1 --2- 172.21.15.45:0/1413859260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fecb0004800 0x7feccc0cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.544+0000 7fecc3e00640 1 -- 172.21.15.45:0/1413859260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feccc0aa730 msgr2=0x7feccc0aabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:57.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.544+0000 7fecc3e00640 1 --2- 172.21.15.45:0/1413859260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feccc0aa730 0x7feccc0aabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.544+0000 7fecc3e00640 1 -- 172.21.15.45:0/1413859260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feccc0ab740 con 0x7feccc0b0c90 2024-04-16T16:19:57.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.544+0000 7fecca800640 1 --2- 172.21.15.45:0/1413859260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feccc0aa730 0x7feccc0aabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:57.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.545+0000 7fecc3e00640 1 --2- 172.21.15.45:0/1413859260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feccc0b0c90 0x7feccc0b1110 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7feccc0b0350 tx=0x7fecb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:57.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.545+0000 7fecc2000640 1 -- 172.21.15.45:0/1413859260 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4043070 con 0x7feccc0b0c90 2024-04-16T16:19:57.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.545+0000 7fecc2000640 1 -- 172.21.15.45:0/1413859260 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecb403e660 con 0x7feccc0b0c90 2024-04-16T16:19:57.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.545+0000 7fecc2000640 1 -- 172.21.15.45:0/1413859260 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4039040 con 0x7feccc0b0c90 2024-04-16T16:19:57.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.545+0000 7fecd0c00640 1 -- 172.21.15.45:0/1413859260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feccc1f4550 con 0x7feccc0b0c90 2024-04-16T16:19:57.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.545+0000 7fecd0c00640 1 -- 172.21.15.45:0/1413859260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feccc132fd0 con 0x7feccc0b0c90 2024-04-16T16:19:57.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.549+0000 7fecc2000640 1 -- 172.21.15.45:0/1413859260 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 40) ==== 91086+0+0 (secure 0 0 0) 0x7fecb4046070 con 0x7feccc0b0c90 2024-04-16T16:19:57.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.549+0000 7fecc2000640 1 --2- 172.21.15.45:0/1413859260 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7feca806c980 0x7feca806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.550+0000 7fecc2000640 1 -- 172.21.15.45:0/1413859260 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5752+0+0 (secure 0 0 0) 0x7fecb40ad760 con 0x7feccc0b0c90 2024-04-16T16:19:57.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.550+0000 7fecd3974740 1 Processor -- start 2024-04-16T16:19:57.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.550+0000 7fecd3974740 1 -- start start 2024-04-16T16:19:57.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.550+0000 7fecc9e00640 1 --2- 172.21.15.45:0/1413859260 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7feca806c980 0x7feca806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.551+0000 7fecd3974740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfc650 0x55c478e1b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.551+0000 7fecd3974740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfca30 0x55c478dd5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.551+0000 7fecd3974740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478dd5870 0x55c478dd3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.551+0000 7fecd3974740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c478d3a9c0 con 0x55c478dfca30 2024-04-16T16:19:57.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.551+0000 7fecd3974740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c478ce1560 con 0x55c478dfc650 2024-04-16T16:19:57.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.551+0000 7fecd3974740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c478d36980 con 0x55c478dd5870 2024-04-16T16:19:57.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.552+0000 7fecc9e00640 1 --2- 172.21.15.45:0/1413859260 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7feca806c980 0x7feca806ee40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0099d0 tx=0x7fecbc008040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:57.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.552+0000 7fecc3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfca30 0x55c478dd5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.552+0000 7fecc3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfca30 0x55c478dd5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57614/0 (socket says 172.21.15.45:57614) 2024-04-16T16:19:57.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.552+0000 7fecc3e00640 1 -- 172.21.15.45:0/1446520589 learned_addr learned my addr 172.21.15.45:0/1446520589 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:57.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.552+0000 7fecc9e00640 1 --2- 172.21.15.45:0/1446520589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfc650 0x55c478e1b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.552+0000 7fecc3e00640 1 -- 172.21.15.45:0/1446520589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478dd5870 msgr2=0x55c478dd3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.552+0000 7fecc3e00640 1 --2- 172.21.15.45:0/1446520589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478dd5870 0x55c478dd3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.552+0000 7fecc3e00640 1 -- 172.21.15.45:0/1446520589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfc650 msgr2=0x55c478e1b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.552+0000 7fecc3e00640 1 --2- 172.21.15.45:0/1446520589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfc650 0x55c478e1b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.552+0000 7fecc3e00640 1 -- 172.21.15.45:0/1446520589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c478a4d2d0 con 0x55c478dfca30 2024-04-16T16:19:57.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.552+0000 7fecc9e00640 1 --2- 172.21.15.45:0/1446520589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfc650 0x55c478e1b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:57.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.552+0000 7fecc3e00640 1 --2- 172.21.15.45:0/1446520589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfca30 0x55c478dd5330 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fecb4037040 tx=0x7fecb407ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fdc3a718791adb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:57.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.553+0000 7fecba600640 1 -- 172.21.15.45:0/1446520589 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb404d070 con 0x55c478dfca30 2024-04-16T16:19:57.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.553+0000 7fecba600640 1 -- 172.21.15.45:0/1446520589 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecb40abcc0 con 0x55c478dfca30 2024-04-16T16:19:57.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.553+0000 7fecba600640 1 -- 172.21.15.45:0/1446520589 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4077980 con 0x55c478dfca30 2024-04-16T16:19:57.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.553+0000 7fecd3974740 1 -- 172.21.15.45:0/1446520589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfca30 msgr2=0x55c478dd5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:57.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.553+0000 7fecd3974740 1 --2- 172.21.15.45:0/1446520589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfca30 0x55c478dd5330 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fecb4037040 tx=0x7fecb407ada0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.553+0000 7fecd3974740 1 -- 172.21.15.45:0/1446520589 shutdown_connections 2024-04-16T16:19:57.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.553+0000 7fecd3974740 1 --2- 172.21.15.45:0/1446520589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478dd5870 0x55c478dd3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.553+0000 7fecd3974740 1 --2- 172.21.15.45:0/1446520589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfca30 0x55c478dd5330 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.553+0000 7fecd3974740 1 --2- 172.21.15.45:0/1446520589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfc650 0x55c478e1b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.553+0000 7fecd3974740 1 -- 172.21.15.45:0/1446520589 >> 172.21.15.45:0/1446520589 conn(0x55c478dd1910 msgr2=0x55c478e1a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:57.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.553+0000 7fecd3974740 1 -- 172.21.15.45:0/1446520589 shutdown_connections 2024-04-16T16:19:57.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.553+0000 7fecd3974740 1 -- 172.21.15.45:0/1446520589 wait complete. 2024-04-16T16:19:57.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.554+0000 7fecd3974740 1 -- start start 2024-04-16T16:19:57.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.554+0000 7fecd3974740 1 Processor -- start 2024-04-16T16:19:57.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.555+0000 7fecd3974740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfc650 0x55c478e16fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.555+0000 7fecd3974740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfca30 0x55c478e174f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.555+0000 7fecd3974740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478df6100 0x55c478e1cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.555+0000 7fecd3974740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c478ce1560 con 0x55c478dfc650 2024-04-16T16:19:57.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.555+0000 7fecd3974740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c478cf4920 con 0x55c478dfca30 2024-04-16T16:19:57.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.555+0000 7fecd3974740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c478d3a9c0 con 0x55c478df6100 2024-04-16T16:19:57.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.555+0000 7fecd3974740 10 client.0 _ll_register_callbacks cb 0x55c478cf4310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:19:57.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.556+0000 7fecc9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfc650 0x55c478e16fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.556+0000 7fecc9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfc650 0x55c478e16fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57616/0 (socket says 172.21.15.45:57616) 2024-04-16T16:19:57.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.556+0000 7fecc9e00640 1 -- 172.21.15.45:0/862076595 learned_addr learned my addr 172.21.15.45:0/862076595 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:57.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.556+0000 7fecc3e00640 1 --2- 172.21.15.45:0/862076595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfca30 0x55c478e174f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.556+0000 7fecca800640 1 --2- 172.21.15.45:0/862076595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478df6100 0x55c478e1cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.556+0000 7fecc9e00640 1 -- 172.21.15.45:0/862076595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478df6100 msgr2=0x55c478e1cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.556+0000 7fecc9e00640 1 --2- 172.21.15.45:0/862076595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478df6100 0x55c478e1cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.556+0000 7fecc9e00640 1 -- 172.21.15.45:0/862076595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfca30 msgr2=0x55c478e174f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.556+0000 7fecc9e00640 1 --2- 172.21.15.45:0/862076595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfca30 0x55c478e174f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.556+0000 7fecc9e00640 1 -- 172.21.15.45:0/862076595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c478b176b0 con 0x55c478dfc650 2024-04-16T16:19:57.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.556+0000 7fecc3e00640 1 --2- 172.21.15.45:0/862076595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfca30 0x55c478e174f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:57.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.556+0000 7fecca800640 1 --2- 172.21.15.45:0/862076595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478df6100 0x55c478e1cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:57.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.557+0000 7fecc9e00640 1 --2- 172.21.15.45:0/862076595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfc650 0x55c478e16fb0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fecbc015a00 tx=0x7fecbc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675d1852845cf19e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:57.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.557+0000 7fecaea00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:19:57.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.557+0000 7fecaea00640 1 -- 172.21.15.45:0/862076595 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc026070 con 0x55c478dfc650 2024-04-16T16:19:57.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.557+0000 7fecd3974740 1 -- 172.21.15.45:0/862076595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecb40abcc0 con 0x55c478dfc650 2024-04-16T16:19:57.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.557+0000 7fecaea00640 1 -- 172.21.15.45:0/862076595 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 5752+0+0 (secure 0 0 0) 0x7fecbc014d40 con 0x55c478dfc650 2024-04-16T16:19:57.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.557+0000 7fecaea00640 1 -- 172.21.15.45:0/862076595 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fecbc030050 con 0x55c478dfc650 2024-04-16T16:19:57.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.557+0000 7fecd3974740 10 client.8313 fetch_fsmap learned FSMap version 82 2024-04-16T16:19:57.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.557+0000 7fecd3974740 1 -- 172.21.15.45:0/862076595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) -- 0x55c478e2a170 con 0x55c478dfc650 2024-04-16T16:19:57.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.557+0000 7fecaea00640 1 -- 172.21.15.45:0/862076595 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecb40abcc0 con 0x55c478dfc650 2024-04-16T16:19:57.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.557+0000 7fecaea00640 1 -- 172.21.15.45:0/862076595 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 82) ==== 3494+0+0 (secure 0 0 0) 0x7fecbc033020 con 0x55c478dfc650 2024-04-16T16:19:57.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.558+0000 7fecd3974740 10 client.8313 fetch_fsmap finished waiting for FSMap version 82 2024-04-16T16:19:57.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.558+0000 7fecd3974740 10 client.8313 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-16T16:19:57.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.558+0000 7fecd3974740 20 client.8313 populate_metadata read hostname 'smithi045' 2024-04-16T16:19:57.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.558+0000 7fecd3974740 1 --2- 172.21.15.45:0/862076595 >> [v2:172.21.15.45:6832/3429980864,v1:172.21.15.45:6834/3429980864] conn(0x55c478e1d480 0x55c478e15260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.558+0000 7fecd3974740 4 client.8313 mds_command: new command op to 8160 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-16T16:19:57.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.558+0000 7fecd3974740 1 -- 172.21.15.45:0/862076595 --> [v2:172.21.15.45:6832/3429980864,v1:172.21.15.45:6834/3429980864] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c478ce0b70 con 0x55c478e1d480 2024-04-16T16:19:57.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.558+0000 7fecc3e00640 1 --2- 172.21.15.45:0/862076595 >> [v2:172.21.15.45:6832/3429980864,v1:172.21.15.45:6834/3429980864] conn(0x55c478e1d480 0x55c478e15260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.558+0000 7fecc3e00640 1 --2- 172.21.15.45:0/862076595 >> [v2:172.21.15.45:6832/3429980864,v1:172.21.15.45:6834/3429980864] conn(0x55c478e1d480 0x55c478e15260 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f079d9c7ddb5847 server_cookie=59adde1e64ed32e7 in_seq=0 out_seq=0 2024-04-16T16:19:57.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.558+0000 7fecaea00640 10 client.8313 ms_handle_connect on v2:172.21.15.45:6832/3429980864 2024-04-16T16:19:57.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.559+0000 7fecaea00640 1 -- 172.21.15.45:0/862076595 <== mds.0 v2:172.21.15.45:6832/3429980864 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c478ce0b70 con 0x55c478e1d480 2024-04-16T16:19:57.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.559+0000 7fecaea00640 10 client.8313 handle_command_reply: tid=0 2024-04-16T16:19:57.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.559+0000 7fecd3974740 1 client.8313 shutdown 2024-04-16T16:19:57.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.560+0000 7fecd3974740 1 -- 172.21.15.45:0/862076595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfc650 msgr2=0x55c478e16fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:57.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.560+0000 7fecd3974740 1 --2- 172.21.15.45:0/862076595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfc650 0x55c478e16fb0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fecbc015a00 tx=0x7fecbc004730 comp rx=0 tx=0).stop 2024-04-16T16:19:57.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.560+0000 7fecd3974740 1 -- 172.21.15.45:0/862076595 shutdown_connections 2024-04-16T16:19:57.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.560+0000 7fecd3974740 1 --2- 172.21.15.45:0/862076595 >> [v2:172.21.15.45:6832/3429980864,v1:172.21.15.45:6834/3429980864] conn(0x55c478e1d480 0x55c478e15260 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.560+0000 7fecd3974740 1 --2- 172.21.15.45:0/862076595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478df6100 0x55c478e1cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.560+0000 7fecd3974740 1 --2- 172.21.15.45:0/862076595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfca30 0x55c478e174f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.560+0000 7fecd3974740 1 --2- 172.21.15.45:0/862076595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfc650 0x55c478e16fb0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.560+0000 7fecaea00640 0 client.8313 ms_handle_reset on v2:172.21.15.45:6832/3429980864 2024-04-16T16:19:57.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.560+0000 7fecd3974740 1 -- 172.21.15.45:0/862076595 >> 172.21.15.45:0/862076595 conn(0x55c478dd5870 msgr2=0x55c478dcf7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:57.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.560+0000 7fecd3974740 1 -- 172.21.15.45:0/862076595 shutdown_connections 2024-04-16T16:19:57.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.560+0000 7fecd3974740 1 -- 172.21.15.45:0/862076595 wait complete. 2024-04-16T16:19:57.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.560+0000 7fecd3974740 20 client.8313 trim_cache size 0 max 16384 2024-04-16T16:19:57.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.580+0000 7fecd3974740 1 Processor -- start 2024-04-16T16:19:57.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.580+0000 7fecd3974740 1 -- start start 2024-04-16T16:19:57.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.581+0000 7fecd3974740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478df6100 0x55c478e3dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.581+0000 7fecd3974740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfc650 0x55c478e1cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.581+0000 7fecd3974740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478dfca30 0x55c478e19c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.581+0000 7fecd3974740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feccc2008c0 con 0x55c478df6100 2024-04-16T16:19:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.581+0000 7fecd3974740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feccc134c70 con 0x55c478dfc650 2024-04-16T16:19:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.581+0000 7fecd3974740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feccc2002a0 con 0x55c478dfca30 2024-04-16T16:19:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.581+0000 7fecc9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478df6100 0x55c478e3dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.581+0000 7fecc9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478df6100 0x55c478e3dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57618/0 (socket says 172.21.15.45:57618) 2024-04-16T16:19:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.581+0000 7fecc9e00640 1 -- 172.21.15.45:0/3791852575 learned_addr learned my addr 172.21.15.45:0/3791852575 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.581+0000 7fecc9e00640 1 -- 172.21.15.45:0/3791852575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478dfca30 msgr2=0x55c478e19c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.581+0000 7fecc9e00640 1 --2- 172.21.15.45:0/3791852575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478dfca30 0x55c478e19c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.581+0000 7fecc9e00640 1 -- 172.21.15.45:0/3791852575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfc650 msgr2=0x55c478e1cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.581+0000 7fecc9e00640 1 --2- 172.21.15.45:0/3791852575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfc650 0x55c478e1cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.581+0000 7fecc9e00640 1 -- 172.21.15.45:0/3791852575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c478dd13c0 con 0x55c478df6100 2024-04-16T16:19:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.582+0000 7fecc9e00640 1 --2- 172.21.15.45:0/3791852575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478df6100 0x55c478e3dab0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fecbc012eb0 tx=0x7fecbc022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6da24b71bf43ca server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.582+0000 7fecb9c00640 1 -- 172.21.15.45:0/3791852575 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc026050 con 0x55c478df6100 2024-04-16T16:19:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.582+0000 7fecb9c00640 1 -- 172.21.15.45:0/3791852575 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecbc01b040 con 0x55c478df6100 2024-04-16T16:19:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.582+0000 7fecb9c00640 1 -- 172.21.15.45:0/3791852575 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc014d40 con 0x55c478df6100 2024-04-16T16:19:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.583+0000 7fecd3974740 1 -- 172.21.15.45:0/3791852575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478df6100 msgr2=0x55c478e3dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.583+0000 7fecd3974740 1 --2- 172.21.15.45:0/3791852575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478df6100 0x55c478e3dab0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fecbc012eb0 tx=0x7fecbc022000 comp rx=0 tx=0).stop 2024-04-16T16:19:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.583+0000 7fecd3974740 1 -- 172.21.15.45:0/3791852575 shutdown_connections 2024-04-16T16:19:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.583+0000 7fecd3974740 1 --2- 172.21.15.45:0/3791852575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478dfca30 0x55c478e19c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.583+0000 7fecd3974740 1 --2- 172.21.15.45:0/3791852575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478dfc650 0x55c478e1cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.583+0000 7fecd3974740 1 --2- 172.21.15.45:0/3791852575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478df6100 0x55c478e3dab0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.583+0000 7fecd3974740 1 -- 172.21.15.45:0/3791852575 >> 172.21.15.45:0/3791852575 conn(0x55c478dd5870 msgr2=0x55c478dd64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.583+0000 7fecd3974740 1 -- 172.21.15.45:0/3791852575 shutdown_connections 2024-04-16T16:19:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.583+0000 7fecd3974740 1 -- 172.21.15.45:0/3791852575 wait complete. 2024-04-16T16:19:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.583+0000 7fecd3974740 1 -- start start 2024-04-16T16:19:57.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.584+0000 7fecd3974740 1 Processor -- start 2024-04-16T16:19:57.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecd3974740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478df6100 0x55c478e294d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecd3974740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478dfc650 0x55c478df2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecd3974740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfca30 0x55c478df3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecd3974740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c478df68c0 con 0x55c478dfca30 2024-04-16T16:19:57.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecd3974740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c478e29ef0 con 0x55c478df6100 2024-04-16T16:19:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecd3974740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c478e1d8a0 con 0x55c478dfc650 2024-04-16T16:19:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecd3974740 10 client.0 _ll_register_callbacks cb 0x7fecbc015680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:19:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecca800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfca30 0x55c478df3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecca800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfca30 0x55c478df3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57620/0 (socket says 172.21.15.45:57620) 2024-04-16T16:19:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecca800640 1 -- 172.21.15.45:0/3444775989 learned_addr learned my addr 172.21.15.45:0/3444775989 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecc9e00640 1 --2- 172.21.15.45:0/3444775989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478df6100 0x55c478e294d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecca800640 1 -- 172.21.15.45:0/3444775989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478dfc650 msgr2=0x55c478df2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecca800640 1 --2- 172.21.15.45:0/3444775989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478dfc650 0x55c478df2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecca800640 1 -- 172.21.15.45:0/3444775989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478df6100 msgr2=0x55c478e294d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecca800640 1 --2- 172.21.15.45:0/3444775989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478df6100 0x55c478e294d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.585+0000 7fecca800640 1 -- 172.21.15.45:0/3444775989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c478e17630 con 0x55c478dfca30 2024-04-16T16:19:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.586+0000 7fecca800640 1 --2- 172.21.15.45:0/3444775989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfca30 0x55c478df3360 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7feccc06fb60 tx=0x7fecc400e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2ca677dc17e851 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.586+0000 7fecaea00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:19:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.586+0000 7fecaea00640 1 -- 172.21.15.45:0/3444775989 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4018060 con 0x55c478dfca30 2024-04-16T16:19:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.586+0000 7fecd3974740 1 -- 172.21.15.45:0/3444775989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecbc01b040 con 0x55c478dfca30 2024-04-16T16:19:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.586+0000 7fecaea00640 1 -- 172.21.15.45:0/3444775989 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 5752+0+0 (secure 0 0 0) 0x7fecc4013ac0 con 0x55c478dfca30 2024-04-16T16:19:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.586+0000 7fecaea00640 1 -- 172.21.15.45:0/3444775989 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fecc4013cc0 con 0x55c478dfca30 2024-04-16T16:19:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.586+0000 7fecd3974740 10 client.8319 fetch_fsmap learned FSMap version 82 2024-04-16T16:19:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.586+0000 7fecd3974740 1 -- 172.21.15.45:0/3444775989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) -- 0x55c478e171b0 con 0x55c478dfca30 2024-04-16T16:19:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.586+0000 7fecaea00640 1 -- 172.21.15.45:0/3444775989 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecbc01b040 con 0x55c478dfca30 2024-04-16T16:19:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.587+0000 7fecaea00640 1 -- 172.21.15.45:0/3444775989 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 82) ==== 3494+0+0 (secure 0 0 0) 0x7fecc4013430 con 0x55c478dfca30 2024-04-16T16:19:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.587+0000 7fecd3974740 10 client.8319 fetch_fsmap finished waiting for FSMap version 82 2024-04-16T16:19:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.587+0000 7fecd3974740 10 client.8319 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-16T16:19:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.587+0000 7fecd3974740 20 client.8319 populate_metadata read hostname 'smithi045' 2024-04-16T16:19:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.587+0000 7fecd3974740 1 --2- 172.21.15.45:0/3444775989 >> [v2:172.21.15.45:6832/3429980864,v1:172.21.15.45:6834/3429980864] conn(0x55c478e32a30 0x55c478e32e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.587+0000 7fecd3974740 4 client.8319 mds_command: new command op to 8160 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-16T16:19:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.587+0000 7fecd3974740 1 -- 172.21.15.45:0/3444775989 --> [v2:172.21.15.45:6832/3429980864,v1:172.21.15.45:6834/3429980864] -- command(tid 0: {"prefix": "status"}) -- 0x55c478e18f40 con 0x55c478e32a30 2024-04-16T16:19:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.587+0000 7fecc9e00640 1 --2- 172.21.15.45:0/3444775989 >> [v2:172.21.15.45:6832/3429980864,v1:172.21.15.45:6834/3429980864] conn(0x55c478e32a30 0x55c478e32e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.587+0000 7fecc9e00640 1 --2- 172.21.15.45:0/3444775989 >> [v2:172.21.15.45:6832/3429980864,v1:172.21.15.45:6834/3429980864] conn(0x55c478e32a30 0x55c478e32e10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29aacd00f2477354 server_cookie=529db5d015a15977 in_seq=0 out_seq=0 2024-04-16T16:19:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.587+0000 7fecaea00640 10 client.8319 ms_handle_connect on v2:172.21.15.45:6832/3429980864 2024-04-16T16:19:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.588+0000 7fecaea00640 1 -- 172.21.15.45:0/3444775989 <== mds.0 v2:172.21.15.45:6832/3429980864 1 ==== command_reply(tid 0: 0 ) ==== 8+0+318 (crc 0 0 0) 0x55c478e18f40 con 0x55c478e32a30 2024-04-16T16:19:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.588+0000 7fecaea00640 10 client.8319 handle_command_reply: tid=0 2024-04-16T16:19:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.588+0000 7fecd3974740 1 client.8319 shutdown 2024-04-16T16:19:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.588+0000 7fecd3974740 1 -- 172.21.15.45:0/3444775989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfca30 msgr2=0x55c478df3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.588+0000 7fecd3974740 1 --2- 172.21.15.45:0/3444775989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfca30 0x55c478df3360 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7feccc06fb60 tx=0x7fecc400e9c0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.588+0000 7fecd3974740 1 -- 172.21.15.45:0/3444775989 shutdown_connections 2024-04-16T16:19:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.588+0000 7fecd3974740 1 --2- 172.21.15.45:0/3444775989 >> [v2:172.21.15.45:6832/3429980864,v1:172.21.15.45:6834/3429980864] conn(0x55c478e32a30 0x55c478e32e10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.588+0000 7fecd3974740 1 --2- 172.21.15.45:0/3444775989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c478dfca30 0x55c478df3360 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.588+0000 7fecd3974740 1 --2- 172.21.15.45:0/3444775989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c478dfc650 0x55c478df2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.588+0000 7fecd3974740 1 --2- 172.21.15.45:0/3444775989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c478df6100 0x55c478e294d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.588+0000 7fecaea00640 0 client.8319 ms_handle_reset on v2:172.21.15.45:6832/3429980864 2024-04-16T16:19:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.588+0000 7fecd3974740 1 -- 172.21.15.45:0/3444775989 >> 172.21.15.45:0/3444775989 conn(0x55c478dd5870 msgr2=0x55c478dcf7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:57.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.588+0000 7fecd3974740 1 -- 172.21.15.45:0/3444775989 shutdown_connections 2024-04-16T16:19:57.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.588+0000 7fecd3974740 1 -- 172.21.15.45:0/3444775989 wait complete. 2024-04-16T16:19:57.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.588+0000 7fecd3974740 20 client.8319 trim_cache size 0 max 16384 2024-04-16T16:19:57.592 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:19:57.592 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "748c4b85-0190-4c59-80ad-098a6a5007c0", 2024-04-16T16:19:57.592 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-04-16T16:19:57.592 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 8160, 2024-04-16T16:19:57.592 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-04-16T16:19:57.592 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-04-16T16:19:57.592 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-04-16T16:19:57.592 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 6.930520885, 2024-04-16T16:19:57.592 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 82, 2024-04-16T16:19:57.593 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 115, 2024-04-16T16:19:57.593 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 114, 2024-04-16T16:19:57.593 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 13.2522389 2024-04-16T16:19:57.593 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:19:57.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.589+0000 7fecd0c00640 1 -- 172.21.15.45:0/1413859260 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7feca806c980 msgr2=0x7feca806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.589+0000 7fecd0c00640 1 --2- 172.21.15.45:0/1413859260 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7feca806c980 0x7feca806ee40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0099d0 tx=0x7fecbc008040 comp rx=0 tx=0).stop 2024-04-16T16:19:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.589+0000 7fecd0c00640 1 -- 172.21.15.45:0/1413859260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feccc0b0c90 msgr2=0x7feccc0b1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.589+0000 7fecd0c00640 1 --2- 172.21.15.45:0/1413859260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feccc0b0c90 0x7feccc0b1110 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7feccc0b0350 tx=0x7fecb4004730 comp rx=0 tx=0).stop 2024-04-16T16:19:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.589+0000 7fecd0c00640 1 -- 172.21.15.45:0/1413859260 shutdown_connections 2024-04-16T16:19:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.589+0000 7fecd0c00640 1 --2- 172.21.15.45:0/1413859260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feccc0aa730 0x7feccc0aabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.589+0000 7fecd0c00640 1 --2- 172.21.15.45:0/1413859260 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7feca806c980 0x7feca806ee40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.589+0000 7fecd0c00640 1 --2- 172.21.15.45:0/1413859260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feccc0b0c90 0x7feccc0b1110 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.589+0000 7fecd0c00640 1 --2- 172.21.15.45:0/1413859260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fecb0004800 0x7feccc0cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.589+0000 7fecd0c00640 1 -- 172.21.15.45:0/1413859260 >> 172.21.15.45:0/1413859260 conn(0x7feccc06b930 msgr2=0x7feccc0b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.589+0000 7fecd0c00640 1 -- 172.21.15.45:0/1413859260 shutdown_connections 2024-04-16T16:19:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.589+0000 7fecd0c00640 1 -- 172.21.15.45:0/1413859260 wait complete. 2024-04-16T16:19:57.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:19:57.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.812+0000 7f3c7f200640 1 Processor -- start 2024-04-16T16:19:57.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.813+0000 7f3c7f200640 1 -- start start 2024-04-16T16:19:57.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.814+0000 7f3c7f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c80146220 0x7f3c8015ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.814+0000 7f3c7f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3c801608a0 0x7f3c80160d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.814+0000 7f3c7f200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c8015d360 con 0x7f3c8006f7a0 2024-04-16T16:19:57.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.814+0000 7f3c7f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3c80160410 con 0x7f3c801608a0 2024-04-16T16:19:57.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.814+0000 7f3c7f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c8015d640 con 0x7f3c80146220 2024-04-16T16:19:57.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.815+0000 7f3c7d400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3c8006f7a0 0x7f3c8006df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33800/0 (socket says 172.21.15.45:33800) 2024-04-16T16:19:57.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.815+0000 7f3c7d400640 1 -- 172.21.15.45:0/3911730101 learned_addr learned my addr 172.21.15.45:0/3911730101 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:19:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.815+0000 7f3c7de00640 1 --2- 172.21.15.45:0/3911730101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c80146220 0x7f3c8015ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.815+0000 7f3c7ca00640 1 -- 172.21.15.45:0/3911730101 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3395251474 0 0) 0x7f3c8015d360 con 0x7f3c8006f7a0 2024-04-16T16:19:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.815+0000 7f3c7e800640 1 --2- 172.21.15.45:0/3911730101 >> v2:172.21.15.103:3300/0 conn(0x7f3c801608a0 0x7f3c80160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.815+0000 7f3c7ca00640 1 -- 172.21.15.45:0/3911730101 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c64003520 con 0x7f3c8006f7a0 2024-04-16T16:19:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.816+0000 7f3c7ca00640 1 -- 172.21.15.45:0/3911730101 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 402945719 0 0) 0x7f3c64003520 con 0x7f3c8006f7a0 2024-04-16T16:19:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.816+0000 7f3c7ca00640 1 -- 172.21.15.45:0/3911730101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c80146220 msgr2=0x7f3c8015ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.816+0000 7f3c7ca00640 1 --2- 172.21.15.45:0/3911730101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c80146220 0x7f3c8015ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.816+0000 7f3c7ca00640 1 -- 172.21.15.45:0/3911730101 >> v2:172.21.15.103:3300/0 conn(0x7f3c801608a0 msgr2=0x7f3c80160d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.816+0000 7f3c7ca00640 1 --2- 172.21.15.45:0/3911730101 >> v2:172.21.15.103:3300/0 conn(0x7f3c801608a0 0x7f3c80160d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.816+0000 7f3c7ca00640 1 -- 172.21.15.45:0/3911730101 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c8015d920 con 0x7f3c8006f7a0 2024-04-16T16:19:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.816+0000 7f3c7ca00640 1 -- 172.21.15.45:0/3911730101 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f3c68001670 con 0x7f3c8006f7a0 2024-04-16T16:19:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.816+0000 7f3c7ca00640 1 -- 172.21.15.45:0/3911730101 >> v1:172.21.15.45:6789/0 conn(0x7f3c8006f7a0 legacy=0x7f3c8006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.816+0000 7f3c7ca00640 1 --2- 172.21.15.45:0/3911730101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c640039b0 0x7f3c64003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.816+0000 7f3c7ca00640 1 --2- 172.21.15.45:0/3911730101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c80146220 0x7f3c64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.816+0000 7f3c7ca00640 1 --2- 172.21.15.45:0/3911730101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c64004d40 0x7f3c640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.816+0000 7f3c7ca00640 1 -- 172.21.15.45:0/3911730101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3c80160410 con 0x7f3c64004d40 2024-04-16T16:19:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.816+0000 7f3c7ca00640 1 -- 172.21.15.45:0/3911730101 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8015d640 con 0x7f3c640039b0 2024-04-16T16:19:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.816+0000 7f3c7ca00640 1 -- 172.21.15.45:0/3911730101 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c6400b9c0 con 0x7f3c80146220 2024-04-16T16:19:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.816+0000 7f3c7e800640 1 --2- 172.21.15.45:0/3911730101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c64004d40 0x7f3c640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.817+0000 7f3c7de00640 1 --2- 172.21.15.45:0/3911730101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c640039b0 0x7f3c64003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.817+0000 7f3c7d400640 1 --2- 172.21.15.45:0/3911730101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c80146220 0x7f3c64004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.817+0000 7f3c7e800640 1 -- 172.21.15.45:0/3911730101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c80146220 msgr2=0x7f3c64004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.817+0000 7f3c7e800640 1 --2- 172.21.15.45:0/3911730101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c80146220 0x7f3c64004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.817+0000 7f3c7e800640 1 -- 172.21.15.45:0/3911730101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c640039b0 msgr2=0x7f3c64003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.817+0000 7f3c7e800640 1 --2- 172.21.15.45:0/3911730101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c640039b0 0x7f3c64003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.817+0000 7f3c7e800640 1 -- 172.21.15.45:0/3911730101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c64003520 con 0x7f3c64004d40 2024-04-16T16:19:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.817+0000 7f3c7d400640 1 --2- 172.21.15.45:0/3911730101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c80146220 0x7f3c64004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.818+0000 7f3c7e800640 1 --2- 172.21.15.45:0/3911730101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c64004d40 0x7f3c640051c0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009fc0 tx=0x7f3c64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aecf22758684d6f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.818+0000 7f3c7ca00640 1 -- 172.21.15.45:0/3911730101 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c78039070 con 0x7f3c64004d40 2024-04-16T16:19:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.818+0000 7f3c7ca00640 1 -- 172.21.15.45:0/3911730101 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c7802cc60 con 0x7f3c64004d40 2024-04-16T16:19:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.818+0000 7f3c7ca00640 1 -- 172.21.15.45:0/3911730101 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c78009c10 con 0x7f3c64004d40 2024-04-16T16:19:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.819+0000 7f3c7f200640 1 -- 172.21.15.45:0/3911730101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c64004d40 msgr2=0x7f3c640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.819+0000 7f3c7f200640 1 --2- 172.21.15.45:0/3911730101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c64004d40 0x7f3c640051c0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009fc0 tx=0x7f3c64005830 comp rx=0 tx=0).stop 2024-04-16T16:19:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.819+0000 7f3c7e800640 1 -- 172.21.15.45:0/3911730101 reap_dead start 2024-04-16T16:19:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.819+0000 7f3c7f200640 1 -- 172.21.15.45:0/3911730101 shutdown_connections 2024-04-16T16:19:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.819+0000 7f3c7f200640 1 -- 172.21.15.45:0/3911730101 >> 172.21.15.45:0/3911730101 conn(0x7f3c8006b930 msgr2=0x7f3c8006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.819+0000 7f3c7f200640 1 -- 172.21.15.45:0/3911730101 shutdown_connections 2024-04-16T16:19:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.819+0000 7f3c7f200640 1 -- 172.21.15.45:0/3911730101 wait complete. 2024-04-16T16:19:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.821+0000 7f3c7f200640 1 Processor -- start 2024-04-16T16:19:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.821+0000 7f3c7f200640 1 -- start start 2024-04-16T16:19:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.822+0000 7f3c7f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c64004d40 0x7f3c801f6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.822+0000 7f3c7f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c801526f0 0x7f3c801f6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.822+0000 7f3c7f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c80145670 0x7f3c801f1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.822+0000 7f3c7f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8006fc20 con 0x7f3c801526f0 2024-04-16T16:19:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.822+0000 7f3c7f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8015dc00 con 0x7f3c64004d40 2024-04-16T16:19:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.822+0000 7f3c7f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c8006fda0 con 0x7f3c80145670 2024-04-16T16:19:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.822+0000 7f3c7d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c801526f0 0x7f3c801f6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.822+0000 7f3c7d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c801526f0 0x7f3c801f6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57646/0 (socket says 172.21.15.45:57646) 2024-04-16T16:19:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.822+0000 7f3c7d400640 1 -- 172.21.15.45:0/4283062351 learned_addr learned my addr 172.21.15.45:0/4283062351 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.823+0000 7f3c7de00640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c64004d40 0x7f3c801f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.823+0000 7f3c7e800640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c80145670 0x7f3c801f1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.823+0000 7f3c7d400640 1 -- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c80145670 msgr2=0x7f3c801f1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.823+0000 7f3c7d400640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c80145670 0x7f3c801f1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.823+0000 7f3c7d400640 1 -- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c64004d40 msgr2=0x7f3c801f6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.823+0000 7f3c7d400640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c64004d40 0x7f3c801f6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.823+0000 7f3c7d400640 1 -- 172.21.15.45:0/4283062351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c80142740 con 0x7f3c801526f0 2024-04-16T16:19:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.823+0000 7f3c7de00640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c64004d40 0x7f3c801f6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.823+0000 7f3c7e800640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c80145670 0x7f3c801f1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.823+0000 7f3c7d400640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c801526f0 0x7f3c801f6770 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800dae0 tx=0x7f3c68007e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.823+0000 7f3c76000640 1 -- 172.21.15.45:0/4283062351 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c68019070 con 0x7f3c801526f0 2024-04-16T16:19:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.824+0000 7f3c76000640 1 -- 172.21.15.45:0/4283062351 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c68004bf0 con 0x7f3c801526f0 2024-04-16T16:19:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.824+0000 7f3c76000640 1 -- 172.21.15.45:0/4283062351 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c680070d0 con 0x7f3c801526f0 2024-04-16T16:19:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.824+0000 7f3c7f200640 1 -- 172.21.15.45:0/4283062351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c80142a20 con 0x7f3c801526f0 2024-04-16T16:19:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.824+0000 7f3c7f200640 1 -- 172.21.15.45:0/4283062351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c80144e70 con 0x7f3c801526f0 2024-04-16T16:19:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.825+0000 7f3c7f200640 1 -- 172.21.15.45:0/4283062351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c800b69f0 con 0x7f3c801526f0 2024-04-16T16:19:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.826+0000 7f3c76000640 1 -- 172.21.15.45:0/4283062351 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 41) ==== 91142+0+0 (secure 0 0 0) 0x7f3c68007270 con 0x7f3c801526f0 2024-04-16T16:19:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.827+0000 7f3c76000640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3c5c06ca40 0x7f3c5c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.827+0000 7f3c76000640 1 -- 172.21.15.45:0/4283062351 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5752+0+0 (secure 0 0 0) 0x7f3c6808ba70 con 0x7f3c801526f0 2024-04-16T16:19:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.827+0000 7f3c7de00640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3c5c06ca40 0x7f3c5c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:57.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.829+0000 7f3c7de00640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3c5c06ca40 0x7f3c5c06ef00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3c70009980 tx=0x7f3c70008040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:57.831+0000 7f3c76000640 1 -- 172.21.15.45:0/4283062351 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3c800b69f0 con 0x7f3c801526f0 2024-04-16T16:19:58.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58.077+0000 7f3c7f200640 1 -- 172.21.15.45:0/4283062351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f3c80075780 con 0x7f3c801526f0 2024-04-16T16:19:58.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58.078+0000 7f3c76000640 1 -- 172.21.15.45:0/4283062351 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v82) ==== 53+0+73 (secure 0 0 0) 0x7f3c68059bb0 con 0x7f3c801526f0 2024-04-16T16:19:58.078 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:19:58.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58.080+0000 7f3c7f200640 1 -- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3c5c06ca40 msgr2=0x7f3c5c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:58.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58.080+0000 7f3c7f200640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3c5c06ca40 0x7f3c5c06ef00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3c70009980 tx=0x7f3c70008040 comp rx=0 tx=0).stop 2024-04-16T16:19:58.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58.080+0000 7f3c7f200640 1 -- 172.21.15.45:0/4283062351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c801526f0 msgr2=0x7f3c801f6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:58.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58.080+0000 7f3c7f200640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c801526f0 0x7f3c801f6770 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800dae0 tx=0x7f3c68007e90 comp rx=0 tx=0).stop 2024-04-16T16:19:58.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58.081+0000 7f3c7f200640 1 -- 172.21.15.45:0/4283062351 shutdown_connections 2024-04-16T16:19:58.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58.081+0000 7f3c7f200640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c80145670 0x7f3c801f1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:58.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58.081+0000 7f3c7f200640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f3c5c06ca40 0x7f3c5c06ef00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:58.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58.081+0000 7f3c7f200640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c801526f0 0x7f3c801f6770 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:58.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58.081+0000 7f3c7f200640 1 --2- 172.21.15.45:0/4283062351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c64004d40 0x7f3c801f6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:58.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58.081+0000 7f3c7f200640 1 -- 172.21.15.45:0/4283062351 >> 172.21.15.45:0/4283062351 conn(0x7f3c8006b930 msgr2=0x7f3c8015f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:58.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58.081+0000 7f3c7f200640 1 -- 172.21.15.45:0/4283062351 shutdown_connections 2024-04-16T16:19:58.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58.081+0000 7f3c7f200640 1 -- 172.21.15.45:0/4283062351 wait complete. 2024-04-16T16:19:58.101 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-16T16:19:58.101 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:19:58.101 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-04-16T16:19:58.101 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-16T16:19:58.101 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T16:19:58.101 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-16T16:19:58.101 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:19:58.101 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:19:58.101 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:19:58.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:58.102 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-04-16T16:19:58.153 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:19:58.154 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:19:58.154 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:19:58.154 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:58.154 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-04-16T16:19:58.154 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:58.154 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:19:58.154 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:58.154 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-04-16T16:19:58.154 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:19:58.154 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:58.155 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-04-16T16:19:58.155 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:19:58.155 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:58.155 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-04-16T16:19:58.155 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-16T16:19:58.155 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42203sec preferred_lft 42203sec 2024-04-16T16:19:58.155 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-04-16T16:19:58.155 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:58.155 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:19:58.155 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:58.155 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-04-16T16:19:58.155 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:19:58.155 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:58.155 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:19:58.156 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:58.156 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::50f1:99ff:fe2b:225b/64 scope link 2024-04-16T16:19:58.156 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:58.156 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:19:58.156 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T16:19:58.156 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::7007:c3ff:fe13:7bcd/64 scope link 2024-04-16T16:19:58.156 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:58.156 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:19:58.156 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether ae:10:ea:fc:0a:13 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-16T16:19:58.156 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ac10:eaff:fefc:a13/64 scope link 2024-04-16T16:19:58.156 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:58.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:58.157 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:19:58.248 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-04-16T16:19:58.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:58.253 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-04-16T16:19:58.308 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-04-16T16:19:58.309 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:19:58.309 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-16T16:19:58.309 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-16T16:19:58.309 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:19:58.310 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:19:58.310 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:19:58.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-16T16:19:58.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-16T16:19:58.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-16T16:19:58.310 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:19:58.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:19:58.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:19:58.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:58.573 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T16:19:58.627 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-16T16:19:58.627 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-16T16:19:58.628 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:19:58.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:58.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-04-16T16:19:58.734 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:19:58.735 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-16T16:19:58.788 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:19:58.788 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:19:58.788 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:19:58.840 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-16T16:19:58.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:58.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:19:58.986 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:19:58.986 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:19:59.023 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:19:59.023 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:19:59.023 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:19:59.023 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 0@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:19:59.023 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:19:59.026 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-16T16:19:59.026 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:19:59.026 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:19:59.026 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:19:59.026 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:19:59.030 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:19:59.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:59.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:19:59.118 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:19:59.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:19:59.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:19:59.253 INFO:teuthology.orchestra.run.smithi045.stdout:client6125 2024-04-16T16:19:59.253 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:19:59.253 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:19:59.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:19:59.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.536+0000 7f642b200640 1 Processor -- start 2024-04-16T16:19:59.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.536+0000 7f642b200640 1 -- start start 2024-04-16T16:19:59.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.537+0000 7f642b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6424162270 0x7f64240d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:59.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.537+0000 7f642b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f64240d52d0 0x7f642406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:59.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.537+0000 7f642b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64240d3080 con 0x7f6424161e90 2024-04-16T16:19:59.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.537+0000 7f642b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f64240d2e30 con 0x7f64240d52d0 2024-04-16T16:19:59.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.537+0000 7f642b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f64240d3360 con 0x7f6424162270 2024-04-16T16:19:59.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.538+0000 7f642a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f64240d52d0 0x7f642406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:59.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.538+0000 7f642a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f64240d52d0 0x7f642406ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:43926/0 (socket says 172.21.15.45:43926) 2024-04-16T16:19:59.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.538+0000 7f6429400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6424162270 0x7f64240d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:59.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.538+0000 7f642a800640 1 -- 172.21.15.45:0/2054126640 learned_addr learned my addr 172.21.15.45:0/2054126640 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:59.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.539+0000 7f642a800640 1 -- 172.21.15.45:0/2054126640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6424162270 msgr2=0x7f64240d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:19:59.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.539+0000 7f642a800640 1 --2- 172.21.15.45:0/2054126640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6424162270 0x7f64240d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:59.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.539+0000 7f642a800640 1 -- 172.21.15.45:0/2054126640 >> v1:172.21.15.103:6789/0 conn(0x7f6424161e90 legacy=0x7f64240cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:59.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.539+0000 7f642a800640 1 -- 172.21.15.45:0/2054126640 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64240d3640 con 0x7f64240d52d0 2024-04-16T16:19:59.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.540+0000 7f642a800640 1 --2- 172.21.15.45:0/2054126640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64240d52d0 0x7f642406ce40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6420009e50 tx=0x7f642002bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a7a3e071515dc9c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:59.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.541+0000 7f6428a00640 1 -- 172.21.15.45:0/2054126640 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642002cb90 con 0x7f64240d52d0 2024-04-16T16:19:59.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.541+0000 7f6428a00640 1 -- 172.21.15.45:0/2054126640 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f642002cd30 con 0x7f64240d52d0 2024-04-16T16:19:59.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.541+0000 7f6428a00640 1 -- 172.21.15.45:0/2054126640 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6420034610 con 0x7f64240d52d0 2024-04-16T16:19:59.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.542+0000 7f642b200640 1 -- 172.21.15.45:0/2054126640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64240d52d0 msgr2=0x7f642406ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:59.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.542+0000 7f642b200640 1 --2- 172.21.15.45:0/2054126640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64240d52d0 0x7f642406ce40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6420009e50 tx=0x7f642002bff0 comp rx=0 tx=0).stop 2024-04-16T16:19:59.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.542+0000 7f642b200640 1 -- 172.21.15.45:0/2054126640 shutdown_connections 2024-04-16T16:19:59.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.542+0000 7f642b200640 1 --2- 172.21.15.45:0/2054126640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64240d52d0 0x7f642406ce40 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:59.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.542+0000 7f642b200640 1 --2- 172.21.15.45:0/2054126640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6424162270 0x7f64240d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:59.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.542+0000 7f642b200640 1 -- 172.21.15.45:0/2054126640 >> 172.21.15.45:0/2054126640 conn(0x7f64240d1940 msgr2=0x7f64240d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:59.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.542+0000 7f642b200640 1 -- 172.21.15.45:0/2054126640 shutdown_connections 2024-04-16T16:19:59.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.542+0000 7f642b200640 1 -- 172.21.15.45:0/2054126640 wait complete. 2024-04-16T16:19:59.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.544+0000 7f642b200640 1 Processor -- start 2024-04-16T16:19:59.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.544+0000 7f642b200640 1 -- start start 2024-04-16T16:19:59.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.545+0000 7f642b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64240d52d0 0x7f642414c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:59.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.545+0000 7f642b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6424161e90 0x7f642415c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.545+0000 7f642b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6424162270 0x7f64240b7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.545+0000 7f642b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f64240d4500 con 0x7f64240d52d0 2024-04-16T16:19:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.545+0000 7f642b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f64240d4380 con 0x7f6424162270 2024-04-16T16:19:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.545+0000 7f642b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f64240d4680 con 0x7f6424161e90 2024-04-16T16:19:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.545+0000 7f6429e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64240d52d0 0x7f642414c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.545+0000 7f6429e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64240d52d0 0x7f642414c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57658/0 (socket says 172.21.15.45:57658) 2024-04-16T16:19:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.545+0000 7f6429e00640 1 -- 172.21.15.45:0/1666720653 learned_addr learned my addr 172.21.15.45:0/1666720653 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:19:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.545+0000 7f642a800640 1 --2- 172.21.15.45:0/1666720653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6424162270 0x7f64240b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.545+0000 7f6429400640 1 --2- 172.21.15.45:0/1666720653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6424161e90 0x7f642415c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.546+0000 7f6429e00640 1 -- 172.21.15.45:0/1666720653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6424161e90 msgr2=0x7f642415c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.546+0000 7f6429e00640 1 --2- 172.21.15.45:0/1666720653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6424161e90 0x7f642415c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.546+0000 7f6429e00640 1 -- 172.21.15.45:0/1666720653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6424162270 msgr2=0x7f64240b7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.546+0000 7f6429e00640 1 --2- 172.21.15.45:0/1666720653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6424162270 0x7f64240b7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.546+0000 7f6429e00640 1 -- 172.21.15.45:0/1666720653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6424159980 con 0x7f64240d52d0 2024-04-16T16:19:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.546+0000 7f642a800640 1 --2- 172.21.15.45:0/1666720653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6424162270 0x7f64240b7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:19:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.546+0000 7f6429e00640 1 --2- 172.21.15.45:0/1666720653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64240d52d0 0x7f642414c430 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f641800c430 tx=0x7f641800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.546+0000 7f641e000640 1 -- 172.21.15.45:0/1666720653 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64180052d0 con 0x7f64240d52d0 2024-04-16T16:19:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.546+0000 7f641e000640 1 -- 172.21.15.45:0/1666720653 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6418005490 con 0x7f64240d52d0 2024-04-16T16:19:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.546+0000 7f641e000640 1 -- 172.21.15.45:0/1666720653 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6418007260 con 0x7f64240d52d0 2024-04-16T16:19:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.546+0000 7f642b200640 1 -- 172.21.15.45:0/1666720653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f642414cd40 con 0x7f64240d52d0 2024-04-16T16:19:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.546+0000 7f642b200640 1 -- 172.21.15.45:0/1666720653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6424159c20 con 0x7f64240d52d0 2024-04-16T16:19:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.547+0000 7f642b200640 1 -- 172.21.15.45:0/1666720653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f642414d360 con 0x7f64240d52d0 2024-04-16T16:19:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.548+0000 7f641e000640 1 -- 172.21.15.45:0/1666720653 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 41) ==== 91142+0+0 (secure 0 0 0) 0x7f6418007400 con 0x7f64240d52d0 2024-04-16T16:19:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.550+0000 7f641e000640 1 --2- 172.21.15.45:0/1666720653 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f640806ca10 0x7f640806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:19:59.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.550+0000 7f641e000640 1 -- 172.21.15.45:0/1666720653 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5752+0+0 (secure 0 0 0) 0x7f641808b300 con 0x7f64240d52d0 2024-04-16T16:19:59.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.551+0000 7f6429400640 1 --2- 172.21.15.45:0/1666720653 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f640806ca10 0x7f640806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:19:59.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.552+0000 7f6429400640 1 --2- 172.21.15.45:0/1666720653 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f640806ca10 0x7f640806eed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f64240b9b30 tx=0x7f6414008040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:19:59.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.555+0000 7f641e000640 1 -- 172.21.15.45:0/1666720653 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6418059550 con 0x7f64240d52d0 2024-04-16T16:19:59.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.808+0000 7f642b200640 1 -- 172.21.15.45:0/1666720653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f642406dae0 con 0x7f64240d52d0 2024-04-16T16:19:59.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.809+0000 7f641e000640 1 -- 172.21.15.45:0/1666720653 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v82) ==== 53+0+73 (secure 0 0 0) 0x7f6418058e50 con 0x7f64240d52d0 2024-04-16T16:19:59.809 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:19:59.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.811+0000 7f642b200640 1 -- 172.21.15.45:0/1666720653 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f640806ca10 msgr2=0x7f640806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:59.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.811+0000 7f642b200640 1 --2- 172.21.15.45:0/1666720653 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f640806ca10 0x7f640806eed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f64240b9b30 tx=0x7f6414008040 comp rx=0 tx=0).stop 2024-04-16T16:19:59.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.811+0000 7f642b200640 1 -- 172.21.15.45:0/1666720653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64240d52d0 msgr2=0x7f642414c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:19:59.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.811+0000 7f642b200640 1 --2- 172.21.15.45:0/1666720653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64240d52d0 0x7f642414c430 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f641800c430 tx=0x7f641800c8f0 comp rx=0 tx=0).stop 2024-04-16T16:19:59.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.811+0000 7f642b200640 1 -- 172.21.15.45:0/1666720653 shutdown_connections 2024-04-16T16:19:59.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.811+0000 7f642b200640 1 --2- 172.21.15.45:0/1666720653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6424162270 0x7f64240b7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:59.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.811+0000 7f642b200640 1 --2- 172.21.15.45:0/1666720653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6424161e90 0x7f642415c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:59.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.811+0000 7f642b200640 1 --2- 172.21.15.45:0/1666720653 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f640806ca10 0x7f640806eed0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:59.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.811+0000 7f642b200640 1 --2- 172.21.15.45:0/1666720653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64240d52d0 0x7f642414c430 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:19:59.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.811+0000 7f642b200640 1 -- 172.21.15.45:0/1666720653 >> 172.21.15.45:0/1666720653 conn(0x7f64240d1940 msgr2=0x7f64240b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:19:59.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.812+0000 7f642b200640 1 -- 172.21.15.45:0/1666720653 shutdown_connections 2024-04-16T16:19:59.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:19:59.812+0000 7f642b200640 1 -- 172.21.15.45:0/1666720653 wait complete. 2024-04-16T16:19:59.830 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-16T16:19:59.830 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:19:59.831 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-04-16T16:19:59.831 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-16T16:19:59.831 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T16:19:59.831 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-16T16:19:59.831 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:19:59.831 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:19:59.831 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:19:59.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:59.831 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-04-16T16:19:59.888 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:19:59.888 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:19:59.888 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:19:59.888 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:59.888 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-16T16:19:59.888 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:59.888 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:19:59.888 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:59.888 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-04-16T16:19:59.889 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:19:59.889 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:59.889 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-04-16T16:19:59.889 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:19:59.889 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:59.889 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:19:59.889 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:59.889 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-16T16:19:59.889 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42225sec preferred_lft 42225sec 2024-04-16T16:19:59.889 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-04-16T16:19:59.889 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:59.889 INFO:teuthology.orchestra.run.smithi103.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:19:59.889 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:19:59.889 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:19:59.889 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:59.890 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::74ce:ceff:fec6:2a00/64 scope link 2024-04-16T16:19:59.890 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:59.890 INFO:teuthology.orchestra.run.smithi103.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:19:59.890 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T16:19:59.890 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::3cf7:ffff:feca:f879/64 scope link 2024-04-16T16:19:59.890 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:59.890 INFO:teuthology.orchestra.run.smithi103.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:19:59.890 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 86:4c:6e:ce:c3:07 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-16T16:19:59.890 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::844c:6eff:fece:c307/64 scope link 2024-04-16T16:19:59.890 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:19:59.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:59.891 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:19:59.982 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-16T16:19:59.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:19:59.987 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-04-16T16:20:00.036 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-04-16T16:20:00.036 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:20:00.036 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-16T16:20:00.036 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-16T16:20:00.036 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:20:00.037 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:20:00.037 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:20:00.037 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-16T16:20:00.037 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-16T16:20:00.037 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-16T16:20:00.037 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:20:00.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:20:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:20:00.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:20:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:20:00.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:00.283 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-16T16:20:00.330 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-16T16:20:00.330 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-16T16:20:00.330 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:20:00.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:00.331 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.0 up 2024-04-16T16:20:00.455 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:20:00.455 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-16T16:20:00.496 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:20:00.496 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:20:00.496 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:20:00.586 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-16T16:20:00.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:00.587 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:20:00.715 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:20:00.715 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:20:00.750 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:20:00.750 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:20:00.750 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:20:00.750 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with new device syntax: 1@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:20:00.750 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:20:00.753 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-16T16:20:00.753 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:20:00.753 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:20:00.753 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:20:00.753 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:20:00.756 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:20:00.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:00.757 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:20:00.829 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:20:00.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:20:00 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:20:00.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:20:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:20:00.935 INFO:teuthology.orchestra.run.smithi103.stdout:client6730 2024-04-16T16:20:00.935 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:20:00.935 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:20:00.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:20:01.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.220+0000 7f0c43200640 1 Processor -- start 2024-04-16T16:20:01.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.220+0000 7f0c43200640 1 -- start start 2024-04-16T16:20:01.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.222+0000 7f0c43200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c44146220 0x7f0c4415ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:01.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.222+0000 7f0c43200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0c441608a0 0x7f0c44160d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:01.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.222+0000 7f0c43200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c4415d360 con 0x7f0c4406f7a0 2024-04-16T16:20:01.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.222+0000 7f0c43200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0c44160410 con 0x7f0c441608a0 2024-04-16T16:20:01.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.222+0000 7f0c43200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0c4415d640 con 0x7f0c44146220 2024-04-16T16:20:01.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.223+0000 7f0c41e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c44146220 0x7f0c4415ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:01.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.223+0000 7f0c41e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c44146220 0x7f0c4415ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:35210/0 (socket says 172.21.15.45:35210) 2024-04-16T16:20:01.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.223+0000 7f0c41e00640 1 -- 172.21.15.45:0/1235898673 learned_addr learned my addr 172.21.15.45:0/1235898673 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:01.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.223+0000 7f0c42800640 1 --2- 172.21.15.45:0/1235898673 >> v2:172.21.15.103:3300/0 conn(0x7f0c441608a0 0x7f0c44160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:01.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.223+0000 7f0c41e00640 1 -- 172.21.15.45:0/1235898673 >> v2:172.21.15.103:3300/0 conn(0x7f0c441608a0 msgr2=0x7f0c44160d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:01.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.223+0000 7f0c41e00640 1 --2- 172.21.15.45:0/1235898673 >> v2:172.21.15.103:3300/0 conn(0x7f0c441608a0 0x7f0c44160d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:01.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.223+0000 7f0c41e00640 1 -- 172.21.15.45:0/1235898673 >> v1:172.21.15.103:6789/0 conn(0x7f0c4406f7a0 legacy=0x7f0c4406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:01.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.223+0000 7f0c41e00640 1 -- 172.21.15.45:0/1235898673 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c4415d920 con 0x7f0c44146220 2024-04-16T16:20:01.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.225+0000 7f0c41e00640 1 --2- 172.21.15.45:0/1235898673 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c44146220 0x7f0c4415ccb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009870 tx=0x7f0c3402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3f0d7eac21f35b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:01.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.225+0000 7f0c40a00640 1 -- 172.21.15.45:0/1235898673 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34034070 con 0x7f0c44146220 2024-04-16T16:20:01.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.225+0000 7f0c40a00640 1 -- 172.21.15.45:0/1235898673 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c34030840 con 0x7f0c44146220 2024-04-16T16:20:01.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.225+0000 7f0c40a00640 1 -- 172.21.15.45:0/1235898673 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34030c10 con 0x7f0c44146220 2024-04-16T16:20:01.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.226+0000 7f0c43200640 1 -- 172.21.15.45:0/1235898673 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c44146220 msgr2=0x7f0c4415ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:01.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.226+0000 7f0c43200640 1 --2- 172.21.15.45:0/1235898673 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c44146220 0x7f0c4415ccb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009870 tx=0x7f0c3402b980 comp rx=0 tx=0).stop 2024-04-16T16:20:01.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.226+0000 7f0c43200640 1 -- 172.21.15.45:0/1235898673 shutdown_connections 2024-04-16T16:20:01.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.226+0000 7f0c43200640 1 --2- 172.21.15.45:0/1235898673 >> v2:172.21.15.103:3300/0 conn(0x7f0c441608a0 0x7f0c44160d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:01.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.226+0000 7f0c43200640 1 --2- 172.21.15.45:0/1235898673 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c44146220 0x7f0c4415ccb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:01.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.226+0000 7f0c43200640 1 -- 172.21.15.45:0/1235898673 >> 172.21.15.45:0/1235898673 conn(0x7f0c4406b930 msgr2=0x7f0c4406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:01.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.226+0000 7f0c43200640 1 -- 172.21.15.45:0/1235898673 shutdown_connections 2024-04-16T16:20:01.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.226+0000 7f0c43200640 1 -- 172.21.15.45:0/1235898673 wait complete. 2024-04-16T16:20:01.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.228+0000 7f0c43200640 1 Processor -- start 2024-04-16T16:20:01.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.228+0000 7f0c43200640 1 -- start start 2024-04-16T16:20:01.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.229+0000 7f0c43200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c4406f7a0 0x7f0c440cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:01.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.230+0000 7f0c43200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c44146220 0x7f0c4415c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:01.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.230+0000 7f0c43200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c441608a0 0x7f0c4413ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:01.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.230+0000 7f0c43200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4406fc20 con 0x7f0c4406f7a0 2024-04-16T16:20:01.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.230+0000 7f0c43200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4415dc00 con 0x7f0c44146220 2024-04-16T16:20:01.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.230+0000 7f0c43200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0c4406fda0 con 0x7f0c441608a0 2024-04-16T16:20:01.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.230+0000 7f0c41e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c4406f7a0 0x7f0c440cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:01.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.230+0000 7f0c41e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c4406f7a0 0x7f0c440cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57662/0 (socket says 172.21.15.45:57662) 2024-04-16T16:20:01.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.230+0000 7f0c41e00640 1 -- 172.21.15.45:0/13325640 learned_addr learned my addr 172.21.15.45:0/13325640 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:01.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.230+0000 7f0c42800640 1 --2- 172.21.15.45:0/13325640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c441608a0 0x7f0c4413ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.230+0000 7f0c41e00640 1 -- 172.21.15.45:0/13325640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c441608a0 msgr2=0x7f0c4413ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.230+0000 7f0c41e00640 1 --2- 172.21.15.45:0/13325640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c441608a0 0x7f0c4413ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.230+0000 7f0c41400640 1 --2- 172.21.15.45:0/13325640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c44146220 0x7f0c4415c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.230+0000 7f0c41e00640 1 -- 172.21.15.45:0/13325640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c44146220 msgr2=0x7f0c4415c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.231+0000 7f0c41e00640 1 --2- 172.21.15.45:0/13325640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c44146220 0x7f0c4415c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.231+0000 7f0c41e00640 1 -- 172.21.15.45:0/13325640 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c440cfad0 con 0x7f0c4406f7a0 2024-04-16T16:20:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.231+0000 7f0c41e00640 1 --2- 172.21.15.45:0/13325640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c4406f7a0 0x7f0c440cf590 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f0c34000c00 tx=0x7f0c34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.231+0000 7f0c3a000640 1 -- 172.21.15.45:0/13325640 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34048070 con 0x7f0c4406f7a0 2024-04-16T16:20:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.231+0000 7f0c3a000640 1 -- 172.21.15.45:0/13325640 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c34030640 con 0x7f0c4406f7a0 2024-04-16T16:20:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.231+0000 7f0c3a000640 1 -- 172.21.15.45:0/13325640 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34034040 con 0x7f0c4406f7a0 2024-04-16T16:20:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.231+0000 7f0c43200640 1 -- 172.21.15.45:0/13325640 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c440ce4c0 con 0x7f0c4406f7a0 2024-04-16T16:20:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.232+0000 7f0c43200640 1 -- 172.21.15.45:0/13325640 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c440ca5f0 con 0x7f0c4406f7a0 2024-04-16T16:20:01.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.233+0000 7f0c43200640 1 -- 172.21.15.45:0/13325640 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c44075780 con 0x7f0c4406f7a0 2024-04-16T16:20:01.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.234+0000 7f0c3a000640 1 -- 172.21.15.45:0/13325640 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 41) ==== 91142+0+0 (secure 0 0 0) 0x7f0c3404b070 con 0x7f0c4406f7a0 2024-04-16T16:20:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.234+0000 7f0c3a000640 1 --2- 172.21.15.45:0/13325640 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f0c2406ca10 0x7f0c2406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.235+0000 7f0c3a000640 1 -- 172.21.15.45:0/13325640 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5752+0+0 (secure 0 0 0) 0x7f0c340adde0 con 0x7f0c4406f7a0 2024-04-16T16:20:01.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.235+0000 7f0c41400640 1 --2- 172.21.15.45:0/13325640 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f0c2406ca10 0x7f0c2406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:01.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.236+0000 7f0c41400640 1 --2- 172.21.15.45:0/13325640 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f0c2406ca10 0x7f0c2406eed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0c440ccb20 tx=0x7f0c30009040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.239+0000 7f0c3a000640 1 -- 172.21.15.45:0/13325640 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0c3407bfc0 con 0x7f0c4406f7a0 2024-04-16T16:20:01.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.483+0000 7f0c43200640 1 -- 172.21.15.45:0/13325640 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c440cff60 con 0x7f0c4406f7a0 2024-04-16T16:20:01.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.484+0000 7f0c3a000640 1 -- 172.21.15.45:0/13325640 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5186 (secure 0 0 0) 0x7f0c3407b8c0 con 0x7f0c4406f7a0 2024-04-16T16:20:01.484 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:01.484 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6056,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1032791151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":1032791151},{"type":"v1","addr":"172.21.15.45:6835","nonce":1032791151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6062,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/498835619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":498835619},{"type":"v1","addr":"172.21.15.103:6837","nonce":498835619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/3611628422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3611628422},{"type":"v1","addr":"172.21.15.103:6836","nonce":3611628422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:19:50.154928+0000","modified":"2024-04-16T16:19:53.221144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8160},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8160":{"gid":8160,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/3429980864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3429980864},{"type":"v1","addr":"172.21.15.45:6834","nonce":3429980864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8160,"qdb_cluster":[8160]},"id":8}]} 2024-04-16T16:20:01.485 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 82 2024-04-16T16:20:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.486+0000 7f0c43200640 1 -- 172.21.15.45:0/13325640 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f0c2406ca10 msgr2=0x7f0c2406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.486+0000 7f0c43200640 1 --2- 172.21.15.45:0/13325640 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f0c2406ca10 0x7f0c2406eed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0c440ccb20 tx=0x7f0c30009040 comp rx=0 tx=0).stop 2024-04-16T16:20:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.486+0000 7f0c43200640 1 -- 172.21.15.45:0/13325640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c4406f7a0 msgr2=0x7f0c440cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.486+0000 7f0c43200640 1 --2- 172.21.15.45:0/13325640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c4406f7a0 0x7f0c440cf590 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f0c34000c00 tx=0x7f0c34004730 comp rx=0 tx=0).stop 2024-04-16T16:20:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.487+0000 7f0c43200640 1 -- 172.21.15.45:0/13325640 shutdown_connections 2024-04-16T16:20:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.487+0000 7f0c43200640 1 --2- 172.21.15.45:0/13325640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c441608a0 0x7f0c4413ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.487+0000 7f0c43200640 1 --2- 172.21.15.45:0/13325640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c44146220 0x7f0c4415c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.487+0000 7f0c43200640 1 --2- 172.21.15.45:0/13325640 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7f0c2406ca10 0x7f0c2406eed0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.487+0000 7f0c43200640 1 --2- 172.21.15.45:0/13325640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c4406f7a0 0x7f0c440cf590 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.487+0000 7f0c43200640 1 -- 172.21.15.45:0/13325640 >> 172.21.15.45:0/13325640 conn(0x7f0c4406b930 msgr2=0x7f0c4415e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.487+0000 7f0c43200640 1 -- 172.21.15.45:0/13325640 shutdown_connections 2024-04-16T16:20:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.487+0000 7f0c43200640 1 -- 172.21.15.45:0/13325640 wait complete. 2024-04-16T16:20:01.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable stats 2024-04-16T16:20:01.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.740+0000 7fbbda400640 1 Processor -- start 2024-04-16T16:20:01.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.741+0000 7fbbda400640 1 -- start start 2024-04-16T16:20:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.742+0000 7fbbda400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbbd40afda0 0x7fbbd40b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.742+0000 7fbbda400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbd40ccb70 con 0x7fbbd415b920 2024-04-16T16:20:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.742+0000 7fbbda400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbd40cce50 con 0x7fbbd40d0e80 2024-04-16T16:20:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.742+0000 7fbbda400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fbbd406e7a0 con 0x7fbbd40afda0 2024-04-16T16:20:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.743+0000 7fbbd3400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbbd415b920 0x7fbbd40d0770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33812/0 (socket says 172.21.15.45:33812) 2024-04-16T16:20:01.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.743+0000 7fbbd3400640 1 -- 172.21.15.45:0/921914171 learned_addr learned my addr 172.21.15.45:0/921914171 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:01.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.743+0000 7fbbd3e00640 1 --2- 172.21.15.45:0/921914171 >> v2:172.21.15.45:3300/0 conn(0x7fbbd40afda0 0x7fbbd40b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:01.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.743+0000 7fbbd2000640 1 -- 172.21.15.45:0/921914171 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 436290083 0 0) 0x7fbbd40ccb70 con 0x7fbbd415b920 2024-04-16T16:20:01.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.743+0000 7fbbd2000640 1 -- 172.21.15.45:0/921914171 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbb8005730 con 0x7fbbd415b920 2024-04-16T16:20:01.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.743+0000 7fbbd3e00640 1 -- 172.21.15.45:0/921914171 >> v1:172.21.15.103:6789/0 conn(0x7fbbd40d0e80 legacy=0x7fbbd40ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:01.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.743+0000 7fbbd3e00640 1 -- 172.21.15.45:0/921914171 >> v1:172.21.15.45:6789/0 conn(0x7fbbd415b920 legacy=0x7fbbd40d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:01.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.743+0000 7fbbd3e00640 1 -- 172.21.15.45:0/921914171 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd406e430 con 0x7fbbd40afda0 2024-04-16T16:20:01.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.744+0000 7fbbd3e00640 1 --2- 172.21.15.45:0/921914171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd40afda0 0x7fbbd40b4e20 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fbbd406e920 tx=0x7fbbcc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f507399986a4ed3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:01.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.745+0000 7fbbd2000640 1 -- 172.21.15.45:0/921914171 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc034070 con 0x7fbbd40afda0 2024-04-16T16:20:01.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.745+0000 7fbbd2000640 1 -- 172.21.15.45:0/921914171 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbcc030610 con 0x7fbbd40afda0 2024-04-16T16:20:01.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.745+0000 7fbbd2000640 1 -- 172.21.15.45:0/921914171 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc0309e0 con 0x7fbbd40afda0 2024-04-16T16:20:01.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.745+0000 7fbbda400640 1 -- 172.21.15.45:0/921914171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd40afda0 msgr2=0x7fbbd40b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:01.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.745+0000 7fbbda400640 1 --2- 172.21.15.45:0/921914171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd40afda0 0x7fbbd40b4e20 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fbbd406e920 tx=0x7fbbcc02b960 comp rx=0 tx=0).stop 2024-04-16T16:20:01.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.745+0000 7fbbda400640 1 -- 172.21.15.45:0/921914171 shutdown_connections 2024-04-16T16:20:01.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.745+0000 7fbbda400640 1 --2- 172.21.15.45:0/921914171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd40afda0 0x7fbbd40b4e20 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:01.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.745+0000 7fbbda400640 1 -- 172.21.15.45:0/921914171 >> 172.21.15.45:0/921914171 conn(0x7fbbd40b3970 msgr2=0x7fbbd40b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:01.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.746+0000 7fbbda400640 1 -- 172.21.15.45:0/921914171 shutdown_connections 2024-04-16T16:20:01.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.746+0000 7fbbda400640 1 -- 172.21.15.45:0/921914171 wait complete. 2024-04-16T16:20:01.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.747+0000 7fbbda400640 1 Processor -- start 2024-04-16T16:20:01.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.747+0000 7fbbda400640 1 -- start start 2024-04-16T16:20:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.748+0000 7fbbda400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbd40afda0 0x7fbbd4145fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.748+0000 7fbbda400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd40d0e80 0x7fbbd4146500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.748+0000 7fbbda400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbd415b920 0x7fbbd40ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.748+0000 7fbbda400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd406f000 con 0x7fbbd40d0e80 2024-04-16T16:20:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.748+0000 7fbbda400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd406ee80 con 0x7fbbd40afda0 2024-04-16T16:20:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.748+0000 7fbbda400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbbd406f180 con 0x7fbbd415b920 2024-04-16T16:20:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.749+0000 7fbbd2a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd40d0e80 0x7fbbd4146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.749+0000 7fbbd2a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd40d0e80 0x7fbbd4146500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57672/0 (socket says 172.21.15.45:57672) 2024-04-16T16:20:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.749+0000 7fbbd2a00640 1 -- 172.21.15.45:0/3085065026 learned_addr learned my addr 172.21.15.45:0/3085065026 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.749+0000 7fbbd3e00640 1 --2- 172.21.15.45:0/3085065026 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbd415b920 0x7fbbd40ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.749+0000 7fbbd3400640 1 --2- 172.21.15.45:0/3085065026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbd40afda0 0x7fbbd4145fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.749+0000 7fbbd2a00640 1 -- 172.21.15.45:0/3085065026 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbd415b920 msgr2=0x7fbbd40ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.749+0000 7fbbd2a00640 1 --2- 172.21.15.45:0/3085065026 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbd415b920 0x7fbbd40ce5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.749+0000 7fbbd2a00640 1 -- 172.21.15.45:0/3085065026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbd40afda0 msgr2=0x7fbbd4145fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.749+0000 7fbbd2a00640 1 --2- 172.21.15.45:0/3085065026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbd40afda0 0x7fbbd4145fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.749+0000 7fbbd2a00640 1 -- 172.21.15.45:0/3085065026 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd40cf650 con 0x7fbbd40d0e80 2024-04-16T16:20:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.750+0000 7fbbd2a00640 1 --2- 172.21.15.45:0/3085065026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd40d0e80 0x7fbbd4146500 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc00c400 tx=0x7fbbbc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.750+0000 7fbbcb400640 1 -- 172.21.15.45:0/3085065026 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc017070 con 0x7fbbd40d0e80 2024-04-16T16:20:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.750+0000 7fbbcb400640 1 -- 172.21.15.45:0/3085065026 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbbc0052d0 con 0x7fbbd40d0e80 2024-04-16T16:20:01.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.750+0000 7fbbcb400640 1 -- 172.21.15.45:0/3085065026 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc003830 con 0x7fbbd40d0e80 2024-04-16T16:20:01.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.750+0000 7fbbda400640 1 -- 172.21.15.45:0/3085065026 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbd40cf930 con 0x7fbbd40d0e80 2024-04-16T16:20:01.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.750+0000 7fbbda400640 1 -- 172.21.15.45:0/3085065026 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbd40cfef0 con 0x7fbbd40d0e80 2024-04-16T16:20:01.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.751+0000 7fbbda400640 1 -- 172.21.15.45:0/3085065026 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbd4075780 con 0x7fbbd40d0e80 2024-04-16T16:20:01.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.752+0000 7fbbcb400640 1 -- 172.21.15.45:0/3085065026 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 41) ==== 91142+0+0 (secure 0 0 0) 0x7fbbbc01a070 con 0x7fbbd40d0e80 2024-04-16T16:20:01.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.753+0000 7fbbcb400640 1 --2- 172.21.15.45:0/3085065026 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fbbb006ca40 0x7fbbb006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:01.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.753+0000 7fbbcb400640 1 -- 172.21.15.45:0/3085065026 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5752+0+0 (secure 0 0 0) 0x7fbbbc08b0e0 con 0x7fbbd40d0e80 2024-04-16T16:20:01.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.754+0000 7fbbd3400640 1 --2- 172.21.15.45:0/3085065026 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fbbb006ca40 0x7fbbb006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:01.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.754+0000 7fbbd3400640 1 --2- 172.21.15.45:0/3085065026 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fbbb006ca40 0x7fbbb006ef00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4001560 tx=0x7fbbc4009040 comp rx=0 tx=0).ready entity=mgr.5921 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:01.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:01.758+0000 7fbbcb400640 1 -- 172.21.15.45:0/3085065026 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbbbc0593d0 con 0x7fbbd40d0e80 2024-04-16T16:20:02.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:02.018+0000 7fbbda400640 1 -- 172.21.15.45:0/3085065026 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "stats"} v 0) -- 0x7fbbd40b5190 con 0x7fbbd40d0e80 2024-04-16T16:20:02.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:02.781+0000 7fbbcb400640 1 -- 172.21.15.45:0/3085065026 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "stats"}]=0 v42) ==== 84+0+0 (secure 0 0 0) 0x7fbbbc058cd0 con 0x7fbbd40d0e80 2024-04-16T16:20:02.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:02.785+0000 7fbbda400640 1 -- 172.21.15.45:0/3085065026 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fbbb006ca40 msgr2=0x7fbbb006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:02.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:02.785+0000 7fbbda400640 1 --2- 172.21.15.45:0/3085065026 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fbbb006ca40 0x7fbbb006ef00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4001560 tx=0x7fbbc4009040 comp rx=0 tx=0).stop 2024-04-16T16:20:02.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:02.785+0000 7fbbda400640 1 -- 172.21.15.45:0/3085065026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd40d0e80 msgr2=0x7fbbd4146500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:02.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:02.785+0000 7fbbda400640 1 --2- 172.21.15.45:0/3085065026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd40d0e80 0x7fbbd4146500 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc00c400 tx=0x7fbbbc00c8c0 comp rx=0 tx=0).stop 2024-04-16T16:20:02.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:02.786+0000 7fbbda400640 1 -- 172.21.15.45:0/3085065026 shutdown_connections 2024-04-16T16:20:02.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:02.786+0000 7fbbda400640 1 --2- 172.21.15.45:0/3085065026 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbd415b920 0x7fbbd40ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:02.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:02.786+0000 7fbbda400640 1 --2- 172.21.15.45:0/3085065026 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fbbb006ca40 0x7fbbb006ef00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:02.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:02.786+0000 7fbbda400640 1 --2- 172.21.15.45:0/3085065026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbd40d0e80 0x7fbbd4146500 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:02.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:02.786+0000 7fbbda400640 1 --2- 172.21.15.45:0/3085065026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbd40afda0 0x7fbbd4145fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:02.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:02.786+0000 7fbbda400640 1 -- 172.21.15.45:0/3085065026 >> 172.21.15.45:0/3085065026 conn(0x7fbbd40b3970 msgr2=0x7fbbd40b2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:02.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:02.787+0000 7fbbda400640 1 -- 172.21.15.45:0/3085065026 shutdown_connections 2024-04-16T16:20:02.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:02.787+0000 7fbbda400640 1 -- 172.21.15.45:0/3085065026 wait complete. 2024-04-16T16:20:02.815 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats --mds_rank 1, 2024-04-16T16:20:03.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.029+0000 7fce1ca00640 1 Processor -- start 2024-04-16T16:20:03.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.029+0000 7fce1ca00640 1 -- start start 2024-04-16T16:20:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.031+0000 7fce1ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce1809ac30 0x7fce1809b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.031+0000 7fce1ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fce1806d450 0x7fce1806d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.031+0000 7fce1ca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce180d0a00 con 0x7fce181607f0 2024-04-16T16:20:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.031+0000 7fce1ca00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fce180d15d0 con 0x7fce1806d450 2024-04-16T16:20:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.031+0000 7fce1ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fce1806df10 con 0x7fce1809ac30 2024-04-16T16:20:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.031+0000 7fce15400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fce1806d450 0x7fce1806d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.031+0000 7fce15400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fce1806d450 0x7fce1806d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57692/0 (socket says 172.21.15.45:57692) 2024-04-16T16:20:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.031+0000 7fce15400640 1 -- 172.21.15.45:0/400227045 learned_addr learned my addr 172.21.15.45:0/400227045 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:03.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.031+0000 7fce15e00640 1 --2- 172.21.15.45:0/400227045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce1809ac30 0x7fce1809b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:03.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.032+0000 7fce15400640 1 -- 172.21.15.45:0/400227045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce1809ac30 msgr2=0x7fce1809b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:03.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.032+0000 7fce15400640 1 --2- 172.21.15.45:0/400227045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce1809ac30 0x7fce1809b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:03.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.032+0000 7fce15400640 1 -- 172.21.15.45:0/400227045 >> v1:172.21.15.103:6789/0 conn(0x7fce181607f0 legacy=0x7fce180d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:03.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.032+0000 7fce15400640 1 -- 172.21.15.45:0/400227045 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce180d0ca0 con 0x7fce1806d450 2024-04-16T16:20:03.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.033+0000 7fce15400640 1 --2- 172.21.15.45:0/400227045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce1806d450 0x7fce1806d8b0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fce04009870 tx=0x7fce0402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483cc3d5f9c86e8e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:03.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.033+0000 7fce14a00640 1 -- 172.21.15.45:0/400227045 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04035020 con 0x7fce1806d450 2024-04-16T16:20:03.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.033+0000 7fce14a00640 1 -- 172.21.15.45:0/400227045 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce04031640 con 0x7fce1806d450 2024-04-16T16:20:03.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.033+0000 7fce14a00640 1 -- 172.21.15.45:0/400227045 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0402d040 con 0x7fce1806d450 2024-04-16T16:20:03.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.034+0000 7fce1ca00640 1 -- 172.21.15.45:0/400227045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce1806d450 msgr2=0x7fce1806d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:03.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.034+0000 7fce1ca00640 1 --2- 172.21.15.45:0/400227045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce1806d450 0x7fce1806d8b0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fce04009870 tx=0x7fce0402b8f0 comp rx=0 tx=0).stop 2024-04-16T16:20:03.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.034+0000 7fce1ca00640 1 -- 172.21.15.45:0/400227045 shutdown_connections 2024-04-16T16:20:03.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.034+0000 7fce1ca00640 1 --2- 172.21.15.45:0/400227045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce1806d450 0x7fce1806d8b0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:03.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.034+0000 7fce1ca00640 1 --2- 172.21.15.45:0/400227045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce1809ac30 0x7fce1809b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:03.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.035+0000 7fce1ca00640 1 -- 172.21.15.45:0/400227045 >> 172.21.15.45:0/400227045 conn(0x7fce180d2870 msgr2=0x7fce180d2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:03.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.035+0000 7fce1ca00640 1 -- 172.21.15.45:0/400227045 shutdown_connections 2024-04-16T16:20:03.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.035+0000 7fce1ca00640 1 -- 172.21.15.45:0/400227045 wait complete. 2024-04-16T16:20:03.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.036+0000 7fce1ca00640 1 Processor -- start 2024-04-16T16:20:03.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.036+0000 7fce1ca00640 1 -- start start 2024-04-16T16:20:03.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.037+0000 7fce1ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce1806d450 0x7fce180c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:03.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.037+0000 7fce1ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce1809ac30 0x7fce180c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:03.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.037+0000 7fce1ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce181607f0 0x7fce18156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:03.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.037+0000 7fce1ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fce1809b550 con 0x7fce1809ac30 2024-04-16T16:20:03.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.037+0000 7fce1ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fce180d1260 con 0x7fce181607f0 2024-04-16T16:20:03.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.037+0000 7fce1ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fce18160330 con 0x7fce1806d450 2024-04-16T16:20:03.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.038+0000 7fce15400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce1809ac30 0x7fce180c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:03.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.038+0000 7fce15400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce1809ac30 0x7fce180c63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57708/0 (socket says 172.21.15.45:57708) 2024-04-16T16:20:03.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.038+0000 7fce15400640 1 -- 172.21.15.45:0/46332171 learned_addr learned my addr 172.21.15.45:0/46332171 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:03.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.038+0000 7fce15e00640 1 --2- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce1806d450 0x7fce180c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:03.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.038+0000 7fce16800640 1 --2- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce181607f0 0x7fce18156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:03.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.038+0000 7fce15400640 1 -- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce1806d450 msgr2=0x7fce180c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:03.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.038+0000 7fce15400640 1 --2- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce1806d450 0x7fce180c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:03.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.038+0000 7fce15400640 1 -- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce181607f0 msgr2=0x7fce18156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:03.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.038+0000 7fce15400640 1 --2- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce181607f0 0x7fce18156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:03.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.038+0000 7fce15400640 1 -- 172.21.15.45:0/46332171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce180c6dd0 con 0x7fce1809ac30 2024-04-16T16:20:03.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.039+0000 7fce15400640 1 --2- 172.21.15.45:0/46332171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce1809ac30 0x7fce180c63a0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fce0402b930 tx=0x7fce040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:03.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.039+0000 7fce0e000640 1 -- 172.21.15.45:0/46332171 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04040070 con 0x7fce1809ac30 2024-04-16T16:20:03.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.039+0000 7fce0e000640 1 -- 172.21.15.45:0/46332171 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce040399f0 con 0x7fce1809ac30 2024-04-16T16:20:03.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.039+0000 7fce0e000640 1 -- 172.21.15.45:0/46332171 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04035040 con 0x7fce1809ac30 2024-04-16T16:20:03.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.039+0000 7fce1ca00640 1 -- 172.21.15.45:0/46332171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce180c70b0 con 0x7fce1809ac30 2024-04-16T16:20:03.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.039+0000 7fce1ca00640 1 -- 172.21.15.45:0/46332171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce180c7670 con 0x7fce1809ac30 2024-04-16T16:20:03.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.041+0000 7fce1ca00640 1 -- 172.21.15.45:0/46332171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce18075780 con 0x7fce1809ac30 2024-04-16T16:20:03.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.041+0000 7fce0e000640 1 -- 172.21.15.45:0/46332171 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 42) ==== 91151+0+0 (secure 0 0 0) 0x7fce0404b060 con 0x7fce1809ac30 2024-04-16T16:20:03.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.042+0000 7fce0e000640 1 --2- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fcdf806ca60 0x7fcdf806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:03.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.042+0000 7fce0e000640 1 -- 172.21.15.45:0/46332171 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5752+0+0 (secure 0 0 0) 0x7fce040ad6a0 con 0x7fce1809ac30 2024-04-16T16:20:03.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.043+0000 7fce15e00640 1 -- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fcdf806ca60 msgr2=0x7fcdf806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2293932214 2024-04-16T16:20:03.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.043+0000 7fce15e00640 1 --2- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fcdf806ca60 0x7fcdf806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:20:03.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.047+0000 7fce0e000640 1 -- 172.21.15.45:0/46332171 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fce0407b980 con 0x7fce1809ac30 2024-04-16T16:20:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.244+0000 7fce15e00640 1 -- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fcdf806ca60 msgr2=0x7fcdf806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2293932214 2024-04-16T16:20:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.244+0000 7fce15e00640 1 --2- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fcdf806ca60 0x7fcdf806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:20:03.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.280+0000 7fce1ca00640 1 -- 172.21.15.45:0/46332171 --> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] -- mgr_command(tid 0: {"prefix": "fs perf stats", "mds_rank": "1,", "target": ["mon-mgr", ""]}) -- 0x7fce1809b7f0 con 0x7fcdf806ca60 2024-04-16T16:20:03.329 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:03.329+0000 7f821e00e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:20:03.372 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:03.372+0000 7f617442c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:20:03.383 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:03.384+0000 7f821e00e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:20:03.432 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:03.432+0000 7f617442c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:20:03.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.645+0000 7fce15e00640 1 -- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fcdf806ca60 msgr2=0x7fcdf806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2293932214 2024-04-16T16:20:03.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:03.645+0000 7fce15e00640 1 --2- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fcdf806ca60 0x7fcdf806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T16:20:03.830 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:03.830+0000 7f821e00e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:20:03.882 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:03.882+0000 7f821e00e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:20:03.896 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:03.896+0000 7f617442c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:20:03.933 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:03.933+0000 7f821e00e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:20:03.950 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:03.950+0000 7f617442c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:20:03.994 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:03.995+0000 7f821e00e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:20:04.003 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:04.003+0000 7f617442c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:20:04.045 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:04.045+0000 7f821e00e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:20:04.067 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:04.067+0000 7f617442c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:20:04.119 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:04.119+0000 7f617442c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:20:04.176 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:04.176+0000 7f821e00e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:20:04.238 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:04.239+0000 7f821e00e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:20:04.254 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:04.254+0000 7f617442c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:20:04.289 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:04.289+0000 7f821e00e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:20:04.320 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:04.320+0000 7f617442c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:20:04.346 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:04.347+0000 7f821e00e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:20:04.372 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:04.372+0000 7f617442c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:20:04.397 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:04.398+0000 7f821e00e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:20:04.431 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:04.431+0000 7f617442c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:20:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:04.446+0000 7fce15e00640 1 -- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fcdf806ca60 msgr2=0x7fcdf806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2293932214 2024-04-16T16:20:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:04.446+0000 7fce15e00640 1 --2- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fcdf806ca60 0x7fcdf806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T16:20:04.484 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:04.484+0000 7f617442c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:20:04.781 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:04.781+0000 7f821e00e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:20:04.884 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:04.884+0000 7f617442c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:20:04.939 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:04.940+0000 7f821e00e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:20:05.050 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:05.050+0000 7f821e00e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:20:05.052 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:05.052+0000 7f617442c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:20:05.101 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:05.101+0000 7f821e00e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:20:05.165 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:05.165+0000 7f617442c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:20:05.218 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:05.218+0000 7f617442c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:20:05.226 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:05.227+0000 7f821e00e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:20:05.335 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:05.335+0000 7f821e00e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:20:05.351 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:05.351+0000 7f617442c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:20:05.464 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:05.464+0000 7f617442c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:20:05.509 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:05.509+0000 7f821e00e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:20:05.559 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:05.560+0000 7f821e00e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:20:05.647 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:05.647+0000 7f617442c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:20:05.698 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:05.699+0000 7f821e00e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:20:05.702 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:05.702+0000 7f617442c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:20:05.846 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:05.846+0000 7f617442c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:20:06.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:06.048+0000 7fce15e00640 1 -- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fcdf806ca60 msgr2=0x7fcdf806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/2293932214 2024-04-16T16:20:06.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:06.048+0000 7fce15e00640 1 --2- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fcdf806ca60 0x7fcdf806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T16:20:06.533 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:06.533+0000 7f821e00e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:20:06.726 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:06.726+0000 7f617442c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:20:06.843 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:06.844+0000 7f821e00e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:20:06.902 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:06.902+0000 7f821e00e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:20:07.034 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:07.035+0000 7f821e00e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:20:07.047 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:07.047+0000 7f617442c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:20:07.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:07.054+0000 7fce0e000640 1 -- 172.21.15.45:0/46332171 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 43) ==== 90760+0+0 (secure 0 0 0) 0x7fce04038020 con 0x7fce1809ac30 2024-04-16T16:20:07.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:07.055+0000 7fce0e000640 1 -- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fcdf806ca60 msgr2=0x7fcdf806ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:20:07.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:07.055+0000 7fce0e000640 1 --2- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/2293932214,v1:172.21.15.103:6833/2293932214] conn(0x7fcdf806ca60 0x7fcdf806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:07.115 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:07.115+0000 7f617442c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:20:07.127 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:07.127+0000 7f8198e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:07.127 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:07.127+0000 7f8198e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:07.127 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:07.127+0000 7f8198e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:07.127 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:07.127+0000 7f8198e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:07.127 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:07.127+0000 7f8198e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:07.127 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:07.127+0000 7f8195c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:07.127 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:07.127+0000 7f8195c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:07.127 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:07.127+0000 7f8195c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:07.127 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:07.127+0000 7f8195c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:07.128 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:07.127+0000 7f8195c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:07.257 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:07.257+0000 7f617442c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:20:08.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.064+0000 7fce0e000640 1 -- 172.21.15.45:0/46332171 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 44) ==== 90887+0+0 (secure 0 0 0) 0x7fce040778f0 con 0x7fce1809ac30 2024-04-16T16:20:08.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.065+0000 7fce0e000640 1 --2- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fcdf80749d0 0x7fcdf8076dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:08.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.066+0000 7fce0e000640 1 -- 172.21.15.45:0/46332171 --> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] -- mgr_command(tid 0: {"prefix": "fs perf stats", "mds_rank": "1,", "target": ["mon-mgr", ""]}) -- 0x7fce04081b80 con 0x7fcdf80749d0 2024-04-16T16:20:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.067+0000 7fce15e00640 1 --2- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fcdf80749d0 0x7fcdf8076dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:08.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.071+0000 7fce15e00640 1 --2- 172.21.15.45:0/46332171 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fcdf80749d0 0x7fcdf8076dc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fce08006f30 tx=0x7fce08008040 comp rx=0 tx=0).ready entity=mgr.6146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:08.083 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:08.084+0000 7f81a4c00640 -1 mgr.server reply reply (22) Invalid argument invalid mds filter spec: 1, 2024-04-16T16:20:08.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.084+0000 7fce0e000640 1 -- 172.21.15.45:0/46332171 <== mgr.6146 v2:172.21.15.103:6832/1934318167 1 ==== mgr_command_reply(tid 0: -22 invalid mds filter spec: 1,) ==== 35+0+0 (secure 0 0 0) 0x7fce04081b80 con 0x7fcdf80749d0 2024-04-16T16:20:08.085 INFO:teuthology.orchestra.run.smithi045.stderr:Error EINVAL: invalid mds filter spec: 1, 2024-04-16T16:20:08.101 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T16:20:08.101 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable stats 2024-04-16T16:20:08.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.329+0000 7efd6ec00640 1 Processor -- start 2024-04-16T16:20:08.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.329+0000 7efd6ec00640 1 -- start start 2024-04-16T16:20:08.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.330+0000 7efd6ec00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7efd680c55f0 0x7efd680b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:08.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.330+0000 7efd6ec00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd68163fb0 0x7efd680c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:08.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.330+0000 7efd6ec00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd680c5f10 con 0x7efd680ccc70 2024-04-16T16:20:08.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.331+0000 7efd6ec00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7efd680b2f10 con 0x7efd680c55f0 2024-04-16T16:20:08.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.331+0000 7efd6ec00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efd681644e0 con 0x7efd68163fb0 2024-04-16T16:20:08.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.331+0000 7efd6e200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7efd680ccc70 0x7efd6809b030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49062/0 (socket says 172.21.15.45:49062) 2024-04-16T16:20:08.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.331+0000 7efd6e200640 1 -- 172.21.15.45:0/642679289 learned_addr learned my addr 172.21.15.45:0/642679289 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:08.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.331+0000 7efd6d800640 1 --2- 172.21.15.45:0/642679289 >> v2:172.21.15.45:3300/0 conn(0x7efd680c55f0 0x7efd680b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:08.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.331+0000 7efd6ce00640 1 --2- 172.21.15.45:0/642679289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd68163fb0 0x7efd680c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:08.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.331+0000 7efd67e00640 1 -- 172.21.15.45:0/642679289 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3175638269 0 0) 0x7efd680c5f10 con 0x7efd680ccc70 2024-04-16T16:20:08.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.332+0000 7efd6d800640 1 -- 172.21.15.45:0/642679289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd68163fb0 msgr2=0x7efd680c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:08.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.332+0000 7efd6d800640 1 --2- 172.21.15.45:0/642679289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd68163fb0 0x7efd680c59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:08.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.332+0000 7efd6d800640 1 -- 172.21.15.45:0/642679289 >> v1:172.21.15.45:6789/0 conn(0x7efd680ccc70 legacy=0x7efd6809b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:08.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.332+0000 7efd6d800640 1 -- 172.21.15.45:0/642679289 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6806cb70 con 0x7efd680c55f0 2024-04-16T16:20:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.333+0000 7efd6d800640 1 --2- 172.21.15.45:0/642679289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd680c55f0 0x7efd680b35c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7efd5c009d90 tx=0x7efd5c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784a1b0b31447775 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.333+0000 7efd67e00640 1 -- 172.21.15.45:0/642679289 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c039070 con 0x7efd680c55f0 2024-04-16T16:20:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.333+0000 7efd67e00640 1 -- 172.21.15.45:0/642679289 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd5c02cbc0 con 0x7efd680c55f0 2024-04-16T16:20:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.333+0000 7efd67e00640 1 -- 172.21.15.45:0/642679289 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c034440 con 0x7efd680c55f0 2024-04-16T16:20:08.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.334+0000 7efd6ec00640 1 -- 172.21.15.45:0/642679289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd680c55f0 msgr2=0x7efd680b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:08.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.334+0000 7efd6ec00640 1 --2- 172.21.15.45:0/642679289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd680c55f0 0x7efd680b35c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7efd5c009d90 tx=0x7efd5c02be10 comp rx=0 tx=0).stop 2024-04-16T16:20:08.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.334+0000 7efd6ec00640 1 -- 172.21.15.45:0/642679289 shutdown_connections 2024-04-16T16:20:08.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.334+0000 7efd6ec00640 1 --2- 172.21.15.45:0/642679289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd68163fb0 0x7efd680c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:08.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.334+0000 7efd6ec00640 1 --2- 172.21.15.45:0/642679289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd680c55f0 0x7efd680b35c0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:08.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.334+0000 7efd6ec00640 1 -- 172.21.15.45:0/642679289 >> 172.21.15.45:0/642679289 conn(0x7efd680b4110 msgr2=0x7efd680b4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:08.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.334+0000 7efd6ec00640 1 -- 172.21.15.45:0/642679289 shutdown_connections 2024-04-16T16:20:08.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.334+0000 7efd6ec00640 1 -- 172.21.15.45:0/642679289 wait complete. 2024-04-16T16:20:08.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.336+0000 7efd6ec00640 1 Processor -- start 2024-04-16T16:20:08.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.336+0000 7efd6ec00640 1 -- start start 2024-04-16T16:20:08.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.337+0000 7efd6ec00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd680c55f0 0x7efd681fccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:08.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.337+0000 7efd6ec00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd680ccc70 0x7efd681f8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:08.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.337+0000 7efd6ec00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd68163fb0 0x7efd680cfef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:08.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.337+0000 7efd6ec00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efd680b3e40 con 0x7efd680c55f0 2024-04-16T16:20:08.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.337+0000 7efd6ec00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efd6809ab10 con 0x7efd680ccc70 2024-04-16T16:20:08.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.337+0000 7efd6ec00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efd680cd100 con 0x7efd68163fb0 2024-04-16T16:20:08.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.337+0000 7efd6d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd680c55f0 0x7efd681fccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:08.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.337+0000 7efd6d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd680c55f0 0x7efd681fccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60090/0 (socket says 172.21.15.45:60090) 2024-04-16T16:20:08.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.337+0000 7efd6d800640 1 -- 172.21.15.45:0/953553528 learned_addr learned my addr 172.21.15.45:0/953553528 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:08.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.338+0000 7efd6d800640 1 -- 172.21.15.45:0/953553528 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd68163fb0 msgr2=0x7efd680cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:08.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.338+0000 7efd6d800640 1 --2- 172.21.15.45:0/953553528 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd68163fb0 0x7efd680cfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:08.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.338+0000 7efd6d800640 1 -- 172.21.15.45:0/953553528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd680ccc70 msgr2=0x7efd681f8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:08.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.338+0000 7efd6d800640 1 --2- 172.21.15.45:0/953553528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd680ccc70 0x7efd681f8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:08.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.338+0000 7efd6d800640 1 -- 172.21.15.45:0/953553528 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd681f9500 con 0x7efd680c55f0 2024-04-16T16:20:08.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.338+0000 7efd6d800640 1 --2- 172.21.15.45:0/953553528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd680c55f0 0x7efd681fccc0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7efd5c037040 tx=0x7efd5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.338+0000 7efd65600640 1 -- 172.21.15.45:0/953553528 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c039050 con 0x7efd680c55f0 2024-04-16T16:20:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.339+0000 7efd65600640 1 -- 172.21.15.45:0/953553528 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd5c045070 con 0x7efd680c55f0 2024-04-16T16:20:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.339+0000 7efd65600640 1 -- 172.21.15.45:0/953553528 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c03e850 con 0x7efd680c55f0 2024-04-16T16:20:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.339+0000 7efd6ec00640 1 -- 172.21.15.45:0/953553528 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd681f97e0 con 0x7efd680c55f0 2024-04-16T16:20:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.339+0000 7efd6ec00640 1 -- 172.21.15.45:0/953553528 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd680cb880 con 0x7efd680c55f0 2024-04-16T16:20:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.340+0000 7efd6ec00640 1 -- 172.21.15.45:0/953553528 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd68075780 con 0x7efd680c55f0 2024-04-16T16:20:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.341+0000 7efd65600640 1 -- 172.21.15.45:0/953553528 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 44) ==== 90887+0+0 (secure 0 0 0) 0x7efd5c03e9f0 con 0x7efd680c55f0 2024-04-16T16:20:08.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.342+0000 7efd65600640 1 --2- 172.21.15.45:0/953553528 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7efd4806c790 0x7efd4806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:08.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.342+0000 7efd65600640 1 -- 172.21.15.45:0/953553528 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 6096+0+0 (secure 0 0 0) 0x7efd5c0ae650 con 0x7efd680c55f0 2024-04-16T16:20:08.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.343+0000 7efd6ce00640 1 --2- 172.21.15.45:0/953553528 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7efd4806c790 0x7efd4806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:08.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.344+0000 7efd6ce00640 1 --2- 172.21.15.45:0/953553528 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7efd4806c790 0x7efd4806ec50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efd68070bb0 tx=0x7efd54008040 comp rx=0 tx=0).ready entity=mgr.6146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:08.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.347+0000 7efd65600640 1 -- 172.21.15.45:0/953553528 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efd5c07cc40 con 0x7efd680c55f0 2024-04-16T16:20:08.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:08.606+0000 7efd6ec00640 1 -- 172.21.15.45:0/953553528 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "stats"} v 0) -- 0x7efd680b3190 con 0x7efd680c55f0 2024-04-16T16:20:09.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.114+0000 7efd65600640 1 -- 172.21.15.45:0/953553528 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "stats"}]=0 v45) ==== 85+0+0 (secure 0 0 0) 0x7efd5c0ace20 con 0x7efd680c55f0 2024-04-16T16:20:09.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.119+0000 7efd6ec00640 1 -- 172.21.15.45:0/953553528 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7efd4806c790 msgr2=0x7efd4806ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:09.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.119+0000 7efd6ec00640 1 --2- 172.21.15.45:0/953553528 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7efd4806c790 0x7efd4806ec50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efd68070bb0 tx=0x7efd54008040 comp rx=0 tx=0).stop 2024-04-16T16:20:09.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.119+0000 7efd6ec00640 1 -- 172.21.15.45:0/953553528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd680c55f0 msgr2=0x7efd681fccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:09.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.119+0000 7efd6ec00640 1 --2- 172.21.15.45:0/953553528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd680c55f0 0x7efd681fccc0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7efd5c037040 tx=0x7efd5c004a40 comp rx=0 tx=0).stop 2024-04-16T16:20:09.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.119+0000 7efd6ec00640 1 -- 172.21.15.45:0/953553528 shutdown_connections 2024-04-16T16:20:09.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.119+0000 7efd6ec00640 1 --2- 172.21.15.45:0/953553528 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7efd4806c790 0x7efd4806ec50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.119+0000 7efd6ec00640 1 --2- 172.21.15.45:0/953553528 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd68163fb0 0x7efd680cfef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.119+0000 7efd6ec00640 1 --2- 172.21.15.45:0/953553528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd680ccc70 0x7efd681f8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.119+0000 7efd6ec00640 1 --2- 172.21.15.45:0/953553528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd680c55f0 0x7efd681fccc0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.119+0000 7efd6ec00640 1 -- 172.21.15.45:0/953553528 >> 172.21.15.45:0/953553528 conn(0x7efd680b4110 msgr2=0x7efd681648b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:09.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.120+0000 7efd6ec00640 1 -- 172.21.15.45:0/953553528 shutdown_connections 2024-04-16T16:20:09.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.120+0000 7efd6ec00640 1 -- 172.21.15.45:0/953553528 wait complete. 2024-04-16T16:20:09.144 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-16T16:20:09.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:20:09.250 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:20:09.359 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-16T16:20:09.359 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:20:09.359 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:20:09.359 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:20:09.410 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:20:09.410 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:20:09.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:20:09.690 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:09.691+0000 7fb78902b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:20:09.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.706+0000 7fc239200640 1 Processor -- start 2024-04-16T16:20:09.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.706+0000 7fc239200640 1 -- start start 2024-04-16T16:20:09.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.708+0000 7fc239200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc234142ff0 0x7fc23406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:09.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.708+0000 7fc239200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340716b0 0x7fc23415d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:09.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.708+0000 7fc239200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2341433d0 con 0x7fc23406c0a0 2024-04-16T16:20:09.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.708+0000 7fc239200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc2341605e0 con 0x7fc234142ff0 2024-04-16T16:20:09.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.708+0000 7fc239200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc2341608c0 con 0x7fc2340716b0 2024-04-16T16:20:09.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.708+0000 7fc232a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc23406c0a0 0x7fc234070fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49072/0 (socket says 172.21.15.45:49072) 2024-04-16T16:20:09.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.708+0000 7fc232a00640 1 -- 172.21.15.45:0/2265287502 learned_addr learned my addr 172.21.15.45:0/2265287502 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.708+0000 7fc233400640 1 --2- 172.21.15.45:0/2265287502 >> v2:172.21.15.103:3300/0 conn(0x7fc234142ff0 0x7fc23406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.708+0000 7fc233e00640 1 --2- 172.21.15.45:0/2265287502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340716b0 0x7fc23415d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 -- 172.21.15.45:0/2265287502 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2833102583 0 0) 0x7fc2341433d0 con 0x7fc23406c0a0 2024-04-16T16:20:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 -- 172.21.15.45:0/2265287502 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc218003520 con 0x7fc23406c0a0 2024-04-16T16:20:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 -- 172.21.15.45:0/2265287502 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3806392079 0 0) 0x7fc218003520 con 0x7fc23406c0a0 2024-04-16T16:20:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 -- 172.21.15.45:0/2265287502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340716b0 msgr2=0x7fc23415d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 --2- 172.21.15.45:0/2265287502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340716b0 0x7fc23415d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 -- 172.21.15.45:0/2265287502 >> v2:172.21.15.103:3300/0 conn(0x7fc234142ff0 msgr2=0x7fc23406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 --2- 172.21.15.45:0/2265287502 >> v2:172.21.15.103:3300/0 conn(0x7fc234142ff0 0x7fc23406bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 -- 172.21.15.45:0/2265287502 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc234160ba0 con 0x7fc23406c0a0 2024-04-16T16:20:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 -- 172.21.15.45:0/2265287502 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fc220001690 con 0x7fc23406c0a0 2024-04-16T16:20:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 -- 172.21.15.45:0/2265287502 >> v1:172.21.15.45:6789/0 conn(0x7fc23406c0a0 legacy=0x7fc234070fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 --2- 172.21.15.45:0/2265287502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340716b0 0x7fc2180039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 --2- 172.21.15.45:0/2265287502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc218003f30 0x7fc218004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 --2- 172.21.15.45:0/2265287502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc218004d40 0x7fc2180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 -- 172.21.15.45:0/2265287502 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc2341605e0 con 0x7fc218003f30 2024-04-16T16:20:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 -- 172.21.15.45:0/2265287502 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc2341608c0 con 0x7fc218004d40 2024-04-16T16:20:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.709+0000 7fc232000640 1 -- 172.21.15.45:0/2265287502 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc21800b9c0 con 0x7fc2340716b0 2024-04-16T16:20:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.710+0000 7fc232a00640 1 --2- 172.21.15.45:0/2265287502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc218003f30 0x7fc218004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:09.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.710+0000 7fc233400640 1 --2- 172.21.15.45:0/2265287502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340716b0 0x7fc2180039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:09.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.710+0000 7fc233e00640 1 --2- 172.21.15.45:0/2265287502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc218004d40 0x7fc2180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:09.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.710+0000 7fc232a00640 1 -- 172.21.15.45:0/2265287502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340716b0 msgr2=0x7fc2180039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:09.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.710+0000 7fc232a00640 1 --2- 172.21.15.45:0/2265287502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340716b0 0x7fc2180039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.710+0000 7fc232a00640 1 -- 172.21.15.45:0/2265287502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc218004d40 msgr2=0x7fc2180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:09.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.710+0000 7fc232a00640 1 --2- 172.21.15.45:0/2265287502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc218004d40 0x7fc2180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.710+0000 7fc232a00640 1 -- 172.21.15.45:0/2265287502 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc218003520 con 0x7fc218003f30 2024-04-16T16:20:09.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.711+0000 7fc232a00640 1 --2- 172.21.15.45:0/2265287502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc218003f30 0x7fc218004800 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fc22000c210 tx=0x7fc218005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f21ef972d69dbebc server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:09.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.712+0000 7fc232000640 1 -- 172.21.15.45:0/2265287502 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc220007d60 con 0x7fc218003f30 2024-04-16T16:20:09.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.712+0000 7fc232000640 1 -- 172.21.15.45:0/2265287502 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc220030070 con 0x7fc218003f30 2024-04-16T16:20:09.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.712+0000 7fc232000640 1 -- 172.21.15.45:0/2265287502 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc220005d10 con 0x7fc218003f30 2024-04-16T16:20:09.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.712+0000 7fc239200640 1 -- 172.21.15.45:0/2265287502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc218003f30 msgr2=0x7fc218004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:09.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.712+0000 7fc239200640 1 --2- 172.21.15.45:0/2265287502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc218003f30 0x7fc218004800 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fc22000c210 tx=0x7fc218005830 comp rx=0 tx=0).stop 2024-04-16T16:20:09.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.712+0000 7fc233e00640 1 -- 172.21.15.45:0/2265287502 reap_dead start 2024-04-16T16:20:09.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.713+0000 7fc239200640 1 -- 172.21.15.45:0/2265287502 shutdown_connections 2024-04-16T16:20:09.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.713+0000 7fc239200640 1 -- 172.21.15.45:0/2265287502 >> 172.21.15.45:0/2265287502 conn(0x7fc23406f0e0 msgr2=0x7fc23406f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:09.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.713+0000 7fc239200640 1 -- 172.21.15.45:0/2265287502 shutdown_connections 2024-04-16T16:20:09.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.713+0000 7fc239200640 1 -- 172.21.15.45:0/2265287502 wait complete. 2024-04-16T16:20:09.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.715+0000 7fc239200640 1 Processor -- start 2024-04-16T16:20:09.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.715+0000 7fc239200640 1 -- start start 2024-04-16T16:20:09.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.716+0000 7fc239200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc218003f30 0x7fc234138420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:09.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.716+0000 7fc239200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc23413a9d0 0x7fc234138960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:09.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.716+0000 7fc239200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc234138ea0 0x7fc234139320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:09.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.716+0000 7fc239200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc234161440 con 0x7fc218003f30 2024-04-16T16:20:09.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.716+0000 7fc239200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc234161160 con 0x7fc23413a9d0 2024-04-16T16:20:09.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.716+0000 7fc239200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc23415d340 con 0x7fc234138ea0 2024-04-16T16:20:09.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.716+0000 7fc233400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc218003f30 0x7fc234138420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:09.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.716+0000 7fc233400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc218003f30 0x7fc234138420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60112/0 (socket says 172.21.15.45:60112) 2024-04-16T16:20:09.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.716+0000 7fc233400640 1 -- 172.21.15.45:0/2833216597 learned_addr learned my addr 172.21.15.45:0/2833216597 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:09.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.717+0000 7fc232a00640 1 --2- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc23413a9d0 0x7fc234138960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:09.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.717+0000 7fc233400640 1 -- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc234138ea0 msgr2=0x7fc234139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:09.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.717+0000 7fc233e00640 1 --2- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc234138ea0 0x7fc234139320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:09.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.717+0000 7fc233400640 1 --2- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc234138ea0 0x7fc234139320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.717+0000 7fc233400640 1 -- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc23413a9d0 msgr2=0x7fc234138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:09.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.717+0000 7fc233400640 1 --2- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc23413a9d0 0x7fc234138960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.717+0000 7fc233400640 1 -- 172.21.15.45:0/2833216597 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23415bef0 con 0x7fc218003f30 2024-04-16T16:20:09.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.717+0000 7fc233e00640 1 --2- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc234138ea0 0x7fc234139320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:09.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.717+0000 7fc233400640 1 --2- 172.21.15.45:0/2833216597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc218003f30 0x7fc234138420 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fc224011630 tx=0x7fc224011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:09.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.717+0000 7fc22b400640 1 -- 172.21.15.45:0/2833216597 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22400f8c0 con 0x7fc218003f30 2024-04-16T16:20:09.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.717+0000 7fc22b400640 1 -- 172.21.15.45:0/2833216597 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc22400fa60 con 0x7fc218003f30 2024-04-16T16:20:09.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.717+0000 7fc239200640 1 -- 172.21.15.45:0/2833216597 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2340afdb0 con 0x7fc218003f30 2024-04-16T16:20:09.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.717+0000 7fc22b400640 1 -- 172.21.15.45:0/2833216597 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22400fe10 con 0x7fc218003f30 2024-04-16T16:20:09.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.718+0000 7fc239200640 1 -- 172.21.15.45:0/2833216597 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2340b0370 con 0x7fc218003f30 2024-04-16T16:20:09.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.719+0000 7fc239200640 1 -- 172.21.15.45:0/2833216597 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc234075780 con 0x7fc218003f30 2024-04-16T16:20:09.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.721+0000 7fc22b400640 1 -- 172.21.15.45:0/2833216597 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) ==== 91086+0+0 (secure 0 0 0) 0x7fc224015460 con 0x7fc218003f30 2024-04-16T16:20:09.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.722+0000 7fc22b400640 1 --2- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fc21406c980 0x7fc21406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:09.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.722+0000 7fc22b400640 1 -- 172.21.15.45:0/2833216597 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 6096+0+0 (secure 0 0 0) 0x7fc22408c610 con 0x7fc218003f30 2024-04-16T16:20:09.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.722+0000 7fc232a00640 1 -- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fc21406c980 msgr2=0x7fc21406ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:09.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.722+0000 7fc232a00640 1 --2- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fc21406c980 0x7fc21406ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:20:09.726 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:09.726+0000 7fddf422d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:20:09.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.727+0000 7fc22b400640 1 -- 172.21.15.45:0/2833216597 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc224019030 con 0x7fc218003f30 2024-04-16T16:20:09.752 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:09.752+0000 7fb78902b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:20:09.789 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:09.789+0000 7fddf422d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:20:09.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.923+0000 7fc232a00640 1 -- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fc21406c980 msgr2=0x7fc21406ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:09.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.923+0000 7fc232a00640 1 --2- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fc21406c980 0x7fc21406ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:20:09.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.931+0000 7fc239200640 1 -- 172.21.15.45:0/2833216597 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2340ce9e0 con 0x7fc218003f30 2024-04-16T16:20:09.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.933+0000 7fc22b400640 1 -- 172.21.15.45:0/2833216597 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+17197 (secure 0 0 0) 0x7fc22405a980 con 0x7fc218003f30 2024-04-16T16:20:09.934 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:09.934 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":116,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:20:07.040908+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:48.114834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:49.135741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3506765807":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/3547880684":"2024-04-17T16:20:07.040813+0000","172.21.15.103:6833/2293932214":"2024-04-17T16:20:07.040813+0000","172.21.15.103:6832/2293932214":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/1347839066":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/546409297":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/3566291251":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/714185246":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/2784388736":"2024-04-16T17:19:54.469649+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:20:09.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.936+0000 7fc239200640 1 -- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fc21406c980 msgr2=0x7fc21406ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:20:09.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.936+0000 7fc239200640 1 --2- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fc21406c980 0x7fc21406ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.936+0000 7fc239200640 1 -- 172.21.15.45:0/2833216597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc218003f30 msgr2=0x7fc234138420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:09.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.936+0000 7fc239200640 1 --2- 172.21.15.45:0/2833216597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc218003f30 0x7fc234138420 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fc224011630 tx=0x7fc224011af0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.936+0000 7fc239200640 1 -- 172.21.15.45:0/2833216597 shutdown_connections 2024-04-16T16:20:09.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.936+0000 7fc239200640 1 --2- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fc21406c980 0x7fc21406ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.936+0000 7fc239200640 1 --2- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc234138ea0 0x7fc234139320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.936+0000 7fc239200640 1 --2- 172.21.15.45:0/2833216597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc23413a9d0 0x7fc234138960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.936+0000 7fc239200640 1 --2- 172.21.15.45:0/2833216597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc218003f30 0x7fc234138420 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:09.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.936+0000 7fc239200640 1 -- 172.21.15.45:0/2833216597 >> 172.21.15.45:0/2833216597 conn(0x7fc23406f0e0 msgr2=0x7fc23415f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:09.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.937+0000 7fc239200640 1 -- 172.21.15.45:0/2833216597 shutdown_connections 2024-04-16T16:20:09.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:09.937+0000 7fc239200640 1 -- 172.21.15.45:0/2833216597 wait complete. 2024-04-16T16:20:09.955 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:19:48.114834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:19:49.135741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:20:09.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:20:10.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.155+0000 7f31b1e00640 1 Processor -- start 2024-04-16T16:20:10.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.156+0000 7f31b1e00640 1 -- start start 2024-04-16T16:20:10.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.157+0000 7f31b1e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f31ac146170 0x7f31ac15cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:10.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.157+0000 7f31b1e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f31ac0afc10 0x7f31ac06b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:10.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.157+0000 7f31b1e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31ac06be70 0x7f31ac06c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:10.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.157+0000 7f31b1e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f31ac146550 con 0x7f31ac146170 2024-04-16T16:20:10.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.157+0000 7f31b1e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f31ac15e1e0 con 0x7f31ac0afc10 2024-04-16T16:20:10.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.157+0000 7f31b1e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f31ac15e4c0 con 0x7f31ac06be70 2024-04-16T16:20:10.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.157+0000 7f31b0a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f31ac146170 0x7f31ac15cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:10.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.158+0000 7f31b0a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f31ac146170 0x7f31ac15cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60126/0 (socket says 172.21.15.45:60126) 2024-04-16T16:20:10.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.158+0000 7f31b0a00640 1 -- 172.21.15.45:0/537766621 learned_addr learned my addr 172.21.15.45:0/537766621 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:10.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.158+0000 7f31b1400640 1 --2- 172.21.15.45:0/537766621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31ac06be70 0x7f31ac06c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:10.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.158+0000 7f31abe00640 1 --2- 172.21.15.45:0/537766621 >> v2:172.21.15.103:3300/0 conn(0x7f31ac0afc10 0x7f31ac06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:10.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.158+0000 7f31b0a00640 1 -- 172.21.15.45:0/537766621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31ac06be70 msgr2=0x7f31ac06c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:10.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.158+0000 7f31b0a00640 1 --2- 172.21.15.45:0/537766621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31ac06be70 0x7f31ac06c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.158+0000 7f31b0a00640 1 -- 172.21.15.45:0/537766621 >> v2:172.21.15.103:3300/0 conn(0x7f31ac0afc10 msgr2=0x7f31ac06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:10.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.158+0000 7f31b0a00640 1 --2- 172.21.15.45:0/537766621 >> v2:172.21.15.103:3300/0 conn(0x7f31ac0afc10 0x7f31ac06b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.158+0000 7f31b0a00640 1 -- 172.21.15.45:0/537766621 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31ac15e7a0 con 0x7f31ac146170 2024-04-16T16:20:10.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.158+0000 7f31b1400640 1 --2- 172.21.15.45:0/537766621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31ac06be70 0x7f31ac06c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:10.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.158+0000 7f31abe00640 1 --2- 172.21.15.45:0/537766621 >> v2:172.21.15.103:3300/0 conn(0x7f31ac0afc10 0x7f31ac06b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:10.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.159+0000 7f31b0a00640 1 --2- 172.21.15.45:0/537766621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac146170 0x7f31ac15cd40 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0d2640 tx=0x7f31a002b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88025d7906864708 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:10.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.160+0000 7f31ab400640 1 -- 172.21.15.45:0/537766621 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a0035020 con 0x7f31ac146170 2024-04-16T16:20:10.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.160+0000 7f31ab400640 1 -- 172.21.15.45:0/537766621 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31a0031880 con 0x7f31ac146170 2024-04-16T16:20:10.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.160+0000 7f31ab400640 1 -- 172.21.15.45:0/537766621 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a0031c50 con 0x7f31ac146170 2024-04-16T16:20:10.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.160+0000 7f31b1e00640 1 -- 172.21.15.45:0/537766621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac146170 msgr2=0x7f31ac15cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:10.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.160+0000 7f31b1e00640 1 --2- 172.21.15.45:0/537766621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac146170 0x7f31ac15cd40 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0d2640 tx=0x7f31a002b920 comp rx=0 tx=0).stop 2024-04-16T16:20:10.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.161+0000 7f31b1e00640 1 -- 172.21.15.45:0/537766621 shutdown_connections 2024-04-16T16:20:10.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.161+0000 7f31b1e00640 1 --2- 172.21.15.45:0/537766621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31ac06be70 0x7f31ac06c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.161+0000 7f31b1e00640 1 --2- 172.21.15.45:0/537766621 >> v2:172.21.15.103:3300/0 conn(0x7f31ac0afc10 0x7f31ac06b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.161+0000 7f31b1e00640 1 --2- 172.21.15.45:0/537766621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac146170 0x7f31ac15cd40 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.161+0000 7f31b1e00640 1 -- 172.21.15.45:0/537766621 >> 172.21.15.45:0/537766621 conn(0x7f31ac0735e0 msgr2=0x7f31ac09aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:10.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.161+0000 7f31b1e00640 1 -- 172.21.15.45:0/537766621 shutdown_connections 2024-04-16T16:20:10.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.161+0000 7f31b1e00640 1 -- 172.21.15.45:0/537766621 wait complete. 2024-04-16T16:20:10.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.163+0000 7f31b1e00640 1 Processor -- start 2024-04-16T16:20:10.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.163+0000 7f31b1e00640 1 -- start start 2024-04-16T16:20:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.165+0000 7f31b1e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31ac06be70 0x7f31ac140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.165+0000 7f31b1e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31ac0afc10 0x7f31ac142bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.165+0000 7f31b1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac1430f0 0x7f31ac0b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.165+0000 7f31b1e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f31ac160370 con 0x7f31ac1430f0 2024-04-16T16:20:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.165+0000 7f31b1e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f31ac15de50 con 0x7f31ac06be70 2024-04-16T16:20:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.165+0000 7f31b1e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f31ac1604f0 con 0x7f31ac0afc10 2024-04-16T16:20:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.165+0000 7f31b1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac1430f0 0x7f31ac0b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.165+0000 7f31b1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac1430f0 0x7f31ac0b24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60132/0 (socket says 172.21.15.45:60132) 2024-04-16T16:20:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.165+0000 7f31b1400640 1 -- 172.21.15.45:0/3467715317 learned_addr learned my addr 172.21.15.45:0/3467715317 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.165+0000 7f31abe00640 1 --2- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31ac0afc10 0x7f31ac142bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.165+0000 7f31b0a00640 1 --2- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31ac06be70 0x7f31ac140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.166+0000 7f31b1400640 1 -- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31ac0afc10 msgr2=0x7f31ac142bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.166+0000 7f31b1400640 1 --2- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31ac0afc10 0x7f31ac142bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.166+0000 7f31b1400640 1 -- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31ac06be70 msgr2=0x7f31ac140cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.166+0000 7f31b1400640 1 --2- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31ac06be70 0x7f31ac140cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.166+0000 7f31b1400640 1 -- 172.21.15.45:0/3467715317 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31ac09b7e0 con 0x7f31ac1430f0 2024-04-16T16:20:10.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.166+0000 7f31b1400640 1 --2- 172.21.15.45:0/3467715317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac1430f0 0x7f31ac0b24f0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f31a400cf20 tx=0x7f31a400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:10.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.166+0000 7f31a8c00640 1 -- 172.21.15.45:0/3467715317 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a401a070 con 0x7f31ac1430f0 2024-04-16T16:20:10.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.166+0000 7f31a8c00640 1 -- 172.21.15.45:0/3467715317 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31a400d840 con 0x7f31ac1430f0 2024-04-16T16:20:10.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.166+0000 7f31a8c00640 1 -- 172.21.15.45:0/3467715317 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a400dc30 con 0x7f31ac1430f0 2024-04-16T16:20:10.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.166+0000 7f31b1e00640 1 -- 172.21.15.45:0/3467715317 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31ac09bac0 con 0x7f31ac1430f0 2024-04-16T16:20:10.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.166+0000 7f31b1e00640 1 -- 172.21.15.45:0/3467715317 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31ac09c080 con 0x7f31ac1430f0 2024-04-16T16:20:10.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.168+0000 7f31b1e00640 1 -- 172.21.15.45:0/3467715317 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31ac06d9d0 con 0x7f31ac1430f0 2024-04-16T16:20:10.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.168+0000 7f31a8c00640 1 -- 172.21.15.45:0/3467715317 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) ==== 91086+0+0 (secure 0 0 0) 0x7f31a401d070 con 0x7f31ac1430f0 2024-04-16T16:20:10.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.169+0000 7f31a8c00640 1 --2- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f318c06c9b0 0x7f318c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:10.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.169+0000 7f31a8c00640 1 -- 172.21.15.45:0/3467715317 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 6096+0+0 (secure 0 0 0) 0x7f31a408c540 con 0x7f31ac1430f0 2024-04-16T16:20:10.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.170+0000 7f31b0a00640 1 -- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f318c06c9b0 msgr2=0x7f318c06ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:10.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.170+0000 7f31b0a00640 1 --2- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f318c06c9b0 0x7f318c06ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:20:10.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.174+0000 7f31a8c00640 1 -- 172.21.15.45:0/3467715317 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f31a405a7b0 con 0x7f31ac1430f0 2024-04-16T16:20:10.204 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:10.204+0000 7fb78902b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:20:10.256 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:10.257+0000 7fb78902b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:20:10.269 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:10.269+0000 7fddf422d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:20:10.307 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:10.308+0000 7fb78902b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:20:10.324 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:10.324+0000 7fddf422d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:20:10.369 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:10.370+0000 7fb78902b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:20:10.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.371+0000 7f31b0a00640 1 -- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f318c06c9b0 msgr2=0x7f318c06ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:10.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.371+0000 7f31b0a00640 1 --2- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f318c06c9b0 0x7f318c06ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:20:10.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.376+0000 7f31b1e00640 1 -- 172.21.15.45:0/3467715317 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f31ac1416b0 con 0x7f31ac1430f0 2024-04-16T16:20:10.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.377+0000 7f31a8c00640 1 -- 172.21.15.45:0/3467715317 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) ==== 93+0+10 (secure 0 0 0) 0x7f31a405a0b0 con 0x7f31ac1430f0 2024-04-16T16:20:10.376 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:20:10.379 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:10.379+0000 7fddf422d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:20:10.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.379+0000 7f31b1e00640 1 -- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f318c06c9b0 msgr2=0x7f318c06ee70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:20:10.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.379+0000 7f31b1e00640 1 --2- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f318c06c9b0 0x7f318c06ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.379+0000 7f31b1e00640 1 -- 172.21.15.45:0/3467715317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac1430f0 msgr2=0x7f31ac0b24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:10.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.379+0000 7f31b1e00640 1 --2- 172.21.15.45:0/3467715317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac1430f0 0x7f31ac0b24f0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f31a400cf20 tx=0x7f31a400a710 comp rx=0 tx=0).stop 2024-04-16T16:20:10.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.379+0000 7f31b1e00640 1 -- 172.21.15.45:0/3467715317 shutdown_connections 2024-04-16T16:20:10.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.379+0000 7f31b1e00640 1 --2- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f318c06c9b0 0x7f318c06ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.379+0000 7f31b1e00640 1 --2- 172.21.15.45:0/3467715317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac1430f0 0x7f31ac0b24f0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.379+0000 7f31b1e00640 1 --2- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31ac0afc10 0x7f31ac142bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.379+0000 7f31b1e00640 1 --2- 172.21.15.45:0/3467715317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31ac06be70 0x7f31ac140cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.379+0000 7f31b1e00640 1 -- 172.21.15.45:0/3467715317 >> 172.21.15.45:0/3467715317 conn(0x7f31ac0735e0 msgr2=0x7f31ac143d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:10.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.379+0000 7f31b1e00640 1 -- 172.21.15.45:0/3467715317 shutdown_connections 2024-04-16T16:20:10.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.379+0000 7f31b1e00640 1 -- 172.21.15.45:0/3467715317 wait complete. 2024-04-16T16:20:10.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:20:10.420 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:10.420+0000 7fb78902b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:20:10.446 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:10.446+0000 7fddf422d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:20:10.499 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:10.499+0000 7fddf422d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:20:10.549 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:10.549+0000 7fb78902b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:20:10.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.593+0000 7f15e4000640 1 Processor -- start 2024-04-16T16:20:10.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.594+0000 7f15e4000640 1 -- start start 2024-04-16T16:20:10.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.595+0000 7f15e4000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f15dc06cf10 0x7f15dc06d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:10.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.595+0000 7f15e4000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15dc0d35d0 0x7f15dc0d3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:10.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.595+0000 7f15e4000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15dc06d8a0 con 0x7f15dc15ecc0 2024-04-16T16:20:10.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.595+0000 7f15e4000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f15dc0d2b10 con 0x7f15dc06cf10 2024-04-16T16:20:10.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.595+0000 7f15e4000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f15dc163210 con 0x7f15dc0d35d0 2024-04-16T16:20:10.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.595+0000 7f15e0c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f15dc06cf10 0x7f15dc06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:10.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.595+0000 7f15e1600640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f15dc15ecc0 0x7f15dc1626a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49086/0 (socket says 172.21.15.45:49086) 2024-04-16T16:20:10.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.595+0000 7f15e1600640 1 -- 172.21.15.45:0/2470241652 learned_addr learned my addr 172.21.15.45:0/2470241652 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:10.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.595+0000 7f15e2000640 1 --2- 172.21.15.45:0/2470241652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15dc0d35d0 0x7f15dc0d3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:10.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.596+0000 7f15e0c00640 1 -- 172.21.15.45:0/2470241652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15dc0d35d0 msgr2=0x7f15dc0d3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:10.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.596+0000 7f15e0c00640 1 --2- 172.21.15.45:0/2470241652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15dc0d35d0 0x7f15dc0d3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.596+0000 7f15e0c00640 1 -- 172.21.15.45:0/2470241652 >> v1:172.21.15.45:6789/0 conn(0x7f15dc15ecc0 legacy=0x7f15dc1626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:10.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.596+0000 7f15e0c00640 1 -- 172.21.15.45:0/2470241652 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15dc1634f0 con 0x7f15dc06cf10 2024-04-16T16:20:10.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.597+0000 7f15e0c00640 1 --2- 172.21.15.45:0/2470241652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc06cf10 0x7f15dc06d360 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f15c80097a0 tx=0x7f15c802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f28edc7bb822fac9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:10.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.597+0000 7f15dbe00640 1 -- 172.21.15.45:0/2470241652 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c8034070 con 0x7f15dc06cf10 2024-04-16T16:20:10.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.598+0000 7f15dbe00640 1 -- 172.21.15.45:0/2470241652 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15c8030800 con 0x7f15dc06cf10 2024-04-16T16:20:10.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.598+0000 7f15dbe00640 1 -- 172.21.15.45:0/2470241652 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c8030bd0 con 0x7f15dc06cf10 2024-04-16T16:20:10.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.598+0000 7f15e4000640 1 -- 172.21.15.45:0/2470241652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc06cf10 msgr2=0x7f15dc06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:10.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.598+0000 7f15e4000640 1 --2- 172.21.15.45:0/2470241652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc06cf10 0x7f15dc06d360 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f15c80097a0 tx=0x7f15c802b940 comp rx=0 tx=0).stop 2024-04-16T16:20:10.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.599+0000 7f15e4000640 1 -- 172.21.15.45:0/2470241652 shutdown_connections 2024-04-16T16:20:10.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.599+0000 7f15e4000640 1 --2- 172.21.15.45:0/2470241652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15dc0d35d0 0x7f15dc0d3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.599+0000 7f15e4000640 1 --2- 172.21.15.45:0/2470241652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc06cf10 0x7f15dc06d360 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.599+0000 7f15e4000640 1 -- 172.21.15.45:0/2470241652 >> 172.21.15.45:0/2470241652 conn(0x7f15dc073690 msgr2=0x7f15dc0d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:10.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.599+0000 7f15e4000640 1 -- 172.21.15.45:0/2470241652 shutdown_connections 2024-04-16T16:20:10.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.599+0000 7f15e4000640 1 -- 172.21.15.45:0/2470241652 wait complete. 2024-04-16T16:20:10.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.601+0000 7f15e4000640 1 Processor -- start 2024-04-16T16:20:10.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.601+0000 7f15e4000640 1 -- start start 2024-04-16T16:20:10.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.603+0000 7f15e4000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15dc06cf10 0x7f15dc152e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:10.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.603+0000 7f15e4000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15dc0d35d0 0x7f15dc0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:10.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.603+0000 7f15e4000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc15ecc0 0x7f15dc0cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:10.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.603+0000 7f15e4000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f15dc0ba540 con 0x7f15dc15ecc0 2024-04-16T16:20:10.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.603+0000 7f15e4000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f15dc0d30b0 con 0x7f15dc06cf10 2024-04-16T16:20:10.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.603+0000 7f15e4000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f15dc162db0 con 0x7f15dc0d35d0 2024-04-16T16:20:10.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.603+0000 7f15e2000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc15ecc0 0x7f15dc0cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:10.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.603+0000 7f15e2000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc15ecc0 0x7f15dc0cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60152/0 (socket says 172.21.15.45:60152) 2024-04-16T16:20:10.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.603+0000 7f15e2000640 1 -- 172.21.15.45:0/3968207265 learned_addr learned my addr 172.21.15.45:0/3968207265 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:10.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.604+0000 7f15e0c00640 1 --2- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15dc0d35d0 0x7f15dc0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:10.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.604+0000 7f15e1600640 1 --2- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15dc06cf10 0x7f15dc152e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:10.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.604+0000 7f15e2000640 1 -- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15dc0d35d0 msgr2=0x7f15dc0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:10.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.604+0000 7f15e2000640 1 --2- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15dc0d35d0 0x7f15dc0cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.604+0000 7f15e2000640 1 -- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15dc06cf10 msgr2=0x7f15dc152e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:10.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.604+0000 7f15e2000640 1 --2- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15dc06cf10 0x7f15dc152e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.604+0000 7f15e2000640 1 -- 172.21.15.45:0/3968207265 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15dc0cddd0 con 0x7f15dc15ecc0 2024-04-16T16:20:10.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.604+0000 7f15e2000640 1 --2- 172.21.15.45:0/3968207265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc15ecc0 0x7f15dc0cbfc0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f15d4007e70 tx=0x7f15d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:10.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.605+0000 7f15d9600640 1 -- 172.21.15.45:0/3968207265 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4014620 con 0x7f15dc15ecc0 2024-04-16T16:20:10.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.605+0000 7f15e4000640 1 -- 172.21.15.45:0/3968207265 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15dc0ce0b0 con 0x7f15dc15ecc0 2024-04-16T16:20:10.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.605+0000 7f15d9600640 1 -- 172.21.15.45:0/3968207265 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15d4019070 con 0x7f15dc15ecc0 2024-04-16T16:20:10.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.605+0000 7f15d9600640 1 -- 172.21.15.45:0/3968207265 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4014a20 con 0x7f15dc15ecc0 2024-04-16T16:20:10.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.605+0000 7f15e4000640 1 -- 172.21.15.45:0/3968207265 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15dc15f2f0 con 0x7f15dc15ecc0 2024-04-16T16:20:10.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.606+0000 7f15e4000640 1 -- 172.21.15.45:0/3968207265 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15dc0ce5d0 con 0x7f15dc15ecc0 2024-04-16T16:20:10.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.607+0000 7f15d9600640 1 -- 172.21.15.45:0/3968207265 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) ==== 91086+0+0 (secure 0 0 0) 0x7f15d4014bc0 con 0x7f15dc15ecc0 2024-04-16T16:20:10.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.608+0000 7f15d9600640 1 --2- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f15bc06c980 0x7f15bc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:10.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.608+0000 7f15d9600640 1 -- 172.21.15.45:0/3968207265 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 6096+0+0 (secure 0 0 0) 0x7f15d408c6e0 con 0x7f15dc15ecc0 2024-04-16T16:20:10.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.608+0000 7f15e1600640 1 -- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f15bc06c980 msgr2=0x7f15bc06ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:10.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.608+0000 7f15e1600640 1 --2- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f15bc06c980 0x7f15bc06ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:20:10.613 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:10.614+0000 7fb78902b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:20:10.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.616+0000 7f15d9600640 1 -- 172.21.15.45:0/3968207265 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f15d405a8d0 con 0x7f15dc15ecc0 2024-04-16T16:20:10.645 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:10.645+0000 7fddf422d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:20:10.664 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:10.665+0000 7fb78902b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:20:10.713 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:10.713+0000 7fddf422d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:20:10.722 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:10.722+0000 7fb78902b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:20:10.765 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:10.765+0000 7fddf422d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:20:10.773 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:10.774+0000 7fb78902b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:20:10.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.809+0000 7f15e1600640 1 -- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f15bc06c980 msgr2=0x7f15bc06ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:10.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.809+0000 7f15e1600640 1 --2- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f15bc06c980 0x7f15bc06ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:20:10.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.818+0000 7f15e4000640 1 -- 172.21.15.45:0/3968207265 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f15dc075780 con 0x7f15dc15ecc0 2024-04-16T16:20:10.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.819+0000 7f15d9600640 1 -- 172.21.15.45:0/3968207265 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) ==== 104+0+11 (secure 0 0 0) 0x7f15d405a1d0 con 0x7f15dc15ecc0 2024-04-16T16:20:10.819 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:20:10.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.822+0000 7f15e4000640 1 -- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f15bc06c980 msgr2=0x7f15bc06ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:20:10.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.822+0000 7f15e4000640 1 --2- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f15bc06c980 0x7f15bc06ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.822+0000 7f15e4000640 1 -- 172.21.15.45:0/3968207265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc15ecc0 msgr2=0x7f15dc0cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:10.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.822+0000 7f15e4000640 1 --2- 172.21.15.45:0/3968207265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc15ecc0 0x7f15dc0cbfc0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f15d4007e70 tx=0x7f15d400e4b0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.822+0000 7f15e4000640 1 -- 172.21.15.45:0/3968207265 shutdown_connections 2024-04-16T16:20:10.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.822+0000 7f15e4000640 1 --2- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f15bc06c980 0x7f15bc06ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.822+0000 7f15e4000640 1 --2- 172.21.15.45:0/3968207265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc15ecc0 0x7f15dc0cbfc0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.822+0000 7f15e4000640 1 --2- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15dc0d35d0 0x7f15dc0cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.822+0000 7f15e4000640 1 --2- 172.21.15.45:0/3968207265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15dc06cf10 0x7f15dc152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:10.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.822+0000 7f15e4000640 1 -- 172.21.15.45:0/3968207265 >> 172.21.15.45:0/3968207265 conn(0x7f15dc073690 msgr2=0x7f15dc0d49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:10.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.822+0000 7f15e4000640 1 -- 172.21.15.45:0/3968207265 shutdown_connections 2024-04-16T16:20:10.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:10.822+0000 7f15e4000640 1 -- 172.21.15.45:0/3968207265 wait complete. 2024-04-16T16:20:10.826 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:10.827+0000 7fddf422d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:20:10.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:20:10.884 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:10.885+0000 7fddf422d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:20:11.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.055+0000 7fd937200640 1 Processor -- start 2024-04-16T16:20:11.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.055+0000 7fd937200640 1 -- start start 2024-04-16T16:20:11.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.057+0000 7fd937200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9300b0e60 0x7fd93015eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:11.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.057+0000 7fd937200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd930070c00 con 0x7fd930071350 2024-04-16T16:20:11.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.057+0000 7fd937200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd93015c8d0 con 0x7fd93015c4f0 2024-04-16T16:20:11.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.057+0000 7fd937200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd93015fe90 con 0x7fd9300b0e60 2024-04-16T16:20:11.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.057+0000 7fd935400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd930071350 0x7fd9300b0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49088/0 (socket says 172.21.15.45:49088) 2024-04-16T16:20:11.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.057+0000 7fd935400640 1 -- 172.21.15.45:0/96061923 learned_addr learned my addr 172.21.15.45:0/96061923 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:11.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.058+0000 7fd936800640 1 --2- 172.21.15.45:0/96061923 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9300b0e60 0x7fd93015eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:11.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.058+0000 7fd934a00640 1 -- 172.21.15.45:0/96061923 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 85374844 0 0) 0x7fd930070c00 con 0x7fd930071350 2024-04-16T16:20:11.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.058+0000 7fd934a00640 1 -- 172.21.15.45:0/96061923 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd918003520 con 0x7fd930071350 2024-04-16T16:20:11.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.058+0000 7fd934a00640 1 -- 172.21.15.45:0/96061923 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2648652991 0 0) 0x7fd918003520 con 0x7fd930071350 2024-04-16T16:20:11.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.058+0000 7fd934a00640 1 -- 172.21.15.45:0/96061923 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9300b0e60 msgr2=0x7fd93015eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:11.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.058+0000 7fd934a00640 1 --2- 172.21.15.45:0/96061923 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9300b0e60 0x7fd93015eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.058+0000 7fd934a00640 1 -- 172.21.15.45:0/96061923 >> v1:172.21.15.103:6789/0 conn(0x7fd93015c4f0 legacy=0x7fd930162ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:11.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.058+0000 7fd934a00640 1 -- 172.21.15.45:0/96061923 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd93015f5f0 con 0x7fd930071350 2024-04-16T16:20:11.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.058+0000 7fd934a00640 1 -- 172.21.15.45:0/96061923 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fd920001670 con 0x7fd930071350 2024-04-16T16:20:11.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.058+0000 7fd934a00640 1 -- 172.21.15.45:0/96061923 >> v1:172.21.15.45:6789/0 conn(0x7fd930071350 legacy=0x7fd9300b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:11.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.058+0000 7fd934a00640 1 --2- 172.21.15.45:0/96061923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9180039e0 0x7fd918003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:11.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.058+0000 7fd934a00640 1 --2- 172.21.15.45:0/96061923 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9300b0e60 0x7fd918004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:11.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.059+0000 7fd934a00640 1 --2- 172.21.15.45:0/96061923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd918004d40 0x7fd9180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:11.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.059+0000 7fd934a00640 1 -- 172.21.15.45:0/96061923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd91800b9c0 con 0x7fd9180039e0 2024-04-16T16:20:11.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.059+0000 7fd934a00640 1 -- 172.21.15.45:0/96061923 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd91800bb70 con 0x7fd918004d40 2024-04-16T16:20:11.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.059+0000 7fd934a00640 1 -- 172.21.15.45:0/96061923 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd91800bd70 con 0x7fd9300b0e60 2024-04-16T16:20:11.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.059+0000 7fd935e00640 1 --2- 172.21.15.45:0/96061923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9180039e0 0x7fd918003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:11.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.059+0000 7fd935400640 1 --2- 172.21.15.45:0/96061923 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9300b0e60 0x7fd918004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:11.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.059+0000 7fd935e00640 1 -- 172.21.15.45:0/96061923 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9300b0e60 msgr2=0x7fd918004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:11.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.059+0000 7fd935e00640 1 --2- 172.21.15.45:0/96061923 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9300b0e60 0x7fd918004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.059+0000 7fd935e00640 1 -- 172.21.15.45:0/96061923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd918004d40 msgr2=0x7fd9180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:11.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.059+0000 7fd935e00640 1 --2- 172.21.15.45:0/96061923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd918004d40 0x7fd9180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.059+0000 7fd935e00640 1 -- 172.21.15.45:0/96061923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd93015f5f0 con 0x7fd9180039e0 2024-04-16T16:20:11.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.060+0000 7fd935e00640 1 --2- 172.21.15.45:0/96061923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9180039e0 0x7fd918003dc0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fd924001860 tx=0x7fd918005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8305900fb890e01 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:11.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.061+0000 7fd934a00640 1 -- 172.21.15.45:0/96061923 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd924038070 con 0x7fd9180039e0 2024-04-16T16:20:11.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.061+0000 7fd934a00640 1 -- 172.21.15.45:0/96061923 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd924033660 con 0x7fd9180039e0 2024-04-16T16:20:11.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.061+0000 7fd934a00640 1 -- 172.21.15.45:0/96061923 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd924001430 con 0x7fd9180039e0 2024-04-16T16:20:11.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.061+0000 7fd937200640 1 -- 172.21.15.45:0/96061923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9180039e0 msgr2=0x7fd918003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:11.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.061+0000 7fd937200640 1 --2- 172.21.15.45:0/96061923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9180039e0 0x7fd918003dc0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fd924001860 tx=0x7fd918005830 comp rx=0 tx=0).stop 2024-04-16T16:20:11.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.061+0000 7fd936800640 1 -- 172.21.15.45:0/96061923 reap_dead start 2024-04-16T16:20:11.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.061+0000 7fd937200640 1 -- 172.21.15.45:0/96061923 shutdown_connections 2024-04-16T16:20:11.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.061+0000 7fd937200640 1 -- 172.21.15.45:0/96061923 >> 172.21.15.45:0/96061923 conn(0x7fd93006f120 msgr2=0x7fd93006f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:11.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.062+0000 7fd937200640 1 -- 172.21.15.45:0/96061923 shutdown_connections 2024-04-16T16:20:11.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.062+0000 7fd937200640 1 -- 172.21.15.45:0/96061923 wait complete. 2024-04-16T16:20:11.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.063+0000 7fd937200640 1 Processor -- start 2024-04-16T16:20:11.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.063+0000 7fd937200640 1 -- start start 2024-04-16T16:20:11.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.064+0000 7fd937200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9180039e0 0x7fd9301f5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:11.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.064+0000 7fd937200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd93015fdf0 0x7fd9301f8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:11.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.064+0000 7fd937200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9301f6fc0 0x7fd9300b2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:11.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.064+0000 7fd937200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd93015fbb0 con 0x7fd9180039e0 2024-04-16T16:20:11.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.064+0000 7fd937200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd930162710 con 0x7fd9301f6fc0 2024-04-16T16:20:11.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.064+0000 7fd937200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd9301632b0 con 0x7fd93015fdf0 2024-04-16T16:20:11.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.064+0000 7fd935e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9180039e0 0x7fd9301f5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:11.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.065+0000 7fd935e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9180039e0 0x7fd9301f5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60182/0 (socket says 172.21.15.45:60182) 2024-04-16T16:20:11.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.065+0000 7fd935e00640 1 -- 172.21.15.45:0/979725610 learned_addr learned my addr 172.21.15.45:0/979725610 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:11.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.065+0000 7fd935400640 1 --2- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd93015fdf0 0x7fd9301f8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:11.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.065+0000 7fd936800640 1 --2- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9301f6fc0 0x7fd9300b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:11.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.065+0000 7fd935e00640 1 -- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd93015fdf0 msgr2=0x7fd9301f8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:11.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.065+0000 7fd935e00640 1 --2- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd93015fdf0 0x7fd9301f8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.065+0000 7fd935e00640 1 -- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9301f6fc0 msgr2=0x7fd9300b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:11.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.065+0000 7fd935e00640 1 --2- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9301f6fc0 0x7fd9300b2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.065+0000 7fd935e00640 1 -- 172.21.15.45:0/979725610 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9301602b0 con 0x7fd9180039e0 2024-04-16T16:20:11.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.065+0000 7fd935e00640 1 --2- 172.21.15.45:0/979725610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9180039e0 0x7fd9301f5390 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fd9240018c0 tx=0x7fd924001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:11.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.066+0000 7fd92a000640 1 -- 172.21.15.45:0/979725610 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd924043020 con 0x7fd9180039e0 2024-04-16T16:20:11.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.066+0000 7fd92a000640 1 -- 172.21.15.45:0/979725610 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd924039040 con 0x7fd9180039e0 2024-04-16T16:20:11.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.066+0000 7fd92a000640 1 -- 172.21.15.45:0/979725610 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd924038070 con 0x7fd9180039e0 2024-04-16T16:20:11.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.066+0000 7fd937200640 1 -- 172.21.15.45:0/979725610 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd930160590 con 0x7fd9180039e0 2024-04-16T16:20:11.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.066+0000 7fd937200640 1 -- 172.21.15.45:0/979725610 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd930160b50 con 0x7fd9180039e0 2024-04-16T16:20:11.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.067+0000 7fd937200640 1 -- 172.21.15.45:0/979725610 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd930075780 con 0x7fd9180039e0 2024-04-16T16:20:11.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.068+0000 7fd92a000640 1 -- 172.21.15.45:0/979725610 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) ==== 91086+0+0 (secure 0 0 0) 0x7fd924046020 con 0x7fd9180039e0 2024-04-16T16:20:11.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.069+0000 7fd92a000640 1 --2- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fd91406c9b0 0x7fd91406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:11.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.070+0000 7fd92a000640 1 -- 172.21.15.45:0/979725610 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 6096+0+0 (secure 0 0 0) 0x7fd9240ad950 con 0x7fd9180039e0 2024-04-16T16:20:11.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.070+0000 7fd935400640 1 -- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fd91406c9b0 msgr2=0x7fd91406ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:11.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.070+0000 7fd935400640 1 --2- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fd91406c9b0 0x7fd91406ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:20:11.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.074+0000 7fd92a000640 1 -- 172.21.15.45:0/979725610 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd92404d030 con 0x7fd9180039e0 2024-04-16T16:20:11.164 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:11.165+0000 7fb78902b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:20:11.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.271+0000 7fd935400640 1 -- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fd91406c9b0 msgr2=0x7fd91406ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:11.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.271+0000 7fd935400640 1 --2- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fd91406c9b0 0x7fd91406ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:20:11.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.276+0000 7fd937200640 1 -- 172.21.15.45:0/979725610 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd930162d20 con 0x7fd9180039e0 2024-04-16T16:20:11.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.276+0000 7fd92a000640 1 -- 172.21.15.45:0/979725610 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) ==== 100+0+11 (secure 0 0 0) 0x7fd9240323a0 con 0x7fd9180039e0 2024-04-16T16:20:11.276 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:20:11.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.279+0000 7fd937200640 1 -- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fd91406c9b0 msgr2=0x7fd91406ee70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:20:11.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.279+0000 7fd937200640 1 --2- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fd91406c9b0 0x7fd91406ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.279+0000 7fd937200640 1 -- 172.21.15.45:0/979725610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9180039e0 msgr2=0x7fd9301f5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:11.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.279+0000 7fd937200640 1 --2- 172.21.15.45:0/979725610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9180039e0 0x7fd9301f5390 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fd9240018c0 tx=0x7fd924001230 comp rx=0 tx=0).stop 2024-04-16T16:20:11.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.279+0000 7fd937200640 1 -- 172.21.15.45:0/979725610 shutdown_connections 2024-04-16T16:20:11.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.279+0000 7fd937200640 1 --2- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fd91406c9b0 0x7fd91406ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.279+0000 7fd937200640 1 --2- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9301f6fc0 0x7fd9300b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.279+0000 7fd937200640 1 --2- 172.21.15.45:0/979725610 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd93015fdf0 0x7fd9301f8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.279+0000 7fd937200640 1 --2- 172.21.15.45:0/979725610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9180039e0 0x7fd9301f5390 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.279+0000 7fd937200640 1 -- 172.21.15.45:0/979725610 >> 172.21.15.45:0/979725610 conn(0x7fd93006f120 msgr2=0x7fd9300d00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:11.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.279+0000 7fd937200640 1 -- 172.21.15.45:0/979725610 shutdown_connections 2024-04-16T16:20:11.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.279+0000 7fd937200640 1 -- 172.21.15.45:0/979725610 wait complete. 2024-04-16T16:20:11.298 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:20:11.301 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:11.301+0000 7fddf422d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:20:11.329 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:11.329+0000 7fb78902b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:20:11.438 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:11.439+0000 7fb78902b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:20:11.473 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:11.473+0000 7fddf422d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:20:11.489 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:11.489+0000 7fb78902b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:20:11.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.505+0000 7f5db3200640 1 Processor -- start 2024-04-16T16:20:11.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.506+0000 7f5db3200640 1 -- start start 2024-04-16T16:20:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.507+0000 7f5db3200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5db40ba140 0x7f5db40d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.507+0000 7f5db3200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5db40cfc20 0x7f5db40d35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.507+0000 7f5db3200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5db40d2600 con 0x7f5db4162090 2024-04-16T16:20:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.507+0000 7f5db3200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5db40ba8b0 con 0x7f5db40ba140 2024-04-16T16:20:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.507+0000 7f5db3200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5db40ba5d0 con 0x7f5db40cfc20 2024-04-16T16:20:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.508+0000 7f5dabe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5db40ba140 0x7f5db40d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.508+0000 7f5db1e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5db4162090 0x7f5db40cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49104/0 (socket says 172.21.15.45:49104) 2024-04-16T16:20:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.508+0000 7f5db1e00640 1 -- 172.21.15.45:0/1464608381 learned_addr learned my addr 172.21.15.45:0/1464608381 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.508+0000 7f5db2800640 1 --2- 172.21.15.45:0/1464608381 >> v2:172.21.15.103:3300/0 conn(0x7f5db40cfc20 0x7f5db40d35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.508+0000 7f5dabe00640 1 -- 172.21.15.45:0/1464608381 >> v2:172.21.15.103:3300/0 conn(0x7f5db40cfc20 msgr2=0x7f5db40d35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.508+0000 7f5dabe00640 1 --2- 172.21.15.45:0/1464608381 >> v2:172.21.15.103:3300/0 conn(0x7f5db40cfc20 0x7f5db40d35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.508+0000 7f5dabe00640 1 -- 172.21.15.45:0/1464608381 >> v1:172.21.15.45:6789/0 conn(0x7f5db4162090 legacy=0x7f5db40cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:11.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.508+0000 7f5db1400640 1 -- 172.21.15.45:0/1464608381 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 897985254 0 0) 0x7f5db40d2600 con 0x7f5db4162090 2024-04-16T16:20:11.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.508+0000 7f5dabe00640 1 -- 172.21.15.45:0/1464608381 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db40d28a0 con 0x7f5db40ba140 2024-04-16T16:20:11.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.509+0000 7f5dabe00640 1 --2- 172.21.15.45:0/1464608381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db40ba140 0x7f5db40d30a0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f5db40cb8f0 tx=0x7f5d9c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456bb3302dd4a9ae server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.510+0000 7f5db1400640 1 -- 172.21.15.45:0/1464608381 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c035020 con 0x7f5db40ba140 2024-04-16T16:20:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.510+0000 7f5db1400640 1 -- 172.21.15.45:0/1464608381 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d9c031830 con 0x7f5db40ba140 2024-04-16T16:20:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.510+0000 7f5db1400640 1 -- 172.21.15.45:0/1464608381 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c031c00 con 0x7f5db40ba140 2024-04-16T16:20:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.510+0000 7f5db3200640 1 -- 172.21.15.45:0/1464608381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db40ba140 msgr2=0x7f5db40d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.510+0000 7f5db3200640 1 --2- 172.21.15.45:0/1464608381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db40ba140 0x7f5db40d30a0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f5db40cb8f0 tx=0x7f5d9c02b8d0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.511+0000 7f5db3200640 1 -- 172.21.15.45:0/1464608381 shutdown_connections 2024-04-16T16:20:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.511+0000 7f5db3200640 1 --2- 172.21.15.45:0/1464608381 >> v2:172.21.15.103:3300/0 conn(0x7f5db40cfc20 0x7f5db40d35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.511+0000 7f5db3200640 1 --2- 172.21.15.45:0/1464608381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db40ba140 0x7f5db40d30a0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.511+0000 7f5db3200640 1 -- 172.21.15.45:0/1464608381 >> 172.21.15.45:0/1464608381 conn(0x7f5db40d1940 msgr2=0x7f5db40d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.511+0000 7f5db3200640 1 -- 172.21.15.45:0/1464608381 shutdown_connections 2024-04-16T16:20:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.511+0000 7f5db3200640 1 -- 172.21.15.45:0/1464608381 wait complete. 2024-04-16T16:20:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.512+0000 7f5db3200640 1 Processor -- start 2024-04-16T16:20:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.513+0000 7f5db3200640 1 -- start start 2024-04-16T16:20:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.513+0000 7f5db3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5db40cfc20 0x7f5db41f8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.513+0000 7f5db3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5db4162090 0x7f5db41f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.513+0000 7f5db3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db40ce1b0 0x7f5db40b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.513+0000 7f5db3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5db41625f0 con 0x7f5db40ce1b0 2024-04-16T16:20:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.513+0000 7f5db3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5db4162470 con 0x7f5db4162090 2024-04-16T16:20:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.513+0000 7f5db3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5db40d5520 con 0x7f5db40cfc20 2024-04-16T16:20:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.514+0000 7f5db2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db40ce1b0 0x7f5db40b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.514+0000 7f5db2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db40ce1b0 0x7f5db40b53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60190/0 (socket says 172.21.15.45:60190) 2024-04-16T16:20:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.514+0000 7f5db2800640 1 -- 172.21.15.45:0/1138252767 learned_addr learned my addr 172.21.15.45:0/1138252767 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.514+0000 7f5dabe00640 1 --2- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5db4162090 0x7f5db41f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.514+0000 7f5db1e00640 1 --2- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5db40cfc20 0x7f5db41f8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.514+0000 7f5db2800640 1 -- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5db40cfc20 msgr2=0x7f5db41f8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.514+0000 7f5db2800640 1 --2- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5db40cfc20 0x7f5db41f8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.514+0000 7f5db2800640 1 -- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5db4162090 msgr2=0x7f5db41f49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.514+0000 7f5db2800640 1 --2- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5db4162090 0x7f5db41f49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.514+0000 7f5db2800640 1 -- 172.21.15.45:0/1138252767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db41f94f0 con 0x7f5db40ce1b0 2024-04-16T16:20:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.514+0000 7f5db1e00640 1 --2- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5db40cfc20 0x7f5db41f8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.515+0000 7f5db2800640 1 --2- 172.21.15.45:0/1138252767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db40ce1b0 0x7f5db40b53f0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f5dac007e40 tx=0x7f5dac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.515+0000 7f5daa000640 1 -- 172.21.15.45:0/1138252767 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac014620 con 0x7f5db40ce1b0 2024-04-16T16:20:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.515+0000 7f5daa000640 1 -- 172.21.15.45:0/1138252767 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dac019070 con 0x7f5db40ce1b0 2024-04-16T16:20:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.515+0000 7f5daa000640 1 -- 172.21.15.45:0/1138252767 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac014a20 con 0x7f5db40ce1b0 2024-04-16T16:20:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.515+0000 7f5db3200640 1 -- 172.21.15.45:0/1138252767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db41f4ef0 con 0x7f5db40ce1b0 2024-04-16T16:20:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.515+0000 7f5db3200640 1 -- 172.21.15.45:0/1138252767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db40cd6d0 con 0x7f5db40ce1b0 2024-04-16T16:20:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.516+0000 7f5db3200640 1 -- 172.21.15.45:0/1138252767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5db40cec00 con 0x7f5db40ce1b0 2024-04-16T16:20:11.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.518+0000 7f5daa000640 1 -- 172.21.15.45:0/1138252767 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) ==== 91086+0+0 (secure 0 0 0) 0x7f5dac014bc0 con 0x7f5db40ce1b0 2024-04-16T16:20:11.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.519+0000 7f5daa000640 1 --2- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f5d9006c9b0 0x7f5d9006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:11.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.519+0000 7f5daa000640 1 -- 172.21.15.45:0/1138252767 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 6096+0+0 (secure 0 0 0) 0x7f5dac08c560 con 0x7f5db40ce1b0 2024-04-16T16:20:11.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.520+0000 7f5db1e00640 1 -- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f5d9006c9b0 msgr2=0x7f5d9006ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:11.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.520+0000 7f5db1e00640 1 --2- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f5d9006c9b0 0x7f5d9006ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:20:11.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.524+0000 7f5daa000640 1 -- 172.21.15.45:0/1138252767 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5db40cec00 con 0x7f5db40ce1b0 2024-04-16T16:20:11.593 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:11.593+0000 7fddf422d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:20:11.618 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:11.618+0000 7fb78902b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:20:11.647 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:11.647+0000 7fddf422d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:20:11.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.721+0000 7f5db1e00640 1 -- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f5d9006c9b0 msgr2=0x7f5d9006ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:11.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.721+0000 7f5db1e00640 1 --2- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f5d9006c9b0 0x7f5d9006ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:20:11.727 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:11.727+0000 7fb78902b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:20:11.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.741+0000 7f5db3200640 1 -- 172.21.15.45:0/1138252767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5db4075780 con 0x7f5db40ce1b0 2024-04-16T16:20:11.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.742+0000 7f5daa000640 1 -- 172.21.15.45:0/1138252767 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 9 entries v116) ==== 80+0+535 (secure 0 0 0) 0x7f5db41f4ef0 con 0x7f5db40ce1b0 2024-04-16T16:20:11.742 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3506765807 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:11.743 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3547880684 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:11.743 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/2293932214 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:11.743 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/2293932214 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:11.743 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1347839066 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:11.743 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/546409297 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:11.743 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3566291251 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:11.743 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/714185246 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:11.743 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2784388736 2024-04-16T17:19:54.469649+0000 2024-04-16T16:20:11.743 INFO:teuthology.orchestra.run.smithi045.stderr:listed 9 entries 2024-04-16T16:20:11.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.745+0000 7f5db3200640 1 -- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f5d9006c9b0 msgr2=0x7f5d9006ee70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:20:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.745+0000 7f5db3200640 1 --2- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f5d9006c9b0 0x7f5d9006ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.745+0000 7f5db3200640 1 -- 172.21.15.45:0/1138252767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db40ce1b0 msgr2=0x7f5db40b53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.745+0000 7f5db3200640 1 --2- 172.21.15.45:0/1138252767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db40ce1b0 0x7f5db40b53f0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f5dac007e40 tx=0x7f5dac00e4b0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.745+0000 7f5db3200640 1 -- 172.21.15.45:0/1138252767 shutdown_connections 2024-04-16T16:20:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.745+0000 7f5db3200640 1 --2- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f5d9006c9b0 0x7f5d9006ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.745+0000 7f5db3200640 1 --2- 172.21.15.45:0/1138252767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db40ce1b0 0x7f5db40b53f0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.745+0000 7f5db3200640 1 --2- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5db4162090 0x7f5db41f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.745+0000 7f5db3200640 1 --2- 172.21.15.45:0/1138252767 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5db40cfc20 0x7f5db41f8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.745+0000 7f5db3200640 1 -- 172.21.15.45:0/1138252767 >> 172.21.15.45:0/1138252767 conn(0x7f5db40d1940 msgr2=0x7f5db40b7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.745+0000 7f5db3200640 1 -- 172.21.15.45:0/1138252767 shutdown_connections 2024-04-16T16:20:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:11.745+0000 7f5db3200640 1 -- 172.21.15.45:0/1138252767 wait complete. 2024-04-16T16:20:11.763 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-16T16:20:11.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:11.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-16T16:20:11.782 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:11.783+0000 7fddf422d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:20:11.845 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:20:11.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:11.846 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-16T16:20:11.901 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:11.901+0000 7fddf422d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:20:11.926 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:11.927+0000 7fb78902b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:20:11.932 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:20:11.932 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:20:11.932 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:20:11.982 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:11.982+0000 7fb78902b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:20:11.986 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:20:11.986 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:20:12.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:20:12.088 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:12.088+0000 7fddf422d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:20:12.132 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:12.132+0000 7fb78902b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:20:12.142 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:12.142+0000 7fddf422d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:20:12.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.289+0000 7fd021c00640 1 Processor -- start 2024-04-16T16:20:12.290 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:12.289+0000 7fddf422d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:20:12.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.290+0000 7fd021c00640 1 -- start start 2024-04-16T16:20:12.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.291+0000 7fd021c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd01c071a40 0x7fd01c0b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:12.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.291+0000 7fd021c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd01c1604d0 con 0x7fd01c0b7750 2024-04-16T16:20:12.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.291+0000 7fd021c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd01c1607b0 con 0x7fd01c15cd40 2024-04-16T16:20:12.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.291+0000 7fd021c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd01c160a90 con 0x7fd01c071a40 2024-04-16T16:20:12.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.292+0000 7fd01b400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd01c0b7750 0x7fd01c071e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49110/0 (socket says 172.21.15.45:49110) 2024-04-16T16:20:12.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.292+0000 7fd01b400640 1 -- 172.21.15.45:0/2692858617 learned_addr learned my addr 172.21.15.45:0/2692858617 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:12.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.292+0000 7fd01be00640 1 --2- 172.21.15.45:0/2692858617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd01c071a40 0x7fd01c0b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.292+0000 7fd01aa00640 1 -- 172.21.15.45:0/2692858617 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4081508807 0 0) 0x7fd01c1604d0 con 0x7fd01c0b7750 2024-04-16T16:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.292+0000 7fd01aa00640 1 -- 172.21.15.45:0/2692858617 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd000003520 con 0x7fd01c0b7750 2024-04-16T16:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd01aa00640 1 -- 172.21.15.45:0/2692858617 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2181912135 0 0) 0x7fd000003520 con 0x7fd01c0b7750 2024-04-16T16:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd01aa00640 1 -- 172.21.15.45:0/2692858617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd01c071a40 msgr2=0x7fd01c0b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd01aa00640 1 --2- 172.21.15.45:0/2692858617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd01c071a40 0x7fd01c0b56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd01aa00640 1 -- 172.21.15.45:0/2692858617 >> v1:172.21.15.103:6789/0 conn(0x7fd01c15cd40 legacy=0x7fd01c0d1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd01aa00640 1 -- 172.21.15.45:0/2692858617 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd01c160d70 con 0x7fd01c0b7750 2024-04-16T16:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd01aa00640 1 -- 172.21.15.45:0/2692858617 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fd004001670 con 0x7fd01c0b7750 2024-04-16T16:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd01aa00640 1 -- 172.21.15.45:0/2692858617 >> v1:172.21.15.45:6789/0 conn(0x7fd01c0b7750 legacy=0x7fd01c071e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd01aa00640 1 --2- 172.21.15.45:0/2692858617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0000039e0 0x7fd000003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd01aa00640 1 --2- 172.21.15.45:0/2692858617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd01c071a40 0x7fd000004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd01aa00640 1 --2- 172.21.15.45:0/2692858617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd000004d40 0x7fd0000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd01aa00640 1 -- 172.21.15.45:0/2692858617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd01c160a90 con 0x7fd000004d40 2024-04-16T16:20:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd01aa00640 1 -- 172.21.15.45:0/2692858617 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd00000ba70 con 0x7fd0000039e0 2024-04-16T16:20:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd01aa00640 1 -- 172.21.15.45:0/2692858617 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd00000bc20 con 0x7fd01c071a40 2024-04-16T16:20:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd021200640 1 --2- 172.21.15.45:0/2692858617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd000004d40 0x7fd0000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd01b400640 1 --2- 172.21.15.45:0/2692858617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd01c071a40 0x7fd000004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd01be00640 1 --2- 172.21.15.45:0/2692858617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0000039e0 0x7fd000003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd021200640 1 -- 172.21.15.45:0/2692858617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd01c071a40 msgr2=0x7fd000004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.293+0000 7fd021200640 1 --2- 172.21.15.45:0/2692858617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd01c071a40 0x7fd000004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.294+0000 7fd021200640 1 -- 172.21.15.45:0/2692858617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0000039e0 msgr2=0x7fd000003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.294+0000 7fd021200640 1 --2- 172.21.15.45:0/2692858617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0000039e0 0x7fd000003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.294+0000 7fd021200640 1 -- 172.21.15.45:0/2692858617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd000003520 con 0x7fd000004d40 2024-04-16T16:20:12.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.295+0000 7fd021200640 1 --2- 172.21.15.45:0/2692858617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd000004d40 0x7fd0000051c0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fd014001860 tx=0x7fd000005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f618286d7fb2174 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:12.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.295+0000 7fd01aa00640 1 -- 172.21.15.45:0/2692858617 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd014038070 con 0x7fd000004d40 2024-04-16T16:20:12.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.295+0000 7fd01aa00640 1 -- 172.21.15.45:0/2692858617 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd014033410 con 0x7fd000004d40 2024-04-16T16:20:12.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.295+0000 7fd01aa00640 1 -- 172.21.15.45:0/2692858617 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd014001430 con 0x7fd000004d40 2024-04-16T16:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.295+0000 7fd021c00640 1 -- 172.21.15.45:0/2692858617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd000004d40 msgr2=0x7fd0000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.295+0000 7fd021c00640 1 --2- 172.21.15.45:0/2692858617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd000004d40 0x7fd0000051c0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fd014001860 tx=0x7fd000005830 comp rx=0 tx=0).stop 2024-04-16T16:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.295+0000 7fd021200640 1 -- 172.21.15.45:0/2692858617 reap_dead start 2024-04-16T16:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.296+0000 7fd021c00640 1 -- 172.21.15.45:0/2692858617 shutdown_connections 2024-04-16T16:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.296+0000 7fd021c00640 1 -- 172.21.15.45:0/2692858617 >> 172.21.15.45:0/2692858617 conn(0x7fd01c0b43a0 msgr2=0x7fd01c0b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.296+0000 7fd021c00640 1 -- 172.21.15.45:0/2692858617 shutdown_connections 2024-04-16T16:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.296+0000 7fd021c00640 1 -- 172.21.15.45:0/2692858617 wait complete. 2024-04-16T16:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.297+0000 7fd021c00640 1 Processor -- start 2024-04-16T16:20:12.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.298+0000 7fd021c00640 1 -- start start 2024-04-16T16:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.298+0000 7fd021c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd000004d40 0x7fd01c155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.299+0000 7fd021c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd01c145170 0x7fd01c153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.299+0000 7fd021c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd01c1e78e0 0x7fd01c1e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.299+0000 7fd021c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd01c072b20 con 0x7fd01c145170 2024-04-16T16:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.299+0000 7fd021c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd01c15a670 con 0x7fd000004d40 2024-04-16T16:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.299+0000 7fd021c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd01c0b56a0 con 0x7fd01c1e78e0 2024-04-16T16:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.299+0000 7fd01b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd01c145170 0x7fd01c153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.299+0000 7fd01b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd01c145170 0x7fd01c153720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60206/0 (socket says 172.21.15.45:60206) 2024-04-16T16:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.299+0000 7fd01b400640 1 -- 172.21.15.45:0/3462383674 learned_addr learned my addr 172.21.15.45:0/3462383674 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.299+0000 7fd01be00640 1 --2- 172.21.15.45:0/3462383674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd000004d40 0x7fd01c155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.299+0000 7fd01b400640 1 -- 172.21.15.45:0/3462383674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd01c1e78e0 msgr2=0x7fd01c1e7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.299+0000 7fd01b400640 1 --2- 172.21.15.45:0/3462383674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd01c1e78e0 0x7fd01c1e7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.299+0000 7fd01b400640 1 -- 172.21.15.45:0/3462383674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd000004d40 msgr2=0x7fd01c155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.299+0000 7fd01b400640 1 --2- 172.21.15.45:0/3462383674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd000004d40 0x7fd01c155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:12.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.299+0000 7fd01b400640 1 -- 172.21.15.45:0/3462383674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd01c153c60 con 0x7fd01c145170 2024-04-16T16:20:12.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.300+0000 7fd01b400640 1 --2- 172.21.15.45:0/3462383674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd01c145170 0x7fd01c153720 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fd00400d7a0 tx=0x7fd00400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:12.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.300+0000 7fd013e00640 1 -- 172.21.15.45:0/3462383674 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd004019070 con 0x7fd01c145170 2024-04-16T16:20:12.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.300+0000 7fd013e00640 1 -- 172.21.15.45:0/3462383674 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd00400de50 con 0x7fd01c145170 2024-04-16T16:20:12.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.300+0000 7fd013e00640 1 -- 172.21.15.45:0/3462383674 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0040047e0 con 0x7fd01c145170 2024-04-16T16:20:12.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.300+0000 7fd021c00640 1 -- 172.21.15.45:0/3462383674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd01c153f30 con 0x7fd01c145170 2024-04-16T16:20:12.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.300+0000 7fd021c00640 1 -- 172.21.15.45:0/3462383674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd01c1544f0 con 0x7fd01c145170 2024-04-16T16:20:12.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.301+0000 7fd021c00640 1 -- 172.21.15.45:0/3462383674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd01c0b4f50 con 0x7fd01c145170 2024-04-16T16:20:12.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.302+0000 7fd013e00640 1 -- 172.21.15.45:0/3462383674 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) ==== 91086+0+0 (secure 0 0 0) 0x7fd004004980 con 0x7fd01c145170 2024-04-16T16:20:12.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.303+0000 7fd013e00640 1 --2- 172.21.15.45:0/3462383674 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fcff806c9b0 0x7fcff806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:12.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.303+0000 7fd013e00640 1 -- 172.21.15.45:0/3462383674 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 6096+0+0 (secure 0 0 0) 0x7fd00408b800 con 0x7fd01c145170 2024-04-16T16:20:12.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.303+0000 7fd01be00640 1 -- 172.21.15.45:0/3462383674 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fcff806c9b0 msgr2=0x7fcff806ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:12.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.303+0000 7fd01be00640 1 --2- 172.21.15.45:0/3462383674 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fcff806c9b0 0x7fcff806ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:20:12.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.307+0000 7fd013e00640 1 -- 172.21.15.45:0/3462383674 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd0040598d0 con 0x7fd01c145170 2024-04-16T16:20:12.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.504+0000 7fd01be00640 1 -- 172.21.15.45:0/3462383674 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fcff806c9b0 msgr2=0x7fcff806ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:12.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.504+0000 7fd01be00640 1 --2- 172.21.15.45:0/3462383674 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fcff806c9b0 0x7fcff806ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:20:12.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.512+0000 7fd021c00640 1 -- 172.21.15.45:0/3462383674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd01c0d00b0 con 0x7fd01c145170 2024-04-16T16:20:12.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.514+0000 7fd013e00640 1 -- 172.21.15.45:0/3462383674 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+17197 (secure 0 0 0) 0x7fd0040591d0 con 0x7fd01c145170 2024-04-16T16:20:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:12.515 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":116,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:20:07.040908+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:48.114834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:49.135741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3506765807":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/3547880684":"2024-04-17T16:20:07.040813+0000","172.21.15.103:6833/2293932214":"2024-04-17T16:20:07.040813+0000","172.21.15.103:6832/2293932214":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/1347839066":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/546409297":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/3566291251":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/714185246":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/2784388736":"2024-04-16T17:19:54.469649+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:20:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.517+0000 7fd021c00640 1 -- 172.21.15.45:0/3462383674 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fcff806c9b0 msgr2=0x7fcff806ee70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:20:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.517+0000 7fd021c00640 1 --2- 172.21.15.45:0/3462383674 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fcff806c9b0 0x7fcff806ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.517+0000 7fd021c00640 1 -- 172.21.15.45:0/3462383674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd01c145170 msgr2=0x7fd01c153720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.517+0000 7fd021c00640 1 --2- 172.21.15.45:0/3462383674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd01c145170 0x7fd01c153720 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fd00400d7a0 tx=0x7fd00400dc60 comp rx=0 tx=0).stop 2024-04-16T16:20:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.517+0000 7fd021c00640 1 -- 172.21.15.45:0/3462383674 shutdown_connections 2024-04-16T16:20:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.517+0000 7fd021c00640 1 --2- 172.21.15.45:0/3462383674 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fcff806c9b0 0x7fcff806ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.517+0000 7fd021c00640 1 --2- 172.21.15.45:0/3462383674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd01c1e78e0 0x7fd01c1e7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.517+0000 7fd021c00640 1 --2- 172.21.15.45:0/3462383674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd01c145170 0x7fd01c153720 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.517+0000 7fd021c00640 1 --2- 172.21.15.45:0/3462383674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd000004d40 0x7fd01c155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.517+0000 7fd021c00640 1 -- 172.21.15.45:0/3462383674 >> 172.21.15.45:0/3462383674 conn(0x7fd01c0b43a0 msgr2=0x7fd01c0b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.517+0000 7fd021c00640 1 -- 172.21.15.45:0/3462383674 shutdown_connections 2024-04-16T16:20:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.517+0000 7fd021c00640 1 -- 172.21.15.45:0/3462383674 wait complete. 2024-04-16T16:20:12.535 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:19:48.114834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:19:49.135741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:20:12.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:20:12.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.788+0000 7fabb7e00640 1 Processor -- start 2024-04-16T16:20:12.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.788+0000 7fabb7e00640 1 -- start start 2024-04-16T16:20:12.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.790+0000 7fabb7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fabb015b0d0 0x7fabb00b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:12.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.790+0000 7fabb7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fabb00af960 0x7fabb00b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:12.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.790+0000 7fabb7e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb00b4b20 con 0x7fabb00b5bc0 2024-04-16T16:20:12.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.790+0000 7fabb7e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fabb00ced80 con 0x7fabb00af960 2024-04-16T16:20:12.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.790+0000 7fabb7e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fabb00b4e00 con 0x7fabb015b0d0 2024-04-16T16:20:12.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.790+0000 7fabb5e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fabb00af960 0x7fabb00b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:12.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.790+0000 7fabb5e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fabb00af960 0x7fabb00b2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60220/0 (socket says 172.21.15.45:60220) 2024-04-16T16:20:12.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.790+0000 7fabb5e00640 1 -- 172.21.15.45:0/3007918891 learned_addr learned my addr 172.21.15.45:0/3007918891 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:12.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.791+0000 7fabb5400640 1 --2- 172.21.15.45:0/3007918891 >> v2:172.21.15.103:3300/0 conn(0x7fabb015b0d0 0x7fabb00b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:12.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.791+0000 7fabb5e00640 1 -- 172.21.15.45:0/3007918891 >> v2:172.21.15.103:3300/0 conn(0x7fabb015b0d0 msgr2=0x7fabb00b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:12.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.791+0000 7fabb5e00640 1 --2- 172.21.15.45:0/3007918891 >> v2:172.21.15.103:3300/0 conn(0x7fabb015b0d0 0x7fabb00b2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:12.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.791+0000 7fabb5e00640 1 -- 172.21.15.45:0/3007918891 >> v1:172.21.15.103:6789/0 conn(0x7fabb00b5bc0 legacy=0x7fabb00721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:12.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.791+0000 7fabb5e00640 1 -- 172.21.15.45:0/3007918891 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb00b50e0 con 0x7fabb00af960 2024-04-16T16:20:12.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.792+0000 7fabb5e00640 1 --2- 172.21.15.45:0/3007918891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb00af960 0x7fabb00b2aa0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fabb00706b0 tx=0x7faba802e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed20128ff5017f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:12.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.792+0000 7fabafe00640 1 -- 172.21.15.45:0/3007918891 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba802ea00 con 0x7fabb00af960 2024-04-16T16:20:12.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.792+0000 7fabafe00640 1 -- 172.21.15.45:0/3007918891 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba802eba0 con 0x7fabb00af960 2024-04-16T16:20:12.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.792+0000 7fabafe00640 1 -- 172.21.15.45:0/3007918891 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba8009600 con 0x7fabb00af960 2024-04-16T16:20:12.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.793+0000 7fabb7e00640 1 -- 172.21.15.45:0/3007918891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb00af960 msgr2=0x7fabb00b2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:12.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.793+0000 7fabb7e00640 1 --2- 172.21.15.45:0/3007918891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb00af960 0x7fabb00b2aa0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fabb00706b0 tx=0x7faba802e060 comp rx=0 tx=0).stop 2024-04-16T16:20:12.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.793+0000 7fabb7e00640 1 -- 172.21.15.45:0/3007918891 shutdown_connections 2024-04-16T16:20:12.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.793+0000 7fabb7e00640 1 --2- 172.21.15.45:0/3007918891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb00af960 0x7fabb00b2aa0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:12.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.793+0000 7fabb7e00640 1 --2- 172.21.15.45:0/3007918891 >> v2:172.21.15.103:3300/0 conn(0x7fabb015b0d0 0x7fabb00b2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:12.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.793+0000 7fabb7e00640 1 -- 172.21.15.45:0/3007918891 >> 172.21.15.45:0/3007918891 conn(0x7fabb00b3970 msgr2=0x7fabb00b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:12.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.793+0000 7fabb7e00640 1 -- 172.21.15.45:0/3007918891 shutdown_connections 2024-04-16T16:20:12.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.793+0000 7fabb7e00640 1 -- 172.21.15.45:0/3007918891 wait complete. 2024-04-16T16:20:12.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.795+0000 7fabb7e00640 1 Processor -- start 2024-04-16T16:20:12.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.795+0000 7fabb7e00640 1 -- start start 2024-04-16T16:20:12.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.796+0000 7fabb7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabb00af960 0x7fabb0149250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:12.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.796+0000 7fabb7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb00b5bc0 0x7fabb01f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:12.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.796+0000 7fabb7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabb015b0d0 0x7fabb00b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:12.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.796+0000 7fabb7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fabb006e440 con 0x7fabb00b5bc0 2024-04-16T16:20:12.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.796+0000 7fabb7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fabb00b56a0 con 0x7fabb00af960 2024-04-16T16:20:12.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.796+0000 7fabb7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fabb00b0040 con 0x7fabb015b0d0 2024-04-16T16:20:12.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.796+0000 7fabb4a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb00b5bc0 0x7fabb01f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:12.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.796+0000 7fabb4a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb00b5bc0 0x7fabb01f4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60236/0 (socket says 172.21.15.45:60236) 2024-04-16T16:20:12.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.796+0000 7fabb4a00640 1 -- 172.21.15.45:0/2833913999 learned_addr learned my addr 172.21.15.45:0/2833913999 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:12.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.796+0000 7fabb5400640 1 --2- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabb00af960 0x7fabb0149250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:12.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.797+0000 7fabb5e00640 1 --2- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabb015b0d0 0x7fabb00b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:12.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.797+0000 7fabb4a00640 1 -- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabb015b0d0 msgr2=0x7fabb00b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:12.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.797+0000 7fabb4a00640 1 --2- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabb015b0d0 0x7fabb00b3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:12.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.797+0000 7fabb4a00640 1 -- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabb00af960 msgr2=0x7fabb0149250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:12.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.797+0000 7fabb4a00640 1 --2- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabb00af960 0x7fabb0149250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:12.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.797+0000 7fabb4a00640 1 -- 172.21.15.45:0/2833913999 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb015e870 con 0x7fabb00b5bc0 2024-04-16T16:20:12.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.797+0000 7fabb4a00640 1 --2- 172.21.15.45:0/2833913999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb00b5bc0 0x7fabb01f4100 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00c430 tx=0x7fab9c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:12.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.797+0000 7fabad600640 1 -- 172.21.15.45:0/2833913999 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c011790 con 0x7fabb00b5bc0 2024-04-16T16:20:12.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.797+0000 7fabad600640 1 -- 172.21.15.45:0/2833913999 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab9c017070 con 0x7fabb00b5bc0 2024-04-16T16:20:12.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.797+0000 7fabb7e00640 1 -- 172.21.15.45:0/2833913999 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabb015e590 con 0x7fabb00b5bc0 2024-04-16T16:20:12.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.797+0000 7fabad600640 1 -- 172.21.15.45:0/2833913999 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c011b70 con 0x7fabb00b5bc0 2024-04-16T16:20:12.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.797+0000 7fabb7e00640 1 -- 172.21.15.45:0/2833913999 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabb015bcb0 con 0x7fabb00b5bc0 2024-04-16T16:20:12.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.799+0000 7fabb7e00640 1 -- 172.21.15.45:0/2833913999 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabb0075780 con 0x7fabb00b5bc0 2024-04-16T16:20:12.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.799+0000 7fabad600640 1 -- 172.21.15.45:0/2833913999 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) ==== 91086+0+0 (secure 0 0 0) 0x7fab9c011d10 con 0x7fabb00b5bc0 2024-04-16T16:20:12.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.800+0000 7fabad600640 1 --2- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fab9006c9d0 0x7fab9006ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:12.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.800+0000 7fabad600640 1 -- 172.21.15.45:0/2833913999 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 6096+0+0 (secure 0 0 0) 0x7fab9c08bc10 con 0x7fabb00b5bc0 2024-04-16T16:20:12.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.801+0000 7fabb5400640 1 -- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fab9006c9d0 msgr2=0x7fab9006ee90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:12.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.801+0000 7fabb5400640 1 --2- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fab9006c9d0 0x7fab9006ee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:20:12.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:12.805+0000 7fabad600640 1 -- 172.21.15.45:0/2833913999 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fab9c059e70 con 0x7fabb00b5bc0 2024-04-16T16:20:12.960 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:12.960+0000 7fb78902b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:20:13.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.002+0000 7fabb5400640 1 -- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fab9006c9d0 msgr2=0x7fab9006ee90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:13.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.002+0000 7fabb5400640 1 --2- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fab9006c9d0 0x7fab9006ee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:20:13.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.007+0000 7fabb7e00640 1 -- 172.21.15.45:0/2833913999 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fabb00cfb60 con 0x7fabb00b5bc0 2024-04-16T16:20:13.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.007+0000 7fabad600640 1 -- 172.21.15.45:0/2833913999 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) ==== 93+0+10 (secure 0 0 0) 0x7fab9c059770 con 0x7fabb00b5bc0 2024-04-16T16:20:13.007 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:20:13.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.010+0000 7fabb7e00640 1 -- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fab9006c9d0 msgr2=0x7fab9006ee90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:20:13.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.010+0000 7fabb7e00640 1 --2- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fab9006c9d0 0x7fab9006ee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.010+0000 7fabb7e00640 1 -- 172.21.15.45:0/2833913999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb00b5bc0 msgr2=0x7fabb01f4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.010+0000 7fabb7e00640 1 --2- 172.21.15.45:0/2833913999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb00b5bc0 0x7fabb01f4100 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00c430 tx=0x7fab9c00c8f0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.010+0000 7fabb7e00640 1 -- 172.21.15.45:0/2833913999 shutdown_connections 2024-04-16T16:20:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.010+0000 7fabb7e00640 1 --2- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7fab9006c9d0 0x7fab9006ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.010+0000 7fabb7e00640 1 --2- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabb015b0d0 0x7fabb00b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.010+0000 7fabb7e00640 1 --2- 172.21.15.45:0/2833913999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb00b5bc0 0x7fabb01f4100 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.010+0000 7fabb7e00640 1 --2- 172.21.15.45:0/2833913999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabb00af960 0x7fabb0149250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.010+0000 7fabb7e00640 1 -- 172.21.15.45:0/2833913999 >> 172.21.15.45:0/2833913999 conn(0x7fabb00b3970 msgr2=0x7fabb0159b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.010+0000 7fabb7e00640 1 -- 172.21.15.45:0/2833913999 shutdown_connections 2024-04-16T16:20:13.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.010+0000 7fabb7e00640 1 -- 172.21.15.45:0/2833913999 wait complete. 2024-04-16T16:20:13.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:20:13.165 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:13.165+0000 7fddf422d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:20:13.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.243+0000 7f33d2800640 1 Processor -- start 2024-04-16T16:20:13.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.243+0000 7f33d2800640 1 -- start start 2024-04-16T16:20:13.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.245+0000 7f33d2800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33cc15ecc0 0x7f33cc1626a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:13.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.245+0000 7f33d2800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33cc0d2b10 con 0x7f33cc06d620 2024-04-16T16:20:13.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.245+0000 7f33d2800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33cc163210 con 0x7f33cc162be0 2024-04-16T16:20:13.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.245+0000 7f33d2800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f33cc1634f0 con 0x7f33cc15ecc0 2024-04-16T16:20:13.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.245+0000 7f33cbe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f33cc06d620 0x7f33cc0d35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49116/0 (socket says 172.21.15.45:49116) 2024-04-16T16:20:13.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.245+0000 7f33cbe00640 1 -- 172.21.15.45:0/1071097845 learned_addr learned my addr 172.21.15.45:0/1071097845 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:13.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.246+0000 7f33cb400640 1 --2- 172.21.15.45:0/1071097845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33cc15ecc0 0x7f33cc1626a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:13.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.246+0000 7f33ca000640 1 -- 172.21.15.45:0/1071097845 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1176525574 0 0) 0x7f33cc0d2b10 con 0x7f33cc06d620 2024-04-16T16:20:13.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.246+0000 7f33ca000640 1 -- 172.21.15.45:0/1071097845 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33b0003520 con 0x7f33cc06d620 2024-04-16T16:20:13.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.246+0000 7f33ca000640 1 -- 172.21.15.45:0/1071097845 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3119482887 0 0) 0x7f33b0003520 con 0x7f33cc06d620 2024-04-16T16:20:13.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.246+0000 7f33ca000640 1 -- 172.21.15.45:0/1071097845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33cc15ecc0 msgr2=0x7f33cc1626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:13.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.246+0000 7f33ca000640 1 --2- 172.21.15.45:0/1071097845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33cc15ecc0 0x7f33cc1626a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.246+0000 7f33ca000640 1 -- 172.21.15.45:0/1071097845 >> v1:172.21.15.103:6789/0 conn(0x7f33cc162be0 legacy=0x7f33cc06cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:13.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.246+0000 7f33ca000640 1 -- 172.21.15.45:0/1071097845 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33cc0d2dd0 con 0x7f33cc06d620 2024-04-16T16:20:13.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.246+0000 7f33ca000640 1 -- 172.21.15.45:0/1071097845 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f33c40019f0 con 0x7f33cc06d620 2024-04-16T16:20:13.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.246+0000 7f33ca000640 1 -- 172.21.15.45:0/1071097845 >> v1:172.21.15.45:6789/0 conn(0x7f33cc06d620 legacy=0x7f33cc0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:13.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.247+0000 7f33ca000640 1 --2- 172.21.15.45:0/1071097845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33cc15ecc0 0x7f33b00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:13.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.247+0000 7f33ca000640 1 --2- 172.21.15.45:0/1071097845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b0003f20 0x7f33b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:13.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.247+0000 7f33ca000640 1 --2- 172.21.15.45:0/1071097845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33b0004d40 0x7f33b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:13.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.247+0000 7f33ca000640 1 -- 172.21.15.45:0/1071097845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc1634f0 con 0x7f33b0003f20 2024-04-16T16:20:13.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.247+0000 7f33ca000640 1 -- 172.21.15.45:0/1071097845 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33b000ba20 con 0x7f33b0004d40 2024-04-16T16:20:13.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.247+0000 7f33ca000640 1 -- 172.21.15.45:0/1071097845 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f33b000bc20 con 0x7f33cc15ecc0 2024-04-16T16:20:13.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.247+0000 7f33caa00640 1 --2- 172.21.15.45:0/1071097845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b0003f20 0x7f33b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:13.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.247+0000 7f33cb400640 1 --2- 172.21.15.45:0/1071097845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33cc15ecc0 0x7f33b00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:13.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.247+0000 7f33cbe00640 1 --2- 172.21.15.45:0/1071097845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33b0004d40 0x7f33b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:13.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.247+0000 7f33caa00640 1 -- 172.21.15.45:0/1071097845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33cc15ecc0 msgr2=0x7f33b00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:13.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.247+0000 7f33caa00640 1 --2- 172.21.15.45:0/1071097845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33cc15ecc0 0x7f33b00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.247+0000 7f33caa00640 1 -- 172.21.15.45:0/1071097845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33b0004d40 msgr2=0x7f33b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:13.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.247+0000 7f33caa00640 1 --2- 172.21.15.45:0/1071097845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33b0004d40 0x7f33b00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.247+0000 7f33caa00640 1 -- 172.21.15.45:0/1071097845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b0003520 con 0x7f33b0003f20 2024-04-16T16:20:13.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.249+0000 7f33caa00640 1 --2- 172.21.15.45:0/1071097845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b0003f20 0x7f33b0004800 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f33b8001910 tx=0x7f33b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a365767426f44e53 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:13.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.249+0000 7f33ca000640 1 -- 172.21.15.45:0/1071097845 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b8037070 con 0x7f33b0003f20 2024-04-16T16:20:13.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.249+0000 7f33ca000640 1 -- 172.21.15.45:0/1071097845 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33b8032600 con 0x7f33b0003f20 2024-04-16T16:20:13.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.249+0000 7f33ca000640 1 -- 172.21.15.45:0/1071097845 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b80014e0 con 0x7f33b0003f20 2024-04-16T16:20:13.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.250+0000 7f33d2800640 1 -- 172.21.15.45:0/1071097845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b0003f20 msgr2=0x7f33b0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:13.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.250+0000 7f33d2800640 1 --2- 172.21.15.45:0/1071097845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b0003f20 0x7f33b0004800 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f33b8001910 tx=0x7f33b0005830 comp rx=0 tx=0).stop 2024-04-16T16:20:13.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.250+0000 7f33cbe00640 1 -- 172.21.15.45:0/1071097845 reap_dead start 2024-04-16T16:20:13.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.250+0000 7f33d2800640 1 -- 172.21.15.45:0/1071097845 shutdown_connections 2024-04-16T16:20:13.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.250+0000 7f33d2800640 1 -- 172.21.15.45:0/1071097845 >> 172.21.15.45:0/1071097845 conn(0x7f33cc073690 msgr2=0x7f33cc0d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:13.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.250+0000 7f33d2800640 1 -- 172.21.15.45:0/1071097845 shutdown_connections 2024-04-16T16:20:13.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.250+0000 7f33d2800640 1 -- 172.21.15.45:0/1071097845 wait complete. 2024-04-16T16:20:13.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.252+0000 7f33d2800640 1 Processor -- start 2024-04-16T16:20:13.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.252+0000 7f33d2800640 1 -- start start 2024-04-16T16:20:13.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.252+0000 7f33d2800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b0003f20 0x7f33cc152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:13.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33d2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33cc156b50 0x7f33cc0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:13.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33d2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33cc0cbfc0 0x7f33cc0b8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:13.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33d2800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc1626a0 con 0x7f33cc0cbfc0 2024-04-16T16:20:13.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33d2800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc0ba540 con 0x7f33cc156b50 2024-04-16T16:20:13.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33d2800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f33cc162820 con 0x7f33b0003f20 2024-04-16T16:20:13.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33cbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33cc0cbfc0 0x7f33cc0b8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:13.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33cbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33cc0cbfc0 0x7f33cc0b8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60252/0 (socket says 172.21.15.45:60252) 2024-04-16T16:20:13.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33cbe00640 1 -- 172.21.15.45:0/2017670253 learned_addr learned my addr 172.21.15.45:0/2017670253 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:13.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33cb400640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b0003f20 0x7f33cc152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:13.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33caa00640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33cc156b50 0x7f33cc0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:13.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33cbe00640 1 -- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b0003f20 msgr2=0x7f33cc152c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:13.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33cbe00640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b0003f20 0x7f33cc152c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33cbe00640 1 -- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33cc156b50 msgr2=0x7f33cc0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:13.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33cbe00640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33cc156b50 0x7f33cc0cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33cbe00640 1 -- 172.21.15.45:0/2017670253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33cc1531c0 con 0x7f33cc0cbfc0 2024-04-16T16:20:13.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33cb400640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b0003f20 0x7f33cc152c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:13.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.253+0000 7f33caa00640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33cc156b50 0x7f33cc0cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:13.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.254+0000 7f33cbe00640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33cc0cbfc0 0x7f33cc0b8bf0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f33cc06e000 tx=0x7f33c400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:13.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.254+0000 7f33c3400640 1 -- 172.21.15.45:0/2017670253 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c4004d60 con 0x7f33cc0cbfc0 2024-04-16T16:20:13.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.254+0000 7f33c3400640 1 -- 172.21.15.45:0/2017670253 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33c400bb20 con 0x7f33cc0cbfc0 2024-04-16T16:20:13.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.254+0000 7f33c3400640 1 -- 172.21.15.45:0/2017670253 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c4007070 con 0x7f33cc0cbfc0 2024-04-16T16:20:13.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.254+0000 7f33d2800640 1 -- 172.21.15.45:0/2017670253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33cc0cc540 con 0x7f33cc0cbfc0 2024-04-16T16:20:13.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.254+0000 7f33d2800640 1 -- 172.21.15.45:0/2017670253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33cc0ccb00 con 0x7f33cc0cbfc0 2024-04-16T16:20:13.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.255+0000 7f33d2800640 1 -- 172.21.15.45:0/2017670253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33cc075780 con 0x7f33cc0cbfc0 2024-04-16T16:20:13.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.256+0000 7f33c3400640 1 -- 172.21.15.45:0/2017670253 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) ==== 91086+0+0 (secure 0 0 0) 0x7f33c401b020 con 0x7f33cc0cbfc0 2024-04-16T16:20:13.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.257+0000 7f33c3400640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f33ac06c980 0x7f33ac06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:13.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.257+0000 7f33c3400640 1 -- 172.21.15.45:0/2017670253 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 6096+0+0 (secure 0 0 0) 0x7f33c408c070 con 0x7f33cc0cbfc0 2024-04-16T16:20:13.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.257+0000 7f33cb400640 1 -- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f33ac06c980 msgr2=0x7f33ac06ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:13.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.257+0000 7f33cb400640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f33ac06c980 0x7f33ac06ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:20:13.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.262+0000 7f33c3400640 1 -- 172.21.15.45:0/2017670253 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f33c4059af0 con 0x7f33cc0cbfc0 2024-04-16T16:20:13.269 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:13.269+0000 7fb78902b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:20:13.326 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:13.327+0000 7fb78902b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:20:13.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.458+0000 7f33cb400640 1 -- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f33ac06c980 msgr2=0x7f33ac06ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1934318167 2024-04-16T16:20:13.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.458+0000 7f33cb400640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f33ac06c980 0x7f33ac06ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:20:13.458 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:13.459+0000 7fb78902b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:20:13.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.465+0000 7f33d2800640 1 -- 172.21.15.45:0/2017670253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f33cc0d26c0 con 0x7f33cc0cbfc0 2024-04-16T16:20:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.466+0000 7f33c3400640 1 -- 172.21.15.45:0/2017670253 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) ==== 104+0+11 (secure 0 0 0) 0x7f33c405ddf0 con 0x7f33cc0cbfc0 2024-04-16T16:20:13.466 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.469+0000 7f33d2800640 1 -- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f33ac06c980 msgr2=0x7f33ac06ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.469+0000 7f33d2800640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f33ac06c980 0x7f33ac06ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.469+0000 7f33d2800640 1 -- 172.21.15.45:0/2017670253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33cc0cbfc0 msgr2=0x7f33cc0b8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.469+0000 7f33d2800640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33cc0cbfc0 0x7f33cc0b8bf0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f33cc06e000 tx=0x7f33c400b4a0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.469+0000 7f33d2800640 1 -- 172.21.15.45:0/2017670253 shutdown_connections 2024-04-16T16:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.469+0000 7f33d2800640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:6832/1934318167,v1:172.21.15.103:6833/1934318167] conn(0x7f33ac06c980 0x7f33ac06ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.469+0000 7f33d2800640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33cc0cbfc0 0x7f33cc0b8bf0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.469+0000 7f33d2800640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33cc156b50 0x7f33cc0cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.469+0000 7f33d2800640 1 --2- 172.21.15.45:0/2017670253 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33b0003f20 0x7f33cc152c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.469+0000 7f33d2800640 1 -- 172.21.15.45:0/2017670253 >> 172.21.15.45:0/2017670253 conn(0x7f33cc073690 msgr2=0x7f33cc0cfa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.469+0000 7f33d2800640 1 -- 172.21.15.45:0/2017670253 shutdown_connections 2024-04-16T16:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.469+0000 7f33d2800640 1 -- 172.21.15.45:0/2017670253 wait complete. 2024-04-16T16:20:13.493 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:20:13.501 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:13.501+0000 7fddf422d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:20:13.556 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:13.556+0000 7fb702600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:13.556 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:13.556+0000 7fb702600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:13.556 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:13.556+0000 7fb702600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:13.556 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:13.556+0000 7fb702600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:13.556 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:13.556+0000 7fb702600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:13.556 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:13.557+0000 7fb706200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:13.556 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:13.557+0000 7fb706200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:13.557 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:13.557+0000 7fb706200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:13.557 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:13.557+0000 7fb706200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:13.557 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:13.557+0000 7fb706200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:13.567 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:13.568+0000 7fddf422d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:20:13.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.687+0000 7f2717200640 1 Processor -- start 2024-04-16T16:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.687+0000 7f2717200640 1 -- start start 2024-04-16T16:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.689+0000 7f2717200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f27180b6310 0x7f2718161db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.689+0000 7f2717200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27180b6810 con 0x7f27180d18f0 2024-04-16T16:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.689+0000 7f2717200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27180b6af0 con 0x7f27180d3e90 2024-04-16T16:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.689+0000 7f2717200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f27180d29f0 con 0x7f27180b6310 2024-04-16T16:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.689+0000 7f2715e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f27180d18f0 0x7f27180ba220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33554/0 (socket says 172.21.15.45:33554) 2024-04-16T16:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.689+0000 7f2715e00640 1 -- 172.21.15.45:0/2438722543 learned_addr learned my addr 172.21.15.45:0/2438722543 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f270fe00640 1 --2- 172.21.15.45:0/2438722543 >> v2:172.21.15.103:3300/0 conn(0x7f27180b6310 0x7f2718161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f2715400640 1 -- 172.21.15.45:0/2438722543 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 408423128 0 0) 0x7f27180b6810 con 0x7f27180d18f0 2024-04-16T16:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f2715400640 1 -- 172.21.15.45:0/2438722543 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26fc003520 con 0x7f27180d18f0 2024-04-16T16:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f2715400640 1 -- 172.21.15.45:0/2438722543 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3037187100 0 0) 0x7f26fc003520 con 0x7f27180d18f0 2024-04-16T16:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f2715400640 1 -- 172.21.15.45:0/2438722543 >> v2:172.21.15.103:3300/0 conn(0x7f27180b6310 msgr2=0x7f2718161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f2715400640 1 --2- 172.21.15.45:0/2438722543 >> v2:172.21.15.103:3300/0 conn(0x7f27180b6310 0x7f2718161db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f2715400640 1 -- 172.21.15.45:0/2438722543 >> v1:172.21.15.103:6789/0 conn(0x7f27180d3e90 legacy=0x7f27181622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f2715400640 1 -- 172.21.15.45:0/2438722543 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27180b6dd0 con 0x7f27180d18f0 2024-04-16T16:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f2715400640 1 -- 172.21.15.45:0/2438722543 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f2708001670 con 0x7f27180d18f0 2024-04-16T16:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f2715400640 1 -- 172.21.15.45:0/2438722543 >> v1:172.21.15.45:6789/0 conn(0x7f27180d18f0 legacy=0x7f27180ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f2715400640 1 --2- 172.21.15.45:0/2438722543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26fc0039b0 0x7f26fc003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f2715400640 1 --2- 172.21.15.45:0/2438722543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 0x7f26fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f2715400640 1 --2- 172.21.15.45:0/2438722543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26fc005210 0x7f26fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f2715400640 1 -- 172.21.15.45:0/2438722543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f27180d29f0 con 0x7f26fc004800 2024-04-16T16:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f2715400640 1 -- 172.21.15.45:0/2438722543 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc00c140 con 0x7f26fc0039b0 2024-04-16T16:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.690+0000 7f2715400640 1 -- 172.21.15.45:0/2438722543 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f26fc00c340 con 0x7f26fc005210 2024-04-16T16:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.691+0000 7f270fe00640 1 --2- 172.21.15.45:0/2438722543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 0x7f26fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.691+0000 7f2715e00640 1 --2- 172.21.15.45:0/2438722543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26fc0039b0 0x7f26fc003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.691+0000 7f270fe00640 1 -- 172.21.15.45:0/2438722543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26fc005210 msgr2=0x7f26fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.691+0000 7f270fe00640 1 --2- 172.21.15.45:0/2438722543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26fc005210 0x7f26fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.691+0000 7f270fe00640 1 -- 172.21.15.45:0/2438722543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26fc0039b0 msgr2=0x7f26fc003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.691+0000 7f270fe00640 1 --2- 172.21.15.45:0/2438722543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26fc0039b0 0x7f26fc003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.691+0000 7f270fe00640 1 -- 172.21.15.45:0/2438722543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26fc003520 con 0x7f26fc004800 2024-04-16T16:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.691+0000 7f2715e00640 1 --2- 172.21.15.45:0/2438722543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26fc0039b0 0x7f26fc003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.691+0000 7f2716800640 1 -- 172.21.15.45:0/2438722543 reap_dead start 2024-04-16T16:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.692+0000 7f270fe00640 1 --2- 172.21.15.45:0/2438722543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 0x7f26fc004c80 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f2700009a90 tx=0x7f26fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff7ac887ae52f2e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.692+0000 7f2715400640 1 -- 172.21.15.45:0/2438722543 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2700039070 con 0x7f26fc004800 2024-04-16T16:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.692+0000 7f2715400640 1 -- 172.21.15.45:0/2438722543 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f270002c670 con 0x7f26fc004800 2024-04-16T16:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.692+0000 7f2715400640 1 -- 172.21.15.45:0/2438722543 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2700009690 con 0x7f26fc004800 2024-04-16T16:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.693+0000 7f2717200640 1 -- 172.21.15.45:0/2438722543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 msgr2=0x7f26fc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.693+0000 7f2717200640 1 --2- 172.21.15.45:0/2438722543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 0x7f26fc004c80 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f2700009a90 tx=0x7f26fc005830 comp rx=0 tx=0).stop 2024-04-16T16:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.693+0000 7f2717200640 1 -- 172.21.15.45:0/2438722543 shutdown_connections 2024-04-16T16:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.693+0000 7f2717200640 1 --2- 172.21.15.45:0/2438722543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 0x7f26fc004c80 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.693+0000 7f2717200640 1 -- 172.21.15.45:0/2438722543 >> 172.21.15.45:0/2438722543 conn(0x7f271806cf90 msgr2=0x7f271806d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.693+0000 7f2717200640 1 -- 172.21.15.45:0/2438722543 shutdown_connections 2024-04-16T16:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.693+0000 7f2717200640 1 -- 172.21.15.45:0/2438722543 wait complete. 2024-04-16T16:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.695+0000 7f2717200640 1 Processor -- start 2024-04-16T16:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.695+0000 7f2717200640 1 -- start start 2024-04-16T16:20:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.696+0000 7f2717200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 0x7f27180ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.696+0000 7f2717200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f27180c8380 0x7f27181f4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.696+0000 7f2717200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27180c8b60 0x7f27180c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.696+0000 7f2717200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f27180d1e50 con 0x7f26fc004800 2024-04-16T16:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.696+0000 7f2717200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f27180d1cd0 con 0x7f27180c8b60 2024-04-16T16:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.696+0000 7f2717200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f27180d1fd0 con 0x7f27180c8380 2024-04-16T16:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.696+0000 7f2715e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 0x7f27180ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.696+0000 7f2715e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 0x7f27180ca510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38592/0 (socket says 172.21.15.45:38592) 2024-04-16T16:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.696+0000 7f2715e00640 1 -- 172.21.15.45:0/3298851070 learned_addr learned my addr 172.21.15.45:0/3298851070 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.697+0000 7f270fe00640 1 --2- 172.21.15.45:0/3298851070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f27180c8380 0x7f27181f4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.697+0000 7f2716800640 1 --2- 172.21.15.45:0/3298851070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27180c8b60 0x7f27180c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.697+0000 7f2715e00640 1 -- 172.21.15.45:0/3298851070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f27180c8380 msgr2=0x7f27181f4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.697+0000 7f2715e00640 1 --2- 172.21.15.45:0/3298851070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f27180c8380 0x7f27181f4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.697+0000 7f2715e00640 1 -- 172.21.15.45:0/3298851070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27180c8b60 msgr2=0x7f27180c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:13.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.697+0000 7f2715e00640 1 --2- 172.21.15.45:0/3298851070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27180c8b60 0x7f27180c8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.697+0000 7f2715e00640 1 -- 172.21.15.45:0/3298851070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27180cac00 con 0x7f26fc004800 2024-04-16T16:20:13.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.697+0000 7f2715e00640 1 --2- 172.21.15.45:0/3298851070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 0x7f27180ca510 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f270800d8f0 tx=0x7f270800ddb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:13.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.697+0000 7f270e000640 1 -- 172.21.15.45:0/3298851070 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708019070 con 0x7f26fc004800 2024-04-16T16:20:13.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.698+0000 7f270e000640 1 -- 172.21.15.45:0/3298851070 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27080074e0 con 0x7f26fc004800 2024-04-16T16:20:13.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.698+0000 7f2717200640 1 -- 172.21.15.45:0/3298851070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27181f5710 con 0x7f26fc004800 2024-04-16T16:20:13.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.698+0000 7f270e000640 1 -- 172.21.15.45:0/3298851070 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27080049e0 con 0x7f26fc004800 2024-04-16T16:20:13.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.698+0000 7f2717200640 1 -- 172.21.15.45:0/3298851070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27181f4890 con 0x7f26fc004800 2024-04-16T16:20:13.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.699+0000 7f2717200640 1 -- 172.21.15.45:0/3298851070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27180d10d0 con 0x7f26fc004800 2024-04-16T16:20:13.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.700+0000 7f270e000640 1 -- 172.21.15.45:0/3298851070 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 46) ==== 90751+0+0 (secure 0 0 0) 0x7f270801c070 con 0x7f26fc004800 2024-04-16T16:20:13.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.701+0000 7f270e000640 1 -- 172.21.15.45:0/3298851070 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6397+0+0 (secure 0 0 0) 0x7f270808b440 con 0x7f26fc004800 2024-04-16T16:20:13.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.705+0000 7f270e000640 1 -- 172.21.15.45:0/3298851070 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f27180d10d0 con 0x7f26fc004800 2024-04-16T16:20:13.712 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:13.713+0000 7fddf422d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:20:13.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.909+0000 7f2717200640 1 -- 172.21.15.45:0/3298851070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2718075780 con 0x7f26fc004800 2024-04-16T16:20:13.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.910+0000 7f270e000640 1 -- 172.21.15.45:0/3298851070 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v117) ==== 100+0+11 (secure 0 0 0) 0x7f2708059ad0 con 0x7f26fc004800 2024-04-16T16:20:13.911 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:20:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.913+0000 7f2717200640 1 -- 172.21.15.45:0/3298851070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 msgr2=0x7f27180ca510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.913+0000 7f2717200640 1 --2- 172.21.15.45:0/3298851070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 0x7f27180ca510 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f270800d8f0 tx=0x7f270800ddb0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.913+0000 7f2717200640 1 -- 172.21.15.45:0/3298851070 shutdown_connections 2024-04-16T16:20:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.913+0000 7f2717200640 1 --2- 172.21.15.45:0/3298851070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27180c8b60 0x7f27180c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.913+0000 7f2717200640 1 --2- 172.21.15.45:0/3298851070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f27180c8380 0x7f27181f4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.913+0000 7f2717200640 1 --2- 172.21.15.45:0/3298851070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 0x7f27180ca510 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:13.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.913+0000 7f2717200640 1 -- 172.21.15.45:0/3298851070 >> 172.21.15.45:0/3298851070 conn(0x7f271806cf90 msgr2=0x7f27180ba1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:13.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.914+0000 7f2717200640 1 -- 172.21.15.45:0/3298851070 shutdown_connections 2024-04-16T16:20:13.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:13.914+0000 7f2717200640 1 -- 172.21.15.45:0/3298851070 wait complete. 2024-04-16T16:20:13.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:20:14.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.193+0000 7f5897c00640 1 Processor -- start 2024-04-16T16:20:14.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.194+0000 7f5897c00640 1 -- start start 2024-04-16T16:20:14.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.195+0000 7f5897c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5890072890 0x7f58900b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:14.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.195+0000 7f5897c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58900b58e0 0x7f58900b3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:14.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.195+0000 7f5897c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f589006df00 con 0x7f5890147550 2024-04-16T16:20:14.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.195+0000 7f5897c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f58900b5d60 con 0x7f5890072890 2024-04-16T16:20:14.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.195+0000 7f5897c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f589006fc50 con 0x7f58900b58e0 2024-04-16T16:20:14.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.196+0000 7f5895c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58900b58e0 0x7f58900b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:14.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.196+0000 7f5895c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58900b58e0 0x7f58900b3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:44296/0 (socket says 172.21.15.45:44296) 2024-04-16T16:20:14.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.196+0000 7f5895c00640 1 -- 172.21.15.45:0/3177089155 learned_addr learned my addr 172.21.15.45:0/3177089155 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:14.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.196+0000 7f588fe00640 1 --2- 172.21.15.45:0/3177089155 >> v2:172.21.15.103:3300/0 conn(0x7f5890072890 0x7f58900b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:14.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.197+0000 7f5895c00640 1 -- 172.21.15.45:0/3177089155 >> v2:172.21.15.103:3300/0 conn(0x7f5890072890 msgr2=0x7f58900b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:14.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.197+0000 7f5895c00640 1 --2- 172.21.15.45:0/3177089155 >> v2:172.21.15.103:3300/0 conn(0x7f5890072890 0x7f58900b53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:14.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.197+0000 7f5895c00640 1 -- 172.21.15.45:0/3177089155 >> v1:172.21.15.103:6789/0 conn(0x7f5890147550 legacy=0x7f5890072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:14.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.197+0000 7f5895c00640 1 -- 172.21.15.45:0/3177089155 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f589006e1e0 con 0x7f58900b58e0 2024-04-16T16:20:14.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.197+0000 7f588fe00640 1 --2- 172.21.15.45:0/3177089155 >> v2:172.21.15.103:3300/0 conn(0x7f5890072890 0x7f58900b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:20:14.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.198+0000 7f5895c00640 1 --2- 172.21.15.45:0/3177089155 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58900b58e0 0x7f58900b3890 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5888009e50 tx=0x7f588802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbe7924270cbb05e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:14.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.199+0000 7f588f400640 1 -- 172.21.15.45:0/3177089155 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588802cb00 con 0x7f58900b58e0 2024-04-16T16:20:14.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.199+0000 7f588f400640 1 -- 172.21.15.45:0/3177089155 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f588802cca0 con 0x7f58900b58e0 2024-04-16T16:20:14.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.199+0000 7f588f400640 1 -- 172.21.15.45:0/3177089155 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5888034610 con 0x7f58900b58e0 2024-04-16T16:20:14.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.200+0000 7f5897c00640 1 -- 172.21.15.45:0/3177089155 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58900b58e0 msgr2=0x7f58900b3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:14.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.200+0000 7f5897c00640 1 --2- 172.21.15.45:0/3177089155 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58900b58e0 0x7f58900b3890 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5888009e50 tx=0x7f588802bf60 comp rx=0 tx=0).stop 2024-04-16T16:20:14.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.200+0000 7f5897c00640 1 -- 172.21.15.45:0/3177089155 shutdown_connections 2024-04-16T16:20:14.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.200+0000 7f5897c00640 1 --2- 172.21.15.45:0/3177089155 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58900b58e0 0x7f58900b3890 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:14.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.200+0000 7f5897c00640 1 --2- 172.21.15.45:0/3177089155 >> v2:172.21.15.103:3300/0 conn(0x7f5890072890 0x7f58900b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:14.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.200+0000 7f5897c00640 1 -- 172.21.15.45:0/3177089155 >> 172.21.15.45:0/3177089155 conn(0x7f589006b930 msgr2=0x7f589006bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:14.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.200+0000 7f5897c00640 1 -- 172.21.15.45:0/3177089155 shutdown_connections 2024-04-16T16:20:14.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.200+0000 7f5897c00640 1 -- 172.21.15.45:0/3177089155 wait complete. 2024-04-16T16:20:14.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.202+0000 7f5897c00640 1 Processor -- start 2024-04-16T16:20:14.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.202+0000 7f5897c00640 1 -- start start 2024-04-16T16:20:14.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.204+0000 7f5897c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5890072890 0x7f58900af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:14.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.204+0000 7f5897c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58900b58e0 0x7f589013fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:14.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.204+0000 7f5897c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5890147550 0x7f58901509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:14.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.204+0000 7f5897c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f589006ea80 con 0x7f58900b58e0 2024-04-16T16:20:14.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.204+0000 7f5897c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f589006e7a0 con 0x7f5890147550 2024-04-16T16:20:14.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.204+0000 7f5897c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f589006efa0 con 0x7f5890072890 2024-04-16T16:20:14.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.204+0000 7f588fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58900b58e0 0x7f589013fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:14.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.204+0000 7f588fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58900b58e0 0x7f589013fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38604/0 (socket says 172.21.15.45:38604) 2024-04-16T16:20:14.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.204+0000 7f588fe00640 1 -- 172.21.15.45:0/2876135361 learned_addr learned my addr 172.21.15.45:0/2876135361 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:14.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.204+0000 7f5895200640 1 --2- 172.21.15.45:0/2876135361 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5890072890 0x7f58900af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:14.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.204+0000 7f5895c00640 1 --2- 172.21.15.45:0/2876135361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5890147550 0x7f58901509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:14.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.204+0000 7f588fe00640 1 -- 172.21.15.45:0/2876135361 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5890072890 msgr2=0x7f58900af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:14.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.204+0000 7f588fe00640 1 --2- 172.21.15.45:0/2876135361 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5890072890 0x7f58900af8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:14.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.205+0000 7f588fe00640 1 -- 172.21.15.45:0/2876135361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5890147550 msgr2=0x7f58901509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:14.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.205+0000 7f588fe00640 1 --2- 172.21.15.45:0/2876135361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5890147550 0x7f58901509b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:14.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.205+0000 7f588fe00640 1 -- 172.21.15.45:0/2876135361 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58901408b0 con 0x7f58900b58e0 2024-04-16T16:20:14.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.205+0000 7f588fe00640 1 --2- 172.21.15.45:0/2876135361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58900b58e0 0x7f589013fdc0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f587c00e9b0 tx=0x7f587c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:14.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.205+0000 7f588cc00640 1 -- 172.21.15.45:0/2876135361 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587c017070 con 0x7f58900b58e0 2024-04-16T16:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.205+0000 7f588cc00640 1 -- 172.21.15.45:0/2876135361 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f587c00ca00 con 0x7f58900b58e0 2024-04-16T16:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.205+0000 7f588cc00640 1 -- 172.21.15.45:0/2876135361 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587c00cdf0 con 0x7f58900b58e0 2024-04-16T16:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.205+0000 7f5897c00640 1 -- 172.21.15.45:0/2876135361 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58901e84b0 con 0x7f58900b58e0 2024-04-16T16:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.205+0000 7f5897c00640 1 -- 172.21.15.45:0/2876135361 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58901e8a70 con 0x7f58900b58e0 2024-04-16T16:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.206+0000 7f5897c00640 1 -- 172.21.15.45:0/2876135361 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5890075780 con 0x7f58900b58e0 2024-04-16T16:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.207+0000 7f588cc00640 1 -- 172.21.15.45:0/2876135361 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 46) ==== 90751+0+0 (secure 0 0 0) 0x7f587c01a070 con 0x7f58900b58e0 2024-04-16T16:20:14.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.208+0000 7f588cc00640 1 -- 172.21.15.45:0/2876135361 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6397+0+0 (secure 0 0 0) 0x7f587c08b9a0 con 0x7f58900b58e0 2024-04-16T16:20:14.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.212+0000 7f588cc00640 1 -- 172.21.15.45:0/2876135361 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f587c059df0 con 0x7f58900b58e0 2024-04-16T16:20:14.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.420+0000 7f5897c00640 1 -- 172.21.15.45:0/2876135361 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f589006f4b0 con 0x7f58900b58e0 2024-04-16T16:20:14.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.421+0000 7f588cc00640 1 -- 172.21.15.45:0/2876135361 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v117) ==== 81+0+952 (secure 0 0 0) 0x7f587c0596f0 con 0x7f58900b58e0 2024-04-16T16:20:14.421 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/383252954 2024-04-17T16:20:13.464929+0000 2024-04-16T16:20:14.421 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/428120997 2024-04-17T16:20:13.464929+0000 2024-04-16T16:20:14.421 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3229718691 2024-04-17T16:20:13.464929+0000 2024-04-16T16:20:14.421 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/1934318167 2024-04-17T16:20:13.464929+0000 2024-04-16T16:20:14.421 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3469935364 2024-04-17T16:20:13.464929+0000 2024-04-16T16:20:14.421 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3506765807 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:14.421 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/1934318167 2024-04-17T16:20:13.464929+0000 2024-04-16T16:20:14.421 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3547880684 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:14.421 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/2293932214 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:14.422 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1890933611 2024-04-17T16:20:13.464929+0000 2024-04-16T16:20:14.422 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/2293932214 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:14.422 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1347839066 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:14.422 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/546409297 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:14.422 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3566291251 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:14.422 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/714185246 2024-04-17T16:20:07.040813+0000 2024-04-16T16:20:14.422 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2784388736 2024-04-16T17:19:54.469649+0000 2024-04-16T16:20:14.422 INFO:teuthology.orchestra.run.smithi045.stderr:listed 16 entries 2024-04-16T16:20:14.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.423+0000 7f5897c00640 1 -- 172.21.15.45:0/2876135361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58900b58e0 msgr2=0x7f589013fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:14.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.423+0000 7f5897c00640 1 --2- 172.21.15.45:0/2876135361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58900b58e0 0x7f589013fdc0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f587c00e9b0 tx=0x7f587c00ee70 comp rx=0 tx=0).stop 2024-04-16T16:20:14.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.424+0000 7f5897c00640 1 -- 172.21.15.45:0/2876135361 shutdown_connections 2024-04-16T16:20:14.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.424+0000 7f5897c00640 1 --2- 172.21.15.45:0/2876135361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5890147550 0x7f58901509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:14.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.424+0000 7f5897c00640 1 --2- 172.21.15.45:0/2876135361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58900b58e0 0x7f589013fdc0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:14.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.424+0000 7f5897c00640 1 --2- 172.21.15.45:0/2876135361 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5890072890 0x7f58900af8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:14.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.424+0000 7f5897c00640 1 -- 172.21.15.45:0/2876135361 >> 172.21.15.45:0/2876135361 conn(0x7f589006b930 msgr2=0x7f58900b1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:14.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.424+0000 7f5897c00640 1 -- 172.21.15.45:0/2876135361 shutdown_connections 2024-04-16T16:20:14.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.424+0000 7f5897c00640 1 -- 172.21.15.45:0/2876135361 wait complete. 2024-04-16T16:20:14.440 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-16T16:20:14.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:14.440 DEBUG:teuthology.orchestra.run.smithi103:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-04-16T16:20:14.549 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:20:14.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:14.550 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-16T16:20:14.596 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:20:14.596 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:20:14.596 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:20:14.656 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:20:14.656 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:20:14.656 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:20:14.656 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:20:14.711 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:20:14.711 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:20:14.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.939+0000 7f9587e00640 1 Processor -- start 2024-04-16T16:20:14.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.939+0000 7f9587e00640 1 -- start start 2024-04-16T16:20:14.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.941+0000 7f9587e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9588146220 0x7f958815ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:14.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.941+0000 7f9587e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f95881608a0 0x7f9588160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:14.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.941+0000 7f9587e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f958815d360 con 0x7f958806f7a0 2024-04-16T16:20:14.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.941+0000 7f9587e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9588161230 con 0x7f95881608a0 2024-04-16T16:20:14.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.941+0000 7f9587e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9588160410 con 0x7f9588146220 2024-04-16T16:20:14.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.941+0000 7f957e000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f958806f7a0 0x7f958806df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33570/0 (socket says 172.21.15.45:33570) 2024-04-16T16:20:14.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.941+0000 7f957e000640 1 -- 172.21.15.45:0/2158808944 learned_addr learned my addr 172.21.15.45:0/2158808944 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:14.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.941+0000 7f9587400640 1 --2- 172.21.15.45:0/2158808944 >> v2:172.21.15.45:3300/0 conn(0x7f95881608a0 0x7f9588160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:14.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.941+0000 7f9586a00640 1 --2- 172.21.15.45:0/2158808944 >> v2:172.21.15.103:3300/0 conn(0x7f9588146220 0x7f958815ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:14.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.942+0000 7f9587400640 1 -- 172.21.15.45:0/2158808944 >> v2:172.21.15.103:3300/0 conn(0x7f9588146220 msgr2=0x7f958815ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:14.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.942+0000 7f9587400640 1 --2- 172.21.15.45:0/2158808944 >> v2:172.21.15.103:3300/0 conn(0x7f9588146220 0x7f958815ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:14.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.942+0000 7f9587400640 1 -- 172.21.15.45:0/2158808944 >> v1:172.21.15.45:6789/0 conn(0x7f958806f7a0 legacy=0x7f958806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:14.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.942+0000 7f9587400640 1 -- 172.21.15.45:0/2158808944 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958815d640 con 0x7f95881608a0 2024-04-16T16:20:14.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.942+0000 7f9586000640 1 -- 172.21.15.45:0/2158808944 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2930423665 0 0) 0x7f958815d360 con 0x7f958806f7a0 2024-04-16T16:20:14.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.942+0000 7f9586a00640 1 --2- 172.21.15.45:0/2158808944 >> v2:172.21.15.103:3300/0 conn(0x7f9588146220 0x7f958815ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:14.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.943+0000 7f9587400640 1 --2- 172.21.15.45:0/2158808944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95881608a0 0x7f9588160cf0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f9588070210 tx=0x7f958002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64978b52a01b2cc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:14.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.943+0000 7f9586000640 1 -- 172.21.15.45:0/2158808944 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580039070 con 0x7f95881608a0 2024-04-16T16:20:14.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.943+0000 7f9586000640 1 -- 172.21.15.45:0/2158808944 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f958002cc80 con 0x7f95881608a0 2024-04-16T16:20:14.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.943+0000 7f9586000640 1 -- 172.21.15.45:0/2158808944 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580034660 con 0x7f95881608a0 2024-04-16T16:20:14.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.944+0000 7f9587e00640 1 -- 172.21.15.45:0/2158808944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95881608a0 msgr2=0x7f9588160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:14.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.944+0000 7f9587e00640 1 --2- 172.21.15.45:0/2158808944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95881608a0 0x7f9588160cf0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f9588070210 tx=0x7f958002bed0 comp rx=0 tx=0).stop 2024-04-16T16:20:14.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.944+0000 7f9587e00640 1 -- 172.21.15.45:0/2158808944 shutdown_connections 2024-04-16T16:20:14.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.944+0000 7f9587e00640 1 --2- 172.21.15.45:0/2158808944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95881608a0 0x7f9588160cf0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:14.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.944+0000 7f9587e00640 1 --2- 172.21.15.45:0/2158808944 >> v2:172.21.15.103:3300/0 conn(0x7f9588146220 0x7f958815ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:14.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.944+0000 7f9587e00640 1 -- 172.21.15.45:0/2158808944 >> 172.21.15.45:0/2158808944 conn(0x7f958806b930 msgr2=0x7f958806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:14.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.944+0000 7f9587e00640 1 -- 172.21.15.45:0/2158808944 shutdown_connections 2024-04-16T16:20:14.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.944+0000 7f9587e00640 1 -- 172.21.15.45:0/2158808944 wait complete. 2024-04-16T16:20:14.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.946+0000 7f9587e00640 1 Processor -- start 2024-04-16T16:20:14.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.946+0000 7f9587e00640 1 -- start start 2024-04-16T16:20:14.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f9587e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f958806f7a0 0x7f95880ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:14.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f9587e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9588146220 0x7f958815c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:14.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f9587e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95881608a0 0x7f95881f4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f9587e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f958815dc00 con 0x7f9588146220 2024-04-16T16:20:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f9587e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f958806e650 con 0x7f958806f7a0 2024-04-16T16:20:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f9587e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f958806fc20 con 0x7f95881608a0 2024-04-16T16:20:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f957e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9588146220 0x7f958815c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f957e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9588146220 0x7f958815c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38626/0 (socket says 172.21.15.45:38626) 2024-04-16T16:20:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f957e000640 1 -- 172.21.15.45:0/1582311072 learned_addr learned my addr 172.21.15.45:0/1582311072 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f9586a00640 1 --2- 172.21.15.45:0/1582311072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f958806f7a0 0x7f95880ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f957e000640 1 -- 172.21.15.45:0/1582311072 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95881608a0 msgr2=0x7f95881f4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f957e000640 1 --2- 172.21.15.45:0/1582311072 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95881608a0 0x7f95881f4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f957e000640 1 -- 172.21.15.45:0/1582311072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f958806f7a0 msgr2=0x7f95880ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f957e000640 1 --2- 172.21.15.45:0/1582311072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f958806f7a0 0x7f95880ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f957e000640 1 -- 172.21.15.45:0/1582311072 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95881f51d0 con 0x7f9588146220 2024-04-16T16:20:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.947+0000 7f9586a00640 1 --2- 172.21.15.45:0/1582311072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f958806f7a0 0x7f95880ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.948+0000 7f957e000640 1 --2- 172.21.15.45:0/1582311072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9588146220 0x7f958815c2f0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f957000c4d0 tx=0x7f957000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:14.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.948+0000 7f957fe00640 1 -- 172.21.15.45:0/1582311072 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9570018070 con 0x7f9588146220 2024-04-16T16:20:14.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.948+0000 7f957fe00640 1 -- 172.21.15.45:0/1582311072 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9570007500 con 0x7f9588146220 2024-04-16T16:20:14.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.948+0000 7f957fe00640 1 -- 172.21.15.45:0/1582311072 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9570004a20 con 0x7f9588146220 2024-04-16T16:20:14.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.948+0000 7f9587e00640 1 -- 172.21.15.45:0/1582311072 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95881f7800 con 0x7f9588146220 2024-04-16T16:20:14.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.948+0000 7f9587e00640 1 -- 172.21.15.45:0/1582311072 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95881f7d80 con 0x7f9588146220 2024-04-16T16:20:14.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.949+0000 7f9587e00640 1 -- 172.21.15.45:0/1582311072 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95881f8720 con 0x7f9588146220 2024-04-16T16:20:14.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.950+0000 7f957fe00640 1 -- 172.21.15.45:0/1582311072 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 47) ==== 90878+0+0 (secure 0 0 0) 0x7f957001b070 con 0x7f9588146220 2024-04-16T16:20:14.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.951+0000 7f957fe00640 1 --2- 172.21.15.45:0/1582311072 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f956406c740 0x7f956406ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:14.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.951+0000 7f957fe00640 1 -- 172.21.15.45:0/1582311072 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6397+0+0 (secure 0 0 0) 0x7f957008ad40 con 0x7f9588146220 2024-04-16T16:20:14.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.952+0000 7f9586a00640 1 --2- 172.21.15.45:0/1582311072 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f956406c740 0x7f956406ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:14.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.953+0000 7f9586a00640 1 --2- 172.21.15.45:0/1582311072 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f956406c740 0x7f956406ec00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f958815ff70 tx=0x7f9578008040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:14.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:14.956+0000 7f957fe00640 1 -- 172.21.15.45:0/1582311072 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f95881f8720 con 0x7f9588146220 2024-04-16T16:20:15.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.201+0000 7f9587e00640 1 -- 172.21.15.45:0/1582311072 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9588075780 con 0x7f9588146220 2024-04-16T16:20:15.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.202+0000 7f957fe00640 1 -- 172.21.15.45:0/1582311072 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5186 (secure 0 0 0) 0x7f957008e030 con 0x7f9588146220 2024-04-16T16:20:15.203 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:15.203 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6056,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1032791151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":1032791151},{"type":"v1","addr":"172.21.15.45:6835","nonce":1032791151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6062,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/498835619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":498835619},{"type":"v1","addr":"172.21.15.103:6837","nonce":498835619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/3611628422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3611628422},{"type":"v1","addr":"172.21.15.103:6836","nonce":3611628422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:19:50.154928+0000","modified":"2024-04-16T16:19:53.221144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8160},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8160":{"gid":8160,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/3429980864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3429980864},{"type":"v1","addr":"172.21.15.45:6834","nonce":3429980864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8160,"qdb_cluster":[8160]},"id":8}]} 2024-04-16T16:20:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 82 2024-04-16T16:20:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.205+0000 7f9587e00640 1 -- 172.21.15.45:0/1582311072 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f956406c740 msgr2=0x7f956406ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.205+0000 7f9587e00640 1 --2- 172.21.15.45:0/1582311072 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f956406c740 0x7f956406ec00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f958815ff70 tx=0x7f9578008040 comp rx=0 tx=0).stop 2024-04-16T16:20:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.205+0000 7f9587e00640 1 -- 172.21.15.45:0/1582311072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9588146220 msgr2=0x7f958815c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.205+0000 7f9587e00640 1 --2- 172.21.15.45:0/1582311072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9588146220 0x7f958815c2f0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f957000c4d0 tx=0x7f957000c990 comp rx=0 tx=0).stop 2024-04-16T16:20:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.205+0000 7f9587e00640 1 -- 172.21.15.45:0/1582311072 shutdown_connections 2024-04-16T16:20:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.205+0000 7f9587e00640 1 --2- 172.21.15.45:0/1582311072 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f956406c740 0x7f956406ec00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.205+0000 7f9587e00640 1 --2- 172.21.15.45:0/1582311072 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95881608a0 0x7f95881f4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.205+0000 7f9587e00640 1 --2- 172.21.15.45:0/1582311072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9588146220 0x7f958815c2f0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.205+0000 7f9587e00640 1 --2- 172.21.15.45:0/1582311072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f958806f7a0 0x7f95880ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.205+0000 7f9587e00640 1 -- 172.21.15.45:0/1582311072 >> 172.21.15.45:0/1582311072 conn(0x7f958806b930 msgr2=0x7f958815e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.205+0000 7f9587e00640 1 -- 172.21.15.45:0/1582311072 shutdown_connections 2024-04-16T16:20:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.205+0000 7f9587e00640 1 -- 172.21.15.45:0/1582311072 wait complete. 2024-04-16T16:20:15.223 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:20:15.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.442+0000 7ff13a200640 1 Processor -- start 2024-04-16T16:20:15.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.443+0000 7ff13a200640 1 -- start start 2024-04-16T16:20:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.444+0000 7ff13a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1340d4560 0x7ff1340d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.444+0000 7ff13a200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1340d34c0 con 0x7ff13415ec60 2024-04-16T16:20:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.444+0000 7ff13a200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1340d37a0 con 0x7ff1340d18f0 2024-04-16T16:20:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.444+0000 7ff13a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff1340d3a80 con 0x7ff1340d4560 2024-04-16T16:20:15.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.445+0000 7ff133400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff13415ec60 0x7ff1340d2a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33572/0 (socket says 172.21.15.45:33572) 2024-04-16T16:20:15.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.445+0000 7ff133400640 1 -- 172.21.15.45:0/563785349 learned_addr learned my addr 172.21.15.45:0/563785349 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:15.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.445+0000 7ff132a00640 1 --2- 172.21.15.45:0/563785349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1340d4560 0x7ff1340d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:15.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.445+0000 7ff132000640 1 -- 172.21.15.45:0/563785349 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3254800986 0 0) 0x7ff1340d34c0 con 0x7ff13415ec60 2024-04-16T16:20:15.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.445+0000 7ff132000640 1 -- 172.21.15.45:0/563785349 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff118003520 con 0x7ff13415ec60 2024-04-16T16:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.445+0000 7ff132000640 1 -- 172.21.15.45:0/563785349 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2679305089 0 0) 0x7ff118003520 con 0x7ff13415ec60 2024-04-16T16:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff132000640 1 -- 172.21.15.45:0/563785349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1340d4560 msgr2=0x7ff1340d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff132000640 1 --2- 172.21.15.45:0/563785349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1340d4560 0x7ff1340d13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff132000640 1 -- 172.21.15.45:0/563785349 >> v1:172.21.15.103:6789/0 conn(0x7ff1340d18f0 legacy=0x7ff1340b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff132000640 1 -- 172.21.15.45:0/563785349 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1340d3d60 con 0x7ff13415ec60 2024-04-16T16:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff132000640 1 -- 172.21.15.45:0/563785349 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7ff1240019f0 con 0x7ff13415ec60 2024-04-16T16:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff132000640 1 -- 172.21.15.45:0/563785349 >> v1:172.21.15.45:6789/0 conn(0x7ff13415ec60 legacy=0x7ff1340d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff132000640 1 --2- 172.21.15.45:0/563785349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1340d4560 0x7ff1180039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff132000640 1 --2- 172.21.15.45:0/563785349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff118004010 0x7ff118004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff132000640 1 --2- 172.21.15.45:0/563785349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff118004d40 0x7ff1180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff132000640 1 -- 172.21.15.45:0/563785349 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff1340d3a80 con 0x7ff118004010 2024-04-16T16:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff132000640 1 -- 172.21.15.45:0/563785349 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff11800bb80 con 0x7ff118004d40 2024-04-16T16:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff132000640 1 -- 172.21.15.45:0/563785349 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff11800bd80 con 0x7ff1340d4560 2024-04-16T16:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff132a00640 1 --2- 172.21.15.45:0/563785349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff118004010 0x7ff118004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff133e00640 1 --2- 172.21.15.45:0/563785349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff118004d40 0x7ff1180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff133400640 1 --2- 172.21.15.45:0/563785349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1340d4560 0x7ff1180039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.446+0000 7ff132a00640 1 -- 172.21.15.45:0/563785349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1340d4560 msgr2=0x7ff1180039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.447+0000 7ff132a00640 1 --2- 172.21.15.45:0/563785349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1340d4560 0x7ff1180039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.447+0000 7ff132a00640 1 -- 172.21.15.45:0/563785349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff118004d40 msgr2=0x7ff1180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.447+0000 7ff132a00640 1 --2- 172.21.15.45:0/563785349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff118004d40 0x7ff1180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.447+0000 7ff132a00640 1 -- 172.21.15.45:0/563785349 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff118003520 con 0x7ff118004010 2024-04-16T16:20:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.447+0000 7ff133e00640 1 --2- 172.21.15.45:0/563785349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff118004d40 0x7ff1180051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.448+0000 7ff132a00640 1 --2- 172.21.15.45:0/563785349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff118004010 0x7ff118004800 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7ff11c00ba00 tx=0x7ff118005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14610e970e85d334 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.448+0000 7ff132000640 1 -- 172.21.15.45:0/563785349 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c036020 con 0x7ff118004010 2024-04-16T16:20:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.448+0000 7ff132000640 1 -- 172.21.15.45:0/563785349 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff11c032620 con 0x7ff118004010 2024-04-16T16:20:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.448+0000 7ff132000640 1 -- 172.21.15.45:0/563785349 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c00b600 con 0x7ff118004010 2024-04-16T16:20:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.449+0000 7ff13a200640 1 -- 172.21.15.45:0/563785349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff118004010 msgr2=0x7ff118004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.449+0000 7ff13a200640 1 --2- 172.21.15.45:0/563785349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff118004010 0x7ff118004800 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7ff11c00ba00 tx=0x7ff118005830 comp rx=0 tx=0).stop 2024-04-16T16:20:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.449+0000 7ff133e00640 1 -- 172.21.15.45:0/563785349 reap_dead start 2024-04-16T16:20:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.449+0000 7ff13a200640 1 -- 172.21.15.45:0/563785349 shutdown_connections 2024-04-16T16:20:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.449+0000 7ff13a200640 1 -- 172.21.15.45:0/563785349 >> 172.21.15.45:0/563785349 conn(0x7ff134073690 msgr2=0x7ff13406cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.449+0000 7ff13a200640 1 -- 172.21.15.45:0/563785349 shutdown_connections 2024-04-16T16:20:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.449+0000 7ff13a200640 1 -- 172.21.15.45:0/563785349 wait complete. 2024-04-16T16:20:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.451+0000 7ff13a200640 1 Processor -- start 2024-04-16T16:20:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.451+0000 7ff13a200640 1 -- start start 2024-04-16T16:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.452+0000 7ff13a200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff118004010 0x7ff1340c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.452+0000 7ff13a200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1341f81b0 0x7ff1341f8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.452+0000 7ff13a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1340cdbc0 0x7ff1340ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.452+0000 7ff13a200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff1340d3110 con 0x7ff1341f81b0 2024-04-16T16:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.452+0000 7ff13a200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff1340ba460 con 0x7ff118004010 2024-04-16T16:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.452+0000 7ff13a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff1341e3de0 con 0x7ff1340cdbc0 2024-04-16T16:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.452+0000 7ff132a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1341f81b0 0x7ff1341f8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.452+0000 7ff132a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1341f81b0 0x7ff1341f8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38640/0 (socket says 172.21.15.45:38640) 2024-04-16T16:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.452+0000 7ff132a00640 1 -- 172.21.15.45:0/1749450146 learned_addr learned my addr 172.21.15.45:0/1749450146 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.452+0000 7ff132a00640 1 -- 172.21.15.45:0/1749450146 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1340cdbc0 msgr2=0x7ff1340ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.452+0000 7ff132a00640 1 --2- 172.21.15.45:0/1749450146 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1340cdbc0 0x7ff1340ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.452+0000 7ff132a00640 1 -- 172.21.15.45:0/1749450146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff118004010 msgr2=0x7ff1340c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.452+0000 7ff132a00640 1 --2- 172.21.15.45:0/1749450146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff118004010 0x7ff1340c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.452+0000 7ff132a00640 1 -- 172.21.15.45:0/1749450146 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1340ce580 con 0x7ff1341f81b0 2024-04-16T16:20:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.453+0000 7ff132a00640 1 --2- 172.21.15.45:0/1749450146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1341f81b0 0x7ff1341f8630 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7ff11c00bb20 tx=0x7ff11c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.453+0000 7ff12b400640 1 -- 172.21.15.45:0/1749450146 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c041070 con 0x7ff1341f81b0 2024-04-16T16:20:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.453+0000 7ff12b400640 1 -- 172.21.15.45:0/1749450146 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff11c004020 con 0x7ff1341f81b0 2024-04-16T16:20:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.453+0000 7ff12b400640 1 -- 172.21.15.45:0/1749450146 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c036040 con 0x7ff1341f81b0 2024-04-16T16:20:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.453+0000 7ff13a200640 1 -- 172.21.15.45:0/1749450146 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1341f8c70 con 0x7ff1341f81b0 2024-04-16T16:20:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.453+0000 7ff13a200640 1 -- 172.21.15.45:0/1749450146 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1341e4030 con 0x7ff1341f81b0 2024-04-16T16:20:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.454+0000 7ff13a200640 1 -- 172.21.15.45:0/1749450146 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff134075780 con 0x7ff1341f81b0 2024-04-16T16:20:15.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.455+0000 7ff12b400640 1 -- 172.21.15.45:0/1749450146 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 47) ==== 90878+0+0 (secure 0 0 0) 0x7ff11c039020 con 0x7ff1341f81b0 2024-04-16T16:20:15.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.456+0000 7ff12b400640 1 --2- 172.21.15.45:0/1749450146 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff11006c790 0x7ff11006ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.457+0000 7ff12b400640 1 -- 172.21.15.45:0/1749450146 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6397+0+0 (secure 0 0 0) 0x7ff11c0ad610 con 0x7ff1341f81b0 2024-04-16T16:20:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.457+0000 7ff133400640 1 --2- 172.21.15.45:0/1749450146 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff11006c790 0x7ff11006ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:15.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.458+0000 7ff133400640 1 --2- 172.21.15.45:0/1749450146 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff11006c790 0x7ff11006ec50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff124001b60 tx=0x7ff124009040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:15.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.461+0000 7ff12b400640 1 -- 172.21.15.45:0/1749450146 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff11c07ba80 con 0x7ff1341f81b0 2024-04-16T16:20:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.667+0000 7ff13a200640 1 -- 172.21.15.45:0/1749450146 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff1340ba960 con 0x7ff1341f81b0 2024-04-16T16:20:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.671+0000 7ff12b400640 1 -- 172.21.15.45:0/1749450146 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) ==== 74+0+17642 (secure 0 0 0) 0x7ff11c07b380 con 0x7ff1341f81b0 2024-04-16T16:20:15.671 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:15.671 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":117,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:20:13.465090+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:48.114834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:49.135741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/383252954":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/428120997":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/3229718691":"2024-04-17T16:20:13.464929+0000","172.21.15.103:6833/1934318167":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/3469935364":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/3506765807":"2024-04-17T16:20:07.040813+0000","172.21.15.103:6832/1934318167":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/3547880684":"2024-04-17T16:20:07.040813+0000","172.21.15.103:6833/2293932214":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/1890933611":"2024-04-17T16:20:13.464929+0000","172.21.15.103:6832/2293932214":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/1347839066":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/546409297":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/3566291251":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/714185246":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/2784388736":"2024-04-16T17:19:54.469649+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:20:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.673+0000 7ff13a200640 1 -- 172.21.15.45:0/1749450146 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff11006c790 msgr2=0x7ff11006ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.673+0000 7ff13a200640 1 --2- 172.21.15.45:0/1749450146 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff11006c790 0x7ff11006ec50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff124001b60 tx=0x7ff124009040 comp rx=0 tx=0).stop 2024-04-16T16:20:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.673+0000 7ff13a200640 1 -- 172.21.15.45:0/1749450146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1341f81b0 msgr2=0x7ff1341f8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.673+0000 7ff13a200640 1 --2- 172.21.15.45:0/1749450146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1341f81b0 0x7ff1341f8630 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7ff11c00bb20 tx=0x7ff11c004730 comp rx=0 tx=0).stop 2024-04-16T16:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.673+0000 7ff13a200640 1 -- 172.21.15.45:0/1749450146 shutdown_connections 2024-04-16T16:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.673+0000 7ff13a200640 1 --2- 172.21.15.45:0/1749450146 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff11006c790 0x7ff11006ec50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.673+0000 7ff13a200640 1 --2- 172.21.15.45:0/1749450146 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1340cdbc0 0x7ff1340ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.673+0000 7ff13a200640 1 --2- 172.21.15.45:0/1749450146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1341f81b0 0x7ff1341f8630 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.673+0000 7ff13a200640 1 --2- 172.21.15.45:0/1749450146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff118004010 0x7ff1340c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.673+0000 7ff13a200640 1 -- 172.21.15.45:0/1749450146 >> 172.21.15.45:0/1749450146 conn(0x7ff134073690 msgr2=0x7ff1340b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.674+0000 7ff13a200640 1 -- 172.21.15.45:0/1749450146 shutdown_connections 2024-04-16T16:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.674+0000 7ff13a200640 1 -- 172.21.15.45:0/1749450146 wait complete. 2024-04-16T16:20:15.694 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:19:48.114834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:19:49.135741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:20:15.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:20:15.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.922+0000 7f8793200640 1 Processor -- start 2024-04-16T16:20:15.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.922+0000 7f8793200640 1 -- start start 2024-04-16T16:20:15.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.924+0000 7f8793200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f878c0b5a30 0x7f878c1625e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.924+0000 7f8793200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f878c0b5e10 con 0x7f878c06e290 2024-04-16T16:20:15.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.924+0000 7f8793200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f878c15ff70 con 0x7f878c0d1030 2024-04-16T16:20:15.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.924+0000 7f8793200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f878c15f3d0 con 0x7f878c0b5a30 2024-04-16T16:20:15.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.924+0000 7f8791200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f878c06e290 0x7f878c0b4920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33576/0 (socket says 172.21.15.45:33576) 2024-04-16T16:20:15.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.924+0000 7f8791200640 1 -- 172.21.15.45:0/2348649682 learned_addr learned my addr 172.21.15.45:0/2348649682 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:15.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878be00640 1 --2- 172.21.15.45:0/2348649682 >> v2:172.21.15.103:3300/0 conn(0x7f878c0b5a30 0x7f878c1625e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:15.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878aa00640 1 -- 172.21.15.45:0/2348649682 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3136316635 0 0) 0x7f878c0b5e10 con 0x7f878c06e290 2024-04-16T16:20:15.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878aa00640 1 -- 172.21.15.45:0/2348649682 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8770003520 con 0x7f878c06e290 2024-04-16T16:20:15.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878aa00640 1 -- 172.21.15.45:0/2348649682 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2929164921 0 0) 0x7f8770003520 con 0x7f878c06e290 2024-04-16T16:20:15.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878aa00640 1 -- 172.21.15.45:0/2348649682 >> v2:172.21.15.103:3300/0 conn(0x7f878c0b5a30 msgr2=0x7f878c1625e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:15.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878aa00640 1 --2- 172.21.15.45:0/2348649682 >> v2:172.21.15.103:3300/0 conn(0x7f878c0b5a30 0x7f878c1625e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878aa00640 1 -- 172.21.15.45:0/2348649682 >> v1:172.21.15.103:6789/0 conn(0x7f878c0d1030 legacy=0x7f878c15f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:15.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878aa00640 1 -- 172.21.15.45:0/2348649682 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878c162e70 con 0x7f878c06e290 2024-04-16T16:20:15.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878aa00640 1 -- 172.21.15.45:0/2348649682 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f87840019f0 con 0x7f878c06e290 2024-04-16T16:20:15.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878aa00640 1 -- 172.21.15.45:0/2348649682 >> v1:172.21.15.45:6789/0 conn(0x7f878c06e290 legacy=0x7f878c0b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:15.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878aa00640 1 --2- 172.21.15.45:0/2348649682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8770003a00 0x7f8770003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878aa00640 1 --2- 172.21.15.45:0/2348649682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8770004800 0x7f8770004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878aa00640 1 --2- 172.21.15.45:0/2348649682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87700051c0 0x7f877000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878aa00640 1 -- 172.21.15.45:0/2348649682 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f878c15f3d0 con 0x7f87700051c0 2024-04-16T16:20:15.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878aa00640 1 -- 172.21.15.45:0/2348649682 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f877000c0c0 con 0x7f8770003a00 2024-04-16T16:20:15.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.925+0000 7f878aa00640 1 -- 172.21.15.45:0/2348649682 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f877000c2c0 con 0x7f8770004800 2024-04-16T16:20:15.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.926+0000 7f8791200640 1 --2- 172.21.15.45:0/2348649682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87700051c0 0x7f877000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:15.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.926+0000 7f878b400640 1 --2- 172.21.15.45:0/2348649682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8770004800 0x7f8770004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:15.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.926+0000 7f878be00640 1 --2- 172.21.15.45:0/2348649682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8770003a00 0x7f8770003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:15.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.926+0000 7f8791200640 1 -- 172.21.15.45:0/2348649682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8770004800 msgr2=0x7f8770004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:15.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.926+0000 7f8791200640 1 --2- 172.21.15.45:0/2348649682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8770004800 0x7f8770004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.926+0000 7f8791200640 1 -- 172.21.15.45:0/2348649682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8770003a00 msgr2=0x7f8770003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:15.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.926+0000 7f8791200640 1 --2- 172.21.15.45:0/2348649682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8770003a00 0x7f8770003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.926+0000 7f8791200640 1 -- 172.21.15.45:0/2348649682 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8770003520 con 0x7f87700051c0 2024-04-16T16:20:15.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.927+0000 7f8791200640 1 -- 172.21.15.45:0/2348649682 reap_dead start 2024-04-16T16:20:15.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.927+0000 7f8791200640 1 --2- 172.21.15.45:0/2348649682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87700051c0 0x7f877000b9c0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f878400a790 tx=0x7f8770005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf107bddaa293959 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:15.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.928+0000 7f878aa00640 1 -- 172.21.15.45:0/2348649682 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878400a3b0 con 0x7f87700051c0 2024-04-16T16:20:15.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.928+0000 7f878aa00640 1 -- 172.21.15.45:0/2348649682 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8784034670 con 0x7f87700051c0 2024-04-16T16:20:15.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.928+0000 7f878aa00640 1 -- 172.21.15.45:0/2348649682 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8784034a60 con 0x7f87700051c0 2024-04-16T16:20:15.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.928+0000 7f8793200640 1 -- 172.21.15.45:0/2348649682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87700051c0 msgr2=0x7f877000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:15.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.928+0000 7f8793200640 1 --2- 172.21.15.45:0/2348649682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87700051c0 0x7f877000b9c0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f878400a790 tx=0x7f8770005830 comp rx=0 tx=0).stop 2024-04-16T16:20:15.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.928+0000 7f8793200640 1 -- 172.21.15.45:0/2348649682 shutdown_connections 2024-04-16T16:20:15.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.928+0000 7f8793200640 1 --2- 172.21.15.45:0/2348649682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87700051c0 0x7f877000b9c0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.928+0000 7f8793200640 1 -- 172.21.15.45:0/2348649682 >> 172.21.15.45:0/2348649682 conn(0x7f878c0b3970 msgr2=0x7f878c0b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:15.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.928+0000 7f8793200640 1 -- 172.21.15.45:0/2348649682 shutdown_connections 2024-04-16T16:20:15.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.928+0000 7f8793200640 1 -- 172.21.15.45:0/2348649682 wait complete. 2024-04-16T16:20:15.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.930+0000 7f8793200640 1 Processor -- start 2024-04-16T16:20:15.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.930+0000 7f8793200640 1 -- start start 2024-04-16T16:20:15.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.931+0000 7f8793200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87700051c0 0x7f878c150690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.931+0000 7f8793200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f878c1f56b0 0x7f878c157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.931+0000 7f8793200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f878c1f5b10 0x7f878c157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.931+0000 7f8793200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f878c15eff0 con 0x7f878c1f56b0 2024-04-16T16:20:15.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.931+0000 7f8793200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f878c163430 con 0x7f87700051c0 2024-04-16T16:20:15.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.931+0000 7f8793200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f878c06e6e0 con 0x7f878c1f5b10 2024-04-16T16:20:15.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.932+0000 7f878b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f878c1f56b0 0x7f878c157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:15.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.932+0000 7f878b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f878c1f56b0 0x7f878c157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38660/0 (socket says 172.21.15.45:38660) 2024-04-16T16:20:15.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.932+0000 7f878b400640 1 -- 172.21.15.45:0/2533304196 learned_addr learned my addr 172.21.15.45:0/2533304196 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:15.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.932+0000 7f878be00640 1 --2- 172.21.15.45:0/2533304196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87700051c0 0x7f878c150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:15.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.932+0000 7f8791200640 1 --2- 172.21.15.45:0/2533304196 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f878c1f5b10 0x7f878c157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:15.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.932+0000 7f878b400640 1 -- 172.21.15.45:0/2533304196 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f878c1f5b10 msgr2=0x7f878c157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:15.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.932+0000 7f878b400640 1 --2- 172.21.15.45:0/2533304196 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f878c1f5b10 0x7f878c157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.932+0000 7f878b400640 1 -- 172.21.15.45:0/2533304196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87700051c0 msgr2=0x7f878c150690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:15.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.932+0000 7f878b400640 1 --2- 172.21.15.45:0/2533304196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87700051c0 0x7f878c150690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:15.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.932+0000 7f878b400640 1 -- 172.21.15.45:0/2533304196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878c0cd460 con 0x7f878c1f56b0 2024-04-16T16:20:15.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.932+0000 7f878b400640 1 --2- 172.21.15.45:0/2533304196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f878c1f56b0 0x7f878c157a20 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f877800d520 tx=0x7f877800d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:15.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.933+0000 7f8783e00640 1 -- 172.21.15.45:0/2533304196 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8778005480 con 0x7f878c1f56b0 2024-04-16T16:20:15.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.933+0000 7f8783e00640 1 -- 172.21.15.45:0/2533304196 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8778017070 con 0x7f878c1f56b0 2024-04-16T16:20:15.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.933+0000 7f8783e00640 1 -- 172.21.15.45:0/2533304196 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8778003830 con 0x7f878c1f56b0 2024-04-16T16:20:15.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.933+0000 7f8793200640 1 -- 172.21.15.45:0/2533304196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f878c0cd740 con 0x7f878c1f56b0 2024-04-16T16:20:15.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.933+0000 7f8793200640 1 -- 172.21.15.45:0/2533304196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f878c0cdd00 con 0x7f878c1f56b0 2024-04-16T16:20:15.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.934+0000 7f8793200640 1 -- 172.21.15.45:0/2533304196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f878c075780 con 0x7f878c1f56b0 2024-04-16T16:20:15.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.935+0000 7f8783e00640 1 -- 172.21.15.45:0/2533304196 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 47) ==== 90878+0+0 (secure 0 0 0) 0x7f87780039d0 con 0x7f878c1f56b0 2024-04-16T16:20:15.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.936+0000 7f8783e00640 1 --2- 172.21.15.45:0/2533304196 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f876c06c740 0x7f876c06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:15.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.936+0000 7f8783e00640 1 -- 172.21.15.45:0/2533304196 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6397+0+0 (secure 0 0 0) 0x7f877808b990 con 0x7f878c1f56b0 2024-04-16T16:20:15.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.937+0000 7f878be00640 1 --2- 172.21.15.45:0/2533304196 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f876c06c740 0x7f876c06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:15.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.938+0000 7f878be00640 1 --2- 172.21.15.45:0/2533304196 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f876c06c740 0x7f876c06ec00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f877c009a10 tx=0x7f877c008040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:15.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:15.941+0000 7f8783e00640 1 -- 172.21.15.45:0/2533304196 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8778059cf0 con 0x7f878c1f56b0 2024-04-16T16:20:16.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.143+0000 7f8793200640 1 -- 172.21.15.45:0/2533304196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f878c155500 con 0x7f878c1f56b0 2024-04-16T16:20:16.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.149+0000 7f8783e00640 1 -- 172.21.15.45:0/2533304196 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 48) ==== 91086+0+0 (secure 0 0 0) 0x7f8778055c60 con 0x7f878c1f56b0 2024-04-16T16:20:16.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.153+0000 7f8783e00640 1 -- 172.21.15.45:0/2533304196 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) ==== 93+0+10 (secure 0 0 0) 0x7f87780778d0 con 0x7f878c1f56b0 2024-04-16T16:20:16.153 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:20:16.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.156+0000 7f8793200640 1 -- 172.21.15.45:0/2533304196 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f876c06c740 msgr2=0x7f876c06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:16.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.156+0000 7f8793200640 1 --2- 172.21.15.45:0/2533304196 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f876c06c740 0x7f876c06ec00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f877c009a10 tx=0x7f877c008040 comp rx=0 tx=0).stop 2024-04-16T16:20:16.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.156+0000 7f8793200640 1 -- 172.21.15.45:0/2533304196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f878c1f56b0 msgr2=0x7f878c157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:16.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.156+0000 7f8793200640 1 --2- 172.21.15.45:0/2533304196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f878c1f56b0 0x7f878c157a20 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f877800d520 tx=0x7f877800d9e0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.156+0000 7f8793200640 1 -- 172.21.15.45:0/2533304196 shutdown_connections 2024-04-16T16:20:16.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.156+0000 7f8793200640 1 --2- 172.21.15.45:0/2533304196 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f876c06c740 0x7f876c06ec00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.156+0000 7f8793200640 1 --2- 172.21.15.45:0/2533304196 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f878c1f5b10 0x7f878c157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.156+0000 7f8793200640 1 --2- 172.21.15.45:0/2533304196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f878c1f56b0 0x7f878c157a20 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.156+0000 7f8793200640 1 --2- 172.21.15.45:0/2533304196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87700051c0 0x7f878c150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.156+0000 7f8793200640 1 -- 172.21.15.45:0/2533304196 >> 172.21.15.45:0/2533304196 conn(0x7f878c0b3970 msgr2=0x7f878c161950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:16.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.156+0000 7f8793200640 1 -- 172.21.15.45:0/2533304196 shutdown_connections 2024-04-16T16:20:16.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.156+0000 7f8793200640 1 -- 172.21.15.45:0/2533304196 wait complete. 2024-04-16T16:20:16.176 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:20:16.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.400+0000 7ff1cde00640 1 Processor -- start 2024-04-16T16:20:16.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.400+0000 7ff1cde00640 1 -- start start 2024-04-16T16:20:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.401+0000 7ff1cde00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff1c80d18f0 0x7ff1c80ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.401+0000 7ff1cde00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff1c80d1cd0 0x7ff1c8162dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.401+0000 7ff1cde00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1c80b6810 con 0x7ff1c80b6310 2024-04-16T16:20:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.401+0000 7ff1cde00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff1c80d29f0 con 0x7ff1c80d1cd0 2024-04-16T16:20:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.401+0000 7ff1cde00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff1c80b6af0 con 0x7ff1c80d18f0 2024-04-16T16:20:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.402+0000 7ff1cd400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff1c80d1cd0 0x7ff1c8162dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.402+0000 7ff1cd400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff1c80d1cd0 0x7ff1c8162dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38662/0 (socket says 172.21.15.45:38662) 2024-04-16T16:20:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.402+0000 7ff1cd400640 1 -- 172.21.15.45:0/3639651553 learned_addr learned my addr 172.21.15.45:0/3639651553 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.402+0000 7ff1cca00640 1 --2- 172.21.15.45:0/3639651553 >> v2:172.21.15.103:3300/0 conn(0x7ff1c80d18f0 0x7ff1c80ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:16.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.402+0000 7ff1cd400640 1 -- 172.21.15.45:0/3639651553 >> v2:172.21.15.103:3300/0 conn(0x7ff1c80d18f0 msgr2=0x7ff1c80ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:16.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.402+0000 7ff1cd400640 1 --2- 172.21.15.45:0/3639651553 >> v2:172.21.15.103:3300/0 conn(0x7ff1c80d18f0 0x7ff1c80ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.402+0000 7ff1cd400640 1 -- 172.21.15.45:0/3639651553 >> v1:172.21.15.103:6789/0 conn(0x7ff1c80b6310 legacy=0x7ff1c80d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:16.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.402+0000 7ff1cd400640 1 -- 172.21.15.45:0/3639651553 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1c80b6dd0 con 0x7ff1c80d1cd0 2024-04-16T16:20:16.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.404+0000 7ff1cd400640 1 --2- 172.21.15.45:0/3639651553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1c80d1cd0 0x7ff1c8162dc0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8073f70 tx=0x7ff1c002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8922190f96b08530 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:16.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.404+0000 7ff1c7400640 1 -- 172.21.15.45:0/3639651553 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c002c960 con 0x7ff1c80d1cd0 2024-04-16T16:20:16.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.404+0000 7ff1c7400640 1 -- 172.21.15.45:0/3639651553 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1c002cb00 con 0x7ff1c80d1cd0 2024-04-16T16:20:16.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.404+0000 7ff1c7400640 1 -- 172.21.15.45:0/3639651553 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c00343f0 con 0x7ff1c80d1cd0 2024-04-16T16:20:16.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.405+0000 7ff1cde00640 1 -- 172.21.15.45:0/3639651553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1c80d1cd0 msgr2=0x7ff1c8162dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:16.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.405+0000 7ff1cde00640 1 --2- 172.21.15.45:0/3639651553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1c80d1cd0 0x7ff1c8162dc0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8073f70 tx=0x7ff1c002bdc0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.405+0000 7ff1cde00640 1 -- 172.21.15.45:0/3639651553 shutdown_connections 2024-04-16T16:20:16.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.405+0000 7ff1cde00640 1 --2- 172.21.15.45:0/3639651553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1c80d1cd0 0x7ff1c8162dc0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.405+0000 7ff1cde00640 1 --2- 172.21.15.45:0/3639651553 >> v2:172.21.15.103:3300/0 conn(0x7ff1c80d18f0 0x7ff1c80ba220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.405+0000 7ff1cde00640 1 -- 172.21.15.45:0/3639651553 >> 172.21.15.45:0/3639651553 conn(0x7ff1c806cf90 msgr2=0x7ff1c806d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:16.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.405+0000 7ff1cde00640 1 -- 172.21.15.45:0/3639651553 shutdown_connections 2024-04-16T16:20:16.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.405+0000 7ff1cde00640 1 -- 172.21.15.45:0/3639651553 wait complete. 2024-04-16T16:20:16.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.407+0000 7ff1cde00640 1 Processor -- start 2024-04-16T16:20:16.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.407+0000 7ff1cde00640 1 -- start start 2024-04-16T16:20:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.408+0000 7ff1cde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1c80b6310 0x7ff1c80cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.408+0000 7ff1cde00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1c80d18f0 0x7ff1c80cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.408+0000 7ff1cde00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1c80d1cd0 0x7ff1c80cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.408+0000 7ff1cde00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c8163300 con 0x7ff1c80b6310 2024-04-16T16:20:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.408+0000 7ff1cde00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c80ba760 con 0x7ff1c80d1cd0 2024-04-16T16:20:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.408+0000 7ff1cde00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff1c80c8550 con 0x7ff1c80d18f0 2024-04-16T16:20:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.408+0000 7ff1cca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1c80b6310 0x7ff1c80cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.408+0000 7ff1cca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1c80b6310 0x7ff1c80cb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38666/0 (socket says 172.21.15.45:38666) 2024-04-16T16:20:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.408+0000 7ff1cca00640 1 -- 172.21.15.45:0/151525408 learned_addr learned my addr 172.21.15.45:0/151525408 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.409+0000 7ff1cd400640 1 --2- 172.21.15.45:0/151525408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1c80d1cd0 0x7ff1c80cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.409+0000 7ff1cca00640 1 -- 172.21.15.45:0/151525408 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1c80d18f0 msgr2=0x7ff1c80cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.409+0000 7ff1cca00640 1 --2- 172.21.15.45:0/151525408 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1c80d18f0 0x7ff1c80cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.409+0000 7ff1cca00640 1 -- 172.21.15.45:0/151525408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1c80d1cd0 msgr2=0x7ff1c80cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.409+0000 7ff1cca00640 1 --2- 172.21.15.45:0/151525408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1c80d1cd0 0x7ff1c80cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.409+0000 7ff1cca00640 1 -- 172.21.15.45:0/151525408 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1c80b8ab0 con 0x7ff1c80b6310 2024-04-16T16:20:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.409+0000 7ff1cd400640 1 --2- 172.21.15.45:0/151525408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1c80d1cd0 0x7ff1c80cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.409+0000 7ff1cca00640 1 --2- 172.21.15.45:0/151525408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1c80b6310 0x7ff1c80cb960 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7ff1b800d800 tx=0x7ff1b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.409+0000 7ff1c4c00640 1 -- 172.21.15.45:0/151525408 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b800b6c0 con 0x7ff1c80b6310 2024-04-16T16:20:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.409+0000 7ff1c4c00640 1 -- 172.21.15.45:0/151525408 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1b8019070 con 0x7ff1c80b6310 2024-04-16T16:20:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.409+0000 7ff1cde00640 1 -- 172.21.15.45:0/151525408 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1c80b8d90 con 0x7ff1c80b6310 2024-04-16T16:20:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.409+0000 7ff1c4c00640 1 -- 172.21.15.45:0/151525408 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b800ba80 con 0x7ff1c80b6310 2024-04-16T16:20:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.410+0000 7ff1cde00640 1 -- 172.21.15.45:0/151525408 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1c80b9350 con 0x7ff1c80b6310 2024-04-16T16:20:16.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.411+0000 7ff1cde00640 1 -- 172.21.15.45:0/151525408 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1c8075780 con 0x7ff1c80b6310 2024-04-16T16:20:16.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.412+0000 7ff1c4c00640 1 -- 172.21.15.45:0/151525408 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 48) ==== 91086+0+0 (secure 0 0 0) 0x7ff1b800bc20 con 0x7ff1c80b6310 2024-04-16T16:20:16.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.412+0000 7ff1c4c00640 1 --2- 172.21.15.45:0/151525408 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff1a406c980 0x7ff1a406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:16.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.413+0000 7ff1c4c00640 1 -- 172.21.15.45:0/151525408 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6397+0+0 (secure 0 0 0) 0x7ff1b808c300 con 0x7ff1c80b6310 2024-04-16T16:20:16.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.413+0000 7ff1c7e00640 1 --2- 172.21.15.45:0/151525408 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff1a406c980 0x7ff1a406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:16.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.414+0000 7ff1c7e00640 1 --2- 172.21.15.45:0/151525408 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff1a406c980 0x7ff1a406ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80ca610 tx=0x7ff1b00079b0 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.417+0000 7ff1c4c00640 1 -- 172.21.15.45:0/151525408 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff1b805a490 con 0x7ff1c80b6310 2024-04-16T16:20:16.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.623+0000 7ff1cde00640 1 -- 172.21.15.45:0/151525408 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff1c8162f40 con 0x7ff1c80b6310 2024-04-16T16:20:16.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.623+0000 7ff1c4c00640 1 -- 172.21.15.45:0/151525408 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v117) ==== 104+0+11 (secure 0 0 0) 0x7ff1b8059d90 con 0x7ff1c80b6310 2024-04-16T16:20:16.624 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:20:16.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.626+0000 7ff1cde00640 1 -- 172.21.15.45:0/151525408 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff1a406c980 msgr2=0x7ff1a406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:16.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.626+0000 7ff1cde00640 1 --2- 172.21.15.45:0/151525408 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff1a406c980 0x7ff1a406ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80ca610 tx=0x7ff1b00079b0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.626+0000 7ff1cde00640 1 -- 172.21.15.45:0/151525408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1c80b6310 msgr2=0x7ff1c80cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:16.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.626+0000 7ff1cde00640 1 --2- 172.21.15.45:0/151525408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1c80b6310 0x7ff1c80cb960 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7ff1b800d800 tx=0x7ff1b800dcc0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.626+0000 7ff1cde00640 1 -- 172.21.15.45:0/151525408 shutdown_connections 2024-04-16T16:20:16.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.626+0000 7ff1cde00640 1 --2- 172.21.15.45:0/151525408 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff1a406c980 0x7ff1a406ee40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.626+0000 7ff1cde00640 1 --2- 172.21.15.45:0/151525408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1c80d1cd0 0x7ff1c80cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.626+0000 7ff1cde00640 1 --2- 172.21.15.45:0/151525408 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1c80d18f0 0x7ff1c80cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.626+0000 7ff1cde00640 1 --2- 172.21.15.45:0/151525408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1c80b6310 0x7ff1c80cb960 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.626+0000 7ff1cde00640 1 -- 172.21.15.45:0/151525408 >> 172.21.15.45:0/151525408 conn(0x7ff1c806cf90 msgr2=0x7ff1c80b4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:16.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.626+0000 7ff1cde00640 1 -- 172.21.15.45:0/151525408 shutdown_connections 2024-04-16T16:20:16.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.626+0000 7ff1cde00640 1 -- 172.21.15.45:0/151525408 wait complete. 2024-04-16T16:20:16.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:20:16.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.872+0000 7f1600a00640 1 Processor -- start 2024-04-16T16:20:16.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.872+0000 7f1600a00640 1 -- start start 2024-04-16T16:20:16.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.874+0000 7f1600a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f15fc0b6310 0x7f15fc161db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:16.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.874+0000 7f1600a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f15fc0d3e90 0x7f15fc1622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:16.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.874+0000 7f1600a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15fc0b6810 con 0x7f15fc0d18f0 2024-04-16T16:20:16.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.874+0000 7f1600a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f15fc0d29f0 con 0x7f15fc0b6310 2024-04-16T16:20:16.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.874+0000 7f1600a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f15fc0b6af0 con 0x7f15fc0d3e90 2024-04-16T16:20:16.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.874+0000 7f15f9e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f15fc0b6310 0x7f15fc161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:16.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.874+0000 7f15f9e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f15fc0b6310 0x7f15fc161db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38674/0 (socket says 172.21.15.45:38674) 2024-04-16T16:20:16.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.874+0000 7f15f9e00640 1 -- 172.21.15.45:0/1758297460 learned_addr learned my addr 172.21.15.45:0/1758297460 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:16.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.874+0000 7f15fb200640 1 --2- 172.21.15.45:0/1758297460 >> v2:172.21.15.103:3300/0 conn(0x7f15fc0d3e90 0x7f15fc1622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:16.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.875+0000 7f15f9e00640 1 -- 172.21.15.45:0/1758297460 >> v2:172.21.15.103:3300/0 conn(0x7f15fc0d3e90 msgr2=0x7f15fc1622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:16.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.875+0000 7f15f9e00640 1 --2- 172.21.15.45:0/1758297460 >> v2:172.21.15.103:3300/0 conn(0x7f15fc0d3e90 0x7f15fc1622f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.875+0000 7f15f9e00640 1 -- 172.21.15.45:0/1758297460 >> v1:172.21.15.103:6789/0 conn(0x7f15fc0d18f0 legacy=0x7f15fc0ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:16.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.875+0000 7f15f9e00640 1 -- 172.21.15.45:0/1758297460 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15fc0b6dd0 con 0x7f15fc0b6310 2024-04-16T16:20:16.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.877+0000 7f15f9e00640 1 --2- 172.21.15.45:0/1758297460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15fc0b6310 0x7f15fc161db0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f15fc08d500 tx=0x7f15e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=effc6f590bf68307 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:16.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.877+0000 7f15f9400640 1 -- 172.21.15.45:0/1758297460 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e4035020 con 0x7f15fc0b6310 2024-04-16T16:20:16.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.877+0000 7f15f9400640 1 -- 172.21.15.45:0/1758297460 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15e4031870 con 0x7f15fc0b6310 2024-04-16T16:20:16.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.877+0000 7f15f9400640 1 -- 172.21.15.45:0/1758297460 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e4031c40 con 0x7f15fc0b6310 2024-04-16T16:20:16.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.878+0000 7f1600a00640 1 -- 172.21.15.45:0/1758297460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15fc0b6310 msgr2=0x7f15fc161db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:16.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.878+0000 7f1600a00640 1 --2- 172.21.15.45:0/1758297460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15fc0b6310 0x7f15fc161db0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f15fc08d500 tx=0x7f15e402b910 comp rx=0 tx=0).stop 2024-04-16T16:20:16.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.878+0000 7f1600a00640 1 -- 172.21.15.45:0/1758297460 shutdown_connections 2024-04-16T16:20:16.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.878+0000 7f1600a00640 1 --2- 172.21.15.45:0/1758297460 >> v2:172.21.15.103:3300/0 conn(0x7f15fc0d3e90 0x7f15fc1622f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.878+0000 7f1600a00640 1 --2- 172.21.15.45:0/1758297460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15fc0b6310 0x7f15fc161db0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.878+0000 7f1600a00640 1 -- 172.21.15.45:0/1758297460 >> 172.21.15.45:0/1758297460 conn(0x7f15fc06cf90 msgr2=0x7f15fc06d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:16.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.878+0000 7f1600a00640 1 -- 172.21.15.45:0/1758297460 shutdown_connections 2024-04-16T16:20:16.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.878+0000 7f1600a00640 1 -- 172.21.15.45:0/1758297460 wait complete. 2024-04-16T16:20:16.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.880+0000 7f1600a00640 1 Processor -- start 2024-04-16T16:20:16.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.880+0000 7f1600a00640 1 -- start start 2024-04-16T16:20:16.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.881+0000 7f1600a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15fc0b6310 0x7f15fc0c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:16.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.881+0000 7f1600a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15fc0d18f0 0x7f15fc0b8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:16.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.881+0000 7f1600a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15fc0d3e90 0x7f15fc0b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:16.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.881+0000 7f1600a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc0d1e50 con 0x7f15fc0d3e90 2024-04-16T16:20:16.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.881+0000 7f1600a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc0d1cd0 con 0x7f15fc0d18f0 2024-04-16T16:20:16.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.881+0000 7f1600a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f15fc0d1fd0 con 0x7f15fc0b6310 2024-04-16T16:20:16.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.881+0000 7f15fb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15fc0d3e90 0x7f15fc0b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:16.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.881+0000 7f15fb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15fc0d3e90 0x7f15fc0b91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38690/0 (socket says 172.21.15.45:38690) 2024-04-16T16:20:16.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.881+0000 7f15fb200640 1 -- 172.21.15.45:0/3100813305 learned_addr learned my addr 172.21.15.45:0/3100813305 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:16.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.882+0000 7f15f9e00640 1 --2- 172.21.15.45:0/3100813305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15fc0d18f0 0x7f15fc0b8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:16.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.882+0000 7f15fa800640 1 --2- 172.21.15.45:0/3100813305 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15fc0b6310 0x7f15fc0c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:16.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.882+0000 7f15fb200640 1 -- 172.21.15.45:0/3100813305 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15fc0b6310 msgr2=0x7f15fc0c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:16.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.882+0000 7f15fb200640 1 --2- 172.21.15.45:0/3100813305 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15fc0b6310 0x7f15fc0c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.882+0000 7f15fb200640 1 -- 172.21.15.45:0/3100813305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15fc0d18f0 msgr2=0x7f15fc0b8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:16.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.882+0000 7f15fb200640 1 --2- 172.21.15.45:0/3100813305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15fc0d18f0 0x7f15fc0b8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:16.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.882+0000 7f15fb200640 1 -- 172.21.15.45:0/3100813305 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15fc0c9c80 con 0x7f15fc0d3e90 2024-04-16T16:20:16.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.882+0000 7f15fa800640 1 --2- 172.21.15.45:0/3100813305 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15fc0b6310 0x7f15fc0c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:16.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.882+0000 7f15fb200640 1 --2- 172.21.15.45:0/3100813305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15fc0d3e90 0x7f15fc0b91f0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f15f400e7f0 tx=0x7f15f400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:16.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.882+0000 7f15f2a00640 1 -- 172.21.15.45:0/3100813305 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f4017070 con 0x7f15fc0d3e90 2024-04-16T16:20:16.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.882+0000 7f15f2a00640 1 -- 172.21.15.45:0/3100813305 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15f400ee40 con 0x7f15fc0d3e90 2024-04-16T16:20:16.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.882+0000 7f15f2a00640 1 -- 172.21.15.45:0/3100813305 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f400cb80 con 0x7f15fc0d3e90 2024-04-16T16:20:16.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.882+0000 7f1600a00640 1 -- 172.21.15.45:0/3100813305 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15fc0c9f20 con 0x7f15fc0d3e90 2024-04-16T16:20:16.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.883+0000 7f1600a00640 1 -- 172.21.15.45:0/3100813305 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15fc0ca4e0 con 0x7f15fc0d3e90 2024-04-16T16:20:16.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.884+0000 7f1600a00640 1 -- 172.21.15.45:0/3100813305 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15fc075780 con 0x7f15fc0d3e90 2024-04-16T16:20:16.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.885+0000 7f15f2a00640 1 -- 172.21.15.45:0/3100813305 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 48) ==== 91086+0+0 (secure 0 0 0) 0x7f15f401a070 con 0x7f15fc0d3e90 2024-04-16T16:20:16.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.886+0000 7f15f2a00640 1 --2- 172.21.15.45:0/3100813305 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f15d806c980 0x7f15d806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:16.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.886+0000 7f15f2a00640 1 -- 172.21.15.45:0/3100813305 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6397+0+0 (secure 0 0 0) 0x7f15f408bce0 con 0x7f15fc0d3e90 2024-04-16T16:20:16.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.887+0000 7f15fa800640 1 --2- 172.21.15.45:0/3100813305 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f15d806c980 0x7f15d806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:16.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.888+0000 7f15fa800640 1 --2- 172.21.15.45:0/3100813305 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f15d806c980 0x7f15d806ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f15fc0b98a0 tx=0x7f15ec009040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:16.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:16.890+0000 7f15f2a00640 1 -- 172.21.15.45:0/3100813305 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f15f4059e70 con 0x7f15fc0d3e90 2024-04-16T16:20:17.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.094+0000 7f1600a00640 1 -- 172.21.15.45:0/3100813305 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f15fc15f2b0 con 0x7f15fc0d3e90 2024-04-16T16:20:17.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.095+0000 7f15f2a00640 1 -- 172.21.15.45:0/3100813305 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v117) ==== 100+0+11 (secure 0 0 0) 0x7f15f4059770 con 0x7f15fc0d3e90 2024-04-16T16:20:17.095 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:20:17.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.098+0000 7f1600a00640 1 -- 172.21.15.45:0/3100813305 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f15d806c980 msgr2=0x7f15d806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:17.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.098+0000 7f1600a00640 1 --2- 172.21.15.45:0/3100813305 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f15d806c980 0x7f15d806ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f15fc0b98a0 tx=0x7f15ec009040 comp rx=0 tx=0).stop 2024-04-16T16:20:17.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.098+0000 7f1600a00640 1 -- 172.21.15.45:0/3100813305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15fc0d3e90 msgr2=0x7f15fc0b91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:17.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.098+0000 7f1600a00640 1 --2- 172.21.15.45:0/3100813305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15fc0d3e90 0x7f15fc0b91f0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f15f400e7f0 tx=0x7f15f400ebf0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.098+0000 7f1600a00640 1 -- 172.21.15.45:0/3100813305 shutdown_connections 2024-04-16T16:20:17.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.098+0000 7f1600a00640 1 --2- 172.21.15.45:0/3100813305 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f15d806c980 0x7f15d806ee40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.098+0000 7f1600a00640 1 --2- 172.21.15.45:0/3100813305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15fc0d3e90 0x7f15fc0b91f0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.098+0000 7f1600a00640 1 --2- 172.21.15.45:0/3100813305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15fc0d18f0 0x7f15fc0b8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.098+0000 7f1600a00640 1 --2- 172.21.15.45:0/3100813305 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15fc0b6310 0x7f15fc0c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.098+0000 7f1600a00640 1 -- 172.21.15.45:0/3100813305 >> 172.21.15.45:0/3100813305 conn(0x7f15fc06cf90 msgr2=0x7f15fc15eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:17.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.098+0000 7f1600a00640 1 -- 172.21.15.45:0/3100813305 shutdown_connections 2024-04-16T16:20:17.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.098+0000 7f1600a00640 1 -- 172.21.15.45:0/3100813305 wait complete. 2024-04-16T16:20:17.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:20:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.338+0000 7f291f200640 1 Processor -- start 2024-04-16T16:20:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.338+0000 7f291f200640 1 -- start start 2024-04-16T16:20:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.340+0000 7f291f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f292015e8f0 0x7f29200cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.340+0000 7f291f200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29201632a0 con 0x7f29200ce9a0 2024-04-16T16:20:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.340+0000 7f291f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f292015ecd0 con 0x7f29200cf080 2024-04-16T16:20:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.340+0000 7f291f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f29200ce2e0 con 0x7f292015e8f0 2024-04-16T16:20:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.340+0000 7f291d400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f29200ce9a0 0x7f2920162480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33586/0 (socket says 172.21.15.45:33586) 2024-04-16T16:20:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.340+0000 7f291d400640 1 -- 172.21.15.45:0/908281749 learned_addr learned my addr 172.21.15.45:0/908281749 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.341+0000 7f291de00640 1 --2- 172.21.15.45:0/908281749 >> v2:172.21.15.103:3300/0 conn(0x7f292015e8f0 0x7f29200cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.341+0000 7f291ca00640 1 -- 172.21.15.45:0/908281749 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2994986219 0 0) 0x7f29201632a0 con 0x7f29200ce9a0 2024-04-16T16:20:17.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.341+0000 7f291ca00640 1 -- 172.21.15.45:0/908281749 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2904003520 con 0x7f29200ce9a0 2024-04-16T16:20:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.341+0000 7f291ca00640 1 -- 172.21.15.45:0/908281749 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3441130319 0 0) 0x7f2904003520 con 0x7f29200ce9a0 2024-04-16T16:20:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.341+0000 7f291ca00640 1 -- 172.21.15.45:0/908281749 >> v2:172.21.15.103:3300/0 conn(0x7f292015e8f0 msgr2=0x7f29200cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.341+0000 7f291ca00640 1 --2- 172.21.15.45:0/908281749 >> v2:172.21.15.103:3300/0 conn(0x7f292015e8f0 0x7f29200cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.341+0000 7f291ca00640 1 -- 172.21.15.45:0/908281749 >> v1:172.21.15.103:6789/0 conn(0x7f29200cf080 legacy=0x7f2920162b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.341+0000 7f291ca00640 1 -- 172.21.15.45:0/908281749 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29200cd6d0 con 0x7f29200ce9a0 2024-04-16T16:20:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.341+0000 7f291ca00640 1 -- 172.21.15.45:0/908281749 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f290c001670 con 0x7f29200ce9a0 2024-04-16T16:20:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.341+0000 7f291ca00640 1 -- 172.21.15.45:0/908281749 >> v1:172.21.15.45:6789/0 conn(0x7f29200ce9a0 legacy=0x7f2920162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.341+0000 7f291ca00640 1 --2- 172.21.15.45:0/908281749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29040039b0 0x7f2904003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.342+0000 7f291ca00640 1 --2- 172.21.15.45:0/908281749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2904004800 0x7f2904004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.342+0000 7f291ca00640 1 --2- 172.21.15.45:0/908281749 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29040051c0 0x7f290400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.342+0000 7f291ca00640 1 -- 172.21.15.45:0/908281749 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f29200ce2e0 con 0x7f2904004800 2024-04-16T16:20:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.342+0000 7f291ca00640 1 -- 172.21.15.45:0/908281749 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f290400c060 con 0x7f29040039b0 2024-04-16T16:20:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.342+0000 7f291ca00640 1 -- 172.21.15.45:0/908281749 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f290400c260 con 0x7f29040051c0 2024-04-16T16:20:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.342+0000 7f291d400640 1 --2- 172.21.15.45:0/908281749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2904004800 0x7f2904004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.342+0000 7f291de00640 1 --2- 172.21.15.45:0/908281749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29040039b0 0x7f2904003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.342+0000 7f291d400640 1 -- 172.21.15.45:0/908281749 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29040051c0 msgr2=0x7f290400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.342+0000 7f291d400640 1 --2- 172.21.15.45:0/908281749 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29040051c0 0x7f290400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.342+0000 7f291d400640 1 -- 172.21.15.45:0/908281749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29040039b0 msgr2=0x7f2904003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.342+0000 7f291d400640 1 --2- 172.21.15.45:0/908281749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29040039b0 0x7f2904003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.342+0000 7f291d400640 1 -- 172.21.15.45:0/908281749 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2904003520 con 0x7f2904004800 2024-04-16T16:20:17.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.342+0000 7f291e800640 1 -- 172.21.15.45:0/908281749 reap_dead start 2024-04-16T16:20:17.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.344+0000 7f291d400640 1 --2- 172.21.15.45:0/908281749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2904004800 0x7f2904004c80 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f29200cee10 tx=0x7f2904005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f270c14dc7a9b74 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:17.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.344+0000 7f291ca00640 1 -- 172.21.15.45:0/908281749 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290c007d30 con 0x7f2904004800 2024-04-16T16:20:17.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.344+0000 7f291ca00640 1 -- 172.21.15.45:0/908281749 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f290c030070 con 0x7f2904004800 2024-04-16T16:20:17.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.344+0000 7f291ca00640 1 -- 172.21.15.45:0/908281749 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290c005d70 con 0x7f2904004800 2024-04-16T16:20:17.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.345+0000 7f291f200640 1 -- 172.21.15.45:0/908281749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2904004800 msgr2=0x7f2904004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:17.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.345+0000 7f291f200640 1 --2- 172.21.15.45:0/908281749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2904004800 0x7f2904004c80 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f29200cee10 tx=0x7f2904005830 comp rx=0 tx=0).stop 2024-04-16T16:20:17.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.345+0000 7f291f200640 1 -- 172.21.15.45:0/908281749 shutdown_connections 2024-04-16T16:20:17.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.345+0000 7f291f200640 1 --2- 172.21.15.45:0/908281749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2904004800 0x7f2904004c80 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.345+0000 7f291f200640 1 -- 172.21.15.45:0/908281749 >> 172.21.15.45:0/908281749 conn(0x7f292006b930 msgr2=0x7f292006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:17.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.345+0000 7f291f200640 1 -- 172.21.15.45:0/908281749 shutdown_connections 2024-04-16T16:20:17.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.345+0000 7f291f200640 1 -- 172.21.15.45:0/908281749 wait complete. 2024-04-16T16:20:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.347+0000 7f291f200640 1 Processor -- start 2024-04-16T16:20:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.347+0000 7f291f200640 1 -- start start 2024-04-16T16:20:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.348+0000 7f291f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2904004800 0x7f292014f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.348+0000 7f291f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f292014edf0 0x7f2920151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.348+0000 7f291f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2920152330 0x7f29201527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.348+0000 7f291f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f29200cdf70 con 0x7f2904004800 2024-04-16T16:20:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.348+0000 7f291f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f29200cdc90 con 0x7f292014edf0 2024-04-16T16:20:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.348+0000 7f291f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f29200cd360 con 0x7f2920152330 2024-04-16T16:20:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.348+0000 7f291de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2904004800 0x7f292014f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:17.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.348+0000 7f291de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2904004800 0x7f292014f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38702/0 (socket says 172.21.15.45:38702) 2024-04-16T16:20:17.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.348+0000 7f291de00640 1 -- 172.21.15.45:0/1758760522 learned_addr learned my addr 172.21.15.45:0/1758760522 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:17.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.349+0000 7f291d400640 1 --2- 172.21.15.45:0/1758760522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f292014edf0 0x7f2920151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:17.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.349+0000 7f291e800640 1 --2- 172.21.15.45:0/1758760522 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2920152330 0x7f29201527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:17.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.349+0000 7f291de00640 1 -- 172.21.15.45:0/1758760522 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2920152330 msgr2=0x7f29201527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:17.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.349+0000 7f291de00640 1 --2- 172.21.15.45:0/1758760522 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2920152330 0x7f29201527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.349+0000 7f291de00640 1 -- 172.21.15.45:0/1758760522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f292014edf0 msgr2=0x7f2920151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:17.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.349+0000 7f291de00640 1 --2- 172.21.15.45:0/1758760522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f292014edf0 0x7f2920151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.349+0000 7f291de00640 1 -- 172.21.15.45:0/1758760522 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2920150b60 con 0x7f2904004800 2024-04-16T16:20:17.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.349+0000 7f291e800640 1 --2- 172.21.15.45:0/1758760522 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2920152330 0x7f29201527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:17.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.349+0000 7f291de00640 1 --2- 172.21.15.45:0/1758760522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2904004800 0x7f292014f520 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f2910011700 tx=0x7f2910011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:17.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.349+0000 7f2916000640 1 -- 172.21.15.45:0/1758760522 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291000f9c0 con 0x7f2904004800 2024-04-16T16:20:17.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.349+0000 7f2916000640 1 -- 172.21.15.45:0/1758760522 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f291000fb60 con 0x7f2904004800 2024-04-16T16:20:17.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.349+0000 7f2916000640 1 -- 172.21.15.45:0/1758760522 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910015460 con 0x7f2904004800 2024-04-16T16:20:17.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.349+0000 7f291f200640 1 -- 172.21.15.45:0/1758760522 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2920150de0 con 0x7f2904004800 2024-04-16T16:20:17.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.350+0000 7f291f200640 1 -- 172.21.15.45:0/1758760522 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29201513a0 con 0x7f2904004800 2024-04-16T16:20:17.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.351+0000 7f291f200640 1 -- 172.21.15.45:0/1758760522 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2920075780 con 0x7f2904004800 2024-04-16T16:20:17.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.352+0000 7f2916000640 1 -- 172.21.15.45:0/1758760522 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 48) ==== 91086+0+0 (secure 0 0 0) 0x7f2910015600 con 0x7f2904004800 2024-04-16T16:20:17.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.353+0000 7f2916000640 1 --2- 172.21.15.45:0/1758760522 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f290006c980 0x7f290006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:17.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.353+0000 7f2916000640 1 -- 172.21.15.45:0/1758760522 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6397+0+0 (secure 0 0 0) 0x7f291008c410 con 0x7f2904004800 2024-04-16T16:20:17.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.353+0000 7f291d400640 1 --2- 172.21.15.45:0/1758760522 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f290006c980 0x7f290006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:17.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.355+0000 7f291d400640 1 --2- 172.21.15.45:0/1758760522 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f290006c980 0x7f290006ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f290c034000 tx=0x7f290c009030 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:17.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.357+0000 7f2916000640 1 -- 172.21.15.45:0/1758760522 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2920150b60 con 0x7f2904004800 2024-04-16T16:20:17.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.602+0000 7f291f200640 1 -- 172.21.15.45:0/1758760522 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29200b5b90 con 0x7f2904004800 2024-04-16T16:20:17.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.602+0000 7f2916000640 1 -- 172.21.15.45:0/1758760522 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5186 (secure 0 0 0) 0x7f2910019030 con 0x7f2904004800 2024-04-16T16:20:17.603 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:17.603 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6056,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1032791151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":1032791151},{"type":"v1","addr":"172.21.15.45:6835","nonce":1032791151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6062,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/498835619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":498835619},{"type":"v1","addr":"172.21.15.103:6837","nonce":498835619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/3611628422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3611628422},{"type":"v1","addr":"172.21.15.103:6836","nonce":3611628422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:19:50.154928+0000","modified":"2024-04-16T16:19:53.221144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8160},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8160":{"gid":8160,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/3429980864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3429980864},{"type":"v1","addr":"172.21.15.45:6834","nonce":3429980864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8160,"qdb_cluster":[8160]},"id":8}]} 2024-04-16T16:20:17.603 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 82 2024-04-16T16:20:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.605+0000 7f291f200640 1 -- 172.21.15.45:0/1758760522 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f290006c980 msgr2=0x7f290006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:17.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.605+0000 7f291f200640 1 --2- 172.21.15.45:0/1758760522 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f290006c980 0x7f290006ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f290c034000 tx=0x7f290c009030 comp rx=0 tx=0).stop 2024-04-16T16:20:17.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.605+0000 7f291f200640 1 -- 172.21.15.45:0/1758760522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2904004800 msgr2=0x7f292014f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:17.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.605+0000 7f291f200640 1 --2- 172.21.15.45:0/1758760522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2904004800 0x7f292014f520 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f2910011700 tx=0x7f2910011bc0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.605+0000 7f291f200640 1 -- 172.21.15.45:0/1758760522 shutdown_connections 2024-04-16T16:20:17.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.605+0000 7f291f200640 1 --2- 172.21.15.45:0/1758760522 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f290006c980 0x7f290006ee40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.605+0000 7f291f200640 1 --2- 172.21.15.45:0/1758760522 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2920152330 0x7f29201527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.605+0000 7f291f200640 1 --2- 172.21.15.45:0/1758760522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f292014edf0 0x7f2920151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.605+0000 7f291f200640 1 --2- 172.21.15.45:0/1758760522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2904004800 0x7f292014f520 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.605+0000 7f291f200640 1 -- 172.21.15.45:0/1758760522 >> 172.21.15.45:0/1758760522 conn(0x7f292006b930 msgr2=0x7f292006f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:17.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.605+0000 7f291f200640 1 -- 172.21.15.45:0/1758760522 shutdown_connections 2024-04-16T16:20:17.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.605+0000 7f291f200640 1 -- 172.21.15.45:0/1758760522 wait complete. 2024-04-16T16:20:17.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:20:17.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.838+0000 7f6e94c00640 1 Processor -- start 2024-04-16T16:20:17.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.839+0000 7f6e94c00640 1 -- start start 2024-04-16T16:20:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.841+0000 7f6e94c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6e90073760 0x7f6e900d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.841+0000 7f6e94c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e900b54a0 con 0x7f6e900d0a50 2024-04-16T16:20:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.841+0000 7f6e94c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e900b5780 con 0x7f6e90159540 2024-04-16T16:20:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.841+0000 7f6e94c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f6e90159920 con 0x7f6e90073760 2024-04-16T16:20:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.841+0000 7f6e8fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6e90073760 0x7f6e900d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.841+0000 7f6e8ea00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6e900d0a50 0x7f6e90073050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33592/0 (socket says 172.21.15.45:33592) 2024-04-16T16:20:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.841+0000 7f6e8ea00640 1 -- 172.21.15.45:0/2841702275 learned_addr learned my addr 172.21.15.45:0/2841702275 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.841+0000 7f6e8fe00640 1 -- 172.21.15.45:0/2841702275 >> v1:172.21.15.103:6789/0 conn(0x7f6e90159540 legacy=0x7f6e9015bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.841+0000 7f6e8fe00640 1 -- 172.21.15.45:0/2841702275 >> v1:172.21.15.45:6789/0 conn(0x7f6e900d0a50 legacy=0x7f6e90073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.841+0000 7f6e8fe00640 1 -- 172.21.15.45:0/2841702275 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e900b5a60 con 0x7f6e90073760 2024-04-16T16:20:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.841+0000 7f6e8e000640 1 -- 172.21.15.45:0/2841702275 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3080202832 0 0) 0x7f6e900b54a0 con 0x7f6e900d0a50 2024-04-16T16:20:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.842+0000 7f6e8fe00640 1 --2- 172.21.15.45:0/2841702275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90073760 0x7f6e900d0ee0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f6e880097d0 tx=0x7f6e8802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73aa48d3127ec4e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.843+0000 7f6e8e000640 1 -- 172.21.15.45:0/2841702275 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88035020 con 0x7f6e90073760 2024-04-16T16:20:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.843+0000 7f6e8e000640 1 -- 172.21.15.45:0/2841702275 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e88031830 con 0x7f6e90073760 2024-04-16T16:20:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.843+0000 7f6e8e000640 1 -- 172.21.15.45:0/2841702275 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88031c00 con 0x7f6e90073760 2024-04-16T16:20:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.843+0000 7f6e94c00640 1 -- 172.21.15.45:0/2841702275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90073760 msgr2=0x7f6e900d0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.843+0000 7f6e94c00640 1 --2- 172.21.15.45:0/2841702275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90073760 0x7f6e900d0ee0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f6e880097d0 tx=0x7f6e8802b8d0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.843+0000 7f6e94c00640 1 -- 172.21.15.45:0/2841702275 shutdown_connections 2024-04-16T16:20:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.843+0000 7f6e94c00640 1 --2- 172.21.15.45:0/2841702275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90073760 0x7f6e900d0ee0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.843+0000 7f6e94c00640 1 -- 172.21.15.45:0/2841702275 >> 172.21.15.45:0/2841702275 conn(0x7f6e9006f0e0 msgr2=0x7f6e9006f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.843+0000 7f6e94c00640 1 -- 172.21.15.45:0/2841702275 shutdown_connections 2024-04-16T16:20:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.844+0000 7f6e94c00640 1 -- 172.21.15.45:0/2841702275 wait complete. 2024-04-16T16:20:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.845+0000 7f6e94c00640 1 Processor -- start 2024-04-16T16:20:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.845+0000 7f6e94c00640 1 -- start start 2024-04-16T16:20:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.846+0000 7f6e94c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e90073760 0x7f6e9015e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e94c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e900d0a50 0x7f6e901f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e94c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90159540 0x7f6e90143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e94c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9015f770 con 0x7f6e90159540 2024-04-16T16:20:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e94c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6e900b6020 con 0x7f6e90073760 2024-04-16T16:20:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e94c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6e9015f8f0 con 0x7f6e900d0a50 2024-04-16T16:20:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e8fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90159540 0x7f6e90143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e8fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90159540 0x7f6e90143ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38708/0 (socket says 172.21.15.45:38708) 2024-04-16T16:20:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e8fe00640 1 -- 172.21.15.45:0/3805502896 learned_addr learned my addr 172.21.15.45:0/3805502896 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e8f400640 1 --2- 172.21.15.45:0/3805502896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e90073760 0x7f6e9015e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e8ea00640 1 --2- 172.21.15.45:0/3805502896 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e900d0a50 0x7f6e901f50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e8fe00640 1 -- 172.21.15.45:0/3805502896 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e900d0a50 msgr2=0x7f6e901f50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e8fe00640 1 --2- 172.21.15.45:0/3805502896 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e900d0a50 0x7f6e901f50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e8fe00640 1 -- 172.21.15.45:0/3805502896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e90073760 msgr2=0x7f6e9015e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e8fe00640 1 --2- 172.21.15.45:0/3805502896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e90073760 0x7f6e9015e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e8fe00640 1 -- 172.21.15.45:0/3805502896 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e901f55f0 con 0x7f6e90159540 2024-04-16T16:20:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.847+0000 7f6e8f400640 1 --2- 172.21.15.45:0/3805502896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e90073760 0x7f6e9015e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.848+0000 7f6e8fe00640 1 --2- 172.21.15.45:0/3805502896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90159540 0x7f6e90143ab0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f6e880098f0 tx=0x7f6e880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.848+0000 7f6e87400640 1 -- 172.21.15.45:0/3805502896 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8803b940 con 0x7f6e90159540 2024-04-16T16:20:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.848+0000 7f6e87400640 1 -- 172.21.15.45:0/3805502896 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e8803bae0 con 0x7f6e90159540 2024-04-16T16:20:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.848+0000 7f6e87400640 1 -- 172.21.15.45:0/3805502896 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e880453f0 con 0x7f6e90159540 2024-04-16T16:20:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.848+0000 7f6e94c00640 1 -- 172.21.15.45:0/3805502896 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e90143ff0 con 0x7f6e90159540 2024-04-16T16:20:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.848+0000 7f6e94c00640 1 -- 172.21.15.45:0/3805502896 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e900c6a90 con 0x7f6e90159540 2024-04-16T16:20:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.849+0000 7f6e94c00640 1 -- 172.21.15.45:0/3805502896 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e90075780 con 0x7f6e90159540 2024-04-16T16:20:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.850+0000 7f6e87400640 1 -- 172.21.15.45:0/3805502896 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 48) ==== 91086+0+0 (secure 0 0 0) 0x7f6e88038020 con 0x7f6e90159540 2024-04-16T16:20:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.851+0000 7f6e87400640 1 --2- 172.21.15.45:0/3805502896 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f6e6c06c980 0x7f6e6c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.851+0000 7f6e87400640 1 -- 172.21.15.45:0/3805502896 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6397+0+0 (secure 0 0 0) 0x7f6e880add70 con 0x7f6e90159540 2024-04-16T16:20:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.852+0000 7f6e8f400640 1 --2- 172.21.15.45:0/3805502896 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f6e6c06c980 0x7f6e6c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.853+0000 7f6e8f400640 1 --2- 172.21.15.45:0/3805502896 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f6e6c06c980 0x7f6e6c06ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6e8000a830 tx=0x7f6e80008040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:17.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:17.856+0000 7f6e87400640 1 -- 172.21.15.45:0/3805502896 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6e88040070 con 0x7f6e90159540 2024-04-16T16:20:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.061+0000 7f6e94c00640 1 -- 172.21.15.45:0/3805502896 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e900b4bd0 con 0x7f6e90159540 2024-04-16T16:20:18.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.063+0000 7f6e87400640 1 -- 172.21.15.45:0/3805502896 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) ==== 74+0+17642 (secure 0 0 0) 0x7f6e8807bdf0 con 0x7f6e90159540 2024-04-16T16:20:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:18.064 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":117,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:20:13.465090+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:48.114834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:49.135741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/383252954":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/428120997":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/3229718691":"2024-04-17T16:20:13.464929+0000","172.21.15.103:6833/1934318167":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/3469935364":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/3506765807":"2024-04-17T16:20:07.040813+0000","172.21.15.103:6832/1934318167":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/3547880684":"2024-04-17T16:20:07.040813+0000","172.21.15.103:6833/2293932214":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/1890933611":"2024-04-17T16:20:13.464929+0000","172.21.15.103:6832/2293932214":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/1347839066":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/546409297":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/3566291251":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/714185246":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/2784388736":"2024-04-16T17:19:54.469649+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:20:18.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.065+0000 7f6e94c00640 1 -- 172.21.15.45:0/3805502896 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f6e6c06c980 msgr2=0x7f6e6c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:18.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.065+0000 7f6e94c00640 1 --2- 172.21.15.45:0/3805502896 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f6e6c06c980 0x7f6e6c06ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6e8000a830 tx=0x7f6e80008040 comp rx=0 tx=0).stop 2024-04-16T16:20:18.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.065+0000 7f6e94c00640 1 -- 172.21.15.45:0/3805502896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90159540 msgr2=0x7f6e90143ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:18.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.065+0000 7f6e94c00640 1 --2- 172.21.15.45:0/3805502896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90159540 0x7f6e90143ab0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f6e880098f0 tx=0x7f6e880046c0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.066+0000 7f6e94c00640 1 -- 172.21.15.45:0/3805502896 shutdown_connections 2024-04-16T16:20:18.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.066+0000 7f6e94c00640 1 --2- 172.21.15.45:0/3805502896 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f6e6c06c980 0x7f6e6c06ee40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.066+0000 7f6e94c00640 1 --2- 172.21.15.45:0/3805502896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90159540 0x7f6e90143ab0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.066+0000 7f6e94c00640 1 --2- 172.21.15.45:0/3805502896 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e900d0a50 0x7f6e901f50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.066+0000 7f6e94c00640 1 --2- 172.21.15.45:0/3805502896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e90073760 0x7f6e9015e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.066+0000 7f6e94c00640 1 -- 172.21.15.45:0/3805502896 >> 172.21.15.45:0/3805502896 conn(0x7f6e9006f0e0 msgr2=0x7f6e900b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.066+0000 7f6e94c00640 1 -- 172.21.15.45:0/3805502896 shutdown_connections 2024-04-16T16:20:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.066+0000 7f6e94c00640 1 -- 172.21.15.45:0/3805502896 wait complete. 2024-04-16T16:20:18.085 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-16T16:20:18.085 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:20:18.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.315+0000 7f4264c00640 1 Processor -- start 2024-04-16T16:20:18.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.315+0000 7f4264c00640 1 -- start start 2024-04-16T16:20:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.316+0000 7f4264c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f426015e8f0 0x7f42600cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.316+0000 7f4264c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42601632a0 con 0x7f42600cf080 2024-04-16T16:20:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.317+0000 7f4264c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f426015ecd0 con 0x7f42600ce9a0 2024-04-16T16:20:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.317+0000 7f4264c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f42600ce2e0 con 0x7f426015e8f0 2024-04-16T16:20:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.317+0000 7f425f400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f426015e8f0 0x7f42600cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.317+0000 7f425f400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f426015e8f0 0x7f42600cc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38722/0 (socket says 172.21.15.45:38722) 2024-04-16T16:20:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.317+0000 7f425fe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f42600cf080 0x7f4260162b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33598/0 (socket says 172.21.15.45:33598) 2024-04-16T16:20:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.317+0000 7f425f400640 1 -- 172.21.15.45:0/325142431 learned_addr learned my addr 172.21.15.45:0/325142431 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:18.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.317+0000 7f425f400640 1 -- 172.21.15.45:0/325142431 >> v1:172.21.15.103:6789/0 conn(0x7f42600ce9a0 legacy=0x7f4260162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:18.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.317+0000 7f425f400640 1 -- 172.21.15.45:0/325142431 >> v1:172.21.15.45:6789/0 conn(0x7f42600cf080 legacy=0x7f4260162b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:18.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.318+0000 7f425f400640 1 -- 172.21.15.45:0/325142431 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42600cd6d0 con 0x7f426015e8f0 2024-04-16T16:20:18.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.319+0000 7f425f400640 1 --2- 172.21.15.45:0/325142431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f426015e8f0 0x7f42600cc410 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f4250009870 tx=0x7f425002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc782feffb4879e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:18.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.319+0000 7f425ea00640 1 -- 172.21.15.45:0/325142431 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4250034070 con 0x7f426015e8f0 2024-04-16T16:20:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.320+0000 7f425ea00640 1 -- 172.21.15.45:0/325142431 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4250030860 con 0x7f426015e8f0 2024-04-16T16:20:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.320+0000 7f425ea00640 1 -- 172.21.15.45:0/325142431 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4250030c30 con 0x7f426015e8f0 2024-04-16T16:20:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.320+0000 7f4264c00640 1 -- 172.21.15.45:0/325142431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f426015e8f0 msgr2=0x7f42600cc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.320+0000 7f4264c00640 1 --2- 172.21.15.45:0/325142431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f426015e8f0 0x7f42600cc410 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f4250009870 tx=0x7f425002b9a0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.320+0000 7f4264c00640 1 -- 172.21.15.45:0/325142431 shutdown_connections 2024-04-16T16:20:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.320+0000 7f4264c00640 1 --2- 172.21.15.45:0/325142431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f426015e8f0 0x7f42600cc410 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.320+0000 7f4264c00640 1 -- 172.21.15.45:0/325142431 >> 172.21.15.45:0/325142431 conn(0x7f426006b930 msgr2=0x7f426006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.320+0000 7f4264c00640 1 -- 172.21.15.45:0/325142431 shutdown_connections 2024-04-16T16:20:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.320+0000 7f4264c00640 1 -- 172.21.15.45:0/325142431 wait complete. 2024-04-16T16:20:18.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.322+0000 7f4264c00640 1 Processor -- start 2024-04-16T16:20:18.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.322+0000 7f4264c00640 1 -- start start 2024-04-16T16:20:18.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.323+0000 7f4264c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42600ce9a0 0x7f426013ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:18.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.323+0000 7f4264c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42600cf080 0x7f426015a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:18.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.323+0000 7f4264c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f426015e8f0 0x7f4260161450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:18.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.323+0000 7f4264c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f42600cdf70 con 0x7f426015e8f0 2024-04-16T16:20:18.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.323+0000 7f4264c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f42600cdc90 con 0x7f42600ce9a0 2024-04-16T16:20:18.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.323+0000 7f4264c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f42600cd360 con 0x7f42600cf080 2024-04-16T16:20:18.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.324+0000 7f425fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f426015e8f0 0x7f4260161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:18.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.324+0000 7f425fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f426015e8f0 0x7f4260161450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38726/0 (socket says 172.21.15.45:38726) 2024-04-16T16:20:18.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.324+0000 7f425fe00640 1 -- 172.21.15.45:0/4154181947 learned_addr learned my addr 172.21.15.45:0/4154181947 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:18.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.324+0000 7f425f400640 1 --2- 172.21.15.45:0/4154181947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42600ce9a0 0x7f426013ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:18.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.324+0000 7f4256a00640 1 --2- 172.21.15.45:0/4154181947 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42600cf080 0x7f426015a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:18.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.324+0000 7f425fe00640 1 -- 172.21.15.45:0/4154181947 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42600cf080 msgr2=0x7f426015a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:18.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.324+0000 7f425fe00640 1 --2- 172.21.15.45:0/4154181947 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42600cf080 0x7f426015a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.324+0000 7f425fe00640 1 -- 172.21.15.45:0/4154181947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42600ce9a0 msgr2=0x7f426013ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:18.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.324+0000 7f425fe00640 1 --2- 172.21.15.45:0/4154181947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42600ce9a0 0x7f426013ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.324+0000 7f425fe00640 1 -- 172.21.15.45:0/4154181947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42601404e0 con 0x7f426015e8f0 2024-04-16T16:20:18.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.324+0000 7f4256a00640 1 --2- 172.21.15.45:0/4154181947 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42600cf080 0x7f426015a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:20:18.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.325+0000 7f425fe00640 1 --2- 172.21.15.45:0/4154181947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f426015e8f0 0x7f4260161450 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f425800c400 tx=0x7f425800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:18.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.325+0000 7f4257e00640 1 -- 172.21.15.45:0/4154181947 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42580074e0 con 0x7f426015e8f0 2024-04-16T16:20:18.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.325+0000 7f4257e00640 1 -- 172.21.15.45:0/4154181947 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4258007680 con 0x7f426015e8f0 2024-04-16T16:20:18.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.325+0000 7f4257e00640 1 -- 172.21.15.45:0/4154181947 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42580049e0 con 0x7f426015e8f0 2024-04-16T16:20:18.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.325+0000 7f4264c00640 1 -- 172.21.15.45:0/4154181947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42601603b0 con 0x7f426015e8f0 2024-04-16T16:20:18.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.325+0000 7f4264c00640 1 -- 172.21.15.45:0/4154181947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4260160970 con 0x7f426015e8f0 2024-04-16T16:20:18.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.326+0000 7f4264c00640 1 -- 172.21.15.45:0/4154181947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4260075780 con 0x7f426015e8f0 2024-04-16T16:20:18.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.327+0000 7f4257e00640 1 -- 172.21.15.45:0/4154181947 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f425801a020 con 0x7f426015e8f0 2024-04-16T16:20:18.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.328+0000 7f4257e00640 1 --2- 172.21.15.45:0/4154181947 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f423c06ca10 0x7f423c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:18.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.328+0000 7f4257e00640 1 -- 172.21.15.45:0/4154181947 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6397+0+0 (secure 0 0 0) 0x7f425808b830 con 0x7f426015e8f0 2024-04-16T16:20:18.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.329+0000 7f425f400640 1 --2- 172.21.15.45:0/4154181947 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f423c06ca10 0x7f423c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:18.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.330+0000 7f425f400640 1 --2- 172.21.15.45:0/4154181947 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f423c06ca10 0x7f423c06eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4250036000 tx=0x7f425003b040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:18.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.333+0000 7f4257e00640 1 -- 172.21.15.45:0/4154181947 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4258059770 con 0x7f426015e8f0 2024-04-16T16:20:18.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.576+0000 7f4264c00640 1 -- 172.21.15.45:0/4154181947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42600b5b90 con 0x7f426015e8f0 2024-04-16T16:20:18.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.577+0000 7f4257e00640 1 -- 172.21.15.45:0/4154181947 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5186 (secure 0 0 0) 0x7f4258059070 con 0x7f426015e8f0 2024-04-16T16:20:18.578 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:18.578 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6056,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1032791151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":1032791151},{"type":"v1","addr":"172.21.15.45:6835","nonce":1032791151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6062,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/498835619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":498835619},{"type":"v1","addr":"172.21.15.103:6837","nonce":498835619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/3611628422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3611628422},{"type":"v1","addr":"172.21.15.103:6836","nonce":3611628422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:19:50.154928+0000","modified":"2024-04-16T16:19:53.221144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8160},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8160":{"gid":8160,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/3429980864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3429980864},{"type":"v1","addr":"172.21.15.45:6834","nonce":3429980864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8160,"qdb_cluster":[8160]},"id":8}]} 2024-04-16T16:20:18.578 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 82 2024-04-16T16:20:18.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.580+0000 7f4264c00640 1 -- 172.21.15.45:0/4154181947 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f423c06ca10 msgr2=0x7f423c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:18.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.580+0000 7f4264c00640 1 --2- 172.21.15.45:0/4154181947 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f423c06ca10 0x7f423c06eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4250036000 tx=0x7f425003b040 comp rx=0 tx=0).stop 2024-04-16T16:20:18.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.580+0000 7f4264c00640 1 -- 172.21.15.45:0/4154181947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f426015e8f0 msgr2=0x7f4260161450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:18.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.580+0000 7f4264c00640 1 --2- 172.21.15.45:0/4154181947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f426015e8f0 0x7f4260161450 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f425800c400 tx=0x7f425800c8c0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.580+0000 7f4264c00640 1 -- 172.21.15.45:0/4154181947 shutdown_connections 2024-04-16T16:20:18.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.580+0000 7f4264c00640 1 --2- 172.21.15.45:0/4154181947 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f423c06ca10 0x7f423c06eed0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.580+0000 7f4264c00640 1 --2- 172.21.15.45:0/4154181947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f426015e8f0 0x7f4260161450 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.580+0000 7f4264c00640 1 --2- 172.21.15.45:0/4154181947 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42600cf080 0x7f426015a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.580+0000 7f4264c00640 1 --2- 172.21.15.45:0/4154181947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42600ce9a0 0x7f426013ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.580+0000 7f4264c00640 1 -- 172.21.15.45:0/4154181947 >> 172.21.15.45:0/4154181947 conn(0x7f426006b930 msgr2=0x7f42600cf4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:18.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.580+0000 7f4264c00640 1 -- 172.21.15.45:0/4154181947 shutdown_connections 2024-04-16T16:20:18.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.580+0000 7f4264c00640 1 -- 172.21.15.45:0/4154181947 wait complete. 2024-04-16T16:20:18.601 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:20:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.823+0000 7f3bd1400640 1 Processor -- start 2024-04-16T16:20:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.823+0000 7f3bd1400640 1 -- start start 2024-04-16T16:20:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.825+0000 7f3bd1400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3bcc15ccb0 0x7f3bcc06df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.825+0000 7f3bd1400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bcc15d360 con 0x7f3bcc146220 2024-04-16T16:20:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.825+0000 7f3bd1400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bcc15d640 con 0x7f3bcc06e480 2024-04-16T16:20:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.825+0000 7f3bd1400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3bcc15d130 con 0x7f3bcc15ccb0 2024-04-16T16:20:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.825+0000 7f3bcbe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3bcc146220 0x7f3bcc06f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33608/0 (socket says 172.21.15.45:33608) 2024-04-16T16:20:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.825+0000 7f3bcbe00640 1 -- 172.21.15.45:0/2117038747 learned_addr learned my addr 172.21.15.45:0/2117038747 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.825+0000 7f3bcb400640 1 --2- 172.21.15.45:0/2117038747 >> v2:172.21.15.103:3300/0 conn(0x7f3bcc15ccb0 0x7f3bcc06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.826+0000 7f3bcaa00640 1 -- 172.21.15.45:0/2117038747 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3752565686 0 0) 0x7f3bcc15d360 con 0x7f3bcc146220 2024-04-16T16:20:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.826+0000 7f3bcaa00640 1 -- 172.21.15.45:0/2117038747 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bb0003520 con 0x7f3bcc146220 2024-04-16T16:20:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.826+0000 7f3bcaa00640 1 -- 172.21.15.45:0/2117038747 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 41691556 0 0) 0x7f3bb0003520 con 0x7f3bcc146220 2024-04-16T16:20:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.826+0000 7f3bcaa00640 1 -- 172.21.15.45:0/2117038747 >> v2:172.21.15.103:3300/0 conn(0x7f3bcc15ccb0 msgr2=0x7f3bcc06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.826+0000 7f3bcaa00640 1 --2- 172.21.15.45:0/2117038747 >> v2:172.21.15.103:3300/0 conn(0x7f3bcc15ccb0 0x7f3bcc06df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.826+0000 7f3bcaa00640 1 -- 172.21.15.45:0/2117038747 >> v1:172.21.15.103:6789/0 conn(0x7f3bcc06e480 legacy=0x7f3bcc1608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.826+0000 7f3bcaa00640 1 -- 172.21.15.45:0/2117038747 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bcc15d920 con 0x7f3bcc146220 2024-04-16T16:20:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.826+0000 7f3bcaa00640 1 -- 172.21.15.45:0/2117038747 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f3bbc001670 con 0x7f3bcc146220 2024-04-16T16:20:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.826+0000 7f3bcaa00640 1 -- 172.21.15.45:0/2117038747 >> v1:172.21.15.45:6789/0 conn(0x7f3bcc146220 legacy=0x7f3bcc06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.826+0000 7f3bcaa00640 1 --2- 172.21.15.45:0/2117038747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bb0003a00 0x7f3bb0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.826+0000 7f3bcaa00640 1 --2- 172.21.15.45:0/2117038747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bb0004800 0x7f3bb0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.826+0000 7f3bcaa00640 1 --2- 172.21.15.45:0/2117038747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb00051c0 0x7f3bb000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.826+0000 7f3bcaa00640 1 -- 172.21.15.45:0/2117038747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb000c090 con 0x7f3bb00051c0 2024-04-16T16:20:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.826+0000 7f3bcaa00640 1 -- 172.21.15.45:0/2117038747 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb000c240 con 0x7f3bb0004800 2024-04-16T16:20:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.826+0000 7f3bcaa00640 1 -- 172.21.15.45:0/2117038747 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3bb000c440 con 0x7f3bb0003a00 2024-04-16T16:20:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.827+0000 7f3bd0a00640 1 --2- 172.21.15.45:0/2117038747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb00051c0 0x7f3bb000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.827+0000 7f3bcb400640 1 --2- 172.21.15.45:0/2117038747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bb0004800 0x7f3bb0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.827+0000 7f3bcbe00640 1 --2- 172.21.15.45:0/2117038747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bb0003a00 0x7f3bb0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.827+0000 7f3bd0a00640 1 -- 172.21.15.45:0/2117038747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bb0003a00 msgr2=0x7f3bb0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.827+0000 7f3bd0a00640 1 --2- 172.21.15.45:0/2117038747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bb0003a00 0x7f3bb0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.827+0000 7f3bd0a00640 1 -- 172.21.15.45:0/2117038747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bb0004800 msgr2=0x7f3bb0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.827+0000 7f3bd0a00640 1 --2- 172.21.15.45:0/2117038747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bb0004800 0x7f3bb0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.827+0000 7f3bd0a00640 1 -- 172.21.15.45:0/2117038747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb0003520 con 0x7f3bb00051c0 2024-04-16T16:20:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.827+0000 7f3bcb400640 1 --2- 172.21.15.45:0/2117038747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bb0004800 0x7f3bb0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:20:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.828+0000 7f3bd0a00640 1 -- 172.21.15.45:0/2117038747 reap_dead start 2024-04-16T16:20:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.828+0000 7f3bd0a00640 1 --2- 172.21.15.45:0/2117038747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb00051c0 0x7f3bb000b9c0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4001e50 tx=0x7f3bb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786617198b567d9d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.829+0000 7f3bcaa00640 1 -- 172.21.15.45:0/2117038747 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4001a70 con 0x7f3bb00051c0 2024-04-16T16:20:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.829+0000 7f3bcaa00640 1 -- 172.21.15.45:0/2117038747 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bc4003bb0 con 0x7f3bb00051c0 2024-04-16T16:20:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.829+0000 7f3bcaa00640 1 -- 172.21.15.45:0/2117038747 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc40050d0 con 0x7f3bb00051c0 2024-04-16T16:20:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.829+0000 7f3bd1400640 1 -- 172.21.15.45:0/2117038747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb00051c0 msgr2=0x7f3bb000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.829+0000 7f3bd1400640 1 --2- 172.21.15.45:0/2117038747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb00051c0 0x7f3bb000b9c0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4001e50 tx=0x7f3bb0005830 comp rx=0 tx=0).stop 2024-04-16T16:20:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.829+0000 7f3bd1400640 1 -- 172.21.15.45:0/2117038747 shutdown_connections 2024-04-16T16:20:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.829+0000 7f3bd1400640 1 --2- 172.21.15.45:0/2117038747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb00051c0 0x7f3bb000b9c0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.829+0000 7f3bd1400640 1 -- 172.21.15.45:0/2117038747 >> 172.21.15.45:0/2117038747 conn(0x7f3bcc06b930 msgr2=0x7f3bcc06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.829+0000 7f3bd1400640 1 -- 172.21.15.45:0/2117038747 shutdown_connections 2024-04-16T16:20:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.830+0000 7f3bd1400640 1 -- 172.21.15.45:0/2117038747 wait complete. 2024-04-16T16:20:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.831+0000 7f3bd1400640 1 Processor -- start 2024-04-16T16:20:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.831+0000 7f3bd1400640 1 -- start start 2024-04-16T16:20:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.832+0000 7f3bd1400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bb00051c0 0x7f3bcc1f7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.832+0000 7f3bd1400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bcc0b4200 0x7f3bcc0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.832+0000 7f3bd1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bcc13e240 0x7f3bcc13e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.832+0000 7f3bd1400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc06df40 con 0x7f3bcc13e240 2024-04-16T16:20:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.832+0000 7f3bd1400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc160410 con 0x7f3bb00051c0 2024-04-16T16:20:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.832+0000 7f3bd1400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3bcc06e0c0 con 0x7f3bcc0b4200 2024-04-16T16:20:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.833+0000 7f3bd0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bcc13e240 0x7f3bcc13e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.833+0000 7f3bd0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bcc13e240 0x7f3bcc13e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38756/0 (socket says 172.21.15.45:38756) 2024-04-16T16:20:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.833+0000 7f3bd0a00640 1 -- 172.21.15.45:0/3582380198 learned_addr learned my addr 172.21.15.45:0/3582380198 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.833+0000 7f3bcbe00640 1 --2- 172.21.15.45:0/3582380198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bb00051c0 0x7f3bcc1f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.833+0000 7f3bcb400640 1 --2- 172.21.15.45:0/3582380198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bcc0b4200 0x7f3bcc0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.833+0000 7f3bd0a00640 1 -- 172.21.15.45:0/3582380198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bcc0b4200 msgr2=0x7f3bcc0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.833+0000 7f3bd0a00640 1 --2- 172.21.15.45:0/3582380198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bcc0b4200 0x7f3bcc0b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.833+0000 7f3bd0a00640 1 -- 172.21.15.45:0/3582380198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bb00051c0 msgr2=0x7f3bcc1f7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.833+0000 7f3bd0a00640 1 --2- 172.21.15.45:0/3582380198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bb00051c0 0x7f3bcc1f7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.833+0000 7f3bd0a00640 1 -- 172.21.15.45:0/3582380198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bcc13ec00 con 0x7f3bcc13e240 2024-04-16T16:20:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.834+0000 7f3bd0a00640 1 --2- 172.21.15.45:0/3582380198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bcc13e240 0x7f3bcc13e6c0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4001f70 tx=0x7f3bc40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.834+0000 7f3bc3e00640 1 -- 172.21.15.45:0/3582380198 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4006d60 con 0x7f3bcc13e240 2024-04-16T16:20:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.834+0000 7f3bc3e00640 1 -- 172.21.15.45:0/3582380198 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bc4039480 con 0x7f3bcc13e240 2024-04-16T16:20:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.834+0000 7f3bc3e00640 1 -- 172.21.15.45:0/3582380198 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4039830 con 0x7f3bcc13e240 2024-04-16T16:20:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.834+0000 7f3bd1400640 1 -- 172.21.15.45:0/3582380198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bcc138e20 con 0x7f3bcc13e240 2024-04-16T16:20:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.834+0000 7f3bd1400640 1 -- 172.21.15.45:0/3582380198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bcc1393e0 con 0x7f3bcc13e240 2024-04-16T16:20:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.836+0000 7f3bd1400640 1 -- 172.21.15.45:0/3582380198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bcc15d360 con 0x7f3bcc13e240 2024-04-16T16:20:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.836+0000 7f3bc3e00640 1 -- 172.21.15.45:0/3582380198 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f3bc40399d0 con 0x7f3bcc13e240 2024-04-16T16:20:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.837+0000 7f3bc3e00640 1 --2- 172.21.15.45:0/3582380198 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f3bac06ca10 0x7f3bac06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.838+0000 7f3bc3e00640 1 -- 172.21.15.45:0/3582380198 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6397+0+0 (secure 0 0 0) 0x7f3bc40addb0 con 0x7f3bcc13e240 2024-04-16T16:20:18.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.838+0000 7f3bcbe00640 1 --2- 172.21.15.45:0/3582380198 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f3bac06ca10 0x7f3bac06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:18.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.839+0000 7f3bcbe00640 1 --2- 172.21.15.45:0/3582380198 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f3bac06ca10 0x7f3bac06eed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc001050 tx=0x7f3bbc009040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:18.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:18.842+0000 7f3bc3e00640 1 -- 172.21.15.45:0/3582380198 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3bcc15d360 con 0x7f3bcc13e240 2024-04-16T16:20:19.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.082+0000 7f3bd1400640 1 -- 172.21.15.45:0/3582380198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bcc0cff60 con 0x7f3bcc13e240 2024-04-16T16:20:19.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.083+0000 7f3bc3e00640 1 -- 172.21.15.45:0/3582380198 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5186 (secure 0 0 0) 0x7f3bc403e070 con 0x7f3bcc13e240 2024-04-16T16:20:19.083 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:19.083 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6056,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1032791151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":1032791151},{"type":"v1","addr":"172.21.15.45:6835","nonce":1032791151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6062,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/498835619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":498835619},{"type":"v1","addr":"172.21.15.103:6837","nonce":498835619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/3611628422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3611628422},{"type":"v1","addr":"172.21.15.103:6836","nonce":3611628422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:19:50.154928+0000","modified":"2024-04-16T16:19:53.221144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8160},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8160":{"gid":8160,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/3429980864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3429980864},{"type":"v1","addr":"172.21.15.45:6834","nonce":3429980864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8160,"qdb_cluster":[8160]},"id":8}]} 2024-04-16T16:20:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 82 2024-04-16T16:20:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.085+0000 7f3bd1400640 1 -- 172.21.15.45:0/3582380198 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f3bac06ca10 msgr2=0x7f3bac06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.085+0000 7f3bd1400640 1 --2- 172.21.15.45:0/3582380198 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f3bac06ca10 0x7f3bac06eed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc001050 tx=0x7f3bbc009040 comp rx=0 tx=0).stop 2024-04-16T16:20:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.085+0000 7f3bd1400640 1 -- 172.21.15.45:0/3582380198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bcc13e240 msgr2=0x7f3bcc13e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.085+0000 7f3bd1400640 1 --2- 172.21.15.45:0/3582380198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bcc13e240 0x7f3bcc13e6c0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4001f70 tx=0x7f3bc40014a0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.086+0000 7f3bd1400640 1 -- 172.21.15.45:0/3582380198 shutdown_connections 2024-04-16T16:20:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.086+0000 7f3bd1400640 1 --2- 172.21.15.45:0/3582380198 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f3bac06ca10 0x7f3bac06eed0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.086+0000 7f3bd1400640 1 --2- 172.21.15.45:0/3582380198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bcc13e240 0x7f3bcc13e6c0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.086+0000 7f3bd1400640 1 --2- 172.21.15.45:0/3582380198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bcc0b4200 0x7f3bcc0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.086+0000 7f3bd1400640 1 --2- 172.21.15.45:0/3582380198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bb00051c0 0x7f3bcc1f7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.086+0000 7f3bd1400640 1 -- 172.21.15.45:0/3582380198 >> 172.21.15.45:0/3582380198 conn(0x7f3bcc06b930 msgr2=0x7f3bcc06f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.086+0000 7f3bd1400640 1 -- 172.21.15.45:0/3582380198 shutdown_connections 2024-04-16T16:20:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.086+0000 7f3bd1400640 1 -- 172.21.15.45:0/3582380198 wait complete. 2024-04-16T16:20:19.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:20:19.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.310+0000 7f51bf200640 1 Processor -- start 2024-04-16T16:20:19.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.310+0000 7f51bf200640 1 -- start start 2024-04-16T16:20:19.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.311+0000 7f51bf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51c0161e90 0x7f51c00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:19.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.311+0000 7f51bf200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f51c0162270 0x7f51c00d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:19.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.311+0000 7f51bf200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51c00d3080 con 0x7f51c00d52d0 2024-04-16T16:20:19.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.311+0000 7f51bf200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f51c00d2e30 con 0x7f51c0162270 2024-04-16T16:20:19.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.311+0000 7f51bf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f51c00d3360 con 0x7f51c0161e90 2024-04-16T16:20:19.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.312+0000 7f51bd400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f51c0162270 0x7f51c00d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:19.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.312+0000 7f51bde00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51c0161e90 0x7f51c00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:19.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.312+0000 7f51bde00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51c0161e90 0x7f51c00cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:44434/0 (socket says 172.21.15.45:44434) 2024-04-16T16:20:19.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.312+0000 7f51bde00640 1 -- 172.21.15.45:0/715855076 learned_addr learned my addr 172.21.15.45:0/715855076 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:19.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.313+0000 7f51bde00640 1 -- 172.21.15.45:0/715855076 >> v2:172.21.15.103:3300/0 conn(0x7f51c0162270 msgr2=0x7f51c00d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:19.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.313+0000 7f51bde00640 1 --2- 172.21.15.45:0/715855076 >> v2:172.21.15.103:3300/0 conn(0x7f51c0162270 0x7f51c00d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.313+0000 7f51bde00640 1 -- 172.21.15.45:0/715855076 >> v1:172.21.15.103:6789/0 conn(0x7f51c00d52d0 legacy=0x7f51c006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:19.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.313+0000 7f51bde00640 1 -- 172.21.15.45:0/715855076 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c00d3640 con 0x7f51c0161e90 2024-04-16T16:20:19.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.314+0000 7f51bde00640 1 --2- 172.21.15.45:0/715855076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51c0161e90 0x7f51c00cf7d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f51b0009e50 tx=0x7f51b002bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dc4a5173b531f7b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.315+0000 7f51bca00640 1 -- 172.21.15.45:0/715855076 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b002cb90 con 0x7f51c0161e90 2024-04-16T16:20:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.316+0000 7f51bca00640 1 -- 172.21.15.45:0/715855076 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51b002cd30 con 0x7f51c0161e90 2024-04-16T16:20:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.316+0000 7f51bca00640 1 -- 172.21.15.45:0/715855076 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0034610 con 0x7f51c0161e90 2024-04-16T16:20:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.316+0000 7f51bf200640 1 -- 172.21.15.45:0/715855076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51c0161e90 msgr2=0x7f51c00cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.316+0000 7f51bf200640 1 --2- 172.21.15.45:0/715855076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51c0161e90 0x7f51c00cf7d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f51b0009e50 tx=0x7f51b002bff0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.316+0000 7f51bf200640 1 -- 172.21.15.45:0/715855076 shutdown_connections 2024-04-16T16:20:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.316+0000 7f51bf200640 1 --2- 172.21.15.45:0/715855076 >> v2:172.21.15.103:3300/0 conn(0x7f51c0162270 0x7f51c00d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.316+0000 7f51bf200640 1 --2- 172.21.15.45:0/715855076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51c0161e90 0x7f51c00cf7d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.316+0000 7f51bf200640 1 -- 172.21.15.45:0/715855076 >> 172.21.15.45:0/715855076 conn(0x7f51c00d1940 msgr2=0x7f51c00d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:19.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.316+0000 7f51bf200640 1 -- 172.21.15.45:0/715855076 shutdown_connections 2024-04-16T16:20:19.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.316+0000 7f51bf200640 1 -- 172.21.15.45:0/715855076 wait complete. 2024-04-16T16:20:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.318+0000 7f51bf200640 1 Processor -- start 2024-04-16T16:20:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.318+0000 7f51bf200640 1 -- start start 2024-04-16T16:20:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.319+0000 7f51bf200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51c00d52d0 0x7f51c00c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.319+0000 7f51bf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51c0161e90 0x7f51c014a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.319+0000 7f51bf200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51c0162270 0x7f51c01e7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.319+0000 7f51bf200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f51c006d550 con 0x7f51c0162270 2024-04-16T16:20:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.319+0000 7f51bf200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f51c00cfd10 con 0x7f51c00d52d0 2024-04-16T16:20:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.319+0000 7f51bf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f51c006d6d0 con 0x7f51c0161e90 2024-04-16T16:20:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.319+0000 7f51be800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51c0162270 0x7f51c01e7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.319+0000 7f51be800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51c0162270 0x7f51c01e7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38764/0 (socket says 172.21.15.45:38764) 2024-04-16T16:20:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.319+0000 7f51be800640 1 -- 172.21.15.45:0/3891880471 learned_addr learned my addr 172.21.15.45:0/3891880471 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.320+0000 7f51bde00640 1 --2- 172.21.15.45:0/3891880471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51c00d52d0 0x7f51c00c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.320+0000 7f51bd400640 1 --2- 172.21.15.45:0/3891880471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51c0161e90 0x7f51c014a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.320+0000 7f51be800640 1 -- 172.21.15.45:0/3891880471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51c0161e90 msgr2=0x7f51c014a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:19.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.320+0000 7f51be800640 1 --2- 172.21.15.45:0/3891880471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51c0161e90 0x7f51c014a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.320+0000 7f51be800640 1 -- 172.21.15.45:0/3891880471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51c00d52d0 msgr2=0x7f51c00c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:19.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.320+0000 7f51be800640 1 --2- 172.21.15.45:0/3891880471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51c00d52d0 0x7f51c00c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.320+0000 7f51be800640 1 -- 172.21.15.45:0/3891880471 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c00c9c00 con 0x7f51c0162270 2024-04-16T16:20:19.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.320+0000 7f51bd400640 1 --2- 172.21.15.45:0/3891880471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51c0161e90 0x7f51c014a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:19.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.320+0000 7f51be800640 1 --2- 172.21.15.45:0/3891880471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51c0162270 0x7f51c01e7360 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f51b8001eb0 tx=0x7f51b800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.320+0000 7f51b6000640 1 -- 172.21.15.45:0/3891880471 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b8017070 con 0x7f51c0162270 2024-04-16T16:20:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.320+0000 7f51b6000640 1 -- 172.21.15.45:0/3891880471 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51b80052d0 con 0x7f51c0162270 2024-04-16T16:20:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.320+0000 7f51b6000640 1 -- 172.21.15.45:0/3891880471 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b8003810 con 0x7f51c0162270 2024-04-16T16:20:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.320+0000 7f51bf200640 1 -- 172.21.15.45:0/3891880471 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51c00c9ee0 con 0x7f51c0162270 2024-04-16T16:20:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.321+0000 7f51bf200640 1 -- 172.21.15.45:0/3891880471 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51c00ca4a0 con 0x7f51c0162270 2024-04-16T16:20:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.322+0000 7f51bf200640 1 -- 172.21.15.45:0/3891880471 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51c0075780 con 0x7f51c0162270 2024-04-16T16:20:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.323+0000 7f51b6000640 1 -- 172.21.15.45:0/3891880471 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f51b801a070 con 0x7f51c0162270 2024-04-16T16:20:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.324+0000 7f51b6000640 1 --2- 172.21.15.45:0/3891880471 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f519c06ca10 0x7f519c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.324+0000 7f51b6000640 1 -- 172.21.15.45:0/3891880471 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6397+0+0 (secure 0 0 0) 0x7f51b808b470 con 0x7f51c0162270 2024-04-16T16:20:19.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.324+0000 7f51bde00640 1 --2- 172.21.15.45:0/3891880471 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f519c06ca10 0x7f519c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:19.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.325+0000 7f51bde00640 1 --2- 172.21.15.45:0/3891880471 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f519c06ca10 0x7f519c06eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f51b0035000 tx=0x7f51b0030000 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:19.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.328+0000 7f51b6000640 1 -- 172.21.15.45:0/3891880471 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f51b80593c0 con 0x7f51c0162270 2024-04-16T16:20:19.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.534+0000 7f51bf200640 1 -- 172.21.15.45:0/3891880471 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f51c00b9bc0 con 0x7f51c0162270 2024-04-16T16:20:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.536+0000 7f51b6000640 1 -- 172.21.15.45:0/3891880471 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) ==== 74+0+17642 (secure 0 0 0) 0x7f51b8058cc0 con 0x7f51c0162270 2024-04-16T16:20:19.536 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:19.536 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":117,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:20:13.465090+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:19:48.114834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-16T16:19:49.135741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/383252954":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/428120997":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/3229718691":"2024-04-17T16:20:13.464929+0000","172.21.15.103:6833/1934318167":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/3469935364":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/3506765807":"2024-04-17T16:20:07.040813+0000","172.21.15.103:6832/1934318167":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/3547880684":"2024-04-17T16:20:07.040813+0000","172.21.15.103:6833/2293932214":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/1890933611":"2024-04-17T16:20:13.464929+0000","172.21.15.103:6832/2293932214":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/1347839066":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/546409297":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/3566291251":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/714185246":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/2784388736":"2024-04-16T17:19:54.469649+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:20:19.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.538+0000 7f51bf200640 1 -- 172.21.15.45:0/3891880471 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f519c06ca10 msgr2=0x7f519c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:19.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.539+0000 7f51bf200640 1 --2- 172.21.15.45:0/3891880471 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f519c06ca10 0x7f519c06eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f51b0035000 tx=0x7f51b0030000 comp rx=0 tx=0).stop 2024-04-16T16:20:19.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.539+0000 7f51bf200640 1 -- 172.21.15.45:0/3891880471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51c0162270 msgr2=0x7f51c01e7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:19.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.539+0000 7f51bf200640 1 --2- 172.21.15.45:0/3891880471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51c0162270 0x7f51c01e7360 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f51b8001eb0 tx=0x7f51b800c890 comp rx=0 tx=0).stop 2024-04-16T16:20:19.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.539+0000 7f51bf200640 1 -- 172.21.15.45:0/3891880471 shutdown_connections 2024-04-16T16:20:19.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.539+0000 7f51bf200640 1 --2- 172.21.15.45:0/3891880471 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f519c06ca10 0x7f519c06eed0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.539+0000 7f51bf200640 1 --2- 172.21.15.45:0/3891880471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51c0162270 0x7f51c01e7360 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.539+0000 7f51bf200640 1 --2- 172.21.15.45:0/3891880471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51c0161e90 0x7f51c014a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.539+0000 7f51bf200640 1 --2- 172.21.15.45:0/3891880471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51c00d52d0 0x7f51c00c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.539+0000 7f51bf200640 1 -- 172.21.15.45:0/3891880471 >> 172.21.15.45:0/3891880471 conn(0x7f51c00d1940 msgr2=0x7f51c00b91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:19.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.539+0000 7f51bf200640 1 -- 172.21.15.45:0/3891880471 shutdown_connections 2024-04-16T16:20:19.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.539+0000 7f51bf200640 1 -- 172.21.15.45:0/3891880471 wait complete. 2024-04-16T16:20:19.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-16T16:20:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.784+0000 7fc251a00640 1 Processor -- start 2024-04-16T16:20:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.784+0000 7fc251a00640 1 -- start start 2024-04-16T16:20:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.787+0000 7fc251a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc24c145230 0x7fc24c15bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.787+0000 7fc251a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc24c15d3b0 con 0x7fc24c15cbe0 2024-04-16T16:20:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.787+0000 7fc251a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc24c15d690 con 0x7fc24c0c5670 2024-04-16T16:20:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.787+0000 7fc251a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc24c15dfc0 con 0x7fc24c145230 2024-04-16T16:20:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.787+0000 7fc24be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc24c145230 0x7fc24c15bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.787+0000 7fc24be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc24c145230 0x7fc24c15bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38770/0 (socket says 172.21.15.45:38770) 2024-04-16T16:20:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.787+0000 7fc24be00640 1 -- 172.21.15.45:0/3386928330 learned_addr learned my addr 172.21.15.45:0/3386928330 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.787+0000 7fc243e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc24c15cbe0 0x7fc24c0cd0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33622/0 (socket says 172.21.15.45:33622) 2024-04-16T16:20:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.788+0000 7fc24be00640 1 -- 172.21.15.45:0/3386928330 >> v1:172.21.15.103:6789/0 conn(0x7fc24c0c5670 legacy=0x7fc24c15e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.788+0000 7fc24be00640 1 -- 172.21.15.45:0/3386928330 >> v1:172.21.15.45:6789/0 conn(0x7fc24c15cbe0 legacy=0x7fc24c0cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.788+0000 7fc24be00640 1 -- 172.21.15.45:0/3386928330 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c15d970 con 0x7fc24c145230 2024-04-16T16:20:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.788+0000 7fc24b400640 1 -- 172.21.15.45:0/3386928330 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 831384335 0 0) 0x7fc24c15d3b0 con 0x7fc24c15cbe0 2024-04-16T16:20:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.789+0000 7fc24be00640 1 --2- 172.21.15.45:0/3386928330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc24c145230 0x7fc24c15bc80 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fc24c15e140 tx=0x7fc23c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54ad5d6f312406c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:19.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.789+0000 7fc24b400640 1 -- 172.21.15.45:0/3386928330 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c035020 con 0x7fc24c145230 2024-04-16T16:20:19.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.789+0000 7fc24b400640 1 -- 172.21.15.45:0/3386928330 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc23c031640 con 0x7fc24c145230 2024-04-16T16:20:19.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.789+0000 7fc24b400640 1 -- 172.21.15.45:0/3386928330 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c02d040 con 0x7fc24c145230 2024-04-16T16:20:19.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.790+0000 7fc251a00640 1 -- 172.21.15.45:0/3386928330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc24c145230 msgr2=0x7fc24c15bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:19.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.790+0000 7fc251a00640 1 --2- 172.21.15.45:0/3386928330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc24c145230 0x7fc24c15bc80 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fc24c15e140 tx=0x7fc23c02b8f0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.790+0000 7fc251a00640 1 -- 172.21.15.45:0/3386928330 shutdown_connections 2024-04-16T16:20:19.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.790+0000 7fc251a00640 1 --2- 172.21.15.45:0/3386928330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc24c145230 0x7fc24c15bc80 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.790+0000 7fc251a00640 1 -- 172.21.15.45:0/3386928330 >> 172.21.15.45:0/3386928330 conn(0x7fc24c06b930 msgr2=0x7fc24c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:19.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.790+0000 7fc251a00640 1 -- 172.21.15.45:0/3386928330 shutdown_connections 2024-04-16T16:20:19.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.790+0000 7fc251a00640 1 -- 172.21.15.45:0/3386928330 wait complete. 2024-04-16T16:20:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.792+0000 7fc251a00640 1 Processor -- start 2024-04-16T16:20:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.792+0000 7fc251a00640 1 -- start start 2024-04-16T16:20:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.793+0000 7fc251a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc24c0c5670 0x7fc24c0cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.793+0000 7fc251a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc24c145230 0x7fc24c13ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.793+0000 7fc251a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc24c15cbe0 0x7fc24c13d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.793+0000 7fc251a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c15eb60 con 0x7fc24c145230 2024-04-16T16:20:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.793+0000 7fc251a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c15d070 con 0x7fc24c15cbe0 2024-04-16T16:20:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.793+0000 7fc251a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc24c15ece0 con 0x7fc24c0c5670 2024-04-16T16:20:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.793+0000 7fc243e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc24c145230 0x7fc24c13ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.793+0000 7fc243e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc24c145230 0x7fc24c13ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38782/0 (socket says 172.21.15.45:38782) 2024-04-16T16:20:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.793+0000 7fc243e00640 1 -- 172.21.15.45:0/3665874833 learned_addr learned my addr 172.21.15.45:0/3665874833 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.794+0000 7fc24be00640 1 --2- 172.21.15.45:0/3665874833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc24c0c5670 0x7fc24c0cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.794+0000 7fc243e00640 1 -- 172.21.15.45:0/3665874833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc24c0c5670 msgr2=0x7fc24c0cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:19.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.794+0000 7fc251000640 1 --2- 172.21.15.45:0/3665874833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc24c15cbe0 0x7fc24c13d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:19.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.794+0000 7fc243e00640 1 --2- 172.21.15.45:0/3665874833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc24c0c5670 0x7fc24c0cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.794+0000 7fc243e00640 1 -- 172.21.15.45:0/3665874833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc24c15cbe0 msgr2=0x7fc24c13d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:19.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.794+0000 7fc243e00640 1 --2- 172.21.15.45:0/3665874833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc24c15cbe0 0x7fc24c13d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:19.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.794+0000 7fc243e00640 1 -- 172.21.15.45:0/3665874833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c13d930 con 0x7fc24c145230 2024-04-16T16:20:19.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.794+0000 7fc243e00640 1 --2- 172.21.15.45:0/3665874833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc24c145230 0x7fc24c13ce00 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fc23400c430 tx=0x7fc23400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:19.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.794+0000 7fc248c00640 1 -- 172.21.15.45:0/3665874833 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2340074e0 con 0x7fc24c145230 2024-04-16T16:20:19.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.794+0000 7fc248c00640 1 -- 172.21.15.45:0/3665874833 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2340047e0 con 0x7fc24c145230 2024-04-16T16:20:19.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.794+0000 7fc251a00640 1 -- 172.21.15.45:0/3665874833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc24c13dc10 con 0x7fc24c145230 2024-04-16T16:20:19.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.794+0000 7fc248c00640 1 -- 172.21.15.45:0/3665874833 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc234004bd0 con 0x7fc24c145230 2024-04-16T16:20:19.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.795+0000 7fc251a00640 1 -- 172.21.15.45:0/3665874833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc24c0b53c0 con 0x7fc24c145230 2024-04-16T16:20:19.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.796+0000 7fc251a00640 1 -- 172.21.15.45:0/3665874833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc24c13e000 con 0x7fc24c145230 2024-04-16T16:20:19.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.796+0000 7fc248c00640 1 -- 172.21.15.45:0/3665874833 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7fc234011070 con 0x7fc24c145230 2024-04-16T16:20:19.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.797+0000 7fc248c00640 1 --2- 172.21.15.45:0/3665874833 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fc22806ca10 0x7fc22806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:19.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.797+0000 7fc248c00640 1 -- 172.21.15.45:0/3665874833 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6397+0+0 (secure 0 0 0) 0x7fc23408bbd0 con 0x7fc24c145230 2024-04-16T16:20:19.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.798+0000 7fc24be00640 1 --2- 172.21.15.45:0/3665874833 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fc22806ca10 0x7fc22806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:19.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.799+0000 7fc24be00640 1 --2- 172.21.15.45:0/3665874833 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fc22806ca10 0x7fc22806eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc23c036000 tx=0x7fc23c03c040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:19.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:19.802+0000 7fc248c00640 1 -- 172.21.15.45:0/3665874833 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc234059cd0 con 0x7fc24c145230 2024-04-16T16:20:20.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.039+0000 7fc251a00640 1 -- 172.21.15.45:0/3665874833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc24c13dde0 con 0x7fc24c145230 2024-04-16T16:20:20.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-04-16T16:20:20.040+0000 7fe7a4800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T16:20:20.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.044+0000 7fc248c00640 1 -- 172.21.15.45:0/3665874833 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v83) ==== 161+0+0 (secure 0 0 0) 0x7fc2340595d0 con 0x7fc24c145230 2024-04-16T16:20:20.044 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-16T16:20:20.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.052+0000 7fc251a00640 1 -- 172.21.15.45:0/3665874833 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fc22806ca10 msgr2=0x7fc22806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:20.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.052+0000 7fc251a00640 1 --2- 172.21.15.45:0/3665874833 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fc22806ca10 0x7fc22806eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc23c036000 tx=0x7fc23c03c040 comp rx=0 tx=0).stop 2024-04-16T16:20:20.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.052+0000 7fc251a00640 1 -- 172.21.15.45:0/3665874833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc24c145230 msgr2=0x7fc24c13ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:20.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.052+0000 7fc251a00640 1 --2- 172.21.15.45:0/3665874833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc24c145230 0x7fc24c13ce00 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fc23400c430 tx=0x7fc23400c8f0 comp rx=0 tx=0).stop 2024-04-16T16:20:20.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.052+0000 7fc251a00640 1 -- 172.21.15.45:0/3665874833 shutdown_connections 2024-04-16T16:20:20.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.052+0000 7fc251a00640 1 --2- 172.21.15.45:0/3665874833 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fc22806ca10 0x7fc22806eed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:20.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.052+0000 7fc251a00640 1 --2- 172.21.15.45:0/3665874833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc24c15cbe0 0x7fc24c13d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:20.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.052+0000 7fc251a00640 1 --2- 172.21.15.45:0/3665874833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc24c145230 0x7fc24c13ce00 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:20.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.052+0000 7fc251a00640 1 --2- 172.21.15.45:0/3665874833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc24c0c5670 0x7fc24c0cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:20.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.052+0000 7fc251a00640 1 -- 172.21.15.45:0/3665874833 >> 172.21.15.45:0/3665874833 conn(0x7fc24c06b930 msgr2=0x7fc24c1421f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:20.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.053+0000 7fc251a00640 1 -- 172.21.15.45:0/3665874833 shutdown_connections 2024-04-16T16:20:20.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.053+0000 7fc251a00640 1 -- 172.21.15.45:0/3665874833 wait complete. 2024-04-16T16:20:20.059 INFO:tasks.ceph.mds.a.smithi045.stderr: -3636> 2024-04-16T16:19:54.585+0000 7efc7c200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:20:20.070 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-16T16:20:20.122 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2024-04-16T16:20:20.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.270+0000 7f56c7200640 1 Processor -- start 2024-04-16T16:20:20.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.270+0000 7f56c7200640 1 -- start start 2024-04-16T16:20:20.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.272+0000 7f56c7200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56c0162e20 0x7f56c006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:20.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.272+0000 7f56c7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56c006ed00 0x7f56c00b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:20.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.272+0000 7f56c7200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56c0167780 con 0x7f56c0162a40 2024-04-16T16:20:20.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.272+0000 7f56c7200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f56c0166f60 con 0x7f56c0162e20 2024-04-16T16:20:20.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.272+0000 7f56c7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f56c01671c0 con 0x7f56c006ed00 2024-04-16T16:20:20.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.272+0000 7f56c5400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56c0162e20 0x7f56c006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:20.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.272+0000 7f56c5400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56c0162e20 0x7f56c006e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38800/0 (socket says 172.21.15.45:38800) 2024-04-16T16:20:20.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.272+0000 7f56c5400640 1 -- 172.21.15.45:0/336116924 learned_addr learned my addr 172.21.15.45:0/336116924 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:20.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.273+0000 7f56c6800640 1 --2- 172.21.15.45:0/336116924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56c006ed00 0x7f56c00b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:20.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.273+0000 7f56c5400640 1 -- 172.21.15.45:0/336116924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56c006ed00 msgr2=0x7f56c00b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:20.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.273+0000 7f56c5400640 1 --2- 172.21.15.45:0/336116924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56c006ed00 0x7f56c00b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:20.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.273+0000 7f56c5400640 1 -- 172.21.15.45:0/336116924 >> v1:172.21.15.103:6789/0 conn(0x7f56c0162a40 legacy=0x7f56c0166850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:20.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.273+0000 7f56c5400640 1 -- 172.21.15.45:0/336116924 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c01674a0 con 0x7f56c0162e20 2024-04-16T16:20:20.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.274+0000 7f56c5400640 1 --2- 172.21.15.45:0/336116924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56c0162e20 0x7f56c006e7c0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009850 tx=0x7f56b002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf717ca509bec96 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:20.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.275+0000 7f56c4a00640 1 -- 172.21.15.45:0/336116924 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b0004230 con 0x7f56c0162e20 2024-04-16T16:20:20.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.275+0000 7f56c4a00640 1 -- 172.21.15.45:0/336116924 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56b00043d0 con 0x7f56c0162e20 2024-04-16T16:20:20.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.275+0000 7f56c4a00640 1 -- 172.21.15.45:0/336116924 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b0003810 con 0x7f56c0162e20 2024-04-16T16:20:20.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.275+0000 7f56c7200640 1 -- 172.21.15.45:0/336116924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56c0162e20 msgr2=0x7f56c006e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:20.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.275+0000 7f56c7200640 1 --2- 172.21.15.45:0/336116924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56c0162e20 0x7f56c006e7c0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009850 tx=0x7f56b002b960 comp rx=0 tx=0).stop 2024-04-16T16:20:20.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.276+0000 7f56c7200640 1 -- 172.21.15.45:0/336116924 shutdown_connections 2024-04-16T16:20:20.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.276+0000 7f56c7200640 1 --2- 172.21.15.45:0/336116924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56c006ed00 0x7f56c00b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:20.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.276+0000 7f56c7200640 1 --2- 172.21.15.45:0/336116924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56c0162e20 0x7f56c006e7c0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:20.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.276+0000 7f56c7200640 1 -- 172.21.15.45:0/336116924 >> 172.21.15.45:0/336116924 conn(0x7f56c0067e00 msgr2=0x7f56c006b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:20.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.276+0000 7f56c7200640 1 -- 172.21.15.45:0/336116924 shutdown_connections 2024-04-16T16:20:20.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.276+0000 7f56c7200640 1 -- 172.21.15.45:0/336116924 wait complete. 2024-04-16T16:20:20.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.277+0000 7f56c7200640 1 Processor -- start 2024-04-16T16:20:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.278+0000 7f56c7200640 1 -- start start 2024-04-16T16:20:20.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.278+0000 7f56c7200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56c006ed00 0x7f56c015b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:20.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.278+0000 7f56c7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56c0162a40 0x7f56c01fa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:20.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.279+0000 7f56c7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56c0162e20 0x7f56c01519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:20.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.279+0000 7f56c7200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f56c006fd30 con 0x7f56c006ed00 2024-04-16T16:20:20.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.279+0000 7f56c7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f56c006e2a0 con 0x7f56c0162e20 2024-04-16T16:20:20.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.279+0000 7f56c7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f56c006feb0 con 0x7f56c0162a40 2024-04-16T16:20:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.279+0000 7f56c5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56c006ed00 0x7f56c015b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.279+0000 7f56c5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56c006ed00 0x7f56c015b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38810/0 (socket says 172.21.15.45:38810) 2024-04-16T16:20:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.279+0000 7f56c5e00640 1 -- 172.21.15.45:0/4077251497 learned_addr learned my addr 172.21.15.45:0/4077251497 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.279+0000 7f56c5400640 1 --2- 172.21.15.45:0/4077251497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56c0162a40 0x7f56c01fa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.279+0000 7f56c5e00640 1 -- 172.21.15.45:0/4077251497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56c0162a40 msgr2=0x7f56c01fa8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.279+0000 7f56c5e00640 1 --2- 172.21.15.45:0/4077251497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56c0162a40 0x7f56c01fa8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.279+0000 7f56c5e00640 1 -- 172.21.15.45:0/4077251497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56c0162e20 msgr2=0x7f56c01519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.279+0000 7f56c5e00640 1 --2- 172.21.15.45:0/4077251497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56c0162e20 0x7f56c01519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.279+0000 7f56c5e00640 1 -- 172.21.15.45:0/4077251497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c01faf50 con 0x7f56c006ed00 2024-04-16T16:20:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.280+0000 7f56c5e00640 1 --2- 172.21.15.45:0/4077251497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56c006ed00 0x7f56c015b970 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f56b400c3f0 tx=0x7f56b400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.280+0000 7f56ba000640 1 -- 172.21.15.45:0/4077251497 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b40052d0 con 0x7f56c006ed00 2024-04-16T16:20:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.280+0000 7f56ba000640 1 -- 172.21.15.45:0/4077251497 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56b4005490 con 0x7f56c006ed00 2024-04-16T16:20:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.280+0000 7f56ba000640 1 -- 172.21.15.45:0/4077251497 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b4007260 con 0x7f56c006ed00 2024-04-16T16:20:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.280+0000 7f56c7200640 1 -- 172.21.15.45:0/4077251497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c01fb230 con 0x7f56c006ed00 2024-04-16T16:20:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.280+0000 7f56c7200640 1 -- 172.21.15.45:0/4077251497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c01fb7f0 con 0x7f56c006ed00 2024-04-16T16:20:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.281+0000 7f56c7200640 1 -- 172.21.15.45:0/4077251497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56c0075780 con 0x7f56c006ed00 2024-04-16T16:20:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.282+0000 7f56ba000640 1 -- 172.21.15.45:0/4077251497 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f56b4007400 con 0x7f56c006ed00 2024-04-16T16:20:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.283+0000 7f56ba000640 1 --2- 172.21.15.45:0/4077251497 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f56a406ca10 0x7f56a406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.283+0000 7f56ba000640 1 -- 172.21.15.45:0/4077251497 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 6483+0+0 (secure 0 0 0) 0x7f56b408b510 con 0x7f56c006ed00 2024-04-16T16:20:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.284+0000 7f56c5400640 1 --2- 172.21.15.45:0/4077251497 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f56a406ca10 0x7f56a406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.285+0000 7f56c5400640 1 --2- 172.21.15.45:0/4077251497 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f56a406ca10 0x7f56a406eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f56b0035000 tx=0x7f56b0008030 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:20.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.288+0000 7f56ba000640 1 -- 172.21.15.45:0/4077251497 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f56b4059510 con 0x7f56c006ed00 2024-04-16T16:20:20.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:20.539+0000 7f56c7200640 1 -- 172.21.15.45:0/4077251497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f56c00b4430 con 0x7f56c006ed00 2024-04-16T16:20:21.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.051+0000 7f56ba000640 1 -- 172.21.15.45:0/4077251497 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v84) ==== 104+0+0 (secure 0 0 0) 0x7f56b4058e10 con 0x7f56c006ed00 2024-04-16T16:20:21.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.054+0000 7f56c7200640 1 -- 172.21.15.45:0/4077251497 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f56a406ca10 msgr2=0x7f56a406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:21.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.054+0000 7f56c7200640 1 --2- 172.21.15.45:0/4077251497 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f56a406ca10 0x7f56a406eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f56b0035000 tx=0x7f56b0008030 comp rx=0 tx=0).stop 2024-04-16T16:20:21.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.054+0000 7f56c7200640 1 -- 172.21.15.45:0/4077251497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56c006ed00 msgr2=0x7f56c015b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:21.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.054+0000 7f56c7200640 1 --2- 172.21.15.45:0/4077251497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56c006ed00 0x7f56c015b970 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f56b400c3f0 tx=0x7f56b400c8b0 comp rx=0 tx=0).stop 2024-04-16T16:20:21.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.055+0000 7f56c7200640 1 -- 172.21.15.45:0/4077251497 shutdown_connections 2024-04-16T16:20:21.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.055+0000 7f56c7200640 1 --2- 172.21.15.45:0/4077251497 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f56a406ca10 0x7f56a406eed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:21.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.055+0000 7f56c7200640 1 --2- 172.21.15.45:0/4077251497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56c0162e20 0x7f56c01519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:21.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.055+0000 7f56c7200640 1 --2- 172.21.15.45:0/4077251497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56c0162a40 0x7f56c01fa8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:21.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.055+0000 7f56c7200640 1 --2- 172.21.15.45:0/4077251497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56c006ed00 0x7f56c015b970 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:21.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.055+0000 7f56c7200640 1 -- 172.21.15.45:0/4077251497 >> 172.21.15.45:0/4077251497 conn(0x7f56c0067e00 msgr2=0x7f56c00b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:21.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.055+0000 7f56c7200640 1 -- 172.21.15.45:0/4077251497 shutdown_connections 2024-04-16T16:20:21.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.055+0000 7f56c7200640 1 -- 172.21.15.45:0/4077251497 wait complete. 2024-04-16T16:20:21.079 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-16T16:20:21.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.299+0000 7f02eca00640 1 Processor -- start 2024-04-16T16:20:21.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.299+0000 7f02eca00640 1 -- start start 2024-04-16T16:20:21.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.301+0000 7f02eca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f02e815e9c0 0x7f02e8072650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:21.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.301+0000 7f02eca00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02e8072c70 con 0x7f02e8146180 2024-04-16T16:20:21.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.301+0000 7f02eca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02e815ced0 con 0x7f02e806eb40 2024-04-16T16:20:21.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.301+0000 7f02eca00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f02e815cc90 con 0x7f02e815e9c0 2024-04-16T16:20:21.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.301+0000 7f02e7400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f02e8146180 0x7f02e806e430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33640/0 (socket says 172.21.15.45:33640) 2024-04-16T16:20:21.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.301+0000 7f02e7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f02e815e9c0 0x7f02e8072650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:21.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.301+0000 7f02e7400640 1 -- 172.21.15.45:0/514894929 learned_addr learned my addr 172.21.15.45:0/514894929 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:21.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.302+0000 7f02e6a00640 1 -- 172.21.15.45:0/514894929 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1584684391 0 0) 0x7f02e8072c70 con 0x7f02e8146180 2024-04-16T16:20:21.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.302+0000 7f02e6a00640 1 -- 172.21.15.45:0/514894929 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02cc003520 con 0x7f02e8146180 2024-04-16T16:20:21.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.302+0000 7f02e7e00640 1 -- 172.21.15.45:0/514894929 >> v1:172.21.15.103:6789/0 conn(0x7f02e806eb40 legacy=0x7f02e8071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:21.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.302+0000 7f02e7e00640 1 -- 172.21.15.45:0/514894929 >> v1:172.21.15.45:6789/0 conn(0x7f02e8146180 legacy=0x7f02e806e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:21.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.302+0000 7f02e7e00640 1 -- 172.21.15.45:0/514894929 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e815d1b0 con 0x7f02e815e9c0 2024-04-16T16:20:21.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.303+0000 7f02e7e00640 1 --2- 172.21.15.45:0/514894929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e815e9c0 0x7f02e8072650 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f02e0009d90 tx=0x7f02e002bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7673f4b693bfe200 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:21.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.303+0000 7f02e6a00640 1 -- 172.21.15.45:0/514894929 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0039070 con 0x7f02e815e9c0 2024-04-16T16:20:21.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.303+0000 7f02e6a00640 1 -- 172.21.15.45:0/514894929 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02e002cb40 con 0x7f02e815e9c0 2024-04-16T16:20:21.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.303+0000 7f02e6a00640 1 -- 172.21.15.45:0/514894929 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0034440 con 0x7f02e815e9c0 2024-04-16T16:20:21.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.304+0000 7f02eca00640 1 -- 172.21.15.45:0/514894929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e815e9c0 msgr2=0x7f02e8072650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:21.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.304+0000 7f02eca00640 1 --2- 172.21.15.45:0/514894929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e815e9c0 0x7f02e8072650 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f02e0009d90 tx=0x7f02e002bd90 comp rx=0 tx=0).stop 2024-04-16T16:20:21.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.304+0000 7f02eca00640 1 -- 172.21.15.45:0/514894929 shutdown_connections 2024-04-16T16:20:21.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.304+0000 7f02eca00640 1 --2- 172.21.15.45:0/514894929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e815e9c0 0x7f02e8072650 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:21.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.304+0000 7f02eca00640 1 -- 172.21.15.45:0/514894929 >> 172.21.15.45:0/514894929 conn(0x7f02e80b3bb0 msgr2=0x7f02e80b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:21.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.304+0000 7f02eca00640 1 -- 172.21.15.45:0/514894929 shutdown_connections 2024-04-16T16:20:21.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.304+0000 7f02eca00640 1 -- 172.21.15.45:0/514894929 wait complete. 2024-04-16T16:20:21.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.306+0000 7f02eca00640 1 Processor -- start 2024-04-16T16:20:21.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.306+0000 7f02eca00640 1 -- start start 2024-04-16T16:20:21.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.307+0000 7f02eca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e806eb40 0x7f02e80af110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:21.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.307+0000 7f02eca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02e8146180 0x7f02e814de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:21.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.307+0000 7f02eca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e815e9c0 0x7f02e8141df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:21.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.307+0000 7f02eca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f02e8167540 con 0x7f02e815e9c0 2024-04-16T16:20:21.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.307+0000 7f02eca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f02e815d520 con 0x7f02e806eb40 2024-04-16T16:20:21.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.307+0000 7f02eca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f02e81676c0 con 0x7f02e8146180 2024-04-16T16:20:21.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.307+0000 7f02e7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e815e9c0 0x7f02e8141df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:21.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.307+0000 7f02e7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e815e9c0 0x7f02e8141df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38834/0 (socket says 172.21.15.45:38834) 2024-04-16T16:20:21.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.307+0000 7f02e7e00640 1 -- 172.21.15.45:0/4081860840 learned_addr learned my addr 172.21.15.45:0/4081860840 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:21.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.307+0000 7f02e7400640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e806eb40 0x7f02e80af110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:21.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.307+0000 7f02dfe00640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02e8146180 0x7f02e814de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:21.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.307+0000 7f02e7e00640 1 -- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02e8146180 msgr2=0x7f02e814de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:21.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.308+0000 7f02e7e00640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02e8146180 0x7f02e814de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:21.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.308+0000 7f02e7e00640 1 -- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e806eb40 msgr2=0x7f02e80af110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:21.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.308+0000 7f02e7e00640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e806eb40 0x7f02e80af110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:21.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.308+0000 7f02e7e00640 1 -- 172.21.15.45:0/4081860840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e814e390 con 0x7f02e815e9c0 2024-04-16T16:20:21.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.308+0000 7f02dfe00640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02e8146180 0x7f02e814de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:21.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.308+0000 7f02e7400640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e806eb40 0x7f02e80af110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:21.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.308+0000 7f02e7e00640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e815e9c0 0x7f02e8141df0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f02e0009eb0 tx=0x7f02e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:21.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.308+0000 7f02df400640 1 -- 172.21.15.45:0/4081860840 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0044020 con 0x7f02e815e9c0 2024-04-16T16:20:21.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.308+0000 7f02df400640 1 -- 172.21.15.45:0/4081860840 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02e003f470 con 0x7f02e815e9c0 2024-04-16T16:20:21.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.308+0000 7f02eca00640 1 -- 172.21.15.45:0/4081860840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e8142690 con 0x7f02e815e9c0 2024-04-16T16:20:21.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.308+0000 7f02df400640 1 -- 172.21.15.45:0/4081860840 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0039040 con 0x7f02e815e9c0 2024-04-16T16:20:21.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.308+0000 7f02eca00640 1 -- 172.21.15.45:0/4081860840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e8142c50 con 0x7f02e815e9c0 2024-04-16T16:20:21.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.310+0000 7f02eca00640 1 -- 172.21.15.45:0/4081860840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02e8143280 con 0x7f02e815e9c0 2024-04-16T16:20:21.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.311+0000 7f02df400640 1 -- 172.21.15.45:0/4081860840 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f02e0005d10 con 0x7f02e815e9c0 2024-04-16T16:20:21.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.311+0000 7f02df400640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f02c406ca10 0x7f02c406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:21.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.312+0000 7f02df400640 1 -- 172.21.15.45:0/4081860840 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 6483+0+0 (secure 0 0 0) 0x7f02e00aea10 con 0x7f02e815e9c0 2024-04-16T16:20:21.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.312+0000 7f02e7400640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f02c406ca10 0x7f02c406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:21.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.314+0000 7f02e7400640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f02c406ca10 0x7f02c406eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f02d80011e0 tx=0x7f02d8009040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:21.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.316+0000 7f02df400640 1 -- 172.21.15.45:0/4081860840 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f02e8143280 con 0x7f02e815e9c0 2024-04-16T16:20:21.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:21.521+0000 7f02eca00640 1 -- 172.21.15.45:0/4081860840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f02e8075780 con 0x7f02e815e9c0 2024-04-16T16:20:22.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.192+0000 7f02df400640 1 -- 172.21.15.45:0/4081860840 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v119) ==== 181+0+0 (secure 0 0 0) 0x7f02e004e070 con 0x7f02e815e9c0 2024-04-16T16:20:22.192 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2024-04-16T16:20:22.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.194+0000 7f02eca00640 1 -- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f02c406ca10 msgr2=0x7f02c406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:22.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.194+0000 7f02eca00640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f02c406ca10 0x7f02c406eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f02d80011e0 tx=0x7f02d8009040 comp rx=0 tx=0).stop 2024-04-16T16:20:22.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.194+0000 7f02eca00640 1 -- 172.21.15.45:0/4081860840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e815e9c0 msgr2=0x7f02e8141df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:22.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.194+0000 7f02eca00640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e815e9c0 0x7f02e8141df0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f02e0009eb0 tx=0x7f02e0004a40 comp rx=0 tx=0).stop 2024-04-16T16:20:22.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.195+0000 7f02eca00640 1 -- 172.21.15.45:0/4081860840 shutdown_connections 2024-04-16T16:20:22.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.195+0000 7f02eca00640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f02c406ca10 0x7f02c406eed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:22.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.195+0000 7f02eca00640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e815e9c0 0x7f02e8141df0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.195+0000 7f02eca00640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02e8146180 0x7f02e814de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.195+0000 7f02eca00640 1 --2- 172.21.15.45:0/4081860840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e806eb40 0x7f02e80af110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.195+0000 7f02eca00640 1 -- 172.21.15.45:0/4081860840 >> 172.21.15.45:0/4081860840 conn(0x7f02e80b3bb0 msgr2=0x7f02e80b5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.195+0000 7f02eca00640 1 -- 172.21.15.45:0/4081860840 shutdown_connections 2024-04-16T16:20:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.195+0000 7f02eca00640 1 -- 172.21.15.45:0/4081860840 wait complete. 2024-04-16T16:20:22.221 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-16T16:20:22.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.441+0000 7f87c2a00640 1 Processor -- start 2024-04-16T16:20:22.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.442+0000 7f87c2a00640 1 -- start start 2024-04-16T16:20:22.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.444+0000 7f87c2a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87bc0d0050 0x7f87bc0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:22.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.444+0000 7f87c2a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f87bc0cd8a0 0x7f87bc06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:22.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.444+0000 7f87c2a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f87bc0cdd20 0x7f87bc06e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:22.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.445+0000 7f87c2a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f87bc0d0430 con 0x7f87bc0cd8a0 2024-04-16T16:20:22.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.445+0000 7f87c2a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f87bc15bbd0 con 0x7f87bc0cdd20 2024-04-16T16:20:22.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.445+0000 7f87c2a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f87bc0b0fd0 con 0x7f87bc0d0050 2024-04-16T16:20:22.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.445+0000 7f87c0c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f87bc0cd8a0 0x7f87bc06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:22.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.445+0000 7f87c0c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f87bc0cd8a0 0x7f87bc06e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38836/0 (socket says 172.21.15.45:38836) 2024-04-16T16:20:22.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.445+0000 7f87c0c00640 1 -- 172.21.15.45:0/691843674 learned_addr learned my addr 172.21.15.45:0/691843674 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:22.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.445+0000 7f87c1600640 1 --2- 172.21.15.45:0/691843674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87bc0d0050 0x7f87bc0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:22.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.445+0000 7f87c2000640 1 --2- 172.21.15.45:0/691843674 >> v2:172.21.15.103:3300/0 conn(0x7f87bc0cdd20 0x7f87bc06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:22.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.445+0000 7f87c0c00640 1 -- 172.21.15.45:0/691843674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87bc0d0050 msgr2=0x7f87bc0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:22.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.445+0000 7f87c0c00640 1 --2- 172.21.15.45:0/691843674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87bc0d0050 0x7f87bc0b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:22.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.445+0000 7f87c0c00640 1 -- 172.21.15.45:0/691843674 >> v2:172.21.15.103:3300/0 conn(0x7f87bc0cdd20 msgr2=0x7f87bc06e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:22.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.445+0000 7f87c0c00640 1 --2- 172.21.15.45:0/691843674 >> v2:172.21.15.103:3300/0 conn(0x7f87bc0cdd20 0x7f87bc06e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:22.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.445+0000 7f87c0c00640 1 -- 172.21.15.45:0/691843674 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc15be90 con 0x7f87bc0cd8a0 2024-04-16T16:20:22.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.445+0000 7f87c1600640 1 --2- 172.21.15.45:0/691843674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87bc0d0050 0x7f87bc0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:22.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.445+0000 7f87c2000640 1 --2- 172.21.15.45:0/691843674 >> v2:172.21.15.103:3300/0 conn(0x7f87bc0cdd20 0x7f87bc06e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:22.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.447+0000 7f87c0c00640 1 --2- 172.21.15.45:0/691843674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87bc0cd8a0 0x7f87bc06e3e0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0b0820 tx=0x7f87a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae6d68cc6874aba server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:22.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.449+0000 7f87bbe00640 1 -- 172.21.15.45:0/691843674 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8035020 con 0x7f87bc0cd8a0 2024-04-16T16:20:22.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.449+0000 7f87bbe00640 1 -- 172.21.15.45:0/691843674 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87a8031870 con 0x7f87bc0cd8a0 2024-04-16T16:20:22.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.449+0000 7f87bbe00640 1 -- 172.21.15.45:0/691843674 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8031c40 con 0x7f87bc0cd8a0 2024-04-16T16:20:22.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.449+0000 7f87c2a00640 1 -- 172.21.15.45:0/691843674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87bc0cd8a0 msgr2=0x7f87bc06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:22.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.449+0000 7f87c2a00640 1 --2- 172.21.15.45:0/691843674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87bc0cd8a0 0x7f87bc06e3e0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0b0820 tx=0x7f87a802b910 comp rx=0 tx=0).stop 2024-04-16T16:20:22.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.450+0000 7f87c2a00640 1 -- 172.21.15.45:0/691843674 shutdown_connections 2024-04-16T16:20:22.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.450+0000 7f87c2a00640 1 --2- 172.21.15.45:0/691843674 >> v2:172.21.15.103:3300/0 conn(0x7f87bc0cdd20 0x7f87bc06e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:22.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.450+0000 7f87c2a00640 1 --2- 172.21.15.45:0/691843674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87bc0cd8a0 0x7f87bc06e3e0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:22.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.450+0000 7f87c2a00640 1 --2- 172.21.15.45:0/691843674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87bc0d0050 0x7f87bc0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:22.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.450+0000 7f87c2a00640 1 -- 172.21.15.45:0/691843674 >> 172.21.15.45:0/691843674 conn(0x7f87bc0b3bb0 msgr2=0x7f87bc0b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:22.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.450+0000 7f87c2a00640 1 -- 172.21.15.45:0/691843674 shutdown_connections 2024-04-16T16:20:22.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.450+0000 7f87c2a00640 1 -- 172.21.15.45:0/691843674 wait complete. 2024-04-16T16:20:22.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.451+0000 7f87c2a00640 1 Processor -- start 2024-04-16T16:20:22.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.451+0000 7f87c2a00640 1 -- start start 2024-04-16T16:20:22.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.452+0000 7f87c2a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87bc0cd8a0 0x7f87bc1e5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:22.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.452+0000 7f87c2a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87bc0cdd20 0x7f87bc151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:22.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.452+0000 7f87c2a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87bc0d0050 0x7f87bc1517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:22.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.452+0000 7f87c2a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc15c730 con 0x7f87bc0d0050 2024-04-16T16:20:22.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.452+0000 7f87c2a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc15c450 con 0x7f87bc0cd8a0 2024-04-16T16:20:22.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.452+0000 7f87c2a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f87bc15f370 con 0x7f87bc0cdd20 2024-04-16T16:20:22.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.452+0000 7f87c2000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87bc0d0050 0x7f87bc1517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:22.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.452+0000 7f87c2000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87bc0d0050 0x7f87bc1517c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38840/0 (socket says 172.21.15.45:38840) 2024-04-16T16:20:22.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.452+0000 7f87c2000640 1 -- 172.21.15.45:0/3581430644 learned_addr learned my addr 172.21.15.45:0/3581430644 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:22.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.454+0000 7f87c2000640 1 -- 172.21.15.45:0/3581430644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87bc0cdd20 msgr2=0x7f87bc151280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:22.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.454+0000 7f87c2000640 1 --2- 172.21.15.45:0/3581430644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87bc0cdd20 0x7f87bc151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:22.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.454+0000 7f87c2000640 1 -- 172.21.15.45:0/3581430644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87bc0cd8a0 msgr2=0x7f87bc1e5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:22.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.454+0000 7f87c2000640 1 --2- 172.21.15.45:0/3581430644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87bc0cd8a0 0x7f87bc1e5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:22.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.454+0000 7f87c2000640 1 -- 172.21.15.45:0/3581430644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc15ed10 con 0x7f87bc0d0050 2024-04-16T16:20:22.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.454+0000 7f87c2000640 1 --2- 172.21.15.45:0/3581430644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87bc0d0050 0x7f87bc1517c0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f87b400cf20 tx=0x7f87b400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:22.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.454+0000 7f87b9600640 1 -- 172.21.15.45:0/3581430644 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b401a070 con 0x7f87bc0d0050 2024-04-16T16:20:22.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.454+0000 7f87b9600640 1 -- 172.21.15.45:0/3581430644 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b400d840 con 0x7f87bc0d0050 2024-04-16T16:20:22.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.454+0000 7f87b9600640 1 -- 172.21.15.45:0/3581430644 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b400dc30 con 0x7f87bc0d0050 2024-04-16T16:20:22.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.454+0000 7f87c2a00640 1 -- 172.21.15.45:0/3581430644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87bc1f8350 con 0x7f87bc0d0050 2024-04-16T16:20:22.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.454+0000 7f87c2a00640 1 -- 172.21.15.45:0/3581430644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87bc1f8910 con 0x7f87bc0d0050 2024-04-16T16:20:22.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.455+0000 7f87c2a00640 1 -- 172.21.15.45:0/3581430644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87bc075780 con 0x7f87bc0d0050 2024-04-16T16:20:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.456+0000 7f87b9600640 1 -- 172.21.15.45:0/3581430644 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f87b401d070 con 0x7f87bc0d0050 2024-04-16T16:20:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.458+0000 7f87b9600640 1 --2- 172.21.15.45:0/3581430644 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f879c06ca10 0x7f879c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:22.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.458+0000 7f87b9600640 1 -- 172.21.15.45:0/3581430644 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 6045+0+0 (secure 0 0 0) 0x7f87b408c710 con 0x7f87bc0d0050 2024-04-16T16:20:22.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.458+0000 7f87c1600640 1 --2- 172.21.15.45:0/3581430644 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f879c06ca10 0x7f879c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:22.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.460+0000 7f87c1600640 1 --2- 172.21.15.45:0/3581430644 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f879c06ca10 0x7f879c06eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f87b0006b10 tx=0x7f87b0008040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:22.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.463+0000 7f87b9600640 1 -- 172.21.15.45:0/3581430644 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f87b405a910 con 0x7f87bc0d0050 2024-04-16T16:20:22.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:22.667+0000 7f87c2a00640 1 -- 172.21.15.45:0/3581430644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f87bc1e7ff0 con 0x7f87bc0d0050 2024-04-16T16:20:23.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.196+0000 7f87b9600640 1 -- 172.21.15.45:0/3581430644 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v120) ==== 169+0+0 (secure 0 0 0) 0x7f87b405a210 con 0x7f87bc0d0050 2024-04-16T16:20:23.195 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2024-04-16T16:20:23.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.199+0000 7f87c2a00640 1 -- 172.21.15.45:0/3581430644 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f879c06ca10 msgr2=0x7f879c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:23.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.199+0000 7f87c2a00640 1 --2- 172.21.15.45:0/3581430644 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f879c06ca10 0x7f879c06eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f87b0006b10 tx=0x7f87b0008040 comp rx=0 tx=0).stop 2024-04-16T16:20:23.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.199+0000 7f87c2a00640 1 -- 172.21.15.45:0/3581430644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87bc0d0050 msgr2=0x7f87bc1517c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:23.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.199+0000 7f87c2a00640 1 --2- 172.21.15.45:0/3581430644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87bc0d0050 0x7f87bc1517c0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f87b400cf20 tx=0x7f87b400a7d0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.199+0000 7f87c2a00640 1 -- 172.21.15.45:0/3581430644 shutdown_connections 2024-04-16T16:20:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.200+0000 7f87c2a00640 1 --2- 172.21.15.45:0/3581430644 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f879c06ca10 0x7f879c06eed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.200+0000 7f87c2a00640 1 --2- 172.21.15.45:0/3581430644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87bc0d0050 0x7f87bc1517c0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.200+0000 7f87c2a00640 1 --2- 172.21.15.45:0/3581430644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87bc0cdd20 0x7f87bc151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.200+0000 7f87c2a00640 1 --2- 172.21.15.45:0/3581430644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87bc0cd8a0 0x7f87bc1e5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.200+0000 7f87c2a00640 1 -- 172.21.15.45:0/3581430644 >> 172.21.15.45:0/3581430644 conn(0x7f87bc0b3bb0 msgr2=0x7f87bc0cbf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.200+0000 7f87c2a00640 1 -- 172.21.15.45:0/3581430644 shutdown_connections 2024-04-16T16:20:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.200+0000 7f87c2a00640 1 -- 172.21.15.45:0/3581430644 wait complete. 2024-04-16T16:20:23.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-16T16:20:23.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.449+0000 7f3095600640 1 Processor -- start 2024-04-16T16:20:23.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.449+0000 7f3095600640 1 -- start start 2024-04-16T16:20:23.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.450+0000 7f3095600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f309015ecc0 0x7f30901626a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:23.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.450+0000 7f3095600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f309006d620 0x7f30900d35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:23.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.450+0000 7f3095600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30900d2b10 con 0x7f3090162be0 2024-04-16T16:20:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.451+0000 7f3095600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3090163210 con 0x7f309006d620 2024-04-16T16:20:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.451+0000 7f3095600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f30901634f0 con 0x7f309015ecc0 2024-04-16T16:20:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.451+0000 7f308f400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f309006d620 0x7f30900d35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.451+0000 7f308f400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f309006d620 0x7f30900d35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49806/0 (socket says 172.21.15.45:49806) 2024-04-16T16:20:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.451+0000 7f308f400640 1 -- 172.21.15.45:0/4078052370 learned_addr learned my addr 172.21.15.45:0/4078052370 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.451+0000 7f308ea00640 1 --2- 172.21.15.45:0/4078052370 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f309015ecc0 0x7f30901626a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.451+0000 7f308f400640 1 -- 172.21.15.45:0/4078052370 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f309015ecc0 msgr2=0x7f30901626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:23.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.451+0000 7f308f400640 1 --2- 172.21.15.45:0/4078052370 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f309015ecc0 0x7f30901626a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.451+0000 7f308f400640 1 -- 172.21.15.45:0/4078052370 >> v1:172.21.15.103:6789/0 conn(0x7f3090162be0 legacy=0x7f309006cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:23.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.451+0000 7f308f400640 1 -- 172.21.15.45:0/4078052370 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30900d2dd0 con 0x7f309006d620 2024-04-16T16:20:23.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.451+0000 7f308ea00640 1 --2- 172.21.15.45:0/4078052370 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f309015ecc0 0x7f30901626a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:23.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.453+0000 7f308f400640 1 --2- 172.21.15.45:0/4078052370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309006d620 0x7f30900d35d0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f3088009870 tx=0x7f308802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e061ffea024c6ff0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:23.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.453+0000 7f308d600640 1 -- 172.21.15.45:0/4078052370 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3088034070 con 0x7f309006d620 2024-04-16T16:20:23.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.453+0000 7f308d600640 1 -- 172.21.15.45:0/4078052370 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3088030840 con 0x7f309006d620 2024-04-16T16:20:23.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.453+0000 7f308d600640 1 -- 172.21.15.45:0/4078052370 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3088030c10 con 0x7f309006d620 2024-04-16T16:20:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.454+0000 7f3095600640 1 -- 172.21.15.45:0/4078052370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309006d620 msgr2=0x7f30900d35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.454+0000 7f3095600640 1 --2- 172.21.15.45:0/4078052370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309006d620 0x7f30900d35d0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f3088009870 tx=0x7f308802b980 comp rx=0 tx=0).stop 2024-04-16T16:20:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.454+0000 7f3095600640 1 -- 172.21.15.45:0/4078052370 shutdown_connections 2024-04-16T16:20:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.454+0000 7f3095600640 1 --2- 172.21.15.45:0/4078052370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309006d620 0x7f30900d35d0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.454+0000 7f3095600640 1 --2- 172.21.15.45:0/4078052370 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f309015ecc0 0x7f30901626a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.454+0000 7f3095600640 1 -- 172.21.15.45:0/4078052370 >> 172.21.15.45:0/4078052370 conn(0x7f3090073690 msgr2=0x7f30900d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.454+0000 7f3095600640 1 -- 172.21.15.45:0/4078052370 shutdown_connections 2024-04-16T16:20:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.454+0000 7f3095600640 1 -- 172.21.15.45:0/4078052370 wait complete. 2024-04-16T16:20:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.456+0000 7f3095600640 1 Processor -- start 2024-04-16T16:20:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.456+0000 7f3095600640 1 -- start start 2024-04-16T16:20:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.457+0000 7f3095600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309006d620 0x7f309014eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.457+0000 7f3095600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f309015ecc0 0x7f3090160860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.457+0000 7f3095600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3090162be0 0x7f30900b8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.457+0000 7f3095600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f30900d3b10 con 0x7f309006d620 2024-04-16T16:20:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.457+0000 7f3095600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f30900ba540 con 0x7f309015ecc0 2024-04-16T16:20:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.457+0000 7f3095600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f30900d3c90 con 0x7f3090162be0 2024-04-16T16:20:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.457+0000 7f308ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309006d620 0x7f309014eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.457+0000 7f308ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309006d620 0x7f309014eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49808/0 (socket says 172.21.15.45:49808) 2024-04-16T16:20:23.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.457+0000 7f308ea00640 1 -- 172.21.15.45:0/3640527870 learned_addr learned my addr 172.21.15.45:0/3640527870 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:23.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.458+0000 7f308f400640 1 --2- 172.21.15.45:0/3640527870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3090162be0 0x7f30900b8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.458+0000 7f308ea00640 1 -- 172.21.15.45:0/3640527870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3090162be0 msgr2=0x7f30900b8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.458+0000 7f308e000640 1 --2- 172.21.15.45:0/3640527870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f309015ecc0 0x7f3090160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.458+0000 7f308ea00640 1 --2- 172.21.15.45:0/3640527870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3090162be0 0x7f30900b8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.458+0000 7f308ea00640 1 -- 172.21.15.45:0/3640527870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f309015ecc0 msgr2=0x7f3090160860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.458+0000 7f308ea00640 1 --2- 172.21.15.45:0/3640527870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f309015ecc0 0x7f3090160860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.458+0000 7f308ea00640 1 -- 172.21.15.45:0/3640527870 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f309015f2f0 con 0x7f309006d620 2024-04-16T16:20:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.458+0000 7f308ea00640 1 --2- 172.21.15.45:0/3640527870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309006d620 0x7f309014eb50 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f308000c7f0 tx=0x7f308000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.458+0000 7f3086a00640 1 -- 172.21.15.45:0/3640527870 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3080015070 con 0x7f309006d620 2024-04-16T16:20:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.458+0000 7f3086a00640 1 -- 172.21.15.45:0/3640527870 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3080007a30 con 0x7f309006d620 2024-04-16T16:20:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.458+0000 7f3086a00640 1 -- 172.21.15.45:0/3640527870 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3080007e00 con 0x7f309006d620 2024-04-16T16:20:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.459+0000 7f3095600640 1 -- 172.21.15.45:0/3640527870 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f309015f5d0 con 0x7f309006d620 2024-04-16T16:20:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.459+0000 7f3095600640 1 -- 172.21.15.45:0/3640527870 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f309015fb90 con 0x7f309006d620 2024-04-16T16:20:23.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.460+0000 7f3095600640 1 -- 172.21.15.45:0/3640527870 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3090075780 con 0x7f309006d620 2024-04-16T16:20:23.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.461+0000 7f3086a00640 1 -- 172.21.15.45:0/3640527870 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f3080018070 con 0x7f309006d620 2024-04-16T16:20:23.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.462+0000 7f3086a00640 1 --2- 172.21.15.45:0/3640527870 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f306c06ca10 0x7f306c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:23.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.462+0000 7f3086a00640 1 -- 172.21.15.45:0/3640527870 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 5631+0+0 (secure 0 0 0) 0x7f308008be40 con 0x7f309006d620 2024-04-16T16:20:23.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.463+0000 7f308e000640 1 --2- 172.21.15.45:0/3640527870 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f306c06ca10 0x7f306c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:23.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.464+0000 7f308e000640 1 --2- 172.21.15.45:0/3640527870 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f306c06ca10 0x7f306c06eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f30900b9d10 tx=0x7f3078007240 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:23.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.467+0000 7f3086a00640 1 -- 172.21.15.45:0/3640527870 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f308005a000 con 0x7f309006d620 2024-04-16T16:20:23.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.693+0000 7f3095600640 1 -- 172.21.15.45:0/3640527870 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f3090158d30 con 0x7f309006d620 2024-04-16T16:20:23.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.701+0000 7f3086a00640 1 -- 172.21.15.45:0/3640527870 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7f3080059900 con 0x7f309006d620 2024-04-16T16:20:23.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.703+0000 7f3095600640 1 -- 172.21.15.45:0/3640527870 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f306c06ca10 msgr2=0x7f306c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:23.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.703+0000 7f3095600640 1 --2- 172.21.15.45:0/3640527870 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f306c06ca10 0x7f306c06eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f30900b9d10 tx=0x7f3078007240 comp rx=0 tx=0).stop 2024-04-16T16:20:23.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.703+0000 7f3095600640 1 -- 172.21.15.45:0/3640527870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309006d620 msgr2=0x7f309014eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:23.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.703+0000 7f3095600640 1 --2- 172.21.15.45:0/3640527870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309006d620 0x7f309014eb50 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f308000c7f0 tx=0x7f308000cbf0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.704+0000 7f3095600640 1 -- 172.21.15.45:0/3640527870 shutdown_connections 2024-04-16T16:20:23.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.704+0000 7f3095600640 1 --2- 172.21.15.45:0/3640527870 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f306c06ca10 0x7f306c06eed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.704+0000 7f3095600640 1 --2- 172.21.15.45:0/3640527870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3090162be0 0x7f30900b8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.704+0000 7f3095600640 1 --2- 172.21.15.45:0/3640527870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f309015ecc0 0x7f3090160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.704+0000 7f3095600640 1 --2- 172.21.15.45:0/3640527870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309006d620 0x7f309014eb50 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.704+0000 7f3095600640 1 -- 172.21.15.45:0/3640527870 >> 172.21.15.45:0/3640527870 conn(0x7f3090073690 msgr2=0x7f309006cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:23.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.704+0000 7f3095600640 1 -- 172.21.15.45:0/3640527870 shutdown_connections 2024-04-16T16:20:23.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.704+0000 7f3095600640 1 -- 172.21.15.45:0/3640527870 wait complete. 2024-04-16T16:20:23.722 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_mds_rank' 2024-04-16T16:20:23.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.934+0000 7f443a000640 1 Processor -- start 2024-04-16T16:20:23.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.934+0000 7f443a000640 1 -- start start 2024-04-16T16:20:23.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.936+0000 7f443a000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f44340d1530 0x7f44340b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:23.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.936+0000 7f443a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44340b6c60 0x7f44340cf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:23.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.936+0000 7f443a000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44340d19d0 con 0x7f443415c790 2024-04-16T16:20:23.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.936+0000 7f443a000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f44340d3180 con 0x7f44340d1530 2024-04-16T16:20:23.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.936+0000 7f443a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f44340d1cb0 con 0x7f44340b6c60 2024-04-16T16:20:23.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.937+0000 7f4439600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44340b6c60 0x7f44340cf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:23.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.937+0000 7f4433e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f44340d1530 0x7f44340b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:23.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.937+0000 7f4433e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f44340d1530 0x7f44340b6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:37570/0 (socket says 172.21.15.45:37570) 2024-04-16T16:20:23.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.937+0000 7f4439600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44340b6c60 0x7f44340cf300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:45850/0 (socket says 172.21.15.45:45850) 2024-04-16T16:20:23.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.937+0000 7f4433e00640 1 -- 172.21.15.45:0/1254636985 learned_addr learned my addr 172.21.15.45:0/1254636985 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:23.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.938+0000 7f4433e00640 1 -- 172.21.15.45:0/1254636985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44340b6c60 msgr2=0x7f44340cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:23.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.938+0000 7f4433e00640 1 --2- 172.21.15.45:0/1254636985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44340b6c60 0x7f44340cf300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.938+0000 7f4433e00640 1 -- 172.21.15.45:0/1254636985 >> v1:172.21.15.103:6789/0 conn(0x7f443415c790 legacy=0x7f44340d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:23.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.938+0000 7f4433e00640 1 -- 172.21.15.45:0/1254636985 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44340d1f90 con 0x7f44340d1530 2024-04-16T16:20:23.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.939+0000 7f4433e00640 1 --2- 172.21.15.45:0/1254636985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44340d1530 0x7f44340b6720 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f44340748c0 tx=0x7f442002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e3f37fa77edbb5d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:23.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.940+0000 7f4433400640 1 -- 172.21.15.45:0/1254636985 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4420035020 con 0x7f44340d1530 2024-04-16T16:20:23.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.940+0000 7f4433400640 1 -- 172.21.15.45:0/1254636985 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4420031830 con 0x7f44340d1530 2024-04-16T16:20:23.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.940+0000 7f4433400640 1 -- 172.21.15.45:0/1254636985 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4420031c00 con 0x7f44340d1530 2024-04-16T16:20:23.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.940+0000 7f443a000640 1 -- 172.21.15.45:0/1254636985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44340d1530 msgr2=0x7f44340b6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:23.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.940+0000 7f443a000640 1 --2- 172.21.15.45:0/1254636985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44340d1530 0x7f44340b6720 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f44340748c0 tx=0x7f442002b8d0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.940+0000 7f443a000640 1 -- 172.21.15.45:0/1254636985 shutdown_connections 2024-04-16T16:20:23.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.940+0000 7f443a000640 1 --2- 172.21.15.45:0/1254636985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44340b6c60 0x7f44340cf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.940+0000 7f443a000640 1 --2- 172.21.15.45:0/1254636985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44340d1530 0x7f44340b6720 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.940+0000 7f443a000640 1 -- 172.21.15.45:0/1254636985 >> 172.21.15.45:0/1254636985 conn(0x7f44340b43a0 msgr2=0x7f44340b4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:23.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.941+0000 7f443a000640 1 -- 172.21.15.45:0/1254636985 shutdown_connections 2024-04-16T16:20:23.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.941+0000 7f443a000640 1 -- 172.21.15.45:0/1254636985 wait complete. 2024-04-16T16:20:23.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.942+0000 7f443a000640 1 Processor -- start 2024-04-16T16:20:23.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.943+0000 7f443a000640 1 -- start start 2024-04-16T16:20:23.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.943+0000 7f443a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44340b6c60 0x7f44341504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:23.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.943+0000 7f443a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44340d1530 0x7f443406da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:23.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.943+0000 7f443a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f443415c790 0x7f443406dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:23.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.943+0000 7f443a000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f44340cf840 con 0x7f44340d1530 2024-04-16T16:20:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.943+0000 7f443a000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f44340d2550 con 0x7f44340b6c60 2024-04-16T16:20:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.943+0000 7f443a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f44340cf9c0 con 0x7f443415c790 2024-04-16T16:20:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.944+0000 7f4433e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44340d1530 0x7f443406da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.944+0000 7f4433e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44340d1530 0x7f443406da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49824/0 (socket says 172.21.15.45:49824) 2024-04-16T16:20:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.944+0000 7f4433e00640 1 -- 172.21.15.45:0/4197579432 learned_addr learned my addr 172.21.15.45:0/4197579432 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:23.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.944+0000 7f4439600640 1 --2- 172.21.15.45:0/4197579432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f443415c790 0x7f443406dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:23.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.944+0000 7f4438c00640 1 --2- 172.21.15.45:0/4197579432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44340b6c60 0x7f44341504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:23.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.944+0000 7f4433e00640 1 -- 172.21.15.45:0/4197579432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f443415c790 msgr2=0x7f443406dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:23.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.944+0000 7f4433e00640 1 --2- 172.21.15.45:0/4197579432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f443415c790 0x7f443406dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.944+0000 7f4433e00640 1 -- 172.21.15.45:0/4197579432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44340b6c60 msgr2=0x7f44341504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.944+0000 7f4433e00640 1 --2- 172.21.15.45:0/4197579432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44340b6c60 0x7f44341504a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.944+0000 7f4433e00640 1 -- 172.21.15.45:0/4197579432 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443415cc10 con 0x7f44340d1530 2024-04-16T16:20:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.945+0000 7f4438c00640 1 --2- 172.21.15.45:0/4197579432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44340b6c60 0x7f44341504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.945+0000 7f4433e00640 1 --2- 172.21.15.45:0/4197579432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44340d1530 0x7f443406da90 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f4420000c00 tx=0x7f44200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.945+0000 7f4430c00640 1 -- 172.21.15.45:0/4197579432 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4420040070 con 0x7f44340d1530 2024-04-16T16:20:23.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.945+0000 7f4430c00640 1 -- 172.21.15.45:0/4197579432 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f442003a830 con 0x7f44340d1530 2024-04-16T16:20:23.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.945+0000 7f443a000640 1 -- 172.21.15.45:0/4197579432 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f443415ce80 con 0x7f44340d1530 2024-04-16T16:20:23.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.945+0000 7f4430c00640 1 -- 172.21.15.45:0/4197579432 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4420035040 con 0x7f44340d1530 2024-04-16T16:20:23.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.945+0000 7f443a000640 1 -- 172.21.15.45:0/4197579432 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44341509e0 con 0x7f44340d1530 2024-04-16T16:20:23.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.947+0000 7f443a000640 1 -- 172.21.15.45:0/4197579432 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4434075780 con 0x7f44340d1530 2024-04-16T16:20:23.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.948+0000 7f4430c00640 1 -- 172.21.15.45:0/4197579432 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f442004b060 con 0x7f44340d1530 2024-04-16T16:20:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.949+0000 7f4430c00640 1 --2- 172.21.15.45:0/4197579432 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f441406ca10 0x7f441406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.949+0000 7f4430c00640 1 -- 172.21.15.45:0/4197579432 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 5631+0+0 (secure 0 0 0) 0x7f44200ad4e0 con 0x7f44340d1530 2024-04-16T16:20:23.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.950+0000 7f4438c00640 1 --2- 172.21.15.45:0/4197579432 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f441406ca10 0x7f441406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:23.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.951+0000 7f4438c00640 1 --2- 172.21.15.45:0/4197579432 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f441406ca10 0x7f441406eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f44280012e0 tx=0x7f4428009040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:23.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:23.953+0000 7f4430c00640 1 -- 172.21.15.45:0/4197579432 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4420031800 con 0x7f44340d1530 2024-04-16T16:20:24.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:24.236+0000 7f443a000640 1 -- 172.21.15.45:0/4197579432 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_mds_rank"]} v 0) -- 0x7f44340b2b90 con 0x7f44340d1530 2024-04-16T16:20:24.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:20:24.980 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:20:25.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.225+0000 7f4430c00640 1 -- 172.21.15.45:0/4197579432 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_for_invalid_mds_rank"]}]=0 v404) ==== 149+0+0 (secure 0 0 0) 0x7f442007b840 con 0x7f44340d1530 2024-04-16T16:20:25.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.229+0000 7f443a000640 1 -- 172.21.15.45:0/4197579432 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f441406ca10 msgr2=0x7f441406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:25.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.229+0000 7f443a000640 1 --2- 172.21.15.45:0/4197579432 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f441406ca10 0x7f441406eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f44280012e0 tx=0x7f4428009040 comp rx=0 tx=0).stop 2024-04-16T16:20:25.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.229+0000 7f443a000640 1 -- 172.21.15.45:0/4197579432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44340d1530 msgr2=0x7f443406da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:25.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.229+0000 7f443a000640 1 --2- 172.21.15.45:0/4197579432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44340d1530 0x7f443406da90 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f4420000c00 tx=0x7f44200046c0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.229+0000 7f443a000640 1 -- 172.21.15.45:0/4197579432 shutdown_connections 2024-04-16T16:20:25.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.229+0000 7f443a000640 1 --2- 172.21.15.45:0/4197579432 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f441406ca10 0x7f441406eed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.229+0000 7f443a000640 1 --2- 172.21.15.45:0/4197579432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f443415c790 0x7f443406dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.229+0000 7f443a000640 1 --2- 172.21.15.45:0/4197579432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44340d1530 0x7f443406da90 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.229+0000 7f443a000640 1 --2- 172.21.15.45:0/4197579432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44340b6c60 0x7f44341504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.229+0000 7f443a000640 1 -- 172.21.15.45:0/4197579432 >> 172.21.15.45:0/4197579432 conn(0x7f44340b43a0 msgr2=0x7f443415b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:25.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.230+0000 7f443a000640 1 -- 172.21.15.45:0/4197579432 shutdown_connections 2024-04-16T16:20:25.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.230+0000 7f443a000640 1 -- 172.21.15.45:0/4197579432 wait complete. 2024-04-16T16:20:25.248 INFO:tasks.cephfs_test_runner:test_for_invalid_mds_rank (tasks.cephfs.test_mds_metrics.TestMDSMetrics) ... ok 2024-04-16T16:20:25.248 INFO:tasks.cephfs_test_runner:Starting test: test_metrics_from_rank (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2024-04-16T16:20:25.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:20:25.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.461+0000 7fea21c00640 1 Processor -- start 2024-04-16T16:20:25.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.462+0000 7fea21c00640 1 -- start start 2024-04-16T16:20:25.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.463+0000 7fea21c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea1c161e90 0x7fea1c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:25.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.463+0000 7fea21c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fea1c162270 0x7fea1c0d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:25.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.464+0000 7fea21c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea1c0d3080 con 0x7fea1c0d52d0 2024-04-16T16:20:25.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.464+0000 7fea21c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fea1c0d2e30 con 0x7fea1c162270 2024-04-16T16:20:25.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.464+0000 7fea21c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fea1c0d3360 con 0x7fea1c161e90 2024-04-16T16:20:25.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.464+0000 7fea1be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea1c161e90 0x7fea1c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:25.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.464+0000 7fea1b400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fea1c162270 0x7fea1c0d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:25.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.464+0000 7fea1be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea1c161e90 0x7fea1c0cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:45880/0 (socket says 172.21.15.45:45880) 2024-04-16T16:20:25.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.464+0000 7fea1be00640 1 -- 172.21.15.45:0/1526401939 learned_addr learned my addr 172.21.15.45:0/1526401939 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:25.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.465+0000 7fea1be00640 1 -- 172.21.15.45:0/1526401939 >> v2:172.21.15.103:3300/0 conn(0x7fea1c162270 msgr2=0x7fea1c0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:25.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.465+0000 7fea1be00640 1 --2- 172.21.15.45:0/1526401939 >> v2:172.21.15.103:3300/0 conn(0x7fea1c162270 0x7fea1c0d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.465+0000 7fea1be00640 1 -- 172.21.15.45:0/1526401939 >> v1:172.21.15.103:6789/0 conn(0x7fea1c0d52d0 legacy=0x7fea1c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:25.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.465+0000 7fea1be00640 1 -- 172.21.15.45:0/1526401939 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1c0d3640 con 0x7fea1c161e90 2024-04-16T16:20:25.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.467+0000 7fea1be00640 1 --2- 172.21.15.45:0/1526401939 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea1c161e90 0x7fea1c0cf7d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fea0c009e50 tx=0x7fea0c02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=116bdc673af7000 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:25.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.467+0000 7fea1aa00640 1 -- 172.21.15.45:0/1526401939 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c02cb90 con 0x7fea1c161e90 2024-04-16T16:20:25.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.467+0000 7fea1aa00640 1 -- 172.21.15.45:0/1526401939 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea0c02cd30 con 0x7fea1c161e90 2024-04-16T16:20:25.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.467+0000 7fea1aa00640 1 -- 172.21.15.45:0/1526401939 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c034610 con 0x7fea1c161e90 2024-04-16T16:20:25.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.468+0000 7fea21c00640 1 -- 172.21.15.45:0/1526401939 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea1c161e90 msgr2=0x7fea1c0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:25.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.468+0000 7fea21c00640 1 --2- 172.21.15.45:0/1526401939 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea1c161e90 0x7fea1c0cf7d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fea0c009e50 tx=0x7fea0c02bff0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.468+0000 7fea21c00640 1 -- 172.21.15.45:0/1526401939 shutdown_connections 2024-04-16T16:20:25.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.468+0000 7fea21c00640 1 --2- 172.21.15.45:0/1526401939 >> v2:172.21.15.103:3300/0 conn(0x7fea1c162270 0x7fea1c0d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.468+0000 7fea21c00640 1 --2- 172.21.15.45:0/1526401939 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea1c161e90 0x7fea1c0cf7d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.468+0000 7fea21c00640 1 -- 172.21.15.45:0/1526401939 >> 172.21.15.45:0/1526401939 conn(0x7fea1c0d1940 msgr2=0x7fea1c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:25.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.468+0000 7fea21c00640 1 -- 172.21.15.45:0/1526401939 shutdown_connections 2024-04-16T16:20:25.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.468+0000 7fea21c00640 1 -- 172.21.15.45:0/1526401939 wait complete. 2024-04-16T16:20:25.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.470+0000 7fea21c00640 1 Processor -- start 2024-04-16T16:20:25.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.470+0000 7fea21c00640 1 -- start start 2024-04-16T16:20:25.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.471+0000 7fea21c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea1c0d52d0 0x7fea1c0ce790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:25.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.471+0000 7fea21c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea1c161e90 0x7fea1c15de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:25.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.471+0000 7fea21c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea1c162270 0x7fea1c151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:25.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.471+0000 7fea21c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fea1c06d550 con 0x7fea1c162270 2024-04-16T16:20:25.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.471+0000 7fea21c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea1c0cfd10 con 0x7fea1c0d52d0 2024-04-16T16:20:25.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.471+0000 7fea21c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fea1c06d6d0 con 0x7fea1c161e90 2024-04-16T16:20:25.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.472+0000 7fea21200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea1c162270 0x7fea1c151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:25.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.472+0000 7fea21200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea1c162270 0x7fea1c151bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49834/0 (socket says 172.21.15.45:49834) 2024-04-16T16:20:25.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.472+0000 7fea21200640 1 -- 172.21.15.45:0/2723371366 learned_addr learned my addr 172.21.15.45:0/2723371366 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:25.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.472+0000 7fea1be00640 1 --2- 172.21.15.45:0/2723371366 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea1c0d52d0 0x7fea1c0ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:25.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.472+0000 7fea1b400640 1 --2- 172.21.15.45:0/2723371366 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea1c161e90 0x7fea1c15de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:25.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.472+0000 7fea21200640 1 -- 172.21.15.45:0/2723371366 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea1c161e90 msgr2=0x7fea1c15de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:25.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.472+0000 7fea21200640 1 --2- 172.21.15.45:0/2723371366 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea1c161e90 0x7fea1c15de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.472+0000 7fea21200640 1 -- 172.21.15.45:0/2723371366 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea1c0d52d0 msgr2=0x7fea1c0ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:25.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.472+0000 7fea21200640 1 --2- 172.21.15.45:0/2723371366 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea1c0d52d0 0x7fea1c0ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.472+0000 7fea21200640 1 -- 172.21.15.45:0/2723371366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1c15cdd0 con 0x7fea1c162270 2024-04-16T16:20:25.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.473+0000 7fea21200640 1 --2- 172.21.15.45:0/2723371366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea1c162270 0x7fea1c151bb0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fea14001eb0 tx=0x7fea1400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:25.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.473+0000 7fea13e00640 1 -- 172.21.15.45:0/2723371366 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea14017070 con 0x7fea1c162270 2024-04-16T16:20:25.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.473+0000 7fea13e00640 1 -- 172.21.15.45:0/2723371366 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea140052d0 con 0x7fea1c162270 2024-04-16T16:20:25.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.473+0000 7fea13e00640 1 -- 172.21.15.45:0/2723371366 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea14003810 con 0x7fea1c162270 2024-04-16T16:20:25.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.473+0000 7fea21c00640 1 -- 172.21.15.45:0/2723371366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea1c15d0b0 con 0x7fea1c162270 2024-04-16T16:20:25.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.473+0000 7fea21c00640 1 -- 172.21.15.45:0/2723371366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea1c15d670 con 0x7fea1c162270 2024-04-16T16:20:25.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.474+0000 7fea21c00640 1 -- 172.21.15.45:0/2723371366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea1c075780 con 0x7fea1c162270 2024-04-16T16:20:25.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.475+0000 7fea13e00640 1 -- 172.21.15.45:0/2723371366 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7fea1401a070 con 0x7fea1c162270 2024-04-16T16:20:25.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.476+0000 7fea13e00640 1 --2- 172.21.15.45:0/2723371366 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe9f806ca10 0x7fe9f806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:25.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.476+0000 7fea13e00640 1 -- 172.21.15.45:0/2723371366 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 5631+0+0 (secure 0 0 0) 0x7fea1408afa0 con 0x7fea1c162270 2024-04-16T16:20:25.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.477+0000 7fea1be00640 1 --2- 172.21.15.45:0/2723371366 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe9f806ca10 0x7fe9f806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:25.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.478+0000 7fea1be00640 1 --2- 172.21.15.45:0/2723371366 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe9f806ca10 0x7fe9f806eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fea0c035000 tx=0x7fea0c030000 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:25.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.481+0000 7fea13e00640 1 -- 172.21.15.45:0/2723371366 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fea140593a0 con 0x7fea1c162270 2024-04-16T16:20:25.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.687+0000 7fea21c00640 1 -- 172.21.15.45:0/2723371366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fea1c06dae0 con 0x7fea1c162270 2024-04-16T16:20:25.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.689+0000 7fea13e00640 1 -- 172.21.15.45:0/2723371366 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) ==== 74+0+14046 (secure 0 0 0) 0x7fea14058ca0 con 0x7fea1c162270 2024-04-16T16:20:25.689 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:25.689 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":120,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:20:23.192619+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/383252954":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/428120997":"2024-04-17T16:20:13.464929+0000","172.21.15.45:6832/3429980864":"2024-04-17T16:20:20.041556+0000","172.21.15.103:0/3229718691":"2024-04-17T16:20:13.464929+0000","172.21.15.103:6833/1934318167":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/3469935364":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/3506765807":"2024-04-17T16:20:07.040813+0000","172.21.15.103:6832/1934318167":"2024-04-17T16:20:13.464929+0000","172.21.15.103:0/3547880684":"2024-04-17T16:20:07.040813+0000","172.21.15.103:6833/2293932214":"2024-04-17T16:20:07.040813+0000","172.21.15.45:6834/3429980864":"2024-04-17T16:20:20.041556+0000","172.21.15.103:0/1890933611":"2024-04-17T16:20:13.464929+0000","172.21.15.103:6832/2293932214":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/1347839066":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/546409297":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/3566291251":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/714185246":"2024-04-17T16:20:07.040813+0000","172.21.15.103:0/2784388736":"2024-04-16T17:19:54.469649+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:20:25.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.692+0000 7fea21c00640 1 -- 172.21.15.45:0/2723371366 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe9f806ca10 msgr2=0x7fe9f806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:25.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.692+0000 7fea21c00640 1 --2- 172.21.15.45:0/2723371366 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe9f806ca10 0x7fe9f806eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fea0c035000 tx=0x7fea0c030000 comp rx=0 tx=0).stop 2024-04-16T16:20:25.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.692+0000 7fea21c00640 1 -- 172.21.15.45:0/2723371366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea1c162270 msgr2=0x7fea1c151bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:25.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.692+0000 7fea21c00640 1 --2- 172.21.15.45:0/2723371366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea1c162270 0x7fea1c151bb0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fea14001eb0 tx=0x7fea1400c890 comp rx=0 tx=0).stop 2024-04-16T16:20:25.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.692+0000 7fea21c00640 1 -- 172.21.15.45:0/2723371366 shutdown_connections 2024-04-16T16:20:25.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.692+0000 7fea21c00640 1 --2- 172.21.15.45:0/2723371366 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe9f806ca10 0x7fe9f806eed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.692+0000 7fea21c00640 1 --2- 172.21.15.45:0/2723371366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea1c162270 0x7fea1c151bb0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.693+0000 7fea21c00640 1 --2- 172.21.15.45:0/2723371366 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea1c161e90 0x7fea1c15de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.693+0000 7fea21c00640 1 --2- 172.21.15.45:0/2723371366 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea1c0d52d0 0x7fea1c0ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.693+0000 7fea21c00640 1 -- 172.21.15.45:0/2723371366 >> 172.21.15.45:0/2723371366 conn(0x7fea1c0d1940 msgr2=0x7fea1c0b8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:25.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.693+0000 7fea21c00640 1 -- 172.21.15.45:0/2723371366 shutdown_connections 2024-04-16T16:20:25.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.693+0000 7fea21c00640 1 -- 172.21.15.45:0/2723371366 wait complete. 2024-04-16T16:20:25.710 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:20:25.711 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:20:25.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.909+0000 7f6144e00640 1 Processor -- start 2024-04-16T16:20:25.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.909+0000 7f6144e00640 1 -- start start 2024-04-16T16:20:25.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.910+0000 7f6144e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6140162720 0x7f61400d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:25.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.910+0000 7f6144e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f614006ca40 0x7f61400d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:25.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.910+0000 7f6144e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f614006cfe0 con 0x7f614009ab50 2024-04-16T16:20:25.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.910+0000 7f6144e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f614006dbf0 con 0x7f614006ca40 2024-04-16T16:20:25.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.911+0000 7f6144e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f614006d2c0 con 0x7f6140162720 2024-04-16T16:20:25.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.911+0000 7f613de00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f614009ab50 0x7f614009af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57212/0 (socket says 172.21.15.45:57212) 2024-04-16T16:20:25.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.911+0000 7f613de00640 1 -- 172.21.15.45:0/3776181113 learned_addr learned my addr 172.21.15.45:0/3776181113 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:25.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.911+0000 7f613e800640 1 --2- 172.21.15.45:0/3776181113 >> v2:172.21.15.45:3300/0 conn(0x7f614006ca40 0x7f61400d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:25.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.911+0000 7f613d400640 1 --2- 172.21.15.45:0/3776181113 >> v2:172.21.15.103:3300/0 conn(0x7f6140162720 0x7f61400d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:25.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.911+0000 7f613e800640 1 -- 172.21.15.45:0/3776181113 >> v2:172.21.15.103:3300/0 conn(0x7f6140162720 msgr2=0x7f61400d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:25.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.911+0000 7f613e800640 1 --2- 172.21.15.45:0/3776181113 >> v2:172.21.15.103:3300/0 conn(0x7f6140162720 0x7f61400d0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.911+0000 7f613e800640 1 -- 172.21.15.45:0/3776181113 >> v1:172.21.15.45:6789/0 conn(0x7f614009ab50 legacy=0x7f614009af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:25.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.911+0000 7f613e800640 1 -- 172.21.15.45:0/3776181113 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614006d5a0 con 0x7f614006ca40 2024-04-16T16:20:25.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.911+0000 7f613ca00640 1 -- 172.21.15.45:0/3776181113 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1651260735 0 0) 0x7f614006cfe0 con 0x7f614009ab50 2024-04-16T16:20:25.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.911+0000 7f613d400640 1 --2- 172.21.15.45:0/3776181113 >> v2:172.21.15.103:3300/0 conn(0x7f6140162720 0x7f61400d0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:25.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.913+0000 7f613e800640 1 --2- 172.21.15.45:0/3776181113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f614006ca40 0x7f61400d11f0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f61380097a0 tx=0x7f613802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4141980a6dc45a6c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:25.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.914+0000 7f613ca00640 1 -- 172.21.15.45:0/3776181113 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6138038070 con 0x7f614006ca40 2024-04-16T16:20:25.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.914+0000 7f613ca00640 1 -- 172.21.15.45:0/3776181113 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6138031700 con 0x7f614006ca40 2024-04-16T16:20:25.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.914+0000 7f613ca00640 1 -- 172.21.15.45:0/3776181113 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6138031ad0 con 0x7f614006ca40 2024-04-16T16:20:25.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.914+0000 7f6144e00640 1 -- 172.21.15.45:0/3776181113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f614006ca40 msgr2=0x7f61400d11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:25.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.914+0000 7f6144e00640 1 --2- 172.21.15.45:0/3776181113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f614006ca40 0x7f61400d11f0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f61380097a0 tx=0x7f613802b7a0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.914+0000 7f6144e00640 1 -- 172.21.15.45:0/3776181113 shutdown_connections 2024-04-16T16:20:25.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.914+0000 7f6144e00640 1 --2- 172.21.15.45:0/3776181113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f614006ca40 0x7f61400d11f0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.914+0000 7f6144e00640 1 --2- 172.21.15.45:0/3776181113 >> v2:172.21.15.103:3300/0 conn(0x7f6140162720 0x7f61400d0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.914+0000 7f6144e00640 1 -- 172.21.15.45:0/3776181113 >> 172.21.15.45:0/3776181113 conn(0x7f61400d2890 msgr2=0x7f61400d2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:25.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.914+0000 7f6144e00640 1 -- 172.21.15.45:0/3776181113 shutdown_connections 2024-04-16T16:20:25.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.914+0000 7f6144e00640 1 -- 172.21.15.45:0/3776181113 wait complete. 2024-04-16T16:20:25.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.916+0000 7f6144e00640 1 Processor -- start 2024-04-16T16:20:25.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.916+0000 7f6144e00640 1 -- start start 2024-04-16T16:20:25.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.917+0000 7f6144e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f614006ca40 0x7f61400b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:25.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.917+0000 7f6144e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614009ab50 0x7f61401556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:25.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.917+0000 7f6144e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6140162720 0x7f61401e8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:25.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.917+0000 7f6144e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f614015ff00 con 0x7f6140162720 2024-04-16T16:20:25.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.917+0000 7f6144e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f614009b8b0 con 0x7f614006ca40 2024-04-16T16:20:25.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.917+0000 7f6144e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6140160080 con 0x7f614009ab50 2024-04-16T16:20:25.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.918+0000 7f613e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6140162720 0x7f61401e8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:25.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.918+0000 7f613e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6140162720 0x7f61401e8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49856/0 (socket says 172.21.15.45:49856) 2024-04-16T16:20:25.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.918+0000 7f613e800640 1 -- 172.21.15.45:0/2193723790 learned_addr learned my addr 172.21.15.45:0/2193723790 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:25.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.918+0000 7f613d400640 1 --2- 172.21.15.45:0/2193723790 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614009ab50 0x7f61401556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:25.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.918+0000 7f613de00640 1 --2- 172.21.15.45:0/2193723790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f614006ca40 0x7f61400b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:25.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.918+0000 7f613e800640 1 -- 172.21.15.45:0/2193723790 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614009ab50 msgr2=0x7f61401556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:25.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.918+0000 7f613e800640 1 --2- 172.21.15.45:0/2193723790 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614009ab50 0x7f61401556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.918+0000 7f613e800640 1 -- 172.21.15.45:0/2193723790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f614006ca40 msgr2=0x7f61400b7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:25.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.918+0000 7f613e800640 1 --2- 172.21.15.45:0/2193723790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f614006ca40 0x7f61400b7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:25.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.918+0000 7f613e800640 1 -- 172.21.15.45:0/2193723790 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61400b6ec0 con 0x7f6140162720 2024-04-16T16:20:25.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.919+0000 7f613e800640 1 --2- 172.21.15.45:0/2193723790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6140162720 0x7f61401e8e30 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f6138036040 tx=0x7f61380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:25.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.919+0000 7f6136000640 1 -- 172.21.15.45:0/2193723790 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6138042070 con 0x7f6140162720 2024-04-16T16:20:25.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.919+0000 7f6136000640 1 -- 172.21.15.45:0/2193723790 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61380339b0 con 0x7f6140162720 2024-04-16T16:20:25.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.919+0000 7f6136000640 1 -- 172.21.15.45:0/2193723790 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6138038040 con 0x7f6140162720 2024-04-16T16:20:25.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.919+0000 7f6144e00640 1 -- 172.21.15.45:0/2193723790 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61400b71a0 con 0x7f6140162720 2024-04-16T16:20:25.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.919+0000 7f6144e00640 1 -- 172.21.15.45:0/2193723790 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61400b7760 con 0x7f6140162720 2024-04-16T16:20:25.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.921+0000 7f6144e00640 1 -- 172.21.15.45:0/2193723790 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6140075780 con 0x7f6140162720 2024-04-16T16:20:25.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.923+0000 7f6136000640 1 -- 172.21.15.45:0/2193723790 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f6138045070 con 0x7f6140162720 2024-04-16T16:20:25.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.924+0000 7f6136000640 1 --2- 172.21.15.45:0/2193723790 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f611c06ca40 0x7f611c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:25.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.924+0000 7f6136000640 1 -- 172.21.15.45:0/2193723790 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 5631+0+0 (secure 0 0 0) 0x7f61380ad9b0 con 0x7f6140162720 2024-04-16T16:20:25.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.925+0000 7f613de00640 1 --2- 172.21.15.45:0/2193723790 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f611c06ca40 0x7f611c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:25.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.926+0000 7f613de00640 1 --2- 172.21.15.45:0/2193723790 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f611c06ca40 0x7f611c06ef00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6130000f60 tx=0x7f6130007b50 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:25.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:25.929+0000 7f6136000640 1 -- 172.21.15.45:0/2193723790 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6138030070 con 0x7f6140162720 2024-04-16T16:20:26.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.132+0000 7f6144e00640 1 -- 172.21.15.45:0/2193723790 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f61400d3710 con 0x7f6140162720 2024-04-16T16:20:26.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.132+0000 7f6136000640 1 -- 172.21.15.45:0/2193723790 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) ==== 93+0+10 (secure 0 0 0) 0x7f613807bd20 con 0x7f6140162720 2024-04-16T16:20:26.131 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:20:26.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.135+0000 7f6144e00640 1 -- 172.21.15.45:0/2193723790 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f611c06ca40 msgr2=0x7f611c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:26.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.135+0000 7f6144e00640 1 --2- 172.21.15.45:0/2193723790 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f611c06ca40 0x7f611c06ef00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6130000f60 tx=0x7f6130007b50 comp rx=0 tx=0).stop 2024-04-16T16:20:26.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.135+0000 7f6144e00640 1 -- 172.21.15.45:0/2193723790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6140162720 msgr2=0x7f61401e8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:26.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.135+0000 7f6144e00640 1 --2- 172.21.15.45:0/2193723790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6140162720 0x7f61401e8e30 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f6138036040 tx=0x7f61380046c0 comp rx=0 tx=0).stop 2024-04-16T16:20:26.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.135+0000 7f6144e00640 1 -- 172.21.15.45:0/2193723790 shutdown_connections 2024-04-16T16:20:26.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.135+0000 7f6144e00640 1 --2- 172.21.15.45:0/2193723790 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f611c06ca40 0x7f611c06ef00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:26.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.135+0000 7f6144e00640 1 --2- 172.21.15.45:0/2193723790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6140162720 0x7f61401e8e30 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:26.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.135+0000 7f6144e00640 1 --2- 172.21.15.45:0/2193723790 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f614009ab50 0x7f61401556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:26.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.135+0000 7f6144e00640 1 --2- 172.21.15.45:0/2193723790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f614006ca40 0x7f61400b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:26.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.135+0000 7f6144e00640 1 -- 172.21.15.45:0/2193723790 >> 172.21.15.45:0/2193723790 conn(0x7f61400d2890 msgr2=0x7f61400709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:26.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.135+0000 7f6144e00640 1 -- 172.21.15.45:0/2193723790 shutdown_connections 2024-04-16T16:20:26.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.135+0000 7f6144e00640 1 -- 172.21.15.45:0/2193723790 wait complete. 2024-04-16T16:20:26.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_metrics_from_rank' 2024-04-16T16:20:26.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.354+0000 7febd0a00640 1 Processor -- start 2024-04-16T16:20:26.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.354+0000 7febd0a00640 1 -- start start 2024-04-16T16:20:26.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.356+0000 7febd0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febcc15ea70 0x7febcc0d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:26.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.356+0000 7febd0a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7febcc0cfc40 0x7febcc162560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:26.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.356+0000 7febd0a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febcc15ee50 con 0x7febcc0d4370 2024-04-16T16:20:26.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.356+0000 7febd0a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7febcc0d1a10 con 0x7febcc0cfc40 2024-04-16T16:20:26.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.356+0000 7febd0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7febcc0b6480 con 0x7febcc15ea70 2024-04-16T16:20:26.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.357+0000 7febca800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febcc15ea70 0x7febcc0d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:26.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.357+0000 7febcb200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7febcc0cfc40 0x7febcc162560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:26.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.357+0000 7febcb200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7febcc0cfc40 0x7febcc162560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:37616/0 (socket says 172.21.15.45:37616) 2024-04-16T16:20:26.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.357+0000 7febcb200640 1 -- 172.21.15.45:0/3712688786 learned_addr learned my addr 172.21.15.45:0/3712688786 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:26.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.357+0000 7febca800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febcc15ea70 0x7febcc0d11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:45918/0 (socket says 172.21.15.45:45918) 2024-04-16T16:20:26.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.357+0000 7febca800640 1 -- 172.21.15.45:0/3712688786 >> v2:172.21.15.103:3300/0 conn(0x7febcc0cfc40 msgr2=0x7febcc162560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:26.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.357+0000 7febca800640 1 --2- 172.21.15.45:0/3712688786 >> v2:172.21.15.103:3300/0 conn(0x7febcc0cfc40 0x7febcc162560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:26.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.357+0000 7febca800640 1 -- 172.21.15.45:0/3712688786 >> v1:172.21.15.103:6789/0 conn(0x7febcc0d4370 legacy=0x7febcc0cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:26.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.358+0000 7febca800640 1 -- 172.21.15.45:0/3712688786 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febcc0b6760 con 0x7febcc15ea70 2024-04-16T16:20:26.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.359+0000 7febca800640 1 --2- 172.21.15.45:0/3712688786 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febcc15ea70 0x7febcc0d11a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7febbc0097a0 tx=0x7febbc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e269ecd2a6d1e6ba server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:26.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.359+0000 7febc9400640 1 -- 172.21.15.45:0/3712688786 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febbc035020 con 0x7febcc15ea70 2024-04-16T16:20:26.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.360+0000 7febc9400640 1 -- 172.21.15.45:0/3712688786 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febbc031830 con 0x7febcc15ea70 2024-04-16T16:20:26.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.360+0000 7febc9400640 1 -- 172.21.15.45:0/3712688786 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febbc031c00 con 0x7febcc15ea70 2024-04-16T16:20:26.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.360+0000 7febd0a00640 1 -- 172.21.15.45:0/3712688786 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febcc15ea70 msgr2=0x7febcc0d11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:26.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.360+0000 7febd0a00640 1 --2- 172.21.15.45:0/3712688786 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febcc15ea70 0x7febcc0d11a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7febbc0097a0 tx=0x7febbc02b8d0 comp rx=0 tx=0).stop 2024-04-16T16:20:26.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.360+0000 7febd0a00640 1 -- 172.21.15.45:0/3712688786 shutdown_connections 2024-04-16T16:20:26.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.360+0000 7febd0a00640 1 --2- 172.21.15.45:0/3712688786 >> v2:172.21.15.103:3300/0 conn(0x7febcc0cfc40 0x7febcc162560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:26.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.360+0000 7febd0a00640 1 --2- 172.21.15.45:0/3712688786 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febcc15ea70 0x7febcc0d11a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:26.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.360+0000 7febd0a00640 1 -- 172.21.15.45:0/3712688786 >> 172.21.15.45:0/3712688786 conn(0x7febcc06d3b0 msgr2=0x7febcc073060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:26.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.360+0000 7febd0a00640 1 -- 172.21.15.45:0/3712688786 shutdown_connections 2024-04-16T16:20:26.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.360+0000 7febd0a00640 1 -- 172.21.15.45:0/3712688786 wait complete. 2024-04-16T16:20:26.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.362+0000 7febd0a00640 1 Processor -- start 2024-04-16T16:20:26.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.362+0000 7febd0a00640 1 -- start start 2024-04-16T16:20:26.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.363+0000 7febd0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0cfc40 0x7febcc0ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:26.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.363+0000 7febd0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febcc0d4370 0x7febcc15fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:26.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.363+0000 7febd0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febcc15ea70 0x7febcc1f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:26.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.363+0000 7febd0a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7febcc0b6f60 con 0x7febcc0cfc40 2024-04-16T16:20:26.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.363+0000 7febd0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7febcc0d16e0 con 0x7febcc15ea70 2024-04-16T16:20:26.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.363+0000 7febd0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7febcc1604d0 con 0x7febcc0d4370 2024-04-16T16:20:26.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.363+0000 7febca800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0cfc40 0x7febcc0ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:26.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.363+0000 7febca800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0cfc40 0x7febcc0ca910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49866/0 (socket says 172.21.15.45:49866) 2024-04-16T16:20:26.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.363+0000 7febca800640 1 -- 172.21.15.45:0/45820727 learned_addr learned my addr 172.21.15.45:0/45820727 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:26.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.364+0000 7febca800640 1 -- 172.21.15.45:0/45820727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febcc0d4370 msgr2=0x7febcc15fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:26.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.364+0000 7febca800640 1 --2- 172.21.15.45:0/45820727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febcc0d4370 0x7febcc15fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:26.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.364+0000 7febca800640 1 -- 172.21.15.45:0/45820727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febcc15ea70 msgr2=0x7febcc1f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:26.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.364+0000 7febca800640 1 --2- 172.21.15.45:0/45820727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febcc15ea70 0x7febcc1f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:26.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.364+0000 7febca800640 1 -- 172.21.15.45:0/45820727 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febcc1607b0 con 0x7febcc0cfc40 2024-04-16T16:20:26.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.364+0000 7febca800640 1 --2- 172.21.15.45:0/45820727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0cfc40 0x7febcc0ca910 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7febbc02bee0 tx=0x7febbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:26.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.364+0000 7febc2a00640 1 -- 172.21.15.45:0/45820727 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febbc041070 con 0x7febcc0cfc40 2024-04-16T16:20:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.364+0000 7febc2a00640 1 -- 172.21.15.45:0/45820727 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febbc03b920 con 0x7febcc0cfc40 2024-04-16T16:20:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.364+0000 7febd0a00640 1 -- 172.21.15.45:0/45820727 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febcc0c83a0 con 0x7febcc0cfc40 2024-04-16T16:20:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.364+0000 7febc2a00640 1 -- 172.21.15.45:0/45820727 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febbc035040 con 0x7febcc0cfc40 2024-04-16T16:20:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.365+0000 7febd0a00640 1 -- 172.21.15.45:0/45820727 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febcc0c8960 con 0x7febcc0cfc40 2024-04-16T16:20:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.366+0000 7febd0a00640 1 -- 172.21.15.45:0/45820727 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febcc075780 con 0x7febcc0cfc40 2024-04-16T16:20:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.367+0000 7febc2a00640 1 -- 172.21.15.45:0/45820727 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7febbc04c060 con 0x7febcc0cfc40 2024-04-16T16:20:26.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.367+0000 7febc2a00640 1 --2- 172.21.15.45:0/45820727 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7febac06ca40 0x7febac06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:26.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.368+0000 7febc2a00640 1 -- 172.21.15.45:0/45820727 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 5631+0+0 (secure 0 0 0) 0x7febbc0ad520 con 0x7febcc0cfc40 2024-04-16T16:20:26.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.368+0000 7febc9e00640 1 --2- 172.21.15.45:0/45820727 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7febac06ca40 0x7febac06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:26.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.369+0000 7febc9e00640 1 --2- 172.21.15.45:0/45820727 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7febac06ca40 0x7febac06ef00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7febb8000f60 tx=0x7febb80079d0 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:26.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.376+0000 7febc2a00640 1 -- 172.21.15.45:0/45820727 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7febbc07b230 con 0x7febcc0cfc40 2024-04-16T16:20:26.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:26.671+0000 7febd0a00640 1 -- 172.21.15.45:0/45820727 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_metrics_from_rank"]} v 0) -- 0x7febcc0d2500 con 0x7febcc0cfc40 2024-04-16T16:20:27.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.239+0000 7febc2a00640 1 -- 172.21.15.45:0/45820727 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_metrics_from_rank"]}]=0 v406) ==== 149+0+0 (secure 0 0 0) 0x7febbc0abe20 con 0x7febcc0cfc40 2024-04-16T16:20:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.244+0000 7febd0a00640 1 -- 172.21.15.45:0/45820727 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7febac06ca40 msgr2=0x7febac06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.244+0000 7febd0a00640 1 --2- 172.21.15.45:0/45820727 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7febac06ca40 0x7febac06ef00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7febb8000f60 tx=0x7febb80079d0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.244+0000 7febd0a00640 1 -- 172.21.15.45:0/45820727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0cfc40 msgr2=0x7febcc0ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.244+0000 7febd0a00640 1 --2- 172.21.15.45:0/45820727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0cfc40 0x7febcc0ca910 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7febbc02bee0 tx=0x7febbc0046c0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.245+0000 7febd0a00640 1 -- 172.21.15.45:0/45820727 shutdown_connections 2024-04-16T16:20:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.245+0000 7febd0a00640 1 --2- 172.21.15.45:0/45820727 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7febac06ca40 0x7febac06ef00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.245+0000 7febd0a00640 1 --2- 172.21.15.45:0/45820727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febcc15ea70 0x7febcc1f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.245+0000 7febd0a00640 1 --2- 172.21.15.45:0/45820727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febcc0d4370 0x7febcc15fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.245+0000 7febd0a00640 1 --2- 172.21.15.45:0/45820727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0cfc40 0x7febcc0ca910 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.245+0000 7febd0a00640 1 -- 172.21.15.45:0/45820727 >> 172.21.15.45:0/45820727 conn(0x7febcc06d3b0 msgr2=0x7febcc0b8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.245+0000 7febd0a00640 1 -- 172.21.15.45:0/45820727 shutdown_connections 2024-04-16T16:20:27.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.245+0000 7febd0a00640 1 -- 172.21.15.45:0/45820727 wait complete. 2024-04-16T16:20:27.268 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-16T16:20:27.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.493+0000 7fe692e00640 1 Processor -- start 2024-04-16T16:20:27.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.493+0000 7fe692e00640 1 -- start start 2024-04-16T16:20:27.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.495+0000 7fe692e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe68c166840 0x7fe68c163cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:27.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.495+0000 7fe692e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe68c0b58f0 0x7fe68c0b5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:27.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.495+0000 7fe692e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe68c1495a0 con 0x7fe68c1491c0 2024-04-16T16:20:27.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.495+0000 7fe692e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe68c0cd810 con 0x7fe68c166840 2024-04-16T16:20:27.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.495+0000 7fe692e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe68c0b6280 con 0x7fe68c0b58f0 2024-04-16T16:20:27.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.495+0000 7fe691a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe68c1491c0 0x7fe68c0cd100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57214/0 (socket says 172.21.15.45:57214) 2024-04-16T16:20:27.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.495+0000 7fe691a00640 1 -- 172.21.15.45:0/2388739536 learned_addr learned my addr 172.21.15.45:0/2388739536 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:27.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.496+0000 7fe691000640 1 --2- 172.21.15.45:0/2388739536 >> v2:172.21.15.45:3300/0 conn(0x7fe68c166840 0x7fe68c163cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.496+0000 7fe692400640 1 --2- 172.21.15.45:0/2388739536 >> v2:172.21.15.103:3300/0 conn(0x7fe68c0b58f0 0x7fe68c0b5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.496+0000 7fe68be00640 1 -- 172.21.15.45:0/2388739536 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1774212889 0 0) 0x7fe68c1495a0 con 0x7fe68c1491c0 2024-04-16T16:20:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.496+0000 7fe68be00640 1 -- 172.21.15.45:0/2388739536 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe674003520 con 0x7fe68c1491c0 2024-04-16T16:20:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.496+0000 7fe691000640 1 -- 172.21.15.45:0/2388739536 >> v2:172.21.15.103:3300/0 conn(0x7fe68c0b58f0 msgr2=0x7fe68c0b5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.496+0000 7fe691000640 1 --2- 172.21.15.45:0/2388739536 >> v2:172.21.15.103:3300/0 conn(0x7fe68c0b58f0 0x7fe68c0b5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.496+0000 7fe691000640 1 -- 172.21.15.45:0/2388739536 >> v1:172.21.15.45:6789/0 conn(0x7fe68c1491c0 legacy=0x7fe68c0cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.496+0000 7fe691000640 1 -- 172.21.15.45:0/2388739536 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68c166e40 con 0x7fe68c166840 2024-04-16T16:20:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.497+0000 7fe691000640 1 --2- 172.21.15.45:0/2388739536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c166840 0x7fe68c163cd0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fe68c071d90 tx=0x7fe67802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caae4084c165d430 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:27.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.497+0000 7fe68be00640 1 -- 172.21.15.45:0/2388739536 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6780315a0 con 0x7fe68c166840 2024-04-16T16:20:27.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.498+0000 7fe68be00640 1 -- 172.21.15.45:0/2388739536 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe678037070 con 0x7fe68c166840 2024-04-16T16:20:27.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.498+0000 7fe68be00640 1 -- 172.21.15.45:0/2388739536 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe678031960 con 0x7fe68c166840 2024-04-16T16:20:27.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.498+0000 7fe692e00640 1 -- 172.21.15.45:0/2388739536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c166840 msgr2=0x7fe68c163cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:27.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.498+0000 7fe692e00640 1 --2- 172.21.15.45:0/2388739536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c166840 0x7fe68c163cd0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fe68c071d90 tx=0x7fe67802b840 comp rx=0 tx=0).stop 2024-04-16T16:20:27.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.498+0000 7fe692e00640 1 -- 172.21.15.45:0/2388739536 shutdown_connections 2024-04-16T16:20:27.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.498+0000 7fe692e00640 1 --2- 172.21.15.45:0/2388739536 >> v2:172.21.15.103:3300/0 conn(0x7fe68c0b58f0 0x7fe68c0b5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.498+0000 7fe692e00640 1 --2- 172.21.15.45:0/2388739536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c166840 0x7fe68c163cd0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.498+0000 7fe692e00640 1 -- 172.21.15.45:0/2388739536 >> 172.21.15.45:0/2388739536 conn(0x7fe68c06b930 msgr2=0x7fe68c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:27.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.498+0000 7fe692e00640 1 -- 172.21.15.45:0/2388739536 shutdown_connections 2024-04-16T16:20:27.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.498+0000 7fe692e00640 1 -- 172.21.15.45:0/2388739536 wait complete. 2024-04-16T16:20:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.500+0000 7fe692e00640 1 Processor -- start 2024-04-16T16:20:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.500+0000 7fe692e00640 1 -- start start 2024-04-16T16:20:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.501+0000 7fe692e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe68c0b58f0 0x7fe68c15e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.501+0000 7fe692e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c1491c0 0x7fe68c1394e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.501+0000 7fe692e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe68c166840 0x7fe68c0b2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.501+0000 7fe692e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c167400 con 0x7fe68c1491c0 2024-04-16T16:20:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.501+0000 7fe692e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c06e090 con 0x7fe68c166840 2024-04-16T16:20:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.501+0000 7fe692e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe68c1676e0 con 0x7fe68c0b58f0 2024-04-16T16:20:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.501+0000 7fe691000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c1491c0 0x7fe68c1394e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.502+0000 7fe691000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c1491c0 0x7fe68c1394e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49878/0 (socket says 172.21.15.45:49878) 2024-04-16T16:20:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.502+0000 7fe691000640 1 -- 172.21.15.45:0/71308529 learned_addr learned my addr 172.21.15.45:0/71308529 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.502+0000 7fe691a00640 1 --2- 172.21.15.45:0/71308529 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe68c0b58f0 0x7fe68c15e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.502+0000 7fe692400640 1 --2- 172.21.15.45:0/71308529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe68c166840 0x7fe68c0b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.502+0000 7fe691000640 1 -- 172.21.15.45:0/71308529 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe68c0b58f0 msgr2=0x7fe68c15e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.502+0000 7fe691000640 1 --2- 172.21.15.45:0/71308529 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe68c0b58f0 0x7fe68c15e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.502+0000 7fe691000640 1 -- 172.21.15.45:0/71308529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe68c166840 msgr2=0x7fe68c0b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.502+0000 7fe691000640 1 --2- 172.21.15.45:0/71308529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe68c166840 0x7fe68c0b2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.502+0000 7fe691000640 1 -- 172.21.15.45:0/71308529 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68c15eba0 con 0x7fe68c1491c0 2024-04-16T16:20:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.502+0000 7fe691000640 1 --2- 172.21.15.45:0/71308529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c1491c0 0x7fe68c1394e0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fe678000c00 tx=0x7fe6780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.503+0000 7fe689600640 1 -- 172.21.15.45:0/71308529 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe678032630 con 0x7fe68c1491c0 2024-04-16T16:20:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.503+0000 7fe689600640 1 -- 172.21.15.45:0/71308529 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6780327f0 con 0x7fe68c1491c0 2024-04-16T16:20:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.503+0000 7fe689600640 1 -- 172.21.15.45:0/71308529 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe678032be0 con 0x7fe68c1491c0 2024-04-16T16:20:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.503+0000 7fe692e00640 1 -- 172.21.15.45:0/71308529 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe68c15ee80 con 0x7fe68c1491c0 2024-04-16T16:20:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.503+0000 7fe692e00640 1 -- 172.21.15.45:0/71308529 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe68c15f440 con 0x7fe68c1491c0 2024-04-16T16:20:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.504+0000 7fe692e00640 1 -- 172.21.15.45:0/71308529 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe68c075780 con 0x7fe68c1491c0 2024-04-16T16:20:27.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.505+0000 7fe689600640 1 -- 172.21.15.45:0/71308529 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7fe67803b3f0 con 0x7fe68c1491c0 2024-04-16T16:20:27.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.506+0000 7fe689600640 1 --2- 172.21.15.45:0/71308529 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe66c06ca10 0x7fe66c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:27.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.506+0000 7fe689600640 1 -- 172.21.15.45:0/71308529 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 5631+0+0 (secure 0 0 0) 0x7fe67802f030 con 0x7fe68c1491c0 2024-04-16T16:20:27.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.507+0000 7fe691a00640 1 --2- 172.21.15.45:0/71308529 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe66c06ca10 0x7fe66c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:27.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.508+0000 7fe691a00640 1 --2- 172.21.15.45:0/71308529 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe66c06ca10 0x7fe66c06eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe680001560 tx=0x7fe680009040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:27.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.511+0000 7fe689600640 1 -- 172.21.15.45:0/71308529 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe678040070 con 0x7fe68c1491c0 2024-04-16T16:20:27.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.715+0000 7fe692e00640 1 -- 172.21.15.45:0/71308529 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fe68c0b20d0 con 0x7fe68c1491c0 2024-04-16T16:20:27.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.722+0000 7fe689600640 1 -- 172.21.15.45:0/71308529 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) ==== 122+0+0 (secure 0 0 0) 0x7fe67807c070 con 0x7fe68c1491c0 2024-04-16T16:20:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.724+0000 7fe692e00640 1 -- 172.21.15.45:0/71308529 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe66c06ca10 msgr2=0x7fe66c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.724+0000 7fe692e00640 1 --2- 172.21.15.45:0/71308529 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe66c06ca10 0x7fe66c06eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe680001560 tx=0x7fe680009040 comp rx=0 tx=0).stop 2024-04-16T16:20:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.724+0000 7fe692e00640 1 -- 172.21.15.45:0/71308529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c1491c0 msgr2=0x7fe68c1394e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.724+0000 7fe692e00640 1 --2- 172.21.15.45:0/71308529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c1491c0 0x7fe68c1394e0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fe678000c00 tx=0x7fe6780046c0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.725+0000 7fe692e00640 1 -- 172.21.15.45:0/71308529 shutdown_connections 2024-04-16T16:20:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.725+0000 7fe692e00640 1 --2- 172.21.15.45:0/71308529 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe66c06ca10 0x7fe66c06eed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.725+0000 7fe692e00640 1 --2- 172.21.15.45:0/71308529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe68c166840 0x7fe68c0b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.725+0000 7fe692e00640 1 --2- 172.21.15.45:0/71308529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c1491c0 0x7fe68c1394e0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.725+0000 7fe692e00640 1 --2- 172.21.15.45:0/71308529 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe68c0b58f0 0x7fe68c15e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.725+0000 7fe692e00640 1 -- 172.21.15.45:0/71308529 >> 172.21.15.45:0/71308529 conn(0x7fe68c06b930 msgr2=0x7fe68c0ce3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.725+0000 7fe692e00640 1 -- 172.21.15.45:0/71308529 shutdown_connections 2024-04-16T16:20:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:27.725+0000 7fe692e00640 1 -- 172.21.15.45:0/71308529 wait complete. 2024-04-16T16:20:27.743 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-16T16:20:27.743 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:20:27.855 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:20:27.988 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-16T16:20:27.988 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:20:27.988 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:20:28.049 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:20:28.049 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:20:28.049 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:20:28.109 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:20:28.109 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:20:28.109 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:20:28.169 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:20:28.169 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:20:28.169 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:20:28.227 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:20:28.227 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-16T16:20:28.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:20:28.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.452+0000 7fb72f200640 1 Processor -- start 2024-04-16T16:20:28.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.453+0000 7fb72f200640 1 -- start start 2024-04-16T16:20:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.454+0000 7fb72f200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb730163010 0x7fb7300cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.454+0000 7fb72f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb73006f540 0x7fb7301645b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.454+0000 7fb72f200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb730163510 con 0x7fb7300cde00 2024-04-16T16:20:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.454+0000 7fb72f200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb7301637f0 con 0x7fb730163010 2024-04-16T16:20:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.454+0000 7fb72f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb730163ad0 con 0x7fb73006f540 2024-04-16T16:20:28.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.455+0000 7fb725400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb730163010 0x7fb7300cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:28.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.455+0000 7fb72de00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb7300cde00 0x7fb7300ce1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57218/0 (socket says 172.21.15.45:57218) 2024-04-16T16:20:28.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.455+0000 7fb72de00640 1 -- 172.21.15.45:0/1251581049 learned_addr learned my addr 172.21.15.45:0/1251581049 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:28.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.455+0000 7fb72e800640 1 --2- 172.21.15.45:0/1251581049 >> v2:172.21.15.103:3300/0 conn(0x7fb73006f540 0x7fb7301645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:28.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.455+0000 7fb725400640 1 -- 172.21.15.45:0/1251581049 >> v2:172.21.15.103:3300/0 conn(0x7fb73006f540 msgr2=0x7fb7301645b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:28.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.455+0000 7fb725400640 1 --2- 172.21.15.45:0/1251581049 >> v2:172.21.15.103:3300/0 conn(0x7fb73006f540 0x7fb7301645b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:28.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.455+0000 7fb725400640 1 -- 172.21.15.45:0/1251581049 >> v1:172.21.15.45:6789/0 conn(0x7fb7300cde00 legacy=0x7fb7300ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:28.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.455+0000 7fb72d400640 1 -- 172.21.15.45:0/1251581049 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 451699796 0 0) 0x7fb730163510 con 0x7fb7300cde00 2024-04-16T16:20:28.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.455+0000 7fb725400640 1 -- 172.21.15.45:0/1251581049 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb730163db0 con 0x7fb730163010 2024-04-16T16:20:28.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.456+0000 7fb725400640 1 --2- 172.21.15.45:0/1251581049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb730163010 0x7fb7300cff30 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fb7300733e0 tx=0x7fb71802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711af95e7f692d14 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:28.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.457+0000 7fb72d400640 1 -- 172.21.15.45:0/1251581049 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb718035020 con 0x7fb730163010 2024-04-16T16:20:28.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.457+0000 7fb72d400640 1 -- 172.21.15.45:0/1251581049 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb718031830 con 0x7fb730163010 2024-04-16T16:20:28.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.457+0000 7fb72d400640 1 -- 172.21.15.45:0/1251581049 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb718031c00 con 0x7fb730163010 2024-04-16T16:20:28.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.457+0000 7fb72f200640 1 -- 172.21.15.45:0/1251581049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb730163010 msgr2=0x7fb7300cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:28.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.457+0000 7fb72f200640 1 --2- 172.21.15.45:0/1251581049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb730163010 0x7fb7300cff30 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fb7300733e0 tx=0x7fb71802b8d0 comp rx=0 tx=0).stop 2024-04-16T16:20:28.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.457+0000 7fb72f200640 1 -- 172.21.15.45:0/1251581049 shutdown_connections 2024-04-16T16:20:28.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.458+0000 7fb72f200640 1 --2- 172.21.15.45:0/1251581049 >> v2:172.21.15.103:3300/0 conn(0x7fb73006f540 0x7fb7301645b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:28.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.458+0000 7fb72f200640 1 --2- 172.21.15.45:0/1251581049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb730163010 0x7fb7300cff30 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:28.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.458+0000 7fb72f200640 1 -- 172.21.15.45:0/1251581049 >> 172.21.15.45:0/1251581049 conn(0x7fb73006b930 msgr2=0x7fb73006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:28.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.458+0000 7fb72f200640 1 -- 172.21.15.45:0/1251581049 shutdown_connections 2024-04-16T16:20:28.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.458+0000 7fb72f200640 1 -- 172.21.15.45:0/1251581049 wait complete. 2024-04-16T16:20:28.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.459+0000 7fb72f200640 1 Processor -- start 2024-04-16T16:20:28.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.459+0000 7fb72f200640 1 -- start start 2024-04-16T16:20:28.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.460+0000 7fb72f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb73006f540 0x7fb730158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:28.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.460+0000 7fb72f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7300cde00 0x7fb730152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:28.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.460+0000 7fb72f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb730163010 0x7fb73015ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:28.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.460+0000 7fb72f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb730164af0 con 0x7fb7300cde00 2024-04-16T16:20:28.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.460+0000 7fb72f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb73006f970 con 0x7fb73006f540 2024-04-16T16:20:28.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.460+0000 7fb72f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb730164c70 con 0x7fb730163010 2024-04-16T16:20:28.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.461+0000 7fb725400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7300cde00 0x7fb730152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:28.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.461+0000 7fb725400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7300cde00 0x7fb730152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49894/0 (socket says 172.21.15.45:49894) 2024-04-16T16:20:28.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.461+0000 7fb725400640 1 -- 172.21.15.45:0/438877288 learned_addr learned my addr 172.21.15.45:0/438877288 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:28.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.461+0000 7fb725400640 1 -- 172.21.15.45:0/438877288 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb730163010 msgr2=0x7fb73015ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:28.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.461+0000 7fb725400640 1 --2- 172.21.15.45:0/438877288 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb730163010 0x7fb73015ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:28.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.461+0000 7fb725400640 1 -- 172.21.15.45:0/438877288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb73006f540 msgr2=0x7fb730158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:28.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.461+0000 7fb725400640 1 --2- 172.21.15.45:0/438877288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb73006f540 0x7fb730158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:28.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.461+0000 7fb725400640 1 -- 172.21.15.45:0/438877288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb73015b360 con 0x7fb7300cde00 2024-04-16T16:20:28.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.461+0000 7fb725400640 1 --2- 172.21.15.45:0/438877288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7300cde00 0x7fb730152450 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fb718000c00 tx=0x7fb7180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:28.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.462+0000 7fb726a00640 1 -- 172.21.15.45:0/438877288 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb718040070 con 0x7fb7300cde00 2024-04-16T16:20:28.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.462+0000 7fb726a00640 1 -- 172.21.15.45:0/438877288 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb718039a20 con 0x7fb7300cde00 2024-04-16T16:20:28.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.462+0000 7fb726a00640 1 -- 172.21.15.45:0/438877288 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb718035040 con 0x7fb7300cde00 2024-04-16T16:20:28.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.462+0000 7fb72f200640 1 -- 172.21.15.45:0/438877288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb73015b640 con 0x7fb7300cde00 2024-04-16T16:20:28.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.462+0000 7fb72f200640 1 -- 172.21.15.45:0/438877288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb73015bc00 con 0x7fb7300cde00 2024-04-16T16:20:28.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.463+0000 7fb72f200640 1 -- 172.21.15.45:0/438877288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb730075780 con 0x7fb7300cde00 2024-04-16T16:20:28.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.464+0000 7fb726a00640 1 -- 172.21.15.45:0/438877288 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7fb71804b060 con 0x7fb7300cde00 2024-04-16T16:20:28.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.466+0000 7fb726a00640 1 --2- 172.21.15.45:0/438877288 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fb70c06ca10 0x7fb70c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:28.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.466+0000 7fb726a00640 1 -- 172.21.15.45:0/438877288 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 5631+0+0 (secure 0 0 0) 0x7fb7180ad630 con 0x7fb7300cde00 2024-04-16T16:20:28.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.466+0000 7fb72de00640 1 --2- 172.21.15.45:0/438877288 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fb70c06ca10 0x7fb70c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:28.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.468+0000 7fb72de00640 1 --2- 172.21.15.45:0/438877288 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fb70c06ca10 0x7fb70c06eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb720007fd0 tx=0x7fb720007920 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:28.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.471+0000 7fb726a00640 1 -- 172.21.15.45:0/438877288 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb71807b900 con 0x7fb7300cde00 2024-04-16T16:20:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.716+0000 7fb72f200640 1 -- 172.21.15.45:0/438877288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb73015a6b0 con 0x7fb7300cde00 2024-04-16T16:20:28.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.717+0000 7fb726a00640 1 -- 172.21.15.45:0/438877288 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+3872 (secure 0 0 0) 0x7fb71807b200 con 0x7fb7300cde00 2024-04-16T16:20:28.716 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:28.716 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":84,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6056,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1032791151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":1032791151},{"type":"v1","addr":"172.21.15.45:6835","nonce":1032791151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6062,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/498835619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":498835619},{"type":"v1","addr":"172.21.15.103:6837","nonce":498835619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/3611628422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3611628422},{"type":"v1","addr":"172.21.15.103:6836","nonce":3611628422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":8568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/1057850182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1057850182},{"type":"v1","addr":"172.21.15.45:6834","nonce":1057850182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[]} 2024-04-16T16:20:28.716 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 84 2024-04-16T16:20:28.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.719+0000 7fb72f200640 1 -- 172.21.15.45:0/438877288 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fb70c06ca10 msgr2=0x7fb70c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:28.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.719+0000 7fb72f200640 1 --2- 172.21.15.45:0/438877288 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fb70c06ca10 0x7fb70c06eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb720007fd0 tx=0x7fb720007920 comp rx=0 tx=0).stop 2024-04-16T16:20:28.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.719+0000 7fb72f200640 1 -- 172.21.15.45:0/438877288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7300cde00 msgr2=0x7fb730152450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:28.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.719+0000 7fb72f200640 1 --2- 172.21.15.45:0/438877288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7300cde00 0x7fb730152450 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fb718000c00 tx=0x7fb7180046c0 comp rx=0 tx=0).stop 2024-04-16T16:20:28.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.719+0000 7fb72f200640 1 -- 172.21.15.45:0/438877288 shutdown_connections 2024-04-16T16:20:28.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.719+0000 7fb72f200640 1 --2- 172.21.15.45:0/438877288 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fb70c06ca10 0x7fb70c06eed0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:28.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.719+0000 7fb72f200640 1 --2- 172.21.15.45:0/438877288 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb730163010 0x7fb73015ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:28.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.719+0000 7fb72f200640 1 --2- 172.21.15.45:0/438877288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7300cde00 0x7fb730152450 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:28.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.719+0000 7fb72f200640 1 --2- 172.21.15.45:0/438877288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb73006f540 0x7fb730158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:28.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.719+0000 7fb72f200640 1 -- 172.21.15.45:0/438877288 >> 172.21.15.45:0/438877288 conn(0x7fb73006b930 msgr2=0x7fb7300ce1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:28.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.720+0000 7fb72f200640 1 -- 172.21.15.45:0/438877288 shutdown_connections 2024-04-16T16:20:28.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:28.720+0000 7fb72f200640 1 -- 172.21.15.45:0/438877288 wait complete. 2024-04-16T16:20:28.740 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-16T16:20:28.740 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-16T16:20:28.741 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-16T16:20:28.741 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:20:28.741 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-16T16:20:28.741 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-16T16:20:28.742 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-16T16:20:28.742 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:20:28.742 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-16T16:20:28.742 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-16T16:20:28.743 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-16T16:20:28.743 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:20:28.743 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-16T16:20:28.743 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-16T16:20:28.744 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-16T16:20:28.744 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:20:28.759 INFO:tasks.ceph.mds.a:Stopped 2024-04-16T16:20:28.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-16T16:20:28.762 INFO:tasks.ceph.mds.c:Stopped 2024-04-16T16:20:28.762 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-16T16:20:28.765 INFO:tasks.ceph.mds.b:Stopped 2024-04-16T16:20:28.765 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-16T16:20:28.773 INFO:tasks.ceph.mds.d:Stopped 2024-04-16T16:20:28.774 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-16T16:20:28.807 INFO:tasks.ceph.mds.a:Started 2024-04-16T16:20:28.808 DEBUG:teuthology.parallel:result is None 2024-04-16T16:20:28.812 INFO:tasks.ceph.mds.c:Started 2024-04-16T16:20:28.812 DEBUG:teuthology.parallel:result is None 2024-04-16T16:20:28.821 INFO:tasks.ceph.mds.b:Started 2024-04-16T16:20:28.821 DEBUG:teuthology.parallel:result is None 2024-04-16T16:20:28.827 INFO:tasks.ceph.mds.d:Started 2024-04-16T16:20:28.827 DEBUG:teuthology.parallel:result is None 2024-04-16T16:20:28.827 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-16T16:20:28.966 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2024-04-16T16:20:28.968 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2024-04-16T16:20:28.990 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-16T16:20:28.992 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-16T16:20:29.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.021+0000 7fdaa9600640 1 Processor -- start 2024-04-16T16:20:29.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.021+0000 7fdaa9600640 1 -- start start 2024-04-16T16:20:29.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.022+0000 7fdaa9600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdaa4072890 0x7fdaa40cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:29.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.022+0000 7fdaa9600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdaa40cd220 0x7fdaa40cd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:29.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.022+0000 7fdaa9600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdaa4146600 con 0x7fdaa4146220 2024-04-16T16:20:29.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.022+0000 7fdaa9600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fdaa406f840 con 0x7fdaa4072890 2024-04-16T16:20:29.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.022+0000 7fdaa9600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa40ce0e0 con 0x7fdaa40cd220 2024-04-16T16:20:29.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.023+0000 7fdaa2a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdaa4146220 0x7fdaa4072180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57230/0 (socket says 172.21.15.45:57230) 2024-04-16T16:20:29.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.023+0000 7fdaa2a00640 1 -- 172.21.15.45:0/2393437438 learned_addr learned my addr 172.21.15.45:0/2393437438 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:29.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.023+0000 7fdaa2000640 1 --2- 172.21.15.45:0/2393437438 >> v2:172.21.15.103:3300/0 conn(0x7fdaa4072890 0x7fdaa40cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:29.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.023+0000 7fdaa3400640 1 --2- 172.21.15.45:0/2393437438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdaa40cd220 0x7fdaa40cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:29.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.023+0000 7fdaa1600640 1 -- 172.21.15.45:0/2393437438 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1396239498 0 0) 0x7fdaa4146600 con 0x7fdaa4146220 2024-04-16T16:20:29.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.023+0000 7fdaa1600640 1 -- 172.21.15.45:0/2393437438 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda88003520 con 0x7fdaa4146220 2024-04-16T16:20:29.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.023+0000 7fdaa2000640 1 -- 172.21.15.45:0/2393437438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdaa40cd220 msgr2=0x7fdaa40cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:29.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.024+0000 7fdaa2000640 1 --2- 172.21.15.45:0/2393437438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdaa40cd220 0x7fdaa40cd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.024+0000 7fdaa2000640 1 -- 172.21.15.45:0/2393437438 >> v1:172.21.15.45:6789/0 conn(0x7fdaa4146220 legacy=0x7fdaa4072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:29.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.024+0000 7fdaa2000640 1 -- 172.21.15.45:0/2393437438 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa40cde00 con 0x7fdaa4072890 2024-04-16T16:20:29.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.024+0000 7fdaa3400640 1 --2- 172.21.15.45:0/2393437438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdaa40cd220 0x7fdaa40cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:20:29.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.025+0000 7fdaa2000640 1 --2- 172.21.15.45:0/2393437438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4072890 0x7fdaa40cf110 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fda900097a0 tx=0x7fda9002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3112e5445ac908f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:29.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.025+0000 7fdaa1600640 1 -- 172.21.15.45:0/2393437438 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda90035020 con 0x7fdaa4072890 2024-04-16T16:20:29.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.025+0000 7fdaa1600640 1 -- 172.21.15.45:0/2393437438 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda90031810 con 0x7fdaa4072890 2024-04-16T16:20:29.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.025+0000 7fdaa1600640 1 -- 172.21.15.45:0/2393437438 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda90031be0 con 0x7fdaa4072890 2024-04-16T16:20:29.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.026+0000 7fdaa9600640 1 -- 172.21.15.45:0/2393437438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4072890 msgr2=0x7fdaa40cf110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:29.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.026+0000 7fdaa9600640 1 --2- 172.21.15.45:0/2393437438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4072890 0x7fdaa40cf110 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fda900097a0 tx=0x7fda9002b8b0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.026+0000 7fdaa9600640 1 -- 172.21.15.45:0/2393437438 shutdown_connections 2024-04-16T16:20:29.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.026+0000 7fdaa9600640 1 --2- 172.21.15.45:0/2393437438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdaa40cd220 0x7fdaa40cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.026+0000 7fdaa9600640 1 --2- 172.21.15.45:0/2393437438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4072890 0x7fdaa40cf110 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.026+0000 7fdaa9600640 1 -- 172.21.15.45:0/2393437438 >> 172.21.15.45:0/2393437438 conn(0x7fdaa406b930 msgr2=0x7fdaa406bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:29.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.026+0000 7fdaa9600640 1 -- 172.21.15.45:0/2393437438 shutdown_connections 2024-04-16T16:20:29.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.026+0000 7fdaa9600640 1 -- 172.21.15.45:0/2393437438 wait complete. 2024-04-16T16:20:29.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.028+0000 7fdaa9600640 1 Processor -- start 2024-04-16T16:20:29.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.028+0000 7fdaa9600640 1 -- start start 2024-04-16T16:20:29.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.029+0000 7fdaa9600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4072890 0x7fdaa4137a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:29.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.029+0000 7fdaa9600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdaa40cd220 0x7fdaa40aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:29.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.029+0000 7fdaa9600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaa4146220 0x7fdaa4158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:29.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.029+0000 7fdaa9600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa40ce910 con 0x7fdaa4146220 2024-04-16T16:20:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.029+0000 7fdaa9600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa40cdbb0 con 0x7fdaa4072890 2024-04-16T16:20:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.029+0000 7fdaa9600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa40cebf0 con 0x7fdaa40cd220 2024-04-16T16:20:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.029+0000 7fdaa3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaa4146220 0x7fdaa4158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.029+0000 7fdaa3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaa4146220 0x7fdaa4158c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49922/0 (socket says 172.21.15.45:49922) 2024-04-16T16:20:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.029+0000 7fdaa3400640 1 -- 172.21.15.45:0/3802359228 learned_addr learned my addr 172.21.15.45:0/3802359228 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.029+0000 7fdaa2000640 1 --2- 172.21.15.45:0/3802359228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdaa40cd220 0x7fdaa40aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.029+0000 7fdaa2a00640 1 --2- 172.21.15.45:0/3802359228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4072890 0x7fdaa4137a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:29.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.029+0000 7fdaa3400640 1 -- 172.21.15.45:0/3802359228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdaa40cd220 msgr2=0x7fdaa40aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:29.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.029+0000 7fdaa3400640 1 --2- 172.21.15.45:0/3802359228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdaa40cd220 0x7fdaa40aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.030+0000 7fdaa3400640 1 -- 172.21.15.45:0/3802359228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4072890 msgr2=0x7fdaa4137a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:29.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.030+0000 7fdaa3400640 1 --2- 172.21.15.45:0/3802359228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4072890 0x7fdaa4137a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.030+0000 7fdaa3400640 1 -- 172.21.15.45:0/3802359228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa4137fa0 con 0x7fdaa4146220 2024-04-16T16:20:29.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.030+0000 7fdaa2a00640 1 --2- 172.21.15.45:0/3802359228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4072890 0x7fdaa4137a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:20:29.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.030+0000 7fdaa3400640 1 --2- 172.21.15.45:0/3802359228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaa4146220 0x7fdaa4158c70 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fda9c009f00 tx=0x7fda9c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.030+0000 7fda9aa00640 1 -- 172.21.15.45:0/3802359228 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c018030 con 0x7fdaa4146220 2024-04-16T16:20:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.030+0000 7fda9aa00640 1 -- 172.21.15.45:0/3802359228 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda9c00ce50 con 0x7fdaa4146220 2024-04-16T16:20:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.030+0000 7fda9aa00640 1 -- 172.21.15.45:0/3802359228 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c003bb0 con 0x7fdaa4146220 2024-04-16T16:20:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.030+0000 7fdaa9600640 1 -- 172.21.15.45:0/3802359228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa4135360 con 0x7fdaa4146220 2024-04-16T16:20:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.030+0000 7fdaa9600640 1 -- 172.21.15.45:0/3802359228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa4135920 con 0x7fdaa4146220 2024-04-16T16:20:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.032+0000 7fdaa9600640 1 -- 172.21.15.45:0/3802359228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaa4075780 con 0x7fdaa4146220 2024-04-16T16:20:29.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.033+0000 7fda9aa00640 1 -- 172.21.15.45:0/3802359228 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7fda9c01b070 con 0x7fdaa4146220 2024-04-16T16:20:29.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.033+0000 7fda9aa00640 1 --2- 172.21.15.45:0/3802359228 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fda8406ca10 0x7fda8406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:29.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.034+0000 7fda9aa00640 1 -- 172.21.15.45:0/3802359228 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 5631+0+0 (secure 0 0 0) 0x7fda9c08c7b0 con 0x7fdaa4146220 2024-04-16T16:20:29.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.034+0000 7fdaa2a00640 1 --2- 172.21.15.45:0/3802359228 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fda8406ca10 0x7fda8406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:29.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.035+0000 7fdaa2a00640 1 --2- 172.21.15.45:0/3802359228 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fda8406ca10 0x7fda8406eed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fda94007fd0 tx=0x7fda94009040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:29.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.038+0000 7fda9aa00640 1 -- 172.21.15.45:0/3802359228 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdaa4137fa0 con 0x7fdaa4146220 2024-04-16T16:20:29.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.255+0000 7fdaa9600640 1 -- 172.21.15.45:0/3802359228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fdaa41609a0 con 0x7fdaa4146220 2024-04-16T16:20:29.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.274+0000 7fda9aa00640 1 -- 172.21.15.45:0/3802359228 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v122) ==== 97+0+0 (secure 0 0 0) 0x7fda9c010040 con 0x7fdaa4146220 2024-04-16T16:20:29.274 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2024-04-16T16:20:29.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.278+0000 7fdaa9600640 1 -- 172.21.15.45:0/3802359228 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fda8406ca10 msgr2=0x7fda8406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:29.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.278+0000 7fdaa9600640 1 --2- 172.21.15.45:0/3802359228 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fda8406ca10 0x7fda8406eed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fda94007fd0 tx=0x7fda94009040 comp rx=0 tx=0).stop 2024-04-16T16:20:29.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.278+0000 7fdaa9600640 1 -- 172.21.15.45:0/3802359228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaa4146220 msgr2=0x7fdaa4158c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:29.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.278+0000 7fdaa9600640 1 --2- 172.21.15.45:0/3802359228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaa4146220 0x7fdaa4158c70 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fda9c009f00 tx=0x7fda9c00c490 comp rx=0 tx=0).stop 2024-04-16T16:20:29.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.278+0000 7fdaa9600640 1 -- 172.21.15.45:0/3802359228 shutdown_connections 2024-04-16T16:20:29.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.278+0000 7fdaa9600640 1 --2- 172.21.15.45:0/3802359228 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fda8406ca10 0x7fda8406eed0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.278+0000 7fdaa9600640 1 --2- 172.21.15.45:0/3802359228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaa4146220 0x7fdaa4158c70 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.278+0000 7fdaa9600640 1 --2- 172.21.15.45:0/3802359228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdaa40cd220 0x7fdaa40aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.278+0000 7fdaa9600640 1 --2- 172.21.15.45:0/3802359228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4072890 0x7fdaa4137a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.278+0000 7fdaa9600640 1 -- 172.21.15.45:0/3802359228 >> 172.21.15.45:0/3802359228 conn(0x7fdaa406b930 msgr2=0x7fdaa406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:29.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.278+0000 7fdaa9600640 1 -- 172.21.15.45:0/3802359228 shutdown_connections 2024-04-16T16:20:29.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.278+0000 7fdaa9600640 1 -- 172.21.15.45:0/3802359228 wait complete. 2024-04-16T16:20:29.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-16T16:20:29.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.520+0000 7f9e41600640 1 Processor -- start 2024-04-16T16:20:29.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.520+0000 7f9e41600640 1 -- start start 2024-04-16T16:20:29.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.522+0000 7f9e41600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e3c15e8f0 0x7f9e3c0cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:29.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.522+0000 7f9e41600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9e3c0ce9a0 0x7f9e3c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:29.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.522+0000 7f9e41600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e3c15ecd0 con 0x7f9e3c0cf080 2024-04-16T16:20:29.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.522+0000 7f9e41600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9e3c0ce2e0 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.522+0000 7f9e41600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9e3c0cd6d0 con 0x7f9e3c15e8f0 2024-04-16T16:20:29.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.522+0000 7f9e3b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9e3c0ce9a0 0x7f9e3c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:29.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.522+0000 7f9e3b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9e3c0ce9a0 0x7f9e3c0cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49928/0 (socket says 172.21.15.45:49928) 2024-04-16T16:20:29.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.522+0000 7f9e3b400640 1 -- 172.21.15.45:0/3085023857 learned_addr learned my addr 172.21.15.45:0/3085023857 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:29.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.523+0000 7f9e3be00640 1 --2- 172.21.15.45:0/3085023857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e3c15e8f0 0x7f9e3c0cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:29.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.523+0000 7f9e3b400640 1 -- 172.21.15.45:0/3085023857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e3c15e8f0 msgr2=0x7f9e3c0cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:29.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.523+0000 7f9e3b400640 1 --2- 172.21.15.45:0/3085023857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e3c15e8f0 0x7f9e3c0cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.523+0000 7f9e3b400640 1 -- 172.21.15.45:0/3085023857 >> v1:172.21.15.103:6789/0 conn(0x7f9e3c0cf080 legacy=0x7f9e3c162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:29.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.523+0000 7f9e3b400640 1 -- 172.21.15.45:0/3085023857 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e3c0cd9b0 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.524+0000 7f9e3b400640 1 --2- 172.21.15.45:0/3085023857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e3c0ce9a0 0x7f9e3c0cc950 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009870 tx=0x7f9e2802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd95986eda66588 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:29.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.524+0000 7f9e3aa00640 1 -- 172.21.15.45:0/3085023857 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e28035020 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.524+0000 7f9e3aa00640 1 -- 172.21.15.45:0/3085023857 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e28031640 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.524+0000 7f9e3aa00640 1 -- 172.21.15.45:0/3085023857 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2802d040 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.525+0000 7f9e41600640 1 -- 172.21.15.45:0/3085023857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e3c0ce9a0 msgr2=0x7f9e3c0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:29.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.525+0000 7f9e41600640 1 --2- 172.21.15.45:0/3085023857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e3c0ce9a0 0x7f9e3c0cc950 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009870 tx=0x7f9e2802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.525+0000 7f9e41600640 1 -- 172.21.15.45:0/3085023857 shutdown_connections 2024-04-16T16:20:29.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.525+0000 7f9e41600640 1 --2- 172.21.15.45:0/3085023857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e3c0ce9a0 0x7f9e3c0cc950 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.525+0000 7f9e41600640 1 --2- 172.21.15.45:0/3085023857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e3c15e8f0 0x7f9e3c0cc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.525+0000 7f9e41600640 1 -- 172.21.15.45:0/3085023857 >> 172.21.15.45:0/3085023857 conn(0x7f9e3c06b930 msgr2=0x7f9e3c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:29.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.525+0000 7f9e41600640 1 -- 172.21.15.45:0/3085023857 shutdown_connections 2024-04-16T16:20:29.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.525+0000 7f9e41600640 1 -- 172.21.15.45:0/3085023857 wait complete. 2024-04-16T16:20:29.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.527+0000 7f9e41600640 1 Processor -- start 2024-04-16T16:20:29.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.527+0000 7f9e41600640 1 -- start start 2024-04-16T16:20:29.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.528+0000 7f9e41600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e3c0ce9a0 0x7f9e3c142c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:29.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.528+0000 7f9e41600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e3c0cf080 0x7f9e3c13ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:29.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.528+0000 7f9e41600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e3c15e8f0 0x7f9e3c1f3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:29.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.528+0000 7f9e41600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3c0cd360 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.528+0000 7f9e41600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3c0cdf70 con 0x7f9e3c15e8f0 2024-04-16T16:20:29.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.528+0000 7f9e41600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9e3c0cbc40 con 0x7f9e3c0cf080 2024-04-16T16:20:29.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.528+0000 7f9e3be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e3c0ce9a0 0x7f9e3c142c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:29.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.528+0000 7f9e3be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e3c0ce9a0 0x7f9e3c142c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49944/0 (socket says 172.21.15.45:49944) 2024-04-16T16:20:29.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.528+0000 7f9e3be00640 1 -- 172.21.15.45:0/1243372660 learned_addr learned my addr 172.21.15.45:0/1243372660 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:29.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.529+0000 7f9e3b400640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e3c0cf080 0x7f9e3c13ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:29.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.529+0000 7f9e40c00640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e3c15e8f0 0x7f9e3c1f3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:29.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.529+0000 7f9e3be00640 1 -- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e3c0cf080 msgr2=0x7f9e3c13ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:29.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.529+0000 7f9e3be00640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e3c0cf080 0x7f9e3c13ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.529+0000 7f9e3be00640 1 -- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e3c15e8f0 msgr2=0x7f9e3c1f3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:29.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.529+0000 7f9e3be00640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e3c15e8f0 0x7f9e3c1f3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.529+0000 7f9e3be00640 1 -- 172.21.15.45:0/1243372660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e3c13cd20 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.529+0000 7f9e3b400640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e3c0cf080 0x7f9e3c13ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:29.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.529+0000 7f9e40c00640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e3c15e8f0 0x7f9e3c1f3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:29.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.529+0000 7f9e3be00640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e3c0ce9a0 0x7f9e3c142c20 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c00d5f0 tx=0x7f9e2c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:29.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.529+0000 7f9e33e00640 1 -- 172.21.15.45:0/1243372660 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c0125f0 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.529+0000 7f9e33e00640 1 -- 172.21.15.45:0/1243372660 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e2c019070 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.529+0000 7f9e33e00640 1 -- 172.21.15.45:0/1243372660 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c0129d0 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.530+0000 7f9e41600640 1 -- 172.21.15.45:0/1243372660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e3c1e87f0 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.530+0000 7f9e41600640 1 -- 172.21.15.45:0/1243372660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e3c13cfc0 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.531+0000 7f9e41600640 1 -- 172.21.15.45:0/1243372660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e3c075780 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.532+0000 7f9e33e00640 1 -- 172.21.15.45:0/1243372660 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f9e2c012b70 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.533+0000 7f9e33e00640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9e1c06ca10 0x7f9e1c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:29.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.533+0000 7f9e33e00640 1 -- 172.21.15.45:0/1243372660 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 4857+0+0 (secure 0 0 0) 0x7f9e2c08be70 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.533+0000 7f9e3b400640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9e1c06ca10 0x7f9e1c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:29.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.535+0000 7f9e3b400640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9e1c06ca10 0x7f9e1c06eed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9e28036000 tx=0x7f9e2803c040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:29.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.537+0000 7f9e33e00640 1 -- 172.21.15.45:0/1243372660 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9e2c05a440 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.812+0000 7f9e41600640 1 -- 172.21.15.45:0/1243372660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f9e3c0b5b90 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.813+0000 7f9e33e00640 1 -- 172.21.15.45:0/1243372660 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v23) ==== 80+0+5238 (secure 0 0 0) 0x7f9e2c059d40 con 0x7f9e3c0ce9a0 2024-04-16T16:20:29.812 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:29.812 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:20:29.812 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2024-04-16T16:20:29.813 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.813 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2024-04-16T16:20:29.813 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mpjReORAAJMoNf4qvfuxPSiV9+VlKfg==", 2024-04-16T16:20:29.813 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.813 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:20:29.813 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:20:29.814 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:20:29.814 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:20:29.814 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.814 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2024-04-16T16:20:29.815 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mxHeBIBAA3G8maaK65zaOaqXpAOA6xA==", 2024-04-16T16:20:29.815 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.815 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:20:29.815 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:20:29.815 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:20:29.815 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:20:29.816 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.816 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2024-04-16T16:20:29.816 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5m+AaODxAA08XSjc44/i2KYExYQSukaQ==", 2024-04-16T16:20:29.816 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.817 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:20:29.817 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:20:29.817 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:20:29.817 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:20:29.817 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.818 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2024-04-16T16:20:29.818 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mEQ4UMRAAOslSsxQ17bOxPrI6QMlRRw==", 2024-04-16T16:20:29.818 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.818 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:20:29.818 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:20:29.818 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:20:29.819 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:20:29.819 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.819 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2024-04-16T16:20:29.819 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBuox5m3l3OHxAA+UjwZx8Mn6tCu95TTYIG3w==", 2024-04-16T16:20:29.819 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.820 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:20:29.820 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:20:29.820 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:20:29.820 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.821 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2024-04-16T16:20:29.821 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxox5mozt1LhAAJL5NtMQdPVBtKOaVlMQBtw==", 2024-04-16T16:20:29.821 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.821 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:20:29.821 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:20:29.821 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:20:29.822 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.822 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2024-04-16T16:20:29.822 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB1ox5mfE7kABAAIvgZ5Wx+eIJ6QNKj2GVdpw==", 2024-04-16T16:20:29.822 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.823 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:20:29.823 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:20:29.823 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:20:29.823 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.823 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2024-04-16T16:20:29.824 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB4ox5mEvxcDRAAO0yXBKvsC7hZ6bpOAJyuag==", 2024-04-16T16:20:29.824 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.824 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:20:29.824 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:20:29.824 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:20:29.824 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.825 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2024-04-16T16:20:29.825 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+ox5mZubrORAAcGJLyRN5Ob9aBnopumnEUg==", 2024-04-16T16:20:29.825 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.825 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:20:29.825 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:20:29.826 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:20:29.826 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.826 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2024-04-16T16:20:29.826 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCCox5mcRqsChAACktmuUHvqWkp/mHT3UcjRw==", 2024-04-16T16:20:29.827 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.827 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:20:29.827 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:20:29.827 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:20:29.827 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.828 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2024-04-16T16:20:29.828 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFox5mT52FFhAAChhDDfNZmIMgxE1j90LZpQ==", 2024-04-16T16:20:29.828 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.828 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:20:29.828 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:20:29.828 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:20:29.828 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.829 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2024-04-16T16:20:29.829 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCIox5mC+EDIxAAxtvM0on00lxTLaoRjN0Gfw==", 2024-04-16T16:20:29.829 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.829 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:20:29.830 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:20:29.830 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:20:29.830 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.830 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2024-04-16T16:20:29.831 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ==", 2024-04-16T16:20:29.831 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.831 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:20:29.831 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:20:29.831 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-16T16:20:29.831 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.832 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2024-04-16T16:20:29.832 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mYpVUGhAAYdR+40O6zbWqgbI55F0FYA==", 2024-04-16T16:20:29.832 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.832 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:20:29.832 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:20:29.833 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:20:29.833 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.833 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2024-04-16T16:20:29.833 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mWdlUDxAAuO2FsGH1/E+qP1xcyUEe3A==", 2024-04-16T16:20:29.834 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.834 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:20:29.834 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:20:29.834 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:20:29.834 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.834 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.835 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2024-04-16T16:20:29.835 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5miPdMJBAAqjKpctNYu9ot1lff3rouxg==", 2024-04-16T16:20:29.835 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.835 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:20:29.835 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:20:29.835 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:20:29.836 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.836 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2024-04-16T16:20:29.836 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBpox5mO4PMLBAA/0YbQCfHujgdEzpXs0nOLw==", 2024-04-16T16:20:29.836 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.836 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:20:29.837 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:20:29.837 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:20:29.837 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:20:29.837 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.838 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2024-04-16T16:20:29.838 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mGevaLBAAPpA6g+Fdi/XHqAqG0QZNag==", 2024-04-16T16:20:29.838 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.838 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:20:29.838 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:20:29.838 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:20:29.838 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:20:29.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:29.839 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2024-04-16T16:20:29.839 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5moiNRIRAAz5JzOrjKNLkcphT+iMo+zw==", 2024-04-16T16:20:29.839 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:20:29.839 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:20:29.840 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:20:29.840 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:20:29.840 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.840 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:29.840 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-16T16:20:29.840 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:20:29.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.815+0000 7f9e41600640 1 -- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9e1c06ca10 msgr2=0x7f9e1c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:29.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.815+0000 7f9e41600640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9e1c06ca10 0x7f9e1c06eed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9e28036000 tx=0x7f9e2803c040 comp rx=0 tx=0).stop 2024-04-16T16:20:29.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.815+0000 7f9e41600640 1 -- 172.21.15.45:0/1243372660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e3c0ce9a0 msgr2=0x7f9e3c142c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:29.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.815+0000 7f9e41600640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e3c0ce9a0 0x7f9e3c142c20 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c00d5f0 tx=0x7f9e2c00dab0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.815+0000 7f9e41600640 1 -- 172.21.15.45:0/1243372660 shutdown_connections 2024-04-16T16:20:29.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.815+0000 7f9e41600640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9e1c06ca10 0x7f9e1c06eed0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.815+0000 7f9e41600640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e3c15e8f0 0x7f9e3c1f3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.815+0000 7f9e41600640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e3c0cf080 0x7f9e3c13ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.815+0000 7f9e41600640 1 --2- 172.21.15.45:0/1243372660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e3c0ce9a0 0x7f9e3c142c20 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:29.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.815+0000 7f9e41600640 1 -- 172.21.15.45:0/1243372660 >> 172.21.15.45:0/1243372660 conn(0x7f9e3c06b930 msgr2=0x7f9e3c162410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:29.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.815+0000 7f9e41600640 1 -- 172.21.15.45:0/1243372660 shutdown_connections 2024-04-16T16:20:29.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:29.815+0000 7f9e41600640 1 -- 172.21.15.45:0/1243372660 wait complete. 2024-04-16T16:20:29.846 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:20:30.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.066+0000 7fc403e00640 1 Processor -- start 2024-04-16T16:20:30.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.066+0000 7fc403e00640 1 -- start start 2024-04-16T16:20:30.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.067+0000 7fc403e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc4040cde00 0x7fc4040cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.068+0000 7fc403e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc40406f540 0x7fc4041645b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.068+0000 7fc403e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc404163510 con 0x7fc404163010 2024-04-16T16:20:30.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.068+0000 7fc403e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc4041637f0 con 0x7fc4040cde00 2024-04-16T16:20:30.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.068+0000 7fc403e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc404163ad0 con 0x7fc40406f540 2024-04-16T16:20:30.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.068+0000 7fc402a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc4040cde00 0x7fc4040cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.068+0000 7fc3fbe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc404163010 0x7fc4040ce1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57234/0 (socket says 172.21.15.45:57234) 2024-04-16T16:20:30.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.068+0000 7fc3fbe00640 1 -- 172.21.15.45:0/605164183 learned_addr learned my addr 172.21.15.45:0/605164183 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:30.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.068+0000 7fc403400640 1 --2- 172.21.15.45:0/605164183 >> v2:172.21.15.103:3300/0 conn(0x7fc40406f540 0x7fc4041645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.068+0000 7fc402a00640 1 -- 172.21.15.45:0/605164183 >> v2:172.21.15.103:3300/0 conn(0x7fc40406f540 msgr2=0x7fc4041645b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:30.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.068+0000 7fc402a00640 1 --2- 172.21.15.45:0/605164183 >> v2:172.21.15.103:3300/0 conn(0x7fc40406f540 0x7fc4041645b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.068+0000 7fc402a00640 1 -- 172.21.15.45:0/605164183 >> v1:172.21.15.45:6789/0 conn(0x7fc404163010 legacy=0x7fc4040ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.069+0000 7fc402a00640 1 -- 172.21.15.45:0/605164183 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc404163db0 con 0x7fc4040cde00 2024-04-16T16:20:30.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.069+0000 7fc403400640 1 --2- 172.21.15.45:0/605164183 >> v2:172.21.15.103:3300/0 conn(0x7fc40406f540 0x7fc4041645b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:30.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.068+0000 7fc402000640 1 -- 172.21.15.45:0/605164183 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 676476282 0 0) 0x7fc404163510 con 0x7fc404163010 2024-04-16T16:20:30.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.070+0000 7fc402a00640 1 --2- 172.21.15.45:0/605164183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4040cde00 0x7fc4040cff30 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fc4040733e0 tx=0x7fc3f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb853caf62952d5d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:30.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.070+0000 7fc402000640 1 -- 172.21.15.45:0/605164183 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4035020 con 0x7fc4040cde00 2024-04-16T16:20:30.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.070+0000 7fc402000640 1 -- 172.21.15.45:0/605164183 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3f4031830 con 0x7fc4040cde00 2024-04-16T16:20:30.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.070+0000 7fc402000640 1 -- 172.21.15.45:0/605164183 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4031c00 con 0x7fc4040cde00 2024-04-16T16:20:30.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.070+0000 7fc403e00640 1 -- 172.21.15.45:0/605164183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4040cde00 msgr2=0x7fc4040cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.070+0000 7fc403e00640 1 --2- 172.21.15.45:0/605164183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4040cde00 0x7fc4040cff30 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fc4040733e0 tx=0x7fc3f402b8d0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.071+0000 7fc403e00640 1 -- 172.21.15.45:0/605164183 shutdown_connections 2024-04-16T16:20:30.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.071+0000 7fc403e00640 1 --2- 172.21.15.45:0/605164183 >> v2:172.21.15.103:3300/0 conn(0x7fc40406f540 0x7fc4041645b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.071+0000 7fc403e00640 1 --2- 172.21.15.45:0/605164183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4040cde00 0x7fc4040cff30 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.071+0000 7fc403e00640 1 -- 172.21.15.45:0/605164183 >> 172.21.15.45:0/605164183 conn(0x7fc40406b930 msgr2=0x7fc40406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:30.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.071+0000 7fc403e00640 1 -- 172.21.15.45:0/605164183 shutdown_connections 2024-04-16T16:20:30.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.071+0000 7fc403e00640 1 -- 172.21.15.45:0/605164183 wait complete. 2024-04-16T16:20:30.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.072+0000 7fc403e00640 1 Processor -- start 2024-04-16T16:20:30.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.073+0000 7fc403e00640 1 -- start start 2024-04-16T16:20:30.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.073+0000 7fc403e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40406f540 0x7fc4040c5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.073+0000 7fc403e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4040cde00 0x7fc40415d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.073+0000 7fc403e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc404163010 0x7fc404161240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.073+0000 7fc403e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc404164af0 con 0x7fc40406f540 2024-04-16T16:20:30.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.073+0000 7fc403e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc40406f970 con 0x7fc404163010 2024-04-16T16:20:30.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.073+0000 7fc403e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc404164c70 con 0x7fc4040cde00 2024-04-16T16:20:30.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.074+0000 7fc402a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40406f540 0x7fc4040c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.074+0000 7fc402a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40406f540 0x7fc4040c5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49966/0 (socket says 172.21.15.45:49966) 2024-04-16T16:20:30.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.074+0000 7fc402a00640 1 -- 172.21.15.45:0/4172511595 learned_addr learned my addr 172.21.15.45:0/4172511595 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:30.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.074+0000 7fc403400640 1 --2- 172.21.15.45:0/4172511595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc404163010 0x7fc404161240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.074+0000 7fc402a00640 1 -- 172.21.15.45:0/4172511595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4040cde00 msgr2=0x7fc40415d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.074+0000 7fc402a00640 1 --2- 172.21.15.45:0/4172511595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4040cde00 0x7fc40415d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.074+0000 7fc402a00640 1 -- 172.21.15.45:0/4172511595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc404163010 msgr2=0x7fc404161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.074+0000 7fc402a00640 1 --2- 172.21.15.45:0/4172511595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc404163010 0x7fc404161240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.074+0000 7fc402a00640 1 -- 172.21.15.45:0/4172511595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc40415dca0 con 0x7fc40406f540 2024-04-16T16:20:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.075+0000 7fc402a00640 1 --2- 172.21.15.45:0/4172511595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40406f540 0x7fc4040c5670 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4009890 tx=0x7fc3f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.075+0000 7fc3faa00640 1 -- 172.21.15.45:0/4172511595 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4035050 con 0x7fc40406f540 2024-04-16T16:20:30.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.075+0000 7fc3faa00640 1 -- 172.21.15.45:0/4172511595 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3f4004230 con 0x7fc40406f540 2024-04-16T16:20:30.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.075+0000 7fc3faa00640 1 -- 172.21.15.45:0/4172511595 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f403b800 con 0x7fc40406f540 2024-04-16T16:20:30.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.075+0000 7fc403e00640 1 -- 172.21.15.45:0/4172511595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc40415e3d0 con 0x7fc40406f540 2024-04-16T16:20:30.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.075+0000 7fc403e00640 1 -- 172.21.15.45:0/4172511595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc40415e990 con 0x7fc40406f540 2024-04-16T16:20:30.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.076+0000 7fc403e00640 1 -- 172.21.15.45:0/4172511595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc404075780 con 0x7fc40406f540 2024-04-16T16:20:30.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.077+0000 7fc3faa00640 1 -- 172.21.15.45:0/4172511595 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7fc3f4038020 con 0x7fc40406f540 2024-04-16T16:20:30.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.078+0000 7fc3faa00640 1 --2- 172.21.15.45:0/4172511595 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fc3e006ca10 0x7fc3e006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.078+0000 7fc3faa00640 1 -- 172.21.15.45:0/4172511595 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 4857+0+0 (secure 0 0 0) 0x7fc3f40ada40 con 0x7fc40406f540 2024-04-16T16:20:30.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.079+0000 7fc3fbe00640 1 --2- 172.21.15.45:0/4172511595 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fc3e006ca10 0x7fc3e006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.080+0000 7fc3fbe00640 1 --2- 172.21.15.45:0/4172511595 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fc3e006ca10 0x7fc3e006eed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec007fd0 tx=0x7fc3ec007960 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:30.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.083+0000 7fc3faa00640 1 -- 172.21.15.45:0/4172511595 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc40415dca0 con 0x7fc40406f540 2024-04-16T16:20:30.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.295+0000 7fc403e00640 1 -- 172.21.15.45:0/4172511595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc4041577a0 con 0x7fc40406f540 2024-04-16T16:20:30.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.296+0000 7fc3faa00640 1 -- 172.21.15.45:0/4172511595 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) ==== 74+0+12901 (secure 0 0 0) 0x7fc3f4040070 con 0x7fc40406f540 2024-04-16T16:20:30.296 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:30.296 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":124,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:20:29.287480+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:20:30.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.299+0000 7fc403e00640 1 -- 172.21.15.45:0/4172511595 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fc3e006ca10 msgr2=0x7fc3e006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.299+0000 7fc403e00640 1 --2- 172.21.15.45:0/4172511595 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fc3e006ca10 0x7fc3e006eed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec007fd0 tx=0x7fc3ec007960 comp rx=0 tx=0).stop 2024-04-16T16:20:30.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.299+0000 7fc403e00640 1 -- 172.21.15.45:0/4172511595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40406f540 msgr2=0x7fc4040c5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.299+0000 7fc403e00640 1 --2- 172.21.15.45:0/4172511595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40406f540 0x7fc4040c5670 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4009890 tx=0x7fc3f40046c0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.299+0000 7fc403e00640 1 -- 172.21.15.45:0/4172511595 shutdown_connections 2024-04-16T16:20:30.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.299+0000 7fc403e00640 1 --2- 172.21.15.45:0/4172511595 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fc3e006ca10 0x7fc3e006eed0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.299+0000 7fc403e00640 1 --2- 172.21.15.45:0/4172511595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc404163010 0x7fc404161240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.299+0000 7fc403e00640 1 --2- 172.21.15.45:0/4172511595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4040cde00 0x7fc40415d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.299+0000 7fc403e00640 1 --2- 172.21.15.45:0/4172511595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40406f540 0x7fc4040c5670 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.299+0000 7fc403e00640 1 -- 172.21.15.45:0/4172511595 >> 172.21.15.45:0/4172511595 conn(0x7fc40406b930 msgr2=0x7fc40406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:30.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.299+0000 7fc403e00640 1 -- 172.21.15.45:0/4172511595 shutdown_connections 2024-04-16T16:20:30.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.299+0000 7fc403e00640 1 -- 172.21.15.45:0/4172511595 wait complete. 2024-04-16T16:20:30.317 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:20:30.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:20:30.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.519+0000 7f7fa8e00640 1 Processor -- start 2024-04-16T16:20:30.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.519+0000 7f7fa8e00640 1 -- start start 2024-04-16T16:20:30.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.521+0000 7f7fa8e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa40cfc40 0x7f7fa4169890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.521+0000 7f7fa8e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa415ee50 con 0x7f7fa415ea70 2024-04-16T16:20:30.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.521+0000 7f7fa8e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa40d1a10 con 0x7f7fa40d4370 2024-04-16T16:20:30.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.521+0000 7f7fa8e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fa40b6480 con 0x7f7fa40cfc40 2024-04-16T16:20:30.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.521+0000 7f7fa3400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7fa415ea70 0x7f7fa40d11a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57248/0 (socket says 172.21.15.45:57248) 2024-04-16T16:20:30.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.521+0000 7f7fa3400640 1 -- 172.21.15.45:0/3084621942 learned_addr learned my addr 172.21.15.45:0/3084621942 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:30.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa3e00640 1 --2- 172.21.15.45:0/3084621942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa40cfc40 0x7f7fa4169890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa2000640 1 -- 172.21.15.45:0/3084621942 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3300435826 0 0) 0x7f7fa415ee50 con 0x7f7fa415ea70 2024-04-16T16:20:30.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa2000640 1 -- 172.21.15.45:0/3084621942 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f88003520 con 0x7f7fa415ea70 2024-04-16T16:20:30.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa2000640 1 -- 172.21.15.45:0/3084621942 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2995561822 0 0) 0x7f7f88003520 con 0x7f7fa415ea70 2024-04-16T16:20:30.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa2000640 1 -- 172.21.15.45:0/3084621942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa40cfc40 msgr2=0x7f7fa4169890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:30.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa2000640 1 --2- 172.21.15.45:0/3084621942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa40cfc40 0x7f7fa4169890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa2000640 1 -- 172.21.15.45:0/3084621942 >> v1:172.21.15.103:6789/0 conn(0x7f7fa40d4370 legacy=0x7f7fa40cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa2000640 1 -- 172.21.15.45:0/3084621942 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa40b6760 con 0x7f7fa415ea70 2024-04-16T16:20:30.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa2000640 1 -- 172.21.15.45:0/3084621942 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f7f94001670 con 0x7f7fa415ea70 2024-04-16T16:20:30.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa2000640 1 -- 172.21.15.45:0/3084621942 >> v1:172.21.15.45:6789/0 conn(0x7f7fa415ea70 legacy=0x7f7fa40d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa2000640 1 --2- 172.21.15.45:0/3084621942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f880039e0 0x7f7f88003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa2000640 1 --2- 172.21.15.45:0/3084621942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f88004800 0x7f7f88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa2000640 1 --2- 172.21.15.45:0/3084621942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa40cfc40 0x7f7f880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa2000640 1 -- 172.21.15.45:0/3084621942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa40b6480 con 0x7f7f880039e0 2024-04-16T16:20:30.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa2000640 1 -- 172.21.15.45:0/3084621942 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8800bad0 con 0x7f7f88004800 2024-04-16T16:20:30.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa2000640 1 -- 172.21.15.45:0/3084621942 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7f8800bcd0 con 0x7f7fa40cfc40 2024-04-16T16:20:30.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.522+0000 7f7fa3400640 1 --2- 172.21.15.45:0/3084621942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f880039e0 0x7f7f88003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.523+0000 7f7fa3e00640 1 --2- 172.21.15.45:0/3084621942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa40cfc40 0x7f7f880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.523+0000 7f7fa2a00640 1 --2- 172.21.15.45:0/3084621942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f88004800 0x7f7f88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.523+0000 7f7fa3400640 1 -- 172.21.15.45:0/3084621942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa40cfc40 msgr2=0x7f7f880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:30.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.523+0000 7f7fa3400640 1 --2- 172.21.15.45:0/3084621942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa40cfc40 0x7f7f880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.523+0000 7f7fa3400640 1 -- 172.21.15.45:0/3084621942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f88004800 msgr2=0x7f7f88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:30.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.523+0000 7f7fa3400640 1 --2- 172.21.15.45:0/3084621942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f88004800 0x7f7f88004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.523+0000 7f7fa3400640 1 -- 172.21.15.45:0/3084621942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f88003520 con 0x7f7f880039e0 2024-04-16T16:20:30.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.523+0000 7f7fa2a00640 1 --2- 172.21.15.45:0/3084621942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f88004800 0x7f7f88004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:30.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.524+0000 7f7fa3400640 1 --2- 172.21.15.45:0/3084621942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f880039e0 0x7f7f88003e00 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400c160 tx=0x7f7f88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693ffb422a14e6e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:30.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.524+0000 7f7fa2000640 1 -- 172.21.15.45:0/3084621942 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94007c60 con 0x7f7f880039e0 2024-04-16T16:20:30.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.524+0000 7f7fa2000640 1 -- 172.21.15.45:0/3084621942 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f94030070 con 0x7f7f880039e0 2024-04-16T16:20:30.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.524+0000 7f7fa2000640 1 -- 172.21.15.45:0/3084621942 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94005d30 con 0x7f7f880039e0 2024-04-16T16:20:30.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.525+0000 7f7fa8e00640 1 -- 172.21.15.45:0/3084621942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f880039e0 msgr2=0x7f7f88003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.525+0000 7f7fa8e00640 1 --2- 172.21.15.45:0/3084621942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f880039e0 0x7f7f88003e00 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400c160 tx=0x7f7f88005830 comp rx=0 tx=0).stop 2024-04-16T16:20:30.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.525+0000 7f7fa3e00640 1 -- 172.21.15.45:0/3084621942 reap_dead start 2024-04-16T16:20:30.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.525+0000 7f7fa8e00640 1 -- 172.21.15.45:0/3084621942 shutdown_connections 2024-04-16T16:20:30.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.525+0000 7f7fa8e00640 1 -- 172.21.15.45:0/3084621942 >> 172.21.15.45:0/3084621942 conn(0x7f7fa406d3b0 msgr2=0x7f7fa4073060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:30.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.525+0000 7f7fa8e00640 1 -- 172.21.15.45:0/3084621942 shutdown_connections 2024-04-16T16:20:30.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.525+0000 7f7fa8e00640 1 -- 172.21.15.45:0/3084621942 wait complete. 2024-04-16T16:20:30.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.527+0000 7f7fa8e00640 1 Processor -- start 2024-04-16T16:20:30.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.527+0000 7f7fa8e00640 1 -- start start 2024-04-16T16:20:30.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.528+0000 7f7fa8e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f880039e0 0x7f7fa414e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.528+0000 7f7fa8e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fa415f060 0x7f7fa40b7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.528+0000 7f7fa8e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa40b8080 0x7f7fa40cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.528+0000 7f7fa8e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa40b6f60 con 0x7f7f880039e0 2024-04-16T16:20:30.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.528+0000 7f7fa8e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa4167880 con 0x7f7fa415f060 2024-04-16T16:20:30.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.528+0000 7f7fa8e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fa414f030 con 0x7f7fa40b8080 2024-04-16T16:20:30.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.528+0000 7f7fa3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f880039e0 0x7f7fa414e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.528+0000 7f7fa3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f880039e0 0x7f7fa414e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49986/0 (socket says 172.21.15.45:49986) 2024-04-16T16:20:30.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.528+0000 7f7fa3400640 1 -- 172.21.15.45:0/2293433358 learned_addr learned my addr 172.21.15.45:0/2293433358 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:30.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.528+0000 7f7fa3400640 1 -- 172.21.15.45:0/2293433358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa40b8080 msgr2=0x7f7fa40cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.528+0000 7f7fa3400640 1 --2- 172.21.15.45:0/2293433358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa40b8080 0x7f7fa40cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.528+0000 7f7fa3400640 1 -- 172.21.15.45:0/2293433358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fa415f060 msgr2=0x7f7fa40b7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.528+0000 7f7fa3400640 1 --2- 172.21.15.45:0/2293433358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fa415f060 0x7f7fa40b7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.528+0000 7f7fa3400640 1 -- 172.21.15.45:0/2293433358 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa4156770 con 0x7f7f880039e0 2024-04-16T16:20:30.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.529+0000 7f7fa3400640 1 --2- 172.21.15.45:0/2293433358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f880039e0 0x7f7fa414e590 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400c280 tx=0x7f7f94001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:30.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.529+0000 7f7f9b400640 1 -- 172.21.15.45:0/2293433358 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94046070 con 0x7f7f880039e0 2024-04-16T16:20:30.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.529+0000 7f7f9b400640 1 -- 172.21.15.45:0/2293433358 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f94000ed0 con 0x7f7f880039e0 2024-04-16T16:20:30.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.529+0000 7f7f9b400640 1 -- 172.21.15.45:0/2293433358 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94007c60 con 0x7f7f880039e0 2024-04-16T16:20:30.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.529+0000 7f7fa8e00640 1 -- 172.21.15.45:0/2293433358 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fa414ead0 con 0x7f7f880039e0 2024-04-16T16:20:30.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.529+0000 7f7fa8e00640 1 -- 172.21.15.45:0/2293433358 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fa41569e0 con 0x7f7f880039e0 2024-04-16T16:20:30.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.530+0000 7f7fa8e00640 1 -- 172.21.15.45:0/2293433358 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fa40b84a0 con 0x7f7f880039e0 2024-04-16T16:20:30.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.531+0000 7f7f9b400640 1 -- 172.21.15.45:0/2293433358 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f7f94007070 con 0x7f7f880039e0 2024-04-16T16:20:30.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.533+0000 7f7f9b400640 1 --2- 172.21.15.45:0/2293433358 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f7f8406ca90 0x7f7f8406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.533+0000 7f7f9b400640 1 -- 172.21.15.45:0/2293433358 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 4857+0+0 (secure 0 0 0) 0x7f7f940ace90 con 0x7f7f880039e0 2024-04-16T16:20:30.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.533+0000 7f7fa2a00640 1 --2- 172.21.15.45:0/2293433358 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f7f8406ca90 0x7f7f8406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.535+0000 7f7fa2a00640 1 --2- 172.21.15.45:0/2293433358 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f7f8406ca90 0x7f7f8406ef50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7fa41620d0 tx=0x7f7f90009040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:30.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.537+0000 7f7f9b400640 1 -- 172.21.15.45:0/2293433358 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7f940b0030 con 0x7f7f880039e0 2024-04-16T16:20:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.730+0000 7f7fa8e00640 1 -- 172.21.15.45:0/2293433358 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7fa4075780 con 0x7f7f880039e0 2024-04-16T16:20:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.731+0000 7f7f9b400640 1 -- 172.21.15.45:0/2293433358 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v124) ==== 93+0+10 (secure 0 0 0) 0x7f7f9402de20 con 0x7f7f880039e0 2024-04-16T16:20:30.731 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:20:30.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.734+0000 7f7fa8e00640 1 -- 172.21.15.45:0/2293433358 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f7f8406ca90 msgr2=0x7f7f8406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.734+0000 7f7fa8e00640 1 --2- 172.21.15.45:0/2293433358 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f7f8406ca90 0x7f7f8406ef50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7fa41620d0 tx=0x7f7f90009040 comp rx=0 tx=0).stop 2024-04-16T16:20:30.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.734+0000 7f7fa8e00640 1 -- 172.21.15.45:0/2293433358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f880039e0 msgr2=0x7f7fa414e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.734+0000 7f7fa8e00640 1 --2- 172.21.15.45:0/2293433358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f880039e0 0x7f7fa414e590 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400c280 tx=0x7f7f94001970 comp rx=0 tx=0).stop 2024-04-16T16:20:30.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.734+0000 7f7fa8e00640 1 -- 172.21.15.45:0/2293433358 shutdown_connections 2024-04-16T16:20:30.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.734+0000 7f7fa8e00640 1 --2- 172.21.15.45:0/2293433358 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f7f8406ca90 0x7f7f8406ef50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.734+0000 7f7fa8e00640 1 --2- 172.21.15.45:0/2293433358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa40b8080 0x7f7fa40cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.734+0000 7f7fa8e00640 1 --2- 172.21.15.45:0/2293433358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fa415f060 0x7f7fa40b7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.734+0000 7f7fa8e00640 1 --2- 172.21.15.45:0/2293433358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f880039e0 0x7f7fa414e590 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.734+0000 7f7fa8e00640 1 -- 172.21.15.45:0/2293433358 >> 172.21.15.45:0/2293433358 conn(0x7f7fa406d3b0 msgr2=0x7f7fa4158450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:30.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.734+0000 7f7fa8e00640 1 -- 172.21.15.45:0/2293433358 shutdown_connections 2024-04-16T16:20:30.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.734+0000 7f7fa8e00640 1 -- 172.21.15.45:0/2293433358 wait complete. 2024-04-16T16:20:30.753 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-16T16:20:30.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.968+0000 7f1e59600640 1 Processor -- start 2024-04-16T16:20:30.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.968+0000 7f1e59600640 1 -- start start 2024-04-16T16:20:30.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.970+0000 7f1e59600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1e541608a0 0x7f1e54160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.970+0000 7f1e59600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e5415d360 con 0x7f1e54146220 2024-04-16T16:20:30.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.970+0000 7f1e59600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e5415d640 con 0x7f1e5415ccb0 2024-04-16T16:20:30.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.970+0000 7f1e59600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1e5415d130 con 0x7f1e541608a0 2024-04-16T16:20:30.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.970+0000 7f1e53e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1e54146220 0x7f1e5406f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57264/0 (socket says 172.21.15.45:57264) 2024-04-16T16:20:30.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.970+0000 7f1e53e00640 1 -- 172.21.15.45:0/2686833423 learned_addr learned my addr 172.21.15.45:0/2686833423 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:30.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.970+0000 7f1e58c00640 1 --2- 172.21.15.45:0/2686833423 >> v2:172.21.15.45:3300/0 conn(0x7f1e541608a0 0x7f1e54160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.971+0000 7f1e52a00640 1 -- 172.21.15.45:0/2686833423 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2982700250 0 0) 0x7f1e5415d360 con 0x7f1e54146220 2024-04-16T16:20:30.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.971+0000 7f1e58c00640 1 -- 172.21.15.45:0/2686833423 >> v1:172.21.15.103:6789/0 conn(0x7f1e5415ccb0 legacy=0x7f1e5406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.971+0000 7f1e58c00640 1 -- 172.21.15.45:0/2686833423 >> v1:172.21.15.45:6789/0 conn(0x7f1e54146220 legacy=0x7f1e5406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.971+0000 7f1e58c00640 1 -- 172.21.15.45:0/2686833423 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5415d920 con 0x7f1e541608a0 2024-04-16T16:20:30.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.972+0000 7f1e58c00640 1 --2- 172.21.15.45:0/2686833423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e541608a0 0x7f1e54160cf0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009d90 tx=0x7f1e4c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c9decf335e88ee6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:30.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.972+0000 7f1e52a00640 1 -- 172.21.15.45:0/2686833423 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c039070 con 0x7f1e541608a0 2024-04-16T16:20:30.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.972+0000 7f1e52a00640 1 -- 172.21.15.45:0/2686833423 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e4c02cd00 con 0x7f1e541608a0 2024-04-16T16:20:30.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.972+0000 7f1e52a00640 1 -- 172.21.15.45:0/2686833423 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c034660 con 0x7f1e541608a0 2024-04-16T16:20:30.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.973+0000 7f1e59600640 1 -- 172.21.15.45:0/2686833423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e541608a0 msgr2=0x7f1e54160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.973+0000 7f1e59600640 1 --2- 172.21.15.45:0/2686833423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e541608a0 0x7f1e54160cf0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009d90 tx=0x7f1e4c02bf50 comp rx=0 tx=0).stop 2024-04-16T16:20:30.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.973+0000 7f1e59600640 1 -- 172.21.15.45:0/2686833423 shutdown_connections 2024-04-16T16:20:30.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.973+0000 7f1e59600640 1 --2- 172.21.15.45:0/2686833423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e541608a0 0x7f1e54160cf0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.973+0000 7f1e59600640 1 -- 172.21.15.45:0/2686833423 >> 172.21.15.45:0/2686833423 conn(0x7f1e5406b930 msgr2=0x7f1e5406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:30.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.973+0000 7f1e59600640 1 -- 172.21.15.45:0/2686833423 shutdown_connections 2024-04-16T16:20:30.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.973+0000 7f1e59600640 1 -- 172.21.15.45:0/2686833423 wait complete. 2024-04-16T16:20:30.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.975+0000 7f1e59600640 1 Processor -- start 2024-04-16T16:20:30.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.975+0000 7f1e59600640 1 -- start start 2024-04-16T16:20:30.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.976+0000 7f1e59600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e54146220 0x7f1e5413dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.976+0000 7f1e59600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e5415ccb0 0x7f1e541f3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.976+0000 7f1e59600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e541608a0 0x7f1e541f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.976+0000 7f1e59600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1e54160410 con 0x7f1e54146220 2024-04-16T16:20:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.976+0000 7f1e59600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5415dc00 con 0x7f1e541608a0 2024-04-16T16:20:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.976+0000 7f1e59600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1e540cf360 con 0x7f1e5415ccb0 2024-04-16T16:20:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.976+0000 7f1e53e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e54146220 0x7f1e5413dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.976+0000 7f1e53e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e54146220 0x7f1e5413dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49994/0 (socket says 172.21.15.45:49994) 2024-04-16T16:20:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.976+0000 7f1e53e00640 1 -- 172.21.15.45:0/2154963410 learned_addr learned my addr 172.21.15.45:0/2154963410 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.977+0000 7f1e53400640 1 --2- 172.21.15.45:0/2154963410 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e5415ccb0 0x7f1e541f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.977+0000 7f1e58c00640 1 --2- 172.21.15.45:0/2154963410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e541608a0 0x7f1e541f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.977+0000 7f1e53e00640 1 -- 172.21.15.45:0/2154963410 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e5415ccb0 msgr2=0x7f1e541f3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.977+0000 7f1e53e00640 1 --2- 172.21.15.45:0/2154963410 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e5415ccb0 0x7f1e541f3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.977+0000 7f1e53e00640 1 -- 172.21.15.45:0/2154963410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e541608a0 msgr2=0x7f1e541f21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:30.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.977+0000 7f1e53e00640 1 --2- 172.21.15.45:0/2154963410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e541608a0 0x7f1e541f21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:30.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.977+0000 7f1e53e00640 1 -- 172.21.15.45:0/2154963410 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e541f2a30 con 0x7f1e54146220 2024-04-16T16:20:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.977+0000 7f1e53e00640 1 --2- 172.21.15.45:0/2154963410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e54146220 0x7f1e5413dac0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f1e4400c4a0 tx=0x7f1e4400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.977+0000 7f1e4be00640 1 -- 172.21.15.45:0/2154963410 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e44018070 con 0x7f1e54146220 2024-04-16T16:20:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.977+0000 7f1e4be00640 1 -- 172.21.15.45:0/2154963410 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e440074e0 con 0x7f1e54146220 2024-04-16T16:20:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.977+0000 7f1e4be00640 1 -- 172.21.15.45:0/2154963410 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e44004a20 con 0x7f1e54146220 2024-04-16T16:20:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.977+0000 7f1e59600640 1 -- 172.21.15.45:0/2154963410 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e541f2d10 con 0x7f1e54146220 2024-04-16T16:20:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.978+0000 7f1e59600640 1 -- 172.21.15.45:0/2154963410 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e541f32d0 con 0x7f1e54146220 2024-04-16T16:20:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.979+0000 7f1e59600640 1 -- 172.21.15.45:0/2154963410 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e54075780 con 0x7f1e54146220 2024-04-16T16:20:30.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.980+0000 7f1e4be00640 1 -- 172.21.15.45:0/2154963410 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f1e44004bc0 con 0x7f1e54146220 2024-04-16T16:20:30.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.981+0000 7f1e4be00640 1 --2- 172.21.15.45:0/2154963410 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f1e3406ca10 0x7f1e3406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:30.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.981+0000 7f1e4be00640 1 -- 172.21.15.45:0/2154963410 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 4857+0+0 (secure 0 0 0) 0x7f1e4408acf0 con 0x7f1e54146220 2024-04-16T16:20:30.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.981+0000 7f1e53400640 1 --2- 172.21.15.45:0/2154963410 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f1e3406ca10 0x7f1e3406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:30.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.983+0000 7f1e53400640 1 --2- 172.21.15.45:0/2154963410 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f1e3406ca10 0x7f1e3406eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1e54141750 tx=0x7f1e40009040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:30.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:30.986+0000 7f1e4be00640 1 -- 172.21.15.45:0/2154963410 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1e44059350 con 0x7f1e54146220 2024-04-16T16:20:31.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.191+0000 7f1e59600640 1 -- 172.21.15.45:0/2154963410 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f1e540cff60 con 0x7f1e54146220 2024-04-16T16:20:31.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.192+0000 7f1e4be00640 1 -- 172.21.15.45:0/2154963410 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v124) ==== 84+0+65 (secure 0 0 0) 0x7f1e44058c50 con 0x7f1e54146220 2024-04-16T16:20:31.191 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:31.191 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2024-04-16T16:20:31.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:20:31.192 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2024-04-16T16:20:31.192 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2024-04-16T16:20:31.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:20:31.192 INFO:teuthology.orchestra.run.smithi045.stdout:] 2024-04-16T16:20:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.195+0000 7f1e59600640 1 -- 172.21.15.45:0/2154963410 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f1e3406ca10 msgr2=0x7f1e3406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.195+0000 7f1e59600640 1 --2- 172.21.15.45:0/2154963410 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f1e3406ca10 0x7f1e3406eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1e54141750 tx=0x7f1e40009040 comp rx=0 tx=0).stop 2024-04-16T16:20:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.195+0000 7f1e59600640 1 -- 172.21.15.45:0/2154963410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e54146220 msgr2=0x7f1e5413dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.195+0000 7f1e59600640 1 --2- 172.21.15.45:0/2154963410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e54146220 0x7f1e5413dac0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f1e4400c4a0 tx=0x7f1e4400c960 comp rx=0 tx=0).stop 2024-04-16T16:20:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.195+0000 7f1e59600640 1 -- 172.21.15.45:0/2154963410 shutdown_connections 2024-04-16T16:20:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.195+0000 7f1e59600640 1 --2- 172.21.15.45:0/2154963410 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f1e3406ca10 0x7f1e3406eed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.195+0000 7f1e59600640 1 --2- 172.21.15.45:0/2154963410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e541608a0 0x7f1e541f21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.195+0000 7f1e59600640 1 --2- 172.21.15.45:0/2154963410 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e5415ccb0 0x7f1e541f3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.195+0000 7f1e59600640 1 --2- 172.21.15.45:0/2154963410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e54146220 0x7f1e5413dac0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.195+0000 7f1e59600640 1 -- 172.21.15.45:0/2154963410 >> 172.21.15.45:0/2154963410 conn(0x7f1e5406b930 msgr2=0x7f1e5415e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.195+0000 7f1e59600640 1 -- 172.21.15.45:0/2154963410 shutdown_connections 2024-04-16T16:20:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.195+0000 7f1e59600640 1 -- 172.21.15.45:0/2154963410 wait complete. 2024-04-16T16:20:31.212 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-16T16:20:31.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-16T16:20:31.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.444+0000 7fa426800640 1 Processor -- start 2024-04-16T16:20:31.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.444+0000 7fa426800640 1 -- start start 2024-04-16T16:20:31.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.446+0000 7fa426800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa42006eaf0 0x7fa4200ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:31.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.446+0000 7fa426800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa4200ceaf0 0x7fa42015cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:31.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.446+0000 7fa426800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa420071e60 con 0x7fa4201452e0 2024-04-16T16:20:31.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.446+0000 7fa426800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa420072f00 con 0x7fa4200ceaf0 2024-04-16T16:20:31.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.446+0000 7fa426800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa420072140 con 0x7fa42006eaf0 2024-04-16T16:20:31.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.446+0000 7fa425400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa4201452e0 0x7fa42006e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57280/0 (socket says 172.21.15.45:57280) 2024-04-16T16:20:31.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.446+0000 7fa425400640 1 -- 172.21.15.45:0/2987909030 learned_addr learned my addr 172.21.15.45:0/2987909030 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:31.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.446+0000 7fa425e00640 1 --2- 172.21.15.45:0/2987909030 >> v2:172.21.15.45:3300/0 conn(0x7fa4200ceaf0 0x7fa42015cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:31.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.447+0000 7fa424a00640 1 --2- 172.21.15.45:0/2987909030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa42006eaf0 0x7fa4200ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:31.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.447+0000 7fa41fe00640 1 -- 172.21.15.45:0/2987909030 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2090616004 0 0) 0x7fa420071e60 con 0x7fa4201452e0 2024-04-16T16:20:31.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.447+0000 7fa41fe00640 1 -- 172.21.15.45:0/2987909030 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa408003520 con 0x7fa4201452e0 2024-04-16T16:20:31.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.447+0000 7fa425e00640 1 -- 172.21.15.45:0/2987909030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa42006eaf0 msgr2=0x7fa4200ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:31.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.447+0000 7fa425e00640 1 --2- 172.21.15.45:0/2987909030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa42006eaf0 0x7fa4200ce5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:31.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.447+0000 7fa425e00640 1 -- 172.21.15.45:0/2987909030 >> v1:172.21.15.45:6789/0 conn(0x7fa4201452e0 legacy=0x7fa42006e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:31.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.447+0000 7fa425e00640 1 -- 172.21.15.45:0/2987909030 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa420072420 con 0x7fa4200ceaf0 2024-04-16T16:20:31.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.447+0000 7fa424a00640 1 --2- 172.21.15.45:0/2987909030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa42006eaf0 0x7fa4200ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:20:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.448+0000 7fa425e00640 1 --2- 172.21.15.45:0/2987909030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4200ceaf0 0x7fa42015cc90 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fa418009d90 tx=0x7fa41802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81df612b594d6830 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.448+0000 7fa41fe00640 1 -- 172.21.15.45:0/2987909030 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418039070 con 0x7fa4200ceaf0 2024-04-16T16:20:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.449+0000 7fa41fe00640 1 -- 172.21.15.45:0/2987909030 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa41802cce0 con 0x7fa4200ceaf0 2024-04-16T16:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.449+0000 7fa41fe00640 1 -- 172.21.15.45:0/2987909030 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418034660 con 0x7fa4200ceaf0 2024-04-16T16:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.449+0000 7fa426800640 1 -- 172.21.15.45:0/2987909030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4200ceaf0 msgr2=0x7fa42015cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.449+0000 7fa426800640 1 --2- 172.21.15.45:0/2987909030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4200ceaf0 0x7fa42015cc90 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fa418009d90 tx=0x7fa41802bf30 comp rx=0 tx=0).stop 2024-04-16T16:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.449+0000 7fa426800640 1 -- 172.21.15.45:0/2987909030 shutdown_connections 2024-04-16T16:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.449+0000 7fa426800640 1 --2- 172.21.15.45:0/2987909030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4200ceaf0 0x7fa42015cc90 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.449+0000 7fa426800640 1 --2- 172.21.15.45:0/2987909030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa42006eaf0 0x7fa4200ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.449+0000 7fa426800640 1 -- 172.21.15.45:0/2987909030 >> 172.21.15.45:0/2987909030 conn(0x7fa4200b3bb0 msgr2=0x7fa4200b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:31.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.450+0000 7fa426800640 1 -- 172.21.15.45:0/2987909030 shutdown_connections 2024-04-16T16:20:31.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.450+0000 7fa426800640 1 -- 172.21.15.45:0/2987909030 wait complete. 2024-04-16T16:20:31.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.452+0000 7fa426800640 1 Processor -- start 2024-04-16T16:20:31.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.452+0000 7fa426800640 1 -- start start 2024-04-16T16:20:31.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.453+0000 7fa426800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa42006eaf0 0x7fa420140220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:31.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.453+0000 7fa426800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4200ceaf0 0x7fa4200b57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:31.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.453+0000 7fa426800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4201452e0 0x7fa4200b5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:31.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.453+0000 7fa426800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa4200cef80 con 0x7fa4200ceaf0 2024-04-16T16:20:31.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.453+0000 7fa426800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa4200729e0 con 0x7fa4201452e0 2024-04-16T16:20:31.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.453+0000 7fa426800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa4201456c0 con 0x7fa42006eaf0 2024-04-16T16:20:31.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.454+0000 7fa424a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4200ceaf0 0x7fa4200b57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:31.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.454+0000 7fa424a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4200ceaf0 0x7fa4200b57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50004/0 (socket says 172.21.15.45:50004) 2024-04-16T16:20:31.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.454+0000 7fa424a00640 1 -- 172.21.15.45:0/28694573 learned_addr learned my addr 172.21.15.45:0/28694573 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:31.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.454+0000 7fa425400640 1 --2- 172.21.15.45:0/28694573 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa42006eaf0 0x7fa420140220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:31.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.454+0000 7fa425e00640 1 --2- 172.21.15.45:0/28694573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4201452e0 0x7fa4200b5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:31.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.454+0000 7fa424a00640 1 -- 172.21.15.45:0/28694573 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa42006eaf0 msgr2=0x7fa420140220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:31.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.454+0000 7fa424a00640 1 --2- 172.21.15.45:0/28694573 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa42006eaf0 0x7fa420140220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:31.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.454+0000 7fa424a00640 1 -- 172.21.15.45:0/28694573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4201452e0 msgr2=0x7fa4200b5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:31.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.454+0000 7fa424a00640 1 --2- 172.21.15.45:0/28694573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4201452e0 0x7fa4200b5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:31.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.454+0000 7fa424a00640 1 -- 172.21.15.45:0/28694573 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa420140760 con 0x7fa4200ceaf0 2024-04-16T16:20:31.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.455+0000 7fa424a00640 1 --2- 172.21.15.45:0/28694573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4200ceaf0 0x7fa4200b57d0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fa40c00d800 tx=0x7fa40c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:31.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.455+0000 7fa41d600640 1 -- 172.21.15.45:0/28694573 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40c004030 con 0x7fa4200ceaf0 2024-04-16T16:20:31.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.455+0000 7fa41d600640 1 -- 172.21.15.45:0/28694573 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa40c0041d0 con 0x7fa4200ceaf0 2024-04-16T16:20:31.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.455+0000 7fa41d600640 1 -- 172.21.15.45:0/28694573 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40c00b6c0 con 0x7fa4200ceaf0 2024-04-16T16:20:31.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.455+0000 7fa426800640 1 -- 172.21.15.45:0/28694573 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4200ab8b0 con 0x7fa4200ceaf0 2024-04-16T16:20:31.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.455+0000 7fa426800640 1 -- 172.21.15.45:0/28694573 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4200abe30 con 0x7fa4200ceaf0 2024-04-16T16:20:31.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.457+0000 7fa426800640 1 -- 172.21.15.45:0/28694573 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa420075780 con 0x7fa4200ceaf0 2024-04-16T16:20:31.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.458+0000 7fa41d600640 1 -- 172.21.15.45:0/28694573 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7fa40c00b860 con 0x7fa4200ceaf0 2024-04-16T16:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.459+0000 7fa41d600640 1 --2- 172.21.15.45:0/28694573 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fa40006ca10 0x7fa40006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.459+0000 7fa41d600640 1 -- 172.21.15.45:0/28694573 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 4857+0+0 (secure 0 0 0) 0x7fa40c08bce0 con 0x7fa4200ceaf0 2024-04-16T16:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.459+0000 7fa425400640 1 --2- 172.21.15.45:0/28694573 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fa40006ca10 0x7fa40006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.461+0000 7fa425400640 1 --2- 172.21.15.45:0/28694573 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fa40006ca10 0x7fa40006eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa414001200 tx=0x7fa414009040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.463+0000 7fa41d600640 1 -- 172.21.15.45:0/28694573 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa40c05a2b0 con 0x7fa4200ceaf0 2024-04-16T16:20:31.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:31.667+0000 7fa426800640 1 -- 172.21.15.45:0/28694573 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa420144a90 con 0x7fa4200ceaf0 2024-04-16T16:20:32.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.299+0000 7fa41d600640 1 -- 172.21.15.45:0/28694573 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v125) ==== 138+0+0 (secure 0 0 0) 0x7fa40c059bb0 con 0x7fa4200ceaf0 2024-04-16T16:20:32.299 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2024-04-16T16:20:32.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.304+0000 7fa426800640 1 -- 172.21.15.45:0/28694573 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fa40006ca10 msgr2=0x7fa40006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:32.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.304+0000 7fa426800640 1 --2- 172.21.15.45:0/28694573 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fa40006ca10 0x7fa40006eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa414001200 tx=0x7fa414009040 comp rx=0 tx=0).stop 2024-04-16T16:20:32.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.304+0000 7fa426800640 1 -- 172.21.15.45:0/28694573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4200ceaf0 msgr2=0x7fa4200b57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:32.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.304+0000 7fa426800640 1 --2- 172.21.15.45:0/28694573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4200ceaf0 0x7fa4200b57d0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fa40c00d800 tx=0x7fa40c00dcc0 comp rx=0 tx=0).stop 2024-04-16T16:20:32.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.304+0000 7fa426800640 1 -- 172.21.15.45:0/28694573 shutdown_connections 2024-04-16T16:20:32.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.304+0000 7fa426800640 1 --2- 172.21.15.45:0/28694573 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fa40006ca10 0x7fa40006eed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:32.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.304+0000 7fa426800640 1 --2- 172.21.15.45:0/28694573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4201452e0 0x7fa4200b5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:32.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.304+0000 7fa426800640 1 --2- 172.21.15.45:0/28694573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4200ceaf0 0x7fa4200b57d0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:32.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.304+0000 7fa426800640 1 --2- 172.21.15.45:0/28694573 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa42006eaf0 0x7fa420140220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:32.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.304+0000 7fa426800640 1 -- 172.21.15.45:0/28694573 >> 172.21.15.45:0/28694573 conn(0x7fa4200b3bb0 msgr2=0x7fa4200cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:32.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.304+0000 7fa426800640 1 -- 172.21.15.45:0/28694573 shutdown_connections 2024-04-16T16:20:32.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.304+0000 7fa426800640 1 -- 172.21.15.45:0/28694573 wait complete. 2024-04-16T16:20:32.332 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-16T16:20:32.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.547+0000 7f72a7e00640 1 Processor -- start 2024-04-16T16:20:32.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.547+0000 7f72a7e00640 1 -- start start 2024-04-16T16:20:32.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.548+0000 7f72a7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f72a81452e0 0x7f72a806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:32.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.549+0000 7f72a7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72a80cecc0 0x7f72a815cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:32.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.549+0000 7f72a7e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72a8071e60 con 0x7f72a806e920 2024-04-16T16:20:32.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.549+0000 7f72a7e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f72a8072f00 con 0x7f72a81452e0 2024-04-16T16:20:32.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.549+0000 7f72a7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f72a806ed90 con 0x7f72a80cecc0 2024-04-16T16:20:32.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.549+0000 7f72a6a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f72a81452e0 0x7f72a806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:32.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.549+0000 7f72a6a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f72a81452e0 0x7f72a806e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50016/0 (socket says 172.21.15.45:50016) 2024-04-16T16:20:32.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.549+0000 7f72a6a00640 1 -- 172.21.15.45:0/2626032876 learned_addr learned my addr 172.21.15.45:0/2626032876 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:32.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.549+0000 7f72a7400640 1 --2- 172.21.15.45:0/2626032876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72a80cecc0 0x7f72a815cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:32.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.550+0000 7f72a6a00640 1 -- 172.21.15.45:0/2626032876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72a80cecc0 msgr2=0x7f72a815cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:32.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.550+0000 7f72a6a00640 1 --2- 172.21.15.45:0/2626032876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72a80cecc0 0x7f72a815cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:32.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.550+0000 7f72a6a00640 1 -- 172.21.15.45:0/2626032876 >> v1:172.21.15.103:6789/0 conn(0x7f72a806e920 legacy=0x7f72a80ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:32.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.550+0000 7f72a6a00640 1 -- 172.21.15.45:0/2626032876 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a8072140 con 0x7f72a81452e0 2024-04-16T16:20:32.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.550+0000 7f72a7400640 1 --2- 172.21.15.45:0/2626032876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72a80cecc0 0x7f72a815cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:20:32.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.551+0000 7f72a6a00640 1 --2- 172.21.15.45:0/2626032876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72a81452e0 0x7f72a806e3e0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f72980098a0 tx=0x7f729802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ad4daa20da29bd server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:32.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.551+0000 7f72a6000640 1 -- 172.21.15.45:0/2626032876 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729802f830 con 0x7f72a81452e0 2024-04-16T16:20:32.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.551+0000 7f72a6000640 1 -- 172.21.15.45:0/2626032876 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f729802f9d0 con 0x7f72a81452e0 2024-04-16T16:20:32.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.551+0000 7f72a6000640 1 -- 172.21.15.45:0/2626032876 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729802fda0 con 0x7f72a81452e0 2024-04-16T16:20:32.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.552+0000 7f72a7e00640 1 -- 172.21.15.45:0/2626032876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72a81452e0 msgr2=0x7f72a806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:32.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.552+0000 7f72a7e00640 1 --2- 172.21.15.45:0/2626032876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72a81452e0 0x7f72a806e3e0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f72980098a0 tx=0x7f729802ba10 comp rx=0 tx=0).stop 2024-04-16T16:20:32.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.552+0000 7f72a7e00640 1 -- 172.21.15.45:0/2626032876 shutdown_connections 2024-04-16T16:20:32.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.552+0000 7f72a7e00640 1 --2- 172.21.15.45:0/2626032876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72a80cecc0 0x7f72a815cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:32.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.552+0000 7f72a7e00640 1 --2- 172.21.15.45:0/2626032876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72a81452e0 0x7f72a806e3e0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:32.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.552+0000 7f72a7e00640 1 -- 172.21.15.45:0/2626032876 >> 172.21.15.45:0/2626032876 conn(0x7f72a80b3bb0 msgr2=0x7f72a80b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:32.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.552+0000 7f72a7e00640 1 -- 172.21.15.45:0/2626032876 shutdown_connections 2024-04-16T16:20:32.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.552+0000 7f72a7e00640 1 -- 172.21.15.45:0/2626032876 wait complete. 2024-04-16T16:20:32.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.555+0000 7f72a7e00640 1 Processor -- start 2024-04-16T16:20:32.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.555+0000 7f72a7e00640 1 -- start start 2024-04-16T16:20:32.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.556+0000 7f72a7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72a806e920 0x7f72a8144440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:32.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.556+0000 7f72a7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72a80cecc0 0x7f72a81e72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:32.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.556+0000 7f72a7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72a81452e0 0x7f72a81e7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:32.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.556+0000 7f72a7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f72a80729e0 con 0x7f72a80cecc0 2024-04-16T16:20:32.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.556+0000 7f72a7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72a8072700 con 0x7f72a81452e0 2024-04-16T16:20:32.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.556+0000 7f72a7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f72a81456c0 con 0x7f72a806e920 2024-04-16T16:20:32.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.556+0000 7f729fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72a80cecc0 0x7f72a81e72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:32.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.557+0000 7f729fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72a80cecc0 0x7f72a81e72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50024/0 (socket says 172.21.15.45:50024) 2024-04-16T16:20:32.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.557+0000 7f729fe00640 1 -- 172.21.15.45:0/3980401959 learned_addr learned my addr 172.21.15.45:0/3980401959 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:32.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.557+0000 7f72a6a00640 1 --2- 172.21.15.45:0/3980401959 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72a806e920 0x7f72a8144440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:32.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.557+0000 7f72a7400640 1 --2- 172.21.15.45:0/3980401959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72a81452e0 0x7f72a81e7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:32.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.557+0000 7f729fe00640 1 -- 172.21.15.45:0/3980401959 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72a806e920 msgr2=0x7f72a8144440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:32.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.557+0000 7f729fe00640 1 --2- 172.21.15.45:0/3980401959 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72a806e920 0x7f72a8144440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:32.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.557+0000 7f729fe00640 1 -- 172.21.15.45:0/3980401959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72a81452e0 msgr2=0x7f72a81e7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:32.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.557+0000 7f729fe00640 1 --2- 172.21.15.45:0/3980401959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72a81452e0 0x7f72a81e7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:32.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.557+0000 7f729fe00640 1 -- 172.21.15.45:0/3980401959 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a815c300 con 0x7f72a80cecc0 2024-04-16T16:20:32.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.558+0000 7f729fe00640 1 --2- 172.21.15.45:0/3980401959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72a80cecc0 0x7f72a81e72d0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f729000c340 tx=0x7f729000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:32.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.558+0000 7f729ea00640 1 -- 172.21.15.45:0/3980401959 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7290017070 con 0x7f72a80cecc0 2024-04-16T16:20:32.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.558+0000 7f729ea00640 1 -- 172.21.15.45:0/3980401959 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72900052d0 con 0x7f72a80cecc0 2024-04-16T16:20:32.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.558+0000 7f729ea00640 1 -- 172.21.15.45:0/3980401959 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7290003850 con 0x7f72a80cecc0 2024-04-16T16:20:32.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.558+0000 7f72a7e00640 1 -- 172.21.15.45:0/3980401959 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72a81e89c0 con 0x7f72a80cecc0 2024-04-16T16:20:32.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.559+0000 7f72a7e00640 1 -- 172.21.15.45:0/3980401959 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72a81e8120 con 0x7f72a80cecc0 2024-04-16T16:20:32.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.560+0000 7f72a7e00640 1 -- 172.21.15.45:0/3980401959 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72a8075780 con 0x7f72a80cecc0 2024-04-16T16:20:32.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.562+0000 7f729ea00640 1 -- 172.21.15.45:0/3980401959 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f729001a070 con 0x7f72a80cecc0 2024-04-16T16:20:32.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.563+0000 7f729ea00640 1 --2- 172.21.15.45:0/3980401959 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f728406ca10 0x7f728406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:32.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.563+0000 7f729ea00640 1 -- 172.21.15.45:0/3980401959 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5227+0+0 (secure 0 0 0) 0x7f729008ae50 con 0x7f72a80cecc0 2024-04-16T16:20:32.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.564+0000 7f72a6a00640 1 --2- 172.21.15.45:0/3980401959 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f728406ca10 0x7f728406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:32.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.565+0000 7f72a6a00640 1 --2- 172.21.15.45:0/3980401959 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f728406ca10 0x7f728406eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7298036000 tx=0x7f7298008030 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:32.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.568+0000 7f729ea00640 1 -- 172.21.15.45:0/3980401959 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7290058c60 con 0x7f72a80cecc0 2024-04-16T16:20:32.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:32.772+0000 7f72a7e00640 1 -- 172.21.15.45:0/3980401959 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f72a80b07a0 con 0x7f72a80cecc0 2024-04-16T16:20:33.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.318+0000 7f729ea00640 1 -- 172.21.15.45:0/3980401959 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v126) ==== 170+0+0 (secure 0 0 0) 0x7f729005cf60 con 0x7f72a80cecc0 2024-04-16T16:20:33.317 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2024-04-16T16:20:33.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.321+0000 7f72a7e00640 1 -- 172.21.15.45:0/3980401959 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f728406ca10 msgr2=0x7f728406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:33.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.322+0000 7f72a7e00640 1 --2- 172.21.15.45:0/3980401959 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f728406ca10 0x7f728406eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7298036000 tx=0x7f7298008030 comp rx=0 tx=0).stop 2024-04-16T16:20:33.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.322+0000 7f72a7e00640 1 -- 172.21.15.45:0/3980401959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72a80cecc0 msgr2=0x7f72a81e72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:33.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.322+0000 7f72a7e00640 1 --2- 172.21.15.45:0/3980401959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72a80cecc0 0x7f72a81e72d0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f729000c340 tx=0x7f729000c800 comp rx=0 tx=0).stop 2024-04-16T16:20:33.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.322+0000 7f72a7e00640 1 -- 172.21.15.45:0/3980401959 shutdown_connections 2024-04-16T16:20:33.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.322+0000 7f72a7e00640 1 --2- 172.21.15.45:0/3980401959 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f728406ca10 0x7f728406eed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:33.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.322+0000 7f72a7e00640 1 --2- 172.21.15.45:0/3980401959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72a81452e0 0x7f72a81e7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:33.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.322+0000 7f72a7e00640 1 --2- 172.21.15.45:0/3980401959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72a80cecc0 0x7f72a81e72d0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:33.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.322+0000 7f72a7e00640 1 --2- 172.21.15.45:0/3980401959 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72a806e920 0x7f72a8144440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:33.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.322+0000 7f72a7e00640 1 -- 172.21.15.45:0/3980401959 >> 172.21.15.45:0/3980401959 conn(0x7f72a80b3bb0 msgr2=0x7f72a80ce460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:33.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.322+0000 7f72a7e00640 1 -- 172.21.15.45:0/3980401959 shutdown_connections 2024-04-16T16:20:33.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.322+0000 7f72a7e00640 1 -- 172.21.15.45:0/3980401959 wait complete. 2024-04-16T16:20:33.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-16T16:20:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.553+0000 7f8296000640 1 Processor -- start 2024-04-16T16:20:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.553+0000 7f8296000640 1 -- start start 2024-04-16T16:20:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.554+0000 7f8296000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8290160e90 0x7f82900cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.554+0000 7f8296000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f82900cf980 0x7f829006dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.554+0000 7f8296000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82900c6210 con 0x7f8290145fe0 2024-04-16T16:20:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.554+0000 7f8296000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f82900cfdd0 con 0x7f82900cf980 2024-04-16T16:20:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.554+0000 7f8296000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f82901603b0 con 0x7f8290160e90 2024-04-16T16:20:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.555+0000 7f8295600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f82900cf980 0x7f829006dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.555+0000 7f8295600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f82900cf980 0x7f829006dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44208/0 (socket says 172.21.15.45:44208) 2024-04-16T16:20:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.555+0000 7f8294c00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8290145fe0 0x7f82900c5750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43036/0 (socket says 172.21.15.45:43036) 2024-04-16T16:20:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.555+0000 7f8295600640 1 -- 172.21.15.45:0/3767002517 learned_addr learned my addr 172.21.15.45:0/3767002517 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.555+0000 7f828fe00640 1 --2- 172.21.15.45:0/3767002517 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8290160e90 0x7f82900cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.555+0000 7f8295600640 1 -- 172.21.15.45:0/3767002517 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8290160e90 msgr2=0x7f82900cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.555+0000 7f8295600640 1 --2- 172.21.15.45:0/3767002517 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8290160e90 0x7f82900cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.555+0000 7f8295600640 1 -- 172.21.15.45:0/3767002517 >> v1:172.21.15.45:6789/0 conn(0x7f8290145fe0 legacy=0x7f82900c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.555+0000 7f8295600640 1 -- 172.21.15.45:0/3767002517 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8290160690 con 0x7f82900cf980 2024-04-16T16:20:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.555+0000 7f828fe00640 1 --2- 172.21.15.45:0/3767002517 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8290160e90 0x7f82900cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.556+0000 7f8295600640 1 --2- 172.21.15.45:0/3767002517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82900cf980 0x7f829006dfb0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f8288009e50 tx=0x7f828802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4fcf37841c005f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.557+0000 7f828f400640 1 -- 172.21.15.45:0/3767002517 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828802ca80 con 0x7f82900cf980 2024-04-16T16:20:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.557+0000 7f828f400640 1 -- 172.21.15.45:0/3767002517 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f828802cc20 con 0x7f82900cf980 2024-04-16T16:20:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.557+0000 7f828f400640 1 -- 172.21.15.45:0/3767002517 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82880343f0 con 0x7f82900cf980 2024-04-16T16:20:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.557+0000 7f8296000640 1 -- 172.21.15.45:0/3767002517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82900cf980 msgr2=0x7f829006dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.557+0000 7f8296000640 1 --2- 172.21.15.45:0/3767002517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82900cf980 0x7f829006dfb0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f8288009e50 tx=0x7f828802bee0 comp rx=0 tx=0).stop 2024-04-16T16:20:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.557+0000 7f8296000640 1 -- 172.21.15.45:0/3767002517 shutdown_connections 2024-04-16T16:20:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.557+0000 7f8296000640 1 --2- 172.21.15.45:0/3767002517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82900cf980 0x7f829006dfb0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.557+0000 7f8296000640 1 --2- 172.21.15.45:0/3767002517 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8290160e90 0x7f82900cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.558+0000 7f8296000640 1 -- 172.21.15.45:0/3767002517 >> 172.21.15.45:0/3767002517 conn(0x7f8290067e00 msgr2=0x7f829006b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.558+0000 7f8296000640 1 -- 172.21.15.45:0/3767002517 shutdown_connections 2024-04-16T16:20:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.558+0000 7f8296000640 1 -- 172.21.15.45:0/3767002517 wait complete. 2024-04-16T16:20:33.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.559+0000 7f8296000640 1 Processor -- start 2024-04-16T16:20:33.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.559+0000 7f8296000640 1 -- start start 2024-04-16T16:20:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.560+0000 7f8296000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82900cf980 0x7f82901f3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.560+0000 7f8296000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8290145fe0 0x7f82901f3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.560+0000 7f8296000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8290160e90 0x7f82901eff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.560+0000 7f8296000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f82900c5e60 con 0x7f82900cf980 2024-04-16T16:20:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.560+0000 7f8296000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f829015dae0 con 0x7f8290145fe0 2024-04-16T16:20:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.560+0000 7f8296000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f82901440b0 con 0x7f8290160e90 2024-04-16T16:20:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.561+0000 7f8294c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82900cf980 0x7f82901f3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.561+0000 7f8294c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82900cf980 0x7f82901f3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44214/0 (socket says 172.21.15.45:44214) 2024-04-16T16:20:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.561+0000 7f8294c00640 1 -- 172.21.15.45:0/3269422643 learned_addr learned my addr 172.21.15.45:0/3269422643 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.561+0000 7f8295600640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8290160e90 0x7f82901eff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.561+0000 7f828fe00640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8290145fe0 0x7f82901f3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.561+0000 7f8294c00640 1 -- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8290160e90 msgr2=0x7f82901eff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.561+0000 7f8294c00640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8290160e90 0x7f82901eff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.561+0000 7f8294c00640 1 -- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8290145fe0 msgr2=0x7f82901f3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.561+0000 7f8294c00640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8290145fe0 0x7f82901f3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.561+0000 7f8294c00640 1 -- 172.21.15.45:0/3269422643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82901f1230 con 0x7f82900cf980 2024-04-16T16:20:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.561+0000 7f8295600640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8290160e90 0x7f82901eff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.561+0000 7f828fe00640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8290145fe0 0x7f82901f3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.562+0000 7f8294c00640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82900cf980 0x7f82901f3550 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f828400c400 tx=0x7f828400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.562+0000 7f828cc00640 1 -- 172.21.15.45:0/3269422643 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82840074e0 con 0x7f82900cf980 2024-04-16T16:20:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.562+0000 7f828cc00640 1 -- 172.21.15.45:0/3269422643 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8284007680 con 0x7f82900cf980 2024-04-16T16:20:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.562+0000 7f8296000640 1 -- 172.21.15.45:0/3269422643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82901f1510 con 0x7f82900cf980 2024-04-16T16:20:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.562+0000 7f828cc00640 1 -- 172.21.15.45:0/3269422643 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8284004a00 con 0x7f82900cf980 2024-04-16T16:20:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.562+0000 7f8296000640 1 -- 172.21.15.45:0/3269422643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82901f04a0 con 0x7f82900cf980 2024-04-16T16:20:33.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.563+0000 7f8296000640 1 -- 172.21.15.45:0/3269422643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8290075780 con 0x7f82900cf980 2024-04-16T16:20:33.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.566+0000 7f828cc00640 1 -- 172.21.15.45:0/3269422643 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f828401a020 con 0x7f82900cf980 2024-04-16T16:20:33.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.567+0000 7f828cc00640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f827006ca10 0x7f827006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:33.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.567+0000 7f828cc00640 1 -- 172.21.15.45:0/3269422643 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5609+0+0 (secure 0 0 0) 0x7f828408bd50 con 0x7f82900cf980 2024-04-16T16:20:33.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.568+0000 7f828fe00640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f827006ca10 0x7f827006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:33.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.569+0000 7f828fe00640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f827006ca10 0x7f827006eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f827c006fd0 tx=0x7f827c008040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.571+0000 7f828cc00640 1 -- 172.21.15.45:0/3269422643 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8284021070 con 0x7f82900cf980 2024-04-16T16:20:33.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:33.815+0000 7f8296000640 1 -- 172.21.15.45:0/3269422643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f829006c110 con 0x7f82900cf980 2024-04-16T16:20:34.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.345+0000 7f828cc00640 1 -- 172.21.15.45:0/3269422643 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v87) ==== 174+0+0 (secure 0 0 0) 0x7f828405a020 con 0x7f82900cf980 2024-04-16T16:20:34.346 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 18 and data pool 19 2024-04-16T16:20:34.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.356+0000 7f8296000640 1 -- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f827006ca10 msgr2=0x7f827006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:34.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.356+0000 7f8296000640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f827006ca10 0x7f827006eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f827c006fd0 tx=0x7f827c008040 comp rx=0 tx=0).stop 2024-04-16T16:20:34.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.356+0000 7f8296000640 1 -- 172.21.15.45:0/3269422643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82900cf980 msgr2=0x7f82901f3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:34.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.356+0000 7f8296000640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82900cf980 0x7f82901f3550 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f828400c400 tx=0x7f828400c8c0 comp rx=0 tx=0).stop 2024-04-16T16:20:34.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.356+0000 7f8296000640 1 -- 172.21.15.45:0/3269422643 shutdown_connections 2024-04-16T16:20:34.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.356+0000 7f8296000640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f827006ca10 0x7f827006eed0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:34.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.356+0000 7f8296000640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8290160e90 0x7f82901eff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:34.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.356+0000 7f8296000640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8290145fe0 0x7f82901f3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:34.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.356+0000 7f8296000640 1 --2- 172.21.15.45:0/3269422643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82900cf980 0x7f82901f3550 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:34.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.356+0000 7f8296000640 1 -- 172.21.15.45:0/3269422643 >> 172.21.15.45:0/3269422643 conn(0x7f8290067e00 msgr2=0x7f82900b4940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:34.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.358+0000 7f8296000640 1 -- 172.21.15.45:0/3269422643 shutdown_connections 2024-04-16T16:20:34.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.358+0000 7f8296000640 1 -- 172.21.15.45:0/3269422643 wait complete. 2024-04-16T16:20:34.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:20:34.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.569+0000 7f9e23400640 1 Processor -- start 2024-04-16T16:20:34.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.569+0000 7f9e23400640 1 -- start start 2024-04-16T16:20:34.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.571+0000 7f9e23400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c072c50 0x7f9e1c06e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:34.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.571+0000 7f9e23400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e1c15a440 con 0x7f9e1c15a060 2024-04-16T16:20:34.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.571+0000 7f9e23400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e1c0b3c80 con 0x7f9e1c06f880 2024-04-16T16:20:34.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.571+0000 7f9e23400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9e1c0b3a50 con 0x7f9e1c072c50 2024-04-16T16:20:34.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.571+0000 7f9e20a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9e1c15a060 0x7f9e1c072540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43038/0 (socket says 172.21.15.45:43038) 2024-04-16T16:20:34.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.571+0000 7f9e20a00640 1 -- 172.21.15.45:0/3522999632 learned_addr learned my addr 172.21.15.45:0/3522999632 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:34.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e13e00640 1 --2- 172.21.15.45:0/3522999632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c072c50 0x7f9e1c06e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:34.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e1be00640 1 -- 172.21.15.45:0/3522999632 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4103080483 0 0) 0x7f9e1c15a440 con 0x7f9e1c15a060 2024-04-16T16:20:34.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e1be00640 1 -- 172.21.15.45:0/3522999632 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dfc003520 con 0x7f9e1c15a060 2024-04-16T16:20:34.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e1be00640 1 -- 172.21.15.45:0/3522999632 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2108230491 0 0) 0x7f9dfc003520 con 0x7f9e1c15a060 2024-04-16T16:20:34.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e1be00640 1 -- 172.21.15.45:0/3522999632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c072c50 msgr2=0x7f9e1c06e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:34.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e1be00640 1 --2- 172.21.15.45:0/3522999632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c072c50 0x7f9e1c06e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:34.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e1be00640 1 -- 172.21.15.45:0/3522999632 >> v1:172.21.15.103:6789/0 conn(0x7f9e1c06f880 legacy=0x7f9e1c0b4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:34.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e1be00640 1 -- 172.21.15.45:0/3522999632 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1c0b3f60 con 0x7f9e1c15a060 2024-04-16T16:20:34.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e1be00640 1 -- 172.21.15.45:0/3522999632 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f9e04001670 con 0x7f9e1c15a060 2024-04-16T16:20:34.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e1be00640 1 -- 172.21.15.45:0/3522999632 >> v1:172.21.15.45:6789/0 conn(0x7f9e1c15a060 legacy=0x7f9e1c072540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:34.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e1be00640 1 --2- 172.21.15.45:0/3522999632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dfc0039e0 0x7f9dfc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:34.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e1be00640 1 --2- 172.21.15.45:0/3522999632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dfc004800 0x7f9dfc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:34.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e1be00640 1 --2- 172.21.15.45:0/3522999632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9dfc0052e0 0x7f9dfc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:34.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e1be00640 1 -- 172.21.15.45:0/3522999632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9dfc00bfb0 con 0x7f9dfc0039e0 2024-04-16T16:20:34.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e1be00640 1 -- 172.21.15.45:0/3522999632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9dfc00c130 con 0x7f9dfc004800 2024-04-16T16:20:34.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e1be00640 1 -- 172.21.15.45:0/3522999632 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9dfc00c300 con 0x7f9dfc0052e0 2024-04-16T16:20:34.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.572+0000 7f9e20a00640 1 --2- 172.21.15.45:0/3522999632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dfc0039e0 0x7f9dfc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:34.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.573+0000 7f9e20a00640 1 -- 172.21.15.45:0/3522999632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9dfc0052e0 msgr2=0x7f9dfc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:34.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.573+0000 7f9e13e00640 1 --2- 172.21.15.45:0/3522999632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dfc004800 0x7f9dfc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:34.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.573+0000 7f9e20a00640 1 --2- 172.21.15.45:0/3522999632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9dfc0052e0 0x7f9dfc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:34.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.573+0000 7f9e20a00640 1 -- 172.21.15.45:0/3522999632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dfc004800 msgr2=0x7f9dfc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:34.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.573+0000 7f9e20a00640 1 --2- 172.21.15.45:0/3522999632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dfc004800 0x7f9dfc004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:34.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.573+0000 7f9e20a00640 1 -- 172.21.15.45:0/3522999632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dfc003520 con 0x7f9dfc0039e0 2024-04-16T16:20:34.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.574+0000 7f9e20a00640 1 --2- 172.21.15.45:0/3522999632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dfc0039e0 0x7f9dfc003e00 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400c240 tx=0x7f9dfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d46c0736b28ef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:34.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.574+0000 7f9e1be00640 1 -- 172.21.15.45:0/3522999632 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e04007d10 con 0x7f9dfc0039e0 2024-04-16T16:20:34.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.574+0000 7f9e1be00640 1 -- 172.21.15.45:0/3522999632 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e04030070 con 0x7f9dfc0039e0 2024-04-16T16:20:34.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.574+0000 7f9e1be00640 1 -- 172.21.15.45:0/3522999632 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0402cd60 con 0x7f9dfc0039e0 2024-04-16T16:20:34.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.575+0000 7f9e23400640 1 -- 172.21.15.45:0/3522999632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dfc0039e0 msgr2=0x7f9dfc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:34.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.575+0000 7f9e23400640 1 --2- 172.21.15.45:0/3522999632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dfc0039e0 0x7f9dfc003e00 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400c240 tx=0x7f9dfc005830 comp rx=0 tx=0).stop 2024-04-16T16:20:34.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.575+0000 7f9e21400640 1 -- 172.21.15.45:0/3522999632 reap_dead start 2024-04-16T16:20:34.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.575+0000 7f9e23400640 1 -- 172.21.15.45:0/3522999632 shutdown_connections 2024-04-16T16:20:34.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.575+0000 7f9e23400640 1 -- 172.21.15.45:0/3522999632 >> 172.21.15.45:0/3522999632 conn(0x7f9e1c0b2750 msgr2=0x7f9e1c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:34.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.575+0000 7f9e23400640 1 -- 172.21.15.45:0/3522999632 shutdown_connections 2024-04-16T16:20:34.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.575+0000 7f9e23400640 1 -- 172.21.15.45:0/3522999632 wait complete. 2024-04-16T16:20:34.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.577+0000 7f9e23400640 1 Processor -- start 2024-04-16T16:20:34.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.577+0000 7f9e23400640 1 -- start start 2024-04-16T16:20:34.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.578+0000 7f9e23400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dfc0039e0 0x7f9e1c1fc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:34.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.578+0000 7f9e23400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c0b1570 0x7f9e1c14bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:34.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.578+0000 7f9e23400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e1c1f9160 0x7f9e1c1f95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:34.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.578+0000 7f9e23400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c06e2b0 con 0x7f9e1c1f9160 2024-04-16T16:20:34.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.578+0000 7f9e23400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c0b4520 con 0x7f9dfc0039e0 2024-04-16T16:20:34.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.578+0000 7f9e23400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9e1c06e430 con 0x7f9e1c0b1570 2024-04-16T16:20:34.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.578+0000 7f9e21400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e1c1f9160 0x7f9e1c1f95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:34.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.578+0000 7f9e21400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e1c1f9160 0x7f9e1c1f95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44242/0 (socket says 172.21.15.45:44242) 2024-04-16T16:20:34.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.578+0000 7f9e21400640 1 -- 172.21.15.45:0/3214163611 learned_addr learned my addr 172.21.15.45:0/3214163611 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:34.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.579+0000 7f9e21400640 1 -- 172.21.15.45:0/3214163611 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c0b1570 msgr2=0x7f9e1c14bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:34.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.579+0000 7f9e21400640 1 --2- 172.21.15.45:0/3214163611 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c0b1570 0x7f9e1c14bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:34.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.579+0000 7f9e21400640 1 -- 172.21.15.45:0/3214163611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dfc0039e0 msgr2=0x7f9e1c1fc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:34.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.579+0000 7f9e21400640 1 --2- 172.21.15.45:0/3214163611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dfc0039e0 0x7f9e1c1fc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:34.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.579+0000 7f9e21400640 1 -- 172.21.15.45:0/3214163611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1c1fc720 con 0x7f9e1c1f9160 2024-04-16T16:20:34.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.579+0000 7f9e21400640 1 --2- 172.21.15.45:0/3214163611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e1c1f9160 0x7f9e1c1f95e0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c166a40 tx=0x7f9e1400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:34.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.579+0000 7f9e19600640 1 -- 172.21.15.45:0/3214163611 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e14017070 con 0x7f9e1c1f9160 2024-04-16T16:20:34.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.579+0000 7f9e19600640 1 -- 172.21.15.45:0/3214163611 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e14005d60 con 0x7f9e1c1f9160 2024-04-16T16:20:34.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.579+0000 7f9e19600640 1 -- 172.21.15.45:0/3214163611 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1400a850 con 0x7f9e1c1f9160 2024-04-16T16:20:34.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.579+0000 7f9e23400640 1 -- 172.21.15.45:0/3214163611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e1c1fca00 con 0x7f9e1c1f9160 2024-04-16T16:20:34.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.579+0000 7f9e23400640 1 -- 172.21.15.45:0/3214163611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e1c14c090 con 0x7f9e1c1f9160 2024-04-16T16:20:34.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.581+0000 7f9e23400640 1 -- 172.21.15.45:0/3214163611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e1c166790 con 0x7f9e1c1f9160 2024-04-16T16:20:34.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.582+0000 7f9e19600640 1 -- 172.21.15.45:0/3214163611 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f9e1400a9f0 con 0x7f9e1c1f9160 2024-04-16T16:20:34.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.583+0000 7f9e19600640 1 --2- 172.21.15.45:0/3214163611 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9df806ca10 0x7f9df806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:34.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.583+0000 7f9e19600640 1 -- 172.21.15.45:0/3214163611 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f9e1408af90 con 0x7f9e1c1f9160 2024-04-16T16:20:34.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.583+0000 7f9e20a00640 1 --2- 172.21.15.45:0/3214163611 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9df806ca10 0x7f9df806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:34.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.585+0000 7f9e20a00640 1 --2- 172.21.15.45:0/3214163611 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9df806ca10 0x7f9df806eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e04034000 tx=0x7f9e04039040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:34.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.588+0000 7f9e19600640 1 -- 172.21.15.45:0/3214163611 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9e1c166790 con 0x7f9e1c1f9160 2024-04-16T16:20:34.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.794+0000 7f9e23400640 1 -- 172.21.15.45:0/3214163611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e1c075780 con 0x7f9e1c1f9160 2024-04-16T16:20:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.797+0000 7f9e19600640 1 -- 172.21.15.45:0/3214163611 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) ==== 74+0+16692 (secure 0 0 0) 0x7f9e140592a0 con 0x7f9e1c1f9160 2024-04-16T16:20:34.796 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:34.796 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":127,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:20:33.816509+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:20:31.681227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-16T16:20:32.790980+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:20:34.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.799+0000 7f9e23400640 1 -- 172.21.15.45:0/3214163611 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9df806ca10 msgr2=0x7f9df806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:34.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.799+0000 7f9e23400640 1 --2- 172.21.15.45:0/3214163611 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9df806ca10 0x7f9df806eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e04034000 tx=0x7f9e04039040 comp rx=0 tx=0).stop 2024-04-16T16:20:34.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.799+0000 7f9e23400640 1 -- 172.21.15.45:0/3214163611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e1c1f9160 msgr2=0x7f9e1c1f95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:34.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.799+0000 7f9e23400640 1 --2- 172.21.15.45:0/3214163611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e1c1f9160 0x7f9e1c1f95e0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c166a40 tx=0x7f9e1400cb90 comp rx=0 tx=0).stop 2024-04-16T16:20:34.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.800+0000 7f9e23400640 1 -- 172.21.15.45:0/3214163611 shutdown_connections 2024-04-16T16:20:34.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.800+0000 7f9e23400640 1 --2- 172.21.15.45:0/3214163611 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9df806ca10 0x7f9df806eed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:34.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.800+0000 7f9e23400640 1 --2- 172.21.15.45:0/3214163611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e1c1f9160 0x7f9e1c1f95e0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:34.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.800+0000 7f9e23400640 1 --2- 172.21.15.45:0/3214163611 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c0b1570 0x7f9e1c14bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:34.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.800+0000 7f9e23400640 1 --2- 172.21.15.45:0/3214163611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dfc0039e0 0x7f9e1c1fc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:34.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.800+0000 7f9e23400640 1 -- 172.21.15.45:0/3214163611 >> 172.21.15.45:0/3214163611 conn(0x7f9e1c0b2750 msgr2=0x7f9e1c0ccf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:34.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.800+0000 7f9e23400640 1 -- 172.21.15.45:0/3214163611 shutdown_connections 2024-04-16T16:20:34.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:34.800+0000 7f9e23400640 1 -- 172.21.15.45:0/3214163611 wait complete. 2024-04-16T16:20:34.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:20:35.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.038+0000 7f81aa600640 1 Processor -- start 2024-04-16T16:20:35.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.039+0000 7f81aa600640 1 -- start start 2024-04-16T16:20:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.040+0000 7f81aa600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81a4161e90 0x7f81a40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.040+0000 7f81aa600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81a40d3080 con 0x7f81a4162270 2024-04-16T16:20:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.040+0000 7f81aa600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81a40d3360 con 0x7f81a40d52d0 2024-04-16T16:20:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.040+0000 7f81aa600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f81a40d2e30 con 0x7f81a4161e90 2024-04-16T16:20:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.040+0000 7f81a3e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f81a4162270 0x7f81a40d3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43046/0 (socket says 172.21.15.45:43046) 2024-04-16T16:20:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.040+0000 7f81a3e00640 1 -- 172.21.15.45:0/755714396 learned_addr learned my addr 172.21.15.45:0/755714396 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:35.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.041+0000 7f81a9200640 1 --2- 172.21.15.45:0/755714396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81a4161e90 0x7f81a40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:35.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.041+0000 7f81a3400640 1 -- 172.21.15.45:0/755714396 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3669637055 0 0) 0x7f81a40d3080 con 0x7f81a4162270 2024-04-16T16:20:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.041+0000 7f81a3400640 1 -- 172.21.15.45:0/755714396 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8188003520 con 0x7f81a4162270 2024-04-16T16:20:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.041+0000 7f81a3400640 1 -- 172.21.15.45:0/755714396 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4130058557 0 0) 0x7f8188003520 con 0x7f81a4162270 2024-04-16T16:20:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.041+0000 7f81a3400640 1 -- 172.21.15.45:0/755714396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81a4161e90 msgr2=0x7f81a40cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.041+0000 7f81a3400640 1 --2- 172.21.15.45:0/755714396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81a4161e90 0x7f81a40cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.041+0000 7f81a3400640 1 -- 172.21.15.45:0/755714396 >> v1:172.21.15.103:6789/0 conn(0x7f81a40d52d0 legacy=0x7f81a406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.041+0000 7f81a3400640 1 -- 172.21.15.45:0/755714396 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a40d3640 con 0x7f81a4162270 2024-04-16T16:20:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.041+0000 7f81a3400640 1 -- 172.21.15.45:0/755714396 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f8190001670 con 0x7f81a4162270 2024-04-16T16:20:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.041+0000 7f81a3400640 1 -- 172.21.15.45:0/755714396 >> v1:172.21.15.45:6789/0 conn(0x7f81a4162270 legacy=0x7f81a40d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a3400640 1 --2- 172.21.15.45:0/755714396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81880039e0 0x7f8188003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a3400640 1 --2- 172.21.15.45:0/755714396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8188004800 0x7f8188004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a3400640 1 --2- 172.21.15.45:0/755714396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81a4161e90 0x7f81880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a3400640 1 -- 172.21.15.45:0/755714396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f81a40d2e30 con 0x7f8188004800 2024-04-16T16:20:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a3400640 1 -- 172.21.15.45:0/755714396 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f818800bb00 con 0x7f81880039e0 2024-04-16T16:20:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a3400640 1 -- 172.21.15.45:0/755714396 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f818800bd00 con 0x7f81a4161e90 2024-04-16T16:20:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a3e00640 1 --2- 172.21.15.45:0/755714396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8188004800 0x7f8188004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a9200640 1 --2- 172.21.15.45:0/755714396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81880039e0 0x7f8188003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a9c00640 1 --2- 172.21.15.45:0/755714396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81a4161e90 0x7f81880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a3e00640 1 -- 172.21.15.45:0/755714396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81a4161e90 msgr2=0x7f81880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a3e00640 1 --2- 172.21.15.45:0/755714396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81a4161e90 0x7f81880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a3e00640 1 -- 172.21.15.45:0/755714396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81880039e0 msgr2=0x7f8188003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a3e00640 1 --2- 172.21.15.45:0/755714396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81880039e0 0x7f8188003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a3e00640 1 -- 172.21.15.45:0/755714396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8188003520 con 0x7f8188004800 2024-04-16T16:20:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a9c00640 1 --2- 172.21.15.45:0/755714396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81a4161e90 0x7f81880051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.042+0000 7f81a9200640 1 --2- 172.21.15.45:0/755714396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81880039e0 0x7f8188003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:35.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.044+0000 7f81a3e00640 1 --2- 172.21.15.45:0/755714396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8188004800 0x7f8188004c80 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f8190001eb0 tx=0x7f8188005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d4560c97695766c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:35.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.044+0000 7f81a3400640 1 -- 172.21.15.45:0/755714396 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8190007d30 con 0x7f8188004800 2024-04-16T16:20:35.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.044+0000 7f81a3400640 1 -- 172.21.15.45:0/755714396 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8190030070 con 0x7f8188004800 2024-04-16T16:20:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.044+0000 7f81a3400640 1 -- 172.21.15.45:0/755714396 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8190005d30 con 0x7f8188004800 2024-04-16T16:20:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.044+0000 7f81aa600640 1 -- 172.21.15.45:0/755714396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8188004800 msgr2=0x7f8188004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.044+0000 7f81aa600640 1 --2- 172.21.15.45:0/755714396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8188004800 0x7f8188004c80 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f8190001eb0 tx=0x7f8188005830 comp rx=0 tx=0).stop 2024-04-16T16:20:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.044+0000 7f81a9c00640 1 -- 172.21.15.45:0/755714396 reap_dead start 2024-04-16T16:20:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.045+0000 7f81aa600640 1 -- 172.21.15.45:0/755714396 shutdown_connections 2024-04-16T16:20:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.045+0000 7f81aa600640 1 -- 172.21.15.45:0/755714396 >> 172.21.15.45:0/755714396 conn(0x7f81a40d1940 msgr2=0x7f81a40d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.045+0000 7f81aa600640 1 -- 172.21.15.45:0/755714396 shutdown_connections 2024-04-16T16:20:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.045+0000 7f81aa600640 1 -- 172.21.15.45:0/755714396 wait complete. 2024-04-16T16:20:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.046+0000 7f81aa600640 1 Processor -- start 2024-04-16T16:20:35.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.047+0000 7f81aa600640 1 -- start start 2024-04-16T16:20:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.047+0000 7f81aa600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8188004800 0x7f81a40b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.047+0000 7f81aa600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81a4150500 0x7f81a415c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.047+0000 7f81aa600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81a415cb20 0x7f81a415e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.047+0000 7f81aa600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f81a40d46d0 con 0x7f81a4150500 2024-04-16T16:20:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.047+0000 7f81aa600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f81a40d4550 con 0x7f8188004800 2024-04-16T16:20:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.047+0000 7f81aa600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f81a406d550 con 0x7f81a415cb20 2024-04-16T16:20:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.048+0000 7f81a3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81a4150500 0x7f81a415c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.048+0000 7f81a3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81a4150500 0x7f81a415c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44256/0 (socket says 172.21.15.45:44256) 2024-04-16T16:20:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.048+0000 7f81a3e00640 1 -- 172.21.15.45:0/4217639831 learned_addr learned my addr 172.21.15.45:0/4217639831 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.048+0000 7f81a3e00640 1 -- 172.21.15.45:0/4217639831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81a415cb20 msgr2=0x7f81a415e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.048+0000 7f81a3e00640 1 --2- 172.21.15.45:0/4217639831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81a415cb20 0x7f81a415e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.048+0000 7f81a3e00640 1 -- 172.21.15.45:0/4217639831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8188004800 msgr2=0x7f81a40b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:35.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.048+0000 7f81a3e00640 1 --2- 172.21.15.45:0/4217639831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8188004800 0x7f81a40b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:35.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.048+0000 7f81a3e00640 1 -- 172.21.15.45:0/4217639831 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a415d1c0 con 0x7f81a4150500 2024-04-16T16:20:35.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.048+0000 7f81a3e00640 1 --2- 172.21.15.45:0/4217639831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81a4150500 0x7f81a415c5e0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f8190001970 tx=0x7f8190001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:35.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.049+0000 7f81a0c00640 1 -- 172.21.15.45:0/4217639831 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8190004bf0 con 0x7f81a4150500 2024-04-16T16:20:35.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.049+0000 7f81a0c00640 1 -- 172.21.15.45:0/4217639831 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8190004d90 con 0x7f81a4150500 2024-04-16T16:20:35.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.049+0000 7f81aa600640 1 -- 172.21.15.45:0/4217639831 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81a415d4a0 con 0x7f81a4150500 2024-04-16T16:20:35.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.049+0000 7f81a0c00640 1 -- 172.21.15.45:0/4217639831 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8190007270 con 0x7f81a4150500 2024-04-16T16:20:35.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.049+0000 7f81aa600640 1 -- 172.21.15.45:0/4217639831 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81a415da60 con 0x7f81a4150500 2024-04-16T16:20:35.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.050+0000 7f81aa600640 1 -- 172.21.15.45:0/4217639831 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81a415cf40 con 0x7f81a4150500 2024-04-16T16:20:35.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.051+0000 7f81a0c00640 1 -- 172.21.15.45:0/4217639831 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f8190007410 con 0x7f81a4150500 2024-04-16T16:20:35.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.052+0000 7f81a0c00640 1 --2- 172.21.15.45:0/4217639831 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f818406ca10 0x7f818406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:35.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.052+0000 7f81a0c00640 1 -- 172.21.15.45:0/4217639831 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f81900ad210 con 0x7f81a4150500 2024-04-16T16:20:35.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.053+0000 7f81a9200640 1 --2- 172.21.15.45:0/4217639831 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f818406ca10 0x7f818406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:35.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.054+0000 7f81a9200640 1 --2- 172.21.15.45:0/4217639831 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f818406ca10 0x7f818406eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8194007460 tx=0x7f8194009750 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.056+0000 7f81a0c00640 1 -- 172.21.15.45:0/4217639831 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f81a415cf40 con 0x7f81a4150500 2024-04-16T16:20:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.272+0000 7f81aa600640 1 -- 172.21.15.45:0/4217639831 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f81a4075780 con 0x7f81a4150500 2024-04-16T16:20:35.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.273+0000 7f81a0c00640 1 -- 172.21.15.45:0/4217639831 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) ==== 74+0+16692 (secure 0 0 0) 0x7f819007b520 con 0x7f81a4150500 2024-04-16T16:20:35.273 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:35.273 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":127,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:20:33.816509+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:20:31.681227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-16T16:20:32.790980+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:20:35.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.276+0000 7f81aa600640 1 -- 172.21.15.45:0/4217639831 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f818406ca10 msgr2=0x7f818406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:35.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.276+0000 7f81aa600640 1 --2- 172.21.15.45:0/4217639831 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f818406ca10 0x7f818406eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8194007460 tx=0x7f8194009750 comp rx=0 tx=0).stop 2024-04-16T16:20:35.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.276+0000 7f81aa600640 1 -- 172.21.15.45:0/4217639831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81a4150500 msgr2=0x7f81a415c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:35.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.276+0000 7f81aa600640 1 --2- 172.21.15.45:0/4217639831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81a4150500 0x7f81a415c5e0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f8190001970 tx=0x7f8190001a40 comp rx=0 tx=0).stop 2024-04-16T16:20:35.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.276+0000 7f81aa600640 1 -- 172.21.15.45:0/4217639831 shutdown_connections 2024-04-16T16:20:35.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.276+0000 7f81aa600640 1 --2- 172.21.15.45:0/4217639831 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f818406ca10 0x7f818406eed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:35.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.276+0000 7f81aa600640 1 --2- 172.21.15.45:0/4217639831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81a415cb20 0x7f81a415e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:35.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.276+0000 7f81aa600640 1 --2- 172.21.15.45:0/4217639831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81a4150500 0x7f81a415c5e0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:35.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.276+0000 7f81aa600640 1 --2- 172.21.15.45:0/4217639831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8188004800 0x7f81a40b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:35.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.276+0000 7f81aa600640 1 -- 172.21.15.45:0/4217639831 >> 172.21.15.45:0/4217639831 conn(0x7f81a40d1940 msgr2=0x7f81a40b82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:35.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.277+0000 7f81aa600640 1 -- 172.21.15.45:0/4217639831 shutdown_connections 2024-04-16T16:20:35.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.277+0000 7f81aa600640 1 -- 172.21.15.45:0/4217639831 wait complete. 2024-04-16T16:20:35.294 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-16T16:20:35.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.510+0000 7fe2a1400640 1 Processor -- start 2024-04-16T16:20:35.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.510+0000 7fe2a1400640 1 -- start start 2024-04-16T16:20:35.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.512+0000 7fe2a1400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe29c148190 0x7fe29c0b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:35.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.512+0000 7fe2a1400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe29c0ced20 0x7fe29c160040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:35.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.512+0000 7fe2a1400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe29c148570 con 0x7fe29c0b5e30 2024-04-16T16:20:35.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.512+0000 7fe2a1400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe29c0b6290 con 0x7fe29c0ced20 2024-04-16T16:20:35.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.512+0000 7fe2a1400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe29c15fbb0 con 0x7fe29c148190 2024-04-16T16:20:35.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.512+0000 7fe293400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe29c0b5e30 0x7fe29c0ce610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43060/0 (socket says 172.21.15.45:43060) 2024-04-16T16:20:35.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.512+0000 7fe293400640 1 -- 172.21.15.45:0/2814486829 learned_addr learned my addr 172.21.15.45:0/2814486829 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:35.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.512+0000 7fe2a0a00640 1 --2- 172.21.15.45:0/2814486829 >> v2:172.21.15.45:3300/0 conn(0x7fe29c0ced20 0x7fe29c160040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:35.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.513+0000 7fe29be00640 1 --2- 172.21.15.45:0/2814486829 >> v2:172.21.15.103:3300/0 conn(0x7fe29c148190 0x7fe29c0b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:35.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.513+0000 7fe29b400640 1 -- 172.21.15.45:0/2814486829 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 802929761 0 0) 0x7fe29c148570 con 0x7fe29c0b5e30 2024-04-16T16:20:35.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.513+0000 7fe29b400640 1 -- 172.21.15.45:0/2814486829 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe280003520 con 0x7fe29c0b5e30 2024-04-16T16:20:35.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.513+0000 7fe2a0a00640 1 -- 172.21.15.45:0/2814486829 >> v2:172.21.15.103:3300/0 conn(0x7fe29c148190 msgr2=0x7fe29c0b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:35.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.513+0000 7fe2a0a00640 1 --2- 172.21.15.45:0/2814486829 >> v2:172.21.15.103:3300/0 conn(0x7fe29c148190 0x7fe29c0b58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:35.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.513+0000 7fe2a0a00640 1 -- 172.21.15.45:0/2814486829 >> v1:172.21.15.45:6789/0 conn(0x7fe29c0b5e30 legacy=0x7fe29c0ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:35.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.513+0000 7fe2a0a00640 1 -- 172.21.15.45:0/2814486829 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c15efa0 con 0x7fe29c0ced20 2024-04-16T16:20:35.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.514+0000 7fe2a0a00640 1 --2- 172.21.15.45:0/2814486829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29c0ced20 0x7fe29c160040 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fe29c06e0f0 tx=0x7fe29402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24a1ea5c71bbbd1d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:35.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.515+0000 7fe29b400640 1 -- 172.21.15.45:0/2814486829 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe294039070 con 0x7fe29c0ced20 2024-04-16T16:20:35.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.515+0000 7fe29b400640 1 -- 172.21.15.45:0/2814486829 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe29402cc80 con 0x7fe29c0ced20 2024-04-16T16:20:35.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.515+0000 7fe29b400640 1 -- 172.21.15.45:0/2814486829 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe294034660 con 0x7fe29c0ced20 2024-04-16T16:20:35.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.515+0000 7fe2a1400640 1 -- 172.21.15.45:0/2814486829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29c0ced20 msgr2=0x7fe29c160040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:35.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.515+0000 7fe2a1400640 1 --2- 172.21.15.45:0/2814486829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29c0ced20 0x7fe29c160040 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fe29c06e0f0 tx=0x7fe29402bed0 comp rx=0 tx=0).stop 2024-04-16T16:20:35.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.515+0000 7fe2a1400640 1 -- 172.21.15.45:0/2814486829 shutdown_connections 2024-04-16T16:20:35.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.515+0000 7fe2a1400640 1 --2- 172.21.15.45:0/2814486829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29c0ced20 0x7fe29c160040 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:35.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.515+0000 7fe2a1400640 1 --2- 172.21.15.45:0/2814486829 >> v2:172.21.15.103:3300/0 conn(0x7fe29c148190 0x7fe29c0b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:35.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.515+0000 7fe2a1400640 1 -- 172.21.15.45:0/2814486829 >> 172.21.15.45:0/2814486829 conn(0x7fe29c06b930 msgr2=0x7fe29c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:35.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.515+0000 7fe2a1400640 1 -- 172.21.15.45:0/2814486829 shutdown_connections 2024-04-16T16:20:35.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.516+0000 7fe2a1400640 1 -- 172.21.15.45:0/2814486829 wait complete. 2024-04-16T16:20:35.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.517+0000 7fe2a1400640 1 Processor -- start 2024-04-16T16:20:35.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.517+0000 7fe2a1400640 1 -- start start 2024-04-16T16:20:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.518+0000 7fe2a1400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe29c0b5e30 0x7fe29c0ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.518+0000 7fe2a1400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe29c0ced20 0x7fe29c1f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.518+0000 7fe2a1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29c148190 0x7fe29c1612f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.518+0000 7fe2a1400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c15f840 con 0x7fe29c148190 2024-04-16T16:20:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.518+0000 7fe2a1400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c15f560 con 0x7fe29c0b5e30 2024-04-16T16:20:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.518+0000 7fe2a1400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe29c160580 con 0x7fe29c0ced20 2024-04-16T16:20:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.519+0000 7fe2a0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29c148190 0x7fe29c1612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.519+0000 7fe2a0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29c148190 0x7fe29c1612f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44280/0 (socket says 172.21.15.45:44280) 2024-04-16T16:20:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.519+0000 7fe2a0a00640 1 -- 172.21.15.45:0/1421092746 learned_addr learned my addr 172.21.15.45:0/1421092746 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:35.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.519+0000 7fe293400640 1 --2- 172.21.15.45:0/1421092746 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe29c0ced20 0x7fe29c1f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:35.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.519+0000 7fe29be00640 1 --2- 172.21.15.45:0/1421092746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe29c0b5e30 0x7fe29c0ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:35.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.519+0000 7fe2a0a00640 1 -- 172.21.15.45:0/1421092746 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe29c0ced20 msgr2=0x7fe29c1f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:35.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.519+0000 7fe2a0a00640 1 --2- 172.21.15.45:0/1421092746 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe29c0ced20 0x7fe29c1f4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:35.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.519+0000 7fe2a0a00640 1 -- 172.21.15.45:0/1421092746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe29c0b5e30 msgr2=0x7fe29c0ca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:35.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.519+0000 7fe2a0a00640 1 --2- 172.21.15.45:0/1421092746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe29c0b5e30 0x7fe29c0ca000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:35.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.519+0000 7fe2a0a00640 1 -- 172.21.15.45:0/1421092746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c13e2d0 con 0x7fe29c148190 2024-04-16T16:20:35.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.520+0000 7fe2a0a00640 1 --2- 172.21.15.45:0/1421092746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29c148190 0x7fe29c1612f0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fe29402c4e0 tx=0x7fe294004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:35.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.520+0000 7fe298c00640 1 -- 172.21.15.45:0/1421092746 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe294039050 con 0x7fe29c148190 2024-04-16T16:20:35.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.520+0000 7fe298c00640 1 -- 172.21.15.45:0/1421092746 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe294045070 con 0x7fe29c148190 2024-04-16T16:20:35.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.520+0000 7fe298c00640 1 -- 172.21.15.45:0/1421092746 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe294009450 con 0x7fe29c148190 2024-04-16T16:20:35.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.520+0000 7fe2a1400640 1 -- 172.21.15.45:0/1421092746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe29c1f4cc0 con 0x7fe29c148190 2024-04-16T16:20:35.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.520+0000 7fe2a1400640 1 -- 172.21.15.45:0/1421092746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe29c1f4960 con 0x7fe29c148190 2024-04-16T16:20:35.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.521+0000 7fe2a1400640 1 -- 172.21.15.45:0/1421092746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe29c075780 con 0x7fe29c148190 2024-04-16T16:20:35.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.522+0000 7fe298c00640 1 -- 172.21.15.45:0/1421092746 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7fe29403e850 con 0x7fe29c148190 2024-04-16T16:20:35.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.523+0000 7fe298c00640 1 --2- 172.21.15.45:0/1421092746 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe27806ca10 0x7fe27806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:35.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.523+0000 7fe298c00640 1 -- 172.21.15.45:0/1421092746 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7fe2940aea50 con 0x7fe29c148190 2024-04-16T16:20:35.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.524+0000 7fe29be00640 1 --2- 172.21.15.45:0/1421092746 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe27806ca10 0x7fe27806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:35.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.525+0000 7fe29be00640 1 --2- 172.21.15.45:0/1421092746 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe27806ca10 0x7fe27806eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe28c006fd0 tx=0x7fe28c008040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:35.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.527+0000 7fe298c00640 1 -- 172.21.15.45:0/1421092746 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe294032e20 con 0x7fe29c148190 2024-04-16T16:20:35.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:35.771+0000 7fe2a1400640 1 -- 172.21.15.45:0/1421092746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fe29c0b6590 con 0x7fe29c148190 2024-04-16T16:20:36.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.365+0000 7fe298c00640 1 -- 172.21.15.45:0/1421092746 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v90) ==== 118+0+0 (secure 0 0 0) 0x7fe29407ccd0 con 0x7fe29c148190 2024-04-16T16:20:36.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.369+0000 7fe2a1400640 1 -- 172.21.15.45:0/1421092746 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe27806ca10 msgr2=0x7fe27806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:36.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.369+0000 7fe2a1400640 1 --2- 172.21.15.45:0/1421092746 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe27806ca10 0x7fe27806eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe28c006fd0 tx=0x7fe28c008040 comp rx=0 tx=0).stop 2024-04-16T16:20:36.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.369+0000 7fe2a1400640 1 -- 172.21.15.45:0/1421092746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29c148190 msgr2=0x7fe29c1612f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:36.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.369+0000 7fe2a1400640 1 --2- 172.21.15.45:0/1421092746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29c148190 0x7fe29c1612f0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fe29402c4e0 tx=0x7fe294004a40 comp rx=0 tx=0).stop 2024-04-16T16:20:36.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.369+0000 7fe2a1400640 1 -- 172.21.15.45:0/1421092746 shutdown_connections 2024-04-16T16:20:36.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.369+0000 7fe2a1400640 1 --2- 172.21.15.45:0/1421092746 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fe27806ca10 0x7fe27806eed0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:36.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.369+0000 7fe2a1400640 1 --2- 172.21.15.45:0/1421092746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29c148190 0x7fe29c1612f0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:36.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.369+0000 7fe2a1400640 1 --2- 172.21.15.45:0/1421092746 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe29c0ced20 0x7fe29c1f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:36.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.369+0000 7fe2a1400640 1 --2- 172.21.15.45:0/1421092746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe29c0b5e30 0x7fe29c0ca000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:36.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.369+0000 7fe2a1400640 1 -- 172.21.15.45:0/1421092746 >> 172.21.15.45:0/1421092746 conn(0x7fe29c06b930 msgr2=0x7fe29c0b47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:36.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.369+0000 7fe2a1400640 1 -- 172.21.15.45:0/1421092746 shutdown_connections 2024-04-16T16:20:36.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.369+0000 7fe2a1400640 1 -- 172.21.15.45:0/1421092746 wait complete. 2024-04-16T16:20:36.392 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-16T16:20:36.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-16T16:20:36.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.619+0000 7f7d54e00640 1 Processor -- start 2024-04-16T16:20:36.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.620+0000 7f7d54e00640 1 -- start start 2024-04-16T16:20:36.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.621+0000 7f7d54e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d500d18f0 0x7f7d500ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:36.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.621+0000 7f7d54e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7d500d1cd0 0x7f7d50162dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:36.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.621+0000 7f7d54e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d500b6810 con 0x7f7d500b6310 2024-04-16T16:20:36.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.621+0000 7f7d54e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7d500d29f0 con 0x7f7d500d1cd0 2024-04-16T16:20:36.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.621+0000 7f7d54e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7d500b6af0 con 0x7f7d500d18f0 2024-04-16T16:20:36.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.622+0000 7f7d4f400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d500d18f0 0x7f7d500ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:36.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.622+0000 7f7d4fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7d500d1cd0 0x7f7d50162dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:36.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.622+0000 7f7d47e00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7d500b6310 0x7f7d500d2e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:51902/0 (socket says 172.21.15.45:51902) 2024-04-16T16:20:36.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.622+0000 7f7d47e00640 1 -- 172.21.15.45:0/1751818073 learned_addr learned my addr 172.21.15.45:0/1751818073 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:36.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.623+0000 7f7d4f400640 1 -- 172.21.15.45:0/1751818073 >> v2:172.21.15.103:3300/0 conn(0x7f7d500d1cd0 msgr2=0x7f7d50162dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:36.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.623+0000 7f7d4f400640 1 --2- 172.21.15.45:0/1751818073 >> v2:172.21.15.103:3300/0 conn(0x7f7d500d1cd0 0x7f7d50162dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:36.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.623+0000 7f7d4f400640 1 -- 172.21.15.45:0/1751818073 >> v1:172.21.15.103:6789/0 conn(0x7f7d500b6310 legacy=0x7f7d500d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:36.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.623+0000 7f7d4f400640 1 -- 172.21.15.45:0/1751818073 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d500b6dd0 con 0x7f7d500d18f0 2024-04-16T16:20:36.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.623+0000 7f7d4fe00640 1 --2- 172.21.15.45:0/1751818073 >> v2:172.21.15.103:3300/0 conn(0x7f7d500d1cd0 0x7f7d50162dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:20:36.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.624+0000 7f7d4f400640 1 --2- 172.21.15.45:0/1751818073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d500d18f0 0x7f7d500ba220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7d400097a0 tx=0x7f7d4002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f418a7662ccd6200 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:36.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.625+0000 7f7d4ea00640 1 -- 172.21.15.45:0/1751818073 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40031780 con 0x7f7d500d18f0 2024-04-16T16:20:36.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.625+0000 7f7d4ea00640 1 -- 172.21.15.45:0/1751818073 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d40037070 con 0x7f7d500d18f0 2024-04-16T16:20:36.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.625+0000 7f7d4ea00640 1 -- 172.21.15.45:0/1751818073 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40031b40 con 0x7f7d500d18f0 2024-04-16T16:20:36.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.626+0000 7f7d54e00640 1 -- 172.21.15.45:0/1751818073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d500d18f0 msgr2=0x7f7d500ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:36.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.626+0000 7f7d54e00640 1 --2- 172.21.15.45:0/1751818073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d500d18f0 0x7f7d500ba220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7d400097a0 tx=0x7f7d4002b820 comp rx=0 tx=0).stop 2024-04-16T16:20:36.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.626+0000 7f7d54e00640 1 -- 172.21.15.45:0/1751818073 shutdown_connections 2024-04-16T16:20:36.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.626+0000 7f7d54e00640 1 --2- 172.21.15.45:0/1751818073 >> v2:172.21.15.103:3300/0 conn(0x7f7d500d1cd0 0x7f7d50162dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:36.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.626+0000 7f7d54e00640 1 --2- 172.21.15.45:0/1751818073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d500d18f0 0x7f7d500ba220 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:36.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.626+0000 7f7d54e00640 1 -- 172.21.15.45:0/1751818073 >> 172.21.15.45:0/1751818073 conn(0x7f7d5006cf90 msgr2=0x7f7d5006d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:36.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.626+0000 7f7d54e00640 1 -- 172.21.15.45:0/1751818073 shutdown_connections 2024-04-16T16:20:36.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.626+0000 7f7d54e00640 1 -- 172.21.15.45:0/1751818073 wait complete. 2024-04-16T16:20:36.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.628+0000 7f7d54e00640 1 Processor -- start 2024-04-16T16:20:36.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.628+0000 7f7d54e00640 1 -- start start 2024-04-16T16:20:36.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.629+0000 7f7d54e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d500b6310 0x7f7d500ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:36.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.629+0000 7f7d54e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d500d18f0 0x7f7d501f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:36.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.629+0000 7f7d54e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d500d1cd0 0x7f7d50155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:36.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.629+0000 7f7d54e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7d50163300 con 0x7f7d500d18f0 2024-04-16T16:20:36.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.629+0000 7f7d54e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7d500ba760 con 0x7f7d500d1cd0 2024-04-16T16:20:36.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.629+0000 7f7d54e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7d501f7a60 con 0x7f7d500b6310 2024-04-16T16:20:36.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.629+0000 7f7d47e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d500d18f0 0x7f7d501f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:36.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.629+0000 7f7d47e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d500d18f0 0x7f7d501f4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44292/0 (socket says 172.21.15.45:44292) 2024-04-16T16:20:36.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.629+0000 7f7d47e00640 1 -- 172.21.15.45:0/2072400307 learned_addr learned my addr 172.21.15.45:0/2072400307 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:36.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.630+0000 7f7d4fe00640 1 --2- 172.21.15.45:0/2072400307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d500d1cd0 0x7f7d50155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:36.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.630+0000 7f7d4f400640 1 --2- 172.21.15.45:0/2072400307 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d500b6310 0x7f7d500ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:36.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.630+0000 7f7d47e00640 1 -- 172.21.15.45:0/2072400307 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d500b6310 msgr2=0x7f7d500ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:36.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.630+0000 7f7d47e00640 1 --2- 172.21.15.45:0/2072400307 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d500b6310 0x7f7d500ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:36.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.630+0000 7f7d47e00640 1 -- 172.21.15.45:0/2072400307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d500d1cd0 msgr2=0x7f7d50155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:36.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.630+0000 7f7d47e00640 1 --2- 172.21.15.45:0/2072400307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d500d1cd0 0x7f7d50155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:36.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.630+0000 7f7d47e00640 1 -- 172.21.15.45:0/2072400307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d501f4ca0 con 0x7f7d500d18f0 2024-04-16T16:20:36.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.630+0000 7f7d4f400640 1 --2- 172.21.15.45:0/2072400307 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d500b6310 0x7f7d500ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:36.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.630+0000 7f7d47e00640 1 --2- 172.21.15.45:0/2072400307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d500d18f0 0x7f7d501f4470 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f7d3800c400 tx=0x7f7d3800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:36.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.631+0000 7f7d47400640 1 -- 172.21.15.45:0/2072400307 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d38017070 con 0x7f7d500d18f0 2024-04-16T16:20:36.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.631+0000 7f7d47400640 1 -- 172.21.15.45:0/2072400307 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d38005ce0 con 0x7f7d500d18f0 2024-04-16T16:20:36.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.631+0000 7f7d47400640 1 -- 172.21.15.45:0/2072400307 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d380072f0 con 0x7f7d500d18f0 2024-04-16T16:20:36.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.631+0000 7f7d54e00640 1 -- 172.21.15.45:0/2072400307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d501f4f80 con 0x7f7d500d18f0 2024-04-16T16:20:36.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.631+0000 7f7d54e00640 1 -- 172.21.15.45:0/2072400307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d501f7d00 con 0x7f7d500d18f0 2024-04-16T16:20:36.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.632+0000 7f7d54e00640 1 -- 172.21.15.45:0/2072400307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d50075780 con 0x7f7d500d18f0 2024-04-16T16:20:36.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.633+0000 7f7d47400640 1 -- 172.21.15.45:0/2072400307 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f7d3801a070 con 0x7f7d500d18f0 2024-04-16T16:20:36.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.634+0000 7f7d47400640 1 --2- 172.21.15.45:0/2072400307 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f7d2c06ca10 0x7f7d2c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:36.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.634+0000 7f7d47400640 1 -- 172.21.15.45:0/2072400307 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7f7d3808b470 con 0x7f7d500d18f0 2024-04-16T16:20:36.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.635+0000 7f7d4f400640 1 --2- 172.21.15.45:0/2072400307 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f7d2c06ca10 0x7f7d2c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:36.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.636+0000 7f7d4f400640 1 --2- 172.21.15.45:0/2072400307 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f7d2c06ca10 0x7f7d2c06eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d40036000 tx=0x7f7d40008030 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:36.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.638+0000 7f7d47400640 1 -- 172.21.15.45:0/2072400307 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7d38021020 con 0x7f7d500d18f0 2024-04-16T16:20:36.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:36.890+0000 7f7d54e00640 1 -- 172.21.15.45:0/2072400307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f7d500d25a0 con 0x7f7d500d18f0 2024-04-16T16:20:37.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.368+0000 7f7d47400640 1 -- 172.21.15.45:0/2072400307 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v91) ==== 122+0+0 (secure 0 0 0) 0x7f7d38059870 con 0x7f7d500d18f0 2024-04-16T16:20:37.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.372+0000 7f7d54e00640 1 -- 172.21.15.45:0/2072400307 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f7d2c06ca10 msgr2=0x7f7d2c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:37.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.372+0000 7f7d54e00640 1 --2- 172.21.15.45:0/2072400307 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f7d2c06ca10 0x7f7d2c06eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d40036000 tx=0x7f7d40008030 comp rx=0 tx=0).stop 2024-04-16T16:20:37.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.372+0000 7f7d54e00640 1 -- 172.21.15.45:0/2072400307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d500d18f0 msgr2=0x7f7d501f4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:37.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.372+0000 7f7d54e00640 1 --2- 172.21.15.45:0/2072400307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d500d18f0 0x7f7d501f4470 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f7d3800c400 tx=0x7f7d3800c8c0 comp rx=0 tx=0).stop 2024-04-16T16:20:37.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.372+0000 7f7d54e00640 1 -- 172.21.15.45:0/2072400307 shutdown_connections 2024-04-16T16:20:37.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.372+0000 7f7d54e00640 1 --2- 172.21.15.45:0/2072400307 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f7d2c06ca10 0x7f7d2c06eed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:37.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.372+0000 7f7d54e00640 1 --2- 172.21.15.45:0/2072400307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d500d1cd0 0x7f7d50155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:37.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.372+0000 7f7d54e00640 1 --2- 172.21.15.45:0/2072400307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d500d18f0 0x7f7d501f4470 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:37.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.372+0000 7f7d54e00640 1 --2- 172.21.15.45:0/2072400307 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d500b6310 0x7f7d500ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:37.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.372+0000 7f7d54e00640 1 -- 172.21.15.45:0/2072400307 >> 172.21.15.45:0/2072400307 conn(0x7f7d5006cf90 msgr2=0x7f7d500b4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:37.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.372+0000 7f7d54e00640 1 -- 172.21.15.45:0/2072400307 shutdown_connections 2024-04-16T16:20:37.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.372+0000 7f7d54e00640 1 -- 172.21.15.45:0/2072400307 wait complete. 2024-04-16T16:20:37.395 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-16T16:20:37.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:20:37.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.619+0000 7f70a0600640 1 Processor -- start 2024-04-16T16:20:37.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.619+0000 7f70a0600640 1 -- start start 2024-04-16T16:20:37.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.621+0000 7f70a0600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f709815ecc0 0x7f709806fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:37.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.621+0000 7f70a0600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f709806f490 con 0x7f709806c040 2024-04-16T16:20:37.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.621+0000 7f70a0600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f709806f730 con 0x7f709809af60 2024-04-16T16:20:37.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.621+0000 7f70a0600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f70980b3f60 con 0x7f709815ecc0 2024-04-16T16:20:37.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.621+0000 7f709e600640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f709806c040 0x7f70980cf580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43066/0 (socket says 172.21.15.45:43066) 2024-04-16T16:20:37.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.621+0000 7f709e600640 1 -- 172.21.15.45:0/156306873 learned_addr learned my addr 172.21.15.45:0/156306873 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:37.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.622+0000 7f709dc00640 1 --2- 172.21.15.45:0/156306873 >> v2:172.21.15.103:3300/0 conn(0x7f709815ecc0 0x7f709806fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.622+0000 7f7097e00640 1 -- 172.21.15.45:0/156306873 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3600693030 0 0) 0x7f709806f490 con 0x7f709806c040 2024-04-16T16:20:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.622+0000 7f7097e00640 1 -- 172.21.15.45:0/156306873 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7080003520 con 0x7f709806c040 2024-04-16T16:20:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.622+0000 7f7097e00640 1 -- 172.21.15.45:0/156306873 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1773188190 0 0) 0x7f7080003520 con 0x7f709806c040 2024-04-16T16:20:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.622+0000 7f7097e00640 1 -- 172.21.15.45:0/156306873 >> v2:172.21.15.103:3300/0 conn(0x7f709815ecc0 msgr2=0x7f709806fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.622+0000 7f7097e00640 1 --2- 172.21.15.45:0/156306873 >> v2:172.21.15.103:3300/0 conn(0x7f709815ecc0 0x7f709806fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.622+0000 7f7097e00640 1 -- 172.21.15.45:0/156306873 >> v1:172.21.15.103:6789/0 conn(0x7f709809af60 legacy=0x7f709806b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.622+0000 7f7097e00640 1 -- 172.21.15.45:0/156306873 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70980b3bf0 con 0x7f709806c040 2024-04-16T16:20:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.622+0000 7f7097e00640 1 -- 172.21.15.45:0/156306873 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f70900019f0 con 0x7f709806c040 2024-04-16T16:20:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.622+0000 7f7097e00640 1 -- 172.21.15.45:0/156306873 >> v1:172.21.15.45:6789/0 conn(0x7f709806c040 legacy=0x7f70980cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.623+0000 7f7097e00640 1 --2- 172.21.15.45:0/156306873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70800039b0 0x7f7080003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.623+0000 7f7097e00640 1 --2- 172.21.15.45:0/156306873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7080004800 0x7f7080004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.623+0000 7f7097e00640 1 --2- 172.21.15.45:0/156306873 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70800051c0 0x7f708000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.623+0000 7f7097e00640 1 -- 172.21.15.45:0/156306873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f70980b3f60 con 0x7f70800039b0 2024-04-16T16:20:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.623+0000 7f7097e00640 1 -- 172.21.15.45:0/156306873 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f708000c160 con 0x7f7080004800 2024-04-16T16:20:37.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.623+0000 7f7097e00640 1 -- 172.21.15.45:0/156306873 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f708000c360 con 0x7f70800051c0 2024-04-16T16:20:37.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.623+0000 7f709dc00640 1 --2- 172.21.15.45:0/156306873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70800039b0 0x7f7080003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:37.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.623+0000 7f709dc00640 1 -- 172.21.15.45:0/156306873 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70800051c0 msgr2=0x7f708000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:37.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.623+0000 7f709dc00640 1 --2- 172.21.15.45:0/156306873 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70800051c0 0x7f708000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:37.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.623+0000 7f709dc00640 1 -- 172.21.15.45:0/156306873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7080004800 msgr2=0x7f7080004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:37.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.623+0000 7f709dc00640 1 --2- 172.21.15.45:0/156306873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7080004800 0x7f7080004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:37.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.623+0000 7f709dc00640 1 -- 172.21.15.45:0/156306873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7080003520 con 0x7f70800039b0 2024-04-16T16:20:37.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.623+0000 7f709e600640 1 -- 172.21.15.45:0/156306873 reap_dead start 2024-04-16T16:20:37.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.624+0000 7f709dc00640 1 --2- 172.21.15.45:0/156306873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70800039b0 0x7f7080003dd0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f708c009ac0 tx=0x7f7080005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c020587e96a5a5a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:37.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.625+0000 7f7097e00640 1 -- 172.21.15.45:0/156306873 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c039070 con 0x7f70800039b0 2024-04-16T16:20:37.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.625+0000 7f7097e00640 1 -- 172.21.15.45:0/156306873 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708c02c6d0 con 0x7f70800039b0 2024-04-16T16:20:37.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.625+0000 7f7097e00640 1 -- 172.21.15.45:0/156306873 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c009690 con 0x7f70800039b0 2024-04-16T16:20:37.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.625+0000 7f70a0600640 1 -- 172.21.15.45:0/156306873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70800039b0 msgr2=0x7f7080003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:37.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.625+0000 7f70a0600640 1 --2- 172.21.15.45:0/156306873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70800039b0 0x7f7080003dd0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f708c009ac0 tx=0x7f7080005830 comp rx=0 tx=0).stop 2024-04-16T16:20:37.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.626+0000 7f70a0600640 1 -- 172.21.15.45:0/156306873 shutdown_connections 2024-04-16T16:20:37.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.626+0000 7f70a0600640 1 --2- 172.21.15.45:0/156306873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70800039b0 0x7f7080003dd0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:37.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.626+0000 7f70a0600640 1 -- 172.21.15.45:0/156306873 >> 172.21.15.45:0/156306873 conn(0x7f709809aa70 msgr2=0x7f70980b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:37.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.626+0000 7f70a0600640 1 -- 172.21.15.45:0/156306873 shutdown_connections 2024-04-16T16:20:37.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.626+0000 7f70a0600640 1 -- 172.21.15.45:0/156306873 wait complete. 2024-04-16T16:20:37.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.627+0000 7f70a0600640 1 Processor -- start 2024-04-16T16:20:37.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.628+0000 7f70a0600640 1 -- start start 2024-04-16T16:20:37.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.628+0000 7f70a0600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70800039b0 0x7f7098161b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:37.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.628+0000 7f70a0600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7098155010 0x7f709814fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:37.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.629+0000 7f70a0600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70981526f0 0x7f7098152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:37.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.629+0000 7f70a0600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f709806bab0 con 0x7f70981526f0 2024-04-16T16:20:37.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.629+0000 7f70a0600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f709806b930 con 0x7f7098155010 2024-04-16T16:20:37.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.629+0000 7f70a0600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f709806bc30 con 0x7f70800039b0 2024-04-16T16:20:37.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.629+0000 7f709e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70981526f0 0x7f7098152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:37.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.629+0000 7f709e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70981526f0 0x7f7098152ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44316/0 (socket says 172.21.15.45:44316) 2024-04-16T16:20:37.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.629+0000 7f709e600640 1 -- 172.21.15.45:0/2816626569 learned_addr learned my addr 172.21.15.45:0/2816626569 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:37.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.629+0000 7f709d200640 1 --2- 172.21.15.45:0/2816626569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7098155010 0x7f709814fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:37.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.629+0000 7f709dc00640 1 --2- 172.21.15.45:0/2816626569 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70800039b0 0x7f7098161b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:37.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.629+0000 7f709e600640 1 -- 172.21.15.45:0/2816626569 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70800039b0 msgr2=0x7f7098161b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:37.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.629+0000 7f709e600640 1 --2- 172.21.15.45:0/2816626569 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70800039b0 0x7f7098161b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:37.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.629+0000 7f709e600640 1 -- 172.21.15.45:0/2816626569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7098155010 msgr2=0x7f709814fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:37.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.629+0000 7f709e600640 1 --2- 172.21.15.45:0/2816626569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7098155010 0x7f709814fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:37.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.629+0000 7f709e600640 1 -- 172.21.15.45:0/2816626569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7098161390 con 0x7f70981526f0 2024-04-16T16:20:37.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.630+0000 7f709e600640 1 --2- 172.21.15.45:0/2816626569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70981526f0 0x7f7098152ba0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f70980d1580 tx=0x7f709000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:37.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.630+0000 7f7095600640 1 -- 172.21.15.45:0/2816626569 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709000a5b0 con 0x7f70981526f0 2024-04-16T16:20:37.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.630+0000 7f7095600640 1 -- 172.21.15.45:0/2816626569 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f709000a770 con 0x7f70981526f0 2024-04-16T16:20:37.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.630+0000 7f7095600640 1 -- 172.21.15.45:0/2816626569 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709000ab60 con 0x7f70981526f0 2024-04-16T16:20:37.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.630+0000 7f70a0600640 1 -- 172.21.15.45:0/2816626569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70980b6860 con 0x7f70981526f0 2024-04-16T16:20:37.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.630+0000 7f70a0600640 1 -- 172.21.15.45:0/2816626569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7098161600 con 0x7f70981526f0 2024-04-16T16:20:37.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.631+0000 7f70a0600640 1 -- 172.21.15.45:0/2816626569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7098075780 con 0x7f70981526f0 2024-04-16T16:20:37.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.632+0000 7f7095600640 1 -- 172.21.15.45:0/2816626569 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f7090011070 con 0x7f70981526f0 2024-04-16T16:20:37.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.633+0000 7f7095600640 1 --2- 172.21.15.45:0/2816626569 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f707806ca10 0x7f707806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:37.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.633+0000 7f7095600640 1 -- 172.21.15.45:0/2816626569 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7f709008b480 con 0x7f70981526f0 2024-04-16T16:20:37.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.634+0000 7f709dc00640 1 --2- 172.21.15.45:0/2816626569 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f707806ca10 0x7f707806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:37.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.635+0000 7f709dc00640 1 --2- 172.21.15.45:0/2816626569 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f707806ca10 0x7f707806eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f708c036000 tx=0x7f708c030000 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:37.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.638+0000 7f7095600640 1 -- 172.21.15.45:0/2816626569 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7090059670 con 0x7f70981526f0 2024-04-16T16:20:37.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.883+0000 7f70a0600640 1 -- 172.21.15.45:0/2816626569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f709815ce90 con 0x7f70981526f0 2024-04-16T16:20:37.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.884+0000 7f7095600640 1 -- 172.21.15.45:0/2816626569 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5186 (secure 0 0 0) 0x7f7090058f70 con 0x7f70981526f0 2024-04-16T16:20:37.883 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:37.884 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/929218719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":929218719},{"type":"v1","addr":"172.21.15.103:6835","nonce":929218719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6323,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1619719405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1619719405},{"type":"v1","addr":"172.21.15.103:6837","nonce":1619719405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":8640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/3613288118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3613288118},{"type":"v1","addr":"172.21.15.45:6834","nonce":3613288118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:20:33.816422+0000","modified":"2024-04-16T16:20:36.891870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8643":{"gid":8643,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2060312705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2060312705},{"type":"v1","addr":"172.21.15.45:6835","nonce":2060312705}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8643,"qdb_cluster":[8643]},"id":9}]} 2024-04-16T16:20:37.884 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 91 2024-04-16T16:20:37.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.887+0000 7f70a0600640 1 -- 172.21.15.45:0/2816626569 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f707806ca10 msgr2=0x7f707806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:37.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.887+0000 7f70a0600640 1 --2- 172.21.15.45:0/2816626569 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f707806ca10 0x7f707806eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f708c036000 tx=0x7f708c030000 comp rx=0 tx=0).stop 2024-04-16T16:20:37.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.887+0000 7f70a0600640 1 -- 172.21.15.45:0/2816626569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70981526f0 msgr2=0x7f7098152ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:37.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.887+0000 7f70a0600640 1 --2- 172.21.15.45:0/2816626569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70981526f0 0x7f7098152ba0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f70980d1580 tx=0x7f709000cc20 comp rx=0 tx=0).stop 2024-04-16T16:20:37.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.887+0000 7f70a0600640 1 -- 172.21.15.45:0/2816626569 shutdown_connections 2024-04-16T16:20:37.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.887+0000 7f70a0600640 1 --2- 172.21.15.45:0/2816626569 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f707806ca10 0x7f707806eed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:37.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.887+0000 7f70a0600640 1 --2- 172.21.15.45:0/2816626569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70981526f0 0x7f7098152ba0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.887+0000 7f70a0600640 1 --2- 172.21.15.45:0/2816626569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7098155010 0x7f709814fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.887+0000 7f70a0600640 1 --2- 172.21.15.45:0/2816626569 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70800039b0 0x7f7098161b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.887+0000 7f70a0600640 1 -- 172.21.15.45:0/2816626569 >> 172.21.15.45:0/2816626569 conn(0x7f709809aa70 msgr2=0x7f70980d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.887+0000 7f70a0600640 1 -- 172.21.15.45:0/2816626569 shutdown_connections 2024-04-16T16:20:37.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:37.887+0000 7f70a0600640 1 -- 172.21.15.45:0/2816626569 wait complete. 2024-04-16T16:20:37.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:20:38.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.115+0000 7f4881e00640 1 Processor -- start 2024-04-16T16:20:38.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.115+0000 7f4881e00640 1 -- start start 2024-04-16T16:20:38.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.116+0000 7f4881e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f487c09ab50 0x7f487c09af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:38.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.116+0000 7f4881e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f487c0d1cc0 0x7f487c15f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:38.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.116+0000 7f4881e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f487c09b470 con 0x7f487c162720 2024-04-16T16:20:38.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.116+0000 7f4881e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f487c06dbf0 con 0x7f487c09ab50 2024-04-16T16:20:38.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.116+0000 7f4881e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f487c06cfe0 con 0x7f487c0d1cc0 2024-04-16T16:20:38.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.116+0000 7f487aa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f487c09ab50 0x7f487c09af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:38.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.116+0000 7f487aa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f487c09ab50 0x7f487c09af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44332/0 (socket says 172.21.15.45:44332) 2024-04-16T16:20:38.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.117+0000 7f487aa00640 1 -- 172.21.15.45:0/1171391525 learned_addr learned my addr 172.21.15.45:0/1171391525 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:38.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.117+0000 7f487aa00640 1 -- 172.21.15.45:0/1171391525 >> v2:172.21.15.103:3300/0 conn(0x7f487c0d1cc0 msgr2=0x7f487c15f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:38.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.117+0000 7f487b400640 1 --2- 172.21.15.45:0/1171391525 >> v2:172.21.15.103:3300/0 conn(0x7f487c0d1cc0 0x7f487c15f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:38.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.117+0000 7f487aa00640 1 --2- 172.21.15.45:0/1171391525 >> v2:172.21.15.103:3300/0 conn(0x7f487c0d1cc0 0x7f487c15f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.117+0000 7f487aa00640 1 -- 172.21.15.45:0/1171391525 >> v1:172.21.15.103:6789/0 conn(0x7f487c162720 legacy=0x7f487c15eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:38.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.117+0000 7f487aa00640 1 -- 172.21.15.45:0/1171391525 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487c06d2c0 con 0x7f487c09ab50 2024-04-16T16:20:38.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.118+0000 7f487aa00640 1 --2- 172.21.15.45:0/1171391525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f487c09ab50 0x7f487c09af30 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f487c070300 tx=0x7f486c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=250fc010c5d69f95 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:38.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.119+0000 7f4879600640 1 -- 172.21.15.45:0/1171391525 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c035020 con 0x7f487c09ab50 2024-04-16T16:20:38.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.119+0000 7f4879600640 1 -- 172.21.15.45:0/1171391525 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f486c031870 con 0x7f487c09ab50 2024-04-16T16:20:38.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.119+0000 7f4879600640 1 -- 172.21.15.45:0/1171391525 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c031c40 con 0x7f487c09ab50 2024-04-16T16:20:38.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.119+0000 7f4881e00640 1 -- 172.21.15.45:0/1171391525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f487c09ab50 msgr2=0x7f487c09af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:38.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.119+0000 7f4881e00640 1 --2- 172.21.15.45:0/1171391525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f487c09ab50 0x7f487c09af30 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f487c070300 tx=0x7f486c02b910 comp rx=0 tx=0).stop 2024-04-16T16:20:38.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.120+0000 7f4881e00640 1 -- 172.21.15.45:0/1171391525 shutdown_connections 2024-04-16T16:20:38.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.120+0000 7f4881e00640 1 --2- 172.21.15.45:0/1171391525 >> v2:172.21.15.103:3300/0 conn(0x7f487c0d1cc0 0x7f487c15f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.120+0000 7f4881e00640 1 --2- 172.21.15.45:0/1171391525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f487c09ab50 0x7f487c09af30 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.120+0000 7f4881e00640 1 -- 172.21.15.45:0/1171391525 >> 172.21.15.45:0/1171391525 conn(0x7f487c0d2890 msgr2=0x7f487c0d2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:38.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.120+0000 7f4881e00640 1 -- 172.21.15.45:0/1171391525 shutdown_connections 2024-04-16T16:20:38.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.120+0000 7f4881e00640 1 -- 172.21.15.45:0/1171391525 wait complete. 2024-04-16T16:20:38.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.122+0000 7f4881e00640 1 Processor -- start 2024-04-16T16:20:38.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.122+0000 7f4881e00640 1 -- start start 2024-04-16T16:20:38.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.123+0000 7f4881e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f487c09ab50 0x7f487c1f9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:38.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.123+0000 7f4881e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f487c0d1cc0 0x7f487c0c8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:38.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.123+0000 7f4881e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f487c162720 0x7f487c0c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:38.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.123+0000 7f4881e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f487c09b8b0 con 0x7f487c0d1cc0 2024-04-16T16:20:38.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.123+0000 7f4881e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f487c06d880 con 0x7f487c09ab50 2024-04-16T16:20:38.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.123+0000 7f4881e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f487c06ca40 con 0x7f487c162720 2024-04-16T16:20:38.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.123+0000 7f487a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f487c0d1cc0 0x7f487c0c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:38.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.123+0000 7f487a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f487c0d1cc0 0x7f487c0c8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44334/0 (socket says 172.21.15.45:44334) 2024-04-16T16:20:38.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.123+0000 7f487a000640 1 -- 172.21.15.45:0/2427782643 learned_addr learned my addr 172.21.15.45:0/2427782643 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:38.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.123+0000 7f487a000640 1 -- 172.21.15.45:0/2427782643 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f487c162720 msgr2=0x7f487c0c8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:38.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.123+0000 7f487a000640 1 --2- 172.21.15.45:0/2427782643 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f487c162720 0x7f487c0c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.123+0000 7f487a000640 1 -- 172.21.15.45:0/2427782643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f487c09ab50 msgr2=0x7f487c1f9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:38.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.123+0000 7f487aa00640 1 --2- 172.21.15.45:0/2427782643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f487c09ab50 0x7f487c1f9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:38.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.123+0000 7f487a000640 1 --2- 172.21.15.45:0/2427782643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f487c09ab50 0x7f487c1f9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.123+0000 7f487a000640 1 -- 172.21.15.45:0/2427782643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487c0c72f0 con 0x7f487c0d1cc0 2024-04-16T16:20:38.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.124+0000 7f487a000640 1 --2- 172.21.15.45:0/2427782643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f487c0d1cc0 0x7f487c0c8350 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f486400c6e0 tx=0x7f486400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:38.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.124+0000 7f4873400640 1 -- 172.21.15.45:0/2427782643 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4864016020 con 0x7f487c0d1cc0 2024-04-16T16:20:38.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.124+0000 7f4873400640 1 -- 172.21.15.45:0/2427782643 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4864006d50 con 0x7f487c0d1cc0 2024-04-16T16:20:38.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.124+0000 7f4873400640 1 -- 172.21.15.45:0/2427782643 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4864003bb0 con 0x7f487c0d1cc0 2024-04-16T16:20:38.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.124+0000 7f4881e00640 1 -- 172.21.15.45:0/2427782643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f487c0c9030 con 0x7f487c0d1cc0 2024-04-16T16:20:38.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.124+0000 7f4881e00640 1 -- 172.21.15.45:0/2427782643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f487c0c7870 con 0x7f487c0d1cc0 2024-04-16T16:20:38.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.126+0000 7f4881e00640 1 -- 172.21.15.45:0/2427782643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f487c075780 con 0x7f487c0d1cc0 2024-04-16T16:20:38.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.126+0000 7f4873400640 1 -- 172.21.15.45:0/2427782643 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f48640050d0 con 0x7f487c0d1cc0 2024-04-16T16:20:38.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.127+0000 7f4873400640 1 --2- 172.21.15.45:0/2427782643 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f485806ca10 0x7f485806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:38.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.127+0000 7f4873400640 1 -- 172.21.15.45:0/2427782643 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7f486408b980 con 0x7f487c0d1cc0 2024-04-16T16:20:38.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.128+0000 7f487aa00640 1 --2- 172.21.15.45:0/2427782643 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f485806ca10 0x7f485806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:38.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.129+0000 7f487aa00640 1 --2- 172.21.15.45:0/2427782643 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f485806ca10 0x7f485806eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f486c036000 tx=0x7f486c008030 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:38.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.132+0000 7f4873400640 1 -- 172.21.15.45:0/2427782643 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4864020020 con 0x7f487c0d1cc0 2024-04-16T16:20:38.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.335+0000 7f4881e00640 1 -- 172.21.15.45:0/2427782643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f487c0d3710 con 0x7f487c0d1cc0 2024-04-16T16:20:38.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.337+0000 7f4873400640 1 -- 172.21.15.45:0/2427782643 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) ==== 74+0+16679 (secure 0 0 0) 0x7f4864059c90 con 0x7f487c0d1cc0 2024-04-16T16:20:38.336 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:38.337 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":128,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:20:35.342933+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:20:31.681227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-16T16:20:32.790980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:20:38.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.340+0000 7f4881e00640 1 -- 172.21.15.45:0/2427782643 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f485806ca10 msgr2=0x7f485806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:38.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.340+0000 7f4881e00640 1 --2- 172.21.15.45:0/2427782643 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f485806ca10 0x7f485806eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f486c036000 tx=0x7f486c008030 comp rx=0 tx=0).stop 2024-04-16T16:20:38.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.340+0000 7f4881e00640 1 -- 172.21.15.45:0/2427782643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f487c0d1cc0 msgr2=0x7f487c0c8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:38.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.340+0000 7f4881e00640 1 --2- 172.21.15.45:0/2427782643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f487c0d1cc0 0x7f487c0c8350 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f486400c6e0 tx=0x7f486400cba0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.340+0000 7f4881e00640 1 -- 172.21.15.45:0/2427782643 shutdown_connections 2024-04-16T16:20:38.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.340+0000 7f4881e00640 1 --2- 172.21.15.45:0/2427782643 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f485806ca10 0x7f485806eed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.340+0000 7f4881e00640 1 --2- 172.21.15.45:0/2427782643 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f487c162720 0x7f487c0c8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.340+0000 7f4881e00640 1 --2- 172.21.15.45:0/2427782643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f487c0d1cc0 0x7f487c0c8350 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.340+0000 7f4881e00640 1 --2- 172.21.15.45:0/2427782643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f487c09ab50 0x7f487c1f9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.340+0000 7f4881e00640 1 -- 172.21.15.45:0/2427782643 >> 172.21.15.45:0/2427782643 conn(0x7f487c0d2890 msgr2=0x7f487c070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:38.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.340+0000 7f4881e00640 1 -- 172.21.15.45:0/2427782643 shutdown_connections 2024-04-16T16:20:38.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.340+0000 7f4881e00640 1 -- 172.21.15.45:0/2427782643 wait complete. 2024-04-16T16:20:38.360 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-16T16:20:38.361 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:20:38.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.582+0000 7fa86fe00640 1 Processor -- start 2024-04-16T16:20:38.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.582+0000 7fa86fe00640 1 -- start start 2024-04-16T16:20:38.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.584+0000 7fa86fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa87015ccb0 0x7fa87006df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:38.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.585+0000 7fa86fe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa87015d360 con 0x7fa87006e480 2024-04-16T16:20:38.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.585+0000 7fa86fe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa87015d640 con 0x7fa870146220 2024-04-16T16:20:38.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.585+0000 7fa86fe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa87015d130 con 0x7fa87015ccb0 2024-04-16T16:20:38.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.585+0000 7fa86f400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa87006e480 0x7fa8701608a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43080/0 (socket says 172.21.15.45:43080) 2024-04-16T16:20:38.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.585+0000 7fa866000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa87015ccb0 0x7fa87006df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:38.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.585+0000 7fa86f400640 1 -- 172.21.15.45:0/3920588501 learned_addr learned my addr 172.21.15.45:0/3920588501 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.585+0000 7fa866000640 1 -- 172.21.15.45:0/3920588501 >> v1:172.21.15.103:6789/0 conn(0x7fa870146220 legacy=0x7fa87006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.585+0000 7fa866000640 1 -- 172.21.15.45:0/3920588501 >> v1:172.21.15.45:6789/0 conn(0x7fa87006e480 legacy=0x7fa8701608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.585+0000 7fa866000640 1 -- 172.21.15.45:0/3920588501 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87015d920 con 0x7fa87015ccb0 2024-04-16T16:20:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.585+0000 7fa86e000640 1 -- 172.21.15.45:0/3920588501 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1989866759 0 0) 0x7fa87015d360 con 0x7fa87006e480 2024-04-16T16:20:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.587+0000 7fa866000640 1 --2- 172.21.15.45:0/3920588501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa87015ccb0 0x7fa87006df40 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fa8580097a0 tx=0x7fa85802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c9c1614a478528 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.587+0000 7fa86e000640 1 -- 172.21.15.45:0/3920588501 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa858035020 con 0x7fa87015ccb0 2024-04-16T16:20:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.587+0000 7fa86e000640 1 -- 172.21.15.45:0/3920588501 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa858031830 con 0x7fa87015ccb0 2024-04-16T16:20:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.587+0000 7fa86e000640 1 -- 172.21.15.45:0/3920588501 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa858031c00 con 0x7fa87015ccb0 2024-04-16T16:20:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.588+0000 7fa86fe00640 1 -- 172.21.15.45:0/3920588501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa87015ccb0 msgr2=0x7fa87006df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.588+0000 7fa86fe00640 1 --2- 172.21.15.45:0/3920588501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa87015ccb0 0x7fa87006df40 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fa8580097a0 tx=0x7fa85802b8d0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.588+0000 7fa86fe00640 1 -- 172.21.15.45:0/3920588501 shutdown_connections 2024-04-16T16:20:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.588+0000 7fa86fe00640 1 --2- 172.21.15.45:0/3920588501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa87015ccb0 0x7fa87006df40 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.588+0000 7fa86fe00640 1 -- 172.21.15.45:0/3920588501 >> 172.21.15.45:0/3920588501 conn(0x7fa87006b930 msgr2=0x7fa87006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.588+0000 7fa86fe00640 1 -- 172.21.15.45:0/3920588501 shutdown_connections 2024-04-16T16:20:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.588+0000 7fa86fe00640 1 -- 172.21.15.45:0/3920588501 wait complete. 2024-04-16T16:20:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.589+0000 7fa86fe00640 1 Processor -- start 2024-04-16T16:20:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.590+0000 7fa86fe00640 1 -- start start 2024-04-16T16:20:38.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.591+0000 7fa86fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa87006e480 0x7fa87013fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:38.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.591+0000 7fa86fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa870146220 0x7fa87013f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.591+0000 7fa86fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa87015ccb0 0x7fa8700cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.591+0000 7fa86fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa870160fb0 con 0x7fa87015ccb0 2024-04-16T16:20:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.591+0000 7fa86fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa870160410 con 0x7fa87006e480 2024-04-16T16:20:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.591+0000 7fa86fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa870161130 con 0x7fa870146220 2024-04-16T16:20:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.591+0000 7fa86f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa87015ccb0 0x7fa8700cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.591+0000 7fa86f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa87015ccb0 0x7fa8700cdbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44346/0 (socket says 172.21.15.45:44346) 2024-04-16T16:20:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.591+0000 7fa86f400640 1 -- 172.21.15.45:0/3526497091 learned_addr learned my addr 172.21.15.45:0/3526497091 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.591+0000 7fa86ea00640 1 --2- 172.21.15.45:0/3526497091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa87006e480 0x7fa87013fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.591+0000 7fa866000640 1 --2- 172.21.15.45:0/3526497091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa870146220 0x7fa87013f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.591+0000 7fa86f400640 1 -- 172.21.15.45:0/3526497091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa870146220 msgr2=0x7fa87013f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.591+0000 7fa86f400640 1 --2- 172.21.15.45:0/3526497091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa870146220 0x7fa87013f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.591+0000 7fa86f400640 1 -- 172.21.15.45:0/3526497091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa87006e480 msgr2=0x7fa87013fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.592+0000 7fa86f400640 1 --2- 172.21.15.45:0/3526497091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa87006e480 0x7fa87013fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.592+0000 7fa86f400640 1 -- 172.21.15.45:0/3526497091 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8701e8a70 con 0x7fa87015ccb0 2024-04-16T16:20:38.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.592+0000 7fa86f400640 1 --2- 172.21.15.45:0/3526497091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa87015ccb0 0x7fa8700cdbb0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fa86800c7a0 tx=0x7fa86800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:38.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.592+0000 7fa867400640 1 -- 172.21.15.45:0/3526497091 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868018070 con 0x7fa87015ccb0 2024-04-16T16:20:38.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.592+0000 7fa867400640 1 -- 172.21.15.45:0/3526497091 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa86800a640 con 0x7fa87015ccb0 2024-04-16T16:20:38.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.592+0000 7fa867400640 1 -- 172.21.15.45:0/3526497091 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86800aab0 con 0x7fa87015ccb0 2024-04-16T16:20:38.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.592+0000 7fa86fe00640 1 -- 172.21.15.45:0/3526497091 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8701e7ef0 con 0x7fa87015ccb0 2024-04-16T16:20:38.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.592+0000 7fa86fe00640 1 -- 172.21.15.45:0/3526497091 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8701e84b0 con 0x7fa87015ccb0 2024-04-16T16:20:38.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.594+0000 7fa86fe00640 1 -- 172.21.15.45:0/3526497091 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa870075780 con 0x7fa87015ccb0 2024-04-16T16:20:38.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.595+0000 7fa867400640 1 -- 172.21.15.45:0/3526497091 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7fa86801b070 con 0x7fa87015ccb0 2024-04-16T16:20:38.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.596+0000 7fa867400640 1 --2- 172.21.15.45:0/3526497091 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fa84c06ca40 0x7fa84c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:38.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.596+0000 7fa867400640 1 -- 172.21.15.45:0/3526497091 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7fa86808b340 con 0x7fa87015ccb0 2024-04-16T16:20:38.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.597+0000 7fa86ea00640 1 --2- 172.21.15.45:0/3526497091 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fa84c06ca40 0x7fa84c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:38.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.598+0000 7fa86ea00640 1 --2- 172.21.15.45:0/3526497091 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fa84c06ca40 0x7fa84c06ef00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa870140470 tx=0x7fa860009040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:38.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.601+0000 7fa867400640 1 -- 172.21.15.45:0/3526497091 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa8680595c0 con 0x7fa87015ccb0 2024-04-16T16:20:38.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.800+0000 7fa86fe00640 1 -- 172.21.15.45:0/3526497091 --> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa8701420c0 con 0x7fa84c06ca40 2024-04-16T16:20:38.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.823+0000 7fa867400640 1 -- 172.21.15.45:0/3526497091 <== mgr.6793 v2:172.21.15.103:6832/1120913661 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252422 (secure 0 0 0) 0x7fa8600053f0 con 0x7fa84c06ca40 2024-04-16T16:20:38.827 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:38.828 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:20:38.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.831+0000 7fa86fe00640 1 -- 172.21.15.45:0/3526497091 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fa84c06ca40 msgr2=0x7fa84c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:38.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.831+0000 7fa86fe00640 1 --2- 172.21.15.45:0/3526497091 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fa84c06ca40 0x7fa84c06ef00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa870140470 tx=0x7fa860009040 comp rx=0 tx=0).stop 2024-04-16T16:20:38.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.831+0000 7fa86fe00640 1 -- 172.21.15.45:0/3526497091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa87015ccb0 msgr2=0x7fa8700cdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:38.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.831+0000 7fa86fe00640 1 --2- 172.21.15.45:0/3526497091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa87015ccb0 0x7fa8700cdbb0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fa86800c7a0 tx=0x7fa86800cc60 comp rx=0 tx=0).stop 2024-04-16T16:20:38.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.831+0000 7fa86fe00640 1 -- 172.21.15.45:0/3526497091 shutdown_connections 2024-04-16T16:20:38.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.831+0000 7fa86fe00640 1 --2- 172.21.15.45:0/3526497091 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fa84c06ca40 0x7fa84c06ef00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.831+0000 7fa86fe00640 1 --2- 172.21.15.45:0/3526497091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa87015ccb0 0x7fa8700cdbb0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.831+0000 7fa86fe00640 1 --2- 172.21.15.45:0/3526497091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa870146220 0x7fa87013f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.831+0000 7fa86fe00640 1 --2- 172.21.15.45:0/3526497091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa87006e480 0x7fa87013fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:38.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.831+0000 7fa86fe00640 1 -- 172.21.15.45:0/3526497091 >> 172.21.15.45:0/3526497091 conn(0x7fa87006b930 msgr2=0x7fa8701409f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:38.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.831+0000 7fa86fe00640 1 -- 172.21.15.45:0/3526497091 shutdown_connections 2024-04-16T16:20:38.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:38.831+0000 7fa86fe00640 1 -- 172.21.15.45:0/3526497091 wait complete. 2024-04-16T16:20:38.850 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-16T16:20:37.504791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":891,"num_read_kb":1701,"num_write":351,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4685,"ondisk_log_size":4685,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":267100,"kb_used_data":13464,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754707620,"statfs":{"total":773094113280,"available":772820602880,"internally_reserved":0,"allocated":13787136,"data_stored":7146037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.202295"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.372822+0000","last_change":"2024-04-16T16:20:34.361793+0000","last_active":"2024-04-16T16:20:34.372822+0000","last_peered":"2024-04-16T16:20:34.372822+0000","last_clean":"2024-04-16T16:20:34.372822+0000","last_became_active":"2024-04-16T16:20:34.361431+0000","last_became_peered":"2024-04-16T16:20:34.361431+0000","last_unstale":"2024-04-16T16:20:34.372822+0000","last_undegraded":"2024-04-16T16:20:34.372822+0000","last_fullsized":"2024-04-16T16:20:34.372822+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:22:55.733613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.364257+0000","last_change":"2024-04-16T16:20:34.358765+0000","last_active":"2024-04-16T16:20:34.364257+0000","last_peered":"2024-04-16T16:20:34.364257+0000","last_clean":"2024-04-16T16:20:34.364257+0000","last_became_active":"2024-04-16T16:20:34.358417+0000","last_became_peered":"2024-04-16T16:20:34.358417+0000","last_unstale":"2024-04-16T16:20:34.364257+0000","last_undegraded":"2024-04-16T16:20:34.364257+0000","last_fullsized":"2024-04-16T16:20:34.364257+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:48.948213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.368092+0000","last_change":"2024-04-16T16:20:34.359027+0000","last_active":"2024-04-16T16:20:34.368092+0000","last_peered":"2024-04-16T16:20:34.368092+0000","last_clean":"2024-04-16T16:20:34.368092+0000","last_became_active":"2024-04-16T16:20:34.355482+0000","last_became_peered":"2024-04-16T16:20:34.355482+0000","last_unstale":"2024-04-16T16:20:34.368092+0000","last_undegraded":"2024-04-16T16:20:34.368092+0000","last_fullsized":"2024-04-16T16:20:34.368092+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:14:11.178395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.354131+0000","last_change":"2024-04-16T16:20:34.347936+0000","last_active":"2024-04-16T16:20:34.354131+0000","last_peered":"2024-04-16T16:20:34.354131+0000","last_clean":"2024-04-16T16:20:34.354131+0000","last_became_active":"2024-04-16T16:20:34.347408+0000","last_became_peered":"2024-04-16T16:20:34.347408+0000","last_unstale":"2024-04-16T16:20:34.354131+0000","last_undegraded":"2024-04-16T16:20:34.354131+0000","last_fullsized":"2024-04-16T16:20:34.354131+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:33:30.949546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.362735+0000","last_change":"2024-04-16T16:20:34.358353+0000","last_active":"2024-04-16T16:20:34.362735+0000","last_peered":"2024-04-16T16:20:34.362735+0000","last_clean":"2024-04-16T16:20:34.362735+0000","last_became_active":"2024-04-16T16:20:34.357419+0000","last_became_peered":"2024-04-16T16:20:34.357419+0000","last_unstale":"2024-04-16T16:20:34.362735+0000","last_undegraded":"2024-04-16T16:20:34.362735+0000","last_fullsized":"2024-04-16T16:20:34.362735+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:11:35.179675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375099+0000","last_change":"2024-04-16T16:20:34.371848+0000","last_active":"2024-04-16T16:20:34.375099+0000","last_peered":"2024-04-16T16:20:34.375099+0000","last_clean":"2024-04-16T16:20:34.375099+0000","last_became_active":"2024-04-16T16:20:34.371503+0000","last_became_peered":"2024-04-16T16:20:34.371503+0000","last_unstale":"2024-04-16T16:20:34.375099+0000","last_undegraded":"2024-04-16T16:20:34.375099+0000","last_fullsized":"2024-04-16T16:20:34.375099+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:12.236456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.359296+0000","last_change":"2024-04-16T16:20:34.355494+0000","last_active":"2024-04-16T16:20:34.359296+0000","last_peered":"2024-04-16T16:20:34.359296+0000","last_clean":"2024-04-16T16:20:34.359296+0000","last_became_active":"2024-04-16T16:20:34.353830+0000","last_became_peered":"2024-04-16T16:20:34.353830+0000","last_unstale":"2024-04-16T16:20:34.359296+0000","last_undegraded":"2024-04-16T16:20:34.359296+0000","last_fullsized":"2024-04-16T16:20:34.359296+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:27:17.816514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.371747+0000","last_change":"2024-04-16T16:20:34.360213+0000","last_active":"2024-04-16T16:20:34.371747+0000","last_peered":"2024-04-16T16:20:34.371747+0000","last_clean":"2024-04-16T16:20:34.371747+0000","last_became_active":"2024-04-16T16:20:34.359511+0000","last_became_peered":"2024-04-16T16:20:34.359511+0000","last_unstale":"2024-04-16T16:20:34.371747+0000","last_undegraded":"2024-04-16T16:20:34.371747+0000","last_fullsized":"2024-04-16T16:20:34.371747+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:12.236456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.368141+0000","last_change":"2024-04-16T16:20:34.362583+0000","last_active":"2024-04-16T16:20:34.368141+0000","last_peered":"2024-04-16T16:20:34.368141+0000","last_clean":"2024-04-16T16:20:34.368141+0000","last_became_active":"2024-04-16T16:20:34.360510+0000","last_became_peered":"2024-04-16T16:20:34.360510+0000","last_unstale":"2024-04-16T16:20:34.368141+0000","last_undegraded":"2024-04-16T16:20:34.368141+0000","last_fullsized":"2024-04-16T16:20:34.368141+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:58:34.406516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.364230+0000","last_change":"2024-04-16T16:20:34.358409+0000","last_active":"2024-04-16T16:20:34.364230+0000","last_peered":"2024-04-16T16:20:34.364230+0000","last_clean":"2024-04-16T16:20:34.364230+0000","last_became_active":"2024-04-16T16:20:34.358077+0000","last_became_peered":"2024-04-16T16:20:34.358077+0000","last_unstale":"2024-04-16T16:20:34.364230+0000","last_undegraded":"2024-04-16T16:20:34.364230+0000","last_fullsized":"2024-04-16T16:20:34.364230+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:52:36.909650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.372065+0000","last_change":"2024-04-16T16:20:34.365071+0000","last_active":"2024-04-16T16:20:34.372065+0000","last_peered":"2024-04-16T16:20:34.372065+0000","last_clean":"2024-04-16T16:20:34.372065+0000","last_became_active":"2024-04-16T16:20:34.364744+0000","last_became_peered":"2024-04-16T16:20:34.364744+0000","last_unstale":"2024-04-16T16:20:34.372065+0000","last_undegraded":"2024-04-16T16:20:34.372065+0000","last_fullsized":"2024-04-16T16:20:34.372065+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:20:35.843902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.366933+0000","last_change":"2024-04-16T16:20:34.357516+0000","last_active":"2024-04-16T16:20:34.366933+0000","last_peered":"2024-04-16T16:20:34.366933+0000","last_clean":"2024-04-16T16:20:34.366933+0000","last_became_active":"2024-04-16T16:20:34.353848+0000","last_became_peered":"2024-04-16T16:20:34.353848+0000","last_unstale":"2024-04-16T16:20:34.366933+0000","last_undegraded":"2024-04-16T16:20:34.366933+0000","last_fullsized":"2024-04-16T16:20:34.366933+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:47:39.882496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.372812+0000","last_change":"2024-04-16T16:20:34.362085+0000","last_active":"2024-04-16T16:20:34.372812+0000","last_peered":"2024-04-16T16:20:34.372812+0000","last_clean":"2024-04-16T16:20:34.372812+0000","last_became_active":"2024-04-16T16:20:34.361604+0000","last_became_peered":"2024-04-16T16:20:34.361604+0000","last_unstale":"2024-04-16T16:20:34.372812+0000","last_undegraded":"2024-04-16T16:20:34.372812+0000","last_fullsized":"2024-04-16T16:20:34.372812+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:15:59.018794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.381747+0000","last_change":"2024-04-16T16:20:34.378870+0000","last_active":"2024-04-16T16:20:34.381747+0000","last_peered":"2024-04-16T16:20:34.381747+0000","last_clean":"2024-04-16T16:20:34.381747+0000","last_became_active":"2024-04-16T16:20:34.378566+0000","last_became_peered":"2024-04-16T16:20:34.378566+0000","last_unstale":"2024-04-16T16:20:34.381747+0000","last_undegraded":"2024-04-16T16:20:34.381747+0000","last_fullsized":"2024-04-16T16:20:34.381747+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:56:42.934648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.354626+0000","last_change":"2024-04-16T16:20:34.350500+0000","last_active":"2024-04-16T16:20:34.354626+0000","last_peered":"2024-04-16T16:20:34.354626+0000","last_clean":"2024-04-16T16:20:34.354626+0000","last_became_active":"2024-04-16T16:20:34.350184+0000","last_became_peered":"2024-04-16T16:20:34.350184+0000","last_unstale":"2024-04-16T16:20:34.354626+0000","last_undegraded":"2024-04-16T16:20:34.354626+0000","last_fullsized":"2024-04-16T16:20:34.354626+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:45.599894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.381403+0000","last_change":"2024-04-16T16:20:34.376104+0000","last_active":"2024-04-16T16:20:34.381403+0000","last_peered":"2024-04-16T16:20:34.381403+0000","last_clean":"2024-04-16T16:20:34.381403+0000","last_became_active":"2024-04-16T16:20:34.375771+0000","last_became_peered":"2024-04-16T16:20:34.375771+0000","last_unstale":"2024-04-16T16:20:34.381403+0000","last_undegraded":"2024-04-16T16:20:34.381403+0000","last_fullsized":"2024-04-16T16:20:34.381403+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:26:13.741282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375045+0000","last_change":"2024-04-16T16:20:34.367113+0000","last_active":"2024-04-16T16:20:34.375045+0000","last_peered":"2024-04-16T16:20:34.375045+0000","last_clean":"2024-04-16T16:20:34.375045+0000","last_became_active":"2024-04-16T16:20:34.366838+0000","last_became_peered":"2024-04-16T16:20:34.366838+0000","last_unstale":"2024-04-16T16:20:34.375045+0000","last_undegraded":"2024-04-16T16:20:34.375045+0000","last_fullsized":"2024-04-16T16:20:34.375045+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:19.871374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375151+0000","last_change":"2024-04-16T16:20:34.366810+0000","last_active":"2024-04-16T16:20:34.375151+0000","last_peered":"2024-04-16T16:20:34.375151+0000","last_clean":"2024-04-16T16:20:34.375151+0000","last_became_active":"2024-04-16T16:20:34.364143+0000","last_became_peered":"2024-04-16T16:20:34.364143+0000","last_unstale":"2024-04-16T16:20:34.375151+0000","last_undegraded":"2024-04-16T16:20:34.375151+0000","last_fullsized":"2024-04-16T16:20:34.375151+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:30:53.885057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.372039+0000","last_change":"2024-04-16T16:20:34.360432+0000","last_active":"2024-04-16T16:20:34.372039+0000","last_peered":"2024-04-16T16:20:34.372039+0000","last_clean":"2024-04-16T16:20:34.372039+0000","last_became_active":"2024-04-16T16:20:34.360078+0000","last_became_peered":"2024-04-16T16:20:34.360078+0000","last_unstale":"2024-04-16T16:20:34.372039+0000","last_undegraded":"2024-04-16T16:20:34.372039+0000","last_fullsized":"2024-04-16T16:20:34.372039+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:34:53.236404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.370441+0000","last_change":"2024-04-16T16:20:34.360690+0000","last_active":"2024-04-16T16:20:34.370441+0000","last_peered":"2024-04-16T16:20:34.370441+0000","last_clean":"2024-04-16T16:20:34.370441+0000","last_became_active":"2024-04-16T16:20:34.357170+0000","last_became_peered":"2024-04-16T16:20:34.357170+0000","last_unstale":"2024-04-16T16:20:34.370441+0000","last_undegraded":"2024-04-16T16:20:34.370441+0000","last_fullsized":"2024-04-16T16:20:34.370441+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:53:10.676759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.368303+0000","last_change":"2024-04-16T16:20:34.363295+0000","last_active":"2024-04-16T16:20:34.368303+0000","last_peered":"2024-04-16T16:20:34.368303+0000","last_clean":"2024-04-16T16:20:34.368303+0000","last_became_active":"2024-04-16T16:20:34.362980+0000","last_became_peered":"2024-04-16T16:20:34.362980+0000","last_unstale":"2024-04-16T16:20:34.368303+0000","last_undegraded":"2024-04-16T16:20:34.368303+0000","last_fullsized":"2024-04-16T16:20:34.368303+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:01:37.023164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.372002+0000","last_change":"2024-04-16T16:20:34.362680+0000","last_active":"2024-04-16T16:20:34.372002+0000","last_peered":"2024-04-16T16:20:34.372002+0000","last_clean":"2024-04-16T16:20:34.372002+0000","last_became_active":"2024-04-16T16:20:34.361137+0000","last_became_peered":"2024-04-16T16:20:34.361137+0000","last_unstale":"2024-04-16T16:20:34.372002+0000","last_undegraded":"2024-04-16T16:20:34.372002+0000","last_fullsized":"2024-04-16T16:20:34.372002+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:51:03.598091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.359986+0000","last_change":"2024-04-16T16:20:34.352114+0000","last_active":"2024-04-16T16:20:34.359986+0000","last_peered":"2024-04-16T16:20:34.359986+0000","last_clean":"2024-04-16T16:20:34.359986+0000","last_became_active":"2024-04-16T16:20:34.351760+0000","last_became_peered":"2024-04-16T16:20:34.351760+0000","last_unstale":"2024-04-16T16:20:34.359986+0000","last_undegraded":"2024-04-16T16:20:34.359986+0000","last_fullsized":"2024-04-16T16:20:34.359986+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:10:00.520752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.370584+0000","last_change":"2024-04-16T16:20:34.359311+0000","last_active":"2024-04-16T16:20:34.370584+0000","last_peered":"2024-04-16T16:20:34.370584+0000","last_clean":"2024-04-16T16:20:34.370584+0000","last_became_active":"2024-04-16T16:20:34.359033+0000","last_became_peered":"2024-04-16T16:20:34.359033+0000","last_unstale":"2024-04-16T16:20:34.370584+0000","last_undegraded":"2024-04-16T16:20:34.370584+0000","last_fullsized":"2024-04-16T16:20:34.370584+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:47.655663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375122+0000","last_change":"2024-04-16T16:20:34.364311+0000","last_active":"2024-04-16T16:20:34.375122+0000","last_peered":"2024-04-16T16:20:34.375122+0000","last_clean":"2024-04-16T16:20:34.375122+0000","last_became_active":"2024-04-16T16:20:34.362818+0000","last_became_peered":"2024-04-16T16:20:34.362818+0000","last_unstale":"2024-04-16T16:20:34.375122+0000","last_undegraded":"2024-04-16T16:20:34.375122+0000","last_fullsized":"2024-04-16T16:20:34.375122+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:58:20.232704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.380983+0000","last_change":"2024-04-16T16:20:34.375272+0000","last_active":"2024-04-16T16:20:34.380983+0000","last_peered":"2024-04-16T16:20:34.380983+0000","last_clean":"2024-04-16T16:20:34.380983+0000","last_became_active":"2024-04-16T16:20:34.374920+0000","last_became_peered":"2024-04-16T16:20:34.374920+0000","last_unstale":"2024-04-16T16:20:34.380983+0000","last_undegraded":"2024-04-16T16:20:34.380983+0000","last_fullsized":"2024-04-16T16:20:34.380983+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:26:43.829444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.380686+0000","last_change":"2024-04-16T16:20:34.372477+0000","last_active":"2024-04-16T16:20:34.380686+0000","last_peered":"2024-04-16T16:20:34.380686+0000","last_clean":"2024-04-16T16:20:34.380686+0000","last_became_active":"2024-04-16T16:20:34.369430+0000","last_became_peered":"2024-04-16T16:20:34.369430+0000","last_unstale":"2024-04-16T16:20:34.380686+0000","last_undegraded":"2024-04-16T16:20:34.380686+0000","last_fullsized":"2024-04-16T16:20:34.380686+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:25:36.383304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.371070+0000","last_change":"2024-04-16T16:20:34.362286+0000","last_active":"2024-04-16T16:20:34.371070+0000","last_peered":"2024-04-16T16:20:34.371070+0000","last_clean":"2024-04-16T16:20:34.371070+0000","last_became_active":"2024-04-16T16:20:34.359101+0000","last_became_peered":"2024-04-16T16:20:34.359101+0000","last_unstale":"2024-04-16T16:20:34.371070+0000","last_undegraded":"2024-04-16T16:20:34.371070+0000","last_fullsized":"2024-04-16T16:20:34.371070+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:45:34.981948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.367739+0000","last_change":"2024-04-16T16:20:34.353766+0000","last_active":"2024-04-16T16:20:34.367739+0000","last_peered":"2024-04-16T16:20:34.367739+0000","last_clean":"2024-04-16T16:20:34.367739+0000","last_became_active":"2024-04-16T16:20:34.352111+0000","last_became_peered":"2024-04-16T16:20:34.352111+0000","last_unstale":"2024-04-16T16:20:34.367739+0000","last_undegraded":"2024-04-16T16:20:34.367739+0000","last_fullsized":"2024-04-16T16:20:34.367739+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:24:04.314017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.373562+0000","last_change":"2024-04-16T16:20:34.359559+0000","last_active":"2024-04-16T16:20:34.373562+0000","last_peered":"2024-04-16T16:20:34.373562+0000","last_clean":"2024-04-16T16:20:34.373562+0000","last_became_active":"2024-04-16T16:20:34.359194+0000","last_became_peered":"2024-04-16T16:20:34.359194+0000","last_unstale":"2024-04-16T16:20:34.373562+0000","last_undegraded":"2024-04-16T16:20:34.373562+0000","last_fullsized":"2024-04-16T16:20:34.373562+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:02:02.711699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.362179+0000","last_change":"2024-04-16T16:20:34.356912+0000","last_active":"2024-04-16T16:20:34.362179+0000","last_peered":"2024-04-16T16:20:34.362179+0000","last_clean":"2024-04-16T16:20:34.362179+0000","last_became_active":"2024-04-16T16:20:34.356511+0000","last_became_peered":"2024-04-16T16:20:34.356511+0000","last_unstale":"2024-04-16T16:20:34.362179+0000","last_undegraded":"2024-04-16T16:20:34.362179+0000","last_fullsized":"2024-04-16T16:20:34.362179+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:40:20.124474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375504+0000","last_change":"2024-04-16T16:20:34.373025+0000","last_active":"2024-04-16T16:20:34.375504+0000","last_peered":"2024-04-16T16:20:34.375504+0000","last_clean":"2024-04-16T16:20:34.375504+0000","last_became_active":"2024-04-16T16:20:34.372680+0000","last_became_peered":"2024-04-16T16:20:34.372680+0000","last_unstale":"2024-04-16T16:20:34.375504+0000","last_undegraded":"2024-04-16T16:20:34.375504+0000","last_fullsized":"2024-04-16T16:20:34.375504+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:58:34.406516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.370718+0000","last_change":"2024-04-16T16:20:34.363780+0000","last_active":"2024-04-16T16:20:34.370718+0000","last_peered":"2024-04-16T16:20:34.370718+0000","last_clean":"2024-04-16T16:20:34.370718+0000","last_became_active":"2024-04-16T16:20:34.360230+0000","last_became_peered":"2024-04-16T16:20:34.360230+0000","last_unstale":"2024-04-16T16:20:34.370718+0000","last_undegraded":"2024-04-16T16:20:34.370718+0000","last_fullsized":"2024-04-16T16:20:34.370718+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:01:42.148308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"127'4","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.382092+0000","last_change":"2024-04-16T16:20:33.351691+0000","last_active":"2024-04-16T16:20:34.382092+0000","last_peered":"2024-04-16T16:20:34.382092+0000","last_clean":"2024-04-16T16:20:34.382092+0000","last_became_active":"2024-04-16T16:20:33.348087+0000","last_became_peered":"2024-04-16T16:20:33.348087+0000","last_unstale":"2024-04-16T16:20:34.382092+0000","last_undegraded":"2024-04-16T16:20:34.382092+0000","last_fullsized":"2024-04-16T16:20:34.382092+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:23:30.553516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.376004+0000","last_change":"2024-04-16T16:20:34.373883+0000","last_active":"2024-04-16T16:20:34.376004+0000","last_peered":"2024-04-16T16:20:34.376004+0000","last_clean":"2024-04-16T16:20:34.376004+0000","last_became_active":"2024-04-16T16:20:34.372562+0000","last_became_peered":"2024-04-16T16:20:34.372562+0000","last_unstale":"2024-04-16T16:20:34.376004+0000","last_undegraded":"2024-04-16T16:20:34.376004+0000","last_fullsized":"2024-04-16T16:20:34.376004+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:39.314157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"127'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.376613+0000","last_change":"2024-04-16T16:20:33.353448+0000","last_active":"2024-04-16T16:20:34.376613+0000","last_peered":"2024-04-16T16:20:34.376613+0000","last_clean":"2024-04-16T16:20:34.376613+0000","last_became_active":"2024-04-16T16:20:33.353165+0000","last_became_peered":"2024-04-16T16:20:33.353165+0000","last_unstale":"2024-04-16T16:20:34.376613+0000","last_undegraded":"2024-04-16T16:20:34.376613+0000","last_fullsized":"2024-04-16T16:20:34.376613+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:07:39.556240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.363584+0000","last_change":"2024-04-16T16:20:34.359025+0000","last_active":"2024-04-16T16:20:34.363584+0000","last_peered":"2024-04-16T16:20:34.363584+0000","last_clean":"2024-04-16T16:20:34.363584+0000","last_became_active":"2024-04-16T16:20:34.358748+0000","last_became_peered":"2024-04-16T16:20:34.358748+0000","last_unstale":"2024-04-16T16:20:34.363584+0000","last_undegraded":"2024-04-16T16:20:34.363584+0000","last_fullsized":"2024-04-16T16:20:34.363584+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:14:29.749442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"127'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.382431+0000","last_change":"2024-04-16T16:20:33.341261+0000","last_active":"2024-04-16T16:20:34.382431+0000","last_peered":"2024-04-16T16:20:34.382431+0000","last_clean":"2024-04-16T16:20:34.382431+0000","last_became_active":"2024-04-16T16:20:33.340851+0000","last_became_peered":"2024-04-16T16:20:33.340851+0000","last_unstale":"2024-04-16T16:20:34.382431+0000","last_undegraded":"2024-04-16T16:20:34.382431+0000","last_fullsized":"2024-04-16T16:20:34.382431+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:40:55.769480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.370678+0000","last_change":"2024-04-16T16:20:34.365845+0000","last_active":"2024-04-16T16:20:34.370678+0000","last_peered":"2024-04-16T16:20:34.370678+0000","last_clean":"2024-04-16T16:20:34.370678+0000","last_became_active":"2024-04-16T16:20:34.365280+0000","last_became_peered":"2024-04-16T16:20:34.365280+0000","last_unstale":"2024-04-16T16:20:34.370678+0000","last_undegraded":"2024-04-16T16:20:34.370678+0000","last_fullsized":"2024-04-16T16:20:34.370678+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:01:42.148308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"127'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.379029+0000","last_change":"2024-04-16T16:20:33.348338+0000","last_active":"2024-04-16T16:20:34.379029+0000","last_peered":"2024-04-16T16:20:34.379029+0000","last_clean":"2024-04-16T16:20:34.379029+0000","last_became_active":"2024-04-16T16:20:33.348034+0000","last_became_peered":"2024-04-16T16:20:33.348034+0000","last_unstale":"2024-04-16T16:20:34.379029+0000","last_undegraded":"2024-04-16T16:20:34.379029+0000","last_fullsized":"2024-04-16T16:20:34.379029+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:45.738827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.366296+0000","last_change":"2024-04-16T16:20:34.360365+0000","last_active":"2024-04-16T16:20:34.366296+0000","last_peered":"2024-04-16T16:20:34.366296+0000","last_clean":"2024-04-16T16:20:34.366296+0000","last_became_active":"2024-04-16T16:20:34.358751+0000","last_became_peered":"2024-04-16T16:20:34.358751+0000","last_unstale":"2024-04-16T16:20:34.366296+0000","last_undegraded":"2024-04-16T16:20:34.366296+0000","last_fullsized":"2024-04-16T16:20:34.366296+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:48:52.180431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"127'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.376917+0000","last_change":"2024-04-16T16:20:33.345953+0000","last_active":"2024-04-16T16:20:34.376917+0000","last_peered":"2024-04-16T16:20:34.376917+0000","last_clean":"2024-04-16T16:20:34.376917+0000","last_became_active":"2024-04-16T16:20:33.345535+0000","last_became_peered":"2024-04-16T16:20:33.345535+0000","last_unstale":"2024-04-16T16:20:34.376917+0000","last_undegraded":"2024-04-16T16:20:34.376917+0000","last_fullsized":"2024-04-16T16:20:34.376917+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:38:48.896764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375778+0000","last_change":"2024-04-16T16:20:34.373542+0000","last_active":"2024-04-16T16:20:34.375778+0000","last_peered":"2024-04-16T16:20:34.375778+0000","last_clean":"2024-04-16T16:20:34.375778+0000","last_became_active":"2024-04-16T16:20:34.373201+0000","last_became_peered":"2024-04-16T16:20:34.373201+0000","last_unstale":"2024-04-16T16:20:34.375778+0000","last_undegraded":"2024-04-16T16:20:34.375778+0000","last_fullsized":"2024-04-16T16:20:34.375778+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:28:20.010499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"127'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.382087+0000","last_change":"2024-04-16T16:20:33.352927+0000","last_active":"2024-04-16T16:20:34.382087+0000","last_peered":"2024-04-16T16:20:34.382087+0000","last_clean":"2024-04-16T16:20:34.382087+0000","last_became_active":"2024-04-16T16:20:33.352562+0000","last_became_peered":"2024-04-16T16:20:33.352562+0000","last_unstale":"2024-04-16T16:20:34.382087+0000","last_undegraded":"2024-04-16T16:20:34.382087+0000","last_fullsized":"2024-04-16T16:20:34.382087+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:32:33.538843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.371724+0000","last_change":"2024-04-16T16:20:34.359904+0000","last_active":"2024-04-16T16:20:34.371724+0000","last_peered":"2024-04-16T16:20:34.371724+0000","last_clean":"2024-04-16T16:20:34.371724+0000","last_became_active":"2024-04-16T16:20:34.359109+0000","last_became_peered":"2024-04-16T16:20:34.359109+0000","last_unstale":"2024-04-16T16:20:34.371724+0000","last_undegraded":"2024-04-16T16:20:34.371724+0000","last_fullsized":"2024-04-16T16:20:34.371724+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:19.871374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.345875+0000","last_change":"2024-04-16T16:20:33.351294+0000","last_active":"2024-04-16T16:20:34.345875+0000","last_peered":"2024-04-16T16:20:34.345875+0000","last_clean":"2024-04-16T16:20:34.345875+0000","last_became_active":"2024-04-16T16:20:33.350935+0000","last_became_peered":"2024-04-16T16:20:33.350935+0000","last_unstale":"2024-04-16T16:20:34.345875+0000","last_undegraded":"2024-04-16T16:20:34.345875+0000","last_fullsized":"2024-04-16T16:20:34.345875+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:01.065024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.379292+0000","last_change":"2024-04-16T16:20:34.370132+0000","last_active":"2024-04-16T16:20:34.379292+0000","last_peered":"2024-04-16T16:20:34.379292+0000","last_clean":"2024-04-16T16:20:34.379292+0000","last_became_active":"2024-04-16T16:20:34.369818+0000","last_became_peered":"2024-04-16T16:20:34.369818+0000","last_unstale":"2024-04-16T16:20:34.379292+0000","last_undegraded":"2024-04-16T16:20:34.379292+0000","last_fullsized":"2024-04-16T16:20:34.379292+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:07:40.574514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.347541+0000","last_change":"2024-04-16T16:20:33.352013+0000","last_active":"2024-04-16T16:20:34.347541+0000","last_peered":"2024-04-16T16:20:34.347541+0000","last_clean":"2024-04-16T16:20:34.347541+0000","last_became_active":"2024-04-16T16:20:33.351732+0000","last_became_peered":"2024-04-16T16:20:33.351732+0000","last_unstale":"2024-04-16T16:20:34.347541+0000","last_undegraded":"2024-04-16T16:20:34.347541+0000","last_fullsized":"2024-04-16T16:20:34.347541+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:05:59.360238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.355309+0000","last_change":"2024-04-16T16:20:34.350062+0000","last_active":"2024-04-16T16:20:34.355309+0000","last_peered":"2024-04-16T16:20:34.355309+0000","last_clean":"2024-04-16T16:20:34.355309+0000","last_became_active":"2024-04-16T16:20:34.349778+0000","last_became_peered":"2024-04-16T16:20:34.349778+0000","last_unstale":"2024-04-16T16:20:34.355309+0000","last_undegraded":"2024-04-16T16:20:34.355309+0000","last_fullsized":"2024-04-16T16:20:34.355309+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:21:17.060299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.348521+0000","last_change":"2024-04-16T16:20:33.358407+0000","last_active":"2024-04-16T16:20:34.348521+0000","last_peered":"2024-04-16T16:20:34.348521+0000","last_clean":"2024-04-16T16:20:34.348521+0000","last_became_active":"2024-04-16T16:20:33.357978+0000","last_became_peered":"2024-04-16T16:20:33.357978+0000","last_unstale":"2024-04-16T16:20:34.348521+0000","last_undegraded":"2024-04-16T16:20:34.348521+0000","last_fullsized":"2024-04-16T16:20:34.348521+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:12:05.936112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.374806+0000","last_change":"2024-04-16T16:20:34.368759+0000","last_active":"2024-04-16T16:20:34.374806+0000","last_peered":"2024-04-16T16:20:34.374806+0000","last_clean":"2024-04-16T16:20:34.374806+0000","last_became_active":"2024-04-16T16:20:34.368485+0000","last_became_peered":"2024-04-16T16:20:34.368485+0000","last_unstale":"2024-04-16T16:20:34.374806+0000","last_undegraded":"2024-04-16T16:20:34.374806+0000","last_fullsized":"2024-04-16T16:20:34.374806+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:49.049522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.346908+0000","last_change":"2024-04-16T16:20:33.355035+0000","last_active":"2024-04-16T16:20:34.346908+0000","last_peered":"2024-04-16T16:20:34.346908+0000","last_clean":"2024-04-16T16:20:34.346908+0000","last_became_active":"2024-04-16T16:20:33.354764+0000","last_became_peered":"2024-04-16T16:20:33.354764+0000","last_unstale":"2024-04-16T16:20:34.346908+0000","last_undegraded":"2024-04-16T16:20:34.346908+0000","last_fullsized":"2024-04-16T16:20:34.346908+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:20.470756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.369272+0000","last_change":"2024-04-16T16:20:34.363304+0000","last_active":"2024-04-16T16:20:34.369272+0000","last_peered":"2024-04-16T16:20:34.369272+0000","last_clean":"2024-04-16T16:20:34.369272+0000","last_became_active":"2024-04-16T16:20:34.362930+0000","last_became_peered":"2024-04-16T16:20:34.362930+0000","last_unstale":"2024-04-16T16:20:34.369272+0000","last_undegraded":"2024-04-16T16:20:34.369272+0000","last_fullsized":"2024-04-16T16:20:34.369272+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:48:52.180431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"127'3","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.382432+0000","last_change":"2024-04-16T16:20:33.356648+0000","last_active":"2024-04-16T16:20:34.382432+0000","last_peered":"2024-04-16T16:20:34.382432+0000","last_clean":"2024-04-16T16:20:34.382432+0000","last_became_active":"2024-04-16T16:20:33.356007+0000","last_became_peered":"2024-04-16T16:20:33.356007+0000","last_unstale":"2024-04-16T16:20:34.382432+0000","last_undegraded":"2024-04-16T16:20:34.382432+0000","last_fullsized":"2024-04-16T16:20:34.382432+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:02:33.987144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.362054+0000","last_change":"2024-04-16T16:20:34.354863+0000","last_active":"2024-04-16T16:20:34.362054+0000","last_peered":"2024-04-16T16:20:34.362054+0000","last_clean":"2024-04-16T16:20:34.362054+0000","last_became_active":"2024-04-16T16:20:34.354566+0000","last_became_peered":"2024-04-16T16:20:34.354566+0000","last_unstale":"2024-04-16T16:20:34.362054+0000","last_undegraded":"2024-04-16T16:20:34.362054+0000","last_fullsized":"2024-04-16T16:20:34.362054+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:35:18.937612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"127'6","reported_seq":18,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.380813+0000","last_change":"2024-04-16T16:20:33.351465+0000","last_active":"2024-04-16T16:20:34.380813+0000","last_peered":"2024-04-16T16:20:34.380813+0000","last_clean":"2024-04-16T16:20:34.380813+0000","last_became_active":"2024-04-16T16:20:33.350065+0000","last_became_peered":"2024-04-16T16:20:33.350065+0000","last_unstale":"2024-04-16T16:20:34.380813+0000","last_undegraded":"2024-04-16T16:20:34.380813+0000","last_fullsized":"2024-04-16T16:20:34.380813+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:01.065024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.367768+0000","last_change":"2024-04-16T16:20:34.354121+0000","last_active":"2024-04-16T16:20:34.367768+0000","last_peered":"2024-04-16T16:20:34.367768+0000","last_clean":"2024-04-16T16:20:34.367768+0000","last_became_active":"2024-04-16T16:20:34.350876+0000","last_became_peered":"2024-04-16T16:20:34.350876+0000","last_unstale":"2024-04-16T16:20:34.367768+0000","last_undegraded":"2024-04-16T16:20:34.367768+0000","last_fullsized":"2024-04-16T16:20:34.367768+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:09:25.348090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.329339+0000","last_change":"2024-04-16T16:20:33.350668+0000","last_active":"2024-04-16T16:20:34.329339+0000","last_peered":"2024-04-16T16:20:34.329339+0000","last_clean":"2024-04-16T16:20:34.329339+0000","last_became_active":"2024-04-16T16:20:33.350371+0000","last_became_peered":"2024-04-16T16:20:33.350371+0000","last_unstale":"2024-04-16T16:20:34.329339+0000","last_undegraded":"2024-04-16T16:20:34.329339+0000","last_fullsized":"2024-04-16T16:20:34.329339+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:58:48.712234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375728+0000","last_change":"2024-04-16T16:20:34.362014+0000","last_active":"2024-04-16T16:20:34.375728+0000","last_peered":"2024-04-16T16:20:34.375728+0000","last_clean":"2024-04-16T16:20:34.375728+0000","last_became_active":"2024-04-16T16:20:34.361731+0000","last_became_peered":"2024-04-16T16:20:34.361731+0000","last_unstale":"2024-04-16T16:20:34.375728+0000","last_undegraded":"2024-04-16T16:20:34.375728+0000","last_fullsized":"2024-04-16T16:20:34.375728+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:49.542874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.354696+0000","last_change":"2024-04-16T16:20:33.351770+0000","last_active":"2024-04-16T16:20:34.354696+0000","last_peered":"2024-04-16T16:20:34.354696+0000","last_clean":"2024-04-16T16:20:34.354696+0000","last_became_active":"2024-04-16T16:20:33.351436+0000","last_became_peered":"2024-04-16T16:20:33.351436+0000","last_unstale":"2024-04-16T16:20:34.354696+0000","last_undegraded":"2024-04-16T16:20:34.354696+0000","last_fullsized":"2024-04-16T16:20:34.354696+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:15:43.843828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.360013+0000","last_change":"2024-04-16T16:20:34.355768+0000","last_active":"2024-04-16T16:20:34.360013+0000","last_peered":"2024-04-16T16:20:34.360013+0000","last_clean":"2024-04-16T16:20:34.360013+0000","last_became_active":"2024-04-16T16:20:34.354533+0000","last_became_peered":"2024-04-16T16:20:34.354533+0000","last_unstale":"2024-04-16T16:20:34.360013+0000","last_undegraded":"2024-04-16T16:20:34.360013+0000","last_fullsized":"2024-04-16T16:20:34.360013+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:27:17.816514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"127'8","reported_seq":20,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.381716+0000","last_change":"2024-04-16T16:20:33.352247+0000","last_active":"2024-04-16T16:20:34.381716+0000","last_peered":"2024-04-16T16:20:34.381716+0000","last_clean":"2024-04-16T16:20:34.381716+0000","last_became_active":"2024-04-16T16:20:33.350241+0000","last_became_peered":"2024-04-16T16:20:33.350241+0000","last_unstale":"2024-04-16T16:20:34.381716+0000","last_undegraded":"2024-04-16T16:20:34.381716+0000","last_fullsized":"2024-04-16T16:20:34.381716+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:48.031248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.376410+0000","last_change":"2024-04-16T16:20:34.372467+0000","last_active":"2024-04-16T16:20:34.376410+0000","last_peered":"2024-04-16T16:20:34.376410+0000","last_clean":"2024-04-16T16:20:34.376410+0000","last_became_active":"2024-04-16T16:20:34.371120+0000","last_became_peered":"2024-04-16T16:20:34.371120+0000","last_unstale":"2024-04-16T16:20:34.376410+0000","last_undegraded":"2024-04-16T16:20:34.376410+0000","last_fullsized":"2024-04-16T16:20:34.376410+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:28:20.010499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"127'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.376903+0000","last_change":"2024-04-16T16:20:33.354841+0000","last_active":"2024-04-16T16:20:34.376903+0000","last_peered":"2024-04-16T16:20:34.376903+0000","last_clean":"2024-04-16T16:20:34.376903+0000","last_became_active":"2024-04-16T16:20:33.354166+0000","last_became_peered":"2024-04-16T16:20:33.354166+0000","last_unstale":"2024-04-16T16:20:34.376903+0000","last_undegraded":"2024-04-16T16:20:34.376903+0000","last_fullsized":"2024-04-16T16:20:34.376903+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:56:41.916375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.374784+0000","last_change":"2024-04-16T16:20:34.367167+0000","last_active":"2024-04-16T16:20:34.374784+0000","last_peered":"2024-04-16T16:20:34.374784+0000","last_clean":"2024-04-16T16:20:34.374784+0000","last_became_active":"2024-04-16T16:20:34.366888+0000","last_became_peered":"2024-04-16T16:20:34.366888+0000","last_unstale":"2024-04-16T16:20:34.374784+0000","last_undegraded":"2024-04-16T16:20:34.374784+0000","last_fullsized":"2024-04-16T16:20:34.374784+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:04:57.438239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"127'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.379328+0000","last_change":"2024-04-16T16:20:33.356844+0000","last_active":"2024-04-16T16:20:34.379328+0000","last_peered":"2024-04-16T16:20:34.379328+0000","last_clean":"2024-04-16T16:20:34.379328+0000","last_became_active":"2024-04-16T16:20:33.356540+0000","last_became_peered":"2024-04-16T16:20:33.356540+0000","last_unstale":"2024-04-16T16:20:34.379328+0000","last_undegraded":"2024-04-16T16:20:34.379328+0000","last_fullsized":"2024-04-16T16:20:34.379328+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:18:45.381044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.365351+0000","last_change":"2024-04-16T16:20:34.353407+0000","last_active":"2024-04-16T16:20:34.365351+0000","last_peered":"2024-04-16T16:20:34.365351+0000","last_clean":"2024-04-16T16:20:34.365351+0000","last_became_active":"2024-04-16T16:20:34.353105+0000","last_became_peered":"2024-04-16T16:20:34.353105+0000","last_unstale":"2024-04-16T16:20:34.365351+0000","last_undegraded":"2024-04-16T16:20:34.365351+0000","last_fullsized":"2024-04-16T16:20:34.365351+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:32.514450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"127'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.382326+0000","last_change":"2024-04-16T16:20:33.352540+0000","last_active":"2024-04-16T16:20:34.382326+0000","last_peered":"2024-04-16T16:20:34.382326+0000","last_clean":"2024-04-16T16:20:34.382326+0000","last_became_active":"2024-04-16T16:20:33.351173+0000","last_became_peered":"2024-04-16T16:20:33.351173+0000","last_unstale":"2024-04-16T16:20:34.382326+0000","last_undegraded":"2024-04-16T16:20:34.382326+0000","last_fullsized":"2024-04-16T16:20:34.382326+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:09:59.502479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.378890+0000","last_change":"2024-04-16T16:20:34.369205+0000","last_active":"2024-04-16T16:20:34.378890+0000","last_peered":"2024-04-16T16:20:34.378890+0000","last_clean":"2024-04-16T16:20:34.378890+0000","last_became_active":"2024-04-16T16:20:34.368890+0000","last_became_peered":"2024-04-16T16:20:34.368890+0000","last_unstale":"2024-04-16T16:20:34.378890+0000","last_undegraded":"2024-04-16T16:20:34.378890+0000","last_fullsized":"2024-04-16T16:20:34.378890+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:34:30.635566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.348042+0000","last_change":"2024-04-16T16:20:33.353316+0000","last_active":"2024-04-16T16:20:34.348042+0000","last_peered":"2024-04-16T16:20:34.348042+0000","last_clean":"2024-04-16T16:20:34.348042+0000","last_became_active":"2024-04-16T16:20:33.352996+0000","last_became_peered":"2024-04-16T16:20:33.352996+0000","last_unstale":"2024-04-16T16:20:34.348042+0000","last_undegraded":"2024-04-16T16:20:34.348042+0000","last_fullsized":"2024-04-16T16:20:34.348042+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:02.973918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.373583+0000","last_change":"2024-04-16T16:20:34.361841+0000","last_active":"2024-04-16T16:20:34.373583+0000","last_peered":"2024-04-16T16:20:34.373583+0000","last_clean":"2024-04-16T16:20:34.373583+0000","last_became_active":"2024-04-16T16:20:34.361507+0000","last_became_peered":"2024-04-16T16:20:34.361507+0000","last_unstale":"2024-04-16T16:20:34.373583+0000","last_undegraded":"2024-04-16T16:20:34.373583+0000","last_fullsized":"2024-04-16T16:20:34.373583+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:38:47.368385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"127'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.380823+0000","last_change":"2024-04-16T16:20:33.357401+0000","last_active":"2024-04-16T16:20:34.380823+0000","last_peered":"2024-04-16T16:20:34.380823+0000","last_clean":"2024-04-16T16:20:34.380823+0000","last_became_active":"2024-04-16T16:20:33.356996+0000","last_became_peered":"2024-04-16T16:20:33.356996+0000","last_unstale":"2024-04-16T16:20:34.380823+0000","last_undegraded":"2024-04-16T16:20:34.380823+0000","last_fullsized":"2024-04-16T16:20:34.380823+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:34:29.617292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.359376+0000","last_change":"2024-04-16T16:20:34.357044+0000","last_active":"2024-04-16T16:20:34.359376+0000","last_peered":"2024-04-16T16:20:34.359376+0000","last_clean":"2024-04-16T16:20:34.359376+0000","last_became_active":"2024-04-16T16:20:34.356477+0000","last_became_peered":"2024-04-16T16:20:34.356477+0000","last_unstale":"2024-04-16T16:20:34.359376+0000","last_undegraded":"2024-04-16T16:20:34.359376+0000","last_fullsized":"2024-04-16T16:20:34.359376+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:47:39.882496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.345840+0000","last_change":"2024-04-16T16:20:33.349863+0000","last_active":"2024-04-16T16:20:34.345840+0000","last_peered":"2024-04-16T16:20:34.345840+0000","last_clean":"2024-04-16T16:20:34.345840+0000","last_became_active":"2024-04-16T16:20:33.349537+0000","last_became_peered":"2024-04-16T16:20:33.349537+0000","last_unstale":"2024-04-16T16:20:34.345840+0000","last_undegraded":"2024-04-16T16:20:34.345840+0000","last_fullsized":"2024-04-16T16:20:34.345840+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:11:34.161402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.378961+0000","last_change":"2024-04-16T16:20:34.371470+0000","last_active":"2024-04-16T16:20:34.378961+0000","last_peered":"2024-04-16T16:20:34.378961+0000","last_clean":"2024-04-16T16:20:34.378961+0000","last_became_active":"2024-04-16T16:20:34.371135+0000","last_became_peered":"2024-04-16T16:20:34.371135+0000","last_unstale":"2024-04-16T16:20:34.378961+0000","last_undegraded":"2024-04-16T16:20:34.378961+0000","last_fullsized":"2024-04-16T16:20:34.378961+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:03.992192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.349817+0000","last_change":"2024-04-16T16:20:33.353720+0000","last_active":"2024-04-16T16:20:34.349817+0000","last_peered":"2024-04-16T16:20:34.349817+0000","last_clean":"2024-04-16T16:20:34.349817+0000","last_became_active":"2024-04-16T16:20:33.353456+0000","last_became_peered":"2024-04-16T16:20:33.353456+0000","last_unstale":"2024-04-16T16:20:34.349817+0000","last_undegraded":"2024-04-16T16:20:34.349817+0000","last_fullsized":"2024-04-16T16:20:34.349817+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:14:28.731168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.362361+0000","last_change":"2024-04-16T16:20:34.358214+0000","last_active":"2024-04-16T16:20:34.362361+0000","last_peered":"2024-04-16T16:20:34.362361+0000","last_clean":"2024-04-16T16:20:34.362361+0000","last_became_active":"2024-04-16T16:20:34.356119+0000","last_became_peered":"2024-04-16T16:20:34.356119+0000","last_unstale":"2024-04-16T16:20:34.362361+0000","last_undegraded":"2024-04-16T16:20:34.362361+0000","last_fullsized":"2024-04-16T16:20:34.362361+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:32:34.557117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.338121+0000","last_change":"2024-04-16T16:20:33.354868+0000","last_active":"2024-04-16T16:20:34.338121+0000","last_peered":"2024-04-16T16:20:34.338121+0000","last_clean":"2024-04-16T16:20:34.338121+0000","last_became_active":"2024-04-16T16:20:33.354435+0000","last_became_peered":"2024-04-16T16:20:33.354435+0000","last_unstale":"2024-04-16T16:20:34.338121+0000","last_undegraded":"2024-04-16T16:20:34.338121+0000","last_fullsized":"2024-04-16T16:20:34.338121+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:45:15.064132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375171+0000","last_change":"2024-04-16T16:20:34.366623+0000","last_active":"2024-04-16T16:20:34.375171+0000","last_peered":"2024-04-16T16:20:34.375171+0000","last_clean":"2024-04-16T16:20:34.375171+0000","last_became_active":"2024-04-16T16:20:34.366351+0000","last_became_peered":"2024-04-16T16:20:34.366351+0000","last_unstale":"2024-04-16T16:20:34.375171+0000","last_undegraded":"2024-04-16T16:20:34.375171+0000","last_fullsized":"2024-04-16T16:20:34.375171+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:14:11.178395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"127'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.378924+0000","last_change":"2024-04-16T16:20:33.356462+0000","last_active":"2024-04-16T16:20:34.378924+0000","last_peered":"2024-04-16T16:20:34.378924+0000","last_clean":"2024-04-16T16:20:34.378924+0000","last_became_active":"2024-04-16T16:20:33.356186+0000","last_became_peered":"2024-04-16T16:20:33.356186+0000","last_unstale":"2024-04-16T16:20:34.378924+0000","last_undegraded":"2024-04-16T16:20:34.378924+0000","last_fullsized":"2024-04-16T16:20:34.378924+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:58:13.375230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.358471+0000","last_change":"2024-04-16T16:20:34.352959+0000","last_active":"2024-04-16T16:20:34.358471+0000","last_peered":"2024-04-16T16:20:34.358471+0000","last_clean":"2024-04-16T16:20:34.358471+0000","last_became_active":"2024-04-16T16:20:34.352686+0000","last_became_peered":"2024-04-16T16:20:34.352686+0000","last_unstale":"2024-04-16T16:20:34.358471+0000","last_undegraded":"2024-04-16T16:20:34.358471+0000","last_fullsized":"2024-04-16T16:20:34.358471+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:24:59.680663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.335693+0000","last_change":"2024-04-16T16:20:33.351516+0000","last_active":"2024-04-16T16:20:34.335693+0000","last_peered":"2024-04-16T16:20:34.335693+0000","last_clean":"2024-04-16T16:20:34.335693+0000","last_became_active":"2024-04-16T16:20:33.351205+0000","last_became_peered":"2024-04-16T16:20:33.351205+0000","last_unstale":"2024-04-16T16:20:34.335693+0000","last_undegraded":"2024-04-16T16:20:34.335693+0000","last_fullsized":"2024-04-16T16:20:34.335693+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:03:14.310460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375784+0000","last_change":"2024-04-16T16:20:34.362519+0000","last_active":"2024-04-16T16:20:34.375784+0000","last_peered":"2024-04-16T16:20:34.375784+0000","last_clean":"2024-04-16T16:20:34.375784+0000","last_became_active":"2024-04-16T16:20:34.362237+0000","last_became_peered":"2024-04-16T16:20:34.362237+0000","last_unstale":"2024-04-16T16:20:34.375784+0000","last_undegraded":"2024-04-16T16:20:34.375784+0000","last_fullsized":"2024-04-16T16:20:34.375784+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:20:03.802988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.356391+0000","last_change":"2024-04-16T16:20:33.348806+0000","last_active":"2024-04-16T16:20:34.356391+0000","last_peered":"2024-04-16T16:20:34.356391+0000","last_clean":"2024-04-16T16:20:34.356391+0000","last_became_active":"2024-04-16T16:20:33.348466+0000","last_became_peered":"2024-04-16T16:20:33.348466+0000","last_unstale":"2024-04-16T16:20:34.356391+0000","last_undegraded":"2024-04-16T16:20:34.356391+0000","last_fullsized":"2024-04-16T16:20:34.356391+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:50:08.297094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.377291+0000","last_change":"2024-04-16T16:20:34.376458+0000","last_active":"2024-04-16T16:20:34.377291+0000","last_peered":"2024-04-16T16:20:34.377291+0000","last_clean":"2024-04-16T16:20:34.377291+0000","last_became_active":"2024-04-16T16:20:34.376128+0000","last_became_peered":"2024-04-16T16:20:34.376128+0000","last_unstale":"2024-04-16T16:20:34.377291+0000","last_undegraded":"2024-04-16T16:20:34.377291+0000","last_fullsized":"2024-04-16T16:20:34.377291+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:39.314157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.341665+0000","last_change":"2024-04-16T16:20:33.350044+0000","last_active":"2024-04-16T16:20:34.341665+0000","last_peered":"2024-04-16T16:20:34.341665+0000","last_clean":"2024-04-16T16:20:34.341665+0000","last_became_active":"2024-04-16T16:20:33.349757+0000","last_became_peered":"2024-04-16T16:20:33.349757+0000","last_unstale":"2024-04-16T16:20:34.341665+0000","last_undegraded":"2024-04-16T16:20:34.341665+0000","last_fullsized":"2024-04-16T16:20:34.341665+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:00:00.700933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.365679+0000","last_change":"2024-04-16T16:20:34.361254+0000","last_active":"2024-04-16T16:20:34.365679+0000","last_peered":"2024-04-16T16:20:34.365679+0000","last_clean":"2024-04-16T16:20:34.365679+0000","last_became_active":"2024-04-16T16:20:34.357920+0000","last_became_peered":"2024-04-16T16:20:34.357920+0000","last_unstale":"2024-04-16T16:20:34.365679+0000","last_undegraded":"2024-04-16T16:20:34.365679+0000","last_fullsized":"2024-04-16T16:20:34.365679+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:32.514450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.356939+0000","last_change":"2024-04-16T16:20:33.356517+0000","last_active":"2024-04-16T16:20:34.356939+0000","last_peered":"2024-04-16T16:20:34.356939+0000","last_clean":"2024-04-16T16:20:34.356939+0000","last_became_active":"2024-04-16T16:20:33.355627+0000","last_became_peered":"2024-04-16T16:20:33.355627+0000","last_unstale":"2024-04-16T16:20:34.356939+0000","last_undegraded":"2024-04-16T16:20:34.356939+0000","last_fullsized":"2024-04-16T16:20:34.356939+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:30:04.765668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.374567+0000","last_change":"2024-04-16T16:20:34.366563+0000","last_active":"2024-04-16T16:20:34.374567+0000","last_peered":"2024-04-16T16:20:34.374567+0000","last_clean":"2024-04-16T16:20:34.374567+0000","last_became_active":"2024-04-16T16:20:34.364194+0000","last_became_peered":"2024-04-16T16:20:34.364194+0000","last_unstale":"2024-04-16T16:20:34.374567+0000","last_undegraded":"2024-04-16T16:20:34.374567+0000","last_fullsized":"2024-04-16T16:20:34.374567+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:53:36.365314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.337720+0000","last_change":"2024-04-16T16:20:33.344860+0000","last_active":"2024-04-16T16:20:34.337720+0000","last_peered":"2024-04-16T16:20:34.337720+0000","last_clean":"2024-04-16T16:20:34.337720+0000","last_became_active":"2024-04-16T16:20:33.342430+0000","last_became_peered":"2024-04-16T16:20:33.342430+0000","last_unstale":"2024-04-16T16:20:34.337720+0000","last_undegraded":"2024-04-16T16:20:34.337720+0000","last_fullsized":"2024-04-16T16:20:34.337720+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:03.458798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.368742+0000","last_change":"2024-04-16T16:20:34.357036+0000","last_active":"2024-04-16T16:20:34.368742+0000","last_peered":"2024-04-16T16:20:34.368742+0000","last_clean":"2024-04-16T16:20:34.368742+0000","last_became_active":"2024-04-16T16:20:34.356719+0000","last_became_peered":"2024-04-16T16:20:34.356719+0000","last_unstale":"2024-04-16T16:20:34.368742+0000","last_undegraded":"2024-04-16T16:20:34.368742+0000","last_fullsized":"2024-04-16T16:20:34.368742+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:01:37.023164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.345165+0000","last_change":"2024-04-16T16:20:33.351715+0000","last_active":"2024-04-16T16:20:34.345165+0000","last_peered":"2024-04-16T16:20:34.345165+0000","last_clean":"2024-04-16T16:20:34.345165+0000","last_became_active":"2024-04-16T16:20:33.351294+0000","last_became_peered":"2024-04-16T16:20:33.351294+0000","last_unstale":"2024-04-16T16:20:34.345165+0000","last_undegraded":"2024-04-16T16:20:34.345165+0000","last_fullsized":"2024-04-16T16:20:34.345165+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:40:19.106200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.374338+0000","last_change":"2024-04-16T16:20:34.363956+0000","last_active":"2024-04-16T16:20:34.374338+0000","last_peered":"2024-04-16T16:20:34.374338+0000","last_clean":"2024-04-16T16:20:34.374338+0000","last_became_active":"2024-04-16T16:20:34.363577+0000","last_became_peered":"2024-04-16T16:20:34.363577+0000","last_unstale":"2024-04-16T16:20:34.374338+0000","last_undegraded":"2024-04-16T16:20:34.374338+0000","last_fullsized":"2024-04-16T16:20:34.374338+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:54:42.925759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.338544+0000","last_change":"2024-04-16T16:20:33.348451+0000","last_active":"2024-04-16T16:20:34.338544+0000","last_peered":"2024-04-16T16:20:34.338544+0000","last_clean":"2024-04-16T16:20:34.338544+0000","last_became_active":"2024-04-16T16:20:33.347483+0000","last_became_peered":"2024-04-16T16:20:33.347483+0000","last_unstale":"2024-04-16T16:20:34.338544+0000","last_undegraded":"2024-04-16T16:20:34.338544+0000","last_fullsized":"2024-04-16T16:20:34.338544+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:21:08.945797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.365656+0000","last_change":"2024-04-16T16:20:34.359962+0000","last_active":"2024-04-16T16:20:34.365656+0000","last_peered":"2024-04-16T16:20:34.365656+0000","last_clean":"2024-04-16T16:20:34.365656+0000","last_became_active":"2024-04-16T16:20:34.357564+0000","last_became_peered":"2024-04-16T16:20:34.357564+0000","last_unstale":"2024-04-16T16:20:34.365656+0000","last_undegraded":"2024-04-16T16:20:34.365656+0000","last_fullsized":"2024-04-16T16:20:34.365656+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:21.489030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"127'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.382100+0000","last_change":"2024-04-16T16:20:33.353294+0000","last_active":"2024-04-16T16:20:34.382100+0000","last_peered":"2024-04-16T16:20:34.382100+0000","last_clean":"2024-04-16T16:20:34.382100+0000","last_became_active":"2024-04-16T16:20:33.352773+0000","last_became_peered":"2024-04-16T16:20:33.352773+0000","last_unstale":"2024-04-16T16:20:34.382100+0000","last_undegraded":"2024-04-16T16:20:34.382100+0000","last_fullsized":"2024-04-16T16:20:34.382100+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:03.036667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"117'4647","reported_seq":5175,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.335164+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:20:34.335164+0000","last_peered":"2024-04-16T16:20:34.335164+0000","last_clean":"2024-04-16T16:20:34.335164+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:20:34.335164+0000","last_undegraded":"2024-04-16T16:20:34.335164+0000","last_fullsized":"2024-04-16T16:20:34.335164+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":4647,"log_dups_size":0,"ondisk_log_size":4647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":891,"num_read_kb":1701,"num_write":307,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":891,"num_read_kb":1701,"num_write":307,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4647,"ondisk_log_size":4647,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2166784,"data_stored":1335175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574933,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1572864,"data_stored":744807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1572864,"data_stored":744807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574934,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32740,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339100,"statfs":{"total":96636764160,"available":96603238400,"internally_reserved":0,"allocated":1589248,"data_stored":751652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1572864,"data_stored":744807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1572864,"data_stored":744807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1572864,"data_stored":744807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574933,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37460,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334380,"statfs":{"total":96636764160,"available":96598405120,"internally_reserved":0,"allocated":2166784,"data_stored":1335175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:20:38.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:20:39.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.111+0000 7f8641400640 1 Processor -- start 2024-04-16T16:20:39.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.112+0000 7f8641400640 1 -- start start 2024-04-16T16:20:39.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.113+0000 7f8641400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f863c161e90 0x7f863c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:39.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.113+0000 7f8641400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f863c0d52d0 0x7f863c06ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:39.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.113+0000 7f8641400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f863c0d3080 con 0x7f863c162270 2024-04-16T16:20:39.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.113+0000 7f8641400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f863c0d2e30 con 0x7f863c0d52d0 2024-04-16T16:20:39.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.113+0000 7f8641400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f863c0d3360 con 0x7f863c161e90 2024-04-16T16:20:39.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.114+0000 7f863b400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f863c162270 0x7f863c0d3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43090/0 (socket says 172.21.15.45:43090) 2024-04-16T16:20:39.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.114+0000 7f863b400640 1 -- 172.21.15.45:0/2773477480 learned_addr learned my addr 172.21.15.45:0/2773477480 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:39.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.114+0000 7f8640a00640 1 --2- 172.21.15.45:0/2773477480 >> v2:172.21.15.45:3300/0 conn(0x7f863c0d52d0 0x7f863c06ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:39.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.114+0000 7f863be00640 1 --2- 172.21.15.45:0/2773477480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f863c161e90 0x7f863c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:39.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.114+0000 7f863aa00640 1 -- 172.21.15.45:0/2773477480 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3805745365 0 0) 0x7f863c0d3080 con 0x7f863c162270 2024-04-16T16:20:39.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.114+0000 7f863aa00640 1 -- 172.21.15.45:0/2773477480 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8620003520 con 0x7f863c162270 2024-04-16T16:20:39.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.114+0000 7f8640a00640 1 -- 172.21.15.45:0/2773477480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f863c161e90 msgr2=0x7f863c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:39.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.114+0000 7f8640a00640 1 --2- 172.21.15.45:0/2773477480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f863c161e90 0x7f863c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.114+0000 7f8640a00640 1 -- 172.21.15.45:0/2773477480 >> v1:172.21.15.45:6789/0 conn(0x7f863c162270 legacy=0x7f863c0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:39.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.114+0000 7f8640a00640 1 -- 172.21.15.45:0/2773477480 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f863c0d3640 con 0x7f863c0d52d0 2024-04-16T16:20:39.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.114+0000 7f863be00640 1 --2- 172.21.15.45:0/2773477480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f863c161e90 0x7f863c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:20:39.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.116+0000 7f8640a00640 1 --2- 172.21.15.45:0/2773477480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c0d52d0 0x7f863c06ce40 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f86340097a0 tx=0x7f863402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83fb656bcbcb8de server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:39.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.116+0000 7f863aa00640 1 -- 172.21.15.45:0/2773477480 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8634034070 con 0x7f863c0d52d0 2024-04-16T16:20:39.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.116+0000 7f863aa00640 1 -- 172.21.15.45:0/2773477480 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8634030610 con 0x7f863c0d52d0 2024-04-16T16:20:39.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.116+0000 7f863aa00640 1 -- 172.21.15.45:0/2773477480 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86340309e0 con 0x7f863c0d52d0 2024-04-16T16:20:39.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.117+0000 7f8641400640 1 -- 172.21.15.45:0/2773477480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c0d52d0 msgr2=0x7f863c06ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:39.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.117+0000 7f8641400640 1 --2- 172.21.15.45:0/2773477480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c0d52d0 0x7f863c06ce40 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f86340097a0 tx=0x7f863402b960 comp rx=0 tx=0).stop 2024-04-16T16:20:39.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.117+0000 7f8641400640 1 -- 172.21.15.45:0/2773477480 shutdown_connections 2024-04-16T16:20:39.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.117+0000 7f8641400640 1 --2- 172.21.15.45:0/2773477480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c0d52d0 0x7f863c06ce40 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.117+0000 7f8641400640 1 --2- 172.21.15.45:0/2773477480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f863c161e90 0x7f863c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.117+0000 7f8641400640 1 -- 172.21.15.45:0/2773477480 >> 172.21.15.45:0/2773477480 conn(0x7f863c0d1940 msgr2=0x7f863c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:39.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.117+0000 7f8641400640 1 -- 172.21.15.45:0/2773477480 shutdown_connections 2024-04-16T16:20:39.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.117+0000 7f8641400640 1 -- 172.21.15.45:0/2773477480 wait complete. 2024-04-16T16:20:39.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.119+0000 7f8641400640 1 Processor -- start 2024-04-16T16:20:39.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.119+0000 7f8641400640 1 -- start start 2024-04-16T16:20:39.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.120+0000 7f8641400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c0d52d0 0x7f863c156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:39.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.120+0000 7f8641400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f863c161e90 0x7f863c0b1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:39.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.120+0000 7f8641400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c162270 0x7f863c0b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:39.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.120+0000 7f8641400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f863c0d4550 con 0x7f863c162270 2024-04-16T16:20:39.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.120+0000 7f8641400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f863c0cfd10 con 0x7f863c0d52d0 2024-04-16T16:20:39.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.120+0000 7f8641400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f863c0d46d0 con 0x7f863c161e90 2024-04-16T16:20:39.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.121+0000 7f8640a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c162270 0x7f863c0b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:39.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.121+0000 7f8640a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c162270 0x7f863c0b17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44352/0 (socket says 172.21.15.45:44352) 2024-04-16T16:20:39.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.121+0000 7f8640a00640 1 -- 172.21.15.45:0/3359545499 learned_addr learned my addr 172.21.15.45:0/3359545499 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:39.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.121+0000 7f863be00640 1 --2- 172.21.15.45:0/3359545499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c0d52d0 0x7f863c156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:39.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.121+0000 7f863b400640 1 --2- 172.21.15.45:0/3359545499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f863c161e90 0x7f863c0b1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:39.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.121+0000 7f8640a00640 1 -- 172.21.15.45:0/3359545499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f863c161e90 msgr2=0x7f863c0b1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:39.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.121+0000 7f8640a00640 1 --2- 172.21.15.45:0/3359545499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f863c161e90 0x7f863c0b1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.121+0000 7f8640a00640 1 -- 172.21.15.45:0/3359545499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c0d52d0 msgr2=0x7f863c156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:39.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.121+0000 7f8640a00640 1 --2- 172.21.15.45:0/3359545499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c0d52d0 0x7f863c156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.121+0000 7f8640a00640 1 -- 172.21.15.45:0/3359545499 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f863c157160 con 0x7f863c162270 2024-04-16T16:20:39.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.121+0000 7f863be00640 1 --2- 172.21.15.45:0/3359545499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c0d52d0 0x7f863c156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:20:39.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.122+0000 7f8640a00640 1 --2- 172.21.15.45:0/3359545499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c162270 0x7f863c0b17b0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f86340097a0 tx=0x7f86340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:39.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.122+0000 7f8633e00640 1 -- 172.21.15.45:0/3359545499 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8634034040 con 0x7f863c162270 2024-04-16T16:20:39.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.122+0000 7f8633e00640 1 -- 172.21.15.45:0/3359545499 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8634030410 con 0x7f863c162270 2024-04-16T16:20:39.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.122+0000 7f8633e00640 1 -- 172.21.15.45:0/3359545499 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8634039bb0 con 0x7f863c162270 2024-04-16T16:20:39.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.122+0000 7f8641400640 1 -- 172.21.15.45:0/3359545499 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f863c0ad960 con 0x7f863c162270 2024-04-16T16:20:39.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.122+0000 7f8641400640 1 -- 172.21.15.45:0/3359545499 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f863c0aee90 con 0x7f863c162270 2024-04-16T16:20:39.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.124+0000 7f8641400640 1 -- 172.21.15.45:0/3359545499 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f863c0ba7a0 con 0x7f863c162270 2024-04-16T16:20:39.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.125+0000 7f8633e00640 1 -- 172.21.15.45:0/3359545499 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f8634037070 con 0x7f863c162270 2024-04-16T16:20:39.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.125+0000 7f8633e00640 1 --2- 172.21.15.45:0/3359545499 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f861806ca10 0x7f861806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:39.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.126+0000 7f8633e00640 1 -- 172.21.15.45:0/3359545499 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7f86340add80 con 0x7f863c162270 2024-04-16T16:20:39.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.126+0000 7f863be00640 1 --2- 172.21.15.45:0/3359545499 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f861806ca10 0x7f861806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:39.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.128+0000 7f863be00640 1 --2- 172.21.15.45:0/3359545499 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f861806ca10 0x7f861806eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f862c009b40 tx=0x7f862c003a30 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:39.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.130+0000 7f8633e00640 1 -- 172.21.15.45:0/3359545499 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f863c0ba7a0 con 0x7f863c162270 2024-04-16T16:20:39.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.331+0000 7f8641400640 1 -- 172.21.15.45:0/3359545499 --> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f863c075780 con 0x7f861806ca10 2024-04-16T16:20:39.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.352+0000 7f8633e00640 1 -- 172.21.15.45:0/3359545499 <== mgr.6793 v2:172.21.15.103:6832/1120913661 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252422 (secure 0 0 0) 0x7f863c075780 con 0x7f861806ca10 2024-04-16T16:20:39.353 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:20:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.358+0000 7f8641400640 1 -- 172.21.15.45:0/3359545499 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f861806ca10 msgr2=0x7f861806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.358+0000 7f8641400640 1 --2- 172.21.15.45:0/3359545499 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f861806ca10 0x7f861806eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f862c009b40 tx=0x7f862c003a30 comp rx=0 tx=0).stop 2024-04-16T16:20:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.358+0000 7f8641400640 1 -- 172.21.15.45:0/3359545499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c162270 msgr2=0x7f863c0b17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.358+0000 7f8641400640 1 --2- 172.21.15.45:0/3359545499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c162270 0x7f863c0b17b0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f86340097a0 tx=0x7f86340046c0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.358+0000 7f8641400640 1 -- 172.21.15.45:0/3359545499 shutdown_connections 2024-04-16T16:20:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.358+0000 7f8641400640 1 --2- 172.21.15.45:0/3359545499 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f861806ca10 0x7f861806eed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.358+0000 7f8641400640 1 --2- 172.21.15.45:0/3359545499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f863c162270 0x7f863c0b17b0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.358+0000 7f8641400640 1 --2- 172.21.15.45:0/3359545499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f863c161e90 0x7f863c0b1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.358+0000 7f8641400640 1 --2- 172.21.15.45:0/3359545499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c0d52d0 0x7f863c156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.358+0000 7f8641400640 1 -- 172.21.15.45:0/3359545499 >> 172.21.15.45:0/3359545499 conn(0x7f863c0d1940 msgr2=0x7f863c06ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.358+0000 7f8641400640 1 -- 172.21.15.45:0/3359545499 shutdown_connections 2024-04-16T16:20:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.358+0000 7f8641400640 1 -- 172.21.15.45:0/3359545499 wait complete. 2024-04-16T16:20:39.378 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-16T16:20:37.504791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":891,"num_read_kb":1701,"num_write":351,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4685,"ondisk_log_size":4685,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":267100,"kb_used_data":13464,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754707620,"statfs":{"total":773094113280,"available":772820602880,"internally_reserved":0,"allocated":13787136,"data_stored":7146037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.202295"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.372822+0000","last_change":"2024-04-16T16:20:34.361793+0000","last_active":"2024-04-16T16:20:34.372822+0000","last_peered":"2024-04-16T16:20:34.372822+0000","last_clean":"2024-04-16T16:20:34.372822+0000","last_became_active":"2024-04-16T16:20:34.361431+0000","last_became_peered":"2024-04-16T16:20:34.361431+0000","last_unstale":"2024-04-16T16:20:34.372822+0000","last_undegraded":"2024-04-16T16:20:34.372822+0000","last_fullsized":"2024-04-16T16:20:34.372822+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:22:55.733613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.364257+0000","last_change":"2024-04-16T16:20:34.358765+0000","last_active":"2024-04-16T16:20:34.364257+0000","last_peered":"2024-04-16T16:20:34.364257+0000","last_clean":"2024-04-16T16:20:34.364257+0000","last_became_active":"2024-04-16T16:20:34.358417+0000","last_became_peered":"2024-04-16T16:20:34.358417+0000","last_unstale":"2024-04-16T16:20:34.364257+0000","last_undegraded":"2024-04-16T16:20:34.364257+0000","last_fullsized":"2024-04-16T16:20:34.364257+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:48.948213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.368092+0000","last_change":"2024-04-16T16:20:34.359027+0000","last_active":"2024-04-16T16:20:34.368092+0000","last_peered":"2024-04-16T16:20:34.368092+0000","last_clean":"2024-04-16T16:20:34.368092+0000","last_became_active":"2024-04-16T16:20:34.355482+0000","last_became_peered":"2024-04-16T16:20:34.355482+0000","last_unstale":"2024-04-16T16:20:34.368092+0000","last_undegraded":"2024-04-16T16:20:34.368092+0000","last_fullsized":"2024-04-16T16:20:34.368092+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:14:11.178395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.354131+0000","last_change":"2024-04-16T16:20:34.347936+0000","last_active":"2024-04-16T16:20:34.354131+0000","last_peered":"2024-04-16T16:20:34.354131+0000","last_clean":"2024-04-16T16:20:34.354131+0000","last_became_active":"2024-04-16T16:20:34.347408+0000","last_became_peered":"2024-04-16T16:20:34.347408+0000","last_unstale":"2024-04-16T16:20:34.354131+0000","last_undegraded":"2024-04-16T16:20:34.354131+0000","last_fullsized":"2024-04-16T16:20:34.354131+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:33:30.949546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.362735+0000","last_change":"2024-04-16T16:20:34.358353+0000","last_active":"2024-04-16T16:20:34.362735+0000","last_peered":"2024-04-16T16:20:34.362735+0000","last_clean":"2024-04-16T16:20:34.362735+0000","last_became_active":"2024-04-16T16:20:34.357419+0000","last_became_peered":"2024-04-16T16:20:34.357419+0000","last_unstale":"2024-04-16T16:20:34.362735+0000","last_undegraded":"2024-04-16T16:20:34.362735+0000","last_fullsized":"2024-04-16T16:20:34.362735+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:11:35.179675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375099+0000","last_change":"2024-04-16T16:20:34.371848+0000","last_active":"2024-04-16T16:20:34.375099+0000","last_peered":"2024-04-16T16:20:34.375099+0000","last_clean":"2024-04-16T16:20:34.375099+0000","last_became_active":"2024-04-16T16:20:34.371503+0000","last_became_peered":"2024-04-16T16:20:34.371503+0000","last_unstale":"2024-04-16T16:20:34.375099+0000","last_undegraded":"2024-04-16T16:20:34.375099+0000","last_fullsized":"2024-04-16T16:20:34.375099+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:12.236456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.359296+0000","last_change":"2024-04-16T16:20:34.355494+0000","last_active":"2024-04-16T16:20:34.359296+0000","last_peered":"2024-04-16T16:20:34.359296+0000","last_clean":"2024-04-16T16:20:34.359296+0000","last_became_active":"2024-04-16T16:20:34.353830+0000","last_became_peered":"2024-04-16T16:20:34.353830+0000","last_unstale":"2024-04-16T16:20:34.359296+0000","last_undegraded":"2024-04-16T16:20:34.359296+0000","last_fullsized":"2024-04-16T16:20:34.359296+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:27:17.816514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.371747+0000","last_change":"2024-04-16T16:20:34.360213+0000","last_active":"2024-04-16T16:20:34.371747+0000","last_peered":"2024-04-16T16:20:34.371747+0000","last_clean":"2024-04-16T16:20:34.371747+0000","last_became_active":"2024-04-16T16:20:34.359511+0000","last_became_peered":"2024-04-16T16:20:34.359511+0000","last_unstale":"2024-04-16T16:20:34.371747+0000","last_undegraded":"2024-04-16T16:20:34.371747+0000","last_fullsized":"2024-04-16T16:20:34.371747+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:12.236456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.368141+0000","last_change":"2024-04-16T16:20:34.362583+0000","last_active":"2024-04-16T16:20:34.368141+0000","last_peered":"2024-04-16T16:20:34.368141+0000","last_clean":"2024-04-16T16:20:34.368141+0000","last_became_active":"2024-04-16T16:20:34.360510+0000","last_became_peered":"2024-04-16T16:20:34.360510+0000","last_unstale":"2024-04-16T16:20:34.368141+0000","last_undegraded":"2024-04-16T16:20:34.368141+0000","last_fullsized":"2024-04-16T16:20:34.368141+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:58:34.406516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.364230+0000","last_change":"2024-04-16T16:20:34.358409+0000","last_active":"2024-04-16T16:20:34.364230+0000","last_peered":"2024-04-16T16:20:34.364230+0000","last_clean":"2024-04-16T16:20:34.364230+0000","last_became_active":"2024-04-16T16:20:34.358077+0000","last_became_peered":"2024-04-16T16:20:34.358077+0000","last_unstale":"2024-04-16T16:20:34.364230+0000","last_undegraded":"2024-04-16T16:20:34.364230+0000","last_fullsized":"2024-04-16T16:20:34.364230+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:52:36.909650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.372065+0000","last_change":"2024-04-16T16:20:34.365071+0000","last_active":"2024-04-16T16:20:34.372065+0000","last_peered":"2024-04-16T16:20:34.372065+0000","last_clean":"2024-04-16T16:20:34.372065+0000","last_became_active":"2024-04-16T16:20:34.364744+0000","last_became_peered":"2024-04-16T16:20:34.364744+0000","last_unstale":"2024-04-16T16:20:34.372065+0000","last_undegraded":"2024-04-16T16:20:34.372065+0000","last_fullsized":"2024-04-16T16:20:34.372065+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:20:35.843902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.366933+0000","last_change":"2024-04-16T16:20:34.357516+0000","last_active":"2024-04-16T16:20:34.366933+0000","last_peered":"2024-04-16T16:20:34.366933+0000","last_clean":"2024-04-16T16:20:34.366933+0000","last_became_active":"2024-04-16T16:20:34.353848+0000","last_became_peered":"2024-04-16T16:20:34.353848+0000","last_unstale":"2024-04-16T16:20:34.366933+0000","last_undegraded":"2024-04-16T16:20:34.366933+0000","last_fullsized":"2024-04-16T16:20:34.366933+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:47:39.882496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.372812+0000","last_change":"2024-04-16T16:20:34.362085+0000","last_active":"2024-04-16T16:20:34.372812+0000","last_peered":"2024-04-16T16:20:34.372812+0000","last_clean":"2024-04-16T16:20:34.372812+0000","last_became_active":"2024-04-16T16:20:34.361604+0000","last_became_peered":"2024-04-16T16:20:34.361604+0000","last_unstale":"2024-04-16T16:20:34.372812+0000","last_undegraded":"2024-04-16T16:20:34.372812+0000","last_fullsized":"2024-04-16T16:20:34.372812+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:15:59.018794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.381747+0000","last_change":"2024-04-16T16:20:34.378870+0000","last_active":"2024-04-16T16:20:34.381747+0000","last_peered":"2024-04-16T16:20:34.381747+0000","last_clean":"2024-04-16T16:20:34.381747+0000","last_became_active":"2024-04-16T16:20:34.378566+0000","last_became_peered":"2024-04-16T16:20:34.378566+0000","last_unstale":"2024-04-16T16:20:34.381747+0000","last_undegraded":"2024-04-16T16:20:34.381747+0000","last_fullsized":"2024-04-16T16:20:34.381747+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:56:42.934648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.354626+0000","last_change":"2024-04-16T16:20:34.350500+0000","last_active":"2024-04-16T16:20:34.354626+0000","last_peered":"2024-04-16T16:20:34.354626+0000","last_clean":"2024-04-16T16:20:34.354626+0000","last_became_active":"2024-04-16T16:20:34.350184+0000","last_became_peered":"2024-04-16T16:20:34.350184+0000","last_unstale":"2024-04-16T16:20:34.354626+0000","last_undegraded":"2024-04-16T16:20:34.354626+0000","last_fullsized":"2024-04-16T16:20:34.354626+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:45.599894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.381403+0000","last_change":"2024-04-16T16:20:34.376104+0000","last_active":"2024-04-16T16:20:34.381403+0000","last_peered":"2024-04-16T16:20:34.381403+0000","last_clean":"2024-04-16T16:20:34.381403+0000","last_became_active":"2024-04-16T16:20:34.375771+0000","last_became_peered":"2024-04-16T16:20:34.375771+0000","last_unstale":"2024-04-16T16:20:34.381403+0000","last_undegraded":"2024-04-16T16:20:34.381403+0000","last_fullsized":"2024-04-16T16:20:34.381403+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:26:13.741282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375045+0000","last_change":"2024-04-16T16:20:34.367113+0000","last_active":"2024-04-16T16:20:34.375045+0000","last_peered":"2024-04-16T16:20:34.375045+0000","last_clean":"2024-04-16T16:20:34.375045+0000","last_became_active":"2024-04-16T16:20:34.366838+0000","last_became_peered":"2024-04-16T16:20:34.366838+0000","last_unstale":"2024-04-16T16:20:34.375045+0000","last_undegraded":"2024-04-16T16:20:34.375045+0000","last_fullsized":"2024-04-16T16:20:34.375045+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:19.871374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375151+0000","last_change":"2024-04-16T16:20:34.366810+0000","last_active":"2024-04-16T16:20:34.375151+0000","last_peered":"2024-04-16T16:20:34.375151+0000","last_clean":"2024-04-16T16:20:34.375151+0000","last_became_active":"2024-04-16T16:20:34.364143+0000","last_became_peered":"2024-04-16T16:20:34.364143+0000","last_unstale":"2024-04-16T16:20:34.375151+0000","last_undegraded":"2024-04-16T16:20:34.375151+0000","last_fullsized":"2024-04-16T16:20:34.375151+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:30:53.885057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.372039+0000","last_change":"2024-04-16T16:20:34.360432+0000","last_active":"2024-04-16T16:20:34.372039+0000","last_peered":"2024-04-16T16:20:34.372039+0000","last_clean":"2024-04-16T16:20:34.372039+0000","last_became_active":"2024-04-16T16:20:34.360078+0000","last_became_peered":"2024-04-16T16:20:34.360078+0000","last_unstale":"2024-04-16T16:20:34.372039+0000","last_undegraded":"2024-04-16T16:20:34.372039+0000","last_fullsized":"2024-04-16T16:20:34.372039+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:34:53.236404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.370441+0000","last_change":"2024-04-16T16:20:34.360690+0000","last_active":"2024-04-16T16:20:34.370441+0000","last_peered":"2024-04-16T16:20:34.370441+0000","last_clean":"2024-04-16T16:20:34.370441+0000","last_became_active":"2024-04-16T16:20:34.357170+0000","last_became_peered":"2024-04-16T16:20:34.357170+0000","last_unstale":"2024-04-16T16:20:34.370441+0000","last_undegraded":"2024-04-16T16:20:34.370441+0000","last_fullsized":"2024-04-16T16:20:34.370441+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:53:10.676759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.368303+0000","last_change":"2024-04-16T16:20:34.363295+0000","last_active":"2024-04-16T16:20:34.368303+0000","last_peered":"2024-04-16T16:20:34.368303+0000","last_clean":"2024-04-16T16:20:34.368303+0000","last_became_active":"2024-04-16T16:20:34.362980+0000","last_became_peered":"2024-04-16T16:20:34.362980+0000","last_unstale":"2024-04-16T16:20:34.368303+0000","last_undegraded":"2024-04-16T16:20:34.368303+0000","last_fullsized":"2024-04-16T16:20:34.368303+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:01:37.023164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.372002+0000","last_change":"2024-04-16T16:20:34.362680+0000","last_active":"2024-04-16T16:20:34.372002+0000","last_peered":"2024-04-16T16:20:34.372002+0000","last_clean":"2024-04-16T16:20:34.372002+0000","last_became_active":"2024-04-16T16:20:34.361137+0000","last_became_peered":"2024-04-16T16:20:34.361137+0000","last_unstale":"2024-04-16T16:20:34.372002+0000","last_undegraded":"2024-04-16T16:20:34.372002+0000","last_fullsized":"2024-04-16T16:20:34.372002+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:51:03.598091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.359986+0000","last_change":"2024-04-16T16:20:34.352114+0000","last_active":"2024-04-16T16:20:34.359986+0000","last_peered":"2024-04-16T16:20:34.359986+0000","last_clean":"2024-04-16T16:20:34.359986+0000","last_became_active":"2024-04-16T16:20:34.351760+0000","last_became_peered":"2024-04-16T16:20:34.351760+0000","last_unstale":"2024-04-16T16:20:34.359986+0000","last_undegraded":"2024-04-16T16:20:34.359986+0000","last_fullsized":"2024-04-16T16:20:34.359986+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:10:00.520752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.370584+0000","last_change":"2024-04-16T16:20:34.359311+0000","last_active":"2024-04-16T16:20:34.370584+0000","last_peered":"2024-04-16T16:20:34.370584+0000","last_clean":"2024-04-16T16:20:34.370584+0000","last_became_active":"2024-04-16T16:20:34.359033+0000","last_became_peered":"2024-04-16T16:20:34.359033+0000","last_unstale":"2024-04-16T16:20:34.370584+0000","last_undegraded":"2024-04-16T16:20:34.370584+0000","last_fullsized":"2024-04-16T16:20:34.370584+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:47.655663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375122+0000","last_change":"2024-04-16T16:20:34.364311+0000","last_active":"2024-04-16T16:20:34.375122+0000","last_peered":"2024-04-16T16:20:34.375122+0000","last_clean":"2024-04-16T16:20:34.375122+0000","last_became_active":"2024-04-16T16:20:34.362818+0000","last_became_peered":"2024-04-16T16:20:34.362818+0000","last_unstale":"2024-04-16T16:20:34.375122+0000","last_undegraded":"2024-04-16T16:20:34.375122+0000","last_fullsized":"2024-04-16T16:20:34.375122+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:58:20.232704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.380983+0000","last_change":"2024-04-16T16:20:34.375272+0000","last_active":"2024-04-16T16:20:34.380983+0000","last_peered":"2024-04-16T16:20:34.380983+0000","last_clean":"2024-04-16T16:20:34.380983+0000","last_became_active":"2024-04-16T16:20:34.374920+0000","last_became_peered":"2024-04-16T16:20:34.374920+0000","last_unstale":"2024-04-16T16:20:34.380983+0000","last_undegraded":"2024-04-16T16:20:34.380983+0000","last_fullsized":"2024-04-16T16:20:34.380983+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:26:43.829444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.380686+0000","last_change":"2024-04-16T16:20:34.372477+0000","last_active":"2024-04-16T16:20:34.380686+0000","last_peered":"2024-04-16T16:20:34.380686+0000","last_clean":"2024-04-16T16:20:34.380686+0000","last_became_active":"2024-04-16T16:20:34.369430+0000","last_became_peered":"2024-04-16T16:20:34.369430+0000","last_unstale":"2024-04-16T16:20:34.380686+0000","last_undegraded":"2024-04-16T16:20:34.380686+0000","last_fullsized":"2024-04-16T16:20:34.380686+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:25:36.383304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.371070+0000","last_change":"2024-04-16T16:20:34.362286+0000","last_active":"2024-04-16T16:20:34.371070+0000","last_peered":"2024-04-16T16:20:34.371070+0000","last_clean":"2024-04-16T16:20:34.371070+0000","last_became_active":"2024-04-16T16:20:34.359101+0000","last_became_peered":"2024-04-16T16:20:34.359101+0000","last_unstale":"2024-04-16T16:20:34.371070+0000","last_undegraded":"2024-04-16T16:20:34.371070+0000","last_fullsized":"2024-04-16T16:20:34.371070+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:45:34.981948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.367739+0000","last_change":"2024-04-16T16:20:34.353766+0000","last_active":"2024-04-16T16:20:34.367739+0000","last_peered":"2024-04-16T16:20:34.367739+0000","last_clean":"2024-04-16T16:20:34.367739+0000","last_became_active":"2024-04-16T16:20:34.352111+0000","last_became_peered":"2024-04-16T16:20:34.352111+0000","last_unstale":"2024-04-16T16:20:34.367739+0000","last_undegraded":"2024-04-16T16:20:34.367739+0000","last_fullsized":"2024-04-16T16:20:34.367739+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:24:04.314017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.373562+0000","last_change":"2024-04-16T16:20:34.359559+0000","last_active":"2024-04-16T16:20:34.373562+0000","last_peered":"2024-04-16T16:20:34.373562+0000","last_clean":"2024-04-16T16:20:34.373562+0000","last_became_active":"2024-04-16T16:20:34.359194+0000","last_became_peered":"2024-04-16T16:20:34.359194+0000","last_unstale":"2024-04-16T16:20:34.373562+0000","last_undegraded":"2024-04-16T16:20:34.373562+0000","last_fullsized":"2024-04-16T16:20:34.373562+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:02:02.711699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.362179+0000","last_change":"2024-04-16T16:20:34.356912+0000","last_active":"2024-04-16T16:20:34.362179+0000","last_peered":"2024-04-16T16:20:34.362179+0000","last_clean":"2024-04-16T16:20:34.362179+0000","last_became_active":"2024-04-16T16:20:34.356511+0000","last_became_peered":"2024-04-16T16:20:34.356511+0000","last_unstale":"2024-04-16T16:20:34.362179+0000","last_undegraded":"2024-04-16T16:20:34.362179+0000","last_fullsized":"2024-04-16T16:20:34.362179+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:40:20.124474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375504+0000","last_change":"2024-04-16T16:20:34.373025+0000","last_active":"2024-04-16T16:20:34.375504+0000","last_peered":"2024-04-16T16:20:34.375504+0000","last_clean":"2024-04-16T16:20:34.375504+0000","last_became_active":"2024-04-16T16:20:34.372680+0000","last_became_peered":"2024-04-16T16:20:34.372680+0000","last_unstale":"2024-04-16T16:20:34.375504+0000","last_undegraded":"2024-04-16T16:20:34.375504+0000","last_fullsized":"2024-04-16T16:20:34.375504+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:58:34.406516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.370718+0000","last_change":"2024-04-16T16:20:34.363780+0000","last_active":"2024-04-16T16:20:34.370718+0000","last_peered":"2024-04-16T16:20:34.370718+0000","last_clean":"2024-04-16T16:20:34.370718+0000","last_became_active":"2024-04-16T16:20:34.360230+0000","last_became_peered":"2024-04-16T16:20:34.360230+0000","last_unstale":"2024-04-16T16:20:34.370718+0000","last_undegraded":"2024-04-16T16:20:34.370718+0000","last_fullsized":"2024-04-16T16:20:34.370718+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:01:42.148308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"127'4","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.382092+0000","last_change":"2024-04-16T16:20:33.351691+0000","last_active":"2024-04-16T16:20:34.382092+0000","last_peered":"2024-04-16T16:20:34.382092+0000","last_clean":"2024-04-16T16:20:34.382092+0000","last_became_active":"2024-04-16T16:20:33.348087+0000","last_became_peered":"2024-04-16T16:20:33.348087+0000","last_unstale":"2024-04-16T16:20:34.382092+0000","last_undegraded":"2024-04-16T16:20:34.382092+0000","last_fullsized":"2024-04-16T16:20:34.382092+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:23:30.553516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.376004+0000","last_change":"2024-04-16T16:20:34.373883+0000","last_active":"2024-04-16T16:20:34.376004+0000","last_peered":"2024-04-16T16:20:34.376004+0000","last_clean":"2024-04-16T16:20:34.376004+0000","last_became_active":"2024-04-16T16:20:34.372562+0000","last_became_peered":"2024-04-16T16:20:34.372562+0000","last_unstale":"2024-04-16T16:20:34.376004+0000","last_undegraded":"2024-04-16T16:20:34.376004+0000","last_fullsized":"2024-04-16T16:20:34.376004+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:39.314157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"127'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.376613+0000","last_change":"2024-04-16T16:20:33.353448+0000","last_active":"2024-04-16T16:20:34.376613+0000","last_peered":"2024-04-16T16:20:34.376613+0000","last_clean":"2024-04-16T16:20:34.376613+0000","last_became_active":"2024-04-16T16:20:33.353165+0000","last_became_peered":"2024-04-16T16:20:33.353165+0000","last_unstale":"2024-04-16T16:20:34.376613+0000","last_undegraded":"2024-04-16T16:20:34.376613+0000","last_fullsized":"2024-04-16T16:20:34.376613+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:07:39.556240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.363584+0000","last_change":"2024-04-16T16:20:34.359025+0000","last_active":"2024-04-16T16:20:34.363584+0000","last_peered":"2024-04-16T16:20:34.363584+0000","last_clean":"2024-04-16T16:20:34.363584+0000","last_became_active":"2024-04-16T16:20:34.358748+0000","last_became_peered":"2024-04-16T16:20:34.358748+0000","last_unstale":"2024-04-16T16:20:34.363584+0000","last_undegraded":"2024-04-16T16:20:34.363584+0000","last_fullsized":"2024-04-16T16:20:34.363584+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:14:29.749442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"127'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.382431+0000","last_change":"2024-04-16T16:20:33.341261+0000","last_active":"2024-04-16T16:20:34.382431+0000","last_peered":"2024-04-16T16:20:34.382431+0000","last_clean":"2024-04-16T16:20:34.382431+0000","last_became_active":"2024-04-16T16:20:33.340851+0000","last_became_peered":"2024-04-16T16:20:33.340851+0000","last_unstale":"2024-04-16T16:20:34.382431+0000","last_undegraded":"2024-04-16T16:20:34.382431+0000","last_fullsized":"2024-04-16T16:20:34.382431+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:40:55.769480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.370678+0000","last_change":"2024-04-16T16:20:34.365845+0000","last_active":"2024-04-16T16:20:34.370678+0000","last_peered":"2024-04-16T16:20:34.370678+0000","last_clean":"2024-04-16T16:20:34.370678+0000","last_became_active":"2024-04-16T16:20:34.365280+0000","last_became_peered":"2024-04-16T16:20:34.365280+0000","last_unstale":"2024-04-16T16:20:34.370678+0000","last_undegraded":"2024-04-16T16:20:34.370678+0000","last_fullsized":"2024-04-16T16:20:34.370678+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:01:42.148308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"127'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.379029+0000","last_change":"2024-04-16T16:20:33.348338+0000","last_active":"2024-04-16T16:20:34.379029+0000","last_peered":"2024-04-16T16:20:34.379029+0000","last_clean":"2024-04-16T16:20:34.379029+0000","last_became_active":"2024-04-16T16:20:33.348034+0000","last_became_peered":"2024-04-16T16:20:33.348034+0000","last_unstale":"2024-04-16T16:20:34.379029+0000","last_undegraded":"2024-04-16T16:20:34.379029+0000","last_fullsized":"2024-04-16T16:20:34.379029+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:45.738827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.366296+0000","last_change":"2024-04-16T16:20:34.360365+0000","last_active":"2024-04-16T16:20:34.366296+0000","last_peered":"2024-04-16T16:20:34.366296+0000","last_clean":"2024-04-16T16:20:34.366296+0000","last_became_active":"2024-04-16T16:20:34.358751+0000","last_became_peered":"2024-04-16T16:20:34.358751+0000","last_unstale":"2024-04-16T16:20:34.366296+0000","last_undegraded":"2024-04-16T16:20:34.366296+0000","last_fullsized":"2024-04-16T16:20:34.366296+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:48:52.180431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"127'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.376917+0000","last_change":"2024-04-16T16:20:33.345953+0000","last_active":"2024-04-16T16:20:34.376917+0000","last_peered":"2024-04-16T16:20:34.376917+0000","last_clean":"2024-04-16T16:20:34.376917+0000","last_became_active":"2024-04-16T16:20:33.345535+0000","last_became_peered":"2024-04-16T16:20:33.345535+0000","last_unstale":"2024-04-16T16:20:34.376917+0000","last_undegraded":"2024-04-16T16:20:34.376917+0000","last_fullsized":"2024-04-16T16:20:34.376917+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:38:48.896764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375778+0000","last_change":"2024-04-16T16:20:34.373542+0000","last_active":"2024-04-16T16:20:34.375778+0000","last_peered":"2024-04-16T16:20:34.375778+0000","last_clean":"2024-04-16T16:20:34.375778+0000","last_became_active":"2024-04-16T16:20:34.373201+0000","last_became_peered":"2024-04-16T16:20:34.373201+0000","last_unstale":"2024-04-16T16:20:34.375778+0000","last_undegraded":"2024-04-16T16:20:34.375778+0000","last_fullsized":"2024-04-16T16:20:34.375778+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:28:20.010499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"127'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.382087+0000","last_change":"2024-04-16T16:20:33.352927+0000","last_active":"2024-04-16T16:20:34.382087+0000","last_peered":"2024-04-16T16:20:34.382087+0000","last_clean":"2024-04-16T16:20:34.382087+0000","last_became_active":"2024-04-16T16:20:33.352562+0000","last_became_peered":"2024-04-16T16:20:33.352562+0000","last_unstale":"2024-04-16T16:20:34.382087+0000","last_undegraded":"2024-04-16T16:20:34.382087+0000","last_fullsized":"2024-04-16T16:20:34.382087+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:32:33.538843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.371724+0000","last_change":"2024-04-16T16:20:34.359904+0000","last_active":"2024-04-16T16:20:34.371724+0000","last_peered":"2024-04-16T16:20:34.371724+0000","last_clean":"2024-04-16T16:20:34.371724+0000","last_became_active":"2024-04-16T16:20:34.359109+0000","last_became_peered":"2024-04-16T16:20:34.359109+0000","last_unstale":"2024-04-16T16:20:34.371724+0000","last_undegraded":"2024-04-16T16:20:34.371724+0000","last_fullsized":"2024-04-16T16:20:34.371724+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:19.871374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.345875+0000","last_change":"2024-04-16T16:20:33.351294+0000","last_active":"2024-04-16T16:20:34.345875+0000","last_peered":"2024-04-16T16:20:34.345875+0000","last_clean":"2024-04-16T16:20:34.345875+0000","last_became_active":"2024-04-16T16:20:33.350935+0000","last_became_peered":"2024-04-16T16:20:33.350935+0000","last_unstale":"2024-04-16T16:20:34.345875+0000","last_undegraded":"2024-04-16T16:20:34.345875+0000","last_fullsized":"2024-04-16T16:20:34.345875+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:01.065024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.379292+0000","last_change":"2024-04-16T16:20:34.370132+0000","last_active":"2024-04-16T16:20:34.379292+0000","last_peered":"2024-04-16T16:20:34.379292+0000","last_clean":"2024-04-16T16:20:34.379292+0000","last_became_active":"2024-04-16T16:20:34.369818+0000","last_became_peered":"2024-04-16T16:20:34.369818+0000","last_unstale":"2024-04-16T16:20:34.379292+0000","last_undegraded":"2024-04-16T16:20:34.379292+0000","last_fullsized":"2024-04-16T16:20:34.379292+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:07:40.574514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.347541+0000","last_change":"2024-04-16T16:20:33.352013+0000","last_active":"2024-04-16T16:20:34.347541+0000","last_peered":"2024-04-16T16:20:34.347541+0000","last_clean":"2024-04-16T16:20:34.347541+0000","last_became_active":"2024-04-16T16:20:33.351732+0000","last_became_peered":"2024-04-16T16:20:33.351732+0000","last_unstale":"2024-04-16T16:20:34.347541+0000","last_undegraded":"2024-04-16T16:20:34.347541+0000","last_fullsized":"2024-04-16T16:20:34.347541+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:05:59.360238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.355309+0000","last_change":"2024-04-16T16:20:34.350062+0000","last_active":"2024-04-16T16:20:34.355309+0000","last_peered":"2024-04-16T16:20:34.355309+0000","last_clean":"2024-04-16T16:20:34.355309+0000","last_became_active":"2024-04-16T16:20:34.349778+0000","last_became_peered":"2024-04-16T16:20:34.349778+0000","last_unstale":"2024-04-16T16:20:34.355309+0000","last_undegraded":"2024-04-16T16:20:34.355309+0000","last_fullsized":"2024-04-16T16:20:34.355309+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:21:17.060299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.348521+0000","last_change":"2024-04-16T16:20:33.358407+0000","last_active":"2024-04-16T16:20:34.348521+0000","last_peered":"2024-04-16T16:20:34.348521+0000","last_clean":"2024-04-16T16:20:34.348521+0000","last_became_active":"2024-04-16T16:20:33.357978+0000","last_became_peered":"2024-04-16T16:20:33.357978+0000","last_unstale":"2024-04-16T16:20:34.348521+0000","last_undegraded":"2024-04-16T16:20:34.348521+0000","last_fullsized":"2024-04-16T16:20:34.348521+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:12:05.936112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.374806+0000","last_change":"2024-04-16T16:20:34.368759+0000","last_active":"2024-04-16T16:20:34.374806+0000","last_peered":"2024-04-16T16:20:34.374806+0000","last_clean":"2024-04-16T16:20:34.374806+0000","last_became_active":"2024-04-16T16:20:34.368485+0000","last_became_peered":"2024-04-16T16:20:34.368485+0000","last_unstale":"2024-04-16T16:20:34.374806+0000","last_undegraded":"2024-04-16T16:20:34.374806+0000","last_fullsized":"2024-04-16T16:20:34.374806+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:49.049522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.346908+0000","last_change":"2024-04-16T16:20:33.355035+0000","last_active":"2024-04-16T16:20:34.346908+0000","last_peered":"2024-04-16T16:20:34.346908+0000","last_clean":"2024-04-16T16:20:34.346908+0000","last_became_active":"2024-04-16T16:20:33.354764+0000","last_became_peered":"2024-04-16T16:20:33.354764+0000","last_unstale":"2024-04-16T16:20:34.346908+0000","last_undegraded":"2024-04-16T16:20:34.346908+0000","last_fullsized":"2024-04-16T16:20:34.346908+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:20.470756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.369272+0000","last_change":"2024-04-16T16:20:34.363304+0000","last_active":"2024-04-16T16:20:34.369272+0000","last_peered":"2024-04-16T16:20:34.369272+0000","last_clean":"2024-04-16T16:20:34.369272+0000","last_became_active":"2024-04-16T16:20:34.362930+0000","last_became_peered":"2024-04-16T16:20:34.362930+0000","last_unstale":"2024-04-16T16:20:34.369272+0000","last_undegraded":"2024-04-16T16:20:34.369272+0000","last_fullsized":"2024-04-16T16:20:34.369272+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:48:52.180431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"127'3","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.382432+0000","last_change":"2024-04-16T16:20:33.356648+0000","last_active":"2024-04-16T16:20:34.382432+0000","last_peered":"2024-04-16T16:20:34.382432+0000","last_clean":"2024-04-16T16:20:34.382432+0000","last_became_active":"2024-04-16T16:20:33.356007+0000","last_became_peered":"2024-04-16T16:20:33.356007+0000","last_unstale":"2024-04-16T16:20:34.382432+0000","last_undegraded":"2024-04-16T16:20:34.382432+0000","last_fullsized":"2024-04-16T16:20:34.382432+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:02:33.987144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.362054+0000","last_change":"2024-04-16T16:20:34.354863+0000","last_active":"2024-04-16T16:20:34.362054+0000","last_peered":"2024-04-16T16:20:34.362054+0000","last_clean":"2024-04-16T16:20:34.362054+0000","last_became_active":"2024-04-16T16:20:34.354566+0000","last_became_peered":"2024-04-16T16:20:34.354566+0000","last_unstale":"2024-04-16T16:20:34.362054+0000","last_undegraded":"2024-04-16T16:20:34.362054+0000","last_fullsized":"2024-04-16T16:20:34.362054+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:35:18.937612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"127'6","reported_seq":18,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.380813+0000","last_change":"2024-04-16T16:20:33.351465+0000","last_active":"2024-04-16T16:20:34.380813+0000","last_peered":"2024-04-16T16:20:34.380813+0000","last_clean":"2024-04-16T16:20:34.380813+0000","last_became_active":"2024-04-16T16:20:33.350065+0000","last_became_peered":"2024-04-16T16:20:33.350065+0000","last_unstale":"2024-04-16T16:20:34.380813+0000","last_undegraded":"2024-04-16T16:20:34.380813+0000","last_fullsized":"2024-04-16T16:20:34.380813+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:01.065024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.367768+0000","last_change":"2024-04-16T16:20:34.354121+0000","last_active":"2024-04-16T16:20:34.367768+0000","last_peered":"2024-04-16T16:20:34.367768+0000","last_clean":"2024-04-16T16:20:34.367768+0000","last_became_active":"2024-04-16T16:20:34.350876+0000","last_became_peered":"2024-04-16T16:20:34.350876+0000","last_unstale":"2024-04-16T16:20:34.367768+0000","last_undegraded":"2024-04-16T16:20:34.367768+0000","last_fullsized":"2024-04-16T16:20:34.367768+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:09:25.348090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.329339+0000","last_change":"2024-04-16T16:20:33.350668+0000","last_active":"2024-04-16T16:20:34.329339+0000","last_peered":"2024-04-16T16:20:34.329339+0000","last_clean":"2024-04-16T16:20:34.329339+0000","last_became_active":"2024-04-16T16:20:33.350371+0000","last_became_peered":"2024-04-16T16:20:33.350371+0000","last_unstale":"2024-04-16T16:20:34.329339+0000","last_undegraded":"2024-04-16T16:20:34.329339+0000","last_fullsized":"2024-04-16T16:20:34.329339+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:58:48.712234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375728+0000","last_change":"2024-04-16T16:20:34.362014+0000","last_active":"2024-04-16T16:20:34.375728+0000","last_peered":"2024-04-16T16:20:34.375728+0000","last_clean":"2024-04-16T16:20:34.375728+0000","last_became_active":"2024-04-16T16:20:34.361731+0000","last_became_peered":"2024-04-16T16:20:34.361731+0000","last_unstale":"2024-04-16T16:20:34.375728+0000","last_undegraded":"2024-04-16T16:20:34.375728+0000","last_fullsized":"2024-04-16T16:20:34.375728+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:49.542874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.354696+0000","last_change":"2024-04-16T16:20:33.351770+0000","last_active":"2024-04-16T16:20:34.354696+0000","last_peered":"2024-04-16T16:20:34.354696+0000","last_clean":"2024-04-16T16:20:34.354696+0000","last_became_active":"2024-04-16T16:20:33.351436+0000","last_became_peered":"2024-04-16T16:20:33.351436+0000","last_unstale":"2024-04-16T16:20:34.354696+0000","last_undegraded":"2024-04-16T16:20:34.354696+0000","last_fullsized":"2024-04-16T16:20:34.354696+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:15:43.843828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.360013+0000","last_change":"2024-04-16T16:20:34.355768+0000","last_active":"2024-04-16T16:20:34.360013+0000","last_peered":"2024-04-16T16:20:34.360013+0000","last_clean":"2024-04-16T16:20:34.360013+0000","last_became_active":"2024-04-16T16:20:34.354533+0000","last_became_peered":"2024-04-16T16:20:34.354533+0000","last_unstale":"2024-04-16T16:20:34.360013+0000","last_undegraded":"2024-04-16T16:20:34.360013+0000","last_fullsized":"2024-04-16T16:20:34.360013+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:27:17.816514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"127'8","reported_seq":20,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.381716+0000","last_change":"2024-04-16T16:20:33.352247+0000","last_active":"2024-04-16T16:20:34.381716+0000","last_peered":"2024-04-16T16:20:34.381716+0000","last_clean":"2024-04-16T16:20:34.381716+0000","last_became_active":"2024-04-16T16:20:33.350241+0000","last_became_peered":"2024-04-16T16:20:33.350241+0000","last_unstale":"2024-04-16T16:20:34.381716+0000","last_undegraded":"2024-04-16T16:20:34.381716+0000","last_fullsized":"2024-04-16T16:20:34.381716+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:03:48.031248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.376410+0000","last_change":"2024-04-16T16:20:34.372467+0000","last_active":"2024-04-16T16:20:34.376410+0000","last_peered":"2024-04-16T16:20:34.376410+0000","last_clean":"2024-04-16T16:20:34.376410+0000","last_became_active":"2024-04-16T16:20:34.371120+0000","last_became_peered":"2024-04-16T16:20:34.371120+0000","last_unstale":"2024-04-16T16:20:34.376410+0000","last_undegraded":"2024-04-16T16:20:34.376410+0000","last_fullsized":"2024-04-16T16:20:34.376410+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:28:20.010499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"127'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.376903+0000","last_change":"2024-04-16T16:20:33.354841+0000","last_active":"2024-04-16T16:20:34.376903+0000","last_peered":"2024-04-16T16:20:34.376903+0000","last_clean":"2024-04-16T16:20:34.376903+0000","last_became_active":"2024-04-16T16:20:33.354166+0000","last_became_peered":"2024-04-16T16:20:33.354166+0000","last_unstale":"2024-04-16T16:20:34.376903+0000","last_undegraded":"2024-04-16T16:20:34.376903+0000","last_fullsized":"2024-04-16T16:20:34.376903+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:56:41.916375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.374784+0000","last_change":"2024-04-16T16:20:34.367167+0000","last_active":"2024-04-16T16:20:34.374784+0000","last_peered":"2024-04-16T16:20:34.374784+0000","last_clean":"2024-04-16T16:20:34.374784+0000","last_became_active":"2024-04-16T16:20:34.366888+0000","last_became_peered":"2024-04-16T16:20:34.366888+0000","last_unstale":"2024-04-16T16:20:34.374784+0000","last_undegraded":"2024-04-16T16:20:34.374784+0000","last_fullsized":"2024-04-16T16:20:34.374784+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:04:57.438239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"127'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.379328+0000","last_change":"2024-04-16T16:20:33.356844+0000","last_active":"2024-04-16T16:20:34.379328+0000","last_peered":"2024-04-16T16:20:34.379328+0000","last_clean":"2024-04-16T16:20:34.379328+0000","last_became_active":"2024-04-16T16:20:33.356540+0000","last_became_peered":"2024-04-16T16:20:33.356540+0000","last_unstale":"2024-04-16T16:20:34.379328+0000","last_undegraded":"2024-04-16T16:20:34.379328+0000","last_fullsized":"2024-04-16T16:20:34.379328+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:18:45.381044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.365351+0000","last_change":"2024-04-16T16:20:34.353407+0000","last_active":"2024-04-16T16:20:34.365351+0000","last_peered":"2024-04-16T16:20:34.365351+0000","last_clean":"2024-04-16T16:20:34.365351+0000","last_became_active":"2024-04-16T16:20:34.353105+0000","last_became_peered":"2024-04-16T16:20:34.353105+0000","last_unstale":"2024-04-16T16:20:34.365351+0000","last_undegraded":"2024-04-16T16:20:34.365351+0000","last_fullsized":"2024-04-16T16:20:34.365351+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:32.514450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"127'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.382326+0000","last_change":"2024-04-16T16:20:33.352540+0000","last_active":"2024-04-16T16:20:34.382326+0000","last_peered":"2024-04-16T16:20:34.382326+0000","last_clean":"2024-04-16T16:20:34.382326+0000","last_became_active":"2024-04-16T16:20:33.351173+0000","last_became_peered":"2024-04-16T16:20:33.351173+0000","last_unstale":"2024-04-16T16:20:34.382326+0000","last_undegraded":"2024-04-16T16:20:34.382326+0000","last_fullsized":"2024-04-16T16:20:34.382326+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:09:59.502479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.378890+0000","last_change":"2024-04-16T16:20:34.369205+0000","last_active":"2024-04-16T16:20:34.378890+0000","last_peered":"2024-04-16T16:20:34.378890+0000","last_clean":"2024-04-16T16:20:34.378890+0000","last_became_active":"2024-04-16T16:20:34.368890+0000","last_became_peered":"2024-04-16T16:20:34.368890+0000","last_unstale":"2024-04-16T16:20:34.378890+0000","last_undegraded":"2024-04-16T16:20:34.378890+0000","last_fullsized":"2024-04-16T16:20:34.378890+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:34:30.635566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.348042+0000","last_change":"2024-04-16T16:20:33.353316+0000","last_active":"2024-04-16T16:20:34.348042+0000","last_peered":"2024-04-16T16:20:34.348042+0000","last_clean":"2024-04-16T16:20:34.348042+0000","last_became_active":"2024-04-16T16:20:33.352996+0000","last_became_peered":"2024-04-16T16:20:33.352996+0000","last_unstale":"2024-04-16T16:20:34.348042+0000","last_undegraded":"2024-04-16T16:20:34.348042+0000","last_fullsized":"2024-04-16T16:20:34.348042+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:02.973918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.373583+0000","last_change":"2024-04-16T16:20:34.361841+0000","last_active":"2024-04-16T16:20:34.373583+0000","last_peered":"2024-04-16T16:20:34.373583+0000","last_clean":"2024-04-16T16:20:34.373583+0000","last_became_active":"2024-04-16T16:20:34.361507+0000","last_became_peered":"2024-04-16T16:20:34.361507+0000","last_unstale":"2024-04-16T16:20:34.373583+0000","last_undegraded":"2024-04-16T16:20:34.373583+0000","last_fullsized":"2024-04-16T16:20:34.373583+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:38:47.368385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"127'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.380823+0000","last_change":"2024-04-16T16:20:33.357401+0000","last_active":"2024-04-16T16:20:34.380823+0000","last_peered":"2024-04-16T16:20:34.380823+0000","last_clean":"2024-04-16T16:20:34.380823+0000","last_became_active":"2024-04-16T16:20:33.356996+0000","last_became_peered":"2024-04-16T16:20:33.356996+0000","last_unstale":"2024-04-16T16:20:34.380823+0000","last_undegraded":"2024-04-16T16:20:34.380823+0000","last_fullsized":"2024-04-16T16:20:34.380823+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:34:29.617292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.359376+0000","last_change":"2024-04-16T16:20:34.357044+0000","last_active":"2024-04-16T16:20:34.359376+0000","last_peered":"2024-04-16T16:20:34.359376+0000","last_clean":"2024-04-16T16:20:34.359376+0000","last_became_active":"2024-04-16T16:20:34.356477+0000","last_became_peered":"2024-04-16T16:20:34.356477+0000","last_unstale":"2024-04-16T16:20:34.359376+0000","last_undegraded":"2024-04-16T16:20:34.359376+0000","last_fullsized":"2024-04-16T16:20:34.359376+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:47:39.882496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.345840+0000","last_change":"2024-04-16T16:20:33.349863+0000","last_active":"2024-04-16T16:20:34.345840+0000","last_peered":"2024-04-16T16:20:34.345840+0000","last_clean":"2024-04-16T16:20:34.345840+0000","last_became_active":"2024-04-16T16:20:33.349537+0000","last_became_peered":"2024-04-16T16:20:33.349537+0000","last_unstale":"2024-04-16T16:20:34.345840+0000","last_undegraded":"2024-04-16T16:20:34.345840+0000","last_fullsized":"2024-04-16T16:20:34.345840+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:11:34.161402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.378961+0000","last_change":"2024-04-16T16:20:34.371470+0000","last_active":"2024-04-16T16:20:34.378961+0000","last_peered":"2024-04-16T16:20:34.378961+0000","last_clean":"2024-04-16T16:20:34.378961+0000","last_became_active":"2024-04-16T16:20:34.371135+0000","last_became_peered":"2024-04-16T16:20:34.371135+0000","last_unstale":"2024-04-16T16:20:34.378961+0000","last_undegraded":"2024-04-16T16:20:34.378961+0000","last_fullsized":"2024-04-16T16:20:34.378961+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:03.992192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.349817+0000","last_change":"2024-04-16T16:20:33.353720+0000","last_active":"2024-04-16T16:20:34.349817+0000","last_peered":"2024-04-16T16:20:34.349817+0000","last_clean":"2024-04-16T16:20:34.349817+0000","last_became_active":"2024-04-16T16:20:33.353456+0000","last_became_peered":"2024-04-16T16:20:33.353456+0000","last_unstale":"2024-04-16T16:20:34.349817+0000","last_undegraded":"2024-04-16T16:20:34.349817+0000","last_fullsized":"2024-04-16T16:20:34.349817+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:14:28.731168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.362361+0000","last_change":"2024-04-16T16:20:34.358214+0000","last_active":"2024-04-16T16:20:34.362361+0000","last_peered":"2024-04-16T16:20:34.362361+0000","last_clean":"2024-04-16T16:20:34.362361+0000","last_became_active":"2024-04-16T16:20:34.356119+0000","last_became_peered":"2024-04-16T16:20:34.356119+0000","last_unstale":"2024-04-16T16:20:34.362361+0000","last_undegraded":"2024-04-16T16:20:34.362361+0000","last_fullsized":"2024-04-16T16:20:34.362361+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:32:34.557117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.338121+0000","last_change":"2024-04-16T16:20:33.354868+0000","last_active":"2024-04-16T16:20:34.338121+0000","last_peered":"2024-04-16T16:20:34.338121+0000","last_clean":"2024-04-16T16:20:34.338121+0000","last_became_active":"2024-04-16T16:20:33.354435+0000","last_became_peered":"2024-04-16T16:20:33.354435+0000","last_unstale":"2024-04-16T16:20:34.338121+0000","last_undegraded":"2024-04-16T16:20:34.338121+0000","last_fullsized":"2024-04-16T16:20:34.338121+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:45:15.064132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375171+0000","last_change":"2024-04-16T16:20:34.366623+0000","last_active":"2024-04-16T16:20:34.375171+0000","last_peered":"2024-04-16T16:20:34.375171+0000","last_clean":"2024-04-16T16:20:34.375171+0000","last_became_active":"2024-04-16T16:20:34.366351+0000","last_became_peered":"2024-04-16T16:20:34.366351+0000","last_unstale":"2024-04-16T16:20:34.375171+0000","last_undegraded":"2024-04-16T16:20:34.375171+0000","last_fullsized":"2024-04-16T16:20:34.375171+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:14:11.178395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"127'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.378924+0000","last_change":"2024-04-16T16:20:33.356462+0000","last_active":"2024-04-16T16:20:34.378924+0000","last_peered":"2024-04-16T16:20:34.378924+0000","last_clean":"2024-04-16T16:20:34.378924+0000","last_became_active":"2024-04-16T16:20:33.356186+0000","last_became_peered":"2024-04-16T16:20:33.356186+0000","last_unstale":"2024-04-16T16:20:34.378924+0000","last_undegraded":"2024-04-16T16:20:34.378924+0000","last_fullsized":"2024-04-16T16:20:34.378924+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:58:13.375230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.358471+0000","last_change":"2024-04-16T16:20:34.352959+0000","last_active":"2024-04-16T16:20:34.358471+0000","last_peered":"2024-04-16T16:20:34.358471+0000","last_clean":"2024-04-16T16:20:34.358471+0000","last_became_active":"2024-04-16T16:20:34.352686+0000","last_became_peered":"2024-04-16T16:20:34.352686+0000","last_unstale":"2024-04-16T16:20:34.358471+0000","last_undegraded":"2024-04-16T16:20:34.358471+0000","last_fullsized":"2024-04-16T16:20:34.358471+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:24:59.680663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.335693+0000","last_change":"2024-04-16T16:20:33.351516+0000","last_active":"2024-04-16T16:20:34.335693+0000","last_peered":"2024-04-16T16:20:34.335693+0000","last_clean":"2024-04-16T16:20:34.335693+0000","last_became_active":"2024-04-16T16:20:33.351205+0000","last_became_peered":"2024-04-16T16:20:33.351205+0000","last_unstale":"2024-04-16T16:20:34.335693+0000","last_undegraded":"2024-04-16T16:20:34.335693+0000","last_fullsized":"2024-04-16T16:20:34.335693+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:03:14.310460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.375784+0000","last_change":"2024-04-16T16:20:34.362519+0000","last_active":"2024-04-16T16:20:34.375784+0000","last_peered":"2024-04-16T16:20:34.375784+0000","last_clean":"2024-04-16T16:20:34.375784+0000","last_became_active":"2024-04-16T16:20:34.362237+0000","last_became_peered":"2024-04-16T16:20:34.362237+0000","last_unstale":"2024-04-16T16:20:34.375784+0000","last_undegraded":"2024-04-16T16:20:34.375784+0000","last_fullsized":"2024-04-16T16:20:34.375784+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:20:03.802988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.356391+0000","last_change":"2024-04-16T16:20:33.348806+0000","last_active":"2024-04-16T16:20:34.356391+0000","last_peered":"2024-04-16T16:20:34.356391+0000","last_clean":"2024-04-16T16:20:34.356391+0000","last_became_active":"2024-04-16T16:20:33.348466+0000","last_became_peered":"2024-04-16T16:20:33.348466+0000","last_unstale":"2024-04-16T16:20:34.356391+0000","last_undegraded":"2024-04-16T16:20:34.356391+0000","last_fullsized":"2024-04-16T16:20:34.356391+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:50:08.297094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.377291+0000","last_change":"2024-04-16T16:20:34.376458+0000","last_active":"2024-04-16T16:20:34.377291+0000","last_peered":"2024-04-16T16:20:34.377291+0000","last_clean":"2024-04-16T16:20:34.377291+0000","last_became_active":"2024-04-16T16:20:34.376128+0000","last_became_peered":"2024-04-16T16:20:34.376128+0000","last_unstale":"2024-04-16T16:20:34.377291+0000","last_undegraded":"2024-04-16T16:20:34.377291+0000","last_fullsized":"2024-04-16T16:20:34.377291+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:39.314157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.341665+0000","last_change":"2024-04-16T16:20:33.350044+0000","last_active":"2024-04-16T16:20:34.341665+0000","last_peered":"2024-04-16T16:20:34.341665+0000","last_clean":"2024-04-16T16:20:34.341665+0000","last_became_active":"2024-04-16T16:20:33.349757+0000","last_became_peered":"2024-04-16T16:20:33.349757+0000","last_unstale":"2024-04-16T16:20:34.341665+0000","last_undegraded":"2024-04-16T16:20:34.341665+0000","last_fullsized":"2024-04-16T16:20:34.341665+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:00:00.700933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.365679+0000","last_change":"2024-04-16T16:20:34.361254+0000","last_active":"2024-04-16T16:20:34.365679+0000","last_peered":"2024-04-16T16:20:34.365679+0000","last_clean":"2024-04-16T16:20:34.365679+0000","last_became_active":"2024-04-16T16:20:34.357920+0000","last_became_peered":"2024-04-16T16:20:34.357920+0000","last_unstale":"2024-04-16T16:20:34.365679+0000","last_undegraded":"2024-04-16T16:20:34.365679+0000","last_fullsized":"2024-04-16T16:20:34.365679+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:32.514450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.356939+0000","last_change":"2024-04-16T16:20:33.356517+0000","last_active":"2024-04-16T16:20:34.356939+0000","last_peered":"2024-04-16T16:20:34.356939+0000","last_clean":"2024-04-16T16:20:34.356939+0000","last_became_active":"2024-04-16T16:20:33.355627+0000","last_became_peered":"2024-04-16T16:20:33.355627+0000","last_unstale":"2024-04-16T16:20:34.356939+0000","last_undegraded":"2024-04-16T16:20:34.356939+0000","last_fullsized":"2024-04-16T16:20:34.356939+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:30:04.765668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.374567+0000","last_change":"2024-04-16T16:20:34.366563+0000","last_active":"2024-04-16T16:20:34.374567+0000","last_peered":"2024-04-16T16:20:34.374567+0000","last_clean":"2024-04-16T16:20:34.374567+0000","last_became_active":"2024-04-16T16:20:34.364194+0000","last_became_peered":"2024-04-16T16:20:34.364194+0000","last_unstale":"2024-04-16T16:20:34.374567+0000","last_undegraded":"2024-04-16T16:20:34.374567+0000","last_fullsized":"2024-04-16T16:20:34.374567+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:53:36.365314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.337720+0000","last_change":"2024-04-16T16:20:33.344860+0000","last_active":"2024-04-16T16:20:34.337720+0000","last_peered":"2024-04-16T16:20:34.337720+0000","last_clean":"2024-04-16T16:20:34.337720+0000","last_became_active":"2024-04-16T16:20:33.342430+0000","last_became_peered":"2024-04-16T16:20:33.342430+0000","last_unstale":"2024-04-16T16:20:34.337720+0000","last_undegraded":"2024-04-16T16:20:34.337720+0000","last_fullsized":"2024-04-16T16:20:34.337720+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:03.458798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.368742+0000","last_change":"2024-04-16T16:20:34.357036+0000","last_active":"2024-04-16T16:20:34.368742+0000","last_peered":"2024-04-16T16:20:34.368742+0000","last_clean":"2024-04-16T16:20:34.368742+0000","last_became_active":"2024-04-16T16:20:34.356719+0000","last_became_peered":"2024-04-16T16:20:34.356719+0000","last_unstale":"2024-04-16T16:20:34.368742+0000","last_undegraded":"2024-04-16T16:20:34.368742+0000","last_fullsized":"2024-04-16T16:20:34.368742+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:01:37.023164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.345165+0000","last_change":"2024-04-16T16:20:33.351715+0000","last_active":"2024-04-16T16:20:34.345165+0000","last_peered":"2024-04-16T16:20:34.345165+0000","last_clean":"2024-04-16T16:20:34.345165+0000","last_became_active":"2024-04-16T16:20:33.351294+0000","last_became_peered":"2024-04-16T16:20:33.351294+0000","last_unstale":"2024-04-16T16:20:34.345165+0000","last_undegraded":"2024-04-16T16:20:34.345165+0000","last_fullsized":"2024-04-16T16:20:34.345165+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:40:19.106200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.374338+0000","last_change":"2024-04-16T16:20:34.363956+0000","last_active":"2024-04-16T16:20:34.374338+0000","last_peered":"2024-04-16T16:20:34.374338+0000","last_clean":"2024-04-16T16:20:34.374338+0000","last_became_active":"2024-04-16T16:20:34.363577+0000","last_became_peered":"2024-04-16T16:20:34.363577+0000","last_unstale":"2024-04-16T16:20:34.374338+0000","last_undegraded":"2024-04-16T16:20:34.374338+0000","last_fullsized":"2024-04-16T16:20:34.374338+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:54:42.925759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.338544+0000","last_change":"2024-04-16T16:20:33.348451+0000","last_active":"2024-04-16T16:20:34.338544+0000","last_peered":"2024-04-16T16:20:34.338544+0000","last_clean":"2024-04-16T16:20:34.338544+0000","last_became_active":"2024-04-16T16:20:33.347483+0000","last_became_peered":"2024-04-16T16:20:33.347483+0000","last_unstale":"2024-04-16T16:20:34.338544+0000","last_undegraded":"2024-04-16T16:20:34.338544+0000","last_fullsized":"2024-04-16T16:20:34.338544+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:21:08.945797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.365656+0000","last_change":"2024-04-16T16:20:34.359962+0000","last_active":"2024-04-16T16:20:34.365656+0000","last_peered":"2024-04-16T16:20:34.365656+0000","last_clean":"2024-04-16T16:20:34.365656+0000","last_became_active":"2024-04-16T16:20:34.357564+0000","last_became_peered":"2024-04-16T16:20:34.357564+0000","last_unstale":"2024-04-16T16:20:34.365656+0000","last_undegraded":"2024-04-16T16:20:34.365656+0000","last_fullsized":"2024-04-16T16:20:34.365656+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:33.300791+0000","last_clean_scrub_stamp":"2024-04-16T16:20:33.300791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:21.489030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"127'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.382100+0000","last_change":"2024-04-16T16:20:33.353294+0000","last_active":"2024-04-16T16:20:34.382100+0000","last_peered":"2024-04-16T16:20:34.382100+0000","last_clean":"2024-04-16T16:20:34.382100+0000","last_became_active":"2024-04-16T16:20:33.352773+0000","last_became_peered":"2024-04-16T16:20:33.352773+0000","last_unstale":"2024-04-16T16:20:34.382100+0000","last_undegraded":"2024-04-16T16:20:34.382100+0000","last_fullsized":"2024-04-16T16:20:34.382100+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:20:32.282517+0000","last_clean_scrub_stamp":"2024-04-16T16:20:32.282517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:03.036667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"117'4647","reported_seq":5175,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-16T16:20:34.335164+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:20:34.335164+0000","last_peered":"2024-04-16T16:20:34.335164+0000","last_clean":"2024-04-16T16:20:34.335164+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:20:34.335164+0000","last_undegraded":"2024-04-16T16:20:34.335164+0000","last_fullsized":"2024-04-16T16:20:34.335164+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":4647,"log_dups_size":0,"ondisk_log_size":4647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":891,"num_read_kb":1701,"num_write":307,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":891,"num_read_kb":1701,"num_write":307,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4647,"ondisk_log_size":4647,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2166784,"data_stored":1335175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574933,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1572864,"data_stored":744807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1572864,"data_stored":744807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574934,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32740,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339100,"statfs":{"total":96636764160,"available":96603238400,"internally_reserved":0,"allocated":1589248,"data_stored":751652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1572864,"data_stored":744807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1572864,"data_stored":744807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1572864,"data_stored":744807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574933,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37460,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334380,"statfs":{"total":96636764160,"available":96598405120,"internally_reserved":0,"allocated":2166784,"data_stored":1335175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:20:39.381 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-16T16:20:39.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-16T16:20:39.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.667+0000 7f0f6b000640 1 Processor -- start 2024-04-16T16:20:39.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.667+0000 7f0f6b000640 1 -- start start 2024-04-16T16:20:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.668+0000 7f0f6b000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f6406cf10 0x7f0f6406d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.668+0000 7f0f6b000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f640d2b10 con 0x7f0f6415ecc0 2024-04-16T16:20:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.668+0000 7f0f6b000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f6406d870 con 0x7f0f640d35d0 2024-04-16T16:20:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.668+0000 7f0f6b000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0f64163210 con 0x7f0f6406cf10 2024-04-16T16:20:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.669+0000 7f0f63e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0f6415ecc0 0x7f0f641626a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43096/0 (socket says 172.21.15.45:43096) 2024-04-16T16:20:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.669+0000 7f0f63e00640 1 -- 172.21.15.45:0/2791658254 learned_addr learned my addr 172.21.15.45:0/2791658254 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.669+0000 7f0f62a00640 1 -- 172.21.15.45:0/2791658254 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1141953599 0 0) 0x7f0f640d2b10 con 0x7f0f6415ecc0 2024-04-16T16:20:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.669+0000 7f0f62a00640 1 -- 172.21.15.45:0/2791658254 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f44003520 con 0x7f0f6415ecc0 2024-04-16T16:20:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.669+0000 7f0f63400640 1 --2- 172.21.15.45:0/2791658254 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f6406cf10 0x7f0f6406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.669+0000 7f0f62a00640 1 -- 172.21.15.45:0/2791658254 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 880598519 0 0) 0x7f0f44003520 con 0x7f0f6415ecc0 2024-04-16T16:20:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.669+0000 7f0f62a00640 1 -- 172.21.15.45:0/2791658254 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f6406cf10 msgr2=0x7f0f6406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.669+0000 7f0f62a00640 1 --2- 172.21.15.45:0/2791658254 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f6406cf10 0x7f0f6406d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.669+0000 7f0f62a00640 1 -- 172.21.15.45:0/2791658254 >> v1:172.21.15.103:6789/0 conn(0x7f0f640d35d0 legacy=0x7f0f640d3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.669+0000 7f0f62a00640 1 -- 172.21.15.45:0/2791658254 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f641634f0 con 0x7f0f6415ecc0 2024-04-16T16:20:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.669+0000 7f0f62a00640 1 -- 172.21.15.45:0/2791658254 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f0f54001670 con 0x7f0f6415ecc0 2024-04-16T16:20:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.670+0000 7f0f62a00640 1 -- 172.21.15.45:0/2791658254 >> v1:172.21.15.45:6789/0 conn(0x7f0f6415ecc0 legacy=0x7f0f641626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.670+0000 7f0f62a00640 1 --2- 172.21.15.45:0/2791658254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f440039e0 0x7f0f44003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.670+0000 7f0f62a00640 1 --2- 172.21.15.45:0/2791658254 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f44004800 0x7f0f44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.670+0000 7f0f62a00640 1 --2- 172.21.15.45:0/2791658254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f440051c0 0x7f0f4400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:39.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.670+0000 7f0f62a00640 1 -- 172.21.15.45:0/2791658254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0f64163210 con 0x7f0f440039e0 2024-04-16T16:20:39.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.670+0000 7f0f62a00640 1 -- 172.21.15.45:0/2791658254 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4400c070 con 0x7f0f440051c0 2024-04-16T16:20:39.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.670+0000 7f0f62a00640 1 -- 172.21.15.45:0/2791658254 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0f4400c270 con 0x7f0f44004800 2024-04-16T16:20:39.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.670+0000 7f0f63e00640 1 --2- 172.21.15.45:0/2791658254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f440039e0 0x7f0f44003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.670+0000 7f0f63400640 1 --2- 172.21.15.45:0/2791658254 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f44004800 0x7f0f44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.670+0000 7f0f63e00640 1 -- 172.21.15.45:0/2791658254 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f44004800 msgr2=0x7f0f44004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.670+0000 7f0f63e00640 1 --2- 172.21.15.45:0/2791658254 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f44004800 0x7f0f44004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.670+0000 7f0f63e00640 1 -- 172.21.15.45:0/2791658254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f440051c0 msgr2=0x7f0f4400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.670+0000 7f0f63e00640 1 --2- 172.21.15.45:0/2791658254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f440051c0 0x7f0f4400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.670+0000 7f0f63e00640 1 -- 172.21.15.45:0/2791658254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f44003520 con 0x7f0f440039e0 2024-04-16T16:20:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.671+0000 7f0f63e00640 1 --2- 172.21.15.45:0/2791658254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f440039e0 0x7f0f44003de0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400c240 tx=0x7f0f44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7355d3c742300cb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.672+0000 7f0f62a00640 1 -- 172.21.15.45:0/2791658254 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f54007d30 con 0x7f0f440039e0 2024-04-16T16:20:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.672+0000 7f0f62a00640 1 -- 172.21.15.45:0/2791658254 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f54030070 con 0x7f0f440039e0 2024-04-16T16:20:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.672+0000 7f0f62a00640 1 -- 172.21.15.45:0/2791658254 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5402cd60 con 0x7f0f440039e0 2024-04-16T16:20:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.672+0000 7f0f6b000640 1 -- 172.21.15.45:0/2791658254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f440039e0 msgr2=0x7f0f44003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.672+0000 7f0f6b000640 1 --2- 172.21.15.45:0/2791658254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f440039e0 0x7f0f44003de0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400c240 tx=0x7f0f44005830 comp rx=0 tx=0).stop 2024-04-16T16:20:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.672+0000 7f0f69000640 1 -- 172.21.15.45:0/2791658254 reap_dead start 2024-04-16T16:20:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.673+0000 7f0f6b000640 1 -- 172.21.15.45:0/2791658254 shutdown_connections 2024-04-16T16:20:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.673+0000 7f0f6b000640 1 -- 172.21.15.45:0/2791658254 >> 172.21.15.45:0/2791658254 conn(0x7f0f64073690 msgr2=0x7f0f640d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.673+0000 7f0f6b000640 1 -- 172.21.15.45:0/2791658254 shutdown_connections 2024-04-16T16:20:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.673+0000 7f0f6b000640 1 -- 172.21.15.45:0/2791658254 wait complete. 2024-04-16T16:20:39.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.675+0000 7f0f6b000640 1 Processor -- start 2024-04-16T16:20:39.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.675+0000 7f0f6b000640 1 -- start start 2024-04-16T16:20:39.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.676+0000 7f0f6b000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f440039e0 0x7f0f64150ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:39.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.676+0000 7f0f6b000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f64161220 0x7f0f641e84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:39.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.676+0000 7f0f6b000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f641616a0 0x7f0f641e8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:39.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.676+0000 7f0f6b000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0f640ba540 con 0x7f0f440039e0 2024-04-16T16:20:39.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.676+0000 7f0f6b000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0f640d30b0 con 0x7f0f64161220 2024-04-16T16:20:39.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.676+0000 7f0f6b000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0f64162db0 con 0x7f0f641616a0 2024-04-16T16:20:39.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.676+0000 7f0f63e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f440039e0 0x7f0f64150ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:39.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.676+0000 7f0f63e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f440039e0 0x7f0f64150ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44370/0 (socket says 172.21.15.45:44370) 2024-04-16T16:20:39.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.676+0000 7f0f63e00640 1 -- 172.21.15.45:0/917515243 learned_addr learned my addr 172.21.15.45:0/917515243 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:39.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.676+0000 7f0f69000640 1 --2- 172.21.15.45:0/917515243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f641616a0 0x7f0f641e8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:39.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.677+0000 7f0f63400640 1 --2- 172.21.15.45:0/917515243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f64161220 0x7f0f641e84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:39.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.677+0000 7f0f63e00640 1 -- 172.21.15.45:0/917515243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f641616a0 msgr2=0x7f0f641e8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:39.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.677+0000 7f0f63e00640 1 --2- 172.21.15.45:0/917515243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f641616a0 0x7f0f641e8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.677+0000 7f0f63e00640 1 -- 172.21.15.45:0/917515243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f64161220 msgr2=0x7f0f641e84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:39.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.677+0000 7f0f63e00640 1 --2- 172.21.15.45:0/917515243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f64161220 0x7f0f641e84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.677+0000 7f0f63e00640 1 -- 172.21.15.45:0/917515243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f64160180 con 0x7f0f440039e0 2024-04-16T16:20:39.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.677+0000 7f0f63e00640 1 --2- 172.21.15.45:0/917515243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f440039e0 0x7f0f64150ac0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f0f5402c570 tx=0x7f0f640b7a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:39.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.677+0000 7f0f5be00640 1 -- 172.21.15.45:0/917515243 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f54007d30 con 0x7f0f440039e0 2024-04-16T16:20:39.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.677+0000 7f0f5be00640 1 -- 172.21.15.45:0/917515243 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f54004bf0 con 0x7f0f440039e0 2024-04-16T16:20:39.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.677+0000 7f0f5be00640 1 -- 172.21.15.45:0/917515243 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f54007070 con 0x7f0f440039e0 2024-04-16T16:20:39.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.678+0000 7f0f6b000640 1 -- 172.21.15.45:0/917515243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f64160460 con 0x7f0f440039e0 2024-04-16T16:20:39.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.678+0000 7f0f6b000640 1 -- 172.21.15.45:0/917515243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f64160a20 con 0x7f0f440039e0 2024-04-16T16:20:39.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.679+0000 7f0f6b000640 1 -- 172.21.15.45:0/917515243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f64075780 con 0x7f0f440039e0 2024-04-16T16:20:39.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.680+0000 7f0f5be00640 1 -- 172.21.15.45:0/917515243 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f0f54047020 con 0x7f0f440039e0 2024-04-16T16:20:39.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.681+0000 7f0f5be00640 1 --2- 172.21.15.45:0/917515243 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f0f4006ca40 0x7f0f4006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:39.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.681+0000 7f0f5be00640 1 -- 172.21.15.45:0/917515243 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7f0f540ad090 con 0x7f0f440039e0 2024-04-16T16:20:39.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.682+0000 7f0f63400640 1 --2- 172.21.15.45:0/917515243 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f0f4006ca40 0x7f0f4006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:39.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.683+0000 7f0f63400640 1 --2- 172.21.15.45:0/917515243 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f0f4006ca40 0x7f0f4006ef00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0f640d1b80 tx=0x7f0f4c00b040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:39.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.686+0000 7f0f5be00640 1 -- 172.21.15.45:0/917515243 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0f5407b280 con 0x7f0f440039e0 2024-04-16T16:20:39.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.966+0000 7f0f6b000640 1 -- 172.21.15.45:0/917515243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f0f640d26c0 con 0x7f0f440039e0 2024-04-16T16:20:39.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.972+0000 7f0f5be00640 1 -- 172.21.15.45:0/917515243 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v24) ==== 190+0+147 (secure 0 0 0) 0x7f0f5407ab80 con 0x7f0f440039e0 2024-04-16T16:20:39.971 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-04-16T16:20:39.971 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ== 2024-04-16T16:20:39.971 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:20:39.971 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:20:39.971 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-16T16:20:39.972 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2024-04-16T16:20:39.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.975+0000 7f0f6b000640 1 -- 172.21.15.45:0/917515243 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f0f4006ca40 msgr2=0x7f0f4006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:39.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.975+0000 7f0f6b000640 1 --2- 172.21.15.45:0/917515243 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f0f4006ca40 0x7f0f4006ef00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0f640d1b80 tx=0x7f0f4c00b040 comp rx=0 tx=0).stop 2024-04-16T16:20:39.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.975+0000 7f0f6b000640 1 -- 172.21.15.45:0/917515243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f440039e0 msgr2=0x7f0f64150ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:39.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.975+0000 7f0f6b000640 1 --2- 172.21.15.45:0/917515243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f440039e0 0x7f0f64150ac0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f0f5402c570 tx=0x7f0f640b7a00 comp rx=0 tx=0).stop 2024-04-16T16:20:39.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.976+0000 7f0f6b000640 1 -- 172.21.15.45:0/917515243 shutdown_connections 2024-04-16T16:20:39.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.976+0000 7f0f6b000640 1 --2- 172.21.15.45:0/917515243 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f0f4006ca40 0x7f0f4006ef00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.976+0000 7f0f6b000640 1 --2- 172.21.15.45:0/917515243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f641616a0 0x7f0f641e8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.976+0000 7f0f6b000640 1 --2- 172.21.15.45:0/917515243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f64161220 0x7f0f641e84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.976+0000 7f0f6b000640 1 --2- 172.21.15.45:0/917515243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f440039e0 0x7f0f64150ac0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:39.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.976+0000 7f0f6b000640 1 -- 172.21.15.45:0/917515243 >> 172.21.15.45:0/917515243 conn(0x7f0f64073690 msgr2=0x7f0f640b8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:39.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.976+0000 7f0f6b000640 1 -- 172.21.15.45:0/917515243 shutdown_connections 2024-04-16T16:20:39.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:39.976+0000 7f0f6b000640 1 -- 172.21.15.45:0/917515243 wait complete. 2024-04-16T16:20:39.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:20:40.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.212+0000 7f917f200640 1 Processor -- start 2024-04-16T16:20:40.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.212+0000 7f917f200640 1 -- start start 2024-04-16T16:20:40.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.214+0000 7f917f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9180149460 0x7f91800b59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:40.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.214+0000 7f917f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9180072450 0x7f91800b5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:40.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.214+0000 7f917f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91800728e0 con 0x7f918014c1b0 2024-04-16T16:20:40.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.214+0000 7f917f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9180072bc0 con 0x7f9180072450 2024-04-16T16:20:40.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.214+0000 7f917f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f918006cb70 con 0x7f9180149460 2024-04-16T16:20:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.214+0000 7f917de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9180149460 0x7f91800b59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.215+0000 7f917d400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9180072450 0x7f91800b5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.215+0000 7f917de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9180149460 0x7f91800b59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:39902/0 (socket says 172.21.15.45:39902) 2024-04-16T16:20:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.215+0000 7f917e800640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f918014c1b0 0x7f91801669c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:51942/0 (socket says 172.21.15.45:51942) 2024-04-16T16:20:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.215+0000 7f917de00640 1 -- 172.21.15.45:0/1243571883 learned_addr learned my addr 172.21.15.45:0/1243571883 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.215+0000 7f917de00640 1 -- 172.21.15.45:0/1243571883 >> v2:172.21.15.103:3300/0 conn(0x7f9180072450 msgr2=0x7f91800b5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:40.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.215+0000 7f917de00640 1 --2- 172.21.15.45:0/1243571883 >> v2:172.21.15.103:3300/0 conn(0x7f9180072450 0x7f91800b5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:40.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.215+0000 7f917de00640 1 -- 172.21.15.45:0/1243571883 >> v1:172.21.15.103:6789/0 conn(0x7f918014c1b0 legacy=0x7f91801669c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:40.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.215+0000 7f917de00640 1 -- 172.21.15.45:0/1243571883 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f918006ce50 con 0x7f9180149460 2024-04-16T16:20:40.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.217+0000 7f917de00640 1 --2- 172.21.15.45:0/1243571883 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9180149460 0x7f91800b59b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9170009e50 tx=0x7f917002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c60ba0ad32b1f98 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:40.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.217+0000 7f917ca00640 1 -- 172.21.15.45:0/1243571883 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917002c9e0 con 0x7f9180149460 2024-04-16T16:20:40.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.218+0000 7f917ca00640 1 -- 172.21.15.45:0/1243571883 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f917002cb80 con 0x7f9180149460 2024-04-16T16:20:40.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.218+0000 7f917ca00640 1 -- 172.21.15.45:0/1243571883 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91700343f0 con 0x7f9180149460 2024-04-16T16:20:40.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.218+0000 7f917f200640 1 -- 172.21.15.45:0/1243571883 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9180149460 msgr2=0x7f91800b59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:40.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.218+0000 7f917f200640 1 --2- 172.21.15.45:0/1243571883 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9180149460 0x7f91800b59b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9170009e50 tx=0x7f917002be40 comp rx=0 tx=0).stop 2024-04-16T16:20:40.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.218+0000 7f917f200640 1 -- 172.21.15.45:0/1243571883 shutdown_connections 2024-04-16T16:20:40.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.218+0000 7f917f200640 1 --2- 172.21.15.45:0/1243571883 >> v2:172.21.15.103:3300/0 conn(0x7f9180072450 0x7f91800b5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:40.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.218+0000 7f917f200640 1 --2- 172.21.15.45:0/1243571883 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9180149460 0x7f91800b59b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:40.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.218+0000 7f917f200640 1 -- 172.21.15.45:0/1243571883 >> 172.21.15.45:0/1243571883 conn(0x7f91800b43a0 msgr2=0x7f91800b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:40.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.218+0000 7f917f200640 1 -- 172.21.15.45:0/1243571883 shutdown_connections 2024-04-16T16:20:40.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.218+0000 7f917f200640 1 -- 172.21.15.45:0/1243571883 wait complete. 2024-04-16T16:20:40.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.220+0000 7f917f200640 1 Processor -- start 2024-04-16T16:20:40.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.220+0000 7f917f200640 1 -- start start 2024-04-16T16:20:40.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.221+0000 7f917f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9180149460 0x7f918013ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:40.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.221+0000 7f917f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f918014c1b0 0x7f9180166140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:40.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.221+0000 7f917f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f918013c3e0 0x7f91801fc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:40.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.221+0000 7f917f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f918014c9d0 con 0x7f918013c3e0 2024-04-16T16:20:40.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.221+0000 7f917f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f918014c850 con 0x7f9180149460 2024-04-16T16:20:40.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.221+0000 7f917f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f918006e660 con 0x7f918014c1b0 2024-04-16T16:20:40.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.222+0000 7f917e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f918013c3e0 0x7f91801fc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:40.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.222+0000 7f917e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f918013c3e0 0x7f91801fc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44376/0 (socket says 172.21.15.45:44376) 2024-04-16T16:20:40.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.222+0000 7f917e800640 1 -- 172.21.15.45:0/2132063284 learned_addr learned my addr 172.21.15.45:0/2132063284 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:40.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.222+0000 7f917de00640 1 --2- 172.21.15.45:0/2132063284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9180149460 0x7f918013ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:40.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.222+0000 7f917d400640 1 --2- 172.21.15.45:0/2132063284 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f918014c1b0 0x7f9180166140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:40.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.222+0000 7f917e800640 1 -- 172.21.15.45:0/2132063284 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f918014c1b0 msgr2=0x7f9180166140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:40.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.222+0000 7f917e800640 1 --2- 172.21.15.45:0/2132063284 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f918014c1b0 0x7f9180166140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:40.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.222+0000 7f917e800640 1 -- 172.21.15.45:0/2132063284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9180149460 msgr2=0x7f918013ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:40.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.222+0000 7f917e800640 1 --2- 172.21.15.45:0/2132063284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9180149460 0x7f918013ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:40.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.222+0000 7f917e800640 1 -- 172.21.15.45:0/2132063284 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f918013f320 con 0x7f918013c3e0 2024-04-16T16:20:40.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.223+0000 7f917e800640 1 --2- 172.21.15.45:0/2132063284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f918013c3e0 0x7f91801fc130 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f9178001eb0 tx=0x7f917800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.223+0000 7f9176000640 1 -- 172.21.15.45:0/2132063284 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91780074e0 con 0x7f918013c3e0 2024-04-16T16:20:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.223+0000 7f9176000640 1 -- 172.21.15.45:0/2132063284 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9178007680 con 0x7f918013c3e0 2024-04-16T16:20:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.223+0000 7f9176000640 1 -- 172.21.15.45:0/2132063284 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91780049e0 con 0x7f918013c3e0 2024-04-16T16:20:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.223+0000 7f917f200640 1 -- 172.21.15.45:0/2132063284 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f918013bdf0 con 0x7f918013c3e0 2024-04-16T16:20:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.223+0000 7f917f200640 1 -- 172.21.15.45:0/2132063284 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9180146470 con 0x7f918013c3e0 2024-04-16T16:20:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.224+0000 7f917f200640 1 -- 172.21.15.45:0/2132063284 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91800715f0 con 0x7f918013c3e0 2024-04-16T16:20:40.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.225+0000 7f9176000640 1 -- 172.21.15.45:0/2132063284 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f917801a020 con 0x7f918013c3e0 2024-04-16T16:20:40.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.226+0000 7f9176000640 1 --2- 172.21.15.45:0/2132063284 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f915c06ca60 0x7f915c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:40.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.226+0000 7f9176000640 1 -- 172.21.15.45:0/2132063284 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7f917808b380 con 0x7f918013c3e0 2024-04-16T16:20:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.227+0000 7f917de00640 1 --2- 172.21.15.45:0/2132063284 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f915c06ca60 0x7f915c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:40.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.228+0000 7f917de00640 1 --2- 172.21.15.45:0/2132063284 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f915c06ca60 0x7f915c06ef20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9170035000 tx=0x7f9170030000 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:40.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.231+0000 7f9176000640 1 -- 172.21.15.45:0/2132063284 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9178059710 con 0x7f918013c3e0 2024-04-16T16:20:40.289 INFO:tasks.ceph.mds.c.smithi045.stderr:2024-04-16T16:20:40.290+0000 7fba8c000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:20:40.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.482+0000 7f917f200640 1 -- 172.21.15.45:0/2132063284 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91800b4820 con 0x7f918013c3e0 2024-04-16T16:20:40.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.483+0000 7f9176000640 1 -- 172.21.15.45:0/2132063284 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5186 (secure 0 0 0) 0x7f9178059010 con 0x7f918013c3e0 2024-04-16T16:20:40.483 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:40.483 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/929218719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":929218719},{"type":"v1","addr":"172.21.15.103:6835","nonce":929218719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6323,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1619719405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1619719405},{"type":"v1","addr":"172.21.15.103:6837","nonce":1619719405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":8640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/3613288118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3613288118},{"type":"v1","addr":"172.21.15.45:6834","nonce":3613288118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:20:33.816422+0000","modified":"2024-04-16T16:20:36.891870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8643":{"gid":8643,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2060312705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2060312705},{"type":"v1","addr":"172.21.15.45:6835","nonce":2060312705}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8643,"qdb_cluster":[8643]},"id":9}]} 2024-04-16T16:20:40.483 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 91 2024-04-16T16:20:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.486+0000 7f917f200640 1 -- 172.21.15.45:0/2132063284 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f915c06ca60 msgr2=0x7f915c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.486+0000 7f917f200640 1 --2- 172.21.15.45:0/2132063284 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f915c06ca60 0x7f915c06ef20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9170035000 tx=0x7f9170030000 comp rx=0 tx=0).stop 2024-04-16T16:20:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.486+0000 7f917f200640 1 -- 172.21.15.45:0/2132063284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f918013c3e0 msgr2=0x7f91801fc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.486+0000 7f917f200640 1 --2- 172.21.15.45:0/2132063284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f918013c3e0 0x7f91801fc130 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f9178001eb0 tx=0x7f917800c820 comp rx=0 tx=0).stop 2024-04-16T16:20:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.486+0000 7f917f200640 1 -- 172.21.15.45:0/2132063284 shutdown_connections 2024-04-16T16:20:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.486+0000 7f917f200640 1 --2- 172.21.15.45:0/2132063284 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f915c06ca60 0x7f915c06ef20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:40.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.486+0000 7f917f200640 1 --2- 172.21.15.45:0/2132063284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f918013c3e0 0x7f91801fc130 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:40.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.486+0000 7f917f200640 1 --2- 172.21.15.45:0/2132063284 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f918014c1b0 0x7f9180166140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:40.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.486+0000 7f917f200640 1 --2- 172.21.15.45:0/2132063284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9180149460 0x7f918013ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:40.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.486+0000 7f917f200640 1 -- 172.21.15.45:0/2132063284 >> 172.21.15.45:0/2132063284 conn(0x7f91800b43a0 msgr2=0x7f918014a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:40.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.486+0000 7f917f200640 1 -- 172.21.15.45:0/2132063284 shutdown_connections 2024-04-16T16:20:40.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:40.486+0000 7f917f200640 1 -- 172.21.15.45:0/2132063284 wait complete. 2024-04-16T16:20:40.907 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 91, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:20:33.816422+0000', 'modified': '2024-04-16T16:20:36.891870+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8643}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8643': {'gid': 8643, 'name': 'c', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6835/2060312705', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6833', 'nonce': 2060312705}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 2060312705}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8643, 'qdb_cluster': [8643]} 2024-04-16T16:20:40.908 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-16T16:20:40.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-16T16:20:41.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.139+0000 7f9a0a200640 1 Processor -- start 2024-04-16T16:20:41.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.139+0000 7f9a0a200640 1 -- start start 2024-04-16T16:20:41.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.140+0000 7f9a0a200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9a04147550 0x7f9a04072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.140+0000 7f9a0a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a040726c0 0x7f9a040b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.140+0000 7f9a0a200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a040b5d00 con 0x7f9a040b58e0 2024-04-16T16:20:41.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.140+0000 7f9a0a200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9a0406fc50 con 0x7f9a04147550 2024-04-16T16:20:41.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.140+0000 7f9a0a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9a04072b20 con 0x7f9a040726c0 2024-04-16T16:20:41.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.140+0000 7f9a03e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9a040b58e0 0x7f9a04162670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43112/0 (socket says 172.21.15.45:43112) 2024-04-16T16:20:41.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.140+0000 7f9a03e00640 1 -- 172.21.15.45:0/803036763 learned_addr learned my addr 172.21.15.45:0/803036763 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:41.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.140+0000 7f9a03400640 1 --2- 172.21.15.45:0/803036763 >> v2:172.21.15.45:3300/0 conn(0x7f9a04147550 0x7f9a04072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.141+0000 7f9a02a00640 1 --2- 172.21.15.45:0/803036763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a040726c0 0x7f9a040b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.141+0000 7f9a02000640 1 -- 172.21.15.45:0/803036763 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3354188845 0 0) 0x7f99fc001490 con 0x7f9a040b58e0 2024-04-16T16:20:41.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.141+0000 7f9a02000640 1 -- 172.21.15.45:0/803036763 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99e8003520 con 0x7f9a040b58e0 2024-04-16T16:20:41.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.141+0000 7f9a03400640 1 -- 172.21.15.45:0/803036763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a040726c0 msgr2=0x7f9a040b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:41.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.141+0000 7f9a03400640 1 --2- 172.21.15.45:0/803036763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a040726c0 0x7f9a040b53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.141+0000 7f9a03400640 1 -- 172.21.15.45:0/803036763 >> v1:172.21.15.45:6789/0 conn(0x7f9a040b58e0 legacy=0x7f9a04162670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.141+0000 7f9a03400640 1 -- 172.21.15.45:0/803036763 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a0406df00 con 0x7f9a04147550 2024-04-16T16:20:41.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.142+0000 7f9a03400640 1 --2- 172.21.15.45:0/803036763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a04147550 0x7f9a04072180 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f99f40097a0 tx=0x7f99f402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30bcbfc42cb16ff2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:41.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.143+0000 7f9a02000640 1 -- 172.21.15.45:0/803036763 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f4035020 con 0x7f9a04147550 2024-04-16T16:20:41.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.143+0000 7f9a02000640 1 -- 172.21.15.45:0/803036763 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f4031810 con 0x7f9a04147550 2024-04-16T16:20:41.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.143+0000 7f9a02000640 1 -- 172.21.15.45:0/803036763 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f4031bc0 con 0x7f9a04147550 2024-04-16T16:20:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.143+0000 7f9a0a200640 1 -- 172.21.15.45:0/803036763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a04147550 msgr2=0x7f9a04072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.143+0000 7f9a0a200640 1 --2- 172.21.15.45:0/803036763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a04147550 0x7f9a04072180 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f99f40097a0 tx=0x7f99f402b8b0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.144+0000 7f9a0a200640 1 -- 172.21.15.45:0/803036763 shutdown_connections 2024-04-16T16:20:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.144+0000 7f9a0a200640 1 --2- 172.21.15.45:0/803036763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a040726c0 0x7f9a040b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.144+0000 7f9a0a200640 1 --2- 172.21.15.45:0/803036763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a04147550 0x7f9a04072180 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.144+0000 7f9a0a200640 1 -- 172.21.15.45:0/803036763 >> 172.21.15.45:0/803036763 conn(0x7f9a0406b930 msgr2=0x7f9a0406bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.144+0000 7f9a0a200640 1 -- 172.21.15.45:0/803036763 shutdown_connections 2024-04-16T16:20:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.144+0000 7f9a0a200640 1 -- 172.21.15.45:0/803036763 wait complete. 2024-04-16T16:20:41.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.146+0000 7f9a0a200640 1 Processor -- start 2024-04-16T16:20:41.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.146+0000 7f9a0a200640 1 -- start start 2024-04-16T16:20:41.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.147+0000 7f9a0a200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a040726c0 0x7f9a041f51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.147+0000 7f9a0a200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a040b58e0 0x7f9a041f1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.147+0000 7f9a0a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a04147550 0x7f9a0413f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.147+0000 7f9a0a200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0406e7a0 con 0x7f9a040726c0 2024-04-16T16:20:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.147+0000 7f9a0a200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0406e4c0 con 0x7f9a040b58e0 2024-04-16T16:20:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.147+0000 7f9a0a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9a0406ea80 con 0x7f9a04147550 2024-04-16T16:20:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.147+0000 7f9a03400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a040726c0 0x7f9a041f51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.147+0000 7f9a03400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a040726c0 0x7f9a041f51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44398/0 (socket says 172.21.15.45:44398) 2024-04-16T16:20:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.147+0000 7f9a03400640 1 -- 172.21.15.45:0/1137867605 learned_addr learned my addr 172.21.15.45:0/1137867605 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.147+0000 7f9a03400640 1 -- 172.21.15.45:0/1137867605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a04147550 msgr2=0x7f9a0413f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.147+0000 7f9a03e00640 1 --2- 172.21.15.45:0/1137867605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a04147550 0x7f9a0413f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.147+0000 7f9a02a00640 1 --2- 172.21.15.45:0/1137867605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a040b58e0 0x7f9a041f1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.147+0000 7f9a03400640 1 --2- 172.21.15.45:0/1137867605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a04147550 0x7f9a0413f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.148+0000 7f9a03400640 1 -- 172.21.15.45:0/1137867605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a040b58e0 msgr2=0x7f9a041f1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.148+0000 7f9a03400640 1 --2- 172.21.15.45:0/1137867605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a040b58e0 0x7f9a041f1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.148+0000 7f9a03400640 1 -- 172.21.15.45:0/1137867605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a041f5700 con 0x7f9a040726c0 2024-04-16T16:20:41.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.148+0000 7f9a03400640 1 --2- 172.21.15.45:0/1137867605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a040726c0 0x7f9a041f51c0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f99f40098c0 tx=0x7f99f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.148+0000 7f99fb400640 1 -- 172.21.15.45:0/1137867605 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f4035070 con 0x7f9a040726c0 2024-04-16T16:20:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.148+0000 7f99fb400640 1 -- 172.21.15.45:0/1137867605 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f4003610 con 0x7f9a040726c0 2024-04-16T16:20:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.148+0000 7f99fb400640 1 -- 172.21.15.45:0/1137867605 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f40039c0 con 0x7f9a040726c0 2024-04-16T16:20:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.148+0000 7f9a0a200640 1 -- 172.21.15.45:0/1137867605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a0413da50 con 0x7f9a040726c0 2024-04-16T16:20:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.148+0000 7f9a0a200640 1 -- 172.21.15.45:0/1137867605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a0413e010 con 0x7f9a040726c0 2024-04-16T16:20:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.151+0000 7f99fb400640 1 -- 172.21.15.45:0/1137867605 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f99f4038020 con 0x7f9a040726c0 2024-04-16T16:20:41.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.152+0000 7f99fb400640 1 --2- 172.21.15.45:0/1137867605 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f99e406ca10 0x7f99e406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.152+0000 7f99fb400640 1 -- 172.21.15.45:0/1137867605 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7f99f40ad8d0 con 0x7f9a040726c0 2024-04-16T16:20:41.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.152+0000 7f9a0cfba740 1 Processor -- start 2024-04-16T16:20:41.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.152+0000 7f9a02a00640 1 --2- 172.21.15.45:0/1137867605 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f99e406ca10 0x7f99e406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.153+0000 7f9a0cfba740 1 -- start start 2024-04-16T16:20:41.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.153+0000 7f9a0cfba740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3526650 0x5639e3545790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.153+0000 7f9a0cfba740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e34ff330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.153+0000 7f9a0cfba740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e34ff870 0x5639e34fd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.153+0000 7f9a0cfba740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5639e34649c0 con 0x5639e3526a30 2024-04-16T16:20:41.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.153+0000 7f9a0cfba740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5639e340b560 con 0x5639e3526650 2024-04-16T16:20:41.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.153+0000 7f9a0cfba740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5639e3460980 con 0x5639e34ff870 2024-04-16T16:20:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.154+0000 7f9a02a00640 1 --2- 172.21.15.45:0/1137867605 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f99e406ca10 0x7f99e406eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9a04163320 tx=0x7f99f0008040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.154+0000 7f9a02a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e34ff330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.154+0000 7f9a02a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e34ff330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44414/0 (socket says 172.21.15.45:44414) 2024-04-16T16:20:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.154+0000 7f9a02a00640 1 -- 172.21.15.45:0/257120967 learned_addr learned my addr 172.21.15.45:0/257120967 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.154+0000 7f9a03e00640 1 --2- 172.21.15.45:0/257120967 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e34ff870 0x5639e34fd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.154+0000 7f9a02a00640 1 -- 172.21.15.45:0/257120967 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e34ff870 msgr2=0x5639e34fd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.154+0000 7f9a02a00640 1 --2- 172.21.15.45:0/257120967 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e34ff870 0x5639e34fd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.154+0000 7f9a02a00640 1 -- 172.21.15.45:0/257120967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3526650 msgr2=0x5639e3545790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.154+0000 7f9a02a00640 1 --2- 172.21.15.45:0/257120967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3526650 0x5639e3545790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.154+0000 7f9a02a00640 1 -- 172.21.15.45:0/257120967 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639e31772d0 con 0x5639e3526a30 2024-04-16T16:20:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.154+0000 7f9a03e00640 1 --2- 172.21.15.45:0/257120967 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e34ff870 0x5639e34fd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.155+0000 7f9a02a00640 1 --2- 172.21.15.45:0/257120967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e34ff330 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f99f000e8f0 tx=0x7f99f0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac37a13e3fb3b8a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.155+0000 7f99ef400640 1 -- 172.21.15.45:0/257120967 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0025070 con 0x5639e3526a30 2024-04-16T16:20:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.155+0000 7f99ef400640 1 -- 172.21.15.45:0/257120967 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f0015700 con 0x5639e3526a30 2024-04-16T16:20:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.155+0000 7f99ef400640 1 -- 172.21.15.45:0/257120967 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f000f040 con 0x5639e3526a30 2024-04-16T16:20:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.156+0000 7f9a0cfba740 1 -- 172.21.15.45:0/257120967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 msgr2=0x5639e34ff330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.156+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/257120967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e34ff330 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f99f000e8f0 tx=0x7f99f0011000 comp rx=0 tx=0).stop 2024-04-16T16:20:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.156+0000 7f9a0cfba740 1 -- 172.21.15.45:0/257120967 shutdown_connections 2024-04-16T16:20:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.156+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/257120967 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e34ff870 0x5639e34fd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.156+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/257120967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e34ff330 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.156+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/257120967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3526650 0x5639e3545790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.156+0000 7f9a0cfba740 1 -- 172.21.15.45:0/257120967 >> 172.21.15.45:0/257120967 conn(0x5639e34fb910 msgr2=0x5639e3544490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.156+0000 7f9a0cfba740 1 -- 172.21.15.45:0/257120967 shutdown_connections 2024-04-16T16:20:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.156+0000 7f9a0cfba740 1 -- 172.21.15.45:0/257120967 wait complete. 2024-04-16T16:20:41.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.157+0000 7f9a0cfba740 1 -- start start 2024-04-16T16:20:41.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.157+0000 7f9a0cfba740 1 Processor -- start 2024-04-16T16:20:41.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.158+0000 7f9a0cfba740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526650 0x5639e3520ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.158+0000 7f9a0cfba740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3526a30 0x5639e353fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.158+0000 7f9a0cfba740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e3547260 0x5639e35476e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.158+0000 7f9a0cfba740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5639e34f9740 con 0x5639e3526650 2024-04-16T16:20:41.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.158+0000 7f9a0cfba740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5639e341e920 con 0x5639e3526a30 2024-04-16T16:20:41.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.158+0000 7f9a0cfba740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5639e34ffcf0 con 0x5639e3547260 2024-04-16T16:20:41.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.158+0000 7f9a0cfba740 10 client.0 _ll_register_callbacks cb 0x5639e341e310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:20:41.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.158+0000 7f9a03400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526650 0x5639e3520ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.158+0000 7f9a03400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526650 0x5639e3520ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44418/0 (socket says 172.21.15.45:44418) 2024-04-16T16:20:41.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.158+0000 7f9a03400640 1 -- 172.21.15.45:0/3440103157 learned_addr learned my addr 172.21.15.45:0/3440103157 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:41.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.158+0000 7f9a03400640 1 -- 172.21.15.45:0/3440103157 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e3547260 msgr2=0x5639e35476e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:41.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.158+0000 7f9a02a00640 1 --2- 172.21.15.45:0/3440103157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3526a30 0x5639e353fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.158+0000 7f9a03400640 1 --2- 172.21.15.45:0/3440103157 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e3547260 0x5639e35476e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.158+0000 7f9a03400640 1 -- 172.21.15.45:0/3440103157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3526a30 msgr2=0x5639e353fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:41.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.159+0000 7f9a03400640 1 --2- 172.21.15.45:0/3440103157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3526a30 0x5639e353fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.159+0000 7f9a03400640 1 -- 172.21.15.45:0/3440103157 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639e32416b0 con 0x5639e3526650 2024-04-16T16:20:41.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.159+0000 7f9a03400640 1 --2- 172.21.15.45:0/3440103157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526650 0x5639e3520ac0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f99f4034040 tx=0x7f99f407bb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb5c4b6ee364d24 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:41.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.159+0000 7f99e3e00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:20:41.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.159+0000 7f99e3e00640 1 -- 172.21.15.45:0/3440103157 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f407f810 con 0x5639e3526650 2024-04-16T16:20:41.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.159+0000 7f99e3e00640 1 -- 172.21.15.45:0/3440103157 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7f99f4077ad0 con 0x5639e3526650 2024-04-16T16:20:41.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.159+0000 7f9a0cfba740 1 -- 172.21.15.45:0/3440103157 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99f0015700 con 0x5639e3526650 2024-04-16T16:20:41.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.159+0000 7f99e3e00640 1 -- 172.21.15.45:0/3440103157 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f99f4084040 con 0x5639e3526650 2024-04-16T16:20:41.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.159+0000 7f9a0cfba740 10 client.8787 fetch_fsmap learned FSMap version 91 2024-04-16T16:20:41.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.159+0000 7f9a0cfba740 1 -- 172.21.15.45:0/3440103157 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=2+}) -- 0x5639e3543380 con 0x5639e3526650 2024-04-16T16:20:41.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.160+0000 7f99e3e00640 1 -- 172.21.15.45:0/3440103157 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f0015700 con 0x5639e3526650 2024-04-16T16:20:41.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.160+0000 7f99e3e00640 1 -- 172.21.15.45:0/3440103157 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 91) ==== 3494+0+0 (secure 0 0 0) 0x7f99f4083bb0 con 0x5639e3526650 2024-04-16T16:20:41.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.160+0000 7f9a0cfba740 10 client.8787 fetch_fsmap finished waiting for FSMap version 91 2024-04-16T16:20:41.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.160+0000 7f9a0cfba740 10 client.8787 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-16T16:20:41.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.160+0000 7f9a0cfba740 20 client.8787 populate_metadata read hostname 'smithi045' 2024-04-16T16:20:41.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.160+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/3440103157 >> [v2:172.21.15.45:6833/2060312705,v1:172.21.15.45:6835/2060312705] conn(0x5639e3554f30 0x5639e3562d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.160+0000 7f9a0cfba740 4 client.8787 mds_command: new command op to 8643 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-16T16:20:41.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.160+0000 7f9a0cfba740 1 -- 172.21.15.45:0/3440103157 --> [v2:172.21.15.45:6833/2060312705,v1:172.21.15.45:6835/2060312705] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639e340ab70 con 0x5639e3554f30 2024-04-16T16:20:41.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.160+0000 7f9a02a00640 1 --2- 172.21.15.45:0/3440103157 >> [v2:172.21.15.45:6833/2060312705,v1:172.21.15.45:6835/2060312705] conn(0x5639e3554f30 0x5639e3562d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.161+0000 7f9a02a00640 1 --2- 172.21.15.45:0/3440103157 >> [v2:172.21.15.45:6833/2060312705,v1:172.21.15.45:6835/2060312705] conn(0x5639e3554f30 0x5639e3562d70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c7aea45216aae0d server_cookie=1996f894a6d875f9 in_seq=0 out_seq=0 2024-04-16T16:20:41.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.161+0000 7f99e3e00640 10 client.8787 ms_handle_connect on v2:172.21.15.45:6833/2060312705 2024-04-16T16:20:41.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.162+0000 7f99e3e00640 1 -- 172.21.15.45:0/3440103157 <== mds.0 v2:172.21.15.45:6833/2060312705 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5639e340ab70 con 0x5639e3554f30 2024-04-16T16:20:41.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.162+0000 7f99e3e00640 10 client.8787 handle_command_reply: tid=0 2024-04-16T16:20:41.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.162+0000 7f9a0cfba740 1 client.8787 shutdown 2024-04-16T16:20:41.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.162+0000 7f9a0cfba740 1 -- 172.21.15.45:0/3440103157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526650 msgr2=0x5639e3520ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.162+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/3440103157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526650 0x5639e3520ac0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f99f4034040 tx=0x7f99f407bb60 comp rx=0 tx=0).stop 2024-04-16T16:20:41.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.162+0000 7f9a0cfba740 1 -- 172.21.15.45:0/3440103157 shutdown_connections 2024-04-16T16:20:41.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.162+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/3440103157 >> [v2:172.21.15.45:6833/2060312705,v1:172.21.15.45:6835/2060312705] conn(0x5639e3554f30 0x5639e3562d70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.162+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/3440103157 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e3547260 0x5639e35476e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.162+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/3440103157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3526a30 0x5639e353fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.162+0000 7f99e3e00640 0 client.8787 ms_handle_reset on v2:172.21.15.45:6833/2060312705 2024-04-16T16:20:41.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.162+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/3440103157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526650 0x5639e3520ac0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.162+0000 7f9a0cfba740 1 -- 172.21.15.45:0/3440103157 >> 172.21.15.45:0/3440103157 conn(0x5639e34ff870 msgr2=0x5639e34f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:41.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.162+0000 7f9a0cfba740 1 -- 172.21.15.45:0/3440103157 shutdown_connections 2024-04-16T16:20:41.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.162+0000 7f9a0cfba740 1 -- 172.21.15.45:0/3440103157 wait complete. 2024-04-16T16:20:41.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.163+0000 7f9a0cfba740 20 client.8787 trim_cache size 0 max 16384 2024-04-16T16:20:41.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.183+0000 7f9a0cfba740 1 Processor -- start 2024-04-16T16:20:41.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.183+0000 7f9a0cfba740 1 -- start start 2024-04-16T16:20:41.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.184+0000 7f9a0cfba740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3526650 0x5639e35004c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.184+0000 7f9a0cfba740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e34fd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.184+0000 7f9a0cfba740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e3547260 0x5639e34fd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.184+0000 7f9a0cfba740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9a04200fb0 con 0x5639e3526a30 2024-04-16T16:20:41.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.184+0000 7f9a0cfba740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9a042000a0 con 0x5639e3526650 2024-04-16T16:20:41.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.184+0000 7f9a0cfba740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9a040cdb00 con 0x5639e3547260 2024-04-16T16:20:41.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.184+0000 7f9a02a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e34fd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.184+0000 7f9a02a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e34fd200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44424/0 (socket says 172.21.15.45:44424) 2024-04-16T16:20:41.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.184+0000 7f9a02a00640 1 -- 172.21.15.45:0/226847174 learned_addr learned my addr 172.21.15.45:0/226847174 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:41.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.185+0000 7f9a02a00640 1 -- 172.21.15.45:0/226847174 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e3547260 msgr2=0x5639e34fd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:41.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.185+0000 7f9a03e00640 1 --2- 172.21.15.45:0/226847174 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e3547260 0x5639e34fd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.185+0000 7f9a03400640 1 --2- 172.21.15.45:0/226847174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3526650 0x5639e35004c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.185+0000 7f9a02a00640 1 --2- 172.21.15.45:0/226847174 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e3547260 0x5639e34fd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.185+0000 7f9a02a00640 1 -- 172.21.15.45:0/226847174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3526650 msgr2=0x5639e35004c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:41.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.185+0000 7f9a02a00640 1 --2- 172.21.15.45:0/226847174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3526650 0x5639e35004c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.185+0000 7f9a02a00640 1 -- 172.21.15.45:0/226847174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639e3527500 con 0x5639e3526a30 2024-04-16T16:20:41.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.185+0000 7f9a03400640 1 --2- 172.21.15.45:0/226847174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3526650 0x5639e35004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:41.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.185+0000 7f9a02a00640 1 --2- 172.21.15.45:0/226847174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e34fd200 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f99f000e8c0 tx=0x7f99f0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f74c1ec36fe30e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:41.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.185+0000 7f99eea00640 1 -- 172.21.15.45:0/226847174 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0035070 con 0x5639e3526a30 2024-04-16T16:20:41.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.185+0000 7f99eea00640 1 -- 172.21.15.45:0/226847174 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f0030e50 con 0x5639e3526a30 2024-04-16T16:20:41.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.185+0000 7f99eea00640 1 -- 172.21.15.45:0/226847174 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0030070 con 0x5639e3526a30 2024-04-16T16:20:41.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.186+0000 7f9a0cfba740 1 -- 172.21.15.45:0/226847174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 msgr2=0x5639e34fd200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.186+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/226847174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e34fd200 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f99f000e8c0 tx=0x7f99f0013000 comp rx=0 tx=0).stop 2024-04-16T16:20:41.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.186+0000 7f9a0cfba740 1 -- 172.21.15.45:0/226847174 shutdown_connections 2024-04-16T16:20:41.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.186+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/226847174 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e3547260 0x5639e34fd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.186+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/226847174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e34fd200 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.186+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/226847174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3526650 0x5639e35004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.186+0000 7f9a0cfba740 1 -- 172.21.15.45:0/226847174 >> 172.21.15.45:0/226847174 conn(0x5639e34ff870 msgr2=0x5639e34feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:41.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.186+0000 7f9a0cfba740 1 -- 172.21.15.45:0/226847174 shutdown_connections 2024-04-16T16:20:41.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.186+0000 7f9a0cfba740 1 -- 172.21.15.45:0/226847174 wait complete. 2024-04-16T16:20:41.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.187+0000 7f9a0cfba740 1 -- start start 2024-04-16T16:20:41.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.187+0000 7f9a0cfba740 1 Processor -- start 2024-04-16T16:20:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.188+0000 7f9a0cfba740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e3526650 0x5639e35bdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.188+0000 7f9a0cfba740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e3543780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.188+0000 7f9a0cfba740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3547260 0x5639e3543cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.188+0000 7f9a0cfba740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5639e3540df0 con 0x5639e3526a30 2024-04-16T16:20:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.188+0000 7f9a0cfba740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9a040ce150 con 0x5639e3547260 2024-04-16T16:20:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.188+0000 7f9a0cfba740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5639e3555830 con 0x5639e3526650 2024-04-16T16:20:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.188+0000 7f9a0cfba740 10 client.0 _ll_register_callbacks cb 0x7f99f4009150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:20:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.188+0000 7f9a02a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e3543780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.188+0000 7f9a02a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e3543780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44432/0 (socket says 172.21.15.45:44432) 2024-04-16T16:20:41.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.188+0000 7f9a02a00640 1 -- 172.21.15.45:0/2670819934 learned_addr learned my addr 172.21.15.45:0/2670819934 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:41.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.189+0000 7f9a02a00640 1 -- 172.21.15.45:0/2670819934 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e3526650 msgr2=0x5639e35bdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:41.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.189+0000 7f9a02a00640 1 --2- 172.21.15.45:0/2670819934 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e3526650 0x5639e35bdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.189+0000 7f9a02a00640 1 -- 172.21.15.45:0/2670819934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3547260 msgr2=0x5639e3543cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:41.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.189+0000 7f9a02a00640 1 --2- 172.21.15.45:0/2670819934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3547260 0x5639e3543cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.189+0000 7f9a02a00640 1 -- 172.21.15.45:0/2670819934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639e3553e20 con 0x5639e3526a30 2024-04-16T16:20:41.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.189+0000 7f9a02a00640 1 --2- 172.21.15.45:0/2670819934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e3543780 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f99f000e8f0 tx=0x7f99f0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d63af320178a0d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:41.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.189+0000 7f99e3e00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:20:41.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.189+0000 7f99e3e00640 1 -- 172.21.15.45:0/2670819934 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0035030 con 0x5639e3526a30 2024-04-16T16:20:41.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.189+0000 7f99e3e00640 1 -- 172.21.15.45:0/2670819934 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7f99f00301e0 con 0x5639e3526a30 2024-04-16T16:20:41.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.189+0000 7f9a0cfba740 1 -- 172.21.15.45:0/2670819934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99f0030e50 con 0x5639e3526a30 2024-04-16T16:20:41.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.190+0000 7f99e3e00640 1 -- 172.21.15.45:0/2670819934 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f99f0030040 con 0x5639e3526a30 2024-04-16T16:20:41.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.190+0000 7f9a0cfba740 10 client.8793 fetch_fsmap learned FSMap version 91 2024-04-16T16:20:41.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.190+0000 7f9a0cfba740 1 -- 172.21.15.45:0/2670819934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=2+}) -- 0x5639e352a6e0 con 0x5639e3526a30 2024-04-16T16:20:41.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.190+0000 7f99e3e00640 1 -- 172.21.15.45:0/2670819934 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f0030e50 con 0x5639e3526a30 2024-04-16T16:20:41.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.190+0000 7f99e3e00640 1 -- 172.21.15.45:0/2670819934 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 91) ==== 3494+0+0 (secure 0 0 0) 0x7f99f0012030 con 0x5639e3526a30 2024-04-16T16:20:41.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.190+0000 7f9a0cfba740 10 client.8793 fetch_fsmap finished waiting for FSMap version 91 2024-04-16T16:20:41.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.190+0000 7f9a0cfba740 10 client.8793 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-16T16:20:41.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.190+0000 7f9a0cfba740 20 client.8793 populate_metadata read hostname 'smithi045' 2024-04-16T16:20:41.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.190+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/2670819934 >> [v2:172.21.15.45:6833/2060312705,v1:172.21.15.45:6835/2060312705] conn(0x5639e35684b0 0x5639e35c1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.190+0000 7f9a0cfba740 4 client.8793 mds_command: new command op to 8643 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-16T16:20:41.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.190+0000 7f9a0cfba740 1 -- 172.21.15.45:0/2670819934 --> [v2:172.21.15.45:6833/2060312705,v1:172.21.15.45:6835/2060312705] -- command(tid 0: {"prefix": "status"}) -- 0x5639e34723e0 con 0x5639e35684b0 2024-04-16T16:20:41.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.190+0000 7f9a03400640 1 --2- 172.21.15.45:0/2670819934 >> [v2:172.21.15.45:6833/2060312705,v1:172.21.15.45:6835/2060312705] conn(0x5639e35684b0 0x5639e35c1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.191+0000 7f9a03400640 1 --2- 172.21.15.45:0/2670819934 >> [v2:172.21.15.45:6833/2060312705,v1:172.21.15.45:6835/2060312705] conn(0x5639e35684b0 0x5639e35c1ae0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=325a791bda210d0b server_cookie=c29cbeeef706d633 in_seq=0 out_seq=0 2024-04-16T16:20:41.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.191+0000 7f99e3e00640 10 client.8793 ms_handle_connect on v2:172.21.15.45:6833/2060312705 2024-04-16T16:20:41.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.191+0000 7f99e3e00640 1 -- 172.21.15.45:0/2670819934 <== mds.0 v2:172.21.15.45:6833/2060312705 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x5639e34723e0 con 0x5639e35684b0 2024-04-16T16:20:41.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.191+0000 7f99e3e00640 10 client.8793 handle_command_reply: tid=0 2024-04-16T16:20:41.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.191+0000 7f9a0cfba740 1 client.8793 shutdown 2024-04-16T16:20:41.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.191+0000 7f9a0cfba740 1 -- 172.21.15.45:0/2670819934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 msgr2=0x5639e3543780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.191+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/2670819934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e3543780 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f99f000e8f0 tx=0x7f99f0015000 comp rx=0 tx=0).stop 2024-04-16T16:20:41.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.191+0000 7f9a0cfba740 1 -- 172.21.15.45:0/2670819934 shutdown_connections 2024-04-16T16:20:41.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.191+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/2670819934 >> [v2:172.21.15.45:6833/2060312705,v1:172.21.15.45:6835/2060312705] conn(0x5639e35684b0 0x5639e35c1ae0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.191+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/2670819934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5639e3547260 0x5639e3543cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.191+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/2670819934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639e3526a30 0x5639e3543780 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.191+0000 7f9a0cfba740 1 --2- 172.21.15.45:0/2670819934 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5639e3526650 0x5639e35bdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.191+0000 7f99e3e00640 0 client.8793 ms_handle_reset on v2:172.21.15.45:6833/2060312705 2024-04-16T16:20:41.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.191+0000 7f9a0cfba740 1 -- 172.21.15.45:0/2670819934 >> 172.21.15.45:0/2670819934 conn(0x5639e34ff870 msgr2=0x5639e34fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:41.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.192+0000 7f9a0cfba740 1 -- 172.21.15.45:0/2670819934 shutdown_connections 2024-04-16T16:20:41.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.192+0000 7f9a0cfba740 1 -- 172.21.15.45:0/2670819934 wait complete. 2024-04-16T16:20:41.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.192+0000 7f9a0cfba740 20 client.8793 trim_cache size 0 max 16384 2024-04-16T16:20:41.194 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:20:41.194 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "748c4b85-0190-4c59-80ad-098a6a5007c0", 2024-04-16T16:20:41.194 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-04-16T16:20:41.195 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 8643, 2024-04-16T16:20:41.195 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-04-16T16:20:41.195 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-04-16T16:20:41.195 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-04-16T16:20:41.195 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 6.8446801559999999, 2024-04-16T16:20:41.195 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 91, 2024-04-16T16:20:41.195 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 128, 2024-04-16T16:20:41.195 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 128, 2024-04-16T16:20:41.195 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 12.217293142000001 2024-04-16T16:20:41.195 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:20:41.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.192+0000 7f9a0a200640 1 -- 172.21.15.45:0/1137867605 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f99e406ca10 msgr2=0x7f99e406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.192+0000 7f9a0a200640 1 --2- 172.21.15.45:0/1137867605 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f99e406ca10 0x7f99e406eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9a04163320 tx=0x7f99f0008040 comp rx=0 tx=0).stop 2024-04-16T16:20:41.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.192+0000 7f9a0a200640 1 -- 172.21.15.45:0/1137867605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a040726c0 msgr2=0x7f9a041f51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.192+0000 7f9a0a200640 1 --2- 172.21.15.45:0/1137867605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a040726c0 0x7f9a041f51c0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f99f40098c0 tx=0x7f99f40046c0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.192+0000 7f9a0a200640 1 -- 172.21.15.45:0/1137867605 shutdown_connections 2024-04-16T16:20:41.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.192+0000 7f9a0a200640 1 --2- 172.21.15.45:0/1137867605 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f99e406ca10 0x7f99e406eed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.192+0000 7f9a0a200640 1 --2- 172.21.15.45:0/1137867605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a04147550 0x7f9a0413f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.192+0000 7f9a0a200640 1 --2- 172.21.15.45:0/1137867605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a040b58e0 0x7f9a041f1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.192+0000 7f9a0a200640 1 --2- 172.21.15.45:0/1137867605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a040726c0 0x7f9a041f51c0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.192+0000 7f9a0a200640 1 -- 172.21.15.45:0/1137867605 >> 172.21.15.45:0/1137867605 conn(0x7f9a0406b930 msgr2=0x7f9a0406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:41.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.193+0000 7f9a0a200640 1 -- 172.21.15.45:0/1137867605 shutdown_connections 2024-04-16T16:20:41.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.193+0000 7f9a0a200640 1 -- 172.21.15.45:0/1137867605 wait complete. 2024-04-16T16:20:41.208 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:20:41.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.423+0000 7f14cca00640 1 Processor -- start 2024-04-16T16:20:41.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.423+0000 7f14cca00640 1 -- start start 2024-04-16T16:20:41.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.425+0000 7f14cca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f14c815ccb0 0x7f14c806df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.425+0000 7f14cca00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14c815d360 con 0x7f14c8146220 2024-04-16T16:20:41.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.425+0000 7f14cca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14c815d640 con 0x7f14c806e480 2024-04-16T16:20:41.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.425+0000 7f14cca00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f14c815d130 con 0x7f14c815ccb0 2024-04-16T16:20:41.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.425+0000 7f14c5e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f14c815ccb0 0x7f14c806df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.425+0000 7f14c6800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f14c8146220 0x7f14c806f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43116/0 (socket says 172.21.15.45:43116) 2024-04-16T16:20:41.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.425+0000 7f14c6800640 1 -- 172.21.15.45:0/3255654316 learned_addr learned my addr 172.21.15.45:0/3255654316 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:41.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.426+0000 7f14c5e00640 1 -- 172.21.15.45:0/3255654316 >> v1:172.21.15.103:6789/0 conn(0x7f14c806e480 legacy=0x7f14c81608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.426+0000 7f14c5e00640 1 -- 172.21.15.45:0/3255654316 >> v1:172.21.15.45:6789/0 conn(0x7f14c8146220 legacy=0x7f14c806f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.426+0000 7f14c5e00640 1 -- 172.21.15.45:0/3255654316 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c815d920 con 0x7f14c815ccb0 2024-04-16T16:20:41.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.426+0000 7f14c5400640 1 -- 172.21.15.45:0/3255654316 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2561450616 0 0) 0x7f14c815d360 con 0x7f14c8146220 2024-04-16T16:20:41.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.427+0000 7f14c5e00640 1 --2- 172.21.15.45:0/3255654316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c815ccb0 0x7f14c806df40 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f14b40097a0 tx=0x7f14b402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f5334c60214ca9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:41.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.428+0000 7f14c5400640 1 -- 172.21.15.45:0/3255654316 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4034070 con 0x7f14c815ccb0 2024-04-16T16:20:41.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.428+0000 7f14c5400640 1 -- 172.21.15.45:0/3255654316 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b4030610 con 0x7f14c815ccb0 2024-04-16T16:20:41.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.428+0000 7f14c5400640 1 -- 172.21.15.45:0/3255654316 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b40309e0 con 0x7f14c815ccb0 2024-04-16T16:20:41.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.428+0000 7f14cca00640 1 -- 172.21.15.45:0/3255654316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c815ccb0 msgr2=0x7f14c806df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.428+0000 7f14cca00640 1 --2- 172.21.15.45:0/3255654316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c815ccb0 0x7f14c806df40 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f14b40097a0 tx=0x7f14b402b960 comp rx=0 tx=0).stop 2024-04-16T16:20:41.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.429+0000 7f14cca00640 1 -- 172.21.15.45:0/3255654316 shutdown_connections 2024-04-16T16:20:41.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.429+0000 7f14cca00640 1 --2- 172.21.15.45:0/3255654316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c815ccb0 0x7f14c806df40 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.429+0000 7f14cca00640 1 -- 172.21.15.45:0/3255654316 >> 172.21.15.45:0/3255654316 conn(0x7f14c806b930 msgr2=0x7f14c806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:41.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.429+0000 7f14cca00640 1 -- 172.21.15.45:0/3255654316 shutdown_connections 2024-04-16T16:20:41.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.429+0000 7f14cca00640 1 -- 172.21.15.45:0/3255654316 wait complete. 2024-04-16T16:20:41.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.431+0000 7f14cca00640 1 Processor -- start 2024-04-16T16:20:41.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.431+0000 7f14cca00640 1 -- start start 2024-04-16T16:20:41.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.432+0000 7f14cca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c806e480 0x7f14c8142da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.433+0000 7f14cca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c8146220 0x7f14c813ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.433+0000 7f14cca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c815ccb0 0x7f14c80b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.433+0000 7f14cca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f14c8160fb0 con 0x7f14c8146220 2024-04-16T16:20:41.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.433+0000 7f14cca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f14c8160410 con 0x7f14c806e480 2024-04-16T16:20:41.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.433+0000 7f14cca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f14c8161130 con 0x7f14c815ccb0 2024-04-16T16:20:41.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.433+0000 7f14c5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c8146220 0x7f14c813ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.433+0000 7f14c5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c8146220 0x7f14c813ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44452/0 (socket says 172.21.15.45:44452) 2024-04-16T16:20:41.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.433+0000 7f14c5e00640 1 -- 172.21.15.45:0/2282239735 learned_addr learned my addr 172.21.15.45:0/2282239735 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:41.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.433+0000 7f14c6800640 1 --2- 172.21.15.45:0/2282239735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c806e480 0x7f14c8142da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.433+0000 7f14c7200640 1 --2- 172.21.15.45:0/2282239735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c815ccb0 0x7f14c80b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.434+0000 7f14c5e00640 1 -- 172.21.15.45:0/2282239735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c815ccb0 msgr2=0x7f14c80b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.434+0000 7f14c5e00640 1 --2- 172.21.15.45:0/2282239735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c815ccb0 0x7f14c80b1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.434+0000 7f14c5e00640 1 -- 172.21.15.45:0/2282239735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c806e480 msgr2=0x7f14c8142da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.434+0000 7f14c5e00640 1 --2- 172.21.15.45:0/2282239735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c806e480 0x7f14c8142da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.434+0000 7f14c5e00640 1 -- 172.21.15.45:0/2282239735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c8141fe0 con 0x7f14c8146220 2024-04-16T16:20:41.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.434+0000 7f14c7200640 1 --2- 172.21.15.45:0/2282239735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c815ccb0 0x7f14c80b1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:41.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.434+0000 7f14c5e00640 1 --2- 172.21.15.45:0/2282239735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c8146220 0x7f14c813ee30 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f14b40098c0 tx=0x7f14b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:41.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.434+0000 7f14bea00640 1 -- 172.21.15.45:0/2282239735 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4047070 con 0x7f14c8146220 2024-04-16T16:20:41.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.434+0000 7f14bea00640 1 -- 172.21.15.45:0/2282239735 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b4039a10 con 0x7f14c8146220 2024-04-16T16:20:41.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.434+0000 7f14bea00640 1 -- 172.21.15.45:0/2282239735 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4034040 con 0x7f14c8146220 2024-04-16T16:20:41.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.434+0000 7f14cca00640 1 -- 172.21.15.45:0/2282239735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14c81422c0 con 0x7f14c8146220 2024-04-16T16:20:41.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.435+0000 7f14cca00640 1 -- 172.21.15.45:0/2282239735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14c8142880 con 0x7f14c8146220 2024-04-16T16:20:41.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.436+0000 7f14cca00640 1 -- 172.21.15.45:0/2282239735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14c8075780 con 0x7f14c8146220 2024-04-16T16:20:41.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.437+0000 7f14bea00640 1 -- 172.21.15.45:0/2282239735 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f14b404a070 con 0x7f14c8146220 2024-04-16T16:20:41.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.438+0000 7f14bea00640 1 --2- 172.21.15.45:0/2282239735 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f14a806ca10 0x7f14a806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:41.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.438+0000 7f14bea00640 1 -- 172.21.15.45:0/2282239735 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7f14b40adaa0 con 0x7f14c8146220 2024-04-16T16:20:41.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.439+0000 7f14c6800640 1 --2- 172.21.15.45:0/2282239735 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f14a806ca10 0x7f14a806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:41.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.440+0000 7f14c6800640 1 --2- 172.21.15.45:0/2282239735 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f14a806ca10 0x7f14a806eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f14b8000f60 tx=0x7f14b8007b90 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:41.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.443+0000 7f14bea00640 1 -- 172.21.15.45:0/2282239735 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f14b407bc10 con 0x7f14c8146220 2024-04-16T16:20:41.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.692+0000 7f14cca00640 1 -- 172.21.15.45:0/2282239735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f14c80cff40 con 0x7f14c8146220 2024-04-16T16:20:41.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.693+0000 7f14bea00640 1 -- 172.21.15.45:0/2282239735 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v91) ==== 53+0+73 (secure 0 0 0) 0x7f14b407b510 con 0x7f14c8146220 2024-04-16T16:20:41.692 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:20:41.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.696+0000 7f14cca00640 1 -- 172.21.15.45:0/2282239735 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f14a806ca10 msgr2=0x7f14a806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.696+0000 7f14cca00640 1 --2- 172.21.15.45:0/2282239735 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f14a806ca10 0x7f14a806eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f14b8000f60 tx=0x7f14b8007b90 comp rx=0 tx=0).stop 2024-04-16T16:20:41.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.696+0000 7f14cca00640 1 -- 172.21.15.45:0/2282239735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c8146220 msgr2=0x7f14c813ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:41.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.696+0000 7f14cca00640 1 --2- 172.21.15.45:0/2282239735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c8146220 0x7f14c813ee30 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f14b40098c0 tx=0x7f14b40046c0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.696+0000 7f14cca00640 1 -- 172.21.15.45:0/2282239735 shutdown_connections 2024-04-16T16:20:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.696+0000 7f14cca00640 1 --2- 172.21.15.45:0/2282239735 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f14a806ca10 0x7f14a806eed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.696+0000 7f14cca00640 1 --2- 172.21.15.45:0/2282239735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14c815ccb0 0x7f14c80b1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.696+0000 7f14cca00640 1 --2- 172.21.15.45:0/2282239735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14c8146220 0x7f14c813ee30 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.696+0000 7f14cca00640 1 --2- 172.21.15.45:0/2282239735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c806e480 0x7f14c8142da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.696+0000 7f14cca00640 1 -- 172.21.15.45:0/2282239735 >> 172.21.15.45:0/2282239735 conn(0x7f14c806b930 msgr2=0x7f14c80c5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.696+0000 7f14cca00640 1 -- 172.21.15.45:0/2282239735 shutdown_connections 2024-04-16T16:20:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:41.696+0000 7f14cca00640 1 -- 172.21.15.45:0/2282239735 wait complete. 2024-04-16T16:20:41.713 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-16T16:20:41.714 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:20:41.714 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-04-16T16:20:41.714 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-16T16:20:41.714 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T16:20:41.714 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-16T16:20:41.714 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:20:41.714 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:20:41.714 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:20:41.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:41.714 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-04-16T16:20:41.768 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:20:41.768 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:20:41.768 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:20:41.768 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:20:41.768 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-04-16T16:20:41.768 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:20:41.768 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:20:41.768 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:20:41.768 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-04-16T16:20:41.768 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:20:41.768 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:20:41.769 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-04-16T16:20:41.769 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:20:41.769 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-04-16T16:20:41.769 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-04-16T16:20:41.769 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-16T16:20:41.769 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42159sec preferred_lft 42159sec 2024-04-16T16:20:41.769 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-04-16T16:20:41.769 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:20:41.769 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:20:41.769 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-04-16T16:20:41.769 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-04-16T16:20:41.769 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:20:41.769 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff 2024-04-16T16:20:41.769 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:20:41.770 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:20:41.770 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::50f1:99ff:fe2b:225b/64 scope link 2024-04-16T16:20:41.770 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:20:41.770 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:20:41.770 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T16:20:41.770 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::7007:c3ff:fe13:7bcd/64 scope link 2024-04-16T16:20:41.770 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:20:41.770 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:20:41.770 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether ae:10:ea:fc:0a:13 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-16T16:20:41.770 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ac10:eaff:fefc:a13/64 scope link 2024-04-16T16:20:41.770 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:20:41.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:41.771 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:20:41.866 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-04-16T16:20:41.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:41.870 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-04-16T16:20:41.929 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-04-16T16:20:41.929 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:20:41.929 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-16T16:20:41.930 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-16T16:20:41.930 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:20:41.931 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:20:41.931 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:20:41.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-16T16:20:41.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-16T16:20:41.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-16T16:20:41.931 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:20:42.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:20:42.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:20:42.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:42.196 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T16:20:42.247 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-16T16:20:42.247 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-16T16:20:42.248 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:20:42.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:42.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-04-16T16:20:42.349 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:20:42.349 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-16T16:20:42.404 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:20:42.404 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:20:42.404 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:20:42.451 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-16T16:20:42.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:42.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:20:42.642 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:20:42.643 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:20:42.678 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:20:42.678 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:20:42.678 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:20:42.678 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 0@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:20:42.678 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:20:42.680 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-16T16:20:42.680 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:20:42.680 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:20:42.680 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:20:42.680 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:20:42.684 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:20:42.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:42.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:20:42.775 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:20:42.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:20:42.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:20:42.921 INFO:teuthology.orchestra.run.smithi045.stdout:client8805 2024-04-16T16:20:42.921 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:20:42.921 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:20:42.974 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:20:43.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.198+0000 7ff2e1200640 1 Processor -- start 2024-04-16T16:20:43.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.199+0000 7ff2e1200640 1 -- start start 2024-04-16T16:20:43.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.200+0000 7ff2e1200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff2dc0b6730 0x7ff2dc0b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:43.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.200+0000 7ff2e1200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff2dc0b7ab0 0x7ff2dc0b7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:43.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.200+0000 7ff2e1200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2dc0d46e0 con 0x7ff2dc161ff0 2024-04-16T16:20:43.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.200+0000 7ff2e1200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff2dc0ba560 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.200+0000 7ff2e1200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff2dc0d3640 con 0x7ff2dc0b6730 2024-04-16T16:20:43.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.201+0000 7ff2dbe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff2dc0b7ab0 0x7ff2dc0b7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:43.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.201+0000 7ff2dbe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff2dc0b7ab0 0x7ff2dc0b7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44462/0 (socket says 172.21.15.45:44462) 2024-04-16T16:20:43.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.201+0000 7ff2dbe00640 1 -- 172.21.15.45:0/2718739301 learned_addr learned my addr 172.21.15.45:0/2718739301 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:43.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.201+0000 7ff2daa00640 1 --2- 172.21.15.45:0/2718739301 >> v2:172.21.15.103:3300/0 conn(0x7ff2dc0b6730 0x7ff2dc0b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:43.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.201+0000 7ff2dbe00640 1 -- 172.21.15.45:0/2718739301 >> v2:172.21.15.103:3300/0 conn(0x7ff2dc0b6730 msgr2=0x7ff2dc0b6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:43.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.201+0000 7ff2dbe00640 1 --2- 172.21.15.45:0/2718739301 >> v2:172.21.15.103:3300/0 conn(0x7ff2dc0b6730 0x7ff2dc0b6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:43.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.201+0000 7ff2dbe00640 1 -- 172.21.15.45:0/2718739301 >> v1:172.21.15.103:6789/0 conn(0x7ff2dc161ff0 legacy=0x7ff2dc0cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:43.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.201+0000 7ff2dbe00640 1 -- 172.21.15.45:0/2718739301 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2dc0d3920 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.202+0000 7ff2dbe00640 1 --2- 172.21.15.45:0/2718739301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2dc0b7ab0 0x7ff2dc0b7ee0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc0b4320 tx=0x7ff2d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f42c69f54f2c2fa server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:43.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.203+0000 7ff2da000640 1 -- 172.21.15.45:0/2718739301 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4035020 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.203+0000 7ff2da000640 1 -- 172.21.15.45:0/2718739301 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2d4031870 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.203+0000 7ff2da000640 1 -- 172.21.15.45:0/2718739301 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4031c40 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.203+0000 7ff2e1200640 1 -- 172.21.15.45:0/2718739301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2dc0b7ab0 msgr2=0x7ff2dc0b7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:43.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.203+0000 7ff2e1200640 1 --2- 172.21.15.45:0/2718739301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2dc0b7ab0 0x7ff2dc0b7ee0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc0b4320 tx=0x7ff2d402b910 comp rx=0 tx=0).stop 2024-04-16T16:20:43.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.204+0000 7ff2e1200640 1 -- 172.21.15.45:0/2718739301 shutdown_connections 2024-04-16T16:20:43.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.204+0000 7ff2e1200640 1 --2- 172.21.15.45:0/2718739301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2dc0b7ab0 0x7ff2dc0b7ee0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:43.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.204+0000 7ff2e1200640 1 --2- 172.21.15.45:0/2718739301 >> v2:172.21.15.103:3300/0 conn(0x7ff2dc0b6730 0x7ff2dc0b6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:43.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.204+0000 7ff2e1200640 1 -- 172.21.15.45:0/2718739301 >> 172.21.15.45:0/2718739301 conn(0x7ff2dc0d17b0 msgr2=0x7ff2dc0d1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:43.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.204+0000 7ff2e1200640 1 -- 172.21.15.45:0/2718739301 shutdown_connections 2024-04-16T16:20:43.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.204+0000 7ff2e1200640 1 -- 172.21.15.45:0/2718739301 wait complete. 2024-04-16T16:20:43.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.206+0000 7ff2e1200640 1 Processor -- start 2024-04-16T16:20:43.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.206+0000 7ff2e1200640 1 -- start start 2024-04-16T16:20:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2e1200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2dc0b6730 0x7ff2dc0ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2e1200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2dc0b7ab0 0x7ff2dc1f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2e1200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2dc161ff0 0x7ff2dc0abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2e1200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff2dc0d41c0 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2e1200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff2dc0d3ee0 con 0x7ff2dc161ff0 2024-04-16T16:20:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2e1200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff2dc1623d0 con 0x7ff2dc0b6730 2024-04-16T16:20:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2daa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2dc0b7ab0 0x7ff2dc1f6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2daa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2dc0b7ab0 0x7ff2dc1f6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44472/0 (socket says 172.21.15.45:44472) 2024-04-16T16:20:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2daa00640 1 -- 172.21.15.45:0/2544222412 learned_addr learned my addr 172.21.15.45:0/2544222412 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2db400640 1 --2- 172.21.15.45:0/2544222412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2dc0b6730 0x7ff2dc0ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2dbe00640 1 --2- 172.21.15.45:0/2544222412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2dc161ff0 0x7ff2dc0abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2daa00640 1 -- 172.21.15.45:0/2544222412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2dc0b6730 msgr2=0x7ff2dc0ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2daa00640 1 --2- 172.21.15.45:0/2544222412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2dc0b6730 0x7ff2dc0ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2daa00640 1 -- 172.21.15.45:0/2544222412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2dc161ff0 msgr2=0x7ff2dc0abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2daa00640 1 --2- 172.21.15.45:0/2544222412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2dc161ff0 0x7ff2dc0abe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.207+0000 7ff2daa00640 1 -- 172.21.15.45:0/2544222412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2dc0ced80 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.208+0000 7ff2db400640 1 --2- 172.21.15.45:0/2544222412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2dc0b6730 0x7ff2dc0ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.208+0000 7ff2daa00640 1 --2- 172.21.15.45:0/2544222412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2dc0b7ab0 0x7ff2dc1f6a50 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4007e40 tx=0x7ff2c400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.208+0000 7ff2d3400640 1 -- 172.21.15.45:0/2544222412 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4014620 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.208+0000 7ff2d3400640 1 -- 172.21.15.45:0/2544222412 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2c4019070 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.208+0000 7ff2e1200640 1 -- 172.21.15.45:0/2544222412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2dc1f73c0 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.208+0000 7ff2d3400640 1 -- 172.21.15.45:0/2544222412 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4014a00 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.208+0000 7ff2e1200640 1 -- 172.21.15.45:0/2544222412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2dc1f7980 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.209+0000 7ff2e1200640 1 -- 172.21.15.45:0/2544222412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2dc075780 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.210+0000 7ff2d3400640 1 -- 172.21.15.45:0/2544222412 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7ff2c4014ba0 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.211+0000 7ff2d3400640 1 --2- 172.21.15.45:0/2544222412 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff2b806ca10 0x7ff2b806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:43.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.211+0000 7ff2d3400640 1 -- 172.21.15.45:0/2544222412 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7ff2c408c6c0 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.212+0000 7ff2db400640 1 --2- 172.21.15.45:0/2544222412 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff2b806ca10 0x7ff2b806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:43.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.214+0000 7ff2db400640 1 --2- 172.21.15.45:0/2544222412 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff2b806ca10 0x7ff2b806eed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc003eb0 tx=0x7ff2cc009040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:43.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.216+0000 7ff2d3400640 1 -- 172.21.15.45:0/2544222412 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff2dc0ced80 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.466+0000 7ff2e1200640 1 -- 172.21.15.45:0/2544222412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff2dc0d0270 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.467+0000 7ff2d3400640 1 -- 172.21.15.45:0/2544222412 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v91) ==== 53+0+73 (secure 0 0 0) 0x7ff2c405a8b0 con 0x7ff2dc0b7ab0 2024-04-16T16:20:43.466 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:20:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.469+0000 7ff2e1200640 1 -- 172.21.15.45:0/2544222412 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff2b806ca10 msgr2=0x7ff2b806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.469+0000 7ff2e1200640 1 --2- 172.21.15.45:0/2544222412 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff2b806ca10 0x7ff2b806eed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc003eb0 tx=0x7ff2cc009040 comp rx=0 tx=0).stop 2024-04-16T16:20:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.469+0000 7ff2e1200640 1 -- 172.21.15.45:0/2544222412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2dc0b7ab0 msgr2=0x7ff2dc1f6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.469+0000 7ff2e1200640 1 --2- 172.21.15.45:0/2544222412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2dc0b7ab0 0x7ff2dc1f6a50 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4007e40 tx=0x7ff2c400e4b0 comp rx=0 tx=0).stop 2024-04-16T16:20:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.470+0000 7ff2e1200640 1 -- 172.21.15.45:0/2544222412 shutdown_connections 2024-04-16T16:20:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.470+0000 7ff2e1200640 1 --2- 172.21.15.45:0/2544222412 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7ff2b806ca10 0x7ff2b806eed0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.470+0000 7ff2e1200640 1 --2- 172.21.15.45:0/2544222412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2dc161ff0 0x7ff2dc0abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.470+0000 7ff2e1200640 1 --2- 172.21.15.45:0/2544222412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2dc0b7ab0 0x7ff2dc1f6a50 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.470+0000 7ff2e1200640 1 --2- 172.21.15.45:0/2544222412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2dc0b6730 0x7ff2dc0ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.470+0000 7ff2e1200640 1 -- 172.21.15.45:0/2544222412 >> 172.21.15.45:0/2544222412 conn(0x7ff2dc0d17b0 msgr2=0x7ff2dc0b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.470+0000 7ff2e1200640 1 -- 172.21.15.45:0/2544222412 shutdown_connections 2024-04-16T16:20:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:43.470+0000 7ff2e1200640 1 -- 172.21.15.45:0/2544222412 wait complete. 2024-04-16T16:20:43.490 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-16T16:20:43.490 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:20:43.490 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-04-16T16:20:43.490 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-16T16:20:43.490 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T16:20:43.491 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-16T16:20:43.491 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:20:43.491 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:20:43.491 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:20:43.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:43.491 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-04-16T16:20:43.549 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:20:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:20:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:20:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:20:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-16T16:20:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:20:43.550 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:20:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:20:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-04-16T16:20:43.550 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:20:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:20:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-04-16T16:20:43.550 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:20:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-04-16T16:20:43.550 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:20:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-04-16T16:20:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-16T16:20:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42181sec preferred_lft 42181sec 2024-04-16T16:20:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-04-16T16:20:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:20:43.551 INFO:teuthology.orchestra.run.smithi103.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:20:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:20:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:20:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:20:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::74ce:ceff:fec6:2a00/64 scope link 2024-04-16T16:20:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:20:43.551 INFO:teuthology.orchestra.run.smithi103.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:20:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T16:20:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::3cf7:ffff:feca:f879/64 scope link 2024-04-16T16:20:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:20:43.552 INFO:teuthology.orchestra.run.smithi103.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:20:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 86:4c:6e:ce:c3:07 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-16T16:20:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::844c:6eff:fece:c307/64 scope link 2024-04-16T16:20:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:20:43.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:43.553 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:20:43.640 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-16T16:20:43.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:43.644 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-04-16T16:20:43.697 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-04-16T16:20:43.698 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:20:43.698 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-16T16:20:43.698 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-16T16:20:43.698 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:20:43.699 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:20:43.699 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:20:43.699 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-16T16:20:43.699 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-16T16:20:43.699 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-16T16:20:43.699 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:20:43.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:20:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:20:43.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:20:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:20:43.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:43.937 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-16T16:20:43.982 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-16T16:20:43.982 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-16T16:20:43.983 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:20:43.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:43.983 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.0 up 2024-04-16T16:20:44.123 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:20:44.123 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-16T16:20:44.178 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:20:44.178 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:20:44.179 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:20:44.230 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-16T16:20:44.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:44.230 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:20:44.370 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:20:44.370 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:20:44.403 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:20:44.403 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:20:44.403 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:20:44.403 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with new device syntax: 1@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:20:44.403 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:20:44.405 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-16T16:20:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:20:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:20:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:20:44.406 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:20:44.409 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:20:44.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:20:44.409 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:20:44.490 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:20:44.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:20:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:20:44.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:20:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:20:44.608 INFO:teuthology.orchestra.run.smithi103.stdout:client6380 2024-04-16T16:20:44.608 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:20:44.608 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:20:44.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:20:44.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.892+0000 7fa006c00640 1 Processor -- start 2024-04-16T16:20:44.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.893+0000 7fa006c00640 1 -- start start 2024-04-16T16:20:44.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.894+0000 7fa006c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa00015e8f0 0x7fa0000cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:44.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.894+0000 7fa006c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa0000ce9a0 0x7fa0000cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:44.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.894+0000 7fa006c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa00015ecd0 con 0x7fa0000cf080 2024-04-16T16:20:44.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.894+0000 7fa006c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa0000ce2e0 con 0x7fa0000ce9a0 2024-04-16T16:20:44.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.894+0000 7fa006c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa0000cd6d0 con 0x7fa00015e8f0 2024-04-16T16:20:44.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.895+0000 7fa006200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa0000cf080 0x7fa000162480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41926/0 (socket says 172.21.15.45:41926) 2024-04-16T16:20:44.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.895+0000 7fa006200640 1 -- 172.21.15.45:0/919962631 learned_addr learned my addr 172.21.15.45:0/919962631 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:44.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.895+0000 7fa004e00640 1 --2- 172.21.15.45:0/919962631 >> v2:172.21.15.45:3300/0 conn(0x7fa0000ce9a0 0x7fa0000cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:44.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.895+0000 7fa005800640 1 --2- 172.21.15.45:0/919962631 >> v2:172.21.15.103:3300/0 conn(0x7fa00015e8f0 0x7fa0000cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:44.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.895+0000 7fa004e00640 1 -- 172.21.15.45:0/919962631 >> v2:172.21.15.103:3300/0 conn(0x7fa00015e8f0 msgr2=0x7fa0000cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:44.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.895+0000 7fa004e00640 1 --2- 172.21.15.45:0/919962631 >> v2:172.21.15.103:3300/0 conn(0x7fa00015e8f0 0x7fa0000cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:44.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.895+0000 7fa004e00640 1 -- 172.21.15.45:0/919962631 >> v1:172.21.15.45:6789/0 conn(0x7fa0000cf080 legacy=0x7fa000162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:44.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.895+0000 7fa004e00640 1 -- 172.21.15.45:0/919962631 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0000cd9b0 con 0x7fa0000ce9a0 2024-04-16T16:20:44.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.897+0000 7fa004e00640 1 --2- 172.21.15.45:0/919962631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0000ce9a0 0x7fa0000cc950 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fa0000b5dd0 tx=0x7f9fec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf88965e42799e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:44.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.897+0000 7f9fffe00640 1 -- 172.21.15.45:0/919962631 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fec035020 con 0x7fa0000ce9a0 2024-04-16T16:20:44.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.897+0000 7f9fffe00640 1 -- 172.21.15.45:0/919962631 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fec031830 con 0x7fa0000ce9a0 2024-04-16T16:20:44.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.897+0000 7f9fffe00640 1 -- 172.21.15.45:0/919962631 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fec031c00 con 0x7fa0000ce9a0 2024-04-16T16:20:44.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.898+0000 7fa006c00640 1 -- 172.21.15.45:0/919962631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0000ce9a0 msgr2=0x7fa0000cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:44.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.898+0000 7fa006c00640 1 --2- 172.21.15.45:0/919962631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0000ce9a0 0x7fa0000cc950 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fa0000b5dd0 tx=0x7f9fec02b8d0 comp rx=0 tx=0).stop 2024-04-16T16:20:44.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.898+0000 7fa006c00640 1 -- 172.21.15.45:0/919962631 shutdown_connections 2024-04-16T16:20:44.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.898+0000 7fa006c00640 1 --2- 172.21.15.45:0/919962631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0000ce9a0 0x7fa0000cc950 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:44.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.898+0000 7fa006c00640 1 --2- 172.21.15.45:0/919962631 >> v2:172.21.15.103:3300/0 conn(0x7fa00015e8f0 0x7fa0000cc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:44.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.898+0000 7fa006c00640 1 -- 172.21.15.45:0/919962631 >> 172.21.15.45:0/919962631 conn(0x7fa00006b930 msgr2=0x7fa00006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:44.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.898+0000 7fa006c00640 1 -- 172.21.15.45:0/919962631 shutdown_connections 2024-04-16T16:20:44.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.898+0000 7fa006c00640 1 -- 172.21.15.45:0/919962631 wait complete. 2024-04-16T16:20:44.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.900+0000 7fa006c00640 1 Processor -- start 2024-04-16T16:20:44.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.900+0000 7fa006c00640 1 -- start start 2024-04-16T16:20:44.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.901+0000 7fa006c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0000ce9a0 0x7fa00015ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:44.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.901+0000 7fa006c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0000cf080 0x7fa000140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:44.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.901+0000 7fa006c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa00015e8f0 0x7fa00015da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:44.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.901+0000 7fa006c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa0000cd360 con 0x7fa0000cf080 2024-04-16T16:20:44.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.901+0000 7fa006c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa0000cdf70 con 0x7fa00015e8f0 2024-04-16T16:20:44.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.901+0000 7fa006c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa0000cbc40 con 0x7fa0000ce9a0 2024-04-16T16:20:44.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.901+0000 7fa004e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0000cf080 0x7fa000140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:44.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.901+0000 7fa004e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0000cf080 0x7fa000140ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54586/0 (socket says 172.21.15.45:54586) 2024-04-16T16:20:44.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.901+0000 7fa004e00640 1 -- 172.21.15.45:0/3491101792 learned_addr learned my addr 172.21.15.45:0/3491101792 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:44.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.902+0000 7fa005800640 1 --2- 172.21.15.45:0/3491101792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0000ce9a0 0x7fa00015ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:44.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.902+0000 7fa006200640 1 --2- 172.21.15.45:0/3491101792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa00015e8f0 0x7fa00015da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:44.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.902+0000 7fa004e00640 1 -- 172.21.15.45:0/3491101792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0000ce9a0 msgr2=0x7fa00015ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:44.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.902+0000 7fa004e00640 1 --2- 172.21.15.45:0/3491101792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0000ce9a0 0x7fa00015ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:44.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.902+0000 7fa004e00640 1 -- 172.21.15.45:0/3491101792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa00015e8f0 msgr2=0x7fa00015da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:44.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.902+0000 7fa004e00640 1 --2- 172.21.15.45:0/3491101792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa00015e8f0 0x7fa00015da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:44.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.902+0000 7fa004e00640 1 -- 172.21.15.45:0/3491101792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00015c9f0 con 0x7fa0000cf080 2024-04-16T16:20:44.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.902+0000 7fa004e00640 1 --2- 172.21.15.45:0/3491101792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0000cf080 0x7fa000140ed0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f9fec009890 tx=0x7f9fec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:44.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.902+0000 7f9ffd600640 1 -- 172.21.15.45:0/3491101792 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fec041070 con 0x7fa0000cf080 2024-04-16T16:20:44.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.902+0000 7f9ffd600640 1 -- 172.21.15.45:0/3491101792 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fec03aa30 con 0x7fa0000cf080 2024-04-16T16:20:44.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.902+0000 7f9ffd600640 1 -- 172.21.15.45:0/3491101792 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fec035040 con 0x7fa0000cf080 2024-04-16T16:20:44.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.902+0000 7fa006c00640 1 -- 172.21.15.45:0/3491101792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa00015ccd0 con 0x7fa0000cf080 2024-04-16T16:20:44.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.903+0000 7fa006c00640 1 -- 172.21.15.45:0/3491101792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa00015d290 con 0x7fa0000cf080 2024-04-16T16:20:44.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.904+0000 7fa006c00640 1 -- 172.21.15.45:0/3491101792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa000075780 con 0x7fa0000cf080 2024-04-16T16:20:44.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.905+0000 7f9ffd600640 1 -- 172.21.15.45:0/3491101792 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7f9fec038020 con 0x7fa0000cf080 2024-04-16T16:20:44.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.906+0000 7f9ffd600640 1 --2- 172.21.15.45:0/3491101792 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9fe006ca40 0x7f9fe006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:44.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.906+0000 7f9ffd600640 1 -- 172.21.15.45:0/3491101792 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7f9fec0ad8d0 con 0x7fa0000cf080 2024-04-16T16:20:44.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.906+0000 7fa005800640 1 --2- 172.21.15.45:0/3491101792 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9fe006ca40 0x7f9fe006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:44.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.908+0000 7fa005800640 1 --2- 172.21.15.45:0/3491101792 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9fe006ca40 0x7f9fe006ef00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4006fd0 tx=0x7f9ff4008040 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:44.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:44.910+0000 7f9ffd600640 1 -- 172.21.15.45:0/3491101792 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9fec07bac0 con 0x7fa0000cf080 2024-04-16T16:20:45.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.157+0000 7fa006c00640 1 -- 172.21.15.45:0/3491101792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa00015bc20 con 0x7fa0000cf080 2024-04-16T16:20:45.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.158+0000 7f9ffd600640 1 -- 172.21.15.45:0/3491101792 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5186 (secure 0 0 0) 0x7f9fec07b3c0 con 0x7fa0000cf080 2024-04-16T16:20:45.157 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:45.158 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/929218719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":929218719},{"type":"v1","addr":"172.21.15.103:6835","nonce":929218719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6323,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1619719405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1619719405},{"type":"v1","addr":"172.21.15.103:6837","nonce":1619719405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":8640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/3613288118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3613288118},{"type":"v1","addr":"172.21.15.45:6834","nonce":3613288118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:20:33.816422+0000","modified":"2024-04-16T16:20:36.891870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8643":{"gid":8643,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2060312705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2060312705},{"type":"v1","addr":"172.21.15.45:6835","nonce":2060312705}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8643,"qdb_cluster":[8643]},"id":9}]} 2024-04-16T16:20:45.158 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 91 2024-04-16T16:20:45.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.161+0000 7fa006c00640 1 -- 172.21.15.45:0/3491101792 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9fe006ca40 msgr2=0x7f9fe006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:45.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.161+0000 7fa006c00640 1 --2- 172.21.15.45:0/3491101792 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9fe006ca40 0x7f9fe006ef00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4006fd0 tx=0x7f9ff4008040 comp rx=0 tx=0).stop 2024-04-16T16:20:45.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.161+0000 7fa006c00640 1 -- 172.21.15.45:0/3491101792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0000cf080 msgr2=0x7fa000140ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:45.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.161+0000 7fa006c00640 1 --2- 172.21.15.45:0/3491101792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0000cf080 0x7fa000140ed0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f9fec009890 tx=0x7f9fec0046c0 comp rx=0 tx=0).stop 2024-04-16T16:20:45.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.161+0000 7fa006c00640 1 -- 172.21.15.45:0/3491101792 shutdown_connections 2024-04-16T16:20:45.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.161+0000 7fa006c00640 1 --2- 172.21.15.45:0/3491101792 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f9fe006ca40 0x7f9fe006ef00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:45.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.161+0000 7fa006c00640 1 --2- 172.21.15.45:0/3491101792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa00015e8f0 0x7fa00015da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:45.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.161+0000 7fa006c00640 1 --2- 172.21.15.45:0/3491101792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0000cf080 0x7fa000140ed0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:45.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.161+0000 7fa006c00640 1 --2- 172.21.15.45:0/3491101792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0000ce9a0 0x7fa00015ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:45.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.161+0000 7fa006c00640 1 -- 172.21.15.45:0/3491101792 >> 172.21.15.45:0/3491101792 conn(0x7fa00006b930 msgr2=0x7fa00006df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:45.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.161+0000 7fa006c00640 1 -- 172.21.15.45:0/3491101792 shutdown_connections 2024-04-16T16:20:45.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.161+0000 7fa006c00640 1 -- 172.21.15.45:0/3491101792 wait complete. 2024-04-16T16:20:45.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable stats 2024-04-16T16:20:45.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.407+0000 7fd514800640 1 Processor -- start 2024-04-16T16:20:45.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.408+0000 7fd514800640 1 -- start start 2024-04-16T16:20:45.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.409+0000 7fd514800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd50c06d8b0 0x7fd50c0d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:45.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.409+0000 7fd514800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd50c0d0a00 con 0x7fd50c06d450 2024-04-16T16:20:45.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.409+0000 7fd514800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd50c0d0ca0 con 0x7fd50c09ac30 2024-04-16T16:20:45.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.409+0000 7fd514800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd50c0d15d0 con 0x7fd50c06d8b0 2024-04-16T16:20:45.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.409+0000 7fd511400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd50c06d450 0x7fd50c1607f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41932/0 (socket says 172.21.15.45:41932) 2024-04-16T16:20:45.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.409+0000 7fd511400640 1 -- 172.21.15.45:0/3278852509 learned_addr learned my addr 172.21.15.45:0/3278852509 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:45.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.410+0000 7fd510a00640 1 -- 172.21.15.45:0/3278852509 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2831072967 0 0) 0x7fd50c0d0a00 con 0x7fd50c06d450 2024-04-16T16:20:45.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.410+0000 7fd510a00640 1 -- 172.21.15.45:0/3278852509 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4f4003520 con 0x7fd50c06d450 2024-04-16T16:20:45.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.410+0000 7fd510a00640 1 -- 172.21.15.45:0/3278852509 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 401551027 0 0) 0x7fd4f4003520 con 0x7fd50c06d450 2024-04-16T16:20:45.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.410+0000 7fd512800640 1 --2- 172.21.15.45:0/3278852509 >> v2:172.21.15.103:3300/0 conn(0x7fd50c06d8b0 0x7fd50c0d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:45.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.410+0000 7fd510a00640 1 -- 172.21.15.45:0/3278852509 >> v2:172.21.15.103:3300/0 conn(0x7fd50c06d8b0 msgr2=0x7fd50c0d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:45.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.410+0000 7fd510a00640 1 --2- 172.21.15.45:0/3278852509 >> v2:172.21.15.103:3300/0 conn(0x7fd50c06d8b0 0x7fd50c0d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:45.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.410+0000 7fd510a00640 1 -- 172.21.15.45:0/3278852509 >> v1:172.21.15.103:6789/0 conn(0x7fd50c09ac30 legacy=0x7fd50c09b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:45.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.410+0000 7fd510a00640 1 -- 172.21.15.45:0/3278852509 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd50c0d0f80 con 0x7fd50c06d450 2024-04-16T16:20:45.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.410+0000 7fd510a00640 1 -- 172.21.15.45:0/3278852509 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fd4fc001670 con 0x7fd50c06d450 2024-04-16T16:20:45.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.411+0000 7fd512800640 1 --2- 172.21.15.45:0/3278852509 >> v2:172.21.15.103:3300/0 conn(0x7fd50c06d8b0 0x7fd50c0d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:45.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.411+0000 7fd510a00640 1 -- 172.21.15.45:0/3278852509 >> v1:172.21.15.45:6789/0 conn(0x7fd50c06d450 legacy=0x7fd50c1607f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:45.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.411+0000 7fd510a00640 1 --2- 172.21.15.45:0/3278852509 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039b0 0x7fd4f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:45.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.411+0000 7fd510a00640 1 --2- 172.21.15.45:0/3278852509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4f4004800 0x7fd4f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:45.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.411+0000 7fd510a00640 1 --2- 172.21.15.45:0/3278852509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f40053c0 0x7fd4f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.411+0000 7fd510a00640 1 -- 172.21.15.45:0/3278852509 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c0d15d0 con 0x7fd4f40039b0 2024-04-16T16:20:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.411+0000 7fd510a00640 1 -- 172.21.15.45:0/3278852509 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f400c280 con 0x7fd4f40053c0 2024-04-16T16:20:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.411+0000 7fd510a00640 1 -- 172.21.15.45:0/3278852509 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd4f400c480 con 0x7fd4f4004800 2024-04-16T16:20:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.411+0000 7fd511e00640 1 --2- 172.21.15.45:0/3278852509 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039b0 0x7fd4f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.411+0000 7fd511400640 1 --2- 172.21.15.45:0/3278852509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4f4004800 0x7fd4f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.411+0000 7fd512800640 1 --2- 172.21.15.45:0/3278852509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f40053c0 0x7fd4f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.412+0000 7fd511e00640 1 -- 172.21.15.45:0/3278852509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4f4004800 msgr2=0x7fd4f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.412+0000 7fd511e00640 1 --2- 172.21.15.45:0/3278852509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd4f4004800 0x7fd4f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.412+0000 7fd511e00640 1 -- 172.21.15.45:0/3278852509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f40053c0 msgr2=0x7fd4f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.412+0000 7fd511e00640 1 --2- 172.21.15.45:0/3278852509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f40053c0 0x7fd4f400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.412+0000 7fd511e00640 1 -- 172.21.15.45:0/3278852509 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f4003520 con 0x7fd4f40039b0 2024-04-16T16:20:45.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.412+0000 7fd512800640 1 --2- 172.21.15.45:0/3278852509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f40053c0 0x7fd4f400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:45.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.412+0000 7fd512800640 1 -- 172.21.15.45:0/3278852509 reap_dead start 2024-04-16T16:20:45.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.413+0000 7fd511e00640 1 --2- 172.21.15.45:0/3278852509 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039b0 0x7fd4f4003dd0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fd500001750 tx=0x7fd4f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef627a469720b0e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:45.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.413+0000 7fd510a00640 1 -- 172.21.15.45:0/3278852509 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd500001370 con 0x7fd4f40039b0 2024-04-16T16:20:45.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.413+0000 7fd510a00640 1 -- 172.21.15.45:0/3278852509 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5000052d0 con 0x7fd4f40039b0 2024-04-16T16:20:45.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.413+0000 7fd510a00640 1 -- 172.21.15.45:0/3278852509 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd500003830 con 0x7fd4f40039b0 2024-04-16T16:20:45.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.414+0000 7fd514800640 1 -- 172.21.15.45:0/3278852509 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039b0 msgr2=0x7fd4f4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:45.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.414+0000 7fd514800640 1 --2- 172.21.15.45:0/3278852509 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039b0 0x7fd4f4003dd0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fd500001750 tx=0x7fd4f4005830 comp rx=0 tx=0).stop 2024-04-16T16:20:45.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.414+0000 7fd514800640 1 -- 172.21.15.45:0/3278852509 shutdown_connections 2024-04-16T16:20:45.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.414+0000 7fd514800640 1 --2- 172.21.15.45:0/3278852509 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039b0 0x7fd4f4003dd0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:45.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.414+0000 7fd514800640 1 -- 172.21.15.45:0/3278852509 >> 172.21.15.45:0/3278852509 conn(0x7fd50c0d2870 msgr2=0x7fd50c0d2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:45.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.414+0000 7fd514800640 1 -- 172.21.15.45:0/3278852509 shutdown_connections 2024-04-16T16:20:45.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.414+0000 7fd514800640 1 -- 172.21.15.45:0/3278852509 wait complete. 2024-04-16T16:20:45.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.416+0000 7fd514800640 1 Processor -- start 2024-04-16T16:20:45.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.416+0000 7fd514800640 1 -- start start 2024-04-16T16:20:45.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.417+0000 7fd514800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039b0 0x7fd50c157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:45.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.417+0000 7fd514800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd50c0c8660 0x7fd50c0c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.417+0000 7fd514800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd50c0c64f0 0x7fd50c0c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.417+0000 7fd514800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c160330 con 0x7fd4f40039b0 2024-04-16T16:20:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.417+0000 7fd514800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c06df10 con 0x7fd50c0c8660 2024-04-16T16:20:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.417+0000 7fd514800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd50c0b3b00 con 0x7fd50c0c64f0 2024-04-16T16:20:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.417+0000 7fd511e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039b0 0x7fd50c157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.418+0000 7fd511e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039b0 0x7fd50c157210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54608/0 (socket says 172.21.15.45:54608) 2024-04-16T16:20:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.418+0000 7fd511e00640 1 -- 172.21.15.45:0/2633077212 learned_addr learned my addr 172.21.15.45:0/2633077212 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.418+0000 7fd512800640 1 --2- 172.21.15.45:0/2633077212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd50c0c64f0 0x7fd50c0c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.418+0000 7fd511400640 1 --2- 172.21.15.45:0/2633077212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd50c0c8660 0x7fd50c0c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.418+0000 7fd511e00640 1 -- 172.21.15.45:0/2633077212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd50c0c64f0 msgr2=0x7fd50c0c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.418+0000 7fd511e00640 1 --2- 172.21.15.45:0/2633077212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd50c0c64f0 0x7fd50c0c6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.418+0000 7fd511e00640 1 -- 172.21.15.45:0/2633077212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd50c0c8660 msgr2=0x7fd50c0c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.418+0000 7fd511e00640 1 --2- 172.21.15.45:0/2633077212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd50c0c8660 0x7fd50c0c8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.418+0000 7fd511e00640 1 -- 172.21.15.45:0/2633077212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd50c0c7240 con 0x7fd4f40039b0 2024-04-16T16:20:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.418+0000 7fd511e00640 1 --2- 172.21.15.45:0/2633077212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039b0 0x7fd50c157210 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fd500001870 tx=0x7fd500001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.419+0000 7fd506000640 1 -- 172.21.15.45:0/2633077212 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd500041070 con 0x7fd4f40039b0 2024-04-16T16:20:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.419+0000 7fd506000640 1 -- 172.21.15.45:0/2633077212 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd500005480 con 0x7fd4f40039b0 2024-04-16T16:20:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.419+0000 7fd506000640 1 -- 172.21.15.45:0/2633077212 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50003d640 con 0x7fd4f40039b0 2024-04-16T16:20:45.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.419+0000 7fd514800640 1 -- 172.21.15.45:0/2633077212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd50c0c7520 con 0x7fd4f40039b0 2024-04-16T16:20:45.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.419+0000 7fd514800640 1 -- 172.21.15.45:0/2633077212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd50c0c7ae0 con 0x7fd4f40039b0 2024-04-16T16:20:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.420+0000 7fd514800640 1 -- 172.21.15.45:0/2633077212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd50c0c7d20 con 0x7fd4f40039b0 2024-04-16T16:20:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.421+0000 7fd506000640 1 -- 172.21.15.45:0/2633077212 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) ==== 91142+0+0 (secure 0 0 0) 0x7fd500044070 con 0x7fd4f40039b0 2024-04-16T16:20:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.422+0000 7fd506000640 1 --2- 172.21.15.45:0/2633077212 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fd4f006ca40 0x7fd4f006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.422+0000 7fd506000640 1 -- 172.21.15.45:0/2633077212 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7fd5000ad660 con 0x7fd4f40039b0 2024-04-16T16:20:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.423+0000 7fd511400640 1 --2- 172.21.15.45:0/2633077212 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fd4f006ca40 0x7fd4f006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:45.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.424+0000 7fd511400640 1 --2- 172.21.15.45:0/2633077212 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fd4f006ca40 0x7fd4f006ef00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc001eb0 tx=0x7fd4fc001de0 comp rx=0 tx=0).ready entity=mgr.6793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:45.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.427+0000 7fd506000640 1 -- 172.21.15.45:0/2633077212 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd50c0c7d20 con 0x7fd4f40039b0 2024-04-16T16:20:45.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:45.687+0000 7fd514800640 1 -- 172.21.15.45:0/2633077212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "stats"} v 0) -- 0x7fd50c075780 con 0x7fd4f40039b0 2024-04-16T16:20:46.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.594+0000 7fd506000640 1 -- 172.21.15.45:0/2633077212 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "stats"}]=0 v50) ==== 84+0+0 (secure 0 0 0) 0x7fd50007b750 con 0x7fd4f40039b0 2024-04-16T16:20:46.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.597+0000 7fd514800640 1 -- 172.21.15.45:0/2633077212 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fd4f006ca40 msgr2=0x7fd4f006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:46.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.597+0000 7fd514800640 1 --2- 172.21.15.45:0/2633077212 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fd4f006ca40 0x7fd4f006ef00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc001eb0 tx=0x7fd4fc001de0 comp rx=0 tx=0).stop 2024-04-16T16:20:46.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.597+0000 7fd514800640 1 -- 172.21.15.45:0/2633077212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039b0 msgr2=0x7fd50c157210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:46.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.597+0000 7fd514800640 1 --2- 172.21.15.45:0/2633077212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039b0 0x7fd50c157210 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fd500001870 tx=0x7fd500001120 comp rx=0 tx=0).stop 2024-04-16T16:20:46.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.598+0000 7fd514800640 1 -- 172.21.15.45:0/2633077212 shutdown_connections 2024-04-16T16:20:46.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.598+0000 7fd514800640 1 --2- 172.21.15.45:0/2633077212 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7fd4f006ca40 0x7fd4f006ef00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:46.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.598+0000 7fd514800640 1 --2- 172.21.15.45:0/2633077212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd50c0c64f0 0x7fd50c0c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:46.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.598+0000 7fd514800640 1 --2- 172.21.15.45:0/2633077212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd50c0c8660 0x7fd50c0c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:46.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.598+0000 7fd514800640 1 --2- 172.21.15.45:0/2633077212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4f40039b0 0x7fd50c157210 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:46.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.598+0000 7fd514800640 1 -- 172.21.15.45:0/2633077212 >> 172.21.15.45:0/2633077212 conn(0x7fd50c0d2870 msgr2=0x7fd50c0ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:46.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.599+0000 7fd514800640 1 -- 172.21.15.45:0/2633077212 shutdown_connections 2024-04-16T16:20:46.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.599+0000 7fd514800640 1 -- 172.21.15.45:0/2633077212 wait complete. 2024-04-16T16:20:46.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:20:46.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.847+0000 7f0914400640 1 Processor -- start 2024-04-16T16:20:46.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.847+0000 7f0914400640 1 -- start start 2024-04-16T16:20:46.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.848+0000 7f0914400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f090c1453c0 0x7f090c0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:46.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.848+0000 7f0914400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f090c06b930 0x7f090c06bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:46.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.849+0000 7f0914400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f090c0b6930 con 0x7f090c0b63c0 2024-04-16T16:20:46.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.849+0000 7f0914400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f090c06c2c0 con 0x7f090c1453c0 2024-04-16T16:20:46.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.849+0000 7f0914400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f090c0b6c10 con 0x7f090c06b930 2024-04-16T16:20:46.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.849+0000 7f0911a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f090c1453c0 0x7f090c0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:46.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.849+0000 7f0911a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f090c1453c0 0x7f090c0b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54628/0 (socket says 172.21.15.45:54628) 2024-04-16T16:20:46.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.849+0000 7f0911a00640 1 -- 172.21.15.45:0/4073882774 learned_addr learned my addr 172.21.15.45:0/4073882774 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:46.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.849+0000 7f0912400640 1 --2- 172.21.15.45:0/4073882774 >> v2:172.21.15.103:3300/0 conn(0x7f090c06b930 0x7f090c06bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:46.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.849+0000 7f0911a00640 1 -- 172.21.15.45:0/4073882774 >> v2:172.21.15.103:3300/0 conn(0x7f090c06b930 msgr2=0x7f090c06bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:46.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.849+0000 7f0911a00640 1 --2- 172.21.15.45:0/4073882774 >> v2:172.21.15.103:3300/0 conn(0x7f090c06b930 0x7f090c06bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:46.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.850+0000 7f0911a00640 1 -- 172.21.15.45:0/4073882774 >> v1:172.21.15.103:6789/0 conn(0x7f090c0b63c0 legacy=0x7f090c15f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:46.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.850+0000 7f0911a00640 1 -- 172.21.15.45:0/4073882774 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090c147590 con 0x7f090c1453c0 2024-04-16T16:20:46.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.851+0000 7f0911a00640 1 --2- 172.21.15.45:0/4073882774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f090c1453c0 0x7f090c0b47d0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f090c074140 tx=0x7f090002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ec815a3c90c6ee server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:46.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.851+0000 7f090be00640 1 -- 172.21.15.45:0/4073882774 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0900035020 con 0x7f090c1453c0 2024-04-16T16:20:46.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.851+0000 7f090be00640 1 -- 172.21.15.45:0/4073882774 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0900031870 con 0x7f090c1453c0 2024-04-16T16:20:46.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.851+0000 7f090be00640 1 -- 172.21.15.45:0/4073882774 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0900031c40 con 0x7f090c1453c0 2024-04-16T16:20:46.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.852+0000 7f0914400640 1 -- 172.21.15.45:0/4073882774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f090c1453c0 msgr2=0x7f090c0b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:46.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.852+0000 7f0914400640 1 --2- 172.21.15.45:0/4073882774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f090c1453c0 0x7f090c0b47d0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f090c074140 tx=0x7f090002b910 comp rx=0 tx=0).stop 2024-04-16T16:20:46.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.852+0000 7f0914400640 1 -- 172.21.15.45:0/4073882774 shutdown_connections 2024-04-16T16:20:46.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.852+0000 7f0914400640 1 --2- 172.21.15.45:0/4073882774 >> v2:172.21.15.103:3300/0 conn(0x7f090c06b930 0x7f090c06bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:46.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.852+0000 7f0914400640 1 --2- 172.21.15.45:0/4073882774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f090c1453c0 0x7f090c0b47d0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:46.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.852+0000 7f0914400640 1 -- 172.21.15.45:0/4073882774 >> 172.21.15.45:0/4073882774 conn(0x7f090c0b6ea0 msgr2=0x7f090c09aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:46.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.852+0000 7f0914400640 1 -- 172.21.15.45:0/4073882774 shutdown_connections 2024-04-16T16:20:46.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.852+0000 7f0914400640 1 -- 172.21.15.45:0/4073882774 wait complete. 2024-04-16T16:20:46.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.854+0000 7f0914400640 1 Processor -- start 2024-04-16T16:20:46.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.854+0000 7f0914400640 1 -- start start 2024-04-16T16:20:46.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.855+0000 7f0914400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f090c06b930 0x7f090c1401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:46.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.855+0000 7f0914400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f090c0b63c0 0x7f090c1549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:46.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.855+0000 7f0914400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f090c1453c0 0x7f090c0b1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:46.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.855+0000 7f0914400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f090c15fa00 con 0x7f090c06b930 2024-04-16T16:20:46.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.855+0000 7f0914400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f090c147b50 con 0x7f090c0b63c0 2024-04-16T16:20:46.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.855+0000 7f0914400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f090c0b4d10 con 0x7f090c1453c0 2024-04-16T16:20:46.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.855+0000 7f0911a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f090c06b930 0x7f090c1401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:46.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.855+0000 7f0911a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f090c06b930 0x7f090c1401c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54632/0 (socket says 172.21.15.45:54632) 2024-04-16T16:20:46.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.855+0000 7f0911a00640 1 -- 172.21.15.45:0/1118256428 learned_addr learned my addr 172.21.15.45:0/1118256428 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.856+0000 7f0911000640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f090c0b63c0 0x7f090c1549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.856+0000 7f0912400640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f090c1453c0 0x7f090c0b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.856+0000 7f0911a00640 1 -- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f090c1453c0 msgr2=0x7f090c0b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.856+0000 7f0911a00640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f090c1453c0 0x7f090c0b1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.856+0000 7f0911a00640 1 -- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f090c0b63c0 msgr2=0x7f090c1549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.856+0000 7f0911a00640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f090c0b63c0 0x7f090c1549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.856+0000 7f0911a00640 1 -- 172.21.15.45:0/1118256428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090c140700 con 0x7f090c06b930 2024-04-16T16:20:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.856+0000 7f0911000640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f090c0b63c0 0x7f090c1549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.856+0000 7f0911a00640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f090c06b930 0x7f090c1401c0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f0900000c00 tx=0x7f09000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.856+0000 7f0909600640 1 -- 172.21.15.45:0/1118256428 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0900040070 con 0x7f090c06b930 2024-04-16T16:20:46.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.856+0000 7f0909600640 1 -- 172.21.15.45:0/1118256428 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0900030d00 con 0x7f090c06b930 2024-04-16T16:20:46.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.856+0000 7f0909600640 1 -- 172.21.15.45:0/1118256428 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0900035040 con 0x7f090c06b930 2024-04-16T16:20:46.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.856+0000 7f0914400640 1 -- 172.21.15.45:0/1118256428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f090c154ef0 con 0x7f090c06b930 2024-04-16T16:20:46.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.857+0000 7f0914400640 1 -- 172.21.15.45:0/1118256428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f090c09ccf0 con 0x7f090c06b930 2024-04-16T16:20:46.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.858+0000 7f0914400640 1 -- 172.21.15.45:0/1118256428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f090c075780 con 0x7f090c06b930 2024-04-16T16:20:46.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.859+0000 7f0909600640 1 -- 172.21.15.45:0/1118256428 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 50) ==== 91151+0+0 (secure 0 0 0) 0x7f090004b060 con 0x7f090c06b930 2024-04-16T16:20:46.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.860+0000 7f0909600640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f08ec06ca60 0x7f08ec06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:46.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.860+0000 7f0909600640 1 -- 172.21.15.45:0/1118256428 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5709+0+0 (secure 0 0 0) 0x7f09000adbb0 con 0x7f090c06b930 2024-04-16T16:20:46.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.860+0000 7f0911000640 1 -- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f08ec06ca60 msgr2=0x7f08ec06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1120913661 2024-04-16T16:20:46.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.860+0000 7f0911000640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f08ec06ca60 0x7f08ec06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:20:46.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:46.864+0000 7f0909600640 1 -- 172.21.15.45:0/1118256428 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f090c140700 con 0x7f090c06b930 2024-04-16T16:20:47.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:47.061+0000 7f0911000640 1 -- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f08ec06ca60 msgr2=0x7f08ec06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1120913661 2024-04-16T16:20:47.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:47.061+0000 7f0911000640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f08ec06ca60 0x7f08ec06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:20:47.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:47.097+0000 7f0914400640 1 -- 172.21.15.45:0/1118256428 --> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f090c0afbb0 con 0x7f08ec06ca60 2024-04-16T16:20:47.176 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:47.177+0000 7f2d8e83b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:20:47.228 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:47.229+0000 7fa9bc814040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:20:47.233 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:47.234+0000 7f2d8e83b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:20:47.286 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:47.287+0000 7fa9bc814040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:20:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:47.462+0000 7f0911000640 1 -- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f08ec06ca60 msgr2=0x7f08ec06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1120913661 2024-04-16T16:20:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:47.462+0000 7f0911000640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f08ec06ca60 0x7f08ec06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T16:20:47.685 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:47.687+0000 7f2d8e83b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:20:47.738 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:47.740+0000 7f2d8e83b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:20:47.753 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:47.755+0000 7fa9bc814040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:20:47.790 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:47.791+0000 7f2d8e83b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:20:47.808 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:47.809+0000 7fa9bc814040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:20:47.852 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:47.854+0000 7f2d8e83b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:20:47.861 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:47.862+0000 7fa9bc814040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:20:47.902 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:47.904+0000 7f2d8e83b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:20:47.925 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:47.926+0000 7fa9bc814040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:20:47.977 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:47.978+0000 7fa9bc814040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:20:48.034 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:48.036+0000 7f2d8e83b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:20:48.100 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:48.102+0000 7f2d8e83b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:20:48.113 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:48.114+0000 7fa9bc814040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:20:48.150 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:48.152+0000 7f2d8e83b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:20:48.179 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:48.180+0000 7fa9bc814040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:20:48.209 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:48.210+0000 7f2d8e83b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:20:48.231 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:48.232+0000 7fa9bc814040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:20:48.260 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:48.261+0000 7f2d8e83b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:20:48.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:48.263+0000 7f0911000640 1 -- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f08ec06ca60 msgr2=0x7f08ec06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1120913661 2024-04-16T16:20:48.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:48.263+0000 7f0911000640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f08ec06ca60 0x7f08ec06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T16:20:48.292 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:48.292+0000 7fa9bc814040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:20:48.345 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:48.346+0000 7fa9bc814040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:20:48.650 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:48.651+0000 7f2d8e83b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:20:48.748 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:48.749+0000 7fa9bc814040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:20:48.811 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:48.812+0000 7f2d8e83b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:20:48.915 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:48.916+0000 7fa9bc814040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:20:48.921 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:48.923+0000 7f2d8e83b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:20:48.974 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:48.975+0000 7f2d8e83b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:20:49.030 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:49.031+0000 7fa9bc814040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:20:49.083 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:49.084+0000 7fa9bc814040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:20:49.104 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:49.105+0000 7f2d8e83b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:20:49.214 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:49.215+0000 7f2d8e83b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:20:49.216 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:49.217+0000 7fa9bc814040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:20:49.329 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:49.330+0000 7fa9bc814040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:20:49.394 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:49.395+0000 7f2d8e83b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:20:49.445 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:49.446+0000 7f2d8e83b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:20:49.517 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:49.517+0000 7fa9bc814040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:20:49.568 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:49.569+0000 7fa9bc814040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:20:49.584 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:49.585+0000 7f2d8e83b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:20:49.712 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:49.713+0000 7fa9bc814040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:20:49.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:49.865+0000 7f0911000640 1 -- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f08ec06ca60 msgr2=0x7f08ec06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1120913661 2024-04-16T16:20:49.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:49.865+0000 7f0911000640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f08ec06ca60 0x7f08ec06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T16:20:50.422 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:50.424+0000 7f2d8e83b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:20:50.573 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:50.573+0000 7fa9bc814040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:20:50.739 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:50.741+0000 7f2d8e83b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:20:50.797 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:50.799+0000 7f2d8e83b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:20:50.892 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:50.893+0000 7fa9bc814040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:20:50.929 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:50.931+0000 7f2d8e83b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:20:50.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:50.950+0000 7f0909600640 1 -- 172.21.15.45:0/1118256428 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 51) ==== 90760+0+0 (secure 0 0 0) 0x7f0900038020 con 0x7f090c06b930 2024-04-16T16:20:50.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:50.951+0000 7f0909600640 1 -- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f08ec06ca60 msgr2=0x7f08ec06ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:20:50.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:50.951+0000 7f0909600640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/1120913661,v1:172.21.15.103:6833/1120913661] conn(0x7f08ec06ca60 0x7f08ec06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:50.957 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:50.958+0000 7fa9bc814040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:20:51.024 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:51.026+0000 7f2d0a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:51.025 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:51.026+0000 7f2d0a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:51.025 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:51.026+0000 7f2d0a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:51.025 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:51.026+0000 7f2d0a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:51.025 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:51.026+0000 7f2d0a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:51.026 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:51.027+0000 7f2d06600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:51.026 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:51.027+0000 7f2d06600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:51.026 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:51.027+0000 7f2d06600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:51.026 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:51.027+0000 7f2d06600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:51.026 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:20:51.027+0000 7f2d06600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:20:51.099 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:20:51.100+0000 7fa9bc814040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:20:51.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:51.963+0000 7f0909600640 1 -- 172.21.15.45:0/1118256428 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 52) ==== 90887+0+0 (secure 0 0 0) 0x7f0900077da0 con 0x7f090c06b930 2024-04-16T16:20:51.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:51.964+0000 7f0909600640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f08ec074910 0x7f08ec076d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:51.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:51.964+0000 7f0909600640 1 -- 172.21.15.45:0/1118256428 --> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f0900082bb0 con 0x7f08ec074910 2024-04-16T16:20:51.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:51.971+0000 7f0911000640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f08ec074910 0x7f08ec076d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:51.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:51.979+0000 7f0911000640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f08ec074910 0x7f08ec076d00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f090c09f380 tx=0x7f08f80039f0 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:51.990+0000 7f0909600640 1 -- 172.21.15.45:0/1118256428 <== mgr.6389 v2:172.21.15.103:6832/549890383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7f0900082bb0 con 0x7f08ec074910 2024-04-16T16:20:51.989 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:20:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:51.994+0000 7f0914400640 1 -- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f08ec074910 msgr2=0x7f08ec076d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:51.995+0000 7f0914400640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f08ec074910 0x7f08ec076d00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f090c09f380 tx=0x7f08f80039f0 comp rx=0 tx=0).stop 2024-04-16T16:20:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:51.995+0000 7f0914400640 1 -- 172.21.15.45:0/1118256428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f090c06b930 msgr2=0x7f090c1401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:51.995+0000 7f0914400640 1 --2- 172.21.15.45:0/1118256428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f090c06b930 0x7f090c1401c0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f0900000c00 tx=0x7f09000046c0 comp rx=0 tx=0).stop 2024-04-16T16:20:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:51.995+0000 7f0912400640 1 -- 172.21.15.45:0/1118256428 reap_dead start 2024-04-16T16:20:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:51.995+0000 7f0914400640 1 -- 172.21.15.45:0/1118256428 shutdown_connections 2024-04-16T16:20:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:51.995+0000 7f0914400640 1 -- 172.21.15.45:0/1118256428 >> 172.21.15.45:0/1118256428 conn(0x7f090c0b6ea0 msgr2=0x7f090c09ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:51.995+0000 7f0914400640 1 -- 172.21.15.45:0/1118256428 shutdown_connections 2024-04-16T16:20:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:51.995+0000 7f0914400640 1 -- 172.21.15.45:0/1118256428 wait complete. 2024-04-16T16:20:53.018 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:20:53.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.234+0000 7f898c400640 1 Processor -- start 2024-04-16T16:20:53.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.235+0000 7f898c400640 1 -- start start 2024-04-16T16:20:53.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.236+0000 7f898c400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f89840cdfe0 0x7f89840b4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:53.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.236+0000 7f898c400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f898406fab0 con 0x7f898415ca00 2024-04-16T16:20:53.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.236+0000 7f898c400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f898415bf20 con 0x7f89840afe20 2024-04-16T16:20:53.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.236+0000 7f898c400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f898415c200 con 0x7f89840cdfe0 2024-04-16T16:20:53.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.237+0000 7f8989000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f898415ca00 0x7f89840cd8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41944/0 (socket says 172.21.15.45:41944) 2024-04-16T16:20:53.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.237+0000 7f8989000640 1 -- 172.21.15.45:0/2294732875 learned_addr learned my addr 172.21.15.45:0/2294732875 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:53.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.237+0000 7f8983e00640 1 -- 172.21.15.45:0/2294732875 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2252356105 0 0) 0x7f898406fab0 con 0x7f898415ca00 2024-04-16T16:20:53.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.237+0000 7f8983e00640 1 -- 172.21.15.45:0/2294732875 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f896c005730 con 0x7f898415ca00 2024-04-16T16:20:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.237+0000 7f898a400640 1 --2- 172.21.15.45:0/2294732875 >> v2:172.21.15.103:3300/0 conn(0x7f89840cdfe0 0x7f89840b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.237+0000 7f8983e00640 1 -- 172.21.15.45:0/2294732875 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 731437062 0 0) 0x7f896c005730 con 0x7f898415ca00 2024-04-16T16:20:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.237+0000 7f8983e00640 1 -- 172.21.15.45:0/2294732875 >> v2:172.21.15.103:3300/0 conn(0x7f89840cdfe0 msgr2=0x7f89840b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.237+0000 7f8983e00640 1 --2- 172.21.15.45:0/2294732875 >> v2:172.21.15.103:3300/0 conn(0x7f89840cdfe0 0x7f89840b4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.237+0000 7f8983e00640 1 -- 172.21.15.45:0/2294732875 >> v1:172.21.15.103:6789/0 conn(0x7f89840afe20 legacy=0x7f898406f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.238+0000 7f8983e00640 1 -- 172.21.15.45:0/2294732875 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f898415c4e0 con 0x7f898415ca00 2024-04-16T16:20:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.238+0000 7f8983e00640 1 -- 172.21.15.45:0/2294732875 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f8970001670 con 0x7f898415ca00 2024-04-16T16:20:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.238+0000 7f8983e00640 1 -- 172.21.15.45:0/2294732875 >> v1:172.21.15.45:6789/0 conn(0x7f898415ca00 legacy=0x7f89840cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.238+0000 7f8983e00640 1 --2- 172.21.15.45:0/2294732875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f896c005bc0 0x7f896c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.238+0000 7f8983e00640 1 --2- 172.21.15.45:0/2294732875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f896c006a10 0x7f896c006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.238+0000 7f8983e00640 1 --2- 172.21.15.45:0/2294732875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f896c0073d0 0x7f896c00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.238+0000 7f8983e00640 1 -- 172.21.15.45:0/2294732875 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f898415c200 con 0x7f896c006a10 2024-04-16T16:20:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.238+0000 7f8983e00640 1 -- 172.21.15.45:0/2294732875 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f896c00e320 con 0x7f896c005bc0 2024-04-16T16:20:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.238+0000 7f8983e00640 1 -- 172.21.15.45:0/2294732875 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f896c00e520 con 0x7f896c0073d0 2024-04-16T16:20:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.238+0000 7f8989000640 1 --2- 172.21.15.45:0/2294732875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f896c006a10 0x7f896c006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.238+0000 7f8989a00640 1 --2- 172.21.15.45:0/2294732875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f896c005bc0 0x7f896c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.238+0000 7f8989000640 1 -- 172.21.15.45:0/2294732875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f896c0073d0 msgr2=0x7f896c00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.238+0000 7f8989000640 1 --2- 172.21.15.45:0/2294732875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f896c0073d0 0x7f896c00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.238+0000 7f8989000640 1 -- 172.21.15.45:0/2294732875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f896c005bc0 msgr2=0x7f896c005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.238+0000 7f8989000640 1 --2- 172.21.15.45:0/2294732875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f896c005bc0 0x7f896c005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.239+0000 7f8989000640 1 -- 172.21.15.45:0/2294732875 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896c005730 con 0x7f896c006a10 2024-04-16T16:20:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.239+0000 7f898a400640 1 -- 172.21.15.45:0/2294732875 reap_dead start 2024-04-16T16:20:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.239+0000 7f8989a00640 1 --2- 172.21.15.45:0/2294732875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f896c005bc0 0x7f896c005fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.240+0000 7f8989000640 1 --2- 172.21.15.45:0/2294732875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f896c006a10 0x7f896c006e90 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f897000c240 tx=0x7f896c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56e3448b69ff3ee server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.240+0000 7f8983e00640 1 -- 172.21.15.45:0/2294732875 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8970007cf0 con 0x7f896c006a10 2024-04-16T16:20:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.240+0000 7f8983e00640 1 -- 172.21.15.45:0/2294732875 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8970030070 con 0x7f896c006a10 2024-04-16T16:20:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.240+0000 7f8983e00640 1 -- 172.21.15.45:0/2294732875 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8970005d30 con 0x7f896c006a10 2024-04-16T16:20:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.240+0000 7f898c400640 1 -- 172.21.15.45:0/2294732875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f896c006a10 msgr2=0x7f896c006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.240+0000 7f898c400640 1 --2- 172.21.15.45:0/2294732875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f896c006a10 0x7f896c006e90 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f897000c240 tx=0x7f896c007a40 comp rx=0 tx=0).stop 2024-04-16T16:20:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.241+0000 7f898c400640 1 -- 172.21.15.45:0/2294732875 shutdown_connections 2024-04-16T16:20:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.241+0000 7f898c400640 1 --2- 172.21.15.45:0/2294732875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f896c006a10 0x7f896c006e90 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.241+0000 7f898c400640 1 -- 172.21.15.45:0/2294732875 >> 172.21.15.45:0/2294732875 conn(0x7f898406b930 msgr2=0x7f898406bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.241+0000 7f898c400640 1 -- 172.21.15.45:0/2294732875 shutdown_connections 2024-04-16T16:20:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.241+0000 7f898c400640 1 -- 172.21.15.45:0/2294732875 wait complete. 2024-04-16T16:20:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.242+0000 7f898c400640 1 Processor -- start 2024-04-16T16:20:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.243+0000 7f898c400640 1 -- start start 2024-04-16T16:20:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.243+0000 7f898c400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f896c006a10 0x7f89841f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.244+0000 7f898c400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f898415b150 0x7f898415b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.244+0000 7f898c400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89841563d0 0x7f8984156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.244+0000 7f898c400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f89840b59d0 con 0x7f896c006a10 2024-04-16T16:20:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.244+0000 7f898c400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89840b5850 con 0x7f898415b150 2024-04-16T16:20:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.244+0000 7f898c400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f89840cd8d0 con 0x7f89841563d0 2024-04-16T16:20:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.244+0000 7f8989a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f896c006a10 0x7f89841f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.244+0000 7f8989a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f896c006a10 0x7f89841f4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54648/0 (socket says 172.21.15.45:54648) 2024-04-16T16:20:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.244+0000 7f8989a00640 1 -- 172.21.15.45:0/3886637641 learned_addr learned my addr 172.21.15.45:0/3886637641 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.244+0000 7f898a400640 1 --2- 172.21.15.45:0/3886637641 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89841563d0 0x7f8984156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.244+0000 7f8989000640 1 --2- 172.21.15.45:0/3886637641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f898415b150 0x7f898415b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.244+0000 7f8989a00640 1 -- 172.21.15.45:0/3886637641 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89841563d0 msgr2=0x7f8984156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.244+0000 7f8989a00640 1 --2- 172.21.15.45:0/3886637641 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89841563d0 0x7f8984156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.244+0000 7f8989a00640 1 -- 172.21.15.45:0/3886637641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f898415b150 msgr2=0x7f898415b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.245+0000 7f8989a00640 1 --2- 172.21.15.45:0/3886637641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f898415b150 0x7f898415b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.245+0000 7f8989a00640 1 -- 172.21.15.45:0/3886637641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f898415a0b0 con 0x7f896c006a10 2024-04-16T16:20:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.245+0000 7f8989a00640 1 --2- 172.21.15.45:0/3886637641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f896c006a10 0x7f89841f4600 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f897800d430 tx=0x7f897800d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.245+0000 7f8981600640 1 -- 172.21.15.45:0/3886637641 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8978018070 con 0x7f896c006a10 2024-04-16T16:20:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.245+0000 7f8981600640 1 -- 172.21.15.45:0/3886637641 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8978003610 con 0x7f896c006a10 2024-04-16T16:20:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.245+0000 7f8981600640 1 -- 172.21.15.45:0/3886637641 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8978003a00 con 0x7f896c006a10 2024-04-16T16:20:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.245+0000 7f898c400640 1 -- 172.21.15.45:0/3886637641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f898415a390 con 0x7f896c006a10 2024-04-16T16:20:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.245+0000 7f898c400640 1 -- 172.21.15.45:0/3886637641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f898415a950 con 0x7f896c006a10 2024-04-16T16:20:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.247+0000 7f898c400640 1 -- 172.21.15.45:0/3886637641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f898406fe50 con 0x7f896c006a10 2024-04-16T16:20:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.248+0000 7f8981600640 1 -- 172.21.15.45:0/3886637641 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 53) ==== 91095+0+0 (secure 0 0 0) 0x7f897801b070 con 0x7f896c006a10 2024-04-16T16:20:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.248+0000 7f8981600640 1 --2- 172.21.15.45:0/3886637641 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f896406c9d0 0x7f896406ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.249+0000 7f8981600640 1 -- 172.21.15.45:0/3886637641 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6053+0+0 (secure 0 0 0) 0x7f897808b5e0 con 0x7f896c006a10 2024-04-16T16:20:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.249+0000 7f8989000640 1 --2- 172.21.15.45:0/3886637641 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f896406c9d0 0x7f896406ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.250+0000 7f8989000640 1 --2- 172.21.15.45:0/3886637641 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f896406c9d0 0x7f896406ee90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8970034000 tx=0x7f8970007000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.254+0000 7f8981600640 1 -- 172.21.15.45:0/3886637641 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f898406fe50 con 0x7f896c006a10 2024-04-16T16:20:53.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.483+0000 7f898c400640 1 -- 172.21.15.45:0/3886637641 --> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f8984075780 con 0x7f896406c9d0 2024-04-16T16:20:53.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.486+0000 7f8981600640 1 -- 172.21.15.45:0/3886637641 <== mgr.6389 v2:172.21.15.103:6832/549890383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7f8984075780 con 0x7f896406c9d0 2024-04-16T16:20:53.485 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:20:53.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.489+0000 7f898c400640 1 -- 172.21.15.45:0/3886637641 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f896406c9d0 msgr2=0x7f896406ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:53.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.489+0000 7f898c400640 1 --2- 172.21.15.45:0/3886637641 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f896406c9d0 0x7f896406ee90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8970034000 tx=0x7f8970007000 comp rx=0 tx=0).stop 2024-04-16T16:20:53.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.489+0000 7f898c400640 1 -- 172.21.15.45:0/3886637641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f896c006a10 msgr2=0x7f89841f4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:53.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.489+0000 7f898c400640 1 --2- 172.21.15.45:0/3886637641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f896c006a10 0x7f89841f4600 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f897800d430 tx=0x7f897800d8f0 comp rx=0 tx=0).stop 2024-04-16T16:20:53.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.489+0000 7f898c400640 1 -- 172.21.15.45:0/3886637641 shutdown_connections 2024-04-16T16:20:53.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.489+0000 7f898c400640 1 --2- 172.21.15.45:0/3886637641 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89841563d0 0x7f8984156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:53.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.489+0000 7f898c400640 1 --2- 172.21.15.45:0/3886637641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f898415b150 0x7f898415b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:53.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.489+0000 7f898c400640 1 --2- 172.21.15.45:0/3886637641 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f896406c9d0 0x7f896406ee90 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:53.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.489+0000 7f898c400640 1 --2- 172.21.15.45:0/3886637641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f896c006a10 0x7f89841f4600 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:53.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.489+0000 7f898c400640 1 -- 172.21.15.45:0/3886637641 >> 172.21.15.45:0/3886637641 conn(0x7f898406b930 msgr2=0x7f89840ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:53.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.489+0000 7f898c400640 1 -- 172.21.15.45:0/3886637641 shutdown_connections 2024-04-16T16:20:53.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:53.489+0000 7f898c400640 1 -- 172.21.15.45:0/3886637641 wait complete. 2024-04-16T16:20:54.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:20:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.758+0000 7f0841e00640 1 Processor -- start 2024-04-16T16:20:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.758+0000 7f0841e00640 1 -- start start 2024-04-16T16:20:54.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.760+0000 7f0841e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f083c0cb7d0 0x7f083c0cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:54.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.760+0000 7f0841e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f083c15f1b0 con 0x7f083c0cbc80 2024-04-16T16:20:54.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.760+0000 7f0841e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f083c0cf280 con 0x7f083c15e830 2024-04-16T16:20:54.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.760+0000 7f0841e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f083c0ce190 con 0x7f083c0cb7d0 2024-04-16T16:20:54.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.760+0000 7f0841400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f083c0cbc80 0x7f083c0cc410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58614/0 (socket says 172.21.15.45:58614) 2024-04-16T16:20:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.760+0000 7f0841400640 1 -- 172.21.15.45:0/4282153497 learned_addr learned my addr 172.21.15.45:0/4282153497 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.761+0000 7f0833e00640 1 --2- 172.21.15.45:0/4282153497 >> v2:172.21.15.103:3300/0 conn(0x7f083c0cb7d0 0x7f083c0cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.761+0000 7f083be00640 1 -- 172.21.15.45:0/4282153497 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3811892142 0 0) 0x7f0834001490 con 0x7f083c0cbc80 2024-04-16T16:20:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.761+0000 7f083be00640 1 -- 172.21.15.45:0/4282153497 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0820003520 con 0x7f083c0cbc80 2024-04-16T16:20:54.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.761+0000 7f083be00640 1 -- 172.21.15.45:0/4282153497 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2364190188 0 0) 0x7f0820003520 con 0x7f083c0cbc80 2024-04-16T16:20:54.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.761+0000 7f083be00640 1 -- 172.21.15.45:0/4282153497 >> v2:172.21.15.103:3300/0 conn(0x7f083c0cb7d0 msgr2=0x7f083c0cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:54.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.761+0000 7f083be00640 1 --2- 172.21.15.45:0/4282153497 >> v2:172.21.15.103:3300/0 conn(0x7f083c0cb7d0 0x7f083c0cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:54.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.761+0000 7f083be00640 1 -- 172.21.15.45:0/4282153497 >> v1:172.21.15.103:6789/0 conn(0x7f083c15e830 legacy=0x7f083c0ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:54.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.761+0000 7f083be00640 1 -- 172.21.15.45:0/4282153497 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083c0ce470 con 0x7f083c0cbc80 2024-04-16T16:20:54.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.761+0000 7f083be00640 1 -- 172.21.15.45:0/4282153497 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f0834001bc0 con 0x7f083c0cbc80 2024-04-16T16:20:54.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.761+0000 7f083be00640 1 -- 172.21.15.45:0/4282153497 >> v1:172.21.15.45:6789/0 conn(0x7f083c0cbc80 legacy=0x7f083c0cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:54.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.762+0000 7f083be00640 1 --2- 172.21.15.45:0/4282153497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0820003a00 0x7f0820003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:54.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.762+0000 7f083be00640 1 --2- 172.21.15.45:0/4282153497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0820004800 0x7f0820004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:54.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.762+0000 7f083be00640 1 --2- 172.21.15.45:0/4282153497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0820005210 0x7f082000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:54.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.762+0000 7f083be00640 1 -- 172.21.15.45:0/4282153497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f083c0ce190 con 0x7f0820003a00 2024-04-16T16:20:54.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.762+0000 7f083be00640 1 -- 172.21.15.45:0/4282153497 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f082000c160 con 0x7f0820004800 2024-04-16T16:20:54.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.762+0000 7f083be00640 1 -- 172.21.15.45:0/4282153497 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f082000c360 con 0x7f0820005210 2024-04-16T16:20:54.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.762+0000 7f0840a00640 1 --2- 172.21.15.45:0/4282153497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0820003a00 0x7f0820003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:54.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.762+0000 7f0833e00640 1 --2- 172.21.15.45:0/4282153497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0820004800 0x7f0820004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:54.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.762+0000 7f0841400640 1 --2- 172.21.15.45:0/4282153497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0820005210 0x7f082000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:54.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.762+0000 7f0840a00640 1 -- 172.21.15.45:0/4282153497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0820005210 msgr2=0x7f082000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:54.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.762+0000 7f0840a00640 1 --2- 172.21.15.45:0/4282153497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0820005210 0x7f082000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:54.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.762+0000 7f0840a00640 1 -- 172.21.15.45:0/4282153497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0820004800 msgr2=0x7f0820004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:54.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.762+0000 7f0840a00640 1 --2- 172.21.15.45:0/4282153497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0820004800 0x7f0820004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:54.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.762+0000 7f0840a00640 1 -- 172.21.15.45:0/4282153497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0820003520 con 0x7f0820003a00 2024-04-16T16:20:54.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.763+0000 7f0841400640 1 --2- 172.21.15.45:0/4282153497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0820005210 0x7f082000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:54.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.763+0000 7f0841400640 1 -- 172.21.15.45:0/4282153497 reap_dead start 2024-04-16T16:20:54.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.764+0000 7f0840a00640 1 --2- 172.21.15.45:0/4282153497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0820003a00 0x7f0820003e20 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f082c001720 tx=0x7f0820005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3c1840b3cc3c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:54.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.764+0000 7f083be00640 1 -- 172.21.15.45:0/4282153497 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082c001370 con 0x7f0820003a00 2024-04-16T16:20:54.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.764+0000 7f083be00640 1 -- 172.21.15.45:0/4282153497 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f082c0052d0 con 0x7f0820003a00 2024-04-16T16:20:54.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.764+0000 7f083be00640 1 -- 172.21.15.45:0/4282153497 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082c003830 con 0x7f0820003a00 2024-04-16T16:20:54.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.764+0000 7f0841e00640 1 -- 172.21.15.45:0/4282153497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0820003a00 msgr2=0x7f0820003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:54.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.764+0000 7f0841e00640 1 --2- 172.21.15.45:0/4282153497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0820003a00 0x7f0820003e20 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f082c001720 tx=0x7f0820005830 comp rx=0 tx=0).stop 2024-04-16T16:20:54.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.765+0000 7f0841e00640 1 -- 172.21.15.45:0/4282153497 shutdown_connections 2024-04-16T16:20:54.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.765+0000 7f0841e00640 1 --2- 172.21.15.45:0/4282153497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0820003a00 0x7f0820003e20 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:54.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.765+0000 7f0841e00640 1 -- 172.21.15.45:0/4282153497 >> 172.21.15.45:0/4282153497 conn(0x7f083c06b930 msgr2=0x7f083c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:54.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.765+0000 7f0841e00640 1 -- 172.21.15.45:0/4282153497 shutdown_connections 2024-04-16T16:20:54.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.765+0000 7f0841e00640 1 -- 172.21.15.45:0/4282153497 wait complete. 2024-04-16T16:20:54.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.766+0000 7f0841e00640 1 Processor -- start 2024-04-16T16:20:54.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.767+0000 7f0841e00640 1 -- start start 2024-04-16T16:20:54.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.767+0000 7f0841e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0820003a00 0x7f083c142ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:54.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.767+0000 7f0841e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f083c154490 0x7f083c15cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:54.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.768+0000 7f0841e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f083c154910 0x7f083c15d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:54.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.768+0000 7f0841e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f083c15ec10 con 0x7f083c154490 2024-04-16T16:20:54.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.768+0000 7f0841e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f083c0cb2b0 con 0x7f083c154910 2024-04-16T16:20:54.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.768+0000 7f0841e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f083c0cac30 con 0x7f0820003a00 2024-04-16T16:20:54.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.768+0000 7f0833e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f083c154490 0x7f083c15cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:54.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.768+0000 7f0833e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f083c154490 0x7f083c15cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34700/0 (socket says 172.21.15.45:34700) 2024-04-16T16:20:54.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.768+0000 7f0833e00640 1 -- 172.21.15.45:0/3221302782 learned_addr learned my addr 172.21.15.45:0/3221302782 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:54.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.768+0000 7f0840a00640 1 --2- 172.21.15.45:0/3221302782 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0820003a00 0x7f083c142ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:54.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.768+0000 7f0841400640 1 --2- 172.21.15.45:0/3221302782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f083c154910 0x7f083c15d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:54.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.768+0000 7f0833e00640 1 -- 172.21.15.45:0/3221302782 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0820003a00 msgr2=0x7f083c142ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:54.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.768+0000 7f0833e00640 1 --2- 172.21.15.45:0/3221302782 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0820003a00 0x7f083c142ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:54.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.768+0000 7f0833e00640 1 -- 172.21.15.45:0/3221302782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f083c154910 msgr2=0x7f083c15d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:54.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.768+0000 7f0833e00640 1 --2- 172.21.15.45:0/3221302782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f083c154910 0x7f083c15d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:54.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.768+0000 7f0833e00640 1 -- 172.21.15.45:0/3221302782 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083c157ac0 con 0x7f083c154490 2024-04-16T16:20:54.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.768+0000 7f0841400640 1 --2- 172.21.15.45:0/3221302782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f083c154910 0x7f083c15d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:54.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.769+0000 7f0833e00640 1 --2- 172.21.15.45:0/3221302782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f083c154490 0x7f083c15cc10 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f0824011730 tx=0x7f0824011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:54.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.769+0000 7f0839600640 1 -- 172.21.15.45:0/3221302782 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082400f960 con 0x7f083c154490 2024-04-16T16:20:54.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.769+0000 7f0839600640 1 -- 172.21.15.45:0/3221302782 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f082400fb00 con 0x7f083c154490 2024-04-16T16:20:54.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.769+0000 7f0839600640 1 -- 172.21.15.45:0/3221302782 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0824015460 con 0x7f083c154490 2024-04-16T16:20:54.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.769+0000 7f0841e00640 1 -- 172.21.15.45:0/3221302782 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f083c1539b0 con 0x7f083c154490 2024-04-16T16:20:54.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.769+0000 7f0841e00640 1 -- 172.21.15.45:0/3221302782 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f083c153f70 con 0x7f083c154490 2024-04-16T16:20:54.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.770+0000 7f0841e00640 1 -- 172.21.15.45:0/3221302782 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f083c075780 con 0x7f083c154490 2024-04-16T16:20:54.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.771+0000 7f0839600640 1 -- 172.21.15.45:0/3221302782 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 53) ==== 91095+0+0 (secure 0 0 0) 0x7f0824015600 con 0x7f083c154490 2024-04-16T16:20:54.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.772+0000 7f0839600640 1 --2- 172.21.15.45:0/3221302782 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f081806c9d0 0x7f081806ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:54.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.772+0000 7f0839600640 1 -- 172.21.15.45:0/3221302782 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6053+0+0 (secure 0 0 0) 0x7f082408c610 con 0x7f083c154490 2024-04-16T16:20:54.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.773+0000 7f0840a00640 1 --2- 172.21.15.45:0/3221302782 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f081806c9d0 0x7f081806ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:54.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.774+0000 7f0840a00640 1 --2- 172.21.15.45:0/3221302782 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f081806c9d0 0x7f081806ee90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f082c033000 tx=0x7f082c009030 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:54.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:54.777+0000 7f0839600640 1 -- 172.21.15.45:0/3221302782 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0824019030 con 0x7f083c154490 2024-04-16T16:20:55.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:55.010+0000 7f0841e00640 1 -- 172.21.15.45:0/3221302782 --> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f083c0b5b90 con 0x7f081806c9d0 2024-04-16T16:20:55.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:55.014+0000 7f0839600640 1 -- 172.21.15.45:0/3221302782 <== mgr.6389 v2:172.21.15.103:6832/549890383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7f083c0b5b90 con 0x7f081806c9d0 2024-04-16T16:20:55.012 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:20:55.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:55.017+0000 7f0841e00640 1 -- 172.21.15.45:0/3221302782 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f081806c9d0 msgr2=0x7f081806ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:55.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:55.017+0000 7f0841e00640 1 --2- 172.21.15.45:0/3221302782 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f081806c9d0 0x7f081806ee90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f082c033000 tx=0x7f082c009030 comp rx=0 tx=0).stop 2024-04-16T16:20:55.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:55.017+0000 7f0841e00640 1 -- 172.21.15.45:0/3221302782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f083c154490 msgr2=0x7f083c15cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:55.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:55.017+0000 7f0841e00640 1 --2- 172.21.15.45:0/3221302782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f083c154490 0x7f083c15cc10 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f0824011730 tx=0x7f0824011bf0 comp rx=0 tx=0).stop 2024-04-16T16:20:55.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:55.017+0000 7f0841e00640 1 -- 172.21.15.45:0/3221302782 shutdown_connections 2024-04-16T16:20:55.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:55.017+0000 7f0841e00640 1 --2- 172.21.15.45:0/3221302782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f083c154910 0x7f083c15d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:55.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:55.017+0000 7f0841e00640 1 --2- 172.21.15.45:0/3221302782 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f081806c9d0 0x7f081806ee90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:55.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:55.017+0000 7f0841e00640 1 --2- 172.21.15.45:0/3221302782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f083c154490 0x7f083c15cc10 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:55.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:55.017+0000 7f0841e00640 1 --2- 172.21.15.45:0/3221302782 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0820003a00 0x7f083c142ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:55.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:55.017+0000 7f0841e00640 1 -- 172.21.15.45:0/3221302782 >> 172.21.15.45:0/3221302782 conn(0x7f083c06b930 msgr2=0x7f083c0cf890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:55.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:55.017+0000 7f0841e00640 1 -- 172.21.15.45:0/3221302782 shutdown_connections 2024-04-16T16:20:55.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:55.017+0000 7f0841e00640 1 -- 172.21.15.45:0/3221302782 wait complete. 2024-04-16T16:20:55.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:20:55.128 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:20:56.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:20:56.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.274+0000 7fc340000640 1 Processor -- start 2024-04-16T16:20:56.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.274+0000 7fc340000640 1 -- start start 2024-04-16T16:20:56.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.275+0000 7fc340000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc3380afe90 0x7fc3380b5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:56.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.276+0000 7fc340000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc338164440 con 0x7fc3380b5640 2024-04-16T16:20:56.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.276+0000 7fc340000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3380b5ca0 con 0x7fc338072890 2024-04-16T16:20:56.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.276+0000 7fc340000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc33806ef10 con 0x7fc3380afe90 2024-04-16T16:20:56.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.276+0000 7fc33cc00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc3380b5640 0x7fc338072180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58620/0 (socket says 172.21.15.45:58620) 2024-04-16T16:20:56.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.276+0000 7fc33cc00640 1 -- 172.21.15.45:0/787853430 learned_addr learned my addr 172.21.15.45:0/787853430 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:56.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.276+0000 7fc33d600640 1 --2- 172.21.15.45:0/787853430 >> v2:172.21.15.103:3300/0 conn(0x7fc3380afe90 0x7fc3380b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:56.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.276+0000 7fc337e00640 1 -- 172.21.15.45:0/787853430 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3359384321 0 0) 0x7fc338164440 con 0x7fc3380b5640 2024-04-16T16:20:56.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.277+0000 7fc337e00640 1 -- 172.21.15.45:0/787853430 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc320003520 con 0x7fc3380b5640 2024-04-16T16:20:56.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.277+0000 7fc337e00640 1 -- 172.21.15.45:0/787853430 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2918641239 0 0) 0x7fc320003520 con 0x7fc3380b5640 2024-04-16T16:20:56.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.277+0000 7fc337e00640 1 -- 172.21.15.45:0/787853430 >> v2:172.21.15.103:3300/0 conn(0x7fc3380afe90 msgr2=0x7fc3380b5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:56.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.277+0000 7fc337e00640 1 --2- 172.21.15.45:0/787853430 >> v2:172.21.15.103:3300/0 conn(0x7fc3380afe90 0x7fc3380b5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:56.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.277+0000 7fc337e00640 1 -- 172.21.15.45:0/787853430 >> v1:172.21.15.103:6789/0 conn(0x7fc338072890 legacy=0x7fc338160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:56.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.277+0000 7fc337e00640 1 -- 172.21.15.45:0/787853430 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc338166a60 con 0x7fc3380b5640 2024-04-16T16:20:56.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.277+0000 7fc337e00640 1 -- 172.21.15.45:0/787853430 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fc324001670 con 0x7fc3380b5640 2024-04-16T16:20:56.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.277+0000 7fc337e00640 1 -- 172.21.15.45:0/787853430 >> v1:172.21.15.45:6789/0 conn(0x7fc3380b5640 legacy=0x7fc338072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:56.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.277+0000 7fc337e00640 1 --2- 172.21.15.45:0/787853430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc320003a00 0x7fc320003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:56.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.277+0000 7fc337e00640 1 --2- 172.21.15.45:0/787853430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc320004800 0x7fc320004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:56.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.277+0000 7fc337e00640 1 --2- 172.21.15.45:0/787853430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc320005210 0x7fc32000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:56.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.277+0000 7fc337e00640 1 -- 172.21.15.45:0/787853430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc33806ef10 con 0x7fc320004800 2024-04-16T16:20:56.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.277+0000 7fc337e00640 1 -- 172.21.15.45:0/787853430 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc32000c160 con 0x7fc320003a00 2024-04-16T16:20:56.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.277+0000 7fc337e00640 1 -- 172.21.15.45:0/787853430 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc32000c360 con 0x7fc320005210 2024-04-16T16:20:56.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.277+0000 7fc33cc00640 1 --2- 172.21.15.45:0/787853430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc320004800 0x7fc320004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:56.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.278+0000 7fc33d600640 1 --2- 172.21.15.45:0/787853430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc320003a00 0x7fc320003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:56.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.278+0000 7fc33cc00640 1 -- 172.21.15.45:0/787853430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc320005210 msgr2=0x7fc32000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:56.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.278+0000 7fc33e000640 1 --2- 172.21.15.45:0/787853430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc320005210 0x7fc32000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:56.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.278+0000 7fc33cc00640 1 --2- 172.21.15.45:0/787853430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc320005210 0x7fc32000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:56.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.278+0000 7fc33cc00640 1 -- 172.21.15.45:0/787853430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc320003a00 msgr2=0x7fc320003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:56.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.278+0000 7fc33cc00640 1 --2- 172.21.15.45:0/787853430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc320003a00 0x7fc320003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:56.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.278+0000 7fc33cc00640 1 -- 172.21.15.45:0/787853430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc320003520 con 0x7fc320004800 2024-04-16T16:20:56.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.278+0000 7fc33d600640 1 --2- 172.21.15.45:0/787853430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc320003a00 0x7fc320003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:20:56.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.278+0000 7fc33e000640 1 -- 172.21.15.45:0/787853430 reap_dead start 2024-04-16T16:20:56.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.279+0000 7fc33cc00640 1 --2- 172.21.15.45:0/787853430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc320004800 0x7fc320004c80 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fc324007fd0 tx=0x7fc320005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166f6edd81f1bd31 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:56.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.279+0000 7fc337e00640 1 -- 172.21.15.45:0/787853430 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324007c20 con 0x7fc320004800 2024-04-16T16:20:56.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.279+0000 7fc337e00640 1 -- 172.21.15.45:0/787853430 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc324030070 con 0x7fc320004800 2024-04-16T16:20:56.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.279+0000 7fc337e00640 1 -- 172.21.15.45:0/787853430 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324005d70 con 0x7fc320004800 2024-04-16T16:20:56.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.280+0000 7fc340000640 1 -- 172.21.15.45:0/787853430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc320004800 msgr2=0x7fc320004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:56.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.280+0000 7fc340000640 1 --2- 172.21.15.45:0/787853430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc320004800 0x7fc320004c80 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fc324007fd0 tx=0x7fc320005830 comp rx=0 tx=0).stop 2024-04-16T16:20:56.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.280+0000 7fc340000640 1 -- 172.21.15.45:0/787853430 shutdown_connections 2024-04-16T16:20:56.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.280+0000 7fc340000640 1 --2- 172.21.15.45:0/787853430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc320004800 0x7fc320004c80 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:56.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.280+0000 7fc340000640 1 -- 172.21.15.45:0/787853430 >> 172.21.15.45:0/787853430 conn(0x7fc33806b930 msgr2=0x7fc33806bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:56.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.280+0000 7fc340000640 1 -- 172.21.15.45:0/787853430 shutdown_connections 2024-04-16T16:20:56.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.280+0000 7fc340000640 1 -- 172.21.15.45:0/787853430 wait complete. 2024-04-16T16:20:56.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.282+0000 7fc340000640 1 Processor -- start 2024-04-16T16:20:56.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.282+0000 7fc340000640 1 -- start start 2024-04-16T16:20:56.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.283+0000 7fc340000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc320004800 0x7fc338147790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:56.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.283+0000 7fc340000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc338165200 0x7fc338165660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:56.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.283+0000 7fc340000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc33814a230 0x7fc33814a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:56.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.283+0000 7fc340000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc338167300 con 0x7fc33814a230 2024-04-16T16:20:56.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.283+0000 7fc340000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc338167020 con 0x7fc320004800 2024-04-16T16:20:56.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.283+0000 7fc340000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc3380b0270 con 0x7fc338165200 2024-04-16T16:20:56.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.283+0000 7fc33e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc33814a230 0x7fc33814a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:56.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.283+0000 7fc33e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc33814a230 0x7fc33814a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34712/0 (socket says 172.21.15.45:34712) 2024-04-16T16:20:56.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.283+0000 7fc33e000640 1 -- 172.21.15.45:0/3137103723 learned_addr learned my addr 172.21.15.45:0/3137103723 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:56.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.284+0000 7fc33cc00640 1 --2- 172.21.15.45:0/3137103723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc338165200 0x7fc338165660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:56.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.284+0000 7fc33d600640 1 --2- 172.21.15.45:0/3137103723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc320004800 0x7fc338147790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:56.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.284+0000 7fc33e000640 1 -- 172.21.15.45:0/3137103723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc338165200 msgr2=0x7fc338165660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:56.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.284+0000 7fc33e000640 1 --2- 172.21.15.45:0/3137103723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc338165200 0x7fc338165660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:56.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.284+0000 7fc33e000640 1 -- 172.21.15.45:0/3137103723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc320004800 msgr2=0x7fc338147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:56.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.284+0000 7fc33e000640 1 --2- 172.21.15.45:0/3137103723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc320004800 0x7fc338147790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:56.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.284+0000 7fc33e000640 1 -- 172.21.15.45:0/3137103723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc338158af0 con 0x7fc33814a230 2024-04-16T16:20:56.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.284+0000 7fc33d600640 1 --2- 172.21.15.45:0/3137103723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc320004800 0x7fc338147790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:56.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.284+0000 7fc33e000640 1 --2- 172.21.15.45:0/3137103723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc33814a230 0x7fc33814a6e0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fc338070350 tx=0x7fc33000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:56.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.284+0000 7fc335600640 1 -- 172.21.15.45:0/3137103723 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330018070 con 0x7fc33814a230 2024-04-16T16:20:56.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.284+0000 7fc335600640 1 -- 172.21.15.45:0/3137103723 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc330003b70 con 0x7fc33814a230 2024-04-16T16:20:56.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.284+0000 7fc335600640 1 -- 172.21.15.45:0/3137103723 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330005070 con 0x7fc33814a230 2024-04-16T16:20:56.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.284+0000 7fc340000640 1 -- 172.21.15.45:0/3137103723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc338158dd0 con 0x7fc33814a230 2024-04-16T16:20:56.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.285+0000 7fc340000640 1 -- 172.21.15.45:0/3137103723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc338159390 con 0x7fc33814a230 2024-04-16T16:20:56.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.286+0000 7fc340000640 1 -- 172.21.15.45:0/3137103723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc338159590 con 0x7fc33814a230 2024-04-16T16:20:56.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.287+0000 7fc335600640 1 -- 172.21.15.45:0/3137103723 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 54) ==== 91151+0+0 (secure 0 0 0) 0x7fc33001b070 con 0x7fc33814a230 2024-04-16T16:20:56.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.288+0000 7fc335600640 1 --2- 172.21.15.45:0/3137103723 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fc31806ca90 0x7fc31806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:56.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.288+0000 7fc335600640 1 -- 172.21.15.45:0/3137103723 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6053+0+0 (secure 0 0 0) 0x7fc33008b790 con 0x7fc33814a230 2024-04-16T16:20:56.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.289+0000 7fc33d600640 1 --2- 172.21.15.45:0/3137103723 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fc31806ca90 0x7fc31806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:56.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.290+0000 7fc33d600640 1 --2- 172.21.15.45:0/3137103723 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fc31806ca90 0x7fc31806ef50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc338069180 tx=0x7fc32c008040 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:56.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.294+0000 7fc335600640 1 -- 172.21.15.45:0/3137103723 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc338159590 con 0x7fc33814a230 2024-04-16T16:20:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.523+0000 7fc340000640 1 -- 172.21.15.45:0/3137103723 --> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7fc338070020 con 0x7fc31806ca90 2024-04-16T16:20:56.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.526+0000 7fc335600640 1 -- 172.21.15.45:0/3137103723 <== mgr.6389 v2:172.21.15.103:6832/549890383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7fc338070020 con 0x7fc31806ca90 2024-04-16T16:20:56.525 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:20:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.529+0000 7fc340000640 1 -- 172.21.15.45:0/3137103723 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fc31806ca90 msgr2=0x7fc31806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.529+0000 7fc340000640 1 --2- 172.21.15.45:0/3137103723 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fc31806ca90 0x7fc31806ef50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc338069180 tx=0x7fc32c008040 comp rx=0 tx=0).stop 2024-04-16T16:20:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.529+0000 7fc340000640 1 -- 172.21.15.45:0/3137103723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc33814a230 msgr2=0x7fc33814a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.529+0000 7fc340000640 1 --2- 172.21.15.45:0/3137103723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc33814a230 0x7fc33814a6e0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fc338070350 tx=0x7fc33000dc00 comp rx=0 tx=0).stop 2024-04-16T16:20:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.529+0000 7fc340000640 1 -- 172.21.15.45:0/3137103723 shutdown_connections 2024-04-16T16:20:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.529+0000 7fc340000640 1 --2- 172.21.15.45:0/3137103723 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fc31806ca90 0x7fc31806ef50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.529+0000 7fc340000640 1 --2- 172.21.15.45:0/3137103723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc33814a230 0x7fc33814a6e0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.529+0000 7fc340000640 1 --2- 172.21.15.45:0/3137103723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc338165200 0x7fc338165660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.529+0000 7fc340000640 1 --2- 172.21.15.45:0/3137103723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc320004800 0x7fc338147790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.529+0000 7fc340000640 1 -- 172.21.15.45:0/3137103723 >> 172.21.15.45:0/3137103723 conn(0x7fc33806b930 msgr2=0x7fc338161f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.529+0000 7fc340000640 1 -- 172.21.15.45:0/3137103723 shutdown_connections 2024-04-16T16:20:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:56.529+0000 7fc340000640 1 -- 172.21.15.45:0/3137103723 wait complete. 2024-04-16T16:20:57.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:20:57.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.781+0000 7f62b1000640 1 Processor -- start 2024-04-16T16:20:57.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.781+0000 7f62b1000640 1 -- start start 2024-04-16T16:20:57.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.783+0000 7f62b1000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62ac0cdfe0 0x7f62ac0b4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:57.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.783+0000 7f62b1000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62ac06fab0 con 0x7f62ac0afe20 2024-04-16T16:20:57.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.783+0000 7f62b1000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62ac15bf20 con 0x7f62ac15ca00 2024-04-16T16:20:57.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.783+0000 7f62b1000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f62ac15c200 con 0x7f62ac0cdfe0 2024-04-16T16:20:57.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.783+0000 7f62a9e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f62ac0afe20 0x7f62ac06f010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58636/0 (socket says 172.21.15.45:58636) 2024-04-16T16:20:57.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.783+0000 7f62a9e00640 1 -- 172.21.15.45:0/2870055288 learned_addr learned my addr 172.21.15.45:0/2870055288 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:57.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62aa800640 1 --2- 172.21.15.45:0/2870055288 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62ac0cdfe0 0x7f62ac0b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:57.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62a9400640 1 -- 172.21.15.45:0/2870055288 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2023951293 0 0) 0x7f62ac06fab0 con 0x7f62ac0afe20 2024-04-16T16:20:57.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62a9400640 1 -- 172.21.15.45:0/2870055288 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6290003520 con 0x7f62ac0afe20 2024-04-16T16:20:57.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62a9400640 1 -- 172.21.15.45:0/2870055288 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 553270479 0 0) 0x7f6290003520 con 0x7f62ac0afe20 2024-04-16T16:20:57.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62a9400640 1 -- 172.21.15.45:0/2870055288 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62ac0cdfe0 msgr2=0x7f62ac0b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:57.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62a9400640 1 --2- 172.21.15.45:0/2870055288 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62ac0cdfe0 0x7f62ac0b4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:57.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62a9400640 1 -- 172.21.15.45:0/2870055288 >> v1:172.21.15.103:6789/0 conn(0x7f62ac15ca00 legacy=0x7f62ac0cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:57.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62a9400640 1 -- 172.21.15.45:0/2870055288 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ac15c4e0 con 0x7f62ac0afe20 2024-04-16T16:20:57.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62a9400640 1 -- 172.21.15.45:0/2870055288 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f629c001670 con 0x7f62ac0afe20 2024-04-16T16:20:57.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62a9400640 1 -- 172.21.15.45:0/2870055288 >> v1:172.21.15.45:6789/0 conn(0x7f62ac0afe20 legacy=0x7f62ac06f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:57.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62a9400640 1 --2- 172.21.15.45:0/2870055288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62900039e0 0x7f6290003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:57.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62a9400640 1 --2- 172.21.15.45:0/2870055288 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62ac0cdfe0 0x7f6290004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:57.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62a9400640 1 --2- 172.21.15.45:0/2870055288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6290004d40 0x7f62900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:57.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62a9400640 1 -- 172.21.15.45:0/2870055288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac15c200 con 0x7f6290004d40 2024-04-16T16:20:57.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62a9400640 1 -- 172.21.15.45:0/2870055288 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f629000bbd0 con 0x7f62900039e0 2024-04-16T16:20:57.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.784+0000 7f62a9400640 1 -- 172.21.15.45:0/2870055288 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f629000bdd0 con 0x7f62ac0cdfe0 2024-04-16T16:20:57.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.785+0000 7f62aa800640 1 --2- 172.21.15.45:0/2870055288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6290004d40 0x7f62900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:57.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.785+0000 7f62a1400640 1 --2- 172.21.15.45:0/2870055288 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62ac0cdfe0 0x7f6290004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:57.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.785+0000 7f62a9e00640 1 --2- 172.21.15.45:0/2870055288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62900039e0 0x7f6290003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:57.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.785+0000 7f62aa800640 1 -- 172.21.15.45:0/2870055288 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62ac0cdfe0 msgr2=0x7f6290004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:57.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.785+0000 7f62aa800640 1 --2- 172.21.15.45:0/2870055288 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62ac0cdfe0 0x7f6290004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:57.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.785+0000 7f62aa800640 1 -- 172.21.15.45:0/2870055288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62900039e0 msgr2=0x7f6290003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:57.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.785+0000 7f62aa800640 1 --2- 172.21.15.45:0/2870055288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62900039e0 0x7f6290003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:57.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.785+0000 7f62aa800640 1 -- 172.21.15.45:0/2870055288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6290003520 con 0x7f6290004d40 2024-04-16T16:20:57.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.786+0000 7f62aa800640 1 --2- 172.21.15.45:0/2870055288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6290004d40 0x7f62900051c0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f62a4007360 tx=0x7f6290005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632db7fa905e0be3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:57.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.787+0000 7f62a9400640 1 -- 172.21.15.45:0/2870055288 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a4039070 con 0x7f6290004d40 2024-04-16T16:20:57.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.787+0000 7f62a9400640 1 -- 172.21.15.45:0/2870055288 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62a402ccd0 con 0x7f6290004d40 2024-04-16T16:20:57.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.787+0000 7f62a9400640 1 -- 172.21.15.45:0/2870055288 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a4009be0 con 0x7f6290004d40 2024-04-16T16:20:57.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.787+0000 7f62b1000640 1 -- 172.21.15.45:0/2870055288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6290004d40 msgr2=0x7f62900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:57.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.787+0000 7f62b1000640 1 --2- 172.21.15.45:0/2870055288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6290004d40 0x7f62900051c0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f62a4007360 tx=0x7f6290005830 comp rx=0 tx=0).stop 2024-04-16T16:20:57.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.787+0000 7f62aa800640 1 -- 172.21.15.45:0/2870055288 reap_dead start 2024-04-16T16:20:57.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.787+0000 7f62b1000640 1 -- 172.21.15.45:0/2870055288 shutdown_connections 2024-04-16T16:20:57.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.787+0000 7f62b1000640 1 -- 172.21.15.45:0/2870055288 >> 172.21.15.45:0/2870055288 conn(0x7f62ac06b930 msgr2=0x7f62ac06bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:57.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.787+0000 7f62b1000640 1 -- 172.21.15.45:0/2870055288 shutdown_connections 2024-04-16T16:20:57.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.787+0000 7f62b1000640 1 -- 172.21.15.45:0/2870055288 wait complete. 2024-04-16T16:20:57.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.789+0000 7f62b1000640 1 Processor -- start 2024-04-16T16:20:57.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.789+0000 7f62b1000640 1 -- start start 2024-04-16T16:20:57.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.790+0000 7f62b1000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290004d40 0x7f62ac0acac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.790+0000 7f62b1000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62ac149fa0 0x7f62ac152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.790+0000 7f62b1000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ac153100 0x7f62ac151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.790+0000 7f62b1000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac0b0380 con 0x7f62ac153100 2024-04-16T16:20:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.790+0000 7f62b1000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac0b0200 con 0x7f6290004d40 2024-04-16T16:20:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.790+0000 7f62b1000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f62ac0b5850 con 0x7f62ac149fa0 2024-04-16T16:20:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.791+0000 7f62aa800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ac153100 0x7f62ac151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.791+0000 7f62aa800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ac153100 0x7f62ac151b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34744/0 (socket says 172.21.15.45:34744) 2024-04-16T16:20:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.791+0000 7f62aa800640 1 -- 172.21.15.45:0/730107763 learned_addr learned my addr 172.21.15.45:0/730107763 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.791+0000 7f62a1400640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62ac149fa0 0x7f62ac152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.791+0000 7f62a9e00640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290004d40 0x7f62ac0acac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.791+0000 7f62aa800640 1 -- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62ac149fa0 msgr2=0x7f62ac152bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.791+0000 7f62aa800640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62ac149fa0 0x7f62ac152bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.791+0000 7f62aa800640 1 -- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290004d40 msgr2=0x7f62ac0acac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.791+0000 7f62aa800640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290004d40 0x7f62ac0acac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:57.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.791+0000 7f62aa800640 1 -- 172.21.15.45:0/730107763 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ac0a9720 con 0x7f62ac153100 2024-04-16T16:20:57.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.791+0000 7f62a1400640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62ac149fa0 0x7f62ac152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:57.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.791+0000 7f62a9e00640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290004d40 0x7f62ac0acac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:57.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.791+0000 7f62aa800640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ac153100 0x7f62ac151b70 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f62a4007360 tx=0x7f62a4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:57.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.792+0000 7f62a2a00640 1 -- 172.21.15.45:0/730107763 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a4039070 con 0x7f62ac153100 2024-04-16T16:20:57.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.792+0000 7f62a2a00640 1 -- 172.21.15.45:0/730107763 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62a4044070 con 0x7f62ac153100 2024-04-16T16:20:57.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.792+0000 7f62a2a00640 1 -- 172.21.15.45:0/730107763 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a4009be0 con 0x7f62ac153100 2024-04-16T16:20:57.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.792+0000 7f62b1000640 1 -- 172.21.15.45:0/730107763 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ac0a9a00 con 0x7f62ac153100 2024-04-16T16:20:57.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.792+0000 7f62b1000640 1 -- 172.21.15.45:0/730107763 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ac0a9fc0 con 0x7f62ac153100 2024-04-16T16:20:57.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.793+0000 7f62b1000640 1 -- 172.21.15.45:0/730107763 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62ac075780 con 0x7f62ac153100 2024-04-16T16:20:57.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.794+0000 7f62a2a00640 1 -- 172.21.15.45:0/730107763 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 54) ==== 91151+0+0 (secure 0 0 0) 0x7f62a403e830 con 0x7f62ac153100 2024-04-16T16:20:57.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.795+0000 7f62a2a00640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f628806ca90 0x7f628806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:57.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.795+0000 7f62a2a00640 1 -- 172.21.15.45:0/730107763 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6053+0+0 (secure 0 0 0) 0x7f62a40adfd0 con 0x7f62ac153100 2024-04-16T16:20:57.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.796+0000 7f62a9e00640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f628806ca90 0x7f628806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:57.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.797+0000 7f62a9e00640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f628806ca90 0x7f628806ef50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f629c001eb0 tx=0x7f629c000ed0 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:57.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:57.799+0000 7f62a2a00640 1 -- 172.21.15.45:0/730107763 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f62a407c060 con 0x7f62ac153100 2024-04-16T16:20:58.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.030+0000 7f62b1000640 1 -- 172.21.15.45:0/730107763 --> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f62ac0a94d0 con 0x7f628806ca90 2024-04-16T16:20:58.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.034+0000 7f62a2a00640 1 -- 172.21.15.45:0/730107763 <== mgr.6389 v2:172.21.15.103:6832/549890383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+2956 (secure 0 0 0) 0x7f62ac0a94d0 con 0x7f628806ca90 2024-04-16T16:20:58.033 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.6380": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "192.168.0.1"}, "client.6422": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.7111": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.8805": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "192.168.0.1"}}}, "global_metrics": {"cephfs": {"client.6380": [[24, 0], [0, 0], [0, 0], [0, 8932351], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 2233089], [19354710870695, 4]], "client.6422": [[2, 0], [0, 0], [0, 0], [0, 813612], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 813612], [0, 1]], "client.7111": [[2, 0], [0, 0], [0, 0], [0, 1127228], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1127228], [0, 1]], "client.8805": [[5, 19], [0, 0], [0, 0], [0, 25258439], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1052440], [46275762452433, 24]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.6380": [], "client.6422": [], "client.7111": [], "client.8805": []}}} 2024-04-16T16:20:58.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.036+0000 7f62b1000640 1 -- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f628806ca90 msgr2=0x7f628806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:58.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.036+0000 7f62b1000640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f628806ca90 0x7f628806ef50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f629c001eb0 tx=0x7f629c000ed0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.036+0000 7f62b1000640 1 -- 172.21.15.45:0/730107763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ac153100 msgr2=0x7f62ac151b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:58.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.036+0000 7f62b1000640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ac153100 0x7f62ac151b70 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f62a4007360 tx=0x7f62a4004ab0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.036+0000 7f62b1000640 1 -- 172.21.15.45:0/730107763 shutdown_connections 2024-04-16T16:20:58.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.036+0000 7f62b1000640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f628806ca90 0x7f628806ef50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.036+0000 7f62b1000640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ac153100 0x7f62ac151b70 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.036+0000 7f62b1000640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f62ac149fa0 0x7f62ac152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.036+0000 7f62b1000640 1 --2- 172.21.15.45:0/730107763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290004d40 0x7f62ac0acac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.036+0000 7f62b1000640 1 -- 172.21.15.45:0/730107763 >> 172.21.15.45:0/730107763 conn(0x7f62ac06b930 msgr2=0x7f62ac06bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:58.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.036+0000 7f62b1000640 1 -- 172.21.15.45:0/730107763 shutdown_connections 2024-04-16T16:20:58.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.036+0000 7f62b1000640 1 -- 172.21.15.45:0/730107763 wait complete. 2024-04-16T16:20:58.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:20:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.285+0000 7fbe47e00640 1 Processor -- start 2024-04-16T16:20:58.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.285+0000 7fbe47e00640 1 -- start start 2024-04-16T16:20:58.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.287+0000 7fbe47e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbe4815ccb0 0x7fbe4806df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.287+0000 7fbe47e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbe4806e480 0x7fbe481608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.287+0000 7fbe47e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe4815d360 con 0x7fbe48146220 2024-04-16T16:20:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.287+0000 7fbe47e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fbe4815d130 con 0x7fbe4806e480 2024-04-16T16:20:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.287+0000 7fbe47e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbe48160410 con 0x7fbe4815ccb0 2024-04-16T16:20:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.287+0000 7fbe47400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbe4806e480 0x7fbe481608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.287+0000 7fbe46a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbe48146220 0x7fbe4806f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58648/0 (socket says 172.21.15.45:58648) 2024-04-16T16:20:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.287+0000 7fbe46a00640 1 -- 172.21.15.45:0/3946278913 learned_addr learned my addr 172.21.15.45:0/3946278913 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.288+0000 7fbe46000640 1 --2- 172.21.15.45:0/3946278913 >> v2:172.21.15.103:3300/0 conn(0x7fbe4815ccb0 0x7fbe4806df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.288+0000 7fbe47400640 1 -- 172.21.15.45:0/3946278913 >> v2:172.21.15.103:3300/0 conn(0x7fbe4815ccb0 msgr2=0x7fbe4806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:20:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.288+0000 7fbe47400640 1 --2- 172.21.15.45:0/3946278913 >> v2:172.21.15.103:3300/0 conn(0x7fbe4815ccb0 0x7fbe4806df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.288+0000 7fbe47400640 1 -- 172.21.15.45:0/3946278913 >> v1:172.21.15.45:6789/0 conn(0x7fbe48146220 legacy=0x7fbe4806f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.288+0000 7fbe47400640 1 -- 172.21.15.45:0/3946278913 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe4815d640 con 0x7fbe4806e480 2024-04-16T16:20:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.289+0000 7fbe47400640 1 --2- 172.21.15.45:0/3946278913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe4806e480 0x7fbe481608a0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fbe48160590 tx=0x7fbe4002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def94d7686b7be9b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:58.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.289+0000 7fbe45600640 1 -- 172.21.15.45:0/3946278913 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe40039070 con 0x7fbe4806e480 2024-04-16T16:20:58.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.289+0000 7fbe45600640 1 -- 172.21.15.45:0/3946278913 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe4002cc60 con 0x7fbe4806e480 2024-04-16T16:20:58.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.289+0000 7fbe45600640 1 -- 172.21.15.45:0/3946278913 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe40034660 con 0x7fbe4806e480 2024-04-16T16:20:58.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.290+0000 7fbe47e00640 1 -- 172.21.15.45:0/3946278913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe4806e480 msgr2=0x7fbe481608a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:58.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.290+0000 7fbe47e00640 1 --2- 172.21.15.45:0/3946278913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe4806e480 0x7fbe481608a0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fbe48160590 tx=0x7fbe4002beb0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.290+0000 7fbe47e00640 1 -- 172.21.15.45:0/3946278913 shutdown_connections 2024-04-16T16:20:58.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.290+0000 7fbe47e00640 1 --2- 172.21.15.45:0/3946278913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe4806e480 0x7fbe481608a0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.290+0000 7fbe47e00640 1 --2- 172.21.15.45:0/3946278913 >> v2:172.21.15.103:3300/0 conn(0x7fbe4815ccb0 0x7fbe4806df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.290+0000 7fbe47e00640 1 -- 172.21.15.45:0/3946278913 >> 172.21.15.45:0/3946278913 conn(0x7fbe4806b930 msgr2=0x7fbe4806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:58.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.290+0000 7fbe47e00640 1 -- 172.21.15.45:0/3946278913 shutdown_connections 2024-04-16T16:20:58.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.290+0000 7fbe47e00640 1 -- 172.21.15.45:0/3946278913 wait complete. 2024-04-16T16:20:58.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.292+0000 7fbe47e00640 1 Processor -- start 2024-04-16T16:20:58.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.293+0000 7fbe47e00640 1 -- start start 2024-04-16T16:20:58.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.294+0000 7fbe47e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe4806e480 0x7fbe480b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:58.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.294+0000 7fbe47e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe48146220 0x7fbe481e6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:58.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.294+0000 7fbe47e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe4815ccb0 0x7fbe481f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:58.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.294+0000 7fbe47e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbe48160de0 con 0x7fbe4815ccb0 2024-04-16T16:20:58.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.294+0000 7fbe47e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4815dc00 con 0x7fbe48146220 2024-04-16T16:20:58.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.294+0000 7fbe47e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbe48160f60 con 0x7fbe4806e480 2024-04-16T16:20:58.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.294+0000 7fbe47400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe4815ccb0 0x7fbe481f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:58.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.294+0000 7fbe47400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe4815ccb0 0x7fbe481f84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34764/0 (socket says 172.21.15.45:34764) 2024-04-16T16:20:58.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.294+0000 7fbe47400640 1 -- 172.21.15.45:0/3453351647 learned_addr learned my addr 172.21.15.45:0/3453351647 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:58.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.294+0000 7fbe46a00640 1 --2- 172.21.15.45:0/3453351647 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe4806e480 0x7fbe480b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:58.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.294+0000 7fbe46000640 1 --2- 172.21.15.45:0/3453351647 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe48146220 0x7fbe481e6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:58.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.294+0000 7fbe47400640 1 -- 172.21.15.45:0/3453351647 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe4806e480 msgr2=0x7fbe480b53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:58.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.295+0000 7fbe47400640 1 --2- 172.21.15.45:0/3453351647 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe4806e480 0x7fbe480b53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.295+0000 7fbe47400640 1 -- 172.21.15.45:0/3453351647 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe48146220 msgr2=0x7fbe481e6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:58.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.295+0000 7fbe47400640 1 --2- 172.21.15.45:0/3453351647 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe48146220 0x7fbe481e6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.295+0000 7fbe47400640 1 -- 172.21.15.45:0/3453351647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe480b2400 con 0x7fbe4815ccb0 2024-04-16T16:20:58.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.295+0000 7fbe46000640 1 --2- 172.21.15.45:0/3453351647 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe48146220 0x7fbe481e6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:58.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.295+0000 7fbe47400640 1 --2- 172.21.15.45:0/3453351647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe4815ccb0 0x7fbe481f84c0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009e50 tx=0x7fbe40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:58.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.295+0000 7fbe3ea00640 1 -- 172.21.15.45:0/3453351647 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe40039050 con 0x7fbe4815ccb0 2024-04-16T16:20:58.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.295+0000 7fbe3ea00640 1 -- 172.21.15.45:0/3453351647 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe40045070 con 0x7fbe4815ccb0 2024-04-16T16:20:58.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.295+0000 7fbe3ea00640 1 -- 172.21.15.45:0/3453351647 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4003e830 con 0x7fbe4815ccb0 2024-04-16T16:20:58.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.295+0000 7fbe47e00640 1 -- 172.21.15.45:0/3453351647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe480b1e80 con 0x7fbe4815ccb0 2024-04-16T16:20:58.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.295+0000 7fbe47e00640 1 -- 172.21.15.45:0/3453351647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe481e6570 con 0x7fbe4815ccb0 2024-04-16T16:20:58.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.297+0000 7fbe47e00640 1 -- 172.21.15.45:0/3453351647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe48075780 con 0x7fbe4815ccb0 2024-04-16T16:20:58.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.297+0000 7fbe3ea00640 1 -- 172.21.15.45:0/3453351647 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 54) ==== 91151+0+0 (secure 0 0 0) 0x7fbe4003e9d0 con 0x7fbe4815ccb0 2024-04-16T16:20:58.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.298+0000 7fbe3ea00640 1 --2- 172.21.15.45:0/3453351647 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fbe2406ca60 0x7fbe2406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:58.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.299+0000 7fbe3ea00640 1 -- 172.21.15.45:0/3453351647 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6053+0+0 (secure 0 0 0) 0x7fbe400aec90 con 0x7fbe4815ccb0 2024-04-16T16:20:58.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.299+0000 7fbe46a00640 1 --2- 172.21.15.45:0/3453351647 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fbe2406ca60 0x7fbe2406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:58.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.300+0000 7fbe46a00640 1 --2- 172.21.15.45:0/3453351647 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fbe2406ca60 0x7fbe2406ef20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbe38000f60 tx=0x7fbe38009040 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:58.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.303+0000 7fbe3ea00640 1 -- 172.21.15.45:0/3453351647 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe4007cf60 con 0x7fbe4815ccb0 2024-04-16T16:20:58.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.556+0000 7fbe47e00640 1 -- 172.21.15.45:0/3453351647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe48141c90 con 0x7fbe4815ccb0 2024-04-16T16:20:58.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.557+0000 7fbe3ea00640 1 -- 172.21.15.45:0/3453351647 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5186 (secure 0 0 0) 0x7fbe4007c860 con 0x7fbe4815ccb0 2024-04-16T16:20:58.556 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:20:58.556 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/929218719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":929218719},{"type":"v1","addr":"172.21.15.103:6835","nonce":929218719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6323,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1619719405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1619719405},{"type":"v1","addr":"172.21.15.103:6837","nonce":1619719405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":8640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/3613288118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3613288118},{"type":"v1","addr":"172.21.15.45:6834","nonce":3613288118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:20:33.816422+0000","modified":"2024-04-16T16:20:36.891870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8643":{"gid":8643,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2060312705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2060312705},{"type":"v1","addr":"172.21.15.45:6835","nonce":2060312705}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8643,"qdb_cluster":[8643]},"id":9}]} 2024-04-16T16:20:58.557 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 91 2024-04-16T16:20:58.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.559+0000 7fbe47e00640 1 -- 172.21.15.45:0/3453351647 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fbe2406ca60 msgr2=0x7fbe2406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:58.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.559+0000 7fbe47e00640 1 --2- 172.21.15.45:0/3453351647 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fbe2406ca60 0x7fbe2406ef20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbe38000f60 tx=0x7fbe38009040 comp rx=0 tx=0).stop 2024-04-16T16:20:58.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.559+0000 7fbe47e00640 1 -- 172.21.15.45:0/3453351647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe4815ccb0 msgr2=0x7fbe481f84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:58.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.559+0000 7fbe47e00640 1 --2- 172.21.15.45:0/3453351647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe4815ccb0 0x7fbe481f84c0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009e50 tx=0x7fbe40004a40 comp rx=0 tx=0).stop 2024-04-16T16:20:58.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.560+0000 7fbe47e00640 1 -- 172.21.15.45:0/3453351647 shutdown_connections 2024-04-16T16:20:58.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.560+0000 7fbe47e00640 1 --2- 172.21.15.45:0/3453351647 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fbe2406ca60 0x7fbe2406ef20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.560+0000 7fbe47e00640 1 --2- 172.21.15.45:0/3453351647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe4815ccb0 0x7fbe481f84c0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.560+0000 7fbe47e00640 1 --2- 172.21.15.45:0/3453351647 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe48146220 0x7fbe481e6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.560+0000 7fbe47e00640 1 --2- 172.21.15.45:0/3453351647 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe4806e480 0x7fbe480b53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.560+0000 7fbe47e00640 1 -- 172.21.15.45:0/3453351647 >> 172.21.15.45:0/3453351647 conn(0x7fbe4806b930 msgr2=0x7fbe480c5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:58.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.560+0000 7fbe47e00640 1 -- 172.21.15.45:0/3453351647 shutdown_connections 2024-04-16T16:20:58.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.560+0000 7fbe47e00640 1 -- 172.21.15.45:0/3453351647 wait complete. 2024-04-16T16:20:58.578 DEBUG:tasks.cephfs.test_mds_metrics:metrics={'version': 2, 'global_counters': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'counters': [], 'client_metadata': {'cephfs': {'client.6380': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '192.168.0.1'}, 'client.6422': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'IP': '172.21.15.103'}, 'client.7111': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'IP': '172.21.15.103'}, 'client.8805': {'hostname': 'smithi045', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '192.168.0.1'}}}, 'global_metrics': {'cephfs': {'client.6380': [[24, 0], [0, 0], [0, 0], [0, 8932351], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 2233089], [19354710870695, 4]], 'client.6422': [[2, 0], [0, 0], [0, 0], [0, 813612], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 813612], [0, 1]], 'client.7111': [[2, 0], [0, 0], [0, 0], [0, 1127228], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1127228], [0, 1]], 'client.8805': [[5, 19], [0, 0], [0, 0], [0, 25258439], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1052440], [46275762452433, 24]]}}, 'metrics': {'delayed_ranks': [], 'mds.0': {'client.6380': [], 'client.6422': [], 'client.7111': [], 'client.8805': []}}} 2024-04-16T16:20:58.579 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable stats 2024-04-16T16:20:58.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.803+0000 7fb003400640 1 Processor -- start 2024-04-16T16:20:58.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.804+0000 7fb003400640 1 -- start start 2024-04-16T16:20:58.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.805+0000 7fb003400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faffc15ec80 0x7faffc0d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:58.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.805+0000 7fb003400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faffc0b7ca0 con 0x7faffc0b6c00 2024-04-16T16:20:58.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.805+0000 7fb003400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faffc0d3450 con 0x7faffc15f060 2024-04-16T16:20:58.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.805+0000 7fb003400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7faffc0d3220 con 0x7faffc15ec80 2024-04-16T16:20:58.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.806+0000 7fb001400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7faffc0b6c00 0x7faffc168990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58662/0 (socket says 172.21.15.45:58662) 2024-04-16T16:20:58.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.806+0000 7fb000a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faffc15ec80 0x7faffc0d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:58.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.806+0000 7fb001400640 1 -- 172.21.15.45:0/2001273020 learned_addr learned my addr 172.21.15.45:0/2001273020 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:20:58.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.806+0000 7faffb400640 1 -- 172.21.15.45:0/2001273020 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1980885647 0 0) 0x7faffc0b7ca0 con 0x7faffc0b6c00 2024-04-16T16:20:58.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.806+0000 7faffb400640 1 -- 172.21.15.45:0/2001273020 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafe0003520 con 0x7faffc0b6c00 2024-04-16T16:20:58.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.806+0000 7fb000a00640 1 -- 172.21.15.45:0/2001273020 >> v1:172.21.15.103:6789/0 conn(0x7faffc15f060 legacy=0x7faffc0b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:58.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.806+0000 7fb000a00640 1 -- 172.21.15.45:0/2001273020 >> v1:172.21.15.45:6789/0 conn(0x7faffc0b6c00 legacy=0x7faffc168990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:58.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.806+0000 7fb000a00640 1 -- 172.21.15.45:0/2001273020 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faffc0d3730 con 0x7faffc15ec80 2024-04-16T16:20:58.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.808+0000 7fb000a00640 1 --2- 172.21.15.45:0/2001273020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc15ec80 0x7faffc0d2ce0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fafec0097a0 tx=0x7fafec02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21375ec56f80d950 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:58.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.808+0000 7faffb400640 1 -- 172.21.15.45:0/2001273020 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec02f810 con 0x7faffc15ec80 2024-04-16T16:20:58.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.808+0000 7faffb400640 1 -- 172.21.15.45:0/2001273020 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafec02f9b0 con 0x7faffc15ec80 2024-04-16T16:20:58.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.808+0000 7faffb400640 1 -- 172.21.15.45:0/2001273020 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec02fd60 con 0x7faffc15ec80 2024-04-16T16:20:58.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.809+0000 7fb003400640 1 -- 172.21.15.45:0/2001273020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc15ec80 msgr2=0x7faffc0d2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:58.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.809+0000 7fb003400640 1 --2- 172.21.15.45:0/2001273020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc15ec80 0x7faffc0d2ce0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fafec0097a0 tx=0x7fafec02b9f0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.809+0000 7fb003400640 1 -- 172.21.15.45:0/2001273020 shutdown_connections 2024-04-16T16:20:58.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.809+0000 7fb003400640 1 --2- 172.21.15.45:0/2001273020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc15ec80 0x7faffc0d2ce0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.809+0000 7fb003400640 1 -- 172.21.15.45:0/2001273020 >> 172.21.15.45:0/2001273020 conn(0x7faffc0d16d0 msgr2=0x7faffc0d1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:20:58.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.809+0000 7fb003400640 1 -- 172.21.15.45:0/2001273020 shutdown_connections 2024-04-16T16:20:58.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.809+0000 7fb003400640 1 -- 172.21.15.45:0/2001273020 wait complete. 2024-04-16T16:20:58.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.811+0000 7fb003400640 1 Processor -- start 2024-04-16T16:20:58.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.811+0000 7fb003400640 1 -- start start 2024-04-16T16:20:58.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.812+0000 7fb003400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faffc0b6c00 0x7faffc151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:58.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.812+0000 7fb003400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc15ec80 0x7faffc06cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:58.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.813+0000 7fb003400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faffc06d320 0x7faffc0cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:58.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.813+0000 7fb003400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7faffc0ba190 con 0x7faffc15ec80 2024-04-16T16:20:58.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.813+0000 7fb003400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faffc0d3cf0 con 0x7faffc06d320 2024-04-16T16:20:58.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.813+0000 7fb003400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faffc0ba310 con 0x7faffc0b6c00 2024-04-16T16:20:58.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.813+0000 7faffbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc15ec80 0x7faffc06cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:58.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.813+0000 7faffbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc15ec80 0x7faffc06cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34778/0 (socket says 172.21.15.45:34778) 2024-04-16T16:20:58.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.813+0000 7faffbe00640 1 -- 172.21.15.45:0/3266910941 learned_addr learned my addr 172.21.15.45:0/3266910941 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:20:58.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.813+0000 7fb000a00640 1 --2- 172.21.15.45:0/3266910941 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faffc0b6c00 0x7faffc151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:58.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.813+0000 7faffbe00640 1 -- 172.21.15.45:0/3266910941 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faffc0b6c00 msgr2=0x7faffc151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:58.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.813+0000 7faffbe00640 1 --2- 172.21.15.45:0/3266910941 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faffc0b6c00 0x7faffc151d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.813+0000 7faffbe00640 1 -- 172.21.15.45:0/3266910941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faffc06d320 msgr2=0x7faffc0cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:20:58.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.813+0000 7faffbe00640 1 --2- 172.21.15.45:0/3266910941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faffc06d320 0x7faffc0cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:20:58.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.813+0000 7faffbe00640 1 -- 172.21.15.45:0/3266910941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faffc06d7b0 con 0x7faffc15ec80 2024-04-16T16:20:58.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.813+0000 7fb000a00640 1 --2- 172.21.15.45:0/3266910941 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faffc0b6c00 0x7faffc151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:20:58.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.814+0000 7faffbe00640 1 --2- 172.21.15.45:0/3266910941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc15ec80 0x7faffc06cde0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fafe400c3c0 tx=0x7fafe400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:58.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.814+0000 7faff8c00640 1 -- 172.21.15.45:0/3266910941 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe40052d0 con 0x7faffc15ec80 2024-04-16T16:20:58.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.814+0000 7faff8c00640 1 -- 172.21.15.45:0/3266910941 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafe4005490 con 0x7faffc15ec80 2024-04-16T16:20:58.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.814+0000 7faff8c00640 1 -- 172.21.15.45:0/3266910941 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe4007260 con 0x7faffc15ec80 2024-04-16T16:20:58.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.814+0000 7fb003400640 1 -- 172.21.15.45:0/3266910941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faffc15f5d0 con 0x7faffc15ec80 2024-04-16T16:20:58.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.814+0000 7fb003400640 1 -- 172.21.15.45:0/3266910941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faffc15fb90 con 0x7faffc15ec80 2024-04-16T16:20:58.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.816+0000 7fb003400640 1 -- 172.21.15.45:0/3266910941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faffc075780 con 0x7faffc15ec80 2024-04-16T16:20:58.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.817+0000 7faff8c00640 1 -- 172.21.15.45:0/3266910941 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 54) ==== 91151+0+0 (secure 0 0 0) 0x7fafe4007400 con 0x7faffc15ec80 2024-04-16T16:20:58.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.818+0000 7faff8c00640 1 --2- 172.21.15.45:0/3266910941 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fafd806ca60 0x7fafd806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:20:58.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.818+0000 7faff8c00640 1 -- 172.21.15.45:0/3266910941 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6053+0+0 (secure 0 0 0) 0x7fafe408b410 con 0x7faffc15ec80 2024-04-16T16:20:58.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.818+0000 7fb000a00640 1 --2- 172.21.15.45:0/3266910941 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fafd806ca60 0x7fafd806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:20:58.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.820+0000 7fb000a00640 1 --2- 172.21.15.45:0/3266910941 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fafd806ca60 0x7fafd806ef20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fafec036000 tx=0x7fafec008030 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:20:58.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:58.822+0000 7faff8c00640 1 -- 172.21.15.45:0/3266910941 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fafe4059530 con 0x7faffc15ec80 2024-04-16T16:20:59.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:20:59.084+0000 7fb003400640 1 -- 172.21.15.45:0/3266910941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "stats"} v 0) -- 0x7faffc0b66a0 con 0x7faffc15ec80 2024-04-16T16:21:00.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.064+0000 7faff8c00640 1 -- 172.21.15.45:0/3266910941 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "stats"}]=0 v55) ==== 85+0+0 (secure 0 0 0) 0x7fafe4058e30 con 0x7faffc15ec80 2024-04-16T16:21:00.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.067+0000 7fb003400640 1 -- 172.21.15.45:0/3266910941 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fafd806ca60 msgr2=0x7fafd806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:00.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.067+0000 7fb003400640 1 --2- 172.21.15.45:0/3266910941 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fafd806ca60 0x7fafd806ef20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fafec036000 tx=0x7fafec008030 comp rx=0 tx=0).stop 2024-04-16T16:21:00.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.067+0000 7fb003400640 1 -- 172.21.15.45:0/3266910941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc15ec80 msgr2=0x7faffc06cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:00.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.067+0000 7fb003400640 1 --2- 172.21.15.45:0/3266910941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc15ec80 0x7faffc06cde0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fafe400c3c0 tx=0x7fafe400c880 comp rx=0 tx=0).stop 2024-04-16T16:21:00.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.068+0000 7fb003400640 1 -- 172.21.15.45:0/3266910941 shutdown_connections 2024-04-16T16:21:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.068+0000 7fb003400640 1 --2- 172.21.15.45:0/3266910941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faffc06d320 0x7faffc0cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.068+0000 7fb003400640 1 --2- 172.21.15.45:0/3266910941 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fafd806ca60 0x7fafd806ef20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.069+0000 7fb003400640 1 --2- 172.21.15.45:0/3266910941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc15ec80 0x7faffc06cde0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.069+0000 7fb003400640 1 --2- 172.21.15.45:0/3266910941 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faffc0b6c00 0x7faffc151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.069+0000 7fb003400640 1 -- 172.21.15.45:0/3266910941 >> 172.21.15.45:0/3266910941 conn(0x7faffc0d16d0 msgr2=0x7faffc0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:00.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.071+0000 7fb003400640 1 -- 172.21.15.45:0/3266910941 shutdown_connections 2024-04-16T16:21:00.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.071+0000 7fb003400640 1 -- 172.21.15.45:0/3266910941 wait complete. 2024-04-16T16:21:00.105 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-16T16:21:00.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:21:00.227 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:21:00.330 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-16T16:21:00.330 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:21:00.330 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:21:00.331 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:21:00.386 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:21:00.386 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:21:00.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:21:00.619 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:00.620+0000 7f9f6da0d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:21:00.658 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:00.658+0000 7f7fa9c46040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:21:00.674 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:00.675+0000 7f9f6da0d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:21:00.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.695+0000 7f024d600640 1 Processor -- start 2024-04-16T16:21:00.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.695+0000 7f024d600640 1 -- start start 2024-04-16T16:21:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.696+0000 7f024d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0248161e90 0x7f02480cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.697+0000 7f024d600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0248162270 0x7f02480d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.697+0000 7f024d600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02480d3080 con 0x7f02480d52d0 2024-04-16T16:21:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.697+0000 7f024d600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f02480d2e30 con 0x7f0248162270 2024-04-16T16:21:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.697+0000 7f024d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f02480d3360 con 0x7f0248161e90 2024-04-16T16:21:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.697+0000 7f024cc00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f02480d52d0 0x7f024806ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58678/0 (socket says 172.21.15.45:58678) 2024-04-16T16:21:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.697+0000 7f024cc00640 1 -- 172.21.15.45:0/1791578932 learned_addr learned my addr 172.21.15.45:0/1791578932 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.697+0000 7f0247400640 1 --2- 172.21.15.45:0/1791578932 >> v2:172.21.15.103:3300/0 conn(0x7f0248162270 0x7f02480d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.698+0000 7f0247e00640 1 --2- 172.21.15.45:0/1791578932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0248161e90 0x7f02480cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.698+0000 7f0246a00640 1 -- 172.21.15.45:0/1791578932 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1988438299 0 0) 0x7f02480d3080 con 0x7f02480d52d0 2024-04-16T16:21:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.698+0000 7f0246a00640 1 -- 172.21.15.45:0/1791578932 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f022c003520 con 0x7f02480d52d0 2024-04-16T16:21:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.698+0000 7f0247400640 1 -- 172.21.15.45:0/1791578932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0248161e90 msgr2=0x7f02480cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.698+0000 7f0247400640 1 --2- 172.21.15.45:0/1791578932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0248161e90 0x7f02480cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.698+0000 7f0247400640 1 -- 172.21.15.45:0/1791578932 >> v1:172.21.15.45:6789/0 conn(0x7f02480d52d0 legacy=0x7f024806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.698+0000 7f0247400640 1 -- 172.21.15.45:0/1791578932 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02480d3640 con 0x7f0248162270 2024-04-16T16:21:00.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.698+0000 7f0247e00640 1 --2- 172.21.15.45:0/1791578932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0248161e90 0x7f02480cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:21:00.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.699+0000 7f0247400640 1 --2- 172.21.15.45:0/1791578932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0248162270 0x7f02480d3e40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f02340097a0 tx=0x7f023402b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97af171a7172aaf server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:00.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.700+0000 7f0246a00640 1 -- 172.21.15.45:0/1791578932 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0234034070 con 0x7f0248162270 2024-04-16T16:21:00.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.700+0000 7f0246a00640 1 -- 172.21.15.45:0/1791578932 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0234030800 con 0x7f0248162270 2024-04-16T16:21:00.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.700+0000 7f0246a00640 1 -- 172.21.15.45:0/1791578932 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0234030bb0 con 0x7f0248162270 2024-04-16T16:21:00.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.700+0000 7f024d600640 1 -- 172.21.15.45:0/1791578932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0248162270 msgr2=0x7f02480d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:00.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.700+0000 7f024d600640 1 --2- 172.21.15.45:0/1791578932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0248162270 0x7f02480d3e40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f02340097a0 tx=0x7f023402b940 comp rx=0 tx=0).stop 2024-04-16T16:21:00.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.700+0000 7f024d600640 1 -- 172.21.15.45:0/1791578932 shutdown_connections 2024-04-16T16:21:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.700+0000 7f024d600640 1 --2- 172.21.15.45:0/1791578932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0248162270 0x7f02480d3e40 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.700+0000 7f024d600640 1 --2- 172.21.15.45:0/1791578932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0248161e90 0x7f02480cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.700+0000 7f024d600640 1 -- 172.21.15.45:0/1791578932 >> 172.21.15.45:0/1791578932 conn(0x7f02480d1940 msgr2=0x7f02480d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.700+0000 7f024d600640 1 -- 172.21.15.45:0/1791578932 shutdown_connections 2024-04-16T16:21:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.701+0000 7f024d600640 1 -- 172.21.15.45:0/1791578932 wait complete. 2024-04-16T16:21:00.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.702+0000 7f024d600640 1 Processor -- start 2024-04-16T16:21:00.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.702+0000 7f024d600640 1 -- start start 2024-04-16T16:21:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.703+0000 7f024d600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02480d52d0 0x7f02480acf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.703+0000 7f024d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0248161e90 0x7f0248153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.703+0000 7f024d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0248162270 0x7f02480d3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.703+0000 7f024d600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f024806d550 con 0x7f02480d52d0 2024-04-16T16:21:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.703+0000 7f024d600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f02480cfd10 con 0x7f0248162270 2024-04-16T16:21:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.703+0000 7f024d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f024806d6d0 con 0x7f0248161e90 2024-04-16T16:21:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.704+0000 7f0247e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02480d52d0 0x7f02480acf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.704+0000 7f0247e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02480d52d0 0x7f02480acf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34806/0 (socket says 172.21.15.45:34806) 2024-04-16T16:21:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.704+0000 7f0247e00640 1 -- 172.21.15.45:0/2949927245 learned_addr learned my addr 172.21.15.45:0/2949927245 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.704+0000 7f024cc00640 1 --2- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0248162270 0x7f02480d3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.704+0000 7f0247400640 1 --2- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0248161e90 0x7f0248153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.704+0000 7f0247e00640 1 -- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0248161e90 msgr2=0x7f0248153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.704+0000 7f0247e00640 1 --2- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0248161e90 0x7f0248153bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.704+0000 7f0247e00640 1 -- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0248162270 msgr2=0x7f02480d3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.704+0000 7f0247e00640 1 --2- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0248162270 0x7f02480d3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.704+0000 7f0247e00640 1 -- 172.21.15.45:0/2949927245 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02480abe90 con 0x7f02480d52d0 2024-04-16T16:21:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.704+0000 7f024cc00640 1 --2- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0248162270 0x7f02480d3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:21:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.705+0000 7f0247e00640 1 --2- 172.21.15.45:0/2949927245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02480d52d0 0x7f02480acf30 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f023800e7e0 tx=0x7f023800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.705+0000 7f023fe00640 1 -- 172.21.15.45:0/2949927245 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023800f040 con 0x7f02480d52d0 2024-04-16T16:21:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.705+0000 7f023fe00640 1 -- 172.21.15.45:0/2949927245 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0238004030 con 0x7f02480d52d0 2024-04-16T16:21:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.705+0000 7f023fe00640 1 -- 172.21.15.45:0/2949927245 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238004400 con 0x7f02480d52d0 2024-04-16T16:21:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.705+0000 7f024d600640 1 -- 172.21.15.45:0/2949927245 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02480ac170 con 0x7f02480d52d0 2024-04-16T16:21:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.705+0000 7f024d600640 1 -- 172.21.15.45:0/2949927245 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02480ac730 con 0x7f02480d52d0 2024-04-16T16:21:00.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.707+0000 7f024d600640 1 -- 172.21.15.45:0/2949927245 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0248075780 con 0x7f02480d52d0 2024-04-16T16:21:00.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.707+0000 7f023fe00640 1 -- 172.21.15.45:0/2949927245 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 55) ==== 91142+0+0 (secure 0 0 0) 0x7f023800c6d0 con 0x7f02480d52d0 2024-04-16T16:21:00.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.708+0000 7f023fe00640 1 --2- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f022806ca40 0x7f022806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:00.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.709+0000 7f023fe00640 1 -- 172.21.15.45:0/2949927245 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6053+0+0 (secure 0 0 0) 0x7f023808be20 con 0x7f02480d52d0 2024-04-16T16:21:00.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.709+0000 7f0247400640 1 -- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f022806ca40 msgr2=0x7f022806ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/549890383 2024-04-16T16:21:00.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.709+0000 7f0247400640 1 --2- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f022806ca40 0x7f022806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:21:00.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.713+0000 7f023fe00640 1 -- 172.21.15.45:0/2949927245 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0238017020 con 0x7f02480d52d0 2024-04-16T16:21:00.723 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:00.723+0000 7f7fa9c46040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:21:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.910+0000 7f0247400640 1 -- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f022806ca40 msgr2=0x7f022806ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/549890383 2024-04-16T16:21:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.910+0000 7f0247400640 1 --2- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f022806ca40 0x7f022806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:21:00.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.945+0000 7f024d600640 1 -- 172.21.15.45:0/2949927245 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f024806dae0 con 0x7f02480d52d0 2024-04-16T16:21:00.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.948+0000 7f023fe00640 1 -- 172.21.15.45:0/2949927245 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) ==== 74+0+17188 (secure 0 0 0) 0x7f0238059dc0 con 0x7f02480d52d0 2024-04-16T16:21:00.948 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:00.949 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":129,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:20:50.935396+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:20:31.681227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-16T16:20:32.790980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6833/1120913661":"2024-04-17T16:20:50.935324+0000","172.21.15.103:6832/1120913661":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/4103173879":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/4177084908":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1961421675":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/2190202958":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/3679729711":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1543358977":"2024-04-17T16:20:50.935324+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:21:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.951+0000 7f024d600640 1 -- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f022806ca40 msgr2=0x7f022806ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:21:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.951+0000 7f024d600640 1 --2- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f022806ca40 0x7f022806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.951+0000 7f024d600640 1 -- 172.21.15.45:0/2949927245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02480d52d0 msgr2=0x7f02480acf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.951+0000 7f024d600640 1 --2- 172.21.15.45:0/2949927245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02480d52d0 0x7f02480acf30 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f023800e7e0 tx=0x7f023800eca0 comp rx=0 tx=0).stop 2024-04-16T16:21:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.951+0000 7f024d600640 1 -- 172.21.15.45:0/2949927245 shutdown_connections 2024-04-16T16:21:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.951+0000 7f024d600640 1 --2- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0248162270 0x7f02480d3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.951+0000 7f024d600640 1 --2- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0248161e90 0x7f0248153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.951+0000 7f024d600640 1 --2- 172.21.15.45:0/2949927245 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f022806ca40 0x7f022806ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.951+0000 7f024d600640 1 --2- 172.21.15.45:0/2949927245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02480d52d0 0x7f02480acf30 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.951+0000 7f024d600640 1 -- 172.21.15.45:0/2949927245 >> 172.21.15.45:0/2949927245 conn(0x7f02480d1940 msgr2=0x7f02480b78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.951+0000 7f024d600640 1 -- 172.21.15.45:0/2949927245 shutdown_connections 2024-04-16T16:21:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:00.951+0000 7f024d600640 1 -- 172.21.15.45:0/2949927245 wait complete. 2024-04-16T16:21:00.978 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:20:31.681227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:20:32.790980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:21:00.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:21:01.125 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:01.126+0000 7f9f6da0d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:21:01.177 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:01.178+0000 7f9f6da0d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:21:01.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.198+0000 7fb801000640 1 Processor -- start 2024-04-16T16:21:01.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.198+0000 7fb801000640 1 -- start start 2024-04-16T16:21:01.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.200+0000 7fb801000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb7fc15ea70 0x7fb7fc0d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.200+0000 7fb801000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb7fc0d4370 0x7fb7fc0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.200+0000 7fb801000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7fc0cfa70 0x7fb7fc0d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.200+0000 7fb801000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb7fc15ee50 con 0x7fb7fc0d4370 2024-04-16T16:21:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.200+0000 7fb801000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb7fc0d1a10 con 0x7fb7fc15ea70 2024-04-16T16:21:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.200+0000 7fb801000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb7fc0b6480 con 0x7fb7fc0cfa70 2024-04-16T16:21:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.200+0000 7fb7faa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb7fc0d4370 0x7fb7fc0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.200+0000 7fb7faa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb7fc0d4370 0x7fb7fc0cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34816/0 (socket says 172.21.15.45:34816) 2024-04-16T16:21:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.200+0000 7fb7faa00640 1 -- 172.21.15.45:0/656780227 learned_addr learned my addr 172.21.15.45:0/656780227 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.201+0000 7fb7fb400640 1 --2- 172.21.15.45:0/656780227 >> v2:172.21.15.103:3300/0 conn(0x7fb7fc15ea70 0x7fb7fc0d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.201+0000 7fb7fbe00640 1 --2- 172.21.15.45:0/656780227 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7fc0cfa70 0x7fb7fc0d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.201+0000 7fb7faa00640 1 -- 172.21.15.45:0/656780227 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7fc0cfa70 msgr2=0x7fb7fc0d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.201+0000 7fb7faa00640 1 --2- 172.21.15.45:0/656780227 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7fc0cfa70 0x7fb7fc0d2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.201+0000 7fb7faa00640 1 -- 172.21.15.45:0/656780227 >> v2:172.21.15.103:3300/0 conn(0x7fb7fc15ea70 msgr2=0x7fb7fc0d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.201+0000 7fb7faa00640 1 --2- 172.21.15.45:0/656780227 >> v2:172.21.15.103:3300/0 conn(0x7fb7fc15ea70 0x7fb7fc0d11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.201+0000 7fb7faa00640 1 -- 172.21.15.45:0/656780227 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7fc0b6760 con 0x7fb7fc0d4370 2024-04-16T16:21:01.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.201+0000 7fb7fb400640 1 --2- 172.21.15.45:0/656780227 >> v2:172.21.15.103:3300/0 conn(0x7fb7fc15ea70 0x7fb7fc0d11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:21:01.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.202+0000 7fb7faa00640 1 --2- 172.21.15.45:0/656780227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0d4370 0x7fb7fc0cf530 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0d5890 tx=0x7fb7e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f95222621959e834 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:01.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.202+0000 7fb7fa000640 1 -- 172.21.15.45:0/656780227 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8035020 con 0x7fb7fc0d4370 2024-04-16T16:21:01.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.202+0000 7fb7fa000640 1 -- 172.21.15.45:0/656780227 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7e8031870 con 0x7fb7fc0d4370 2024-04-16T16:21:01.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.202+0000 7fb7fa000640 1 -- 172.21.15.45:0/656780227 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8031c40 con 0x7fb7fc0d4370 2024-04-16T16:21:01.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.203+0000 7fb801000640 1 -- 172.21.15.45:0/656780227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0d4370 msgr2=0x7fb7fc0cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:01.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.203+0000 7fb801000640 1 --2- 172.21.15.45:0/656780227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0d4370 0x7fb7fc0cf530 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0d5890 tx=0x7fb7e802b910 comp rx=0 tx=0).stop 2024-04-16T16:21:01.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.203+0000 7fb801000640 1 -- 172.21.15.45:0/656780227 shutdown_connections 2024-04-16T16:21:01.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.203+0000 7fb801000640 1 --2- 172.21.15.45:0/656780227 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7fc0cfa70 0x7fb7fc0d2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.203+0000 7fb801000640 1 --2- 172.21.15.45:0/656780227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0d4370 0x7fb7fc0cf530 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.203+0000 7fb801000640 1 --2- 172.21.15.45:0/656780227 >> v2:172.21.15.103:3300/0 conn(0x7fb7fc15ea70 0x7fb7fc0d11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.203+0000 7fb801000640 1 -- 172.21.15.45:0/656780227 >> 172.21.15.45:0/656780227 conn(0x7fb7fc06d3b0 msgr2=0x7fb7fc073060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:01.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.203+0000 7fb801000640 1 -- 172.21.15.45:0/656780227 shutdown_connections 2024-04-16T16:21:01.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.203+0000 7fb801000640 1 -- 172.21.15.45:0/656780227 wait complete. 2024-04-16T16:21:01.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.205+0000 7fb801000640 1 Processor -- start 2024-04-16T16:21:01.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.205+0000 7fb801000640 1 -- start start 2024-04-16T16:21:01.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.206+0000 7fb801000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7fc0cfa70 0x7fb7fc15caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:01.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.206+0000 7fb801000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7fc0d4370 0x7fb7fc1f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:01.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.206+0000 7fb801000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc15ea70 0x7fb7fc1f7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:01.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.206+0000 7fb801000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc0b6f60 con 0x7fb7fc15ea70 2024-04-16T16:21:01.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.206+0000 7fb801000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc0d16e0 con 0x7fb7fc0d4370 2024-04-16T16:21:01.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.206+0000 7fb801000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb7fc0ce770 con 0x7fb7fc0cfa70 2024-04-16T16:21:01.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.206+0000 7fb7fbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc15ea70 0x7fb7fc1f7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:01.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.206+0000 7fb7fbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc15ea70 0x7fb7fc1f7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34830/0 (socket says 172.21.15.45:34830) 2024-04-16T16:21:01.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.206+0000 7fb7fbe00640 1 -- 172.21.15.45:0/2600157905 learned_addr learned my addr 172.21.15.45:0/2600157905 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:01.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.207+0000 7fb7faa00640 1 --2- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7fc0d4370 0x7fb7fc1f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:01.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.207+0000 7fb7fb400640 1 --2- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7fc0cfa70 0x7fb7fc15caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:01.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.207+0000 7fb7fbe00640 1 -- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7fc0cfa70 msgr2=0x7fb7fc15caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:01.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.207+0000 7fb7fbe00640 1 --2- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7fc0cfa70 0x7fb7fc15caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.207+0000 7fb7fbe00640 1 -- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7fc0d4370 msgr2=0x7fb7fc1f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:01.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.207+0000 7fb7fbe00640 1 --2- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7fc0d4370 0x7fb7fc1f76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.207+0000 7fb7fbe00640 1 -- 172.21.15.45:0/2600157905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7fc15c2e0 con 0x7fb7fc15ea70 2024-04-16T16:21:01.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.207+0000 7fb7fbe00640 1 --2- 172.21.15.45:0/2600157905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc15ea70 0x7fb7fc1f7be0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4007d20 tx=0x7fb7f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:01.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.207+0000 7fb7f3400640 1 -- 172.21.15.45:0/2600157905 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4014420 con 0x7fb7fc15ea70 2024-04-16T16:21:01.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.208+0000 7fb7f3400640 1 -- 172.21.15.45:0/2600157905 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7f4019070 con 0x7fb7fc15ea70 2024-04-16T16:21:01.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.208+0000 7fb7f3400640 1 -- 172.21.15.45:0/2600157905 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4014820 con 0x7fb7fc15ea70 2024-04-16T16:21:01.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.208+0000 7fb801000640 1 -- 172.21.15.45:0/2600157905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7fc1f8120 con 0x7fb7fc15ea70 2024-04-16T16:21:01.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.208+0000 7fb801000640 1 -- 172.21.15.45:0/2600157905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7fc0ce490 con 0x7fb7fc15ea70 2024-04-16T16:21:01.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.209+0000 7fb801000640 1 -- 172.21.15.45:0/2600157905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7fc075780 con 0x7fb7fc15ea70 2024-04-16T16:21:01.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.210+0000 7fb7f3400640 1 -- 172.21.15.45:0/2600157905 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 55) ==== 91142+0+0 (secure 0 0 0) 0x7fb7f40149c0 con 0x7fb7fc15ea70 2024-04-16T16:21:01.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.210+0000 7fb7f3400640 1 --2- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fb7dc06ca10 0x7fb7dc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:01.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.211+0000 7fb7f3400640 1 -- 172.21.15.45:0/2600157905 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6053+0+0 (secure 0 0 0) 0x7fb7f408c540 con 0x7fb7fc15ea70 2024-04-16T16:21:01.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.211+0000 7fb7fb400640 1 -- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fb7dc06ca10 msgr2=0x7fb7dc06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/549890383 2024-04-16T16:21:01.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.211+0000 7fb7fb400640 1 --2- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fb7dc06ca10 0x7fb7dc06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:21:01.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.215+0000 7fb7f3400640 1 -- 172.21.15.45:0/2600157905 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb7f405a6f0 con 0x7fb7fc15ea70 2024-04-16T16:21:01.230 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:01.230+0000 7f9f6da0d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:21:01.238 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:01.238+0000 7f7fa9c46040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:21:01.291 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:01.292+0000 7f9f6da0d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:21:01.293 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:01.294+0000 7f7fa9c46040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:21:01.340 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:01.341+0000 7f9f6da0d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:21:01.347 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:01.347+0000 7f7fa9c46040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:21:01.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.411+0000 7fb801000640 1 -- 172.21.15.45:0/2600157905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb7fc0d2500 con 0x7fb7fc15ea70 2024-04-16T16:21:01.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.412+0000 7fb7fb400640 1 -- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fb7dc06ca10 msgr2=0x7fb7dc06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/549890383 2024-04-16T16:21:01.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.412+0000 7fb7fb400640 1 --2- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fb7dc06ca10 0x7fb7dc06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:21:01.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.412+0000 7fb7f3400640 1 -- 172.21.15.45:0/2600157905 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) ==== 93+0+10 (secure 0 0 0) 0x7fb7f4059ff0 con 0x7fb7fc15ea70 2024-04-16T16:21:01.412 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:21:01.412 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:01.412+0000 7f7fa9c46040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:21:01.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.414+0000 7fb801000640 1 -- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fb7dc06ca10 msgr2=0x7fb7dc06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:21:01.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.414+0000 7fb801000640 1 --2- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fb7dc06ca10 0x7fb7dc06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.414+0000 7fb801000640 1 -- 172.21.15.45:0/2600157905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc15ea70 msgr2=0x7fb7fc1f7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:01.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.414+0000 7fb801000640 1 --2- 172.21.15.45:0/2600157905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc15ea70 0x7fb7fc1f7be0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4007d20 tx=0x7fb7f400e4b0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.415+0000 7fb801000640 1 -- 172.21.15.45:0/2600157905 shutdown_connections 2024-04-16T16:21:01.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.415+0000 7fb801000640 1 --2- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fb7dc06ca10 0x7fb7dc06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.415+0000 7fb801000640 1 --2- 172.21.15.45:0/2600157905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc15ea70 0x7fb7fc1f7be0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.415+0000 7fb801000640 1 --2- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7fc0d4370 0x7fb7fc1f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.415+0000 7fb801000640 1 --2- 172.21.15.45:0/2600157905 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7fc0cfa70 0x7fb7fc15caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.415+0000 7fb801000640 1 -- 172.21.15.45:0/2600157905 >> 172.21.15.45:0/2600157905 conn(0x7fb7fc06d3b0 msgr2=0x7fb7fc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:01.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.415+0000 7fb801000640 1 -- 172.21.15.45:0/2600157905 shutdown_connections 2024-04-16T16:21:01.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.415+0000 7fb801000640 1 -- 172.21.15.45:0/2600157905 wait complete. 2024-04-16T16:21:01.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:21:01.467 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:01.468+0000 7f7fa9c46040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:21:01.469 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:01.470+0000 7f9f6da0d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:21:01.532 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:01.533+0000 7f9f6da0d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:21:01.584 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:01.584+0000 7f9f6da0d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:21:01.608 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:01.609+0000 7f7fa9c46040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:21:01.641 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:01.641+0000 7f9f6da0d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:21:01.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.649+0000 7f980f000640 1 Processor -- start 2024-04-16T16:21:01.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.650+0000 7f980f000640 1 -- start start 2024-04-16T16:21:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.651+0000 7f980f000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9808161f50 0x7f9808073ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.651+0000 7f980f000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f98080cf410 0x7f9808162e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.651+0000 7f980f000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9808162330 con 0x7f98080b36e0 2024-04-16T16:21:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.651+0000 7f980f000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f98080cfc60 con 0x7f98080cf410 2024-04-16T16:21:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.651+0000 7f980f000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f98080cff40 con 0x7f9808161f50 2024-04-16T16:21:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.652+0000 7f980d200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f98080cf410 0x7f9808162e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.652+0000 7f980d200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f98080cf410 0x7f9808162e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34840/0 (socket says 172.21.15.45:34840) 2024-04-16T16:21:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.652+0000 7f980d200640 1 -- 172.21.15.45:0/2764791180 learned_addr learned my addr 172.21.15.45:0/2764791180 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.652+0000 7f980dc00640 1 --2- 172.21.15.45:0/2764791180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9808161f50 0x7f9808073ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.652+0000 7f980d200640 1 -- 172.21.15.45:0/2764791180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9808161f50 msgr2=0x7f9808073ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.652+0000 7f980d200640 1 --2- 172.21.15.45:0/2764791180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9808161f50 0x7f9808073ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.652+0000 7f980d200640 1 -- 172.21.15.45:0/2764791180 >> v1:172.21.15.103:6789/0 conn(0x7f98080b36e0 legacy=0x7f98080cdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:01.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.652+0000 7f980d200640 1 -- 172.21.15.45:0/2764791180 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9808162680 con 0x7f98080cf410 2024-04-16T16:21:01.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.652+0000 7f980dc00640 1 --2- 172.21.15.45:0/2764791180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9808161f50 0x7f9808073ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:01.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.653+0000 7f980d200640 1 --2- 172.21.15.45:0/2764791180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98080cf410 0x7f9808162e80 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009870 tx=0x7f97f402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d25969158ea1c8f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:01.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.654+0000 7f9807e00640 1 -- 172.21.15.45:0/2764791180 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f40317c0 con 0x7f98080cf410 2024-04-16T16:21:01.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.654+0000 7f9807e00640 1 -- 172.21.15.45:0/2764791180 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97f4037070 con 0x7f98080cf410 2024-04-16T16:21:01.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.654+0000 7f9807e00640 1 -- 172.21.15.45:0/2764791180 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f4031b80 con 0x7f98080cf410 2024-04-16T16:21:01.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.654+0000 7f980f000640 1 -- 172.21.15.45:0/2764791180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98080cf410 msgr2=0x7f9808162e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:01.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.654+0000 7f980f000640 1 --2- 172.21.15.45:0/2764791180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98080cf410 0x7f9808162e80 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009870 tx=0x7f97f402b860 comp rx=0 tx=0).stop 2024-04-16T16:21:01.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.654+0000 7f980f000640 1 -- 172.21.15.45:0/2764791180 shutdown_connections 2024-04-16T16:21:01.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.654+0000 7f980f000640 1 --2- 172.21.15.45:0/2764791180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98080cf410 0x7f9808162e80 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.654+0000 7f980f000640 1 --2- 172.21.15.45:0/2764791180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9808161f50 0x7f9808073ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.654+0000 7f980f000640 1 -- 172.21.15.45:0/2764791180 >> 172.21.15.45:0/2764791180 conn(0x7f98080b6ae0 msgr2=0x7f98080b6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:01.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.655+0000 7f980f000640 1 -- 172.21.15.45:0/2764791180 shutdown_connections 2024-04-16T16:21:01.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.655+0000 7f980f000640 1 -- 172.21.15.45:0/2764791180 wait complete. 2024-04-16T16:21:01.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.656+0000 7f980f000640 1 Processor -- start 2024-04-16T16:21:01.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.656+0000 7f980f000640 1 -- start start 2024-04-16T16:21:01.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.657+0000 7f980f000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98080b36e0 0x7f9808146b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:01.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.657+0000 7f980f000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98080cf410 0x7f980815ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:01.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.657+0000 7f980f000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9808161f50 0x7f98080cc810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:01.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.657+0000 7f980f000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f98080d03a0 con 0x7f98080b36e0 2024-04-16T16:21:01.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.657+0000 7f980f000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f98080d0220 con 0x7f98080cf410 2024-04-16T16:21:01.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.657+0000 7f980f000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f98080cf870 con 0x7f9808161f50 2024-04-16T16:21:01.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.657+0000 7f980dc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98080b36e0 0x7f9808146b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:01.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.658+0000 7f980dc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98080b36e0 0x7f9808146b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34842/0 (socket says 172.21.15.45:34842) 2024-04-16T16:21:01.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.658+0000 7f980dc00640 1 -- 172.21.15.45:0/3838437179 learned_addr learned my addr 172.21.15.45:0/3838437179 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:01.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.658+0000 7f980d200640 1 --2- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98080cf410 0x7f980815ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:01.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.658+0000 7f980e600640 1 --2- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9808161f50 0x7f98080cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:01.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.658+0000 7f980dc00640 1 -- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9808161f50 msgr2=0x7f98080cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:01.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.658+0000 7f980dc00640 1 --2- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9808161f50 0x7f98080cc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.658+0000 7f980dc00640 1 -- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98080cf410 msgr2=0x7f980815ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:01.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.658+0000 7f980dc00640 1 --2- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98080cf410 0x7f980815ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.658+0000 7f980dc00640 1 -- 172.21.15.45:0/3838437179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98081477f0 con 0x7f98080b36e0 2024-04-16T16:21:01.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.658+0000 7f980d200640 1 --2- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98080cf410 0x7f980815ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:01.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.658+0000 7f980dc00640 1 --2- 172.21.15.45:0/3838437179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98080b36e0 0x7f9808146b00 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00cf60 tx=0x7f97fc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:01.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.659+0000 7f9805600640 1 -- 172.21.15.45:0/3838437179 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97fc01a070 con 0x7f98080b36e0 2024-04-16T16:21:01.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.659+0000 7f9805600640 1 -- 172.21.15.45:0/3838437179 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97fc00d840 con 0x7f98080b36e0 2024-04-16T16:21:01.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.659+0000 7f9805600640 1 -- 172.21.15.45:0/3838437179 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97fc00dc10 con 0x7f98080b36e0 2024-04-16T16:21:01.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.659+0000 7f980f000640 1 -- 172.21.15.45:0/3838437179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9808147ad0 con 0x7f98080b36e0 2024-04-16T16:21:01.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.659+0000 7f980f000640 1 -- 172.21.15.45:0/3838437179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9808148090 con 0x7f98080b36e0 2024-04-16T16:21:01.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.660+0000 7f980f000640 1 -- 172.21.15.45:0/3838437179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9808075780 con 0x7f98080b36e0 2024-04-16T16:21:01.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.661+0000 7f9805600640 1 -- 172.21.15.45:0/3838437179 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 55) ==== 91142+0+0 (secure 0 0 0) 0x7f97fc01d070 con 0x7f98080b36e0 2024-04-16T16:21:01.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.662+0000 7f9805600640 1 --2- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f97e806ca10 0x7f97e806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:01.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.662+0000 7f9805600640 1 -- 172.21.15.45:0/3838437179 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6053+0+0 (secure 0 0 0) 0x7f97fc08c6c0 con 0x7f98080b36e0 2024-04-16T16:21:01.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.662+0000 7f980d200640 1 -- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f97e806ca10 msgr2=0x7f97e806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/549890383 2024-04-16T16:21:01.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.662+0000 7f980d200640 1 --2- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f97e806ca10 0x7f97e806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:21:01.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.666+0000 7f9805600640 1 -- 172.21.15.45:0/3838437179 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f97fc05a220 con 0x7f98080b36e0 2024-04-16T16:21:01.677 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:01.677+0000 7f7fa9c46040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:21:01.693 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:01.693+0000 7f9f6da0d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:21:01.731 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:01.730+0000 7f7fa9c46040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:21:01.791 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:01.791+0000 7f7fa9c46040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:21:01.845 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:01.844+0000 7f7fa9c46040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:21:01.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.862+0000 7f980f000640 1 -- 172.21.15.45:0/3838437179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f98080b6160 con 0x7f98080b36e0 2024-04-16T16:21:01.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.863+0000 7f980d200640 1 -- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f97e806ca10 msgr2=0x7f97e806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/549890383 2024-04-16T16:21:01.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.863+0000 7f980d200640 1 --2- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f97e806ca10 0x7f97e806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:21:01.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.863+0000 7f9805600640 1 -- 172.21.15.45:0/3838437179 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) ==== 104+0+11 (secure 0 0 0) 0x7f97fc05e520 con 0x7f98080b36e0 2024-04-16T16:21:01.864 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:21:01.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.866+0000 7f980f000640 1 -- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f97e806ca10 msgr2=0x7f97e806eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:21:01.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.866+0000 7f980f000640 1 --2- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f97e806ca10 0x7f97e806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.866+0000 7f980f000640 1 -- 172.21.15.45:0/3838437179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98080b36e0 msgr2=0x7f9808146b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:01.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.866+0000 7f980f000640 1 --2- 172.21.15.45:0/3838437179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98080b36e0 0x7f9808146b00 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00cf60 tx=0x7f97fc00cf90 comp rx=0 tx=0).stop 2024-04-16T16:21:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.866+0000 7f980f000640 1 -- 172.21.15.45:0/3838437179 shutdown_connections 2024-04-16T16:21:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.866+0000 7f980f000640 1 --2- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9808161f50 0x7f98080cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.866+0000 7f980f000640 1 --2- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98080cf410 0x7f980815ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.866+0000 7f980f000640 1 --2- 172.21.15.45:0/3838437179 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f97e806ca10 0x7f97e806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.866+0000 7f980f000640 1 --2- 172.21.15.45:0/3838437179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98080b36e0 0x7f9808146b00 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.866+0000 7f980f000640 1 -- 172.21.15.45:0/3838437179 >> 172.21.15.45:0/3838437179 conn(0x7f98080b6ae0 msgr2=0x7f98080b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.866+0000 7f980f000640 1 -- 172.21.15.45:0/3838437179 shutdown_connections 2024-04-16T16:21:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:01.866+0000 7f980f000640 1 -- 172.21.15.45:0/3838437179 wait complete. 2024-04-16T16:21:01.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:21:02.080 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:02.080+0000 7f9f6da0d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:21:02.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.143+0000 7fa3d6e00640 1 Processor -- start 2024-04-16T16:21:02.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.143+0000 7fa3d6e00640 1 -- start start 2024-04-16T16:21:02.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.145+0000 7fa3d6e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3d00d18f0 0x7fa3d00ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.145+0000 7fa3d6e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa3d00b6310 0x7fa3d00d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.145+0000 7fa3d6e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3d00b6810 con 0x7fa3d00d1cd0 2024-04-16T16:21:02.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.145+0000 7fa3d6e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa3d00d29f0 con 0x7fa3d00b6310 2024-04-16T16:21:02.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.145+0000 7fa3d6e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa3d00b6af0 con 0x7fa3d00d18f0 2024-04-16T16:21:02.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.145+0000 7fa3d6400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa3d00d1cd0 0x7fa3d00d33c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58694/0 (socket says 172.21.15.45:58694) 2024-04-16T16:21:02.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.145+0000 7fa3d6400640 1 -- 172.21.15.45:0/619814792 learned_addr learned my addr 172.21.15.45:0/619814792 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:02.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.145+0000 7fa3d5a00640 1 --2- 172.21.15.45:0/619814792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3d00d18f0 0x7fa3d00ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3d5000640 1 --2- 172.21.15.45:0/619814792 >> v2:172.21.15.103:3300/0 conn(0x7fa3d00b6310 0x7fa3d00d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 -- 172.21.15.45:0/619814792 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4194514887 0 0) 0x7fa3d00b6810 con 0x7fa3d00d1cd0 2024-04-16T16:21:02.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 -- 172.21.15.45:0/619814792 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3b8003520 con 0x7fa3d00d1cd0 2024-04-16T16:21:02.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 -- 172.21.15.45:0/619814792 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2212919457 0 0) 0x7fa3b8003520 con 0x7fa3d00d1cd0 2024-04-16T16:21:02.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 -- 172.21.15.45:0/619814792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3d00d18f0 msgr2=0x7fa3d00ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:02.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 --2- 172.21.15.45:0/619814792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3d00d18f0 0x7fa3d00ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 -- 172.21.15.45:0/619814792 >> v2:172.21.15.103:3300/0 conn(0x7fa3d00b6310 msgr2=0x7fa3d00d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:02.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 --2- 172.21.15.45:0/619814792 >> v2:172.21.15.103:3300/0 conn(0x7fa3d00b6310 0x7fa3d00d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 -- 172.21.15.45:0/619814792 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d00b6dd0 con 0x7fa3d00d1cd0 2024-04-16T16:21:02.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 -- 172.21.15.45:0/619814792 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fa3c80019f0 con 0x7fa3d00d1cd0 2024-04-16T16:21:02.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 -- 172.21.15.45:0/619814792 >> v1:172.21.15.45:6789/0 conn(0x7fa3d00d1cd0 legacy=0x7fa3d00d33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:02.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 --2- 172.21.15.45:0/619814792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3b80039b0 0x7fa3b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 --2- 172.21.15.45:0/619814792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3d00d18f0 0x7fa3b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 --2- 172.21.15.45:0/619814792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b8004d40 0x7fa3b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 -- 172.21.15.45:0/619814792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d00d29f0 con 0x7fa3b8004d40 2024-04-16T16:21:02.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 -- 172.21.15.45:0/619814792 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d00b6af0 con 0x7fa3b80039b0 2024-04-16T16:21:02.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.146+0000 7fa3cfe00640 1 -- 172.21.15.45:0/619814792 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa3b800b9f0 con 0x7fa3d00d18f0 2024-04-16T16:21:02.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.147+0000 7fa3d6400640 1 --2- 172.21.15.45:0/619814792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b8004d40 0x7fa3b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.147+0000 7fa3d5000640 1 --2- 172.21.15.45:0/619814792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3d00d18f0 0x7fa3b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.147+0000 7fa3d5a00640 1 --2- 172.21.15.45:0/619814792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3b80039b0 0x7fa3b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.147+0000 7fa3d6400640 1 -- 172.21.15.45:0/619814792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3d00d18f0 msgr2=0x7fa3b8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:02.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.147+0000 7fa3d6400640 1 --2- 172.21.15.45:0/619814792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3d00d18f0 0x7fa3b8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.147+0000 7fa3d6400640 1 -- 172.21.15.45:0/619814792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3b80039b0 msgr2=0x7fa3b8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:02.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.147+0000 7fa3d6400640 1 --2- 172.21.15.45:0/619814792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3b80039b0 0x7fa3b8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.147+0000 7fa3d6400640 1 -- 172.21.15.45:0/619814792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b8003520 con 0x7fa3b8004d40 2024-04-16T16:21:02.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.148+0000 7fa3d6400640 1 --2- 172.21.15.45:0/619814792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b8004d40 0x7fa3b80051c0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800a7a0 tx=0x7fa3b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0d23c1831886980 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:02.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.148+0000 7fa3cfe00640 1 -- 172.21.15.45:0/619814792 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c800a3f0 con 0x7fa3b8004d40 2024-04-16T16:21:02.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.148+0000 7fa3cfe00640 1 -- 172.21.15.45:0/619814792 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3c8034630 con 0x7fa3b8004d40 2024-04-16T16:21:02.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.148+0000 7fa3cfe00640 1 -- 172.21.15.45:0/619814792 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8034ac0 con 0x7fa3b8004d40 2024-04-16T16:21:02.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.149+0000 7fa3d6e00640 1 -- 172.21.15.45:0/619814792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b8004d40 msgr2=0x7fa3b80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:02.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.149+0000 7fa3d6e00640 1 --2- 172.21.15.45:0/619814792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b8004d40 0x7fa3b80051c0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800a7a0 tx=0x7fa3b8005830 comp rx=0 tx=0).stop 2024-04-16T16:21:02.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.149+0000 7fa3d6400640 1 -- 172.21.15.45:0/619814792 reap_dead start 2024-04-16T16:21:02.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.149+0000 7fa3d6e00640 1 -- 172.21.15.45:0/619814792 shutdown_connections 2024-04-16T16:21:02.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.149+0000 7fa3d6e00640 1 -- 172.21.15.45:0/619814792 >> 172.21.15.45:0/619814792 conn(0x7fa3d006cf90 msgr2=0x7fa3d006d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:02.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.149+0000 7fa3d6e00640 1 -- 172.21.15.45:0/619814792 shutdown_connections 2024-04-16T16:21:02.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.149+0000 7fa3d6e00640 1 -- 172.21.15.45:0/619814792 wait complete. 2024-04-16T16:21:02.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.151+0000 7fa3d6e00640 1 Processor -- start 2024-04-16T16:21:02.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.151+0000 7fa3d6e00640 1 -- start start 2024-04-16T16:21:02.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.152+0000 7fa3d6e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b8004d40 0x7fa3d0160170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.152+0000 7fa3d6e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d00cb960 0x7fa3d015f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.152+0000 7fa3d6e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3d0154920 0x7fa3d0154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.152+0000 7fa3d6e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d00ba3a0 con 0x7fa3b8004d40 2024-04-16T16:21:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.152+0000 7fa3d6e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d00ba220 con 0x7fa3d00cb960 2024-04-16T16:21:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.152+0000 7fa3d6e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa3d00ba520 con 0x7fa3d0154920 2024-04-16T16:21:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.152+0000 7fa3d5a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b8004d40 0x7fa3d0160170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.153+0000 7fa3d5a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b8004d40 0x7fa3d0160170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34856/0 (socket says 172.21.15.45:34856) 2024-04-16T16:21:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.153+0000 7fa3d5a00640 1 -- 172.21.15.45:0/2264208580 learned_addr learned my addr 172.21.15.45:0/2264208580 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.153+0000 7fa3d6400640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3d0154920 0x7fa3d0154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.153+0000 7fa3d5000640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d00cb960 0x7fa3d015f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.153+0000 7fa3d5a00640 1 -- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3d0154920 msgr2=0x7fa3d0154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.153+0000 7fa3d5a00640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3d0154920 0x7fa3d0154d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.153+0000 7fa3d5a00640 1 -- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d00cb960 msgr2=0x7fa3d015f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.153+0000 7fa3d5a00640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d00cb960 0x7fa3d015f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.153+0000 7fa3d5a00640 1 -- 172.21.15.45:0/2264208580 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d01552a0 con 0x7fa3b8004d40 2024-04-16T16:21:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.153+0000 7fa3d6400640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3d0154920 0x7fa3d0154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:21:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.153+0000 7fa3d5000640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d00cb960 0x7fa3d015f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:21:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.153+0000 7fa3d5a00640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b8004d40 0x7fa3d0160170 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4011540 tx=0x7fa3c4011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.154+0000 7fa3cd600640 1 -- 172.21.15.45:0/2264208580 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4015930 con 0x7fa3b8004d40 2024-04-16T16:21:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.154+0000 7fa3cd600640 1 -- 172.21.15.45:0/2264208580 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3c4015ad0 con 0x7fa3b8004d40 2024-04-16T16:21:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.154+0000 7fa3cd600640 1 -- 172.21.15.45:0/2264208580 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c400f460 con 0x7fa3b8004d40 2024-04-16T16:21:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.154+0000 7fa3d6e00640 1 -- 172.21.15.45:0/2264208580 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3d00cbf20 con 0x7fa3b8004d40 2024-04-16T16:21:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.154+0000 7fa3d6e00640 1 -- 172.21.15.45:0/2264208580 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d00cc4e0 con 0x7fa3b8004d40 2024-04-16T16:21:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.155+0000 7fa3d6e00640 1 -- 172.21.15.45:0/2264208580 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3d0075780 con 0x7fa3b8004d40 2024-04-16T16:21:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.156+0000 7fa3cd600640 1 -- 172.21.15.45:0/2264208580 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 55) ==== 91142+0+0 (secure 0 0 0) 0x7fa3c400f600 con 0x7fa3b8004d40 2024-04-16T16:21:02.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.157+0000 7fa3cd600640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fa3b006ca10 0x7fa3b006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.157+0000 7fa3cd600640 1 -- 172.21.15.45:0/2264208580 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6053+0+0 (secure 0 0 0) 0x7fa3c408c800 con 0x7fa3b8004d40 2024-04-16T16:21:02.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.157+0000 7fa3d5000640 1 -- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fa3b006ca10 msgr2=0x7fa3b006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/549890383 2024-04-16T16:21:02.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.157+0000 7fa3d5000640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fa3b006ca10 0x7fa3b006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:21:02.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.161+0000 7fa3cd600640 1 -- 172.21.15.45:0/2264208580 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa3c4019030 con 0x7fa3b8004d40 2024-04-16T16:21:02.241 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:02.242+0000 7f9f6da0d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:21:02.258 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:02.258+0000 7f7fa9c46040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:21:02.350 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:02.350+0000 7f9f6da0d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:21:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.357+0000 7fa3d6e00640 1 -- 172.21.15.45:0/2264208580 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa3d00b89c0 con 0x7fa3b8004d40 2024-04-16T16:21:02.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.358+0000 7fa3d5000640 1 -- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fa3b006ca10 msgr2=0x7fa3b006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/549890383 2024-04-16T16:21:02.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.358+0000 7fa3cd600640 1 -- 172.21.15.45:0/2264208580 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) ==== 100+0+11 (secure 0 0 0) 0x7fa3c405a890 con 0x7fa3b8004d40 2024-04-16T16:21:02.358 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:21:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.358+0000 7fa3d5000640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fa3b006ca10 0x7fa3b006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:21:02.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.360+0000 7fa3d6e00640 1 -- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fa3b006ca10 msgr2=0x7fa3b006eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:21:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.360+0000 7fa3d6e00640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fa3b006ca10 0x7fa3b006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.360+0000 7fa3d6e00640 1 -- 172.21.15.45:0/2264208580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b8004d40 msgr2=0x7fa3d0160170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.360+0000 7fa3d6e00640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b8004d40 0x7fa3d0160170 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4011540 tx=0x7fa3c4011a00 comp rx=0 tx=0).stop 2024-04-16T16:21:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.360+0000 7fa3d6e00640 1 -- 172.21.15.45:0/2264208580 shutdown_connections 2024-04-16T16:21:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.360+0000 7fa3d6e00640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3d0154920 0x7fa3d0154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.360+0000 7fa3d6e00640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d00cb960 0x7fa3d015f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.360+0000 7fa3d6e00640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fa3b006ca10 0x7fa3b006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.360+0000 7fa3d6e00640 1 --2- 172.21.15.45:0/2264208580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b8004d40 0x7fa3d0160170 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.360+0000 7fa3d6e00640 1 -- 172.21.15.45:0/2264208580 >> 172.21.15.45:0/2264208580 conn(0x7fa3d006cf90 msgr2=0x7fa3d00b42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.360+0000 7fa3d6e00640 1 -- 172.21.15.45:0/2264208580 shutdown_connections 2024-04-16T16:21:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.360+0000 7fa3d6e00640 1 -- 172.21.15.45:0/2264208580 wait complete. 2024-04-16T16:21:02.378 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:21:02.400 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:02.401+0000 7f9f6da0d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:21:02.428 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:02.428+0000 7f7fa9c46040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:21:02.525 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:02.526+0000 7f9f6da0d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:21:02.545 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:02.545+0000 7f7fa9c46040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:21:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.581+0000 7efd7e600640 1 Processor -- start 2024-04-16T16:21:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.582+0000 7efd7e600640 1 -- start start 2024-04-16T16:21:02.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.584+0000 7efd7e600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd780ba140 0x7efd780d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.584+0000 7efd7e600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efd780cfc20 0x7efd780d35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.584+0000 7efd7e600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd780d2600 con 0x7efd78162090 2024-04-16T16:21:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.584+0000 7efd7e600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7efd780ba8b0 con 0x7efd780cfc20 2024-04-16T16:21:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.584+0000 7efd7e600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efd780ba5d0 con 0x7efd780ba140 2024-04-16T16:21:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.584+0000 7efd7d200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7efd78162090 0x7efd780cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58704/0 (socket says 172.21.15.45:58704) 2024-04-16T16:21:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.584+0000 7efd7d200640 1 -- 172.21.15.45:0/417962041 learned_addr learned my addr 172.21.15.45:0/417962041 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.584+0000 7efd77400640 1 -- 172.21.15.45:0/417962041 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 6976159 0 0) 0x7efd780d2600 con 0x7efd78162090 2024-04-16T16:21:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.584+0000 7efd77400640 1 -- 172.21.15.45:0/417962041 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd60003520 con 0x7efd78162090 2024-04-16T16:21:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.584+0000 7efd77e00640 1 --2- 172.21.15.45:0/417962041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd780ba140 0x7efd780d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.584+0000 7efd7dc00640 1 --2- 172.21.15.45:0/417962041 >> v2:172.21.15.103:3300/0 conn(0x7efd780cfc20 0x7efd780d35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd77400640 1 -- 172.21.15.45:0/417962041 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 643773822 0 0) 0x7efd60003520 con 0x7efd78162090 2024-04-16T16:21:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd77400640 1 -- 172.21.15.45:0/417962041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd780ba140 msgr2=0x7efd780d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd77400640 1 --2- 172.21.15.45:0/417962041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd780ba140 0x7efd780d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd77400640 1 -- 172.21.15.45:0/417962041 >> v2:172.21.15.103:3300/0 conn(0x7efd780cfc20 msgr2=0x7efd780d35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd77400640 1 --2- 172.21.15.45:0/417962041 >> v2:172.21.15.103:3300/0 conn(0x7efd780cfc20 0x7efd780d35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd77400640 1 -- 172.21.15.45:0/417962041 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd780d28a0 con 0x7efd78162090 2024-04-16T16:21:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd77400640 1 -- 172.21.15.45:0/417962041 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7efd6c001670 con 0x7efd78162090 2024-04-16T16:21:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd77400640 1 -- 172.21.15.45:0/417962041 >> v1:172.21.15.45:6789/0 conn(0x7efd78162090 legacy=0x7efd780cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd77400640 1 --2- 172.21.15.45:0/417962041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd600039b0 0x7efd60003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd77400640 1 --2- 172.21.15.45:0/417962041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd600043c0 0x7efd600048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd77400640 1 --2- 172.21.15.45:0/417962041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd60004fc0 0x7efd6000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd77400640 1 -- 172.21.15.45:0/417962041 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efd780ba8b0 con 0x7efd60004fc0 2024-04-16T16:21:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd77400640 1 -- 172.21.15.45:0/417962041 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efd780ba5d0 con 0x7efd600043c0 2024-04-16T16:21:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd77400640 1 -- 172.21.15.45:0/417962041 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efd6000bf00 con 0x7efd600039b0 2024-04-16T16:21:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd7dc00640 1 --2- 172.21.15.45:0/417962041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd60004fc0 0x7efd6000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.585+0000 7efd7d200640 1 --2- 172.21.15.45:0/417962041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd600039b0 0x7efd60003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.586+0000 7efd77e00640 1 --2- 172.21.15.45:0/417962041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd600043c0 0x7efd600048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.586+0000 7efd7dc00640 1 -- 172.21.15.45:0/417962041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd600039b0 msgr2=0x7efd60003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.586+0000 7efd7dc00640 1 --2- 172.21.15.45:0/417962041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd600039b0 0x7efd60003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.586+0000 7efd7dc00640 1 -- 172.21.15.45:0/417962041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd600043c0 msgr2=0x7efd600048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.586+0000 7efd7dc00640 1 --2- 172.21.15.45:0/417962041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd600043c0 0x7efd600048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.586+0000 7efd7dc00640 1 -- 172.21.15.45:0/417962041 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd60003520 con 0x7efd60004fc0 2024-04-16T16:21:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.586+0000 7efd77e00640 1 --2- 172.21.15.45:0/417962041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd600043c0 0x7efd600048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.586+0000 7efd7d200640 1 --2- 172.21.15.45:0/417962041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd600039b0 0x7efd60003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:21:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.587+0000 7efd7dc00640 1 --2- 172.21.15.45:0/417962041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd60004fc0 0x7efd6000b9c0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7efd7808ce70 tx=0x7efd60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19c8735586f67350 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.587+0000 7efd77400640 1 -- 172.21.15.45:0/417962041 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70039070 con 0x7efd60004fc0 2024-04-16T16:21:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.587+0000 7efd77400640 1 -- 172.21.15.45:0/417962041 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd7002cca0 con 0x7efd60004fc0 2024-04-16T16:21:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.587+0000 7efd77400640 1 -- 172.21.15.45:0/417962041 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70009be0 con 0x7efd60004fc0 2024-04-16T16:21:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.588+0000 7efd7e600640 1 -- 172.21.15.45:0/417962041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd60004fc0 msgr2=0x7efd6000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.588+0000 7efd7e600640 1 --2- 172.21.15.45:0/417962041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd60004fc0 0x7efd6000b9c0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7efd7808ce70 tx=0x7efd60005830 comp rx=0 tx=0).stop 2024-04-16T16:21:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.588+0000 7efd7dc00640 1 -- 172.21.15.45:0/417962041 reap_dead start 2024-04-16T16:21:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.588+0000 7efd7e600640 1 -- 172.21.15.45:0/417962041 shutdown_connections 2024-04-16T16:21:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.588+0000 7efd7e600640 1 -- 172.21.15.45:0/417962041 >> 172.21.15.45:0/417962041 conn(0x7efd780d1940 msgr2=0x7efd780d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.588+0000 7efd7e600640 1 -- 172.21.15.45:0/417962041 shutdown_connections 2024-04-16T16:21:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.588+0000 7efd7e600640 1 -- 172.21.15.45:0/417962041 wait complete. 2024-04-16T16:21:02.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.590+0000 7efd7e600640 1 Processor -- start 2024-04-16T16:21:02.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.590+0000 7efd7e600640 1 -- start start 2024-04-16T16:21:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.591+0000 7efd7e600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd78153b80 0x7efd781573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.591+0000 7efd7e600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd781f4af0 0x7efd781545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.591+0000 7efd7e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd780b5780 0x7efd780b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.591+0000 7efd7e600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efd781625f0 con 0x7efd780b5780 2024-04-16T16:21:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.591+0000 7efd7e600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efd78162470 con 0x7efd78153b80 2024-04-16T16:21:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.591+0000 7efd7e600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efd780d5520 con 0x7efd781f4af0 2024-04-16T16:21:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.591+0000 7efd7dc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd780b5780 0x7efd780b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.591+0000 7efd7dc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd780b5780 0x7efd780b5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34884/0 (socket says 172.21.15.45:34884) 2024-04-16T16:21:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.591+0000 7efd7dc00640 1 -- 172.21.15.45:0/2408331527 learned_addr learned my addr 172.21.15.45:0/2408331527 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.592+0000 7efd77e00640 1 --2- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd781f4af0 0x7efd781545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.592+0000 7efd7d200640 1 --2- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd78153b80 0x7efd781573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.592+0000 7efd7dc00640 1 -- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd781f4af0 msgr2=0x7efd781545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.592+0000 7efd7dc00640 1 --2- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd781f4af0 0x7efd781545d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.592+0000 7efd7dc00640 1 -- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd78153b80 msgr2=0x7efd781573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.592+0000 7efd7dc00640 1 --2- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd78153b80 0x7efd781573d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.592+0000 7efd7dc00640 1 -- 172.21.15.45:0/2408331527 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd78154b10 con 0x7efd780b5780 2024-04-16T16:21:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.592+0000 7efd7dc00640 1 --2- 172.21.15.45:0/2408331527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd780b5780 0x7efd780b5c00 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7efd7002c710 tx=0x7efd70004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.592+0000 7efd74c00640 1 -- 172.21.15.45:0/2408331527 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70039070 con 0x7efd780b5780 2024-04-16T16:21:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.592+0000 7efd74c00640 1 -- 172.21.15.45:0/2408331527 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd70044070 con 0x7efd780b5780 2024-04-16T16:21:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.592+0000 7efd74c00640 1 -- 172.21.15.45:0/2408331527 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70009be0 con 0x7efd780b5780 2024-04-16T16:21:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.592+0000 7efd7e600640 1 -- 172.21.15.45:0/2408331527 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd78154de0 con 0x7efd780b5780 2024-04-16T16:21:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.593+0000 7efd7e600640 1 -- 172.21.15.45:0/2408331527 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd780cd4a0 con 0x7efd780b5780 2024-04-16T16:21:02.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.594+0000 7efd7e600640 1 -- 172.21.15.45:0/2408331527 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd78075780 con 0x7efd780b5780 2024-04-16T16:21:02.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.596+0000 7efd74c00640 1 -- 172.21.15.45:0/2408331527 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 55) ==== 91142+0+0 (secure 0 0 0) 0x7efd7003e810 con 0x7efd780b5780 2024-04-16T16:21:02.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.596+0000 7efd74c00640 1 --2- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7efd5806ca40 0x7efd5806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:02.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.597+0000 7efd74c00640 1 -- 172.21.15.45:0/2408331527 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6053+0+0 (secure 0 0 0) 0x7efd700adbe0 con 0x7efd780b5780 2024-04-16T16:21:02.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.597+0000 7efd7d200640 1 -- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7efd5806ca40 msgr2=0x7efd5806ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/549890383 2024-04-16T16:21:02.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.597+0000 7efd7d200640 1 --2- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7efd5806ca40 0x7efd5806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:21:02.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.601+0000 7efd74c00640 1 -- 172.21.15.45:0/2408331527 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efd7007bda0 con 0x7efd780b5780 2024-04-16T16:21:02.602 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:02.602+0000 7f7fa9c46040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:21:02.634 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:02.634+0000 7f9f6da0d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:21:02.736 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:02.736+0000 7f7fa9c46040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:21:02.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.798+0000 7efd7d200640 1 -- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7efd5806ca40 msgr2=0x7efd5806ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/549890383 2024-04-16T16:21:02.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.798+0000 7efd7d200640 1 --2- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7efd5806ca40 0x7efd5806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:21:02.810 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:02.810+0000 7f9f6da0d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:21:02.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.814+0000 7efd7e600640 1 -- 172.21.15.45:0/2408331527 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efd780b71c0 con 0x7efd780b5780 2024-04-16T16:21:02.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.815+0000 7efd74c00640 1 -- 172.21.15.45:0/2408331527 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v129) ==== 80+0+478 (secure 0 0 0) 0x7efd7007b6a0 con 0x7efd780b5780 2024-04-16T16:21:02.815 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/1120913661 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:02.815 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/1120913661 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:02.815 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/4103173879 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:02.816 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/4177084908 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:02.816 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1961421675 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:02.816 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2190202958 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:02.816 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3679729711 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:02.816 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1543358977 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:02.816 INFO:teuthology.orchestra.run.smithi045.stderr:listed 8 entries 2024-04-16T16:21:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.817+0000 7efd7e600640 1 -- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7efd5806ca40 msgr2=0x7efd5806ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:21:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.817+0000 7efd7e600640 1 --2- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7efd5806ca40 0x7efd5806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.817+0000 7efd7e600640 1 -- 172.21.15.45:0/2408331527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd780b5780 msgr2=0x7efd780b5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.817+0000 7efd7e600640 1 --2- 172.21.15.45:0/2408331527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd780b5780 0x7efd780b5c00 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7efd7002c710 tx=0x7efd70004ab0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.818+0000 7efd7e600640 1 -- 172.21.15.45:0/2408331527 shutdown_connections 2024-04-16T16:21:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.818+0000 7efd7e600640 1 --2- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7efd5806ca40 0x7efd5806ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.818+0000 7efd7e600640 1 --2- 172.21.15.45:0/2408331527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd780b5780 0x7efd780b5c00 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.818+0000 7efd7e600640 1 --2- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd781f4af0 0x7efd781545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.818+0000 7efd7e600640 1 --2- 172.21.15.45:0/2408331527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd78153b80 0x7efd781573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:02.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.818+0000 7efd7e600640 1 -- 172.21.15.45:0/2408331527 >> 172.21.15.45:0/2408331527 conn(0x7efd780d1940 msgr2=0x7efd78161110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:02.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.818+0000 7efd7e600640 1 -- 172.21.15.45:0/2408331527 shutdown_connections 2024-04-16T16:21:02.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:02.818+0000 7efd7e600640 1 -- 172.21.15.45:0/2408331527 wait complete. 2024-04-16T16:21:02.836 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-16T16:21:02.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:02.837 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-16T16:21:02.850 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:02.850+0000 7f7fa9c46040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:21:02.858 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:02.859+0000 7f9f6da0d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:21:02.930 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:21:02.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:02.931 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-16T16:21:02.997 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:02.998+0000 7f9f6da0d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:21:03.022 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:21:03.022 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:21:03.023 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:21:03.039 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:03.040+0000 7f7fa9c46040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:21:03.076 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:21:03.076 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:21:03.092 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:03.093+0000 7f7fa9c46040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:21:03.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:21:03.239 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:03.239+0000 7f7fa9c46040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:21:03.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.324+0000 7f3e1d600640 1 Processor -- start 2024-04-16T16:21:03.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.324+0000 7f3e1d600640 1 -- start start 2024-04-16T16:21:03.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.325+0000 7f3e1d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e180b6c00 0x7f3e180cf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:03.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.326+0000 7f3e1d600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e180d2e20 con 0x7f3e1815edf0 2024-04-16T16:21:03.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.326+0000 7f3e1d600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e180d3100 con 0x7f3e180cf880 2024-04-16T16:21:03.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.326+0000 7f3e1d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3e180d3be0 con 0x7f3e180b6c00 2024-04-16T16:21:03.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.326+0000 7f3e16a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3e1815edf0 0x7f3e180b64f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36056/0 (socket says 172.21.15.45:36056) 2024-04-16T16:21:03.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.326+0000 7f3e16a00640 1 -- 172.21.15.45:0/849474906 learned_addr learned my addr 172.21.15.45:0/849474906 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:03.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.326+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/849474906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e180b6c00 0x7f3e180cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:03.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e16000640 1 -- 172.21.15.45:0/849474906 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2445774157 0 0) 0x7f3e180d2e20 con 0x7f3e1815edf0 2024-04-16T16:21:03.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e16000640 1 -- 172.21.15.45:0/849474906 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dfc003520 con 0x7f3e1815edf0 2024-04-16T16:21:03.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e16000640 1 -- 172.21.15.45:0/849474906 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3929816067 0 0) 0x7f3dfc003520 con 0x7f3e1815edf0 2024-04-16T16:21:03.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e16000640 1 -- 172.21.15.45:0/849474906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e180b6c00 msgr2=0x7f3e180cf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:03.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e16000640 1 --2- 172.21.15.45:0/849474906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e180b6c00 0x7f3e180cf340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:03.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e16000640 1 -- 172.21.15.45:0/849474906 >> v1:172.21.15.103:6789/0 conn(0x7f3e180cf880 legacy=0x7f3e180b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:03.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e16000640 1 -- 172.21.15.45:0/849474906 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e180d33e0 con 0x7f3e1815edf0 2024-04-16T16:21:03.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e16000640 1 -- 172.21.15.45:0/849474906 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f3e08001670 con 0x7f3e1815edf0 2024-04-16T16:21:03.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e16000640 1 -- 172.21.15.45:0/849474906 >> v1:172.21.15.45:6789/0 conn(0x7f3e1815edf0 legacy=0x7f3e180b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e16000640 1 --2- 172.21.15.45:0/849474906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dfc0039e0 0x7f3dfc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e16000640 1 --2- 172.21.15.45:0/849474906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc004800 0x7f3dfc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e16000640 1 --2- 172.21.15.45:0/849474906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc0051c0 0x7f3dfc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e16000640 1 -- 172.21.15.45:0/849474906 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3e180d3be0 con 0x7f3dfc004800 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e16000640 1 -- 172.21.15.45:0/849474906 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc00c010 con 0x7f3dfc0051c0 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e16000640 1 -- 172.21.15.45:0/849474906 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3dfc00c210 con 0x7f3dfc0039e0 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/849474906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e180b6c00 0x7f3e180cf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.327+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/849474906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc004800 0x7f3dfc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.328+0000 7f3e16a00640 1 --2- 172.21.15.45:0/849474906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dfc0039e0 0x7f3dfc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.328+0000 7f3e17400640 1 --2- 172.21.15.45:0/849474906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc0051c0 0x7f3dfc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.328+0000 7f3e0fe00640 1 -- 172.21.15.45:0/849474906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dfc0039e0 msgr2=0x7f3dfc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.328+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/849474906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dfc0039e0 0x7f3dfc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.328+0000 7f3e0fe00640 1 -- 172.21.15.45:0/849474906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc0051c0 msgr2=0x7f3dfc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.328+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/849474906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc0051c0 0x7f3dfc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.328+0000 7f3e0fe00640 1 -- 172.21.15.45:0/849474906 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dfc003520 con 0x7f3dfc004800 2024-04-16T16:21:03.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.328+0000 7f3e17400640 1 --2- 172.21.15.45:0/849474906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc0051c0 0x7f3dfc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:03.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.329+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/849474906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc004800 0x7f3dfc004c80 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f3e1808dff0 tx=0x7f3dfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e50547a7e74d44f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:03.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.329+0000 7f3e16000640 1 -- 172.21.15.45:0/849474906 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e000097d0 con 0x7f3dfc004800 2024-04-16T16:21:03.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.329+0000 7f3e16000640 1 -- 172.21.15.45:0/849474906 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e00004030 con 0x7f3dfc004800 2024-04-16T16:21:03.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.329+0000 7f3e16000640 1 -- 172.21.15.45:0/849474906 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e00004400 con 0x7f3dfc004800 2024-04-16T16:21:03.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.330+0000 7f3e1d600640 1 -- 172.21.15.45:0/849474906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc004800 msgr2=0x7f3dfc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:03.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.330+0000 7f3e1d600640 1 --2- 172.21.15.45:0/849474906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dfc004800 0x7f3dfc004c80 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f3e1808dff0 tx=0x7f3dfc005830 comp rx=0 tx=0).stop 2024-04-16T16:21:03.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.330+0000 7f3e17400640 1 -- 172.21.15.45:0/849474906 reap_dead start 2024-04-16T16:21:03.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.330+0000 7f3e1d600640 1 -- 172.21.15.45:0/849474906 shutdown_connections 2024-04-16T16:21:03.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.330+0000 7f3e1d600640 1 -- 172.21.15.45:0/849474906 >> 172.21.15.45:0/849474906 conn(0x7f3e18073690 msgr2=0x7f3e180d14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:03.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.330+0000 7f3e1d600640 1 -- 172.21.15.45:0/849474906 shutdown_connections 2024-04-16T16:21:03.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.330+0000 7f3e1d600640 1 -- 172.21.15.45:0/849474906 wait complete. 2024-04-16T16:21:03.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.333+0000 7f3e1d600640 1 Processor -- start 2024-04-16T16:21:03.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.333+0000 7f3e1d600640 1 -- start start 2024-04-16T16:21:03.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.335+0000 7f3e1d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc004800 0x7f3e181f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:03.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.335+0000 7f3e1d600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e180c98c0 0x7f3e180cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:03.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.335+0000 7f3e1d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e180ca020 0x7f3e180ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.335+0000 7f3e1d600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3e180cf340 con 0x7f3e180c98c0 2024-04-16T16:21:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.335+0000 7f3e1d600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3e180d3750 con 0x7f3dfc004800 2024-04-16T16:21:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.335+0000 7f3e1d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3e180cf4c0 con 0x7f3e180ca020 2024-04-16T16:21:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.335+0000 7f3e0fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e180c98c0 0x7f3e180cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.335+0000 7f3e0fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e180c98c0 0x7f3e180cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48314/0 (socket says 172.21.15.45:48314) 2024-04-16T16:21:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.335+0000 7f3e0fe00640 1 -- 172.21.15.45:0/1574249441 learned_addr learned my addr 172.21.15.45:0/1574249441 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.335+0000 7f3e16a00640 1 --2- 172.21.15.45:0/1574249441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc004800 0x7f3e181f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.335+0000 7f3e0fe00640 1 -- 172.21.15.45:0/1574249441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e180ca020 msgr2=0x7f3e180ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.335+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/1574249441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e180ca020 0x7f3e180ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.335+0000 7f3e0fe00640 1 -- 172.21.15.45:0/1574249441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc004800 msgr2=0x7f3e181f4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.335+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/1574249441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc004800 0x7f3e181f4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.335+0000 7f3e0fe00640 1 -- 172.21.15.45:0/1574249441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e180ca5e0 con 0x7f3e180c98c0 2024-04-16T16:21:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.336+0000 7f3e0fe00640 1 --2- 172.21.15.45:0/1574249441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e180c98c0 0x7f3e180cdb50 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f3e0002c380 tx=0x7f3e00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:03.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.336+0000 7f3e0ea00640 1 -- 172.21.15.45:0/1574249441 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e00042070 con 0x7f3e180c98c0 2024-04-16T16:21:03.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.336+0000 7f3e0ea00640 1 -- 172.21.15.45:0/1574249441 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e0002cb20 con 0x7f3e180c98c0 2024-04-16T16:21:03.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.336+0000 7f3e0ea00640 1 -- 172.21.15.45:0/1574249441 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e00003610 con 0x7f3e180c98c0 2024-04-16T16:21:03.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.336+0000 7f3e1d600640 1 -- 172.21.15.45:0/1574249441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e180ca8c0 con 0x7f3e180c98c0 2024-04-16T16:21:03.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.336+0000 7f3e1d600640 1 -- 172.21.15.45:0/1574249441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e180cae80 con 0x7f3e180c98c0 2024-04-16T16:21:03.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.337+0000 7f3e1d600640 1 -- 172.21.15.45:0/1574249441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e180cb260 con 0x7f3e180c98c0 2024-04-16T16:21:03.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.338+0000 7f3e0ea00640 1 -- 172.21.15.45:0/1574249441 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 55) ==== 91142+0+0 (secure 0 0 0) 0x7f3e00045070 con 0x7f3e180c98c0 2024-04-16T16:21:03.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.339+0000 7f3e0ea00640 1 --2- 172.21.15.45:0/1574249441 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f3df406ca10 0x7f3df406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:03.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.339+0000 7f3e0ea00640 1 -- 172.21.15.45:0/1574249441 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6053+0+0 (secure 0 0 0) 0x7f3e000adde0 con 0x7f3e180c98c0 2024-04-16T16:21:03.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.340+0000 7f3e16a00640 1 -- 172.21.15.45:0/1574249441 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f3df406ca10 msgr2=0x7f3df406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/549890383 2024-04-16T16:21:03.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.340+0000 7f3e16a00640 1 --2- 172.21.15.45:0/1574249441 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f3df406ca10 0x7f3df406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:21:03.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.344+0000 7f3e0ea00640 1 -- 172.21.15.45:0/1574249441 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3e180cb260 con 0x7f3e180c98c0 2024-04-16T16:21:03.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.541+0000 7f3e16a00640 1 -- 172.21.15.45:0/1574249441 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f3df406ca10 msgr2=0x7f3df406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/549890383 2024-04-16T16:21:03.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.541+0000 7f3e16a00640 1 --2- 172.21.15.45:0/1574249441 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f3df406ca10 0x7f3df406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:21:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.546+0000 7f3e1d600640 1 -- 172.21.15.45:0/1574249441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e18075780 con 0x7f3e180c98c0 2024-04-16T16:21:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.548+0000 7f3e0ea00640 1 -- 172.21.15.45:0/1574249441 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) ==== 74+0+17188 (secure 0 0 0) 0x7f3e0007bf90 con 0x7f3e180c98c0 2024-04-16T16:21:03.548 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:03.548 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":129,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:20:50.935396+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:20:31.681227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-16T16:20:32.790980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6833/1120913661":"2024-04-17T16:20:50.935324+0000","172.21.15.103:6832/1120913661":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/4103173879":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/4177084908":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1961421675":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/2190202958":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/3679729711":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1543358977":"2024-04-17T16:20:50.935324+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:21:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.550+0000 7f3e1d600640 1 -- 172.21.15.45:0/1574249441 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f3df406ca10 msgr2=0x7f3df406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:21:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.550+0000 7f3e1d600640 1 --2- 172.21.15.45:0/1574249441 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f3df406ca10 0x7f3df406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.550+0000 7f3e1d600640 1 -- 172.21.15.45:0/1574249441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e180c98c0 msgr2=0x7f3e180cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.550+0000 7f3e1d600640 1 --2- 172.21.15.45:0/1574249441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e180c98c0 0x7f3e180cdb50 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f3e0002c380 tx=0x7f3e00004730 comp rx=0 tx=0).stop 2024-04-16T16:21:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.551+0000 7f3e1d600640 1 -- 172.21.15.45:0/1574249441 shutdown_connections 2024-04-16T16:21:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.551+0000 7f3e1d600640 1 --2- 172.21.15.45:0/1574249441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e180ca020 0x7f3e180ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.551+0000 7f3e1d600640 1 --2- 172.21.15.45:0/1574249441 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f3df406ca10 0x7f3df406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.551+0000 7f3e1d600640 1 --2- 172.21.15.45:0/1574249441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e180c98c0 0x7f3e180cdb50 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.551+0000 7f3e1d600640 1 --2- 172.21.15.45:0/1574249441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dfc004800 0x7f3e181f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.551+0000 7f3e1d600640 1 -- 172.21.15.45:0/1574249441 >> 172.21.15.45:0/1574249441 conn(0x7f3e18073690 msgr2=0x7f3e180d4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:03.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.551+0000 7f3e1d600640 1 -- 172.21.15.45:0/1574249441 shutdown_connections 2024-04-16T16:21:03.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.551+0000 7f3e1d600640 1 -- 172.21.15.45:0/1574249441 wait complete. 2024-04-16T16:21:03.569 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:20:31.681227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:20:32.790980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:21:03.570 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:21:03.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.773+0000 7f508d000640 1 Processor -- start 2024-04-16T16:21:03.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.773+0000 7f508d000640 1 -- start start 2024-04-16T16:21:03.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.774+0000 7f508d000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f508815ea70 0x7f50880d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:03.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.774+0000 7f508d000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f50880d4370 0x7f50880cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:03.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.774+0000 7f508d000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f508815ee50 con 0x7f50880cfa70 2024-04-16T16:21:03.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.774+0000 7f508d000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f50880d1a10 con 0x7f50880d4370 2024-04-16T16:21:03.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.774+0000 7f508d000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f50880b6480 con 0x7f508815ea70 2024-04-16T16:21:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.775+0000 7f5087400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f508815ea70 0x7f50880d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.775+0000 7f5087400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f508815ea70 0x7f50880d11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:59674/0 (socket says 172.21.15.45:59674) 2024-04-16T16:21:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.775+0000 7f5087400640 1 -- 172.21.15.45:0/3658970740 learned_addr learned my addr 172.21.15.45:0/3658970740 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.775+0000 7f5086a00640 1 --2- 172.21.15.45:0/3658970740 >> v2:172.21.15.103:3300/0 conn(0x7f50880d4370 0x7f50880cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.776+0000 7f5087400640 1 -- 172.21.15.45:0/3658970740 >> v2:172.21.15.103:3300/0 conn(0x7f50880d4370 msgr2=0x7f50880cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.776+0000 7f5087400640 1 --2- 172.21.15.45:0/3658970740 >> v2:172.21.15.103:3300/0 conn(0x7f50880d4370 0x7f50880cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.776+0000 7f5087400640 1 -- 172.21.15.45:0/3658970740 >> v1:172.21.15.103:6789/0 conn(0x7f50880cfa70 legacy=0x7f50880d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.776+0000 7f5087400640 1 -- 172.21.15.45:0/3658970740 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50880b6760 con 0x7f508815ea70 2024-04-16T16:21:03.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.776+0000 7f5086a00640 1 --2- 172.21.15.45:0/3658970740 >> v2:172.21.15.103:3300/0 conn(0x7f50880d4370 0x7f50880cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:21:03.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.777+0000 7f5087400640 1 --2- 172.21.15.45:0/3658970740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f508815ea70 0x7f50880d11a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5078009870 tx=0x7f507802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f9c767b33dbd447 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:03.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.777+0000 7f5086000640 1 -- 172.21.15.45:0/3658970740 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5078034070 con 0x7f508815ea70 2024-04-16T16:21:03.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.777+0000 7f5086000640 1 -- 172.21.15.45:0/3658970740 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5078030840 con 0x7f508815ea70 2024-04-16T16:21:03.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.777+0000 7f5086000640 1 -- 172.21.15.45:0/3658970740 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5078030c10 con 0x7f508815ea70 2024-04-16T16:21:03.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.778+0000 7f508d000640 1 -- 172.21.15.45:0/3658970740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f508815ea70 msgr2=0x7f50880d11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:03.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.778+0000 7f508d000640 1 --2- 172.21.15.45:0/3658970740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f508815ea70 0x7f50880d11a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5078009870 tx=0x7f507802b980 comp rx=0 tx=0).stop 2024-04-16T16:21:03.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.778+0000 7f508d000640 1 -- 172.21.15.45:0/3658970740 shutdown_connections 2024-04-16T16:21:03.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.778+0000 7f508d000640 1 --2- 172.21.15.45:0/3658970740 >> v2:172.21.15.103:3300/0 conn(0x7f50880d4370 0x7f50880cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:03.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.778+0000 7f508d000640 1 --2- 172.21.15.45:0/3658970740 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f508815ea70 0x7f50880d11a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:03.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.778+0000 7f508d000640 1 -- 172.21.15.45:0/3658970740 >> 172.21.15.45:0/3658970740 conn(0x7f508806d3b0 msgr2=0x7f5088073060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:03.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.778+0000 7f508d000640 1 -- 172.21.15.45:0/3658970740 shutdown_connections 2024-04-16T16:21:03.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.778+0000 7f508d000640 1 -- 172.21.15.45:0/3658970740 wait complete. 2024-04-16T16:21:03.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.780+0000 7f508d000640 1 Processor -- start 2024-04-16T16:21:03.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.780+0000 7f508d000640 1 -- start start 2024-04-16T16:21:03.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.781+0000 7f508d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50880cfa70 0x7f50880ab2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:03.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.781+0000 7f508d000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50880d4370 0x7f50880ab800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:03.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.781+0000 7f508d000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f508815ea70 0x7f50881f8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:03.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.781+0000 7f508d000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f50880b6f60 con 0x7f50880cfa70 2024-04-16T16:21:03.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.781+0000 7f508d000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f50880d16e0 con 0x7f508815ea70 2024-04-16T16:21:03.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.781+0000 7f508d000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5088161bb0 con 0x7f50880d4370 2024-04-16T16:21:03.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.782+0000 7f5087400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50880cfa70 0x7f50880ab2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:03.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.782+0000 7f5087400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50880cfa70 0x7f50880ab2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48328/0 (socket says 172.21.15.45:48328) 2024-04-16T16:21:03.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.782+0000 7f5087400640 1 -- 172.21.15.45:0/3362571391 learned_addr learned my addr 172.21.15.45:0/3362571391 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:03.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.782+0000 7f5086a00640 1 --2- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50880d4370 0x7f50880ab800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:03.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.782+0000 7f5087e00640 1 --2- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f508815ea70 0x7f50881f8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:03.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.782+0000 7f5087400640 1 -- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50880d4370 msgr2=0x7f50880ab800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:03.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.782+0000 7f5087400640 1 --2- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50880d4370 0x7f50880ab800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:03.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.782+0000 7f5087400640 1 -- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f508815ea70 msgr2=0x7f50881f8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:03.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.782+0000 7f5087400640 1 --2- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f508815ea70 0x7f50881f8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:03.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.783+0000 7f5087400640 1 -- 172.21.15.45:0/3362571391 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50880c90a0 con 0x7f50880cfa70 2024-04-16T16:21:03.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.783+0000 7f5087400640 1 --2- 172.21.15.45:0/3362571391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50880cfa70 0x7f50880ab2c0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f5078000c00 tx=0x7f50780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:03.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.783+0000 7f507f400640 1 -- 172.21.15.45:0/3362571391 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5078034040 con 0x7f50880cfa70 2024-04-16T16:21:03.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.783+0000 7f507f400640 1 -- 172.21.15.45:0/3362571391 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5078039d90 con 0x7f50880cfa70 2024-04-16T16:21:03.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.783+0000 7f508d000640 1 -- 172.21.15.45:0/3362571391 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50880c9340 con 0x7f50880cfa70 2024-04-16T16:21:03.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.783+0000 7f507f400640 1 -- 172.21.15.45:0/3362571391 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5078043610 con 0x7f50880cfa70 2024-04-16T16:21:03.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.783+0000 7f508d000640 1 -- 172.21.15.45:0/3362571391 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50880c9900 con 0x7f50880cfa70 2024-04-16T16:21:03.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.785+0000 7f508d000640 1 -- 172.21.15.45:0/3362571391 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5088075780 con 0x7f50880cfa70 2024-04-16T16:21:03.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.786+0000 7f507f400640 1 -- 172.21.15.45:0/3362571391 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 55) ==== 91142+0+0 (secure 0 0 0) 0x7f507804a060 con 0x7f50880cfa70 2024-04-16T16:21:03.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.787+0000 7f507f400640 1 --2- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f506806ca10 0x7f506806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:03.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.788+0000 7f507f400640 1 -- 172.21.15.45:0/3362571391 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6053+0+0 (secure 0 0 0) 0x7f50780ade90 con 0x7f50880cfa70 2024-04-16T16:21:03.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.788+0000 7f5086a00640 1 -- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f506806ca10 msgr2=0x7f506806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/549890383 2024-04-16T16:21:03.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.788+0000 7f5086a00640 1 --2- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f506806ca10 0x7f506806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:21:03.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.792+0000 7f507f400640 1 -- 172.21.15.45:0/3362571391 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f507803e070 con 0x7f50880cfa70 2024-04-16T16:21:03.836 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:03.836+0000 7f9f6da0d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:21:03.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.989+0000 7f5086a00640 1 -- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f506806ca10 msgr2=0x7f506806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/549890383 2024-04-16T16:21:03.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.989+0000 7f5086a00640 1 --2- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f506806ca10 0x7f506806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:21:03.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.997+0000 7f508d000640 1 -- 172.21.15.45:0/3362571391 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f50880b3fc0 con 0x7f50880cfa70 2024-04-16T16:21:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:03.998+0000 7f507f400640 1 -- 172.21.15.45:0/3362571391 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) ==== 93+0+10 (secure 0 0 0) 0x7f507807bfd0 con 0x7f50880cfa70 2024-04-16T16:21:03.998 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:21:04.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.002+0000 7f508d000640 1 -- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f506806ca10 msgr2=0x7f506806eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:21:04.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.002+0000 7f508d000640 1 --2- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f506806ca10 0x7f506806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.002+0000 7f508d000640 1 -- 172.21.15.45:0/3362571391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50880cfa70 msgr2=0x7f50880ab2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:04.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.002+0000 7f508d000640 1 --2- 172.21.15.45:0/3362571391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50880cfa70 0x7f50880ab2c0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f5078000c00 tx=0x7f50780046c0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.002+0000 7f508d000640 1 -- 172.21.15.45:0/3362571391 shutdown_connections 2024-04-16T16:21:04.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.002+0000 7f508d000640 1 --2- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f508815ea70 0x7f50881f8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.002+0000 7f508d000640 1 --2- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50880d4370 0x7f50880ab800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.002+0000 7f508d000640 1 --2- 172.21.15.45:0/3362571391 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7f506806ca10 0x7f506806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.002+0000 7f508d000640 1 --2- 172.21.15.45:0/3362571391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50880cfa70 0x7f50880ab2c0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.002+0000 7f508d000640 1 -- 172.21.15.45:0/3362571391 >> 172.21.15.45:0/3362571391 conn(0x7f508806d3b0 msgr2=0x7f50880b79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:04.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.002+0000 7f508d000640 1 -- 172.21.15.45:0/3362571391 shutdown_connections 2024-04-16T16:21:04.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.002+0000 7f508d000640 1 -- 172.21.15.45:0/3362571391 wait complete. 2024-04-16T16:21:04.020 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:21:04.111 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:04.111+0000 7f7fa9c46040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:21:04.143 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:04.143+0000 7f9f6da0d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:21:04.201 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:04.201+0000 7f9f6da0d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:21:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.237+0000 7fb775c00640 1 Processor -- start 2024-04-16T16:21:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.238+0000 7fb775c00640 1 -- start start 2024-04-16T16:21:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.239+0000 7fb775c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb770146120 0x7fb77006b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.239+0000 7fb775c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7700cdf70 0x7fb77006be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.239+0000 7fb775c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb7700cf5a0 0x7fb7700cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.239+0000 7fb775c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb770160f60 con 0x7fb770146120 2024-04-16T16:21:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.239+0000 7fb775c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb770160390 con 0x7fb7700cf5a0 2024-04-16T16:21:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.239+0000 7fb775c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb770160630 con 0x7fb7700cdf70 2024-04-16T16:21:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.240+0000 7fb76fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb770146120 0x7fb77006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.240+0000 7fb76fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb770146120 0x7fb77006b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48332/0 (socket says 172.21.15.45:48332) 2024-04-16T16:21:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.240+0000 7fb76fe00640 1 -- 172.21.15.45:0/3658792674 learned_addr learned my addr 172.21.15.45:0/3658792674 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.240+0000 7fb76f400640 1 --2- 172.21.15.45:0/3658792674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7700cdf70 0x7fb77006be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.240+0000 7fb775200640 1 --2- 172.21.15.45:0/3658792674 >> v2:172.21.15.103:3300/0 conn(0x7fb7700cf5a0 0x7fb7700cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.240+0000 7fb76fe00640 1 -- 172.21.15.45:0/3658792674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7700cdf70 msgr2=0x7fb77006be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.240+0000 7fb76fe00640 1 --2- 172.21.15.45:0/3658792674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7700cdf70 0x7fb77006be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.240+0000 7fb76fe00640 1 -- 172.21.15.45:0/3658792674 >> v2:172.21.15.103:3300/0 conn(0x7fb7700cf5a0 msgr2=0x7fb7700cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.240+0000 7fb76fe00640 1 --2- 172.21.15.45:0/3658792674 >> v2:172.21.15.103:3300/0 conn(0x7fb7700cf5a0 0x7fb7700cfa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.240+0000 7fb76fe00640 1 -- 172.21.15.45:0/3658792674 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb770160910 con 0x7fb770146120 2024-04-16T16:21:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.241+0000 7fb76fe00640 1 --2- 172.21.15.45:0/3658792674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb770146120 0x7fb77006b930 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fb7700b6c90 tx=0x7fb76002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a05fcf3862be3fe1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.242+0000 7fb76ea00640 1 -- 172.21.15.45:0/3658792674 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb760035020 con 0x7fb770146120 2024-04-16T16:21:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.242+0000 7fb76ea00640 1 -- 172.21.15.45:0/3658792674 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb76002d040 con 0x7fb770146120 2024-04-16T16:21:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.242+0000 7fb76ea00640 1 -- 172.21.15.45:0/3658792674 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb760031a80 con 0x7fb770146120 2024-04-16T16:21:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.242+0000 7fb775c00640 1 -- 172.21.15.45:0/3658792674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb770146120 msgr2=0x7fb77006b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.242+0000 7fb775c00640 1 --2- 172.21.15.45:0/3658792674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb770146120 0x7fb77006b930 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fb7700b6c90 tx=0x7fb76002b900 comp rx=0 tx=0).stop 2024-04-16T16:21:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.243+0000 7fb775c00640 1 -- 172.21.15.45:0/3658792674 shutdown_connections 2024-04-16T16:21:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.243+0000 7fb775c00640 1 --2- 172.21.15.45:0/3658792674 >> v2:172.21.15.103:3300/0 conn(0x7fb7700cf5a0 0x7fb7700cfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.243+0000 7fb775c00640 1 --2- 172.21.15.45:0/3658792674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7700cdf70 0x7fb77006be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.243+0000 7fb775c00640 1 --2- 172.21.15.45:0/3658792674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb770146120 0x7fb77006b930 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.243+0000 7fb775c00640 1 -- 172.21.15.45:0/3658792674 >> 172.21.15.45:0/3658792674 conn(0x7fb77006f120 msgr2=0x7fb77006f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.243+0000 7fb775c00640 1 -- 172.21.15.45:0/3658792674 shutdown_connections 2024-04-16T16:21:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.243+0000 7fb775c00640 1 -- 172.21.15.45:0/3658792674 wait complete. 2024-04-16T16:21:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.245+0000 7fb775c00640 1 Processor -- start 2024-04-16T16:21:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.245+0000 7fb775c00640 1 -- start start 2024-04-16T16:21:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.247+0000 7fb775c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7700cdf70 0x7fb770153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.247+0000 7fb775c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7700cf5a0 0x7fb7700b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.247+0000 7fb775c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb770146120 0x7fb77013ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.247+0000 7fb775c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb7700d00f0 con 0x7fb7700cdf70 2024-04-16T16:21:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.247+0000 7fb775c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb7700cff70 con 0x7fb7700cf5a0 2024-04-16T16:21:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.247+0000 7fb775c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb770073010 con 0x7fb770146120 2024-04-16T16:21:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.247+0000 7fb76fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7700cdf70 0x7fb770153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.247+0000 7fb76fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7700cdf70 0x7fb770153870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48340/0 (socket says 172.21.15.45:48340) 2024-04-16T16:21:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.247+0000 7fb76fe00640 1 -- 172.21.15.45:0/2706207106 learned_addr learned my addr 172.21.15.45:0/2706207106 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.248+0000 7fb775200640 1 --2- 172.21.15.45:0/2706207106 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb770146120 0x7fb77013ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.248+0000 7fb76f400640 1 --2- 172.21.15.45:0/2706207106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7700cf5a0 0x7fb7700b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.248+0000 7fb76fe00640 1 -- 172.21.15.45:0/2706207106 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb770146120 msgr2=0x7fb77013ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.248+0000 7fb76fe00640 1 --2- 172.21.15.45:0/2706207106 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb770146120 0x7fb77013ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.248+0000 7fb76fe00640 1 -- 172.21.15.45:0/2706207106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7700cf5a0 msgr2=0x7fb7700b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.248+0000 7fb76fe00640 1 --2- 172.21.15.45:0/2706207106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7700cf5a0 0x7fb7700b0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.248+0000 7fb76fe00640 1 -- 172.21.15.45:0/2706207106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7701411d0 con 0x7fb7700cdf70 2024-04-16T16:21:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.248+0000 7fb76fe00640 1 --2- 172.21.15.45:0/2706207106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7700cdf70 0x7fb770153870 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fb76002b940 tx=0x7fb7600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:04.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.248+0000 7fb767e00640 1 -- 172.21.15.45:0/2706207106 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76003ac60 con 0x7fb7700cdf70 2024-04-16T16:21:04.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.249+0000 7fb767e00640 1 -- 172.21.15.45:0/2706207106 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb76003ae00 con 0x7fb7700cdf70 2024-04-16T16:21:04.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.249+0000 7fb775c00640 1 -- 172.21.15.45:0/2706207106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb770141470 con 0x7fb7700cdf70 2024-04-16T16:21:04.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.249+0000 7fb767e00640 1 -- 172.21.15.45:0/2706207106 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb760044610 con 0x7fb7700cdf70 2024-04-16T16:21:04.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.249+0000 7fb775c00640 1 -- 172.21.15.45:0/2706207106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7700b48d0 con 0x7fb7700cdf70 2024-04-16T16:21:04.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.250+0000 7fb775c00640 1 -- 172.21.15.45:0/2706207106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb770141750 con 0x7fb7700cdf70 2024-04-16T16:21:04.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.251+0000 7fb767e00640 1 -- 172.21.15.45:0/2706207106 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 55) ==== 91142+0+0 (secure 0 0 0) 0x7fb760038020 con 0x7fb7700cdf70 2024-04-16T16:21:04.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.252+0000 7fb767e00640 1 --2- 172.21.15.45:0/2706207106 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fb74c06ca10 0x7fb74c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:04.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.252+0000 7fb767e00640 1 -- 172.21.15.45:0/2706207106 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6053+0+0 (secure 0 0 0) 0x7fb7600ade90 con 0x7fb7700cdf70 2024-04-16T16:21:04.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.253+0000 7fb76f400640 1 -- 172.21.15.45:0/2706207106 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fb74c06ca10 msgr2=0x7fb74c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/549890383 2024-04-16T16:21:04.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.253+0000 7fb76f400640 1 --2- 172.21.15.45:0/2706207106 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fb74c06ca10 0x7fb74c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:21:04.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.257+0000 7fb767e00640 1 -- 172.21.15.45:0/2706207106 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb76003f070 con 0x7fb7700cdf70 2024-04-16T16:21:04.334 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:04.334+0000 7f9f6da0d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:21:04.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.357+0000 7fb767e00640 1 -- 172.21.15.45:0/2706207106 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 56) ==== 90751+0+0 (secure 0 0 0) 0x7fb7600ac250 con 0x7fb7700cdf70 2024-04-16T16:21:04.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.357+0000 7fb767e00640 1 -- 172.21.15.45:0/2706207106 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fb74c06ca10 msgr2=0x7fb74c06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:21:04.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.357+0000 7fb767e00640 1 --2- 172.21.15.45:0/2706207106 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fb74c06ca10 0x7fb74c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.438 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:04.439+0000 7f9ee7200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:04.438 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:04.439+0000 7f9ee7200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:04.439 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:04.439+0000 7f9ee7200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:04.439 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:04.439+0000 7f9ee7200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:04.439 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:04.439+0000 7f9ee7200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:04.439 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:04.440+0000 7f9eeae00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:04.439 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:04.440+0000 7f9eeae00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:04.439 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:04.440+0000 7f9eeae00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:04.440 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:04.440+0000 7f9eeae00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:04.440 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:04.440+0000 7f9eeae00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:04.473 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:04.473+0000 7f7fa9c46040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:21:04.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.488+0000 7fb775c00640 1 -- 172.21.15.45:0/2706207106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb770070850 con 0x7fb7700cdf70 2024-04-16T16:21:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.489+0000 7fb767e00640 1 -- 172.21.15.45:0/2706207106 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v130) ==== 104+0+11 (secure 0 0 0) 0x7fb76007bf40 con 0x7fb7700cdf70 2024-04-16T16:21:04.489 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:21:04.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.492+0000 7fb775c00640 1 -- 172.21.15.45:0/2706207106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7700cdf70 msgr2=0x7fb770153870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:04.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.492+0000 7fb775c00640 1 --2- 172.21.15.45:0/2706207106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7700cdf70 0x7fb770153870 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fb76002b940 tx=0x7fb7600046c0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.492+0000 7fb775c00640 1 -- 172.21.15.45:0/2706207106 shutdown_connections 2024-04-16T16:21:04.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.492+0000 7fb775c00640 1 --2- 172.21.15.45:0/2706207106 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb770146120 0x7fb77013ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.492+0000 7fb775c00640 1 --2- 172.21.15.45:0/2706207106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7700cf5a0 0x7fb7700b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.492+0000 7fb775c00640 1 --2- 172.21.15.45:0/2706207106 >> [v2:172.21.15.103:6832/549890383,v1:172.21.15.103:6833/549890383] conn(0x7fb74c06ca10 0x7fb74c06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.492+0000 7fb775c00640 1 --2- 172.21.15.45:0/2706207106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7700cdf70 0x7fb770153870 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.492+0000 7fb775c00640 1 -- 172.21.15.45:0/2706207106 >> 172.21.15.45:0/2706207106 conn(0x7fb77006f120 msgr2=0x7fb770070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:04.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.492+0000 7fb775c00640 1 -- 172.21.15.45:0/2706207106 shutdown_connections 2024-04-16T16:21:04.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.492+0000 7fb775c00640 1 -- 172.21.15.45:0/2706207106 wait complete. 2024-04-16T16:21:04.510 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:21:04.538 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:04.539+0000 7f7fa9c46040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:21:04.687 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:04.687+0000 7f7fa9c46040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:21:04.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.720+0000 7fe92e400640 1 Processor -- start 2024-04-16T16:21:04.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.720+0000 7fe92e400640 1 -- start start 2024-04-16T16:21:04.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.722+0000 7fe92e400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe9280b6c00 0x7fe9280d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:04.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.722+0000 7fe92e400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9280b7ca0 con 0x7fe92815ec80 2024-04-16T16:21:04.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.722+0000 7fe92e400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9280d3450 con 0x7fe92815f060 2024-04-16T16:21:04.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.722+0000 7fe92e400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe9280d3220 con 0x7fe9280b6c00 2024-04-16T16:21:04.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.722+0000 7fe927400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe92815ec80 0x7fe9280b64f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36058/0 (socket says 172.21.15.45:36058) 2024-04-16T16:21:04.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.722+0000 7fe927400640 1 -- 172.21.15.45:0/1332928729 learned_addr learned my addr 172.21.15.45:0/1332928729 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:04.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe927e00640 1 --2- 172.21.15.45:0/1332928729 >> v2:172.21.15.103:3300/0 conn(0x7fe9280b6c00 0x7fe9280d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:04.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe926000640 1 -- 172.21.15.45:0/1332928729 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1598881694 0 0) 0x7fe9280b7ca0 con 0x7fe92815ec80 2024-04-16T16:21:04.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe926000640 1 -- 172.21.15.45:0/1332928729 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe90c003520 con 0x7fe92815ec80 2024-04-16T16:21:04.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe926000640 1 -- 172.21.15.45:0/1332928729 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4266749465 0 0) 0x7fe90c003520 con 0x7fe92815ec80 2024-04-16T16:21:04.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe926000640 1 -- 172.21.15.45:0/1332928729 >> v2:172.21.15.103:3300/0 conn(0x7fe9280b6c00 msgr2=0x7fe9280d2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:04.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe926000640 1 --2- 172.21.15.45:0/1332928729 >> v2:172.21.15.103:3300/0 conn(0x7fe9280b6c00 0x7fe9280d2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe926000640 1 -- 172.21.15.45:0/1332928729 >> v1:172.21.15.103:6789/0 conn(0x7fe92815f060 legacy=0x7fe928166980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:04.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe926000640 1 -- 172.21.15.45:0/1332928729 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9280d3730 con 0x7fe92815ec80 2024-04-16T16:21:04.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe926000640 1 -- 172.21.15.45:0/1332928729 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fe918001a10 con 0x7fe92815ec80 2024-04-16T16:21:04.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe926000640 1 -- 172.21.15.45:0/1332928729 >> v1:172.21.15.45:6789/0 conn(0x7fe92815ec80 legacy=0x7fe9280b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:04.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe926000640 1 --2- 172.21.15.45:0/1332928729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90c003a00 0x7fe90c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:04.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe926000640 1 --2- 172.21.15.45:0/1332928729 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90c004800 0x7fe90c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:04.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe926000640 1 --2- 172.21.15.45:0/1332928729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe90c0051c0 0x7fe90c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:04.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe926000640 1 -- 172.21.15.45:0/1332928729 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c00c070 con 0x7fe90c003a00 2024-04-16T16:21:04.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe926000640 1 -- 172.21.15.45:0/1332928729 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c00c220 con 0x7fe90c0051c0 2024-04-16T16:21:04.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.723+0000 7fe926000640 1 -- 172.21.15.45:0/1332928729 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe90c00c420 con 0x7fe90c004800 2024-04-16T16:21:04.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.724+0000 7fe927400640 1 --2- 172.21.15.45:0/1332928729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90c003a00 0x7fe90c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:04.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.724+0000 7fe927e00640 1 --2- 172.21.15.45:0/1332928729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe90c0051c0 0x7fe90c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:04.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.724+0000 7fe926a00640 1 --2- 172.21.15.45:0/1332928729 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90c004800 0x7fe90c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:04.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.724+0000 7fe927400640 1 -- 172.21.15.45:0/1332928729 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90c004800 msgr2=0x7fe90c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:04.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.724+0000 7fe927400640 1 --2- 172.21.15.45:0/1332928729 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90c004800 0x7fe90c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.724+0000 7fe927400640 1 -- 172.21.15.45:0/1332928729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe90c0051c0 msgr2=0x7fe90c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:04.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.724+0000 7fe927400640 1 --2- 172.21.15.45:0/1332928729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe90c0051c0 0x7fe90c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.724+0000 7fe927400640 1 -- 172.21.15.45:0/1332928729 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90c003520 con 0x7fe90c003a00 2024-04-16T16:21:04.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.724+0000 7fe927e00640 1 --2- 172.21.15.45:0/1332928729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe90c0051c0 0x7fe90c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:04.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.724+0000 7fe927e00640 1 -- 172.21.15.45:0/1332928729 reap_dead start 2024-04-16T16:21:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.725+0000 7fe927400640 1 --2- 172.21.15.45:0/1332928729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90c003a00 0x7fe90c003e20 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fe91800a780 tx=0x7fe90c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da6398a217fe832 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.726+0000 7fe926000640 1 -- 172.21.15.45:0/1332928729 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91800a3d0 con 0x7fe90c003a00 2024-04-16T16:21:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.726+0000 7fe926000640 1 -- 172.21.15.45:0/1332928729 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe918034630 con 0x7fe90c003a00 2024-04-16T16:21:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.726+0000 7fe926000640 1 -- 172.21.15.45:0/1332928729 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe918034a20 con 0x7fe90c003a00 2024-04-16T16:21:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.726+0000 7fe92e400640 1 -- 172.21.15.45:0/1332928729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90c003a00 msgr2=0x7fe90c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.726+0000 7fe92e400640 1 --2- 172.21.15.45:0/1332928729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90c003a00 0x7fe90c003e20 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fe91800a780 tx=0x7fe90c005830 comp rx=0 tx=0).stop 2024-04-16T16:21:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.726+0000 7fe92e400640 1 -- 172.21.15.45:0/1332928729 shutdown_connections 2024-04-16T16:21:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.726+0000 7fe92e400640 1 --2- 172.21.15.45:0/1332928729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90c003a00 0x7fe90c003e20 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.726+0000 7fe92e400640 1 -- 172.21.15.45:0/1332928729 >> 172.21.15.45:0/1332928729 conn(0x7fe9280d16d0 msgr2=0x7fe9280d1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.726+0000 7fe92e400640 1 -- 172.21.15.45:0/1332928729 shutdown_connections 2024-04-16T16:21:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.726+0000 7fe92e400640 1 -- 172.21.15.45:0/1332928729 wait complete. 2024-04-16T16:21:04.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.728+0000 7fe92e400640 1 Processor -- start 2024-04-16T16:21:04.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.728+0000 7fe92e400640 1 -- start start 2024-04-16T16:21:04.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.729+0000 7fe92e400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90c003a00 0x7fe9280c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:04.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.729+0000 7fe92e400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe928158dd0 0x7fe9280ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:04.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.729+0000 7fe92e400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe92815a370 0x7fe9281610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:04.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.729+0000 7fe92e400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe9280d4160 con 0x7fe928158dd0 2024-04-16T16:21:04.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.729+0000 7fe92e400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe9280d3cf0 con 0x7fe92815a370 2024-04-16T16:21:04.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.729+0000 7fe92e400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe9280d42e0 con 0x7fe90c003a00 2024-04-16T16:21:04.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.729+0000 7fe926a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe928158dd0 0x7fe9280ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:04.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.729+0000 7fe926a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe928158dd0 0x7fe9280ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48362/0 (socket says 172.21.15.45:48362) 2024-04-16T16:21:04.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.729+0000 7fe926a00640 1 -- 172.21.15.45:0/2588639497 learned_addr learned my addr 172.21.15.45:0/2588639497 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:04.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.730+0000 7fe926a00640 1 -- 172.21.15.45:0/2588639497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90c003a00 msgr2=0x7fe9280c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:04.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.730+0000 7fe926a00640 1 --2- 172.21.15.45:0/2588639497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90c003a00 0x7fe9280c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.730+0000 7fe926a00640 1 -- 172.21.15.45:0/2588639497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe92815a370 msgr2=0x7fe9281610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:04.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.730+0000 7fe926a00640 1 --2- 172.21.15.45:0/2588639497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe92815a370 0x7fe9281610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.730+0000 7fe926a00640 1 -- 172.21.15.45:0/2588639497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9281592d0 con 0x7fe928158dd0 2024-04-16T16:21:04.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.730+0000 7fe926a00640 1 --2- 172.21.15.45:0/2588639497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe928158dd0 0x7fe9280ca830 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fe91400d5a0 tx=0x7fe91400da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:04.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.730+0000 7fe91f400640 1 -- 172.21.15.45:0/2588639497 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe914005480 con 0x7fe928158dd0 2024-04-16T16:21:04.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.730+0000 7fe91f400640 1 -- 172.21.15.45:0/2588639497 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe914017070 con 0x7fe928158dd0 2024-04-16T16:21:04.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.730+0000 7fe91f400640 1 -- 172.21.15.45:0/2588639497 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe914003830 con 0x7fe928158dd0 2024-04-16T16:21:04.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.730+0000 7fe92e400640 1 -- 172.21.15.45:0/2588639497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9281595b0 con 0x7fe928158dd0 2024-04-16T16:21:04.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.730+0000 7fe92e400640 1 -- 172.21.15.45:0/2588639497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe928159b70 con 0x7fe928158dd0 2024-04-16T16:21:04.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.731+0000 7fe92e400640 1 -- 172.21.15.45:0/2588639497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9280b7840 con 0x7fe928158dd0 2024-04-16T16:21:04.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.732+0000 7fe91f400640 1 -- 172.21.15.45:0/2588639497 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 56) ==== 90751+0+0 (secure 0 0 0) 0x7fe9140039d0 con 0x7fe928158dd0 2024-04-16T16:21:04.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.733+0000 7fe91f400640 1 -- 172.21.15.45:0/2588639497 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6397+0+0 (secure 0 0 0) 0x7fe91408b760 con 0x7fe928158dd0 2024-04-16T16:21:04.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.737+0000 7fe91f400640 1 -- 172.21.15.45:0/2588639497 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe9280b7840 con 0x7fe928158dd0 2024-04-16T16:21:04.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.941+0000 7fe92e400640 1 -- 172.21.15.45:0/2588639497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe9280d1c40 con 0x7fe928158dd0 2024-04-16T16:21:04.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.942+0000 7fe91f400640 1 -- 172.21.15.45:0/2588639497 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) ==== 100+0+11 (secure 0 0 0) 0x7fe914059c40 con 0x7fe928158dd0 2024-04-16T16:21:04.942 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:21:04.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.944+0000 7fe92e400640 1 -- 172.21.15.45:0/2588639497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe928158dd0 msgr2=0x7fe9280ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:04.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.944+0000 7fe92e400640 1 --2- 172.21.15.45:0/2588639497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe928158dd0 0x7fe9280ca830 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fe91400d5a0 tx=0x7fe91400da60 comp rx=0 tx=0).stop 2024-04-16T16:21:04.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.945+0000 7fe92e400640 1 -- 172.21.15.45:0/2588639497 shutdown_connections 2024-04-16T16:21:04.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.945+0000 7fe92e400640 1 --2- 172.21.15.45:0/2588639497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe92815a370 0x7fe9281610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.945+0000 7fe92e400640 1 --2- 172.21.15.45:0/2588639497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe928158dd0 0x7fe9280ca830 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.945+0000 7fe92e400640 1 --2- 172.21.15.45:0/2588639497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe90c003a00 0x7fe9280c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:04.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.945+0000 7fe92e400640 1 -- 172.21.15.45:0/2588639497 >> 172.21.15.45:0/2588639497 conn(0x7fe9280d16d0 msgr2=0x7fe92806d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:04.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.945+0000 7fe92e400640 1 -- 172.21.15.45:0/2588639497 shutdown_connections 2024-04-16T16:21:04.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:04.945+0000 7fe92e400640 1 -- 172.21.15.45:0/2588639497 wait complete. 2024-04-16T16:21:04.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:21:05.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.233+0000 7f1c1e200640 1 Processor -- start 2024-04-16T16:21:05.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.234+0000 7f1c1e200640 1 -- start start 2024-04-16T16:21:05.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.235+0000 7f1c1e200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1c1815ca00 0x7f1c180cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:05.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.235+0000 7f1c1e200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c1806fab0 con 0x7f1c180cde10 2024-04-16T16:21:05.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.235+0000 7f1c1e200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c1815bf20 con 0x7f1c180afe20 2024-04-16T16:21:05.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.235+0000 7f1c1e200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1c180ce240 con 0x7f1c1815ca00 2024-04-16T16:21:05.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.236+0000 7f1c17e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1c180cde10 0x7f1c180b4160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36068/0 (socket says 172.21.15.45:36068) 2024-04-16T16:21:05.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.236+0000 7f1c17e00640 1 -- 172.21.15.45:0/3065375701 learned_addr learned my addr 172.21.15.45:0/3065375701 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:05.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.236+0000 7f1c16a00640 1 --2- 172.21.15.45:0/3065375701 >> v2:172.21.15.45:3300/0 conn(0x7f1c1815ca00 0x7f1c180cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:05.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.236+0000 7f1c16a00640 1 -- 172.21.15.45:0/3065375701 >> v1:172.21.15.103:6789/0 conn(0x7f1c180afe20 legacy=0x7f1c1806f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:05.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.236+0000 7f1c16a00640 1 -- 172.21.15.45:0/3065375701 >> v1:172.21.15.45:6789/0 conn(0x7f1c180cde10 legacy=0x7f1c180b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:05.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.236+0000 7f1c16a00640 1 -- 172.21.15.45:0/3065375701 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1815c200 con 0x7f1c1815ca00 2024-04-16T16:21:05.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.236+0000 7f1c16000640 1 -- 172.21.15.45:0/3065375701 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3223372467 0 0) 0x7f1c1806fab0 con 0x7f1c180cde10 2024-04-16T16:21:05.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.237+0000 7f1c16a00640 1 --2- 172.21.15.45:0/3065375701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c1815ca00 0x7f1c180cd8d0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f1c00009d90 tx=0x7f1c0002e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af90a385437f565e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:05.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.238+0000 7f1c16000640 1 -- 172.21.15.45:0/3065375701 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0003b070 con 0x7f1c1815ca00 2024-04-16T16:21:05.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.238+0000 7f1c16000640 1 -- 172.21.15.45:0/3065375701 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c00036440 con 0x7f1c1815ca00 2024-04-16T16:21:05.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.238+0000 7f1c16000640 1 -- 172.21.15.45:0/3065375701 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c00036810 con 0x7f1c1815ca00 2024-04-16T16:21:05.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.238+0000 7f1c1e200640 1 -- 172.21.15.45:0/3065375701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c1815ca00 msgr2=0x7f1c180cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:05.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.238+0000 7f1c1e200640 1 --2- 172.21.15.45:0/3065375701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c1815ca00 0x7f1c180cd8d0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f1c00009d90 tx=0x7f1c0002e0c0 comp rx=0 tx=0).stop 2024-04-16T16:21:05.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.238+0000 7f1c1e200640 1 -- 172.21.15.45:0/3065375701 shutdown_connections 2024-04-16T16:21:05.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.238+0000 7f1c1e200640 1 --2- 172.21.15.45:0/3065375701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c1815ca00 0x7f1c180cd8d0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:05.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.238+0000 7f1c1e200640 1 -- 172.21.15.45:0/3065375701 >> 172.21.15.45:0/3065375701 conn(0x7f1c1806b930 msgr2=0x7f1c1806bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:05.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.239+0000 7f1c1e200640 1 -- 172.21.15.45:0/3065375701 shutdown_connections 2024-04-16T16:21:05.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.239+0000 7f1c1e200640 1 -- 172.21.15.45:0/3065375701 wait complete. 2024-04-16T16:21:05.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.240+0000 7f1c1e200640 1 Processor -- start 2024-04-16T16:21:05.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.240+0000 7f1c1e200640 1 -- start start 2024-04-16T16:21:05.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.241+0000 7f1c1e200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c180afe20 0x7f1c1815e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:05.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.241+0000 7f1c1e200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c180cde10 0x7f1c181e4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:05.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.241+0000 7f1c1e200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c1815ca00 0x7f1c1814dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:05.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.241+0000 7f1c1e200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1c180b0200 con 0x7f1c180cde10 2024-04-16T16:21:05.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.241+0000 7f1c1e200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1806f720 con 0x7f1c1815ca00 2024-04-16T16:21:05.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.241+0000 7f1c1e200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c180b0380 con 0x7f1c180afe20 2024-04-16T16:21:05.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.242+0000 7f1c16a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c180cde10 0x7f1c181e4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:05.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.242+0000 7f1c16a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c180cde10 0x7f1c181e4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48376/0 (socket says 172.21.15.45:48376) 2024-04-16T16:21:05.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.242+0000 7f1c16a00640 1 -- 172.21.15.45:0/1548534785 learned_addr learned my addr 172.21.15.45:0/1548534785 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:05.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.242+0000 7f1c16a00640 1 -- 172.21.15.45:0/1548534785 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c180afe20 msgr2=0x7f1c1815e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:05.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.242+0000 7f1c16a00640 1 --2- 172.21.15.45:0/1548534785 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c180afe20 0x7f1c1815e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:05.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.242+0000 7f1c16a00640 1 -- 172.21.15.45:0/1548534785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c1815ca00 msgr2=0x7f1c1814dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:05.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.242+0000 7f1c16a00640 1 --2- 172.21.15.45:0/1548534785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c1815ca00 0x7f1c1814dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:05.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.242+0000 7f1c16a00640 1 -- 172.21.15.45:0/1548534785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c18153370 con 0x7f1c180cde10 2024-04-16T16:21:05.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.242+0000 7f1c16a00640 1 --2- 172.21.15.45:0/1548534785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c180cde10 0x7f1c181e4930 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f1c00039040 tx=0x7f1c00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:05.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.243+0000 7f1c0f400640 1 -- 172.21.15.45:0/1548534785 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c00046070 con 0x7f1c180cde10 2024-04-16T16:21:05.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.243+0000 7f1c0f400640 1 -- 172.21.15.45:0/1548534785 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c00041630 con 0x7f1c180cde10 2024-04-16T16:21:05.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.243+0000 7f1c0f400640 1 -- 172.21.15.45:0/1548534785 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0003b040 con 0x7f1c180cde10 2024-04-16T16:21:05.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.243+0000 7f1c1e200640 1 -- 172.21.15.45:0/1548534785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c181536b0 con 0x7f1c180cde10 2024-04-16T16:21:05.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.243+0000 7f1c1e200640 1 -- 172.21.15.45:0/1548534785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c18153ef0 con 0x7f1c180cde10 2024-04-16T16:21:05.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.244+0000 7f1c1e200640 1 -- 172.21.15.45:0/1548534785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c18075780 con 0x7f1c180cde10 2024-04-16T16:21:05.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.245+0000 7f1c0f400640 1 -- 172.21.15.45:0/1548534785 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 56) ==== 90751+0+0 (secure 0 0 0) 0x7f1c00049070 con 0x7f1c180cde10 2024-04-16T16:21:05.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.247+0000 7f1c0f400640 1 -- 172.21.15.45:0/1548534785 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6397+0+0 (secure 0 0 0) 0x7f1c000ada00 con 0x7f1c180cde10 2024-04-16T16:21:05.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.251+0000 7f1c0f400640 1 -- 172.21.15.45:0/1548534785 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1c181536b0 con 0x7f1c180cde10 2024-04-16T16:21:05.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.366+0000 7f1c0f400640 1 -- 172.21.15.45:0/1548534785 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 57) ==== 90878+0+0 (secure 0 0 0) 0x7f1c00077570 con 0x7f1c180cde10 2024-04-16T16:21:05.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.367+0000 7f1c0f400640 1 --2- 172.21.15.45:0/1548534785 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1bf4072000 0x7f1bf40743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:05.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.370+0000 7f1c17400640 1 --2- 172.21.15.45:0/1548534785 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1bf4072000 0x7f1bf40743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:05.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.379+0000 7f1c17400640 1 --2- 172.21.15.45:0/1548534785 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1bf4072000 0x7f1bf40743f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1c08006fd0 tx=0x7f1c08006bc0 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:05.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.479+0000 7f1c1e200640 1 -- 172.21.15.45:0/1548534785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1c18073330 con 0x7f1c180cde10 2024-04-16T16:21:05.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.480+0000 7f1c0f400640 1 -- 172.21.15.45:0/1548534785 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v130) ==== 81+0+954 (secure 0 0 0) 0x7f1c0000ae50 con 0x7f1c180cde10 2024-04-16T16:21:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1715153913 2024-04-17T16:21:04.341295+0000 2024-04-16T16:21:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/4225560501 2024-04-17T16:21:04.341295+0000 2024-04-16T16:21:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/549890383 2024-04-17T16:21:04.341295+0000 2024-04-16T16:21:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1495495989 2024-04-17T16:21:04.341295+0000 2024-04-16T16:21:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3982280724 2024-04-17T16:21:04.341295+0000 2024-04-16T16:21:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/1120913661 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/549890383 2024-04-17T16:21:04.341295+0000 2024-04-16T16:21:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/1120913661 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1543548744 2024-04-17T16:21:04.341295+0000 2024-04-16T16:21:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/4103173879 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1578451701 2024-04-17T16:21:04.341295+0000 2024-04-16T16:21:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/4177084908 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1961421675 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2190202958 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3679729711 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1543358977 2024-04-17T16:20:50.935324+0000 2024-04-16T16:21:05.482 INFO:teuthology.orchestra.run.smithi045.stderr:listed 16 entries 2024-04-16T16:21:05.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.483+0000 7f1c1e200640 1 -- 172.21.15.45:0/1548534785 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1bf4072000 msgr2=0x7f1bf40743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:05.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.483+0000 7f1c1e200640 1 --2- 172.21.15.45:0/1548534785 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1bf4072000 0x7f1bf40743f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1c08006fd0 tx=0x7f1c08006bc0 comp rx=0 tx=0).stop 2024-04-16T16:21:05.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.483+0000 7f1c1e200640 1 -- 172.21.15.45:0/1548534785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c180cde10 msgr2=0x7f1c181e4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:05.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.483+0000 7f1c1e200640 1 --2- 172.21.15.45:0/1548534785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c180cde10 0x7f1c181e4930 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f1c00039040 tx=0x7f1c00004a40 comp rx=0 tx=0).stop 2024-04-16T16:21:05.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.483+0000 7f1c1e200640 1 -- 172.21.15.45:0/1548534785 shutdown_connections 2024-04-16T16:21:05.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.483+0000 7f1c1e200640 1 --2- 172.21.15.45:0/1548534785 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1bf4072000 0x7f1bf40743f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:05.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.483+0000 7f1c1e200640 1 --2- 172.21.15.45:0/1548534785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c1815ca00 0x7f1c1814dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:05.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.483+0000 7f1c1e200640 1 --2- 172.21.15.45:0/1548534785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c180cde10 0x7f1c181e4930 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:05.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.483+0000 7f1c1e200640 1 --2- 172.21.15.45:0/1548534785 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c180afe20 0x7f1c1815e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:05.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.483+0000 7f1c1e200640 1 -- 172.21.15.45:0/1548534785 >> 172.21.15.45:0/1548534785 conn(0x7f1c1806b930 msgr2=0x7f1c1806df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:05.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.483+0000 7f1c1e200640 1 -- 172.21.15.45:0/1548534785 shutdown_connections 2024-04-16T16:21:05.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:05.483+0000 7f1c1e200640 1 -- 172.21.15.45:0/1548534785 wait complete. 2024-04-16T16:21:05.501 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-16T16:21:05.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:05.502 DEBUG:teuthology.orchestra.run.smithi103:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-04-16T16:21:05.609 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:21:05.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:05.609 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-16T16:21:05.664 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:21:05.664 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:21:05.664 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:21:05.725 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:21:05.725 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:21:05.725 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:21:05.725 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:21:05.781 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:21:05.781 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:21:06.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.010+0000 7f648b200640 1 Processor -- start 2024-04-16T16:21:06.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.011+0000 7f648b200640 1 -- start start 2024-04-16T16:21:06.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.012+0000 7f648b200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f648c161e90 0x7f648c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:06.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.012+0000 7f648b200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f648c0d3080 con 0x7f648c0d52d0 2024-04-16T16:21:06.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.012+0000 7f648b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f648c0d3360 con 0x7f648c162270 2024-04-16T16:21:06.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.012+0000 7f648b200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f648c0d2e30 con 0x7f648c161e90 2024-04-16T16:21:06.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.013+0000 7f6489e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f648c161e90 0x7f648c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:06.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.013+0000 7f6489e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f648c161e90 0x7f648c0cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48386/0 (socket says 172.21.15.45:48386) 2024-04-16T16:21:06.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.013+0000 7f6489e00640 1 -- 172.21.15.45:0/3714772149 learned_addr learned my addr 172.21.15.45:0/3714772149 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:06.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.013+0000 7f6489e00640 1 -- 172.21.15.45:0/3714772149 >> v1:172.21.15.103:6789/0 conn(0x7f648c162270 legacy=0x7f648c0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:06.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.013+0000 7f6489e00640 1 -- 172.21.15.45:0/3714772149 >> v1:172.21.15.45:6789/0 conn(0x7f648c0d52d0 legacy=0x7f648c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:06.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.013+0000 7f6489e00640 1 -- 172.21.15.45:0/3714772149 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c0d3640 con 0x7f648c161e90 2024-04-16T16:21:06.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.014+0000 7f6489e00640 1 --2- 172.21.15.45:0/3714772149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c161e90 0x7f648c0cf7d0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f64740098a0 tx=0x7f647402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66dce63f45ae7cf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:06.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.015+0000 7f6488a00640 1 -- 172.21.15.45:0/3714772149 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474034070 con 0x7f648c161e90 2024-04-16T16:21:06.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.015+0000 7f6488a00640 1 -- 172.21.15.45:0/3714772149 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6474030840 con 0x7f648c161e90 2024-04-16T16:21:06.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.015+0000 7f6488a00640 1 -- 172.21.15.45:0/3714772149 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474030c10 con 0x7f648c161e90 2024-04-16T16:21:06.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.015+0000 7f648b200640 1 -- 172.21.15.45:0/3714772149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c161e90 msgr2=0x7f648c0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:06.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.015+0000 7f648b200640 1 --2- 172.21.15.45:0/3714772149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c161e90 0x7f648c0cf7d0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f64740098a0 tx=0x7f647402b980 comp rx=0 tx=0).stop 2024-04-16T16:21:06.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.016+0000 7f648b200640 1 -- 172.21.15.45:0/3714772149 shutdown_connections 2024-04-16T16:21:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.016+0000 7f648b200640 1 --2- 172.21.15.45:0/3714772149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c161e90 0x7f648c0cf7d0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.016+0000 7f648b200640 1 -- 172.21.15.45:0/3714772149 >> 172.21.15.45:0/3714772149 conn(0x7f648c0d1940 msgr2=0x7f648c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.016+0000 7f648b200640 1 -- 172.21.15.45:0/3714772149 shutdown_connections 2024-04-16T16:21:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.016+0000 7f648b200640 1 -- 172.21.15.45:0/3714772149 wait complete. 2024-04-16T16:21:06.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.017+0000 7f648b200640 1 Processor -- start 2024-04-16T16:21:06.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.018+0000 7f648b200640 1 -- start start 2024-04-16T16:21:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.019+0000 7f648b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c0d52d0 0x7f648c1f7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.019+0000 7f648b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c161e90 0x7f648c15e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.019+0000 7f648b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f648c162270 0x7f648c15ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.019+0000 7f648b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f648c0d4550 con 0x7f648c0d52d0 2024-04-16T16:21:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.019+0000 7f648b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f648c0cfd10 con 0x7f648c162270 2024-04-16T16:21:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.019+0000 7f648b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f648c0d46d0 con 0x7f648c161e90 2024-04-16T16:21:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.019+0000 7f6489e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c0d52d0 0x7f648c1f7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.019+0000 7f6489e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c0d52d0 0x7f648c1f7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48396/0 (socket says 172.21.15.45:48396) 2024-04-16T16:21:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.019+0000 7f6489e00640 1 -- 172.21.15.45:0/3133215392 learned_addr learned my addr 172.21.15.45:0/3133215392 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.019+0000 7f6489e00640 1 -- 172.21.15.45:0/3133215392 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c161e90 msgr2=0x7f648c15e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.019+0000 7f6489e00640 1 --2- 172.21.15.45:0/3133215392 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c161e90 0x7f648c15e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.019+0000 7f6489e00640 1 -- 172.21.15.45:0/3133215392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f648c162270 msgr2=0x7f648c15ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.019+0000 7f6489e00640 1 --2- 172.21.15.45:0/3133215392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f648c162270 0x7f648c15ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.019+0000 7f6489e00640 1 -- 172.21.15.45:0/3133215392 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c0cc4a0 con 0x7f648c0d52d0 2024-04-16T16:21:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.020+0000 7f6489e00640 1 --2- 172.21.15.45:0/3133215392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c0d52d0 0x7f648c1f7770 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f6474000c00 tx=0x7f6474004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.020+0000 7f6482000640 1 -- 172.21.15.45:0/3133215392 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474048070 con 0x7f648c0d52d0 2024-04-16T16:21:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.020+0000 7f6482000640 1 -- 172.21.15.45:0/3133215392 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6474030640 con 0x7f648c0d52d0 2024-04-16T16:21:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.020+0000 7f6482000640 1 -- 172.21.15.45:0/3133215392 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474034040 con 0x7f648c0d52d0 2024-04-16T16:21:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.020+0000 7f648b200640 1 -- 172.21.15.45:0/3133215392 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f648c0cc780 con 0x7f648c0d52d0 2024-04-16T16:21:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.020+0000 7f648b200640 1 -- 172.21.15.45:0/3133215392 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f648c0ccd40 con 0x7f648c0d52d0 2024-04-16T16:21:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.022+0000 7f648b200640 1 -- 172.21.15.45:0/3133215392 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f648c075780 con 0x7f648c0d52d0 2024-04-16T16:21:06.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.023+0000 7f6482000640 1 -- 172.21.15.45:0/3133215392 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) ==== 90878+0+0 (secure 0 0 0) 0x7f647404b070 con 0x7f648c0d52d0 2024-04-16T16:21:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.024+0000 7f6482000640 1 --2- 172.21.15.45:0/3133215392 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f646806c740 0x7f646806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.024+0000 7f6482000640 1 -- 172.21.15.45:0/3133215392 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6397+0+0 (secure 0 0 0) 0x7f64740adb90 con 0x7f648c0d52d0 2024-04-16T16:21:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.025+0000 7f6489400640 1 --2- 172.21.15.45:0/3133215392 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f646806c740 0x7f646806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:06.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.026+0000 7f6489400640 1 --2- 172.21.15.45:0/3133215392 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f646806c740 0x7f646806ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f648c0b5cb0 tx=0x7f647c009230 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:06.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.029+0000 7f6482000640 1 -- 172.21.15.45:0/3133215392 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f647407c040 con 0x7f648c0d52d0 2024-04-16T16:21:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.284+0000 7f648b200640 1 -- 172.21.15.45:0/3133215392 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f648c06dae0 con 0x7f648c0d52d0 2024-04-16T16:21:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.286+0000 7f6482000640 1 -- 172.21.15.45:0/3133215392 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5186 (secure 0 0 0) 0x7f647407b940 con 0x7f648c0d52d0 2024-04-16T16:21:06.286 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:06.286 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/929218719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":929218719},{"type":"v1","addr":"172.21.15.103:6835","nonce":929218719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6323,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1619719405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1619719405},{"type":"v1","addr":"172.21.15.103:6837","nonce":1619719405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":8640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/3613288118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3613288118},{"type":"v1","addr":"172.21.15.45:6834","nonce":3613288118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:20:33.816422+0000","modified":"2024-04-16T16:20:36.891870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8643":{"gid":8643,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2060312705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2060312705},{"type":"v1","addr":"172.21.15.45:6835","nonce":2060312705}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8643,"qdb_cluster":[8643]},"id":9}]} 2024-04-16T16:21:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 91 2024-04-16T16:21:06.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.289+0000 7f648b200640 1 -- 172.21.15.45:0/3133215392 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f646806c740 msgr2=0x7f646806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:06.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.289+0000 7f648b200640 1 --2- 172.21.15.45:0/3133215392 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f646806c740 0x7f646806ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f648c0b5cb0 tx=0x7f647c009230 comp rx=0 tx=0).stop 2024-04-16T16:21:06.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.289+0000 7f648b200640 1 -- 172.21.15.45:0/3133215392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c0d52d0 msgr2=0x7f648c1f7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:06.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.289+0000 7f648b200640 1 --2- 172.21.15.45:0/3133215392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c0d52d0 0x7f648c1f7770 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f6474000c00 tx=0x7f6474004730 comp rx=0 tx=0).stop 2024-04-16T16:21:06.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.289+0000 7f648b200640 1 -- 172.21.15.45:0/3133215392 shutdown_connections 2024-04-16T16:21:06.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.289+0000 7f648b200640 1 --2- 172.21.15.45:0/3133215392 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f646806c740 0x7f646806ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.289+0000 7f648b200640 1 --2- 172.21.15.45:0/3133215392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f648c162270 0x7f648c15ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.289+0000 7f648b200640 1 --2- 172.21.15.45:0/3133215392 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648c161e90 0x7f648c15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.289+0000 7f648b200640 1 --2- 172.21.15.45:0/3133215392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648c0d52d0 0x7f648c1f7770 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.289+0000 7f648b200640 1 -- 172.21.15.45:0/3133215392 >> 172.21.15.45:0/3133215392 conn(0x7f648c0d1940 msgr2=0x7f648c0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.289+0000 7f648b200640 1 -- 172.21.15.45:0/3133215392 shutdown_connections 2024-04-16T16:21:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.289+0000 7f648b200640 1 -- 172.21.15.45:0/3133215392 wait complete. 2024-04-16T16:21:06.307 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:21:06.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.531+0000 7f6fe2600640 1 Processor -- start 2024-04-16T16:21:06.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.531+0000 7f6fe2600640 1 -- start start 2024-04-16T16:21:06.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.532+0000 7f6fe2600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6fdc1462e0 0x7f6fdc072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:06.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.532+0000 7f6fe2600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fdc0728c0 0x7f6fdc0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:06.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.532+0000 7f6fe2600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fdc0cf6d0 con 0x7f6fdc0cec00 2024-04-16T16:21:06.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.532+0000 7f6fe2600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6fdc072d20 con 0x7f6fdc1462e0 2024-04-16T16:21:06.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.533+0000 7f6fe2600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6fdc1466c0 con 0x7f6fdc0728c0 2024-04-16T16:21:06.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.533+0000 7f6fdb400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6fdc1462e0 0x7f6fdc072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:06.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.533+0000 7f6fdaa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fdc0728c0 0x7f6fdc0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:06.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.533+0000 7f6fdb400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6fdc1462e0 0x7f6fdc072380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:54284/0 (socket says 172.21.15.45:54284) 2024-04-16T16:21:06.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.533+0000 7f6fdb400640 1 -- 172.21.15.45:0/2898798035 learned_addr learned my addr 172.21.15.45:0/2898798035 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:06.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.534+0000 7f6fdb400640 1 -- 172.21.15.45:0/2898798035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fdc0728c0 msgr2=0x7f6fdc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:06.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.534+0000 7f6fdb400640 1 --2- 172.21.15.45:0/2898798035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fdc0728c0 0x7f6fdc0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.534+0000 7f6fdb400640 1 -- 172.21.15.45:0/2898798035 >> v1:172.21.15.103:6789/0 conn(0x7f6fdc0cec00 legacy=0x7f6fdc15da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:06.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.534+0000 7f6fdb400640 1 -- 172.21.15.45:0/2898798035 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fdc0cf3f0 con 0x7f6fdc1462e0 2024-04-16T16:21:06.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.535+0000 7f6fdb400640 1 --2- 172.21.15.45:0/2898798035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fdc1462e0 0x7f6fdc072380 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc009870 tx=0x7f6fcc02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5ab72083eec177f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:06.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.536+0000 7f6fda000640 1 -- 172.21.15.45:0/2898798035 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc035020 con 0x7f6fdc1462e0 2024-04-16T16:21:06.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.536+0000 7f6fda000640 1 -- 172.21.15.45:0/2898798035 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fcc031640 con 0x7f6fdc1462e0 2024-04-16T16:21:06.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.536+0000 7f6fda000640 1 -- 172.21.15.45:0/2898798035 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc02d040 con 0x7f6fdc1462e0 2024-04-16T16:21:06.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.537+0000 7f6fe2600640 1 -- 172.21.15.45:0/2898798035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fdc1462e0 msgr2=0x7f6fdc072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:06.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.537+0000 7f6fe2600640 1 --2- 172.21.15.45:0/2898798035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fdc1462e0 0x7f6fdc072380 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc009870 tx=0x7f6fcc02b8f0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.537+0000 7f6fe2600640 1 -- 172.21.15.45:0/2898798035 shutdown_connections 2024-04-16T16:21:06.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.537+0000 7f6fe2600640 1 --2- 172.21.15.45:0/2898798035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fdc0728c0 0x7f6fdc0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.537+0000 7f6fe2600640 1 --2- 172.21.15.45:0/2898798035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fdc1462e0 0x7f6fdc072380 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.537+0000 7f6fe2600640 1 -- 172.21.15.45:0/2898798035 >> 172.21.15.45:0/2898798035 conn(0x7f6fdc0b2750 msgr2=0x7f6fdc0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:06.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.537+0000 7f6fe2600640 1 -- 172.21.15.45:0/2898798035 shutdown_connections 2024-04-16T16:21:06.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.537+0000 7f6fe2600640 1 -- 172.21.15.45:0/2898798035 wait complete. 2024-04-16T16:21:06.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.539+0000 7f6fe2600640 1 Processor -- start 2024-04-16T16:21:06.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.539+0000 7f6fe2600640 1 -- start start 2024-04-16T16:21:06.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.540+0000 7f6fe2600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fdc0728c0 0x7f6fdc1e4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:06.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.540+0000 7f6fe2600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fdc0cec00 0x7f6fdc15c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:06.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.540+0000 7f6fe2600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc1462e0 0x7f6fdc13e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:06.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.540+0000 7f6fe2600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6fdc15d540 con 0x7f6fdc1462e0 2024-04-16T16:21:06.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.540+0000 7f6fe2600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6fdc15d260 con 0x7f6fdc0cec00 2024-04-16T16:21:06.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.540+0000 7f6fe2600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6fdc0cf090 con 0x7f6fdc0728c0 2024-04-16T16:21:06.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.540+0000 7f6fdbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc1462e0 0x7f6fdc13e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:06.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.540+0000 7f6fdbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc1462e0 0x7f6fdc13e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48406/0 (socket says 172.21.15.45:48406) 2024-04-16T16:21:06.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.540+0000 7f6fdbe00640 1 -- 172.21.15.45:0/2562815490 learned_addr learned my addr 172.21.15.45:0/2562815490 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:06.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.540+0000 7f6fdb400640 1 --2- 172.21.15.45:0/2562815490 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fdc0728c0 0x7f6fdc1e4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:06.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.540+0000 7f6fdaa00640 1 --2- 172.21.15.45:0/2562815490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fdc0cec00 0x7f6fdc15c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:06.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.541+0000 7f6fdbe00640 1 -- 172.21.15.45:0/2562815490 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fdc0728c0 msgr2=0x7f6fdc1e4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:06.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.541+0000 7f6fdbe00640 1 --2- 172.21.15.45:0/2562815490 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fdc0728c0 0x7f6fdc1e4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.541+0000 7f6fdbe00640 1 -- 172.21.15.45:0/2562815490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fdc0cec00 msgr2=0x7f6fdc15c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:06.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.541+0000 7f6fdbe00640 1 --2- 172.21.15.45:0/2562815490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fdc0cec00 0x7f6fdc15c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.541+0000 7f6fdbe00640 1 -- 172.21.15.45:0/2562815490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fdc13eac0 con 0x7f6fdc1462e0 2024-04-16T16:21:06.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.541+0000 7f6fdaa00640 1 --2- 172.21.15.45:0/2562815490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fdc0cec00 0x7f6fdc15c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:21:06.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.541+0000 7f6fdbe00640 1 --2- 172.21.15.45:0/2562815490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc1462e0 0x7f6fdc13e580 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400c7a0 tx=0x7f6fd400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:06.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.541+0000 7f6fd3400640 1 -- 172.21.15.45:0/2562815490 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4003b70 con 0x7f6fdc1462e0 2024-04-16T16:21:06.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.541+0000 7f6fd3400640 1 -- 172.21.15.45:0/2562815490 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd4003d30 con 0x7f6fdc1462e0 2024-04-16T16:21:06.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.541+0000 7f6fd3400640 1 -- 172.21.15.45:0/2562815490 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd400aa90 con 0x7f6fdc1462e0 2024-04-16T16:21:06.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.541+0000 7f6fe2600640 1 -- 172.21.15.45:0/2562815490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fdc1e54f0 con 0x7f6fdc1462e0 2024-04-16T16:21:06.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.541+0000 7f6fe2600640 1 -- 172.21.15.45:0/2562815490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fdc1e5ab0 con 0x7f6fdc1462e0 2024-04-16T16:21:06.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.543+0000 7f6fe2600640 1 -- 172.21.15.45:0/2562815490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fdc075780 con 0x7f6fdc1462e0 2024-04-16T16:21:06.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.543+0000 7f6fd3400640 1 -- 172.21.15.45:0/2562815490 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 58) ==== 91086+0+0 (secure 0 0 0) 0x7f6fd400ac30 con 0x7f6fdc1462e0 2024-04-16T16:21:06.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.544+0000 7f6fd3400640 1 --2- 172.21.15.45:0/2562815490 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f6fbc06c980 0x7f6fbc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:06.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.545+0000 7f6fd3400640 1 -- 172.21.15.45:0/2562815490 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6397+0+0 (secure 0 0 0) 0x7f6fd408bc60 con 0x7f6fdc1462e0 2024-04-16T16:21:06.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.545+0000 7f6fdb400640 1 --2- 172.21.15.45:0/2562815490 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f6fbc06c980 0x7f6fbc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:06.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.546+0000 7f6fdb400640 1 --2- 172.21.15.45:0/2562815490 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f6fbc06c980 0x7f6fbc06ee40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc036000 tx=0x7f6fcc03c040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:06.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.549+0000 7f6fd3400640 1 -- 172.21.15.45:0/2562815490 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6fdc13eac0 con 0x7f6fdc1462e0 2024-04-16T16:21:06.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.756+0000 7f6fe2600640 1 -- 172.21.15.45:0/2562815490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6fdc0b1370 con 0x7f6fdc1462e0 2024-04-16T16:21:06.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.759+0000 7f6fd3400640 1 -- 172.21.15.45:0/2562815490 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) ==== 74+0+17696 (secure 0 0 0) 0x7f6fd4059e80 con 0x7f6fdc1462e0 2024-04-16T16:21:06.759 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:06.760 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":130,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:21:04.341365+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:20:31.681227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-16T16:20:32.790980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/1715153913":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/4225560501":"2024-04-17T16:21:04.341295+0000","172.21.15.103:6832/549890383":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/1495495989":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/3982280724":"2024-04-17T16:21:04.341295+0000","172.21.15.103:6833/1120913661":"2024-04-17T16:20:50.935324+0000","172.21.15.103:6833/549890383":"2024-04-17T16:21:04.341295+0000","172.21.15.103:6832/1120913661":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1543548744":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/4103173879":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1578451701":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/4177084908":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1961421675":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/2190202958":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/3679729711":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1543358977":"2024-04-17T16:20:50.935324+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:21:06.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.762+0000 7f6fe2600640 1 -- 172.21.15.45:0/2562815490 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f6fbc06c980 msgr2=0x7f6fbc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:06.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.762+0000 7f6fe2600640 1 --2- 172.21.15.45:0/2562815490 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f6fbc06c980 0x7f6fbc06ee40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc036000 tx=0x7f6fcc03c040 comp rx=0 tx=0).stop 2024-04-16T16:21:06.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.762+0000 7f6fe2600640 1 -- 172.21.15.45:0/2562815490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc1462e0 msgr2=0x7f6fdc13e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:06.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.762+0000 7f6fe2600640 1 --2- 172.21.15.45:0/2562815490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc1462e0 0x7f6fdc13e580 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400c7a0 tx=0x7f6fd400cc60 comp rx=0 tx=0).stop 2024-04-16T16:21:06.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.762+0000 7f6fe2600640 1 -- 172.21.15.45:0/2562815490 shutdown_connections 2024-04-16T16:21:06.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.762+0000 7f6fe2600640 1 --2- 172.21.15.45:0/2562815490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc1462e0 0x7f6fdc13e580 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.762+0000 7f6fe2600640 1 --2- 172.21.15.45:0/2562815490 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f6fbc06c980 0x7f6fbc06ee40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.762+0000 7f6fe2600640 1 --2- 172.21.15.45:0/2562815490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fdc0cec00 0x7f6fdc15c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.762+0000 7f6fe2600640 1 --2- 172.21.15.45:0/2562815490 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fdc0728c0 0x7f6fdc1e4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:06.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.762+0000 7f6fe2600640 1 -- 172.21.15.45:0/2562815490 >> 172.21.15.45:0/2562815490 conn(0x7f6fdc0b2750 msgr2=0x7f6fdc0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:06.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.762+0000 7f6fe2600640 1 -- 172.21.15.45:0/2562815490 shutdown_connections 2024-04-16T16:21:06.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:06.762+0000 7f6fe2600640 1 -- 172.21.15.45:0/2562815490 wait complete. 2024-04-16T16:21:06.781 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:20:31.681227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:20:32.790980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:21:06.781 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:21:07.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.006+0000 7fe7bbe00640 1 Processor -- start 2024-04-16T16:21:07.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.007+0000 7fe7bbe00640 1 -- start start 2024-04-16T16:21:07.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.008+0000 7fe7bbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc145fe0 0x7fe7bc0c5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.008+0000 7fe7bbe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7bc0c6210 con 0x7fe7bc160e90 2024-04-16T16:21:07.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.008+0000 7fe7bbe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7bc1603b0 con 0x7fe7bc0cfb50 2024-04-16T16:21:07.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.008+0000 7fe7bbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe7bc160690 con 0x7fe7bc145fe0 2024-04-16T16:21:07.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.009+0000 7fe7b2000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe7bc160e90 0x7fe7bc0cf440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36084/0 (socket says 172.21.15.45:36084) 2024-04-16T16:21:07.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.009+0000 7fe7b2000640 1 -- 172.21.15.45:0/4103380719 learned_addr learned my addr 172.21.15.45:0/4103380719 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:07.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.009+0000 7fe7baa00640 1 --2- 172.21.15.45:0/4103380719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc145fe0 0x7fe7bc0c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.009+0000 7fe7ba000640 1 -- 172.21.15.45:0/4103380719 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 896395819 0 0) 0x7fe7bc0c6210 con 0x7fe7bc160e90 2024-04-16T16:21:07.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.009+0000 7fe7ba000640 1 -- 172.21.15.45:0/4103380719 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7a0003520 con 0x7fe7bc160e90 2024-04-16T16:21:07.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.010+0000 7fe7ba000640 1 -- 172.21.15.45:0/4103380719 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4273475675 0 0) 0x7fe7a0003520 con 0x7fe7bc160e90 2024-04-16T16:21:07.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.010+0000 7fe7ba000640 1 -- 172.21.15.45:0/4103380719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc145fe0 msgr2=0x7fe7bc0c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:07.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.010+0000 7fe7ba000640 1 --2- 172.21.15.45:0/4103380719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc145fe0 0x7fe7bc0c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.010+0000 7fe7ba000640 1 -- 172.21.15.45:0/4103380719 >> v1:172.21.15.103:6789/0 conn(0x7fe7bc0cfb50 legacy=0x7fe7bc06dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.010+0000 7fe7ba000640 1 -- 172.21.15.45:0/4103380719 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7bc160970 con 0x7fe7bc160e90 2024-04-16T16:21:07.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.010+0000 7fe7ba000640 1 -- 172.21.15.45:0/4103380719 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fe7a4001670 con 0x7fe7bc160e90 2024-04-16T16:21:07.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.010+0000 7fe7ba000640 1 -- 172.21.15.45:0/4103380719 >> v1:172.21.15.45:6789/0 conn(0x7fe7bc160e90 legacy=0x7fe7bc0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.010+0000 7fe7ba000640 1 --2- 172.21.15.45:0/4103380719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7a00039e0 0x7fe7a0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.010+0000 7fe7ba000640 1 --2- 172.21.15.45:0/4103380719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc145fe0 0x7fe7a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.010+0000 7fe7ba000640 1 --2- 172.21.15.45:0/4103380719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a0004d40 0x7fe7a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.010+0000 7fe7ba000640 1 -- 172.21.15.45:0/4103380719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc160690 con 0x7fe7a0004d40 2024-04-16T16:21:07.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.010+0000 7fe7ba000640 1 -- 172.21.15.45:0/4103380719 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a000baf0 con 0x7fe7a00039e0 2024-04-16T16:21:07.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.010+0000 7fe7ba000640 1 -- 172.21.15.45:0/4103380719 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe7a000bcf0 con 0x7fe7bc145fe0 2024-04-16T16:21:07.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.010+0000 7fe7bb400640 1 --2- 172.21.15.45:0/4103380719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a0004d40 0x7fe7a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.011+0000 7fe7baa00640 1 --2- 172.21.15.45:0/4103380719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7a00039e0 0x7fe7a0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.011+0000 7fe7bb400640 1 -- 172.21.15.45:0/4103380719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc145fe0 msgr2=0x7fe7a0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:07.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.011+0000 7fe7bb400640 1 --2- 172.21.15.45:0/4103380719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc145fe0 0x7fe7a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.011+0000 7fe7bb400640 1 -- 172.21.15.45:0/4103380719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7a00039e0 msgr2=0x7fe7a0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:07.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.011+0000 7fe7bb400640 1 --2- 172.21.15.45:0/4103380719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7a00039e0 0x7fe7a0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.011+0000 7fe7bb400640 1 -- 172.21.15.45:0/4103380719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a0003520 con 0x7fe7a0004d40 2024-04-16T16:21:07.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.011+0000 7fe7baa00640 1 --2- 172.21.15.45:0/4103380719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7a00039e0 0x7fe7a0003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:07.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.012+0000 7fe7bb400640 1 --2- 172.21.15.45:0/4103380719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a0004d40 0x7fe7a00051c0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4001e50 tx=0x7fe7a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a82abe307de990 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:07.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.012+0000 7fe7ba000640 1 -- 172.21.15.45:0/4103380719 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4001a70 con 0x7fe7a0004d40 2024-04-16T16:21:07.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.012+0000 7fe7ba000640 1 -- 172.21.15.45:0/4103380719 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7b4003b70 con 0x7fe7a0004d40 2024-04-16T16:21:07.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.012+0000 7fe7ba000640 1 -- 172.21.15.45:0/4103380719 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b40050d0 con 0x7fe7a0004d40 2024-04-16T16:21:07.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.013+0000 7fe7bbe00640 1 -- 172.21.15.45:0/4103380719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a0004d40 msgr2=0x7fe7a00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.013+0000 7fe7bbe00640 1 --2- 172.21.15.45:0/4103380719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a0004d40 0x7fe7a00051c0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4001e50 tx=0x7fe7a0005830 comp rx=0 tx=0).stop 2024-04-16T16:21:07.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.013+0000 7fe7bb400640 1 -- 172.21.15.45:0/4103380719 reap_dead start 2024-04-16T16:21:07.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.013+0000 7fe7bbe00640 1 -- 172.21.15.45:0/4103380719 shutdown_connections 2024-04-16T16:21:07.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.013+0000 7fe7bbe00640 1 -- 172.21.15.45:0/4103380719 >> 172.21.15.45:0/4103380719 conn(0x7fe7bc067e00 msgr2=0x7fe7bc06b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:07.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.013+0000 7fe7bbe00640 1 -- 172.21.15.45:0/4103380719 shutdown_connections 2024-04-16T16:21:07.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.013+0000 7fe7bbe00640 1 -- 172.21.15.45:0/4103380719 wait complete. 2024-04-16T16:21:07.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.015+0000 7fe7bbe00640 1 Processor -- start 2024-04-16T16:21:07.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.015+0000 7fe7bbe00640 1 -- start start 2024-04-16T16:21:07.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.016+0000 7fe7bbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7bc15c320 0x7fe7bc0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.016+0000 7fe7bbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc1402d0 0x7fe7bc0c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.016+0000 7fe7bbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7bc0c94c0 0x7fe7bc0c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.016+0000 7fe7bbe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc146540 con 0x7fe7bc0c94c0 2024-04-16T16:21:07.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.016+0000 7fe7bbe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc1463c0 con 0x7fe7bc15c320 2024-04-16T16:21:07.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.016+0000 7fe7bbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe7bc06e6c0 con 0x7fe7bc1402d0 2024-04-16T16:21:07.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.016+0000 7fe7bb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7bc0c94c0 0x7fe7bc0c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.016+0000 7fe7bb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7bc0c94c0 0x7fe7bc0c7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48430/0 (socket says 172.21.15.45:48430) 2024-04-16T16:21:07.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.016+0000 7fe7bb400640 1 -- 172.21.15.45:0/4437006 learned_addr learned my addr 172.21.15.45:0/4437006 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:07.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.016+0000 7fe7bb400640 1 -- 172.21.15.45:0/4437006 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc1402d0 msgr2=0x7fe7bc0c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.017+0000 7fe7b2000640 1 --2- 172.21.15.45:0/4437006 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc1402d0 0x7fe7bc0c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.017+0000 7fe7baa00640 1 --2- 172.21.15.45:0/4437006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7bc15c320 0x7fe7bc0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.017+0000 7fe7bb400640 1 --2- 172.21.15.45:0/4437006 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc1402d0 0x7fe7bc0c8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.017+0000 7fe7bb400640 1 -- 172.21.15.45:0/4437006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7bc15c320 msgr2=0x7fe7bc0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.017+0000 7fe7bb400640 1 --2- 172.21.15.45:0/4437006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7bc15c320 0x7fe7bc0cba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.017+0000 7fe7bb400640 1 -- 172.21.15.45:0/4437006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7bc15c700 con 0x7fe7bc0c94c0 2024-04-16T16:21:07.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.017+0000 7fe7bb400640 1 --2- 172.21.15.45:0/4437006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7bc0c94c0 0x7fe7bc0c7980 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4001f40 tx=0x7fe7b40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:07.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.017+0000 7fe7b3400640 1 -- 172.21.15.45:0/4437006 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4001a70 con 0x7fe7bc0c94c0 2024-04-16T16:21:07.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.017+0000 7fe7b3400640 1 -- 172.21.15.45:0/4437006 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7b4003970 con 0x7fe7bc0c94c0 2024-04-16T16:21:07.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.017+0000 7fe7bbe00640 1 -- 172.21.15.45:0/4437006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7bc0cc1f0 con 0x7fe7bc0c94c0 2024-04-16T16:21:07.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.017+0000 7fe7b3400640 1 -- 172.21.15.45:0/4437006 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4007db0 con 0x7fe7bc0c94c0 2024-04-16T16:21:07.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.017+0000 7fe7bbe00640 1 -- 172.21.15.45:0/4437006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7bc0cc7b0 con 0x7fe7bc0c94c0 2024-04-16T16:21:07.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.019+0000 7fe7bbe00640 1 -- 172.21.15.45:0/4437006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7bc075780 con 0x7fe7bc0c94c0 2024-04-16T16:21:07.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.020+0000 7fe7b3400640 1 -- 172.21.15.45:0/4437006 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 58) ==== 91086+0+0 (secure 0 0 0) 0x7fe7b4007030 con 0x7fe7bc0c94c0 2024-04-16T16:21:07.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.021+0000 7fe7b3400640 1 --2- 172.21.15.45:0/4437006 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fe79806c980 0x7fe79806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.021+0000 7fe7b3400640 1 -- 172.21.15.45:0/4437006 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6397+0+0 (secure 0 0 0) 0x7fe7b40ad1e0 con 0x7fe7bc0c94c0 2024-04-16T16:21:07.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.022+0000 7fe7baa00640 1 --2- 172.21.15.45:0/4437006 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fe79806c980 0x7fe79806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.023+0000 7fe7baa00640 1 --2- 172.21.15.45:0/4437006 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fe79806c980 0x7fe79806ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0099e0 tx=0x7fe7ac008040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:07.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.025+0000 7fe7b3400640 1 -- 172.21.15.45:0/4437006 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe7b4033020 con 0x7fe7bc0c94c0 2024-04-16T16:21:07.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.229+0000 7fe7bbe00640 1 -- 172.21.15.45:0/4437006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe7bc06c110 con 0x7fe7bc0c94c0 2024-04-16T16:21:07.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.229+0000 7fe7b3400640 1 -- 172.21.15.45:0/4437006 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v130) ==== 93+0+10 (secure 0 0 0) 0x7fe7b407b160 con 0x7fe7bc0c94c0 2024-04-16T16:21:07.229 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:21:07.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.232+0000 7fe7bbe00640 1 -- 172.21.15.45:0/4437006 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fe79806c980 msgr2=0x7fe79806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.232+0000 7fe7bbe00640 1 --2- 172.21.15.45:0/4437006 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fe79806c980 0x7fe79806ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0099e0 tx=0x7fe7ac008040 comp rx=0 tx=0).stop 2024-04-16T16:21:07.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.232+0000 7fe7bbe00640 1 -- 172.21.15.45:0/4437006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7bc0c94c0 msgr2=0x7fe7bc0c7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.232+0000 7fe7bbe00640 1 --2- 172.21.15.45:0/4437006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7bc0c94c0 0x7fe7bc0c7980 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4001f40 tx=0x7fe7b40014a0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.232+0000 7fe7bbe00640 1 -- 172.21.15.45:0/4437006 shutdown_connections 2024-04-16T16:21:07.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.232+0000 7fe7bbe00640 1 --2- 172.21.15.45:0/4437006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7bc0c94c0 0x7fe7bc0c7980 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.232+0000 7fe7bbe00640 1 --2- 172.21.15.45:0/4437006 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc1402d0 0x7fe7bc0c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.232+0000 7fe7bbe00640 1 --2- 172.21.15.45:0/4437006 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fe79806c980 0x7fe79806ee40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.232+0000 7fe7bbe00640 1 --2- 172.21.15.45:0/4437006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7bc15c320 0x7fe7bc0cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.232+0000 7fe7bbe00640 1 -- 172.21.15.45:0/4437006 >> 172.21.15.45:0/4437006 conn(0x7fe7bc067e00 msgr2=0x7fe7bc15dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:07.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.232+0000 7fe7bbe00640 1 -- 172.21.15.45:0/4437006 shutdown_connections 2024-04-16T16:21:07.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.232+0000 7fe7bbe00640 1 -- 172.21.15.45:0/4437006 wait complete. 2024-04-16T16:21:07.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:21:07.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.478+0000 7fa4b4200640 1 Processor -- start 2024-04-16T16:21:07.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.479+0000 7fa4b4200640 1 -- start start 2024-04-16T16:21:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.480+0000 7fa4b4200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa4ac0b5a30 0x7fa4ac1625e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.480+0000 7fa4b4200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac06e290 0x7fa4ac06e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.480+0000 7fa4b4200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4ac0b5e10 con 0x7fa4ac0d1030 2024-04-16T16:21:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.480+0000 7fa4b4200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa4ac15f3d0 con 0x7fa4ac0b5a30 2024-04-16T16:21:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.480+0000 7fa4b4200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa4ac15ff70 con 0x7fa4ac06e290 2024-04-16T16:21:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.481+0000 7fa4b0e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa4ac0d1030 0x7fa4ac15f860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36090/0 (socket says 172.21.15.45:36090) 2024-04-16T16:21:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.481+0000 7fa4b0e00640 1 -- 172.21.15.45:0/400206140 learned_addr learned my addr 172.21.15.45:0/400206140 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.481+0000 7fa4b1800640 1 --2- 172.21.15.45:0/400206140 >> v2:172.21.15.103:3300/0 conn(0x7fa4ac0b5a30 0x7fa4ac1625e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.481+0000 7fa4b2200640 1 --2- 172.21.15.45:0/400206140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac06e290 0x7fa4ac06e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.481+0000 7fa4abe00640 1 -- 172.21.15.45:0/400206140 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3852208461 0 0) 0x7fa4ac0b5e10 con 0x7fa4ac0d1030 2024-04-16T16:21:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.481+0000 7fa4abe00640 1 -- 172.21.15.45:0/400206140 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa494003520 con 0x7fa4ac0d1030 2024-04-16T16:21:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.482+0000 7fa4abe00640 1 -- 172.21.15.45:0/400206140 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1518010092 0 0) 0x7fa494003520 con 0x7fa4ac0d1030 2024-04-16T16:21:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.482+0000 7fa4abe00640 1 -- 172.21.15.45:0/400206140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac06e290 msgr2=0x7fa4ac06e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.482+0000 7fa4abe00640 1 --2- 172.21.15.45:0/400206140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac06e290 0x7fa4ac06e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.482+0000 7fa4abe00640 1 -- 172.21.15.45:0/400206140 >> v2:172.21.15.103:3300/0 conn(0x7fa4ac0b5a30 msgr2=0x7fa4ac1625e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.482+0000 7fa4abe00640 1 --2- 172.21.15.45:0/400206140 >> v2:172.21.15.103:3300/0 conn(0x7fa4ac0b5a30 0x7fa4ac1625e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.482+0000 7fa4abe00640 1 -- 172.21.15.45:0/400206140 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4ac162e70 con 0x7fa4ac0d1030 2024-04-16T16:21:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.482+0000 7fa4abe00640 1 -- 172.21.15.45:0/400206140 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fa4a0001670 con 0x7fa4ac0d1030 2024-04-16T16:21:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.482+0000 7fa4abe00640 1 -- 172.21.15.45:0/400206140 >> v1:172.21.15.45:6789/0 conn(0x7fa4ac0d1030 legacy=0x7fa4ac15f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.482+0000 7fa4abe00640 1 --2- 172.21.15.45:0/400206140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4940039b0 0x7fa494003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.482+0000 7fa4abe00640 1 --2- 172.21.15.45:0/400206140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac06e290 0x7fa494004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.482+0000 7fa4abe00640 1 --2- 172.21.15.45:0/400206140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa494004d40 0x7fa4940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.482+0000 7fa4abe00640 1 -- 172.21.15.45:0/400206140 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ac15f3d0 con 0x7fa4940039b0 2024-04-16T16:21:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.482+0000 7fa4abe00640 1 -- 172.21.15.45:0/400206140 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ac15ff70 con 0x7fa494004d40 2024-04-16T16:21:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.482+0000 7fa4abe00640 1 -- 172.21.15.45:0/400206140 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa49400ba40 con 0x7fa4ac06e290 2024-04-16T16:21:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.482+0000 7fa4b1800640 1 --2- 172.21.15.45:0/400206140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4940039b0 0x7fa494003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.483+0000 7fa4b0e00640 1 --2- 172.21.15.45:0/400206140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac06e290 0x7fa494004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.483+0000 7fa4b1800640 1 -- 172.21.15.45:0/400206140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac06e290 msgr2=0x7fa494004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.483+0000 7fa4b1800640 1 --2- 172.21.15.45:0/400206140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac06e290 0x7fa494004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.483+0000 7fa4b1800640 1 -- 172.21.15.45:0/400206140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa494004d40 msgr2=0x7fa4940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.483+0000 7fa4b1800640 1 --2- 172.21.15.45:0/400206140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa494004d40 0x7fa4940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.483+0000 7fa4b1800640 1 -- 172.21.15.45:0/400206140 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa494003520 con 0x7fa4940039b0 2024-04-16T16:21:07.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.484+0000 7fa4b1800640 1 --2- 172.21.15.45:0/400206140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4940039b0 0x7fa494003dd0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009ac0 tx=0x7fa494005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a15ca8410f02dc server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:07.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.484+0000 7fa4abe00640 1 -- 172.21.15.45:0/400206140 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c039070 con 0x7fa4940039b0 2024-04-16T16:21:07.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.485+0000 7fa4abe00640 1 -- 172.21.15.45:0/400206140 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa49c02c710 con 0x7fa4940039b0 2024-04-16T16:21:07.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.485+0000 7fa4abe00640 1 -- 172.21.15.45:0/400206140 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c009690 con 0x7fa4940039b0 2024-04-16T16:21:07.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.485+0000 7fa4b4200640 1 -- 172.21.15.45:0/400206140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4940039b0 msgr2=0x7fa494003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.485+0000 7fa4b4200640 1 --2- 172.21.15.45:0/400206140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4940039b0 0x7fa494003dd0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009ac0 tx=0x7fa494005830 comp rx=0 tx=0).stop 2024-04-16T16:21:07.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.485+0000 7fa4b2200640 1 -- 172.21.15.45:0/400206140 reap_dead start 2024-04-16T16:21:07.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.485+0000 7fa4b4200640 1 -- 172.21.15.45:0/400206140 shutdown_connections 2024-04-16T16:21:07.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.485+0000 7fa4b4200640 1 -- 172.21.15.45:0/400206140 >> 172.21.15.45:0/400206140 conn(0x7fa4ac0b3970 msgr2=0x7fa4ac0b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:07.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.485+0000 7fa4b4200640 1 -- 172.21.15.45:0/400206140 shutdown_connections 2024-04-16T16:21:07.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.486+0000 7fa4b4200640 1 -- 172.21.15.45:0/400206140 wait complete. 2024-04-16T16:21:07.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.487+0000 7fa4b4200640 1 Processor -- start 2024-04-16T16:21:07.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.487+0000 7fa4b4200640 1 -- start start 2024-04-16T16:21:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.488+0000 7fa4b4200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4940039b0 0x7fa4ac147810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.488+0000 7fa4b4200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac161be0 0x7fa4ac156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.488+0000 7fa4b4200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac0b2720 0x7fa4ac0b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.488+0000 7fa4b4200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ac15eff0 con 0x7fa4940039b0 2024-04-16T16:21:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.488+0000 7fa4b4200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ac163430 con 0x7fa4ac161be0 2024-04-16T16:21:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.488+0000 7fa4b4200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa4ac162b20 con 0x7fa4ac0b2720 2024-04-16T16:21:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.489+0000 7fa4b1800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4940039b0 0x7fa4ac147810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.489+0000 7fa4b1800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4940039b0 0x7fa4ac147810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48446/0 (socket says 172.21.15.45:48446) 2024-04-16T16:21:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.489+0000 7fa4b1800640 1 -- 172.21.15.45:0/1454305460 learned_addr learned my addr 172.21.15.45:0/1454305460 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.489+0000 7fa4b1800640 1 -- 172.21.15.45:0/1454305460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac0b2720 msgr2=0x7fa4ac0b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.489+0000 7fa4b1800640 1 --2- 172.21.15.45:0/1454305460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac0b2720 0x7fa4ac0b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.489+0000 7fa4b1800640 1 -- 172.21.15.45:0/1454305460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac161be0 msgr2=0x7fa4ac156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.489+0000 7fa4b1800640 1 --2- 172.21.15.45:0/1454305460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac161be0 0x7fa4ac156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.489+0000 7fa4b1800640 1 -- 172.21.15.45:0/1454305460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4ac144d90 con 0x7fa4940039b0 2024-04-16T16:21:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.489+0000 7fa4b1800640 1 --2- 172.21.15.45:0/1454305460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4940039b0 0x7fa4ac147810 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009be0 tx=0x7fa49c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.490+0000 7fa4a9600640 1 -- 172.21.15.45:0/1454305460 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c042020 con 0x7fa4940039b0 2024-04-16T16:21:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.490+0000 7fa4a9600640 1 -- 172.21.15.45:0/1454305460 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa49c03a040 con 0x7fa4940039b0 2024-04-16T16:21:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.490+0000 7fa4a9600640 1 -- 172.21.15.45:0/1454305460 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c039070 con 0x7fa4940039b0 2024-04-16T16:21:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.490+0000 7fa4b4200640 1 -- 172.21.15.45:0/1454305460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4ac145070 con 0x7fa4940039b0 2024-04-16T16:21:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.490+0000 7fa4b4200640 1 -- 172.21.15.45:0/1454305460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4ac145630 con 0x7fa4940039b0 2024-04-16T16:21:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.491+0000 7fa4b4200640 1 -- 172.21.15.45:0/1454305460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4ac1621d0 con 0x7fa4940039b0 2024-04-16T16:21:07.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.492+0000 7fa4a9600640 1 -- 172.21.15.45:0/1454305460 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 58) ==== 91086+0+0 (secure 0 0 0) 0x7fa49c045020 con 0x7fa4940039b0 2024-04-16T16:21:07.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.494+0000 7fa4a9600640 1 --2- 172.21.15.45:0/1454305460 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa49006c980 0x7fa49006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.494+0000 7fa4a9600640 1 -- 172.21.15.45:0/1454305460 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6397+0+0 (secure 0 0 0) 0x7fa49c0ad2e0 con 0x7fa4940039b0 2024-04-16T16:21:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.494+0000 7fa4b0e00640 1 --2- 172.21.15.45:0/1454305460 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa49006c980 0x7fa49006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.495+0000 7fa4b0e00640 1 --2- 172.21.15.45:0/1454305460 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa49006c980 0x7fa49006ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0001eb0 tx=0x7fa4a0009040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:07.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.498+0000 7fa4a9600640 1 -- 172.21.15.45:0/1454305460 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa49c04c070 con 0x7fa4940039b0 2024-04-16T16:21:07.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.702+0000 7fa4b4200640 1 -- 172.21.15.45:0/1454305460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa4ac153cd0 con 0x7fa4940039b0 2024-04-16T16:21:07.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.703+0000 7fa4a9600640 1 -- 172.21.15.45:0/1454305460 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v130) ==== 104+0+11 (secure 0 0 0) 0x7fa49c07b470 con 0x7fa4940039b0 2024-04-16T16:21:07.703 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:21:07.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.706+0000 7fa4b4200640 1 -- 172.21.15.45:0/1454305460 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa49006c980 msgr2=0x7fa49006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.706+0000 7fa4b4200640 1 --2- 172.21.15.45:0/1454305460 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa49006c980 0x7fa49006ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0001eb0 tx=0x7fa4a0009040 comp rx=0 tx=0).stop 2024-04-16T16:21:07.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.706+0000 7fa4b4200640 1 -- 172.21.15.45:0/1454305460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4940039b0 msgr2=0x7fa4ac147810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.706+0000 7fa4b4200640 1 --2- 172.21.15.45:0/1454305460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4940039b0 0x7fa4ac147810 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009be0 tx=0x7fa49c004730 comp rx=0 tx=0).stop 2024-04-16T16:21:07.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.706+0000 7fa4b4200640 1 -- 172.21.15.45:0/1454305460 shutdown_connections 2024-04-16T16:21:07.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.706+0000 7fa4b4200640 1 --2- 172.21.15.45:0/1454305460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4ac0b2720 0x7fa4ac0b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.706+0000 7fa4b4200640 1 --2- 172.21.15.45:0/1454305460 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa49006c980 0x7fa49006ee40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.706+0000 7fa4b4200640 1 --2- 172.21.15.45:0/1454305460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac161be0 0x7fa4ac156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.706+0000 7fa4b4200640 1 --2- 172.21.15.45:0/1454305460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4940039b0 0x7fa4ac147810 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.706+0000 7fa4b4200640 1 -- 172.21.15.45:0/1454305460 >> 172.21.15.45:0/1454305460 conn(0x7fa4ac0b3970 msgr2=0x7fa4ac06e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:07.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.706+0000 7fa4b4200640 1 -- 172.21.15.45:0/1454305460 shutdown_connections 2024-04-16T16:21:07.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.707+0000 7fa4b4200640 1 -- 172.21.15.45:0/1454305460 wait complete. 2024-04-16T16:21:07.726 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:21:07.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.950+0000 7fee33200640 1 Processor -- start 2024-04-16T16:21:07.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.950+0000 7fee33200640 1 -- start start 2024-04-16T16:21:07.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.952+0000 7fee33200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fee341461b0 0x7fee34160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.952+0000 7fee33200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee340b58f0 0x7fee340b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.952+0000 7fee33200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee34146590 con 0x7fee340cf440 2024-04-16T16:21:07.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.952+0000 7fee33200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fee3406dfd0 con 0x7fee341461b0 2024-04-16T16:21:07.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.952+0000 7fee33200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fee340b6290 con 0x7fee340b58f0 2024-04-16T16:21:07.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.953+0000 7fee29400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee340b58f0 0x7fee340b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.953+0000 7fee29400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee340b58f0 0x7fee340b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:59822/0 (socket says 172.21.15.45:59822) 2024-04-16T16:21:07.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.953+0000 7fee29400640 1 -- 172.21.15.45:0/3173118548 learned_addr learned my addr 172.21.15.45:0/3173118548 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:07.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.953+0000 7fee31e00640 1 --2- 172.21.15.45:0/3173118548 >> v2:172.21.15.103:3300/0 conn(0x7fee341461b0 0x7fee34160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.953+0000 7fee29400640 1 -- 172.21.15.45:0/3173118548 >> v2:172.21.15.103:3300/0 conn(0x7fee341461b0 msgr2=0x7fee34160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:07.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.953+0000 7fee29400640 1 --2- 172.21.15.45:0/3173118548 >> v2:172.21.15.103:3300/0 conn(0x7fee341461b0 0x7fee34160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.953+0000 7fee29400640 1 -- 172.21.15.45:0/3173118548 >> v1:172.21.15.103:6789/0 conn(0x7fee340cf440 legacy=0x7fee340cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.953+0000 7fee29400640 1 -- 172.21.15.45:0/3173118548 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee340c5750 con 0x7fee340b58f0 2024-04-16T16:21:07.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.955+0000 7fee29400640 1 --2- 172.21.15.45:0/3173118548 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee340b58f0 0x7fee340b5d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fee1c009870 tx=0x7fee1c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ce5377825a71ed0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:07.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.956+0000 7fee31400640 1 -- 172.21.15.45:0/3173118548 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c034070 con 0x7fee340b58f0 2024-04-16T16:21:07.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.956+0000 7fee31400640 1 -- 172.21.15.45:0/3173118548 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee1c030840 con 0x7fee340b58f0 2024-04-16T16:21:07.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.956+0000 7fee31400640 1 -- 172.21.15.45:0/3173118548 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c030c10 con 0x7fee340b58f0 2024-04-16T16:21:07.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.956+0000 7fee33200640 1 -- 172.21.15.45:0/3173118548 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee340b58f0 msgr2=0x7fee340b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.956+0000 7fee33200640 1 --2- 172.21.15.45:0/3173118548 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee340b58f0 0x7fee340b5d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fee1c009870 tx=0x7fee1c02b980 comp rx=0 tx=0).stop 2024-04-16T16:21:07.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.956+0000 7fee33200640 1 -- 172.21.15.45:0/3173118548 shutdown_connections 2024-04-16T16:21:07.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.956+0000 7fee33200640 1 --2- 172.21.15.45:0/3173118548 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee340b58f0 0x7fee340b5d50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.956+0000 7fee33200640 1 --2- 172.21.15.45:0/3173118548 >> v2:172.21.15.103:3300/0 conn(0x7fee341461b0 0x7fee34160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.956+0000 7fee33200640 1 -- 172.21.15.45:0/3173118548 >> 172.21.15.45:0/3173118548 conn(0x7fee3406b930 msgr2=0x7fee3406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.957+0000 7fee33200640 1 -- 172.21.15.45:0/3173118548 shutdown_connections 2024-04-16T16:21:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.957+0000 7fee33200640 1 -- 172.21.15.45:0/3173118548 wait complete. 2024-04-16T16:21:07.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.958+0000 7fee33200640 1 Processor -- start 2024-04-16T16:21:07.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.958+0000 7fee33200640 1 -- start start 2024-04-16T16:21:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.959+0000 7fee33200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee340b58f0 0x7fee340cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.959+0000 7fee33200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee340cf440 0x7fee341e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.960+0000 7fee33200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee341461b0 0x7fee341437b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.960+0000 7fee33200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fee340c59f0 con 0x7fee340b58f0 2024-04-16T16:21:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.960+0000 7fee33200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee34161230 con 0x7fee341461b0 2024-04-16T16:21:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.960+0000 7fee33200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fee340c5cd0 con 0x7fee340cf440 2024-04-16T16:21:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.960+0000 7fee31e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee340b58f0 0x7fee340cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.960+0000 7fee31e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee340b58f0 0x7fee340cdc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48460/0 (socket says 172.21.15.45:48460) 2024-04-16T16:21:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.960+0000 7fee31e00640 1 -- 172.21.15.45:0/3297114877 learned_addr learned my addr 172.21.15.45:0/3297114877 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.960+0000 7fee29400640 1 --2- 172.21.15.45:0/3297114877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee340cf440 0x7fee341e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.960+0000 7fee32800640 1 --2- 172.21.15.45:0/3297114877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee341461b0 0x7fee341437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.960+0000 7fee31e00640 1 -- 172.21.15.45:0/3297114877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee340cf440 msgr2=0x7fee341e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.960+0000 7fee31e00640 1 --2- 172.21.15.45:0/3297114877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee340cf440 0x7fee341e8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.960+0000 7fee31e00640 1 -- 172.21.15.45:0/3297114877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee341461b0 msgr2=0x7fee341437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.960+0000 7fee31e00640 1 --2- 172.21.15.45:0/3297114877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee341461b0 0x7fee341437b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.960+0000 7fee31e00640 1 -- 172.21.15.45:0/3297114877 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee340ce340 con 0x7fee340b58f0 2024-04-16T16:21:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.961+0000 7fee31e00640 1 --2- 172.21.15.45:0/3297114877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee340b58f0 0x7fee340cdc40 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fee2400efc0 tx=0x7fee2400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.961+0000 7fee2aa00640 1 -- 172.21.15.45:0/3297114877 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee24018020 con 0x7fee340b58f0 2024-04-16T16:21:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.961+0000 7fee2aa00640 1 -- 172.21.15.45:0/3297114877 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee24014480 con 0x7fee340b58f0 2024-04-16T16:21:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.961+0000 7fee2aa00640 1 -- 172.21.15.45:0/3297114877 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee24014850 con 0x7fee340b58f0 2024-04-16T16:21:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.961+0000 7fee33200640 1 -- 172.21.15.45:0/3297114877 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee340ce620 con 0x7fee340b58f0 2024-04-16T16:21:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.961+0000 7fee33200640 1 -- 172.21.15.45:0/3297114877 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee340cebe0 con 0x7fee340b58f0 2024-04-16T16:21:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.963+0000 7fee33200640 1 -- 172.21.15.45:0/3297114877 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee34075780 con 0x7fee340b58f0 2024-04-16T16:21:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.963+0000 7fee2aa00640 1 -- 172.21.15.45:0/3297114877 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 58) ==== 91086+0+0 (secure 0 0 0) 0x7fee24005d10 con 0x7fee340b58f0 2024-04-16T16:21:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.964+0000 7fee2aa00640 1 --2- 172.21.15.45:0/3297114877 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fee1006c980 0x7fee1006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.964+0000 7fee2aa00640 1 -- 172.21.15.45:0/3297114877 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6397+0+0 (secure 0 0 0) 0x7fee2408cb50 con 0x7fee340b58f0 2024-04-16T16:21:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.965+0000 7fee29400640 1 --2- 172.21.15.45:0/3297114877 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fee1006c980 0x7fee1006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.966+0000 7fee29400640 1 --2- 172.21.15.45:0/3297114877 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fee1006c980 0x7fee1006ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fee1c036000 tx=0x7fee1c03b040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:07.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:07.969+0000 7fee2aa00640 1 -- 172.21.15.45:0/3297114877 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fee24022070 con 0x7fee340b58f0 2024-04-16T16:21:08.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.172+0000 7fee33200640 1 -- 172.21.15.45:0/3297114877 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fee34143590 con 0x7fee340b58f0 2024-04-16T16:21:08.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.173+0000 7fee2aa00640 1 -- 172.21.15.45:0/3297114877 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) ==== 100+0+11 (secure 0 0 0) 0x7fee24010040 con 0x7fee340b58f0 2024-04-16T16:21:08.173 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:21:08.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.176+0000 7fee33200640 1 -- 172.21.15.45:0/3297114877 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fee1006c980 msgr2=0x7fee1006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:08.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.176+0000 7fee33200640 1 --2- 172.21.15.45:0/3297114877 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fee1006c980 0x7fee1006ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fee1c036000 tx=0x7fee1c03b040 comp rx=0 tx=0).stop 2024-04-16T16:21:08.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.176+0000 7fee33200640 1 -- 172.21.15.45:0/3297114877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee340b58f0 msgr2=0x7fee340cdc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:08.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.176+0000 7fee33200640 1 --2- 172.21.15.45:0/3297114877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee340b58f0 0x7fee340cdc40 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fee2400efc0 tx=0x7fee2400c490 comp rx=0 tx=0).stop 2024-04-16T16:21:08.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.176+0000 7fee33200640 1 -- 172.21.15.45:0/3297114877 shutdown_connections 2024-04-16T16:21:08.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.176+0000 7fee33200640 1 --2- 172.21.15.45:0/3297114877 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fee1006c980 0x7fee1006ee40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.176+0000 7fee33200640 1 --2- 172.21.15.45:0/3297114877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee341461b0 0x7fee341437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.176+0000 7fee33200640 1 --2- 172.21.15.45:0/3297114877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee340cf440 0x7fee341e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.176+0000 7fee33200640 1 --2- 172.21.15.45:0/3297114877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee340b58f0 0x7fee340cdc40 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.176+0000 7fee33200640 1 -- 172.21.15.45:0/3297114877 >> 172.21.15.45:0/3297114877 conn(0x7fee3406b930 msgr2=0x7fee3415cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:08.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.176+0000 7fee33200640 1 -- 172.21.15.45:0/3297114877 shutdown_connections 2024-04-16T16:21:08.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.176+0000 7fee33200640 1 -- 172.21.15.45:0/3297114877 wait complete. 2024-04-16T16:21:08.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:21:08.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.412+0000 7f3264600640 1 Processor -- start 2024-04-16T16:21:08.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.412+0000 7f3264600640 1 -- start start 2024-04-16T16:21:08.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.414+0000 7f3264600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f325c15ed60 0x7f325c0ba2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:08.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.414+0000 7f3264600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f325c0cf800 0x7f325c0d3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:08.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.414+0000 7f3264600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f325c15f140 con 0x7f325c0ba7f0 2024-04-16T16:21:08.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.414+0000 7f3264600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f325c0d34f0 con 0x7f325c0cf800 2024-04-16T16:21:08.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.414+0000 7f3264600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f325c0cfc90 con 0x7f325c15ed60 2024-04-16T16:21:08.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.414+0000 7f3261200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f325c0cf800 0x7f325c0d3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:08.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.414+0000 7f3261200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f325c0cf800 0x7f325c0d3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48470/0 (socket says 172.21.15.45:48470) 2024-04-16T16:21:08.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.414+0000 7f3261200640 1 -- 172.21.15.45:0/2485708962 learned_addr learned my addr 172.21.15.45:0/2485708962 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:08.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.414+0000 7f3261c00640 1 --2- 172.21.15.45:0/2485708962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f325c15ed60 0x7f325c0ba2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:08.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.415+0000 7f3261200640 1 -- 172.21.15.45:0/2485708962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f325c15ed60 msgr2=0x7f325c0ba2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:08.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.415+0000 7f3261200640 1 --2- 172.21.15.45:0/2485708962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f325c15ed60 0x7f325c0ba2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.415+0000 7f3261200640 1 -- 172.21.15.45:0/2485708962 >> v1:172.21.15.45:6789/0 conn(0x7f325c0ba7f0 legacy=0x7f325c0b7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:08.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.415+0000 7f3261200640 1 -- 172.21.15.45:0/2485708962 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f325c0d3790 con 0x7f325c0cf800 2024-04-16T16:21:08.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.416+0000 7f3261200640 1 --2- 172.21.15.45:0/2485708962 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f325c0cf800 0x7f325c0d3f90 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f3248009870 tx=0x7f324802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b242f4edcab7754 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:08.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.416+0000 7f325be00640 1 -- 172.21.15.45:0/2485708962 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324802f840 con 0x7f325c0cf800 2024-04-16T16:21:08.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.416+0000 7f325be00640 1 -- 172.21.15.45:0/2485708962 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f324802f9e0 con 0x7f325c0cf800 2024-04-16T16:21:08.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.416+0000 7f325be00640 1 -- 172.21.15.45:0/2485708962 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324802fdb0 con 0x7f325c0cf800 2024-04-16T16:21:08.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.417+0000 7f3264600640 1 -- 172.21.15.45:0/2485708962 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f325c0cf800 msgr2=0x7f325c0d3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:08.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.417+0000 7f3264600640 1 --2- 172.21.15.45:0/2485708962 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f325c0cf800 0x7f325c0d3f90 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f3248009870 tx=0x7f324802ba20 comp rx=0 tx=0).stop 2024-04-16T16:21:08.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.417+0000 7f3264600640 1 -- 172.21.15.45:0/2485708962 shutdown_connections 2024-04-16T16:21:08.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.417+0000 7f3264600640 1 --2- 172.21.15.45:0/2485708962 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f325c0cf800 0x7f325c0d3f90 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.417+0000 7f3264600640 1 --2- 172.21.15.45:0/2485708962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f325c15ed60 0x7f325c0ba2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.417+0000 7f3264600640 1 -- 172.21.15.45:0/2485708962 >> 172.21.15.45:0/2485708962 conn(0x7f325c073480 msgr2=0x7f325c073890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:08.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.417+0000 7f3264600640 1 -- 172.21.15.45:0/2485708962 shutdown_connections 2024-04-16T16:21:08.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.417+0000 7f3264600640 1 -- 172.21.15.45:0/2485708962 wait complete. 2024-04-16T16:21:08.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.419+0000 7f3264600640 1 Processor -- start 2024-04-16T16:21:08.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.419+0000 7f3264600640 1 -- start start 2024-04-16T16:21:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.420+0000 7f3264600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f325c0ba7f0 0x7f325c1f7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.420+0000 7f3264600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f325c15ed60 0x7f325c0b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.420+0000 7f3264600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f325c15dd60 0x7f325c0cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.420+0000 7f3264600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f325c0d00f0 con 0x7f325c15dd60 2024-04-16T16:21:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.420+0000 7f3264600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f325c0cff70 con 0x7f325c15ed60 2024-04-16T16:21:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.420+0000 7f3264600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f325c0b8160 con 0x7f325c0ba7f0 2024-04-16T16:21:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.420+0000 7f3262600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f325c15dd60 0x7f325c0cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.420+0000 7f3262600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f325c15dd60 0x7f325c0cc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48472/0 (socket says 172.21.15.45:48472) 2024-04-16T16:21:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.420+0000 7f3262600640 1 -- 172.21.15.45:0/1821658071 learned_addr learned my addr 172.21.15.45:0/1821658071 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:08.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.421+0000 7f3261c00640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f325c0ba7f0 0x7f325c1f7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:08.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.421+0000 7f3261200640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f325c15ed60 0x7f325c0b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:08.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.421+0000 7f3262600640 1 -- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f325c0ba7f0 msgr2=0x7f325c1f7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:08.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.421+0000 7f3262600640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f325c0ba7f0 0x7f325c1f7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.421+0000 7f3262600640 1 -- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f325c15ed60 msgr2=0x7f325c0b53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:08.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.421+0000 7f3262600640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f325c15ed60 0x7f325c0b53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.421+0000 7f3262600640 1 -- 172.21.15.45:0/1821658071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f325c0b5cb0 con 0x7f325c15dd60 2024-04-16T16:21:08.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.421+0000 7f3261c00640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f325c0ba7f0 0x7f325c1f7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:08.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.421+0000 7f3261200640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f325c15ed60 0x7f325c0b53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:08.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.421+0000 7f3262600640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f325c15dd60 0x7f325c0cc190 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f325400c730 tx=0x7f325400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:08.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.421+0000 7f3259600640 1 -- 172.21.15.45:0/1821658071 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254003b70 con 0x7f325c15dd60 2024-04-16T16:21:08.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.421+0000 7f3259600640 1 -- 172.21.15.45:0/1821658071 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3254003d10 con 0x7f325c15dd60 2024-04-16T16:21:08.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.421+0000 7f3264600640 1 -- 172.21.15.45:0/1821658071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f325c0b5f90 con 0x7f325c15dd60 2024-04-16T16:21:08.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.421+0000 7f3259600640 1 -- 172.21.15.45:0/1821658071 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325400aa90 con 0x7f325c15dd60 2024-04-16T16:21:08.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.421+0000 7f3264600640 1 -- 172.21.15.45:0/1821658071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f325c0b6550 con 0x7f325c15dd60 2024-04-16T16:21:08.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.422+0000 7f3264600640 1 -- 172.21.15.45:0/1821658071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f325c0d4e70 con 0x7f325c15dd60 2024-04-16T16:21:08.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.423+0000 7f3259600640 1 -- 172.21.15.45:0/1821658071 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f325400ac30 con 0x7f325c15dd60 2024-04-16T16:21:08.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.425+0000 7f3259600640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f323c06ca10 0x7f323c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:08.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.425+0000 7f3259600640 1 -- 172.21.15.45:0/1821658071 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6397+0+0 (secure 0 0 0) 0x7f325408be20 con 0x7f325c15dd60 2024-04-16T16:21:08.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.425+0000 7f3261c00640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f323c06ca10 0x7f323c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:08.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.427+0000 7f3261c00640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f323c06ca10 0x7f323c06eed0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3250006fd0 tx=0x7f3250008040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:08.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.429+0000 7f3259600640 1 -- 172.21.15.45:0/1821658071 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3254059e90 con 0x7f325c15dd60 2024-04-16T16:21:08.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.672+0000 7f3264600640 1 -- 172.21.15.45:0/1821658071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f325c075780 con 0x7f325c15dd60 2024-04-16T16:21:08.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.673+0000 7f3259600640 1 -- 172.21.15.45:0/1821658071 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5186 (secure 0 0 0) 0x7f3254059790 con 0x7f325c15dd60 2024-04-16T16:21:08.674 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:08.674 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/929218719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":929218719},{"type":"v1","addr":"172.21.15.103:6835","nonce":929218719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6323,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1619719405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1619719405},{"type":"v1","addr":"172.21.15.103:6837","nonce":1619719405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":8640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/3613288118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3613288118},{"type":"v1","addr":"172.21.15.45:6834","nonce":3613288118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:20:33.816422+0000","modified":"2024-04-16T16:20:36.891870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8643":{"gid":8643,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2060312705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2060312705},{"type":"v1","addr":"172.21.15.45:6835","nonce":2060312705}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8643,"qdb_cluster":[8643]},"id":9}]} 2024-04-16T16:21:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 91 2024-04-16T16:21:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.675+0000 7f3264600640 1 -- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f323c06ca10 msgr2=0x7f323c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.675+0000 7f3264600640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f323c06ca10 0x7f323c06eed0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3250006fd0 tx=0x7f3250008040 comp rx=0 tx=0).stop 2024-04-16T16:21:08.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.675+0000 7f3264600640 1 -- 172.21.15.45:0/1821658071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f325c15dd60 msgr2=0x7f325c0cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:08.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.675+0000 7f3264600640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f325c15dd60 0x7f325c0cc190 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f325400c730 tx=0x7f325400cbf0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.675+0000 7f3264600640 1 -- 172.21.15.45:0/1821658071 shutdown_connections 2024-04-16T16:21:08.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.675+0000 7f3264600640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f325c15dd60 0x7f325c0cc190 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.675+0000 7f3264600640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f323c06ca10 0x7f323c06eed0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.675+0000 7f3264600640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f325c15ed60 0x7f325c0b53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.675+0000 7f3264600640 1 --2- 172.21.15.45:0/1821658071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f325c0ba7f0 0x7f325c1f7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.676+0000 7f3264600640 1 -- 172.21.15.45:0/1821658071 >> 172.21.15.45:0/1821658071 conn(0x7f325c073480 msgr2=0x7f325c073860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.676+0000 7f3264600640 1 -- 172.21.15.45:0/1821658071 shutdown_connections 2024-04-16T16:21:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.676+0000 7f3264600640 1 -- 172.21.15.45:0/1821658071 wait complete. 2024-04-16T16:21:08.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:21:08.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.916+0000 7f8ba1e00640 1 Processor -- start 2024-04-16T16:21:08.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.916+0000 7f8ba1e00640 1 -- start start 2024-04-16T16:21:08.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.917+0000 7f8ba1e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8b9c072a90 0x7f8b9c1699a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:08.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.917+0000 7f8ba1e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b9c147c20 con 0x7f8b9c147840 2024-04-16T16:21:08.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.917+0000 7f8ba1e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b9c160210 con 0x7f8b9c0aed90 2024-04-16T16:21:08.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.917+0000 7f8ba1e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8b9c0afe80 con 0x7f8b9c072a90 2024-04-16T16:21:08.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.918+0000 7f8b9b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8b9c072a90 0x7f8b9c1699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:08.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.918+0000 7f8b9aa00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8b9c147840 0x7f8b9c15fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36110/0 (socket says 172.21.15.45:36110) 2024-04-16T16:21:08.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.918+0000 7f8b9aa00640 1 -- 172.21.15.45:0/718789764 learned_addr learned my addr 172.21.15.45:0/718789764 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:08.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.918+0000 7f8b9b400640 1 -- 172.21.15.45:0/718789764 >> v1:172.21.15.103:6789/0 conn(0x7f8b9c0aed90 legacy=0x7f8b9c072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:08.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.918+0000 7f8b9b400640 1 -- 172.21.15.45:0/718789764 >> v1:172.21.15.45:6789/0 conn(0x7f8b9c147840 legacy=0x7f8b9c15fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:08.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.919+0000 7f8b9b400640 1 -- 172.21.15.45:0/718789764 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b9c0af270 con 0x7f8b9c072a90 2024-04-16T16:21:08.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.920+0000 7f8b9b400640 1 --2- 172.21.15.45:0/718789764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c072a90 0x7f8b9c1699a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0b0000 tx=0x7f8b9402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=266830896af50ef1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:08.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.920+0000 7f8b9a000640 1 -- 172.21.15.45:0/718789764 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94039070 con 0x7f8b9c072a90 2024-04-16T16:21:08.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.921+0000 7f8b9a000640 1 -- 172.21.15.45:0/718789764 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b9402cce0 con 0x7f8b9c072a90 2024-04-16T16:21:08.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.921+0000 7f8b9a000640 1 -- 172.21.15.45:0/718789764 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94034660 con 0x7f8b9c072a90 2024-04-16T16:21:08.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.921+0000 7f8ba1e00640 1 -- 172.21.15.45:0/718789764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c072a90 msgr2=0x7f8b9c1699a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:08.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.921+0000 7f8ba1e00640 1 --2- 172.21.15.45:0/718789764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c072a90 0x7f8b9c1699a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0b0000 tx=0x7f8b9402bf30 comp rx=0 tx=0).stop 2024-04-16T16:21:08.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.921+0000 7f8ba1e00640 1 -- 172.21.15.45:0/718789764 shutdown_connections 2024-04-16T16:21:08.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.921+0000 7f8ba1e00640 1 --2- 172.21.15.45:0/718789764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c072a90 0x7f8b9c1699a0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.921+0000 7f8ba1e00640 1 -- 172.21.15.45:0/718789764 >> 172.21.15.45:0/718789764 conn(0x7f8b9c0b2750 msgr2=0x7f8b9c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:08.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.921+0000 7f8ba1e00640 1 -- 172.21.15.45:0/718789764 shutdown_connections 2024-04-16T16:21:08.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.921+0000 7f8ba1e00640 1 -- 172.21.15.45:0/718789764 wait complete. 2024-04-16T16:21:08.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.923+0000 7f8ba1e00640 1 Processor -- start 2024-04-16T16:21:08.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.923+0000 7f8ba1e00640 1 -- start start 2024-04-16T16:21:08.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.924+0000 7f8ba1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c072a90 0x7f8b9c12f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:08.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.924+0000 7f8ba1e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b9c0aed90 0x7f8b9c1412d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:08.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.924+0000 7f8ba1e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b9c147840 0x7f8b9c141810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:08.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.924+0000 7f8ba1e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c0afb10 con 0x7f8b9c072a90 2024-04-16T16:21:08.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.924+0000 7f8ba1e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c0af830 con 0x7f8b9c0aed90 2024-04-16T16:21:08.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.924+0000 7f8ba1e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8b9c06f2c0 con 0x7f8b9c147840 2024-04-16T16:21:08.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.924+0000 7f8b9aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c072a90 0x7f8b9c12f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:08.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.924+0000 7f8b9aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c072a90 0x7f8b9c12f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48484/0 (socket says 172.21.15.45:48484) 2024-04-16T16:21:08.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.924+0000 7f8b9aa00640 1 -- 172.21.15.45:0/1372751833 learned_addr learned my addr 172.21.15.45:0/1372751833 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:08.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.925+0000 7f8b93e00640 1 --2- 172.21.15.45:0/1372751833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b9c0aed90 0x7f8b9c1412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:08.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.925+0000 7f8b9aa00640 1 -- 172.21.15.45:0/1372751833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b9c147840 msgr2=0x7f8b9c141810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:08.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.925+0000 7f8b9aa00640 1 --2- 172.21.15.45:0/1372751833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b9c147840 0x7f8b9c141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.925+0000 7f8b9aa00640 1 -- 172.21.15.45:0/1372751833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b9c0aed90 msgr2=0x7f8b9c1412d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:08.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.925+0000 7f8b9aa00640 1 --2- 172.21.15.45:0/1372751833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b9c0aed90 0x7f8b9c1412d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:08.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.925+0000 7f8b9aa00640 1 -- 172.21.15.45:0/1372751833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b9c12f840 con 0x7f8b9c072a90 2024-04-16T16:21:08.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.925+0000 7f8b93e00640 1 --2- 172.21.15.45:0/1372751833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b9c0aed90 0x7f8b9c1412d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:08.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.925+0000 7f8b9aa00640 1 --2- 172.21.15.45:0/1372751833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c072a90 0x7f8b9c12f300 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00c480 tx=0x7f8b8c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:08.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.925+0000 7f8b93400640 1 -- 172.21.15.45:0/1372751833 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c018070 con 0x7f8b9c072a90 2024-04-16T16:21:08.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.926+0000 7f8b93400640 1 -- 172.21.15.45:0/1372751833 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b8c005ce0 con 0x7f8b9c072a90 2024-04-16T16:21:08.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.926+0000 7f8b93400640 1 -- 172.21.15.45:0/1372751833 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c0072d0 con 0x7f8b9c072a90 2024-04-16T16:21:08.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.926+0000 7f8ba1e00640 1 -- 172.21.15.45:0/1372751833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b9c146360 con 0x7f8b9c072a90 2024-04-16T16:21:08.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.926+0000 7f8ba1e00640 1 -- 172.21.15.45:0/1372751833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b9c143e80 con 0x7f8b9c072a90 2024-04-16T16:21:08.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.927+0000 7f8ba1e00640 1 -- 172.21.15.45:0/1372751833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b9c075780 con 0x7f8b9c072a90 2024-04-16T16:21:08.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.928+0000 7f8b93400640 1 -- 172.21.15.45:0/1372751833 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f8b8c007470 con 0x7f8b9c072a90 2024-04-16T16:21:08.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.929+0000 7f8b93400640 1 --2- 172.21.15.45:0/1372751833 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f8b7806ca10 0x7f8b7806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:08.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.929+0000 7f8b93400640 1 -- 172.21.15.45:0/1372751833 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6397+0+0 (secure 0 0 0) 0x7f8b8c08b0f0 con 0x7f8b9c072a90 2024-04-16T16:21:08.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.930+0000 7f8b93e00640 1 --2- 172.21.15.45:0/1372751833 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f8b7806ca10 0x7f8b7806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:08.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.931+0000 7f8b93e00640 1 --2- 172.21.15.45:0/1372751833 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f8b7806ca10 0x7f8b7806eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c1328b0 tx=0x7f8b84009040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:08.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:08.934+0000 7f8b93400640 1 -- 172.21.15.45:0/1372751833 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b9c12f840 con 0x7f8b9c072a90 2024-04-16T16:21:09.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.140+0000 7f8ba1e00640 1 -- 172.21.15.45:0/1372751833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b9c070220 con 0x7f8b9c072a90 2024-04-16T16:21:09.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.143+0000 7f8b93400640 1 -- 172.21.15.45:0/1372751833 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) ==== 74+0+17696 (secure 0 0 0) 0x7f8b8c058b20 con 0x7f8b9c072a90 2024-04-16T16:21:09.143 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:09.143 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":130,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:21:04.341365+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:20:31.681227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-16T16:20:32.790980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/1715153913":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/4225560501":"2024-04-17T16:21:04.341295+0000","172.21.15.103:6832/549890383":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/1495495989":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/3982280724":"2024-04-17T16:21:04.341295+0000","172.21.15.103:6833/1120913661":"2024-04-17T16:20:50.935324+0000","172.21.15.103:6833/549890383":"2024-04-17T16:21:04.341295+0000","172.21.15.103:6832/1120913661":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1543548744":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/4103173879":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1578451701":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/4177084908":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1961421675":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/2190202958":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/3679729711":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1543358977":"2024-04-17T16:20:50.935324+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:21:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.145+0000 7f8ba1e00640 1 -- 172.21.15.45:0/1372751833 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f8b7806ca10 msgr2=0x7f8b7806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.145+0000 7f8ba1e00640 1 --2- 172.21.15.45:0/1372751833 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f8b7806ca10 0x7f8b7806eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c1328b0 tx=0x7f8b84009040 comp rx=0 tx=0).stop 2024-04-16T16:21:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.146+0000 7f8ba1e00640 1 -- 172.21.15.45:0/1372751833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c072a90 msgr2=0x7f8b9c12f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.146+0000 7f8ba1e00640 1 --2- 172.21.15.45:0/1372751833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c072a90 0x7f8b9c12f300 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00c480 tx=0x7f8b8c00c940 comp rx=0 tx=0).stop 2024-04-16T16:21:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.146+0000 7f8ba1e00640 1 -- 172.21.15.45:0/1372751833 shutdown_connections 2024-04-16T16:21:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.146+0000 7f8ba1e00640 1 --2- 172.21.15.45:0/1372751833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b9c147840 0x7f8b9c141810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.146+0000 7f8ba1e00640 1 --2- 172.21.15.45:0/1372751833 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f8b7806ca10 0x7f8b7806eed0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.146+0000 7f8ba1e00640 1 --2- 172.21.15.45:0/1372751833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b9c0aed90 0x7f8b9c1412d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.146+0000 7f8ba1e00640 1 --2- 172.21.15.45:0/1372751833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c072a90 0x7f8b9c12f300 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.146+0000 7f8ba1e00640 1 -- 172.21.15.45:0/1372751833 >> 172.21.15.45:0/1372751833 conn(0x7f8b9c0b2750 msgr2=0x7f8b9c142330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:09.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.146+0000 7f8ba1e00640 1 -- 172.21.15.45:0/1372751833 shutdown_connections 2024-04-16T16:21:09.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.146+0000 7f8ba1e00640 1 -- 172.21.15.45:0/1372751833 wait complete. 2024-04-16T16:21:09.166 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-16T16:21:09.166 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:21:09.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.388+0000 7f1f08c00640 1 Processor -- start 2024-04-16T16:21:09.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.388+0000 7f1f08c00640 1 -- start start 2024-04-16T16:21:09.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.390+0000 7f1f08c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1f04147840 0x7f1f0415fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:09.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.390+0000 7f1f08c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1f040aed90 0x7f1f04072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:09.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.390+0000 7f1f08c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f04147c20 con 0x7f1f040728c0 2024-04-16T16:21:09.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.390+0000 7f1f08c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1f04072d10 con 0x7f1f040aed90 2024-04-16T16:21:09.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.390+0000 7f1f08c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1f040afe80 con 0x7f1f04147840 2024-04-16T16:21:09.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.390+0000 7f1f01400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1f040aed90 0x7f1f04072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:09.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.390+0000 7f1f01400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1f040aed90 0x7f1f04072380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48494/0 (socket says 172.21.15.45:48494) 2024-04-16T16:21:09.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.390+0000 7f1f01400640 1 -- 172.21.15.45:0/1125885805 learned_addr learned my addr 172.21.15.45:0/1125885805 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:09.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.390+0000 7f1f01e00640 1 --2- 172.21.15.45:0/1125885805 >> v2:172.21.15.103:3300/0 conn(0x7f1f04147840 0x7f1f0415fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:09.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.391+0000 7f1f01400640 1 -- 172.21.15.45:0/1125885805 >> v2:172.21.15.103:3300/0 conn(0x7f1f04147840 msgr2=0x7f1f0415fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:09.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.391+0000 7f1f01400640 1 --2- 172.21.15.45:0/1125885805 >> v2:172.21.15.103:3300/0 conn(0x7f1f04147840 0x7f1f0415fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.391+0000 7f1f01400640 1 -- 172.21.15.45:0/1125885805 >> v1:172.21.15.45:6789/0 conn(0x7f1f040728c0 legacy=0x7f1f04162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:09.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.391+0000 7f1f01400640 1 -- 172.21.15.45:0/1125885805 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f040af270 con 0x7f1f040aed90 2024-04-16T16:21:09.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.392+0000 7f1f01400640 1 --2- 172.21.15.45:0/1125885805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f040aed90 0x7f1f04072380 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f1f0406ab80 tx=0x7f1ef402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4caf9d81085ddf8c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:09.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.392+0000 7f1f00a00640 1 -- 172.21.15.45:0/1125885805 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef402cb20 con 0x7f1f040aed90 2024-04-16T16:21:09.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.392+0000 7f1f00a00640 1 -- 172.21.15.45:0/1125885805 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ef402ccc0 con 0x7f1f040aed90 2024-04-16T16:21:09.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.392+0000 7f1f00a00640 1 -- 172.21.15.45:0/1125885805 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4034610 con 0x7f1f040aed90 2024-04-16T16:21:09.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.393+0000 7f1f08c00640 1 -- 172.21.15.45:0/1125885805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f040aed90 msgr2=0x7f1f04072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:09.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.393+0000 7f1f08c00640 1 --2- 172.21.15.45:0/1125885805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f040aed90 0x7f1f04072380 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f1f0406ab80 tx=0x7f1ef402bf80 comp rx=0 tx=0).stop 2024-04-16T16:21:09.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.393+0000 7f1f08c00640 1 -- 172.21.15.45:0/1125885805 shutdown_connections 2024-04-16T16:21:09.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.393+0000 7f1f08c00640 1 --2- 172.21.15.45:0/1125885805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f040aed90 0x7f1f04072380 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.393+0000 7f1f08c00640 1 --2- 172.21.15.45:0/1125885805 >> v2:172.21.15.103:3300/0 conn(0x7f1f04147840 0x7f1f0415fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.393+0000 7f1f08c00640 1 -- 172.21.15.45:0/1125885805 >> 172.21.15.45:0/1125885805 conn(0x7f1f040b2750 msgr2=0x7f1f040b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:09.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.393+0000 7f1f08c00640 1 -- 172.21.15.45:0/1125885805 shutdown_connections 2024-04-16T16:21:09.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.393+0000 7f1f08c00640 1 -- 172.21.15.45:0/1125885805 wait complete. 2024-04-16T16:21:09.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.395+0000 7f1f08c00640 1 Processor -- start 2024-04-16T16:21:09.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.395+0000 7f1f08c00640 1 -- start start 2024-04-16T16:21:09.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.396+0000 7f1f08c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f040728c0 0x7f1f040b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:09.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.396+0000 7f1f08c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f040aed90 0x7f1f041348e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:09.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.396+0000 7f1f08c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f04147840 0x7f1f04134e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:09.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.396+0000 7f1f08c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1f040afb10 con 0x7f1f040aed90 2024-04-16T16:21:09.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.396+0000 7f1f08c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1f040af830 con 0x7f1f04147840 2024-04-16T16:21:09.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.396+0000 7f1f08c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1f04160040 con 0x7f1f040728c0 2024-04-16T16:21:09.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.396+0000 7f1f01400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f040aed90 0x7f1f041348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:09.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.396+0000 7f1f01400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f040aed90 0x7f1f041348e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48504/0 (socket says 172.21.15.45:48504) 2024-04-16T16:21:09.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.396+0000 7f1f01400640 1 -- 172.21.15.45:0/4145993594 learned_addr learned my addr 172.21.15.45:0/4145993594 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:09.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.397+0000 7f1f01400640 1 -- 172.21.15.45:0/4145993594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f040728c0 msgr2=0x7f1f040b3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:09.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.397+0000 7f1f01400640 1 --2- 172.21.15.45:0/4145993594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f040728c0 0x7f1f040b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.397+0000 7f1f01400640 1 -- 172.21.15.45:0/4145993594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f04147840 msgr2=0x7f1f04134e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:09.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.397+0000 7f1f01400640 1 --2- 172.21.15.45:0/4145993594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f04147840 0x7f1f04134e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.397+0000 7f1f01400640 1 -- 172.21.15.45:0/4145993594 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f04131dd0 con 0x7f1f040aed90 2024-04-16T16:21:09.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.397+0000 7f1f01400640 1 --2- 172.21.15.45:0/4145993594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f040aed90 0x7f1f041348e0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4036040 tx=0x7f1ef4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:09.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.400+0000 7f1efa000640 1 -- 172.21.15.45:0/4145993594 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4043020 con 0x7f1f040aed90 2024-04-16T16:21:09.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.400+0000 7f1efa000640 1 -- 172.21.15.45:0/4145993594 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ef403e450 con 0x7f1f040aed90 2024-04-16T16:21:09.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.400+0000 7f1efa000640 1 -- 172.21.15.45:0/4145993594 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4009510 con 0x7f1f040aed90 2024-04-16T16:21:09.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.400+0000 7f1f08c00640 1 -- 172.21.15.45:0/4145993594 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f041320b0 con 0x7f1f040aed90 2024-04-16T16:21:09.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.400+0000 7f1f08c00640 1 -- 172.21.15.45:0/4145993594 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f04145270 con 0x7f1f040aed90 2024-04-16T16:21:09.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.402+0000 7f1f08c00640 1 -- 172.21.15.45:0/4145993594 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f04075780 con 0x7f1f040aed90 2024-04-16T16:21:09.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.403+0000 7f1efa000640 1 -- 172.21.15.45:0/4145993594 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f1ef4046020 con 0x7f1f040aed90 2024-04-16T16:21:09.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.404+0000 7f1efa000640 1 --2- 172.21.15.45:0/4145993594 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1ee006ca60 0x7f1ee006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:09.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.404+0000 7f1efa000640 1 -- 172.21.15.45:0/4145993594 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6397+0+0 (secure 0 0 0) 0x7f1ef40ae870 con 0x7f1f040aed90 2024-04-16T16:21:09.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.405+0000 7f1f01e00640 1 --2- 172.21.15.45:0/4145993594 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1ee006ca60 0x7f1ee006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:09.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.406+0000 7f1f01e00640 1 --2- 172.21.15.45:0/4145993594 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1ee006ca60 0x7f1ee006ef20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1eec006fd0 tx=0x7f1eec008040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:09.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.409+0000 7f1efa000640 1 -- 172.21.15.45:0/4145993594 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1ef404d030 con 0x7f1f040aed90 2024-04-16T16:21:09.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.654+0000 7f1f08c00640 1 -- 172.21.15.45:0/4145993594 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f04070220 con 0x7f1f040aed90 2024-04-16T16:21:09.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.655+0000 7f1efa000640 1 -- 172.21.15.45:0/4145993594 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5186 (secure 0 0 0) 0x7f1ef407c8c0 con 0x7f1f040aed90 2024-04-16T16:21:09.655 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:09.655 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/929218719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":929218719},{"type":"v1","addr":"172.21.15.103:6835","nonce":929218719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6323,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1619719405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1619719405},{"type":"v1","addr":"172.21.15.103:6837","nonce":1619719405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":8640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/3613288118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3613288118},{"type":"v1","addr":"172.21.15.45:6834","nonce":3613288118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:20:33.816422+0000","modified":"2024-04-16T16:20:36.891870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8643":{"gid":8643,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2060312705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2060312705},{"type":"v1","addr":"172.21.15.45:6835","nonce":2060312705}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8643,"qdb_cluster":[8643]},"id":9}]} 2024-04-16T16:21:09.655 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 91 2024-04-16T16:21:09.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.657+0000 7f1f08c00640 1 -- 172.21.15.45:0/4145993594 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1ee006ca60 msgr2=0x7f1ee006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:09.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.657+0000 7f1f08c00640 1 --2- 172.21.15.45:0/4145993594 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1ee006ca60 0x7f1ee006ef20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1eec006fd0 tx=0x7f1eec008040 comp rx=0 tx=0).stop 2024-04-16T16:21:09.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.657+0000 7f1f08c00640 1 -- 172.21.15.45:0/4145993594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f040aed90 msgr2=0x7f1f041348e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:09.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.657+0000 7f1f08c00640 1 --2- 172.21.15.45:0/4145993594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f040aed90 0x7f1f041348e0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4036040 tx=0x7f1ef4004a40 comp rx=0 tx=0).stop 2024-04-16T16:21:09.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.657+0000 7f1f08c00640 1 -- 172.21.15.45:0/4145993594 shutdown_connections 2024-04-16T16:21:09.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.657+0000 7f1f08c00640 1 --2- 172.21.15.45:0/4145993594 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1ee006ca60 0x7f1ee006ef20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.657+0000 7f1f08c00640 1 --2- 172.21.15.45:0/4145993594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f04147840 0x7f1f04134e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.657+0000 7f1f08c00640 1 --2- 172.21.15.45:0/4145993594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f040aed90 0x7f1f041348e0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.657+0000 7f1f08c00640 1 --2- 172.21.15.45:0/4145993594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f040728c0 0x7f1f040b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.657+0000 7f1f08c00640 1 -- 172.21.15.45:0/4145993594 >> 172.21.15.45:0/4145993594 conn(0x7f1f040b2750 msgr2=0x7f1f040b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:09.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.658+0000 7f1f08c00640 1 -- 172.21.15.45:0/4145993594 shutdown_connections 2024-04-16T16:21:09.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.658+0000 7f1f08c00640 1 -- 172.21.15.45:0/4145993594 wait complete. 2024-04-16T16:21:09.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:21:09.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.897+0000 7fae53e00640 1 Processor -- start 2024-04-16T16:21:09.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.897+0000 7fae53e00640 1 -- start start 2024-04-16T16:21:09.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.898+0000 7fae53e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fae54161e90 0x7fae540cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:09.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.898+0000 7fae53e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fae54162270 0x7fae540d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:09.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.898+0000 7fae53e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae540d3080 con 0x7fae540d52d0 2024-04-16T16:21:09.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.898+0000 7fae53e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fae540d2e30 con 0x7fae54161e90 2024-04-16T16:21:09.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.898+0000 7fae53e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fae540d3360 con 0x7fae54162270 2024-04-16T16:21:09.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.899+0000 7fae52a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fae54161e90 0x7fae540cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:09.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.899+0000 7fae52a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fae54161e90 0x7fae540cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48516/0 (socket says 172.21.15.45:48516) 2024-04-16T16:21:09.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.899+0000 7fae52a00640 1 -- 172.21.15.45:0/4079170054 learned_addr learned my addr 172.21.15.45:0/4079170054 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:09.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.899+0000 7fae52a00640 1 -- 172.21.15.45:0/4079170054 >> v2:172.21.15.103:3300/0 conn(0x7fae54162270 msgr2=0x7fae540d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:09.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.899+0000 7fae52a00640 1 --2- 172.21.15.45:0/4079170054 >> v2:172.21.15.103:3300/0 conn(0x7fae54162270 0x7fae540d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.899+0000 7fae52a00640 1 -- 172.21.15.45:0/4079170054 >> v1:172.21.15.103:6789/0 conn(0x7fae540d52d0 legacy=0x7fae5406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:09.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.899+0000 7fae52a00640 1 -- 172.21.15.45:0/4079170054 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae540d3640 con 0x7fae54161e90 2024-04-16T16:21:09.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.900+0000 7fae52a00640 1 --2- 172.21.15.45:0/4079170054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae54161e90 0x7fae540cf7d0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fae44009e50 tx=0x7fae4402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d940ce9ef67a864c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:09.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.900+0000 7fae51600640 1 -- 172.21.15.45:0/4079170054 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4402ca80 con 0x7fae54161e90 2024-04-16T16:21:09.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.900+0000 7fae51600640 1 -- 172.21.15.45:0/4079170054 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae4402cc20 con 0x7fae54161e90 2024-04-16T16:21:09.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.901+0000 7fae51600640 1 -- 172.21.15.45:0/4079170054 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae440343f0 con 0x7fae54161e90 2024-04-16T16:21:09.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.901+0000 7fae53e00640 1 -- 172.21.15.45:0/4079170054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae54161e90 msgr2=0x7fae540cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:09.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.901+0000 7fae53e00640 1 --2- 172.21.15.45:0/4079170054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae54161e90 0x7fae540cf7d0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fae44009e50 tx=0x7fae4402bee0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.901+0000 7fae53e00640 1 -- 172.21.15.45:0/4079170054 shutdown_connections 2024-04-16T16:21:09.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.901+0000 7fae53e00640 1 --2- 172.21.15.45:0/4079170054 >> v2:172.21.15.103:3300/0 conn(0x7fae54162270 0x7fae540d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.901+0000 7fae53e00640 1 --2- 172.21.15.45:0/4079170054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae54161e90 0x7fae540cf7d0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.901+0000 7fae53e00640 1 -- 172.21.15.45:0/4079170054 >> 172.21.15.45:0/4079170054 conn(0x7fae540d1940 msgr2=0x7fae540d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:09.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.901+0000 7fae53e00640 1 -- 172.21.15.45:0/4079170054 shutdown_connections 2024-04-16T16:21:09.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.901+0000 7fae53e00640 1 -- 172.21.15.45:0/4079170054 wait complete. 2024-04-16T16:21:09.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.904+0000 7fae53e00640 1 Processor -- start 2024-04-16T16:21:09.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.904+0000 7fae53e00640 1 -- start start 2024-04-16T16:21:09.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.905+0000 7fae53e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae540d52d0 0x7fae541e73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:09.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.905+0000 7fae53e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae54161e90 0x7fae5414c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:09.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.905+0000 7fae53e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae54162270 0x7fae540cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:09.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.905+0000 7fae53e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fae5406d550 con 0x7fae54161e90 2024-04-16T16:21:09.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.905+0000 7fae53e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fae540cfd10 con 0x7fae54162270 2024-04-16T16:21:09.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.905+0000 7fae53e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fae5406d6d0 con 0x7fae540d52d0 2024-04-16T16:21:09.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.905+0000 7fae52000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae54161e90 0x7fae5414c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:09.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.905+0000 7fae52000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae54161e90 0x7fae5414c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48522/0 (socket says 172.21.15.45:48522) 2024-04-16T16:21:09.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.905+0000 7fae52000640 1 -- 172.21.15.45:0/1924485753 learned_addr learned my addr 172.21.15.45:0/1924485753 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:09.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.906+0000 7fae53400640 1 --2- 172.21.15.45:0/1924485753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae54162270 0x7fae540cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:09.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.906+0000 7fae52000640 1 -- 172.21.15.45:0/1924485753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae540d52d0 msgr2=0x7fae541e73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:09.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.906+0000 7fae52a00640 1 --2- 172.21.15.45:0/1924485753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae540d52d0 0x7fae541e73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:09.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.906+0000 7fae52000640 1 --2- 172.21.15.45:0/1924485753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae540d52d0 0x7fae541e73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.906+0000 7fae52000640 1 -- 172.21.15.45:0/1924485753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae54162270 msgr2=0x7fae540cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:09.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.906+0000 7fae52000640 1 --2- 172.21.15.45:0/1924485753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae54162270 0x7fae540cd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:09.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.906+0000 7fae52000640 1 -- 172.21.15.45:0/1924485753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae541e78e0 con 0x7fae54161e90 2024-04-16T16:21:09.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.906+0000 7fae53400640 1 --2- 172.21.15.45:0/1924485753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae54162270 0x7fae540cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:09.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.906+0000 7fae52000640 1 --2- 172.21.15.45:0/1924485753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae54161e90 0x7fae5414c8d0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fae4000b410 tx=0x7fae4000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:09.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.906+0000 7fae4b400640 1 -- 172.21.15.45:0/1924485753 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae40015020 con 0x7fae54161e90 2024-04-16T16:21:09.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.906+0000 7fae4b400640 1 -- 172.21.15.45:0/1924485753 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae40011830 con 0x7fae54161e90 2024-04-16T16:21:09.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.906+0000 7fae4b400640 1 -- 172.21.15.45:0/1924485753 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae40011c20 con 0x7fae54161e90 2024-04-16T16:21:09.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.906+0000 7fae53e00640 1 -- 172.21.15.45:0/1924485753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae5414d450 con 0x7fae54161e90 2024-04-16T16:21:09.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.907+0000 7fae53e00640 1 -- 172.21.15.45:0/1924485753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae5414da10 con 0x7fae54161e90 2024-04-16T16:21:09.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.908+0000 7fae53e00640 1 -- 172.21.15.45:0/1924485753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae54075780 con 0x7fae54161e90 2024-04-16T16:21:09.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.909+0000 7fae4b400640 1 -- 172.21.15.45:0/1924485753 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7fae40018020 con 0x7fae54161e90 2024-04-16T16:21:09.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.910+0000 7fae4b400640 1 --2- 172.21.15.45:0/1924485753 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fae3406ca10 0x7fae3406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:09.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.910+0000 7fae4b400640 1 -- 172.21.15.45:0/1924485753 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6397+0+0 (secure 0 0 0) 0x7fae4008be10 con 0x7fae54161e90 2024-04-16T16:21:09.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.911+0000 7fae52a00640 1 --2- 172.21.15.45:0/1924485753 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fae3406ca10 0x7fae3406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:09.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.912+0000 7fae52a00640 1 --2- 172.21.15.45:0/1924485753 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fae3406ca10 0x7fae3406eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fae44035000 tx=0x7fae44030000 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:09.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:09.915+0000 7fae4b400640 1 -- 172.21.15.45:0/1924485753 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae4001f020 con 0x7fae54161e90 2024-04-16T16:21:10.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.163+0000 7fae53e00640 1 -- 172.21.15.45:0/1924485753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae540b7c90 con 0x7fae54161e90 2024-04-16T16:21:10.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.164+0000 7fae4b400640 1 -- 172.21.15.45:0/1924485753 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5186 (secure 0 0 0) 0x7fae400172c0 con 0x7fae54161e90 2024-04-16T16:21:10.164 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:10.165 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/929218719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":929218719},{"type":"v1","addr":"172.21.15.103:6835","nonce":929218719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6323,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1619719405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1619719405},{"type":"v1","addr":"172.21.15.103:6837","nonce":1619719405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":8640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/3613288118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3613288118},{"type":"v1","addr":"172.21.15.45:6834","nonce":3613288118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:20:33.816422+0000","modified":"2024-04-16T16:20:36.891870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8643":{"gid":8643,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2060312705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2060312705},{"type":"v1","addr":"172.21.15.45:6835","nonce":2060312705}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8643,"qdb_cluster":[8643]},"id":9}]} 2024-04-16T16:21:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 91 2024-04-16T16:21:10.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.168+0000 7fae53e00640 1 -- 172.21.15.45:0/1924485753 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fae3406ca10 msgr2=0x7fae3406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:10.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.168+0000 7fae53e00640 1 --2- 172.21.15.45:0/1924485753 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fae3406ca10 0x7fae3406eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fae44035000 tx=0x7fae44030000 comp rx=0 tx=0).stop 2024-04-16T16:21:10.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.168+0000 7fae53e00640 1 -- 172.21.15.45:0/1924485753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae54161e90 msgr2=0x7fae5414c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:10.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.168+0000 7fae53e00640 1 --2- 172.21.15.45:0/1924485753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae54161e90 0x7fae5414c8d0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fae4000b410 tx=0x7fae4000b8d0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.168+0000 7fae53e00640 1 -- 172.21.15.45:0/1924485753 shutdown_connections 2024-04-16T16:21:10.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.168+0000 7fae53e00640 1 --2- 172.21.15.45:0/1924485753 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fae3406ca10 0x7fae3406eed0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.168+0000 7fae53e00640 1 --2- 172.21.15.45:0/1924485753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae54162270 0x7fae540cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.168+0000 7fae53e00640 1 --2- 172.21.15.45:0/1924485753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae54161e90 0x7fae5414c8d0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.168+0000 7fae53e00640 1 --2- 172.21.15.45:0/1924485753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae540d52d0 0x7fae541e73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.168+0000 7fae53e00640 1 -- 172.21.15.45:0/1924485753 >> 172.21.15.45:0/1924485753 conn(0x7fae540d1940 msgr2=0x7fae540ca390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:10.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.168+0000 7fae53e00640 1 -- 172.21.15.45:0/1924485753 shutdown_connections 2024-04-16T16:21:10.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.168+0000 7fae53e00640 1 -- 172.21.15.45:0/1924485753 wait complete. 2024-04-16T16:21:10.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:21:10.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.421+0000 7f2b65600640 1 Processor -- start 2024-04-16T16:21:10.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.422+0000 7f2b65600640 1 -- start start 2024-04-16T16:21:10.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.423+0000 7f2b65600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b600cdeb0 0x7f2b600cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:10.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.423+0000 7f2b65600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2b600d0060 0x7f2b600b4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:10.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.423+0000 7f2b65600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b600ce310 con 0x7f2b6015b730 2024-04-16T16:21:10.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.423+0000 7f2b65600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2b600ce5b0 con 0x7f2b600d0060 2024-04-16T16:21:10.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.423+0000 7f2b65600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2b60072420 con 0x7f2b600cdeb0 2024-04-16T16:21:10.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.424+0000 7f2b5e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b600cdeb0 0x7f2b600cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:10.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.424+0000 7f2b5e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b600cdeb0 0x7f2b600cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:59872/0 (socket says 172.21.15.45:59872) 2024-04-16T16:21:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.424+0000 7f2b5e000640 1 -- 172.21.15.45:0/1270155785 learned_addr learned my addr 172.21.15.45:0/1270155785 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.424+0000 7f2b5f400640 1 --2- 172.21.15.45:0/1270155785 >> v2:172.21.15.103:3300/0 conn(0x7f2b600d0060 0x7f2b600b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.425+0000 7f2b5e000640 1 -- 172.21.15.45:0/1270155785 >> v2:172.21.15.103:3300/0 conn(0x7f2b600d0060 msgr2=0x7f2b600b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.425+0000 7f2b5e000640 1 --2- 172.21.15.45:0/1270155785 >> v2:172.21.15.103:3300/0 conn(0x7f2b600d0060 0x7f2b600b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.425+0000 7f2b5e000640 1 -- 172.21.15.45:0/1270155785 >> v1:172.21.15.103:6789/0 conn(0x7f2b6015b730 legacy=0x7f2b600b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.425+0000 7f2b5d600640 1 -- 172.21.15.45:0/1270155785 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1950064103 0 0) 0x7f2b600ce310 con 0x7f2b6015b730 2024-04-16T16:21:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.425+0000 7f2b5e000640 1 -- 172.21.15.45:0/1270155785 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b60072700 con 0x7f2b600cdeb0 2024-04-16T16:21:10.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.426+0000 7f2b5e000640 1 --2- 172.21.15.45:0/1270155785 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b600cdeb0 0x7f2b600cfb20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2b48009870 tx=0x7f2b4802db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b41bb5c03e2a764 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:10.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.427+0000 7f2b5d600640 1 -- 172.21.15.45:0/1270155785 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4803b070 con 0x7f2b600cdeb0 2024-04-16T16:21:10.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.427+0000 7f2b5d600640 1 -- 172.21.15.45:0/1270155785 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b4802e940 con 0x7f2b600cdeb0 2024-04-16T16:21:10.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.427+0000 7f2b5d600640 1 -- 172.21.15.45:0/1270155785 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4802ed10 con 0x7f2b600cdeb0 2024-04-16T16:21:10.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.427+0000 7f2b65600640 1 -- 172.21.15.45:0/1270155785 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b600cdeb0 msgr2=0x7f2b600cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:10.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.427+0000 7f2b65600640 1 --2- 172.21.15.45:0/1270155785 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b600cdeb0 0x7f2b600cfb20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2b48009870 tx=0x7f2b4802db90 comp rx=0 tx=0).stop 2024-04-16T16:21:10.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.428+0000 7f2b65600640 1 -- 172.21.15.45:0/1270155785 shutdown_connections 2024-04-16T16:21:10.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.428+0000 7f2b65600640 1 --2- 172.21.15.45:0/1270155785 >> v2:172.21.15.103:3300/0 conn(0x7f2b600d0060 0x7f2b600b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.428+0000 7f2b65600640 1 --2- 172.21.15.45:0/1270155785 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b600cdeb0 0x7f2b600cfb20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.428+0000 7f2b65600640 1 -- 172.21.15.45:0/1270155785 >> 172.21.15.45:0/1270155785 conn(0x7f2b600b2750 msgr2=0x7f2b600b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:10.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.428+0000 7f2b65600640 1 -- 172.21.15.45:0/1270155785 shutdown_connections 2024-04-16T16:21:10.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.428+0000 7f2b65600640 1 -- 172.21.15.45:0/1270155785 wait complete. 2024-04-16T16:21:10.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.430+0000 7f2b65600640 1 Processor -- start 2024-04-16T16:21:10.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.430+0000 7f2b65600640 1 -- start start 2024-04-16T16:21:10.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.431+0000 7f2b65600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b600cdeb0 0x7f2b60151440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:10.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.431+0000 7f2b65600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b600d0060 0x7f2b601423a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:10.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.431+0000 7f2b65600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b6015b730 0x7f2b6014f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:10.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.431+0000 7f2b65600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2b600d04f0 con 0x7f2b600cdeb0 2024-04-16T16:21:10.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.431+0000 7f2b65600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2b60072cc0 con 0x7f2b600d0060 2024-04-16T16:21:10.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.431+0000 7f2b65600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2b6015bb10 con 0x7f2b6015b730 2024-04-16T16:21:10.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.431+0000 7f2b5ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b600cdeb0 0x7f2b60151440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:10.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.432+0000 7f2b5ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b600cdeb0 0x7f2b60151440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48530/0 (socket says 172.21.15.45:48530) 2024-04-16T16:21:10.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.432+0000 7f2b5ea00640 1 -- 172.21.15.45:0/3927168891 learned_addr learned my addr 172.21.15.45:0/3927168891 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:10.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.432+0000 7f2b5f400640 1 --2- 172.21.15.45:0/3927168891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b6015b730 0x7f2b6014f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:10.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.432+0000 7f2b5e000640 1 --2- 172.21.15.45:0/3927168891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b600d0060 0x7f2b601423a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:10.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.432+0000 7f2b5ea00640 1 -- 172.21.15.45:0/3927168891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b6015b730 msgr2=0x7f2b6014f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:10.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.432+0000 7f2b5ea00640 1 --2- 172.21.15.45:0/3927168891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b6015b730 0x7f2b6014f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.432+0000 7f2b5ea00640 1 -- 172.21.15.45:0/3927168891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b600d0060 msgr2=0x7f2b601423a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:10.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.432+0000 7f2b5ea00640 1 --2- 172.21.15.45:0/3927168891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b600d0060 0x7f2b601423a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.432+0000 7f2b5ea00640 1 -- 172.21.15.45:0/3927168891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b60151980 con 0x7f2b600cdeb0 2024-04-16T16:21:10.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.432+0000 7f2b5e000640 1 --2- 172.21.15.45:0/3927168891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b600d0060 0x7f2b601423a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:10.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.432+0000 7f2b5ea00640 1 --2- 172.21.15.45:0/3927168891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b600cdeb0 0x7f2b60151440 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000c410 tx=0x7f2b5000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:10.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.433+0000 7f2b56a00640 1 -- 172.21.15.45:0/3927168891 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b50017070 con 0x7f2b600cdeb0 2024-04-16T16:21:10.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.433+0000 7f2b65600640 1 -- 172.21.15.45:0/3927168891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b601503a0 con 0x7f2b600cdeb0 2024-04-16T16:21:10.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.433+0000 7f2b56a00640 1 -- 172.21.15.45:0/3927168891 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b500035f0 con 0x7f2b600cdeb0 2024-04-16T16:21:10.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.433+0000 7f2b56a00640 1 -- 172.21.15.45:0/3927168891 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b500039e0 con 0x7f2b600cdeb0 2024-04-16T16:21:10.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.433+0000 7f2b65600640 1 -- 172.21.15.45:0/3927168891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b60150960 con 0x7f2b600cdeb0 2024-04-16T16:21:10.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.434+0000 7f2b65600640 1 -- 172.21.15.45:0/3927168891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b60075780 con 0x7f2b600cdeb0 2024-04-16T16:21:10.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.435+0000 7f2b56a00640 1 -- 172.21.15.45:0/3927168891 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f2b5001a070 con 0x7f2b600cdeb0 2024-04-16T16:21:10.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.436+0000 7f2b56a00640 1 --2- 172.21.15.45:0/3927168891 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f2b3c06ca10 0x7f2b3c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:10.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.436+0000 7f2b56a00640 1 -- 172.21.15.45:0/3927168891 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6397+0+0 (secure 0 0 0) 0x7f2b5008b4a0 con 0x7f2b600cdeb0 2024-04-16T16:21:10.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.437+0000 7f2b5e000640 1 --2- 172.21.15.45:0/3927168891 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f2b3c06ca10 0x7f2b3c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:10.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.438+0000 7f2b5e000640 1 --2- 172.21.15.45:0/3927168891 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f2b3c06ca10 0x7f2b3c06eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2b48038000 tx=0x7f2b48032000 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:10.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.442+0000 7f2b56a00640 1 -- 172.21.15.45:0/3927168891 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2b50059480 con 0x7f2b600cdeb0 2024-04-16T16:21:10.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.648+0000 7f2b65600640 1 -- 172.21.15.45:0/3927168891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2b600b3d00 con 0x7f2b600cdeb0 2024-04-16T16:21:10.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.650+0000 7f2b56a00640 1 -- 172.21.15.45:0/3927168891 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) ==== 74+0+17696 (secure 0 0 0) 0x7f2b50058d80 con 0x7f2b600cdeb0 2024-04-16T16:21:10.650 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:10.651 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":130,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:21:04.341365+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:20:31.681227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-16T16:20:32.790980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/1715153913":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/4225560501":"2024-04-17T16:21:04.341295+0000","172.21.15.103:6832/549890383":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/1495495989":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/3982280724":"2024-04-17T16:21:04.341295+0000","172.21.15.103:6833/1120913661":"2024-04-17T16:20:50.935324+0000","172.21.15.103:6833/549890383":"2024-04-17T16:21:04.341295+0000","172.21.15.103:6832/1120913661":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1543548744":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/4103173879":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1578451701":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/4177084908":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1961421675":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/2190202958":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/3679729711":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1543358977":"2024-04-17T16:20:50.935324+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:21:10.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.653+0000 7f2b65600640 1 -- 172.21.15.45:0/3927168891 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f2b3c06ca10 msgr2=0x7f2b3c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:10.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.653+0000 7f2b65600640 1 --2- 172.21.15.45:0/3927168891 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f2b3c06ca10 0x7f2b3c06eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2b48038000 tx=0x7f2b48032000 comp rx=0 tx=0).stop 2024-04-16T16:21:10.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.653+0000 7f2b65600640 1 -- 172.21.15.45:0/3927168891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b600cdeb0 msgr2=0x7f2b60151440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:10.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.653+0000 7f2b65600640 1 --2- 172.21.15.45:0/3927168891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b600cdeb0 0x7f2b60151440 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000c410 tx=0x7f2b5000c8d0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.653+0000 7f2b65600640 1 -- 172.21.15.45:0/3927168891 shutdown_connections 2024-04-16T16:21:10.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.653+0000 7f2b65600640 1 --2- 172.21.15.45:0/3927168891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b6015b730 0x7f2b6014f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.653+0000 7f2b65600640 1 --2- 172.21.15.45:0/3927168891 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f2b3c06ca10 0x7f2b3c06eed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.653+0000 7f2b65600640 1 --2- 172.21.15.45:0/3927168891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b600d0060 0x7f2b601423a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.653+0000 7f2b65600640 1 --2- 172.21.15.45:0/3927168891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b600cdeb0 0x7f2b60151440 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.653+0000 7f2b65600640 1 -- 172.21.15.45:0/3927168891 >> 172.21.15.45:0/3927168891 conn(0x7f2b600b2750 msgr2=0x7f2b600b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:10.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.653+0000 7f2b65600640 1 -- 172.21.15.45:0/3927168891 shutdown_connections 2024-04-16T16:21:10.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.653+0000 7f2b65600640 1 -- 172.21.15.45:0/3927168891 wait complete. 2024-04-16T16:21:10.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-16T16:21:10.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.900+0000 7f500e200640 1 Processor -- start 2024-04-16T16:21:10.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.901+0000 7f500e200640 1 -- start start 2024-04-16T16:21:10.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.902+0000 7f500e200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f50080cfc20 0x7f50080d37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:10.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.902+0000 7f500e200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50080d2600 con 0x7f50080ba140 2024-04-16T16:21:10.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.902+0000 7f500e200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50080ba5d0 con 0x7f5008162090 2024-04-16T16:21:10.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.902+0000 7f500e200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f50080ba8b0 con 0x7f50080cfc20 2024-04-16T16:21:10.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.903+0000 7f5007e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f50080ba140 0x7f50080d30a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36134/0 (socket says 172.21.15.45:36134) 2024-04-16T16:21:10.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.903+0000 7f5007e00640 1 -- 172.21.15.45:0/3470465805 learned_addr learned my addr 172.21.15.45:0/3470465805 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:10.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.903+0000 7f500d800640 1 --2- 172.21.15.45:0/3470465805 >> v2:172.21.15.103:3300/0 conn(0x7f50080cfc20 0x7f50080d37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:10.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.903+0000 7f5007400640 1 -- 172.21.15.45:0/3470465805 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 461773808 0 0) 0x7f50080d2600 con 0x7f50080ba140 2024-04-16T16:21:10.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.903+0000 7f5007400640 1 -- 172.21.15.45:0/3470465805 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fec003520 con 0x7f50080ba140 2024-04-16T16:21:10.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.904+0000 7f5007400640 1 -- 172.21.15.45:0/3470465805 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 230146712 0 0) 0x7f4fec003520 con 0x7f50080ba140 2024-04-16T16:21:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.904+0000 7f5007400640 1 -- 172.21.15.45:0/3470465805 >> v2:172.21.15.103:3300/0 conn(0x7f50080cfc20 msgr2=0x7f50080d37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.904+0000 7f5007400640 1 --2- 172.21.15.45:0/3470465805 >> v2:172.21.15.103:3300/0 conn(0x7f50080cfc20 0x7f50080d37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.904+0000 7f5007400640 1 -- 172.21.15.45:0/3470465805 >> v1:172.21.15.103:6789/0 conn(0x7f5008162090 legacy=0x7f50080cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.904+0000 7f5007400640 1 -- 172.21.15.45:0/3470465805 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50080d28a0 con 0x7f50080ba140 2024-04-16T16:21:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.904+0000 7f5007400640 1 -- 172.21.15.45:0/3470465805 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f4ff0001690 con 0x7f50080ba140 2024-04-16T16:21:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.904+0000 7f5007400640 1 -- 172.21.15.45:0/3470465805 >> v1:172.21.15.45:6789/0 conn(0x7f50080ba140 legacy=0x7f50080d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.904+0000 7f5007400640 1 --2- 172.21.15.45:0/3470465805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fec0039b0 0x7f4fec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.904+0000 7f5007400640 1 --2- 172.21.15.45:0/3470465805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4fec004800 0x7f4fec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.904+0000 7f5007400640 1 --2- 172.21.15.45:0/3470465805 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fec0051c0 0x7f4fec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.904+0000 7f5007400640 1 -- 172.21.15.45:0/3470465805 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f50080ba8b0 con 0x7f4fec004800 2024-04-16T16:21:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.904+0000 7f5007400640 1 -- 172.21.15.45:0/3470465805 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4fec00c060 con 0x7f4fec0039b0 2024-04-16T16:21:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.904+0000 7f5007400640 1 -- 172.21.15.45:0/3470465805 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4fec00c260 con 0x7f4fec0051c0 2024-04-16T16:21:10.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.904+0000 7f5007e00640 1 --2- 172.21.15.45:0/3470465805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4fec004800 0x7f4fec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:10.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.905+0000 7f500d800640 1 --2- 172.21.15.45:0/3470465805 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fec0051c0 0x7f4fec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:10.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.905+0000 7f5007e00640 1 -- 172.21.15.45:0/3470465805 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fec0051c0 msgr2=0x7f4fec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:10.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.905+0000 7f5007e00640 1 --2- 172.21.15.45:0/3470465805 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fec0051c0 0x7f4fec00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.905+0000 7f5007e00640 1 -- 172.21.15.45:0/3470465805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fec0039b0 msgr2=0x7f4fec003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:10.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.905+0000 7f500ce00640 1 --2- 172.21.15.45:0/3470465805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fec0039b0 0x7f4fec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:10.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.905+0000 7f5007e00640 1 --2- 172.21.15.45:0/3470465805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fec0039b0 0x7f4fec003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.905+0000 7f5007e00640 1 -- 172.21.15.45:0/3470465805 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fec003520 con 0x7f4fec004800 2024-04-16T16:21:10.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.905+0000 7f500d800640 1 -- 172.21.15.45:0/3470465805 reap_dead start 2024-04-16T16:21:10.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.905+0000 7f500ce00640 1 --2- 172.21.15.45:0/3470465805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fec0039b0 0x7f4fec003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:10.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.906+0000 7f5007e00640 1 --2- 172.21.15.45:0/3470465805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4fec004800 0x7f4fec004c80 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0007fd0 tx=0x7f4fec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f0aa6c92c93042 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:10.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.906+0000 7f5007400640 1 -- 172.21.15.45:0/3470465805 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0007d50 con 0x7f4fec004800 2024-04-16T16:21:10.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.906+0000 7f5007400640 1 -- 172.21.15.45:0/3470465805 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ff0030070 con 0x7f4fec004800 2024-04-16T16:21:10.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.906+0000 7f5007400640 1 -- 172.21.15.45:0/3470465805 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0005d10 con 0x7f4fec004800 2024-04-16T16:21:10.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.907+0000 7f500e200640 1 -- 172.21.15.45:0/3470465805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4fec004800 msgr2=0x7f4fec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:10.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.907+0000 7f500e200640 1 --2- 172.21.15.45:0/3470465805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4fec004800 0x7f4fec004c80 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0007fd0 tx=0x7f4fec005830 comp rx=0 tx=0).stop 2024-04-16T16:21:10.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.907+0000 7f500e200640 1 -- 172.21.15.45:0/3470465805 shutdown_connections 2024-04-16T16:21:10.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.907+0000 7f500e200640 1 --2- 172.21.15.45:0/3470465805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4fec004800 0x7f4fec004c80 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.907+0000 7f500e200640 1 -- 172.21.15.45:0/3470465805 >> 172.21.15.45:0/3470465805 conn(0x7f50080d1940 msgr2=0x7f50080d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:10.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.907+0000 7f500e200640 1 -- 172.21.15.45:0/3470465805 shutdown_connections 2024-04-16T16:21:10.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.907+0000 7f500e200640 1 -- 172.21.15.45:0/3470465805 wait complete. 2024-04-16T16:21:10.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.909+0000 7f500e200640 1 Processor -- start 2024-04-16T16:21:10.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.909+0000 7f500e200640 1 -- start start 2024-04-16T16:21:10.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.910+0000 7f500e200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fec004800 0x7f5008150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:10.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.910+0000 7f500e200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50080cdfd0 0x7f5008161070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:10.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.910+0000 7f500e200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f500814d3d0 0x7f500814d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:10.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.910+0000 7f500e200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f50081625f0 con 0x7f50080cdfd0 2024-04-16T16:21:10.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.910+0000 7f500e200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5008162470 con 0x7f500814d3d0 2024-04-16T16:21:10.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.910+0000 7f500e200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f50080d5520 con 0x7f4fec004800 2024-04-16T16:21:10.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.910+0000 7f5007e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50080cdfd0 0x7f5008161070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:10.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.910+0000 7f5007e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50080cdfd0 0x7f5008161070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48542/0 (socket says 172.21.15.45:48542) 2024-04-16T16:21:10.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.910+0000 7f5007e00640 1 -- 172.21.15.45:0/1474476215 learned_addr learned my addr 172.21.15.45:0/1474476215 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:10.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.911+0000 7f500d800640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f500814d3d0 0x7f500814d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:10.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.911+0000 7f500ce00640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fec004800 0x7f5008150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:10.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.911+0000 7f5007e00640 1 -- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fec004800 msgr2=0x7f5008150b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:10.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.911+0000 7f5007e00640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fec004800 0x7f5008150b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.911+0000 7f5007e00640 1 -- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f500814d3d0 msgr2=0x7f500814d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:10.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.911+0000 7f5007e00640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f500814d3d0 0x7f500814d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:10.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.911+0000 7f5007e00640 1 -- 172.21.15.45:0/1474476215 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50080ab3b0 con 0x7f50080cdfd0 2024-04-16T16:21:10.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.911+0000 7f500d800640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f500814d3d0 0x7f500814d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:10.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.911+0000 7f500ce00640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fec004800 0x7f5008150b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:10.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.911+0000 7f5007e00640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50080cdfd0 0x7f5008161070 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0001990 tx=0x7f50080b3ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:10.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.911+0000 7f5004c00640 1 -- 172.21.15.45:0/1474476215 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0045070 con 0x7f50080cdfd0 2024-04-16T16:21:10.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.911+0000 7f5004c00640 1 -- 172.21.15.45:0/1474476215 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ff0004bf0 con 0x7f50080cdfd0 2024-04-16T16:21:10.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.911+0000 7f500e200640 1 -- 172.21.15.45:0/1474476215 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50080cd4f0 con 0x7f50080cdfd0 2024-04-16T16:21:10.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.911+0000 7f5004c00640 1 -- 172.21.15.45:0/1474476215 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0007d50 con 0x7f50080cdfd0 2024-04-16T16:21:10.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.911+0000 7f500e200640 1 -- 172.21.15.45:0/1474476215 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50080cdab0 con 0x7f50080cdfd0 2024-04-16T16:21:10.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.913+0000 7f500e200640 1 -- 172.21.15.45:0/1474476215 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5008075780 con 0x7f50080cdfd0 2024-04-16T16:21:10.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.914+0000 7f5004c00640 1 -- 172.21.15.45:0/1474476215 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f4ff0007070 con 0x7f50080cdfd0 2024-04-16T16:21:10.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.915+0000 7f5004c00640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f4fe406ca40 0x7f4fe406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:10.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.915+0000 7f5004c00640 1 -- 172.21.15.45:0/1474476215 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6397+0+0 (secure 0 0 0) 0x7f4ff00ad2d0 con 0x7f50080cdfd0 2024-04-16T16:21:10.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.915+0000 7f500ce00640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f4fe406ca40 0x7f4fe406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:10.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.916+0000 7f500ce00640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f4fe406ca40 0x7f4fe406ef00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5008051690 tx=0x7f4ff8009040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:10.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:10.920+0000 7f5004c00640 1 -- 172.21.15.45:0/1474476215 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ff007b4b0 con 0x7f50080cdfd0 2024-04-16T16:21:11.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.163+0000 7f500e200640 1 -- 172.21.15.45:0/1474476215 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f50080b3320 con 0x7f50080cdfd0 2024-04-16T16:21:11.163 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-04-16T16:21:11.164+0000 7fe7a4800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T16:21:11.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.166+0000 7f5004c00640 1 -- 172.21.15.45:0/1474476215 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v92) ==== 161+0+0 (secure 0 0 0) 0x7f4ff007adb0 con 0x7f50080cdfd0 2024-04-16T16:21:11.166 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-16T16:21:11.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.170+0000 7f500e200640 1 -- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f4fe406ca40 msgr2=0x7f4fe406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:11.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.170+0000 7f500e200640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f4fe406ca40 0x7f4fe406ef00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5008051690 tx=0x7f4ff8009040 comp rx=0 tx=0).stop 2024-04-16T16:21:11.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.170+0000 7f500e200640 1 -- 172.21.15.45:0/1474476215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50080cdfd0 msgr2=0x7f5008161070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:11.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.170+0000 7f500e200640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50080cdfd0 0x7f5008161070 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0001990 tx=0x7f50080b3ff0 comp rx=0 tx=0).stop 2024-04-16T16:21:11.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.170+0000 7f500e200640 1 -- 172.21.15.45:0/1474476215 shutdown_connections 2024-04-16T16:21:11.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.170+0000 7f500e200640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f4fe406ca40 0x7f4fe406ef00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:11.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.170+0000 7f500e200640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f500814d3d0 0x7f500814d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:11.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.170+0000 7f500e200640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50080cdfd0 0x7f5008161070 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:11.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.170+0000 7f500e200640 1 --2- 172.21.15.45:0/1474476215 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fec004800 0x7f5008150b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:11.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.170+0000 7f500e200640 1 -- 172.21.15.45:0/1474476215 >> 172.21.15.45:0/1474476215 conn(0x7f50080d1940 msgr2=0x7f50080b96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:11.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.171+0000 7f500e200640 1 -- 172.21.15.45:0/1474476215 shutdown_connections 2024-04-16T16:21:11.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.171+0000 7f500e200640 1 -- 172.21.15.45:0/1474476215 wait complete. 2024-04-16T16:21:11.190 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-16T16:21:11.201 INFO:tasks.ceph.mds.c.smithi045.stderr: -4834> 2024-04-16T16:20:40.290+0000 7fba8c000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:21:11.270 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2024-04-16T16:21:11.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.386+0000 7f05f0e00640 1 Processor -- start 2024-04-16T16:21:11.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.386+0000 7f05f0e00640 1 -- start start 2024-04-16T16:21:11.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.388+0000 7f05f0e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f05ec0d0e80 0x7f05ec0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:11.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.388+0000 7f05f0e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05ec0afda0 0x7f05ec06ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:11.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.388+0000 7f05f0e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05ec0ccb70 con 0x7f05ec15b920 2024-04-16T16:21:11.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.388+0000 7f05f0e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f05ec06e7a0 con 0x7f05ec0d0e80 2024-04-16T16:21:11.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.388+0000 7f05f0e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f05ec06f170 con 0x7f05ec0afda0 2024-04-16T16:21:11.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.389+0000 7f05e9400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f05ec0d0e80 0x7f05ec0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:11.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.389+0000 7f05e9400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f05ec0d0e80 0x7f05ec0b4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48576/0 (socket says 172.21.15.45:48576) 2024-04-16T16:21:11.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.389+0000 7f05e9400640 1 -- 172.21.15.45:0/170717283 learned_addr learned my addr 172.21.15.45:0/170717283 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:11.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.389+0000 7f05ea800640 1 --2- 172.21.15.45:0/170717283 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05ec0afda0 0x7f05ec06ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:11.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.389+0000 7f05e9400640 1 -- 172.21.15.45:0/170717283 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05ec0afda0 msgr2=0x7f05ec06ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:11.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.389+0000 7f05e9400640 1 --2- 172.21.15.45:0/170717283 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05ec0afda0 0x7f05ec06ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:11.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.389+0000 7f05e9400640 1 -- 172.21.15.45:0/170717283 >> v1:172.21.15.103:6789/0 conn(0x7f05ec15b920 legacy=0x7f05ec0d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:11.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.389+0000 7f05e9400640 1 -- 172.21.15.45:0/170717283 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05ec0cce50 con 0x7f05ec0d0e80 2024-04-16T16:21:11.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.390+0000 7f05e9400640 1 --2- 172.21.15.45:0/170717283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05ec0d0e80 0x7f05ec0b4e20 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f05ec073470 tx=0x7f05dc02e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6864ad2631b6324f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:11.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.391+0000 7f05e8a00640 1 -- 172.21.15.45:0/170717283 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc02ea40 con 0x7f05ec0d0e80 2024-04-16T16:21:11.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.391+0000 7f05e8a00640 1 -- 172.21.15.45:0/170717283 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05dc02ebe0 con 0x7f05ec0d0e80 2024-04-16T16:21:11.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.391+0000 7f05e8a00640 1 -- 172.21.15.45:0/170717283 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc009600 con 0x7f05ec0d0e80 2024-04-16T16:21:11.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.391+0000 7f05f0e00640 1 -- 172.21.15.45:0/170717283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05ec0d0e80 msgr2=0x7f05ec0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:11.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.391+0000 7f05f0e00640 1 --2- 172.21.15.45:0/170717283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05ec0d0e80 0x7f05ec0b4e20 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f05ec073470 tx=0x7f05dc02e0a0 comp rx=0 tx=0).stop 2024-04-16T16:21:11.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.391+0000 7f05f0e00640 1 -- 172.21.15.45:0/170717283 shutdown_connections 2024-04-16T16:21:11.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.391+0000 7f05f0e00640 1 --2- 172.21.15.45:0/170717283 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05ec0afda0 0x7f05ec06ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:11.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.391+0000 7f05f0e00640 1 --2- 172.21.15.45:0/170717283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05ec0d0e80 0x7f05ec0b4e20 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:11.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.391+0000 7f05f0e00640 1 -- 172.21.15.45:0/170717283 >> 172.21.15.45:0/170717283 conn(0x7f05ec0b3970 msgr2=0x7f05ec0b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:11.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.391+0000 7f05f0e00640 1 -- 172.21.15.45:0/170717283 shutdown_connections 2024-04-16T16:21:11.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.392+0000 7f05f0e00640 1 -- 172.21.15.45:0/170717283 wait complete. 2024-04-16T16:21:11.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.393+0000 7f05f0e00640 1 Processor -- start 2024-04-16T16:21:11.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.393+0000 7f05f0e00640 1 -- start start 2024-04-16T16:21:11.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.394+0000 7f05f0e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05ec0afda0 0x7f05ec1530c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:11.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.394+0000 7f05f0e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05ec0d0e80 0x7f05ec1516b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:11.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.394+0000 7f05f0e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05ec15b920 0x7f05ec15c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:11.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.394+0000 7f05f0e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f05ec0ce760 con 0x7f05ec0d0e80 2024-04-16T16:21:11.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.394+0000 7f05f0e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f05ec0ce5e0 con 0x7f05ec15b920 2024-04-16T16:21:11.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.394+0000 7f05f0e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f05ec0ce8e0 con 0x7f05ec0afda0 2024-04-16T16:21:11.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.395+0000 7f05e9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05ec0d0e80 0x7f05ec1516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:11.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.395+0000 7f05e9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05ec0d0e80 0x7f05ec1516b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48588/0 (socket says 172.21.15.45:48588) 2024-04-16T16:21:11.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.395+0000 7f05e9400640 1 -- 172.21.15.45:0/3671454531 learned_addr learned my addr 172.21.15.45:0/3671454531 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:11.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.395+0000 7f05e9e00640 1 --2- 172.21.15.45:0/3671454531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05ec0afda0 0x7f05ec1530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:11.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.395+0000 7f05ea800640 1 --2- 172.21.15.45:0/3671454531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05ec15b920 0x7f05ec15c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:11.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.395+0000 7f05e9400640 1 -- 172.21.15.45:0/3671454531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05ec0afda0 msgr2=0x7f05ec1530c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:11.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.395+0000 7f05e9400640 1 --2- 172.21.15.45:0/3671454531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05ec0afda0 0x7f05ec1530c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:11.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.395+0000 7f05e9400640 1 -- 172.21.15.45:0/3671454531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05ec15b920 msgr2=0x7f05ec15c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:11.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.395+0000 7f05e9400640 1 --2- 172.21.15.45:0/3671454531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05ec15b920 0x7f05ec15c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:11.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.395+0000 7f05e9400640 1 -- 172.21.15.45:0/3671454531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05ec152020 con 0x7f05ec0d0e80 2024-04-16T16:21:11.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.396+0000 7f05e9400640 1 --2- 172.21.15.45:0/3671454531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05ec0d0e80 0x7f05ec1516b0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00d240 tx=0x7f05dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:11.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.396+0000 7f05e2000640 1 -- 172.21.15.45:0/3671454531 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc02ea40 con 0x7f05ec0d0e80 2024-04-16T16:21:11.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.396+0000 7f05e2000640 1 -- 172.21.15.45:0/3671454531 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05dc045070 con 0x7f05ec0d0e80 2024-04-16T16:21:11.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.396+0000 7f05e2000640 1 -- 172.21.15.45:0/3671454531 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc00b720 con 0x7f05ec0d0e80 2024-04-16T16:21:11.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.396+0000 7f05f0e00640 1 -- 172.21.15.45:0/3671454531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05ec152300 con 0x7f05ec0d0e80 2024-04-16T16:21:11.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.396+0000 7f05f0e00640 1 -- 172.21.15.45:0/3671454531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05ec1528c0 con 0x7f05ec0d0e80 2024-04-16T16:21:11.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.397+0000 7f05f0e00640 1 -- 172.21.15.45:0/3671454531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05ec075780 con 0x7f05ec0d0e80 2024-04-16T16:21:11.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.398+0000 7f05e2000640 1 -- 172.21.15.45:0/3671454531 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f05dc036a40 con 0x7f05ec0d0e80 2024-04-16T16:21:11.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.399+0000 7f05e2000640 1 --2- 172.21.15.45:0/3671454531 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f05c806ca40 0x7f05c806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:11.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.399+0000 7f05e2000640 1 -- 172.21.15.45:0/3671454531 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 6483+0+0 (secure 0 0 0) 0x7f05dc0ae8d0 con 0x7f05ec0d0e80 2024-04-16T16:21:11.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.400+0000 7f05e9e00640 1 --2- 172.21.15.45:0/3671454531 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f05c806ca40 0x7f05c806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:11.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.401+0000 7f05e9e00640 1 --2- 172.21.15.45:0/3671454531 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f05c806ca40 0x7f05c806ef00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f05d4006fd0 tx=0x7f05d4009060 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:11.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.404+0000 7f05e2000640 1 -- 172.21.15.45:0/3671454531 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f05dc04e020 con 0x7f05ec0d0e80 2024-04-16T16:21:11.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:11.651+0000 7f05f0e00640 1 -- 172.21.15.45:0/3671454531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f05ec0cdbd0 con 0x7f05ec0d0e80 2024-04-16T16:21:12.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.171+0000 7f05e2000640 1 -- 172.21.15.45:0/3671454531 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v93) ==== 104+0+0 (secure 0 0 0) 0x7f05dc07c850 con 0x7f05ec0d0e80 2024-04-16T16:21:12.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.174+0000 7f05f0e00640 1 -- 172.21.15.45:0/3671454531 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f05c806ca40 msgr2=0x7f05c806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:12.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.174+0000 7f05f0e00640 1 --2- 172.21.15.45:0/3671454531 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f05c806ca40 0x7f05c806ef00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f05d4006fd0 tx=0x7f05d4009060 comp rx=0 tx=0).stop 2024-04-16T16:21:12.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.174+0000 7f05f0e00640 1 -- 172.21.15.45:0/3671454531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05ec0d0e80 msgr2=0x7f05ec1516b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:12.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.174+0000 7f05f0e00640 1 --2- 172.21.15.45:0/3671454531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05ec0d0e80 0x7f05ec1516b0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00d240 tx=0x7f05dc004a40 comp rx=0 tx=0).stop 2024-04-16T16:21:12.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.174+0000 7f05f0e00640 1 -- 172.21.15.45:0/3671454531 shutdown_connections 2024-04-16T16:21:12.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.174+0000 7f05f0e00640 1 --2- 172.21.15.45:0/3671454531 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f05c806ca40 0x7f05c806ef00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:12.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.174+0000 7f05f0e00640 1 --2- 172.21.15.45:0/3671454531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05ec15b920 0x7f05ec15c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:12.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.174+0000 7f05f0e00640 1 --2- 172.21.15.45:0/3671454531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05ec0d0e80 0x7f05ec1516b0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:12.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.174+0000 7f05f0e00640 1 --2- 172.21.15.45:0/3671454531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f05ec0afda0 0x7f05ec1530c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:12.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.174+0000 7f05f0e00640 1 -- 172.21.15.45:0/3671454531 >> 172.21.15.45:0/3671454531 conn(0x7f05ec0b3970 msgr2=0x7f05ec0b2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:12.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.175+0000 7f05f0e00640 1 -- 172.21.15.45:0/3671454531 shutdown_connections 2024-04-16T16:21:12.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.175+0000 7f05f0e00640 1 -- 172.21.15.45:0/3671454531 wait complete. 2024-04-16T16:21:12.201 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-16T16:21:12.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.472+0000 7f5b17e00640 1 Processor -- start 2024-04-16T16:21:12.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.472+0000 7f5b17e00640 1 -- start start 2024-04-16T16:21:12.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.474+0000 7f5b17e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b18146180 0x7f5b1806e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:12.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.474+0000 7f5b17e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b1806e970 0x7f5b18071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:12.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.474+0000 7f5b17e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5b18072480 0x7f5b181602b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:12.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.474+0000 7f5b17e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5b1806edd0 con 0x7f5b1806e970 2024-04-16T16:21:12.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.474+0000 7f5b17e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5b1815cc90 con 0x7f5b18072480 2024-04-16T16:21:12.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.474+0000 7f5b17e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5b1815d520 con 0x7f5b18146180 2024-04-16T16:21:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.474+0000 7f5b16000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b1806e970 0x7f5b18071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.474+0000 7f5b16000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b1806e970 0x7f5b18071f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48604/0 (socket says 172.21.15.45:48604) 2024-04-16T16:21:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.474+0000 7f5b16000640 1 -- 172.21.15.45:0/2848673600 learned_addr learned my addr 172.21.15.45:0/2848673600 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.475+0000 7f5b17400640 1 --2- 172.21.15.45:0/2848673600 >> v2:172.21.15.103:3300/0 conn(0x7f5b18072480 0x7f5b181602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:12.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.475+0000 7f5b16a00640 1 --2- 172.21.15.45:0/2848673600 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b18146180 0x7f5b1806e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:12.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.475+0000 7f5b16000640 1 -- 172.21.15.45:0/2848673600 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b18146180 msgr2=0x7f5b1806e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:12.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.475+0000 7f5b16000640 1 --2- 172.21.15.45:0/2848673600 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b18146180 0x7f5b1806e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:12.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.475+0000 7f5b16000640 1 -- 172.21.15.45:0/2848673600 >> v2:172.21.15.103:3300/0 conn(0x7f5b18072480 msgr2=0x7f5b181602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:12.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.475+0000 7f5b16000640 1 --2- 172.21.15.45:0/2848673600 >> v2:172.21.15.103:3300/0 conn(0x7f5b18072480 0x7f5b181602b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:12.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.475+0000 7f5b16000640 1 -- 172.21.15.45:0/2848673600 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b18072c70 con 0x7f5b1806e970 2024-04-16T16:21:12.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.475+0000 7f5b16a00640 1 --2- 172.21.15.45:0/2848673600 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b18146180 0x7f5b1806e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:12.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.476+0000 7f5b16000640 1 --2- 172.21.15.45:0/2848673600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b1806e970 0x7f5b18071f40 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f5b180737c0 tx=0x7f5b0402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3fd7387b964c321 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:12.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.477+0000 7f5b15600640 1 -- 172.21.15.45:0/2848673600 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b040317c0 con 0x7f5b1806e970 2024-04-16T16:21:12.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.477+0000 7f5b15600640 1 -- 172.21.15.45:0/2848673600 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b04037070 con 0x7f5b1806e970 2024-04-16T16:21:12.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.477+0000 7f5b15600640 1 -- 172.21.15.45:0/2848673600 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b04031b80 con 0x7f5b1806e970 2024-04-16T16:21:12.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.477+0000 7f5b17e00640 1 -- 172.21.15.45:0/2848673600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b1806e970 msgr2=0x7f5b18071f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:12.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.477+0000 7f5b17e00640 1 --2- 172.21.15.45:0/2848673600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b1806e970 0x7f5b18071f40 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f5b180737c0 tx=0x7f5b0402b860 comp rx=0 tx=0).stop 2024-04-16T16:21:12.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.477+0000 7f5b17e00640 1 -- 172.21.15.45:0/2848673600 shutdown_connections 2024-04-16T16:21:12.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.477+0000 7f5b17e00640 1 --2- 172.21.15.45:0/2848673600 >> v2:172.21.15.103:3300/0 conn(0x7f5b18072480 0x7f5b181602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:12.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.477+0000 7f5b17e00640 1 --2- 172.21.15.45:0/2848673600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b1806e970 0x7f5b18071f40 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:12.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.477+0000 7f5b17e00640 1 --2- 172.21.15.45:0/2848673600 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b18146180 0x7f5b1806e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:12.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.477+0000 7f5b17e00640 1 -- 172.21.15.45:0/2848673600 >> 172.21.15.45:0/2848673600 conn(0x7f5b180b3bb0 msgr2=0x7f5b180b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:12.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.478+0000 7f5b17e00640 1 -- 172.21.15.45:0/2848673600 shutdown_connections 2024-04-16T16:21:12.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.478+0000 7f5b17e00640 1 -- 172.21.15.45:0/2848673600 wait complete. 2024-04-16T16:21:12.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.480+0000 7f5b17e00640 1 Processor -- start 2024-04-16T16:21:12.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.480+0000 7f5b17e00640 1 -- start start 2024-04-16T16:21:12.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.481+0000 7f5b17e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b1806e970 0x7f5b1815c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:12.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.481+0000 7f5b17e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b18072480 0x7f5b181f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:12.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.481+0000 7f5b17e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b18146180 0x7f5b180c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:12.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.481+0000 7f5b17e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5b180728f0 con 0x7f5b1806e970 2024-04-16T16:21:12.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.481+0000 7f5b17e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1815d1b0 con 0x7f5b18146180 2024-04-16T16:21:12.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.481+0000 7f5b17e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5b181607f0 con 0x7f5b18072480 2024-04-16T16:21:12.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.481+0000 7f5b16a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b1806e970 0x7f5b1815c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:12.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.482+0000 7f5b16a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b1806e970 0x7f5b1815c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48610/0 (socket says 172.21.15.45:48610) 2024-04-16T16:21:12.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.482+0000 7f5b16a00640 1 -- 172.21.15.45:0/572948750 learned_addr learned my addr 172.21.15.45:0/572948750 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.482+0000 7f5b16a00640 1 -- 172.21.15.45:0/572948750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b18072480 msgr2=0x7f5b181f5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.482+0000 7f5b16000640 1 --2- 172.21.15.45:0/572948750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b18072480 0x7f5b181f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.482+0000 7f5b17400640 1 --2- 172.21.15.45:0/572948750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b18146180 0x7f5b180c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.482+0000 7f5b16a00640 1 --2- 172.21.15.45:0/572948750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b18072480 0x7f5b181f5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.482+0000 7f5b16a00640 1 -- 172.21.15.45:0/572948750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b18146180 msgr2=0x7f5b180c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.482+0000 7f5b16a00640 1 --2- 172.21.15.45:0/572948750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b18146180 0x7f5b180c97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:12.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.482+0000 7f5b16a00640 1 -- 172.21.15.45:0/572948750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b1815c5c0 con 0x7f5b1806e970 2024-04-16T16:21:12.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.482+0000 7f5b16000640 1 --2- 172.21.15.45:0/572948750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b18072480 0x7f5b181f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:12.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.482+0000 7f5b16a00640 1 --2- 172.21.15.45:0/572948750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b1806e970 0x7f5b1815c080 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800c7f0 tx=0x7f5b0800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:12.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.483+0000 7f5b0ea00640 1 -- 172.21.15.45:0/572948750 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08015070 con 0x7f5b1806e970 2024-04-16T16:21:12.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.483+0000 7f5b0ea00640 1 -- 172.21.15.45:0/572948750 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b08007820 con 0x7f5b1806e970 2024-04-16T16:21:12.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.483+0000 7f5b17e00640 1 -- 172.21.15.45:0/572948750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b180c6cd0 con 0x7f5b1806e970 2024-04-16T16:21:12.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.483+0000 7f5b0ea00640 1 -- 172.21.15.45:0/572948750 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08007bd0 con 0x7f5b1806e970 2024-04-16T16:21:12.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.483+0000 7f5b17e00640 1 -- 172.21.15.45:0/572948750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b180c7290 con 0x7f5b1806e970 2024-04-16T16:21:12.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.484+0000 7f5b17e00640 1 -- 172.21.15.45:0/572948750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b18075780 con 0x7f5b1806e970 2024-04-16T16:21:12.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.486+0000 7f5b0ea00640 1 -- 172.21.15.45:0/572948750 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f5b08018070 con 0x7f5b1806e970 2024-04-16T16:21:12.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.487+0000 7f5b0ea00640 1 --2- 172.21.15.45:0/572948750 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5af806ca10 0x7f5af806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:12.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.487+0000 7f5b0ea00640 1 -- 172.21.15.45:0/572948750 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 6483+0+0 (secure 0 0 0) 0x7f5b0808c090 con 0x7f5b1806e970 2024-04-16T16:21:12.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.488+0000 7f5b16000640 1 --2- 172.21.15.45:0/572948750 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5af806ca10 0x7f5af806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:12.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.489+0000 7f5b16000640 1 --2- 172.21.15.45:0/572948750 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5af806ca10 0x7f5af806eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5b04036000 tx=0x7f5b04008030 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.492+0000 7f5b0ea00640 1 -- 172.21.15.45:0/572948750 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5b08059ef0 con 0x7f5b1806e970 2024-04-16T16:21:12.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:12.697+0000 7f5b17e00640 1 -- 172.21.15.45:0/572948750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5b181f61e0 con 0x7f5b1806e970 2024-04-16T16:21:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.430+0000 7f5b0ea00640 1 -- 172.21.15.45:0/572948750 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v132) ==== 181+0+0 (secure 0 0 0) 0x7f5b080597f0 con 0x7f5b1806e970 2024-04-16T16:21:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2024-04-16T16:21:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.436+0000 7f5b17e00640 1 -- 172.21.15.45:0/572948750 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5af806ca10 msgr2=0x7f5af806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.436+0000 7f5b17e00640 1 --2- 172.21.15.45:0/572948750 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5af806ca10 0x7f5af806eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5b04036000 tx=0x7f5b04008030 comp rx=0 tx=0).stop 2024-04-16T16:21:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.436+0000 7f5b17e00640 1 -- 172.21.15.45:0/572948750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b1806e970 msgr2=0x7f5b1815c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.436+0000 7f5b17e00640 1 --2- 172.21.15.45:0/572948750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b1806e970 0x7f5b1815c080 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800c7f0 tx=0x7f5b0800ccb0 comp rx=0 tx=0).stop 2024-04-16T16:21:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.436+0000 7f5b17e00640 1 -- 172.21.15.45:0/572948750 shutdown_connections 2024-04-16T16:21:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.436+0000 7f5b17e00640 1 --2- 172.21.15.45:0/572948750 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5af806ca10 0x7f5af806eed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.436+0000 7f5b17e00640 1 --2- 172.21.15.45:0/572948750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b18146180 0x7f5b180c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.436+0000 7f5b17e00640 1 --2- 172.21.15.45:0/572948750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b18072480 0x7f5b181f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.436+0000 7f5b17e00640 1 --2- 172.21.15.45:0/572948750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b1806e970 0x7f5b1815c080 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.436+0000 7f5b17e00640 1 -- 172.21.15.45:0/572948750 >> 172.21.15.45:0/572948750 conn(0x7f5b180b3bb0 msgr2=0x7f5b18142d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.436+0000 7f5b17e00640 1 -- 172.21.15.45:0/572948750 shutdown_connections 2024-04-16T16:21:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.436+0000 7f5b17e00640 1 -- 172.21.15.45:0/572948750 wait complete. 2024-04-16T16:21:13.460 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-16T16:21:13.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.686+0000 7fa243e00640 1 Processor -- start 2024-04-16T16:21:13.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.686+0000 7fa243e00640 1 -- start start 2024-04-16T16:21:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.687+0000 7fa243e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa244155ab0 0x7fa244162ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.687+0000 7fa243e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2440ba610 0x7fa24415efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.687+0000 7fa243e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa2440d18c0 0x7fa2440d1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.687+0000 7fa243e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa24415ff00 con 0x7fa244155ab0 2024-04-16T16:21:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.687+0000 7fa243e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa244155e90 con 0x7fa2440d18c0 2024-04-16T16:21:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.687+0000 7fa243e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa24415f940 con 0x7fa2440ba610 2024-04-16T16:21:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.688+0000 7fa242a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa244155ab0 0x7fa244162ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.688+0000 7fa242a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa244155ab0 0x7fa244162ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56142/0 (socket says 172.21.15.45:56142) 2024-04-16T16:21:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.688+0000 7fa242a00640 1 -- 172.21.15.45:0/1398758879 learned_addr learned my addr 172.21.15.45:0/1398758879 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.688+0000 7fa243400640 1 --2- 172.21.15.45:0/1398758879 >> v2:172.21.15.103:3300/0 conn(0x7fa2440d18c0 0x7fa2440d1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.688+0000 7fa242a00640 1 -- 172.21.15.45:0/1398758879 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2440ba610 msgr2=0x7fa24415efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.688+0000 7fa242a00640 1 --2- 172.21.15.45:0/1398758879 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2440ba610 0x7fa24415efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.688+0000 7fa242a00640 1 -- 172.21.15.45:0/1398758879 >> v2:172.21.15.103:3300/0 conn(0x7fa2440d18c0 msgr2=0x7fa2440d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.688+0000 7fa242a00640 1 --2- 172.21.15.45:0/1398758879 >> v2:172.21.15.103:3300/0 conn(0x7fa2440d18c0 0x7fa2440d1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.688+0000 7fa242a00640 1 -- 172.21.15.45:0/1398758879 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa24415fc20 con 0x7fa244155ab0 2024-04-16T16:21:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.689+0000 7fa242a00640 1 --2- 172.21.15.45:0/1398758879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa244155ab0 0x7fa244162ef0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fa2440baa70 tx=0x7fa23402b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af15a9d7322eddee server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.690+0000 7fa241600640 1 -- 172.21.15.45:0/1398758879 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2340317f0 con 0x7fa244155ab0 2024-04-16T16:21:13.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.690+0000 7fa241600640 1 -- 172.21.15.45:0/1398758879 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa234037070 con 0x7fa244155ab0 2024-04-16T16:21:13.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.690+0000 7fa241600640 1 -- 172.21.15.45:0/1398758879 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa234031bb0 con 0x7fa244155ab0 2024-04-16T16:21:13.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.690+0000 7fa243e00640 1 -- 172.21.15.45:0/1398758879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa244155ab0 msgr2=0x7fa244162ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:13.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.690+0000 7fa243e00640 1 --2- 172.21.15.45:0/1398758879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa244155ab0 0x7fa244162ef0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fa2440baa70 tx=0x7fa23402b890 comp rx=0 tx=0).stop 2024-04-16T16:21:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.690+0000 7fa243e00640 1 -- 172.21.15.45:0/1398758879 shutdown_connections 2024-04-16T16:21:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.690+0000 7fa243e00640 1 --2- 172.21.15.45:0/1398758879 >> v2:172.21.15.103:3300/0 conn(0x7fa2440d18c0 0x7fa2440d1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.690+0000 7fa243e00640 1 --2- 172.21.15.45:0/1398758879 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2440ba610 0x7fa24415efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.690+0000 7fa243e00640 1 --2- 172.21.15.45:0/1398758879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa244155ab0 0x7fa244162ef0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.690+0000 7fa243e00640 1 -- 172.21.15.45:0/1398758879 >> 172.21.15.45:0/1398758879 conn(0x7fa24406d1a0 msgr2=0x7fa24406d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.691+0000 7fa243e00640 1 -- 172.21.15.45:0/1398758879 shutdown_connections 2024-04-16T16:21:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.691+0000 7fa243e00640 1 -- 172.21.15.45:0/1398758879 wait complete. 2024-04-16T16:21:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.692+0000 7fa243e00640 1 Processor -- start 2024-04-16T16:21:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.693+0000 7fa243e00640 1 -- start start 2024-04-16T16:21:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.693+0000 7fa243e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2440ba610 0x7fa2440ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.693+0000 7fa243e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2440d18c0 0x7fa2440cc410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.693+0000 7fa243e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa244155ab0 0x7fa2441f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.694+0000 7fa243e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa244163430 con 0x7fa2440d18c0 2024-04-16T16:21:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.694+0000 7fa243e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa2441629d0 con 0x7fa2440ba610 2024-04-16T16:21:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.694+0000 7fa243e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa24415f510 con 0x7fa244155ab0 2024-04-16T16:21:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.694+0000 7fa242000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2440d18c0 0x7fa2440cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.694+0000 7fa242000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2440d18c0 0x7fa2440cc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56148/0 (socket says 172.21.15.45:56148) 2024-04-16T16:21:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.694+0000 7fa242000640 1 -- 172.21.15.45:0/2374856352 learned_addr learned my addr 172.21.15.45:0/2374856352 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.694+0000 7fa242a00640 1 --2- 172.21.15.45:0/2374856352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2440ba610 0x7fa2440ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.694+0000 7fa242000640 1 -- 172.21.15.45:0/2374856352 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa244155ab0 msgr2=0x7fa2441f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.694+0000 7fa242000640 1 --2- 172.21.15.45:0/2374856352 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa244155ab0 0x7fa2441f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.694+0000 7fa242000640 1 -- 172.21.15.45:0/2374856352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2440ba610 msgr2=0x7fa2440ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.694+0000 7fa242000640 1 --2- 172.21.15.45:0/2374856352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2440ba610 0x7fa2440ce0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.694+0000 7fa242000640 1 -- 172.21.15.45:0/2374856352 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2441f5290 con 0x7fa2440d18c0 2024-04-16T16:21:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.695+0000 7fa242000640 1 --2- 172.21.15.45:0/2374856352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2440d18c0 0x7fa2440cc410 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fa23000ba50 tx=0x7fa23000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.695+0000 7fa23b400640 1 -- 172.21.15.45:0/2374856352 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230019070 con 0x7fa2440d18c0 2024-04-16T16:21:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.695+0000 7fa23b400640 1 -- 172.21.15.45:0/2374856352 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa23000cc00 con 0x7fa2440d18c0 2024-04-16T16:21:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.695+0000 7fa23b400640 1 -- 172.21.15.45:0/2374856352 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230014440 con 0x7fa2440d18c0 2024-04-16T16:21:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.695+0000 7fa243e00640 1 -- 172.21.15.45:0/2374856352 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2441f5570 con 0x7fa2440d18c0 2024-04-16T16:21:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.695+0000 7fa243e00640 1 -- 172.21.15.45:0/2374856352 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2440cbef0 con 0x7fa2440d18c0 2024-04-16T16:21:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.697+0000 7fa243e00640 1 -- 172.21.15.45:0/2374856352 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa244075780 con 0x7fa2440d18c0 2024-04-16T16:21:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.697+0000 7fa23b400640 1 -- 172.21.15.45:0/2374856352 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7fa23001c070 con 0x7fa2440d18c0 2024-04-16T16:21:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.698+0000 7fa23b400640 1 --2- 172.21.15.45:0/2374856352 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa22406ca10 0x7fa22406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.698+0000 7fa23b400640 1 -- 172.21.15.45:0/2374856352 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 6045+0+0 (secure 0 0 0) 0x7fa23008bb60 con 0x7fa2440d18c0 2024-04-16T16:21:13.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.699+0000 7fa242a00640 1 --2- 172.21.15.45:0/2374856352 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa22406ca10 0x7fa22406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:13.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.700+0000 7fa242a00640 1 --2- 172.21.15.45:0/2374856352 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa22406ca10 0x7fa22406eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa234036000 tx=0x7fa23403d040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:13.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.703+0000 7fa23b400640 1 -- 172.21.15.45:0/2374856352 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa230059dc0 con 0x7fa2440d18c0 2024-04-16T16:21:13.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:13.927+0000 7fa243e00640 1 -- 172.21.15.45:0/2374856352 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa2440b9f10 con 0x7fa2440d18c0 2024-04-16T16:21:14.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.434+0000 7fa23b400640 1 -- 172.21.15.45:0/2374856352 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v133) ==== 169+0+0 (secure 0 0 0) 0x7fa2300596c0 con 0x7fa2440d18c0 2024-04-16T16:21:14.435 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2024-04-16T16:21:14.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.438+0000 7fa243e00640 1 -- 172.21.15.45:0/2374856352 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa22406ca10 msgr2=0x7fa22406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:14.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.438+0000 7fa243e00640 1 --2- 172.21.15.45:0/2374856352 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa22406ca10 0x7fa22406eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa234036000 tx=0x7fa23403d040 comp rx=0 tx=0).stop 2024-04-16T16:21:14.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.438+0000 7fa243e00640 1 -- 172.21.15.45:0/2374856352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2440d18c0 msgr2=0x7fa2440cc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:14.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.438+0000 7fa243e00640 1 --2- 172.21.15.45:0/2374856352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2440d18c0 0x7fa2440cc410 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fa23000ba50 tx=0x7fa23000bf10 comp rx=0 tx=0).stop 2024-04-16T16:21:14.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.438+0000 7fa243e00640 1 -- 172.21.15.45:0/2374856352 shutdown_connections 2024-04-16T16:21:14.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.438+0000 7fa243e00640 1 --2- 172.21.15.45:0/2374856352 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa244155ab0 0x7fa2441f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.438+0000 7fa243e00640 1 --2- 172.21.15.45:0/2374856352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2440d18c0 0x7fa2440cc410 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.438+0000 7fa243e00640 1 --2- 172.21.15.45:0/2374856352 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa22406ca10 0x7fa22406eed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.438+0000 7fa243e00640 1 --2- 172.21.15.45:0/2374856352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2440ba610 0x7fa2440ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.438+0000 7fa243e00640 1 -- 172.21.15.45:0/2374856352 >> 172.21.15.45:0/2374856352 conn(0x7fa24406d1a0 msgr2=0x7fa244160340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.438+0000 7fa243e00640 1 -- 172.21.15.45:0/2374856352 shutdown_connections 2024-04-16T16:21:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.438+0000 7fa243e00640 1 -- 172.21.15.45:0/2374856352 wait complete. 2024-04-16T16:21:14.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-16T16:21:14.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.681+0000 7f0f4dc00640 1 Processor -- start 2024-04-16T16:21:14.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.681+0000 7f0f4dc00640 1 -- start start 2024-04-16T16:21:14.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.683+0000 7f0f4dc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f48067c20 0x7f0f480b1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:14.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.683+0000 7f0f4dc00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f4806c040 con 0x7f0f480b2ee0 2024-04-16T16:21:14.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.683+0000 7f0f4dc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f480b21b0 con 0x7f0f4815eca0 2024-04-16T16:21:14.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.683+0000 7f0f4dc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0f4815f080 con 0x7f0f48067c20 2024-04-16T16:21:14.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.684+0000 7f0f47400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0f480b2ee0 0x7f0f480af990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41334/0 (socket says 172.21.15.45:41334) 2024-04-16T16:21:14.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.684+0000 7f0f47400640 1 -- 172.21.15.45:0/2348643947 learned_addr learned my addr 172.21.15.45:0/2348643947 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:14.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.684+0000 7f0f3fe00640 1 --2- 172.21.15.45:0/2348643947 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f48067c20 0x7f0f480b1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:14.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.684+0000 7f0f46000640 1 -- 172.21.15.45:0/2348643947 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2588505279 0 0) 0x7f0f4806c040 con 0x7f0f480b2ee0 2024-04-16T16:21:14.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.684+0000 7f0f46000640 1 -- 172.21.15.45:0/2348643947 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f2c003520 con 0x7f0f480b2ee0 2024-04-16T16:21:14.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.685+0000 7f0f46000640 1 -- 172.21.15.45:0/2348643947 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 717765813 0 0) 0x7f0f2c003520 con 0x7f0f480b2ee0 2024-04-16T16:21:14.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.685+0000 7f0f46000640 1 -- 172.21.15.45:0/2348643947 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f48067c20 msgr2=0x7f0f480b1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:14.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.685+0000 7f0f46000640 1 --2- 172.21.15.45:0/2348643947 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f48067c20 0x7f0f480b1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:14.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.685+0000 7f0f46000640 1 -- 172.21.15.45:0/2348643947 >> v1:172.21.15.103:6789/0 conn(0x7f0f4815eca0 legacy=0x7f0f4806b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:14.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.685+0000 7f0f46000640 1 -- 172.21.15.45:0/2348643947 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f480aef10 con 0x7f0f480b2ee0 2024-04-16T16:21:14.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.685+0000 7f0f46000640 1 -- 172.21.15.45:0/2348643947 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f0f400019f0 con 0x7f0f480b2ee0 2024-04-16T16:21:14.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.685+0000 7f0f46000640 1 -- 172.21.15.45:0/2348643947 >> v1:172.21.15.45:6789/0 conn(0x7f0f480b2ee0 legacy=0x7f0f480af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:14.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.685+0000 7f0f46000640 1 --2- 172.21.15.45:0/2348643947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f2c0039e0 0x7f0f2c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:14.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.685+0000 7f0f46000640 1 --2- 172.21.15.45:0/2348643947 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f48067c20 0x7f0f2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:14.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.685+0000 7f0f46000640 1 --2- 172.21.15.45:0/2348643947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f2c004d40 0x7f0f2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:14.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.685+0000 7f0f46000640 1 -- 172.21.15.45:0/2348643947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4815f080 con 0x7f0f2c0039e0 2024-04-16T16:21:14.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.685+0000 7f0f46000640 1 -- 172.21.15.45:0/2348643947 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c00ba70 con 0x7f0f2c004d40 2024-04-16T16:21:14.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.685+0000 7f0f46000640 1 -- 172.21.15.45:0/2348643947 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0f2c00bc70 con 0x7f0f48067c20 2024-04-16T16:21:14.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.685+0000 7f0f46a00640 1 --2- 172.21.15.45:0/2348643947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f2c0039e0 0x7f0f2c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:14.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.686+0000 7f0f3fe00640 1 --2- 172.21.15.45:0/2348643947 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f48067c20 0x7f0f2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:14.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.686+0000 7f0f47400640 1 --2- 172.21.15.45:0/2348643947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f2c004d40 0x7f0f2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:14.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.686+0000 7f0f46a00640 1 -- 172.21.15.45:0/2348643947 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f48067c20 msgr2=0x7f0f2c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:14.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.686+0000 7f0f46a00640 1 --2- 172.21.15.45:0/2348643947 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f48067c20 0x7f0f2c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:14.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.686+0000 7f0f46a00640 1 -- 172.21.15.45:0/2348643947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f2c004d40 msgr2=0x7f0f2c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:14.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.686+0000 7f0f46a00640 1 --2- 172.21.15.45:0/2348643947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f2c004d40 0x7f0f2c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:14.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.686+0000 7f0f46a00640 1 -- 172.21.15.45:0/2348643947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2c003520 con 0x7f0f2c0039e0 2024-04-16T16:21:14.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.686+0000 7f0f47400640 1 --2- 172.21.15.45:0/2348643947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f2c004d40 0x7f0f2c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:14.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.687+0000 7f0f46a00640 1 --2- 172.21.15.45:0/2348643947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f2c0039e0 0x7f0f2c003e00 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f0f38001720 tx=0x7f0f2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=542dd2ca481deafb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:14.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.687+0000 7f0f46000640 1 -- 172.21.15.45:0/2348643947 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f38001370 con 0x7f0f2c0039e0 2024-04-16T16:21:14.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.687+0000 7f0f46000640 1 -- 172.21.15.45:0/2348643947 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f380052d0 con 0x7f0f2c0039e0 2024-04-16T16:21:14.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.687+0000 7f0f46000640 1 -- 172.21.15.45:0/2348643947 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f38003830 con 0x7f0f2c0039e0 2024-04-16T16:21:14.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.688+0000 7f0f4dc00640 1 -- 172.21.15.45:0/2348643947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f2c0039e0 msgr2=0x7f0f2c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:14.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.688+0000 7f0f4dc00640 1 --2- 172.21.15.45:0/2348643947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f2c0039e0 0x7f0f2c003e00 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f0f38001720 tx=0x7f0f2c005830 comp rx=0 tx=0).stop 2024-04-16T16:21:14.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.688+0000 7f0f47400640 1 -- 172.21.15.45:0/2348643947 reap_dead start 2024-04-16T16:21:14.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.688+0000 7f0f4dc00640 1 -- 172.21.15.45:0/2348643947 shutdown_connections 2024-04-16T16:21:14.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.688+0000 7f0f4dc00640 1 -- 172.21.15.45:0/2348643947 >> 172.21.15.45:0/2348643947 conn(0x7f0f4809aa70 msgr2=0x7f0f480b6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:14.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.688+0000 7f0f4dc00640 1 -- 172.21.15.45:0/2348643947 shutdown_connections 2024-04-16T16:21:14.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.688+0000 7f0f4dc00640 1 -- 172.21.15.45:0/2348643947 wait complete. 2024-04-16T16:21:14.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.690+0000 7f0f4dc00640 1 Processor -- start 2024-04-16T16:21:14.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.690+0000 7f0f4dc00640 1 -- start start 2024-04-16T16:21:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.691+0000 7f0f4dc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f2c0039e0 0x7f0f481531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.691+0000 7f0f4dc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f481536f0 0x7f0f48157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.691+0000 7f0f4dc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f48157d80 0x7f0f481e7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.691+0000 7f0f4dc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4806d470 con 0x7f0f481536f0 2024-04-16T16:21:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.691+0000 7f0f4dc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0f480af500 con 0x7f0f48157d80 2024-04-16T16:21:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.691+0000 7f0f4dc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0f480b2450 con 0x7f0f2c0039e0 2024-04-16T16:21:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.692+0000 7f0f3fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f481536f0 0x7f0f48157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.692+0000 7f0f3fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f481536f0 0x7f0f48157840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56164/0 (socket says 172.21.15.45:56164) 2024-04-16T16:21:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.692+0000 7f0f3fe00640 1 -- 172.21.15.45:0/2062163500 learned_addr learned my addr 172.21.15.45:0/2062163500 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.692+0000 7f0f47400640 1 --2- 172.21.15.45:0/2062163500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f48157d80 0x7f0f481e7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.692+0000 7f0f46a00640 1 --2- 172.21.15.45:0/2062163500 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f2c0039e0 0x7f0f481531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.692+0000 7f0f3fe00640 1 -- 172.21.15.45:0/2062163500 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f2c0039e0 msgr2=0x7f0f481531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.692+0000 7f0f3fe00640 1 --2- 172.21.15.45:0/2062163500 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f2c0039e0 0x7f0f481531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.692+0000 7f0f3fe00640 1 -- 172.21.15.45:0/2062163500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f48157d80 msgr2=0x7f0f481e7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.692+0000 7f0f3fe00640 1 --2- 172.21.15.45:0/2062163500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f48157d80 0x7f0f481e7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.692+0000 7f0f3fe00640 1 -- 172.21.15.45:0/2062163500 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f481567a0 con 0x7f0f481536f0 2024-04-16T16:21:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.693+0000 7f0f3fe00640 1 --2- 172.21.15.45:0/2062163500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f481536f0 0x7f0f48157840 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f0f30011600 tx=0x7f0f30011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.693+0000 7f0f3ea00640 1 -- 172.21.15.45:0/2062163500 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3000f920 con 0x7f0f481536f0 2024-04-16T16:21:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.693+0000 7f0f3ea00640 1 -- 172.21.15.45:0/2062163500 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f3000fac0 con 0x7f0f481536f0 2024-04-16T16:21:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.693+0000 7f0f3ea00640 1 -- 172.21.15.45:0/2062163500 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3000fe70 con 0x7f0f481536f0 2024-04-16T16:21:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.693+0000 7f0f4dc00640 1 -- 172.21.15.45:0/2062163500 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f48156a80 con 0x7f0f481536f0 2024-04-16T16:21:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.693+0000 7f0f4dc00640 1 -- 172.21.15.45:0/2062163500 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f48157040 con 0x7f0f481536f0 2024-04-16T16:21:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.694+0000 7f0f4dc00640 1 -- 172.21.15.45:0/2062163500 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f48075780 con 0x7f0f481536f0 2024-04-16T16:21:14.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.695+0000 7f0f3ea00640 1 -- 172.21.15.45:0/2062163500 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f0f30015460 con 0x7f0f481536f0 2024-04-16T16:21:14.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.696+0000 7f0f3ea00640 1 --2- 172.21.15.45:0/2062163500 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f0f2406ca40 0x7f0f2406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:14.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.696+0000 7f0f3ea00640 1 -- 172.21.15.45:0/2062163500 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5631+0+0 (secure 0 0 0) 0x7f0f3008c530 con 0x7f0f481536f0 2024-04-16T16:21:14.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.697+0000 7f0f46a00640 1 --2- 172.21.15.45:0/2062163500 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f0f2406ca40 0x7f0f2406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:14.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.698+0000 7f0f46a00640 1 --2- 172.21.15.45:0/2062163500 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f0f2406ca40 0x7f0f2406ef00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0f38033000 tx=0x7f0f38009030 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:14.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.701+0000 7f0f3ea00640 1 -- 172.21.15.45:0/2062163500 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0f30019030 con 0x7f0f481536f0 2024-04-16T16:21:14.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.909+0000 7f0f4dc00640 1 -- 172.21.15.45:0/2062163500 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f0f48158a90 con 0x7f0f481536f0 2024-04-16T16:21:14.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.914+0000 7f0f3ea00640 1 -- 172.21.15.45:0/2062163500 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) ==== 104+0+0 (secure 0 0 0) 0x7f0f3005a890 con 0x7f0f481536f0 2024-04-16T16:21:14.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.917+0000 7f0f4dc00640 1 -- 172.21.15.45:0/2062163500 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f0f2406ca40 msgr2=0x7f0f2406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:14.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.917+0000 7f0f4dc00640 1 --2- 172.21.15.45:0/2062163500 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f0f2406ca40 0x7f0f2406ef00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0f38033000 tx=0x7f0f38009030 comp rx=0 tx=0).stop 2024-04-16T16:21:14.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.917+0000 7f0f4dc00640 1 -- 172.21.15.45:0/2062163500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f481536f0 msgr2=0x7f0f48157840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:14.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.917+0000 7f0f4dc00640 1 --2- 172.21.15.45:0/2062163500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f481536f0 0x7f0f48157840 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f0f30011600 tx=0x7f0f30011ac0 comp rx=0 tx=0).stop 2024-04-16T16:21:14.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.918+0000 7f0f4dc00640 1 -- 172.21.15.45:0/2062163500 shutdown_connections 2024-04-16T16:21:14.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.918+0000 7f0f4dc00640 1 --2- 172.21.15.45:0/2062163500 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f0f2406ca40 0x7f0f2406ef00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:14.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.918+0000 7f0f4dc00640 1 --2- 172.21.15.45:0/2062163500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f48157d80 0x7f0f481e7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:14.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.918+0000 7f0f4dc00640 1 --2- 172.21.15.45:0/2062163500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f481536f0 0x7f0f48157840 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:14.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.918+0000 7f0f4dc00640 1 --2- 172.21.15.45:0/2062163500 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f2c0039e0 0x7f0f481531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:14.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.918+0000 7f0f4dc00640 1 -- 172.21.15.45:0/2062163500 >> 172.21.15.45:0/2062163500 conn(0x7f0f4809aa70 msgr2=0x7f0f48161a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:14.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.918+0000 7f0f4dc00640 1 -- 172.21.15.45:0/2062163500 shutdown_connections 2024-04-16T16:21:14.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:14.918+0000 7f0f4dc00640 1 -- 172.21.15.45:0/2062163500 wait complete. 2024-04-16T16:21:14.937 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_metrics_from_rank' 2024-04-16T16:21:15.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.151+0000 7f5594400640 1 Processor -- start 2024-04-16T16:21:15.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.151+0000 7f5594400640 1 -- start start 2024-04-16T16:21:15.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.153+0000 7f5594400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f558c0aed90 0x7f558c072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:15.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.153+0000 7f5594400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f558c147c20 con 0x7f558c0728c0 2024-04-16T16:21:15.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.153+0000 7f5594400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f558c160210 con 0x7f558c147840 2024-04-16T16:21:15.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.153+0000 7f5594400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f558c072d10 con 0x7f558c0aed90 2024-04-16T16:21:15.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.154+0000 7f5591000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f558c0aed90 0x7f558c072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:15.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.154+0000 7f5592400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f558c0728c0 0x7f558c06e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41336/0 (socket says 172.21.15.45:41336) 2024-04-16T16:21:15.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.154+0000 7f5592400640 1 -- 172.21.15.45:0/2707894985 learned_addr learned my addr 172.21.15.45:0/2707894985 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:15.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.154+0000 7f5591000640 1 -- 172.21.15.45:0/2707894985 >> v1:172.21.15.103:6789/0 conn(0x7f558c147840 legacy=0x7f558c15fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:15.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.154+0000 7f5591000640 1 -- 172.21.15.45:0/2707894985 >> v1:172.21.15.45:6789/0 conn(0x7f558c0728c0 legacy=0x7f558c06e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:15.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.154+0000 7f5591000640 1 -- 172.21.15.45:0/2707894985 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558c0af270 con 0x7f558c0aed90 2024-04-16T16:21:15.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.154+0000 7f558be00640 1 -- 172.21.15.45:0/2707894985 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3192018573 0 0) 0x7f558c147c20 con 0x7f558c0728c0 2024-04-16T16:21:15.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.155+0000 7f5591000640 1 --2- 172.21.15.45:0/2707894985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c0aed90 0x7f558c072380 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f5578009d90 tx=0x7f557802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683168d1e6ed75d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.156+0000 7f558be00640 1 -- 172.21.15.45:0/2707894985 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5578039070 con 0x7f558c0aed90 2024-04-16T16:21:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.156+0000 7f558be00640 1 -- 172.21.15.45:0/2707894985 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f557802cc60 con 0x7f558c0aed90 2024-04-16T16:21:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.156+0000 7f558be00640 1 -- 172.21.15.45:0/2707894985 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5578034660 con 0x7f558c0aed90 2024-04-16T16:21:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.156+0000 7f5594400640 1 -- 172.21.15.45:0/2707894985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c0aed90 msgr2=0x7f558c072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:15.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.156+0000 7f5594400640 1 --2- 172.21.15.45:0/2707894985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c0aed90 0x7f558c072380 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f5578009d90 tx=0x7f557802beb0 comp rx=0 tx=0).stop 2024-04-16T16:21:15.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.157+0000 7f5594400640 1 -- 172.21.15.45:0/2707894985 shutdown_connections 2024-04-16T16:21:15.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.157+0000 7f5594400640 1 --2- 172.21.15.45:0/2707894985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c0aed90 0x7f558c072380 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:15.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.157+0000 7f5594400640 1 -- 172.21.15.45:0/2707894985 >> 172.21.15.45:0/2707894985 conn(0x7f558c0b2750 msgr2=0x7f558c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:15.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.157+0000 7f5594400640 1 -- 172.21.15.45:0/2707894985 shutdown_connections 2024-04-16T16:21:15.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.157+0000 7f5594400640 1 -- 172.21.15.45:0/2707894985 wait complete. 2024-04-16T16:21:15.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.159+0000 7f5594400640 1 Processor -- start 2024-04-16T16:21:15.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.159+0000 7f5594400640 1 -- start start 2024-04-16T16:21:15.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.160+0000 7f5594400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f558c0728c0 0x7f558c13f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:15.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.160+0000 7f5594400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c0aed90 0x7f558c1462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:15.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.160+0000 7f5594400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f558c147840 0x7f558c145130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:15.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.160+0000 7f5594400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f558c0af830 con 0x7f558c0aed90 2024-04-16T16:21:15.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.160+0000 7f5594400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f558c0afe80 con 0x7f558c147840 2024-04-16T16:21:15.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.160+0000 7f5594400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f558c0afb10 con 0x7f558c0728c0 2024-04-16T16:21:15.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.161+0000 7f5591000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c0aed90 0x7f558c1462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:15.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.161+0000 7f5591000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c0aed90 0x7f558c1462c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56180/0 (socket says 172.21.15.45:56180) 2024-04-16T16:21:15.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.161+0000 7f5591000640 1 -- 172.21.15.45:0/3411961490 learned_addr learned my addr 172.21.15.45:0/3411961490 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:15.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.161+0000 7f5591a00640 1 --2- 172.21.15.45:0/3411961490 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f558c0728c0 0x7f558c13f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.161+0000 7f5592400640 1 --2- 172.21.15.45:0/3411961490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f558c147840 0x7f558c145130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.161+0000 7f5591000640 1 -- 172.21.15.45:0/3411961490 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f558c0728c0 msgr2=0x7f558c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.161+0000 7f5591000640 1 --2- 172.21.15.45:0/3411961490 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f558c0728c0 0x7f558c13f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.161+0000 7f5591000640 1 -- 172.21.15.45:0/3411961490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f558c147840 msgr2=0x7f558c145130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.161+0000 7f5591000640 1 --2- 172.21.15.45:0/3411961490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f558c147840 0x7f558c145130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.161+0000 7f5591000640 1 -- 172.21.15.45:0/3411961490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558c1469c0 con 0x7f558c0aed90 2024-04-16T16:21:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.162+0000 7f5591000640 1 --2- 172.21.15.45:0/3411961490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c0aed90 0x7f558c1462c0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f5578037040 tx=0x7f5578004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.162+0000 7f5589600640 1 -- 172.21.15.45:0/3411961490 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5578044020 con 0x7f558c0aed90 2024-04-16T16:21:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.162+0000 7f5589600640 1 -- 172.21.15.45:0/3411961490 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f557803f670 con 0x7f558c0aed90 2024-04-16T16:21:15.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.162+0000 7f5589600640 1 -- 172.21.15.45:0/3411961490 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5578039040 con 0x7f558c0aed90 2024-04-16T16:21:15.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.162+0000 7f5594400640 1 -- 172.21.15.45:0/3411961490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f558c0cf710 con 0x7f558c0aed90 2024-04-16T16:21:15.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.162+0000 7f5594400640 1 -- 172.21.15.45:0/3411961490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f558c0cfcd0 con 0x7f558c0aed90 2024-04-16T16:21:15.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.163+0000 7f5594400640 1 -- 172.21.15.45:0/3411961490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f558c075780 con 0x7f558c0aed90 2024-04-16T16:21:15.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.164+0000 7f5589600640 1 -- 172.21.15.45:0/3411961490 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f5578005d10 con 0x7f558c0aed90 2024-04-16T16:21:15.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.165+0000 7f5589600640 1 --2- 172.21.15.45:0/3411961490 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f556c06ca10 0x7f556c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:15.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.165+0000 7f5589600640 1 -- 172.21.15.45:0/3411961490 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5631+0+0 (secure 0 0 0) 0x7f55780ae830 con 0x7f558c0aed90 2024-04-16T16:21:15.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.166+0000 7f5591a00640 1 --2- 172.21.15.45:0/3411961490 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f556c06ca10 0x7f556c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:15.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.167+0000 7f5591a00640 1 --2- 172.21.15.45:0/3411961490 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f556c06ca10 0x7f556c06eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f557c007fd0 tx=0x7f557c009040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.170+0000 7f5589600640 1 -- 172.21.15.45:0/3411961490 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f557804e070 con 0x7f558c0aed90 2024-04-16T16:21:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:15.452+0000 7f5594400640 1 -- 172.21.15.45:0/3411961490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_metrics_from_rank"]} v 0) -- 0x7f558c070220 con 0x7f558c0aed90 2024-04-16T16:21:16.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.459+0000 7f5589600640 1 -- 172.21.15.45:0/3411961490 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_metrics_from_rank"]}]=0 v451) ==== 146+0+0 (secure 0 0 0) 0x7f557807cc30 con 0x7f558c0aed90 2024-04-16T16:21:16.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.464+0000 7f5594400640 1 -- 172.21.15.45:0/3411961490 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f556c06ca10 msgr2=0x7f556c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:16.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.464+0000 7f5594400640 1 --2- 172.21.15.45:0/3411961490 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f556c06ca10 0x7f556c06eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f557c007fd0 tx=0x7f557c009040 comp rx=0 tx=0).stop 2024-04-16T16:21:16.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.464+0000 7f5594400640 1 -- 172.21.15.45:0/3411961490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c0aed90 msgr2=0x7f558c1462c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:16.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.464+0000 7f5594400640 1 --2- 172.21.15.45:0/3411961490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c0aed90 0x7f558c1462c0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f5578037040 tx=0x7f5578004a40 comp rx=0 tx=0).stop 2024-04-16T16:21:16.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.464+0000 7f5594400640 1 -- 172.21.15.45:0/3411961490 shutdown_connections 2024-04-16T16:21:16.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.464+0000 7f5594400640 1 --2- 172.21.15.45:0/3411961490 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f556c06ca10 0x7f556c06eed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:16.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.464+0000 7f5594400640 1 --2- 172.21.15.45:0/3411961490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f558c147840 0x7f558c145130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:16.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.464+0000 7f5594400640 1 --2- 172.21.15.45:0/3411961490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c0aed90 0x7f558c1462c0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:16.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.464+0000 7f5594400640 1 --2- 172.21.15.45:0/3411961490 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f558c0728c0 0x7f558c13f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:16.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.464+0000 7f5594400640 1 -- 172.21.15.45:0/3411961490 >> 172.21.15.45:0/3411961490 conn(0x7f558c0b2750 msgr2=0x7f558c0b3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:16.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.465+0000 7f5594400640 1 -- 172.21.15.45:0/3411961490 shutdown_connections 2024-04-16T16:21:16.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.465+0000 7f5594400640 1 -- 172.21.15.45:0/3411961490 wait complete. 2024-04-16T16:21:16.491 INFO:tasks.cephfs_test_runner:test_metrics_from_rank (tasks.cephfs.test_mds_metrics.TestMDSMetrics) ... ok 2024-04-16T16:21:16.491 INFO:tasks.cephfs_test_runner:Starting test: test_metrics_mds_grow (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2024-04-16T16:21:16.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:21:16.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.712+0000 7f597e000640 1 Processor -- start 2024-04-16T16:21:16.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.712+0000 7f597e000640 1 -- start start 2024-04-16T16:21:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.714+0000 7f597e000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5978146220 0x7f597815ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.714+0000 7f597e000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f59781608a0 0x7f5978160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.714+0000 7f597e000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f597815d360 con 0x7f597806f7a0 2024-04-16T16:21:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.714+0000 7f597e000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5978161230 con 0x7f5978146220 2024-04-16T16:21:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.714+0000 7f597e000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5978160410 con 0x7f59781608a0 2024-04-16T16:21:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.714+0000 7f597cc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5978146220 0x7f597815ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.714+0000 7f5977e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f597806f7a0 0x7f597806df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41344/0 (socket says 172.21.15.45:41344) 2024-04-16T16:21:16.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.714+0000 7f5977e00640 1 -- 172.21.15.45:0/1461407676 learned_addr learned my addr 172.21.15.45:0/1461407676 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:16.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.715+0000 7f597cc00640 1 -- 172.21.15.45:0/1461407676 >> v2:172.21.15.103:3300/0 conn(0x7f59781608a0 msgr2=0x7f5978160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:16.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.715+0000 7f597cc00640 1 --2- 172.21.15.45:0/1461407676 >> v2:172.21.15.103:3300/0 conn(0x7f59781608a0 0x7f5978160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:16.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.715+0000 7f597cc00640 1 -- 172.21.15.45:0/1461407676 >> v1:172.21.15.45:6789/0 conn(0x7f597806f7a0 legacy=0x7f597806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:16.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.715+0000 7f597cc00640 1 -- 172.21.15.45:0/1461407676 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597815d640 con 0x7f5978146220 2024-04-16T16:21:16.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.715+0000 7f5977400640 1 -- 172.21.15.45:0/1461407676 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 516985580 0 0) 0x7f597815d360 con 0x7f597806f7a0 2024-04-16T16:21:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.716+0000 7f597cc00640 1 --2- 172.21.15.45:0/1461407676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5978146220 0x7f597815ccb0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f59780b3790 tx=0x7f596c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31f549faf5ec91e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.716+0000 7f5977400640 1 -- 172.21.15.45:0/1461407676 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596c035020 con 0x7f5978146220 2024-04-16T16:21:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.716+0000 7f5977400640 1 -- 172.21.15.45:0/1461407676 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f596c031830 con 0x7f5978146220 2024-04-16T16:21:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.716+0000 7f5977400640 1 -- 172.21.15.45:0/1461407676 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596c031c00 con 0x7f5978146220 2024-04-16T16:21:16.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.717+0000 7f597e000640 1 -- 172.21.15.45:0/1461407676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5978146220 msgr2=0x7f597815ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:16.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.717+0000 7f597e000640 1 --2- 172.21.15.45:0/1461407676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5978146220 0x7f597815ccb0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f59780b3790 tx=0x7f596c02b8d0 comp rx=0 tx=0).stop 2024-04-16T16:21:16.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.717+0000 7f597e000640 1 -- 172.21.15.45:0/1461407676 shutdown_connections 2024-04-16T16:21:16.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.717+0000 7f597e000640 1 --2- 172.21.15.45:0/1461407676 >> v2:172.21.15.103:3300/0 conn(0x7f59781608a0 0x7f5978160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:16.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.717+0000 7f597e000640 1 --2- 172.21.15.45:0/1461407676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5978146220 0x7f597815ccb0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:16.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.717+0000 7f597e000640 1 -- 172.21.15.45:0/1461407676 >> 172.21.15.45:0/1461407676 conn(0x7f597806b930 msgr2=0x7f597806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:16.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.717+0000 7f597e000640 1 -- 172.21.15.45:0/1461407676 shutdown_connections 2024-04-16T16:21:16.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.717+0000 7f597e000640 1 -- 172.21.15.45:0/1461407676 wait complete. 2024-04-16T16:21:16.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.719+0000 7f597e000640 1 Processor -- start 2024-04-16T16:21:16.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.719+0000 7f597e000640 1 -- start start 2024-04-16T16:21:16.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.720+0000 7f597e000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f597806f7a0 0x7f597815b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.720+0000 7f597e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5978146220 0x7f5978137980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.720+0000 7f597e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59781608a0 0x7f597813ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.720+0000 7f597e000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f597815dc00 con 0x7f5978146220 2024-04-16T16:21:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.720+0000 7f597e000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f597806e650 con 0x7f597806f7a0 2024-04-16T16:21:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.720+0000 7f597e000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f597806fc20 con 0x7f59781608a0 2024-04-16T16:21:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.721+0000 7f5977e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5978146220 0x7f5978137980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.721+0000 7f5977e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5978146220 0x7f5978137980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56198/0 (socket says 172.21.15.45:56198) 2024-04-16T16:21:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.721+0000 7f5977e00640 1 -- 172.21.15.45:0/2848039010 learned_addr learned my addr 172.21.15.45:0/2848039010 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.721+0000 7f597cc00640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f597806f7a0 0x7f597815b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.721+0000 7f597d600640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59781608a0 0x7f597813ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.721+0000 7f5977e00640 1 -- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59781608a0 msgr2=0x7f597813ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.721+0000 7f5977e00640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59781608a0 0x7f597813ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.721+0000 7f5977e00640 1 -- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f597806f7a0 msgr2=0x7f597815b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.721+0000 7f5977e00640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f597806f7a0 0x7f597815b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.721+0000 7f5977e00640 1 -- 172.21.15.45:0/2848039010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5978144dc0 con 0x7f5978146220 2024-04-16T16:21:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.721+0000 7f597cc00640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f597806f7a0 0x7f597815b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.721+0000 7f597d600640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59781608a0 0x7f597813ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.721+0000 7f5977e00640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5978146220 0x7f5978137980 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f596400c4d0 tx=0x7f596400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.722+0000 7f5974c00640 1 -- 172.21.15.45:0/2848039010 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5964018070 con 0x7f5978146220 2024-04-16T16:21:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.722+0000 7f5974c00640 1 -- 172.21.15.45:0/2848039010 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5964007500 con 0x7f5978146220 2024-04-16T16:21:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.722+0000 7f5974c00640 1 -- 172.21.15.45:0/2848039010 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5964004a20 con 0x7f5978146220 2024-04-16T16:21:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.722+0000 7f597e000640 1 -- 172.21.15.45:0/2848039010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5978145100 con 0x7f5978146220 2024-04-16T16:21:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.722+0000 7f597e000640 1 -- 172.21.15.45:0/2848039010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f597815bce0 con 0x7f5978146220 2024-04-16T16:21:16.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.723+0000 7f597e000640 1 -- 172.21.15.45:0/2848039010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5978075780 con 0x7f5978146220 2024-04-16T16:21:16.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.724+0000 7f5974c00640 1 -- 172.21.15.45:0/2848039010 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f596401b070 con 0x7f5978146220 2024-04-16T16:21:16.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.725+0000 7f5974c00640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f595806ca40 0x7f595806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:16.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.725+0000 7f5974c00640 1 -- 172.21.15.45:0/2848039010 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5631+0+0 (secure 0 0 0) 0x7f596408aff0 con 0x7f5978146220 2024-04-16T16:21:16.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.726+0000 7f597cc00640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f595806ca40 0x7f595806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:16.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.727+0000 7f597cc00640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f595806ca40 0x7f595806ef00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f596c036000 tx=0x7f596c008030 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:16.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.730+0000 7f5974c00640 1 -- 172.21.15.45:0/2848039010 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5978145100 con 0x7f5978146220 2024-04-16T16:21:16.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.935+0000 7f597e000640 1 -- 172.21.15.45:0/2848039010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59780cff60 con 0x7f5978146220 2024-04-16T16:21:16.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.937+0000 7f5974c00640 1 -- 172.21.15.45:0/2848039010 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) ==== 74+0+14048 (secure 0 0 0) 0x7f5964059390 con 0x7f5978146220 2024-04-16T16:21:16.937 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:16.938 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":133,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:21:14.431898+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6833/2060312705":"2024-04-17T16:21:11.165319+0000","172.21.15.103:0/1715153913":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/4225560501":"2024-04-17T16:21:04.341295+0000","172.21.15.103:6832/549890383":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/1495495989":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/3982280724":"2024-04-17T16:21:04.341295+0000","172.21.15.103:6833/1120913661":"2024-04-17T16:20:50.935324+0000","172.21.15.103:6833/549890383":"2024-04-17T16:21:04.341295+0000","172.21.15.103:6832/1120913661":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1543548744":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/4103173879":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1578451701":"2024-04-17T16:21:04.341295+0000","172.21.15.103:0/4177084908":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1961421675":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/2190202958":"2024-04-17T16:20:50.935324+0000","172.21.15.45:6835/2060312705":"2024-04-17T16:21:11.165319+0000","172.21.15.103:0/3679729711":"2024-04-17T16:20:50.935324+0000","172.21.15.103:0/1543358977":"2024-04-17T16:20:50.935324+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:21:16.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.940+0000 7f597e000640 1 -- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f595806ca40 msgr2=0x7f595806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:16.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.940+0000 7f597e000640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f595806ca40 0x7f595806ef00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f596c036000 tx=0x7f596c008030 comp rx=0 tx=0).stop 2024-04-16T16:21:16.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.940+0000 7f597e000640 1 -- 172.21.15.45:0/2848039010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5978146220 msgr2=0x7f5978137980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:16.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.940+0000 7f597e000640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5978146220 0x7f5978137980 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f596400c4d0 tx=0x7f596400c990 comp rx=0 tx=0).stop 2024-04-16T16:21:16.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.940+0000 7f597e000640 1 -- 172.21.15.45:0/2848039010 shutdown_connections 2024-04-16T16:21:16.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.940+0000 7f597e000640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59781608a0 0x7f597813ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:16.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.940+0000 7f597e000640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5978146220 0x7f5978137980 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:16.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.940+0000 7f597e000640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f595806ca40 0x7f595806ef00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:16.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.940+0000 7f597e000640 1 --2- 172.21.15.45:0/2848039010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f597806f7a0 0x7f597815b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:16.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.940+0000 7f597e000640 1 -- 172.21.15.45:0/2848039010 >> 172.21.15.45:0/2848039010 conn(0x7f597806b930 msgr2=0x7f59780c6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:16.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.940+0000 7f597e000640 1 -- 172.21.15.45:0/2848039010 shutdown_connections 2024-04-16T16:21:16.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:16.940+0000 7f597e000640 1 -- 172.21.15.45:0/2848039010 wait complete. 2024-04-16T16:21:16.960 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:21:16.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:21:17.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.190+0000 7f6613200640 1 Processor -- start 2024-04-16T16:21:17.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.190+0000 7f6613200640 1 -- start start 2024-04-16T16:21:17.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.192+0000 7f6613200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f661415ea70 0x7f66140d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:17.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.192+0000 7f6613200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66140cfc40 0x7f6614162560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:17.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.192+0000 7f6613200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f661415ee50 con 0x7f66140d4370 2024-04-16T16:21:17.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.192+0000 7f6613200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f66140d1a10 con 0x7f661415ea70 2024-04-16T16:21:17.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.192+0000 7f6613200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f66140b6480 con 0x7f66140cfc40 2024-04-16T16:21:17.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.192+0000 7f6612800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66140cfc40 0x7f6614162560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:17.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.193+0000 7f6611400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f66140d4370 0x7f66140cf530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:39494/0 (socket says 172.21.15.45:39494) 2024-04-16T16:21:17.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.193+0000 7f6611400640 1 -- 172.21.15.45:0/1129379325 learned_addr learned my addr 172.21.15.45:0/1129379325 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:17.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.193+0000 7f6612800640 1 --2- 172.21.15.45:0/1129379325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66140cfc40 0x7f6614162560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:46408/0 (socket says 172.21.15.45:46408) 2024-04-16T16:21:17.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.193+0000 7f6611e00640 1 --2- 172.21.15.45:0/1129379325 >> v2:172.21.15.103:3300/0 conn(0x7f661415ea70 0x7f66140d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:17.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.193+0000 7f6612800640 1 -- 172.21.15.45:0/1129379325 >> v2:172.21.15.103:3300/0 conn(0x7f661415ea70 msgr2=0x7f66140d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:17.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.193+0000 7f6612800640 1 --2- 172.21.15.45:0/1129379325 >> v2:172.21.15.103:3300/0 conn(0x7f661415ea70 0x7f66140d11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.193+0000 7f6612800640 1 -- 172.21.15.45:0/1129379325 >> v1:172.21.15.103:6789/0 conn(0x7f66140d4370 legacy=0x7f66140cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:17.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.193+0000 7f6612800640 1 -- 172.21.15.45:0/1129379325 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66140b6760 con 0x7f66140cfc40 2024-04-16T16:21:17.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.194+0000 7f6611e00640 1 --2- 172.21.15.45:0/1129379325 >> v2:172.21.15.103:3300/0 conn(0x7f661415ea70 0x7f66140d11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:21:17.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.195+0000 7f6612800640 1 --2- 172.21.15.45:0/1129379325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66140cfc40 0x7f6614162560 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f660c009d90 tx=0x7f660c02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbb0dd02623b8ba6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:17.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.196+0000 7f6610a00640 1 -- 172.21.15.45:0/1129379325 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c039070 con 0x7f66140cfc40 2024-04-16T16:21:17.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.196+0000 7f6610a00640 1 -- 172.21.15.45:0/1129379325 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f660c02cc60 con 0x7f66140cfc40 2024-04-16T16:21:17.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.196+0000 7f6610a00640 1 -- 172.21.15.45:0/1129379325 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c034660 con 0x7f66140cfc40 2024-04-16T16:21:17.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.196+0000 7f6613200640 1 -- 172.21.15.45:0/1129379325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66140cfc40 msgr2=0x7f6614162560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:17.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.196+0000 7f6613200640 1 --2- 172.21.15.45:0/1129379325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66140cfc40 0x7f6614162560 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f660c009d90 tx=0x7f660c02beb0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.196+0000 7f6613200640 1 -- 172.21.15.45:0/1129379325 shutdown_connections 2024-04-16T16:21:17.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.196+0000 7f6613200640 1 --2- 172.21.15.45:0/1129379325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66140cfc40 0x7f6614162560 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.196+0000 7f6613200640 1 --2- 172.21.15.45:0/1129379325 >> v2:172.21.15.103:3300/0 conn(0x7f661415ea70 0x7f66140d11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.196+0000 7f6613200640 1 -- 172.21.15.45:0/1129379325 >> 172.21.15.45:0/1129379325 conn(0x7f661406d3b0 msgr2=0x7f6614073060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:17.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.196+0000 7f6613200640 1 -- 172.21.15.45:0/1129379325 shutdown_connections 2024-04-16T16:21:17.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.197+0000 7f6613200640 1 -- 172.21.15.45:0/1129379325 wait complete. 2024-04-16T16:21:17.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.198+0000 7f6613200640 1 Processor -- start 2024-04-16T16:21:17.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.198+0000 7f6613200640 1 -- start start 2024-04-16T16:21:17.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.199+0000 7f6613200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66140cfc40 0x7f6614153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:17.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.199+0000 7f6613200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66140d4370 0x7f66140cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:17.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.199+0000 7f6613200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f661415ea70 0x7f66140cd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:17.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.199+0000 7f6613200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f66140b6f60 con 0x7f661415ea70 2024-04-16T16:21:17.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.199+0000 7f6613200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f66140d16e0 con 0x7f66140cfc40 2024-04-16T16:21:17.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.199+0000 7f6613200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f66140c9ae0 con 0x7f66140d4370 2024-04-16T16:21:17.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.200+0000 7f6612800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f661415ea70 0x7f66140cd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:17.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.200+0000 7f6612800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f661415ea70 0x7f66140cd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56200/0 (socket says 172.21.15.45:56200) 2024-04-16T16:21:17.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.200+0000 7f6612800640 1 -- 172.21.15.45:0/3034296293 learned_addr learned my addr 172.21.15.45:0/3034296293 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:17.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.200+0000 7f6611400640 1 --2- 172.21.15.45:0/3034296293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66140d4370 0x7f66140cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:17.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.200+0000 7f6611e00640 1 --2- 172.21.15.45:0/3034296293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66140cfc40 0x7f6614153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:17.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.200+0000 7f6612800640 1 -- 172.21.15.45:0/3034296293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66140d4370 msgr2=0x7f66140cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:17.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.200+0000 7f6612800640 1 --2- 172.21.15.45:0/3034296293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66140d4370 0x7f66140cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.200+0000 7f6612800640 1 -- 172.21.15.45:0/3034296293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66140cfc40 msgr2=0x7f6614153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:17.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.200+0000 7f6612800640 1 --2- 172.21.15.45:0/3034296293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66140cfc40 0x7f6614153e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.200+0000 7f6612800640 1 -- 172.21.15.45:0/3034296293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66141543a0 con 0x7f661415ea70 2024-04-16T16:21:17.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.200+0000 7f6611e00640 1 --2- 172.21.15.45:0/3034296293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66140cfc40 0x7f6614153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:17.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.201+0000 7f6612800640 1 --2- 172.21.15.45:0/3034296293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f661415ea70 0x7f66140cd9a0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f660c009d90 tx=0x7f660c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:17.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.201+0000 7f660a000640 1 -- 172.21.15.45:0/3034296293 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c039050 con 0x7f661415ea70 2024-04-16T16:21:17.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.201+0000 7f660a000640 1 -- 172.21.15.45:0/3034296293 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f660c045070 con 0x7f661415ea70 2024-04-16T16:21:17.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.201+0000 7f660a000640 1 -- 172.21.15.45:0/3034296293 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c009450 con 0x7f661415ea70 2024-04-16T16:21:17.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.201+0000 7f6613200640 1 -- 172.21.15.45:0/3034296293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66140c9240 con 0x7f661415ea70 2024-04-16T16:21:17.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.201+0000 7f6613200640 1 -- 172.21.15.45:0/3034296293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66140c9800 con 0x7f661415ea70 2024-04-16T16:21:17.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.202+0000 7f6613200640 1 -- 172.21.15.45:0/3034296293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6614075780 con 0x7f661415ea70 2024-04-16T16:21:17.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.203+0000 7f660a000640 1 -- 172.21.15.45:0/3034296293 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f660c03e850 con 0x7f661415ea70 2024-04-16T16:21:17.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.204+0000 7f660a000640 1 --2- 172.21.15.45:0/3034296293 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f65f406ca10 0x7f65f406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:17.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.204+0000 7f660a000640 1 -- 172.21.15.45:0/3034296293 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5631+0+0 (secure 0 0 0) 0x7f660c0aeae0 con 0x7f661415ea70 2024-04-16T16:21:17.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.205+0000 7f6611e00640 1 --2- 172.21.15.45:0/3034296293 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f65f406ca10 0x7f65f406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:17.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.206+0000 7f6611e00640 1 --2- 172.21.15.45:0/3034296293 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f65f406ca10 0x7f65f406eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f66141546a0 tx=0x7f6604008040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:17.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.208+0000 7f660a000640 1 -- 172.21.15.45:0/3034296293 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f66141543a0 con 0x7f661415ea70 2024-04-16T16:21:17.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.420+0000 7f6613200640 1 -- 172.21.15.45:0/3034296293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f66140d2500 con 0x7f661415ea70 2024-04-16T16:21:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.421+0000 7f660a000640 1 -- 172.21.15.45:0/3034296293 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v133) ==== 93+0+10 (secure 0 0 0) 0x7f660c032e20 con 0x7f661415ea70 2024-04-16T16:21:17.421 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:21:17.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.424+0000 7f6613200640 1 -- 172.21.15.45:0/3034296293 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f65f406ca10 msgr2=0x7f65f406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:17.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.424+0000 7f6613200640 1 --2- 172.21.15.45:0/3034296293 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f65f406ca10 0x7f65f406eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f66141546a0 tx=0x7f6604008040 comp rx=0 tx=0).stop 2024-04-16T16:21:17.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.424+0000 7f6613200640 1 -- 172.21.15.45:0/3034296293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f661415ea70 msgr2=0x7f66140cd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:17.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.424+0000 7f6613200640 1 --2- 172.21.15.45:0/3034296293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f661415ea70 0x7f66140cd9a0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f660c009d90 tx=0x7f660c004a40 comp rx=0 tx=0).stop 2024-04-16T16:21:17.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.424+0000 7f6613200640 1 -- 172.21.15.45:0/3034296293 shutdown_connections 2024-04-16T16:21:17.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.424+0000 7f6613200640 1 --2- 172.21.15.45:0/3034296293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f661415ea70 0x7f66140cd9a0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.424+0000 7f6613200640 1 --2- 172.21.15.45:0/3034296293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66140d4370 0x7f66140cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.424+0000 7f6613200640 1 --2- 172.21.15.45:0/3034296293 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f65f406ca10 0x7f65f406eed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.424+0000 7f6613200640 1 --2- 172.21.15.45:0/3034296293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66140cfc40 0x7f6614153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.424+0000 7f6613200640 1 -- 172.21.15.45:0/3034296293 >> 172.21.15.45:0/3034296293 conn(0x7f661406d3b0 msgr2=0x7f66140b79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:17.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.424+0000 7f6613200640 1 -- 172.21.15.45:0/3034296293 shutdown_connections 2024-04-16T16:21:17.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.424+0000 7f6613200640 1 -- 172.21.15.45:0/3034296293 wait complete. 2024-04-16T16:21:17.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_metrics_mds_grow' 2024-04-16T16:21:17.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.666+0000 7f075e600640 1 Processor -- start 2024-04-16T16:21:17.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.666+0000 7f075e600640 1 -- start start 2024-04-16T16:21:17.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.667+0000 7f075e600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f07580d0e70 0x7f07580d1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:17.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.668+0000 7f075e600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07580d0b70 con 0x7f075815cd40 2024-04-16T16:21:17.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.668+0000 7f075e600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07580d1790 con 0x7f07580cd5a0 2024-04-16T16:21:17.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.668+0000 7f075e600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f075815d1e0 con 0x7f07580d0e70 2024-04-16T16:21:17.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.668+0000 7f0756a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f075815cd40 0x7f07580d0460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41358/0 (socket says 172.21.15.45:41358) 2024-04-16T16:21:17.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.668+0000 7f0756a00640 1 -- 172.21.15.45:0/3799189199 learned_addr learned my addr 172.21.15.45:0/3799189199 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:17.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0757400640 1 --2- 172.21.15.45:0/3799189199 >> v2:172.21.15.103:3300/0 conn(0x7f07580d0e70 0x7f07580d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:17.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0756000640 1 -- 172.21.15.45:0/3799189199 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 607521004 0 0) 0x7f07580d0b70 con 0x7f075815cd40 2024-04-16T16:21:17.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0756000640 1 -- 172.21.15.45:0/3799189199 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f073c003520 con 0x7f075815cd40 2024-04-16T16:21:17.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0756000640 1 -- 172.21.15.45:0/3799189199 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1084361521 0 0) 0x7f073c003520 con 0x7f075815cd40 2024-04-16T16:21:17.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0756000640 1 -- 172.21.15.45:0/3799189199 >> v2:172.21.15.103:3300/0 conn(0x7f07580d0e70 msgr2=0x7f07580d1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:17.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0756000640 1 --2- 172.21.15.45:0/3799189199 >> v2:172.21.15.103:3300/0 conn(0x7f07580d0e70 0x7f07580d1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0756000640 1 -- 172.21.15.45:0/3799189199 >> v1:172.21.15.103:6789/0 conn(0x7f07580cd5a0 legacy=0x7f07580cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:17.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0756000640 1 -- 172.21.15.45:0/3799189199 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075815d4c0 con 0x7f075815cd40 2024-04-16T16:21:17.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0756000640 1 -- 172.21.15.45:0/3799189199 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f0744001670 con 0x7f075815cd40 2024-04-16T16:21:17.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0756000640 1 -- 172.21.15.45:0/3799189199 >> v1:172.21.15.45:6789/0 conn(0x7f075815cd40 legacy=0x7f07580d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:17.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0756000640 1 --2- 172.21.15.45:0/3799189199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c003a00 0x7f073c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:17.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0756000640 1 --2- 172.21.15.45:0/3799189199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f073c004800 0x7f073c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:17.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0756000640 1 --2- 172.21.15.45:0/3799189199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f073c0051c0 0x7f073c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:17.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0756000640 1 -- 172.21.15.45:0/3799189199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f075815d1e0 con 0x7f073c003a00 2024-04-16T16:21:17.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0756000640 1 -- 172.21.15.45:0/3799189199 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f073c00c0c0 con 0x7f073c004800 2024-04-16T16:21:17.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.669+0000 7f0756000640 1 -- 172.21.15.45:0/3799189199 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f073c00c2c0 con 0x7f073c0051c0 2024-04-16T16:21:17.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.670+0000 7f0757400640 1 --2- 172.21.15.45:0/3799189199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c003a00 0x7f073c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:17.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.670+0000 7f0756a00640 1 --2- 172.21.15.45:0/3799189199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f073c004800 0x7f073c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:17.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.670+0000 7f0757400640 1 -- 172.21.15.45:0/3799189199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f073c0051c0 msgr2=0x7f073c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:17.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.670+0000 7f0757e00640 1 --2- 172.21.15.45:0/3799189199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f073c0051c0 0x7f073c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:17.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.670+0000 7f0757400640 1 --2- 172.21.15.45:0/3799189199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f073c0051c0 0x7f073c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.670+0000 7f0757400640 1 -- 172.21.15.45:0/3799189199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f073c004800 msgr2=0x7f073c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:17.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.670+0000 7f0757400640 1 --2- 172.21.15.45:0/3799189199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f073c004800 0x7f073c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.670+0000 7f0757400640 1 -- 172.21.15.45:0/3799189199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073c003520 con 0x7f073c003a00 2024-04-16T16:21:17.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.670+0000 7f0757e00640 1 -- 172.21.15.45:0/3799189199 reap_dead start 2024-04-16T16:21:17.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.671+0000 7f0757400640 1 --2- 172.21.15.45:0/3799189199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c003a00 0x7f073c003e20 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f0748009ac0 tx=0x7f073c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be5e23ef33a63f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:17.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.672+0000 7f0756000640 1 -- 172.21.15.45:0/3799189199 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0748039070 con 0x7f073c003a00 2024-04-16T16:21:17.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.672+0000 7f0756000640 1 -- 172.21.15.45:0/3799189199 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f074802c6b0 con 0x7f073c003a00 2024-04-16T16:21:17.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.672+0000 7f0756000640 1 -- 172.21.15.45:0/3799189199 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0748009690 con 0x7f073c003a00 2024-04-16T16:21:17.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.672+0000 7f075e600640 1 -- 172.21.15.45:0/3799189199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c003a00 msgr2=0x7f073c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:17.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.672+0000 7f075e600640 1 --2- 172.21.15.45:0/3799189199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c003a00 0x7f073c003e20 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f0748009ac0 tx=0x7f073c005830 comp rx=0 tx=0).stop 2024-04-16T16:21:17.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.672+0000 7f075e600640 1 -- 172.21.15.45:0/3799189199 shutdown_connections 2024-04-16T16:21:17.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.672+0000 7f075e600640 1 --2- 172.21.15.45:0/3799189199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c003a00 0x7f073c003e20 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.672+0000 7f075e600640 1 -- 172.21.15.45:0/3799189199 >> 172.21.15.45:0/3799189199 conn(0x7f07580d2890 msgr2=0x7f07580d2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:17.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.673+0000 7f075e600640 1 -- 172.21.15.45:0/3799189199 shutdown_connections 2024-04-16T16:21:17.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.673+0000 7f075e600640 1 -- 172.21.15.45:0/3799189199 wait complete. 2024-04-16T16:21:17.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.674+0000 7f075e600640 1 Processor -- start 2024-04-16T16:21:17.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.674+0000 7f075e600640 1 -- start start 2024-04-16T16:21:17.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.675+0000 7f075e600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f073c003a00 0x7f075815b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:17.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.675+0000 7f075e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07581f48f0 0x7f075806e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.675+0000 7f075e600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0758157c50 0x7f07580c8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.675+0000 7f075e600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f075809b7f0 con 0x7f07581f48f0 2024-04-16T16:21:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.675+0000 7f075e600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f075815da80 con 0x7f073c003a00 2024-04-16T16:21:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.675+0000 7f075e600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f075809b970 con 0x7f0758157c50 2024-04-16T16:21:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.676+0000 7f0756a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07581f48f0 0x7f075806e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.676+0000 7f0756a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07581f48f0 0x7f075806e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56212/0 (socket says 172.21.15.45:56212) 2024-04-16T16:21:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.676+0000 7f0756a00640 1 -- 172.21.15.45:0/3748309521 learned_addr learned my addr 172.21.15.45:0/3748309521 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.676+0000 7f0757400640 1 --2- 172.21.15.45:0/3748309521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f073c003a00 0x7f075815b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.676+0000 7f0756a00640 1 -- 172.21.15.45:0/3748309521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0758157c50 msgr2=0x7f07580c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.676+0000 7f0756a00640 1 --2- 172.21.15.45:0/3748309521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0758157c50 0x7f07580c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.676+0000 7f0756a00640 1 -- 172.21.15.45:0/3748309521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f073c003a00 msgr2=0x7f075815b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.676+0000 7f0756a00640 1 --2- 172.21.15.45:0/3748309521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f073c003a00 0x7f075815b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.676+0000 7f0756a00640 1 -- 172.21.15.45:0/3748309521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075806e9d0 con 0x7f07581f48f0 2024-04-16T16:21:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.677+0000 7f0756a00640 1 --2- 172.21.15.45:0/3748309521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07581f48f0 0x7f075806e490 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f074400e710 tx=0x7f074400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.677+0000 7f074f400640 1 -- 172.21.15.45:0/3748309521 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074400ee20 con 0x7f07581f48f0 2024-04-16T16:21:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.677+0000 7f074f400640 1 -- 172.21.15.45:0/3748309521 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0744018070 con 0x7f07581f48f0 2024-04-16T16:21:17.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.677+0000 7f074f400640 1 -- 172.21.15.45:0/3748309521 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074400cb10 con 0x7f07581f48f0 2024-04-16T16:21:17.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.677+0000 7f075e600640 1 -- 172.21.15.45:0/3748309521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07580b7dc0 con 0x7f07581f48f0 2024-04-16T16:21:17.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.677+0000 7f075e600640 1 -- 172.21.15.45:0/3748309521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07580b8380 con 0x7f07581f48f0 2024-04-16T16:21:17.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.678+0000 7f075e600640 1 -- 172.21.15.45:0/3748309521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0758075780 con 0x7f07581f48f0 2024-04-16T16:21:17.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.679+0000 7f074f400640 1 -- 172.21.15.45:0/3748309521 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f074400ccb0 con 0x7f07581f48f0 2024-04-16T16:21:17.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.681+0000 7f074f400640 1 --2- 172.21.15.45:0/3748309521 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f073806ca10 0x7f073806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:17.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.681+0000 7f074f400640 1 -- 172.21.15.45:0/3748309521 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5631+0+0 (secure 0 0 0) 0x7f074408bc20 con 0x7f07581f48f0 2024-04-16T16:21:17.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.681+0000 7f0757400640 1 --2- 172.21.15.45:0/3748309521 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f073806ca10 0x7f073806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.683+0000 7f0757400640 1 --2- 172.21.15.45:0/3748309521 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f073806ca10 0x7f073806eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0748036000 tx=0x7f0748030000 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.685+0000 7f074f400640 1 -- 172.21.15.45:0/3748309521 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f075806e9d0 con 0x7f07581f48f0 2024-04-16T16:21:17.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:17.970+0000 7f075e600640 1 -- 172.21.15.45:0/3748309521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_metrics_mds_grow"]} v 0) -- 0x7f07580d3710 con 0x7f07581f48f0 2024-04-16T16:21:18.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.472+0000 7f074f400640 1 -- 172.21.15.45:0/3748309521 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_metrics_mds_grow"]}]=0 v453) ==== 148+0+0 (secure 0 0 0) 0x7f0744059ef0 con 0x7f07581f48f0 2024-04-16T16:21:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.477+0000 7f075e600640 1 -- 172.21.15.45:0/3748309521 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f073806ca10 msgr2=0x7f073806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.477+0000 7f075e600640 1 --2- 172.21.15.45:0/3748309521 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f073806ca10 0x7f073806eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0748036000 tx=0x7f0748030000 comp rx=0 tx=0).stop 2024-04-16T16:21:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.477+0000 7f075e600640 1 -- 172.21.15.45:0/3748309521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07581f48f0 msgr2=0x7f075806e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.477+0000 7f075e600640 1 --2- 172.21.15.45:0/3748309521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07581f48f0 0x7f075806e490 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f074400e710 tx=0x7f074400ebd0 comp rx=0 tx=0).stop 2024-04-16T16:21:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.477+0000 7f075e600640 1 -- 172.21.15.45:0/3748309521 shutdown_connections 2024-04-16T16:21:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.477+0000 7f075e600640 1 --2- 172.21.15.45:0/3748309521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0758157c50 0x7f07580c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.478+0000 7f075e600640 1 --2- 172.21.15.45:0/3748309521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07581f48f0 0x7f075806e490 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.478+0000 7f075e600640 1 --2- 172.21.15.45:0/3748309521 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f073806ca10 0x7f073806eed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.478+0000 7f075e600640 1 --2- 172.21.15.45:0/3748309521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f073c003a00 0x7f075815b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.478+0000 7f075e600640 1 -- 172.21.15.45:0/3748309521 >> 172.21.15.45:0/3748309521 conn(0x7f07580d2890 msgr2=0x7f075809ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.478+0000 7f075e600640 1 -- 172.21.15.45:0/3748309521 shutdown_connections 2024-04-16T16:21:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.478+0000 7f075e600640 1 -- 172.21.15.45:0/3748309521 wait complete. 2024-04-16T16:21:18.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-16T16:21:18.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.736+0000 7fb023e00640 1 Processor -- start 2024-04-16T16:21:18.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.736+0000 7fb023e00640 1 -- start start 2024-04-16T16:21:18.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.738+0000 7fb023e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb0241491c0 0x7fb024166840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:18.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.738+0000 7fb023e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb024163cd0 0x7fb0240cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:18.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.738+0000 7fb023e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0241495a0 con 0x7fb0240b58f0 2024-04-16T16:21:18.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.738+0000 7fb023e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb02406e090 con 0x7fb024163cd0 2024-04-16T16:21:18.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.738+0000 7fb023e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb024164140 con 0x7fb0241491c0 2024-04-16T16:21:18.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.738+0000 7fb01be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb024163cd0 0x7fb0240cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:18.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.738+0000 7fb01be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb024163cd0 0x7fb0240cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56216/0 (socket says 172.21.15.45:56216) 2024-04-16T16:21:18.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.738+0000 7fb023400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb0240b58f0 0x7fb02406e520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41362/0 (socket says 172.21.15.45:41362) 2024-04-16T16:21:18.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.738+0000 7fb01be00640 1 -- 172.21.15.45:0/1301536851 learned_addr learned my addr 172.21.15.45:0/1301536851 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:18.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.739+0000 7fb022a00640 1 --2- 172.21.15.45:0/1301536851 >> v2:172.21.15.103:3300/0 conn(0x7fb0241491c0 0x7fb024166840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:18.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.739+0000 7fb01be00640 1 -- 172.21.15.45:0/1301536851 >> v2:172.21.15.103:3300/0 conn(0x7fb0241491c0 msgr2=0x7fb024166840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:18.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.739+0000 7fb01be00640 1 --2- 172.21.15.45:0/1301536851 >> v2:172.21.15.103:3300/0 conn(0x7fb0241491c0 0x7fb024166840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:18.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.739+0000 7fb022000640 1 -- 172.21.15.45:0/1301536851 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3069511796 0 0) 0x7fb0241495a0 con 0x7fb0240b58f0 2024-04-16T16:21:18.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.739+0000 7fb01be00640 1 -- 172.21.15.45:0/1301536851 >> v1:172.21.15.45:6789/0 conn(0x7fb0240b58f0 legacy=0x7fb02406e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:18.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.739+0000 7fb01be00640 1 -- 172.21.15.45:0/1301536851 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb024166e40 con 0x7fb024163cd0 2024-04-16T16:21:18.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.742+0000 7fb01be00640 1 --2- 172.21.15.45:0/1301536851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb024163cd0 0x7fb0240cd100 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fb024070230 tx=0x7fb00c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=757276138584e38 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:18.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.743+0000 7fb022000640 1 -- 172.21.15.45:0/1301536851 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c035020 con 0x7fb024163cd0 2024-04-16T16:21:18.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.743+0000 7fb022000640 1 -- 172.21.15.45:0/1301536851 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00c031870 con 0x7fb024163cd0 2024-04-16T16:21:18.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.743+0000 7fb022000640 1 -- 172.21.15.45:0/1301536851 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c031c40 con 0x7fb024163cd0 2024-04-16T16:21:18.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.743+0000 7fb023e00640 1 -- 172.21.15.45:0/1301536851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb024163cd0 msgr2=0x7fb0240cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:18.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.743+0000 7fb023e00640 1 --2- 172.21.15.45:0/1301536851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb024163cd0 0x7fb0240cd100 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fb024070230 tx=0x7fb00c02b910 comp rx=0 tx=0).stop 2024-04-16T16:21:18.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.744+0000 7fb023e00640 1 -- 172.21.15.45:0/1301536851 shutdown_connections 2024-04-16T16:21:18.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.744+0000 7fb023e00640 1 --2- 172.21.15.45:0/1301536851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb024163cd0 0x7fb0240cd100 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:18.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.744+0000 7fb023e00640 1 --2- 172.21.15.45:0/1301536851 >> v2:172.21.15.103:3300/0 conn(0x7fb0241491c0 0x7fb024166840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:18.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.744+0000 7fb023e00640 1 -- 172.21.15.45:0/1301536851 >> 172.21.15.45:0/1301536851 conn(0x7fb02406b930 msgr2=0x7fb02406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:18.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.744+0000 7fb023e00640 1 -- 172.21.15.45:0/1301536851 shutdown_connections 2024-04-16T16:21:18.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.744+0000 7fb023e00640 1 -- 172.21.15.45:0/1301536851 wait complete. 2024-04-16T16:21:18.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.746+0000 7fb023e00640 1 Processor -- start 2024-04-16T16:21:18.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.746+0000 7fb023e00640 1 -- start start 2024-04-16T16:21:18.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.748+0000 7fb023e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0240b58f0 0x7fb0241466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:18.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.748+0000 7fb023e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0241491c0 0x7fb02413b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:18.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.748+0000 7fb023e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb024163cd0 0x7fb02415c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:18.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.748+0000 7fb023e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb0241676e0 con 0x7fb0240b58f0 2024-04-16T16:21:18.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.748+0000 7fb023e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb024167400 con 0x7fb024163cd0 2024-04-16T16:21:18.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.748+0000 7fb023e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb0240cd640 con 0x7fb0241491c0 2024-04-16T16:21:18.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.748+0000 7fb022a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0240b58f0 0x7fb0241466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:18.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.748+0000 7fb022a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0240b58f0 0x7fb0241466e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56228/0 (socket says 172.21.15.45:56228) 2024-04-16T16:21:18.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.748+0000 7fb022a00640 1 -- 172.21.15.45:0/2297999587 learned_addr learned my addr 172.21.15.45:0/2297999587 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:18.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.748+0000 7fb01be00640 1 --2- 172.21.15.45:0/2297999587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0241491c0 0x7fb02413b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:18.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.749+0000 7fb022a00640 1 -- 172.21.15.45:0/2297999587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0241491c0 msgr2=0x7fb02413b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:18.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.749+0000 7fb022a00640 1 --2- 172.21.15.45:0/2297999587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0241491c0 0x7fb02413b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:18.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.749+0000 7fb023400640 1 --2- 172.21.15.45:0/2297999587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb024163cd0 0x7fb02415c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:18.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.749+0000 7fb022a00640 1 -- 172.21.15.45:0/2297999587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb024163cd0 msgr2=0x7fb02415c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:18.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.749+0000 7fb022a00640 1 --2- 172.21.15.45:0/2297999587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb024163cd0 0x7fb02415c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:18.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.749+0000 7fb022a00640 1 -- 172.21.15.45:0/2297999587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb024146d70 con 0x7fb0240b58f0 2024-04-16T16:21:18.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.749+0000 7fb022a00640 1 --2- 172.21.15.45:0/2297999587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0240b58f0 0x7fb0241466e0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fb01400e9b0 tx=0x7fb01400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:18.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.749+0000 7fb01aa00640 1 -- 172.21.15.45:0/2297999587 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb014017070 con 0x7fb0240b58f0 2024-04-16T16:21:18.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.749+0000 7fb01aa00640 1 -- 172.21.15.45:0/2297999587 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb01400c9a0 con 0x7fb0240b58f0 2024-04-16T16:21:18.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.749+0000 7fb01aa00640 1 -- 172.21.15.45:0/2297999587 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01400cd90 con 0x7fb0240b58f0 2024-04-16T16:21:18.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.749+0000 7fb023e00640 1 -- 172.21.15.45:0/2297999587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb024147050 con 0x7fb0240b58f0 2024-04-16T16:21:18.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.750+0000 7fb023e00640 1 -- 172.21.15.45:0/2297999587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb02415da40 con 0x7fb0240b58f0 2024-04-16T16:21:18.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.751+0000 7fb023e00640 1 -- 172.21.15.45:0/2297999587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb024075780 con 0x7fb0240b58f0 2024-04-16T16:21:18.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.752+0000 7fb01aa00640 1 -- 172.21.15.45:0/2297999587 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7fb01401a070 con 0x7fb0240b58f0 2024-04-16T16:21:18.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.753+0000 7fb01aa00640 1 --2- 172.21.15.45:0/2297999587 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb00006ca10 0x7fb00006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:18.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.753+0000 7fb01aa00640 1 -- 172.21.15.45:0/2297999587 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5631+0+0 (secure 0 0 0) 0x7fb01408bcc0 con 0x7fb0240b58f0 2024-04-16T16:21:18.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.754+0000 7fb01be00640 1 --2- 172.21.15.45:0/2297999587 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb00006ca10 0x7fb00006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:18.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.755+0000 7fb01be00640 1 --2- 172.21.15.45:0/2297999587 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb00006ca10 0x7fb00006eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb00c036000 tx=0x7fb00c008030 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:18.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.758+0000 7fb01aa00640 1 -- 172.21.15.45:0/2297999587 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb014092080 con 0x7fb0240b58f0 2024-04-16T16:21:18.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.969+0000 7fb023e00640 1 -- 172.21.15.45:0/2297999587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fb0240b20d0 con 0x7fb0240b58f0 2024-04-16T16:21:18.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.975+0000 7fb01aa00640 1 -- 172.21.15.45:0/2297999587 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) ==== 122+0+0 (secure 0 0 0) 0x7fb01405a020 con 0x7fb0240b58f0 2024-04-16T16:21:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.978+0000 7fb023e00640 1 -- 172.21.15.45:0/2297999587 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb00006ca10 msgr2=0x7fb00006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.978+0000 7fb023e00640 1 --2- 172.21.15.45:0/2297999587 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb00006ca10 0x7fb00006eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb00c036000 tx=0x7fb00c008030 comp rx=0 tx=0).stop 2024-04-16T16:21:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.978+0000 7fb023e00640 1 -- 172.21.15.45:0/2297999587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0240b58f0 msgr2=0x7fb0241466e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.978+0000 7fb023e00640 1 --2- 172.21.15.45:0/2297999587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0240b58f0 0x7fb0241466e0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fb01400e9b0 tx=0x7fb01400ee70 comp rx=0 tx=0).stop 2024-04-16T16:21:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.978+0000 7fb023e00640 1 -- 172.21.15.45:0/2297999587 shutdown_connections 2024-04-16T16:21:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.978+0000 7fb023e00640 1 --2- 172.21.15.45:0/2297999587 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb00006ca10 0x7fb00006eed0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.978+0000 7fb023e00640 1 --2- 172.21.15.45:0/2297999587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb024163cd0 0x7fb02415c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.978+0000 7fb023e00640 1 --2- 172.21.15.45:0/2297999587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0241491c0 0x7fb02413b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.978+0000 7fb023e00640 1 --2- 172.21.15.45:0/2297999587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0240b58f0 0x7fb0241466e0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.978+0000 7fb023e00640 1 -- 172.21.15.45:0/2297999587 >> 172.21.15.45:0/2297999587 conn(0x7fb02406b930 msgr2=0x7fb0240cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.978+0000 7fb023e00640 1 -- 172.21.15.45:0/2297999587 shutdown_connections 2024-04-16T16:21:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:18.978+0000 7fb023e00640 1 -- 172.21.15.45:0/2297999587 wait complete. 2024-04-16T16:21:18.998 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-16T16:21:18.998 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:21:19.146 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:21:19.273 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-16T16:21:19.273 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:21:19.273 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:21:19.335 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:21:19.335 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:21:19.335 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:21:19.392 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:21:19.392 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:21:19.392 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:21:19.454 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:21:19.454 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:21:19.454 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:21:19.509 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:21:19.509 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-16T16:21:19.510 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:21:19.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.744+0000 7f24a7e00640 1 Processor -- start 2024-04-16T16:21:19.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.745+0000 7f24a7e00640 1 -- start start 2024-04-16T16:21:19.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.746+0000 7f24a7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f24a80ccb20 0x7f24a80cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:19.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.746+0000 7f24a7e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24a815ecd0 con 0x7f24a80ce9a0 2024-04-16T16:21:19.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.746+0000 7f24a7e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24a80cd6d0 con 0x7f24a815e8f0 2024-04-16T16:21:19.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.746+0000 7f24a7e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f24a80ce2e0 con 0x7f24a80ccb20 2024-04-16T16:21:19.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.747+0000 7f24a7400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f24a80ce9a0 0x7f24a80cb170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41374/0 (socket says 172.21.15.45:41374) 2024-04-16T16:21:19.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.747+0000 7f24a7400640 1 -- 172.21.15.45:0/1074716651 learned_addr learned my addr 172.21.15.45:0/1074716651 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:19.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.747+0000 7f24a6000640 1 --2- 172.21.15.45:0/1074716651 >> v2:172.21.15.103:3300/0 conn(0x7f24a80ccb20 0x7f24a80cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:19.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.747+0000 7f24a5600640 1 -- 172.21.15.45:0/1074716651 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1148962963 0 0) 0x7f24a815ecd0 con 0x7f24a80ce9a0 2024-04-16T16:21:19.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.747+0000 7f24a5600640 1 -- 172.21.15.45:0/1074716651 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2488003520 con 0x7f24a80ce9a0 2024-04-16T16:21:19.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.748+0000 7f24a5600640 1 -- 172.21.15.45:0/1074716651 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 497309867 0 0) 0x7f2488003520 con 0x7f24a80ce9a0 2024-04-16T16:21:19.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.748+0000 7f24a5600640 1 -- 172.21.15.45:0/1074716651 >> v2:172.21.15.103:3300/0 conn(0x7f24a80ccb20 msgr2=0x7f24a80cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:19.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.748+0000 7f24a5600640 1 --2- 172.21.15.45:0/1074716651 >> v2:172.21.15.103:3300/0 conn(0x7f24a80ccb20 0x7f24a80cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:19.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.748+0000 7f24a5600640 1 -- 172.21.15.45:0/1074716651 >> v1:172.21.15.103:6789/0 conn(0x7f24a815e8f0 legacy=0x7f24a80cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:19.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.748+0000 7f24a5600640 1 -- 172.21.15.45:0/1074716651 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a80cd9b0 con 0x7f24a80ce9a0 2024-04-16T16:21:19.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.748+0000 7f24a5600640 1 -- 172.21.15.45:0/1074716651 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f24a0001670 con 0x7f24a80ce9a0 2024-04-16T16:21:19.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.748+0000 7f24a5600640 1 -- 172.21.15.45:0/1074716651 >> v1:172.21.15.45:6789/0 conn(0x7f24a80ce9a0 legacy=0x7f24a80cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:19.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.748+0000 7f24a5600640 1 --2- 172.21.15.45:0/1074716651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2488003a00 0x7f2488003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:19.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.748+0000 7f24a5600640 1 --2- 172.21.15.45:0/1074716651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2488004800 0x7f2488004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:19.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.748+0000 7f24a5600640 1 --2- 172.21.15.45:0/1074716651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2488005210 0x7f248800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:19.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.748+0000 7f24a5600640 1 -- 172.21.15.45:0/1074716651 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f24a80ce2e0 con 0x7f2488004800 2024-04-16T16:21:19.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.748+0000 7f24a5600640 1 -- 172.21.15.45:0/1074716651 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f248800c160 con 0x7f2488005210 2024-04-16T16:21:19.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.748+0000 7f24a5600640 1 -- 172.21.15.45:0/1074716651 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f248800c360 con 0x7f2488003a00 2024-04-16T16:21:19.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.748+0000 7f24a6000640 1 --2- 172.21.15.45:0/1074716651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2488004800 0x7f2488004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:19.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.749+0000 7f24a6a00640 1 --2- 172.21.15.45:0/1074716651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2488003a00 0x7f2488003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:19.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.749+0000 7f24a7400640 1 --2- 172.21.15.45:0/1074716651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2488005210 0x7f248800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:19.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.749+0000 7f24a6000640 1 -- 172.21.15.45:0/1074716651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2488003a00 msgr2=0x7f2488003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:19.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.749+0000 7f24a6000640 1 --2- 172.21.15.45:0/1074716651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2488003a00 0x7f2488003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:19.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.749+0000 7f24a6000640 1 -- 172.21.15.45:0/1074716651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2488005210 msgr2=0x7f248800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:19.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.749+0000 7f24a6000640 1 --2- 172.21.15.45:0/1074716651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2488005210 0x7f248800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:19.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.749+0000 7f24a6000640 1 -- 172.21.15.45:0/1074716651 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2488003520 con 0x7f2488004800 2024-04-16T16:21:19.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.749+0000 7f24a7400640 1 --2- 172.21.15.45:0/1074716651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2488005210 0x7f248800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:19.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.749+0000 7f24a7400640 1 -- 172.21.15.45:0/1074716651 reap_dead start 2024-04-16T16:21:19.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.750+0000 7f24a6000640 1 --2- 172.21.15.45:0/1074716651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2488004800 0x7f2488004c80 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f2490009f90 tx=0x7f2488005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3d55302a543c48 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:19.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.750+0000 7f24a5600640 1 -- 172.21.15.45:0/1074716651 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2490039070 con 0x7f2488004800 2024-04-16T16:21:19.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.750+0000 7f24a5600640 1 -- 172.21.15.45:0/1074716651 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f249002cbd0 con 0x7f2488004800 2024-04-16T16:21:19.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.750+0000 7f24a5600640 1 -- 172.21.15.45:0/1074716651 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2490009c10 con 0x7f2488004800 2024-04-16T16:21:19.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.751+0000 7f24a7e00640 1 -- 172.21.15.45:0/1074716651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2488004800 msgr2=0x7f2488004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:19.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.751+0000 7f24a7e00640 1 --2- 172.21.15.45:0/1074716651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2488004800 0x7f2488004c80 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f2490009f90 tx=0x7f2488005830 comp rx=0 tx=0).stop 2024-04-16T16:21:19.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.751+0000 7f24a7e00640 1 -- 172.21.15.45:0/1074716651 shutdown_connections 2024-04-16T16:21:19.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.751+0000 7f24a7e00640 1 --2- 172.21.15.45:0/1074716651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2488004800 0x7f2488004c80 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:19.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.751+0000 7f24a7e00640 1 -- 172.21.15.45:0/1074716651 >> 172.21.15.45:0/1074716651 conn(0x7f24a806b930 msgr2=0x7f24a806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:19.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.751+0000 7f24a7e00640 1 -- 172.21.15.45:0/1074716651 shutdown_connections 2024-04-16T16:21:19.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.751+0000 7f24a7e00640 1 -- 172.21.15.45:0/1074716651 wait complete. 2024-04-16T16:21:19.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.753+0000 7f24a7e00640 1 Processor -- start 2024-04-16T16:21:19.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.753+0000 7f24a7e00640 1 -- start start 2024-04-16T16:21:19.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.754+0000 7f24a7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2488004800 0x7f24a8145010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:19.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.754+0000 7f24a7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24a815f160 0x7f24a815f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:19.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.754+0000 7f24a7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24a8152a50 0x7f24a8152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:19.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.754+0000 7f24a7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f24a80cd360 con 0x7f24a815f160 2024-04-16T16:21:19.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.754+0000 7f24a7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24a80cdf70 con 0x7f2488004800 2024-04-16T16:21:19.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.754+0000 7f24a7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f24a80cf080 con 0x7f24a8152a50 2024-04-16T16:21:19.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.754+0000 7f24a6000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24a815f160 0x7f24a815f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:19.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.754+0000 7f24a6000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24a815f160 0x7f24a815f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56246/0 (socket says 172.21.15.45:56246) 2024-04-16T16:21:19.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.754+0000 7f24a6000640 1 -- 172.21.15.45:0/3846505836 learned_addr learned my addr 172.21.15.45:0/3846505836 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:19.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.755+0000 7f24a6a00640 1 --2- 172.21.15.45:0/3846505836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2488004800 0x7f24a8145010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:19.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.755+0000 7f24a7400640 1 --2- 172.21.15.45:0/3846505836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24a8152a50 0x7f24a8152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.755+0000 7f24a6000640 1 -- 172.21.15.45:0/3846505836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24a8152a50 msgr2=0x7f24a8152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.755+0000 7f24a6000640 1 --2- 172.21.15.45:0/3846505836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24a8152a50 0x7f24a8152ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.755+0000 7f24a6000640 1 -- 172.21.15.45:0/3846505836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2488004800 msgr2=0x7f24a8145010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.755+0000 7f24a6000640 1 --2- 172.21.15.45:0/3846505836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2488004800 0x7f24a8145010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.755+0000 7f24a6000640 1 -- 172.21.15.45:0/3846505836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a8153410 con 0x7f24a815f160 2024-04-16T16:21:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.755+0000 7f24a7400640 1 --2- 172.21.15.45:0/3846505836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24a8152a50 0x7f24a8152ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.755+0000 7f24a6000640 1 --2- 172.21.15.45:0/3846505836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24a815f160 0x7f24a815f5e0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f249002c640 tx=0x7f2490009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:19.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.756+0000 7f249ea00640 1 -- 172.21.15.45:0/3846505836 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2490042020 con 0x7f24a815f160 2024-04-16T16:21:19.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.756+0000 7f249ea00640 1 -- 172.21.15.45:0/3846505836 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f249003a040 con 0x7f24a815f160 2024-04-16T16:21:19.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.756+0000 7f249ea00640 1 -- 172.21.15.45:0/3846505836 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2490039070 con 0x7f24a815f160 2024-04-16T16:21:19.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.756+0000 7f24a7e00640 1 -- 172.21.15.45:0/3846505836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24a815fbe0 con 0x7f24a815f160 2024-04-16T16:21:19.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.756+0000 7f24a7e00640 1 -- 172.21.15.45:0/3846505836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24a81601a0 con 0x7f24a815f160 2024-04-16T16:21:19.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.757+0000 7f24a7e00640 1 -- 172.21.15.45:0/3846505836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24a8075780 con 0x7f24a815f160 2024-04-16T16:21:19.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.758+0000 7f249ea00640 1 -- 172.21.15.45:0/3846505836 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f2490045020 con 0x7f24a815f160 2024-04-16T16:21:19.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.759+0000 7f249ea00640 1 --2- 172.21.15.45:0/3846505836 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f248406ca40 0x7f248406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:19.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.760+0000 7f249ea00640 1 -- 172.21.15.45:0/3846505836 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5631+0+0 (secure 0 0 0) 0x7f24900ae2c0 con 0x7f24a815f160 2024-04-16T16:21:19.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.760+0000 7f24a6a00640 1 --2- 172.21.15.45:0/3846505836 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f248406ca40 0x7f248406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:19.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.762+0000 7f24a6a00640 1 --2- 172.21.15.45:0/3846505836 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f248406ca40 0x7f248406ef00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2498007fd0 tx=0x7f2498009040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:19.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:19.764+0000 7f249ea00640 1 -- 172.21.15.45:0/3846505836 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f249004c020 con 0x7f24a815f160 2024-04-16T16:21:20.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.006+0000 7f24a7e00640 1 -- 172.21.15.45:0/3846505836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24a80b5b90 con 0x7f24a815f160 2024-04-16T16:21:20.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.008+0000 7f249ea00640 1 -- 172.21.15.45:0/3846505836 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) ==== 94+0+3872 (secure 0 0 0) 0x7f249007c590 con 0x7f24a815f160 2024-04-16T16:21:20.008 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:20.008 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":93,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/929218719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":929218719},{"type":"v1","addr":"172.21.15.103:6835","nonce":929218719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6323,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1619719405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1619719405},{"type":"v1","addr":"172.21.15.103:6837","nonce":1619719405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":8640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/3613288118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3613288118},{"type":"v1","addr":"172.21.15.45:6834","nonce":3613288118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":9084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3426596276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3426596276},{"type":"v1","addr":"172.21.15.45:6835","nonce":3426596276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[]} 2024-04-16T16:21:20.008 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 93 2024-04-16T16:21:20.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.010+0000 7f24a7e00640 1 -- 172.21.15.45:0/3846505836 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f248406ca40 msgr2=0x7f248406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:20.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.010+0000 7f24a7e00640 1 --2- 172.21.15.45:0/3846505836 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f248406ca40 0x7f248406ef00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2498007fd0 tx=0x7f2498009040 comp rx=0 tx=0).stop 2024-04-16T16:21:20.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.010+0000 7f24a7e00640 1 -- 172.21.15.45:0/3846505836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24a815f160 msgr2=0x7f24a815f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:20.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.010+0000 7f24a7e00640 1 --2- 172.21.15.45:0/3846505836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24a815f160 0x7f24a815f5e0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f249002c640 tx=0x7f2490009750 comp rx=0 tx=0).stop 2024-04-16T16:21:20.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.010+0000 7f24a7e00640 1 -- 172.21.15.45:0/3846505836 shutdown_connections 2024-04-16T16:21:20.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.010+0000 7f24a7e00640 1 --2- 172.21.15.45:0/3846505836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24a8152a50 0x7f24a8152ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:20.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.010+0000 7f24a7e00640 1 --2- 172.21.15.45:0/3846505836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24a815f160 0x7f24a815f5e0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:20.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.010+0000 7f24a7e00640 1 --2- 172.21.15.45:0/3846505836 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f248406ca40 0x7f248406ef00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:20.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.010+0000 7f24a7e00640 1 --2- 172.21.15.45:0/3846505836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2488004800 0x7f24a8145010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:20.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.010+0000 7f24a7e00640 1 -- 172.21.15.45:0/3846505836 >> 172.21.15.45:0/3846505836 conn(0x7f24a806b930 msgr2=0x7f24a806ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:20.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.011+0000 7f24a7e00640 1 -- 172.21.15.45:0/3846505836 shutdown_connections 2024-04-16T16:21:20.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.011+0000 7f24a7e00640 1 -- 172.21.15.45:0/3846505836 wait complete. 2024-04-16T16:21:20.030 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-16T16:21:20.031 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-16T16:21:20.031 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-16T16:21:20.031 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:21:20.031 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-16T16:21:20.031 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-16T16:21:20.031 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-16T16:21:20.031 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:21:20.032 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-16T16:21:20.032 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-16T16:21:20.032 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-16T16:21:20.032 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:21:20.032 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-16T16:21:20.032 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-16T16:21:20.032 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-16T16:21:20.032 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:21:20.048 INFO:tasks.ceph.mds.c:Stopped 2024-04-16T16:21:20.048 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-16T16:21:20.049 INFO:tasks.ceph.mds.a:Stopped 2024-04-16T16:21:20.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-16T16:21:20.052 INFO:tasks.ceph.mds.b:Stopped 2024-04-16T16:21:20.052 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-16T16:21:20.058 INFO:tasks.ceph.mds.d:Stopped 2024-04-16T16:21:20.058 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-16T16:21:20.096 INFO:tasks.ceph.mds.c:Started 2024-04-16T16:21:20.096 DEBUG:teuthology.parallel:result is None 2024-04-16T16:21:20.101 INFO:tasks.ceph.mds.a:Started 2024-04-16T16:21:20.101 DEBUG:teuthology.parallel:result is None 2024-04-16T16:21:20.105 INFO:tasks.ceph.mds.b:Started 2024-04-16T16:21:20.105 DEBUG:teuthology.parallel:result is None 2024-04-16T16:21:20.111 INFO:tasks.ceph.mds.d:Started 2024-04-16T16:21:20.112 DEBUG:teuthology.parallel:result is None 2024-04-16T16:21:20.112 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-16T16:21:20.259 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2024-04-16T16:21:20.259 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2024-04-16T16:21:20.263 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-16T16:21:20.271 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-16T16:21:20.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.301+0000 7f85e7000640 1 Processor -- start 2024-04-16T16:21:20.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.301+0000 7f85e7000640 1 -- start start 2024-04-16T16:21:20.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.302+0000 7f85e7000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f85e015ebd0 0x7f85e009bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:20.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.302+0000 7f85e7000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f85e00b7510 0x7f85e009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:20.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.302+0000 7f85e7000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85e00b6470 con 0x7f85e00b79a0 2024-04-16T16:21:20.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.302+0000 7f85e7000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f85e00b6240 con 0x7f85e00b7510 2024-04-16T16:21:20.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.302+0000 7f85e7000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f85e015efb0 con 0x7f85e015ebd0 2024-04-16T16:21:20.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.302+0000 7f85d7400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f85e00b7510 0x7f85e009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:20.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.302+0000 7f85d7400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f85e00b7510 0x7f85e009ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56268/0 (socket says 172.21.15.45:56268) 2024-04-16T16:21:20.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.302+0000 7f85d7400640 1 -- 172.21.15.45:0/1120043011 learned_addr learned my addr 172.21.15.45:0/1120043011 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:20.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.303+0000 7f85dfe00640 1 --2- 172.21.15.45:0/1120043011 >> v2:172.21.15.103:3300/0 conn(0x7f85e015ebd0 0x7f85e009bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:20.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.303+0000 7f85d7400640 1 -- 172.21.15.45:0/1120043011 >> v2:172.21.15.103:3300/0 conn(0x7f85e015ebd0 msgr2=0x7f85e009bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:20.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.303+0000 7f85d7400640 1 --2- 172.21.15.45:0/1120043011 >> v2:172.21.15.103:3300/0 conn(0x7f85e015ebd0 0x7f85e009bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:20.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.303+0000 7f85d7400640 1 -- 172.21.15.45:0/1120043011 >> v1:172.21.15.103:6789/0 conn(0x7f85e00b79a0 legacy=0x7f85e00af710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:20.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.303+0000 7f85d7400640 1 -- 172.21.15.45:0/1120043011 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e00b6750 con 0x7f85e00b7510 2024-04-16T16:21:20.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.304+0000 7f85d7400640 1 --2- 172.21.15.45:0/1120043011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e00b7510 0x7f85e009ac30 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f85e00b5540 tx=0x7f85c802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ec7fddf9cd5474 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:20.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.304+0000 7f85df400640 1 -- 172.21.15.45:0/1120043011 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c8034070 con 0x7f85e00b7510 2024-04-16T16:21:20.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.304+0000 7f85df400640 1 -- 172.21.15.45:0/1120043011 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85c8030860 con 0x7f85e00b7510 2024-04-16T16:21:20.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.304+0000 7f85df400640 1 -- 172.21.15.45:0/1120043011 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c8030c30 con 0x7f85e00b7510 2024-04-16T16:21:20.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.305+0000 7f85e7000640 1 -- 172.21.15.45:0/1120043011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e00b7510 msgr2=0x7f85e009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:20.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.305+0000 7f85e7000640 1 --2- 172.21.15.45:0/1120043011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e00b7510 0x7f85e009ac30 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f85e00b5540 tx=0x7f85c802b9a0 comp rx=0 tx=0).stop 2024-04-16T16:21:20.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.305+0000 7f85e7000640 1 -- 172.21.15.45:0/1120043011 shutdown_connections 2024-04-16T16:21:20.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.305+0000 7f85e7000640 1 --2- 172.21.15.45:0/1120043011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e00b7510 0x7f85e009ac30 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:20.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.305+0000 7f85e7000640 1 --2- 172.21.15.45:0/1120043011 >> v2:172.21.15.103:3300/0 conn(0x7f85e015ebd0 0x7f85e009bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:20.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.305+0000 7f85e7000640 1 -- 172.21.15.45:0/1120043011 >> 172.21.15.45:0/1120043011 conn(0x7f85e00b7ee0 msgr2=0x7f85e00b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:20.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.305+0000 7f85e7000640 1 -- 172.21.15.45:0/1120043011 shutdown_connections 2024-04-16T16:21:20.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.305+0000 7f85e7000640 1 -- 172.21.15.45:0/1120043011 wait complete. 2024-04-16T16:21:20.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.307+0000 7f85e7000640 1 Processor -- start 2024-04-16T16:21:20.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.307+0000 7f85e7000640 1 -- start start 2024-04-16T16:21:20.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.308+0000 7f85e7000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85e00b7510 0x7f85e0145320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:20.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.308+0000 7f85e7000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e00b79a0 0x7f85e0141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:20.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.308+0000 7f85e7000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e015ebd0 0x7f85e0141f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:20.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.308+0000 7f85e7000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f85e00b6ff0 con 0x7f85e00b79a0 2024-04-16T16:21:20.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.308+0000 7f85e7000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f85e00b6d10 con 0x7f85e015ebd0 2024-04-16T16:21:20.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.308+0000 7f85e7000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f85e009c210 con 0x7f85e00b7510 2024-04-16T16:21:20.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.309+0000 7f85d7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e00b79a0 0x7f85e0141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:20.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.309+0000 7f85dfe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85e00b7510 0x7f85e0145320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:20.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.309+0000 7f85e5000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e015ebd0 0x7f85e0141f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:20.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.309+0000 7f85dfe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85e00b7510 0x7f85e0145320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:46518/0 (socket says 172.21.15.45:46518) 2024-04-16T16:21:20.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.309+0000 7f85dfe00640 1 -- 172.21.15.45:0/3479944941 learned_addr learned my addr 172.21.15.45:0/3479944941 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.309+0000 7f85e5000640 1 -- 172.21.15.45:0/3479944941 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85e00b7510 msgr2=0x7f85e0145320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.309+0000 7f85e5000640 1 --2- 172.21.15.45:0/3479944941 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85e00b7510 0x7f85e0145320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.309+0000 7f85e5000640 1 -- 172.21.15.45:0/3479944941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e00b79a0 msgr2=0x7f85e0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.309+0000 7f85e5000640 1 --2- 172.21.15.45:0/3479944941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e00b79a0 0x7f85e0141a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.309+0000 7f85e5000640 1 -- 172.21.15.45:0/3479944941 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e0160b60 con 0x7f85e015ebd0 2024-04-16T16:21:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.309+0000 7f85e5000640 1 --2- 172.21.15.45:0/3479944941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e015ebd0 0x7f85e0141f60 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f85d800c730 tx=0x7f85d800cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.310+0000 7f85dcc00640 1 -- 172.21.15.45:0/3479944941 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d8016020 con 0x7f85e015ebd0 2024-04-16T16:21:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.310+0000 7f85dcc00640 1 -- 172.21.15.45:0/3479944941 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85d8006d50 con 0x7f85e015ebd0 2024-04-16T16:21:20.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.310+0000 7f85e7000640 1 -- 172.21.15.45:0/3479944941 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85e0160e40 con 0x7f85e015ebd0 2024-04-16T16:21:20.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.310+0000 7f85dcc00640 1 -- 172.21.15.45:0/3479944941 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d8003bb0 con 0x7f85e015ebd0 2024-04-16T16:21:20.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.310+0000 7f85e7000640 1 -- 172.21.15.45:0/3479944941 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85e0154370 con 0x7f85e015ebd0 2024-04-16T16:21:20.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.311+0000 7f85e7000640 1 -- 172.21.15.45:0/3479944941 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85e0154570 con 0x7f85e015ebd0 2024-04-16T16:21:20.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.312+0000 7f85dcc00640 1 -- 172.21.15.45:0/3479944941 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f85d80050d0 con 0x7f85e015ebd0 2024-04-16T16:21:20.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.313+0000 7f85dcc00640 1 --2- 172.21.15.45:0/3479944941 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f85bc06ca10 0x7f85bc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:20.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.313+0000 7f85dcc00640 1 -- 172.21.15.45:0/3479944941 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5631+0+0 (secure 0 0 0) 0x7f85d808b6e0 con 0x7f85e015ebd0 2024-04-16T16:21:20.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.313+0000 7f85dfe00640 1 --2- 172.21.15.45:0/3479944941 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f85bc06ca10 0x7f85bc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:20.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.314+0000 7f85dfe00640 1 --2- 172.21.15.45:0/3479944941 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f85bc06ca10 0x7f85bc06eed0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f85d0009b60 tx=0x7f85d0009190 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:20.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.321+0000 7f85dcc00640 1 -- 172.21.15.45:0/3479944941 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f85e0154570 con 0x7f85e015ebd0 2024-04-16T16:21:20.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:20.543+0000 7f85e7000640 1 -- 172.21.15.45:0/3479944941 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f85e0075780 con 0x7f85e015ebd0 2024-04-16T16:21:21.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.544+0000 7f85dcc00640 1 -- 172.21.15.45:0/3479944941 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v138) ==== 97+0+0 (secure 0 0 0) 0x7f85d8020020 con 0x7f85e015ebd0 2024-04-16T16:21:21.543 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2024-04-16T16:21:21.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.546+0000 7f85e7000640 1 -- 172.21.15.45:0/3479944941 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f85bc06ca10 msgr2=0x7f85bc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:21.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.546+0000 7f85e7000640 1 --2- 172.21.15.45:0/3479944941 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f85bc06ca10 0x7f85bc06eed0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f85d0009b60 tx=0x7f85d0009190 comp rx=0 tx=0).stop 2024-04-16T16:21:21.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.547+0000 7f85e7000640 1 -- 172.21.15.45:0/3479944941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e015ebd0 msgr2=0x7f85e0141f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:21.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.547+0000 7f85e7000640 1 --2- 172.21.15.45:0/3479944941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e015ebd0 0x7f85e0141f60 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f85d800c730 tx=0x7f85d800cbf0 comp rx=0 tx=0).stop 2024-04-16T16:21:21.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.547+0000 7f85e7000640 1 -- 172.21.15.45:0/3479944941 shutdown_connections 2024-04-16T16:21:21.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.547+0000 7f85e7000640 1 --2- 172.21.15.45:0/3479944941 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f85bc06ca10 0x7f85bc06eed0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:21.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.547+0000 7f85e7000640 1 --2- 172.21.15.45:0/3479944941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e015ebd0 0x7f85e0141f60 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:21.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.547+0000 7f85e7000640 1 --2- 172.21.15.45:0/3479944941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e00b79a0 0x7f85e0141a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:21.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.547+0000 7f85e7000640 1 --2- 172.21.15.45:0/3479944941 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85e00b7510 0x7f85e0145320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:21.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.547+0000 7f85e7000640 1 -- 172.21.15.45:0/3479944941 >> 172.21.15.45:0/3479944941 conn(0x7f85e00b7ee0 msgr2=0x7f85e0070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:21.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.547+0000 7f85e7000640 1 -- 172.21.15.45:0/3479944941 shutdown_connections 2024-04-16T16:21:21.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.547+0000 7f85e7000640 1 -- 172.21.15.45:0/3479944941 wait complete. 2024-04-16T16:21:21.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-16T16:21:21.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.780+0000 7f8135000640 1 Processor -- start 2024-04-16T16:21:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.781+0000 7f8135000640 1 -- start start 2024-04-16T16:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.782+0000 7f8135000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81300d1070 0x7f8130071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.782+0000 7f8135000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f81300b1b00 0x7f813009aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.782+0000 7f8135000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8130071f10 con 0x7f813015fea0 2024-04-16T16:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.782+0000 7f8135000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8130160280 con 0x7f81300b1b00 2024-04-16T16:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.782+0000 7f8135000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8130071ce0 con 0x7f81300d1070 2024-04-16T16:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.782+0000 7f812f400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f813015fea0 0x7f8130160d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41404/0 (socket says 172.21.15.45:41404) 2024-04-16T16:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.782+0000 7f812f400640 1 -- 172.21.15.45:0/307821582 learned_addr learned my addr 172.21.15.45:0/307821582 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.782+0000 7f812fe00640 1 --2- 172.21.15.45:0/307821582 >> v2:172.21.15.45:3300/0 conn(0x7f81300b1b00 0x7f813009aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:21.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 -- 172.21.15.45:0/307821582 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2960904135 0 0) 0x7f8130071f10 con 0x7f813015fea0 2024-04-16T16:21:21.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812ea00640 1 --2- 172.21.15.45:0/307821582 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81300d1070 0x7f8130071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:21.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 -- 172.21.15.45:0/307821582 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8114003520 con 0x7f813015fea0 2024-04-16T16:21:21.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 -- 172.21.15.45:0/307821582 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3088891579 0 0) 0x7f8114003520 con 0x7f813015fea0 2024-04-16T16:21:21.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 -- 172.21.15.45:0/307821582 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81300d1070 msgr2=0x7f8130071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:21.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 --2- 172.21.15.45:0/307821582 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81300d1070 0x7f8130071240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:21.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 -- 172.21.15.45:0/307821582 >> v2:172.21.15.45:3300/0 conn(0x7f81300b1b00 msgr2=0x7f813009aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 --2- 172.21.15.45:0/307821582 >> v2:172.21.15.45:3300/0 conn(0x7f81300b1b00 0x7f813009aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 -- 172.21.15.45:0/307821582 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8130160520 con 0x7f813015fea0 2024-04-16T16:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 -- 172.21.15.45:0/307821582 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f81200019f0 con 0x7f813015fea0 2024-04-16T16:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 -- 172.21.15.45:0/307821582 >> v1:172.21.15.45:6789/0 conn(0x7f813015fea0 legacy=0x7f8130160d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 --2- 172.21.15.45:0/307821582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81140039b0 0x7f8114003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 --2- 172.21.15.45:0/307821582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8114004800 0x7f8114004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 --2- 172.21.15.45:0/307821582 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81140051c0 0x7f811400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 -- 172.21.15.45:0/307821582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f811400bf00 con 0x7f8114004800 2024-04-16T16:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 -- 172.21.15.45:0/307821582 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f811400c080 con 0x7f81140039b0 2024-04-16T16:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.783+0000 7f812e000640 1 -- 172.21.15.45:0/307821582 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f811400c200 con 0x7f81140051c0 2024-04-16T16:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.784+0000 7f812ea00640 1 --2- 172.21.15.45:0/307821582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8114004800 0x7f8114004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.784+0000 7f812f400640 1 --2- 172.21.15.45:0/307821582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81140039b0 0x7f8114003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.784+0000 7f812fe00640 1 --2- 172.21.15.45:0/307821582 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81140051c0 0x7f811400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.784+0000 7f812ea00640 1 -- 172.21.15.45:0/307821582 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81140051c0 msgr2=0x7f811400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.784+0000 7f812ea00640 1 --2- 172.21.15.45:0/307821582 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81140051c0 0x7f811400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:21.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.784+0000 7f812ea00640 1 -- 172.21.15.45:0/307821582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81140039b0 msgr2=0x7f8114003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:21.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.784+0000 7f812ea00640 1 --2- 172.21.15.45:0/307821582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81140039b0 0x7f8114003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:21.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.784+0000 7f812ea00640 1 -- 172.21.15.45:0/307821582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8114003520 con 0x7f8114004800 2024-04-16T16:21:21.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.784+0000 7f812f400640 1 --2- 172.21.15.45:0/307821582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81140039b0 0x7f8114003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:21.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.785+0000 7f812ea00640 1 --2- 172.21.15.45:0/307821582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8114004800 0x7f8114004c80 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f8118009ac0 tx=0x7f8114005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec40dbf97b9e66 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:21.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.786+0000 7f812e000640 1 -- 172.21.15.45:0/307821582 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8118039070 con 0x7f8114004800 2024-04-16T16:21:21.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.786+0000 7f812e000640 1 -- 172.21.15.45:0/307821582 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f811802c680 con 0x7f8114004800 2024-04-16T16:21:21.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.786+0000 7f812e000640 1 -- 172.21.15.45:0/307821582 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8118009690 con 0x7f8114004800 2024-04-16T16:21:21.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.786+0000 7f8135000640 1 -- 172.21.15.45:0/307821582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8114004800 msgr2=0x7f8114004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:21.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.786+0000 7f8135000640 1 --2- 172.21.15.45:0/307821582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8114004800 0x7f8114004c80 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f8118009ac0 tx=0x7f8114005830 comp rx=0 tx=0).stop 2024-04-16T16:21:21.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.786+0000 7f812fe00640 1 -- 172.21.15.45:0/307821582 reap_dead start 2024-04-16T16:21:21.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.786+0000 7f8135000640 1 -- 172.21.15.45:0/307821582 shutdown_connections 2024-04-16T16:21:21.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.786+0000 7f8135000640 1 -- 172.21.15.45:0/307821582 >> 172.21.15.45:0/307821582 conn(0x7f81300d2610 msgr2=0x7f81300d2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:21.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.786+0000 7f8135000640 1 -- 172.21.15.45:0/307821582 shutdown_connections 2024-04-16T16:21:21.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.786+0000 7f8135000640 1 -- 172.21.15.45:0/307821582 wait complete. 2024-04-16T16:21:21.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.788+0000 7f8135000640 1 Processor -- start 2024-04-16T16:21:21.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.788+0000 7f8135000640 1 -- start start 2024-04-16T16:21:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.789+0000 7f8135000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8114004800 0x7f81301f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.789+0000 7f8135000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81300af720 0x7f81300b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.789+0000 7f8135000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81300afba0 0x7f81300b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.789+0000 7f8135000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8130071240 con 0x7f81300afba0 2024-04-16T16:21:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.789+0000 7f8135000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f81300721f0 con 0x7f81300af720 2024-04-16T16:21:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.789+0000 7f8135000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f81300713c0 con 0x7f8114004800 2024-04-16T16:21:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.790+0000 7f812fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81300afba0 0x7f81300b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.790+0000 7f812fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81300afba0 0x7f81300b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56300/0 (socket says 172.21.15.45:56300) 2024-04-16T16:21:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.790+0000 7f812fe00640 1 -- 172.21.15.45:0/2436286261 learned_addr learned my addr 172.21.15.45:0/2436286261 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.790+0000 7f812ea00640 1 --2- 172.21.15.45:0/2436286261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81300af720 0x7f81300b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.790+0000 7f812fe00640 1 -- 172.21.15.45:0/2436286261 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8114004800 msgr2=0x7f81301f3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.790+0000 7f812fe00640 1 --2- 172.21.15.45:0/2436286261 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8114004800 0x7f81301f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.790+0000 7f812fe00640 1 -- 172.21.15.45:0/2436286261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81300af720 msgr2=0x7f81300b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.790+0000 7f812fe00640 1 --2- 172.21.15.45:0/2436286261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81300af720 0x7f81300b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.790+0000 7f812fe00640 1 -- 172.21.15.45:0/2436286261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81300b0160 con 0x7f81300afba0 2024-04-16T16:21:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.790+0000 7f812fe00640 1 --2- 172.21.15.45:0/2436286261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81300afba0 0x7f81300b7be0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f81300c6110 tx=0x7f8128011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.791+0000 7f8127400640 1 -- 172.21.15.45:0/2436286261 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8128003610 con 0x7f81300afba0 2024-04-16T16:21:21.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.791+0000 7f8127400640 1 -- 172.21.15.45:0/2436286261 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f812801b070 con 0x7f81300afba0 2024-04-16T16:21:21.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.791+0000 7f8127400640 1 -- 172.21.15.45:0/2436286261 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8128003a30 con 0x7f81300afba0 2024-04-16T16:21:21.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.791+0000 7f8135000640 1 -- 172.21.15.45:0/2436286261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81300b0440 con 0x7f81300afba0 2024-04-16T16:21:21.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.791+0000 7f8135000640 1 -- 172.21.15.45:0/2436286261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81300b0a00 con 0x7f81300afba0 2024-04-16T16:21:21.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.792+0000 7f8135000640 1 -- 172.21.15.45:0/2436286261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8130075780 con 0x7f81300afba0 2024-04-16T16:21:21.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.793+0000 7f8127400640 1 -- 172.21.15.45:0/2436286261 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f8128003bd0 con 0x7f81300afba0 2024-04-16T16:21:21.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.794+0000 7f8127400640 1 --2- 172.21.15.45:0/2436286261 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f810c06ca40 0x7f810c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:21.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.794+0000 7f8127400640 1 -- 172.21.15.45:0/2436286261 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4857+0+0 (secure 0 0 0) 0x7f812808bab0 con 0x7f81300afba0 2024-04-16T16:21:21.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.795+0000 7f812f400640 1 --2- 172.21.15.45:0/2436286261 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f810c06ca40 0x7f810c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:21.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.796+0000 7f812f400640 1 --2- 172.21.15.45:0/2436286261 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f810c06ca40 0x7f810c06ef00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f81300af540 tx=0x7f8120009040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:21.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:21.799+0000 7f8127400640 1 -- 172.21.15.45:0/2436286261 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8128025020 con 0x7f81300afba0 2024-04-16T16:21:22.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.079+0000 7f8135000640 1 -- 172.21.15.45:0/2436286261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f81300d3520 con 0x7f81300afba0 2024-04-16T16:21:22.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.080+0000 7f8127400640 1 -- 172.21.15.45:0/2436286261 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v25) ==== 80+0+5238 (secure 0 0 0) 0x7f812801d310 con 0x7f81300afba0 2024-04-16T16:21:22.080 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:22.081 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:21:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2024-04-16T16:21:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2024-04-16T16:21:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mpjReORAAJMoNf4qvfuxPSiV9+VlKfg==", 2024-04-16T16:21:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:21:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:21:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:21:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:21:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2024-04-16T16:21:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mxHeBIBAA3G8maaK65zaOaqXpAOA6xA==", 2024-04-16T16:21:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:21:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:21:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:21:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:21:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2024-04-16T16:21:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5m+AaODxAA08XSjc44/i2KYExYQSukaQ==", 2024-04-16T16:21:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:21:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mEQ4UMRAAOslSsxQ17bOxPrI6QMlRRw==", 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBuox5m3l3OHxAA+UjwZx8Mn6tCu95TTYIG3w==", 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxox5mozt1LhAAJL5NtMQdPVBtKOaVlMQBtw==", 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.085 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.085 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2024-04-16T16:21:22.085 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB1ox5mfE7kABAAIvgZ5Wx+eIJ6QNKj2GVdpw==", 2024-04-16T16:21:22.085 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.085 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:21:22.085 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:21:22.085 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:21:22.085 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.085 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.085 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.085 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2024-04-16T16:21:22.085 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB4ox5mEvxcDRAAO0yXBKvsC7hZ6bpOAJyuag==", 2024-04-16T16:21:22.085 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.085 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:21:22.085 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:21:22.086 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:21:22.086 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.086 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.086 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.086 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2024-04-16T16:21:22.086 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+ox5mZubrORAAcGJLyRN5Ob9aBnopumnEUg==", 2024-04-16T16:21:22.086 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.086 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:21:22.086 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:21:22.086 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:21:22.086 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.086 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.086 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.086 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2024-04-16T16:21:22.086 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCCox5mcRqsChAACktmuUHvqWkp/mHT3UcjRw==", 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFox5mT52FFhAAChhDDfNZmIMgxE1j90LZpQ==", 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.087 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.088 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2024-04-16T16:21:22.088 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCIox5mC+EDIxAAxtvM0on00lxTLaoRjN0Gfw==", 2024-04-16T16:21:22.088 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.088 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:21:22.088 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:21:22.088 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:21:22.088 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.088 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.088 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.088 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2024-04-16T16:21:22.088 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ==", 2024-04-16T16:21:22.088 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.088 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:21:22.088 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mYpVUGhAAYdR+40O6zbWqgbI55F0FYA==", 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mWdlUDxAAuO2FsGH1/E+qP1xcyUEe3A==", 2024-04-16T16:21:22.089 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5miPdMJBAAqjKpctNYu9ot1lff3rouxg==", 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.090 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2024-04-16T16:21:22.091 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBpox5mO4PMLBAA/0YbQCfHujgdEzpXs0nOLw==", 2024-04-16T16:21:22.091 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.091 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:21:22.091 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:21:22.091 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:21:22.091 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:21:22.091 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.091 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.091 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.091 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2024-04-16T16:21:22.091 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mGevaLBAAPpA6g+Fdi/XHqAqG0QZNag==", 2024-04-16T16:21:22.091 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.091 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:21:22.092 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:21:22.092 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:21:22.092 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.092 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:21:22.092 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:22.092 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2024-04-16T16:21:22.092 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5moiNRIRAAz5JzOrjKNLkcphT+iMo+zw==", 2024-04-16T16:21:22.092 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:21:22.092 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:21:22.092 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:21:22.092 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:21:22.092 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.092 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:22.092 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-16T16:21:22.092 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:21:22.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.083+0000 7f8135000640 1 -- 172.21.15.45:0/2436286261 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f810c06ca40 msgr2=0x7f810c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:22.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.083+0000 7f8135000640 1 --2- 172.21.15.45:0/2436286261 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f810c06ca40 0x7f810c06ef00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f81300af540 tx=0x7f8120009040 comp rx=0 tx=0).stop 2024-04-16T16:21:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.083+0000 7f8135000640 1 -- 172.21.15.45:0/2436286261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81300afba0 msgr2=0x7f81300b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.083+0000 7f8135000640 1 --2- 172.21.15.45:0/2436286261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81300afba0 0x7f81300b7be0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f81300c6110 tx=0x7f8128011920 comp rx=0 tx=0).stop 2024-04-16T16:21:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.084+0000 7f8135000640 1 -- 172.21.15.45:0/2436286261 shutdown_connections 2024-04-16T16:21:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.084+0000 7f8135000640 1 --2- 172.21.15.45:0/2436286261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81300afba0 0x7f81300b7be0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.084+0000 7f8135000640 1 --2- 172.21.15.45:0/2436286261 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f810c06ca40 0x7f810c06ef00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.084+0000 7f8135000640 1 --2- 172.21.15.45:0/2436286261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81300af720 0x7f81300b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.084+0000 7f8135000640 1 --2- 172.21.15.45:0/2436286261 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8114004800 0x7f81301f3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.084+0000 7f8135000640 1 -- 172.21.15.45:0/2436286261 >> 172.21.15.45:0/2436286261 conn(0x7f81300d2610 msgr2=0x7f813009aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.084+0000 7f8135000640 1 -- 172.21.15.45:0/2436286261 shutdown_connections 2024-04-16T16:21:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.084+0000 7f8135000640 1 -- 172.21.15.45:0/2436286261 wait complete. 2024-04-16T16:21:22.103 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:21:22.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.310+0000 7fb517e00640 1 Processor -- start 2024-04-16T16:21:22.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.310+0000 7fb517e00640 1 -- start start 2024-04-16T16:21:22.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.312+0000 7fb517e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb5181608a0 0x7fb518160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:22.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.312+0000 7fb517e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb51815d360 con 0x7fb51815ccb0 2024-04-16T16:21:22.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.312+0000 7fb517e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb51815d640 con 0x7fb518146220 2024-04-16T16:21:22.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.312+0000 7fb517e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb51815d130 con 0x7fb5181608a0 2024-04-16T16:21:22.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.312+0000 7fb516000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb51815ccb0 0x7fb51806df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41412/0 (socket says 172.21.15.45:41412) 2024-04-16T16:21:22.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.312+0000 7fb517400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb5181608a0 0x7fb518160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:22.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.312+0000 7fb516000640 1 -- 172.21.15.45:0/950703652 learned_addr learned my addr 172.21.15.45:0/950703652 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:22.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.313+0000 7fb517400640 1 -- 172.21.15.45:0/950703652 >> v1:172.21.15.103:6789/0 conn(0x7fb518146220 legacy=0x7fb51806f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:22.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.313+0000 7fb517400640 1 -- 172.21.15.45:0/950703652 >> v1:172.21.15.45:6789/0 conn(0x7fb51815ccb0 legacy=0x7fb51806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:22.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.313+0000 7fb517400640 1 -- 172.21.15.45:0/950703652 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51815d920 con 0x7fb5181608a0 2024-04-16T16:21:22.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.314+0000 7fb517400640 1 --2- 172.21.15.45:0/950703652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5181608a0 0x7fb518160cf0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fb510009d90 tx=0x7fb51002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86cbc8db3eb2761d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:22.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.314+0000 7fb515600640 1 -- 172.21.15.45:0/950703652 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb510039070 con 0x7fb5181608a0 2024-04-16T16:21:22.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.314+0000 7fb515600640 1 -- 172.21.15.45:0/950703652 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb51002cc60 con 0x7fb5181608a0 2024-04-16T16:21:22.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.314+0000 7fb515600640 1 -- 172.21.15.45:0/950703652 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb510034660 con 0x7fb5181608a0 2024-04-16T16:21:22.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.315+0000 7fb517e00640 1 -- 172.21.15.45:0/950703652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5181608a0 msgr2=0x7fb518160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:22.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.315+0000 7fb517e00640 1 --2- 172.21.15.45:0/950703652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5181608a0 0x7fb518160cf0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fb510009d90 tx=0x7fb51002beb0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.315+0000 7fb517e00640 1 -- 172.21.15.45:0/950703652 shutdown_connections 2024-04-16T16:21:22.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.315+0000 7fb517e00640 1 --2- 172.21.15.45:0/950703652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5181608a0 0x7fb518160cf0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.315+0000 7fb517e00640 1 -- 172.21.15.45:0/950703652 >> 172.21.15.45:0/950703652 conn(0x7fb51806b930 msgr2=0x7fb51806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:22.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.315+0000 7fb517e00640 1 -- 172.21.15.45:0/950703652 shutdown_connections 2024-04-16T16:21:22.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.315+0000 7fb517e00640 1 -- 172.21.15.45:0/950703652 wait complete. 2024-04-16T16:21:22.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.317+0000 7fb517e00640 1 Processor -- start 2024-04-16T16:21:22.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.317+0000 7fb517e00640 1 -- start start 2024-04-16T16:21:22.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.318+0000 7fb517e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb518146220 0x7fb51815bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:22.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.318+0000 7fb517e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb51815ccb0 0x7fb51812b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:22.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.318+0000 7fb517e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5181608a0 0x7fb51812be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:22.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.318+0000 7fb517e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb518160410 con 0x7fb5181608a0 2024-04-16T16:21:22.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.318+0000 7fb517e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb51815dc00 con 0x7fb51815ccb0 2024-04-16T16:21:22.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.318+0000 7fb517e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb5180cf360 con 0x7fb518146220 2024-04-16T16:21:22.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.318+0000 7fb517400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5181608a0 0x7fb51812be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:22.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.318+0000 7fb517400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5181608a0 0x7fb51812be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56316/0 (socket says 172.21.15.45:56316) 2024-04-16T16:21:22.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.318+0000 7fb517400640 1 -- 172.21.15.45:0/4181955783 learned_addr learned my addr 172.21.15.45:0/4181955783 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:22.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.318+0000 7fb516000640 1 --2- 172.21.15.45:0/4181955783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb51815ccb0 0x7fb51812b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:22.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.318+0000 7fb517400640 1 -- 172.21.15.45:0/4181955783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb518146220 msgr2=0x7fb51815bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:22.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.318+0000 7fb517400640 1 --2- 172.21.15.45:0/4181955783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb518146220 0x7fb51815bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.318+0000 7fb517400640 1 -- 172.21.15.45:0/4181955783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb51815ccb0 msgr2=0x7fb51812b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:22.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.318+0000 7fb517400640 1 --2- 172.21.15.45:0/4181955783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb51815ccb0 0x7fb51812b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.319+0000 7fb517400640 1 -- 172.21.15.45:0/4181955783 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51813e230 con 0x7fb5181608a0 2024-04-16T16:21:22.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.319+0000 7fb517400640 1 --2- 172.21.15.45:0/4181955783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5181608a0 0x7fb51812be00 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fb510009d90 tx=0x7fb510004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:22.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.319+0000 7fb50ea00640 1 -- 172.21.15.45:0/4181955783 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb510039050 con 0x7fb5181608a0 2024-04-16T16:21:22.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.319+0000 7fb50ea00640 1 -- 172.21.15.45:0/4181955783 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb51003a040 con 0x7fb5181608a0 2024-04-16T16:21:22.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.319+0000 7fb50ea00640 1 -- 172.21.15.45:0/4181955783 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51003e610 con 0x7fb5181608a0 2024-04-16T16:21:22.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.319+0000 7fb517e00640 1 -- 172.21.15.45:0/4181955783 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb51812c4a0 con 0x7fb5181608a0 2024-04-16T16:21:22.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.319+0000 7fb517e00640 1 -- 172.21.15.45:0/4181955783 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb51812ca60 con 0x7fb5181608a0 2024-04-16T16:21:22.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.321+0000 7fb517e00640 1 -- 172.21.15.45:0/4181955783 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb518075780 con 0x7fb5181608a0 2024-04-16T16:21:22.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.322+0000 7fb50ea00640 1 -- 172.21.15.45:0/4181955783 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7fb510047020 con 0x7fb5181608a0 2024-04-16T16:21:22.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.323+0000 7fb50ea00640 1 --2- 172.21.15.45:0/4181955783 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb4f806ca10 0x7fb4f806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:22.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.323+0000 7fb50ea00640 1 -- 172.21.15.45:0/4181955783 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4857+0+0 (secure 0 0 0) 0x7fb5100addc0 con 0x7fb5181608a0 2024-04-16T16:21:22.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.324+0000 7fb516a00640 1 --2- 172.21.15.45:0/4181955783 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb4f806ca10 0x7fb4f806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:22.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.325+0000 7fb516a00640 1 --2- 172.21.15.45:0/4181955783 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb4f806ca10 0x7fb4f806eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb5180c8e30 tx=0x7fb5080079b0 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:22.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.327+0000 7fb50ea00640 1 -- 172.21.15.45:0/4181955783 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb51004e020 con 0x7fb5181608a0 2024-04-16T16:21:22.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.538+0000 7fb517e00640 1 -- 172.21.15.45:0/4181955783 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5180cff60 con 0x7fb5181608a0 2024-04-16T16:21:22.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.540+0000 7fb50ea00640 1 -- 172.21.15.45:0/4181955783 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) ==== 74+0+12901 (secure 0 0 0) 0x7fb51813e230 con 0x7fb5181608a0 2024-04-16T16:21:22.540 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:22.541 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":138,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:21:21.531822+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:21:22.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.543+0000 7fb517e00640 1 -- 172.21.15.45:0/4181955783 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb4f806ca10 msgr2=0x7fb4f806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:22.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.543+0000 7fb517e00640 1 --2- 172.21.15.45:0/4181955783 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb4f806ca10 0x7fb4f806eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb5180c8e30 tx=0x7fb5080079b0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.543+0000 7fb517e00640 1 -- 172.21.15.45:0/4181955783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5181608a0 msgr2=0x7fb51812be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:22.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.543+0000 7fb517e00640 1 --2- 172.21.15.45:0/4181955783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5181608a0 0x7fb51812be00 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fb510009d90 tx=0x7fb510004a40 comp rx=0 tx=0).stop 2024-04-16T16:21:22.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.543+0000 7fb517e00640 1 -- 172.21.15.45:0/4181955783 shutdown_connections 2024-04-16T16:21:22.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.543+0000 7fb517e00640 1 --2- 172.21.15.45:0/4181955783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5181608a0 0x7fb51812be00 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.543+0000 7fb517e00640 1 --2- 172.21.15.45:0/4181955783 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb4f806ca10 0x7fb4f806eed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.543+0000 7fb517e00640 1 --2- 172.21.15.45:0/4181955783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb51815ccb0 0x7fb51812b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.543+0000 7fb517e00640 1 --2- 172.21.15.45:0/4181955783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb518146220 0x7fb51815bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.543+0000 7fb517e00640 1 -- 172.21.15.45:0/4181955783 >> 172.21.15.45:0/4181955783 conn(0x7fb51806b930 msgr2=0x7fb5180c66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:22.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.544+0000 7fb517e00640 1 -- 172.21.15.45:0/4181955783 shutdown_connections 2024-04-16T16:21:22.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.544+0000 7fb517e00640 1 -- 172.21.15.45:0/4181955783 wait complete. 2024-04-16T16:21:22.563 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:21:22.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:21:22.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.789+0000 7f5522e00640 1 Processor -- start 2024-04-16T16:21:22.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.789+0000 7f5522e00640 1 -- start start 2024-04-16T16:21:22.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.790+0000 7f5522e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f551c0d4370 0x7f551c0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:22.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.790+0000 7f5522e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f551c15ee50 con 0x7f551c15ea70 2024-04-16T16:21:22.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.790+0000 7f5522e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f551c0d1a10 con 0x7f551c0cfa70 2024-04-16T16:21:22.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.790+0000 7f5522e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f551c0b6480 con 0x7f551c0d4370 2024-04-16T16:21:22.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.790+0000 7f5521a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f551c15ea70 0x7f551c0d11a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41416/0 (socket says 172.21.15.45:41416) 2024-04-16T16:21:22.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.790+0000 7f5521a00640 1 -- 172.21.15.45:0/1813304233 learned_addr learned my addr 172.21.15.45:0/1813304233 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:22.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.791+0000 7f551be00640 1 -- 172.21.15.45:0/1813304233 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1524903649 0 0) 0x7f551c15ee50 con 0x7f551c15ea70 2024-04-16T16:21:22.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.791+0000 7f551be00640 1 -- 172.21.15.45:0/1813304233 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5500003520 con 0x7f551c15ea70 2024-04-16T16:21:22.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.791+0000 7f5521000640 1 --2- 172.21.15.45:0/1813304233 >> v2:172.21.15.103:3300/0 conn(0x7f551c0d4370 0x7f551c0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:22.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.791+0000 7f551be00640 1 -- 172.21.15.45:0/1813304233 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2081887306 0 0) 0x7f5500003520 con 0x7f551c15ea70 2024-04-16T16:21:22.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.791+0000 7f551be00640 1 -- 172.21.15.45:0/1813304233 >> v2:172.21.15.103:3300/0 conn(0x7f551c0d4370 msgr2=0x7f551c0cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:22.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.791+0000 7f551be00640 1 --2- 172.21.15.45:0/1813304233 >> v2:172.21.15.103:3300/0 conn(0x7f551c0d4370 0x7f551c0cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.791+0000 7f551be00640 1 -- 172.21.15.45:0/1813304233 >> v1:172.21.15.103:6789/0 conn(0x7f551c0cfa70 legacy=0x7f551c162560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:22.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.791+0000 7f551be00640 1 -- 172.21.15.45:0/1813304233 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551c0b6760 con 0x7f551c15ea70 2024-04-16T16:21:22.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.791+0000 7f551be00640 1 -- 172.21.15.45:0/1813304233 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f550c001670 con 0x7f551c15ea70 2024-04-16T16:21:22.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.791+0000 7f551be00640 1 -- 172.21.15.45:0/1813304233 >> v1:172.21.15.45:6789/0 conn(0x7f551c15ea70 legacy=0x7f551c0d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:22.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.791+0000 7f551be00640 1 --2- 172.21.15.45:0/1813304233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5500003a00 0x7f5500003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:22.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.791+0000 7f551be00640 1 --2- 172.21.15.45:0/1813304233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5500004800 0x7f5500004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:22.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.792+0000 7f551be00640 1 --2- 172.21.15.45:0/1813304233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5500005210 0x7f550000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:22.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.792+0000 7f551be00640 1 -- 172.21.15.45:0/1813304233 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f551c0b6480 con 0x7f5500003a00 2024-04-16T16:21:22.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.792+0000 7f551be00640 1 -- 172.21.15.45:0/1813304233 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f550000c160 con 0x7f5500005210 2024-04-16T16:21:22.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.792+0000 7f551be00640 1 -- 172.21.15.45:0/1813304233 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f550000c360 con 0x7f5500004800 2024-04-16T16:21:22.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.792+0000 7f5521a00640 1 --2- 172.21.15.45:0/1813304233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5500003a00 0x7f5500003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:22.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.792+0000 7f5522400640 1 --2- 172.21.15.45:0/1813304233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5500005210 0x7f550000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:22.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.792+0000 7f5521a00640 1 -- 172.21.15.45:0/1813304233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5500004800 msgr2=0x7f5500004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:22.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.792+0000 7f5521a00640 1 --2- 172.21.15.45:0/1813304233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5500004800 0x7f5500004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.792+0000 7f5521a00640 1 -- 172.21.15.45:0/1813304233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5500005210 msgr2=0x7f550000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:22.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.792+0000 7f5521a00640 1 --2- 172.21.15.45:0/1813304233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5500005210 0x7f550000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.792+0000 7f5521a00640 1 -- 172.21.15.45:0/1813304233 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5500003520 con 0x7f5500003a00 2024-04-16T16:21:22.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.792+0000 7f5522400640 1 --2- 172.21.15.45:0/1813304233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5500005210 0x7f550000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:22.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.792+0000 7f5522400640 1 -- 172.21.15.45:0/1813304233 reap_dead start 2024-04-16T16:21:22.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.793+0000 7f5521a00640 1 --2- 172.21.15.45:0/1813304233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5500003a00 0x7f5500003e20 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f550c00c190 tx=0x7f5500005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5108533caf015eb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:22.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.794+0000 7f551be00640 1 -- 172.21.15.45:0/1813304233 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c007c60 con 0x7f5500003a00 2024-04-16T16:21:22.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.794+0000 7f551be00640 1 -- 172.21.15.45:0/1813304233 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f550c030070 con 0x7f5500003a00 2024-04-16T16:21:22.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.794+0000 7f551be00640 1 -- 172.21.15.45:0/1813304233 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c005d90 con 0x7f5500003a00 2024-04-16T16:21:22.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.794+0000 7f5522e00640 1 -- 172.21.15.45:0/1813304233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5500003a00 msgr2=0x7f5500003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:22.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.794+0000 7f5522e00640 1 --2- 172.21.15.45:0/1813304233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5500003a00 0x7f5500003e20 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f550c00c190 tx=0x7f5500005830 comp rx=0 tx=0).stop 2024-04-16T16:21:22.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.794+0000 7f5522e00640 1 -- 172.21.15.45:0/1813304233 shutdown_connections 2024-04-16T16:21:22.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.794+0000 7f5522e00640 1 --2- 172.21.15.45:0/1813304233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5500003a00 0x7f5500003e20 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.794+0000 7f5522e00640 1 -- 172.21.15.45:0/1813304233 >> 172.21.15.45:0/1813304233 conn(0x7f551c06d3b0 msgr2=0x7f551c073060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:22.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.794+0000 7f5522e00640 1 -- 172.21.15.45:0/1813304233 shutdown_connections 2024-04-16T16:21:22.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.794+0000 7f5522e00640 1 -- 172.21.15.45:0/1813304233 wait complete. 2024-04-16T16:21:22.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.796+0000 7f5522e00640 1 Processor -- start 2024-04-16T16:21:22.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.796+0000 7f5522e00640 1 -- start start 2024-04-16T16:21:22.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.797+0000 7f5522e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5500003a00 0x7f551c14bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:22.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.797+0000 7f5522e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f551c0c97a0 0x7f551c0cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:22.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.797+0000 7f5522e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f551c15f040 0x7f551c0cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:22.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.797+0000 7f5522e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f551c0b6f60 con 0x7f5500003a00 2024-04-16T16:21:22.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.797+0000 7f5522e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f551c0cf530 con 0x7f551c0c97a0 2024-04-16T16:21:22.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.797+0000 7f5522e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f551c0cb210 con 0x7f551c15f040 2024-04-16T16:21:22.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.798+0000 7f5521a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5500003a00 0x7f551c14bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:22.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.798+0000 7f5521a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5500003a00 0x7f551c14bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56332/0 (socket says 172.21.15.45:56332) 2024-04-16T16:21:22.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.798+0000 7f5521a00640 1 -- 172.21.15.45:0/3607666598 learned_addr learned my addr 172.21.15.45:0/3607666598 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:22.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.798+0000 7f5522400640 1 --2- 172.21.15.45:0/3607666598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f551c15f040 0x7f551c0cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.798+0000 7f5521000640 1 --2- 172.21.15.45:0/3607666598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f551c0c97a0 0x7f551c0cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.798+0000 7f5521a00640 1 -- 172.21.15.45:0/3607666598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f551c15f040 msgr2=0x7f551c0cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.798+0000 7f5521a00640 1 --2- 172.21.15.45:0/3607666598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f551c15f040 0x7f551c0cc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.798+0000 7f5521a00640 1 -- 172.21.15.45:0/3607666598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f551c0c97a0 msgr2=0x7f551c0cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.798+0000 7f5521a00640 1 --2- 172.21.15.45:0/3607666598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f551c0c97a0 0x7f551c0cacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.798+0000 7f5521a00640 1 -- 172.21.15.45:0/3607666598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551c158340 con 0x7f5500003a00 2024-04-16T16:21:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.798+0000 7f5522400640 1 --2- 172.21.15.45:0/3607666598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f551c15f040 0x7f551c0cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.798+0000 7f5521a00640 1 --2- 172.21.15.45:0/3607666598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5500003a00 0x7f551c14bd90 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f550c00c2b0 tx=0x7f550c001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.799+0000 7f5519600640 1 -- 172.21.15.45:0/3607666598 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c045070 con 0x7f5500003a00 2024-04-16T16:21:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.799+0000 7f5519600640 1 -- 172.21.15.45:0/3607666598 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f550c0049f0 con 0x7f5500003a00 2024-04-16T16:21:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.799+0000 7f5519600640 1 -- 172.21.15.45:0/3607666598 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c02de60 con 0x7f5500003a00 2024-04-16T16:21:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.799+0000 7f5522e00640 1 -- 172.21.15.45:0/3607666598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f551c0c9c30 con 0x7f5500003a00 2024-04-16T16:21:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.799+0000 7f5522e00640 1 -- 172.21.15.45:0/3607666598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f551c0ca1f0 con 0x7f5500003a00 2024-04-16T16:21:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.800+0000 7f5522e00640 1 -- 172.21.15.45:0/3607666598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f551c075780 con 0x7f5500003a00 2024-04-16T16:21:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.801+0000 7f5519600640 1 -- 172.21.15.45:0/3607666598 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f550c007070 con 0x7f5500003a00 2024-04-16T16:21:22.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.802+0000 7f5519600640 1 --2- 172.21.15.45:0/3607666598 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f54fc06ca60 0x7f54fc06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:22.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.802+0000 7f5519600640 1 -- 172.21.15.45:0/3607666598 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4857+0+0 (secure 0 0 0) 0x7f550c0acac0 con 0x7f5500003a00 2024-04-16T16:21:22.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.803+0000 7f5521000640 1 --2- 172.21.15.45:0/3607666598 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f54fc06ca60 0x7f54fc06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:22.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.804+0000 7f5521000640 1 --2- 172.21.15.45:0/3607666598 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f54fc06ca60 0x7f54fc06ef20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f551c15ce40 tx=0x7f5508008040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:22.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:22.807+0000 7f5519600640 1 -- 172.21.15.45:0/3607666598 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f550c07b110 con 0x7f5500003a00 2024-04-16T16:21:23.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.010+0000 7f5522e00640 1 -- 172.21.15.45:0/3607666598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f551c0d3680 con 0x7f5500003a00 2024-04-16T16:21:23.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.011+0000 7f5519600640 1 -- 172.21.15.45:0/3607666598 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v138) ==== 93+0+10 (secure 0 0 0) 0x7f550c07aa10 con 0x7f5500003a00 2024-04-16T16:21:23.011 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:21:23.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.013+0000 7f5522e00640 1 -- 172.21.15.45:0/3607666598 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f54fc06ca60 msgr2=0x7f54fc06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:23.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.013+0000 7f5522e00640 1 --2- 172.21.15.45:0/3607666598 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f54fc06ca60 0x7f54fc06ef20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f551c15ce40 tx=0x7f5508008040 comp rx=0 tx=0).stop 2024-04-16T16:21:23.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.013+0000 7f5522e00640 1 -- 172.21.15.45:0/3607666598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5500003a00 msgr2=0x7f551c14bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:23.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.013+0000 7f5522e00640 1 --2- 172.21.15.45:0/3607666598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5500003a00 0x7f551c14bd90 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f550c00c2b0 tx=0x7f550c001970 comp rx=0 tx=0).stop 2024-04-16T16:21:23.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.013+0000 7f5522e00640 1 -- 172.21.15.45:0/3607666598 shutdown_connections 2024-04-16T16:21:23.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.013+0000 7f5522e00640 1 --2- 172.21.15.45:0/3607666598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f551c15f040 0x7f551c0cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.013+0000 7f5522e00640 1 --2- 172.21.15.45:0/3607666598 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f54fc06ca60 0x7f54fc06ef20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.013+0000 7f5522e00640 1 --2- 172.21.15.45:0/3607666598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f551c0c97a0 0x7f551c0cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.013+0000 7f5522e00640 1 --2- 172.21.15.45:0/3607666598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5500003a00 0x7f551c14bd90 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.013+0000 7f5522e00640 1 -- 172.21.15.45:0/3607666598 >> 172.21.15.45:0/3607666598 conn(0x7f551c06d3b0 msgr2=0x7f551c0b9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:23.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.014+0000 7f5522e00640 1 -- 172.21.15.45:0/3607666598 shutdown_connections 2024-04-16T16:21:23.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.014+0000 7f5522e00640 1 -- 172.21.15.45:0/3607666598 wait complete. 2024-04-16T16:21:23.034 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-16T16:21:23.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.255+0000 7fa9b3e00640 1 Processor -- start 2024-04-16T16:21:23.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.256+0000 7fa9b3e00640 1 -- start start 2024-04-16T16:21:23.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.257+0000 7fa9b3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9b40d52d0 0x7fa9b406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:23.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.257+0000 7fa9b3e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9b40d3080 con 0x7fa9b4162270 2024-04-16T16:21:23.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.257+0000 7fa9b3e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9b40d3360 con 0x7fa9b4161e90 2024-04-16T16:21:23.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.257+0000 7fa9b3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b40d2e30 con 0x7fa9b40d52d0 2024-04-16T16:21:23.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.258+0000 7fa9b2000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa9b4162270 0x7fa9b40d3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41430/0 (socket says 172.21.15.45:41430) 2024-04-16T16:21:23.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.258+0000 7fa9b2000640 1 -- 172.21.15.45:0/4216657569 learned_addr learned my addr 172.21.15.45:0/4216657569 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:23.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.258+0000 7fa9b1600640 1 -- 172.21.15.45:0/4216657569 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 797836802 0 0) 0x7fa9b40d3080 con 0x7fa9b4162270 2024-04-16T16:21:23.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.258+0000 7fa9b1600640 1 -- 172.21.15.45:0/4216657569 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa998003520 con 0x7fa9b4162270 2024-04-16T16:21:23.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.258+0000 7fa9b3400640 1 --2- 172.21.15.45:0/4216657569 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9b40d52d0 0x7fa9b406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:23.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b1600640 1 -- 172.21.15.45:0/4216657569 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2051725521 0 0) 0x7fa998003520 con 0x7fa9b4162270 2024-04-16T16:21:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b1600640 1 -- 172.21.15.45:0/4216657569 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9b40d52d0 msgr2=0x7fa9b406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b1600640 1 --2- 172.21.15.45:0/4216657569 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9b40d52d0 0x7fa9b406ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b1600640 1 -- 172.21.15.45:0/4216657569 >> v1:172.21.15.103:6789/0 conn(0x7fa9b4161e90 legacy=0x7fa9b40cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b1600640 1 -- 172.21.15.45:0/4216657569 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b40d3640 con 0x7fa9b4162270 2024-04-16T16:21:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b1600640 1 -- 172.21.15.45:0/4216657569 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fa9a0001670 con 0x7fa9b4162270 2024-04-16T16:21:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b1600640 1 -- 172.21.15.45:0/4216657569 >> v1:172.21.15.45:6789/0 conn(0x7fa9b4162270 legacy=0x7fa9b40d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b1600640 1 --2- 172.21.15.45:0/4216657569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9980039e0 0x7fa998003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b1600640 1 --2- 172.21.15.45:0/4216657569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa998004800 0x7fa998004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b1600640 1 --2- 172.21.15.45:0/4216657569 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9b40d52d0 0x7fa998005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b1600640 1 -- 172.21.15.45:0/4216657569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b40d2e30 con 0x7fa998004800 2024-04-16T16:21:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b1600640 1 -- 172.21.15.45:0/4216657569 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa99800bbb0 con 0x7fa9980039e0 2024-04-16T16:21:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b1600640 1 -- 172.21.15.45:0/4216657569 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa99800bdb0 con 0x7fa9b40d52d0 2024-04-16T16:21:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b2000640 1 --2- 172.21.15.45:0/4216657569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa998004800 0x7fa998004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b2a00640 1 --2- 172.21.15.45:0/4216657569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9980039e0 0x7fa998003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b3400640 1 --2- 172.21.15.45:0/4216657569 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9b40d52d0 0x7fa998005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:23.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b2000640 1 -- 172.21.15.45:0/4216657569 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9b40d52d0 msgr2=0x7fa998005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:23.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b2000640 1 --2- 172.21.15.45:0/4216657569 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9b40d52d0 0x7fa998005200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b2000640 1 -- 172.21.15.45:0/4216657569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9980039e0 msgr2=0x7fa998003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:23.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.259+0000 7fa9b2000640 1 --2- 172.21.15.45:0/4216657569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9980039e0 0x7fa998003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.260+0000 7fa9b2000640 1 -- 172.21.15.45:0/4216657569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa998003520 con 0x7fa998004800 2024-04-16T16:21:23.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.261+0000 7fa9b2000640 1 --2- 172.21.15.45:0/4216657569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa998004800 0x7fa998004c80 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fa9b406dab0 tx=0x7fa998005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8678974514465e6c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:23.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.261+0000 7fa9b1600640 1 -- 172.21.15.45:0/4216657569 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0007d30 con 0x7fa998004800 2024-04-16T16:21:23.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.261+0000 7fa9b1600640 1 -- 172.21.15.45:0/4216657569 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9a0030070 con 0x7fa998004800 2024-04-16T16:21:23.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.261+0000 7fa9b1600640 1 -- 172.21.15.45:0/4216657569 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0005d30 con 0x7fa998004800 2024-04-16T16:21:23.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.262+0000 7fa9b3e00640 1 -- 172.21.15.45:0/4216657569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa998004800 msgr2=0x7fa998004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:23.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.262+0000 7fa9b3e00640 1 --2- 172.21.15.45:0/4216657569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa998004800 0x7fa998004c80 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fa9b406dab0 tx=0x7fa998005830 comp rx=0 tx=0).stop 2024-04-16T16:21:23.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.262+0000 7fa9b3400640 1 -- 172.21.15.45:0/4216657569 reap_dead start 2024-04-16T16:21:23.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.262+0000 7fa9b3e00640 1 -- 172.21.15.45:0/4216657569 shutdown_connections 2024-04-16T16:21:23.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.262+0000 7fa9b3e00640 1 -- 172.21.15.45:0/4216657569 >> 172.21.15.45:0/4216657569 conn(0x7fa9b40d1940 msgr2=0x7fa9b40d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:23.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.262+0000 7fa9b3e00640 1 -- 172.21.15.45:0/4216657569 shutdown_connections 2024-04-16T16:21:23.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.262+0000 7fa9b3e00640 1 -- 172.21.15.45:0/4216657569 wait complete. 2024-04-16T16:21:23.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.264+0000 7fa9b3e00640 1 Processor -- start 2024-04-16T16:21:23.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.264+0000 7fa9b3e00640 1 -- start start 2024-04-16T16:21:23.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.265+0000 7fa9b3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa998004800 0x7fa9b40aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:23.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.265+0000 7fa9b3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b40cd930 0x7fa9b40cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:23.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.265+0000 7fa9b3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9b4150c10 0x7fa9b41510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.265+0000 7fa9b3e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b40d46d0 con 0x7fa9b4150c10 2024-04-16T16:21:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.265+0000 7fa9b3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b40d4550 con 0x7fa9b40cd930 2024-04-16T16:21:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.265+0000 7fa9b3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b406ce40 con 0x7fa998004800 2024-04-16T16:21:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.265+0000 7fa9b3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9b4150c10 0x7fa9b41510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.265+0000 7fa9b3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9b4150c10 0x7fa9b41510c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56350/0 (socket says 172.21.15.45:56350) 2024-04-16T16:21:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.265+0000 7fa9b3400640 1 -- 172.21.15.45:0/2025306187 learned_addr learned my addr 172.21.15.45:0/2025306187 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.266+0000 7fa9b2a00640 1 --2- 172.21.15.45:0/2025306187 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa998004800 0x7fa9b40aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.266+0000 7fa9b2000640 1 --2- 172.21.15.45:0/2025306187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b40cd930 0x7fa9b40cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.266+0000 7fa9b3400640 1 -- 172.21.15.45:0/2025306187 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa998004800 msgr2=0x7fa9b40aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.266+0000 7fa9b3400640 1 --2- 172.21.15.45:0/2025306187 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa998004800 0x7fa9b40aea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.266+0000 7fa9b3400640 1 -- 172.21.15.45:0/2025306187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b40cd930 msgr2=0x7fa9b40cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.266+0000 7fa9b3400640 1 --2- 172.21.15.45:0/2025306187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b40cd930 0x7fa9b40cdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.266+0000 7fa9b3400640 1 -- 172.21.15.45:0/2025306187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b41519b0 con 0x7fa9b4150c10 2024-04-16T16:21:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.266+0000 7fa9b3400640 1 --2- 172.21.15.45:0/2025306187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9b4150c10 0x7fa9b41510c0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fa9b406c680 tx=0x7fa9ac00e510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.267+0000 7fa9aaa00640 1 -- 172.21.15.45:0/2025306187 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac007e40 con 0x7fa9b4150c10 2024-04-16T16:21:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.267+0000 7fa9aaa00640 1 -- 172.21.15.45:0/2025306187 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9ac01a070 con 0x7fa9b4150c10 2024-04-16T16:21:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.267+0000 7fa9aaa00640 1 -- 172.21.15.45:0/2025306187 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac015650 con 0x7fa9b4150c10 2024-04-16T16:21:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.267+0000 7fa9b3e00640 1 -- 172.21.15.45:0/2025306187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b4151600 con 0x7fa9b4150c10 2024-04-16T16:21:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.267+0000 7fa9b3e00640 1 -- 172.21.15.45:0/2025306187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b4151c20 con 0x7fa9b4150c10 2024-04-16T16:21:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.268+0000 7fa9b3e00640 1 -- 172.21.15.45:0/2025306187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9b4075780 con 0x7fa9b4150c10 2024-04-16T16:21:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.269+0000 7fa9aaa00640 1 -- 172.21.15.45:0/2025306187 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7fa9ac012030 con 0x7fa9b4150c10 2024-04-16T16:21:23.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.270+0000 7fa9aaa00640 1 --2- 172.21.15.45:0/2025306187 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa99406ca10 0x7fa99406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:23.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.270+0000 7fa9aaa00640 1 -- 172.21.15.45:0/2025306187 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4857+0+0 (secure 0 0 0) 0x7fa9ac08c240 con 0x7fa9b4150c10 2024-04-16T16:21:23.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.271+0000 7fa9b2a00640 1 --2- 172.21.15.45:0/2025306187 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa99406ca10 0x7fa99406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:23.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.272+0000 7fa9b2a00640 1 --2- 172.21.15.45:0/2025306187 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa99406ca10 0x7fa99406eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4007fd0 tx=0x7fa9a4009040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:23.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.274+0000 7fa9aaa00640 1 -- 172.21.15.45:0/2025306187 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa9ac010040 con 0x7fa9b4150c10 2024-04-16T16:21:23.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.483+0000 7fa9b3e00640 1 -- 172.21.15.45:0/2025306187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa9b406dae0 con 0x7fa9b4150c10 2024-04-16T16:21:23.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.484+0000 7fa9aaa00640 1 -- 172.21.15.45:0/2025306187 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v138) ==== 84+0+65 (secure 0 0 0) 0x7fa9ac05a7a0 con 0x7fa9b4150c10 2024-04-16T16:21:23.484 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:23.484 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2024-04-16T16:21:23.484 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:21:23.485 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2024-04-16T16:21:23.485 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2024-04-16T16:21:23.485 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:21:23.485 INFO:teuthology.orchestra.run.smithi045.stdout:] 2024-04-16T16:21:23.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.487+0000 7fa9b3e00640 1 -- 172.21.15.45:0/2025306187 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa99406ca10 msgr2=0x7fa99406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:23.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.487+0000 7fa9b3e00640 1 --2- 172.21.15.45:0/2025306187 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa99406ca10 0x7fa99406eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4007fd0 tx=0x7fa9a4009040 comp rx=0 tx=0).stop 2024-04-16T16:21:23.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.487+0000 7fa9b3e00640 1 -- 172.21.15.45:0/2025306187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9b4150c10 msgr2=0x7fa9b41510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:23.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.487+0000 7fa9b3e00640 1 --2- 172.21.15.45:0/2025306187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9b4150c10 0x7fa9b41510c0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fa9b406c680 tx=0x7fa9ac00e510 comp rx=0 tx=0).stop 2024-04-16T16:21:23.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.487+0000 7fa9b3e00640 1 -- 172.21.15.45:0/2025306187 shutdown_connections 2024-04-16T16:21:23.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.487+0000 7fa9b3e00640 1 --2- 172.21.15.45:0/2025306187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9b4150c10 0x7fa9b41510c0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.487+0000 7fa9b3e00640 1 --2- 172.21.15.45:0/2025306187 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fa99406ca10 0x7fa99406eed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.487+0000 7fa9b3e00640 1 --2- 172.21.15.45:0/2025306187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9b40cd930 0x7fa9b40cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.487+0000 7fa9b3e00640 1 --2- 172.21.15.45:0/2025306187 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa998004800 0x7fa9b40aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.487+0000 7fa9b3e00640 1 -- 172.21.15.45:0/2025306187 >> 172.21.15.45:0/2025306187 conn(0x7fa9b40d1940 msgr2=0x7fa9b40ca3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:23.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.487+0000 7fa9b3e00640 1 -- 172.21.15.45:0/2025306187 shutdown_connections 2024-04-16T16:21:23.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.487+0000 7fa9b3e00640 1 -- 172.21.15.45:0/2025306187 wait complete. 2024-04-16T16:21:23.508 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-16T16:21:23.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-16T16:21:23.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.727+0000 7f3867200640 1 Processor -- start 2024-04-16T16:21:23.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.728+0000 7f3867200640 1 -- start start 2024-04-16T16:21:23.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.729+0000 7f3867200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f386815e970 0x7f38680ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:23.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.729+0000 7f3867200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f386806bff0 0x7f386806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:23.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.729+0000 7f3867200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f386806d2a0 con 0x7f386806ce10 2024-04-16T16:21:23.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.729+0000 7f3867200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3868070650 con 0x7f386815e970 2024-04-16T16:21:23.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.729+0000 7f3867200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f386806c970 con 0x7f386806bff0 2024-04-16T16:21:23.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.730+0000 7f3866800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f386806bff0 0x7f386806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:23.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.730+0000 7f3866800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f386806bff0 0x7f386806c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:47852/0 (socket says 172.21.15.45:47852) 2024-04-16T16:21:23.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.730+0000 7f3866800640 1 -- 172.21.15.45:0/370638170 learned_addr learned my addr 172.21.15.45:0/370638170 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:23.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.730+0000 7f3865e00640 1 --2- 172.21.15.45:0/370638170 >> v2:172.21.15.103:3300/0 conn(0x7f386815e970 0x7f38680ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:23.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.731+0000 7f3866800640 1 -- 172.21.15.45:0/370638170 >> v2:172.21.15.103:3300/0 conn(0x7f386815e970 msgr2=0x7f38680ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:23.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.731+0000 7f3866800640 1 --2- 172.21.15.45:0/370638170 >> v2:172.21.15.103:3300/0 conn(0x7f386815e970 0x7f38680ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.731+0000 7f3866800640 1 -- 172.21.15.45:0/370638170 >> v1:172.21.15.103:6789/0 conn(0x7f386806ce10 legacy=0x7f386806e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:23.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.731+0000 7f3865400640 1 -- 172.21.15.45:0/370638170 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3622448381 0 0) 0x7f386806d2a0 con 0x7f386806ce10 2024-04-16T16:21:23.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.731+0000 7f3866800640 1 -- 172.21.15.45:0/370638170 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3868070090 con 0x7f386806bff0 2024-04-16T16:21:23.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.733+0000 7f3866800640 1 --2- 172.21.15.45:0/370638170 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f386806bff0 0x7f386806c430 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3860009e50 tx=0x7f386002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e195cb6bdc68891a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:23.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.733+0000 7f3865400640 1 -- 172.21.15.45:0/370638170 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386002cb00 con 0x7f386806bff0 2024-04-16T16:21:23.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.733+0000 7f3865400640 1 -- 172.21.15.45:0/370638170 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f386002cca0 con 0x7f386806bff0 2024-04-16T16:21:23.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.733+0000 7f3865400640 1 -- 172.21.15.45:0/370638170 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3860034610 con 0x7f386806bff0 2024-04-16T16:21:23.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.734+0000 7f3867200640 1 -- 172.21.15.45:0/370638170 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f386806bff0 msgr2=0x7f386806c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:23.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.734+0000 7f3867200640 1 --2- 172.21.15.45:0/370638170 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f386806bff0 0x7f386806c430 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3860009e50 tx=0x7f386002bf60 comp rx=0 tx=0).stop 2024-04-16T16:21:23.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.734+0000 7f3867200640 1 -- 172.21.15.45:0/370638170 shutdown_connections 2024-04-16T16:21:23.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.734+0000 7f3867200640 1 --2- 172.21.15.45:0/370638170 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f386806bff0 0x7f386806c430 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.734+0000 7f3867200640 1 --2- 172.21.15.45:0/370638170 >> v2:172.21.15.103:3300/0 conn(0x7f386815e970 0x7f38680ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.734+0000 7f3867200640 1 -- 172.21.15.45:0/370638170 >> 172.21.15.45:0/370638170 conn(0x7f38680b36b0 msgr2=0x7f38680cd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:23.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.734+0000 7f3867200640 1 -- 172.21.15.45:0/370638170 shutdown_connections 2024-04-16T16:21:23.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.734+0000 7f3867200640 1 -- 172.21.15.45:0/370638170 wait complete. 2024-04-16T16:21:23.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.736+0000 7f3867200640 1 Processor -- start 2024-04-16T16:21:23.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.736+0000 7f3867200640 1 -- start start 2024-04-16T16:21:23.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.737+0000 7f3867200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f386806bff0 0x7f38680ae050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:23.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.737+0000 7f3867200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f386815e970 0x7f38680ae590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:23.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.737+0000 7f3867200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38680b4ae0 0x7f38680b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:23.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.737+0000 7f3867200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f386806ee50 con 0x7f38680b4ae0 2024-04-16T16:21:23.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.737+0000 7f3867200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f386806bb60 con 0x7f386806bff0 2024-04-16T16:21:23.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.737+0000 7f3867200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f386815ed50 con 0x7f386815e970 2024-04-16T16:21:23.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.737+0000 7f3866800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38680b4ae0 0x7f38680b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:23.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.737+0000 7f3866800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38680b4ae0 0x7f38680b4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58842/0 (socket says 172.21.15.45:58842) 2024-04-16T16:21:23.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.737+0000 7f3866800640 1 -- 172.21.15.45:0/225533419 learned_addr learned my addr 172.21.15.45:0/225533419 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:23.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.738+0000 7f385fe00640 1 --2- 172.21.15.45:0/225533419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f386815e970 0x7f38680ae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:23.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.738+0000 7f3865e00640 1 --2- 172.21.15.45:0/225533419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f386806bff0 0x7f38680ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:23.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.738+0000 7f3866800640 1 -- 172.21.15.45:0/225533419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f386815e970 msgr2=0x7f38680ae590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:23.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.738+0000 7f3866800640 1 --2- 172.21.15.45:0/225533419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f386815e970 0x7f38680ae590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.738+0000 7f3866800640 1 -- 172.21.15.45:0/225533419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f386806bff0 msgr2=0x7f38680ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:23.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.738+0000 7f3866800640 1 --2- 172.21.15.45:0/225533419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f386806bff0 0x7f38680ae050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:23.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.738+0000 7f3866800640 1 -- 172.21.15.45:0/225533419 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f386814bb10 con 0x7f38680b4ae0 2024-04-16T16:21:23.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.738+0000 7f3866800640 1 --2- 172.21.15.45:0/225533419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38680b4ae0 0x7f38680b4f70 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f3860009e50 tx=0x7f3860004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:23.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.738+0000 7f385e000640 1 -- 172.21.15.45:0/225533419 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3860043020 con 0x7f38680b4ae0 2024-04-16T16:21:23.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.739+0000 7f385e000640 1 -- 172.21.15.45:0/225533419 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f386002ce50 con 0x7f38680b4ae0 2024-04-16T16:21:23.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.739+0000 7f3867200640 1 -- 172.21.15.45:0/225533419 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f386814c450 con 0x7f38680b4ae0 2024-04-16T16:21:23.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.739+0000 7f385e000640 1 -- 172.21.15.45:0/225533419 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386003e690 con 0x7f38680b4ae0 2024-04-16T16:21:23.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.739+0000 7f3867200640 1 -- 172.21.15.45:0/225533419 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38680b54b0 con 0x7f38680b4ae0 2024-04-16T16:21:23.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.740+0000 7f3867200640 1 -- 172.21.15.45:0/225533419 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38680b02d0 con 0x7f38680b4ae0 2024-04-16T16:21:23.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.741+0000 7f385e000640 1 -- 172.21.15.45:0/225533419 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f3860046020 con 0x7f38680b4ae0 2024-04-16T16:21:23.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.742+0000 7f385e000640 1 --2- 172.21.15.45:0/225533419 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f384406ca40 0x7f384406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:23.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.742+0000 7f385e000640 1 -- 172.21.15.45:0/225533419 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4857+0+0 (secure 0 0 0) 0x7f38600adfb0 con 0x7f38680b4ae0 2024-04-16T16:21:23.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.743+0000 7f3865e00640 1 --2- 172.21.15.45:0/225533419 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f384406ca40 0x7f384406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:23.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.744+0000 7f3865e00640 1 --2- 172.21.15.45:0/225533419 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f384406ca40 0x7f384406ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3858006fd0 tx=0x7f3858008040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:23.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.747+0000 7f385e000640 1 -- 172.21.15.45:0/225533419 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f38680b02d0 con 0x7f38680b4ae0 2024-04-16T16:21:23.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:23.957+0000 7f3867200640 1 -- 172.21.15.45:0/225533419 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f3868075780 con 0x7f38680b4ae0 2024-04-16T16:21:24.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.555+0000 7f385e000640 1 -- 172.21.15.45:0/225533419 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v139) ==== 138+0+0 (secure 0 0 0) 0x7f386004d030 con 0x7f38680b4ae0 2024-04-16T16:21:24.555 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2024-04-16T16:21:24.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.558+0000 7f3867200640 1 -- 172.21.15.45:0/225533419 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f384406ca40 msgr2=0x7f384406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:24.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.558+0000 7f3867200640 1 --2- 172.21.15.45:0/225533419 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f384406ca40 0x7f384406ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3858006fd0 tx=0x7f3858008040 comp rx=0 tx=0).stop 2024-04-16T16:21:24.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.558+0000 7f3867200640 1 -- 172.21.15.45:0/225533419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38680b4ae0 msgr2=0x7f38680b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:24.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.558+0000 7f3867200640 1 --2- 172.21.15.45:0/225533419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38680b4ae0 0x7f38680b4f70 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f3860009e50 tx=0x7f3860004a40 comp rx=0 tx=0).stop 2024-04-16T16:21:24.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.559+0000 7f3867200640 1 -- 172.21.15.45:0/225533419 shutdown_connections 2024-04-16T16:21:24.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.559+0000 7f3867200640 1 --2- 172.21.15.45:0/225533419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38680b4ae0 0x7f38680b4f70 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:24.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.559+0000 7f3867200640 1 --2- 172.21.15.45:0/225533419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f386815e970 0x7f38680ae590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:24.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.559+0000 7f3867200640 1 --2- 172.21.15.45:0/225533419 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f384406ca40 0x7f384406ef00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:24.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.559+0000 7f3867200640 1 --2- 172.21.15.45:0/225533419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f386806bff0 0x7f38680ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:24.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.559+0000 7f3867200640 1 -- 172.21.15.45:0/225533419 >> 172.21.15.45:0/225533419 conn(0x7f38680b36b0 msgr2=0x7f386815dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:24.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.559+0000 7f3867200640 1 -- 172.21.15.45:0/225533419 shutdown_connections 2024-04-16T16:21:24.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.559+0000 7f3867200640 1 -- 172.21.15.45:0/225533419 wait complete. 2024-04-16T16:21:24.590 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-16T16:21:24.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.800+0000 7fd618c00640 1 Processor -- start 2024-04-16T16:21:24.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.801+0000 7fd618c00640 1 -- start start 2024-04-16T16:21:24.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.802+0000 7fd618c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd614155ab0 0x7fd614162ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:24.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.802+0000 7fd618c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd6140ba610 0x7fd61415efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:24.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.802+0000 7fd618c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd6140d18c0 0x7fd6140d1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:24.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.802+0000 7fd618c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd61415ff00 con 0x7fd6140d18c0 2024-04-16T16:21:24.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.802+0000 7fd618c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd614155e90 con 0x7fd6140ba610 2024-04-16T16:21:24.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.802+0000 7fd618c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd61415f940 con 0x7fd614155ab0 2024-04-16T16:21:24.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.803+0000 7fd613e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd6140d18c0 0x7fd6140d1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:24.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.803+0000 7fd613e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd6140d18c0 0x7fd6140d1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58856/0 (socket says 172.21.15.45:58856) 2024-04-16T16:21:24.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.803+0000 7fd613e00640 1 -- 172.21.15.45:0/3582637688 learned_addr learned my addr 172.21.15.45:0/3582637688 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:24.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.803+0000 7fd612a00640 1 --2- 172.21.15.45:0/3582637688 >> v2:172.21.15.103:3300/0 conn(0x7fd6140ba610 0x7fd61415efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:24.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.803+0000 7fd613400640 1 --2- 172.21.15.45:0/3582637688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd614155ab0 0x7fd614162ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:24.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.803+0000 7fd613e00640 1 -- 172.21.15.45:0/3582637688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd614155ab0 msgr2=0x7fd614162ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:24.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.803+0000 7fd613e00640 1 --2- 172.21.15.45:0/3582637688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd614155ab0 0x7fd614162ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:24.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.803+0000 7fd613e00640 1 -- 172.21.15.45:0/3582637688 >> v2:172.21.15.103:3300/0 conn(0x7fd6140ba610 msgr2=0x7fd61415efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:24.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.803+0000 7fd613e00640 1 --2- 172.21.15.45:0/3582637688 >> v2:172.21.15.103:3300/0 conn(0x7fd6140ba610 0x7fd61415efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:24.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.803+0000 7fd613e00640 1 -- 172.21.15.45:0/3582637688 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd61415fc20 con 0x7fd6140d18c0 2024-04-16T16:21:24.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.803+0000 7fd612a00640 1 --2- 172.21.15.45:0/3582637688 >> v2:172.21.15.103:3300/0 conn(0x7fd6140ba610 0x7fd61415efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:24.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.804+0000 7fd613e00640 1 --2- 172.21.15.45:0/3582637688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6140d18c0 0x7fd6140d1d50 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fd6140d59c0 tx=0x7fd60c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66fff65ec1712e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:24.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.805+0000 7fd612000640 1 -- 172.21.15.45:0/3582637688 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60c02c9e0 con 0x7fd6140d18c0 2024-04-16T16:21:24.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.805+0000 7fd612000640 1 -- 172.21.15.45:0/3582637688 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd60c02cb80 con 0x7fd6140d18c0 2024-04-16T16:21:24.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.805+0000 7fd612000640 1 -- 172.21.15.45:0/3582637688 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60c0343f0 con 0x7fd6140d18c0 2024-04-16T16:21:24.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.805+0000 7fd618c00640 1 -- 172.21.15.45:0/3582637688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6140d18c0 msgr2=0x7fd6140d1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:24.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.805+0000 7fd618c00640 1 --2- 172.21.15.45:0/3582637688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6140d18c0 0x7fd6140d1d50 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fd6140d59c0 tx=0x7fd60c02be40 comp rx=0 tx=0).stop 2024-04-16T16:21:24.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.806+0000 7fd618c00640 1 -- 172.21.15.45:0/3582637688 shutdown_connections 2024-04-16T16:21:24.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.806+0000 7fd618c00640 1 --2- 172.21.15.45:0/3582637688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6140d18c0 0x7fd6140d1d50 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:24.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.806+0000 7fd618c00640 1 --2- 172.21.15.45:0/3582637688 >> v2:172.21.15.103:3300/0 conn(0x7fd6140ba610 0x7fd61415efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:24.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.806+0000 7fd618c00640 1 --2- 172.21.15.45:0/3582637688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd614155ab0 0x7fd614162ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:24.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.806+0000 7fd618c00640 1 -- 172.21.15.45:0/3582637688 >> 172.21.15.45:0/3582637688 conn(0x7fd61406d1a0 msgr2=0x7fd61406d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:24.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.806+0000 7fd618c00640 1 -- 172.21.15.45:0/3582637688 shutdown_connections 2024-04-16T16:21:24.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.806+0000 7fd618c00640 1 -- 172.21.15.45:0/3582637688 wait complete. 2024-04-16T16:21:24.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.807+0000 7fd618c00640 1 Processor -- start 2024-04-16T16:21:24.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.808+0000 7fd618c00640 1 -- start start 2024-04-16T16:21:24.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.808+0000 7fd618c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6140ba610 0x7fd61414e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:24.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.809+0000 7fd618c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6140d18c0 0x7fd614158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:24.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.809+0000 7fd618c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd614155ab0 0x7fd6140ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:24.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.809+0000 7fd618c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd614163430 con 0x7fd614155ab0 2024-04-16T16:21:24.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.809+0000 7fd618c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd6141629d0 con 0x7fd6140ba610 2024-04-16T16:21:24.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.809+0000 7fd618c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd61415f510 con 0x7fd6140d18c0 2024-04-16T16:21:24.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.809+0000 7fd613e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd614155ab0 0x7fd6140ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:24.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.809+0000 7fd613e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd614155ab0 0x7fd6140ca5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58862/0 (socket says 172.21.15.45:58862) 2024-04-16T16:21:24.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.809+0000 7fd613e00640 1 -- 172.21.15.45:0/766240784 learned_addr learned my addr 172.21.15.45:0/766240784 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:24.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.809+0000 7fd613e00640 1 -- 172.21.15.45:0/766240784 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6140d18c0 msgr2=0x7fd614158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:24.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.809+0000 7fd612a00640 1 --2- 172.21.15.45:0/766240784 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6140d18c0 0x7fd614158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:24.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.809+0000 7fd613400640 1 --2- 172.21.15.45:0/766240784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6140ba610 0x7fd61414e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:24.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.809+0000 7fd613e00640 1 --2- 172.21.15.45:0/766240784 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6140d18c0 0x7fd614158b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:24.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.809+0000 7fd613e00640 1 -- 172.21.15.45:0/766240784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6140ba610 msgr2=0x7fd61414e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:24.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.809+0000 7fd613e00640 1 --2- 172.21.15.45:0/766240784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6140ba610 0x7fd61414e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:24.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.809+0000 7fd613e00640 1 -- 172.21.15.45:0/766240784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd614160850 con 0x7fd614155ab0 2024-04-16T16:21:24.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.810+0000 7fd613e00640 1 --2- 172.21.15.45:0/766240784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd614155ab0 0x7fd6140ca5b0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fd60c009f40 tx=0x7fd60c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:24.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.810+0000 7fd60b400640 1 -- 172.21.15.45:0/766240784 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60c043020 con 0x7fd614155ab0 2024-04-16T16:21:24.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.810+0000 7fd60b400640 1 -- 172.21.15.45:0/766240784 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd60c03e450 con 0x7fd614155ab0 2024-04-16T16:21:24.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.810+0000 7fd60b400640 1 -- 172.21.15.45:0/766240784 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60c03e820 con 0x7fd614155ab0 2024-04-16T16:21:24.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.810+0000 7fd618c00640 1 -- 172.21.15.45:0/766240784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd614160b10 con 0x7fd614155ab0 2024-04-16T16:21:24.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.810+0000 7fd618c00640 1 -- 172.21.15.45:0/766240784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6141610d0 con 0x7fd614155ab0 2024-04-16T16:21:24.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.811+0000 7fd618c00640 1 -- 172.21.15.45:0/766240784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd61415ef90 con 0x7fd614155ab0 2024-04-16T16:21:24.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.812+0000 7fd60b400640 1 -- 172.21.15.45:0/766240784 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7fd60c005d10 con 0x7fd614155ab0 2024-04-16T16:21:24.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.813+0000 7fd60b400640 1 --2- 172.21.15.45:0/766240784 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fd5f406ca10 0x7fd5f406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:24.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.813+0000 7fd60b400640 1 -- 172.21.15.45:0/766240784 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5227+0+0 (secure 0 0 0) 0x7fd60c0ae780 con 0x7fd614155ab0 2024-04-16T16:21:24.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.814+0000 7fd613400640 1 --2- 172.21.15.45:0/766240784 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fd5f406ca10 0x7fd5f406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:24.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.816+0000 7fd613400640 1 --2- 172.21.15.45:0/766240784 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fd5f406ca10 0x7fd5f406eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd60400b7c0 tx=0x7fd604008040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:24.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:24.819+0000 7fd60b400640 1 -- 172.21.15.45:0/766240784 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd61415ef90 con 0x7fd614155ab0 2024-04-16T16:21:25.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.023+0000 7fd618c00640 1 -- 172.21.15.45:0/766240784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd6141612d0 con 0x7fd614155ab0 2024-04-16T16:21:25.277 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:21:25.286 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:21:25.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.562+0000 7fd60b400640 1 -- 172.21.15.45:0/766240784 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v140) ==== 170+0+0 (secure 0 0 0) 0x7fd6141612d0 con 0x7fd614155ab0 2024-04-16T16:21:25.562 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2024-04-16T16:21:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.566+0000 7fd618c00640 1 -- 172.21.15.45:0/766240784 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fd5f406ca10 msgr2=0x7fd5f406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.566+0000 7fd618c00640 1 --2- 172.21.15.45:0/766240784 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fd5f406ca10 0x7fd5f406eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd60400b7c0 tx=0x7fd604008040 comp rx=0 tx=0).stop 2024-04-16T16:21:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.566+0000 7fd618c00640 1 -- 172.21.15.45:0/766240784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd614155ab0 msgr2=0x7fd6140ca5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.566+0000 7fd618c00640 1 --2- 172.21.15.45:0/766240784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd614155ab0 0x7fd6140ca5b0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fd60c009f40 tx=0x7fd60c004a40 comp rx=0 tx=0).stop 2024-04-16T16:21:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.566+0000 7fd618c00640 1 -- 172.21.15.45:0/766240784 shutdown_connections 2024-04-16T16:21:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.566+0000 7fd618c00640 1 --2- 172.21.15.45:0/766240784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd614155ab0 0x7fd6140ca5b0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.566+0000 7fd618c00640 1 --2- 172.21.15.45:0/766240784 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6140d18c0 0x7fd614158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.566+0000 7fd618c00640 1 --2- 172.21.15.45:0/766240784 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fd5f406ca10 0x7fd5f406eed0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.566+0000 7fd618c00640 1 --2- 172.21.15.45:0/766240784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6140ba610 0x7fd61414e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.566+0000 7fd618c00640 1 -- 172.21.15.45:0/766240784 >> 172.21.15.45:0/766240784 conn(0x7fd61406d1a0 msgr2=0x7fd6140d48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.567+0000 7fd618c00640 1 -- 172.21.15.45:0/766240784 shutdown_connections 2024-04-16T16:21:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.567+0000 7fd618c00640 1 -- 172.21.15.45:0/766240784 wait complete. 2024-04-16T16:21:25.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-16T16:21:25.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.796+0000 7fb09f600640 1 Processor -- start 2024-04-16T16:21:25.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.796+0000 7fb09f600640 1 -- start start 2024-04-16T16:21:25.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.798+0000 7fb09f600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0980afda0 0x7fb0980b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:25.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.798+0000 7fb09f600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0980ccb70 con 0x7fb0980d0e80 2024-04-16T16:21:25.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.798+0000 7fb09f600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0980cce50 con 0x7fb09815b920 2024-04-16T16:21:25.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.798+0000 7fb09f600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb09806e7a0 con 0x7fb0980afda0 2024-04-16T16:21:25.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.798+0000 7fb097e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb0980d0e80 0x7fb0980ce5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60508/0 (socket says 172.21.15.45:60508) 2024-04-16T16:21:25.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.798+0000 7fb097e00640 1 -- 172.21.15.45:0/2411379255 learned_addr learned my addr 172.21.15.45:0/2411379255 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:25.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb09d600640 1 --2- 172.21.15.45:0/2411379255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0980afda0 0x7fb0980b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:25.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb097400640 1 -- 172.21.15.45:0/2411379255 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 726977384 0 0) 0x7fb0980ccb70 con 0x7fb0980d0e80 2024-04-16T16:21:25.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb097400640 1 -- 172.21.15.45:0/2411379255 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb07c003520 con 0x7fb0980d0e80 2024-04-16T16:21:25.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb097400640 1 -- 172.21.15.45:0/2411379255 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3392554179 0 0) 0x7fb07c003520 con 0x7fb0980d0e80 2024-04-16T16:21:25.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb097400640 1 -- 172.21.15.45:0/2411379255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0980afda0 msgr2=0x7fb0980b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb097400640 1 --2- 172.21.15.45:0/2411379255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0980afda0 0x7fb0980b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb097400640 1 -- 172.21.15.45:0/2411379255 >> v1:172.21.15.103:6789/0 conn(0x7fb09815b920 legacy=0x7fb0980d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb097400640 1 -- 172.21.15.45:0/2411379255 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09806e430 con 0x7fb0980d0e80 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb097400640 1 -- 172.21.15.45:0/2411379255 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fb084001a10 con 0x7fb0980d0e80 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb097400640 1 -- 172.21.15.45:0/2411379255 >> v1:172.21.15.45:6789/0 conn(0x7fb0980d0e80 legacy=0x7fb0980ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb097400640 1 --2- 172.21.15.45:0/2411379255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0980afda0 0x7fb07c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb097400640 1 --2- 172.21.15.45:0/2411379255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb07c003f60 0x7fb07c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb097400640 1 --2- 172.21.15.45:0/2411379255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb07c004d40 0x7fb07c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb097400640 1 -- 172.21.15.45:0/2411379255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb09806e7a0 con 0x7fb07c003f60 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb097400640 1 -- 172.21.15.45:0/2411379255 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb07c00dcb0 con 0x7fb07c004d40 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.799+0000 7fb097400640 1 -- 172.21.15.45:0/2411379255 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb07c00ba70 con 0x7fb0980afda0 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.800+0000 7fb097e00640 1 --2- 172.21.15.45:0/2411379255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb07c003f60 0x7fb07c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.800+0000 7fb09d600640 1 --2- 172.21.15.45:0/2411379255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb07c004d40 0x7fb07c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.800+0000 7fb09cc00640 1 --2- 172.21.15.45:0/2411379255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0980afda0 0x7fb07c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.800+0000 7fb097e00640 1 -- 172.21.15.45:0/2411379255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0980afda0 msgr2=0x7fb07c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.800+0000 7fb097e00640 1 --2- 172.21.15.45:0/2411379255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0980afda0 0x7fb07c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.800+0000 7fb097e00640 1 -- 172.21.15.45:0/2411379255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb07c004d40 msgr2=0x7fb07c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.800+0000 7fb097e00640 1 --2- 172.21.15.45:0/2411379255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb07c004d40 0x7fb07c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.800+0000 7fb097e00640 1 -- 172.21.15.45:0/2411379255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07c003520 con 0x7fb07c003f60 2024-04-16T16:21:25.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.801+0000 7fb097e00640 1 --2- 172.21.15.45:0/2411379255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb07c003f60 0x7fb07c004800 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fb0980b5fb0 tx=0x7fb07c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528618c47971118c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:25.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.802+0000 7fb097400640 1 -- 172.21.15.45:0/2411379255 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08400a480 con 0x7fb07c003f60 2024-04-16T16:21:25.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.802+0000 7fb097400640 1 -- 172.21.15.45:0/2411379255 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb084034630 con 0x7fb07c003f60 2024-04-16T16:21:25.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.802+0000 7fb097400640 1 -- 172.21.15.45:0/2411379255 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08402ce60 con 0x7fb07c003f60 2024-04-16T16:21:25.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.802+0000 7fb09f600640 1 -- 172.21.15.45:0/2411379255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb07c003f60 msgr2=0x7fb07c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:25.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.802+0000 7fb09f600640 1 --2- 172.21.15.45:0/2411379255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb07c003f60 0x7fb07c004800 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fb0980b5fb0 tx=0x7fb07c005830 comp rx=0 tx=0).stop 2024-04-16T16:21:25.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.802+0000 7fb09d600640 1 -- 172.21.15.45:0/2411379255 reap_dead start 2024-04-16T16:21:25.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.802+0000 7fb09f600640 1 -- 172.21.15.45:0/2411379255 shutdown_connections 2024-04-16T16:21:25.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.802+0000 7fb09f600640 1 -- 172.21.15.45:0/2411379255 >> 172.21.15.45:0/2411379255 conn(0x7fb0980b3970 msgr2=0x7fb0980b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:25.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.802+0000 7fb09f600640 1 -- 172.21.15.45:0/2411379255 shutdown_connections 2024-04-16T16:21:25.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.802+0000 7fb09f600640 1 -- 172.21.15.45:0/2411379255 wait complete. 2024-04-16T16:21:25.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.804+0000 7fb09f600640 1 Processor -- start 2024-04-16T16:21:25.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.804+0000 7fb09f600640 1 -- start start 2024-04-16T16:21:25.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.805+0000 7fb09f600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb07c003f60 0x7fb098154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:25.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.805+0000 7fb09f600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb09815bd00 0x7fb098147790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:25.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.805+0000 7fb09f600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09815e070 0x7fb09815e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:25.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.805+0000 7fb09f600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb0980b51b0 con 0x7fb09815e070 2024-04-16T16:21:25.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.805+0000 7fb09f600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb0980b5030 con 0x7fb09815bd00 2024-04-16T16:21:25.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.805+0000 7fb09f600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb09806ec30 con 0x7fb07c003f60 2024-04-16T16:21:25.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.805+0000 7fb09d600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09815e070 0x7fb09815e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:25.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.806+0000 7fb09d600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09815e070 0x7fb09815e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58884/0 (socket says 172.21.15.45:58884) 2024-04-16T16:21:25.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.806+0000 7fb09d600640 1 -- 172.21.15.45:0/2952923712 learned_addr learned my addr 172.21.15.45:0/2952923712 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:25.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.806+0000 7fb09d600640 1 -- 172.21.15.45:0/2952923712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb07c003f60 msgr2=0x7fb098154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:25.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.806+0000 7fb09cc00640 1 --2- 172.21.15.45:0/2952923712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb07c003f60 0x7fb098154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:25.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.806+0000 7fb09d600640 1 --2- 172.21.15.45:0/2952923712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb07c003f60 0x7fb098154680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:25.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.806+0000 7fb09d600640 1 -- 172.21.15.45:0/2952923712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb09815bd00 msgr2=0x7fb098147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:25.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.806+0000 7fb09d600640 1 --2- 172.21.15.45:0/2952923712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb09815bd00 0x7fb098147790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:25.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.806+0000 7fb09d600640 1 -- 172.21.15.45:0/2952923712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0980b1030 con 0x7fb09815e070 2024-04-16T16:21:25.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.806+0000 7fb09d600640 1 --2- 172.21.15.45:0/2952923712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09815e070 0x7fb09815e4f0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fb09815f2a0 tx=0x7fb090011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:25.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.807+0000 7fb094c00640 1 -- 172.21.15.45:0/2952923712 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb090019020 con 0x7fb09815e070 2024-04-16T16:21:25.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.807+0000 7fb094c00640 1 -- 172.21.15.45:0/2952923712 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb090003610 con 0x7fb09815e070 2024-04-16T16:21:25.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.807+0000 7fb094c00640 1 -- 172.21.15.45:0/2952923712 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb090003a40 con 0x7fb09815e070 2024-04-16T16:21:25.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.807+0000 7fb09f600640 1 -- 172.21.15.45:0/2952923712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb09815c4e0 con 0x7fb09815e070 2024-04-16T16:21:25.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.807+0000 7fb09f600640 1 -- 172.21.15.45:0/2952923712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb098147cd0 con 0x7fb09815e070 2024-04-16T16:21:25.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.808+0000 7fb09f600640 1 -- 172.21.15.45:0/2952923712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb098075780 con 0x7fb09815e070 2024-04-16T16:21:25.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.810+0000 7fb094c00640 1 -- 172.21.15.45:0/2952923712 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7fb09001c020 con 0x7fb09815e070 2024-04-16T16:21:25.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.810+0000 7fb094c00640 1 --2- 172.21.15.45:0/2952923712 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb07806ca10 0x7fb07806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:25.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.811+0000 7fb094c00640 1 -- 172.21.15.45:0/2952923712 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5609+0+0 (secure 0 0 0) 0x7fb09008c3c0 con 0x7fb09815e070 2024-04-16T16:21:25.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.811+0000 7fb09cc00640 1 --2- 172.21.15.45:0/2952923712 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb07806ca10 0x7fb07806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:25.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.812+0000 7fb09cc00640 1 --2- 172.21.15.45:0/2952923712 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb07806ca10 0x7fb07806eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb088007fd0 tx=0x7fb088009040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:25.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:25.815+0000 7fb094c00640 1 -- 172.21.15.45:0/2952923712 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb090023030 con 0x7fb09815e070 2024-04-16T16:21:26.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.064+0000 7fb09f600640 1 -- 172.21.15.45:0/2952923712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb0980cffe0 con 0x7fb09815e070 2024-04-16T16:21:26.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.588+0000 7fb094c00640 1 -- 172.21.15.45:0/2952923712 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v96) ==== 174+0+0 (secure 0 0 0) 0x7fb090026390 con 0x7fb09815e070 2024-04-16T16:21:26.588 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 20 and data pool 21 2024-04-16T16:21:26.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.593+0000 7fb09f600640 1 -- 172.21.15.45:0/2952923712 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb07806ca10 msgr2=0x7fb07806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:26.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.593+0000 7fb09f600640 1 --2- 172.21.15.45:0/2952923712 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb07806ca10 0x7fb07806eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb088007fd0 tx=0x7fb088009040 comp rx=0 tx=0).stop 2024-04-16T16:21:26.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.593+0000 7fb09f600640 1 -- 172.21.15.45:0/2952923712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09815e070 msgr2=0x7fb09815e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:26.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.593+0000 7fb09f600640 1 --2- 172.21.15.45:0/2952923712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09815e070 0x7fb09815e4f0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fb09815f2a0 tx=0x7fb090011920 comp rx=0 tx=0).stop 2024-04-16T16:21:26.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.594+0000 7fb09f600640 1 -- 172.21.15.45:0/2952923712 shutdown_connections 2024-04-16T16:21:26.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.594+0000 7fb09f600640 1 --2- 172.21.15.45:0/2952923712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09815e070 0x7fb09815e4f0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:26.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.594+0000 7fb09f600640 1 --2- 172.21.15.45:0/2952923712 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fb07806ca10 0x7fb07806eed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:26.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.594+0000 7fb09f600640 1 --2- 172.21.15.45:0/2952923712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb09815bd00 0x7fb098147790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:26.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.594+0000 7fb09f600640 1 --2- 172.21.15.45:0/2952923712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb07c003f60 0x7fb098154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:26.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.594+0000 7fb09f600640 1 -- 172.21.15.45:0/2952923712 >> 172.21.15.45:0/2952923712 conn(0x7fb0980b3970 msgr2=0x7fb0980ce670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:26.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.597+0000 7fb09f600640 1 -- 172.21.15.45:0/2952923712 shutdown_connections 2024-04-16T16:21:26.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.597+0000 7fb09f600640 1 -- 172.21.15.45:0/2952923712 wait complete. 2024-04-16T16:21:26.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:21:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.827+0000 7f4a02000640 1 Processor -- start 2024-04-16T16:21:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.827+0000 7f4a02000640 1 -- start start 2024-04-16T16:21:26.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.828+0000 7f4a02000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f49fc146220 0x7f49fc15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:26.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.828+0000 7f4a02000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49fc06f7a0 0x7f49fc06df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:26.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.828+0000 7f4a02000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49fc15d360 con 0x7f49fc06e480 2024-04-16T16:21:26.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.828+0000 7f4a02000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f49fc160410 con 0x7f49fc146220 2024-04-16T16:21:26.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.828+0000 7f4a02000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f49fc15d640 con 0x7f49fc06f7a0 2024-04-16T16:21:26.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.828+0000 7f4a00c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f49fc146220 0x7f49fc15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:26.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.828+0000 7f4a01600640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f49fc06e480 0x7f49fc1608a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60518/0 (socket says 172.21.15.45:60518) 2024-04-16T16:21:26.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.828+0000 7f4a01600640 1 -- 172.21.15.45:0/3852187331 learned_addr learned my addr 172.21.15.45:0/3852187331 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:26.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.829+0000 7f49fbe00640 1 --2- 172.21.15.45:0/3852187331 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49fc06f7a0 0x7f49fc06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:26.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.829+0000 7f4a00c00640 1 -- 172.21.15.45:0/3852187331 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49fc06f7a0 msgr2=0x7f49fc06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:26.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.829+0000 7f4a00c00640 1 --2- 172.21.15.45:0/3852187331 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49fc06f7a0 0x7f49fc06df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:26.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.829+0000 7f4a00c00640 1 -- 172.21.15.45:0/3852187331 >> v1:172.21.15.45:6789/0 conn(0x7f49fc06e480 legacy=0x7f49fc1608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:26.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.829+0000 7f4a00c00640 1 -- 172.21.15.45:0/3852187331 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49fc15d920 con 0x7f49fc146220 2024-04-16T16:21:26.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.829+0000 7f49fb400640 1 -- 172.21.15.45:0/3852187331 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2974551773 0 0) 0x7f49fc15d360 con 0x7f49fc06e480 2024-04-16T16:21:26.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.829+0000 7f49fbe00640 1 --2- 172.21.15.45:0/3852187331 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49fc06f7a0 0x7f49fc06df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:26.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.830+0000 7f4a00c00640 1 --2- 172.21.15.45:0/3852187331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49fc146220 0x7f49fc15ccb0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f49f00097a0 tx=0x7f49f002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab8e87968697f22b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:26.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.830+0000 7f49fb400640 1 -- 172.21.15.45:0/3852187331 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0035020 con 0x7f49fc146220 2024-04-16T16:21:26.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.830+0000 7f49fb400640 1 -- 172.21.15.45:0/3852187331 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49f0031810 con 0x7f49fc146220 2024-04-16T16:21:26.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.830+0000 7f49fb400640 1 -- 172.21.15.45:0/3852187331 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0031be0 con 0x7f49fc146220 2024-04-16T16:21:26.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.831+0000 7f4a02000640 1 -- 172.21.15.45:0/3852187331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49fc146220 msgr2=0x7f49fc15ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:26.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.831+0000 7f4a02000640 1 --2- 172.21.15.45:0/3852187331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49fc146220 0x7f49fc15ccb0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f49f00097a0 tx=0x7f49f002b8b0 comp rx=0 tx=0).stop 2024-04-16T16:21:26.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.831+0000 7f4a02000640 1 -- 172.21.15.45:0/3852187331 shutdown_connections 2024-04-16T16:21:26.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.831+0000 7f4a02000640 1 --2- 172.21.15.45:0/3852187331 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49fc06f7a0 0x7f49fc06df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:26.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.831+0000 7f4a02000640 1 --2- 172.21.15.45:0/3852187331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49fc146220 0x7f49fc15ccb0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:26.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.831+0000 7f4a02000640 1 -- 172.21.15.45:0/3852187331 >> 172.21.15.45:0/3852187331 conn(0x7f49fc06b930 msgr2=0x7f49fc06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:26.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.831+0000 7f4a02000640 1 -- 172.21.15.45:0/3852187331 shutdown_connections 2024-04-16T16:21:26.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.831+0000 7f4a02000640 1 -- 172.21.15.45:0/3852187331 wait complete. 2024-04-16T16:21:26.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.832+0000 7f4a02000640 1 Processor -- start 2024-04-16T16:21:26.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.833+0000 7f4a02000640 1 -- start start 2024-04-16T16:21:26.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.833+0000 7f4a02000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49fc06e480 0x7f49fc15c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:26.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.833+0000 7f4a02000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49fc06f7a0 0x7f49fc0cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:26.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.834+0000 7f4a02000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49fc146220 0x7f49fc0cea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:26.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.834+0000 7f4a02000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc06fda0 con 0x7f49fc06f7a0 2024-04-16T16:21:26.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.834+0000 7f4a02000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc06fc20 con 0x7f49fc06e480 2024-04-16T16:21:26.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.834+0000 7f4a02000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f49fc160fb0 con 0x7f49fc146220 2024-04-16T16:21:26.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.834+0000 7f49fbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49fc06f7a0 0x7f49fc0cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:26.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.834+0000 7f49fbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49fc06f7a0 0x7f49fc0cc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58892/0 (socket says 172.21.15.45:58892) 2024-04-16T16:21:26.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.834+0000 7f49fbe00640 1 -- 172.21.15.45:0/3439531869 learned_addr learned my addr 172.21.15.45:0/3439531869 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:26.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.834+0000 7f4a00c00640 1 --2- 172.21.15.45:0/3439531869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49fc06e480 0x7f49fc15c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:26.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.834+0000 7f4a01600640 1 --2- 172.21.15.45:0/3439531869 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49fc146220 0x7f49fc0cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:26.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.834+0000 7f49fbe00640 1 -- 172.21.15.45:0/3439531869 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49fc146220 msgr2=0x7f49fc0cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:26.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.834+0000 7f49fbe00640 1 --2- 172.21.15.45:0/3439531869 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49fc146220 0x7f49fc0cea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:26.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.834+0000 7f49fbe00640 1 -- 172.21.15.45:0/3439531869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49fc06e480 msgr2=0x7f49fc15c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:26.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.834+0000 7f49fbe00640 1 --2- 172.21.15.45:0/3439531869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49fc06e480 0x7f49fc15c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:26.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.834+0000 7f49fbe00640 1 -- 172.21.15.45:0/3439531869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49fc15caf0 con 0x7f49fc06f7a0 2024-04-16T16:21:26.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.835+0000 7f49fbe00640 1 --2- 172.21.15.45:0/3439531869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49fc06f7a0 0x7f49fc0cc0f0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f49e800c7f0 tx=0x7f49e800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:26.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.835+0000 7f49f8c00640 1 -- 172.21.15.45:0/3439531869 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e8015070 con 0x7f49fc06f7a0 2024-04-16T16:21:26.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.835+0000 7f49f8c00640 1 -- 172.21.15.45:0/3439531869 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49e80079d0 con 0x7f49fc06f7a0 2024-04-16T16:21:26.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.835+0000 7f49f8c00640 1 -- 172.21.15.45:0/3439531869 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e8007da0 con 0x7f49fc06f7a0 2024-04-16T16:21:26.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.835+0000 7f4a02000640 1 -- 172.21.15.45:0/3439531869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49fc1f4660 con 0x7f49fc06f7a0 2024-04-16T16:21:26.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.835+0000 7f4a02000640 1 -- 172.21.15.45:0/3439531869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49fc1f4c20 con 0x7f49fc06f7a0 2024-04-16T16:21:26.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.836+0000 7f4a02000640 1 -- 172.21.15.45:0/3439531869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49fc075780 con 0x7f49fc06f7a0 2024-04-16T16:21:26.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.838+0000 7f49f8c00640 1 -- 172.21.15.45:0/3439531869 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f49e8018070 con 0x7f49fc06f7a0 2024-04-16T16:21:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.838+0000 7f49f8c00640 1 --2- 172.21.15.45:0/3439531869 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f49dc06ca10 0x7f49dc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.838+0000 7f49f8c00640 1 -- 172.21.15.45:0/3439531869 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5709+0+0 (secure 0 0 0) 0x7f49e808bdd0 con 0x7f49fc06f7a0 2024-04-16T16:21:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.839+0000 7f4a00c00640 1 --2- 172.21.15.45:0/3439531869 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f49dc06ca10 0x7f49dc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:26.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.840+0000 7f4a00c00640 1 --2- 172.21.15.45:0/3439531869 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f49dc06ca10 0x7f49dc06eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f49f0036000 tx=0x7f49f003c040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:26.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:26.843+0000 7f49f8c00640 1 -- 172.21.15.45:0/3439531869 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f49e8059f40 con 0x7f49fc06f7a0 2024-04-16T16:21:27.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.044+0000 7f4a02000640 1 -- 172.21.15.45:0/3439531869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49fc1f43a0 con 0x7f49fc06f7a0 2024-04-16T16:21:27.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.046+0000 7f49f8c00640 1 -- 172.21.15.45:0/3439531869 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) ==== 74+0+16692 (secure 0 0 0) 0x7f49e8059840 con 0x7f49fc06f7a0 2024-04-16T16:21:27.047 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:27.047 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":141,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:21:26.066901+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:21:23.973149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-16T16:21:25.043255+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:21:27.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.049+0000 7f4a02000640 1 -- 172.21.15.45:0/3439531869 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f49dc06ca10 msgr2=0x7f49dc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:27.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.049+0000 7f4a02000640 1 --2- 172.21.15.45:0/3439531869 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f49dc06ca10 0x7f49dc06eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f49f0036000 tx=0x7f49f003c040 comp rx=0 tx=0).stop 2024-04-16T16:21:27.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.049+0000 7f4a02000640 1 -- 172.21.15.45:0/3439531869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49fc06f7a0 msgr2=0x7f49fc0cc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:27.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.049+0000 7f4a02000640 1 --2- 172.21.15.45:0/3439531869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49fc06f7a0 0x7f49fc0cc0f0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f49e800c7f0 tx=0x7f49e800cbf0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.049+0000 7f4a02000640 1 -- 172.21.15.45:0/3439531869 shutdown_connections 2024-04-16T16:21:27.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.049+0000 7f4a02000640 1 --2- 172.21.15.45:0/3439531869 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49fc146220 0x7f49fc0cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.049+0000 7f4a02000640 1 --2- 172.21.15.45:0/3439531869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49fc06f7a0 0x7f49fc0cc0f0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.049+0000 7f4a02000640 1 --2- 172.21.15.45:0/3439531869 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f49dc06ca10 0x7f49dc06eed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.049+0000 7f4a02000640 1 --2- 172.21.15.45:0/3439531869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49fc06e480 0x7f49fc15c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.049+0000 7f4a02000640 1 -- 172.21.15.45:0/3439531869 >> 172.21.15.45:0/3439531869 conn(0x7f49fc06b930 msgr2=0x7f49fc15e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:27.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.049+0000 7f4a02000640 1 -- 172.21.15.45:0/3439531869 shutdown_connections 2024-04-16T16:21:27.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.049+0000 7f4a02000640 1 -- 172.21.15.45:0/3439531869 wait complete. 2024-04-16T16:21:27.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:21:27.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.285+0000 7f82ff200640 1 Processor -- start 2024-04-16T16:21:27.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.285+0000 7f82ff200640 1 -- start start 2024-04-16T16:21:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.287+0000 7f82ff200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f83000b2de0 0x7f830006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.287+0000 7f82ff200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f830015c580 con 0x7f8300158a40 2024-04-16T16:21:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.287+0000 7f82ff200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83000cfff0 con 0x7f830006ed10 2024-04-16T16:21:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.287+0000 7f82ff200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f830015c890 con 0x7f83000b2de0 2024-04-16T16:21:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.287+0000 7f82fde00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8300158a40 0x7f830015be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60530/0 (socket says 172.21.15.45:60530) 2024-04-16T16:21:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.287+0000 7f82fde00640 1 -- 172.21.15.45:0/4169666014 learned_addr learned my addr 172.21.15.45:0/4169666014 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:27.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fd400640 1 --2- 172.21.15.45:0/4169666014 >> v2:172.21.15.103:3300/0 conn(0x7f83000b2de0 0x7f830006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:27.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fca00640 1 -- 172.21.15.45:0/4169666014 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1784617950 0 0) 0x7f82f0001110 con 0x7f8300158a40 2024-04-16T16:21:27.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fca00640 1 -- 172.21.15.45:0/4169666014 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82e4005730 con 0x7f8300158a40 2024-04-16T16:21:27.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fca00640 1 -- 172.21.15.45:0/4169666014 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1124298000 0 0) 0x7f82e4005730 con 0x7f8300158a40 2024-04-16T16:21:27.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fca00640 1 -- 172.21.15.45:0/4169666014 >> v2:172.21.15.103:3300/0 conn(0x7f83000b2de0 msgr2=0x7f830006e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:27.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fca00640 1 --2- 172.21.15.45:0/4169666014 >> v2:172.21.15.103:3300/0 conn(0x7f83000b2de0 0x7f830006e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fca00640 1 -- 172.21.15.45:0/4169666014 >> v1:172.21.15.103:6789/0 conn(0x7f830006ed10 legacy=0x7f830015cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:27.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fca00640 1 -- 172.21.15.45:0/4169666014 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83000d02d0 con 0x7f8300158a40 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fca00640 1 -- 172.21.15.45:0/4169666014 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f82f0001840 con 0x7f8300158a40 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fca00640 1 -- 172.21.15.45:0/4169666014 >> v1:172.21.15.45:6789/0 conn(0x7f8300158a40 legacy=0x7f830015be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fca00640 1 --2- 172.21.15.45:0/4169666014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82e4005bc0 0x7f82e4005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fca00640 1 --2- 172.21.15.45:0/4169666014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82e4006a10 0x7f82e4006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fca00640 1 --2- 172.21.15.45:0/4169666014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82e40073d0 0x7f82e400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fca00640 1 -- 172.21.15.45:0/4169666014 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f830015c890 con 0x7f82e4006a10 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fca00640 1 -- 172.21.15.45:0/4169666014 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f82e400e270 con 0x7f82e4005bc0 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.288+0000 7f82fca00640 1 -- 172.21.15.45:0/4169666014 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f82e400e470 con 0x7f82e40073d0 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.289+0000 7f82fd400640 1 --2- 172.21.15.45:0/4169666014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82e4006a10 0x7f82e4006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.289+0000 7f82fd400640 1 -- 172.21.15.45:0/4169666014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82e40073d0 msgr2=0x7f82e400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.289+0000 7f82fe800640 1 --2- 172.21.15.45:0/4169666014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82e40073d0 0x7f82e400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.289+0000 7f82fde00640 1 --2- 172.21.15.45:0/4169666014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82e4005bc0 0x7f82e4005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.289+0000 7f82fd400640 1 --2- 172.21.15.45:0/4169666014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82e40073d0 0x7f82e400dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.289+0000 7f82fd400640 1 -- 172.21.15.45:0/4169666014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82e4005bc0 msgr2=0x7f82e4005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.289+0000 7f82fd400640 1 --2- 172.21.15.45:0/4169666014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82e4005bc0 0x7f82e4005fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.289+0000 7f82fd400640 1 -- 172.21.15.45:0/4169666014 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e4005730 con 0x7f82e4006a10 2024-04-16T16:21:27.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.289+0000 7f82fe800640 1 -- 172.21.15.45:0/4169666014 reap_dead start 2024-04-16T16:21:27.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.290+0000 7f82fd400640 1 --2- 172.21.15.45:0/4169666014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82e4006a10 0x7f82e4006e90 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009ac0 tx=0x7f82e4007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9ee933174628fe server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:27.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.291+0000 7f82fca00640 1 -- 172.21.15.45:0/4169666014 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec039070 con 0x7f82e4006a10 2024-04-16T16:21:27.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.291+0000 7f82fca00640 1 -- 172.21.15.45:0/4169666014 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82ec02c6b0 con 0x7f82e4006a10 2024-04-16T16:21:27.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.291+0000 7f82fca00640 1 -- 172.21.15.45:0/4169666014 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec009690 con 0x7f82e4006a10 2024-04-16T16:21:27.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.291+0000 7f82ff200640 1 -- 172.21.15.45:0/4169666014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82e4006a10 msgr2=0x7f82e4006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:27.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.291+0000 7f82ff200640 1 --2- 172.21.15.45:0/4169666014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82e4006a10 0x7f82e4006e90 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009ac0 tx=0x7f82e4007a40 comp rx=0 tx=0).stop 2024-04-16T16:21:27.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.291+0000 7f82ff200640 1 -- 172.21.15.45:0/4169666014 shutdown_connections 2024-04-16T16:21:27.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.291+0000 7f82ff200640 1 --2- 172.21.15.45:0/4169666014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82e4006a10 0x7f82e4006e90 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.291+0000 7f82ff200640 1 -- 172.21.15.45:0/4169666014 >> 172.21.15.45:0/4169666014 conn(0x7f830006b930 msgr2=0x7f830006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:27.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.291+0000 7f82ff200640 1 -- 172.21.15.45:0/4169666014 shutdown_connections 2024-04-16T16:21:27.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.291+0000 7f82ff200640 1 -- 172.21.15.45:0/4169666014 wait complete. 2024-04-16T16:21:27.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.293+0000 7f82ff200640 1 Processor -- start 2024-04-16T16:21:27.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.293+0000 7f82ff200640 1 -- start start 2024-04-16T16:21:27.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.294+0000 7f82ff200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82e4006a10 0x7f83001544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:27.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.294+0000 7f82ff200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83000cbe10 0x7f830014aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:27.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.294+0000 7f82ff200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83000cc290 0x7f830014b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:27.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.294+0000 7f82ff200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f830015c200 con 0x7f82e4006a10 2024-04-16T16:21:27.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.294+0000 7f82ff200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f830015c080 con 0x7f83000cbe10 2024-04-16T16:21:27.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.294+0000 7f82ff200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f830015c380 con 0x7f83000cc290 2024-04-16T16:21:27.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.295+0000 7f82fde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82e4006a10 0x7f83001544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:27.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.295+0000 7f82fde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82e4006a10 0x7f83001544b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58904/0 (socket says 172.21.15.45:58904) 2024-04-16T16:21:27.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.295+0000 7f82fde00640 1 -- 172.21.15.45:0/2317397623 learned_addr learned my addr 172.21.15.45:0/2317397623 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:27.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.295+0000 7f82fde00640 1 -- 172.21.15.45:0/2317397623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83000cc290 msgr2=0x7f830014b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:27.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.295+0000 7f82fde00640 1 --2- 172.21.15.45:0/2317397623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83000cc290 0x7f830014b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.295+0000 7f82fde00640 1 -- 172.21.15.45:0/2317397623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83000cbe10 msgr2=0x7f830014aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:27.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.295+0000 7f82fd400640 1 --2- 172.21.15.45:0/2317397623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83000cbe10 0x7f830014aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:27.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.295+0000 7f82fde00640 1 --2- 172.21.15.45:0/2317397623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83000cbe10 0x7f830014aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.295+0000 7f82fde00640 1 -- 172.21.15.45:0/2317397623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83001f8f20 con 0x7f82e4006a10 2024-04-16T16:21:27.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.295+0000 7f82fd400640 1 --2- 172.21.15.45:0/2317397623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83000cbe10 0x7f830014aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:27.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.295+0000 7f82fde00640 1 --2- 172.21.15.45:0/2317397623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82e4006a10 0x7f83001544b0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f82f000d7a0 tx=0x7f82f000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:27.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.296+0000 7f82f6000640 1 -- 172.21.15.45:0/2317397623 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f0019070 con 0x7f82e4006a10 2024-04-16T16:21:27.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.296+0000 7f82f6000640 1 -- 172.21.15.45:0/2317397623 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f00074e0 con 0x7f82e4006a10 2024-04-16T16:21:27.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.296+0000 7f82ff200640 1 -- 172.21.15.45:0/2317397623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83001539d0 con 0x7f82e4006a10 2024-04-16T16:21:27.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.296+0000 7f82f6000640 1 -- 172.21.15.45:0/2317397623 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f00049e0 con 0x7f82e4006a10 2024-04-16T16:21:27.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.296+0000 7f82ff200640 1 -- 172.21.15.45:0/2317397623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8300153f90 con 0x7f82e4006a10 2024-04-16T16:21:27.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.297+0000 7f82ff200640 1 -- 172.21.15.45:0/2317397623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8300068400 con 0x7f82e4006a10 2024-04-16T16:21:27.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.298+0000 7f82f6000640 1 -- 172.21.15.45:0/2317397623 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f82f001c070 con 0x7f82e4006a10 2024-04-16T16:21:27.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.299+0000 7f82f6000640 1 --2- 172.21.15.45:0/2317397623 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f82e006ca40 0x7f82e006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:27.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.299+0000 7f82f6000640 1 -- 172.21.15.45:0/2317397623 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5709+0+0 (secure 0 0 0) 0x7f82f008b810 con 0x7f82e4006a10 2024-04-16T16:21:27.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.299+0000 7f82fd400640 1 --2- 172.21.15.45:0/2317397623 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f82e006ca40 0x7f82e006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:27.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.301+0000 7f82fd400640 1 --2- 172.21.15.45:0/2317397623 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f82e006ca40 0x7f82e006ef00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f82ec036000 tx=0x7f82ec030000 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:27.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.303+0000 7f82f6000640 1 -- 172.21.15.45:0/2317397623 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8300068400 con 0x7f82e4006a10 2024-04-16T16:21:27.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.507+0000 7f82ff200640 1 -- 172.21.15.45:0/2317397623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8300075780 con 0x7f82e4006a10 2024-04-16T16:21:27.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.509+0000 7f82f6000640 1 -- 172.21.15.45:0/2317397623 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) ==== 74+0+16692 (secure 0 0 0) 0x7f830015c580 con 0x7f82e4006a10 2024-04-16T16:21:27.509 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:27.509 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":141,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:21:26.066901+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:21:23.973149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-16T16:21:25.043255+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:21:27.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.512+0000 7f82ff200640 1 -- 172.21.15.45:0/2317397623 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f82e006ca40 msgr2=0x7f82e006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.512+0000 7f82ff200640 1 --2- 172.21.15.45:0/2317397623 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f82e006ca40 0x7f82e006ef00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f82ec036000 tx=0x7f82ec030000 comp rx=0 tx=0).stop 2024-04-16T16:21:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.512+0000 7f82ff200640 1 -- 172.21.15.45:0/2317397623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82e4006a10 msgr2=0x7f83001544b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.512+0000 7f82ff200640 1 --2- 172.21.15.45:0/2317397623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82e4006a10 0x7f83001544b0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f82f000d7a0 tx=0x7f82f000dc60 comp rx=0 tx=0).stop 2024-04-16T16:21:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.512+0000 7f82ff200640 1 -- 172.21.15.45:0/2317397623 shutdown_connections 2024-04-16T16:21:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.512+0000 7f82ff200640 1 --2- 172.21.15.45:0/2317397623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f83000cc290 0x7f830014b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.512+0000 7f82ff200640 1 --2- 172.21.15.45:0/2317397623 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f82e006ca40 0x7f82e006ef00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.512+0000 7f82ff200640 1 --2- 172.21.15.45:0/2317397623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83000cbe10 0x7f830014aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.512+0000 7f82ff200640 1 --2- 172.21.15.45:0/2317397623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82e4006a10 0x7f83001544b0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.512+0000 7f82ff200640 1 -- 172.21.15.45:0/2317397623 >> 172.21.15.45:0/2317397623 conn(0x7f830006b930 msgr2=0x7f830006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.512+0000 7f82ff200640 1 -- 172.21.15.45:0/2317397623 shutdown_connections 2024-04-16T16:21:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.513+0000 7f82ff200640 1 -- 172.21.15.45:0/2317397623 wait complete. 2024-04-16T16:21:27.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-16T16:21:27.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.734+0000 7f6719800640 1 Processor -- start 2024-04-16T16:21:27.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.734+0000 7f6719800640 1 -- start start 2024-04-16T16:21:27.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.735+0000 7f6719800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6714160070 0x7f67141605e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:27.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.735+0000 7f6719800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f67140cee10 0x7f6714160b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:27.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.735+0000 7f6719800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67140b3980 0x7f67140cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:27.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.735+0000 7f6719800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f67140d0ba0 con 0x7f67140cee10 2024-04-16T16:21:27.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.735+0000 7f6719800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f67141611b0 con 0x7f6714160070 2024-04-16T16:21:27.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.735+0000 7f6719800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f67140d0550 con 0x7f67140b3980 2024-04-16T16:21:27.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.736+0000 7f6712000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f67140cee10 0x7f6714160b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:27.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.736+0000 7f6712000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f67140cee10 0x7f6714160b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58918/0 (socket says 172.21.15.45:58918) 2024-04-16T16:21:27.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.736+0000 7f6712000640 1 -- 172.21.15.45:0/845279097 learned_addr learned my addr 172.21.15.45:0/845279097 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.736+0000 7f6712000640 1 -- 172.21.15.45:0/845279097 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67140b3980 msgr2=0x7f67140cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.736+0000 7f6712000640 1 --2- 172.21.15.45:0/845279097 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67140b3980 0x7f67140cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.736+0000 7f6712000640 1 -- 172.21.15.45:0/845279097 >> v2:172.21.15.103:3300/0 conn(0x7f6714160070 msgr2=0x7f67141605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.736+0000 7f6712000640 1 --2- 172.21.15.45:0/845279097 >> v2:172.21.15.103:3300/0 conn(0x7f6714160070 0x7f67141605e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.736+0000 7f6712000640 1 -- 172.21.15.45:0/845279097 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67140d0830 con 0x7f67140cee10 2024-04-16T16:21:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.737+0000 7f6712000640 1 --2- 172.21.15.45:0/845279097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67140cee10 0x7f6714160b20 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f67140734e0 tx=0x7f66fc02bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ce7c7bf765d50d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.738+0000 7f6711600640 1 -- 172.21.15.45:0/845279097 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66fc02caa0 con 0x7f67140cee10 2024-04-16T16:21:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.738+0000 7f6711600640 1 -- 172.21.15.45:0/845279097 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66fc02cc40 con 0x7f67140cee10 2024-04-16T16:21:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.738+0000 7f6711600640 1 -- 172.21.15.45:0/845279097 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66fc0343f0 con 0x7f67140cee10 2024-04-16T16:21:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.738+0000 7f6719800640 1 -- 172.21.15.45:0/845279097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67140cee10 msgr2=0x7f6714160b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.738+0000 7f6719800640 1 --2- 172.21.15.45:0/845279097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67140cee10 0x7f6714160b20 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f67140734e0 tx=0x7f66fc02bf00 comp rx=0 tx=0).stop 2024-04-16T16:21:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.738+0000 7f6719800640 1 -- 172.21.15.45:0/845279097 shutdown_connections 2024-04-16T16:21:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.738+0000 7f6719800640 1 --2- 172.21.15.45:0/845279097 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67140b3980 0x7f67140cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.739+0000 7f6719800640 1 --2- 172.21.15.45:0/845279097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67140cee10 0x7f6714160b20 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.739+0000 7f6719800640 1 --2- 172.21.15.45:0/845279097 >> v2:172.21.15.103:3300/0 conn(0x7f6714160070 0x7f67141605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.739+0000 7f6719800640 1 -- 172.21.15.45:0/845279097 >> 172.21.15.45:0/845279097 conn(0x7f67140b42d0 msgr2=0x7f67140b46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.739+0000 7f6719800640 1 -- 172.21.15.45:0/845279097 shutdown_connections 2024-04-16T16:21:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.739+0000 7f6719800640 1 -- 172.21.15.45:0/845279097 wait complete. 2024-04-16T16:21:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.741+0000 7f6719800640 1 Processor -- start 2024-04-16T16:21:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.741+0000 7f6719800640 1 -- start start 2024-04-16T16:21:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.742+0000 7f6719800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67140b3980 0x7f67140b8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.742+0000 7f6719800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67140cee10 0x7f6714150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.742+0000 7f6719800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6714160070 0x7f67140cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.742+0000 7f6719800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f67140cdc90 con 0x7f6714160070 2024-04-16T16:21:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.742+0000 7f6719800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f67140cdb10 con 0x7f67140cee10 2024-04-16T16:21:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.742+0000 7f6719800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f67140cde10 con 0x7f67140b3980 2024-04-16T16:21:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.742+0000 7f6713400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6714160070 0x7f67140cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.742+0000 7f6713400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6714160070 0x7f67140cc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58928/0 (socket says 172.21.15.45:58928) 2024-04-16T16:21:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.742+0000 7f6713400640 1 -- 172.21.15.45:0/1884113084 learned_addr learned my addr 172.21.15.45:0/1884113084 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.743+0000 7f6712000640 1 --2- 172.21.15.45:0/1884113084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67140cee10 0x7f6714150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.743+0000 7f6712a00640 1 --2- 172.21.15.45:0/1884113084 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67140b3980 0x7f67140b8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.743+0000 7f6713400640 1 -- 172.21.15.45:0/1884113084 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67140b3980 msgr2=0x7f67140b8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.743+0000 7f6713400640 1 --2- 172.21.15.45:0/1884113084 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67140b3980 0x7f67140b8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.743+0000 7f6713400640 1 -- 172.21.15.45:0/1884113084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67140cee10 msgr2=0x7f6714150370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:27.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.743+0000 7f6713400640 1 --2- 172.21.15.45:0/1884113084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67140cee10 0x7f6714150370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:27.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.743+0000 7f6713400640 1 -- 172.21.15.45:0/1884113084 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67140cb620 con 0x7f6714160070 2024-04-16T16:21:27.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.743+0000 7f6712000640 1 --2- 172.21.15.45:0/1884113084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67140cee10 0x7f6714150370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:27.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.743+0000 7f6713400640 1 --2- 172.21.15.45:0/1884113084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6714160070 0x7f67140cc6c0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f670c00b3c0 tx=0x7f670c00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.743+0000 7f670aa00640 1 -- 172.21.15.45:0/1884113084 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c011830 con 0x7f6714160070 2024-04-16T16:21:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.744+0000 7f670aa00640 1 -- 172.21.15.45:0/1884113084 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f670c016070 con 0x7f6714160070 2024-04-16T16:21:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.744+0000 7f670aa00640 1 -- 172.21.15.45:0/1884113084 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c011c30 con 0x7f6714160070 2024-04-16T16:21:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.744+0000 7f6719800640 1 -- 172.21.15.45:0/1884113084 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67140cb900 con 0x7f6714160070 2024-04-16T16:21:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.744+0000 7f6719800640 1 -- 172.21.15.45:0/1884113084 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67140cbec0 con 0x7f6714160070 2024-04-16T16:21:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.745+0000 7f6719800640 1 -- 172.21.15.45:0/1884113084 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6714075780 con 0x7f6714160070 2024-04-16T16:21:27.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.746+0000 7f670aa00640 1 -- 172.21.15.45:0/1884113084 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f670c01b460 con 0x7f6714160070 2024-04-16T16:21:27.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.747+0000 7f670aa00640 1 --2- 172.21.15.45:0/1884113084 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f66f006ca10 0x7f66f006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:27.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.748+0000 7f670aa00640 1 -- 172.21.15.45:0/1884113084 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7f670c08ba90 con 0x7f6714160070 2024-04-16T16:21:27.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.748+0000 7f6712a00640 1 --2- 172.21.15.45:0/1884113084 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f66f006ca10 0x7f66f006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:27.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.749+0000 7f6712a00640 1 --2- 172.21.15.45:0/1884113084 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f66f006ca10 0x7f66f006eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f670400b810 tx=0x7f6704008040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:27.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.752+0000 7f670aa00640 1 -- 172.21.15.45:0/1884113084 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f670c059d10 con 0x7f6714160070 2024-04-16T16:21:27.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:27.992+0000 7f6719800640 1 -- 172.21.15.45:0/1884113084 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f67140b50c0 con 0x7f6714160070 2024-04-16T16:21:28.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.631+0000 7f670aa00640 1 -- 172.21.15.45:0/1884113084 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v99) ==== 118+0+0 (secure 0 0 0) 0x7f670c059610 con 0x7f6714160070 2024-04-16T16:21:28.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.635+0000 7f6719800640 1 -- 172.21.15.45:0/1884113084 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f66f006ca10 msgr2=0x7f66f006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:28.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.635+0000 7f6719800640 1 --2- 172.21.15.45:0/1884113084 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f66f006ca10 0x7f66f006eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f670400b810 tx=0x7f6704008040 comp rx=0 tx=0).stop 2024-04-16T16:21:28.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.635+0000 7f6719800640 1 -- 172.21.15.45:0/1884113084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6714160070 msgr2=0x7f67140cc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:28.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.635+0000 7f6719800640 1 --2- 172.21.15.45:0/1884113084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6714160070 0x7f67140cc6c0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f670c00b3c0 tx=0x7f670c00b880 comp rx=0 tx=0).stop 2024-04-16T16:21:28.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.635+0000 7f6719800640 1 -- 172.21.15.45:0/1884113084 shutdown_connections 2024-04-16T16:21:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.635+0000 7f6719800640 1 --2- 172.21.15.45:0/1884113084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6714160070 0x7f67140cc6c0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.635+0000 7f6719800640 1 --2- 172.21.15.45:0/1884113084 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f66f006ca10 0x7f66f006eed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.635+0000 7f6719800640 1 --2- 172.21.15.45:0/1884113084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67140cee10 0x7f6714150370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.635+0000 7f6719800640 1 --2- 172.21.15.45:0/1884113084 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67140b3980 0x7f67140b8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.635+0000 7f6719800640 1 -- 172.21.15.45:0/1884113084 >> 172.21.15.45:0/1884113084 conn(0x7f67140b42d0 msgr2=0x7f67140704b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.635+0000 7f6719800640 1 -- 172.21.15.45:0/1884113084 shutdown_connections 2024-04-16T16:21:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.636+0000 7f6719800640 1 -- 172.21.15.45:0/1884113084 wait complete. 2024-04-16T16:21:28.662 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-16T16:21:28.662 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-16T16:21:28.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.890+0000 7f1cc7200640 1 Processor -- start 2024-04-16T16:21:28.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.890+0000 7f1cc7200640 1 -- start start 2024-04-16T16:21:28.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.891+0000 7f1cc7200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1cc81461b0 0x7f1cc8160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:28.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.891+0000 7f1cc7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cc80b5d50 0x7f1cc80cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:28.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.891+0000 7f1cc7200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cc8146590 con 0x7f1cc80b58f0 2024-04-16T16:21:28.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.891+0000 7f1cc7200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1cc806dfd0 con 0x7f1cc81461b0 2024-04-16T16:21:28.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.891+0000 7f1cc7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1cc80c5750 con 0x7f1cc80b5d50 2024-04-16T16:21:28.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.892+0000 7f1cc6800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cc80b5d50 0x7f1cc80cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:28.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.892+0000 7f1cc6800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cc80b5d50 0x7f1cc80cfb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:47980/0 (socket says 172.21.15.45:47980) 2024-04-16T16:21:28.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.892+0000 7f1cc6800640 1 -- 172.21.15.45:0/3446666959 learned_addr learned my addr 172.21.15.45:0/3446666959 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:28.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.892+0000 7f1cc5e00640 1 --2- 172.21.15.45:0/3446666959 >> v2:172.21.15.103:3300/0 conn(0x7f1cc81461b0 0x7f1cc8160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:28.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.893+0000 7f1cc4a00640 1 -- 172.21.15.45:0/3446666959 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 145121178 0 0) 0x7f1cc8146590 con 0x7f1cc80b58f0 2024-04-16T16:21:28.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.894+0000 7f1cc4a00640 1 -- 172.21.15.45:0/3446666959 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cac003520 con 0x7f1cc80b58f0 2024-04-16T16:21:28.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.894+0000 7f1cc5e00640 1 -- 172.21.15.45:0/3446666959 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cc80b5d50 msgr2=0x7f1cc80cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:28.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.894+0000 7f1cc5e00640 1 --2- 172.21.15.45:0/3446666959 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cc80b5d50 0x7f1cc80cfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:28.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.894+0000 7f1cc5e00640 1 -- 172.21.15.45:0/3446666959 >> v1:172.21.15.103:6789/0 conn(0x7f1cc80b58f0 legacy=0x7f1cc80cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:28.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.894+0000 7f1cc5e00640 1 -- 172.21.15.45:0/3446666959 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc8161230 con 0x7f1cc81461b0 2024-04-16T16:21:28.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.894+0000 7f1cc6800640 1 --2- 172.21.15.45:0/3446666959 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cc80b5d50 0x7f1cc80cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:21:28.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.895+0000 7f1cc5e00640 1 --2- 172.21.15.45:0/3446666959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cc81461b0 0x7f1cc8160cf0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80097a0 tx=0x7f1cb802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f784b91677210f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:28.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.896+0000 7f1cc4a00640 1 -- 172.21.15.45:0/3446666959 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8035020 con 0x7f1cc81461b0 2024-04-16T16:21:28.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.896+0000 7f1cc4a00640 1 -- 172.21.15.45:0/3446666959 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cb8031810 con 0x7f1cc81461b0 2024-04-16T16:21:28.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.896+0000 7f1cc4a00640 1 -- 172.21.15.45:0/3446666959 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8031bc0 con 0x7f1cc81461b0 2024-04-16T16:21:28.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.896+0000 7f1cc7200640 1 -- 172.21.15.45:0/3446666959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cc81461b0 msgr2=0x7f1cc8160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:28.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.896+0000 7f1cc7200640 1 --2- 172.21.15.45:0/3446666959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cc81461b0 0x7f1cc8160cf0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80097a0 tx=0x7f1cb802b8b0 comp rx=0 tx=0).stop 2024-04-16T16:21:28.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.896+0000 7f1cc7200640 1 -- 172.21.15.45:0/3446666959 shutdown_connections 2024-04-16T16:21:28.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.896+0000 7f1cc7200640 1 --2- 172.21.15.45:0/3446666959 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cc80b5d50 0x7f1cc80cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:28.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.896+0000 7f1cc7200640 1 --2- 172.21.15.45:0/3446666959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cc81461b0 0x7f1cc8160cf0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:28.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.896+0000 7f1cc7200640 1 -- 172.21.15.45:0/3446666959 >> 172.21.15.45:0/3446666959 conn(0x7f1cc806b930 msgr2=0x7f1cc806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:28.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.896+0000 7f1cc7200640 1 -- 172.21.15.45:0/3446666959 shutdown_connections 2024-04-16T16:21:28.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.896+0000 7f1cc7200640 1 -- 172.21.15.45:0/3446666959 wait complete. 2024-04-16T16:21:28.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.898+0000 7f1cc7200640 1 Processor -- start 2024-04-16T16:21:28.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.898+0000 7f1cc7200640 1 -- start start 2024-04-16T16:21:28.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.899+0000 7f1cc7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cc80b58f0 0x7f1cc80ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:28.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.899+0000 7f1cc7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cc80b5d50 0x7f1cc80cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:28.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.899+0000 7f1cc7200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc81461b0 0x7f1cc81311f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:28.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.899+0000 7f1cc7200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc80c5cd0 con 0x7f1cc81461b0 2024-04-16T16:21:28.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.899+0000 7f1cc7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc80c59f0 con 0x7f1cc80b58f0 2024-04-16T16:21:28.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.899+0000 7f1cc7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1cc80c5fb0 con 0x7f1cc80b5d50 2024-04-16T16:21:28.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.899+0000 7f1cc6800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc81461b0 0x7f1cc81311f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:28.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.900+0000 7f1cc6800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc81461b0 0x7f1cc81311f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58944/0 (socket says 172.21.15.45:58944) 2024-04-16T16:21:28.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.900+0000 7f1cc6800640 1 -- 172.21.15.45:0/74568703 learned_addr learned my addr 172.21.15.45:0/74568703 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:28.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.900+0000 7f1cc5e00640 1 --2- 172.21.15.45:0/74568703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cc80b58f0 0x7f1cc80ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:28.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.900+0000 7f1cc5400640 1 --2- 172.21.15.45:0/74568703 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cc80b5d50 0x7f1cc80cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:28.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.900+0000 7f1cc6800640 1 -- 172.21.15.45:0/74568703 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cc80b5d50 msgr2=0x7f1cc80cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:28.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.900+0000 7f1cc6800640 1 --2- 172.21.15.45:0/74568703 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cc80b5d50 0x7f1cc80cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:28.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.900+0000 7f1cc6800640 1 -- 172.21.15.45:0/74568703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cc80b58f0 msgr2=0x7f1cc80ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:28.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.900+0000 7f1cc6800640 1 --2- 172.21.15.45:0/74568703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cc80b58f0 0x7f1cc80ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:28.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.900+0000 7f1cc6800640 1 -- 172.21.15.45:0/74568703 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc812f300 con 0x7f1cc81461b0 2024-04-16T16:21:28.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.900+0000 7f1cc5e00640 1 --2- 172.21.15.45:0/74568703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cc80b58f0 0x7f1cc80ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:21:28.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.900+0000 7f1cc6800640 1 --2- 172.21.15.45:0/74568703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc81461b0 0x7f1cc81311f0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0009f10 tx=0x7f1cc000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:28.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.901+0000 7f1cbe000640 1 -- 172.21.15.45:0/74568703 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc000cc40 con 0x7f1cc81461b0 2024-04-16T16:21:28.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.901+0000 7f1cbe000640 1 -- 172.21.15.45:0/74568703 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cc000cde0 con 0x7f1cc81461b0 2024-04-16T16:21:28.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.901+0000 7f1cbe000640 1 -- 172.21.15.45:0/74568703 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc0003bb0 con 0x7f1cc81461b0 2024-04-16T16:21:28.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.901+0000 7f1cc7200640 1 -- 172.21.15.45:0/74568703 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cc812f5e0 con 0x7f1cc81461b0 2024-04-16T16:21:28.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.901+0000 7f1cc7200640 1 -- 172.21.15.45:0/74568703 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cc812fba0 con 0x7f1cc81461b0 2024-04-16T16:21:28.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.902+0000 7f1cc7200640 1 -- 172.21.15.45:0/74568703 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cc812fda0 con 0x7f1cc81461b0 2024-04-16T16:21:28.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.903+0000 7f1cbe000640 1 -- 172.21.15.45:0/74568703 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f1cc001b030 con 0x7f1cc81461b0 2024-04-16T16:21:28.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.904+0000 7f1cbe000640 1 --2- 172.21.15.45:0/74568703 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1ca806ca10 0x7f1ca806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:28.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.904+0000 7f1cbe000640 1 -- 172.21.15.45:0/74568703 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7f1cc008c2a0 con 0x7f1cc81461b0 2024-04-16T16:21:28.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.905+0000 7f1cc5e00640 1 --2- 172.21.15.45:0/74568703 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1ca806ca10 0x7f1ca806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:28.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.906+0000 7f1cc5e00640 1 --2- 172.21.15.45:0/74568703 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1ca806ca10 0x7f1ca806eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8036000 tx=0x7f1cb8008030 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:28.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:28.909+0000 7f1cbe000640 1 -- 172.21.15.45:0/74568703 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1cc812fda0 con 0x7f1cc81461b0 2024-04-16T16:21:29.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.154+0000 7f1cc7200640 1 -- 172.21.15.45:0/74568703 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f1cc8075780 con 0x7f1cc81461b0 2024-04-16T16:21:29.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.634+0000 7f1cbe000640 1 -- 172.21.15.45:0/74568703 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v100) ==== 122+0+0 (secure 0 0 0) 0x7f1cc0022020 con 0x7f1cc81461b0 2024-04-16T16:21:29.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.638+0000 7f1cc7200640 1 -- 172.21.15.45:0/74568703 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1ca806ca10 msgr2=0x7f1ca806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:29.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.638+0000 7f1cc7200640 1 --2- 172.21.15.45:0/74568703 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1ca806ca10 0x7f1ca806eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8036000 tx=0x7f1cb8008030 comp rx=0 tx=0).stop 2024-04-16T16:21:29.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.638+0000 7f1cc7200640 1 -- 172.21.15.45:0/74568703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc81461b0 msgr2=0x7f1cc81311f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:29.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.638+0000 7f1cc7200640 1 --2- 172.21.15.45:0/74568703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc81461b0 0x7f1cc81311f0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0009f10 tx=0x7f1cc000c490 comp rx=0 tx=0).stop 2024-04-16T16:21:29.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.638+0000 7f1cc7200640 1 -- 172.21.15.45:0/74568703 shutdown_connections 2024-04-16T16:21:29.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.638+0000 7f1cc7200640 1 --2- 172.21.15.45:0/74568703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc81461b0 0x7f1cc81311f0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:29.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.638+0000 7f1cc7200640 1 --2- 172.21.15.45:0/74568703 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cc80b5d50 0x7f1cc80cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:29.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.638+0000 7f1cc7200640 1 --2- 172.21.15.45:0/74568703 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f1ca806ca10 0x7f1ca806eed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:29.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.638+0000 7f1cc7200640 1 --2- 172.21.15.45:0/74568703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cc80b58f0 0x7f1cc80ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:29.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.638+0000 7f1cc7200640 1 -- 172.21.15.45:0/74568703 >> 172.21.15.45:0/74568703 conn(0x7f1cc806b930 msgr2=0x7f1cc806e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:29.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.638+0000 7f1cc7200640 1 -- 172.21.15.45:0/74568703 shutdown_connections 2024-04-16T16:21:29.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.638+0000 7f1cc7200640 1 -- 172.21.15.45:0/74568703 wait complete. 2024-04-16T16:21:29.659 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-16T16:21:29.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:21:29.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.886+0000 7f92e7200640 1 Processor -- start 2024-04-16T16:21:29.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.886+0000 7f92e7200640 1 -- start start 2024-04-16T16:21:29.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.887+0000 7f92e7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8162270 0x7f92e80d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:29.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.887+0000 7f92e7200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f92e80d52d0 0x7f92e806ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.887+0000 7f92e7200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92e80d3080 con 0x7f92e8161e90 2024-04-16T16:21:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.887+0000 7f92e7200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f92e80d2e30 con 0x7f92e80d52d0 2024-04-16T16:21:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.887+0000 7f92e7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f92e80d3360 con 0x7f92e8162270 2024-04-16T16:21:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.888+0000 7f92e5400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8162270 0x7f92e80d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.888+0000 7f92e5e00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f92e8161e90 0x7f92e80cf7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:38296/0 (socket says 172.21.15.45:38296) 2024-04-16T16:21:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.888+0000 7f92e5e00640 1 -- 172.21.15.45:0/3387120827 learned_addr learned my addr 172.21.15.45:0/3387120827 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.888+0000 7f92e6800640 1 --2- 172.21.15.45:0/3387120827 >> v2:172.21.15.103:3300/0 conn(0x7f92e80d52d0 0x7f92e806ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:29.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.889+0000 7f92e5400640 1 -- 172.21.15.45:0/3387120827 >> v2:172.21.15.103:3300/0 conn(0x7f92e80d52d0 msgr2=0x7f92e806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:29.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.889+0000 7f92e5400640 1 --2- 172.21.15.45:0/3387120827 >> v2:172.21.15.103:3300/0 conn(0x7f92e80d52d0 0x7f92e806ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:29.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.889+0000 7f92e5400640 1 -- 172.21.15.45:0/3387120827 >> v1:172.21.15.103:6789/0 conn(0x7f92e8161e90 legacy=0x7f92e80cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:29.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.889+0000 7f92e5400640 1 -- 172.21.15.45:0/3387120827 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e80d3640 con 0x7f92e8162270 2024-04-16T16:21:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.890+0000 7f92e5400640 1 --2- 172.21.15.45:0/3387120827 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8162270 0x7f92e80d3e40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f92d4009d90 tx=0x7f92d402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2814e7244852e8d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.891+0000 7f92e4a00640 1 -- 172.21.15.45:0/3387120827 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d4039070 con 0x7f92e8162270 2024-04-16T16:21:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.891+0000 7f92e4a00640 1 -- 172.21.15.45:0/3387120827 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92d402cce0 con 0x7f92e8162270 2024-04-16T16:21:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.891+0000 7f92e4a00640 1 -- 172.21.15.45:0/3387120827 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d4034660 con 0x7f92e8162270 2024-04-16T16:21:29.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.891+0000 7f92e7200640 1 -- 172.21.15.45:0/3387120827 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8162270 msgr2=0x7f92e80d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:29.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.891+0000 7f92e7200640 1 --2- 172.21.15.45:0/3387120827 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8162270 0x7f92e80d3e40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f92d4009d90 tx=0x7f92d402bf30 comp rx=0 tx=0).stop 2024-04-16T16:21:29.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.891+0000 7f92e7200640 1 -- 172.21.15.45:0/3387120827 shutdown_connections 2024-04-16T16:21:29.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.891+0000 7f92e7200640 1 --2- 172.21.15.45:0/3387120827 >> v2:172.21.15.103:3300/0 conn(0x7f92e80d52d0 0x7f92e806ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.891+0000 7f92e7200640 1 --2- 172.21.15.45:0/3387120827 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8162270 0x7f92e80d3e40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.891+0000 7f92e7200640 1 -- 172.21.15.45:0/3387120827 >> 172.21.15.45:0/3387120827 conn(0x7f92e80d1940 msgr2=0x7f92e80d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.892+0000 7f92e7200640 1 -- 172.21.15.45:0/3387120827 shutdown_connections 2024-04-16T16:21:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.892+0000 7f92e7200640 1 -- 172.21.15.45:0/3387120827 wait complete. 2024-04-16T16:21:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.893+0000 7f92e7200640 1 Processor -- start 2024-04-16T16:21:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.893+0000 7f92e7200640 1 -- start start 2024-04-16T16:21:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.894+0000 7f92e7200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92e80d52d0 0x7f92e815b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.894+0000 7f92e7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8161e90 0x7f92e814f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.894+0000 7f92e7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92e8162270 0x7f92e814fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.894+0000 7f92e7200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f92e80d4500 con 0x7f92e80d52d0 2024-04-16T16:21:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.894+0000 7f92e7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f92e80d4380 con 0x7f92e8162270 2024-04-16T16:21:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.894+0000 7f92e7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f92e80d4680 con 0x7f92e8161e90 2024-04-16T16:21:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.895+0000 7f92e5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92e80d52d0 0x7f92e815b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.895+0000 7f92e5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92e80d52d0 0x7f92e815b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58960/0 (socket says 172.21.15.45:58960) 2024-04-16T16:21:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.895+0000 7f92e5e00640 1 -- 172.21.15.45:0/3193703021 learned_addr learned my addr 172.21.15.45:0/3193703021 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.895+0000 7f92e5400640 1 --2- 172.21.15.45:0/3193703021 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8161e90 0x7f92e814f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.895+0000 7f92e5e00640 1 -- 172.21.15.45:0/3193703021 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8161e90 msgr2=0x7f92e814f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.895+0000 7f92e5e00640 1 --2- 172.21.15.45:0/3193703021 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8161e90 0x7f92e814f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.895+0000 7f92e6800640 1 --2- 172.21.15.45:0/3193703021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92e8162270 0x7f92e814fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.895+0000 7f92e5e00640 1 -- 172.21.15.45:0/3193703021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92e8162270 msgr2=0x7f92e814fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.895+0000 7f92e5e00640 1 --2- 172.21.15.45:0/3193703021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92e8162270 0x7f92e814fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.895+0000 7f92e5e00640 1 -- 172.21.15.45:0/3193703021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e80cc650 con 0x7f92e80d52d0 2024-04-16T16:21:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.895+0000 7f92e5e00640 1 --2- 172.21.15.45:0/3193703021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92e80d52d0 0x7f92e815b970 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f92d800c400 tx=0x7f92d800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.896+0000 7f92de000640 1 -- 172.21.15.45:0/3193703021 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d8017070 con 0x7f92e80d52d0 2024-04-16T16:21:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.896+0000 7f92de000640 1 -- 172.21.15.45:0/3193703021 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92d8005ce0 con 0x7f92e80d52d0 2024-04-16T16:21:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.896+0000 7f92de000640 1 -- 172.21.15.45:0/3193703021 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d80072f0 con 0x7f92e80d52d0 2024-04-16T16:21:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.896+0000 7f92e7200640 1 -- 172.21.15.45:0/3193703021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92e80cc930 con 0x7f92e80d52d0 2024-04-16T16:21:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.896+0000 7f92e7200640 1 -- 172.21.15.45:0/3193703021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92e80cc340 con 0x7f92e80d52d0 2024-04-16T16:21:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.897+0000 7f92e7200640 1 -- 172.21.15.45:0/3193703021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92e8075780 con 0x7f92e80d52d0 2024-04-16T16:21:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.898+0000 7f92de000640 1 -- 172.21.15.45:0/3193703021 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f92d801a070 con 0x7f92e80d52d0 2024-04-16T16:21:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.900+0000 7f92de000640 1 --2- 172.21.15.45:0/3193703021 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f92c806ca10 0x7f92c806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.900+0000 7f92de000640 1 -- 172.21.15.45:0/3193703021 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7f92d808b040 con 0x7f92e80d52d0 2024-04-16T16:21:29.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.900+0000 7f92e5400640 1 --2- 172.21.15.45:0/3193703021 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f92c806ca10 0x7f92c806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:29.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.902+0000 7f92e5400640 1 --2- 172.21.15.45:0/3193703021 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f92c806ca10 0x7f92c806eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f92d4036000 tx=0x7f92d4030000 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:29.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:29.905+0000 7f92de000640 1 -- 172.21.15.45:0/3193703021 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f92d80593f0 con 0x7f92e80d52d0 2024-04-16T16:21:30.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.147+0000 7f92e7200640 1 -- 172.21.15.45:0/3193703021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92e806dae0 con 0x7f92e80d52d0 2024-04-16T16:21:30.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.148+0000 7f92de000640 1 -- 172.21.15.45:0/3193703021 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5190 (secure 0 0 0) 0x7f92d8058cf0 con 0x7f92e80d52d0 2024-04-16T16:21:30.147 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:30.147 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2585865379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2585865379},{"type":"v1","addr":"172.21.15.45:6834","nonce":2585865379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6590,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/777087937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":777087937},{"type":"v1","addr":"172.21.15.45:6835","nonce":777087937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6593,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2561260581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2561260581},{"type":"v1","addr":"172.21.15.103:6835","nonce":2561260581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:21:26.066855+0000","modified":"2024-04-16T16:21:29.155869+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6596":{"gid":6596,"name":"d","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6837/3252460318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3252460318},{"type":"v1","addr":"172.21.15.103:6837","nonce":3252460318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6596,"qdb_cluster":[6596]},"id":10}]} 2024-04-16T16:21:30.147 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 100 2024-04-16T16:21:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.150+0000 7f92e7200640 1 -- 172.21.15.45:0/3193703021 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f92c806ca10 msgr2=0x7f92c806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.150+0000 7f92e7200640 1 --2- 172.21.15.45:0/3193703021 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f92c806ca10 0x7f92c806eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f92d4036000 tx=0x7f92d4030000 comp rx=0 tx=0).stop 2024-04-16T16:21:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.150+0000 7f92e7200640 1 -- 172.21.15.45:0/3193703021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92e80d52d0 msgr2=0x7f92e815b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.150+0000 7f92e7200640 1 --2- 172.21.15.45:0/3193703021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92e80d52d0 0x7f92e815b970 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f92d800c400 tx=0x7f92d800c8c0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.150+0000 7f92e7200640 1 -- 172.21.15.45:0/3193703021 shutdown_connections 2024-04-16T16:21:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.150+0000 7f92e7200640 1 --2- 172.21.15.45:0/3193703021 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f92c806ca10 0x7f92c806eed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.150+0000 7f92e7200640 1 --2- 172.21.15.45:0/3193703021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92e8162270 0x7f92e814fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.150+0000 7f92e7200640 1 --2- 172.21.15.45:0/3193703021 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8161e90 0x7f92e814f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.150+0000 7f92e7200640 1 --2- 172.21.15.45:0/3193703021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92e80d52d0 0x7f92e815b970 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.150+0000 7f92e7200640 1 -- 172.21.15.45:0/3193703021 >> 172.21.15.45:0/3193703021 conn(0x7f92e80d1940 msgr2=0x7f92e80b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:30.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.150+0000 7f92e7200640 1 -- 172.21.15.45:0/3193703021 shutdown_connections 2024-04-16T16:21:30.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.150+0000 7f92e7200640 1 -- 172.21.15.45:0/3193703021 wait complete. 2024-04-16T16:21:30.168 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:21:30.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.389+0000 7effa7e00640 1 Processor -- start 2024-04-16T16:21:30.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.390+0000 7effa7e00640 1 -- start start 2024-04-16T16:21:30.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.391+0000 7effa7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effa80ccb20 0x7effa80cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:30.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.391+0000 7effa7e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effa815ecd0 con 0x7effa80ce9a0 2024-04-16T16:21:30.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.391+0000 7effa7e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effa80cd6d0 con 0x7effa815e8f0 2024-04-16T16:21:30.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.391+0000 7effa7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7effa80ce2e0 con 0x7effa80ccb20 2024-04-16T16:21:30.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.391+0000 7effa7400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7effa80ce9a0 0x7effa80cb170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60538/0 (socket says 172.21.15.45:60538) 2024-04-16T16:21:30.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.391+0000 7effa7400640 1 -- 172.21.15.45:0/2378284062 learned_addr learned my addr 172.21.15.45:0/2378284062 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:30.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.392+0000 7effa6000640 1 --2- 172.21.15.45:0/2378284062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effa80ccb20 0x7effa80cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:30.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.392+0000 7effa5600640 1 -- 172.21.15.45:0/2378284062 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3934385381 0 0) 0x7effa815ecd0 con 0x7effa80ce9a0 2024-04-16T16:21:30.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.392+0000 7effa5600640 1 -- 172.21.15.45:0/2378284062 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff8c003520 con 0x7effa80ce9a0 2024-04-16T16:21:30.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.392+0000 7effa5600640 1 -- 172.21.15.45:0/2378284062 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3310841268 0 0) 0x7eff8c003520 con 0x7effa80ce9a0 2024-04-16T16:21:30.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.392+0000 7effa5600640 1 -- 172.21.15.45:0/2378284062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effa80ccb20 msgr2=0x7effa80cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:30.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.392+0000 7effa5600640 1 --2- 172.21.15.45:0/2378284062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effa80ccb20 0x7effa80cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.392+0000 7effa5600640 1 -- 172.21.15.45:0/2378284062 >> v1:172.21.15.103:6789/0 conn(0x7effa815e8f0 legacy=0x7effa80cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:30.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.392+0000 7effa5600640 1 -- 172.21.15.45:0/2378284062 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effa80cd9b0 con 0x7effa80ce9a0 2024-04-16T16:21:30.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.392+0000 7effa5600640 1 -- 172.21.15.45:0/2378284062 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7effa0001670 con 0x7effa80ce9a0 2024-04-16T16:21:30.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.392+0000 7effa5600640 1 -- 172.21.15.45:0/2378284062 >> v1:172.21.15.45:6789/0 conn(0x7effa80ce9a0 legacy=0x7effa80cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:30.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.393+0000 7effa5600640 1 --2- 172.21.15.45:0/2378284062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effa80ccb20 0x7eff8c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:30.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.393+0000 7effa5600640 1 --2- 172.21.15.45:0/2378284062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff8c004010 0x7eff8c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:30.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.393+0000 7effa5600640 1 --2- 172.21.15.45:0/2378284062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff8c004d40 0x7eff8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:30.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.393+0000 7effa5600640 1 -- 172.21.15.45:0/2378284062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7effa80ce2e0 con 0x7eff8c004010 2024-04-16T16:21:30.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.393+0000 7effa5600640 1 -- 172.21.15.45:0/2378284062 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7eff8c00bb80 con 0x7eff8c004d40 2024-04-16T16:21:30.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.393+0000 7effa5600640 1 -- 172.21.15.45:0/2378284062 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7eff8c00bd80 con 0x7effa80ccb20 2024-04-16T16:21:30.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.393+0000 7effa6000640 1 --2- 172.21.15.45:0/2378284062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff8c004010 0x7eff8c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:30.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.393+0000 7effa6000640 1 -- 172.21.15.45:0/2378284062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effa80ccb20 msgr2=0x7eff8c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:30.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.393+0000 7effa6000640 1 --2- 172.21.15.45:0/2378284062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effa80ccb20 0x7eff8c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.393+0000 7effa6000640 1 -- 172.21.15.45:0/2378284062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff8c004d40 msgr2=0x7eff8c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:30.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.393+0000 7effa6000640 1 --2- 172.21.15.45:0/2378284062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff8c004d40 0x7eff8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.393+0000 7effa6000640 1 -- 172.21.15.45:0/2378284062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8c003520 con 0x7eff8c004010 2024-04-16T16:21:30.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.394+0000 7effa6000640 1 --2- 172.21.15.45:0/2378284062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff8c004010 0x7eff8c004800 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7eff9000ba30 tx=0x7eff8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1507b3ae852c932a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:30.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.395+0000 7effa5600640 1 -- 172.21.15.45:0/2378284062 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff90036020 con 0x7eff8c004010 2024-04-16T16:21:30.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.395+0000 7effa5600640 1 -- 172.21.15.45:0/2378284062 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff90032680 con 0x7eff8c004010 2024-04-16T16:21:30.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.395+0000 7effa5600640 1 -- 172.21.15.45:0/2378284062 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff9000b600 con 0x7eff8c004010 2024-04-16T16:21:30.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.395+0000 7effa7e00640 1 -- 172.21.15.45:0/2378284062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff8c004010 msgr2=0x7eff8c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:30.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.395+0000 7effa7e00640 1 --2- 172.21.15.45:0/2378284062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff8c004010 0x7eff8c004800 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7eff9000ba30 tx=0x7eff8c005830 comp rx=0 tx=0).stop 2024-04-16T16:21:30.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.395+0000 7effa7400640 1 -- 172.21.15.45:0/2378284062 reap_dead start 2024-04-16T16:21:30.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.395+0000 7effa7e00640 1 -- 172.21.15.45:0/2378284062 shutdown_connections 2024-04-16T16:21:30.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.395+0000 7effa7e00640 1 -- 172.21.15.45:0/2378284062 >> 172.21.15.45:0/2378284062 conn(0x7effa806b930 msgr2=0x7effa806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:30.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.396+0000 7effa7e00640 1 -- 172.21.15.45:0/2378284062 shutdown_connections 2024-04-16T16:21:30.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.396+0000 7effa7e00640 1 -- 172.21.15.45:0/2378284062 wait complete. 2024-04-16T16:21:30.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.397+0000 7effa7e00640 1 Processor -- start 2024-04-16T16:21:30.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.397+0000 7effa7e00640 1 -- start start 2024-04-16T16:21:30.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.398+0000 7effa7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff8c004010 0x7effa815ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:30.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.398+0000 7effa7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa814d4e0 0x7effa8150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:30.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.398+0000 7effa7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effa8150c50 0x7effa81531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:30.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.398+0000 7effa7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7effa80cd360 con 0x7eff8c004010 2024-04-16T16:21:30.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.398+0000 7effa7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7effa80cdf70 con 0x7effa814d4e0 2024-04-16T16:21:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.398+0000 7effa7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7effa80cf080 con 0x7effa8150c50 2024-04-16T16:21:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.399+0000 7effa6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff8c004010 0x7effa815ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.399+0000 7effa6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff8c004010 0x7effa815ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58978/0 (socket says 172.21.15.45:58978) 2024-04-16T16:21:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.399+0000 7effa6a00640 1 -- 172.21.15.45:0/1394973446 learned_addr learned my addr 172.21.15.45:0/1394973446 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.399+0000 7effa7400640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effa8150c50 0x7effa81531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.399+0000 7effa6000640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa814d4e0 0x7effa8150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.399+0000 7effa6a00640 1 -- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effa8150c50 msgr2=0x7effa81531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.399+0000 7effa6a00640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effa8150c50 0x7effa81531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.399+0000 7effa6a00640 1 -- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa814d4e0 msgr2=0x7effa8150710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.399+0000 7effa6a00640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa814d4e0 0x7effa8150710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.399+0000 7effa6a00640 1 -- 172.21.15.45:0/1394973446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effa8151100 con 0x7eff8c004010 2024-04-16T16:21:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.399+0000 7effa7400640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effa8150c50 0x7effa81531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.399+0000 7effa6000640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa814d4e0 0x7effa8150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.400+0000 7effa6a00640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff8c004010 0x7effa815ef00 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7eff9800c7c0 tx=0x7eff9800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:30.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.400+0000 7eff9ea00640 1 -- 172.21.15.45:0/1394973446 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff98003b70 con 0x7eff8c004010 2024-04-16T16:21:30.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.400+0000 7eff9ea00640 1 -- 172.21.15.45:0/1394973446 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff98003d10 con 0x7eff8c004010 2024-04-16T16:21:30.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.400+0000 7effa7e00640 1 -- 172.21.15.45:0/1394973446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effa8153e90 con 0x7eff8c004010 2024-04-16T16:21:30.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.400+0000 7eff9ea00640 1 -- 172.21.15.45:0/1394973446 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff9800aa90 con 0x7eff8c004010 2024-04-16T16:21:30.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.400+0000 7effa7e00640 1 -- 172.21.15.45:0/1394973446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effa814ff10 con 0x7eff8c004010 2024-04-16T16:21:30.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.401+0000 7effa7e00640 1 -- 172.21.15.45:0/1394973446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effa814fa60 con 0x7eff8c004010 2024-04-16T16:21:30.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.402+0000 7eff9ea00640 1 -- 172.21.15.45:0/1394973446 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7eff9800ac30 con 0x7eff8c004010 2024-04-16T16:21:30.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.403+0000 7eff9ea00640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7eff8406ca40 0x7eff8406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:30.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.403+0000 7eff9ea00640 1 -- 172.21.15.45:0/1394973446 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7eff9808bbe0 con 0x7eff8c004010 2024-04-16T16:21:30.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.404+0000 7effa6000640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7eff8406ca40 0x7eff8406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:30.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.405+0000 7effa6000640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7eff8406ca40 0x7eff8406ef00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7eff9000ba00 tx=0x7eff90030000 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:30.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.408+0000 7eff9ea00640 1 -- 172.21.15.45:0/1394973446 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7effa8151100 con 0x7eff8c004010 2024-04-16T16:21:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.613+0000 7effa7e00640 1 -- 172.21.15.45:0/1394973446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7effa8075780 con 0x7eff8c004010 2024-04-16T16:21:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.616+0000 7eff9ea00640 1 -- 172.21.15.45:0/1394973446 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) ==== 74+0+16679 (secure 0 0 0) 0x7eff98059f90 con 0x7eff8c004010 2024-04-16T16:21:30.616 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:30.616 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":142,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:21:27.608290+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:21:23.973149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-16T16:21:25.043255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:21:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.619+0000 7effa7e00640 1 -- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7eff8406ca40 msgr2=0x7eff8406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.619+0000 7effa7e00640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7eff8406ca40 0x7eff8406ef00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7eff9000ba00 tx=0x7eff90030000 comp rx=0 tx=0).stop 2024-04-16T16:21:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.619+0000 7effa7e00640 1 -- 172.21.15.45:0/1394973446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff8c004010 msgr2=0x7effa815ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.619+0000 7effa7e00640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff8c004010 0x7effa815ef00 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7eff9800c7c0 tx=0x7eff9800cc80 comp rx=0 tx=0).stop 2024-04-16T16:21:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.619+0000 7effa7e00640 1 -- 172.21.15.45:0/1394973446 shutdown_connections 2024-04-16T16:21:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.619+0000 7effa7e00640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effa8150c50 0x7effa81531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.619+0000 7effa7e00640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7eff8406ca40 0x7eff8406ef00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.619+0000 7effa7e00640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa814d4e0 0x7effa8150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.619+0000 7effa7e00640 1 --2- 172.21.15.45:0/1394973446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff8c004010 0x7effa815ef00 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.619+0000 7effa7e00640 1 -- 172.21.15.45:0/1394973446 >> 172.21.15.45:0/1394973446 conn(0x7effa806b930 msgr2=0x7effa806ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.619+0000 7effa7e00640 1 -- 172.21.15.45:0/1394973446 shutdown_connections 2024-04-16T16:21:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.619+0000 7effa7e00640 1 -- 172.21.15.45:0/1394973446 wait complete. 2024-04-16T16:21:30.637 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-16T16:21:30.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:21:30.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.850+0000 7f5d0a000640 1 Processor -- start 2024-04-16T16:21:30.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.851+0000 7f5d0a000640 1 -- start start 2024-04-16T16:21:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.852+0000 7f5d0a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0415edf0 0x7f5d040b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.852+0000 7f5d0a000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5d040cfa50 0x7f5d040b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.852+0000 7f5d0a000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d040d2e20 con 0x7f5d040b6a30 2024-04-16T16:21:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.852+0000 7f5d0a000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5d040d3be0 con 0x7f5d040cfa50 2024-04-16T16:21:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.852+0000 7f5d0a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5d040d3750 con 0x7f5d0415edf0 2024-04-16T16:21:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.852+0000 7f5d03e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5d040cfa50 0x7f5d040b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.852+0000 7f5d03e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5d040cfa50 0x7f5d040b7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58988/0 (socket says 172.21.15.45:58988) 2024-04-16T16:21:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.853+0000 7f5d03e00640 1 -- 172.21.15.45:0/2639665830 learned_addr learned my addr 172.21.15.45:0/2639665830 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.853+0000 7f5d03400640 1 --2- 172.21.15.45:0/2639665830 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0415edf0 0x7f5d040b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.853+0000 7f5d03e00640 1 -- 172.21.15.45:0/2639665830 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0415edf0 msgr2=0x7f5d040b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.853+0000 7f5d03e00640 1 --2- 172.21.15.45:0/2639665830 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0415edf0 0x7f5d040b64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.853+0000 7f5d03e00640 1 -- 172.21.15.45:0/2639665830 >> v1:172.21.15.103:6789/0 conn(0x7f5d040b6a30 legacy=0x7f5d040cf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:30.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.853+0000 7f5d03e00640 1 -- 172.21.15.45:0/2639665830 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d040d3100 con 0x7f5d040cfa50 2024-04-16T16:21:30.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.854+0000 7f5d03e00640 1 --2- 172.21.15.45:0/2639665830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cfa50 0x7f5d040b7a70 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc009870 tx=0x7f5cfc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef1de5256b24295 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:30.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.855+0000 7f5d02000640 1 -- 172.21.15.45:0/2639665830 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc034070 con 0x7f5d040cfa50 2024-04-16T16:21:30.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.855+0000 7f5d02000640 1 -- 172.21.15.45:0/2639665830 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cfc030840 con 0x7f5d040cfa50 2024-04-16T16:21:30.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.855+0000 7f5d02000640 1 -- 172.21.15.45:0/2639665830 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc030c10 con 0x7f5d040cfa50 2024-04-16T16:21:30.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.855+0000 7f5d0a000640 1 -- 172.21.15.45:0/2639665830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cfa50 msgr2=0x7f5d040b7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:30.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.855+0000 7f5d0a000640 1 --2- 172.21.15.45:0/2639665830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cfa50 0x7f5d040b7a70 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc009870 tx=0x7f5cfc02b980 comp rx=0 tx=0).stop 2024-04-16T16:21:30.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.856+0000 7f5d0a000640 1 -- 172.21.15.45:0/2639665830 shutdown_connections 2024-04-16T16:21:30.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.856+0000 7f5d0a000640 1 --2- 172.21.15.45:0/2639665830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cfa50 0x7f5d040b7a70 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.856+0000 7f5d0a000640 1 --2- 172.21.15.45:0/2639665830 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0415edf0 0x7f5d040b64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.856+0000 7f5d0a000640 1 -- 172.21.15.45:0/2639665830 >> 172.21.15.45:0/2639665830 conn(0x7f5d04073690 msgr2=0x7f5d040d14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:30.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.856+0000 7f5d0a000640 1 -- 172.21.15.45:0/2639665830 shutdown_connections 2024-04-16T16:21:30.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.856+0000 7f5d0a000640 1 -- 172.21.15.45:0/2639665830 wait complete. 2024-04-16T16:21:30.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.857+0000 7f5d0a000640 1 Processor -- start 2024-04-16T16:21:30.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.858+0000 7f5d0a000640 1 -- start start 2024-04-16T16:21:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d0a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d040b6a30 0x7f5d0415af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d0a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cfa50 0x7f5d040ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d0a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0415edf0 0x7f5d040cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d0a000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5d040b7fb0 con 0x7f5d040cfa50 2024-04-16T16:21:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d0a000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5d040b6e70 con 0x7f5d040b6a30 2024-04-16T16:21:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d0a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5d040b8130 con 0x7f5d0415edf0 2024-04-16T16:21:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d02a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cfa50 0x7f5d040ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d02a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cfa50 0x7f5d040ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58994/0 (socket says 172.21.15.45:58994) 2024-04-16T16:21:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d02a00640 1 -- 172.21.15.45:0/2882135679 learned_addr learned my addr 172.21.15.45:0/2882135679 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:30.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d03400640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d040b6a30 0x7f5d0415af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d03e00640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0415edf0 0x7f5d040cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d02a00640 1 -- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0415edf0 msgr2=0x7f5d040cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d02a00640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0415edf0 0x7f5d040cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d02a00640 1 -- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d040b6a30 msgr2=0x7f5d0415af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d02a00640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d040b6a30 0x7f5d0415af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d02a00640 1 -- 172.21.15.45:0/2882135679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d040c87b0 con 0x7f5d040cfa50 2024-04-16T16:21:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.859+0000 7f5d03e00640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0415edf0 0x7f5d040cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.860+0000 7f5d03400640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d040b6a30 0x7f5d0415af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.860+0000 7f5d02a00640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cfa50 0x7f5d040ccb90 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f5cec00c400 tx=0x7f5cec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.860+0000 7f5cfb400640 1 -- 172.21.15.45:0/2882135679 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cec0052d0 con 0x7f5d040cfa50 2024-04-16T16:21:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.860+0000 7f5cfb400640 1 -- 172.21.15.45:0/2882135679 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cec005490 con 0x7f5d040cfa50 2024-04-16T16:21:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.860+0000 7f5cfb400640 1 -- 172.21.15.45:0/2882135679 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cec007260 con 0x7f5d040cfa50 2024-04-16T16:21:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.860+0000 7f5d0a000640 1 -- 172.21.15.45:0/2882135679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d040c8a90 con 0x7f5d040cfa50 2024-04-16T16:21:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.860+0000 7f5d0a000640 1 -- 172.21.15.45:0/2882135679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d040c9050 con 0x7f5d040cfa50 2024-04-16T16:21:30.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.861+0000 7f5d0a000640 1 -- 172.21.15.45:0/2882135679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d04075780 con 0x7f5d040cfa50 2024-04-16T16:21:30.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.862+0000 7f5cfb400640 1 -- 172.21.15.45:0/2882135679 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f5cec007400 con 0x7f5d040cfa50 2024-04-16T16:21:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.863+0000 7f5cfb400640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5ce006ca10 0x7f5ce006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.863+0000 7f5cfb400640 1 -- 172.21.15.45:0/2882135679 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7f5cec08b320 con 0x7f5d040cfa50 2024-04-16T16:21:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.864+0000 7f5d03400640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5ce006ca10 0x7f5ce006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:30.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.865+0000 7f5d03400640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5ce006ca10 0x7f5ce006eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5d0415e590 tx=0x7f5cf400b040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:30.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:30.868+0000 7f5cfb400640 1 -- 172.21.15.45:0/2882135679 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5cec0595a0 con 0x7f5d040cfa50 2024-04-16T16:21:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.068+0000 7f5d0a000640 1 -- 172.21.15.45:0/2882135679 --> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5d040c9220 con 0x7f5ce006ca10 2024-04-16T16:21:31.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.090+0000 7f5cfb400640 1 -- 172.21.15.45:0/2882135679 <== mgr.7153 v2:172.21.15.103:6832/1478535274 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+241854 (secure 0 0 0) 0x7f5d040c9220 con 0x7f5ce006ca10 2024-04-16T16:21:31.091 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:31.094 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:21:31.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.096+0000 7f5d0a000640 1 -- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5ce006ca10 msgr2=0x7f5ce006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.096+0000 7f5d0a000640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5ce006ca10 0x7f5ce006eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5d0415e590 tx=0x7f5cf400b040 comp rx=0 tx=0).stop 2024-04-16T16:21:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.096+0000 7f5d0a000640 1 -- 172.21.15.45:0/2882135679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cfa50 msgr2=0x7f5d040ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.096+0000 7f5d0a000640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cfa50 0x7f5d040ccb90 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f5cec00c400 tx=0x7f5cec00c8c0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.096+0000 7f5d0a000640 1 -- 172.21.15.45:0/2882135679 shutdown_connections 2024-04-16T16:21:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.096+0000 7f5d0a000640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0415edf0 0x7f5d040cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.096+0000 7f5d0a000640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cfa50 0x7f5d040ccb90 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.096+0000 7f5d0a000640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5ce006ca10 0x7f5ce006eed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.096+0000 7f5d0a000640 1 --2- 172.21.15.45:0/2882135679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d040b6a30 0x7f5d0415af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.096+0000 7f5d0a000640 1 -- 172.21.15.45:0/2882135679 >> 172.21.15.45:0/2882135679 conn(0x7f5d04073690 msgr2=0x7f5d040d1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.097+0000 7f5d0a000640 1 -- 172.21.15.45:0/2882135679 shutdown_connections 2024-04-16T16:21:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.097+0000 7f5d0a000640 1 -- 172.21.15.45:0/2882135679 wait complete. 2024-04-16T16:21:31.116 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-04-16T16:21:30.390379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1083,"num_read_kb":2113,"num_write":360,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4679,"ondisk_log_size":4679,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":272564,"kb_used_data":14788,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754702156,"statfs":{"total":773094113280,"available":772815007744,"internally_reserved":0,"allocated":15142912,"data_stored":7797822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.830661"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643268+0000","last_change":"2024-04-16T16:21:26.619336+0000","last_active":"2024-04-16T16:21:28.643268+0000","last_peered":"2024-04-16T16:21:28.643268+0000","last_clean":"2024-04-16T16:21:28.643268+0000","last_became_active":"2024-04-16T16:21:26.618973+0000","last_became_peered":"2024-04-16T16:21:26.618973+0000","last_unstale":"2024-04-16T16:21:28.643268+0000","last_undegraded":"2024-04-16T16:21:28.643268+0000","last_fullsized":"2024-04-16T16:21:28.643268+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:56.732655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643963+0000","last_change":"2024-04-16T16:21:26.618141+0000","last_active":"2024-04-16T16:21:28.643963+0000","last_peered":"2024-04-16T16:21:28.643963+0000","last_clean":"2024-04-16T16:21:28.643963+0000","last_became_active":"2024-04-16T16:21:26.614289+0000","last_became_peered":"2024-04-16T16:21:26.614289+0000","last_unstale":"2024-04-16T16:21:28.643963+0000","last_undegraded":"2024-04-16T16:21:28.643963+0000","last_fullsized":"2024-04-16T16:21:28.643963+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:15:22.005025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643682+0000","last_change":"2024-04-16T16:21:26.623659+0000","last_active":"2024-04-16T16:21:28.643682+0000","last_peered":"2024-04-16T16:21:28.643682+0000","last_clean":"2024-04-16T16:21:28.643682+0000","last_became_active":"2024-04-16T16:21:26.623344+0000","last_became_peered":"2024-04-16T16:21:26.623344+0000","last_unstale":"2024-04-16T16:21:28.643682+0000","last_undegraded":"2024-04-16T16:21:28.643682+0000","last_fullsized":"2024-04-16T16:21:28.643682+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:36:11.193195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643714+0000","last_change":"2024-04-16T16:21:26.636365+0000","last_active":"2024-04-16T16:21:28.643714+0000","last_peered":"2024-04-16T16:21:28.643714+0000","last_clean":"2024-04-16T16:21:28.643714+0000","last_became_active":"2024-04-16T16:21:26.636019+0000","last_became_peered":"2024-04-16T16:21:26.636019+0000","last_unstale":"2024-04-16T16:21:28.643714+0000","last_undegraded":"2024-04-16T16:21:28.643714+0000","last_fullsized":"2024-04-16T16:21:28.643714+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:12.380057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643829+0000","last_change":"2024-04-16T16:21:26.622683+0000","last_active":"2024-04-16T16:21:28.643829+0000","last_peered":"2024-04-16T16:21:28.643829+0000","last_clean":"2024-04-16T16:21:28.643829+0000","last_became_active":"2024-04-16T16:21:26.622308+0000","last_became_peered":"2024-04-16T16:21:26.622308+0000","last_unstale":"2024-04-16T16:21:28.643829+0000","last_undegraded":"2024-04-16T16:21:28.643829+0000","last_fullsized":"2024-04-16T16:21:28.643829+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:26.812700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.568474+0000","last_change":"2024-04-16T16:21:24.568474+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.568474+0000","last_undegraded":"2024-04-16T16:21:24.568474+0000","last_fullsized":"2024-04-16T16:21:24.568474+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.566193+0000","last_change":"2024-04-16T16:21:24.566193+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.566193+0000","last_undegraded":"2024-04-16T16:21:24.566193+0000","last_fullsized":"2024-04-16T16:21:24.566193+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.578608+0000","last_change":"2024-04-16T16:21:24.578608+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.578608+0000","last_undegraded":"2024-04-16T16:21:24.578608+0000","last_fullsized":"2024-04-16T16:21:24.578608+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.578075+0000","last_change":"2024-04-16T16:21:24.578075+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.578075+0000","last_undegraded":"2024-04-16T16:21:24.578075+0000","last_fullsized":"2024-04-16T16:21:24.578075+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1b","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644284+0000","last_change":"2024-04-16T16:21:25.588337+0000","last_active":"2024-04-16T16:21:28.644284+0000","last_peered":"2024-04-16T16:21:28.644284+0000","last_clean":"2024-04-16T16:21:28.644284+0000","last_became_active":"2024-04-16T16:21:25.583483+0000","last_became_peered":"2024-04-16T16:21:25.583483+0000","last_unstale":"2024-04-16T16:21:28.644284+0000","last_undegraded":"2024-04-16T16:21:28.644284+0000","last_fullsized":"2024-04-16T16:21:28.644284+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:10:51.761699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.565860+0000","last_change":"2024-04-16T16:21:24.565860+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.565860+0000","last_undegraded":"2024-04-16T16:21:24.565860+0000","last_fullsized":"2024-04-16T16:21:24.565860+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643197+0000","last_change":"2024-04-16T16:21:25.595882+0000","last_active":"2024-04-16T16:21:28.643197+0000","last_peered":"2024-04-16T16:21:28.643197+0000","last_clean":"2024-04-16T16:21:28.643197+0000","last_became_active":"2024-04-16T16:21:25.595359+0000","last_became_peered":"2024-04-16T16:21:25.595359+0000","last_unstale":"2024-04-16T16:21:28.643197+0000","last_undegraded":"2024-04-16T16:21:28.643197+0000","last_fullsized":"2024-04-16T16:21:28.643197+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:53.324244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.577970+0000","last_change":"2024-04-16T16:21:24.577970+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.577970+0000","last_undegraded":"2024-04-16T16:21:24.577970+0000","last_fullsized":"2024-04-16T16:21:24.577970+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.565010+0000","last_change":"2024-04-16T16:21:24.565010+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.565010+0000","last_undegraded":"2024-04-16T16:21:24.565010+0000","last_fullsized":"2024-04-16T16:21:24.565010+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.565036+0000","last_change":"2024-04-16T16:21:24.565036+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.565036+0000","last_undegraded":"2024-04-16T16:21:24.565036+0000","last_fullsized":"2024-04-16T16:21:24.565036+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644198+0000","last_change":"2024-04-16T16:21:26.618860+0000","last_active":"2024-04-16T16:21:28.644198+0000","last_peered":"2024-04-16T16:21:28.644198+0000","last_clean":"2024-04-16T16:21:28.644198+0000","last_became_active":"2024-04-16T16:21:26.618520+0000","last_became_peered":"2024-04-16T16:21:26.618520+0000","last_unstale":"2024-04-16T16:21:28.644198+0000","last_undegraded":"2024-04-16T16:21:28.644198+0000","last_fullsized":"2024-04-16T16:21:28.644198+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:13.744613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.571965+0000","last_change":"2024-04-16T16:21:24.571965+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.571965+0000","last_undegraded":"2024-04-16T16:21:24.571965+0000","last_fullsized":"2024-04-16T16:21:24.571965+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.571669+0000","last_change":"2024-04-16T16:21:24.571669+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.571669+0000","last_undegraded":"2024-04-16T16:21:24.571669+0000","last_fullsized":"2024-04-16T16:21:24.571669+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.570182+0000","last_change":"2024-04-16T16:21:24.570182+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.570182+0000","last_undegraded":"2024-04-16T16:21:24.570182+0000","last_fullsized":"2024-04-16T16:21:24.570182+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.571981+0000","last_change":"2024-04-16T16:21:24.571981+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.571981+0000","last_undegraded":"2024-04-16T16:21:24.571981+0000","last_fullsized":"2024-04-16T16:21:24.571981+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.10","version":"141'8","reported_seq":22,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643881+0000","last_change":"2024-04-16T16:21:25.600083+0000","last_active":"2024-04-16T16:21:28.643881+0000","last_peered":"2024-04-16T16:21:28.643881+0000","last_clean":"2024-04-16T16:21:28.643881+0000","last_became_active":"2024-04-16T16:21:25.599688+0000","last_became_peered":"2024-04-16T16:21:25.599688+0000","last_unstale":"2024-04-16T16:21:28.643881+0000","last_undegraded":"2024-04-16T16:21:28.643881+0000","last_fullsized":"2024-04-16T16:21:28.643881+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:31.123443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564359+0000","last_change":"2024-04-16T16:21:24.564359+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564359+0000","last_undegraded":"2024-04-16T16:21:24.564359+0000","last_fullsized":"2024-04-16T16:21:24.564359+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643880+0000","last_change":"2024-04-16T16:21:26.623525+0000","last_active":"2024-04-16T16:21:28.643880+0000","last_peered":"2024-04-16T16:21:28.643880+0000","last_clean":"2024-04-16T16:21:28.643880+0000","last_became_active":"2024-04-16T16:21:26.620561+0000","last_became_peered":"2024-04-16T16:21:26.620561+0000","last_unstale":"2024-04-16T16:21:28.643880+0000","last_undegraded":"2024-04-16T16:21:28.643880+0000","last_fullsized":"2024-04-16T16:21:28.643880+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:27.435258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564908+0000","last_change":"2024-04-16T16:21:24.564908+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564908+0000","last_undegraded":"2024-04-16T16:21:24.564908+0000","last_fullsized":"2024-04-16T16:21:24.564908+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.d","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644128+0000","last_change":"2024-04-16T16:21:25.586171+0000","last_active":"2024-04-16T16:21:28.644128+0000","last_peered":"2024-04-16T16:21:28.644128+0000","last_clean":"2024-04-16T16:21:28.644128+0000","last_became_active":"2024-04-16T16:21:25.585863+0000","last_became_peered":"2024-04-16T16:21:25.585863+0000","last_unstale":"2024-04-16T16:21:28.644128+0000","last_undegraded":"2024-04-16T16:21:28.644128+0000","last_fullsized":"2024-04-16T16:21:28.644128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:28:09.057461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564985+0000","last_change":"2024-04-16T16:21:24.564985+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564985+0000","last_undegraded":"2024-04-16T16:21:24.564985+0000","last_fullsized":"2024-04-16T16:21:24.564985+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.573816+0000","last_change":"2024-04-16T16:21:24.573816+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.573816+0000","last_undegraded":"2024-04-16T16:21:24.573816+0000","last_fullsized":"2024-04-16T16:21:24.573816+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564374+0000","last_change":"2024-04-16T16:21:24.564374+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564374+0000","last_undegraded":"2024-04-16T16:21:24.564374+0000","last_fullsized":"2024-04-16T16:21:24.564374+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564355+0000","last_change":"2024-04-16T16:21:24.564355+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564355+0000","last_undegraded":"2024-04-16T16:21:24.564355+0000","last_fullsized":"2024-04-16T16:21:24.564355+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.570375+0000","last_change":"2024-04-16T16:21:24.570375+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.570375+0000","last_undegraded":"2024-04-16T16:21:24.570375+0000","last_fullsized":"2024-04-16T16:21:24.570375+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564516+0000","last_change":"2024-04-16T16:21:24.564516+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564516+0000","last_undegraded":"2024-04-16T16:21:24.564516+0000","last_fullsized":"2024-04-16T16:21:24.564516+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.572258+0000","last_change":"2024-04-16T16:21:24.572258+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.572258+0000","last_undegraded":"2024-04-16T16:21:24.572258+0000","last_fullsized":"2024-04-16T16:21:24.572258+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.572392+0000","last_change":"2024-04-16T16:21:24.572392+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.572392+0000","last_undegraded":"2024-04-16T16:21:24.572392+0000","last_fullsized":"2024-04-16T16:21:24.572392+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.574026+0000","last_change":"2024-04-16T16:21:24.574026+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.574026+0000","last_undegraded":"2024-04-16T16:21:24.574026+0000","last_fullsized":"2024-04-16T16:21:24.574026+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643444+0000","last_change":"2024-04-16T16:21:26.629185+0000","last_active":"2024-04-16T16:21:28.643444+0000","last_peered":"2024-04-16T16:21:28.643444+0000","last_clean":"2024-04-16T16:21:28.643444+0000","last_became_active":"2024-04-16T16:21:26.628800+0000","last_became_peered":"2024-04-16T16:21:26.628800+0000","last_unstale":"2024-04-16T16:21:28.643444+0000","last_undegraded":"2024-04-16T16:21:28.643444+0000","last_fullsized":"2024-04-16T16:21:28.643444+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:56.310523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.571269+0000","last_change":"2024-04-16T16:21:24.571269+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.571269+0000","last_undegraded":"2024-04-16T16:21:24.571269+0000","last_fullsized":"2024-04-16T16:21:24.571269+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.565821+0000","last_change":"2024-04-16T16:21:24.565821+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.565821+0000","last_undegraded":"2024-04-16T16:21:24.565821+0000","last_fullsized":"2024-04-16T16:21:24.565821+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.570696+0000","last_change":"2024-04-16T16:21:24.570696+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.570696+0000","last_undegraded":"2024-04-16T16:21:24.570696+0000","last_fullsized":"2024-04-16T16:21:24.570696+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564958+0000","last_change":"2024-04-16T16:21:24.564958+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564958+0000","last_undegraded":"2024-04-16T16:21:24.564958+0000","last_fullsized":"2024-04-16T16:21:24.564958+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.570853+0000","last_change":"2024-04-16T16:21:24.570853+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.570853+0000","last_undegraded":"2024-04-16T16:21:24.570853+0000","last_fullsized":"2024-04-16T16:21:24.570853+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"130'4669","reported_seq":5277,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-16T16:21:24.571361+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:21:24.571361+0000","last_peered":"2024-04-16T16:21:24.571361+0000","last_clean":"2024-04-16T16:21:24.571361+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:21:24.571361+0000","last_undegraded":"2024-04-16T16:21:24.571361+0000","last_fullsized":"2024-04-16T16:21:24.571361+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1083,"num_read_kb":2113,"num_write":347,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":1},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1083,"num_read_kb":2113,"num_write":347,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4669,"ondisk_log_size":4669,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33460,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338380,"statfs":{"total":96636764160,"available":96602501120,"internally_reserved":0,"allocated":2334720,"data_stored":1416329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574942,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1740800,"data_stored":825961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1740800,"data_stored":825961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574943,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1740800,"data_stored":825961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1728512,"data_stored":820144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574942,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37096,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334744,"statfs":{"total":96636764160,"available":96598777856,"internally_reserved":0,"allocated":1794048,"data_stored":846993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1740800,"data_stored":825961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37612,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334228,"statfs":{"total":96636764160,"available":96598249472,"internally_reserved":0,"allocated":2322432,"data_stored":1410512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:21:31.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:21:31.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.381+0000 7f89fbe00640 1 Processor -- start 2024-04-16T16:21:31.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.382+0000 7f89fbe00640 1 -- start start 2024-04-16T16:21:31.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.383+0000 7f89fbe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f89fc161e90 0x7f89fc0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:31.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.383+0000 7f89fbe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f89fc162270 0x7f89fc0d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:31.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.383+0000 7f89fbe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89fc0d3080 con 0x7f89fc0d52d0 2024-04-16T16:21:31.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.383+0000 7f89fbe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f89fc0d2e30 con 0x7f89fc161e90 2024-04-16T16:21:31.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.383+0000 7f89fbe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f89fc0d3360 con 0x7f89fc162270 2024-04-16T16:21:31.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.383+0000 7f89fb400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f89fc0d52d0 0x7f89fc06ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60546/0 (socket says 172.21.15.45:60546) 2024-04-16T16:21:31.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.383+0000 7f89faa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f89fc161e90 0x7f89fc0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:31.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.383+0000 7f89fb400640 1 -- 172.21.15.45:0/3052987934 learned_addr learned my addr 172.21.15.45:0/3052987934 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:31.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.384+0000 7f89f2000640 1 --2- 172.21.15.45:0/3052987934 >> v2:172.21.15.103:3300/0 conn(0x7f89fc162270 0x7f89fc0d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:31.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.384+0000 7f89faa00640 1 -- 172.21.15.45:0/3052987934 >> v2:172.21.15.103:3300/0 conn(0x7f89fc162270 msgr2=0x7f89fc0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:31.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.384+0000 7f89faa00640 1 --2- 172.21.15.45:0/3052987934 >> v2:172.21.15.103:3300/0 conn(0x7f89fc162270 0x7f89fc0d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.384+0000 7f89faa00640 1 -- 172.21.15.45:0/3052987934 >> v1:172.21.15.45:6789/0 conn(0x7f89fc0d52d0 legacy=0x7f89fc06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:31.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.384+0000 7f89faa00640 1 -- 172.21.15.45:0/3052987934 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89fc0d3640 con 0x7f89fc161e90 2024-04-16T16:21:31.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.385+0000 7f89faa00640 1 --2- 172.21.15.45:0/3052987934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89fc161e90 0x7f89fc0cf7d0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f89ec0097a0 tx=0x7f89ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1bce269a8953d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:31.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.385+0000 7f89fa000640 1 -- 172.21.15.45:0/3052987934 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec035020 con 0x7f89fc161e90 2024-04-16T16:21:31.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.385+0000 7f89fa000640 1 -- 172.21.15.45:0/3052987934 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89ec031830 con 0x7f89fc161e90 2024-04-16T16:21:31.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.385+0000 7f89fa000640 1 -- 172.21.15.45:0/3052987934 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec031c00 con 0x7f89fc161e90 2024-04-16T16:21:31.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.386+0000 7f89fbe00640 1 -- 172.21.15.45:0/3052987934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89fc161e90 msgr2=0x7f89fc0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:31.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.386+0000 7f89fbe00640 1 --2- 172.21.15.45:0/3052987934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89fc161e90 0x7f89fc0cf7d0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f89ec0097a0 tx=0x7f89ec02b8d0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.386+0000 7f89fbe00640 1 -- 172.21.15.45:0/3052987934 shutdown_connections 2024-04-16T16:21:31.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.386+0000 7f89fbe00640 1 --2- 172.21.15.45:0/3052987934 >> v2:172.21.15.103:3300/0 conn(0x7f89fc162270 0x7f89fc0d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.386+0000 7f89fbe00640 1 --2- 172.21.15.45:0/3052987934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89fc161e90 0x7f89fc0cf7d0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.386+0000 7f89fbe00640 1 -- 172.21.15.45:0/3052987934 >> 172.21.15.45:0/3052987934 conn(0x7f89fc0d1940 msgr2=0x7f89fc0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:31.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.386+0000 7f89fbe00640 1 -- 172.21.15.45:0/3052987934 shutdown_connections 2024-04-16T16:21:31.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.386+0000 7f89fbe00640 1 -- 172.21.15.45:0/3052987934 wait complete. 2024-04-16T16:21:31.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.388+0000 7f89fbe00640 1 Processor -- start 2024-04-16T16:21:31.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.388+0000 7f89fbe00640 1 -- start start 2024-04-16T16:21:31.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.389+0000 7f89fbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89fc0d52d0 0x7f89fc14c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:31.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.389+0000 7f89fbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89fc161e90 0x7f89fc0cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:31.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.389+0000 7f89fbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89fc162270 0x7f89fc1e73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:31.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.389+0000 7f89fbe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc06d550 con 0x7f89fc162270 2024-04-16T16:21:31.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.389+0000 7f89fbe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc0cfd10 con 0x7f89fc161e90 2024-04-16T16:21:31.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.389+0000 7f89fbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f89fc06d6d0 con 0x7f89fc0d52d0 2024-04-16T16:21:31.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.389+0000 7f89fb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89fc162270 0x7f89fc1e73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:31.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.390+0000 7f89fb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89fc162270 0x7f89fc1e73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59010/0 (socket says 172.21.15.45:59010) 2024-04-16T16:21:31.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.390+0000 7f89fb400640 1 -- 172.21.15.45:0/4233904925 learned_addr learned my addr 172.21.15.45:0/4233904925 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:31.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.390+0000 7f89faa00640 1 --2- 172.21.15.45:0/4233904925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89fc0d52d0 0x7f89fc14c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:31.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.390+0000 7f89f2000640 1 --2- 172.21.15.45:0/4233904925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89fc161e90 0x7f89fc0cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:31.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.390+0000 7f89fb400640 1 -- 172.21.15.45:0/4233904925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89fc0d52d0 msgr2=0x7f89fc14c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:31.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.390+0000 7f89fb400640 1 --2- 172.21.15.45:0/4233904925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89fc0d52d0 0x7f89fc14c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.390+0000 7f89fb400640 1 -- 172.21.15.45:0/4233904925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89fc161e90 msgr2=0x7f89fc0cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:31.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.390+0000 7f89fb400640 1 --2- 172.21.15.45:0/4233904925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89fc161e90 0x7f89fc0cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.390+0000 7f89fb400640 1 -- 172.21.15.45:0/4233904925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89fc0ce600 con 0x7f89fc162270 2024-04-16T16:21:31.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.390+0000 7f89fb400640 1 --2- 172.21.15.45:0/4233904925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89fc162270 0x7f89fc1e73a0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f89f400cad0 tx=0x7f89f400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:31.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.391+0000 7f89f3400640 1 -- 172.21.15.45:0/4233904925 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f4017070 con 0x7f89fc162270 2024-04-16T16:21:31.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.391+0000 7f89f3400640 1 -- 172.21.15.45:0/4233904925 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89f400ac30 con 0x7f89fc162270 2024-04-16T16:21:31.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.391+0000 7f89f3400640 1 -- 172.21.15.45:0/4233904925 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f4012400 con 0x7f89fc162270 2024-04-16T16:21:31.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.391+0000 7f89fbe00640 1 -- 172.21.15.45:0/4233904925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89fc0cf4e0 con 0x7f89fc162270 2024-04-16T16:21:31.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.391+0000 7f89fbe00640 1 -- 172.21.15.45:0/4233904925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89fc14d3e0 con 0x7f89fc162270 2024-04-16T16:21:31.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.392+0000 7f89fbe00640 1 -- 172.21.15.45:0/4233904925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89fc075780 con 0x7f89fc162270 2024-04-16T16:21:31.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.393+0000 7f89f3400640 1 -- 172.21.15.45:0/4233904925 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f89f401a070 con 0x7f89fc162270 2024-04-16T16:21:31.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.394+0000 7f89f3400640 1 --2- 172.21.15.45:0/4233904925 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f89d806ca10 0x7f89d806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:31.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.394+0000 7f89f3400640 1 -- 172.21.15.45:0/4233904925 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7f89f408ba20 con 0x7f89fc162270 2024-04-16T16:21:31.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.395+0000 7f89faa00640 1 --2- 172.21.15.45:0/4233904925 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f89d806ca10 0x7f89d806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:31.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.396+0000 7f89faa00640 1 --2- 172.21.15.45:0/4233904925 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f89d806ca10 0x7f89d806eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f89ec036000 tx=0x7f89ec008030 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:31.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.398+0000 7f89f3400640 1 -- 172.21.15.45:0/4233904925 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f89fc0cf4e0 con 0x7f89fc162270 2024-04-16T16:21:31.521 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-16T16:21:31.521+0000 7f0d31200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:21:31.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.599+0000 7f89fbe00640 1 -- 172.21.15.45:0/4233904925 --> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f89fc0d4bf0 con 0x7f89d806ca10 2024-04-16T16:21:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.617+0000 7f89f3400640 1 -- 172.21.15.45:0/4233904925 <== mgr.7153 v2:172.21.15.103:6832/1478535274 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+241854 (secure 0 0 0) 0x7f89ec0840d0 con 0x7f89d806ca10 2024-04-16T16:21:31.618 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:21:31.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.624+0000 7f89fbe00640 1 -- 172.21.15.45:0/4233904925 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f89d806ca10 msgr2=0x7f89d806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:31.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.624+0000 7f89fbe00640 1 --2- 172.21.15.45:0/4233904925 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f89d806ca10 0x7f89d806eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f89ec036000 tx=0x7f89ec008030 comp rx=0 tx=0).stop 2024-04-16T16:21:31.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.624+0000 7f89fbe00640 1 -- 172.21.15.45:0/4233904925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89fc162270 msgr2=0x7f89fc1e73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:31.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.624+0000 7f89fbe00640 1 --2- 172.21.15.45:0/4233904925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89fc162270 0x7f89fc1e73a0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f89f400cad0 tx=0x7f89f400cf90 comp rx=0 tx=0).stop 2024-04-16T16:21:31.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.624+0000 7f89fbe00640 1 -- 172.21.15.45:0/4233904925 shutdown_connections 2024-04-16T16:21:31.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.624+0000 7f89fbe00640 1 --2- 172.21.15.45:0/4233904925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89fc162270 0x7f89fc1e73a0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.624+0000 7f89fbe00640 1 --2- 172.21.15.45:0/4233904925 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f89d806ca10 0x7f89d806eed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.624+0000 7f89fbe00640 1 --2- 172.21.15.45:0/4233904925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89fc161e90 0x7f89fc0cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.624+0000 7f89fbe00640 1 --2- 172.21.15.45:0/4233904925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f89fc0d52d0 0x7f89fc14c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.624+0000 7f89fbe00640 1 -- 172.21.15.45:0/4233904925 >> 172.21.15.45:0/4233904925 conn(0x7f89fc0d1940 msgr2=0x7f89fc0ca320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:31.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.624+0000 7f89fbe00640 1 -- 172.21.15.45:0/4233904925 shutdown_connections 2024-04-16T16:21:31.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.624+0000 7f89fbe00640 1 -- 172.21.15.45:0/4233904925 wait complete. 2024-04-16T16:21:31.643 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-04-16T16:21:30.390379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1083,"num_read_kb":2113,"num_write":360,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4679,"ondisk_log_size":4679,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":272564,"kb_used_data":14788,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754702156,"statfs":{"total":773094113280,"available":772815007744,"internally_reserved":0,"allocated":15142912,"data_stored":7797822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.830661"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643268+0000","last_change":"2024-04-16T16:21:26.619336+0000","last_active":"2024-04-16T16:21:28.643268+0000","last_peered":"2024-04-16T16:21:28.643268+0000","last_clean":"2024-04-16T16:21:28.643268+0000","last_became_active":"2024-04-16T16:21:26.618973+0000","last_became_peered":"2024-04-16T16:21:26.618973+0000","last_unstale":"2024-04-16T16:21:28.643268+0000","last_undegraded":"2024-04-16T16:21:28.643268+0000","last_fullsized":"2024-04-16T16:21:28.643268+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:56.732655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643963+0000","last_change":"2024-04-16T16:21:26.618141+0000","last_active":"2024-04-16T16:21:28.643963+0000","last_peered":"2024-04-16T16:21:28.643963+0000","last_clean":"2024-04-16T16:21:28.643963+0000","last_became_active":"2024-04-16T16:21:26.614289+0000","last_became_peered":"2024-04-16T16:21:26.614289+0000","last_unstale":"2024-04-16T16:21:28.643963+0000","last_undegraded":"2024-04-16T16:21:28.643963+0000","last_fullsized":"2024-04-16T16:21:28.643963+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:15:22.005025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643682+0000","last_change":"2024-04-16T16:21:26.623659+0000","last_active":"2024-04-16T16:21:28.643682+0000","last_peered":"2024-04-16T16:21:28.643682+0000","last_clean":"2024-04-16T16:21:28.643682+0000","last_became_active":"2024-04-16T16:21:26.623344+0000","last_became_peered":"2024-04-16T16:21:26.623344+0000","last_unstale":"2024-04-16T16:21:28.643682+0000","last_undegraded":"2024-04-16T16:21:28.643682+0000","last_fullsized":"2024-04-16T16:21:28.643682+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:36:11.193195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643714+0000","last_change":"2024-04-16T16:21:26.636365+0000","last_active":"2024-04-16T16:21:28.643714+0000","last_peered":"2024-04-16T16:21:28.643714+0000","last_clean":"2024-04-16T16:21:28.643714+0000","last_became_active":"2024-04-16T16:21:26.636019+0000","last_became_peered":"2024-04-16T16:21:26.636019+0000","last_unstale":"2024-04-16T16:21:28.643714+0000","last_undegraded":"2024-04-16T16:21:28.643714+0000","last_fullsized":"2024-04-16T16:21:28.643714+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:12.380057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643829+0000","last_change":"2024-04-16T16:21:26.622683+0000","last_active":"2024-04-16T16:21:28.643829+0000","last_peered":"2024-04-16T16:21:28.643829+0000","last_clean":"2024-04-16T16:21:28.643829+0000","last_became_active":"2024-04-16T16:21:26.622308+0000","last_became_peered":"2024-04-16T16:21:26.622308+0000","last_unstale":"2024-04-16T16:21:28.643829+0000","last_undegraded":"2024-04-16T16:21:28.643829+0000","last_fullsized":"2024-04-16T16:21:28.643829+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:26.812700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.568474+0000","last_change":"2024-04-16T16:21:24.568474+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.568474+0000","last_undegraded":"2024-04-16T16:21:24.568474+0000","last_fullsized":"2024-04-16T16:21:24.568474+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.566193+0000","last_change":"2024-04-16T16:21:24.566193+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.566193+0000","last_undegraded":"2024-04-16T16:21:24.566193+0000","last_fullsized":"2024-04-16T16:21:24.566193+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.578608+0000","last_change":"2024-04-16T16:21:24.578608+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.578608+0000","last_undegraded":"2024-04-16T16:21:24.578608+0000","last_fullsized":"2024-04-16T16:21:24.578608+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.578075+0000","last_change":"2024-04-16T16:21:24.578075+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.578075+0000","last_undegraded":"2024-04-16T16:21:24.578075+0000","last_fullsized":"2024-04-16T16:21:24.578075+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1b","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644284+0000","last_change":"2024-04-16T16:21:25.588337+0000","last_active":"2024-04-16T16:21:28.644284+0000","last_peered":"2024-04-16T16:21:28.644284+0000","last_clean":"2024-04-16T16:21:28.644284+0000","last_became_active":"2024-04-16T16:21:25.583483+0000","last_became_peered":"2024-04-16T16:21:25.583483+0000","last_unstale":"2024-04-16T16:21:28.644284+0000","last_undegraded":"2024-04-16T16:21:28.644284+0000","last_fullsized":"2024-04-16T16:21:28.644284+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:10:51.761699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.565860+0000","last_change":"2024-04-16T16:21:24.565860+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.565860+0000","last_undegraded":"2024-04-16T16:21:24.565860+0000","last_fullsized":"2024-04-16T16:21:24.565860+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643197+0000","last_change":"2024-04-16T16:21:25.595882+0000","last_active":"2024-04-16T16:21:28.643197+0000","last_peered":"2024-04-16T16:21:28.643197+0000","last_clean":"2024-04-16T16:21:28.643197+0000","last_became_active":"2024-04-16T16:21:25.595359+0000","last_became_peered":"2024-04-16T16:21:25.595359+0000","last_unstale":"2024-04-16T16:21:28.643197+0000","last_undegraded":"2024-04-16T16:21:28.643197+0000","last_fullsized":"2024-04-16T16:21:28.643197+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:53.324244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.577970+0000","last_change":"2024-04-16T16:21:24.577970+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.577970+0000","last_undegraded":"2024-04-16T16:21:24.577970+0000","last_fullsized":"2024-04-16T16:21:24.577970+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.565010+0000","last_change":"2024-04-16T16:21:24.565010+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.565010+0000","last_undegraded":"2024-04-16T16:21:24.565010+0000","last_fullsized":"2024-04-16T16:21:24.565010+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.565036+0000","last_change":"2024-04-16T16:21:24.565036+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.565036+0000","last_undegraded":"2024-04-16T16:21:24.565036+0000","last_fullsized":"2024-04-16T16:21:24.565036+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644198+0000","last_change":"2024-04-16T16:21:26.618860+0000","last_active":"2024-04-16T16:21:28.644198+0000","last_peered":"2024-04-16T16:21:28.644198+0000","last_clean":"2024-04-16T16:21:28.644198+0000","last_became_active":"2024-04-16T16:21:26.618520+0000","last_became_peered":"2024-04-16T16:21:26.618520+0000","last_unstale":"2024-04-16T16:21:28.644198+0000","last_undegraded":"2024-04-16T16:21:28.644198+0000","last_fullsized":"2024-04-16T16:21:28.644198+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:13.744613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.571965+0000","last_change":"2024-04-16T16:21:24.571965+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.571965+0000","last_undegraded":"2024-04-16T16:21:24.571965+0000","last_fullsized":"2024-04-16T16:21:24.571965+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.571669+0000","last_change":"2024-04-16T16:21:24.571669+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.571669+0000","last_undegraded":"2024-04-16T16:21:24.571669+0000","last_fullsized":"2024-04-16T16:21:24.571669+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.570182+0000","last_change":"2024-04-16T16:21:24.570182+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.570182+0000","last_undegraded":"2024-04-16T16:21:24.570182+0000","last_fullsized":"2024-04-16T16:21:24.570182+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.571981+0000","last_change":"2024-04-16T16:21:24.571981+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.571981+0000","last_undegraded":"2024-04-16T16:21:24.571981+0000","last_fullsized":"2024-04-16T16:21:24.571981+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.10","version":"141'8","reported_seq":22,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643881+0000","last_change":"2024-04-16T16:21:25.600083+0000","last_active":"2024-04-16T16:21:28.643881+0000","last_peered":"2024-04-16T16:21:28.643881+0000","last_clean":"2024-04-16T16:21:28.643881+0000","last_became_active":"2024-04-16T16:21:25.599688+0000","last_became_peered":"2024-04-16T16:21:25.599688+0000","last_unstale":"2024-04-16T16:21:28.643881+0000","last_undegraded":"2024-04-16T16:21:28.643881+0000","last_fullsized":"2024-04-16T16:21:28.643881+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:31.123443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564359+0000","last_change":"2024-04-16T16:21:24.564359+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564359+0000","last_undegraded":"2024-04-16T16:21:24.564359+0000","last_fullsized":"2024-04-16T16:21:24.564359+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643880+0000","last_change":"2024-04-16T16:21:26.623525+0000","last_active":"2024-04-16T16:21:28.643880+0000","last_peered":"2024-04-16T16:21:28.643880+0000","last_clean":"2024-04-16T16:21:28.643880+0000","last_became_active":"2024-04-16T16:21:26.620561+0000","last_became_peered":"2024-04-16T16:21:26.620561+0000","last_unstale":"2024-04-16T16:21:28.643880+0000","last_undegraded":"2024-04-16T16:21:28.643880+0000","last_fullsized":"2024-04-16T16:21:28.643880+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:27.435258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564908+0000","last_change":"2024-04-16T16:21:24.564908+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564908+0000","last_undegraded":"2024-04-16T16:21:24.564908+0000","last_fullsized":"2024-04-16T16:21:24.564908+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.d","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644128+0000","last_change":"2024-04-16T16:21:25.586171+0000","last_active":"2024-04-16T16:21:28.644128+0000","last_peered":"2024-04-16T16:21:28.644128+0000","last_clean":"2024-04-16T16:21:28.644128+0000","last_became_active":"2024-04-16T16:21:25.585863+0000","last_became_peered":"2024-04-16T16:21:25.585863+0000","last_unstale":"2024-04-16T16:21:28.644128+0000","last_undegraded":"2024-04-16T16:21:28.644128+0000","last_fullsized":"2024-04-16T16:21:28.644128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:28:09.057461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564985+0000","last_change":"2024-04-16T16:21:24.564985+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564985+0000","last_undegraded":"2024-04-16T16:21:24.564985+0000","last_fullsized":"2024-04-16T16:21:24.564985+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.573816+0000","last_change":"2024-04-16T16:21:24.573816+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.573816+0000","last_undegraded":"2024-04-16T16:21:24.573816+0000","last_fullsized":"2024-04-16T16:21:24.573816+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564374+0000","last_change":"2024-04-16T16:21:24.564374+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564374+0000","last_undegraded":"2024-04-16T16:21:24.564374+0000","last_fullsized":"2024-04-16T16:21:24.564374+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564355+0000","last_change":"2024-04-16T16:21:24.564355+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564355+0000","last_undegraded":"2024-04-16T16:21:24.564355+0000","last_fullsized":"2024-04-16T16:21:24.564355+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.570375+0000","last_change":"2024-04-16T16:21:24.570375+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.570375+0000","last_undegraded":"2024-04-16T16:21:24.570375+0000","last_fullsized":"2024-04-16T16:21:24.570375+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564516+0000","last_change":"2024-04-16T16:21:24.564516+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564516+0000","last_undegraded":"2024-04-16T16:21:24.564516+0000","last_fullsized":"2024-04-16T16:21:24.564516+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.572258+0000","last_change":"2024-04-16T16:21:24.572258+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.572258+0000","last_undegraded":"2024-04-16T16:21:24.572258+0000","last_fullsized":"2024-04-16T16:21:24.572258+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.572392+0000","last_change":"2024-04-16T16:21:24.572392+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.572392+0000","last_undegraded":"2024-04-16T16:21:24.572392+0000","last_fullsized":"2024-04-16T16:21:24.572392+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.574026+0000","last_change":"2024-04-16T16:21:24.574026+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.574026+0000","last_undegraded":"2024-04-16T16:21:24.574026+0000","last_fullsized":"2024-04-16T16:21:24.574026+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643444+0000","last_change":"2024-04-16T16:21:26.629185+0000","last_active":"2024-04-16T16:21:28.643444+0000","last_peered":"2024-04-16T16:21:28.643444+0000","last_clean":"2024-04-16T16:21:28.643444+0000","last_became_active":"2024-04-16T16:21:26.628800+0000","last_became_peered":"2024-04-16T16:21:26.628800+0000","last_unstale":"2024-04-16T16:21:28.643444+0000","last_undegraded":"2024-04-16T16:21:28.643444+0000","last_fullsized":"2024-04-16T16:21:28.643444+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:56.310523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.571269+0000","last_change":"2024-04-16T16:21:24.571269+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.571269+0000","last_undegraded":"2024-04-16T16:21:24.571269+0000","last_fullsized":"2024-04-16T16:21:24.571269+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.565821+0000","last_change":"2024-04-16T16:21:24.565821+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.565821+0000","last_undegraded":"2024-04-16T16:21:24.565821+0000","last_fullsized":"2024-04-16T16:21:24.565821+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.570696+0000","last_change":"2024-04-16T16:21:24.570696+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.570696+0000","last_undegraded":"2024-04-16T16:21:24.570696+0000","last_fullsized":"2024-04-16T16:21:24.570696+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564958+0000","last_change":"2024-04-16T16:21:24.564958+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564958+0000","last_undegraded":"2024-04-16T16:21:24.564958+0000","last_fullsized":"2024-04-16T16:21:24.564958+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.570853+0000","last_change":"2024-04-16T16:21:24.570853+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.570853+0000","last_undegraded":"2024-04-16T16:21:24.570853+0000","last_fullsized":"2024-04-16T16:21:24.570853+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"130'4669","reported_seq":5277,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-16T16:21:24.571361+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:21:24.571361+0000","last_peered":"2024-04-16T16:21:24.571361+0000","last_clean":"2024-04-16T16:21:24.571361+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:21:24.571361+0000","last_undegraded":"2024-04-16T16:21:24.571361+0000","last_fullsized":"2024-04-16T16:21:24.571361+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1083,"num_read_kb":2113,"num_write":347,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":1},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1083,"num_read_kb":2113,"num_write":347,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4669,"ondisk_log_size":4669,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33460,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338380,"statfs":{"total":96636764160,"available":96602501120,"internally_reserved":0,"allocated":2334720,"data_stored":1416329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574942,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1740800,"data_stored":825961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1740800,"data_stored":825961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574943,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1740800,"data_stored":825961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1728512,"data_stored":820144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574942,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37096,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334744,"statfs":{"total":96636764160,"available":96598777856,"internally_reserved":0,"allocated":1794048,"data_stored":846993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1740800,"data_stored":825961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37612,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334228,"statfs":{"total":96636764160,"available":96598249472,"internally_reserved":0,"allocated":2322432,"data_stored":1410512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:21:31.647 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:21:31.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.903+0000 7f191b200640 1 Processor -- start 2024-04-16T16:21:31.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.903+0000 7f191b200640 1 -- start start 2024-04-16T16:21:31.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.905+0000 7f191b200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f19140ce9a0 0x7f19140cb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:31.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.905+0000 7f191b200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f191415ecd0 con 0x7f191415e8f0 2024-04-16T16:21:31.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.905+0000 7f191b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19140cd6d0 con 0x7f19140ccb20 2024-04-16T16:21:31.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.905+0000 7f191b200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f19140ce2e0 con 0x7f19140ce9a0 2024-04-16T16:21:31.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.905+0000 7f191a800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f19140ce9a0 0x7f19140cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:31.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.905+0000 7f1919e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f191415e8f0 0x7f19140cc410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60550/0 (socket says 172.21.15.45:60550) 2024-04-16T16:21:31.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.905+0000 7f1919e00640 1 -- 172.21.15.45:0/622710639 learned_addr learned my addr 172.21.15.45:0/622710639 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:31.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.906+0000 7f191a800640 1 -- 172.21.15.45:0/622710639 >> v1:172.21.15.103:6789/0 conn(0x7f19140ccb20 legacy=0x7f19140cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:31.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.906+0000 7f191a800640 1 -- 172.21.15.45:0/622710639 >> v1:172.21.15.45:6789/0 conn(0x7f191415e8f0 legacy=0x7f19140cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:31.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.906+0000 7f191a800640 1 -- 172.21.15.45:0/622710639 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19140cd9b0 con 0x7f19140ce9a0 2024-04-16T16:21:31.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.907+0000 7f191a800640 1 --2- 172.21.15.45:0/622710639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19140ce9a0 0x7f19140cb340 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f1910009d90 tx=0x7f191002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ba20252641ea49 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:31.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.907+0000 7f1918a00640 1 -- 172.21.15.45:0/622710639 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910039070 con 0x7f19140ce9a0 2024-04-16T16:21:31.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.907+0000 7f1918a00640 1 -- 172.21.15.45:0/622710639 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f191002cbd0 con 0x7f19140ce9a0 2024-04-16T16:21:31.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.907+0000 7f1918a00640 1 -- 172.21.15.45:0/622710639 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910034440 con 0x7f19140ce9a0 2024-04-16T16:21:31.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.908+0000 7f191b200640 1 -- 172.21.15.45:0/622710639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19140ce9a0 msgr2=0x7f19140cb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:31.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.908+0000 7f191b200640 1 --2- 172.21.15.45:0/622710639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19140ce9a0 0x7f19140cb340 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f1910009d90 tx=0x7f191002be20 comp rx=0 tx=0).stop 2024-04-16T16:21:31.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.908+0000 7f191b200640 1 -- 172.21.15.45:0/622710639 shutdown_connections 2024-04-16T16:21:31.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.908+0000 7f191b200640 1 --2- 172.21.15.45:0/622710639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19140ce9a0 0x7f19140cb340 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.908+0000 7f191b200640 1 -- 172.21.15.45:0/622710639 >> 172.21.15.45:0/622710639 conn(0x7f191406b930 msgr2=0x7f191406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:31.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.908+0000 7f191b200640 1 -- 172.21.15.45:0/622710639 shutdown_connections 2024-04-16T16:21:31.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.908+0000 7f191b200640 1 -- 172.21.15.45:0/622710639 wait complete. 2024-04-16T16:21:31.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.910+0000 7f191b200640 1 Processor -- start 2024-04-16T16:21:31.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.910+0000 7f191b200640 1 -- start start 2024-04-16T16:21:31.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.911+0000 7f191b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19140ccb20 0x7f1914142ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:31.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.911+0000 7f191b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19140ce9a0 0x7f19141f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:31.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.911+0000 7f191b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f191415e8f0 0x7f191415ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:31.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.911+0000 7f191b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f19140cd360 con 0x7f19140ccb20 2024-04-16T16:21:31.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.911+0000 7f191b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f19140cdf70 con 0x7f191415e8f0 2024-04-16T16:21:31.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.911+0000 7f191b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f19140cf080 con 0x7f19140ce9a0 2024-04-16T16:21:31.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.911+0000 7f1919e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19140ccb20 0x7f1914142ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:31.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.912+0000 7f1919e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19140ccb20 0x7f1914142ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59038/0 (socket says 172.21.15.45:59038) 2024-04-16T16:21:31.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.912+0000 7f1919e00640 1 -- 172.21.15.45:0/987449068 learned_addr learned my addr 172.21.15.45:0/987449068 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:31.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.912+0000 7f1919400640 1 --2- 172.21.15.45:0/987449068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19140ce9a0 0x7f19141f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:31.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.912+0000 7f191a800640 1 --2- 172.21.15.45:0/987449068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f191415e8f0 0x7f191415ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:31.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.912+0000 7f1919e00640 1 -- 172.21.15.45:0/987449068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19140ce9a0 msgr2=0x7f19141f4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:31.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.912+0000 7f1919e00640 1 --2- 172.21.15.45:0/987449068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19140ce9a0 0x7f19141f4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.912+0000 7f1919e00640 1 -- 172.21.15.45:0/987449068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f191415e8f0 msgr2=0x7f191415ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:31.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.912+0000 7f1919e00640 1 --2- 172.21.15.45:0/987449068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f191415e8f0 0x7f191415ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:31.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.912+0000 7f1919e00640 1 -- 172.21.15.45:0/987449068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19141f5640 con 0x7f19140ccb20 2024-04-16T16:21:31.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.912+0000 7f191a800640 1 --2- 172.21.15.45:0/987449068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f191415e8f0 0x7f191415ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:31.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.912+0000 7f1919e00640 1 --2- 172.21.15.45:0/987449068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19140ccb20 0x7f1914142ed0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f190800c3f0 tx=0x7f190800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:31.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.912+0000 7f190e000640 1 -- 172.21.15.45:0/987449068 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908017070 con 0x7f19140ccb20 2024-04-16T16:21:31.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.913+0000 7f190e000640 1 -- 172.21.15.45:0/987449068 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1908005ce0 con 0x7f19140ccb20 2024-04-16T16:21:31.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.913+0000 7f191b200640 1 -- 172.21.15.45:0/987449068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19141f5920 con 0x7f19140ccb20 2024-04-16T16:21:31.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.913+0000 7f190e000640 1 -- 172.21.15.45:0/987449068 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19080072f0 con 0x7f19140ccb20 2024-04-16T16:21:31.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.913+0000 7f191b200640 1 -- 172.21.15.45:0/987449068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1914143410 con 0x7f19140ccb20 2024-04-16T16:21:31.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.914+0000 7f191b200640 1 -- 172.21.15.45:0/987449068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1914075780 con 0x7f19140ccb20 2024-04-16T16:21:31.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.915+0000 7f190e000640 1 -- 172.21.15.45:0/987449068 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f190801a070 con 0x7f19140ccb20 2024-04-16T16:21:31.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.915+0000 7f190e000640 1 --2- 172.21.15.45:0/987449068 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f18f806ca10 0x7f18f806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:31.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.916+0000 7f190e000640 1 -- 172.21.15.45:0/987449068 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7f190808afd0 con 0x7f19140ccb20 2024-04-16T16:21:31.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.916+0000 7f1919400640 1 --2- 172.21.15.45:0/987449068 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f18f806ca10 0x7f18f806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:31.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.918+0000 7f1919400640 1 --2- 172.21.15.45:0/987449068 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f18f806ca10 0x7f18f806eed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f19141f5d60 tx=0x7f1904009040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:31.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:31.920+0000 7f190e000640 1 -- 172.21.15.45:0/987449068 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1908059380 con 0x7f19140ccb20 2024-04-16T16:21:32.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.119+0000 7f191b200640 1 -- 172.21.15.45:0/987449068 --> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f19140b5b90 con 0x7f18f806ca10 2024-04-16T16:21:32.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.137+0000 7f190e000640 1 -- 172.21.15.45:0/987449068 <== mgr.7153 v2:172.21.15.103:6832/1478535274 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+241854 (secure 0 0 0) 0x7f19140b5b90 con 0x7f18f806ca10 2024-04-16T16:21:32.137 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:21:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.143+0000 7f191b200640 1 -- 172.21.15.45:0/987449068 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f18f806ca10 msgr2=0x7f18f806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.143+0000 7f191b200640 1 --2- 172.21.15.45:0/987449068 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f18f806ca10 0x7f18f806eed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f19141f5d60 tx=0x7f1904009040 comp rx=0 tx=0).stop 2024-04-16T16:21:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.143+0000 7f191b200640 1 -- 172.21.15.45:0/987449068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19140ccb20 msgr2=0x7f1914142ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.143+0000 7f191b200640 1 --2- 172.21.15.45:0/987449068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19140ccb20 0x7f1914142ed0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f190800c3f0 tx=0x7f190800c8b0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.143+0000 7f191b200640 1 -- 172.21.15.45:0/987449068 shutdown_connections 2024-04-16T16:21:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.143+0000 7f191b200640 1 --2- 172.21.15.45:0/987449068 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f18f806ca10 0x7f18f806eed0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f19141f5d60 tx=0x7f1904009040 comp rx=0 tx=0).stop 2024-04-16T16:21:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.143+0000 7f191b200640 1 --2- 172.21.15.45:0/987449068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f191415e8f0 0x7f191415ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.143+0000 7f191b200640 1 --2- 172.21.15.45:0/987449068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19140ce9a0 0x7f19141f4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.143+0000 7f191b200640 1 --2- 172.21.15.45:0/987449068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19140ccb20 0x7f1914142ed0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.143+0000 7f191b200640 1 -- 172.21.15.45:0/987449068 >> 172.21.15.45:0/987449068 conn(0x7f191406b930 msgr2=0x7f19140cabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.143+0000 7f191b200640 1 -- 172.21.15.45:0/987449068 shutdown_connections 2024-04-16T16:21:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.143+0000 7f191b200640 1 -- 172.21.15.45:0/987449068 wait complete. 2024-04-16T16:21:32.160 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-04-16T16:21:30.390379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1083,"num_read_kb":2113,"num_write":360,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4679,"ondisk_log_size":4679,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":272564,"kb_used_data":14788,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754702156,"statfs":{"total":773094113280,"available":772815007744,"internally_reserved":0,"allocated":15142912,"data_stored":7797822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.830661"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643268+0000","last_change":"2024-04-16T16:21:26.619336+0000","last_active":"2024-04-16T16:21:28.643268+0000","last_peered":"2024-04-16T16:21:28.643268+0000","last_clean":"2024-04-16T16:21:28.643268+0000","last_became_active":"2024-04-16T16:21:26.618973+0000","last_became_peered":"2024-04-16T16:21:26.618973+0000","last_unstale":"2024-04-16T16:21:28.643268+0000","last_undegraded":"2024-04-16T16:21:28.643268+0000","last_fullsized":"2024-04-16T16:21:28.643268+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:56.732655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643963+0000","last_change":"2024-04-16T16:21:26.618141+0000","last_active":"2024-04-16T16:21:28.643963+0000","last_peered":"2024-04-16T16:21:28.643963+0000","last_clean":"2024-04-16T16:21:28.643963+0000","last_became_active":"2024-04-16T16:21:26.614289+0000","last_became_peered":"2024-04-16T16:21:26.614289+0000","last_unstale":"2024-04-16T16:21:28.643963+0000","last_undegraded":"2024-04-16T16:21:28.643963+0000","last_fullsized":"2024-04-16T16:21:28.643963+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:15:22.005025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643682+0000","last_change":"2024-04-16T16:21:26.623659+0000","last_active":"2024-04-16T16:21:28.643682+0000","last_peered":"2024-04-16T16:21:28.643682+0000","last_clean":"2024-04-16T16:21:28.643682+0000","last_became_active":"2024-04-16T16:21:26.623344+0000","last_became_peered":"2024-04-16T16:21:26.623344+0000","last_unstale":"2024-04-16T16:21:28.643682+0000","last_undegraded":"2024-04-16T16:21:28.643682+0000","last_fullsized":"2024-04-16T16:21:28.643682+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:36:11.193195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643714+0000","last_change":"2024-04-16T16:21:26.636365+0000","last_active":"2024-04-16T16:21:28.643714+0000","last_peered":"2024-04-16T16:21:28.643714+0000","last_clean":"2024-04-16T16:21:28.643714+0000","last_became_active":"2024-04-16T16:21:26.636019+0000","last_became_peered":"2024-04-16T16:21:26.636019+0000","last_unstale":"2024-04-16T16:21:28.643714+0000","last_undegraded":"2024-04-16T16:21:28.643714+0000","last_fullsized":"2024-04-16T16:21:28.643714+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:12.380057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643829+0000","last_change":"2024-04-16T16:21:26.622683+0000","last_active":"2024-04-16T16:21:28.643829+0000","last_peered":"2024-04-16T16:21:28.643829+0000","last_clean":"2024-04-16T16:21:28.643829+0000","last_became_active":"2024-04-16T16:21:26.622308+0000","last_became_peered":"2024-04-16T16:21:26.622308+0000","last_unstale":"2024-04-16T16:21:28.643829+0000","last_undegraded":"2024-04-16T16:21:28.643829+0000","last_fullsized":"2024-04-16T16:21:28.643829+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:26.812700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.568474+0000","last_change":"2024-04-16T16:21:24.568474+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.568474+0000","last_undegraded":"2024-04-16T16:21:24.568474+0000","last_fullsized":"2024-04-16T16:21:24.568474+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.566193+0000","last_change":"2024-04-16T16:21:24.566193+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.566193+0000","last_undegraded":"2024-04-16T16:21:24.566193+0000","last_fullsized":"2024-04-16T16:21:24.566193+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.578608+0000","last_change":"2024-04-16T16:21:24.578608+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.578608+0000","last_undegraded":"2024-04-16T16:21:24.578608+0000","last_fullsized":"2024-04-16T16:21:24.578608+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.578075+0000","last_change":"2024-04-16T16:21:24.578075+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.578075+0000","last_undegraded":"2024-04-16T16:21:24.578075+0000","last_fullsized":"2024-04-16T16:21:24.578075+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1b","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644284+0000","last_change":"2024-04-16T16:21:25.588337+0000","last_active":"2024-04-16T16:21:28.644284+0000","last_peered":"2024-04-16T16:21:28.644284+0000","last_clean":"2024-04-16T16:21:28.644284+0000","last_became_active":"2024-04-16T16:21:25.583483+0000","last_became_peered":"2024-04-16T16:21:25.583483+0000","last_unstale":"2024-04-16T16:21:28.644284+0000","last_undegraded":"2024-04-16T16:21:28.644284+0000","last_fullsized":"2024-04-16T16:21:28.644284+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:10:51.761699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.565860+0000","last_change":"2024-04-16T16:21:24.565860+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.565860+0000","last_undegraded":"2024-04-16T16:21:24.565860+0000","last_fullsized":"2024-04-16T16:21:24.565860+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643197+0000","last_change":"2024-04-16T16:21:25.595882+0000","last_active":"2024-04-16T16:21:28.643197+0000","last_peered":"2024-04-16T16:21:28.643197+0000","last_clean":"2024-04-16T16:21:28.643197+0000","last_became_active":"2024-04-16T16:21:25.595359+0000","last_became_peered":"2024-04-16T16:21:25.595359+0000","last_unstale":"2024-04-16T16:21:28.643197+0000","last_undegraded":"2024-04-16T16:21:28.643197+0000","last_fullsized":"2024-04-16T16:21:28.643197+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:53.324244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.577970+0000","last_change":"2024-04-16T16:21:24.577970+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.577970+0000","last_undegraded":"2024-04-16T16:21:24.577970+0000","last_fullsized":"2024-04-16T16:21:24.577970+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.565010+0000","last_change":"2024-04-16T16:21:24.565010+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.565010+0000","last_undegraded":"2024-04-16T16:21:24.565010+0000","last_fullsized":"2024-04-16T16:21:24.565010+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.565036+0000","last_change":"2024-04-16T16:21:24.565036+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.565036+0000","last_undegraded":"2024-04-16T16:21:24.565036+0000","last_fullsized":"2024-04-16T16:21:24.565036+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644198+0000","last_change":"2024-04-16T16:21:26.618860+0000","last_active":"2024-04-16T16:21:28.644198+0000","last_peered":"2024-04-16T16:21:28.644198+0000","last_clean":"2024-04-16T16:21:28.644198+0000","last_became_active":"2024-04-16T16:21:26.618520+0000","last_became_peered":"2024-04-16T16:21:26.618520+0000","last_unstale":"2024-04-16T16:21:28.644198+0000","last_undegraded":"2024-04-16T16:21:28.644198+0000","last_fullsized":"2024-04-16T16:21:28.644198+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:13.744613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.571965+0000","last_change":"2024-04-16T16:21:24.571965+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.571965+0000","last_undegraded":"2024-04-16T16:21:24.571965+0000","last_fullsized":"2024-04-16T16:21:24.571965+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.571669+0000","last_change":"2024-04-16T16:21:24.571669+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.571669+0000","last_undegraded":"2024-04-16T16:21:24.571669+0000","last_fullsized":"2024-04-16T16:21:24.571669+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.570182+0000","last_change":"2024-04-16T16:21:24.570182+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.570182+0000","last_undegraded":"2024-04-16T16:21:24.570182+0000","last_fullsized":"2024-04-16T16:21:24.570182+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.571981+0000","last_change":"2024-04-16T16:21:24.571981+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.571981+0000","last_undegraded":"2024-04-16T16:21:24.571981+0000","last_fullsized":"2024-04-16T16:21:24.571981+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.10","version":"141'8","reported_seq":22,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643881+0000","last_change":"2024-04-16T16:21:25.600083+0000","last_active":"2024-04-16T16:21:28.643881+0000","last_peered":"2024-04-16T16:21:28.643881+0000","last_clean":"2024-04-16T16:21:28.643881+0000","last_became_active":"2024-04-16T16:21:25.599688+0000","last_became_peered":"2024-04-16T16:21:25.599688+0000","last_unstale":"2024-04-16T16:21:28.643881+0000","last_undegraded":"2024-04-16T16:21:28.643881+0000","last_fullsized":"2024-04-16T16:21:28.643881+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:31.123443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564359+0000","last_change":"2024-04-16T16:21:24.564359+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564359+0000","last_undegraded":"2024-04-16T16:21:24.564359+0000","last_fullsized":"2024-04-16T16:21:24.564359+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643880+0000","last_change":"2024-04-16T16:21:26.623525+0000","last_active":"2024-04-16T16:21:28.643880+0000","last_peered":"2024-04-16T16:21:28.643880+0000","last_clean":"2024-04-16T16:21:28.643880+0000","last_became_active":"2024-04-16T16:21:26.620561+0000","last_became_peered":"2024-04-16T16:21:26.620561+0000","last_unstale":"2024-04-16T16:21:28.643880+0000","last_undegraded":"2024-04-16T16:21:28.643880+0000","last_fullsized":"2024-04-16T16:21:28.643880+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:27.435258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564908+0000","last_change":"2024-04-16T16:21:24.564908+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564908+0000","last_undegraded":"2024-04-16T16:21:24.564908+0000","last_fullsized":"2024-04-16T16:21:24.564908+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.d","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644128+0000","last_change":"2024-04-16T16:21:25.586171+0000","last_active":"2024-04-16T16:21:28.644128+0000","last_peered":"2024-04-16T16:21:28.644128+0000","last_clean":"2024-04-16T16:21:28.644128+0000","last_became_active":"2024-04-16T16:21:25.585863+0000","last_became_peered":"2024-04-16T16:21:25.585863+0000","last_unstale":"2024-04-16T16:21:28.644128+0000","last_undegraded":"2024-04-16T16:21:28.644128+0000","last_fullsized":"2024-04-16T16:21:28.644128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:28:09.057461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564985+0000","last_change":"2024-04-16T16:21:24.564985+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564985+0000","last_undegraded":"2024-04-16T16:21:24.564985+0000","last_fullsized":"2024-04-16T16:21:24.564985+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.573816+0000","last_change":"2024-04-16T16:21:24.573816+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.573816+0000","last_undegraded":"2024-04-16T16:21:24.573816+0000","last_fullsized":"2024-04-16T16:21:24.573816+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564374+0000","last_change":"2024-04-16T16:21:24.564374+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564374+0000","last_undegraded":"2024-04-16T16:21:24.564374+0000","last_fullsized":"2024-04-16T16:21:24.564374+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564355+0000","last_change":"2024-04-16T16:21:24.564355+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564355+0000","last_undegraded":"2024-04-16T16:21:24.564355+0000","last_fullsized":"2024-04-16T16:21:24.564355+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.570375+0000","last_change":"2024-04-16T16:21:24.570375+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.570375+0000","last_undegraded":"2024-04-16T16:21:24.570375+0000","last_fullsized":"2024-04-16T16:21:24.570375+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564516+0000","last_change":"2024-04-16T16:21:24.564516+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564516+0000","last_undegraded":"2024-04-16T16:21:24.564516+0000","last_fullsized":"2024-04-16T16:21:24.564516+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.572258+0000","last_change":"2024-04-16T16:21:24.572258+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.572258+0000","last_undegraded":"2024-04-16T16:21:24.572258+0000","last_fullsized":"2024-04-16T16:21:24.572258+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.572392+0000","last_change":"2024-04-16T16:21:24.572392+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.572392+0000","last_undegraded":"2024-04-16T16:21:24.572392+0000","last_fullsized":"2024-04-16T16:21:24.572392+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.574026+0000","last_change":"2024-04-16T16:21:24.574026+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.574026+0000","last_undegraded":"2024-04-16T16:21:24.574026+0000","last_fullsized":"2024-04-16T16:21:24.574026+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643444+0000","last_change":"2024-04-16T16:21:26.629185+0000","last_active":"2024-04-16T16:21:28.643444+0000","last_peered":"2024-04-16T16:21:28.643444+0000","last_clean":"2024-04-16T16:21:28.643444+0000","last_became_active":"2024-04-16T16:21:26.628800+0000","last_became_peered":"2024-04-16T16:21:26.628800+0000","last_unstale":"2024-04-16T16:21:28.643444+0000","last_undegraded":"2024-04-16T16:21:28.643444+0000","last_fullsized":"2024-04-16T16:21:28.643444+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:56.310523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.571269+0000","last_change":"2024-04-16T16:21:24.571269+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.571269+0000","last_undegraded":"2024-04-16T16:21:24.571269+0000","last_fullsized":"2024-04-16T16:21:24.571269+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.565821+0000","last_change":"2024-04-16T16:21:24.565821+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.565821+0000","last_undegraded":"2024-04-16T16:21:24.565821+0000","last_fullsized":"2024-04-16T16:21:24.565821+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.570696+0000","last_change":"2024-04-16T16:21:24.570696+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.570696+0000","last_undegraded":"2024-04-16T16:21:24.570696+0000","last_fullsized":"2024-04-16T16:21:24.570696+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.564958+0000","last_change":"2024-04-16T16:21:24.564958+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.564958+0000","last_undegraded":"2024-04-16T16:21:24.564958+0000","last_fullsized":"2024-04-16T16:21:24.564958+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-16T16:21:25.556374+0000","last_change":"2024-04-16T16:21:25.556374+0000","last_active":"2024-04-16T16:21:25.556374+0000","last_peered":"2024-04-16T16:21:25.556374+0000","last_clean":"2024-04-16T16:21:25.556374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:25.556374+0000","last_undegraded":"2024-04-16T16:21:25.556374+0000","last_fullsized":"2024-04-16T16:21:25.556374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":1,"reported_epoch":139,"state":"creating+peering","last_fresh":"2024-04-16T16:21:24.570853+0000","last_change":"2024-04-16T16:21:24.570853+0000","last_active":"2024-04-16T16:21:24.541738+0000","last_peered":"2024-04-16T16:21:24.541738+0000","last_clean":"2024-04-16T16:21:24.541738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T16:21:24.570853+0000","last_undegraded":"2024-04-16T16:21:24.570853+0000","last_fullsized":"2024-04-16T16:21:24.570853+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"130'4669","reported_seq":5277,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-16T16:21:24.571361+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:21:24.571361+0000","last_peered":"2024-04-16T16:21:24.571361+0000","last_clean":"2024-04-16T16:21:24.571361+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:21:24.571361+0000","last_undegraded":"2024-04-16T16:21:24.571361+0000","last_fullsized":"2024-04-16T16:21:24.571361+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1083,"num_read_kb":2113,"num_write":347,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":1},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1083,"num_read_kb":2113,"num_write":347,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4669,"ondisk_log_size":4669,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33460,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338380,"statfs":{"total":96636764160,"available":96602501120,"internally_reserved":0,"allocated":2334720,"data_stored":1416329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574942,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1740800,"data_stored":825961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1740800,"data_stored":825961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574943,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1740800,"data_stored":825961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1728512,"data_stored":820144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574942,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37096,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334744,"statfs":{"total":96636764160,"available":96598777856,"internally_reserved":0,"allocated":1794048,"data_stored":846993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1740800,"data_stored":825961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37612,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334228,"statfs":{"total":96636764160,"available":96598249472,"internally_reserved":0,"allocated":2322432,"data_stored":1410512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:21:32.163 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.3f is not active+clean 2024-04-16T16:21:32.163 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.3e is not active+clean 2024-04-16T16:21:32.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.3d is not active+clean 2024-04-16T16:21:32.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.3c is not active+clean 2024-04-16T16:21:32.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.3b is not active+clean 2024-04-16T16:21:32.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.3a is not active+clean 2024-04-16T16:21:32.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.39 is not active+clean 2024-04-16T16:21:32.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.38 is not active+clean 2024-04-16T16:21:32.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.37 is not active+clean 2024-04-16T16:21:32.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.36 is not active+clean 2024-04-16T16:21:32.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.35 is not active+clean 2024-04-16T16:21:32.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.34 is not active+clean 2024-04-16T16:21:32.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.33 is not active+clean 2024-04-16T16:21:32.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.32 is not active+clean 2024-04-16T16:21:32.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.31 is not active+clean 2024-04-16T16:21:32.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.2f is not active+clean 2024-04-16T16:21:32.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.2e is not active+clean 2024-04-16T16:21:32.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.2d is not active+clean 2024-04-16T16:21:32.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.2b is not active+clean 2024-04-16T16:21:32.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.2a is not active+clean 2024-04-16T16:21:32.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.29 is not active+clean 2024-04-16T16:21:32.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.28 is not active+clean 2024-04-16T16:21:32.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.26 is not active+clean 2024-04-16T16:21:32.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.24 is not active+clean 2024-04-16T16:21:32.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.23 is not active+clean 2024-04-16T16:21:32.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.22 is not active+clean 2024-04-16T16:21:32.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.21 is not active+clean 2024-04-16T16:21:32.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.1e is not active+clean 2024-04-16T16:21:32.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1f is not active+clean 2024-04-16T16:21:32.169 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.568474+0000', 'last_change': '2024-04-16T16:21:24.568474+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.568474+0000', 'last_undegraded': '2024-04-16T16:21:24.568474+0000', 'last_fullsized': '2024-04-16T16:21:24.568474+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-16T16:21:32.169 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.1f is not active+clean 2024-04-16T16:21:32.169 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.169 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1e is not active+clean 2024-04-16T16:21:32.169 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.566193+0000', 'last_change': '2024-04-16T16:21:24.566193+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.566193+0000', 'last_undegraded': '2024-04-16T16:21:24.566193+0000', 'last_fullsized': '2024-04-16T16:21:24.566193+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-16T16:21:32.169 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.1c is not active+clean 2024-04-16T16:21:32.169 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.169 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1d is not active+clean 2024-04-16T16:21:32.169 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.578608+0000', 'last_change': '2024-04-16T16:21:24.578608+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.578608+0000', 'last_undegraded': '2024-04-16T16:21:24.578608+0000', 'last_fullsized': '2024-04-16T16:21:24.578608+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-16T16:21:32.169 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.1d is not active+clean 2024-04-16T16:21:32.170 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.170 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1c is not active+clean 2024-04-16T16:21:32.170 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.578075+0000', 'last_change': '2024-04-16T16:21:24.578075+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.578075+0000', 'last_undegraded': '2024-04-16T16:21:24.578075+0000', 'last_fullsized': '2024-04-16T16:21:24.578075+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-16T16:21:32.170 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.1a is not active+clean 2024-04-16T16:21:32.170 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.170 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.1b is not active+clean 2024-04-16T16:21:32.170 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.170 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1a is not active+clean 2024-04-16T16:21:32.170 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.565860+0000', 'last_change': '2024-04-16T16:21:24.565860+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.565860+0000', 'last_undegraded': '2024-04-16T16:21:24.565860+0000', 'last_fullsized': '2024-04-16T16:21:24.565860+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-16T16:21:32.170 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.18 is not active+clean 2024-04-16T16:21:32.170 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.170 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.19 is not active+clean 2024-04-16T16:21:32.171 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.171 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.18 is not active+clean 2024-04-16T16:21:32.171 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.577970+0000', 'last_change': '2024-04-16T16:21:24.577970+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.577970+0000', 'last_undegraded': '2024-04-16T16:21:24.577970+0000', 'last_fullsized': '2024-04-16T16:21:24.577970+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-16T16:21:32.171 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.16 is not active+clean 2024-04-16T16:21:32.171 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.171 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.17 is not active+clean 2024-04-16T16:21:32.171 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.565010+0000', 'last_change': '2024-04-16T16:21:24.565010+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.565010+0000', 'last_undegraded': '2024-04-16T16:21:24.565010+0000', 'last_fullsized': '2024-04-16T16:21:24.565010+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-16T16:21:32.171 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.17 is not active+clean 2024-04-16T16:21:32.171 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.171 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.16 is not active+clean 2024-04-16T16:21:32.171 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.565036+0000', 'last_change': '2024-04-16T16:21:24.565036+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.565036+0000', 'last_undegraded': '2024-04-16T16:21:24.565036+0000', 'last_fullsized': '2024-04-16T16:21:24.565036+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-16T16:21:32.172 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.14 is not active+clean 2024-04-16T16:21:32.172 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.571965+0000', 'last_change': '2024-04-16T16:21:24.571965+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.571965+0000', 'last_undegraded': '2024-04-16T16:21:24.571965+0000', 'last_fullsized': '2024-04-16T16:21:24.571965+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-16T16:21:32.172 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.12 is not active+clean 2024-04-16T16:21:32.172 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.172 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.13 is not active+clean 2024-04-16T16:21:32.172 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.571669+0000', 'last_change': '2024-04-16T16:21:24.571669+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.571669+0000', 'last_undegraded': '2024-04-16T16:21:24.571669+0000', 'last_fullsized': '2024-04-16T16:21:24.571669+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:21:32.172 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.13 is not active+clean 2024-04-16T16:21:32.172 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.172 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.12 is not active+clean 2024-04-16T16:21:32.172 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.570182+0000', 'last_change': '2024-04-16T16:21:24.570182+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.570182+0000', 'last_undegraded': '2024-04-16T16:21:24.570182+0000', 'last_fullsized': '2024-04-16T16:21:24.570182+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:21:32.172 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.10 is not active+clean 2024-04-16T16:21:32.173 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.173 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.11 is not active+clean 2024-04-16T16:21:32.173 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.571981+0000', 'last_change': '2024-04-16T16:21:24.571981+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.571981+0000', 'last_undegraded': '2024-04-16T16:21:24.571981+0000', 'last_fullsized': '2024-04-16T16:21:24.571981+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-16T16:21:32.173 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.11 is not active+clean 2024-04-16T16:21:32.173 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.173 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.e is not active+clean 2024-04-16T16:21:32.173 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.173 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.f is not active+clean 2024-04-16T16:21:32.173 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.564359+0000', 'last_change': '2024-04-16T16:21:24.564359+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.564359+0000', 'last_undegraded': '2024-04-16T16:21:24.564359+0000', 'last_fullsized': '2024-04-16T16:21:24.564359+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-16T16:21:32.173 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.e is not active+clean 2024-04-16T16:21:32.173 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.564908+0000', 'last_change': '2024-04-16T16:21:24.564908+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.564908+0000', 'last_undegraded': '2024-04-16T16:21:24.564908+0000', 'last_fullsized': '2024-04-16T16:21:24.564908+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-16T16:21:32.173 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.c is not active+clean 2024-04-16T16:21:32.174 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.174 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.d is not active+clean 2024-04-16T16:21:32.174 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.174 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.c is not active+clean 2024-04-16T16:21:32.174 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.564985+0000', 'last_change': '2024-04-16T16:21:24.564985+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.564985+0000', 'last_undegraded': '2024-04-16T16:21:24.564985+0000', 'last_fullsized': '2024-04-16T16:21:24.564985+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-16T16:21:32.174 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.a is not active+clean 2024-04-16T16:21:32.174 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.174 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.b is not active+clean 2024-04-16T16:21:32.174 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.573816+0000', 'last_change': '2024-04-16T16:21:24.573816+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.573816+0000', 'last_undegraded': '2024-04-16T16:21:24.573816+0000', 'last_fullsized': '2024-04-16T16:21:24.573816+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-16T16:21:32.174 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.b is not active+clean 2024-04-16T16:21:32.174 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.174 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.a is not active+clean 2024-04-16T16:21:32.174 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.564374+0000', 'last_change': '2024-04-16T16:21:24.564374+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.564374+0000', 'last_undegraded': '2024-04-16T16:21:24.564374+0000', 'last_fullsized': '2024-04-16T16:21:24.564374+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-16T16:21:32.175 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.8 is not active+clean 2024-04-16T16:21:32.175 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.175 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.9 is not active+clean 2024-04-16T16:21:32.175 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.564355+0000', 'last_change': '2024-04-16T16:21:24.564355+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.564355+0000', 'last_undegraded': '2024-04-16T16:21:24.564355+0000', 'last_fullsized': '2024-04-16T16:21:24.564355+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-16T16:21:32.175 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.9 is not active+clean 2024-04-16T16:21:32.175 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.175 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.8 is not active+clean 2024-04-16T16:21:32.175 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.570375+0000', 'last_change': '2024-04-16T16:21:24.570375+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.570375+0000', 'last_undegraded': '2024-04-16T16:21:24.570375+0000', 'last_fullsized': '2024-04-16T16:21:24.570375+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:21:32.175 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.6 is not active+clean 2024-04-16T16:21:32.175 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.175 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.7 is not active+clean 2024-04-16T16:21:32.175 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.564516+0000', 'last_change': '2024-04-16T16:21:24.564516+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.564516+0000', 'last_undegraded': '2024-04-16T16:21:24.564516+0000', 'last_fullsized': '2024-04-16T16:21:24.564516+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-16T16:21:32.176 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.7 is not active+clean 2024-04-16T16:21:32.176 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.176 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.6 is not active+clean 2024-04-16T16:21:32.176 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.572258+0000', 'last_change': '2024-04-16T16:21:24.572258+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.572258+0000', 'last_undegraded': '2024-04-16T16:21:24.572258+0000', 'last_fullsized': '2024-04-16T16:21:24.572258+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-16T16:21:32.176 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.4 is not active+clean 2024-04-16T16:21:32.176 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.176 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.5 is not active+clean 2024-04-16T16:21:32.176 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.572392+0000', 'last_change': '2024-04-16T16:21:24.572392+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.572392+0000', 'last_undegraded': '2024-04-16T16:21:24.572392+0000', 'last_fullsized': '2024-04-16T16:21:24.572392+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-16T16:21:32.176 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.5 is not active+clean 2024-04-16T16:21:32.176 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.176 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.4 is not active+clean 2024-04-16T16:21:32.176 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.574026+0000', 'last_change': '2024-04-16T16:21:24.574026+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.574026+0000', 'last_undegraded': '2024-04-16T16:21:24.574026+0000', 'last_fullsized': '2024-04-16T16:21:24.574026+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-16T16:21:32.176 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3 is not active+clean 2024-04-16T16:21:32.177 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.571269+0000', 'last_change': '2024-04-16T16:21:24.571269+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.571269+0000', 'last_undegraded': '2024-04-16T16:21:24.571269+0000', 'last_fullsized': '2024-04-16T16:21:24.571269+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-16T16:21:32.177 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.3 is not active+clean 2024-04-16T16:21:32.177 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.177 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2 is not active+clean 2024-04-16T16:21:32.177 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.565821+0000', 'last_change': '2024-04-16T16:21:24.565821+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.565821+0000', 'last_undegraded': '2024-04-16T16:21:24.565821+0000', 'last_fullsized': '2024-04-16T16:21:24.565821+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-16T16:21:32.177 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.0 is not active+clean 2024-04-16T16:21:32.177 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.177 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1 is not active+clean 2024-04-16T16:21:32.177 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.570696+0000', 'last_change': '2024-04-16T16:21:24.570696+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.570696+0000', 'last_undegraded': '2024-04-16T16:21:24.570696+0000', 'last_fullsized': '2024-04-16T16:21:24.570696+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:21:32.177 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.1 is not active+clean 2024-04-16T16:21:32.177 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.177 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.0 is not active+clean 2024-04-16T16:21:32.178 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.564958+0000', 'last_change': '2024-04-16T16:21:24.564958+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.564958+0000', 'last_undegraded': '2024-04-16T16:21:24.564958+0000', 'last_fullsized': '2024-04-16T16:21:24.564958+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-16T16:21:32.178 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.14 is not active+clean 2024-04-16T16:21:32.178 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-16T16:21:25.556374+0000', 'last_change': '2024-04-16T16:21:25.556374+0000', 'last_active': '2024-04-16T16:21:25.556374+0000', 'last_peered': '2024-04-16T16:21:25.556374+0000', 'last_clean': '2024-04-16T16:21:25.556374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:25.556374+0000', 'last_undegraded': '2024-04-16T16:21:25.556374+0000', 'last_fullsized': '2024-04-16T16:21:25.556374+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:25.556374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-16T16:21:32.178 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.15 is not active+clean 2024-04-16T16:21:32.178 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 139, 'state': 'creating+peering', 'last_fresh': '2024-04-16T16:21:24.570853+0000', 'last_change': '2024-04-16T16:21:24.570853+0000', 'last_active': '2024-04-16T16:21:24.541738+0000', 'last_peered': '2024-04-16T16:21:24.541738+0000', 'last_clean': '2024-04-16T16:21:24.541738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-16T16:21:24.570853+0000', 'last_undegraded': '2024-04-16T16:21:24.570853+0000', 'last_fullsized': '2024-04-16T16:21:24.570853+0000', 'mapping_epoch': 139, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 139, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'last_clean_scrub_stamp': '2024-04-16T16:21:24.541738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-16T16:21:32.178 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T16:21:32.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.428+0000 7fc1ef000640 1 Processor -- start 2024-04-16T16:21:32.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.428+0000 7fc1ef000640 1 -- start start 2024-04-16T16:21:32.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.430+0000 7fc1ef000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1e80ccb20 0x7fc1e80cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.430+0000 7fc1ef000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1e80ce9a0 0x7fc1e80cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.430+0000 7fc1ef000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e815ecd0 con 0x7fc1e815e8f0 2024-04-16T16:21:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.430+0000 7fc1ef000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc1e80ce2e0 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.430+0000 7fc1ef000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc1e80cd6d0 con 0x7fc1e80ccb20 2024-04-16T16:21:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.430+0000 7fc1ee600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1e80ce9a0 0x7fc1e80cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.430+0000 7fc1ee600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1e80ce9a0 0x7fc1e80cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59042/0 (socket says 172.21.15.45:59042) 2024-04-16T16:21:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.430+0000 7fc1ee600640 1 -- 172.21.15.45:0/3099802502 learned_addr learned my addr 172.21.15.45:0/3099802502 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.431+0000 7fc1ed200640 1 --2- 172.21.15.45:0/3099802502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1e80ccb20 0x7fc1e80cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.431+0000 7fc1ee600640 1 -- 172.21.15.45:0/3099802502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1e80ccb20 msgr2=0x7fc1e80cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.431+0000 7fc1ee600640 1 --2- 172.21.15.45:0/3099802502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1e80ccb20 0x7fc1e80cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.431+0000 7fc1ee600640 1 -- 172.21.15.45:0/3099802502 >> v1:172.21.15.103:6789/0 conn(0x7fc1e815e8f0 legacy=0x7fc1e80cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.431+0000 7fc1ee600640 1 -- 172.21.15.45:0/3099802502 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e80cd9b0 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.431+0000 7fc1ed200640 1 --2- 172.21.15.45:0/3099802502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1e80ccb20 0x7fc1e80cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:32.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.432+0000 7fc1ee600640 1 --2- 172.21.15.45:0/3099802502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1e80ce9a0 0x7fc1e80cb170 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009e50 tx=0x7fc1e002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad5dad790cfe2bf server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:32.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.432+0000 7fc1e7e00640 1 -- 172.21.15.45:0/3099802502 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e002ca80 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.432+0000 7fc1e7e00640 1 -- 172.21.15.45:0/3099802502 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1e002cc20 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.432+0000 7fc1e7e00640 1 -- 172.21.15.45:0/3099802502 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e00343f0 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.433+0000 7fc1ef000640 1 -- 172.21.15.45:0/3099802502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1e80ce9a0 msgr2=0x7fc1e80cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:32.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.433+0000 7fc1ef000640 1 --2- 172.21.15.45:0/3099802502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1e80ce9a0 0x7fc1e80cb170 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009e50 tx=0x7fc1e002bee0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.433+0000 7fc1ef000640 1 -- 172.21.15.45:0/3099802502 shutdown_connections 2024-04-16T16:21:32.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.433+0000 7fc1ef000640 1 --2- 172.21.15.45:0/3099802502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1e80ce9a0 0x7fc1e80cb170 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.433+0000 7fc1ef000640 1 --2- 172.21.15.45:0/3099802502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1e80ccb20 0x7fc1e80cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.433+0000 7fc1ef000640 1 -- 172.21.15.45:0/3099802502 >> 172.21.15.45:0/3099802502 conn(0x7fc1e806b930 msgr2=0x7fc1e806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:32.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.433+0000 7fc1ef000640 1 -- 172.21.15.45:0/3099802502 shutdown_connections 2024-04-16T16:21:32.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.433+0000 7fc1ef000640 1 -- 172.21.15.45:0/3099802502 wait complete. 2024-04-16T16:21:32.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.435+0000 7fc1ef000640 1 Processor -- start 2024-04-16T16:21:32.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.435+0000 7fc1ef000640 1 -- start start 2024-04-16T16:21:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.436+0000 7fc1ef000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1e80ccb20 0x7fc1e815a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.436+0000 7fc1ef000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1e80ce9a0 0x7fc1e815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.436+0000 7fc1ef000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1e815e8f0 0x7fc1e8146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.436+0000 7fc1ef000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e80cd360 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.436+0000 7fc1ef000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e80cdf70 con 0x7fc1e80ccb20 2024-04-16T16:21:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.436+0000 7fc1ef000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc1e80cf080 con 0x7fc1e815e8f0 2024-04-16T16:21:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.437+0000 7fc1ed200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1e80ce9a0 0x7fc1e815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.437+0000 7fc1ed200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1e80ce9a0 0x7fc1e815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59056/0 (socket says 172.21.15.45:59056) 2024-04-16T16:21:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.437+0000 7fc1ed200640 1 -- 172.21.15.45:0/520246905 learned_addr learned my addr 172.21.15.45:0/520246905 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.437+0000 7fc1edc00640 1 --2- 172.21.15.45:0/520246905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1e80ccb20 0x7fc1e815a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:32.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.437+0000 7fc1ed200640 1 -- 172.21.15.45:0/520246905 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1e815e8f0 msgr2=0x7fc1e8146e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:32.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.437+0000 7fc1ed200640 1 --2- 172.21.15.45:0/520246905 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1e815e8f0 0x7fc1e8146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.437+0000 7fc1ed200640 1 -- 172.21.15.45:0/520246905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1e80ccb20 msgr2=0x7fc1e815a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:32.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.437+0000 7fc1ed200640 1 --2- 172.21.15.45:0/520246905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1e80ccb20 0x7fc1e815a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.437+0000 7fc1ed200640 1 -- 172.21.15.45:0/520246905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e815c5a0 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.437+0000 7fc1ed200640 1 --2- 172.21.15.45:0/520246905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1e80ce9a0 0x7fc1e815bee0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fc1d400c820 tx=0x7fc1d400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:32.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.438+0000 7fc1e5600640 1 -- 172.21.15.45:0/520246905 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4015070 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.438+0000 7fc1e5600640 1 -- 172.21.15.45:0/520246905 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1d4007a30 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.438+0000 7fc1ef000640 1 -- 172.21.15.45:0/520246905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1e815c880 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.438+0000 7fc1e5600640 1 -- 172.21.15.45:0/520246905 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4007e00 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.438+0000 7fc1ef000640 1 -- 172.21.15.45:0/520246905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1e815ce40 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.439+0000 7fc1ef000640 1 -- 172.21.15.45:0/520246905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1e8075780 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.440+0000 7fc1e5600640 1 -- 172.21.15.45:0/520246905 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7fc1d4018070 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.441+0000 7fc1e5600640 1 --2- 172.21.15.45:0/520246905 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fc1c806ca10 0x7fc1c806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.441+0000 7fc1e5600640 1 -- 172.21.15.45:0/520246905 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7fc1d408bdd0 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.442+0000 7fc1edc00640 1 --2- 172.21.15.45:0/520246905 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fc1c806ca10 0x7fc1c806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:32.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.443+0000 7fc1edc00640 1 --2- 172.21.15.45:0/520246905 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fc1c806ca10 0x7fc1c806eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc006fd0 tx=0x7fc1dc008040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:32.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.446+0000 7fc1e5600640 1 -- 172.21.15.45:0/520246905 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc1d4059fe0 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.725+0000 7fc1ef000640 1 -- 172.21.15.45:0/520246905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fc1e80b5b90 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.727+0000 7fc1e5600640 1 -- 172.21.15.45:0/520246905 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+3212 (secure 0 0 0) 0x7fc1d405ee20 con 0x7fc1e80ce9a0 2024-04-16T16:21:32.726 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:32.726 INFO:teuthology.orchestra.run.smithi045.stdout:{"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":506,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":142,"num_osds":8,"num_up_osds":8,"osd_up_since":1713283997,"num_in_osds":8,"osd_in_since":1713283989,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":279453696,"bytes_avail":772814659584,"bytes_total":773094113280,"write_bytes_sec":2100,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":100,"id":10,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":10,"rank":0,"name":"d","status":"up:active","gid":6596}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-16T16:16:25.060027+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"y":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"6":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"7":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-16T16:21:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.729+0000 7fc1ef000640 1 -- 172.21.15.45:0/520246905 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fc1c806ca10 msgr2=0x7fc1c806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.729+0000 7fc1ef000640 1 --2- 172.21.15.45:0/520246905 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fc1c806ca10 0x7fc1c806eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc006fd0 tx=0x7fc1dc008040 comp rx=0 tx=0).stop 2024-04-16T16:21:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.729+0000 7fc1ef000640 1 -- 172.21.15.45:0/520246905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1e80ce9a0 msgr2=0x7fc1e815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.729+0000 7fc1ef000640 1 --2- 172.21.15.45:0/520246905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1e80ce9a0 0x7fc1e815bee0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fc1d400c820 tx=0x7fc1d400cc20 comp rx=0 tx=0).stop 2024-04-16T16:21:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.729+0000 7fc1ef000640 1 -- 172.21.15.45:0/520246905 shutdown_connections 2024-04-16T16:21:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.729+0000 7fc1ef000640 1 --2- 172.21.15.45:0/520246905 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1e815e8f0 0x7fc1e8146e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.729+0000 7fc1ef000640 1 --2- 172.21.15.45:0/520246905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1e80ce9a0 0x7fc1e815bee0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.729+0000 7fc1ef000640 1 --2- 172.21.15.45:0/520246905 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fc1c806ca10 0x7fc1c806eed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.729+0000 7fc1ef000640 1 --2- 172.21.15.45:0/520246905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1e80ccb20 0x7fc1e815a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.729+0000 7fc1ef000640 1 -- 172.21.15.45:0/520246905 >> 172.21.15.45:0/520246905 conn(0x7fc1e806b930 msgr2=0x7fc1e8157800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.730+0000 7fc1ef000640 1 -- 172.21.15.45:0/520246905 shutdown_connections 2024-04-16T16:21:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.730+0000 7fc1ef000640 1 -- 172.21.15.45:0/520246905 wait complete. 2024-04-16T16:21:32.746 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-16T16:21:32.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:21:32.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.959+0000 7fd959c00640 1 Processor -- start 2024-04-16T16:21:32.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.959+0000 7fd959c00640 1 -- start start 2024-04-16T16:21:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.961+0000 7fd959c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd954146220 0x7fd95415ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.961+0000 7fd959c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd9541608a0 0x7fd954160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.961+0000 7fd959c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd95415d360 con 0x7fd95406f7a0 2024-04-16T16:21:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.961+0000 7fd959c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd954161230 con 0x7fd9541608a0 2024-04-16T16:21:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.961+0000 7fd959c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd954160410 con 0x7fd954146220 2024-04-16T16:21:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.961+0000 7fd953400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd95406f7a0 0x7fd95406df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60566/0 (socket says 172.21.15.45:60566) 2024-04-16T16:21:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.961+0000 7fd953400640 1 -- 172.21.15.45:0/506316121 learned_addr learned my addr 172.21.15.45:0/506316121 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.962+0000 7fd959200640 1 --2- 172.21.15.45:0/506316121 >> v2:172.21.15.45:3300/0 conn(0x7fd9541608a0 0x7fd954160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.962+0000 7fd952a00640 1 -- 172.21.15.45:0/506316121 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 678015388 0 0) 0x7fd95415d360 con 0x7fd95406f7a0 2024-04-16T16:21:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.962+0000 7fd953e00640 1 --2- 172.21.15.45:0/506316121 >> v2:172.21.15.103:3300/0 conn(0x7fd954146220 0x7fd95415ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.962+0000 7fd952a00640 1 -- 172.21.15.45:0/506316121 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd938003520 con 0x7fd95406f7a0 2024-04-16T16:21:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.962+0000 7fd952a00640 1 -- 172.21.15.45:0/506316121 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2709377164 0 0) 0x7fd938003520 con 0x7fd95406f7a0 2024-04-16T16:21:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.962+0000 7fd959200640 1 -- 172.21.15.45:0/506316121 >> v2:172.21.15.103:3300/0 conn(0x7fd954146220 msgr2=0x7fd95415ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.962+0000 7fd959200640 1 --2- 172.21.15.45:0/506316121 >> v2:172.21.15.103:3300/0 conn(0x7fd954146220 0x7fd95415ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.962+0000 7fd959200640 1 -- 172.21.15.45:0/506316121 >> v1:172.21.15.45:6789/0 conn(0x7fd95406f7a0 legacy=0x7fd95406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.962+0000 7fd959200640 1 -- 172.21.15.45:0/506316121 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd95415d640 con 0x7fd9541608a0 2024-04-16T16:21:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.963+0000 7fd959200640 1 --2- 172.21.15.45:0/506316121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9541608a0 0x7fd954160cf0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fd954070210 tx=0x7fd94c02bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f70242b791620c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.964+0000 7fd952a00640 1 -- 172.21.15.45:0/506316121 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd94c039070 con 0x7fd9541608a0 2024-04-16T16:21:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.964+0000 7fd952a00640 1 -- 172.21.15.45:0/506316121 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd94c02c8c0 con 0x7fd9541608a0 2024-04-16T16:21:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.964+0000 7fd952a00640 1 -- 172.21.15.45:0/506316121 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd940001670 con 0x7fd9541608a0 2024-04-16T16:21:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.964+0000 7fd959c00640 1 -- 172.21.15.45:0/506316121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9541608a0 msgr2=0x7fd954160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.964+0000 7fd959c00640 1 --2- 172.21.15.45:0/506316121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9541608a0 0x7fd954160cf0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fd954070210 tx=0x7fd94c02bd90 comp rx=0 tx=0).stop 2024-04-16T16:21:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.964+0000 7fd959c00640 1 -- 172.21.15.45:0/506316121 shutdown_connections 2024-04-16T16:21:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.964+0000 7fd959c00640 1 --2- 172.21.15.45:0/506316121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9541608a0 0x7fd954160cf0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.964+0000 7fd959c00640 1 --2- 172.21.15.45:0/506316121 >> v2:172.21.15.103:3300/0 conn(0x7fd954146220 0x7fd95415ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.964+0000 7fd959c00640 1 -- 172.21.15.45:0/506316121 >> 172.21.15.45:0/506316121 conn(0x7fd95406b930 msgr2=0x7fd95406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.964+0000 7fd959c00640 1 -- 172.21.15.45:0/506316121 shutdown_connections 2024-04-16T16:21:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.965+0000 7fd959c00640 1 -- 172.21.15.45:0/506316121 wait complete. 2024-04-16T16:21:32.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.966+0000 7fd959c00640 1 Processor -- start 2024-04-16T16:21:32.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.966+0000 7fd959c00640 1 -- start start 2024-04-16T16:21:32.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.967+0000 7fd959c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd95406f7a0 0x7fd9541411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:32.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.967+0000 7fd959c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd954146220 0x7fd9540ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:32.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.967+0000 7fd959c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9541608a0 0x7fd954144140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:32.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.967+0000 7fd959c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd95415dc00 con 0x7fd9541608a0 2024-04-16T16:21:32.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.967+0000 7fd959c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd95406e650 con 0x7fd954146220 2024-04-16T16:21:32.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.967+0000 7fd959c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd95406fc20 con 0x7fd95406f7a0 2024-04-16T16:21:32.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.968+0000 7fd959200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9541608a0 0x7fd954144140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:32.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.968+0000 7fd959200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9541608a0 0x7fd954144140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59080/0 (socket says 172.21.15.45:59080) 2024-04-16T16:21:32.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.968+0000 7fd959200640 1 -- 172.21.15.45:0/4214904430 learned_addr learned my addr 172.21.15.45:0/4214904430 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:32.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.968+0000 7fd953400640 1 --2- 172.21.15.45:0/4214904430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd954146220 0x7fd9540ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.968+0000 7fd953e00640 1 --2- 172.21.15.45:0/4214904430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd95406f7a0 0x7fd9541411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.968+0000 7fd959200640 1 -- 172.21.15.45:0/4214904430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd95406f7a0 msgr2=0x7fd9541411f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.968+0000 7fd959200640 1 --2- 172.21.15.45:0/4214904430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd95406f7a0 0x7fd9541411f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.968+0000 7fd959200640 1 -- 172.21.15.45:0/4214904430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd954146220 msgr2=0x7fd9540ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.968+0000 7fd959200640 1 --2- 172.21.15.45:0/4214904430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd954146220 0x7fd9540ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.968+0000 7fd959200640 1 -- 172.21.15.45:0/4214904430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9540caf70 con 0x7fd9541608a0 2024-04-16T16:21:32.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.968+0000 7fd959200640 1 --2- 172.21.15.45:0/4214904430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9541608a0 0x7fd954144140 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fd94c009e80 tx=0x7fd94c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:32.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.969+0000 7fd94be00640 1 -- 172.21.15.45:0/4214904430 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd94c039050 con 0x7fd9541608a0 2024-04-16T16:21:32.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.969+0000 7fd94be00640 1 -- 172.21.15.45:0/4214904430 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd94c045070 con 0x7fd9541608a0 2024-04-16T16:21:32.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.969+0000 7fd94be00640 1 -- 172.21.15.45:0/4214904430 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd94c009450 con 0x7fd9541608a0 2024-04-16T16:21:32.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.969+0000 7fd959c00640 1 -- 172.21.15.45:0/4214904430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9540c8c20 con 0x7fd9541608a0 2024-04-16T16:21:32.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.969+0000 7fd959c00640 1 -- 172.21.15.45:0/4214904430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9540c91e0 con 0x7fd9541608a0 2024-04-16T16:21:32.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.970+0000 7fd959c00640 1 -- 172.21.15.45:0/4214904430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9540c93e0 con 0x7fd9541608a0 2024-04-16T16:21:32.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.971+0000 7fd94be00640 1 -- 172.21.15.45:0/4214904430 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7fd94c040690 con 0x7fd9541608a0 2024-04-16T16:21:32.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.972+0000 7fd94be00640 1 --2- 172.21.15.45:0/4214904430 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fd93406ca10 0x7fd93406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.972+0000 7fd94be00640 1 -- 172.21.15.45:0/4214904430 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7fd94c0ae740 con 0x7fd9541608a0 2024-04-16T16:21:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.973+0000 7fd953e00640 1 --2- 172.21.15.45:0/4214904430 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fd93406ca10 0x7fd93406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:32.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.974+0000 7fd953e00640 1 --2- 172.21.15.45:0/4214904430 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fd93406ca10 0x7fd93406eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd944006fd0 tx=0x7fd944008040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:32.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:32.976+0000 7fd94be00640 1 -- 172.21.15.45:0/4214904430 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9540c93e0 con 0x7fd9541608a0 2024-04-16T16:21:33.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:33.169+0000 7fd959c00640 1 -- 172.21.15.45:0/4214904430 --> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd954075780 con 0x7fd93406ca10 2024-04-16T16:21:33.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:33.189+0000 7fd94be00640 1 -- 172.21.15.45:0/4214904430 <== mgr.7153 v2:172.21.15.103:6832/1478535274 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252501 (secure 0 0 0) 0x7fd954075780 con 0x7fd93406ca10 2024-04-16T16:21:33.191 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:33.193 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:21:33.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:33.194+0000 7fd959c00640 1 -- 172.21.15.45:0/4214904430 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fd93406ca10 msgr2=0x7fd93406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:33.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:33.194+0000 7fd959c00640 1 --2- 172.21.15.45:0/4214904430 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fd93406ca10 0x7fd93406eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd944006fd0 tx=0x7fd944008040 comp rx=0 tx=0).stop 2024-04-16T16:21:33.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:33.194+0000 7fd959c00640 1 -- 172.21.15.45:0/4214904430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9541608a0 msgr2=0x7fd954144140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:33.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:33.194+0000 7fd959c00640 1 --2- 172.21.15.45:0/4214904430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9541608a0 0x7fd954144140 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fd94c009e80 tx=0x7fd94c004a40 comp rx=0 tx=0).stop 2024-04-16T16:21:33.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:33.194+0000 7fd959c00640 1 -- 172.21.15.45:0/4214904430 shutdown_connections 2024-04-16T16:21:33.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:33.194+0000 7fd959c00640 1 --2- 172.21.15.45:0/4214904430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9541608a0 0x7fd954144140 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:33.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:33.194+0000 7fd959c00640 1 --2- 172.21.15.45:0/4214904430 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7fd93406ca10 0x7fd93406eed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:33.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:33.194+0000 7fd959c00640 1 --2- 172.21.15.45:0/4214904430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd954146220 0x7fd9540ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:33.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:33.194+0000 7fd959c00640 1 --2- 172.21.15.45:0/4214904430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd95406f7a0 0x7fd9541411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:33.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:33.194+0000 7fd959c00640 1 -- 172.21.15.45:0/4214904430 >> 172.21.15.45:0/4214904430 conn(0x7fd95406b930 msgr2=0x7fd9540c5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:33.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:33.195+0000 7fd959c00640 1 -- 172.21.15.45:0/4214904430 shutdown_connections 2024-04-16T16:21:33.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:33.195+0000 7fd959c00640 1 -- 172.21.15.45:0/4214904430 wait complete. 2024-04-16T16:21:33.212 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-04-16T16:21:32.391263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1083,"num_read_kb":2113,"num_write":392,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4708,"ondisk_log_size":4708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":169,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":272904,"kb_used_data":15112,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754701816,"statfs":{"total":773094113280,"available":772814659584,"internally_reserved":0,"allocated":15474688,"data_stored":7948044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.824721"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.611281+0000","last_change":"2024-04-16T16:21:26.626458+0000","last_active":"2024-04-16T16:21:28.611281+0000","last_peered":"2024-04-16T16:21:28.611281+0000","last_clean":"2024-04-16T16:21:28.611281+0000","last_became_active":"2024-04-16T16:21:26.626085+0000","last_became_peered":"2024-04-16T16:21:26.626085+0000","last_unstale":"2024-04-16T16:21:28.611281+0000","last_undegraded":"2024-04-16T16:21:28.611281+0000","last_fullsized":"2024-04-16T16:21:28.611281+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:00:53.974790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.623203+0000","last_change":"2024-04-16T16:21:26.618211+0000","last_active":"2024-04-16T16:21:27.623203+0000","last_peered":"2024-04-16T16:21:27.623203+0000","last_clean":"2024-04-16T16:21:27.623203+0000","last_became_active":"2024-04-16T16:21:26.617930+0000","last_became_peered":"2024-04-16T16:21:26.617930+0000","last_unstale":"2024-04-16T16:21:27.623203+0000","last_undegraded":"2024-04-16T16:21:27.623203+0000","last_fullsized":"2024-04-16T16:21:27.623203+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:16:09.049767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643996+0000","last_change":"2024-04-16T16:21:26.635568+0000","last_active":"2024-04-16T16:21:28.643996+0000","last_peered":"2024-04-16T16:21:28.643996+0000","last_clean":"2024-04-16T16:21:28.643996+0000","last_became_active":"2024-04-16T16:21:26.635193+0000","last_became_peered":"2024-04-16T16:21:26.635193+0000","last_unstale":"2024-04-16T16:21:28.643996+0000","last_undegraded":"2024-04-16T16:21:28.643996+0000","last_fullsized":"2024-04-16T16:21:28.643996+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:49:46.979105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626780+0000","last_change":"2024-04-16T16:21:26.636857+0000","last_active":"2024-04-16T16:21:27.626780+0000","last_peered":"2024-04-16T16:21:27.626780+0000","last_clean":"2024-04-16T16:21:27.626780+0000","last_became_active":"2024-04-16T16:21:26.636573+0000","last_became_peered":"2024-04-16T16:21:26.636573+0000","last_unstale":"2024-04-16T16:21:27.626780+0000","last_undegraded":"2024-04-16T16:21:27.626780+0000","last_fullsized":"2024-04-16T16:21:27.626780+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:55.067333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644598+0000","last_change":"2024-04-16T16:21:26.622281+0000","last_active":"2024-04-16T16:21:28.644598+0000","last_peered":"2024-04-16T16:21:28.644598+0000","last_clean":"2024-04-16T16:21:28.644598+0000","last_became_active":"2024-04-16T16:21:26.621705+0000","last_became_peered":"2024-04-16T16:21:26.621705+0000","last_unstale":"2024-04-16T16:21:28.644598+0000","last_undegraded":"2024-04-16T16:21:28.644598+0000","last_fullsized":"2024-04-16T16:21:28.644598+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:36.349501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610344+0000","last_change":"2024-04-16T16:21:26.634720+0000","last_active":"2024-04-16T16:21:28.610344+0000","last_peered":"2024-04-16T16:21:28.610344+0000","last_clean":"2024-04-16T16:21:28.610344+0000","last_became_active":"2024-04-16T16:21:26.634339+0000","last_became_peered":"2024-04-16T16:21:26.634339+0000","last_unstale":"2024-04-16T16:21:28.610344+0000","last_undegraded":"2024-04-16T16:21:28.610344+0000","last_fullsized":"2024-04-16T16:21:28.610344+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:20:31.043392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.641429+0000","last_change":"2024-04-16T16:21:26.638223+0000","last_active":"2024-04-16T16:21:28.641429+0000","last_peered":"2024-04-16T16:21:28.641429+0000","last_clean":"2024-04-16T16:21:28.641429+0000","last_became_active":"2024-04-16T16:21:26.637894+0000","last_became_peered":"2024-04-16T16:21:26.637894+0000","last_unstale":"2024-04-16T16:21:28.641429+0000","last_undegraded":"2024-04-16T16:21:28.641429+0000","last_fullsized":"2024-04-16T16:21:28.641429+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:06.714681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.609807+0000","last_change":"2024-04-16T16:21:26.622007+0000","last_active":"2024-04-16T16:21:28.609807+0000","last_peered":"2024-04-16T16:21:28.609807+0000","last_clean":"2024-04-16T16:21:28.609807+0000","last_became_active":"2024-04-16T16:21:26.621724+0000","last_became_peered":"2024-04-16T16:21:26.621724+0000","last_unstale":"2024-04-16T16:21:28.609807+0000","last_undegraded":"2024-04-16T16:21:28.609807+0000","last_fullsized":"2024-04-16T16:21:28.609807+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:20.296700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.623030+0000","last_change":"2024-04-16T16:21:26.621820+0000","last_active":"2024-04-16T16:21:27.623030+0000","last_peered":"2024-04-16T16:21:27.623030+0000","last_clean":"2024-04-16T16:21:27.623030+0000","last_became_active":"2024-04-16T16:21:26.621472+0000","last_became_peered":"2024-04-16T16:21:26.621472+0000","last_unstale":"2024-04-16T16:21:27.623030+0000","last_undegraded":"2024-04-16T16:21:27.623030+0000","last_fullsized":"2024-04-16T16:21:27.623030+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:42.748859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628125+0000","last_change":"2024-04-16T16:21:26.629761+0000","last_active":"2024-04-16T16:21:27.628125+0000","last_peered":"2024-04-16T16:21:27.628125+0000","last_clean":"2024-04-16T16:21:27.628125+0000","last_became_active":"2024-04-16T16:21:26.629483+0000","last_became_peered":"2024-04-16T16:21:26.629483+0000","last_unstale":"2024-04-16T16:21:27.628125+0000","last_undegraded":"2024-04-16T16:21:27.628125+0000","last_fullsized":"2024-04-16T16:21:27.628125+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:41:45.291950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610211+0000","last_change":"2024-04-16T16:21:26.624559+0000","last_active":"2024-04-16T16:21:28.610211+0000","last_peered":"2024-04-16T16:21:28.610211+0000","last_clean":"2024-04-16T16:21:28.610211+0000","last_became_active":"2024-04-16T16:21:26.622580+0000","last_became_peered":"2024-04-16T16:21:26.622580+0000","last_unstale":"2024-04-16T16:21:28.610211+0000","last_undegraded":"2024-04-16T16:21:28.610211+0000","last_fullsized":"2024-04-16T16:21:28.610211+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:04:09.629772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627808+0000","last_change":"2024-04-16T16:21:26.623657+0000","last_active":"2024-04-16T16:21:27.627808+0000","last_peered":"2024-04-16T16:21:27.627808+0000","last_clean":"2024-04-16T16:21:27.627808+0000","last_became_active":"2024-04-16T16:21:26.623238+0000","last_became_peered":"2024-04-16T16:21:26.623238+0000","last_unstale":"2024-04-16T16:21:27.627808+0000","last_undegraded":"2024-04-16T16:21:27.627808+0000","last_fullsized":"2024-04-16T16:21:27.627808+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:16:09.049767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644691+0000","last_change":"2024-04-16T16:21:26.621932+0000","last_active":"2024-04-16T16:21:28.644691+0000","last_peered":"2024-04-16T16:21:28.644691+0000","last_clean":"2024-04-16T16:21:28.644691+0000","last_became_active":"2024-04-16T16:21:26.621590+0000","last_became_peered":"2024-04-16T16:21:26.621590+0000","last_unstale":"2024-04-16T16:21:28.644691+0000","last_undegraded":"2024-04-16T16:21:28.644691+0000","last_fullsized":"2024-04-16T16:21:28.644691+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:22:41.361055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626204+0000","last_change":"2024-04-16T16:21:26.626405+0000","last_active":"2024-04-16T16:21:27.626204+0000","last_peered":"2024-04-16T16:21:27.626204+0000","last_clean":"2024-04-16T16:21:27.626204+0000","last_became_active":"2024-04-16T16:21:26.625738+0000","last_became_peered":"2024-04-16T16:21:26.625738+0000","last_unstale":"2024-04-16T16:21:27.626204+0000","last_undegraded":"2024-04-16T16:21:27.626204+0000","last_fullsized":"2024-04-16T16:21:27.626204+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:35:45.491987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.625529+0000","last_change":"2024-04-16T16:21:26.618881+0000","last_active":"2024-04-16T16:21:27.625529+0000","last_peered":"2024-04-16T16:21:27.625529+0000","last_clean":"2024-04-16T16:21:27.625529+0000","last_became_active":"2024-04-16T16:21:26.618499+0000","last_became_peered":"2024-04-16T16:21:26.618499+0000","last_unstale":"2024-04-16T16:21:27.625529+0000","last_undegraded":"2024-04-16T16:21:27.625529+0000","last_fullsized":"2024-04-16T16:21:27.625529+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:28.393986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643268+0000","last_change":"2024-04-16T16:21:26.619336+0000","last_active":"2024-04-16T16:21:28.643268+0000","last_peered":"2024-04-16T16:21:28.643268+0000","last_clean":"2024-04-16T16:21:28.643268+0000","last_became_active":"2024-04-16T16:21:26.618973+0000","last_became_peered":"2024-04-16T16:21:26.618973+0000","last_unstale":"2024-04-16T16:21:28.643268+0000","last_undegraded":"2024-04-16T16:21:28.643268+0000","last_fullsized":"2024-04-16T16:21:28.643268+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:56.732655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628517+0000","last_change":"2024-04-16T16:21:26.621609+0000","last_active":"2024-04-16T16:21:27.628517+0000","last_peered":"2024-04-16T16:21:27.628517+0000","last_clean":"2024-04-16T16:21:27.628517+0000","last_became_active":"2024-04-16T16:21:26.621328+0000","last_became_peered":"2024-04-16T16:21:26.621328+0000","last_unstale":"2024-04-16T16:21:27.628517+0000","last_undegraded":"2024-04-16T16:21:27.628517+0000","last_fullsized":"2024-04-16T16:21:27.628517+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:16.919182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644161+0000","last_change":"2024-04-16T16:21:26.628792+0000","last_active":"2024-04-16T16:21:28.644161+0000","last_peered":"2024-04-16T16:21:28.644161+0000","last_clean":"2024-04-16T16:21:28.644161+0000","last_became_active":"2024-04-16T16:21:26.628437+0000","last_became_peered":"2024-04-16T16:21:26.628437+0000","last_unstale":"2024-04-16T16:21:28.644161+0000","last_undegraded":"2024-04-16T16:21:28.644161+0000","last_fullsized":"2024-04-16T16:21:28.644161+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:38:03.914829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610959+0000","last_change":"2024-04-16T16:21:26.627834+0000","last_active":"2024-04-16T16:21:28.610959+0000","last_peered":"2024-04-16T16:21:28.610959+0000","last_clean":"2024-04-16T16:21:28.610959+0000","last_became_active":"2024-04-16T16:21:26.626611+0000","last_became_peered":"2024-04-16T16:21:26.626611+0000","last_unstale":"2024-04-16T16:21:28.610959+0000","last_undegraded":"2024-04-16T16:21:28.610959+0000","last_fullsized":"2024-04-16T16:21:28.610959+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:38.654900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643963+0000","last_change":"2024-04-16T16:21:26.618141+0000","last_active":"2024-04-16T16:21:28.643963+0000","last_peered":"2024-04-16T16:21:28.643963+0000","last_clean":"2024-04-16T16:21:28.643963+0000","last_became_active":"2024-04-16T16:21:26.614289+0000","last_became_peered":"2024-04-16T16:21:26.614289+0000","last_unstale":"2024-04-16T16:21:28.643963+0000","last_undegraded":"2024-04-16T16:21:28.643963+0000","last_fullsized":"2024-04-16T16:21:28.643963+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:15:22.005025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627713+0000","last_change":"2024-04-16T16:21:26.633948+0000","last_active":"2024-04-16T16:21:27.627713+0000","last_peered":"2024-04-16T16:21:27.627713+0000","last_clean":"2024-04-16T16:21:27.627713+0000","last_became_active":"2024-04-16T16:21:26.631959+0000","last_became_peered":"2024-04-16T16:21:26.631959+0000","last_unstale":"2024-04-16T16:21:27.627713+0000","last_undegraded":"2024-04-16T16:21:27.627713+0000","last_fullsized":"2024-04-16T16:21:27.627713+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:42.748859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626655+0000","last_change":"2024-04-16T16:21:26.626656+0000","last_active":"2024-04-16T16:21:27.626655+0000","last_peered":"2024-04-16T16:21:27.626655+0000","last_clean":"2024-04-16T16:21:27.626655+0000","last_became_active":"2024-04-16T16:21:26.626102+0000","last_became_peered":"2024-04-16T16:21:26.626102+0000","last_unstale":"2024-04-16T16:21:27.626655+0000","last_undegraded":"2024-04-16T16:21:27.626655+0000","last_fullsized":"2024-04-16T16:21:27.626655+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:21:28.099485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.625579+0000","last_change":"2024-04-16T16:21:26.640121+0000","last_active":"2024-04-16T16:21:27.625579+0000","last_peered":"2024-04-16T16:21:27.625579+0000","last_clean":"2024-04-16T16:21:27.625579+0000","last_became_active":"2024-04-16T16:21:26.639851+0000","last_became_peered":"2024-04-16T16:21:26.639851+0000","last_unstale":"2024-04-16T16:21:27.625579+0000","last_undegraded":"2024-04-16T16:21:27.625579+0000","last_fullsized":"2024-04-16T16:21:27.625579+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:51:55.853674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.625755+0000","last_change":"2024-04-16T16:21:26.634952+0000","last_active":"2024-04-16T16:21:27.625755+0000","last_peered":"2024-04-16T16:21:27.625755+0000","last_clean":"2024-04-16T16:21:27.625755+0000","last_became_active":"2024-04-16T16:21:26.634627+0000","last_became_peered":"2024-04-16T16:21:26.634627+0000","last_unstale":"2024-04-16T16:21:27.625755+0000","last_undegraded":"2024-04-16T16:21:27.625755+0000","last_fullsized":"2024-04-16T16:21:27.625755+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:38.752487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643682+0000","last_change":"2024-04-16T16:21:26.623659+0000","last_active":"2024-04-16T16:21:28.643682+0000","last_peered":"2024-04-16T16:21:28.643682+0000","last_clean":"2024-04-16T16:21:28.643682+0000","last_became_active":"2024-04-16T16:21:26.623344+0000","last_became_peered":"2024-04-16T16:21:26.623344+0000","last_unstale":"2024-04-16T16:21:28.643682+0000","last_undegraded":"2024-04-16T16:21:28.643682+0000","last_fullsized":"2024-04-16T16:21:28.643682+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:36:11.193195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610902+0000","last_change":"2024-04-16T16:21:26.625781+0000","last_active":"2024-04-16T16:21:28.610902+0000","last_peered":"2024-04-16T16:21:28.610902+0000","last_clean":"2024-04-16T16:21:28.610902+0000","last_became_active":"2024-04-16T16:21:26.625477+0000","last_became_peered":"2024-04-16T16:21:26.625477+0000","last_unstale":"2024-04-16T16:21:28.610902+0000","last_undegraded":"2024-04-16T16:21:28.610902+0000","last_fullsized":"2024-04-16T16:21:28.610902+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:02.219653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643714+0000","last_change":"2024-04-16T16:21:26.636365+0000","last_active":"2024-04-16T16:21:28.643714+0000","last_peered":"2024-04-16T16:21:28.643714+0000","last_clean":"2024-04-16T16:21:28.643714+0000","last_became_active":"2024-04-16T16:21:26.636019+0000","last_became_peered":"2024-04-16T16:21:26.636019+0000","last_unstale":"2024-04-16T16:21:28.643714+0000","last_undegraded":"2024-04-16T16:21:28.643714+0000","last_fullsized":"2024-04-16T16:21:28.643714+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:12.380057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.641995+0000","last_change":"2024-04-16T16:21:26.639338+0000","last_active":"2024-04-16T16:21:28.641995+0000","last_peered":"2024-04-16T16:21:28.641995+0000","last_clean":"2024-04-16T16:21:28.641995+0000","last_became_active":"2024-04-16T16:21:26.639071+0000","last_became_peered":"2024-04-16T16:21:26.639071+0000","last_unstale":"2024-04-16T16:21:28.641995+0000","last_undegraded":"2024-04-16T16:21:28.641995+0000","last_fullsized":"2024-04-16T16:21:28.641995+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:35.235737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610040+0000","last_change":"2024-04-16T16:21:26.638327+0000","last_active":"2024-04-16T16:21:28.610040+0000","last_peered":"2024-04-16T16:21:28.610040+0000","last_clean":"2024-04-16T16:21:28.610040+0000","last_became_active":"2024-04-16T16:21:26.638007+0000","last_became_peered":"2024-04-16T16:21:26.638007+0000","last_unstale":"2024-04-16T16:21:28.610040+0000","last_undegraded":"2024-04-16T16:21:28.610040+0000","last_fullsized":"2024-04-16T16:21:28.610040+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:37:16.481358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627008+0000","last_change":"2024-04-16T16:21:26.631148+0000","last_active":"2024-04-16T16:21:27.627008+0000","last_peered":"2024-04-16T16:21:27.627008+0000","last_clean":"2024-04-16T16:21:27.627008+0000","last_became_active":"2024-04-16T16:21:26.629486+0000","last_became_peered":"2024-04-16T16:21:26.629486+0000","last_unstale":"2024-04-16T16:21:27.627008+0000","last_undegraded":"2024-04-16T16:21:27.627008+0000","last_fullsized":"2024-04-16T16:21:27.627008+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:49.214360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.622409+0000","last_change":"2024-04-16T16:21:26.633930+0000","last_active":"2024-04-16T16:21:27.622409+0000","last_peered":"2024-04-16T16:21:27.622409+0000","last_clean":"2024-04-16T16:21:27.622409+0000","last_became_active":"2024-04-16T16:21:26.633235+0000","last_became_peered":"2024-04-16T16:21:26.633235+0000","last_unstale":"2024-04-16T16:21:27.622409+0000","last_undegraded":"2024-04-16T16:21:27.622409+0000","last_fullsized":"2024-04-16T16:21:27.622409+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:06:10.334071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643829+0000","last_change":"2024-04-16T16:21:26.622683+0000","last_active":"2024-04-16T16:21:28.643829+0000","last_peered":"2024-04-16T16:21:28.643829+0000","last_clean":"2024-04-16T16:21:28.643829+0000","last_became_active":"2024-04-16T16:21:26.622308+0000","last_became_peered":"2024-04-16T16:21:26.622308+0000","last_unstale":"2024-04-16T16:21:28.643829+0000","last_undegraded":"2024-04-16T16:21:28.643829+0000","last_fullsized":"2024-04-16T16:21:28.643829+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:26.812700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628420+0000","last_change":"2024-04-16T16:21:26.624637+0000","last_active":"2024-04-16T16:21:27.628420+0000","last_peered":"2024-04-16T16:21:27.628420+0000","last_clean":"2024-04-16T16:21:27.628420+0000","last_became_active":"2024-04-16T16:21:26.623808+0000","last_became_peered":"2024-04-16T16:21:26.623808+0000","last_unstale":"2024-04-16T16:21:27.628420+0000","last_undegraded":"2024-04-16T16:21:27.628420+0000","last_fullsized":"2024-04-16T16:21:27.628420+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:06.714681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"141'4","reported_seq":18,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628451+0000","last_change":"2024-04-16T16:21:25.597818+0000","last_active":"2024-04-16T16:21:27.628451+0000","last_peered":"2024-04-16T16:21:27.628451+0000","last_clean":"2024-04-16T16:21:27.628451+0000","last_became_active":"2024-04-16T16:21:25.597506+0000","last_became_peered":"2024-04-16T16:21:25.597506+0000","last_unstale":"2024-04-16T16:21:27.628451+0000","last_undegraded":"2024-04-16T16:21:27.628451+0000","last_fullsized":"2024-04-16T16:21:27.628451+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:29:15.239160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.623236+0000","last_change":"2024-04-16T16:21:26.619178+0000","last_active":"2024-04-16T16:21:27.623236+0000","last_peered":"2024-04-16T16:21:27.623236+0000","last_clean":"2024-04-16T16:21:27.623236+0000","last_became_active":"2024-04-16T16:21:26.618887+0000","last_became_peered":"2024-04-16T16:21:26.618887+0000","last_unstale":"2024-04-16T16:21:27.623236+0000","last_undegraded":"2024-04-16T16:21:27.623236+0000","last_fullsized":"2024-04-16T16:21:27.623236+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:07:10.543796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.623327+0000","last_change":"2024-04-16T16:21:25.603465+0000","last_active":"2024-04-16T16:21:27.623327+0000","last_peered":"2024-04-16T16:21:27.623327+0000","last_clean":"2024-04-16T16:21:27.623327+0000","last_became_active":"2024-04-16T16:21:25.602980+0000","last_became_peered":"2024-04-16T16:21:25.602980+0000","last_unstale":"2024-04-16T16:21:27.623327+0000","last_undegraded":"2024-04-16T16:21:27.623327+0000","last_fullsized":"2024-04-16T16:21:27.623327+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:39.706251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610226+0000","last_change":"2024-04-16T16:21:26.635213+0000","last_active":"2024-04-16T16:21:28.610226+0000","last_peered":"2024-04-16T16:21:28.610226+0000","last_clean":"2024-04-16T16:21:28.610226+0000","last_became_active":"2024-04-16T16:21:26.634409+0000","last_became_peered":"2024-04-16T16:21:26.634409+0000","last_unstale":"2024-04-16T16:21:28.610226+0000","last_undegraded":"2024-04-16T16:21:28.610226+0000","last_fullsized":"2024-04-16T16:21:28.610226+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:03:26.310046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627258+0000","last_change":"2024-04-16T16:21:25.596356+0000","last_active":"2024-04-16T16:21:27.627258+0000","last_peered":"2024-04-16T16:21:27.627258+0000","last_clean":"2024-04-16T16:21:27.627258+0000","last_became_active":"2024-04-16T16:21:25.594551+0000","last_became_peered":"2024-04-16T16:21:25.594551+0000","last_unstale":"2024-04-16T16:21:27.627258+0000","last_undegraded":"2024-04-16T16:21:27.627258+0000","last_fullsized":"2024-04-16T16:21:27.627258+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:20:30.028756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627848+0000","last_change":"2024-04-16T16:21:26.623606+0000","last_active":"2024-04-16T16:21:27.627848+0000","last_peered":"2024-04-16T16:21:27.627848+0000","last_clean":"2024-04-16T16:21:27.627848+0000","last_became_active":"2024-04-16T16:21:26.623161+0000","last_became_peered":"2024-04-16T16:21:26.623161+0000","last_unstale":"2024-04-16T16:21:27.627848+0000","last_undegraded":"2024-04-16T16:21:27.627848+0000","last_fullsized":"2024-04-16T16:21:27.627848+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:07:10.543796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626876+0000","last_change":"2024-04-16T16:21:25.592173+0000","last_active":"2024-04-16T16:21:27.626876+0000","last_peered":"2024-04-16T16:21:27.626876+0000","last_clean":"2024-04-16T16:21:27.626876+0000","last_became_active":"2024-04-16T16:21:25.591898+0000","last_became_peered":"2024-04-16T16:21:25.591898+0000","last_unstale":"2024-04-16T16:21:27.626876+0000","last_undegraded":"2024-04-16T16:21:27.626876+0000","last_fullsized":"2024-04-16T16:21:27.626876+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:03:25.295410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.622502+0000","last_change":"2024-04-16T16:21:26.633513+0000","last_active":"2024-04-16T16:21:27.622502+0000","last_peered":"2024-04-16T16:21:27.622502+0000","last_clean":"2024-04-16T16:21:27.622502+0000","last_became_active":"2024-04-16T16:21:26.633154+0000","last_became_peered":"2024-04-16T16:21:26.633154+0000","last_unstale":"2024-04-16T16:21:27.622502+0000","last_undegraded":"2024-04-16T16:21:27.622502+0000","last_fullsized":"2024-04-16T16:21:27.622502+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:27.121241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644284+0000","last_change":"2024-04-16T16:21:25.588337+0000","last_active":"2024-04-16T16:21:28.644284+0000","last_peered":"2024-04-16T16:21:28.644284+0000","last_clean":"2024-04-16T16:21:28.644284+0000","last_became_active":"2024-04-16T16:21:25.583483+0000","last_became_peered":"2024-04-16T16:21:25.583483+0000","last_unstale":"2024-04-16T16:21:28.644284+0000","last_undegraded":"2024-04-16T16:21:28.644284+0000","last_fullsized":"2024-04-16T16:21:28.644284+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:10:51.761699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627525+0000","last_change":"2024-04-16T16:21:26.638002+0000","last_active":"2024-04-16T16:21:27.627525+0000","last_peered":"2024-04-16T16:21:27.627525+0000","last_clean":"2024-04-16T16:21:27.627525+0000","last_became_active":"2024-04-16T16:21:26.637712+0000","last_became_peered":"2024-04-16T16:21:26.637712+0000","last_unstale":"2024-04-16T16:21:27.627525+0000","last_undegraded":"2024-04-16T16:21:27.627525+0000","last_fullsized":"2024-04-16T16:21:27.627525+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:38:03.914829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644045+0000","last_change":"2024-04-16T16:21:25.600581+0000","last_active":"2024-04-16T16:21:28.644045+0000","last_peered":"2024-04-16T16:21:28.644045+0000","last_clean":"2024-04-16T16:21:28.644045+0000","last_became_active":"2024-04-16T16:21:25.600245+0000","last_became_peered":"2024-04-16T16:21:25.600245+0000","last_unstale":"2024-04-16T16:21:28.644045+0000","last_undegraded":"2024-04-16T16:21:28.644045+0000","last_fullsized":"2024-04-16T16:21:28.644045+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:49:43.421377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.622668+0000","last_change":"2024-04-16T16:21:26.620432+0000","last_active":"2024-04-16T16:21:27.622668+0000","last_peered":"2024-04-16T16:21:27.622668+0000","last_clean":"2024-04-16T16:21:27.622668+0000","last_became_active":"2024-04-16T16:21:26.620036+0000","last_became_peered":"2024-04-16T16:21:26.620036+0000","last_unstale":"2024-04-16T16:21:27.622668+0000","last_undegraded":"2024-04-16T16:21:27.622668+0000","last_fullsized":"2024-04-16T16:21:27.622668+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:28.393986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643197+0000","last_change":"2024-04-16T16:21:25.595882+0000","last_active":"2024-04-16T16:21:28.643197+0000","last_peered":"2024-04-16T16:21:28.643197+0000","last_clean":"2024-04-16T16:21:28.643197+0000","last_became_active":"2024-04-16T16:21:25.595359+0000","last_became_peered":"2024-04-16T16:21:25.595359+0000","last_unstale":"2024-04-16T16:21:28.643197+0000","last_undegraded":"2024-04-16T16:21:28.643197+0000","last_fullsized":"2024-04-16T16:21:28.643197+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:53.324244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.645266+0000","last_change":"2024-04-16T16:21:26.622580+0000","last_active":"2024-04-16T16:21:28.645266+0000","last_peered":"2024-04-16T16:21:28.645266+0000","last_clean":"2024-04-16T16:21:28.645266+0000","last_became_active":"2024-04-16T16:21:26.621727+0000","last_became_peered":"2024-04-16T16:21:26.621727+0000","last_unstale":"2024-04-16T16:21:28.645266+0000","last_undegraded":"2024-04-16T16:21:28.645266+0000","last_fullsized":"2024-04-16T16:21:28.645266+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:04:09.629772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.625268+0000","last_change":"2024-04-16T16:21:25.598600+0000","last_active":"2024-04-16T16:21:27.625268+0000","last_peered":"2024-04-16T16:21:27.625268+0000","last_clean":"2024-04-16T16:21:27.625268+0000","last_became_active":"2024-04-16T16:21:25.598251+0000","last_became_peered":"2024-04-16T16:21:25.598251+0000","last_unstale":"2024-04-16T16:21:27.625268+0000","last_undegraded":"2024-04-16T16:21:27.625268+0000","last_fullsized":"2024-04-16T16:21:27.625268+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:56:21.905428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644889+0000","last_change":"2024-04-16T16:21:26.626469+0000","last_active":"2024-04-16T16:21:28.644889+0000","last_peered":"2024-04-16T16:21:28.644889+0000","last_clean":"2024-04-16T16:21:28.644889+0000","last_became_active":"2024-04-16T16:21:26.625974+0000","last_became_peered":"2024-04-16T16:21:26.625974+0000","last_unstale":"2024-04-16T16:21:28.644889+0000","last_undegraded":"2024-04-16T16:21:28.644889+0000","last_fullsized":"2024-04-16T16:21:28.644889+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:56:22.920064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.623204+0000","last_change":"2024-04-16T16:21:25.592714+0000","last_active":"2024-04-16T16:21:27.623204+0000","last_peered":"2024-04-16T16:21:27.623204+0000","last_clean":"2024-04-16T16:21:27.623204+0000","last_became_active":"2024-04-16T16:21:25.592418+0000","last_became_peered":"2024-04-16T16:21:25.592418+0000","last_unstale":"2024-04-16T16:21:27.623204+0000","last_undegraded":"2024-04-16T16:21:27.623204+0000","last_fullsized":"2024-04-16T16:21:27.623204+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:29:50.499359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628355+0000","last_change":"2024-04-16T16:21:26.619071+0000","last_active":"2024-04-16T16:21:27.628355+0000","last_peered":"2024-04-16T16:21:27.628355+0000","last_clean":"2024-04-16T16:21:27.628355+0000","last_became_active":"2024-04-16T16:21:26.618660+0000","last_became_peered":"2024-04-16T16:21:26.618660+0000","last_unstale":"2024-04-16T16:21:27.628355+0000","last_undegraded":"2024-04-16T16:21:27.628355+0000","last_fullsized":"2024-04-16T16:21:27.628355+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:02:34.201782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628390+0000","last_change":"2024-04-16T16:21:25.601618+0000","last_active":"2024-04-16T16:21:27.628390+0000","last_peered":"2024-04-16T16:21:27.628390+0000","last_clean":"2024-04-16T16:21:27.628390+0000","last_became_active":"2024-04-16T16:21:25.600831+0000","last_became_peered":"2024-04-16T16:21:25.600831+0000","last_unstale":"2024-04-16T16:21:27.628390+0000","last_undegraded":"2024-04-16T16:21:27.628390+0000","last_fullsized":"2024-04-16T16:21:27.628390+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:29:50.499359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644198+0000","last_change":"2024-04-16T16:21:26.618860+0000","last_active":"2024-04-16T16:21:28.644198+0000","last_peered":"2024-04-16T16:21:28.644198+0000","last_clean":"2024-04-16T16:21:28.644198+0000","last_became_active":"2024-04-16T16:21:26.618520+0000","last_became_peered":"2024-04-16T16:21:26.618520+0000","last_unstale":"2024-04-16T16:21:28.644198+0000","last_undegraded":"2024-04-16T16:21:28.644198+0000","last_fullsized":"2024-04-16T16:21:28.644198+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:13.744613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"141'6","reported_seq":20,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.642299+0000","last_change":"2024-04-16T16:21:25.595231+0000","last_active":"2024-04-16T16:21:28.642299+0000","last_peered":"2024-04-16T16:21:28.642299+0000","last_clean":"2024-04-16T16:21:28.642299+0000","last_became_active":"2024-04-16T16:21:25.594379+0000","last_became_peered":"2024-04-16T16:21:25.594379+0000","last_unstale":"2024-04-16T16:21:28.642299+0000","last_undegraded":"2024-04-16T16:21:28.642299+0000","last_fullsized":"2024-04-16T16:21:28.642299+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:52:49.593567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.642913+0000","last_change":"2024-04-16T16:21:26.632827+0000","last_active":"2024-04-16T16:21:28.642913+0000","last_peered":"2024-04-16T16:21:28.642913+0000","last_clean":"2024-04-16T16:21:28.642913+0000","last_became_active":"2024-04-16T16:21:26.632152+0000","last_became_peered":"2024-04-16T16:21:26.632152+0000","last_unstale":"2024-04-16T16:21:28.642913+0000","last_undegraded":"2024-04-16T16:21:28.642913+0000","last_fullsized":"2024-04-16T16:21:28.642913+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:55.067333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.642992+0000","last_change":"2024-04-16T16:21:25.604040+0000","last_active":"2024-04-16T16:21:28.642992+0000","last_peered":"2024-04-16T16:21:28.642992+0000","last_clean":"2024-04-16T16:21:28.642992+0000","last_became_active":"2024-04-16T16:21:25.603726+0000","last_became_peered":"2024-04-16T16:21:25.603726+0000","last_unstale":"2024-04-16T16:21:28.642992+0000","last_undegraded":"2024-04-16T16:21:28.642992+0000","last_fullsized":"2024-04-16T16:21:28.642992+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:20:30.028756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.622865+0000","last_change":"2024-04-16T16:21:26.618805+0000","last_active":"2024-04-16T16:21:27.622865+0000","last_peered":"2024-04-16T16:21:27.622865+0000","last_clean":"2024-04-16T16:21:27.622865+0000","last_became_active":"2024-04-16T16:21:26.618424+0000","last_became_peered":"2024-04-16T16:21:26.618424+0000","last_unstale":"2024-04-16T16:21:27.622865+0000","last_undegraded":"2024-04-16T16:21:27.622865+0000","last_fullsized":"2024-04-16T16:21:27.622865+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:38.752487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643213+0000","last_change":"2024-04-16T16:21:25.603576+0000","last_active":"2024-04-16T16:21:28.643213+0000","last_peered":"2024-04-16T16:21:28.643213+0000","last_clean":"2024-04-16T16:21:28.643213+0000","last_became_active":"2024-04-16T16:21:25.602512+0000","last_became_peered":"2024-04-16T16:21:25.602512+0000","last_unstale":"2024-04-16T16:21:28.643213+0000","last_undegraded":"2024-04-16T16:21:28.643213+0000","last_fullsized":"2024-04-16T16:21:28.643213+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:46:07.323352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643715+0000","last_change":"2024-04-16T16:21:26.619141+0000","last_active":"2024-04-16T16:21:28.643715+0000","last_peered":"2024-04-16T16:21:28.643715+0000","last_clean":"2024-04-16T16:21:28.643715+0000","last_became_active":"2024-04-16T16:21:26.618765+0000","last_became_peered":"2024-04-16T16:21:26.618765+0000","last_unstale":"2024-04-16T16:21:28.643715+0000","last_undegraded":"2024-04-16T16:21:28.643715+0000","last_fullsized":"2024-04-16T16:21:28.643715+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:05:37.498029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.609518+0000","last_change":"2024-04-16T16:21:25.593260+0000","last_active":"2024-04-16T16:21:28.609518+0000","last_peered":"2024-04-16T16:21:28.609518+0000","last_clean":"2024-04-16T16:21:28.609518+0000","last_became_active":"2024-04-16T16:21:25.592707+0000","last_became_peered":"2024-04-16T16:21:25.592707+0000","last_unstale":"2024-04-16T16:21:28.609518+0000","last_undegraded":"2024-04-16T16:21:28.609518+0000","last_fullsized":"2024-04-16T16:21:28.609518+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:49:43.421377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.642498+0000","last_change":"2024-04-16T16:21:26.628634+0000","last_active":"2024-04-16T16:21:28.642498+0000","last_peered":"2024-04-16T16:21:28.642498+0000","last_clean":"2024-04-16T16:21:28.642498+0000","last_became_active":"2024-04-16T16:21:26.628341+0000","last_became_peered":"2024-04-16T16:21:26.628341+0000","last_unstale":"2024-04-16T16:21:28.642498+0000","last_undegraded":"2024-04-16T16:21:28.642498+0000","last_fullsized":"2024-04-16T16:21:28.642498+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:21:28.099485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"141'8","reported_seq":22,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643881+0000","last_change":"2024-04-16T16:21:25.600083+0000","last_active":"2024-04-16T16:21:28.643881+0000","last_peered":"2024-04-16T16:21:28.643881+0000","last_clean":"2024-04-16T16:21:28.643881+0000","last_became_active":"2024-04-16T16:21:25.599688+0000","last_became_peered":"2024-04-16T16:21:25.599688+0000","last_unstale":"2024-04-16T16:21:28.643881+0000","last_undegraded":"2024-04-16T16:21:28.643881+0000","last_fullsized":"2024-04-16T16:21:28.643881+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:31.123443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628305+0000","last_change":"2024-04-16T16:21:26.619132+0000","last_active":"2024-04-16T16:21:27.628305+0000","last_peered":"2024-04-16T16:21:27.628305+0000","last_clean":"2024-04-16T16:21:27.628305+0000","last_became_active":"2024-04-16T16:21:26.618806+0000","last_became_peered":"2024-04-16T16:21:26.618806+0000","last_unstale":"2024-04-16T16:21:27.628305+0000","last_undegraded":"2024-04-16T16:21:27.628305+0000","last_fullsized":"2024-04-16T16:21:27.628305+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:51:38.560726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.645024+0000","last_change":"2024-04-16T16:21:25.604548+0000","last_active":"2024-04-16T16:21:28.645024+0000","last_peered":"2024-04-16T16:21:28.645024+0000","last_clean":"2024-04-16T16:21:28.645024+0000","last_became_active":"2024-04-16T16:21:25.603623+0000","last_became_peered":"2024-04-16T16:21:25.603623+0000","last_unstale":"2024-04-16T16:21:28.645024+0000","last_undegraded":"2024-04-16T16:21:28.645024+0000","last_fullsized":"2024-04-16T16:21:28.645024+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:02:28.264111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643880+0000","last_change":"2024-04-16T16:21:26.623525+0000","last_active":"2024-04-16T16:21:28.643880+0000","last_peered":"2024-04-16T16:21:28.643880+0000","last_clean":"2024-04-16T16:21:28.643880+0000","last_became_active":"2024-04-16T16:21:26.620561+0000","last_became_peered":"2024-04-16T16:21:26.620561+0000","last_unstale":"2024-04-16T16:21:28.643880+0000","last_undegraded":"2024-04-16T16:21:28.643880+0000","last_fullsized":"2024-04-16T16:21:28.643880+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:27.435258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.623337+0000","last_change":"2024-04-16T16:21:25.604810+0000","last_active":"2024-04-16T16:21:27.623337+0000","last_peered":"2024-04-16T16:21:27.623337+0000","last_clean":"2024-04-16T16:21:27.623337+0000","last_became_active":"2024-04-16T16:21:25.604495+0000","last_became_peered":"2024-04-16T16:21:25.604495+0000","last_unstale":"2024-04-16T16:21:27.623337+0000","last_undegraded":"2024-04-16T16:21:27.623337+0000","last_fullsized":"2024-04-16T16:21:27.623337+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:39:03.212104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643558+0000","last_change":"2024-04-16T16:21:26.632649+0000","last_active":"2024-04-16T16:21:28.643558+0000","last_peered":"2024-04-16T16:21:28.643558+0000","last_clean":"2024-04-16T16:21:28.643558+0000","last_became_active":"2024-04-16T16:21:26.632334+0000","last_became_peered":"2024-04-16T16:21:26.632334+0000","last_unstale":"2024-04-16T16:21:28.643558+0000","last_undegraded":"2024-04-16T16:21:28.643558+0000","last_fullsized":"2024-04-16T16:21:28.643558+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:49.214360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644128+0000","last_change":"2024-04-16T16:21:25.586171+0000","last_active":"2024-04-16T16:21:28.644128+0000","last_peered":"2024-04-16T16:21:28.644128+0000","last_clean":"2024-04-16T16:21:28.644128+0000","last_became_active":"2024-04-16T16:21:25.585863+0000","last_became_peered":"2024-04-16T16:21:25.585863+0000","last_unstale":"2024-04-16T16:21:28.644128+0000","last_undegraded":"2024-04-16T16:21:28.644128+0000","last_fullsized":"2024-04-16T16:21:28.644128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:28:09.057461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610500+0000","last_change":"2024-04-16T16:21:26.639412+0000","last_active":"2024-04-16T16:21:28.610500+0000","last_peered":"2024-04-16T16:21:28.610500+0000","last_clean":"2024-04-16T16:21:28.610500+0000","last_became_active":"2024-04-16T16:21:26.638086+0000","last_became_peered":"2024-04-16T16:21:26.638086+0000","last_unstale":"2024-04-16T16:21:28.610500+0000","last_undegraded":"2024-04-16T16:21:28.610500+0000","last_fullsized":"2024-04-16T16:21:28.610500+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:07:07.900771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.623013+0000","last_change":"2024-04-16T16:21:25.596529+0000","last_active":"2024-04-16T16:21:27.623013+0000","last_peered":"2024-04-16T16:21:27.623013+0000","last_clean":"2024-04-16T16:21:27.623013+0000","last_became_active":"2024-04-16T16:21:25.594866+0000","last_became_peered":"2024-04-16T16:21:25.594866+0000","last_unstale":"2024-04-16T16:21:27.623013+0000","last_undegraded":"2024-04-16T16:21:27.623013+0000","last_fullsized":"2024-04-16T16:21:27.623013+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:29:15.239160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627424+0000","last_change":"2024-04-16T16:21:26.627838+0000","last_active":"2024-04-16T16:21:27.627424+0000","last_peered":"2024-04-16T16:21:27.627424+0000","last_clean":"2024-04-16T16:21:27.627424+0000","last_became_active":"2024-04-16T16:21:26.627034+0000","last_became_peered":"2024-04-16T16:21:26.627034+0000","last_unstale":"2024-04-16T16:21:27.627424+0000","last_undegraded":"2024-04-16T16:21:27.627424+0000","last_fullsized":"2024-04-16T16:21:27.627424+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:49:46.979105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.641478+0000","last_change":"2024-04-16T16:21:25.600929+0000","last_active":"2024-04-16T16:21:28.641478+0000","last_peered":"2024-04-16T16:21:28.641478+0000","last_clean":"2024-04-16T16:21:28.641478+0000","last_became_active":"2024-04-16T16:21:25.600350+0000","last_became_peered":"2024-04-16T16:21:25.600350+0000","last_unstale":"2024-04-16T16:21:28.641478+0000","last_undegraded":"2024-04-16T16:21:28.641478+0000","last_fullsized":"2024-04-16T16:21:28.641478+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:27.379350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.641517+0000","last_change":"2024-04-16T16:21:26.637010+0000","last_active":"2024-04-16T16:21:28.641517+0000","last_peered":"2024-04-16T16:21:28.641517+0000","last_clean":"2024-04-16T16:21:28.641517+0000","last_became_active":"2024-04-16T16:21:26.636730+0000","last_became_peered":"2024-04-16T16:21:26.636730+0000","last_unstale":"2024-04-16T16:21:28.641517+0000","last_undegraded":"2024-04-16T16:21:28.641517+0000","last_fullsized":"2024-04-16T16:21:28.641517+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:16.919182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627173+0000","last_change":"2024-04-16T16:21:25.598152+0000","last_active":"2024-04-16T16:21:27.627173+0000","last_peered":"2024-04-16T16:21:27.627173+0000","last_clean":"2024-04-16T16:21:27.627173+0000","last_became_active":"2024-04-16T16:21:25.597731+0000","last_became_peered":"2024-04-16T16:21:25.597731+0000","last_unstale":"2024-04-16T16:21:27.627173+0000","last_undegraded":"2024-04-16T16:21:27.627173+0000","last_fullsized":"2024-04-16T16:21:27.627173+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:54:51.270429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.625193+0000","last_change":"2024-04-16T16:21:26.633445+0000","last_active":"2024-04-16T16:21:27.625193+0000","last_peered":"2024-04-16T16:21:27.625193+0000","last_clean":"2024-04-16T16:21:27.625193+0000","last_became_active":"2024-04-16T16:21:26.633143+0000","last_became_peered":"2024-04-16T16:21:26.633143+0000","last_unstale":"2024-04-16T16:21:27.625193+0000","last_undegraded":"2024-04-16T16:21:27.625193+0000","last_fullsized":"2024-04-16T16:21:27.625193+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:39.911246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626337+0000","last_change":"2024-04-16T16:21:25.597534+0000","last_active":"2024-04-16T16:21:27.626337+0000","last_peered":"2024-04-16T16:21:27.626337+0000","last_clean":"2024-04-16T16:21:27.626337+0000","last_became_active":"2024-04-16T16:21:25.597218+0000","last_became_peered":"2024-04-16T16:21:25.597218+0000","last_unstale":"2024-04-16T16:21:27.626337+0000","last_undegraded":"2024-04-16T16:21:27.626337+0000","last_fullsized":"2024-04-16T16:21:27.626337+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:39.706251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.641220+0000","last_change":"2024-04-16T16:21:26.638223+0000","last_active":"2024-04-16T16:21:28.641220+0000","last_peered":"2024-04-16T16:21:28.641220+0000","last_clean":"2024-04-16T16:21:28.641220+0000","last_became_active":"2024-04-16T16:21:26.637929+0000","last_became_peered":"2024-04-16T16:21:26.637929+0000","last_unstale":"2024-04-16T16:21:28.641220+0000","last_undegraded":"2024-04-16T16:21:28.641220+0000","last_fullsized":"2024-04-16T16:21:28.641220+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:02:34.201782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.642044+0000","last_change":"2024-04-16T16:21:25.601886+0000","last_active":"2024-04-16T16:21:28.642044+0000","last_peered":"2024-04-16T16:21:28.642044+0000","last_clean":"2024-04-16T16:21:28.642044+0000","last_became_active":"2024-04-16T16:21:25.600879+0000","last_became_peered":"2024-04-16T16:21:25.600879+0000","last_unstale":"2024-04-16T16:21:28.642044+0000","last_undegraded":"2024-04-16T16:21:28.642044+0000","last_fullsized":"2024-04-16T16:21:28.642044+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:03:25.295410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.645104+0000","last_change":"2024-04-16T16:21:26.617017+0000","last_active":"2024-04-16T16:21:28.645104+0000","last_peered":"2024-04-16T16:21:28.645104+0000","last_clean":"2024-04-16T16:21:28.645104+0000","last_became_active":"2024-04-16T16:21:26.616626+0000","last_became_peered":"2024-04-16T16:21:26.616626+0000","last_unstale":"2024-04-16T16:21:28.645104+0000","last_undegraded":"2024-04-16T16:21:28.645104+0000","last_fullsized":"2024-04-16T16:21:28.645104+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:37:16.481358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628575+0000","last_change":"2024-04-16T16:21:25.605051+0000","last_active":"2024-04-16T16:21:27.628575+0000","last_peered":"2024-04-16T16:21:27.628575+0000","last_clean":"2024-04-16T16:21:27.628575+0000","last_became_active":"2024-04-16T16:21:25.604679+0000","last_became_peered":"2024-04-16T16:21:25.604679+0000","last_unstale":"2024-04-16T16:21:27.628575+0000","last_undegraded":"2024-04-16T16:21:27.628575+0000","last_fullsized":"2024-04-16T16:21:27.628575+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:39:03.212104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.645393+0000","last_change":"2024-04-16T16:21:26.632757+0000","last_active":"2024-04-16T16:21:28.645393+0000","last_peered":"2024-04-16T16:21:28.645393+0000","last_clean":"2024-04-16T16:21:28.645393+0000","last_became_active":"2024-04-16T16:21:26.632460+0000","last_became_peered":"2024-04-16T16:21:26.632460+0000","last_unstale":"2024-04-16T16:21:28.645393+0000","last_undegraded":"2024-04-16T16:21:28.645393+0000","last_fullsized":"2024-04-16T16:21:28.645393+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:26.590328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"142'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610449+0000","last_change":"2024-04-16T16:21:25.585117+0000","last_active":"2024-04-16T16:21:28.610449+0000","last_peered":"2024-04-16T16:21:28.610449+0000","last_clean":"2024-04-16T16:21:28.610449+0000","last_became_active":"2024-04-16T16:21:25.584609+0000","last_became_peered":"2024-04-16T16:21:25.584609+0000","last_unstale":"2024-04-16T16:21:28.610449+0000","last_undegraded":"2024-04-16T16:21:28.610449+0000","last_fullsized":"2024-04-16T16:21:28.610449+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:01.917706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.622948+0000","last_change":"2024-04-16T16:21:26.635364+0000","last_active":"2024-04-16T16:21:27.622948+0000","last_peered":"2024-04-16T16:21:27.622948+0000","last_clean":"2024-04-16T16:21:27.622948+0000","last_became_active":"2024-04-16T16:21:26.635101+0000","last_became_peered":"2024-04-16T16:21:26.635101+0000","last_unstale":"2024-04-16T16:21:27.622948+0000","last_undegraded":"2024-04-16T16:21:27.622948+0000","last_fullsized":"2024-04-16T16:21:27.622948+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:52:50.608203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610772+0000","last_change":"2024-04-16T16:21:25.589018+0000","last_active":"2024-04-16T16:21:28.610772+0000","last_peered":"2024-04-16T16:21:28.610772+0000","last_clean":"2024-04-16T16:21:28.610772+0000","last_became_active":"2024-04-16T16:21:25.588722+0000","last_became_peered":"2024-04-16T16:21:25.588722+0000","last_unstale":"2024-04-16T16:21:28.610772+0000","last_undegraded":"2024-04-16T16:21:28.610772+0000","last_fullsized":"2024-04-16T16:21:28.610772+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:31:45.126004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610676+0000","last_change":"2024-04-16T16:21:26.630528+0000","last_active":"2024-04-16T16:21:28.610676+0000","last_peered":"2024-04-16T16:21:28.610676+0000","last_clean":"2024-04-16T16:21:28.610676+0000","last_became_active":"2024-04-16T16:21:26.630235+0000","last_became_peered":"2024-04-16T16:21:26.630235+0000","last_unstale":"2024-04-16T16:21:28.610676+0000","last_undegraded":"2024-04-16T16:21:28.610676+0000","last_fullsized":"2024-04-16T16:21:28.610676+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:46:08.337988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610296+0000","last_change":"2024-04-16T16:21:25.582293+0000","last_active":"2024-04-16T16:21:28.610296+0000","last_peered":"2024-04-16T16:21:28.610296+0000","last_clean":"2024-04-16T16:21:28.610296+0000","last_became_active":"2024-04-16T16:21:25.581912+0000","last_became_peered":"2024-04-16T16:21:25.581912+0000","last_unstale":"2024-04-16T16:21:28.610296+0000","last_undegraded":"2024-04-16T16:21:28.610296+0000","last_fullsized":"2024-04-16T16:21:28.610296+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:02:33.389254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643444+0000","last_change":"2024-04-16T16:21:26.629185+0000","last_active":"2024-04-16T16:21:28.643444+0000","last_peered":"2024-04-16T16:21:28.643444+0000","last_clean":"2024-04-16T16:21:28.643444+0000","last_became_active":"2024-04-16T16:21:26.628800+0000","last_became_peered":"2024-04-16T16:21:26.628800+0000","last_unstale":"2024-04-16T16:21:28.643444+0000","last_undegraded":"2024-04-16T16:21:28.643444+0000","last_fullsized":"2024-04-16T16:21:28.643444+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:56.310523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.641618+0000","last_change":"2024-04-16T16:21:25.598027+0000","last_active":"2024-04-16T16:21:28.641618+0000","last_peered":"2024-04-16T16:21:28.641618+0000","last_clean":"2024-04-16T16:21:28.641618+0000","last_became_active":"2024-04-16T16:21:25.597733+0000","last_became_peered":"2024-04-16T16:21:25.597733+0000","last_unstale":"2024-04-16T16:21:28.641618+0000","last_undegraded":"2024-04-16T16:21:28.641618+0000","last_fullsized":"2024-04-16T16:21:28.641618+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:37.737851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644991+0000","last_change":"2024-04-16T16:21:26.631846+0000","last_active":"2024-04-16T16:21:28.644991+0000","last_peered":"2024-04-16T16:21:28.644991+0000","last_clean":"2024-04-16T16:21:28.644991+0000","last_became_active":"2024-04-16T16:21:26.631515+0000","last_became_peered":"2024-04-16T16:21:26.631515+0000","last_unstale":"2024-04-16T16:21:28.644991+0000","last_undegraded":"2024-04-16T16:21:28.644991+0000","last_fullsized":"2024-04-16T16:21:28.644991+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:20.296700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626953+0000","last_change":"2024-04-16T16:21:25.600334+0000","last_active":"2024-04-16T16:21:27.626953+0000","last_peered":"2024-04-16T16:21:27.626953+0000","last_clean":"2024-04-16T16:21:27.626953+0000","last_became_active":"2024-04-16T16:21:25.599921+0000","last_became_peered":"2024-04-16T16:21:25.599921+0000","last_unstale":"2024-04-16T16:21:27.626953+0000","last_undegraded":"2024-04-16T16:21:27.626953+0000","last_fullsized":"2024-04-16T16:21:27.626953+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:06.016163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626017+0000","last_change":"2024-04-16T16:21:26.624365+0000","last_active":"2024-04-16T16:21:27.626017+0000","last_peered":"2024-04-16T16:21:27.626017+0000","last_clean":"2024-04-16T16:21:27.626017+0000","last_became_active":"2024-04-16T16:21:26.624052+0000","last_became_peered":"2024-04-16T16:21:26.624052+0000","last_unstale":"2024-04-16T16:21:27.626017+0000","last_undegraded":"2024-04-16T16:21:27.626017+0000","last_fullsized":"2024-04-16T16:21:27.626017+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:52:50.608203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.642377+0000","last_change":"2024-04-16T16:21:25.602774+0000","last_active":"2024-04-16T16:21:28.642377+0000","last_peered":"2024-04-16T16:21:28.642377+0000","last_clean":"2024-04-16T16:21:28.642377+0000","last_became_active":"2024-04-16T16:21:25.602465+0000","last_became_peered":"2024-04-16T16:21:25.602465+0000","last_unstale":"2024-04-16T16:21:28.642377+0000","last_undegraded":"2024-04-16T16:21:28.642377+0000","last_fullsized":"2024-04-16T16:21:28.642377+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:07:06.886135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644754+0000","last_change":"2024-04-16T16:21:26.620146+0000","last_active":"2024-04-16T16:21:28.644754+0000","last_peered":"2024-04-16T16:21:28.644754+0000","last_clean":"2024-04-16T16:21:28.644754+0000","last_became_active":"2024-04-16T16:21:26.619872+0000","last_became_peered":"2024-04-16T16:21:26.619872+0000","last_unstale":"2024-04-16T16:21:28.644754+0000","last_undegraded":"2024-04-16T16:21:28.644754+0000","last_fullsized":"2024-04-16T16:21:28.644754+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:15.375653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.622489+0000","last_change":"2024-04-16T16:21:25.601334+0000","last_active":"2024-04-16T16:21:27.622489+0000","last_peered":"2024-04-16T16:21:27.622489+0000","last_clean":"2024-04-16T16:21:27.622489+0000","last_became_active":"2024-04-16T16:21:25.600462+0000","last_became_peered":"2024-04-16T16:21:25.600462+0000","last_unstale":"2024-04-16T16:21:27.622489+0000","last_undegraded":"2024-04-16T16:21:27.622489+0000","last_fullsized":"2024-04-16T16:21:27.622489+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:54:51.270429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.642187+0000","last_change":"2024-04-16T16:21:26.633261+0000","last_active":"2024-04-16T16:21:28.642187+0000","last_peered":"2024-04-16T16:21:28.642187+0000","last_clean":"2024-04-16T16:21:28.642187+0000","last_became_active":"2024-04-16T16:21:26.632970+0000","last_became_peered":"2024-04-16T16:21:26.632970+0000","last_unstale":"2024-04-16T16:21:28.642187+0000","last_undegraded":"2024-04-16T16:21:28.642187+0000","last_fullsized":"2024-04-16T16:21:28.642187+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:48:48.721677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"141'3","reported_seq":17,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643844+0000","last_change":"2024-04-16T16:21:25.604880+0000","last_active":"2024-04-16T16:21:28.643844+0000","last_peered":"2024-04-16T16:21:28.643844+0000","last_clean":"2024-04-16T16:21:28.643844+0000","last_became_active":"2024-04-16T16:21:25.604027+0000","last_became_peered":"2024-04-16T16:21:25.604027+0000","last_unstale":"2024-04-16T16:21:28.643844+0000","last_undegraded":"2024-04-16T16:21:28.643844+0000","last_fullsized":"2024-04-16T16:21:28.643844+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:01.205017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"130'4669","reported_seq":5284,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626166+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:21:27.626166+0000","last_peered":"2024-04-16T16:21:27.626166+0000","last_clean":"2024-04-16T16:21:27.626166+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:21:27.626166+0000","last_undegraded":"2024-04-16T16:21:27.626166+0000","last_fullsized":"2024-04-16T16:21:27.626166+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1083,"num_read_kb":2113,"num_write":347,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1083,"num_read_kb":2113,"num_write":347,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4669,"ondisk_log_size":4669,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574942,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33512,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338328,"statfs":{"total":96636764160,"available":96602447872,"internally_reserved":0,"allocated":2379776,"data_stored":1436331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574943,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32932,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338908,"statfs":{"total":96636764160,"available":96603041792,"internally_reserved":0,"allocated":1789952,"data_stored":845457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574943,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1781760,"data_stored":845333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574944,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":1777664,"data_stored":845311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574942,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1781760,"data_stored":845401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574942,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37096,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334744,"statfs":{"total":96636764160,"available":96598777856,"internally_reserved":0,"allocated":1794048,"data_stored":846993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1781760,"data_stored":845893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574943,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37672,"kb_used_data":2332,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334168,"statfs":{"total":96636764160,"available":96598188032,"internally_reserved":0,"allocated":2387968,"data_stored":1437325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:21:36.216 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:21:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.455+0000 7f5da1e00640 1 Processor -- start 2024-04-16T16:21:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.455+0000 7f5da1e00640 1 -- start start 2024-04-16T16:21:36.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.457+0000 7f5da1e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d9c0d52d0 0x7f5d9c06ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:36.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.457+0000 7f5da1e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d9c0d3080 con 0x7f5d9c161e90 2024-04-16T16:21:36.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.457+0000 7f5da1e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d9c0d3360 con 0x7f5d9c162270 2024-04-16T16:21:36.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.457+0000 7f5da1e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5d9c0d2e30 con 0x7f5d9c0d52d0 2024-04-16T16:21:36.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.457+0000 7f5da0a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5d9c161e90 0x7f5d9c0cf7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55198/0 (socket says 172.21.15.45:55198) 2024-04-16T16:21:36.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.457+0000 7f5da0a00640 1 -- 172.21.15.45:0/2757653102 learned_addr learned my addr 172.21.15.45:0/2757653102 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.457+0000 7f5da1400640 1 --2- 172.21.15.45:0/2757653102 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d9c0d52d0 0x7f5d9c06ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.457+0000 7f5d9b400640 1 -- 172.21.15.45:0/2757653102 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3048893283 0 0) 0x7f5d9c0d3080 con 0x7f5d9c161e90 2024-04-16T16:21:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.458+0000 7f5d9b400640 1 -- 172.21.15.45:0/2757653102 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d80003520 con 0x7f5d9c161e90 2024-04-16T16:21:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.458+0000 7f5d9b400640 1 -- 172.21.15.45:0/2757653102 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 294680842 0 0) 0x7f5d80003520 con 0x7f5d9c161e90 2024-04-16T16:21:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.458+0000 7f5d9b400640 1 -- 172.21.15.45:0/2757653102 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d9c0d52d0 msgr2=0x7f5d9c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.458+0000 7f5d9b400640 1 --2- 172.21.15.45:0/2757653102 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d9c0d52d0 0x7f5d9c06ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.458+0000 7f5d9b400640 1 -- 172.21.15.45:0/2757653102 >> v1:172.21.15.103:6789/0 conn(0x7f5d9c162270 legacy=0x7f5d9c0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.458+0000 7f5d9b400640 1 -- 172.21.15.45:0/2757653102 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d9c0d3640 con 0x7f5d9c161e90 2024-04-16T16:21:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.458+0000 7f5d9b400640 1 -- 172.21.15.45:0/2757653102 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f5d8c0019f0 con 0x7f5d9c161e90 2024-04-16T16:21:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.458+0000 7f5d9b400640 1 -- 172.21.15.45:0/2757653102 >> v1:172.21.15.45:6789/0 conn(0x7f5d9c161e90 legacy=0x7f5d9c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.458+0000 7f5d9b400640 1 --2- 172.21.15.45:0/2757653102 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d9c0d52d0 0x7f5d800039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.458+0000 7f5d9b400640 1 --2- 172.21.15.45:0/2757653102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d80003f60 0x7f5d80004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.458+0000 7f5d9b400640 1 --2- 172.21.15.45:0/2757653102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80004d40 0x7f5d800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.458+0000 7f5d9b400640 1 -- 172.21.15.45:0/2757653102 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5d9c0d2e30 con 0x7f5d80004d40 2024-04-16T16:21:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.458+0000 7f5d9b400640 1 -- 172.21.15.45:0/2757653102 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8000bb30 con 0x7f5d80003f60 2024-04-16T16:21:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.458+0000 7f5d9b400640 1 -- 172.21.15.45:0/2757653102 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5d8000bd30 con 0x7f5d9c0d52d0 2024-04-16T16:21:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.458+0000 7f5da1400640 1 --2- 172.21.15.45:0/2757653102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80004d40 0x7f5d800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.459+0000 7f5da0a00640 1 --2- 172.21.15.45:0/2757653102 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d9c0d52d0 0x7f5d800039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.459+0000 7f5d9be00640 1 --2- 172.21.15.45:0/2757653102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d80003f60 0x7f5d80004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.459+0000 7f5da1400640 1 -- 172.21.15.45:0/2757653102 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d9c0d52d0 msgr2=0x7f5d800039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.459+0000 7f5da1400640 1 --2- 172.21.15.45:0/2757653102 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d9c0d52d0 0x7f5d800039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:36.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.459+0000 7f5da1400640 1 -- 172.21.15.45:0/2757653102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d80003f60 msgr2=0x7f5d80004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:36.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.459+0000 7f5da1400640 1 --2- 172.21.15.45:0/2757653102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d80003f60 0x7f5d80004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:36.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.459+0000 7f5da1400640 1 -- 172.21.15.45:0/2757653102 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d80003520 con 0x7f5d80004d40 2024-04-16T16:21:36.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.459+0000 7f5d9be00640 1 --2- 172.21.15.45:0/2757653102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d80003f60 0x7f5d80004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.460+0000 7f5da1400640 1 --2- 172.21.15.45:0/2757653102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80004d40 0x7f5d800051c0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009a90 tx=0x7f5d80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814e42326c8108a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.460+0000 7f5d9b400640 1 -- 172.21.15.45:0/2757653102 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d94039070 con 0x7f5d80004d40 2024-04-16T16:21:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.460+0000 7f5d9b400640 1 -- 172.21.15.45:0/2757653102 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d9402c680 con 0x7f5d80004d40 2024-04-16T16:21:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.460+0000 7f5d9b400640 1 -- 172.21.15.45:0/2757653102 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d94009690 con 0x7f5d80004d40 2024-04-16T16:21:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.461+0000 7f5da1e00640 1 -- 172.21.15.45:0/2757653102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80004d40 msgr2=0x7f5d800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.461+0000 7f5da1e00640 1 --2- 172.21.15.45:0/2757653102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80004d40 0x7f5d800051c0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009a90 tx=0x7f5d80005830 comp rx=0 tx=0).stop 2024-04-16T16:21:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.461+0000 7f5da1400640 1 -- 172.21.15.45:0/2757653102 reap_dead start 2024-04-16T16:21:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.461+0000 7f5da1e00640 1 -- 172.21.15.45:0/2757653102 shutdown_connections 2024-04-16T16:21:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.461+0000 7f5da1e00640 1 -- 172.21.15.45:0/2757653102 >> 172.21.15.45:0/2757653102 conn(0x7f5d9c0d1940 msgr2=0x7f5d9c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.461+0000 7f5da1e00640 1 -- 172.21.15.45:0/2757653102 shutdown_connections 2024-04-16T16:21:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.461+0000 7f5da1e00640 1 -- 172.21.15.45:0/2757653102 wait complete. 2024-04-16T16:21:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.463+0000 7f5da1e00640 1 Processor -- start 2024-04-16T16:21:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.463+0000 7f5da1e00640 1 -- start start 2024-04-16T16:21:36.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.464+0000 7f5da1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80004d40 0x7f5d9c150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:36.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.464+0000 7f5da1e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d9c14d8d0 0x7f5d9c14f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:36.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.464+0000 7f5da1e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d9c1f3e20 0x7f5d9c14f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:36.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.464+0000 7f5da1e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5d9c0d46d0 con 0x7f5d80004d40 2024-04-16T16:21:36.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.464+0000 7f5da1e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5d9c0d4550 con 0x7f5d9c14d8d0 2024-04-16T16:21:36.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.464+0000 7f5da1e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5d9c06ce40 con 0x7f5d9c1f3e20 2024-04-16T16:21:36.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.465+0000 7f5da0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80004d40 0x7f5d9c150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:36.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.465+0000 7f5da0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80004d40 0x7f5d9c150500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60102/0 (socket says 172.21.15.45:60102) 2024-04-16T16:21:36.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.465+0000 7f5da0a00640 1 -- 172.21.15.45:0/225938681 learned_addr learned my addr 172.21.15.45:0/225938681 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:36.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.465+0000 7f5da1400640 1 --2- 172.21.15.45:0/225938681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d9c1f3e20 0x7f5d9c14f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:36.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.465+0000 7f5d9be00640 1 --2- 172.21.15.45:0/225938681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d9c14d8d0 0x7f5d9c14f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:36.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.465+0000 7f5da0a00640 1 -- 172.21.15.45:0/225938681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d9c1f3e20 msgr2=0x7f5d9c14f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:36.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.465+0000 7f5da0a00640 1 --2- 172.21.15.45:0/225938681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d9c1f3e20 0x7f5d9c14f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:36.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.465+0000 7f5da0a00640 1 -- 172.21.15.45:0/225938681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d9c14d8d0 msgr2=0x7f5d9c14f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:36.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.465+0000 7f5da0a00640 1 --2- 172.21.15.45:0/225938681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d9c14d8d0 0x7f5d9c14f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.465+0000 7f5da0a00640 1 -- 172.21.15.45:0/225938681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d9c14e0b0 con 0x7f5d80004d40 2024-04-16T16:21:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.465+0000 7f5da0a00640 1 --2- 172.21.15.45:0/225938681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80004d40 0x7f5d9c150500 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c005fd0 tx=0x7f5d8c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.466+0000 7f5d98c00640 1 -- 172.21.15.45:0/225938681 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c013420 con 0x7f5d80004d40 2024-04-16T16:21:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.466+0000 7f5d98c00640 1 -- 172.21.15.45:0/225938681 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d8c018070 con 0x7f5d80004d40 2024-04-16T16:21:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.466+0000 7f5d98c00640 1 -- 172.21.15.45:0/225938681 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c013820 con 0x7f5d80004d40 2024-04-16T16:21:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.466+0000 7f5da1e00640 1 -- 172.21.15.45:0/225938681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d9c14e390 con 0x7f5d80004d40 2024-04-16T16:21:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.466+0000 7f5da1e00640 1 -- 172.21.15.45:0/225938681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d9c14e950 con 0x7f5d80004d40 2024-04-16T16:21:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.467+0000 7f5da1e00640 1 -- 172.21.15.45:0/225938681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d9c075780 con 0x7f5d80004d40 2024-04-16T16:21:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.468+0000 7f5d98c00640 1 -- 172.21.15.45:0/225938681 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f5d8c0139c0 con 0x7f5d80004d40 2024-04-16T16:21:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.469+0000 7f5d98c00640 1 --2- 172.21.15.45:0/225938681 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5d7806ca40 0x7f5d7806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.469+0000 7f5d98c00640 1 -- 172.21.15.45:0/225938681 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7f5d8c08c4d0 con 0x7f5d80004d40 2024-04-16T16:21:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.470+0000 7f5d9be00640 1 --2- 172.21.15.45:0/225938681 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5d7806ca40 0x7f5d7806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:36.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.471+0000 7f5d9be00640 1 --2- 172.21.15.45:0/225938681 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5d7806ca40 0x7f5d7806ef00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5d84007fd0 tx=0x7f5d84009040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:36.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.474+0000 7f5d98c00640 1 -- 172.21.15.45:0/225938681 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d8c05a810 con 0x7f5d80004d40 2024-04-16T16:21:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.673+0000 7f5da1e00640 1 -- 172.21.15.45:0/225938681 --> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5d9c075330 con 0x7f5d7806ca40 2024-04-16T16:21:36.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.694+0000 7f5d98c00640 1 -- 172.21.15.45:0/225938681 <== mgr.7153 v2:172.21.15.103:6832/1478535274 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252390 (secure 0 0 0) 0x7f5d9c075330 con 0x7f5d7806ca40 2024-04-16T16:21:36.696 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:36.698 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:21:36.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.700+0000 7f5da1e00640 1 -- 172.21.15.45:0/225938681 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5d7806ca40 msgr2=0x7f5d7806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:36.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.700+0000 7f5da1e00640 1 --2- 172.21.15.45:0/225938681 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5d7806ca40 0x7f5d7806ef00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5d84007fd0 tx=0x7f5d84009040 comp rx=0 tx=0).stop 2024-04-16T16:21:36.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.700+0000 7f5da1e00640 1 -- 172.21.15.45:0/225938681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80004d40 msgr2=0x7f5d9c150500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:36.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.700+0000 7f5da1e00640 1 --2- 172.21.15.45:0/225938681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80004d40 0x7f5d9c150500 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c005fd0 tx=0x7f5d8c00e530 comp rx=0 tx=0).stop 2024-04-16T16:21:36.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.700+0000 7f5da1e00640 1 -- 172.21.15.45:0/225938681 shutdown_connections 2024-04-16T16:21:36.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.700+0000 7f5da1e00640 1 --2- 172.21.15.45:0/225938681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d9c1f3e20 0x7f5d9c14f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:36.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.701+0000 7f5da1e00640 1 --2- 172.21.15.45:0/225938681 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f5d7806ca40 0x7f5d7806ef00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:36.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.701+0000 7f5da1e00640 1 --2- 172.21.15.45:0/225938681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d9c14d8d0 0x7f5d9c14f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:36.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.701+0000 7f5da1e00640 1 --2- 172.21.15.45:0/225938681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80004d40 0x7f5d9c150500 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:36.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.701+0000 7f5da1e00640 1 -- 172.21.15.45:0/225938681 >> 172.21.15.45:0/225938681 conn(0x7f5d9c0d1940 msgr2=0x7f5d9c0b9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:36.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.701+0000 7f5da1e00640 1 -- 172.21.15.45:0/225938681 shutdown_connections 2024-04-16T16:21:36.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:36.701+0000 7f5da1e00640 1 -- 172.21.15.45:0/225938681 wait complete. 2024-04-16T16:21:36.719 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-04-16T16:21:36.392826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1083,"num_read_kb":2113,"num_write":392,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4708,"ondisk_log_size":4708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":272904,"kb_used_data":15112,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754701816,"statfs":{"total":773094113280,"available":772814659584,"internally_reserved":0,"allocated":15474688,"data_stored":7948044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.799207"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.611281+0000","last_change":"2024-04-16T16:21:26.626458+0000","last_active":"2024-04-16T16:21:28.611281+0000","last_peered":"2024-04-16T16:21:28.611281+0000","last_clean":"2024-04-16T16:21:28.611281+0000","last_became_active":"2024-04-16T16:21:26.626085+0000","last_became_peered":"2024-04-16T16:21:26.626085+0000","last_unstale":"2024-04-16T16:21:28.611281+0000","last_undegraded":"2024-04-16T16:21:28.611281+0000","last_fullsized":"2024-04-16T16:21:28.611281+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:00:53.974790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.623203+0000","last_change":"2024-04-16T16:21:26.618211+0000","last_active":"2024-04-16T16:21:27.623203+0000","last_peered":"2024-04-16T16:21:27.623203+0000","last_clean":"2024-04-16T16:21:27.623203+0000","last_became_active":"2024-04-16T16:21:26.617930+0000","last_became_peered":"2024-04-16T16:21:26.617930+0000","last_unstale":"2024-04-16T16:21:27.623203+0000","last_undegraded":"2024-04-16T16:21:27.623203+0000","last_fullsized":"2024-04-16T16:21:27.623203+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:16:09.049767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643996+0000","last_change":"2024-04-16T16:21:26.635568+0000","last_active":"2024-04-16T16:21:28.643996+0000","last_peered":"2024-04-16T16:21:28.643996+0000","last_clean":"2024-04-16T16:21:28.643996+0000","last_became_active":"2024-04-16T16:21:26.635193+0000","last_became_peered":"2024-04-16T16:21:26.635193+0000","last_unstale":"2024-04-16T16:21:28.643996+0000","last_undegraded":"2024-04-16T16:21:28.643996+0000","last_fullsized":"2024-04-16T16:21:28.643996+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:49:46.979105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626780+0000","last_change":"2024-04-16T16:21:26.636857+0000","last_active":"2024-04-16T16:21:27.626780+0000","last_peered":"2024-04-16T16:21:27.626780+0000","last_clean":"2024-04-16T16:21:27.626780+0000","last_became_active":"2024-04-16T16:21:26.636573+0000","last_became_peered":"2024-04-16T16:21:26.636573+0000","last_unstale":"2024-04-16T16:21:27.626780+0000","last_undegraded":"2024-04-16T16:21:27.626780+0000","last_fullsized":"2024-04-16T16:21:27.626780+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:55.067333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644598+0000","last_change":"2024-04-16T16:21:26.622281+0000","last_active":"2024-04-16T16:21:28.644598+0000","last_peered":"2024-04-16T16:21:28.644598+0000","last_clean":"2024-04-16T16:21:28.644598+0000","last_became_active":"2024-04-16T16:21:26.621705+0000","last_became_peered":"2024-04-16T16:21:26.621705+0000","last_unstale":"2024-04-16T16:21:28.644598+0000","last_undegraded":"2024-04-16T16:21:28.644598+0000","last_fullsized":"2024-04-16T16:21:28.644598+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:36.349501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610344+0000","last_change":"2024-04-16T16:21:26.634720+0000","last_active":"2024-04-16T16:21:28.610344+0000","last_peered":"2024-04-16T16:21:28.610344+0000","last_clean":"2024-04-16T16:21:28.610344+0000","last_became_active":"2024-04-16T16:21:26.634339+0000","last_became_peered":"2024-04-16T16:21:26.634339+0000","last_unstale":"2024-04-16T16:21:28.610344+0000","last_undegraded":"2024-04-16T16:21:28.610344+0000","last_fullsized":"2024-04-16T16:21:28.610344+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:20:31.043392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.641429+0000","last_change":"2024-04-16T16:21:26.638223+0000","last_active":"2024-04-16T16:21:28.641429+0000","last_peered":"2024-04-16T16:21:28.641429+0000","last_clean":"2024-04-16T16:21:28.641429+0000","last_became_active":"2024-04-16T16:21:26.637894+0000","last_became_peered":"2024-04-16T16:21:26.637894+0000","last_unstale":"2024-04-16T16:21:28.641429+0000","last_undegraded":"2024-04-16T16:21:28.641429+0000","last_fullsized":"2024-04-16T16:21:28.641429+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:06.714681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.609807+0000","last_change":"2024-04-16T16:21:26.622007+0000","last_active":"2024-04-16T16:21:28.609807+0000","last_peered":"2024-04-16T16:21:28.609807+0000","last_clean":"2024-04-16T16:21:28.609807+0000","last_became_active":"2024-04-16T16:21:26.621724+0000","last_became_peered":"2024-04-16T16:21:26.621724+0000","last_unstale":"2024-04-16T16:21:28.609807+0000","last_undegraded":"2024-04-16T16:21:28.609807+0000","last_fullsized":"2024-04-16T16:21:28.609807+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:20.296700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.623030+0000","last_change":"2024-04-16T16:21:26.621820+0000","last_active":"2024-04-16T16:21:27.623030+0000","last_peered":"2024-04-16T16:21:27.623030+0000","last_clean":"2024-04-16T16:21:27.623030+0000","last_became_active":"2024-04-16T16:21:26.621472+0000","last_became_peered":"2024-04-16T16:21:26.621472+0000","last_unstale":"2024-04-16T16:21:27.623030+0000","last_undegraded":"2024-04-16T16:21:27.623030+0000","last_fullsized":"2024-04-16T16:21:27.623030+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:42.748859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628125+0000","last_change":"2024-04-16T16:21:26.629761+0000","last_active":"2024-04-16T16:21:27.628125+0000","last_peered":"2024-04-16T16:21:27.628125+0000","last_clean":"2024-04-16T16:21:27.628125+0000","last_became_active":"2024-04-16T16:21:26.629483+0000","last_became_peered":"2024-04-16T16:21:26.629483+0000","last_unstale":"2024-04-16T16:21:27.628125+0000","last_undegraded":"2024-04-16T16:21:27.628125+0000","last_fullsized":"2024-04-16T16:21:27.628125+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:41:45.291950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610211+0000","last_change":"2024-04-16T16:21:26.624559+0000","last_active":"2024-04-16T16:21:28.610211+0000","last_peered":"2024-04-16T16:21:28.610211+0000","last_clean":"2024-04-16T16:21:28.610211+0000","last_became_active":"2024-04-16T16:21:26.622580+0000","last_became_peered":"2024-04-16T16:21:26.622580+0000","last_unstale":"2024-04-16T16:21:28.610211+0000","last_undegraded":"2024-04-16T16:21:28.610211+0000","last_fullsized":"2024-04-16T16:21:28.610211+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:04:09.629772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627808+0000","last_change":"2024-04-16T16:21:26.623657+0000","last_active":"2024-04-16T16:21:27.627808+0000","last_peered":"2024-04-16T16:21:27.627808+0000","last_clean":"2024-04-16T16:21:27.627808+0000","last_became_active":"2024-04-16T16:21:26.623238+0000","last_became_peered":"2024-04-16T16:21:26.623238+0000","last_unstale":"2024-04-16T16:21:27.627808+0000","last_undegraded":"2024-04-16T16:21:27.627808+0000","last_fullsized":"2024-04-16T16:21:27.627808+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:16:09.049767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644691+0000","last_change":"2024-04-16T16:21:26.621932+0000","last_active":"2024-04-16T16:21:28.644691+0000","last_peered":"2024-04-16T16:21:28.644691+0000","last_clean":"2024-04-16T16:21:28.644691+0000","last_became_active":"2024-04-16T16:21:26.621590+0000","last_became_peered":"2024-04-16T16:21:26.621590+0000","last_unstale":"2024-04-16T16:21:28.644691+0000","last_undegraded":"2024-04-16T16:21:28.644691+0000","last_fullsized":"2024-04-16T16:21:28.644691+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:22:41.361055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626204+0000","last_change":"2024-04-16T16:21:26.626405+0000","last_active":"2024-04-16T16:21:27.626204+0000","last_peered":"2024-04-16T16:21:27.626204+0000","last_clean":"2024-04-16T16:21:27.626204+0000","last_became_active":"2024-04-16T16:21:26.625738+0000","last_became_peered":"2024-04-16T16:21:26.625738+0000","last_unstale":"2024-04-16T16:21:27.626204+0000","last_undegraded":"2024-04-16T16:21:27.626204+0000","last_fullsized":"2024-04-16T16:21:27.626204+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:35:45.491987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.625529+0000","last_change":"2024-04-16T16:21:26.618881+0000","last_active":"2024-04-16T16:21:27.625529+0000","last_peered":"2024-04-16T16:21:27.625529+0000","last_clean":"2024-04-16T16:21:27.625529+0000","last_became_active":"2024-04-16T16:21:26.618499+0000","last_became_peered":"2024-04-16T16:21:26.618499+0000","last_unstale":"2024-04-16T16:21:27.625529+0000","last_undegraded":"2024-04-16T16:21:27.625529+0000","last_fullsized":"2024-04-16T16:21:27.625529+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:28.393986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643268+0000","last_change":"2024-04-16T16:21:26.619336+0000","last_active":"2024-04-16T16:21:28.643268+0000","last_peered":"2024-04-16T16:21:28.643268+0000","last_clean":"2024-04-16T16:21:28.643268+0000","last_became_active":"2024-04-16T16:21:26.618973+0000","last_became_peered":"2024-04-16T16:21:26.618973+0000","last_unstale":"2024-04-16T16:21:28.643268+0000","last_undegraded":"2024-04-16T16:21:28.643268+0000","last_fullsized":"2024-04-16T16:21:28.643268+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:56.732655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628517+0000","last_change":"2024-04-16T16:21:26.621609+0000","last_active":"2024-04-16T16:21:27.628517+0000","last_peered":"2024-04-16T16:21:27.628517+0000","last_clean":"2024-04-16T16:21:27.628517+0000","last_became_active":"2024-04-16T16:21:26.621328+0000","last_became_peered":"2024-04-16T16:21:26.621328+0000","last_unstale":"2024-04-16T16:21:27.628517+0000","last_undegraded":"2024-04-16T16:21:27.628517+0000","last_fullsized":"2024-04-16T16:21:27.628517+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:16.919182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644161+0000","last_change":"2024-04-16T16:21:26.628792+0000","last_active":"2024-04-16T16:21:28.644161+0000","last_peered":"2024-04-16T16:21:28.644161+0000","last_clean":"2024-04-16T16:21:28.644161+0000","last_became_active":"2024-04-16T16:21:26.628437+0000","last_became_peered":"2024-04-16T16:21:26.628437+0000","last_unstale":"2024-04-16T16:21:28.644161+0000","last_undegraded":"2024-04-16T16:21:28.644161+0000","last_fullsized":"2024-04-16T16:21:28.644161+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:38:03.914829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610959+0000","last_change":"2024-04-16T16:21:26.627834+0000","last_active":"2024-04-16T16:21:28.610959+0000","last_peered":"2024-04-16T16:21:28.610959+0000","last_clean":"2024-04-16T16:21:28.610959+0000","last_became_active":"2024-04-16T16:21:26.626611+0000","last_became_peered":"2024-04-16T16:21:26.626611+0000","last_unstale":"2024-04-16T16:21:28.610959+0000","last_undegraded":"2024-04-16T16:21:28.610959+0000","last_fullsized":"2024-04-16T16:21:28.610959+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:38.654900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643963+0000","last_change":"2024-04-16T16:21:26.618141+0000","last_active":"2024-04-16T16:21:28.643963+0000","last_peered":"2024-04-16T16:21:28.643963+0000","last_clean":"2024-04-16T16:21:28.643963+0000","last_became_active":"2024-04-16T16:21:26.614289+0000","last_became_peered":"2024-04-16T16:21:26.614289+0000","last_unstale":"2024-04-16T16:21:28.643963+0000","last_undegraded":"2024-04-16T16:21:28.643963+0000","last_fullsized":"2024-04-16T16:21:28.643963+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:15:22.005025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627713+0000","last_change":"2024-04-16T16:21:26.633948+0000","last_active":"2024-04-16T16:21:27.627713+0000","last_peered":"2024-04-16T16:21:27.627713+0000","last_clean":"2024-04-16T16:21:27.627713+0000","last_became_active":"2024-04-16T16:21:26.631959+0000","last_became_peered":"2024-04-16T16:21:26.631959+0000","last_unstale":"2024-04-16T16:21:27.627713+0000","last_undegraded":"2024-04-16T16:21:27.627713+0000","last_fullsized":"2024-04-16T16:21:27.627713+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:42.748859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626655+0000","last_change":"2024-04-16T16:21:26.626656+0000","last_active":"2024-04-16T16:21:27.626655+0000","last_peered":"2024-04-16T16:21:27.626655+0000","last_clean":"2024-04-16T16:21:27.626655+0000","last_became_active":"2024-04-16T16:21:26.626102+0000","last_became_peered":"2024-04-16T16:21:26.626102+0000","last_unstale":"2024-04-16T16:21:27.626655+0000","last_undegraded":"2024-04-16T16:21:27.626655+0000","last_fullsized":"2024-04-16T16:21:27.626655+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:21:28.099485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.625579+0000","last_change":"2024-04-16T16:21:26.640121+0000","last_active":"2024-04-16T16:21:27.625579+0000","last_peered":"2024-04-16T16:21:27.625579+0000","last_clean":"2024-04-16T16:21:27.625579+0000","last_became_active":"2024-04-16T16:21:26.639851+0000","last_became_peered":"2024-04-16T16:21:26.639851+0000","last_unstale":"2024-04-16T16:21:27.625579+0000","last_undegraded":"2024-04-16T16:21:27.625579+0000","last_fullsized":"2024-04-16T16:21:27.625579+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:51:55.853674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.625755+0000","last_change":"2024-04-16T16:21:26.634952+0000","last_active":"2024-04-16T16:21:27.625755+0000","last_peered":"2024-04-16T16:21:27.625755+0000","last_clean":"2024-04-16T16:21:27.625755+0000","last_became_active":"2024-04-16T16:21:26.634627+0000","last_became_peered":"2024-04-16T16:21:26.634627+0000","last_unstale":"2024-04-16T16:21:27.625755+0000","last_undegraded":"2024-04-16T16:21:27.625755+0000","last_fullsized":"2024-04-16T16:21:27.625755+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:38.752487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643682+0000","last_change":"2024-04-16T16:21:26.623659+0000","last_active":"2024-04-16T16:21:28.643682+0000","last_peered":"2024-04-16T16:21:28.643682+0000","last_clean":"2024-04-16T16:21:28.643682+0000","last_became_active":"2024-04-16T16:21:26.623344+0000","last_became_peered":"2024-04-16T16:21:26.623344+0000","last_unstale":"2024-04-16T16:21:28.643682+0000","last_undegraded":"2024-04-16T16:21:28.643682+0000","last_fullsized":"2024-04-16T16:21:28.643682+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:36:11.193195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610902+0000","last_change":"2024-04-16T16:21:26.625781+0000","last_active":"2024-04-16T16:21:28.610902+0000","last_peered":"2024-04-16T16:21:28.610902+0000","last_clean":"2024-04-16T16:21:28.610902+0000","last_became_active":"2024-04-16T16:21:26.625477+0000","last_became_peered":"2024-04-16T16:21:26.625477+0000","last_unstale":"2024-04-16T16:21:28.610902+0000","last_undegraded":"2024-04-16T16:21:28.610902+0000","last_fullsized":"2024-04-16T16:21:28.610902+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:02.219653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643714+0000","last_change":"2024-04-16T16:21:26.636365+0000","last_active":"2024-04-16T16:21:28.643714+0000","last_peered":"2024-04-16T16:21:28.643714+0000","last_clean":"2024-04-16T16:21:28.643714+0000","last_became_active":"2024-04-16T16:21:26.636019+0000","last_became_peered":"2024-04-16T16:21:26.636019+0000","last_unstale":"2024-04-16T16:21:28.643714+0000","last_undegraded":"2024-04-16T16:21:28.643714+0000","last_fullsized":"2024-04-16T16:21:28.643714+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:12.380057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.641995+0000","last_change":"2024-04-16T16:21:26.639338+0000","last_active":"2024-04-16T16:21:28.641995+0000","last_peered":"2024-04-16T16:21:28.641995+0000","last_clean":"2024-04-16T16:21:28.641995+0000","last_became_active":"2024-04-16T16:21:26.639071+0000","last_became_peered":"2024-04-16T16:21:26.639071+0000","last_unstale":"2024-04-16T16:21:28.641995+0000","last_undegraded":"2024-04-16T16:21:28.641995+0000","last_fullsized":"2024-04-16T16:21:28.641995+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:35.235737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610040+0000","last_change":"2024-04-16T16:21:26.638327+0000","last_active":"2024-04-16T16:21:28.610040+0000","last_peered":"2024-04-16T16:21:28.610040+0000","last_clean":"2024-04-16T16:21:28.610040+0000","last_became_active":"2024-04-16T16:21:26.638007+0000","last_became_peered":"2024-04-16T16:21:26.638007+0000","last_unstale":"2024-04-16T16:21:28.610040+0000","last_undegraded":"2024-04-16T16:21:28.610040+0000","last_fullsized":"2024-04-16T16:21:28.610040+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:37:16.481358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627008+0000","last_change":"2024-04-16T16:21:26.631148+0000","last_active":"2024-04-16T16:21:27.627008+0000","last_peered":"2024-04-16T16:21:27.627008+0000","last_clean":"2024-04-16T16:21:27.627008+0000","last_became_active":"2024-04-16T16:21:26.629486+0000","last_became_peered":"2024-04-16T16:21:26.629486+0000","last_unstale":"2024-04-16T16:21:27.627008+0000","last_undegraded":"2024-04-16T16:21:27.627008+0000","last_fullsized":"2024-04-16T16:21:27.627008+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:49.214360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.622409+0000","last_change":"2024-04-16T16:21:26.633930+0000","last_active":"2024-04-16T16:21:27.622409+0000","last_peered":"2024-04-16T16:21:27.622409+0000","last_clean":"2024-04-16T16:21:27.622409+0000","last_became_active":"2024-04-16T16:21:26.633235+0000","last_became_peered":"2024-04-16T16:21:26.633235+0000","last_unstale":"2024-04-16T16:21:27.622409+0000","last_undegraded":"2024-04-16T16:21:27.622409+0000","last_fullsized":"2024-04-16T16:21:27.622409+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:06:10.334071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643829+0000","last_change":"2024-04-16T16:21:26.622683+0000","last_active":"2024-04-16T16:21:28.643829+0000","last_peered":"2024-04-16T16:21:28.643829+0000","last_clean":"2024-04-16T16:21:28.643829+0000","last_became_active":"2024-04-16T16:21:26.622308+0000","last_became_peered":"2024-04-16T16:21:26.622308+0000","last_unstale":"2024-04-16T16:21:28.643829+0000","last_undegraded":"2024-04-16T16:21:28.643829+0000","last_fullsized":"2024-04-16T16:21:28.643829+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:26.812700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628420+0000","last_change":"2024-04-16T16:21:26.624637+0000","last_active":"2024-04-16T16:21:27.628420+0000","last_peered":"2024-04-16T16:21:27.628420+0000","last_clean":"2024-04-16T16:21:27.628420+0000","last_became_active":"2024-04-16T16:21:26.623808+0000","last_became_peered":"2024-04-16T16:21:26.623808+0000","last_unstale":"2024-04-16T16:21:27.628420+0000","last_undegraded":"2024-04-16T16:21:27.628420+0000","last_fullsized":"2024-04-16T16:21:27.628420+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:10:06.714681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"141'4","reported_seq":18,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628451+0000","last_change":"2024-04-16T16:21:25.597818+0000","last_active":"2024-04-16T16:21:27.628451+0000","last_peered":"2024-04-16T16:21:27.628451+0000","last_clean":"2024-04-16T16:21:27.628451+0000","last_became_active":"2024-04-16T16:21:25.597506+0000","last_became_peered":"2024-04-16T16:21:25.597506+0000","last_unstale":"2024-04-16T16:21:27.628451+0000","last_undegraded":"2024-04-16T16:21:27.628451+0000","last_fullsized":"2024-04-16T16:21:27.628451+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:29:15.239160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.623236+0000","last_change":"2024-04-16T16:21:26.619178+0000","last_active":"2024-04-16T16:21:27.623236+0000","last_peered":"2024-04-16T16:21:27.623236+0000","last_clean":"2024-04-16T16:21:27.623236+0000","last_became_active":"2024-04-16T16:21:26.618887+0000","last_became_peered":"2024-04-16T16:21:26.618887+0000","last_unstale":"2024-04-16T16:21:27.623236+0000","last_undegraded":"2024-04-16T16:21:27.623236+0000","last_fullsized":"2024-04-16T16:21:27.623236+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:07:10.543796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.623327+0000","last_change":"2024-04-16T16:21:25.603465+0000","last_active":"2024-04-16T16:21:27.623327+0000","last_peered":"2024-04-16T16:21:27.623327+0000","last_clean":"2024-04-16T16:21:27.623327+0000","last_became_active":"2024-04-16T16:21:25.602980+0000","last_became_peered":"2024-04-16T16:21:25.602980+0000","last_unstale":"2024-04-16T16:21:27.623327+0000","last_undegraded":"2024-04-16T16:21:27.623327+0000","last_fullsized":"2024-04-16T16:21:27.623327+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:39.706251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610226+0000","last_change":"2024-04-16T16:21:26.635213+0000","last_active":"2024-04-16T16:21:28.610226+0000","last_peered":"2024-04-16T16:21:28.610226+0000","last_clean":"2024-04-16T16:21:28.610226+0000","last_became_active":"2024-04-16T16:21:26.634409+0000","last_became_peered":"2024-04-16T16:21:26.634409+0000","last_unstale":"2024-04-16T16:21:28.610226+0000","last_undegraded":"2024-04-16T16:21:28.610226+0000","last_fullsized":"2024-04-16T16:21:28.610226+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:03:26.310046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627258+0000","last_change":"2024-04-16T16:21:25.596356+0000","last_active":"2024-04-16T16:21:27.627258+0000","last_peered":"2024-04-16T16:21:27.627258+0000","last_clean":"2024-04-16T16:21:27.627258+0000","last_became_active":"2024-04-16T16:21:25.594551+0000","last_became_peered":"2024-04-16T16:21:25.594551+0000","last_unstale":"2024-04-16T16:21:27.627258+0000","last_undegraded":"2024-04-16T16:21:27.627258+0000","last_fullsized":"2024-04-16T16:21:27.627258+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:20:30.028756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627848+0000","last_change":"2024-04-16T16:21:26.623606+0000","last_active":"2024-04-16T16:21:27.627848+0000","last_peered":"2024-04-16T16:21:27.627848+0000","last_clean":"2024-04-16T16:21:27.627848+0000","last_became_active":"2024-04-16T16:21:26.623161+0000","last_became_peered":"2024-04-16T16:21:26.623161+0000","last_unstale":"2024-04-16T16:21:27.627848+0000","last_undegraded":"2024-04-16T16:21:27.627848+0000","last_fullsized":"2024-04-16T16:21:27.627848+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:07:10.543796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626876+0000","last_change":"2024-04-16T16:21:25.592173+0000","last_active":"2024-04-16T16:21:27.626876+0000","last_peered":"2024-04-16T16:21:27.626876+0000","last_clean":"2024-04-16T16:21:27.626876+0000","last_became_active":"2024-04-16T16:21:25.591898+0000","last_became_peered":"2024-04-16T16:21:25.591898+0000","last_unstale":"2024-04-16T16:21:27.626876+0000","last_undegraded":"2024-04-16T16:21:27.626876+0000","last_fullsized":"2024-04-16T16:21:27.626876+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:03:25.295410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.622502+0000","last_change":"2024-04-16T16:21:26.633513+0000","last_active":"2024-04-16T16:21:27.622502+0000","last_peered":"2024-04-16T16:21:27.622502+0000","last_clean":"2024-04-16T16:21:27.622502+0000","last_became_active":"2024-04-16T16:21:26.633154+0000","last_became_peered":"2024-04-16T16:21:26.633154+0000","last_unstale":"2024-04-16T16:21:27.622502+0000","last_undegraded":"2024-04-16T16:21:27.622502+0000","last_fullsized":"2024-04-16T16:21:27.622502+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:27.121241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644284+0000","last_change":"2024-04-16T16:21:25.588337+0000","last_active":"2024-04-16T16:21:28.644284+0000","last_peered":"2024-04-16T16:21:28.644284+0000","last_clean":"2024-04-16T16:21:28.644284+0000","last_became_active":"2024-04-16T16:21:25.583483+0000","last_became_peered":"2024-04-16T16:21:25.583483+0000","last_unstale":"2024-04-16T16:21:28.644284+0000","last_undegraded":"2024-04-16T16:21:28.644284+0000","last_fullsized":"2024-04-16T16:21:28.644284+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:10:51.761699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627525+0000","last_change":"2024-04-16T16:21:26.638002+0000","last_active":"2024-04-16T16:21:27.627525+0000","last_peered":"2024-04-16T16:21:27.627525+0000","last_clean":"2024-04-16T16:21:27.627525+0000","last_became_active":"2024-04-16T16:21:26.637712+0000","last_became_peered":"2024-04-16T16:21:26.637712+0000","last_unstale":"2024-04-16T16:21:27.627525+0000","last_undegraded":"2024-04-16T16:21:27.627525+0000","last_fullsized":"2024-04-16T16:21:27.627525+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:38:03.914829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644045+0000","last_change":"2024-04-16T16:21:25.600581+0000","last_active":"2024-04-16T16:21:28.644045+0000","last_peered":"2024-04-16T16:21:28.644045+0000","last_clean":"2024-04-16T16:21:28.644045+0000","last_became_active":"2024-04-16T16:21:25.600245+0000","last_became_peered":"2024-04-16T16:21:25.600245+0000","last_unstale":"2024-04-16T16:21:28.644045+0000","last_undegraded":"2024-04-16T16:21:28.644045+0000","last_fullsized":"2024-04-16T16:21:28.644045+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:49:43.421377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.622668+0000","last_change":"2024-04-16T16:21:26.620432+0000","last_active":"2024-04-16T16:21:27.622668+0000","last_peered":"2024-04-16T16:21:27.622668+0000","last_clean":"2024-04-16T16:21:27.622668+0000","last_became_active":"2024-04-16T16:21:26.620036+0000","last_became_peered":"2024-04-16T16:21:26.620036+0000","last_unstale":"2024-04-16T16:21:27.622668+0000","last_undegraded":"2024-04-16T16:21:27.622668+0000","last_fullsized":"2024-04-16T16:21:27.622668+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:28.393986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643197+0000","last_change":"2024-04-16T16:21:25.595882+0000","last_active":"2024-04-16T16:21:28.643197+0000","last_peered":"2024-04-16T16:21:28.643197+0000","last_clean":"2024-04-16T16:21:28.643197+0000","last_became_active":"2024-04-16T16:21:25.595359+0000","last_became_peered":"2024-04-16T16:21:25.595359+0000","last_unstale":"2024-04-16T16:21:28.643197+0000","last_undegraded":"2024-04-16T16:21:28.643197+0000","last_fullsized":"2024-04-16T16:21:28.643197+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:53.324244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.645266+0000","last_change":"2024-04-16T16:21:26.622580+0000","last_active":"2024-04-16T16:21:28.645266+0000","last_peered":"2024-04-16T16:21:28.645266+0000","last_clean":"2024-04-16T16:21:28.645266+0000","last_became_active":"2024-04-16T16:21:26.621727+0000","last_became_peered":"2024-04-16T16:21:26.621727+0000","last_unstale":"2024-04-16T16:21:28.645266+0000","last_undegraded":"2024-04-16T16:21:28.645266+0000","last_fullsized":"2024-04-16T16:21:28.645266+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:04:09.629772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.625268+0000","last_change":"2024-04-16T16:21:25.598600+0000","last_active":"2024-04-16T16:21:27.625268+0000","last_peered":"2024-04-16T16:21:27.625268+0000","last_clean":"2024-04-16T16:21:27.625268+0000","last_became_active":"2024-04-16T16:21:25.598251+0000","last_became_peered":"2024-04-16T16:21:25.598251+0000","last_unstale":"2024-04-16T16:21:27.625268+0000","last_undegraded":"2024-04-16T16:21:27.625268+0000","last_fullsized":"2024-04-16T16:21:27.625268+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:56:21.905428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644889+0000","last_change":"2024-04-16T16:21:26.626469+0000","last_active":"2024-04-16T16:21:28.644889+0000","last_peered":"2024-04-16T16:21:28.644889+0000","last_clean":"2024-04-16T16:21:28.644889+0000","last_became_active":"2024-04-16T16:21:26.625974+0000","last_became_peered":"2024-04-16T16:21:26.625974+0000","last_unstale":"2024-04-16T16:21:28.644889+0000","last_undegraded":"2024-04-16T16:21:28.644889+0000","last_fullsized":"2024-04-16T16:21:28.644889+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:56:22.920064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.623204+0000","last_change":"2024-04-16T16:21:25.592714+0000","last_active":"2024-04-16T16:21:27.623204+0000","last_peered":"2024-04-16T16:21:27.623204+0000","last_clean":"2024-04-16T16:21:27.623204+0000","last_became_active":"2024-04-16T16:21:25.592418+0000","last_became_peered":"2024-04-16T16:21:25.592418+0000","last_unstale":"2024-04-16T16:21:27.623204+0000","last_undegraded":"2024-04-16T16:21:27.623204+0000","last_fullsized":"2024-04-16T16:21:27.623204+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:29:50.499359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628355+0000","last_change":"2024-04-16T16:21:26.619071+0000","last_active":"2024-04-16T16:21:27.628355+0000","last_peered":"2024-04-16T16:21:27.628355+0000","last_clean":"2024-04-16T16:21:27.628355+0000","last_became_active":"2024-04-16T16:21:26.618660+0000","last_became_peered":"2024-04-16T16:21:26.618660+0000","last_unstale":"2024-04-16T16:21:27.628355+0000","last_undegraded":"2024-04-16T16:21:27.628355+0000","last_fullsized":"2024-04-16T16:21:27.628355+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:02:34.201782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628390+0000","last_change":"2024-04-16T16:21:25.601618+0000","last_active":"2024-04-16T16:21:27.628390+0000","last_peered":"2024-04-16T16:21:27.628390+0000","last_clean":"2024-04-16T16:21:27.628390+0000","last_became_active":"2024-04-16T16:21:25.600831+0000","last_became_peered":"2024-04-16T16:21:25.600831+0000","last_unstale":"2024-04-16T16:21:27.628390+0000","last_undegraded":"2024-04-16T16:21:27.628390+0000","last_fullsized":"2024-04-16T16:21:27.628390+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:29:50.499359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644198+0000","last_change":"2024-04-16T16:21:26.618860+0000","last_active":"2024-04-16T16:21:28.644198+0000","last_peered":"2024-04-16T16:21:28.644198+0000","last_clean":"2024-04-16T16:21:28.644198+0000","last_became_active":"2024-04-16T16:21:26.618520+0000","last_became_peered":"2024-04-16T16:21:26.618520+0000","last_unstale":"2024-04-16T16:21:28.644198+0000","last_undegraded":"2024-04-16T16:21:28.644198+0000","last_fullsized":"2024-04-16T16:21:28.644198+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:13.744613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"141'6","reported_seq":20,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.642299+0000","last_change":"2024-04-16T16:21:25.595231+0000","last_active":"2024-04-16T16:21:28.642299+0000","last_peered":"2024-04-16T16:21:28.642299+0000","last_clean":"2024-04-16T16:21:28.642299+0000","last_became_active":"2024-04-16T16:21:25.594379+0000","last_became_peered":"2024-04-16T16:21:25.594379+0000","last_unstale":"2024-04-16T16:21:28.642299+0000","last_undegraded":"2024-04-16T16:21:28.642299+0000","last_fullsized":"2024-04-16T16:21:28.642299+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:52:49.593567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.642913+0000","last_change":"2024-04-16T16:21:26.632827+0000","last_active":"2024-04-16T16:21:28.642913+0000","last_peered":"2024-04-16T16:21:28.642913+0000","last_clean":"2024-04-16T16:21:28.642913+0000","last_became_active":"2024-04-16T16:21:26.632152+0000","last_became_peered":"2024-04-16T16:21:26.632152+0000","last_unstale":"2024-04-16T16:21:28.642913+0000","last_undegraded":"2024-04-16T16:21:28.642913+0000","last_fullsized":"2024-04-16T16:21:28.642913+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:55.067333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.642992+0000","last_change":"2024-04-16T16:21:25.604040+0000","last_active":"2024-04-16T16:21:28.642992+0000","last_peered":"2024-04-16T16:21:28.642992+0000","last_clean":"2024-04-16T16:21:28.642992+0000","last_became_active":"2024-04-16T16:21:25.603726+0000","last_became_peered":"2024-04-16T16:21:25.603726+0000","last_unstale":"2024-04-16T16:21:28.642992+0000","last_undegraded":"2024-04-16T16:21:28.642992+0000","last_fullsized":"2024-04-16T16:21:28.642992+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:20:30.028756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.622865+0000","last_change":"2024-04-16T16:21:26.618805+0000","last_active":"2024-04-16T16:21:27.622865+0000","last_peered":"2024-04-16T16:21:27.622865+0000","last_clean":"2024-04-16T16:21:27.622865+0000","last_became_active":"2024-04-16T16:21:26.618424+0000","last_became_peered":"2024-04-16T16:21:26.618424+0000","last_unstale":"2024-04-16T16:21:27.622865+0000","last_undegraded":"2024-04-16T16:21:27.622865+0000","last_fullsized":"2024-04-16T16:21:27.622865+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:38.752487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643213+0000","last_change":"2024-04-16T16:21:25.603576+0000","last_active":"2024-04-16T16:21:28.643213+0000","last_peered":"2024-04-16T16:21:28.643213+0000","last_clean":"2024-04-16T16:21:28.643213+0000","last_became_active":"2024-04-16T16:21:25.602512+0000","last_became_peered":"2024-04-16T16:21:25.602512+0000","last_unstale":"2024-04-16T16:21:28.643213+0000","last_undegraded":"2024-04-16T16:21:28.643213+0000","last_fullsized":"2024-04-16T16:21:28.643213+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:46:07.323352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643715+0000","last_change":"2024-04-16T16:21:26.619141+0000","last_active":"2024-04-16T16:21:28.643715+0000","last_peered":"2024-04-16T16:21:28.643715+0000","last_clean":"2024-04-16T16:21:28.643715+0000","last_became_active":"2024-04-16T16:21:26.618765+0000","last_became_peered":"2024-04-16T16:21:26.618765+0000","last_unstale":"2024-04-16T16:21:28.643715+0000","last_undegraded":"2024-04-16T16:21:28.643715+0000","last_fullsized":"2024-04-16T16:21:28.643715+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:05:37.498029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.609518+0000","last_change":"2024-04-16T16:21:25.593260+0000","last_active":"2024-04-16T16:21:28.609518+0000","last_peered":"2024-04-16T16:21:28.609518+0000","last_clean":"2024-04-16T16:21:28.609518+0000","last_became_active":"2024-04-16T16:21:25.592707+0000","last_became_peered":"2024-04-16T16:21:25.592707+0000","last_unstale":"2024-04-16T16:21:28.609518+0000","last_undegraded":"2024-04-16T16:21:28.609518+0000","last_fullsized":"2024-04-16T16:21:28.609518+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:49:43.421377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.642498+0000","last_change":"2024-04-16T16:21:26.628634+0000","last_active":"2024-04-16T16:21:28.642498+0000","last_peered":"2024-04-16T16:21:28.642498+0000","last_clean":"2024-04-16T16:21:28.642498+0000","last_became_active":"2024-04-16T16:21:26.628341+0000","last_became_peered":"2024-04-16T16:21:26.628341+0000","last_unstale":"2024-04-16T16:21:28.642498+0000","last_undegraded":"2024-04-16T16:21:28.642498+0000","last_fullsized":"2024-04-16T16:21:28.642498+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:21:28.099485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"141'8","reported_seq":22,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643881+0000","last_change":"2024-04-16T16:21:25.600083+0000","last_active":"2024-04-16T16:21:28.643881+0000","last_peered":"2024-04-16T16:21:28.643881+0000","last_clean":"2024-04-16T16:21:28.643881+0000","last_became_active":"2024-04-16T16:21:25.599688+0000","last_became_peered":"2024-04-16T16:21:25.599688+0000","last_unstale":"2024-04-16T16:21:28.643881+0000","last_undegraded":"2024-04-16T16:21:28.643881+0000","last_fullsized":"2024-04-16T16:21:28.643881+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:48:31.123443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628305+0000","last_change":"2024-04-16T16:21:26.619132+0000","last_active":"2024-04-16T16:21:27.628305+0000","last_peered":"2024-04-16T16:21:27.628305+0000","last_clean":"2024-04-16T16:21:27.628305+0000","last_became_active":"2024-04-16T16:21:26.618806+0000","last_became_peered":"2024-04-16T16:21:26.618806+0000","last_unstale":"2024-04-16T16:21:27.628305+0000","last_undegraded":"2024-04-16T16:21:27.628305+0000","last_fullsized":"2024-04-16T16:21:27.628305+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:51:38.560726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.645024+0000","last_change":"2024-04-16T16:21:25.604548+0000","last_active":"2024-04-16T16:21:28.645024+0000","last_peered":"2024-04-16T16:21:28.645024+0000","last_clean":"2024-04-16T16:21:28.645024+0000","last_became_active":"2024-04-16T16:21:25.603623+0000","last_became_peered":"2024-04-16T16:21:25.603623+0000","last_unstale":"2024-04-16T16:21:28.645024+0000","last_undegraded":"2024-04-16T16:21:28.645024+0000","last_fullsized":"2024-04-16T16:21:28.645024+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:02:28.264111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643880+0000","last_change":"2024-04-16T16:21:26.623525+0000","last_active":"2024-04-16T16:21:28.643880+0000","last_peered":"2024-04-16T16:21:28.643880+0000","last_clean":"2024-04-16T16:21:28.643880+0000","last_became_active":"2024-04-16T16:21:26.620561+0000","last_became_peered":"2024-04-16T16:21:26.620561+0000","last_unstale":"2024-04-16T16:21:28.643880+0000","last_undegraded":"2024-04-16T16:21:28.643880+0000","last_fullsized":"2024-04-16T16:21:28.643880+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:12:27.435258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.623337+0000","last_change":"2024-04-16T16:21:25.604810+0000","last_active":"2024-04-16T16:21:27.623337+0000","last_peered":"2024-04-16T16:21:27.623337+0000","last_clean":"2024-04-16T16:21:27.623337+0000","last_became_active":"2024-04-16T16:21:25.604495+0000","last_became_peered":"2024-04-16T16:21:25.604495+0000","last_unstale":"2024-04-16T16:21:27.623337+0000","last_undegraded":"2024-04-16T16:21:27.623337+0000","last_fullsized":"2024-04-16T16:21:27.623337+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:39:03.212104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643558+0000","last_change":"2024-04-16T16:21:26.632649+0000","last_active":"2024-04-16T16:21:28.643558+0000","last_peered":"2024-04-16T16:21:28.643558+0000","last_clean":"2024-04-16T16:21:28.643558+0000","last_became_active":"2024-04-16T16:21:26.632334+0000","last_became_peered":"2024-04-16T16:21:26.632334+0000","last_unstale":"2024-04-16T16:21:28.643558+0000","last_undegraded":"2024-04-16T16:21:28.643558+0000","last_fullsized":"2024-04-16T16:21:28.643558+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:49.214360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644128+0000","last_change":"2024-04-16T16:21:25.586171+0000","last_active":"2024-04-16T16:21:28.644128+0000","last_peered":"2024-04-16T16:21:28.644128+0000","last_clean":"2024-04-16T16:21:28.644128+0000","last_became_active":"2024-04-16T16:21:25.585863+0000","last_became_peered":"2024-04-16T16:21:25.585863+0000","last_unstale":"2024-04-16T16:21:28.644128+0000","last_undegraded":"2024-04-16T16:21:28.644128+0000","last_fullsized":"2024-04-16T16:21:28.644128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:28:09.057461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610500+0000","last_change":"2024-04-16T16:21:26.639412+0000","last_active":"2024-04-16T16:21:28.610500+0000","last_peered":"2024-04-16T16:21:28.610500+0000","last_clean":"2024-04-16T16:21:28.610500+0000","last_became_active":"2024-04-16T16:21:26.638086+0000","last_became_peered":"2024-04-16T16:21:26.638086+0000","last_unstale":"2024-04-16T16:21:28.610500+0000","last_undegraded":"2024-04-16T16:21:28.610500+0000","last_fullsized":"2024-04-16T16:21:28.610500+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:07:07.900771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.623013+0000","last_change":"2024-04-16T16:21:25.596529+0000","last_active":"2024-04-16T16:21:27.623013+0000","last_peered":"2024-04-16T16:21:27.623013+0000","last_clean":"2024-04-16T16:21:27.623013+0000","last_became_active":"2024-04-16T16:21:25.594866+0000","last_became_peered":"2024-04-16T16:21:25.594866+0000","last_unstale":"2024-04-16T16:21:27.623013+0000","last_undegraded":"2024-04-16T16:21:27.623013+0000","last_fullsized":"2024-04-16T16:21:27.623013+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:29:15.239160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627424+0000","last_change":"2024-04-16T16:21:26.627838+0000","last_active":"2024-04-16T16:21:27.627424+0000","last_peered":"2024-04-16T16:21:27.627424+0000","last_clean":"2024-04-16T16:21:27.627424+0000","last_became_active":"2024-04-16T16:21:26.627034+0000","last_became_peered":"2024-04-16T16:21:26.627034+0000","last_unstale":"2024-04-16T16:21:27.627424+0000","last_undegraded":"2024-04-16T16:21:27.627424+0000","last_fullsized":"2024-04-16T16:21:27.627424+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:49:46.979105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.641478+0000","last_change":"2024-04-16T16:21:25.600929+0000","last_active":"2024-04-16T16:21:28.641478+0000","last_peered":"2024-04-16T16:21:28.641478+0000","last_clean":"2024-04-16T16:21:28.641478+0000","last_became_active":"2024-04-16T16:21:25.600350+0000","last_became_peered":"2024-04-16T16:21:25.600350+0000","last_unstale":"2024-04-16T16:21:28.641478+0000","last_undegraded":"2024-04-16T16:21:28.641478+0000","last_fullsized":"2024-04-16T16:21:28.641478+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:31:27.379350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.641517+0000","last_change":"2024-04-16T16:21:26.637010+0000","last_active":"2024-04-16T16:21:28.641517+0000","last_peered":"2024-04-16T16:21:28.641517+0000","last_clean":"2024-04-16T16:21:28.641517+0000","last_became_active":"2024-04-16T16:21:26.636730+0000","last_became_peered":"2024-04-16T16:21:26.636730+0000","last_unstale":"2024-04-16T16:21:28.641517+0000","last_undegraded":"2024-04-16T16:21:28.641517+0000","last_fullsized":"2024-04-16T16:21:28.641517+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:08:16.919182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.627173+0000","last_change":"2024-04-16T16:21:25.598152+0000","last_active":"2024-04-16T16:21:27.627173+0000","last_peered":"2024-04-16T16:21:27.627173+0000","last_clean":"2024-04-16T16:21:27.627173+0000","last_became_active":"2024-04-16T16:21:25.597731+0000","last_became_peered":"2024-04-16T16:21:25.597731+0000","last_unstale":"2024-04-16T16:21:27.627173+0000","last_undegraded":"2024-04-16T16:21:27.627173+0000","last_fullsized":"2024-04-16T16:21:27.627173+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:54:51.270429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.625193+0000","last_change":"2024-04-16T16:21:26.633445+0000","last_active":"2024-04-16T16:21:27.625193+0000","last_peered":"2024-04-16T16:21:27.625193+0000","last_clean":"2024-04-16T16:21:27.625193+0000","last_became_active":"2024-04-16T16:21:26.633143+0000","last_became_peered":"2024-04-16T16:21:26.633143+0000","last_unstale":"2024-04-16T16:21:27.625193+0000","last_undegraded":"2024-04-16T16:21:27.625193+0000","last_fullsized":"2024-04-16T16:21:27.625193+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:39.911246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626337+0000","last_change":"2024-04-16T16:21:25.597534+0000","last_active":"2024-04-16T16:21:27.626337+0000","last_peered":"2024-04-16T16:21:27.626337+0000","last_clean":"2024-04-16T16:21:27.626337+0000","last_became_active":"2024-04-16T16:21:25.597218+0000","last_became_peered":"2024-04-16T16:21:25.597218+0000","last_unstale":"2024-04-16T16:21:27.626337+0000","last_undegraded":"2024-04-16T16:21:27.626337+0000","last_fullsized":"2024-04-16T16:21:27.626337+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:39.706251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.641220+0000","last_change":"2024-04-16T16:21:26.638223+0000","last_active":"2024-04-16T16:21:28.641220+0000","last_peered":"2024-04-16T16:21:28.641220+0000","last_clean":"2024-04-16T16:21:28.641220+0000","last_became_active":"2024-04-16T16:21:26.637929+0000","last_became_peered":"2024-04-16T16:21:26.637929+0000","last_unstale":"2024-04-16T16:21:28.641220+0000","last_undegraded":"2024-04-16T16:21:28.641220+0000","last_fullsized":"2024-04-16T16:21:28.641220+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:02:34.201782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.642044+0000","last_change":"2024-04-16T16:21:25.601886+0000","last_active":"2024-04-16T16:21:28.642044+0000","last_peered":"2024-04-16T16:21:28.642044+0000","last_clean":"2024-04-16T16:21:28.642044+0000","last_became_active":"2024-04-16T16:21:25.600879+0000","last_became_peered":"2024-04-16T16:21:25.600879+0000","last_unstale":"2024-04-16T16:21:28.642044+0000","last_undegraded":"2024-04-16T16:21:28.642044+0000","last_fullsized":"2024-04-16T16:21:28.642044+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:03:25.295410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.645104+0000","last_change":"2024-04-16T16:21:26.617017+0000","last_active":"2024-04-16T16:21:28.645104+0000","last_peered":"2024-04-16T16:21:28.645104+0000","last_clean":"2024-04-16T16:21:28.645104+0000","last_became_active":"2024-04-16T16:21:26.616626+0000","last_became_peered":"2024-04-16T16:21:26.616626+0000","last_unstale":"2024-04-16T16:21:28.645104+0000","last_undegraded":"2024-04-16T16:21:28.645104+0000","last_fullsized":"2024-04-16T16:21:28.645104+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:37:16.481358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.628575+0000","last_change":"2024-04-16T16:21:25.605051+0000","last_active":"2024-04-16T16:21:27.628575+0000","last_peered":"2024-04-16T16:21:27.628575+0000","last_clean":"2024-04-16T16:21:27.628575+0000","last_became_active":"2024-04-16T16:21:25.604679+0000","last_became_peered":"2024-04-16T16:21:25.604679+0000","last_unstale":"2024-04-16T16:21:27.628575+0000","last_undegraded":"2024-04-16T16:21:27.628575+0000","last_fullsized":"2024-04-16T16:21:27.628575+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:39:03.212104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.645393+0000","last_change":"2024-04-16T16:21:26.632757+0000","last_active":"2024-04-16T16:21:28.645393+0000","last_peered":"2024-04-16T16:21:28.645393+0000","last_clean":"2024-04-16T16:21:28.645393+0000","last_became_active":"2024-04-16T16:21:26.632460+0000","last_became_peered":"2024-04-16T16:21:26.632460+0000","last_unstale":"2024-04-16T16:21:28.645393+0000","last_undegraded":"2024-04-16T16:21:28.645393+0000","last_fullsized":"2024-04-16T16:21:28.645393+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:26.590328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"142'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610449+0000","last_change":"2024-04-16T16:21:25.585117+0000","last_active":"2024-04-16T16:21:28.610449+0000","last_peered":"2024-04-16T16:21:28.610449+0000","last_clean":"2024-04-16T16:21:28.610449+0000","last_became_active":"2024-04-16T16:21:25.584609+0000","last_became_peered":"2024-04-16T16:21:25.584609+0000","last_unstale":"2024-04-16T16:21:28.610449+0000","last_undegraded":"2024-04-16T16:21:28.610449+0000","last_fullsized":"2024-04-16T16:21:28.610449+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:01.917706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.622948+0000","last_change":"2024-04-16T16:21:26.635364+0000","last_active":"2024-04-16T16:21:27.622948+0000","last_peered":"2024-04-16T16:21:27.622948+0000","last_clean":"2024-04-16T16:21:27.622948+0000","last_became_active":"2024-04-16T16:21:26.635101+0000","last_became_peered":"2024-04-16T16:21:26.635101+0000","last_unstale":"2024-04-16T16:21:27.622948+0000","last_undegraded":"2024-04-16T16:21:27.622948+0000","last_fullsized":"2024-04-16T16:21:27.622948+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:52:50.608203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610772+0000","last_change":"2024-04-16T16:21:25.589018+0000","last_active":"2024-04-16T16:21:28.610772+0000","last_peered":"2024-04-16T16:21:28.610772+0000","last_clean":"2024-04-16T16:21:28.610772+0000","last_became_active":"2024-04-16T16:21:25.588722+0000","last_became_peered":"2024-04-16T16:21:25.588722+0000","last_unstale":"2024-04-16T16:21:28.610772+0000","last_undegraded":"2024-04-16T16:21:28.610772+0000","last_fullsized":"2024-04-16T16:21:28.610772+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:31:45.126004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610676+0000","last_change":"2024-04-16T16:21:26.630528+0000","last_active":"2024-04-16T16:21:28.610676+0000","last_peered":"2024-04-16T16:21:28.610676+0000","last_clean":"2024-04-16T16:21:28.610676+0000","last_became_active":"2024-04-16T16:21:26.630235+0000","last_became_peered":"2024-04-16T16:21:26.630235+0000","last_unstale":"2024-04-16T16:21:28.610676+0000","last_undegraded":"2024-04-16T16:21:28.610676+0000","last_fullsized":"2024-04-16T16:21:28.610676+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:46:08.337988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.610296+0000","last_change":"2024-04-16T16:21:25.582293+0000","last_active":"2024-04-16T16:21:28.610296+0000","last_peered":"2024-04-16T16:21:28.610296+0000","last_clean":"2024-04-16T16:21:28.610296+0000","last_became_active":"2024-04-16T16:21:25.581912+0000","last_became_peered":"2024-04-16T16:21:25.581912+0000","last_unstale":"2024-04-16T16:21:28.610296+0000","last_undegraded":"2024-04-16T16:21:28.610296+0000","last_fullsized":"2024-04-16T16:21:28.610296+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:02:33.389254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643444+0000","last_change":"2024-04-16T16:21:26.629185+0000","last_active":"2024-04-16T16:21:28.643444+0000","last_peered":"2024-04-16T16:21:28.643444+0000","last_clean":"2024-04-16T16:21:28.643444+0000","last_became_active":"2024-04-16T16:21:26.628800+0000","last_became_peered":"2024-04-16T16:21:26.628800+0000","last_unstale":"2024-04-16T16:21:28.643444+0000","last_undegraded":"2024-04-16T16:21:28.643444+0000","last_fullsized":"2024-04-16T16:21:28.643444+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:33:56.310523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.641618+0000","last_change":"2024-04-16T16:21:25.598027+0000","last_active":"2024-04-16T16:21:28.641618+0000","last_peered":"2024-04-16T16:21:28.641618+0000","last_clean":"2024-04-16T16:21:28.641618+0000","last_became_active":"2024-04-16T16:21:25.597733+0000","last_became_peered":"2024-04-16T16:21:25.597733+0000","last_unstale":"2024-04-16T16:21:28.641618+0000","last_undegraded":"2024-04-16T16:21:28.641618+0000","last_fullsized":"2024-04-16T16:21:28.641618+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:37.737851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644991+0000","last_change":"2024-04-16T16:21:26.631846+0000","last_active":"2024-04-16T16:21:28.644991+0000","last_peered":"2024-04-16T16:21:28.644991+0000","last_clean":"2024-04-16T16:21:28.644991+0000","last_became_active":"2024-04-16T16:21:26.631515+0000","last_became_peered":"2024-04-16T16:21:26.631515+0000","last_unstale":"2024-04-16T16:21:28.644991+0000","last_undegraded":"2024-04-16T16:21:28.644991+0000","last_fullsized":"2024-04-16T16:21:28.644991+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:20.296700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626953+0000","last_change":"2024-04-16T16:21:25.600334+0000","last_active":"2024-04-16T16:21:27.626953+0000","last_peered":"2024-04-16T16:21:27.626953+0000","last_clean":"2024-04-16T16:21:27.626953+0000","last_became_active":"2024-04-16T16:21:25.599921+0000","last_became_peered":"2024-04-16T16:21:25.599921+0000","last_unstale":"2024-04-16T16:21:27.626953+0000","last_undegraded":"2024-04-16T16:21:27.626953+0000","last_fullsized":"2024-04-16T16:21:27.626953+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:06.016163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626017+0000","last_change":"2024-04-16T16:21:26.624365+0000","last_active":"2024-04-16T16:21:27.626017+0000","last_peered":"2024-04-16T16:21:27.626017+0000","last_clean":"2024-04-16T16:21:27.626017+0000","last_became_active":"2024-04-16T16:21:26.624052+0000","last_became_peered":"2024-04-16T16:21:26.624052+0000","last_unstale":"2024-04-16T16:21:27.626017+0000","last_undegraded":"2024-04-16T16:21:27.626017+0000","last_fullsized":"2024-04-16T16:21:27.626017+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:52:50.608203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.642377+0000","last_change":"2024-04-16T16:21:25.602774+0000","last_active":"2024-04-16T16:21:28.642377+0000","last_peered":"2024-04-16T16:21:28.642377+0000","last_clean":"2024-04-16T16:21:28.642377+0000","last_became_active":"2024-04-16T16:21:25.602465+0000","last_became_peered":"2024-04-16T16:21:25.602465+0000","last_unstale":"2024-04-16T16:21:28.642377+0000","last_undegraded":"2024-04-16T16:21:28.642377+0000","last_fullsized":"2024-04-16T16:21:28.642377+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:07:06.886135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.644754+0000","last_change":"2024-04-16T16:21:26.620146+0000","last_active":"2024-04-16T16:21:28.644754+0000","last_peered":"2024-04-16T16:21:28.644754+0000","last_clean":"2024-04-16T16:21:28.644754+0000","last_became_active":"2024-04-16T16:21:26.619872+0000","last_became_peered":"2024-04-16T16:21:26.619872+0000","last_unstale":"2024-04-16T16:21:28.644754+0000","last_undegraded":"2024-04-16T16:21:28.644754+0000","last_fullsized":"2024-04-16T16:21:28.644754+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:59:15.375653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.622489+0000","last_change":"2024-04-16T16:21:25.601334+0000","last_active":"2024-04-16T16:21:27.622489+0000","last_peered":"2024-04-16T16:21:27.622489+0000","last_clean":"2024-04-16T16:21:27.622489+0000","last_became_active":"2024-04-16T16:21:25.600462+0000","last_became_peered":"2024-04-16T16:21:25.600462+0000","last_unstale":"2024-04-16T16:21:27.622489+0000","last_undegraded":"2024-04-16T16:21:27.622489+0000","last_fullsized":"2024-04-16T16:21:27.622489+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:54:51.270429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.642187+0000","last_change":"2024-04-16T16:21:26.633261+0000","last_active":"2024-04-16T16:21:28.642187+0000","last_peered":"2024-04-16T16:21:28.642187+0000","last_clean":"2024-04-16T16:21:28.642187+0000","last_became_active":"2024-04-16T16:21:26.632970+0000","last_became_peered":"2024-04-16T16:21:26.632970+0000","last_unstale":"2024-04-16T16:21:28.642187+0000","last_undegraded":"2024-04-16T16:21:28.642187+0000","last_fullsized":"2024-04-16T16:21:28.642187+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:25.556374+0000","last_clean_scrub_stamp":"2024-04-16T16:21:25.556374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:48:48.721677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"141'3","reported_seq":17,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:28.643844+0000","last_change":"2024-04-16T16:21:25.604880+0000","last_active":"2024-04-16T16:21:28.643844+0000","last_peered":"2024-04-16T16:21:28.643844+0000","last_clean":"2024-04-16T16:21:28.643844+0000","last_became_active":"2024-04-16T16:21:25.604027+0000","last_became_peered":"2024-04-16T16:21:25.604027+0000","last_unstale":"2024-04-16T16:21:28.643844+0000","last_undegraded":"2024-04-16T16:21:28.643844+0000","last_fullsized":"2024-04-16T16:21:28.643844+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:21:24.541738+0000","last_clean_scrub_stamp":"2024-04-16T16:21:24.541738+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:01.205017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"130'4669","reported_seq":5284,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-16T16:21:27.626166+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:21:27.626166+0000","last_peered":"2024-04-16T16:21:27.626166+0000","last_clean":"2024-04-16T16:21:27.626166+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:21:27.626166+0000","last_undegraded":"2024-04-16T16:21:27.626166+0000","last_fullsized":"2024-04-16T16:21:27.626166+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1083,"num_read_kb":2113,"num_write":347,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1083,"num_read_kb":2113,"num_write":347,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4669,"ondisk_log_size":4669,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574943,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33512,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338328,"statfs":{"total":96636764160,"available":96602447872,"internally_reserved":0,"allocated":2379776,"data_stored":1436331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574944,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32932,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338908,"statfs":{"total":96636764160,"available":96603041792,"internally_reserved":0,"allocated":1789952,"data_stored":845457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574944,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1781760,"data_stored":845333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574945,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":1777664,"data_stored":845311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574943,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1781760,"data_stored":845401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574943,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37096,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334744,"statfs":{"total":96636764160,"available":96598777856,"internally_reserved":0,"allocated":1794048,"data_stored":846993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1781760,"data_stored":845893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574944,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37672,"kb_used_data":2332,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334168,"statfs":{"total":96636764160,"available":96598188032,"internally_reserved":0,"allocated":2387968,"data_stored":1437325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:21:36.722 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-16T16:21:36.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-16T16:21:37.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.001+0000 7ffab4600640 1 Processor -- start 2024-04-16T16:21:37.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.001+0000 7ffab4600640 1 -- start start 2024-04-16T16:21:37.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.003+0000 7ffab4600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ffaac160950 0x7ffaac167370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:37.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.003+0000 7ffab4600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaac1678b0 con 0x7ffaac0730e0 2024-04-16T16:21:37.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.003+0000 7ffab4600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaac0b5260 con 0x7ffaac0d07c0 2024-04-16T16:21:37.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.003+0000 7ffab4600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ffaac0b5010 con 0x7ffaac160950 2024-04-16T16:21:37.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.003+0000 7ffab1200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ffaac0730e0 0x7ffaac0b4900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55212/0 (socket says 172.21.15.45:55212) 2024-04-16T16:21:37.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.003+0000 7ffab1200640 1 -- 172.21.15.45:0/3425939008 learned_addr learned my addr 172.21.15.45:0/3425939008 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.004+0000 7ffab1c00640 1 --2- 172.21.15.45:0/3425939008 >> v2:172.21.15.103:3300/0 conn(0x7ffaac160950 0x7ffaac167370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.004+0000 7ffaabe00640 1 -- 172.21.15.45:0/3425939008 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 146688964 0 0) 0x7ffaac1678b0 con 0x7ffaac0730e0 2024-04-16T16:21:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.004+0000 7ffaabe00640 1 -- 172.21.15.45:0/3425939008 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa94003520 con 0x7ffaac0730e0 2024-04-16T16:21:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.004+0000 7ffaabe00640 1 -- 172.21.15.45:0/3425939008 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4094566951 0 0) 0x7ffa94003520 con 0x7ffaac0730e0 2024-04-16T16:21:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.004+0000 7ffaabe00640 1 -- 172.21.15.45:0/3425939008 >> v2:172.21.15.103:3300/0 conn(0x7ffaac160950 msgr2=0x7ffaac167370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.004+0000 7ffaabe00640 1 --2- 172.21.15.45:0/3425939008 >> v2:172.21.15.103:3300/0 conn(0x7ffaac160950 0x7ffaac167370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.004+0000 7ffaabe00640 1 -- 172.21.15.45:0/3425939008 >> v1:172.21.15.103:6789/0 conn(0x7ffaac0d07c0 legacy=0x7ffaac1633e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.004+0000 7ffaabe00640 1 -- 172.21.15.45:0/3425939008 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac0b5540 con 0x7ffaac0730e0 2024-04-16T16:21:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.004+0000 7ffaabe00640 1 -- 172.21.15.45:0/3425939008 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7ffa98001670 con 0x7ffaac0730e0 2024-04-16T16:21:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.004+0000 7ffaabe00640 1 -- 172.21.15.45:0/3425939008 >> v1:172.21.15.45:6789/0 conn(0x7ffaac0730e0 legacy=0x7ffaac0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.004+0000 7ffaabe00640 1 --2- 172.21.15.45:0/3425939008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa940039b0 0x7ffa94003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.004+0000 7ffaabe00640 1 --2- 172.21.15.45:0/3425939008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa94004800 0x7ffa94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.005+0000 7ffaabe00640 1 --2- 172.21.15.45:0/3425939008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa940051c0 0x7ffa9400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.005+0000 7ffaabe00640 1 -- 172.21.15.45:0/3425939008 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac0b5010 con 0x7ffa940051c0 2024-04-16T16:21:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.005+0000 7ffaabe00640 1 -- 172.21.15.45:0/3425939008 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9400c110 con 0x7ffa940039b0 2024-04-16T16:21:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.005+0000 7ffaabe00640 1 -- 172.21.15.45:0/3425939008 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffa9400c310 con 0x7ffa94004800 2024-04-16T16:21:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.005+0000 7ffab2600640 1 --2- 172.21.15.45:0/3425939008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa940051c0 0x7ffa9400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.005+0000 7ffab1c00640 1 --2- 172.21.15.45:0/3425939008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa940039b0 0x7ffa94003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.005+0000 7ffab1200640 1 --2- 172.21.15.45:0/3425939008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa94004800 0x7ffa94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.005+0000 7ffab2600640 1 -- 172.21.15.45:0/3425939008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa94004800 msgr2=0x7ffa94004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.005+0000 7ffab2600640 1 --2- 172.21.15.45:0/3425939008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa94004800 0x7ffa94004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.006+0000 7ffab2600640 1 -- 172.21.15.45:0/3425939008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa940039b0 msgr2=0x7ffa94003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.006+0000 7ffab2600640 1 --2- 172.21.15.45:0/3425939008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa940039b0 0x7ffa94003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.006+0000 7ffab2600640 1 -- 172.21.15.45:0/3425939008 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa94003520 con 0x7ffa940051c0 2024-04-16T16:21:37.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.006+0000 7ffab1200640 1 --2- 172.21.15.45:0/3425939008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa94004800 0x7ffa94004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:21:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.007+0000 7ffab2600640 1 -- 172.21.15.45:0/3425939008 reap_dead start 2024-04-16T16:21:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.007+0000 7ffab2600640 1 --2- 172.21.15.45:0/3425939008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa940051c0 0x7ffa9400b9c0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4001e50 tx=0x7ffa94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a05d53f81f5f68 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.007+0000 7ffaabe00640 1 -- 172.21.15.45:0/3425939008 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4001a70 con 0x7ffa940051c0 2024-04-16T16:21:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.007+0000 7ffaabe00640 1 -- 172.21.15.45:0/3425939008 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaa4003bb0 con 0x7ffa940051c0 2024-04-16T16:21:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.007+0000 7ffaabe00640 1 -- 172.21.15.45:0/3425939008 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa40050d0 con 0x7ffa940051c0 2024-04-16T16:21:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.008+0000 7ffab4600640 1 -- 172.21.15.45:0/3425939008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa940051c0 msgr2=0x7ffa9400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.008+0000 7ffab4600640 1 --2- 172.21.15.45:0/3425939008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa940051c0 0x7ffa9400b9c0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4001e50 tx=0x7ffa94005830 comp rx=0 tx=0).stop 2024-04-16T16:21:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.008+0000 7ffab4600640 1 -- 172.21.15.45:0/3425939008 shutdown_connections 2024-04-16T16:21:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.008+0000 7ffab4600640 1 --2- 172.21.15.45:0/3425939008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa940051c0 0x7ffa9400b9c0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.008+0000 7ffab4600640 1 -- 172.21.15.45:0/3425939008 >> 172.21.15.45:0/3425939008 conn(0x7ffaac0b3970 msgr2=0x7ffaac0b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.008+0000 7ffab4600640 1 -- 172.21.15.45:0/3425939008 shutdown_connections 2024-04-16T16:21:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.008+0000 7ffab4600640 1 -- 172.21.15.45:0/3425939008 wait complete. 2024-04-16T16:21:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.010+0000 7ffab4600640 1 Processor -- start 2024-04-16T16:21:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.010+0000 7ffab4600640 1 -- start start 2024-04-16T16:21:37.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.011+0000 7ffab4600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa940051c0 0x7ffaac158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:37.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.011+0000 7ffab4600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaac146b10 0x7ffaac146f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:37.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.011+0000 7ffab4600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffaac15e690 0x7ffaac15eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:37.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.011+0000 7ffab4600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac160d30 con 0x7ffaac146b10 2024-04-16T16:21:37.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.011+0000 7ffab4600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac166e50 con 0x7ffaac15e690 2024-04-16T16:21:37.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.011+0000 7ffab4600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffaac167370 con 0x7ffa940051c0 2024-04-16T16:21:37.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.012+0000 7ffab1200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaac146b10 0x7ffaac146f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:37.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.012+0000 7ffab1200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaac146b10 0x7ffaac146f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60132/0 (socket says 172.21.15.45:60132) 2024-04-16T16:21:37.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.012+0000 7ffab1200640 1 -- 172.21.15.45:0/4065419877 learned_addr learned my addr 172.21.15.45:0/4065419877 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:37.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.012+0000 7ffab1c00640 1 --2- 172.21.15.45:0/4065419877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa940051c0 0x7ffaac158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.012+0000 7ffab2600640 1 --2- 172.21.15.45:0/4065419877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffaac15e690 0x7ffaac15eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.012+0000 7ffab1200640 1 -- 172.21.15.45:0/4065419877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa940051c0 msgr2=0x7ffaac158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.012+0000 7ffab1200640 1 --2- 172.21.15.45:0/4065419877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa940051c0 0x7ffaac158430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.012+0000 7ffab1200640 1 -- 172.21.15.45:0/4065419877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffaac15e690 msgr2=0x7ffaac15eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.012+0000 7ffab1200640 1 --2- 172.21.15.45:0/4065419877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffaac15e690 0x7ffaac15eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.012+0000 7ffab1200640 1 -- 172.21.15.45:0/4065419877 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac158970 con 0x7ffaac146b10 2024-04-16T16:21:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.012+0000 7ffab1200640 1 --2- 172.21.15.45:0/4065419877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaac146b10 0x7ffaac146f90 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7ffa98001200 tx=0x7ffa9800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.013+0000 7ffaa9600640 1 -- 172.21.15.45:0/4065419877 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9800c8d0 con 0x7ffaac146b10 2024-04-16T16:21:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.013+0000 7ffaa9600640 1 -- 172.21.15.45:0/4065419877 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa98018070 con 0x7ffaac146b10 2024-04-16T16:21:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.013+0000 7ffaa9600640 1 -- 172.21.15.45:0/4065419877 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9800cc70 con 0x7ffaac146b10 2024-04-16T16:21:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.013+0000 7ffab4600640 1 -- 172.21.15.45:0/4065419877 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaac15f3e0 con 0x7ffaac146b10 2024-04-16T16:21:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.013+0000 7ffab4600640 1 -- 172.21.15.45:0/4065419877 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaac15f9a0 con 0x7ffaac146b10 2024-04-16T16:21:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.014+0000 7ffab4600640 1 -- 172.21.15.45:0/4065419877 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaac075780 con 0x7ffaac146b10 2024-04-16T16:21:37.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.015+0000 7ffaa9600640 1 -- 172.21.15.45:0/4065419877 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7ffa9801d460 con 0x7ffaac146b10 2024-04-16T16:21:37.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.016+0000 7ffaa9600640 1 --2- 172.21.15.45:0/4065419877 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7ffa8c06ca40 0x7ffa8c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:37.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.016+0000 7ffaa9600640 1 -- 172.21.15.45:0/4065419877 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7ffa9808bb80 con 0x7ffaac146b10 2024-04-16T16:21:37.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.017+0000 7ffab1c00640 1 --2- 172.21.15.45:0/4065419877 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7ffa8c06ca40 0x7ffa8c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:37.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.018+0000 7ffab1c00640 1 --2- 172.21.15.45:0/4065419877 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7ffa8c06ca40 0x7ffa8c06ef00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0006fd0 tx=0x7ffaa0008040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:37.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.021+0000 7ffaa9600640 1 -- 172.21.15.45:0/4065419877 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffaac158970 con 0x7ffaac146b10 2024-04-16T16:21:37.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.303+0000 7ffab4600640 1 -- 172.21.15.45:0/4065419877 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7ffaac06e9b0 con 0x7ffaac146b10 2024-04-16T16:21:37.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.307+0000 7ffaa9600640 1 -- 172.21.15.45:0/4065419877 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v26) ==== 190+0+147 (secure 0 0 0) 0x7ffa98059e00 con 0x7ffaac146b10 2024-04-16T16:21:37.307 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-04-16T16:21:37.307 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ== 2024-04-16T16:21:37.307 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:21:37.307 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:21:37.307 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-16T16:21:37.307 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2024-04-16T16:21:37.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.311+0000 7ffab4600640 1 -- 172.21.15.45:0/4065419877 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7ffa8c06ca40 msgr2=0x7ffa8c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:37.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.311+0000 7ffab4600640 1 --2- 172.21.15.45:0/4065419877 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7ffa8c06ca40 0x7ffa8c06ef00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0006fd0 tx=0x7ffaa0008040 comp rx=0 tx=0).stop 2024-04-16T16:21:37.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.311+0000 7ffab4600640 1 -- 172.21.15.45:0/4065419877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaac146b10 msgr2=0x7ffaac146f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:37.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.311+0000 7ffab4600640 1 --2- 172.21.15.45:0/4065419877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaac146b10 0x7ffaac146f90 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7ffa98001200 tx=0x7ffa9800ec60 comp rx=0 tx=0).stop 2024-04-16T16:21:37.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.311+0000 7ffab4600640 1 -- 172.21.15.45:0/4065419877 shutdown_connections 2024-04-16T16:21:37.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.311+0000 7ffab4600640 1 --2- 172.21.15.45:0/4065419877 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7ffa8c06ca40 0x7ffa8c06ef00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.311+0000 7ffab4600640 1 --2- 172.21.15.45:0/4065419877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffaac15e690 0x7ffaac15eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.311+0000 7ffab4600640 1 --2- 172.21.15.45:0/4065419877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaac146b10 0x7ffaac146f90 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.311+0000 7ffab4600640 1 --2- 172.21.15.45:0/4065419877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa940051c0 0x7ffaac158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.311+0000 7ffab4600640 1 -- 172.21.15.45:0/4065419877 >> 172.21.15.45:0/4065419877 conn(0x7ffaac0b3970 msgr2=0x7ffaac15afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:37.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.311+0000 7ffab4600640 1 -- 172.21.15.45:0/4065419877 shutdown_connections 2024-04-16T16:21:37.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.312+0000 7ffab4600640 1 -- 172.21.15.45:0/4065419877 wait complete. 2024-04-16T16:21:37.330 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:21:37.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.549+0000 7f3acdc00640 1 Processor -- start 2024-04-16T16:21:37.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.549+0000 7f3acdc00640 1 -- start start 2024-04-16T16:21:37.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.551+0000 7f3acdc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3ac80cde00 0x7f3ac80cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:37.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.552+0000 7f3acdc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3ac806f540 0x7f3ac81645b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:37.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.552+0000 7f3acdc00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ac8163510 con 0x7f3ac8163010 2024-04-16T16:21:37.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.552+0000 7f3acdc00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3ac81637f0 con 0x7f3ac806f540 2024-04-16T16:21:37.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.552+0000 7f3acdc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3ac8163ad0 con 0x7f3ac80cde00 2024-04-16T16:21:37.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.552+0000 7f3abfe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3ac8163010 0x7f3ac80ce1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55224/0 (socket says 172.21.15.45:55224) 2024-04-16T16:21:37.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.552+0000 7f3abfe00640 1 -- 172.21.15.45:0/674709798 learned_addr learned my addr 172.21.15.45:0/674709798 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:37.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.552+0000 7f3acd200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3ac806f540 0x7f3ac81645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:37.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.552+0000 7f3ac7e00640 1 --2- 172.21.15.45:0/674709798 >> v2:172.21.15.103:3300/0 conn(0x7f3ac80cde00 0x7f3ac80cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:37.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.553+0000 7f3ac7400640 1 -- 172.21.15.45:0/674709798 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2859333992 0 0) 0x7f3ac8163510 con 0x7f3ac8163010 2024-04-16T16:21:37.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.553+0000 7f3ac7400640 1 -- 172.21.15.45:0/674709798 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3aac003520 con 0x7f3ac8163010 2024-04-16T16:21:37.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.553+0000 7f3acd200640 1 -- 172.21.15.45:0/674709798 >> v2:172.21.15.103:3300/0 conn(0x7f3ac80cde00 msgr2=0x7f3ac80cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:37.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.553+0000 7f3acd200640 1 --2- 172.21.15.45:0/674709798 >> v2:172.21.15.103:3300/0 conn(0x7f3ac80cde00 0x7f3ac80cff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.553+0000 7f3acd200640 1 -- 172.21.15.45:0/674709798 >> v1:172.21.15.45:6789/0 conn(0x7f3ac8163010 legacy=0x7f3ac80ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:37.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.553+0000 7f3acd200640 1 -- 172.21.15.45:0/674709798 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac8163db0 con 0x7f3ac806f540 2024-04-16T16:21:37.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.554+0000 7f3acd200640 1 --2- 172.21.15.45:0/674709798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ac806f540 0x7f3ac81645b0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8073060 tx=0x7f3ac002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ea6a097e083cad server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:37.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.554+0000 7f3ac7400640 1 -- 172.21.15.45:0/674709798 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0039070 con 0x7f3ac806f540 2024-04-16T16:21:37.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.554+0000 7f3ac7400640 1 -- 172.21.15.45:0/674709798 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ac002cc80 con 0x7f3ac806f540 2024-04-16T16:21:37.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.554+0000 7f3ac7400640 1 -- 172.21.15.45:0/674709798 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0034660 con 0x7f3ac806f540 2024-04-16T16:21:37.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.555+0000 7f3acdc00640 1 -- 172.21.15.45:0/674709798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ac806f540 msgr2=0x7f3ac81645b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:37.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.555+0000 7f3acdc00640 1 --2- 172.21.15.45:0/674709798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ac806f540 0x7f3ac81645b0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8073060 tx=0x7f3ac002bed0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.555+0000 7f3acdc00640 1 -- 172.21.15.45:0/674709798 shutdown_connections 2024-04-16T16:21:37.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.555+0000 7f3acdc00640 1 --2- 172.21.15.45:0/674709798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ac806f540 0x7f3ac81645b0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.555+0000 7f3acdc00640 1 --2- 172.21.15.45:0/674709798 >> v2:172.21.15.103:3300/0 conn(0x7f3ac80cde00 0x7f3ac80cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.555+0000 7f3acdc00640 1 -- 172.21.15.45:0/674709798 >> 172.21.15.45:0/674709798 conn(0x7f3ac806b930 msgr2=0x7f3ac806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:37.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.555+0000 7f3acdc00640 1 -- 172.21.15.45:0/674709798 shutdown_connections 2024-04-16T16:21:37.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.555+0000 7f3acdc00640 1 -- 172.21.15.45:0/674709798 wait complete. 2024-04-16T16:21:37.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.557+0000 7f3acdc00640 1 Processor -- start 2024-04-16T16:21:37.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.557+0000 7f3acdc00640 1 -- start start 2024-04-16T16:21:37.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.558+0000 7f3acdc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac806f540 0x7f3ac80b5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:37.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.558+0000 7f3acdc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ac80cde00 0x7f3ac814d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:37.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.558+0000 7f3acdc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ac8163010 0x7f3ac8161ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:37.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.558+0000 7f3acdc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac8164af0 con 0x7f3ac80cde00 2024-04-16T16:21:37.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.558+0000 7f3acdc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac806f970 con 0x7f3ac806f540 2024-04-16T16:21:37.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.558+0000 7f3acdc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3ac8164c70 con 0x7f3ac8163010 2024-04-16T16:21:37.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.558+0000 7f3abfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ac80cde00 0x7f3ac814d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:37.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.559+0000 7f3abfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ac80cde00 0x7f3ac814d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60150/0 (socket says 172.21.15.45:60150) 2024-04-16T16:21:37.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.559+0000 7f3abfe00640 1 -- 172.21.15.45:0/3233010623 learned_addr learned my addr 172.21.15.45:0/3233010623 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:37.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.559+0000 7f3ac7e00640 1 --2- 172.21.15.45:0/3233010623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac806f540 0x7f3ac80b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:37.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.559+0000 7f3acd200640 1 --2- 172.21.15.45:0/3233010623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ac8163010 0x7f3ac8161ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:37.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.559+0000 7f3abfe00640 1 -- 172.21.15.45:0/3233010623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ac8163010 msgr2=0x7f3ac8161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:37.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.559+0000 7f3abfe00640 1 --2- 172.21.15.45:0/3233010623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ac8163010 0x7f3ac8161ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.559+0000 7f3abfe00640 1 -- 172.21.15.45:0/3233010623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac806f540 msgr2=0x7f3ac80b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:37.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.559+0000 7f3abfe00640 1 --2- 172.21.15.45:0/3233010623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac806f540 0x7f3ac80b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.559+0000 7f3abfe00640 1 -- 172.21.15.45:0/3233010623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac8151db0 con 0x7f3ac80cde00 2024-04-16T16:21:37.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.559+0000 7f3abfe00640 1 --2- 172.21.15.45:0/3233010623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ac80cde00 0x7f3ac814d4e0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0001eb0 tx=0x7f3ab000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:37.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.560+0000 7f3ac4c00640 1 -- 172.21.15.45:0/3233010623 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0017070 con 0x7f3ac80cde00 2024-04-16T16:21:37.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.560+0000 7f3ac4c00640 1 -- 172.21.15.45:0/3233010623 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ab0005d00 con 0x7f3ac80cde00 2024-04-16T16:21:37.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.560+0000 7f3ac4c00640 1 -- 172.21.15.45:0/3233010623 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0007310 con 0x7f3ac80cde00 2024-04-16T16:21:37.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.560+0000 7f3acdc00640 1 -- 172.21.15.45:0/3233010623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac8152090 con 0x7f3ac80cde00 2024-04-16T16:21:37.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.560+0000 7f3acdc00640 1 -- 172.21.15.45:0/3233010623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac8152650 con 0x7f3ac80cde00 2024-04-16T16:21:37.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.561+0000 7f3acdc00640 1 -- 172.21.15.45:0/3233010623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ac8075780 con 0x7f3ac80cde00 2024-04-16T16:21:37.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.562+0000 7f3ac4c00640 1 -- 172.21.15.45:0/3233010623 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f3ab001a070 con 0x7f3ac80cde00 2024-04-16T16:21:37.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.563+0000 7f3ac4c00640 1 --2- 172.21.15.45:0/3233010623 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f3aa406ca10 0x7f3aa406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:37.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.563+0000 7f3ac4c00640 1 -- 172.21.15.45:0/3233010623 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7f3ab008b1b0 con 0x7f3ac80cde00 2024-04-16T16:21:37.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.564+0000 7f3ac7e00640 1 --2- 172.21.15.45:0/3233010623 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f3aa406ca10 0x7f3aa406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:37.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.565+0000 7f3ac7e00640 1 --2- 172.21.15.45:0/3233010623 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f3aa406ca10 0x7f3aa406eed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8009ad0 tx=0x7f3ab8008040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:37.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.568+0000 7f3ac4c00640 1 -- 172.21.15.45:0/3233010623 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3ab0059430 con 0x7f3ac80cde00 2024-04-16T16:21:37.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.815+0000 7f3acdc00640 1 -- 172.21.15.45:0/3233010623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ac806fc20 con 0x7f3ac80cde00 2024-04-16T16:21:37.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.816+0000 7f3ac4c00640 1 -- 172.21.15.45:0/3233010623 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5190 (secure 0 0 0) 0x7f3ab0058d30 con 0x7f3ac80cde00 2024-04-16T16:21:37.815 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:37.816 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2585865379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2585865379},{"type":"v1","addr":"172.21.15.45:6834","nonce":2585865379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6590,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/777087937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":777087937},{"type":"v1","addr":"172.21.15.45:6835","nonce":777087937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6593,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2561260581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2561260581},{"type":"v1","addr":"172.21.15.103:6835","nonce":2561260581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:21:26.066855+0000","modified":"2024-04-16T16:21:29.155869+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6596":{"gid":6596,"name":"d","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6837/3252460318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3252460318},{"type":"v1","addr":"172.21.15.103:6837","nonce":3252460318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6596,"qdb_cluster":[6596]},"id":10}]} 2024-04-16T16:21:37.816 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 100 2024-04-16T16:21:37.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.819+0000 7f3acdc00640 1 -- 172.21.15.45:0/3233010623 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f3aa406ca10 msgr2=0x7f3aa406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:37.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.819+0000 7f3acdc00640 1 --2- 172.21.15.45:0/3233010623 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f3aa406ca10 0x7f3aa406eed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8009ad0 tx=0x7f3ab8008040 comp rx=0 tx=0).stop 2024-04-16T16:21:37.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.819+0000 7f3acdc00640 1 -- 172.21.15.45:0/3233010623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ac80cde00 msgr2=0x7f3ac814d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:37.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.819+0000 7f3acdc00640 1 --2- 172.21.15.45:0/3233010623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ac80cde00 0x7f3ac814d4e0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0001eb0 tx=0x7f3ab000c820 comp rx=0 tx=0).stop 2024-04-16T16:21:37.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.819+0000 7f3acdc00640 1 -- 172.21.15.45:0/3233010623 shutdown_connections 2024-04-16T16:21:37.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.819+0000 7f3acdc00640 1 --2- 172.21.15.45:0/3233010623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ac8163010 0x7f3ac8161ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.819+0000 7f3acdc00640 1 --2- 172.21.15.45:0/3233010623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ac80cde00 0x7f3ac814d4e0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.819+0000 7f3acdc00640 1 --2- 172.21.15.45:0/3233010623 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f3aa406ca10 0x7f3aa406eed0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.819+0000 7f3acdc00640 1 --2- 172.21.15.45:0/3233010623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac806f540 0x7f3ac80b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:37.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.819+0000 7f3acdc00640 1 -- 172.21.15.45:0/3233010623 >> 172.21.15.45:0/3233010623 conn(0x7f3ac806b930 msgr2=0x7f3ac8159e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:37.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.820+0000 7f3acdc00640 1 -- 172.21.15.45:0/3233010623 shutdown_connections 2024-04-16T16:21:37.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:37.820+0000 7f3acdc00640 1 -- 172.21.15.45:0/3233010623 wait complete. 2024-04-16T16:21:38.239 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:21:26.066855+0000', 'modified': '2024-04-16T16:21:29.155869+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6596}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6596': {'gid': 6596, 'name': 'd', 'rank': 0, 'incarnation': 96, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.103:6837/3252460318', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6836', 'nonce': 3252460318}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 3252460318}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6596, 'qdb_cluster': [6596]} 2024-04-16T16:21:38.239 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-16T16:21:38.240 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-16T16:21:38.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.475+0000 7f79a6200640 1 Processor -- start 2024-04-16T16:21:38.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.475+0000 7f79a6200640 1 -- start start 2024-04-16T16:21:38.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.477+0000 7f79a6200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f79a00c6fc0 0x7f79a006f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.477+0000 7f79a6200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79a015cdc0 0x7f79a00c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.477+0000 7f79a6200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f79a00c6320 0x7f79a006df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.477+0000 7f79a6200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f79a00c5670 con 0x7f79a00c6320 2024-04-16T16:21:38.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.477+0000 7f79a6200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f79a015d600 con 0x7f79a00c6fc0 2024-04-16T16:21:38.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.477+0000 7f79a6200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f79a015d8e0 con 0x7f79a015cdc0 2024-04-16T16:21:38.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.477+0000 7f79a5800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f79a00c6320 0x7f79a006df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.477+0000 7f79a5800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f79a00c6320 0x7f79a006df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60156/0 (socket says 172.21.15.45:60156) 2024-04-16T16:21:38.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.477+0000 7f79a5800640 1 -- 172.21.15.45:0/2859620166 learned_addr learned my addr 172.21.15.45:0/2859620166 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:38.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.478+0000 7f79a4e00640 1 --2- 172.21.15.45:0/2859620166 >> v2:172.21.15.103:3300/0 conn(0x7f79a00c6fc0 0x7f79a006f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.478+0000 7f799fe00640 1 --2- 172.21.15.45:0/2859620166 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79a015cdc0 0x7f79a00c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.478+0000 7f79a5800640 1 -- 172.21.15.45:0/2859620166 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79a015cdc0 msgr2=0x7f79a00c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:38.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.478+0000 7f79a5800640 1 --2- 172.21.15.45:0/2859620166 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79a015cdc0 0x7f79a00c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.478+0000 7f79a5800640 1 -- 172.21.15.45:0/2859620166 >> v2:172.21.15.103:3300/0 conn(0x7f79a00c6fc0 msgr2=0x7f79a006f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:38.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.478+0000 7f79a5800640 1 --2- 172.21.15.45:0/2859620166 >> v2:172.21.15.103:3300/0 conn(0x7f79a00c6fc0 0x7f79a006f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.478+0000 7f79a5800640 1 -- 172.21.15.45:0/2859620166 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a015dbc0 con 0x7f79a00c6320 2024-04-16T16:21:38.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.479+0000 7f79a5800640 1 --2- 172.21.15.45:0/2859620166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a00c6320 0x7f79a006df40 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f79a00700f0 tx=0x7f799802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4886653b3ca7c50b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:38.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.479+0000 7f799f400640 1 -- 172.21.15.45:0/2859620166 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799802ca80 con 0x7f79a00c6320 2024-04-16T16:21:38.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.479+0000 7f799f400640 1 -- 172.21.15.45:0/2859620166 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f799802cc20 con 0x7f79a00c6320 2024-04-16T16:21:38.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.479+0000 7f799f400640 1 -- 172.21.15.45:0/2859620166 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79980343f0 con 0x7f79a00c6320 2024-04-16T16:21:38.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.480+0000 7f79a6200640 1 -- 172.21.15.45:0/2859620166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a00c6320 msgr2=0x7f79a006df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:38.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.480+0000 7f79a6200640 1 --2- 172.21.15.45:0/2859620166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a00c6320 0x7f79a006df40 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f79a00700f0 tx=0x7f799802bee0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.480+0000 7f79a6200640 1 -- 172.21.15.45:0/2859620166 shutdown_connections 2024-04-16T16:21:38.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.480+0000 7f79a6200640 1 --2- 172.21.15.45:0/2859620166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a00c6320 0x7f79a006df40 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.480+0000 7f79a6200640 1 --2- 172.21.15.45:0/2859620166 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79a015cdc0 0x7f79a00c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.480+0000 7f79a6200640 1 --2- 172.21.15.45:0/2859620166 >> v2:172.21.15.103:3300/0 conn(0x7f79a00c6fc0 0x7f79a006f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.480+0000 7f79a6200640 1 -- 172.21.15.45:0/2859620166 >> 172.21.15.45:0/2859620166 conn(0x7f79a006b930 msgr2=0x7f79a006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:38.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.480+0000 7f79a6200640 1 -- 172.21.15.45:0/2859620166 shutdown_connections 2024-04-16T16:21:38.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.480+0000 7f79a6200640 1 -- 172.21.15.45:0/2859620166 wait complete. 2024-04-16T16:21:38.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.482+0000 7f79a6200640 1 Processor -- start 2024-04-16T16:21:38.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.482+0000 7f79a6200640 1 -- start start 2024-04-16T16:21:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.483+0000 7f79a6200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79a00c6320 0x7f79a015c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.483+0000 7f79a6200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79a00c6fc0 0x7f79a015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.483+0000 7f79a6200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a015cdc0 0x7f79a0157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.483+0000 7f79a6200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f79a00c73a0 con 0x7f79a015cdc0 2024-04-16T16:21:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.483+0000 7f79a6200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79a015d270 con 0x7f79a00c6320 2024-04-16T16:21:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.483+0000 7f79a6200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f79a00c7520 con 0x7f79a00c6fc0 2024-04-16T16:21:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.484+0000 7f79a5800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a015cdc0 0x7f79a0157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.484+0000 7f79a5800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a015cdc0 0x7f79a0157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60170/0 (socket says 172.21.15.45:60170) 2024-04-16T16:21:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.484+0000 7f79a5800640 1 -- 172.21.15.45:0/2952003521 learned_addr learned my addr 172.21.15.45:0/2952003521 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.484+0000 7f79a4e00640 1 --2- 172.21.15.45:0/2952003521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79a00c6320 0x7f79a015c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.484+0000 7f799fe00640 1 --2- 172.21.15.45:0/2952003521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79a00c6fc0 0x7f79a015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.484+0000 7f79a5800640 1 -- 172.21.15.45:0/2952003521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79a00c6fc0 msgr2=0x7f79a015a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.484+0000 7f79a5800640 1 --2- 172.21.15.45:0/2952003521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79a00c6fc0 0x7f79a015a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.484+0000 7f79a5800640 1 -- 172.21.15.45:0/2952003521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79a00c6320 msgr2=0x7f79a015c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:38.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.484+0000 7f79a5800640 1 --2- 172.21.15.45:0/2952003521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79a00c6320 0x7f79a015c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.484+0000 7f79a5800640 1 -- 172.21.15.45:0/2952003521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a015b000 con 0x7f79a015cdc0 2024-04-16T16:21:38.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.485+0000 7f79a5800640 1 --2- 172.21.15.45:0/2952003521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a015cdc0 0x7f79a0157ca0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f7998036040 tx=0x7f7998004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:38.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.485+0000 7f799cc00640 1 -- 172.21.15.45:0/2952003521 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7998043020 con 0x7f79a015cdc0 2024-04-16T16:21:38.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.485+0000 7f799cc00640 1 -- 172.21.15.45:0/2952003521 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f799803e450 con 0x7f79a015cdc0 2024-04-16T16:21:38.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.485+0000 7f799cc00640 1 -- 172.21.15.45:0/2952003521 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799803e820 con 0x7f79a015cdc0 2024-04-16T16:21:38.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.485+0000 7f79a6200640 1 -- 172.21.15.45:0/2952003521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79a015b2e0 con 0x7f79a015cdc0 2024-04-16T16:21:38.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.485+0000 7f79a6200640 1 -- 172.21.15.45:0/2952003521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a015b8a0 con 0x7f79a015cdc0 2024-04-16T16:21:38.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.487+0000 7f799cc00640 1 -- 172.21.15.45:0/2952003521 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f7998005d10 con 0x7f79a015cdc0 2024-04-16T16:21:38.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.489+0000 7f799cc00640 1 --2- 172.21.15.45:0/2952003521 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f798406ca10 0x7f798406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.489+0000 7f799cc00640 1 -- 172.21.15.45:0/2952003521 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7f79980ae7d0 con 0x7f79a015cdc0 2024-04-16T16:21:38.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.490+0000 7f79aa45f740 1 Processor -- start 2024-04-16T16:21:38.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.490+0000 7f79a4e00640 1 --2- 172.21.15.45:0/2952003521 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f798406ca10 0x7f798406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.490+0000 7f79aa45f740 1 -- start start 2024-04-16T16:21:38.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.491+0000 7f79aa45f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5b50200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.491+0000 7f79aa45f740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b4b8e0 0x55c8b5b52750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.491+0000 7f79aa45f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b509f0 0x55c8b5b50e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.491+0000 7f79aa45f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c8b5ab69c0 con 0x55c8b5b78f20 2024-04-16T16:21:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.491+0000 7f79aa45f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c8b5a5d560 con 0x55c8b5b509f0 2024-04-16T16:21:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.491+0000 7f79aa45f740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c8b5ab2980 con 0x55c8b5b4b8e0 2024-04-16T16:21:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.491+0000 7f79a4e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5b50200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.491+0000 7f79a4e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5b50200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60178/0 (socket says 172.21.15.45:60178) 2024-04-16T16:21:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.491+0000 7f79a4e00640 1 -- 172.21.15.45:0/2086897623 learned_addr learned my addr 172.21.15.45:0/2086897623 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.491+0000 7f79a4e00640 1 --2- 172.21.15.45:0/2952003521 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f798406ca10 0x7f798406eed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7994006fd0 tx=0x7f7994008040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.492+0000 7f79a5800640 1 --2- 172.21.15.45:0/2086897623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b509f0 0x55c8b5b50e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.492+0000 7f799fe00640 1 --2- 172.21.15.45:0/2086897623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b4b8e0 0x55c8b5b52750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.492+0000 7f79a4e00640 1 -- 172.21.15.45:0/2086897623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b4b8e0 msgr2=0x55c8b5b52750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.492+0000 7f79a4e00640 1 --2- 172.21.15.45:0/2086897623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b4b8e0 0x55c8b5b52750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.492+0000 7f79a4e00640 1 -- 172.21.15.45:0/2086897623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b509f0 msgr2=0x55c8b5b50e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.492+0000 7f79a4e00640 1 --2- 172.21.15.45:0/2086897623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b509f0 0x55c8b5b50e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.492+0000 7f79a4e00640 1 -- 172.21.15.45:0/2086897623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8b57c92d0 con 0x55c8b5b78f20 2024-04-16T16:21:38.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.492+0000 7f79a4e00640 1 --2- 172.21.15.45:0/2086897623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5b50200 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f799400fb10 tx=0x7f7994012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec349f8334d9e94a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:38.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.492+0000 7f798d600640 1 -- 172.21.15.45:0/2086897623 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799401b050 con 0x55c8b5b78f20 2024-04-16T16:21:38.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.493+0000 7f798d600640 1 -- 172.21.15.45:0/2086897623 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7994025070 con 0x55c8b5b78f20 2024-04-16T16:21:38.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.493+0000 7f798d600640 1 -- 172.21.15.45:0/2086897623 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7994020440 con 0x55c8b5b78f20 2024-04-16T16:21:38.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.493+0000 7f79aa45f740 1 -- 172.21.15.45:0/2086897623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 msgr2=0x55c8b5b50200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:38.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.493+0000 7f79aa45f740 1 --2- 172.21.15.45:0/2086897623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5b50200 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f799400fb10 tx=0x7f7994012000 comp rx=0 tx=0).stop 2024-04-16T16:21:38.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.493+0000 7f79aa45f740 1 -- 172.21.15.45:0/2086897623 shutdown_connections 2024-04-16T16:21:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.493+0000 7f79aa45f740 1 --2- 172.21.15.45:0/2086897623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b509f0 0x55c8b5b50e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.493+0000 7f79aa45f740 1 --2- 172.21.15.45:0/2086897623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b4b8e0 0x55c8b5b52750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.493+0000 7f79aa45f740 1 --2- 172.21.15.45:0/2086897623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5b50200 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.493+0000 7f79aa45f740 1 -- 172.21.15.45:0/2086897623 >> 172.21.15.45:0/2086897623 conn(0x55c8b5b4db50 msgr2=0x55c8b5b966d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.493+0000 7f79aa45f740 1 -- 172.21.15.45:0/2086897623 shutdown_connections 2024-04-16T16:21:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.493+0000 7f79aa45f740 1 -- 172.21.15.45:0/2086897623 wait complete. 2024-04-16T16:21:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.494+0000 7f79aa45f740 1 -- start start 2024-04-16T16:21:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.494+0000 7f79aa45f740 1 Processor -- start 2024-04-16T16:21:38.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.495+0000 7f79aa45f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b509f0 0x55c8b5b786f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.495+0000 7f79aa45f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5b79680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.495+0000 7f79aa45f740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b79bc0 0x55c8b5b8bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.495+0000 7f79aa45f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c8b5a70920 con 0x55c8b5b78f20 2024-04-16T16:21:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.495+0000 7f79aa45f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c8b5b4bf70 con 0x55c8b5b509f0 2024-04-16T16:21:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.495+0000 7f79aa45f740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c8b5b7bb70 con 0x55c8b5b79bc0 2024-04-16T16:21:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.495+0000 7f79aa45f740 10 client.0 _ll_register_callbacks cb 0x55c8b5a70310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:21:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.496+0000 7f799fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5b79680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.496+0000 7f799fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5b79680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60194/0 (socket says 172.21.15.45:60194) 2024-04-16T16:21:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.496+0000 7f799fe00640 1 -- 172.21.15.45:0/1370871862 learned_addr learned my addr 172.21.15.45:0/1370871862 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.496+0000 7f79a4e00640 1 --2- 172.21.15.45:0/1370871862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b509f0 0x55c8b5b786f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.496+0000 7f79a5800640 1 --2- 172.21.15.45:0/1370871862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b79bc0 0x55c8b5b8bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.496+0000 7f799fe00640 1 -- 172.21.15.45:0/1370871862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b79bc0 msgr2=0x55c8b5b8bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.496+0000 7f799fe00640 1 --2- 172.21.15.45:0/1370871862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b79bc0 0x55c8b5b8bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.496+0000 7f799fe00640 1 -- 172.21.15.45:0/1370871862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b509f0 msgr2=0x55c8b5b786f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.496+0000 7f799fe00640 1 --2- 172.21.15.45:0/1370871862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b509f0 0x55c8b5b786f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.496+0000 7f799fe00640 1 -- 172.21.15.45:0/1370871862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8b58936b0 con 0x55c8b5b78f20 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.496+0000 7f79a5800640 1 --2- 172.21.15.45:0/1370871862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b79bc0 0x55c8b5b8bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.497+0000 7f799fe00640 1 --2- 172.21.15.45:0/1370871862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5b79680 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f79a0150ac0 tx=0x7f799000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afcb9fa4038fd2eb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.497+0000 7f7982000640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.497+0000 7f7982000640 1 -- 172.21.15.45:0/1370871862 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799000f040 con 0x55c8b5b78f20 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.497+0000 7f7982000640 1 -- 172.21.15.45:0/1370871862 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7f7990004230 con 0x55c8b5b78f20 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.497+0000 7f79aa45f740 1 -- 172.21.15.45:0/1370871862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7994025070 con 0x55c8b5b78f20 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.497+0000 7f7982000640 1 -- 172.21.15.45:0/1370871862 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f7990023050 con 0x55c8b5b78f20 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.497+0000 7f79aa45f740 10 client.9333 fetch_fsmap learned FSMap version 100 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.497+0000 7f79aa45f740 1 -- 172.21.15.45:0/1370871862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) -- 0x55c8b5b7b8b0 con 0x55c8b5b78f20 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.497+0000 7f7982000640 1 -- 172.21.15.45:0/1370871862 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7994025070 con 0x55c8b5b78f20 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.498+0000 7f7982000640 1 -- 172.21.15.45:0/1370871862 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 100) ==== 3494+0+0 (secure 0 0 0) 0x7f7990026020 con 0x55c8b5b78f20 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.498+0000 7f79aa45f740 10 client.9333 fetch_fsmap finished waiting for FSMap version 100 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.498+0000 7f79aa45f740 10 client.9333 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.498+0000 7f79aa45f740 20 client.9333 populate_metadata read hostname 'smithi045' 2024-04-16T16:21:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.498+0000 7f79aa45f740 1 --2- 172.21.15.45:0/1370871862 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55c8b5b7b250 0x55c8b5bbdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.498+0000 7f79aa45f740 4 client.9333 mds_command: new command op to 6596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-16T16:21:38.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.498+0000 7f79aa45f740 1 -- 172.21.15.45:0/1370871862 --> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8b5a5cb70 con 0x55c8b5b7b250 2024-04-16T16:21:38.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.498+0000 7f79a4e00640 1 --2- 172.21.15.45:0/1370871862 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55c8b5b7b250 0x55c8b5bbdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.500+0000 7f79a4e00640 1 --2- 172.21.15.45:0/1370871862 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55c8b5b7b250 0x55c8b5bbdff0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40d8da8d36869cdf server_cookie=f54469b7e7e384e8 in_seq=0 out_seq=0 2024-04-16T16:21:38.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.500+0000 7f7982000640 10 client.9333 ms_handle_connect on v2:172.21.15.103:6836/3252460318 2024-04-16T16:21:38.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.502+0000 7f7982000640 1 -- 172.21.15.45:0/1370871862 <== mds.0 v2:172.21.15.103:6836/3252460318 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c8b5a5cb70 con 0x55c8b5b7b250 2024-04-16T16:21:38.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.502+0000 7f7982000640 10 client.9333 handle_command_reply: tid=0 2024-04-16T16:21:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.502+0000 7f79aa45f740 1 client.9333 shutdown 2024-04-16T16:21:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.502+0000 7f79aa45f740 1 -- 172.21.15.45:0/1370871862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 msgr2=0x55c8b5b79680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.502+0000 7f79aa45f740 1 --2- 172.21.15.45:0/1370871862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5b79680 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f79a0150ac0 tx=0x7f799000ef90 comp rx=0 tx=0).stop 2024-04-16T16:21:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.503+0000 7f79aa45f740 1 -- 172.21.15.45:0/1370871862 shutdown_connections 2024-04-16T16:21:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.503+0000 7f79aa45f740 1 --2- 172.21.15.45:0/1370871862 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55c8b5b7b250 0x55c8b5bbdff0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.503+0000 7f79aa45f740 1 --2- 172.21.15.45:0/1370871862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b79bc0 0x55c8b5b8bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.503+0000 7f79aa45f740 1 --2- 172.21.15.45:0/1370871862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5b79680 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.503+0000 7f79aa45f740 1 --2- 172.21.15.45:0/1370871862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b509f0 0x55c8b5b786f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.503+0000 7f7982000640 0 client.9333 ms_handle_reset on v2:172.21.15.103:6836/3252460318 2024-04-16T16:21:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.503+0000 7f79aa45f740 1 -- 172.21.15.45:0/1370871862 >> 172.21.15.45:0/1370871862 conn(0x55c8b5b4b8e0 msgr2=0x55c8b5b4b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:38.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.503+0000 7f79aa45f740 1 -- 172.21.15.45:0/1370871862 shutdown_connections 2024-04-16T16:21:38.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.503+0000 7f79aa45f740 1 -- 172.21.15.45:0/1370871862 wait complete. 2024-04-16T16:21:38.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.503+0000 7f79aa45f740 20 client.9333 trim_cache size 0 max 16384 2024-04-16T16:21:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.523+0000 7f79aa45f740 1 Processor -- start 2024-04-16T16:21:38.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.523+0000 7f79aa45f740 1 -- start start 2024-04-16T16:21:38.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.524+0000 7f79aa45f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b509f0 0x55c8b5ba8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.524+0000 7f79aa45f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5bc3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.524+0000 7f79aa45f740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b79bc0 0x55c8b5bc3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.524+0000 7f79aa45f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f79a00ca050 con 0x55c8b5b78f20 2024-04-16T16:21:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.524+0000 7f79aa45f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01f4810 con 0x55c8b5b509f0 2024-04-16T16:21:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.524+0000 7f79aa45f740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f79a01f4e60 con 0x55c8b5b79bc0 2024-04-16T16:21:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.524+0000 7f799fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5bc3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.524+0000 7f799fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5bc3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60198/0 (socket says 172.21.15.45:60198) 2024-04-16T16:21:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.524+0000 7f799fe00640 1 -- 172.21.15.45:0/470319982 learned_addr learned my addr 172.21.15.45:0/470319982 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.524+0000 7f79a4e00640 1 --2- 172.21.15.45:0/470319982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b509f0 0x55c8b5ba8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.525+0000 7f79a5800640 1 --2- 172.21.15.45:0/470319982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b79bc0 0x55c8b5bc3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.525+0000 7f799fe00640 1 -- 172.21.15.45:0/470319982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b79bc0 msgr2=0x55c8b5bc3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.525+0000 7f799fe00640 1 --2- 172.21.15.45:0/470319982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b79bc0 0x55c8b5bc3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.525+0000 7f799fe00640 1 -- 172.21.15.45:0/470319982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b509f0 msgr2=0x55c8b5ba8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.525+0000 7f799fe00640 1 --2- 172.21.15.45:0/470319982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b509f0 0x55c8b5ba8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.525+0000 7f799fe00640 1 -- 172.21.15.45:0/470319982 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8b5b7af00 con 0x55c8b5b78f20 2024-04-16T16:21:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.525+0000 7f799fe00640 1 --2- 172.21.15.45:0/470319982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5bc3400 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f7990000c00 tx=0x7f7990016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8177c02db3f725ff server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.526+0000 7f798cc00640 1 -- 172.21.15.45:0/470319982 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799002b070 con 0x55c8b5b78f20 2024-04-16T16:21:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.526+0000 7f798cc00640 1 -- 172.21.15.45:0/470319982 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f799001b070 con 0x55c8b5b78f20 2024-04-16T16:21:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.526+0000 7f798cc00640 1 -- 172.21.15.45:0/470319982 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7990003ba0 con 0x55c8b5b78f20 2024-04-16T16:21:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.526+0000 7f79aa45f740 1 -- 172.21.15.45:0/470319982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 msgr2=0x55c8b5bc3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.526+0000 7f79aa45f740 1 --2- 172.21.15.45:0/470319982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5bc3400 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f7990000c00 tx=0x7f7990016000 comp rx=0 tx=0).stop 2024-04-16T16:21:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.526+0000 7f79aa45f740 1 -- 172.21.15.45:0/470319982 shutdown_connections 2024-04-16T16:21:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.526+0000 7f79aa45f740 1 --2- 172.21.15.45:0/470319982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b79bc0 0x55c8b5bc3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.526+0000 7f79aa45f740 1 --2- 172.21.15.45:0/470319982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b78f20 0x55c8b5bc3400 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.526+0000 7f79aa45f740 1 --2- 172.21.15.45:0/470319982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b509f0 0x55c8b5ba8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.526+0000 7f79aa45f740 1 -- 172.21.15.45:0/470319982 >> 172.21.15.45:0/470319982 conn(0x55c8b5b4b8e0 msgr2=0x55c8b5b7a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.526+0000 7f79aa45f740 1 -- 172.21.15.45:0/470319982 shutdown_connections 2024-04-16T16:21:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.526+0000 7f79aa45f740 1 -- 172.21.15.45:0/470319982 wait complete. 2024-04-16T16:21:38.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.527+0000 7f79aa45f740 1 -- start start 2024-04-16T16:21:38.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.527+0000 7f79aa45f740 1 Processor -- start 2024-04-16T16:21:38.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.528+0000 7f79aa45f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b509f0 0x55c8b5c18890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.528+0000 7f79aa45f740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b78f20 0x55c8b5b8a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.528+0000 7f79aa45f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b79bc0 0x55c8b5b79fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.528+0000 7f79aa45f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c8b5b7b0d0 con 0x55c8b5b509f0 2024-04-16T16:21:38.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.528+0000 7f79aa45f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c8b5b7abe0 con 0x55c8b5b79bc0 2024-04-16T16:21:38.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.528+0000 7f79aa45f740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c8b5bb2b80 con 0x55c8b5b78f20 2024-04-16T16:21:38.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.528+0000 7f79aa45f740 10 client.0 _ll_register_callbacks cb 0x7f799000eb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:21:38.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.528+0000 7f79a4e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b509f0 0x55c8b5c18890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.529+0000 7f79a4e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b509f0 0x55c8b5c18890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60214/0 (socket says 172.21.15.45:60214) 2024-04-16T16:21:38.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.529+0000 7f79a4e00640 1 -- 172.21.15.45:0/3823617173 learned_addr learned my addr 172.21.15.45:0/3823617173 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:38.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.529+0000 7f79a4e00640 1 -- 172.21.15.45:0/3823617173 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b78f20 msgr2=0x55c8b5b8a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:38.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.529+0000 7f799fe00640 1 --2- 172.21.15.45:0/3823617173 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b78f20 0x55c8b5b8a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.529+0000 7f79a4e00640 1 --2- 172.21.15.45:0/3823617173 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b78f20 0x55c8b5b8a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.529+0000 7f79a5800640 1 --2- 172.21.15.45:0/3823617173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b79bc0 0x55c8b5b79fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.529+0000 7f79a4e00640 1 -- 172.21.15.45:0/3823617173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b79bc0 msgr2=0x55c8b5b79fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:38.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.529+0000 7f79a4e00640 1 --2- 172.21.15.45:0/3823617173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b79bc0 0x55c8b5b79fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.529+0000 7f79a4e00640 1 -- 172.21.15.45:0/3823617173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8b5ba8d80 con 0x55c8b5b509f0 2024-04-16T16:21:38.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.529+0000 7f79a5800640 1 --2- 172.21.15.45:0/3823617173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b79bc0 0x55c8b5b79fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:38.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.529+0000 7f79a4e00640 1 --2- 172.21.15.45:0/3823617173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b509f0 0x55c8b5c18890 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f7994008010 tx=0x7f7994015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ab5ade690becec server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:38.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.529+0000 7f7982000640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:21:38.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.530+0000 7f7982000640 1 -- 172.21.15.45:0/3823617173 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799403c060 con 0x55c8b5b509f0 2024-04-16T16:21:38.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.530+0000 7f7982000640 1 -- 172.21.15.45:0/3823617173 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7f7994037840 con 0x55c8b5b509f0 2024-04-16T16:21:38.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.530+0000 7f79aa45f740 1 -- 172.21.15.45:0/3823617173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f799001b070 con 0x55c8b5b509f0 2024-04-16T16:21:38.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.530+0000 7f7982000640 1 -- 172.21.15.45:0/3823617173 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f7994042050 con 0x55c8b5b509f0 2024-04-16T16:21:38.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.530+0000 7f79aa45f740 10 client.9339 fetch_fsmap learned FSMap version 100 2024-04-16T16:21:38.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.530+0000 7f79aa45f740 1 -- 172.21.15.45:0/3823617173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) -- 0x55c8b5b50220 con 0x55c8b5b509f0 2024-04-16T16:21:38.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.530+0000 7f7982000640 1 -- 172.21.15.45:0/3823617173 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f799001b070 con 0x55c8b5b509f0 2024-04-16T16:21:38.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.530+0000 7f7982000640 1 -- 172.21.15.45:0/3823617173 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 100) ==== 3494+0+0 (secure 0 0 0) 0x7f7994045020 con 0x55c8b5b509f0 2024-04-16T16:21:38.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.530+0000 7f79aa45f740 10 client.9339 fetch_fsmap finished waiting for FSMap version 100 2024-04-16T16:21:38.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.530+0000 7f79aa45f740 10 client.9339 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-16T16:21:38.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.530+0000 7f79aa45f740 20 client.9339 populate_metadata read hostname 'smithi045' 2024-04-16T16:21:38.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.530+0000 7f79aa45f740 1 --2- 172.21.15.45:0/3823617173 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55c8b5b77380 0x55c8b5b77760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.530+0000 7f79aa45f740 4 client.9339 mds_command: new command op to 6596 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-16T16:21:38.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.530+0000 7f79aa45f740 1 -- 172.21.15.45:0/3823617173 --> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] -- command(tid 0: {"prefix": "status"}) -- 0x55c8b5ac43e0 con 0x55c8b5b77380 2024-04-16T16:21:38.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.531+0000 7f799fe00640 1 --2- 172.21.15.45:0/3823617173 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55c8b5b77380 0x55c8b5b77760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.532+0000 7f799fe00640 1 --2- 172.21.15.45:0/3823617173 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55c8b5b77380 0x55c8b5b77760 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=587d6db4c366b6b0 server_cookie=3dd2b5f232047250 in_seq=0 out_seq=0 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.532+0000 7f7982000640 10 client.9339 ms_handle_connect on v2:172.21.15.103:6836/3252460318 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.532+0000 7f7982000640 1 -- 172.21.15.45:0/3823617173 <== mds.0 v2:172.21.15.103:6836/3252460318 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55c8b5ac43e0 con 0x55c8b5b77380 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.532+0000 7f7982000640 10 client.9339 handle_command_reply: tid=0 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.532+0000 7f79aa45f740 1 client.9339 shutdown 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.533+0000 7f79aa45f740 1 -- 172.21.15.45:0/3823617173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b509f0 msgr2=0x55c8b5c18890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.533+0000 7f79aa45f740 1 --2- 172.21.15.45:0/3823617173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b509f0 0x55c8b5c18890 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f7994008010 tx=0x7f7994015000 comp rx=0 tx=0).stop 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.533+0000 7f79aa45f740 1 -- 172.21.15.45:0/3823617173 shutdown_connections 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.533+0000 7f79aa45f740 1 --2- 172.21.15.45:0/3823617173 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55c8b5b77380 0x55c8b5b77760 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.533+0000 7f79aa45f740 1 --2- 172.21.15.45:0/3823617173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c8b5b79bc0 0x55c8b5b79fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.533+0000 7f79aa45f740 1 --2- 172.21.15.45:0/3823617173 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c8b5b78f20 0x55c8b5b8a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.533+0000 7f79aa45f740 1 --2- 172.21.15.45:0/3823617173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c8b5b509f0 0x55c8b5c18890 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.533+0000 7f7982000640 0 client.9339 ms_handle_reset on v2:172.21.15.103:6836/3252460318 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.533+0000 7f79aa45f740 1 -- 172.21.15.45:0/3823617173 >> 172.21.15.45:0/3823617173 conn(0x55c8b5b4b8e0 msgr2=0x55c8b5bafc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.533+0000 7f79aa45f740 1 -- 172.21.15.45:0/3823617173 shutdown_connections 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.533+0000 7f79aa45f740 1 -- 172.21.15.45:0/3823617173 wait complete. 2024-04-16T16:21:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.533+0000 7f79aa45f740 20 client.9339 trim_cache size 0 max 16384 2024-04-16T16:21:38.534 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:21:38.534 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "748c4b85-0190-4c59-80ad-098a6a5007c0", 2024-04-16T16:21:38.534 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-04-16T16:21:38.534 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 6596, 2024-04-16T16:21:38.534 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-04-16T16:21:38.534 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-04-16T16:21:38.534 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-04-16T16:21:38.534 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 11.939843944, 2024-04-16T16:21:38.534 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 100, 2024-04-16T16:21:38.534 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 142, 2024-04-16T16:21:38.535 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 142, 2024-04-16T16:21:38.535 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 18.256099143 2024-04-16T16:21:38.535 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:21:38.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.534+0000 7f79a6200640 1 -- 172.21.15.45:0/2952003521 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f798406ca10 msgr2=0x7f798406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:38.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.534+0000 7f79a6200640 1 --2- 172.21.15.45:0/2952003521 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f798406ca10 0x7f798406eed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7994006fd0 tx=0x7f7994008040 comp rx=0 tx=0).stop 2024-04-16T16:21:38.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.534+0000 7f79a6200640 1 -- 172.21.15.45:0/2952003521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a015cdc0 msgr2=0x7f79a0157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:38.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.534+0000 7f79a6200640 1 --2- 172.21.15.45:0/2952003521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a015cdc0 0x7f79a0157ca0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f7998036040 tx=0x7f7998004a40 comp rx=0 tx=0).stop 2024-04-16T16:21:38.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.534+0000 7f79a6200640 1 -- 172.21.15.45:0/2952003521 shutdown_connections 2024-04-16T16:21:38.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.534+0000 7f79a6200640 1 --2- 172.21.15.45:0/2952003521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a015cdc0 0x7f79a0157ca0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.534+0000 7f79a6200640 1 --2- 172.21.15.45:0/2952003521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79a00c6fc0 0x7f79a015a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.534+0000 7f79a6200640 1 --2- 172.21.15.45:0/2952003521 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f798406ca10 0x7f798406eed0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.534+0000 7f79a6200640 1 --2- 172.21.15.45:0/2952003521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79a00c6320 0x7f79a015c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.534+0000 7f79a6200640 1 -- 172.21.15.45:0/2952003521 >> 172.21.15.45:0/2952003521 conn(0x7f79a006b930 msgr2=0x7f79a015ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:38.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.534+0000 7f79a6200640 1 -- 172.21.15.45:0/2952003521 shutdown_connections 2024-04-16T16:21:38.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.534+0000 7f79a6200640 1 -- 172.21.15.45:0/2952003521 wait complete. 2024-04-16T16:21:38.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:21:38.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.771+0000 7f8ecf200640 1 Processor -- start 2024-04-16T16:21:38.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.772+0000 7f8ecf200640 1 -- start start 2024-04-16T16:21:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.773+0000 7f8ecf200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8ed0161e90 0x7f8ed00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.773+0000 7f8ecf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8ed00d52d0 0x7f8ed006ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.773+0000 7f8ecf200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ed00d3080 con 0x7f8ed0162270 2024-04-16T16:21:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.773+0000 7f8ecf200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8ed00d2e30 con 0x7f8ed0161e90 2024-04-16T16:21:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.773+0000 7f8ecf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8ed00d3360 con 0x7f8ed00d52d0 2024-04-16T16:21:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.773+0000 7f8ecde00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8ed0161e90 0x7f8ed00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.773+0000 7f8ecde00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8ed0161e90 0x7f8ed00cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60216/0 (socket says 172.21.15.45:60216) 2024-04-16T16:21:38.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.773+0000 7f8ecde00640 1 -- 172.21.15.45:0/2258292797 learned_addr learned my addr 172.21.15.45:0/2258292797 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:38.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.774+0000 7f8ece800640 1 --2- 172.21.15.45:0/2258292797 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8ed00d52d0 0x7f8ed006ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.774+0000 7f8ecde00640 1 -- 172.21.15.45:0/2258292797 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8ed00d52d0 msgr2=0x7f8ed006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:38.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.774+0000 7f8ecde00640 1 --2- 172.21.15.45:0/2258292797 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8ed00d52d0 0x7f8ed006ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.774+0000 7f8ecde00640 1 -- 172.21.15.45:0/2258292797 >> v1:172.21.15.103:6789/0 conn(0x7f8ed0162270 legacy=0x7f8ed00d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:38.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.774+0000 7f8ecde00640 1 -- 172.21.15.45:0/2258292797 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed00d3640 con 0x7f8ed0161e90 2024-04-16T16:21:38.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.775+0000 7f8ecde00640 1 --2- 172.21.15.45:0/2258292797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ed0161e90 0x7f8ed00cf7d0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f8ec00098a0 tx=0x7f8ec002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2920af1f9d106b17 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:38.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.775+0000 7f8ecd400640 1 -- 172.21.15.45:0/2258292797 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec002f830 con 0x7f8ed0161e90 2024-04-16T16:21:38.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.775+0000 7f8ecd400640 1 -- 172.21.15.45:0/2258292797 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ec002f9d0 con 0x7f8ed0161e90 2024-04-16T16:21:38.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.775+0000 7f8ecd400640 1 -- 172.21.15.45:0/2258292797 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec002fda0 con 0x7f8ed0161e90 2024-04-16T16:21:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.776+0000 7f8ecf200640 1 -- 172.21.15.45:0/2258292797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ed0161e90 msgr2=0x7f8ed00cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.776+0000 7f8ecf200640 1 --2- 172.21.15.45:0/2258292797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ed0161e90 0x7f8ed00cf7d0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f8ec00098a0 tx=0x7f8ec002ba10 comp rx=0 tx=0).stop 2024-04-16T16:21:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.776+0000 7f8ecf200640 1 -- 172.21.15.45:0/2258292797 shutdown_connections 2024-04-16T16:21:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.776+0000 7f8ecf200640 1 --2- 172.21.15.45:0/2258292797 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8ed00d52d0 0x7f8ed006ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.776+0000 7f8ecf200640 1 --2- 172.21.15.45:0/2258292797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ed0161e90 0x7f8ed00cf7d0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.776+0000 7f8ecf200640 1 -- 172.21.15.45:0/2258292797 >> 172.21.15.45:0/2258292797 conn(0x7f8ed00d1940 msgr2=0x7f8ed00d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.776+0000 7f8ecf200640 1 -- 172.21.15.45:0/2258292797 shutdown_connections 2024-04-16T16:21:38.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.776+0000 7f8ecf200640 1 -- 172.21.15.45:0/2258292797 wait complete. 2024-04-16T16:21:38.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.778+0000 7f8ecf200640 1 Processor -- start 2024-04-16T16:21:38.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.778+0000 7f8ecf200640 1 -- start start 2024-04-16T16:21:38.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.779+0000 7f8ecf200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ed00d52d0 0x7f8ed00c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.779+0000 7f8ecf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8ed0161e90 0x7f8ed0155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.779+0000 7f8ecf200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ed0162270 0x7f8ed00b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.779+0000 7f8ecf200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed00d4550 con 0x7f8ed00d52d0 2024-04-16T16:21:38.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.779+0000 7f8ecf200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed00cfd10 con 0x7f8ed0162270 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.779+0000 7f8ecf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8ed00d46d0 con 0x7f8ed0161e90 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.779+0000 7f8ecde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ed00d52d0 0x7f8ed00c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.779+0000 7f8ecde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ed00d52d0 0x7f8ed00c9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60228/0 (socket says 172.21.15.45:60228) 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.779+0000 7f8ecde00640 1 -- 172.21.15.45:0/1952993055 learned_addr learned my addr 172.21.15.45:0/1952993055 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.780+0000 7f8ece800640 1 --2- 172.21.15.45:0/1952993055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ed0162270 0x7f8ed00b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.780+0000 7f8ec5400640 1 --2- 172.21.15.45:0/1952993055 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8ed0161e90 0x7f8ed0155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.780+0000 7f8ecde00640 1 -- 172.21.15.45:0/1952993055 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8ed0161e90 msgr2=0x7f8ed0155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.780+0000 7f8ecde00640 1 --2- 172.21.15.45:0/1952993055 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8ed0161e90 0x7f8ed0155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.780+0000 7f8ecde00640 1 -- 172.21.15.45:0/1952993055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ed0162270 msgr2=0x7f8ed00b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.780+0000 7f8ecde00640 1 --2- 172.21.15.45:0/1952993055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ed0162270 0x7f8ed00b56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.780+0000 7f8ecde00640 1 -- 172.21.15.45:0/1952993055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed00cb600 con 0x7f8ed00d52d0 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.780+0000 7f8ec5400640 1 --2- 172.21.15.45:0/1952993055 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8ed0161e90 0x7f8ed0155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.780+0000 7f8ecde00640 1 --2- 172.21.15.45:0/1952993055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ed00d52d0 0x7f8ed00c9e70 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0000c00 tx=0x7f8ec0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.780+0000 7f8ec6a00640 1 -- 172.21.15.45:0/1952993055 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0037d10 con 0x7f8ed00d52d0 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.781+0000 7f8ec6a00640 1 -- 172.21.15.45:0/1952993055 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ec0046070 con 0x7f8ed00d52d0 2024-04-16T16:21:38.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.781+0000 7f8ecf200640 1 -- 172.21.15.45:0/1952993055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ed00ca560 con 0x7f8ed00d52d0 2024-04-16T16:21:38.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.781+0000 7f8ec6a00640 1 -- 172.21.15.45:0/1952993055 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0041640 con 0x7f8ed00d52d0 2024-04-16T16:21:38.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.781+0000 7f8ecf200640 1 -- 172.21.15.45:0/1952993055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ed00cab20 con 0x7f8ed00d52d0 2024-04-16T16:21:38.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.782+0000 7f8ecf200640 1 -- 172.21.15.45:0/1952993055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ed00cf5a0 con 0x7f8ed00d52d0 2024-04-16T16:21:38.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.783+0000 7f8ec6a00640 1 -- 172.21.15.45:0/1952993055 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f8ec0040070 con 0x7f8ed00d52d0 2024-04-16T16:21:38.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.784+0000 7f8ec6a00640 1 --2- 172.21.15.45:0/1952993055 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f8eac06ca10 0x7f8eac06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:38.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.784+0000 7f8ec6a00640 1 -- 172.21.15.45:0/1952993055 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7f8ec00ae4e0 con 0x7f8ed00d52d0 2024-04-16T16:21:38.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.784+0000 7f8ec5400640 1 --2- 172.21.15.45:0/1952993055 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f8eac06ca10 0x7f8eac06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:38.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.786+0000 7f8ec5400640 1 --2- 172.21.15.45:0/1952993055 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f8eac06ca10 0x7f8eac06eed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00c9dc0 tx=0x7f8eb8009040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:38.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:38.788+0000 7f8ec6a00640 1 -- 172.21.15.45:0/1952993055 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8ec0033030 con 0x7f8ed00d52d0 2024-04-16T16:21:39.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39.036+0000 7f8ecf200640 1 -- 172.21.15.45:0/1952993055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f8ed0075780 con 0x7f8ed00d52d0 2024-04-16T16:21:39.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39.036+0000 7f8ec6a00640 1 -- 172.21.15.45:0/1952993055 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v100) ==== 53+0+73 (secure 0 0 0) 0x7f8ec007c6e0 con 0x7f8ed00d52d0 2024-04-16T16:21:39.035 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:21:39.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39.039+0000 7f8ecf200640 1 -- 172.21.15.45:0/1952993055 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f8eac06ca10 msgr2=0x7f8eac06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:39.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39.039+0000 7f8ecf200640 1 --2- 172.21.15.45:0/1952993055 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f8eac06ca10 0x7f8eac06eed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00c9dc0 tx=0x7f8eb8009040 comp rx=0 tx=0).stop 2024-04-16T16:21:39.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39.039+0000 7f8ecf200640 1 -- 172.21.15.45:0/1952993055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ed00d52d0 msgr2=0x7f8ed00c9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:39.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39.039+0000 7f8ecf200640 1 --2- 172.21.15.45:0/1952993055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ed00d52d0 0x7f8ed00c9e70 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0000c00 tx=0x7f8ec0004730 comp rx=0 tx=0).stop 2024-04-16T16:21:39.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39.039+0000 7f8ecf200640 1 -- 172.21.15.45:0/1952993055 shutdown_connections 2024-04-16T16:21:39.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39.039+0000 7f8ecf200640 1 --2- 172.21.15.45:0/1952993055 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f8eac06ca10 0x7f8eac06eed0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:39.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39.040+0000 7f8ecf200640 1 --2- 172.21.15.45:0/1952993055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ed0162270 0x7f8ed00b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:39.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39.040+0000 7f8ecf200640 1 --2- 172.21.15.45:0/1952993055 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8ed0161e90 0x7f8ed0155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:39.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39.040+0000 7f8ecf200640 1 --2- 172.21.15.45:0/1952993055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ed00d52d0 0x7f8ed00c9e70 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:39.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39.040+0000 7f8ecf200640 1 -- 172.21.15.45:0/1952993055 >> 172.21.15.45:0/1952993055 conn(0x7f8ed00d1940 msgr2=0x7f8ed00b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:39.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39.040+0000 7f8ecf200640 1 -- 172.21.15.45:0/1952993055 shutdown_connections 2024-04-16T16:21:39.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39.040+0000 7f8ecf200640 1 -- 172.21.15.45:0/1952993055 wait complete. 2024-04-16T16:21:39.058 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-16T16:21:39.058 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:21:39.058 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-04-16T16:21:39.059 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-16T16:21:39.059 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T16:21:39.059 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-16T16:21:39.059 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:21:39.059 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:21:39.059 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:21:39.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:39.059 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-04-16T16:21:39.114 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:21:39.115 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:21:39.115 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:21:39.115 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:21:39.115 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-04-16T16:21:39.115 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:21:39.115 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:21:39.115 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:21:39.115 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-04-16T16:21:39.115 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:21:39.115 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:21:39.115 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-04-16T16:21:39.115 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:21:39.115 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-04-16T16:21:39.116 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-04-16T16:21:39.116 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-16T16:21:39.116 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42102sec preferred_lft 42102sec 2024-04-16T16:21:39.116 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-04-16T16:21:39.116 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:21:39.116 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:21:39.116 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-04-16T16:21:39.116 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-04-16T16:21:39.116 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:21:39.116 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff 2024-04-16T16:21:39.116 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:21:39.116 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:21:39.116 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::50f1:99ff:fe2b:225b/64 scope link 2024-04-16T16:21:39.116 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:21:39.117 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:21:39.117 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 72:07:c3:13:7b:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T16:21:39.117 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::7007:c3ff:fe13:7bcd/64 scope link 2024-04-16T16:21:39.117 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:21:39.117 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:21:39.117 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether ae:10:ea:fc:0a:13 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-16T16:21:39.117 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ac10:eaff:fefc:a13/64 scope link 2024-04-16T16:21:39.117 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:21:39.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:39.118 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:21:39.212 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-04-16T16:21:39.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:39.217 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-04-16T16:21:39.273 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-04-16T16:21:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:21:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-16T16:21:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-16T16:21:39.274 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:21:39.275 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:21:39.275 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T16:21:39.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-16T16:21:39.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-16T16:21:39.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-16T16:21:39.276 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T16:21:39.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:21:39.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:21:39.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:39.544 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T16:21:39.596 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-16T16:21:39.597 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-16T16:21:39.597 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:21:39.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:39.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-04-16T16:21:39.699 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:21:39.699 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-16T16:21:39.754 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-16T16:21:39.754 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:21:39.754 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:21:39.803 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-16T16:21:39.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:39.803 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:21:39.942 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:21:39.942 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:21:39.978 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:21:39.978 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:21:39.978 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:21:39.978 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 0@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:21:39.979 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:21:39.982 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-16T16:21:39.982 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:21:39.982 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:21:39.982 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:21:39.982 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:21:39.986 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:21:39.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:39.987 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-16T16:21:40.072 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:21:40.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:21:40.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:21:40.223 INFO:teuthology.orchestra.run.smithi045.stdout:client9348 2024-04-16T16:21:40.223 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:21:40.223 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:21:40.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-16T16:21:40.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.498+0000 7f9eb3000640 1 Processor -- start 2024-04-16T16:21:40.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.498+0000 7f9eb3000640 1 -- start start 2024-04-16T16:21:40.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.499+0000 7f9eb3000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9eac15ccb0 0x7f9eac06df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:40.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.500+0000 7f9eb3000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9eac15d360 con 0x7f9eac06e480 2024-04-16T16:21:40.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.500+0000 7f9eb3000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9eac15d640 con 0x7f9eac146220 2024-04-16T16:21:40.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.500+0000 7f9eb3000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9eac15d130 con 0x7f9eac15ccb0 2024-04-16T16:21:40.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.500+0000 7f9eb2600640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9eac06e480 0x7f9eac1608a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55238/0 (socket says 172.21.15.45:55238) 2024-04-16T16:21:40.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.500+0000 7f9eb2600640 1 -- 172.21.15.45:0/50350272 learned_addr learned my addr 172.21.15.45:0/50350272 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:40.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.500+0000 7f9eb1200640 1 --2- 172.21.15.45:0/50350272 >> v2:172.21.15.103:3300/0 conn(0x7f9eac15ccb0 0x7f9eac06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.500+0000 7f9eabe00640 1 -- 172.21.15.45:0/50350272 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 845175057 0 0) 0x7f9eac15d360 con 0x7f9eac06e480 2024-04-16T16:21:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.500+0000 7f9eabe00640 1 -- 172.21.15.45:0/50350272 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e94003520 con 0x7f9eac06e480 2024-04-16T16:21:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.501+0000 7f9eabe00640 1 -- 172.21.15.45:0/50350272 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1046357433 0 0) 0x7f9e94003520 con 0x7f9eac06e480 2024-04-16T16:21:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.501+0000 7f9eabe00640 1 -- 172.21.15.45:0/50350272 >> v2:172.21.15.103:3300/0 conn(0x7f9eac15ccb0 msgr2=0x7f9eac06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.501+0000 7f9eabe00640 1 --2- 172.21.15.45:0/50350272 >> v2:172.21.15.103:3300/0 conn(0x7f9eac15ccb0 0x7f9eac06df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.501+0000 7f9eabe00640 1 -- 172.21.15.45:0/50350272 >> v1:172.21.15.103:6789/0 conn(0x7f9eac146220 legacy=0x7f9eac06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.501+0000 7f9eabe00640 1 -- 172.21.15.45:0/50350272 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eac15d920 con 0x7f9eac06e480 2024-04-16T16:21:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.501+0000 7f9eabe00640 1 -- 172.21.15.45:0/50350272 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f9ea40019f0 con 0x7f9eac06e480 2024-04-16T16:21:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.501+0000 7f9eabe00640 1 -- 172.21.15.45:0/50350272 >> v1:172.21.15.45:6789/0 conn(0x7f9eac06e480 legacy=0x7f9eac1608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.501+0000 7f9eabe00640 1 --2- 172.21.15.45:0/50350272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e940039b0 0x7f9e94003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.501+0000 7f9eabe00640 1 --2- 172.21.15.45:0/50350272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e94004800 0x7f9e94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.501+0000 7f9eabe00640 1 --2- 172.21.15.45:0/50350272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e940051c0 0x7f9e9400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.501+0000 7f9eabe00640 1 -- 172.21.15.45:0/50350272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9400c030 con 0x7f9e940039b0 2024-04-16T16:21:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.501+0000 7f9eabe00640 1 -- 172.21.15.45:0/50350272 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9400c1e0 con 0x7f9e940051c0 2024-04-16T16:21:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.501+0000 7f9eabe00640 1 -- 172.21.15.45:0/50350272 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9e9400c3e0 con 0x7f9e94004800 2024-04-16T16:21:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.501+0000 7f9eb1c00640 1 --2- 172.21.15.45:0/50350272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e940039b0 0x7f9e94003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.502+0000 7f9eb1200640 1 --2- 172.21.15.45:0/50350272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e94004800 0x7f9e94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.502+0000 7f9eb1c00640 1 -- 172.21.15.45:0/50350272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e94004800 msgr2=0x7f9e94004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.502+0000 7f9eb1c00640 1 --2- 172.21.15.45:0/50350272 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e94004800 0x7f9e94004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.502+0000 7f9eb1c00640 1 -- 172.21.15.45:0/50350272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e940051c0 msgr2=0x7f9e9400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.502+0000 7f9eb1c00640 1 --2- 172.21.15.45:0/50350272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e940051c0 0x7f9e9400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.502+0000 7f9eb1c00640 1 -- 172.21.15.45:0/50350272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e94003520 con 0x7f9e940039b0 2024-04-16T16:21:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.502+0000 7f9eb2600640 1 -- 172.21.15.45:0/50350272 reap_dead start 2024-04-16T16:21:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.503+0000 7f9eb1c00640 1 --2- 172.21.15.45:0/50350272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e940039b0 0x7f9e94003dd0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0001750 tx=0x7f9e94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da88a34d1717265 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.503+0000 7f9eabe00640 1 -- 172.21.15.45:0/50350272 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0001370 con 0x7f9e940039b0 2024-04-16T16:21:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.503+0000 7f9eabe00640 1 -- 172.21.15.45:0/50350272 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ea00052d0 con 0x7f9e940039b0 2024-04-16T16:21:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.503+0000 7f9eabe00640 1 -- 172.21.15.45:0/50350272 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0003830 con 0x7f9e940039b0 2024-04-16T16:21:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.504+0000 7f9eb3000640 1 -- 172.21.15.45:0/50350272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e940039b0 msgr2=0x7f9e94003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.504+0000 7f9eb3000640 1 --2- 172.21.15.45:0/50350272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e940039b0 0x7f9e94003dd0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0001750 tx=0x7f9e94005830 comp rx=0 tx=0).stop 2024-04-16T16:21:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.504+0000 7f9eb3000640 1 -- 172.21.15.45:0/50350272 shutdown_connections 2024-04-16T16:21:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.504+0000 7f9eb3000640 1 --2- 172.21.15.45:0/50350272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e940039b0 0x7f9e94003dd0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.504+0000 7f9eb3000640 1 -- 172.21.15.45:0/50350272 >> 172.21.15.45:0/50350272 conn(0x7f9eac06b930 msgr2=0x7f9eac06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.504+0000 7f9eb3000640 1 -- 172.21.15.45:0/50350272 shutdown_connections 2024-04-16T16:21:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.504+0000 7f9eb3000640 1 -- 172.21.15.45:0/50350272 wait complete. 2024-04-16T16:21:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.506+0000 7f9eb3000640 1 Processor -- start 2024-04-16T16:21:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.506+0000 7f9eb3000640 1 -- start start 2024-04-16T16:21:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.507+0000 7f9eb3000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e940039b0 0x7f9eac0cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.507+0000 7f9eb3000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eac1f8710 0x7f9eac0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.507+0000 7f9eb3000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9eac13db00 0x7f9eac13df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.507+0000 7f9eb3000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac06df40 con 0x7f9eac13db00 2024-04-16T16:21:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.507+0000 7f9eb3000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac160410 con 0x7f9eac1f8710 2024-04-16T16:21:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.507+0000 7f9eb3000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9eac06e0c0 con 0x7f9e940039b0 2024-04-16T16:21:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.507+0000 7f9eb2600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9eac13db00 0x7f9eac13df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.507+0000 7f9eb2600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9eac13db00 0x7f9eac13df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60250/0 (socket says 172.21.15.45:60250) 2024-04-16T16:21:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.507+0000 7f9eb2600640 1 -- 172.21.15.45:0/3739976437 learned_addr learned my addr 172.21.15.45:0/3739976437 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:40.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.508+0000 7f9eb1c00640 1 --2- 172.21.15.45:0/3739976437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e940039b0 0x7f9eac0cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.508+0000 7f9eb2600640 1 -- 172.21.15.45:0/3739976437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e940039b0 msgr2=0x7f9eac0cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.508+0000 7f9eb2600640 1 --2- 172.21.15.45:0/3739976437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e940039b0 0x7f9eac0cc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.508+0000 7f9eb2600640 1 -- 172.21.15.45:0/3739976437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eac1f8710 msgr2=0x7f9eac0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.508+0000 7f9eb2600640 1 --2- 172.21.15.45:0/3739976437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eac1f8710 0x7f9eac0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.508+0000 7f9eb2600640 1 -- 172.21.15.45:0/3739976437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eac0cb340 con 0x7f9eac13db00 2024-04-16T16:21:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.508+0000 7f9eb2600640 1 --2- 172.21.15.45:0/3739976437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9eac13db00 0x7f9eac13df90 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4001350 tx=0x7f9ea400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.508+0000 7f9ea9600640 1 -- 172.21.15.45:0/3739976437 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea400ce70 con 0x7f9eac13db00 2024-04-16T16:21:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.508+0000 7f9ea9600640 1 -- 172.21.15.45:0/3739976437 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ea400a610 con 0x7f9eac13db00 2024-04-16T16:21:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.508+0000 7f9eb3000640 1 -- 172.21.15.45:0/3739976437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eac0cb620 con 0x7f9eac13db00 2024-04-16T16:21:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.508+0000 7f9ea9600640 1 -- 172.21.15.45:0/3739976437 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea400aa00 con 0x7f9eac13db00 2024-04-16T16:21:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.508+0000 7f9eb3000640 1 -- 172.21.15.45:0/3739976437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eac0cbbe0 con 0x7f9eac13db00 2024-04-16T16:21:40.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.510+0000 7f9eb3000640 1 -- 172.21.15.45:0/3739976437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eac075780 con 0x7f9eac13db00 2024-04-16T16:21:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.511+0000 7f9ea9600640 1 -- 172.21.15.45:0/3739976437 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f9ea4011070 con 0x7f9eac13db00 2024-04-16T16:21:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.512+0000 7f9ea9600640 1 --2- 172.21.15.45:0/3739976437 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f9e8c06ca10 0x7f9e8c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.512+0000 7f9ea9600640 1 -- 172.21.15.45:0/3739976437 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7f9ea408b6f0 con 0x7f9eac13db00 2024-04-16T16:21:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.513+0000 7f9eb1c00640 1 --2- 172.21.15.45:0/3739976437 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f9e8c06ca10 0x7f9e8c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:40.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.514+0000 7f9eb1c00640 1 --2- 172.21.15.45:0/3739976437 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f9e8c06ca10 0x7f9e8c06eed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0033000 tx=0x7f9ea0009030 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:40.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.517+0000 7f9ea9600640 1 -- 172.21.15.45:0/3739976437 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9ea401a060 con 0x7f9eac13db00 2024-04-16T16:21:40.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.765+0000 7f9eb3000640 1 -- 172.21.15.45:0/3739976437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9eac0b5280 con 0x7f9eac13db00 2024-04-16T16:21:40.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.766+0000 7f9ea9600640 1 -- 172.21.15.45:0/3739976437 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v100) ==== 53+0+73 (secure 0 0 0) 0x7f9ea4059aa0 con 0x7f9eac13db00 2024-04-16T16:21:40.765 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-16T16:21:40.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.769+0000 7f9eb3000640 1 -- 172.21.15.45:0/3739976437 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f9e8c06ca10 msgr2=0x7f9e8c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:40.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.769+0000 7f9eb3000640 1 --2- 172.21.15.45:0/3739976437 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f9e8c06ca10 0x7f9e8c06eed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0033000 tx=0x7f9ea0009030 comp rx=0 tx=0).stop 2024-04-16T16:21:40.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.769+0000 7f9eb3000640 1 -- 172.21.15.45:0/3739976437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9eac13db00 msgr2=0x7f9eac13df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:40.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.769+0000 7f9eb3000640 1 --2- 172.21.15.45:0/3739976437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9eac13db00 0x7f9eac13df90 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4001350 tx=0x7f9ea400cc20 comp rx=0 tx=0).stop 2024-04-16T16:21:40.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.769+0000 7f9eb3000640 1 -- 172.21.15.45:0/3739976437 shutdown_connections 2024-04-16T16:21:40.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.769+0000 7f9eb3000640 1 --2- 172.21.15.45:0/3739976437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9eac13db00 0x7f9eac13df90 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:40.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.769+0000 7f9eb3000640 1 --2- 172.21.15.45:0/3739976437 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f9e8c06ca10 0x7f9e8c06eed0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:40.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.769+0000 7f9eb3000640 1 --2- 172.21.15.45:0/3739976437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eac1f8710 0x7f9eac0c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:40.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.769+0000 7f9eb3000640 1 --2- 172.21.15.45:0/3739976437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e940039b0 0x7f9eac0cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:40.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.769+0000 7f9eb3000640 1 -- 172.21.15.45:0/3739976437 >> 172.21.15.45:0/3739976437 conn(0x7f9eac06b930 msgr2=0x7f9eac074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:40.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.769+0000 7f9eb3000640 1 -- 172.21.15.45:0/3739976437 shutdown_connections 2024-04-16T16:21:40.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:40.769+0000 7f9eb3000640 1 -- 172.21.15.45:0/3739976437 wait complete. 2024-04-16T16:21:40.789 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-16T16:21:40.789 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T16:21:40.789 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-04-16T16:21:40.789 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-16T16:21:40.789 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T16:21:40.789 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-16T16:21:40.789 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-16T16:21:40.789 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T16:21:40.789 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:21:40.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:40.790 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-04-16T16:21:40.849 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T16:21:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T16:21:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T16:21:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:21:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-16T16:21:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:21:40.849 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:21:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:21:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-04-16T16:21:40.850 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:21:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:21:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-04-16T16:21:40.850 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T16:21:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-04-16T16:21:40.850 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T16:21:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-04-16T16:21:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-16T16:21:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42124sec preferred_lft 42124sec 2024-04-16T16:21:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-04-16T16:21:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:21:40.851 INFO:teuthology.orchestra.run.smithi103.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-16T16:21:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff 2024-04-16T16:21:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-16T16:21:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:21:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::74ce:ceff:fec6:2a00/64 scope link 2024-04-16T16:21:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:21:40.851 INFO:teuthology.orchestra.run.smithi103.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:21:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 3e:f7:ff:ca:f8:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T16:21:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::3cf7:ffff:feca:f879/64 scope link 2024-04-16T16:21:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:21:40.851 INFO:teuthology.orchestra.run.smithi103.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-16T16:21:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 86:4c:6e:ce:c3:07 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-16T16:21:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::844c:6eff:fece:c307/64 scope link 2024-04-16T16:21:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-16T16:21:40.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:40.852 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T16:21:40.947 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-16T16:21:40.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:40.952 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-04-16T16:21:41.004 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-04-16T16:21:41.004 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T16:21:41.005 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-16T16:21:41.005 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-16T16:21:41.005 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T16:21:41.005 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T16:21:41.005 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-16T16:21:41.005 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-16T16:21:41.005 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-16T16:21:41.005 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-16T16:21:41.005 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-16T16:21:41.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:21:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T16:21:41.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:21:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:21:41.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:41.252 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-16T16:21:41.301 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-16T16:21:41.302 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-16T16:21:41.303 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:21:41.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:41.303 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.0 up 2024-04-16T16:21:41.389 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:21:41.389 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-16T16:21:41.442 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-16T16:21:41.443 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T16:21:41.443 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:21:41.497 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-16T16:21:41.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:41.497 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:21:41.635 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:21:41.636 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:21:41.672 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-16T16:21:41.672 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-16T16:21:41.672 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-16T16:21:41.672 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with new device syntax: 1@748c4b85-0190-4c59-80ad-098a6a5007c0.cephfs=/ 2024-04-16T16:21:41.672 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.45:3300/172.21.15.103:3300/172.21.15.103:3301" will pass to kernel 2024-04-16T16:21:41.675 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-16T16:21:41.675 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-04-16T16:21:41.675 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-16T16:21:41.675 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-04-16T16:21:41.675 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-16T16:21:41.680 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-16T16:21:41.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:21:41.680 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-16T16:21:41.755 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-16T16:21:41.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:21:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-16T16:21:41.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-16T16:21:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T16:21:41.866 INFO:teuthology.orchestra.run.smithi103.stdout:client6659 2024-04-16T16:21:41.866 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:21:41.867 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:21:41.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:21:42.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.147+0000 7f9ede600640 1 Processor -- start 2024-04-16T16:21:42.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.147+0000 7f9ede600640 1 -- start start 2024-04-16T16:21:42.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.149+0000 7f9ede600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9ed81608a0 0x7f9ed8160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.149+0000 7f9ede600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ed815d360 con 0x7f9ed815ccb0 2024-04-16T16:21:42.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.149+0000 7f9ede600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ed815d640 con 0x7f9ed8146220 2024-04-16T16:21:42.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.149+0000 7f9ede600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9ed815d130 con 0x7f9ed81608a0 2024-04-16T16:21:42.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.149+0000 7f9ed7e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9ed815ccb0 0x7f9ed806df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55254/0 (socket says 172.21.15.45:55254) 2024-04-16T16:21:42.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.149+0000 7f9ed7e00640 1 -- 172.21.15.45:0/1307563211 learned_addr learned my addr 172.21.15.45:0/1307563211 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:42.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.149+0000 7f9ed7400640 1 -- 172.21.15.45:0/1307563211 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2407064228 0 0) 0x7f9ed815d360 con 0x7f9ed815ccb0 2024-04-16T16:21:42.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.149+0000 7f9ed7400640 1 -- 172.21.15.45:0/1307563211 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ec0003520 con 0x7f9ed815ccb0 2024-04-16T16:21:42.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.150+0000 7f9eddc00640 1 --2- 172.21.15.45:0/1307563211 >> v2:172.21.15.103:3300/0 conn(0x7f9ed81608a0 0x7f9ed8160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:42.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.150+0000 7f9ed7400640 1 -- 172.21.15.45:0/1307563211 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3385983060 0 0) 0x7f9ec0003520 con 0x7f9ed815ccb0 2024-04-16T16:21:42.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.150+0000 7f9ed7400640 1 -- 172.21.15.45:0/1307563211 >> v2:172.21.15.103:3300/0 conn(0x7f9ed81608a0 msgr2=0x7f9ed8160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:42.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.150+0000 7f9ed7400640 1 --2- 172.21.15.45:0/1307563211 >> v2:172.21.15.103:3300/0 conn(0x7f9ed81608a0 0x7f9ed8160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.150+0000 7f9ed7400640 1 -- 172.21.15.45:0/1307563211 >> v1:172.21.15.103:6789/0 conn(0x7f9ed8146220 legacy=0x7f9ed806f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:42.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.150+0000 7f9ed7400640 1 -- 172.21.15.45:0/1307563211 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed815d920 con 0x7f9ed815ccb0 2024-04-16T16:21:42.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.150+0000 7f9ed7400640 1 -- 172.21.15.45:0/1307563211 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f9ec4001670 con 0x7f9ed815ccb0 2024-04-16T16:21:42.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.150+0000 7f9ed7400640 1 -- 172.21.15.45:0/1307563211 >> v1:172.21.15.45:6789/0 conn(0x7f9ed815ccb0 legacy=0x7f9ed806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:42.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.150+0000 7f9ed7400640 1 --2- 172.21.15.45:0/1307563211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ec0003a00 0x7f9ec0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.150+0000 7f9ed7400640 1 --2- 172.21.15.45:0/1307563211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ec0004800 0x7f9ec0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.150+0000 7f9ed7400640 1 --2- 172.21.15.45:0/1307563211 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ec00051c0 0x7f9ec000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.150+0000 7f9ed7400640 1 -- 172.21.15.45:0/1307563211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec000c090 con 0x7f9ec0004800 2024-04-16T16:21:42.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.150+0000 7f9ed7400640 1 -- 172.21.15.45:0/1307563211 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec000c240 con 0x7f9ec0003a00 2024-04-16T16:21:42.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.150+0000 7f9ed7400640 1 -- 172.21.15.45:0/1307563211 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ec000c440 con 0x7f9ec00051c0 2024-04-16T16:21:42.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.150+0000 7f9ed7e00640 1 --2- 172.21.15.45:0/1307563211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ec0004800 0x7f9ec0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:42.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.151+0000 7f9eddc00640 1 --2- 172.21.15.45:0/1307563211 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ec00051c0 0x7f9ec000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:42.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.151+0000 7f9ed7e00640 1 -- 172.21.15.45:0/1307563211 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ec00051c0 msgr2=0x7f9ec000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:42.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.151+0000 7f9ed7e00640 1 --2- 172.21.15.45:0/1307563211 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ec00051c0 0x7f9ec000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.151+0000 7f9ed7e00640 1 -- 172.21.15.45:0/1307563211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ec0003a00 msgr2=0x7f9ec0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:42.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.151+0000 7f9ed7e00640 1 --2- 172.21.15.45:0/1307563211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ec0003a00 0x7f9ec0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.151+0000 7f9ed7e00640 1 -- 172.21.15.45:0/1307563211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ec0003520 con 0x7f9ec0004800 2024-04-16T16:21:42.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.151+0000 7f9eddc00640 1 -- 172.21.15.45:0/1307563211 reap_dead start 2024-04-16T16:21:42.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.152+0000 7f9ed7e00640 1 --2- 172.21.15.45:0/1307563211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ec0004800 0x7f9ec0004c80 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400c110 tx=0x7f9ec0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fbff85ae994bd86 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:42.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.152+0000 7f9ed7400640 1 -- 172.21.15.45:0/1307563211 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4007c40 con 0x7f9ec0004800 2024-04-16T16:21:42.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.152+0000 7f9ed7400640 1 -- 172.21.15.45:0/1307563211 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ec4030070 con 0x7f9ec0004800 2024-04-16T16:21:42.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.152+0000 7f9ed7400640 1 -- 172.21.15.45:0/1307563211 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4005d70 con 0x7f9ec0004800 2024-04-16T16:21:42.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.153+0000 7f9ede600640 1 -- 172.21.15.45:0/1307563211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ec0004800 msgr2=0x7f9ec0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:42.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.153+0000 7f9ede600640 1 --2- 172.21.15.45:0/1307563211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ec0004800 0x7f9ec0004c80 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400c110 tx=0x7f9ec0005830 comp rx=0 tx=0).stop 2024-04-16T16:21:42.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.153+0000 7f9ede600640 1 -- 172.21.15.45:0/1307563211 shutdown_connections 2024-04-16T16:21:42.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.153+0000 7f9ede600640 1 --2- 172.21.15.45:0/1307563211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ec0004800 0x7f9ec0004c80 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.153+0000 7f9ede600640 1 -- 172.21.15.45:0/1307563211 >> 172.21.15.45:0/1307563211 conn(0x7f9ed806b930 msgr2=0x7f9ed806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:42.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.153+0000 7f9ede600640 1 -- 172.21.15.45:0/1307563211 shutdown_connections 2024-04-16T16:21:42.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.153+0000 7f9ede600640 1 -- 172.21.15.45:0/1307563211 wait complete. 2024-04-16T16:21:42.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.155+0000 7f9ede600640 1 Processor -- start 2024-04-16T16:21:42.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.155+0000 7f9ede600640 1 -- start start 2024-04-16T16:21:42.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.156+0000 7f9ede600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ec0004800 0x7f9ed813a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.156+0000 7f9ede600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed815be20 0x7f9ed80c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.156+0000 7f9ede600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ed80cb290 0x7f9ed80cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.156+0000 7f9ede600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed8160cb0 con 0x7f9ed815be20 2024-04-16T16:21:42.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.156+0000 7f9ede600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed8160410 con 0x7f9ec0004800 2024-04-16T16:21:42.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.156+0000 7f9ede600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ed8160e30 con 0x7f9ed80cb290 2024-04-16T16:21:42.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.157+0000 7f9ed7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed815be20 0x7f9ed80c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:42.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.157+0000 7f9ed7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed815be20 0x7f9ed80c9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60282/0 (socket says 172.21.15.45:60282) 2024-04-16T16:21:42.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.157+0000 7f9ed7e00640 1 -- 172.21.15.45:0/1775575070 learned_addr learned my addr 172.21.15.45:0/1775575070 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:42.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.157+0000 7f9edd200640 1 --2- 172.21.15.45:0/1775575070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ec0004800 0x7f9ed813a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:42.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.157+0000 7f9eddc00640 1 --2- 172.21.15.45:0/1775575070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ed80cb290 0x7f9ed80cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:42.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.157+0000 7f9ed7e00640 1 -- 172.21.15.45:0/1775575070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ed80cb290 msgr2=0x7f9ed80cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:42.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.157+0000 7f9ed7e00640 1 --2- 172.21.15.45:0/1775575070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ed80cb290 0x7f9ed80cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.157+0000 7f9ed7e00640 1 -- 172.21.15.45:0/1775575070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ec0004800 msgr2=0x7f9ed813a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:42.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.157+0000 7f9ed7e00640 1 --2- 172.21.15.45:0/1775575070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ec0004800 0x7f9ed813a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.157+0000 7f9ed7e00640 1 -- 172.21.15.45:0/1775575070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed813a710 con 0x7f9ed815be20 2024-04-16T16:21:42.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.158+0000 7f9ed7e00640 1 --2- 172.21.15.45:0/1775575070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed815be20 0x7f9ed80c9e10 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400c230 tx=0x7f9ec402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:42.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.158+0000 7f9ed4c00640 1 -- 172.21.15.45:0/1775575070 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec402cbd0 con 0x7f9ed815be20 2024-04-16T16:21:42.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.158+0000 7f9ed4c00640 1 -- 172.21.15.45:0/1775575070 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ec4000ed0 con 0x7f9ed815be20 2024-04-16T16:21:42.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.158+0000 7f9ed4c00640 1 -- 172.21.15.45:0/1775575070 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4007070 con 0x7f9ed815be20 2024-04-16T16:21:42.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.158+0000 7f9ede600640 1 -- 172.21.15.45:0/1775575070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ed813a9f0 con 0x7f9ed815be20 2024-04-16T16:21:42.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.158+0000 7f9ede600640 1 -- 172.21.15.45:0/1775575070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ed813afb0 con 0x7f9ed815be20 2024-04-16T16:21:42.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.159+0000 7f9ede600640 1 -- 172.21.15.45:0/1775575070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ed8156900 con 0x7f9ed815be20 2024-04-16T16:21:42.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.160+0000 7f9ed4c00640 1 -- 172.21.15.45:0/1775575070 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7f9ec4007210 con 0x7f9ed815be20 2024-04-16T16:21:42.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.161+0000 7f9ed4c00640 1 --2- 172.21.15.45:0/1775575070 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f9eb806ca10 0x7f9eb806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.161+0000 7f9ed4c00640 1 -- 172.21.15.45:0/1775575070 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7f9ec40ad230 con 0x7f9ed815be20 2024-04-16T16:21:42.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.162+0000 7f9edd200640 1 --2- 172.21.15.45:0/1775575070 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f9eb806ca10 0x7f9eb806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:42.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.163+0000 7f9edd200640 1 --2- 172.21.15.45:0/1775575070 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f9eb806ca10 0x7f9eb806eed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9ed80b2860 tx=0x7f9ecc009040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:42.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.166+0000 7f9ed4c00640 1 -- 172.21.15.45:0/1775575070 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9ed8156900 con 0x7f9ed815be20 2024-04-16T16:21:42.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.412+0000 7f9ede600640 1 -- 172.21.15.45:0/1775575070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ed80cff60 con 0x7f9ed815be20 2024-04-16T16:21:42.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.414+0000 7f9ed4c00640 1 -- 172.21.15.45:0/1775575070 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5190 (secure 0 0 0) 0x7f9ec407b540 con 0x7f9ed815be20 2024-04-16T16:21:42.413 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:21:42.413 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2585865379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2585865379},{"type":"v1","addr":"172.21.15.45:6834","nonce":2585865379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6590,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/777087937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":777087937},{"type":"v1","addr":"172.21.15.45:6835","nonce":777087937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6593,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2561260581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2561260581},{"type":"v1","addr":"172.21.15.103:6835","nonce":2561260581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:21:26.066855+0000","modified":"2024-04-16T16:21:29.155869+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6596":{"gid":6596,"name":"d","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6837/3252460318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3252460318},{"type":"v1","addr":"172.21.15.103:6837","nonce":3252460318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6596,"qdb_cluster":[6596]},"id":10}]} 2024-04-16T16:21:42.413 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 100 2024-04-16T16:21:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.416+0000 7f9ede600640 1 -- 172.21.15.45:0/1775575070 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f9eb806ca10 msgr2=0x7f9eb806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.416+0000 7f9ede600640 1 --2- 172.21.15.45:0/1775575070 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f9eb806ca10 0x7f9eb806eed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9ed80b2860 tx=0x7f9ecc009040 comp rx=0 tx=0).stop 2024-04-16T16:21:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.416+0000 7f9ede600640 1 -- 172.21.15.45:0/1775575070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed815be20 msgr2=0x7f9ed80c9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.416+0000 7f9ede600640 1 --2- 172.21.15.45:0/1775575070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed815be20 0x7f9ed80c9e10 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400c230 tx=0x7f9ec402e070 comp rx=0 tx=0).stop 2024-04-16T16:21:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.417+0000 7f9ede600640 1 -- 172.21.15.45:0/1775575070 shutdown_connections 2024-04-16T16:21:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.417+0000 7f9ede600640 1 --2- 172.21.15.45:0/1775575070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ed80cb290 0x7f9ed80cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.417+0000 7f9ede600640 1 --2- 172.21.15.45:0/1775575070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed815be20 0x7f9ed80c9e10 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.417+0000 7f9ede600640 1 --2- 172.21.15.45:0/1775575070 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f9eb806ca10 0x7f9eb806eed0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.417+0000 7f9ede600640 1 --2- 172.21.15.45:0/1775575070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ec0004800 0x7f9ed813a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.417+0000 7f9ede600640 1 -- 172.21.15.45:0/1775575070 >> 172.21.15.45:0/1775575070 conn(0x7f9ed806b930 msgr2=0x7f9ed806f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.417+0000 7f9ede600640 1 -- 172.21.15.45:0/1775575070 shutdown_connections 2024-04-16T16:21:42.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.417+0000 7f9ede600640 1 -- 172.21.15.45:0/1775575070 wait complete. 2024-04-16T16:21:42.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable stats 2024-04-16T16:21:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.669+0000 7ff0fdc00640 1 Processor -- start 2024-04-16T16:21:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.669+0000 7ff0fdc00640 1 -- start start 2024-04-16T16:21:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.671+0000 7ff0fdc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff0f8164370 0x7ff0f8166760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.671+0000 7ff0fdc00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f815bc00 con 0x7ff0f806f030 2024-04-16T16:21:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.671+0000 7ff0fdc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f815bed0 con 0x7ff0f80cf7d0 2024-04-16T16:21:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.671+0000 7ff0fdc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff0f80d0340 con 0x7ff0f8164370 2024-04-16T16:21:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.671+0000 7ff0f7e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff0f806f030 0x7ff0f806f410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55258/0 (socket says 172.21.15.45:55258) 2024-04-16T16:21:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.671+0000 7ff0f7e00640 1 -- 172.21.15.45:0/4259394290 learned_addr learned my addr 172.21.15.45:0/4259394290 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.672+0000 7ff0f7400640 1 -- 172.21.15.45:0/4259394290 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1311044670 0 0) 0x7ff0f815bc00 con 0x7ff0f806f030 2024-04-16T16:21:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.672+0000 7ff0f7400640 1 -- 172.21.15.45:0/4259394290 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0dc005730 con 0x7ff0f806f030 2024-04-16T16:21:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.672+0000 7ff0fd200640 1 --2- 172.21.15.45:0/4259394290 >> v2:172.21.15.103:3300/0 conn(0x7ff0f8164370 0x7ff0f8166760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.672+0000 7ff0f7400640 1 -- 172.21.15.45:0/4259394290 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3155716139 0 0) 0x7ff0dc005730 con 0x7ff0f806f030 2024-04-16T16:21:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.672+0000 7ff0f7400640 1 -- 172.21.15.45:0/4259394290 >> v2:172.21.15.103:3300/0 conn(0x7ff0f8164370 msgr2=0x7ff0f8166760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.672+0000 7ff0f7400640 1 --2- 172.21.15.45:0/4259394290 >> v2:172.21.15.103:3300/0 conn(0x7ff0f8164370 0x7ff0f8166760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.672+0000 7ff0f7400640 1 -- 172.21.15.45:0/4259394290 >> v1:172.21.15.103:6789/0 conn(0x7ff0f80cf7d0 legacy=0x7ff0f80cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.672+0000 7ff0f7400640 1 -- 172.21.15.45:0/4259394290 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f815c1b0 con 0x7ff0f806f030 2024-04-16T16:21:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.672+0000 7ff0f7400640 1 -- 172.21.15.45:0/4259394290 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7ff0e8001670 con 0x7ff0f806f030 2024-04-16T16:21:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.672+0000 7ff0f7400640 1 -- 172.21.15.45:0/4259394290 >> v1:172.21.15.45:6789/0 conn(0x7ff0f806f030 legacy=0x7ff0f806f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.672+0000 7ff0f7400640 1 --2- 172.21.15.45:0/4259394290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0dc005bc0 0x7ff0dc005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.672+0000 7ff0f7400640 1 --2- 172.21.15.45:0/4259394290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0dc006a10 0x7ff0dc006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.673+0000 7ff0f7400640 1 --2- 172.21.15.45:0/4259394290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0dc0073d0 0x7ff0dc00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.673+0000 7ff0f7400640 1 -- 172.21.15.45:0/4259394290 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc00e2f0 con 0x7ff0dc006a10 2024-04-16T16:21:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.673+0000 7ff0f7400640 1 -- 172.21.15.45:0/4259394290 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc00e4a0 con 0x7ff0dc0073d0 2024-04-16T16:21:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.673+0000 7ff0f7400640 1 -- 172.21.15.45:0/4259394290 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff0dc00e6a0 con 0x7ff0dc005bc0 2024-04-16T16:21:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.673+0000 7ff0ef400640 1 --2- 172.21.15.45:0/4259394290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0dc006a10 0x7ff0dc006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.673+0000 7ff0f7e00640 1 --2- 172.21.15.45:0/4259394290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0dc005bc0 0x7ff0dc005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.673+0000 7ff0ef400640 1 -- 172.21.15.45:0/4259394290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0dc005bc0 msgr2=0x7ff0dc005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.673+0000 7ff0ef400640 1 --2- 172.21.15.45:0/4259394290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0dc005bc0 0x7ff0dc005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.673+0000 7ff0ef400640 1 -- 172.21.15.45:0/4259394290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0dc0073d0 msgr2=0x7ff0dc00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:42.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.673+0000 7ff0ef400640 1 --2- 172.21.15.45:0/4259394290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0dc0073d0 0x7ff0dc00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.673+0000 7ff0ef400640 1 -- 172.21.15.45:0/4259394290 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0dc005730 con 0x7ff0dc006a10 2024-04-16T16:21:42.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.673+0000 7ff0fd200640 1 -- 172.21.15.45:0/4259394290 reap_dead start 2024-04-16T16:21:42.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.673+0000 7ff0f7e00640 1 --2- 172.21.15.45:0/4259394290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0dc005bc0 0x7ff0dc005fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.674+0000 7ff0ef400640 1 --2- 172.21.15.45:0/4259394290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0dc006a10 0x7ff0dc006e90 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0001770 tx=0x7ff0dc007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4fbda9151a8bb75 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.675+0000 7ff0f7400640 1 -- 172.21.15.45:0/4259394290 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e00013c0 con 0x7ff0dc006a10 2024-04-16T16:21:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.675+0000 7ff0f7400640 1 -- 172.21.15.45:0/4259394290 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e00052d0 con 0x7ff0dc006a10 2024-04-16T16:21:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.675+0000 7ff0f7400640 1 -- 172.21.15.45:0/4259394290 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0003640 con 0x7ff0dc006a10 2024-04-16T16:21:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.675+0000 7ff0fdc00640 1 -- 172.21.15.45:0/4259394290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0dc006a10 msgr2=0x7ff0dc006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.675+0000 7ff0fdc00640 1 --2- 172.21.15.45:0/4259394290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0dc006a10 0x7ff0dc006e90 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0001770 tx=0x7ff0dc007a40 comp rx=0 tx=0).stop 2024-04-16T16:21:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.675+0000 7ff0fdc00640 1 -- 172.21.15.45:0/4259394290 shutdown_connections 2024-04-16T16:21:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.675+0000 7ff0fdc00640 1 --2- 172.21.15.45:0/4259394290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0dc006a10 0x7ff0dc006e90 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.675+0000 7ff0fdc00640 1 -- 172.21.15.45:0/4259394290 >> 172.21.15.45:0/4259394290 conn(0x7ff0f8067e00 msgr2=0x7ff0f806b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:42.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.675+0000 7ff0fdc00640 1 -- 172.21.15.45:0/4259394290 shutdown_connections 2024-04-16T16:21:42.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.675+0000 7ff0fdc00640 1 -- 172.21.15.45:0/4259394290 wait complete. 2024-04-16T16:21:42.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.677+0000 7ff0fdc00640 1 Processor -- start 2024-04-16T16:21:42.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.677+0000 7ff0fdc00640 1 -- start start 2024-04-16T16:21:42.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.678+0000 7ff0fdc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0dc006a10 0x7ff0f8140620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.678+0000 7ff0fdc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f815b990 0x7ff0f815a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.678+0000 7ff0fdc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f815ad30 0x7ff0f815b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.678+0000 7ff0fdc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f80b3e60 con 0x7ff0f815ad30 2024-04-16T16:21:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.678+0000 7ff0fdc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f815c770 con 0x7ff0f815b990 2024-04-16T16:21:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.678+0000 7ff0fdc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff0f80b3fe0 con 0x7ff0dc006a10 2024-04-16T16:21:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.679+0000 7ff0fd200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f815ad30 0x7ff0f815b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.679+0000 7ff0fd200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f815ad30 0x7ff0f815b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60300/0 (socket says 172.21.15.45:60300) 2024-04-16T16:21:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.679+0000 7ff0fd200640 1 -- 172.21.15.45:0/3878983004 learned_addr learned my addr 172.21.15.45:0/3878983004 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.679+0000 7ff0f7e00640 1 --2- 172.21.15.45:0/3878983004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0dc006a10 0x7ff0f8140620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.679+0000 7ff0ef400640 1 --2- 172.21.15.45:0/3878983004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f815b990 0x7ff0f815a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.679+0000 7ff0fd200640 1 -- 172.21.15.45:0/3878983004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0dc006a10 msgr2=0x7ff0f8140620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.679+0000 7ff0fd200640 1 --2- 172.21.15.45:0/3878983004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0dc006a10 0x7ff0f8140620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.679+0000 7ff0fd200640 1 -- 172.21.15.45:0/3878983004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f815b990 msgr2=0x7ff0f815a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.679+0000 7ff0fd200640 1 --2- 172.21.15.45:0/3878983004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f815b990 0x7ff0f815a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.679+0000 7ff0fd200640 1 -- 172.21.15.45:0/3878983004 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f8158660 con 0x7ff0f815ad30 2024-04-16T16:21:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.679+0000 7ff0fd200640 1 --2- 172.21.15.45:0/3878983004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f815ad30 0x7ff0f815b1c0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8070240 tx=0x7ff0f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.680+0000 7ff0f4c00640 1 -- 172.21.15.45:0/3878983004 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f000edc0 con 0x7ff0f815ad30 2024-04-16T16:21:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.680+0000 7ff0f4c00640 1 -- 172.21.15.45:0/3878983004 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f0019070 con 0x7ff0f815ad30 2024-04-16T16:21:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.680+0000 7ff0f4c00640 1 -- 172.21.15.45:0/3878983004 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0014660 con 0x7ff0f815ad30 2024-04-16T16:21:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.680+0000 7ff0fdc00640 1 -- 172.21.15.45:0/3878983004 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0f8156210 con 0x7ff0f815ad30 2024-04-16T16:21:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.680+0000 7ff0fdc00640 1 -- 172.21.15.45:0/3878983004 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0f81588e0 con 0x7ff0f815ad30 2024-04-16T16:21:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.681+0000 7ff0fdc00640 1 -- 172.21.15.45:0/3878983004 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0f8156730 con 0x7ff0f815ad30 2024-04-16T16:21:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.682+0000 7ff0f4c00640 1 -- 172.21.15.45:0/3878983004 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 59) ==== 91142+0+0 (secure 0 0 0) 0x7ff0f0005d10 con 0x7ff0f815ad30 2024-04-16T16:21:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.683+0000 7ff0f4c00640 1 --2- 172.21.15.45:0/3878983004 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7ff0d406ca10 0x7ff0d406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.683+0000 7ff0f4c00640 1 -- 172.21.15.45:0/3878983004 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7ff0f008be60 con 0x7ff0f815ad30 2024-04-16T16:21:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.684+0000 7ff0f7e00640 1 --2- 172.21.15.45:0/3878983004 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7ff0d406ca10 0x7ff0d406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:42.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.685+0000 7ff0f7e00640 1 --2- 172.21.15.45:0/3878983004 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7ff0d406ca10 0x7ff0d406eed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8000ff0 tx=0x7ff0e8009040 comp rx=0 tx=0).ready entity=mgr.7153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:42.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.688+0000 7ff0f4c00640 1 -- 172.21.15.45:0/3878983004 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff0f8158660 con 0x7ff0f815ad30 2024-04-16T16:21:42.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:42.946+0000 7ff0fdc00640 1 -- 172.21.15.45:0/3878983004 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "stats"} v 0) -- 0x7ff0f80b0020 con 0x7ff0f815ad30 2024-04-16T16:21:43.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.726+0000 7ff0f4c00640 1 -- 172.21.15.45:0/3878983004 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "stats"}]=0 v60) ==== 84+0+0 (secure 0 0 0) 0x7ff0f005a210 con 0x7ff0f815ad30 2024-04-16T16:21:43.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.731+0000 7ff0fdc00640 1 -- 172.21.15.45:0/3878983004 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7ff0d406ca10 msgr2=0x7ff0d406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:43.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.731+0000 7ff0fdc00640 1 --2- 172.21.15.45:0/3878983004 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7ff0d406ca10 0x7ff0d406eed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8000ff0 tx=0x7ff0e8009040 comp rx=0 tx=0).stop 2024-04-16T16:21:43.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.731+0000 7ff0fdc00640 1 -- 172.21.15.45:0/3878983004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f815ad30 msgr2=0x7ff0f815b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:43.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.731+0000 7ff0fdc00640 1 --2- 172.21.15.45:0/3878983004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f815ad30 0x7ff0f815b1c0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8070240 tx=0x7ff0f000e4b0 comp rx=0 tx=0).stop 2024-04-16T16:21:43.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.731+0000 7ff0fdc00640 1 -- 172.21.15.45:0/3878983004 shutdown_connections 2024-04-16T16:21:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.731+0000 7ff0fdc00640 1 --2- 172.21.15.45:0/3878983004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f815ad30 0x7ff0f815b1c0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.731+0000 7ff0fdc00640 1 --2- 172.21.15.45:0/3878983004 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7ff0d406ca10 0x7ff0d406eed0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.731+0000 7ff0fdc00640 1 --2- 172.21.15.45:0/3878983004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f815b990 0x7ff0f815a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.731+0000 7ff0fdc00640 1 --2- 172.21.15.45:0/3878983004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0dc006a10 0x7ff0f8140620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.732+0000 7ff0fdc00640 1 -- 172.21.15.45:0/3878983004 >> 172.21.15.45:0/3878983004 conn(0x7ff0f8067e00 msgr2=0x7ff0f80cd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.732+0000 7ff0fdc00640 1 -- 172.21.15.45:0/3878983004 shutdown_connections 2024-04-16T16:21:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.732+0000 7ff0fdc00640 1 -- 172.21.15.45:0/3878983004 wait complete. 2024-04-16T16:21:43.755 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:21:43.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.954+0000 7f788f400640 1 Processor -- start 2024-04-16T16:21:43.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.954+0000 7f788f400640 1 -- start start 2024-04-16T16:21:43.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.955+0000 7f788f400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7888146220 0x7f78880cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:43.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.955+0000 7f788f400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7888072180 0x7f78880725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:43.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.955+0000 7f788f400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7888146600 con 0x7f78880cd220 2024-04-16T16:21:43.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.955+0000 7f788f400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f788806f840 con 0x7f7888146220 2024-04-16T16:21:43.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.955+0000 7f788f400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f78880ce0e0 con 0x7f7888072180 2024-04-16T16:21:43.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.956+0000 7f7887e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7888072180 0x7f78880725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:43.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.956+0000 7f7887e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7888072180 0x7f78880725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:52418/0 (socket says 172.21.15.45:52418) 2024-04-16T16:21:43.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.956+0000 7f7887e00640 1 -- 172.21.15.45:0/2761371830 learned_addr learned my addr 172.21.15.45:0/2761371830 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:43.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.956+0000 7f788ca00640 1 --2- 172.21.15.45:0/2761371830 >> v2:172.21.15.103:3300/0 conn(0x7f7888146220 0x7f78880cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:43.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.956+0000 7f7887e00640 1 -- 172.21.15.45:0/2761371830 >> v2:172.21.15.103:3300/0 conn(0x7f7888146220 msgr2=0x7f78880cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:43.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.956+0000 7f7887e00640 1 --2- 172.21.15.45:0/2761371830 >> v2:172.21.15.103:3300/0 conn(0x7f7888146220 0x7f78880cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:43.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.956+0000 7f7887e00640 1 -- 172.21.15.45:0/2761371830 >> v1:172.21.15.103:6789/0 conn(0x7f78880cd220 legacy=0x7f788806df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:43.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.956+0000 7f7887e00640 1 -- 172.21.15.45:0/2761371830 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78880cde00 con 0x7f7888072180 2024-04-16T16:21:43.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.957+0000 7f7887e00640 1 --2- 172.21.15.45:0/2761371830 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7888072180 0x7f78880725c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7870009870 tx=0x7f787002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d7bd55043f5f5cb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:43.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.958+0000 7f7887400640 1 -- 172.21.15.45:0/2761371830 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7870035020 con 0x7f7888072180 2024-04-16T16:21:43.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.958+0000 7f7887400640 1 -- 172.21.15.45:0/2761371830 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7870031640 con 0x7f7888072180 2024-04-16T16:21:43.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.958+0000 7f7887400640 1 -- 172.21.15.45:0/2761371830 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787002d040 con 0x7f7888072180 2024-04-16T16:21:43.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.958+0000 7f788f400640 1 -- 172.21.15.45:0/2761371830 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7888072180 msgr2=0x7f78880725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:43.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.958+0000 7f788f400640 1 --2- 172.21.15.45:0/2761371830 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7888072180 0x7f78880725c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7870009870 tx=0x7f787002b8f0 comp rx=0 tx=0).stop 2024-04-16T16:21:43.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.959+0000 7f788f400640 1 -- 172.21.15.45:0/2761371830 shutdown_connections 2024-04-16T16:21:43.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.959+0000 7f788f400640 1 --2- 172.21.15.45:0/2761371830 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7888072180 0x7f78880725c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:43.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.959+0000 7f788f400640 1 --2- 172.21.15.45:0/2761371830 >> v2:172.21.15.103:3300/0 conn(0x7f7888146220 0x7f78880cf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:43.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.959+0000 7f788f400640 1 -- 172.21.15.45:0/2761371830 >> 172.21.15.45:0/2761371830 conn(0x7f788806b930 msgr2=0x7f788806bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:43.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.959+0000 7f788f400640 1 -- 172.21.15.45:0/2761371830 shutdown_connections 2024-04-16T16:21:43.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.959+0000 7f788f400640 1 -- 172.21.15.45:0/2761371830 wait complete. 2024-04-16T16:21:43.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.960+0000 7f788f400640 1 Processor -- start 2024-04-16T16:21:43.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.961+0000 7f788f400640 1 -- start start 2024-04-16T16:21:43.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.961+0000 7f788f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7888072180 0x7f7888143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:43.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.961+0000 7f788f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78880cd220 0x7f788813e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:43.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.961+0000 7f788f400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7888146220 0x7f788813f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:43.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.961+0000 7f788f400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f78880ce910 con 0x7f7888072180 2024-04-16T16:21:43.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.961+0000 7f788f400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7888072b00 con 0x7f78880cd220 2024-04-16T16:21:43.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.961+0000 7f788f400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f78880cebf0 con 0x7f7888146220 2024-04-16T16:21:43.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.962+0000 7f788ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7888072180 0x7f7888143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:43.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.962+0000 7f788ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7888072180 0x7f7888143fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36942/0 (socket says 172.21.15.45:36942) 2024-04-16T16:21:43.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.962+0000 7f788ca00640 1 -- 172.21.15.45:0/4285904868 learned_addr learned my addr 172.21.15.45:0/4285904868 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:43.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.962+0000 7f788d400640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7888146220 0x7f788813f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:43.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.962+0000 7f7887e00640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78880cd220 0x7f788813e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:43.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.962+0000 7f788ca00640 1 -- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7888146220 msgr2=0x7f788813f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:43.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.962+0000 7f788ca00640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7888146220 0x7f788813f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:43.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.962+0000 7f788ca00640 1 -- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78880cd220 msgr2=0x7f788813e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:43.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.962+0000 7f788ca00640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78880cd220 0x7f788813e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:43.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.962+0000 7f788ca00640 1 -- 172.21.15.45:0/4285904868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78881411b0 con 0x7f7888072180 2024-04-16T16:21:43.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.962+0000 7f788d400640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7888146220 0x7f788813f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:21:43.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.963+0000 7f788ca00640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7888072180 0x7f7888143fd0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f7878007e40 tx=0x7f787800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:43.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.963+0000 7f7884c00640 1 -- 172.21.15.45:0/4285904868 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7878014620 con 0x7f7888072180 2024-04-16T16:21:43.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.963+0000 7f7884c00640 1 -- 172.21.15.45:0/4285904868 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7878019070 con 0x7f7888072180 2024-04-16T16:21:43.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.963+0000 7f7884c00640 1 -- 172.21.15.45:0/4285904868 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7878014a00 con 0x7f7888072180 2024-04-16T16:21:43.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.963+0000 7f788f400640 1 -- 172.21.15.45:0/4285904868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f788813ed20 con 0x7f7888072180 2024-04-16T16:21:43.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.963+0000 7f788f400640 1 -- 172.21.15.45:0/4285904868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f788813f000 con 0x7f7888072180 2024-04-16T16:21:43.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.964+0000 7f788f400640 1 -- 172.21.15.45:0/4285904868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78880cfd50 con 0x7f7888072180 2024-04-16T16:21:43.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.965+0000 7f7884c00640 1 -- 172.21.15.45:0/4285904868 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 60) ==== 91151+0+0 (secure 0 0 0) 0x7f7878014ba0 con 0x7f7888072180 2024-04-16T16:21:43.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.966+0000 7f7884c00640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f786406ca60 0x7f786406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:43.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.966+0000 7f7884c00640 1 -- 172.21.15.45:0/4285904868 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5709+0+0 (secure 0 0 0) 0x7f787808c5f0 con 0x7f7888072180 2024-04-16T16:21:43.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.966+0000 7f7887e00640 1 -- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f786406ca60 msgr2=0x7f786406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1478535274 2024-04-16T16:21:43.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.966+0000 7f7887e00640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f786406ca60 0x7f786406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:21:43.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:43.970+0000 7f7884c00640 1 -- 172.21.15.45:0/4285904868 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78880cfd50 con 0x7f7888072180 2024-04-16T16:21:44.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:44.167+0000 7f7887e00640 1 -- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f786406ca60 msgr2=0x7f786406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1478535274 2024-04-16T16:21:44.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:44.167+0000 7f7887e00640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f786406ca60 0x7f786406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:21:44.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:44.191+0000 7f788f400640 1 -- 172.21.15.45:0/4285904868 --> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f7888075780 con 0x7f786406ca60 2024-04-16T16:21:44.283 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:44.285+0000 7fd08be30040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:21:44.285 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:44.287+0000 7f40889f1040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:21:44.338 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:44.339+0000 7fd08be30040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:21:44.342 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:44.344+0000 7f40889f1040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:21:44.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:44.568+0000 7f7887e00640 1 -- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f786406ca60 msgr2=0x7f786406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1478535274 2024-04-16T16:21:44.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:44.568+0000 7f7887e00640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f786406ca60 0x7f786406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T16:21:44.778 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:44.779+0000 7fd08be30040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:21:44.808 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:44.809+0000 7f40889f1040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:21:44.829 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:44.831+0000 7fd08be30040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:21:44.862 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:44.864+0000 7f40889f1040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:21:44.880 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:44.881+0000 7fd08be30040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:21:44.915 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:44.917+0000 7f40889f1040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:21:44.940 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:44.942+0000 7fd08be30040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:21:44.980 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:44.981+0000 7f40889f1040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:21:44.989 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:44.991+0000 7fd08be30040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:21:45.032 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:45.033+0000 7f40889f1040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:21:45.119 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:45.121+0000 7fd08be30040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:21:45.167 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:45.168+0000 7f40889f1040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:21:45.181 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:45.183+0000 7fd08be30040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:21:45.232 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:45.233+0000 7fd08be30040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:21:45.234 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:45.235+0000 7f40889f1040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:21:45.286 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:45.287+0000 7f40889f1040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:21:45.288 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:45.290+0000 7fd08be30040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:21:45.339 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:45.341+0000 7fd08be30040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:21:45.345 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:45.347+0000 7f40889f1040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:21:45.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:45.369+0000 7f7887e00640 1 -- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f786406ca60 msgr2=0x7f786406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1478535274 2024-04-16T16:21:45.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:45.369+0000 7f7887e00640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f786406ca60 0x7f786406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T16:21:45.399 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:45.400+0000 7f40889f1040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:21:45.722 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:45.723+0000 7fd08be30040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:21:45.802 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:45.803+0000 7f40889f1040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:21:45.880 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:45.882+0000 7fd08be30040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:21:45.969 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:45.971+0000 7f40889f1040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:21:45.988 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:45.990+0000 7fd08be30040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:21:46.039 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:46.040+0000 7fd08be30040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:21:46.083 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:46.084+0000 7f40889f1040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:21:46.136 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:46.137+0000 7f40889f1040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:21:46.165 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:46.167+0000 7fd08be30040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:21:46.269 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:46.270+0000 7f40889f1040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:21:46.272 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:46.274+0000 7fd08be30040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:21:46.382 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:46.383+0000 7f40889f1040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:21:46.446 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:46.448+0000 7fd08be30040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:21:46.496 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:46.497+0000 7fd08be30040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:21:46.566 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:46.566+0000 7f40889f1040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:21:46.617 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:46.618+0000 7f40889f1040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:21:46.632 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:46.633+0000 7fd08be30040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:21:46.760 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:46.761+0000 7f40889f1040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:21:46.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:46.971+0000 7f7887e00640 1 -- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f786406ca60 msgr2=0x7f786406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1478535274 2024-04-16T16:21:46.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:46.972+0000 7f7887e00640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f786406ca60 0x7f786406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T16:21:47.438 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:47.439+0000 7fd08be30040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:21:47.673 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:47.673+0000 7f40889f1040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:21:47.739 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:47.739+0000 7fd08be30040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:21:47.795 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:47.796+0000 7fd08be30040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:21:47.926 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:47.927+0000 7fd08be30040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:21:47.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:47.949+0000 7f7884c00640 1 -- 172.21.15.45:0/4285904868 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 61) ==== 90760+0+0 (secure 0 0 0) 0x7f7878005d10 con 0x7f7888072180 2024-04-16T16:21:47.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:47.950+0000 7f7884c00640 1 -- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f786406ca60 msgr2=0x7f786406ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:21:47.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:47.950+0000 7f7884c00640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1478535274,v1:172.21.15.103:6833/1478535274] conn(0x7f786406ca60 0x7f786406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:48.000 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:48.001+0000 7f40889f1040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:21:48.024 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:48.024+0000 7fd003600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:48.024 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:48.024+0000 7fd003600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:48.024 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:48.024+0000 7fd003600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:48.024 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:48.024+0000 7fd003600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:48.024 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:48.024+0000 7fd003600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:48.024 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:48.024+0000 7fd007200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:48.024 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:48.024+0000 7fd007200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:48.024 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:48.024+0000 7fd007200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:48.024 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:48.024+0000 7fd007200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:48.024 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:21:48.024+0000 7fd007200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:21:48.063 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:48.064+0000 7f40889f1040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:21:48.204 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:21:48.205+0000 7f40889f1040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:21:48.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:48.961+0000 7f7884c00640 1 -- 172.21.15.45:0/4285904868 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 62) ==== 90887+0+0 (secure 0 0 0) 0x7f7878056870 con 0x7f7888072180 2024-04-16T16:21:48.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:48.962+0000 7f7884c00640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f7864074910 0x7f7864076d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:48.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:48.962+0000 7f7884c00640 1 -- 172.21.15.45:0/4285904868 --> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f7864007ea0 con 0x7f7864074910 2024-04-16T16:21:48.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:48.964+0000 7f7887e00640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f7864074910 0x7f7864076d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:48.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:48.969+0000 7f7887e00640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f7864074910 0x7f7864076d00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7870036000 tx=0x7f7870008030 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:48.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:48.986+0000 7f7884c00640 1 -- 172.21.15.45:0/4285904868 <== mgr.6674 v2:172.21.15.103:6832/1506110537 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7f7864007ea0 con 0x7f7864074910 2024-04-16T16:21:48.985 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:21:48.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:48.989+0000 7f788f400640 1 -- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f7864074910 msgr2=0x7f7864076d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:48.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:48.989+0000 7f788f400640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f7864074910 0x7f7864076d00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7870036000 tx=0x7f7870008030 comp rx=0 tx=0).stop 2024-04-16T16:21:48.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:48.989+0000 7f788f400640 1 -- 172.21.15.45:0/4285904868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7888072180 msgr2=0x7f7888143fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:48.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:48.989+0000 7f788f400640 1 --2- 172.21.15.45:0/4285904868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7888072180 0x7f7888143fd0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f7878007e40 tx=0x7f787800e4b0 comp rx=0 tx=0).stop 2024-04-16T16:21:48.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:48.989+0000 7f788d400640 1 -- 172.21.15.45:0/4285904868 reap_dead start 2024-04-16T16:21:48.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:48.989+0000 7f788f400640 1 -- 172.21.15.45:0/4285904868 shutdown_connections 2024-04-16T16:21:48.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:48.989+0000 7f788f400640 1 -- 172.21.15.45:0/4285904868 >> 172.21.15.45:0/4285904868 conn(0x7f788806b930 msgr2=0x7f788806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:48.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:48.990+0000 7f788f400640 1 -- 172.21.15.45:0/4285904868 shutdown_connections 2024-04-16T16:21:48.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:48.990+0000 7f788f400640 1 -- 172.21.15.45:0/4285904868 wait complete. 2024-04-16T16:21:50.013 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:21:50.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.242+0000 7ff7e2800640 1 Processor -- start 2024-04-16T16:21:50.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.242+0000 7ff7e2800640 1 -- start start 2024-04-16T16:21:50.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.244+0000 7ff7e2800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7dc0c6fc0 0x7ff7dc06f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:50.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.244+0000 7ff7e2800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff7dc15cdc0 0x7ff7dc0c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:50.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.244+0000 7ff7e2800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7dc15d600 con 0x7ff7dc0c6320 2024-04-16T16:21:50.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.244+0000 7ff7e2800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff7dc0c5670 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.244+0000 7ff7e2800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff7dc15d8e0 con 0x7ff7dc0c6fc0 2024-04-16T16:21:50.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.244+0000 7ff7e0a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff7dc15cdc0 0x7ff7dc0c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:50.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.245+0000 7ff7e0a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff7dc15cdc0 0x7ff7dc0c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36946/0 (socket says 172.21.15.45:36946) 2024-04-16T16:21:50.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.245+0000 7ff7e0a00640 1 -- 172.21.15.45:0/2679473529 learned_addr learned my addr 172.21.15.45:0/2679473529 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:50.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.245+0000 7ff7e1400640 1 --2- 172.21.15.45:0/2679473529 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7dc0c6fc0 0x7ff7dc06f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:50.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.245+0000 7ff7e0a00640 1 -- 172.21.15.45:0/2679473529 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7dc0c6fc0 msgr2=0x7ff7dc06f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:50.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.245+0000 7ff7e0a00640 1 --2- 172.21.15.45:0/2679473529 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7dc0c6fc0 0x7ff7dc06f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:50.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.245+0000 7ff7e0a00640 1 -- 172.21.15.45:0/2679473529 >> v1:172.21.15.103:6789/0 conn(0x7ff7dc0c6320 legacy=0x7ff7dc06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:50.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.245+0000 7ff7e0a00640 1 -- 172.21.15.45:0/2679473529 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7dc15dbc0 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.245+0000 7ff7e1400640 1 --2- 172.21.15.45:0/2679473529 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7dc0c6fc0 0x7ff7dc06f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:50.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.246+0000 7ff7e0a00640 1 --2- 172.21.15.45:0/2679473529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7dc15cdc0 0x7ff7dc0c5de0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8009870 tx=0x7ff7c802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9092272a944c719f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:50.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.247+0000 7ff7dbe00640 1 -- 172.21.15.45:0/2679473529 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8035020 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.247+0000 7ff7dbe00640 1 -- 172.21.15.45:0/2679473529 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7c8031640 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.247+0000 7ff7dbe00640 1 -- 172.21.15.45:0/2679473529 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c802d040 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.247+0000 7ff7e2800640 1 -- 172.21.15.45:0/2679473529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7dc15cdc0 msgr2=0x7ff7dc0c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:50.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.247+0000 7ff7e2800640 1 --2- 172.21.15.45:0/2679473529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7dc15cdc0 0x7ff7dc0c5de0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8009870 tx=0x7ff7c802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:21:50.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.248+0000 7ff7e2800640 1 -- 172.21.15.45:0/2679473529 shutdown_connections 2024-04-16T16:21:50.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.248+0000 7ff7e2800640 1 --2- 172.21.15.45:0/2679473529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7dc15cdc0 0x7ff7dc0c5de0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:50.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.248+0000 7ff7e2800640 1 --2- 172.21.15.45:0/2679473529 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7dc0c6fc0 0x7ff7dc06f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:50.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.248+0000 7ff7e2800640 1 -- 172.21.15.45:0/2679473529 >> 172.21.15.45:0/2679473529 conn(0x7ff7dc06b930 msgr2=0x7ff7dc06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:50.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.248+0000 7ff7e2800640 1 -- 172.21.15.45:0/2679473529 shutdown_connections 2024-04-16T16:21:50.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.248+0000 7ff7e2800640 1 -- 172.21.15.45:0/2679473529 wait complete. 2024-04-16T16:21:50.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.249+0000 7ff7e2800640 1 Processor -- start 2024-04-16T16:21:50.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.250+0000 7ff7e2800640 1 -- start start 2024-04-16T16:21:50.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7dc0c6320 0x7ff7dc0cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:50.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e2800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7dc0c6fc0 0x7ff7dc150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:50.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7dc15cdc0 0x7ff7dc0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:50.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e2800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc15d3c0 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e2800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc15d240 con 0x7ff7dc0c6320 2024-04-16T16:21:50.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e2800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff7dc0c73a0 con 0x7ff7dc0c6fc0 2024-04-16T16:21:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7dc15cdc0 0x7ff7dc0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7dc15cdc0 0x7ff7dc0ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36956/0 (socket says 172.21.15.45:36956) 2024-04-16T16:21:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e1e00640 1 -- 172.21.15.45:0/1989196377 learned_addr learned my addr 172.21.15.45:0/1989196377 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e0a00640 1 --2- 172.21.15.45:0/1989196377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7dc0c6fc0 0x7ff7dc150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e1400640 1 --2- 172.21.15.45:0/1989196377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7dc0c6320 0x7ff7dc0cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e1e00640 1 -- 172.21.15.45:0/1989196377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7dc0c6fc0 msgr2=0x7ff7dc150cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e1e00640 1 --2- 172.21.15.45:0/1989196377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7dc0c6fc0 0x7ff7dc150cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e1e00640 1 -- 172.21.15.45:0/1989196377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7dc0c6320 msgr2=0x7ff7dc0cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e1e00640 1 --2- 172.21.15.45:0/1989196377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7dc0c6320 0x7ff7dc0cc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.251+0000 7ff7e1e00640 1 -- 172.21.15.45:0/1989196377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7dc0cfff0 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.252+0000 7ff7e1400640 1 --2- 172.21.15.45:0/1989196377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7dc0c6320 0x7ff7dc0cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.252+0000 7ff7e1e00640 1 --2- 172.21.15.45:0/1989196377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7dc15cdc0 0x7ff7dc0ca840 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7ff7d400c780 tx=0x7ff7d400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.252+0000 7ff7d9600640 1 -- 172.21.15.45:0/1989196377 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d4004b60 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.252+0000 7ff7d9600640 1 -- 172.21.15.45:0/1989196377 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7d4004d00 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.252+0000 7ff7d9600640 1 -- 172.21.15.45:0/1989196377 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d400aa90 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.252+0000 7ff7e2800640 1 -- 172.21.15.45:0/1989196377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7dc0cfa30 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.252+0000 7ff7e2800640 1 -- 172.21.15.45:0/1989196377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7dc1507d0 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.254+0000 7ff7e2800640 1 -- 172.21.15.45:0/1989196377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7dc075780 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.255+0000 7ff7d9600640 1 -- 172.21.15.45:0/1989196377 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 63) ==== 91095+0+0 (secure 0 0 0) 0x7ff7d400ac30 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.256+0000 7ff7d9600640 1 --2- 172.21.15.45:0/1989196377 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7ff7bc06c9d0 0x7ff7bc06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:50.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.256+0000 7ff7d9600640 1 -- 172.21.15.45:0/1989196377 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7ff7d408c090 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.256+0000 7ff7e1400640 1 --2- 172.21.15.45:0/1989196377 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7ff7bc06c9d0 0x7ff7bc06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.257+0000 7ff7e1400640 1 --2- 172.21.15.45:0/1989196377 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7ff7bc06c9d0 0x7ff7bc06ee90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0006fd0 tx=0x7ff7d0008040 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:50.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.261+0000 7ff7d9600640 1 -- 172.21.15.45:0/1989196377 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff7d4020070 con 0x7ff7dc15cdc0 2024-04-16T16:21:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.493+0000 7ff7e2800640 1 -- 172.21.15.45:0/1989196377 --> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7ff7dc0cfe30 con 0x7ff7bc06c9d0 2024-04-16T16:21:50.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.497+0000 7ff7d9600640 1 -- 172.21.15.45:0/1989196377 <== mgr.6674 v2:172.21.15.103:6832/1506110537 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7ff7d000d840 con 0x7ff7bc06c9d0 2024-04-16T16:21:50.496 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:21:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.500+0000 7ff7e2800640 1 -- 172.21.15.45:0/1989196377 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7ff7bc06c9d0 msgr2=0x7ff7bc06ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.500+0000 7ff7e2800640 1 --2- 172.21.15.45:0/1989196377 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7ff7bc06c9d0 0x7ff7bc06ee90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0006fd0 tx=0x7ff7d0008040 comp rx=0 tx=0).stop 2024-04-16T16:21:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.500+0000 7ff7e2800640 1 -- 172.21.15.45:0/1989196377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7dc15cdc0 msgr2=0x7ff7dc0ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.500+0000 7ff7e2800640 1 --2- 172.21.15.45:0/1989196377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7dc15cdc0 0x7ff7dc0ca840 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7ff7d400c780 tx=0x7ff7d400cc40 comp rx=0 tx=0).stop 2024-04-16T16:21:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.500+0000 7ff7e2800640 1 -- 172.21.15.45:0/1989196377 shutdown_connections 2024-04-16T16:21:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.500+0000 7ff7e2800640 1 --2- 172.21.15.45:0/1989196377 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7ff7bc06c9d0 0x7ff7bc06ee90 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.500+0000 7ff7e2800640 1 --2- 172.21.15.45:0/1989196377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7dc15cdc0 0x7ff7dc0ca840 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.500+0000 7ff7e2800640 1 --2- 172.21.15.45:0/1989196377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7dc0c6fc0 0x7ff7dc150cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.500+0000 7ff7e2800640 1 --2- 172.21.15.45:0/1989196377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7dc0c6320 0x7ff7dc0cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.500+0000 7ff7e2800640 1 -- 172.21.15.45:0/1989196377 >> 172.21.15.45:0/1989196377 conn(0x7ff7dc06b930 msgr2=0x7ff7dc06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.500+0000 7ff7e2800640 1 -- 172.21.15.45:0/1989196377 shutdown_connections 2024-04-16T16:21:50.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:50.500+0000 7ff7e2800640 1 -- 172.21.15.45:0/1989196377 wait complete. 2024-04-16T16:21:51.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:21:51.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.757+0000 7fd7be000640 1 Processor -- start 2024-04-16T16:21:51.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.757+0000 7fd7be000640 1 -- start start 2024-04-16T16:21:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.759+0000 7fd7be000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd7b80c6fc0 0x7fd7b806f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.759+0000 7fd7be000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b806df40 0x7fd7b806e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.759+0000 7fd7be000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7b815d600 con 0x7fd7b815cdc0 2024-04-16T16:21:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.759+0000 7fd7be000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd7b80c5670 con 0x7fd7b80c6fc0 2024-04-16T16:21:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.759+0000 7fd7be000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd7b815d8e0 con 0x7fd7b806df40 2024-04-16T16:21:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.759+0000 7fd7bcc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd7b80c6fc0 0x7fd7b806f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.759+0000 7fd7bcc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd7b80c6fc0 0x7fd7b806f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36968/0 (socket says 172.21.15.45:36968) 2024-04-16T16:21:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.759+0000 7fd7bcc00640 1 -- 172.21.15.45:0/1181452826 learned_addr learned my addr 172.21.15.45:0/1181452826 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.760+0000 7fd7bd600640 1 --2- 172.21.15.45:0/1181452826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b806df40 0x7fd7b806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.760+0000 7fd7bcc00640 1 -- 172.21.15.45:0/1181452826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b806df40 msgr2=0x7fd7b806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.760+0000 7fd7bcc00640 1 --2- 172.21.15.45:0/1181452826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b806df40 0x7fd7b806e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.760+0000 7fd7bcc00640 1 -- 172.21.15.45:0/1181452826 >> v1:172.21.15.45:6789/0 conn(0x7fd7b815cdc0 legacy=0x7fd7b80c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.760+0000 7fd7bcc00640 1 -- 172.21.15.45:0/1181452826 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b815dbc0 con 0x7fd7b80c6fc0 2024-04-16T16:21:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.760+0000 7fd7bd600640 1 --2- 172.21.15.45:0/1181452826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b806df40 0x7fd7b806e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:51.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.761+0000 7fd7bcc00640 1 --2- 172.21.15.45:0/1181452826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b80c6fc0 0x7fd7b806f730 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac009870 tx=0x7fd7ac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59be6a90e1946391 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.762+0000 7fd7b7400640 1 -- 172.21.15.45:0/1181452826 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac034070 con 0x7fd7b80c6fc0 2024-04-16T16:21:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.762+0000 7fd7b7400640 1 -- 172.21.15.45:0/1181452826 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ac030840 con 0x7fd7b80c6fc0 2024-04-16T16:21:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.762+0000 7fd7b7400640 1 -- 172.21.15.45:0/1181452826 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac030c10 con 0x7fd7b80c6fc0 2024-04-16T16:21:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.762+0000 7fd7be000640 1 -- 172.21.15.45:0/1181452826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b80c6fc0 msgr2=0x7fd7b806f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.762+0000 7fd7be000640 1 --2- 172.21.15.45:0/1181452826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b80c6fc0 0x7fd7b806f730 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac009870 tx=0x7fd7ac02b980 comp rx=0 tx=0).stop 2024-04-16T16:21:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.762+0000 7fd7be000640 1 -- 172.21.15.45:0/1181452826 shutdown_connections 2024-04-16T16:21:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.762+0000 7fd7be000640 1 --2- 172.21.15.45:0/1181452826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b806df40 0x7fd7b806e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.762+0000 7fd7be000640 1 --2- 172.21.15.45:0/1181452826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b80c6fc0 0x7fd7b806f730 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.762+0000 7fd7be000640 1 -- 172.21.15.45:0/1181452826 >> 172.21.15.45:0/1181452826 conn(0x7fd7b806b930 msgr2=0x7fd7b806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.763+0000 7fd7be000640 1 -- 172.21.15.45:0/1181452826 shutdown_connections 2024-04-16T16:21:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.763+0000 7fd7be000640 1 -- 172.21.15.45:0/1181452826 wait complete. 2024-04-16T16:21:51.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.764+0000 7fd7be000640 1 Processor -- start 2024-04-16T16:21:51.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.764+0000 7fd7be000640 1 -- start start 2024-04-16T16:21:51.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.765+0000 7fd7be000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b806df40 0x7fd7b81540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:51.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.765+0000 7fd7be000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7b80c6fc0 0x7fd7b81f7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:51.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.765+0000 7fd7be000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b815cdc0 0x7fd7b806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:51.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.765+0000 7fd7be000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b80c64f0 con 0x7fd7b815cdc0 2024-04-16T16:21:51.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.765+0000 7fd7be000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b815d270 con 0x7fd7b80c6fc0 2024-04-16T16:21:51.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.765+0000 7fd7be000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd7b80c6670 con 0x7fd7b806df40 2024-04-16T16:21:51.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.766+0000 7fd7bd600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b815cdc0 0x7fd7b806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:51.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.766+0000 7fd7bd600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b815cdc0 0x7fd7b806e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36980/0 (socket says 172.21.15.45:36980) 2024-04-16T16:21:51.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.766+0000 7fd7bd600640 1 -- 172.21.15.45:0/376423351 learned_addr learned my addr 172.21.15.45:0/376423351 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:51.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.766+0000 7fd7bd600640 1 -- 172.21.15.45:0/376423351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b806df40 msgr2=0x7fd7b81540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:51.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.766+0000 7fd7bd600640 1 --2- 172.21.15.45:0/376423351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b806df40 0x7fd7b81540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:51.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.766+0000 7fd7bd600640 1 -- 172.21.15.45:0/376423351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7b80c6fc0 msgr2=0x7fd7b81f7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:51.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.766+0000 7fd7bd600640 1 --2- 172.21.15.45:0/376423351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7b80c6fc0 0x7fd7b81f7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:51.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.766+0000 7fd7bd600640 1 -- 172.21.15.45:0/376423351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b815bad0 con 0x7fd7b815cdc0 2024-04-16T16:21:51.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.766+0000 7fd7bd600640 1 --2- 172.21.15.45:0/376423351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b815cdc0 0x7fd7b806e320 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fd7b000cf20 tx=0x7fd7b000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:51.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.767+0000 7fd7b4c00640 1 -- 172.21.15.45:0/376423351 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b001a070 con 0x7fd7b815cdc0 2024-04-16T16:21:51.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.767+0000 7fd7b4c00640 1 -- 172.21.15.45:0/376423351 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b000d840 con 0x7fd7b815cdc0 2024-04-16T16:21:51.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.767+0000 7fd7be000640 1 -- 172.21.15.45:0/376423351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b8154620 con 0x7fd7b815cdc0 2024-04-16T16:21:51.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.767+0000 7fd7b4c00640 1 -- 172.21.15.45:0/376423351 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b000dc30 con 0x7fd7b815cdc0 2024-04-16T16:21:51.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.767+0000 7fd7be000640 1 -- 172.21.15.45:0/376423351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b81f6170 con 0x7fd7b815cdc0 2024-04-16T16:21:51.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.768+0000 7fd7be000640 1 -- 172.21.15.45:0/376423351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7b8075780 con 0x7fd7b815cdc0 2024-04-16T16:21:51.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.769+0000 7fd7b4c00640 1 -- 172.21.15.45:0/376423351 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 63) ==== 91095+0+0 (secure 0 0 0) 0x7fd7b001d070 con 0x7fd7b815cdc0 2024-04-16T16:21:51.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.770+0000 7fd7b4c00640 1 --2- 172.21.15.45:0/376423351 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd79806c9d0 0x7fd79806ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:51.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.771+0000 7fd7b4c00640 1 -- 172.21.15.45:0/376423351 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7fd7b008c720 con 0x7fd7b815cdc0 2024-04-16T16:21:51.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.771+0000 7fd7bcc00640 1 --2- 172.21.15.45:0/376423351 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd79806c9d0 0x7fd79806ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:51.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.773+0000 7fd7bcc00640 1 --2- 172.21.15.45:0/376423351 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd79806c9d0 0x7fd79806ee90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac036000 tx=0x7fd7ac03b040 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:51.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:51.775+0000 7fd7b4c00640 1 -- 172.21.15.45:0/376423351 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd7b8154620 con 0x7fd7b815cdc0 2024-04-16T16:21:52.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:52.006+0000 7fd7be000640 1 -- 172.21.15.45:0/376423351 --> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7fd7b815f1f0 con 0x7fd79806c9d0 2024-04-16T16:21:52.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:52.010+0000 7fd7b4c00640 1 -- 172.21.15.45:0/376423351 <== mgr.6674 v2:172.21.15.103:6832/1506110537 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7fd7b815f1f0 con 0x7fd79806c9d0 2024-04-16T16:21:52.009 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:21:52.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:52.012+0000 7fd7be000640 1 -- 172.21.15.45:0/376423351 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd79806c9d0 msgr2=0x7fd79806ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:52.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:52.012+0000 7fd7be000640 1 --2- 172.21.15.45:0/376423351 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd79806c9d0 0x7fd79806ee90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac036000 tx=0x7fd7ac03b040 comp rx=0 tx=0).stop 2024-04-16T16:21:52.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:52.012+0000 7fd7be000640 1 -- 172.21.15.45:0/376423351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b815cdc0 msgr2=0x7fd7b806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:52.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:52.012+0000 7fd7be000640 1 --2- 172.21.15.45:0/376423351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b815cdc0 0x7fd7b806e320 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fd7b000cf20 tx=0x7fd7b000a7d0 comp rx=0 tx=0).stop 2024-04-16T16:21:52.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:52.012+0000 7fd7be000640 1 -- 172.21.15.45:0/376423351 shutdown_connections 2024-04-16T16:21:52.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:52.012+0000 7fd7be000640 1 --2- 172.21.15.45:0/376423351 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd79806c9d0 0x7fd79806ee90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:52.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:52.012+0000 7fd7be000640 1 --2- 172.21.15.45:0/376423351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b815cdc0 0x7fd7b806e320 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:52.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:52.012+0000 7fd7be000640 1 --2- 172.21.15.45:0/376423351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7b80c6fc0 0x7fd7b81f7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:52.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:52.012+0000 7fd7be000640 1 --2- 172.21.15.45:0/376423351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b806df40 0x7fd7b81540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:52.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:52.012+0000 7fd7be000640 1 -- 172.21.15.45:0/376423351 >> 172.21.15.45:0/376423351 conn(0x7fd7b806b930 msgr2=0x7fd7b80b4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:52.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:52.013+0000 7fd7be000640 1 -- 172.21.15.45:0/376423351 shutdown_connections 2024-04-16T16:21:52.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:52.013+0000 7fd7be000640 1 -- 172.21.15.45:0/376423351 wait complete. 2024-04-16T16:21:53.036 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:21:53.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.269+0000 7fa31d400640 1 Processor -- start 2024-04-16T16:21:53.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.269+0000 7fa31d400640 1 -- start start 2024-04-16T16:21:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.271+0000 7fa31d400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa31815e830 0x7fa3180ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.271+0000 7fa31d400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa31815f1b0 con 0x7fa3180cbc80 2024-04-16T16:21:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.271+0000 7fa31d400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3180cf280 con 0x7fa3180cb7d0 2024-04-16T16:21:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.271+0000 7fa31d400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa3180ce190 con 0x7fa31815e830 2024-04-16T16:21:53.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.272+0000 7fa31ca00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa3180cbc80 0x7fa3180cc410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33218/0 (socket says 172.21.15.45:33218) 2024-04-16T16:21:53.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.272+0000 7fa31ca00640 1 -- 172.21.15.45:0/1627127430 learned_addr learned my addr 172.21.15.45:0/1627127430 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.272+0000 7fa317e00640 1 --2- 172.21.15.45:0/1627127430 >> v2:172.21.15.103:3300/0 conn(0x7fa31815e830 0x7fa3180ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.272+0000 7fa316a00640 1 -- 172.21.15.45:0/1627127430 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1472498602 0 0) 0x7fa310001110 con 0x7fa3180cbc80 2024-04-16T16:21:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.272+0000 7fa316a00640 1 -- 172.21.15.45:0/1627127430 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2fc003520 con 0x7fa3180cbc80 2024-04-16T16:21:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.273+0000 7fa316a00640 1 -- 172.21.15.45:0/1627127430 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2855813283 0 0) 0x7fa2fc003520 con 0x7fa3180cbc80 2024-04-16T16:21:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.273+0000 7fa316a00640 1 -- 172.21.15.45:0/1627127430 >> v2:172.21.15.103:3300/0 conn(0x7fa31815e830 msgr2=0x7fa3180ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.273+0000 7fa316a00640 1 --2- 172.21.15.45:0/1627127430 >> v2:172.21.15.103:3300/0 conn(0x7fa31815e830 0x7fa3180ce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.273+0000 7fa316a00640 1 -- 172.21.15.45:0/1627127430 >> v1:172.21.15.103:6789/0 conn(0x7fa3180cb7d0 legacy=0x7fa3180cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.273+0000 7fa316a00640 1 -- 172.21.15.45:0/1627127430 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3180ce470 con 0x7fa3180cbc80 2024-04-16T16:21:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.273+0000 7fa316a00640 1 -- 172.21.15.45:0/1627127430 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fa310001840 con 0x7fa3180cbc80 2024-04-16T16:21:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.273+0000 7fa316a00640 1 -- 172.21.15.45:0/1627127430 >> v1:172.21.15.45:6789/0 conn(0x7fa3180cbc80 legacy=0x7fa3180cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.273+0000 7fa316a00640 1 --2- 172.21.15.45:0/1627127430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2fc0039b0 0x7fa2fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.273+0000 7fa316a00640 1 --2- 172.21.15.45:0/1627127430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2fc004800 0x7fa2fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.273+0000 7fa316a00640 1 --2- 172.21.15.45:0/1627127430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc0051c0 0x7fa2fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.273+0000 7fa316a00640 1 -- 172.21.15.45:0/1627127430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa3180ce190 con 0x7fa2fc0051c0 2024-04-16T16:21:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.273+0000 7fa316a00640 1 -- 172.21.15.45:0/1627127430 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc00c160 con 0x7fa2fc0039b0 2024-04-16T16:21:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.273+0000 7fa316a00640 1 -- 172.21.15.45:0/1627127430 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa2fc00c360 con 0x7fa2fc004800 2024-04-16T16:21:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.273+0000 7fa31ca00640 1 --2- 172.21.15.45:0/1627127430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc0051c0 0x7fa2fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.273+0000 7fa317e00640 1 --2- 172.21.15.45:0/1627127430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2fc0039b0 0x7fa2fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.274+0000 7fa317400640 1 --2- 172.21.15.45:0/1627127430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2fc004800 0x7fa2fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.274+0000 7fa31ca00640 1 -- 172.21.15.45:0/1627127430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2fc004800 msgr2=0x7fa2fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.274+0000 7fa31ca00640 1 --2- 172.21.15.45:0/1627127430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2fc004800 0x7fa2fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.274+0000 7fa31ca00640 1 -- 172.21.15.45:0/1627127430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2fc0039b0 msgr2=0x7fa2fc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.274+0000 7fa31ca00640 1 --2- 172.21.15.45:0/1627127430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2fc0039b0 0x7fa2fc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.274+0000 7fa31ca00640 1 -- 172.21.15.45:0/1627127430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2fc003520 con 0x7fa2fc0051c0 2024-04-16T16:21:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.274+0000 7fa317400640 1 --2- 172.21.15.45:0/1627127430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2fc004800 0x7fa2fc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.275+0000 7fa31ca00640 1 -- 172.21.15.45:0/1627127430 reap_dead start 2024-04-16T16:21:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.275+0000 7fa31ca00640 1 --2- 172.21.15.45:0/1627127430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc0051c0 0x7fa2fc00b9c0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fa31000c4c0 tx=0x7fa2fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baecb2e527f9f3c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.275+0000 7fa316a00640 1 -- 172.21.15.45:0/1627127430 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31000c0e0 con 0x7fa2fc0051c0 2024-04-16T16:21:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.275+0000 7fa316a00640 1 -- 172.21.15.45:0/1627127430 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa310030070 con 0x7fa2fc0051c0 2024-04-16T16:21:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.275+0000 7fa316a00640 1 -- 172.21.15.45:0/1627127430 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3100058d0 con 0x7fa2fc0051c0 2024-04-16T16:21:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.276+0000 7fa31d400640 1 -- 172.21.15.45:0/1627127430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc0051c0 msgr2=0x7fa2fc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.276+0000 7fa31d400640 1 --2- 172.21.15.45:0/1627127430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc0051c0 0x7fa2fc00b9c0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fa31000c4c0 tx=0x7fa2fc005830 comp rx=0 tx=0).stop 2024-04-16T16:21:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.276+0000 7fa31d400640 1 -- 172.21.15.45:0/1627127430 shutdown_connections 2024-04-16T16:21:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.276+0000 7fa31d400640 1 --2- 172.21.15.45:0/1627127430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2fc0051c0 0x7fa2fc00b9c0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.276+0000 7fa31d400640 1 -- 172.21.15.45:0/1627127430 >> 172.21.15.45:0/1627127430 conn(0x7fa31806b930 msgr2=0x7fa31806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.276+0000 7fa31d400640 1 -- 172.21.15.45:0/1627127430 shutdown_connections 2024-04-16T16:21:53.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.276+0000 7fa31d400640 1 -- 172.21.15.45:0/1627127430 wait complete. 2024-04-16T16:21:53.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.278+0000 7fa31d400640 1 Processor -- start 2024-04-16T16:21:53.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.278+0000 7fa31d400640 1 -- start start 2024-04-16T16:21:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.279+0000 7fa31d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2fc0051c0 0x7fa3180b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.279+0000 7fa31d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa318156480 0x7fa318156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.279+0000 7fa31d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa318142cd0 0x7fa318143150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.279+0000 7fa31d400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa31815ec10 con 0x7fa318142cd0 2024-04-16T16:21:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.279+0000 7fa31d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa3180cb2b0 con 0x7fa318156480 2024-04-16T16:21:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.279+0000 7fa31d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa3180cac30 con 0x7fa2fc0051c0 2024-04-16T16:21:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.279+0000 7fa31ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa318142cd0 0x7fa318143150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.279+0000 7fa31ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa318142cd0 0x7fa318143150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37004/0 (socket says 172.21.15.45:37004) 2024-04-16T16:21:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.279+0000 7fa31ca00640 1 -- 172.21.15.45:0/996205730 learned_addr learned my addr 172.21.15.45:0/996205730 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.280+0000 7fa317e00640 1 --2- 172.21.15.45:0/996205730 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2fc0051c0 0x7fa3180b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.280+0000 7fa317400640 1 --2- 172.21.15.45:0/996205730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa318156480 0x7fa318156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.280+0000 7fa31ca00640 1 -- 172.21.15.45:0/996205730 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2fc0051c0 msgr2=0x7fa3180b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:53.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.280+0000 7fa31ca00640 1 --2- 172.21.15.45:0/996205730 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2fc0051c0 0x7fa3180b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:53.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.280+0000 7fa31ca00640 1 -- 172.21.15.45:0/996205730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa318156480 msgr2=0x7fa318156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:53.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.280+0000 7fa31ca00640 1 --2- 172.21.15.45:0/996205730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa318156480 0x7fa318156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:53.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.280+0000 7fa31ca00640 1 -- 172.21.15.45:0/996205730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa318148ff0 con 0x7fa318142cd0 2024-04-16T16:21:53.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.280+0000 7fa31ca00640 1 --2- 172.21.15.45:0/996205730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa318142cd0 0x7fa318143150 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fa31002c720 tx=0x7fa3180ce640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:53.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.281+0000 7fa30fe00640 1 -- 172.21.15.45:0/996205730 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa310039030 con 0x7fa318142cd0 2024-04-16T16:21:53.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.281+0000 7fa30fe00640 1 -- 172.21.15.45:0/996205730 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa310004bf0 con 0x7fa318142cd0 2024-04-16T16:21:53.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.281+0000 7fa30fe00640 1 -- 172.21.15.45:0/996205730 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa310001110 con 0x7fa318142cd0 2024-04-16T16:21:53.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.281+0000 7fa31d400640 1 -- 172.21.15.45:0/996205730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa318143690 con 0x7fa318142cd0 2024-04-16T16:21:53.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.281+0000 7fa31d400640 1 -- 172.21.15.45:0/996205730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa318149540 con 0x7fa318142cd0 2024-04-16T16:21:53.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.282+0000 7fa31d400640 1 -- 172.21.15.45:0/996205730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa318075780 con 0x7fa318142cd0 2024-04-16T16:21:53.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.283+0000 7fa30fe00640 1 -- 172.21.15.45:0/996205730 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7fa310007070 con 0x7fa318142cd0 2024-04-16T16:21:53.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.284+0000 7fa30fe00640 1 --2- 172.21.15.45:0/996205730 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fa2f406ca90 0x7fa2f406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:53.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.285+0000 7fa30fe00640 1 -- 172.21.15.45:0/996205730 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7fa3100acec0 con 0x7fa318142cd0 2024-04-16T16:21:53.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.285+0000 7fa317e00640 1 --2- 172.21.15.45:0/996205730 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fa2f406ca90 0x7fa2f406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:53.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.286+0000 7fa317e00640 1 --2- 172.21.15.45:0/996205730 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fa2f406ca90 0x7fa2f406ef50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa300009b40 tx=0x7fa300009190 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:53.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.289+0000 7fa30fe00640 1 -- 172.21.15.45:0/996205730 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa31815f1b0 con 0x7fa318142cd0 2024-04-16T16:21:53.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.523+0000 7fa31d400640 1 -- 172.21.15.45:0/996205730 --> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7fa3180b2800 con 0x7fa2f406ca90 2024-04-16T16:21:53.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.526+0000 7fa30fe00640 1 -- 172.21.15.45:0/996205730 <== mgr.6674 v2:172.21.15.103:6832/1506110537 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7fa3180b2800 con 0x7fa2f406ca90 2024-04-16T16:21:53.525 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:21:53.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.529+0000 7fa31d400640 1 -- 172.21.15.45:0/996205730 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fa2f406ca90 msgr2=0x7fa2f406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:53.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.529+0000 7fa31d400640 1 --2- 172.21.15.45:0/996205730 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fa2f406ca90 0x7fa2f406ef50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa300009b40 tx=0x7fa300009190 comp rx=0 tx=0).stop 2024-04-16T16:21:53.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.529+0000 7fa31d400640 1 -- 172.21.15.45:0/996205730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa318142cd0 msgr2=0x7fa318143150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:53.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.529+0000 7fa31d400640 1 --2- 172.21.15.45:0/996205730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa318142cd0 0x7fa318143150 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fa31002c720 tx=0x7fa3180ce640 comp rx=0 tx=0).stop 2024-04-16T16:21:53.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.530+0000 7fa31d400640 1 -- 172.21.15.45:0/996205730 shutdown_connections 2024-04-16T16:21:53.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.530+0000 7fa31d400640 1 --2- 172.21.15.45:0/996205730 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fa2f406ca90 0x7fa2f406ef50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:53.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.530+0000 7fa31d400640 1 --2- 172.21.15.45:0/996205730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa318142cd0 0x7fa318143150 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:53.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.530+0000 7fa31d400640 1 --2- 172.21.15.45:0/996205730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa318156480 0x7fa318156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:53.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.530+0000 7fa31d400640 1 --2- 172.21.15.45:0/996205730 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2fc0051c0 0x7fa3180b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:53.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.530+0000 7fa31d400640 1 -- 172.21.15.45:0/996205730 >> 172.21.15.45:0/996205730 conn(0x7fa31806b930 msgr2=0x7fa3181618f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:53.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.530+0000 7fa31d400640 1 -- 172.21.15.45:0/996205730 shutdown_connections 2024-04-16T16:21:53.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:53.530+0000 7fa31d400640 1 -- 172.21.15.45:0/996205730 wait complete. 2024-04-16T16:21:54.548 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:21:54.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.784+0000 7fe8c5400640 1 Processor -- start 2024-04-16T16:21:54.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.785+0000 7fe8c5400640 1 -- start start 2024-04-16T16:21:54.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.786+0000 7fe8c5400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe8c006f730 0x7fe8c00c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:54.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.786+0000 7fe8c5400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe8c00c6320 0x7fe8c006df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:54.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.786+0000 7fe8c5400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8c015d600 con 0x7fe8c00c6fc0 2024-04-16T16:21:54.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.786+0000 7fe8c5400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe8c00c5670 con 0x7fe8c00c6320 2024-04-16T16:21:54.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.787+0000 7fe8c5400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe8c015d8e0 con 0x7fe8c006f730 2024-04-16T16:21:54.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.787+0000 7fe8c4a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe8c00c6320 0x7fe8c006df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:54.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.787+0000 7fe8c4a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe8c00c6320 0x7fe8c006df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57672/0 (socket says 172.21.15.45:57672) 2024-04-16T16:21:54.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.787+0000 7fe8c4a00640 1 -- 172.21.15.45:0/1915130739 learned_addr learned my addr 172.21.15.45:0/1915130739 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:54.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.787+0000 7fe8c4a00640 1 -- 172.21.15.45:0/1915130739 >> v2:172.21.15.103:3300/0 conn(0x7fe8c006f730 msgr2=0x7fe8c00c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:54.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.787+0000 7fe8bf400640 1 --2- 172.21.15.45:0/1915130739 >> v2:172.21.15.103:3300/0 conn(0x7fe8c006f730 0x7fe8c00c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:54.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.787+0000 7fe8c4a00640 1 --2- 172.21.15.45:0/1915130739 >> v2:172.21.15.103:3300/0 conn(0x7fe8c006f730 0x7fe8c00c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:54.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.787+0000 7fe8c4a00640 1 -- 172.21.15.45:0/1915130739 >> v1:172.21.15.103:6789/0 conn(0x7fe8c00c6fc0 legacy=0x7fe8c015cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:54.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.787+0000 7fe8c4a00640 1 -- 172.21.15.45:0/1915130739 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c015dbc0 con 0x7fe8c00c6320 2024-04-16T16:21:54.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.787+0000 7fe8bf400640 1 --2- 172.21.15.45:0/1915130739 >> v2:172.21.15.103:3300/0 conn(0x7fe8c006f730 0x7fe8c00c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:54.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.789+0000 7fe8c4a00640 1 --2- 172.21.15.45:0/1915130739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8c00c6320 0x7fe8c006df40 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0073370 tx=0x7fe8b802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654825c2b22f21db server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:54.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.789+0000 7fe8bea00640 1 -- 172.21.15.45:0/1915130739 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b802c9f0 con 0x7fe8c00c6320 2024-04-16T16:21:54.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.789+0000 7fe8bea00640 1 -- 172.21.15.45:0/1915130739 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8b802cb90 con 0x7fe8c00c6320 2024-04-16T16:21:54.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.789+0000 7fe8bea00640 1 -- 172.21.15.45:0/1915130739 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b80343f0 con 0x7fe8c00c6320 2024-04-16T16:21:54.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.790+0000 7fe8c5400640 1 -- 172.21.15.45:0/1915130739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8c00c6320 msgr2=0x7fe8c006df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:54.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.790+0000 7fe8c5400640 1 --2- 172.21.15.45:0/1915130739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8c00c6320 0x7fe8c006df40 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0073370 tx=0x7fe8b802be50 comp rx=0 tx=0).stop 2024-04-16T16:21:54.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.790+0000 7fe8c5400640 1 -- 172.21.15.45:0/1915130739 shutdown_connections 2024-04-16T16:21:54.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.790+0000 7fe8c5400640 1 --2- 172.21.15.45:0/1915130739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8c00c6320 0x7fe8c006df40 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:54.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.790+0000 7fe8c5400640 1 --2- 172.21.15.45:0/1915130739 >> v2:172.21.15.103:3300/0 conn(0x7fe8c006f730 0x7fe8c00c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:54.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.790+0000 7fe8c5400640 1 -- 172.21.15.45:0/1915130739 >> 172.21.15.45:0/1915130739 conn(0x7fe8c006b930 msgr2=0x7fe8c006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:54.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.790+0000 7fe8c5400640 1 -- 172.21.15.45:0/1915130739 shutdown_connections 2024-04-16T16:21:54.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.790+0000 7fe8c5400640 1 -- 172.21.15.45:0/1915130739 wait complete. 2024-04-16T16:21:54.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.791+0000 7fe8c5400640 1 Processor -- start 2024-04-16T16:21:54.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.792+0000 7fe8c5400640 1 -- start start 2024-04-16T16:21:54.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.792+0000 7fe8c5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8c006f730 0x7fe8c00ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:54.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8c5400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8c00c6320 0x7fe8c0153c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:54.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8c5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8c00c6fc0 0x7fe8c0145b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:54.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8c5400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c00c7520 con 0x7fe8c006f730 2024-04-16T16:21:54.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8c5400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c00c73a0 con 0x7fe8c00c6fc0 2024-04-16T16:21:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8c5400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe8c006e480 con 0x7fe8c00c6320 2024-04-16T16:21:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8bfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8c006f730 0x7fe8c00ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8bfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8c006f730 0x7fe8c00ce2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57674/0 (socket says 172.21.15.45:57674) 2024-04-16T16:21:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8bfe00640 1 -- 172.21.15.45:0/2943739298 learned_addr learned my addr 172.21.15.45:0/2943739298 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8bf400640 1 --2- 172.21.15.45:0/2943739298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8c00c6320 0x7fe8c0153c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8bfe00640 1 -- 172.21.15.45:0/2943739298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8c00c6320 msgr2=0x7fe8c0153c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8c4a00640 1 --2- 172.21.15.45:0/2943739298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8c00c6fc0 0x7fe8c0145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8bfe00640 1 --2- 172.21.15.45:0/2943739298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8c00c6320 0x7fe8c0153c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8bfe00640 1 -- 172.21.15.45:0/2943739298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8c00c6fc0 msgr2=0x7fe8c0145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8bfe00640 1 --2- 172.21.15.45:0/2943739298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8c00c6fc0 0x7fe8c0145b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8bfe00640 1 -- 172.21.15.45:0/2943739298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c01576e0 con 0x7fe8c006f730 2024-04-16T16:21:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.793+0000 7fe8bf400640 1 --2- 172.21.15.45:0/2943739298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8c00c6320 0x7fe8c0153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.794+0000 7fe8bfe00640 1 --2- 172.21.15.45:0/2943739298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8c006f730 0x7fe8c00ce2e0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000b3f0 tx=0x7fe8b000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:54.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.794+0000 7fe8b7e00640 1 -- 172.21.15.45:0/2943739298 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0015020 con 0x7fe8c006f730 2024-04-16T16:21:54.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.794+0000 7fe8b7e00640 1 -- 172.21.15.45:0/2943739298 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8b0011830 con 0x7fe8c006f730 2024-04-16T16:21:54.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.794+0000 7fe8b7e00640 1 -- 172.21.15.45:0/2943739298 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0011c20 con 0x7fe8c006f730 2024-04-16T16:21:54.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.794+0000 7fe8c5400640 1 -- 172.21.15.45:0/2943739298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8c01579c0 con 0x7fe8c006f730 2024-04-16T16:21:54.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.794+0000 7fe8c5400640 1 -- 172.21.15.45:0/2943739298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8c0157f80 con 0x7fe8c006f730 2024-04-16T16:21:54.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.795+0000 7fe8c5400640 1 -- 172.21.15.45:0/2943739298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8c0075780 con 0x7fe8c006f730 2024-04-16T16:21:54.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.796+0000 7fe8b7e00640 1 -- 172.21.15.45:0/2943739298 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7fe8b0018020 con 0x7fe8c006f730 2024-04-16T16:21:54.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.797+0000 7fe8b7e00640 1 --2- 172.21.15.45:0/2943739298 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fe8a006cae0 0x7fe8a006efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:54.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.798+0000 7fe8b7e00640 1 -- 172.21.15.45:0/2943739298 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7fe8b008bb90 con 0x7fe8c006f730 2024-04-16T16:21:54.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.798+0000 7fe8bf400640 1 --2- 172.21.15.45:0/2943739298 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fe8a006cae0 0x7fe8a006efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:54.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.800+0000 7fe8bf400640 1 --2- 172.21.15.45:0/2943739298 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fe8a006cae0 0x7fe8a006efa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac006fd0 tx=0x7fe8ac008040 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:54.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:54.802+0000 7fe8b7e00640 1 -- 172.21.15.45:0/2943739298 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe8b001f030 con 0x7fe8c006f730 2024-04-16T16:21:55.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:55.034+0000 7fe8c5400640 1 -- 172.21.15.45:0/2943739298 --> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7fe8c00c5e80 con 0x7fe8a006cae0 2024-04-16T16:21:55.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:55.037+0000 7fe8b7e00640 1 -- 172.21.15.45:0/2943739298 <== mgr.6674 v2:172.21.15.103:6832/1506110537 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7fe8c00c5e80 con 0x7fe8a006cae0 2024-04-16T16:21:55.036 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:21:55.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:55.041+0000 7fe8c5400640 1 -- 172.21.15.45:0/2943739298 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fe8a006cae0 msgr2=0x7fe8a006efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:55.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:55.041+0000 7fe8c5400640 1 --2- 172.21.15.45:0/2943739298 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fe8a006cae0 0x7fe8a006efa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac006fd0 tx=0x7fe8ac008040 comp rx=0 tx=0).stop 2024-04-16T16:21:55.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:55.041+0000 7fe8c5400640 1 -- 172.21.15.45:0/2943739298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8c006f730 msgr2=0x7fe8c00ce2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:55.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:55.041+0000 7fe8c5400640 1 --2- 172.21.15.45:0/2943739298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8c006f730 0x7fe8c00ce2e0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000b3f0 tx=0x7fe8b000b8b0 comp rx=0 tx=0).stop 2024-04-16T16:21:55.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:55.041+0000 7fe8c5400640 1 -- 172.21.15.45:0/2943739298 shutdown_connections 2024-04-16T16:21:55.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:55.041+0000 7fe8c5400640 1 --2- 172.21.15.45:0/2943739298 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fe8a006cae0 0x7fe8a006efa0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:55.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:55.041+0000 7fe8c5400640 1 --2- 172.21.15.45:0/2943739298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8c00c6fc0 0x7fe8c0145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:55.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:55.041+0000 7fe8c5400640 1 --2- 172.21.15.45:0/2943739298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8c00c6320 0x7fe8c0153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:55.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:55.041+0000 7fe8c5400640 1 --2- 172.21.15.45:0/2943739298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8c006f730 0x7fe8c00ce2e0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:55.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:55.041+0000 7fe8c5400640 1 -- 172.21.15.45:0/2943739298 >> 172.21.15.45:0/2943739298 conn(0x7fe8c006b930 msgr2=0x7fe8c006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:55.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:55.041+0000 7fe8c5400640 1 -- 172.21.15.45:0/2943739298 shutdown_connections 2024-04-16T16:21:55.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:55.041+0000 7fe8c5400640 1 -- 172.21.15.45:0/2943739298 wait complete. 2024-04-16T16:21:55.418 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:21:55.426 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:21:56.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:21:56.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.297+0000 7fb399800640 1 Processor -- start 2024-04-16T16:21:56.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.298+0000 7fb399800640 1 -- start start 2024-04-16T16:21:56.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.300+0000 7fb399800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb39406f730 0x7fb3940c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:56.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.300+0000 7fb399800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb3940c6320 0x7fb39406df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:56.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.300+0000 7fb399800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb39415d600 con 0x7fb3940c6fc0 2024-04-16T16:21:56.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.300+0000 7fb399800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb3940c5670 con 0x7fb3940c6320 2024-04-16T16:21:56.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.300+0000 7fb399800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb39415d8e0 con 0x7fb39406f730 2024-04-16T16:21:56.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.300+0000 7fb393e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb3940c6fc0 0x7fb39415cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53592/0 (socket says 172.21.15.45:53592) 2024-04-16T16:21:56.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.300+0000 7fb393e00640 1 -- 172.21.15.45:0/2235263730 learned_addr learned my addr 172.21.15.45:0/2235263730 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:56.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.300+0000 7fb398e00640 1 --2- 172.21.15.45:0/2235263730 >> v2:172.21.15.45:3300/0 conn(0x7fb3940c6320 0x7fb39406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:56.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.300+0000 7fb38be00640 1 --2- 172.21.15.45:0/2235263730 >> v2:172.21.15.103:3300/0 conn(0x7fb39406f730 0x7fb3940c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:56.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.300+0000 7fb393400640 1 -- 172.21.15.45:0/2235263730 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 991757396 0 0) 0x7fb39415d600 con 0x7fb3940c6fc0 2024-04-16T16:21:56.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.301+0000 7fb393400640 1 -- 172.21.15.45:0/2235263730 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb378003520 con 0x7fb3940c6fc0 2024-04-16T16:21:56.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.301+0000 7fb398e00640 1 -- 172.21.15.45:0/2235263730 >> v2:172.21.15.103:3300/0 conn(0x7fb39406f730 msgr2=0x7fb3940c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:56.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.301+0000 7fb398e00640 1 --2- 172.21.15.45:0/2235263730 >> v2:172.21.15.103:3300/0 conn(0x7fb39406f730 0x7fb3940c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:56.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.301+0000 7fb398e00640 1 -- 172.21.15.45:0/2235263730 >> v1:172.21.15.45:6789/0 conn(0x7fb3940c6fc0 legacy=0x7fb39415cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:56.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.301+0000 7fb398e00640 1 -- 172.21.15.45:0/2235263730 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39415dbc0 con 0x7fb3940c6320 2024-04-16T16:21:56.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.302+0000 7fb398e00640 1 --2- 172.21.15.45:0/2235263730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3940c6320 0x7fb39406df40 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fb394073370 tx=0x7fb38c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9142b8ee21ed9f57 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:56.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.302+0000 7fb393400640 1 -- 172.21.15.45:0/2235263730 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c039070 con 0x7fb3940c6320 2024-04-16T16:21:56.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.302+0000 7fb393400640 1 -- 172.21.15.45:0/2235263730 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb38c02cbf0 con 0x7fb3940c6320 2024-04-16T16:21:56.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.302+0000 7fb393400640 1 -- 172.21.15.45:0/2235263730 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c034440 con 0x7fb3940c6320 2024-04-16T16:21:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.303+0000 7fb399800640 1 -- 172.21.15.45:0/2235263730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3940c6320 msgr2=0x7fb39406df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.303+0000 7fb399800640 1 --2- 172.21.15.45:0/2235263730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3940c6320 0x7fb39406df40 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fb394073370 tx=0x7fb38c02be40 comp rx=0 tx=0).stop 2024-04-16T16:21:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.303+0000 7fb399800640 1 -- 172.21.15.45:0/2235263730 shutdown_connections 2024-04-16T16:21:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.303+0000 7fb399800640 1 --2- 172.21.15.45:0/2235263730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3940c6320 0x7fb39406df40 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.303+0000 7fb399800640 1 --2- 172.21.15.45:0/2235263730 >> v2:172.21.15.103:3300/0 conn(0x7fb39406f730 0x7fb3940c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.303+0000 7fb399800640 1 -- 172.21.15.45:0/2235263730 >> 172.21.15.45:0/2235263730 conn(0x7fb39406b930 msgr2=0x7fb39406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.303+0000 7fb399800640 1 -- 172.21.15.45:0/2235263730 shutdown_connections 2024-04-16T16:21:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.303+0000 7fb399800640 1 -- 172.21.15.45:0/2235263730 wait complete. 2024-04-16T16:21:56.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.305+0000 7fb399800640 1 Processor -- start 2024-04-16T16:21:56.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.305+0000 7fb399800640 1 -- start start 2024-04-16T16:21:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.306+0000 7fb399800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb39406f730 0x7fb3941f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.306+0000 7fb399800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3940c6320 0x7fb394152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.306+0000 7fb399800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3940c6fc0 0x7fb3940cef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.306+0000 7fb399800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb3940c7520 con 0x7fb3940c6320 2024-04-16T16:21:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.306+0000 7fb399800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb3940c73a0 con 0x7fb39406f730 2024-04-16T16:21:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.306+0000 7fb399800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb39406e480 con 0x7fb3940c6fc0 2024-04-16T16:21:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.306+0000 7fb38be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3940c6320 0x7fb394152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.306+0000 7fb38be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3940c6320 0x7fb394152430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57694/0 (socket says 172.21.15.45:57694) 2024-04-16T16:21:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.306+0000 7fb38be00640 1 -- 172.21.15.45:0/2852314325 learned_addr learned my addr 172.21.15.45:0/2852314325 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:56.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.306+0000 7fb393e00640 1 --2- 172.21.15.45:0/2852314325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb39406f730 0x7fb3941f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:56.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.307+0000 7fb38be00640 1 -- 172.21.15.45:0/2852314325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3940c6fc0 msgr2=0x7fb3940cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:56.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.307+0000 7fb38be00640 1 --2- 172.21.15.45:0/2852314325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3940c6fc0 0x7fb3940cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.307+0000 7fb38be00640 1 -- 172.21.15.45:0/2852314325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb39406f730 msgr2=0x7fb3941f3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.307+0000 7fb38be00640 1 --2- 172.21.15.45:0/2852314325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb39406f730 0x7fb3941f3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.307+0000 7fb38be00640 1 -- 172.21.15.45:0/2852314325 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb394152da0 con 0x7fb3940c6320 2024-04-16T16:21:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.307+0000 7fb38be00640 1 --2- 172.21.15.45:0/2852314325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3940c6320 0x7fb394152430 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00d800 tx=0x7fb37c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.307+0000 7fb390c00640 1 -- 172.21.15.45:0/2852314325 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c00b6c0 con 0x7fb3940c6320 2024-04-16T16:21:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.307+0000 7fb390c00640 1 -- 172.21.15.45:0/2852314325 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb37c019070 con 0x7fb3940c6320 2024-04-16T16:21:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.307+0000 7fb390c00640 1 -- 172.21.15.45:0/2852314325 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c00ba80 con 0x7fb3940c6320 2024-04-16T16:21:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.307+0000 7fb399800640 1 -- 172.21.15.45:0/2852314325 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb394153080 con 0x7fb3940c6320 2024-04-16T16:21:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.307+0000 7fb399800640 1 -- 172.21.15.45:0/2852314325 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb394153640 con 0x7fb3940c6320 2024-04-16T16:21:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.309+0000 7fb399800640 1 -- 172.21.15.45:0/2852314325 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb394153b30 con 0x7fb3940c6320 2024-04-16T16:21:56.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.310+0000 7fb390c00640 1 -- 172.21.15.45:0/2852314325 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7fb37c00bc20 con 0x7fb3940c6320 2024-04-16T16:21:56.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.310+0000 7fb390c00640 1 --2- 172.21.15.45:0/2852314325 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fb37006ca90 0x7fb37006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:56.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.311+0000 7fb390c00640 1 -- 172.21.15.45:0/2852314325 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7fb37c08c410 con 0x7fb3940c6320 2024-04-16T16:21:56.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.311+0000 7fb393e00640 1 --2- 172.21.15.45:0/2852314325 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fb37006ca90 0x7fb37006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:56.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.313+0000 7fb393e00640 1 --2- 172.21.15.45:0/2852314325 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fb37006ca90 0x7fb37006ef50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb384005880 tx=0x7fb384009040 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:56.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.315+0000 7fb390c00640 1 -- 172.21.15.45:0/2852314325 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb37c01c360 con 0x7fb3940c6320 2024-04-16T16:21:56.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.545+0000 7fb399800640 1 -- 172.21.15.45:0/2852314325 --> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7fb3941f4060 con 0x7fb37006ca90 2024-04-16T16:21:56.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.549+0000 7fb390c00640 1 -- 172.21.15.45:0/2852314325 <== mgr.6674 v2:172.21.15.103:6832/1506110537 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7fb3941f4060 con 0x7fb37006ca90 2024-04-16T16:21:56.548 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:21:56.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.551+0000 7fb399800640 1 -- 172.21.15.45:0/2852314325 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fb37006ca90 msgr2=0x7fb37006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:56.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.551+0000 7fb399800640 1 --2- 172.21.15.45:0/2852314325 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fb37006ca90 0x7fb37006ef50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb384005880 tx=0x7fb384009040 comp rx=0 tx=0).stop 2024-04-16T16:21:56.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.551+0000 7fb399800640 1 -- 172.21.15.45:0/2852314325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3940c6320 msgr2=0x7fb394152430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:56.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.551+0000 7fb399800640 1 --2- 172.21.15.45:0/2852314325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3940c6320 0x7fb394152430 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00d800 tx=0x7fb37c00dcc0 comp rx=0 tx=0).stop 2024-04-16T16:21:56.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.551+0000 7fb399800640 1 -- 172.21.15.45:0/2852314325 shutdown_connections 2024-04-16T16:21:56.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.551+0000 7fb399800640 1 --2- 172.21.15.45:0/2852314325 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fb37006ca90 0x7fb37006ef50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:56.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.551+0000 7fb399800640 1 --2- 172.21.15.45:0/2852314325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3940c6fc0 0x7fb3940cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:56.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.551+0000 7fb399800640 1 --2- 172.21.15.45:0/2852314325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3940c6320 0x7fb394152430 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:56.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.551+0000 7fb399800640 1 --2- 172.21.15.45:0/2852314325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb39406f730 0x7fb3941f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:56.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.551+0000 7fb399800640 1 -- 172.21.15.45:0/2852314325 >> 172.21.15.45:0/2852314325 conn(0x7fb39406b930 msgr2=0x7fb39406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:56.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.552+0000 7fb399800640 1 -- 172.21.15.45:0/2852314325 shutdown_connections 2024-04-16T16:21:56.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:56.552+0000 7fb399800640 1 -- 172.21.15.45:0/2852314325 wait complete. 2024-04-16T16:21:57.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:21:57.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.805+0000 7f023a800640 1 Processor -- start 2024-04-16T16:21:57.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.806+0000 7f023a800640 1 -- start start 2024-04-16T16:21:57.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.807+0000 7f023a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f023415e830 0x7f02340ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:57.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.807+0000 7f023a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02340cbc80 0x7f02340cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:57.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.807+0000 7f023a800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023415f1b0 con 0x7f02340cb7d0 2024-04-16T16:21:57.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.807+0000 7f023a800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f02340cf280 con 0x7f023415e830 2024-04-16T16:21:57.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.807+0000 7f023a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f02340ce190 con 0x7f02340cbc80 2024-04-16T16:21:57.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.808+0000 7f0239e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02340cbc80 0x7f02340cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:57.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.808+0000 7f0239e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02340cbc80 0x7f02340cc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:33062/0 (socket says 172.21.15.45:33062) 2024-04-16T16:21:57.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.808+0000 7f0239e00640 1 -- 172.21.15.45:0/403435957 learned_addr learned my addr 172.21.15.45:0/403435957 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:57.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.808+0000 7f0239400640 1 --2- 172.21.15.45:0/403435957 >> v2:172.21.15.103:3300/0 conn(0x7f023415e830 0x7f02340ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:57.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.809+0000 7f0239e00640 1 -- 172.21.15.45:0/403435957 >> v2:172.21.15.103:3300/0 conn(0x7f023415e830 msgr2=0x7f02340ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:57.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.809+0000 7f0239e00640 1 --2- 172.21.15.45:0/403435957 >> v2:172.21.15.103:3300/0 conn(0x7f023415e830 0x7f02340ce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:57.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.809+0000 7f0239e00640 1 -- 172.21.15.45:0/403435957 >> v1:172.21.15.103:6789/0 conn(0x7f02340cb7d0 legacy=0x7f02340cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:57.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.809+0000 7f0239e00640 1 -- 172.21.15.45:0/403435957 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02340ce470 con 0x7f02340cbc80 2024-04-16T16:21:57.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.810+0000 7f0239e00640 1 --2- 172.21.15.45:0/403435957 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02340cbc80 0x7f02340cc410 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f022c009870 tx=0x7f022c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b72b6748287af2a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:57.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.811+0000 7f0233e00640 1 -- 172.21.15.45:0/403435957 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c035020 con 0x7f02340cbc80 2024-04-16T16:21:57.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.811+0000 7f0233e00640 1 -- 172.21.15.45:0/403435957 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f022c031640 con 0x7f02340cbc80 2024-04-16T16:21:57.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.811+0000 7f0233e00640 1 -- 172.21.15.45:0/403435957 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c02d040 con 0x7f02340cbc80 2024-04-16T16:21:57.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.812+0000 7f023a800640 1 -- 172.21.15.45:0/403435957 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02340cbc80 msgr2=0x7f02340cc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:57.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.812+0000 7f023a800640 1 --2- 172.21.15.45:0/403435957 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02340cbc80 0x7f02340cc410 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f022c009870 tx=0x7f022c02b8f0 comp rx=0 tx=0).stop 2024-04-16T16:21:57.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.812+0000 7f023a800640 1 -- 172.21.15.45:0/403435957 shutdown_connections 2024-04-16T16:21:57.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.812+0000 7f023a800640 1 --2- 172.21.15.45:0/403435957 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02340cbc80 0x7f02340cc410 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:57.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.812+0000 7f023a800640 1 --2- 172.21.15.45:0/403435957 >> v2:172.21.15.103:3300/0 conn(0x7f023415e830 0x7f02340ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:57.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.812+0000 7f023a800640 1 -- 172.21.15.45:0/403435957 >> 172.21.15.45:0/403435957 conn(0x7f023406b930 msgr2=0x7f023406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:57.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.812+0000 7f023a800640 1 -- 172.21.15.45:0/403435957 shutdown_connections 2024-04-16T16:21:57.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.812+0000 7f023a800640 1 -- 172.21.15.45:0/403435957 wait complete. 2024-04-16T16:21:57.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.814+0000 7f023a800640 1 Processor -- start 2024-04-16T16:21:57.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.814+0000 7f023a800640 1 -- start start 2024-04-16T16:21:57.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.815+0000 7f023a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02340cb7d0 0x7f02340b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:57.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.815+0000 7f023a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02340cbc80 0x7f0234153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:57.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.815+0000 7f023a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023415e830 0x7f02341f1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:57.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.815+0000 7f023a800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f023415ec10 con 0x7f023415e830 2024-04-16T16:21:57.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.815+0000 7f023a800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f02340cb2b0 con 0x7f02340cb7d0 2024-04-16T16:21:57.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.815+0000 7f023a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f02340cac30 con 0x7f02340cbc80 2024-04-16T16:21:57.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.815+0000 7f0239e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023415e830 0x7f02341f1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:57.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.815+0000 7f0239e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023415e830 0x7f02341f1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57696/0 (socket says 172.21.15.45:57696) 2024-04-16T16:21:57.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.815+0000 7f0239e00640 1 -- 172.21.15.45:0/1638177679 learned_addr learned my addr 172.21.15.45:0/1638177679 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:57.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.816+0000 7f0238a00640 1 --2- 172.21.15.45:0/1638177679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02340cbc80 0x7f0234153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:57.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.816+0000 7f0239400640 1 --2- 172.21.15.45:0/1638177679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02340cb7d0 0x7f02340b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:57.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.816+0000 7f0239e00640 1 -- 172.21.15.45:0/1638177679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02340cbc80 msgr2=0x7f0234153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:57.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.816+0000 7f0239e00640 1 --2- 172.21.15.45:0/1638177679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02340cbc80 0x7f0234153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:57.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.816+0000 7f0239e00640 1 -- 172.21.15.45:0/1638177679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02340cb7d0 msgr2=0x7f02340b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:57.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.816+0000 7f0239e00640 1 --2- 172.21.15.45:0/1638177679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02340cb7d0 0x7f02340b4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:57.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.816+0000 7f0239e00640 1 -- 172.21.15.45:0/1638177679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02341f4880 con 0x7f023415e830 2024-04-16T16:21:57.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.816+0000 7f0239e00640 1 --2- 172.21.15.45:0/1638177679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023415e830 0x7f02341f1480 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f022c02b930 tx=0x7f022c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.817+0000 7f0231600640 1 -- 172.21.15.45:0/1638177679 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c035040 con 0x7f023415e830 2024-04-16T16:21:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.817+0000 7f0231600640 1 -- 172.21.15.45:0/1638177679 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f022c039b10 con 0x7f023415e830 2024-04-16T16:21:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.817+0000 7f0231600640 1 -- 172.21.15.45:0/1638177679 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c045480 con 0x7f023415e830 2024-04-16T16:21:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.817+0000 7f023a800640 1 -- 172.21.15.45:0/1638177679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02341f4b60 con 0x7f023415e830 2024-04-16T16:21:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.817+0000 7f023a800640 1 -- 172.21.15.45:0/1638177679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02341f5120 con 0x7f023415e830 2024-04-16T16:21:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.818+0000 7f023a800640 1 -- 172.21.15.45:0/1638177679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0234075780 con 0x7f023415e830 2024-04-16T16:21:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.819+0000 7f0231600640 1 -- 172.21.15.45:0/1638177679 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7f022c038020 con 0x7f023415e830 2024-04-16T16:21:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.820+0000 7f0231600640 1 --2- 172.21.15.45:0/1638177679 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f021406ca60 0x7f021406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.820+0000 7f0231600640 1 -- 172.21.15.45:0/1638177679 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f022c0adcc0 con 0x7f023415e830 2024-04-16T16:21:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.821+0000 7f0239400640 1 --2- 172.21.15.45:0/1638177679 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f021406ca60 0x7f021406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.822+0000 7f0239400640 1 --2- 172.21.15.45:0/1638177679 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f021406ca60 0x7f021406ef20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0228009a60 tx=0x7f0228003990 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:57.825+0000 7f0231600640 1 -- 172.21.15.45:0/1638177679 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f022c0b4080 con 0x7f023415e830 2024-04-16T16:21:58.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:58.058+0000 7f023a800640 1 -- 172.21.15.45:0/1638177679 --> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f02340b5b90 con 0x7f021406ca60 2024-04-16T16:21:58.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:58.061+0000 7f0231600640 1 -- 172.21.15.45:0/1638177679 <== mgr.6674 v2:172.21.15.103:6832/1506110537 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+431 (secure 0 0 0) 0x7f02340b5b90 con 0x7f021406ca60 2024-04-16T16:21:58.060 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} 2024-04-16T16:21:58.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:58.064+0000 7f023a800640 1 -- 172.21.15.45:0/1638177679 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f021406ca60 msgr2=0x7f021406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:58.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:58.064+0000 7f023a800640 1 --2- 172.21.15.45:0/1638177679 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f021406ca60 0x7f021406ef20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0228009a60 tx=0x7f0228003990 comp rx=0 tx=0).stop 2024-04-16T16:21:58.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:58.064+0000 7f023a800640 1 -- 172.21.15.45:0/1638177679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023415e830 msgr2=0x7f02341f1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:58.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:58.064+0000 7f023a800640 1 --2- 172.21.15.45:0/1638177679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023415e830 0x7f02341f1480 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f022c02b930 tx=0x7f022c0046c0 comp rx=0 tx=0).stop 2024-04-16T16:21:58.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:58.064+0000 7f023a800640 1 -- 172.21.15.45:0/1638177679 shutdown_connections 2024-04-16T16:21:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:58.064+0000 7f023a800640 1 --2- 172.21.15.45:0/1638177679 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f021406ca60 0x7f021406ef20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:58.064+0000 7f023a800640 1 --2- 172.21.15.45:0/1638177679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023415e830 0x7f02341f1480 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:58.064+0000 7f023a800640 1 --2- 172.21.15.45:0/1638177679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02340cbc80 0x7f0234153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:58.064+0000 7f023a800640 1 --2- 172.21.15.45:0/1638177679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02340cb7d0 0x7f02340b4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:58.064+0000 7f023a800640 1 -- 172.21.15.45:0/1638177679 >> 172.21.15.45:0/1638177679 conn(0x7f023406b930 msgr2=0x7f0234160560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:58.065+0000 7f023a800640 1 -- 172.21.15.45:0/1638177679 shutdown_connections 2024-04-16T16:21:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:58.065+0000 7f023a800640 1 -- 172.21.15.45:0/1638177679 wait complete. 2024-04-16T16:21:59.085 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:21:59.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.319+0000 7fe2efe00640 1 Processor -- start 2024-04-16T16:21:59.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.319+0000 7fe2efe00640 1 -- start start 2024-04-16T16:21:59.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.321+0000 7fe2efe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2f00ba140 0x7fe2f00d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:59.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.321+0000 7fe2efe00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f00d2600 con 0x7fe2f0162090 2024-04-16T16:21:59.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.321+0000 7fe2efe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f00ba5d0 con 0x7fe2f00cfc20 2024-04-16T16:21:59.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.321+0000 7fe2efe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f00ba8b0 con 0x7fe2f00ba140 2024-04-16T16:21:59.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.321+0000 7fe2eea00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe2f0162090 0x7fe2f00cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53608/0 (socket says 172.21.15.45:53608) 2024-04-16T16:21:59.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.321+0000 7fe2eea00640 1 -- 172.21.15.45:0/1935091938 learned_addr learned my addr 172.21.15.45:0/1935091938 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:21:59.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.321+0000 7fe2ee000640 1 --2- 172.21.15.45:0/1935091938 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2f00ba140 0x7fe2f00d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:59.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.321+0000 7fe2ed600640 1 -- 172.21.15.45:0/1935091938 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 951742979 0 0) 0x7fe2f00d2600 con 0x7fe2f0162090 2024-04-16T16:21:59.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.322+0000 7fe2ed600640 1 -- 172.21.15.45:0/1935091938 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2d4003520 con 0x7fe2f0162090 2024-04-16T16:21:59.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.322+0000 7fe2ed600640 1 -- 172.21.15.45:0/1935091938 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3918368693 0 0) 0x7fe2d4003520 con 0x7fe2f0162090 2024-04-16T16:21:59.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.322+0000 7fe2ed600640 1 -- 172.21.15.45:0/1935091938 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2f00ba140 msgr2=0x7fe2f00d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:59.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.322+0000 7fe2ed600640 1 --2- 172.21.15.45:0/1935091938 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2f00ba140 0x7fe2f00d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:59.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.322+0000 7fe2ed600640 1 -- 172.21.15.45:0/1935091938 >> v1:172.21.15.103:6789/0 conn(0x7fe2f00cfc20 legacy=0x7fe2f00d35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:59.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.322+0000 7fe2ed600640 1 -- 172.21.15.45:0/1935091938 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f00d28a0 con 0x7fe2f0162090 2024-04-16T16:21:59.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.322+0000 7fe2ed600640 1 -- 172.21.15.45:0/1935091938 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fe2e0001670 con 0x7fe2f0162090 2024-04-16T16:21:59.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.322+0000 7fe2ed600640 1 -- 172.21.15.45:0/1935091938 >> v1:172.21.15.45:6789/0 conn(0x7fe2f0162090 legacy=0x7fe2f00cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:59.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.322+0000 7fe2ed600640 1 --2- 172.21.15.45:0/1935091938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d40039e0 0x7fe2d4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:59.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.322+0000 7fe2ed600640 1 --2- 172.21.15.45:0/1935091938 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d4004800 0x7fe2d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:59.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.322+0000 7fe2ed600640 1 --2- 172.21.15.45:0/1935091938 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4005180 0x7fe2d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:59.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.322+0000 7fe2ed600640 1 -- 172.21.15.45:0/1935091938 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f00ba8b0 con 0x7fe2d4005180 2024-04-16T16:21:59.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.322+0000 7fe2ed600640 1 -- 172.21.15.45:0/1935091938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d400bf00 con 0x7fe2d40039e0 2024-04-16T16:21:59.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.322+0000 7fe2ed600640 1 -- 172.21.15.45:0/1935091938 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d400c080 con 0x7fe2d4004800 2024-04-16T16:21:59.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.322+0000 7fe2ef400640 1 --2- 172.21.15.45:0/1935091938 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4005180 0x7fe2d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:59.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.323+0000 7fe2eea00640 1 --2- 172.21.15.45:0/1935091938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d40039e0 0x7fe2d4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:59.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.323+0000 7fe2ee000640 1 --2- 172.21.15.45:0/1935091938 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d4004800 0x7fe2d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:59.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.323+0000 7fe2ef400640 1 -- 172.21.15.45:0/1935091938 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d4004800 msgr2=0x7fe2d4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:59.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.323+0000 7fe2ef400640 1 --2- 172.21.15.45:0/1935091938 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d4004800 0x7fe2d4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:59.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.323+0000 7fe2ef400640 1 -- 172.21.15.45:0/1935091938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d40039e0 msgr2=0x7fe2d4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:59.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.323+0000 7fe2ef400640 1 --2- 172.21.15.45:0/1935091938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d40039e0 0x7fe2d4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:59.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.323+0000 7fe2ef400640 1 -- 172.21.15.45:0/1935091938 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d4003520 con 0x7fe2d4005180 2024-04-16T16:21:59.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.323+0000 7fe2ee000640 1 --2- 172.21.15.45:0/1935091938 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d4004800 0x7fe2d4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:59.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.323+0000 7fe2eea00640 1 --2- 172.21.15.45:0/1935091938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d40039e0 0x7fe2d4003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:21:59.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.324+0000 7fe2ef400640 1 --2- 172.21.15.45:0/1935091938 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4005180 0x7fe2d400b9c0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fe2f008ce70 tx=0x7fe2d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7801a3698754743 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:59.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.324+0000 7fe2ed600640 1 -- 172.21.15.45:0/1935091938 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8001a70 con 0x7fe2d4005180 2024-04-16T16:21:59.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.324+0000 7fe2ed600640 1 -- 172.21.15.45:0/1935091938 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e8003b70 con 0x7fe2d4005180 2024-04-16T16:21:59.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.324+0000 7fe2ed600640 1 -- 172.21.15.45:0/1935091938 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e80050d0 con 0x7fe2d4005180 2024-04-16T16:21:59.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.325+0000 7fe2efe00640 1 -- 172.21.15.45:0/1935091938 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4005180 msgr2=0x7fe2d400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:59.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.325+0000 7fe2efe00640 1 --2- 172.21.15.45:0/1935091938 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2d4005180 0x7fe2d400b9c0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fe2f008ce70 tx=0x7fe2d4005830 comp rx=0 tx=0).stop 2024-04-16T16:21:59.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.325+0000 7fe2ef400640 1 -- 172.21.15.45:0/1935091938 reap_dead start 2024-04-16T16:21:59.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.325+0000 7fe2efe00640 1 -- 172.21.15.45:0/1935091938 shutdown_connections 2024-04-16T16:21:59.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.325+0000 7fe2efe00640 1 -- 172.21.15.45:0/1935091938 >> 172.21.15.45:0/1935091938 conn(0x7fe2f00d1940 msgr2=0x7fe2f00d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:59.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.325+0000 7fe2efe00640 1 -- 172.21.15.45:0/1935091938 shutdown_connections 2024-04-16T16:21:59.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.325+0000 7fe2efe00640 1 -- 172.21.15.45:0/1935091938 wait complete. 2024-04-16T16:21:59.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.327+0000 7fe2efe00640 1 Processor -- start 2024-04-16T16:21:59.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.327+0000 7fe2efe00640 1 -- start start 2024-04-16T16:21:59.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.328+0000 7fe2efe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d4005180 0x7fe2f00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:59.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.328+0000 7fe2efe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f00ce950 0x7fe2f00b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:59.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.328+0000 7fe2efe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f01f7a20 0x7fe2f01f7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:59.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.328+0000 7fe2efe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f01625f0 con 0x7fe2f01f7a20 2024-04-16T16:21:59.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.328+0000 7fe2efe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f0162470 con 0x7fe2f00ce950 2024-04-16T16:21:59.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.328+0000 7fe2efe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f00d5520 con 0x7fe2d4005180 2024-04-16T16:21:59.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.328+0000 7fe2ef400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f01f7a20 0x7fe2f01f7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:59.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.328+0000 7fe2ef400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f01f7a20 0x7fe2f01f7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57716/0 (socket says 172.21.15.45:57716) 2024-04-16T16:21:59.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.328+0000 7fe2ef400640 1 -- 172.21.15.45:0/3294016461 learned_addr learned my addr 172.21.15.45:0/3294016461 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:59.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.329+0000 7fe2ee000640 1 --2- 172.21.15.45:0/3294016461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f00ce950 0x7fe2f00b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:59.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.329+0000 7fe2eea00640 1 --2- 172.21.15.45:0/3294016461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d4005180 0x7fe2f00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:59.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.329+0000 7fe2ef400640 1 -- 172.21.15.45:0/3294016461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d4005180 msgr2=0x7fe2f00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:59.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.329+0000 7fe2ef400640 1 --2- 172.21.15.45:0/3294016461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d4005180 0x7fe2f00cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:59.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.329+0000 7fe2ef400640 1 -- 172.21.15.45:0/3294016461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f00ce950 msgr2=0x7fe2f00b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:59.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.329+0000 7fe2ef400640 1 --2- 172.21.15.45:0/3294016461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f00ce950 0x7fe2f00b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:59.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.329+0000 7fe2ef400640 1 -- 172.21.15.45:0/3294016461 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f00b57c0 con 0x7fe2f01f7a20 2024-04-16T16:21:59.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.329+0000 7fe2ef400640 1 --2- 172.21.15.45:0/3294016461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f01f7a20 0x7fe2f01f7eb0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8001ee0 tx=0x7fe2e80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:59.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.329+0000 7fe2e6a00640 1 -- 172.21.15.45:0/3294016461 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8007db0 con 0x7fe2f01f7a20 2024-04-16T16:21:59.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.330+0000 7fe2e6a00640 1 -- 172.21.15.45:0/3294016461 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e8003970 con 0x7fe2f01f7a20 2024-04-16T16:21:59.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.330+0000 7fe2e6a00640 1 -- 172.21.15.45:0/3294016461 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8005330 con 0x7fe2f01f7a20 2024-04-16T16:21:59.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.330+0000 7fe2efe00640 1 -- 172.21.15.45:0/3294016461 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f00b5aa0 con 0x7fe2f01f7a20 2024-04-16T16:21:59.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.330+0000 7fe2efe00640 1 -- 172.21.15.45:0/3294016461 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f00cb2e0 con 0x7fe2f01f7a20 2024-04-16T16:21:59.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.331+0000 7fe2efe00640 1 -- 172.21.15.45:0/3294016461 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f0075780 con 0x7fe2f01f7a20 2024-04-16T16:21:59.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.332+0000 7fe2e6a00640 1 -- 172.21.15.45:0/3294016461 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7fe2e80054d0 con 0x7fe2f01f7a20 2024-04-16T16:21:59.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.333+0000 7fe2e6a00640 1 --2- 172.21.15.45:0/3294016461 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fe2cc06ca90 0x7fe2cc06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:59.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.333+0000 7fe2e6a00640 1 -- 172.21.15.45:0/3294016461 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7fe2e80acd00 con 0x7fe2f01f7a20 2024-04-16T16:21:59.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.333+0000 7fe2eea00640 1 --2- 172.21.15.45:0/3294016461 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fe2cc06ca90 0x7fe2cc06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:59.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.335+0000 7fe2eea00640 1 --2- 172.21.15.45:0/3294016461 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fe2cc06ca90 0x7fe2cc06ef50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0001eb0 tx=0x7fe2e0001150 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:59.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.337+0000 7fe2e6a00640 1 -- 172.21.15.45:0/3294016461 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe2e8033020 con 0x7fe2f01f7a20 2024-04-16T16:21:59.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.565+0000 7fe2efe00640 1 -- 172.21.15.45:0/3294016461 --> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7fe2f006dab0 con 0x7fe2cc06ca90 2024-04-16T16:21:59.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.568+0000 7fe2e6a00640 1 -- 172.21.15.45:0/3294016461 <== mgr.6674 v2:172.21.15.103:6832/1506110537 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+2954 (secure 0 0 0) 0x7fe2f006dab0 con 0x7fe2cc06ca90 2024-04-16T16:21:59.567 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.6659": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "192.168.0.1"}, "client.6710": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.7498": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.9348": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "192.168.0.1"}}}, "global_metrics": {"cephfs": {"client.6659": [[24, 0], [0, 0], [0, 0], [0, 8994062], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 2248517], [37136336141620, 4]], "client.6710": [[2, 0], [0, 0], [0, 0], [0, 980564], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 980564], [0, 1]], "client.7498": [[2, 0], [0, 0], [0, 0], [0, 592826], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 592826], [0, 1]], "client.9348": [[5, 19], [0, 0], [0, 0], [0, 30535869], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1272333], [36535092943149, 24]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.6659": [], "client.6710": [], "client.7498": [], "client.9348": []}}} 2024-04-16T16:21:59.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.571+0000 7fe2efe00640 1 -- 172.21.15.45:0/3294016461 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fe2cc06ca90 msgr2=0x7fe2cc06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:59.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.571+0000 7fe2efe00640 1 --2- 172.21.15.45:0/3294016461 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fe2cc06ca90 0x7fe2cc06ef50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0001eb0 tx=0x7fe2e0001150 comp rx=0 tx=0).stop 2024-04-16T16:21:59.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.571+0000 7fe2efe00640 1 -- 172.21.15.45:0/3294016461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f01f7a20 msgr2=0x7fe2f01f7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:59.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.571+0000 7fe2efe00640 1 --2- 172.21.15.45:0/3294016461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f01f7a20 0x7fe2f01f7eb0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8001ee0 tx=0x7fe2e80014a0 comp rx=0 tx=0).stop 2024-04-16T16:21:59.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.571+0000 7fe2efe00640 1 -- 172.21.15.45:0/3294016461 shutdown_connections 2024-04-16T16:21:59.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.571+0000 7fe2efe00640 1 --2- 172.21.15.45:0/3294016461 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fe2cc06ca90 0x7fe2cc06ef50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:59.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.571+0000 7fe2efe00640 1 --2- 172.21.15.45:0/3294016461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f01f7a20 0x7fe2f01f7eb0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:59.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.571+0000 7fe2efe00640 1 --2- 172.21.15.45:0/3294016461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f00ce950 0x7fe2f00b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:59.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.571+0000 7fe2efe00640 1 --2- 172.21.15.45:0/3294016461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d4005180 0x7fe2f00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:59.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.571+0000 7fe2efe00640 1 -- 172.21.15.45:0/3294016461 >> 172.21.15.45:0/3294016461 conn(0x7fe2f00d1940 msgr2=0x7fe2f015c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:59.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.571+0000 7fe2efe00640 1 -- 172.21.15.45:0/3294016461 shutdown_connections 2024-04-16T16:21:59.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.571+0000 7fe2efe00640 1 -- 172.21.15.45:0/3294016461 wait complete. 2024-04-16T16:21:59.588 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:21:59.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.809+0000 7f3839200640 1 Processor -- start 2024-04-16T16:21:59.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.810+0000 7f3839200640 1 -- start start 2024-04-16T16:21:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.811+0000 7f3839200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3834146220 0x7f383415ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.811+0000 7f3839200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f383415d360 con 0x7f383406f7a0 2024-04-16T16:21:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.811+0000 7f3839200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f383415d640 con 0x7f38341608a0 2024-04-16T16:21:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.811+0000 7f3839200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3834160410 con 0x7f3834146220 2024-04-16T16:21:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.811+0000 7f3833400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3834146220 0x7f383415ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.811+0000 7f3833400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3834146220 0x7f383415ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57726/0 (socket says 172.21.15.45:57726) 2024-04-16T16:21:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.811+0000 7f3833400640 1 -- 172.21.15.45:0/2887601410 learned_addr learned my addr 172.21.15.45:0/2887601410 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.811+0000 7f3832a00640 1 --1- 172.21.15.45:0/2887601410 >> v1:172.21.15.45:6789/0 conn(0x7f383406f7a0 0x7f383406df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53616/0 (socket says 172.21.15.45:53616) 2024-04-16T16:21:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.812+0000 7f3833400640 1 -- 172.21.15.45:0/2887601410 >> v1:172.21.15.103:6789/0 conn(0x7f38341608a0 legacy=0x7f38340cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:21:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.812+0000 7f3833400640 1 -- 172.21.15.45:0/2887601410 >> v1:172.21.15.45:6789/0 conn(0x7f383406f7a0 legacy=0x7f383406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.812+0000 7f3833400640 1 -- 172.21.15.45:0/2887601410 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383415d920 con 0x7f3834146220 2024-04-16T16:21:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.813+0000 7f3833400640 1 --2- 172.21.15.45:0/2887601410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3834146220 0x7f383415ccb0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f3824009870 tx=0x7f382402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3279c327661056fe server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.813+0000 7f3832000640 1 -- 172.21.15.45:0/2887601410 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3824034070 con 0x7f3834146220 2024-04-16T16:21:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.813+0000 7f3832000640 1 -- 172.21.15.45:0/2887601410 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3824030850 con 0x7f3834146220 2024-04-16T16:21:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.813+0000 7f3832000640 1 -- 172.21.15.45:0/2887601410 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3824030c20 con 0x7f3834146220 2024-04-16T16:21:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.814+0000 7f3839200640 1 -- 172.21.15.45:0/2887601410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3834146220 msgr2=0x7f383415ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.814+0000 7f3839200640 1 --2- 172.21.15.45:0/2887601410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3834146220 0x7f383415ccb0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f3824009870 tx=0x7f382402b990 comp rx=0 tx=0).stop 2024-04-16T16:21:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.814+0000 7f3839200640 1 -- 172.21.15.45:0/2887601410 shutdown_connections 2024-04-16T16:21:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.814+0000 7f3839200640 1 --2- 172.21.15.45:0/2887601410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3834146220 0x7f383415ccb0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.814+0000 7f3839200640 1 -- 172.21.15.45:0/2887601410 >> 172.21.15.45:0/2887601410 conn(0x7f383406b930 msgr2=0x7f383406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:21:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.814+0000 7f3839200640 1 -- 172.21.15.45:0/2887601410 shutdown_connections 2024-04-16T16:21:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.815+0000 7f3839200640 1 -- 172.21.15.45:0/2887601410 wait complete. 2024-04-16T16:21:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.816+0000 7f3839200640 1 Processor -- start 2024-04-16T16:21:59.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.816+0000 7f3839200640 1 -- start start 2024-04-16T16:21:59.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.817+0000 7f3839200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f383406f7a0 0x7f38340ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:59.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.817+0000 7f3839200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3834146220 0x7f383413e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:59.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.817+0000 7f3839200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38341608a0 0x7f38341f8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:59.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.817+0000 7f3839200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f383406fc20 con 0x7f38341608a0 2024-04-16T16:21:59.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.817+0000 7f3839200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f383415dc00 con 0x7f383406f7a0 2024-04-16T16:21:59.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.817+0000 7f3839200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f383406fda0 con 0x7f3834146220 2024-04-16T16:21:59.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.818+0000 7f3833e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38341608a0 0x7f38341f8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:59.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.818+0000 7f3833e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38341608a0 0x7f38341f8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57732/0 (socket says 172.21.15.45:57732) 2024-04-16T16:21:59.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.818+0000 7f3833e00640 1 -- 172.21.15.45:0/921074623 learned_addr learned my addr 172.21.15.45:0/921074623 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:21:59.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.818+0000 7f3833400640 1 --2- 172.21.15.45:0/921074623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f383406f7a0 0x7f38340ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:59.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.818+0000 7f3833e00640 1 -- 172.21.15.45:0/921074623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3834146220 msgr2=0x7f383413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:59.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.818+0000 7f3832a00640 1 --2- 172.21.15.45:0/921074623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3834146220 0x7f383413e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:59.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.818+0000 7f3833e00640 1 --2- 172.21.15.45:0/921074623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3834146220 0x7f383413e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.818+0000 7f3833e00640 1 -- 172.21.15.45:0/921074623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f383406f7a0 msgr2=0x7f38340ca650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:21:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.818+0000 7f3833e00640 1 --2- 172.21.15.45:0/921074623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f383406f7a0 0x7f38340ca650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:21:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.818+0000 7f3833e00640 1 -- 172.21.15.45:0/921074623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38340cad10 con 0x7f38341608a0 2024-04-16T16:21:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.818+0000 7f3833400640 1 --2- 172.21.15.45:0/921074623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f383406f7a0 0x7f38340ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:21:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.818+0000 7f3833e00640 1 --2- 172.21.15.45:0/921074623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38341608a0 0x7f38341f8d10 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f382c00b970 tx=0x7f382c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.819+0000 7f382b400640 1 -- 172.21.15.45:0/921074623 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c019070 con 0x7f38341608a0 2024-04-16T16:21:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.819+0000 7f382b400640 1 -- 172.21.15.45:0/921074623 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f382c00cb40 con 0x7f38341608a0 2024-04-16T16:21:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.819+0000 7f382b400640 1 -- 172.21.15.45:0/921074623 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c014440 con 0x7f38341608a0 2024-04-16T16:21:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.819+0000 7f3839200640 1 -- 172.21.15.45:0/921074623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38340caff0 con 0x7f38341608a0 2024-04-16T16:21:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.819+0000 7f3839200640 1 -- 172.21.15.45:0/921074623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f383415e5b0 con 0x7f38341608a0 2024-04-16T16:21:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.820+0000 7f3839200640 1 -- 172.21.15.45:0/921074623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3834075780 con 0x7f38341608a0 2024-04-16T16:21:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.821+0000 7f382b400640 1 -- 172.21.15.45:0/921074623 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7f382c01c070 con 0x7f38341608a0 2024-04-16T16:21:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.822+0000 7f382b400640 1 --2- 172.21.15.45:0/921074623 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f381406ca60 0x7f381406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:21:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.822+0000 7f382b400640 1 -- 172.21.15.45:0/921074623 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f382c08ba50 con 0x7f38341608a0 2024-04-16T16:21:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.823+0000 7f3833400640 1 --2- 172.21.15.45:0/921074623 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f381406ca60 0x7f381406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:21:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.824+0000 7f3833400640 1 --2- 172.21.15.45:0/921074623 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f381406ca60 0x7f381406ef20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3824036000 tx=0x7f382403b040 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:21:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:21:59.827+0000 7f382b400640 1 -- 172.21.15.45:0/921074623 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f382c059ca0 con 0x7f38341608a0 2024-04-16T16:22:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.072+0000 7f3839200640 1 -- 172.21.15.45:0/921074623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38340cff60 con 0x7f38341608a0 2024-04-16T16:22:00.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.073+0000 7f382b400640 1 -- 172.21.15.45:0/921074623 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5190 (secure 0 0 0) 0x7f382c0595a0 con 0x7f38341608a0 2024-04-16T16:22:00.072 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:00.073 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2585865379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2585865379},{"type":"v1","addr":"172.21.15.45:6834","nonce":2585865379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6590,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/777087937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":777087937},{"type":"v1","addr":"172.21.15.45:6835","nonce":777087937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6593,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2561260581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2561260581},{"type":"v1","addr":"172.21.15.103:6835","nonce":2561260581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:21:26.066855+0000","modified":"2024-04-16T16:21:29.155869+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6596":{"gid":6596,"name":"d","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6837/3252460318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3252460318},{"type":"v1","addr":"172.21.15.103:6837","nonce":3252460318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6596,"qdb_cluster":[6596]},"id":10}]} 2024-04-16T16:22:00.073 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 100 2024-04-16T16:22:00.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.076+0000 7f3839200640 1 -- 172.21.15.45:0/921074623 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f381406ca60 msgr2=0x7f381406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.076+0000 7f3839200640 1 --2- 172.21.15.45:0/921074623 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f381406ca60 0x7f381406ef20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3824036000 tx=0x7f382403b040 comp rx=0 tx=0).stop 2024-04-16T16:22:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.076+0000 7f3839200640 1 -- 172.21.15.45:0/921074623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38341608a0 msgr2=0x7f38341f8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.076+0000 7f3839200640 1 --2- 172.21.15.45:0/921074623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38341608a0 0x7f38341f8d10 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f382c00b970 tx=0x7f382c00be30 comp rx=0 tx=0).stop 2024-04-16T16:22:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.077+0000 7f3839200640 1 -- 172.21.15.45:0/921074623 shutdown_connections 2024-04-16T16:22:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.077+0000 7f3839200640 1 --2- 172.21.15.45:0/921074623 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f381406ca60 0x7f381406ef20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.077+0000 7f3839200640 1 --2- 172.21.15.45:0/921074623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38341608a0 0x7f38341f8d10 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.077+0000 7f3839200640 1 --2- 172.21.15.45:0/921074623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3834146220 0x7f383413e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.077+0000 7f3839200640 1 --2- 172.21.15.45:0/921074623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f383406f7a0 0x7f38340ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.077+0000 7f3839200640 1 -- 172.21.15.45:0/921074623 >> 172.21.15.45:0/921074623 conn(0x7f383406b930 msgr2=0x7f38340cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.077+0000 7f3839200640 1 -- 172.21.15.45:0/921074623 shutdown_connections 2024-04-16T16:22:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.077+0000 7f3839200640 1 -- 172.21.15.45:0/921074623 wait complete. 2024-04-16T16:22:00.097 DEBUG:tasks.cephfs.test_mds_metrics:metrics={'version': 2, 'global_counters': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'counters': [], 'client_metadata': {'cephfs': {'client.6659': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '192.168.0.1'}, 'client.6710': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'IP': '172.21.15.103'}, 'client.7498': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'IP': '172.21.15.103'}, 'client.9348': {'hostname': 'smithi045', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '192.168.0.1'}}}, 'global_metrics': {'cephfs': {'client.6659': [[24, 0], [0, 0], [0, 0], [0, 8994062], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 2248517], [37136336141620, 4]], 'client.6710': [[2, 0], [0, 0], [0, 0], [0, 980564], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 980564], [0, 1]], 'client.7498': [[2, 0], [0, 0], [0, 0], [0, 592826], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 592826], [0, 1]], 'client.9348': [[5, 19], [0, 0], [0, 0], [0, 30535869], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 1272333], [36535092943149, 24]]}}, 'metrics': {'delayed_ranks': [], 'mds.0': {'client.6659': [], 'client.6710': [], 'client.7498': [], 'client.9348': []}}} 2024-04-16T16:22:00.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:22:00.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.319+0000 7fcfcb200640 1 Processor -- start 2024-04-16T16:22:00.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.319+0000 7fcfcb200640 1 -- start start 2024-04-16T16:22:00.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.321+0000 7fcfcb200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcfcc162270 0x7fcfcc0d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:00.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.321+0000 7fcfcb200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcfcc0d52d0 0x7fcfcc06ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:00.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.321+0000 7fcfcb200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfcc0d3080 con 0x7fcfcc161e90 2024-04-16T16:22:00.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.321+0000 7fcfcb200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fcfcc0d2e30 con 0x7fcfcc162270 2024-04-16T16:22:00.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.321+0000 7fcfcb200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcfcc0d3360 con 0x7fcfcc0d52d0 2024-04-16T16:22:00.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.321+0000 7fcfc9e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcfcc161e90 0x7fcfcc0cf7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53632/0 (socket says 172.21.15.45:53632) 2024-04-16T16:22:00.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.321+0000 7fcfc9e00640 1 -- 172.21.15.45:0/2829908206 learned_addr learned my addr 172.21.15.45:0/2829908206 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:00.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.321+0000 7fcfc3e00640 1 --2- 172.21.15.45:0/2829908206 >> v2:172.21.15.45:3300/0 conn(0x7fcfcc162270 0x7fcfcc0d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:00.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.322+0000 7fcfc9400640 1 -- 172.21.15.45:0/2829908206 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 990514733 0 0) 0x7fcfcc0d3080 con 0x7fcfcc161e90 2024-04-16T16:22:00.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.322+0000 7fcfca800640 1 --2- 172.21.15.45:0/2829908206 >> v2:172.21.15.103:3300/0 conn(0x7fcfcc0d52d0 0x7fcfcc06ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:00.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.322+0000 7fcfc9400640 1 -- 172.21.15.45:0/2829908206 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfac003520 con 0x7fcfcc161e90 2024-04-16T16:22:00.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.322+0000 7fcfc3e00640 1 -- 172.21.15.45:0/2829908206 >> v2:172.21.15.103:3300/0 conn(0x7fcfcc0d52d0 msgr2=0x7fcfcc06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:00.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.322+0000 7fcfc3e00640 1 --2- 172.21.15.45:0/2829908206 >> v2:172.21.15.103:3300/0 conn(0x7fcfcc0d52d0 0x7fcfcc06ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.322+0000 7fcfc3e00640 1 -- 172.21.15.45:0/2829908206 >> v1:172.21.15.45:6789/0 conn(0x7fcfcc161e90 legacy=0x7fcfcc0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:00.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.322+0000 7fcfc3e00640 1 -- 172.21.15.45:0/2829908206 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfcc0d3640 con 0x7fcfcc162270 2024-04-16T16:22:00.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.322+0000 7fcfca800640 1 --2- 172.21.15.45:0/2829908206 >> v2:172.21.15.103:3300/0 conn(0x7fcfcc0d52d0 0x7fcfcc06ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:00.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.323+0000 7fcfc3e00640 1 --2- 172.21.15.45:0/2829908206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc162270 0x7fcfcc0d3e40 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40097a0 tx=0x7fcfb402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0d0321e4611218 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:00.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.323+0000 7fcfc9400640 1 -- 172.21.15.45:0/2829908206 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4035020 con 0x7fcfcc162270 2024-04-16T16:22:00.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.323+0000 7fcfc9400640 1 -- 172.21.15.45:0/2829908206 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfb4031830 con 0x7fcfcc162270 2024-04-16T16:22:00.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.323+0000 7fcfc9400640 1 -- 172.21.15.45:0/2829908206 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4031c00 con 0x7fcfcc162270 2024-04-16T16:22:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.324+0000 7fcfcb200640 1 -- 172.21.15.45:0/2829908206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc162270 msgr2=0x7fcfcc0d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.324+0000 7fcfcb200640 1 --2- 172.21.15.45:0/2829908206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc162270 0x7fcfcc0d3e40 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40097a0 tx=0x7fcfb402b8d0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.324+0000 7fcfcb200640 1 -- 172.21.15.45:0/2829908206 shutdown_connections 2024-04-16T16:22:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.324+0000 7fcfcb200640 1 --2- 172.21.15.45:0/2829908206 >> v2:172.21.15.103:3300/0 conn(0x7fcfcc0d52d0 0x7fcfcc06ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.324+0000 7fcfcb200640 1 --2- 172.21.15.45:0/2829908206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc162270 0x7fcfcc0d3e40 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.324+0000 7fcfcb200640 1 -- 172.21.15.45:0/2829908206 >> 172.21.15.45:0/2829908206 conn(0x7fcfcc0d1940 msgr2=0x7fcfcc0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.324+0000 7fcfcb200640 1 -- 172.21.15.45:0/2829908206 shutdown_connections 2024-04-16T16:22:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.324+0000 7fcfcb200640 1 -- 172.21.15.45:0/2829908206 wait complete. 2024-04-16T16:22:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.326+0000 7fcfcb200640 1 Processor -- start 2024-04-16T16:22:00.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.326+0000 7fcfcb200640 1 -- start start 2024-04-16T16:22:00.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.327+0000 7fcfcb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfcc0d52d0 0x7fcfcc15c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:00.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.327+0000 7fcfcb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc161e90 0x7fcfcc1f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:00.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.327+0000 7fcfcb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfcc162270 0x7fcfcc1f5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:00.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.327+0000 7fcfcb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcfcc0d4500 con 0x7fcfcc161e90 2024-04-16T16:22:00.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.327+0000 7fcfcb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcfcc0d4380 con 0x7fcfcc0d52d0 2024-04-16T16:22:00.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.327+0000 7fcfcb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcfcc0d4680 con 0x7fcfcc162270 2024-04-16T16:22:00.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.328+0000 7fcfc3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc161e90 0x7fcfcc1f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:00.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.328+0000 7fcfc3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc161e90 0x7fcfcc1f4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57750/0 (socket says 172.21.15.45:57750) 2024-04-16T16:22:00.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.328+0000 7fcfc3e00640 1 -- 172.21.15.45:0/2670109291 learned_addr learned my addr 172.21.15.45:0/2670109291 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:00.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.328+0000 7fcfc9e00640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfcc0d52d0 0x7fcfcc15c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:00.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.328+0000 7fcfca800640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfcc162270 0x7fcfcc1f5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:00.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.328+0000 7fcfc3e00640 1 -- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfcc162270 msgr2=0x7fcfcc1f5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:00.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.328+0000 7fcfc3e00640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfcc162270 0x7fcfcc1f5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.328+0000 7fcfc3e00640 1 -- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfcc0d52d0 msgr2=0x7fcfcc15c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:00.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.328+0000 7fcfc3e00640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfcc0d52d0 0x7fcfcc15c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.328+0000 7fcfc3e00640 1 -- 172.21.15.45:0/2670109291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfcc0cc510 con 0x7fcfcc161e90 2024-04-16T16:22:00.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.328+0000 7fcfc9e00640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfcc0d52d0 0x7fcfcc15c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:22:00.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.328+0000 7fcfca800640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfcc162270 0x7fcfcc1f5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:00.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.329+0000 7fcfc3e00640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc161e90 0x7fcfcc1f4d10 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4034040 tx=0x7fcfb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:00.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.329+0000 7fcfc2000640 1 -- 172.21.15.45:0/2670109291 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4040070 con 0x7fcfcc161e90 2024-04-16T16:22:00.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.329+0000 7fcfc2000640 1 -- 172.21.15.45:0/2670109291 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfb403a910 con 0x7fcfcc161e90 2024-04-16T16:22:00.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.329+0000 7fcfc2000640 1 -- 172.21.15.45:0/2670109291 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4035040 con 0x7fcfcc161e90 2024-04-16T16:22:00.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.329+0000 7fcfcb200640 1 -- 172.21.15.45:0/2670109291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfcc0cc7f0 con 0x7fcfcc161e90 2024-04-16T16:22:00.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.329+0000 7fcfcb200640 1 -- 172.21.15.45:0/2670109291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfcc0ccdb0 con 0x7fcfcc161e90 2024-04-16T16:22:00.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.330+0000 7fcfcb200640 1 -- 172.21.15.45:0/2670109291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfcc075780 con 0x7fcfcc161e90 2024-04-16T16:22:00.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.331+0000 7fcfc2000640 1 -- 172.21.15.45:0/2670109291 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7fcfb4038020 con 0x7fcfcc161e90 2024-04-16T16:22:00.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.332+0000 7fcfc2000640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fcfa806ca60 0x7fcfa806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:00.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.333+0000 7fcfc2000640 1 -- 172.21.15.45:0/2670109291 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7fcfb40ad980 con 0x7fcfcc161e90 2024-04-16T16:22:00.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.333+0000 7fcfc9e00640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fcfa806ca60 0x7fcfa806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:00.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.334+0000 7fcfc9e00640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fcfa806ca60 0x7fcfa806ef20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc007fd0 tx=0x7fcfbc009040 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:00.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.337+0000 7fcfc2000640 1 -- 172.21.15.45:0/2670109291 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcfb4031800 con 0x7fcfcc161e90 2024-04-16T16:22:00.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.587+0000 7fcfcb200640 1 -- 172.21.15.45:0/2670109291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfcc0cf800 con 0x7fcfcc161e90 2024-04-16T16:22:00.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.588+0000 7fcfc2000640 1 -- 172.21.15.45:0/2670109291 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5190 (secure 0 0 0) 0x7fcfb407bb30 con 0x7fcfcc161e90 2024-04-16T16:22:00.587 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:00.587 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2585865379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2585865379},{"type":"v1","addr":"172.21.15.45:6834","nonce":2585865379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6590,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/777087937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":777087937},{"type":"v1","addr":"172.21.15.45:6835","nonce":777087937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6593,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2561260581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2561260581},{"type":"v1","addr":"172.21.15.103:6835","nonce":2561260581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:21:26.066855+0000","modified":"2024-04-16T16:21:29.155869+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6596":{"gid":6596,"name":"d","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6837/3252460318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3252460318},{"type":"v1","addr":"172.21.15.103:6837","nonce":3252460318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6596,"qdb_cluster":[6596]},"id":10}]} 2024-04-16T16:22:00.587 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 100 2024-04-16T16:22:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.590+0000 7fcfcb200640 1 -- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fcfa806ca60 msgr2=0x7fcfa806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.590+0000 7fcfcb200640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fcfa806ca60 0x7fcfa806ef20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc007fd0 tx=0x7fcfbc009040 comp rx=0 tx=0).stop 2024-04-16T16:22:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.590+0000 7fcfcb200640 1 -- 172.21.15.45:0/2670109291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc161e90 msgr2=0x7fcfcc1f4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.590+0000 7fcfcb200640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc161e90 0x7fcfcc1f4d10 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4034040 tx=0x7fcfb40046c0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.591+0000 7fcfcb200640 1 -- 172.21.15.45:0/2670109291 shutdown_connections 2024-04-16T16:22:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.591+0000 7fcfcb200640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fcfa806ca60 0x7fcfa806ef20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.591+0000 7fcfcb200640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfcc162270 0x7fcfcc1f5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.591+0000 7fcfcb200640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc161e90 0x7fcfcc1f4d10 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.591+0000 7fcfcb200640 1 --2- 172.21.15.45:0/2670109291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfcc0d52d0 0x7fcfcc15c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.591+0000 7fcfcb200640 1 -- 172.21.15.45:0/2670109291 >> 172.21.15.45:0/2670109291 conn(0x7fcfcc0d1940 msgr2=0x7fcfcc0b9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:00.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.591+0000 7fcfcb200640 1 -- 172.21.15.45:0/2670109291 shutdown_connections 2024-04-16T16:22:00.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.591+0000 7fcfcb200640 1 -- 172.21.15.45:0/2670109291 wait complete. 2024-04-16T16:22:00.610 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-04-16T16:22:00.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.832+0000 7fba3ec00640 1 Processor -- start 2024-04-16T16:22:00.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.832+0000 7fba3ec00640 1 -- start start 2024-04-16T16:22:00.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.833+0000 7fba3ec00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fba380d0e80 0x7fba380b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:00.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.834+0000 7fba3ec00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba380afda0 0x7fba3806ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:00.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.834+0000 7fba3ec00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba380ccb70 con 0x7fba3815b920 2024-04-16T16:22:00.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.834+0000 7fba3ec00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fba3806e7a0 con 0x7fba380d0e80 2024-04-16T16:22:00.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.834+0000 7fba3ec00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fba3806f170 con 0x7fba380afda0 2024-04-16T16:22:00.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.834+0000 7fba37400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fba380d0e80 0x7fba380b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:00.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.834+0000 7fba37400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fba380d0e80 0x7fba380b4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57758/0 (socket says 172.21.15.45:57758) 2024-04-16T16:22:00.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.834+0000 7fba37400640 1 -- 172.21.15.45:0/4124824200 learned_addr learned my addr 172.21.15.45:0/4124824200 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:00.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.834+0000 7fba3cc00640 1 --2- 172.21.15.45:0/4124824200 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba380afda0 0x7fba3806ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:00.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.834+0000 7fba37400640 1 -- 172.21.15.45:0/4124824200 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba380afda0 msgr2=0x7fba3806ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:00.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.834+0000 7fba37400640 1 --2- 172.21.15.45:0/4124824200 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba380afda0 0x7fba3806ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.835+0000 7fba37400640 1 -- 172.21.15.45:0/4124824200 >> v1:172.21.15.103:6789/0 conn(0x7fba3815b920 legacy=0x7fba380d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:00.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.835+0000 7fba37400640 1 -- 172.21.15.45:0/4124824200 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba380cce50 con 0x7fba380d0e80 2024-04-16T16:22:00.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.836+0000 7fba37400640 1 --2- 172.21.15.45:0/4124824200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba380d0e80 0x7fba380b4e20 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fba38073470 tx=0x7fba2402e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c80ee9e176128be server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:00.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.836+0000 7fba36a00640 1 -- 172.21.15.45:0/4124824200 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2403b070 con 0x7fba380d0e80 2024-04-16T16:22:00.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.836+0000 7fba36a00640 1 -- 172.21.15.45:0/4124824200 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba2402ee50 con 0x7fba380d0e80 2024-04-16T16:22:00.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.836+0000 7fba36a00640 1 -- 172.21.15.45:0/4124824200 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba24036660 con 0x7fba380d0e80 2024-04-16T16:22:00.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.837+0000 7fba3ec00640 1 -- 172.21.15.45:0/4124824200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba380d0e80 msgr2=0x7fba380b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:00.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.837+0000 7fba3ec00640 1 --2- 172.21.15.45:0/4124824200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba380d0e80 0x7fba380b4e20 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fba38073470 tx=0x7fba2402e0a0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.837+0000 7fba3ec00640 1 -- 172.21.15.45:0/4124824200 shutdown_connections 2024-04-16T16:22:00.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.837+0000 7fba3ec00640 1 --2- 172.21.15.45:0/4124824200 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba380afda0 0x7fba3806ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.837+0000 7fba3ec00640 1 --2- 172.21.15.45:0/4124824200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba380d0e80 0x7fba380b4e20 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.837+0000 7fba3ec00640 1 -- 172.21.15.45:0/4124824200 >> 172.21.15.45:0/4124824200 conn(0x7fba380b3970 msgr2=0x7fba380b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:00.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.837+0000 7fba3ec00640 1 -- 172.21.15.45:0/4124824200 shutdown_connections 2024-04-16T16:22:00.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.837+0000 7fba3ec00640 1 -- 172.21.15.45:0/4124824200 wait complete. 2024-04-16T16:22:00.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.839+0000 7fba3ec00640 1 Processor -- start 2024-04-16T16:22:00.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.839+0000 7fba3ec00640 1 -- start start 2024-04-16T16:22:00.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.840+0000 7fba3ec00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba380afda0 0x7fba381531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:00.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.840+0000 7fba3ec00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba380d0e80 0x7fba38151930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:00.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.840+0000 7fba3ec00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3815b920 0x7fba3815ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:00.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.840+0000 7fba3ec00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fba380ce760 con 0x7fba3815b920 2024-04-16T16:22:00.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.840+0000 7fba3ec00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fba380ce5e0 con 0x7fba380afda0 2024-04-16T16:22:00.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.840+0000 7fba3ec00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fba380ce8e0 con 0x7fba380d0e80 2024-04-16T16:22:00.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.841+0000 7fba3cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3815b920 0x7fba3815ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:00.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.841+0000 7fba37e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba380afda0 0x7fba381531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:00.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.841+0000 7fba3cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3815b920 0x7fba3815ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57760/0 (socket says 172.21.15.45:57760) 2024-04-16T16:22:00.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.841+0000 7fba37e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba380afda0 0x7fba381531e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:42414/0 (socket says 172.21.15.45:42414) 2024-04-16T16:22:00.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.841+0000 7fba3cc00640 1 -- 172.21.15.45:0/2325331708 learned_addr learned my addr 172.21.15.45:0/2325331708 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:00.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.841+0000 7fba37400640 1 --2- 172.21.15.45:0/2325331708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba380d0e80 0x7fba38151930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:00.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.841+0000 7fba3cc00640 1 -- 172.21.15.45:0/2325331708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba380d0e80 msgr2=0x7fba38151930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:00.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.841+0000 7fba3cc00640 1 --2- 172.21.15.45:0/2325331708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba380d0e80 0x7fba38151930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.841+0000 7fba3cc00640 1 -- 172.21.15.45:0/2325331708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba380afda0 msgr2=0x7fba381531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:00.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.841+0000 7fba3cc00640 1 --2- 172.21.15.45:0/2325331708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba380afda0 0x7fba381531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:00.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.841+0000 7fba3cc00640 1 -- 172.21.15.45:0/2325331708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba38153830 con 0x7fba3815b920 2024-04-16T16:22:00.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.842+0000 7fba3cc00640 1 --2- 172.21.15.45:0/2325331708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3815b920 0x7fba3815ab40 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fba3000d830 tx=0x7fba3000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:00.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.842+0000 7fba2fe00640 1 -- 172.21.15.45:0/2325331708 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3000b6c0 con 0x7fba3815b920 2024-04-16T16:22:00.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.842+0000 7fba2fe00640 1 -- 172.21.15.45:0/2325331708 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba30019070 con 0x7fba3815b920 2024-04-16T16:22:00.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.842+0000 7fba2fe00640 1 -- 172.21.15.45:0/2325331708 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3000bac0 con 0x7fba3815b920 2024-04-16T16:22:00.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.842+0000 7fba3ec00640 1 -- 172.21.15.45:0/2325331708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba38153b10 con 0x7fba3815b920 2024-04-16T16:22:00.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.842+0000 7fba3ec00640 1 -- 172.21.15.45:0/2325331708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba381540d0 con 0x7fba3815b920 2024-04-16T16:22:00.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.843+0000 7fba3ec00640 1 -- 172.21.15.45:0/2325331708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba380d1920 con 0x7fba3815b920 2024-04-16T16:22:00.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.844+0000 7fba2fe00640 1 -- 172.21.15.45:0/2325331708 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7fba3001e460 con 0x7fba3815b920 2024-04-16T16:22:00.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.845+0000 7fba2fe00640 1 --2- 172.21.15.45:0/2325331708 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fba1806ca90 0x7fba1806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:00.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.845+0000 7fba2fe00640 1 -- 172.21.15.45:0/2325331708 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7fba3008c510 con 0x7fba3815b920 2024-04-16T16:22:00.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.846+0000 7fba37e00640 1 --2- 172.21.15.45:0/2325331708 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fba1806ca90 0x7fba1806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:00.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.847+0000 7fba37e00640 1 --2- 172.21.15.45:0/2325331708 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fba1806ca90 0x7fba1806ef50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fba2800b450 tx=0x7fba2800d040 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:00.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:00.850+0000 7fba2fe00640 1 -- 172.21.15.45:0/2325331708 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fba380d1920 con 0x7fba3815b920 2024-04-16T16:22:01.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:01.100+0000 7fba3ec00640 1 -- 172.21.15.45:0/2325331708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) -- 0x7fba38075780 con 0x7fba3815b920 2024-04-16T16:22:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.083+0000 7fba2fe00640 1 -- 172.21.15.45:0/2325331708 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v101) ==== 105+0+0 (secure 0 0 0) 0x7fba3005a630 con 0x7fba3815b920 2024-04-16T16:22:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.088+0000 7fba3ec00640 1 -- 172.21.15.45:0/2325331708 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fba1806ca90 msgr2=0x7fba1806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.088+0000 7fba3ec00640 1 --2- 172.21.15.45:0/2325331708 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fba1806ca90 0x7fba1806ef50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fba2800b450 tx=0x7fba2800d040 comp rx=0 tx=0).stop 2024-04-16T16:22:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.088+0000 7fba3ec00640 1 -- 172.21.15.45:0/2325331708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3815b920 msgr2=0x7fba3815ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.088+0000 7fba3ec00640 1 --2- 172.21.15.45:0/2325331708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3815b920 0x7fba3815ab40 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fba3000d830 tx=0x7fba3000dcf0 comp rx=0 tx=0).stop 2024-04-16T16:22:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.088+0000 7fba3ec00640 1 -- 172.21.15.45:0/2325331708 shutdown_connections 2024-04-16T16:22:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.088+0000 7fba3ec00640 1 --2- 172.21.15.45:0/2325331708 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fba1806ca90 0x7fba1806ef50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.088+0000 7fba3ec00640 1 --2- 172.21.15.45:0/2325331708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3815b920 0x7fba3815ab40 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:02.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.088+0000 7fba3ec00640 1 --2- 172.21.15.45:0/2325331708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba380d0e80 0x7fba38151930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:02.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.088+0000 7fba3ec00640 1 --2- 172.21.15.45:0/2325331708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba380afda0 0x7fba381531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:02.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.088+0000 7fba3ec00640 1 -- 172.21.15.45:0/2325331708 >> 172.21.15.45:0/2325331708 conn(0x7fba380b3970 msgr2=0x7fba38072ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:02.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.088+0000 7fba3ec00640 1 -- 172.21.15.45:0/2325331708 shutdown_connections 2024-04-16T16:22:02.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.088+0000 7fba3ec00640 1 -- 172.21.15.45:0/2325331708 wait complete. 2024-04-16T16:22:02.117 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:22:02.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.323+0000 7f525f200640 1 Processor -- start 2024-04-16T16:22:02.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.323+0000 7f525f200640 1 -- start start 2024-04-16T16:22:02.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.325+0000 7f525f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5260146220 0x7f526015ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:02.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.325+0000 7f525f200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f526006f7a0 0x7f526006df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:02.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.325+0000 7f525f200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f526015d360 con 0x7f526006e480 2024-04-16T16:22:02.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.325+0000 7f525f200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5260160410 con 0x7f526006f7a0 2024-04-16T16:22:02.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.325+0000 7f525f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f526015d640 con 0x7f5260146220 2024-04-16T16:22:02.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.326+0000 7f5257e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f526006f7a0 0x7f526006df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:02.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.326+0000 7f5257e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f526006f7a0 0x7f526006df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57764/0 (socket says 172.21.15.45:57764) 2024-04-16T16:22:02.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.326+0000 7f525e800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f526006e480 0x7f52601608a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53636/0 (socket says 172.21.15.45:53636) 2024-04-16T16:22:02.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.326+0000 7f5257e00640 1 -- 172.21.15.45:0/2620237685 learned_addr learned my addr 172.21.15.45:0/2620237685 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:02.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.326+0000 7f525de00640 1 --2- 172.21.15.45:0/2620237685 >> v2:172.21.15.103:3300/0 conn(0x7f5260146220 0x7f526015ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:02.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.326+0000 7f5257e00640 1 -- 172.21.15.45:0/2620237685 >> v2:172.21.15.103:3300/0 conn(0x7f5260146220 msgr2=0x7f526015ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:02.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.326+0000 7f5257e00640 1 --2- 172.21.15.45:0/2620237685 >> v2:172.21.15.103:3300/0 conn(0x7f5260146220 0x7f526015ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:02.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.326+0000 7f5257e00640 1 -- 172.21.15.45:0/2620237685 >> v1:172.21.15.45:6789/0 conn(0x7f526006e480 legacy=0x7f52601608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:02.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.326+0000 7f5257e00640 1 -- 172.21.15.45:0/2620237685 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526015d920 con 0x7f526006f7a0 2024-04-16T16:22:02.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.326+0000 7f525de00640 1 --2- 172.21.15.45:0/2620237685 >> v2:172.21.15.103:3300/0 conn(0x7f5260146220 0x7f526015ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:22:02.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.327+0000 7f5257e00640 1 --2- 172.21.15.45:0/2620237685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526006f7a0 0x7f526006df40 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f5260160590 tx=0x7f524802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d3c57eb146faeb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:02.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.328+0000 7f525d400640 1 -- 172.21.15.45:0/2620237685 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5248034070 con 0x7f526006f7a0 2024-04-16T16:22:02.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.328+0000 7f525d400640 1 -- 172.21.15.45:0/2620237685 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5248030850 con 0x7f526006f7a0 2024-04-16T16:22:02.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.328+0000 7f525d400640 1 -- 172.21.15.45:0/2620237685 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5248030c20 con 0x7f526006f7a0 2024-04-16T16:22:02.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.328+0000 7f525f200640 1 -- 172.21.15.45:0/2620237685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526006f7a0 msgr2=0x7f526006df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:02.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.328+0000 7f525f200640 1 --2- 172.21.15.45:0/2620237685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526006f7a0 0x7f526006df40 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f5260160590 tx=0x7f524802b990 comp rx=0 tx=0).stop 2024-04-16T16:22:02.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.328+0000 7f525f200640 1 -- 172.21.15.45:0/2620237685 shutdown_connections 2024-04-16T16:22:02.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.328+0000 7f525f200640 1 --2- 172.21.15.45:0/2620237685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f526006f7a0 0x7f526006df40 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:02.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.328+0000 7f525f200640 1 --2- 172.21.15.45:0/2620237685 >> v2:172.21.15.103:3300/0 conn(0x7f5260146220 0x7f526015ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:02.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.328+0000 7f525f200640 1 -- 172.21.15.45:0/2620237685 >> 172.21.15.45:0/2620237685 conn(0x7f526006b930 msgr2=0x7f526006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:02.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.329+0000 7f525f200640 1 -- 172.21.15.45:0/2620237685 shutdown_connections 2024-04-16T16:22:02.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.329+0000 7f525f200640 1 -- 172.21.15.45:0/2620237685 wait complete. 2024-04-16T16:22:02.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.330+0000 7f525f200640 1 Processor -- start 2024-04-16T16:22:02.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.331+0000 7f525f200640 1 -- start start 2024-04-16T16:22:02.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.331+0000 7f525f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f526006e480 0x7f526013c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:02.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.332+0000 7f525f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5260146220 0x7f526013c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:02.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.332+0000 7f525f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52601e6150 0x7f52601e6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:02.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.332+0000 7f525f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f526006fda0 con 0x7f52601e6150 2024-04-16T16:22:02.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.332+0000 7f525f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f526006fc20 con 0x7f526006e480 2024-04-16T16:22:02.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.332+0000 7f525f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5260160fb0 con 0x7f5260146220 2024-04-16T16:22:02.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.332+0000 7f525e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52601e6150 0x7f52601e6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:02.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.332+0000 7f525e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52601e6150 0x7f52601e6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57776/0 (socket says 172.21.15.45:57776) 2024-04-16T16:22:02.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.332+0000 7f525e800640 1 -- 172.21.15.45:0/1435294428 learned_addr learned my addr 172.21.15.45:0/1435294428 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:02.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.332+0000 7f525de00640 1 --2- 172.21.15.45:0/1435294428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f526006e480 0x7f526013c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:02.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.332+0000 7f525e800640 1 -- 172.21.15.45:0/1435294428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5260146220 msgr2=0x7f526013c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:02.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.332+0000 7f525e800640 1 --2- 172.21.15.45:0/1435294428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5260146220 0x7f526013c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:02.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.333+0000 7f525e800640 1 -- 172.21.15.45:0/1435294428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f526006e480 msgr2=0x7f526013c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:02.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.333+0000 7f525e800640 1 --2- 172.21.15.45:0/1435294428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f526006e480 0x7f526013c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:02.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.333+0000 7f525e800640 1 -- 172.21.15.45:0/1435294428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526013cee0 con 0x7f52601e6150 2024-04-16T16:22:02.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.333+0000 7f525e800640 1 --2- 172.21.15.45:0/1435294428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52601e6150 0x7f52601e6600 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f525800c710 tx=0x7f525800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:02.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.333+0000 7f5256000640 1 -- 172.21.15.45:0/1435294428 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5258017070 con 0x7f52601e6150 2024-04-16T16:22:02.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.333+0000 7f5256000640 1 -- 172.21.15.45:0/1435294428 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f525800ce20 con 0x7f52601e6150 2024-04-16T16:22:02.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.333+0000 7f5256000640 1 -- 172.21.15.45:0/1435294428 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525800a860 con 0x7f52601e6150 2024-04-16T16:22:02.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.333+0000 7f525f200640 1 -- 172.21.15.45:0/1435294428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f526013d1c0 con 0x7f52601e6150 2024-04-16T16:22:02.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.333+0000 7f525f200640 1 -- 172.21.15.45:0/1435294428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f526013d780 con 0x7f52601e6150 2024-04-16T16:22:02.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.334+0000 7f525f200640 1 -- 172.21.15.45:0/1435294428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5260075780 con 0x7f52601e6150 2024-04-16T16:22:02.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.336+0000 7f5256000640 1 -- 172.21.15.45:0/1435294428 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7f525801a070 con 0x7f52601e6150 2024-04-16T16:22:02.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.336+0000 7f5256000640 1 --2- 172.21.15.45:0/1435294428 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f523c06ca90 0x7f523c06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:02.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.337+0000 7f5256000640 1 -- 172.21.15.45:0/1435294428 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f525808b570 con 0x7f52601e6150 2024-04-16T16:22:02.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.337+0000 7f525de00640 1 --2- 172.21.15.45:0/1435294428 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f523c06ca90 0x7f523c06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:02.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.338+0000 7f525de00640 1 --2- 172.21.15.45:0/1435294428 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f523c06ca90 0x7f523c06ef50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5250006fd0 tx=0x7f5250008040 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:02.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.341+0000 7f5256000640 1 -- 172.21.15.45:0/1435294428 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5258059690 con 0x7f52601e6150 2024-04-16T16:22:02.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.586+0000 7f525f200640 1 -- 172.21.15.45:0/1435294428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5260145a40 con 0x7f52601e6150 2024-04-16T16:22:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.587+0000 7f5256000640 1 -- 172.21.15.45:0/1435294428 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5207 (secure 0 0 0) 0x7f5258058f90 con 0x7f52601e6150 2024-04-16T16:22:02.587 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:02.587 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2585865379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2585865379},{"type":"v1","addr":"172.21.15.45:6834","nonce":2585865379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6590,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/777087937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":777087937},{"type":"v1","addr":"172.21.15.45:6835","nonce":777087937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:21:26.066855+0000","modified":"2024-04-16T16:22:02.087032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6596,"mds_1":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":1,"incarnation":102,"state":"up:creating","state_seq":1,"addr":"172.21.15.103:6835/2561260581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2561260581},{"type":"v1","addr":"172.21.15.103:6835","nonce":2561260581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6596":{"gid":6596,"name":"d","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6837/3252460318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3252460318},{"type":"v1","addr":"172.21.15.103:6837","nonce":3252460318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6596,"qdb_cluster":[6596]},"id":10}]} 2024-04-16T16:22:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 102 2024-04-16T16:22:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.590+0000 7f525f200640 1 -- 172.21.15.45:0/1435294428 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f523c06ca90 msgr2=0x7f523c06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.590+0000 7f525f200640 1 --2- 172.21.15.45:0/1435294428 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f523c06ca90 0x7f523c06ef50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5250006fd0 tx=0x7f5250008040 comp rx=0 tx=0).stop 2024-04-16T16:22:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.590+0000 7f525f200640 1 -- 172.21.15.45:0/1435294428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52601e6150 msgr2=0x7f52601e6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.590+0000 7f525f200640 1 --2- 172.21.15.45:0/1435294428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52601e6150 0x7f52601e6600 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f525800c710 tx=0x7f525800cbd0 comp rx=0 tx=0).stop 2024-04-16T16:22:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.590+0000 7f525f200640 1 -- 172.21.15.45:0/1435294428 shutdown_connections 2024-04-16T16:22:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.590+0000 7f525f200640 1 --2- 172.21.15.45:0/1435294428 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f523c06ca90 0x7f523c06ef50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.590+0000 7f525f200640 1 --2- 172.21.15.45:0/1435294428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52601e6150 0x7f52601e6600 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.590+0000 7f525f200640 1 --2- 172.21.15.45:0/1435294428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5260146220 0x7f526013c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.590+0000 7f525f200640 1 --2- 172.21.15.45:0/1435294428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f526006e480 0x7f526013c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.590+0000 7f525f200640 1 -- 172.21.15.45:0/1435294428 >> 172.21.15.45:0/1435294428 conn(0x7f526006b930 msgr2=0x7f5260074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.591+0000 7f525f200640 1 -- 172.21.15.45:0/1435294428 shutdown_connections 2024-04-16T16:22:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:02.591+0000 7f525f200640 1 -- 172.21.15.45:0/1435294428 wait complete. 2024-04-16T16:22:03.011 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:21:26.066855+0000', 'modified': '2024-04-16T16:22:02.087032+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 6596, 'mds_1': 6593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6593': {'gid': 6593, 'name': 'b', 'rank': 1, 'incarnation': 102, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.103:6835/2561260581', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6834', 'nonce': 2561260581}, {'type': 'v1', 'addr': '172.21.15.103:6835', 'nonce': 2561260581}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_6596': {'gid': 6596, 'name': 'd', 'rank': 0, 'incarnation': 96, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.103:6837/3252460318', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6836', 'nonce': 3252460318}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 3252460318}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6596, 'qdb_cluster': [6596]} 2024-04-16T16:22:03.012 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6593:up:creating 2024-04-16T16:22:03.985 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-16T16:22:03.985+0000 7f0d31200640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-16T16:22:04.012 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:22:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.246+0000 7fc299600640 1 Processor -- start 2024-04-16T16:22:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.246+0000 7fc299600640 1 -- start start 2024-04-16T16:22:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.247+0000 7fc299600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc294146220 0x7fc29415ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.247+0000 7fc299600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc29415d360 con 0x7fc2941608a0 2024-04-16T16:22:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.247+0000 7fc299600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc29415d640 con 0x7fc29406f7a0 2024-04-16T16:22:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.247+0000 7fc299600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc294160410 con 0x7fc294146220 2024-04-16T16:22:04.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.248+0000 7fc298c00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc2941608a0 0x7fc2940cf360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37370/0 (socket says 172.21.15.45:37370) 2024-04-16T16:22:04.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.248+0000 7fc298c00640 1 -- 172.21.15.45:0/2054366399 learned_addr learned my addr 172.21.15.45:0/2054366399 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:04.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.248+0000 7fc293e00640 1 --2- 172.21.15.45:0/2054366399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc294146220 0x7fc29415ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:04.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.248+0000 7fc292a00640 1 -- 172.21.15.45:0/2054366399 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1633636407 0 0) 0x7fc29415d360 con 0x7fc2941608a0 2024-04-16T16:22:04.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.248+0000 7fc292a00640 1 -- 172.21.15.45:0/2054366399 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc278003520 con 0x7fc2941608a0 2024-04-16T16:22:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.248+0000 7fc292a00640 1 -- 172.21.15.45:0/2054366399 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1781730361 0 0) 0x7fc278003520 con 0x7fc2941608a0 2024-04-16T16:22:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.248+0000 7fc292a00640 1 -- 172.21.15.45:0/2054366399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc294146220 msgr2=0x7fc29415ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.248+0000 7fc292a00640 1 --2- 172.21.15.45:0/2054366399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc294146220 0x7fc29415ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.248+0000 7fc292a00640 1 -- 172.21.15.45:0/2054366399 >> v1:172.21.15.103:6789/0 conn(0x7fc29406f7a0 legacy=0x7fc29406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.249+0000 7fc292a00640 1 -- 172.21.15.45:0/2054366399 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc29415d920 con 0x7fc2941608a0 2024-04-16T16:22:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.249+0000 7fc292a00640 1 -- 172.21.15.45:0/2054366399 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fc28c0019f0 con 0x7fc2941608a0 2024-04-16T16:22:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.249+0000 7fc292a00640 1 -- 172.21.15.45:0/2054366399 >> v1:172.21.15.45:6789/0 conn(0x7fc2941608a0 legacy=0x7fc2940cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.249+0000 7fc292a00640 1 --2- 172.21.15.45:0/2054366399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2780039e0 0x7fc278003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.249+0000 7fc292a00640 1 --2- 172.21.15.45:0/2054366399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc278004800 0x7fc278004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.249+0000 7fc292a00640 1 --2- 172.21.15.45:0/2054366399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc294146220 0x7fc2780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.249+0000 7fc292a00640 1 -- 172.21.15.45:0/2054366399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc294160410 con 0x7fc2780039e0 2024-04-16T16:22:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.249+0000 7fc292a00640 1 -- 172.21.15.45:0/2054366399 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc27800ba70 con 0x7fc278004800 2024-04-16T16:22:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.249+0000 7fc292a00640 1 -- 172.21.15.45:0/2054366399 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc27800bc70 con 0x7fc294146220 2024-04-16T16:22:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.249+0000 7fc293e00640 1 --2- 172.21.15.45:0/2054366399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2780039e0 0x7fc278003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.249+0000 7fc293e00640 1 -- 172.21.15.45:0/2054366399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc294146220 msgr2=0x7fc2780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.249+0000 7fc293e00640 1 --2- 172.21.15.45:0/2054366399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc294146220 0x7fc2780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.249+0000 7fc293e00640 1 -- 172.21.15.45:0/2054366399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc278004800 msgr2=0x7fc278004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.249+0000 7fc293e00640 1 --2- 172.21.15.45:0/2054366399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc278004800 0x7fc278004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:04.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.249+0000 7fc293e00640 1 -- 172.21.15.45:0/2054366399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc278003520 con 0x7fc2780039e0 2024-04-16T16:22:04.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.251+0000 7fc293e00640 1 --2- 172.21.15.45:0/2054366399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2780039e0 0x7fc278003e00 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fc284009a90 tx=0x7fc278005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e853cf640d279e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:04.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.251+0000 7fc292a00640 1 -- 172.21.15.45:0/2054366399 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc284039070 con 0x7fc2780039e0 2024-04-16T16:22:04.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.251+0000 7fc292a00640 1 -- 172.21.15.45:0/2054366399 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc28402c6b0 con 0x7fc2780039e0 2024-04-16T16:22:04.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.251+0000 7fc292a00640 1 -- 172.21.15.45:0/2054366399 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc284009690 con 0x7fc2780039e0 2024-04-16T16:22:04.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.252+0000 7fc299600640 1 -- 172.21.15.45:0/2054366399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2780039e0 msgr2=0x7fc278003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:04.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.252+0000 7fc299600640 1 --2- 172.21.15.45:0/2054366399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2780039e0 0x7fc278003e00 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fc284009a90 tx=0x7fc278005830 comp rx=0 tx=0).stop 2024-04-16T16:22:04.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.252+0000 7fc298c00640 1 -- 172.21.15.45:0/2054366399 reap_dead start 2024-04-16T16:22:04.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.252+0000 7fc299600640 1 -- 172.21.15.45:0/2054366399 shutdown_connections 2024-04-16T16:22:04.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.252+0000 7fc299600640 1 -- 172.21.15.45:0/2054366399 >> 172.21.15.45:0/2054366399 conn(0x7fc29406b930 msgr2=0x7fc29406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:04.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.252+0000 7fc299600640 1 -- 172.21.15.45:0/2054366399 shutdown_connections 2024-04-16T16:22:04.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.252+0000 7fc299600640 1 -- 172.21.15.45:0/2054366399 wait complete. 2024-04-16T16:22:04.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.254+0000 7fc299600640 1 Processor -- start 2024-04-16T16:22:04.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.254+0000 7fc299600640 1 -- start start 2024-04-16T16:22:04.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.255+0000 7fc299600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2780039e0 0x7fc2940cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.255+0000 7fc299600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc29415bfd0 0x7fc294138b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.255+0000 7fc299600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc29415c450 0x7fc2940b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.255+0000 7fc299600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc29406fc20 con 0x7fc2780039e0 2024-04-16T16:22:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.255+0000 7fc299600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc29415dc00 con 0x7fc29415c450 2024-04-16T16:22:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.255+0000 7fc299600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc29406fda0 con 0x7fc29415bfd0 2024-04-16T16:22:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.255+0000 7fc293e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2780039e0 0x7fc2940cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.255+0000 7fc293e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2780039e0 0x7fc2940cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58082/0 (socket says 172.21.15.45:58082) 2024-04-16T16:22:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.255+0000 7fc293e00640 1 -- 172.21.15.45:0/525211537 learned_addr learned my addr 172.21.15.45:0/525211537 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.255+0000 7fc293e00640 1 -- 172.21.15.45:0/525211537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc29415bfd0 msgr2=0x7fc294138b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.255+0000 7fc293e00640 1 --2- 172.21.15.45:0/525211537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc29415bfd0 0x7fc294138b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.255+0000 7fc298c00640 1 --2- 172.21.15.45:0/525211537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc29415c450 0x7fc2940b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.255+0000 7fc293e00640 1 -- 172.21.15.45:0/525211537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc29415c450 msgr2=0x7fc2940b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.256+0000 7fc293e00640 1 --2- 172.21.15.45:0/525211537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc29415c450 0x7fc2940b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.256+0000 7fc293e00640 1 -- 172.21.15.45:0/525211537 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2940cd980 con 0x7fc2780039e0 2024-04-16T16:22:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.256+0000 7fc298c00640 1 --2- 172.21.15.45:0/525211537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc29415c450 0x7fc2940b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:04.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.256+0000 7fc293e00640 1 --2- 172.21.15.45:0/525211537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2780039e0 0x7fc2940cd440 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fc284009bb0 tx=0x7fc284004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:04.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.256+0000 7fc28be00640 1 -- 172.21.15.45:0/525211537 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc284042020 con 0x7fc2780039e0 2024-04-16T16:22:04.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.256+0000 7fc28be00640 1 -- 172.21.15.45:0/525211537 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc28402de40 con 0x7fc2780039e0 2024-04-16T16:22:04.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.256+0000 7fc28be00640 1 -- 172.21.15.45:0/525211537 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc284039040 con 0x7fc2780039e0 2024-04-16T16:22:04.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.256+0000 7fc299600640 1 -- 172.21.15.45:0/525211537 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc294139630 con 0x7fc2780039e0 2024-04-16T16:22:04.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.256+0000 7fc299600640 1 -- 172.21.15.45:0/525211537 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc29413cd60 con 0x7fc2780039e0 2024-04-16T16:22:04.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.258+0000 7fc299600640 1 -- 172.21.15.45:0/525211537 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc294075780 con 0x7fc2780039e0 2024-04-16T16:22:04.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.258+0000 7fc28be00640 1 -- 172.21.15.45:0/525211537 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7fc284045020 con 0x7fc2780039e0 2024-04-16T16:22:04.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.260+0000 7fc28be00640 1 --2- 172.21.15.45:0/525211537 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fc27406ca60 0x7fc27406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:04.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.260+0000 7fc28be00640 1 -- 172.21.15.45:0/525211537 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7fc2840ad220 con 0x7fc2780039e0 2024-04-16T16:22:04.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.261+0000 7fc293400640 1 --2- 172.21.15.45:0/525211537 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fc27406ca60 0x7fc27406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:04.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.262+0000 7fc293400640 1 --2- 172.21.15.45:0/525211537 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fc27406ca60 0x7fc27406ef20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc29415f2c0 tx=0x7fc280007990 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:04.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.265+0000 7fc28be00640 1 -- 172.21.15.45:0/525211537 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc28404c070 con 0x7fc2780039e0 2024-04-16T16:22:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.509+0000 7fc299600640 1 -- 172.21.15.45:0/525211537 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2940cff60 con 0x7fc2780039e0 2024-04-16T16:22:04.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.510+0000 7fc28be00640 1 -- 172.21.15.45:0/525211537 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+5211 (secure 0 0 0) 0x7fc2840044a0 con 0x7fc2780039e0 2024-04-16T16:22:04.510 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:04.510 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2585865379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2585865379},{"type":"v1","addr":"172.21.15.45:6834","nonce":2585865379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6590,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/777087937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":777087937},{"type":"v1","addr":"172.21.15.45:6835","nonce":777087937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:21:26.066855+0000","modified":"2024-04-16T16:22:03.091438+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6596,"mds_1":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":1,"incarnation":102,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6835/2561260581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2561260581},{"type":"v1","addr":"172.21.15.103:6835","nonce":2561260581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6596":{"gid":6596,"name":"d","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6837/3252460318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3252460318},{"type":"v1","addr":"172.21.15.103:6837","nonce":3252460318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6596,"qdb_cluster":[6593,6596]},"id":10}]} 2024-04-16T16:22:04.510 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 103 2024-04-16T16:22:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.513+0000 7fc299600640 1 -- 172.21.15.45:0/525211537 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fc27406ca60 msgr2=0x7fc27406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.513+0000 7fc299600640 1 --2- 172.21.15.45:0/525211537 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fc27406ca60 0x7fc27406ef20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc29415f2c0 tx=0x7fc280007990 comp rx=0 tx=0).stop 2024-04-16T16:22:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.513+0000 7fc299600640 1 -- 172.21.15.45:0/525211537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2780039e0 msgr2=0x7fc2940cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.513+0000 7fc299600640 1 --2- 172.21.15.45:0/525211537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2780039e0 0x7fc2940cd440 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fc284009bb0 tx=0x7fc284004730 comp rx=0 tx=0).stop 2024-04-16T16:22:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.513+0000 7fc299600640 1 -- 172.21.15.45:0/525211537 shutdown_connections 2024-04-16T16:22:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.513+0000 7fc299600640 1 --2- 172.21.15.45:0/525211537 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fc27406ca60 0x7fc27406ef20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.513+0000 7fc299600640 1 --2- 172.21.15.45:0/525211537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc29415c450 0x7fc2940b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.513+0000 7fc299600640 1 --2- 172.21.15.45:0/525211537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc29415bfd0 0x7fc294138b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.513+0000 7fc299600640 1 --2- 172.21.15.45:0/525211537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2780039e0 0x7fc2940cd440 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.513+0000 7fc299600640 1 -- 172.21.15.45:0/525211537 >> 172.21.15.45:0/525211537 conn(0x7fc29406b930 msgr2=0x7fc2941379a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.513+0000 7fc299600640 1 -- 172.21.15.45:0/525211537 shutdown_connections 2024-04-16T16:22:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:04.514+0000 7fc299600640 1 -- 172.21.15.45:0/525211537 wait complete. 2024-04-16T16:22:04.936 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-16T16:21:26.066855+0000', 'modified': '2024-04-16T16:22:03.091438+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 6596, 'mds_1': 6593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6593': {'gid': 6593, 'name': 'b', 'rank': 1, 'incarnation': 102, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.103:6835/2561260581', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6834', 'nonce': 2561260581}, {'type': 'v1', 'addr': '172.21.15.103:6835', 'nonce': 2561260581}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_6596': {'gid': 6596, 'name': 'd', 'rank': 0, 'incarnation': 96, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.103:6837/3252460318', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6836', 'nonce': 3252460318}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 3252460318}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6596, 'qdb_cluster': [6593, 6596]} 2024-04-16T16:22:04.936 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-04-16T16:22:04.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-16T16:22:05.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.170+0000 7f940ce00640 1 Processor -- start 2024-04-16T16:22:05.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.171+0000 7f940ce00640 1 -- start start 2024-04-16T16:22:05.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.172+0000 7f940ce00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f940815e830 0x7f94080ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.172+0000 7f940ce00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f94080cbc80 0x7f94080cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.173+0000 7f940ce00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f940815f1b0 con 0x7f94080cb7d0 2024-04-16T16:22:05.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.173+0000 7f940ce00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f94080cf280 con 0x7f94080cbc80 2024-04-16T16:22:05.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.173+0000 7f940ce00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f94080ce190 con 0x7f940815e830 2024-04-16T16:22:05.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.173+0000 7f9407e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f94080cbc80 0x7f94080cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.173+0000 7f93fea00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f94080cb7d0 0x7f94080cd620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37376/0 (socket says 172.21.15.45:37376) 2024-04-16T16:22:05.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.173+0000 7f93fea00640 1 -- 172.21.15.45:0/2890672302 learned_addr learned my addr 172.21.15.45:0/2890672302 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:05.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.173+0000 7f9407400640 1 --2- 172.21.15.45:0/2890672302 >> v2:172.21.15.103:3300/0 conn(0x7f940815e830 0x7f94080ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.173+0000 7f9407e00640 1 -- 172.21.15.45:0/2890672302 >> v2:172.21.15.103:3300/0 conn(0x7f940815e830 msgr2=0x7f94080ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.173+0000 7f9407e00640 1 --2- 172.21.15.45:0/2890672302 >> v2:172.21.15.103:3300/0 conn(0x7f940815e830 0x7f94080ce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.173+0000 7f9407e00640 1 -- 172.21.15.45:0/2890672302 >> v1:172.21.15.45:6789/0 conn(0x7f94080cb7d0 legacy=0x7f94080cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.174+0000 7f9407e00640 1 -- 172.21.15.45:0/2890672302 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94080ce470 con 0x7f94080cbc80 2024-04-16T16:22:05.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.175+0000 7f9407e00640 1 --2- 172.21.15.45:0/2890672302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94080cbc80 0x7f94080cc410 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f9408070350 tx=0x7f940002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f986f4323001613 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.175+0000 7f9406a00640 1 -- 172.21.15.45:0/2890672302 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9400039070 con 0x7f94080cbc80 2024-04-16T16:22:05.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.175+0000 7f9406a00640 1 -- 172.21.15.45:0/2890672302 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f940002cbe0 con 0x7f94080cbc80 2024-04-16T16:22:05.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.175+0000 7f9406a00640 1 -- 172.21.15.45:0/2890672302 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9400034440 con 0x7f94080cbc80 2024-04-16T16:22:05.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.175+0000 7f940ce00640 1 -- 172.21.15.45:0/2890672302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94080cbc80 msgr2=0x7f94080cc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.175+0000 7f940ce00640 1 --2- 172.21.15.45:0/2890672302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94080cbc80 0x7f94080cc410 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f9408070350 tx=0x7f940002be30 comp rx=0 tx=0).stop 2024-04-16T16:22:05.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.176+0000 7f940ce00640 1 -- 172.21.15.45:0/2890672302 shutdown_connections 2024-04-16T16:22:05.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.176+0000 7f940ce00640 1 --2- 172.21.15.45:0/2890672302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94080cbc80 0x7f94080cc410 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.176+0000 7f940ce00640 1 --2- 172.21.15.45:0/2890672302 >> v2:172.21.15.103:3300/0 conn(0x7f940815e830 0x7f94080ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.176+0000 7f940ce00640 1 -- 172.21.15.45:0/2890672302 >> 172.21.15.45:0/2890672302 conn(0x7f940806b930 msgr2=0x7f940806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:05.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.176+0000 7f940ce00640 1 -- 172.21.15.45:0/2890672302 shutdown_connections 2024-04-16T16:22:05.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.176+0000 7f940ce00640 1 -- 172.21.15.45:0/2890672302 wait complete. 2024-04-16T16:22:05.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.178+0000 7f940ce00640 1 Processor -- start 2024-04-16T16:22:05.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.178+0000 7f940ce00640 1 -- start start 2024-04-16T16:22:05.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.179+0000 7f940ce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94080cb7d0 0x7f9408158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.179+0000 7f940ce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94080cbc80 0x7f9408155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.179+0000 7f940ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f940815e830 0x7f9408149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.179+0000 7f940ce00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f940815ec10 con 0x7f940815e830 2024-04-16T16:22:05.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.179+0000 7f940ce00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f94080cb2b0 con 0x7f94080cb7d0 2024-04-16T16:22:05.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.179+0000 7f940ce00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f94080cac30 con 0x7f94080cbc80 2024-04-16T16:22:05.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.179+0000 7f9407e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f940815e830 0x7f9408149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.179+0000 7f9407e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f940815e830 0x7f9408149e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58094/0 (socket says 172.21.15.45:58094) 2024-04-16T16:22:05.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.180+0000 7f9407e00640 1 -- 172.21.15.45:0/3820134105 learned_addr learned my addr 172.21.15.45:0/3820134105 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:05.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.180+0000 7f93fea00640 1 --2- 172.21.15.45:0/3820134105 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94080cbc80 0x7f9408155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.180+0000 7f9407400640 1 --2- 172.21.15.45:0/3820134105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94080cb7d0 0x7f9408158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.180+0000 7f9407e00640 1 -- 172.21.15.45:0/3820134105 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94080cbc80 msgr2=0x7f9408155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.180+0000 7f9407e00640 1 --2- 172.21.15.45:0/3820134105 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94080cbc80 0x7f9408155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.180+0000 7f9407e00640 1 -- 172.21.15.45:0/3820134105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94080cb7d0 msgr2=0x7f9408158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.180+0000 7f9407e00640 1 --2- 172.21.15.45:0/3820134105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94080cb7d0 0x7f9408158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.180+0000 7f9407e00640 1 -- 172.21.15.45:0/3820134105 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9408156480 con 0x7f940815e830 2024-04-16T16:22:05.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.180+0000 7f9407e00640 1 --2- 172.21.15.45:0/3820134105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f940815e830 0x7f9408149e80 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f9400009e80 tx=0x7f9400004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.181+0000 7f93ffe00640 1 -- 172.21.15.45:0/3820134105 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9400039050 con 0x7f940815e830 2024-04-16T16:22:05.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.181+0000 7f93ffe00640 1 -- 172.21.15.45:0/3820134105 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9400045070 con 0x7f940815e830 2024-04-16T16:22:05.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.181+0000 7f940ce00640 1 -- 172.21.15.45:0/3820134105 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9408156700 con 0x7f940815e830 2024-04-16T16:22:05.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.181+0000 7f93ffe00640 1 -- 172.21.15.45:0/3820134105 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940003e830 con 0x7f940815e830 2024-04-16T16:22:05.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.181+0000 7f940ce00640 1 -- 172.21.15.45:0/3820134105 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9408158970 con 0x7f940815e830 2024-04-16T16:22:05.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.184+0000 7f93ffe00640 1 -- 172.21.15.45:0/3820134105 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7f940003e9d0 con 0x7f940815e830 2024-04-16T16:22:05.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.185+0000 7f93ffe00640 1 --2- 172.21.15.45:0/3820134105 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f93e406ca60 0x7f93e406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.185+0000 7f93ffe00640 1 -- 172.21.15.45:0/3820134105 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f94000aec90 con 0x7f940815e830 2024-04-16T16:22:05.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.185+0000 7f94111d8740 1 Processor -- start 2024-04-16T16:22:05.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.186+0000 7f94111d8740 1 -- start start 2024-04-16T16:22:05.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.186+0000 7f9407400640 1 --2- 172.21.15.45:0/3820134105 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f93e406ca60 0x7f93e406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.186+0000 7f94111d8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499f890 0x5598e49be9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.186+0000 7f94111d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499fc70 0x5598e4978570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.186+0000 7f94111d8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e4978ab0 0x5598e4976440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.186+0000 7f94111d8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5598e48dd9c0 con 0x5598e499f890 2024-04-16T16:22:05.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.186+0000 7f94111d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5598e4884560 con 0x5598e499fc70 2024-04-16T16:22:05.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.186+0000 7f94111d8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5598e48d9980 con 0x5598e4978ab0 2024-04-16T16:22:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.187+0000 7f9407400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499f890 0x5598e49be9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.187+0000 7f9407400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499f890 0x5598e49be9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58096/0 (socket says 172.21.15.45:58096) 2024-04-16T16:22:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.187+0000 7f9407400640 1 -- 172.21.15.45:0/2671572405 learned_addr learned my addr 172.21.15.45:0/2671572405 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.187+0000 7f93fea00640 1 --2- 172.21.15.45:0/2671572405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499fc70 0x5598e4978570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.187+0000 7f9407400640 1 --2- 172.21.15.45:0/3820134105 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f93e406ca60 0x7f93e406ef20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f93f8006fd0 tx=0x7f93f800be10 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.187+0000 7f9407400640 1 -- 172.21.15.45:0/2671572405 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e4978ab0 msgr2=0x5598e4976440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.187+0000 7f9407400640 1 --2- 172.21.15.45:0/2671572405 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e4978ab0 0x5598e4976440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.187+0000 7f9407400640 1 -- 172.21.15.45:0/2671572405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499fc70 msgr2=0x5598e4978570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.187+0000 7f9407400640 1 --2- 172.21.15.45:0/2671572405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499fc70 0x5598e4978570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.187+0000 7f9407400640 1 -- 172.21.15.45:0/2671572405 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598e45f02d0 con 0x5598e499f890 2024-04-16T16:22:05.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.188+0000 7f9407400640 1 --2- 172.21.15.45:0/2671572405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499f890 0x5598e49be9d0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f93f800fb10 tx=0x7f93f801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d12f920dd22f88 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.188+0000 7f93f7e00640 1 -- 172.21.15.45:0/2671572405 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f801c040 con 0x5598e499f890 2024-04-16T16:22:05.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.188+0000 7f93f7e00640 1 -- 172.21.15.45:0/2671572405 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93f8025070 con 0x5598e499f890 2024-04-16T16:22:05.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.188+0000 7f93f7e00640 1 -- 172.21.15.45:0/2671572405 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f8020430 con 0x5598e499f890 2024-04-16T16:22:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.189+0000 7f94111d8740 1 -- 172.21.15.45:0/2671572405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499f890 msgr2=0x5598e49be9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.189+0000 7f94111d8740 1 --2- 172.21.15.45:0/2671572405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499f890 0x5598e49be9d0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f93f800fb10 tx=0x7f93f801a000 comp rx=0 tx=0).stop 2024-04-16T16:22:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.189+0000 7f94111d8740 1 -- 172.21.15.45:0/2671572405 shutdown_connections 2024-04-16T16:22:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.189+0000 7f94111d8740 1 --2- 172.21.15.45:0/2671572405 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e4978ab0 0x5598e4976440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.189+0000 7f94111d8740 1 --2- 172.21.15.45:0/2671572405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499fc70 0x5598e4978570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.189+0000 7f94111d8740 1 --2- 172.21.15.45:0/2671572405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499f890 0x5598e49be9d0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.189+0000 7f94111d8740 1 -- 172.21.15.45:0/2671572405 >> 172.21.15.45:0/2671572405 conn(0x5598e4974b50 msgr2=0x5598e49bd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.189+0000 7f94111d8740 1 -- 172.21.15.45:0/2671572405 shutdown_connections 2024-04-16T16:22:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.189+0000 7f94111d8740 1 -- 172.21.15.45:0/2671572405 wait complete. 2024-04-16T16:22:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.190+0000 7f94111d8740 1 -- start start 2024-04-16T16:22:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.190+0000 7f94111d8740 1 Processor -- start 2024-04-16T16:22:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.191+0000 7f94111d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499f890 0x5598e49a64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.191+0000 7f94111d8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e499fc70 0x5598e49b8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.191+0000 7f94111d8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e49b8b70 0x5598e498c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.191+0000 7f94111d8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5598e4972980 con 0x5598e49b8b70 2024-04-16T16:22:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.191+0000 7f94111d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5598e4897920 con 0x5598e499f890 2024-04-16T16:22:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.191+0000 7f94111d8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5598e4978f30 con 0x5598e499fc70 2024-04-16T16:22:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.191+0000 7f94111d8740 10 client.0 _ll_register_callbacks cb 0x5598e4897310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:22:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.191+0000 7f9407e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e49b8b70 0x5598e498c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.191+0000 7f9407e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e49b8b70 0x5598e498c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58110/0 (socket says 172.21.15.45:58110) 2024-04-16T16:22:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.191+0000 7f9407e00640 1 -- 172.21.15.45:0/2821262803 learned_addr learned my addr 172.21.15.45:0/2821262803 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.192+0000 7f9407e00640 1 -- 172.21.15.45:0/2821262803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e499fc70 msgr2=0x5598e49b8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.192+0000 7f93fea00640 1 --2- 172.21.15.45:0/2821262803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e499fc70 0x5598e49b8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.192+0000 7f9407400640 1 --2- 172.21.15.45:0/2821262803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499f890 0x5598e49a64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.192+0000 7f9407e00640 1 --2- 172.21.15.45:0/2821262803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e499fc70 0x5598e49b8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.192+0000 7f9407e00640 1 -- 172.21.15.45:0/2821262803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499f890 msgr2=0x5598e49a64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.192+0000 7f9407e00640 1 --2- 172.21.15.45:0/2821262803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499f890 0x5598e49a64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.192+0000 7f9407e00640 1 -- 172.21.15.45:0/2821262803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598e46ba6b0 con 0x5598e49b8b70 2024-04-16T16:22:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.192+0000 7f9407e00640 1 --2- 172.21.15.45:0/2821262803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e49b8b70 0x5598e498c2b0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f940007c7f0 tx=0x7f940002cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc550c9516094427 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.192+0000 7f93f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:22:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.192+0000 7f93f4c00640 1 -- 172.21.15.45:0/2821262803 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9400080c10 con 0x5598e49b8b70 2024-04-16T16:22:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.192+0000 7f93f4c00640 1 -- 172.21.15.45:0/2821262803 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f9400078ed0 con 0x5598e49b8b70 2024-04-16T16:22:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.192+0000 7f94111d8740 1 -- 172.21.15.45:0/2821262803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93f8025070 con 0x5598e49b8b70 2024-04-16T16:22:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.192+0000 7f93f4c00640 1 -- 172.21.15.45:0/2821262803 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f9400085440 con 0x5598e49b8b70 2024-04-16T16:22:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.193+0000 7f94111d8740 10 client.9495 fetch_fsmap learned FSMap version 103 2024-04-16T16:22:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.193+0000 7f94111d8740 1 -- 172.21.15.45:0/2821262803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=2+}) -- 0x5598e498db70 con 0x5598e49b8b70 2024-04-16T16:22:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.193+0000 7f93f4c00640 1 -- 172.21.15.45:0/2821262803 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93f8025070 con 0x5598e49b8b70 2024-04-16T16:22:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.193+0000 7f93f4c00640 1 -- 172.21.15.45:0/2821262803 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 103) ==== 3514+0+0 (secure 0 0 0) 0x7f940003ecc0 con 0x5598e49b8b70 2024-04-16T16:22:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.193+0000 7f94111d8740 10 client.9495 fetch_fsmap finished waiting for FSMap version 103 2024-04-16T16:22:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.193+0000 7f94111d8740 10 client.9495 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-16T16:22:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.193+0000 7f94111d8740 20 client.9495 populate_metadata read hostname 'smithi045' 2024-04-16T16:22:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.193+0000 7f94111d8740 1 --2- 172.21.15.45:0/2821262803 >> [v2:172.21.15.103:6834/2561260581,v1:172.21.15.103:6835/2561260581] conn(0x5598e49d15e0 0x5598e498eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.193+0000 7f94111d8740 4 client.9495 mds_command: new command op to 6593 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-16T16:22:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.193+0000 7f94111d8740 1 -- 172.21.15.45:0/2821262803 --> [v2:172.21.15.103:6834/2561260581,v1:172.21.15.103:6835/2561260581] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598e4883b70 con 0x5598e49d15e0 2024-04-16T16:22:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.194+0000 7f9407400640 1 --2- 172.21.15.45:0/2821262803 >> [v2:172.21.15.103:6834/2561260581,v1:172.21.15.103:6835/2561260581] conn(0x5598e49d15e0 0x5598e498eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.195+0000 7f9407400640 1 --2- 172.21.15.45:0/2821262803 >> [v2:172.21.15.103:6834/2561260581,v1:172.21.15.103:6835/2561260581] conn(0x5598e49d15e0 0x5598e498eb60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8f51d1824bdeb5c7 server_cookie=957fb77291cd037d in_seq=0 out_seq=0 2024-04-16T16:22:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.195+0000 7f93f4c00640 10 client.9495 ms_handle_connect on v2:172.21.15.103:6834/2561260581 2024-04-16T16:22:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.198+0000 7f93f4c00640 1 -- 172.21.15.45:0/2821262803 <== mds.1 v2:172.21.15.103:6834/2561260581 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5598e4883b70 con 0x5598e49d15e0 2024-04-16T16:22:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.198+0000 7f93f4c00640 10 client.9495 handle_command_reply: tid=0 2024-04-16T16:22:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.198+0000 7f94111d8740 1 client.9495 shutdown 2024-04-16T16:22:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.198+0000 7f94111d8740 1 -- 172.21.15.45:0/2821262803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e49b8b70 msgr2=0x5598e498c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.198+0000 7f94111d8740 1 --2- 172.21.15.45:0/2821262803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e49b8b70 0x5598e498c2b0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f940007c7f0 tx=0x7f940002cbe0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.198+0000 7f94111d8740 1 -- 172.21.15.45:0/2821262803 shutdown_connections 2024-04-16T16:22:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.198+0000 7f94111d8740 1 --2- 172.21.15.45:0/2821262803 >> [v2:172.21.15.103:6834/2561260581,v1:172.21.15.103:6835/2561260581] conn(0x5598e49d15e0 0x5598e498eb60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.198+0000 7f94111d8740 1 --2- 172.21.15.45:0/2821262803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e49b8b70 0x5598e498c2b0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.198+0000 7f94111d8740 1 --2- 172.21.15.45:0/2821262803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e499fc70 0x5598e49b8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.198+0000 7f93f4c00640 0 client.9495 ms_handle_reset on v2:172.21.15.103:6834/2561260581 2024-04-16T16:22:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.198+0000 7f94111d8740 1 --2- 172.21.15.45:0/2821262803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499f890 0x5598e49a64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.198+0000 7f94111d8740 1 -- 172.21.15.45:0/2821262803 >> 172.21.15.45:0/2821262803 conn(0x5598e4978ab0 msgr2=0x5598e49bd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.198+0000 7f94111d8740 1 -- 172.21.15.45:0/2821262803 shutdown_connections 2024-04-16T16:22:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.198+0000 7f94111d8740 1 -- 172.21.15.45:0/2821262803 wait complete. 2024-04-16T16:22:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.199+0000 7f94111d8740 20 client.9495 trim_cache size 0 max 16384 2024-04-16T16:22:05.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.218+0000 7f94111d8740 1 Processor -- start 2024-04-16T16:22:05.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.219+0000 7f94111d8740 1 -- start start 2024-04-16T16:22:05.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.219+0000 7f94111d8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499f890 0x5598e49bd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.219+0000 7f94111d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499fc70 0x5598e49bdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.219+0000 7f94111d8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e49b8b70 0x5598e498b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.219+0000 7f94111d8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f94082011d0 con 0x5598e499f890 2024-04-16T16:22:05.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.219+0000 7f94111d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9408157b10 con 0x5598e499fc70 2024-04-16T16:22:05.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.219+0000 7f94111d8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9408159e80 con 0x5598e49b8b70 2024-04-16T16:22:05.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.220+0000 7f9407400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499f890 0x5598e49bd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.220+0000 7f9407400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499f890 0x5598e49bd660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58114/0 (socket says 172.21.15.45:58114) 2024-04-16T16:22:05.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.220+0000 7f9407400640 1 -- 172.21.15.45:0/3654210433 learned_addr learned my addr 172.21.15.45:0/3654210433 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:05.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.220+0000 7f9407400640 1 -- 172.21.15.45:0/3654210433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e49b8b70 msgr2=0x5598e498b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.220+0000 7f9407400640 1 --2- 172.21.15.45:0/3654210433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e49b8b70 0x5598e498b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.220+0000 7f93fea00640 1 --2- 172.21.15.45:0/3654210433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499fc70 0x5598e49bdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.220+0000 7f9407400640 1 -- 172.21.15.45:0/3654210433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499fc70 msgr2=0x5598e49bdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.220+0000 7f9407400640 1 --2- 172.21.15.45:0/3654210433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499fc70 0x5598e49bdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.220+0000 7f9407400640 1 -- 172.21.15.45:0/3654210433 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598e4971bd0 con 0x5598e499f890 2024-04-16T16:22:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.221+0000 7f9407400640 1 --2- 172.21.15.45:0/3654210433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499f890 0x5598e49bd660 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f93f8021000 tx=0x7f93f8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14dd5f51cfc4dac2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.221+0000 7f93f7400640 1 -- 172.21.15.45:0/3654210433 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f801c040 con 0x5598e499f890 2024-04-16T16:22:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.221+0000 7f93f7400640 1 -- 172.21.15.45:0/3654210433 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93f803a070 con 0x5598e499f890 2024-04-16T16:22:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.221+0000 7f93f7400640 1 -- 172.21.15.45:0/3654210433 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f8018ad0 con 0x5598e499f890 2024-04-16T16:22:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.221+0000 7f94111d8740 1 -- 172.21.15.45:0/3654210433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499f890 msgr2=0x5598e49bd660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.221+0000 7f94111d8740 1 --2- 172.21.15.45:0/3654210433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499f890 0x5598e49bd660 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f93f8021000 tx=0x7f93f8008040 comp rx=0 tx=0).stop 2024-04-16T16:22:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.222+0000 7f94111d8740 1 -- 172.21.15.45:0/3654210433 shutdown_connections 2024-04-16T16:22:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.222+0000 7f94111d8740 1 --2- 172.21.15.45:0/3654210433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e49b8b70 0x5598e498b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.222+0000 7f94111d8740 1 --2- 172.21.15.45:0/3654210433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499fc70 0x5598e49bdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.222+0000 7f94111d8740 1 --2- 172.21.15.45:0/3654210433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499f890 0x5598e49bd660 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.222+0000 7f94111d8740 1 -- 172.21.15.45:0/3654210433 >> 172.21.15.45:0/3654210433 conn(0x5598e4978ab0 msgr2=0x5598e49e26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.222+0000 7f94111d8740 1 -- 172.21.15.45:0/3654210433 shutdown_connections 2024-04-16T16:22:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.222+0000 7f94111d8740 1 -- 172.21.15.45:0/3654210433 wait complete. 2024-04-16T16:22:05.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.222+0000 7f94111d8740 1 -- start start 2024-04-16T16:22:05.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.223+0000 7f94111d8740 1 Processor -- start 2024-04-16T16:22:05.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.224+0000 7f94111d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499f890 0x5598e4a372f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.224+0000 7f94111d8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499fc70 0x5598e49bd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.224+0000 7f94111d8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e49b8b70 0x5598e49bd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.224+0000 7f94111d8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5598e49d1460 con 0x5598e499fc70 2024-04-16T16:22:05.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.224+0000 7f94111d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5598e4972980 con 0x5598e499f890 2024-04-16T16:22:05.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.224+0000 7f94111d8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5598e49d1ab0 con 0x5598e49b8b70 2024-04-16T16:22:05.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.224+0000 7f94111d8740 10 client.0 _ll_register_callbacks cb 0x7f940007c410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:22:05.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.224+0000 7f93fea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499fc70 0x5598e49bd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.224+0000 7f93fea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499fc70 0x5598e49bd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58130/0 (socket says 172.21.15.45:58130) 2024-04-16T16:22:05.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.224+0000 7f93fea00640 1 -- 172.21.15.45:0/3377982319 learned_addr learned my addr 172.21.15.45:0/3377982319 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.224+0000 7f93fea00640 1 -- 172.21.15.45:0/3377982319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e49b8b70 msgr2=0x5598e49bd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.224+0000 7f93fea00640 1 --2- 172.21.15.45:0/3377982319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e49b8b70 0x5598e49bd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.224+0000 7f93fea00640 1 -- 172.21.15.45:0/3377982319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499f890 msgr2=0x5598e4a372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.224+0000 7f93fea00640 1 --2- 172.21.15.45:0/3377982319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499f890 0x5598e4a372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.224+0000 7f93fea00640 1 -- 172.21.15.45:0/3377982319 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598e49b83c0 con 0x5598e499fc70 2024-04-16T16:22:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.225+0000 7f93fea00640 1 --2- 172.21.15.45:0/3377982319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499fc70 0x5598e49bd430 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f9408153c40 tx=0x7f93f00111f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca5d458239c1de7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.225+0000 7f93f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:22:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.225+0000 7f93f4c00640 1 -- 172.21.15.45:0/3377982319 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f001c060 con 0x5598e499fc70 2024-04-16T16:22:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.225+0000 7f93f4c00640 1 -- 172.21.15.45:0/3377982319 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f93f0011da0 con 0x5598e499fc70 2024-04-16T16:22:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.225+0000 7f94111d8740 1 -- 172.21.15.45:0/3377982319 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93f803a070 con 0x5598e499fc70 2024-04-16T16:22:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.225+0000 7f93f4c00640 1 -- 172.21.15.45:0/3377982319 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f93f0010040 con 0x5598e499fc70 2024-04-16T16:22:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.225+0000 7f94111d8740 10 client.9501 fetch_fsmap learned FSMap version 103 2024-04-16T16:22:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.225+0000 7f94111d8740 1 -- 172.21.15.45:0/3377982319 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=2+}) -- 0x5598e49b71d0 con 0x5598e499fc70 2024-04-16T16:22:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.226+0000 7f93f4c00640 1 -- 172.21.15.45:0/3377982319 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93f803a070 con 0x5598e499fc70 2024-04-16T16:22:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.226+0000 7f93f4c00640 1 -- 172.21.15.45:0/3377982319 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 103) ==== 3514+0+0 (secure 0 0 0) 0x7f93f0006110 con 0x5598e499fc70 2024-04-16T16:22:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.226+0000 7f94111d8740 10 client.9501 fetch_fsmap finished waiting for FSMap version 103 2024-04-16T16:22:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.226+0000 7f94111d8740 10 client.9501 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-16T16:22:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.226+0000 7f94111d8740 20 client.9501 populate_metadata read hostname 'smithi045' 2024-04-16T16:22:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.226+0000 7f94111d8740 1 --2- 172.21.15.45:0/3377982319 >> [v2:172.21.15.103:6834/2561260581,v1:172.21.15.103:6835/2561260581] conn(0x5598e49bb680 0x5598e49bba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.226+0000 7f94111d8740 4 client.9501 mds_command: new command op to 6593 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-16T16:22:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.226+0000 7f94111d8740 1 -- 172.21.15.45:0/3377982319 --> [v2:172.21.15.103:6834/2561260581,v1:172.21.15.103:6835/2561260581] -- command(tid 0: {"prefix": "status"}) -- 0x5598e48eb3e0 con 0x5598e49bb680 2024-04-16T16:22:05.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.227+0000 7f9407400640 1 --2- 172.21.15.45:0/3377982319 >> [v2:172.21.15.103:6834/2561260581,v1:172.21.15.103:6835/2561260581] conn(0x5598e49bb680 0x5598e49bba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.228+0000 7f9407400640 1 --2- 172.21.15.45:0/3377982319 >> [v2:172.21.15.103:6834/2561260581,v1:172.21.15.103:6835/2561260581] conn(0x5598e49bb680 0x5598e49bba60 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8f6eb3131ba4db29 server_cookie=9bdcb9f800fa151b in_seq=0 out_seq=0 2024-04-16T16:22:05.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.228+0000 7f93f4c00640 10 client.9501 ms_handle_connect on v2:172.21.15.103:6834/2561260581 2024-04-16T16:22:05.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.228+0000 7f93f4c00640 1 -- 172.21.15.45:0/3377982319 <== mds.1 v2:172.21.15.103:6834/2561260581 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x5598e48eb3e0 con 0x5598e49bb680 2024-04-16T16:22:05.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.228+0000 7f93f4c00640 10 client.9501 handle_command_reply: tid=0 2024-04-16T16:22:05.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.228+0000 7f94111d8740 1 client.9501 shutdown 2024-04-16T16:22:05.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.228+0000 7f94111d8740 1 -- 172.21.15.45:0/3377982319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499fc70 msgr2=0x5598e49bd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.228+0000 7f94111d8740 1 --2- 172.21.15.45:0/3377982319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499fc70 0x5598e49bd430 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f9408153c40 tx=0x7f93f00111f0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.229+0000 7f94111d8740 1 -- 172.21.15.45:0/3377982319 shutdown_connections 2024-04-16T16:22:05.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.229+0000 7f94111d8740 1 --2- 172.21.15.45:0/3377982319 >> [v2:172.21.15.103:6834/2561260581,v1:172.21.15.103:6835/2561260581] conn(0x5598e49bb680 0x5598e49bba60 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.229+0000 7f94111d8740 1 --2- 172.21.15.45:0/3377982319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5598e49b8b70 0x5598e49bd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.229+0000 7f94111d8740 1 --2- 172.21.15.45:0/3377982319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5598e499fc70 0x5598e49bd430 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.229+0000 7f94111d8740 1 --2- 172.21.15.45:0/3377982319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5598e499f890 0x5598e4a372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.229+0000 7f93f4c00640 0 client.9501 ms_handle_reset on v2:172.21.15.103:6834/2561260581 2024-04-16T16:22:05.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.229+0000 7f94111d8740 1 -- 172.21.15.45:0/3377982319 >> 172.21.15.45:0/3377982319 conn(0x5598e4978ab0 msgr2=0x5598e4972b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:05.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.229+0000 7f94111d8740 1 -- 172.21.15.45:0/3377982319 shutdown_connections 2024-04-16T16:22:05.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.229+0000 7f94111d8740 1 -- 172.21.15.45:0/3377982319 wait complete. 2024-04-16T16:22:05.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.229+0000 7f94111d8740 20 client.9501 trim_cache size 0 max 16384 2024-04-16T16:22:05.230 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:22:05.230 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "748c4b85-0190-4c59-80ad-098a6a5007c0", 2024-04-16T16:22:05.230 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 1, 2024-04-16T16:22:05.230 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 6593, 2024-04-16T16:22:05.230 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-04-16T16:22:05.230 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-04-16T16:22:05.230 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-04-16T16:22:05.231 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 3.1346363209999999, 2024-04-16T16:22:05.231 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 103, 2024-04-16T16:22:05.231 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 143, 2024-04-16T16:22:05.231 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 143, 2024-04-16T16:22:05.231 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 44.958172603999998 2024-04-16T16:22:05.231 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:22:05.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.230+0000 7f940ce00640 1 -- 172.21.15.45:0/3820134105 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f93e406ca60 msgr2=0x7f93e406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.230+0000 7f940ce00640 1 --2- 172.21.15.45:0/3820134105 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f93e406ca60 0x7f93e406ef20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f93f8006fd0 tx=0x7f93f800be10 comp rx=0 tx=0).stop 2024-04-16T16:22:05.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.230+0000 7f940ce00640 1 -- 172.21.15.45:0/3820134105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f940815e830 msgr2=0x7f9408149e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.230+0000 7f940ce00640 1 --2- 172.21.15.45:0/3820134105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f940815e830 0x7f9408149e80 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f9400009e80 tx=0x7f9400004a40 comp rx=0 tx=0).stop 2024-04-16T16:22:05.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.230+0000 7f940ce00640 1 -- 172.21.15.45:0/3820134105 shutdown_connections 2024-04-16T16:22:05.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.230+0000 7f940ce00640 1 --2- 172.21.15.45:0/3820134105 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f93e406ca60 0x7f93e406ef20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.230+0000 7f940ce00640 1 --2- 172.21.15.45:0/3820134105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f940815e830 0x7f9408149e80 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.230+0000 7f940ce00640 1 --2- 172.21.15.45:0/3820134105 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94080cbc80 0x7f9408155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.230+0000 7f940ce00640 1 --2- 172.21.15.45:0/3820134105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94080cb7d0 0x7f9408158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.230+0000 7f940ce00640 1 -- 172.21.15.45:0/3820134105 >> 172.21.15.45:0/3820134105 conn(0x7f940806b930 msgr2=0x7f94080ce8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:05.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.230+0000 7f940ce00640 1 -- 172.21.15.45:0/3820134105 shutdown_connections 2024-04-16T16:22:05.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.230+0000 7f940ce00640 1 -- 172.21.15.45:0/3820134105 wait complete. 2024-04-16T16:22:05.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-16T16:22:05.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.473+0000 7f7634a00640 1 Processor -- start 2024-04-16T16:22:05.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.473+0000 7f7634a00640 1 -- start start 2024-04-16T16:22:05.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.474+0000 7f7634a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76300c6fc0 0x7f763006f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.474+0000 7f7634a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f763015cdc0 0x7f76300c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.474+0000 7f7634a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f763015d600 con 0x7f76300c6320 2024-04-16T16:22:05.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.474+0000 7f7634a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f76300c5670 con 0x7f763015cdc0 2024-04-16T16:22:05.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.474+0000 7f7634a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f763015d8e0 con 0x7f76300c6fc0 2024-04-16T16:22:05.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.475+0000 7f762e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76300c6fc0 0x7f763006f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.475+0000 7f762e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76300c6fc0 0x7f763006f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:47268/0 (socket says 172.21.15.45:47268) 2024-04-16T16:22:05.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.475+0000 7f762e800640 1 -- 172.21.15.45:0/1325217732 learned_addr learned my addr 172.21.15.45:0/1325217732 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:05.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.476+0000 7f7625e00640 1 --2- 172.21.15.45:0/1325217732 >> v2:172.21.15.103:3300/0 conn(0x7f763015cdc0 0x7f76300c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.476+0000 7f762e800640 1 -- 172.21.15.45:0/1325217732 >> v2:172.21.15.103:3300/0 conn(0x7f763015cdc0 msgr2=0x7f76300c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.476+0000 7f762e800640 1 --2- 172.21.15.45:0/1325217732 >> v2:172.21.15.103:3300/0 conn(0x7f763015cdc0 0x7f76300c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.476+0000 7f762e800640 1 -- 172.21.15.45:0/1325217732 >> v1:172.21.15.103:6789/0 conn(0x7f76300c6320 legacy=0x7f763006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.476+0000 7f762e800640 1 -- 172.21.15.45:0/1325217732 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f763015dbc0 con 0x7f76300c6fc0 2024-04-16T16:22:05.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.476+0000 7f7625e00640 1 --2- 172.21.15.45:0/1325217732 >> v2:172.21.15.103:3300/0 conn(0x7f763015cdc0 0x7f76300c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:22:05.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.477+0000 7f762e800640 1 --2- 172.21.15.45:0/1325217732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76300c6fc0 0x7f763006f730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7620009870 tx=0x7f762002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9c7fe7b8b0c948c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.478+0000 7f762de00640 1 -- 172.21.15.45:0/1325217732 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620034070 con 0x7f76300c6fc0 2024-04-16T16:22:05.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.478+0000 7f762de00640 1 -- 172.21.15.45:0/1325217732 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7620030840 con 0x7f76300c6fc0 2024-04-16T16:22:05.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.478+0000 7f762de00640 1 -- 172.21.15.45:0/1325217732 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620030c10 con 0x7f76300c6fc0 2024-04-16T16:22:05.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.479+0000 7f7634a00640 1 -- 172.21.15.45:0/1325217732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76300c6fc0 msgr2=0x7f763006f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.479+0000 7f7634a00640 1 --2- 172.21.15.45:0/1325217732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76300c6fc0 0x7f763006f730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7620009870 tx=0x7f762002b980 comp rx=0 tx=0).stop 2024-04-16T16:22:05.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.479+0000 7f7634a00640 1 -- 172.21.15.45:0/1325217732 shutdown_connections 2024-04-16T16:22:05.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.479+0000 7f7634a00640 1 --2- 172.21.15.45:0/1325217732 >> v2:172.21.15.103:3300/0 conn(0x7f763015cdc0 0x7f76300c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.479+0000 7f7634a00640 1 --2- 172.21.15.45:0/1325217732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76300c6fc0 0x7f763006f730 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.479+0000 7f7634a00640 1 -- 172.21.15.45:0/1325217732 >> 172.21.15.45:0/1325217732 conn(0x7f763006b930 msgr2=0x7f763006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:05.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.479+0000 7f7634a00640 1 -- 172.21.15.45:0/1325217732 shutdown_connections 2024-04-16T16:22:05.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.479+0000 7f7634a00640 1 -- 172.21.15.45:0/1325217732 wait complete. 2024-04-16T16:22:05.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.481+0000 7f7634a00640 1 Processor -- start 2024-04-16T16:22:05.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.481+0000 7f7634a00640 1 -- start start 2024-04-16T16:22:05.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.482+0000 7f7634a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76300c6320 0x7f76301f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.482+0000 7f7634a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76300c6fc0 0x7f763015b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.482+0000 7f7634a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f763015cdc0 0x7f7630159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.482+0000 7f7634a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f763015d3c0 con 0x7f763015cdc0 2024-04-16T16:22:05.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.482+0000 7f7634a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f763015d240 con 0x7f76300c6fc0 2024-04-16T16:22:05.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.482+0000 7f7634a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f76300c73a0 con 0x7f76300c6320 2024-04-16T16:22:05.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.482+0000 7f762f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f763015cdc0 0x7f7630159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.482+0000 7f762f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f763015cdc0 0x7f7630159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58144/0 (socket says 172.21.15.45:58144) 2024-04-16T16:22:05.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.482+0000 7f762f200640 1 -- 172.21.15.45:0/69474023 learned_addr learned my addr 172.21.15.45:0/69474023 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:05.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.483+0000 7f762e800640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76300c6320 0x7f76301f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.483+0000 7f7625e00640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76300c6fc0 0x7f763015b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.483+0000 7f762f200640 1 -- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76300c6320 msgr2=0x7f76301f1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.483+0000 7f762f200640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76300c6320 0x7f76301f1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.483+0000 7f762f200640 1 -- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76300c6fc0 msgr2=0x7f763015b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.483+0000 7f762f200640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76300c6fc0 0x7f763015b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.483+0000 7f762f200640 1 -- 172.21.15.45:0/69474023 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f763015b8c0 con 0x7f763015cdc0 2024-04-16T16:22:05.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.483+0000 7f7625e00640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76300c6fc0 0x7f763015b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:05.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.483+0000 7f762e800640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76300c6320 0x7f76301f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:05.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.483+0000 7f762f200640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f763015cdc0 0x7f7630159010 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f762800c7d0 tx=0x7f762800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.484+0000 7f7627400640 1 -- 172.21.15.45:0/69474023 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7628003b70 con 0x7f763015cdc0 2024-04-16T16:22:05.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.484+0000 7f7627400640 1 -- 172.21.15.45:0/69474023 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7628003d30 con 0x7f763015cdc0 2024-04-16T16:22:05.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.484+0000 7f7634a00640 1 -- 172.21.15.45:0/69474023 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f763015bba0 con 0x7f763015cdc0 2024-04-16T16:22:05.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.484+0000 7f7627400640 1 -- 172.21.15.45:0/69474023 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762800aa90 con 0x7f763015cdc0 2024-04-16T16:22:05.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.484+0000 7f7634a00640 1 -- 172.21.15.45:0/69474023 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f763015c160 con 0x7f763015cdc0 2024-04-16T16:22:05.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.486+0000 7f7627400640 1 -- 172.21.15.45:0/69474023 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7f762800ac30 con 0x7f763015cdc0 2024-04-16T16:22:05.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.487+0000 7f7627400640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f760c06ca60 0x7f760c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.487+0000 7f7627400640 1 -- 172.21.15.45:0/69474023 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f762808be20 con 0x7f763015cdc0 2024-04-16T16:22:05.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.488+0000 7f7637ff9740 1 Processor -- start 2024-04-16T16:22:05.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.488+0000 7f7637ff9740 1 -- start start 2024-04-16T16:22:05.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.488+0000 7f762e800640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f760c06ca60 0x7f760c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.489+0000 7f7637ff9740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a6f0 0x55d34ada98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.489+0000 7f7637ff9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34ad64910 0x55d34ad63490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.489+0000 7f7637ff9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad639d0 0x55d34ad61440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.489+0000 7f7637ff9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d34acc89c0 con 0x55d34ad639d0 2024-04-16T16:22:05.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.489+0000 7f7637ff9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d34ac6f560 con 0x55d34ad64910 2024-04-16T16:22:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.489+0000 7f7637ff9740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d34acc4980 con 0x55d34ad8a6f0 2024-04-16T16:22:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.489+0000 7f762f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad639d0 0x55d34ad61440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.489+0000 7f762e800640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f760c06ca60 0x7f760c06ef20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7620036000 tx=0x7f762003b040 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.489+0000 7f762f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad639d0 0x55d34ad61440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58158/0 (socket says 172.21.15.45:58158) 2024-04-16T16:22:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.489+0000 7f762f200640 1 -- 172.21.15.45:0/632214464 learned_addr learned my addr 172.21.15.45:0/632214464 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.490+0000 7f762e800640 1 --2- 172.21.15.45:0/632214464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a6f0 0x55d34ada98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.490+0000 7f7625e00640 1 --2- 172.21.15.45:0/632214464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34ad64910 0x55d34ad63490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.490+0000 7f762f200640 1 -- 172.21.15.45:0/632214464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a6f0 msgr2=0x55d34ada98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.490+0000 7f762f200640 1 --2- 172.21.15.45:0/632214464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a6f0 0x55d34ada98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.490+0000 7f762f200640 1 -- 172.21.15.45:0/632214464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34ad64910 msgr2=0x55d34ad63490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.490+0000 7f762f200640 1 --2- 172.21.15.45:0/632214464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34ad64910 0x55d34ad63490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.490+0000 7f762f200640 1 -- 172.21.15.45:0/632214464 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d34a9db2d0 con 0x55d34ad639d0 2024-04-16T16:22:05.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.490+0000 7f762e800640 1 --2- 172.21.15.45:0/632214464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a6f0 0x55d34ada98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:05.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.490+0000 7f762f200640 1 --2- 172.21.15.45:0/632214464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad639d0 0x55d34ad61440 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f762808a0c0 tx=0x7f7628059230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895b79cf03c87b46 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.490+0000 7f761f000640 1 -- 172.21.15.45:0/632214464 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762805dc80 con 0x55d34ad639d0 2024-04-16T16:22:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.490+0000 7f761f000640 1 -- 172.21.15.45:0/632214464 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76280624b0 con 0x55d34ad639d0 2024-04-16T16:22:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.490+0000 7f761f000640 1 -- 172.21.15.45:0/632214464 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762808a140 con 0x55d34ad639d0 2024-04-16T16:22:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.491+0000 7f7637ff9740 1 -- 172.21.15.45:0/632214464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad639d0 msgr2=0x55d34ad61440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.491+0000 7f7637ff9740 1 --2- 172.21.15.45:0/632214464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad639d0 0x55d34ad61440 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f762808a0c0 tx=0x7f7628059230 comp rx=0 tx=0).stop 2024-04-16T16:22:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.491+0000 7f7637ff9740 1 -- 172.21.15.45:0/632214464 shutdown_connections 2024-04-16T16:22:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.491+0000 7f7637ff9740 1 --2- 172.21.15.45:0/632214464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad639d0 0x55d34ad61440 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.491+0000 7f7637ff9740 1 --2- 172.21.15.45:0/632214464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34ad64910 0x55d34ad63490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.491+0000 7f7637ff9740 1 --2- 172.21.15.45:0/632214464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a6f0 0x55d34ada98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.491+0000 7f7637ff9740 1 -- 172.21.15.45:0/632214464 >> 172.21.15.45:0/632214464 conn(0x55d34ad5fb50 msgr2=0x55d34ada86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.491+0000 7f7637ff9740 1 -- 172.21.15.45:0/632214464 shutdown_connections 2024-04-16T16:22:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.491+0000 7f7637ff9740 1 -- 172.21.15.45:0/632214464 wait complete. 2024-04-16T16:22:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.492+0000 7f7637ff9740 1 -- start start 2024-04-16T16:22:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.492+0000 7f7637ff9740 1 Processor -- start 2024-04-16T16:22:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.493+0000 7f7637ff9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad64910 0x55d34ad84c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.493+0000 7f7637ff9740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a6f0 0x55d34ad5c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.493+0000 7f7637ff9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34adac2b0 0x55d34ad5ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.493+0000 7f7637ff9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d34ad5d8e0 con 0x55d34ad64910 2024-04-16T16:22:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.493+0000 7f7637ff9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d34ac82920 con 0x55d34adac2b0 2024-04-16T16:22:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.493+0000 7f7637ff9740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d34ada8de0 con 0x55d34ad8a6f0 2024-04-16T16:22:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.493+0000 7f7637ff9740 10 client.0 _ll_register_callbacks cb 0x55d34ac82310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:22:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.493+0000 7f762e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad64910 0x55d34ad84c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.493+0000 7f762e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad64910 0x55d34ad84c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58174/0 (socket says 172.21.15.45:58174) 2024-04-16T16:22:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.493+0000 7f762e800640 1 -- 172.21.15.45:0/1026159987 learned_addr learned my addr 172.21.15.45:0/1026159987 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.494+0000 7f762f200640 1 --2- 172.21.15.45:0/1026159987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34adac2b0 0x55d34ad5ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.494+0000 7f762e800640 1 -- 172.21.15.45:0/1026159987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a6f0 msgr2=0x55d34ad5c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.494+0000 7f7625e00640 1 --2- 172.21.15.45:0/1026159987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a6f0 0x55d34ad5c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.494+0000 7f762e800640 1 --2- 172.21.15.45:0/1026159987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a6f0 0x55d34ad5c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.494+0000 7f762e800640 1 -- 172.21.15.45:0/1026159987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34adac2b0 msgr2=0x55d34ad5ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.494+0000 7f762e800640 1 --2- 172.21.15.45:0/1026159987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34adac2b0 0x55d34ad5ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.494+0000 7f762e800640 1 -- 172.21.15.45:0/1026159987 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d34aaa56b0 con 0x55d34ad64910 2024-04-16T16:22:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.494+0000 7f762f200640 1 --2- 172.21.15.45:0/1026159987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34adac2b0 0x55d34ad5ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.494+0000 7f7625e00640 1 --2- 172.21.15.45:0/1026159987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a6f0 0x55d34ad5c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.494+0000 7f762e800640 1 --2- 172.21.15.45:0/1026159987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad64910 0x55d34ad84c40 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f7620009960 tx=0x7f762002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cafcec0019fcba4 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.494+0000 7f7613400640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:22:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.495+0000 7f7613400640 1 -- 172.21.15.45:0/1026159987 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620053060 con 0x55d34ad64910 2024-04-16T16:22:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.495+0000 7f7613400640 1 -- 172.21.15.45:0/1026159987 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f76200438d0 con 0x55d34ad64910 2024-04-16T16:22:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.495+0000 7f7637ff9740 1 -- 172.21.15.45:0/1026159987 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76280624b0 con 0x55d34ad64910 2024-04-16T16:22:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.495+0000 7f7613400640 1 -- 172.21.15.45:0/1026159987 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f7620034040 con 0x55d34ad64910 2024-04-16T16:22:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.495+0000 7f7637ff9740 10 client.9510 fetch_fsmap learned FSMap version 103 2024-04-16T16:22:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.495+0000 7f7637ff9740 1 -- 172.21.15.45:0/1026159987 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=2+}) -- 0x55d34ad8bfc0 con 0x55d34ad64910 2024-04-16T16:22:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.495+0000 7f7613400640 1 -- 172.21.15.45:0/1026159987 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76280624b0 con 0x55d34ad64910 2024-04-16T16:22:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.495+0000 7f7613400640 1 -- 172.21.15.45:0/1026159987 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 103) ==== 3514+0+0 (secure 0 0 0) 0x7f7620043ad0 con 0x55d34ad64910 2024-04-16T16:22:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.495+0000 7f7637ff9740 10 client.9510 fetch_fsmap finished waiting for FSMap version 103 2024-04-16T16:22:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.495+0000 7f7637ff9740 10 client.9510 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-16T16:22:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.496+0000 7f7637ff9740 20 client.9510 populate_metadata read hostname 'smithi045' 2024-04-16T16:22:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.496+0000 7f7637ff9740 1 --2- 172.21.15.45:0/1026159987 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55d34ad8d5a0 0x55d34ad8fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.496+0000 7f7637ff9740 4 client.9510 mds_command: new command op to 6596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-16T16:22:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.496+0000 7f7637ff9740 1 -- 172.21.15.45:0/1026159987 --> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d34ac6eb70 con 0x55d34ad8d5a0 2024-04-16T16:22:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.496+0000 7f7625e00640 1 --2- 172.21.15.45:0/1026159987 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55d34ad8d5a0 0x55d34ad8fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.497+0000 7f7613400640 10 client.9510 ms_handle_connect on v2:172.21.15.103:6836/3252460318 2024-04-16T16:22:05.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.497+0000 7f7625e00640 1 --2- 172.21.15.45:0/1026159987 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55d34ad8d5a0 0x55d34ad8fbb0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26803fa9c1a6ce49 server_cookie=eecde88de589aa23 in_seq=0 out_seq=0 2024-04-16T16:22:05.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.500+0000 7f7613400640 1 -- 172.21.15.45:0/1026159987 <== mds.0 v2:172.21.15.103:6836/3252460318 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d34ac6eb70 con 0x55d34ad8d5a0 2024-04-16T16:22:05.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.500+0000 7f7613400640 10 client.9510 handle_command_reply: tid=0 2024-04-16T16:22:05.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.500+0000 7f7637ff9740 1 client.9510 shutdown 2024-04-16T16:22:05.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.500+0000 7f7637ff9740 1 -- 172.21.15.45:0/1026159987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad64910 msgr2=0x55d34ad84c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.500+0000 7f7637ff9740 1 --2- 172.21.15.45:0/1026159987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad64910 0x55d34ad84c40 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f7620009960 tx=0x7f762002e000 comp rx=0 tx=0).stop 2024-04-16T16:22:05.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.500+0000 7f7637ff9740 1 -- 172.21.15.45:0/1026159987 shutdown_connections 2024-04-16T16:22:05.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.500+0000 7f7637ff9740 1 --2- 172.21.15.45:0/1026159987 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55d34ad8d5a0 0x55d34ad8fbb0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.500+0000 7f7637ff9740 1 --2- 172.21.15.45:0/1026159987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34adac2b0 0x55d34ad5ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.500+0000 7f7637ff9740 1 --2- 172.21.15.45:0/1026159987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a6f0 0x55d34ad5c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.500+0000 7f7637ff9740 1 --2- 172.21.15.45:0/1026159987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad64910 0x55d34ad84c40 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.500+0000 7f7613400640 0 client.9510 ms_handle_reset on v2:172.21.15.103:6836/3252460318 2024-04-16T16:22:05.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.500+0000 7f7637ff9740 1 -- 172.21.15.45:0/1026159987 >> 172.21.15.45:0/1026159987 conn(0x55d34ad639d0 msgr2=0x55d34adaad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:05.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.500+0000 7f7637ff9740 1 -- 172.21.15.45:0/1026159987 shutdown_connections 2024-04-16T16:22:05.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.500+0000 7f7637ff9740 1 -- 172.21.15.45:0/1026159987 wait complete. 2024-04-16T16:22:05.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.500+0000 7f7637ff9740 20 client.9510 trim_cache size 0 max 16384 2024-04-16T16:22:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.529+0000 7f7637ff9740 1 Processor -- start 2024-04-16T16:22:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.530+0000 7f7637ff9740 1 -- start start 2024-04-16T16:22:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.531+0000 7f7637ff9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34ad64910 0x55d34ad8e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.531+0000 7f7637ff9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad8a6f0 0x55d34ad873c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.531+0000 7f7637ff9740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34adac2b0 0x55d34ad84c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.531+0000 7f7637ff9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f76301fce70 con 0x55d34ad8a6f0 2024-04-16T16:22:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.531+0000 7f7637ff9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f76301586a0 con 0x55d34ad64910 2024-04-16T16:22:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.531+0000 7f7637ff9740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f76300ce360 con 0x55d34adac2b0 2024-04-16T16:22:05.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.531+0000 7f7625e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad8a6f0 0x55d34ad873c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.531+0000 7f7625e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad8a6f0 0x55d34ad873c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58184/0 (socket says 172.21.15.45:58184) 2024-04-16T16:22:05.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.531+0000 7f7625e00640 1 -- 172.21.15.45:0/192361853 learned_addr learned my addr 172.21.15.45:0/192361853 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:05.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.531+0000 7f762e800640 1 --2- 172.21.15.45:0/192361853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34ad64910 0x55d34ad8e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.531+0000 7f762f200640 1 --2- 172.21.15.45:0/192361853 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34adac2b0 0x55d34ad84c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.532+0000 7f7625e00640 1 -- 172.21.15.45:0/192361853 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34adac2b0 msgr2=0x55d34ad84c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.532+0000 7f7625e00640 1 --2- 172.21.15.45:0/192361853 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34adac2b0 0x55d34ad84c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.532+0000 7f7625e00640 1 -- 172.21.15.45:0/192361853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34ad64910 msgr2=0x55d34ad8e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.532+0000 7f7625e00640 1 --2- 172.21.15.45:0/192361853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34ad64910 0x55d34ad8e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.532+0000 7f7625e00640 1 -- 172.21.15.45:0/192361853 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d34ada7620 con 0x55d34ad8a6f0 2024-04-16T16:22:05.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.533+0000 7f7625e00640 1 --2- 172.21.15.45:0/192361853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad8a6f0 0x55d34ad873c0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x55d34ad60ae0 tx=0x7f761800bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d60bf161fcc3db server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.533+0000 7f761e600640 1 -- 172.21.15.45:0/192361853 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7618015870 con 0x55d34ad8a6f0 2024-04-16T16:22:05.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.533+0000 7f761e600640 1 -- 172.21.15.45:0/192361853 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f761801e030 con 0x55d34ad8a6f0 2024-04-16T16:22:05.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.533+0000 7f761e600640 1 -- 172.21.15.45:0/192361853 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7618015c70 con 0x55d34ad8a6f0 2024-04-16T16:22:05.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.534+0000 7f7637ff9740 1 -- 172.21.15.45:0/192361853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad8a6f0 msgr2=0x55d34ad873c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.534+0000 7f7637ff9740 1 --2- 172.21.15.45:0/192361853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad8a6f0 0x55d34ad873c0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x55d34ad60ae0 tx=0x7f761800bce0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.534+0000 7f7637ff9740 1 -- 172.21.15.45:0/192361853 shutdown_connections 2024-04-16T16:22:05.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.534+0000 7f7637ff9740 1 --2- 172.21.15.45:0/192361853 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34adac2b0 0x55d34ad84c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.534+0000 7f7637ff9740 1 --2- 172.21.15.45:0/192361853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad8a6f0 0x55d34ad873c0 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.534+0000 7f7637ff9740 1 --2- 172.21.15.45:0/192361853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34ad64910 0x55d34ad8e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.534+0000 7f7637ff9740 1 -- 172.21.15.45:0/192361853 >> 172.21.15.45:0/192361853 conn(0x55d34ad639d0 msgr2=0x55d34ad62e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:05.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.534+0000 7f7637ff9740 1 -- 172.21.15.45:0/192361853 shutdown_connections 2024-04-16T16:22:05.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.534+0000 7f7637ff9740 1 -- 172.21.15.45:0/192361853 wait complete. 2024-04-16T16:22:05.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.535+0000 7f7637ff9740 1 -- start start 2024-04-16T16:22:05.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.535+0000 7f7637ff9740 1 Processor -- start 2024-04-16T16:22:05.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.537+0000 7f7637ff9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad64910 0x55d34ae269c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.537+0000 7f7637ff9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34adac2b0 0x55d34ad90570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.537+0000 7f7637ff9740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a040 0x55d34adaade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.537+0000 7f7637ff9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d34ad8d420 con 0x55d34ad64910 2024-04-16T16:22:05.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.537+0000 7f7637ff9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d34ad903f0 con 0x55d34adac2b0 2024-04-16T16:22:05.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.537+0000 7f7637ff9740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d34ad8d9e0 con 0x55d34ad8a040 2024-04-16T16:22:05.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.537+0000 7f7637ff9740 10 client.0 _ll_register_callbacks cb 0x7f7620030d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-16T16:22:05.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.537+0000 7f762e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad64910 0x55d34ae269c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.537+0000 7f762e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad64910 0x55d34ae269c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58198/0 (socket says 172.21.15.45:58198) 2024-04-16T16:22:05.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.537+0000 7f762e800640 1 -- 172.21.15.45:0/768896640 learned_addr learned my addr 172.21.15.45:0/768896640 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:05.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.538+0000 7f762f200640 1 --2- 172.21.15.45:0/768896640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a040 0x55d34adaade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.538+0000 7f762e800640 1 -- 172.21.15.45:0/768896640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a040 msgr2=0x55d34adaade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.538+0000 7f762e800640 1 --2- 172.21.15.45:0/768896640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a040 0x55d34adaade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.538+0000 7f762e800640 1 -- 172.21.15.45:0/768896640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34adac2b0 msgr2=0x55d34ad90570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.538+0000 7f762e800640 1 --2- 172.21.15.45:0/768896640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34adac2b0 0x55d34ad90570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.538+0000 7f762e800640 1 -- 172.21.15.45:0/768896640 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d34ad5cd40 con 0x55d34ad64910 2024-04-16T16:22:05.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.538+0000 7f762e800640 1 --2- 172.21.15.45:0/768896640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad64910 0x55d34ae269c0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f762002bf90 tx=0x7f7620004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a2e8f7c64e39297 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.538+0000 7f7613400640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-16T16:22:05.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.538+0000 7f7613400640 1 -- 172.21.15.45:0/768896640 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620062070 con 0x55d34ad64910 2024-04-16T16:22:05.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.538+0000 7f7613400640 1 -- 172.21.15.45:0/768896640 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f76200436d0 con 0x55d34ad64910 2024-04-16T16:22:05.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.538+0000 7f7637ff9740 1 -- 172.21.15.45:0/768896640 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f761801e030 con 0x55d34ad64910 2024-04-16T16:22:05.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.539+0000 7f7613400640 1 -- 172.21.15.45:0/768896640 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f762006d050 con 0x55d34ad64910 2024-04-16T16:22:05.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.539+0000 7f7637ff9740 10 client.9516 fetch_fsmap learned FSMap version 103 2024-04-16T16:22:05.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.539+0000 7f7637ff9740 1 -- 172.21.15.45:0/768896640 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=2+}) -- 0x55d34ad90da0 con 0x55d34ad64910 2024-04-16T16:22:05.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.539+0000 7f7613400640 1 -- 172.21.15.45:0/768896640 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f761801e030 con 0x55d34ad64910 2024-04-16T16:22:05.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.539+0000 7f7613400640 1 -- 172.21.15.45:0/768896640 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 103) ==== 3514+0+0 (secure 0 0 0) 0x7f7620070020 con 0x55d34ad64910 2024-04-16T16:22:05.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.539+0000 7f7637ff9740 10 client.9516 fetch_fsmap finished waiting for FSMap version 103 2024-04-16T16:22:05.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.539+0000 7f7637ff9740 10 client.9516 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-16T16:22:05.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.539+0000 7f7637ff9740 20 client.9516 populate_metadata read hostname 'smithi045' 2024-04-16T16:22:05.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.539+0000 7f7637ff9740 1 --2- 172.21.15.45:0/768896640 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55d34ae30600 0x55d34ae329f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.539+0000 7f7637ff9740 4 client.9516 mds_command: new command op to 6596 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-16T16:22:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.539+0000 7f7637ff9740 1 -- 172.21.15.45:0/768896640 --> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] -- command(tid 0: {"prefix": "status"}) -- 0x55d34acd63e0 con 0x55d34ae30600 2024-04-16T16:22:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.540+0000 7f7625e00640 1 --2- 172.21.15.45:0/768896640 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55d34ae30600 0x55d34ae329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.541+0000 7f7625e00640 1 --2- 172.21.15.45:0/768896640 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55d34ae30600 0x55d34ae329f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f3fc794cb91a3d6 server_cookie=a747302148378550 in_seq=0 out_seq=0 2024-04-16T16:22:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.541+0000 7f7613400640 10 client.9516 ms_handle_connect on v2:172.21.15.103:6836/3252460318 2024-04-16T16:22:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.542+0000 7f7613400640 1 -- 172.21.15.45:0/768896640 <== mds.0 v2:172.21.15.103:6836/3252460318 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x55d34acd63e0 con 0x55d34ae30600 2024-04-16T16:22:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.542+0000 7f7613400640 10 client.9516 handle_command_reply: tid=0 2024-04-16T16:22:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.542+0000 7f7637ff9740 1 client.9516 shutdown 2024-04-16T16:22:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.542+0000 7f7637ff9740 1 -- 172.21.15.45:0/768896640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad64910 msgr2=0x55d34ae269c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.542+0000 7f7637ff9740 1 --2- 172.21.15.45:0/768896640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad64910 0x55d34ae269c0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f762002bf90 tx=0x7f7620004730 comp rx=0 tx=0).stop 2024-04-16T16:22:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.542+0000 7f7637ff9740 1 -- 172.21.15.45:0/768896640 shutdown_connections 2024-04-16T16:22:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.542+0000 7f7637ff9740 1 --2- 172.21.15.45:0/768896640 >> [v2:172.21.15.103:6836/3252460318,v1:172.21.15.103:6837/3252460318] conn(0x55d34ae30600 0x55d34ae329f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.542+0000 7f7637ff9740 1 --2- 172.21.15.45:0/768896640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d34ad8a040 0x55d34adaade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.542+0000 7f7637ff9740 1 --2- 172.21.15.45:0/768896640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d34adac2b0 0x55d34ad90570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.542+0000 7f7613400640 0 client.9516 ms_handle_reset on v2:172.21.15.103:6836/3252460318 2024-04-16T16:22:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.542+0000 7f7637ff9740 1 --2- 172.21.15.45:0/768896640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d34ad64910 0x55d34ae269c0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.542+0000 7f7637ff9740 1 -- 172.21.15.45:0/768896640 >> 172.21.15.45:0/768896640 conn(0x55d34ad639d0 msgr2=0x55d34ad58e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.542+0000 7f7637ff9740 1 -- 172.21.15.45:0/768896640 shutdown_connections 2024-04-16T16:22:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.542+0000 7f7637ff9740 1 -- 172.21.15.45:0/768896640 wait complete. 2024-04-16T16:22:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.542+0000 7f7637ff9740 20 client.9516 trim_cache size 0 max 16384 2024-04-16T16:22:05.549 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:22:05.550 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "748c4b85-0190-4c59-80ad-098a6a5007c0", 2024-04-16T16:22:05.550 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-04-16T16:22:05.550 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 6596, 2024-04-16T16:22:05.550 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-04-16T16:22:05.550 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-04-16T16:22:05.550 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-04-16T16:22:05.550 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 38.949943646000001, 2024-04-16T16:22:05.551 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 103, 2024-04-16T16:22:05.551 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 143, 2024-04-16T16:22:05.551 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 143, 2024-04-16T16:22:05.551 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 45.266199041 2024-04-16T16:22:05.551 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:22:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.543+0000 7f7634a00640 1 -- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f760c06ca60 msgr2=0x7f760c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.543+0000 7f7634a00640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f760c06ca60 0x7f760c06ef20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7620036000 tx=0x7f762003b040 comp rx=0 tx=0).stop 2024-04-16T16:22:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.543+0000 7f7634a00640 1 -- 172.21.15.45:0/69474023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f763015cdc0 msgr2=0x7f7630159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.543+0000 7f7634a00640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f763015cdc0 0x7f7630159010 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f762800c7d0 tx=0x7f762800cc90 comp rx=0 tx=0).stop 2024-04-16T16:22:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.543+0000 7f7634a00640 1 -- 172.21.15.45:0/69474023 shutdown_connections 2024-04-16T16:22:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.543+0000 7f7634a00640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f760c06ca60 0x7f760c06ef20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.543+0000 7f7634a00640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f763015cdc0 0x7f7630159010 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.543+0000 7f7634a00640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76300c6fc0 0x7f763015b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.543+0000 7f7634a00640 1 --2- 172.21.15.45:0/69474023 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76300c6320 0x7f76301f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.543+0000 7f7634a00640 1 -- 172.21.15.45:0/69474023 >> 172.21.15.45:0/69474023 conn(0x7f763006b930 msgr2=0x7f763015eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.543+0000 7f7634a00640 1 -- 172.21.15.45:0/69474023 shutdown_connections 2024-04-16T16:22:05.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.543+0000 7f7634a00640 1 -- 172.21.15.45:0/69474023 wait complete. 2024-04-16T16:22:05.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:22:05.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.783+0000 7f17ee600640 1 Processor -- start 2024-04-16T16:22:05.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.784+0000 7f17ee600640 1 -- start start 2024-04-16T16:22:05.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.785+0000 7f17ee600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17e815eb00 0x7f17e80b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.785+0000 7f17ee600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f17e8072150 0x7f17e806cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.786+0000 7f17ee600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17e80b5d40 con 0x7f17e8162f40 2024-04-16T16:22:05.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.786+0000 7f17ee600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f17e80b6020 con 0x7f17e8072150 2024-04-16T16:22:05.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.786+0000 7f17ee600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f17e80b6300 con 0x7f17e815eb00 2024-04-16T16:22:05.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.786+0000 7f17ed200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17e815eb00 0x7f17e80b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.787+0000 7f17ed200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17e815eb00 0x7f17e80b56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:47316/0 (socket says 172.21.15.45:47316) 2024-04-16T16:22:05.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.787+0000 7f17ed200640 1 -- 172.21.15.45:0/4149927030 learned_addr learned my addr 172.21.15.45:0/4149927030 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:05.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.787+0000 7f17edc00640 1 --2- 172.21.15.45:0/4149927030 >> v2:172.21.15.103:3300/0 conn(0x7f17e8072150 0x7f17e806cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.788+0000 7f17ed200640 1 -- 172.21.15.45:0/4149927030 >> v2:172.21.15.103:3300/0 conn(0x7f17e8072150 msgr2=0x7f17e806cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:05.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.788+0000 7f17ed200640 1 --2- 172.21.15.45:0/4149927030 >> v2:172.21.15.103:3300/0 conn(0x7f17e8072150 0x7f17e806cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.788+0000 7f17ed200640 1 -- 172.21.15.45:0/4149927030 >> v1:172.21.15.103:6789/0 conn(0x7f17e8162f40 legacy=0x7f17e8071a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.788+0000 7f17ed200640 1 -- 172.21.15.45:0/4149927030 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e80b65e0 con 0x7f17e815eb00 2024-04-16T16:22:05.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.789+0000 7f17ed200640 1 --2- 172.21.15.45:0/4149927030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17e815eb00 0x7f17e80b56a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009870 tx=0x7f17d802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4260514af9722b5d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.790+0000 7f17e7400640 1 -- 172.21.15.45:0/4149927030 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8035020 con 0x7f17e815eb00 2024-04-16T16:22:05.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.790+0000 7f17e7400640 1 -- 172.21.15.45:0/4149927030 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17d8031640 con 0x7f17e815eb00 2024-04-16T16:22:05.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.790+0000 7f17e7400640 1 -- 172.21.15.45:0/4149927030 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d802d040 con 0x7f17e815eb00 2024-04-16T16:22:05.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.790+0000 7f17ee600640 1 -- 172.21.15.45:0/4149927030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17e815eb00 msgr2=0x7f17e80b56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.790+0000 7f17ee600640 1 --2- 172.21.15.45:0/4149927030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17e815eb00 0x7f17e80b56a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009870 tx=0x7f17d802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.791+0000 7f17ee600640 1 -- 172.21.15.45:0/4149927030 shutdown_connections 2024-04-16T16:22:05.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.791+0000 7f17ee600640 1 --2- 172.21.15.45:0/4149927030 >> v2:172.21.15.103:3300/0 conn(0x7f17e8072150 0x7f17e806cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.791+0000 7f17ee600640 1 --2- 172.21.15.45:0/4149927030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17e815eb00 0x7f17e80b56a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.791+0000 7f17ee600640 1 -- 172.21.15.45:0/4149927030 >> 172.21.15.45:0/4149927030 conn(0x7f17e80b43a0 msgr2=0x7f17e80b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:05.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.791+0000 7f17ee600640 1 -- 172.21.15.45:0/4149927030 shutdown_connections 2024-04-16T16:22:05.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.791+0000 7f17ee600640 1 -- 172.21.15.45:0/4149927030 wait complete. 2024-04-16T16:22:05.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.793+0000 7f17ee600640 1 Processor -- start 2024-04-16T16:22:05.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.793+0000 7f17ee600640 1 -- start start 2024-04-16T16:22:05.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.794+0000 7f17ee600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17e8072150 0x7f17e81f4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.794+0000 7f17ee600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e815eb00 0x7f17e815d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.794+0000 7f17ee600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17e8162f40 0x7f17e815da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.794+0000 7f17ee600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f17e806d2f0 con 0x7f17e815eb00 2024-04-16T16:22:05.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.794+0000 7f17ee600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f17e8163380 con 0x7f17e8162f40 2024-04-16T16:22:05.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.794+0000 7f17ee600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f17e806d470 con 0x7f17e8072150 2024-04-16T16:22:05.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.794+0000 7f17e7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e815eb00 0x7f17e815d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.794+0000 7f17e7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e815eb00 0x7f17e815d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58200/0 (socket says 172.21.15.45:58200) 2024-04-16T16:22:05.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.794+0000 7f17e7e00640 1 -- 172.21.15.45:0/2694979212 learned_addr learned my addr 172.21.15.45:0/2694979212 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:05.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.795+0000 7f17edc00640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17e8162f40 0x7f17e815da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.795+0000 7f17ed200640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17e8072150 0x7f17e81f4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.795+0000 7f17e7e00640 1 -- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17e8072150 msgr2=0x7f17e81f4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.795+0000 7f17e7e00640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17e8072150 0x7f17e81f4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.795+0000 7f17e7e00640 1 -- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17e8162f40 msgr2=0x7f17e815da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:05.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.795+0000 7f17e7e00640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17e8162f40 0x7f17e815da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:05.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.795+0000 7f17e7e00640 1 -- 172.21.15.45:0/2694979212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e815c750 con 0x7f17e815eb00 2024-04-16T16:22:05.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.795+0000 7f17edc00640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17e8162f40 0x7f17e815da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:05.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.795+0000 7f17ed200640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17e8072150 0x7f17e81f4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:05.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.795+0000 7f17e7e00640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e815eb00 0x7f17e815d510 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f17d000c470 tx=0x7f17d000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.795+0000 7f17e4c00640 1 -- 172.21.15.45:0/2694979212 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d00052d0 con 0x7f17e815eb00 2024-04-16T16:22:05.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.795+0000 7f17e4c00640 1 -- 172.21.15.45:0/2694979212 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17d0005490 con 0x7f17e815eb00 2024-04-16T16:22:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.795+0000 7f17ee600640 1 -- 172.21.15.45:0/2694979212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17e815ca30 con 0x7f17e815eb00 2024-04-16T16:22:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.795+0000 7f17e4c00640 1 -- 172.21.15.45:0/2694979212 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d0007260 con 0x7f17e815eb00 2024-04-16T16:22:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.796+0000 7f17ee600640 1 -- 172.21.15.45:0/2694979212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17e815c370 con 0x7f17e815eb00 2024-04-16T16:22:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.797+0000 7f17ee600640 1 -- 172.21.15.45:0/2694979212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17e80b4f50 con 0x7f17e815eb00 2024-04-16T16:22:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.798+0000 7f17e4c00640 1 -- 172.21.15.45:0/2694979212 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7f17d0007400 con 0x7f17e815eb00 2024-04-16T16:22:05.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.799+0000 7f17e4c00640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f17c406ca90 0x7f17c406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:05.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.799+0000 7f17e4c00640 1 -- 172.21.15.45:0/2694979212 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f17d008b4f0 con 0x7f17e815eb00 2024-04-16T16:22:05.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.799+0000 7f17ed200640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f17c406ca90 0x7f17c406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:05.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.801+0000 7f17ed200640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f17c406ca90 0x7f17c406ef50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f17d8036000 tx=0x7f17d8008030 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:05.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:05.804+0000 7f17e4c00640 1 -- 172.21.15.45:0/2694979212 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f17d0059500 con 0x7f17e815eb00 2024-04-16T16:22:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:06.051+0000 7f17ee600640 1 -- 172.21.15.45:0/2694979212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17e80b1a40 con 0x7f17e815eb00 2024-04-16T16:22:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:06.053+0000 7f17e4c00640 1 -- 172.21.15.45:0/2694979212 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+5211 (secure 0 0 0) 0x7f17d0058e00 con 0x7f17e815eb00 2024-04-16T16:22:06.053 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:06.053 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2585865379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2585865379},{"type":"v1","addr":"172.21.15.45:6834","nonce":2585865379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6590,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/777087937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":777087937},{"type":"v1","addr":"172.21.15.45:6835","nonce":777087937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:21:26.066855+0000","modified":"2024-04-16T16:22:03.091438+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6596,"mds_1":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":1,"incarnation":102,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6835/2561260581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2561260581},{"type":"v1","addr":"172.21.15.103:6835","nonce":2561260581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6596":{"gid":6596,"name":"d","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6837/3252460318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3252460318},{"type":"v1","addr":"172.21.15.103:6837","nonce":3252460318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6596,"qdb_cluster":[6593,6596]},"id":10}]} 2024-04-16T16:22:06.053 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 103 2024-04-16T16:22:06.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:06.055+0000 7f17ee600640 1 -- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f17c406ca90 msgr2=0x7f17c406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:06.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:06.055+0000 7f17ee600640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f17c406ca90 0x7f17c406ef50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f17d8036000 tx=0x7f17d8008030 comp rx=0 tx=0).stop 2024-04-16T16:22:06.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:06.055+0000 7f17ee600640 1 -- 172.21.15.45:0/2694979212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e815eb00 msgr2=0x7f17e815d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:06.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:06.055+0000 7f17ee600640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e815eb00 0x7f17e815d510 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f17d000c470 tx=0x7f17d000c930 comp rx=0 tx=0).stop 2024-04-16T16:22:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:06.055+0000 7f17ee600640 1 -- 172.21.15.45:0/2694979212 shutdown_connections 2024-04-16T16:22:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:06.055+0000 7f17ee600640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f17c406ca90 0x7f17c406ef50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:06.055+0000 7f17ee600640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17e8162f40 0x7f17e815da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:06.055+0000 7f17ee600640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e815eb00 0x7f17e815d510 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:06.055+0000 7f17ee600640 1 --2- 172.21.15.45:0/2694979212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f17e8072150 0x7f17e81f4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:06.055+0000 7f17ee600640 1 -- 172.21.15.45:0/2694979212 >> 172.21.15.45:0/2694979212 conn(0x7f17e80b43a0 msgr2=0x7f17e81624c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:06.056+0000 7f17ee600640 1 -- 172.21.15.45:0/2694979212 shutdown_connections 2024-04-16T16:22:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:06.056+0000 7f17ee600640 1 -- 172.21.15.45:0/2694979212 wait complete. 2024-04-16T16:22:06.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:06.076 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir test_mds_metrics_0) 2024-04-16T16:22:06.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:06.149 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 test_mds_metrics_0) 2024-04-16T16:22:06.209 INFO:tasks.cephfs.test_mds_metrics:pinning directory test_mds_metrics_0 to rank 0 2024-04-16T16:22:06.209 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:06.209 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:06.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:06.266 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.0 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:06.359 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:06.359 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:06.359 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347851 s, 30.1 MB/s 2024-04-16T16:22:06.361 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:06.361 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:06.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:06.415 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.1 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:06.497 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:06.497 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:06.497 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272741 s, 38.4 MB/s 2024-04-16T16:22:06.499 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:06.499 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:06.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:06.557 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.2 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:06.650 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:06.650 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:06.650 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355437 s, 29.5 MB/s 2024-04-16T16:22:06.652 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:06.652 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:06.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:06.707 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.3 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265489 s, 39.5 MB/s 2024-04-16T16:22:06.790 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:06.790 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:06.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:06.850 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.4 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:06.932 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:06.932 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:06.932 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256633 s, 40.9 MB/s 2024-04-16T16:22:06.934 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:06.934 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:06.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:06.993 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.5 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:07.079 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:07.079 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:07.079 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293335 s, 35.7 MB/s 2024-04-16T16:22:07.080 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:07.080 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:07.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:07.135 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.6 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:07.217 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:07.217 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:07.217 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293611 s, 35.7 MB/s 2024-04-16T16:22:07.219 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:07.219 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:07.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:07.279 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.7 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:07.362 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:07.362 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:07.362 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243739 s, 43.0 MB/s 2024-04-16T16:22:07.367 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:07.367 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:07.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:07.427 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.8 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:07.515 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:07.515 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:07.516 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270664 s, 38.7 MB/s 2024-04-16T16:22:07.517 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:07.517 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:07.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:07.572 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.9 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:07.656 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:07.656 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:07.656 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029057 s, 36.1 MB/s 2024-04-16T16:22:07.658 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:07.658 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:07.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:07.717 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.10 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:07.797 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:07.797 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:07.797 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226368 s, 46.3 MB/s 2024-04-16T16:22:07.799 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:07.799 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:07.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:07.857 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.11 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:07.939 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:07.939 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:07.939 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260305 s, 40.3 MB/s 2024-04-16T16:22:07.941 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:07.941 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:08.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:08.001 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.12 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:08.087 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:08.087 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:08.088 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316757 s, 33.1 MB/s 2024-04-16T16:22:08.089 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:08.090 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:08.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:08.150 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.13 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:08.232 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:08.232 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:08.232 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246957 s, 42.5 MB/s 2024-04-16T16:22:08.234 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:08.234 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:08.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:08.288 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.14 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:08.366 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:08.366 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:08.366 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237264 s, 44.2 MB/s 2024-04-16T16:22:08.367 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:08.368 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:08.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:08.424 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_0/test.15 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:08.508 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:08.508 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:08.508 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261139 s, 40.2 MB/s 2024-04-16T16:22:08.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:08.510 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir test_mds_metrics_1) 2024-04-16T16:22:08.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:08.578 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 test_mds_metrics_1) 2024-04-16T16:22:08.659 INFO:tasks.cephfs.test_mds_metrics:pinning directory test_mds_metrics_1 to rank 1 2024-04-16T16:22:08.660 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:08.660 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:08.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:08.718 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.0 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:08.796 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:08.796 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:08.796 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208261 s, 50.3 MB/s 2024-04-16T16:22:08.798 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:08.798 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:08.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:08.857 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.1 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:08.942 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:08.942 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:08.942 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255008 s, 41.1 MB/s 2024-04-16T16:22:08.944 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:08.945 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:09.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:09.007 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.2 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:09.091 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:09.091 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:09.091 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256611 s, 40.9 MB/s 2024-04-16T16:22:09.093 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:09.093 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:09.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:09.150 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.3 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:09.234 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:09.234 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:09.234 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281619 s, 37.2 MB/s 2024-04-16T16:22:09.236 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:09.236 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:09.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:09.293 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.4 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:09.371 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:09.371 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:09.371 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022769 s, 46.1 MB/s 2024-04-16T16:22:09.373 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:09.373 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:09.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:09.426 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.5 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:09.510 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:09.510 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:09.510 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292844 s, 35.8 MB/s 2024-04-16T16:22:09.512 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:09.512 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:09.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:09.567 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.6 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:09.645 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:09.645 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:09.645 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214885 s, 48.8 MB/s 2024-04-16T16:22:09.646 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:09.646 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:09.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:09.698 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.7 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248762 s, 42.2 MB/s 2024-04-16T16:22:09.780 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:09.780 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:09.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:09.839 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.8 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:09.921 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:09.921 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:09.921 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290556 s, 36.1 MB/s 2024-04-16T16:22:09.923 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:09.923 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:09.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:09.982 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.9 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:10.072 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:10.072 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:10.072 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323919 s, 32.4 MB/s 2024-04-16T16:22:10.074 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:10.074 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:10.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:10.136 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.10 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:10.217 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:10.217 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:10.217 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232363 s, 45.1 MB/s 2024-04-16T16:22:10.219 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:10.219 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:10.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:10.276 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.11 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:10.360 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:10.360 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:10.361 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249263 s, 42.1 MB/s 2024-04-16T16:22:10.362 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:10.363 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:10.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:10.421 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.12 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:10.503 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:10.503 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:10.503 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242643 s, 43.2 MB/s 2024-04-16T16:22:10.505 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:10.505 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:10.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:10.563 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.13 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:10.640 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:10.640 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:10.640 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230221 s, 45.5 MB/s 2024-04-16T16:22:10.641 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:10.641 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:10.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:10.698 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.14 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:10.779 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:10.779 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:10.779 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261221 s, 40.1 MB/s 2024-04-16T16:22:10.781 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:10.781 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:10.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:10.838 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=test_mds_metrics_1/test.15 bs=1M conv=fdatasync count=1 seek=0) 2024-04-16T16:22:10.917 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T16:22:10.918 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T16:22:10.918 INFO:teuthology.orchestra.run.smithi045.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228528 s, 45.9 MB/s 2024-04-16T16:22:10.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:10.919 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec find .) 2024-04-16T16:22:10.996 INFO:teuthology.orchestra.run.smithi103.stdout:. 2024-04-16T16:22:10.996 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0 2024-04-16T16:22:10.996 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.5 2024-04-16T16:22:10.996 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.7 2024-04-16T16:22:10.996 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.6 2024-04-16T16:22:10.996 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.10 2024-04-16T16:22:10.996 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.4 2024-04-16T16:22:10.997 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.12 2024-04-16T16:22:10.997 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.1 2024-04-16T16:22:10.997 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.9 2024-04-16T16:22:10.997 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.13 2024-04-16T16:22:10.997 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.0 2024-04-16T16:22:10.997 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.11 2024-04-16T16:22:10.997 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.8 2024-04-16T16:22:10.997 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.3 2024-04-16T16:22:10.997 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.2 2024-04-16T16:22:10.997 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.14 2024-04-16T16:22:10.997 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_0/test.15 2024-04-16T16:22:10.997 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1 2024-04-16T16:22:10.997 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.5 2024-04-16T16:22:10.997 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.7 2024-04-16T16:22:10.998 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.6 2024-04-16T16:22:10.998 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.10 2024-04-16T16:22:10.998 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.4 2024-04-16T16:22:10.998 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.12 2024-04-16T16:22:10.998 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.1 2024-04-16T16:22:10.998 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.9 2024-04-16T16:22:10.998 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.13 2024-04-16T16:22:10.998 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.0 2024-04-16T16:22:10.998 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.11 2024-04-16T16:22:10.998 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.8 2024-04-16T16:22:10.998 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.3 2024-04-16T16:22:10.998 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.2 2024-04-16T16:22:10.999 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.14 2024-04-16T16:22:10.999 INFO:teuthology.orchestra.run.smithi103.stdout:./test_mds_metrics_1/test.15 2024-04-16T16:22:16.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:22:16.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.238+0000 7f181d800640 1 Processor -- start 2024-04-16T16:22:16.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.238+0000 7f181d800640 1 -- start start 2024-04-16T16:22:16.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.240+0000 7f181d800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f181806f730 0x7f18180c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:16.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.240+0000 7f181d800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f18180c6320 0x7f181806df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:16.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.240+0000 7f181d800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f181815d600 con 0x7f18180c6fc0 2024-04-16T16:22:16.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.240+0000 7f181d800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f18180c5670 con 0x7f18180c6320 2024-04-16T16:22:16.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.240+0000 7f181d800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f181815d8e0 con 0x7f181806f730 2024-04-16T16:22:16.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.240+0000 7f1817e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f18180c6fc0 0x7f181815cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45542/0 (socket says 172.21.15.45:45542) 2024-04-16T16:22:16.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.240+0000 7f1817e00640 1 -- 172.21.15.45:0/449519397 learned_addr learned my addr 172.21.15.45:0/449519397 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:16.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.241+0000 7f181ce00640 1 --2- 172.21.15.45:0/449519397 >> v2:172.21.15.45:3300/0 conn(0x7f18180c6320 0x7f181806df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:16.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.241+0000 7f1817400640 1 --2- 172.21.15.45:0/449519397 >> v2:172.21.15.103:3300/0 conn(0x7f181806f730 0x7f18180c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:16.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.241+0000 7f1816a00640 1 -- 172.21.15.45:0/449519397 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3733329052 0 0) 0x7f181815d600 con 0x7f18180c6fc0 2024-04-16T16:22:16.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.241+0000 7f1816a00640 1 -- 172.21.15.45:0/449519397 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17fc003520 con 0x7f18180c6fc0 2024-04-16T16:22:16.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.241+0000 7f181ce00640 1 -- 172.21.15.45:0/449519397 >> v2:172.21.15.103:3300/0 conn(0x7f181806f730 msgr2=0x7f18180c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:16.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.241+0000 7f181ce00640 1 --2- 172.21.15.45:0/449519397 >> v2:172.21.15.103:3300/0 conn(0x7f181806f730 0x7f18180c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:16.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.241+0000 7f181ce00640 1 -- 172.21.15.45:0/449519397 >> v1:172.21.15.45:6789/0 conn(0x7f18180c6fc0 legacy=0x7f181815cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:16.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.241+0000 7f181ce00640 1 -- 172.21.15.45:0/449519397 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f181815dbc0 con 0x7f18180c6320 2024-04-16T16:22:16.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.242+0000 7f181ce00640 1 --2- 172.21.15.45:0/449519397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18180c6320 0x7f181806df40 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f1818073370 tx=0x7f181002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ac07e5dfc86833 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:16.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.243+0000 7f1816a00640 1 -- 172.21.15.45:0/449519397 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1810039070 con 0x7f18180c6320 2024-04-16T16:22:16.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.243+0000 7f1816a00640 1 -- 172.21.15.45:0/449519397 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f181002cbf0 con 0x7f18180c6320 2024-04-16T16:22:16.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.243+0000 7f1816a00640 1 -- 172.21.15.45:0/449519397 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1810034440 con 0x7f18180c6320 2024-04-16T16:22:16.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.243+0000 7f181d800640 1 -- 172.21.15.45:0/449519397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18180c6320 msgr2=0x7f181806df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:16.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.243+0000 7f181d800640 1 --2- 172.21.15.45:0/449519397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18180c6320 0x7f181806df40 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f1818073370 tx=0x7f181002be40 comp rx=0 tx=0).stop 2024-04-16T16:22:16.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.243+0000 7f181d800640 1 -- 172.21.15.45:0/449519397 shutdown_connections 2024-04-16T16:22:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.243+0000 7f181d800640 1 --2- 172.21.15.45:0/449519397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18180c6320 0x7f181806df40 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.243+0000 7f181d800640 1 --2- 172.21.15.45:0/449519397 >> v2:172.21.15.103:3300/0 conn(0x7f181806f730 0x7f18180c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.243+0000 7f181d800640 1 -- 172.21.15.45:0/449519397 >> 172.21.15.45:0/449519397 conn(0x7f181806b930 msgr2=0x7f181806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.243+0000 7f181d800640 1 -- 172.21.15.45:0/449519397 shutdown_connections 2024-04-16T16:22:16.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.243+0000 7f181d800640 1 -- 172.21.15.45:0/449519397 wait complete. 2024-04-16T16:22:16.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.245+0000 7f181d800640 1 Processor -- start 2024-04-16T16:22:16.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.245+0000 7f181d800640 1 -- start start 2024-04-16T16:22:16.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.246+0000 7f181d800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f181806f730 0x7f18180cef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:16.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.246+0000 7f181d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18180c6320 0x7f18181546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:16.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.246+0000 7f181d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18180c6fc0 0x7f18181587c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:16.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.246+0000 7f181d800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f18180c7520 con 0x7f18180c6fc0 2024-04-16T16:22:16.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.246+0000 7f181d800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18180c73a0 con 0x7f181806f730 2024-04-16T16:22:16.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.246+0000 7f181d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f181806e480 con 0x7f18180c6320 2024-04-16T16:22:16.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.247+0000 7f181ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18180c6fc0 0x7f18181587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:16.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.247+0000 7f181ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18180c6fc0 0x7f18181587c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39720/0 (socket says 172.21.15.45:39720) 2024-04-16T16:22:16.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.247+0000 7f181ce00640 1 -- 172.21.15.45:0/2105034052 learned_addr learned my addr 172.21.15.45:0/2105034052 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:16.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.247+0000 7f1817e00640 1 --2- 172.21.15.45:0/2105034052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f181806f730 0x7f18180cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:16.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.247+0000 7f1817400640 1 --2- 172.21.15.45:0/2105034052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18180c6320 0x7f18181546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:16.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.247+0000 7f181ce00640 1 -- 172.21.15.45:0/2105034052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18180c6320 msgr2=0x7f18181546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:16.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.247+0000 7f181ce00640 1 --2- 172.21.15.45:0/2105034052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18180c6320 0x7f18181546c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:16.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.247+0000 7f181ce00640 1 -- 172.21.15.45:0/2105034052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f181806f730 msgr2=0x7f18180cef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:16.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.247+0000 7f181ce00640 1 --2- 172.21.15.45:0/2105034052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f181806f730 0x7f18180cef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:16.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.247+0000 7f181ce00640 1 -- 172.21.15.45:0/2105034052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18180cf580 con 0x7f18180c6fc0 2024-04-16T16:22:16.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.248+0000 7f181ce00640 1 --2- 172.21.15.45:0/2105034052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18180c6fc0 0x7f18181587c0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f1810009e80 tx=0x7f1810004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:16.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.248+0000 7f180fe00640 1 -- 172.21.15.45:0/2105034052 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1810039050 con 0x7f18180c6fc0 2024-04-16T16:22:16.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.248+0000 7f180fe00640 1 -- 172.21.15.45:0/2105034052 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1810045070 con 0x7f18180c6fc0 2024-04-16T16:22:16.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.248+0000 7f180fe00640 1 -- 172.21.15.45:0/2105034052 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181003e830 con 0x7f18180c6fc0 2024-04-16T16:22:16.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.248+0000 7f181d800640 1 -- 172.21.15.45:0/2105034052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18180cf860 con 0x7f18180c6fc0 2024-04-16T16:22:16.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.248+0000 7f181d800640 1 -- 172.21.15.45:0/2105034052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18180cfe20 con 0x7f18180c6fc0 2024-04-16T16:22:16.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.249+0000 7f181d800640 1 -- 172.21.15.45:0/2105034052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1818075780 con 0x7f18180c6fc0 2024-04-16T16:22:16.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.250+0000 7f180fe00640 1 -- 172.21.15.45:0/2105034052 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7f181003e9d0 con 0x7f18180c6fc0 2024-04-16T16:22:16.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.251+0000 7f180fe00640 1 --2- 172.21.15.45:0/2105034052 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f17f806ca90 0x7f17f806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:16.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.251+0000 7f180fe00640 1 -- 172.21.15.45:0/2105034052 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f18100aed20 con 0x7f18180c6fc0 2024-04-16T16:22:16.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.252+0000 7f1817e00640 1 --2- 172.21.15.45:0/2105034052 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f17f806ca90 0x7f17f806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:16.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.253+0000 7f1817e00640 1 --2- 172.21.15.45:0/2105034052 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f17f806ca90 0x7f17f806ef50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18181f7760 tx=0x7f1808009040 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:16.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.257+0000 7f180fe00640 1 -- 172.21.15.45:0/2105034052 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f181007cff0 con 0x7f18180c6fc0 2024-04-16T16:22:16.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.488+0000 7f181d800640 1 -- 172.21.15.45:0/2105034052 --> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f18180b2fc0 con 0x7f17f806ca90 2024-04-16T16:22:16.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.492+0000 7f180fe00640 1 -- 172.21.15.45:0/2105034052 <== mgr.6674 v2:172.21.15.103:6832/1506110537 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3003 (secure 0 0 0) 0x7f18180b2fc0 con 0x7f17f806ca90 2024-04-16T16:22:16.491 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.6659": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "192.168.0.1"}, "client.6710": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.7498": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.9348": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "192.168.0.1"}}}, "global_metrics": {"cephfs": {"client.6659": [[52, 2], [0, 0], [0, 0], [0, 26614178], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 2957132], [102926059387199, 9]], "client.6710": [[2, 0], [0, 0], [0, 0], [0, 980564], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 980564], [0, 1]], "client.7498": [[2, 0], [0, 0], [0, 0], [0, 592826], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 592826], [0, 1]], "client.9348": [[419, 22], [0, 0], [0, 507173266], [0, 150907705], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 15849167], [271636030061266, 32], [0, 2357924], [388853860915318, 64]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.6659": [], "client.6710": [], "client.7498": [], "client.9348": []}}} 2024-04-16T16:22:16.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.494+0000 7f181d800640 1 -- 172.21.15.45:0/2105034052 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f17f806ca90 msgr2=0x7f17f806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:16.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.494+0000 7f181d800640 1 --2- 172.21.15.45:0/2105034052 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f17f806ca90 0x7f17f806ef50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18181f7760 tx=0x7f1808009040 comp rx=0 tx=0).stop 2024-04-16T16:22:16.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.494+0000 7f181d800640 1 -- 172.21.15.45:0/2105034052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18180c6fc0 msgr2=0x7f18181587c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:16.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.494+0000 7f181d800640 1 --2- 172.21.15.45:0/2105034052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18180c6fc0 0x7f18181587c0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f1810009e80 tx=0x7f1810004a40 comp rx=0 tx=0).stop 2024-04-16T16:22:16.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.494+0000 7f181d800640 1 -- 172.21.15.45:0/2105034052 shutdown_connections 2024-04-16T16:22:16.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.494+0000 7f181d800640 1 --2- 172.21.15.45:0/2105034052 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f17f806ca90 0x7f17f806ef50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:16.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.494+0000 7f181d800640 1 --2- 172.21.15.45:0/2105034052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18180c6fc0 0x7f18181587c0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:16.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.494+0000 7f181d800640 1 --2- 172.21.15.45:0/2105034052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18180c6320 0x7f18181546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:16.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.494+0000 7f181d800640 1 --2- 172.21.15.45:0/2105034052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f181806f730 0x7f18180cef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:16.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.494+0000 7f181d800640 1 -- 172.21.15.45:0/2105034052 >> 172.21.15.45:0/2105034052 conn(0x7f181806b930 msgr2=0x7f18181563d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:16.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.495+0000 7f181d800640 1 -- 172.21.15.45:0/2105034052 shutdown_connections 2024-04-16T16:22:16.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:16.495+0000 7f181d800640 1 -- 172.21.15.45:0/2105034052 wait complete. 2024-04-16T16:22:17.513 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:22:17.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.749+0000 7f9b73e00640 1 Processor -- start 2024-04-16T16:22:17.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.750+0000 7f9b73e00640 1 -- start start 2024-04-16T16:22:17.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.751+0000 7f9b73e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9b7406f730 0x7f9b740c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:17.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.751+0000 7f9b73e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9b740c6320 0x7f9b7406df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:17.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.751+0000 7f9b73e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b7415d600 con 0x7f9b740c6fc0 2024-04-16T16:22:17.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.751+0000 7f9b73e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9b740c5670 con 0x7f9b740c6320 2024-04-16T16:22:17.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.751+0000 7f9b73e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9b7415d8e0 con 0x7f9b7406f730 2024-04-16T16:22:17.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.752+0000 7f9b73400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9b740c6320 0x7f9b7406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:17.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.752+0000 7f9b73400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9b740c6320 0x7f9b7406df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39730/0 (socket says 172.21.15.45:39730) 2024-04-16T16:22:17.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.752+0000 7f9b73400640 1 -- 172.21.15.45:0/664778033 learned_addr learned my addr 172.21.15.45:0/664778033 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:17.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.752+0000 7f9b73400640 1 -- 172.21.15.45:0/664778033 >> v2:172.21.15.103:3300/0 conn(0x7f9b7406f730 msgr2=0x7f9b740c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:17.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.752+0000 7f9b73400640 1 --2- 172.21.15.45:0/664778033 >> v2:172.21.15.103:3300/0 conn(0x7f9b7406f730 0x7f9b740c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:17.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.752+0000 7f9b73400640 1 -- 172.21.15.45:0/664778033 >> v1:172.21.15.103:6789/0 conn(0x7f9b740c6fc0 legacy=0x7f9b7415cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:17.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.752+0000 7f9b73400640 1 -- 172.21.15.45:0/664778033 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b7415dbc0 con 0x7f9b740c6320 2024-04-16T16:22:17.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.753+0000 7f9b73400640 1 --2- 172.21.15.45:0/664778033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b740c6320 0x7f9b7406df40 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f9b74073370 tx=0x7f9b6c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1ea61d6fe0e15d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:17.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.754+0000 7f9b72000640 1 -- 172.21.15.45:0/664778033 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c02e070 con 0x7f9b740c6320 2024-04-16T16:22:17.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.754+0000 7f9b72000640 1 -- 172.21.15.45:0/664778033 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b6c037070 con 0x7f9b740c6320 2024-04-16T16:22:17.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.754+0000 7f9b72000640 1 -- 172.21.15.45:0/664778033 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c031a00 con 0x7f9b740c6320 2024-04-16T16:22:17.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.754+0000 7f9b73e00640 1 -- 172.21.15.45:0/664778033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b740c6320 msgr2=0x7f9b7406df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:17.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.754+0000 7f9b73e00640 1 --2- 172.21.15.45:0/664778033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b740c6320 0x7f9b7406df40 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f9b74073370 tx=0x7f9b6c02b880 comp rx=0 tx=0).stop 2024-04-16T16:22:17.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.755+0000 7f9b73e00640 1 -- 172.21.15.45:0/664778033 shutdown_connections 2024-04-16T16:22:17.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.755+0000 7f9b73e00640 1 --2- 172.21.15.45:0/664778033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b740c6320 0x7f9b7406df40 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:17.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.755+0000 7f9b73e00640 1 --2- 172.21.15.45:0/664778033 >> v2:172.21.15.103:3300/0 conn(0x7f9b7406f730 0x7f9b740c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:17.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.755+0000 7f9b73e00640 1 -- 172.21.15.45:0/664778033 >> 172.21.15.45:0/664778033 conn(0x7f9b7406b930 msgr2=0x7f9b7406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:17.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.755+0000 7f9b73e00640 1 -- 172.21.15.45:0/664778033 shutdown_connections 2024-04-16T16:22:17.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.755+0000 7f9b73e00640 1 -- 172.21.15.45:0/664778033 wait complete. 2024-04-16T16:22:17.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.756+0000 7f9b73e00640 1 Processor -- start 2024-04-16T16:22:17.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.756+0000 7f9b73e00640 1 -- start start 2024-04-16T16:22:17.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.757+0000 7f9b73e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b7406f730 0x7f9b7415bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:17.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.757+0000 7f9b73e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b740c6320 0x7f9b741556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:17.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.757+0000 7f9b73e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b740c6fc0 0x7f9b74155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:17.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.757+0000 7f9b73e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b740c7520 con 0x7f9b740c6320 2024-04-16T16:22:17.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.757+0000 7f9b73e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9b740c73a0 con 0x7f9b740c6fc0 2024-04-16T16:22:17.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.757+0000 7f9b73e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9b7406e480 con 0x7f9b7406f730 2024-04-16T16:22:17.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.758+0000 7f9b6a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b740c6320 0x7f9b741556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:17.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.758+0000 7f9b6a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b740c6320 0x7f9b741556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39742/0 (socket says 172.21.15.45:39742) 2024-04-16T16:22:17.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.758+0000 7f9b6a000640 1 -- 172.21.15.45:0/123566446 learned_addr learned my addr 172.21.15.45:0/123566446 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:17.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.758+0000 7f9b72a00640 1 --2- 172.21.15.45:0/123566446 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b7406f730 0x7f9b7415bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:17.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.758+0000 7f9b6a000640 1 -- 172.21.15.45:0/123566446 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b7406f730 msgr2=0x7f9b7415bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:17.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.758+0000 7f9b6a000640 1 --2- 172.21.15.45:0/123566446 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b7406f730 0x7f9b7415bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:17.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.758+0000 7f9b6a000640 1 -- 172.21.15.45:0/123566446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b740c6fc0 msgr2=0x7f9b74155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:17.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.758+0000 7f9b73400640 1 --2- 172.21.15.45:0/123566446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b740c6fc0 0x7f9b74155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:17.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.758+0000 7f9b6a000640 1 --2- 172.21.15.45:0/123566446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b740c6fc0 0x7f9b74155c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:17.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.758+0000 7f9b6a000640 1 -- 172.21.15.45:0/123566446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b741f2e50 con 0x7f9b740c6320 2024-04-16T16:22:17.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.759+0000 7f9b6a000640 1 --2- 172.21.15.45:0/123566446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b740c6320 0x7f9b741556e0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c00b4a0 tx=0x7f9b5c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:17.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.759+0000 7f9b6b400640 1 -- 172.21.15.45:0/123566446 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c014070 con 0x7f9b740c6320 2024-04-16T16:22:17.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.759+0000 7f9b6b400640 1 -- 172.21.15.45:0/123566446 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b5c010610 con 0x7f9b740c6320 2024-04-16T16:22:17.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.759+0000 7f9b6b400640 1 -- 172.21.15.45:0/123566446 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c010a00 con 0x7f9b740c6320 2024-04-16T16:22:17.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.759+0000 7f9b73e00640 1 -- 172.21.15.45:0/123566446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b741f2090 con 0x7f9b740c6320 2024-04-16T16:22:17.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.759+0000 7f9b73e00640 1 -- 172.21.15.45:0/123566446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b741f2650 con 0x7f9b740c6320 2024-04-16T16:22:17.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.761+0000 7f9b73e00640 1 -- 172.21.15.45:0/123566446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b74075780 con 0x7f9b740c6320 2024-04-16T16:22:17.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.761+0000 7f9b6b400640 1 -- 172.21.15.45:0/123566446 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7f9b5c017070 con 0x7f9b740c6320 2024-04-16T16:22:17.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.762+0000 7f9b6b400640 1 --2- 172.21.15.45:0/123566446 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f9b5006ca60 0x7f9b5006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:17.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.762+0000 7f9b6b400640 1 -- 172.21.15.45:0/123566446 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f9b5c08b940 con 0x7f9b740c6320 2024-04-16T16:22:17.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.763+0000 7f9b72a00640 1 --2- 172.21.15.45:0/123566446 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f9b5006ca60 0x7f9b5006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:17.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.764+0000 7f9b72a00640 1 --2- 172.21.15.45:0/123566446 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f9b5006ca60 0x7f9b5006ef20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9b64009f60 tx=0x7f9b640094f0 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:17.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.767+0000 7f9b6b400640 1 -- 172.21.15.45:0/123566446 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9b5c059af0 con 0x7f9b740c6320 2024-04-16T16:22:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:17.998+0000 7f9b73e00640 1 -- 172.21.15.45:0/123566446 --> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f9b7406cf70 con 0x7f9b5006ca60 2024-04-16T16:22:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:18.002+0000 7f9b6b400640 1 -- 172.21.15.45:0/123566446 <== mgr.6674 v2:172.21.15.103:6832/1506110537 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3003 (secure 0 0 0) 0x7f9b7406cf70 con 0x7f9b5006ca60 2024-04-16T16:22:18.001 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.6659": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "192.168.0.1"}, "client.6710": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.7498": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.9348": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "192.168.0.1"}}}, "global_metrics": {"cephfs": {"client.6659": [[52, 2], [0, 0], [0, 0], [0, 26614178], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 2957132], [102926059387199, 9]], "client.6710": [[2, 0], [0, 0], [0, 0], [0, 980564], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 980564], [0, 1]], "client.7498": [[2, 0], [0, 0], [0, 0], [0, 592826], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 592826], [0, 1]], "client.9348": [[419, 22], [0, 0], [0, 507173266], [0, 150907705], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 15849167], [271636030061266, 32], [0, 2357924], [388853860915318, 64]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.6659": [], "client.6710": [], "client.7498": [], "client.9348": []}}} 2024-04-16T16:22:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:18.004+0000 7f9b73e00640 1 -- 172.21.15.45:0/123566446 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f9b5006ca60 msgr2=0x7f9b5006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:18.004+0000 7f9b73e00640 1 --2- 172.21.15.45:0/123566446 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f9b5006ca60 0x7f9b5006ef20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9b64009f60 tx=0x7f9b640094f0 comp rx=0 tx=0).stop 2024-04-16T16:22:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:18.004+0000 7f9b73e00640 1 -- 172.21.15.45:0/123566446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b740c6320 msgr2=0x7f9b741556e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:18.004+0000 7f9b73e00640 1 --2- 172.21.15.45:0/123566446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b740c6320 0x7f9b741556e0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c00b4a0 tx=0x7f9b5c00b960 comp rx=0 tx=0).stop 2024-04-16T16:22:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:18.004+0000 7f9b73e00640 1 -- 172.21.15.45:0/123566446 shutdown_connections 2024-04-16T16:22:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:18.004+0000 7f9b73e00640 1 --2- 172.21.15.45:0/123566446 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f9b5006ca60 0x7f9b5006ef20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:18.004+0000 7f9b73e00640 1 --2- 172.21.15.45:0/123566446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b740c6fc0 0x7f9b74155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:18.004+0000 7f9b73e00640 1 --2- 172.21.15.45:0/123566446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b740c6320 0x7f9b741556e0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:18.004+0000 7f9b73e00640 1 --2- 172.21.15.45:0/123566446 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b7406f730 0x7f9b7415bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:18.004+0000 7f9b73e00640 1 -- 172.21.15.45:0/123566446 >> 172.21.15.45:0/123566446 conn(0x7f9b7406b930 msgr2=0x7f9b74154e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:18.004+0000 7f9b73e00640 1 -- 172.21.15.45:0/123566446 shutdown_connections 2024-04-16T16:22:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:18.005+0000 7f9b73e00640 1 -- 172.21.15.45:0/123566446 wait complete. 2024-04-16T16:22:19.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs perf stats 2024-04-16T16:22:19.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.259+0000 7f25d9200640 1 Processor -- start 2024-04-16T16:22:19.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.260+0000 7f25d9200640 1 -- start start 2024-04-16T16:22:19.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.261+0000 7f25d9200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25d40c6fc0 0x7f25d406f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:19.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.261+0000 7f25d9200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f25d406df40 0x7f25d406e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:19.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.262+0000 7f25d9200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25d415d600 con 0x7f25d415cdc0 2024-04-16T16:22:19.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.262+0000 7f25d9200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f25d40c5670 con 0x7f25d406df40 2024-04-16T16:22:19.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.262+0000 7f25d9200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f25d415d8e0 con 0x7f25d40c6fc0 2024-04-16T16:22:19.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.262+0000 7f25d2a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f25d415cdc0 0x7f25d40c5de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45552/0 (socket says 172.21.15.45:45552) 2024-04-16T16:22:19.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.262+0000 7f25d2a00640 1 -- 172.21.15.45:0/1892309015 learned_addr learned my addr 172.21.15.45:0/1892309015 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:19.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.262+0000 7f25d3e00640 1 --2- 172.21.15.45:0/1892309015 >> v2:172.21.15.103:3300/0 conn(0x7f25d406df40 0x7f25d406e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:19.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.262+0000 7f25d3400640 1 --2- 172.21.15.45:0/1892309015 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25d40c6fc0 0x7f25d406f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:19.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.262+0000 7f25d2000640 1 -- 172.21.15.45:0/1892309015 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2718350993 0 0) 0x7f25d415d600 con 0x7f25d415cdc0 2024-04-16T16:22:19.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d2000640 1 -- 172.21.15.45:0/1892309015 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25b8003520 con 0x7f25d415cdc0 2024-04-16T16:22:19.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d2000640 1 -- 172.21.15.45:0/1892309015 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 401884930 0 0) 0x7f25b8003520 con 0x7f25d415cdc0 2024-04-16T16:22:19.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d2000640 1 -- 172.21.15.45:0/1892309015 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25d40c6fc0 msgr2=0x7f25d406f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:19.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d2000640 1 --2- 172.21.15.45:0/1892309015 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25d40c6fc0 0x7f25d406f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:19.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d2000640 1 -- 172.21.15.45:0/1892309015 >> v2:172.21.15.103:3300/0 conn(0x7f25d406df40 msgr2=0x7f25d406e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:19.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d2000640 1 --2- 172.21.15.45:0/1892309015 >> v2:172.21.15.103:3300/0 conn(0x7f25d406df40 0x7f25d406e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:19.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d2000640 1 -- 172.21.15.45:0/1892309015 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d415dbc0 con 0x7f25d415cdc0 2024-04-16T16:22:19.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d2000640 1 -- 172.21.15.45:0/1892309015 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f25c0001670 con 0x7f25d415cdc0 2024-04-16T16:22:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d3e00640 1 --2- 172.21.15.45:0/1892309015 >> v2:172.21.15.103:3300/0 conn(0x7f25d406df40 0x7f25d406e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:22:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d2000640 1 -- 172.21.15.45:0/1892309015 >> v1:172.21.15.45:6789/0 conn(0x7f25d415cdc0 legacy=0x7f25d40c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d2000640 1 --2- 172.21.15.45:0/1892309015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25b80039b0 0x7f25b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d2000640 1 --2- 172.21.15.45:0/1892309015 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25d40c6fc0 0x7f25b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d2000640 1 --2- 172.21.15.45:0/1892309015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b8004d40 0x7f25b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d2000640 1 -- 172.21.15.45:0/1892309015 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f25d40c5670 con 0x7f25b80039b0 2024-04-16T16:22:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d2000640 1 -- 172.21.15.45:0/1892309015 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f25d415d8e0 con 0x7f25b8004d40 2024-04-16T16:22:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.263+0000 7f25d2000640 1 -- 172.21.15.45:0/1892309015 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f25b800b9c0 con 0x7f25d40c6fc0 2024-04-16T16:22:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.264+0000 7f25d3400640 1 --2- 172.21.15.45:0/1892309015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25b80039b0 0x7f25b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.264+0000 7f25d2a00640 1 --2- 172.21.15.45:0/1892309015 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25d40c6fc0 0x7f25b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.264+0000 7f25d3400640 1 -- 172.21.15.45:0/1892309015 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25d40c6fc0 msgr2=0x7f25b8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.264+0000 7f25d3400640 1 --2- 172.21.15.45:0/1892309015 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25d40c6fc0 0x7f25b8004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.264+0000 7f25d3400640 1 -- 172.21.15.45:0/1892309015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b8004d40 msgr2=0x7f25b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.264+0000 7f25d3400640 1 --2- 172.21.15.45:0/1892309015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b8004d40 0x7f25b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.264+0000 7f25d3400640 1 -- 172.21.15.45:0/1892309015 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b8003520 con 0x7f25b80039b0 2024-04-16T16:22:19.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.265+0000 7f25d3400640 1 --2- 172.21.15.45:0/1892309015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25b80039b0 0x7f25b8003dd0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f25c4009ac0 tx=0x7f25b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3ff4b4d8b6b22f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:19.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.265+0000 7f25d2000640 1 -- 172.21.15.45:0/1892309015 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c4039070 con 0x7f25b80039b0 2024-04-16T16:22:19.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.266+0000 7f25d2000640 1 -- 172.21.15.45:0/1892309015 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25c402c6b0 con 0x7f25b80039b0 2024-04-16T16:22:19.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.266+0000 7f25d2000640 1 -- 172.21.15.45:0/1892309015 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c4009690 con 0x7f25b80039b0 2024-04-16T16:22:19.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.266+0000 7f25d9200640 1 -- 172.21.15.45:0/1892309015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25b80039b0 msgr2=0x7f25b8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:19.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.266+0000 7f25d9200640 1 --2- 172.21.15.45:0/1892309015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25b80039b0 0x7f25b8003dd0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f25c4009ac0 tx=0x7f25b8005830 comp rx=0 tx=0).stop 2024-04-16T16:22:19.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.266+0000 7f25d3e00640 1 -- 172.21.15.45:0/1892309015 reap_dead start 2024-04-16T16:22:19.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.266+0000 7f25d9200640 1 -- 172.21.15.45:0/1892309015 shutdown_connections 2024-04-16T16:22:19.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.266+0000 7f25d9200640 1 -- 172.21.15.45:0/1892309015 >> 172.21.15.45:0/1892309015 conn(0x7f25d406b930 msgr2=0x7f25d406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:19.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.266+0000 7f25d9200640 1 -- 172.21.15.45:0/1892309015 shutdown_connections 2024-04-16T16:22:19.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.266+0000 7f25d9200640 1 -- 172.21.15.45:0/1892309015 wait complete. 2024-04-16T16:22:19.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.268+0000 7f25d9200640 1 Processor -- start 2024-04-16T16:22:19.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.268+0000 7f25d9200640 1 -- start start 2024-04-16T16:22:19.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.269+0000 7f25d9200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25b80039b0 0x7f25d4142e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:19.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.269+0000 7f25d9200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25d4151470 0x7f25d4153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:19.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.269+0000 7f25d9200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d41518f0 0x7f25d4154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:19.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.269+0000 7f25d9200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f25d40c6670 con 0x7f25d41518f0 2024-04-16T16:22:19.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.269+0000 7f25d9200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f25d40c64f0 con 0x7f25d4151470 2024-04-16T16:22:19.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.269+0000 7f25d9200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f25d415d240 con 0x7f25b80039b0 2024-04-16T16:22:19.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.270+0000 7f25d3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d41518f0 0x7f25d4154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:19.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.270+0000 7f25d3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d41518f0 0x7f25d4154230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39768/0 (socket says 172.21.15.45:39768) 2024-04-16T16:22:19.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.270+0000 7f25d3e00640 1 -- 172.21.15.45:0/2991802186 learned_addr learned my addr 172.21.15.45:0/2991802186 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:19.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.270+0000 7f25d2a00640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25d4151470 0x7f25d4153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:19.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.270+0000 7f25d3400640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25b80039b0 0x7f25d4142e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:19.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.270+0000 7f25d3e00640 1 -- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25b80039b0 msgr2=0x7f25d4142e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:19.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.270+0000 7f25d3e00640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25b80039b0 0x7f25d4142e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:19.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.270+0000 7f25d3e00640 1 -- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25d4151470 msgr2=0x7f25d4153cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:19.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.270+0000 7f25d3e00640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25d4151470 0x7f25d4153cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:19.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.270+0000 7f25d3e00640 1 -- 172.21.15.45:0/2991802186 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d414f900 con 0x7f25d41518f0 2024-04-16T16:22:19.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.270+0000 7f25d2a00640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25d4151470 0x7f25d4153cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:19.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.270+0000 7f25d3400640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25b80039b0 0x7f25d4142e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:19.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.271+0000 7f25d3e00640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d41518f0 0x7f25d4154230 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f25d40b5be0 tx=0x7f25cc00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:19.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.271+0000 7f25cb400640 1 -- 172.21.15.45:0/2991802186 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc01b070 con 0x7f25d41518f0 2024-04-16T16:22:19.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.271+0000 7f25cb400640 1 -- 172.21.15.45:0/2991802186 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25cc003990 con 0x7f25d41518f0 2024-04-16T16:22:19.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.271+0000 7f25cb400640 1 -- 172.21.15.45:0/2991802186 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc003d80 con 0x7f25d41518f0 2024-04-16T16:22:19.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.271+0000 7f25d9200640 1 -- 172.21.15.45:0/2991802186 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25d414fbe0 con 0x7f25d41518f0 2024-04-16T16:22:19.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.271+0000 7f25d9200640 1 -- 172.21.15.45:0/2991802186 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25d41501a0 con 0x7f25d41518f0 2024-04-16T16:22:19.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.272+0000 7f25d9200640 1 -- 172.21.15.45:0/2991802186 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25d4075780 con 0x7f25d41518f0 2024-04-16T16:22:19.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.273+0000 7f25cb400640 1 -- 172.21.15.45:0/2991802186 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7f25cc01e070 con 0x7f25d41518f0 2024-04-16T16:22:19.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.274+0000 7f25cb400640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f25b406ca60 0x7f25b406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:19.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.274+0000 7f25cb400640 1 -- 172.21.15.45:0/2991802186 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f25cc08bc50 con 0x7f25d41518f0 2024-04-16T16:22:19.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.275+0000 7f25d3400640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f25b406ca60 0x7f25b406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:19.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.276+0000 7f25d3400640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f25b406ca60 0x7f25b406ef20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f25c4036000 tx=0x7f25c4030000 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:19.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.279+0000 7f25cb400640 1 -- 172.21.15.45:0/2991802186 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f25cc059c60 con 0x7f25d41518f0 2024-04-16T16:22:19.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.515+0000 7f25d9200640 1 -- 172.21.15.45:0/2991802186 --> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] -- mgr_command(tid 0: {"prefix": "fs perf stats", "target": ["mon-mgr", ""]}) -- 0x7f25d4155ec0 con 0x7f25b406ca60 2024-04-16T16:22:19.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.520+0000 7f25cb400640 1 -- 172.21.15.45:0/2991802186 <== mgr.6674 v2:172.21.15.103:6832/1506110537 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3033 (secure 0 0 0) 0x7f25d4155ec0 con 0x7f25b406ca60 2024-04-16T16:22:19.519 INFO:teuthology.orchestra.run.smithi045.stdout:{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.6659": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "192.168.0.1"}, "client.6710": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.7498": {"hostname": "smithi103", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "IP": "172.21.15.103"}, "client.9348": {"hostname": "smithi045", "root": "/", "mount_point": "N/A", "valid_metrics": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "kernel_version": "6.9.0-rc3-g3f60ed14f7c1", "IP": "192.168.0.1"}}}, "global_metrics": {"cephfs": {"client.6659": [[52, 2], [0, 0], [0, 0], [0, 26614178], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 2957132], [102926059387199, 9]], "client.6710": [[2, 0], [0, 0], [0, 0], [0, 980564], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 980564], [0, 1]], "client.7498": [[2, 0], [0, 0], [0, 0], [0, 592826], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 592826], [0, 1]], "client.9348": [[419, 22], [0, 0], [0, 507173266], [0, 150907705], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 15849167], [271636030061266, 32], [0, 2357924], [388853860915318, 64]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.6659": [], "client.6710": [], "client.7498": [], "client.9348": []}, "mds.1": {"client.9348": []}}} 2024-04-16T16:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.522+0000 7f25d9200640 1 -- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f25b406ca60 msgr2=0x7f25b406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.522+0000 7f25d9200640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f25b406ca60 0x7f25b406ef20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f25c4036000 tx=0x7f25c4030000 comp rx=0 tx=0).stop 2024-04-16T16:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.522+0000 7f25d9200640 1 -- 172.21.15.45:0/2991802186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d41518f0 msgr2=0x7f25d4154230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.522+0000 7f25d9200640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d41518f0 0x7f25d4154230 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f25d40b5be0 tx=0x7f25cc00b480 comp rx=0 tx=0).stop 2024-04-16T16:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.522+0000 7f25d9200640 1 -- 172.21.15.45:0/2991802186 shutdown_connections 2024-04-16T16:22:19.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.522+0000 7f25d9200640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f25b406ca60 0x7f25b406ef20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:19.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.522+0000 7f25d9200640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d41518f0 0x7f25d4154230 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:19.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.522+0000 7f25d9200640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25d4151470 0x7f25d4153cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:19.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.522+0000 7f25d9200640 1 --2- 172.21.15.45:0/2991802186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25b80039b0 0x7f25d4142e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:19.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.522+0000 7f25d9200640 1 -- 172.21.15.45:0/2991802186 >> 172.21.15.45:0/2991802186 conn(0x7f25d406b930 msgr2=0x7f25d406e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:19.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.522+0000 7f25d9200640 1 -- 172.21.15.45:0/2991802186 shutdown_connections 2024-04-16T16:22:19.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.522+0000 7f25d9200640 1 -- 172.21.15.45:0/2991802186 wait complete. 2024-04-16T16:22:19.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:22:19.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.761+0000 7f14b1000640 1 Processor -- start 2024-04-16T16:22:19.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.762+0000 7f14b1000640 1 -- start start 2024-04-16T16:22:19.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.763+0000 7f14b1000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14ac1462e0 0x7f14ac072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:19.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.763+0000 7f14b1000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f14ac0728c0 0x7f14ac0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:19.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.763+0000 7f14b1000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14ac0cf6d0 con 0x7f14ac0cec00 2024-04-16T16:22:19.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.763+0000 7f14b1000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f14ac072d20 con 0x7f14ac0728c0 2024-04-16T16:22:19.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.763+0000 7f14b1000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f14ac1466c0 con 0x7f14ac1462e0 2024-04-16T16:22:19.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.764+0000 7f14aa800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f14ac0cec00 0x7f14ac15da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45566/0 (socket says 172.21.15.45:45566) 2024-04-16T16:22:19.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.764+0000 7f14aa800640 1 -- 172.21.15.45:0/3326369616 learned_addr learned my addr 172.21.15.45:0/3326369616 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:19.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.764+0000 7f14a9400640 1 --2- 172.21.15.45:0/3326369616 >> v2:172.21.15.45:3300/0 conn(0x7f14ac0728c0 0x7f14ac0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:19.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.764+0000 7f14a9e00640 1 --2- 172.21.15.45:0/3326369616 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14ac1462e0 0x7f14ac072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:19.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.764+0000 7f14a8a00640 1 -- 172.21.15.45:0/3326369616 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 704744664 0 0) 0x7f14ac0cf6d0 con 0x7f14ac0cec00 2024-04-16T16:22:19.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.764+0000 7f14a8a00640 1 -- 172.21.15.45:0/3326369616 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f148c003520 con 0x7f14ac0cec00 2024-04-16T16:22:19.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.765+0000 7f14a9400640 1 -- 172.21.15.45:0/3326369616 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14ac1462e0 msgr2=0x7f14ac072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:19.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.765+0000 7f14a9400640 1 --2- 172.21.15.45:0/3326369616 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14ac1462e0 0x7f14ac072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:19.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.765+0000 7f14a9400640 1 -- 172.21.15.45:0/3326369616 >> v1:172.21.15.45:6789/0 conn(0x7f14ac0cec00 legacy=0x7f14ac15da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:19.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.765+0000 7f14a9400640 1 -- 172.21.15.45:0/3326369616 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14ac0cf3f0 con 0x7f14ac0728c0 2024-04-16T16:22:19.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.766+0000 7f14a9400640 1 --2- 172.21.15.45:0/3326369616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac0728c0 0x7f14ac0ce6c0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f1494009d90 tx=0x7f149402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa21300da931881a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:19.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.766+0000 7f14a8a00640 1 -- 172.21.15.45:0/3326369616 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1494039070 con 0x7f14ac0728c0 2024-04-16T16:22:19.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.766+0000 7f14a8a00640 1 -- 172.21.15.45:0/3326369616 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f149402cbc0 con 0x7f14ac0728c0 2024-04-16T16:22:19.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.766+0000 7f14a8a00640 1 -- 172.21.15.45:0/3326369616 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1494034440 con 0x7f14ac0728c0 2024-04-16T16:22:19.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.767+0000 7f14b1000640 1 -- 172.21.15.45:0/3326369616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac0728c0 msgr2=0x7f14ac0ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:19.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.767+0000 7f14b1000640 1 --2- 172.21.15.45:0/3326369616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac0728c0 0x7f14ac0ce6c0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f1494009d90 tx=0x7f149402be10 comp rx=0 tx=0).stop 2024-04-16T16:22:19.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.767+0000 7f14b1000640 1 -- 172.21.15.45:0/3326369616 shutdown_connections 2024-04-16T16:22:19.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.767+0000 7f14b1000640 1 --2- 172.21.15.45:0/3326369616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac0728c0 0x7f14ac0ce6c0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:19.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.767+0000 7f14b1000640 1 --2- 172.21.15.45:0/3326369616 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14ac1462e0 0x7f14ac072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:19.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.767+0000 7f14b1000640 1 -- 172.21.15.45:0/3326369616 >> 172.21.15.45:0/3326369616 conn(0x7f14ac0b2750 msgr2=0x7f14ac0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:19.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.767+0000 7f14b1000640 1 -- 172.21.15.45:0/3326369616 shutdown_connections 2024-04-16T16:22:19.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.767+0000 7f14b1000640 1 -- 172.21.15.45:0/3326369616 wait complete. 2024-04-16T16:22:19.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.769+0000 7f14b1000640 1 Processor -- start 2024-04-16T16:22:19.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.769+0000 7f14b1000640 1 -- start start 2024-04-16T16:22:19.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.770+0000 7f14b1000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac0728c0 0x7f14ac0b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:19.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.770+0000 7f14b1000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14ac0cec00 0x7f14ac13fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:19.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.770+0000 7f14b1000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14ac1462e0 0x7f14ac143c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:19.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.770+0000 7f14b1000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f14ac15d540 con 0x7f14ac0728c0 2024-04-16T16:22:19.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.770+0000 7f14b1000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f14ac15d260 con 0x7f14ac1462e0 2024-04-16T16:22:19.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.770+0000 7f14b1000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f14ac0cf090 con 0x7f14ac0cec00 2024-04-16T16:22:19.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.770+0000 7f14a9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac0728c0 0x7f14ac0b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:19.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.770+0000 7f14a9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac0728c0 0x7f14ac0b56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39772/0 (socket says 172.21.15.45:39772) 2024-04-16T16:22:19.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.770+0000 7f14a9e00640 1 -- 172.21.15.45:0/1989081627 learned_addr learned my addr 172.21.15.45:0/1989081627 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:19.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.771+0000 7f14a9400640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14ac0cec00 0x7f14ac13fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:19.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.771+0000 7f14aa800640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14ac1462e0 0x7f14ac143c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:19.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.771+0000 7f14a9e00640 1 -- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14ac0cec00 msgr2=0x7f14ac13fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:19.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.771+0000 7f14a9e00640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14ac0cec00 0x7f14ac13fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:19.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.771+0000 7f14a9e00640 1 -- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14ac1462e0 msgr2=0x7f14ac143c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:19.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.771+0000 7f14a9e00640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14ac1462e0 0x7f14ac143c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:19.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.771+0000 7f14a9e00640 1 -- 172.21.15.45:0/1989081627 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14ac141af0 con 0x7f14ac0728c0 2024-04-16T16:22:19.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.771+0000 7f14aa800640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14ac1462e0 0x7f14ac143c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:19.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.771+0000 7f14a9400640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14ac0cec00 0x7f14ac13fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:19.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.771+0000 7f14a9e00640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac0728c0 0x7f14ac0b56f0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f149c00e9b0 tx=0x7f149c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:19.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.771+0000 7f14a2000640 1 -- 172.21.15.45:0/1989081627 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149c017070 con 0x7f14ac0728c0 2024-04-16T16:22:19.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.771+0000 7f14a2000640 1 -- 172.21.15.45:0/1989081627 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f149c00c9a0 con 0x7f14ac0728c0 2024-04-16T16:22:19.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.771+0000 7f14a2000640 1 -- 172.21.15.45:0/1989081627 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149c00cd90 con 0x7f14ac0728c0 2024-04-16T16:22:19.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.771+0000 7f14b1000640 1 -- 172.21.15.45:0/1989081627 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14ac141dd0 con 0x7f14ac0728c0 2024-04-16T16:22:19.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.772+0000 7f14b1000640 1 -- 172.21.15.45:0/1989081627 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14ac142390 con 0x7f14ac0728c0 2024-04-16T16:22:19.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.773+0000 7f14b1000640 1 -- 172.21.15.45:0/1989081627 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14ac075780 con 0x7f14ac0728c0 2024-04-16T16:22:19.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.774+0000 7f14a2000640 1 -- 172.21.15.45:0/1989081627 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7f149c01a070 con 0x7f14ac0728c0 2024-04-16T16:22:19.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.775+0000 7f14a2000640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f148806ca60 0x7f148806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:19.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.775+0000 7f14a2000640 1 -- 172.21.15.45:0/1989081627 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f149c08c240 con 0x7f14ac0728c0 2024-04-16T16:22:19.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.776+0000 7f14a9400640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f148806ca60 0x7f148806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:19.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.777+0000 7f14a9400640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f148806ca60 0x7f148806ef20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1494036000 tx=0x7f1494030000 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:19.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:19.780+0000 7f14a2000640 1 -- 172.21.15.45:0/1989081627 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f149c05a3f0 con 0x7f14ac0728c0 2024-04-16T16:22:20.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.023+0000 7f14b1000640 1 -- 172.21.15.45:0/1989081627 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14ac0b3020 con 0x7f14ac0728c0 2024-04-16T16:22:20.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.024+0000 7f14a2000640 1 -- 172.21.15.45:0/1989081627 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) ==== 95+0+5212 (secure 0 0 0) 0x7f149c059cf0 con 0x7f14ac0728c0 2024-04-16T16:22:20.025 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:20.025 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":104,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2585865379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2585865379},{"type":"v1","addr":"172.21.15.45:6834","nonce":2585865379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6590,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/777087937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":777087937},{"type":"v1","addr":"172.21.15.45:6835","nonce":777087937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:21:26.066855+0000","modified":"2024-04-16T16:22:15.282978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6596,"mds_1":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":1,"incarnation":102,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6835/2561260581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2561260581},{"type":"v1","addr":"172.21.15.103:6835","nonce":2561260581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6596":{"gid":6596,"name":"d","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6837/3252460318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3252460318},{"type":"v1","addr":"172.21.15.103:6837","nonce":3252460318}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6596,"qdb_cluster":[6596,6593]},"id":10}]} 2024-04-16T16:22:20.025 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 104 2024-04-16T16:22:20.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.027+0000 7f14b1000640 1 -- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f148806ca60 msgr2=0x7f148806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:20.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.027+0000 7f14b1000640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f148806ca60 0x7f148806ef20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1494036000 tx=0x7f1494030000 comp rx=0 tx=0).stop 2024-04-16T16:22:20.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.027+0000 7f14b1000640 1 -- 172.21.15.45:0/1989081627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac0728c0 msgr2=0x7f14ac0b56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:20.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.027+0000 7f14b1000640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac0728c0 0x7f14ac0b56f0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f149c00e9b0 tx=0x7f149c00ee70 comp rx=0 tx=0).stop 2024-04-16T16:22:20.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.027+0000 7f14b1000640 1 -- 172.21.15.45:0/1989081627 shutdown_connections 2024-04-16T16:22:20.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.027+0000 7f14b1000640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f148806ca60 0x7f148806ef20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:20.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.027+0000 7f14b1000640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14ac1462e0 0x7f14ac143c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:20.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.027+0000 7f14b1000640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14ac0cec00 0x7f14ac13fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:20.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.027+0000 7f14b1000640 1 --2- 172.21.15.45:0/1989081627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac0728c0 0x7f14ac0b56f0 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:20.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.027+0000 7f14b1000640 1 -- 172.21.15.45:0/1989081627 >> 172.21.15.45:0/1989081627 conn(0x7f14ac0b2750 msgr2=0x7f14ac15f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:20.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.027+0000 7f14b1000640 1 -- 172.21.15.45:0/1989081627 shutdown_connections 2024-04-16T16:22:20.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.027+0000 7f14b1000640 1 -- 172.21.15.45:0/1989081627 wait complete. 2024-04-16T16:22:20.048 DEBUG:tasks.cephfs.test_mds_metrics:metrics={'version': 2, 'global_counters': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'counters': [], 'client_metadata': {'cephfs': {'client.6659': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '192.168.0.1'}, 'client.6710': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'IP': '172.21.15.103'}, 'client.7498': {'hostname': 'smithi103', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'IP': '172.21.15.103'}, 'client.9348': {'hostname': 'smithi045', 'root': '/', 'mount_point': 'N/A', 'valid_metrics': ['cap_hit', 'read_latency', 'write_latency', 'metadata_latency', 'dentry_lease', 'opened_files', 'pinned_icaps', 'opened_inodes', 'read_io_sizes', 'write_io_sizes', 'avg_read_latency', 'stdev_read_latency', 'avg_write_latency', 'stdev_write_latency', 'avg_metadata_latency', 'stdev_metadata_latency'], 'kernel_version': '6.9.0-rc3-g3f60ed14f7c1', 'IP': '192.168.0.1'}}}, 'global_metrics': {'cephfs': {'client.6659': [[52, 2], [0, 0], [0, 0], [0, 26614178], [6, 0], [0, 3], [3, 3], [0, 3], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 2957132], [102926059387199, 9]], 'client.6710': [[2, 0], [0, 0], [0, 0], [0, 980564], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 980564], [0, 1]], 'client.7498': [[2, 0], [0, 0], [0, 0], [0, 592826], [0, 0], [0, 1], [1, 1], [0, 1], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 592826], [0, 1]], 'client.9348': [[419, 22], [0, 0], [0, 507173266], [0, 150907705], [34, 0], [0, 35], [35, 35], [0, 35], [0, 0], [32, 33554432], [0, 0], [0, 0], [0, 15849167], [271636030061266, 32], [0, 2357924], [388853860915318, 64]]}}, 'metrics': {'delayed_ranks': [], 'mds.0': {'client.6659': [], 'client.6710': [], 'client.7498': [], 'client.9348': []}, 'mds.1': {'client.9348': []}}} 2024-04-16T16:22:20.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:20.048 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls) 2024-04-16T16:22:20.101 INFO:teuthology.orchestra.run.smithi045.stdout:test_mds_metrics_0 2024-04-16T16:22:20.102 INFO:teuthology.orchestra.run.smithi045.stdout:test_mds_metrics_1 2024-04-16T16:22:20.103 INFO:tasks.cephfs.test_mds_metrics:cleaning directory test_mds_metrics_0 2024-04-16T16:22:20.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:20.103 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf test_mds_metrics_0) 2024-04-16T16:22:20.289 INFO:tasks.cephfs.test_mds_metrics:cleaning directory test_mds_metrics_1 2024-04-16T16:22:20.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:20.290 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf test_mds_metrics_1) 2024-04-16T16:22:20.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable stats 2024-04-16T16:22:20.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.753+0000 7f2603200640 1 Processor -- start 2024-04-16T16:22:20.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.754+0000 7f2603200640 1 -- start start 2024-04-16T16:22:20.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.755+0000 7f2603200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f26040cf5c0 0x7f26040d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:20.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.755+0000 7f2603200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f260415eef0 0x7f26040cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:20.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.755+0000 7f2603200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26040d35a0 con 0x7f26040d42d0 2024-04-16T16:22:20.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.755+0000 7f2603200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f26040cfee0 con 0x7f26040cf5c0 2024-04-16T16:22:20.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.755+0000 7f2603200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f26040d2740 con 0x7f260415eef0 2024-04-16T16:22:20.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.755+0000 7f2601e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f26040cf5c0 0x7f26040d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:20.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.755+0000 7f2601e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f26040cf5c0 0x7f26040d3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39776/0 (socket says 172.21.15.45:39776) 2024-04-16T16:22:20.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.755+0000 7f2601e00640 1 -- 172.21.15.45:0/3923749224 learned_addr learned my addr 172.21.15.45:0/3923749224 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:20.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.756+0000 7f2601e00640 1 -- 172.21.15.45:0/3923749224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f260415eef0 msgr2=0x7f26040cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:20.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.756+0000 7f2601e00640 1 --2- 172.21.15.45:0/3923749224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f260415eef0 0x7f26040cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:20.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.756+0000 7f2601e00640 1 -- 172.21.15.45:0/3923749224 >> v1:172.21.15.103:6789/0 conn(0x7f26040d42d0 legacy=0x7f26040b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:20.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.756+0000 7f2601e00640 1 -- 172.21.15.45:0/3923749224 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26040d2a20 con 0x7f26040cf5c0 2024-04-16T16:22:20.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.757+0000 7f2601e00640 1 --2- 172.21.15.45:0/3923749224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26040cf5c0 0x7f26040d3980 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f25f40098a0 tx=0x7f25f402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f04a7a4b95b656 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:20.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.757+0000 7f2600a00640 1 -- 172.21.15.45:0/3923749224 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f4035020 con 0x7f26040cf5c0 2024-04-16T16:22:20.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.757+0000 7f2600a00640 1 -- 172.21.15.45:0/3923749224 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25f4031640 con 0x7f26040cf5c0 2024-04-16T16:22:20.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.757+0000 7f2600a00640 1 -- 172.21.15.45:0/3923749224 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f402d040 con 0x7f26040cf5c0 2024-04-16T16:22:20.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.758+0000 7f2603200640 1 -- 172.21.15.45:0/3923749224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26040cf5c0 msgr2=0x7f26040d3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:20.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.758+0000 7f2603200640 1 --2- 172.21.15.45:0/3923749224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26040cf5c0 0x7f26040d3980 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f25f40098a0 tx=0x7f25f402b8f0 comp rx=0 tx=0).stop 2024-04-16T16:22:20.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.758+0000 7f2603200640 1 -- 172.21.15.45:0/3923749224 shutdown_connections 2024-04-16T16:22:20.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.758+0000 7f2603200640 1 --2- 172.21.15.45:0/3923749224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f260415eef0 0x7f26040cf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:20.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.758+0000 7f2603200640 1 --2- 172.21.15.45:0/3923749224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26040cf5c0 0x7f26040d3980 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:20.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.758+0000 7f2603200640 1 -- 172.21.15.45:0/3923749224 >> 172.21.15.45:0/3923749224 conn(0x7f260406d300 msgr2=0x7f26040d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:20.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.758+0000 7f2603200640 1 -- 172.21.15.45:0/3923749224 shutdown_connections 2024-04-16T16:22:20.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.758+0000 7f2603200640 1 -- 172.21.15.45:0/3923749224 wait complete. 2024-04-16T16:22:20.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.761+0000 7f2603200640 1 Processor -- start 2024-04-16T16:22:20.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.761+0000 7f2603200640 1 -- start start 2024-04-16T16:22:20.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.762+0000 7f2603200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26040cf5c0 0x7f260414f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.762+0000 7f2603200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26040d42d0 0x7f2604159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.762+0000 7f2603200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f260415eef0 0x7f26041f5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.762+0000 7f2603200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f26040d32c0 con 0x7f26040cf5c0 2024-04-16T16:22:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.762+0000 7f2603200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f26040d2fe0 con 0x7f260415eef0 2024-04-16T16:22:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.762+0000 7f2603200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2604162b50 con 0x7f26040d42d0 2024-04-16T16:22:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.762+0000 7f2601e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26040cf5c0 0x7f260414f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.762+0000 7f2601e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26040cf5c0 0x7f260414f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39782/0 (socket says 172.21.15.45:39782) 2024-04-16T16:22:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.762+0000 7f2601e00640 1 -- 172.21.15.45:0/1268325999 learned_addr learned my addr 172.21.15.45:0/1268325999 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.763+0000 7f2602800640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f260415eef0 0x7f26041f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.763+0000 7f2601e00640 1 -- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26040d42d0 msgr2=0x7f2604159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.763+0000 7f2601400640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26040d42d0 0x7f2604159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.763+0000 7f2601e00640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26040d42d0 0x7f2604159540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.763+0000 7f2601e00640 1 -- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f260415eef0 msgr2=0x7f26041f5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.763+0000 7f2601e00640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f260415eef0 0x7f26041f5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:20.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.763+0000 7f2601e00640 1 -- 172.21.15.45:0/1268325999 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2604159da0 con 0x7f26040cf5c0 2024-04-16T16:22:20.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.763+0000 7f2602800640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f260415eef0 0x7f26041f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:20.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.763+0000 7f2601400640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26040d42d0 0x7f2604159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:20.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.763+0000 7f2601e00640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26040cf5c0 0x7f260414f110 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f25f4000c00 tx=0x7f25f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:20.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.764+0000 7f25fa000640 1 -- 172.21.15.45:0/1268325999 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f403ac60 con 0x7f26040cf5c0 2024-04-16T16:22:20.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.764+0000 7f25fa000640 1 -- 172.21.15.45:0/1268325999 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25f403ae20 con 0x7f26040cf5c0 2024-04-16T16:22:20.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.764+0000 7f25fa000640 1 -- 172.21.15.45:0/1268325999 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f4044630 con 0x7f26040cf5c0 2024-04-16T16:22:20.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.764+0000 7f2603200640 1 -- 172.21.15.45:0/1268325999 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f260415a080 con 0x7f26040cf5c0 2024-04-16T16:22:20.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.764+0000 7f2603200640 1 -- 172.21.15.45:0/1268325999 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26041608b0 con 0x7f26040cf5c0 2024-04-16T16:22:20.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.765+0000 7f2603200640 1 -- 172.21.15.45:0/1268325999 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26040d5960 con 0x7f26040cf5c0 2024-04-16T16:22:20.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.766+0000 7f25fa000640 1 -- 172.21.15.45:0/1268325999 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) ==== 91151+0+0 (secure 0 0 0) 0x7f25f4038020 con 0x7f26040cf5c0 2024-04-16T16:22:20.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.767+0000 7f25fa000640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f25e006ca60 0x7f25e006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:20.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.767+0000 7f25fa000640 1 -- 172.21.15.45:0/1268325999 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f25f40ade50 con 0x7f26040cf5c0 2024-04-16T16:22:20.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.768+0000 7f2601400640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f25e006ca60 0x7f25e006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:20.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.769+0000 7f2601400640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f25e006ca60 0x7f25e006ef20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2604161e50 tx=0x7f25ec0073e0 comp rx=0 tx=0).ready entity=mgr.6674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:20.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:20.773+0000 7f25fa000640 1 -- 172.21.15.45:0/1268325999 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f26040d5960 con 0x7f26040cf5c0 2024-04-16T16:22:21.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.033+0000 7f2603200640 1 -- 172.21.15.45:0/1268325999 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "stats"} v 0) -- 0x7f2604075780 con 0x7f26040cf5c0 2024-04-16T16:22:21.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.192+0000 7f25fa000640 1 -- 172.21.15.45:0/1268325999 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "stats"}]=0 v65) ==== 85+0+0 (secure 0 0 0) 0x7f25f403f070 con 0x7f26040cf5c0 2024-04-16T16:22:21.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.195+0000 7f2603200640 1 -- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f25e006ca60 msgr2=0x7f25e006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:21.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.195+0000 7f2603200640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f25e006ca60 0x7f25e006ef20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2604161e50 tx=0x7f25ec0073e0 comp rx=0 tx=0).stop 2024-04-16T16:22:21.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.196+0000 7f2603200640 1 -- 172.21.15.45:0/1268325999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26040cf5c0 msgr2=0x7f260414f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:21.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.196+0000 7f2603200640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26040cf5c0 0x7f260414f110 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f25f4000c00 tx=0x7f25f40046c0 comp rx=0 tx=0).stop 2024-04-16T16:22:21.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.196+0000 7f2603200640 1 -- 172.21.15.45:0/1268325999 shutdown_connections 2024-04-16T16:22:21.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.196+0000 7f2603200640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f25e006ca60 0x7f25e006ef20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:21.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.196+0000 7f2603200640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f260415eef0 0x7f26041f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:21.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.196+0000 7f2603200640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f26040d42d0 0x7f2604159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:21.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.196+0000 7f2603200640 1 --2- 172.21.15.45:0/1268325999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26040cf5c0 0x7f260414f110 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:21.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.196+0000 7f2603200640 1 -- 172.21.15.45:0/1268325999 >> 172.21.15.45:0/1268325999 conn(0x7f260406d300 msgr2=0x7f26040d3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:21.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.197+0000 7f2603200640 1 -- 172.21.15.45:0/1268325999 shutdown_connections 2024-04-16T16:22:21.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.197+0000 7f2603200640 1 -- 172.21.15.45:0/1268325999 wait complete. 2024-04-16T16:22:21.225 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-16T16:22:21.226 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:22:21.331 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:22:21.425 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-16T16:22:21.425 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:22:21.425 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:21.425 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:21.474 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:21.475 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:21.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:22:21.763 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:21.764+0000 7f7e2aa2c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:22:21.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.779+0000 7f1fe2400640 1 Processor -- start 2024-04-16T16:22:21.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.779+0000 7f1fe2400640 1 -- start start 2024-04-16T16:22:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.781+0000 7f1fe2400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1fdc09ae30 0x7f1fdc0720d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.781+0000 7f1fe2400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fdc09b210 0x7f1fdc15fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.781+0000 7f1fe2400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fdc09be50 con 0x7f1fdc0717f0 2024-04-16T16:22:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.781+0000 7f1fe2400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1fdc09bc00 con 0x7f1fdc09ae30 2024-04-16T16:22:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.781+0000 7f1fe2400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1fdc0cdc00 con 0x7f1fdc09b210 2024-04-16T16:22:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.781+0000 7f1fdb400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1fdc09ae30 0x7f1fdc0720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.781+0000 7f1fdb400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1fdc09ae30 0x7f1fdc0720d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39802/0 (socket says 172.21.15.45:39802) 2024-04-16T16:22:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.781+0000 7f1fdbe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1fdc0717f0 0x7f1fdc070420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45576/0 (socket says 172.21.15.45:45576) 2024-04-16T16:22:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.781+0000 7f1fdb400640 1 -- 172.21.15.45:0/3250465684 learned_addr learned my addr 172.21.15.45:0/3250465684 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.782+0000 7f1fd3e00640 1 --2- 172.21.15.45:0/3250465684 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fdc09b210 0x7f1fdc15fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.782+0000 7f1fdb400640 1 -- 172.21.15.45:0/3250465684 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fdc09b210 msgr2=0x7f1fdc15fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.782+0000 7f1fdb400640 1 --2- 172.21.15.45:0/3250465684 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fdc09b210 0x7f1fdc15fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.782+0000 7f1fdb400640 1 -- 172.21.15.45:0/3250465684 >> v1:172.21.15.45:6789/0 conn(0x7f1fdc0717f0 legacy=0x7f1fdc070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.782+0000 7f1fdb400640 1 -- 172.21.15.45:0/3250465684 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fdc0cd5b0 con 0x7f1fdc09ae30 2024-04-16T16:22:21.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.783+0000 7f1fdb400640 1 --2- 172.21.15.45:0/3250465684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fdc09ae30 0x7f1fdc0720d0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009870 tx=0x7f1fcc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d01ea8a5d951821 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:21.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.783+0000 7f1fdaa00640 1 -- 172.21.15.45:0/3250465684 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc034070 con 0x7f1fdc09ae30 2024-04-16T16:22:21.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.784+0000 7f1fdaa00640 1 -- 172.21.15.45:0/3250465684 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fcc030840 con 0x7f1fdc09ae30 2024-04-16T16:22:21.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.784+0000 7f1fdaa00640 1 -- 172.21.15.45:0/3250465684 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc030c10 con 0x7f1fdc09ae30 2024-04-16T16:22:21.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.784+0000 7f1fe2400640 1 -- 172.21.15.45:0/3250465684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fdc09ae30 msgr2=0x7f1fdc0720d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:21.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.784+0000 7f1fe2400640 1 --2- 172.21.15.45:0/3250465684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fdc09ae30 0x7f1fdc0720d0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009870 tx=0x7f1fcc02b980 comp rx=0 tx=0).stop 2024-04-16T16:22:21.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.784+0000 7f1fe2400640 1 -- 172.21.15.45:0/3250465684 shutdown_connections 2024-04-16T16:22:21.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.784+0000 7f1fe2400640 1 --2- 172.21.15.45:0/3250465684 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fdc09b210 0x7f1fdc15fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:21.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.784+0000 7f1fe2400640 1 --2- 172.21.15.45:0/3250465684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fdc09ae30 0x7f1fdc0720d0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.784+0000 7f1fe2400640 1 -- 172.21.15.45:0/3250465684 >> 172.21.15.45:0/3250465684 conn(0x7f1fdc0b4350 msgr2=0x7f1fdc0b4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.785+0000 7f1fe2400640 1 -- 172.21.15.45:0/3250465684 shutdown_connections 2024-04-16T16:22:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.785+0000 7f1fe2400640 1 -- 172.21.15.45:0/3250465684 wait complete. 2024-04-16T16:22:21.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.786+0000 7f1fe2400640 1 Processor -- start 2024-04-16T16:22:21.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.786+0000 7f1fe2400640 1 -- start start 2024-04-16T16:22:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.787+0000 7f1fe2400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fdc0717f0 0x7f1fdc157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.787+0000 7f1fe2400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fdc09ae30 0x7f1fdc0b5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.787+0000 7f1fe2400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fdc09b210 0x7f1fdc06dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.787+0000 7f1fe2400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1fdc0af610 con 0x7f1fdc09ae30 2024-04-16T16:22:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.787+0000 7f1fe2400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1fdc0cf0a0 con 0x7f1fdc0717f0 2024-04-16T16:22:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.787+0000 7f1fe2400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1fdc0cd1d0 con 0x7f1fdc09b210 2024-04-16T16:22:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.788+0000 7f1fd3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fdc09ae30 0x7f1fdc0b5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.788+0000 7f1fd3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fdc09ae30 0x7f1fdc0b5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39808/0 (socket says 172.21.15.45:39808) 2024-04-16T16:22:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.788+0000 7f1fd3e00640 1 -- 172.21.15.45:0/2380784614 learned_addr learned my addr 172.21.15.45:0/2380784614 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.788+0000 7f1fdbe00640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fdc09b210 0x7f1fdc06dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.788+0000 7f1fdb400640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fdc0717f0 0x7f1fdc157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.788+0000 7f1fd3e00640 1 -- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fdc09b210 msgr2=0x7f1fdc06dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.788+0000 7f1fd3e00640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fdc09b210 0x7f1fdc06dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.788+0000 7f1fd3e00640 1 -- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fdc0717f0 msgr2=0x7f1fdc157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.788+0000 7f1fd3e00640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fdc0717f0 0x7f1fdc157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.788+0000 7f1fd3e00640 1 -- 172.21.15.45:0/2380784614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fdc1f57c0 con 0x7f1fdc09ae30 2024-04-16T16:22:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.788+0000 7f1fdbe00640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fdc09b210 0x7f1fdc06dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.788+0000 7f1fdb400640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fdc0717f0 0x7f1fdc157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.789+0000 7f1fd3e00640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fdc09ae30 0x7f1fdc0b5320 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400d830 tx=0x7f1fc400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.789+0000 7f1fd3400640 1 -- 172.21.15.45:0/2380784614 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc400b6c0 con 0x7f1fdc09ae30 2024-04-16T16:22:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.789+0000 7f1fd3400640 1 -- 172.21.15.45:0/2380784614 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fc4019070 con 0x7f1fdc09ae30 2024-04-16T16:22:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.789+0000 7f1fd3400640 1 -- 172.21.15.45:0/2380784614 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc400ba80 con 0x7f1fdc09ae30 2024-04-16T16:22:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.789+0000 7f1fe2400640 1 -- 172.21.15.45:0/2380784614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fdc0c8100 con 0x7f1fdc09ae30 2024-04-16T16:22:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.789+0000 7f1fe2400640 1 -- 172.21.15.45:0/2380784614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fdc0c86c0 con 0x7f1fdc09ae30 2024-04-16T16:22:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.790+0000 7f1fe2400640 1 -- 172.21.15.45:0/2380784614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fdc075780 con 0x7f1fdc09ae30 2024-04-16T16:22:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.791+0000 7f1fd3400640 1 -- 172.21.15.45:0/2380784614 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 65) ==== 91142+0+0 (secure 0 0 0) 0x7f1fc401e460 con 0x7f1fdc09ae30 2024-04-16T16:22:21.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.792+0000 7f1fd3400640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f1fb806ca40 0x7f1fb806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:21.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.792+0000 7f1fd3400640 1 -- 172.21.15.45:0/2380784614 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f1fc408c450 con 0x7f1fdc09ae30 2024-04-16T16:22:21.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.792+0000 7f1fdb400640 1 -- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f1fb806ca40 msgr2=0x7f1fb806ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1506110537 2024-04-16T16:22:21.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.792+0000 7f1fdb400640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f1fb806ca40 0x7f1fb806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:22:21.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.796+0000 7f1fd3400640 1 -- 172.21.15.45:0/2380784614 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1fc405a6a0 con 0x7f1fdc09ae30 2024-04-16T16:22:21.799 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:21.799+0000 7fd7fb02e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T16:22:21.819 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:21.820+0000 7f7e2aa2c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:22:21.860 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:21.860+0000 7fd7fb02e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T16:22:21.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.993+0000 7f1fdb400640 1 -- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f1fb806ca40 msgr2=0x7f1fb806ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1506110537 2024-04-16T16:22:21.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:21.993+0000 7f1fdb400640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f1fb806ca40 0x7f1fb806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:22:22.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.000+0000 7f1fe2400640 1 -- 172.21.15.45:0/2380784614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1fdc1e63d0 con 0x7f1fdc09ae30 2024-04-16T16:22:22.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.002+0000 7f1fd3400640 1 -- 172.21.15.45:0/2380784614 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) ==== 74+0+17187 (secure 0 0 0) 0x7f1fc4059fa0 con 0x7f1fdc09ae30 2024-04-16T16:22:22.003 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:22.003 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":143,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:21:47.933077+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:21:23.973149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-16T16:21:25.043255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/884289453":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/1437683872":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/2877575799":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/2644878831":"2024-04-17T16:21:47.933003+0000","172.21.15.103:6833/1478535274":"2024-04-17T16:21:47.933003+0000","172.21.15.103:6832/1478535274":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/1111841625":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/1114329424":"2024-04-17T16:21:47.933003+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:22:22.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.005+0000 7f1fe2400640 1 -- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f1fb806ca40 msgr2=0x7f1fb806ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:22:22.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.005+0000 7f1fe2400640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f1fb806ca40 0x7f1fb806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.005+0000 7f1fe2400640 1 -- 172.21.15.45:0/2380784614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fdc09ae30 msgr2=0x7f1fdc0b5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:22.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.005+0000 7f1fe2400640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fdc09ae30 0x7f1fdc0b5320 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400d830 tx=0x7f1fc400dcf0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.005+0000 7f1fe2400640 1 -- 172.21.15.45:0/2380784614 shutdown_connections 2024-04-16T16:22:22.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.005+0000 7f1fe2400640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f1fb806ca40 0x7f1fb806ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.005+0000 7f1fe2400640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fdc09b210 0x7f1fdc06dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.005+0000 7f1fe2400640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fdc09ae30 0x7f1fdc0b5320 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.005+0000 7f1fe2400640 1 --2- 172.21.15.45:0/2380784614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fdc0717f0 0x7f1fdc157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.005+0000 7f1fe2400640 1 -- 172.21.15.45:0/2380784614 >> 172.21.15.45:0/2380784614 conn(0x7f1fdc0b4350 msgr2=0x7f1fdc0cf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:22.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.005+0000 7f1fe2400640 1 -- 172.21.15.45:0/2380784614 shutdown_connections 2024-04-16T16:22:22.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.005+0000 7f1fe2400640 1 -- 172.21.15.45:0/2380784614 wait complete. 2024-04-16T16:22:22.024 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:21:23.973149+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:21:25.043255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:22:22.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:22:22.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.221+0000 7f98cf800640 1 Processor -- start 2024-04-16T16:22:22.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.221+0000 7f98cf800640 1 -- start start 2024-04-16T16:22:22.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.223+0000 7f98cf800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f98c80cfe00 0x7f98c80d2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:22.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.223+0000 7f98cf800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98c80d1f10 0x7f98c80d2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:22.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.223+0000 7f98cf800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98c80d4640 con 0x7f98c81622d0 2024-04-16T16:22:22.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.223+0000 7f98cf800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f98c80d43f0 con 0x7f98c80cfe00 2024-04-16T16:22:22.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.223+0000 7f98cf800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f98c80ba4c0 con 0x7f98c80d1f10 2024-04-16T16:22:22.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.223+0000 7f98c7e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f98c80cfe00 0x7f98c80d2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:22.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.223+0000 7f98cce00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f98c81622d0 0x7f98c80cf6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45584/0 (socket says 172.21.15.45:45584) 2024-04-16T16:22:22.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.223+0000 7f98cce00640 1 -- 172.21.15.45:0/939719111 learned_addr learned my addr 172.21.15.45:0/939719111 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:22.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.224+0000 7f98cd800640 1 --2- 172.21.15.45:0/939719111 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98c80d1f10 0x7f98c80d2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:22.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.224+0000 7f98c7e00640 1 -- 172.21.15.45:0/939719111 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98c80d1f10 msgr2=0x7f98c80d2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:22.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.224+0000 7f98c7e00640 1 --2- 172.21.15.45:0/939719111 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98c80d1f10 0x7f98c80d2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.224+0000 7f98c7e00640 1 -- 172.21.15.45:0/939719111 >> v1:172.21.15.45:6789/0 conn(0x7f98c81622d0 legacy=0x7f98c80cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:22.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.224+0000 7f98c7e00640 1 -- 172.21.15.45:0/939719111 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c80b7b50 con 0x7f98c80cfe00 2024-04-16T16:22:22.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.225+0000 7f98c7e00640 1 --2- 172.21.15.45:0/939719111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98c80cfe00 0x7f98c80d2680 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f98b40097a0 tx=0x7f98b402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab665c3aeccb970f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:22.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.225+0000 7f98c7400640 1 -- 172.21.15.45:0/939719111 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b4035020 con 0x7f98c80cfe00 2024-04-16T16:22:22.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.225+0000 7f98c7400640 1 -- 172.21.15.45:0/939719111 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98b4031810 con 0x7f98c80cfe00 2024-04-16T16:22:22.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.225+0000 7f98c7400640 1 -- 172.21.15.45:0/939719111 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b4031be0 con 0x7f98c80cfe00 2024-04-16T16:22:22.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.226+0000 7f98cf800640 1 -- 172.21.15.45:0/939719111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98c80cfe00 msgr2=0x7f98c80d2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:22.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.226+0000 7f98cf800640 1 --2- 172.21.15.45:0/939719111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98c80cfe00 0x7f98c80d2680 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f98b40097a0 tx=0x7f98b402b8b0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.226+0000 7f98cf800640 1 -- 172.21.15.45:0/939719111 shutdown_connections 2024-04-16T16:22:22.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.226+0000 7f98cf800640 1 --2- 172.21.15.45:0/939719111 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98c80d1f10 0x7f98c80d2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.226+0000 7f98cf800640 1 --2- 172.21.15.45:0/939719111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98c80cfe00 0x7f98c80d2680 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.226+0000 7f98cf800640 1 -- 172.21.15.45:0/939719111 >> 172.21.15.45:0/939719111 conn(0x7f98c806d400 msgr2=0x7f98c806d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:22.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.226+0000 7f98cf800640 1 -- 172.21.15.45:0/939719111 shutdown_connections 2024-04-16T16:22:22.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.226+0000 7f98cf800640 1 -- 172.21.15.45:0/939719111 wait complete. 2024-04-16T16:22:22.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.228+0000 7f98cf800640 1 Processor -- start 2024-04-16T16:22:22.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.228+0000 7f98cf800640 1 -- start start 2024-04-16T16:22:22.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.229+0000 7f98cf800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98c80cfe00 0x7f98c814acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:22.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.229+0000 7f98cf800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98c80d1f10 0x7f98c80cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:22.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.229+0000 7f98cf800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98c81622d0 0x7f98c80cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:22.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.229+0000 7f98cf800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f98c80b70e0 con 0x7f98c80cfe00 2024-04-16T16:22:22.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.229+0000 7f98cf800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f98c80b6f60 con 0x7f98c81622d0 2024-04-16T16:22:22.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.229+0000 7f98cf800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f98c80d3100 con 0x7f98c80d1f10 2024-04-16T16:22:22.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.229+0000 7f98cce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98c80cfe00 0x7f98c814acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:22.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.229+0000 7f98cce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98c80cfe00 0x7f98c814acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39836/0 (socket says 172.21.15.45:39836) 2024-04-16T16:22:22.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.229+0000 7f98cce00640 1 -- 172.21.15.45:0/3950376263 learned_addr learned my addr 172.21.15.45:0/3950376263 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:22.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.230+0000 7f98c7e00640 1 --2- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98c80d1f10 0x7f98c80cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:22.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.230+0000 7f98cce00640 1 -- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98c80d1f10 msgr2=0x7f98c80cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:22.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.230+0000 7f98cce00640 1 --2- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98c80d1f10 0x7f98c80cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.230+0000 7f98cce00640 1 -- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98c81622d0 msgr2=0x7f98c80cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:22.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.230+0000 7f98cd800640 1 --2- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98c81622d0 0x7f98c80cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:22.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.230+0000 7f98cce00640 1 --2- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98c81622d0 0x7f98c80cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.230+0000 7f98cce00640 1 -- 172.21.15.45:0/3950376263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c80ceb00 con 0x7f98c80cfe00 2024-04-16T16:22:22.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.230+0000 7f98cce00640 1 --2- 172.21.15.45:0/3950376263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98c80cfe00 0x7f98c814acf0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00c3c0 tx=0x7f98bc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:22.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.230+0000 7f98c4c00640 1 -- 172.21.15.45:0/3950376263 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc017070 con 0x7f98c80cfe00 2024-04-16T16:22:22.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.230+0000 7f98c4c00640 1 -- 172.21.15.45:0/3950376263 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98bc005ce0 con 0x7f98c80cfe00 2024-04-16T16:22:22.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.230+0000 7f98c4c00640 1 -- 172.21.15.45:0/3950376263 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc0072f0 con 0x7f98c80cfe00 2024-04-16T16:22:22.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.230+0000 7f98cf800640 1 -- 172.21.15.45:0/3950376263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c81f4af0 con 0x7f98c80cfe00 2024-04-16T16:22:22.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.231+0000 7f98cf800640 1 -- 172.21.15.45:0/3950376263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c80cdf80 con 0x7f98c80cfe00 2024-04-16T16:22:22.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.232+0000 7f98cf800640 1 -- 172.21.15.45:0/3950376263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98c81482d0 con 0x7f98c80cfe00 2024-04-16T16:22:22.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.233+0000 7f98c4c00640 1 -- 172.21.15.45:0/3950376263 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 65) ==== 91142+0+0 (secure 0 0 0) 0x7f98bc01a070 con 0x7f98c80cfe00 2024-04-16T16:22:22.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.233+0000 7f98c4c00640 1 --2- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f98a806ca10 0x7f98a806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:22.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.234+0000 7f98c4c00640 1 -- 172.21.15.45:0/3950376263 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f98bc08b130 con 0x7f98c80cfe00 2024-04-16T16:22:22.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.234+0000 7f98c7e00640 1 -- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f98a806ca10 msgr2=0x7f98a806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1506110537 2024-04-16T16:22:22.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.234+0000 7f98c7e00640 1 --2- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f98a806ca10 0x7f98a806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:22:22.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.238+0000 7f98c4c00640 1 -- 172.21.15.45:0/3950376263 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f98c81482d0 con 0x7f98c80cfe00 2024-04-16T16:22:22.263 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:22.264+0000 7f7e2aa2c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:22:22.315 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:22.316+0000 7f7e2aa2c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:22:22.355 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:22.356+0000 7fd7fb02e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T16:22:22.365 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:22.366+0000 7f7e2aa2c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:22:22.412 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:22.412+0000 7fd7fb02e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T16:22:22.425 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:22.426+0000 7f7e2aa2c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:22:22.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.435+0000 7f98c7e00640 1 -- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f98a806ca10 msgr2=0x7f98a806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1506110537 2024-04-16T16:22:22.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.435+0000 7f98c7e00640 1 --2- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f98a806ca10 0x7f98a806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:22:22.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.456+0000 7f98cf800640 1 -- 172.21.15.45:0/3950376263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f98c8075780 con 0x7f98c80cfe00 2024-04-16T16:22:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.456+0000 7f98c4c00640 1 -- 172.21.15.45:0/3950376263 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) ==== 93+0+10 (secure 0 0 0) 0x7f98bc059380 con 0x7f98c80cfe00 2024-04-16T16:22:22.456 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:22:22.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.459+0000 7f98cf800640 1 -- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f98a806ca10 msgr2=0x7f98a806eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:22:22.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.459+0000 7f98cf800640 1 --2- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f98a806ca10 0x7f98a806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.459+0000 7f98cf800640 1 -- 172.21.15.45:0/3950376263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98c80cfe00 msgr2=0x7f98c814acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:22.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.459+0000 7f98cf800640 1 --2- 172.21.15.45:0/3950376263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98c80cfe00 0x7f98c814acf0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00c3c0 tx=0x7f98bc00c880 comp rx=0 tx=0).stop 2024-04-16T16:22:22.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.459+0000 7f98cf800640 1 -- 172.21.15.45:0/3950376263 shutdown_connections 2024-04-16T16:22:22.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.459+0000 7f98cf800640 1 --2- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f98a806ca10 0x7f98a806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.459+0000 7f98cf800640 1 --2- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98c81622d0 0x7f98c80cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.459+0000 7f98cf800640 1 --2- 172.21.15.45:0/3950376263 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98c80d1f10 0x7f98c80cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.459+0000 7f98cf800640 1 --2- 172.21.15.45:0/3950376263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98c80cfe00 0x7f98c814acf0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.459+0000 7f98cf800640 1 -- 172.21.15.45:0/3950376263 >> 172.21.15.45:0/3950376263 conn(0x7f98c806d400 msgr2=0x7f98c806d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:22.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.460+0000 7f98cf800640 1 -- 172.21.15.45:0/3950376263 shutdown_connections 2024-04-16T16:22:22.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.460+0000 7f98cf800640 1 -- 172.21.15.45:0/3950376263 wait complete. 2024-04-16T16:22:22.467 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:22.468+0000 7fd7fb02e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T16:22:22.474 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:22.475+0000 7f7e2aa2c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:22:22.479 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:22:22.536 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:22.536+0000 7fd7fb02e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T16:22:22.589 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:22.590+0000 7fd7fb02e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T16:22:22.602 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:22.603+0000 7f7e2aa2c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:22:22.671 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:22.671+0000 7f7e2aa2c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:22:22.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.694+0000 7fd296000640 1 Processor -- start 2024-04-16T16:22:22.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.694+0000 7fd296000640 1 -- start start 2024-04-16T16:22:22.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.695+0000 7fd296000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2900d10e0 0x7fd290072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:22.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.695+0000 7fd296000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2900b5530 con 0x7fd29015c7c0 2024-04-16T16:22:22.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.695+0000 7fd296000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2900b5810 con 0x7fd2900b4930 2024-04-16T16:22:22.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.695+0000 7fd296000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd2900b52e0 con 0x7fd2900d10e0 2024-04-16T16:22:22.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.696+0000 7fd28f400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd29015c7c0 0x7fd2900d09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45594/0 (socket says 172.21.15.45:45594) 2024-04-16T16:22:22.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.696+0000 7fd28f400640 1 -- 172.21.15.45:0/3684624296 learned_addr learned my addr 172.21.15.45:0/3684624296 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:22.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.696+0000 7fd28ea00640 1 --2- 172.21.15.45:0/3684624296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2900d10e0 0x7fd290072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:22.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.696+0000 7fd28e000640 1 -- 172.21.15.45:0/3684624296 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3034829092 0 0) 0x7fd2900b5530 con 0x7fd29015c7c0 2024-04-16T16:22:22.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.696+0000 7fd28e000640 1 -- 172.21.15.45:0/3684624296 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd274003520 con 0x7fd29015c7c0 2024-04-16T16:22:22.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28e000640 1 -- 172.21.15.45:0/3684624296 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 513172624 0 0) 0x7fd274003520 con 0x7fd29015c7c0 2024-04-16T16:22:22.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28e000640 1 -- 172.21.15.45:0/3684624296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2900d10e0 msgr2=0x7fd290072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:22.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28e000640 1 --2- 172.21.15.45:0/3684624296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2900d10e0 0x7fd290072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28e000640 1 -- 172.21.15.45:0/3684624296 >> v1:172.21.15.103:6789/0 conn(0x7fd2900b4930 legacy=0x7fd29006f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:22.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28e000640 1 -- 172.21.15.45:0/3684624296 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2900d04b0 con 0x7fd29015c7c0 2024-04-16T16:22:22.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28e000640 1 -- 172.21.15.45:0/3684624296 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fd280001670 con 0x7fd29015c7c0 2024-04-16T16:22:22.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28e000640 1 -- 172.21.15.45:0/3684624296 >> v1:172.21.15.45:6789/0 conn(0x7fd29015c7c0 legacy=0x7fd2900d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:22.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28e000640 1 --2- 172.21.15.45:0/3684624296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2900d10e0 0x7fd2740039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:22.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28e000640 1 --2- 172.21.15.45:0/3684624296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd274003f60 0x7fd274004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:22.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28e000640 1 --2- 172.21.15.45:0/3684624296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd274004d40 0x7fd2740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:22.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28e000640 1 -- 172.21.15.45:0/3684624296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd2900b52e0 con 0x7fd274003f60 2024-04-16T16:22:22.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28e000640 1 -- 172.21.15.45:0/3684624296 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd27400bc00 con 0x7fd274004d40 2024-04-16T16:22:22.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28e000640 1 -- 172.21.15.45:0/3684624296 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd27400be00 con 0x7fd2900d10e0 2024-04-16T16:22:22.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28ea00640 1 --2- 172.21.15.45:0/3684624296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd274003f60 0x7fd274004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28f400640 1 --2- 172.21.15.45:0/3684624296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2900d10e0 0x7fd2740039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28ea00640 1 -- 172.21.15.45:0/3684624296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2900d10e0 msgr2=0x7fd2740039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28ea00640 1 --2- 172.21.15.45:0/3684624296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2900d10e0 0x7fd2740039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28ea00640 1 -- 172.21.15.45:0/3684624296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd274004d40 msgr2=0x7fd2740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28ea00640 1 --2- 172.21.15.45:0/3684624296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd274004d40 0x7fd2740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.697+0000 7fd28ea00640 1 -- 172.21.15.45:0/3684624296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2900d04b0 con 0x7fd274003f60 2024-04-16T16:22:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.699+0000 7fd28ea00640 1 --2- 172.21.15.45:0/3684624296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd274003f60 0x7fd274004800 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009a90 tx=0x7fd274005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1cfdf4d45771b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.699+0000 7fd28e000640 1 -- 172.21.15.45:0/3684624296 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c039070 con 0x7fd274003f60 2024-04-16T16:22:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.699+0000 7fd28e000640 1 -- 172.21.15.45:0/3684624296 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd27c02c680 con 0x7fd274003f60 2024-04-16T16:22:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.699+0000 7fd28e000640 1 -- 172.21.15.45:0/3684624296 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c009690 con 0x7fd274003f60 2024-04-16T16:22:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.699+0000 7fd296000640 1 -- 172.21.15.45:0/3684624296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd274003f60 msgr2=0x7fd274004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:22.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.699+0000 7fd296000640 1 --2- 172.21.15.45:0/3684624296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd274003f60 0x7fd274004800 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009a90 tx=0x7fd274005830 comp rx=0 tx=0).stop 2024-04-16T16:22:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.699+0000 7fd28fe00640 1 -- 172.21.15.45:0/3684624296 reap_dead start 2024-04-16T16:22:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.700+0000 7fd296000640 1 -- 172.21.15.45:0/3684624296 shutdown_connections 2024-04-16T16:22:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.700+0000 7fd296000640 1 -- 172.21.15.45:0/3684624296 >> 172.21.15.45:0/3684624296 conn(0x7fd2900b3970 msgr2=0x7fd2900b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.700+0000 7fd296000640 1 -- 172.21.15.45:0/3684624296 shutdown_connections 2024-04-16T16:22:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.700+0000 7fd296000640 1 -- 172.21.15.45:0/3684624296 wait complete. 2024-04-16T16:22:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.701+0000 7fd296000640 1 Processor -- start 2024-04-16T16:22:22.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.701+0000 7fd296000640 1 -- start start 2024-04-16T16:22:22.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.702+0000 7fd296000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd274003f60 0x7fd290152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:22.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.702+0000 7fd296000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd290155af0 0x7fd2901f7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:22.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.702+0000 7fd296000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2901f8150 0x7fd2901f85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:22.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.702+0000 7fd296000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd29006fa80 con 0x7fd290155af0 2024-04-16T16:22:22.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.702+0000 7fd296000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd2900d0260 con 0x7fd274003f60 2024-04-16T16:22:22.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.702+0000 7fd296000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd290154d90 con 0x7fd2901f8150 2024-04-16T16:22:22.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.703+0000 7fd28ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd290155af0 0x7fd2901f7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:22.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.703+0000 7fd28ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd290155af0 0x7fd2901f7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39844/0 (socket says 172.21.15.45:39844) 2024-04-16T16:22:22.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.703+0000 7fd28ea00640 1 -- 172.21.15.45:0/696442251 learned_addr learned my addr 172.21.15.45:0/696442251 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:22.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.703+0000 7fd28f400640 1 --2- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd274003f60 0x7fd290152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:22.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.703+0000 7fd28fe00640 1 --2- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2901f8150 0x7fd2901f85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:22.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.703+0000 7fd28ea00640 1 -- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2901f8150 msgr2=0x7fd2901f85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:22.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.703+0000 7fd28ea00640 1 --2- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2901f8150 0x7fd2901f85d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.703+0000 7fd28ea00640 1 -- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd274003f60 msgr2=0x7fd290152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.703+0000 7fd28ea00640 1 --2- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd274003f60 0x7fd290152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.703+0000 7fd28ea00640 1 -- 172.21.15.45:0/696442251 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd290154fe0 con 0x7fd290155af0 2024-04-16T16:22:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.703+0000 7fd28f400640 1 --2- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd274003f60 0x7fd290152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.703+0000 7fd28ea00640 1 --2- 172.21.15.45:0/696442251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd290155af0 0x7fd2901f7c10 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fd27c037040 tx=0x7fd27c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.704+0000 7fd287400640 1 -- 172.21.15.45:0/696442251 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c039050 con 0x7fd290155af0 2024-04-16T16:22:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.704+0000 7fd287400640 1 -- 172.21.15.45:0/696442251 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd27c045070 con 0x7fd290155af0 2024-04-16T16:22:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.704+0000 7fd287400640 1 -- 172.21.15.45:0/696442251 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c0044e0 con 0x7fd290155af0 2024-04-16T16:22:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.704+0000 7fd296000640 1 -- 172.21.15.45:0/696442251 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2901552c0 con 0x7fd290155af0 2024-04-16T16:22:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.704+0000 7fd296000640 1 -- 172.21.15.45:0/696442251 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd290152b60 con 0x7fd290155af0 2024-04-16T16:22:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.705+0000 7fd296000640 1 -- 172.21.15.45:0/696442251 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd290075780 con 0x7fd290155af0 2024-04-16T16:22:22.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.706+0000 7fd287400640 1 -- 172.21.15.45:0/696442251 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 65) ==== 91142+0+0 (secure 0 0 0) 0x7fd27c003850 con 0x7fd290155af0 2024-04-16T16:22:22.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.707+0000 7fd287400640 1 --2- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd27006ca40 0x7fd27006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:22.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.707+0000 7fd287400640 1 -- 172.21.15.45:0/696442251 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7fd27c0ad470 con 0x7fd290155af0 2024-04-16T16:22:22.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.707+0000 7fd28f400640 1 -- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd27006ca40 msgr2=0x7fd27006ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1506110537 2024-04-16T16:22:22.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.707+0000 7fd28f400640 1 --2- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd27006ca40 0x7fd27006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:22:22.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.711+0000 7fd287400640 1 -- 172.21.15.45:0/696442251 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd27c07b620 con 0x7fd290155af0 2024-04-16T16:22:22.723 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:22.723+0000 7f7e2aa2c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:22:22.736 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:22.736+0000 7fd7fb02e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T16:22:22.780 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:22.780+0000 7f7e2aa2c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:22:22.802 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:22.803+0000 7fd7fb02e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T16:22:22.830 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:22.830+0000 7f7e2aa2c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:22:22.855 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:22.855+0000 7fd7fb02e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T16:22:22.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.908+0000 7fd28f400640 1 -- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd27006ca40 msgr2=0x7fd27006ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1506110537 2024-04-16T16:22:22.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.908+0000 7fd28f400640 1 --2- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd27006ca40 0x7fd27006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:22:22.915 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:22.915+0000 7fd7fb02e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T16:22:22.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.918+0000 7fd296000640 1 -- 172.21.15.45:0/696442251 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd29015a9f0 con 0x7fd290155af0 2024-04-16T16:22:22.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.919+0000 7fd287400640 1 -- 172.21.15.45:0/696442251 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) ==== 104+0+11 (secure 0 0 0) 0x7fd27c07af20 con 0x7fd290155af0 2024-04-16T16:22:22.919 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:22:22.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.921+0000 7fd296000640 1 -- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd27006ca40 msgr2=0x7fd27006ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:22:22.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.921+0000 7fd296000640 1 --2- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd27006ca40 0x7fd27006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.921+0000 7fd296000640 1 -- 172.21.15.45:0/696442251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd290155af0 msgr2=0x7fd2901f7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:22.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.921+0000 7fd296000640 1 --2- 172.21.15.45:0/696442251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd290155af0 0x7fd2901f7c10 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fd27c037040 tx=0x7fd27c004730 comp rx=0 tx=0).stop 2024-04-16T16:22:22.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.921+0000 7fd296000640 1 -- 172.21.15.45:0/696442251 shutdown_connections 2024-04-16T16:22:22.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.921+0000 7fd296000640 1 --2- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd27006ca40 0x7fd27006ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.921+0000 7fd296000640 1 --2- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2901f8150 0x7fd2901f85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.921+0000 7fd296000640 1 --2- 172.21.15.45:0/696442251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd290155af0 0x7fd2901f7c10 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.921+0000 7fd296000640 1 --2- 172.21.15.45:0/696442251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd274003f60 0x7fd290152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:22.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.921+0000 7fd296000640 1 -- 172.21.15.45:0/696442251 >> 172.21.15.45:0/696442251 conn(0x7fd2900b3970 msgr2=0x7fd2900b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:22.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.922+0000 7fd296000640 1 -- 172.21.15.45:0/696442251 shutdown_connections 2024-04-16T16:22:22.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:22.922+0000 7fd296000640 1 -- 172.21.15.45:0/696442251 wait complete. 2024-04-16T16:22:22.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:22:22.969 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:22.970+0000 7fd7fb02e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T16:22:23.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.156+0000 7fc6b0e00640 1 Processor -- start 2024-04-16T16:22:23.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.156+0000 7fc6b0e00640 1 -- start start 2024-04-16T16:22:23.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.158+0000 7fc6b0e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc6ac145560 0x7fc6ac15cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:23.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.158+0000 7fc6b0e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc6ac15d470 0x7fc6ac0cfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:23.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.158+0000 7fc6b0e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6ac15d980 con 0x7fc6ac0d01f0 2024-04-16T16:22:23.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.158+0000 7fc6b0e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc6ac0cf820 con 0x7fc6ac15d470 2024-04-16T16:22:23.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.158+0000 7fc6b0e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc6ac15dc60 con 0x7fc6ac145560 2024-04-16T16:22:23.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.158+0000 7fc6aaa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc6ac15d470 0x7fc6ac0cfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:23.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.158+0000 7fc6aaa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc6ac15d470 0x7fc6ac0cfcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39860/0 (socket says 172.21.15.45:39860) 2024-04-16T16:22:23.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.158+0000 7fc6abe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc6ac0d01f0 0x7fc6ac070ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45598/0 (socket says 172.21.15.45:45598) 2024-04-16T16:22:23.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.158+0000 7fc6aaa00640 1 -- 172.21.15.45:0/1865245973 learned_addr learned my addr 172.21.15.45:0/1865245973 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:23.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.158+0000 7fc6ab400640 1 --2- 172.21.15.45:0/1865245973 >> v2:172.21.15.103:3300/0 conn(0x7fc6ac145560 0x7fc6ac15cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:23.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.159+0000 7fc6aaa00640 1 -- 172.21.15.45:0/1865245973 >> v2:172.21.15.103:3300/0 conn(0x7fc6ac145560 msgr2=0x7fc6ac15cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:23.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.159+0000 7fc6aaa00640 1 --2- 172.21.15.45:0/1865245973 >> v2:172.21.15.103:3300/0 conn(0x7fc6ac145560 0x7fc6ac15cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.159+0000 7fc6aaa00640 1 -- 172.21.15.45:0/1865245973 >> v1:172.21.15.45:6789/0 conn(0x7fc6ac0d01f0 legacy=0x7fc6ac070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:23.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.159+0000 7fc6aaa00640 1 -- 172.21.15.45:0/1865245973 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ac0cf4b0 con 0x7fc6ac15d470 2024-04-16T16:22:23.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.160+0000 7fc6aaa00640 1 --2- 172.21.15.45:0/1865245973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6ac15d470 0x7fc6ac0cfcb0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac06ca10 tx=0x7fc69c02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338bf23683669f6d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:23.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.160+0000 7fc6aa000640 1 -- 172.21.15.45:0/1865245973 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c02cb10 con 0x7fc6ac15d470 2024-04-16T16:22:23.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.160+0000 7fc6aa000640 1 -- 172.21.15.45:0/1865245973 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc69c02ccb0 con 0x7fc6ac15d470 2024-04-16T16:22:23.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.160+0000 7fc6aa000640 1 -- 172.21.15.45:0/1865245973 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c02ce60 con 0x7fc6ac15d470 2024-04-16T16:22:23.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.161+0000 7fc6b0e00640 1 -- 172.21.15.45:0/1865245973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6ac15d470 msgr2=0x7fc6ac0cfcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:23.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.161+0000 7fc6b0e00640 1 --2- 172.21.15.45:0/1865245973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6ac15d470 0x7fc6ac0cfcb0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac06ca10 tx=0x7fc69c02bf70 comp rx=0 tx=0).stop 2024-04-16T16:22:23.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.161+0000 7fc6b0e00640 1 -- 172.21.15.45:0/1865245973 shutdown_connections 2024-04-16T16:22:23.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.161+0000 7fc6b0e00640 1 --2- 172.21.15.45:0/1865245973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6ac15d470 0x7fc6ac0cfcb0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.161+0000 7fc6b0e00640 1 --2- 172.21.15.45:0/1865245973 >> v2:172.21.15.103:3300/0 conn(0x7fc6ac145560 0x7fc6ac15cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.161+0000 7fc6b0e00640 1 -- 172.21.15.45:0/1865245973 >> 172.21.15.45:0/1865245973 conn(0x7fc6ac06f0c0 msgr2=0x7fc6ac06f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:23.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.161+0000 7fc6b0e00640 1 -- 172.21.15.45:0/1865245973 shutdown_connections 2024-04-16T16:22:23.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.161+0000 7fc6b0e00640 1 -- 172.21.15.45:0/1865245973 wait complete. 2024-04-16T16:22:23.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.163+0000 7fc6b0e00640 1 Processor -- start 2024-04-16T16:22:23.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.163+0000 7fc6b0e00640 1 -- start start 2024-04-16T16:22:23.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.164+0000 7fc6b0e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6ac0d01f0 0x7fc6ac14e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:23.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.164+0000 7fc6b0e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6ac145560 0x7fc6ac1415a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:23.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.164+0000 7fc6b0e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6ac15d470 0x7fc6ac13f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:23.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.164+0000 7fc6b0e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac145f00 con 0x7fc6ac145560 2024-04-16T16:22:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.164+0000 7fc6b0e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac145d80 con 0x7fc6ac0d01f0 2024-04-16T16:22:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.164+0000 7fc6b0e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc6ac071700 con 0x7fc6ac15d470 2024-04-16T16:22:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.164+0000 7fc6aaa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6ac145560 0x7fc6ac1415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.164+0000 7fc6aaa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6ac145560 0x7fc6ac1415a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39862/0 (socket says 172.21.15.45:39862) 2024-04-16T16:22:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.164+0000 7fc6aaa00640 1 -- 172.21.15.45:0/1815912237 learned_addr learned my addr 172.21.15.45:0/1815912237 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.164+0000 7fc6ab400640 1 --2- 172.21.15.45:0/1815912237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6ac0d01f0 0x7fc6ac14e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.164+0000 7fc6aaa00640 1 -- 172.21.15.45:0/1815912237 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6ac15d470 msgr2=0x7fc6ac13f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.164+0000 7fc6aaa00640 1 --2- 172.21.15.45:0/1815912237 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6ac15d470 0x7fc6ac13f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.164+0000 7fc6aaa00640 1 -- 172.21.15.45:0/1815912237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6ac0d01f0 msgr2=0x7fc6ac14e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.164+0000 7fc6aaa00640 1 --2- 172.21.15.45:0/1815912237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6ac0d01f0 0x7fc6ac14e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.164+0000 7fc6aaa00640 1 -- 172.21.15.45:0/1815912237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ac13ed50 con 0x7fc6ac145560 2024-04-16T16:22:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.165+0000 7fc6aaa00640 1 --2- 172.21.15.45:0/1815912237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6ac145560 0x7fc6ac1415a0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fc69c036040 tx=0x7fc69c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:23.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.165+0000 7fc6a3400640 1 -- 172.21.15.45:0/1815912237 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c043020 con 0x7fc6ac145560 2024-04-16T16:22:23.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.165+0000 7fc6a3400640 1 -- 172.21.15.45:0/1815912237 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc69c03e450 con 0x7fc6ac145560 2024-04-16T16:22:23.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.165+0000 7fc6a3400640 1 -- 172.21.15.45:0/1815912237 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c02ce60 con 0x7fc6ac145560 2024-04-16T16:22:23.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.165+0000 7fc6b0e00640 1 -- 172.21.15.45:0/1815912237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6ac0b4ce0 con 0x7fc6ac145560 2024-04-16T16:22:23.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.165+0000 7fc6b0e00640 1 -- 172.21.15.45:0/1815912237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ac0b52a0 con 0x7fc6ac145560 2024-04-16T16:22:23.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.166+0000 7fc6b0e00640 1 -- 172.21.15.45:0/1815912237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6ac075780 con 0x7fc6ac145560 2024-04-16T16:22:23.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.167+0000 7fc6a3400640 1 -- 172.21.15.45:0/1815912237 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 65) ==== 91142+0+0 (secure 0 0 0) 0x7fc69c046020 con 0x7fc6ac145560 2024-04-16T16:22:23.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.168+0000 7fc6a3400640 1 --2- 172.21.15.45:0/1815912237 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fc68806ca10 0x7fc68806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:23.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.168+0000 7fc6a3400640 1 -- 172.21.15.45:0/1815912237 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7fc69c0ae4b0 con 0x7fc6ac145560 2024-04-16T16:22:23.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.168+0000 7fc6ab400640 1 -- 172.21.15.45:0/1815912237 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fc68806ca10 msgr2=0x7fc68806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1506110537 2024-04-16T16:22:23.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.168+0000 7fc6ab400640 1 --2- 172.21.15.45:0/1815912237 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fc68806ca10 0x7fc68806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:22:23.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.172+0000 7fc6a3400640 1 -- 172.21.15.45:0/1815912237 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc69c04d030 con 0x7fc6ac145560 2024-04-16T16:22:23.213 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:23.213+0000 7f7e2aa2c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:22:23.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.369+0000 7fc6ab400640 1 -- 172.21.15.45:0/1815912237 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fc68806ca10 msgr2=0x7fc68806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1506110537 2024-04-16T16:22:23.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.369+0000 7fc6ab400640 1 --2- 172.21.15.45:0/1815912237 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fc68806ca10 0x7fc68806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:22:23.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.371+0000 7fc6b0e00640 1 -- 172.21.15.45:0/1815912237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc6ac06f8b0 con 0x7fc6ac145560 2024-04-16T16:22:23.371 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:23.372+0000 7f7e2aa2c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:22:23.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.372+0000 7fc6a3400640 1 -- 172.21.15.45:0/1815912237 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) ==== 100+0+11 (secure 0 0 0) 0x7fc69c07c700 con 0x7fc6ac145560 2024-04-16T16:22:23.372 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:22:23.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.375+0000 7fc6b0e00640 1 -- 172.21.15.45:0/1815912237 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fc68806ca10 msgr2=0x7fc68806eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:22:23.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.375+0000 7fc6b0e00640 1 --2- 172.21.15.45:0/1815912237 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fc68806ca10 0x7fc68806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.375+0000 7fc6b0e00640 1 -- 172.21.15.45:0/1815912237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6ac145560 msgr2=0x7fc6ac1415a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:23.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.375+0000 7fc6b0e00640 1 --2- 172.21.15.45:0/1815912237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6ac145560 0x7fc6ac1415a0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fc69c036040 tx=0x7fc69c004a40 comp rx=0 tx=0).stop 2024-04-16T16:22:23.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.375+0000 7fc6b0e00640 1 -- 172.21.15.45:0/1815912237 shutdown_connections 2024-04-16T16:22:23.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.375+0000 7fc6b0e00640 1 --2- 172.21.15.45:0/1815912237 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fc68806ca10 0x7fc68806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.375+0000 7fc6b0e00640 1 --2- 172.21.15.45:0/1815912237 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6ac15d470 0x7fc6ac13f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.375+0000 7fc6b0e00640 1 --2- 172.21.15.45:0/1815912237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6ac145560 0x7fc6ac1415a0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.375+0000 7fc6b0e00640 1 --2- 172.21.15.45:0/1815912237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6ac0d01f0 0x7fc6ac14e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.375+0000 7fc6b0e00640 1 -- 172.21.15.45:0/1815912237 >> 172.21.15.45:0/1815912237 conn(0x7fc6ac06f0c0 msgr2=0x7fc6ac1428e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:23.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.375+0000 7fc6b0e00640 1 -- 172.21.15.45:0/1815912237 shutdown_connections 2024-04-16T16:22:23.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.375+0000 7fc6b0e00640 1 -- 172.21.15.45:0/1815912237 wait complete. 2024-04-16T16:22:23.378 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:23.379+0000 7fd7fb02e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T16:22:23.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:22:23.479 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:23.480+0000 7f7e2aa2c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:22:23.529 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:23.530+0000 7f7e2aa2c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:22:23.552 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:23.552+0000 7fd7fb02e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T16:22:23.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.649+0000 7f67daa00640 1 Processor -- start 2024-04-16T16:22:23.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.650+0000 7f67daa00640 1 -- start start 2024-04-16T16:22:23.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.651+0000 7f67daa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67d4162090 0x7f67d40cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:23.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.651+0000 7f67daa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f67d40ba140 0x7f67d40cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:23.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.651+0000 7f67daa00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67d40d2600 con 0x7f67d40d5520 2024-04-16T16:22:23.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.651+0000 7f67daa00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f67d40ba8b0 con 0x7f67d40ba140 2024-04-16T16:22:23.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.651+0000 7f67daa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f67d40ba5d0 con 0x7f67d4162090 2024-04-16T16:22:23.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.652+0000 7f67da000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f67d40d5520 0x7f67d41628a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39472/0 (socket says 172.21.15.45:39472) 2024-04-16T16:22:23.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.652+0000 7f67da000640 1 -- 172.21.15.45:0/515022986 learned_addr learned my addr 172.21.15.45:0/515022986 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:23.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.652+0000 7f67d9600640 1 --2- 172.21.15.45:0/515022986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67d4162090 0x7f67d40cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:23.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.652+0000 7f67d8c00640 1 --2- 172.21.15.45:0/515022986 >> v2:172.21.15.103:3300/0 conn(0x7f67d40ba140 0x7f67d40cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:23.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.652+0000 7f67d3e00640 1 -- 172.21.15.45:0/515022986 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2716585706 0 0) 0x7f67d40d2600 con 0x7f67d40d5520 2024-04-16T16:22:23.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.652+0000 7f67d3e00640 1 -- 172.21.15.45:0/515022986 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67bc003520 con 0x7f67d40d5520 2024-04-16T16:22:23.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.652+0000 7f67d3e00640 1 -- 172.21.15.45:0/515022986 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 76582188 0 0) 0x7f67bc003520 con 0x7f67d40d5520 2024-04-16T16:22:23.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.652+0000 7f67d3e00640 1 -- 172.21.15.45:0/515022986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67d4162090 msgr2=0x7f67d40cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:23.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.652+0000 7f67d3e00640 1 --2- 172.21.15.45:0/515022986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67d4162090 0x7f67d40cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.652+0000 7f67d3e00640 1 -- 172.21.15.45:0/515022986 >> v2:172.21.15.103:3300/0 conn(0x7f67d40ba140 msgr2=0x7f67d40cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:23.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.652+0000 7f67d3e00640 1 --2- 172.21.15.45:0/515022986 >> v2:172.21.15.103:3300/0 conn(0x7f67d40ba140 0x7f67d40cfa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.652+0000 7f67d3e00640 1 -- 172.21.15.45:0/515022986 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d40d28a0 con 0x7f67d40d5520 2024-04-16T16:22:23.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.652+0000 7f67d3e00640 1 -- 172.21.15.45:0/515022986 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f67cc0019f0 con 0x7f67d40d5520 2024-04-16T16:22:23.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.653+0000 7f67d3e00640 1 -- 172.21.15.45:0/515022986 >> v1:172.21.15.45:6789/0 conn(0x7f67d40d5520 legacy=0x7f67d41628a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:23.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.653+0000 7f67d3e00640 1 --2- 172.21.15.45:0/515022986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67bc0039b0 0x7f67bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:23.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.653+0000 7f67d3e00640 1 --2- 172.21.15.45:0/515022986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67bc004800 0x7f67bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:23.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.653+0000 7f67d3e00640 1 --2- 172.21.15.45:0/515022986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67d4162090 0x7f67bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:23.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.653+0000 7f67d3e00640 1 -- 172.21.15.45:0/515022986 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f67d40ba8b0 con 0x7f67bc004800 2024-04-16T16:22:23.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.653+0000 7f67d3e00640 1 -- 172.21.15.45:0/515022986 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f67d40ba5d0 con 0x7f67bc0039b0 2024-04-16T16:22:23.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.653+0000 7f67d3e00640 1 -- 172.21.15.45:0/515022986 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f67bc00b9c0 con 0x7f67d4162090 2024-04-16T16:22:23.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.653+0000 7f67d8c00640 1 --2- 172.21.15.45:0/515022986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67bc004800 0x7f67bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:23.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.653+0000 7f67da000640 1 --2- 172.21.15.45:0/515022986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67d4162090 0x7f67bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:23.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.653+0000 7f67d8c00640 1 -- 172.21.15.45:0/515022986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67d4162090 msgr2=0x7f67bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:23.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.653+0000 7f67d8c00640 1 --2- 172.21.15.45:0/515022986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67d4162090 0x7f67bc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.653+0000 7f67d8c00640 1 -- 172.21.15.45:0/515022986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67bc0039b0 msgr2=0x7f67bc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:23.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.653+0000 7f67d9600640 1 --2- 172.21.15.45:0/515022986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67bc0039b0 0x7f67bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:23.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.653+0000 7f67d8c00640 1 --2- 172.21.15.45:0/515022986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67bc0039b0 0x7f67bc003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.653+0000 7f67d8c00640 1 -- 172.21.15.45:0/515022986 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc003520 con 0x7f67bc004800 2024-04-16T16:22:23.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.654+0000 7f67d8c00640 1 --2- 172.21.15.45:0/515022986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67bc004800 0x7f67bc004c80 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f67c0009ac0 tx=0x7f67bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35258b53132678ce server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:23.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.655+0000 7f67d3e00640 1 -- 172.21.15.45:0/515022986 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0039070 con 0x7f67bc004800 2024-04-16T16:22:23.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.655+0000 7f67d3e00640 1 -- 172.21.15.45:0/515022986 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67c002c6d0 con 0x7f67bc004800 2024-04-16T16:22:23.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.655+0000 7f67d3e00640 1 -- 172.21.15.45:0/515022986 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0009690 con 0x7f67bc004800 2024-04-16T16:22:23.657 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:23.655+0000 7f7e2aa2c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:22:23.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.655+0000 7f67daa00640 1 -- 172.21.15.45:0/515022986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67bc004800 msgr2=0x7f67bc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:23.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.655+0000 7f67daa00640 1 --2- 172.21.15.45:0/515022986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67bc004800 0x7f67bc004c80 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f67c0009ac0 tx=0x7f67bc005830 comp rx=0 tx=0).stop 2024-04-16T16:22:23.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.655+0000 7f67da000640 1 -- 172.21.15.45:0/515022986 reap_dead start 2024-04-16T16:22:23.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.655+0000 7f67daa00640 1 -- 172.21.15.45:0/515022986 shutdown_connections 2024-04-16T16:22:23.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.655+0000 7f67daa00640 1 -- 172.21.15.45:0/515022986 >> 172.21.15.45:0/515022986 conn(0x7f67d40d1940 msgr2=0x7f67d40d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:23.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.656+0000 7f67daa00640 1 -- 172.21.15.45:0/515022986 shutdown_connections 2024-04-16T16:22:23.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.656+0000 7f67daa00640 1 -- 172.21.15.45:0/515022986 wait complete. 2024-04-16T16:22:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.657+0000 7f67daa00640 1 Processor -- start 2024-04-16T16:22:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.657+0000 7f67daa00640 1 -- start start 2024-04-16T16:22:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.658+0000 7f67daa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67bc004800 0x7f67d40b6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.658+0000 7f67daa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d4153eb0 0x7f67d4159830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.658+0000 7f67daa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67d4158660 0x7f67d41568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.658+0000 7f67daa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f67d41625f0 con 0x7f67d4153eb0 2024-04-16T16:22:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.658+0000 7f67daa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f67d4162470 con 0x7f67bc004800 2024-04-16T16:22:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.658+0000 7f67daa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f67d40d40b0 con 0x7f67d4158660 2024-04-16T16:22:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.659+0000 7f67d8c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d4153eb0 0x7f67d4159830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.659+0000 7f67d8c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d4153eb0 0x7f67d4159830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53130/0 (socket says 172.21.15.45:53130) 2024-04-16T16:22:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.659+0000 7f67d8c00640 1 -- 172.21.15.45:0/4127934021 learned_addr learned my addr 172.21.15.45:0/4127934021 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.659+0000 7f67da000640 1 --2- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67d4158660 0x7f67d41568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.659+0000 7f67d9600640 1 --2- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67bc004800 0x7f67d40b6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:23.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.659+0000 7f67d8c00640 1 -- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67d4158660 msgr2=0x7f67d41568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:23.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.659+0000 7f67d8c00640 1 --2- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67d4158660 0x7f67d41568a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.659+0000 7f67d8c00640 1 -- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67bc004800 msgr2=0x7f67d40b6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:23.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.659+0000 7f67d8c00640 1 --2- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67bc004800 0x7f67d40b6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.659+0000 7f67d8c00640 1 -- 172.21.15.45:0/4127934021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d40b5f70 con 0x7f67d4153eb0 2024-04-16T16:22:23.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.659+0000 7f67d9600640 1 --2- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67bc004800 0x7f67d40b6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:23.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.659+0000 7f67d8c00640 1 --2- 172.21.15.45:0/4127934021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d4153eb0 0x7f67d4159830 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f67c0000c00 tx=0x7f67c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:23.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.660+0000 7f67d1600640 1 -- 172.21.15.45:0/4127934021 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0039050 con 0x7f67d4153eb0 2024-04-16T16:22:23.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.660+0000 7f67d1600640 1 -- 172.21.15.45:0/4127934021 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67c0045070 con 0x7f67d4153eb0 2024-04-16T16:22:23.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.660+0000 7f67d1600640 1 -- 172.21.15.45:0/4127934021 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c00044e0 con 0x7f67d4153eb0 2024-04-16T16:22:23.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.660+0000 7f67daa00640 1 -- 172.21.15.45:0/4127934021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67d40b6250 con 0x7f67d4153eb0 2024-04-16T16:22:23.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.660+0000 7f67daa00640 1 -- 172.21.15.45:0/4127934021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d40b6810 con 0x7f67d4153eb0 2024-04-16T16:22:23.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.661+0000 7f67daa00640 1 -- 172.21.15.45:0/4127934021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67d4075780 con 0x7f67d4153eb0 2024-04-16T16:22:23.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.662+0000 7f67d1600640 1 -- 172.21.15.45:0/4127934021 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 65) ==== 91142+0+0 (secure 0 0 0) 0x7f67c0003a50 con 0x7f67d4153eb0 2024-04-16T16:22:23.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.663+0000 7f67d1600640 1 --2- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f67b406ca10 0x7f67b406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:23.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.663+0000 7f67d1600640 1 -- 172.21.15.45:0/4127934021 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f67c00addc0 con 0x7f67d4153eb0 2024-04-16T16:22:23.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.663+0000 7f67d9600640 1 -- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f67b406ca10 msgr2=0x7f67b406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1506110537 2024-04-16T16:22:23.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.663+0000 7f67d9600640 1 --2- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f67b406ca10 0x7f67b406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:22:23.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.667+0000 7f67d1600640 1 -- 172.21.15.45:0/4127934021 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f67c004f020 con 0x7f67d4153eb0 2024-04-16T16:22:23.669 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:23.669+0000 7fd7fb02e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T16:22:23.723 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:23.723+0000 7fd7fb02e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T16:22:23.761 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:23.762+0000 7f7e2aa2c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:22:23.855 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:23.855+0000 7fd7fb02e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:22:23.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.864+0000 7f67d9600640 1 -- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f67b406ca10 msgr2=0x7f67b406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1506110537 2024-04-16T16:22:23.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.864+0000 7f67d9600640 1 --2- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f67b406ca10 0x7f67b406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:22:23.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.883+0000 7f67daa00640 1 -- 172.21.15.45:0/4127934021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f67d406dab0 con 0x7f67d4153eb0 2024-04-16T16:22:23.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.883+0000 7f67d1600640 1 -- 172.21.15.45:0/4127934021 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v143) ==== 80+0+477 (secure 0 0 0) 0x7f67c007c010 con 0x7f67d4153eb0 2024-04-16T16:22:23.884 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/884289453 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:23.884 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1437683872 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:23.884 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2877575799 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:23.884 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2644878831 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:23.884 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/1478535274 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:23.884 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/1478535274 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:23.884 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1111841625 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:23.884 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1114329424 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:23.884 INFO:teuthology.orchestra.run.smithi045.stderr:listed 8 entries 2024-04-16T16:22:23.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.886+0000 7f67daa00640 1 -- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f67b406ca10 msgr2=0x7f67b406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:22:23.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.886+0000 7f67daa00640 1 --2- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f67b406ca10 0x7f67b406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.886+0000 7f67daa00640 1 -- 172.21.15.45:0/4127934021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d4153eb0 msgr2=0x7f67d4159830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:23.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.886+0000 7f67daa00640 1 --2- 172.21.15.45:0/4127934021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d4153eb0 0x7f67d4159830 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f67c0000c00 tx=0x7f67c0004730 comp rx=0 tx=0).stop 2024-04-16T16:22:23.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.886+0000 7f67daa00640 1 -- 172.21.15.45:0/4127934021 shutdown_connections 2024-04-16T16:22:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.886+0000 7f67daa00640 1 --2- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f67b406ca10 0x7f67b406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.886+0000 7f67daa00640 1 --2- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67d4158660 0x7f67d41568a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.886+0000 7f67daa00640 1 --2- 172.21.15.45:0/4127934021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d4153eb0 0x7f67d4159830 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.886+0000 7f67daa00640 1 --2- 172.21.15.45:0/4127934021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67bc004800 0x7f67d40b6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.886+0000 7f67daa00640 1 -- 172.21.15.45:0/4127934021 >> 172.21.15.45:0/4127934021 conn(0x7f67d40d1940 msgr2=0x7f67d40b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.886+0000 7f67daa00640 1 -- 172.21.15.45:0/4127934021 shutdown_connections 2024-04-16T16:22:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:23.886+0000 7f67daa00640 1 -- 172.21.15.45:0/4127934021 wait complete. 2024-04-16T16:22:23.904 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-16T16:22:23.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:23.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-16T16:22:23.935 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:23.935+0000 7f7e2aa2c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:22:23.971 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:23.971+0000 7fd7fb02e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T16:22:23.984 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:23.985+0000 7f7e2aa2c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:22:24.027 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:22:24.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:24.027 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-16T16:22:24.077 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:22:24.077 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:22:24.077 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:24.115 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:22:24.115 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:24.163 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:24.164+0000 7f7e2aa2c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:22:24.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:22:24.196 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:24.194+0000 7fd7fb02e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T16:22:24.248 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:24.248+0000 7fd7fb02e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T16:22:24.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.387+0000 7f06ab600640 1 Processor -- start 2024-04-16T16:22:24.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.387+0000 7f06ab600640 1 -- start start 2024-04-16T16:22:24.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.389+0000 7f06ab600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f06a4072a90 0x7f06a40ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:24.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.389+0000 7f06ab600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06a415ea70 0x7f06a40cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:24.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.389+0000 7f06ab600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06a40cf140 con 0x7f06a41462e0 2024-04-16T16:22:24.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.389+0000 7f06ab600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f06a40cf6d0 con 0x7f06a4072a90 2024-04-16T16:22:24.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.389+0000 7f06ab600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f06a41466c0 con 0x7f06a415ea70 2024-04-16T16:22:24.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.389+0000 7f06a3e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f06a4072a90 0x7f06a40ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:24.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.389+0000 7f06a3e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f06a4072a90 0x7f06a40ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53136/0 (socket says 172.21.15.45:53136) 2024-04-16T16:22:24.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.389+0000 7f06a3e00640 1 -- 172.21.15.45:0/534443381 learned_addr learned my addr 172.21.15.45:0/534443381 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:24.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.389+0000 7f06a3e00640 1 -- 172.21.15.45:0/534443381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06a415ea70 msgr2=0x7f06a40cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:24.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.389+0000 7f06a9600640 1 --2- 172.21.15.45:0/534443381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06a415ea70 0x7f06a40cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:24.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.389+0000 7f06a3e00640 1 --2- 172.21.15.45:0/534443381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06a415ea70 0x7f06a40cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.389+0000 7f06a3e00640 1 -- 172.21.15.45:0/534443381 >> v1:172.21.15.103:6789/0 conn(0x7f06a41462e0 legacy=0x7f06a4072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:24.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.389+0000 7f06a3e00640 1 -- 172.21.15.45:0/534443381 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a40cf3f0 con 0x7f06a4072a90 2024-04-16T16:22:24.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.391+0000 7f06a3e00640 1 --2- 172.21.15.45:0/534443381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06a4072a90 0x7f06a40ce6c0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f068c009870 tx=0x7f068c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9b3eb18bf8b79e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:24.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.391+0000 7f06a3400640 1 -- 172.21.15.45:0/534443381 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f068c0317c0 con 0x7f06a4072a90 2024-04-16T16:22:24.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.391+0000 7f06a3400640 1 -- 172.21.15.45:0/534443381 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f068c037070 con 0x7f06a4072a90 2024-04-16T16:22:24.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.391+0000 7f06a3400640 1 -- 172.21.15.45:0/534443381 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f068c031b80 con 0x7f06a4072a90 2024-04-16T16:22:24.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.392+0000 7f06ab600640 1 -- 172.21.15.45:0/534443381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06a4072a90 msgr2=0x7f06a40ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:24.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.392+0000 7f06ab600640 1 --2- 172.21.15.45:0/534443381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06a4072a90 0x7f06a40ce6c0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f068c009870 tx=0x7f068c02b860 comp rx=0 tx=0).stop 2024-04-16T16:22:24.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.392+0000 7f06ab600640 1 -- 172.21.15.45:0/534443381 shutdown_connections 2024-04-16T16:22:24.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.392+0000 7f06ab600640 1 --2- 172.21.15.45:0/534443381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06a415ea70 0x7f06a40cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.392+0000 7f06ab600640 1 --2- 172.21.15.45:0/534443381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06a4072a90 0x7f06a40ce6c0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.392+0000 7f06ab600640 1 -- 172.21.15.45:0/534443381 >> 172.21.15.45:0/534443381 conn(0x7f06a40b2750 msgr2=0x7f06a40b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:24.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.392+0000 7f06ab600640 1 -- 172.21.15.45:0/534443381 shutdown_connections 2024-04-16T16:22:24.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.392+0000 7f06ab600640 1 -- 172.21.15.45:0/534443381 wait complete. 2024-04-16T16:22:24.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.393+0000 7f06ab600640 1 Processor -- start 2024-04-16T16:22:24.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.394+0000 7f06ab600640 1 -- start start 2024-04-16T16:22:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.394+0000 7f06ab600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06a4072a90 0x7f06a413da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.394+0000 7f06ab600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06a41462e0 0x7f06a415bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.394+0000 7f06ab600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06a415ea70 0x7f06a40b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.395+0000 7f06ab600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f06a415d540 con 0x7f06a415ea70 2024-04-16T16:22:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.395+0000 7f06ab600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f06a415d260 con 0x7f06a4072a90 2024-04-16T16:22:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.395+0000 7f06ab600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f06a4166490 con 0x7f06a41462e0 2024-04-16T16:22:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.395+0000 7f06a9600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06a415ea70 0x7f06a40b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.395+0000 7f06a9600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06a415ea70 0x7f06a40b5000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53138/0 (socket says 172.21.15.45:53138) 2024-04-16T16:22:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.395+0000 7f06a9600640 1 -- 172.21.15.45:0/1158470104 learned_addr learned my addr 172.21.15.45:0/1158470104 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.395+0000 7f06a3e00640 1 --2- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06a41462e0 0x7f06a415bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.395+0000 7f06a8c00640 1 --2- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06a4072a90 0x7f06a413da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:24.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.395+0000 7f06a9600640 1 -- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06a41462e0 msgr2=0x7f06a415bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:24.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.395+0000 7f06a9600640 1 --2- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06a41462e0 0x7f06a415bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.395+0000 7f06a9600640 1 -- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06a4072a90 msgr2=0x7f06a413da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:24.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.395+0000 7f06a9600640 1 --2- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06a4072a90 0x7f06a413da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.395+0000 7f06a9600640 1 -- 172.21.15.45:0/1158470104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a415c200 con 0x7f06a415ea70 2024-04-16T16:22:24.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.396+0000 7f06a9600640 1 --2- 172.21.15.45:0/1158470104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06a415ea70 0x7f06a40b5000 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f069c00c580 tx=0x7f069c00ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:24.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.396+0000 7f06a0c00640 1 -- 172.21.15.45:0/1158470104 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c015070 con 0x7f06a415ea70 2024-04-16T16:22:24.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.396+0000 7f06a0c00640 1 -- 172.21.15.45:0/1158470104 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f069c004220 con 0x7f06a415ea70 2024-04-16T16:22:24.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.396+0000 7f06ab600640 1 -- 172.21.15.45:0/1158470104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a40b5540 con 0x7f06a415ea70 2024-04-16T16:22:24.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.396+0000 7f06a0c00640 1 -- 172.21.15.45:0/1158470104 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c003610 con 0x7f06a415ea70 2024-04-16T16:22:24.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.396+0000 7f06ab600640 1 -- 172.21.15.45:0/1158470104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a40b5aa0 con 0x7f06a415ea70 2024-04-16T16:22:24.398 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:24.396+0000 7fd7fb02e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T16:22:24.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.397+0000 7f06ab600640 1 -- 172.21.15.45:0/1158470104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06a4075780 con 0x7f06a415ea70 2024-04-16T16:22:24.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.399+0000 7f06a0c00640 1 -- 172.21.15.45:0/1158470104 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 65) ==== 91142+0+0 (secure 0 0 0) 0x7f069c0037b0 con 0x7f06a415ea70 2024-04-16T16:22:24.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.399+0000 7f06a0c00640 1 --2- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f068006ca10 0x7f068006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:24.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.400+0000 7f06a0c00640 1 -- 172.21.15.45:0/1158470104 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f069c08bd30 con 0x7f06a415ea70 2024-04-16T16:22:24.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.400+0000 7f06a8c00640 1 -- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f068006ca10 msgr2=0x7f068006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1506110537 2024-04-16T16:22:24.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.400+0000 7f06a8c00640 1 --2- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f068006ca10 0x7f068006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:22:24.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.404+0000 7f06a0c00640 1 -- 172.21.15.45:0/1158470104 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06a40b5540 con 0x7f06a415ea70 2024-04-16T16:22:24.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.601+0000 7f06a8c00640 1 -- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f068006ca10 msgr2=0x7f068006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1506110537 2024-04-16T16:22:24.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.601+0000 7f06a8c00640 1 --2- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f068006ca10 0x7f068006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:22:24.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.611+0000 7f06ab600640 1 -- 172.21.15.45:0/1158470104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06a4144a60 con 0x7f06a415ea70 2024-04-16T16:22:24.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.613+0000 7f06a0c00640 1 -- 172.21.15.45:0/1158470104 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) ==== 74+0+17187 (secure 0 0 0) 0x7f069c059df0 con 0x7f06a415ea70 2024-04-16T16:22:24.614 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:24.614 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":143,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:21:47.933077+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:21:23.973149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-16T16:21:25.043255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/884289453":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/1437683872":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/2877575799":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/2644878831":"2024-04-17T16:21:47.933003+0000","172.21.15.103:6833/1478535274":"2024-04-17T16:21:47.933003+0000","172.21.15.103:6832/1478535274":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/1111841625":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/1114329424":"2024-04-17T16:21:47.933003+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:22:24.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.616+0000 7f06ab600640 1 -- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f068006ca10 msgr2=0x7f068006eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:22:24.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.616+0000 7f06ab600640 1 --2- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f068006ca10 0x7f068006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.616+0000 7f06ab600640 1 -- 172.21.15.45:0/1158470104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06a415ea70 msgr2=0x7f06a40b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:24.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.616+0000 7f06ab600640 1 --2- 172.21.15.45:0/1158470104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06a415ea70 0x7f06a40b5000 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f069c00c580 tx=0x7f069c00ca40 comp rx=0 tx=0).stop 2024-04-16T16:22:24.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.616+0000 7f06ab600640 1 -- 172.21.15.45:0/1158470104 shutdown_connections 2024-04-16T16:22:24.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.616+0000 7f06ab600640 1 --2- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f068006ca10 0x7f068006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.616+0000 7f06ab600640 1 --2- 172.21.15.45:0/1158470104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06a415ea70 0x7f06a40b5000 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.616+0000 7f06ab600640 1 --2- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06a41462e0 0x7f06a415bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.616+0000 7f06ab600640 1 --2- 172.21.15.45:0/1158470104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06a4072a90 0x7f06a413da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.616+0000 7f06ab600640 1 -- 172.21.15.45:0/1158470104 >> 172.21.15.45:0/1158470104 conn(0x7f06a40b2750 msgr2=0x7f06a40b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:24.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.616+0000 7f06ab600640 1 -- 172.21.15.45:0/1158470104 shutdown_connections 2024-04-16T16:22:24.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.616+0000 7f06ab600640 1 -- 172.21.15.45:0/1158470104 wait complete. 2024-04-16T16:22:24.636 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:21:23.973149+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:21:25.043255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:22:24.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:22:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.851+0000 7fd9bde00640 1 Processor -- start 2024-04-16T16:22:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.851+0000 7fd9bde00640 1 -- start start 2024-04-16T16:22:24.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.852+0000 7fd9bde00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd9b815eca0 0x7fd9b80b1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.852+0000 7fd9bde00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9b8067c20 0x7fd9b806b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.852+0000 7fd9bde00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9b80b21b0 con 0x7fd9b80b2ee0 2024-04-16T16:22:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.853+0000 7fd9bde00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd9b815f080 con 0x7fd9b815eca0 2024-04-16T16:22:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.853+0000 7fd9bde00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd9b80af500 con 0x7fd9b8067c20 2024-04-16T16:22:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.853+0000 7fd9b7400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd9b80b2ee0 0x7fd9b80af990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39486/0 (socket says 172.21.15.45:39486) 2024-04-16T16:22:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.853+0000 7fd9b7400640 1 -- 172.21.15.45:0/2986346273 learned_addr learned my addr 172.21.15.45:0/2986346273 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.853+0000 7fd9b6000640 1 --2- 172.21.15.45:0/2986346273 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9b8067c20 0x7fd9b806b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.853+0000 7fd9b6a00640 1 --2- 172.21.15.45:0/2986346273 >> v2:172.21.15.103:3300/0 conn(0x7fd9b815eca0 0x7fd9b80b1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.853+0000 7fd9b5600640 1 -- 172.21.15.45:0/2986346273 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1570235784 0 0) 0x7fd9b80b21b0 con 0x7fd9b80b2ee0 2024-04-16T16:22:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.853+0000 7fd9b5600640 1 -- 172.21.15.45:0/2986346273 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd99c003520 con 0x7fd9b80b2ee0 2024-04-16T16:22:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.854+0000 7fd9b5600640 1 -- 172.21.15.45:0/2986346273 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2647414286 0 0) 0x7fd99c003520 con 0x7fd9b80b2ee0 2024-04-16T16:22:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.854+0000 7fd9b5600640 1 -- 172.21.15.45:0/2986346273 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9b8067c20 msgr2=0x7fd9b806b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.854+0000 7fd9b5600640 1 --2- 172.21.15.45:0/2986346273 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9b8067c20 0x7fd9b806b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.854+0000 7fd9b5600640 1 -- 172.21.15.45:0/2986346273 >> v2:172.21.15.103:3300/0 conn(0x7fd9b815eca0 msgr2=0x7fd9b80b1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.854+0000 7fd9b5600640 1 --2- 172.21.15.45:0/2986346273 >> v2:172.21.15.103:3300/0 conn(0x7fd9b815eca0 0x7fd9b80b1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.854+0000 7fd9b5600640 1 -- 172.21.15.45:0/2986346273 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b80aef10 con 0x7fd9b80b2ee0 2024-04-16T16:22:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.854+0000 7fd9b5600640 1 -- 172.21.15.45:0/2986346273 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fd9b0001670 con 0x7fd9b80b2ee0 2024-04-16T16:22:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.854+0000 7fd9b5600640 1 -- 172.21.15.45:0/2986346273 >> v1:172.21.15.45:6789/0 conn(0x7fd9b80b2ee0 legacy=0x7fd9b80af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.854+0000 7fd9b5600640 1 --2- 172.21.15.45:0/2986346273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c0039b0 0x7fd99c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.854+0000 7fd9b5600640 1 --2- 172.21.15.45:0/2986346273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd99c004800 0x7fd99c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.854+0000 7fd9b5600640 1 --2- 172.21.15.45:0/2986346273 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9b8067c20 0x7fd99c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.854+0000 7fd9b5600640 1 -- 172.21.15.45:0/2986346273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b815f080 con 0x7fd99c004800 2024-04-16T16:22:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.854+0000 7fd9b5600640 1 -- 172.21.15.45:0/2986346273 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b80af500 con 0x7fd99c0039b0 2024-04-16T16:22:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.854+0000 7fd9b5600640 1 -- 172.21.15.45:0/2986346273 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd99c00b9c0 con 0x7fd9b8067c20 2024-04-16T16:22:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.854+0000 7fd9b6000640 1 --2- 172.21.15.45:0/2986346273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd99c004800 0x7fd99c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.855+0000 7fd9b7400640 1 --2- 172.21.15.45:0/2986346273 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9b8067c20 0x7fd99c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.855+0000 7fd9b6a00640 1 --2- 172.21.15.45:0/2986346273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c0039b0 0x7fd99c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.855+0000 7fd9b6000640 1 -- 172.21.15.45:0/2986346273 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9b8067c20 msgr2=0x7fd99c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.855+0000 7fd9b6000640 1 --2- 172.21.15.45:0/2986346273 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9b8067c20 0x7fd99c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.855+0000 7fd9b6000640 1 -- 172.21.15.45:0/2986346273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c0039b0 msgr2=0x7fd99c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.855+0000 7fd9b6000640 1 --2- 172.21.15.45:0/2986346273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c0039b0 0x7fd99c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.855+0000 7fd9b6000640 1 -- 172.21.15.45:0/2986346273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99c003520 con 0x7fd99c004800 2024-04-16T16:22:24.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.855+0000 7fd9b7400640 1 --2- 172.21.15.45:0/2986346273 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9b8067c20 0x7fd99c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.856+0000 7fd9b6000640 1 --2- 172.21.15.45:0/2986346273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd99c004800 0x7fd99c004c80 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0009ac0 tx=0x7fd99c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c00b737f5f80328 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.856+0000 7fd9b5600640 1 -- 172.21.15.45:0/2986346273 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0039070 con 0x7fd99c004800 2024-04-16T16:22:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.856+0000 7fd9b5600640 1 -- 172.21.15.45:0/2986346273 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9a002c6b0 con 0x7fd99c004800 2024-04-16T16:22:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.856+0000 7fd9b5600640 1 -- 172.21.15.45:0/2986346273 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0009690 con 0x7fd99c004800 2024-04-16T16:22:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.857+0000 7fd9bde00640 1 -- 172.21.15.45:0/2986346273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd99c004800 msgr2=0x7fd99c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.857+0000 7fd9bde00640 1 --2- 172.21.15.45:0/2986346273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd99c004800 0x7fd99c004c80 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0009ac0 tx=0x7fd99c005830 comp rx=0 tx=0).stop 2024-04-16T16:22:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.857+0000 7fd9b7400640 1 -- 172.21.15.45:0/2986346273 reap_dead start 2024-04-16T16:22:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.857+0000 7fd9bde00640 1 -- 172.21.15.45:0/2986346273 shutdown_connections 2024-04-16T16:22:24.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.857+0000 7fd9bde00640 1 -- 172.21.15.45:0/2986346273 >> 172.21.15.45:0/2986346273 conn(0x7fd9b809aa70 msgr2=0x7fd9b80b6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:24.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.857+0000 7fd9bde00640 1 -- 172.21.15.45:0/2986346273 shutdown_connections 2024-04-16T16:22:24.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.857+0000 7fd9bde00640 1 -- 172.21.15.45:0/2986346273 wait complete. 2024-04-16T16:22:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.859+0000 7fd9bde00640 1 Processor -- start 2024-04-16T16:22:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.859+0000 7fd9bde00640 1 -- start start 2024-04-16T16:22:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.860+0000 7fd9bde00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c004800 0x7fd9b814eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.860+0000 7fd9bde00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9b809bd30 0x7fd9b809c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.860+0000 7fd9bde00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9b81e8000 0x7fd9b81e8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.860+0000 7fd9bde00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b80b2450 con 0x7fd9b81e8000 2024-04-16T16:22:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.860+0000 7fd9bde00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b806d470 con 0x7fd99c004800 2024-04-16T16:22:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.860+0000 7fd9bde00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd9b80b25d0 con 0x7fd9b809bd30 2024-04-16T16:22:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.860+0000 7fd9b7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9b81e8000 0x7fd9b81e8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.860+0000 7fd9b7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9b81e8000 0x7fd9b81e8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53150/0 (socket says 172.21.15.45:53150) 2024-04-16T16:22:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.860+0000 7fd9b7400640 1 -- 172.21.15.45:0/1661465948 learned_addr learned my addr 172.21.15.45:0/1661465948 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.860+0000 7fd9b6000640 1 --2- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9b809bd30 0x7fd9b809c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.861+0000 7fd9b6a00640 1 --2- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c004800 0x7fd9b814eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.861+0000 7fd9b7400640 1 -- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9b809bd30 msgr2=0x7fd9b809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.861+0000 7fd9b7400640 1 --2- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9b809bd30 0x7fd9b809c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.861+0000 7fd9b7400640 1 -- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c004800 msgr2=0x7fd9b814eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.861+0000 7fd9b7400640 1 --2- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c004800 0x7fd9b814eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.861+0000 7fd9b7400640 1 -- 172.21.15.45:0/1661465948 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b80ca710 con 0x7fd9b81e8000 2024-04-16T16:22:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.861+0000 7fd9b7400640 1 --2- 172.21.15.45:0/1661465948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9b81e8000 0x7fd9b81e8490 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fd9b806e070 tx=0x7fd9b000d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.861+0000 7fd9af400640 1 -- 172.21.15.45:0/1661465948 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b00074e0 con 0x7fd9b81e8000 2024-04-16T16:22:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.861+0000 7fd9af400640 1 -- 172.21.15.45:0/1661465948 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9b00047e0 con 0x7fd9b81e8000 2024-04-16T16:22:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.861+0000 7fd9bde00640 1 -- 172.21.15.45:0/1661465948 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b80ca9f0 con 0x7fd9b81e8000 2024-04-16T16:22:24.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.861+0000 7fd9af400640 1 -- 172.21.15.45:0/1661465948 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0004bd0 con 0x7fd9b81e8000 2024-04-16T16:22:24.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.861+0000 7fd9bde00640 1 -- 172.21.15.45:0/1661465948 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b80cafb0 con 0x7fd9b81e8000 2024-04-16T16:22:24.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.862+0000 7fd9bde00640 1 -- 172.21.15.45:0/1661465948 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9b8075780 con 0x7fd9b81e8000 2024-04-16T16:22:24.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.864+0000 7fd9af400640 1 -- 172.21.15.45:0/1661465948 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 65) ==== 91142+0+0 (secure 0 0 0) 0x7fd9b00133d0 con 0x7fd9b81e8000 2024-04-16T16:22:24.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.865+0000 7fd9af400640 1 --2- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd99406ca10 0x7fd99406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:24.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.865+0000 7fd9af400640 1 -- 172.21.15.45:0/1661465948 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7fd9b008bba0 con 0x7fd9b81e8000 2024-04-16T16:22:24.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.866+0000 7fd9b6a00640 1 -- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd99406ca10 msgr2=0x7fd99406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1506110537 2024-04-16T16:22:24.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.866+0000 7fd9b6a00640 1 --2- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd99406ca10 0x7fd99406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:22:24.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:24.870+0000 7fd9af400640 1 -- 172.21.15.45:0/1661465948 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9b001a060 con 0x7fd9b81e8000 2024-04-16T16:22:25.001 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:25.002+0000 7f7e2aa2c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:22:25.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.067+0000 7fd9b6a00640 1 -- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd99406ca10 msgr2=0x7fd99406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1506110537 2024-04-16T16:22:25.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.067+0000 7fd9b6a00640 1 --2- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd99406ca10 0x7fd99406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T16:22:25.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.123+0000 7fd9bde00640 1 -- 172.21.15.45:0/1661465948 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd9b80cb180 con 0x7fd9b81e8000 2024-04-16T16:22:25.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.124+0000 7fd9af400640 1 -- 172.21.15.45:0/1661465948 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) ==== 93+0+10 (secure 0 0 0) 0x7fd9b0059c40 con 0x7fd9b81e8000 2024-04-16T16:22:25.124 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:22:25.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.127+0000 7fd9bde00640 1 -- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd99406ca10 msgr2=0x7fd99406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:22:25.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.127+0000 7fd9bde00640 1 --2- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd99406ca10 0x7fd99406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.127+0000 7fd9bde00640 1 -- 172.21.15.45:0/1661465948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9b81e8000 msgr2=0x7fd9b81e8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:25.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.127+0000 7fd9bde00640 1 --2- 172.21.15.45:0/1661465948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9b81e8000 0x7fd9b81e8490 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fd9b806e070 tx=0x7fd9b000d950 comp rx=0 tx=0).stop 2024-04-16T16:22:25.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.127+0000 7fd9bde00640 1 -- 172.21.15.45:0/1661465948 shutdown_connections 2024-04-16T16:22:25.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.127+0000 7fd9bde00640 1 --2- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7fd99406ca10 0x7fd99406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.127+0000 7fd9bde00640 1 --2- 172.21.15.45:0/1661465948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9b81e8000 0x7fd9b81e8490 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.127+0000 7fd9bde00640 1 --2- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd9b809bd30 0x7fd9b809c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.127+0000 7fd9bde00640 1 --2- 172.21.15.45:0/1661465948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c004800 0x7fd9b814eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.127+0000 7fd9bde00640 1 -- 172.21.15.45:0/1661465948 >> 172.21.15.45:0/1661465948 conn(0x7fd9b809aa70 msgr2=0x7fd9b80af890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:25.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.127+0000 7fd9bde00640 1 -- 172.21.15.45:0/1661465948 shutdown_connections 2024-04-16T16:22:25.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.127+0000 7fd9bde00640 1 -- 172.21.15.45:0/1661465948 wait complete. 2024-04-16T16:22:25.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:22:25.307 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:25.308+0000 7f7e2aa2c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:22:25.331 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:25.331+0000 7fd7fb02e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T16:22:25.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.363+0000 7f2359c00640 1 Processor -- start 2024-04-16T16:22:25.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.364+0000 7f2359c00640 1 -- start start 2024-04-16T16:22:25.364 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:25.365+0000 7f7e2aa2c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:22:25.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.366+0000 7f2359c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f23540d1030 0x7f235415f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:25.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.366+0000 7f2359c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f235415fda0 0x7f235406e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:25.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.366+0000 7f2359c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23540b5e10 con 0x7f23540b5a30 2024-04-16T16:22:25.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.366+0000 7f2359c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f235415f3d0 con 0x7f23540d1030 2024-04-16T16:22:25.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.366+0000 7f2359c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2354162e70 con 0x7f235415fda0 2024-04-16T16:22:25.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.366+0000 7f234be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f23540d1030 0x7f235415f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:25.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.366+0000 7f234be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f23540d1030 0x7f235415f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53158/0 (socket says 172.21.15.45:53158) 2024-04-16T16:22:25.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.366+0000 7f234be00640 1 -- 172.21.15.45:0/1806339299 learned_addr learned my addr 172.21.15.45:0/1806339299 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:25.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.366+0000 7f2353400640 1 --2- 172.21.15.45:0/1806339299 >> v2:172.21.15.103:3300/0 conn(0x7f235415fda0 0x7f235406e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:25.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.367+0000 7f234be00640 1 -- 172.21.15.45:0/1806339299 >> v2:172.21.15.103:3300/0 conn(0x7f235415fda0 msgr2=0x7f235406e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:25.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.367+0000 7f234be00640 1 --2- 172.21.15.45:0/1806339299 >> v2:172.21.15.103:3300/0 conn(0x7f235415fda0 0x7f235406e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.367+0000 7f234be00640 1 -- 172.21.15.45:0/1806339299 >> v1:172.21.15.103:6789/0 conn(0x7f23540b5a30 legacy=0x7f23541625e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:25.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.367+0000 7f234be00640 1 -- 172.21.15.45:0/1806339299 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2354163150 con 0x7f23540d1030 2024-04-16T16:22:25.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.367+0000 7f2353400640 1 --2- 172.21.15.45:0/1806339299 >> v2:172.21.15.103:3300/0 conn(0x7f235415fda0 0x7f235406e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:22:25.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.368+0000 7f234be00640 1 --2- 172.21.15.45:0/1806339299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23540d1030 0x7f235415f860 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f23540b3900 tx=0x7f233c02b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c2c54b37430fa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:25.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.368+0000 7f2352000640 1 -- 172.21.15.45:0/1806339299 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c034070 con 0x7f23540d1030 2024-04-16T16:22:25.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.369+0000 7f2352000640 1 -- 172.21.15.45:0/1806339299 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f233c030880 con 0x7f23540d1030 2024-04-16T16:22:25.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.369+0000 7f2352000640 1 -- 172.21.15.45:0/1806339299 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c030c50 con 0x7f23540d1030 2024-04-16T16:22:25.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.369+0000 7f2359c00640 1 -- 172.21.15.45:0/1806339299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23540d1030 msgr2=0x7f235415f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:25.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.369+0000 7f2359c00640 1 --2- 172.21.15.45:0/1806339299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23540d1030 0x7f235415f860 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f23540b3900 tx=0x7f233c02b9c0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.369+0000 7f2359c00640 1 -- 172.21.15.45:0/1806339299 shutdown_connections 2024-04-16T16:22:25.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.369+0000 7f2359c00640 1 --2- 172.21.15.45:0/1806339299 >> v2:172.21.15.103:3300/0 conn(0x7f235415fda0 0x7f235406e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.369+0000 7f2359c00640 1 --2- 172.21.15.45:0/1806339299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23540d1030 0x7f235415f860 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.369+0000 7f2359c00640 1 -- 172.21.15.45:0/1806339299 >> 172.21.15.45:0/1806339299 conn(0x7f23540b3970 msgr2=0x7f23540b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:25.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.370+0000 7f2359c00640 1 -- 172.21.15.45:0/1806339299 shutdown_connections 2024-04-16T16:22:25.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.370+0000 7f2359c00640 1 -- 172.21.15.45:0/1806339299 wait complete. 2024-04-16T16:22:25.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.371+0000 7f2359c00640 1 Processor -- start 2024-04-16T16:22:25.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.371+0000 7f2359c00640 1 -- start start 2024-04-16T16:22:25.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.372+0000 7f2359c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23540b5a30 0x7f235414ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:25.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.372+0000 7f2359c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23540d1030 0x7f2354156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:25.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.372+0000 7f2359c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23541e9b00 0x7f2354154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:25.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.372+0000 7f2359c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f235406e7d0 con 0x7f23540b5a30 2024-04-16T16:22:25.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.372+0000 7f2359c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f235415eff0 con 0x7f23540d1030 2024-04-16T16:22:25.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.372+0000 7f2359c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f235406e950 con 0x7f23541e9b00 2024-04-16T16:22:25.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.373+0000 7f2352a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23540b5a30 0x7f235414ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:25.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.373+0000 7f2352a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23540b5a30 0x7f235414ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53172/0 (socket says 172.21.15.45:53172) 2024-04-16T16:22:25.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.373+0000 7f2352a00640 1 -- 172.21.15.45:0/1500066723 learned_addr learned my addr 172.21.15.45:0/1500066723 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:25.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.373+0000 7f234be00640 1 --2- 172.21.15.45:0/1500066723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23540d1030 0x7f2354156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:25.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.373+0000 7f2352a00640 1 -- 172.21.15.45:0/1500066723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23541e9b00 msgr2=0x7f2354154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:25.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.373+0000 7f2352a00640 1 --2- 172.21.15.45:0/1500066723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23541e9b00 0x7f2354154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.373+0000 7f2352a00640 1 -- 172.21.15.45:0/1500066723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23540d1030 msgr2=0x7f2354156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:25.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.373+0000 7f2352a00640 1 --2- 172.21.15.45:0/1500066723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23540d1030 0x7f2354156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.373+0000 7f2352a00640 1 -- 172.21.15.45:0/1500066723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235414f220 con 0x7f23540b5a30 2024-04-16T16:22:25.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.374+0000 7f2352a00640 1 --2- 172.21.15.45:0/1500066723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23540b5a30 0x7f235414ece0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f234400c450 tx=0x7f234400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:25.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.374+0000 7f234aa00640 1 -- 172.21.15.45:0/1500066723 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23440052d0 con 0x7f23540b5a30 2024-04-16T16:22:25.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.374+0000 7f234aa00640 1 -- 172.21.15.45:0/1500066723 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2344005470 con 0x7f23540b5a30 2024-04-16T16:22:25.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.374+0000 7f2359c00640 1 -- 172.21.15.45:0/1500066723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23540cd2c0 con 0x7f23540b5a30 2024-04-16T16:22:25.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.374+0000 7f234aa00640 1 -- 172.21.15.45:0/1500066723 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2344012870 con 0x7f23540b5a30 2024-04-16T16:22:25.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.374+0000 7f2359c00640 1 -- 172.21.15.45:0/1500066723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23540cdb60 con 0x7f23540b5a30 2024-04-16T16:22:25.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.376+0000 7f2359c00640 1 -- 172.21.15.45:0/1500066723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2354075780 con 0x7f23540b5a30 2024-04-16T16:22:25.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.376+0000 7f234aa00640 1 -- 172.21.15.45:0/1500066723 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 65) ==== 91142+0+0 (secure 0 0 0) 0x7f2344012a10 con 0x7f23540b5a30 2024-04-16T16:22:25.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.377+0000 7f234aa00640 1 --2- 172.21.15.45:0/1500066723 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f233006ca40 0x7f233006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:25.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.377+0000 7f234aa00640 1 -- 172.21.15.45:0/1500066723 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6053+0+0 (secure 0 0 0) 0x7f234408b990 con 0x7f23540b5a30 2024-04-16T16:22:25.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.377+0000 7f234be00640 1 -- 172.21.15.45:0/1500066723 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f233006ca40 msgr2=0x7f233006ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6832/1506110537 2024-04-16T16:22:25.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.377+0000 7f234be00640 1 --2- 172.21.15.45:0/1500066723 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f233006ca40 0x7f233006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T16:22:25.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.381+0000 7f234aa00640 1 -- 172.21.15.45:0/1500066723 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2344059b40 con 0x7f23540b5a30 2024-04-16T16:22:25.497 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:25.497+0000 7f7e2aa2c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:22:25.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.515+0000 7f234aa00640 1 -- 172.21.15.45:0/1500066723 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 66) ==== 90751+0+0 (secure 0 0 0) 0x7f2344012d00 con 0x7f23540b5a30 2024-04-16T16:22:25.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.516+0000 7f234aa00640 1 -- 172.21.15.45:0/1500066723 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f233006ca40 msgr2=0x7f233006ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T16:22:25.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.516+0000 7f234aa00640 1 --2- 172.21.15.45:0/1500066723 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f233006ca40 0x7f233006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:22:25.579 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:22:25.591 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:25.592+0000 7f7da4000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:22:25.591 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:25.592+0000 7f7da4000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:22:25.591 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:25.592+0000 7f7da4000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:22:25.591 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:25.592+0000 7f7da4000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:22:25.591 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:25.592+0000 7f7da4000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:22:25.592 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:25.593+0000 7f7da7200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:22:25.592 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:25.593+0000 7f7da7200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:22:25.592 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:25.593+0000 7f7da7200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:22:25.592 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:25.593+0000 7f7da7200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:22:25.592 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-16T16:22:25.593+0000 7f7da7200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-16T16:22:25.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.618+0000 7f2359c00640 1 -- 172.21.15.45:0/1500066723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2354161360 con 0x7f23540b5a30 2024-04-16T16:22:25.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.619+0000 7f234aa00640 1 -- 172.21.15.45:0/1500066723 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v144) ==== 104+0+11 (secure 0 0 0) 0x7f2344079200 con 0x7f23540b5a30 2024-04-16T16:22:25.618 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:22:25.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.621+0000 7f2359c00640 1 -- 172.21.15.45:0/1500066723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23540b5a30 msgr2=0x7f235414ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:25.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.621+0000 7f2359c00640 1 --2- 172.21.15.45:0/1500066723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23540b5a30 0x7f235414ece0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f234400c450 tx=0x7f234400c910 comp rx=0 tx=0).stop 2024-04-16T16:22:25.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.621+0000 7f2359c00640 1 -- 172.21.15.45:0/1500066723 shutdown_connections 2024-04-16T16:22:25.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.621+0000 7f2359c00640 1 --2- 172.21.15.45:0/1500066723 >> [v2:172.21.15.103:6832/1506110537,v1:172.21.15.103:6833/1506110537] conn(0x7f233006ca40 0x7f233006ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.621+0000 7f2359c00640 1 --2- 172.21.15.45:0/1500066723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23541e9b00 0x7f2354154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.621+0000 7f2359c00640 1 --2- 172.21.15.45:0/1500066723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23540d1030 0x7f2354156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.621+0000 7f2359c00640 1 --2- 172.21.15.45:0/1500066723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23540b5a30 0x7f235414ece0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.621+0000 7f2359c00640 1 -- 172.21.15.45:0/1500066723 >> 172.21.15.45:0/1500066723 conn(0x7f23540b3970 msgr2=0x7f23540adac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:25.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.621+0000 7f2359c00640 1 -- 172.21.15.45:0/1500066723 shutdown_connections 2024-04-16T16:22:25.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.621+0000 7f2359c00640 1 -- 172.21.15.45:0/1500066723 wait complete. 2024-04-16T16:22:25.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:22:25.699 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:25.699+0000 7fd7fb02e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T16:22:25.761 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:25.761+0000 7fd7fb02e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T16:22:25.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.866+0000 7fcd7ea00640 1 Processor -- start 2024-04-16T16:22:25.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.867+0000 7fcd7ea00640 1 -- start start 2024-04-16T16:22:25.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.868+0000 7fcd7ea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd780b6310 0x7fcd78161db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:25.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.868+0000 7fcd7ea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcd780d3e90 0x7fcd781622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:25.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.868+0000 7fcd7ea00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd780b6810 con 0x7fcd780d18f0 2024-04-16T16:22:25.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.868+0000 7fcd7ea00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fcd780d29f0 con 0x7fcd780d3e90 2024-04-16T16:22:25.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.868+0000 7fcd7ea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcd780b6af0 con 0x7fcd780b6310 2024-04-16T16:22:25.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.869+0000 7fcd7d600640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcd780d18f0 0x7fcd780ba220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39494/0 (socket says 172.21.15.45:39494) 2024-04-16T16:22:25.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.869+0000 7fcd7d600640 1 -- 172.21.15.45:0/2233878955 learned_addr learned my addr 172.21.15.45:0/2233878955 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:25.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.869+0000 7fcd7e000640 1 --2- 172.21.15.45:0/2233878955 >> v2:172.21.15.45:3300/0 conn(0x7fcd780d3e90 0x7fcd781622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:25.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.869+0000 7fcd7cc00640 1 --2- 172.21.15.45:0/2233878955 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd780b6310 0x7fcd78161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:25.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.869+0000 7fcd77e00640 1 -- 172.21.15.45:0/2233878955 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3513862543 0 0) 0x7fcd780b6810 con 0x7fcd780d18f0 2024-04-16T16:22:25.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.869+0000 7fcd77e00640 1 -- 172.21.15.45:0/2233878955 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd60003520 con 0x7fcd780d18f0 2024-04-16T16:22:25.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.869+0000 7fcd7e000640 1 -- 172.21.15.45:0/2233878955 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd780b6310 msgr2=0x7fcd78161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:25.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.869+0000 7fcd7e000640 1 --2- 172.21.15.45:0/2233878955 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd780b6310 0x7fcd78161db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.869+0000 7fcd7e000640 1 -- 172.21.15.45:0/2233878955 >> v1:172.21.15.45:6789/0 conn(0x7fcd780d18f0 legacy=0x7fcd780ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:25.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.869+0000 7fcd7e000640 1 -- 172.21.15.45:0/2233878955 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd780b6dd0 con 0x7fcd780d3e90 2024-04-16T16:22:25.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.870+0000 7fcd7e000640 1 --2- 172.21.15.45:0/2233878955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd780d3e90 0x7fcd781622f0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009d90 tx=0x7fcd7002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3764d95ea1e8d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:25.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.871+0000 7fcd77e00640 1 -- 172.21.15.45:0/2233878955 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70039070 con 0x7fcd780d3e90 2024-04-16T16:22:25.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.871+0000 7fcd77e00640 1 -- 172.21.15.45:0/2233878955 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd7002cbd0 con 0x7fcd780d3e90 2024-04-16T16:22:25.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.871+0000 7fcd77e00640 1 -- 172.21.15.45:0/2233878955 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70034440 con 0x7fcd780d3e90 2024-04-16T16:22:25.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.871+0000 7fcd7ea00640 1 -- 172.21.15.45:0/2233878955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd780d3e90 msgr2=0x7fcd781622f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:25.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.871+0000 7fcd7ea00640 1 --2- 172.21.15.45:0/2233878955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd780d3e90 0x7fcd781622f0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009d90 tx=0x7fcd7002be20 comp rx=0 tx=0).stop 2024-04-16T16:22:25.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.871+0000 7fcd7ea00640 1 -- 172.21.15.45:0/2233878955 shutdown_connections 2024-04-16T16:22:25.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.871+0000 7fcd7ea00640 1 --2- 172.21.15.45:0/2233878955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd780d3e90 0x7fcd781622f0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.871+0000 7fcd7ea00640 1 --2- 172.21.15.45:0/2233878955 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd780b6310 0x7fcd78161db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.871+0000 7fcd7ea00640 1 -- 172.21.15.45:0/2233878955 >> 172.21.15.45:0/2233878955 conn(0x7fcd7806cf90 msgr2=0x7fcd7806d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:25.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.871+0000 7fcd7ea00640 1 -- 172.21.15.45:0/2233878955 shutdown_connections 2024-04-16T16:22:25.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.872+0000 7fcd7ea00640 1 -- 172.21.15.45:0/2233878955 wait complete. 2024-04-16T16:22:25.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.873+0000 7fcd7ea00640 1 Processor -- start 2024-04-16T16:22:25.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.873+0000 7fcd7ea00640 1 -- start start 2024-04-16T16:22:25.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.874+0000 7fcd7ea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd780b6310 0x7fcd78154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:25.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.874+0000 7fcd7ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd780d18f0 0x7fcd780b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:25.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.874+0000 7fcd7ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd780d3e90 0x7fcd780b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:25.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.874+0000 7fcd7ea00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcd780d1e50 con 0x7fcd780d3e90 2024-04-16T16:22:25.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.874+0000 7fcd7ea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcd780d1cd0 con 0x7fcd780d18f0 2024-04-16T16:22:25.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.874+0000 7fcd7ea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcd780d1fd0 con 0x7fcd780b6310 2024-04-16T16:22:25.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.874+0000 7fcd7e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd780d3e90 0x7fcd780b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:25.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.874+0000 7fcd7e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd780d3e90 0x7fcd780b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53188/0 (socket says 172.21.15.45:53188) 2024-04-16T16:22:25.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.874+0000 7fcd7e000640 1 -- 172.21.15.45:0/2155745453 learned_addr learned my addr 172.21.15.45:0/2155745453 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:25.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.875+0000 7fcd7cc00640 1 --2- 172.21.15.45:0/2155745453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd780d18f0 0x7fcd780b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:25.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.875+0000 7fcd7d600640 1 --2- 172.21.15.45:0/2155745453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd780b6310 0x7fcd78154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:25.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.875+0000 7fcd7e000640 1 -- 172.21.15.45:0/2155745453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd780b6310 msgr2=0x7fcd78154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:25.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.875+0000 7fcd7e000640 1 --2- 172.21.15.45:0/2155745453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd780b6310 0x7fcd78154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.875+0000 7fcd7e000640 1 -- 172.21.15.45:0/2155745453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd780d18f0 msgr2=0x7fcd780b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:25.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.875+0000 7fcd7e000640 1 --2- 172.21.15.45:0/2155745453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd780d18f0 0x7fcd780b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:25.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.875+0000 7fcd7e000640 1 -- 172.21.15.45:0/2155745453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd780c9d70 con 0x7fcd780d3e90 2024-04-16T16:22:25.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.875+0000 7fcd7d600640 1 --2- 172.21.15.45:0/2155745453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd780b6310 0x7fcd78154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:25.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.875+0000 7fcd7cc00640 1 --2- 172.21.15.45:0/2155745453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd780d18f0 0x7fcd780b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:25.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.875+0000 7fcd7e000640 1 --2- 172.21.15.45:0/2155745453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd780d3e90 0x7fcd780b4830 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009eb0 tx=0x7fcd70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:25.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.876+0000 7fcd75600640 1 -- 172.21.15.45:0/2155745453 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70039050 con 0x7fcd780d3e90 2024-04-16T16:22:25.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.876+0000 7fcd75600640 1 -- 172.21.15.45:0/2155745453 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd70045070 con 0x7fcd780d3e90 2024-04-16T16:22:25.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.876+0000 7fcd75600640 1 -- 172.21.15.45:0/2155745453 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70009450 con 0x7fcd780d3e90 2024-04-16T16:22:25.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.876+0000 7fcd7ea00640 1 -- 172.21.15.45:0/2155745453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd78156570 con 0x7fcd780d3e90 2024-04-16T16:22:25.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.876+0000 7fcd7ea00640 1 -- 172.21.15.45:0/2155745453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd78156b30 con 0x7fcd780d3e90 2024-04-16T16:22:25.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.877+0000 7fcd7ea00640 1 -- 172.21.15.45:0/2155745453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd78068300 con 0x7fcd780d3e90 2024-04-16T16:22:25.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.878+0000 7fcd75600640 1 -- 172.21.15.45:0/2155745453 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 66) ==== 90751+0+0 (secure 0 0 0) 0x7fcd7003e850 con 0x7fcd780d3e90 2024-04-16T16:22:25.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.879+0000 7fcd75600640 1 -- 172.21.15.45:0/2155745453 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6397+0+0 (secure 0 0 0) 0x7fcd700adeb0 con 0x7fcd780d3e90 2024-04-16T16:22:25.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:25.883+0000 7fcd75600640 1 -- 172.21.15.45:0/2155745453 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcd78068300 con 0x7fcd780d3e90 2024-04-16T16:22:25.902 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-04-16T16:22:25.902+0000 7fd7fb02e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T16:22:26.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.085+0000 7fcd7ea00640 1 -- 172.21.15.45:0/2155745453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcd780b3b10 con 0x7fcd780d3e90 2024-04-16T16:22:26.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.086+0000 7fcd75600640 1 -- 172.21.15.45:0/2155745453 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v144) ==== 100+0+11 (secure 0 0 0) 0x7fcd780b3b10 con 0x7fcd780d3e90 2024-04-16T16:22:26.085 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:22:26.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.088+0000 7fcd7ea00640 1 -- 172.21.15.45:0/2155745453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd780d3e90 msgr2=0x7fcd780b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:26.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.088+0000 7fcd7ea00640 1 --2- 172.21.15.45:0/2155745453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd780d3e90 0x7fcd780b4830 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009eb0 tx=0x7fcd70004a40 comp rx=0 tx=0).stop 2024-04-16T16:22:26.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.088+0000 7fcd7ea00640 1 -- 172.21.15.45:0/2155745453 shutdown_connections 2024-04-16T16:22:26.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.088+0000 7fcd7ea00640 1 --2- 172.21.15.45:0/2155745453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd780d3e90 0x7fcd780b4830 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:26.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.088+0000 7fcd7ea00640 1 --2- 172.21.15.45:0/2155745453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd780d18f0 0x7fcd780b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:26.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.088+0000 7fcd7ea00640 1 --2- 172.21.15.45:0/2155745453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd780b6310 0x7fcd78154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:26.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.088+0000 7fcd7ea00640 1 -- 172.21.15.45:0/2155745453 >> 172.21.15.45:0/2155745453 conn(0x7fcd7806cf90 msgr2=0x7fcd780b5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:26.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.088+0000 7fcd7ea00640 1 -- 172.21.15.45:0/2155745453 shutdown_connections 2024-04-16T16:22:26.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.088+0000 7fcd7ea00640 1 -- 172.21.15.45:0/2155745453 wait complete. 2024-04-16T16:22:26.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T16:22:26.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.333+0000 7fe630a00640 1 Processor -- start 2024-04-16T16:22:26.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.334+0000 7fe630a00640 1 -- start start 2024-04-16T16:22:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.335+0000 7fe630a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe62c0b1ce0 0x7fe62c163010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.335+0000 7fe630a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe62c0ce710 0x7fe62c163ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.335+0000 7fe630a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe62c0cd210 con 0x7fe62c06f410 2024-04-16T16:22:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.335+0000 7fe630a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe62c0d02f0 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.335+0000 7fe630a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe62c0cd4f0 con 0x7fe62c0ce710 2024-04-16T16:22:26.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.336+0000 7fe62b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe62c0ce710 0x7fe62c163ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:26.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.336+0000 7fe62a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe62c0b1ce0 0x7fe62c163010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:26.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.337+0000 7fe623e00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fe62c06f410 0x7fe62c0b20c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:46702/0 (socket says 172.21.15.45:46702) 2024-04-16T16:22:26.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.337+0000 7fe623e00640 1 -- 172.21.15.45:0/813376942 learned_addr learned my addr 172.21.15.45:0/813376942 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:26.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.337+0000 7fe62b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe62c0ce710 0x7fe62c163ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:59800/0 (socket says 172.21.15.45:59800) 2024-04-16T16:22:26.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.337+0000 7fe62a800640 1 -- 172.21.15.45:0/813376942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe62c0ce710 msgr2=0x7fe62c163ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:26.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.337+0000 7fe62a800640 1 --2- 172.21.15.45:0/813376942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe62c0ce710 0x7fe62c163ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:26.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.337+0000 7fe62a800640 1 -- 172.21.15.45:0/813376942 >> v1:172.21.15.103:6789/0 conn(0x7fe62c06f410 legacy=0x7fe62c0b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:26.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.337+0000 7fe62a800640 1 -- 172.21.15.45:0/813376942 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62c163680 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.337+0000 7fe62b200640 1 --2- 172.21.15.45:0/813376942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe62c0ce710 0x7fe62c163ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:22:26.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.339+0000 7fe62a800640 1 --2- 172.21.15.45:0/813376942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe62c0b1ce0 0x7fe62c163010 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe61c0097a0 tx=0x7fe61c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0e4fa1bce306954 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:26.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.339+0000 7fe629e00640 1 -- 172.21.15.45:0/813376942 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c039070 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.339+0000 7fe629e00640 1 -- 172.21.15.45:0/813376942 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe61c0316f0 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.339+0000 7fe629e00640 1 -- 172.21.15.45:0/813376942 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c031ac0 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.340+0000 7fe630a00640 1 -- 172.21.15.45:0/813376942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe62c0b1ce0 msgr2=0x7fe62c163010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:26.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.340+0000 7fe630a00640 1 --2- 172.21.15.45:0/813376942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe62c0b1ce0 0x7fe62c163010 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe61c0097a0 tx=0x7fe61c02b790 comp rx=0 tx=0).stop 2024-04-16T16:22:26.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.340+0000 7fe630a00640 1 -- 172.21.15.45:0/813376942 shutdown_connections 2024-04-16T16:22:26.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.340+0000 7fe630a00640 1 --2- 172.21.15.45:0/813376942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe62c0ce710 0x7fe62c163ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:26.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.340+0000 7fe630a00640 1 --2- 172.21.15.45:0/813376942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe62c0b1ce0 0x7fe62c163010 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:26.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.340+0000 7fe630a00640 1 -- 172.21.15.45:0/813376942 >> 172.21.15.45:0/813376942 conn(0x7fe62c06b930 msgr2=0x7fe62c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:26.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.340+0000 7fe630a00640 1 -- 172.21.15.45:0/813376942 shutdown_connections 2024-04-16T16:22:26.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.340+0000 7fe630a00640 1 -- 172.21.15.45:0/813376942 wait complete. 2024-04-16T16:22:26.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.342+0000 7fe630a00640 1 Processor -- start 2024-04-16T16:22:26.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.342+0000 7fe630a00640 1 -- start start 2024-04-16T16:22:26.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.343+0000 7fe630a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe62c06f410 0x7fe62c157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:26.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.343+0000 7fe630a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe62c0b1ce0 0x7fe62c1f9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:26.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.343+0000 7fe630a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe62c0ce710 0x7fe62c158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:26.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.343+0000 7fe630a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c164260 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.343+0000 7fe630a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c1640e0 con 0x7fe62c06f410 2024-04-16T16:22:26.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.343+0000 7fe630a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe62c1643e0 con 0x7fe62c0ce710 2024-04-16T16:22:26.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.343+0000 7fe623e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe62c0b1ce0 0x7fe62c1f9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:26.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.343+0000 7fe623e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe62c0b1ce0 0x7fe62c1f9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53198/0 (socket says 172.21.15.45:53198) 2024-04-16T16:22:26.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.343+0000 7fe623e00640 1 -- 172.21.15.45:0/78777437 learned_addr learned my addr 172.21.15.45:0/78777437 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:26.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.344+0000 7fe62b200640 1 --2- 172.21.15.45:0/78777437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe62c0ce710 0x7fe62c158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:26.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.344+0000 7fe62a800640 1 --2- 172.21.15.45:0/78777437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe62c06f410 0x7fe62c157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:26.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.344+0000 7fe623e00640 1 -- 172.21.15.45:0/78777437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe62c0ce710 msgr2=0x7fe62c158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:26.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.344+0000 7fe623e00640 1 --2- 172.21.15.45:0/78777437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe62c0ce710 0x7fe62c158a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:26.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.344+0000 7fe623e00640 1 -- 172.21.15.45:0/78777437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe62c06f410 msgr2=0x7fe62c157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:26.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.344+0000 7fe623e00640 1 --2- 172.21.15.45:0/78777437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe62c06f410 0x7fe62c157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:26.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.344+0000 7fe623e00640 1 -- 172.21.15.45:0/78777437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62c1f9770 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.344+0000 7fe623e00640 1 --2- 172.21.15.45:0/78777437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe62c0b1ce0 0x7fe62c1f9230 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fe61400c430 tx=0x7fe61400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:26.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.345+0000 7fe622a00640 1 -- 172.21.15.45:0/78777437 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe614017070 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.345+0000 7fe622a00640 1 -- 172.21.15.45:0/78777437 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe614005ce0 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.345+0000 7fe630a00640 1 -- 172.21.15.45:0/78777437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe62c15ce90 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.345+0000 7fe622a00640 1 -- 172.21.15.45:0/78777437 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6140072f0 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.345+0000 7fe630a00640 1 -- 172.21.15.45:0/78777437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe62c15d450 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.346+0000 7fe630a00640 1 -- 172.21.15.45:0/78777437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe62c075780 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.347+0000 7fe622a00640 1 -- 172.21.15.45:0/78777437 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 66) ==== 90751+0+0 (secure 0 0 0) 0x7fe61401a070 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.348+0000 7fe622a00640 1 -- 172.21.15.45:0/78777437 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6397+0+0 (secure 0 0 0) 0x7fe61408ad90 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.352+0000 7fe622a00640 1 -- 172.21.15.45:0/78777437 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe62c1f9770 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.525+0000 7fe622a00640 1 -- 172.21.15.45:0/78777437 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 67) ==== 90878+0+0 (secure 0 0 0) 0x7fe6140548c0 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.526+0000 7fe622a00640 1 --2- 172.21.15.45:0/78777437 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fe608072040 0x7fe608074430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:26.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.537+0000 7fe62a800640 1 --2- 172.21.15.45:0/78777437 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fe608072040 0x7fe608074430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:26.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.549+0000 7fe62a800640 1 --2- 172.21.15.45:0/78777437 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fe608072040 0x7fe608074430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe61c036000 tx=0x7fe61c008030 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:26.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.581+0000 7fe630a00640 1 -- 172.21.15.45:0/78777437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe62c166430 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.582+0000 7fe622a00640 1 -- 172.21.15.45:0/78777437 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v144) ==== 81+0+954 (secure 0 0 0) 0x7fe614059270 con 0x7fe62c0b1ce0 2024-04-16T16:22:26.582 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/741531900 2024-04-17T16:22:25.503233+0000 2024-04-16T16:22:26.582 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3249203677 2024-04-17T16:22:25.503233+0000 2024-04-16T16:22:26.582 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1497652063 2024-04-17T16:22:25.503233+0000 2024-04-16T16:22:26.582 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/1506110537 2024-04-17T16:22:25.503233+0000 2024-04-16T16:22:26.582 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3222629764 2024-04-17T16:22:25.503233+0000 2024-04-16T16:22:26.582 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/1506110537 2024-04-17T16:22:25.503233+0000 2024-04-16T16:22:26.583 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1298902321 2024-04-17T16:22:25.503233+0000 2024-04-16T16:22:26.583 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/884289453 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:26.583 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/3228977328 2024-04-17T16:22:25.503233+0000 2024-04-16T16:22:26.583 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1437683872 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:26.583 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2877575799 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:26.583 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/2644878831 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:26.583 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6833/1478535274 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:26.583 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:6832/1478535274 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:26.583 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1111841625 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:26.583 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.103:0/1114329424 2024-04-17T16:21:47.933003+0000 2024-04-16T16:22:26.584 INFO:teuthology.orchestra.run.smithi045.stderr:listed 16 entries 2024-04-16T16:22:26.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.584+0000 7fe630a00640 1 -- 172.21.15.45:0/78777437 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fe608072040 msgr2=0x7fe608074430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:26.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.584+0000 7fe630a00640 1 --2- 172.21.15.45:0/78777437 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fe608072040 0x7fe608074430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe61c036000 tx=0x7fe61c008030 comp rx=0 tx=0).stop 2024-04-16T16:22:26.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.584+0000 7fe630a00640 1 -- 172.21.15.45:0/78777437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe62c0b1ce0 msgr2=0x7fe62c1f9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:26.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.584+0000 7fe630a00640 1 --2- 172.21.15.45:0/78777437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe62c0b1ce0 0x7fe62c1f9230 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fe61400c430 tx=0x7fe61400c8f0 comp rx=0 tx=0).stop 2024-04-16T16:22:26.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.585+0000 7fe630a00640 1 -- 172.21.15.45:0/78777437 shutdown_connections 2024-04-16T16:22:26.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.585+0000 7fe630a00640 1 --2- 172.21.15.45:0/78777437 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fe608072040 0x7fe608074430 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:26.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.585+0000 7fe630a00640 1 --2- 172.21.15.45:0/78777437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe62c0ce710 0x7fe62c158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:26.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.585+0000 7fe630a00640 1 --2- 172.21.15.45:0/78777437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe62c0b1ce0 0x7fe62c1f9230 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:26.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.585+0000 7fe630a00640 1 --2- 172.21.15.45:0/78777437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe62c06f410 0x7fe62c157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:26.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.585+0000 7fe630a00640 1 -- 172.21.15.45:0/78777437 >> 172.21.15.45:0/78777437 conn(0x7fe62c06b930 msgr2=0x7fe62c0b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:26.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.585+0000 7fe630a00640 1 -- 172.21.15.45:0/78777437 shutdown_connections 2024-04-16T16:22:26.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:26.585+0000 7fe630a00640 1 -- 172.21.15.45:0/78777437 wait complete. 2024-04-16T16:22:26.606 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-16T16:22:26.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:26.606 DEBUG:teuthology.orchestra.run.smithi103:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-04-16T16:22:26.704 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:22:26.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:26.704 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-16T16:22:26.750 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:22:26.750 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:26.750 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:26.813 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:22:26.813 INFO:tasks.cephfs.mount:Terminating background process 2024-04-16T16:22:26.813 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:22:26.813 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:26.867 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:22:26.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:22:27.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.105+0000 7fcb4de00640 1 Processor -- start 2024-04-16T16:22:27.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.105+0000 7fcb4de00640 1 -- start start 2024-04-16T16:22:27.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.107+0000 7fcb4de00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcb480ccb20 0x7fcb480cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:27.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.107+0000 7fcb4de00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcb480ce9a0 0x7fcb480cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:27.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.107+0000 7fcb4de00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb4815ecd0 con 0x7fcb4815e8f0 2024-04-16T16:22:27.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.107+0000 7fcb4de00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fcb480ce2e0 con 0x7fcb480ccb20 2024-04-16T16:22:27.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.107+0000 7fcb4de00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcb480cd6d0 con 0x7fcb480ce9a0 2024-04-16T16:22:27.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.107+0000 7fcb4ca00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcb4815e8f0 0x7fcb480cc410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39510/0 (socket says 172.21.15.45:39510) 2024-04-16T16:22:27.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.107+0000 7fcb4ca00640 1 -- 172.21.15.45:0/3328261813 learned_addr learned my addr 172.21.15.45:0/3328261813 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:27.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.107+0000 7fcb47e00640 1 --2- 172.21.15.45:0/3328261813 >> v2:172.21.15.45:3300/0 conn(0x7fcb480ccb20 0x7fcb480cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:27.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.108+0000 7fcb4d400640 1 --2- 172.21.15.45:0/3328261813 >> v2:172.21.15.103:3300/0 conn(0x7fcb480ce9a0 0x7fcb480cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:27.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.108+0000 7fcb47400640 1 -- 172.21.15.45:0/3328261813 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2634814512 0 0) 0x7fcb4815ecd0 con 0x7fcb4815e8f0 2024-04-16T16:22:27.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.108+0000 7fcb47400640 1 -- 172.21.15.45:0/3328261813 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb30003520 con 0x7fcb4815e8f0 2024-04-16T16:22:27.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.108+0000 7fcb47e00640 1 -- 172.21.15.45:0/3328261813 >> v2:172.21.15.103:3300/0 conn(0x7fcb480ce9a0 msgr2=0x7fcb480cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:27.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.108+0000 7fcb47e00640 1 --2- 172.21.15.45:0/3328261813 >> v2:172.21.15.103:3300/0 conn(0x7fcb480ce9a0 0x7fcb480cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.108+0000 7fcb47e00640 1 -- 172.21.15.45:0/3328261813 >> v1:172.21.15.45:6789/0 conn(0x7fcb4815e8f0 legacy=0x7fcb480cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:27.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.108+0000 7fcb47e00640 1 -- 172.21.15.45:0/3328261813 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb480cd9b0 con 0x7fcb480ccb20 2024-04-16T16:22:27.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.111+0000 7fcb47e00640 1 --2- 172.21.15.45:0/3328261813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb480ccb20 0x7fcb480cac30 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fcb4806ff60 tx=0x7fcb3402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eef2e60b5e1278d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:27.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.112+0000 7fcb47400640 1 -- 172.21.15.45:0/3328261813 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb340315a0 con 0x7fcb480ccb20 2024-04-16T16:22:27.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.112+0000 7fcb47400640 1 -- 172.21.15.45:0/3328261813 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb34037070 con 0x7fcb480ccb20 2024-04-16T16:22:27.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.112+0000 7fcb47400640 1 -- 172.21.15.45:0/3328261813 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb34031960 con 0x7fcb480ccb20 2024-04-16T16:22:27.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.112+0000 7fcb4de00640 1 -- 172.21.15.45:0/3328261813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb480ccb20 msgr2=0x7fcb480cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:27.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.112+0000 7fcb4de00640 1 --2- 172.21.15.45:0/3328261813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb480ccb20 0x7fcb480cac30 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fcb4806ff60 tx=0x7fcb3402b840 comp rx=0 tx=0).stop 2024-04-16T16:22:27.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.112+0000 7fcb4de00640 1 -- 172.21.15.45:0/3328261813 shutdown_connections 2024-04-16T16:22:27.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.112+0000 7fcb4de00640 1 --2- 172.21.15.45:0/3328261813 >> v2:172.21.15.103:3300/0 conn(0x7fcb480ce9a0 0x7fcb480cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.112+0000 7fcb4de00640 1 --2- 172.21.15.45:0/3328261813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb480ccb20 0x7fcb480cac30 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.112+0000 7fcb4de00640 1 -- 172.21.15.45:0/3328261813 >> 172.21.15.45:0/3328261813 conn(0x7fcb4806b930 msgr2=0x7fcb4806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:27.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.113+0000 7fcb4de00640 1 -- 172.21.15.45:0/3328261813 shutdown_connections 2024-04-16T16:22:27.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.113+0000 7fcb4de00640 1 -- 172.21.15.45:0/3328261813 wait complete. 2024-04-16T16:22:27.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.114+0000 7fcb4de00640 1 Processor -- start 2024-04-16T16:22:27.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.114+0000 7fcb4de00640 1 -- start start 2024-04-16T16:22:27.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.115+0000 7fcb4de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb480ce9a0 0x7fcb48143180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:27.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.115+0000 7fcb4de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb4815e8f0 0x7fcb480b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:27.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.115+0000 7fcb4de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb480b4d70 0x7fcb481f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:27.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.115+0000 7fcb4de00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb480cd360 con 0x7fcb480ce9a0 2024-04-16T16:22:27.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.115+0000 7fcb4de00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcb480cdf70 con 0x7fcb480b4d70 2024-04-16T16:22:27.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.115+0000 7fcb4de00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcb480cf080 con 0x7fcb4815e8f0 2024-04-16T16:22:27.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.116+0000 7fcb4ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb480ce9a0 0x7fcb48143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:27.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.116+0000 7fcb4ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb480ce9a0 0x7fcb48143180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53220/0 (socket says 172.21.15.45:53220) 2024-04-16T16:22:27.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.116+0000 7fcb4ca00640 1 -- 172.21.15.45:0/1060913968 learned_addr learned my addr 172.21.15.45:0/1060913968 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:27.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.116+0000 7fcb47e00640 1 --2- 172.21.15.45:0/1060913968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb4815e8f0 0x7fcb480b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:27.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.116+0000 7fcb4ca00640 1 -- 172.21.15.45:0/1060913968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb4815e8f0 msgr2=0x7fcb480b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:27.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.116+0000 7fcb4ca00640 1 --2- 172.21.15.45:0/1060913968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb4815e8f0 0x7fcb480b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.116+0000 7fcb4ca00640 1 -- 172.21.15.45:0/1060913968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb480b4d70 msgr2=0x7fcb481f7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.116+0000 7fcb4ca00640 1 --2- 172.21.15.45:0/1060913968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb480b4d70 0x7fcb481f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.116+0000 7fcb4ca00640 1 -- 172.21.15.45:0/1060913968 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb481436c0 con 0x7fcb480ce9a0 2024-04-16T16:22:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.117+0000 7fcb4ca00640 1 --2- 172.21.15.45:0/1060913968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb480ce9a0 0x7fcb48143180 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c00c390 tx=0x7fcb3c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.117+0000 7fcb44c00640 1 -- 172.21.15.45:0/1060913968 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c017070 con 0x7fcb480ce9a0 2024-04-16T16:22:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.117+0000 7fcb44c00640 1 -- 172.21.15.45:0/1060913968 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb3c005ce0 con 0x7fcb480ce9a0 2024-04-16T16:22:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.117+0000 7fcb44c00640 1 -- 172.21.15.45:0/1060913968 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c0072f0 con 0x7fcb480ce9a0 2024-04-16T16:22:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.117+0000 7fcb4de00640 1 -- 172.21.15.45:0/1060913968 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb481f8910 con 0x7fcb480ce9a0 2024-04-16T16:22:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.117+0000 7fcb4de00640 1 -- 172.21.15.45:0/1060913968 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb4815f450 con 0x7fcb480ce9a0 2024-04-16T16:22:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.118+0000 7fcb4de00640 1 -- 172.21.15.45:0/1060913968 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb480b51c0 con 0x7fcb480ce9a0 2024-04-16T16:22:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.119+0000 7fcb44c00640 1 -- 172.21.15.45:0/1060913968 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 67) ==== 90878+0+0 (secure 0 0 0) 0x7fcb3c01a070 con 0x7fcb480ce9a0 2024-04-16T16:22:27.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.120+0000 7fcb44c00640 1 --2- 172.21.15.45:0/1060913968 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fcb2806c740 0x7fcb2806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:27.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.120+0000 7fcb44c00640 1 -- 172.21.15.45:0/1060913968 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6397+0+0 (secure 0 0 0) 0x7fcb3c08ae70 con 0x7fcb480ce9a0 2024-04-16T16:22:27.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.121+0000 7fcb47e00640 1 --2- 172.21.15.45:0/1060913968 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fcb2806c740 0x7fcb2806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:27.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.122+0000 7fcb47e00640 1 --2- 172.21.15.45:0/1060913968 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fcb2806c740 0x7fcb2806ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcb34036000 tx=0x7fcb34008030 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:27.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.124+0000 7fcb44c00640 1 -- 172.21.15.45:0/1060913968 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcb3c059260 con 0x7fcb480ce9a0 2024-04-16T16:22:27.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.375+0000 7fcb4de00640 1 -- 172.21.15.45:0/1060913968 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb48075780 con 0x7fcb480ce9a0 2024-04-16T16:22:27.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.377+0000 7fcb44c00640 1 -- 172.21.15.45:0/1060913968 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5213 (secure 0 0 0) 0x7fcb3c058b60 con 0x7fcb480ce9a0 2024-04-16T16:22:27.377 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:27.377 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2585865379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2585865379},{"type":"v1","addr":"172.21.15.45:6834","nonce":2585865379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6590,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/777087937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":777087937},{"type":"v1","addr":"172.21.15.45:6835","nonce":777087937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:21:26.066855+0000","modified":"2024-04-16T16:22:25.276355+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6596,"mds_1":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":1,"incarnation":102,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6835/2561260581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2561260581},{"type":"v1","addr":"172.21.15.103:6835","nonce":2561260581}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6596":{"gid":6596,"name":"d","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6837/3252460318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3252460318},{"type":"v1","addr":"172.21.15.103:6837","nonce":3252460318}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6596,"qdb_cluster":[6593,6596]},"id":10}]} 2024-04-16T16:22:27.377 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 105 2024-04-16T16:22:27.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.379+0000 7fcb4de00640 1 -- 172.21.15.45:0/1060913968 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fcb2806c740 msgr2=0x7fcb2806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:27.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.379+0000 7fcb4de00640 1 --2- 172.21.15.45:0/1060913968 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fcb2806c740 0x7fcb2806ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcb34036000 tx=0x7fcb34008030 comp rx=0 tx=0).stop 2024-04-16T16:22:27.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.379+0000 7fcb4de00640 1 -- 172.21.15.45:0/1060913968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb480ce9a0 msgr2=0x7fcb48143180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:27.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.379+0000 7fcb4de00640 1 --2- 172.21.15.45:0/1060913968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb480ce9a0 0x7fcb48143180 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c00c390 tx=0x7fcb3c00c850 comp rx=0 tx=0).stop 2024-04-16T16:22:27.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.379+0000 7fcb4de00640 1 -- 172.21.15.45:0/1060913968 shutdown_connections 2024-04-16T16:22:27.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.379+0000 7fcb4de00640 1 --2- 172.21.15.45:0/1060913968 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fcb2806c740 0x7fcb2806ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.379+0000 7fcb4de00640 1 --2- 172.21.15.45:0/1060913968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb480b4d70 0x7fcb481f7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.379+0000 7fcb4de00640 1 --2- 172.21.15.45:0/1060913968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb4815e8f0 0x7fcb480b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.379+0000 7fcb4de00640 1 --2- 172.21.15.45:0/1060913968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb480ce9a0 0x7fcb48143180 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.379+0000 7fcb4de00640 1 -- 172.21.15.45:0/1060913968 >> 172.21.15.45:0/1060913968 conn(0x7fcb4806b930 msgr2=0x7fcb480cb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:27.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.379+0000 7fcb4de00640 1 -- 172.21.15.45:0/1060913968 shutdown_connections 2024-04-16T16:22:27.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.380+0000 7fcb4de00640 1 -- 172.21.15.45:0/1060913968 wait complete. 2024-04-16T16:22:27.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:22:27.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.626+0000 7fa64de00640 1 Processor -- start 2024-04-16T16:22:27.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.626+0000 7fa64de00640 1 -- start start 2024-04-16T16:22:27.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.628+0000 7fa64de00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa64815ea60 0x7fa64806e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:27.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.628+0000 7fa64de00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa64806cd20 0x7fa64806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:27.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.628+0000 7fa64de00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6480cf4b0 con 0x7fa648067be0 2024-04-16T16:22:27.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.628+0000 7fa64de00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa6480d0030 con 0x7fa64815ea60 2024-04-16T16:22:27.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.628+0000 7fa64de00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa6480cf790 con 0x7fa64806cd20 2024-04-16T16:22:27.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.628+0000 7fa64ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa64815ea60 0x7fa64806e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:27.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.628+0000 7fa64ca00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa64815ea60 0x7fa64806e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53232/0 (socket says 172.21.15.45:53232) 2024-04-16T16:22:27.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.628+0000 7fa647e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa648067be0 0x7fa64815ee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39522/0 (socket says 172.21.15.45:39522) 2024-04-16T16:22:27.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.628+0000 7fa64ca00640 1 -- 172.21.15.45:0/523089316 learned_addr learned my addr 172.21.15.45:0/523089316 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:27.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.629+0000 7fa64d400640 1 --2- 172.21.15.45:0/523089316 >> v2:172.21.15.103:3300/0 conn(0x7fa64806cd20 0x7fa64806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:27.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.629+0000 7fa64ca00640 1 -- 172.21.15.45:0/523089316 >> v2:172.21.15.103:3300/0 conn(0x7fa64806cd20 msgr2=0x7fa64806ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:27.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.629+0000 7fa64ca00640 1 --2- 172.21.15.45:0/523089316 >> v2:172.21.15.103:3300/0 conn(0x7fa64806cd20 0x7fa64806ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.629+0000 7fa64ca00640 1 -- 172.21.15.45:0/523089316 >> v1:172.21.15.45:6789/0 conn(0x7fa648067be0 legacy=0x7fa64815ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:27.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.629+0000 7fa64ca00640 1 -- 172.21.15.45:0/523089316 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6480cfa70 con 0x7fa64815ea60 2024-04-16T16:22:27.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.629+0000 7fa64d400640 1 --2- 172.21.15.45:0/523089316 >> v2:172.21.15.103:3300/0 conn(0x7fa64806cd20 0x7fa64806ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:27.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.630+0000 7fa64ca00640 1 --2- 172.21.15.45:0/523089316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa64815ea60 0x7fa64806e790 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fa6480d06c0 tx=0x7fa63c02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7c1abd339180c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:27.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.630+0000 7fa647400640 1 -- 172.21.15.45:0/523089316 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c034070 con 0x7fa64815ea60 2024-04-16T16:22:27.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.630+0000 7fa647400640 1 -- 172.21.15.45:0/523089316 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa63c030870 con 0x7fa64815ea60 2024-04-16T16:22:27.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.630+0000 7fa647400640 1 -- 172.21.15.45:0/523089316 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c030c40 con 0x7fa64815ea60 2024-04-16T16:22:27.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.631+0000 7fa64de00640 1 -- 172.21.15.45:0/523089316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa64815ea60 msgr2=0x7fa64806e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:27.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.631+0000 7fa64de00640 1 --2- 172.21.15.45:0/523089316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa64815ea60 0x7fa64806e790 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fa6480d06c0 tx=0x7fa63c02b9b0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.631+0000 7fa64de00640 1 -- 172.21.15.45:0/523089316 shutdown_connections 2024-04-16T16:22:27.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.631+0000 7fa64de00640 1 --2- 172.21.15.45:0/523089316 >> v2:172.21.15.103:3300/0 conn(0x7fa64806cd20 0x7fa64806ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.631+0000 7fa64de00640 1 --2- 172.21.15.45:0/523089316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa64815ea60 0x7fa64806e790 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.631+0000 7fa64de00640 1 -- 172.21.15.45:0/523089316 >> 172.21.15.45:0/523089316 conn(0x7fa6480cd820 msgr2=0x7fa6480cdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:27.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.631+0000 7fa64de00640 1 -- 172.21.15.45:0/523089316 shutdown_connections 2024-04-16T16:22:27.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.631+0000 7fa64de00640 1 -- 172.21.15.45:0/523089316 wait complete. 2024-04-16T16:22:27.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.633+0000 7fa64de00640 1 Processor -- start 2024-04-16T16:22:27.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.633+0000 7fa64de00640 1 -- start start 2024-04-16T16:22:27.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.634+0000 7fa64de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa648067be0 0x7fa6481f9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:27.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.634+0000 7fa64de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa64806cd20 0x7fa6481f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:27.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.634+0000 7fa64de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa64815ea60 0x7fa64815ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:27.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.634+0000 7fa64de00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa6480cf270 con 0x7fa64806cd20 2024-04-16T16:22:27.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.634+0000 7fa64de00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa6480cf0f0 con 0x7fa64815ea60 2024-04-16T16:22:27.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.634+0000 7fa64de00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa6481625a0 con 0x7fa648067be0 2024-04-16T16:22:27.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.634+0000 7fa647e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa64806cd20 0x7fa6481f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:27.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.635+0000 7fa647e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa64806cd20 0x7fa6481f49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53234/0 (socket says 172.21.15.45:53234) 2024-04-16T16:22:27.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.635+0000 7fa647e00640 1 -- 172.21.15.45:0/3381180888 learned_addr learned my addr 172.21.15.45:0/3381180888 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:27.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.635+0000 7fa64ca00640 1 --2- 172.21.15.45:0/3381180888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa648067be0 0x7fa6481f9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:27.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.635+0000 7fa64d400640 1 --2- 172.21.15.45:0/3381180888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa64815ea60 0x7fa64815ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:27.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.635+0000 7fa647e00640 1 -- 172.21.15.45:0/3381180888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa648067be0 msgr2=0x7fa6481f9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:27.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.635+0000 7fa647e00640 1 --2- 172.21.15.45:0/3381180888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa648067be0 0x7fa6481f9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.635+0000 7fa647e00640 1 -- 172.21.15.45:0/3381180888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa64815ea60 msgr2=0x7fa64815ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:27.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.635+0000 7fa647e00640 1 --2- 172.21.15.45:0/3381180888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa64815ea60 0x7fa64815ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.635+0000 7fa647e00640 1 -- 172.21.15.45:0/3381180888 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6480c5d40 con 0x7fa64806cd20 2024-04-16T16:22:27.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.635+0000 7fa64d400640 1 --2- 172.21.15.45:0/3381180888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa64815ea60 0x7fa64815ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:27.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.635+0000 7fa647e00640 1 --2- 172.21.15.45:0/3381180888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa64806cd20 0x7fa6481f49b0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fa63400c440 tx=0x7fa634001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:27.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.636+0000 7fa644c00640 1 -- 172.21.15.45:0/3381180888 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6340074e0 con 0x7fa64806cd20 2024-04-16T16:22:27.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.636+0000 7fa644c00640 1 -- 172.21.15.45:0/3381180888 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa634007680 con 0x7fa64806cd20 2024-04-16T16:22:27.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.636+0000 7fa644c00640 1 -- 172.21.15.45:0/3381180888 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa634004a20 con 0x7fa64806cd20 2024-04-16T16:22:27.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.636+0000 7fa64de00640 1 -- 172.21.15.45:0/3381180888 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6480c6020 con 0x7fa64806cd20 2024-04-16T16:22:27.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.636+0000 7fa64de00640 1 -- 172.21.15.45:0/3381180888 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa64814b4d0 con 0x7fa64806cd20 2024-04-16T16:22:27.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.637+0000 7fa64de00640 1 -- 172.21.15.45:0/3381180888 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa648075780 con 0x7fa64806cd20 2024-04-16T16:22:27.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.638+0000 7fa644c00640 1 -- 172.21.15.45:0/3381180888 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 67) ==== 90878+0+0 (secure 0 0 0) 0x7fa63401a020 con 0x7fa64806cd20 2024-04-16T16:22:27.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.639+0000 7fa644c00640 1 --2- 172.21.15.45:0/3381180888 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fa62806c740 0x7fa62806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:27.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.639+0000 7fa644c00640 1 -- 172.21.15.45:0/3381180888 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6397+0+0 (secure 0 0 0) 0x7fa63408b330 con 0x7fa64806cd20 2024-04-16T16:22:27.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.640+0000 7fa64ca00640 1 --2- 172.21.15.45:0/3381180888 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fa62806c740 0x7fa62806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:27.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.641+0000 7fa64ca00640 1 --2- 172.21.15.45:0/3381180888 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fa62806c740 0x7fa62806ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa63c036000 tx=0x7fa63c03b040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:27.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.644+0000 7fa644c00640 1 -- 172.21.15.45:0/3381180888 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa634010300 con 0x7fa64806cd20 2024-04-16T16:22:27.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.850+0000 7fa64de00640 1 -- 172.21.15.45:0/3381180888 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa64815e280 con 0x7fa64806cd20 2024-04-16T16:22:27.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.853+0000 7fa644c00640 1 -- 172.21.15.45:0/3381180888 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) ==== 74+0+17696 (secure 0 0 0) 0x7fa634059720 con 0x7fa64806cd20 2024-04-16T16:22:27.854 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:27.854 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":144,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:22:25.503282+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:21:23.973149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-16T16:21:25.043255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/741531900":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/3249203677":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/1497652063":"2024-04-17T16:22:25.503233+0000","172.21.15.103:6833/1506110537":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/3222629764":"2024-04-17T16:22:25.503233+0000","172.21.15.103:6832/1506110537":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/1298902321":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/884289453":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/3228977328":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/1437683872":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/2877575799":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/2644878831":"2024-04-17T16:21:47.933003+0000","172.21.15.103:6833/1478535274":"2024-04-17T16:21:47.933003+0000","172.21.15.103:6832/1478535274":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/1111841625":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/1114329424":"2024-04-17T16:21:47.933003+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:22:27.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.856+0000 7fa64de00640 1 -- 172.21.15.45:0/3381180888 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fa62806c740 msgr2=0x7fa62806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:27.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.856+0000 7fa64de00640 1 --2- 172.21.15.45:0/3381180888 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fa62806c740 0x7fa62806ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa63c036000 tx=0x7fa63c03b040 comp rx=0 tx=0).stop 2024-04-16T16:22:27.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.856+0000 7fa64de00640 1 -- 172.21.15.45:0/3381180888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa64806cd20 msgr2=0x7fa6481f49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:27.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.856+0000 7fa64de00640 1 --2- 172.21.15.45:0/3381180888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa64806cd20 0x7fa6481f49b0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fa63400c440 tx=0x7fa634001e70 comp rx=0 tx=0).stop 2024-04-16T16:22:27.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.856+0000 7fa64de00640 1 -- 172.21.15.45:0/3381180888 shutdown_connections 2024-04-16T16:22:27.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.856+0000 7fa64de00640 1 --2- 172.21.15.45:0/3381180888 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fa62806c740 0x7fa62806ec00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.856+0000 7fa64de00640 1 --2- 172.21.15.45:0/3381180888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa64815ea60 0x7fa64815ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.856+0000 7fa64de00640 1 --2- 172.21.15.45:0/3381180888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa64806cd20 0x7fa6481f49b0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.856+0000 7fa64de00640 1 --2- 172.21.15.45:0/3381180888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa648067be0 0x7fa6481f9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:27.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.856+0000 7fa64de00640 1 -- 172.21.15.45:0/3381180888 >> 172.21.15.45:0/3381180888 conn(0x7fa6480cd820 msgr2=0x7fa648161040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:27.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.856+0000 7fa64de00640 1 -- 172.21.15.45:0/3381180888 shutdown_connections 2024-04-16T16:22:27.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:27.857+0000 7fa64de00640 1 -- 172.21.15.45:0/3381180888 wait complete. 2024-04-16T16:22:27.876 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-16T16:21:23.973149+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-04-16T16:21:25.043255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:22:27.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:22:28.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.087+0000 7fbf71000640 1 Processor -- start 2024-04-16T16:22:28.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.088+0000 7fbf71000640 1 -- start start 2024-04-16T16:22:28.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.089+0000 7fbf71000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf6c0b6270 0x7fbf6c0d2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:28.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.089+0000 7fbf71000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbf6c0b7920 0x7fbf6c0b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:28.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.089+0000 7fbf71000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf6c0d32c0 con 0x7fbf6c0ba2a0 2024-04-16T16:22:28.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.089+0000 7fbf71000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fbf6c0d35a0 con 0x7fbf6c0b7920 2024-04-16T16:22:28.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.089+0000 7fbf71000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbf6c0d3880 con 0x7fbf6c0b6270 2024-04-16T16:22:28.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.089+0000 7fbf6be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbf6c0b7920 0x7fbf6c0b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:28.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.089+0000 7fbf6be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbf6c0b7920 0x7fbf6c0b7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53242/0 (socket says 172.21.15.45:53242) 2024-04-16T16:22:28.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.089+0000 7fbf6be00640 1 -- 172.21.15.45:0/2458412722 learned_addr learned my addr 172.21.15.45:0/2458412722 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:28.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.090+0000 7fbf6b400640 1 --2- 172.21.15.45:0/2458412722 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf6c0b6270 0x7fbf6c0d2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:28.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.090+0000 7fbf6be00640 1 -- 172.21.15.45:0/2458412722 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf6c0b6270 msgr2=0x7fbf6c0d2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:28.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.090+0000 7fbf6be00640 1 --2- 172.21.15.45:0/2458412722 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf6c0b6270 0x7fbf6c0d2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.090+0000 7fbf6be00640 1 -- 172.21.15.45:0/2458412722 >> v1:172.21.15.103:6789/0 conn(0x7fbf6c0ba2a0 legacy=0x7fbf6c0b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:28.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.090+0000 7fbf6be00640 1 -- 172.21.15.45:0/2458412722 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf6c0d3b60 con 0x7fbf6c0b7920 2024-04-16T16:22:28.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.091+0000 7fbf6be00640 1 --2- 172.21.15.45:0/2458412722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf6c0b7920 0x7fbf6c0b7db0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fbf64009870 tx=0x7fbf6402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=725001457bc8ac17 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:28.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.091+0000 7fbf6a000640 1 -- 172.21.15.45:0/2458412722 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf64035020 con 0x7fbf6c0b7920 2024-04-16T16:22:28.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.091+0000 7fbf6a000640 1 -- 172.21.15.45:0/2458412722 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf64031640 con 0x7fbf6c0b7920 2024-04-16T16:22:28.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.091+0000 7fbf6a000640 1 -- 172.21.15.45:0/2458412722 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6402d040 con 0x7fbf6c0b7920 2024-04-16T16:22:28.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.092+0000 7fbf71000640 1 -- 172.21.15.45:0/2458412722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf6c0b7920 msgr2=0x7fbf6c0b7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:28.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.092+0000 7fbf71000640 1 --2- 172.21.15.45:0/2458412722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf6c0b7920 0x7fbf6c0b7db0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fbf64009870 tx=0x7fbf6402b8f0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.092+0000 7fbf71000640 1 -- 172.21.15.45:0/2458412722 shutdown_connections 2024-04-16T16:22:28.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.092+0000 7fbf71000640 1 --2- 172.21.15.45:0/2458412722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf6c0b7920 0x7fbf6c0b7db0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.092+0000 7fbf71000640 1 --2- 172.21.15.45:0/2458412722 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf6c0b6270 0x7fbf6c0d2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.092+0000 7fbf71000640 1 -- 172.21.15.45:0/2458412722 >> 172.21.15.45:0/2458412722 conn(0x7fbf6c06d300 msgr2=0x7fbf6c0d1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:28.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.092+0000 7fbf71000640 1 -- 172.21.15.45:0/2458412722 shutdown_connections 2024-04-16T16:22:28.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.092+0000 7fbf71000640 1 -- 172.21.15.45:0/2458412722 wait complete. 2024-04-16T16:22:28.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.094+0000 7fbf71000640 1 Processor -- start 2024-04-16T16:22:28.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.094+0000 7fbf71000640 1 -- start start 2024-04-16T16:22:28.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.095+0000 7fbf71000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf6c0b6270 0x7fbf6c0ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:28.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.095+0000 7fbf71000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf6c0b7920 0x7fbf6c15b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:28.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.095+0000 7fbf71000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf6c0ba2a0 0x7fbf6c1f4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:28.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.095+0000 7fbf71000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c0d4360 con 0x7fbf6c0b7920 2024-04-16T16:22:28.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.095+0000 7fbf71000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c0ba700 con 0x7fbf6c0ba2a0 2024-04-16T16:22:28.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.095+0000 7fbf71000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbf6c1606a0 con 0x7fbf6c0b6270 2024-04-16T16:22:28.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.095+0000 7fbf6aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf6c0b7920 0x7fbf6c15b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.095+0000 7fbf6aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf6c0b7920 0x7fbf6c15b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53248/0 (socket says 172.21.15.45:53248) 2024-04-16T16:22:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.095+0000 7fbf6aa00640 1 -- 172.21.15.45:0/1349398591 learned_addr learned my addr 172.21.15.45:0/1349398591 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.096+0000 7fbf6be00640 1 --2- 172.21.15.45:0/1349398591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf6c0ba2a0 0x7fbf6c1f4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.096+0000 7fbf6b400640 1 --2- 172.21.15.45:0/1349398591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf6c0b6270 0x7fbf6c0ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.096+0000 7fbf6aa00640 1 -- 172.21.15.45:0/1349398591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf6c0b6270 msgr2=0x7fbf6c0ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.096+0000 7fbf6aa00640 1 --2- 172.21.15.45:0/1349398591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf6c0b6270 0x7fbf6c0ce5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.096+0000 7fbf6aa00640 1 -- 172.21.15.45:0/1349398591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf6c0ba2a0 msgr2=0x7fbf6c1f4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.096+0000 7fbf6aa00640 1 --2- 172.21.15.45:0/1349398591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf6c0ba2a0 0x7fbf6c1f4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.096+0000 7fbf6aa00640 1 -- 172.21.15.45:0/1349398591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf6c160980 con 0x7fbf6c0b7920 2024-04-16T16:22:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.096+0000 7fbf6b400640 1 --2- 172.21.15.45:0/1349398591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf6c0b6270 0x7fbf6c0ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.096+0000 7fbf6aa00640 1 --2- 172.21.15.45:0/1349398591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf6c0b7920 0x7fbf6c15b3c0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fbf5400c430 tx=0x7fbf5400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.096+0000 7fbf63400640 1 -- 172.21.15.45:0/1349398591 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf54011790 con 0x7fbf6c0b7920 2024-04-16T16:22:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.096+0000 7fbf63400640 1 -- 172.21.15.45:0/1349398591 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf54017070 con 0x7fbf6c0b7920 2024-04-16T16:22:28.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.096+0000 7fbf71000640 1 -- 172.21.15.45:0/1349398591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf6c160c60 con 0x7fbf6c0b7920 2024-04-16T16:22:28.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.096+0000 7fbf63400640 1 -- 172.21.15.45:0/1349398591 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf54011bb0 con 0x7fbf6c0b7920 2024-04-16T16:22:28.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.097+0000 7fbf71000640 1 -- 172.21.15.45:0/1349398591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf6c161220 con 0x7fbf6c0b7920 2024-04-16T16:22:28.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.098+0000 7fbf71000640 1 -- 172.21.15.45:0/1349398591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf6c0b4e20 con 0x7fbf6c0b7920 2024-04-16T16:22:28.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.099+0000 7fbf63400640 1 -- 172.21.15.45:0/1349398591 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 67) ==== 90878+0+0 (secure 0 0 0) 0x7fbf5401c460 con 0x7fbf6c0b7920 2024-04-16T16:22:28.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.099+0000 7fbf63400640 1 --2- 172.21.15.45:0/1349398591 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbf4806c740 0x7fbf4806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:28.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.100+0000 7fbf63400640 1 -- 172.21.15.45:0/1349398591 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6397+0+0 (secure 0 0 0) 0x7fbf5408ba10 con 0x7fbf6c0b7920 2024-04-16T16:22:28.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.100+0000 7fbf6b400640 1 --2- 172.21.15.45:0/1349398591 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbf4806c740 0x7fbf4806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:28.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.101+0000 7fbf6b400640 1 --2- 172.21.15.45:0/1349398591 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbf4806c740 0x7fbf4806ec00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c007eb0 tx=0x7fbf5c007400 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:28.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.104+0000 7fbf63400640 1 -- 172.21.15.45:0/1349398591 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbf6c0b4e20 con 0x7fbf6c0b7920 2024-04-16T16:22:28.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.241+0000 7fbf63400640 1 -- 172.21.15.45:0/1349398591 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 68) ==== 91086+0+0 (secure 0 0 0) 0x7fbf54055d70 con 0x7fbf6c0b7920 2024-04-16T16:22:28.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.312+0000 7fbf71000640 1 -- 172.21.15.45:0/1349398591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbf6c075780 con 0x7fbf6c0b7920 2024-04-16T16:22:28.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.313+0000 7fbf63400640 1 -- 172.21.15.45:0/1349398591 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v144) ==== 93+0+10 (secure 0 0 0) 0x7fbf54078180 con 0x7fbf6c0b7920 2024-04-16T16:22:28.313 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:22:28.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.315+0000 7fbf71000640 1 -- 172.21.15.45:0/1349398591 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbf4806c740 msgr2=0x7fbf4806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:28.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.315+0000 7fbf71000640 1 --2- 172.21.15.45:0/1349398591 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbf4806c740 0x7fbf4806ec00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c007eb0 tx=0x7fbf5c007400 comp rx=0 tx=0).stop 2024-04-16T16:22:28.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.315+0000 7fbf71000640 1 -- 172.21.15.45:0/1349398591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf6c0b7920 msgr2=0x7fbf6c15b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:28.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.315+0000 7fbf71000640 1 --2- 172.21.15.45:0/1349398591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf6c0b7920 0x7fbf6c15b3c0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fbf5400c430 tx=0x7fbf5400c8f0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.315+0000 7fbf71000640 1 -- 172.21.15.45:0/1349398591 shutdown_connections 2024-04-16T16:22:28.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.315+0000 7fbf71000640 1 --2- 172.21.15.45:0/1349398591 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbf4806c740 0x7fbf4806ec00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.315+0000 7fbf71000640 1 --2- 172.21.15.45:0/1349398591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf6c0ba2a0 0x7fbf6c1f4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.315+0000 7fbf71000640 1 --2- 172.21.15.45:0/1349398591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf6c0b7920 0x7fbf6c15b3c0 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.315+0000 7fbf71000640 1 --2- 172.21.15.45:0/1349398591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf6c0b6270 0x7fbf6c0ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.315+0000 7fbf71000640 1 -- 172.21.15.45:0/1349398591 >> 172.21.15.45:0/1349398591 conn(0x7fbf6c06d300 msgr2=0x7fbf6c0ca330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:28.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.316+0000 7fbf71000640 1 -- 172.21.15.45:0/1349398591 shutdown_connections 2024-04-16T16:22:28.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.316+0000 7fbf71000640 1 -- 172.21.15.45:0/1349398591 wait complete. 2024-04-16T16:22:28.336 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-16T16:22:28.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.565+0000 7f965be00640 1 Processor -- start 2024-04-16T16:22:28.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.565+0000 7f965be00640 1 -- start start 2024-04-16T16:22:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.567+0000 7f965be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f96540b5a30 0x7f96541625e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.567+0000 7f965be00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96540b5e10 con 0x7f96540d1030 2024-04-16T16:22:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.567+0000 7f965be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f965415ff70 con 0x7f965406e290 2024-04-16T16:22:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.567+0000 7f965be00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f965415f3d0 con 0x7f96540b5a30 2024-04-16T16:22:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.567+0000 7f9658a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f96540d1030 0x7f965415f860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39528/0 (socket says 172.21.15.45:39528) 2024-04-16T16:22:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.567+0000 7f9658a00640 1 -- 172.21.15.45:0/1808837691 learned_addr learned my addr 172.21.15.45:0/1808837691 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:28.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.568+0000 7f9659400640 1 --2- 172.21.15.45:0/1808837691 >> v2:172.21.15.103:3300/0 conn(0x7f96540b5a30 0x7f96541625e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.568+0000 7f9653e00640 1 -- 172.21.15.45:0/1808837691 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2523819157 0 0) 0x7f96540b5e10 con 0x7f96540d1030 2024-04-16T16:22:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.568+0000 7f9653e00640 1 -- 172.21.15.45:0/1808837691 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f963c003520 con 0x7f96540d1030 2024-04-16T16:22:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.568+0000 7f9653e00640 1 -- 172.21.15.45:0/1808837691 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1626741909 0 0) 0x7f963c003520 con 0x7f96540d1030 2024-04-16T16:22:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.568+0000 7f9653e00640 1 -- 172.21.15.45:0/1808837691 >> v2:172.21.15.103:3300/0 conn(0x7f96540b5a30 msgr2=0x7f96541625e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.568+0000 7f9653e00640 1 --2- 172.21.15.45:0/1808837691 >> v2:172.21.15.103:3300/0 conn(0x7f96540b5a30 0x7f96541625e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.568+0000 7f9653e00640 1 -- 172.21.15.45:0/1808837691 >> v1:172.21.15.103:6789/0 conn(0x7f965406e290 legacy=0x7f96540b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.568+0000 7f9653e00640 1 -- 172.21.15.45:0/1808837691 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9654162e70 con 0x7f96540d1030 2024-04-16T16:22:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.568+0000 7f9653e00640 1 -- 172.21.15.45:0/1808837691 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f9640001670 con 0x7f96540d1030 2024-04-16T16:22:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.568+0000 7f9653e00640 1 -- 172.21.15.45:0/1808837691 >> v1:172.21.15.45:6789/0 conn(0x7f96540d1030 legacy=0x7f965415f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9653e00640 1 --2- 172.21.15.45:0/1808837691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f963c003a00 0x7f963c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9653e00640 1 --2- 172.21.15.45:0/1808837691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963c004800 0x7f963c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9653e00640 1 --2- 172.21.15.45:0/1808837691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f963c0051c0 0x7f963c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9653e00640 1 -- 172.21.15.45:0/1808837691 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f965415f3d0 con 0x7f963c004800 2024-04-16T16:22:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9653e00640 1 -- 172.21.15.45:0/1808837691 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f963c00c0c0 con 0x7f963c0051c0 2024-04-16T16:22:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9653e00640 1 -- 172.21.15.45:0/1808837691 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f963c00c2c0 con 0x7f963c003a00 2024-04-16T16:22:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9658a00640 1 --2- 172.21.15.45:0/1808837691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963c004800 0x7f963c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9659400640 1 --2- 172.21.15.45:0/1808837691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f963c003a00 0x7f963c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9658a00640 1 -- 172.21.15.45:0/1808837691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f963c003a00 msgr2=0x7f963c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9658a00640 1 --2- 172.21.15.45:0/1808837691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f963c003a00 0x7f963c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9659e00640 1 --2- 172.21.15.45:0/1808837691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f963c0051c0 0x7f963c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9658a00640 1 -- 172.21.15.45:0/1808837691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f963c0051c0 msgr2=0x7f963c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9658a00640 1 --2- 172.21.15.45:0/1808837691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f963c0051c0 0x7f963c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9658a00640 1 -- 172.21.15.45:0/1808837691 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f963c003520 con 0x7f963c004800 2024-04-16T16:22:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9659400640 1 --2- 172.21.15.45:0/1808837691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f963c003a00 0x7f963c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9659e00640 1 --2- 172.21.15.45:0/1808837691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f963c0051c0 0x7f963c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.569+0000 7f9659e00640 1 -- 172.21.15.45:0/1808837691 reap_dead start 2024-04-16T16:22:28.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.571+0000 7f9658a00640 1 --2- 172.21.15.45:0/1808837691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963c004800 0x7f963c004c80 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f964000c110 tx=0x7f963c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e0ea85350987fe server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:28.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.571+0000 7f9653e00640 1 -- 172.21.15.45:0/1808837691 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9640007c40 con 0x7f963c004800 2024-04-16T16:22:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.571+0000 7f9653e00640 1 -- 172.21.15.45:0/1808837691 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9640030070 con 0x7f963c004800 2024-04-16T16:22:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.571+0000 7f9653e00640 1 -- 172.21.15.45:0/1808837691 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9640005d70 con 0x7f963c004800 2024-04-16T16:22:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.571+0000 7f965be00640 1 -- 172.21.15.45:0/1808837691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963c004800 msgr2=0x7f963c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.571+0000 7f965be00640 1 --2- 172.21.15.45:0/1808837691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963c004800 0x7f963c004c80 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f964000c110 tx=0x7f963c005830 comp rx=0 tx=0).stop 2024-04-16T16:22:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.571+0000 7f965be00640 1 -- 172.21.15.45:0/1808837691 shutdown_connections 2024-04-16T16:22:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.571+0000 7f965be00640 1 --2- 172.21.15.45:0/1808837691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963c004800 0x7f963c004c80 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.571+0000 7f965be00640 1 -- 172.21.15.45:0/1808837691 >> 172.21.15.45:0/1808837691 conn(0x7f96540b3970 msgr2=0x7f96540b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.572+0000 7f965be00640 1 -- 172.21.15.45:0/1808837691 shutdown_connections 2024-04-16T16:22:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.572+0000 7f965be00640 1 -- 172.21.15.45:0/1808837691 wait complete. 2024-04-16T16:22:28.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.573+0000 7f965be00640 1 Processor -- start 2024-04-16T16:22:28.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.573+0000 7f965be00640 1 -- start start 2024-04-16T16:22:28.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.574+0000 7f965be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963c004800 0x7f9654158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:28.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.574+0000 7f965be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f965414d8f0 0x7f9654156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:28.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.574+0000 7f965be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f96540ce020 0x7f96540b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:28.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.574+0000 7f965be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f965415eff0 con 0x7f963c004800 2024-04-16T16:22:28.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.574+0000 7f965be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9654163430 con 0x7f965414d8f0 2024-04-16T16:22:28.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.574+0000 7f965be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f965406e6e0 con 0x7f96540ce020 2024-04-16T16:22:28.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.575+0000 7f9659400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963c004800 0x7f9654158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:28.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.575+0000 7f9659400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963c004800 0x7f9654158560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53266/0 (socket says 172.21.15.45:53266) 2024-04-16T16:22:28.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.575+0000 7f9659400640 1 -- 172.21.15.45:0/1038750232 learned_addr learned my addr 172.21.15.45:0/1038750232 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.575+0000 7f9659400640 1 -- 172.21.15.45:0/1038750232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f96540ce020 msgr2=0x7f96540b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.575+0000 7f9659400640 1 --2- 172.21.15.45:0/1038750232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f96540ce020 0x7f96540b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.575+0000 7f9659400640 1 -- 172.21.15.45:0/1038750232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f965414d8f0 msgr2=0x7f9654156aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.575+0000 7f9658a00640 1 --2- 172.21.15.45:0/1038750232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f965414d8f0 0x7f9654156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.575+0000 7f9659400640 1 --2- 172.21.15.45:0/1038750232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f965414d8f0 0x7f9654156aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.575+0000 7f9659400640 1 -- 172.21.15.45:0/1038750232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9654157220 con 0x7f963c004800 2024-04-16T16:22:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.575+0000 7f9659400640 1 --2- 172.21.15.45:0/1038750232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963c004800 0x7f9654158560 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f96480107f0 tx=0x7f9648010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.576+0000 7f9651600640 1 -- 172.21.15.45:0/1038750232 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964800f040 con 0x7f963c004800 2024-04-16T16:22:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.576+0000 7f9651600640 1 -- 172.21.15.45:0/1038750232 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f964800eb00 con 0x7f963c004800 2024-04-16T16:22:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.576+0000 7f9651600640 1 -- 172.21.15.45:0/1038750232 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9648010ea0 con 0x7f963c004800 2024-04-16T16:22:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.576+0000 7f965be00640 1 -- 172.21.15.45:0/1038750232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96540b0660 con 0x7f963c004800 2024-04-16T16:22:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.576+0000 7f965be00640 1 -- 172.21.15.45:0/1038750232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96540b0c20 con 0x7f963c004800 2024-04-16T16:22:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.577+0000 7f965be00640 1 -- 172.21.15.45:0/1038750232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9654075780 con 0x7f963c004800 2024-04-16T16:22:28.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.578+0000 7f9651600640 1 -- 172.21.15.45:0/1038750232 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) ==== 91086+0+0 (secure 0 0 0) 0x7f964801c400 con 0x7f963c004800 2024-04-16T16:22:28.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.579+0000 7f9651600640 1 --2- 172.21.15.45:0/1038750232 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f963406c980 0x7f963406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:28.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.579+0000 7f9651600640 1 -- 172.21.15.45:0/1038750232 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6397+0+0 (secure 0 0 0) 0x7f964808c400 con 0x7f963c004800 2024-04-16T16:22:28.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.580+0000 7f9658a00640 1 --2- 172.21.15.45:0/1038750232 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f963406c980 0x7f963406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:28.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.581+0000 7f9658a00640 1 --2- 172.21.15.45:0/1038750232 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f963406c980 0x7f963406ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9640034000 tx=0x7f9640039040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:28.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.584+0000 7f9651600640 1 -- 172.21.15.45:0/1038750232 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9648018030 con 0x7f963c004800 2024-04-16T16:22:28.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.790+0000 7f965be00640 1 -- 172.21.15.45:0/1038750232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9654159380 con 0x7f963c004800 2024-04-16T16:22:28.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.791+0000 7f9651600640 1 -- 172.21.15.45:0/1038750232 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v144) ==== 104+0+11 (secure 0 0 0) 0x7f9648059ed0 con 0x7f963c004800 2024-04-16T16:22:28.791 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-04-16T16:22:28.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.793+0000 7f965be00640 1 -- 172.21.15.45:0/1038750232 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f963406c980 msgr2=0x7f963406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:28.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.793+0000 7f965be00640 1 --2- 172.21.15.45:0/1038750232 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f963406c980 0x7f963406ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9640034000 tx=0x7f9640039040 comp rx=0 tx=0).stop 2024-04-16T16:22:28.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.793+0000 7f965be00640 1 -- 172.21.15.45:0/1038750232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963c004800 msgr2=0x7f9654158560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:28.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.793+0000 7f965be00640 1 --2- 172.21.15.45:0/1038750232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963c004800 0x7f9654158560 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f96480107f0 tx=0x7f9648010cb0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.794+0000 7f965be00640 1 -- 172.21.15.45:0/1038750232 shutdown_connections 2024-04-16T16:22:28.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.794+0000 7f965be00640 1 --2- 172.21.15.45:0/1038750232 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f963406c980 0x7f963406ee40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.794+0000 7f965be00640 1 --2- 172.21.15.45:0/1038750232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f96540ce020 0x7f96540b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.794+0000 7f965be00640 1 --2- 172.21.15.45:0/1038750232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f965414d8f0 0x7f9654156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.794+0000 7f965be00640 1 --2- 172.21.15.45:0/1038750232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963c004800 0x7f9654158560 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:28.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.794+0000 7f965be00640 1 -- 172.21.15.45:0/1038750232 >> 172.21.15.45:0/1038750232 conn(0x7f96540b3970 msgr2=0x7f965406b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:28.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.794+0000 7f965be00640 1 -- 172.21.15.45:0/1038750232 shutdown_connections 2024-04-16T16:22:28.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:28.794+0000 7f965be00640 1 -- 172.21.15.45:0/1038750232 wait complete. 2024-04-16T16:22:28.811 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-16T16:22:29.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.032+0000 7fc13fe00640 1 Processor -- start 2024-04-16T16:22:29.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.032+0000 7fc13fe00640 1 -- start start 2024-04-16T16:22:29.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.034+0000 7fc13fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1400b58f0 0x7fc1400b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:29.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.034+0000 7fc13fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc14006e460 0x7fc14006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:29.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.034+0000 7fc13fe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc140146590 con 0x7fc1401461b0 2024-04-16T16:22:29.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.034+0000 7fc13fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc14006dfd0 con 0x7fc14006e460 2024-04-16T16:22:29.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.034+0000 7fc13fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc1400b6290 con 0x7fc1400b58f0 2024-04-16T16:22:29.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.035+0000 7fc13e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1400b58f0 0x7fc1400b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:29.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.035+0000 7fc13f400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc14006e460 0x7fc14006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:29.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.035+0000 7fc13f400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc14006e460 0x7fc14006e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:53482/0 (socket says 172.21.15.45:53482) 2024-04-16T16:22:29.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.035+0000 7fc13f400640 1 -- 172.21.15.45:0/596893086 learned_addr learned my addr 172.21.15.45:0/596893086 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:29.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.035+0000 7fc13ea00640 1 --1- 172.21.15.45:0/596893086 >> v1:172.21.15.103:6789/0 conn(0x7fc1401461b0 0x7fc140160cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:46734/0 (socket says 172.21.15.45:46734) 2024-04-16T16:22:29.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.036+0000 7fc13f400640 1 -- 172.21.15.45:0/596893086 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1400b58f0 msgr2=0x7fc1400b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.036+0000 7fc13f400640 1 --2- 172.21.15.45:0/596893086 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1400b58f0 0x7fc1400b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.036+0000 7fc13f400640 1 -- 172.21.15.45:0/596893086 >> v1:172.21.15.103:6789/0 conn(0x7fc1401461b0 legacy=0x7fc140160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.036+0000 7fc13f400640 1 -- 172.21.15.45:0/596893086 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1400c5750 con 0x7fc14006e460 2024-04-16T16:22:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.037+0000 7fc13f400640 1 --2- 172.21.15.45:0/596893086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc14006e460 0x7fc14006e8f0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc138009e50 tx=0x7fc13802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97341eac60c58a01 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:29.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.037+0000 7fc13d600640 1 -- 172.21.15.45:0/596893086 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13802cb00 con 0x7fc14006e460 2024-04-16T16:22:29.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.037+0000 7fc13d600640 1 -- 172.21.15.45:0/596893086 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc13802cca0 con 0x7fc14006e460 2024-04-16T16:22:29.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.037+0000 7fc13d600640 1 -- 172.21.15.45:0/596893086 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138034610 con 0x7fc14006e460 2024-04-16T16:22:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.038+0000 7fc13fe00640 1 -- 172.21.15.45:0/596893086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc14006e460 msgr2=0x7fc14006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.038+0000 7fc13fe00640 1 --2- 172.21.15.45:0/596893086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc14006e460 0x7fc14006e8f0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc138009e50 tx=0x7fc13802bf60 comp rx=0 tx=0).stop 2024-04-16T16:22:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.038+0000 7fc13fe00640 1 -- 172.21.15.45:0/596893086 shutdown_connections 2024-04-16T16:22:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.038+0000 7fc13fe00640 1 --2- 172.21.15.45:0/596893086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc14006e460 0x7fc14006e8f0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.038+0000 7fc13fe00640 1 --2- 172.21.15.45:0/596893086 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1400b58f0 0x7fc1400b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.038+0000 7fc13fe00640 1 -- 172.21.15.45:0/596893086 >> 172.21.15.45:0/596893086 conn(0x7fc14006b930 msgr2=0x7fc14006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.038+0000 7fc13fe00640 1 -- 172.21.15.45:0/596893086 shutdown_connections 2024-04-16T16:22:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.038+0000 7fc13fe00640 1 -- 172.21.15.45:0/596893086 wait complete. 2024-04-16T16:22:29.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.040+0000 7fc13fe00640 1 Processor -- start 2024-04-16T16:22:29.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.040+0000 7fc13fe00640 1 -- start start 2024-04-16T16:22:29.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.041+0000 7fc13fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14006e460 0x7fc1401f4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:29.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.041+0000 7fc13fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1400b58f0 0x7fc140144190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:29.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.041+0000 7fc13fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1401461b0 0x7fc1401e4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:29.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.041+0000 7fc13fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc1400c5cd0 con 0x7fc14006e460 2024-04-16T16:22:29.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.041+0000 7fc13fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc1400c59f0 con 0x7fc1401461b0 2024-04-16T16:22:29.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.041+0000 7fc13fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc1400c5fb0 con 0x7fc1400b58f0 2024-04-16T16:22:29.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.041+0000 7fc13ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14006e460 0x7fc1401f4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:29.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.041+0000 7fc13ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14006e460 0x7fc1401f4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53276/0 (socket says 172.21.15.45:53276) 2024-04-16T16:22:29.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.041+0000 7fc13ea00640 1 -- 172.21.15.45:0/3785089785 learned_addr learned my addr 172.21.15.45:0/3785089785 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:29.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.042+0000 7fc13f400640 1 --2- 172.21.15.45:0/3785089785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1401461b0 0x7fc1401e4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:29.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.042+0000 7fc13e000640 1 --2- 172.21.15.45:0/3785089785 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1400b58f0 0x7fc140144190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:29.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.042+0000 7fc13ea00640 1 -- 172.21.15.45:0/3785089785 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1400b58f0 msgr2=0x7fc140144190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:29.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.042+0000 7fc13ea00640 1 --2- 172.21.15.45:0/3785089785 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1400b58f0 0x7fc140144190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.042+0000 7fc13ea00640 1 -- 172.21.15.45:0/3785089785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1401461b0 msgr2=0x7fc1401e4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:29.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.042+0000 7fc13ea00640 1 --2- 172.21.15.45:0/3785089785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1401461b0 0x7fc1401e4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.042+0000 7fc13ea00640 1 -- 172.21.15.45:0/3785089785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1401430f0 con 0x7fc14006e460 2024-04-16T16:22:29.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.042+0000 7fc13f400640 1 --2- 172.21.15.45:0/3785089785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1401461b0 0x7fc1401e4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:29.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.042+0000 7fc13ea00640 1 --2- 172.21.15.45:0/3785089785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14006e460 0x7fc1401f4f90 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fc13000c3f0 tx=0x7fc13000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:29.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.042+0000 7fc136a00640 1 -- 172.21.15.45:0/3785089785 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1300074e0 con 0x7fc14006e460 2024-04-16T16:22:29.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.042+0000 7fc136a00640 1 -- 172.21.15.45:0/3785089785 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc130007680 con 0x7fc14006e460 2024-04-16T16:22:29.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.043+0000 7fc13fe00640 1 -- 172.21.15.45:0/3785089785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1401433d0 con 0x7fc14006e460 2024-04-16T16:22:29.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.043+0000 7fc136a00640 1 -- 172.21.15.45:0/3785089785 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130004a00 con 0x7fc14006e460 2024-04-16T16:22:29.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.043+0000 7fc13fe00640 1 -- 172.21.15.45:0/3785089785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc140143990 con 0x7fc14006e460 2024-04-16T16:22:29.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.044+0000 7fc13fe00640 1 -- 172.21.15.45:0/3785089785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc140075780 con 0x7fc14006e460 2024-04-16T16:22:29.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.045+0000 7fc136a00640 1 -- 172.21.15.45:0/3785089785 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) ==== 91086+0+0 (secure 0 0 0) 0x7fc13001a020 con 0x7fc14006e460 2024-04-16T16:22:29.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.046+0000 7fc136a00640 1 --2- 172.21.15.45:0/3785089785 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc12006c980 0x7fc12006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:29.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.046+0000 7fc136a00640 1 -- 172.21.15.45:0/3785089785 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6397+0+0 (secure 0 0 0) 0x7fc13008b680 con 0x7fc14006e460 2024-04-16T16:22:29.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.047+0000 7fc13e000640 1 --2- 172.21.15.45:0/3785089785 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc12006c980 0x7fc12006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:29.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.048+0000 7fc13e000640 1 --2- 172.21.15.45:0/3785089785 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc12006c980 0x7fc12006ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc140152350 tx=0x7fc12c008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:29.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.051+0000 7fc136a00640 1 -- 172.21.15.45:0/3785089785 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc130010300 con 0x7fc14006e460 2024-04-16T16:22:29.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.248+0000 7fc13fe00640 1 -- 172.21.15.45:0/3785089785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc1400b30f0 con 0x7fc14006e460 2024-04-16T16:22:29.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.249+0000 7fc136a00640 1 -- 172.21.15.45:0/3785089785 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v144) ==== 100+0+11 (secure 0 0 0) 0x7fc1300597a0 con 0x7fc14006e460 2024-04-16T16:22:29.249 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-04-16T16:22:29.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.252+0000 7fc13fe00640 1 -- 172.21.15.45:0/3785089785 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc12006c980 msgr2=0x7fc12006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:29.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.252+0000 7fc13fe00640 1 --2- 172.21.15.45:0/3785089785 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc12006c980 0x7fc12006ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc140152350 tx=0x7fc12c008040 comp rx=0 tx=0).stop 2024-04-16T16:22:29.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.252+0000 7fc13fe00640 1 -- 172.21.15.45:0/3785089785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14006e460 msgr2=0x7fc1401f4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:29.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.252+0000 7fc13fe00640 1 --2- 172.21.15.45:0/3785089785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14006e460 0x7fc1401f4f90 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fc13000c3f0 tx=0x7fc13000c8b0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.252+0000 7fc13fe00640 1 -- 172.21.15.45:0/3785089785 shutdown_connections 2024-04-16T16:22:29.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.252+0000 7fc13fe00640 1 --2- 172.21.15.45:0/3785089785 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc12006c980 0x7fc12006ee40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.252+0000 7fc13fe00640 1 --2- 172.21.15.45:0/3785089785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1401461b0 0x7fc1401e4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.252+0000 7fc13fe00640 1 --2- 172.21.15.45:0/3785089785 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc1400b58f0 0x7fc140144190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.252+0000 7fc13fe00640 1 --2- 172.21.15.45:0/3785089785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14006e460 0x7fc1401f4f90 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.252+0000 7fc13fe00640 1 -- 172.21.15.45:0/3785089785 >> 172.21.15.45:0/3785089785 conn(0x7fc14006b930 msgr2=0x7fc14006f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:29.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.252+0000 7fc13fe00640 1 -- 172.21.15.45:0/3785089785 shutdown_connections 2024-04-16T16:22:29.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.252+0000 7fc13fe00640 1 -- 172.21.15.45:0/3785089785 wait complete. 2024-04-16T16:22:29.272 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:22:29.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.491+0000 7f0b36e00640 1 Processor -- start 2024-04-16T16:22:29.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.492+0000 7f0b36e00640 1 -- start start 2024-04-16T16:22:29.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.494+0000 7f0b36e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b300d1180 0x7f0b300b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:29.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.494+0000 7f0b36e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0b30162f70 0x7f0b3006b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:29.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.494+0000 7f0b36e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b3015fa20 con 0x7f0b300d0da0 2024-04-16T16:22:29.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.494+0000 7f0b36e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0b3015fd00 con 0x7f0b30162f70 2024-04-16T16:22:29.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.494+0000 7f0b36e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0b3015ffe0 con 0x7f0b300d1180 2024-04-16T16:22:29.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.494+0000 7f0b35a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0b300d0da0 0x7f0b3015f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39530/0 (socket says 172.21.15.45:39530) 2024-04-16T16:22:29.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.494+0000 7f0b35a00640 1 -- 172.21.15.45:0/854357922 learned_addr learned my addr 172.21.15.45:0/854357922 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:29.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.495+0000 7f0b35000640 1 --2- 172.21.15.45:0/854357922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b300d1180 0x7f0b300b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:29.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.495+0000 7f0b36400640 1 --2- 172.21.15.45:0/854357922 >> v2:172.21.15.103:3300/0 conn(0x7f0b30162f70 0x7f0b3006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:29.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.495+0000 7f0b2fe00640 1 -- 172.21.15.45:0/854357922 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3547174030 0 0) 0x7f0b3015fa20 con 0x7f0b300d0da0 2024-04-16T16:22:29.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.495+0000 7f0b2fe00640 1 -- 172.21.15.45:0/854357922 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b18003520 con 0x7f0b300d0da0 2024-04-16T16:22:29.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.495+0000 7f0b2fe00640 1 -- 172.21.15.45:0/854357922 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1321502627 0 0) 0x7f0b18003520 con 0x7f0b300d0da0 2024-04-16T16:22:29.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.495+0000 7f0b2fe00640 1 -- 172.21.15.45:0/854357922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b300d1180 msgr2=0x7f0b300b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:29.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.495+0000 7f0b2fe00640 1 --2- 172.21.15.45:0/854357922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b300d1180 0x7f0b300b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.496+0000 7f0b2fe00640 1 -- 172.21.15.45:0/854357922 >> v2:172.21.15.103:3300/0 conn(0x7f0b30162f70 msgr2=0x7f0b3006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.496+0000 7f0b2fe00640 1 --2- 172.21.15.45:0/854357922 >> v2:172.21.15.103:3300/0 conn(0x7f0b30162f70 0x7f0b3006b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.496+0000 7f0b2fe00640 1 -- 172.21.15.45:0/854357922 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b30162770 con 0x7f0b300d0da0 2024-04-16T16:22:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.496+0000 7f0b2fe00640 1 -- 172.21.15.45:0/854357922 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f0b240019f0 con 0x7f0b300d0da0 2024-04-16T16:22:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.496+0000 7f0b2fe00640 1 -- 172.21.15.45:0/854357922 >> v1:172.21.15.45:6789/0 conn(0x7f0b300d0da0 legacy=0x7f0b3015f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.496+0000 7f0b2fe00640 1 --2- 172.21.15.45:0/854357922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b180039b0 0x7f0b18003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.496+0000 7f0b2fe00640 1 --2- 172.21.15.45:0/854357922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b18004800 0x7f0b18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.496+0000 7f0b2fe00640 1 --2- 172.21.15.45:0/854357922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b180051c0 0x7f0b1800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.496+0000 7f0b2fe00640 1 -- 172.21.15.45:0/854357922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3015fd00 con 0x7f0b180039b0 2024-04-16T16:22:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.496+0000 7f0b2fe00640 1 -- 172.21.15.45:0/854357922 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3015ffe0 con 0x7f0b18004800 2024-04-16T16:22:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.496+0000 7f0b2fe00640 1 -- 172.21.15.45:0/854357922 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0b1800bf80 con 0x7f0b180051c0 2024-04-16T16:22:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.496+0000 7f0b35000640 1 --2- 172.21.15.45:0/854357922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b300d1180 0x7f0b300b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:22:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.496+0000 7f0b35a00640 1 --2- 172.21.15.45:0/854357922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b180039b0 0x7f0b18003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.496+0000 7f0b36400640 1 --2- 172.21.15.45:0/854357922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b180051c0 0x7f0b1800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.497+0000 7f0b35a00640 1 -- 172.21.15.45:0/854357922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b180051c0 msgr2=0x7f0b1800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:29.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.497+0000 7f0b35a00640 1 --2- 172.21.15.45:0/854357922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b180051c0 0x7f0b1800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.497+0000 7f0b35a00640 1 -- 172.21.15.45:0/854357922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b18004800 msgr2=0x7f0b18004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:29.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.497+0000 7f0b35000640 1 --2- 172.21.15.45:0/854357922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b18004800 0x7f0b18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:29.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.497+0000 7f0b35a00640 1 --2- 172.21.15.45:0/854357922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b18004800 0x7f0b18004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.497+0000 7f0b35a00640 1 -- 172.21.15.45:0/854357922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b18003520 con 0x7f0b180039b0 2024-04-16T16:22:29.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.497+0000 7f0b35000640 1 --2- 172.21.15.45:0/854357922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b18004800 0x7f0b18004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:29.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.498+0000 7f0b35a00640 1 --2- 172.21.15.45:0/854357922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b180039b0 0x7f0b18003dd0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400a7a0 tx=0x7f0b18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de48502d420c53f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:29.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.498+0000 7f0b2fe00640 1 -- 172.21.15.45:0/854357922 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2400a3f0 con 0x7f0b180039b0 2024-04-16T16:22:29.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.498+0000 7f0b2fe00640 1 -- 172.21.15.45:0/854357922 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b24034630 con 0x7f0b180039b0 2024-04-16T16:22:29.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.498+0000 7f0b2fe00640 1 -- 172.21.15.45:0/854357922 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b24034a40 con 0x7f0b180039b0 2024-04-16T16:22:29.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.499+0000 7f0b36e00640 1 -- 172.21.15.45:0/854357922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b180039b0 msgr2=0x7f0b18003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:29.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.499+0000 7f0b36e00640 1 --2- 172.21.15.45:0/854357922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b180039b0 0x7f0b18003dd0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400a7a0 tx=0x7f0b18005830 comp rx=0 tx=0).stop 2024-04-16T16:22:29.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.499+0000 7f0b36400640 1 -- 172.21.15.45:0/854357922 reap_dead start 2024-04-16T16:22:29.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.499+0000 7f0b36e00640 1 -- 172.21.15.45:0/854357922 shutdown_connections 2024-04-16T16:22:29.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.499+0000 7f0b36e00640 1 -- 172.21.15.45:0/854357922 >> 172.21.15.45:0/854357922 conn(0x7f0b3006f0e0 msgr2=0x7f0b3006f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:29.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.499+0000 7f0b36e00640 1 -- 172.21.15.45:0/854357922 shutdown_connections 2024-04-16T16:22:29.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.499+0000 7f0b36e00640 1 -- 172.21.15.45:0/854357922 wait complete. 2024-04-16T16:22:29.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.501+0000 7f0b36e00640 1 Processor -- start 2024-04-16T16:22:29.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.501+0000 7f0b36e00640 1 -- start start 2024-04-16T16:22:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.502+0000 7f0b36e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b180039b0 0x7f0b3013da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.502+0000 7f0b36e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b30151ad0 0x7f0b300b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.502+0000 7f0b36e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b300b12c0 0x7f0b30140640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.502+0000 7f0b36e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0b30163580 con 0x7f0b300b12c0 2024-04-16T16:22:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.502+0000 7f0b36e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0b30163400 con 0x7f0b180039b0 2024-04-16T16:22:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.502+0000 7f0b36e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0b30163700 con 0x7f0b30151ad0 2024-04-16T16:22:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.503+0000 7f0b36400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b300b12c0 0x7f0b30140640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.503+0000 7f0b36400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b300b12c0 0x7f0b30140640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53286/0 (socket says 172.21.15.45:53286) 2024-04-16T16:22:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.503+0000 7f0b36400640 1 -- 172.21.15.45:0/1264028706 learned_addr learned my addr 172.21.15.45:0/1264028706 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.503+0000 7f0b35a00640 1 --2- 172.21.15.45:0/1264028706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b180039b0 0x7f0b3013da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.503+0000 7f0b35000640 1 --2- 172.21.15.45:0/1264028706 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b30151ad0 0x7f0b300b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.503+0000 7f0b36400640 1 -- 172.21.15.45:0/1264028706 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b30151ad0 msgr2=0x7f0b300b0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.503+0000 7f0b36400640 1 --2- 172.21.15.45:0/1264028706 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b30151ad0 0x7f0b300b0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.503+0000 7f0b36400640 1 -- 172.21.15.45:0/1264028706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b180039b0 msgr2=0x7f0b3013da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.503+0000 7f0b36400640 1 --2- 172.21.15.45:0/1264028706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b180039b0 0x7f0b3013da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.503+0000 7f0b36400640 1 -- 172.21.15.45:0/1264028706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b3013df90 con 0x7f0b300b12c0 2024-04-16T16:22:29.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.503+0000 7f0b36400640 1 --2- 172.21.15.45:0/1264028706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b300b12c0 0x7f0b30140640 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f0b300715c0 tx=0x7f0b28011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:29.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.504+0000 7f0b2d600640 1 -- 172.21.15.45:0/1264028706 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b28019020 con 0x7f0b300b12c0 2024-04-16T16:22:29.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.504+0000 7f0b2d600640 1 -- 172.21.15.45:0/1264028706 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b28003610 con 0x7f0b300b12c0 2024-04-16T16:22:29.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.504+0000 7f0b2d600640 1 -- 172.21.15.45:0/1264028706 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b28003a00 con 0x7f0b300b12c0 2024-04-16T16:22:29.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.504+0000 7f0b36e00640 1 -- 172.21.15.45:0/1264028706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b30150d10 con 0x7f0b300b12c0 2024-04-16T16:22:29.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.504+0000 7f0b36e00640 1 -- 172.21.15.45:0/1264028706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b301512d0 con 0x7f0b300b12c0 2024-04-16T16:22:29.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.505+0000 7f0b36e00640 1 -- 172.21.15.45:0/1264028706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b30075780 con 0x7f0b300b12c0 2024-04-16T16:22:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.506+0000 7f0b2d600640 1 -- 172.21.15.45:0/1264028706 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) ==== 91086+0+0 (secure 0 0 0) 0x7f0b2801c020 con 0x7f0b300b12c0 2024-04-16T16:22:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.507+0000 7f0b2d600640 1 --2- 172.21.15.45:0/1264028706 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0b1006c980 0x7f0b1006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.507+0000 7f0b2d600640 1 -- 172.21.15.45:0/1264028706 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6397+0+0 (secure 0 0 0) 0x7f0b2808bff0 con 0x7f0b300b12c0 2024-04-16T16:22:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.508+0000 7f0b35a00640 1 --2- 172.21.15.45:0/1264028706 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0b1006c980 0x7f0b1006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:29.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.509+0000 7f0b35a00640 1 --2- 172.21.15.45:0/1264028706 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0b1006c980 0x7f0b1006ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0b24034000 tx=0x7f0b24009030 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:29.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.512+0000 7f0b2d600640 1 -- 172.21.15.45:0/1264028706 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0b28023030 con 0x7f0b300b12c0 2024-04-16T16:22:29.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.758+0000 7f0b36e00640 1 -- 172.21.15.45:0/1264028706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b30070f40 con 0x7f0b300b12c0 2024-04-16T16:22:29.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.759+0000 7f0b2d600640 1 -- 172.21.15.45:0/1264028706 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5213 (secure 0 0 0) 0x7f0b3013df90 con 0x7f0b300b12c0 2024-04-16T16:22:29.760 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:29.760 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2585865379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2585865379},{"type":"v1","addr":"172.21.15.45:6834","nonce":2585865379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6590,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/777087937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":777087937},{"type":"v1","addr":"172.21.15.45:6835","nonce":777087937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:21:26.066855+0000","modified":"2024-04-16T16:22:25.276355+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6596,"mds_1":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":1,"incarnation":102,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6835/2561260581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2561260581},{"type":"v1","addr":"172.21.15.103:6835","nonce":2561260581}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6596":{"gid":6596,"name":"d","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6837/3252460318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3252460318},{"type":"v1","addr":"172.21.15.103:6837","nonce":3252460318}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6596,"qdb_cluster":[6593,6596]},"id":10}]} 2024-04-16T16:22:29.760 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 105 2024-04-16T16:22:29.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.762+0000 7f0b36e00640 1 -- 172.21.15.45:0/1264028706 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0b1006c980 msgr2=0x7f0b1006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:29.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.762+0000 7f0b36e00640 1 --2- 172.21.15.45:0/1264028706 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0b1006c980 0x7f0b1006ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0b24034000 tx=0x7f0b24009030 comp rx=0 tx=0).stop 2024-04-16T16:22:29.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.762+0000 7f0b36e00640 1 -- 172.21.15.45:0/1264028706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b300b12c0 msgr2=0x7f0b30140640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:29.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.762+0000 7f0b36e00640 1 --2- 172.21.15.45:0/1264028706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b300b12c0 0x7f0b30140640 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f0b300715c0 tx=0x7f0b28011920 comp rx=0 tx=0).stop 2024-04-16T16:22:29.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.762+0000 7f0b36e00640 1 -- 172.21.15.45:0/1264028706 shutdown_connections 2024-04-16T16:22:29.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.762+0000 7f0b36e00640 1 --2- 172.21.15.45:0/1264028706 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0b1006c980 0x7f0b1006ee40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.762+0000 7f0b36e00640 1 --2- 172.21.15.45:0/1264028706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b300b12c0 0x7f0b30140640 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.762+0000 7f0b36e00640 1 --2- 172.21.15.45:0/1264028706 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b30151ad0 0x7f0b300b0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.762+0000 7f0b36e00640 1 --2- 172.21.15.45:0/1264028706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b180039b0 0x7f0b3013da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:29.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.762+0000 7f0b36e00640 1 -- 172.21.15.45:0/1264028706 >> 172.21.15.45:0/1264028706 conn(0x7f0b3006f0e0 msgr2=0x7f0b300b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:29.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.763+0000 7f0b36e00640 1 -- 172.21.15.45:0/1264028706 shutdown_connections 2024-04-16T16:22:29.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:29.763+0000 7f0b36e00640 1 -- 172.21.15.45:0/1264028706 wait complete. 2024-04-16T16:22:29.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:22:30.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.004+0000 7f363ee00640 1 Processor -- start 2024-04-16T16:22:30.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.004+0000 7f363ee00640 1 -- start start 2024-04-16T16:22:30.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.007+0000 7f363ee00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3638165b30 0x7f363816f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:30.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.007+0000 7f363ee00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f363816f590 0x7f3638071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:30.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.007+0000 7f363ee00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3638165f10 con 0x7f3638072480 2024-04-16T16:22:30.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.007+0000 7f363ee00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f363816fff0 con 0x7f363816f590 2024-04-16T16:22:30.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.007+0000 7f363ee00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f363816d0c0 con 0x7f3638165b30 2024-04-16T16:22:30.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.007+0000 7f363e400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3638072480 0x7f36380b2560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39538/0 (socket says 172.21.15.45:39538) 2024-04-16T16:22:30.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.007+0000 7f363e400640 1 -- 172.21.15.45:0/2874807073 learned_addr learned my addr 172.21.15.45:0/2874807073 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:30.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.008+0000 7f363d000640 1 --2- 172.21.15.45:0/2874807073 >> v2:172.21.15.103:3300/0 conn(0x7f363816f590 0x7f3638071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:30.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.008+0000 7f363da00640 1 --2- 172.21.15.45:0/2874807073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3638165b30 0x7f363816f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:30.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.008+0000 7f3637e00640 1 -- 172.21.15.45:0/2874807073 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2352171227 0 0) 0x7f3638165f10 con 0x7f3638072480 2024-04-16T16:22:30.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.016+0000 7f3637e00640 1 -- 172.21.15.45:0/2874807073 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3620003500 con 0x7f3638072480 2024-04-16T16:22:30.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.016+0000 7f3637e00640 1 -- 172.21.15.45:0/2874807073 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3018545457 0 0) 0x7f3620003500 con 0x7f3638072480 2024-04-16T16:22:30.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.016+0000 7f3637e00640 1 -- 172.21.15.45:0/2874807073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3638165b30 msgr2=0x7f363816f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:30.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.016+0000 7f3637e00640 1 --2- 172.21.15.45:0/2874807073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3638165b30 0x7f363816f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.016+0000 7f3637e00640 1 -- 172.21.15.45:0/2874807073 >> v2:172.21.15.103:3300/0 conn(0x7f363816f590 msgr2=0x7f3638071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:30.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.016+0000 7f3637e00640 1 --2- 172.21.15.45:0/2874807073 >> v2:172.21.15.103:3300/0 conn(0x7f363816f590 0x7f3638071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.016+0000 7f3637e00640 1 -- 172.21.15.45:0/2874807073 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f363816fa30 con 0x7f3638072480 2024-04-16T16:22:30.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.016+0000 7f3637e00640 1 -- 172.21.15.45:0/2874807073 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f36300019f0 con 0x7f3638072480 2024-04-16T16:22:30.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.016+0000 7f3637e00640 1 -- 172.21.15.45:0/2874807073 >> v1:172.21.15.45:6789/0 conn(0x7f3638072480 legacy=0x7f36380b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:30.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.016+0000 7f3637e00640 1 --2- 172.21.15.45:0/2874807073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3638165b30 0x7f3620003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:30.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.016+0000 7f3637e00640 1 --2- 172.21.15.45:0/2874807073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36200040e0 0x7f3620004560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:30.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.017+0000 7f3637e00640 1 --2- 172.21.15.45:0/2874807073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3620004bd0 0x7f3620005050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:30.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.017+0000 7f3637e00640 1 -- 172.21.15.45:0/2874807073 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f363816fff0 con 0x7f3620004bd0 2024-04-16T16:22:30.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.017+0000 7f3637e00640 1 -- 172.21.15.45:0/2874807073 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f363816d0c0 con 0x7f36200040e0 2024-04-16T16:22:30.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.017+0000 7f3637e00640 1 -- 172.21.15.45:0/2874807073 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f362000bad0 con 0x7f3638165b30 2024-04-16T16:22:30.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.017+0000 7f363e400640 1 --2- 172.21.15.45:0/2874807073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3620004bd0 0x7f3620005050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:30.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.017+0000 7f363da00640 1 --2- 172.21.15.45:0/2874807073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3638165b30 0x7f3620003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:30.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.017+0000 7f363d000640 1 --2- 172.21.15.45:0/2874807073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36200040e0 0x7f3620004560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:30.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.017+0000 7f363e400640 1 -- 172.21.15.45:0/2874807073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3638165b30 msgr2=0x7f3620003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:30.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.017+0000 7f363e400640 1 --2- 172.21.15.45:0/2874807073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3638165b30 0x7f3620003b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.018+0000 7f363e400640 1 -- 172.21.15.45:0/2874807073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36200040e0 msgr2=0x7f3620004560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:30.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.018+0000 7f363e400640 1 --2- 172.21.15.45:0/2874807073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36200040e0 0x7f3620004560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.018+0000 7f363e400640 1 -- 172.21.15.45:0/2874807073 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3620003500 con 0x7f3620004bd0 2024-04-16T16:22:30.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.018+0000 7f363d000640 1 --2- 172.21.15.45:0/2874807073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36200040e0 0x7f3620004560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:30.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.019+0000 7f363e400640 1 --2- 172.21.15.45:0/2874807073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3620004bd0 0x7f3620005050 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f363000a7a0 tx=0x7f3620003890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97532cf7ce104e74 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:30.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.019+0000 7f3637e00640 1 -- 172.21.15.45:0/2874807073 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363000a3f0 con 0x7f3620004bd0 2024-04-16T16:22:30.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.019+0000 7f3637e00640 1 -- 172.21.15.45:0/2874807073 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3630034650 con 0x7f3620004bd0 2024-04-16T16:22:30.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.019+0000 7f3637e00640 1 -- 172.21.15.45:0/2874807073 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3630034a40 con 0x7f3620004bd0 2024-04-16T16:22:30.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.020+0000 7f363ee00640 1 -- 172.21.15.45:0/2874807073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3620004bd0 msgr2=0x7f3620005050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:30.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.020+0000 7f363ee00640 1 --2- 172.21.15.45:0/2874807073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3620004bd0 0x7f3620005050 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f363000a7a0 tx=0x7f3620003890 comp rx=0 tx=0).stop 2024-04-16T16:22:30.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.020+0000 7f363e400640 1 -- 172.21.15.45:0/2874807073 reap_dead start 2024-04-16T16:22:30.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.020+0000 7f363ee00640 1 -- 172.21.15.45:0/2874807073 shutdown_connections 2024-04-16T16:22:30.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.020+0000 7f363ee00640 1 -- 172.21.15.45:0/2874807073 >> 172.21.15.45:0/2874807073 conn(0x7f36380677d0 msgr2=0x7f3638067bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:30.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.020+0000 7f363ee00640 1 -- 172.21.15.45:0/2874807073 shutdown_connections 2024-04-16T16:22:30.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.020+0000 7f363ee00640 1 -- 172.21.15.45:0/2874807073 wait complete. 2024-04-16T16:22:30.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.023+0000 7f363ee00640 1 Processor -- start 2024-04-16T16:22:30.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.023+0000 7f363ee00640 1 -- start start 2024-04-16T16:22:30.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.025+0000 7f363ee00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3620004bd0 0x7f363815a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:30.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.025+0000 7f363ee00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f363815c350 0x7f363815c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:30.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.025+0000 7f363ee00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f363815f0b0 0x7f363815f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:30.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.025+0000 7f363ee00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f363816bc60 con 0x7f363815c350 2024-04-16T16:22:30.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.025+0000 7f363ee00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f363816cd50 con 0x7f363815f0b0 2024-04-16T16:22:30.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.025+0000 7f363ee00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f363816bde0 con 0x7f3620004bd0 2024-04-16T16:22:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.025+0000 7f363d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f363815c350 0x7f363815c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.025+0000 7f363d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f363815c350 0x7f363815c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53302/0 (socket says 172.21.15.45:53302) 2024-04-16T16:22:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.025+0000 7f363d000640 1 -- 172.21.15.45:0/234276461 learned_addr learned my addr 172.21.15.45:0/234276461 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.025+0000 7f363e400640 1 --2- 172.21.15.45:0/234276461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f363815f0b0 0x7f363815f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.025+0000 7f363da00640 1 --2- 172.21.15.45:0/234276461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3620004bd0 0x7f363815a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.026+0000 7f363d000640 1 -- 172.21.15.45:0/234276461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3620004bd0 msgr2=0x7f363815a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.026+0000 7f363d000640 1 --2- 172.21.15.45:0/234276461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3620004bd0 0x7f363815a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.026+0000 7f363d000640 1 -- 172.21.15.45:0/234276461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f363815f0b0 msgr2=0x7f363815f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.026+0000 7f363d000640 1 --2- 172.21.15.45:0/234276461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f363815f0b0 0x7f363815f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.026+0000 7f363d000640 1 -- 172.21.15.45:0/234276461 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f363815b2b0 con 0x7f363815c350 2024-04-16T16:22:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.026+0000 7f363da00640 1 --2- 172.21.15.45:0/234276461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3620004bd0 0x7f363815a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.026+0000 7f363d000640 1 --2- 172.21.15.45:0/234276461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f363815c350 0x7f363815c7d0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f362400bde0 tx=0x7f3624013520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.026+0000 7f3635600640 1 -- 172.21.15.45:0/234276461 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362400de30 con 0x7f363815c350 2024-04-16T16:22:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.026+0000 7f3635600640 1 -- 172.21.15.45:0/234276461 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f362401f070 con 0x7f363815c350 2024-04-16T16:22:30.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.027+0000 7f363ee00640 1 -- 172.21.15.45:0/234276461 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f363815b590 con 0x7f363815c350 2024-04-16T16:22:30.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.027+0000 7f3635600640 1 -- 172.21.15.45:0/234276461 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362401a640 con 0x7f363815c350 2024-04-16T16:22:30.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.027+0000 7f363ee00640 1 -- 172.21.15.45:0/234276461 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f363815bb50 con 0x7f363815c350 2024-04-16T16:22:30.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.028+0000 7f363ee00640 1 -- 172.21.15.45:0/234276461 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3638075780 con 0x7f363815c350 2024-04-16T16:22:30.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.029+0000 7f3635600640 1 -- 172.21.15.45:0/234276461 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) ==== 91086+0+0 (secure 0 0 0) 0x7f362401a7e0 con 0x7f363815c350 2024-04-16T16:22:30.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.030+0000 7f3635600640 1 --2- 172.21.15.45:0/234276461 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f361806c9b0 0x7f361806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:30.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.030+0000 7f3635600640 1 -- 172.21.15.45:0/234276461 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6397+0+0 (secure 0 0 0) 0x7f3624090bc0 con 0x7f363815c350 2024-04-16T16:22:30.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.031+0000 7f363da00640 1 --2- 172.21.15.45:0/234276461 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f361806c9b0 0x7f361806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:30.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.032+0000 7f363da00640 1 --2- 172.21.15.45:0/234276461 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f361806c9b0 0x7f361806ee70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f362c004580 tx=0x7f362c00d040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:30.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.035+0000 7f3635600640 1 -- 172.21.15.45:0/234276461 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3624029020 con 0x7f363815c350 2024-04-16T16:22:30.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.239+0000 7f363ee00640 1 -- 172.21.15.45:0/234276461 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f36380730f0 con 0x7f363815c350 2024-04-16T16:22:30.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.242+0000 7f3635600640 1 -- 172.21.15.45:0/234276461 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) ==== 74+0+17696 (secure 0 0 0) 0x7f362405eac0 con 0x7f363815c350 2024-04-16T16:22:30.242 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:30.242 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":144,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:22:25.503282+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:21:23.973149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-16T16:21:25.043255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/741531900":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/3249203677":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/1497652063":"2024-04-17T16:22:25.503233+0000","172.21.15.103:6833/1506110537":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/3222629764":"2024-04-17T16:22:25.503233+0000","172.21.15.103:6832/1506110537":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/1298902321":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/884289453":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/3228977328":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/1437683872":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/2877575799":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/2644878831":"2024-04-17T16:21:47.933003+0000","172.21.15.103:6833/1478535274":"2024-04-17T16:21:47.933003+0000","172.21.15.103:6832/1478535274":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/1111841625":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/1114329424":"2024-04-17T16:21:47.933003+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:22:30.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.246+0000 7f363ee00640 1 -- 172.21.15.45:0/234276461 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f361806c9b0 msgr2=0x7f361806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.246+0000 7f363ee00640 1 --2- 172.21.15.45:0/234276461 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f361806c9b0 0x7f361806ee70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f362c004580 tx=0x7f362c00d040 comp rx=0 tx=0).stop 2024-04-16T16:22:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.246+0000 7f363ee00640 1 -- 172.21.15.45:0/234276461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f363815c350 msgr2=0x7f363815c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.246+0000 7f363ee00640 1 --2- 172.21.15.45:0/234276461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f363815c350 0x7f363815c7d0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f362400bde0 tx=0x7f3624013520 comp rx=0 tx=0).stop 2024-04-16T16:22:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.246+0000 7f363ee00640 1 -- 172.21.15.45:0/234276461 shutdown_connections 2024-04-16T16:22:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.246+0000 7f363ee00640 1 --2- 172.21.15.45:0/234276461 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f361806c9b0 0x7f361806ee70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.246+0000 7f363ee00640 1 --2- 172.21.15.45:0/234276461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f363815f0b0 0x7f363815f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.246+0000 7f363ee00640 1 --2- 172.21.15.45:0/234276461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f363815c350 0x7f363815c7d0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.246+0000 7f363ee00640 1 --2- 172.21.15.45:0/234276461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3620004bd0 0x7f363815a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.246+0000 7f363ee00640 1 -- 172.21.15.45:0/234276461 >> 172.21.15.45:0/234276461 conn(0x7f36380677d0 msgr2=0x7f36380b42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.247+0000 7f363ee00640 1 -- 172.21.15.45:0/234276461 shutdown_connections 2024-04-16T16:22:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.247+0000 7f363ee00640 1 -- 172.21.15.45:0/234276461 wait complete. 2024-04-16T16:22:30.267 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-16T16:22:30.267 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:22:30.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.480+0000 7fbd2cc00640 1 Processor -- start 2024-04-16T16:22:30.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.480+0000 7fbd2cc00640 1 -- start start 2024-04-16T16:22:30.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.482+0000 7fbd2cc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbd28161e90 0x7fbd280cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:30.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.482+0000 7fbd2cc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbd28162270 0x7fbd280d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:30.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.482+0000 7fbd2cc00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd280d3080 con 0x7fbd280d52d0 2024-04-16T16:22:30.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.482+0000 7fbd2cc00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fbd280d2e30 con 0x7fbd28161e90 2024-04-16T16:22:30.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.482+0000 7fbd2cc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbd280d3360 con 0x7fbd28162270 2024-04-16T16:22:30.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.482+0000 7fbd27400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbd28161e90 0x7fbd280cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:30.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.482+0000 7fbd27400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbd28161e90 0x7fbd280cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53306/0 (socket says 172.21.15.45:53306) 2024-04-16T16:22:30.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.482+0000 7fbd27400640 1 -- 172.21.15.45:0/1244444117 learned_addr learned my addr 172.21.15.45:0/1244444117 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:30.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.483+0000 7fbd26a00640 1 --2- 172.21.15.45:0/1244444117 >> v2:172.21.15.103:3300/0 conn(0x7fbd28162270 0x7fbd280d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:30.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.483+0000 7fbd27400640 1 -- 172.21.15.45:0/1244444117 >> v2:172.21.15.103:3300/0 conn(0x7fbd28162270 msgr2=0x7fbd280d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:30.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.483+0000 7fbd27400640 1 --2- 172.21.15.45:0/1244444117 >> v2:172.21.15.103:3300/0 conn(0x7fbd28162270 0x7fbd280d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.483+0000 7fbd27400640 1 -- 172.21.15.45:0/1244444117 >> v1:172.21.15.45:6789/0 conn(0x7fbd280d52d0 legacy=0x7fbd2806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:30.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.483+0000 7fbd27400640 1 -- 172.21.15.45:0/1244444117 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd280d3640 con 0x7fbd28161e90 2024-04-16T16:22:30.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.484+0000 7fbd27400640 1 --2- 172.21.15.45:0/1244444117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd28161e90 0x7fbd280cf7d0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fbd18009870 tx=0x7fbd1802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce6ae4245db7a8e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:30.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.484+0000 7fbd26000640 1 -- 172.21.15.45:0/1244444117 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd18034070 con 0x7fbd28161e90 2024-04-16T16:22:30.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.484+0000 7fbd26000640 1 -- 172.21.15.45:0/1244444117 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd18030860 con 0x7fbd28161e90 2024-04-16T16:22:30.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.484+0000 7fbd26000640 1 -- 172.21.15.45:0/1244444117 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd18030c30 con 0x7fbd28161e90 2024-04-16T16:22:30.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.485+0000 7fbd2cc00640 1 -- 172.21.15.45:0/1244444117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd28161e90 msgr2=0x7fbd280cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:30.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.485+0000 7fbd2cc00640 1 --2- 172.21.15.45:0/1244444117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd28161e90 0x7fbd280cf7d0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fbd18009870 tx=0x7fbd1802b9a0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.485+0000 7fbd2cc00640 1 -- 172.21.15.45:0/1244444117 shutdown_connections 2024-04-16T16:22:30.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.485+0000 7fbd2cc00640 1 --2- 172.21.15.45:0/1244444117 >> v2:172.21.15.103:3300/0 conn(0x7fbd28162270 0x7fbd280d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.485+0000 7fbd2cc00640 1 --2- 172.21.15.45:0/1244444117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd28161e90 0x7fbd280cf7d0 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.485+0000 7fbd2cc00640 1 -- 172.21.15.45:0/1244444117 >> 172.21.15.45:0/1244444117 conn(0x7fbd280d1940 msgr2=0x7fbd280d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:30.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.485+0000 7fbd2cc00640 1 -- 172.21.15.45:0/1244444117 shutdown_connections 2024-04-16T16:22:30.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.485+0000 7fbd2cc00640 1 -- 172.21.15.45:0/1244444117 wait complete. 2024-04-16T16:22:30.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.487+0000 7fbd2cc00640 1 Processor -- start 2024-04-16T16:22:30.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.487+0000 7fbd2cc00640 1 -- start start 2024-04-16T16:22:30.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.488+0000 7fbd2cc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd280d52d0 0x7fbd2814c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:30.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.488+0000 7fbd2cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd28161e90 0x7fbd280cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:30.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.488+0000 7fbd2cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd28162270 0x7fbd281e73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:30.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.488+0000 7fbd2cc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2806d550 con 0x7fbd28162270 2024-04-16T16:22:30.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.488+0000 7fbd2cc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbd280cfd10 con 0x7fbd28161e90 2024-04-16T16:22:30.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.488+0000 7fbd2cc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbd2806d6d0 con 0x7fbd280d52d0 2024-04-16T16:22:30.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.489+0000 7fbd27e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd28162270 0x7fbd281e73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:30.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.489+0000 7fbd27e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd28162270 0x7fbd281e73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53320/0 (socket says 172.21.15.45:53320) 2024-04-16T16:22:30.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.489+0000 7fbd27e00640 1 -- 172.21.15.45:0/523817 learned_addr learned my addr 172.21.15.45:0/523817 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:30.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.489+0000 7fbd27400640 1 --2- 172.21.15.45:0/523817 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd280d52d0 0x7fbd2814c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:30.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.489+0000 7fbd26a00640 1 --2- 172.21.15.45:0/523817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd28161e90 0x7fbd280cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:30.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.489+0000 7fbd27e00640 1 -- 172.21.15.45:0/523817 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd280d52d0 msgr2=0x7fbd2814c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:30.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.489+0000 7fbd27e00640 1 --2- 172.21.15.45:0/523817 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd280d52d0 0x7fbd2814c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.490+0000 7fbd27e00640 1 -- 172.21.15.45:0/523817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd28161e90 msgr2=0x7fbd280cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:30.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.490+0000 7fbd27e00640 1 --2- 172.21.15.45:0/523817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd28161e90 0x7fbd280cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.490+0000 7fbd27e00640 1 -- 172.21.15.45:0/523817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd280ce600 con 0x7fbd28162270 2024-04-16T16:22:30.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.490+0000 7fbd27e00640 1 --2- 172.21.15.45:0/523817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd28162270 0x7fbd281e73a0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fbd2000c6b0 tx=0x7fbd2000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:30.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.490+0000 7fbd1f400640 1 -- 172.21.15.45:0/523817 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20003b70 con 0x7fbd28162270 2024-04-16T16:22:30.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.490+0000 7fbd1f400640 1 -- 172.21.15.45:0/523817 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd20003d10 con 0x7fbd28162270 2024-04-16T16:22:30.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.490+0000 7fbd1f400640 1 -- 172.21.15.45:0/523817 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20005310 con 0x7fbd28162270 2024-04-16T16:22:30.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.490+0000 7fbd2cc00640 1 -- 172.21.15.45:0/523817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd280cf4e0 con 0x7fbd28162270 2024-04-16T16:22:30.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.490+0000 7fbd2cc00640 1 -- 172.21.15.45:0/523817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd2814d3e0 con 0x7fbd28162270 2024-04-16T16:22:30.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.492+0000 7fbd2cc00640 1 -- 172.21.15.45:0/523817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd28075780 con 0x7fbd28162270 2024-04-16T16:22:30.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.493+0000 7fbd1f400640 1 -- 172.21.15.45:0/523817 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fbd200054b0 con 0x7fbd28162270 2024-04-16T16:22:30.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.494+0000 7fbd1f400640 1 --2- 172.21.15.45:0/523817 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbd0806ca10 0x7fbd0806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:30.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.494+0000 7fbd1f400640 1 -- 172.21.15.45:0/523817 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6397+0+0 (secure 0 0 0) 0x7fbd2008bef0 con 0x7fbd28162270 2024-04-16T16:22:30.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.494+0000 7fbd27400640 1 --2- 172.21.15.45:0/523817 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbd0806ca10 0x7fbd0806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:30.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.495+0000 7fbd27400640 1 --2- 172.21.15.45:0/523817 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbd0806ca10 0x7fbd0806eed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbd18036000 tx=0x7fbd1803b040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:30.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.498+0000 7fbd1f400640 1 -- 172.21.15.45:0/523817 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbd280cf4e0 con 0x7fbd28162270 2024-04-16T16:22:30.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.747+0000 7fbd2cc00640 1 -- 172.21.15.45:0/523817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd280d4bf0 con 0x7fbd28162270 2024-04-16T16:22:30.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.751+0000 7fbd1f400640 1 -- 172.21.15.45:0/523817 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5213 (secure 0 0 0) 0x7fbd20059e30 con 0x7fbd28162270 2024-04-16T16:22:30.751 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:30.751 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2585865379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2585865379},{"type":"v1","addr":"172.21.15.45:6834","nonce":2585865379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6590,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/777087937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":777087937},{"type":"v1","addr":"172.21.15.45:6835","nonce":777087937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:21:26.066855+0000","modified":"2024-04-16T16:22:25.276355+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6596,"mds_1":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":1,"incarnation":102,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6835/2561260581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2561260581},{"type":"v1","addr":"172.21.15.103:6835","nonce":2561260581}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6596":{"gid":6596,"name":"d","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6837/3252460318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3252460318},{"type":"v1","addr":"172.21.15.103:6837","nonce":3252460318}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6596,"qdb_cluster":[6593,6596]},"id":10}]} 2024-04-16T16:22:30.751 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 105 2024-04-16T16:22:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.754+0000 7fbd2cc00640 1 -- 172.21.15.45:0/523817 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbd0806ca10 msgr2=0x7fbd0806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.754+0000 7fbd2cc00640 1 --2- 172.21.15.45:0/523817 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbd0806ca10 0x7fbd0806eed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbd18036000 tx=0x7fbd1803b040 comp rx=0 tx=0).stop 2024-04-16T16:22:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.754+0000 7fbd2cc00640 1 -- 172.21.15.45:0/523817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd28162270 msgr2=0x7fbd281e73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.754+0000 7fbd2cc00640 1 --2- 172.21.15.45:0/523817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd28162270 0x7fbd281e73a0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fbd2000c6b0 tx=0x7fbd2000cb70 comp rx=0 tx=0).stop 2024-04-16T16:22:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.754+0000 7fbd2cc00640 1 -- 172.21.15.45:0/523817 shutdown_connections 2024-04-16T16:22:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.754+0000 7fbd2cc00640 1 --2- 172.21.15.45:0/523817 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbd0806ca10 0x7fbd0806eed0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.754+0000 7fbd2cc00640 1 --2- 172.21.15.45:0/523817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd28162270 0x7fbd281e73a0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.754+0000 7fbd2cc00640 1 --2- 172.21.15.45:0/523817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd28161e90 0x7fbd280cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.754+0000 7fbd2cc00640 1 --2- 172.21.15.45:0/523817 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd280d52d0 0x7fbd2814c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.754+0000 7fbd2cc00640 1 -- 172.21.15.45:0/523817 >> 172.21.15.45:0/523817 conn(0x7fbd280d1940 msgr2=0x7fbd280ca320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.755+0000 7fbd2cc00640 1 -- 172.21.15.45:0/523817 shutdown_connections 2024-04-16T16:22:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:30.755+0000 7fbd2cc00640 1 -- 172.21.15.45:0/523817 wait complete. 2024-04-16T16:22:30.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:22:31.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.003+0000 7fe461e00640 1 Processor -- start 2024-04-16T16:22:31.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.003+0000 7fe461e00640 1 -- start start 2024-04-16T16:22:31.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.005+0000 7fe461e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe45c163010 0x7fe45c0cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.005+0000 7fe461e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe45c163510 con 0x7fe45c0cde00 2024-04-16T16:22:31.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.005+0000 7fe461e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe45c1637f0 con 0x7fe45c06f540 2024-04-16T16:22:31.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.005+0000 7fe461e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe45c163ad0 con 0x7fe45c163010 2024-04-16T16:22:31.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.005+0000 7fe45be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe45c163010 0x7fe45c0cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:31.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.006+0000 7fe45be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe45c163010 0x7fe45c0cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53324/0 (socket says 172.21.15.45:53324) 2024-04-16T16:22:31.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.006+0000 7fe45be00640 1 -- 172.21.15.45:0/2618760695 learned_addr learned my addr 172.21.15.45:0/2618760695 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:31.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.006+0000 7fe45be00640 1 -- 172.21.15.45:0/2618760695 >> v1:172.21.15.103:6789/0 conn(0x7fe45c06f540 legacy=0x7fe45c1645b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:31.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.006+0000 7fe45be00640 1 -- 172.21.15.45:0/2618760695 >> v1:172.21.15.45:6789/0 conn(0x7fe45c0cde00 legacy=0x7fe45c0ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.006+0000 7fe45be00640 1 -- 172.21.15.45:0/2618760695 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe45c163db0 con 0x7fe45c163010 2024-04-16T16:22:31.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.007+0000 7fe45be00640 1 --2- 172.21.15.45:0/2618760695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c163010 0x7fe45c0cff30 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fe448009870 tx=0x7fe44802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec659e34558894 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:31.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.007+0000 7fe45b400640 1 -- 172.21.15.45:0/2618760695 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe448034070 con 0x7fe45c163010 2024-04-16T16:22:31.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.007+0000 7fe45b400640 1 -- 172.21.15.45:0/2618760695 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe448030860 con 0x7fe45c163010 2024-04-16T16:22:31.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.008+0000 7fe45b400640 1 -- 172.21.15.45:0/2618760695 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe448030c30 con 0x7fe45c163010 2024-04-16T16:22:31.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.008+0000 7fe461e00640 1 -- 172.21.15.45:0/2618760695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c163010 msgr2=0x7fe45c0cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.008+0000 7fe461e00640 1 --2- 172.21.15.45:0/2618760695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c163010 0x7fe45c0cff30 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fe448009870 tx=0x7fe44802b9a0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.008+0000 7fe461e00640 1 -- 172.21.15.45:0/2618760695 shutdown_connections 2024-04-16T16:22:31.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.008+0000 7fe461e00640 1 --2- 172.21.15.45:0/2618760695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c163010 0x7fe45c0cff30 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.008+0000 7fe461e00640 1 -- 172.21.15.45:0/2618760695 >> 172.21.15.45:0/2618760695 conn(0x7fe45c06b930 msgr2=0x7fe45c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:31.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.009+0000 7fe461e00640 1 -- 172.21.15.45:0/2618760695 shutdown_connections 2024-04-16T16:22:31.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.009+0000 7fe461e00640 1 -- 172.21.15.45:0/2618760695 wait complete. 2024-04-16T16:22:31.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.010+0000 7fe461e00640 1 Processor -- start 2024-04-16T16:22:31.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.010+0000 7fe461e00640 1 -- start start 2024-04-16T16:22:31.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.011+0000 7fe461e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe45c06f540 0x7fe45c15cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.011+0000 7fe461e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c0cde00 0x7fe45c160a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.011+0000 7fe461e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe45c163010 0x7fe45c15c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.011+0000 7fe461e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c164cc0 con 0x7fe45c0cde00 2024-04-16T16:22:31.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.012+0000 7fe461e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c06f970 con 0x7fe45c06f540 2024-04-16T16:22:31.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.012+0000 7fe461e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe45c164e40 con 0x7fe45c163010 2024-04-16T16:22:31.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.012+0000 7fe45be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c0cde00 0x7fe45c160a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:31.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.012+0000 7fe45be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c0cde00 0x7fe45c160a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53340/0 (socket says 172.21.15.45:53340) 2024-04-16T16:22:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.012+0000 7fe45be00640 1 -- 172.21.15.45:0/876016328 learned_addr learned my addr 172.21.15.45:0/876016328 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.012+0000 7fe460a00640 1 --2- 172.21.15.45:0/876016328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe45c06f540 0x7fe45c15cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.012+0000 7fe45be00640 1 -- 172.21.15.45:0/876016328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe45c163010 msgr2=0x7fe45c15c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.012+0000 7fe45be00640 1 --2- 172.21.15.45:0/876016328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe45c163010 0x7fe45c15c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.012+0000 7fe45be00640 1 -- 172.21.15.45:0/876016328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe45c06f540 msgr2=0x7fe45c15cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.012+0000 7fe45be00640 1 --2- 172.21.15.45:0/876016328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe45c06f540 0x7fe45c15cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.012+0000 7fe45be00640 1 -- 172.21.15.45:0/876016328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe45c15c5b0 con 0x7fe45c0cde00 2024-04-16T16:22:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.012+0000 7fe460a00640 1 --2- 172.21.15.45:0/876016328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe45c06f540 0x7fe45c15cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.013+0000 7fe45be00640 1 --2- 172.21.15.45:0/876016328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c0cde00 0x7fe45c160a50 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fe448009870 tx=0x7fe4480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.013+0000 7fe458c00640 1 -- 172.21.15.45:0/876016328 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe448034040 con 0x7fe45c0cde00 2024-04-16T16:22:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.013+0000 7fe458c00640 1 -- 172.21.15.45:0/876016328 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4480433f0 con 0x7fe45c0cde00 2024-04-16T16:22:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.013+0000 7fe458c00640 1 -- 172.21.15.45:0/876016328 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4480437e0 con 0x7fe45c0cde00 2024-04-16T16:22:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.013+0000 7fe461e00640 1 -- 172.21.15.45:0/876016328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe45c15d450 con 0x7fe45c0cde00 2024-04-16T16:22:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.013+0000 7fe461e00640 1 -- 172.21.15.45:0/876016328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe45c15f280 con 0x7fe45c0cde00 2024-04-16T16:22:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.014+0000 7fe461e00640 1 -- 172.21.15.45:0/876016328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe45c075780 con 0x7fe45c0cde00 2024-04-16T16:22:31.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.016+0000 7fe458c00640 1 -- 172.21.15.45:0/876016328 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fe44804a060 con 0x7fe45c0cde00 2024-04-16T16:22:31.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.016+0000 7fe458c00640 1 --2- 172.21.15.45:0/876016328 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fe43c06ca10 0x7fe43c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.017+0000 7fe458c00640 1 -- 172.21.15.45:0/876016328 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6397+0+0 (secure 0 0 0) 0x7fe4480ae3a0 con 0x7fe45c0cde00 2024-04-16T16:22:31.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.017+0000 7fe460a00640 1 --2- 172.21.15.45:0/876016328 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fe43c06ca10 0x7fe43c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:31.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.018+0000 7fe460a00640 1 --2- 172.21.15.45:0/876016328 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fe43c06ca10 0x7fe43c06eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe450001680 tx=0x7fe450009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:31.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.021+0000 7fe458c00640 1 -- 172.21.15.45:0/876016328 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe44803e070 con 0x7fe45c0cde00 2024-04-16T16:22:31.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.270+0000 7fe461e00640 1 -- 172.21.15.45:0/876016328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe45c06fc20 con 0x7fe45c0cde00 2024-04-16T16:22:31.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.271+0000 7fe458c00640 1 -- 172.21.15.45:0/876016328 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5213 (secure 0 0 0) 0x7fe44804be90 con 0x7fe45c0cde00 2024-04-16T16:22:31.271 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:31.271 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2585865379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2585865379},{"type":"v1","addr":"172.21.15.45:6834","nonce":2585865379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6590,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/777087937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":777087937},{"type":"v1","addr":"172.21.15.45:6835","nonce":777087937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:21:26.066855+0000","modified":"2024-04-16T16:22:25.276355+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6596,"mds_1":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":1,"incarnation":102,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6835/2561260581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2561260581},{"type":"v1","addr":"172.21.15.103:6835","nonce":2561260581}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6596":{"gid":6596,"name":"d","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6837/3252460318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3252460318},{"type":"v1","addr":"172.21.15.103:6837","nonce":3252460318}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6596,"qdb_cluster":[6593,6596]},"id":10}]} 2024-04-16T16:22:31.271 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 105 2024-04-16T16:22:31.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.274+0000 7fe461e00640 1 -- 172.21.15.45:0/876016328 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fe43c06ca10 msgr2=0x7fe43c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.274+0000 7fe461e00640 1 --2- 172.21.15.45:0/876016328 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fe43c06ca10 0x7fe43c06eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe450001680 tx=0x7fe450009040 comp rx=0 tx=0).stop 2024-04-16T16:22:31.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.274+0000 7fe461e00640 1 -- 172.21.15.45:0/876016328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c0cde00 msgr2=0x7fe45c160a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.274+0000 7fe461e00640 1 --2- 172.21.15.45:0/876016328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c0cde00 0x7fe45c160a50 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fe448009870 tx=0x7fe4480046c0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.274+0000 7fe461e00640 1 -- 172.21.15.45:0/876016328 shutdown_connections 2024-04-16T16:22:31.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.274+0000 7fe461e00640 1 --2- 172.21.15.45:0/876016328 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fe43c06ca10 0x7fe43c06eed0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.274+0000 7fe461e00640 1 --2- 172.21.15.45:0/876016328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe45c163010 0x7fe45c15c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.274+0000 7fe461e00640 1 --2- 172.21.15.45:0/876016328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c0cde00 0x7fe45c160a50 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.274+0000 7fe461e00640 1 --2- 172.21.15.45:0/876016328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe45c06f540 0x7fe45c15cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.274+0000 7fe461e00640 1 -- 172.21.15.45:0/876016328 >> 172.21.15.45:0/876016328 conn(0x7fe45c06b930 msgr2=0x7fe45c161af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:31.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.274+0000 7fe461e00640 1 -- 172.21.15.45:0/876016328 shutdown_connections 2024-04-16T16:22:31.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.274+0000 7fe461e00640 1 -- 172.21.15.45:0/876016328 wait complete. 2024-04-16T16:22:31.293 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:22:31.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.496+0000 7f0a3d800640 1 Processor -- start 2024-04-16T16:22:31.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.496+0000 7f0a3d800640 1 -- start start 2024-04-16T16:22:31.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.498+0000 7f0a3d800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0a381462e0 0x7f0a38072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.498+0000 7f0a3d800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0a380728c0 0x7f0a380ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.498+0000 7f0a3d800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a380cf6d0 con 0x7f0a380cec00 2024-04-16T16:22:31.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.498+0000 7f0a3d800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0a381466c0 con 0x7f0a381462e0 2024-04-16T16:22:31.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.498+0000 7f0a3d800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0a380cf3f0 con 0x7f0a380728c0 2024-04-16T16:22:31.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.498+0000 7f0a36a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0a381462e0 0x7f0a38072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:31.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.498+0000 7f0a36a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0a381462e0 0x7f0a38072380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53346/0 (socket says 172.21.15.45:53346) 2024-04-16T16:22:31.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.498+0000 7f0a36a00640 1 -- 172.21.15.45:0/2017324990 learned_addr learned my addr 172.21.15.45:0/2017324990 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:31.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.498+0000 7f0a2e000640 1 --2- 172.21.15.45:0/2017324990 >> v2:172.21.15.103:3300/0 conn(0x7f0a380728c0 0x7f0a380ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:31.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.498+0000 7f0a36a00640 1 -- 172.21.15.45:0/2017324990 >> v2:172.21.15.103:3300/0 conn(0x7f0a380728c0 msgr2=0x7f0a380ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:31.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.498+0000 7f0a36a00640 1 --2- 172.21.15.45:0/2017324990 >> v2:172.21.15.103:3300/0 conn(0x7f0a380728c0 0x7f0a380ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.498+0000 7f0a36a00640 1 -- 172.21.15.45:0/2017324990 >> v1:172.21.15.45:6789/0 conn(0x7f0a380cec00 legacy=0x7f0a3815da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.498+0000 7f0a36a00640 1 -- 172.21.15.45:0/2017324990 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3815cf80 con 0x7f0a381462e0 2024-04-16T16:22:31.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.499+0000 7f0a36a00640 1 --2- 172.21.15.45:0/2017324990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a381462e0 0x7f0a38072380 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f0a3806fc10 tx=0x7f0a2802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca351e0f21bfbd3d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:31.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.499+0000 7f0a36000640 1 -- 172.21.15.45:0/2017324990 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a28035020 con 0x7f0a381462e0 2024-04-16T16:22:31.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.499+0000 7f0a36000640 1 -- 172.21.15.45:0/2017324990 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a28031870 con 0x7f0a381462e0 2024-04-16T16:22:31.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.499+0000 7f0a36000640 1 -- 172.21.15.45:0/2017324990 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a28031c40 con 0x7f0a381462e0 2024-04-16T16:22:31.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.500+0000 7f0a3d800640 1 -- 172.21.15.45:0/2017324990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a381462e0 msgr2=0x7f0a38072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.500+0000 7f0a3d800640 1 --2- 172.21.15.45:0/2017324990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a381462e0 0x7f0a38072380 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f0a3806fc10 tx=0x7f0a2802b910 comp rx=0 tx=0).stop 2024-04-16T16:22:31.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.500+0000 7f0a3d800640 1 -- 172.21.15.45:0/2017324990 shutdown_connections 2024-04-16T16:22:31.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.500+0000 7f0a3d800640 1 --2- 172.21.15.45:0/2017324990 >> v2:172.21.15.103:3300/0 conn(0x7f0a380728c0 0x7f0a380ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.500+0000 7f0a3d800640 1 --2- 172.21.15.45:0/2017324990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a381462e0 0x7f0a38072380 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.500+0000 7f0a3d800640 1 -- 172.21.15.45:0/2017324990 >> 172.21.15.45:0/2017324990 conn(0x7f0a380b2750 msgr2=0x7f0a380b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:31.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.500+0000 7f0a3d800640 1 -- 172.21.15.45:0/2017324990 shutdown_connections 2024-04-16T16:22:31.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.500+0000 7f0a3d800640 1 -- 172.21.15.45:0/2017324990 wait complete. 2024-04-16T16:22:31.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.502+0000 7f0a3d800640 1 Processor -- start 2024-04-16T16:22:31.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.502+0000 7f0a3d800640 1 -- start start 2024-04-16T16:22:31.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.503+0000 7f0a3d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a380728c0 0x7f0a381f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.503+0000 7f0a3d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a380cec00 0x7f0a3813ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.503+0000 7f0a3d800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a381462e0 0x7f0a38139df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.503+0000 7f0a3d800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0a380cf020 con 0x7f0a380cec00 2024-04-16T16:22:31.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.503+0000 7f0a3d800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3815d540 con 0x7f0a381462e0 2024-04-16T16:22:31.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.503+0000 7f0a3d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0a380cf1a0 con 0x7f0a380728c0 2024-04-16T16:22:31.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.503+0000 7f0a2e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a380cec00 0x7f0a3813ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:31.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.503+0000 7f0a2e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a380cec00 0x7f0a3813ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53360/0 (socket says 172.21.15.45:53360) 2024-04-16T16:22:31.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.503+0000 7f0a2e000640 1 -- 172.21.15.45:0/3764080963 learned_addr learned my addr 172.21.15.45:0/3764080963 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:31.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.504+0000 7f0a36a00640 1 --2- 172.21.15.45:0/3764080963 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a380728c0 0x7f0a381f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:31.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.504+0000 7f0a37400640 1 --2- 172.21.15.45:0/3764080963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a381462e0 0x7f0a38139df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:31.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.504+0000 7f0a2e000640 1 -- 172.21.15.45:0/3764080963 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a380728c0 msgr2=0x7f0a381f5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.504+0000 7f0a2e000640 1 --2- 172.21.15.45:0/3764080963 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a380728c0 0x7f0a381f5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.504+0000 7f0a2e000640 1 -- 172.21.15.45:0/3764080963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a381462e0 msgr2=0x7f0a38139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.504+0000 7f0a2e000640 1 --2- 172.21.15.45:0/3764080963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a381462e0 0x7f0a38139df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.504+0000 7f0a2e000640 1 -- 172.21.15.45:0/3764080963 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3814db50 con 0x7f0a380cec00 2024-04-16T16:22:31.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.504+0000 7f0a37400640 1 --2- 172.21.15.45:0/3764080963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a381462e0 0x7f0a38139df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:31.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.504+0000 7f0a2e000640 1 --2- 172.21.15.45:0/3764080963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a380cec00 0x7f0a3813ac70 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000d800 tx=0x7f0a2000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:31.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.504+0000 7f0a2f400640 1 -- 172.21.15.45:0/3764080963 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2000b6c0 con 0x7f0a380cec00 2024-04-16T16:22:31.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.504+0000 7f0a2f400640 1 -- 172.21.15.45:0/3764080963 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a20019070 con 0x7f0a380cec00 2024-04-16T16:22:31.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.504+0000 7f0a2f400640 1 -- 172.21.15.45:0/3764080963 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2000ba80 con 0x7f0a380cec00 2024-04-16T16:22:31.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.505+0000 7f0a3d800640 1 -- 172.21.15.45:0/3764080963 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a3813d5f0 con 0x7f0a380cec00 2024-04-16T16:22:31.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.505+0000 7f0a3d800640 1 -- 172.21.15.45:0/3764080963 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a3813db70 con 0x7f0a380cec00 2024-04-16T16:22:31.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.506+0000 7f0a3d800640 1 -- 172.21.15.45:0/3764080963 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a3813dd40 con 0x7f0a380cec00 2024-04-16T16:22:31.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.507+0000 7f0a2f400640 1 -- 172.21.15.45:0/3764080963 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f0a2000bc20 con 0x7f0a380cec00 2024-04-16T16:22:31.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.508+0000 7f0a2f400640 1 --2- 172.21.15.45:0/3764080963 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0a1406ca10 0x7f0a1406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.508+0000 7f0a2f400640 1 -- 172.21.15.45:0/3764080963 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6397+0+0 (secure 0 0 0) 0x7f0a2008c520 con 0x7f0a380cec00 2024-04-16T16:22:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.508+0000 7f0a36a00640 1 --2- 172.21.15.45:0/3764080963 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0a1406ca10 0x7f0a1406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.509+0000 7f0a36a00640 1 --2- 172.21.15.45:0/3764080963 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0a1406ca10 0x7f0a1406eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0a28036000 tx=0x7f0a2803c040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:31.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.513+0000 7f0a2f400640 1 -- 172.21.15.45:0/3764080963 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a2005a4f0 con 0x7f0a380cec00 2024-04-16T16:22:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.718+0000 7f0a3d800640 1 -- 172.21.15.45:0/3764080963 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a38075780 con 0x7f0a380cec00 2024-04-16T16:22:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.720+0000 7f0a2f400640 1 -- 172.21.15.45:0/3764080963 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) ==== 74+0+17696 (secure 0 0 0) 0x7f0a20059df0 con 0x7f0a380cec00 2024-04-16T16:22:31.720 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:31.720 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":144,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:22:25.503282+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:21:23.973149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-16T16:21:25.043255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/741531900":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/3249203677":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/1497652063":"2024-04-17T16:22:25.503233+0000","172.21.15.103:6833/1506110537":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/3222629764":"2024-04-17T16:22:25.503233+0000","172.21.15.103:6832/1506110537":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/1298902321":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/884289453":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/3228977328":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/1437683872":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/2877575799":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/2644878831":"2024-04-17T16:21:47.933003+0000","172.21.15.103:6833/1478535274":"2024-04-17T16:21:47.933003+0000","172.21.15.103:6832/1478535274":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/1111841625":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/1114329424":"2024-04-17T16:21:47.933003+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:22:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.722+0000 7f0a3d800640 1 -- 172.21.15.45:0/3764080963 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0a1406ca10 msgr2=0x7f0a1406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.722+0000 7f0a3d800640 1 --2- 172.21.15.45:0/3764080963 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0a1406ca10 0x7f0a1406eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0a28036000 tx=0x7f0a2803c040 comp rx=0 tx=0).stop 2024-04-16T16:22:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.722+0000 7f0a3d800640 1 -- 172.21.15.45:0/3764080963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a380cec00 msgr2=0x7f0a3813ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.722+0000 7f0a3d800640 1 --2- 172.21.15.45:0/3764080963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a380cec00 0x7f0a3813ac70 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000d800 tx=0x7f0a2000dcc0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.723+0000 7f0a3d800640 1 -- 172.21.15.45:0/3764080963 shutdown_connections 2024-04-16T16:22:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.723+0000 7f0a3d800640 1 --2- 172.21.15.45:0/3764080963 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0a1406ca10 0x7f0a1406eed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.723+0000 7f0a3d800640 1 --2- 172.21.15.45:0/3764080963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a381462e0 0x7f0a38139df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.723+0000 7f0a3d800640 1 --2- 172.21.15.45:0/3764080963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a380cec00 0x7f0a3813ac70 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.723+0000 7f0a3d800640 1 --2- 172.21.15.45:0/3764080963 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a380728c0 0x7f0a381f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.723+0000 7f0a3d800640 1 -- 172.21.15.45:0/3764080963 >> 172.21.15.45:0/3764080963 conn(0x7f0a380b2750 msgr2=0x7f0a380b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.723+0000 7f0a3d800640 1 -- 172.21.15.45:0/3764080963 shutdown_connections 2024-04-16T16:22:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.723+0000 7f0a3d800640 1 -- 172.21.15.45:0/3764080963 wait complete. 2024-04-16T16:22:31.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-16T16:22:31.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.966+0000 7fb35d800640 1 Processor -- start 2024-04-16T16:22:31.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.966+0000 7fb35d800640 1 -- start start 2024-04-16T16:22:31.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.968+0000 7fb35d800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb358146220 0x7fb3580cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.968+0000 7fb35d800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb358072180 0x7fb3580725f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.968+0000 7fb35d800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb358146600 con 0x7fb3580cd220 2024-04-16T16:22:31.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.968+0000 7fb35d800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb358072b30 con 0x7fb358072180 2024-04-16T16:22:31.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.968+0000 7fb35d800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb35806f840 con 0x7fb358146220 2024-04-16T16:22:31.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.968+0000 7fb34e000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb358072180 0x7fb3580725f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:31.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.968+0000 7fb34e000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb358072180 0x7fb3580725f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53362/0 (socket says 172.21.15.45:53362) 2024-04-16T16:22:31.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.968+0000 7fb34e000640 1 -- 172.21.15.45:0/83716856 learned_addr learned my addr 172.21.15.45:0/83716856 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:31.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.968+0000 7fb34e000640 1 -- 172.21.15.45:0/83716856 >> v2:172.21.15.103:3300/0 conn(0x7fb358146220 msgr2=0x7fb3580cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.968+0000 7fb356a00640 1 --2- 172.21.15.45:0/83716856 >> v2:172.21.15.103:3300/0 conn(0x7fb358146220 0x7fb3580cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.968+0000 7fb34e000640 1 --2- 172.21.15.45:0/83716856 >> v2:172.21.15.103:3300/0 conn(0x7fb358146220 0x7fb3580cf110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.968+0000 7fb34e000640 1 -- 172.21.15.45:0/83716856 >> v1:172.21.15.45:6789/0 conn(0x7fb3580cd220 legacy=0x7fb3580cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.968+0000 7fb34e000640 1 -- 172.21.15.45:0/83716856 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3580cde00 con 0x7fb358072180 2024-04-16T16:22:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.969+0000 7fb34e000640 1 --2- 172.21.15.45:0/83716856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb358072180 0x7fb3580725f0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fb358073510 tx=0x7fb34002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d84f9e394514e6f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:31.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.970+0000 7fb356000640 1 -- 172.21.15.45:0/83716856 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb340034070 con 0x7fb358072180 2024-04-16T16:22:31.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.970+0000 7fb356000640 1 -- 172.21.15.45:0/83716856 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb340030860 con 0x7fb358072180 2024-04-16T16:22:31.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.970+0000 7fb356000640 1 -- 172.21.15.45:0/83716856 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb340030c30 con 0x7fb358072180 2024-04-16T16:22:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.970+0000 7fb35d800640 1 -- 172.21.15.45:0/83716856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb358072180 msgr2=0x7fb3580725f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.970+0000 7fb35d800640 1 --2- 172.21.15.45:0/83716856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb358072180 0x7fb3580725f0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fb358073510 tx=0x7fb34002b9a0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.970+0000 7fb35d800640 1 -- 172.21.15.45:0/83716856 shutdown_connections 2024-04-16T16:22:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.970+0000 7fb35d800640 1 --2- 172.21.15.45:0/83716856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb358072180 0x7fb3580725f0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.970+0000 7fb35d800640 1 --2- 172.21.15.45:0/83716856 >> v2:172.21.15.103:3300/0 conn(0x7fb358146220 0x7fb3580cf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.970+0000 7fb35d800640 1 -- 172.21.15.45:0/83716856 >> 172.21.15.45:0/83716856 conn(0x7fb35806b930 msgr2=0x7fb35806bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.970+0000 7fb35d800640 1 -- 172.21.15.45:0/83716856 shutdown_connections 2024-04-16T16:22:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.971+0000 7fb35d800640 1 -- 172.21.15.45:0/83716856 wait complete. 2024-04-16T16:22:31.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.972+0000 7fb35d800640 1 Processor -- start 2024-04-16T16:22:31.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.972+0000 7fb35d800640 1 -- start start 2024-04-16T16:22:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.973+0000 7fb35d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb358072180 0x7fb3580b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.973+0000 7fb35d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3580cd220 0x7fb358154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.973+0000 7fb35d800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb358146220 0x7fb35813ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.973+0000 7fb35d800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb3580ce910 con 0x7fb358072180 2024-04-16T16:22:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.973+0000 7fb35d800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb3580ce0e0 con 0x7fb358146220 2024-04-16T16:22:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.973+0000 7fb35d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb3580cebf0 con 0x7fb3580cd220 2024-04-16T16:22:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.974+0000 7fb356a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb358072180 0x7fb3580b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.974+0000 7fb356a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb358072180 0x7fb3580b1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53368/0 (socket says 172.21.15.45:53368) 2024-04-16T16:22:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.974+0000 7fb356a00640 1 -- 172.21.15.45:0/2995700991 learned_addr learned my addr 172.21.15.45:0/2995700991 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.974+0000 7fb356a00640 1 -- 172.21.15.45:0/2995700991 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3580cd220 msgr2=0x7fb358154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.974+0000 7fb356a00640 1 --2- 172.21.15.45:0/2995700991 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3580cd220 0x7fb358154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.974+0000 7fb356a00640 1 -- 172.21.15.45:0/2995700991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb358146220 msgr2=0x7fb35813ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.974+0000 7fb356a00640 1 --2- 172.21.15.45:0/2995700991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb358146220 0x7fb35813ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.974+0000 7fb356a00640 1 -- 172.21.15.45:0/2995700991 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb358155590 con 0x7fb358072180 2024-04-16T16:22:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.974+0000 7fb356a00640 1 --2- 172.21.15.45:0/2995700991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb358072180 0x7fb3580b1b20 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7fb34800c7f0 tx=0x7fb34800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.975+0000 7fb34f400640 1 -- 172.21.15.45:0/2995700991 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348015070 con 0x7fb358072180 2024-04-16T16:22:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.975+0000 7fb34f400640 1 -- 172.21.15.45:0/2995700991 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb348004240 con 0x7fb358072180 2024-04-16T16:22:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.975+0000 7fb34f400640 1 -- 172.21.15.45:0/2995700991 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348007620 con 0x7fb358072180 2024-04-16T16:22:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.975+0000 7fb35d800640 1 -- 172.21.15.45:0/2995700991 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb358155870 con 0x7fb358072180 2024-04-16T16:22:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.975+0000 7fb35d800640 1 -- 172.21.15.45:0/2995700991 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb358155e30 con 0x7fb358072180 2024-04-16T16:22:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.976+0000 7fb35d800640 1 -- 172.21.15.45:0/2995700991 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb35815dd10 con 0x7fb358072180 2024-04-16T16:22:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.977+0000 7fb34f400640 1 -- 172.21.15.45:0/2995700991 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fb3480079d0 con 0x7fb358072180 2024-04-16T16:22:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.979+0000 7fb34f400640 1 --2- 172.21.15.45:0/2995700991 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb33406ca10 0x7fb33406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.979+0000 7fb34f400640 1 -- 172.21.15.45:0/2995700991 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6397+0+0 (secure 0 0 0) 0x7fb34808c120 con 0x7fb358072180 2024-04-16T16:22:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.979+0000 7fb34e000640 1 --2- 172.21.15.45:0/2995700991 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb33406ca10 0x7fb33406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:31.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.980+0000 7fb34e000640 1 --2- 172.21.15.45:0/2995700991 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb33406ca10 0x7fb33406eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb340036000 tx=0x7fb340030000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:31.983+0000 7fb34f400640 1 -- 172.21.15.45:0/2995700991 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb35815dd10 con 0x7fb358072180 2024-04-16T16:22:32.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.228+0000 7fb35d800640 1 -- 172.21.15.45:0/2995700991 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb358075780 con 0x7fb358072180 2024-04-16T16:22:32.230 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-04-16T16:22:32.231+0000 7fe7a4800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T16:22:32.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.234+0000 7fb34f400640 1 -- 172.21.15.45:0/2995700991 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v106) ==== 161+0+0 (secure 0 0 0) 0x7fb34805a110 con 0x7fb358072180 2024-04-16T16:22:32.233 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-16T16:22:32.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.237+0000 7fb35d800640 1 -- 172.21.15.45:0/2995700991 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb33406ca10 msgr2=0x7fb33406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:32.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.237+0000 7fb35d800640 1 --2- 172.21.15.45:0/2995700991 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb33406ca10 0x7fb33406eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb340036000 tx=0x7fb340030000 comp rx=0 tx=0).stop 2024-04-16T16:22:32.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.237+0000 7fb35d800640 1 -- 172.21.15.45:0/2995700991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb358072180 msgr2=0x7fb3580b1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:32.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.237+0000 7fb35d800640 1 --2- 172.21.15.45:0/2995700991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb358072180 0x7fb3580b1b20 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7fb34800c7f0 tx=0x7fb34800ccb0 comp rx=0 tx=0).stop 2024-04-16T16:22:32.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.237+0000 7fb35d800640 1 -- 172.21.15.45:0/2995700991 shutdown_connections 2024-04-16T16:22:32.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.237+0000 7fb35d800640 1 --2- 172.21.15.45:0/2995700991 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb33406ca10 0x7fb33406eed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:32.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.237+0000 7fb35d800640 1 --2- 172.21.15.45:0/2995700991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb358146220 0x7fb35813ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:32.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.237+0000 7fb35d800640 1 --2- 172.21.15.45:0/2995700991 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3580cd220 0x7fb358154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:32.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.237+0000 7fb35d800640 1 --2- 172.21.15.45:0/2995700991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb358072180 0x7fb3580b1b20 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:32.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.237+0000 7fb35d800640 1 -- 172.21.15.45:0/2995700991 >> 172.21.15.45:0/2995700991 conn(0x7fb35806b930 msgr2=0x7fb3580b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:32.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.237+0000 7fb35d800640 1 -- 172.21.15.45:0/2995700991 shutdown_connections 2024-04-16T16:22:32.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.237+0000 7fb35d800640 1 -- 172.21.15.45:0/2995700991 wait complete. 2024-04-16T16:22:32.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-16T16:22:32.373 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-16T16:22:32.374 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-16T16:22:32.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.485+0000 7f259b200640 1 Processor -- start 2024-04-16T16:22:32.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.485+0000 7f259b200640 1 -- start start 2024-04-16T16:22:32.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.486+0000 7f259b200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f259c06ed60 0x7f259c071f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:32.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.486+0000 7f259b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f259c072460 0x7f259c15d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:32.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.486+0000 7f259b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f259c0cece0 con 0x7f259c1460f0 2024-04-16T16:22:32.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.487+0000 7f259b200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f259c06f1d0 con 0x7f259c06ed60 2024-04-16T16:22:32.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.487+0000 7f259b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f259c0728d0 con 0x7f259c072460 2024-04-16T16:22:32.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.487+0000 7f2593e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f259c06ed60 0x7f259c071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:32.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.487+0000 7f2593e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f259c06ed60 0x7f259c071f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53382/0 (socket says 172.21.15.45:53382) 2024-04-16T16:22:32.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.487+0000 7f2593e00640 1 -- 172.21.15.45:0/1696547382 learned_addr learned my addr 172.21.15.45:0/1696547382 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:32.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.487+0000 7f2593e00640 1 -- 172.21.15.45:0/1696547382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f259c072460 msgr2=0x7f259c15d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:32.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.487+0000 7f2593e00640 1 --2- 172.21.15.45:0/1696547382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f259c072460 0x7f259c15d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:32.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.487+0000 7f2593e00640 1 -- 172.21.15.45:0/1696547382 >> v1:172.21.15.103:6789/0 conn(0x7f259c1460f0 legacy=0x7f259c0ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:32.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.487+0000 7f2593e00640 1 -- 172.21.15.45:0/1696547382 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f259c0cefc0 con 0x7f259c06ed60 2024-04-16T16:22:32.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.489+0000 7f2593e00640 1 --2- 172.21.15.45:0/1696547382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f259c06ed60 0x7f259c071f20 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f2584009850 tx=0x7f258402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749ce175fcba0c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:32.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.489+0000 7f2599400640 1 -- 172.21.15.45:0/1696547382 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584004230 con 0x7f259c06ed60 2024-04-16T16:22:32.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.489+0000 7f2599400640 1 -- 172.21.15.45:0/1696547382 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25840043d0 con 0x7f259c06ed60 2024-04-16T16:22:32.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.489+0000 7f2599400640 1 -- 172.21.15.45:0/1696547382 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584003810 con 0x7f259c06ed60 2024-04-16T16:22:32.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.490+0000 7f259b200640 1 -- 172.21.15.45:0/1696547382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f259c06ed60 msgr2=0x7f259c071f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:32.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.490+0000 7f259b200640 1 --2- 172.21.15.45:0/1696547382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f259c06ed60 0x7f259c071f20 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f2584009850 tx=0x7f258402b960 comp rx=0 tx=0).stop 2024-04-16T16:22:32.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.490+0000 7f259b200640 1 -- 172.21.15.45:0/1696547382 shutdown_connections 2024-04-16T16:22:32.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.490+0000 7f259b200640 1 --2- 172.21.15.45:0/1696547382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f259c072460 0x7f259c15d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:32.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.490+0000 7f259b200640 1 --2- 172.21.15.45:0/1696547382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f259c06ed60 0x7f259c071f20 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:32.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.490+0000 7f259b200640 1 -- 172.21.15.45:0/1696547382 >> 172.21.15.45:0/1696547382 conn(0x7f259c067e00 msgr2=0x7f259c06b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:32.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.490+0000 7f259b200640 1 -- 172.21.15.45:0/1696547382 shutdown_connections 2024-04-16T16:22:32.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.490+0000 7f259b200640 1 -- 172.21.15.45:0/1696547382 wait complete. 2024-04-16T16:22:32.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.492+0000 7f259b200640 1 Processor -- start 2024-04-16T16:22:32.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.492+0000 7f259b200640 1 -- start start 2024-04-16T16:22:32.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.493+0000 7f259b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f259c06ed60 0x7f259c1380a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:32.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.493+0000 7f259b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f259c072460 0x7f259c0c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:32.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.493+0000 7f259b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f259c1460f0 0x7f259c1446f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:32.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.493+0000 7f259b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f259c15d200 con 0x7f259c06ed60 2024-04-16T16:22:32.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.493+0000 7f259b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f259c1464d0 con 0x7f259c1460f0 2024-04-16T16:22:32.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.493+0000 7f259b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f259c15cc20 con 0x7f259c072460 2024-04-16T16:22:32.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.493+0000 7f2599e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f259c06ed60 0x7f259c1380a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:32.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.493+0000 7f2599e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f259c06ed60 0x7f259c1380a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53384/0 (socket says 172.21.15.45:53384) 2024-04-16T16:22:32.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.493+0000 7f2599e00640 1 -- 172.21.15.45:0/1468963583 learned_addr learned my addr 172.21.15.45:0/1468963583 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:32.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.493+0000 7f2599e00640 1 -- 172.21.15.45:0/1468963583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f259c072460 msgr2=0x7f259c0c5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:32.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.493+0000 7f2599e00640 1 --2- 172.21.15.45:0/1468963583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f259c072460 0x7f259c0c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:32.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.493+0000 7f2599e00640 1 -- 172.21.15.45:0/1468963583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f259c1460f0 msgr2=0x7f259c1446f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:32.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.493+0000 7f259a800640 1 --2- 172.21.15.45:0/1468963583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f259c1460f0 0x7f259c1446f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:32.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.493+0000 7f2599e00640 1 --2- 172.21.15.45:0/1468963583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f259c1460f0 0x7f259c1446f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:32.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.493+0000 7f2599e00640 1 -- 172.21.15.45:0/1468963583 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f259c0b45e0 con 0x7f259c06ed60 2024-04-16T16:22:32.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.494+0000 7f2599e00640 1 --2- 172.21.15.45:0/1468963583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f259c06ed60 0x7f259c1380a0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f258c00b3f0 tx=0x7f258c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:32.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.494+0000 7f2592000640 1 -- 172.21.15.45:0/1468963583 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c015020 con 0x7f259c06ed60 2024-04-16T16:22:32.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.494+0000 7f2592000640 1 -- 172.21.15.45:0/1468963583 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f258c011830 con 0x7f259c06ed60 2024-04-16T16:22:32.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.494+0000 7f2592000640 1 -- 172.21.15.45:0/1468963583 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c011c20 con 0x7f259c06ed60 2024-04-16T16:22:32.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.494+0000 7f259b200640 1 -- 172.21.15.45:0/1468963583 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f259c0b48c0 con 0x7f259c06ed60 2024-04-16T16:22:32.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.494+0000 7f259b200640 1 -- 172.21.15.45:0/1468963583 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f259c0b4e80 con 0x7f259c06ed60 2024-04-16T16:22:32.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.495+0000 7f259b200640 1 -- 172.21.15.45:0/1468963583 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f259c075780 con 0x7f259c06ed60 2024-04-16T16:22:32.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.497+0000 7f2592000640 1 -- 172.21.15.45:0/1468963583 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f258c018020 con 0x7f259c06ed60 2024-04-16T16:22:32.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.498+0000 7f2592000640 1 --2- 172.21.15.45:0/1468963583 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f257806ca10 0x7f257806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:32.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.498+0000 7f2592000640 1 -- 172.21.15.45:0/1468963583 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 6569+0+0 (secure 0 0 0) 0x7f258c08bd50 con 0x7f259c06ed60 2024-04-16T16:22:32.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.499+0000 7f2593e00640 1 --2- 172.21.15.45:0/1468963583 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f257806ca10 0x7f257806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:32.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.500+0000 7f2593e00640 1 --2- 172.21.15.45:0/1468963583 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f257806ca10 0x7f257806eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2584035000 tx=0x7f2584030000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:32.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.503+0000 7f2592000640 1 -- 172.21.15.45:0/1468963583 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f258c01f030 con 0x7f259c06ed60 2024-04-16T16:22:32.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:32.747+0000 7f259b200640 1 -- 172.21.15.45:0/1468963583 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f259c06b9d0 con 0x7f259c06ed60 2024-04-16T16:22:33.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.280+0000 7f2592000640 1 -- 172.21.15.45:0/1468963583 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v107) ==== 104+0+0 (secure 0 0 0) 0x7f258c059c70 con 0x7f259c06ed60 2024-04-16T16:22:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.284+0000 7f259b200640 1 -- 172.21.15.45:0/1468963583 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f257806ca10 msgr2=0x7f257806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.284+0000 7f259b200640 1 --2- 172.21.15.45:0/1468963583 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f257806ca10 0x7f257806eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2584035000 tx=0x7f2584030000 comp rx=0 tx=0).stop 2024-04-16T16:22:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.284+0000 7f259b200640 1 -- 172.21.15.45:0/1468963583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f259c06ed60 msgr2=0x7f259c1380a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.284+0000 7f259b200640 1 --2- 172.21.15.45:0/1468963583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f259c06ed60 0x7f259c1380a0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f258c00b3f0 tx=0x7f258c00b8b0 comp rx=0 tx=0).stop 2024-04-16T16:22:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.284+0000 7f259b200640 1 -- 172.21.15.45:0/1468963583 shutdown_connections 2024-04-16T16:22:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.284+0000 7f259b200640 1 --2- 172.21.15.45:0/1468963583 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f257806ca10 0x7f257806eed0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.284+0000 7f259b200640 1 --2- 172.21.15.45:0/1468963583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f259c1460f0 0x7f259c1446f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.284+0000 7f259b200640 1 --2- 172.21.15.45:0/1468963583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f259c072460 0x7f259c0c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.284+0000 7f259b200640 1 --2- 172.21.15.45:0/1468963583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f259c06ed60 0x7f259c1380a0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:33.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.284+0000 7f259b200640 1 -- 172.21.15.45:0/1468963583 >> 172.21.15.45:0/1468963583 conn(0x7f259c067e00 msgr2=0x7f259c0b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:33.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.284+0000 7f259b200640 1 -- 172.21.15.45:0/1468963583 shutdown_connections 2024-04-16T16:22:33.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.285+0000 7f259b200640 1 -- 172.21.15.45:0/1468963583 wait complete. 2024-04-16T16:22:33.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-16T16:22:33.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.537+0000 7fd49be00640 1 Processor -- start 2024-04-16T16:22:33.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.537+0000 7fd49be00640 1 -- start start 2024-04-16T16:22:33.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.538+0000 7fd49be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd49c15c7f0 0x7fd49c09aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:33.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.538+0000 7fd49be00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd49c0b3180 con 0x7fd49c0af3b0 2024-04-16T16:22:33.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.538+0000 7fd49be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd49c0af840 con 0x7fd49c09b680 2024-04-16T16:22:33.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.538+0000 7fd49be00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd49c0afb20 con 0x7fd49c15c7f0 2024-04-16T16:22:33.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.539+0000 7fd49a000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd49c0af3b0 0x7fd49c0b2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53058/0 (socket says 172.21.15.45:53058) 2024-04-16T16:22:33.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.539+0000 7fd49a000640 1 -- 172.21.15.45:0/3647364266 learned_addr learned my addr 172.21.15.45:0/3647364266 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:33.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.539+0000 7fd49aa00640 1 --2- 172.21.15.45:0/3647364266 >> v2:172.21.15.45:3300/0 conn(0x7fd49c15c7f0 0x7fd49c09aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.539+0000 7fd499600640 1 -- 172.21.15.45:0/3647364266 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1716422093 0 0) 0x7fd49c0b3180 con 0x7fd49c0af3b0 2024-04-16T16:22:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.539+0000 7fd499600640 1 -- 172.21.15.45:0/3647364266 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd480003520 con 0x7fd49c0af3b0 2024-04-16T16:22:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.540+0000 7fd49aa00640 1 -- 172.21.15.45:0/3647364266 >> v1:172.21.15.103:6789/0 conn(0x7fd49c09b680 legacy=0x7fd49c0b2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.540+0000 7fd49aa00640 1 -- 172.21.15.45:0/3647364266 >> v1:172.21.15.45:6789/0 conn(0x7fd49c0af3b0 legacy=0x7fd49c0b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.540+0000 7fd49aa00640 1 -- 172.21.15.45:0/3647364266 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd49c0afe00 con 0x7fd49c15c7f0 2024-04-16T16:22:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.541+0000 7fd49aa00640 1 --2- 172.21.15.45:0/3647364266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd49c15c7f0 0x7fd49c09aa70 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0097d0 tx=0x7fd48c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c4518bea7d92d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.541+0000 7fd499600640 1 -- 172.21.15.45:0/3647364266 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48c0315a0 con 0x7fd49c15c7f0 2024-04-16T16:22:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.542+0000 7fd499600640 1 -- 172.21.15.45:0/3647364266 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd48c037070 con 0x7fd49c15c7f0 2024-04-16T16:22:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.542+0000 7fd499600640 1 -- 172.21.15.45:0/3647364266 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48c031940 con 0x7fd49c15c7f0 2024-04-16T16:22:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.542+0000 7fd49be00640 1 -- 172.21.15.45:0/3647364266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd49c15c7f0 msgr2=0x7fd49c09aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.542+0000 7fd49be00640 1 --2- 172.21.15.45:0/3647364266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd49c15c7f0 0x7fd49c09aa70 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0097d0 tx=0x7fd48c02b840 comp rx=0 tx=0).stop 2024-04-16T16:22:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.542+0000 7fd49be00640 1 -- 172.21.15.45:0/3647364266 shutdown_connections 2024-04-16T16:22:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.542+0000 7fd49be00640 1 --2- 172.21.15.45:0/3647364266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd49c15c7f0 0x7fd49c09aa70 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.542+0000 7fd49be00640 1 -- 172.21.15.45:0/3647364266 >> 172.21.15.45:0/3647364266 conn(0x7fd49c0d2600 msgr2=0x7fd49c0d29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:33.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.542+0000 7fd49be00640 1 -- 172.21.15.45:0/3647364266 shutdown_connections 2024-04-16T16:22:33.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.542+0000 7fd49be00640 1 -- 172.21.15.45:0/3647364266 wait complete. 2024-04-16T16:22:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.544+0000 7fd49be00640 1 Processor -- start 2024-04-16T16:22:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.544+0000 7fd49be00640 1 -- start start 2024-04-16T16:22:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.545+0000 7fd49be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd49c09b680 0x7fd49c06cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.545+0000 7fd49be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd49c15c7f0 0x7fd49c0b5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.545+0000 7fd49be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd49c1f7480 0x7fd49c15f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.545+0000 7fd49be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd49c15cbd0 con 0x7fd49c1f7480 2024-04-16T16:22:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.545+0000 7fd49be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd49c0b03c0 con 0x7fd49c09b680 2024-04-16T16:22:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.545+0000 7fd49be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd49c0b18f0 con 0x7fd49c15c7f0 2024-04-16T16:22:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.545+0000 7fd49b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd49c1f7480 0x7fd49c15f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.545+0000 7fd49b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd49c1f7480 0x7fd49c15f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58132/0 (socket says 172.21.15.45:58132) 2024-04-16T16:22:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.546+0000 7fd49b400640 1 -- 172.21.15.45:0/609661423 learned_addr learned my addr 172.21.15.45:0/609661423 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.546+0000 7fd49a000640 1 --2- 172.21.15.45:0/609661423 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd49c15c7f0 0x7fd49c0b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.546+0000 7fd49aa00640 1 --2- 172.21.15.45:0/609661423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd49c09b680 0x7fd49c06cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.546+0000 7fd49b400640 1 -- 172.21.15.45:0/609661423 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd49c15c7f0 msgr2=0x7fd49c0b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.546+0000 7fd49b400640 1 --2- 172.21.15.45:0/609661423 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd49c15c7f0 0x7fd49c0b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.546+0000 7fd49b400640 1 -- 172.21.15.45:0/609661423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd49c09b680 msgr2=0x7fd49c06cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.546+0000 7fd49b400640 1 --2- 172.21.15.45:0/609661423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd49c09b680 0x7fd49c06cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.546+0000 7fd49b400640 1 -- 172.21.15.45:0/609661423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd49c0b5760 con 0x7fd49c1f7480 2024-04-16T16:22:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.546+0000 7fd49a000640 1 --2- 172.21.15.45:0/609661423 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd49c15c7f0 0x7fd49c0b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.546+0000 7fd49b400640 1 --2- 172.21.15.45:0/609661423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd49c1f7480 0x7fd49c15f5d0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7fd49400c730 tx=0x7fd49400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.547+0000 7fd492a00640 1 -- 172.21.15.45:0/609661423 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd494016020 con 0x7fd49c1f7480 2024-04-16T16:22:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.547+0000 7fd492a00640 1 -- 172.21.15.45:0/609661423 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd494006d50 con 0x7fd49c1f7480 2024-04-16T16:22:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.547+0000 7fd49be00640 1 -- 172.21.15.45:0/609661423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd49c06dc30 con 0x7fd49c1f7480 2024-04-16T16:22:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.547+0000 7fd492a00640 1 -- 172.21.15.45:0/609661423 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd494003bb0 con 0x7fd49c1f7480 2024-04-16T16:22:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.547+0000 7fd49be00640 1 -- 172.21.15.45:0/609661423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd49c06e1f0 con 0x7fd49c1f7480 2024-04-16T16:22:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.548+0000 7fd49be00640 1 -- 172.21.15.45:0/609661423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd49c075070 con 0x7fd49c1f7480 2024-04-16T16:22:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.549+0000 7fd492a00640 1 -- 172.21.15.45:0/609661423 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fd49400a7f0 con 0x7fd49c1f7480 2024-04-16T16:22:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.550+0000 7fd492a00640 1 --2- 172.21.15.45:0/609661423 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd47c06ca10 0x7fd47c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.550+0000 7fd492a00640 1 -- 172.21.15.45:0/609661423 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 6569+0+0 (secure 0 0 0) 0x7fd49408bc70 con 0x7fd49c1f7480 2024-04-16T16:22:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.551+0000 7fd49aa00640 1 --2- 172.21.15.45:0/609661423 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd47c06ca10 0x7fd47c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.552+0000 7fd49aa00640 1 --2- 172.21.15.45:0/609661423 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd47c06ca10 0x7fd47c06eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd48c036000 tx=0x7fd48c008030 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.555+0000 7fd492a00640 1 -- 172.21.15.45:0/609661423 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd494020020 con 0x7fd49c1f7480 2024-04-16T16:22:33.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:33.756+0000 7fd49be00640 1 -- 172.21.15.45:0/609661423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd49c070f80 con 0x7fd49c1f7480 2024-04-16T16:22:34.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.301+0000 7fd492a00640 1 -- 172.21.15.45:0/609661423 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v146) ==== 181+0+0 (secure 0 0 0) 0x7fd494059cc0 con 0x7fd49c1f7480 2024-04-16T16:22:34.300 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2024-04-16T16:22:34.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.305+0000 7fd49be00640 1 -- 172.21.15.45:0/609661423 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd47c06ca10 msgr2=0x7fd47c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:34.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.305+0000 7fd49be00640 1 --2- 172.21.15.45:0/609661423 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd47c06ca10 0x7fd47c06eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd48c036000 tx=0x7fd48c008030 comp rx=0 tx=0).stop 2024-04-16T16:22:34.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.305+0000 7fd49be00640 1 -- 172.21.15.45:0/609661423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd49c1f7480 msgr2=0x7fd49c15f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:34.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.305+0000 7fd49be00640 1 --2- 172.21.15.45:0/609661423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd49c1f7480 0x7fd49c15f5d0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7fd49400c730 tx=0x7fd49400cbf0 comp rx=0 tx=0).stop 2024-04-16T16:22:34.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.305+0000 7fd49be00640 1 -- 172.21.15.45:0/609661423 shutdown_connections 2024-04-16T16:22:34.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.305+0000 7fd49be00640 1 --2- 172.21.15.45:0/609661423 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd47c06ca10 0x7fd47c06eed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:34.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.305+0000 7fd49be00640 1 --2- 172.21.15.45:0/609661423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd49c1f7480 0x7fd49c15f5d0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:34.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.305+0000 7fd49be00640 1 --2- 172.21.15.45:0/609661423 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd49c15c7f0 0x7fd49c0b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:34.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.305+0000 7fd49be00640 1 --2- 172.21.15.45:0/609661423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd49c09b680 0x7fd49c06cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:34.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.305+0000 7fd49be00640 1 -- 172.21.15.45:0/609661423 >> 172.21.15.45:0/609661423 conn(0x7fd49c0d2600 msgr2=0x7fd49c070160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:34.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.306+0000 7fd49be00640 1 -- 172.21.15.45:0/609661423 shutdown_connections 2024-04-16T16:22:34.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.306+0000 7fd49be00640 1 -- 172.21.15.45:0/609661423 wait complete. 2024-04-16T16:22:34.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-16T16:22:34.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.537+0000 7f2994400640 1 Processor -- start 2024-04-16T16:22:34.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.538+0000 7f2994400640 1 -- start start 2024-04-16T16:22:34.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.539+0000 7f2994400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f298c1622d0 0x7f298c0cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:34.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.539+0000 7f2994400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f298c0d1f10 0x7f298c0cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:34.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.539+0000 7f2994400640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f298c0d4640 con 0x7f298c162a50 2024-04-16T16:22:34.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.539+0000 7f2994400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f298c0d43f0 con 0x7f298c0d1f10 2024-04-16T16:22:34.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.539+0000 7f2994400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f298c0ba4c0 con 0x7f298c1622d0 2024-04-16T16:22:34.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.539+0000 7f2991000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f298c0d1f10 0x7f298c0cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:34.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.540+0000 7f2991000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f298c0d1f10 0x7f298c0cfc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58134/0 (socket says 172.21.15.45:58134) 2024-04-16T16:22:34.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.540+0000 7f2991000640 1 -- 172.21.15.45:0/2914076677 learned_addr learned my addr 172.21.15.45:0/2914076677 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:34.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.540+0000 7f2991a00640 1 --2- 172.21.15.45:0/2914076677 >> v2:172.21.15.103:3300/0 conn(0x7f298c1622d0 0x7f298c0cf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:34.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.540+0000 7f2991000640 1 -- 172.21.15.45:0/2914076677 >> v2:172.21.15.103:3300/0 conn(0x7f298c1622d0 msgr2=0x7f298c0cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:34.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.540+0000 7f2991000640 1 --2- 172.21.15.45:0/2914076677 >> v2:172.21.15.103:3300/0 conn(0x7f298c1622d0 0x7f298c0cf6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:34.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.540+0000 7f2991000640 1 -- 172.21.15.45:0/2914076677 >> v1:172.21.15.45:6789/0 conn(0x7f298c162a50 legacy=0x7f298c162e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:34.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.540+0000 7f2991000640 1 -- 172.21.15.45:0/2914076677 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f298c0b7b50 con 0x7f298c0d1f10 2024-04-16T16:22:34.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.540+0000 7f2991a00640 1 --2- 172.21.15.45:0/2914076677 >> v2:172.21.15.103:3300/0 conn(0x7f298c1622d0 0x7f298c0cf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:34.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.541+0000 7f2991000640 1 --2- 172.21.15.45:0/2914076677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f298c0d1f10 0x7f298c0cfc30 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f298c08da50 tx=0x7f297802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5644c97322f97354 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:34.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.542+0000 7f298be00640 1 -- 172.21.15.45:0/2914076677 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2978034070 con 0x7f298c0d1f10 2024-04-16T16:22:34.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.542+0000 7f298be00640 1 -- 172.21.15.45:0/2914076677 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2978030870 con 0x7f298c0d1f10 2024-04-16T16:22:34.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.542+0000 7f298be00640 1 -- 172.21.15.45:0/2914076677 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2978030c40 con 0x7f298c0d1f10 2024-04-16T16:22:34.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.542+0000 7f2994400640 1 -- 172.21.15.45:0/2914076677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f298c0d1f10 msgr2=0x7f298c0cfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:34.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.542+0000 7f2994400640 1 --2- 172.21.15.45:0/2914076677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f298c0d1f10 0x7f298c0cfc30 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f298c08da50 tx=0x7f297802b9b0 comp rx=0 tx=0).stop 2024-04-16T16:22:34.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.543+0000 7f2994400640 1 -- 172.21.15.45:0/2914076677 shutdown_connections 2024-04-16T16:22:34.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.543+0000 7f2994400640 1 --2- 172.21.15.45:0/2914076677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f298c0d1f10 0x7f298c0cfc30 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:34.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.543+0000 7f2994400640 1 --2- 172.21.15.45:0/2914076677 >> v2:172.21.15.103:3300/0 conn(0x7f298c1622d0 0x7f298c0cf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:34.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.543+0000 7f2994400640 1 -- 172.21.15.45:0/2914076677 >> 172.21.15.45:0/2914076677 conn(0x7f298c06d400 msgr2=0x7f298c06d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:34.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.543+0000 7f2994400640 1 -- 172.21.15.45:0/2914076677 shutdown_connections 2024-04-16T16:22:34.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.543+0000 7f2994400640 1 -- 172.21.15.45:0/2914076677 wait complete. 2024-04-16T16:22:34.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.544+0000 7f2994400640 1 Processor -- start 2024-04-16T16:22:34.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.544+0000 7f2994400640 1 -- start start 2024-04-16T16:22:34.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.545+0000 7f2994400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f298c0d1f10 0x7f298c1f7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:34.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.545+0000 7f2994400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f298c1622d0 0x7f298c14ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:34.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.545+0000 7f2994400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f298c162a50 0x7f298c0b55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:34.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.545+0000 7f2994400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f298c0b70e0 con 0x7f298c0d1f10 2024-04-16T16:22:34.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.545+0000 7f2994400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f298c0b6f60 con 0x7f298c1622d0 2024-04-16T16:22:34.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.545+0000 7f2994400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f298c163570 con 0x7f298c162a50 2024-04-16T16:22:34.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.546+0000 7f2991a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f298c0d1f10 0x7f298c1f7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:34.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.546+0000 7f2991a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f298c0d1f10 0x7f298c1f7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58138/0 (socket says 172.21.15.45:58138) 2024-04-16T16:22:34.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.546+0000 7f2991a00640 1 -- 172.21.15.45:0/4063006506 learned_addr learned my addr 172.21.15.45:0/4063006506 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:34.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.546+0000 7f2991000640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f298c1622d0 0x7f298c14ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:34.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.546+0000 7f2992400640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f298c162a50 0x7f298c0b55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:34.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.546+0000 7f2991a00640 1 -- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f298c162a50 msgr2=0x7f298c0b55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:34.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.546+0000 7f2991a00640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f298c162a50 0x7f298c0b55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:34.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.546+0000 7f2991a00640 1 -- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f298c1622d0 msgr2=0x7f298c14ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:34.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.546+0000 7f2991a00640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f298c1622d0 0x7f298c14ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:34.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.546+0000 7f2991a00640 1 -- 172.21.15.45:0/4063006506 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f298c1f87e0 con 0x7f298c0d1f10 2024-04-16T16:22:34.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.546+0000 7f2991000640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f298c1622d0 0x7f298c14ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:34.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.546+0000 7f2992400640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f298c162a50 0x7f298c0b55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:34.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.547+0000 7f2991a00640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f298c0d1f10 0x7f298c1f7ee0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f298000c820 tx=0x7f298000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:34.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.547+0000 7f2989600640 1 -- 172.21.15.45:0/4063006506 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980015070 con 0x7f298c0d1f10 2024-04-16T16:22:34.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.547+0000 7f2989600640 1 -- 172.21.15.45:0/4063006506 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2980007a30 con 0x7f298c0d1f10 2024-04-16T16:22:34.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.547+0000 7f2989600640 1 -- 172.21.15.45:0/4063006506 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980007e00 con 0x7f298c0d1f10 2024-04-16T16:22:34.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.547+0000 7f2994400640 1 -- 172.21.15.45:0/4063006506 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f298c1f8ac0 con 0x7f298c0d1f10 2024-04-16T16:22:34.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.547+0000 7f2994400640 1 -- 172.21.15.45:0/4063006506 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f298c1f8420 con 0x7f298c0d1f10 2024-04-16T16:22:34.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.549+0000 7f2994400640 1 -- 172.21.15.45:0/4063006506 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f298c075780 con 0x7f298c0d1f10 2024-04-16T16:22:34.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.550+0000 7f2989600640 1 -- 172.21.15.45:0/4063006506 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f2980018070 con 0x7f298c0d1f10 2024-04-16T16:22:34.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.551+0000 7f2989600640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f296c06ca10 0x7f296c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:34.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.551+0000 7f2989600640 1 -- 172.21.15.45:0/4063006506 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 6131+0+0 (secure 0 0 0) 0x7f298008bef0 con 0x7f298c0d1f10 2024-04-16T16:22:34.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.552+0000 7f2991000640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f296c06ca10 0x7f296c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:34.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.553+0000 7f2991000640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f296c06ca10 0x7f296c06eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2978036000 tx=0x7f297803b040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:34.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.556+0000 7f2989600640 1 -- 172.21.15.45:0/4063006506 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2980059fe0 con 0x7f298c0d1f10 2024-04-16T16:22:34.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:34.763+0000 7f2994400640 1 -- 172.21.15.45:0/4063006506 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f298c163140 con 0x7f298c0d1f10 2024-04-16T16:22:35.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.305+0000 7f2989600640 1 -- 172.21.15.45:0/4063006506 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v147) ==== 169+0+0 (secure 0 0 0) 0x7f298005ee20 con 0x7f298c0d1f10 2024-04-16T16:22:35.304 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2024-04-16T16:22:35.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.309+0000 7f2994400640 1 -- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f296c06ca10 msgr2=0x7f296c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:35.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.309+0000 7f2994400640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f296c06ca10 0x7f296c06eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2978036000 tx=0x7f297803b040 comp rx=0 tx=0).stop 2024-04-16T16:22:35.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.309+0000 7f2994400640 1 -- 172.21.15.45:0/4063006506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f298c0d1f10 msgr2=0x7f298c1f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:35.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.309+0000 7f2994400640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f298c0d1f10 0x7f298c1f7ee0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f298000c820 tx=0x7f298000cc20 comp rx=0 tx=0).stop 2024-04-16T16:22:35.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.309+0000 7f2994400640 1 -- 172.21.15.45:0/4063006506 shutdown_connections 2024-04-16T16:22:35.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.309+0000 7f2994400640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f296c06ca10 0x7f296c06eed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:35.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.309+0000 7f2994400640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f298c162a50 0x7f298c0b55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:35.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.309+0000 7f2994400640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f298c1622d0 0x7f298c14ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:35.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.309+0000 7f2994400640 1 --2- 172.21.15.45:0/4063006506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f298c0d1f10 0x7f298c1f7ee0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:35.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.309+0000 7f2994400640 1 -- 172.21.15.45:0/4063006506 >> 172.21.15.45:0/4063006506 conn(0x7f298c06d400 msgr2=0x7f298c161530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:35.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.310+0000 7f2994400640 1 -- 172.21.15.45:0/4063006506 shutdown_connections 2024-04-16T16:22:35.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.310+0000 7f2994400640 1 -- 172.21.15.45:0/4063006506 wait complete. 2024-04-16T16:22:35.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-16T16:22:35.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.539+0000 7f6e6ea00640 1 Processor -- start 2024-04-16T16:22:35.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.539+0000 7f6e6ea00640 1 -- start start 2024-04-16T16:22:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.541+0000 7f6e6ea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6e6806ed60 0x7f6e68071f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.541+0000 7f6e6ea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6e68072460 0x7f6e6815d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.541+0000 7f6e6ea00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e680cece0 con 0x7f6e681460f0 2024-04-16T16:22:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.541+0000 7f6e6ea00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f6e6806f1d0 con 0x7f6e6806ed60 2024-04-16T16:22:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.541+0000 7f6e6ea00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6e680728b0 con 0x7f6e68072460 2024-04-16T16:22:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.541+0000 7f6e6d600640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6e681460f0 0x7f6e680ce520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53082/0 (socket says 172.21.15.45:53082) 2024-04-16T16:22:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.541+0000 7f6e6d600640 1 -- 172.21.15.45:0/2677079833 learned_addr learned my addr 172.21.15.45:0/2677079833 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:35.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.541+0000 7f6e6cc00640 1 --2- 172.21.15.45:0/2677079833 >> v2:172.21.15.45:3300/0 conn(0x7f6e6806ed60 0x7f6e68071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:35.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.542+0000 7f6e6e000640 1 --2- 172.21.15.45:0/2677079833 >> v2:172.21.15.103:3300/0 conn(0x7f6e68072460 0x7f6e6815d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:35.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.542+0000 7f6e6cc00640 1 -- 172.21.15.45:0/2677079833 >> v2:172.21.15.103:3300/0 conn(0x7f6e68072460 msgr2=0x7f6e6815d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:35.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.542+0000 7f6e6cc00640 1 --2- 172.21.15.45:0/2677079833 >> v2:172.21.15.103:3300/0 conn(0x7f6e68072460 0x7f6e6815d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:35.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.542+0000 7f6e6cc00640 1 -- 172.21.15.45:0/2677079833 >> v1:172.21.15.45:6789/0 conn(0x7f6e681460f0 legacy=0x7f6e680ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:35.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.542+0000 7f6e6cc00640 1 -- 172.21.15.45:0/2677079833 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e680cefc0 con 0x7f6e6806ed60 2024-04-16T16:22:35.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.542+0000 7f6e6e000640 1 --2- 172.21.15.45:0/2677079833 >> v2:172.21.15.103:3300/0 conn(0x7f6e68072460 0x7f6e6815d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:35.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.543+0000 7f6e6cc00640 1 --2- 172.21.15.45:0/2677079833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e6806ed60 0x7f6e68071f20 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f6e680a94d0 tx=0x7f6e5402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3dce13de51e498c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:35.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.543+0000 7f6e67e00640 1 -- 172.21.15.45:0/2677079833 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e54035020 con 0x7f6e6806ed60 2024-04-16T16:22:35.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.543+0000 7f6e67e00640 1 -- 172.21.15.45:0/2677079833 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e54031830 con 0x7f6e6806ed60 2024-04-16T16:22:35.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.543+0000 7f6e67e00640 1 -- 172.21.15.45:0/2677079833 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e54031c00 con 0x7f6e6806ed60 2024-04-16T16:22:35.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.544+0000 7f6e6ea00640 1 -- 172.21.15.45:0/2677079833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e6806ed60 msgr2=0x7f6e68071f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:35.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.544+0000 7f6e6ea00640 1 --2- 172.21.15.45:0/2677079833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e6806ed60 0x7f6e68071f20 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f6e680a94d0 tx=0x7f6e5402b8d0 comp rx=0 tx=0).stop 2024-04-16T16:22:35.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.544+0000 7f6e6ea00640 1 -- 172.21.15.45:0/2677079833 shutdown_connections 2024-04-16T16:22:35.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.544+0000 7f6e6ea00640 1 --2- 172.21.15.45:0/2677079833 >> v2:172.21.15.103:3300/0 conn(0x7f6e68072460 0x7f6e6815d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:35.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.544+0000 7f6e6ea00640 1 --2- 172.21.15.45:0/2677079833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e6806ed60 0x7f6e68071f20 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:35.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.544+0000 7f6e6ea00640 1 -- 172.21.15.45:0/2677079833 >> 172.21.15.45:0/2677079833 conn(0x7f6e68067e00 msgr2=0x7f6e6806b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:35.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.544+0000 7f6e6ea00640 1 -- 172.21.15.45:0/2677079833 shutdown_connections 2024-04-16T16:22:35.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.544+0000 7f6e6ea00640 1 -- 172.21.15.45:0/2677079833 wait complete. 2024-04-16T16:22:35.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.546+0000 7f6e6ea00640 1 Processor -- start 2024-04-16T16:22:35.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.546+0000 7f6e6ea00640 1 -- start start 2024-04-16T16:22:35.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.547+0000 7f6e6ea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e6806ed60 0x7f6e6815c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:35.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.547+0000 7f6e6ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e68072460 0x7f6e680b2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:35.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.547+0000 7f6e6ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e681460f0 0x7f6e680b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:35.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.547+0000 7f6e6ea00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6815d200 con 0x7f6e681460f0 2024-04-16T16:22:35.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.547+0000 7f6e6ea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6e681464d0 con 0x7f6e68072460 2024-04-16T16:22:35.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.547+0000 7f6e6ea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6e6815cc20 con 0x7f6e6806ed60 2024-04-16T16:22:35.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.547+0000 7f6e6e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e681460f0 0x7f6e680b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:35.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.547+0000 7f6e6e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e681460f0 0x7f6e680b2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58160/0 (socket says 172.21.15.45:58160) 2024-04-16T16:22:35.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.547+0000 7f6e6e000640 1 -- 172.21.15.45:0/905617301 learned_addr learned my addr 172.21.15.45:0/905617301 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:35.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.548+0000 7f6e6cc00640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e68072460 0x7f6e680b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:35.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.548+0000 7f6e6e000640 1 -- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e6806ed60 msgr2=0x7f6e6815c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:35.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.548+0000 7f6e6d600640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e6806ed60 0x7f6e6815c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:35.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.548+0000 7f6e6e000640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e6806ed60 0x7f6e6815c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:35.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.548+0000 7f6e6e000640 1 -- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e68072460 msgr2=0x7f6e680b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:35.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.548+0000 7f6e6e000640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e68072460 0x7f6e680b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:35.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.548+0000 7f6e6e000640 1 -- 172.21.15.45:0/905617301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e680ac2c0 con 0x7f6e681460f0 2024-04-16T16:22:35.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.548+0000 7f6e6d600640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e6806ed60 0x7f6e6815c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:35.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.548+0000 7f6e6cc00640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e68072460 0x7f6e680b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:35.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.548+0000 7f6e6e000640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e681460f0 0x7f6e680b2d60 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000cf20 tx=0x7f6e6000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:35.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.548+0000 7f6e65600640 1 -- 172.21.15.45:0/905617301 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6001a070 con 0x7f6e681460f0 2024-04-16T16:22:35.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.549+0000 7f6e65600640 1 -- 172.21.15.45:0/905617301 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e6000d840 con 0x7f6e681460f0 2024-04-16T16:22:35.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.549+0000 7f6e65600640 1 -- 172.21.15.45:0/905617301 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6000dc30 con 0x7f6e681460f0 2024-04-16T16:22:35.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.549+0000 7f6e6ea00640 1 -- 172.21.15.45:0/905617301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e680ac5a0 con 0x7f6e681460f0 2024-04-16T16:22:35.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.549+0000 7f6e6ea00640 1 -- 172.21.15.45:0/905617301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e680acb60 con 0x7f6e681460f0 2024-04-16T16:22:35.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.550+0000 7f6e6ea00640 1 -- 172.21.15.45:0/905617301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e68075780 con 0x7f6e681460f0 2024-04-16T16:22:35.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.551+0000 7f6e65600640 1 -- 172.21.15.45:0/905617301 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f6e6001d070 con 0x7f6e681460f0 2024-04-16T16:22:35.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.552+0000 7f6e65600640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f6e4806ca10 0x7f6e4806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:35.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.552+0000 7f6e65600640 1 -- 172.21.15.45:0/905617301 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5717+0+0 (secure 0 0 0) 0x7f6e6008c5d0 con 0x7f6e681460f0 2024-04-16T16:22:35.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.552+0000 7f6e6d600640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f6e4806ca10 0x7f6e4806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:35.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.553+0000 7f6e6d600640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f6e4806ca10 0x7f6e4806eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c000f60 tx=0x7f6e5c009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:35.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.556+0000 7f6e65600640 1 -- 172.21.15.45:0/905617301 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6e6005a8e0 con 0x7f6e681460f0 2024-04-16T16:22:35.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.761+0000 7f6e6ea00640 1 -- 172.21.15.45:0/905617301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f6e680ce9e0 con 0x7f6e681460f0 2024-04-16T16:22:35.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.769+0000 7f6e65600640 1 -- 172.21.15.45:0/905617301 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) ==== 104+0+0 (secure 0 0 0) 0x7f6e6005a1e0 con 0x7f6e681460f0 2024-04-16T16:22:35.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.772+0000 7f6e6ea00640 1 -- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f6e4806ca10 msgr2=0x7f6e4806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:35.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.772+0000 7f6e6ea00640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f6e4806ca10 0x7f6e4806eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c000f60 tx=0x7f6e5c009040 comp rx=0 tx=0).stop 2024-04-16T16:22:35.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.772+0000 7f6e6ea00640 1 -- 172.21.15.45:0/905617301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e681460f0 msgr2=0x7f6e680b2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:35.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.772+0000 7f6e6ea00640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e681460f0 0x7f6e680b2d60 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000cf20 tx=0x7f6e6000a7d0 comp rx=0 tx=0).stop 2024-04-16T16:22:35.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.772+0000 7f6e6ea00640 1 -- 172.21.15.45:0/905617301 shutdown_connections 2024-04-16T16:22:35.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.772+0000 7f6e6ea00640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f6e4806ca10 0x7f6e4806eed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:35.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.772+0000 7f6e6ea00640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e681460f0 0x7f6e680b2d60 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:35.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.772+0000 7f6e6ea00640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e68072460 0x7f6e680b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:35.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.772+0000 7f6e6ea00640 1 --2- 172.21.15.45:0/905617301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e6806ed60 0x7f6e6815c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:35.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.772+0000 7f6e6ea00640 1 -- 172.21.15.45:0/905617301 >> 172.21.15.45:0/905617301 conn(0x7f6e68067e00 msgr2=0x7f6e68145240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:35.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.772+0000 7f6e6ea00640 1 -- 172.21.15.45:0/905617301 shutdown_connections 2024-04-16T16:22:35.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:35.772+0000 7f6e6ea00640 1 -- 172.21.15.45:0/905617301 wait complete. 2024-04-16T16:22:35.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_metrics_mds_grow' 2024-04-16T16:22:36.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.015+0000 7f230d800640 1 Processor -- start 2024-04-16T16:22:36.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.016+0000 7f230d800640 1 -- start start 2024-04-16T16:22:36.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.017+0000 7f230d800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2308144ff0 0x7f23080cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:36.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.017+0000 7f230d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23080c5d80 0x7f230815d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:36.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.017+0000 7f230d800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f230815c700 con 0x7f230806dfb0 2024-04-16T16:22:36.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.017+0000 7f230d800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f230815d2f0 con 0x7f2308144ff0 2024-04-16T16:22:36.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.017+0000 7f230d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f230815c9c0 con 0x7f23080c5d80 2024-04-16T16:22:36.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.018+0000 7f2307e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2308144ff0 0x7f23080cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:36.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.018+0000 7f2307e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2308144ff0 0x7f23080cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58172/0 (socket says 172.21.15.45:58172) 2024-04-16T16:22:36.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.018+0000 7f2307e00640 1 -- 172.21.15.45:0/1265331582 learned_addr learned my addr 172.21.15.45:0/1265331582 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:36.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.018+0000 7f230ce00640 1 --2- 172.21.15.45:0/1265331582 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23080c5d80 0x7f230815d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:36.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.018+0000 7f2307e00640 1 -- 172.21.15.45:0/1265331582 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23080c5d80 msgr2=0x7f230815d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:36.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.018+0000 7f2307e00640 1 --2- 172.21.15.45:0/1265331582 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23080c5d80 0x7f230815d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.018+0000 7f2307e00640 1 -- 172.21.15.45:0/1265331582 >> v1:172.21.15.103:6789/0 conn(0x7f230806dfb0 legacy=0x7f23080c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:36.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.018+0000 7f2307e00640 1 -- 172.21.15.45:0/1265331582 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230815cca0 con 0x7f2308144ff0 2024-04-16T16:22:36.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.019+0000 7f2307e00640 1 --2- 172.21.15.45:0/1265331582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2308144ff0 0x7f23080cfd30 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009870 tx=0x7f22f802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4605e9e34466130a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:36.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.020+0000 7f2306a00640 1 -- 172.21.15.45:0/1265331582 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f8035020 con 0x7f2308144ff0 2024-04-16T16:22:36.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.020+0000 7f2306a00640 1 -- 172.21.15.45:0/1265331582 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f8031640 con 0x7f2308144ff0 2024-04-16T16:22:36.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.020+0000 7f2306a00640 1 -- 172.21.15.45:0/1265331582 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f802d040 con 0x7f2308144ff0 2024-04-16T16:22:36.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.020+0000 7f230d800640 1 -- 172.21.15.45:0/1265331582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2308144ff0 msgr2=0x7f23080cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:36.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.020+0000 7f230d800640 1 --2- 172.21.15.45:0/1265331582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2308144ff0 0x7f23080cfd30 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009870 tx=0x7f22f802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.020+0000 7f230d800640 1 -- 172.21.15.45:0/1265331582 shutdown_connections 2024-04-16T16:22:36.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.020+0000 7f230d800640 1 --2- 172.21.15.45:0/1265331582 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23080c5d80 0x7f230815d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.020+0000 7f230d800640 1 --2- 172.21.15.45:0/1265331582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2308144ff0 0x7f23080cfd30 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.020+0000 7f230d800640 1 -- 172.21.15.45:0/1265331582 >> 172.21.15.45:0/1265331582 conn(0x7f230806b930 msgr2=0x7f230806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:36.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.021+0000 7f230d800640 1 -- 172.21.15.45:0/1265331582 shutdown_connections 2024-04-16T16:22:36.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.021+0000 7f230d800640 1 -- 172.21.15.45:0/1265331582 wait complete. 2024-04-16T16:22:36.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.022+0000 7f230d800640 1 Processor -- start 2024-04-16T16:22:36.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.022+0000 7f230d800640 1 -- start start 2024-04-16T16:22:36.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.023+0000 7f230d800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f230806dfb0 0x7f2308132780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:36.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.023+0000 7f230d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23080c5d80 0x7f230815d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:36.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.023+0000 7f230d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2308144ff0 0x7f230815b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:36.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.023+0000 7f230d800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f23080d0270 con 0x7f23080c5d80 2024-04-16T16:22:36.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.023+0000 7f230d800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f23081453d0 con 0x7f230806dfb0 2024-04-16T16:22:36.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.023+0000 7f230d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f23080d03f0 con 0x7f2308144ff0 2024-04-16T16:22:36.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.024+0000 7f2307400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23080c5d80 0x7f230815d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:36.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.024+0000 7f2307400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23080c5d80 0x7f230815d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58182/0 (socket says 172.21.15.45:58182) 2024-04-16T16:22:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.024+0000 7f2307400640 1 -- 172.21.15.45:0/1933193142 learned_addr learned my addr 172.21.15.45:0/1933193142 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.024+0000 7f230ce00640 1 --2- 172.21.15.45:0/1933193142 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2308144ff0 0x7f230815b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.024+0000 7f2307e00640 1 --2- 172.21.15.45:0/1933193142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f230806dfb0 0x7f2308132780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.024+0000 7f2307400640 1 -- 172.21.15.45:0/1933193142 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2308144ff0 msgr2=0x7f230815b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.024+0000 7f2307400640 1 --2- 172.21.15.45:0/1933193142 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2308144ff0 0x7f230815b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.024+0000 7f2307400640 1 -- 172.21.15.45:0/1933193142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f230806dfb0 msgr2=0x7f2308132780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.024+0000 7f2307400640 1 --2- 172.21.15.45:0/1933193142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f230806dfb0 0x7f2308132780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.024+0000 7f2307400640 1 -- 172.21.15.45:0/1933193142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2308132cc0 con 0x7f23080c5d80 2024-04-16T16:22:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.025+0000 7f2307400640 1 --2- 172.21.15.45:0/1933193142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23080c5d80 0x7f230815d9f0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f22f400c400 tx=0x7f22f400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.025+0000 7f22ffe00640 1 -- 172.21.15.45:0/1933193142 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4011790 con 0x7f23080c5d80 2024-04-16T16:22:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.025+0000 7f22ffe00640 1 -- 172.21.15.45:0/1933193142 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f4017070 con 0x7f23080c5d80 2024-04-16T16:22:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.025+0000 7f22ffe00640 1 -- 172.21.15.45:0/1933193142 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4011bb0 con 0x7f23080c5d80 2024-04-16T16:22:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.025+0000 7f230d800640 1 -- 172.21.15.45:0/1933193142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2308132f60 con 0x7f23080c5d80 2024-04-16T16:22:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.025+0000 7f230d800640 1 -- 172.21.15.45:0/1933193142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2308133520 con 0x7f23080c5d80 2024-04-16T16:22:36.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.026+0000 7f230d800640 1 -- 172.21.15.45:0/1933193142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2308075780 con 0x7f23080c5d80 2024-04-16T16:22:36.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.028+0000 7f22ffe00640 1 -- 172.21.15.45:0/1933193142 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f22f401c460 con 0x7f23080c5d80 2024-04-16T16:22:36.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.029+0000 7f22ffe00640 1 --2- 172.21.15.45:0/1933193142 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f22e806ca10 0x7f22e806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:36.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.029+0000 7f22ffe00640 1 -- 172.21.15.45:0/1933193142 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5717+0+0 (secure 0 0 0) 0x7f22f408bc50 con 0x7f23080c5d80 2024-04-16T16:22:36.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.029+0000 7f2307e00640 1 --2- 172.21.15.45:0/1933193142 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f22e806ca10 0x7f22e806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:36.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.030+0000 7f2307e00640 1 --2- 172.21.15.45:0/1933193142 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f22e806ca10 0x7f22e806eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f22f8036000 tx=0x7f22f8008030 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:36.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.033+0000 7f22ffe00640 1 -- 172.21.15.45:0/1933193142 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f22f4059ec0 con 0x7f23080c5d80 2024-04-16T16:22:36.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.314+0000 7f230d800640 1 -- 172.21.15.45:0/1933193142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_metrics_mds_grow"]} v 0) -- 0x7f23080b59c0 con 0x7f23080c5d80 2024-04-16T16:22:36.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.329+0000 7f22ffe00640 1 -- 172.21.15.45:0/1933193142 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_metrics_mds_grow"]}]=0 v517) ==== 145+0+0 (secure 0 0 0) 0x7f22f40597c0 con 0x7f23080c5d80 2024-04-16T16:22:36.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.331+0000 7f230d800640 1 -- 172.21.15.45:0/1933193142 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f22e806ca10 msgr2=0x7f22e806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:36.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.331+0000 7f230d800640 1 --2- 172.21.15.45:0/1933193142 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f22e806ca10 0x7f22e806eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f22f8036000 tx=0x7f22f8008030 comp rx=0 tx=0).stop 2024-04-16T16:22:36.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.331+0000 7f230d800640 1 -- 172.21.15.45:0/1933193142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23080c5d80 msgr2=0x7f230815d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:36.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.331+0000 7f230d800640 1 --2- 172.21.15.45:0/1933193142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23080c5d80 0x7f230815d9f0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f22f400c400 tx=0x7f22f400c8c0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.332+0000 7f230d800640 1 -- 172.21.15.45:0/1933193142 shutdown_connections 2024-04-16T16:22:36.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.332+0000 7f230d800640 1 --2- 172.21.15.45:0/1933193142 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f22e806ca10 0x7f22e806eed0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.332+0000 7f230d800640 1 --2- 172.21.15.45:0/1933193142 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2308144ff0 0x7f230815b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.332+0000 7f230d800640 1 --2- 172.21.15.45:0/1933193142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23080c5d80 0x7f230815d9f0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.332+0000 7f230d800640 1 --2- 172.21.15.45:0/1933193142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f230806dfb0 0x7f2308132780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.332+0000 7f230d800640 1 -- 172.21.15.45:0/1933193142 >> 172.21.15.45:0/1933193142 conn(0x7f230806b930 msgr2=0x7f230806f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:36.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.332+0000 7f230d800640 1 -- 172.21.15.45:0/1933193142 shutdown_connections 2024-04-16T16:22:36.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.332+0000 7f230d800640 1 -- 172.21.15.45:0/1933193142 wait complete. 2024-04-16T16:22:36.352 INFO:tasks.cephfs_test_runner:test_metrics_mds_grow (tasks.cephfs.test_mds_metrics.TestMDSMetrics) ... ok 2024-04-16T16:22:36.353 INFO:tasks.cephfs_test_runner:Starting test: test_metrics_mds_grow_and_shrink (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2024-04-16T16:22:36.353 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:22:36.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.577+0000 7f7933200640 1 Processor -- start 2024-04-16T16:22:36.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.577+0000 7f7933200640 1 -- start start 2024-04-16T16:22:36.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.578+0000 7f7933200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f792c0d4560 0x7f792c0d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:36.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.578+0000 7f7933200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f792c0d18f0 0x7f792c0b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.578+0000 7f7933200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f792c0d34c0 con 0x7f792c15ec60 2024-04-16T16:22:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.578+0000 7f7933200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f792c0d1d60 con 0x7f792c0d4560 2024-04-16T16:22:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.578+0000 7f7933200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f792c0d37a0 con 0x7f792c0d18f0 2024-04-16T16:22:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.579+0000 7f792be00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f792c15ec60 0x7f792c0d2a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53090/0 (socket says 172.21.15.45:53090) 2024-04-16T16:22:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.579+0000 7f792be00640 1 -- 172.21.15.45:0/3445233248 learned_addr learned my addr 172.21.15.45:0/3445233248 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.579+0000 7f792b400640 1 --2- 172.21.15.45:0/3445233248 >> v2:172.21.15.103:3300/0 conn(0x7f792c0d4560 0x7f792c0d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.579+0000 7f7931200640 1 --2- 172.21.15.45:0/3445233248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f792c0d18f0 0x7f792c0b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.579+0000 7f792aa00640 1 -- 172.21.15.45:0/3445233248 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3111333368 0 0) 0x7f792c0d34c0 con 0x7f792c15ec60 2024-04-16T16:22:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.579+0000 7f792aa00640 1 -- 172.21.15.45:0/3445233248 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7910003520 con 0x7f792c15ec60 2024-04-16T16:22:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.580+0000 7f792aa00640 1 -- 172.21.15.45:0/3445233248 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3324310221 0 0) 0x7f7910003520 con 0x7f792c15ec60 2024-04-16T16:22:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.580+0000 7f792aa00640 1 -- 172.21.15.45:0/3445233248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f792c0d18f0 msgr2=0x7f792c0b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.580+0000 7f792aa00640 1 --2- 172.21.15.45:0/3445233248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f792c0d18f0 0x7f792c0b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.580+0000 7f792aa00640 1 -- 172.21.15.45:0/3445233248 >> v2:172.21.15.103:3300/0 conn(0x7f792c0d4560 msgr2=0x7f792c0d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.580+0000 7f792aa00640 1 --2- 172.21.15.45:0/3445233248 >> v2:172.21.15.103:3300/0 conn(0x7f792c0d4560 0x7f792c0d13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.580+0000 7f792aa00640 1 -- 172.21.15.45:0/3445233248 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792c0d3a80 con 0x7f792c15ec60 2024-04-16T16:22:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.580+0000 7f792aa00640 1 -- 172.21.15.45:0/3445233248 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f791c001670 con 0x7f792c15ec60 2024-04-16T16:22:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.580+0000 7f792aa00640 1 -- 172.21.15.45:0/3445233248 >> v1:172.21.15.45:6789/0 conn(0x7f792c15ec60 legacy=0x7f792c0d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.580+0000 7f792aa00640 1 --2- 172.21.15.45:0/3445233248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79100039b0 0x7f7910003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.580+0000 7f792aa00640 1 --2- 172.21.15.45:0/3445233248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7910004800 0x7f7910004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.580+0000 7f792aa00640 1 --2- 172.21.15.45:0/3445233248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f792c0d18f0 0x7f79100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.580+0000 7f792aa00640 1 -- 172.21.15.45:0/3445233248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f792c0d1d60 con 0x7f79100039b0 2024-04-16T16:22:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.580+0000 7f792aa00640 1 -- 172.21.15.45:0/3445233248 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f792c0d37a0 con 0x7f7910004800 2024-04-16T16:22:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.580+0000 7f792aa00640 1 -- 172.21.15.45:0/3445233248 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f791000b9c0 con 0x7f792c0d18f0 2024-04-16T16:22:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.580+0000 7f792be00640 1 --2- 172.21.15.45:0/3445233248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79100039b0 0x7f7910003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.581+0000 7f792b400640 1 --2- 172.21.15.45:0/3445233248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7910004800 0x7f7910004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.581+0000 7f792be00640 1 -- 172.21.15.45:0/3445233248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f792c0d18f0 msgr2=0x7f79100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.581+0000 7f792be00640 1 --2- 172.21.15.45:0/3445233248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f792c0d18f0 0x7f79100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.581+0000 7f792be00640 1 -- 172.21.15.45:0/3445233248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7910004800 msgr2=0x7f7910004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.581+0000 7f792be00640 1 --2- 172.21.15.45:0/3445233248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7910004800 0x7f7910004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.581+0000 7f792be00640 1 -- 172.21.15.45:0/3445233248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7910003520 con 0x7f79100039b0 2024-04-16T16:22:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.581+0000 7f792b400640 1 --2- 172.21.15.45:0/3445233248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7910004800 0x7f7910004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:36.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.582+0000 7f792be00640 1 --2- 172.21.15.45:0/3445233248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79100039b0 0x7f7910003dd0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f791c00c240 tx=0x7f7910005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8a1da10b1a0d71 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:36.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.582+0000 7f792aa00640 1 -- 172.21.15.45:0/3445233248 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791c007d30 con 0x7f79100039b0 2024-04-16T16:22:36.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.582+0000 7f792aa00640 1 -- 172.21.15.45:0/3445233248 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f791c030070 con 0x7f79100039b0 2024-04-16T16:22:36.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.582+0000 7f792aa00640 1 -- 172.21.15.45:0/3445233248 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791c02cd60 con 0x7f79100039b0 2024-04-16T16:22:36.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.583+0000 7f7933200640 1 -- 172.21.15.45:0/3445233248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79100039b0 msgr2=0x7f7910003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:36.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.583+0000 7f7933200640 1 --2- 172.21.15.45:0/3445233248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79100039b0 0x7f7910003dd0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f791c00c240 tx=0x7f7910005830 comp rx=0 tx=0).stop 2024-04-16T16:22:36.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.583+0000 7f7931200640 1 -- 172.21.15.45:0/3445233248 reap_dead start 2024-04-16T16:22:36.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.583+0000 7f7933200640 1 -- 172.21.15.45:0/3445233248 shutdown_connections 2024-04-16T16:22:36.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.583+0000 7f7933200640 1 -- 172.21.15.45:0/3445233248 >> 172.21.15.45:0/3445233248 conn(0x7f792c073690 msgr2=0x7f792c06cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:36.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.583+0000 7f7933200640 1 -- 172.21.15.45:0/3445233248 shutdown_connections 2024-04-16T16:22:36.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.583+0000 7f7933200640 1 -- 172.21.15.45:0/3445233248 wait complete. 2024-04-16T16:22:36.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.585+0000 7f7933200640 1 Processor -- start 2024-04-16T16:22:36.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.585+0000 7f7933200640 1 -- start start 2024-04-16T16:22:36.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.586+0000 7f7933200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79100039b0 0x7f792c1e7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:36.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.586+0000 7f7933200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f792c1e4180 0x7f792c1e5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:36.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.586+0000 7f7933200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f792c1e61e0 0x7f792c15a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:36.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.586+0000 7f7933200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f792c0ba460 con 0x7f79100039b0 2024-04-16T16:22:36.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.586+0000 7f7933200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f792c0d4040 con 0x7f792c1e4180 2024-04-16T16:22:36.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.586+0000 7f7933200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f792c0d3110 con 0x7f792c1e61e0 2024-04-16T16:22:36.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.586+0000 7f792be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79100039b0 0x7f792c1e7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:36.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.586+0000 7f792be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79100039b0 0x7f792c1e7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58202/0 (socket says 172.21.15.45:58202) 2024-04-16T16:22:36.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.586+0000 7f792be00640 1 -- 172.21.15.45:0/3113513076 learned_addr learned my addr 172.21.15.45:0/3113513076 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.587+0000 7f792b400640 1 --2- 172.21.15.45:0/3113513076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f792c1e4180 0x7f792c1e5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.587+0000 7f7931200640 1 --2- 172.21.15.45:0/3113513076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f792c1e61e0 0x7f792c15a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.587+0000 7f792be00640 1 -- 172.21.15.45:0/3113513076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f792c1e61e0 msgr2=0x7f792c15a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.587+0000 7f792be00640 1 --2- 172.21.15.45:0/3113513076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f792c1e61e0 0x7f792c15a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.587+0000 7f792be00640 1 -- 172.21.15.45:0/3113513076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f792c1e4180 msgr2=0x7f792c1e5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.587+0000 7f792be00640 1 --2- 172.21.15.45:0/3113513076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f792c1e4180 0x7f792c1e5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.587+0000 7f792be00640 1 -- 172.21.15.45:0/3113513076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792c1e6820 con 0x7f79100039b0 2024-04-16T16:22:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.587+0000 7f792b400640 1 --2- 172.21.15.45:0/3113513076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f792c1e4180 0x7f792c1e5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.587+0000 7f792be00640 1 --2- 172.21.15.45:0/3113513076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79100039b0 0x7f792c1e7880 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f791c000f60 tx=0x7f792c0d3c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.587+0000 7f7923e00640 1 -- 172.21.15.45:0/3113513076 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791c001c70 con 0x7f79100039b0 2024-04-16T16:22:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.587+0000 7f7923e00640 1 -- 172.21.15.45:0/3113513076 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f791c004bf0 con 0x7f79100039b0 2024-04-16T16:22:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.587+0000 7f7923e00640 1 -- 172.21.15.45:0/3113513076 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791c007070 con 0x7f79100039b0 2024-04-16T16:22:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.587+0000 7f7933200640 1 -- 172.21.15.45:0/3113513076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f792c1e6b00 con 0x7f79100039b0 2024-04-16T16:22:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.588+0000 7f7933200640 1 -- 172.21.15.45:0/3113513076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f792c1e7dc0 con 0x7f79100039b0 2024-04-16T16:22:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.589+0000 7f7933200640 1 -- 172.21.15.45:0/3113513076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f792c075780 con 0x7f79100039b0 2024-04-16T16:22:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.590+0000 7f7923e00640 1 -- 172.21.15.45:0/3113513076 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f791c007230 con 0x7f79100039b0 2024-04-16T16:22:36.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.591+0000 7f7923e00640 1 --2- 172.21.15.45:0/3113513076 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f790c06ca40 0x7f790c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:36.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.591+0000 7f7923e00640 1 -- 172.21.15.45:0/3113513076 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5717+0+0 (secure 0 0 0) 0x7f791c0ace70 con 0x7f79100039b0 2024-04-16T16:22:36.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.592+0000 7f792b400640 1 --2- 172.21.15.45:0/3113513076 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f790c06ca40 0x7f790c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:36.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.593+0000 7f792b400640 1 --2- 172.21.15.45:0/3113513076 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f790c06ca40 0x7f790c06ef00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f792c1e4580 tx=0x7f791800b040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:36.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.596+0000 7f7923e00640 1 -- 172.21.15.45:0/3113513076 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f791c07b0e0 con 0x7f79100039b0 2024-04-16T16:22:36.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.800+0000 7f7933200640 1 -- 172.21.15.45:0/3113513076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f792c0ceda0 con 0x7f79100039b0 2024-04-16T16:22:36.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.802+0000 7f7923e00640 1 -- 172.21.15.45:0/3113513076 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) ==== 74+0+14182 (secure 0 0 0) 0x7f791c07a9e0 con 0x7f79100039b0 2024-04-16T16:22:36.801 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:36.802 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":147,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:22:35.301308+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6835/2561260581":"2024-04-17T16:22:32.231207+0000","172.21.15.103:6834/2561260581":"2024-04-17T16:22:32.231207+0000","172.21.15.103:6837/3252460318":"2024-04-17T16:22:32.231237+0000","172.21.15.103:6836/3252460318":"2024-04-17T16:22:32.231237+0000","172.21.15.103:0/741531900":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/3249203677":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/1497652063":"2024-04-17T16:22:25.503233+0000","172.21.15.103:6833/1506110537":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/3222629764":"2024-04-17T16:22:25.503233+0000","172.21.15.103:6832/1506110537":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/1298902321":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/884289453":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/3228977328":"2024-04-17T16:22:25.503233+0000","172.21.15.103:0/1437683872":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/2877575799":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/2644878831":"2024-04-17T16:21:47.933003+0000","172.21.15.103:6833/1478535274":"2024-04-17T16:21:47.933003+0000","172.21.15.103:6832/1478535274":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/1111841625":"2024-04-17T16:21:47.933003+0000","172.21.15.103:0/1114329424":"2024-04-17T16:21:47.933003+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:22:36.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.805+0000 7f7933200640 1 -- 172.21.15.45:0/3113513076 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f790c06ca40 msgr2=0x7f790c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:36.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.805+0000 7f7933200640 1 --2- 172.21.15.45:0/3113513076 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f790c06ca40 0x7f790c06ef00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f792c1e4580 tx=0x7f791800b040 comp rx=0 tx=0).stop 2024-04-16T16:22:36.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.805+0000 7f7933200640 1 -- 172.21.15.45:0/3113513076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79100039b0 msgr2=0x7f792c1e7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:36.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.805+0000 7f7933200640 1 --2- 172.21.15.45:0/3113513076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79100039b0 0x7f792c1e7880 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f791c000f60 tx=0x7f792c0d3c50 comp rx=0 tx=0).stop 2024-04-16T16:22:36.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.805+0000 7f7933200640 1 -- 172.21.15.45:0/3113513076 shutdown_connections 2024-04-16T16:22:36.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.805+0000 7f7933200640 1 --2- 172.21.15.45:0/3113513076 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f790c06ca40 0x7f790c06ef00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.805+0000 7f7933200640 1 --2- 172.21.15.45:0/3113513076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f792c1e61e0 0x7f792c15a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.805+0000 7f7933200640 1 --2- 172.21.15.45:0/3113513076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f792c1e4180 0x7f792c1e5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.805+0000 7f7933200640 1 --2- 172.21.15.45:0/3113513076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79100039b0 0x7f792c1e7880 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:36.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.805+0000 7f7933200640 1 -- 172.21.15.45:0/3113513076 >> 172.21.15.45:0/3113513076 conn(0x7f792c073690 msgr2=0x7f792c0c8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:36.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.805+0000 7f7933200640 1 -- 172.21.15.45:0/3113513076 shutdown_connections 2024-04-16T16:22:36.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:36.806+0000 7f7933200640 1 -- 172.21.15.45:0/3113513076 wait complete. 2024-04-16T16:22:36.825 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:22:36.825 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:22:37.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.043+0000 7f4541800640 1 Processor -- start 2024-04-16T16:22:37.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.043+0000 7f4541800640 1 -- start start 2024-04-16T16:22:37.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.044+0000 7f4541800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f453c15ecc0 0x7f453c1626a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:37.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.044+0000 7f4541800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f453c162be0 0x7f453c06cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:37.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.045+0000 7f4541800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f453c06d450 0x7f453c0b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:37.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.045+0000 7f4541800640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f453c06d8c0 con 0x7f453c06d450 2024-04-16T16:22:37.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.045+0000 7f4541800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f453c0d2b10 con 0x7f453c15ecc0 2024-04-16T16:22:37.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.045+0000 7f4541800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f453c163210 con 0x7f453c162be0 2024-04-16T16:22:37.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.045+0000 7f453b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f453c06d450 0x7f453c0b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:37.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.045+0000 7f453b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f453c06d450 0x7f453c0b64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58204/0 (socket says 172.21.15.45:58204) 2024-04-16T16:22:37.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.045+0000 7f453b400640 1 -- 172.21.15.45:0/1599276399 learned_addr learned my addr 172.21.15.45:0/1599276399 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:37.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.045+0000 7f453aa00640 1 --2- 172.21.15.45:0/1599276399 >> v2:172.21.15.103:3300/0 conn(0x7f453c15ecc0 0x7f453c1626a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:37.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.045+0000 7f453a000640 1 --2- 172.21.15.45:0/1599276399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f453c162be0 0x7f453c06cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:37.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.045+0000 7f453b400640 1 -- 172.21.15.45:0/1599276399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f453c162be0 msgr2=0x7f453c06cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:37.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.045+0000 7f453b400640 1 --2- 172.21.15.45:0/1599276399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f453c162be0 0x7f453c06cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.045+0000 7f453b400640 1 -- 172.21.15.45:0/1599276399 >> v2:172.21.15.103:3300/0 conn(0x7f453c15ecc0 msgr2=0x7f453c1626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:37.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.045+0000 7f453b400640 1 --2- 172.21.15.45:0/1599276399 >> v2:172.21.15.103:3300/0 conn(0x7f453c15ecc0 0x7f453c1626a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.045+0000 7f453b400640 1 -- 172.21.15.45:0/1599276399 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f453c1634f0 con 0x7f453c06d450 2024-04-16T16:22:37.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.045+0000 7f453aa00640 1 --2- 172.21.15.45:0/1599276399 >> v2:172.21.15.103:3300/0 conn(0x7f453c15ecc0 0x7f453c1626a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:37.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.047+0000 7f453b400640 1 --2- 172.21.15.45:0/1599276399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f453c06d450 0x7f453c0b64c0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f453c06de90 tx=0x7f453402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80923a287e09d8a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:37.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.047+0000 7f4539600640 1 -- 172.21.15.45:0/1599276399 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4534034070 con 0x7f453c06d450 2024-04-16T16:22:37.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.047+0000 7f4539600640 1 -- 172.21.15.45:0/1599276399 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4534030850 con 0x7f453c06d450 2024-04-16T16:22:37.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.047+0000 7f4539600640 1 -- 172.21.15.45:0/1599276399 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4534030c20 con 0x7f453c06d450 2024-04-16T16:22:37.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.048+0000 7f4541800640 1 -- 172.21.15.45:0/1599276399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f453c06d450 msgr2=0x7f453c0b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:37.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.048+0000 7f4541800640 1 --2- 172.21.15.45:0/1599276399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f453c06d450 0x7f453c0b64c0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f453c06de90 tx=0x7f453402b990 comp rx=0 tx=0).stop 2024-04-16T16:22:37.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.048+0000 7f4541800640 1 -- 172.21.15.45:0/1599276399 shutdown_connections 2024-04-16T16:22:37.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.048+0000 7f4541800640 1 --2- 172.21.15.45:0/1599276399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f453c06d450 0x7f453c0b64c0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.048+0000 7f4541800640 1 --2- 172.21.15.45:0/1599276399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f453c162be0 0x7f453c06cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.048+0000 7f4541800640 1 --2- 172.21.15.45:0/1599276399 >> v2:172.21.15.103:3300/0 conn(0x7f453c15ecc0 0x7f453c1626a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.048+0000 7f4541800640 1 -- 172.21.15.45:0/1599276399 >> 172.21.15.45:0/1599276399 conn(0x7f453c073690 msgr2=0x7f453c0d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:37.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.048+0000 7f4541800640 1 -- 172.21.15.45:0/1599276399 shutdown_connections 2024-04-16T16:22:37.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.048+0000 7f4541800640 1 -- 172.21.15.45:0/1599276399 wait complete. 2024-04-16T16:22:37.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.050+0000 7f4541800640 1 Processor -- start 2024-04-16T16:22:37.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.050+0000 7f4541800640 1 -- start start 2024-04-16T16:22:37.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f4541800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f453c06d450 0x7f453c150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:37.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f4541800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f453c15ecc0 0x7f453c0b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:37.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f4541800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f453c162be0 0x7f453c0b4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:37.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f4541800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f453c0ba540 con 0x7f453c162be0 2024-04-16T16:22:37.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f4541800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f453c0d30b0 con 0x7f453c15ecc0 2024-04-16T16:22:37.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f4541800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f453c0b6a00 con 0x7f453c06d450 2024-04-16T16:22:37.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f453b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f453c162be0 0x7f453c0b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:37.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f453b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f453c162be0 0x7f453c0b4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58206/0 (socket says 172.21.15.45:58206) 2024-04-16T16:22:37.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f453b400640 1 -- 172.21.15.45:0/931254079 learned_addr learned my addr 172.21.15.45:0/931254079 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:37.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f453aa00640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f453c06d450 0x7f453c150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:37.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f453b400640 1 -- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f453c06d450 msgr2=0x7f453c150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:37.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f453a000640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f453c15ecc0 0x7f453c0b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:37.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f453b400640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f453c06d450 0x7f453c150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f453b400640 1 -- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f453c15ecc0 msgr2=0x7f453c0b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:37.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f453b400640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f453c15ecc0 0x7f453c0b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.051+0000 7f453b400640 1 -- 172.21.15.45:0/931254079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f453c0c8580 con 0x7f453c162be0 2024-04-16T16:22:37.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.052+0000 7f453aa00640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f453c06d450 0x7f453c150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:37.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.052+0000 7f453a000640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f453c15ecc0 0x7f453c0b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:37.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.052+0000 7f453b400640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f453c162be0 0x7f453c0b4120 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f453402b9d0 tx=0x7f4534004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:37.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.052+0000 7f4532a00640 1 -- 172.21.15.45:0/931254079 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4534048070 con 0x7f453c162be0 2024-04-16T16:22:37.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.052+0000 7f4532a00640 1 -- 172.21.15.45:0/931254079 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4534039ce0 con 0x7f453c162be0 2024-04-16T16:22:37.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.052+0000 7f4532a00640 1 -- 172.21.15.45:0/931254079 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4534034040 con 0x7f453c162be0 2024-04-16T16:22:37.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.052+0000 7f4541800640 1 -- 172.21.15.45:0/931254079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f453c0c8860 con 0x7f453c162be0 2024-04-16T16:22:37.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.052+0000 7f4541800640 1 -- 172.21.15.45:0/931254079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f453c0c8e20 con 0x7f453c162be0 2024-04-16T16:22:37.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.054+0000 7f4541800640 1 -- 172.21.15.45:0/931254079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f453c075780 con 0x7f453c162be0 2024-04-16T16:22:37.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.055+0000 7f4532a00640 1 -- 172.21.15.45:0/931254079 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f453404b070 con 0x7f453c162be0 2024-04-16T16:22:37.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.055+0000 7f4532a00640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f451c06ca10 0x7f451c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:37.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.056+0000 7f4532a00640 1 -- 172.21.15.45:0/931254079 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5717+0+0 (secure 0 0 0) 0x7f45340ae070 con 0x7f453c162be0 2024-04-16T16:22:37.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.056+0000 7f453aa00640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f451c06ca10 0x7f451c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:37.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.057+0000 7f453aa00640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f451c06ca10 0x7f451c06eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f452c008fc0 tx=0x7f452c008510 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:37.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.060+0000 7f4532a00640 1 -- 172.21.15.45:0/931254079 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f453407c260 con 0x7f453c162be0 2024-04-16T16:22:37.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.262+0000 7f4541800640 1 -- 172.21.15.45:0/931254079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f453c0b7850 con 0x7f453c162be0 2024-04-16T16:22:37.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.263+0000 7f4532a00640 1 -- 172.21.15.45:0/931254079 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v147) ==== 93+0+10 (secure 0 0 0) 0x7f453407bb60 con 0x7f453c162be0 2024-04-16T16:22:37.262 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:22:37.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.265+0000 7f4541800640 1 -- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f451c06ca10 msgr2=0x7f451c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:37.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.265+0000 7f4541800640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f451c06ca10 0x7f451c06eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f452c008fc0 tx=0x7f452c008510 comp rx=0 tx=0).stop 2024-04-16T16:22:37.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.265+0000 7f4541800640 1 -- 172.21.15.45:0/931254079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f453c162be0 msgr2=0x7f453c0b4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:37.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.265+0000 7f4541800640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f453c162be0 0x7f453c0b4120 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f453402b9d0 tx=0x7f4534004730 comp rx=0 tx=0).stop 2024-04-16T16:22:37.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.266+0000 7f4541800640 1 -- 172.21.15.45:0/931254079 shutdown_connections 2024-04-16T16:22:37.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.266+0000 7f4541800640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f451c06ca10 0x7f451c06eed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.266+0000 7f4541800640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f453c162be0 0x7f453c0b4120 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.266+0000 7f4541800640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f453c15ecc0 0x7f453c0b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.266+0000 7f4541800640 1 --2- 172.21.15.45:0/931254079 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f453c06d450 0x7f453c150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.266+0000 7f4541800640 1 -- 172.21.15.45:0/931254079 >> 172.21.15.45:0/931254079 conn(0x7f453c073690 msgr2=0x7f453c0b8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:37.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.266+0000 7f4541800640 1 -- 172.21.15.45:0/931254079 shutdown_connections 2024-04-16T16:22:37.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.266+0000 7f4541800640 1 -- 172.21.15.45:0/931254079 wait complete. 2024-04-16T16:22:37.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_metrics_mds_grow_and_shrink' 2024-04-16T16:22:37.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.516+0000 7fac86a00640 1 Processor -- start 2024-04-16T16:22:37.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.516+0000 7fac86a00640 1 -- start start 2024-04-16T16:22:37.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.517+0000 7fac86a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fac80144ff0 0x7fac800cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:37.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.518+0000 7fac86a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac8015c700 con 0x7fac800c5d80 2024-04-16T16:22:37.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.518+0000 7fac86a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac8015c9c0 con 0x7fac8006dfb0 2024-04-16T16:22:37.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.518+0000 7fac86a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fac8015d2f0 con 0x7fac80144ff0 2024-04-16T16:22:37.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.518+0000 7fac86000640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fac800c5d80 0x7fac8015d780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53094/0 (socket says 172.21.15.45:53094) 2024-04-16T16:22:37.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.518+0000 7fac86000640 1 -- 172.21.15.45:0/1673321347 learned_addr learned my addr 172.21.15.45:0/1673321347 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.518+0000 7fac85600640 1 --2- 172.21.15.45:0/1673321347 >> v2:172.21.15.103:3300/0 conn(0x7fac80144ff0 0x7fac800cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.519+0000 7fac7fe00640 1 -- 172.21.15.45:0/1673321347 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1039171779 0 0) 0x7fac8015c700 con 0x7fac800c5d80 2024-04-16T16:22:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.519+0000 7fac7fe00640 1 -- 172.21.15.45:0/1673321347 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac68005730 con 0x7fac800c5d80 2024-04-16T16:22:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.519+0000 7fac7fe00640 1 -- 172.21.15.45:0/1673321347 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1794191744 0 0) 0x7fac68005730 con 0x7fac800c5d80 2024-04-16T16:22:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.519+0000 7fac7fe00640 1 -- 172.21.15.45:0/1673321347 >> v2:172.21.15.103:3300/0 conn(0x7fac80144ff0 msgr2=0x7fac800cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.519+0000 7fac7fe00640 1 --2- 172.21.15.45:0/1673321347 >> v2:172.21.15.103:3300/0 conn(0x7fac80144ff0 0x7fac800cfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.519+0000 7fac7fe00640 1 -- 172.21.15.45:0/1673321347 >> v1:172.21.15.103:6789/0 conn(0x7fac8006dfb0 legacy=0x7fac800c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.519+0000 7fac7fe00640 1 -- 172.21.15.45:0/1673321347 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac8015cca0 con 0x7fac800c5d80 2024-04-16T16:22:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.519+0000 7fac7fe00640 1 -- 172.21.15.45:0/1673321347 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fac780019f0 con 0x7fac800c5d80 2024-04-16T16:22:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.519+0000 7fac7fe00640 1 -- 172.21.15.45:0/1673321347 >> v1:172.21.15.45:6789/0 conn(0x7fac800c5d80 legacy=0x7fac8015d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.519+0000 7fac7fe00640 1 --2- 172.21.15.45:0/1673321347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac68005bc0 0x7fac68006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.519+0000 7fac7fe00640 1 --2- 172.21.15.45:0/1673321347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac68006a10 0x7fac68006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.519+0000 7fac7fe00640 1 --2- 172.21.15.45:0/1673321347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac680073d0 0x7fac6800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.519+0000 7fac7fe00640 1 -- 172.21.15.45:0/1673321347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fac8015d2f0 con 0x7fac68006a10 2024-04-16T16:22:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.519+0000 7fac7fe00640 1 -- 172.21.15.45:0/1673321347 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fac6800e2a0 con 0x7fac68005bc0 2024-04-16T16:22:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.519+0000 7fac7fe00640 1 -- 172.21.15.45:0/1673321347 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fac6800e4a0 con 0x7fac680073d0 2024-04-16T16:22:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.520+0000 7fac84c00640 1 --2- 172.21.15.45:0/1673321347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac68006a10 0x7fac68006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.520+0000 7fac84c00640 1 -- 172.21.15.45:0/1673321347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac680073d0 msgr2=0x7fac6800dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.520+0000 7fac86000640 1 --2- 172.21.15.45:0/1673321347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac680073d0 0x7fac6800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.520+0000 7fac85600640 1 --2- 172.21.15.45:0/1673321347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac68005bc0 0x7fac68006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.520+0000 7fac84c00640 1 --2- 172.21.15.45:0/1673321347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac680073d0 0x7fac6800dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.520+0000 7fac84c00640 1 -- 172.21.15.45:0/1673321347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac68005bc0 msgr2=0x7fac68006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.520+0000 7fac84c00640 1 --2- 172.21.15.45:0/1673321347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac68005bc0 0x7fac68006000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.520+0000 7fac84c00640 1 -- 172.21.15.45:0/1673321347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac68005730 con 0x7fac68006a10 2024-04-16T16:22:37.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.520+0000 7fac86000640 1 --2- 172.21.15.45:0/1673321347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac680073d0 0x7fac6800dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:37.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.520+0000 7fac86000640 1 -- 172.21.15.45:0/1673321347 reap_dead start 2024-04-16T16:22:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.521+0000 7fac84c00640 1 --2- 172.21.15.45:0/1673321347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac68006a10 0x7fac68006e90 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fac6c001720 tx=0x7fac68007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad054f243b0b0b12 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.522+0000 7fac7fe00640 1 -- 172.21.15.45:0/1673321347 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6c001370 con 0x7fac68006a10 2024-04-16T16:22:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.522+0000 7fac7fe00640 1 -- 172.21.15.45:0/1673321347 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac6c0052d0 con 0x7fac68006a10 2024-04-16T16:22:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.522+0000 7fac7fe00640 1 -- 172.21.15.45:0/1673321347 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6c003830 con 0x7fac68006a10 2024-04-16T16:22:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.522+0000 7fac86a00640 1 -- 172.21.15.45:0/1673321347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac68006a10 msgr2=0x7fac68006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.522+0000 7fac86a00640 1 --2- 172.21.15.45:0/1673321347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac68006a10 0x7fac68006e90 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fac6c001720 tx=0x7fac68007a40 comp rx=0 tx=0).stop 2024-04-16T16:22:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.522+0000 7fac86a00640 1 -- 172.21.15.45:0/1673321347 shutdown_connections 2024-04-16T16:22:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.522+0000 7fac86a00640 1 --2- 172.21.15.45:0/1673321347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac68006a10 0x7fac68006e90 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.522+0000 7fac86a00640 1 -- 172.21.15.45:0/1673321347 >> 172.21.15.45:0/1673321347 conn(0x7fac8006b930 msgr2=0x7fac8006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.522+0000 7fac86a00640 1 -- 172.21.15.45:0/1673321347 shutdown_connections 2024-04-16T16:22:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.522+0000 7fac86a00640 1 -- 172.21.15.45:0/1673321347 wait complete. 2024-04-16T16:22:37.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.524+0000 7fac86a00640 1 Processor -- start 2024-04-16T16:22:37.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.524+0000 7fac86a00640 1 -- start start 2024-04-16T16:22:37.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.525+0000 7fac86a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac68006a10 0x7fac801e86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:37.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.525+0000 7fac86a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac801e3f10 0x7fac801e4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:37.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.525+0000 7fac86a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac800cd9b0 0x7fac800cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:37.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.525+0000 7fac86a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fac800c5670 con 0x7fac68006a10 2024-04-16T16:22:37.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.525+0000 7fac86a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fac801453d0 con 0x7fac800cd9b0 2024-04-16T16:22:37.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.525+0000 7fac86a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fac800c57f0 con 0x7fac801e3f10 2024-04-16T16:22:37.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.526+0000 7fac85600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac68006a10 0x7fac801e86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:37.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.526+0000 7fac85600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac68006a10 0x7fac801e86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58236/0 (socket says 172.21.15.45:58236) 2024-04-16T16:22:37.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.526+0000 7fac85600640 1 -- 172.21.15.45:0/1179171803 learned_addr learned my addr 172.21.15.45:0/1179171803 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:37.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.526+0000 7fac84c00640 1 --2- 172.21.15.45:0/1179171803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac801e3f10 0x7fac801e4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:37.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.526+0000 7fac86000640 1 --2- 172.21.15.45:0/1179171803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac800cd9b0 0x7fac800cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:37.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.526+0000 7fac85600640 1 -- 172.21.15.45:0/1179171803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac801e3f10 msgr2=0x7fac801e4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:37.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.526+0000 7fac85600640 1 --2- 172.21.15.45:0/1179171803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac801e3f10 0x7fac801e4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.526+0000 7fac85600640 1 -- 172.21.15.45:0/1179171803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac800cd9b0 msgr2=0x7fac800cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:37.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.526+0000 7fac85600640 1 --2- 172.21.15.45:0/1179171803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac800cd9b0 0x7fac800cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:37.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.526+0000 7fac85600640 1 -- 172.21.15.45:0/1179171803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac80140600 con 0x7fac68006a10 2024-04-16T16:22:37.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.526+0000 7fac85600640 1 --2- 172.21.15.45:0/1179171803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac68006a10 0x7fac801e86b0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fac740108e0 tx=0x7fac74010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:37.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.527+0000 7fac7d600640 1 -- 172.21.15.45:0/1179171803 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7400f040 con 0x7fac68006a10 2024-04-16T16:22:37.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.527+0000 7fac7d600640 1 -- 172.21.15.45:0/1179171803 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac7400eb60 con 0x7fac68006a10 2024-04-16T16:22:37.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.527+0000 7fac7d600640 1 -- 172.21.15.45:0/1179171803 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7401c400 con 0x7fac68006a10 2024-04-16T16:22:37.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.527+0000 7fac86a00640 1 -- 172.21.15.45:0/1179171803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac801e4a80 con 0x7fac68006a10 2024-04-16T16:22:37.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.527+0000 7fac86a00640 1 -- 172.21.15.45:0/1179171803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac801e5a90 con 0x7fac68006a10 2024-04-16T16:22:37.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.528+0000 7fac86a00640 1 -- 172.21.15.45:0/1179171803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac80075780 con 0x7fac68006a10 2024-04-16T16:22:37.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.529+0000 7fac7d600640 1 -- 172.21.15.45:0/1179171803 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fac7401c5a0 con 0x7fac68006a10 2024-04-16T16:22:37.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.530+0000 7fac7d600640 1 --2- 172.21.15.45:0/1179171803 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fac6006ca10 0x7fac6006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:37.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.530+0000 7fac7d600640 1 -- 172.21.15.45:0/1179171803 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5717+0+0 (secure 0 0 0) 0x7fac7408c420 con 0x7fac68006a10 2024-04-16T16:22:37.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.531+0000 7fac84c00640 1 --2- 172.21.15.45:0/1179171803 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fac6006ca10 0x7fac6006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:37.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.532+0000 7fac84c00640 1 --2- 172.21.15.45:0/1179171803 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fac6006ca10 0x7fac6006eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fac6c001720 tx=0x7fac6c009030 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:37.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.535+0000 7fac7d600640 1 -- 172.21.15.45:0/1179171803 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac74018030 con 0x7fac68006a10 2024-04-16T16:22:37.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:37.815+0000 7fac86a00640 1 -- 172.21.15.45:0/1179171803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_metrics_mds_grow_and_shrink"]} v 0) -- 0x7fac800b59c0 con 0x7fac68006a10 2024-04-16T16:22:38.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.342+0000 7fac7d600640 1 -- 172.21.15.45:0/1179171803 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_mds_metrics.TestMDSMetrics.test_metrics_mds_grow_and_shrink"]}]=0 v519) ==== 159+0+0 (secure 0 0 0) 0x7fac7405a170 con 0x7fac68006a10 2024-04-16T16:22:38.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.347+0000 7fac86a00640 1 -- 172.21.15.45:0/1179171803 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fac6006ca10 msgr2=0x7fac6006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:38.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.347+0000 7fac86a00640 1 --2- 172.21.15.45:0/1179171803 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fac6006ca10 0x7fac6006eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fac6c001720 tx=0x7fac6c009030 comp rx=0 tx=0).stop 2024-04-16T16:22:38.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.347+0000 7fac86a00640 1 -- 172.21.15.45:0/1179171803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac68006a10 msgr2=0x7fac801e86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:38.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.347+0000 7fac86a00640 1 --2- 172.21.15.45:0/1179171803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac68006a10 0x7fac801e86b0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fac740108e0 tx=0x7fac74010da0 comp rx=0 tx=0).stop 2024-04-16T16:22:38.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.347+0000 7fac86a00640 1 -- 172.21.15.45:0/1179171803 shutdown_connections 2024-04-16T16:22:38.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.347+0000 7fac86a00640 1 --2- 172.21.15.45:0/1179171803 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fac6006ca10 0x7fac6006eed0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:38.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.347+0000 7fac86a00640 1 --2- 172.21.15.45:0/1179171803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac800cd9b0 0x7fac800cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:38.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.347+0000 7fac86a00640 1 --2- 172.21.15.45:0/1179171803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac801e3f10 0x7fac801e4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:38.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.347+0000 7fac86a00640 1 --2- 172.21.15.45:0/1179171803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac68006a10 0x7fac801e86b0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:38.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.347+0000 7fac86a00640 1 -- 172.21.15.45:0/1179171803 >> 172.21.15.45:0/1179171803 conn(0x7fac8006b930 msgr2=0x7fac8015bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:38.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.347+0000 7fac86a00640 1 -- 172.21.15.45:0/1179171803 shutdown_connections 2024-04-16T16:22:38.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.347+0000 7fac86a00640 1 -- 172.21.15.45:0/1179171803 wait complete. 2024-04-16T16:22:38.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-16T16:22:38.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.595+0000 7f18aca00640 1 Processor -- start 2024-04-16T16:22:38.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.596+0000 7f18aca00640 1 -- start start 2024-04-16T16:22:38.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.597+0000 7f18aca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18a80d18f0 0x7f18a80ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:38.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.597+0000 7f18aca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f18a80b6310 0x7f18a80d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:38.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.597+0000 7f18aca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18a80b6810 con 0x7f18a80d1cd0 2024-04-16T16:22:38.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.597+0000 7f18aca00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f18a80d29f0 con 0x7f18a80b6310 2024-04-16T16:22:38.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.597+0000 7f18aca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f18a80b6af0 con 0x7f18a80d18f0 2024-04-16T16:22:38.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.598+0000 7f18a7400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18a80d18f0 0x7f18a80ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:38.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.598+0000 7f18a7e00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f18a80d1cd0 0x7f18a80d33c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:41380/0 (socket says 172.21.15.45:41380) 2024-04-16T16:22:38.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.598+0000 7f189ea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f18a80b6310 0x7f18a80d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:38.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.598+0000 7f18a7e00640 1 -- 172.21.15.45:0/1640422245 learned_addr learned my addr 172.21.15.45:0/1640422245 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:38.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.599+0000 7f18a7400640 1 -- 172.21.15.45:0/1640422245 >> v2:172.21.15.103:3300/0 conn(0x7f18a80b6310 msgr2=0x7f18a80d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:38.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.599+0000 7f18a7400640 1 --2- 172.21.15.45:0/1640422245 >> v2:172.21.15.103:3300/0 conn(0x7f18a80b6310 0x7f18a80d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:38.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.599+0000 7f18a7400640 1 -- 172.21.15.45:0/1640422245 >> v1:172.21.15.103:6789/0 conn(0x7f18a80d1cd0 legacy=0x7f18a80d33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:38.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.599+0000 7f18a7400640 1 -- 172.21.15.45:0/1640422245 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a80b6dd0 con 0x7f18a80d18f0 2024-04-16T16:22:38.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.599+0000 7f189ea00640 1 --2- 172.21.15.45:0/1640422245 >> v2:172.21.15.103:3300/0 conn(0x7f18a80b6310 0x7f18a80d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:22:38.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.601+0000 7f18a7400640 1 --2- 172.21.15.45:0/1640422245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18a80d18f0 0x7f18a80ba220 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f18980097a0 tx=0x7f189802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2a6e313ea7458c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:38.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.601+0000 7f18a6a00640 1 -- 172.21.15.45:0/1640422245 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1898031780 con 0x7f18a80d18f0 2024-04-16T16:22:38.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.601+0000 7f18a6a00640 1 -- 172.21.15.45:0/1640422245 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1898037070 con 0x7f18a80d18f0 2024-04-16T16:22:38.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.601+0000 7f18a6a00640 1 -- 172.21.15.45:0/1640422245 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1898031b40 con 0x7f18a80d18f0 2024-04-16T16:22:38.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.602+0000 7f18aca00640 1 -- 172.21.15.45:0/1640422245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18a80d18f0 msgr2=0x7f18a80ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:38.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.602+0000 7f18aca00640 1 --2- 172.21.15.45:0/1640422245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18a80d18f0 0x7f18a80ba220 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f18980097a0 tx=0x7f189802b820 comp rx=0 tx=0).stop 2024-04-16T16:22:38.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.602+0000 7f18aca00640 1 -- 172.21.15.45:0/1640422245 shutdown_connections 2024-04-16T16:22:38.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.602+0000 7f18aca00640 1 --2- 172.21.15.45:0/1640422245 >> v2:172.21.15.103:3300/0 conn(0x7f18a80b6310 0x7f18a80d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:38.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.602+0000 7f18aca00640 1 --2- 172.21.15.45:0/1640422245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18a80d18f0 0x7f18a80ba220 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:38.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.602+0000 7f18aca00640 1 -- 172.21.15.45:0/1640422245 >> 172.21.15.45:0/1640422245 conn(0x7f18a806cf90 msgr2=0x7f18a806d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:38.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.602+0000 7f18aca00640 1 -- 172.21.15.45:0/1640422245 shutdown_connections 2024-04-16T16:22:38.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.602+0000 7f18aca00640 1 -- 172.21.15.45:0/1640422245 wait complete. 2024-04-16T16:22:38.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.604+0000 7f18aca00640 1 Processor -- start 2024-04-16T16:22:38.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.604+0000 7f18aca00640 1 -- start start 2024-04-16T16:22:38.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18aca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18a80b6310 0x7f18a80ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:38.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18aca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18a80d18f0 0x7f18a80c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:38.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18aca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18a80d1cd0 0x7f18a80b8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:38.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18aca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f18a8162dc0 con 0x7f18a80b6310 2024-04-16T16:22:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18aca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18a80ba760 con 0x7f18a80d18f0 2024-04-16T16:22:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18aca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f18a8162f40 con 0x7f18a80d1cd0 2024-04-16T16:22:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18a7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18a80b6310 0x7f18a80ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18a7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18a80b6310 0x7f18a80ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58244/0 (socket says 172.21.15.45:58244) 2024-04-16T16:22:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18a7400640 1 -- 172.21.15.45:0/2575643148 learned_addr learned my addr 172.21.15.45:0/2575643148 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f189ea00640 1 --2- 172.21.15.45:0/2575643148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18a80d18f0 0x7f18a80c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18a7e00640 1 --2- 172.21.15.45:0/2575643148 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18a80d1cd0 0x7f18a80b8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18a7400640 1 -- 172.21.15.45:0/2575643148 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18a80d1cd0 msgr2=0x7f18a80b8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18a7400640 1 --2- 172.21.15.45:0/2575643148 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18a80d1cd0 0x7f18a80b8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18a7400640 1 -- 172.21.15.45:0/2575643148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18a80d18f0 msgr2=0x7f18a80c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18a7400640 1 --2- 172.21.15.45:0/2575643148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18a80d18f0 0x7f18a80c8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18a7400640 1 -- 172.21.15.45:0/2575643148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a81544d0 con 0x7f18a80b6310 2024-04-16T16:22:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.605+0000 7f18a7e00640 1 --2- 172.21.15.45:0/2575643148 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18a80d1cd0 0x7f18a80b8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:38.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.606+0000 7f18a7400640 1 --2- 172.21.15.45:0/2575643148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18a80b6310 0x7f18a80ce300 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f1898034040 tx=0x7f18980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:38.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.606+0000 7f189fe00640 1 -- 172.21.15.45:0/2575643148 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1898032620 con 0x7f18a80b6310 2024-04-16T16:22:38.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.606+0000 7f189fe00640 1 -- 172.21.15.45:0/2575643148 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18980327e0 con 0x7f18a80b6310 2024-04-16T16:22:38.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.606+0000 7f189fe00640 1 -- 172.21.15.45:0/2575643148 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1898032bd0 con 0x7f18a80b6310 2024-04-16T16:22:38.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.606+0000 7f18aca00640 1 -- 172.21.15.45:0/2575643148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a81547b0 con 0x7f18a80b6310 2024-04-16T16:22:38.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.606+0000 7f18aca00640 1 -- 172.21.15.45:0/2575643148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a8154d70 con 0x7f18a80b6310 2024-04-16T16:22:38.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.608+0000 7f18aca00640 1 -- 172.21.15.45:0/2575643148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18a8075780 con 0x7f18a80b6310 2024-04-16T16:22:38.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.609+0000 7f189fe00640 1 -- 172.21.15.45:0/2575643148 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f189804e440 con 0x7f18a80b6310 2024-04-16T16:22:38.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.610+0000 7f189fe00640 1 --2- 172.21.15.45:0/2575643148 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f188406ca10 0x7f188406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:38.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.610+0000 7f189fe00640 1 -- 172.21.15.45:0/2575643148 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5717+0+0 (secure 0 0 0) 0x7f18980addb0 con 0x7f18a80b6310 2024-04-16T16:22:38.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.611+0000 7f189ea00640 1 --2- 172.21.15.45:0/2575643148 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f188406ca10 0x7f188406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:38.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.612+0000 7f189ea00640 1 --2- 172.21.15.45:0/2575643148 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f188406ca10 0x7f188406eed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1890009560 tx=0x7f1890008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:38.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.615+0000 7f189fe00640 1 -- 172.21.15.45:0/2575643148 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f18980b2070 con 0x7f18a80b6310 2024-04-16T16:22:38.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.823+0000 7f18aca00640 1 -- 172.21.15.45:0/2575643148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f18a806d370 con 0x7f18a80b6310 2024-04-16T16:22:38.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.829+0000 7f189fe00640 1 -- 172.21.15.45:0/2575643148 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) ==== 122+0+0 (secure 0 0 0) 0x7f1898040070 con 0x7f18a80b6310 2024-04-16T16:22:38.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.831+0000 7f18aca00640 1 -- 172.21.15.45:0/2575643148 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f188406ca10 msgr2=0x7f188406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:38.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.831+0000 7f18aca00640 1 --2- 172.21.15.45:0/2575643148 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f188406ca10 0x7f188406eed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1890009560 tx=0x7f1890008040 comp rx=0 tx=0).stop 2024-04-16T16:22:38.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.831+0000 7f18aca00640 1 -- 172.21.15.45:0/2575643148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18a80b6310 msgr2=0x7f18a80ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:38.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.831+0000 7f18aca00640 1 --2- 172.21.15.45:0/2575643148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18a80b6310 0x7f18a80ce300 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f1898034040 tx=0x7f18980046c0 comp rx=0 tx=0).stop 2024-04-16T16:22:38.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.832+0000 7f18aca00640 1 -- 172.21.15.45:0/2575643148 shutdown_connections 2024-04-16T16:22:38.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.832+0000 7f18aca00640 1 --2- 172.21.15.45:0/2575643148 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f188406ca10 0x7f188406eed0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:38.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.832+0000 7f18aca00640 1 --2- 172.21.15.45:0/2575643148 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18a80d1cd0 0x7f18a80b8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:38.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.832+0000 7f18aca00640 1 --2- 172.21.15.45:0/2575643148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18a80d18f0 0x7f18a80c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:38.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.832+0000 7f18aca00640 1 --2- 172.21.15.45:0/2575643148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18a80b6310 0x7f18a80ce300 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:38.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.832+0000 7f18aca00640 1 -- 172.21.15.45:0/2575643148 >> 172.21.15.45:0/2575643148 conn(0x7f18a806cf90 msgr2=0x7f18a80b4cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:38.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.832+0000 7f18aca00640 1 -- 172.21.15.45:0/2575643148 shutdown_connections 2024-04-16T16:22:38.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:38.832+0000 7f18aca00640 1 -- 172.21.15.45:0/2575643148 wait complete. 2024-04-16T16:22:38.851 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-16T16:22:38.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:22:38.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-16T16:22:39.098 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-16T16:22:39.099 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:39.099 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:39.161 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:22:39.161 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:22:39.161 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:39.220 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:22:39.221 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:39.221 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:39.282 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:22:39.283 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:22:39.283 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:39.341 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:22:39.341 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-16T16:22:39.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:22:39.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.575+0000 7feac0a00640 1 Processor -- start 2024-04-16T16:22:39.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.575+0000 7feac0a00640 1 -- start start 2024-04-16T16:22:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.577+0000 7feac0a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feabc15ccb0 0x7feabc06df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.577+0000 7feac0a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feabc15d360 con 0x7feabc146220 2024-04-16T16:22:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.577+0000 7feac0a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feabc15d640 con 0x7feabc06e480 2024-04-16T16:22:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.577+0000 7feac0a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7feabc15d130 con 0x7feabc15ccb0 2024-04-16T16:22:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.577+0000 7feabaa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feabc15ccb0 0x7feabc06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.577+0000 7feabaa00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feabc15ccb0 0x7feabc06df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58260/0 (socket says 172.21.15.45:58260) 2024-04-16T16:22:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.577+0000 7feabaa00640 1 -- 172.21.15.45:0/1324171579 learned_addr learned my addr 172.21.15.45:0/1324171579 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.578+0000 7feabaa00640 1 -- 172.21.15.45:0/1324171579 >> v1:172.21.15.103:6789/0 conn(0x7feabc06e480 legacy=0x7feabc1608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.578+0000 7feabaa00640 1 -- 172.21.15.45:0/1324171579 >> v1:172.21.15.45:6789/0 conn(0x7feabc146220 legacy=0x7feabc06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.578+0000 7feabaa00640 1 -- 172.21.15.45:0/1324171579 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feabc15d920 con 0x7feabc15ccb0 2024-04-16T16:22:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.579+0000 7feabaa00640 1 --2- 172.21.15.45:0/1324171579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feabc15ccb0 0x7feabc06df40 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009870 tx=0x7feaa802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9564814a9d38e3d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:39.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.579+0000 7feaba000640 1 -- 172.21.15.45:0/1324171579 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa8034070 con 0x7feabc15ccb0 2024-04-16T16:22:39.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.579+0000 7feaba000640 1 -- 172.21.15.45:0/1324171579 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaa8030860 con 0x7feabc15ccb0 2024-04-16T16:22:39.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.579+0000 7feaba000640 1 -- 172.21.15.45:0/1324171579 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa8030c30 con 0x7feabc15ccb0 2024-04-16T16:22:39.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.580+0000 7feac0a00640 1 -- 172.21.15.45:0/1324171579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feabc15ccb0 msgr2=0x7feabc06df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:39.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.580+0000 7feac0a00640 1 --2- 172.21.15.45:0/1324171579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feabc15ccb0 0x7feabc06df40 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009870 tx=0x7feaa802b9a0 comp rx=0 tx=0).stop 2024-04-16T16:22:39.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.580+0000 7feac0a00640 1 -- 172.21.15.45:0/1324171579 shutdown_connections 2024-04-16T16:22:39.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.580+0000 7feac0a00640 1 --2- 172.21.15.45:0/1324171579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feabc15ccb0 0x7feabc06df40 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:39.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.580+0000 7feac0a00640 1 -- 172.21.15.45:0/1324171579 >> 172.21.15.45:0/1324171579 conn(0x7feabc06b930 msgr2=0x7feabc06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:39.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.580+0000 7feac0a00640 1 -- 172.21.15.45:0/1324171579 shutdown_connections 2024-04-16T16:22:39.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.580+0000 7feac0a00640 1 -- 172.21.15.45:0/1324171579 wait complete. 2024-04-16T16:22:39.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.583+0000 7feac0a00640 1 Processor -- start 2024-04-16T16:22:39.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.583+0000 7feac0a00640 1 -- start start 2024-04-16T16:22:39.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.584+0000 7feac0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feabc06e480 0x7feabc1f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:39.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.584+0000 7feac0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feabc146220 0x7feabc0b1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:39.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.584+0000 7feac0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feabc0b2070 0x7feabc0cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:39.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.584+0000 7feac0a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feabc160fb0 con 0x7feabc06e480 2024-04-16T16:22:39.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.584+0000 7feac0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feabc160410 con 0x7feabc146220 2024-04-16T16:22:39.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.584+0000 7feac0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feabc161130 con 0x7feabc0b2070 2024-04-16T16:22:39.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.584+0000 7feabb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feabc06e480 0x7feabc1f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:39.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.584+0000 7feabb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feabc06e480 0x7feabc1f7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58274/0 (socket says 172.21.15.45:58274) 2024-04-16T16:22:39.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.584+0000 7feabb400640 1 -- 172.21.15.45:0/76767551 learned_addr learned my addr 172.21.15.45:0/76767551 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:39.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.585+0000 7feabaa00640 1 --2- 172.21.15.45:0/76767551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feabc146220 0x7feabc0b1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:39.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.585+0000 7feabbe00640 1 --2- 172.21.15.45:0/76767551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feabc0b2070 0x7feabc0cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:39.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.585+0000 7feabb400640 1 -- 172.21.15.45:0/76767551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feabc0b2070 msgr2=0x7feabc0cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:39.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.585+0000 7feabb400640 1 --2- 172.21.15.45:0/76767551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feabc0b2070 0x7feabc0cd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:39.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.585+0000 7feabb400640 1 -- 172.21.15.45:0/76767551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feabc146220 msgr2=0x7feabc0b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:39.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.585+0000 7feabb400640 1 --2- 172.21.15.45:0/76767551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feabc146220 0x7feabc0b1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:39.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.585+0000 7feabb400640 1 -- 172.21.15.45:0/76767551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feabc0cddd0 con 0x7feabc06e480 2024-04-16T16:22:39.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.585+0000 7feabb400640 1 --2- 172.21.15.45:0/76767551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feabc06e480 0x7feabc1f7a40 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7feaac00c3c0 tx=0x7feaac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:39.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.586+0000 7feab3400640 1 -- 172.21.15.45:0/76767551 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac0052d0 con 0x7feabc06e480 2024-04-16T16:22:39.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.586+0000 7feab3400640 1 -- 172.21.15.45:0/76767551 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaac005490 con 0x7feabc06e480 2024-04-16T16:22:39.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.586+0000 7feac0a00640 1 -- 172.21.15.45:0/76767551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feabc0ce0b0 con 0x7feabc06e480 2024-04-16T16:22:39.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.586+0000 7feab3400640 1 -- 172.21.15.45:0/76767551 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac007260 con 0x7feabc06e480 2024-04-16T16:22:39.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.586+0000 7feac0a00640 1 -- 172.21.15.45:0/76767551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feabc0ce670 con 0x7feabc06e480 2024-04-16T16:22:39.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.587+0000 7feac0a00640 1 -- 172.21.15.45:0/76767551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feabc0b2480 con 0x7feabc06e480 2024-04-16T16:22:39.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.588+0000 7feab3400640 1 -- 172.21.15.45:0/76767551 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7feaac007400 con 0x7feabc06e480 2024-04-16T16:22:39.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.589+0000 7feab3400640 1 --2- 172.21.15.45:0/76767551 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fea9c06ca10 0x7fea9c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:39.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.589+0000 7feab3400640 1 -- 172.21.15.45:0/76767551 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5717+0+0 (secure 0 0 0) 0x7feaac08b270 con 0x7feabc06e480 2024-04-16T16:22:39.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.590+0000 7feabaa00640 1 --2- 172.21.15.45:0/76767551 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fea9c06ca10 0x7fea9c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:39.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.591+0000 7feabaa00640 1 --2- 172.21.15.45:0/76767551 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fea9c06ca10 0x7fea9c06eed0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feaa8036000 tx=0x7feaa803b040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:39.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.593+0000 7feab3400640 1 -- 172.21.15.45:0/76767551 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feabc0b2480 con 0x7feabc06e480 2024-04-16T16:22:39.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.837+0000 7feac0a00640 1 -- 172.21.15.45:0/76767551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feabc075780 con 0x7feabc06e480 2024-04-16T16:22:39.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.838+0000 7feab3400640 1 -- 172.21.15.45:0/76767551 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+3875 (secure 0 0 0) 0x7feaac0594e0 con 0x7feabc06e480 2024-04-16T16:22:39.837 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:39.838 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":107,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2585865379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2585865379},{"type":"v1","addr":"172.21.15.45:6834","nonce":2585865379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6590,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/777087937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":777087937},{"type":"v1","addr":"172.21.15.45:6835","nonce":777087937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":7711,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/3801897458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3801897458},{"type":"v1","addr":"172.21.15.103:6836","nonce":3801897458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":7714,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/4194999181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":4194999181},{"type":"v1","addr":"172.21.15.103:6837","nonce":4194999181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[]} 2024-04-16T16:22:39.838 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 107 2024-04-16T16:22:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.841+0000 7feac0a00640 1 -- 172.21.15.45:0/76767551 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fea9c06ca10 msgr2=0x7fea9c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.841+0000 7feac0a00640 1 --2- 172.21.15.45:0/76767551 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fea9c06ca10 0x7fea9c06eed0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feaa8036000 tx=0x7feaa803b040 comp rx=0 tx=0).stop 2024-04-16T16:22:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.841+0000 7feac0a00640 1 -- 172.21.15.45:0/76767551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feabc06e480 msgr2=0x7feabc1f7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.841+0000 7feac0a00640 1 --2- 172.21.15.45:0/76767551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feabc06e480 0x7feabc1f7a40 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7feaac00c3c0 tx=0x7feaac00c880 comp rx=0 tx=0).stop 2024-04-16T16:22:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.841+0000 7feac0a00640 1 -- 172.21.15.45:0/76767551 shutdown_connections 2024-04-16T16:22:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.841+0000 7feac0a00640 1 --2- 172.21.15.45:0/76767551 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fea9c06ca10 0x7fea9c06eed0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.841+0000 7feac0a00640 1 --2- 172.21.15.45:0/76767551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feabc0b2070 0x7feabc0cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.841+0000 7feac0a00640 1 --2- 172.21.15.45:0/76767551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feabc146220 0x7feabc0b1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.841+0000 7feac0a00640 1 --2- 172.21.15.45:0/76767551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feabc06e480 0x7feabc1f7a40 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.841+0000 7feac0a00640 1 -- 172.21.15.45:0/76767551 >> 172.21.15.45:0/76767551 conn(0x7feabc06b930 msgr2=0x7feabc06f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.841+0000 7feac0a00640 1 -- 172.21.15.45:0/76767551 shutdown_connections 2024-04-16T16:22:39.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:39.841+0000 7feac0a00640 1 -- 172.21.15.45:0/76767551 wait complete. 2024-04-16T16:22:39.861 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-16T16:22:39.861 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-16T16:22:39.861 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-16T16:22:39.861 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:22:39.861 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-16T16:22:39.861 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-16T16:22:39.862 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-16T16:22:39.862 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:22:39.862 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-16T16:22:39.862 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-16T16:22:39.862 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-16T16:22:39.862 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:22:39.862 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-16T16:22:39.862 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-16T16:22:39.863 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-16T16:22:39.863 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:22:39.881 INFO:tasks.ceph.mds.a:Stopped 2024-04-16T16:22:39.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-16T16:22:39.882 INFO:tasks.ceph.mds.c:Stopped 2024-04-16T16:22:39.882 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-16T16:22:39.885 INFO:tasks.ceph.mds.b:Stopped 2024-04-16T16:22:39.885 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-16T16:22:39.885 INFO:tasks.ceph.mds.d:Stopped 2024-04-16T16:22:39.885 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-16T16:22:39.928 INFO:tasks.ceph.mds.a:Started 2024-04-16T16:22:39.928 DEBUG:teuthology.parallel:result is None 2024-04-16T16:22:39.932 INFO:tasks.ceph.mds.b:Started 2024-04-16T16:22:39.933 DEBUG:teuthology.parallel:result is None 2024-04-16T16:22:39.933 INFO:tasks.ceph.mds.c:Started 2024-04-16T16:22:39.933 DEBUG:teuthology.parallel:result is None 2024-04-16T16:22:39.938 INFO:tasks.ceph.mds.d:Started 2024-04-16T16:22:39.938 DEBUG:teuthology.parallel:result is None 2024-04-16T16:22:39.938 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-16T16:22:40.090 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2024-04-16T16:22:40.093 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2024-04-16T16:22:40.099 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-16T16:22:40.103 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-16T16:22:40.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.133+0000 7fc893000640 1 Processor -- start 2024-04-16T16:22:40.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.133+0000 7fc893000640 1 -- start start 2024-04-16T16:22:40.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.135+0000 7fc893000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc88c0afe20 0x7fc88c06f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:40.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.135+0000 7fc893000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc88c0cdfe0 0x7fc88c0b4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:40.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.135+0000 7fc893000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88c06fab0 con 0x7fc88c06f550 2024-04-16T16:22:40.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.135+0000 7fc893000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc88c15bf20 con 0x7fc88c0cdfe0 2024-04-16T16:22:40.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.135+0000 7fc893000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc88c15c200 con 0x7fc88c0afe20 2024-04-16T16:22:40.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.135+0000 7fc891000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc88c0cdfe0 0x7fc88c0b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:40.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.135+0000 7fc891000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc88c0cdfe0 0x7fc88c0b4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58316/0 (socket says 172.21.15.45:58316) 2024-04-16T16:22:40.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.135+0000 7fc891000640 1 -- 172.21.15.45:0/2153233965 learned_addr learned my addr 172.21.15.45:0/2153233965 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:40.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.135+0000 7fc88be00640 1 --2- 172.21.15.45:0/2153233965 >> v2:172.21.15.103:3300/0 conn(0x7fc88c0afe20 0x7fc88c06f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:40.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.136+0000 7fc891000640 1 -- 172.21.15.45:0/2153233965 >> v2:172.21.15.103:3300/0 conn(0x7fc88c0afe20 msgr2=0x7fc88c06f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:40.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.136+0000 7fc891000640 1 --2- 172.21.15.45:0/2153233965 >> v2:172.21.15.103:3300/0 conn(0x7fc88c0afe20 0x7fc88c06f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:40.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.136+0000 7fc891000640 1 -- 172.21.15.45:0/2153233965 >> v1:172.21.15.103:6789/0 conn(0x7fc88c06f550 legacy=0x7fc88c0cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:40.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.136+0000 7fc891000640 1 -- 172.21.15.45:0/2153233965 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c15c4e0 con 0x7fc88c0cdfe0 2024-04-16T16:22:40.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.136+0000 7fc88be00640 1 --2- 172.21.15.45:0/2153233965 >> v2:172.21.15.103:3300/0 conn(0x7fc88c0afe20 0x7fc88c06f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:22:40.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.137+0000 7fc891000640 1 --2- 172.21.15.45:0/2153233965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc88c0cdfe0 0x7fc88c0b4160 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fc88c070350 tx=0x7fc88402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0a174382d27028e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:40.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.137+0000 7fc88aa00640 1 -- 172.21.15.45:0/2153233965 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88402cb20 con 0x7fc88c0cdfe0 2024-04-16T16:22:40.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.137+0000 7fc88aa00640 1 -- 172.21.15.45:0/2153233965 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc88402ccc0 con 0x7fc88c0cdfe0 2024-04-16T16:22:40.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.137+0000 7fc88aa00640 1 -- 172.21.15.45:0/2153233965 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc884034610 con 0x7fc88c0cdfe0 2024-04-16T16:22:40.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.138+0000 7fc893000640 1 -- 172.21.15.45:0/2153233965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc88c0cdfe0 msgr2=0x7fc88c0b4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:40.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.138+0000 7fc893000640 1 --2- 172.21.15.45:0/2153233965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc88c0cdfe0 0x7fc88c0b4160 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fc88c070350 tx=0x7fc88402bf80 comp rx=0 tx=0).stop 2024-04-16T16:22:40.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.138+0000 7fc893000640 1 -- 172.21.15.45:0/2153233965 shutdown_connections 2024-04-16T16:22:40.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.138+0000 7fc893000640 1 --2- 172.21.15.45:0/2153233965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc88c0cdfe0 0x7fc88c0b4160 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:40.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.138+0000 7fc893000640 1 --2- 172.21.15.45:0/2153233965 >> v2:172.21.15.103:3300/0 conn(0x7fc88c0afe20 0x7fc88c06f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:40.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.138+0000 7fc893000640 1 -- 172.21.15.45:0/2153233965 >> 172.21.15.45:0/2153233965 conn(0x7fc88c06b930 msgr2=0x7fc88c06bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:40.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.138+0000 7fc893000640 1 -- 172.21.15.45:0/2153233965 shutdown_connections 2024-04-16T16:22:40.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.138+0000 7fc893000640 1 -- 172.21.15.45:0/2153233965 wait complete. 2024-04-16T16:22:40.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.140+0000 7fc893000640 1 Processor -- start 2024-04-16T16:22:40.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.140+0000 7fc893000640 1 -- start start 2024-04-16T16:22:40.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.141+0000 7fc893000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc88c06f550 0x7fc88c14f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:40.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.141+0000 7fc893000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc88c0afe20 0x7fc88c0c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:40.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.141+0000 7fc893000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc88c0cdfe0 0x7fc88c14cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:40.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.141+0000 7fc893000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c0b59d0 con 0x7fc88c0afe20 2024-04-16T16:22:40.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.141+0000 7fc893000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c0b5850 con 0x7fc88c06f550 2024-04-16T16:22:40.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.141+0000 7fc893000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc88c15ca00 con 0x7fc88c0cdfe0 2024-04-16T16:22:40.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.141+0000 7fc88b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc88c0afe20 0x7fc88c0c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:40.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.141+0000 7fc88b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc88c0afe20 0x7fc88c0c61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58322/0 (socket says 172.21.15.45:58322) 2024-04-16T16:22:40.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.141+0000 7fc88b400640 1 -- 172.21.15.45:0/1327784713 learned_addr learned my addr 172.21.15.45:0/1327784713 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:40.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.141+0000 7fc88be00640 1 --2- 172.21.15.45:0/1327784713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc88c06f550 0x7fc88c14f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:40.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.141+0000 7fc891000640 1 --2- 172.21.15.45:0/1327784713 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc88c0cdfe0 0x7fc88c14cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:40.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.142+0000 7fc88b400640 1 -- 172.21.15.45:0/1327784713 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc88c0cdfe0 msgr2=0x7fc88c14cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:40.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.142+0000 7fc88b400640 1 --2- 172.21.15.45:0/1327784713 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc88c0cdfe0 0x7fc88c14cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:40.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.142+0000 7fc88b400640 1 -- 172.21.15.45:0/1327784713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc88c06f550 msgr2=0x7fc88c14f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:40.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.142+0000 7fc88b400640 1 --2- 172.21.15.45:0/1327784713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc88c06f550 0x7fc88c14f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:40.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.142+0000 7fc88b400640 1 -- 172.21.15.45:0/1327784713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c1583a0 con 0x7fc88c0afe20 2024-04-16T16:22:40.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.142+0000 7fc88b400640 1 --2- 172.21.15.45:0/1327784713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc88c0afe20 0x7fc88c0c61a0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fc87800c400 tx=0x7fc87800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:40.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.142+0000 7fc883e00640 1 -- 172.21.15.45:0/1327784713 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8780052d0 con 0x7fc88c0afe20 2024-04-16T16:22:40.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.142+0000 7fc883e00640 1 -- 172.21.15.45:0/1327784713 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc878005490 con 0x7fc88c0afe20 2024-04-16T16:22:40.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.142+0000 7fc883e00640 1 -- 172.21.15.45:0/1327784713 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc878007260 con 0x7fc88c0afe20 2024-04-16T16:22:40.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.143+0000 7fc893000640 1 -- 172.21.15.45:0/1327784713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc88c158680 con 0x7fc88c0afe20 2024-04-16T16:22:40.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.143+0000 7fc893000640 1 -- 172.21.15.45:0/1327784713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc88c158c40 con 0x7fc88c0afe20 2024-04-16T16:22:40.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.144+0000 7fc893000640 1 -- 172.21.15.45:0/1327784713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc88c158e40 con 0x7fc88c0afe20 2024-04-16T16:22:40.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.145+0000 7fc883e00640 1 -- 172.21.15.45:0/1327784713 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fc878007400 con 0x7fc88c0afe20 2024-04-16T16:22:40.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.146+0000 7fc883e00640 1 --2- 172.21.15.45:0/1327784713 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc86c06ca10 0x7fc86c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:40.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.146+0000 7fc883e00640 1 -- 172.21.15.45:0/1327784713 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5717+0+0 (secure 0 0 0) 0x7fc87808b200 con 0x7fc88c0afe20 2024-04-16T16:22:40.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.146+0000 7fc88be00640 1 --2- 172.21.15.45:0/1327784713 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc86c06ca10 0x7fc86c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:40.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.147+0000 7fc88be00640 1 --2- 172.21.15.45:0/1327784713 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc86c06ca10 0x7fc86c06eed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc88c06c160 tx=0x7fc87c008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:40.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.150+0000 7fc883e00640 1 -- 172.21.15.45:0/1327784713 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc88c158e40 con 0x7fc88c0afe20 2024-04-16T16:22:40.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:40.369+0000 7fc893000640 1 -- 172.21.15.45:0/1327784713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fc88c075780 con 0x7fc88c0afe20 2024-04-16T16:22:41.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.392+0000 7fc883e00640 1 -- 172.21.15.45:0/1327784713 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v152) ==== 97+0+0 (secure 0 0 0) 0x7fc878058e20 con 0x7fc88c0afe20 2024-04-16T16:22:41.391 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2024-04-16T16:22:41.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.396+0000 7fc893000640 1 -- 172.21.15.45:0/1327784713 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc86c06ca10 msgr2=0x7fc86c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:41.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.396+0000 7fc893000640 1 --2- 172.21.15.45:0/1327784713 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc86c06ca10 0x7fc86c06eed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc88c06c160 tx=0x7fc87c008040 comp rx=0 tx=0).stop 2024-04-16T16:22:41.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.396+0000 7fc893000640 1 -- 172.21.15.45:0/1327784713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc88c0afe20 msgr2=0x7fc88c0c61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:41.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.396+0000 7fc893000640 1 --2- 172.21.15.45:0/1327784713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc88c0afe20 0x7fc88c0c61a0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fc87800c400 tx=0x7fc87800c8c0 comp rx=0 tx=0).stop 2024-04-16T16:22:41.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.396+0000 7fc893000640 1 -- 172.21.15.45:0/1327784713 shutdown_connections 2024-04-16T16:22:41.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.396+0000 7fc893000640 1 --2- 172.21.15.45:0/1327784713 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc86c06ca10 0x7fc86c06eed0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:41.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.396+0000 7fc893000640 1 --2- 172.21.15.45:0/1327784713 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc88c0cdfe0 0x7fc88c14cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:41.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.396+0000 7fc893000640 1 --2- 172.21.15.45:0/1327784713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc88c0afe20 0x7fc88c0c61a0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:41.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.396+0000 7fc893000640 1 --2- 172.21.15.45:0/1327784713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc88c06f550 0x7fc88c14f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:41.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.396+0000 7fc893000640 1 -- 172.21.15.45:0/1327784713 >> 172.21.15.45:0/1327784713 conn(0x7fc88c06b930 msgr2=0x7fc88c0b1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:41.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.396+0000 7fc893000640 1 -- 172.21.15.45:0/1327784713 shutdown_connections 2024-04-16T16:22:41.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.397+0000 7fc893000640 1 -- 172.21.15.45:0/1327784713 wait complete. 2024-04-16T16:22:41.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-16T16:22:41.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.640+0000 7f5ce7000640 1 Processor -- start 2024-04-16T16:22:41.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.641+0000 7f5ce7000640 1 -- start start 2024-04-16T16:22:41.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.642+0000 7f5ce7000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ce015eea0 0x7f5ce0162b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:41.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.642+0000 7f5ce7000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5ce00b35d0 0x7f5ce00d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:41.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.642+0000 7f5ce7000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ce0163920 con 0x7f5ce00d24d0 2024-04-16T16:22:41.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.642+0000 7f5ce7000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5ce00d0940 con 0x7f5ce00b35d0 2024-04-16T16:22:41.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.642+0000 7f5ce7000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce006cc50 con 0x7f5ce015eea0 2024-04-16T16:22:41.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.643+0000 7f5cdfe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ce015eea0 0x7f5ce0162b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:41.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.643+0000 7f5ce5000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5ce00b35d0 0x7f5ce00d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:41.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.643+0000 7f5ce5000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5ce00b35d0 0x7f5ce00d0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:48972/0 (socket says 172.21.15.45:48972) 2024-04-16T16:22:41.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.643+0000 7f5ce5000640 1 -- 172.21.15.45:0/3958660284 learned_addr learned my addr 172.21.15.45:0/3958660284 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:41.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.643+0000 7f5cdf400640 1 --1- 172.21.15.45:0/3958660284 >> v1:172.21.15.103:6789/0 conn(0x7f5ce00d24d0 0x7f5ce0163050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.45:41416/0 (socket says 172.21.15.45:41416) 2024-04-16T16:22:41.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.644+0000 7f5ce5000640 1 -- 172.21.15.45:0/3958660284 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ce015eea0 msgr2=0x7f5ce0162b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:41.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.644+0000 7f5ce5000640 1 --2- 172.21.15.45:0/3958660284 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ce015eea0 0x7f5ce0162b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:41.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.644+0000 7f5ce5000640 1 -- 172.21.15.45:0/3958660284 >> v1:172.21.15.103:6789/0 conn(0x7f5ce00d24d0 legacy=0x7f5ce0163050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:41.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.644+0000 7f5ce5000640 1 -- 172.21.15.45:0/3958660284 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce006cf30 con 0x7f5ce00b35d0 2024-04-16T16:22:41.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.644+0000 7f5cdea00640 1 -- 172.21.15.45:0/3958660284 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2578760124 0 0) 0x7f5ce0163920 con 0x7f5ce00d24d0 2024-04-16T16:22:41.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.645+0000 7f5ce5000640 1 --2- 172.21.15.45:0/3958660284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ce00b35d0 0x7f5ce00d0db0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009e50 tx=0x7f5cd802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43904c74ded70313 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:41.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.646+0000 7f5cdea00640 1 -- 172.21.15.45:0/3958660284 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd802cb00 con 0x7f5ce00b35d0 2024-04-16T16:22:41.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.646+0000 7f5cdea00640 1 -- 172.21.15.45:0/3958660284 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cd802cca0 con 0x7f5ce00b35d0 2024-04-16T16:22:41.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.646+0000 7f5cdea00640 1 -- 172.21.15.45:0/3958660284 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8034610 con 0x7f5ce00b35d0 2024-04-16T16:22:41.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.647+0000 7f5ce7000640 1 -- 172.21.15.45:0/3958660284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ce00b35d0 msgr2=0x7f5ce00d0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:41.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.647+0000 7f5ce7000640 1 --2- 172.21.15.45:0/3958660284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ce00b35d0 0x7f5ce00d0db0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009e50 tx=0x7f5cd802bf60 comp rx=0 tx=0).stop 2024-04-16T16:22:41.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.647+0000 7f5ce7000640 1 -- 172.21.15.45:0/3958660284 shutdown_connections 2024-04-16T16:22:41.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.647+0000 7f5ce7000640 1 --2- 172.21.15.45:0/3958660284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ce00b35d0 0x7f5ce00d0db0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:41.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.647+0000 7f5ce7000640 1 --2- 172.21.15.45:0/3958660284 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ce015eea0 0x7f5ce0162b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:41.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.647+0000 7f5ce7000640 1 -- 172.21.15.45:0/3958660284 >> 172.21.15.45:0/3958660284 conn(0x7f5ce00d28b0 msgr2=0x7f5ce00d2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:41.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.647+0000 7f5ce7000640 1 -- 172.21.15.45:0/3958660284 shutdown_connections 2024-04-16T16:22:41.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.647+0000 7f5ce7000640 1 -- 172.21.15.45:0/3958660284 wait complete. 2024-04-16T16:22:41.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.649+0000 7f5ce7000640 1 Processor -- start 2024-04-16T16:22:41.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.649+0000 7f5ce7000640 1 -- start start 2024-04-16T16:22:41.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.650+0000 7f5ce7000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ce00b35d0 0x7f5ce00c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:41.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.650+0000 7f5ce7000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ce00d24d0 0x7f5ce014f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:41.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.650+0000 7f5ce7000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ce015eea0 0x7f5ce006d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:41.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.650+0000 7f5ce7000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce009adb0 con 0x7f5ce00d24d0 2024-04-16T16:22:41.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.650+0000 7f5ce7000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce009ac30 con 0x7f5ce00b35d0 2024-04-16T16:22:41.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.650+0000 7f5ce7000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce00b3b20 con 0x7f5ce015eea0 2024-04-16T16:22:41.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.651+0000 7f5cdf400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ce00d24d0 0x7f5ce014f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:41.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.651+0000 7f5cdf400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ce00d24d0 0x7f5ce014f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58332/0 (socket says 172.21.15.45:58332) 2024-04-16T16:22:41.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.651+0000 7f5cdf400640 1 -- 172.21.15.45:0/708842369 learned_addr learned my addr 172.21.15.45:0/708842369 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:41.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.651+0000 7f5cdfe00640 1 --2- 172.21.15.45:0/708842369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ce00b35d0 0x7f5ce00c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:41.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.651+0000 7f5ce5000640 1 --2- 172.21.15.45:0/708842369 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ce015eea0 0x7f5ce006d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:41.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.651+0000 7f5cdf400640 1 -- 172.21.15.45:0/708842369 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ce015eea0 msgr2=0x7f5ce006d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:41.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.651+0000 7f5cdf400640 1 --2- 172.21.15.45:0/708842369 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ce015eea0 0x7f5ce006d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:41.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.651+0000 7f5cdf400640 1 -- 172.21.15.45:0/708842369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ce00b35d0 msgr2=0x7f5ce00c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:41.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.651+0000 7f5cdf400640 1 --2- 172.21.15.45:0/708842369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ce00b35d0 0x7f5ce00c8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:41.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.651+0000 7f5cdf400640 1 -- 172.21.15.45:0/708842369 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce00c8070 con 0x7f5ce00d24d0 2024-04-16T16:22:41.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.651+0000 7f5ce5000640 1 --2- 172.21.15.45:0/708842369 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ce015eea0 0x7f5ce006d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:41.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.652+0000 7f5cdf400640 1 --2- 172.21.15.45:0/708842369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ce00d24d0 0x7f5ce014f6a0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00c4d0 tx=0x7f5ccc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:41.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.652+0000 7f5cd7e00640 1 -- 172.21.15.45:0/708842369 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc018070 con 0x7f5ce00d24d0 2024-04-16T16:22:41.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.652+0000 7f5cd7e00640 1 -- 172.21.15.45:0/708842369 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ccc0074e0 con 0x7f5ce00d24d0 2024-04-16T16:22:41.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.652+0000 7f5cd7e00640 1 -- 172.21.15.45:0/708842369 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc004a20 con 0x7f5ce00d24d0 2024-04-16T16:22:41.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.652+0000 7f5ce7000640 1 -- 172.21.15.45:0/708842369 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce00c8ae0 con 0x7f5ce00d24d0 2024-04-16T16:22:41.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.652+0000 7f5ce7000640 1 -- 172.21.15.45:0/708842369 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce00b6050 con 0x7f5ce00d24d0 2024-04-16T16:22:41.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.653+0000 7f5ce7000640 1 -- 172.21.15.45:0/708842369 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ce0075780 con 0x7f5ce00d24d0 2024-04-16T16:22:41.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.654+0000 7f5cd7e00640 1 -- 172.21.15.45:0/708842369 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f5ccc004bc0 con 0x7f5ce00d24d0 2024-04-16T16:22:41.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.655+0000 7f5cd7e00640 1 --2- 172.21.15.45:0/708842369 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5cc006ca10 0x7f5cc006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:41.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.656+0000 7f5cd7e00640 1 -- 172.21.15.45:0/708842369 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 4857+0+0 (secure 0 0 0) 0x7f5ccc08adc0 con 0x7f5ce00d24d0 2024-04-16T16:22:41.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.656+0000 7f5cdfe00640 1 --2- 172.21.15.45:0/708842369 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5cc006ca10 0x7f5cc006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:41.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.657+0000 7f5cdfe00640 1 --2- 172.21.15.45:0/708842369 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5cc006ca10 0x7f5cc006eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0009a30 tx=0x7f5cd0008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:41.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.661+0000 7f5cd7e00640 1 -- 172.21.15.45:0/708842369 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5ccc059390 con 0x7f5ce00d24d0 2024-04-16T16:22:41.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.937+0000 7f5ce7000640 1 -- 172.21.15.45:0/708842369 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f5ce00b1de0 con 0x7f5ce00d24d0 2024-04-16T16:22:41.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.939+0000 7f5cd7e00640 1 -- 172.21.15.45:0/708842369 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v26) ==== 80+0+5238 (secure 0 0 0) 0x7f5ccc058c90 con 0x7f5ce00d24d0 2024-04-16T16:22:41.938 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:41.938 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:22:41.938 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2024-04-16T16:22:41.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.938 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2024-04-16T16:22:41.938 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mpjReORAAJMoNf4qvfuxPSiV9+VlKfg==", 2024-04-16T16:22:41.938 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.938 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:22:41.938 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:22:41.938 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:22:41.939 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:22:41.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.939 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2024-04-16T16:22:41.939 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mxHeBIBAA3G8maaK65zaOaqXpAOA6xA==", 2024-04-16T16:22:41.939 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.939 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:22:41.939 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:22:41.939 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:22:41.939 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:22:41.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.939 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2024-04-16T16:22:41.940 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5m+AaODxAA08XSjc44/i2KYExYQSukaQ==", 2024-04-16T16:22:41.940 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.940 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:22:41.940 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:22:41.940 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:22:41.940 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:22:41.940 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.940 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2024-04-16T16:22:41.940 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBrox5mEQ4UMRAAOslSsxQ17bOxPrI6QMlRRw==", 2024-04-16T16:22:41.940 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.940 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBuox5m3l3OHxAA+UjwZx8Mn6tCu95TTYIG3w==", 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.941 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2024-04-16T16:22:41.942 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxox5mozt1LhAAJL5NtMQdPVBtKOaVlMQBtw==", 2024-04-16T16:22:41.942 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.942 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:22:41.942 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:22:41.942 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:22:41.942 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.942 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2024-04-16T16:22:41.942 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB1ox5mfE7kABAAIvgZ5Wx+eIJ6QNKj2GVdpw==", 2024-04-16T16:22:41.942 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.942 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:22:41.942 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:22:41.942 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:22:41.942 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB4ox5mEvxcDRAAO0yXBKvsC7hZ6bpOAJyuag==", 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+ox5mZubrORAAcGJLyRN5Ob9aBnopumnEUg==", 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:22:41.943 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCCox5mcRqsChAACktmuUHvqWkp/mHT3UcjRw==", 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFox5mT52FFhAAChhDDfNZmIMgxE1j90LZpQ==", 2024-04-16T16:22:41.944 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCIox5mC+EDIxAAxtvM0on00lxTLaoRjN0Gfw==", 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.945 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2024-04-16T16:22:41.946 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ==", 2024-04-16T16:22:41.946 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.946 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:22:41.946 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:22:41.946 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-16T16:22:41.946 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.946 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2024-04-16T16:22:41.946 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mYpVUGhAAYdR+40O6zbWqgbI55F0FYA==", 2024-04-16T16:22:41.946 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.946 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:22:41.946 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:22:41.946 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:22:41.946 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.947 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2024-04-16T16:22:41.947 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5mWdlUDxAAuO2FsGH1/E+qP1xcyUEe3A==", 2024-04-16T16:22:41.947 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.947 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:22:41.947 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:22:41.947 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:22:41.947 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.947 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2024-04-16T16:22:41.947 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBsox5miPdMJBAAqjKpctNYu9ot1lff3rouxg==", 2024-04-16T16:22:41.947 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.947 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2024-04-16T16:22:41.948 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2024-04-16T16:22:41.948 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw pool=fs2_data" 2024-04-16T16:22:41.948 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.948 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2024-04-16T16:22:41.948 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBpox5mO4PMLBAA/0YbQCfHujgdEzpXs0nOLw==", 2024-04-16T16:22:41.948 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.948 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:22:41.948 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2024-04-16T16:22:41.948 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2024-04-16T16:22:41.948 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:22:41.948 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5mGevaLBAAPpA6g+Fdi/XHqAqG0QZNag==", 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBqox5moiNRIRAAz5JzOrjKNLkcphT+iMo+zw==", 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2024-04-16T16:22:41.949 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.950 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:41.950 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-16T16:22:41.950 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:22:41.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.941+0000 7f5ce7000640 1 -- 172.21.15.45:0/708842369 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5cc006ca10 msgr2=0x7f5cc006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:41.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.941+0000 7f5ce7000640 1 --2- 172.21.15.45:0/708842369 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5cc006ca10 0x7f5cc006eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0009a30 tx=0x7f5cd0008040 comp rx=0 tx=0).stop 2024-04-16T16:22:41.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.941+0000 7f5ce7000640 1 -- 172.21.15.45:0/708842369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ce00d24d0 msgr2=0x7f5ce014f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:41.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.941+0000 7f5ce7000640 1 --2- 172.21.15.45:0/708842369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ce00d24d0 0x7f5ce014f6a0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00c4d0 tx=0x7f5ccc00c990 comp rx=0 tx=0).stop 2024-04-16T16:22:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.941+0000 7f5ce7000640 1 -- 172.21.15.45:0/708842369 shutdown_connections 2024-04-16T16:22:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.941+0000 7f5ce7000640 1 --2- 172.21.15.45:0/708842369 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5cc006ca10 0x7f5cc006eed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.941+0000 7f5ce7000640 1 --2- 172.21.15.45:0/708842369 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ce015eea0 0x7f5ce006d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.941+0000 7f5ce7000640 1 --2- 172.21.15.45:0/708842369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ce00d24d0 0x7f5ce014f6a0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.941+0000 7f5ce7000640 1 --2- 172.21.15.45:0/708842369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ce00b35d0 0x7f5ce00c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.941+0000 7f5ce7000640 1 -- 172.21.15.45:0/708842369 >> 172.21.15.45:0/708842369 conn(0x7f5ce00d28b0 msgr2=0x7f5ce0071c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.941+0000 7f5ce7000640 1 -- 172.21.15.45:0/708842369 shutdown_connections 2024-04-16T16:22:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:41.941+0000 7f5ce7000640 1 -- 172.21.15.45:0/708842369 wait complete. 2024-04-16T16:22:41.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:22:42.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.184+0000 7fa519e00640 1 Processor -- start 2024-04-16T16:22:42.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.184+0000 7fa519e00640 1 -- start start 2024-04-16T16:22:42.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.186+0000 7fa519e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51415ea70 0x7fa5140cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:42.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.186+0000 7fa519e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5140cf6d0 con 0x7fa5141462e0 2024-04-16T16:22:42.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.186+0000 7fa519e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5141466c0 con 0x7fa514072a90 2024-04-16T16:22:42.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.186+0000 7fa519e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa5140cf3f0 con 0x7fa51415ea70 2024-04-16T16:22:42.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.186+0000 7fa512a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa5141462e0 0x7fa514072380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53124/0 (socket says 172.21.15.45:53124) 2024-04-16T16:22:42.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.186+0000 7fa512a00640 1 -- 172.21.15.45:0/71290589 learned_addr learned my addr 172.21.15.45:0/71290589 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:42.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.187+0000 7fa513400640 1 --2- 172.21.15.45:0/71290589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51415ea70 0x7fa5140cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:42.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.187+0000 7fa511600640 1 -- 172.21.15.45:0/71290589 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1884793337 0 0) 0x7fa5140cf6d0 con 0x7fa5141462e0 2024-04-16T16:22:42.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.187+0000 7fa511600640 1 -- 172.21.15.45:0/71290589 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4f4003520 con 0x7fa5141462e0 2024-04-16T16:22:42.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.187+0000 7fa511600640 1 -- 172.21.15.45:0/71290589 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1676879298 0 0) 0x7fa4f4003520 con 0x7fa5141462e0 2024-04-16T16:22:42.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.187+0000 7fa511600640 1 -- 172.21.15.45:0/71290589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51415ea70 msgr2=0x7fa5140cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:42.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.187+0000 7fa511600640 1 --2- 172.21.15.45:0/71290589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51415ea70 0x7fa5140cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.187+0000 7fa511600640 1 -- 172.21.15.45:0/71290589 >> v1:172.21.15.103:6789/0 conn(0x7fa514072a90 legacy=0x7fa5140ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:42.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.187+0000 7fa511600640 1 -- 172.21.15.45:0/71290589 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa51415cf80 con 0x7fa5141462e0 2024-04-16T16:22:42.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.187+0000 7fa511600640 1 -- 172.21.15.45:0/71290589 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fa504001670 con 0x7fa5141462e0 2024-04-16T16:22:42.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.187+0000 7fa511600640 1 -- 172.21.15.45:0/71290589 >> v1:172.21.15.45:6789/0 conn(0x7fa5141462e0 legacy=0x7fa514072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:42.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.188+0000 7fa511600640 1 --2- 172.21.15.45:0/71290589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4f40039e0 0x7fa4f4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:42.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.188+0000 7fa511600640 1 --2- 172.21.15.45:0/71290589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51415ea70 0x7fa4f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:42.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.188+0000 7fa511600640 1 --2- 172.21.15.45:0/71290589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f4004d40 0x7fa4f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:42.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.188+0000 7fa511600640 1 -- 172.21.15.45:0/71290589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa5140cf3f0 con 0x7fa4f40039e0 2024-04-16T16:22:42.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.188+0000 7fa511600640 1 -- 172.21.15.45:0/71290589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f400baf0 con 0x7fa4f4004d40 2024-04-16T16:22:42.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.188+0000 7fa511600640 1 -- 172.21.15.45:0/71290589 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa4f400bcf0 con 0x7fa51415ea70 2024-04-16T16:22:42.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.188+0000 7fa512a00640 1 --2- 172.21.15.45:0/71290589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4f40039e0 0x7fa4f4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:42.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.188+0000 7fa512a00640 1 -- 172.21.15.45:0/71290589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51415ea70 msgr2=0x7fa4f4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:42.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.188+0000 7fa512000640 1 --2- 172.21.15.45:0/71290589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51415ea70 0x7fa4f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:42.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.188+0000 7fa513400640 1 --2- 172.21.15.45:0/71290589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f4004d40 0x7fa4f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:42.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.188+0000 7fa512a00640 1 --2- 172.21.15.45:0/71290589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51415ea70 0x7fa4f4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.188+0000 7fa512a00640 1 -- 172.21.15.45:0/71290589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f4004d40 msgr2=0x7fa4f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:42.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.188+0000 7fa512a00640 1 --2- 172.21.15.45:0/71290589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f4004d40 0x7fa4f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.188+0000 7fa512a00640 1 -- 172.21.15.45:0/71290589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f4003520 con 0x7fa4f40039e0 2024-04-16T16:22:42.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.189+0000 7fa513400640 1 --2- 172.21.15.45:0/71290589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f4004d40 0x7fa4f40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:42.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.189+0000 7fa512000640 1 --2- 172.21.15.45:0/71290589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51415ea70 0x7fa4f4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:42.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.190+0000 7fa512a00640 1 --2- 172.21.15.45:0/71290589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4f40039e0 0x7fa4f4003e00 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fa50400c240 tx=0x7fa4f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a637ea46a9e297 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:42.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.190+0000 7fa511600640 1 -- 172.21.15.45:0/71290589 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa504007cf0 con 0x7fa4f40039e0 2024-04-16T16:22:42.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.190+0000 7fa511600640 1 -- 172.21.15.45:0/71290589 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa504030070 con 0x7fa4f40039e0 2024-04-16T16:22:42.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.190+0000 7fa511600640 1 -- 172.21.15.45:0/71290589 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa504005d30 con 0x7fa4f40039e0 2024-04-16T16:22:42.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.191+0000 7fa519e00640 1 -- 172.21.15.45:0/71290589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4f40039e0 msgr2=0x7fa4f4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:42.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.191+0000 7fa519e00640 1 --2- 172.21.15.45:0/71290589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4f40039e0 0x7fa4f4003e00 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fa50400c240 tx=0x7fa4f4005830 comp rx=0 tx=0).stop 2024-04-16T16:22:42.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.191+0000 7fa513400640 1 -- 172.21.15.45:0/71290589 reap_dead start 2024-04-16T16:22:42.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.191+0000 7fa519e00640 1 -- 172.21.15.45:0/71290589 shutdown_connections 2024-04-16T16:22:42.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.191+0000 7fa519e00640 1 -- 172.21.15.45:0/71290589 >> 172.21.15.45:0/71290589 conn(0x7fa5140b2750 msgr2=0x7fa5140b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:42.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.191+0000 7fa519e00640 1 -- 172.21.15.45:0/71290589 shutdown_connections 2024-04-16T16:22:42.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.191+0000 7fa519e00640 1 -- 172.21.15.45:0/71290589 wait complete. 2024-04-16T16:22:42.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.193+0000 7fa519e00640 1 Processor -- start 2024-04-16T16:22:42.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.193+0000 7fa519e00640 1 -- start start 2024-04-16T16:22:42.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.194+0000 7fa519e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f40039e0 0x7fa5141402f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:42.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.194+0000 7fa519e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5141e5070 0x7fa51415bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:42.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.194+0000 7fa519e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51413e300 0x7fa51413e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:42.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.194+0000 7fa519e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa5140cedd0 con 0x7fa5141e5070 2024-04-16T16:22:42.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.194+0000 7fa519e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa51415d540 con 0x7fa4f40039e0 2024-04-16T16:22:42.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.194+0000 7fa519e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa5140cef50 con 0x7fa51413e300 2024-04-16T16:22:42.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.194+0000 7fa512000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5141e5070 0x7fa51415bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:42.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.194+0000 7fa512000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5141e5070 0x7fa51415bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58350/0 (socket says 172.21.15.45:58350) 2024-04-16T16:22:42.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.194+0000 7fa512000640 1 -- 172.21.15.45:0/4208226239 learned_addr learned my addr 172.21.15.45:0/4208226239 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:42.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.195+0000 7fa513400640 1 --2- 172.21.15.45:0/4208226239 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51413e300 0x7fa51413e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:42.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.195+0000 7fa512000640 1 -- 172.21.15.45:0/4208226239 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51413e300 msgr2=0x7fa51413e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:42.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.195+0000 7fa512a00640 1 --2- 172.21.15.45:0/4208226239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f40039e0 0x7fa5141402f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:42.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.195+0000 7fa512000640 1 --2- 172.21.15.45:0/4208226239 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51413e300 0x7fa51413e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.195+0000 7fa512000640 1 -- 172.21.15.45:0/4208226239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f40039e0 msgr2=0x7fa5141402f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:42.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.195+0000 7fa512000640 1 --2- 172.21.15.45:0/4208226239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f40039e0 0x7fa5141402f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.195+0000 7fa512000640 1 -- 172.21.15.45:0/4208226239 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5141e5880 con 0x7fa5141e5070 2024-04-16T16:22:42.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.195+0000 7fa512000640 1 --2- 172.21.15.45:0/4208226239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5141e5070 0x7fa51415bcc0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00db10 tx=0x7fa4fc0020c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:42.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.195+0000 7fa50b400640 1 -- 172.21.15.45:0/4208226239 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc018070 con 0x7fa5141e5070 2024-04-16T16:22:42.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.196+0000 7fa50b400640 1 -- 172.21.15.45:0/4208226239 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4fc003da0 con 0x7fa5141e5070 2024-04-16T16:22:42.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.196+0000 7fa50b400640 1 -- 172.21.15.45:0/4208226239 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc006d70 con 0x7fa5141e5070 2024-04-16T16:22:42.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.196+0000 7fa519e00640 1 -- 172.21.15.45:0/4208226239 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5141e5b60 con 0x7fa5141e5070 2024-04-16T16:22:42.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.196+0000 7fa519e00640 1 -- 172.21.15.45:0/4208226239 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5141e6120 con 0x7fa5141e5070 2024-04-16T16:22:42.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.197+0000 7fa519e00640 1 -- 172.21.15.45:0/4208226239 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa514075780 con 0x7fa5141e5070 2024-04-16T16:22:42.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.198+0000 7fa50b400640 1 -- 172.21.15.45:0/4208226239 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fa4fc00b740 con 0x7fa5141e5070 2024-04-16T16:22:42.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.199+0000 7fa50b400640 1 --2- 172.21.15.45:0/4208226239 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fa4f006ca10 0x7fa4f006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:42.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.199+0000 7fa50b400640 1 -- 172.21.15.45:0/4208226239 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 4857+0+0 (secure 0 0 0) 0x7fa4fc08b100 con 0x7fa5141e5070 2024-04-16T16:22:42.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.200+0000 7fa512a00640 1 --2- 172.21.15.45:0/4208226239 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fa4f006ca10 0x7fa4f006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:42.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.201+0000 7fa512a00640 1 --2- 172.21.15.45:0/4208226239 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fa4f006ca10 0x7fa4f006eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa504034000 tx=0x7fa504007000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:42.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.204+0000 7fa50b400640 1 -- 172.21.15.45:0/4208226239 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa4fc059000 con 0x7fa5141e5070 2024-04-16T16:22:42.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.419+0000 7fa519e00640 1 -- 172.21.15.45:0/4208226239 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa51415c200 con 0x7fa5141e5070 2024-04-16T16:22:42.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.421+0000 7fa50b400640 1 -- 172.21.15.45:0/4208226239 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) ==== 74+0+12901 (secure 0 0 0) 0x7fa4fc05d300 con 0x7fa5141e5070 2024-04-16T16:22:42.421 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:42.421 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":152,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:22:41.388670+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:22:42.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.424+0000 7fa519e00640 1 -- 172.21.15.45:0/4208226239 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fa4f006ca10 msgr2=0x7fa4f006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:42.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.424+0000 7fa519e00640 1 --2- 172.21.15.45:0/4208226239 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fa4f006ca10 0x7fa4f006eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa504034000 tx=0x7fa504007000 comp rx=0 tx=0).stop 2024-04-16T16:22:42.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.424+0000 7fa519e00640 1 -- 172.21.15.45:0/4208226239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5141e5070 msgr2=0x7fa51415bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:42.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.424+0000 7fa519e00640 1 --2- 172.21.15.45:0/4208226239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5141e5070 0x7fa51415bcc0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00db10 tx=0x7fa4fc0020c0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.424+0000 7fa519e00640 1 -- 172.21.15.45:0/4208226239 shutdown_connections 2024-04-16T16:22:42.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.425+0000 7fa519e00640 1 --2- 172.21.15.45:0/4208226239 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fa4f006ca10 0x7fa4f006eed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.425+0000 7fa519e00640 1 --2- 172.21.15.45:0/4208226239 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa51413e300 0x7fa51413e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.425+0000 7fa519e00640 1 --2- 172.21.15.45:0/4208226239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5141e5070 0x7fa51415bcc0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.425+0000 7fa519e00640 1 --2- 172.21.15.45:0/4208226239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f40039e0 0x7fa5141402f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.425+0000 7fa519e00640 1 -- 172.21.15.45:0/4208226239 >> 172.21.15.45:0/4208226239 conn(0x7fa5140b2750 msgr2=0x7fa514072380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:42.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.425+0000 7fa519e00640 1 -- 172.21.15.45:0/4208226239 shutdown_connections 2024-04-16T16:22:42.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.425+0000 7fa519e00640 1 -- 172.21.15.45:0/4208226239 wait complete. 2024-04-16T16:22:42.444 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T16:13:19.130583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T16:22:42.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T16:22:42.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.662+0000 7f304a000640 1 Processor -- start 2024-04-16T16:22:42.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.662+0000 7f304a000640 1 -- start start 2024-04-16T16:22:42.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.664+0000 7f304a000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f304406cf10 0x7f304406d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:42.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.664+0000 7f304a000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f30440d35d0 0x7f30440d3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:42.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.664+0000 7f304a000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f304406d8a0 con 0x7f304415ecc0 2024-04-16T16:22:42.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.664+0000 7f304a000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f30440d2b10 con 0x7f30440d35d0 2024-04-16T16:22:42.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.664+0000 7f304a000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3044163210 con 0x7f304406cf10 2024-04-16T16:22:42.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.664+0000 7f3043e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f30440d35d0 0x7f30440d3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:42.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.664+0000 7f3043e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f30440d35d0 0x7f30440d3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58360/0 (socket says 172.21.15.45:58360) 2024-04-16T16:22:42.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.664+0000 7f3043e00640 1 -- 172.21.15.45:0/597658357 learned_addr learned my addr 172.21.15.45:0/597658357 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.664+0000 7f3042a00640 1 --2- 172.21.15.45:0/597658357 >> v2:172.21.15.103:3300/0 conn(0x7f304406cf10 0x7f304406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.665+0000 7f3043e00640 1 -- 172.21.15.45:0/597658357 >> v2:172.21.15.103:3300/0 conn(0x7f304406cf10 msgr2=0x7f304406d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.665+0000 7f3043e00640 1 --2- 172.21.15.45:0/597658357 >> v2:172.21.15.103:3300/0 conn(0x7f304406cf10 0x7f304406d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.665+0000 7f3043e00640 1 -- 172.21.15.45:0/597658357 >> v1:172.21.15.103:6789/0 conn(0x7f304415ecc0 legacy=0x7f30441626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.665+0000 7f3043e00640 1 -- 172.21.15.45:0/597658357 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30441634f0 con 0x7f30440d35d0 2024-04-16T16:22:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.666+0000 7f3043e00640 1 --2- 172.21.15.45:0/597658357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30440d35d0 0x7f30440d3a00 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f304406e000 tx=0x7f303c02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896a902e40978fcb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.667+0000 7f3042000640 1 -- 172.21.15.45:0/597658357 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c02cbb0 con 0x7f30440d35d0 2024-04-16T16:22:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.667+0000 7f3042000640 1 -- 172.21.15.45:0/597658357 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f303c02cd50 con 0x7f30440d35d0 2024-04-16T16:22:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.667+0000 7f3042000640 1 -- 172.21.15.45:0/597658357 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c034610 con 0x7f30440d35d0 2024-04-16T16:22:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.667+0000 7f304a000640 1 -- 172.21.15.45:0/597658357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30440d35d0 msgr2=0x7f30440d3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.667+0000 7f304a000640 1 --2- 172.21.15.45:0/597658357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30440d35d0 0x7f30440d3a00 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f304406e000 tx=0x7f303c02c010 comp rx=0 tx=0).stop 2024-04-16T16:22:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.667+0000 7f304a000640 1 -- 172.21.15.45:0/597658357 shutdown_connections 2024-04-16T16:22:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.667+0000 7f304a000640 1 --2- 172.21.15.45:0/597658357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30440d35d0 0x7f30440d3a00 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.667+0000 7f304a000640 1 --2- 172.21.15.45:0/597658357 >> v2:172.21.15.103:3300/0 conn(0x7f304406cf10 0x7f304406d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.667+0000 7f304a000640 1 -- 172.21.15.45:0/597658357 >> 172.21.15.45:0/597658357 conn(0x7f3044073690 msgr2=0x7f30440d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:42.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.668+0000 7f304a000640 1 -- 172.21.15.45:0/597658357 shutdown_connections 2024-04-16T16:22:42.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.668+0000 7f304a000640 1 -- 172.21.15.45:0/597658357 wait complete. 2024-04-16T16:22:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.669+0000 7f304a000640 1 Processor -- start 2024-04-16T16:22:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.670+0000 7f304a000640 1 -- start start 2024-04-16T16:22:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.671+0000 7f304a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f304406cf10 0x7f30440cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.671+0000 7f304a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30440d35d0 0x7f3044150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.671+0000 7f304a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f304415ecc0 0x7f30440c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.671+0000 7f304a000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f30440ba540 con 0x7f304415ecc0 2024-04-16T16:22:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.671+0000 7f304a000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f30440d30b0 con 0x7f30440d35d0 2024-04-16T16:22:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.671+0000 7f304a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3044162db0 con 0x7f304406cf10 2024-04-16T16:22:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.671+0000 7f3043e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f304415ecc0 0x7f30440c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.671+0000 7f3043e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f304415ecc0 0x7f30440c84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58376/0 (socket says 172.21.15.45:58376) 2024-04-16T16:22:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.671+0000 7f3043e00640 1 -- 172.21.15.45:0/4015595264 learned_addr learned my addr 172.21.15.45:0/4015595264 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.671+0000 7f3042a00640 1 --2- 172.21.15.45:0/4015595264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30440d35d0 0x7f3044150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.671+0000 7f3043400640 1 --2- 172.21.15.45:0/4015595264 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f304406cf10 0x7f30440cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.672+0000 7f3043e00640 1 -- 172.21.15.45:0/4015595264 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f304406cf10 msgr2=0x7f30440cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.672+0000 7f3043e00640 1 --2- 172.21.15.45:0/4015595264 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f304406cf10 0x7f30440cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.672+0000 7f3043e00640 1 -- 172.21.15.45:0/4015595264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30440d35d0 msgr2=0x7f3044150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.672+0000 7f3043e00640 1 --2- 172.21.15.45:0/4015595264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30440d35d0 0x7f3044150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.672+0000 7f3043e00640 1 -- 172.21.15.45:0/4015595264 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3044151450 con 0x7f304415ecc0 2024-04-16T16:22:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.672+0000 7f3042a00640 1 --2- 172.21.15.45:0/4015595264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30440d35d0 0x7f3044150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.672+0000 7f3043e00640 1 --2- 172.21.15.45:0/4015595264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f304415ecc0 0x7f30440c84e0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f303c009f40 tx=0x7f303c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.672+0000 7f303b400640 1 -- 172.21.15.45:0/4015595264 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c043020 con 0x7f304415ecc0 2024-04-16T16:22:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.672+0000 7f303b400640 1 -- 172.21.15.45:0/4015595264 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f303c03e450 con 0x7f304415ecc0 2024-04-16T16:22:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.672+0000 7f304a000640 1 -- 172.21.15.45:0/4015595264 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f304415f140 con 0x7f304415ecc0 2024-04-16T16:22:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.672+0000 7f303b400640 1 -- 172.21.15.45:0/4015595264 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c009510 con 0x7f304415ecc0 2024-04-16T16:22:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.672+0000 7f304a000640 1 -- 172.21.15.45:0/4015595264 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f304415f9e0 con 0x7f304415ecc0 2024-04-16T16:22:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.674+0000 7f304a000640 1 -- 172.21.15.45:0/4015595264 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3044075780 con 0x7f304415ecc0 2024-04-16T16:22:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.675+0000 7f303b400640 1 -- 172.21.15.45:0/4015595264 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f303c046020 con 0x7f304415ecc0 2024-04-16T16:22:42.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.675+0000 7f303b400640 1 --2- 172.21.15.45:0/4015595264 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f302406ca40 0x7f302406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:42.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.676+0000 7f303b400640 1 -- 172.21.15.45:0/4015595264 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 4857+0+0 (secure 0 0 0) 0x7f303c0ae200 con 0x7f304415ecc0 2024-04-16T16:22:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.676+0000 7f3043400640 1 --2- 172.21.15.45:0/4015595264 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f302406ca40 0x7f302406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:42.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.677+0000 7f3043400640 1 --2- 172.21.15.45:0/4015595264 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f302406ca40 0x7f302406ef00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3034007fd0 tx=0x7f3034009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.680+0000 7f303b400640 1 -- 172.21.15.45:0/4015595264 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f303c04d030 con 0x7f304415ecc0 2024-04-16T16:22:42.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.891+0000 7f304a000640 1 -- 172.21.15.45:0/4015595264 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f30440d26c0 con 0x7f304415ecc0 2024-04-16T16:22:42.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.892+0000 7f303b400640 1 -- 172.21.15.45:0/4015595264 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v152) ==== 93+0+10 (secure 0 0 0) 0x7f303c07c7d0 con 0x7f304415ecc0 2024-04-16T16:22:42.891 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-04-16T16:22:42.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.894+0000 7f304a000640 1 -- 172.21.15.45:0/4015595264 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f302406ca40 msgr2=0x7f302406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:42.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.894+0000 7f304a000640 1 --2- 172.21.15.45:0/4015595264 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f302406ca40 0x7f302406ef00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3034007fd0 tx=0x7f3034009040 comp rx=0 tx=0).stop 2024-04-16T16:22:42.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.894+0000 7f304a000640 1 -- 172.21.15.45:0/4015595264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f304415ecc0 msgr2=0x7f30440c84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:42.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.894+0000 7f304a000640 1 --2- 172.21.15.45:0/4015595264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f304415ecc0 0x7f30440c84e0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f303c009f40 tx=0x7f303c004a40 comp rx=0 tx=0).stop 2024-04-16T16:22:42.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.895+0000 7f304a000640 1 -- 172.21.15.45:0/4015595264 shutdown_connections 2024-04-16T16:22:42.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.895+0000 7f304a000640 1 --2- 172.21.15.45:0/4015595264 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f302406ca40 0x7f302406ef00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.895+0000 7f304a000640 1 --2- 172.21.15.45:0/4015595264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f304415ecc0 0x7f30440c84e0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.895+0000 7f304a000640 1 --2- 172.21.15.45:0/4015595264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30440d35d0 0x7f3044150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.895+0000 7f304a000640 1 --2- 172.21.15.45:0/4015595264 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f304406cf10 0x7f30440cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:42.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.895+0000 7f304a000640 1 -- 172.21.15.45:0/4015595264 >> 172.21.15.45:0/4015595264 conn(0x7f3044073690 msgr2=0x7f30440b8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:42.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.895+0000 7f304a000640 1 -- 172.21.15.45:0/4015595264 shutdown_connections 2024-04-16T16:22:42.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:42.895+0000 7f304a000640 1 -- 172.21.15.45:0/4015595264 wait complete. 2024-04-16T16:22:42.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-16T16:22:43.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.145+0000 7f9f8be00640 1 Processor -- start 2024-04-16T16:22:43.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.146+0000 7f9f8be00640 1 -- start start 2024-04-16T16:22:43.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.147+0000 7f9f8be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c161e90 0x7f9f8c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:43.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.147+0000 7f9f8be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9f8c162270 0x7f9f8c0d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:43.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.147+0000 7f9f8be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9f8c0d52d0 0x7f9f8c06ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:43.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.147+0000 7f9f8be00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9f8c0d2e30 con 0x7f9f8c162270 2024-04-16T16:22:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.147+0000 7f9f8be00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9f8c0d3080 con 0x7f9f8c0d52d0 2024-04-16T16:22:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.147+0000 7f9f8be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9f8c0d3360 con 0x7f9f8c161e90 2024-04-16T16:22:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.148+0000 7f9f8a000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9f8c162270 0x7f9f8c0d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.148+0000 7f9f8a000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9f8c162270 0x7f9f8c0d3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58378/0 (socket says 172.21.15.45:58378) 2024-04-16T16:22:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.148+0000 7f9f8a000640 1 -- 172.21.15.45:0/2135694469 learned_addr learned my addr 172.21.15.45:0/2135694469 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.148+0000 7f9f8aa00640 1 --2- 172.21.15.45:0/2135694469 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c161e90 0x7f9f8c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.148+0000 7f9f8b400640 1 --2- 172.21.15.45:0/2135694469 >> v2:172.21.15.103:3300/0 conn(0x7f9f8c0d52d0 0x7f9f8c06ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.148+0000 7f9f8a000640 1 -- 172.21.15.45:0/2135694469 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c161e90 msgr2=0x7f9f8c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:43.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.148+0000 7f9f8a000640 1 --2- 172.21.15.45:0/2135694469 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c161e90 0x7f9f8c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.148+0000 7f9f8a000640 1 -- 172.21.15.45:0/2135694469 >> v2:172.21.15.103:3300/0 conn(0x7f9f8c0d52d0 msgr2=0x7f9f8c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:43.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.148+0000 7f9f8a000640 1 --2- 172.21.15.45:0/2135694469 >> v2:172.21.15.103:3300/0 conn(0x7f9f8c0d52d0 0x7f9f8c06ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.148+0000 7f9f8a000640 1 -- 172.21.15.45:0/2135694469 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8c0d3640 con 0x7f9f8c162270 2024-04-16T16:22:43.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.148+0000 7f9f8b400640 1 --2- 172.21.15.45:0/2135694469 >> v2:172.21.15.103:3300/0 conn(0x7f9f8c0d52d0 0x7f9f8c06ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:43.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.149+0000 7f9f8a000640 1 --2- 172.21.15.45:0/2135694469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8c162270 0x7f9f8c0d3e40 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0ba980 tx=0x7f9f7402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=774b564a08f09200 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:43.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.150+0000 7f9f89600640 1 -- 172.21.15.45:0/2135694469 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f74034070 con 0x7f9f8c162270 2024-04-16T16:22:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.150+0000 7f9f89600640 1 -- 172.21.15.45:0/2135694469 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f74030860 con 0x7f9f8c162270 2024-04-16T16:22:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.150+0000 7f9f89600640 1 -- 172.21.15.45:0/2135694469 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f74030c30 con 0x7f9f8c162270 2024-04-16T16:22:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.150+0000 7f9f8be00640 1 -- 172.21.15.45:0/2135694469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8c162270 msgr2=0x7f9f8c0d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.150+0000 7f9f8be00640 1 --2- 172.21.15.45:0/2135694469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8c162270 0x7f9f8c0d3e40 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0ba980 tx=0x7f9f7402b9a0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.150+0000 7f9f8be00640 1 -- 172.21.15.45:0/2135694469 shutdown_connections 2024-04-16T16:22:43.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.150+0000 7f9f8be00640 1 --2- 172.21.15.45:0/2135694469 >> v2:172.21.15.103:3300/0 conn(0x7f9f8c0d52d0 0x7f9f8c06ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.150+0000 7f9f8be00640 1 --2- 172.21.15.45:0/2135694469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8c162270 0x7f9f8c0d3e40 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.150+0000 7f9f8be00640 1 --2- 172.21.15.45:0/2135694469 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c161e90 0x7f9f8c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.151+0000 7f9f8be00640 1 -- 172.21.15.45:0/2135694469 >> 172.21.15.45:0/2135694469 conn(0x7f9f8c0d1940 msgr2=0x7f9f8c0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:43.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.151+0000 7f9f8be00640 1 -- 172.21.15.45:0/2135694469 shutdown_connections 2024-04-16T16:22:43.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.151+0000 7f9f8be00640 1 -- 172.21.15.45:0/2135694469 wait complete. 2024-04-16T16:22:43.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.152+0000 7f9f8be00640 1 Processor -- start 2024-04-16T16:22:43.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.152+0000 7f9f8be00640 1 -- start start 2024-04-16T16:22:43.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.153+0000 7f9f8be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c0d52d0 0x7f9f8c15b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:43.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.153+0000 7f9f8be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f8c161e90 0x7f9f8c14cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:43.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.153+0000 7f9f8be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8c162270 0x7f9f8c1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:43.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.153+0000 7f9f8be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c0d4380 con 0x7f9f8c162270 2024-04-16T16:22:43.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.153+0000 7f9f8be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c0cfd10 con 0x7f9f8c161e90 2024-04-16T16:22:43.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.153+0000 7f9f8be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9f8c0d4500 con 0x7f9f8c0d52d0 2024-04-16T16:22:43.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.154+0000 7f9f8b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8c162270 0x7f9f8c1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:43.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.154+0000 7f9f8b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8c162270 0x7f9f8c1597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58386/0 (socket says 172.21.15.45:58386) 2024-04-16T16:22:43.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.154+0000 7f9f8b400640 1 -- 172.21.15.45:0/1820389166 learned_addr learned my addr 172.21.15.45:0/1820389166 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:43.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.154+0000 7f9f8aa00640 1 --2- 172.21.15.45:0/1820389166 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c0d52d0 0x7f9f8c15b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:43.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.154+0000 7f9f8a000640 1 --2- 172.21.15.45:0/1820389166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f8c161e90 0x7f9f8c14cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:43.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.154+0000 7f9f8b400640 1 -- 172.21.15.45:0/1820389166 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c0d52d0 msgr2=0x7f9f8c15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:43.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.154+0000 7f9f8b400640 1 --2- 172.21.15.45:0/1820389166 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c0d52d0 0x7f9f8c15b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.154+0000 7f9f8b400640 1 -- 172.21.15.45:0/1820389166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f8c161e90 msgr2=0x7f9f8c14cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:43.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.154+0000 7f9f8b400640 1 --2- 172.21.15.45:0/1820389166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f8c161e90 0x7f9f8c14cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.154+0000 7f9f8b400640 1 -- 172.21.15.45:0/1820389166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8c15a200 con 0x7f9f8c162270 2024-04-16T16:22:43.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.154+0000 7f9f8a000640 1 --2- 172.21.15.45:0/1820389166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f8c161e90 0x7f9f8c14cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:43.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.155+0000 7f9f8b400640 1 --2- 172.21.15.45:0/1820389166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8c162270 0x7f9f8c1597b0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400cef0 tx=0x7f9f8400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:43.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.155+0000 7f9f83400640 1 -- 172.21.15.45:0/1820389166 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8401a070 con 0x7f9f8c162270 2024-04-16T16:22:43.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.155+0000 7f9f83400640 1 -- 172.21.15.45:0/1820389166 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f8400d840 con 0x7f9f8c162270 2024-04-16T16:22:43.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.155+0000 7f9f83400640 1 -- 172.21.15.45:0/1820389166 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8400dc30 con 0x7f9f8c162270 2024-04-16T16:22:43.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.155+0000 7f9f8be00640 1 -- 172.21.15.45:0/1820389166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f8c15a4e0 con 0x7f9f8c162270 2024-04-16T16:22:43.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.155+0000 7f9f8be00640 1 -- 172.21.15.45:0/1820389166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f8c15aaa0 con 0x7f9f8c162270 2024-04-16T16:22:43.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.156+0000 7f9f8be00640 1 -- 172.21.15.45:0/1820389166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f8c075780 con 0x7f9f8c162270 2024-04-16T16:22:43.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.157+0000 7f9f83400640 1 -- 172.21.15.45:0/1820389166 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f9f8401d070 con 0x7f9f8c162270 2024-04-16T16:22:43.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.158+0000 7f9f83400640 1 --2- 172.21.15.45:0/1820389166 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f9f6806ca10 0x7f9f6806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:43.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.159+0000 7f9f83400640 1 -- 172.21.15.45:0/1820389166 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 4857+0+0 (secure 0 0 0) 0x7f9f8408c1c0 con 0x7f9f8c162270 2024-04-16T16:22:43.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.159+0000 7f9f8aa00640 1 --2- 172.21.15.45:0/1820389166 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f9f6806ca10 0x7f9f6806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:43.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.160+0000 7f9f8aa00640 1 --2- 172.21.15.45:0/1820389166 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f9f6806ca10 0x7f9f6806eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c006fd0 tx=0x7f9f7c008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:43.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.163+0000 7f9f83400640 1 -- 172.21.15.45:0/1820389166 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9f8405a820 con 0x7f9f8c162270 2024-04-16T16:22:43.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.370+0000 7f9f8be00640 1 -- 172.21.15.45:0/1820389166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f9f8c1608b0 con 0x7f9f8c162270 2024-04-16T16:22:43.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.371+0000 7f9f83400640 1 -- 172.21.15.45:0/1820389166 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v152) ==== 84+0+65 (secure 0 0 0) 0x7f9f8405a120 con 0x7f9f8c162270 2024-04-16T16:22:43.370 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:43.370 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2024-04-16T16:22:43.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-16T16:22:43.371 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2024-04-16T16:22:43.371 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2024-04-16T16:22:43.371 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T16:22:43.371 INFO:teuthology.orchestra.run.smithi045.stdout:] 2024-04-16T16:22:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.374+0000 7f9f8be00640 1 -- 172.21.15.45:0/1820389166 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f9f6806ca10 msgr2=0x7f9f6806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.374+0000 7f9f8be00640 1 --2- 172.21.15.45:0/1820389166 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f9f6806ca10 0x7f9f6806eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c006fd0 tx=0x7f9f7c008040 comp rx=0 tx=0).stop 2024-04-16T16:22:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.374+0000 7f9f8be00640 1 -- 172.21.15.45:0/1820389166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8c162270 msgr2=0x7f9f8c1597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.374+0000 7f9f8be00640 1 --2- 172.21.15.45:0/1820389166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8c162270 0x7f9f8c1597b0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400cef0 tx=0x7f9f8400a710 comp rx=0 tx=0).stop 2024-04-16T16:22:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.374+0000 7f9f8be00640 1 -- 172.21.15.45:0/1820389166 shutdown_connections 2024-04-16T16:22:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.374+0000 7f9f8be00640 1 --2- 172.21.15.45:0/1820389166 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f9f6806ca10 0x7f9f6806eed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.374+0000 7f9f8be00640 1 --2- 172.21.15.45:0/1820389166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f8c162270 0x7f9f8c1597b0 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.374+0000 7f9f8be00640 1 --2- 172.21.15.45:0/1820389166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f8c161e90 0x7f9f8c14cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.374+0000 7f9f8be00640 1 --2- 172.21.15.45:0/1820389166 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c0d52d0 0x7f9f8c15b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.374+0000 7f9f8be00640 1 -- 172.21.15.45:0/1820389166 >> 172.21.15.45:0/1820389166 conn(0x7f9f8c0d1940 msgr2=0x7f9f8c0b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:43.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.374+0000 7f9f8be00640 1 -- 172.21.15.45:0/1820389166 shutdown_connections 2024-04-16T16:22:43.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.374+0000 7f9f8be00640 1 -- 172.21.15.45:0/1820389166 wait complete. 2024-04-16T16:22:43.393 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-16T16:22:43.393 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-16T16:22:43.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.625+0000 7f33ffe00640 1 Processor -- start 2024-04-16T16:22:43.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.625+0000 7f33ffe00640 1 -- start start 2024-04-16T16:22:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.627+0000 7f33ffe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f340006eaf0 0x7f34000ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.627+0000 7f33ffe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34000ceaf0 0x7f340015cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.627+0000 7f33ffe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3400071e60 con 0x7f34001452e0 2024-04-16T16:22:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.627+0000 7f33ffe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3400072f00 con 0x7f340006eaf0 2024-04-16T16:22:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.627+0000 7f33ffe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3400072140 con 0x7f34000ceaf0 2024-04-16T16:22:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.627+0000 7f33ff400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34000ceaf0 0x7f340015cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.628+0000 7f33ff400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34000ceaf0 0x7f340015cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:48336/0 (socket says 172.21.15.45:48336) 2024-04-16T16:22:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.628+0000 7f33ff400640 1 -- 172.21.15.45:0/2699242891 learned_addr learned my addr 172.21.15.45:0/2699242891 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.627+0000 7f33fe000640 1 --2- 172.21.15.45:0/2699242891 >> v2:172.21.15.103:3300/0 conn(0x7f340006eaf0 0x7f34000ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:43.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.628+0000 7f33ff400640 1 -- 172.21.15.45:0/2699242891 >> v2:172.21.15.103:3300/0 conn(0x7f340006eaf0 msgr2=0x7f34000ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:43.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.628+0000 7f33ff400640 1 --2- 172.21.15.45:0/2699242891 >> v2:172.21.15.103:3300/0 conn(0x7f340006eaf0 0x7f34000ce5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.628+0000 7f33ff400640 1 -- 172.21.15.45:0/2699242891 >> v1:172.21.15.103:6789/0 conn(0x7f34001452e0 legacy=0x7f340006e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:43.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.628+0000 7f33ff400640 1 -- 172.21.15.45:0/2699242891 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3400072420 con 0x7f34000ceaf0 2024-04-16T16:22:43.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.630+0000 7f33ff400640 1 --2- 172.21.15.45:0/2699242891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34000ceaf0 0x7f340015cc90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f33f8009e50 tx=0x7f33f802bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26365db75dac6664 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:43.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.630+0000 7f33fd600640 1 -- 172.21.15.45:0/2699242891 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f802cb90 con 0x7f34000ceaf0 2024-04-16T16:22:43.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.630+0000 7f33fd600640 1 -- 172.21.15.45:0/2699242891 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f802cd30 con 0x7f34000ceaf0 2024-04-16T16:22:43.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.630+0000 7f33fd600640 1 -- 172.21.15.45:0/2699242891 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f8034610 con 0x7f34000ceaf0 2024-04-16T16:22:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.631+0000 7f33ffe00640 1 -- 172.21.15.45:0/2699242891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34000ceaf0 msgr2=0x7f340015cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.631+0000 7f33ffe00640 1 --2- 172.21.15.45:0/2699242891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34000ceaf0 0x7f340015cc90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f33f8009e50 tx=0x7f33f802bff0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.631+0000 7f33ffe00640 1 -- 172.21.15.45:0/2699242891 shutdown_connections 2024-04-16T16:22:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.631+0000 7f33ffe00640 1 --2- 172.21.15.45:0/2699242891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34000ceaf0 0x7f340015cc90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.631+0000 7f33ffe00640 1 --2- 172.21.15.45:0/2699242891 >> v2:172.21.15.103:3300/0 conn(0x7f340006eaf0 0x7f34000ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.631+0000 7f33ffe00640 1 -- 172.21.15.45:0/2699242891 >> 172.21.15.45:0/2699242891 conn(0x7f34000b3bb0 msgr2=0x7f34000b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.631+0000 7f33ffe00640 1 -- 172.21.15.45:0/2699242891 shutdown_connections 2024-04-16T16:22:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.631+0000 7f33ffe00640 1 -- 172.21.15.45:0/2699242891 wait complete. 2024-04-16T16:22:43.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.633+0000 7f33ffe00640 1 Processor -- start 2024-04-16T16:22:43.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.634+0000 7f33ffe00640 1 -- start start 2024-04-16T16:22:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.635+0000 7f33ffe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f340006eaf0 0x7f34001f7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.635+0000 7f33ffe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34000ceaf0 0x7f340013e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.635+0000 7f33ffe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34001452e0 0x7f340013ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.635+0000 7f33ffe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f34000cef60 con 0x7f34001452e0 2024-04-16T16:22:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.635+0000 7f33ffe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34000729e0 con 0x7f340006eaf0 2024-04-16T16:22:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.635+0000 7f33ffe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f34000cf0e0 con 0x7f34000ceaf0 2024-04-16T16:22:43.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.635+0000 7f33ff400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34001452e0 0x7f340013ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:43.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.635+0000 7f33ff400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34001452e0 0x7f340013ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32966/0 (socket says 172.21.15.45:32966) 2024-04-16T16:22:43.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.635+0000 7f33ff400640 1 -- 172.21.15.45:0/5643861 learned_addr learned my addr 172.21.15.45:0/5643861 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:43.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.636+0000 7f33fe000640 1 --2- 172.21.15.45:0/5643861 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34000ceaf0 0x7f340013e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:43.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.636+0000 7f33fea00640 1 --2- 172.21.15.45:0/5643861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f340006eaf0 0x7f34001f7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.636+0000 7f33ff400640 1 -- 172.21.15.45:0/5643861 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34000ceaf0 msgr2=0x7f340013e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.636+0000 7f33ff400640 1 --2- 172.21.15.45:0/5643861 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34000ceaf0 0x7f340013e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.636+0000 7f33ff400640 1 -- 172.21.15.45:0/5643861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f340006eaf0 msgr2=0x7f34001f7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.636+0000 7f33ff400640 1 --2- 172.21.15.45:0/5643861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f340006eaf0 0x7f34001f7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.636+0000 7f33ff400640 1 -- 172.21.15.45:0/5643861 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3400136a60 con 0x7f34001452e0 2024-04-16T16:22:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.636+0000 7f33ff400640 1 --2- 172.21.15.45:0/5643861 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34001452e0 0x7f340013ec20 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f33f8036040 tx=0x7f33f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.637+0000 7f33f6a00640 1 -- 172.21.15.45:0/5643861 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f8043020 con 0x7f34001452e0 2024-04-16T16:22:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.637+0000 7f33f6a00640 1 -- 172.21.15.45:0/5643861 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f803e450 con 0x7f34001452e0 2024-04-16T16:22:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.637+0000 7f33f6a00640 1 -- 172.21.15.45:0/5643861 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f8009510 con 0x7f34001452e0 2024-04-16T16:22:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.637+0000 7f33ffe00640 1 -- 172.21.15.45:0/5643861 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3400136d40 con 0x7f34001452e0 2024-04-16T16:22:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.637+0000 7f33ffe00640 1 -- 172.21.15.45:0/5643861 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3400137300 con 0x7f34001452e0 2024-04-16T16:22:43.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.638+0000 7f33ffe00640 1 -- 172.21.15.45:0/5643861 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3400075780 con 0x7f34001452e0 2024-04-16T16:22:43.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.639+0000 7f33f6a00640 1 -- 172.21.15.45:0/5643861 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f33f8046020 con 0x7f34001452e0 2024-04-16T16:22:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.640+0000 7f33f6a00640 1 --2- 172.21.15.45:0/5643861 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f33e006ca10 0x7f33e006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.640+0000 7f33f6a00640 1 -- 172.21.15.45:0/5643861 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 4857+0+0 (secure 0 0 0) 0x7f33f80adf40 con 0x7f34001452e0 2024-04-16T16:22:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.641+0000 7f33fea00640 1 --2- 172.21.15.45:0/5643861 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f33e006ca10 0x7f33e006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.642+0000 7f33fea00640 1 --2- 172.21.15.45:0/5643861 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f33e006ca10 0x7f33e006eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f33f0000f60 tx=0x7f33f0009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:43.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.645+0000 7f33f6a00640 1 -- 172.21.15.45:0/5643861 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f33f804d030 con 0x7f34001452e0 2024-04-16T16:22:43.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:43.849+0000 7f33ffe00640 1 -- 172.21.15.45:0/5643861 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f3400073f40 con 0x7f34001452e0 2024-04-16T16:22:44.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.413+0000 7f33f6a00640 1 -- 172.21.15.45:0/5643861 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v153) ==== 138+0+0 (secure 0 0 0) 0x7f33f807c640 con 0x7f34001452e0 2024-04-16T16:22:44.413 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2024-04-16T16:22:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.417+0000 7f33ffe00640 1 -- 172.21.15.45:0/5643861 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f33e006ca10 msgr2=0x7f33e006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.417+0000 7f33ffe00640 1 --2- 172.21.15.45:0/5643861 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f33e006ca10 0x7f33e006eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f33f0000f60 tx=0x7f33f0009040 comp rx=0 tx=0).stop 2024-04-16T16:22:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.417+0000 7f33ffe00640 1 -- 172.21.15.45:0/5643861 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34001452e0 msgr2=0x7f340013ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.417+0000 7f33ffe00640 1 --2- 172.21.15.45:0/5643861 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34001452e0 0x7f340013ec20 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f33f8036040 tx=0x7f33f8004a40 comp rx=0 tx=0).stop 2024-04-16T16:22:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.418+0000 7f33ffe00640 1 -- 172.21.15.45:0/5643861 shutdown_connections 2024-04-16T16:22:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.418+0000 7f33ffe00640 1 --2- 172.21.15.45:0/5643861 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f33e006ca10 0x7f33e006eed0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.418+0000 7f33ffe00640 1 --2- 172.21.15.45:0/5643861 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34001452e0 0x7f340013ec20 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.418+0000 7f33ffe00640 1 --2- 172.21.15.45:0/5643861 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34000ceaf0 0x7f340013e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.418+0000 7f33ffe00640 1 --2- 172.21.15.45:0/5643861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f340006eaf0 0x7f34001f7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.418+0000 7f33ffe00640 1 -- 172.21.15.45:0/5643861 >> 172.21.15.45:0/5643861 conn(0x7f34000b3bb0 msgr2=0x7f3400142af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:44.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.418+0000 7f33ffe00640 1 -- 172.21.15.45:0/5643861 shutdown_connections 2024-04-16T16:22:44.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.419+0000 7f33ffe00640 1 -- 172.21.15.45:0/5643861 wait complete. 2024-04-16T16:22:44.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-16T16:22:44.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.659+0000 7fbe29200640 1 Processor -- start 2024-04-16T16:22:44.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.659+0000 7fbe29200640 1 -- start start 2024-04-16T16:22:44.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.660+0000 7fbe29200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe24155ab0 0x7fbe24162ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:44.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.660+0000 7fbe29200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbe240ba610 0x7fbe2415efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:44.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.660+0000 7fbe29200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe24155e90 con 0x7fbe240d18c0 2024-04-16T16:22:44.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.660+0000 7fbe29200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbe2415ff00 con 0x7fbe240ba610 2024-04-16T16:22:44.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.660+0000 7fbe29200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbe2415f940 con 0x7fbe24155ab0 2024-04-16T16:22:44.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.661+0000 7fbe23e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbe240d18c0 0x7fbe240d2a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38378/0 (socket says 172.21.15.45:38378) 2024-04-16T16:22:44.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.661+0000 7fbe23e00640 1 -- 172.21.15.45:0/758130598 learned_addr learned my addr 172.21.15.45:0/758130598 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:44.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.661+0000 7fbe1aa00640 1 --2- 172.21.15.45:0/758130598 >> v2:172.21.15.103:3300/0 conn(0x7fbe240ba610 0x7fbe2415efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:44.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.661+0000 7fbe23400640 1 --2- 172.21.15.45:0/758130598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe24155ab0 0x7fbe24162ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:44.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.661+0000 7fbe22a00640 1 -- 172.21.15.45:0/758130598 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1335601260 0 0) 0x7fbe24155e90 con 0x7fbe240d18c0 2024-04-16T16:22:44.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.661+0000 7fbe22a00640 1 -- 172.21.15.45:0/758130598 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe08003520 con 0x7fbe240d18c0 2024-04-16T16:22:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe22a00640 1 -- 172.21.15.45:0/758130598 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 221150843 0 0) 0x7fbe08003520 con 0x7fbe240d18c0 2024-04-16T16:22:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe22a00640 1 -- 172.21.15.45:0/758130598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe24155ab0 msgr2=0x7fbe24162ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe22a00640 1 --2- 172.21.15.45:0/758130598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe24155ab0 0x7fbe24162ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe22a00640 1 -- 172.21.15.45:0/758130598 >> v2:172.21.15.103:3300/0 conn(0x7fbe240ba610 msgr2=0x7fbe2415efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe22a00640 1 --2- 172.21.15.45:0/758130598 >> v2:172.21.15.103:3300/0 conn(0x7fbe240ba610 0x7fbe2415efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe22a00640 1 -- 172.21.15.45:0/758130598 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2415fc20 con 0x7fbe240d18c0 2024-04-16T16:22:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe22a00640 1 -- 172.21.15.45:0/758130598 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fbe1c0019f0 con 0x7fbe240d18c0 2024-04-16T16:22:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe22a00640 1 -- 172.21.15.45:0/758130598 >> v1:172.21.15.45:6789/0 conn(0x7fbe240d18c0 legacy=0x7fbe240d2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe22a00640 1 --2- 172.21.15.45:0/758130598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe24155ab0 0x7fbe080039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe22a00640 1 --2- 172.21.15.45:0/758130598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe08003f80 0x7fbe08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe22a00640 1 --2- 172.21.15.45:0/758130598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe08004d40 0x7fbe080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe22a00640 1 -- 172.21.15.45:0/758130598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2415ff00 con 0x7fbe08004d40 2024-04-16T16:22:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe22a00640 1 -- 172.21.15.45:0/758130598 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2415f940 con 0x7fbe08003f80 2024-04-16T16:22:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe22a00640 1 -- 172.21.15.45:0/758130598 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbe0800ba70 con 0x7fbe24155ab0 2024-04-16T16:22:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe23e00640 1 --2- 172.21.15.45:0/758130598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe08004d40 0x7fbe080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.662+0000 7fbe1aa00640 1 --2- 172.21.15.45:0/758130598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe08003f80 0x7fbe08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.663+0000 7fbe23400640 1 --2- 172.21.15.45:0/758130598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe24155ab0 0x7fbe080039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.663+0000 7fbe23e00640 1 -- 172.21.15.45:0/758130598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe24155ab0 msgr2=0x7fbe080039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.663+0000 7fbe23e00640 1 --2- 172.21.15.45:0/758130598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe24155ab0 0x7fbe080039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.663+0000 7fbe23e00640 1 -- 172.21.15.45:0/758130598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe08003f80 msgr2=0x7fbe08004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.663+0000 7fbe23e00640 1 --2- 172.21.15.45:0/758130598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe08003f80 0x7fbe08004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.663+0000 7fbe23e00640 1 -- 172.21.15.45:0/758130598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe08003520 con 0x7fbe08004d40 2024-04-16T16:22:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.664+0000 7fbe23e00640 1 --2- 172.21.15.45:0/758130598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe08004d40 0x7fbe080051c0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00a7a0 tx=0x7fbe08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e24fd3a246f4a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.664+0000 7fbe22a00640 1 -- 172.21.15.45:0/758130598 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c00a3f0 con 0x7fbe08004d40 2024-04-16T16:22:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.664+0000 7fbe22a00640 1 -- 172.21.15.45:0/758130598 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe1c034630 con 0x7fbe08004d40 2024-04-16T16:22:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.664+0000 7fbe22a00640 1 -- 172.21.15.45:0/758130598 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c034a80 con 0x7fbe08004d40 2024-04-16T16:22:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.665+0000 7fbe29200640 1 -- 172.21.15.45:0/758130598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe08004d40 msgr2=0x7fbe080051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.665+0000 7fbe29200640 1 --2- 172.21.15.45:0/758130598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe08004d40 0x7fbe080051c0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00a7a0 tx=0x7fbe08005830 comp rx=0 tx=0).stop 2024-04-16T16:22:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.665+0000 7fbe23e00640 1 -- 172.21.15.45:0/758130598 reap_dead start 2024-04-16T16:22:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.665+0000 7fbe29200640 1 -- 172.21.15.45:0/758130598 shutdown_connections 2024-04-16T16:22:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.665+0000 7fbe29200640 1 -- 172.21.15.45:0/758130598 >> 172.21.15.45:0/758130598 conn(0x7fbe2406d1a0 msgr2=0x7fbe2406d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.665+0000 7fbe29200640 1 -- 172.21.15.45:0/758130598 shutdown_connections 2024-04-16T16:22:44.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.665+0000 7fbe29200640 1 -- 172.21.15.45:0/758130598 wait complete. 2024-04-16T16:22:44.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.667+0000 7fbe29200640 1 Processor -- start 2024-04-16T16:22:44.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.667+0000 7fbe29200640 1 -- start start 2024-04-16T16:22:44.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.668+0000 7fbe29200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe08004d40 0x7fbe240cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:44.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.668+0000 7fbe29200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe240b7d60 0x7fbe240ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:44.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.668+0000 7fbe29200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe240b81e0 0x7fbe240cd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:44.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.668+0000 7fbe29200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2415f510 con 0x7fbe08004d40 2024-04-16T16:22:44.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.668+0000 7fbe29200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbe241629d0 con 0x7fbe240b81e0 2024-04-16T16:22:44.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.668+0000 7fbe29200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbe2415f690 con 0x7fbe240b7d60 2024-04-16T16:22:44.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.669+0000 7fbe23400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe08004d40 0x7fbe240cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:44.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.669+0000 7fbe23400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe08004d40 0x7fbe240cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32980/0 (socket says 172.21.15.45:32980) 2024-04-16T16:22:44.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.669+0000 7fbe23400640 1 -- 172.21.15.45:0/4272797973 learned_addr learned my addr 172.21.15.45:0/4272797973 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:44.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.669+0000 7fbe1aa00640 1 --2- 172.21.15.45:0/4272797973 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe240b7d60 0x7fbe240ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:44.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.669+0000 7fbe23e00640 1 --2- 172.21.15.45:0/4272797973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe240b81e0 0x7fbe240cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:44.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.669+0000 7fbe23400640 1 -- 172.21.15.45:0/4272797973 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe240b7d60 msgr2=0x7fbe240ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:44.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.669+0000 7fbe23400640 1 --2- 172.21.15.45:0/4272797973 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe240b7d60 0x7fbe240ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:44.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.669+0000 7fbe23400640 1 -- 172.21.15.45:0/4272797973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe240b81e0 msgr2=0x7fbe240cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:44.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.669+0000 7fbe23400640 1 --2- 172.21.15.45:0/4272797973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe240b81e0 0x7fbe240cd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:44.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.669+0000 7fbe23400640 1 -- 172.21.15.45:0/4272797973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe240cc990 con 0x7fbe08004d40 2024-04-16T16:22:44.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.670+0000 7fbe23400640 1 --2- 172.21.15.45:0/4272797973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe08004d40 0x7fbe240cc450 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fbe14011630 tx=0x7fbe14011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:44.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.670+0000 7fbe1be00640 1 -- 172.21.15.45:0/4272797973 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1400f920 con 0x7fbe08004d40 2024-04-16T16:22:44.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.670+0000 7fbe1be00640 1 -- 172.21.15.45:0/4272797973 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe1400fac0 con 0x7fbe08004d40 2024-04-16T16:22:44.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.670+0000 7fbe1be00640 1 -- 172.21.15.45:0/4272797973 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1400fe70 con 0x7fbe08004d40 2024-04-16T16:22:44.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.670+0000 7fbe29200640 1 -- 172.21.15.45:0/4272797973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe241f4f70 con 0x7fbe08004d40 2024-04-16T16:22:44.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.670+0000 7fbe29200640 1 -- 172.21.15.45:0/4272797973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe240cbc50 con 0x7fbe08004d40 2024-04-16T16:22:44.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.671+0000 7fbe29200640 1 -- 172.21.15.45:0/4272797973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe24075780 con 0x7fbe08004d40 2024-04-16T16:22:44.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.672+0000 7fbe1be00640 1 -- 172.21.15.45:0/4272797973 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fbe14015460 con 0x7fbe08004d40 2024-04-16T16:22:44.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.673+0000 7fbe1be00640 1 --2- 172.21.15.45:0/4272797973 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbe0006ca10 0x7fbe0006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:44.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.673+0000 7fbe1be00640 1 -- 172.21.15.45:0/4272797973 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 5227+0+0 (secure 0 0 0) 0x7fbe1408c490 con 0x7fbe08004d40 2024-04-16T16:22:44.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.674+0000 7fbe1aa00640 1 --2- 172.21.15.45:0/4272797973 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbe0006ca10 0x7fbe0006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:44.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.675+0000 7fbe1aa00640 1 --2- 172.21.15.45:0/4272797973 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbe0006ca10 0x7fbe0006eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbe240b8fa0 tx=0x7fbe0c008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:44.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.678+0000 7fbe1be00640 1 -- 172.21.15.45:0/4272797973 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe14019030 con 0x7fbe08004d40 2024-04-16T16:22:44.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:44.883+0000 7fbe29200640 1 -- 172.21.15.45:0/4272797973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fbe24075100 con 0x7fbe08004d40 2024-04-16T16:22:45.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.432+0000 7fbe1be00640 1 -- 172.21.15.45:0/4272797973 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v154) ==== 170+0+0 (secure 0 0 0) 0x7fbe1405a990 con 0x7fbe08004d40 2024-04-16T16:22:45.431 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2024-04-16T16:22:45.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.436+0000 7fbe29200640 1 -- 172.21.15.45:0/4272797973 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbe0006ca10 msgr2=0x7fbe0006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:45.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.436+0000 7fbe29200640 1 --2- 172.21.15.45:0/4272797973 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbe0006ca10 0x7fbe0006eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbe240b8fa0 tx=0x7fbe0c008040 comp rx=0 tx=0).stop 2024-04-16T16:22:45.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.436+0000 7fbe29200640 1 -- 172.21.15.45:0/4272797973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe08004d40 msgr2=0x7fbe240cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:45.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.436+0000 7fbe29200640 1 --2- 172.21.15.45:0/4272797973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe08004d40 0x7fbe240cc450 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fbe14011630 tx=0x7fbe14011af0 comp rx=0 tx=0).stop 2024-04-16T16:22:45.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.436+0000 7fbe29200640 1 -- 172.21.15.45:0/4272797973 shutdown_connections 2024-04-16T16:22:45.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.436+0000 7fbe29200640 1 --2- 172.21.15.45:0/4272797973 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fbe0006ca10 0x7fbe0006eed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:45.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.436+0000 7fbe29200640 1 --2- 172.21.15.45:0/4272797973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe240b81e0 0x7fbe240cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:45.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.436+0000 7fbe29200640 1 --2- 172.21.15.45:0/4272797973 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe240b7d60 0x7fbe240ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:45.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.436+0000 7fbe29200640 1 --2- 172.21.15.45:0/4272797973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe08004d40 0x7fbe240cc450 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:45.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.436+0000 7fbe29200640 1 -- 172.21.15.45:0/4272797973 >> 172.21.15.45:0/4272797973 conn(0x7fbe2406d1a0 msgr2=0x7fbe240d4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:45.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.436+0000 7fbe29200640 1 -- 172.21.15.45:0/4272797973 shutdown_connections 2024-04-16T16:22:45.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.436+0000 7fbe29200640 1 -- 172.21.15.45:0/4272797973 wait complete. 2024-04-16T16:22:45.469 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-16T16:22:45.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.661+0000 7f9743200640 1 Processor -- start 2024-04-16T16:22:45.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.661+0000 7f9743200640 1 -- start start 2024-04-16T16:22:45.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.663+0000 7f9743200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f973c15c7c0 0x7f973c072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:45.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.663+0000 7f9743200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f973c0b5530 con 0x7f973c0d10e0 2024-04-16T16:22:45.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.663+0000 7f9743200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f973c0b5810 con 0x7f973c0b4930 2024-04-16T16:22:45.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.663+0000 7f9743200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f973c0b52e0 con 0x7f973c15c7c0 2024-04-16T16:22:45.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.664+0000 7f973be00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f973c15c7c0 0x7f973c072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:45.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.664+0000 7f9741200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f973c0d10e0 0x7f973c06f160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38380/0 (socket says 172.21.15.45:38380) 2024-04-16T16:22:45.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.664+0000 7f9741200640 1 -- 172.21.15.45:0/1689080903 learned_addr learned my addr 172.21.15.45:0/1689080903 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:45.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.664+0000 7f973b400640 1 -- 172.21.15.45:0/1689080903 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 360253967 0 0) 0x7f973c0b5530 con 0x7f973c0d10e0 2024-04-16T16:22:45.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.664+0000 7f973b400640 1 -- 172.21.15.45:0/1689080903 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9720003520 con 0x7f973c0d10e0 2024-04-16T16:22:45.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.664+0000 7f973be00640 1 -- 172.21.15.45:0/1689080903 >> v1:172.21.15.103:6789/0 conn(0x7f973c0b4930 legacy=0x7f973c0d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:45.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.664+0000 7f973be00640 1 -- 172.21.15.45:0/1689080903 >> v1:172.21.15.45:6789/0 conn(0x7f973c0d10e0 legacy=0x7f973c06f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:45.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.664+0000 7f973be00640 1 -- 172.21.15.45:0/1689080903 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f973c0d04b0 con 0x7f973c15c7c0 2024-04-16T16:22:45.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.666+0000 7f973be00640 1 --2- 172.21.15.45:0/1689080903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f973c15c7c0 0x7f973c072e10 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f972c009d90 tx=0x7f972c02bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e562ee59eea03ba server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:45.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.666+0000 7f973b400640 1 -- 172.21.15.45:0/1689080903 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972c039070 con 0x7f973c15c7c0 2024-04-16T16:22:45.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.666+0000 7f973b400640 1 -- 172.21.15.45:0/1689080903 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f972c02cb40 con 0x7f973c15c7c0 2024-04-16T16:22:45.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.666+0000 7f973b400640 1 -- 172.21.15.45:0/1689080903 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972c034440 con 0x7f973c15c7c0 2024-04-16T16:22:45.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.667+0000 7f9743200640 1 -- 172.21.15.45:0/1689080903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f973c15c7c0 msgr2=0x7f973c072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:45.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.667+0000 7f9743200640 1 --2- 172.21.15.45:0/1689080903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f973c15c7c0 0x7f973c072e10 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f972c009d90 tx=0x7f972c02bd90 comp rx=0 tx=0).stop 2024-04-16T16:22:45.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.667+0000 7f9743200640 1 -- 172.21.15.45:0/1689080903 shutdown_connections 2024-04-16T16:22:45.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.667+0000 7f9743200640 1 --2- 172.21.15.45:0/1689080903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f973c15c7c0 0x7f973c072e10 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:45.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.667+0000 7f9743200640 1 -- 172.21.15.45:0/1689080903 >> 172.21.15.45:0/1689080903 conn(0x7f973c0b3970 msgr2=0x7f973c0b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:45.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.667+0000 7f9743200640 1 -- 172.21.15.45:0/1689080903 shutdown_connections 2024-04-16T16:22:45.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.667+0000 7f9743200640 1 -- 172.21.15.45:0/1689080903 wait complete. 2024-04-16T16:22:45.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.669+0000 7f9743200640 1 Processor -- start 2024-04-16T16:22:45.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.669+0000 7f9743200640 1 -- start start 2024-04-16T16:22:45.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.670+0000 7f9743200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c0b4930 0x7f973c1f5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:45.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.670+0000 7f9743200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f973c0d10e0 0x7f973c1e7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:45.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.670+0000 7f9743200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f973c15c7c0 0x7f973c0aaad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:45.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.670+0000 7f9743200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f973c0d0260 con 0x7f973c0d10e0 2024-04-16T16:22:45.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.670+0000 7f9743200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f973c06fac0 con 0x7f973c0b4930 2024-04-16T16:22:45.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.670+0000 7f9743200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f973c073350 con 0x7f973c15c7c0 2024-04-16T16:22:45.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.670+0000 7f9733400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f973c0d10e0 0x7f973c1e7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:45.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.670+0000 7f9733400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f973c0d10e0 0x7f973c1e7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32990/0 (socket says 172.21.15.45:32990) 2024-04-16T16:22:45.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.670+0000 7f9733400640 1 -- 172.21.15.45:0/363327059 learned_addr learned my addr 172.21.15.45:0/363327059 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:45.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.671+0000 7f973be00640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c0b4930 0x7f973c1f5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:45.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.671+0000 7f9741200640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f973c15c7c0 0x7f973c0aaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:45.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.671+0000 7f9733400640 1 -- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f973c15c7c0 msgr2=0x7f973c0aaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:45.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.671+0000 7f9733400640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f973c15c7c0 0x7f973c0aaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:45.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.671+0000 7f9733400640 1 -- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c0b4930 msgr2=0x7f973c1f5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:45.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.671+0000 7f9733400640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c0b4930 0x7f973c1f5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:45.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.671+0000 7f9733400640 1 -- 172.21.15.45:0/363327059 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f973c1e6c80 con 0x7f973c0d10e0 2024-04-16T16:22:45.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.671+0000 7f9741200640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f973c15c7c0 0x7f973c0aaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:45.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.671+0000 7f973be00640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c0b4930 0x7f973c1f5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:45.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.671+0000 7f9733400640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f973c0d10e0 0x7f973c1e7de0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f972400b3f0 tx=0x7f972400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:45.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.671+0000 7f9738c00640 1 -- 172.21.15.45:0/363327059 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9724015020 con 0x7f973c0d10e0 2024-04-16T16:22:45.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.672+0000 7f9738c00640 1 -- 172.21.15.45:0/363327059 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9724011830 con 0x7f973c0d10e0 2024-04-16T16:22:45.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.672+0000 7f9738c00640 1 -- 172.21.15.45:0/363327059 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9724011c20 con 0x7f973c0d10e0 2024-04-16T16:22:45.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.672+0000 7f9743200640 1 -- 172.21.15.45:0/363327059 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f973c1e6f20 con 0x7f973c0d10e0 2024-04-16T16:22:45.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.672+0000 7f9743200640 1 -- 172.21.15.45:0/363327059 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f973c1e74e0 con 0x7f973c0d10e0 2024-04-16T16:22:45.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.673+0000 7f9743200640 1 -- 172.21.15.45:0/363327059 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f973c0b5520 con 0x7f973c0d10e0 2024-04-16T16:22:45.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.674+0000 7f9738c00640 1 -- 172.21.15.45:0/363327059 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f9724018020 con 0x7f973c0d10e0 2024-04-16T16:22:45.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.675+0000 7f9738c00640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f971806ca90 0x7f971806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:45.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.676+0000 7f9738c00640 1 -- 172.21.15.45:0/363327059 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 5609+0+0 (secure 0 0 0) 0x7f972408b8f0 con 0x7f973c0d10e0 2024-04-16T16:22:45.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.676+0000 7f973be00640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f971806ca90 0x7f971806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:45.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.677+0000 7f973be00640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f971806ca90 0x7f971806ef50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f972c036000 tx=0x7f972c030000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:45.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.681+0000 7f9738c00640 1 -- 172.21.15.45:0/363327059 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f973c0b5520 con 0x7f973c0d10e0 2024-04-16T16:22:45.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:45.925+0000 7f9743200640 1 -- 172.21.15.45:0/363327059 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f973c06d210 con 0x7f973c0d10e0 2024-04-16T16:22:46.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.435+0000 7f9738c00640 1 -- 172.21.15.45:0/363327059 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v110) ==== 174+0+0 (secure 0 0 0) 0x7f972401f030 con 0x7f973c0d10e0 2024-04-16T16:22:46.435 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 22 and data pool 23 2024-04-16T16:22:46.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.439+0000 7f9743200640 1 -- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f971806ca90 msgr2=0x7f971806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:46.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.439+0000 7f9743200640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f971806ca90 0x7f971806ef50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f972c036000 tx=0x7f972c030000 comp rx=0 tx=0).stop 2024-04-16T16:22:46.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.439+0000 7f9743200640 1 -- 172.21.15.45:0/363327059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f973c0d10e0 msgr2=0x7f973c1e7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:46.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.439+0000 7f9743200640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f973c0d10e0 0x7f973c1e7de0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f972400b3f0 tx=0x7f972400b8b0 comp rx=0 tx=0).stop 2024-04-16T16:22:46.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.439+0000 7f9743200640 1 -- 172.21.15.45:0/363327059 shutdown_connections 2024-04-16T16:22:46.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.439+0000 7f9743200640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f971806ca90 0x7f971806ef50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:46.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.439+0000 7f9743200640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f973c15c7c0 0x7f973c0aaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:46.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.439+0000 7f9743200640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f973c0d10e0 0x7f973c1e7de0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:46.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.439+0000 7f9743200640 1 --2- 172.21.15.45:0/363327059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c0b4930 0x7f973c1f5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:46.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.439+0000 7f9743200640 1 -- 172.21.15.45:0/363327059 >> 172.21.15.45:0/363327059 conn(0x7f973c0b3970 msgr2=0x7f973c0b2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:46.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.439+0000 7f9743200640 1 -- 172.21.15.45:0/363327059 shutdown_connections 2024-04-16T16:22:46.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.440+0000 7f9743200640 1 -- 172.21.15.45:0/363327059 wait complete. 2024-04-16T16:22:46.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:22:46.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.666+0000 7fd749e00640 1 Processor -- start 2024-04-16T16:22:46.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.666+0000 7fd749e00640 1 -- start start 2024-04-16T16:22:46.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.668+0000 7fd749e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd744072a90 0x7fd7441699a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:46.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.668+0000 7fd749e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd744147c20 con 0x7fd744147840 2024-04-16T16:22:46.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.668+0000 7fd749e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd744160210 con 0x7fd7440aed90 2024-04-16T16:22:46.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.668+0000 7fd749e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd7440afe80 con 0x7fd744072a90 2024-04-16T16:22:46.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.668+0000 7fd742a00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd744147840 0x7fd74415fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38396/0 (socket says 172.21.15.45:38396) 2024-04-16T16:22:46.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.668+0000 7fd742a00640 1 -- 172.21.15.45:0/3053603439 learned_addr learned my addr 172.21.15.45:0/3053603439 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:46.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.669+0000 7fd743400640 1 --2- 172.21.15.45:0/3053603439 >> v2:172.21.15.103:3300/0 conn(0x7fd744072a90 0x7fd7441699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:46.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.669+0000 7fd742000640 1 -- 172.21.15.45:0/3053603439 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4186779916 0 0) 0x7fd744147c20 con 0x7fd744147840 2024-04-16T16:22:46.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.669+0000 7fd742000640 1 -- 172.21.15.45:0/3053603439 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd728003520 con 0x7fd744147840 2024-04-16T16:22:46.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.669+0000 7fd742000640 1 -- 172.21.15.45:0/3053603439 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2504597285 0 0) 0x7fd728003520 con 0x7fd744147840 2024-04-16T16:22:46.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.669+0000 7fd742000640 1 -- 172.21.15.45:0/3053603439 >> v2:172.21.15.103:3300/0 conn(0x7fd744072a90 msgr2=0x7fd7441699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:46.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.669+0000 7fd742000640 1 --2- 172.21.15.45:0/3053603439 >> v2:172.21.15.103:3300/0 conn(0x7fd744072a90 0x7fd7441699a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:46.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.669+0000 7fd742000640 1 -- 172.21.15.45:0/3053603439 >> v1:172.21.15.103:6789/0 conn(0x7fd7440aed90 legacy=0x7fd744072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:46.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.669+0000 7fd742000640 1 -- 172.21.15.45:0/3053603439 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7440af270 con 0x7fd744147840 2024-04-16T16:22:46.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.669+0000 7fd742000640 1 -- 172.21.15.45:0/3053603439 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fd734001670 con 0x7fd744147840 2024-04-16T16:22:46.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.669+0000 7fd742000640 1 -- 172.21.15.45:0/3053603439 >> v1:172.21.15.45:6789/0 conn(0x7fd744147840 legacy=0x7fd74415fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:46.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.669+0000 7fd742000640 1 --2- 172.21.15.45:0/3053603439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7280039b0 0x7fd728003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:46.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.669+0000 7fd742000640 1 --2- 172.21.15.45:0/3053603439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd728004800 0x7fd728004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:46.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.670+0000 7fd742000640 1 --2- 172.21.15.45:0/3053603439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7280051c0 0x7fd72800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:46.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.670+0000 7fd742000640 1 -- 172.21.15.45:0/3053603439 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd7440afe80 con 0x7fd7280039b0 2024-04-16T16:22:46.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.670+0000 7fd742000640 1 -- 172.21.15.45:0/3053603439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd72800c090 con 0x7fd728004800 2024-04-16T16:22:46.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.670+0000 7fd742000640 1 -- 172.21.15.45:0/3053603439 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd72800c290 con 0x7fd7280051c0 2024-04-16T16:22:46.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.670+0000 7fd742a00640 1 --2- 172.21.15.45:0/3053603439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7280039b0 0x7fd728003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:46.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.670+0000 7fd742a00640 1 -- 172.21.15.45:0/3053603439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7280051c0 msgr2=0x7fd72800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:46.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.670+0000 7fd742a00640 1 --2- 172.21.15.45:0/3053603439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7280051c0 0x7fd72800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:46.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.670+0000 7fd742a00640 1 -- 172.21.15.45:0/3053603439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd728004800 msgr2=0x7fd728004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:46.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.670+0000 7fd73be00640 1 --2- 172.21.15.45:0/3053603439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd728004800 0x7fd728004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:46.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.670+0000 7fd742a00640 1 --2- 172.21.15.45:0/3053603439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd728004800 0x7fd728004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:46.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.670+0000 7fd742a00640 1 -- 172.21.15.45:0/3053603439 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd728003520 con 0x7fd7280039b0 2024-04-16T16:22:46.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.670+0000 7fd743400640 1 -- 172.21.15.45:0/3053603439 reap_dead start 2024-04-16T16:22:46.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.671+0000 7fd742a00640 1 --2- 172.21.15.45:0/3053603439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7280039b0 0x7fd728003df0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fd73400c240 tx=0x7fd728005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0857d1335c672d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:46.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.672+0000 7fd742000640 1 -- 172.21.15.45:0/3053603439 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd734007d10 con 0x7fd7280039b0 2024-04-16T16:22:46.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.672+0000 7fd742000640 1 -- 172.21.15.45:0/3053603439 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd734030070 con 0x7fd7280039b0 2024-04-16T16:22:46.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.672+0000 7fd742000640 1 -- 172.21.15.45:0/3053603439 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd734005d50 con 0x7fd7280039b0 2024-04-16T16:22:46.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.672+0000 7fd749e00640 1 -- 172.21.15.45:0/3053603439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7280039b0 msgr2=0x7fd728003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:46.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.672+0000 7fd749e00640 1 --2- 172.21.15.45:0/3053603439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7280039b0 0x7fd728003df0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fd73400c240 tx=0x7fd728005830 comp rx=0 tx=0).stop 2024-04-16T16:22:46.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.672+0000 7fd749e00640 1 -- 172.21.15.45:0/3053603439 shutdown_connections 2024-04-16T16:22:46.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.672+0000 7fd749e00640 1 --2- 172.21.15.45:0/3053603439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7280039b0 0x7fd728003df0 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:46.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.672+0000 7fd749e00640 1 -- 172.21.15.45:0/3053603439 >> 172.21.15.45:0/3053603439 conn(0x7fd7440b2750 msgr2=0x7fd7440b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:46.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.673+0000 7fd749e00640 1 -- 172.21.15.45:0/3053603439 shutdown_connections 2024-04-16T16:22:46.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.673+0000 7fd749e00640 1 -- 172.21.15.45:0/3053603439 wait complete. 2024-04-16T16:22:46.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.674+0000 7fd749e00640 1 Processor -- start 2024-04-16T16:22:46.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.674+0000 7fd749e00640 1 -- start start 2024-04-16T16:22:46.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.675+0000 7fd749e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7280039b0 0x7fd7440cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:46.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.675+0000 7fd749e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd744134340 0x7fd7441347a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:46.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.675+0000 7fd749e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7441409b0 0x7fd744140e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:46.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.675+0000 7fd749e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd7440afb10 con 0x7fd7441409b0 2024-04-16T16:22:46.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.676+0000 7fd749e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd7440af830 con 0x7fd7280039b0 2024-04-16T16:22:46.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.676+0000 7fd749e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd744072380 con 0x7fd744134340 2024-04-16T16:22:46.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.676+0000 7fd743400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7441409b0 0x7fd744140e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:46.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.676+0000 7fd743400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7441409b0 0x7fd744140e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33010/0 (socket says 172.21.15.45:33010) 2024-04-16T16:22:46.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.676+0000 7fd743400640 1 -- 172.21.15.45:0/3871874335 learned_addr learned my addr 172.21.15.45:0/3871874335 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:46.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.676+0000 7fd742a00640 1 --2- 172.21.15.45:0/3871874335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7280039b0 0x7fd7440cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:46.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.676+0000 7fd743400640 1 -- 172.21.15.45:0/3871874335 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd744134340 msgr2=0x7fd7441347a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:46.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.676+0000 7fd743400640 1 --2- 172.21.15.45:0/3871874335 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd744134340 0x7fd7441347a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:46.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.676+0000 7fd743400640 1 -- 172.21.15.45:0/3871874335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7280039b0 msgr2=0x7fd7440cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:46.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.676+0000 7fd743400640 1 --2- 172.21.15.45:0/3871874335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7280039b0 0x7fd7440cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:46.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.676+0000 7fd743400640 1 -- 172.21.15.45:0/3871874335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd744131e80 con 0x7fd7441409b0 2024-04-16T16:22:46.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.677+0000 7fd743400640 1 --2- 172.21.15.45:0/3871874335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7441409b0 0x7fd744140e30 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fd7440b36f0 tx=0x7fd73c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:46.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.677+0000 7fd73b400640 1 -- 172.21.15.45:0/3871874335 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73c018020 con 0x7fd7441409b0 2024-04-16T16:22:46.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.677+0000 7fd73b400640 1 -- 172.21.15.45:0/3871874335 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd73c00bc80 con 0x7fd7441409b0 2024-04-16T16:22:46.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.677+0000 7fd749e00640 1 -- 172.21.15.45:0/3871874335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd744132160 con 0x7fd7441409b0 2024-04-16T16:22:46.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.677+0000 7fd73b400640 1 -- 172.21.15.45:0/3871874335 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73c014640 con 0x7fd7441409b0 2024-04-16T16:22:46.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.677+0000 7fd749e00640 1 -- 172.21.15.45:0/3871874335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd744134ce0 con 0x7fd7441409b0 2024-04-16T16:22:46.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.678+0000 7fd749e00640 1 -- 172.21.15.45:0/3871874335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd744075780 con 0x7fd7441409b0 2024-04-16T16:22:46.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.679+0000 7fd73b400640 1 -- 172.21.15.45:0/3871874335 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fd73c01b020 con 0x7fd7441409b0 2024-04-16T16:22:46.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.680+0000 7fd73b400640 1 --2- 172.21.15.45:0/3871874335 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd72006ca10 0x7fd72006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:46.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.680+0000 7fd73b400640 1 -- 172.21.15.45:0/3871874335 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 5709+0+0 (secure 0 0 0) 0x7fd73c08c560 con 0x7fd7441409b0 2024-04-16T16:22:46.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.681+0000 7fd742a00640 1 --2- 172.21.15.45:0/3871874335 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd72006ca10 0x7fd72006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.682+0000 7fd742a00640 1 --2- 172.21.15.45:0/3871874335 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd72006ca10 0x7fd72006eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd73400c240 tx=0x7fd734039040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:46.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.685+0000 7fd73b400640 1 -- 172.21.15.45:0/3871874335 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd73c022020 con 0x7fd7441409b0 2024-04-16T16:22:46.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.890+0000 7fd749e00640 1 -- 172.21.15.45:0/3871874335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd744070220 con 0x7fd7441409b0 2024-04-16T16:22:46.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.892+0000 7fd73b400640 1 -- 172.21.15.45:0/3871874335 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) ==== 74+0+16684 (secure 0 0 0) 0x7fd73c0122c0 con 0x7fd7441409b0 2024-04-16T16:22:46.891 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:46.891 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":155,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:22:45.927111+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:22:43.864367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-16T16:22:44.899356+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:22:46.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.895+0000 7fd749e00640 1 -- 172.21.15.45:0/3871874335 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd72006ca10 msgr2=0x7fd72006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:46.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.895+0000 7fd749e00640 1 --2- 172.21.15.45:0/3871874335 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd72006ca10 0x7fd72006eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd73400c240 tx=0x7fd734039040 comp rx=0 tx=0).stop 2024-04-16T16:22:46.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.895+0000 7fd749e00640 1 -- 172.21.15.45:0/3871874335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7441409b0 msgr2=0x7fd744140e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:46.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.895+0000 7fd749e00640 1 --2- 172.21.15.45:0/3871874335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7441409b0 0x7fd744140e30 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fd7440b36f0 tx=0x7fd73c00b480 comp rx=0 tx=0).stop 2024-04-16T16:22:46.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.895+0000 7fd749e00640 1 -- 172.21.15.45:0/3871874335 shutdown_connections 2024-04-16T16:22:46.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.895+0000 7fd749e00640 1 --2- 172.21.15.45:0/3871874335 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd72006ca10 0x7fd72006eed0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:46.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.895+0000 7fd749e00640 1 --2- 172.21.15.45:0/3871874335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7441409b0 0x7fd744140e30 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:46.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.895+0000 7fd749e00640 1 --2- 172.21.15.45:0/3871874335 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd744134340 0x7fd7441347a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:46.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.895+0000 7fd749e00640 1 --2- 172.21.15.45:0/3871874335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7280039b0 0x7fd7440cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:46.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.895+0000 7fd749e00640 1 -- 172.21.15.45:0/3871874335 >> 172.21.15.45:0/3871874335 conn(0x7fd7440b2750 msgr2=0x7fd7440b06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:46.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.895+0000 7fd749e00640 1 -- 172.21.15.45:0/3871874335 shutdown_connections 2024-04-16T16:22:46.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:46.895+0000 7fd749e00640 1 -- 172.21.15.45:0/3871874335 wait complete. 2024-04-16T16:22:46.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:22:47.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.132+0000 7fd0be000640 1 Processor -- start 2024-04-16T16:22:47.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.132+0000 7fd0be000640 1 -- start start 2024-04-16T16:22:47.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.133+0000 7fd0be000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0b81608a0 0x7fd0b8160d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:47.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.134+0000 7fd0be000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0b815d360 con 0x7fd0b8146220 2024-04-16T16:22:47.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.134+0000 7fd0be000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0b815d640 con 0x7fd0b815ccb0 2024-04-16T16:22:47.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.134+0000 7fd0be000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd0b815d130 con 0x7fd0b81608a0 2024-04-16T16:22:47.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.134+0000 7fd0bcc00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd0b8146220 0x7fd0b806f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38406/0 (socket says 172.21.15.45:38406) 2024-04-16T16:22:47.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.134+0000 7fd0bcc00640 1 -- 172.21.15.45:0/2024614478 learned_addr learned my addr 172.21.15.45:0/2024614478 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:47.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.134+0000 7fd0b7400640 1 -- 172.21.15.45:0/2024614478 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3306680569 0 0) 0x7fd0b815d360 con 0x7fd0b8146220 2024-04-16T16:22:47.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.134+0000 7fd0bd600640 1 --2- 172.21.15.45:0/2024614478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0b81608a0 0x7fd0b8160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:47.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.134+0000 7fd0b7400640 1 -- 172.21.15.45:0/2024614478 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0a0003520 con 0x7fd0b8146220 2024-04-16T16:22:47.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.135+0000 7fd0b7400640 1 -- 172.21.15.45:0/2024614478 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 676027999 0 0) 0x7fd0a0003520 con 0x7fd0b8146220 2024-04-16T16:22:47.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.135+0000 7fd0b7400640 1 -- 172.21.15.45:0/2024614478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0b81608a0 msgr2=0x7fd0b8160d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:47.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.135+0000 7fd0b7400640 1 --2- 172.21.15.45:0/2024614478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0b81608a0 0x7fd0b8160d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:47.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.135+0000 7fd0b7400640 1 -- 172.21.15.45:0/2024614478 >> v1:172.21.15.103:6789/0 conn(0x7fd0b815ccb0 legacy=0x7fd0b806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:47.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.135+0000 7fd0b7400640 1 -- 172.21.15.45:0/2024614478 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b815d920 con 0x7fd0b8146220 2024-04-16T16:22:47.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.135+0000 7fd0b7400640 1 -- 172.21.15.45:0/2024614478 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fd0ac001670 con 0x7fd0b8146220 2024-04-16T16:22:47.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.135+0000 7fd0b7400640 1 -- 172.21.15.45:0/2024614478 >> v1:172.21.15.45:6789/0 conn(0x7fd0b8146220 legacy=0x7fd0b806f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:47.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.135+0000 7fd0b7400640 1 --2- 172.21.15.45:0/2024614478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0b81608a0 0x7fd0a00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:47.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.135+0000 7fd0b7400640 1 --2- 172.21.15.45:0/2024614478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0a0003fb0 0x7fd0a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:47.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.135+0000 7fd0b7400640 1 --2- 172.21.15.45:0/2024614478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0a0004d40 0x7fd0a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:47.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.135+0000 7fd0b7400640 1 -- 172.21.15.45:0/2024614478 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a000bad0 con 0x7fd0a0003fb0 2024-04-16T16:22:47.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.135+0000 7fd0b7400640 1 -- 172.21.15.45:0/2024614478 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a000bc80 con 0x7fd0a0004d40 2024-04-16T16:22:47.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.135+0000 7fd0b7400640 1 -- 172.21.15.45:0/2024614478 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd0a000be80 con 0x7fd0b81608a0 2024-04-16T16:22:47.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.135+0000 7fd0b7e00640 1 --2- 172.21.15.45:0/2024614478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0a0003fb0 0x7fd0a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:47.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.136+0000 7fd0bd600640 1 --2- 172.21.15.45:0/2024614478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0a0004d40 0x7fd0a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:47.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.136+0000 7fd0bcc00640 1 --2- 172.21.15.45:0/2024614478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0b81608a0 0x7fd0a00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:47.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.136+0000 7fd0b7e00640 1 -- 172.21.15.45:0/2024614478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0b81608a0 msgr2=0x7fd0a00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:47.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.136+0000 7fd0b7e00640 1 --2- 172.21.15.45:0/2024614478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0b81608a0 0x7fd0a00039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:47.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.136+0000 7fd0b7e00640 1 -- 172.21.15.45:0/2024614478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0a0004d40 msgr2=0x7fd0a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:47.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.136+0000 7fd0b7e00640 1 --2- 172.21.15.45:0/2024614478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0a0004d40 0x7fd0a00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:47.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.136+0000 7fd0b7e00640 1 -- 172.21.15.45:0/2024614478 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a0003520 con 0x7fd0a0003fb0 2024-04-16T16:22:47.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.137+0000 7fd0b7e00640 1 --2- 172.21.15.45:0/2024614478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0a0003fb0 0x7fd0a0004800 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4001750 tx=0x7fd0a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4851431ffcd81807 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:47.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.137+0000 7fd0b7400640 1 -- 172.21.15.45:0/2024614478 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4001370 con 0x7fd0a0003fb0 2024-04-16T16:22:47.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.137+0000 7fd0b7400640 1 -- 172.21.15.45:0/2024614478 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0a40052d0 con 0x7fd0a0003fb0 2024-04-16T16:22:47.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.137+0000 7fd0b7400640 1 -- 172.21.15.45:0/2024614478 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4003830 con 0x7fd0a0003fb0 2024-04-16T16:22:47.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.138+0000 7fd0be000640 1 -- 172.21.15.45:0/2024614478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0a0003fb0 msgr2=0x7fd0a0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:47.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.138+0000 7fd0be000640 1 --2- 172.21.15.45:0/2024614478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0a0003fb0 0x7fd0a0004800 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4001750 tx=0x7fd0a0005830 comp rx=0 tx=0).stop 2024-04-16T16:22:47.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.138+0000 7fd0bd600640 1 -- 172.21.15.45:0/2024614478 reap_dead start 2024-04-16T16:22:47.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.138+0000 7fd0be000640 1 -- 172.21.15.45:0/2024614478 shutdown_connections 2024-04-16T16:22:47.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.138+0000 7fd0be000640 1 -- 172.21.15.45:0/2024614478 >> 172.21.15.45:0/2024614478 conn(0x7fd0b806b930 msgr2=0x7fd0b806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:47.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.138+0000 7fd0be000640 1 -- 172.21.15.45:0/2024614478 shutdown_connections 2024-04-16T16:22:47.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.138+0000 7fd0be000640 1 -- 172.21.15.45:0/2024614478 wait complete. 2024-04-16T16:22:47.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.140+0000 7fd0be000640 1 Processor -- start 2024-04-16T16:22:47.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.140+0000 7fd0be000640 1 -- start start 2024-04-16T16:22:47.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.141+0000 7fd0be000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0a0003fb0 0x7fd0b81f8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:47.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.141+0000 7fd0be000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b815bbf0 0x7fd0b81f4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:47.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.141+0000 7fd0be000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0b81f4ff0 0x7fd0b813c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:47.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.141+0000 7fd0be000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b815dc00 con 0x7fd0b815bbf0 2024-04-16T16:22:47.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.141+0000 7fd0be000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b8160410 con 0x7fd0a0003fb0 2024-04-16T16:22:47.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.141+0000 7fd0be000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd0b8160d30 con 0x7fd0b81f4ff0 2024-04-16T16:22:47.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.141+0000 7fd0b7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b815bbf0 0x7fd0b81f4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:47.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.141+0000 7fd0b7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b815bbf0 0x7fd0b81f4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33032/0 (socket says 172.21.15.45:33032) 2024-04-16T16:22:47.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.141+0000 7fd0b7e00640 1 -- 172.21.15.45:0/985068723 learned_addr learned my addr 172.21.15.45:0/985068723 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:47.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.141+0000 7fd0bcc00640 1 --2- 172.21.15.45:0/985068723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0a0003fb0 0x7fd0b81f8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:47.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.141+0000 7fd0b7e00640 1 -- 172.21.15.45:0/985068723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0b81f4ff0 msgr2=0x7fd0b813c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:47.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.141+0000 7fd0b7e00640 1 --2- 172.21.15.45:0/985068723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0b81f4ff0 0x7fd0b813c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:47.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.141+0000 7fd0b7e00640 1 -- 172.21.15.45:0/985068723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0a0003fb0 msgr2=0x7fd0b81f8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:47.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.141+0000 7fd0b7e00640 1 --2- 172.21.15.45:0/985068723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0a0003fb0 0x7fd0b81f8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:47.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.141+0000 7fd0b7e00640 1 -- 172.21.15.45:0/985068723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b80cf1c0 con 0x7fd0b815bbf0 2024-04-16T16:22:47.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.142+0000 7fd0b7e00640 1 --2- 172.21.15.45:0/985068723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b815bbf0 0x7fd0b81f4ab0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4008fd0 tx=0x7fd0a4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:47.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.142+0000 7fd0b4c00640 1 -- 172.21.15.45:0/985068723 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4041070 con 0x7fd0b815bbf0 2024-04-16T16:22:47.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.142+0000 7fd0b4c00640 1 -- 172.21.15.45:0/985068723 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0a4005480 con 0x7fd0b815bbf0 2024-04-16T16:22:47.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.142+0000 7fd0b4c00640 1 -- 172.21.15.45:0/985068723 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4001370 con 0x7fd0b815bbf0 2024-04-16T16:22:47.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.142+0000 7fd0be000640 1 -- 172.21.15.45:0/985068723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0b815c070 con 0x7fd0b815bbf0 2024-04-16T16:22:47.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.142+0000 7fd0be000640 1 -- 172.21.15.45:0/985068723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b81f5880 con 0x7fd0b815bbf0 2024-04-16T16:22:47.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.144+0000 7fd0be000640 1 -- 172.21.15.45:0/985068723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0b8075780 con 0x7fd0b815bbf0 2024-04-16T16:22:47.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.144+0000 7fd0b4c00640 1 -- 172.21.15.45:0/985068723 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fd0a4044070 con 0x7fd0b815bbf0 2024-04-16T16:22:47.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.145+0000 7fd0b4c00640 1 --2- 172.21.15.45:0/985068723 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd09806ca40 0x7fd09806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:47.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.145+0000 7fd0b4c00640 1 -- 172.21.15.45:0/985068723 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 5709+0+0 (secure 0 0 0) 0x7fd0a40ada10 con 0x7fd0b815bbf0 2024-04-16T16:22:47.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.146+0000 7fd0bcc00640 1 --2- 172.21.15.45:0/985068723 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd09806ca40 0x7fd09806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:47.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.147+0000 7fd0bcc00640 1 --2- 172.21.15.45:0/985068723 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd09806ca40 0x7fd09806ef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac001ea0 tx=0x7fd0ac009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:47.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.150+0000 7fd0b4c00640 1 -- 172.21.15.45:0/985068723 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd0a407bb90 con 0x7fd0b815bbf0 2024-04-16T16:22:47.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.346+0000 7fd0be000640 1 -- 172.21.15.45:0/985068723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0b80cf9d0 con 0x7fd0b815bbf0 2024-04-16T16:22:47.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.348+0000 7fd0b4c00640 1 -- 172.21.15.45:0/985068723 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) ==== 74+0+16684 (secure 0 0 0) 0x7fd0a407b490 con 0x7fd0b815bbf0 2024-04-16T16:22:47.347 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:47.347 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":155,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:22:45.927111+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:22:43.864367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-16T16:22:44.899356+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:22:47.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.351+0000 7fd0be000640 1 -- 172.21.15.45:0/985068723 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd09806ca40 msgr2=0x7fd09806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:47.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.351+0000 7fd0be000640 1 --2- 172.21.15.45:0/985068723 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd09806ca40 0x7fd09806ef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac001ea0 tx=0x7fd0ac009040 comp rx=0 tx=0).stop 2024-04-16T16:22:47.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.351+0000 7fd0be000640 1 -- 172.21.15.45:0/985068723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b815bbf0 msgr2=0x7fd0b81f4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:47.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.351+0000 7fd0be000640 1 --2- 172.21.15.45:0/985068723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b815bbf0 0x7fd0b81f4ab0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4008fd0 tx=0x7fd0a4001120 comp rx=0 tx=0).stop 2024-04-16T16:22:47.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.351+0000 7fd0be000640 1 -- 172.21.15.45:0/985068723 shutdown_connections 2024-04-16T16:22:47.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.351+0000 7fd0be000640 1 --2- 172.21.15.45:0/985068723 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fd09806ca40 0x7fd09806ef00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:47.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.351+0000 7fd0be000640 1 --2- 172.21.15.45:0/985068723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0b81f4ff0 0x7fd0b813c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:47.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.351+0000 7fd0be000640 1 --2- 172.21.15.45:0/985068723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b815bbf0 0x7fd0b81f4ab0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:47.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.351+0000 7fd0be000640 1 --2- 172.21.15.45:0/985068723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0a0003fb0 0x7fd0b81f8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:47.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.351+0000 7fd0be000640 1 -- 172.21.15.45:0/985068723 >> 172.21.15.45:0/985068723 conn(0x7fd0b806b930 msgr2=0x7fd0b8071f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:47.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.351+0000 7fd0be000640 1 -- 172.21.15.45:0/985068723 shutdown_connections 2024-04-16T16:22:47.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.351+0000 7fd0be000640 1 -- 172.21.15.45:0/985068723 wait complete. 2024-04-16T16:22:47.369 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-16T16:22:47.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.579+0000 7f9ce7200640 1 Processor -- start 2024-04-16T16:22:47.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.580+0000 7f9ce7200640 1 -- start start 2024-04-16T16:22:47.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.581+0000 7f9ce7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ce80d18f0 0x7f9ce80ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:47.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.581+0000 7f9ce7200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9ce80d1cd0 0x7f9ce8162dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:47.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.581+0000 7f9ce7200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ce80b6810 con 0x7f9ce80b6310 2024-04-16T16:22:47.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.581+0000 7f9ce7200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9ce80d29f0 con 0x7f9ce80d1cd0 2024-04-16T16:22:47.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.581+0000 7f9ce7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ce80b6af0 con 0x7f9ce80d18f0 2024-04-16T16:22:47.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.582+0000 7f9ce5e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ce80d18f0 0x7f9ce80ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:47.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.582+0000 7f9ce6800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9ce80d1cd0 0x7f9ce8162dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:47.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.582+0000 7f9ce6800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9ce80d1cd0 0x7f9ce8162dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:55836/0 (socket says 172.21.15.45:55836) 2024-04-16T16:22:47.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.582+0000 7f9ce5e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ce80d18f0 0x7f9ce80ba220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:48408/0 (socket says 172.21.15.45:48408) 2024-04-16T16:22:47.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.582+0000 7f9ce6800640 1 -- 172.21.15.45:0/3605669721 learned_addr learned my addr 172.21.15.45:0/3605669721 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:47.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.583+0000 7f9ce5e00640 1 -- 172.21.15.45:0/3605669721 >> v2:172.21.15.103:3300/0 conn(0x7f9ce80d1cd0 msgr2=0x7f9ce8162dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:47.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.583+0000 7f9ce5e00640 1 --2- 172.21.15.45:0/3605669721 >> v2:172.21.15.103:3300/0 conn(0x7f9ce80d1cd0 0x7f9ce8162dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:47.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.583+0000 7f9ce5e00640 1 -- 172.21.15.45:0/3605669721 >> v1:172.21.15.103:6789/0 conn(0x7f9ce80b6310 legacy=0x7f9ce80d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:47.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.583+0000 7f9ce5e00640 1 -- 172.21.15.45:0/3605669721 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce80b6dd0 con 0x7f9ce80d18f0 2024-04-16T16:22:47.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.583+0000 7f9ce6800640 1 --2- 172.21.15.45:0/3605669721 >> v2:172.21.15.103:3300/0 conn(0x7f9ce80d1cd0 0x7f9ce8162dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T16:22:47.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.584+0000 7f9ce5e00640 1 --2- 172.21.15.45:0/3605669721 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ce80d18f0 0x7f9ce80ba220 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009d90 tx=0x7f9cd802be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9322d694dbe2a88a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:47.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.585+0000 7f9ce4a00640 1 -- 172.21.15.45:0/3605669721 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8039070 con 0x7f9ce80d18f0 2024-04-16T16:22:47.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.585+0000 7f9ce4a00640 1 -- 172.21.15.45:0/3605669721 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cd802cbd0 con 0x7f9ce80d18f0 2024-04-16T16:22:47.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.585+0000 7f9ce4a00640 1 -- 172.21.15.45:0/3605669721 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8034440 con 0x7f9ce80d18f0 2024-04-16T16:22:47.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.585+0000 7f9ce7200640 1 -- 172.21.15.45:0/3605669721 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ce80d18f0 msgr2=0x7f9ce80ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:47.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.585+0000 7f9ce7200640 1 --2- 172.21.15.45:0/3605669721 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ce80d18f0 0x7f9ce80ba220 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009d90 tx=0x7f9cd802be20 comp rx=0 tx=0).stop 2024-04-16T16:22:47.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.585+0000 7f9ce7200640 1 -- 172.21.15.45:0/3605669721 shutdown_connections 2024-04-16T16:22:47.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.585+0000 7f9ce7200640 1 --2- 172.21.15.45:0/3605669721 >> v2:172.21.15.103:3300/0 conn(0x7f9ce80d1cd0 0x7f9ce8162dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:47.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.585+0000 7f9ce7200640 1 --2- 172.21.15.45:0/3605669721 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ce80d18f0 0x7f9ce80ba220 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:47.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.585+0000 7f9ce7200640 1 -- 172.21.15.45:0/3605669721 >> 172.21.15.45:0/3605669721 conn(0x7f9ce806cf90 msgr2=0x7f9ce806d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:47.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.586+0000 7f9ce7200640 1 -- 172.21.15.45:0/3605669721 shutdown_connections 2024-04-16T16:22:47.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.586+0000 7f9ce7200640 1 -- 172.21.15.45:0/3605669721 wait complete. 2024-04-16T16:22:47.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.587+0000 7f9ce7200640 1 Processor -- start 2024-04-16T16:22:47.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.587+0000 7f9ce7200640 1 -- start start 2024-04-16T16:22:47.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.588+0000 7f9ce7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce80b6310 0x7f9ce80cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:47.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.588+0000 7f9ce7200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ce80d18f0 0x7f9ce814c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:47.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.588+0000 7f9ce7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ce80d1cd0 0x7f9ce8158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:47.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.588+0000 7f9ce7200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce8163300 con 0x7f9ce80d18f0 2024-04-16T16:22:47.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.588+0000 7f9ce7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce80ba760 con 0x7f9ce80b6310 2024-04-16T16:22:47.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.588+0000 7f9ce7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ce814cb70 con 0x7f9ce80d1cd0 2024-04-16T16:22:47.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.589+0000 7f9ce5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ce80d18f0 0x7f9ce814c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:47.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.589+0000 7f9ce5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ce80d18f0 0x7f9ce814c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33046/0 (socket says 172.21.15.45:33046) 2024-04-16T16:22:47.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.589+0000 7f9ce5400640 1 -- 172.21.15.45:0/2668590474 learned_addr learned my addr 172.21.15.45:0/2668590474 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:47.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.589+0000 7f9ce5e00640 1 --2- 172.21.15.45:0/2668590474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce80b6310 0x7f9ce80cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:47.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.589+0000 7f9ce6800640 1 --2- 172.21.15.45:0/2668590474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ce80d1cd0 0x7f9ce8158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:47.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.589+0000 7f9ce5400640 1 -- 172.21.15.45:0/2668590474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ce80d1cd0 msgr2=0x7f9ce8158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:47.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.589+0000 7f9ce5400640 1 --2- 172.21.15.45:0/2668590474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ce80d1cd0 0x7f9ce8158c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:47.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.589+0000 7f9ce5400640 1 -- 172.21.15.45:0/2668590474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce80b6310 msgr2=0x7f9ce80cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:47.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.589+0000 7f9ce5400640 1 --2- 172.21.15.45:0/2668590474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce80b6310 0x7f9ce80cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:47.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.589+0000 7f9ce5400640 1 -- 172.21.15.45:0/2668590474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce8159a40 con 0x7f9ce80d18f0 2024-04-16T16:22:47.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.589+0000 7f9ce5e00640 1 --2- 172.21.15.45:0/2668590474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce80b6310 0x7f9ce80cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:47.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.590+0000 7f9ce5400640 1 --2- 172.21.15.45:0/2668590474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ce80d18f0 0x7f9ce814c630 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f9cd000c430 tx=0x7f9cd000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:47.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.590+0000 7f9cde000640 1 -- 172.21.15.45:0/2668590474 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd00052d0 con 0x7f9ce80d18f0 2024-04-16T16:22:47.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.590+0000 7f9cde000640 1 -- 172.21.15.45:0/2668590474 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cd0005490 con 0x7f9ce80d18f0 2024-04-16T16:22:47.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.590+0000 7f9ce7200640 1 -- 172.21.15.45:0/2668590474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ce8159480 con 0x7f9ce80d18f0 2024-04-16T16:22:47.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.590+0000 7f9cde000640 1 -- 172.21.15.45:0/2668590474 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0007260 con 0x7f9ce80d18f0 2024-04-16T16:22:47.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.590+0000 7f9ce7200640 1 -- 172.21.15.45:0/2668590474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ce80cba70 con 0x7f9ce80d18f0 2024-04-16T16:22:47.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.591+0000 7f9ce7200640 1 -- 172.21.15.45:0/2668590474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ce8075780 con 0x7f9ce80d18f0 2024-04-16T16:22:47.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.592+0000 7f9cde000640 1 -- 172.21.15.45:0/2668590474 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f9cd0007400 con 0x7f9ce80d18f0 2024-04-16T16:22:47.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.593+0000 7f9cde000640 1 --2- 172.21.15.45:0/2668590474 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f9cc406ca10 0x7f9cc406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:47.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.593+0000 7f9cde000640 1 -- 172.21.15.45:0/2668590474 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f9cd008b360 con 0x7f9ce80d18f0 2024-04-16T16:22:47.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.594+0000 7f9ce5e00640 1 --2- 172.21.15.45:0/2668590474 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f9cc406ca10 0x7f9cc406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:47.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.595+0000 7f9ce5e00640 1 --2- 172.21.15.45:0/2668590474 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f9cc406ca10 0x7f9cc406eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8036000 tx=0x7f9cd8030000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:47.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.598+0000 7f9cde000640 1 -- 172.21.15.45:0/2668590474 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9cd0059550 con 0x7f9ce80d18f0 2024-04-16T16:22:47.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:47.842+0000 7f9ce7200640 1 -- 172.21.15.45:0/2668590474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f9ce80d25a0 con 0x7f9ce80d18f0 2024-04-16T16:22:48.443 INFO:tasks.ceph.mds.c.smithi045.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: In function 'void MDLog::trim()' thread 7f5bbd600640 time 2024-04-16T16:22:48.445030+0000 2024-04-16T16:22:48.443 INFO:tasks.ceph.mds.c.smithi045.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: 688: FAILED ceph_assert(num_total_segments >= num_replayed_segments) 2024-04-16T16:22:48.444 INFO:tasks.ceph.mds.c.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:22:48.444 INFO:tasks.ceph.mds.c.smithi045.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f5bc9f81f4c] 2024-04-16T16:22:48.444 INFO:tasks.ceph.mds.c.smithi045.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7f5bc9f8210a] 2024-04-16T16:22:48.444 INFO:tasks.ceph.mds.c.smithi045.stderr: 3: (MDLog::trim()+0xf2a) [0x56381c7d31aa] 2024-04-16T16:22:48.444 INFO:tasks.ceph.mds.c.smithi045.stderr: 4: (MDLog::log_trim_upkeep()+0x339) [0x56381c7d3559] 2024-04-16T16:22:48.444 INFO:tasks.ceph.mds.c.smithi045.stderr: 5: /lib64/libstdc++.so.6(+0xdb924) [0x7f5bc9adb924] 2024-04-16T16:22:48.444 INFO:tasks.ceph.mds.c.smithi045.stderr: 6: /lib64/libc.so.6(+0x9f802) [0x7f5bc969f802] 2024-04-16T16:22:48.444 INFO:tasks.ceph.mds.c.smithi045.stderr: 7: /lib64/libc.so.6(+0x3f450) [0x7f5bc963f450] 2024-04-16T16:22:48.444 INFO:tasks.ceph.mds.c.smithi045.stderr:*** Caught signal (Aborted) ** 2024-04-16T16:22:48.445 INFO:tasks.ceph.mds.c.smithi045.stderr: in thread 7f5bbd600640 thread_name:ms_dispatch 2024-04-16T16:22:48.445 INFO:tasks.ceph.mds.c.smithi045.stderr:2024-04-16T16:22:48.445+0000 7f5bbd600640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: In function 'void MDLog::trim()' thread 7f5bbd600640 time 2024-04-16T16:22:48.445030+0000 2024-04-16T16:22:48.445 INFO:tasks.ceph.mds.c.smithi045.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: 688: FAILED ceph_assert(num_total_segments >= num_replayed_segments) 2024-04-16T16:22:48.445 INFO:tasks.ceph.mds.c.smithi045.stderr: 2024-04-16T16:22:48.445 INFO:tasks.ceph.mds.c.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:22:48.445 INFO:tasks.ceph.mds.c.smithi045.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f5bc9f81f4c] 2024-04-16T16:22:48.445 INFO:tasks.ceph.mds.c.smithi045.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7f5bc9f8210a] 2024-04-16T16:22:48.445 INFO:tasks.ceph.mds.c.smithi045.stderr: 3: (MDLog::trim()+0xf2a) [0x56381c7d31aa] 2024-04-16T16:22:48.445 INFO:tasks.ceph.mds.c.smithi045.stderr: 4: (MDLog::log_trim_upkeep()+0x339) [0x56381c7d3559] 2024-04-16T16:22:48.445 INFO:tasks.ceph.mds.c.smithi045.stderr: 5: /lib64/libstdc++.so.6(+0xdb924) [0x7f5bc9adb924] 2024-04-16T16:22:48.445 INFO:tasks.ceph.mds.c.smithi045.stderr: 6: /lib64/libc.so.6(+0x9f802) [0x7f5bc969f802] 2024-04-16T16:22:48.445 INFO:tasks.ceph.mds.c.smithi045.stderr: 7: /lib64/libc.so.6(+0x3f450) [0x7f5bc963f450] 2024-04-16T16:22:48.445 INFO:tasks.ceph.mds.c.smithi045.stderr: 2024-04-16T16:22:48.446 INFO:tasks.ceph.mds.c.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:22:48.446 INFO:tasks.ceph.mds.c.smithi045.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f5bc9654db0] 2024-04-16T16:22:48.446 INFO:tasks.ceph.mds.c.smithi045.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f5bc96a154c] 2024-04-16T16:22:48.446 INFO:tasks.ceph.mds.c.smithi045.stderr: 3: raise() 2024-04-16T16:22:48.446 INFO:tasks.ceph.mds.c.smithi045.stderr: 4: abort() 2024-04-16T16:22:48.446 INFO:tasks.ceph.mds.c.smithi045.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f5bc9f81fa6] 2024-04-16T16:22:48.446 INFO:tasks.ceph.mds.c.smithi045.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7f5bc9f8210a] 2024-04-16T16:22:48.446 INFO:tasks.ceph.mds.c.smithi045.stderr: 7: (MDLog::trim()+0xf2a) [0x56381c7d31aa] 2024-04-16T16:22:48.446 INFO:tasks.ceph.mds.c.smithi045.stderr: 8: (MDLog::log_trim_upkeep()+0x339) [0x56381c7d3559] 2024-04-16T16:22:48.446 INFO:tasks.ceph.mds.c.smithi045.stderr: 9: /lib64/libstdc++.so.6(+0xdb924) [0x7f5bc9adb924] 2024-04-16T16:22:48.447 INFO:tasks.ceph.mds.c.smithi045.stderr: 10: /lib64/libc.so.6(+0x9f802) [0x7f5bc969f802] 2024-04-16T16:22:48.447 INFO:tasks.ceph.mds.c.smithi045.stderr: 11: /lib64/libc.so.6(+0x3f450) [0x7f5bc963f450] 2024-04-16T16:22:48.447 INFO:tasks.ceph.mds.c.smithi045.stderr:2024-04-16T16:22:48.448+0000 7f5bbd600640 -1 *** Caught signal (Aborted) ** 2024-04-16T16:22:48.447 INFO:tasks.ceph.mds.c.smithi045.stderr: in thread 7f5bbd600640 thread_name:ms_dispatch 2024-04-16T16:22:48.447 INFO:tasks.ceph.mds.c.smithi045.stderr: 2024-04-16T16:22:48.447 INFO:tasks.ceph.mds.c.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:22:48.447 INFO:tasks.ceph.mds.c.smithi045.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f5bc9654db0] 2024-04-16T16:22:48.447 INFO:tasks.ceph.mds.c.smithi045.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f5bc96a154c] 2024-04-16T16:22:48.447 INFO:tasks.ceph.mds.c.smithi045.stderr: 3: raise() 2024-04-16T16:22:48.447 INFO:tasks.ceph.mds.c.smithi045.stderr: 4: abort() 2024-04-16T16:22:48.447 INFO:tasks.ceph.mds.c.smithi045.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f5bc9f81fa6] 2024-04-16T16:22:48.448 INFO:tasks.ceph.mds.c.smithi045.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7f5bc9f8210a] 2024-04-16T16:22:48.448 INFO:tasks.ceph.mds.c.smithi045.stderr: 7: (MDLog::trim()+0xf2a) [0x56381c7d31aa] 2024-04-16T16:22:48.448 INFO:tasks.ceph.mds.c.smithi045.stderr: 8: (MDLog::log_trim_upkeep()+0x339) [0x56381c7d3559] 2024-04-16T16:22:48.448 INFO:tasks.ceph.mds.c.smithi045.stderr: 9: /lib64/libstdc++.so.6(+0xdb924) [0x7f5bc9adb924] 2024-04-16T16:22:48.448 INFO:tasks.ceph.mds.c.smithi045.stderr: 10: /lib64/libc.so.6(+0x9f802) [0x7f5bc969f802] 2024-04-16T16:22:48.448 INFO:tasks.ceph.mds.c.smithi045.stderr: 11: /lib64/libc.so.6(+0x3f450) [0x7f5bc963f450] 2024-04-16T16:22:48.448 INFO:tasks.ceph.mds.c.smithi045.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-16T16:22:48.448 INFO:tasks.ceph.mds.c.smithi045.stderr: 2024-04-16T16:22:48.460 INFO:tasks.ceph.mds.c.smithi045.stderr: -1> 2024-04-16T16:22:48.445+0000 7f5bbd600640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: In function 'void MDLog::trim()' thread 7f5bbd600640 time 2024-04-16T16:22:48.445030+0000 2024-04-16T16:22:48.460 INFO:tasks.ceph.mds.c.smithi045.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: 688: FAILED ceph_assert(num_total_segments >= num_replayed_segments) 2024-04-16T16:22:48.460 INFO:tasks.ceph.mds.c.smithi045.stderr: 2024-04-16T16:22:48.460 INFO:tasks.ceph.mds.c.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:22:48.460 INFO:tasks.ceph.mds.c.smithi045.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f5bc9f81f4c] 2024-04-16T16:22:48.460 INFO:tasks.ceph.mds.c.smithi045.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7f5bc9f8210a] 2024-04-16T16:22:48.460 INFO:tasks.ceph.mds.c.smithi045.stderr: 3: (MDLog::trim()+0xf2a) [0x56381c7d31aa] 2024-04-16T16:22:48.460 INFO:tasks.ceph.mds.c.smithi045.stderr: 4: (MDLog::log_trim_upkeep()+0x339) [0x56381c7d3559] 2024-04-16T16:22:48.461 INFO:tasks.ceph.mds.c.smithi045.stderr: 5: /lib64/libstdc++.so.6(+0xdb924) [0x7f5bc9adb924] 2024-04-16T16:22:48.461 INFO:tasks.ceph.mds.c.smithi045.stderr: 6: /lib64/libc.so.6(+0x9f802) [0x7f5bc969f802] 2024-04-16T16:22:48.461 INFO:tasks.ceph.mds.c.smithi045.stderr: 7: /lib64/libc.so.6(+0x3f450) [0x7f5bc963f450] 2024-04-16T16:22:48.461 INFO:tasks.ceph.mds.c.smithi045.stderr: 2024-04-16T16:22:48.461 INFO:tasks.ceph.mds.c.smithi045.stderr: 0> 2024-04-16T16:22:48.448+0000 7f5bbd600640 -1 *** Caught signal (Aborted) ** 2024-04-16T16:22:48.461 INFO:tasks.ceph.mds.c.smithi045.stderr: in thread 7f5bbd600640 thread_name:ms_dispatch 2024-04-16T16:22:48.461 INFO:tasks.ceph.mds.c.smithi045.stderr: 2024-04-16T16:22:48.461 INFO:tasks.ceph.mds.c.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:22:48.461 INFO:tasks.ceph.mds.c.smithi045.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f5bc9654db0] 2024-04-16T16:22:48.461 INFO:tasks.ceph.mds.c.smithi045.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f5bc96a154c] 2024-04-16T16:22:48.461 INFO:tasks.ceph.mds.c.smithi045.stderr: 3: raise() 2024-04-16T16:22:48.461 INFO:tasks.ceph.mds.c.smithi045.stderr: 4: abort() 2024-04-16T16:22:48.461 INFO:tasks.ceph.mds.c.smithi045.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f5bc9f81fa6] 2024-04-16T16:22:48.461 INFO:tasks.ceph.mds.c.smithi045.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7f5bc9f8210a] 2024-04-16T16:22:48.461 INFO:tasks.ceph.mds.c.smithi045.stderr: 7: (MDLog::trim()+0xf2a) [0x56381c7d31aa] 2024-04-16T16:22:48.462 INFO:tasks.ceph.mds.c.smithi045.stderr: 8: (MDLog::log_trim_upkeep()+0x339) [0x56381c7d3559] 2024-04-16T16:22:48.462 INFO:tasks.ceph.mds.c.smithi045.stderr: 9: /lib64/libstdc++.so.6(+0xdb924) [0x7f5bc9adb924] 2024-04-16T16:22:48.462 INFO:tasks.ceph.mds.c.smithi045.stderr: 10: /lib64/libc.so.6(+0x9f802) [0x7f5bc969f802] 2024-04-16T16:22:48.462 INFO:tasks.ceph.mds.c.smithi045.stderr: 11: /lib64/libc.so.6(+0x3f450) [0x7f5bc963f450] 2024-04-16T16:22:48.462 INFO:tasks.ceph.mds.c.smithi045.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-16T16:22:48.462 INFO:tasks.ceph.mds.c.smithi045.stderr: 2024-04-16T16:22:48.471 INFO:tasks.ceph.mds.c.smithi045.stderr: -1> 2024-04-16T16:22:48.445+0000 7f5bbd600640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: In function 'void MDLog::trim()' thread 7f5bbd600640 time 2024-04-16T16:22:48.445030+0000 2024-04-16T16:22:48.471 INFO:tasks.ceph.mds.c.smithi045.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2868-gc3c87ed0/rpm/el9/BUILD/ceph-19.0.0-2868-gc3c87ed0/src/mds/MDLog.cc: 688: FAILED ceph_assert(num_total_segments >= num_replayed_segments) 2024-04-16T16:22:48.471 INFO:tasks.ceph.mds.c.smithi045.stderr: 2024-04-16T16:22:48.471 INFO:tasks.ceph.mds.c.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:22:48.471 INFO:tasks.ceph.mds.c.smithi045.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f5bc9f81f4c] 2024-04-16T16:22:48.471 INFO:tasks.ceph.mds.c.smithi045.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7f5bc9f8210a] 2024-04-16T16:22:48.472 INFO:tasks.ceph.mds.c.smithi045.stderr: 3: (MDLog::trim()+0xf2a) [0x56381c7d31aa] 2024-04-16T16:22:48.472 INFO:tasks.ceph.mds.c.smithi045.stderr: 4: (MDLog::log_trim_upkeep()+0x339) [0x56381c7d3559] 2024-04-16T16:22:48.472 INFO:tasks.ceph.mds.c.smithi045.stderr: 5: /lib64/libstdc++.so.6(+0xdb924) [0x7f5bc9adb924] 2024-04-16T16:22:48.472 INFO:tasks.ceph.mds.c.smithi045.stderr: 6: /lib64/libc.so.6(+0x9f802) [0x7f5bc969f802] 2024-04-16T16:22:48.472 INFO:tasks.ceph.mds.c.smithi045.stderr: 7: /lib64/libc.so.6(+0x3f450) [0x7f5bc963f450] 2024-04-16T16:22:48.472 INFO:tasks.ceph.mds.c.smithi045.stderr: 2024-04-16T16:22:48.472 INFO:tasks.ceph.mds.c.smithi045.stderr: 0> 2024-04-16T16:22:48.448+0000 7f5bbd600640 -1 *** Caught signal (Aborted) ** 2024-04-16T16:22:48.472 INFO:tasks.ceph.mds.c.smithi045.stderr: in thread 7f5bbd600640 thread_name:ms_dispatch 2024-04-16T16:22:48.472 INFO:tasks.ceph.mds.c.smithi045.stderr: 2024-04-16T16:22:48.472 INFO:tasks.ceph.mds.c.smithi045.stderr: ceph version 19.0.0-2868-gc3c87ed0 (c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f) squid (dev) 2024-04-16T16:22:48.472 INFO:tasks.ceph.mds.c.smithi045.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f5bc9654db0] 2024-04-16T16:22:48.472 INFO:tasks.ceph.mds.c.smithi045.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f5bc96a154c] 2024-04-16T16:22:48.472 INFO:tasks.ceph.mds.c.smithi045.stderr: 3: raise() 2024-04-16T16:22:48.472 INFO:tasks.ceph.mds.c.smithi045.stderr: 4: abort() 2024-04-16T16:22:48.473 INFO:tasks.ceph.mds.c.smithi045.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f5bc9f81fa6] 2024-04-16T16:22:48.473 INFO:tasks.ceph.mds.c.smithi045.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x18210a) [0x7f5bc9f8210a] 2024-04-16T16:22:48.473 INFO:tasks.ceph.mds.c.smithi045.stderr: 7: (MDLog::trim()+0xf2a) [0x56381c7d31aa] 2024-04-16T16:22:48.473 INFO:tasks.ceph.mds.c.smithi045.stderr: 8: (MDLog::log_trim_upkeep()+0x339) [0x56381c7d3559] 2024-04-16T16:22:48.473 INFO:tasks.ceph.mds.c.smithi045.stderr: 9: /lib64/libstdc++.so.6(+0xdb924) [0x7f5bc9adb924] 2024-04-16T16:22:48.473 INFO:tasks.ceph.mds.c.smithi045.stderr: 10: /lib64/libc.so.6(+0x9f802) [0x7f5bc969f802] 2024-04-16T16:22:48.473 INFO:tasks.ceph.mds.c.smithi045.stderr: 11: /lib64/libc.so.6(+0x3f450) [0x7f5bc963f450] 2024-04-16T16:22:48.473 INFO:tasks.ceph.mds.c.smithi045.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-16T16:22:48.473 INFO:tasks.ceph.mds.c.smithi045.stderr: 2024-04-16T16:22:48.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.484+0000 7f9cde000640 1 -- 172.21.15.45:0/2668590474 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v113) ==== 118+0+0 (secure 0 0 0) 0x7f9cd0058e50 con 0x7f9ce80d18f0 2024-04-16T16:22:48.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.488+0000 7f9ce7200640 1 -- 172.21.15.45:0/2668590474 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f9cc406ca10 msgr2=0x7f9cc406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:48.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.488+0000 7f9ce7200640 1 --2- 172.21.15.45:0/2668590474 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f9cc406ca10 0x7f9cc406eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8036000 tx=0x7f9cd8030000 comp rx=0 tx=0).stop 2024-04-16T16:22:48.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.488+0000 7f9ce7200640 1 -- 172.21.15.45:0/2668590474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ce80d18f0 msgr2=0x7f9ce814c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:48.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.488+0000 7f9ce7200640 1 --2- 172.21.15.45:0/2668590474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ce80d18f0 0x7f9ce814c630 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f9cd000c430 tx=0x7f9cd000c8f0 comp rx=0 tx=0).stop 2024-04-16T16:22:48.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.488+0000 7f9ce7200640 1 -- 172.21.15.45:0/2668590474 shutdown_connections 2024-04-16T16:22:48.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.488+0000 7f9ce7200640 1 --2- 172.21.15.45:0/2668590474 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f9cc406ca10 0x7f9cc406eed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:48.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.488+0000 7f9ce7200640 1 --2- 172.21.15.45:0/2668590474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ce80d1cd0 0x7f9ce8158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:48.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.488+0000 7f9ce7200640 1 --2- 172.21.15.45:0/2668590474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ce80d18f0 0x7f9ce814c630 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:48.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.488+0000 7f9ce7200640 1 --2- 172.21.15.45:0/2668590474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce80b6310 0x7f9ce80cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:48.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.488+0000 7f9ce7200640 1 -- 172.21.15.45:0/2668590474 >> 172.21.15.45:0/2668590474 conn(0x7f9ce806cf90 msgr2=0x7f9ce80b4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:48.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.488+0000 7f9ce7200640 1 -- 172.21.15.45:0/2668590474 shutdown_connections 2024-04-16T16:22:48.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.489+0000 7f9ce7200640 1 -- 172.21.15.45:0/2668590474 wait complete. 2024-04-16T16:22:48.508 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-16T16:22:48.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-16T16:22:48.668 INFO:tasks.ceph.mds.c.smithi045.stderr:daemon-helper: command crashed with signal 6 2024-04-16T16:22:48.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.728+0000 7fb105a00640 1 Processor -- start 2024-04-16T16:22:48.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.728+0000 7fb105a00640 1 -- start start 2024-04-16T16:22:48.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.730+0000 7fb105a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10015b430 0x7fb1000cd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:48.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.730+0000 7fb105a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1000cdf60 con 0x7fb1000cdac0 2024-04-16T16:22:48.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.730+0000 7fb105a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1000ce240 con 0x7fb1000b6000 2024-04-16T16:22:48.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.730+0000 7fb105a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb1000ce520 con 0x7fb10015b430 2024-04-16T16:22:48.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.730+0000 7fb0ff400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb1000cdac0 0x7fb1000b58f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38422/0 (socket says 172.21.15.45:38422) 2024-04-16T16:22:48.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.730+0000 7fb0ff400640 1 -- 172.21.15.45:0/4204591500 learned_addr learned my addr 172.21.15.45:0/4204591500 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:48.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.731+0000 7fb0ffe00640 1 --2- 172.21.15.45:0/4204591500 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10015b430 0x7fb1000cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:48.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.731+0000 7fb0fea00640 1 -- 172.21.15.45:0/4204591500 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3006980009 0 0) 0x7fb1000cdf60 con 0x7fb1000cdac0 2024-04-16T16:22:48.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.731+0000 7fb0fea00640 1 -- 172.21.15.45:0/4204591500 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0e4003520 con 0x7fb1000cdac0 2024-04-16T16:22:48.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.731+0000 7fb0fea00640 1 -- 172.21.15.45:0/4204591500 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2348908856 0 0) 0x7fb0e4003520 con 0x7fb1000cdac0 2024-04-16T16:22:48.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.731+0000 7fb0fea00640 1 -- 172.21.15.45:0/4204591500 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10015b430 msgr2=0x7fb1000cd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:48.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.731+0000 7fb0fea00640 1 --2- 172.21.15.45:0/4204591500 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10015b430 0x7fb1000cd580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:48.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.731+0000 7fb0fea00640 1 -- 172.21.15.45:0/4204591500 >> v1:172.21.15.103:6789/0 conn(0x7fb1000b6000 legacy=0x7fb1000cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.731+0000 7fb0fea00640 1 -- 172.21.15.45:0/4204591500 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1000ce800 con 0x7fb1000cdac0 2024-04-16T16:22:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.731+0000 7fb0fea00640 1 -- 172.21.15.45:0/4204591500 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7fb0ec001670 con 0x7fb1000cdac0 2024-04-16T16:22:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.732+0000 7fb0fea00640 1 -- 172.21.15.45:0/4204591500 >> v1:172.21.15.45:6789/0 conn(0x7fb1000cdac0 legacy=0x7fb1000b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.732+0000 7fb0fea00640 1 --2- 172.21.15.45:0/4204591500 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10015b430 0x7fb0e40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.732+0000 7fb0fea00640 1 --2- 172.21.15.45:0/4204591500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0e4003f60 0x7fb0e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.732+0000 7fb0fea00640 1 --2- 172.21.15.45:0/4204591500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0e4004d40 0x7fb0e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.732+0000 7fb0fea00640 1 -- 172.21.15.45:0/4204591500 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb1000ce520 con 0x7fb0e4003f60 2024-04-16T16:22:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.732+0000 7fb0fea00640 1 -- 172.21.15.45:0/4204591500 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e400baa0 con 0x7fb0e4004d40 2024-04-16T16:22:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.732+0000 7fb0fea00640 1 -- 172.21.15.45:0/4204591500 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb0e400bca0 con 0x7fb10015b430 2024-04-16T16:22:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.732+0000 7fb0ff400640 1 --2- 172.21.15.45:0/4204591500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0e4003f60 0x7fb0e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.732+0000 7fb0ffe00640 1 --2- 172.21.15.45:0/4204591500 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10015b430 0x7fb0e40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.732+0000 7fb105000640 1 --2- 172.21.15.45:0/4204591500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0e4004d40 0x7fb0e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.733+0000 7fb0ff400640 1 -- 172.21.15.45:0/4204591500 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10015b430 msgr2=0x7fb0e40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.733+0000 7fb0ff400640 1 --2- 172.21.15.45:0/4204591500 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10015b430 0x7fb0e40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.733+0000 7fb0ff400640 1 -- 172.21.15.45:0/4204591500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0e4004d40 msgr2=0x7fb0e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:48.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.733+0000 7fb0ff400640 1 --2- 172.21.15.45:0/4204591500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0e4004d40 0x7fb0e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:48.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.733+0000 7fb0ff400640 1 -- 172.21.15.45:0/4204591500 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e4003520 con 0x7fb0e4003f60 2024-04-16T16:22:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.734+0000 7fb0ff400640 1 --2- 172.21.15.45:0/4204591500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0e4003f60 0x7fb0e4004800 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00c240 tx=0x7fb0e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cca66ee252f94b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.734+0000 7fb0fea00640 1 -- 172.21.15.45:0/4204591500 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec007d10 con 0x7fb0e4003f60 2024-04-16T16:22:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.734+0000 7fb0fea00640 1 -- 172.21.15.45:0/4204591500 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0ec030070 con 0x7fb0e4003f60 2024-04-16T16:22:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.734+0000 7fb0fea00640 1 -- 172.21.15.45:0/4204591500 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec02cda0 con 0x7fb0e4003f60 2024-04-16T16:22:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.735+0000 7fb105a00640 1 -- 172.21.15.45:0/4204591500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0e4003f60 msgr2=0x7fb0e4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.735+0000 7fb105a00640 1 --2- 172.21.15.45:0/4204591500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0e4003f60 0x7fb0e4004800 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00c240 tx=0x7fb0e4005830 comp rx=0 tx=0).stop 2024-04-16T16:22:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.735+0000 7fb105000640 1 -- 172.21.15.45:0/4204591500 reap_dead start 2024-04-16T16:22:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.735+0000 7fb105a00640 1 -- 172.21.15.45:0/4204591500 shutdown_connections 2024-04-16T16:22:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.735+0000 7fb105a00640 1 -- 172.21.15.45:0/4204591500 >> 172.21.15.45:0/4204591500 conn(0x7fb10006b930 msgr2=0x7fb10006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.735+0000 7fb105a00640 1 -- 172.21.15.45:0/4204591500 shutdown_connections 2024-04-16T16:22:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.735+0000 7fb105a00640 1 -- 172.21.15.45:0/4204591500 wait complete. 2024-04-16T16:22:48.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.737+0000 7fb105a00640 1 Processor -- start 2024-04-16T16:22:48.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.737+0000 7fb105a00640 1 -- start start 2024-04-16T16:22:48.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.738+0000 7fb105a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0e4003f60 0x7fb100144980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:48.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.738+0000 7fb105a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10014d580 0x7fb10015e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:48.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.738+0000 7fb105a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb100141ac0 0x7fb10015c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:48.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.738+0000 7fb105a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb10015f490 con 0x7fb100141ac0 2024-04-16T16:22:48.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.738+0000 7fb105a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb10015f310 con 0x7fb0e4003f60 2024-04-16T16:22:48.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.738+0000 7fb105a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb10015b810 con 0x7fb10014d580 2024-04-16T16:22:48.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.738+0000 7fb105000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb100141ac0 0x7fb10015c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:48.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.738+0000 7fb105000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb100141ac0 0x7fb10015c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33058/0 (socket says 172.21.15.45:33058) 2024-04-16T16:22:48.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.738+0000 7fb105000640 1 -- 172.21.15.45:0/1694755632 learned_addr learned my addr 172.21.15.45:0/1694755632 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:48.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.738+0000 7fb0ffe00640 1 --2- 172.21.15.45:0/1694755632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0e4003f60 0x7fb100144980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:48.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.739+0000 7fb0ff400640 1 --2- 172.21.15.45:0/1694755632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10014d580 0x7fb10015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:48.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.739+0000 7fb105000640 1 -- 172.21.15.45:0/1694755632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10014d580 msgr2=0x7fb10015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:48.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.739+0000 7fb105000640 1 --2- 172.21.15.45:0/1694755632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10014d580 0x7fb10015e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:48.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.739+0000 7fb105000640 1 -- 172.21.15.45:0/1694755632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0e4003f60 msgr2=0x7fb100144980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:48.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.739+0000 7fb105000640 1 --2- 172.21.15.45:0/1694755632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0e4003f60 0x7fb100144980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:48.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.739+0000 7fb105000640 1 -- 172.21.15.45:0/1694755632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1001450f0 con 0x7fb100141ac0 2024-04-16T16:22:48.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.739+0000 7fb105000640 1 --2- 172.21.15.45:0/1694755632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb100141ac0 0x7fb10015c880 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fb1000b6a00 tx=0x7fb0f800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:48.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.739+0000 7fb0f7e00640 1 -- 172.21.15.45:0/1694755632 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8003970 con 0x7fb100141ac0 2024-04-16T16:22:48.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.739+0000 7fb0f7e00640 1 -- 172.21.15.45:0/1694755632 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f8018070 con 0x7fb100141ac0 2024-04-16T16:22:48.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.739+0000 7fb105a00640 1 -- 172.21.15.45:0/1694755632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb10015bda0 con 0x7fb100141ac0 2024-04-16T16:22:48.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.739+0000 7fb0f7e00640 1 -- 172.21.15.45:0/1694755632 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8003d50 con 0x7fb100141ac0 2024-04-16T16:22:48.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.740+0000 7fb105a00640 1 -- 172.21.15.45:0/1694755632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb10015c360 con 0x7fb100141ac0 2024-04-16T16:22:48.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.741+0000 7fb105a00640 1 -- 172.21.15.45:0/1694755632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb100075780 con 0x7fb100141ac0 2024-04-16T16:22:48.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.742+0000 7fb0f7e00640 1 -- 172.21.15.45:0/1694755632 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fb0f80050d0 con 0x7fb100141ac0 2024-04-16T16:22:48.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.743+0000 7fb0f7e00640 1 --2- 172.21.15.45:0/1694755632 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb0e006ca40 0x7fb0e006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:48.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.743+0000 7fb0f7e00640 1 -- 172.21.15.45:0/1694755632 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7fb0f808ba30 con 0x7fb100141ac0 2024-04-16T16:22:48.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.743+0000 7fb0ffe00640 1 --2- 172.21.15.45:0/1694755632 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb0e006ca40 0x7fb0e006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:48.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.745+0000 7fb0ffe00640 1 --2- 172.21.15.45:0/1694755632 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb0e006ca40 0x7fb0e006ef00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb10014fa50 tx=0x7fb0f0008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:48.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.747+0000 7fb0f7e00640 1 -- 172.21.15.45:0/1694755632 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb0f8059d40 con 0x7fb100141ac0 2024-04-16T16:22:48.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:48.993+0000 7fb105a00640 1 -- 172.21.15.45:0/1694755632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fb10015c530 con 0x7fb100141ac0 2024-04-16T16:22:49.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-04-16T16:22:49.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.487+0000 7fb0f7e00640 1 -- 172.21.15.45:0/1694755632 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v114) ==== 122+0+0 (secure 0 0 0) 0x7fb0f8059640 con 0x7fb100141ac0 2024-04-16T16:22:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.491+0000 7fb105a00640 1 -- 172.21.15.45:0/1694755632 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb0e006ca40 msgr2=0x7fb0e006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.491+0000 7fb105a00640 1 --2- 172.21.15.45:0/1694755632 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb0e006ca40 0x7fb0e006ef00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb10014fa50 tx=0x7fb0f0008040 comp rx=0 tx=0).stop 2024-04-16T16:22:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.491+0000 7fb105a00640 1 -- 172.21.15.45:0/1694755632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb100141ac0 msgr2=0x7fb10015c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.491+0000 7fb105a00640 1 --2- 172.21.15.45:0/1694755632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb100141ac0 0x7fb10015c880 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fb1000b6a00 tx=0x7fb0f800ec10 comp rx=0 tx=0).stop 2024-04-16T16:22:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.492+0000 7fb105a00640 1 -- 172.21.15.45:0/1694755632 shutdown_connections 2024-04-16T16:22:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.492+0000 7fb105a00640 1 --2- 172.21.15.45:0/1694755632 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb0e006ca40 0x7fb0e006ef00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.492+0000 7fb105a00640 1 --2- 172.21.15.45:0/1694755632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb100141ac0 0x7fb10015c880 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.492+0000 7fb105a00640 1 --2- 172.21.15.45:0/1694755632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb10014d580 0x7fb10015e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.492+0000 7fb105a00640 1 --2- 172.21.15.45:0/1694755632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0e4003f60 0x7fb100144980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.492+0000 7fb105a00640 1 -- 172.21.15.45:0/1694755632 >> 172.21.15.45:0/1694755632 conn(0x7fb10006b930 msgr2=0x7fb1000cc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:49.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.492+0000 7fb105a00640 1 -- 172.21.15.45:0/1694755632 shutdown_connections 2024-04-16T16:22:49.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.492+0000 7fb105a00640 1 -- 172.21.15.45:0/1694755632 wait complete. 2024-04-16T16:22:49.517 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-16T16:22:49.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:22:49.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.788+0000 7f28e1c00640 1 Processor -- start 2024-04-16T16:22:49.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.788+0000 7f28e1c00640 1 -- start start 2024-04-16T16:22:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.790+0000 7f28e1c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f28dc15e8f0 0x7f28dc0cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.790+0000 7f28e1c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28dc1632a0 con 0x7f28dc0ce9a0 2024-04-16T16:22:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.790+0000 7f28e1c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28dc15ecd0 con 0x7f28dc0cf080 2024-04-16T16:22:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.790+0000 7f28e1c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f28dc0ce2e0 con 0x7f28dc15e8f0 2024-04-16T16:22:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.790+0000 7f28db400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f28dc0ce9a0 0x7f28dc162480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38428/0 (socket says 172.21.15.45:38428) 2024-04-16T16:22:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.790+0000 7f28db400640 1 -- 172.21.15.45:0/3240391458 learned_addr learned my addr 172.21.15.45:0/3240391458 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28dbe00640 1 --2- 172.21.15.45:0/3240391458 >> v2:172.21.15.103:3300/0 conn(0x7f28dc15e8f0 0x7f28dc0cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28daa00640 1 -- 172.21.15.45:0/3240391458 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1411511329 0 0) 0x7f28dc1632a0 con 0x7f28dc0ce9a0 2024-04-16T16:22:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28daa00640 1 -- 172.21.15.45:0/3240391458 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28c0003520 con 0x7f28dc0ce9a0 2024-04-16T16:22:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28daa00640 1 -- 172.21.15.45:0/3240391458 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2505500998 0 0) 0x7f28c0003520 con 0x7f28dc0ce9a0 2024-04-16T16:22:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28daa00640 1 -- 172.21.15.45:0/3240391458 >> v2:172.21.15.103:3300/0 conn(0x7f28dc15e8f0 msgr2=0x7f28dc0cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28daa00640 1 --2- 172.21.15.45:0/3240391458 >> v2:172.21.15.103:3300/0 conn(0x7f28dc15e8f0 0x7f28dc0cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28daa00640 1 -- 172.21.15.45:0/3240391458 >> v1:172.21.15.103:6789/0 conn(0x7f28dc0cf080 legacy=0x7f28dc162b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28daa00640 1 -- 172.21.15.45:0/3240391458 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28dc0cd6d0 con 0x7f28dc0ce9a0 2024-04-16T16:22:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28daa00640 1 -- 172.21.15.45:0/3240391458 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f28c8001670 con 0x7f28dc0ce9a0 2024-04-16T16:22:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28daa00640 1 -- 172.21.15.45:0/3240391458 >> v1:172.21.15.45:6789/0 conn(0x7f28dc0ce9a0 legacy=0x7f28dc162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28daa00640 1 --2- 172.21.15.45:0/3240391458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28c0003a00 0x7f28c0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28daa00640 1 --2- 172.21.15.45:0/3240391458 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28c0004800 0x7f28c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28daa00640 1 --2- 172.21.15.45:0/3240391458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c00051c0 0x7f28c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28daa00640 1 -- 172.21.15.45:0/3240391458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc0ce2e0 con 0x7f28c0003a00 2024-04-16T16:22:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28daa00640 1 -- 172.21.15.45:0/3240391458 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f28c000c0c0 con 0x7f28c00051c0 2024-04-16T16:22:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.791+0000 7f28daa00640 1 -- 172.21.15.45:0/3240391458 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f28c000c2c0 con 0x7f28c0004800 2024-04-16T16:22:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.792+0000 7f28dbe00640 1 --2- 172.21.15.45:0/3240391458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28c0003a00 0x7f28c0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.792+0000 7f28dbe00640 1 -- 172.21.15.45:0/3240391458 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28c0004800 msgr2=0x7f28c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.792+0000 7f28db400640 1 --2- 172.21.15.45:0/3240391458 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28c0004800 0x7f28c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.792+0000 7f28dbe00640 1 --2- 172.21.15.45:0/3240391458 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28c0004800 0x7f28c0004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.792+0000 7f28dbe00640 1 -- 172.21.15.45:0/3240391458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c00051c0 msgr2=0x7f28c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.792+0000 7f28e1200640 1 --2- 172.21.15.45:0/3240391458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c00051c0 0x7f28c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.792+0000 7f28dbe00640 1 --2- 172.21.15.45:0/3240391458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c00051c0 0x7f28c000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.792+0000 7f28e1200640 1 -- 172.21.15.45:0/3240391458 reap_dead start 2024-04-16T16:22:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.792+0000 7f28dbe00640 1 -- 172.21.15.45:0/3240391458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c0003520 con 0x7f28c0003a00 2024-04-16T16:22:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.793+0000 7f28dbe00640 1 --2- 172.21.15.45:0/3240391458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28c0003a00 0x7f28c0003e20 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009ac0 tx=0x7f28c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=decba5df1cc2c621 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.794+0000 7f28daa00640 1 -- 172.21.15.45:0/3240391458 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc039070 con 0x7f28c0003a00 2024-04-16T16:22:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.794+0000 7f28daa00640 1 -- 172.21.15.45:0/3240391458 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28cc02c6b0 con 0x7f28c0003a00 2024-04-16T16:22:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.794+0000 7f28daa00640 1 -- 172.21.15.45:0/3240391458 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc009690 con 0x7f28c0003a00 2024-04-16T16:22:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.794+0000 7f28e1c00640 1 -- 172.21.15.45:0/3240391458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28c0003a00 msgr2=0x7f28c0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.794+0000 7f28e1c00640 1 --2- 172.21.15.45:0/3240391458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28c0003a00 0x7f28c0003e20 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009ac0 tx=0x7f28c0005830 comp rx=0 tx=0).stop 2024-04-16T16:22:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.794+0000 7f28e1c00640 1 -- 172.21.15.45:0/3240391458 shutdown_connections 2024-04-16T16:22:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.794+0000 7f28e1c00640 1 --2- 172.21.15.45:0/3240391458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28c0003a00 0x7f28c0003e20 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.794+0000 7f28e1c00640 1 -- 172.21.15.45:0/3240391458 >> 172.21.15.45:0/3240391458 conn(0x7f28dc06b930 msgr2=0x7f28dc06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.794+0000 7f28e1c00640 1 -- 172.21.15.45:0/3240391458 shutdown_connections 2024-04-16T16:22:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.795+0000 7f28e1c00640 1 -- 172.21.15.45:0/3240391458 wait complete. 2024-04-16T16:22:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.796+0000 7f28e1c00640 1 Processor -- start 2024-04-16T16:22:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.796+0000 7f28e1c00640 1 -- start start 2024-04-16T16:22:49.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.797+0000 7f28e1c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c0003a00 0x7f28dc1439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:49.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.797+0000 7f28e1c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28dc146560 0x7f28dc1515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:49.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.797+0000 7f28e1c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28dc1530f0 0x7f28dc153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:49.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.797+0000 7f28e1c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc0cdf70 con 0x7f28dc146560 2024-04-16T16:22:49.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.797+0000 7f28e1c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc0cdc90 con 0x7f28c0003a00 2024-04-16T16:22:49.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.797+0000 7f28e1c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f28dc0cd360 con 0x7f28dc1530f0 2024-04-16T16:22:49.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.798+0000 7f28db400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28dc146560 0x7f28dc1515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:49.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.798+0000 7f28db400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28dc146560 0x7f28dc1515d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33068/0 (socket says 172.21.15.45:33068) 2024-04-16T16:22:49.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.798+0000 7f28db400640 1 -- 172.21.15.45:0/1762310761 learned_addr learned my addr 172.21.15.45:0/1762310761 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.798+0000 7f28e1200640 1 --2- 172.21.15.45:0/1762310761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28dc1530f0 0x7f28dc153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.798+0000 7f28dbe00640 1 --2- 172.21.15.45:0/1762310761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c0003a00 0x7f28dc1439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.798+0000 7f28db400640 1 -- 172.21.15.45:0/1762310761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28dc1530f0 msgr2=0x7f28dc153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.798+0000 7f28db400640 1 --2- 172.21.15.45:0/1762310761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28dc1530f0 0x7f28dc153570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.798+0000 7f28db400640 1 -- 172.21.15.45:0/1762310761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c0003a00 msgr2=0x7f28dc1439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.798+0000 7f28db400640 1 --2- 172.21.15.45:0/1762310761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c0003a00 0x7f28dc1439b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.798+0000 7f28db400640 1 -- 172.21.15.45:0/1762310761 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28dc15fc80 con 0x7f28dc146560 2024-04-16T16:22:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.798+0000 7f28db400640 1 --2- 172.21.15.45:0/1762310761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28dc146560 0x7f28dc1515d0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f28c800d7a0 tx=0x7f28c800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.799+0000 7f28d3e00640 1 -- 172.21.15.45:0/1762310761 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8019070 con 0x7f28dc146560 2024-04-16T16:22:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.799+0000 7f28d3e00640 1 -- 172.21.15.45:0/1762310761 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28c800de50 con 0x7f28dc146560 2024-04-16T16:22:49.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.799+0000 7f28d3e00640 1 -- 172.21.15.45:0/1762310761 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c80047e0 con 0x7f28dc146560 2024-04-16T16:22:49.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.799+0000 7f28e1c00640 1 -- 172.21.15.45:0/1762310761 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28dc15ff20 con 0x7f28dc146560 2024-04-16T16:22:49.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.799+0000 7f28e1c00640 1 -- 172.21.15.45:0/1762310761 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28dc1604e0 con 0x7f28dc146560 2024-04-16T16:22:49.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.800+0000 7f28e1c00640 1 -- 172.21.15.45:0/1762310761 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28dc075780 con 0x7f28dc146560 2024-04-16T16:22:49.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.801+0000 7f28d3e00640 1 -- 172.21.15.45:0/1762310761 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f28c8004980 con 0x7f28dc146560 2024-04-16T16:22:49.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.802+0000 7f28d3e00640 1 --2- 172.21.15.45:0/1762310761 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f28bc06ca10 0x7f28bc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:49.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.802+0000 7f28d3e00640 1 -- 172.21.15.45:0/1762310761 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f28c808b880 con 0x7f28dc146560 2024-04-16T16:22:49.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.803+0000 7f28dbe00640 1 --2- 172.21.15.45:0/1762310761 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f28bc06ca10 0x7f28bc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:49.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.804+0000 7f28dbe00640 1 --2- 172.21.15.45:0/1762310761 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f28bc06ca10 0x7f28bc06eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f28cc036000 tx=0x7f28cc030000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:49.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:49.807+0000 7f28d3e00640 1 -- 172.21.15.45:0/1762310761 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f28c8004c70 con 0x7f28dc146560 2024-04-16T16:22:50.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.052+0000 7f28e1c00640 1 -- 172.21.15.45:0/1762310761 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28dc1469c0 con 0x7f28dc146560 2024-04-16T16:22:50.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.053+0000 7f28d3e00640 1 -- 172.21.15.45:0/1762310761 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) ==== 95+0+5194 (secure 0 0 0) 0x7f28c80599f0 con 0x7f28dc146560 2024-04-16T16:22:50.052 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:50.052 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":114,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6896,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2328390625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2328390625},{"type":"v1","addr":"172.21.15.103:6835","nonce":2328390625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":109},{"gid":7756,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/677052950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":677052950},{"type":"v1","addr":"172.21.15.45:6834","nonce":677052950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":108},{"gid":7765,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3875926287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3875926287},{"type":"v1","addr":"172.21.15.103:6837","nonce":3875926287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:22:45.927071+0000","modified":"2024-04-16T16:22:48.994607+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9840},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9840":{"gid":9840,"name":"c","rank":0,"incarnation":110,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3351492627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3351492627},{"type":"v1","addr":"172.21.15.45:6835","nonce":3351492627}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9840,"qdb_cluster":[9840]},"id":11}]} 2024-04-16T16:22:50.053 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 114 2024-04-16T16:22:50.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.056+0000 7f28e1c00640 1 -- 172.21.15.45:0/1762310761 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f28bc06ca10 msgr2=0x7f28bc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:50.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.056+0000 7f28e1c00640 1 --2- 172.21.15.45:0/1762310761 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f28bc06ca10 0x7f28bc06eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f28cc036000 tx=0x7f28cc030000 comp rx=0 tx=0).stop 2024-04-16T16:22:50.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.056+0000 7f28e1c00640 1 -- 172.21.15.45:0/1762310761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28dc146560 msgr2=0x7f28dc1515d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:50.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.056+0000 7f28e1c00640 1 --2- 172.21.15.45:0/1762310761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28dc146560 0x7f28dc1515d0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f28c800d7a0 tx=0x7f28c800dc60 comp rx=0 tx=0).stop 2024-04-16T16:22:50.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.056+0000 7f28e1c00640 1 -- 172.21.15.45:0/1762310761 shutdown_connections 2024-04-16T16:22:50.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.056+0000 7f28e1c00640 1 --2- 172.21.15.45:0/1762310761 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f28bc06ca10 0x7f28bc06eed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.056+0000 7f28e1c00640 1 --2- 172.21.15.45:0/1762310761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28dc1530f0 0x7f28dc153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.056+0000 7f28e1c00640 1 --2- 172.21.15.45:0/1762310761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28dc146560 0x7f28dc1515d0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.056+0000 7f28e1c00640 1 --2- 172.21.15.45:0/1762310761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c0003a00 0x7f28dc1439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.056+0000 7f28e1c00640 1 -- 172.21.15.45:0/1762310761 >> 172.21.15.45:0/1762310761 conn(0x7f28dc06b930 msgr2=0x7f28dc0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:50.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.056+0000 7f28e1c00640 1 -- 172.21.15.45:0/1762310761 shutdown_connections 2024-04-16T16:22:50.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.056+0000 7f28e1c00640 1 -- 172.21.15.45:0/1762310761 wait complete. 2024-04-16T16:22:50.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:22:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.297+0000 7f94f3200640 1 Processor -- start 2024-04-16T16:22:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.297+0000 7f94f3200640 1 -- start start 2024-04-16T16:22:50.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.299+0000 7f94f3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94f4146220 0x7f94f415ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.299+0000 7f94f3200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f94f41608a0 0x7f94f4160d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.299+0000 7f94f3200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94f415d360 con 0x7f94f406f7a0 2024-04-16T16:22:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.299+0000 7f94f3200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f94f4160410 con 0x7f94f41608a0 2024-04-16T16:22:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.299+0000 7f94f3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f94f415d640 con 0x7f94f4146220 2024-04-16T16:22:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.299+0000 7f94f1e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94f4146220 0x7f94f415ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.300+0000 7f94f1e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94f4146220 0x7f94f415ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:48456/0 (socket says 172.21.15.45:48456) 2024-04-16T16:22:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.300+0000 7f94f1e00640 1 -- 172.21.15.45:0/1163212222 learned_addr learned my addr 172.21.15.45:0/1163212222 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.300+0000 7f94f2800640 1 --2- 172.21.15.45:0/1163212222 >> v2:172.21.15.103:3300/0 conn(0x7f94f41608a0 0x7f94f4160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.300+0000 7f94f1e00640 1 -- 172.21.15.45:0/1163212222 >> v2:172.21.15.103:3300/0 conn(0x7f94f41608a0 msgr2=0x7f94f4160d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.300+0000 7f94f1e00640 1 --2- 172.21.15.45:0/1163212222 >> v2:172.21.15.103:3300/0 conn(0x7f94f41608a0 0x7f94f4160d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.300+0000 7f94f1e00640 1 -- 172.21.15.45:0/1163212222 >> v1:172.21.15.103:6789/0 conn(0x7f94f406f7a0 legacy=0x7f94f406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:50.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.300+0000 7f94f1e00640 1 -- 172.21.15.45:0/1163212222 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f415d920 con 0x7f94f4146220 2024-04-16T16:22:50.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.302+0000 7f94f1e00640 1 --2- 172.21.15.45:0/1163212222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94f4146220 0x7f94f415ccb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f94e4009870 tx=0x7f94e402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4af899506251d645 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:50.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.303+0000 7f94f0a00640 1 -- 172.21.15.45:0/1163212222 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e4034070 con 0x7f94f4146220 2024-04-16T16:22:50.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.303+0000 7f94f0a00640 1 -- 172.21.15.45:0/1163212222 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94e4030840 con 0x7f94f4146220 2024-04-16T16:22:50.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.303+0000 7f94f0a00640 1 -- 172.21.15.45:0/1163212222 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e4030c10 con 0x7f94f4146220 2024-04-16T16:22:50.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.303+0000 7f94f3200640 1 -- 172.21.15.45:0/1163212222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94f4146220 msgr2=0x7f94f415ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:50.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.303+0000 7f94f3200640 1 --2- 172.21.15.45:0/1163212222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94f4146220 0x7f94f415ccb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f94e4009870 tx=0x7f94e402b980 comp rx=0 tx=0).stop 2024-04-16T16:22:50.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.304+0000 7f94f3200640 1 -- 172.21.15.45:0/1163212222 shutdown_connections 2024-04-16T16:22:50.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.304+0000 7f94f3200640 1 --2- 172.21.15.45:0/1163212222 >> v2:172.21.15.103:3300/0 conn(0x7f94f41608a0 0x7f94f4160d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.304+0000 7f94f3200640 1 --2- 172.21.15.45:0/1163212222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94f4146220 0x7f94f415ccb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.304+0000 7f94f3200640 1 -- 172.21.15.45:0/1163212222 >> 172.21.15.45:0/1163212222 conn(0x7f94f406b930 msgr2=0x7f94f406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:50.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.304+0000 7f94f3200640 1 -- 172.21.15.45:0/1163212222 shutdown_connections 2024-04-16T16:22:50.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.304+0000 7f94f3200640 1 -- 172.21.15.45:0/1163212222 wait complete. 2024-04-16T16:22:50.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.305+0000 7f94f3200640 1 Processor -- start 2024-04-16T16:22:50.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.306+0000 7f94f3200640 1 -- start start 2024-04-16T16:22:50.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.306+0000 7f94f3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94f406f7a0 0x7f94f415c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:50.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.306+0000 7f94f3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94f4146220 0x7f94f41f0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:50.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.307+0000 7f94f3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94f41608a0 0x7f94f413bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:50.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.307+0000 7f94f3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f94f406fc20 con 0x7f94f41608a0 2024-04-16T16:22:50.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.307+0000 7f94f3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f94f415dc00 con 0x7f94f406f7a0 2024-04-16T16:22:50.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.307+0000 7f94f3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f94f406fda0 con 0x7f94f4146220 2024-04-16T16:22:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.307+0000 7f94f2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94f41608a0 0x7f94f413bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.307+0000 7f94f2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94f41608a0 0x7f94f413bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33076/0 (socket says 172.21.15.45:33076) 2024-04-16T16:22:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.307+0000 7f94f2800640 1 -- 172.21.15.45:0/589054497 learned_addr learned my addr 172.21.15.45:0/589054497 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.307+0000 7f94f1e00640 1 --2- 172.21.15.45:0/589054497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94f406f7a0 0x7f94f415c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.307+0000 7f94f1400640 1 --2- 172.21.15.45:0/589054497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94f4146220 0x7f94f41f0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.307+0000 7f94f2800640 1 -- 172.21.15.45:0/589054497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94f4146220 msgr2=0x7f94f41f0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.307+0000 7f94f2800640 1 --2- 172.21.15.45:0/589054497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94f4146220 0x7f94f41f0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.307+0000 7f94f2800640 1 -- 172.21.15.45:0/589054497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94f406f7a0 msgr2=0x7f94f415c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.307+0000 7f94f2800640 1 --2- 172.21.15.45:0/589054497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94f406f7a0 0x7f94f415c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.308+0000 7f94f2800640 1 -- 172.21.15.45:0/589054497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f41f1260 con 0x7f94f41608a0 2024-04-16T16:22:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.308+0000 7f94f1400640 1 --2- 172.21.15.45:0/589054497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94f4146220 0x7f94f41f0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.308+0000 7f94f2800640 1 --2- 172.21.15.45:0/589054497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94f41608a0 0x7f94f413bfd0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00efc0 tx=0x7f94ec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.308+0000 7f94ea000640 1 -- 172.21.15.45:0/589054497 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec018020 con 0x7f94f41608a0 2024-04-16T16:22:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.308+0000 7f94ea000640 1 -- 172.21.15.45:0/589054497 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94ec014480 con 0x7f94f41608a0 2024-04-16T16:22:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.308+0000 7f94ea000640 1 -- 172.21.15.45:0/589054497 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec014870 con 0x7f94f41608a0 2024-04-16T16:22:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.308+0000 7f94f3200640 1 -- 172.21.15.45:0/589054497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94f41f1540 con 0x7f94f41608a0 2024-04-16T16:22:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.308+0000 7f94f3200640 1 -- 172.21.15.45:0/589054497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94f41374a0 con 0x7f94f41608a0 2024-04-16T16:22:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.310+0000 7f94f3200640 1 -- 172.21.15.45:0/589054497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94f4075780 con 0x7f94f41608a0 2024-04-16T16:22:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.311+0000 7f94ea000640 1 -- 172.21.15.45:0/589054497 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f94ec005d10 con 0x7f94f41608a0 2024-04-16T16:22:50.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.312+0000 7f94ea000640 1 --2- 172.21.15.45:0/589054497 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f94d406ca10 0x7f94d406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:50.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.312+0000 7f94ea000640 1 -- 172.21.15.45:0/589054497 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f94ec08c870 con 0x7f94f41608a0 2024-04-16T16:22:50.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.312+0000 7f94f1e00640 1 --2- 172.21.15.45:0/589054497 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f94d406ca10 0x7f94d406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:50.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.314+0000 7f94f1e00640 1 --2- 172.21.15.45:0/589054497 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f94d406ca10 0x7f94d406eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f94e4036000 tx=0x7f94e403b040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:50.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.316+0000 7f94ea000640 1 -- 172.21.15.45:0/589054497 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f94ec022070 con 0x7f94f41608a0 2024-04-16T16:22:50.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.526+0000 7f94f3200640 1 -- 172.21.15.45:0/589054497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f94f4137670 con 0x7f94f41608a0 2024-04-16T16:22:50.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.528+0000 7f94ea000640 1 -- 172.21.15.45:0/589054497 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v156) ==== 74+0+16671 (secure 0 0 0) 0x7f94ec010040 con 0x7f94f41608a0 2024-04-16T16:22:50.528 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:50.528 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":156,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:22:47.450563+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:22:43.864367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-16T16:22:44.899356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:22:50.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.532+0000 7f94f3200640 1 -- 172.21.15.45:0/589054497 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f94d406ca10 msgr2=0x7f94d406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:50.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.532+0000 7f94f3200640 1 --2- 172.21.15.45:0/589054497 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f94d406ca10 0x7f94d406eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f94e4036000 tx=0x7f94e403b040 comp rx=0 tx=0).stop 2024-04-16T16:22:50.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.532+0000 7f94f3200640 1 -- 172.21.15.45:0/589054497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94f41608a0 msgr2=0x7f94f413bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:50.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.532+0000 7f94f3200640 1 --2- 172.21.15.45:0/589054497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94f41608a0 0x7f94f413bfd0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00efc0 tx=0x7f94ec00c490 comp rx=0 tx=0).stop 2024-04-16T16:22:50.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.532+0000 7f94f3200640 1 -- 172.21.15.45:0/589054497 shutdown_connections 2024-04-16T16:22:50.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.532+0000 7f94f3200640 1 --2- 172.21.15.45:0/589054497 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f94d406ca10 0x7f94d406eed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.532+0000 7f94f3200640 1 --2- 172.21.15.45:0/589054497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94f41608a0 0x7f94f413bfd0 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.532+0000 7f94f3200640 1 --2- 172.21.15.45:0/589054497 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94f4146220 0x7f94f41f0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.532+0000 7f94f3200640 1 --2- 172.21.15.45:0/589054497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94f406f7a0 0x7f94f415c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.532+0000 7f94f3200640 1 -- 172.21.15.45:0/589054497 >> 172.21.15.45:0/589054497 conn(0x7f94f406b930 msgr2=0x7f94f40b3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:50.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.532+0000 7f94f3200640 1 -- 172.21.15.45:0/589054497 shutdown_connections 2024-04-16T16:22:50.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.532+0000 7f94f3200640 1 -- 172.21.15.45:0/589054497 wait complete. 2024-04-16T16:22:50.552 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-16T16:22:50.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:22:50.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.782+0000 7f21e9200640 1 Processor -- start 2024-04-16T16:22:50.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.782+0000 7f21e9200640 1 -- start start 2024-04-16T16:22:50.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.783+0000 7f21e9200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21e4144ff0 0x7f21e40cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:50.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.784+0000 7f21e9200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f21e40c5d80 0x7f21e415d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:50.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.784+0000 7f21e9200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21e415c700 con 0x7f21e406dfb0 2024-04-16T16:22:50.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.784+0000 7f21e9200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f21e415d2f0 con 0x7f21e40c5d80 2024-04-16T16:22:50.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.784+0000 7f21e9200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f21e415c9c0 con 0x7f21e4144ff0 2024-04-16T16:22:50.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.784+0000 7f21e3e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f21e40c5d80 0x7f21e415d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:50.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.784+0000 7f21e3e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f21e40c5d80 0x7f21e415d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33080/0 (socket says 172.21.15.45:33080) 2024-04-16T16:22:50.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.784+0000 7f21e3e00640 1 -- 172.21.15.45:0/1020753790 learned_addr learned my addr 172.21.15.45:0/1020753790 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:50.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.784+0000 7f21e3400640 1 --2- 172.21.15.45:0/1020753790 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21e4144ff0 0x7f21e40cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:50.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.784+0000 7f21e3e00640 1 -- 172.21.15.45:0/1020753790 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21e4144ff0 msgr2=0x7f21e40cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:50.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.784+0000 7f21e3e00640 1 --2- 172.21.15.45:0/1020753790 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21e4144ff0 0x7f21e40cfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.784+0000 7f21e3e00640 1 -- 172.21.15.45:0/1020753790 >> v1:172.21.15.45:6789/0 conn(0x7f21e406dfb0 legacy=0x7f21e40c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:50.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.785+0000 7f21e3e00640 1 -- 172.21.15.45:0/1020753790 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e415cca0 con 0x7f21e40c5d80 2024-04-16T16:22:50.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.785+0000 7f21e3400640 1 --2- 172.21.15.45:0/1020753790 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21e4144ff0 0x7f21e40cfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:50.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.786+0000 7f21e3e00640 1 --2- 172.21.15.45:0/1020753790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e40c5d80 0x7f21e415d780 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009e50 tx=0x7f21dc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b72ba742495ee2c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:50.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.786+0000 7f21e2000640 1 -- 172.21.15.45:0/1020753790 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02cb00 con 0x7f21e40c5d80 2024-04-16T16:22:50.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.786+0000 7f21e2000640 1 -- 172.21.15.45:0/1020753790 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21dc02cca0 con 0x7f21e40c5d80 2024-04-16T16:22:50.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.786+0000 7f21e2000640 1 -- 172.21.15.45:0/1020753790 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21dc034610 con 0x7f21e40c5d80 2024-04-16T16:22:50.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.787+0000 7f21e9200640 1 -- 172.21.15.45:0/1020753790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e40c5d80 msgr2=0x7f21e415d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:50.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.787+0000 7f21e9200640 1 --2- 172.21.15.45:0/1020753790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e40c5d80 0x7f21e415d780 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009e50 tx=0x7f21dc02bf60 comp rx=0 tx=0).stop 2024-04-16T16:22:50.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.787+0000 7f21e9200640 1 -- 172.21.15.45:0/1020753790 shutdown_connections 2024-04-16T16:22:50.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.787+0000 7f21e9200640 1 --2- 172.21.15.45:0/1020753790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e40c5d80 0x7f21e415d780 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.787+0000 7f21e9200640 1 --2- 172.21.15.45:0/1020753790 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21e4144ff0 0x7f21e40cfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.787+0000 7f21e9200640 1 -- 172.21.15.45:0/1020753790 >> 172.21.15.45:0/1020753790 conn(0x7f21e406b930 msgr2=0x7f21e406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:50.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.787+0000 7f21e9200640 1 -- 172.21.15.45:0/1020753790 shutdown_connections 2024-04-16T16:22:50.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.787+0000 7f21e9200640 1 -- 172.21.15.45:0/1020753790 wait complete. 2024-04-16T16:22:50.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.789+0000 7f21e9200640 1 Processor -- start 2024-04-16T16:22:50.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.789+0000 7f21e9200640 1 -- start start 2024-04-16T16:22:50.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.790+0000 7f21e9200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21e406dfb0 0x7f21e415b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:50.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.790+0000 7f21e9200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21e40c5d80 0x7f21e4136940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:50.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.790+0000 7f21e9200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e4144ff0 0x7f21e413c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:50.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.790+0000 7f21e9200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f21e40d0270 con 0x7f21e4144ff0 2024-04-16T16:22:50.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.790+0000 7f21e9200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f21e41453d0 con 0x7f21e40c5d80 2024-04-16T16:22:50.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.790+0000 7f21e9200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f21e40d03f0 con 0x7f21e406dfb0 2024-04-16T16:22:50.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.790+0000 7f21e3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e4144ff0 0x7f21e413c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:50.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.790+0000 7f21e3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e4144ff0 0x7f21e413c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33090/0 (socket says 172.21.15.45:33090) 2024-04-16T16:22:50.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.790+0000 7f21e3e00640 1 -- 172.21.15.45:0/1806290069 learned_addr learned my addr 172.21.15.45:0/1806290069 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:50.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.791+0000 7f21e3400640 1 --2- 172.21.15.45:0/1806290069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21e406dfb0 0x7f21e415b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:50.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.791+0000 7f21e3e00640 1 -- 172.21.15.45:0/1806290069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21e406dfb0 msgr2=0x7f21e415b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:50.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.791+0000 7f21e2a00640 1 --2- 172.21.15.45:0/1806290069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21e40c5d80 0x7f21e4136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:50.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.791+0000 7f21e3e00640 1 --2- 172.21.15.45:0/1806290069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21e406dfb0 0x7f21e415b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.791+0000 7f21e3e00640 1 -- 172.21.15.45:0/1806290069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21e40c5d80 msgr2=0x7f21e4136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:50.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.791+0000 7f21e3e00640 1 --2- 172.21.15.45:0/1806290069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21e40c5d80 0x7f21e4136940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:50.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.791+0000 7f21e3e00640 1 -- 172.21.15.45:0/1806290069 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e4131a30 con 0x7f21e4144ff0 2024-04-16T16:22:50.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.791+0000 7f21e3e00640 1 --2- 172.21.15.45:0/1806290069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e4144ff0 0x7f21e413c930 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009e50 tx=0x7f21dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:50.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.791+0000 7f21db400640 1 -- 172.21.15.45:0/1806290069 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21dc043020 con 0x7f21e4144ff0 2024-04-16T16:22:50.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.791+0000 7f21db400640 1 -- 172.21.15.45:0/1806290069 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21dc02ce50 con 0x7f21e4144ff0 2024-04-16T16:22:50.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.791+0000 7f21db400640 1 -- 172.21.15.45:0/1806290069 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21dc009510 con 0x7f21e4144ff0 2024-04-16T16:22:50.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.791+0000 7f21e9200640 1 -- 172.21.15.45:0/1806290069 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21e4131cd0 con 0x7f21e4144ff0 2024-04-16T16:22:50.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.792+0000 7f21e9200640 1 -- 172.21.15.45:0/1806290069 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21e4132290 con 0x7f21e4144ff0 2024-04-16T16:22:50.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.793+0000 7f21e9200640 1 -- 172.21.15.45:0/1806290069 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21e4075780 con 0x7f21e4144ff0 2024-04-16T16:22:50.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.794+0000 7f21db400640 1 -- 172.21.15.45:0/1806290069 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f21dc046020 con 0x7f21e4144ff0 2024-04-16T16:22:50.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.794+0000 7f21db400640 1 --2- 172.21.15.45:0/1806290069 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f21c406ca10 0x7f21c406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:50.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.795+0000 7f21db400640 1 -- 172.21.15.45:0/1806290069 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f21dc0ae4c0 con 0x7f21e4144ff0 2024-04-16T16:22:50.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.795+0000 7f21e3400640 1 --2- 172.21.15.45:0/1806290069 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f21c406ca10 0x7f21c406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:50.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.796+0000 7f21e3400640 1 --2- 172.21.15.45:0/1806290069 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f21c406ca10 0x7f21c406eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f21d400b840 tx=0x7f21d4008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:50.800+0000 7f21db400640 1 -- 172.21.15.45:0/1806290069 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f21dc04d030 con 0x7f21e4144ff0 2024-04-16T16:22:51.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.003+0000 7f21e9200640 1 -- 172.21.15.45:0/1806290069 --> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f21e40b59c0 con 0x7f21c406ca10 2024-04-16T16:22:51.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.026+0000 7f21db400640 1 -- 172.21.15.45:0/1806290069 <== mgr.7591 v2:172.21.15.103:6832/1129061436 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252425 (secure 0 0 0) 0x7f21e40b59c0 con 0x7f21c406ca10 2024-04-16T16:22:51.029 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:51.030 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:22:51.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.032+0000 7f21e9200640 1 -- 172.21.15.45:0/1806290069 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f21c406ca10 msgr2=0x7f21c406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:51.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.032+0000 7f21e9200640 1 --2- 172.21.15.45:0/1806290069 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f21c406ca10 0x7f21c406eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f21d400b840 tx=0x7f21d4008040 comp rx=0 tx=0).stop 2024-04-16T16:22:51.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.032+0000 7f21e9200640 1 -- 172.21.15.45:0/1806290069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e4144ff0 msgr2=0x7f21e413c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:51.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.032+0000 7f21e9200640 1 --2- 172.21.15.45:0/1806290069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e4144ff0 0x7f21e413c930 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009e50 tx=0x7f21dc004a40 comp rx=0 tx=0).stop 2024-04-16T16:22:51.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.032+0000 7f21e9200640 1 -- 172.21.15.45:0/1806290069 shutdown_connections 2024-04-16T16:22:51.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.032+0000 7f21e9200640 1 --2- 172.21.15.45:0/1806290069 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f21c406ca10 0x7f21c406eed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.032+0000 7f21e9200640 1 --2- 172.21.15.45:0/1806290069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e4144ff0 0x7f21e413c930 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.032+0000 7f21e9200640 1 --2- 172.21.15.45:0/1806290069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21e40c5d80 0x7f21e4136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.032+0000 7f21e9200640 1 --2- 172.21.15.45:0/1806290069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21e406dfb0 0x7f21e415b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.032+0000 7f21e9200640 1 -- 172.21.15.45:0/1806290069 >> 172.21.15.45:0/1806290069 conn(0x7f21e406b930 msgr2=0x7f21e40b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:51.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.032+0000 7f21e9200640 1 -- 172.21.15.45:0/1806290069 shutdown_connections 2024-04-16T16:22:51.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.032+0000 7f21e9200640 1 -- 172.21.15.45:0/1806290069 wait complete. 2024-04-16T16:22:51.053 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-16T16:22:49.541202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":431,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4729,"ondisk_log_size":4729,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":282356,"kb_used_data":16264,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754692364,"statfs":{"total":773094113280,"available":772804980736,"internally_reserved":0,"allocated":16654336,"data_stored":8544120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.125863"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.489163+0000","last_change":"2024-04-16T16:22:46.488567+0000","last_active":"2024-04-16T16:22:46.489163+0000","last_peered":"2024-04-16T16:22:46.489163+0000","last_clean":"2024-04-16T16:22:46.489163+0000","last_became_active":"2024-04-16T16:22:46.485690+0000","last_became_peered":"2024-04-16T16:22:46.485690+0000","last_unstale":"2024-04-16T16:22:46.489163+0000","last_undegraded":"2024-04-16T16:22:46.489163+0000","last_fullsized":"2024-04-16T16:22:46.489163+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:59:47.867717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.483165+0000","last_change":"2024-04-16T16:22:46.479620+0000","last_active":"2024-04-16T16:22:46.483165+0000","last_peered":"2024-04-16T16:22:46.483165+0000","last_clean":"2024-04-16T16:22:46.483165+0000","last_became_active":"2024-04-16T16:22:46.477410+0000","last_became_peered":"2024-04-16T16:22:46.477410+0000","last_unstale":"2024-04-16T16:22:46.483165+0000","last_undegraded":"2024-04-16T16:22:46.483165+0000","last_fullsized":"2024-04-16T16:22:46.483165+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:35.308909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.486427+0000","last_change":"2024-04-16T16:22:46.480944+0000","last_active":"2024-04-16T16:22:46.486427+0000","last_peered":"2024-04-16T16:22:46.486427+0000","last_clean":"2024-04-16T16:22:46.486427+0000","last_became_active":"2024-04-16T16:22:46.480405+0000","last_became_peered":"2024-04-16T16:22:46.480405+0000","last_unstale":"2024-04-16T16:22:46.486427+0000","last_undegraded":"2024-04-16T16:22:46.486427+0000","last_fullsized":"2024-04-16T16:22:46.486427+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:32:48.252527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.480329+0000","last_change":"2024-04-16T16:22:46.477832+0000","last_active":"2024-04-16T16:22:46.480329+0000","last_peered":"2024-04-16T16:22:46.480329+0000","last_clean":"2024-04-16T16:22:46.480329+0000","last_became_active":"2024-04-16T16:22:46.477516+0000","last_became_peered":"2024-04-16T16:22:46.477516+0000","last_unstale":"2024-04-16T16:22:46.480329+0000","last_undegraded":"2024-04-16T16:22:46.480329+0000","last_fullsized":"2024-04-16T16:22:46.480329+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:13.979255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.485892+0000","last_change":"2024-04-16T16:22:46.483466+0000","last_active":"2024-04-16T16:22:46.485892+0000","last_peered":"2024-04-16T16:22:46.485892+0000","last_clean":"2024-04-16T16:22:46.485892+0000","last_became_active":"2024-04-16T16:22:46.481033+0000","last_became_peered":"2024-04-16T16:22:46.481033+0000","last_unstale":"2024-04-16T16:22:46.485892+0000","last_undegraded":"2024-04-16T16:22:46.485892+0000","last_fullsized":"2024-04-16T16:22:46.485892+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:13.979255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.472885+0000","last_change":"2024-04-16T16:22:46.463298+0000","last_active":"2024-04-16T16:22:46.472885+0000","last_peered":"2024-04-16T16:22:46.472885+0000","last_clean":"2024-04-16T16:22:46.472885+0000","last_became_active":"2024-04-16T16:22:46.462017+0000","last_became_peered":"2024-04-16T16:22:46.462017+0000","last_unstale":"2024-04-16T16:22:46.472885+0000","last_undegraded":"2024-04-16T16:22:46.472885+0000","last_fullsized":"2024-04-16T16:22:46.472885+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:58:14.925874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.471734+0000","last_change":"2024-04-16T16:22:46.462786+0000","last_active":"2024-04-16T16:22:46.471734+0000","last_peered":"2024-04-16T16:22:46.471734+0000","last_clean":"2024-04-16T16:22:46.471734+0000","last_became_active":"2024-04-16T16:22:46.461856+0000","last_became_peered":"2024-04-16T16:22:46.461856+0000","last_unstale":"2024-04-16T16:22:46.471734+0000","last_undegraded":"2024-04-16T16:22:46.471734+0000","last_fullsized":"2024-04-16T16:22:46.471734+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:31.478197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.488167+0000","last_change":"2024-04-16T16:22:46.486917+0000","last_active":"2024-04-16T16:22:46.488167+0000","last_peered":"2024-04-16T16:22:46.488167+0000","last_clean":"2024-04-16T16:22:46.488167+0000","last_became_active":"2024-04-16T16:22:46.485280+0000","last_became_peered":"2024-04-16T16:22:46.485280+0000","last_unstale":"2024-04-16T16:22:46.488167+0000","last_undegraded":"2024-04-16T16:22:46.488167+0000","last_fullsized":"2024-04-16T16:22:46.488167+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:53:06.786966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479938+0000","last_change":"2024-04-16T16:22:46.476513+0000","last_active":"2024-04-16T16:22:46.479938+0000","last_peered":"2024-04-16T16:22:46.479938+0000","last_clean":"2024-04-16T16:22:46.479938+0000","last_became_active":"2024-04-16T16:22:46.476125+0000","last_became_peered":"2024-04-16T16:22:46.476125+0000","last_unstale":"2024-04-16T16:22:46.479938+0000","last_undegraded":"2024-04-16T16:22:46.479938+0000","last_fullsized":"2024-04-16T16:22:46.479938+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:32:25.664832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.485587+0000","last_change":"2024-04-16T16:22:46.483287+0000","last_active":"2024-04-16T16:22:46.485587+0000","last_peered":"2024-04-16T16:22:46.485587+0000","last_clean":"2024-04-16T16:22:46.485587+0000","last_became_active":"2024-04-16T16:22:46.482975+0000","last_became_peered":"2024-04-16T16:22:46.482975+0000","last_unstale":"2024-04-16T16:22:46.485587+0000","last_undegraded":"2024-04-16T16:22:46.485587+0000","last_fullsized":"2024-04-16T16:22:46.485587+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:27.266104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.486131+0000","last_change":"2024-04-16T16:22:46.481301+0000","last_active":"2024-04-16T16:22:46.486131+0000","last_peered":"2024-04-16T16:22:46.486131+0000","last_clean":"2024-04-16T16:22:46.486131+0000","last_became_active":"2024-04-16T16:22:46.481000+0000","last_became_peered":"2024-04-16T16:22:46.481000+0000","last_unstale":"2024-04-16T16:22:46.486131+0000","last_undegraded":"2024-04-16T16:22:46.486131+0000","last_fullsized":"2024-04-16T16:22:46.486131+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:44:40.829325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479690+0000","last_change":"2024-04-16T16:22:46.476179+0000","last_active":"2024-04-16T16:22:46.479690+0000","last_peered":"2024-04-16T16:22:46.479690+0000","last_clean":"2024-04-16T16:22:46.479690+0000","last_became_active":"2024-04-16T16:22:46.475898+0000","last_became_peered":"2024-04-16T16:22:46.475898+0000","last_unstale":"2024-04-16T16:22:46.479690+0000","last_undegraded":"2024-04-16T16:22:46.479690+0000","last_fullsized":"2024-04-16T16:22:46.479690+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:08.580218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.487790+0000","last_change":"2024-04-16T16:22:46.482186+0000","last_active":"2024-04-16T16:22:46.487790+0000","last_peered":"2024-04-16T16:22:46.487790+0000","last_clean":"2024-04-16T16:22:46.487790+0000","last_became_active":"2024-04-16T16:22:46.481776+0000","last_became_peered":"2024-04-16T16:22:46.481776+0000","last_unstale":"2024-04-16T16:22:46.487790+0000","last_undegraded":"2024-04-16T16:22:46.487790+0000","last_fullsized":"2024-04-16T16:22:46.487790+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:43:05.150490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.482401+0000","last_change":"2024-04-16T16:22:46.477861+0000","last_active":"2024-04-16T16:22:46.482401+0000","last_peered":"2024-04-16T16:22:46.482401+0000","last_clean":"2024-04-16T16:22:46.482401+0000","last_became_active":"2024-04-16T16:22:46.476486+0000","last_became_peered":"2024-04-16T16:22:46.476486+0000","last_unstale":"2024-04-16T16:22:46.482401+0000","last_undegraded":"2024-04-16T16:22:46.482401+0000","last_fullsized":"2024-04-16T16:22:46.482401+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:34:35.154021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.472708+0000","last_change":"2024-04-16T16:22:46.468319+0000","last_active":"2024-04-16T16:22:46.472708+0000","last_peered":"2024-04-16T16:22:46.472708+0000","last_clean":"2024-04-16T16:22:46.472708+0000","last_became_active":"2024-04-16T16:22:46.463117+0000","last_became_peered":"2024-04-16T16:22:46.463117+0000","last_unstale":"2024-04-16T16:22:46.472708+0000","last_undegraded":"2024-04-16T16:22:46.472708+0000","last_fullsized":"2024-04-16T16:22:46.472708+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:51:03.090923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479041+0000","last_change":"2024-04-16T16:22:46.467824+0000","last_active":"2024-04-16T16:22:46.479041+0000","last_peered":"2024-04-16T16:22:46.479041+0000","last_clean":"2024-04-16T16:22:46.479041+0000","last_became_active":"2024-04-16T16:22:46.467274+0000","last_became_peered":"2024-04-16T16:22:46.467274+0000","last_unstale":"2024-04-16T16:22:46.479041+0000","last_undegraded":"2024-04-16T16:22:46.479041+0000","last_fullsized":"2024-04-16T16:22:46.479041+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:11.331337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479749+0000","last_change":"2024-04-16T16:22:46.475037+0000","last_active":"2024-04-16T16:22:46.479749+0000","last_peered":"2024-04-16T16:22:46.479749+0000","last_clean":"2024-04-16T16:22:46.479749+0000","last_became_active":"2024-04-16T16:22:46.474757+0000","last_became_peered":"2024-04-16T16:22:46.474757+0000","last_unstale":"2024-04-16T16:22:46.479749+0000","last_undegraded":"2024-04-16T16:22:46.479749+0000","last_fullsized":"2024-04-16T16:22:46.479749+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:39:23.773370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475658+0000","last_change":"2024-04-16T16:22:46.463139+0000","last_active":"2024-04-16T16:22:46.475658+0000","last_peered":"2024-04-16T16:22:46.475658+0000","last_clean":"2024-04-16T16:22:46.475658+0000","last_became_active":"2024-04-16T16:22:46.462857+0000","last_became_peered":"2024-04-16T16:22:46.462857+0000","last_unstale":"2024-04-16T16:22:46.475658+0000","last_undegraded":"2024-04-16T16:22:46.475658+0000","last_fullsized":"2024-04-16T16:22:46.475658+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:18.473865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.462845+0000","last_change":"2024-04-16T16:22:46.457033+0000","last_active":"2024-04-16T16:22:46.462845+0000","last_peered":"2024-04-16T16:22:46.462845+0000","last_clean":"2024-04-16T16:22:46.462845+0000","last_became_active":"2024-04-16T16:22:46.456756+0000","last_became_peered":"2024-04-16T16:22:46.456756+0000","last_unstale":"2024-04-16T16:22:46.462845+0000","last_undegraded":"2024-04-16T16:22:46.462845+0000","last_fullsized":"2024-04-16T16:22:46.462845+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:37:05.350527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.487464+0000","last_change":"2024-04-16T16:22:46.482119+0000","last_active":"2024-04-16T16:22:46.487464+0000","last_peered":"2024-04-16T16:22:46.487464+0000","last_clean":"2024-04-16T16:22:46.487464+0000","last_became_active":"2024-04-16T16:22:46.479857+0000","last_became_peered":"2024-04-16T16:22:46.479857+0000","last_unstale":"2024-04-16T16:22:46.487464+0000","last_undegraded":"2024-04-16T16:22:46.487464+0000","last_fullsized":"2024-04-16T16:22:46.487464+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:02.607399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.481181+0000","last_change":"2024-04-16T16:22:46.479160+0000","last_active":"2024-04-16T16:22:46.481181+0000","last_peered":"2024-04-16T16:22:46.481181+0000","last_clean":"2024-04-16T16:22:46.481181+0000","last_became_active":"2024-04-16T16:22:46.477081+0000","last_became_peered":"2024-04-16T16:22:46.477081+0000","last_unstale":"2024-04-16T16:22:46.481181+0000","last_undegraded":"2024-04-16T16:22:46.481181+0000","last_fullsized":"2024-04-16T16:22:46.481181+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:01:27.421911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479731+0000","last_change":"2024-04-16T16:22:46.473896+0000","last_active":"2024-04-16T16:22:46.479731+0000","last_peered":"2024-04-16T16:22:46.479731+0000","last_clean":"2024-04-16T16:22:46.479731+0000","last_became_active":"2024-04-16T16:22:46.473621+0000","last_became_peered":"2024-04-16T16:22:46.473621+0000","last_unstale":"2024-04-16T16:22:46.479731+0000","last_undegraded":"2024-04-16T16:22:46.479731+0000","last_fullsized":"2024-04-16T16:22:46.479731+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:55.094277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479910+0000","last_change":"2024-04-16T16:22:46.475807+0000","last_active":"2024-04-16T16:22:46.479910+0000","last_peered":"2024-04-16T16:22:46.479910+0000","last_clean":"2024-04-16T16:22:46.479910+0000","last_became_active":"2024-04-16T16:22:46.473183+0000","last_became_peered":"2024-04-16T16:22:46.473183+0000","last_unstale":"2024-04-16T16:22:46.479910+0000","last_undegraded":"2024-04-16T16:22:46.479910+0000","last_fullsized":"2024-04-16T16:22:46.479910+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:50:46.448869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479850+0000","last_change":"2024-04-16T16:22:46.477670+0000","last_active":"2024-04-16T16:22:46.479850+0000","last_peered":"2024-04-16T16:22:46.479850+0000","last_clean":"2024-04-16T16:22:46.479850+0000","last_became_active":"2024-04-16T16:22:46.477303+0000","last_became_peered":"2024-04-16T16:22:46.477303+0000","last_unstale":"2024-04-16T16:22:46.479850+0000","last_undegraded":"2024-04-16T16:22:46.479850+0000","last_fullsized":"2024-04-16T16:22:46.479850+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:05:36.568682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.485748+0000","last_change":"2024-04-16T16:22:46.481055+0000","last_active":"2024-04-16T16:22:46.485748+0000","last_peered":"2024-04-16T16:22:46.485748+0000","last_clean":"2024-04-16T16:22:46.485748+0000","last_became_active":"2024-04-16T16:22:46.480641+0000","last_became_peered":"2024-04-16T16:22:46.480641+0000","last_unstale":"2024-04-16T16:22:46.485748+0000","last_undegraded":"2024-04-16T16:22:46.485748+0000","last_fullsized":"2024-04-16T16:22:46.485748+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:46:58.611028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.478755+0000","last_change":"2024-04-16T16:22:46.467282+0000","last_active":"2024-04-16T16:22:46.478755+0000","last_peered":"2024-04-16T16:22:46.478755+0000","last_clean":"2024-04-16T16:22:46.478755+0000","last_became_active":"2024-04-16T16:22:46.466756+0000","last_became_peered":"2024-04-16T16:22:46.466756+0000","last_unstale":"2024-04-16T16:22:46.478755+0000","last_undegraded":"2024-04-16T16:22:46.478755+0000","last_fullsized":"2024-04-16T16:22:46.478755+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:22.841005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.487969+0000","last_change":"2024-04-16T16:22:46.484066+0000","last_active":"2024-04-16T16:22:46.487969+0000","last_peered":"2024-04-16T16:22:46.487969+0000","last_clean":"2024-04-16T16:22:46.487969+0000","last_became_active":"2024-04-16T16:22:46.483654+0000","last_became_peered":"2024-04-16T16:22:46.483654+0000","last_unstale":"2024-04-16T16:22:46.487969+0000","last_undegraded":"2024-04-16T16:22:46.487969+0000","last_fullsized":"2024-04-16T16:22:46.487969+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:52:58.419267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.485641+0000","last_change":"2024-04-16T16:22:46.484111+0000","last_active":"2024-04-16T16:22:46.485641+0000","last_peered":"2024-04-16T16:22:46.485641+0000","last_clean":"2024-04-16T16:22:46.485641+0000","last_became_active":"2024-04-16T16:22:46.483750+0000","last_became_peered":"2024-04-16T16:22:46.483750+0000","last_unstale":"2024-04-16T16:22:46.485641+0000","last_undegraded":"2024-04-16T16:22:46.485641+0000","last_fullsized":"2024-04-16T16:22:46.485641+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:53:06.786966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.487385+0000","last_change":"2024-04-16T16:22:46.483339+0000","last_active":"2024-04-16T16:22:46.487385+0000","last_peered":"2024-04-16T16:22:46.487385+0000","last_clean":"2024-04-16T16:22:46.487385+0000","last_became_active":"2024-04-16T16:22:46.481358+0000","last_became_peered":"2024-04-16T16:22:46.481358+0000","last_unstale":"2024-04-16T16:22:46.487385+0000","last_undegraded":"2024-04-16T16:22:46.487385+0000","last_fullsized":"2024-04-16T16:22:46.487385+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:30.402337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475943+0000","last_change":"2024-04-16T16:22:46.464311+0000","last_active":"2024-04-16T16:22:46.475943+0000","last_peered":"2024-04-16T16:22:46.475943+0000","last_clean":"2024-04-16T16:22:46.475943+0000","last_became_active":"2024-04-16T16:22:46.463536+0000","last_became_peered":"2024-04-16T16:22:46.463536+0000","last_unstale":"2024-04-16T16:22:46.475943+0000","last_undegraded":"2024-04-16T16:22:46.475943+0000","last_fullsized":"2024-04-16T16:22:46.475943+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:13.979255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475259+0000","last_change":"2024-04-16T16:22:46.460441+0000","last_active":"2024-04-16T16:22:46.475259+0000","last_peered":"2024-04-16T16:22:46.475259+0000","last_clean":"2024-04-16T16:22:46.475259+0000","last_became_active":"2024-04-16T16:22:46.458962+0000","last_became_peered":"2024-04-16T16:22:46.458962+0000","last_unstale":"2024-04-16T16:22:46.475259+0000","last_undegraded":"2024-04-16T16:22:46.475259+0000","last_fullsized":"2024-04-16T16:22:46.475259+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:44:40.829325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475972+0000","last_change":"2024-04-16T16:22:46.468514+0000","last_active":"2024-04-16T16:22:46.475972+0000","last_peered":"2024-04-16T16:22:46.475972+0000","last_clean":"2024-04-16T16:22:46.475972+0000","last_became_active":"2024-04-16T16:22:46.464851+0000","last_became_peered":"2024-04-16T16:22:46.464851+0000","last_unstale":"2024-04-16T16:22:46.475972+0000","last_undegraded":"2024-04-16T16:22:46.475972+0000","last_fullsized":"2024-04-16T16:22:46.475972+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:22.841005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.478711+0000","last_change":"2024-04-16T16:22:46.467039+0000","last_active":"2024-04-16T16:22:46.478711+0000","last_peered":"2024-04-16T16:22:46.478711+0000","last_clean":"2024-04-16T16:22:46.478711+0000","last_became_active":"2024-04-16T16:22:46.466366+0000","last_became_peered":"2024-04-16T16:22:46.466366+0000","last_unstale":"2024-04-16T16:22:46.478711+0000","last_undegraded":"2024-04-16T16:22:46.478711+0000","last_fullsized":"2024-04-16T16:22:46.478711+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:53:48.166033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"155'4","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.488480+0000","last_change":"2024-04-16T16:22:45.470039+0000","last_active":"2024-04-16T16:22:46.488480+0000","last_peered":"2024-04-16T16:22:46.488480+0000","last_clean":"2024-04-16T16:22:46.488480+0000","last_became_active":"2024-04-16T16:22:45.467695+0000","last_became_peered":"2024-04-16T16:22:45.467695+0000","last_unstale":"2024-04-16T16:22:46.488480+0000","last_undegraded":"2024-04-16T16:22:46.488480+0000","last_fullsized":"2024-04-16T16:22:46.488480+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:21.824987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.478465+0000","last_change":"2024-04-16T16:22:46.466378+0000","last_active":"2024-04-16T16:22:46.478465+0000","last_peered":"2024-04-16T16:22:46.478465+0000","last_clean":"2024-04-16T16:22:46.478465+0000","last_became_active":"2024-04-16T16:22:46.462099+0000","last_became_peered":"2024-04-16T16:22:46.462099+0000","last_unstale":"2024-04-16T16:22:46.478465+0000","last_undegraded":"2024-04-16T16:22:46.478465+0000","last_fullsized":"2024-04-16T16:22:46.478465+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:53:06.786966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.484979+0000","last_change":"2024-04-16T16:22:45.464685+0000","last_active":"2024-04-16T16:22:46.484979+0000","last_peered":"2024-04-16T16:22:46.484979+0000","last_clean":"2024-04-16T16:22:46.484979+0000","last_became_active":"2024-04-16T16:22:45.464379+0000","last_became_peered":"2024-04-16T16:22:45.464379+0000","last_unstale":"2024-04-16T16:22:46.484979+0000","last_undegraded":"2024-04-16T16:22:46.484979+0000","last_fullsized":"2024-04-16T16:22:46.484979+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:05:35.552664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.464408+0000","last_change":"2024-04-16T16:22:46.456302+0000","last_active":"2024-04-16T16:22:46.464408+0000","last_peered":"2024-04-16T16:22:46.464408+0000","last_clean":"2024-04-16T16:22:46.464408+0000","last_became_active":"2024-04-16T16:22:46.454610+0000","last_became_peered":"2024-04-16T16:22:46.454610+0000","last_unstale":"2024-04-16T16:22:46.464408+0000","last_undegraded":"2024-04-16T16:22:46.464408+0000","last_fullsized":"2024-04-16T16:22:46.464408+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:53:15.712215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.484025+0000","last_change":"2024-04-16T16:22:45.469877+0000","last_active":"2024-04-16T16:22:46.484025+0000","last_peered":"2024-04-16T16:22:46.484025+0000","last_clean":"2024-04-16T16:22:46.484025+0000","last_became_active":"2024-04-16T16:22:45.469537+0000","last_became_peered":"2024-04-16T16:22:45.469537+0000","last_unstale":"2024-04-16T16:22:46.484025+0000","last_undegraded":"2024-04-16T16:22:46.484025+0000","last_fullsized":"2024-04-16T16:22:46.484025+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:54.078259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.485663+0000","last_change":"2024-04-16T16:22:46.480073+0000","last_active":"2024-04-16T16:22:46.485663+0000","last_peered":"2024-04-16T16:22:46.485663+0000","last_clean":"2024-04-16T16:22:46.485663+0000","last_became_active":"2024-04-16T16:22:46.479206+0000","last_became_peered":"2024-04-16T16:22:46.479206+0000","last_unstale":"2024-04-16T16:22:46.485663+0000","last_undegraded":"2024-04-16T16:22:46.485663+0000","last_fullsized":"2024-04-16T16:22:46.485663+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:46.979782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.474697+0000","last_change":"2024-04-16T16:22:45.462903+0000","last_active":"2024-04-16T16:22:46.474697+0000","last_peered":"2024-04-16T16:22:46.474697+0000","last_clean":"2024-04-16T16:22:46.474697+0000","last_became_active":"2024-04-16T16:22:45.462170+0000","last_became_peered":"2024-04-16T16:22:45.462170+0000","last_unstale":"2024-04-16T16:22:46.474697+0000","last_undegraded":"2024-04-16T16:22:46.474697+0000","last_fullsized":"2024-04-16T16:22:46.474697+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:07.564200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.465111+0000","last_change":"2024-04-16T16:22:46.460202+0000","last_active":"2024-04-16T16:22:46.465111+0000","last_peered":"2024-04-16T16:22:46.465111+0000","last_clean":"2024-04-16T16:22:46.465111+0000","last_became_active":"2024-04-16T16:22:46.459876+0000","last_became_peered":"2024-04-16T16:22:46.459876+0000","last_unstale":"2024-04-16T16:22:46.465111+0000","last_undegraded":"2024-04-16T16:22:46.465111+0000","last_fullsized":"2024-04-16T16:22:46.465111+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:05:07.007726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.484235+0000","last_change":"2024-04-16T16:22:45.472425+0000","last_active":"2024-04-16T16:22:46.484235+0000","last_peered":"2024-04-16T16:22:46.484235+0000","last_clean":"2024-04-16T16:22:46.484235+0000","last_became_active":"2024-04-16T16:22:45.472020+0000","last_became_peered":"2024-04-16T16:22:45.472020+0000","last_unstale":"2024-04-16T16:22:46.484235+0000","last_undegraded":"2024-04-16T16:22:46.484235+0000","last_fullsized":"2024-04-16T16:22:46.484235+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:48:51.793606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476249+0000","last_change":"2024-04-16T16:22:46.470665+0000","last_active":"2024-04-16T16:22:46.476249+0000","last_peered":"2024-04-16T16:22:46.476249+0000","last_clean":"2024-04-16T16:22:46.476249+0000","last_became_active":"2024-04-16T16:22:46.467898+0000","last_became_peered":"2024-04-16T16:22:46.467898+0000","last_unstale":"2024-04-16T16:22:46.476249+0000","last_undegraded":"2024-04-16T16:22:46.476249+0000","last_fullsized":"2024-04-16T16:22:46.476249+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:35.308909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479141+0000","last_change":"2024-04-16T16:22:45.462429+0000","last_active":"2024-04-16T16:22:46.479141+0000","last_peered":"2024-04-16T16:22:46.479141+0000","last_clean":"2024-04-16T16:22:46.479141+0000","last_became_active":"2024-04-16T16:22:45.461674+0000","last_became_peered":"2024-04-16T16:22:45.461674+0000","last_unstale":"2024-04-16T16:22:46.479141+0000","last_undegraded":"2024-04-16T16:22:46.479141+0000","last_fullsized":"2024-04-16T16:22:46.479141+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:29.386319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476205+0000","last_change":"2024-04-16T16:22:46.465672+0000","last_active":"2024-04-16T16:22:46.476205+0000","last_peered":"2024-04-16T16:22:46.476205+0000","last_clean":"2024-04-16T16:22:46.476205+0000","last_became_active":"2024-04-16T16:22:46.465329+0000","last_became_peered":"2024-04-16T16:22:46.465329+0000","last_unstale":"2024-04-16T16:22:46.476205+0000","last_undegraded":"2024-04-16T16:22:46.476205+0000","last_fullsized":"2024-04-16T16:22:46.476205+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:05:36.568682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.465610+0000","last_change":"2024-04-16T16:22:45.466868+0000","last_active":"2024-04-16T16:22:46.465610+0000","last_peered":"2024-04-16T16:22:46.465610+0000","last_clean":"2024-04-16T16:22:46.465610+0000","last_became_active":"2024-04-16T16:22:45.466591+0000","last_became_peered":"2024-04-16T16:22:45.466591+0000","last_unstale":"2024-04-16T16:22:46.465610+0000","last_undegraded":"2024-04-16T16:22:46.465610+0000","last_fullsized":"2024-04-16T16:22:46.465610+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:25.557203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.466619+0000","last_change":"2024-04-16T16:22:46.460701+0000","last_active":"2024-04-16T16:22:46.466619+0000","last_peered":"2024-04-16T16:22:46.466619+0000","last_clean":"2024-04-16T16:22:46.466619+0000","last_became_active":"2024-04-16T16:22:46.460420+0000","last_became_peered":"2024-04-16T16:22:46.460420+0000","last_unstale":"2024-04-16T16:22:46.466619+0000","last_undegraded":"2024-04-16T16:22:46.466619+0000","last_fullsized":"2024-04-16T16:22:46.466619+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:22:47.958026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.451708+0000","last_change":"2024-04-16T16:22:45.456041+0000","last_active":"2024-04-16T16:22:46.451708+0000","last_peered":"2024-04-16T16:22:46.451708+0000","last_clean":"2024-04-16T16:22:46.451708+0000","last_became_active":"2024-04-16T16:22:45.455767+0000","last_became_peered":"2024-04-16T16:22:45.455767+0000","last_unstale":"2024-04-16T16:22:46.451708+0000","last_undegraded":"2024-04-16T16:22:46.451708+0000","last_fullsized":"2024-04-16T16:22:46.451708+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:26:37.373159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475982+0000","last_change":"2024-04-16T16:22:46.467743+0000","last_active":"2024-04-16T16:22:46.475982+0000","last_peered":"2024-04-16T16:22:46.475982+0000","last_clean":"2024-04-16T16:22:46.475982+0000","last_became_active":"2024-04-16T16:22:46.465149+0000","last_became_peered":"2024-04-16T16:22:46.465149+0000","last_unstale":"2024-04-16T16:22:46.475982+0000","last_undegraded":"2024-04-16T16:22:46.475982+0000","last_fullsized":"2024-04-16T16:22:46.475982+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:53:48.166033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.442180+0000","last_change":"2024-04-16T16:22:45.467985+0000","last_active":"2024-04-16T16:22:46.442180+0000","last_peered":"2024-04-16T16:22:46.442180+0000","last_clean":"2024-04-16T16:22:46.442180+0000","last_became_active":"2024-04-16T16:22:45.467704+0000","last_became_peered":"2024-04-16T16:22:45.467704+0000","last_unstale":"2024-04-16T16:22:46.442180+0000","last_undegraded":"2024-04-16T16:22:46.442180+0000","last_fullsized":"2024-04-16T16:22:46.442180+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:58.753768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476055+0000","last_change":"2024-04-16T16:22:46.473841+0000","last_active":"2024-04-16T16:22:46.476055+0000","last_peered":"2024-04-16T16:22:46.476055+0000","last_clean":"2024-04-16T16:22:46.476055+0000","last_became_active":"2024-04-16T16:22:46.473500+0000","last_became_peered":"2024-04-16T16:22:46.473500+0000","last_unstale":"2024-04-16T16:22:46.476055+0000","last_undegraded":"2024-04-16T16:22:46.476055+0000","last_fullsized":"2024-04-16T16:22:46.476055+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:51:03.090923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"155'3","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.467583+0000","last_change":"2024-04-16T16:22:45.461376+0000","last_active":"2024-04-16T16:22:46.467583+0000","last_peered":"2024-04-16T16:22:46.467583+0000","last_clean":"2024-04-16T16:22:46.467583+0000","last_became_active":"2024-04-16T16:22:45.461088+0000","last_became_peered":"2024-04-16T16:22:45.461088+0000","last_unstale":"2024-04-16T16:22:46.467583+0000","last_undegraded":"2024-04-16T16:22:46.467583+0000","last_fullsized":"2024-04-16T16:22:46.467583+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:43:07.885858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.477917+0000","last_change":"2024-04-16T16:22:46.473263+0000","last_active":"2024-04-16T16:22:46.477917+0000","last_peered":"2024-04-16T16:22:46.477917+0000","last_clean":"2024-04-16T16:22:46.477917+0000","last_became_active":"2024-04-16T16:22:46.472979+0000","last_became_peered":"2024-04-16T16:22:46.472979+0000","last_unstale":"2024-04-16T16:22:46.477917+0000","last_undegraded":"2024-04-16T16:22:46.477917+0000","last_fullsized":"2024-04-16T16:22:46.477917+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:22:33.768188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"155'6","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.490256+0000","last_change":"2024-04-16T16:22:45.462782+0000","last_active":"2024-04-16T16:22:46.490256+0000","last_peered":"2024-04-16T16:22:46.490256+0000","last_clean":"2024-04-16T16:22:46.490256+0000","last_became_active":"2024-04-16T16:22:45.461583+0000","last_became_peered":"2024-04-16T16:22:45.461583+0000","last_unstale":"2024-04-16T16:22:46.490256+0000","last_undegraded":"2024-04-16T16:22:46.490256+0000","last_fullsized":"2024-04-16T16:22:46.490256+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476052+0000","last_change":"2024-04-16T16:22:46.466170+0000","last_active":"2024-04-16T16:22:46.476052+0000","last_peered":"2024-04-16T16:22:46.476052+0000","last_clean":"2024-04-16T16:22:46.476052+0000","last_became_active":"2024-04-16T16:22:46.465894+0000","last_became_peered":"2024-04-16T16:22:46.465894+0000","last_unstale":"2024-04-16T16:22:46.476052+0000","last_undegraded":"2024-04-16T16:22:46.476052+0000","last_fullsized":"2024-04-16T16:22:46.476052+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:47.034586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.477731+0000","last_change":"2024-04-16T16:22:45.466273+0000","last_active":"2024-04-16T16:22:46.477731+0000","last_peered":"2024-04-16T16:22:46.477731+0000","last_clean":"2024-04-16T16:22:46.477731+0000","last_became_active":"2024-04-16T16:22:45.465893+0000","last_became_peered":"2024-04-16T16:22:45.465893+0000","last_unstale":"2024-04-16T16:22:46.477731+0000","last_undegraded":"2024-04-16T16:22:46.477731+0000","last_fullsized":"2024-04-16T16:22:46.477731+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:52:57.403249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479098+0000","last_change":"2024-04-16T16:22:46.474507+0000","last_active":"2024-04-16T16:22:46.479098+0000","last_peered":"2024-04-16T16:22:46.479098+0000","last_clean":"2024-04-16T16:22:46.479098+0000","last_became_active":"2024-04-16T16:22:46.474146+0000","last_became_peered":"2024-04-16T16:22:46.474146+0000","last_unstale":"2024-04-16T16:22:46.479098+0000","last_undegraded":"2024-04-16T16:22:46.479098+0000","last_fullsized":"2024-04-16T16:22:46.479098+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:59:47.867717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.449766+0000","last_change":"2024-04-16T16:22:45.463747+0000","last_active":"2024-04-16T16:22:46.449766+0000","last_peered":"2024-04-16T16:22:46.449766+0000","last_clean":"2024-04-16T16:22:46.449766+0000","last_became_active":"2024-04-16T16:22:45.462875+0000","last_became_peered":"2024-04-16T16:22:45.462875+0000","last_unstale":"2024-04-16T16:22:46.449766+0000","last_undegraded":"2024-04-16T16:22:46.449766+0000","last_fullsized":"2024-04-16T16:22:46.449766+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:29:33.964433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476854+0000","last_change":"2024-04-16T16:22:46.473128+0000","last_active":"2024-04-16T16:22:46.476854+0000","last_peered":"2024-04-16T16:22:46.476854+0000","last_clean":"2024-04-16T16:22:46.476854+0000","last_became_active":"2024-04-16T16:22:46.472859+0000","last_became_peered":"2024-04-16T16:22:46.472859+0000","last_unstale":"2024-04-16T16:22:46.476854+0000","last_undegraded":"2024-04-16T16:22:46.476854+0000","last_fullsized":"2024-04-16T16:22:46.476854+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:44:40.829325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.454247+0000","last_change":"2024-04-16T16:22:45.466908+0000","last_active":"2024-04-16T16:22:46.454247+0000","last_peered":"2024-04-16T16:22:46.454247+0000","last_clean":"2024-04-16T16:22:46.454247+0000","last_became_active":"2024-04-16T16:22:45.466535+0000","last_became_peered":"2024-04-16T16:22:45.466535+0000","last_unstale":"2024-04-16T16:22:46.454247+0000","last_undegraded":"2024-04-16T16:22:46.454247+0000","last_fullsized":"2024-04-16T16:22:46.454247+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:25.557203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.487178+0000","last_change":"2024-04-16T16:22:46.483216+0000","last_active":"2024-04-16T16:22:46.487178+0000","last_peered":"2024-04-16T16:22:46.487178+0000","last_clean":"2024-04-16T16:22:46.487178+0000","last_became_active":"2024-04-16T16:22:46.482943+0000","last_became_peered":"2024-04-16T16:22:46.482943+0000","last_unstale":"2024-04-16T16:22:46.487178+0000","last_undegraded":"2024-04-16T16:22:46.487178+0000","last_fullsized":"2024-04-16T16:22:46.487178+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:17:28.908308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"155'8","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.484924+0000","last_change":"2024-04-16T16:22:45.461068+0000","last_active":"2024-04-16T16:22:46.484924+0000","last_peered":"2024-04-16T16:22:46.484924+0000","last_clean":"2024-04-16T16:22:46.484924+0000","last_became_active":"2024-04-16T16:22:45.459760+0000","last_became_peered":"2024-04-16T16:22:45.459760+0000","last_unstale":"2024-04-16T16:22:46.484924+0000","last_undegraded":"2024-04-16T16:22:46.484924+0000","last_fullsized":"2024-04-16T16:22:46.484924+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:32:24.648814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.477198+0000","last_change":"2024-04-16T16:22:46.473695+0000","last_active":"2024-04-16T16:22:46.477198+0000","last_peered":"2024-04-16T16:22:46.477198+0000","last_clean":"2024-04-16T16:22:46.477198+0000","last_became_active":"2024-04-16T16:22:46.471894+0000","last_became_peered":"2024-04-16T16:22:46.471894+0000","last_unstale":"2024-04-16T16:22:46.477198+0000","last_undegraded":"2024-04-16T16:22:46.477198+0000","last_fullsized":"2024-04-16T16:22:46.477198+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:27.266104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.484686+0000","last_change":"2024-04-16T16:22:45.469335+0000","last_active":"2024-04-16T16:22:46.484686+0000","last_peered":"2024-04-16T16:22:46.484686+0000","last_clean":"2024-04-16T16:22:46.484686+0000","last_became_active":"2024-04-16T16:22:45.468966+0000","last_became_peered":"2024-04-16T16:22:45.468966+0000","last_unstale":"2024-04-16T16:22:46.484686+0000","last_undegraded":"2024-04-16T16:22:46.484686+0000","last_fullsized":"2024-04-16T16:22:46.484686+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:09.300484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476015+0000","last_change":"2024-04-16T16:22:46.471925+0000","last_active":"2024-04-16T16:22:46.476015+0000","last_peered":"2024-04-16T16:22:46.476015+0000","last_clean":"2024-04-16T16:22:46.476015+0000","last_became_active":"2024-04-16T16:22:46.470405+0000","last_became_peered":"2024-04-16T16:22:46.470405+0000","last_unstale":"2024-04-16T16:22:46.476015+0000","last_undegraded":"2024-04-16T16:22:46.476015+0000","last_fullsized":"2024-04-16T16:22:46.476015+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:11.331337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.480955+0000","last_change":"2024-04-16T16:22:45.470272+0000","last_active":"2024-04-16T16:22:46.480955+0000","last_peered":"2024-04-16T16:22:46.480955+0000","last_clean":"2024-04-16T16:22:46.480955+0000","last_became_active":"2024-04-16T16:22:45.469959+0000","last_became_peered":"2024-04-16T16:22:45.469959+0000","last_unstale":"2024-04-16T16:22:46.480955+0000","last_undegraded":"2024-04-16T16:22:46.480955+0000","last_fullsized":"2024-04-16T16:22:46.480955+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:34:03.813486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.474133+0000","last_change":"2024-04-16T16:22:46.462951+0000","last_active":"2024-04-16T16:22:46.474133+0000","last_peered":"2024-04-16T16:22:46.474133+0000","last_clean":"2024-04-16T16:22:46.474133+0000","last_became_active":"2024-04-16T16:22:46.462662+0000","last_became_peered":"2024-04-16T16:22:46.462662+0000","last_unstale":"2024-04-16T16:22:46.474133+0000","last_undegraded":"2024-04-16T16:22:46.474133+0000","last_fullsized":"2024-04-16T16:22:46.474133+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:32:25.664832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.482804+0000","last_change":"2024-04-16T16:22:45.462862+0000","last_active":"2024-04-16T16:22:46.482804+0000","last_peered":"2024-04-16T16:22:46.482804+0000","last_clean":"2024-04-16T16:22:46.482804+0000","last_became_active":"2024-04-16T16:22:45.462197+0000","last_became_peered":"2024-04-16T16:22:45.462197+0000","last_unstale":"2024-04-16T16:22:46.482804+0000","last_undegraded":"2024-04-16T16:22:46.482804+0000","last_fullsized":"2024-04-16T16:22:46.482804+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:55.697931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.478880+0000","last_change":"2024-04-16T16:22:46.475567+0000","last_active":"2024-04-16T16:22:46.478880+0000","last_peered":"2024-04-16T16:22:46.478880+0000","last_clean":"2024-04-16T16:22:46.478880+0000","last_became_active":"2024-04-16T16:22:46.475232+0000","last_became_peered":"2024-04-16T16:22:46.475232+0000","last_unstale":"2024-04-16T16:22:46.478880+0000","last_undegraded":"2024-04-16T16:22:46.478880+0000","last_fullsized":"2024-04-16T16:22:46.478880+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:01.225311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.449795+0000","last_change":"2024-04-16T16:22:45.466893+0000","last_active":"2024-04-16T16:22:46.449795+0000","last_peered":"2024-04-16T16:22:46.449795+0000","last_clean":"2024-04-16T16:22:46.449795+0000","last_became_active":"2024-04-16T16:22:45.466614+0000","last_became_peered":"2024-04-16T16:22:45.466614+0000","last_unstale":"2024-04-16T16:22:46.449795+0000","last_undegraded":"2024-04-16T16:22:46.449795+0000","last_fullsized":"2024-04-16T16:22:46.449795+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:53:47.150015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.487053+0000","last_change":"2024-04-16T16:22:46.482036+0000","last_active":"2024-04-16T16:22:46.487053+0000","last_peered":"2024-04-16T16:22:46.487053+0000","last_clean":"2024-04-16T16:22:46.487053+0000","last_became_active":"2024-04-16T16:22:46.481293+0000","last_became_peered":"2024-04-16T16:22:46.481293+0000","last_unstale":"2024-04-16T16:22:46.487053+0000","last_undegraded":"2024-04-16T16:22:46.487053+0000","last_fullsized":"2024-04-16T16:22:46.487053+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:10.466744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.484240+0000","last_change":"2024-04-16T16:22:45.466947+0000","last_active":"2024-04-16T16:22:46.484240+0000","last_peered":"2024-04-16T16:22:46.484240+0000","last_clean":"2024-04-16T16:22:46.484240+0000","last_became_active":"2024-04-16T16:22:45.466624+0000","last_became_peered":"2024-04-16T16:22:45.466624+0000","last_unstale":"2024-04-16T16:22:46.484240+0000","last_undegraded":"2024-04-16T16:22:46.484240+0000","last_fullsized":"2024-04-16T16:22:46.484240+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:53.044305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.478797+0000","last_change":"2024-04-16T16:22:46.469862+0000","last_active":"2024-04-16T16:22:46.478797+0000","last_peered":"2024-04-16T16:22:46.478797+0000","last_clean":"2024-04-16T16:22:46.478797+0000","last_became_active":"2024-04-16T16:22:46.466179+0000","last_became_peered":"2024-04-16T16:22:46.466179+0000","last_unstale":"2024-04-16T16:22:46.478797+0000","last_undegraded":"2024-04-16T16:22:46.478797+0000","last_fullsized":"2024-04-16T16:22:46.478797+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:51:03.090923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.445757+0000","last_change":"2024-04-16T16:22:45.461564+0000","last_active":"2024-04-16T16:22:46.445757+0000","last_peered":"2024-04-16T16:22:46.445757+0000","last_clean":"2024-04-16T16:22:46.445757+0000","last_became_active":"2024-04-16T16:22:45.461253+0000","last_became_peered":"2024-04-16T16:22:45.461253+0000","last_unstale":"2024-04-16T16:22:46.445757+0000","last_undegraded":"2024-04-16T16:22:46.445757+0000","last_fullsized":"2024-04-16T16:22:46.445757+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:46.018568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.472435+0000","last_change":"2024-04-16T16:22:46.465947+0000","last_active":"2024-04-16T16:22:46.472435+0000","last_peered":"2024-04-16T16:22:46.472435+0000","last_clean":"2024-04-16T16:22:46.472435+0000","last_became_active":"2024-04-16T16:22:46.465294+0000","last_became_peered":"2024-04-16T16:22:46.465294+0000","last_unstale":"2024-04-16T16:22:46.472435+0000","last_undegraded":"2024-04-16T16:22:46.472435+0000","last_fullsized":"2024-04-16T16:22:46.472435+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:11.331337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.454403+0000","last_change":"2024-04-16T16:22:45.468883+0000","last_active":"2024-04-16T16:22:46.454403+0000","last_peered":"2024-04-16T16:22:46.454403+0000","last_clean":"2024-04-16T16:22:46.454403+0000","last_became_active":"2024-04-16T16:22:45.468091+0000","last_became_peered":"2024-04-16T16:22:45.468091+0000","last_unstale":"2024-04-16T16:22:46.454403+0000","last_undegraded":"2024-04-16T16:22:46.454403+0000","last_fullsized":"2024-04-16T16:22:46.454403+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476000+0000","last_change":"2024-04-16T16:22:46.471354+0000","last_active":"2024-04-16T16:22:46.476000+0000","last_peered":"2024-04-16T16:22:46.476000+0000","last_clean":"2024-04-16T16:22:46.476000+0000","last_became_active":"2024-04-16T16:22:46.470521+0000","last_became_peered":"2024-04-16T16:22:46.470521+0000","last_unstale":"2024-04-16T16:22:46.476000+0000","last_undegraded":"2024-04-16T16:22:46.476000+0000","last_fullsized":"2024-04-16T16:22:46.476000+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:51:06.837645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.445445+0000","last_change":"2024-04-16T16:22:45.460865+0000","last_active":"2024-04-16T16:22:46.445445+0000","last_peered":"2024-04-16T16:22:46.445445+0000","last_clean":"2024-04-16T16:22:46.445445+0000","last_became_active":"2024-04-16T16:22:45.458861+0000","last_became_peered":"2024-04-16T16:22:45.458861+0000","last_unstale":"2024-04-16T16:22:46.445445+0000","last_undegraded":"2024-04-16T16:22:46.445445+0000","last_fullsized":"2024-04-16T16:22:46.445445+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:30.462179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475680+0000","last_change":"2024-04-16T16:22:46.464764+0000","last_active":"2024-04-16T16:22:46.475680+0000","last_peered":"2024-04-16T16:22:46.475680+0000","last_clean":"2024-04-16T16:22:46.475680+0000","last_became_active":"2024-04-16T16:22:46.464064+0000","last_became_peered":"2024-04-16T16:22:46.464064+0000","last_unstale":"2024-04-16T16:22:46.475680+0000","last_undegraded":"2024-04-16T16:22:46.475680+0000","last_fullsized":"2024-04-16T16:22:46.475680+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:26:38.389177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.480960+0000","last_change":"2024-04-16T16:22:45.466044+0000","last_active":"2024-04-16T16:22:46.480960+0000","last_peered":"2024-04-16T16:22:46.480960+0000","last_clean":"2024-04-16T16:22:46.480960+0000","last_became_active":"2024-04-16T16:22:45.465676+0000","last_became_peered":"2024-04-16T16:22:45.465676+0000","last_unstale":"2024-04-16T16:22:46.480960+0000","last_undegraded":"2024-04-16T16:22:46.480960+0000","last_fullsized":"2024-04-16T16:22:46.480960+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:52.385757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.478576+0000","last_change":"2024-04-16T16:22:46.466356+0000","last_active":"2024-04-16T16:22:46.478576+0000","last_peered":"2024-04-16T16:22:46.478576+0000","last_clean":"2024-04-16T16:22:46.478576+0000","last_became_active":"2024-04-16T16:22:46.465449+0000","last_became_peered":"2024-04-16T16:22:46.465449+0000","last_unstale":"2024-04-16T16:22:46.478576+0000","last_undegraded":"2024-04-16T16:22:46.478576+0000","last_fullsized":"2024-04-16T16:22:46.478576+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:47.034586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.471171+0000","last_change":"2024-04-16T16:22:45.466156+0000","last_active":"2024-04-16T16:22:46.471171+0000","last_peered":"2024-04-16T16:22:46.471171+0000","last_clean":"2024-04-16T16:22:46.471171+0000","last_became_active":"2024-04-16T16:22:45.465791+0000","last_became_peered":"2024-04-16T16:22:45.465791+0000","last_unstale":"2024-04-16T16:22:46.471171+0000","last_undegraded":"2024-04-16T16:22:46.471171+0000","last_fullsized":"2024-04-16T16:22:46.471171+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:42:46.394979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.474067+0000","last_change":"2024-04-16T16:22:46.470854+0000","last_active":"2024-04-16T16:22:46.474067+0000","last_peered":"2024-04-16T16:22:46.474067+0000","last_clean":"2024-04-16T16:22:46.474067+0000","last_became_active":"2024-04-16T16:22:46.470196+0000","last_became_peered":"2024-04-16T16:22:46.470196+0000","last_unstale":"2024-04-16T16:22:46.474067+0000","last_undegraded":"2024-04-16T16:22:46.474067+0000","last_fullsized":"2024-04-16T16:22:46.474067+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:18:09.072901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.460207+0000","last_change":"2024-04-16T16:22:45.471205+0000","last_active":"2024-04-16T16:22:46.460207+0000","last_peered":"2024-04-16T16:22:46.460207+0000","last_clean":"2024-04-16T16:22:46.460207+0000","last_became_active":"2024-04-16T16:22:45.470911+0000","last_became_peered":"2024-04-16T16:22:45.470911+0000","last_unstale":"2024-04-16T16:22:46.460207+0000","last_undegraded":"2024-04-16T16:22:46.460207+0000","last_fullsized":"2024-04-16T16:22:46.460207+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:53.044305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476431+0000","last_change":"2024-04-16T16:22:46.469324+0000","last_active":"2024-04-16T16:22:46.476431+0000","last_peered":"2024-04-16T16:22:46.476431+0000","last_clean":"2024-04-16T16:22:46.476431+0000","last_became_active":"2024-04-16T16:22:46.469009+0000","last_became_peered":"2024-04-16T16:22:46.469009+0000","last_unstale":"2024-04-16T16:22:46.476431+0000","last_undegraded":"2024-04-16T16:22:46.476431+0000","last_fullsized":"2024-04-16T16:22:46.476431+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:56.713949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.455944+0000","last_change":"2024-04-16T16:22:45.468189+0000","last_active":"2024-04-16T16:22:46.455944+0000","last_peered":"2024-04-16T16:22:46.455944+0000","last_clean":"2024-04-16T16:22:46.455944+0000","last_became_active":"2024-04-16T16:22:45.467709+0000","last_became_peered":"2024-04-16T16:22:45.467709+0000","last_unstale":"2024-04-16T16:22:46.455944+0000","last_undegraded":"2024-04-16T16:22:46.455944+0000","last_fullsized":"2024-04-16T16:22:46.455944+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:46.256132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476031+0000","last_change":"2024-04-16T16:22:46.465994+0000","last_active":"2024-04-16T16:22:46.476031+0000","last_peered":"2024-04-16T16:22:46.476031+0000","last_clean":"2024-04-16T16:22:46.476031+0000","last_became_active":"2024-04-16T16:22:46.465007+0000","last_became_peered":"2024-04-16T16:22:46.465007+0000","last_unstale":"2024-04-16T16:22:46.476031+0000","last_undegraded":"2024-04-16T16:22:46.476031+0000","last_fullsized":"2024-04-16T16:22:46.476031+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:31.478197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.448890+0000","last_change":"2024-04-16T16:22:45.470607+0000","last_active":"2024-04-16T16:22:46.448890+0000","last_peered":"2024-04-16T16:22:46.448890+0000","last_clean":"2024-04-16T16:22:46.448890+0000","last_became_active":"2024-04-16T16:22:45.469322+0000","last_became_peered":"2024-04-16T16:22:45.469322+0000","last_unstale":"2024-04-16T16:22:46.448890+0000","last_undegraded":"2024-04-16T16:22:46.448890+0000","last_fullsized":"2024-04-16T16:22:46.448890+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:07.564200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475798+0000","last_change":"2024-04-16T16:22:46.462483+0000","last_active":"2024-04-16T16:22:46.475798+0000","last_peered":"2024-04-16T16:22:46.475798+0000","last_clean":"2024-04-16T16:22:46.475798+0000","last_became_active":"2024-04-16T16:22:46.461432+0000","last_became_peered":"2024-04-16T16:22:46.461432+0000","last_unstale":"2024-04-16T16:22:46.475798+0000","last_undegraded":"2024-04-16T16:22:46.475798+0000","last_fullsized":"2024-04-16T16:22:46.475798+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:27.266104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.451322+0000","last_change":"2024-04-16T16:22:45.462752+0000","last_active":"2024-04-16T16:22:46.451322+0000","last_peered":"2024-04-16T16:22:46.451322+0000","last_clean":"2024-04-16T16:22:46.451322+0000","last_became_active":"2024-04-16T16:22:45.462154+0000","last_became_peered":"2024-04-16T16:22:45.462154+0000","last_unstale":"2024-04-16T16:22:46.451322+0000","last_undegraded":"2024-04-16T16:22:46.451322+0000","last_fullsized":"2024-04-16T16:22:46.451322+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:43:04.134472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479013+0000","last_change":"2024-04-16T16:22:46.467575+0000","last_active":"2024-04-16T16:22:46.479013+0000","last_peered":"2024-04-16T16:22:46.479013+0000","last_clean":"2024-04-16T16:22:46.479013+0000","last_became_active":"2024-04-16T16:22:46.466890+0000","last_became_peered":"2024-04-16T16:22:46.466890+0000","last_unstale":"2024-04-16T16:22:46.479013+0000","last_undegraded":"2024-04-16T16:22:46.479013+0000","last_fullsized":"2024-04-16T16:22:46.479013+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:56.713949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.466502+0000","last_change":"2024-04-16T16:22:45.469689+0000","last_active":"2024-04-16T16:22:46.466502+0000","last_peered":"2024-04-16T16:22:46.466502+0000","last_clean":"2024-04-16T16:22:46.466502+0000","last_became_active":"2024-04-16T16:22:45.469350+0000","last_became_peered":"2024-04-16T16:22:45.469350+0000","last_unstale":"2024-04-16T16:22:46.466502+0000","last_undegraded":"2024-04-16T16:22:46.466502+0000","last_fullsized":"2024-04-16T16:22:46.466502+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.482785+0000","last_change":"2024-04-16T16:22:46.478730+0000","last_active":"2024-04-16T16:22:46.482785+0000","last_peered":"2024-04-16T16:22:46.482785+0000","last_clean":"2024-04-16T16:22:46.482785+0000","last_became_active":"2024-04-16T16:22:46.478459+0000","last_became_peered":"2024-04-16T16:22:46.478459+0000","last_unstale":"2024-04-16T16:22:46.482785+0000","last_undegraded":"2024-04-16T16:22:46.482785+0000","last_fullsized":"2024-04-16T16:22:46.482785+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:07:30.192612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.448825+0000","last_change":"2024-04-16T16:22:45.467505+0000","last_active":"2024-04-16T16:22:46.448825+0000","last_peered":"2024-04-16T16:22:46.448825+0000","last_clean":"2024-04-16T16:22:46.448825+0000","last_became_active":"2024-04-16T16:22:45.465339+0000","last_became_peered":"2024-04-16T16:22:45.465339+0000","last_unstale":"2024-04-16T16:22:46.448825+0000","last_undegraded":"2024-04-16T16:22:46.448825+0000","last_fullsized":"2024-04-16T16:22:46.448825+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:54.078259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475699+0000","last_change":"2024-04-16T16:22:46.465023+0000","last_active":"2024-04-16T16:22:46.475699+0000","last_peered":"2024-04-16T16:22:46.475699+0000","last_clean":"2024-04-16T16:22:46.475699+0000","last_became_active":"2024-04-16T16:22:46.464436+0000","last_became_peered":"2024-04-16T16:22:46.464436+0000","last_unstale":"2024-04-16T16:22:46.475699+0000","last_undegraded":"2024-04-16T16:22:46.475699+0000","last_fullsized":"2024-04-16T16:22:46.475699+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:05:07.007726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.454218+0000","last_change":"2024-04-16T16:22:45.461515+0000","last_active":"2024-04-16T16:22:46.454218+0000","last_peered":"2024-04-16T16:22:46.454218+0000","last_clean":"2024-04-16T16:22:46.454218+0000","last_became_active":"2024-04-16T16:22:45.460350+0000","last_became_peered":"2024-04-16T16:22:45.460350+0000","last_unstale":"2024-04-16T16:22:46.454218+0000","last_undegraded":"2024-04-16T16:22:46.454218+0000","last_fullsized":"2024-04-16T16:22:46.454218+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:34.292891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"144'4691","reported_seq":5388,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.445611+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:22:46.445611+0000","last_peered":"2024-04-16T16:22:46.445611+0000","last_clean":"2024-04-16T16:22:46.445611+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:22:46.445611+0000","last_undegraded":"2024-04-16T16:22:46.445611+0000","last_fullsized":"2024-04-16T16:22:46.445611+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":4691,"log_dups_size":0,"ondisk_log_size":4691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":387,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":387,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4691,"ondisk_log_size":4691,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37812,"kb_used_data":2468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334028,"statfs":{"total":96636764160,"available":96598044672,"internally_reserved":0,"allocated":2527232,"data_stored":1510791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574957,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1933312,"data_stored":920423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574957,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1933312,"data_stored":920423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574958,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1933312,"data_stored":920423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574956,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1933312,"data_stored":920423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574956,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1933312,"data_stored":920423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33072,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338768,"statfs":{"total":96636764160,"available":96602898432,"internally_reserved":0,"allocated":1933312,"data_stored":920423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574957,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37812,"kb_used_data":2468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334028,"statfs":{"total":96636764160,"available":96598044672,"internally_reserved":0,"allocated":2527232,"data_stored":1510791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:22:51.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:22:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.319+0000 7f5a57200640 1 Processor -- start 2024-04-16T16:22:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.320+0000 7f5a57200640 1 -- start start 2024-04-16T16:22:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.321+0000 7f5a57200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a580d52d0 0x7f5a5806ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.321+0000 7f5a57200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a580d3080 con 0x7f5a58161e90 2024-04-16T16:22:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.321+0000 7f5a57200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a580d3360 con 0x7f5a58162270 2024-04-16T16:22:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.321+0000 7f5a57200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5a580d2e30 con 0x7f5a580d52d0 2024-04-16T16:22:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.322+0000 7f5a55e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5a58161e90 0x7f5a580cf7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38444/0 (socket says 172.21.15.45:38444) 2024-04-16T16:22:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.322+0000 7f5a55e00640 1 -- 172.21.15.45:0/1472252190 learned_addr learned my addr 172.21.15.45:0/1472252190 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.322+0000 7f5a56800640 1 --2- 172.21.15.45:0/1472252190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a580d52d0 0x7f5a5806ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.322+0000 7f5a55400640 1 -- 172.21.15.45:0/1472252190 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1394179507 0 0) 0x7f5a580d3080 con 0x7f5a58161e90 2024-04-16T16:22:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.322+0000 7f5a55400640 1 -- 172.21.15.45:0/1472252190 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a3c003520 con 0x7f5a58161e90 2024-04-16T16:22:51.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.323+0000 7f5a55400640 1 -- 172.21.15.45:0/1472252190 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2334479241 0 0) 0x7f5a3c003520 con 0x7f5a58161e90 2024-04-16T16:22:51.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.323+0000 7f5a55400640 1 -- 172.21.15.45:0/1472252190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a580d52d0 msgr2=0x7f5a5806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:51.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.323+0000 7f5a55400640 1 --2- 172.21.15.45:0/1472252190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a580d52d0 0x7f5a5806ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.323+0000 7f5a55400640 1 -- 172.21.15.45:0/1472252190 >> v1:172.21.15.103:6789/0 conn(0x7f5a58162270 legacy=0x7f5a580d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:51.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.323+0000 7f5a55400640 1 -- 172.21.15.45:0/1472252190 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a580d3640 con 0x7f5a58161e90 2024-04-16T16:22:51.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.323+0000 7f5a55400640 1 -- 172.21.15.45:0/1472252190 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f5a48001670 con 0x7f5a58161e90 2024-04-16T16:22:51.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.323+0000 7f5a55400640 1 -- 172.21.15.45:0/1472252190 >> v1:172.21.15.45:6789/0 conn(0x7f5a58161e90 legacy=0x7f5a580cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:51.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.323+0000 7f5a55400640 1 --2- 172.21.15.45:0/1472252190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3c0039e0 0x7f5a3c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:51.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.323+0000 7f5a55400640 1 --2- 172.21.15.45:0/1472252190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a580d52d0 0x7f5a3c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:51.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.323+0000 7f5a55400640 1 --2- 172.21.15.45:0/1472252190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a3c004d40 0x7f5a3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:51.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.323+0000 7f5a55400640 1 -- 172.21.15.45:0/1472252190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5a580d2e30 con 0x7f5a3c0039e0 2024-04-16T16:22:51.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.323+0000 7f5a55400640 1 -- 172.21.15.45:0/1472252190 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c00bb30 con 0x7f5a3c004d40 2024-04-16T16:22:51.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.323+0000 7f5a55400640 1 -- 172.21.15.45:0/1472252190 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5a3c00bd30 con 0x7f5a580d52d0 2024-04-16T16:22:51.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.323+0000 7f5a55e00640 1 --2- 172.21.15.45:0/1472252190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3c0039e0 0x7f5a3c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:51.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.324+0000 7f5a56800640 1 --2- 172.21.15.45:0/1472252190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a3c004d40 0x7f5a3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:51.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.324+0000 7f5a55e00640 1 -- 172.21.15.45:0/1472252190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a580d52d0 msgr2=0x7f5a3c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:51.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.324+0000 7f5a4fe00640 1 --2- 172.21.15.45:0/1472252190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a580d52d0 0x7f5a3c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:51.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.324+0000 7f5a55e00640 1 --2- 172.21.15.45:0/1472252190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a580d52d0 0x7f5a3c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.324+0000 7f5a55e00640 1 -- 172.21.15.45:0/1472252190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a3c004d40 msgr2=0x7f5a3c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:51.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.324+0000 7f5a55e00640 1 --2- 172.21.15.45:0/1472252190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a3c004d40 0x7f5a3c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.324+0000 7f5a55e00640 1 -- 172.21.15.45:0/1472252190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a3c003520 con 0x7f5a3c0039e0 2024-04-16T16:22:51.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.324+0000 7f5a4fe00640 1 --2- 172.21.15.45:0/1472252190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a580d52d0 0x7f5a3c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:51.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.324+0000 7f5a56800640 1 --2- 172.21.15.45:0/1472252190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a3c004d40 0x7f5a3c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.325+0000 7f5a55e00640 1 --2- 172.21.15.45:0/1472252190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3c0039e0 0x7f5a3c003e00 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f5a48001eb0 tx=0x7f5a3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e9b8619ee947290 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.325+0000 7f5a55400640 1 -- 172.21.15.45:0/1472252190 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a48007d30 con 0x7f5a3c0039e0 2024-04-16T16:22:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.325+0000 7f5a55400640 1 -- 172.21.15.45:0/1472252190 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a48030070 con 0x7f5a3c0039e0 2024-04-16T16:22:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.325+0000 7f5a55400640 1 -- 172.21.15.45:0/1472252190 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a48005d30 con 0x7f5a3c0039e0 2024-04-16T16:22:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.326+0000 7f5a57200640 1 -- 172.21.15.45:0/1472252190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3c0039e0 msgr2=0x7f5a3c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.326+0000 7f5a57200640 1 --2- 172.21.15.45:0/1472252190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3c0039e0 0x7f5a3c003e00 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f5a48001eb0 tx=0x7f5a3c005830 comp rx=0 tx=0).stop 2024-04-16T16:22:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.326+0000 7f5a56800640 1 -- 172.21.15.45:0/1472252190 reap_dead start 2024-04-16T16:22:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.326+0000 7f5a57200640 1 -- 172.21.15.45:0/1472252190 shutdown_connections 2024-04-16T16:22:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.326+0000 7f5a57200640 1 -- 172.21.15.45:0/1472252190 >> 172.21.15.45:0/1472252190 conn(0x7f5a580d1940 msgr2=0x7f5a580d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.326+0000 7f5a57200640 1 -- 172.21.15.45:0/1472252190 shutdown_connections 2024-04-16T16:22:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.326+0000 7f5a57200640 1 -- 172.21.15.45:0/1472252190 wait complete. 2024-04-16T16:22:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.328+0000 7f5a57200640 1 Processor -- start 2024-04-16T16:22:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.328+0000 7f5a57200640 1 -- start start 2024-04-16T16:22:51.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.329+0000 7f5a57200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3c0039e0 0x7f5a580ce460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.329+0000 7f5a57200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a5815caa0 0x7f5a580b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.329+0000 7f5a57200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a5814f9e0 0x7f5a5814fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.329+0000 7f5a57200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5a580d46d0 con 0x7f5a3c0039e0 2024-04-16T16:22:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.329+0000 7f5a57200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5a580d4550 con 0x7f5a5815caa0 2024-04-16T16:22:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.329+0000 7f5a57200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5a5806ce40 con 0x7f5a5814f9e0 2024-04-16T16:22:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.329+0000 7f5a55e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3c0039e0 0x7f5a580ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.329+0000 7f5a55e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3c0039e0 0x7f5a580ce460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33110/0 (socket says 172.21.15.45:33110) 2024-04-16T16:22:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.329+0000 7f5a55e00640 1 -- 172.21.15.45:0/189098866 learned_addr learned my addr 172.21.15.45:0/189098866 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.330+0000 7f5a4fe00640 1 --2- 172.21.15.45:0/189098866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a5815caa0 0x7f5a580b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.330+0000 7f5a56800640 1 --2- 172.21.15.45:0/189098866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a5814f9e0 0x7f5a5814fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.330+0000 7f5a55e00640 1 -- 172.21.15.45:0/189098866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a5814f9e0 msgr2=0x7f5a5814fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.330+0000 7f5a55e00640 1 --2- 172.21.15.45:0/189098866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a5814f9e0 0x7f5a5814fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.330+0000 7f5a55e00640 1 -- 172.21.15.45:0/189098866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a5815caa0 msgr2=0x7f5a580b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.330+0000 7f5a55e00640 1 --2- 172.21.15.45:0/189098866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a5815caa0 0x7f5a580b54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.330+0000 7f5a55e00640 1 -- 172.21.15.45:0/189098866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a5815d260 con 0x7f5a3c0039e0 2024-04-16T16:22:51.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.330+0000 7f5a55e00640 1 --2- 172.21.15.45:0/189098866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3c0039e0 0x7f5a580ce460 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f5a4802c540 tx=0x7f5a48001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:51.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.330+0000 7f5a4e000640 1 -- 172.21.15.45:0/189098866 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a48004bf0 con 0x7f5a3c0039e0 2024-04-16T16:22:51.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.331+0000 7f5a4e000640 1 -- 172.21.15.45:0/189098866 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a48004d90 con 0x7f5a3c0039e0 2024-04-16T16:22:51.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.331+0000 7f5a4e000640 1 -- 172.21.15.45:0/189098866 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a48007270 con 0x7f5a3c0039e0 2024-04-16T16:22:51.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.331+0000 7f5a57200640 1 -- 172.21.15.45:0/189098866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a5815d540 con 0x7f5a3c0039e0 2024-04-16T16:22:51.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.331+0000 7f5a57200640 1 -- 172.21.15.45:0/189098866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a5815db00 con 0x7f5a3c0039e0 2024-04-16T16:22:51.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.332+0000 7f5a57200640 1 -- 172.21.15.45:0/189098866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a58068300 con 0x7f5a3c0039e0 2024-04-16T16:22:51.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.333+0000 7f5a4e000640 1 -- 172.21.15.45:0/189098866 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f5a48048020 con 0x7f5a3c0039e0 2024-04-16T16:22:51.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.334+0000 7f5a4e000640 1 --2- 172.21.15.45:0/189098866 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5a3406ca40 0x7f5a3406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:51.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.334+0000 7f5a4e000640 1 -- 172.21.15.45:0/189098866 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f5a480ace90 con 0x7f5a3c0039e0 2024-04-16T16:22:51.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.334+0000 7f5a4fe00640 1 --2- 172.21.15.45:0/189098866 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5a3406ca40 0x7f5a3406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:51.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.336+0000 7f5a4fe00640 1 --2- 172.21.15.45:0/189098866 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5a3406ca40 0x7f5a3406ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5a580ca500 tx=0x7f5a40009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:51.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.338+0000 7f5a4e000640 1 -- 172.21.15.45:0/189098866 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5a58068300 con 0x7f5a3c0039e0 2024-04-16T16:22:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.543+0000 7f5a57200640 1 -- 172.21.15.45:0/189098866 --> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5a58161a40 con 0x7f5a3406ca40 2024-04-16T16:22:51.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.564+0000 7f5a4e000640 1 -- 172.21.15.45:0/189098866 <== mgr.7591 v2:172.21.15.103:6832/1129061436 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252435 (secure 0 0 0) 0x7f5a58161a40 con 0x7f5a3406ca40 2024-04-16T16:22:51.564 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:51.567 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:22:51.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.570+0000 7f5a57200640 1 -- 172.21.15.45:0/189098866 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5a3406ca40 msgr2=0x7f5a3406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:51.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.570+0000 7f5a57200640 1 --2- 172.21.15.45:0/189098866 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5a3406ca40 0x7f5a3406ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5a580ca500 tx=0x7f5a40009040 comp rx=0 tx=0).stop 2024-04-16T16:22:51.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.570+0000 7f5a57200640 1 -- 172.21.15.45:0/189098866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3c0039e0 msgr2=0x7f5a580ce460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:51.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.570+0000 7f5a57200640 1 --2- 172.21.15.45:0/189098866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3c0039e0 0x7f5a580ce460 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f5a4802c540 tx=0x7f5a48001a40 comp rx=0 tx=0).stop 2024-04-16T16:22:51.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.570+0000 7f5a57200640 1 -- 172.21.15.45:0/189098866 shutdown_connections 2024-04-16T16:22:51.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.570+0000 7f5a57200640 1 --2- 172.21.15.45:0/189098866 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5a3406ca40 0x7f5a3406ef00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.570+0000 7f5a57200640 1 --2- 172.21.15.45:0/189098866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a5814f9e0 0x7f5a5814fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.570+0000 7f5a57200640 1 --2- 172.21.15.45:0/189098866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a5815caa0 0x7f5a580b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.570+0000 7f5a57200640 1 --2- 172.21.15.45:0/189098866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3c0039e0 0x7f5a580ce460 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.570+0000 7f5a57200640 1 -- 172.21.15.45:0/189098866 >> 172.21.15.45:0/189098866 conn(0x7f5a580d1940 msgr2=0x7f5a580b9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:51.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.570+0000 7f5a57200640 1 -- 172.21.15.45:0/189098866 shutdown_connections 2024-04-16T16:22:51.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.570+0000 7f5a57200640 1 -- 172.21.15.45:0/189098866 wait complete. 2024-04-16T16:22:51.589 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-04-16T16:22:51.541844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":431,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4729,"ondisk_log_size":4729,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":282384,"kb_used_data":16292,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754692336,"statfs":{"total":773094113280,"available":772804952064,"internally_reserved":0,"allocated":16683008,"data_stored":8557285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.107455"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.489163+0000","last_change":"2024-04-16T16:22:46.488567+0000","last_active":"2024-04-16T16:22:46.489163+0000","last_peered":"2024-04-16T16:22:46.489163+0000","last_clean":"2024-04-16T16:22:46.489163+0000","last_became_active":"2024-04-16T16:22:46.485690+0000","last_became_peered":"2024-04-16T16:22:46.485690+0000","last_unstale":"2024-04-16T16:22:46.489163+0000","last_undegraded":"2024-04-16T16:22:46.489163+0000","last_fullsized":"2024-04-16T16:22:46.489163+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:59:47.867717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.483165+0000","last_change":"2024-04-16T16:22:46.479620+0000","last_active":"2024-04-16T16:22:46.483165+0000","last_peered":"2024-04-16T16:22:46.483165+0000","last_clean":"2024-04-16T16:22:46.483165+0000","last_became_active":"2024-04-16T16:22:46.477410+0000","last_became_peered":"2024-04-16T16:22:46.477410+0000","last_unstale":"2024-04-16T16:22:46.483165+0000","last_undegraded":"2024-04-16T16:22:46.483165+0000","last_fullsized":"2024-04-16T16:22:46.483165+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:35.308909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.486427+0000","last_change":"2024-04-16T16:22:46.480944+0000","last_active":"2024-04-16T16:22:46.486427+0000","last_peered":"2024-04-16T16:22:46.486427+0000","last_clean":"2024-04-16T16:22:46.486427+0000","last_became_active":"2024-04-16T16:22:46.480405+0000","last_became_peered":"2024-04-16T16:22:46.480405+0000","last_unstale":"2024-04-16T16:22:46.486427+0000","last_undegraded":"2024-04-16T16:22:46.486427+0000","last_fullsized":"2024-04-16T16:22:46.486427+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:32:48.252527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.480329+0000","last_change":"2024-04-16T16:22:46.477832+0000","last_active":"2024-04-16T16:22:46.480329+0000","last_peered":"2024-04-16T16:22:46.480329+0000","last_clean":"2024-04-16T16:22:46.480329+0000","last_became_active":"2024-04-16T16:22:46.477516+0000","last_became_peered":"2024-04-16T16:22:46.477516+0000","last_unstale":"2024-04-16T16:22:46.480329+0000","last_undegraded":"2024-04-16T16:22:46.480329+0000","last_fullsized":"2024-04-16T16:22:46.480329+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:13.979255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.485892+0000","last_change":"2024-04-16T16:22:46.483466+0000","last_active":"2024-04-16T16:22:46.485892+0000","last_peered":"2024-04-16T16:22:46.485892+0000","last_clean":"2024-04-16T16:22:46.485892+0000","last_became_active":"2024-04-16T16:22:46.481033+0000","last_became_peered":"2024-04-16T16:22:46.481033+0000","last_unstale":"2024-04-16T16:22:46.485892+0000","last_undegraded":"2024-04-16T16:22:46.485892+0000","last_fullsized":"2024-04-16T16:22:46.485892+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:13.979255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710150+0000","last_change":"2024-04-16T16:22:46.463298+0000","last_active":"2024-04-16T16:22:48.710150+0000","last_peered":"2024-04-16T16:22:48.710150+0000","last_clean":"2024-04-16T16:22:48.710150+0000","last_became_active":"2024-04-16T16:22:46.462017+0000","last_became_peered":"2024-04-16T16:22:46.462017+0000","last_unstale":"2024-04-16T16:22:48.710150+0000","last_undegraded":"2024-04-16T16:22:48.710150+0000","last_fullsized":"2024-04-16T16:22:48.710150+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:30:36.112336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.471734+0000","last_change":"2024-04-16T16:22:46.462786+0000","last_active":"2024-04-16T16:22:46.471734+0000","last_peered":"2024-04-16T16:22:46.471734+0000","last_clean":"2024-04-16T16:22:46.471734+0000","last_became_active":"2024-04-16T16:22:46.461856+0000","last_became_peered":"2024-04-16T16:22:46.461856+0000","last_unstale":"2024-04-16T16:22:46.471734+0000","last_undegraded":"2024-04-16T16:22:46.471734+0000","last_fullsized":"2024-04-16T16:22:46.471734+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:31.478197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.488167+0000","last_change":"2024-04-16T16:22:46.486917+0000","last_active":"2024-04-16T16:22:46.488167+0000","last_peered":"2024-04-16T16:22:46.488167+0000","last_clean":"2024-04-16T16:22:46.488167+0000","last_became_active":"2024-04-16T16:22:46.485280+0000","last_became_peered":"2024-04-16T16:22:46.485280+0000","last_unstale":"2024-04-16T16:22:46.488167+0000","last_undegraded":"2024-04-16T16:22:46.488167+0000","last_fullsized":"2024-04-16T16:22:46.488167+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:53:06.786966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479938+0000","last_change":"2024-04-16T16:22:46.476513+0000","last_active":"2024-04-16T16:22:46.479938+0000","last_peered":"2024-04-16T16:22:46.479938+0000","last_clean":"2024-04-16T16:22:46.479938+0000","last_became_active":"2024-04-16T16:22:46.476125+0000","last_became_peered":"2024-04-16T16:22:46.476125+0000","last_unstale":"2024-04-16T16:22:46.479938+0000","last_undegraded":"2024-04-16T16:22:46.479938+0000","last_fullsized":"2024-04-16T16:22:46.479938+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:32:25.664832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.485587+0000","last_change":"2024-04-16T16:22:46.483287+0000","last_active":"2024-04-16T16:22:46.485587+0000","last_peered":"2024-04-16T16:22:46.485587+0000","last_clean":"2024-04-16T16:22:46.485587+0000","last_became_active":"2024-04-16T16:22:46.482975+0000","last_became_peered":"2024-04-16T16:22:46.482975+0000","last_unstale":"2024-04-16T16:22:46.485587+0000","last_undegraded":"2024-04-16T16:22:46.485587+0000","last_fullsized":"2024-04-16T16:22:46.485587+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:27.266104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.486131+0000","last_change":"2024-04-16T16:22:46.481301+0000","last_active":"2024-04-16T16:22:46.486131+0000","last_peered":"2024-04-16T16:22:46.486131+0000","last_clean":"2024-04-16T16:22:46.486131+0000","last_became_active":"2024-04-16T16:22:46.481000+0000","last_became_peered":"2024-04-16T16:22:46.481000+0000","last_unstale":"2024-04-16T16:22:46.486131+0000","last_undegraded":"2024-04-16T16:22:46.486131+0000","last_fullsized":"2024-04-16T16:22:46.486131+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:44:40.829325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479690+0000","last_change":"2024-04-16T16:22:46.476179+0000","last_active":"2024-04-16T16:22:46.479690+0000","last_peered":"2024-04-16T16:22:46.479690+0000","last_clean":"2024-04-16T16:22:46.479690+0000","last_became_active":"2024-04-16T16:22:46.475898+0000","last_became_peered":"2024-04-16T16:22:46.475898+0000","last_unstale":"2024-04-16T16:22:46.479690+0000","last_undegraded":"2024-04-16T16:22:46.479690+0000","last_fullsized":"2024-04-16T16:22:46.479690+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:08.580218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.487790+0000","last_change":"2024-04-16T16:22:46.482186+0000","last_active":"2024-04-16T16:22:46.487790+0000","last_peered":"2024-04-16T16:22:46.487790+0000","last_clean":"2024-04-16T16:22:46.487790+0000","last_became_active":"2024-04-16T16:22:46.481776+0000","last_became_peered":"2024-04-16T16:22:46.481776+0000","last_unstale":"2024-04-16T16:22:46.487790+0000","last_undegraded":"2024-04-16T16:22:46.487790+0000","last_fullsized":"2024-04-16T16:22:46.487790+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:43:05.150490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.482401+0000","last_change":"2024-04-16T16:22:46.477861+0000","last_active":"2024-04-16T16:22:46.482401+0000","last_peered":"2024-04-16T16:22:46.482401+0000","last_clean":"2024-04-16T16:22:46.482401+0000","last_became_active":"2024-04-16T16:22:46.476486+0000","last_became_peered":"2024-04-16T16:22:46.476486+0000","last_unstale":"2024-04-16T16:22:46.482401+0000","last_undegraded":"2024-04-16T16:22:46.482401+0000","last_fullsized":"2024-04-16T16:22:46.482401+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:34:35.154021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.472708+0000","last_change":"2024-04-16T16:22:46.468319+0000","last_active":"2024-04-16T16:22:46.472708+0000","last_peered":"2024-04-16T16:22:46.472708+0000","last_clean":"2024-04-16T16:22:46.472708+0000","last_became_active":"2024-04-16T16:22:46.463117+0000","last_became_peered":"2024-04-16T16:22:46.463117+0000","last_unstale":"2024-04-16T16:22:46.472708+0000","last_undegraded":"2024-04-16T16:22:46.472708+0000","last_fullsized":"2024-04-16T16:22:46.472708+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:51:03.090923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479041+0000","last_change":"2024-04-16T16:22:46.467824+0000","last_active":"2024-04-16T16:22:46.479041+0000","last_peered":"2024-04-16T16:22:46.479041+0000","last_clean":"2024-04-16T16:22:46.479041+0000","last_became_active":"2024-04-16T16:22:46.467274+0000","last_became_peered":"2024-04-16T16:22:46.467274+0000","last_unstale":"2024-04-16T16:22:46.479041+0000","last_undegraded":"2024-04-16T16:22:46.479041+0000","last_fullsized":"2024-04-16T16:22:46.479041+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:11.331337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710486+0000","last_change":"2024-04-16T16:22:46.475037+0000","last_active":"2024-04-16T16:22:48.710486+0000","last_peered":"2024-04-16T16:22:48.710486+0000","last_clean":"2024-04-16T16:22:48.710486+0000","last_became_active":"2024-04-16T16:22:46.474757+0000","last_became_peered":"2024-04-16T16:22:46.474757+0000","last_unstale":"2024-04-16T16:22:48.710486+0000","last_undegraded":"2024-04-16T16:22:48.710486+0000","last_fullsized":"2024-04-16T16:22:48.710486+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:31:51.916998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475658+0000","last_change":"2024-04-16T16:22:46.463139+0000","last_active":"2024-04-16T16:22:46.475658+0000","last_peered":"2024-04-16T16:22:46.475658+0000","last_clean":"2024-04-16T16:22:46.475658+0000","last_became_active":"2024-04-16T16:22:46.462857+0000","last_became_peered":"2024-04-16T16:22:46.462857+0000","last_unstale":"2024-04-16T16:22:46.475658+0000","last_undegraded":"2024-04-16T16:22:46.475658+0000","last_fullsized":"2024-04-16T16:22:46.475658+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:18.473865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710482+0000","last_change":"2024-04-16T16:22:46.457033+0000","last_active":"2024-04-16T16:22:48.710482+0000","last_peered":"2024-04-16T16:22:48.710482+0000","last_clean":"2024-04-16T16:22:48.710482+0000","last_became_active":"2024-04-16T16:22:46.456756+0000","last_became_peered":"2024-04-16T16:22:46.456756+0000","last_unstale":"2024-04-16T16:22:48.710482+0000","last_undegraded":"2024-04-16T16:22:48.710482+0000","last_fullsized":"2024-04-16T16:22:48.710482+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:18:13.904540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.487464+0000","last_change":"2024-04-16T16:22:46.482119+0000","last_active":"2024-04-16T16:22:46.487464+0000","last_peered":"2024-04-16T16:22:46.487464+0000","last_clean":"2024-04-16T16:22:46.487464+0000","last_became_active":"2024-04-16T16:22:46.479857+0000","last_became_peered":"2024-04-16T16:22:46.479857+0000","last_unstale":"2024-04-16T16:22:46.487464+0000","last_undegraded":"2024-04-16T16:22:46.487464+0000","last_fullsized":"2024-04-16T16:22:46.487464+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:02.607399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710264+0000","last_change":"2024-04-16T16:22:46.479160+0000","last_active":"2024-04-16T16:22:48.710264+0000","last_peered":"2024-04-16T16:22:48.710264+0000","last_clean":"2024-04-16T16:22:48.710264+0000","last_became_active":"2024-04-16T16:22:46.477081+0000","last_became_peered":"2024-04-16T16:22:46.477081+0000","last_unstale":"2024-04-16T16:22:48.710264+0000","last_undegraded":"2024-04-16T16:22:48.710264+0000","last_fullsized":"2024-04-16T16:22:48.710264+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:31:11.372535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479731+0000","last_change":"2024-04-16T16:22:46.473896+0000","last_active":"2024-04-16T16:22:46.479731+0000","last_peered":"2024-04-16T16:22:46.479731+0000","last_clean":"2024-04-16T16:22:46.479731+0000","last_became_active":"2024-04-16T16:22:46.473621+0000","last_became_peered":"2024-04-16T16:22:46.473621+0000","last_unstale":"2024-04-16T16:22:46.479731+0000","last_undegraded":"2024-04-16T16:22:46.479731+0000","last_fullsized":"2024-04-16T16:22:46.479731+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:55.094277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710026+0000","last_change":"2024-04-16T16:22:46.475807+0000","last_active":"2024-04-16T16:22:48.710026+0000","last_peered":"2024-04-16T16:22:48.710026+0000","last_clean":"2024-04-16T16:22:48.710026+0000","last_became_active":"2024-04-16T16:22:46.473183+0000","last_became_peered":"2024-04-16T16:22:46.473183+0000","last_unstale":"2024-04-16T16:22:48.710026+0000","last_undegraded":"2024-04-16T16:22:48.710026+0000","last_fullsized":"2024-04-16T16:22:48.710026+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:26.889339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479850+0000","last_change":"2024-04-16T16:22:46.477670+0000","last_active":"2024-04-16T16:22:46.479850+0000","last_peered":"2024-04-16T16:22:46.479850+0000","last_clean":"2024-04-16T16:22:46.479850+0000","last_became_active":"2024-04-16T16:22:46.477303+0000","last_became_peered":"2024-04-16T16:22:46.477303+0000","last_unstale":"2024-04-16T16:22:46.479850+0000","last_undegraded":"2024-04-16T16:22:46.479850+0000","last_fullsized":"2024-04-16T16:22:46.479850+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:05:36.568682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.485748+0000","last_change":"2024-04-16T16:22:46.481055+0000","last_active":"2024-04-16T16:22:46.485748+0000","last_peered":"2024-04-16T16:22:46.485748+0000","last_clean":"2024-04-16T16:22:46.485748+0000","last_became_active":"2024-04-16T16:22:46.480641+0000","last_became_peered":"2024-04-16T16:22:46.480641+0000","last_unstale":"2024-04-16T16:22:46.485748+0000","last_undegraded":"2024-04-16T16:22:46.485748+0000","last_fullsized":"2024-04-16T16:22:46.485748+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:46:58.611028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.478755+0000","last_change":"2024-04-16T16:22:46.467282+0000","last_active":"2024-04-16T16:22:46.478755+0000","last_peered":"2024-04-16T16:22:46.478755+0000","last_clean":"2024-04-16T16:22:46.478755+0000","last_became_active":"2024-04-16T16:22:46.466756+0000","last_became_peered":"2024-04-16T16:22:46.466756+0000","last_unstale":"2024-04-16T16:22:46.478755+0000","last_undegraded":"2024-04-16T16:22:46.478755+0000","last_fullsized":"2024-04-16T16:22:46.478755+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:22.841005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.487969+0000","last_change":"2024-04-16T16:22:46.484066+0000","last_active":"2024-04-16T16:22:46.487969+0000","last_peered":"2024-04-16T16:22:46.487969+0000","last_clean":"2024-04-16T16:22:46.487969+0000","last_became_active":"2024-04-16T16:22:46.483654+0000","last_became_peered":"2024-04-16T16:22:46.483654+0000","last_unstale":"2024-04-16T16:22:46.487969+0000","last_undegraded":"2024-04-16T16:22:46.487969+0000","last_fullsized":"2024-04-16T16:22:46.487969+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:52:58.419267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.485641+0000","last_change":"2024-04-16T16:22:46.484111+0000","last_active":"2024-04-16T16:22:46.485641+0000","last_peered":"2024-04-16T16:22:46.485641+0000","last_clean":"2024-04-16T16:22:46.485641+0000","last_became_active":"2024-04-16T16:22:46.483750+0000","last_became_peered":"2024-04-16T16:22:46.483750+0000","last_unstale":"2024-04-16T16:22:46.485641+0000","last_undegraded":"2024-04-16T16:22:46.485641+0000","last_fullsized":"2024-04-16T16:22:46.485641+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:53:06.786966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.487385+0000","last_change":"2024-04-16T16:22:46.483339+0000","last_active":"2024-04-16T16:22:46.487385+0000","last_peered":"2024-04-16T16:22:46.487385+0000","last_clean":"2024-04-16T16:22:46.487385+0000","last_became_active":"2024-04-16T16:22:46.481358+0000","last_became_peered":"2024-04-16T16:22:46.481358+0000","last_unstale":"2024-04-16T16:22:46.487385+0000","last_undegraded":"2024-04-16T16:22:46.487385+0000","last_fullsized":"2024-04-16T16:22:46.487385+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:30.402337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475943+0000","last_change":"2024-04-16T16:22:46.464311+0000","last_active":"2024-04-16T16:22:46.475943+0000","last_peered":"2024-04-16T16:22:46.475943+0000","last_clean":"2024-04-16T16:22:46.475943+0000","last_became_active":"2024-04-16T16:22:46.463536+0000","last_became_peered":"2024-04-16T16:22:46.463536+0000","last_unstale":"2024-04-16T16:22:46.475943+0000","last_undegraded":"2024-04-16T16:22:46.475943+0000","last_fullsized":"2024-04-16T16:22:46.475943+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:13.979255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475259+0000","last_change":"2024-04-16T16:22:46.460441+0000","last_active":"2024-04-16T16:22:46.475259+0000","last_peered":"2024-04-16T16:22:46.475259+0000","last_clean":"2024-04-16T16:22:46.475259+0000","last_became_active":"2024-04-16T16:22:46.458962+0000","last_became_peered":"2024-04-16T16:22:46.458962+0000","last_unstale":"2024-04-16T16:22:46.475259+0000","last_undegraded":"2024-04-16T16:22:46.475259+0000","last_fullsized":"2024-04-16T16:22:46.475259+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:44:40.829325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475972+0000","last_change":"2024-04-16T16:22:46.468514+0000","last_active":"2024-04-16T16:22:46.475972+0000","last_peered":"2024-04-16T16:22:46.475972+0000","last_clean":"2024-04-16T16:22:46.475972+0000","last_became_active":"2024-04-16T16:22:46.464851+0000","last_became_peered":"2024-04-16T16:22:46.464851+0000","last_unstale":"2024-04-16T16:22:46.475972+0000","last_undegraded":"2024-04-16T16:22:46.475972+0000","last_fullsized":"2024-04-16T16:22:46.475972+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:22.841005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.478711+0000","last_change":"2024-04-16T16:22:46.467039+0000","last_active":"2024-04-16T16:22:46.478711+0000","last_peered":"2024-04-16T16:22:46.478711+0000","last_clean":"2024-04-16T16:22:46.478711+0000","last_became_active":"2024-04-16T16:22:46.466366+0000","last_became_peered":"2024-04-16T16:22:46.466366+0000","last_unstale":"2024-04-16T16:22:46.478711+0000","last_undegraded":"2024-04-16T16:22:46.478711+0000","last_fullsized":"2024-04-16T16:22:46.478711+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:53:48.166033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"155'4","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.488480+0000","last_change":"2024-04-16T16:22:45.470039+0000","last_active":"2024-04-16T16:22:46.488480+0000","last_peered":"2024-04-16T16:22:46.488480+0000","last_clean":"2024-04-16T16:22:46.488480+0000","last_became_active":"2024-04-16T16:22:45.467695+0000","last_became_peered":"2024-04-16T16:22:45.467695+0000","last_unstale":"2024-04-16T16:22:46.488480+0000","last_undegraded":"2024-04-16T16:22:46.488480+0000","last_fullsized":"2024-04-16T16:22:46.488480+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:21.824987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.478465+0000","last_change":"2024-04-16T16:22:46.466378+0000","last_active":"2024-04-16T16:22:46.478465+0000","last_peered":"2024-04-16T16:22:46.478465+0000","last_clean":"2024-04-16T16:22:46.478465+0000","last_became_active":"2024-04-16T16:22:46.462099+0000","last_became_peered":"2024-04-16T16:22:46.462099+0000","last_unstale":"2024-04-16T16:22:46.478465+0000","last_undegraded":"2024-04-16T16:22:46.478465+0000","last_fullsized":"2024-04-16T16:22:46.478465+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:53:06.786966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.484979+0000","last_change":"2024-04-16T16:22:45.464685+0000","last_active":"2024-04-16T16:22:46.484979+0000","last_peered":"2024-04-16T16:22:46.484979+0000","last_clean":"2024-04-16T16:22:46.484979+0000","last_became_active":"2024-04-16T16:22:45.464379+0000","last_became_peered":"2024-04-16T16:22:45.464379+0000","last_unstale":"2024-04-16T16:22:46.484979+0000","last_undegraded":"2024-04-16T16:22:46.484979+0000","last_fullsized":"2024-04-16T16:22:46.484979+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:05:35.552664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710111+0000","last_change":"2024-04-16T16:22:46.456302+0000","last_active":"2024-04-16T16:22:48.710111+0000","last_peered":"2024-04-16T16:22:48.710111+0000","last_clean":"2024-04-16T16:22:48.710111+0000","last_became_active":"2024-04-16T16:22:46.454610+0000","last_became_peered":"2024-04-16T16:22:46.454610+0000","last_unstale":"2024-04-16T16:22:48.710111+0000","last_undegraded":"2024-04-16T16:22:48.710111+0000","last_fullsized":"2024-04-16T16:22:48.710111+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:56:12.143605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.484025+0000","last_change":"2024-04-16T16:22:45.469877+0000","last_active":"2024-04-16T16:22:46.484025+0000","last_peered":"2024-04-16T16:22:46.484025+0000","last_clean":"2024-04-16T16:22:46.484025+0000","last_became_active":"2024-04-16T16:22:45.469537+0000","last_became_peered":"2024-04-16T16:22:45.469537+0000","last_unstale":"2024-04-16T16:22:46.484025+0000","last_undegraded":"2024-04-16T16:22:46.484025+0000","last_fullsized":"2024-04-16T16:22:46.484025+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:54.078259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.485663+0000","last_change":"2024-04-16T16:22:46.480073+0000","last_active":"2024-04-16T16:22:46.485663+0000","last_peered":"2024-04-16T16:22:46.485663+0000","last_clean":"2024-04-16T16:22:46.485663+0000","last_became_active":"2024-04-16T16:22:46.479206+0000","last_became_peered":"2024-04-16T16:22:46.479206+0000","last_unstale":"2024-04-16T16:22:46.485663+0000","last_undegraded":"2024-04-16T16:22:46.485663+0000","last_fullsized":"2024-04-16T16:22:46.485663+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:46.979782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.474697+0000","last_change":"2024-04-16T16:22:45.462903+0000","last_active":"2024-04-16T16:22:46.474697+0000","last_peered":"2024-04-16T16:22:46.474697+0000","last_clean":"2024-04-16T16:22:46.474697+0000","last_became_active":"2024-04-16T16:22:45.462170+0000","last_became_peered":"2024-04-16T16:22:45.462170+0000","last_unstale":"2024-04-16T16:22:46.474697+0000","last_undegraded":"2024-04-16T16:22:46.474697+0000","last_fullsized":"2024-04-16T16:22:46.474697+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:07.564200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.465111+0000","last_change":"2024-04-16T16:22:46.460202+0000","last_active":"2024-04-16T16:22:46.465111+0000","last_peered":"2024-04-16T16:22:46.465111+0000","last_clean":"2024-04-16T16:22:46.465111+0000","last_became_active":"2024-04-16T16:22:46.459876+0000","last_became_peered":"2024-04-16T16:22:46.459876+0000","last_unstale":"2024-04-16T16:22:46.465111+0000","last_undegraded":"2024-04-16T16:22:46.465111+0000","last_fullsized":"2024-04-16T16:22:46.465111+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:05:07.007726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.484235+0000","last_change":"2024-04-16T16:22:45.472425+0000","last_active":"2024-04-16T16:22:46.484235+0000","last_peered":"2024-04-16T16:22:46.484235+0000","last_clean":"2024-04-16T16:22:46.484235+0000","last_became_active":"2024-04-16T16:22:45.472020+0000","last_became_peered":"2024-04-16T16:22:45.472020+0000","last_unstale":"2024-04-16T16:22:46.484235+0000","last_undegraded":"2024-04-16T16:22:46.484235+0000","last_fullsized":"2024-04-16T16:22:46.484235+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:48:51.793606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476249+0000","last_change":"2024-04-16T16:22:46.470665+0000","last_active":"2024-04-16T16:22:46.476249+0000","last_peered":"2024-04-16T16:22:46.476249+0000","last_clean":"2024-04-16T16:22:46.476249+0000","last_became_active":"2024-04-16T16:22:46.467898+0000","last_became_peered":"2024-04-16T16:22:46.467898+0000","last_unstale":"2024-04-16T16:22:46.476249+0000","last_undegraded":"2024-04-16T16:22:46.476249+0000","last_fullsized":"2024-04-16T16:22:46.476249+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:35.308909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479141+0000","last_change":"2024-04-16T16:22:45.462429+0000","last_active":"2024-04-16T16:22:46.479141+0000","last_peered":"2024-04-16T16:22:46.479141+0000","last_clean":"2024-04-16T16:22:46.479141+0000","last_became_active":"2024-04-16T16:22:45.461674+0000","last_became_peered":"2024-04-16T16:22:45.461674+0000","last_unstale":"2024-04-16T16:22:46.479141+0000","last_undegraded":"2024-04-16T16:22:46.479141+0000","last_fullsized":"2024-04-16T16:22:46.479141+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:29.386319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476205+0000","last_change":"2024-04-16T16:22:46.465672+0000","last_active":"2024-04-16T16:22:46.476205+0000","last_peered":"2024-04-16T16:22:46.476205+0000","last_clean":"2024-04-16T16:22:46.476205+0000","last_became_active":"2024-04-16T16:22:46.465329+0000","last_became_peered":"2024-04-16T16:22:46.465329+0000","last_unstale":"2024-04-16T16:22:46.476205+0000","last_undegraded":"2024-04-16T16:22:46.476205+0000","last_fullsized":"2024-04-16T16:22:46.476205+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:05:36.568682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.465610+0000","last_change":"2024-04-16T16:22:45.466868+0000","last_active":"2024-04-16T16:22:46.465610+0000","last_peered":"2024-04-16T16:22:46.465610+0000","last_clean":"2024-04-16T16:22:46.465610+0000","last_became_active":"2024-04-16T16:22:45.466591+0000","last_became_peered":"2024-04-16T16:22:45.466591+0000","last_unstale":"2024-04-16T16:22:46.465610+0000","last_undegraded":"2024-04-16T16:22:46.465610+0000","last_fullsized":"2024-04-16T16:22:46.465610+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:25.557203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710073+0000","last_change":"2024-04-16T16:22:46.460701+0000","last_active":"2024-04-16T16:22:48.710073+0000","last_peered":"2024-04-16T16:22:48.710073+0000","last_clean":"2024-04-16T16:22:48.710073+0000","last_became_active":"2024-04-16T16:22:46.460420+0000","last_became_peered":"2024-04-16T16:22:46.460420+0000","last_unstale":"2024-04-16T16:22:48.710073+0000","last_undegraded":"2024-04-16T16:22:48.710073+0000","last_fullsized":"2024-04-16T16:22:48.710073+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:35:00.579428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.451708+0000","last_change":"2024-04-16T16:22:45.456041+0000","last_active":"2024-04-16T16:22:46.451708+0000","last_peered":"2024-04-16T16:22:46.451708+0000","last_clean":"2024-04-16T16:22:46.451708+0000","last_became_active":"2024-04-16T16:22:45.455767+0000","last_became_peered":"2024-04-16T16:22:45.455767+0000","last_unstale":"2024-04-16T16:22:46.451708+0000","last_undegraded":"2024-04-16T16:22:46.451708+0000","last_fullsized":"2024-04-16T16:22:46.451708+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:26:37.373159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475982+0000","last_change":"2024-04-16T16:22:46.467743+0000","last_active":"2024-04-16T16:22:46.475982+0000","last_peered":"2024-04-16T16:22:46.475982+0000","last_clean":"2024-04-16T16:22:46.475982+0000","last_became_active":"2024-04-16T16:22:46.465149+0000","last_became_peered":"2024-04-16T16:22:46.465149+0000","last_unstale":"2024-04-16T16:22:46.475982+0000","last_undegraded":"2024-04-16T16:22:46.475982+0000","last_fullsized":"2024-04-16T16:22:46.475982+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:53:48.166033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710644+0000","last_change":"2024-04-16T16:22:45.467985+0000","last_active":"2024-04-16T16:22:48.710644+0000","last_peered":"2024-04-16T16:22:48.710644+0000","last_clean":"2024-04-16T16:22:48.710644+0000","last_became_active":"2024-04-16T16:22:45.467704+0000","last_became_peered":"2024-04-16T16:22:45.467704+0000","last_unstale":"2024-04-16T16:22:48.710644+0000","last_undegraded":"2024-04-16T16:22:48.710644+0000","last_fullsized":"2024-04-16T16:22:48.710644+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:58.753768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476055+0000","last_change":"2024-04-16T16:22:46.473841+0000","last_active":"2024-04-16T16:22:46.476055+0000","last_peered":"2024-04-16T16:22:46.476055+0000","last_clean":"2024-04-16T16:22:46.476055+0000","last_became_active":"2024-04-16T16:22:46.473500+0000","last_became_peered":"2024-04-16T16:22:46.473500+0000","last_unstale":"2024-04-16T16:22:46.476055+0000","last_undegraded":"2024-04-16T16:22:46.476055+0000","last_fullsized":"2024-04-16T16:22:46.476055+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:51:03.090923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"155'3","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710356+0000","last_change":"2024-04-16T16:22:45.461376+0000","last_active":"2024-04-16T16:22:48.710356+0000","last_peered":"2024-04-16T16:22:48.710356+0000","last_clean":"2024-04-16T16:22:48.710356+0000","last_became_active":"2024-04-16T16:22:45.461088+0000","last_became_peered":"2024-04-16T16:22:45.461088+0000","last_unstale":"2024-04-16T16:22:48.710356+0000","last_undegraded":"2024-04-16T16:22:48.710356+0000","last_fullsized":"2024-04-16T16:22:48.710356+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:43:07.885858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.477917+0000","last_change":"2024-04-16T16:22:46.473263+0000","last_active":"2024-04-16T16:22:46.477917+0000","last_peered":"2024-04-16T16:22:46.477917+0000","last_clean":"2024-04-16T16:22:46.477917+0000","last_became_active":"2024-04-16T16:22:46.472979+0000","last_became_peered":"2024-04-16T16:22:46.472979+0000","last_unstale":"2024-04-16T16:22:46.477917+0000","last_undegraded":"2024-04-16T16:22:46.477917+0000","last_fullsized":"2024-04-16T16:22:46.477917+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:22:33.768188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"155'6","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.490256+0000","last_change":"2024-04-16T16:22:45.462782+0000","last_active":"2024-04-16T16:22:46.490256+0000","last_peered":"2024-04-16T16:22:46.490256+0000","last_clean":"2024-04-16T16:22:46.490256+0000","last_became_active":"2024-04-16T16:22:45.461583+0000","last_became_peered":"2024-04-16T16:22:45.461583+0000","last_unstale":"2024-04-16T16:22:46.490256+0000","last_undegraded":"2024-04-16T16:22:46.490256+0000","last_fullsized":"2024-04-16T16:22:46.490256+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476052+0000","last_change":"2024-04-16T16:22:46.466170+0000","last_active":"2024-04-16T16:22:46.476052+0000","last_peered":"2024-04-16T16:22:46.476052+0000","last_clean":"2024-04-16T16:22:46.476052+0000","last_became_active":"2024-04-16T16:22:46.465894+0000","last_became_peered":"2024-04-16T16:22:46.465894+0000","last_unstale":"2024-04-16T16:22:46.476052+0000","last_undegraded":"2024-04-16T16:22:46.476052+0000","last_fullsized":"2024-04-16T16:22:46.476052+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:47.034586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.477731+0000","last_change":"2024-04-16T16:22:45.466273+0000","last_active":"2024-04-16T16:22:46.477731+0000","last_peered":"2024-04-16T16:22:46.477731+0000","last_clean":"2024-04-16T16:22:46.477731+0000","last_became_active":"2024-04-16T16:22:45.465893+0000","last_became_peered":"2024-04-16T16:22:45.465893+0000","last_unstale":"2024-04-16T16:22:46.477731+0000","last_undegraded":"2024-04-16T16:22:46.477731+0000","last_fullsized":"2024-04-16T16:22:46.477731+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:52:57.403249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479098+0000","last_change":"2024-04-16T16:22:46.474507+0000","last_active":"2024-04-16T16:22:46.479098+0000","last_peered":"2024-04-16T16:22:46.479098+0000","last_clean":"2024-04-16T16:22:46.479098+0000","last_became_active":"2024-04-16T16:22:46.474146+0000","last_became_peered":"2024-04-16T16:22:46.474146+0000","last_unstale":"2024-04-16T16:22:46.479098+0000","last_undegraded":"2024-04-16T16:22:46.479098+0000","last_fullsized":"2024-04-16T16:22:46.479098+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:59:47.867717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.449766+0000","last_change":"2024-04-16T16:22:45.463747+0000","last_active":"2024-04-16T16:22:46.449766+0000","last_peered":"2024-04-16T16:22:46.449766+0000","last_clean":"2024-04-16T16:22:46.449766+0000","last_became_active":"2024-04-16T16:22:45.462875+0000","last_became_peered":"2024-04-16T16:22:45.462875+0000","last_unstale":"2024-04-16T16:22:46.449766+0000","last_undegraded":"2024-04-16T16:22:46.449766+0000","last_fullsized":"2024-04-16T16:22:46.449766+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:29:33.964433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476854+0000","last_change":"2024-04-16T16:22:46.473128+0000","last_active":"2024-04-16T16:22:46.476854+0000","last_peered":"2024-04-16T16:22:46.476854+0000","last_clean":"2024-04-16T16:22:46.476854+0000","last_became_active":"2024-04-16T16:22:46.472859+0000","last_became_peered":"2024-04-16T16:22:46.472859+0000","last_unstale":"2024-04-16T16:22:46.476854+0000","last_undegraded":"2024-04-16T16:22:46.476854+0000","last_fullsized":"2024-04-16T16:22:46.476854+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:44:40.829325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.454247+0000","last_change":"2024-04-16T16:22:45.466908+0000","last_active":"2024-04-16T16:22:46.454247+0000","last_peered":"2024-04-16T16:22:46.454247+0000","last_clean":"2024-04-16T16:22:46.454247+0000","last_became_active":"2024-04-16T16:22:45.466535+0000","last_became_peered":"2024-04-16T16:22:45.466535+0000","last_unstale":"2024-04-16T16:22:46.454247+0000","last_undegraded":"2024-04-16T16:22:46.454247+0000","last_fullsized":"2024-04-16T16:22:46.454247+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:25.557203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.487178+0000","last_change":"2024-04-16T16:22:46.483216+0000","last_active":"2024-04-16T16:22:46.487178+0000","last_peered":"2024-04-16T16:22:46.487178+0000","last_clean":"2024-04-16T16:22:46.487178+0000","last_became_active":"2024-04-16T16:22:46.482943+0000","last_became_peered":"2024-04-16T16:22:46.482943+0000","last_unstale":"2024-04-16T16:22:46.487178+0000","last_undegraded":"2024-04-16T16:22:46.487178+0000","last_fullsized":"2024-04-16T16:22:46.487178+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:17:28.908308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"155'8","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.484924+0000","last_change":"2024-04-16T16:22:45.461068+0000","last_active":"2024-04-16T16:22:46.484924+0000","last_peered":"2024-04-16T16:22:46.484924+0000","last_clean":"2024-04-16T16:22:46.484924+0000","last_became_active":"2024-04-16T16:22:45.459760+0000","last_became_peered":"2024-04-16T16:22:45.459760+0000","last_unstale":"2024-04-16T16:22:46.484924+0000","last_undegraded":"2024-04-16T16:22:46.484924+0000","last_fullsized":"2024-04-16T16:22:46.484924+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:32:24.648814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.477198+0000","last_change":"2024-04-16T16:22:46.473695+0000","last_active":"2024-04-16T16:22:46.477198+0000","last_peered":"2024-04-16T16:22:46.477198+0000","last_clean":"2024-04-16T16:22:46.477198+0000","last_became_active":"2024-04-16T16:22:46.471894+0000","last_became_peered":"2024-04-16T16:22:46.471894+0000","last_unstale":"2024-04-16T16:22:46.477198+0000","last_undegraded":"2024-04-16T16:22:46.477198+0000","last_fullsized":"2024-04-16T16:22:46.477198+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:27.266104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.484686+0000","last_change":"2024-04-16T16:22:45.469335+0000","last_active":"2024-04-16T16:22:46.484686+0000","last_peered":"2024-04-16T16:22:46.484686+0000","last_clean":"2024-04-16T16:22:46.484686+0000","last_became_active":"2024-04-16T16:22:45.468966+0000","last_became_peered":"2024-04-16T16:22:45.468966+0000","last_unstale":"2024-04-16T16:22:46.484686+0000","last_undegraded":"2024-04-16T16:22:46.484686+0000","last_fullsized":"2024-04-16T16:22:46.484686+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:09.300484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476015+0000","last_change":"2024-04-16T16:22:46.471925+0000","last_active":"2024-04-16T16:22:46.476015+0000","last_peered":"2024-04-16T16:22:46.476015+0000","last_clean":"2024-04-16T16:22:46.476015+0000","last_became_active":"2024-04-16T16:22:46.470405+0000","last_became_peered":"2024-04-16T16:22:46.470405+0000","last_unstale":"2024-04-16T16:22:46.476015+0000","last_undegraded":"2024-04-16T16:22:46.476015+0000","last_fullsized":"2024-04-16T16:22:46.476015+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:11.331337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.480955+0000","last_change":"2024-04-16T16:22:45.470272+0000","last_active":"2024-04-16T16:22:46.480955+0000","last_peered":"2024-04-16T16:22:46.480955+0000","last_clean":"2024-04-16T16:22:46.480955+0000","last_became_active":"2024-04-16T16:22:45.469959+0000","last_became_peered":"2024-04-16T16:22:45.469959+0000","last_unstale":"2024-04-16T16:22:46.480955+0000","last_undegraded":"2024-04-16T16:22:46.480955+0000","last_fullsized":"2024-04-16T16:22:46.480955+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:34:03.813486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.474133+0000","last_change":"2024-04-16T16:22:46.462951+0000","last_active":"2024-04-16T16:22:46.474133+0000","last_peered":"2024-04-16T16:22:46.474133+0000","last_clean":"2024-04-16T16:22:46.474133+0000","last_became_active":"2024-04-16T16:22:46.462662+0000","last_became_peered":"2024-04-16T16:22:46.462662+0000","last_unstale":"2024-04-16T16:22:46.474133+0000","last_undegraded":"2024-04-16T16:22:46.474133+0000","last_fullsized":"2024-04-16T16:22:46.474133+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:32:25.664832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.482804+0000","last_change":"2024-04-16T16:22:45.462862+0000","last_active":"2024-04-16T16:22:46.482804+0000","last_peered":"2024-04-16T16:22:46.482804+0000","last_clean":"2024-04-16T16:22:46.482804+0000","last_became_active":"2024-04-16T16:22:45.462197+0000","last_became_peered":"2024-04-16T16:22:45.462197+0000","last_unstale":"2024-04-16T16:22:46.482804+0000","last_undegraded":"2024-04-16T16:22:46.482804+0000","last_fullsized":"2024-04-16T16:22:46.482804+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:55.697931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.478880+0000","last_change":"2024-04-16T16:22:46.475567+0000","last_active":"2024-04-16T16:22:46.478880+0000","last_peered":"2024-04-16T16:22:46.478880+0000","last_clean":"2024-04-16T16:22:46.478880+0000","last_became_active":"2024-04-16T16:22:46.475232+0000","last_became_peered":"2024-04-16T16:22:46.475232+0000","last_unstale":"2024-04-16T16:22:46.478880+0000","last_undegraded":"2024-04-16T16:22:46.478880+0000","last_fullsized":"2024-04-16T16:22:46.478880+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:01.225311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.449795+0000","last_change":"2024-04-16T16:22:45.466893+0000","last_active":"2024-04-16T16:22:46.449795+0000","last_peered":"2024-04-16T16:22:46.449795+0000","last_clean":"2024-04-16T16:22:46.449795+0000","last_became_active":"2024-04-16T16:22:45.466614+0000","last_became_peered":"2024-04-16T16:22:45.466614+0000","last_unstale":"2024-04-16T16:22:46.449795+0000","last_undegraded":"2024-04-16T16:22:46.449795+0000","last_fullsized":"2024-04-16T16:22:46.449795+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:53:47.150015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.487053+0000","last_change":"2024-04-16T16:22:46.482036+0000","last_active":"2024-04-16T16:22:46.487053+0000","last_peered":"2024-04-16T16:22:46.487053+0000","last_clean":"2024-04-16T16:22:46.487053+0000","last_became_active":"2024-04-16T16:22:46.481293+0000","last_became_peered":"2024-04-16T16:22:46.481293+0000","last_unstale":"2024-04-16T16:22:46.487053+0000","last_undegraded":"2024-04-16T16:22:46.487053+0000","last_fullsized":"2024-04-16T16:22:46.487053+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:10.466744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.484240+0000","last_change":"2024-04-16T16:22:45.466947+0000","last_active":"2024-04-16T16:22:46.484240+0000","last_peered":"2024-04-16T16:22:46.484240+0000","last_clean":"2024-04-16T16:22:46.484240+0000","last_became_active":"2024-04-16T16:22:45.466624+0000","last_became_peered":"2024-04-16T16:22:45.466624+0000","last_unstale":"2024-04-16T16:22:46.484240+0000","last_undegraded":"2024-04-16T16:22:46.484240+0000","last_fullsized":"2024-04-16T16:22:46.484240+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:53.044305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.478797+0000","last_change":"2024-04-16T16:22:46.469862+0000","last_active":"2024-04-16T16:22:46.478797+0000","last_peered":"2024-04-16T16:22:46.478797+0000","last_clean":"2024-04-16T16:22:46.478797+0000","last_became_active":"2024-04-16T16:22:46.466179+0000","last_became_peered":"2024-04-16T16:22:46.466179+0000","last_unstale":"2024-04-16T16:22:46.478797+0000","last_undegraded":"2024-04-16T16:22:46.478797+0000","last_fullsized":"2024-04-16T16:22:46.478797+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:51:03.090923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.445757+0000","last_change":"2024-04-16T16:22:45.461564+0000","last_active":"2024-04-16T16:22:46.445757+0000","last_peered":"2024-04-16T16:22:46.445757+0000","last_clean":"2024-04-16T16:22:46.445757+0000","last_became_active":"2024-04-16T16:22:45.461253+0000","last_became_peered":"2024-04-16T16:22:45.461253+0000","last_unstale":"2024-04-16T16:22:46.445757+0000","last_undegraded":"2024-04-16T16:22:46.445757+0000","last_fullsized":"2024-04-16T16:22:46.445757+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:46.018568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.472435+0000","last_change":"2024-04-16T16:22:46.465947+0000","last_active":"2024-04-16T16:22:46.472435+0000","last_peered":"2024-04-16T16:22:46.472435+0000","last_clean":"2024-04-16T16:22:46.472435+0000","last_became_active":"2024-04-16T16:22:46.465294+0000","last_became_peered":"2024-04-16T16:22:46.465294+0000","last_unstale":"2024-04-16T16:22:46.472435+0000","last_undegraded":"2024-04-16T16:22:46.472435+0000","last_fullsized":"2024-04-16T16:22:46.472435+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:11.331337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.454403+0000","last_change":"2024-04-16T16:22:45.468883+0000","last_active":"2024-04-16T16:22:46.454403+0000","last_peered":"2024-04-16T16:22:46.454403+0000","last_clean":"2024-04-16T16:22:46.454403+0000","last_became_active":"2024-04-16T16:22:45.468091+0000","last_became_peered":"2024-04-16T16:22:45.468091+0000","last_unstale":"2024-04-16T16:22:46.454403+0000","last_undegraded":"2024-04-16T16:22:46.454403+0000","last_fullsized":"2024-04-16T16:22:46.454403+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710495+0000","last_change":"2024-04-16T16:22:46.471354+0000","last_active":"2024-04-16T16:22:48.710495+0000","last_peered":"2024-04-16T16:22:48.710495+0000","last_clean":"2024-04-16T16:22:48.710495+0000","last_became_active":"2024-04-16T16:22:46.470521+0000","last_became_peered":"2024-04-16T16:22:46.470521+0000","last_unstale":"2024-04-16T16:22:48.710495+0000","last_undegraded":"2024-04-16T16:22:48.710495+0000","last_fullsized":"2024-04-16T16:22:48.710495+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:06:08.736226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.445445+0000","last_change":"2024-04-16T16:22:45.460865+0000","last_active":"2024-04-16T16:22:46.445445+0000","last_peered":"2024-04-16T16:22:46.445445+0000","last_clean":"2024-04-16T16:22:46.445445+0000","last_became_active":"2024-04-16T16:22:45.458861+0000","last_became_peered":"2024-04-16T16:22:45.458861+0000","last_unstale":"2024-04-16T16:22:46.445445+0000","last_undegraded":"2024-04-16T16:22:46.445445+0000","last_fullsized":"2024-04-16T16:22:46.445445+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:30.462179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475680+0000","last_change":"2024-04-16T16:22:46.464764+0000","last_active":"2024-04-16T16:22:46.475680+0000","last_peered":"2024-04-16T16:22:46.475680+0000","last_clean":"2024-04-16T16:22:46.475680+0000","last_became_active":"2024-04-16T16:22:46.464064+0000","last_became_peered":"2024-04-16T16:22:46.464064+0000","last_unstale":"2024-04-16T16:22:46.475680+0000","last_undegraded":"2024-04-16T16:22:46.475680+0000","last_fullsized":"2024-04-16T16:22:46.475680+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:26:38.389177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.480960+0000","last_change":"2024-04-16T16:22:45.466044+0000","last_active":"2024-04-16T16:22:46.480960+0000","last_peered":"2024-04-16T16:22:46.480960+0000","last_clean":"2024-04-16T16:22:46.480960+0000","last_became_active":"2024-04-16T16:22:45.465676+0000","last_became_peered":"2024-04-16T16:22:45.465676+0000","last_unstale":"2024-04-16T16:22:46.480960+0000","last_undegraded":"2024-04-16T16:22:46.480960+0000","last_fullsized":"2024-04-16T16:22:46.480960+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:52.385757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.478576+0000","last_change":"2024-04-16T16:22:46.466356+0000","last_active":"2024-04-16T16:22:46.478576+0000","last_peered":"2024-04-16T16:22:46.478576+0000","last_clean":"2024-04-16T16:22:46.478576+0000","last_became_active":"2024-04-16T16:22:46.465449+0000","last_became_peered":"2024-04-16T16:22:46.465449+0000","last_unstale":"2024-04-16T16:22:46.478576+0000","last_undegraded":"2024-04-16T16:22:46.478576+0000","last_fullsized":"2024-04-16T16:22:46.478576+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:47.034586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.471171+0000","last_change":"2024-04-16T16:22:45.466156+0000","last_active":"2024-04-16T16:22:46.471171+0000","last_peered":"2024-04-16T16:22:46.471171+0000","last_clean":"2024-04-16T16:22:46.471171+0000","last_became_active":"2024-04-16T16:22:45.465791+0000","last_became_peered":"2024-04-16T16:22:45.465791+0000","last_unstale":"2024-04-16T16:22:46.471171+0000","last_undegraded":"2024-04-16T16:22:46.471171+0000","last_fullsized":"2024-04-16T16:22:46.471171+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:42:46.394979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710309+0000","last_change":"2024-04-16T16:22:46.470854+0000","last_active":"2024-04-16T16:22:48.710309+0000","last_peered":"2024-04-16T16:22:48.710309+0000","last_clean":"2024-04-16T16:22:48.710309+0000","last_became_active":"2024-04-16T16:22:46.470196+0000","last_became_peered":"2024-04-16T16:22:46.470196+0000","last_unstale":"2024-04-16T16:22:48.710309+0000","last_undegraded":"2024-04-16T16:22:48.710309+0000","last_fullsized":"2024-04-16T16:22:48.710309+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:40:24.085281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.460207+0000","last_change":"2024-04-16T16:22:45.471205+0000","last_active":"2024-04-16T16:22:46.460207+0000","last_peered":"2024-04-16T16:22:46.460207+0000","last_clean":"2024-04-16T16:22:46.460207+0000","last_became_active":"2024-04-16T16:22:45.470911+0000","last_became_peered":"2024-04-16T16:22:45.470911+0000","last_unstale":"2024-04-16T16:22:46.460207+0000","last_undegraded":"2024-04-16T16:22:46.460207+0000","last_fullsized":"2024-04-16T16:22:46.460207+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:53.044305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476431+0000","last_change":"2024-04-16T16:22:46.469324+0000","last_active":"2024-04-16T16:22:46.476431+0000","last_peered":"2024-04-16T16:22:46.476431+0000","last_clean":"2024-04-16T16:22:46.476431+0000","last_became_active":"2024-04-16T16:22:46.469009+0000","last_became_peered":"2024-04-16T16:22:46.469009+0000","last_unstale":"2024-04-16T16:22:46.476431+0000","last_undegraded":"2024-04-16T16:22:46.476431+0000","last_fullsized":"2024-04-16T16:22:46.476431+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:56.713949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.455944+0000","last_change":"2024-04-16T16:22:45.468189+0000","last_active":"2024-04-16T16:22:46.455944+0000","last_peered":"2024-04-16T16:22:46.455944+0000","last_clean":"2024-04-16T16:22:46.455944+0000","last_became_active":"2024-04-16T16:22:45.467709+0000","last_became_peered":"2024-04-16T16:22:45.467709+0000","last_unstale":"2024-04-16T16:22:46.455944+0000","last_undegraded":"2024-04-16T16:22:46.455944+0000","last_fullsized":"2024-04-16T16:22:46.455944+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:46.256132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.476031+0000","last_change":"2024-04-16T16:22:46.465994+0000","last_active":"2024-04-16T16:22:46.476031+0000","last_peered":"2024-04-16T16:22:46.476031+0000","last_clean":"2024-04-16T16:22:46.476031+0000","last_became_active":"2024-04-16T16:22:46.465007+0000","last_became_peered":"2024-04-16T16:22:46.465007+0000","last_unstale":"2024-04-16T16:22:46.476031+0000","last_undegraded":"2024-04-16T16:22:46.476031+0000","last_fullsized":"2024-04-16T16:22:46.476031+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:31.478197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.448890+0000","last_change":"2024-04-16T16:22:45.470607+0000","last_active":"2024-04-16T16:22:46.448890+0000","last_peered":"2024-04-16T16:22:46.448890+0000","last_clean":"2024-04-16T16:22:46.448890+0000","last_became_active":"2024-04-16T16:22:45.469322+0000","last_became_peered":"2024-04-16T16:22:45.469322+0000","last_unstale":"2024-04-16T16:22:46.448890+0000","last_undegraded":"2024-04-16T16:22:46.448890+0000","last_fullsized":"2024-04-16T16:22:46.448890+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:07.564200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475798+0000","last_change":"2024-04-16T16:22:46.462483+0000","last_active":"2024-04-16T16:22:46.475798+0000","last_peered":"2024-04-16T16:22:46.475798+0000","last_clean":"2024-04-16T16:22:46.475798+0000","last_became_active":"2024-04-16T16:22:46.461432+0000","last_became_peered":"2024-04-16T16:22:46.461432+0000","last_unstale":"2024-04-16T16:22:46.475798+0000","last_undegraded":"2024-04-16T16:22:46.475798+0000","last_fullsized":"2024-04-16T16:22:46.475798+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:27.266104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.451322+0000","last_change":"2024-04-16T16:22:45.462752+0000","last_active":"2024-04-16T16:22:46.451322+0000","last_peered":"2024-04-16T16:22:46.451322+0000","last_clean":"2024-04-16T16:22:46.451322+0000","last_became_active":"2024-04-16T16:22:45.462154+0000","last_became_peered":"2024-04-16T16:22:45.462154+0000","last_unstale":"2024-04-16T16:22:46.451322+0000","last_undegraded":"2024-04-16T16:22:46.451322+0000","last_fullsized":"2024-04-16T16:22:46.451322+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:43:04.134472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.479013+0000","last_change":"2024-04-16T16:22:46.467575+0000","last_active":"2024-04-16T16:22:46.479013+0000","last_peered":"2024-04-16T16:22:46.479013+0000","last_clean":"2024-04-16T16:22:46.479013+0000","last_became_active":"2024-04-16T16:22:46.466890+0000","last_became_peered":"2024-04-16T16:22:46.466890+0000","last_unstale":"2024-04-16T16:22:46.479013+0000","last_undegraded":"2024-04-16T16:22:46.479013+0000","last_fullsized":"2024-04-16T16:22:46.479013+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:56.713949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.466502+0000","last_change":"2024-04-16T16:22:45.469689+0000","last_active":"2024-04-16T16:22:46.466502+0000","last_peered":"2024-04-16T16:22:46.466502+0000","last_clean":"2024-04-16T16:22:46.466502+0000","last_became_active":"2024-04-16T16:22:45.469350+0000","last_became_peered":"2024-04-16T16:22:45.469350+0000","last_unstale":"2024-04-16T16:22:46.466502+0000","last_undegraded":"2024-04-16T16:22:46.466502+0000","last_fullsized":"2024-04-16T16:22:46.466502+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.482785+0000","last_change":"2024-04-16T16:22:46.478730+0000","last_active":"2024-04-16T16:22:46.482785+0000","last_peered":"2024-04-16T16:22:46.482785+0000","last_clean":"2024-04-16T16:22:46.482785+0000","last_became_active":"2024-04-16T16:22:46.478459+0000","last_became_peered":"2024-04-16T16:22:46.478459+0000","last_unstale":"2024-04-16T16:22:46.482785+0000","last_undegraded":"2024-04-16T16:22:46.482785+0000","last_fullsized":"2024-04-16T16:22:46.482785+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:07:30.192612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.448825+0000","last_change":"2024-04-16T16:22:45.467505+0000","last_active":"2024-04-16T16:22:46.448825+0000","last_peered":"2024-04-16T16:22:46.448825+0000","last_clean":"2024-04-16T16:22:46.448825+0000","last_became_active":"2024-04-16T16:22:45.465339+0000","last_became_peered":"2024-04-16T16:22:45.465339+0000","last_unstale":"2024-04-16T16:22:46.448825+0000","last_undegraded":"2024-04-16T16:22:46.448825+0000","last_fullsized":"2024-04-16T16:22:46.448825+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:54.078259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.475699+0000","last_change":"2024-04-16T16:22:46.465023+0000","last_active":"2024-04-16T16:22:46.475699+0000","last_peered":"2024-04-16T16:22:46.475699+0000","last_clean":"2024-04-16T16:22:46.475699+0000","last_became_active":"2024-04-16T16:22:46.464436+0000","last_became_peered":"2024-04-16T16:22:46.464436+0000","last_unstale":"2024-04-16T16:22:46.475699+0000","last_undegraded":"2024-04-16T16:22:46.475699+0000","last_fullsized":"2024-04-16T16:22:46.475699+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:05:07.007726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.454218+0000","last_change":"2024-04-16T16:22:45.461515+0000","last_active":"2024-04-16T16:22:46.454218+0000","last_peered":"2024-04-16T16:22:46.454218+0000","last_clean":"2024-04-16T16:22:46.454218+0000","last_became_active":"2024-04-16T16:22:45.460350+0000","last_became_peered":"2024-04-16T16:22:45.460350+0000","last_unstale":"2024-04-16T16:22:46.454218+0000","last_undegraded":"2024-04-16T16:22:46.454218+0000","last_fullsized":"2024-04-16T16:22:46.454218+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:34.292891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"144'4691","reported_seq":5388,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-16T16:22:46.445611+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:22:46.445611+0000","last_peered":"2024-04-16T16:22:46.445611+0000","last_clean":"2024-04-16T16:22:46.445611+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:22:46.445611+0000","last_undegraded":"2024-04-16T16:22:46.445611+0000","last_fullsized":"2024-04-16T16:22:46.445611+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":4691,"log_dups_size":0,"ondisk_log_size":4691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":387,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":387,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4691,"ondisk_log_size":4691,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37812,"kb_used_data":2468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334028,"statfs":{"total":96636764160,"available":96598044672,"internally_reserved":0,"allocated":2527232,"data_stored":1510791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574957,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1933312,"data_stored":920423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574957,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1933312,"data_stored":920423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574958,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1933312,"data_stored":920423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574956,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1933312,"data_stored":920423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574957,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37256,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334584,"statfs":{"total":96636764160,"available":96598614016,"internally_reserved":0,"allocated":1961984,"data_stored":933588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33072,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338768,"statfs":{"total":96636764160,"available":96602898432,"internally_reserved":0,"allocated":1933312,"data_stored":920423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574957,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37812,"kb_used_data":2468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334028,"statfs":{"total":96636764160,"available":96598044672,"internally_reserved":0,"allocated":2527232,"data_stored":1510791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:22:51.593 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-16T16:22:51.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-16T16:22:51.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.868+0000 7ff771000640 1 Processor -- start 2024-04-16T16:22:51.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.868+0000 7ff771000640 1 -- start start 2024-04-16T16:22:51.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.870+0000 7ff771000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff76c15b920 0x7ff76c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:51.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.870+0000 7ff771000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff76c0d0e80 0x7ff76c06ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:51.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.870+0000 7ff771000640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff76c0ccb70 con 0x7ff76c0afda0 2024-04-16T16:22:51.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.870+0000 7ff771000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff76c06e7a0 con 0x7ff76c15b920 2024-04-16T16:22:51.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.870+0000 7ff771000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff76c06f170 con 0x7ff76c0d0e80 2024-04-16T16:22:51.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.870+0000 7ff769e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff76c15b920 0x7ff76c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:51.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.870+0000 7ff769e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff76c15b920 0x7ff76c0b4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33116/0 (socket says 172.21.15.45:33116) 2024-04-16T16:22:51.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.870+0000 7ff769e00640 1 -- 172.21.15.45:0/1694131006 learned_addr learned my addr 172.21.15.45:0/1694131006 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:51.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.871+0000 7ff76a800640 1 --2- 172.21.15.45:0/1694131006 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff76c0d0e80 0x7ff76c06ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:51.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.871+0000 7ff769e00640 1 -- 172.21.15.45:0/1694131006 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff76c0d0e80 msgr2=0x7ff76c06ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:51.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.871+0000 7ff769e00640 1 --2- 172.21.15.45:0/1694131006 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff76c0d0e80 0x7ff76c06ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.871+0000 7ff769e00640 1 -- 172.21.15.45:0/1694131006 >> v1:172.21.15.45:6789/0 conn(0x7ff76c0afda0 legacy=0x7ff76c0d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:51.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.871+0000 7ff769e00640 1 -- 172.21.15.45:0/1694131006 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff76c0cce50 con 0x7ff76c15b920 2024-04-16T16:22:51.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.872+0000 7ff769e00640 1 --2- 172.21.15.45:0/1694131006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff76c15b920 0x7ff76c0b4e20 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009e50 tx=0x7ff75c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd53fa58b387151 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:51.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.872+0000 7ff768a00640 1 -- 172.21.15.45:0/1694131006 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75c02ec80 con 0x7ff76c15b920 2024-04-16T16:22:51.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.872+0000 7ff768a00640 1 -- 172.21.15.45:0/1694131006 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff75c02ee20 con 0x7ff76c15b920 2024-04-16T16:22:51.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.872+0000 7ff768a00640 1 -- 172.21.15.45:0/1694131006 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75c036610 con 0x7ff76c15b920 2024-04-16T16:22:51.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.873+0000 7ff771000640 1 -- 172.21.15.45:0/1694131006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff76c15b920 msgr2=0x7ff76c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:51.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.873+0000 7ff771000640 1 --2- 172.21.15.45:0/1694131006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff76c15b920 0x7ff76c0b4e20 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009e50 tx=0x7ff75c02e0e0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.873+0000 7ff771000640 1 -- 172.21.15.45:0/1694131006 shutdown_connections 2024-04-16T16:22:51.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.873+0000 7ff771000640 1 --2- 172.21.15.45:0/1694131006 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff76c0d0e80 0x7ff76c06ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.873+0000 7ff771000640 1 --2- 172.21.15.45:0/1694131006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff76c15b920 0x7ff76c0b4e20 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.873+0000 7ff771000640 1 -- 172.21.15.45:0/1694131006 >> 172.21.15.45:0/1694131006 conn(0x7ff76c0b3970 msgr2=0x7ff76c0b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:51.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.873+0000 7ff771000640 1 -- 172.21.15.45:0/1694131006 shutdown_connections 2024-04-16T16:22:51.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.873+0000 7ff771000640 1 -- 172.21.15.45:0/1694131006 wait complete. 2024-04-16T16:22:51.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.875+0000 7ff771000640 1 Processor -- start 2024-04-16T16:22:51.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.875+0000 7ff771000640 1 -- start start 2024-04-16T16:22:51.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.876+0000 7ff771000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff76c0afda0 0x7ff76c1f5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:51.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.876+0000 7ff771000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff76c0d0e80 0x7ff76c14b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:51.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.876+0000 7ff771000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff76c15b920 0x7ff76c157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:51.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.876+0000 7ff771000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff76c0ce760 con 0x7ff76c0afda0 2024-04-16T16:22:51.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.876+0000 7ff771000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff76c0ce5e0 con 0x7ff76c15b920 2024-04-16T16:22:51.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.876+0000 7ff771000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff76c0ce8e0 con 0x7ff76c0d0e80 2024-04-16T16:22:51.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.876+0000 7ff769e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff76c0afda0 0x7ff76c1f5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:51.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.876+0000 7ff769e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff76c0afda0 0x7ff76c1f5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33124/0 (socket says 172.21.15.45:33124) 2024-04-16T16:22:51.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.876+0000 7ff769e00640 1 -- 172.21.15.45:0/2352769858 learned_addr learned my addr 172.21.15.45:0/2352769858 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:51.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.877+0000 7ff769400640 1 --2- 172.21.15.45:0/2352769858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff76c0d0e80 0x7ff76c14b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:51.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.877+0000 7ff769e00640 1 -- 172.21.15.45:0/2352769858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff76c0d0e80 msgr2=0x7ff76c14b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:51.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.877+0000 7ff769e00640 1 --2- 172.21.15.45:0/2352769858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff76c0d0e80 0x7ff76c14b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.877+0000 7ff76a800640 1 --2- 172.21.15.45:0/2352769858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff76c15b920 0x7ff76c157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:51.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.877+0000 7ff769e00640 1 -- 172.21.15.45:0/2352769858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff76c15b920 msgr2=0x7ff76c157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:51.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.877+0000 7ff769e00640 1 --2- 172.21.15.45:0/2352769858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff76c15b920 0x7ff76c157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:51.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.877+0000 7ff769e00640 1 -- 172.21.15.45:0/2352769858 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff76c1f5e40 con 0x7ff76c0afda0 2024-04-16T16:22:51.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.877+0000 7ff769e00640 1 --2- 172.21.15.45:0/2352769858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff76c0afda0 0x7ff76c1f5600 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009f70 tx=0x7ff75c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:51.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.877+0000 7ff762000640 1 -- 172.21.15.45:0/2352769858 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75c009510 con 0x7ff76c0afda0 2024-04-16T16:22:51.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.878+0000 7ff762000640 1 -- 172.21.15.45:0/2352769858 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff75c00e120 con 0x7ff76c0afda0 2024-04-16T16:22:51.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.878+0000 7ff762000640 1 -- 172.21.15.45:0/2352769858 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75c040670 con 0x7ff76c0afda0 2024-04-16T16:22:51.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.878+0000 7ff771000640 1 -- 172.21.15.45:0/2352769858 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff76c1f60e0 con 0x7ff76c0afda0 2024-04-16T16:22:51.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.878+0000 7ff771000640 1 -- 172.21.15.45:0/2352769858 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff76c1f66a0 con 0x7ff76c0afda0 2024-04-16T16:22:51.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.879+0000 7ff771000640 1 -- 172.21.15.45:0/2352769858 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff76c075780 con 0x7ff76c0afda0 2024-04-16T16:22:51.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.880+0000 7ff762000640 1 -- 172.21.15.45:0/2352769858 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7ff75c047020 con 0x7ff76c0afda0 2024-04-16T16:22:51.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.881+0000 7ff762000640 1 --2- 172.21.15.45:0/2352769858 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ff74c06ca90 0x7ff74c06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:51.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.881+0000 7ff762000640 1 -- 172.21.15.45:0/2352769858 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7ff75c0adc20 con 0x7ff76c0afda0 2024-04-16T16:22:51.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.882+0000 7ff769400640 1 --2- 172.21.15.45:0/2352769858 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ff74c06ca90 0x7ff74c06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:51.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.883+0000 7ff769400640 1 --2- 172.21.15.45:0/2352769858 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ff74c06ca90 0x7ff74c06ef50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff76c14bdc0 tx=0x7ff758009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:51.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:51.885+0000 7ff762000640 1 -- 172.21.15.45:0/2352769858 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff75c07bfb0 con 0x7ff76c0afda0 2024-04-16T16:22:52.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.172+0000 7ff771000640 1 -- 172.21.15.45:0/2352769858 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7ff76c158b60 con 0x7ff76c0afda0 2024-04-16T16:22:52.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.177+0000 7ff762000640 1 -- 172.21.15.45:0/2352769858 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v27) ==== 190+0+147 (secure 0 0 0) 0x7ff75c07b8b0 con 0x7ff76c0afda0 2024-04-16T16:22:52.176 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-04-16T16:22:52.176 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBsox5mwI7pAxAAoa/Cls7BFetCxc/TzdQ9lQ== 2024-04-16T16:22:52.176 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2024-04-16T16:22:52.176 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2024-04-16T16:22:52.176 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-16T16:22:52.176 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2024-04-16T16:22:52.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.180+0000 7ff771000640 1 -- 172.21.15.45:0/2352769858 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ff74c06ca90 msgr2=0x7ff74c06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:52.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.180+0000 7ff771000640 1 --2- 172.21.15.45:0/2352769858 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ff74c06ca90 0x7ff74c06ef50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff76c14bdc0 tx=0x7ff758009040 comp rx=0 tx=0).stop 2024-04-16T16:22:52.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.180+0000 7ff771000640 1 -- 172.21.15.45:0/2352769858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff76c0afda0 msgr2=0x7ff76c1f5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:52.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.180+0000 7ff771000640 1 --2- 172.21.15.45:0/2352769858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff76c0afda0 0x7ff76c1f5600 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009f70 tx=0x7ff75c004ab0 comp rx=0 tx=0).stop 2024-04-16T16:22:52.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.180+0000 7ff771000640 1 -- 172.21.15.45:0/2352769858 shutdown_connections 2024-04-16T16:22:52.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.180+0000 7ff771000640 1 --2- 172.21.15.45:0/2352769858 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ff74c06ca90 0x7ff74c06ef50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:52.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.180+0000 7ff771000640 1 --2- 172.21.15.45:0/2352769858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff76c15b920 0x7ff76c157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:52.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.180+0000 7ff771000640 1 --2- 172.21.15.45:0/2352769858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff76c0d0e80 0x7ff76c14b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:52.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.180+0000 7ff771000640 1 --2- 172.21.15.45:0/2352769858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff76c0afda0 0x7ff76c1f5600 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:52.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.180+0000 7ff771000640 1 -- 172.21.15.45:0/2352769858 >> 172.21.15.45:0/2352769858 conn(0x7ff76c0b3970 msgr2=0x7ff76c0cef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:52.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.180+0000 7ff771000640 1 -- 172.21.15.45:0/2352769858 shutdown_connections 2024-04-16T16:22:52.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.180+0000 7ff771000640 1 -- 172.21.15.45:0/2352769858 wait complete. 2024-04-16T16:22:52.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:22:52.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.419+0000 7f0d68a00640 1 Processor -- start 2024-04-16T16:22:52.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.420+0000 7f0d68a00640 1 -- start start 2024-04-16T16:22:52.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.421+0000 7f0d68a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0d6415ea70 0x7f0d640cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:52.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.421+0000 7f0d68a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d640cf6d0 con 0x7f0d641462e0 2024-04-16T16:22:52.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.421+0000 7f0d68a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d641466c0 con 0x7f0d64072a90 2024-04-16T16:22:52.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.421+0000 7f0d68a00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0d640cf3f0 con 0x7f0d6415ea70 2024-04-16T16:22:52.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.422+0000 7f0d61e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0d641462e0 0x7f0d64072380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38452/0 (socket says 172.21.15.45:38452) 2024-04-16T16:22:52.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.422+0000 7f0d61e00640 1 -- 172.21.15.45:0/2334228148 learned_addr learned my addr 172.21.15.45:0/2334228148 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:52.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.422+0000 7f0d62800640 1 --2- 172.21.15.45:0/2334228148 >> v2:172.21.15.45:3300/0 conn(0x7f0d6415ea70 0x7f0d640cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:52.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.422+0000 7f0d62800640 1 -- 172.21.15.45:0/2334228148 >> v1:172.21.15.103:6789/0 conn(0x7f0d64072a90 legacy=0x7f0d640ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:52.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.422+0000 7f0d62800640 1 -- 172.21.15.45:0/2334228148 >> v1:172.21.15.45:6789/0 conn(0x7f0d641462e0 legacy=0x7f0d64072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:52.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.422+0000 7f0d62800640 1 -- 172.21.15.45:0/2334228148 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d6415cf80 con 0x7f0d6415ea70 2024-04-16T16:22:52.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.423+0000 7f0d62800640 1 --2- 172.21.15.45:0/2334228148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d6415ea70 0x7f0d640cedd0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0097a0 tx=0x7f0d5c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f654d1769bb7fe90 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:52.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.424+0000 7f0d61400640 1 -- 172.21.15.45:0/2334228148 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c035020 con 0x7f0d6415ea70 2024-04-16T16:22:52.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.424+0000 7f0d61400640 1 -- 172.21.15.45:0/2334228148 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d5c031830 con 0x7f0d6415ea70 2024-04-16T16:22:52.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.424+0000 7f0d61400640 1 -- 172.21.15.45:0/2334228148 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c031c00 con 0x7f0d6415ea70 2024-04-16T16:22:52.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.424+0000 7f0d68a00640 1 -- 172.21.15.45:0/2334228148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d6415ea70 msgr2=0x7f0d640cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:52.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.424+0000 7f0d68a00640 1 --2- 172.21.15.45:0/2334228148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d6415ea70 0x7f0d640cedd0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0097a0 tx=0x7f0d5c02b8d0 comp rx=0 tx=0).stop 2024-04-16T16:22:52.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.425+0000 7f0d68a00640 1 -- 172.21.15.45:0/2334228148 shutdown_connections 2024-04-16T16:22:52.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.425+0000 7f0d68a00640 1 --2- 172.21.15.45:0/2334228148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d6415ea70 0x7f0d640cedd0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:52.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.425+0000 7f0d68a00640 1 -- 172.21.15.45:0/2334228148 >> 172.21.15.45:0/2334228148 conn(0x7f0d640b2750 msgr2=0x7f0d640b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:52.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.425+0000 7f0d68a00640 1 -- 172.21.15.45:0/2334228148 shutdown_connections 2024-04-16T16:22:52.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.425+0000 7f0d68a00640 1 -- 172.21.15.45:0/2334228148 wait complete. 2024-04-16T16:22:52.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.426+0000 7f0d68a00640 1 Processor -- start 2024-04-16T16:22:52.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.427+0000 7f0d68a00640 1 -- start start 2024-04-16T16:22:52.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.427+0000 7f0d68a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d64072a90 0x7f0d641315e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:52.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.427+0000 7f0d68a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d641462e0 0x7f0d64160bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:52.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.427+0000 7f0d68a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d6415ea70 0x7f0d64161110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:52.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.427+0000 7f0d68a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0d641674a0 con 0x7f0d641462e0 2024-04-16T16:22:52.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.427+0000 7f0d68a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6415d540 con 0x7f0d6415ea70 2024-04-16T16:22:52.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.427+0000 7f0d68a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0d64167620 con 0x7f0d64072a90 2024-04-16T16:22:52.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.428+0000 7f0d59400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d641462e0 0x7f0d64160bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:52.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.428+0000 7f0d59400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d641462e0 0x7f0d64160bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33144/0 (socket says 172.21.15.45:33144) 2024-04-16T16:22:52.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.428+0000 7f0d59400640 1 -- 172.21.15.45:0/1099889310 learned_addr learned my addr 172.21.15.45:0/1099889310 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:52.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.428+0000 7f0d61e00640 1 --2- 172.21.15.45:0/1099889310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d64072a90 0x7f0d641315e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:52.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.428+0000 7f0d62800640 1 --2- 172.21.15.45:0/1099889310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d6415ea70 0x7f0d64161110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:52.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.428+0000 7f0d59400640 1 -- 172.21.15.45:0/1099889310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d64072a90 msgr2=0x7f0d641315e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:52.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.428+0000 7f0d59400640 1 --2- 172.21.15.45:0/1099889310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d64072a90 0x7f0d641315e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:52.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.428+0000 7f0d59400640 1 -- 172.21.15.45:0/1099889310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d6415ea70 msgr2=0x7f0d64161110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:52.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.428+0000 7f0d59400640 1 --2- 172.21.15.45:0/1099889310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d6415ea70 0x7f0d64161110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:52.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.428+0000 7f0d59400640 1 -- 172.21.15.45:0/1099889310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d641409c0 con 0x7f0d641462e0 2024-04-16T16:22:52.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.429+0000 7f0d59400640 1 --2- 172.21.15.45:0/1099889310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d641462e0 0x7f0d64160bd0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00c3c0 tx=0x7f0d4c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:52.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.429+0000 7f0d5aa00640 1 -- 172.21.15.45:0/1099889310 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c0052d0 con 0x7f0d641462e0 2024-04-16T16:22:52.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.429+0000 7f0d5aa00640 1 -- 172.21.15.45:0/1099889310 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d4c005490 con 0x7f0d641462e0 2024-04-16T16:22:52.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.429+0000 7f0d5aa00640 1 -- 172.21.15.45:0/1099889310 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c007260 con 0x7f0d641462e0 2024-04-16T16:22:52.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.429+0000 7f0d68a00640 1 -- 172.21.15.45:0/1099889310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d64140ca0 con 0x7f0d641462e0 2024-04-16T16:22:52.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.429+0000 7f0d68a00640 1 -- 172.21.15.45:0/1099889310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d64141260 con 0x7f0d641462e0 2024-04-16T16:22:52.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.430+0000 7f0d68a00640 1 -- 172.21.15.45:0/1099889310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d64141460 con 0x7f0d641462e0 2024-04-16T16:22:52.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.431+0000 7f0d5aa00640 1 -- 172.21.15.45:0/1099889310 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f0d4c007400 con 0x7f0d641462e0 2024-04-16T16:22:52.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.432+0000 7f0d5aa00640 1 --2- 172.21.15.45:0/1099889310 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0d4006ca40 0x7f0d4006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:52.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.432+0000 7f0d5aa00640 1 -- 172.21.15.45:0/1099889310 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f0d4c08b220 con 0x7f0d641462e0 2024-04-16T16:22:52.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.433+0000 7f0d61e00640 1 --2- 172.21.15.45:0/1099889310 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0d4006ca40 0x7f0d4006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:52.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.434+0000 7f0d61e00640 1 --2- 172.21.15.45:0/1099889310 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0d4006ca40 0x7f0d4006ef00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0d54005c00 tx=0x7f0d54009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:52.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.437+0000 7f0d5aa00640 1 -- 172.21.15.45:0/1099889310 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d64141460 con 0x7f0d641462e0 2024-04-16T16:22:52.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.682+0000 7f0d68a00640 1 -- 172.21.15.45:0/1099889310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d64075780 con 0x7f0d641462e0 2024-04-16T16:22:52.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.683+0000 7f0d5aa00640 1 -- 172.21.15.45:0/1099889310 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) ==== 95+0+5194 (secure 0 0 0) 0x7f0d4c0594a0 con 0x7f0d641462e0 2024-04-16T16:22:52.682 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:52.682 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":114,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6896,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2328390625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2328390625},{"type":"v1","addr":"172.21.15.103:6835","nonce":2328390625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":109},{"gid":7756,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/677052950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":677052950},{"type":"v1","addr":"172.21.15.45:6834","nonce":677052950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":108},{"gid":7765,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3875926287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3875926287},{"type":"v1","addr":"172.21.15.103:6837","nonce":3875926287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:22:45.927071+0000","modified":"2024-04-16T16:22:48.994607+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9840},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9840":{"gid":9840,"name":"c","rank":0,"incarnation":110,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3351492627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3351492627},{"type":"v1","addr":"172.21.15.45:6835","nonce":3351492627}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9840,"qdb_cluster":[9840]},"id":11}]} 2024-04-16T16:22:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 114 2024-04-16T16:22:52.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.686+0000 7f0d68a00640 1 -- 172.21.15.45:0/1099889310 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0d4006ca40 msgr2=0x7f0d4006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:52.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.686+0000 7f0d68a00640 1 --2- 172.21.15.45:0/1099889310 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0d4006ca40 0x7f0d4006ef00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0d54005c00 tx=0x7f0d54009040 comp rx=0 tx=0).stop 2024-04-16T16:22:52.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.686+0000 7f0d68a00640 1 -- 172.21.15.45:0/1099889310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d641462e0 msgr2=0x7f0d64160bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:52.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.686+0000 7f0d68a00640 1 --2- 172.21.15.45:0/1099889310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d641462e0 0x7f0d64160bd0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00c3c0 tx=0x7f0d4c00c880 comp rx=0 tx=0).stop 2024-04-16T16:22:52.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.686+0000 7f0d68a00640 1 -- 172.21.15.45:0/1099889310 shutdown_connections 2024-04-16T16:22:52.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.686+0000 7f0d68a00640 1 --2- 172.21.15.45:0/1099889310 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f0d4006ca40 0x7f0d4006ef00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:52.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.686+0000 7f0d68a00640 1 --2- 172.21.15.45:0/1099889310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d6415ea70 0x7f0d64161110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:52.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.686+0000 7f0d68a00640 1 --2- 172.21.15.45:0/1099889310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d641462e0 0x7f0d64160bd0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:52.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.686+0000 7f0d68a00640 1 --2- 172.21.15.45:0/1099889310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d64072a90 0x7f0d641315e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:52.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.686+0000 7f0d68a00640 1 -- 172.21.15.45:0/1099889310 >> 172.21.15.45:0/1099889310 conn(0x7f0d640b2750 msgr2=0x7f0d640af030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:52.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.686+0000 7f0d68a00640 1 -- 172.21.15.45:0/1099889310 shutdown_connections 2024-04-16T16:22:52.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:52.686+0000 7f0d68a00640 1 -- 172.21.15.45:0/1099889310 wait complete. 2024-04-16T16:22:52.806 INFO:tasks.cephfs_test_runner:test_metrics_mds_grow_and_shrink (tasks.cephfs.test_mds_metrics.TestMDSMetrics) ... ERROR 2024-04-16T16:22:52.806 INFO:tasks.cephfs_test_runner: 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner:ERROR: test_metrics_mds_grow_and_shrink (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/qa/tasks/cephfs/test_mds_metrics.py", line 21, in setUp 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner: super(TestMDSMetrics, self).setUp() 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/qa/tasks/cephfs/cephfs_test_case.py", line 180, in setUp 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons() 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/qa/tasks/cephfs/filesystem.py", line 1270, in wait_for_daemons 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner: if self.are_daemons_healthy(status=status, skip_max_mds_check=skip_max_mds_check): 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/qa/tasks/cephfs/filesystem.py", line 1075, in are_daemons_healthy 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner: mds.check_status() 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 43, in check_status 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner: return self.proc.poll() 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 211, in poll 2024-04-16T16:22:52.807 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-04-16T16:22:52.808 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-04-16T16:22:52.808 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-04-16T16:22:52.808 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c' 2024-04-16T16:22:52.808 INFO:tasks.cephfs_test_runner: 2024-04-16T16:22:52.808 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-16T16:22:52.808 INFO:tasks.cephfs_test_runner:Ran 8 tests in 525.577s 2024-04-16T16:22:52.808 INFO:tasks.cephfs_test_runner: 2024-04-16T16:22:52.808 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-04-16T16:22:52.808 INFO:tasks.cephfs_test_runner: 2024-04-16T16:22:52.808 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-16T16:22:52.808 INFO:tasks.cephfs_test_runner:ERROR: test_metrics_mds_grow_and_shrink (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2024-04-16T16:22:52.808 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-16T16:22:52.808 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-16T16:22:52.808 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/qa/tasks/cephfs/test_mds_metrics.py", line 21, in setUp 2024-04-16T16:22:52.808 INFO:tasks.cephfs_test_runner: super(TestMDSMetrics, self).setUp() 2024-04-16T16:22:52.809 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/qa/tasks/cephfs/cephfs_test_case.py", line 180, in setUp 2024-04-16T16:22:52.809 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons() 2024-04-16T16:22:52.809 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/qa/tasks/cephfs/filesystem.py", line 1270, in wait_for_daemons 2024-04-16T16:22:52.809 INFO:tasks.cephfs_test_runner: if self.are_daemons_healthy(status=status, skip_max_mds_check=skip_max_mds_check): 2024-04-16T16:22:52.809 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/qa/tasks/cephfs/filesystem.py", line 1075, in are_daemons_healthy 2024-04-16T16:22:52.809 INFO:tasks.cephfs_test_runner: mds.check_status() 2024-04-16T16:22:52.809 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 43, in check_status 2024-04-16T16:22:52.809 INFO:tasks.cephfs_test_runner: return self.proc.poll() 2024-04-16T16:22:52.809 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 211, in poll 2024-04-16T16:22:52.809 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-04-16T16:22:52.809 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-04-16T16:22:52.809 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-04-16T16:22:52.809 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c' 2024-04-16T16:22:52.809 INFO:tasks.cephfs_test_runner: 2024-04-16T16:22:52.810 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_metrics_mds_grow_and_shrink (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2024-04-16T16:22:53.006 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5cbbdc8794fd48b2ba7288cde5ff6964 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c3c87ed01bade92fb0ec360baa9bd9c82b8fde4f/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_metrics_mds_grow_and_shrink (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2024-04-16T16:22:53.007 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-16T16:22:53.016 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-16T16:22:53.034 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-16T16:22:53.034 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:53.034 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:53.100 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:22:53.100 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T16:22:53.100 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:53.155 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053546,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-16T16:22:53.155 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:22:53.155 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:53.216 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:22:53.216 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-16T16:22:53.216 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T16:22:53.269 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052774,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-16T16:22:53.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:53.270 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T16:22:53.335 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-16T16:22:53.335 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-16T16:22:53.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:53.336 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T16:22:53.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:53.440 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-16T16:22:53.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:53.530 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2024-04-16T16:22:53.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:53.683 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T16:22:53.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:53.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2024-04-16T16:22:53.862 INFO:teuthology.orchestra.run.smithi045.stderr:Cannot find device "ceph-brx" 2024-04-16T16:22:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T16:22:53.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:53.866 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-16T16:22:53.926 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-16T16:22:53.926 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-16T16:22:53.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:53.927 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T16:22:54.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:54.024 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-16T16:22:54.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:54.106 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-04-16T16:22:54.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:54.246 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-16T16:22:54.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T16:22:54.290 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-04-16T16:22:54.428 INFO:teuthology.orchestra.run.smithi103.stderr:Cannot find device "ceph-brx" 2024-04-16T16:22:54.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T16:22:54.432 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-16T16:22:54.442 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T16:22:54.442 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:22:54.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.678+0000 7f90bd000640 1 Processor -- start 2024-04-16T16:22:54.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.678+0000 7f90bd000640 1 -- start start 2024-04-16T16:22:54.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.680+0000 7f90bd000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f90b8146220 0x7f90b815ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:54.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.680+0000 7f90bd000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90b81608a0 0x7f90b8160d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:54.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.680+0000 7f90bd000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90b815d360 con 0x7f90b806f7a0 2024-04-16T16:22:54.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.680+0000 7f90bd000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f90b8160410 con 0x7f90b8146220 2024-04-16T16:22:54.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.680+0000 7f90bd000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f90b815d640 con 0x7f90b81608a0 2024-04-16T16:22:54.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.680+0000 7f90b7400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f90b8146220 0x7f90b815ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:54.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.680+0000 7f90b7400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f90b8146220 0x7f90b815ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47226/0 (socket says 172.21.15.45:47226) 2024-04-16T16:22:54.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.680+0000 7f90b7400640 1 -- 172.21.15.45:0/3217517440 learned_addr learned my addr 172.21.15.45:0/3217517440 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:54.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.681+0000 7f90b7e00640 1 --2- 172.21.15.45:0/3217517440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90b81608a0 0x7f90b8160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:54.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.681+0000 7f90b7400640 1 -- 172.21.15.45:0/3217517440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90b81608a0 msgr2=0x7f90b8160d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:54.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.681+0000 7f90b7400640 1 --2- 172.21.15.45:0/3217517440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90b81608a0 0x7f90b8160d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:54.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.681+0000 7f90b7400640 1 -- 172.21.15.45:0/3217517440 >> v1:172.21.15.103:6789/0 conn(0x7f90b806f7a0 legacy=0x7f90b806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:54.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.681+0000 7f90b7400640 1 -- 172.21.15.45:0/3217517440 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b815d920 con 0x7f90b8146220 2024-04-16T16:22:54.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.682+0000 7f90b7400640 1 --2- 172.21.15.45:0/3217517440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b8146220 0x7f90b815ccb0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009870 tx=0x7f90a802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7525684c03c626 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:54.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.682+0000 7f90b6000640 1 -- 172.21.15.45:0/3217517440 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8035020 con 0x7f90b8146220 2024-04-16T16:22:54.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.682+0000 7f90b6000640 1 -- 172.21.15.45:0/3217517440 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90a8031640 con 0x7f90b8146220 2024-04-16T16:22:54.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.682+0000 7f90b6000640 1 -- 172.21.15.45:0/3217517440 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a802d040 con 0x7f90b8146220 2024-04-16T16:22:54.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.683+0000 7f90bd000640 1 -- 172.21.15.45:0/3217517440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b8146220 msgr2=0x7f90b815ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:54.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.683+0000 7f90bd000640 1 --2- 172.21.15.45:0/3217517440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b8146220 0x7f90b815ccb0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009870 tx=0x7f90a802b8f0 comp rx=0 tx=0).stop 2024-04-16T16:22:54.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.683+0000 7f90bd000640 1 -- 172.21.15.45:0/3217517440 shutdown_connections 2024-04-16T16:22:54.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.683+0000 7f90bd000640 1 --2- 172.21.15.45:0/3217517440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90b81608a0 0x7f90b8160d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:54.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.683+0000 7f90bd000640 1 --2- 172.21.15.45:0/3217517440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b8146220 0x7f90b815ccb0 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:54.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.683+0000 7f90bd000640 1 -- 172.21.15.45:0/3217517440 >> 172.21.15.45:0/3217517440 conn(0x7f90b806b930 msgr2=0x7f90b806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:54.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.683+0000 7f90bd000640 1 -- 172.21.15.45:0/3217517440 shutdown_connections 2024-04-16T16:22:54.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.683+0000 7f90bd000640 1 -- 172.21.15.45:0/3217517440 wait complete. 2024-04-16T16:22:54.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.685+0000 7f90bd000640 1 Processor -- start 2024-04-16T16:22:54.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.685+0000 7f90bd000640 1 -- start start 2024-04-16T16:22:54.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.686+0000 7f90bd000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90b806f7a0 0x7f90b8139f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:54.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.686+0000 7f90bd000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b8146220 0x7f90b813c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:54.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.686+0000 7f90bd000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b81608a0 0x7f90b814c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:54.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.686+0000 7f90bd000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f90b806fc20 con 0x7f90b8146220 2024-04-16T16:22:54.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.686+0000 7f90bd000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f90b815dc00 con 0x7f90b81608a0 2024-04-16T16:22:54.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.686+0000 7f90bd000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f90b806fda0 con 0x7f90b806f7a0 2024-04-16T16:22:54.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.687+0000 7f90b6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b8146220 0x7f90b813c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:54.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.687+0000 7f90b6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b8146220 0x7f90b813c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47236/0 (socket says 172.21.15.45:47236) 2024-04-16T16:22:54.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.687+0000 7f90b6a00640 1 -- 172.21.15.45:0/2975782181 learned_addr learned my addr 172.21.15.45:0/2975782181 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:54.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.687+0000 7f90b7e00640 1 --2- 172.21.15.45:0/2975782181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b81608a0 0x7f90b814c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:54.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.687+0000 7f90b7400640 1 --2- 172.21.15.45:0/2975782181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90b806f7a0 0x7f90b8139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:54.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.687+0000 7f90b6a00640 1 -- 172.21.15.45:0/2975782181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90b806f7a0 msgr2=0x7f90b8139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:54.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.687+0000 7f90b6a00640 1 --2- 172.21.15.45:0/2975782181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90b806f7a0 0x7f90b8139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:54.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.687+0000 7f90b6a00640 1 -- 172.21.15.45:0/2975782181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b81608a0 msgr2=0x7f90b814c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:54.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.687+0000 7f90b6a00640 1 --2- 172.21.15.45:0/2975782181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b81608a0 0x7f90b814c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:54.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.687+0000 7f90b6a00640 1 -- 172.21.15.45:0/2975782181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b813d390 con 0x7f90b8146220 2024-04-16T16:22:54.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.687+0000 7f90b6a00640 1 --2- 172.21.15.45:0/2975782181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b8146220 0x7f90b813c0a0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f90a400c430 tx=0x7f90a400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:54.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.688+0000 7f90af400640 1 -- 172.21.15.45:0/2975782181 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a4011790 con 0x7f90b8146220 2024-04-16T16:22:54.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.688+0000 7f90af400640 1 -- 172.21.15.45:0/2975782181 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90a4017070 con 0x7f90b8146220 2024-04-16T16:22:54.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.688+0000 7f90af400640 1 -- 172.21.15.45:0/2975782181 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a4011bb0 con 0x7f90b8146220 2024-04-16T16:22:54.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.688+0000 7f90bd000640 1 -- 172.21.15.45:0/2975782181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b813caf0 con 0x7f90b8146220 2024-04-16T16:22:54.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.688+0000 7f90bd000640 1 -- 172.21.15.45:0/2975782181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b813d0b0 con 0x7f90b8146220 2024-04-16T16:22:54.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.689+0000 7f90bd000640 1 -- 172.21.15.45:0/2975782181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90b8075780 con 0x7f90b8146220 2024-04-16T16:22:54.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.690+0000 7f90af400640 1 -- 172.21.15.45:0/2975782181 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f90a401c460 con 0x7f90b8146220 2024-04-16T16:22:54.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.691+0000 7f90af400640 1 --2- 172.21.15.45:0/2975782181 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f909806ca10 0x7f909806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:54.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.692+0000 7f90af400640 1 -- 172.21.15.45:0/2975782181 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f90a408bc00 con 0x7f90b8146220 2024-04-16T16:22:54.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.692+0000 7f90b7400640 1 --2- 172.21.15.45:0/2975782181 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f909806ca10 0x7f909806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:54.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.693+0000 7f90b7400640 1 --2- 172.21.15.45:0/2975782181 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f909806ca10 0x7f909806eed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f90a8036000 tx=0x7f90a8008030 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:54.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.696+0000 7f90af400640 1 -- 172.21.15.45:0/2975782181 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f90a4059e80 con 0x7f90b8146220 2024-04-16T16:22:54.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.897+0000 7f90bd000640 1 -- 172.21.15.45:0/2975782181 --> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f90b80cff60 con 0x7f909806ca10 2024-04-16T16:22:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.917+0000 7f90af400640 1 -- 172.21.15.45:0/2975782181 <== mgr.7591 v2:172.21.15.103:6832/1129061436 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252497 (secure 0 0 0) 0x7f90b80cff60 con 0x7f909806ca10 2024-04-16T16:22:54.919 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:54.920 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:22:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.922+0000 7f90bd000640 1 -- 172.21.15.45:0/2975782181 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f909806ca10 msgr2=0x7f909806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.922+0000 7f90bd000640 1 --2- 172.21.15.45:0/2975782181 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f909806ca10 0x7f909806eed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f90a8036000 tx=0x7f90a8008030 comp rx=0 tx=0).stop 2024-04-16T16:22:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.923+0000 7f90bd000640 1 -- 172.21.15.45:0/2975782181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b8146220 msgr2=0x7f90b813c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.923+0000 7f90bd000640 1 --2- 172.21.15.45:0/2975782181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b8146220 0x7f90b813c0a0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f90a400c430 tx=0x7f90a400c8f0 comp rx=0 tx=0).stop 2024-04-16T16:22:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.923+0000 7f90bd000640 1 -- 172.21.15.45:0/2975782181 shutdown_connections 2024-04-16T16:22:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.923+0000 7f90bd000640 1 --2- 172.21.15.45:0/2975782181 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f909806ca10 0x7f909806eed0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.923+0000 7f90bd000640 1 --2- 172.21.15.45:0/2975782181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b81608a0 0x7f90b814c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.923+0000 7f90bd000640 1 --2- 172.21.15.45:0/2975782181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b8146220 0x7f90b813c0a0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.923+0000 7f90bd000640 1 --2- 172.21.15.45:0/2975782181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90b806f7a0 0x7f90b8139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.923+0000 7f90bd000640 1 -- 172.21.15.45:0/2975782181 >> 172.21.15.45:0/2975782181 conn(0x7f90b806b930 msgr2=0x7f90b80cf490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.923+0000 7f90bd000640 1 -- 172.21.15.45:0/2975782181 shutdown_connections 2024-04-16T16:22:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:54.923+0000 7f90bd000640 1 -- 172.21.15.45:0/2975782181 wait complete. 2024-04-16T16:22:54.940 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-04-16T16:22:53.542715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":431,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4729,"ondisk_log_size":4729,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":286776,"kb_used_data":16520,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754687944,"statfs":{"total":773094113280,"available":772800454656,"internally_reserved":0,"allocated":16916480,"data_stored":8653540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":276614704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002957"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461972+0000","last_change":"2024-04-16T16:22:46.488567+0000","last_active":"2024-04-16T16:22:47.461972+0000","last_peered":"2024-04-16T16:22:47.461972+0000","last_clean":"2024-04-16T16:22:47.461972+0000","last_became_active":"2024-04-16T16:22:46.485690+0000","last_became_peered":"2024-04-16T16:22:46.485690+0000","last_unstale":"2024-04-16T16:22:47.461972+0000","last_undegraded":"2024-04-16T16:22:47.461972+0000","last_fullsized":"2024-04-16T16:22:47.461972+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:50:29.718907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426342+0000","last_change":"2024-04-16T16:22:46.479620+0000","last_active":"2024-04-16T16:22:48.426342+0000","last_peered":"2024-04-16T16:22:48.426342+0000","last_clean":"2024-04-16T16:22:48.426342+0000","last_became_active":"2024-04-16T16:22:46.477410+0000","last_became_peered":"2024-04-16T16:22:46.477410+0000","last_unstale":"2024-04-16T16:22:48.426342+0000","last_undegraded":"2024-04-16T16:22:48.426342+0000","last_fullsized":"2024-04-16T16:22:48.426342+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:35.010598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624440+0000","last_change":"2024-04-16T16:22:46.480944+0000","last_active":"2024-04-16T16:22:48.624440+0000","last_peered":"2024-04-16T16:22:48.624440+0000","last_clean":"2024-04-16T16:22:48.624440+0000","last_became_active":"2024-04-16T16:22:46.480405+0000","last_became_peered":"2024-04-16T16:22:46.480405+0000","last_unstale":"2024-04-16T16:22:48.624440+0000","last_undegraded":"2024-04-16T16:22:48.624440+0000","last_fullsized":"2024-04-16T16:22:48.624440+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:35.308909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077640+0000","last_change":"2024-04-16T16:22:46.477832+0000","last_active":"2024-04-16T16:22:48.077640+0000","last_peered":"2024-04-16T16:22:48.077640+0000","last_clean":"2024-04-16T16:22:48.077640+0000","last_became_active":"2024-04-16T16:22:46.477516+0000","last_became_peered":"2024-04-16T16:22:46.477516+0000","last_unstale":"2024-04-16T16:22:48.077640+0000","last_undegraded":"2024-04-16T16:22:48.077640+0000","last_fullsized":"2024-04-16T16:22:48.077640+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:01.883490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426062+0000","last_change":"2024-04-16T16:22:46.483466+0000","last_active":"2024-04-16T16:22:48.426062+0000","last_peered":"2024-04-16T16:22:48.426062+0000","last_clean":"2024-04-16T16:22:48.426062+0000","last_became_active":"2024-04-16T16:22:46.481033+0000","last_became_peered":"2024-04-16T16:22:46.481033+0000","last_unstale":"2024-04-16T16:22:48.426062+0000","last_undegraded":"2024-04-16T16:22:48.426062+0000","last_fullsized":"2024-04-16T16:22:48.426062+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:36:30.108257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710150+0000","last_change":"2024-04-16T16:22:46.463298+0000","last_active":"2024-04-16T16:22:48.710150+0000","last_peered":"2024-04-16T16:22:48.710150+0000","last_clean":"2024-04-16T16:22:48.710150+0000","last_became_active":"2024-04-16T16:22:46.462017+0000","last_became_peered":"2024-04-16T16:22:46.462017+0000","last_unstale":"2024-04-16T16:22:48.710150+0000","last_undegraded":"2024-04-16T16:22:48.710150+0000","last_fullsized":"2024-04-16T16:22:48.710150+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:30:36.112336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635373+0000","last_change":"2024-04-16T16:22:46.462786+0000","last_active":"2024-04-16T16:22:48.635373+0000","last_peered":"2024-04-16T16:22:48.635373+0000","last_clean":"2024-04-16T16:22:48.635373+0000","last_became_active":"2024-04-16T16:22:46.461856+0000","last_became_peered":"2024-04-16T16:22:46.461856+0000","last_unstale":"2024-04-16T16:22:48.635373+0000","last_undegraded":"2024-04-16T16:22:48.635373+0000","last_fullsized":"2024-04-16T16:22:48.635373+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:29.789672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426662+0000","last_change":"2024-04-16T16:22:46.486917+0000","last_active":"2024-04-16T16:22:48.426662+0000","last_peered":"2024-04-16T16:22:48.426662+0000","last_clean":"2024-04-16T16:22:48.426662+0000","last_became_active":"2024-04-16T16:22:46.485280+0000","last_became_peered":"2024-04-16T16:22:46.485280+0000","last_unstale":"2024-04-16T16:22:48.426662+0000","last_undegraded":"2024-04-16T16:22:48.426662+0000","last_fullsized":"2024-04-16T16:22:48.426662+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:14:27.242053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.427106+0000","last_change":"2024-04-16T16:22:46.476513+0000","last_active":"2024-04-16T16:22:48.427106+0000","last_peered":"2024-04-16T16:22:48.427106+0000","last_clean":"2024-04-16T16:22:48.427106+0000","last_became_active":"2024-04-16T16:22:46.476125+0000","last_became_peered":"2024-04-16T16:22:46.476125+0000","last_unstale":"2024-04-16T16:22:48.427106+0000","last_undegraded":"2024-04-16T16:22:48.427106+0000","last_fullsized":"2024-04-16T16:22:48.427106+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:28.349945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426898+0000","last_change":"2024-04-16T16:22:46.483287+0000","last_active":"2024-04-16T16:22:48.426898+0000","last_peered":"2024-04-16T16:22:48.426898+0000","last_clean":"2024-04-16T16:22:48.426898+0000","last_became_active":"2024-04-16T16:22:46.482975+0000","last_became_peered":"2024-04-16T16:22:46.482975+0000","last_unstale":"2024-04-16T16:22:48.426898+0000","last_undegraded":"2024-04-16T16:22:48.426898+0000","last_fullsized":"2024-04-16T16:22:48.426898+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:21.784358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426384+0000","last_change":"2024-04-16T16:22:46.481301+0000","last_active":"2024-04-16T16:22:48.426384+0000","last_peered":"2024-04-16T16:22:48.426384+0000","last_clean":"2024-04-16T16:22:48.426384+0000","last_became_active":"2024-04-16T16:22:46.481000+0000","last_became_peered":"2024-04-16T16:22:46.481000+0000","last_unstale":"2024-04-16T16:22:48.426384+0000","last_undegraded":"2024-04-16T16:22:48.426384+0000","last_fullsized":"2024-04-16T16:22:48.426384+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:01.883490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426458+0000","last_change":"2024-04-16T16:22:46.476179+0000","last_active":"2024-04-16T16:22:48.426458+0000","last_peered":"2024-04-16T16:22:48.426458+0000","last_clean":"2024-04-16T16:22:48.426458+0000","last_became_active":"2024-04-16T16:22:46.475898+0000","last_became_peered":"2024-04-16T16:22:46.475898+0000","last_unstale":"2024-04-16T16:22:48.426458+0000","last_undegraded":"2024-04-16T16:22:48.426458+0000","last_fullsized":"2024-04-16T16:22:48.426458+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:37.006661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624317+0000","last_change":"2024-04-16T16:22:46.482186+0000","last_active":"2024-04-16T16:22:48.624317+0000","last_peered":"2024-04-16T16:22:48.624317+0000","last_clean":"2024-04-16T16:22:48.624317+0000","last_became_active":"2024-04-16T16:22:46.481776+0000","last_became_peered":"2024-04-16T16:22:46.481776+0000","last_unstale":"2024-04-16T16:22:48.624317+0000","last_undegraded":"2024-04-16T16:22:48.624317+0000","last_fullsized":"2024-04-16T16:22:48.624317+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:32:25.664832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623895+0000","last_change":"2024-04-16T16:22:46.477861+0000","last_active":"2024-04-16T16:22:48.623895+0000","last_peered":"2024-04-16T16:22:48.623895+0000","last_clean":"2024-04-16T16:22:48.623895+0000","last_became_active":"2024-04-16T16:22:46.476486+0000","last_became_peered":"2024-04-16T16:22:46.476486+0000","last_unstale":"2024-04-16T16:22:48.623895+0000","last_undegraded":"2024-04-16T16:22:48.623895+0000","last_fullsized":"2024-04-16T16:22:48.623895+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:08.580218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460332+0000","last_change":"2024-04-16T16:22:46.468319+0000","last_active":"2024-04-16T16:22:47.460332+0000","last_peered":"2024-04-16T16:22:47.460332+0000","last_clean":"2024-04-16T16:22:47.460332+0000","last_became_active":"2024-04-16T16:22:46.463117+0000","last_became_peered":"2024-04-16T16:22:46.463117+0000","last_unstale":"2024-04-16T16:22:47.460332+0000","last_undegraded":"2024-04-16T16:22:47.460332+0000","last_fullsized":"2024-04-16T16:22:47.460332+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:12:58.505334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635241+0000","last_change":"2024-04-16T16:22:46.467824+0000","last_active":"2024-04-16T16:22:48.635241+0000","last_peered":"2024-04-16T16:22:48.635241+0000","last_clean":"2024-04-16T16:22:48.635241+0000","last_became_active":"2024-04-16T16:22:46.467274+0000","last_became_peered":"2024-04-16T16:22:46.467274+0000","last_unstale":"2024-04-16T16:22:48.635241+0000","last_undegraded":"2024-04-16T16:22:48.635241+0000","last_fullsized":"2024-04-16T16:22:48.635241+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:50:29.718907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710486+0000","last_change":"2024-04-16T16:22:46.475037+0000","last_active":"2024-04-16T16:22:48.710486+0000","last_peered":"2024-04-16T16:22:48.710486+0000","last_clean":"2024-04-16T16:22:48.710486+0000","last_became_active":"2024-04-16T16:22:46.474757+0000","last_became_peered":"2024-04-16T16:22:46.474757+0000","last_unstale":"2024-04-16T16:22:48.710486+0000","last_undegraded":"2024-04-16T16:22:48.710486+0000","last_fullsized":"2024-04-16T16:22:48.710486+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:31:51.916998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457334+0000","last_change":"2024-04-16T16:22:46.463139+0000","last_active":"2024-04-16T16:22:47.457334+0000","last_peered":"2024-04-16T16:22:47.457334+0000","last_clean":"2024-04-16T16:22:47.457334+0000","last_became_active":"2024-04-16T16:22:46.462857+0000","last_became_peered":"2024-04-16T16:22:46.462857+0000","last_unstale":"2024-04-16T16:22:47.457334+0000","last_undegraded":"2024-04-16T16:22:47.457334+0000","last_fullsized":"2024-04-16T16:22:47.457334+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:43:19.877365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710482+0000","last_change":"2024-04-16T16:22:46.457033+0000","last_active":"2024-04-16T16:22:48.710482+0000","last_peered":"2024-04-16T16:22:48.710482+0000","last_clean":"2024-04-16T16:22:48.710482+0000","last_became_active":"2024-04-16T16:22:46.456756+0000","last_became_peered":"2024-04-16T16:22:46.456756+0000","last_unstale":"2024-04-16T16:22:48.710482+0000","last_undegraded":"2024-04-16T16:22:48.710482+0000","last_fullsized":"2024-04-16T16:22:48.710482+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:18:13.904540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624366+0000","last_change":"2024-04-16T16:22:46.482119+0000","last_active":"2024-04-16T16:22:48.624366+0000","last_peered":"2024-04-16T16:22:48.624366+0000","last_clean":"2024-04-16T16:22:48.624366+0000","last_became_active":"2024-04-16T16:22:46.479857+0000","last_became_peered":"2024-04-16T16:22:46.479857+0000","last_unstale":"2024-04-16T16:22:48.624366+0000","last_undegraded":"2024-04-16T16:22:48.624366+0000","last_fullsized":"2024-04-16T16:22:48.624366+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:05:36.568682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710264+0000","last_change":"2024-04-16T16:22:46.479160+0000","last_active":"2024-04-16T16:22:48.710264+0000","last_peered":"2024-04-16T16:22:48.710264+0000","last_clean":"2024-04-16T16:22:48.710264+0000","last_became_active":"2024-04-16T16:22:46.477081+0000","last_became_peered":"2024-04-16T16:22:46.477081+0000","last_unstale":"2024-04-16T16:22:48.710264+0000","last_undegraded":"2024-04-16T16:22:48.710264+0000","last_fullsized":"2024-04-16T16:22:48.710264+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:31:11.372535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426469+0000","last_change":"2024-04-16T16:22:46.473896+0000","last_active":"2024-04-16T16:22:48.426469+0000","last_peered":"2024-04-16T16:22:48.426469+0000","last_clean":"2024-04-16T16:22:48.426469+0000","last_became_active":"2024-04-16T16:22:46.473621+0000","last_became_peered":"2024-04-16T16:22:46.473621+0000","last_unstale":"2024-04-16T16:22:48.426469+0000","last_undegraded":"2024-04-16T16:22:48.426469+0000","last_fullsized":"2024-04-16T16:22:48.426469+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:26.785078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710026+0000","last_change":"2024-04-16T16:22:46.475807+0000","last_active":"2024-04-16T16:22:48.710026+0000","last_peered":"2024-04-16T16:22:48.710026+0000","last_clean":"2024-04-16T16:22:48.710026+0000","last_became_active":"2024-04-16T16:22:46.473183+0000","last_became_peered":"2024-04-16T16:22:46.473183+0000","last_unstale":"2024-04-16T16:22:48.710026+0000","last_undegraded":"2024-04-16T16:22:48.710026+0000","last_fullsized":"2024-04-16T16:22:48.710026+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:26.889339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426404+0000","last_change":"2024-04-16T16:22:46.477670+0000","last_active":"2024-04-16T16:22:48.426404+0000","last_peered":"2024-04-16T16:22:48.426404+0000","last_clean":"2024-04-16T16:22:48.426404+0000","last_became_active":"2024-04-16T16:22:46.477303+0000","last_became_peered":"2024-04-16T16:22:46.477303+0000","last_unstale":"2024-04-16T16:22:48.426404+0000","last_undegraded":"2024-04-16T16:22:48.426404+0000","last_fullsized":"2024-04-16T16:22:48.426404+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:02:37.502947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.625109+0000","last_change":"2024-04-16T16:22:46.481055+0000","last_active":"2024-04-16T16:22:48.625109+0000","last_peered":"2024-04-16T16:22:48.625109+0000","last_clean":"2024-04-16T16:22:48.625109+0000","last_became_active":"2024-04-16T16:22:46.480641+0000","last_became_peered":"2024-04-16T16:22:46.480641+0000","last_unstale":"2024-04-16T16:22:48.625109+0000","last_undegraded":"2024-04-16T16:22:48.625109+0000","last_fullsized":"2024-04-16T16:22:48.625109+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:53:06.786966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635707+0000","last_change":"2024-04-16T16:22:46.467282+0000","last_active":"2024-04-16T16:22:48.635707+0000","last_peered":"2024-04-16T16:22:48.635707+0000","last_clean":"2024-04-16T16:22:48.635707+0000","last_became_active":"2024-04-16T16:22:46.466756+0000","last_became_peered":"2024-04-16T16:22:46.466756+0000","last_unstale":"2024-04-16T16:22:48.635707+0000","last_undegraded":"2024-04-16T16:22:48.635707+0000","last_fullsized":"2024-04-16T16:22:48.635707+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:18.072160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624687+0000","last_change":"2024-04-16T16:22:46.484066+0000","last_active":"2024-04-16T16:22:48.624687+0000","last_peered":"2024-04-16T16:22:48.624687+0000","last_clean":"2024-04-16T16:22:48.624687+0000","last_became_active":"2024-04-16T16:22:46.483654+0000","last_became_peered":"2024-04-16T16:22:46.483654+0000","last_unstale":"2024-04-16T16:22:48.624687+0000","last_undegraded":"2024-04-16T16:22:48.624687+0000","last_fullsized":"2024-04-16T16:22:48.624687+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:44:40.829325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077771+0000","last_change":"2024-04-16T16:22:46.484111+0000","last_active":"2024-04-16T16:22:48.077771+0000","last_peered":"2024-04-16T16:22:48.077771+0000","last_clean":"2024-04-16T16:22:48.077771+0000","last_became_active":"2024-04-16T16:22:46.483750+0000","last_became_peered":"2024-04-16T16:22:46.483750+0000","last_unstale":"2024-04-16T16:22:48.077771+0000","last_undegraded":"2024-04-16T16:22:48.077771+0000","last_fullsized":"2024-04-16T16:22:48.077771+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:02:37.502947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.625014+0000","last_change":"2024-04-16T16:22:46.483339+0000","last_active":"2024-04-16T16:22:48.625014+0000","last_peered":"2024-04-16T16:22:48.625014+0000","last_clean":"2024-04-16T16:22:48.625014+0000","last_became_active":"2024-04-16T16:22:46.481358+0000","last_became_peered":"2024-04-16T16:22:46.481358+0000","last_unstale":"2024-04-16T16:22:48.625014+0000","last_undegraded":"2024-04-16T16:22:48.625014+0000","last_fullsized":"2024-04-16T16:22:48.625014+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:13.979255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461048+0000","last_change":"2024-04-16T16:22:46.464311+0000","last_active":"2024-04-16T16:22:47.461048+0000","last_peered":"2024-04-16T16:22:47.461048+0000","last_clean":"2024-04-16T16:22:47.461048+0000","last_became_active":"2024-04-16T16:22:46.463536+0000","last_became_peered":"2024-04-16T16:22:46.463536+0000","last_unstale":"2024-04-16T16:22:47.461048+0000","last_undegraded":"2024-04-16T16:22:47.461048+0000","last_fullsized":"2024-04-16T16:22:47.461048+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:36:30.108257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460440+0000","last_change":"2024-04-16T16:22:46.460441+0000","last_active":"2024-04-16T16:22:47.460440+0000","last_peered":"2024-04-16T16:22:47.460440+0000","last_clean":"2024-04-16T16:22:47.460440+0000","last_became_active":"2024-04-16T16:22:46.458962+0000","last_became_peered":"2024-04-16T16:22:46.458962+0000","last_unstale":"2024-04-16T16:22:47.460440+0000","last_undegraded":"2024-04-16T16:22:47.460440+0000","last_fullsized":"2024-04-16T16:22:47.460440+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:35.010598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.456651+0000","last_change":"2024-04-16T16:22:46.468514+0000","last_active":"2024-04-16T16:22:47.456651+0000","last_peered":"2024-04-16T16:22:47.456651+0000","last_clean":"2024-04-16T16:22:47.456651+0000","last_became_active":"2024-04-16T16:22:46.464851+0000","last_became_peered":"2024-04-16T16:22:46.464851+0000","last_unstale":"2024-04-16T16:22:47.456651+0000","last_undegraded":"2024-04-16T16:22:47.456651+0000","last_fullsized":"2024-04-16T16:22:47.456651+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:59:47.867717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635932+0000","last_change":"2024-04-16T16:22:46.467039+0000","last_active":"2024-04-16T16:22:48.635932+0000","last_peered":"2024-04-16T16:22:48.635932+0000","last_clean":"2024-04-16T16:22:48.635932+0000","last_became_active":"2024-04-16T16:22:46.466366+0000","last_became_peered":"2024-04-16T16:22:46.466366+0000","last_unstale":"2024-04-16T16:22:48.635932+0000","last_undegraded":"2024-04-16T16:22:48.635932+0000","last_fullsized":"2024-04-16T16:22:48.635932+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:54:26.843618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"155'4","reported_seq":18,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461935+0000","last_change":"2024-04-16T16:22:45.470039+0000","last_active":"2024-04-16T16:22:47.461935+0000","last_peered":"2024-04-16T16:22:47.461935+0000","last_clean":"2024-04-16T16:22:47.461935+0000","last_became_active":"2024-04-16T16:22:45.467695+0000","last_became_peered":"2024-04-16T16:22:45.467695+0000","last_unstale":"2024-04-16T16:22:47.461935+0000","last_undegraded":"2024-04-16T16:22:47.461935+0000","last_fullsized":"2024-04-16T16:22:47.461935+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:21.824987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463917+0000","last_change":"2024-04-16T16:22:46.466378+0000","last_active":"2024-04-16T16:22:47.463917+0000","last_peered":"2024-04-16T16:22:47.463917+0000","last_clean":"2024-04-16T16:22:47.463917+0000","last_became_active":"2024-04-16T16:22:46.462099+0000","last_became_peered":"2024-04-16T16:22:46.462099+0000","last_unstale":"2024-04-16T16:22:47.463917+0000","last_undegraded":"2024-04-16T16:22:47.463917+0000","last_fullsized":"2024-04-16T16:22:47.463917+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:02:37.502947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463627+0000","last_change":"2024-04-16T16:22:45.464685+0000","last_active":"2024-04-16T16:22:47.463627+0000","last_peered":"2024-04-16T16:22:47.463627+0000","last_clean":"2024-04-16T16:22:47.463627+0000","last_became_active":"2024-04-16T16:22:45.464379+0000","last_became_peered":"2024-04-16T16:22:45.464379+0000","last_unstale":"2024-04-16T16:22:47.463627+0000","last_undegraded":"2024-04-16T16:22:47.463627+0000","last_fullsized":"2024-04-16T16:22:47.463627+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:05:35.552664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710111+0000","last_change":"2024-04-16T16:22:46.456302+0000","last_active":"2024-04-16T16:22:48.710111+0000","last_peered":"2024-04-16T16:22:48.710111+0000","last_clean":"2024-04-16T16:22:48.710111+0000","last_became_active":"2024-04-16T16:22:46.454610+0000","last_became_peered":"2024-04-16T16:22:46.454610+0000","last_unstale":"2024-04-16T16:22:48.710111+0000","last_undegraded":"2024-04-16T16:22:48.710111+0000","last_fullsized":"2024-04-16T16:22:48.710111+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:56:12.143605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077380+0000","last_change":"2024-04-16T16:22:45.469877+0000","last_active":"2024-04-16T16:22:48.077380+0000","last_peered":"2024-04-16T16:22:48.077380+0000","last_clean":"2024-04-16T16:22:48.077380+0000","last_became_active":"2024-04-16T16:22:45.469537+0000","last_became_peered":"2024-04-16T16:22:45.469537+0000","last_unstale":"2024-04-16T16:22:48.077380+0000","last_undegraded":"2024-04-16T16:22:48.077380+0000","last_fullsized":"2024-04-16T16:22:48.077380+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:54.078259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624791+0000","last_change":"2024-04-16T16:22:46.480073+0000","last_active":"2024-04-16T16:22:48.624791+0000","last_peered":"2024-04-16T16:22:48.624791+0000","last_clean":"2024-04-16T16:22:48.624791+0000","last_became_active":"2024-04-16T16:22:46.479206+0000","last_became_peered":"2024-04-16T16:22:46.479206+0000","last_unstale":"2024-04-16T16:22:48.624791+0000","last_undegraded":"2024-04-16T16:22:48.624791+0000","last_fullsized":"2024-04-16T16:22:48.624791+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:27.266104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.078056+0000","last_change":"2024-04-16T16:22:45.462903+0000","last_active":"2024-04-16T16:22:48.078056+0000","last_peered":"2024-04-16T16:22:48.078056+0000","last_clean":"2024-04-16T16:22:48.078056+0000","last_became_active":"2024-04-16T16:22:45.462170+0000","last_became_peered":"2024-04-16T16:22:45.462170+0000","last_unstale":"2024-04-16T16:22:48.078056+0000","last_undegraded":"2024-04-16T16:22:48.078056+0000","last_fullsized":"2024-04-16T16:22:48.078056+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:07.564200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635999+0000","last_change":"2024-04-16T16:22:46.460202+0000","last_active":"2024-04-16T16:22:48.635999+0000","last_peered":"2024-04-16T16:22:48.635999+0000","last_clean":"2024-04-16T16:22:48.635999+0000","last_became_active":"2024-04-16T16:22:46.459876+0000","last_became_peered":"2024-04-16T16:22:46.459876+0000","last_unstale":"2024-04-16T16:22:48.635999+0000","last_undegraded":"2024-04-16T16:22:48.635999+0000","last_fullsized":"2024-04-16T16:22:48.635999+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:28:07.667824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624602+0000","last_change":"2024-04-16T16:22:45.472425+0000","last_active":"2024-04-16T16:22:48.624602+0000","last_peered":"2024-04-16T16:22:48.624602+0000","last_clean":"2024-04-16T16:22:48.624602+0000","last_became_active":"2024-04-16T16:22:45.472020+0000","last_became_peered":"2024-04-16T16:22:45.472020+0000","last_unstale":"2024-04-16T16:22:48.624602+0000","last_undegraded":"2024-04-16T16:22:48.624602+0000","last_fullsized":"2024-04-16T16:22:48.624602+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:48:51.793606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077212+0000","last_change":"2024-04-16T16:22:46.470665+0000","last_active":"2024-04-16T16:22:48.077212+0000","last_peered":"2024-04-16T16:22:48.077212+0000","last_clean":"2024-04-16T16:22:48.077212+0000","last_became_active":"2024-04-16T16:22:46.467898+0000","last_became_peered":"2024-04-16T16:22:46.467898+0000","last_unstale":"2024-04-16T16:22:48.077212+0000","last_undegraded":"2024-04-16T16:22:48.077212+0000","last_fullsized":"2024-04-16T16:22:48.077212+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:35.010598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077383+0000","last_change":"2024-04-16T16:22:45.462429+0000","last_active":"2024-04-16T16:22:48.077383+0000","last_peered":"2024-04-16T16:22:48.077383+0000","last_clean":"2024-04-16T16:22:48.077383+0000","last_became_active":"2024-04-16T16:22:45.461674+0000","last_became_peered":"2024-04-16T16:22:45.461674+0000","last_unstale":"2024-04-16T16:22:48.077383+0000","last_undegraded":"2024-04-16T16:22:48.077383+0000","last_fullsized":"2024-04-16T16:22:48.077383+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:29.386319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.076898+0000","last_change":"2024-04-16T16:22:46.465672+0000","last_active":"2024-04-16T16:22:48.076898+0000","last_peered":"2024-04-16T16:22:48.076898+0000","last_clean":"2024-04-16T16:22:48.076898+0000","last_became_active":"2024-04-16T16:22:46.465329+0000","last_became_peered":"2024-04-16T16:22:46.465329+0000","last_unstale":"2024-04-16T16:22:48.076898+0000","last_undegraded":"2024-04-16T16:22:48.076898+0000","last_fullsized":"2024-04-16T16:22:48.076898+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:36:42.468881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.425887+0000","last_change":"2024-04-16T16:22:45.466868+0000","last_active":"2024-04-16T16:22:48.425887+0000","last_peered":"2024-04-16T16:22:48.425887+0000","last_clean":"2024-04-16T16:22:48.425887+0000","last_became_active":"2024-04-16T16:22:45.466591+0000","last_became_peered":"2024-04-16T16:22:45.466591+0000","last_unstale":"2024-04-16T16:22:48.425887+0000","last_undegraded":"2024-04-16T16:22:48.425887+0000","last_fullsized":"2024-04-16T16:22:48.425887+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:25.557203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710073+0000","last_change":"2024-04-16T16:22:46.460701+0000","last_active":"2024-04-16T16:22:48.710073+0000","last_peered":"2024-04-16T16:22:48.710073+0000","last_clean":"2024-04-16T16:22:48.710073+0000","last_became_active":"2024-04-16T16:22:46.460420+0000","last_became_peered":"2024-04-16T16:22:46.460420+0000","last_unstale":"2024-04-16T16:22:48.710073+0000","last_undegraded":"2024-04-16T16:22:48.710073+0000","last_fullsized":"2024-04-16T16:22:48.710073+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:35:00.579428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635706+0000","last_change":"2024-04-16T16:22:45.456041+0000","last_active":"2024-04-16T16:22:48.635706+0000","last_peered":"2024-04-16T16:22:48.635706+0000","last_clean":"2024-04-16T16:22:48.635706+0000","last_became_active":"2024-04-16T16:22:45.455767+0000","last_became_peered":"2024-04-16T16:22:45.455767+0000","last_unstale":"2024-04-16T16:22:48.635706+0000","last_undegraded":"2024-04-16T16:22:48.635706+0000","last_fullsized":"2024-04-16T16:22:48.635706+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:26:37.373159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457377+0000","last_change":"2024-04-16T16:22:46.467743+0000","last_active":"2024-04-16T16:22:47.457377+0000","last_peered":"2024-04-16T16:22:47.457377+0000","last_clean":"2024-04-16T16:22:47.457377+0000","last_became_active":"2024-04-16T16:22:46.465149+0000","last_became_peered":"2024-04-16T16:22:46.465149+0000","last_unstale":"2024-04-16T16:22:47.457377+0000","last_undegraded":"2024-04-16T16:22:47.457377+0000","last_fullsized":"2024-04-16T16:22:47.457377+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:18.072160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710644+0000","last_change":"2024-04-16T16:22:45.467985+0000","last_active":"2024-04-16T16:22:48.710644+0000","last_peered":"2024-04-16T16:22:48.710644+0000","last_clean":"2024-04-16T16:22:48.710644+0000","last_became_active":"2024-04-16T16:22:45.467704+0000","last_became_peered":"2024-04-16T16:22:45.467704+0000","last_unstale":"2024-04-16T16:22:48.710644+0000","last_undegraded":"2024-04-16T16:22:48.710644+0000","last_fullsized":"2024-04-16T16:22:48.710644+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:58.753768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457126+0000","last_change":"2024-04-16T16:22:46.473841+0000","last_active":"2024-04-16T16:22:47.457126+0000","last_peered":"2024-04-16T16:22:47.457126+0000","last_clean":"2024-04-16T16:22:47.457126+0000","last_became_active":"2024-04-16T16:22:46.473500+0000","last_became_peered":"2024-04-16T16:22:46.473500+0000","last_unstale":"2024-04-16T16:22:47.457126+0000","last_undegraded":"2024-04-16T16:22:47.457126+0000","last_fullsized":"2024-04-16T16:22:47.457126+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:50:29.718907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"155'3","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710356+0000","last_change":"2024-04-16T16:22:45.461376+0000","last_active":"2024-04-16T16:22:48.710356+0000","last_peered":"2024-04-16T16:22:48.710356+0000","last_clean":"2024-04-16T16:22:48.710356+0000","last_became_active":"2024-04-16T16:22:45.461088+0000","last_became_peered":"2024-04-16T16:22:45.461088+0000","last_unstale":"2024-04-16T16:22:48.710356+0000","last_undegraded":"2024-04-16T16:22:48.710356+0000","last_fullsized":"2024-04-16T16:22:48.710356+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:43:07.885858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.464703+0000","last_change":"2024-04-16T16:22:46.473263+0000","last_active":"2024-04-16T16:22:47.464703+0000","last_peered":"2024-04-16T16:22:47.464703+0000","last_clean":"2024-04-16T16:22:47.464703+0000","last_became_active":"2024-04-16T16:22:46.472979+0000","last_became_peered":"2024-04-16T16:22:46.472979+0000","last_unstale":"2024-04-16T16:22:47.464703+0000","last_undegraded":"2024-04-16T16:22:47.464703+0000","last_fullsized":"2024-04-16T16:22:47.464703+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:37.006661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"155'6","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463168+0000","last_change":"2024-04-16T16:22:45.462782+0000","last_active":"2024-04-16T16:22:47.463168+0000","last_peered":"2024-04-16T16:22:47.463168+0000","last_clean":"2024-04-16T16:22:47.463168+0000","last_became_active":"2024-04-16T16:22:45.461583+0000","last_became_peered":"2024-04-16T16:22:45.461583+0000","last_unstale":"2024-04-16T16:22:47.463168+0000","last_undegraded":"2024-04-16T16:22:47.463168+0000","last_fullsized":"2024-04-16T16:22:47.463168+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.456686+0000","last_change":"2024-04-16T16:22:46.466170+0000","last_active":"2024-04-16T16:22:47.456686+0000","last_peered":"2024-04-16T16:22:47.456686+0000","last_clean":"2024-04-16T16:22:47.456686+0000","last_became_active":"2024-04-16T16:22:46.465894+0000","last_became_peered":"2024-04-16T16:22:46.465894+0000","last_unstale":"2024-04-16T16:22:47.456686+0000","last_undegraded":"2024-04-16T16:22:47.456686+0000","last_fullsized":"2024-04-16T16:22:47.456686+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:01.225311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077288+0000","last_change":"2024-04-16T16:22:45.466273+0000","last_active":"2024-04-16T16:22:48.077288+0000","last_peered":"2024-04-16T16:22:48.077288+0000","last_clean":"2024-04-16T16:22:48.077288+0000","last_became_active":"2024-04-16T16:22:45.465893+0000","last_became_peered":"2024-04-16T16:22:45.465893+0000","last_unstale":"2024-04-16T16:22:48.077288+0000","last_undegraded":"2024-04-16T16:22:48.077288+0000","last_fullsized":"2024-04-16T16:22:48.077288+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:52:57.403249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.636259+0000","last_change":"2024-04-16T16:22:46.474507+0000","last_active":"2024-04-16T16:22:48.636259+0000","last_peered":"2024-04-16T16:22:48.636259+0000","last_clean":"2024-04-16T16:22:48.636259+0000","last_became_active":"2024-04-16T16:22:46.474146+0000","last_became_peered":"2024-04-16T16:22:46.474146+0000","last_unstale":"2024-04-16T16:22:48.636259+0000","last_undegraded":"2024-04-16T16:22:48.636259+0000","last_fullsized":"2024-04-16T16:22:48.636259+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:25:59.259590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.456803+0000","last_change":"2024-04-16T16:22:45.463747+0000","last_active":"2024-04-16T16:22:47.456803+0000","last_peered":"2024-04-16T16:22:47.456803+0000","last_clean":"2024-04-16T16:22:47.456803+0000","last_became_active":"2024-04-16T16:22:45.462875+0000","last_became_peered":"2024-04-16T16:22:45.462875+0000","last_unstale":"2024-04-16T16:22:47.456803+0000","last_undegraded":"2024-04-16T16:22:47.456803+0000","last_fullsized":"2024-04-16T16:22:47.456803+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:29:33.964433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.076960+0000","last_change":"2024-04-16T16:22:46.473128+0000","last_active":"2024-04-16T16:22:48.076960+0000","last_peered":"2024-04-16T16:22:48.076960+0000","last_clean":"2024-04-16T16:22:48.076960+0000","last_became_active":"2024-04-16T16:22:46.472859+0000","last_became_peered":"2024-04-16T16:22:46.472859+0000","last_unstale":"2024-04-16T16:22:48.076960+0000","last_undegraded":"2024-04-16T16:22:48.076960+0000","last_fullsized":"2024-04-16T16:22:48.076960+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:36:30.108257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.076910+0000","last_change":"2024-04-16T16:22:45.466908+0000","last_active":"2024-04-16T16:22:48.076910+0000","last_peered":"2024-04-16T16:22:48.076910+0000","last_clean":"2024-04-16T16:22:48.076910+0000","last_became_active":"2024-04-16T16:22:45.466535+0000","last_became_peered":"2024-04-16T16:22:45.466535+0000","last_unstale":"2024-04-16T16:22:48.076910+0000","last_undegraded":"2024-04-16T16:22:48.076910+0000","last_fullsized":"2024-04-16T16:22:48.076910+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:25.557203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623664+0000","last_change":"2024-04-16T16:22:46.483216+0000","last_active":"2024-04-16T16:22:48.623664+0000","last_peered":"2024-04-16T16:22:48.623664+0000","last_clean":"2024-04-16T16:22:48.623664+0000","last_became_active":"2024-04-16T16:22:46.482943+0000","last_became_peered":"2024-04-16T16:22:46.482943+0000","last_unstale":"2024-04-16T16:22:48.623664+0000","last_undegraded":"2024-04-16T16:22:48.623664+0000","last_fullsized":"2024-04-16T16:22:48.623664+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:26.573221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"155'8","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460436+0000","last_change":"2024-04-16T16:22:45.461068+0000","last_active":"2024-04-16T16:22:47.460436+0000","last_peered":"2024-04-16T16:22:47.460436+0000","last_clean":"2024-04-16T16:22:47.460436+0000","last_became_active":"2024-04-16T16:22:45.459760+0000","last_became_peered":"2024-04-16T16:22:45.459760+0000","last_unstale":"2024-04-16T16:22:47.460436+0000","last_undegraded":"2024-04-16T16:22:47.460436+0000","last_fullsized":"2024-04-16T16:22:47.460436+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:32:24.648814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.078301+0000","last_change":"2024-04-16T16:22:46.473695+0000","last_active":"2024-04-16T16:22:48.078301+0000","last_peered":"2024-04-16T16:22:48.078301+0000","last_clean":"2024-04-16T16:22:48.078301+0000","last_became_active":"2024-04-16T16:22:46.471894+0000","last_became_peered":"2024-04-16T16:22:46.471894+0000","last_unstale":"2024-04-16T16:22:48.078301+0000","last_undegraded":"2024-04-16T16:22:48.078301+0000","last_fullsized":"2024-04-16T16:22:48.078301+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:26.785078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.625205+0000","last_change":"2024-04-16T16:22:45.469335+0000","last_active":"2024-04-16T16:22:48.625205+0000","last_peered":"2024-04-16T16:22:48.625205+0000","last_clean":"2024-04-16T16:22:48.625205+0000","last_became_active":"2024-04-16T16:22:45.468966+0000","last_became_peered":"2024-04-16T16:22:45.468966+0000","last_unstale":"2024-04-16T16:22:48.625205+0000","last_undegraded":"2024-04-16T16:22:48.625205+0000","last_fullsized":"2024-04-16T16:22:48.625205+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:09.300484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457513+0000","last_change":"2024-04-16T16:22:46.471925+0000","last_active":"2024-04-16T16:22:47.457513+0000","last_peered":"2024-04-16T16:22:47.457513+0000","last_clean":"2024-04-16T16:22:47.457513+0000","last_became_active":"2024-04-16T16:22:46.470405+0000","last_became_peered":"2024-04-16T16:22:46.470405+0000","last_unstale":"2024-04-16T16:22:47.457513+0000","last_undegraded":"2024-04-16T16:22:47.457513+0000","last_fullsized":"2024-04-16T16:22:47.457513+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:04.570708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624612+0000","last_change":"2024-04-16T16:22:45.470272+0000","last_active":"2024-04-16T16:22:48.624612+0000","last_peered":"2024-04-16T16:22:48.624612+0000","last_clean":"2024-04-16T16:22:48.624612+0000","last_became_active":"2024-04-16T16:22:45.469959+0000","last_became_peered":"2024-04-16T16:22:45.469959+0000","last_unstale":"2024-04-16T16:22:48.624612+0000","last_undegraded":"2024-04-16T16:22:48.624612+0000","last_fullsized":"2024-04-16T16:22:48.624612+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:34:03.813486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.078101+0000","last_change":"2024-04-16T16:22:46.462951+0000","last_active":"2024-04-16T16:22:48.078101+0000","last_peered":"2024-04-16T16:22:48.078101+0000","last_clean":"2024-04-16T16:22:48.078101+0000","last_became_active":"2024-04-16T16:22:46.462662+0000","last_became_peered":"2024-04-16T16:22:46.462662+0000","last_unstale":"2024-04-16T16:22:48.078101+0000","last_undegraded":"2024-04-16T16:22:48.078101+0000","last_fullsized":"2024-04-16T16:22:48.078101+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:37.006661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462044+0000","last_change":"2024-04-16T16:22:45.462862+0000","last_active":"2024-04-16T16:22:47.462044+0000","last_peered":"2024-04-16T16:22:47.462044+0000","last_clean":"2024-04-16T16:22:47.462044+0000","last_became_active":"2024-04-16T16:22:45.462197+0000","last_became_peered":"2024-04-16T16:22:45.462197+0000","last_unstale":"2024-04-16T16:22:47.462044+0000","last_undegraded":"2024-04-16T16:22:47.462044+0000","last_fullsized":"2024-04-16T16:22:47.462044+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:55.697931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.636077+0000","last_change":"2024-04-16T16:22:46.475567+0000","last_active":"2024-04-16T16:22:48.636077+0000","last_peered":"2024-04-16T16:22:48.636077+0000","last_clean":"2024-04-16T16:22:48.636077+0000","last_became_active":"2024-04-16T16:22:46.475232+0000","last_became_peered":"2024-04-16T16:22:46.475232+0000","last_unstale":"2024-04-16T16:22:48.636077+0000","last_undegraded":"2024-04-16T16:22:48.636077+0000","last_fullsized":"2024-04-16T16:22:48.636077+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:19:43.312174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462377+0000","last_change":"2024-04-16T16:22:45.466893+0000","last_active":"2024-04-16T16:22:47.462377+0000","last_peered":"2024-04-16T16:22:47.462377+0000","last_clean":"2024-04-16T16:22:47.462377+0000","last_became_active":"2024-04-16T16:22:45.466614+0000","last_became_peered":"2024-04-16T16:22:45.466614+0000","last_unstale":"2024-04-16T16:22:47.462377+0000","last_undegraded":"2024-04-16T16:22:47.462377+0000","last_fullsized":"2024-04-16T16:22:47.462377+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:53:47.150015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623905+0000","last_change":"2024-04-16T16:22:46.482036+0000","last_active":"2024-04-16T16:22:48.623905+0000","last_peered":"2024-04-16T16:22:48.623905+0000","last_clean":"2024-04-16T16:22:48.623905+0000","last_became_active":"2024-04-16T16:22:46.481293+0000","last_became_peered":"2024-04-16T16:22:46.481293+0000","last_unstale":"2024-04-16T16:22:48.623905+0000","last_undegraded":"2024-04-16T16:22:48.623905+0000","last_fullsized":"2024-04-16T16:22:48.623905+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:55.094277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077501+0000","last_change":"2024-04-16T16:22:45.466947+0000","last_active":"2024-04-16T16:22:48.077501+0000","last_peered":"2024-04-16T16:22:48.077501+0000","last_clean":"2024-04-16T16:22:48.077501+0000","last_became_active":"2024-04-16T16:22:45.466624+0000","last_became_peered":"2024-04-16T16:22:45.466624+0000","last_unstale":"2024-04-16T16:22:48.077501+0000","last_undegraded":"2024-04-16T16:22:48.077501+0000","last_fullsized":"2024-04-16T16:22:48.077501+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:53.044305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.636145+0000","last_change":"2024-04-16T16:22:46.469862+0000","last_active":"2024-04-16T16:22:48.636145+0000","last_peered":"2024-04-16T16:22:48.636145+0000","last_clean":"2024-04-16T16:22:48.636145+0000","last_became_active":"2024-04-16T16:22:46.466179+0000","last_became_peered":"2024-04-16T16:22:46.466179+0000","last_unstale":"2024-04-16T16:22:48.636145+0000","last_undegraded":"2024-04-16T16:22:48.636145+0000","last_fullsized":"2024-04-16T16:22:48.636145+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:36:56.658760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460683+0000","last_change":"2024-04-16T16:22:45.461564+0000","last_active":"2024-04-16T16:22:47.460683+0000","last_peered":"2024-04-16T16:22:47.460683+0000","last_clean":"2024-04-16T16:22:47.460683+0000","last_became_active":"2024-04-16T16:22:45.461253+0000","last_became_peered":"2024-04-16T16:22:45.461253+0000","last_unstale":"2024-04-16T16:22:47.460683+0000","last_undegraded":"2024-04-16T16:22:47.460683+0000","last_fullsized":"2024-04-16T16:22:47.460683+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:46.018568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462392+0000","last_change":"2024-04-16T16:22:46.465947+0000","last_active":"2024-04-16T16:22:47.462392+0000","last_peered":"2024-04-16T16:22:47.462392+0000","last_clean":"2024-04-16T16:22:47.462392+0000","last_became_active":"2024-04-16T16:22:46.465294+0000","last_became_peered":"2024-04-16T16:22:46.465294+0000","last_unstale":"2024-04-16T16:22:47.462392+0000","last_undegraded":"2024-04-16T16:22:47.462392+0000","last_fullsized":"2024-04-16T16:22:47.462392+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:29.789672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077036+0000","last_change":"2024-04-16T16:22:45.468883+0000","last_active":"2024-04-16T16:22:48.077036+0000","last_peered":"2024-04-16T16:22:48.077036+0000","last_clean":"2024-04-16T16:22:48.077036+0000","last_became_active":"2024-04-16T16:22:45.468091+0000","last_became_peered":"2024-04-16T16:22:45.468091+0000","last_unstale":"2024-04-16T16:22:48.077036+0000","last_undegraded":"2024-04-16T16:22:48.077036+0000","last_fullsized":"2024-04-16T16:22:48.077036+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710495+0000","last_change":"2024-04-16T16:22:46.471354+0000","last_active":"2024-04-16T16:22:48.710495+0000","last_peered":"2024-04-16T16:22:48.710495+0000","last_clean":"2024-04-16T16:22:48.710495+0000","last_became_active":"2024-04-16T16:22:46.470521+0000","last_became_peered":"2024-04-16T16:22:46.470521+0000","last_unstale":"2024-04-16T16:22:48.710495+0000","last_undegraded":"2024-04-16T16:22:48.710495+0000","last_fullsized":"2024-04-16T16:22:48.710495+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:06:08.736226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461147+0000","last_change":"2024-04-16T16:22:45.460865+0000","last_active":"2024-04-16T16:22:47.461147+0000","last_peered":"2024-04-16T16:22:47.461147+0000","last_clean":"2024-04-16T16:22:47.461147+0000","last_became_active":"2024-04-16T16:22:45.458861+0000","last_became_peered":"2024-04-16T16:22:45.458861+0000","last_unstale":"2024-04-16T16:22:47.461147+0000","last_undegraded":"2024-04-16T16:22:47.461147+0000","last_fullsized":"2024-04-16T16:22:47.461147+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:30.462179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457529+0000","last_change":"2024-04-16T16:22:46.464764+0000","last_active":"2024-04-16T16:22:47.457529+0000","last_peered":"2024-04-16T16:22:47.457529+0000","last_clean":"2024-04-16T16:22:47.457529+0000","last_became_active":"2024-04-16T16:22:46.464064+0000","last_became_peered":"2024-04-16T16:22:46.464064+0000","last_unstale":"2024-04-16T16:22:47.457529+0000","last_undegraded":"2024-04-16T16:22:47.457529+0000","last_fullsized":"2024-04-16T16:22:47.457529+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:28:07.667824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457501+0000","last_change":"2024-04-16T16:22:45.466044+0000","last_active":"2024-04-16T16:22:47.457501+0000","last_peered":"2024-04-16T16:22:47.457501+0000","last_clean":"2024-04-16T16:22:47.457501+0000","last_became_active":"2024-04-16T16:22:45.465676+0000","last_became_peered":"2024-04-16T16:22:45.465676+0000","last_unstale":"2024-04-16T16:22:47.457501+0000","last_undegraded":"2024-04-16T16:22:47.457501+0000","last_fullsized":"2024-04-16T16:22:47.457501+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:52.385757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635973+0000","last_change":"2024-04-16T16:22:46.466356+0000","last_active":"2024-04-16T16:22:48.635973+0000","last_peered":"2024-04-16T16:22:48.635973+0000","last_clean":"2024-04-16T16:22:48.635973+0000","last_became_active":"2024-04-16T16:22:46.465449+0000","last_became_peered":"2024-04-16T16:22:46.465449+0000","last_unstale":"2024-04-16T16:22:48.635973+0000","last_undegraded":"2024-04-16T16:22:48.635973+0000","last_fullsized":"2024-04-16T16:22:48.635973+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:04.570708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624728+0000","last_change":"2024-04-16T16:22:45.466156+0000","last_active":"2024-04-16T16:22:48.624728+0000","last_peered":"2024-04-16T16:22:48.624728+0000","last_clean":"2024-04-16T16:22:48.624728+0000","last_became_active":"2024-04-16T16:22:45.465791+0000","last_became_peered":"2024-04-16T16:22:45.465791+0000","last_unstale":"2024-04-16T16:22:48.624728+0000","last_undegraded":"2024-04-16T16:22:48.624728+0000","last_fullsized":"2024-04-16T16:22:48.624728+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:42:46.394979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710309+0000","last_change":"2024-04-16T16:22:46.470854+0000","last_active":"2024-04-16T16:22:48.710309+0000","last_peered":"2024-04-16T16:22:48.710309+0000","last_clean":"2024-04-16T16:22:48.710309+0000","last_became_active":"2024-04-16T16:22:46.470196+0000","last_became_peered":"2024-04-16T16:22:46.470196+0000","last_unstale":"2024-04-16T16:22:48.710309+0000","last_undegraded":"2024-04-16T16:22:48.710309+0000","last_fullsized":"2024-04-16T16:22:48.710309+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:40:24.085281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426634+0000","last_change":"2024-04-16T16:22:45.471205+0000","last_active":"2024-04-16T16:22:48.426634+0000","last_peered":"2024-04-16T16:22:48.426634+0000","last_clean":"2024-04-16T16:22:48.426634+0000","last_became_active":"2024-04-16T16:22:45.470911+0000","last_became_peered":"2024-04-16T16:22:45.470911+0000","last_unstale":"2024-04-16T16:22:48.426634+0000","last_undegraded":"2024-04-16T16:22:48.426634+0000","last_fullsized":"2024-04-16T16:22:48.426634+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:53.044305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457242+0000","last_change":"2024-04-16T16:22:46.469324+0000","last_active":"2024-04-16T16:22:47.457242+0000","last_peered":"2024-04-16T16:22:47.457242+0000","last_clean":"2024-04-16T16:22:47.457242+0000","last_became_active":"2024-04-16T16:22:46.469009+0000","last_became_peered":"2024-04-16T16:22:46.469009+0000","last_unstale":"2024-04-16T16:22:47.457242+0000","last_undegraded":"2024-04-16T16:22:47.457242+0000","last_fullsized":"2024-04-16T16:22:47.457242+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:29.789672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457098+0000","last_change":"2024-04-16T16:22:45.468189+0000","last_active":"2024-04-16T16:22:47.457098+0000","last_peered":"2024-04-16T16:22:47.457098+0000","last_clean":"2024-04-16T16:22:47.457098+0000","last_became_active":"2024-04-16T16:22:45.467709+0000","last_became_peered":"2024-04-16T16:22:45.467709+0000","last_unstale":"2024-04-16T16:22:47.457098+0000","last_undegraded":"2024-04-16T16:22:47.457098+0000","last_fullsized":"2024-04-16T16:22:47.457098+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:46.256132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.456756+0000","last_change":"2024-04-16T16:22:46.465994+0000","last_active":"2024-04-16T16:22:47.456756+0000","last_peered":"2024-04-16T16:22:47.456756+0000","last_clean":"2024-04-16T16:22:47.456756+0000","last_became_active":"2024-04-16T16:22:46.465007+0000","last_became_peered":"2024-04-16T16:22:46.465007+0000","last_unstale":"2024-04-16T16:22:47.456756+0000","last_undegraded":"2024-04-16T16:22:47.456756+0000","last_fullsized":"2024-04-16T16:22:47.456756+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:12:58.505334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462996+0000","last_change":"2024-04-16T16:22:45.470607+0000","last_active":"2024-04-16T16:22:47.462996+0000","last_peered":"2024-04-16T16:22:47.462996+0000","last_clean":"2024-04-16T16:22:47.462996+0000","last_became_active":"2024-04-16T16:22:45.469322+0000","last_became_peered":"2024-04-16T16:22:45.469322+0000","last_unstale":"2024-04-16T16:22:47.462996+0000","last_undegraded":"2024-04-16T16:22:47.462996+0000","last_fullsized":"2024-04-16T16:22:47.462996+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:07.564200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463072+0000","last_change":"2024-04-16T16:22:46.462483+0000","last_active":"2024-04-16T16:22:47.463072+0000","last_peered":"2024-04-16T16:22:47.463072+0000","last_clean":"2024-04-16T16:22:47.463072+0000","last_became_active":"2024-04-16T16:22:46.461432+0000","last_became_peered":"2024-04-16T16:22:46.461432+0000","last_unstale":"2024-04-16T16:22:47.463072+0000","last_undegraded":"2024-04-16T16:22:47.463072+0000","last_fullsized":"2024-04-16T16:22:47.463072+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:01.883490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077289+0000","last_change":"2024-04-16T16:22:45.462752+0000","last_active":"2024-04-16T16:22:48.077289+0000","last_peered":"2024-04-16T16:22:48.077289+0000","last_clean":"2024-04-16T16:22:48.077289+0000","last_became_active":"2024-04-16T16:22:45.462154+0000","last_became_peered":"2024-04-16T16:22:45.462154+0000","last_unstale":"2024-04-16T16:22:48.077289+0000","last_undegraded":"2024-04-16T16:22:48.077289+0000","last_fullsized":"2024-04-16T16:22:48.077289+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:43:04.134472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635368+0000","last_change":"2024-04-16T16:22:46.467575+0000","last_active":"2024-04-16T16:22:48.635368+0000","last_peered":"2024-04-16T16:22:48.635368+0000","last_clean":"2024-04-16T16:22:48.635368+0000","last_became_active":"2024-04-16T16:22:46.466890+0000","last_became_peered":"2024-04-16T16:22:46.466890+0000","last_unstale":"2024-04-16T16:22:48.635368+0000","last_undegraded":"2024-04-16T16:22:48.635368+0000","last_fullsized":"2024-04-16T16:22:48.635368+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:43:19.877365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426288+0000","last_change":"2024-04-16T16:22:45.469689+0000","last_active":"2024-04-16T16:22:48.426288+0000","last_peered":"2024-04-16T16:22:48.426288+0000","last_clean":"2024-04-16T16:22:48.426288+0000","last_became_active":"2024-04-16T16:22:45.469350+0000","last_became_peered":"2024-04-16T16:22:45.469350+0000","last_unstale":"2024-04-16T16:22:48.426288+0000","last_undegraded":"2024-04-16T16:22:48.426288+0000","last_fullsized":"2024-04-16T16:22:48.426288+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623691+0000","last_change":"2024-04-16T16:22:46.478730+0000","last_active":"2024-04-16T16:22:48.623691+0000","last_peered":"2024-04-16T16:22:48.623691+0000","last_clean":"2024-04-16T16:22:48.623691+0000","last_became_active":"2024-04-16T16:22:46.478459+0000","last_became_peered":"2024-04-16T16:22:46.478459+0000","last_unstale":"2024-04-16T16:22:48.623691+0000","last_undegraded":"2024-04-16T16:22:48.623691+0000","last_fullsized":"2024-04-16T16:22:48.623691+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:36.777723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460788+0000","last_change":"2024-04-16T16:22:45.467505+0000","last_active":"2024-04-16T16:22:47.460788+0000","last_peered":"2024-04-16T16:22:47.460788+0000","last_clean":"2024-04-16T16:22:47.460788+0000","last_became_active":"2024-04-16T16:22:45.465339+0000","last_became_peered":"2024-04-16T16:22:45.465339+0000","last_unstale":"2024-04-16T16:22:47.460788+0000","last_undegraded":"2024-04-16T16:22:47.460788+0000","last_fullsized":"2024-04-16T16:22:47.460788+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:54.078259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457379+0000","last_change":"2024-04-16T16:22:46.465023+0000","last_active":"2024-04-16T16:22:47.457379+0000","last_peered":"2024-04-16T16:22:47.457379+0000","last_clean":"2024-04-16T16:22:47.457379+0000","last_became_active":"2024-04-16T16:22:46.464436+0000","last_became_peered":"2024-04-16T16:22:46.464436+0000","last_unstale":"2024-04-16T16:22:47.457379+0000","last_undegraded":"2024-04-16T16:22:47.457379+0000","last_fullsized":"2024-04-16T16:22:47.457379+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:54:26.843618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463842+0000","last_change":"2024-04-16T16:22:45.461515+0000","last_active":"2024-04-16T16:22:47.463842+0000","last_peered":"2024-04-16T16:22:47.463842+0000","last_clean":"2024-04-16T16:22:47.463842+0000","last_became_active":"2024-04-16T16:22:45.460350+0000","last_became_peered":"2024-04-16T16:22:45.460350+0000","last_unstale":"2024-04-16T16:22:47.463842+0000","last_undegraded":"2024-04-16T16:22:47.463842+0000","last_fullsized":"2024-04-16T16:22:47.463842+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:34.292891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"144'4691","reported_seq":5390,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462677+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:22:47.462677+0000","last_peered":"2024-04-16T16:22:47.462677+0000","last_clean":"2024-04-16T16:22:47.462677+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:22:47.462677+0000","last_undegraded":"2024-04-16T16:22:47.462677+0000","last_fullsized":"2024-04-16T16:22:47.462677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":4691,"log_dups_size":0,"ondisk_log_size":4691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":387,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":387,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4691,"ondisk_log_size":4691,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574957,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37856,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333984,"statfs":{"total":96636764160,"available":96597999616,"internally_reserved":0,"allocated":2576384,"data_stored":1526220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574958,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1961984,"data_stored":933520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574958,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":934102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574959,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33116,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338724,"statfs":{"total":96636764160,"available":96602853376,"internally_reserved":0,"allocated":1974272,"data_stored":935180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574957,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37264,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334576,"statfs":{"total":96636764160,"available":96598605824,"internally_reserved":0,"allocated":1961984,"data_stored":933532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574957,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37256,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334584,"statfs":{"total":96636764160,"available":96598614016,"internally_reserved":0,"allocated":1961984,"data_stored":933588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37256,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334584,"statfs":{"total":96636764160,"available":96598614016,"internally_reserved":0,"allocated":1961984,"data_stored":933532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574958,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37832,"kb_used_data":2492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334008,"statfs":{"total":96636764160,"available":96598024192,"internally_reserved":0,"allocated":2551808,"data_stored":1523866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:22:54.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:22:55.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.207+0000 7f9639a00640 1 Processor -- start 2024-04-16T16:22:55.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.207+0000 7f9639a00640 1 -- start start 2024-04-16T16:22:55.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.209+0000 7f9639a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9634163010 0x7f96340cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:55.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.209+0000 7f9639a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f963406f540 0x7f96341645b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:55.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.209+0000 7f9639a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f963406f9b0 con 0x7f96340cde00 2024-04-16T16:22:55.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.209+0000 7f9639a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9634163510 con 0x7f963406f540 2024-04-16T16:22:55.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.209+0000 7f9639a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f96341637f0 con 0x7f9634163010 2024-04-16T16:22:55.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.210+0000 7f962b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9634163010 0x7f96340cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:55.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.210+0000 7f9639000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f963406f540 0x7f96341645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:55.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.210+0000 7f9639000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f963406f540 0x7f96341645b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.45:57478/0 (socket says 172.21.15.45:57478) 2024-04-16T16:22:55.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.210+0000 7f9639000640 1 -- 172.21.15.45:0/401198852 learned_addr learned my addr 172.21.15.45:0/401198852 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.210+0000 7f962b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9634163010 0x7f96340cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:53152/0 (socket says 172.21.15.45:53152) 2024-04-16T16:22:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.210+0000 7f962b400640 1 -- 172.21.15.45:0/401198852 >> v2:172.21.15.103:3300/0 conn(0x7f963406f540 msgr2=0x7f96341645b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.210+0000 7f962b400640 1 --2- 172.21.15.45:0/401198852 >> v2:172.21.15.103:3300/0 conn(0x7f963406f540 0x7f96341645b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.210+0000 7f962b400640 1 -- 172.21.15.45:0/401198852 >> v1:172.21.15.103:6789/0 conn(0x7f96340cde00 legacy=0x7f96340ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.211+0000 7f962b400640 1 -- 172.21.15.45:0/401198852 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9634163ad0 con 0x7f9634163010 2024-04-16T16:22:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.212+0000 7f962b400640 1 --2- 172.21.15.45:0/401198852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9634163010 0x7f96340cff30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f961c0097a0 tx=0x7f961c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bda615a8cf2345a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:55.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.212+0000 7f9633400640 1 -- 172.21.15.45:0/401198852 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c035020 con 0x7f9634163010 2024-04-16T16:22:55.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.213+0000 7f9633400640 1 -- 172.21.15.45:0/401198852 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f961c031810 con 0x7f9634163010 2024-04-16T16:22:55.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.213+0000 7f9633400640 1 -- 172.21.15.45:0/401198852 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c031be0 con 0x7f9634163010 2024-04-16T16:22:55.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.213+0000 7f9639a00640 1 -- 172.21.15.45:0/401198852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9634163010 msgr2=0x7f96340cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:55.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.213+0000 7f9639a00640 1 --2- 172.21.15.45:0/401198852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9634163010 0x7f96340cff30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f961c0097a0 tx=0x7f961c02b8b0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.213+0000 7f9639a00640 1 -- 172.21.15.45:0/401198852 shutdown_connections 2024-04-16T16:22:55.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.213+0000 7f9639a00640 1 --2- 172.21.15.45:0/401198852 >> v2:172.21.15.103:3300/0 conn(0x7f963406f540 0x7f96341645b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.213+0000 7f9639a00640 1 --2- 172.21.15.45:0/401198852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9634163010 0x7f96340cff30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.213+0000 7f9639a00640 1 -- 172.21.15.45:0/401198852 >> 172.21.15.45:0/401198852 conn(0x7f963406b930 msgr2=0x7f963406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:55.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.213+0000 7f9639a00640 1 -- 172.21.15.45:0/401198852 shutdown_connections 2024-04-16T16:22:55.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.213+0000 7f9639a00640 1 -- 172.21.15.45:0/401198852 wait complete. 2024-04-16T16:22:55.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.215+0000 7f9639a00640 1 Processor -- start 2024-04-16T16:22:55.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.215+0000 7f9639a00640 1 -- start start 2024-04-16T16:22:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.216+0000 7f9639a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963406f540 0x7f9634155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.216+0000 7f9639a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f96340cde00 0x7f963414ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.216+0000 7f9639a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9634163010 0x7f963406e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.216+0000 7f9639a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9634164af0 con 0x7f963406f540 2024-04-16T16:22:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.216+0000 7f9639a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9634164090 con 0x7f9634163010 2024-04-16T16:22:55.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.216+0000 7f9639a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9634164c70 con 0x7f96340cde00 2024-04-16T16:22:55.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.216+0000 7f9633e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963406f540 0x7f9634155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:55.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.217+0000 7f9633e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963406f540 0x7f9634155020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47248/0 (socket says 172.21.15.45:47248) 2024-04-16T16:22:55.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.217+0000 7f9633e00640 1 -- 172.21.15.45:0/2162233139 learned_addr learned my addr 172.21.15.45:0/2162233139 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.217+0000 7f962b400640 1 --2- 172.21.15.45:0/2162233139 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f96340cde00 0x7f963414ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.217+0000 7f9639000640 1 --2- 172.21.15.45:0/2162233139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9634163010 0x7f963406e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.217+0000 7f9633e00640 1 -- 172.21.15.45:0/2162233139 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f96340cde00 msgr2=0x7f963414ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.217+0000 7f9633e00640 1 --2- 172.21.15.45:0/2162233139 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f96340cde00 0x7f963414ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.217+0000 7f9633e00640 1 -- 172.21.15.45:0/2162233139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9634163010 msgr2=0x7f963406e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:55.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.217+0000 7f9633e00640 1 --2- 172.21.15.45:0/2162233139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9634163010 0x7f963406e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.217+0000 7f9633e00640 1 -- 172.21.15.45:0/2162233139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96341fca30 con 0x7f963406f540 2024-04-16T16:22:55.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.217+0000 7f9633e00640 1 --2- 172.21.15.45:0/2162233139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963406f540 0x7f9634155020 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f962400c400 tx=0x7f962400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:55.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.218+0000 7f9630c00640 1 -- 172.21.15.45:0/2162233139 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96240052d0 con 0x7f963406f540 2024-04-16T16:22:55.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.218+0000 7f9630c00640 1 -- 172.21.15.45:0/2162233139 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9624005490 con 0x7f963406f540 2024-04-16T16:22:55.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.218+0000 7f9630c00640 1 -- 172.21.15.45:0/2162233139 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9624007260 con 0x7f963406f540 2024-04-16T16:22:55.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.218+0000 7f9639a00640 1 -- 172.21.15.45:0/2162233139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96341fcd10 con 0x7f963406f540 2024-04-16T16:22:55.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.218+0000 7f9639a00640 1 -- 172.21.15.45:0/2162233139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96341fd2d0 con 0x7f963406f540 2024-04-16T16:22:55.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.219+0000 7f9639a00640 1 -- 172.21.15.45:0/2162233139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9634075780 con 0x7f963406f540 2024-04-16T16:22:55.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.220+0000 7f9630c00640 1 -- 172.21.15.45:0/2162233139 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f9624007400 con 0x7f963406f540 2024-04-16T16:22:55.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.221+0000 7f9630c00640 1 --2- 172.21.15.45:0/2162233139 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f961006ca10 0x7f961006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:55.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.221+0000 7f9630c00640 1 -- 172.21.15.45:0/2162233139 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f962408b390 con 0x7f963406f540 2024-04-16T16:22:55.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.222+0000 7f962b400640 1 --2- 172.21.15.45:0/2162233139 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f961006ca10 0x7f961006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:55.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.223+0000 7f962b400640 1 --2- 172.21.15.45:0/2162233139 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f961006ca10 0x7f961006eed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f961c036000 tx=0x7f961c03c040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:55.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.228+0000 7f9630c00640 1 -- 172.21.15.45:0/2162233139 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9624059590 con 0x7f963406f540 2024-04-16T16:22:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.428+0000 7f9639a00640 1 -- 172.21.15.45:0/2162233139 --> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f963406fc20 con 0x7f961006ca10 2024-04-16T16:22:55.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.448+0000 7f9630c00640 1 -- 172.21.15.45:0/2162233139 <== mgr.7591 v2:172.21.15.103:6832/1129061436 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252497 (secure 0 0 0) 0x7f963406fc20 con 0x7f961006ca10 2024-04-16T16:22:55.450 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:55.451 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:22:55.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.454+0000 7f9639a00640 1 -- 172.21.15.45:0/2162233139 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f961006ca10 msgr2=0x7f961006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:55.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.454+0000 7f9639a00640 1 --2- 172.21.15.45:0/2162233139 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f961006ca10 0x7f961006eed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f961c036000 tx=0x7f961c03c040 comp rx=0 tx=0).stop 2024-04-16T16:22:55.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.454+0000 7f9639a00640 1 -- 172.21.15.45:0/2162233139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963406f540 msgr2=0x7f9634155020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:55.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.454+0000 7f9639a00640 1 --2- 172.21.15.45:0/2162233139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963406f540 0x7f9634155020 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f962400c400 tx=0x7f962400c8c0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.455+0000 7f9639a00640 1 -- 172.21.15.45:0/2162233139 shutdown_connections 2024-04-16T16:22:55.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.455+0000 7f9639a00640 1 --2- 172.21.15.45:0/2162233139 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f961006ca10 0x7f961006eed0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.455+0000 7f9639a00640 1 --2- 172.21.15.45:0/2162233139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9634163010 0x7f963406e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.455+0000 7f9639a00640 1 --2- 172.21.15.45:0/2162233139 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f96340cde00 0x7f963414ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.455+0000 7f9639a00640 1 --2- 172.21.15.45:0/2162233139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f963406f540 0x7f9634155020 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.455+0000 7f9639a00640 1 -- 172.21.15.45:0/2162233139 >> 172.21.15.45:0/2162233139 conn(0x7f963406b930 msgr2=0x7f96341608a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:55.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.455+0000 7f9639a00640 1 -- 172.21.15.45:0/2162233139 shutdown_connections 2024-04-16T16:22:55.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.455+0000 7f9639a00640 1 -- 172.21.15.45:0/2162233139 wait complete. 2024-04-16T16:22:55.474 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-04-16T16:22:53.542715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":431,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4729,"ondisk_log_size":4729,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":286776,"kb_used_data":16520,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754687944,"statfs":{"total":773094113280,"available":772800454656,"internally_reserved":0,"allocated":16916480,"data_stored":8653540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":276614704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002957"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461972+0000","last_change":"2024-04-16T16:22:46.488567+0000","last_active":"2024-04-16T16:22:47.461972+0000","last_peered":"2024-04-16T16:22:47.461972+0000","last_clean":"2024-04-16T16:22:47.461972+0000","last_became_active":"2024-04-16T16:22:46.485690+0000","last_became_peered":"2024-04-16T16:22:46.485690+0000","last_unstale":"2024-04-16T16:22:47.461972+0000","last_undegraded":"2024-04-16T16:22:47.461972+0000","last_fullsized":"2024-04-16T16:22:47.461972+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:50:29.718907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426342+0000","last_change":"2024-04-16T16:22:46.479620+0000","last_active":"2024-04-16T16:22:48.426342+0000","last_peered":"2024-04-16T16:22:48.426342+0000","last_clean":"2024-04-16T16:22:48.426342+0000","last_became_active":"2024-04-16T16:22:46.477410+0000","last_became_peered":"2024-04-16T16:22:46.477410+0000","last_unstale":"2024-04-16T16:22:48.426342+0000","last_undegraded":"2024-04-16T16:22:48.426342+0000","last_fullsized":"2024-04-16T16:22:48.426342+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:35.010598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624440+0000","last_change":"2024-04-16T16:22:46.480944+0000","last_active":"2024-04-16T16:22:48.624440+0000","last_peered":"2024-04-16T16:22:48.624440+0000","last_clean":"2024-04-16T16:22:48.624440+0000","last_became_active":"2024-04-16T16:22:46.480405+0000","last_became_peered":"2024-04-16T16:22:46.480405+0000","last_unstale":"2024-04-16T16:22:48.624440+0000","last_undegraded":"2024-04-16T16:22:48.624440+0000","last_fullsized":"2024-04-16T16:22:48.624440+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:35.308909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077640+0000","last_change":"2024-04-16T16:22:46.477832+0000","last_active":"2024-04-16T16:22:48.077640+0000","last_peered":"2024-04-16T16:22:48.077640+0000","last_clean":"2024-04-16T16:22:48.077640+0000","last_became_active":"2024-04-16T16:22:46.477516+0000","last_became_peered":"2024-04-16T16:22:46.477516+0000","last_unstale":"2024-04-16T16:22:48.077640+0000","last_undegraded":"2024-04-16T16:22:48.077640+0000","last_fullsized":"2024-04-16T16:22:48.077640+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:01.883490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426062+0000","last_change":"2024-04-16T16:22:46.483466+0000","last_active":"2024-04-16T16:22:48.426062+0000","last_peered":"2024-04-16T16:22:48.426062+0000","last_clean":"2024-04-16T16:22:48.426062+0000","last_became_active":"2024-04-16T16:22:46.481033+0000","last_became_peered":"2024-04-16T16:22:46.481033+0000","last_unstale":"2024-04-16T16:22:48.426062+0000","last_undegraded":"2024-04-16T16:22:48.426062+0000","last_fullsized":"2024-04-16T16:22:48.426062+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:36:30.108257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710150+0000","last_change":"2024-04-16T16:22:46.463298+0000","last_active":"2024-04-16T16:22:48.710150+0000","last_peered":"2024-04-16T16:22:48.710150+0000","last_clean":"2024-04-16T16:22:48.710150+0000","last_became_active":"2024-04-16T16:22:46.462017+0000","last_became_peered":"2024-04-16T16:22:46.462017+0000","last_unstale":"2024-04-16T16:22:48.710150+0000","last_undegraded":"2024-04-16T16:22:48.710150+0000","last_fullsized":"2024-04-16T16:22:48.710150+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:30:36.112336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635373+0000","last_change":"2024-04-16T16:22:46.462786+0000","last_active":"2024-04-16T16:22:48.635373+0000","last_peered":"2024-04-16T16:22:48.635373+0000","last_clean":"2024-04-16T16:22:48.635373+0000","last_became_active":"2024-04-16T16:22:46.461856+0000","last_became_peered":"2024-04-16T16:22:46.461856+0000","last_unstale":"2024-04-16T16:22:48.635373+0000","last_undegraded":"2024-04-16T16:22:48.635373+0000","last_fullsized":"2024-04-16T16:22:48.635373+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:29.789672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426662+0000","last_change":"2024-04-16T16:22:46.486917+0000","last_active":"2024-04-16T16:22:48.426662+0000","last_peered":"2024-04-16T16:22:48.426662+0000","last_clean":"2024-04-16T16:22:48.426662+0000","last_became_active":"2024-04-16T16:22:46.485280+0000","last_became_peered":"2024-04-16T16:22:46.485280+0000","last_unstale":"2024-04-16T16:22:48.426662+0000","last_undegraded":"2024-04-16T16:22:48.426662+0000","last_fullsized":"2024-04-16T16:22:48.426662+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:14:27.242053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.427106+0000","last_change":"2024-04-16T16:22:46.476513+0000","last_active":"2024-04-16T16:22:48.427106+0000","last_peered":"2024-04-16T16:22:48.427106+0000","last_clean":"2024-04-16T16:22:48.427106+0000","last_became_active":"2024-04-16T16:22:46.476125+0000","last_became_peered":"2024-04-16T16:22:46.476125+0000","last_unstale":"2024-04-16T16:22:48.427106+0000","last_undegraded":"2024-04-16T16:22:48.427106+0000","last_fullsized":"2024-04-16T16:22:48.427106+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:28.349945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426898+0000","last_change":"2024-04-16T16:22:46.483287+0000","last_active":"2024-04-16T16:22:48.426898+0000","last_peered":"2024-04-16T16:22:48.426898+0000","last_clean":"2024-04-16T16:22:48.426898+0000","last_became_active":"2024-04-16T16:22:46.482975+0000","last_became_peered":"2024-04-16T16:22:46.482975+0000","last_unstale":"2024-04-16T16:22:48.426898+0000","last_undegraded":"2024-04-16T16:22:48.426898+0000","last_fullsized":"2024-04-16T16:22:48.426898+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:21.784358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426384+0000","last_change":"2024-04-16T16:22:46.481301+0000","last_active":"2024-04-16T16:22:48.426384+0000","last_peered":"2024-04-16T16:22:48.426384+0000","last_clean":"2024-04-16T16:22:48.426384+0000","last_became_active":"2024-04-16T16:22:46.481000+0000","last_became_peered":"2024-04-16T16:22:46.481000+0000","last_unstale":"2024-04-16T16:22:48.426384+0000","last_undegraded":"2024-04-16T16:22:48.426384+0000","last_fullsized":"2024-04-16T16:22:48.426384+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:01.883490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426458+0000","last_change":"2024-04-16T16:22:46.476179+0000","last_active":"2024-04-16T16:22:48.426458+0000","last_peered":"2024-04-16T16:22:48.426458+0000","last_clean":"2024-04-16T16:22:48.426458+0000","last_became_active":"2024-04-16T16:22:46.475898+0000","last_became_peered":"2024-04-16T16:22:46.475898+0000","last_unstale":"2024-04-16T16:22:48.426458+0000","last_undegraded":"2024-04-16T16:22:48.426458+0000","last_fullsized":"2024-04-16T16:22:48.426458+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:37.006661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624317+0000","last_change":"2024-04-16T16:22:46.482186+0000","last_active":"2024-04-16T16:22:48.624317+0000","last_peered":"2024-04-16T16:22:48.624317+0000","last_clean":"2024-04-16T16:22:48.624317+0000","last_became_active":"2024-04-16T16:22:46.481776+0000","last_became_peered":"2024-04-16T16:22:46.481776+0000","last_unstale":"2024-04-16T16:22:48.624317+0000","last_undegraded":"2024-04-16T16:22:48.624317+0000","last_fullsized":"2024-04-16T16:22:48.624317+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:32:25.664832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623895+0000","last_change":"2024-04-16T16:22:46.477861+0000","last_active":"2024-04-16T16:22:48.623895+0000","last_peered":"2024-04-16T16:22:48.623895+0000","last_clean":"2024-04-16T16:22:48.623895+0000","last_became_active":"2024-04-16T16:22:46.476486+0000","last_became_peered":"2024-04-16T16:22:46.476486+0000","last_unstale":"2024-04-16T16:22:48.623895+0000","last_undegraded":"2024-04-16T16:22:48.623895+0000","last_fullsized":"2024-04-16T16:22:48.623895+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:08.580218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460332+0000","last_change":"2024-04-16T16:22:46.468319+0000","last_active":"2024-04-16T16:22:47.460332+0000","last_peered":"2024-04-16T16:22:47.460332+0000","last_clean":"2024-04-16T16:22:47.460332+0000","last_became_active":"2024-04-16T16:22:46.463117+0000","last_became_peered":"2024-04-16T16:22:46.463117+0000","last_unstale":"2024-04-16T16:22:47.460332+0000","last_undegraded":"2024-04-16T16:22:47.460332+0000","last_fullsized":"2024-04-16T16:22:47.460332+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:12:58.505334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635241+0000","last_change":"2024-04-16T16:22:46.467824+0000","last_active":"2024-04-16T16:22:48.635241+0000","last_peered":"2024-04-16T16:22:48.635241+0000","last_clean":"2024-04-16T16:22:48.635241+0000","last_became_active":"2024-04-16T16:22:46.467274+0000","last_became_peered":"2024-04-16T16:22:46.467274+0000","last_unstale":"2024-04-16T16:22:48.635241+0000","last_undegraded":"2024-04-16T16:22:48.635241+0000","last_fullsized":"2024-04-16T16:22:48.635241+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:50:29.718907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710486+0000","last_change":"2024-04-16T16:22:46.475037+0000","last_active":"2024-04-16T16:22:48.710486+0000","last_peered":"2024-04-16T16:22:48.710486+0000","last_clean":"2024-04-16T16:22:48.710486+0000","last_became_active":"2024-04-16T16:22:46.474757+0000","last_became_peered":"2024-04-16T16:22:46.474757+0000","last_unstale":"2024-04-16T16:22:48.710486+0000","last_undegraded":"2024-04-16T16:22:48.710486+0000","last_fullsized":"2024-04-16T16:22:48.710486+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:31:51.916998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457334+0000","last_change":"2024-04-16T16:22:46.463139+0000","last_active":"2024-04-16T16:22:47.457334+0000","last_peered":"2024-04-16T16:22:47.457334+0000","last_clean":"2024-04-16T16:22:47.457334+0000","last_became_active":"2024-04-16T16:22:46.462857+0000","last_became_peered":"2024-04-16T16:22:46.462857+0000","last_unstale":"2024-04-16T16:22:47.457334+0000","last_undegraded":"2024-04-16T16:22:47.457334+0000","last_fullsized":"2024-04-16T16:22:47.457334+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:43:19.877365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710482+0000","last_change":"2024-04-16T16:22:46.457033+0000","last_active":"2024-04-16T16:22:48.710482+0000","last_peered":"2024-04-16T16:22:48.710482+0000","last_clean":"2024-04-16T16:22:48.710482+0000","last_became_active":"2024-04-16T16:22:46.456756+0000","last_became_peered":"2024-04-16T16:22:46.456756+0000","last_unstale":"2024-04-16T16:22:48.710482+0000","last_undegraded":"2024-04-16T16:22:48.710482+0000","last_fullsized":"2024-04-16T16:22:48.710482+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:18:13.904540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624366+0000","last_change":"2024-04-16T16:22:46.482119+0000","last_active":"2024-04-16T16:22:48.624366+0000","last_peered":"2024-04-16T16:22:48.624366+0000","last_clean":"2024-04-16T16:22:48.624366+0000","last_became_active":"2024-04-16T16:22:46.479857+0000","last_became_peered":"2024-04-16T16:22:46.479857+0000","last_unstale":"2024-04-16T16:22:48.624366+0000","last_undegraded":"2024-04-16T16:22:48.624366+0000","last_fullsized":"2024-04-16T16:22:48.624366+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:05:36.568682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710264+0000","last_change":"2024-04-16T16:22:46.479160+0000","last_active":"2024-04-16T16:22:48.710264+0000","last_peered":"2024-04-16T16:22:48.710264+0000","last_clean":"2024-04-16T16:22:48.710264+0000","last_became_active":"2024-04-16T16:22:46.477081+0000","last_became_peered":"2024-04-16T16:22:46.477081+0000","last_unstale":"2024-04-16T16:22:48.710264+0000","last_undegraded":"2024-04-16T16:22:48.710264+0000","last_fullsized":"2024-04-16T16:22:48.710264+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:31:11.372535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426469+0000","last_change":"2024-04-16T16:22:46.473896+0000","last_active":"2024-04-16T16:22:48.426469+0000","last_peered":"2024-04-16T16:22:48.426469+0000","last_clean":"2024-04-16T16:22:48.426469+0000","last_became_active":"2024-04-16T16:22:46.473621+0000","last_became_peered":"2024-04-16T16:22:46.473621+0000","last_unstale":"2024-04-16T16:22:48.426469+0000","last_undegraded":"2024-04-16T16:22:48.426469+0000","last_fullsized":"2024-04-16T16:22:48.426469+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:26.785078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710026+0000","last_change":"2024-04-16T16:22:46.475807+0000","last_active":"2024-04-16T16:22:48.710026+0000","last_peered":"2024-04-16T16:22:48.710026+0000","last_clean":"2024-04-16T16:22:48.710026+0000","last_became_active":"2024-04-16T16:22:46.473183+0000","last_became_peered":"2024-04-16T16:22:46.473183+0000","last_unstale":"2024-04-16T16:22:48.710026+0000","last_undegraded":"2024-04-16T16:22:48.710026+0000","last_fullsized":"2024-04-16T16:22:48.710026+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:26.889339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426404+0000","last_change":"2024-04-16T16:22:46.477670+0000","last_active":"2024-04-16T16:22:48.426404+0000","last_peered":"2024-04-16T16:22:48.426404+0000","last_clean":"2024-04-16T16:22:48.426404+0000","last_became_active":"2024-04-16T16:22:46.477303+0000","last_became_peered":"2024-04-16T16:22:46.477303+0000","last_unstale":"2024-04-16T16:22:48.426404+0000","last_undegraded":"2024-04-16T16:22:48.426404+0000","last_fullsized":"2024-04-16T16:22:48.426404+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:02:37.502947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.625109+0000","last_change":"2024-04-16T16:22:46.481055+0000","last_active":"2024-04-16T16:22:48.625109+0000","last_peered":"2024-04-16T16:22:48.625109+0000","last_clean":"2024-04-16T16:22:48.625109+0000","last_became_active":"2024-04-16T16:22:46.480641+0000","last_became_peered":"2024-04-16T16:22:46.480641+0000","last_unstale":"2024-04-16T16:22:48.625109+0000","last_undegraded":"2024-04-16T16:22:48.625109+0000","last_fullsized":"2024-04-16T16:22:48.625109+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:53:06.786966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635707+0000","last_change":"2024-04-16T16:22:46.467282+0000","last_active":"2024-04-16T16:22:48.635707+0000","last_peered":"2024-04-16T16:22:48.635707+0000","last_clean":"2024-04-16T16:22:48.635707+0000","last_became_active":"2024-04-16T16:22:46.466756+0000","last_became_peered":"2024-04-16T16:22:46.466756+0000","last_unstale":"2024-04-16T16:22:48.635707+0000","last_undegraded":"2024-04-16T16:22:48.635707+0000","last_fullsized":"2024-04-16T16:22:48.635707+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:18.072160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624687+0000","last_change":"2024-04-16T16:22:46.484066+0000","last_active":"2024-04-16T16:22:48.624687+0000","last_peered":"2024-04-16T16:22:48.624687+0000","last_clean":"2024-04-16T16:22:48.624687+0000","last_became_active":"2024-04-16T16:22:46.483654+0000","last_became_peered":"2024-04-16T16:22:46.483654+0000","last_unstale":"2024-04-16T16:22:48.624687+0000","last_undegraded":"2024-04-16T16:22:48.624687+0000","last_fullsized":"2024-04-16T16:22:48.624687+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:44:40.829325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077771+0000","last_change":"2024-04-16T16:22:46.484111+0000","last_active":"2024-04-16T16:22:48.077771+0000","last_peered":"2024-04-16T16:22:48.077771+0000","last_clean":"2024-04-16T16:22:48.077771+0000","last_became_active":"2024-04-16T16:22:46.483750+0000","last_became_peered":"2024-04-16T16:22:46.483750+0000","last_unstale":"2024-04-16T16:22:48.077771+0000","last_undegraded":"2024-04-16T16:22:48.077771+0000","last_fullsized":"2024-04-16T16:22:48.077771+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:02:37.502947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.625014+0000","last_change":"2024-04-16T16:22:46.483339+0000","last_active":"2024-04-16T16:22:48.625014+0000","last_peered":"2024-04-16T16:22:48.625014+0000","last_clean":"2024-04-16T16:22:48.625014+0000","last_became_active":"2024-04-16T16:22:46.481358+0000","last_became_peered":"2024-04-16T16:22:46.481358+0000","last_unstale":"2024-04-16T16:22:48.625014+0000","last_undegraded":"2024-04-16T16:22:48.625014+0000","last_fullsized":"2024-04-16T16:22:48.625014+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:13.979255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461048+0000","last_change":"2024-04-16T16:22:46.464311+0000","last_active":"2024-04-16T16:22:47.461048+0000","last_peered":"2024-04-16T16:22:47.461048+0000","last_clean":"2024-04-16T16:22:47.461048+0000","last_became_active":"2024-04-16T16:22:46.463536+0000","last_became_peered":"2024-04-16T16:22:46.463536+0000","last_unstale":"2024-04-16T16:22:47.461048+0000","last_undegraded":"2024-04-16T16:22:47.461048+0000","last_fullsized":"2024-04-16T16:22:47.461048+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:36:30.108257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460440+0000","last_change":"2024-04-16T16:22:46.460441+0000","last_active":"2024-04-16T16:22:47.460440+0000","last_peered":"2024-04-16T16:22:47.460440+0000","last_clean":"2024-04-16T16:22:47.460440+0000","last_became_active":"2024-04-16T16:22:46.458962+0000","last_became_peered":"2024-04-16T16:22:46.458962+0000","last_unstale":"2024-04-16T16:22:47.460440+0000","last_undegraded":"2024-04-16T16:22:47.460440+0000","last_fullsized":"2024-04-16T16:22:47.460440+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:35.010598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.456651+0000","last_change":"2024-04-16T16:22:46.468514+0000","last_active":"2024-04-16T16:22:47.456651+0000","last_peered":"2024-04-16T16:22:47.456651+0000","last_clean":"2024-04-16T16:22:47.456651+0000","last_became_active":"2024-04-16T16:22:46.464851+0000","last_became_peered":"2024-04-16T16:22:46.464851+0000","last_unstale":"2024-04-16T16:22:47.456651+0000","last_undegraded":"2024-04-16T16:22:47.456651+0000","last_fullsized":"2024-04-16T16:22:47.456651+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:59:47.867717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635932+0000","last_change":"2024-04-16T16:22:46.467039+0000","last_active":"2024-04-16T16:22:48.635932+0000","last_peered":"2024-04-16T16:22:48.635932+0000","last_clean":"2024-04-16T16:22:48.635932+0000","last_became_active":"2024-04-16T16:22:46.466366+0000","last_became_peered":"2024-04-16T16:22:46.466366+0000","last_unstale":"2024-04-16T16:22:48.635932+0000","last_undegraded":"2024-04-16T16:22:48.635932+0000","last_fullsized":"2024-04-16T16:22:48.635932+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:54:26.843618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"155'4","reported_seq":18,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461935+0000","last_change":"2024-04-16T16:22:45.470039+0000","last_active":"2024-04-16T16:22:47.461935+0000","last_peered":"2024-04-16T16:22:47.461935+0000","last_clean":"2024-04-16T16:22:47.461935+0000","last_became_active":"2024-04-16T16:22:45.467695+0000","last_became_peered":"2024-04-16T16:22:45.467695+0000","last_unstale":"2024-04-16T16:22:47.461935+0000","last_undegraded":"2024-04-16T16:22:47.461935+0000","last_fullsized":"2024-04-16T16:22:47.461935+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:21.824987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463917+0000","last_change":"2024-04-16T16:22:46.466378+0000","last_active":"2024-04-16T16:22:47.463917+0000","last_peered":"2024-04-16T16:22:47.463917+0000","last_clean":"2024-04-16T16:22:47.463917+0000","last_became_active":"2024-04-16T16:22:46.462099+0000","last_became_peered":"2024-04-16T16:22:46.462099+0000","last_unstale":"2024-04-16T16:22:47.463917+0000","last_undegraded":"2024-04-16T16:22:47.463917+0000","last_fullsized":"2024-04-16T16:22:47.463917+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:02:37.502947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463627+0000","last_change":"2024-04-16T16:22:45.464685+0000","last_active":"2024-04-16T16:22:47.463627+0000","last_peered":"2024-04-16T16:22:47.463627+0000","last_clean":"2024-04-16T16:22:47.463627+0000","last_became_active":"2024-04-16T16:22:45.464379+0000","last_became_peered":"2024-04-16T16:22:45.464379+0000","last_unstale":"2024-04-16T16:22:47.463627+0000","last_undegraded":"2024-04-16T16:22:47.463627+0000","last_fullsized":"2024-04-16T16:22:47.463627+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:05:35.552664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710111+0000","last_change":"2024-04-16T16:22:46.456302+0000","last_active":"2024-04-16T16:22:48.710111+0000","last_peered":"2024-04-16T16:22:48.710111+0000","last_clean":"2024-04-16T16:22:48.710111+0000","last_became_active":"2024-04-16T16:22:46.454610+0000","last_became_peered":"2024-04-16T16:22:46.454610+0000","last_unstale":"2024-04-16T16:22:48.710111+0000","last_undegraded":"2024-04-16T16:22:48.710111+0000","last_fullsized":"2024-04-16T16:22:48.710111+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:56:12.143605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077380+0000","last_change":"2024-04-16T16:22:45.469877+0000","last_active":"2024-04-16T16:22:48.077380+0000","last_peered":"2024-04-16T16:22:48.077380+0000","last_clean":"2024-04-16T16:22:48.077380+0000","last_became_active":"2024-04-16T16:22:45.469537+0000","last_became_peered":"2024-04-16T16:22:45.469537+0000","last_unstale":"2024-04-16T16:22:48.077380+0000","last_undegraded":"2024-04-16T16:22:48.077380+0000","last_fullsized":"2024-04-16T16:22:48.077380+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:54.078259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624791+0000","last_change":"2024-04-16T16:22:46.480073+0000","last_active":"2024-04-16T16:22:48.624791+0000","last_peered":"2024-04-16T16:22:48.624791+0000","last_clean":"2024-04-16T16:22:48.624791+0000","last_became_active":"2024-04-16T16:22:46.479206+0000","last_became_peered":"2024-04-16T16:22:46.479206+0000","last_unstale":"2024-04-16T16:22:48.624791+0000","last_undegraded":"2024-04-16T16:22:48.624791+0000","last_fullsized":"2024-04-16T16:22:48.624791+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:27.266104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.078056+0000","last_change":"2024-04-16T16:22:45.462903+0000","last_active":"2024-04-16T16:22:48.078056+0000","last_peered":"2024-04-16T16:22:48.078056+0000","last_clean":"2024-04-16T16:22:48.078056+0000","last_became_active":"2024-04-16T16:22:45.462170+0000","last_became_peered":"2024-04-16T16:22:45.462170+0000","last_unstale":"2024-04-16T16:22:48.078056+0000","last_undegraded":"2024-04-16T16:22:48.078056+0000","last_fullsized":"2024-04-16T16:22:48.078056+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:07.564200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635999+0000","last_change":"2024-04-16T16:22:46.460202+0000","last_active":"2024-04-16T16:22:48.635999+0000","last_peered":"2024-04-16T16:22:48.635999+0000","last_clean":"2024-04-16T16:22:48.635999+0000","last_became_active":"2024-04-16T16:22:46.459876+0000","last_became_peered":"2024-04-16T16:22:46.459876+0000","last_unstale":"2024-04-16T16:22:48.635999+0000","last_undegraded":"2024-04-16T16:22:48.635999+0000","last_fullsized":"2024-04-16T16:22:48.635999+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:28:07.667824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624602+0000","last_change":"2024-04-16T16:22:45.472425+0000","last_active":"2024-04-16T16:22:48.624602+0000","last_peered":"2024-04-16T16:22:48.624602+0000","last_clean":"2024-04-16T16:22:48.624602+0000","last_became_active":"2024-04-16T16:22:45.472020+0000","last_became_peered":"2024-04-16T16:22:45.472020+0000","last_unstale":"2024-04-16T16:22:48.624602+0000","last_undegraded":"2024-04-16T16:22:48.624602+0000","last_fullsized":"2024-04-16T16:22:48.624602+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:48:51.793606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077212+0000","last_change":"2024-04-16T16:22:46.470665+0000","last_active":"2024-04-16T16:22:48.077212+0000","last_peered":"2024-04-16T16:22:48.077212+0000","last_clean":"2024-04-16T16:22:48.077212+0000","last_became_active":"2024-04-16T16:22:46.467898+0000","last_became_peered":"2024-04-16T16:22:46.467898+0000","last_unstale":"2024-04-16T16:22:48.077212+0000","last_undegraded":"2024-04-16T16:22:48.077212+0000","last_fullsized":"2024-04-16T16:22:48.077212+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:35.010598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077383+0000","last_change":"2024-04-16T16:22:45.462429+0000","last_active":"2024-04-16T16:22:48.077383+0000","last_peered":"2024-04-16T16:22:48.077383+0000","last_clean":"2024-04-16T16:22:48.077383+0000","last_became_active":"2024-04-16T16:22:45.461674+0000","last_became_peered":"2024-04-16T16:22:45.461674+0000","last_unstale":"2024-04-16T16:22:48.077383+0000","last_undegraded":"2024-04-16T16:22:48.077383+0000","last_fullsized":"2024-04-16T16:22:48.077383+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:29.386319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.076898+0000","last_change":"2024-04-16T16:22:46.465672+0000","last_active":"2024-04-16T16:22:48.076898+0000","last_peered":"2024-04-16T16:22:48.076898+0000","last_clean":"2024-04-16T16:22:48.076898+0000","last_became_active":"2024-04-16T16:22:46.465329+0000","last_became_peered":"2024-04-16T16:22:46.465329+0000","last_unstale":"2024-04-16T16:22:48.076898+0000","last_undegraded":"2024-04-16T16:22:48.076898+0000","last_fullsized":"2024-04-16T16:22:48.076898+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:36:42.468881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.425887+0000","last_change":"2024-04-16T16:22:45.466868+0000","last_active":"2024-04-16T16:22:48.425887+0000","last_peered":"2024-04-16T16:22:48.425887+0000","last_clean":"2024-04-16T16:22:48.425887+0000","last_became_active":"2024-04-16T16:22:45.466591+0000","last_became_peered":"2024-04-16T16:22:45.466591+0000","last_unstale":"2024-04-16T16:22:48.425887+0000","last_undegraded":"2024-04-16T16:22:48.425887+0000","last_fullsized":"2024-04-16T16:22:48.425887+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:25.557203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710073+0000","last_change":"2024-04-16T16:22:46.460701+0000","last_active":"2024-04-16T16:22:48.710073+0000","last_peered":"2024-04-16T16:22:48.710073+0000","last_clean":"2024-04-16T16:22:48.710073+0000","last_became_active":"2024-04-16T16:22:46.460420+0000","last_became_peered":"2024-04-16T16:22:46.460420+0000","last_unstale":"2024-04-16T16:22:48.710073+0000","last_undegraded":"2024-04-16T16:22:48.710073+0000","last_fullsized":"2024-04-16T16:22:48.710073+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:35:00.579428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635706+0000","last_change":"2024-04-16T16:22:45.456041+0000","last_active":"2024-04-16T16:22:48.635706+0000","last_peered":"2024-04-16T16:22:48.635706+0000","last_clean":"2024-04-16T16:22:48.635706+0000","last_became_active":"2024-04-16T16:22:45.455767+0000","last_became_peered":"2024-04-16T16:22:45.455767+0000","last_unstale":"2024-04-16T16:22:48.635706+0000","last_undegraded":"2024-04-16T16:22:48.635706+0000","last_fullsized":"2024-04-16T16:22:48.635706+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:26:37.373159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457377+0000","last_change":"2024-04-16T16:22:46.467743+0000","last_active":"2024-04-16T16:22:47.457377+0000","last_peered":"2024-04-16T16:22:47.457377+0000","last_clean":"2024-04-16T16:22:47.457377+0000","last_became_active":"2024-04-16T16:22:46.465149+0000","last_became_peered":"2024-04-16T16:22:46.465149+0000","last_unstale":"2024-04-16T16:22:47.457377+0000","last_undegraded":"2024-04-16T16:22:47.457377+0000","last_fullsized":"2024-04-16T16:22:47.457377+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:18.072160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710644+0000","last_change":"2024-04-16T16:22:45.467985+0000","last_active":"2024-04-16T16:22:48.710644+0000","last_peered":"2024-04-16T16:22:48.710644+0000","last_clean":"2024-04-16T16:22:48.710644+0000","last_became_active":"2024-04-16T16:22:45.467704+0000","last_became_peered":"2024-04-16T16:22:45.467704+0000","last_unstale":"2024-04-16T16:22:48.710644+0000","last_undegraded":"2024-04-16T16:22:48.710644+0000","last_fullsized":"2024-04-16T16:22:48.710644+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:58.753768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457126+0000","last_change":"2024-04-16T16:22:46.473841+0000","last_active":"2024-04-16T16:22:47.457126+0000","last_peered":"2024-04-16T16:22:47.457126+0000","last_clean":"2024-04-16T16:22:47.457126+0000","last_became_active":"2024-04-16T16:22:46.473500+0000","last_became_peered":"2024-04-16T16:22:46.473500+0000","last_unstale":"2024-04-16T16:22:47.457126+0000","last_undegraded":"2024-04-16T16:22:47.457126+0000","last_fullsized":"2024-04-16T16:22:47.457126+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:50:29.718907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"155'3","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710356+0000","last_change":"2024-04-16T16:22:45.461376+0000","last_active":"2024-04-16T16:22:48.710356+0000","last_peered":"2024-04-16T16:22:48.710356+0000","last_clean":"2024-04-16T16:22:48.710356+0000","last_became_active":"2024-04-16T16:22:45.461088+0000","last_became_peered":"2024-04-16T16:22:45.461088+0000","last_unstale":"2024-04-16T16:22:48.710356+0000","last_undegraded":"2024-04-16T16:22:48.710356+0000","last_fullsized":"2024-04-16T16:22:48.710356+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:43:07.885858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.464703+0000","last_change":"2024-04-16T16:22:46.473263+0000","last_active":"2024-04-16T16:22:47.464703+0000","last_peered":"2024-04-16T16:22:47.464703+0000","last_clean":"2024-04-16T16:22:47.464703+0000","last_became_active":"2024-04-16T16:22:46.472979+0000","last_became_peered":"2024-04-16T16:22:46.472979+0000","last_unstale":"2024-04-16T16:22:47.464703+0000","last_undegraded":"2024-04-16T16:22:47.464703+0000","last_fullsized":"2024-04-16T16:22:47.464703+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:37.006661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"155'6","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463168+0000","last_change":"2024-04-16T16:22:45.462782+0000","last_active":"2024-04-16T16:22:47.463168+0000","last_peered":"2024-04-16T16:22:47.463168+0000","last_clean":"2024-04-16T16:22:47.463168+0000","last_became_active":"2024-04-16T16:22:45.461583+0000","last_became_peered":"2024-04-16T16:22:45.461583+0000","last_unstale":"2024-04-16T16:22:47.463168+0000","last_undegraded":"2024-04-16T16:22:47.463168+0000","last_fullsized":"2024-04-16T16:22:47.463168+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.456686+0000","last_change":"2024-04-16T16:22:46.466170+0000","last_active":"2024-04-16T16:22:47.456686+0000","last_peered":"2024-04-16T16:22:47.456686+0000","last_clean":"2024-04-16T16:22:47.456686+0000","last_became_active":"2024-04-16T16:22:46.465894+0000","last_became_peered":"2024-04-16T16:22:46.465894+0000","last_unstale":"2024-04-16T16:22:47.456686+0000","last_undegraded":"2024-04-16T16:22:47.456686+0000","last_fullsized":"2024-04-16T16:22:47.456686+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:01.225311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077288+0000","last_change":"2024-04-16T16:22:45.466273+0000","last_active":"2024-04-16T16:22:48.077288+0000","last_peered":"2024-04-16T16:22:48.077288+0000","last_clean":"2024-04-16T16:22:48.077288+0000","last_became_active":"2024-04-16T16:22:45.465893+0000","last_became_peered":"2024-04-16T16:22:45.465893+0000","last_unstale":"2024-04-16T16:22:48.077288+0000","last_undegraded":"2024-04-16T16:22:48.077288+0000","last_fullsized":"2024-04-16T16:22:48.077288+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:52:57.403249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.636259+0000","last_change":"2024-04-16T16:22:46.474507+0000","last_active":"2024-04-16T16:22:48.636259+0000","last_peered":"2024-04-16T16:22:48.636259+0000","last_clean":"2024-04-16T16:22:48.636259+0000","last_became_active":"2024-04-16T16:22:46.474146+0000","last_became_peered":"2024-04-16T16:22:46.474146+0000","last_unstale":"2024-04-16T16:22:48.636259+0000","last_undegraded":"2024-04-16T16:22:48.636259+0000","last_fullsized":"2024-04-16T16:22:48.636259+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:25:59.259590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.456803+0000","last_change":"2024-04-16T16:22:45.463747+0000","last_active":"2024-04-16T16:22:47.456803+0000","last_peered":"2024-04-16T16:22:47.456803+0000","last_clean":"2024-04-16T16:22:47.456803+0000","last_became_active":"2024-04-16T16:22:45.462875+0000","last_became_peered":"2024-04-16T16:22:45.462875+0000","last_unstale":"2024-04-16T16:22:47.456803+0000","last_undegraded":"2024-04-16T16:22:47.456803+0000","last_fullsized":"2024-04-16T16:22:47.456803+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:29:33.964433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.076960+0000","last_change":"2024-04-16T16:22:46.473128+0000","last_active":"2024-04-16T16:22:48.076960+0000","last_peered":"2024-04-16T16:22:48.076960+0000","last_clean":"2024-04-16T16:22:48.076960+0000","last_became_active":"2024-04-16T16:22:46.472859+0000","last_became_peered":"2024-04-16T16:22:46.472859+0000","last_unstale":"2024-04-16T16:22:48.076960+0000","last_undegraded":"2024-04-16T16:22:48.076960+0000","last_fullsized":"2024-04-16T16:22:48.076960+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:36:30.108257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.076910+0000","last_change":"2024-04-16T16:22:45.466908+0000","last_active":"2024-04-16T16:22:48.076910+0000","last_peered":"2024-04-16T16:22:48.076910+0000","last_clean":"2024-04-16T16:22:48.076910+0000","last_became_active":"2024-04-16T16:22:45.466535+0000","last_became_peered":"2024-04-16T16:22:45.466535+0000","last_unstale":"2024-04-16T16:22:48.076910+0000","last_undegraded":"2024-04-16T16:22:48.076910+0000","last_fullsized":"2024-04-16T16:22:48.076910+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:25.557203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623664+0000","last_change":"2024-04-16T16:22:46.483216+0000","last_active":"2024-04-16T16:22:48.623664+0000","last_peered":"2024-04-16T16:22:48.623664+0000","last_clean":"2024-04-16T16:22:48.623664+0000","last_became_active":"2024-04-16T16:22:46.482943+0000","last_became_peered":"2024-04-16T16:22:46.482943+0000","last_unstale":"2024-04-16T16:22:48.623664+0000","last_undegraded":"2024-04-16T16:22:48.623664+0000","last_fullsized":"2024-04-16T16:22:48.623664+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:26.573221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"155'8","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460436+0000","last_change":"2024-04-16T16:22:45.461068+0000","last_active":"2024-04-16T16:22:47.460436+0000","last_peered":"2024-04-16T16:22:47.460436+0000","last_clean":"2024-04-16T16:22:47.460436+0000","last_became_active":"2024-04-16T16:22:45.459760+0000","last_became_peered":"2024-04-16T16:22:45.459760+0000","last_unstale":"2024-04-16T16:22:47.460436+0000","last_undegraded":"2024-04-16T16:22:47.460436+0000","last_fullsized":"2024-04-16T16:22:47.460436+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:32:24.648814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.078301+0000","last_change":"2024-04-16T16:22:46.473695+0000","last_active":"2024-04-16T16:22:48.078301+0000","last_peered":"2024-04-16T16:22:48.078301+0000","last_clean":"2024-04-16T16:22:48.078301+0000","last_became_active":"2024-04-16T16:22:46.471894+0000","last_became_peered":"2024-04-16T16:22:46.471894+0000","last_unstale":"2024-04-16T16:22:48.078301+0000","last_undegraded":"2024-04-16T16:22:48.078301+0000","last_fullsized":"2024-04-16T16:22:48.078301+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:26.785078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.625205+0000","last_change":"2024-04-16T16:22:45.469335+0000","last_active":"2024-04-16T16:22:48.625205+0000","last_peered":"2024-04-16T16:22:48.625205+0000","last_clean":"2024-04-16T16:22:48.625205+0000","last_became_active":"2024-04-16T16:22:45.468966+0000","last_became_peered":"2024-04-16T16:22:45.468966+0000","last_unstale":"2024-04-16T16:22:48.625205+0000","last_undegraded":"2024-04-16T16:22:48.625205+0000","last_fullsized":"2024-04-16T16:22:48.625205+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:09.300484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457513+0000","last_change":"2024-04-16T16:22:46.471925+0000","last_active":"2024-04-16T16:22:47.457513+0000","last_peered":"2024-04-16T16:22:47.457513+0000","last_clean":"2024-04-16T16:22:47.457513+0000","last_became_active":"2024-04-16T16:22:46.470405+0000","last_became_peered":"2024-04-16T16:22:46.470405+0000","last_unstale":"2024-04-16T16:22:47.457513+0000","last_undegraded":"2024-04-16T16:22:47.457513+0000","last_fullsized":"2024-04-16T16:22:47.457513+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:04.570708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624612+0000","last_change":"2024-04-16T16:22:45.470272+0000","last_active":"2024-04-16T16:22:48.624612+0000","last_peered":"2024-04-16T16:22:48.624612+0000","last_clean":"2024-04-16T16:22:48.624612+0000","last_became_active":"2024-04-16T16:22:45.469959+0000","last_became_peered":"2024-04-16T16:22:45.469959+0000","last_unstale":"2024-04-16T16:22:48.624612+0000","last_undegraded":"2024-04-16T16:22:48.624612+0000","last_fullsized":"2024-04-16T16:22:48.624612+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:34:03.813486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.078101+0000","last_change":"2024-04-16T16:22:46.462951+0000","last_active":"2024-04-16T16:22:48.078101+0000","last_peered":"2024-04-16T16:22:48.078101+0000","last_clean":"2024-04-16T16:22:48.078101+0000","last_became_active":"2024-04-16T16:22:46.462662+0000","last_became_peered":"2024-04-16T16:22:46.462662+0000","last_unstale":"2024-04-16T16:22:48.078101+0000","last_undegraded":"2024-04-16T16:22:48.078101+0000","last_fullsized":"2024-04-16T16:22:48.078101+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:37.006661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462044+0000","last_change":"2024-04-16T16:22:45.462862+0000","last_active":"2024-04-16T16:22:47.462044+0000","last_peered":"2024-04-16T16:22:47.462044+0000","last_clean":"2024-04-16T16:22:47.462044+0000","last_became_active":"2024-04-16T16:22:45.462197+0000","last_became_peered":"2024-04-16T16:22:45.462197+0000","last_unstale":"2024-04-16T16:22:47.462044+0000","last_undegraded":"2024-04-16T16:22:47.462044+0000","last_fullsized":"2024-04-16T16:22:47.462044+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:55.697931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.636077+0000","last_change":"2024-04-16T16:22:46.475567+0000","last_active":"2024-04-16T16:22:48.636077+0000","last_peered":"2024-04-16T16:22:48.636077+0000","last_clean":"2024-04-16T16:22:48.636077+0000","last_became_active":"2024-04-16T16:22:46.475232+0000","last_became_peered":"2024-04-16T16:22:46.475232+0000","last_unstale":"2024-04-16T16:22:48.636077+0000","last_undegraded":"2024-04-16T16:22:48.636077+0000","last_fullsized":"2024-04-16T16:22:48.636077+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:19:43.312174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462377+0000","last_change":"2024-04-16T16:22:45.466893+0000","last_active":"2024-04-16T16:22:47.462377+0000","last_peered":"2024-04-16T16:22:47.462377+0000","last_clean":"2024-04-16T16:22:47.462377+0000","last_became_active":"2024-04-16T16:22:45.466614+0000","last_became_peered":"2024-04-16T16:22:45.466614+0000","last_unstale":"2024-04-16T16:22:47.462377+0000","last_undegraded":"2024-04-16T16:22:47.462377+0000","last_fullsized":"2024-04-16T16:22:47.462377+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:53:47.150015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623905+0000","last_change":"2024-04-16T16:22:46.482036+0000","last_active":"2024-04-16T16:22:48.623905+0000","last_peered":"2024-04-16T16:22:48.623905+0000","last_clean":"2024-04-16T16:22:48.623905+0000","last_became_active":"2024-04-16T16:22:46.481293+0000","last_became_peered":"2024-04-16T16:22:46.481293+0000","last_unstale":"2024-04-16T16:22:48.623905+0000","last_undegraded":"2024-04-16T16:22:48.623905+0000","last_fullsized":"2024-04-16T16:22:48.623905+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:55.094277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077501+0000","last_change":"2024-04-16T16:22:45.466947+0000","last_active":"2024-04-16T16:22:48.077501+0000","last_peered":"2024-04-16T16:22:48.077501+0000","last_clean":"2024-04-16T16:22:48.077501+0000","last_became_active":"2024-04-16T16:22:45.466624+0000","last_became_peered":"2024-04-16T16:22:45.466624+0000","last_unstale":"2024-04-16T16:22:48.077501+0000","last_undegraded":"2024-04-16T16:22:48.077501+0000","last_fullsized":"2024-04-16T16:22:48.077501+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:53.044305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.636145+0000","last_change":"2024-04-16T16:22:46.469862+0000","last_active":"2024-04-16T16:22:48.636145+0000","last_peered":"2024-04-16T16:22:48.636145+0000","last_clean":"2024-04-16T16:22:48.636145+0000","last_became_active":"2024-04-16T16:22:46.466179+0000","last_became_peered":"2024-04-16T16:22:46.466179+0000","last_unstale":"2024-04-16T16:22:48.636145+0000","last_undegraded":"2024-04-16T16:22:48.636145+0000","last_fullsized":"2024-04-16T16:22:48.636145+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:36:56.658760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460683+0000","last_change":"2024-04-16T16:22:45.461564+0000","last_active":"2024-04-16T16:22:47.460683+0000","last_peered":"2024-04-16T16:22:47.460683+0000","last_clean":"2024-04-16T16:22:47.460683+0000","last_became_active":"2024-04-16T16:22:45.461253+0000","last_became_peered":"2024-04-16T16:22:45.461253+0000","last_unstale":"2024-04-16T16:22:47.460683+0000","last_undegraded":"2024-04-16T16:22:47.460683+0000","last_fullsized":"2024-04-16T16:22:47.460683+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:46.018568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462392+0000","last_change":"2024-04-16T16:22:46.465947+0000","last_active":"2024-04-16T16:22:47.462392+0000","last_peered":"2024-04-16T16:22:47.462392+0000","last_clean":"2024-04-16T16:22:47.462392+0000","last_became_active":"2024-04-16T16:22:46.465294+0000","last_became_peered":"2024-04-16T16:22:46.465294+0000","last_unstale":"2024-04-16T16:22:47.462392+0000","last_undegraded":"2024-04-16T16:22:47.462392+0000","last_fullsized":"2024-04-16T16:22:47.462392+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:29.789672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077036+0000","last_change":"2024-04-16T16:22:45.468883+0000","last_active":"2024-04-16T16:22:48.077036+0000","last_peered":"2024-04-16T16:22:48.077036+0000","last_clean":"2024-04-16T16:22:48.077036+0000","last_became_active":"2024-04-16T16:22:45.468091+0000","last_became_peered":"2024-04-16T16:22:45.468091+0000","last_unstale":"2024-04-16T16:22:48.077036+0000","last_undegraded":"2024-04-16T16:22:48.077036+0000","last_fullsized":"2024-04-16T16:22:48.077036+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710495+0000","last_change":"2024-04-16T16:22:46.471354+0000","last_active":"2024-04-16T16:22:48.710495+0000","last_peered":"2024-04-16T16:22:48.710495+0000","last_clean":"2024-04-16T16:22:48.710495+0000","last_became_active":"2024-04-16T16:22:46.470521+0000","last_became_peered":"2024-04-16T16:22:46.470521+0000","last_unstale":"2024-04-16T16:22:48.710495+0000","last_undegraded":"2024-04-16T16:22:48.710495+0000","last_fullsized":"2024-04-16T16:22:48.710495+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:06:08.736226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461147+0000","last_change":"2024-04-16T16:22:45.460865+0000","last_active":"2024-04-16T16:22:47.461147+0000","last_peered":"2024-04-16T16:22:47.461147+0000","last_clean":"2024-04-16T16:22:47.461147+0000","last_became_active":"2024-04-16T16:22:45.458861+0000","last_became_peered":"2024-04-16T16:22:45.458861+0000","last_unstale":"2024-04-16T16:22:47.461147+0000","last_undegraded":"2024-04-16T16:22:47.461147+0000","last_fullsized":"2024-04-16T16:22:47.461147+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:30.462179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457529+0000","last_change":"2024-04-16T16:22:46.464764+0000","last_active":"2024-04-16T16:22:47.457529+0000","last_peered":"2024-04-16T16:22:47.457529+0000","last_clean":"2024-04-16T16:22:47.457529+0000","last_became_active":"2024-04-16T16:22:46.464064+0000","last_became_peered":"2024-04-16T16:22:46.464064+0000","last_unstale":"2024-04-16T16:22:47.457529+0000","last_undegraded":"2024-04-16T16:22:47.457529+0000","last_fullsized":"2024-04-16T16:22:47.457529+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:28:07.667824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457501+0000","last_change":"2024-04-16T16:22:45.466044+0000","last_active":"2024-04-16T16:22:47.457501+0000","last_peered":"2024-04-16T16:22:47.457501+0000","last_clean":"2024-04-16T16:22:47.457501+0000","last_became_active":"2024-04-16T16:22:45.465676+0000","last_became_peered":"2024-04-16T16:22:45.465676+0000","last_unstale":"2024-04-16T16:22:47.457501+0000","last_undegraded":"2024-04-16T16:22:47.457501+0000","last_fullsized":"2024-04-16T16:22:47.457501+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:52.385757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635973+0000","last_change":"2024-04-16T16:22:46.466356+0000","last_active":"2024-04-16T16:22:48.635973+0000","last_peered":"2024-04-16T16:22:48.635973+0000","last_clean":"2024-04-16T16:22:48.635973+0000","last_became_active":"2024-04-16T16:22:46.465449+0000","last_became_peered":"2024-04-16T16:22:46.465449+0000","last_unstale":"2024-04-16T16:22:48.635973+0000","last_undegraded":"2024-04-16T16:22:48.635973+0000","last_fullsized":"2024-04-16T16:22:48.635973+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:04.570708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624728+0000","last_change":"2024-04-16T16:22:45.466156+0000","last_active":"2024-04-16T16:22:48.624728+0000","last_peered":"2024-04-16T16:22:48.624728+0000","last_clean":"2024-04-16T16:22:48.624728+0000","last_became_active":"2024-04-16T16:22:45.465791+0000","last_became_peered":"2024-04-16T16:22:45.465791+0000","last_unstale":"2024-04-16T16:22:48.624728+0000","last_undegraded":"2024-04-16T16:22:48.624728+0000","last_fullsized":"2024-04-16T16:22:48.624728+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:42:46.394979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710309+0000","last_change":"2024-04-16T16:22:46.470854+0000","last_active":"2024-04-16T16:22:48.710309+0000","last_peered":"2024-04-16T16:22:48.710309+0000","last_clean":"2024-04-16T16:22:48.710309+0000","last_became_active":"2024-04-16T16:22:46.470196+0000","last_became_peered":"2024-04-16T16:22:46.470196+0000","last_unstale":"2024-04-16T16:22:48.710309+0000","last_undegraded":"2024-04-16T16:22:48.710309+0000","last_fullsized":"2024-04-16T16:22:48.710309+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:40:24.085281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426634+0000","last_change":"2024-04-16T16:22:45.471205+0000","last_active":"2024-04-16T16:22:48.426634+0000","last_peered":"2024-04-16T16:22:48.426634+0000","last_clean":"2024-04-16T16:22:48.426634+0000","last_became_active":"2024-04-16T16:22:45.470911+0000","last_became_peered":"2024-04-16T16:22:45.470911+0000","last_unstale":"2024-04-16T16:22:48.426634+0000","last_undegraded":"2024-04-16T16:22:48.426634+0000","last_fullsized":"2024-04-16T16:22:48.426634+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:53.044305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457242+0000","last_change":"2024-04-16T16:22:46.469324+0000","last_active":"2024-04-16T16:22:47.457242+0000","last_peered":"2024-04-16T16:22:47.457242+0000","last_clean":"2024-04-16T16:22:47.457242+0000","last_became_active":"2024-04-16T16:22:46.469009+0000","last_became_peered":"2024-04-16T16:22:46.469009+0000","last_unstale":"2024-04-16T16:22:47.457242+0000","last_undegraded":"2024-04-16T16:22:47.457242+0000","last_fullsized":"2024-04-16T16:22:47.457242+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:29.789672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457098+0000","last_change":"2024-04-16T16:22:45.468189+0000","last_active":"2024-04-16T16:22:47.457098+0000","last_peered":"2024-04-16T16:22:47.457098+0000","last_clean":"2024-04-16T16:22:47.457098+0000","last_became_active":"2024-04-16T16:22:45.467709+0000","last_became_peered":"2024-04-16T16:22:45.467709+0000","last_unstale":"2024-04-16T16:22:47.457098+0000","last_undegraded":"2024-04-16T16:22:47.457098+0000","last_fullsized":"2024-04-16T16:22:47.457098+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:46.256132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.456756+0000","last_change":"2024-04-16T16:22:46.465994+0000","last_active":"2024-04-16T16:22:47.456756+0000","last_peered":"2024-04-16T16:22:47.456756+0000","last_clean":"2024-04-16T16:22:47.456756+0000","last_became_active":"2024-04-16T16:22:46.465007+0000","last_became_peered":"2024-04-16T16:22:46.465007+0000","last_unstale":"2024-04-16T16:22:47.456756+0000","last_undegraded":"2024-04-16T16:22:47.456756+0000","last_fullsized":"2024-04-16T16:22:47.456756+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:12:58.505334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462996+0000","last_change":"2024-04-16T16:22:45.470607+0000","last_active":"2024-04-16T16:22:47.462996+0000","last_peered":"2024-04-16T16:22:47.462996+0000","last_clean":"2024-04-16T16:22:47.462996+0000","last_became_active":"2024-04-16T16:22:45.469322+0000","last_became_peered":"2024-04-16T16:22:45.469322+0000","last_unstale":"2024-04-16T16:22:47.462996+0000","last_undegraded":"2024-04-16T16:22:47.462996+0000","last_fullsized":"2024-04-16T16:22:47.462996+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:07.564200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463072+0000","last_change":"2024-04-16T16:22:46.462483+0000","last_active":"2024-04-16T16:22:47.463072+0000","last_peered":"2024-04-16T16:22:47.463072+0000","last_clean":"2024-04-16T16:22:47.463072+0000","last_became_active":"2024-04-16T16:22:46.461432+0000","last_became_peered":"2024-04-16T16:22:46.461432+0000","last_unstale":"2024-04-16T16:22:47.463072+0000","last_undegraded":"2024-04-16T16:22:47.463072+0000","last_fullsized":"2024-04-16T16:22:47.463072+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:01.883490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077289+0000","last_change":"2024-04-16T16:22:45.462752+0000","last_active":"2024-04-16T16:22:48.077289+0000","last_peered":"2024-04-16T16:22:48.077289+0000","last_clean":"2024-04-16T16:22:48.077289+0000","last_became_active":"2024-04-16T16:22:45.462154+0000","last_became_peered":"2024-04-16T16:22:45.462154+0000","last_unstale":"2024-04-16T16:22:48.077289+0000","last_undegraded":"2024-04-16T16:22:48.077289+0000","last_fullsized":"2024-04-16T16:22:48.077289+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:43:04.134472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635368+0000","last_change":"2024-04-16T16:22:46.467575+0000","last_active":"2024-04-16T16:22:48.635368+0000","last_peered":"2024-04-16T16:22:48.635368+0000","last_clean":"2024-04-16T16:22:48.635368+0000","last_became_active":"2024-04-16T16:22:46.466890+0000","last_became_peered":"2024-04-16T16:22:46.466890+0000","last_unstale":"2024-04-16T16:22:48.635368+0000","last_undegraded":"2024-04-16T16:22:48.635368+0000","last_fullsized":"2024-04-16T16:22:48.635368+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:43:19.877365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426288+0000","last_change":"2024-04-16T16:22:45.469689+0000","last_active":"2024-04-16T16:22:48.426288+0000","last_peered":"2024-04-16T16:22:48.426288+0000","last_clean":"2024-04-16T16:22:48.426288+0000","last_became_active":"2024-04-16T16:22:45.469350+0000","last_became_peered":"2024-04-16T16:22:45.469350+0000","last_unstale":"2024-04-16T16:22:48.426288+0000","last_undegraded":"2024-04-16T16:22:48.426288+0000","last_fullsized":"2024-04-16T16:22:48.426288+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623691+0000","last_change":"2024-04-16T16:22:46.478730+0000","last_active":"2024-04-16T16:22:48.623691+0000","last_peered":"2024-04-16T16:22:48.623691+0000","last_clean":"2024-04-16T16:22:48.623691+0000","last_became_active":"2024-04-16T16:22:46.478459+0000","last_became_peered":"2024-04-16T16:22:46.478459+0000","last_unstale":"2024-04-16T16:22:48.623691+0000","last_undegraded":"2024-04-16T16:22:48.623691+0000","last_fullsized":"2024-04-16T16:22:48.623691+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:36.777723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460788+0000","last_change":"2024-04-16T16:22:45.467505+0000","last_active":"2024-04-16T16:22:47.460788+0000","last_peered":"2024-04-16T16:22:47.460788+0000","last_clean":"2024-04-16T16:22:47.460788+0000","last_became_active":"2024-04-16T16:22:45.465339+0000","last_became_peered":"2024-04-16T16:22:45.465339+0000","last_unstale":"2024-04-16T16:22:47.460788+0000","last_undegraded":"2024-04-16T16:22:47.460788+0000","last_fullsized":"2024-04-16T16:22:47.460788+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:54.078259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457379+0000","last_change":"2024-04-16T16:22:46.465023+0000","last_active":"2024-04-16T16:22:47.457379+0000","last_peered":"2024-04-16T16:22:47.457379+0000","last_clean":"2024-04-16T16:22:47.457379+0000","last_became_active":"2024-04-16T16:22:46.464436+0000","last_became_peered":"2024-04-16T16:22:46.464436+0000","last_unstale":"2024-04-16T16:22:47.457379+0000","last_undegraded":"2024-04-16T16:22:47.457379+0000","last_fullsized":"2024-04-16T16:22:47.457379+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:54:26.843618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463842+0000","last_change":"2024-04-16T16:22:45.461515+0000","last_active":"2024-04-16T16:22:47.463842+0000","last_peered":"2024-04-16T16:22:47.463842+0000","last_clean":"2024-04-16T16:22:47.463842+0000","last_became_active":"2024-04-16T16:22:45.460350+0000","last_became_peered":"2024-04-16T16:22:45.460350+0000","last_unstale":"2024-04-16T16:22:47.463842+0000","last_undegraded":"2024-04-16T16:22:47.463842+0000","last_fullsized":"2024-04-16T16:22:47.463842+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:34.292891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"144'4691","reported_seq":5390,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462677+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:22:47.462677+0000","last_peered":"2024-04-16T16:22:47.462677+0000","last_clean":"2024-04-16T16:22:47.462677+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:22:47.462677+0000","last_undegraded":"2024-04-16T16:22:47.462677+0000","last_fullsized":"2024-04-16T16:22:47.462677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":4691,"log_dups_size":0,"ondisk_log_size":4691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":387,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":387,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4691,"ondisk_log_size":4691,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574957,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37856,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333984,"statfs":{"total":96636764160,"available":96597999616,"internally_reserved":0,"allocated":2576384,"data_stored":1526220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574958,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1961984,"data_stored":933520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574958,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":934102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574959,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33116,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338724,"statfs":{"total":96636764160,"available":96602853376,"internally_reserved":0,"allocated":1974272,"data_stored":935180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574957,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37264,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334576,"statfs":{"total":96636764160,"available":96598605824,"internally_reserved":0,"allocated":1961984,"data_stored":933532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574957,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37256,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334584,"statfs":{"total":96636764160,"available":96598614016,"internally_reserved":0,"allocated":1961984,"data_stored":933588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37256,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334584,"statfs":{"total":96636764160,"available":96598614016,"internally_reserved":0,"allocated":1961984,"data_stored":933532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574958,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37832,"kb_used_data":2492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334008,"statfs":{"total":96636764160,"available":96598024192,"internally_reserved":0,"allocated":2551808,"data_stored":1523866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:22:55.477 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-16T16:22:55.477 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:22:55.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.746+0000 7f7689600640 1 Processor -- start 2024-04-16T16:22:55.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.746+0000 7f7689600640 1 -- start start 2024-04-16T16:22:55.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.748+0000 7f7689600640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f768415ccb0 0x7f768406df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:55.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.748+0000 7f7689600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f768406e480 0x7f76841608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:55.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.748+0000 7f7689600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f768415d360 con 0x7f7684146220 2024-04-16T16:22:55.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.748+0000 7f7689600640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f768415d130 con 0x7f768415ccb0 2024-04-16T16:22:55.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.748+0000 7f7689600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7684160410 con 0x7f768406e480 2024-04-16T16:22:55.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.748+0000 7f767b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f768415ccb0 0x7f768406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:55.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.748+0000 7f767b400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f768415ccb0 0x7f768406df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47250/0 (socket says 172.21.15.45:47250) 2024-04-16T16:22:55.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.748+0000 7f767b400640 1 -- 172.21.15.45:0/2376715591 learned_addr learned my addr 172.21.15.45:0/2376715591 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:55.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.749+0000 7f767b400640 1 -- 172.21.15.45:0/2376715591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f768406e480 msgr2=0x7f76841608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:55.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.749+0000 7f7688c00640 1 --2- 172.21.15.45:0/2376715591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f768406e480 0x7f76841608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:55.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.749+0000 7f767b400640 1 --2- 172.21.15.45:0/2376715591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f768406e480 0x7f76841608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.749+0000 7f767b400640 1 -- 172.21.15.45:0/2376715591 >> v1:172.21.15.45:6789/0 conn(0x7f7684146220 legacy=0x7f768406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:55.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.749+0000 7f767b400640 1 -- 172.21.15.45:0/2376715591 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f768415d640 con 0x7f768415ccb0 2024-04-16T16:22:55.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.749+0000 7f7688c00640 1 --2- 172.21.15.45:0/2376715591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f768406e480 0x7f76841608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:55.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.750+0000 7f767b400640 1 --2- 172.21.15.45:0/2376715591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768415ccb0 0x7f768406df40 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f766c009870 tx=0x7f766c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a57863e4700e15e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:55.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.750+0000 7f7683400640 1 -- 172.21.15.45:0/2376715591 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766c034070 con 0x7f768415ccb0 2024-04-16T16:22:55.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.751+0000 7f7683400640 1 -- 172.21.15.45:0/2376715591 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f766c030840 con 0x7f768415ccb0 2024-04-16T16:22:55.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.751+0000 7f7683400640 1 -- 172.21.15.45:0/2376715591 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766c030c10 con 0x7f768415ccb0 2024-04-16T16:22:55.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.751+0000 7f7689600640 1 -- 172.21.15.45:0/2376715591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768415ccb0 msgr2=0x7f768406df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:55.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.751+0000 7f7689600640 1 --2- 172.21.15.45:0/2376715591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768415ccb0 0x7f768406df40 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f766c009870 tx=0x7f766c02b980 comp rx=0 tx=0).stop 2024-04-16T16:22:55.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.751+0000 7f7689600640 1 -- 172.21.15.45:0/2376715591 shutdown_connections 2024-04-16T16:22:55.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.751+0000 7f7689600640 1 --2- 172.21.15.45:0/2376715591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f768406e480 0x7f76841608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.751+0000 7f7689600640 1 --2- 172.21.15.45:0/2376715591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768415ccb0 0x7f768406df40 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.751+0000 7f7689600640 1 -- 172.21.15.45:0/2376715591 >> 172.21.15.45:0/2376715591 conn(0x7f768406b930 msgr2=0x7f768406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:55.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.751+0000 7f7689600640 1 -- 172.21.15.45:0/2376715591 shutdown_connections 2024-04-16T16:22:55.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.751+0000 7f7689600640 1 -- 172.21.15.45:0/2376715591 wait complete. 2024-04-16T16:22:55.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.753+0000 7f7689600640 1 Processor -- start 2024-04-16T16:22:55.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.753+0000 7f7689600640 1 -- start start 2024-04-16T16:22:55.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.754+0000 7f7689600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768406e480 0x7f768415c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:55.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.754+0000 7f7689600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7684146220 0x7f76840c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:55.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.754+0000 7f7689600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f768415ccb0 0x7f76840cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:55.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.754+0000 7f7689600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7684160de0 con 0x7f768406e480 2024-04-16T16:22:55.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.754+0000 7f7689600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f768415dc00 con 0x7f7684146220 2024-04-16T16:22:55.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.754+0000 7f7689600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7684160f60 con 0x7f768415ccb0 2024-04-16T16:22:55.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.755+0000 7f7683e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768406e480 0x7f768415c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:55.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.755+0000 7f7683e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768406e480 0x7f768415c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47264/0 (socket says 172.21.15.45:47264) 2024-04-16T16:22:55.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.755+0000 7f7683e00640 1 -- 172.21.15.45:0/3418483171 learned_addr learned my addr 172.21.15.45:0/3418483171 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:55.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.755+0000 7f7688c00640 1 --2- 172.21.15.45:0/3418483171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f768415ccb0 0x7f76840cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:55.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.755+0000 7f767b400640 1 --2- 172.21.15.45:0/3418483171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7684146220 0x7f76840c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:55.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.755+0000 7f7683e00640 1 -- 172.21.15.45:0/3418483171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f768415ccb0 msgr2=0x7f76840cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:55.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.755+0000 7f7683e00640 1 --2- 172.21.15.45:0/3418483171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f768415ccb0 0x7f76840cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.755+0000 7f7683e00640 1 -- 172.21.15.45:0/3418483171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7684146220 msgr2=0x7f76840c93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:55.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.755+0000 7f7683e00640 1 --2- 172.21.15.45:0/3418483171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7684146220 0x7f76840c93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.755+0000 7f7683e00640 1 -- 172.21.15.45:0/3418483171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76840c9d30 con 0x7f768406e480 2024-04-16T16:22:55.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.755+0000 7f7683e00640 1 --2- 172.21.15.45:0/3418483171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768406e480 0x7f768415c0f0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f767400c7c0 tx=0x7f767400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:55.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.756+0000 7f7680c00640 1 -- 172.21.15.45:0/3418483171 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7674016020 con 0x7f768406e480 2024-04-16T16:22:55.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.756+0000 7f7680c00640 1 -- 172.21.15.45:0/3418483171 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7674006d50 con 0x7f768406e480 2024-04-16T16:22:55.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.756+0000 7f7680c00640 1 -- 172.21.15.45:0/3418483171 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7674003bb0 con 0x7f768406e480 2024-04-16T16:22:55.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.756+0000 7f7689600640 1 -- 172.21.15.45:0/3418483171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76840ca010 con 0x7f768406e480 2024-04-16T16:22:55.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.756+0000 7f7689600640 1 -- 172.21.15.45:0/3418483171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76841438f0 con 0x7f768406e480 2024-04-16T16:22:55.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.757+0000 7f7689600640 1 -- 172.21.15.45:0/3418483171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7684075780 con 0x7f768406e480 2024-04-16T16:22:55.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.758+0000 7f7680c00640 1 -- 172.21.15.45:0/3418483171 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f767400a850 con 0x7f768406e480 2024-04-16T16:22:55.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.759+0000 7f7680c00640 1 --2- 172.21.15.45:0/3418483171 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f766006ca10 0x7f766006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:55.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.759+0000 7f7680c00640 1 -- 172.21.15.45:0/3418483171 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f767408ba30 con 0x7f768406e480 2024-04-16T16:22:55.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.760+0000 7f767b400640 1 --2- 172.21.15.45:0/3418483171 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f766006ca10 0x7f766006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:55.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.761+0000 7f767b400640 1 --2- 172.21.15.45:0/3418483171 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f766006ca10 0x7f766006eed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f766c036000 tx=0x7f766c008030 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:55.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.764+0000 7f7680c00640 1 -- 172.21.15.45:0/3418483171 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7674020020 con 0x7f768406e480 2024-04-16T16:22:55.907 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:22:55.915 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:22:55.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.970+0000 7f7689600640 1 -- 172.21.15.45:0/3418483171 --> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f76840b4ba0 con 0x7f766006ca10 2024-04-16T16:22:55.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.980+0000 7f7680c00640 1 -- 172.21.15.45:0/3418483171 <== mgr.7591 v2:172.21.15.103:6832/1129061436 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252497 (secure 0 0 0) 0x7f76840b4ba0 con 0x7f766006ca10 2024-04-16T16:22:55.982 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:55.983 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:22:55.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.985+0000 7f7689600640 1 -- 172.21.15.45:0/3418483171 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f766006ca10 msgr2=0x7f766006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:55.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.985+0000 7f7689600640 1 --2- 172.21.15.45:0/3418483171 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f766006ca10 0x7f766006eed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f766c036000 tx=0x7f766c008030 comp rx=0 tx=0).stop 2024-04-16T16:22:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.985+0000 7f7689600640 1 -- 172.21.15.45:0/3418483171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768406e480 msgr2=0x7f768415c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.985+0000 7f7689600640 1 --2- 172.21.15.45:0/3418483171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768406e480 0x7f768415c0f0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f767400c7c0 tx=0x7f767400cc80 comp rx=0 tx=0).stop 2024-04-16T16:22:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.985+0000 7f7689600640 1 -- 172.21.15.45:0/3418483171 shutdown_connections 2024-04-16T16:22:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.985+0000 7f7689600640 1 --2- 172.21.15.45:0/3418483171 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f766006ca10 0x7f766006eed0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.985+0000 7f7689600640 1 --2- 172.21.15.45:0/3418483171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f768415ccb0 0x7f76840cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.985+0000 7f7689600640 1 --2- 172.21.15.45:0/3418483171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7684146220 0x7f76840c93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.985+0000 7f7689600640 1 --2- 172.21.15.45:0/3418483171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768406e480 0x7f768415c0f0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.985+0000 7f7689600640 1 -- 172.21.15.45:0/3418483171 >> 172.21.15.45:0/3418483171 conn(0x7f768406b930 msgr2=0x7f768415f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.986+0000 7f7689600640 1 -- 172.21.15.45:0/3418483171 shutdown_connections 2024-04-16T16:22:55.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:55.986+0000 7f7689600640 1 -- 172.21.15.45:0/3418483171 wait complete. 2024-04-16T16:22:56.003 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-04-16T16:22:55.543946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":431,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4729,"ondisk_log_size":4729,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":286776,"kb_used_data":16520,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754687944,"statfs":{"total":773094113280,"available":772800454656,"internally_reserved":0,"allocated":16916480,"data_stored":8653540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":276614704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.105888"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461972+0000","last_change":"2024-04-16T16:22:46.488567+0000","last_active":"2024-04-16T16:22:47.461972+0000","last_peered":"2024-04-16T16:22:47.461972+0000","last_clean":"2024-04-16T16:22:47.461972+0000","last_became_active":"2024-04-16T16:22:46.485690+0000","last_became_peered":"2024-04-16T16:22:46.485690+0000","last_unstale":"2024-04-16T16:22:47.461972+0000","last_undegraded":"2024-04-16T16:22:47.461972+0000","last_fullsized":"2024-04-16T16:22:47.461972+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:50:29.718907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426342+0000","last_change":"2024-04-16T16:22:46.479620+0000","last_active":"2024-04-16T16:22:48.426342+0000","last_peered":"2024-04-16T16:22:48.426342+0000","last_clean":"2024-04-16T16:22:48.426342+0000","last_became_active":"2024-04-16T16:22:46.477410+0000","last_became_peered":"2024-04-16T16:22:46.477410+0000","last_unstale":"2024-04-16T16:22:48.426342+0000","last_undegraded":"2024-04-16T16:22:48.426342+0000","last_fullsized":"2024-04-16T16:22:48.426342+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:35.010598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624440+0000","last_change":"2024-04-16T16:22:46.480944+0000","last_active":"2024-04-16T16:22:48.624440+0000","last_peered":"2024-04-16T16:22:48.624440+0000","last_clean":"2024-04-16T16:22:48.624440+0000","last_became_active":"2024-04-16T16:22:46.480405+0000","last_became_peered":"2024-04-16T16:22:46.480405+0000","last_unstale":"2024-04-16T16:22:48.624440+0000","last_undegraded":"2024-04-16T16:22:48.624440+0000","last_fullsized":"2024-04-16T16:22:48.624440+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:35.308909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077640+0000","last_change":"2024-04-16T16:22:46.477832+0000","last_active":"2024-04-16T16:22:48.077640+0000","last_peered":"2024-04-16T16:22:48.077640+0000","last_clean":"2024-04-16T16:22:48.077640+0000","last_became_active":"2024-04-16T16:22:46.477516+0000","last_became_peered":"2024-04-16T16:22:46.477516+0000","last_unstale":"2024-04-16T16:22:48.077640+0000","last_undegraded":"2024-04-16T16:22:48.077640+0000","last_fullsized":"2024-04-16T16:22:48.077640+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:01.883490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426062+0000","last_change":"2024-04-16T16:22:46.483466+0000","last_active":"2024-04-16T16:22:48.426062+0000","last_peered":"2024-04-16T16:22:48.426062+0000","last_clean":"2024-04-16T16:22:48.426062+0000","last_became_active":"2024-04-16T16:22:46.481033+0000","last_became_peered":"2024-04-16T16:22:46.481033+0000","last_unstale":"2024-04-16T16:22:48.426062+0000","last_undegraded":"2024-04-16T16:22:48.426062+0000","last_fullsized":"2024-04-16T16:22:48.426062+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:36:30.108257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710150+0000","last_change":"2024-04-16T16:22:46.463298+0000","last_active":"2024-04-16T16:22:48.710150+0000","last_peered":"2024-04-16T16:22:48.710150+0000","last_clean":"2024-04-16T16:22:48.710150+0000","last_became_active":"2024-04-16T16:22:46.462017+0000","last_became_peered":"2024-04-16T16:22:46.462017+0000","last_unstale":"2024-04-16T16:22:48.710150+0000","last_undegraded":"2024-04-16T16:22:48.710150+0000","last_fullsized":"2024-04-16T16:22:48.710150+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:30:36.112336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635373+0000","last_change":"2024-04-16T16:22:46.462786+0000","last_active":"2024-04-16T16:22:48.635373+0000","last_peered":"2024-04-16T16:22:48.635373+0000","last_clean":"2024-04-16T16:22:48.635373+0000","last_became_active":"2024-04-16T16:22:46.461856+0000","last_became_peered":"2024-04-16T16:22:46.461856+0000","last_unstale":"2024-04-16T16:22:48.635373+0000","last_undegraded":"2024-04-16T16:22:48.635373+0000","last_fullsized":"2024-04-16T16:22:48.635373+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:29.789672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426662+0000","last_change":"2024-04-16T16:22:46.486917+0000","last_active":"2024-04-16T16:22:48.426662+0000","last_peered":"2024-04-16T16:22:48.426662+0000","last_clean":"2024-04-16T16:22:48.426662+0000","last_became_active":"2024-04-16T16:22:46.485280+0000","last_became_peered":"2024-04-16T16:22:46.485280+0000","last_unstale":"2024-04-16T16:22:48.426662+0000","last_undegraded":"2024-04-16T16:22:48.426662+0000","last_fullsized":"2024-04-16T16:22:48.426662+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:14:27.242053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.427106+0000","last_change":"2024-04-16T16:22:46.476513+0000","last_active":"2024-04-16T16:22:48.427106+0000","last_peered":"2024-04-16T16:22:48.427106+0000","last_clean":"2024-04-16T16:22:48.427106+0000","last_became_active":"2024-04-16T16:22:46.476125+0000","last_became_peered":"2024-04-16T16:22:46.476125+0000","last_unstale":"2024-04-16T16:22:48.427106+0000","last_undegraded":"2024-04-16T16:22:48.427106+0000","last_fullsized":"2024-04-16T16:22:48.427106+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:28.349945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426898+0000","last_change":"2024-04-16T16:22:46.483287+0000","last_active":"2024-04-16T16:22:48.426898+0000","last_peered":"2024-04-16T16:22:48.426898+0000","last_clean":"2024-04-16T16:22:48.426898+0000","last_became_active":"2024-04-16T16:22:46.482975+0000","last_became_peered":"2024-04-16T16:22:46.482975+0000","last_unstale":"2024-04-16T16:22:48.426898+0000","last_undegraded":"2024-04-16T16:22:48.426898+0000","last_fullsized":"2024-04-16T16:22:48.426898+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:21.784358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426384+0000","last_change":"2024-04-16T16:22:46.481301+0000","last_active":"2024-04-16T16:22:48.426384+0000","last_peered":"2024-04-16T16:22:48.426384+0000","last_clean":"2024-04-16T16:22:48.426384+0000","last_became_active":"2024-04-16T16:22:46.481000+0000","last_became_peered":"2024-04-16T16:22:46.481000+0000","last_unstale":"2024-04-16T16:22:48.426384+0000","last_undegraded":"2024-04-16T16:22:48.426384+0000","last_fullsized":"2024-04-16T16:22:48.426384+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:01.883490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426458+0000","last_change":"2024-04-16T16:22:46.476179+0000","last_active":"2024-04-16T16:22:48.426458+0000","last_peered":"2024-04-16T16:22:48.426458+0000","last_clean":"2024-04-16T16:22:48.426458+0000","last_became_active":"2024-04-16T16:22:46.475898+0000","last_became_peered":"2024-04-16T16:22:46.475898+0000","last_unstale":"2024-04-16T16:22:48.426458+0000","last_undegraded":"2024-04-16T16:22:48.426458+0000","last_fullsized":"2024-04-16T16:22:48.426458+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:37.006661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624317+0000","last_change":"2024-04-16T16:22:46.482186+0000","last_active":"2024-04-16T16:22:48.624317+0000","last_peered":"2024-04-16T16:22:48.624317+0000","last_clean":"2024-04-16T16:22:48.624317+0000","last_became_active":"2024-04-16T16:22:46.481776+0000","last_became_peered":"2024-04-16T16:22:46.481776+0000","last_unstale":"2024-04-16T16:22:48.624317+0000","last_undegraded":"2024-04-16T16:22:48.624317+0000","last_fullsized":"2024-04-16T16:22:48.624317+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:32:25.664832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623895+0000","last_change":"2024-04-16T16:22:46.477861+0000","last_active":"2024-04-16T16:22:48.623895+0000","last_peered":"2024-04-16T16:22:48.623895+0000","last_clean":"2024-04-16T16:22:48.623895+0000","last_became_active":"2024-04-16T16:22:46.476486+0000","last_became_peered":"2024-04-16T16:22:46.476486+0000","last_unstale":"2024-04-16T16:22:48.623895+0000","last_undegraded":"2024-04-16T16:22:48.623895+0000","last_fullsized":"2024-04-16T16:22:48.623895+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:08.580218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460332+0000","last_change":"2024-04-16T16:22:46.468319+0000","last_active":"2024-04-16T16:22:47.460332+0000","last_peered":"2024-04-16T16:22:47.460332+0000","last_clean":"2024-04-16T16:22:47.460332+0000","last_became_active":"2024-04-16T16:22:46.463117+0000","last_became_peered":"2024-04-16T16:22:46.463117+0000","last_unstale":"2024-04-16T16:22:47.460332+0000","last_undegraded":"2024-04-16T16:22:47.460332+0000","last_fullsized":"2024-04-16T16:22:47.460332+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:12:58.505334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635241+0000","last_change":"2024-04-16T16:22:46.467824+0000","last_active":"2024-04-16T16:22:48.635241+0000","last_peered":"2024-04-16T16:22:48.635241+0000","last_clean":"2024-04-16T16:22:48.635241+0000","last_became_active":"2024-04-16T16:22:46.467274+0000","last_became_peered":"2024-04-16T16:22:46.467274+0000","last_unstale":"2024-04-16T16:22:48.635241+0000","last_undegraded":"2024-04-16T16:22:48.635241+0000","last_fullsized":"2024-04-16T16:22:48.635241+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:50:29.718907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710486+0000","last_change":"2024-04-16T16:22:46.475037+0000","last_active":"2024-04-16T16:22:48.710486+0000","last_peered":"2024-04-16T16:22:48.710486+0000","last_clean":"2024-04-16T16:22:48.710486+0000","last_became_active":"2024-04-16T16:22:46.474757+0000","last_became_peered":"2024-04-16T16:22:46.474757+0000","last_unstale":"2024-04-16T16:22:48.710486+0000","last_undegraded":"2024-04-16T16:22:48.710486+0000","last_fullsized":"2024-04-16T16:22:48.710486+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:31:51.916998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457334+0000","last_change":"2024-04-16T16:22:46.463139+0000","last_active":"2024-04-16T16:22:47.457334+0000","last_peered":"2024-04-16T16:22:47.457334+0000","last_clean":"2024-04-16T16:22:47.457334+0000","last_became_active":"2024-04-16T16:22:46.462857+0000","last_became_peered":"2024-04-16T16:22:46.462857+0000","last_unstale":"2024-04-16T16:22:47.457334+0000","last_undegraded":"2024-04-16T16:22:47.457334+0000","last_fullsized":"2024-04-16T16:22:47.457334+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:43:19.877365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710482+0000","last_change":"2024-04-16T16:22:46.457033+0000","last_active":"2024-04-16T16:22:48.710482+0000","last_peered":"2024-04-16T16:22:48.710482+0000","last_clean":"2024-04-16T16:22:48.710482+0000","last_became_active":"2024-04-16T16:22:46.456756+0000","last_became_peered":"2024-04-16T16:22:46.456756+0000","last_unstale":"2024-04-16T16:22:48.710482+0000","last_undegraded":"2024-04-16T16:22:48.710482+0000","last_fullsized":"2024-04-16T16:22:48.710482+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:18:13.904540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624366+0000","last_change":"2024-04-16T16:22:46.482119+0000","last_active":"2024-04-16T16:22:48.624366+0000","last_peered":"2024-04-16T16:22:48.624366+0000","last_clean":"2024-04-16T16:22:48.624366+0000","last_became_active":"2024-04-16T16:22:46.479857+0000","last_became_peered":"2024-04-16T16:22:46.479857+0000","last_unstale":"2024-04-16T16:22:48.624366+0000","last_undegraded":"2024-04-16T16:22:48.624366+0000","last_fullsized":"2024-04-16T16:22:48.624366+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:05:36.568682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710264+0000","last_change":"2024-04-16T16:22:46.479160+0000","last_active":"2024-04-16T16:22:48.710264+0000","last_peered":"2024-04-16T16:22:48.710264+0000","last_clean":"2024-04-16T16:22:48.710264+0000","last_became_active":"2024-04-16T16:22:46.477081+0000","last_became_peered":"2024-04-16T16:22:46.477081+0000","last_unstale":"2024-04-16T16:22:48.710264+0000","last_undegraded":"2024-04-16T16:22:48.710264+0000","last_fullsized":"2024-04-16T16:22:48.710264+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:31:11.372535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426469+0000","last_change":"2024-04-16T16:22:46.473896+0000","last_active":"2024-04-16T16:22:48.426469+0000","last_peered":"2024-04-16T16:22:48.426469+0000","last_clean":"2024-04-16T16:22:48.426469+0000","last_became_active":"2024-04-16T16:22:46.473621+0000","last_became_peered":"2024-04-16T16:22:46.473621+0000","last_unstale":"2024-04-16T16:22:48.426469+0000","last_undegraded":"2024-04-16T16:22:48.426469+0000","last_fullsized":"2024-04-16T16:22:48.426469+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:26.785078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710026+0000","last_change":"2024-04-16T16:22:46.475807+0000","last_active":"2024-04-16T16:22:48.710026+0000","last_peered":"2024-04-16T16:22:48.710026+0000","last_clean":"2024-04-16T16:22:48.710026+0000","last_became_active":"2024-04-16T16:22:46.473183+0000","last_became_peered":"2024-04-16T16:22:46.473183+0000","last_unstale":"2024-04-16T16:22:48.710026+0000","last_undegraded":"2024-04-16T16:22:48.710026+0000","last_fullsized":"2024-04-16T16:22:48.710026+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:26.889339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426404+0000","last_change":"2024-04-16T16:22:46.477670+0000","last_active":"2024-04-16T16:22:48.426404+0000","last_peered":"2024-04-16T16:22:48.426404+0000","last_clean":"2024-04-16T16:22:48.426404+0000","last_became_active":"2024-04-16T16:22:46.477303+0000","last_became_peered":"2024-04-16T16:22:46.477303+0000","last_unstale":"2024-04-16T16:22:48.426404+0000","last_undegraded":"2024-04-16T16:22:48.426404+0000","last_fullsized":"2024-04-16T16:22:48.426404+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:02:37.502947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.625109+0000","last_change":"2024-04-16T16:22:46.481055+0000","last_active":"2024-04-16T16:22:48.625109+0000","last_peered":"2024-04-16T16:22:48.625109+0000","last_clean":"2024-04-16T16:22:48.625109+0000","last_became_active":"2024-04-16T16:22:46.480641+0000","last_became_peered":"2024-04-16T16:22:46.480641+0000","last_unstale":"2024-04-16T16:22:48.625109+0000","last_undegraded":"2024-04-16T16:22:48.625109+0000","last_fullsized":"2024-04-16T16:22:48.625109+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:53:06.786966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635707+0000","last_change":"2024-04-16T16:22:46.467282+0000","last_active":"2024-04-16T16:22:48.635707+0000","last_peered":"2024-04-16T16:22:48.635707+0000","last_clean":"2024-04-16T16:22:48.635707+0000","last_became_active":"2024-04-16T16:22:46.466756+0000","last_became_peered":"2024-04-16T16:22:46.466756+0000","last_unstale":"2024-04-16T16:22:48.635707+0000","last_undegraded":"2024-04-16T16:22:48.635707+0000","last_fullsized":"2024-04-16T16:22:48.635707+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:18.072160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624687+0000","last_change":"2024-04-16T16:22:46.484066+0000","last_active":"2024-04-16T16:22:48.624687+0000","last_peered":"2024-04-16T16:22:48.624687+0000","last_clean":"2024-04-16T16:22:48.624687+0000","last_became_active":"2024-04-16T16:22:46.483654+0000","last_became_peered":"2024-04-16T16:22:46.483654+0000","last_unstale":"2024-04-16T16:22:48.624687+0000","last_undegraded":"2024-04-16T16:22:48.624687+0000","last_fullsized":"2024-04-16T16:22:48.624687+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:44:40.829325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077771+0000","last_change":"2024-04-16T16:22:46.484111+0000","last_active":"2024-04-16T16:22:48.077771+0000","last_peered":"2024-04-16T16:22:48.077771+0000","last_clean":"2024-04-16T16:22:48.077771+0000","last_became_active":"2024-04-16T16:22:46.483750+0000","last_became_peered":"2024-04-16T16:22:46.483750+0000","last_unstale":"2024-04-16T16:22:48.077771+0000","last_undegraded":"2024-04-16T16:22:48.077771+0000","last_fullsized":"2024-04-16T16:22:48.077771+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:02:37.502947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.625014+0000","last_change":"2024-04-16T16:22:46.483339+0000","last_active":"2024-04-16T16:22:48.625014+0000","last_peered":"2024-04-16T16:22:48.625014+0000","last_clean":"2024-04-16T16:22:48.625014+0000","last_became_active":"2024-04-16T16:22:46.481358+0000","last_became_peered":"2024-04-16T16:22:46.481358+0000","last_unstale":"2024-04-16T16:22:48.625014+0000","last_undegraded":"2024-04-16T16:22:48.625014+0000","last_fullsized":"2024-04-16T16:22:48.625014+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:13.979255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461048+0000","last_change":"2024-04-16T16:22:46.464311+0000","last_active":"2024-04-16T16:22:47.461048+0000","last_peered":"2024-04-16T16:22:47.461048+0000","last_clean":"2024-04-16T16:22:47.461048+0000","last_became_active":"2024-04-16T16:22:46.463536+0000","last_became_peered":"2024-04-16T16:22:46.463536+0000","last_unstale":"2024-04-16T16:22:47.461048+0000","last_undegraded":"2024-04-16T16:22:47.461048+0000","last_fullsized":"2024-04-16T16:22:47.461048+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:36:30.108257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460440+0000","last_change":"2024-04-16T16:22:46.460441+0000","last_active":"2024-04-16T16:22:47.460440+0000","last_peered":"2024-04-16T16:22:47.460440+0000","last_clean":"2024-04-16T16:22:47.460440+0000","last_became_active":"2024-04-16T16:22:46.458962+0000","last_became_peered":"2024-04-16T16:22:46.458962+0000","last_unstale":"2024-04-16T16:22:47.460440+0000","last_undegraded":"2024-04-16T16:22:47.460440+0000","last_fullsized":"2024-04-16T16:22:47.460440+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:35.010598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.456651+0000","last_change":"2024-04-16T16:22:46.468514+0000","last_active":"2024-04-16T16:22:47.456651+0000","last_peered":"2024-04-16T16:22:47.456651+0000","last_clean":"2024-04-16T16:22:47.456651+0000","last_became_active":"2024-04-16T16:22:46.464851+0000","last_became_peered":"2024-04-16T16:22:46.464851+0000","last_unstale":"2024-04-16T16:22:47.456651+0000","last_undegraded":"2024-04-16T16:22:47.456651+0000","last_fullsized":"2024-04-16T16:22:47.456651+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:59:47.867717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635932+0000","last_change":"2024-04-16T16:22:46.467039+0000","last_active":"2024-04-16T16:22:48.635932+0000","last_peered":"2024-04-16T16:22:48.635932+0000","last_clean":"2024-04-16T16:22:48.635932+0000","last_became_active":"2024-04-16T16:22:46.466366+0000","last_became_peered":"2024-04-16T16:22:46.466366+0000","last_unstale":"2024-04-16T16:22:48.635932+0000","last_undegraded":"2024-04-16T16:22:48.635932+0000","last_fullsized":"2024-04-16T16:22:48.635932+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:54:26.843618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"155'4","reported_seq":18,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461935+0000","last_change":"2024-04-16T16:22:45.470039+0000","last_active":"2024-04-16T16:22:47.461935+0000","last_peered":"2024-04-16T16:22:47.461935+0000","last_clean":"2024-04-16T16:22:47.461935+0000","last_became_active":"2024-04-16T16:22:45.467695+0000","last_became_peered":"2024-04-16T16:22:45.467695+0000","last_unstale":"2024-04-16T16:22:47.461935+0000","last_undegraded":"2024-04-16T16:22:47.461935+0000","last_fullsized":"2024-04-16T16:22:47.461935+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:21.824987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463917+0000","last_change":"2024-04-16T16:22:46.466378+0000","last_active":"2024-04-16T16:22:47.463917+0000","last_peered":"2024-04-16T16:22:47.463917+0000","last_clean":"2024-04-16T16:22:47.463917+0000","last_became_active":"2024-04-16T16:22:46.462099+0000","last_became_peered":"2024-04-16T16:22:46.462099+0000","last_unstale":"2024-04-16T16:22:47.463917+0000","last_undegraded":"2024-04-16T16:22:47.463917+0000","last_fullsized":"2024-04-16T16:22:47.463917+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:02:37.502947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463627+0000","last_change":"2024-04-16T16:22:45.464685+0000","last_active":"2024-04-16T16:22:47.463627+0000","last_peered":"2024-04-16T16:22:47.463627+0000","last_clean":"2024-04-16T16:22:47.463627+0000","last_became_active":"2024-04-16T16:22:45.464379+0000","last_became_peered":"2024-04-16T16:22:45.464379+0000","last_unstale":"2024-04-16T16:22:47.463627+0000","last_undegraded":"2024-04-16T16:22:47.463627+0000","last_fullsized":"2024-04-16T16:22:47.463627+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:05:35.552664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710111+0000","last_change":"2024-04-16T16:22:46.456302+0000","last_active":"2024-04-16T16:22:48.710111+0000","last_peered":"2024-04-16T16:22:48.710111+0000","last_clean":"2024-04-16T16:22:48.710111+0000","last_became_active":"2024-04-16T16:22:46.454610+0000","last_became_peered":"2024-04-16T16:22:46.454610+0000","last_unstale":"2024-04-16T16:22:48.710111+0000","last_undegraded":"2024-04-16T16:22:48.710111+0000","last_fullsized":"2024-04-16T16:22:48.710111+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:56:12.143605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077380+0000","last_change":"2024-04-16T16:22:45.469877+0000","last_active":"2024-04-16T16:22:48.077380+0000","last_peered":"2024-04-16T16:22:48.077380+0000","last_clean":"2024-04-16T16:22:48.077380+0000","last_became_active":"2024-04-16T16:22:45.469537+0000","last_became_peered":"2024-04-16T16:22:45.469537+0000","last_unstale":"2024-04-16T16:22:48.077380+0000","last_undegraded":"2024-04-16T16:22:48.077380+0000","last_fullsized":"2024-04-16T16:22:48.077380+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:54.078259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624791+0000","last_change":"2024-04-16T16:22:46.480073+0000","last_active":"2024-04-16T16:22:48.624791+0000","last_peered":"2024-04-16T16:22:48.624791+0000","last_clean":"2024-04-16T16:22:48.624791+0000","last_became_active":"2024-04-16T16:22:46.479206+0000","last_became_peered":"2024-04-16T16:22:46.479206+0000","last_unstale":"2024-04-16T16:22:48.624791+0000","last_undegraded":"2024-04-16T16:22:48.624791+0000","last_fullsized":"2024-04-16T16:22:48.624791+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:27.266104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.078056+0000","last_change":"2024-04-16T16:22:45.462903+0000","last_active":"2024-04-16T16:22:48.078056+0000","last_peered":"2024-04-16T16:22:48.078056+0000","last_clean":"2024-04-16T16:22:48.078056+0000","last_became_active":"2024-04-16T16:22:45.462170+0000","last_became_peered":"2024-04-16T16:22:45.462170+0000","last_unstale":"2024-04-16T16:22:48.078056+0000","last_undegraded":"2024-04-16T16:22:48.078056+0000","last_fullsized":"2024-04-16T16:22:48.078056+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:07.564200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635999+0000","last_change":"2024-04-16T16:22:46.460202+0000","last_active":"2024-04-16T16:22:48.635999+0000","last_peered":"2024-04-16T16:22:48.635999+0000","last_clean":"2024-04-16T16:22:48.635999+0000","last_became_active":"2024-04-16T16:22:46.459876+0000","last_became_peered":"2024-04-16T16:22:46.459876+0000","last_unstale":"2024-04-16T16:22:48.635999+0000","last_undegraded":"2024-04-16T16:22:48.635999+0000","last_fullsized":"2024-04-16T16:22:48.635999+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:28:07.667824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624602+0000","last_change":"2024-04-16T16:22:45.472425+0000","last_active":"2024-04-16T16:22:48.624602+0000","last_peered":"2024-04-16T16:22:48.624602+0000","last_clean":"2024-04-16T16:22:48.624602+0000","last_became_active":"2024-04-16T16:22:45.472020+0000","last_became_peered":"2024-04-16T16:22:45.472020+0000","last_unstale":"2024-04-16T16:22:48.624602+0000","last_undegraded":"2024-04-16T16:22:48.624602+0000","last_fullsized":"2024-04-16T16:22:48.624602+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:48:51.793606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077212+0000","last_change":"2024-04-16T16:22:46.470665+0000","last_active":"2024-04-16T16:22:48.077212+0000","last_peered":"2024-04-16T16:22:48.077212+0000","last_clean":"2024-04-16T16:22:48.077212+0000","last_became_active":"2024-04-16T16:22:46.467898+0000","last_became_peered":"2024-04-16T16:22:46.467898+0000","last_unstale":"2024-04-16T16:22:48.077212+0000","last_undegraded":"2024-04-16T16:22:48.077212+0000","last_fullsized":"2024-04-16T16:22:48.077212+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:35.010598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077383+0000","last_change":"2024-04-16T16:22:45.462429+0000","last_active":"2024-04-16T16:22:48.077383+0000","last_peered":"2024-04-16T16:22:48.077383+0000","last_clean":"2024-04-16T16:22:48.077383+0000","last_became_active":"2024-04-16T16:22:45.461674+0000","last_became_peered":"2024-04-16T16:22:45.461674+0000","last_unstale":"2024-04-16T16:22:48.077383+0000","last_undegraded":"2024-04-16T16:22:48.077383+0000","last_fullsized":"2024-04-16T16:22:48.077383+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:08:29.386319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.076898+0000","last_change":"2024-04-16T16:22:46.465672+0000","last_active":"2024-04-16T16:22:48.076898+0000","last_peered":"2024-04-16T16:22:48.076898+0000","last_clean":"2024-04-16T16:22:48.076898+0000","last_became_active":"2024-04-16T16:22:46.465329+0000","last_became_peered":"2024-04-16T16:22:46.465329+0000","last_unstale":"2024-04-16T16:22:48.076898+0000","last_undegraded":"2024-04-16T16:22:48.076898+0000","last_fullsized":"2024-04-16T16:22:48.076898+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:36:42.468881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.425887+0000","last_change":"2024-04-16T16:22:45.466868+0000","last_active":"2024-04-16T16:22:48.425887+0000","last_peered":"2024-04-16T16:22:48.425887+0000","last_clean":"2024-04-16T16:22:48.425887+0000","last_became_active":"2024-04-16T16:22:45.466591+0000","last_became_peered":"2024-04-16T16:22:45.466591+0000","last_unstale":"2024-04-16T16:22:48.425887+0000","last_undegraded":"2024-04-16T16:22:48.425887+0000","last_fullsized":"2024-04-16T16:22:48.425887+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:25.557203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710073+0000","last_change":"2024-04-16T16:22:46.460701+0000","last_active":"2024-04-16T16:22:48.710073+0000","last_peered":"2024-04-16T16:22:48.710073+0000","last_clean":"2024-04-16T16:22:48.710073+0000","last_became_active":"2024-04-16T16:22:46.460420+0000","last_became_peered":"2024-04-16T16:22:46.460420+0000","last_unstale":"2024-04-16T16:22:48.710073+0000","last_undegraded":"2024-04-16T16:22:48.710073+0000","last_fullsized":"2024-04-16T16:22:48.710073+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:35:00.579428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635706+0000","last_change":"2024-04-16T16:22:45.456041+0000","last_active":"2024-04-16T16:22:48.635706+0000","last_peered":"2024-04-16T16:22:48.635706+0000","last_clean":"2024-04-16T16:22:48.635706+0000","last_became_active":"2024-04-16T16:22:45.455767+0000","last_became_peered":"2024-04-16T16:22:45.455767+0000","last_unstale":"2024-04-16T16:22:48.635706+0000","last_undegraded":"2024-04-16T16:22:48.635706+0000","last_fullsized":"2024-04-16T16:22:48.635706+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:26:37.373159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457377+0000","last_change":"2024-04-16T16:22:46.467743+0000","last_active":"2024-04-16T16:22:47.457377+0000","last_peered":"2024-04-16T16:22:47.457377+0000","last_clean":"2024-04-16T16:22:47.457377+0000","last_became_active":"2024-04-16T16:22:46.465149+0000","last_became_peered":"2024-04-16T16:22:46.465149+0000","last_unstale":"2024-04-16T16:22:47.457377+0000","last_undegraded":"2024-04-16T16:22:47.457377+0000","last_fullsized":"2024-04-16T16:22:47.457377+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:18.072160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710644+0000","last_change":"2024-04-16T16:22:45.467985+0000","last_active":"2024-04-16T16:22:48.710644+0000","last_peered":"2024-04-16T16:22:48.710644+0000","last_clean":"2024-04-16T16:22:48.710644+0000","last_became_active":"2024-04-16T16:22:45.467704+0000","last_became_peered":"2024-04-16T16:22:45.467704+0000","last_unstale":"2024-04-16T16:22:48.710644+0000","last_undegraded":"2024-04-16T16:22:48.710644+0000","last_fullsized":"2024-04-16T16:22:48.710644+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:58.753768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457126+0000","last_change":"2024-04-16T16:22:46.473841+0000","last_active":"2024-04-16T16:22:47.457126+0000","last_peered":"2024-04-16T16:22:47.457126+0000","last_clean":"2024-04-16T16:22:47.457126+0000","last_became_active":"2024-04-16T16:22:46.473500+0000","last_became_peered":"2024-04-16T16:22:46.473500+0000","last_unstale":"2024-04-16T16:22:47.457126+0000","last_undegraded":"2024-04-16T16:22:47.457126+0000","last_fullsized":"2024-04-16T16:22:47.457126+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:50:29.718907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"155'3","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710356+0000","last_change":"2024-04-16T16:22:45.461376+0000","last_active":"2024-04-16T16:22:48.710356+0000","last_peered":"2024-04-16T16:22:48.710356+0000","last_clean":"2024-04-16T16:22:48.710356+0000","last_became_active":"2024-04-16T16:22:45.461088+0000","last_became_peered":"2024-04-16T16:22:45.461088+0000","last_unstale":"2024-04-16T16:22:48.710356+0000","last_undegraded":"2024-04-16T16:22:48.710356+0000","last_fullsized":"2024-04-16T16:22:48.710356+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:43:07.885858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.464703+0000","last_change":"2024-04-16T16:22:46.473263+0000","last_active":"2024-04-16T16:22:47.464703+0000","last_peered":"2024-04-16T16:22:47.464703+0000","last_clean":"2024-04-16T16:22:47.464703+0000","last_became_active":"2024-04-16T16:22:46.472979+0000","last_became_peered":"2024-04-16T16:22:46.472979+0000","last_unstale":"2024-04-16T16:22:47.464703+0000","last_undegraded":"2024-04-16T16:22:47.464703+0000","last_fullsized":"2024-04-16T16:22:47.464703+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:37.006661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"155'6","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463168+0000","last_change":"2024-04-16T16:22:45.462782+0000","last_active":"2024-04-16T16:22:47.463168+0000","last_peered":"2024-04-16T16:22:47.463168+0000","last_clean":"2024-04-16T16:22:47.463168+0000","last_became_active":"2024-04-16T16:22:45.461583+0000","last_became_peered":"2024-04-16T16:22:45.461583+0000","last_unstale":"2024-04-16T16:22:47.463168+0000","last_undegraded":"2024-04-16T16:22:47.463168+0000","last_fullsized":"2024-04-16T16:22:47.463168+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.456686+0000","last_change":"2024-04-16T16:22:46.466170+0000","last_active":"2024-04-16T16:22:47.456686+0000","last_peered":"2024-04-16T16:22:47.456686+0000","last_clean":"2024-04-16T16:22:47.456686+0000","last_became_active":"2024-04-16T16:22:46.465894+0000","last_became_peered":"2024-04-16T16:22:46.465894+0000","last_unstale":"2024-04-16T16:22:47.456686+0000","last_undegraded":"2024-04-16T16:22:47.456686+0000","last_fullsized":"2024-04-16T16:22:47.456686+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:01.225311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077288+0000","last_change":"2024-04-16T16:22:45.466273+0000","last_active":"2024-04-16T16:22:48.077288+0000","last_peered":"2024-04-16T16:22:48.077288+0000","last_clean":"2024-04-16T16:22:48.077288+0000","last_became_active":"2024-04-16T16:22:45.465893+0000","last_became_peered":"2024-04-16T16:22:45.465893+0000","last_unstale":"2024-04-16T16:22:48.077288+0000","last_undegraded":"2024-04-16T16:22:48.077288+0000","last_fullsized":"2024-04-16T16:22:48.077288+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:52:57.403249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.636259+0000","last_change":"2024-04-16T16:22:46.474507+0000","last_active":"2024-04-16T16:22:48.636259+0000","last_peered":"2024-04-16T16:22:48.636259+0000","last_clean":"2024-04-16T16:22:48.636259+0000","last_became_active":"2024-04-16T16:22:46.474146+0000","last_became_peered":"2024-04-16T16:22:46.474146+0000","last_unstale":"2024-04-16T16:22:48.636259+0000","last_undegraded":"2024-04-16T16:22:48.636259+0000","last_fullsized":"2024-04-16T16:22:48.636259+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:25:59.259590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.456803+0000","last_change":"2024-04-16T16:22:45.463747+0000","last_active":"2024-04-16T16:22:47.456803+0000","last_peered":"2024-04-16T16:22:47.456803+0000","last_clean":"2024-04-16T16:22:47.456803+0000","last_became_active":"2024-04-16T16:22:45.462875+0000","last_became_peered":"2024-04-16T16:22:45.462875+0000","last_unstale":"2024-04-16T16:22:47.456803+0000","last_undegraded":"2024-04-16T16:22:47.456803+0000","last_fullsized":"2024-04-16T16:22:47.456803+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:29:33.964433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.076960+0000","last_change":"2024-04-16T16:22:46.473128+0000","last_active":"2024-04-16T16:22:48.076960+0000","last_peered":"2024-04-16T16:22:48.076960+0000","last_clean":"2024-04-16T16:22:48.076960+0000","last_became_active":"2024-04-16T16:22:46.472859+0000","last_became_peered":"2024-04-16T16:22:46.472859+0000","last_unstale":"2024-04-16T16:22:48.076960+0000","last_undegraded":"2024-04-16T16:22:48.076960+0000","last_fullsized":"2024-04-16T16:22:48.076960+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:36:30.108257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.076910+0000","last_change":"2024-04-16T16:22:45.466908+0000","last_active":"2024-04-16T16:22:48.076910+0000","last_peered":"2024-04-16T16:22:48.076910+0000","last_clean":"2024-04-16T16:22:48.076910+0000","last_became_active":"2024-04-16T16:22:45.466535+0000","last_became_peered":"2024-04-16T16:22:45.466535+0000","last_unstale":"2024-04-16T16:22:48.076910+0000","last_undegraded":"2024-04-16T16:22:48.076910+0000","last_fullsized":"2024-04-16T16:22:48.076910+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:25.557203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623664+0000","last_change":"2024-04-16T16:22:46.483216+0000","last_active":"2024-04-16T16:22:48.623664+0000","last_peered":"2024-04-16T16:22:48.623664+0000","last_clean":"2024-04-16T16:22:48.623664+0000","last_became_active":"2024-04-16T16:22:46.482943+0000","last_became_peered":"2024-04-16T16:22:46.482943+0000","last_unstale":"2024-04-16T16:22:48.623664+0000","last_undegraded":"2024-04-16T16:22:48.623664+0000","last_fullsized":"2024-04-16T16:22:48.623664+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:26.573221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"155'8","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460436+0000","last_change":"2024-04-16T16:22:45.461068+0000","last_active":"2024-04-16T16:22:47.460436+0000","last_peered":"2024-04-16T16:22:47.460436+0000","last_clean":"2024-04-16T16:22:47.460436+0000","last_became_active":"2024-04-16T16:22:45.459760+0000","last_became_peered":"2024-04-16T16:22:45.459760+0000","last_unstale":"2024-04-16T16:22:47.460436+0000","last_undegraded":"2024-04-16T16:22:47.460436+0000","last_fullsized":"2024-04-16T16:22:47.460436+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:32:24.648814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.078301+0000","last_change":"2024-04-16T16:22:46.473695+0000","last_active":"2024-04-16T16:22:48.078301+0000","last_peered":"2024-04-16T16:22:48.078301+0000","last_clean":"2024-04-16T16:22:48.078301+0000","last_became_active":"2024-04-16T16:22:46.471894+0000","last_became_peered":"2024-04-16T16:22:46.471894+0000","last_unstale":"2024-04-16T16:22:48.078301+0000","last_undegraded":"2024-04-16T16:22:48.078301+0000","last_fullsized":"2024-04-16T16:22:48.078301+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:26.785078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.625205+0000","last_change":"2024-04-16T16:22:45.469335+0000","last_active":"2024-04-16T16:22:48.625205+0000","last_peered":"2024-04-16T16:22:48.625205+0000","last_clean":"2024-04-16T16:22:48.625205+0000","last_became_active":"2024-04-16T16:22:45.468966+0000","last_became_peered":"2024-04-16T16:22:45.468966+0000","last_unstale":"2024-04-16T16:22:48.625205+0000","last_undegraded":"2024-04-16T16:22:48.625205+0000","last_fullsized":"2024-04-16T16:22:48.625205+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:09.300484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457513+0000","last_change":"2024-04-16T16:22:46.471925+0000","last_active":"2024-04-16T16:22:47.457513+0000","last_peered":"2024-04-16T16:22:47.457513+0000","last_clean":"2024-04-16T16:22:47.457513+0000","last_became_active":"2024-04-16T16:22:46.470405+0000","last_became_peered":"2024-04-16T16:22:46.470405+0000","last_unstale":"2024-04-16T16:22:47.457513+0000","last_undegraded":"2024-04-16T16:22:47.457513+0000","last_fullsized":"2024-04-16T16:22:47.457513+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:04.570708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624612+0000","last_change":"2024-04-16T16:22:45.470272+0000","last_active":"2024-04-16T16:22:48.624612+0000","last_peered":"2024-04-16T16:22:48.624612+0000","last_clean":"2024-04-16T16:22:48.624612+0000","last_became_active":"2024-04-16T16:22:45.469959+0000","last_became_peered":"2024-04-16T16:22:45.469959+0000","last_unstale":"2024-04-16T16:22:48.624612+0000","last_undegraded":"2024-04-16T16:22:48.624612+0000","last_fullsized":"2024-04-16T16:22:48.624612+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:34:03.813486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.078101+0000","last_change":"2024-04-16T16:22:46.462951+0000","last_active":"2024-04-16T16:22:48.078101+0000","last_peered":"2024-04-16T16:22:48.078101+0000","last_clean":"2024-04-16T16:22:48.078101+0000","last_became_active":"2024-04-16T16:22:46.462662+0000","last_became_peered":"2024-04-16T16:22:46.462662+0000","last_unstale":"2024-04-16T16:22:48.078101+0000","last_undegraded":"2024-04-16T16:22:48.078101+0000","last_fullsized":"2024-04-16T16:22:48.078101+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:37.006661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462044+0000","last_change":"2024-04-16T16:22:45.462862+0000","last_active":"2024-04-16T16:22:47.462044+0000","last_peered":"2024-04-16T16:22:47.462044+0000","last_clean":"2024-04-16T16:22:47.462044+0000","last_became_active":"2024-04-16T16:22:45.462197+0000","last_became_peered":"2024-04-16T16:22:45.462197+0000","last_unstale":"2024-04-16T16:22:47.462044+0000","last_undegraded":"2024-04-16T16:22:47.462044+0000","last_fullsized":"2024-04-16T16:22:47.462044+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:55.697931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.636077+0000","last_change":"2024-04-16T16:22:46.475567+0000","last_active":"2024-04-16T16:22:48.636077+0000","last_peered":"2024-04-16T16:22:48.636077+0000","last_clean":"2024-04-16T16:22:48.636077+0000","last_became_active":"2024-04-16T16:22:46.475232+0000","last_became_peered":"2024-04-16T16:22:46.475232+0000","last_unstale":"2024-04-16T16:22:48.636077+0000","last_undegraded":"2024-04-16T16:22:48.636077+0000","last_fullsized":"2024-04-16T16:22:48.636077+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:19:43.312174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462377+0000","last_change":"2024-04-16T16:22:45.466893+0000","last_active":"2024-04-16T16:22:47.462377+0000","last_peered":"2024-04-16T16:22:47.462377+0000","last_clean":"2024-04-16T16:22:47.462377+0000","last_became_active":"2024-04-16T16:22:45.466614+0000","last_became_peered":"2024-04-16T16:22:45.466614+0000","last_unstale":"2024-04-16T16:22:47.462377+0000","last_undegraded":"2024-04-16T16:22:47.462377+0000","last_fullsized":"2024-04-16T16:22:47.462377+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:53:47.150015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623905+0000","last_change":"2024-04-16T16:22:46.482036+0000","last_active":"2024-04-16T16:22:48.623905+0000","last_peered":"2024-04-16T16:22:48.623905+0000","last_clean":"2024-04-16T16:22:48.623905+0000","last_became_active":"2024-04-16T16:22:46.481293+0000","last_became_peered":"2024-04-16T16:22:46.481293+0000","last_unstale":"2024-04-16T16:22:48.623905+0000","last_undegraded":"2024-04-16T16:22:48.623905+0000","last_fullsized":"2024-04-16T16:22:48.623905+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:55.094277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077501+0000","last_change":"2024-04-16T16:22:45.466947+0000","last_active":"2024-04-16T16:22:48.077501+0000","last_peered":"2024-04-16T16:22:48.077501+0000","last_clean":"2024-04-16T16:22:48.077501+0000","last_became_active":"2024-04-16T16:22:45.466624+0000","last_became_peered":"2024-04-16T16:22:45.466624+0000","last_unstale":"2024-04-16T16:22:48.077501+0000","last_undegraded":"2024-04-16T16:22:48.077501+0000","last_fullsized":"2024-04-16T16:22:48.077501+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:53.044305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.636145+0000","last_change":"2024-04-16T16:22:46.469862+0000","last_active":"2024-04-16T16:22:48.636145+0000","last_peered":"2024-04-16T16:22:48.636145+0000","last_clean":"2024-04-16T16:22:48.636145+0000","last_became_active":"2024-04-16T16:22:46.466179+0000","last_became_peered":"2024-04-16T16:22:46.466179+0000","last_unstale":"2024-04-16T16:22:48.636145+0000","last_undegraded":"2024-04-16T16:22:48.636145+0000","last_fullsized":"2024-04-16T16:22:48.636145+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:36:56.658760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460683+0000","last_change":"2024-04-16T16:22:45.461564+0000","last_active":"2024-04-16T16:22:47.460683+0000","last_peered":"2024-04-16T16:22:47.460683+0000","last_clean":"2024-04-16T16:22:47.460683+0000","last_became_active":"2024-04-16T16:22:45.461253+0000","last_became_peered":"2024-04-16T16:22:45.461253+0000","last_unstale":"2024-04-16T16:22:47.460683+0000","last_undegraded":"2024-04-16T16:22:47.460683+0000","last_fullsized":"2024-04-16T16:22:47.460683+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:46.018568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462392+0000","last_change":"2024-04-16T16:22:46.465947+0000","last_active":"2024-04-16T16:22:47.462392+0000","last_peered":"2024-04-16T16:22:47.462392+0000","last_clean":"2024-04-16T16:22:47.462392+0000","last_became_active":"2024-04-16T16:22:46.465294+0000","last_became_peered":"2024-04-16T16:22:46.465294+0000","last_unstale":"2024-04-16T16:22:47.462392+0000","last_undegraded":"2024-04-16T16:22:47.462392+0000","last_fullsized":"2024-04-16T16:22:47.462392+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:29.789672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077036+0000","last_change":"2024-04-16T16:22:45.468883+0000","last_active":"2024-04-16T16:22:48.077036+0000","last_peered":"2024-04-16T16:22:48.077036+0000","last_clean":"2024-04-16T16:22:48.077036+0000","last_became_active":"2024-04-16T16:22:45.468091+0000","last_became_peered":"2024-04-16T16:22:45.468091+0000","last_unstale":"2024-04-16T16:22:48.077036+0000","last_undegraded":"2024-04-16T16:22:48.077036+0000","last_fullsized":"2024-04-16T16:22:48.077036+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710495+0000","last_change":"2024-04-16T16:22:46.471354+0000","last_active":"2024-04-16T16:22:48.710495+0000","last_peered":"2024-04-16T16:22:48.710495+0000","last_clean":"2024-04-16T16:22:48.710495+0000","last_became_active":"2024-04-16T16:22:46.470521+0000","last_became_peered":"2024-04-16T16:22:46.470521+0000","last_unstale":"2024-04-16T16:22:48.710495+0000","last_undegraded":"2024-04-16T16:22:48.710495+0000","last_fullsized":"2024-04-16T16:22:48.710495+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:06:08.736226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461147+0000","last_change":"2024-04-16T16:22:45.460865+0000","last_active":"2024-04-16T16:22:47.461147+0000","last_peered":"2024-04-16T16:22:47.461147+0000","last_clean":"2024-04-16T16:22:47.461147+0000","last_became_active":"2024-04-16T16:22:45.458861+0000","last_became_peered":"2024-04-16T16:22:45.458861+0000","last_unstale":"2024-04-16T16:22:47.461147+0000","last_undegraded":"2024-04-16T16:22:47.461147+0000","last_fullsized":"2024-04-16T16:22:47.461147+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:30.462179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457529+0000","last_change":"2024-04-16T16:22:46.464764+0000","last_active":"2024-04-16T16:22:47.457529+0000","last_peered":"2024-04-16T16:22:47.457529+0000","last_clean":"2024-04-16T16:22:47.457529+0000","last_became_active":"2024-04-16T16:22:46.464064+0000","last_became_peered":"2024-04-16T16:22:46.464064+0000","last_unstale":"2024-04-16T16:22:47.457529+0000","last_undegraded":"2024-04-16T16:22:47.457529+0000","last_fullsized":"2024-04-16T16:22:47.457529+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:28:07.667824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457501+0000","last_change":"2024-04-16T16:22:45.466044+0000","last_active":"2024-04-16T16:22:47.457501+0000","last_peered":"2024-04-16T16:22:47.457501+0000","last_clean":"2024-04-16T16:22:47.457501+0000","last_became_active":"2024-04-16T16:22:45.465676+0000","last_became_peered":"2024-04-16T16:22:45.465676+0000","last_unstale":"2024-04-16T16:22:47.457501+0000","last_undegraded":"2024-04-16T16:22:47.457501+0000","last_fullsized":"2024-04-16T16:22:47.457501+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:52.385757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635973+0000","last_change":"2024-04-16T16:22:46.466356+0000","last_active":"2024-04-16T16:22:48.635973+0000","last_peered":"2024-04-16T16:22:48.635973+0000","last_clean":"2024-04-16T16:22:48.635973+0000","last_became_active":"2024-04-16T16:22:46.465449+0000","last_became_peered":"2024-04-16T16:22:46.465449+0000","last_unstale":"2024-04-16T16:22:48.635973+0000","last_undegraded":"2024-04-16T16:22:48.635973+0000","last_fullsized":"2024-04-16T16:22:48.635973+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:04.570708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624728+0000","last_change":"2024-04-16T16:22:45.466156+0000","last_active":"2024-04-16T16:22:48.624728+0000","last_peered":"2024-04-16T16:22:48.624728+0000","last_clean":"2024-04-16T16:22:48.624728+0000","last_became_active":"2024-04-16T16:22:45.465791+0000","last_became_peered":"2024-04-16T16:22:45.465791+0000","last_unstale":"2024-04-16T16:22:48.624728+0000","last_undegraded":"2024-04-16T16:22:48.624728+0000","last_fullsized":"2024-04-16T16:22:48.624728+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:42:46.394979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710309+0000","last_change":"2024-04-16T16:22:46.470854+0000","last_active":"2024-04-16T16:22:48.710309+0000","last_peered":"2024-04-16T16:22:48.710309+0000","last_clean":"2024-04-16T16:22:48.710309+0000","last_became_active":"2024-04-16T16:22:46.470196+0000","last_became_peered":"2024-04-16T16:22:46.470196+0000","last_unstale":"2024-04-16T16:22:48.710309+0000","last_undegraded":"2024-04-16T16:22:48.710309+0000","last_fullsized":"2024-04-16T16:22:48.710309+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:40:24.085281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426634+0000","last_change":"2024-04-16T16:22:45.471205+0000","last_active":"2024-04-16T16:22:48.426634+0000","last_peered":"2024-04-16T16:22:48.426634+0000","last_clean":"2024-04-16T16:22:48.426634+0000","last_became_active":"2024-04-16T16:22:45.470911+0000","last_became_peered":"2024-04-16T16:22:45.470911+0000","last_unstale":"2024-04-16T16:22:48.426634+0000","last_undegraded":"2024-04-16T16:22:48.426634+0000","last_fullsized":"2024-04-16T16:22:48.426634+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:53.044305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457242+0000","last_change":"2024-04-16T16:22:46.469324+0000","last_active":"2024-04-16T16:22:47.457242+0000","last_peered":"2024-04-16T16:22:47.457242+0000","last_clean":"2024-04-16T16:22:47.457242+0000","last_became_active":"2024-04-16T16:22:46.469009+0000","last_became_peered":"2024-04-16T16:22:46.469009+0000","last_unstale":"2024-04-16T16:22:47.457242+0000","last_undegraded":"2024-04-16T16:22:47.457242+0000","last_fullsized":"2024-04-16T16:22:47.457242+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:29.789672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457098+0000","last_change":"2024-04-16T16:22:45.468189+0000","last_active":"2024-04-16T16:22:47.457098+0000","last_peered":"2024-04-16T16:22:47.457098+0000","last_clean":"2024-04-16T16:22:47.457098+0000","last_became_active":"2024-04-16T16:22:45.467709+0000","last_became_peered":"2024-04-16T16:22:45.467709+0000","last_unstale":"2024-04-16T16:22:47.457098+0000","last_undegraded":"2024-04-16T16:22:47.457098+0000","last_fullsized":"2024-04-16T16:22:47.457098+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:46.256132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.456756+0000","last_change":"2024-04-16T16:22:46.465994+0000","last_active":"2024-04-16T16:22:47.456756+0000","last_peered":"2024-04-16T16:22:47.456756+0000","last_clean":"2024-04-16T16:22:47.456756+0000","last_became_active":"2024-04-16T16:22:46.465007+0000","last_became_peered":"2024-04-16T16:22:46.465007+0000","last_unstale":"2024-04-16T16:22:47.456756+0000","last_undegraded":"2024-04-16T16:22:47.456756+0000","last_fullsized":"2024-04-16T16:22:47.456756+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:12:58.505334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462996+0000","last_change":"2024-04-16T16:22:45.470607+0000","last_active":"2024-04-16T16:22:47.462996+0000","last_peered":"2024-04-16T16:22:47.462996+0000","last_clean":"2024-04-16T16:22:47.462996+0000","last_became_active":"2024-04-16T16:22:45.469322+0000","last_became_peered":"2024-04-16T16:22:45.469322+0000","last_unstale":"2024-04-16T16:22:47.462996+0000","last_undegraded":"2024-04-16T16:22:47.462996+0000","last_fullsized":"2024-04-16T16:22:47.462996+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:07.564200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463072+0000","last_change":"2024-04-16T16:22:46.462483+0000","last_active":"2024-04-16T16:22:47.463072+0000","last_peered":"2024-04-16T16:22:47.463072+0000","last_clean":"2024-04-16T16:22:47.463072+0000","last_became_active":"2024-04-16T16:22:46.461432+0000","last_became_peered":"2024-04-16T16:22:46.461432+0000","last_unstale":"2024-04-16T16:22:47.463072+0000","last_undegraded":"2024-04-16T16:22:47.463072+0000","last_fullsized":"2024-04-16T16:22:47.463072+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:01.883490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077289+0000","last_change":"2024-04-16T16:22:45.462752+0000","last_active":"2024-04-16T16:22:48.077289+0000","last_peered":"2024-04-16T16:22:48.077289+0000","last_clean":"2024-04-16T16:22:48.077289+0000","last_became_active":"2024-04-16T16:22:45.462154+0000","last_became_peered":"2024-04-16T16:22:45.462154+0000","last_unstale":"2024-04-16T16:22:48.077289+0000","last_undegraded":"2024-04-16T16:22:48.077289+0000","last_fullsized":"2024-04-16T16:22:48.077289+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:43:04.134472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635368+0000","last_change":"2024-04-16T16:22:46.467575+0000","last_active":"2024-04-16T16:22:48.635368+0000","last_peered":"2024-04-16T16:22:48.635368+0000","last_clean":"2024-04-16T16:22:48.635368+0000","last_became_active":"2024-04-16T16:22:46.466890+0000","last_became_peered":"2024-04-16T16:22:46.466890+0000","last_unstale":"2024-04-16T16:22:48.635368+0000","last_undegraded":"2024-04-16T16:22:48.635368+0000","last_fullsized":"2024-04-16T16:22:48.635368+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:43:19.877365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426288+0000","last_change":"2024-04-16T16:22:45.469689+0000","last_active":"2024-04-16T16:22:48.426288+0000","last_peered":"2024-04-16T16:22:48.426288+0000","last_clean":"2024-04-16T16:22:48.426288+0000","last_became_active":"2024-04-16T16:22:45.469350+0000","last_became_peered":"2024-04-16T16:22:45.469350+0000","last_unstale":"2024-04-16T16:22:48.426288+0000","last_undegraded":"2024-04-16T16:22:48.426288+0000","last_fullsized":"2024-04-16T16:22:48.426288+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623691+0000","last_change":"2024-04-16T16:22:46.478730+0000","last_active":"2024-04-16T16:22:48.623691+0000","last_peered":"2024-04-16T16:22:48.623691+0000","last_clean":"2024-04-16T16:22:48.623691+0000","last_became_active":"2024-04-16T16:22:46.478459+0000","last_became_peered":"2024-04-16T16:22:46.478459+0000","last_unstale":"2024-04-16T16:22:48.623691+0000","last_undegraded":"2024-04-16T16:22:48.623691+0000","last_fullsized":"2024-04-16T16:22:48.623691+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:36.777723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460788+0000","last_change":"2024-04-16T16:22:45.467505+0000","last_active":"2024-04-16T16:22:47.460788+0000","last_peered":"2024-04-16T16:22:47.460788+0000","last_clean":"2024-04-16T16:22:47.460788+0000","last_became_active":"2024-04-16T16:22:45.465339+0000","last_became_peered":"2024-04-16T16:22:45.465339+0000","last_unstale":"2024-04-16T16:22:47.460788+0000","last_undegraded":"2024-04-16T16:22:47.460788+0000","last_fullsized":"2024-04-16T16:22:47.460788+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:54.078259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457379+0000","last_change":"2024-04-16T16:22:46.465023+0000","last_active":"2024-04-16T16:22:47.457379+0000","last_peered":"2024-04-16T16:22:47.457379+0000","last_clean":"2024-04-16T16:22:47.457379+0000","last_became_active":"2024-04-16T16:22:46.464436+0000","last_became_peered":"2024-04-16T16:22:46.464436+0000","last_unstale":"2024-04-16T16:22:47.457379+0000","last_undegraded":"2024-04-16T16:22:47.457379+0000","last_fullsized":"2024-04-16T16:22:47.457379+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:54:26.843618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463842+0000","last_change":"2024-04-16T16:22:45.461515+0000","last_active":"2024-04-16T16:22:47.463842+0000","last_peered":"2024-04-16T16:22:47.463842+0000","last_clean":"2024-04-16T16:22:47.463842+0000","last_became_active":"2024-04-16T16:22:45.460350+0000","last_became_peered":"2024-04-16T16:22:45.460350+0000","last_unstale":"2024-04-16T16:22:47.463842+0000","last_undegraded":"2024-04-16T16:22:47.463842+0000","last_fullsized":"2024-04-16T16:22:47.463842+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:34.292891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"144'4691","reported_seq":5390,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462677+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:22:47.462677+0000","last_peered":"2024-04-16T16:22:47.462677+0000","last_clean":"2024-04-16T16:22:47.462677+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:22:47.462677+0000","last_undegraded":"2024-04-16T16:22:47.462677+0000","last_fullsized":"2024-04-16T16:22:47.462677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":4691,"log_dups_size":0,"ondisk_log_size":4691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":387,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":387,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4691,"ondisk_log_size":4691,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574957,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37856,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333984,"statfs":{"total":96636764160,"available":96597999616,"internally_reserved":0,"allocated":2576384,"data_stored":1526220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574958,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1961984,"data_stored":933520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574958,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":934102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574959,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33116,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338724,"statfs":{"total":96636764160,"available":96602853376,"internally_reserved":0,"allocated":1974272,"data_stored":935180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574957,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37264,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334576,"statfs":{"total":96636764160,"available":96598605824,"internally_reserved":0,"allocated":1961984,"data_stored":933532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574957,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37256,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334584,"statfs":{"total":96636764160,"available":96598614016,"internally_reserved":0,"allocated":1961984,"data_stored":933588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37256,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334584,"statfs":{"total":96636764160,"available":96598614016,"internally_reserved":0,"allocated":1961984,"data_stored":933532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574958,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37832,"kb_used_data":2492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334008,"statfs":{"total":96636764160,"available":96598024192,"internally_reserved":0,"allocated":2551808,"data_stored":1523866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:22:56.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T16:22:56.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~7s 2024-04-16T16:22:56.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.297+0000 7fb24ae00640 1 Processor -- start 2024-04-16T16:22:56.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.298+0000 7fb24ae00640 1 -- start start 2024-04-16T16:22:56.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.299+0000 7fb24ae00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb244161e90 0x7fb2440cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:56.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.299+0000 7fb24ae00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb2440d52d0 0x7fb24406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:56.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.299+0000 7fb24ae00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2440d3080 con 0x7fb244162270 2024-04-16T16:22:56.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.299+0000 7fb24ae00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb2440d2e30 con 0x7fb2440d52d0 2024-04-16T16:22:56.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.299+0000 7fb24ae00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb2440d3360 con 0x7fb244161e90 2024-04-16T16:22:56.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.300+0000 7fb24a400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb2440d52d0 0x7fb24406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:56.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.300+0000 7fb24a400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb2440d52d0 0x7fb24406ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47268/0 (socket says 172.21.15.45:47268) 2024-04-16T16:22:56.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.300+0000 7fb24a400640 1 -- 172.21.15.45:0/4136305714 learned_addr learned my addr 172.21.15.45:0/4136305714 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:56.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.300+0000 7fb249a00640 1 --2- 172.21.15.45:0/4136305714 >> v2:172.21.15.103:3300/0 conn(0x7fb244161e90 0x7fb2440cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:56.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.300+0000 7fb24a400640 1 -- 172.21.15.45:0/4136305714 >> v2:172.21.15.103:3300/0 conn(0x7fb244161e90 msgr2=0x7fb2440cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:56.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.300+0000 7fb24a400640 1 --2- 172.21.15.45:0/4136305714 >> v2:172.21.15.103:3300/0 conn(0x7fb244161e90 0x7fb2440cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:56.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.300+0000 7fb24a400640 1 -- 172.21.15.45:0/4136305714 >> v1:172.21.15.45:6789/0 conn(0x7fb244162270 legacy=0x7fb2440d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:56.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.300+0000 7fb24a400640 1 -- 172.21.15.45:0/4136305714 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2440d3640 con 0x7fb2440d52d0 2024-04-16T16:22:56.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.300+0000 7fb249a00640 1 --2- 172.21.15.45:0/4136305714 >> v2:172.21.15.103:3300/0 conn(0x7fb244161e90 0x7fb2440cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:56.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.301+0000 7fb24a400640 1 --2- 172.21.15.45:0/4136305714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2440d52d0 0x7fb24406ce40 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009870 tx=0x7fb23c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c5049a1c4f8117 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:56.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.302+0000 7fb243e00640 1 -- 172.21.15.45:0/4136305714 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c034070 con 0x7fb2440d52d0 2024-04-16T16:22:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.302+0000 7fb243e00640 1 -- 172.21.15.45:0/4136305714 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb23c030860 con 0x7fb2440d52d0 2024-04-16T16:22:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.302+0000 7fb243e00640 1 -- 172.21.15.45:0/4136305714 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c030c30 con 0x7fb2440d52d0 2024-04-16T16:22:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.302+0000 7fb24ae00640 1 -- 172.21.15.45:0/4136305714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2440d52d0 msgr2=0x7fb24406ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.302+0000 7fb24ae00640 1 --2- 172.21.15.45:0/4136305714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2440d52d0 0x7fb24406ce40 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009870 tx=0x7fb23c02b9a0 comp rx=0 tx=0).stop 2024-04-16T16:22:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.303+0000 7fb24ae00640 1 -- 172.21.15.45:0/4136305714 shutdown_connections 2024-04-16T16:22:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.303+0000 7fb24ae00640 1 --2- 172.21.15.45:0/4136305714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2440d52d0 0x7fb24406ce40 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.303+0000 7fb24ae00640 1 --2- 172.21.15.45:0/4136305714 >> v2:172.21.15.103:3300/0 conn(0x7fb244161e90 0x7fb2440cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.303+0000 7fb24ae00640 1 -- 172.21.15.45:0/4136305714 >> 172.21.15.45:0/4136305714 conn(0x7fb2440d1940 msgr2=0x7fb2440d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.303+0000 7fb24ae00640 1 -- 172.21.15.45:0/4136305714 shutdown_connections 2024-04-16T16:22:56.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.303+0000 7fb24ae00640 1 -- 172.21.15.45:0/4136305714 wait complete. 2024-04-16T16:22:56.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.305+0000 7fb24ae00640 1 Processor -- start 2024-04-16T16:22:56.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.305+0000 7fb24ae00640 1 -- start start 2024-04-16T16:22:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.306+0000 7fb24ae00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2440d52d0 0x7fb2441f5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.306+0000 7fb24ae00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb244161e90 0x7fb24414df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:56.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.306+0000 7fb24ae00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb244162270 0x7fb24406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:56.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.306+0000 7fb24ae00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb2440d4550 con 0x7fb244162270 2024-04-16T16:22:56.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.306+0000 7fb24ae00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2440cfd10 con 0x7fb244161e90 2024-04-16T16:22:56.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.306+0000 7fb24ae00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb2440d46d0 con 0x7fb2440d52d0 2024-04-16T16:22:56.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.306+0000 7fb24a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb244162270 0x7fb24406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:56.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.306+0000 7fb24a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb244162270 0x7fb24406ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47284/0 (socket says 172.21.15.45:47284) 2024-04-16T16:22:56.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.306+0000 7fb24a400640 1 -- 172.21.15.45:0/3509937062 learned_addr learned my addr 172.21.15.45:0/3509937062 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:56.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.306+0000 7fb249a00640 1 --2- 172.21.15.45:0/3509937062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2440d52d0 0x7fb2441f5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.307+0000 7fb249000640 1 --2- 172.21.15.45:0/3509937062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb244161e90 0x7fb24414df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.307+0000 7fb24a400640 1 -- 172.21.15.45:0/3509937062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2440d52d0 msgr2=0x7fb2441f5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.307+0000 7fb24a400640 1 --2- 172.21.15.45:0/3509937062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2440d52d0 0x7fb2441f5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.307+0000 7fb24a400640 1 -- 172.21.15.45:0/3509937062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb244161e90 msgr2=0x7fb24414df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.307+0000 7fb24a400640 1 --2- 172.21.15.45:0/3509937062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb244161e90 0x7fb24414df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.307+0000 7fb24a400640 1 -- 172.21.15.45:0/3509937062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2441f5990 con 0x7fb244162270 2024-04-16T16:22:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.307+0000 7fb24a400640 1 --2- 172.21.15.45:0/3509937062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb244162270 0x7fb24406ce40 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009870 tx=0x7fb23c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:56.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.307+0000 7fb241600640 1 -- 172.21.15.45:0/3509937062 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c048070 con 0x7fb244162270 2024-04-16T16:22:56.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.307+0000 7fb241600640 1 -- 172.21.15.45:0/3509937062 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb23c039d90 con 0x7fb244162270 2024-04-16T16:22:56.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.307+0000 7fb24ae00640 1 -- 172.21.15.45:0/3509937062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb24415ca60 con 0x7fb244162270 2024-04-16T16:22:56.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.307+0000 7fb241600640 1 -- 172.21.15.45:0/3509937062 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c034040 con 0x7fb244162270 2024-04-16T16:22:56.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.307+0000 7fb24ae00640 1 -- 172.21.15.45:0/3509937062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb24415d020 con 0x7fb244162270 2024-04-16T16:22:56.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.309+0000 7fb24ae00640 1 -- 172.21.15.45:0/3509937062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb244075780 con 0x7fb244162270 2024-04-16T16:22:56.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.310+0000 7fb241600640 1 -- 172.21.15.45:0/3509937062 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fb23c04b070 con 0x7fb244162270 2024-04-16T16:22:56.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.311+0000 7fb241600640 1 --2- 172.21.15.45:0/3509937062 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb22806ca10 0x7fb22806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:56.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.311+0000 7fb241600640 1 -- 172.21.15.45:0/3509937062 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7fb23c0adfb0 con 0x7fb244162270 2024-04-16T16:22:56.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.312+0000 7fb249a00640 1 --2- 172.21.15.45:0/3509937062 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb22806ca10 0x7fb22806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:56.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.313+0000 7fb249a00640 1 --2- 172.21.15.45:0/3509937062 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb22806ca10 0x7fb22806eed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb23800a990 tx=0x7fb238008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:56.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.315+0000 7fb241600640 1 -- 172.21.15.45:0/3509937062 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb23c07c1b0 con 0x7fb244162270 2024-04-16T16:22:56.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.520+0000 7fb24ae00640 1 -- 172.21.15.45:0/3509937062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb2440b3420 con 0x7fb244162270 2024-04-16T16:22:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.523+0000 7fb241600640 1 -- 172.21.15.45:0/3509937062 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v156) ==== 74+0+16671 (secure 0 0 0) 0x7fb23c07bab0 con 0x7fb244162270 2024-04-16T16:22:56.523 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:22:56.523 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":156,"fsid":"748c4b85-0190-4c59-80ad-098a6a5007c0","created":"2024-04-16T16:13:05.835202+0000","modified":"2024-04-16T16:22:47.450563+0000","last_up_change":"2024-04-16T16:13:17.804403+0000","last_in_change":"2024-04-16T16:13:09.473563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T16:13:19.130583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-16T16:22:43.864367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-16T16:22:44.899356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3887da33-8db2-468b-a30b-3835ff05c375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6801","nonce":1377072140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6803","nonce":1377072140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6807","nonce":1377072140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1377072140},{"type":"v1","addr":"172.21.15.45:6805","nonce":1377072140}]},"public_addr":"172.21.15.45:6801/1377072140","cluster_addr":"172.21.15.45:6803/1377072140","heartbeat_back_addr":"172.21.15.45:6807/1377072140","heartbeat_front_addr":"172.21.15.45:6805/1377072140","state":["exists","up"]},{"osd":1,"uuid":"361acb09-4d66-455b-a5b9-c1d9a8794735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6824","nonce":3039880665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6827","nonce":3039880665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6831","nonce":3039880665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3039880665},{"type":"v1","addr":"172.21.15.45:6829","nonce":3039880665}]},"public_addr":"172.21.15.45:6824/3039880665","cluster_addr":"172.21.15.45:6827/3039880665","heartbeat_back_addr":"172.21.15.45:6831/3039880665","heartbeat_front_addr":"172.21.15.45:6829/3039880665","state":["exists","up"]},{"osd":2,"uuid":"13d87862-41f8-4b4f-b19f-10984092f273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6809","nonce":3148487306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6811","nonce":3148487306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6815","nonce":3148487306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3148487306},{"type":"v1","addr":"172.21.15.45:6813","nonce":3148487306}]},"public_addr":"172.21.15.45:6809/3148487306","cluster_addr":"172.21.15.45:6811/3148487306","heartbeat_back_addr":"172.21.15.45:6815/3148487306","heartbeat_front_addr":"172.21.15.45:6813/3148487306","state":["exists","up"]},{"osd":3,"uuid":"d1387f39-070b-4cd8-8abe-8a700504e189","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6817","nonce":707516363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6819","nonce":707516363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6823","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6825","nonce":707516363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":707516363},{"type":"v1","addr":"172.21.15.45:6821","nonce":707516363}]},"public_addr":"172.21.15.45:6817/707516363","cluster_addr":"172.21.15.45:6819/707516363","heartbeat_back_addr":"172.21.15.45:6825/707516363","heartbeat_front_addr":"172.21.15.45:6821/707516363","state":["exists","up"]},{"osd":4,"uuid":"b53b6063-c9a0-487a-a7f8-401c8032ea6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6801","nonce":4229500350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6803","nonce":4229500350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6807","nonce":4229500350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4229500350},{"type":"v1","addr":"172.21.15.103:6805","nonce":4229500350}]},"public_addr":"172.21.15.103:6801/4229500350","cluster_addr":"172.21.15.103:6803/4229500350","heartbeat_back_addr":"172.21.15.103:6807/4229500350","heartbeat_front_addr":"172.21.15.103:6805/4229500350","state":["exists","up"]},{"osd":5,"uuid":"b9ca6760-7092-4896-977b-585d9887e13c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6824","nonce":2127910843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6827","nonce":2127910843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6831","nonce":2127910843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2127910843},{"type":"v1","addr":"172.21.15.103:6829","nonce":2127910843}]},"public_addr":"172.21.15.103:6824/2127910843","cluster_addr":"172.21.15.103:6827/2127910843","heartbeat_back_addr":"172.21.15.103:6831/2127910843","heartbeat_front_addr":"172.21.15.103:6829/2127910843","state":["exists","up"]},{"osd":6,"uuid":"f3e011e1-0ece-4d60-aa69-44310d9c6a8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6809","nonce":3613143327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6811","nonce":3613143327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6815","nonce":3613143327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3613143327},{"type":"v1","addr":"172.21.15.103:6813","nonce":3613143327}]},"public_addr":"172.21.15.103:6809/3613143327","cluster_addr":"172.21.15.103:6811/3613143327","heartbeat_back_addr":"172.21.15.103:6815/3613143327","heartbeat_front_addr":"172.21.15.103:6813/3613143327","state":["exists","up"]},{"osd":7,"uuid":"122cbe72-024a-41a0-aa44-fc95cb04511c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6817","nonce":1303888027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6819","nonce":1303888027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6825","nonce":1303888027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1303888027},{"type":"v1","addr":"172.21.15.103:6821","nonce":1303888027}]},"public_addr":"172.21.15.103:6817/1303888027","cluster_addr":"172.21.15.103:6819/1303888027","heartbeat_back_addr":"172.21.15.103:6825/1303888027","heartbeat_front_addr":"172.21.15.103:6821/1303888027","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.876761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.652232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.895693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.943260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.812967+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.692620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.693886+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T16:13:14.814116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T16:22:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.526+0000 7fb24ae00640 1 -- 172.21.15.45:0/3509937062 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb22806ca10 msgr2=0x7fb22806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.526+0000 7fb24ae00640 1 --2- 172.21.15.45:0/3509937062 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb22806ca10 0x7fb22806eed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb23800a990 tx=0x7fb238008040 comp rx=0 tx=0).stop 2024-04-16T16:22:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.526+0000 7fb24ae00640 1 -- 172.21.15.45:0/3509937062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb244162270 msgr2=0x7fb24406ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.526+0000 7fb24ae00640 1 --2- 172.21.15.45:0/3509937062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb244162270 0x7fb24406ce40 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009870 tx=0x7fb23c004730 comp rx=0 tx=0).stop 2024-04-16T16:22:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.527+0000 7fb24ae00640 1 -- 172.21.15.45:0/3509937062 shutdown_connections 2024-04-16T16:22:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.527+0000 7fb24ae00640 1 --2- 172.21.15.45:0/3509937062 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fb22806ca10 0x7fb22806eed0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.527+0000 7fb24ae00640 1 --2- 172.21.15.45:0/3509937062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb244162270 0x7fb24406ce40 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.527+0000 7fb24ae00640 1 --2- 172.21.15.45:0/3509937062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb244161e90 0x7fb24414df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:56.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.527+0000 7fb24ae00640 1 --2- 172.21.15.45:0/3509937062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2440d52d0 0x7fb2441f5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:56.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.527+0000 7fb24ae00640 1 -- 172.21.15.45:0/3509937062 >> 172.21.15.45:0/3509937062 conn(0x7fb2440d1940 msgr2=0x7fb2440b9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:56.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.527+0000 7fb24ae00640 1 -- 172.21.15.45:0/3509937062 shutdown_connections 2024-04-16T16:22:56.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:56.527+0000 7fb24ae00640 1 -- 172.21.15.45:0/3509937062 wait complete. 2024-04-16T16:22:57.549 INFO:tasks.ceph:Scrubbing osd.0 2024-04-16T16:22:57.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-16T16:22:57.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.787+0000 7ffa6fe00640 1 Processor -- start 2024-04-16T16:22:57.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.787+0000 7ffa6fe00640 1 -- start start 2024-04-16T16:22:57.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.788+0000 7ffa6fe00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffa70155ab0 0x7ffa70162ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:57.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.788+0000 7ffa6fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa700d18c0 0x7ffa700d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:57.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.788+0000 7ffa6fe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa70155e90 con 0x7ffa700ba610 2024-04-16T16:22:57.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.788+0000 7ffa6fe00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ffa7015ff00 con 0x7ffa70155ab0 2024-04-16T16:22:57.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.788+0000 7ffa6fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffa7015f6e0 con 0x7ffa700d18c0 2024-04-16T16:22:57.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.789+0000 7ffa6ea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffa70155ab0 0x7ffa70162ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:57.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.789+0000 7ffa6ea00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffa70155ab0 0x7ffa70162ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47294/0 (socket says 172.21.15.45:47294) 2024-04-16T16:22:57.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.789+0000 7ffa6ea00640 1 -- 172.21.15.45:0/50120141 learned_addr learned my addr 172.21.15.45:0/50120141 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:57.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.789+0000 7ffa6ea00640 1 -- 172.21.15.45:0/50120141 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa700d18c0 msgr2=0x7ffa700d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:57.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.789+0000 7ffa6ea00640 1 --2- 172.21.15.45:0/50120141 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa700d18c0 0x7ffa700d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:57.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.789+0000 7ffa6ea00640 1 -- 172.21.15.45:0/50120141 >> v1:172.21.15.103:6789/0 conn(0x7ffa700ba610 legacy=0x7ffa7015efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:57.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.789+0000 7ffa6ea00640 1 -- 172.21.15.45:0/50120141 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7015f940 con 0x7ffa70155ab0 2024-04-16T16:22:57.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.790+0000 7ffa6ea00640 1 --2- 172.21.15.45:0/50120141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa70155ab0 0x7ffa70162ef0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009870 tx=0x7ffa6002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24559b6f4f4963a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.791+0000 7ffa6d600640 1 -- 172.21.15.45:0/50120141 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa600317c0 con 0x7ffa70155ab0 2024-04-16T16:22:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.791+0000 7ffa6d600640 1 -- 172.21.15.45:0/50120141 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa60037070 con 0x7ffa70155ab0 2024-04-16T16:22:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.791+0000 7ffa6d600640 1 -- 172.21.15.45:0/50120141 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60031b80 con 0x7ffa70155ab0 2024-04-16T16:22:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.791+0000 7ffa6fe00640 1 -- 172.21.15.45:0/50120141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa70155ab0 msgr2=0x7ffa70162ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.791+0000 7ffa6fe00640 1 --2- 172.21.15.45:0/50120141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa70155ab0 0x7ffa70162ef0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009870 tx=0x7ffa6002b860 comp rx=0 tx=0).stop 2024-04-16T16:22:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.791+0000 7ffa6fe00640 1 -- 172.21.15.45:0/50120141 shutdown_connections 2024-04-16T16:22:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.791+0000 7ffa6fe00640 1 --2- 172.21.15.45:0/50120141 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa700d18c0 0x7ffa700d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.791+0000 7ffa6fe00640 1 --2- 172.21.15.45:0/50120141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa70155ab0 0x7ffa70162ef0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:57.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.791+0000 7ffa6fe00640 1 -- 172.21.15.45:0/50120141 >> 172.21.15.45:0/50120141 conn(0x7ffa7006d1a0 msgr2=0x7ffa7006d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:57.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.791+0000 7ffa6fe00640 1 -- 172.21.15.45:0/50120141 shutdown_connections 2024-04-16T16:22:57.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.791+0000 7ffa6fe00640 1 -- 172.21.15.45:0/50120141 wait complete. 2024-04-16T16:22:57.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.793+0000 7ffa6fe00640 1 Processor -- start 2024-04-16T16:22:57.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.793+0000 7ffa6fe00640 1 -- start start 2024-04-16T16:22:57.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.794+0000 7ffa6fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa700ba610 0x7ffa700ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:57.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.794+0000 7ffa6fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa700d18c0 0x7ffa700d2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:57.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.794+0000 7ffa6fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa70155ab0 0x7ffa700d3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:57.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.794+0000 7ffa6fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffa701629d0 con 0x7ffa700ba610 2024-04-16T16:22:57.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.794+0000 7ffa6fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa701626f0 con 0x7ffa70155ab0 2024-04-16T16:22:57.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.794+0000 7ffa6fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffa70163430 con 0x7ffa700d18c0 2024-04-16T16:22:57.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.794+0000 7ffa6ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa700ba610 0x7ffa700ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:57.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.794+0000 7ffa6ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa700ba610 0x7ffa700ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47310/0 (socket says 172.21.15.45:47310) 2024-04-16T16:22:57.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.794+0000 7ffa6ea00640 1 -- 172.21.15.45:0/3672821935 learned_addr learned my addr 172.21.15.45:0/3672821935 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:57.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.795+0000 7ffa6f400640 1 --2- 172.21.15.45:0/3672821935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa70155ab0 0x7ffa700d3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:57.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.795+0000 7ffa6e000640 1 --2- 172.21.15.45:0/3672821935 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa700d18c0 0x7ffa700d2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:57.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.795+0000 7ffa6ea00640 1 -- 172.21.15.45:0/3672821935 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa700d18c0 msgr2=0x7ffa700d2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:57.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.795+0000 7ffa6ea00640 1 --2- 172.21.15.45:0/3672821935 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa700d18c0 0x7ffa700d2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:57.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.795+0000 7ffa6ea00640 1 -- 172.21.15.45:0/3672821935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa70155ab0 msgr2=0x7ffa700d3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:57.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.795+0000 7ffa6ea00640 1 --2- 172.21.15.45:0/3672821935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa70155ab0 0x7ffa700d3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:57.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.795+0000 7ffa6ea00640 1 -- 172.21.15.45:0/3672821935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa701f8b30 con 0x7ffa700ba610 2024-04-16T16:22:57.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.795+0000 7ffa6ea00640 1 --2- 172.21.15.45:0/3672821935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa700ba610 0x7ffa700ca380 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7ffa60000c00 tx=0x7ffa600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:57.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.796+0000 7ffa66a00640 1 -- 172.21.15.45:0/3672821935 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60032860 con 0x7ffa700ba610 2024-04-16T16:22:57.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.796+0000 7ffa66a00640 1 -- 172.21.15.45:0/3672821935 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa60032a00 con 0x7ffa700ba610 2024-04-16T16:22:57.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.796+0000 7ffa6fe00640 1 -- 172.21.15.45:0/3672821935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa700b8b40 con 0x7ffa700ba610 2024-04-16T16:22:57.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.796+0000 7ffa66a00640 1 -- 172.21.15.45:0/3672821935 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60032df0 con 0x7ffa700ba610 2024-04-16T16:22:57.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.796+0000 7ffa6fe00640 1 -- 172.21.15.45:0/3672821935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa700b9100 con 0x7ffa700ba610 2024-04-16T16:22:57.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.797+0000 7ffa6fe00640 1 -- 172.21.15.45:0/3672821935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffa700b94e0 con 0x7ffa700ba610 2024-04-16T16:22:57.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.798+0000 7ffa66a00640 1 -- 172.21.15.45:0/3672821935 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7ffa6004c060 con 0x7ffa700ba610 2024-04-16T16:22:57.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.799+0000 7ffa66a00640 1 --2- 172.21.15.45:0/3672821935 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ffa5006ca10 0x7ffa5006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:57.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.799+0000 7ffa66a00640 1 -- 172.21.15.45:0/3672821935 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7ffa600adcc0 con 0x7ffa700ba610 2024-04-16T16:22:57.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.799+0000 7ffa66a00640 1 --2- 172.21.15.45:0/3672821935 >> [v2:172.21.15.45:6800/1377072140,v1:172.21.15.45:6801/1377072140] conn(0x7ffa50074c10 0x7ffa50077030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:57.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.799+0000 7ffa66a00640 1 -- 172.21.15.45:0/3672821935 --> [v2:172.21.15.45:6800/1377072140,v1:172.21.15.45:6801/1377072140] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffa500776e0 con 0x7ffa50074c10 2024-04-16T16:22:57.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.799+0000 7ffa66a00640 1 -- 172.21.15.45:0/3672821935 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=156) ==== 24+0+0 (secure 0 0 0) 0x7ffa600ae070 con 0x7ffa700ba610 2024-04-16T16:22:57.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.800+0000 7ffa6e000640 1 --2- 172.21.15.45:0/3672821935 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ffa5006ca10 0x7ffa5006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:57.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.800+0000 7ffa6f400640 1 --2- 172.21.15.45:0/3672821935 >> [v2:172.21.15.45:6800/1377072140,v1:172.21.15.45:6801/1377072140] conn(0x7ffa50074c10 0x7ffa50077030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:57.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.800+0000 7ffa6f400640 1 --2- 172.21.15.45:0/3672821935 >> [v2:172.21.15.45:6800/1377072140,v1:172.21.15.45:6801/1377072140] conn(0x7ffa50074c10 0x7ffa50077030 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:57.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.801+0000 7ffa6e000640 1 --2- 172.21.15.45:0/3672821935 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ffa5006ca10 0x7ffa5006eed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffa7014f660 tx=0x7ffa5c009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:57.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.801+0000 7ffa66a00640 1 -- 172.21.15.45:0/3672821935 <== osd.0 v2:172.21.15.45:6800/1377072140 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffa500776e0 con 0x7ffa50074c10 2024-04-16T16:22:57.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.816+0000 7ffa6fe00640 1 -- 172.21.15.45:0/3672821935 --> [v2:172.21.15.45:6800/1377072140,v1:172.21.15.45:6801/1377072140] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ffa70075780 con 0x7ffa50074c10 2024-04-16T16:22:57.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.816+0000 7ffa66a00640 1 -- 172.21.15.45:0/3672821935 <== osd.0 v2:172.21.15.45:6800/1377072140 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ffa70075780 con 0x7ffa50074c10 2024-04-16T16:22:57.815 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:22:57.815 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-16T16:22:57.816 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:22:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.817+0000 7ffa6fe00640 1 -- 172.21.15.45:0/3672821935 >> [v2:172.21.15.45:6800/1377072140,v1:172.21.15.45:6801/1377072140] conn(0x7ffa50074c10 msgr2=0x7ffa50077030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.817+0000 7ffa6fe00640 1 --2- 172.21.15.45:0/3672821935 >> [v2:172.21.15.45:6800/1377072140,v1:172.21.15.45:6801/1377072140] conn(0x7ffa50074c10 0x7ffa50077030 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.817+0000 7ffa6fe00640 1 -- 172.21.15.45:0/3672821935 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ffa5006ca10 msgr2=0x7ffa5006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.817+0000 7ffa6fe00640 1 --2- 172.21.15.45:0/3672821935 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ffa5006ca10 0x7ffa5006eed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffa7014f660 tx=0x7ffa5c009040 comp rx=0 tx=0).stop 2024-04-16T16:22:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.817+0000 7ffa6fe00640 1 -- 172.21.15.45:0/3672821935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa700ba610 msgr2=0x7ffa700ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.817+0000 7ffa6fe00640 1 --2- 172.21.15.45:0/3672821935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa700ba610 0x7ffa700ca380 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7ffa60000c00 tx=0x7ffa600046c0 comp rx=0 tx=0).stop 2024-04-16T16:22:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.817+0000 7ffa6f400640 1 -- 172.21.15.45:0/3672821935 reap_dead start 2024-04-16T16:22:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.817+0000 7ffa6fe00640 1 -- 172.21.15.45:0/3672821935 shutdown_connections 2024-04-16T16:22:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.817+0000 7ffa6fe00640 1 -- 172.21.15.45:0/3672821935 >> 172.21.15.45:0/3672821935 conn(0x7ffa7006d1a0 msgr2=0x7ffa7015e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.817+0000 7ffa6fe00640 1 -- 172.21.15.45:0/3672821935 shutdown_connections 2024-04-16T16:22:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:57.817+0000 7ffa6fe00640 1 -- 172.21.15.45:0/3672821935 wait complete. 2024-04-16T16:22:57.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-16T16:22:58.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.048+0000 7ff976400640 1 Processor -- start 2024-04-16T16:22:58.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.048+0000 7ff976400640 1 -- start start 2024-04-16T16:22:58.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.049+0000 7ff976400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff970072890 0x7ff9700cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.050+0000 7ff976400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9700cd220 0x7ff9700cd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.050+0000 7ff976400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff970146600 con 0x7ff970146220 2024-04-16T16:22:58.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.050+0000 7ff976400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff97006f840 con 0x7ff970072890 2024-04-16T16:22:58.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.050+0000 7ff976400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff9700ce0e0 con 0x7ff9700cd220 2024-04-16T16:22:58.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.050+0000 7ff967e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff970072890 0x7ff9700cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.050+0000 7ff967e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff970072890 0x7ff9700cf110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47322/0 (socket says 172.21.15.45:47322) 2024-04-16T16:22:58.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.050+0000 7ff967e00640 1 -- 172.21.15.45:0/1305117982 learned_addr learned my addr 172.21.15.45:0/1305117982 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:58.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.050+0000 7ff967e00640 1 -- 172.21.15.45:0/1305117982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9700cd220 msgr2=0x7ff9700cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:58.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.050+0000 7ff967e00640 1 --2- 172.21.15.45:0/1305117982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9700cd220 0x7ff9700cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.050+0000 7ff967e00640 1 -- 172.21.15.45:0/1305117982 >> v1:172.21.15.103:6789/0 conn(0x7ff970146220 legacy=0x7ff970072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:58.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.050+0000 7ff967e00640 1 -- 172.21.15.45:0/1305117982 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9700cde00 con 0x7ff970072890 2024-04-16T16:22:58.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.052+0000 7ff967e00640 1 --2- 172.21.15.45:0/1305117982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff970072890 0x7ff9700cf110 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7ff9700b5d50 tx=0x7ff95802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b6b7aba4ce10251 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:58.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.052+0000 7ff96ea00640 1 -- 172.21.15.45:0/1305117982 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff958035020 con 0x7ff970072890 2024-04-16T16:22:58.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.052+0000 7ff96ea00640 1 -- 172.21.15.45:0/1305117982 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff958031830 con 0x7ff970072890 2024-04-16T16:22:58.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.052+0000 7ff96ea00640 1 -- 172.21.15.45:0/1305117982 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff958031c00 con 0x7ff970072890 2024-04-16T16:22:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.053+0000 7ff976400640 1 -- 172.21.15.45:0/1305117982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff970072890 msgr2=0x7ff9700cf110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.053+0000 7ff976400640 1 --2- 172.21.15.45:0/1305117982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff970072890 0x7ff9700cf110 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7ff9700b5d50 tx=0x7ff95802b8d0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.053+0000 7ff976400640 1 -- 172.21.15.45:0/1305117982 shutdown_connections 2024-04-16T16:22:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.053+0000 7ff976400640 1 --2- 172.21.15.45:0/1305117982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9700cd220 0x7ff9700cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.053+0000 7ff976400640 1 --2- 172.21.15.45:0/1305117982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff970072890 0x7ff9700cf110 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.053+0000 7ff976400640 1 -- 172.21.15.45:0/1305117982 >> 172.21.15.45:0/1305117982 conn(0x7ff97006b930 msgr2=0x7ff97006bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:58.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.053+0000 7ff976400640 1 -- 172.21.15.45:0/1305117982 shutdown_connections 2024-04-16T16:22:58.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.053+0000 7ff976400640 1 -- 172.21.15.45:0/1305117982 wait complete. 2024-04-16T16:22:58.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.055+0000 7ff976400640 1 Processor -- start 2024-04-16T16:22:58.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.055+0000 7ff976400640 1 -- start start 2024-04-16T16:22:58.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.056+0000 7ff976400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff970072890 0x7ff970153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.056+0000 7ff976400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9700cd220 0x7ff9701f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.056+0000 7ff976400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff970146220 0x7ff9700b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.056+0000 7ff976400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff9700ce910 con 0x7ff970072890 2024-04-16T16:22:58.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.056+0000 7ff976400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff9700cdbb0 con 0x7ff970146220 2024-04-16T16:22:58.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.056+0000 7ff976400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff9700cebf0 con 0x7ff9700cd220 2024-04-16T16:22:58.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.056+0000 7ff96f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff970072890 0x7ff970153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.056+0000 7ff96f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff970072890 0x7ff970153ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47324/0 (socket says 172.21.15.45:47324) 2024-04-16T16:22:58.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.056+0000 7ff96f400640 1 -- 172.21.15.45:0/1579663971 learned_addr learned my addr 172.21.15.45:0/1579663971 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:58.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.057+0000 7ff967e00640 1 --2- 172.21.15.45:0/1579663971 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9700cd220 0x7ff9701f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.057+0000 7ff96fe00640 1 --2- 172.21.15.45:0/1579663971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff970146220 0x7ff9700b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.057+0000 7ff96f400640 1 -- 172.21.15.45:0/1579663971 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9700cd220 msgr2=0x7ff9701f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.057+0000 7ff96f400640 1 --2- 172.21.15.45:0/1579663971 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9700cd220 0x7ff9701f3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.057+0000 7ff96f400640 1 -- 172.21.15.45:0/1579663971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff970146220 msgr2=0x7ff9700b49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.057+0000 7ff96f400640 1 --2- 172.21.15.45:0/1579663971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff970146220 0x7ff9700b49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.057+0000 7ff96f400640 1 -- 172.21.15.45:0/1579663971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9701547d0 con 0x7ff970072890 2024-04-16T16:22:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.057+0000 7ff96fe00640 1 --2- 172.21.15.45:0/1579663971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff970146220 0x7ff9700b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:22:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.057+0000 7ff96f400640 1 --2- 172.21.15.45:0/1579663971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff970072890 0x7ff970153ed0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7ff96000b3f0 tx=0x7ff96000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.058+0000 7ff967400640 1 -- 172.21.15.45:0/1579663971 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff960015020 con 0x7ff970072890 2024-04-16T16:22:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.058+0000 7ff967400640 1 -- 172.21.15.45:0/1579663971 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff960011830 con 0x7ff970072890 2024-04-16T16:22:58.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.058+0000 7ff967400640 1 -- 172.21.15.45:0/1579663971 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff960011c20 con 0x7ff970072890 2024-04-16T16:22:58.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.058+0000 7ff976400640 1 -- 172.21.15.45:0/1579663971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff970154ab0 con 0x7ff970072890 2024-04-16T16:22:58.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.058+0000 7ff976400640 1 -- 172.21.15.45:0/1579663971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9701e6250 con 0x7ff970072890 2024-04-16T16:22:58.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.059+0000 7ff976400640 1 -- 172.21.15.45:0/1579663971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff970075780 con 0x7ff970072890 2024-04-16T16:22:58.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.060+0000 7ff967400640 1 -- 172.21.15.45:0/1579663971 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7ff960018020 con 0x7ff970072890 2024-04-16T16:22:58.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.061+0000 7ff967400640 1 --2- 172.21.15.45:0/1579663971 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ff94c06ca10 0x7ff94c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.061+0000 7ff967400640 1 -- 172.21.15.45:0/1579663971 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7ff96008b9c0 con 0x7ff970072890 2024-04-16T16:22:58.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.062+0000 7ff967e00640 1 --2- 172.21.15.45:0/1579663971 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ff94c06ca10 0x7ff94c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.063+0000 7ff967e00640 1 --2- 172.21.15.45:0/1579663971 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ff94c06ca10 0x7ff94c06eed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff958036000 tx=0x7ff95803c040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.066+0000 7ff967400640 1 -- 172.21.15.45:0/1579663971 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff96001f030 con 0x7ff970072890 2024-04-16T16:22:58.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.270+0000 7ff976400640 1 -- 172.21.15.45:0/1579663971 --> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7ff9701e6630 con 0x7ff94c06ca10 2024-04-16T16:22:58.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.271+0000 7ff967400640 1 -- 172.21.15.45:0/1579663971 <== mgr.7591 v2:172.21.15.103:6832/1129061436 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff9701e6630 con 0x7ff94c06ca10 2024-04-16T16:22:58.270 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 0 to deep-scrub 2024-04-16T16:22:58.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.274+0000 7ff976400640 1 -- 172.21.15.45:0/1579663971 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ff94c06ca10 msgr2=0x7ff94c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.274+0000 7ff976400640 1 --2- 172.21.15.45:0/1579663971 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ff94c06ca10 0x7ff94c06eed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff958036000 tx=0x7ff95803c040 comp rx=0 tx=0).stop 2024-04-16T16:22:58.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.274+0000 7ff976400640 1 -- 172.21.15.45:0/1579663971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff970072890 msgr2=0x7ff970153ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.274+0000 7ff976400640 1 --2- 172.21.15.45:0/1579663971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff970072890 0x7ff970153ed0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7ff96000b3f0 tx=0x7ff96000b8b0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.274+0000 7ff976400640 1 -- 172.21.15.45:0/1579663971 shutdown_connections 2024-04-16T16:22:58.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.274+0000 7ff976400640 1 --2- 172.21.15.45:0/1579663971 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ff94c06ca10 0x7ff94c06eed0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.274+0000 7ff976400640 1 --2- 172.21.15.45:0/1579663971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff970146220 0x7ff9700b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.274+0000 7ff976400640 1 --2- 172.21.15.45:0/1579663971 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9700cd220 0x7ff9701f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.274+0000 7ff976400640 1 --2- 172.21.15.45:0/1579663971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff970072890 0x7ff970153ed0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.274+0000 7ff976400640 1 -- 172.21.15.45:0/1579663971 >> 172.21.15.45:0/1579663971 conn(0x7ff97006b930 msgr2=0x7ff9700b2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:58.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.274+0000 7ff976400640 1 -- 172.21.15.45:0/1579663971 shutdown_connections 2024-04-16T16:22:58.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.274+0000 7ff976400640 1 -- 172.21.15.45:0/1579663971 wait complete. 2024-04-16T16:22:58.293 INFO:tasks.ceph:Scrubbing osd.1 2024-04-16T16:22:58.293 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-16T16:22:58.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.526+0000 7f162e000640 1 Processor -- start 2024-04-16T16:22:58.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.527+0000 7f162e000640 1 -- start start 2024-04-16T16:22:58.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.528+0000 7f162e000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1628155ab0 0x7f1628162ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.528+0000 7f162e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16280ba610 0x7f162815efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.528+0000 7f162e000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1628155e90 con 0x7f16280d18c0 2024-04-16T16:22:58.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.528+0000 7f162e000640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f162815ff00 con 0x7f1628155ab0 2024-04-16T16:22:58.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.528+0000 7f162e000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f162815f940 con 0x7f16280ba610 2024-04-16T16:22:58.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.528+0000 7f162cc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1628155ab0 0x7f1628162ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.529+0000 7f162cc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1628155ab0 0x7f1628162ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47328/0 (socket says 172.21.15.45:47328) 2024-04-16T16:22:58.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.529+0000 7f162cc00640 1 -- 172.21.15.45:0/3746728695 learned_addr learned my addr 172.21.15.45:0/3746728695 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:58.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.529+0000 7f1627e00640 1 --2- 172.21.15.45:0/3746728695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16280ba610 0x7f162815efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.529+0000 7f162cc00640 1 -- 172.21.15.45:0/3746728695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16280ba610 msgr2=0x7f162815efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:58.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.529+0000 7f162cc00640 1 --2- 172.21.15.45:0/3746728695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16280ba610 0x7f162815efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.529+0000 7f162cc00640 1 -- 172.21.15.45:0/3746728695 >> v1:172.21.15.103:6789/0 conn(0x7f16280d18c0 legacy=0x7f16280d2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:58.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.529+0000 7f162cc00640 1 -- 172.21.15.45:0/3746728695 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f162815fc20 con 0x7f1628155ab0 2024-04-16T16:22:58.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.530+0000 7f162cc00640 1 --2- 172.21.15.45:0/3746728695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1628155ab0 0x7f1628162ef0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f161c009870 tx=0x7f161c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec30f40ad30bf097 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:58.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.531+0000 7f1627400640 1 -- 172.21.15.45:0/3746728695 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c0317c0 con 0x7f1628155ab0 2024-04-16T16:22:58.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.531+0000 7f1627400640 1 -- 172.21.15.45:0/3746728695 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f161c037070 con 0x7f1628155ab0 2024-04-16T16:22:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.531+0000 7f1627400640 1 -- 172.21.15.45:0/3746728695 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c031b80 con 0x7f1628155ab0 2024-04-16T16:22:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.531+0000 7f162e000640 1 -- 172.21.15.45:0/3746728695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1628155ab0 msgr2=0x7f1628162ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.531+0000 7f162e000640 1 --2- 172.21.15.45:0/3746728695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1628155ab0 0x7f1628162ef0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f161c009870 tx=0x7f161c02b860 comp rx=0 tx=0).stop 2024-04-16T16:22:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.531+0000 7f162e000640 1 -- 172.21.15.45:0/3746728695 shutdown_connections 2024-04-16T16:22:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.531+0000 7f162e000640 1 --2- 172.21.15.45:0/3746728695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16280ba610 0x7f162815efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.531+0000 7f162e000640 1 --2- 172.21.15.45:0/3746728695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1628155ab0 0x7f1628162ef0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.531+0000 7f162e000640 1 -- 172.21.15.45:0/3746728695 >> 172.21.15.45:0/3746728695 conn(0x7f162806d1a0 msgr2=0x7f162806d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.531+0000 7f162e000640 1 -- 172.21.15.45:0/3746728695 shutdown_connections 2024-04-16T16:22:58.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.532+0000 7f162e000640 1 -- 172.21.15.45:0/3746728695 wait complete. 2024-04-16T16:22:58.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.533+0000 7f162e000640 1 Processor -- start 2024-04-16T16:22:58.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.533+0000 7f162e000640 1 -- start start 2024-04-16T16:22:58.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.534+0000 7f162e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16280ba610 0x7f1628152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.534+0000 7f162e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16280d18c0 0x7f1628156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.534+0000 7f162e000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1628155ab0 0x7f16280d2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.534+0000 7f162e000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1628163430 con 0x7f16280ba610 2024-04-16T16:22:58.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.534+0000 7f162e000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f16281629d0 con 0x7f1628155ab0 2024-04-16T16:22:58.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.534+0000 7f162e000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f162815f510 con 0x7f16280d18c0 2024-04-16T16:22:58.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.535+0000 7f162cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16280ba610 0x7f1628152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.535+0000 7f162cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16280ba610 0x7f1628152d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47330/0 (socket says 172.21.15.45:47330) 2024-04-16T16:22:58.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.535+0000 7f162cc00640 1 -- 172.21.15.45:0/2687878011 learned_addr learned my addr 172.21.15.45:0/2687878011 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:58.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.535+0000 7f1627e00640 1 --2- 172.21.15.45:0/2687878011 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16280d18c0 0x7f1628156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.535+0000 7f162d600640 1 --2- 172.21.15.45:0/2687878011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1628155ab0 0x7f16280d2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.535+0000 7f162cc00640 1 -- 172.21.15.45:0/2687878011 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16280d18c0 msgr2=0x7f1628156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.535+0000 7f162cc00640 1 --2- 172.21.15.45:0/2687878011 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16280d18c0 0x7f1628156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.535+0000 7f162cc00640 1 -- 172.21.15.45:0/2687878011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1628155ab0 msgr2=0x7f16280d2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.535+0000 7f162cc00640 1 --2- 172.21.15.45:0/2687878011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1628155ab0 0x7f16280d2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.535+0000 7f162cc00640 1 -- 172.21.15.45:0/2687878011 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16280cc8a0 con 0x7f16280ba610 2024-04-16T16:22:58.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.536+0000 7f162cc00640 1 --2- 172.21.15.45:0/2687878011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16280ba610 0x7f1628152d20 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f161c009870 tx=0x7f161c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:58.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.536+0000 7f1624c00640 1 -- 172.21.15.45:0/2687878011 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c032650 con 0x7f16280ba610 2024-04-16T16:22:58.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.536+0000 7f1624c00640 1 -- 172.21.15.45:0/2687878011 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f161c0327f0 con 0x7f16280ba610 2024-04-16T16:22:58.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.536+0000 7f1624c00640 1 -- 172.21.15.45:0/2687878011 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c032be0 con 0x7f16280ba610 2024-04-16T16:22:58.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.536+0000 7f162e000640 1 -- 172.21.15.45:0/2687878011 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16280ccb80 con 0x7f16280ba610 2024-04-16T16:22:58.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.536+0000 7f162e000640 1 -- 172.21.15.45:0/2687878011 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16280cd440 con 0x7f16280ba610 2024-04-16T16:22:58.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.537+0000 7f162e000640 1 -- 172.21.15.45:0/2687878011 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1628075780 con 0x7f16280ba610 2024-04-16T16:22:58.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.538+0000 7f1624c00640 1 -- 172.21.15.45:0/2687878011 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f161c04c060 con 0x7f16280ba610 2024-04-16T16:22:58.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.539+0000 7f1624c00640 1 --2- 172.21.15.45:0/2687878011 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f160806ca10 0x7f160806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.540+0000 7f1624c00640 1 -- 172.21.15.45:0/2687878011 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f161c0ad5b0 con 0x7f16280ba610 2024-04-16T16:22:58.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.540+0000 7f1624c00640 1 --2- 172.21.15.45:0/2687878011 >> [v2:172.21.15.45:6822/3039880665,v1:172.21.15.45:6824/3039880665] conn(0x7f1608074bb0 0x7f1608076fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.540+0000 7f1624c00640 1 -- 172.21.15.45:0/2687878011 --> [v2:172.21.15.45:6822/3039880665,v1:172.21.15.45:6824/3039880665] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f161c034e70 con 0x7f1608074bb0 2024-04-16T16:22:58.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.540+0000 7f1624c00640 1 -- 172.21.15.45:0/2687878011 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=156) ==== 24+0+0 (secure 0 0 0) 0x7f161c0b4050 con 0x7f16280ba610 2024-04-16T16:22:58.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.540+0000 7f162d600640 1 --2- 172.21.15.45:0/2687878011 >> [v2:172.21.15.45:6822/3039880665,v1:172.21.15.45:6824/3039880665] conn(0x7f1608074bb0 0x7f1608076fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.540+0000 7f1627e00640 1 --2- 172.21.15.45:0/2687878011 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f160806ca10 0x7f160806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.541+0000 7f162d600640 1 --2- 172.21.15.45:0/2687878011 >> [v2:172.21.15.45:6822/3039880665,v1:172.21.15.45:6824/3039880665] conn(0x7f1608074bb0 0x7f1608076fd0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:58.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.541+0000 7f1627e00640 1 --2- 172.21.15.45:0/2687878011 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f160806ca10 0x7f160806eed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f16281f7940 tx=0x7f1614008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:58.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.541+0000 7f1624c00640 1 -- 172.21.15.45:0/2687878011 <== osd.1 v2:172.21.15.45:6822/3039880665 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f161c034e70 con 0x7f1608074bb0 2024-04-16T16:22:58.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.556+0000 7f162e000640 1 -- 172.21.15.45:0/2687878011 --> [v2:172.21.15.45:6822/3039880665,v1:172.21.15.45:6824/3039880665] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f16280d5640 con 0x7f1608074bb0 2024-04-16T16:22:58.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.557+0000 7f1624c00640 1 -- 172.21.15.45:0/2687878011 <== osd.1 v2:172.21.15.45:6822/3039880665 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f16280d5640 con 0x7f1608074bb0 2024-04-16T16:22:58.556 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:22:58.556 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-16T16:22:58.556 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:22:58.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.557+0000 7f162e000640 1 -- 172.21.15.45:0/2687878011 >> [v2:172.21.15.45:6822/3039880665,v1:172.21.15.45:6824/3039880665] conn(0x7f1608074bb0 msgr2=0x7f1608076fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.557+0000 7f162e000640 1 --2- 172.21.15.45:0/2687878011 >> [v2:172.21.15.45:6822/3039880665,v1:172.21.15.45:6824/3039880665] conn(0x7f1608074bb0 0x7f1608076fd0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.557+0000 7f162e000640 1 -- 172.21.15.45:0/2687878011 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f160806ca10 msgr2=0x7f160806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.557+0000 7f162e000640 1 --2- 172.21.15.45:0/2687878011 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f160806ca10 0x7f160806eed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f16281f7940 tx=0x7f1614008040 comp rx=0 tx=0).stop 2024-04-16T16:22:58.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.557+0000 7f162e000640 1 -- 172.21.15.45:0/2687878011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16280ba610 msgr2=0x7f1628152d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.557+0000 7f162e000640 1 --2- 172.21.15.45:0/2687878011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16280ba610 0x7f1628152d20 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f161c009870 tx=0x7f161c0046c0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.557+0000 7f162d600640 1 -- 172.21.15.45:0/2687878011 reap_dead start 2024-04-16T16:22:58.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.557+0000 7f162e000640 1 -- 172.21.15.45:0/2687878011 shutdown_connections 2024-04-16T16:22:58.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.557+0000 7f162e000640 1 -- 172.21.15.45:0/2687878011 >> 172.21.15.45:0/2687878011 conn(0x7f162806d1a0 msgr2=0x7f16280d48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:58.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.558+0000 7f162e000640 1 -- 172.21.15.45:0/2687878011 shutdown_connections 2024-04-16T16:22:58.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.558+0000 7f162e000640 1 -- 172.21.15.45:0/2687878011 wait complete. 2024-04-16T16:22:58.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-16T16:22:58.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.788+0000 7fdaa3200640 1 Processor -- start 2024-04-16T16:22:58.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.788+0000 7fdaa3200640 1 -- start start 2024-04-16T16:22:58.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.790+0000 7fdaa3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cd930 0x7fda9c0cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.790+0000 7fdaa3200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda9c146600 con 0x7fda9c072890 2024-04-16T16:22:58.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.790+0000 7fdaa3200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda9c0cde00 con 0x7fda9c146220 2024-04-16T16:22:58.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.790+0000 7fdaa3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda9c06f840 con 0x7fda9c0cd930 2024-04-16T16:22:58.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.791+0000 7fda93e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fda9c072890 0x7fda9c0cd220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45934/0 (socket says 172.21.15.45:45934) 2024-04-16T16:22:58.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.791+0000 7fda93e00640 1 -- 172.21.15.45:0/1005048605 learned_addr learned my addr 172.21.15.45:0/1005048605 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:58.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.791+0000 7fdaa1200640 1 --2- 172.21.15.45:0/1005048605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cd930 0x7fda9c0cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.791+0000 7fda9b400640 1 -- 172.21.15.45:0/1005048605 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 914498949 0 0) 0x7fda9c146600 con 0x7fda9c072890 2024-04-16T16:22:58.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.791+0000 7fda9b400640 1 -- 172.21.15.45:0/1005048605 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda80003520 con 0x7fda9c072890 2024-04-16T16:22:58.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.792+0000 7fda9b400640 1 -- 172.21.15.45:0/1005048605 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2280838082 0 0) 0x7fda9c0cde00 con 0x7fda9c146220 2024-04-16T16:22:58.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.792+0000 7fdaa1200640 1 -- 172.21.15.45:0/1005048605 >> v1:172.21.15.103:6789/0 conn(0x7fda9c146220 legacy=0x7fda9c072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.792+0000 7fdaa1200640 1 -- 172.21.15.45:0/1005048605 >> v1:172.21.15.45:6789/0 conn(0x7fda9c072890 legacy=0x7fda9c0cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.792+0000 7fdaa1200640 1 -- 172.21.15.45:0/1005048605 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c0ce630 con 0x7fda9c0cd930 2024-04-16T16:22:58.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.793+0000 7fdaa1200640 1 --2- 172.21.15.45:0/1005048605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cd930 0x7fda9c0cf110 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fda94009dc0 tx=0x7fda9402bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51fdadd503499945 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:58.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.794+0000 7fda9b400640 1 -- 172.21.15.45:0/1005048605 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94039070 con 0x7fda9c0cd930 2024-04-16T16:22:58.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.794+0000 7fda9b400640 1 -- 172.21.15.45:0/1005048605 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda9402ccc0 con 0x7fda9c0cd930 2024-04-16T16:22:58.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.794+0000 7fda9b400640 1 -- 172.21.15.45:0/1005048605 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94034660 con 0x7fda9c0cd930 2024-04-16T16:22:58.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.795+0000 7fdaa3200640 1 -- 172.21.15.45:0/1005048605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cd930 msgr2=0x7fda9c0cf110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.795+0000 7fdaa3200640 1 --2- 172.21.15.45:0/1005048605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cd930 0x7fda9c0cf110 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fda94009dc0 tx=0x7fda9402bf10 comp rx=0 tx=0).stop 2024-04-16T16:22:58.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.795+0000 7fdaa3200640 1 -- 172.21.15.45:0/1005048605 shutdown_connections 2024-04-16T16:22:58.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.795+0000 7fdaa3200640 1 --2- 172.21.15.45:0/1005048605 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cd930 0x7fda9c0cf110 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.795+0000 7fdaa3200640 1 -- 172.21.15.45:0/1005048605 >> 172.21.15.45:0/1005048605 conn(0x7fda9c06b930 msgr2=0x7fda9c06bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:58.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.795+0000 7fdaa3200640 1 -- 172.21.15.45:0/1005048605 shutdown_connections 2024-04-16T16:22:58.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.795+0000 7fdaa3200640 1 -- 172.21.15.45:0/1005048605 wait complete. 2024-04-16T16:22:58.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.797+0000 7fdaa3200640 1 Processor -- start 2024-04-16T16:22:58.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.798+0000 7fdaa3200640 1 -- start start 2024-04-16T16:22:58.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.799+0000 7fdaa3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda9c072890 0x7fda9c1568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.799+0000 7fdaa3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cd930 0x7fda9c160a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.799+0000 7fdaa3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda9c146220 0x7fda9c128990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.799+0000 7fdaa3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c0cebf0 con 0x7fda9c072890 2024-04-16T16:22:58.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.799+0000 7fdaa3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c0ce0e0 con 0x7fda9c146220 2024-04-16T16:22:58.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.799+0000 7fdaa3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda9c06df00 con 0x7fda9c0cd930 2024-04-16T16:22:58.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.799+0000 7fda9be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda9c072890 0x7fda9c1568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.799+0000 7fda9be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda9c072890 0x7fda9c1568f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47332/0 (socket says 172.21.15.45:47332) 2024-04-16T16:22:58.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.799+0000 7fda9be00640 1 -- 172.21.15.45:0/3058487388 learned_addr learned my addr 172.21.15.45:0/3058487388 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:58.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.800+0000 7fda93e00640 1 --2- 172.21.15.45:0/3058487388 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cd930 0x7fda9c160a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.800+0000 7fdaa1200640 1 --2- 172.21.15.45:0/3058487388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda9c146220 0x7fda9c128990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.800+0000 7fda9be00640 1 -- 172.21.15.45:0/3058487388 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cd930 msgr2=0x7fda9c160a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.800+0000 7fda9be00640 1 --2- 172.21.15.45:0/3058487388 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cd930 0x7fda9c160a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.800+0000 7fda9be00640 1 -- 172.21.15.45:0/3058487388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda9c146220 msgr2=0x7fda9c128990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:58.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.800+0000 7fda9be00640 1 --2- 172.21.15.45:0/3058487388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda9c146220 0x7fda9c128990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:58.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.800+0000 7fda9be00640 1 -- 172.21.15.45:0/3058487388 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c156e30 con 0x7fda9c072890 2024-04-16T16:22:58.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.800+0000 7fda9be00640 1 --2- 172.21.15.45:0/3058487388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda9c072890 0x7fda9c1568f0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00c3c0 tx=0x7fda8c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:58.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.800+0000 7fda98c00640 1 -- 172.21.15.45:0/3058487388 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8c017070 con 0x7fda9c072890 2024-04-16T16:22:58.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.800+0000 7fda98c00640 1 -- 172.21.15.45:0/3058487388 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda8c0035f0 con 0x7fda9c072890 2024-04-16T16:22:58.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.800+0000 7fdaa3200640 1 -- 172.21.15.45:0/3058487388 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda9c141880 con 0x7fda9c072890 2024-04-16T16:22:58.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.800+0000 7fda98c00640 1 -- 172.21.15.45:0/3058487388 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8c0039e0 con 0x7fda9c072890 2024-04-16T16:22:58.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.801+0000 7fdaa3200640 1 -- 172.21.15.45:0/3058487388 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda9c141e40 con 0x7fda9c072890 2024-04-16T16:22:58.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.802+0000 7fdaa3200640 1 -- 172.21.15.45:0/3058487388 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda9c075780 con 0x7fda9c072890 2024-04-16T16:22:58.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.803+0000 7fda98c00640 1 -- 172.21.15.45:0/3058487388 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fda8c01a070 con 0x7fda9c072890 2024-04-16T16:22:58.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.804+0000 7fda98c00640 1 --2- 172.21.15.45:0/3058487388 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fda7806ca40 0x7fda7806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:58.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.804+0000 7fda98c00640 1 -- 172.21.15.45:0/3058487388 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7fda8c08aff0 con 0x7fda9c072890 2024-04-16T16:22:58.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.805+0000 7fda93e00640 1 --2- 172.21.15.45:0/3058487388 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fda7806ca40 0x7fda7806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:58.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.806+0000 7fda93e00640 1 --2- 172.21.15.45:0/3058487388 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fda7806ca40 0x7fda7806ef00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0b1910 tx=0x7fda84009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:58.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:58.809+0000 7fda98c00640 1 -- 172.21.15.45:0/3058487388 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda8c059360 con 0x7fda9c072890 2024-04-16T16:22:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.019+0000 7fdaa3200640 1 -- 172.21.15.45:0/3058487388 --> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fda9c06d160 con 0x7fda7806ca40 2024-04-16T16:22:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.020+0000 7fda98c00640 1 -- 172.21.15.45:0/3058487388 <== mgr.7591 v2:172.21.15.103:6832/1129061436 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fda9c06d160 con 0x7fda7806ca40 2024-04-16T16:22:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 1 to deep-scrub 2024-04-16T16:22:59.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.023+0000 7fdaa3200640 1 -- 172.21.15.45:0/3058487388 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fda7806ca40 msgr2=0x7fda7806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.023+0000 7fdaa3200640 1 --2- 172.21.15.45:0/3058487388 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fda7806ca40 0x7fda7806ef00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0b1910 tx=0x7fda84009040 comp rx=0 tx=0).stop 2024-04-16T16:22:59.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.023+0000 7fdaa3200640 1 -- 172.21.15.45:0/3058487388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda9c072890 msgr2=0x7fda9c1568f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.023+0000 7fdaa3200640 1 --2- 172.21.15.45:0/3058487388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda9c072890 0x7fda9c1568f0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00c3c0 tx=0x7fda8c00c880 comp rx=0 tx=0).stop 2024-04-16T16:22:59.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.023+0000 7fdaa3200640 1 -- 172.21.15.45:0/3058487388 shutdown_connections 2024-04-16T16:22:59.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.023+0000 7fdaa3200640 1 --2- 172.21.15.45:0/3058487388 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fda7806ca40 0x7fda7806ef00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.023+0000 7fdaa3200640 1 --2- 172.21.15.45:0/3058487388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda9c146220 0x7fda9c128990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.023+0000 7fdaa3200640 1 --2- 172.21.15.45:0/3058487388 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cd930 0x7fda9c160a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.023+0000 7fdaa3200640 1 --2- 172.21.15.45:0/3058487388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda9c072890 0x7fda9c1568f0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.023+0000 7fdaa3200640 1 -- 172.21.15.45:0/3058487388 >> 172.21.15.45:0/3058487388 conn(0x7fda9c06b930 msgr2=0x7fda9c072180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:59.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.023+0000 7fdaa3200640 1 -- 172.21.15.45:0/3058487388 shutdown_connections 2024-04-16T16:22:59.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.023+0000 7fdaa3200640 1 -- 172.21.15.45:0/3058487388 wait complete. 2024-04-16T16:22:59.041 INFO:tasks.ceph:Scrubbing osd.2 2024-04-16T16:22:59.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-16T16:22:59.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.263+0000 7f97a9c00640 1 Processor -- start 2024-04-16T16:22:59.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.263+0000 7f97a9c00640 1 -- start start 2024-04-16T16:22:59.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.265+0000 7f97a9c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97a4147ff0 0x7f97a406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.265+0000 7f97a9c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97a415fc90 con 0x7f97a406e920 2024-04-16T16:22:59.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.265+0000 7f97a9c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97a4162730 con 0x7f97a415f600 2024-04-16T16:22:59.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.265+0000 7f97a9c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f97a41624e0 con 0x7f97a4147ff0 2024-04-16T16:22:59.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.265+0000 7f97a3400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f97a406e920 0x7f97a415eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45946/0 (socket says 172.21.15.45:45946) 2024-04-16T16:22:59.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.265+0000 7f97a3400640 1 -- 172.21.15.45:0/3538598679 learned_addr learned my addr 172.21.15.45:0/3538598679 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:59.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.266+0000 7f97a3e00640 1 --2- 172.21.15.45:0/3538598679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97a4147ff0 0x7f97a406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.266+0000 7f97a2a00640 1 -- 172.21.15.45:0/3538598679 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3862802698 0 0) 0x7f97a415fc90 con 0x7f97a406e920 2024-04-16T16:22:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.266+0000 7f97a2a00640 1 -- 172.21.15.45:0/3538598679 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9788003520 con 0x7f97a406e920 2024-04-16T16:22:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.266+0000 7f97a2a00640 1 -- 172.21.15.45:0/3538598679 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1129087573 0 0) 0x7f9788003520 con 0x7f97a406e920 2024-04-16T16:22:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.266+0000 7f97a2a00640 1 -- 172.21.15.45:0/3538598679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97a4147ff0 msgr2=0x7f97a406e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.266+0000 7f97a2a00640 1 --2- 172.21.15.45:0/3538598679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97a4147ff0 0x7f97a406e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.266+0000 7f97a2a00640 1 -- 172.21.15.45:0/3538598679 >> v1:172.21.15.103:6789/0 conn(0x7f97a415f600 legacy=0x7f97a40b1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.266+0000 7f97a2a00640 1 -- 172.21.15.45:0/3538598679 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a4162a10 con 0x7f97a406e920 2024-04-16T16:22:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.266+0000 7f97a2a00640 1 -- 172.21.15.45:0/3538598679 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f9790001670 con 0x7f97a406e920 2024-04-16T16:22:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.266+0000 7f97a2a00640 1 -- 172.21.15.45:0/3538598679 >> v1:172.21.15.45:6789/0 conn(0x7f97a406e920 legacy=0x7f97a415eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.266+0000 7f97a2a00640 1 --2- 172.21.15.45:0/3538598679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97a4147ff0 0x7f97880039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.266+0000 7f97a2a00640 1 --2- 172.21.15.45:0/3538598679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9788003fb0 0x7f9788004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.267+0000 7f97a2a00640 1 --2- 172.21.15.45:0/3538598679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9788004d40 0x7f97880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.267+0000 7f97a2a00640 1 -- 172.21.15.45:0/3538598679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f97a41624e0 con 0x7f9788003fb0 2024-04-16T16:22:59.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.267+0000 7f97a2a00640 1 -- 172.21.15.45:0/3538598679 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f978800bbb0 con 0x7f9788004d40 2024-04-16T16:22:59.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.267+0000 7f97a2a00640 1 -- 172.21.15.45:0/3538598679 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f978800bdb0 con 0x7f97a4147ff0 2024-04-16T16:22:59.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.267+0000 7f97a3400640 1 --2- 172.21.15.45:0/3538598679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9788003fb0 0x7f9788004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.267+0000 7f97a3e00640 1 --2- 172.21.15.45:0/3538598679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97a4147ff0 0x7f97880039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.267+0000 7f97a9200640 1 --2- 172.21.15.45:0/3538598679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9788004d40 0x7f97880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.267+0000 7f97a3400640 1 -- 172.21.15.45:0/3538598679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97a4147ff0 msgr2=0x7f97880039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:59.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.267+0000 7f97a3400640 1 --2- 172.21.15.45:0/3538598679 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97a4147ff0 0x7f97880039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.267+0000 7f97a3400640 1 -- 172.21.15.45:0/3538598679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9788004d40 msgr2=0x7f97880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:59.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.267+0000 7f97a3400640 1 --2- 172.21.15.45:0/3538598679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9788004d40 0x7f97880051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.267+0000 7f97a3400640 1 -- 172.21.15.45:0/3538598679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9788003520 con 0x7f9788003fb0 2024-04-16T16:22:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.268+0000 7f97a3400640 1 --2- 172.21.15.45:0/3538598679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9788003fb0 0x7f9788004800 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f979000c0e0 tx=0x7f9788005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a6dd116d5fdf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.269+0000 7f97a2a00640 1 -- 172.21.15.45:0/3538598679 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9790007c20 con 0x7f9788003fb0 2024-04-16T16:22:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.269+0000 7f97a2a00640 1 -- 172.21.15.45:0/3538598679 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9790030070 con 0x7f9788003fb0 2024-04-16T16:22:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.269+0000 7f97a2a00640 1 -- 172.21.15.45:0/3538598679 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9790005d70 con 0x7f9788003fb0 2024-04-16T16:22:59.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.269+0000 7f97a9c00640 1 -- 172.21.15.45:0/3538598679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9788003fb0 msgr2=0x7f9788004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.269+0000 7f97a9c00640 1 --2- 172.21.15.45:0/3538598679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9788003fb0 0x7f9788004800 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f979000c0e0 tx=0x7f9788005830 comp rx=0 tx=0).stop 2024-04-16T16:22:59.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.269+0000 7f97a9200640 1 -- 172.21.15.45:0/3538598679 reap_dead start 2024-04-16T16:22:59.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.269+0000 7f97a9c00640 1 -- 172.21.15.45:0/3538598679 shutdown_connections 2024-04-16T16:22:59.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.269+0000 7f97a9c00640 1 -- 172.21.15.45:0/3538598679 >> 172.21.15.45:0/3538598679 conn(0x7f97a40b3bb0 msgr2=0x7f97a40b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:59.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.269+0000 7f97a9c00640 1 -- 172.21.15.45:0/3538598679 shutdown_connections 2024-04-16T16:22:59.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.270+0000 7f97a9c00640 1 -- 172.21.15.45:0/3538598679 wait complete. 2024-04-16T16:22:59.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.271+0000 7f97a9c00640 1 Processor -- start 2024-04-16T16:22:59.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.271+0000 7f97a9c00640 1 -- start start 2024-04-16T16:22:59.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.272+0000 7f97a9c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9788003fb0 0x7f97a40c5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.272+0000 7f97a9c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97a4136f70 0x7f97a4141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.272+0000 7f97a9c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a40b1df0 0x7f97a40b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.272+0000 7f97a9c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f97a4163210 con 0x7f97a40b1df0 2024-04-16T16:22:59.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.272+0000 7f97a9c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f97a41483d0 con 0x7f97a4136f70 2024-04-16T16:22:59.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.272+0000 7f97a9c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f97a4141020 con 0x7f9788003fb0 2024-04-16T16:22:59.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.273+0000 7f97a9200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a40b1df0 0x7f97a40b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.273+0000 7f97a9200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a40b1df0 0x7f97a40b22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47350/0 (socket says 172.21.15.45:47350) 2024-04-16T16:22:59.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.273+0000 7f97a9200640 1 -- 172.21.15.45:0/3598058831 learned_addr learned my addr 172.21.15.45:0/3598058831 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:59.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.273+0000 7f97a3e00640 1 --2- 172.21.15.45:0/3598058831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9788003fb0 0x7f97a40c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.273+0000 7f97a3400640 1 --2- 172.21.15.45:0/3598058831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97a4136f70 0x7f97a4141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.273+0000 7f97a9200640 1 -- 172.21.15.45:0/3598058831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9788003fb0 msgr2=0x7f97a40c5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.273+0000 7f97a9200640 1 --2- 172.21.15.45:0/3598058831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9788003fb0 0x7f97a40c5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.273+0000 7f97a9200640 1 -- 172.21.15.45:0/3598058831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97a4136f70 msgr2=0x7f97a4141b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.273+0000 7f97a9200640 1 --2- 172.21.15.45:0/3598058831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97a4136f70 0x7f97a4141b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.273+0000 7f97a9200640 1 -- 172.21.15.45:0/3598058831 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a4144790 con 0x7f97a40b1df0 2024-04-16T16:22:59.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.273+0000 7f97a9200640 1 --2- 172.21.15.45:0/3598058831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a40b1df0 0x7f97a40b22a0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f97a40add30 tx=0x7f979c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:59.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.274+0000 7f979be00640 1 -- 172.21.15.45:0/3598058831 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c018070 con 0x7f97a40b1df0 2024-04-16T16:22:59.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.274+0000 7f979be00640 1 -- 172.21.15.45:0/3598058831 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f979c003b70 con 0x7f97a40b1df0 2024-04-16T16:22:59.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.274+0000 7f97a9c00640 1 -- 172.21.15.45:0/3598058831 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a4144a70 con 0x7f97a40b1df0 2024-04-16T16:22:59.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.274+0000 7f979be00640 1 -- 172.21.15.45:0/3598058831 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c005070 con 0x7f97a40b1df0 2024-04-16T16:22:59.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.274+0000 7f97a9c00640 1 -- 172.21.15.45:0/3598058831 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a41432b0 con 0x7f97a40b1df0 2024-04-16T16:22:59.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.275+0000 7f97a9c00640 1 -- 172.21.15.45:0/3598058831 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f97a4144540 con 0x7f97a40b1df0 2024-04-16T16:22:59.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.276+0000 7f979be00640 1 -- 172.21.15.45:0/3598058831 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f979c01b070 con 0x7f97a40b1df0 2024-04-16T16:22:59.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.277+0000 7f979be00640 1 --2- 172.21.15.45:0/3598058831 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f978406ca40 0x7f978406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.277+0000 7f979be00640 1 -- 172.21.15.45:0/3598058831 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f979c08b720 con 0x7f97a40b1df0 2024-04-16T16:22:59.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.277+0000 7f979be00640 1 --2- 172.21.15.45:0/3598058831 >> [v2:172.21.15.45:6808/3148487306,v1:172.21.15.45:6809/3148487306] conn(0x7f9784074be0 0x7f9784077000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.278+0000 7f979be00640 1 -- 172.21.15.45:0/3598058831 --> [v2:172.21.15.45:6808/3148487306,v1:172.21.15.45:6809/3148487306] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97840776b0 con 0x7f9784074be0 2024-04-16T16:22:59.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.278+0000 7f979be00640 1 -- 172.21.15.45:0/3598058831 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=156) ==== 24+0+0 (secure 0 0 0) 0x7f979c092050 con 0x7f97a40b1df0 2024-04-16T16:22:59.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.278+0000 7f97a3e00640 1 --2- 172.21.15.45:0/3598058831 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f978406ca40 0x7f978406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.278+0000 7f97a3400640 1 --2- 172.21.15.45:0/3598058831 >> [v2:172.21.15.45:6808/3148487306,v1:172.21.15.45:6809/3148487306] conn(0x7f9784074be0 0x7f9784077000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.279+0000 7f97a3400640 1 --2- 172.21.15.45:0/3598058831 >> [v2:172.21.15.45:6808/3148487306,v1:172.21.15.45:6809/3148487306] conn(0x7f9784074be0 0x7f9784077000 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:59.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.279+0000 7f97a3e00640 1 --2- 172.21.15.45:0/3598058831 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f978406ca40 0x7f978406ef00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9794009a10 tx=0x7f9794008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:59.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.279+0000 7f979be00640 1 -- 172.21.15.45:0/3598058831 <== osd.2 v2:172.21.15.45:6808/3148487306 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97840776b0 con 0x7f9784074be0 2024-04-16T16:22:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.294+0000 7f97a9c00640 1 -- 172.21.15.45:0/3598058831 --> [v2:172.21.15.45:6808/3148487306,v1:172.21.15.45:6809/3148487306] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f97a4075780 con 0x7f9784074be0 2024-04-16T16:22:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.295+0000 7f979be00640 1 -- 172.21.15.45:0/3598058831 <== osd.2 v2:172.21.15.45:6808/3148487306 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f97a4075780 con 0x7f9784074be0 2024-04-16T16:22:59.294 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:22:59.294 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-16T16:22:59.294 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:22:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.295+0000 7f97a9c00640 1 -- 172.21.15.45:0/3598058831 >> [v2:172.21.15.45:6808/3148487306,v1:172.21.15.45:6809/3148487306] conn(0x7f9784074be0 msgr2=0x7f9784077000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.295+0000 7f97a9c00640 1 --2- 172.21.15.45:0/3598058831 >> [v2:172.21.15.45:6808/3148487306,v1:172.21.15.45:6809/3148487306] conn(0x7f9784074be0 0x7f9784077000 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.295+0000 7f97a9c00640 1 -- 172.21.15.45:0/3598058831 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f978406ca40 msgr2=0x7f978406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.295+0000 7f97a9c00640 1 --2- 172.21.15.45:0/3598058831 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f978406ca40 0x7f978406ef00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9794009a10 tx=0x7f9794008040 comp rx=0 tx=0).stop 2024-04-16T16:22:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.295+0000 7f97a9c00640 1 -- 172.21.15.45:0/3598058831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a40b1df0 msgr2=0x7f97a40b22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.295+0000 7f97a9c00640 1 --2- 172.21.15.45:0/3598058831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a40b1df0 0x7f97a40b22a0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f97a40add30 tx=0x7f979c00dc00 comp rx=0 tx=0).stop 2024-04-16T16:22:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.295+0000 7f97a9200640 1 -- 172.21.15.45:0/3598058831 reap_dead start 2024-04-16T16:22:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.295+0000 7f97a9c00640 1 -- 172.21.15.45:0/3598058831 shutdown_connections 2024-04-16T16:22:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.295+0000 7f97a9c00640 1 -- 172.21.15.45:0/3598058831 >> 172.21.15.45:0/3598058831 conn(0x7f97a40b3bb0 msgr2=0x7f97a40b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.295+0000 7f97a9c00640 1 -- 172.21.15.45:0/3598058831 shutdown_connections 2024-04-16T16:22:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.296+0000 7f97a9c00640 1 -- 172.21.15.45:0/3598058831 wait complete. 2024-04-16T16:22:59.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-16T16:22:59.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.539+0000 7f71b7600640 1 Processor -- start 2024-04-16T16:22:59.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.540+0000 7f71b7600640 1 -- start start 2024-04-16T16:22:59.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.541+0000 7f71b7600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f71b00cd930 0x7f71b00cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.541+0000 7f71b7600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71b0146600 con 0x7f71b0072890 2024-04-16T16:22:59.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.541+0000 7f71b7600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71b00cde00 con 0x7f71b0146220 2024-04-16T16:22:59.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.541+0000 7f71b7600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f71b006f840 con 0x7f71b00cd930 2024-04-16T16:22:59.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.541+0000 7f71afe00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f71b0072890 0x7f71b00cd220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45956/0 (socket says 172.21.15.45:45956) 2024-04-16T16:22:59.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.541+0000 7f71afe00640 1 -- 172.21.15.45:0/1601933379 learned_addr learned my addr 172.21.15.45:0/1601933379 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:22:59.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.541+0000 7f71b5600640 1 --2- 172.21.15.45:0/1601933379 >> v2:172.21.15.103:3300/0 conn(0x7f71b00cd930 0x7f71b00cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.542+0000 7f71af400640 1 -- 172.21.15.45:0/1601933379 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2740243647 0 0) 0x7f71b0146600 con 0x7f71b0072890 2024-04-16T16:22:59.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.542+0000 7f71af400640 1 -- 172.21.15.45:0/1601933379 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7198003520 con 0x7f71b0072890 2024-04-16T16:22:59.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.542+0000 7f71af400640 1 -- 172.21.15.45:0/1601933379 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2709950083 0 0) 0x7f7198003520 con 0x7f71b0072890 2024-04-16T16:22:59.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.542+0000 7f71af400640 1 -- 172.21.15.45:0/1601933379 >> v2:172.21.15.103:3300/0 conn(0x7f71b00cd930 msgr2=0x7f71b00cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:59.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.542+0000 7f71af400640 1 --2- 172.21.15.45:0/1601933379 >> v2:172.21.15.103:3300/0 conn(0x7f71b00cd930 0x7f71b00cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.542+0000 7f71af400640 1 -- 172.21.15.45:0/1601933379 >> v1:172.21.15.103:6789/0 conn(0x7f71b0146220 legacy=0x7f71b0072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.542+0000 7f71af400640 1 -- 172.21.15.45:0/1601933379 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b00ce630 con 0x7f71b0072890 2024-04-16T16:22:59.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.542+0000 7f71af400640 1 -- 172.21.15.45:0/1601933379 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f719c001690 con 0x7f71b0072890 2024-04-16T16:22:59.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.542+0000 7f71af400640 1 -- 172.21.15.45:0/1601933379 >> v1:172.21.15.45:6789/0 conn(0x7f71b0072890 legacy=0x7f71b00cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.542+0000 7f71af400640 1 --2- 172.21.15.45:0/1601933379 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7198003a00 0x7f7198003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.542+0000 7f71af400640 1 --2- 172.21.15.45:0/1601933379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7198004800 0x7f7198004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.542+0000 7f71af400640 1 --2- 172.21.15.45:0/1601933379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71980051c0 0x7f719800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.542+0000 7f71af400640 1 -- 172.21.15.45:0/1601933379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f719800c090 con 0x7f7198004800 2024-04-16T16:22:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.542+0000 7f71af400640 1 -- 172.21.15.45:0/1601933379 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f719800c240 con 0x7f71980051c0 2024-04-16T16:22:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.542+0000 7f71af400640 1 -- 172.21.15.45:0/1601933379 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f719800c440 con 0x7f7198003a00 2024-04-16T16:22:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.543+0000 7f71afe00640 1 --2- 172.21.15.45:0/1601933379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7198004800 0x7f7198004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.543+0000 7f71b4c00640 1 --2- 172.21.15.45:0/1601933379 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7198003a00 0x7f7198003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.543+0000 7f71b5600640 1 --2- 172.21.15.45:0/1601933379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71980051c0 0x7f719800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.543+0000 7f71afe00640 1 -- 172.21.15.45:0/1601933379 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7198003a00 msgr2=0x7f7198003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:59.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.543+0000 7f71afe00640 1 --2- 172.21.15.45:0/1601933379 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7198003a00 0x7f7198003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.543+0000 7f71afe00640 1 -- 172.21.15.45:0/1601933379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71980051c0 msgr2=0x7f719800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:22:59.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.543+0000 7f71afe00640 1 --2- 172.21.15.45:0/1601933379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71980051c0 0x7f719800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.543+0000 7f71afe00640 1 -- 172.21.15.45:0/1601933379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7198003520 con 0x7f7198004800 2024-04-16T16:22:59.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.543+0000 7f71b5600640 1 -- 172.21.15.45:0/1601933379 reap_dead start 2024-04-16T16:22:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.544+0000 7f71afe00640 1 --2- 172.21.15.45:0/1601933379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7198004800 0x7f7198004c80 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f719c00c160 tx=0x7f7198005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7014eb881a946886 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.545+0000 7f71af400640 1 -- 172.21.15.45:0/1601933379 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c007c60 con 0x7f7198004800 2024-04-16T16:22:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.545+0000 7f71af400640 1 -- 172.21.15.45:0/1601933379 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f719c030070 con 0x7f7198004800 2024-04-16T16:22:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.545+0000 7f71af400640 1 -- 172.21.15.45:0/1601933379 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c005d30 con 0x7f7198004800 2024-04-16T16:22:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.545+0000 7f71b7600640 1 -- 172.21.15.45:0/1601933379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7198004800 msgr2=0x7f7198004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.545+0000 7f71b7600640 1 --2- 172.21.15.45:0/1601933379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7198004800 0x7f7198004c80 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f719c00c160 tx=0x7f7198005830 comp rx=0 tx=0).stop 2024-04-16T16:22:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.545+0000 7f71b7600640 1 -- 172.21.15.45:0/1601933379 shutdown_connections 2024-04-16T16:22:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.545+0000 7f71b7600640 1 --2- 172.21.15.45:0/1601933379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7198004800 0x7f7198004c80 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.545+0000 7f71b7600640 1 -- 172.21.15.45:0/1601933379 >> 172.21.15.45:0/1601933379 conn(0x7f71b006b930 msgr2=0x7f71b006bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.546+0000 7f71b7600640 1 -- 172.21.15.45:0/1601933379 shutdown_connections 2024-04-16T16:22:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.546+0000 7f71b7600640 1 -- 172.21.15.45:0/1601933379 wait complete. 2024-04-16T16:22:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.548+0000 7f71b7600640 1 Processor -- start 2024-04-16T16:22:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.548+0000 7f71b7600640 1 -- start start 2024-04-16T16:22:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71b7600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7198004800 0x7f71b0158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71b7600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71b0159de0 0x7f71b00a9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71b7600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71b015a260 0x7f71b00a9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71b7600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f71b00cebf0 con 0x7f7198004800 2024-04-16T16:22:59.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71b7600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f71b00ce0e0 con 0x7f71b015a260 2024-04-16T16:22:59.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71b7600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f71b00cf110 con 0x7f71b0159de0 2024-04-16T16:22:59.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71b4c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7198004800 0x7f71b0158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71b4c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7198004800 0x7f71b0158920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47364/0 (socket says 172.21.15.45:47364) 2024-04-16T16:22:59.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71b4c00640 1 -- 172.21.15.45:0/767951589 learned_addr learned my addr 172.21.15.45:0/767951589 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:22:59.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71afe00640 1 --2- 172.21.15.45:0/767951589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71b0159de0 0x7f71b00a9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71b5600640 1 --2- 172.21.15.45:0/767951589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71b015a260 0x7f71b00a9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71b4c00640 1 -- 172.21.15.45:0/767951589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71b0159de0 msgr2=0x7f71b00a9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71b4c00640 1 --2- 172.21.15.45:0/767951589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71b0159de0 0x7f71b00a9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71b4c00640 1 -- 172.21.15.45:0/767951589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71b015a260 msgr2=0x7f71b00a9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71b4c00640 1 --2- 172.21.15.45:0/767951589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71b015a260 0x7f71b00a9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.549+0000 7f71b4c00640 1 -- 172.21.15.45:0/767951589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b01355c0 con 0x7f7198004800 2024-04-16T16:22:59.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.550+0000 7f71afe00640 1 --2- 172.21.15.45:0/767951589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71b0159de0 0x7f71b00a9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:22:59.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.550+0000 7f71b4c00640 1 --2- 172.21.15.45:0/767951589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7198004800 0x7f71b0158920 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f71a400e480 tx=0x7f71a400e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:59.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.550+0000 7f71acc00640 1 -- 172.21.15.45:0/767951589 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a40052e0 con 0x7f7198004800 2024-04-16T16:22:59.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.550+0000 7f71acc00640 1 -- 172.21.15.45:0/767951589 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71a4018070 con 0x7f7198004800 2024-04-16T16:22:59.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.550+0000 7f71acc00640 1 -- 172.21.15.45:0/767951589 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a4003810 con 0x7f7198004800 2024-04-16T16:22:59.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.550+0000 7f71b7600640 1 -- 172.21.15.45:0/767951589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71b01358a0 con 0x7f7198004800 2024-04-16T16:22:59.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.550+0000 7f71b7600640 1 -- 172.21.15.45:0/767951589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71b0135e60 con 0x7f7198004800 2024-04-16T16:22:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.552+0000 7f71b7600640 1 -- 172.21.15.45:0/767951589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71b0075780 con 0x7f7198004800 2024-04-16T16:22:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.553+0000 7f71acc00640 1 -- 172.21.15.45:0/767951589 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f71a40039b0 con 0x7f7198004800 2024-04-16T16:22:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.553+0000 7f71acc00640 1 --2- 172.21.15.45:0/767951589 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f719006ca40 0x7f719006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:22:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.553+0000 7f71acc00640 1 -- 172.21.15.45:0/767951589 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f71a408ba20 con 0x7f7198004800 2024-04-16T16:22:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.554+0000 7f71afe00640 1 --2- 172.21.15.45:0/767951589 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f719006ca40 0x7f719006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:22:59.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.555+0000 7f71afe00640 1 --2- 172.21.15.45:0/767951589 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f719006ca40 0x7f719006ef00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f719c034000 tx=0x7f719c039040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:22:59.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.558+0000 7f71acc00640 1 -- 172.21.15.45:0/767951589 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f71a4059d30 con 0x7f7198004800 2024-04-16T16:22:59.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.765+0000 7f71b7600640 1 -- 172.21.15.45:0/767951589 --> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f71b00aff40 con 0x7f719006ca40 2024-04-16T16:22:59.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.767+0000 7f71acc00640 1 -- 172.21.15.45:0/767951589 <== mgr.7591 v2:172.21.15.103:6832/1129061436 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f71b00aff40 con 0x7f719006ca40 2024-04-16T16:22:59.766 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 2 to deep-scrub 2024-04-16T16:22:59.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.769+0000 7f71b7600640 1 -- 172.21.15.45:0/767951589 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f719006ca40 msgr2=0x7f719006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.769+0000 7f71b7600640 1 --2- 172.21.15.45:0/767951589 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f719006ca40 0x7f719006ef00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f719c034000 tx=0x7f719c039040 comp rx=0 tx=0).stop 2024-04-16T16:22:59.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.770+0000 7f71b7600640 1 -- 172.21.15.45:0/767951589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7198004800 msgr2=0x7f71b0158920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:22:59.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.770+0000 7f71b7600640 1 --2- 172.21.15.45:0/767951589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7198004800 0x7f71b0158920 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f71a400e480 tx=0x7f71a400e940 comp rx=0 tx=0).stop 2024-04-16T16:22:59.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.770+0000 7f71b7600640 1 -- 172.21.15.45:0/767951589 shutdown_connections 2024-04-16T16:22:59.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.770+0000 7f71b7600640 1 --2- 172.21.15.45:0/767951589 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f719006ca40 0x7f719006ef00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.770+0000 7f71b7600640 1 --2- 172.21.15.45:0/767951589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71b015a260 0x7f71b00a9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.770+0000 7f71b7600640 1 --2- 172.21.15.45:0/767951589 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71b0159de0 0x7f71b00a9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.770+0000 7f71b7600640 1 --2- 172.21.15.45:0/767951589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7198004800 0x7f71b0158920 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:22:59.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.770+0000 7f71b7600640 1 -- 172.21.15.45:0/767951589 >> 172.21.15.45:0/767951589 conn(0x7f71b006b930 msgr2=0x7f71b015df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:22:59.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.770+0000 7f71b7600640 1 -- 172.21.15.45:0/767951589 shutdown_connections 2024-04-16T16:22:59.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:22:59.770+0000 7f71b7600640 1 -- 172.21.15.45:0/767951589 wait complete. 2024-04-16T16:22:59.789 INFO:tasks.ceph:Scrubbing osd.3 2024-04-16T16:22:59.789 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-16T16:23:00.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.032+0000 7fc3bda00640 1 Processor -- start 2024-04-16T16:23:00.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.033+0000 7fc3bda00640 1 -- start start 2024-04-16T16:23:00.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.034+0000 7fc3bda00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3b8155ab0 0x7fc3b8162ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.034+0000 7fc3bda00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc3b80ba610 0x7fc3b815efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.034+0000 7fc3bda00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3b8155e90 con 0x7fc3b80d18c0 2024-04-16T16:23:00.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.034+0000 7fc3bda00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc3b815ff00 con 0x7fc3b80ba610 2024-04-16T16:23:00.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.034+0000 7fc3bda00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc3b815f940 con 0x7fc3b8155ab0 2024-04-16T16:23:00.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.035+0000 7fc3b7400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc3b80ba610 0x7fc3b815efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.035+0000 7fc3b7400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc3b80ba610 0x7fc3b815efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47368/0 (socket says 172.21.15.45:47368) 2024-04-16T16:23:00.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.035+0000 7fc3b7400640 1 -- 172.21.15.45:0/3089233739 learned_addr learned my addr 172.21.15.45:0/3089233739 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:00.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.035+0000 7fc3b7e00640 1 --2- 172.21.15.45:0/3089233739 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3b8155ab0 0x7fc3b8162ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.035+0000 7fc3b7400640 1 -- 172.21.15.45:0/3089233739 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3b8155ab0 msgr2=0x7fc3b8162ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:00.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.035+0000 7fc3b7400640 1 --2- 172.21.15.45:0/3089233739 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3b8155ab0 0x7fc3b8162ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.035+0000 7fc3b7400640 1 -- 172.21.15.45:0/3089233739 >> v1:172.21.15.45:6789/0 conn(0x7fc3b80d18c0 legacy=0x7fc3b80d2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.035+0000 7fc3b7400640 1 -- 172.21.15.45:0/3089233739 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b815fc20 con 0x7fc3b80ba610 2024-04-16T16:23:00.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.037+0000 7fc3b7400640 1 --2- 172.21.15.45:0/3089233739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3b80ba610 0x7fc3b815efd0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0009ea0 tx=0x7fc3a002bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57287e36ea76929 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:00.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.037+0000 7fc3b6a00640 1 -- 172.21.15.45:0/3089233739 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a002cac0 con 0x7fc3b80ba610 2024-04-16T16:23:00.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.037+0000 7fc3b6a00640 1 -- 172.21.15.45:0/3089233739 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3a002cc60 con 0x7fc3b80ba610 2024-04-16T16:23:00.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.037+0000 7fc3b6a00640 1 -- 172.21.15.45:0/3089233739 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0034610 con 0x7fc3b80ba610 2024-04-16T16:23:00.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.038+0000 7fc3bda00640 1 -- 172.21.15.45:0/3089233739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3b80ba610 msgr2=0x7fc3b815efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.038+0000 7fc3bda00640 1 --2- 172.21.15.45:0/3089233739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3b80ba610 0x7fc3b815efd0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0009ea0 tx=0x7fc3a002bf20 comp rx=0 tx=0).stop 2024-04-16T16:23:00.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.038+0000 7fc3bda00640 1 -- 172.21.15.45:0/3089233739 shutdown_connections 2024-04-16T16:23:00.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.038+0000 7fc3bda00640 1 --2- 172.21.15.45:0/3089233739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3b80ba610 0x7fc3b815efd0 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.038+0000 7fc3bda00640 1 --2- 172.21.15.45:0/3089233739 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3b8155ab0 0x7fc3b8162ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.038+0000 7fc3bda00640 1 -- 172.21.15.45:0/3089233739 >> 172.21.15.45:0/3089233739 conn(0x7fc3b806d1a0 msgr2=0x7fc3b806d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:00.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.038+0000 7fc3bda00640 1 -- 172.21.15.45:0/3089233739 shutdown_connections 2024-04-16T16:23:00.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.038+0000 7fc3bda00640 1 -- 172.21.15.45:0/3089233739 wait complete. 2024-04-16T16:23:00.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.040+0000 7fc3bda00640 1 Processor -- start 2024-04-16T16:23:00.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.040+0000 7fc3bda00640 1 -- start start 2024-04-16T16:23:00.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.041+0000 7fc3bda00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3b80ba610 0x7fc3b815d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.041+0000 7fc3bda00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3b80d18c0 0x7fc3b8158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.041+0000 7fc3bda00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3b8155ab0 0x7fc3b8158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.041+0000 7fc3bda00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b8163430 con 0x7fc3b8155ab0 2024-04-16T16:23:00.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.041+0000 7fc3bda00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b81629d0 con 0x7fc3b80d18c0 2024-04-16T16:23:00.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.041+0000 7fc3bda00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc3b815f510 con 0x7fc3b80ba610 2024-04-16T16:23:00.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.041+0000 7fc3bd000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3b8155ab0 0x7fc3b8158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.041+0000 7fc3bd000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3b8155ab0 0x7fc3b8158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47374/0 (socket says 172.21.15.45:47374) 2024-04-16T16:23:00.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.041+0000 7fc3bd000640 1 -- 172.21.15.45:0/3014261595 learned_addr learned my addr 172.21.15.45:0/3014261595 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:00.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.041+0000 7fc3b7e00640 1 --2- 172.21.15.45:0/3014261595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3b80ba610 0x7fc3b815d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.042+0000 7fc3b7400640 1 --2- 172.21.15.45:0/3014261595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3b80d18c0 0x7fc3b8158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.042+0000 7fc3bd000640 1 -- 172.21.15.45:0/3014261595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3b80ba610 msgr2=0x7fc3b815d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.042+0000 7fc3bd000640 1 --2- 172.21.15.45:0/3014261595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3b80ba610 0x7fc3b815d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.042+0000 7fc3bd000640 1 -- 172.21.15.45:0/3014261595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3b80d18c0 msgr2=0x7fc3b8158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.042+0000 7fc3bd000640 1 --2- 172.21.15.45:0/3014261595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3b80d18c0 0x7fc3b8158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.042+0000 7fc3bd000640 1 -- 172.21.15.45:0/3014261595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b8147250 con 0x7fc3b8155ab0 2024-04-16T16:23:00.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.042+0000 7fc3b7400640 1 --2- 172.21.15.45:0/3014261595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3b80d18c0 0x7fc3b8158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:23:00.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.042+0000 7fc3bd000640 1 --2- 172.21.15.45:0/3014261595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3b8155ab0 0x7fc3b8158770 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0001eb0 tx=0x7fc3b000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:00.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.042+0000 7fc3afe00640 1 -- 172.21.15.45:0/3014261595 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0017070 con 0x7fc3b8155ab0 2024-04-16T16:23:00.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.042+0000 7fc3afe00640 1 -- 172.21.15.45:0/3014261595 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3b00052d0 con 0x7fc3b8155ab0 2024-04-16T16:23:00.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.042+0000 7fc3afe00640 1 -- 172.21.15.45:0/3014261595 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0003810 con 0x7fc3b8155ab0 2024-04-16T16:23:00.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.042+0000 7fc3bda00640 1 -- 172.21.15.45:0/3014261595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3b80cbc50 con 0x7fc3b8155ab0 2024-04-16T16:23:00.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.042+0000 7fc3bda00640 1 -- 172.21.15.45:0/3014261595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3b815df10 con 0x7fc3b8155ab0 2024-04-16T16:23:00.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.043+0000 7fc3bda00640 1 -- 172.21.15.45:0/3014261595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc3b80cc310 con 0x7fc3b8155ab0 2024-04-16T16:23:00.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.045+0000 7fc3afe00640 1 -- 172.21.15.45:0/3014261595 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fc3b001a070 con 0x7fc3b8155ab0 2024-04-16T16:23:00.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.045+0000 7fc3afe00640 1 --2- 172.21.15.45:0/3014261595 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc39406ca10 0x7fc39406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.046+0000 7fc3afe00640 1 -- 172.21.15.45:0/3014261595 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7fc3b008b120 con 0x7fc3b8155ab0 2024-04-16T16:23:00.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.046+0000 7fc3afe00640 1 --2- 172.21.15.45:0/3014261595 >> [v2:172.21.15.45:6816/707516363,v1:172.21.15.45:6817/707516363] conn(0x7fc394074bb0 0x7fc394076fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.046+0000 7fc3afe00640 1 -- 172.21.15.45:0/3014261595 --> [v2:172.21.15.45:6816/707516363,v1:172.21.15.45:6817/707516363] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc3b0015e70 con 0x7fc394074bb0 2024-04-16T16:23:00.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.046+0000 7fc3afe00640 1 -- 172.21.15.45:0/3014261595 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=156) ==== 24+0+0 (secure 0 0 0) 0x7fc3b0092050 con 0x7fc3b8155ab0 2024-04-16T16:23:00.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.046+0000 7fc3b7400640 1 --2- 172.21.15.45:0/3014261595 >> [v2:172.21.15.45:6816/707516363,v1:172.21.15.45:6817/707516363] conn(0x7fc394074bb0 0x7fc394076fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.046+0000 7fc3b7e00640 1 --2- 172.21.15.45:0/3014261595 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc39406ca10 0x7fc39406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.047+0000 7fc3b7400640 1 --2- 172.21.15.45:0/3014261595 >> [v2:172.21.15.45:6816/707516363,v1:172.21.15.45:6817/707516363] conn(0x7fc394074bb0 0x7fc394076fd0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:00.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.047+0000 7fc3b7e00640 1 --2- 172.21.15.45:0/3014261595 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc39406ca10 0x7fc39406eed0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8009a30 tx=0x7fc3a8008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:00.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.048+0000 7fc3afe00640 1 -- 172.21.15.45:0/3014261595 <== osd.3 v2:172.21.15.45:6816/707516363 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc3b0015e70 con 0x7fc394074bb0 2024-04-16T16:23:00.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.062+0000 7fc3bda00640 1 -- 172.21.15.45:0/3014261595 --> [v2:172.21.15.45:6816/707516363,v1:172.21.15.45:6817/707516363] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc3b8075780 con 0x7fc394074bb0 2024-04-16T16:23:00.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.063+0000 7fc3afe00640 1 -- 172.21.15.45:0/3014261595 <== osd.3 v2:172.21.15.45:6816/707516363 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc3b8075780 con 0x7fc394074bb0 2024-04-16T16:23:00.062 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:23:00.062 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-16T16:23:00.062 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:23:00.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.063+0000 7fc3bda00640 1 -- 172.21.15.45:0/3014261595 >> [v2:172.21.15.45:6816/707516363,v1:172.21.15.45:6817/707516363] conn(0x7fc394074bb0 msgr2=0x7fc394076fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.063+0000 7fc3bda00640 1 --2- 172.21.15.45:0/3014261595 >> [v2:172.21.15.45:6816/707516363,v1:172.21.15.45:6817/707516363] conn(0x7fc394074bb0 0x7fc394076fd0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.064+0000 7fc3bda00640 1 -- 172.21.15.45:0/3014261595 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc39406ca10 msgr2=0x7fc39406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.064+0000 7fc3bda00640 1 --2- 172.21.15.45:0/3014261595 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fc39406ca10 0x7fc39406eed0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8009a30 tx=0x7fc3a8008040 comp rx=0 tx=0).stop 2024-04-16T16:23:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.064+0000 7fc3bda00640 1 -- 172.21.15.45:0/3014261595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3b8155ab0 msgr2=0x7fc3b8158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.064+0000 7fc3bda00640 1 --2- 172.21.15.45:0/3014261595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3b8155ab0 0x7fc3b8158770 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0001eb0 tx=0x7fc3b000c890 comp rx=0 tx=0).stop 2024-04-16T16:23:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.064+0000 7fc3bd000640 1 -- 172.21.15.45:0/3014261595 reap_dead start 2024-04-16T16:23:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.064+0000 7fc3bda00640 1 -- 172.21.15.45:0/3014261595 shutdown_connections 2024-04-16T16:23:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.064+0000 7fc3bda00640 1 -- 172.21.15.45:0/3014261595 >> 172.21.15.45:0/3014261595 conn(0x7fc3b806d1a0 msgr2=0x7fc3b80737b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.064+0000 7fc3bda00640 1 -- 172.21.15.45:0/3014261595 shutdown_connections 2024-04-16T16:23:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.064+0000 7fc3bda00640 1 -- 172.21.15.45:0/3014261595 wait complete. 2024-04-16T16:23:00.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-16T16:23:00.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.301+0000 7f6483e00640 1 Processor -- start 2024-04-16T16:23:00.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.302+0000 7f6483e00640 1 -- start start 2024-04-16T16:23:00.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.303+0000 7f6483e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f64840c6fc0 0x7f648406f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.303+0000 7f6483e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648415cdc0 0x7f64840c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.303+0000 7f6483e00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f648415d600 con 0x7f64840c6320 2024-04-16T16:23:00.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.303+0000 7f6483e00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f64840c5670 con 0x7f64840c6fc0 2024-04-16T16:23:00.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.303+0000 7f6483e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f648415d8e0 con 0x7f648415cdc0 2024-04-16T16:23:00.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.304+0000 7f6482a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f64840c6fc0 0x7f648406f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.304+0000 7f6482a00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f64840c6fc0 0x7f648406f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47382/0 (socket says 172.21.15.45:47382) 2024-04-16T16:23:00.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.304+0000 7f6482a00640 1 -- 172.21.15.45:0/807776915 learned_addr learned my addr 172.21.15.45:0/807776915 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:00.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.304+0000 7f647be00640 1 --2- 172.21.15.45:0/807776915 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648415cdc0 0x7f64840c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.304+0000 7f6482a00640 1 -- 172.21.15.45:0/807776915 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648415cdc0 msgr2=0x7f64840c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:00.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.304+0000 7f6482a00640 1 --2- 172.21.15.45:0/807776915 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648415cdc0 0x7f64840c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.304+0000 7f6482a00640 1 -- 172.21.15.45:0/807776915 >> v1:172.21.15.45:6789/0 conn(0x7f64840c6320 legacy=0x7f648406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.304+0000 7f6482a00640 1 -- 172.21.15.45:0/807776915 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648415dbc0 con 0x7f64840c6fc0 2024-04-16T16:23:00.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.305+0000 7f6482a00640 1 --2- 172.21.15.45:0/807776915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64840c6fc0 0x7f648406f730 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f6474009870 tx=0x7f647402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6232ead871204da server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:00.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.306+0000 7f6482000640 1 -- 172.21.15.45:0/807776915 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474034070 con 0x7f64840c6fc0 2024-04-16T16:23:00.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.306+0000 7f6482000640 1 -- 172.21.15.45:0/807776915 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6474030840 con 0x7f64840c6fc0 2024-04-16T16:23:00.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.306+0000 7f6482000640 1 -- 172.21.15.45:0/807776915 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474030c10 con 0x7f64840c6fc0 2024-04-16T16:23:00.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.307+0000 7f6483e00640 1 -- 172.21.15.45:0/807776915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64840c6fc0 msgr2=0x7f648406f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.307+0000 7f6483e00640 1 --2- 172.21.15.45:0/807776915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64840c6fc0 0x7f648406f730 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f6474009870 tx=0x7f647402b980 comp rx=0 tx=0).stop 2024-04-16T16:23:00.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.307+0000 7f6483e00640 1 -- 172.21.15.45:0/807776915 shutdown_connections 2024-04-16T16:23:00.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.307+0000 7f6483e00640 1 --2- 172.21.15.45:0/807776915 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648415cdc0 0x7f64840c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.307+0000 7f6483e00640 1 --2- 172.21.15.45:0/807776915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64840c6fc0 0x7f648406f730 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.307+0000 7f6483e00640 1 -- 172.21.15.45:0/807776915 >> 172.21.15.45:0/807776915 conn(0x7f648406b930 msgr2=0x7f648406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:00.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.307+0000 7f6483e00640 1 -- 172.21.15.45:0/807776915 shutdown_connections 2024-04-16T16:23:00.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.307+0000 7f6483e00640 1 -- 172.21.15.45:0/807776915 wait complete. 2024-04-16T16:23:00.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.309+0000 7f6483e00640 1 Processor -- start 2024-04-16T16:23:00.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.309+0000 7f6483e00640 1 -- start start 2024-04-16T16:23:00.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.310+0000 7f6483e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64840c6320 0x7f6484148900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.310+0000 7f6483e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64840c6fc0 0x7f64841455f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.310+0000 7f6483e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648415cdc0 0x7f6484147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.310+0000 7f6483e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f64840c73a0 con 0x7f64840c6320 2024-04-16T16:23:00.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.310+0000 7f6483e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f648415d270 con 0x7f64840c6fc0 2024-04-16T16:23:00.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.310+0000 7f6483e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f64840c7520 con 0x7f648415cdc0 2024-04-16T16:23:00.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.310+0000 7f6482a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64840c6320 0x7f6484148900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.310+0000 7f6482a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64840c6320 0x7f6484148900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47386/0 (socket says 172.21.15.45:47386) 2024-04-16T16:23:00.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.310+0000 7f6482a00640 1 -- 172.21.15.45:0/262427373 learned_addr learned my addr 172.21.15.45:0/262427373 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:00.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.310+0000 7f647be00640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64840c6fc0 0x7f64841455f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.310+0000 7f6482a00640 1 -- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648415cdc0 msgr2=0x7f6484147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.310+0000 7f6483400640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648415cdc0 0x7f6484147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.311+0000 7f6482a00640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648415cdc0 0x7f6484147510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.311+0000 7f6482a00640 1 -- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64840c6fc0 msgr2=0x7f64841455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.311+0000 7f6482a00640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64840c6fc0 0x7f64841455f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.311+0000 7f6482a00640 1 -- 172.21.15.45:0/262427373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6484148e40 con 0x7f64840c6320 2024-04-16T16:23:00.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.311+0000 7f647be00640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64840c6fc0 0x7f64841455f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:23:00.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.311+0000 7f6483400640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648415cdc0 0x7f6484147510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:23:00.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.311+0000 7f6482a00640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64840c6320 0x7f6484148900 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f6474000c00 tx=0x7f6474004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:00.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.311+0000 7f647b400640 1 -- 172.21.15.45:0/262427373 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474048070 con 0x7f64840c6320 2024-04-16T16:23:00.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.311+0000 7f647b400640 1 -- 172.21.15.45:0/262427373 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6474030640 con 0x7f64840c6320 2024-04-16T16:23:00.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.311+0000 7f647b400640 1 -- 172.21.15.45:0/262427373 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474034040 con 0x7f64840c6320 2024-04-16T16:23:00.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.311+0000 7f6483e00640 1 -- 172.21.15.45:0/262427373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6484146470 con 0x7f64840c6320 2024-04-16T16:23:00.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.312+0000 7f6483e00640 1 -- 172.21.15.45:0/262427373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6484146a30 con 0x7f64840c6320 2024-04-16T16:23:00.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.313+0000 7f6483e00640 1 -- 172.21.15.45:0/262427373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6484075780 con 0x7f64840c6320 2024-04-16T16:23:00.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.314+0000 7f647b400640 1 -- 172.21.15.45:0/262427373 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f647404b070 con 0x7f64840c6320 2024-04-16T16:23:00.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.314+0000 7f647b400640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f646006ca10 0x7f646006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.315+0000 7f647b400640 1 -- 172.21.15.45:0/262427373 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f64740adda0 con 0x7f64840c6320 2024-04-16T16:23:00.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.315+0000 7f647be00640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f646006ca10 0x7f646006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.316+0000 7f647be00640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f646006ca10 0x7f646006eed0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f646c006fd0 tx=0x7f646c008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:00.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.319+0000 7f647b400640 1 -- 172.21.15.45:0/262427373 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f647407bf10 con 0x7f64840c6320 2024-04-16T16:23:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.523+0000 7f6483e00640 1 -- 172.21.15.45:0/262427373 --> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f64840b2920 con 0x7f646006ca10 2024-04-16T16:23:00.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.525+0000 7f647b400640 1 -- 172.21.15.45:0/262427373 <== mgr.7591 v2:172.21.15.103:6832/1129061436 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f64840b2920 con 0x7f646006ca10 2024-04-16T16:23:00.524 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 3 to deep-scrub 2024-04-16T16:23:00.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.527+0000 7f6483e00640 1 -- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f646006ca10 msgr2=0x7f646006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.527+0000 7f6483e00640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f646006ca10 0x7f646006eed0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f646c006fd0 tx=0x7f646c008040 comp rx=0 tx=0).stop 2024-04-16T16:23:00.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.527+0000 7f6483e00640 1 -- 172.21.15.45:0/262427373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64840c6320 msgr2=0x7f6484148900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.527+0000 7f6483e00640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64840c6320 0x7f6484148900 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f6474000c00 tx=0x7f6474004730 comp rx=0 tx=0).stop 2024-04-16T16:23:00.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.528+0000 7f6483e00640 1 -- 172.21.15.45:0/262427373 shutdown_connections 2024-04-16T16:23:00.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.528+0000 7f6483e00640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f646006ca10 0x7f646006eed0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.528+0000 7f6483e00640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f648415cdc0 0x7f6484147510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.528+0000 7f6483e00640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64840c6fc0 0x7f64841455f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.528+0000 7f6483e00640 1 --2- 172.21.15.45:0/262427373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64840c6320 0x7f6484148900 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.528+0000 7f6483e00640 1 -- 172.21.15.45:0/262427373 >> 172.21.15.45:0/262427373 conn(0x7f648406b930 msgr2=0x7f648415f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:00.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.528+0000 7f6483e00640 1 -- 172.21.15.45:0/262427373 shutdown_connections 2024-04-16T16:23:00.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.528+0000 7f6483e00640 1 -- 172.21.15.45:0/262427373 wait complete. 2024-04-16T16:23:00.545 INFO:tasks.ceph:Scrubbing osd.4 2024-04-16T16:23:00.545 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-16T16:23:00.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.780+0000 7f1ebc800640 1 Processor -- start 2024-04-16T16:23:00.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.780+0000 7f1ebc800640 1 -- start start 2024-04-16T16:23:00.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.781+0000 7f1ebc800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb40d10e0 0x7f1eb4072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.781+0000 7f1ebc800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1eb40b4930 0x7f1eb406f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.782+0000 7f1ebc800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eb40b5530 con 0x7f1eb415c7c0 2024-04-16T16:23:00.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.782+0000 7f1ebc800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1eb40b52e0 con 0x7f1eb40b4930 2024-04-16T16:23:00.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.782+0000 7f1ebc800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1eb40d0260 con 0x7f1eb40d10e0 2024-04-16T16:23:00.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.782+0000 7f1eba800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1eb40b4930 0x7f1eb406f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.782+0000 7f1eb9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb40d10e0 0x7f1eb4072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.782+0000 7f1eb9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb40d10e0 0x7f1eb4072e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:53334/0 (socket says 172.21.15.45:53334) 2024-04-16T16:23:00.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.782+0000 7f1eb9400640 1 -- 172.21.15.45:0/1393359915 learned_addr learned my addr 172.21.15.45:0/1393359915 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.783+0000 7f1eb9400640 1 -- 172.21.15.45:0/1393359915 >> v2:172.21.15.103:3300/0 conn(0x7f1eb40b4930 msgr2=0x7f1eb406f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.783+0000 7f1eb9400640 1 --2- 172.21.15.45:0/1393359915 >> v2:172.21.15.103:3300/0 conn(0x7f1eb40b4930 0x7f1eb406f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.783+0000 7f1eb9400640 1 -- 172.21.15.45:0/1393359915 >> v1:172.21.15.103:6789/0 conn(0x7f1eb415c7c0 legacy=0x7f1eb40d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.783+0000 7f1eb9400640 1 -- 172.21.15.45:0/1393359915 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb40b5810 con 0x7f1eb40d10e0 2024-04-16T16:23:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.783+0000 7f1eb8a00640 1 -- 172.21.15.45:0/1393359915 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3704667712 0 0) 0x7f1eb40b5530 con 0x7f1eb415c7c0 2024-04-16T16:23:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.784+0000 7f1eb9400640 1 --2- 172.21.15.45:0/1393359915 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb40d10e0 0x7f1eb4072e10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009870 tx=0x7f1ea402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29e4c08eaaa942f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:00.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.785+0000 7f1eb8a00640 1 -- 172.21.15.45:0/1393359915 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea40317c0 con 0x7f1eb40d10e0 2024-04-16T16:23:00.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.785+0000 7f1eb8a00640 1 -- 172.21.15.45:0/1393359915 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ea4037070 con 0x7f1eb40d10e0 2024-04-16T16:23:00.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.785+0000 7f1eb8a00640 1 -- 172.21.15.45:0/1393359915 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4031b80 con 0x7f1eb40d10e0 2024-04-16T16:23:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.786+0000 7f1ebc800640 1 -- 172.21.15.45:0/1393359915 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb40d10e0 msgr2=0x7f1eb4072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.786+0000 7f1ebc800640 1 --2- 172.21.15.45:0/1393359915 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb40d10e0 0x7f1eb4072e10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009870 tx=0x7f1ea402b860 comp rx=0 tx=0).stop 2024-04-16T16:23:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.786+0000 7f1ebc800640 1 -- 172.21.15.45:0/1393359915 shutdown_connections 2024-04-16T16:23:00.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.786+0000 7f1ebc800640 1 --2- 172.21.15.45:0/1393359915 >> v2:172.21.15.103:3300/0 conn(0x7f1eb40b4930 0x7f1eb406f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.786+0000 7f1ebc800640 1 --2- 172.21.15.45:0/1393359915 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb40d10e0 0x7f1eb4072e10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.786+0000 7f1ebc800640 1 -- 172.21.15.45:0/1393359915 >> 172.21.15.45:0/1393359915 conn(0x7f1eb40b3970 msgr2=0x7f1eb40b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:00.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.786+0000 7f1ebc800640 1 -- 172.21.15.45:0/1393359915 shutdown_connections 2024-04-16T16:23:00.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.786+0000 7f1ebc800640 1 -- 172.21.15.45:0/1393359915 wait complete. 2024-04-16T16:23:00.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.788+0000 7f1ebc800640 1 Processor -- start 2024-04-16T16:23:00.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.788+0000 7f1ebc800640 1 -- start start 2024-04-16T16:23:00.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.789+0000 7f1ebc800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eb40b4930 0x7f1eb4158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.789+0000 7f1ebc800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eb40d10e0 0x7f1eb415ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.789+0000 7f1ebc800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb415c7c0 0x7f1eb415f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.789+0000 7f1ebc800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb406f6a0 con 0x7f1eb40d10e0 2024-04-16T16:23:00.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.789+0000 7f1ebc800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb4073350 con 0x7f1eb40b4930 2024-04-16T16:23:00.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.789+0000 7f1ebc800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1eb406f820 con 0x7f1eb415c7c0 2024-04-16T16:23:00.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.789+0000 7f1eb9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eb40d10e0 0x7f1eb415ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.789+0000 7f1eb9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eb40d10e0 0x7f1eb415ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47398/0 (socket says 172.21.15.45:47398) 2024-04-16T16:23:00.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.789+0000 7f1eb9400640 1 -- 172.21.15.45:0/4238246202 learned_addr learned my addr 172.21.15.45:0/4238246202 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:00.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.790+0000 7f1eb9e00640 1 --2- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eb40b4930 0x7f1eb4158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.790+0000 7f1eba800640 1 --2- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb415c7c0 0x7f1eb415f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.790+0000 7f1eb9400640 1 -- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb415c7c0 msgr2=0x7f1eb415f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.790+0000 7f1eb9400640 1 --2- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1eb415c7c0 0x7f1eb415f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.790+0000 7f1eb9400640 1 -- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eb40b4930 msgr2=0x7f1eb4158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.790+0000 7f1eb9400640 1 --2- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eb40b4930 0x7f1eb4158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.790+0000 7f1eb9400640 1 -- 172.21.15.45:0/4238246202 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb41e74d0 con 0x7f1eb40d10e0 2024-04-16T16:23:00.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.790+0000 7f1eb9400640 1 --2- 172.21.15.45:0/4238246202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eb40d10e0 0x7f1eb415ed70 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4034040 tx=0x7f1ea40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:00.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.791+0000 7f1eae000640 1 -- 172.21.15.45:0/4238246202 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4004030 con 0x7f1eb40d10e0 2024-04-16T16:23:00.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.791+0000 7f1eae000640 1 -- 172.21.15.45:0/4238246202 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ea40041d0 con 0x7f1eb40d10e0 2024-04-16T16:23:00.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.791+0000 7f1eae000640 1 -- 172.21.15.45:0/4238246202 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4003610 con 0x7f1eb40d10e0 2024-04-16T16:23:00.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.791+0000 7f1ebc800640 1 -- 172.21.15.45:0/4238246202 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eb41e77b0 con 0x7f1eb40d10e0 2024-04-16T16:23:00.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.791+0000 7f1ebc800640 1 -- 172.21.15.45:0/4238246202 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eb41e7d70 con 0x7f1eb40d10e0 2024-04-16T16:23:00.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.792+0000 7f1ebc800640 1 -- 172.21.15.45:0/4238246202 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1eb4075780 con 0x7f1eb40d10e0 2024-04-16T16:23:00.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.793+0000 7f1eae000640 1 -- 172.21.15.45:0/4238246202 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f1ea404c060 con 0x7f1eb40d10e0 2024-04-16T16:23:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.794+0000 7f1eae000640 1 --2- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f1e9806ca10 0x7f1e9806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.794+0000 7f1eae000640 1 -- 172.21.15.45:0/4238246202 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f1ea40ad670 con 0x7f1eb40d10e0 2024-04-16T16:23:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.794+0000 7f1eae000640 1 --2- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:6800/4229500350,v1:172.21.15.103:6801/4229500350] conn(0x7f1e98074bb0 0x7f1e98076fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.794+0000 7f1eae000640 1 -- 172.21.15.45:0/4238246202 --> [v2:172.21.15.103:6800/4229500350,v1:172.21.15.103:6801/4229500350] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ea4003e50 con 0x7f1e98074bb0 2024-04-16T16:23:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.794+0000 7f1eae000640 1 -- 172.21.15.45:0/4238246202 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=156) ==== 24+0+0 (secure 0 0 0) 0x7f1ea40b4050 con 0x7f1eb40d10e0 2024-04-16T16:23:00.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.795+0000 7f1eb9e00640 1 --2- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f1e9806ca10 0x7f1e9806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.795+0000 7f1eba800640 1 --2- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:6800/4229500350,v1:172.21.15.103:6801/4229500350] conn(0x7f1e98074bb0 0x7f1e98076fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.796+0000 7f1eba800640 1 --2- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:6800/4229500350,v1:172.21.15.103:6801/4229500350] conn(0x7f1e98074bb0 0x7f1e98076fd0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.796+0000 7f1eb9e00640 1 --2- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f1e9806ca10 0x7f1e9806eed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8000f60 tx=0x7f1ea8009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:00.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.799+0000 7f1eae000640 1 -- 172.21.15.45:0/4238246202 <== osd.4 v2:172.21.15.103:6800/4229500350 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1ea4003e50 con 0x7f1e98074bb0 2024-04-16T16:23:00.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.814+0000 7f1ebc800640 1 -- 172.21.15.45:0/4238246202 --> [v2:172.21.15.103:6800/4229500350,v1:172.21.15.103:6801/4229500350] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1eb40d1700 con 0x7f1e98074bb0 2024-04-16T16:23:00.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.815+0000 7f1eae000640 1 -- 172.21.15.45:0/4238246202 <== osd.4 v2:172.21.15.103:6800/4229500350 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1eb40d1700 con 0x7f1e98074bb0 2024-04-16T16:23:00.814 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:23:00.814 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-16T16:23:00.814 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:23:00.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.815+0000 7f1ebc800640 1 -- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:6800/4229500350,v1:172.21.15.103:6801/4229500350] conn(0x7f1e98074bb0 msgr2=0x7f1e98076fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.815+0000 7f1ebc800640 1 --2- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:6800/4229500350,v1:172.21.15.103:6801/4229500350] conn(0x7f1e98074bb0 0x7f1e98076fd0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.816+0000 7f1ebc800640 1 -- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f1e9806ca10 msgr2=0x7f1e9806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.816+0000 7f1ebc800640 1 --2- 172.21.15.45:0/4238246202 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f1e9806ca10 0x7f1e9806eed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8000f60 tx=0x7f1ea8009040 comp rx=0 tx=0).stop 2024-04-16T16:23:00.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.816+0000 7f1ebc800640 1 -- 172.21.15.45:0/4238246202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eb40d10e0 msgr2=0x7f1eb415ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:00.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.816+0000 7f1ebc800640 1 --2- 172.21.15.45:0/4238246202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eb40d10e0 0x7f1eb415ed70 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4034040 tx=0x7f1ea40046c0 comp rx=0 tx=0).stop 2024-04-16T16:23:00.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.816+0000 7f1eba800640 1 -- 172.21.15.45:0/4238246202 reap_dead start 2024-04-16T16:23:00.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.816+0000 7f1ebc800640 1 -- 172.21.15.45:0/4238246202 shutdown_connections 2024-04-16T16:23:00.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.816+0000 7f1ebc800640 1 -- 172.21.15.45:0/4238246202 >> 172.21.15.45:0/4238246202 conn(0x7f1eb40b3970 msgr2=0x7f1eb40729e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:00.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.816+0000 7f1ebc800640 1 -- 172.21.15.45:0/4238246202 shutdown_connections 2024-04-16T16:23:00.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:00.816+0000 7f1ebc800640 1 -- 172.21.15.45:0/4238246202 wait complete. 2024-04-16T16:23:00.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-16T16:23:01.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.057+0000 7f3c59800640 1 Processor -- start 2024-04-16T16:23:01.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.058+0000 7f3c59800640 1 -- start start 2024-04-16T16:23:01.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.059+0000 7f3c59800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3c5406f730 0x7f3c540c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.059+0000 7f3c59800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c5415d600 con 0x7f3c540c6320 2024-04-16T16:23:01.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.059+0000 7f3c59800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c5415d8e0 con 0x7f3c540c6fc0 2024-04-16T16:23:01.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.059+0000 7f3c59800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3c540c5670 con 0x7f3c5406f730 2024-04-16T16:23:01.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.060+0000 7f3c58e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3c540c6320 0x7f3c5406df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45986/0 (socket says 172.21.15.45:45986) 2024-04-16T16:23:01.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.060+0000 7f3c58e00640 1 -- 172.21.15.45:0/2846981623 learned_addr learned my addr 172.21.15.45:0/2846981623 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:23:01.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.060+0000 7f3c53400640 1 --2- 172.21.15.45:0/2846981623 >> v2:172.21.15.103:3300/0 conn(0x7f3c5406f730 0x7f3c540c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.060+0000 7f3c52a00640 1 -- 172.21.15.45:0/2846981623 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1780711094 0 0) 0x7f3c5415d600 con 0x7f3c540c6320 2024-04-16T16:23:01.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.060+0000 7f3c52a00640 1 -- 172.21.15.45:0/2846981623 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c38003520 con 0x7f3c540c6320 2024-04-16T16:23:01.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.061+0000 7f3c52a00640 1 -- 172.21.15.45:0/2846981623 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3466894384 0 0) 0x7f3c38003520 con 0x7f3c540c6320 2024-04-16T16:23:01.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.061+0000 7f3c52a00640 1 -- 172.21.15.45:0/2846981623 >> v2:172.21.15.103:3300/0 conn(0x7f3c5406f730 msgr2=0x7f3c540c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:01.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.061+0000 7f3c52a00640 1 --2- 172.21.15.45:0/2846981623 >> v2:172.21.15.103:3300/0 conn(0x7f3c5406f730 0x7f3c540c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.061+0000 7f3c52a00640 1 -- 172.21.15.45:0/2846981623 >> v1:172.21.15.103:6789/0 conn(0x7f3c540c6fc0 legacy=0x7f3c5415cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.061+0000 7f3c52a00640 1 -- 172.21.15.45:0/2846981623 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5415dbc0 con 0x7f3c540c6320 2024-04-16T16:23:01.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.061+0000 7f3c52a00640 1 -- 172.21.15.45:0/2846981623 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f3c4c001670 con 0x7f3c540c6320 2024-04-16T16:23:01.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.061+0000 7f3c52a00640 1 -- 172.21.15.45:0/2846981623 >> v1:172.21.15.45:6789/0 conn(0x7f3c540c6320 legacy=0x7f3c5406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.061+0000 7f3c52a00640 1 --2- 172.21.15.45:0/2846981623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c380039b0 0x7f3c38003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.061+0000 7f3c52a00640 1 --2- 172.21.15.45:0/2846981623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c38004800 0x7f3c38004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.061+0000 7f3c52a00640 1 --2- 172.21.15.45:0/2846981623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c380051c0 0x7f3c3800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.061+0000 7f3c52a00640 1 -- 172.21.15.45:0/2846981623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3c540c5670 con 0x7f3c380051c0 2024-04-16T16:23:01.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.061+0000 7f3c52a00640 1 -- 172.21.15.45:0/2846981623 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3800c160 con 0x7f3c380039b0 2024-04-16T16:23:01.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.061+0000 7f3c52a00640 1 -- 172.21.15.45:0/2846981623 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c3800c360 con 0x7f3c38004800 2024-04-16T16:23:01.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.061+0000 7f3c58e00640 1 --2- 172.21.15.45:0/2846981623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c380051c0 0x7f3c3800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.062+0000 7f3c53e00640 1 --2- 172.21.15.45:0/2846981623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c380039b0 0x7f3c38003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.062+0000 7f3c53400640 1 --2- 172.21.15.45:0/2846981623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c38004800 0x7f3c38004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.062+0000 7f3c58e00640 1 -- 172.21.15.45:0/2846981623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c38004800 msgr2=0x7f3c38004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:01.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.062+0000 7f3c58e00640 1 --2- 172.21.15.45:0/2846981623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c38004800 0x7f3c38004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.062+0000 7f3c58e00640 1 -- 172.21.15.45:0/2846981623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c380039b0 msgr2=0x7f3c38003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:01.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.062+0000 7f3c58e00640 1 --2- 172.21.15.45:0/2846981623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c380039b0 0x7f3c38003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.062+0000 7f3c58e00640 1 -- 172.21.15.45:0/2846981623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c38003520 con 0x7f3c380051c0 2024-04-16T16:23:01.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.062+0000 7f3c53e00640 1 --2- 172.21.15.45:0/2846981623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c380039b0 0x7f3c38003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:23:01.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.063+0000 7f3c58e00640 1 -- 172.21.15.45:0/2846981623 reap_dead start 2024-04-16T16:23:01.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.063+0000 7f3c58e00640 1 --2- 172.21.15.45:0/2846981623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c380051c0 0x7f3c3800b9c0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00c240 tx=0x7f3c38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57df0e8f8aef04b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:01.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.064+0000 7f3c52a00640 1 -- 172.21.15.45:0/2846981623 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c007cf0 con 0x7f3c380051c0 2024-04-16T16:23:01.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.064+0000 7f3c52a00640 1 -- 172.21.15.45:0/2846981623 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c4c030070 con 0x7f3c380051c0 2024-04-16T16:23:01.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.064+0000 7f3c52a00640 1 -- 172.21.15.45:0/2846981623 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c005d30 con 0x7f3c380051c0 2024-04-16T16:23:01.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.064+0000 7f3c59800640 1 -- 172.21.15.45:0/2846981623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c380051c0 msgr2=0x7f3c3800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.064+0000 7f3c59800640 1 --2- 172.21.15.45:0/2846981623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c380051c0 0x7f3c3800b9c0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00c240 tx=0x7f3c38005830 comp rx=0 tx=0).stop 2024-04-16T16:23:01.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.064+0000 7f3c59800640 1 -- 172.21.15.45:0/2846981623 shutdown_connections 2024-04-16T16:23:01.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.064+0000 7f3c59800640 1 --2- 172.21.15.45:0/2846981623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c380051c0 0x7f3c3800b9c0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.064+0000 7f3c59800640 1 -- 172.21.15.45:0/2846981623 >> 172.21.15.45:0/2846981623 conn(0x7f3c5406b930 msgr2=0x7f3c5406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:01.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.064+0000 7f3c59800640 1 -- 172.21.15.45:0/2846981623 shutdown_connections 2024-04-16T16:23:01.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.065+0000 7f3c59800640 1 -- 172.21.15.45:0/2846981623 wait complete. 2024-04-16T16:23:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.066+0000 7f3c59800640 1 Processor -- start 2024-04-16T16:23:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.066+0000 7f3c59800640 1 -- start start 2024-04-16T16:23:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.067+0000 7f3c59800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c380051c0 0x7f3c5415b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.067+0000 7f3c59800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c541f5230 0x7f3c54156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.067+0000 7f3c59800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c54156bd0 0x7f3c541f3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.067+0000 7f3c59800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3c540c7520 con 0x7f3c54156bd0 2024-04-16T16:23:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.067+0000 7f3c59800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c540c73a0 con 0x7f3c541f5230 2024-04-16T16:23:01.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.067+0000 7f3c59800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c540c5de0 con 0x7f3c380051c0 2024-04-16T16:23:01.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.068+0000 7f3c53e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c380051c0 0x7f3c5415b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.068+0000 7f3c58e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c54156bd0 0x7f3c541f3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.068+0000 7f3c53e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c380051c0 0x7f3c5415b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.45:53362/0 (socket says 172.21.15.45:53362) 2024-04-16T16:23:01.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.068+0000 7f3c53e00640 1 -- 172.21.15.45:0/469429363 learned_addr learned my addr 172.21.15.45:0/469429363 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:01.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.068+0000 7f3c53400640 1 --2- 172.21.15.45:0/469429363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c541f5230 0x7f3c54156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.068+0000 7f3c58e00640 1 -- 172.21.15.45:0/469429363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c380051c0 msgr2=0x7f3c5415b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.068+0000 7f3c58e00640 1 --2- 172.21.15.45:0/469429363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c380051c0 0x7f3c5415b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.068+0000 7f3c58e00640 1 -- 172.21.15.45:0/469429363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c541f5230 msgr2=0x7f3c54156690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.068+0000 7f3c58e00640 1 --2- 172.21.15.45:0/469429363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c541f5230 0x7f3c54156690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.069+0000 7f3c58e00640 1 -- 172.21.15.45:0/469429363 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c541f4190 con 0x7f3c54156bd0 2024-04-16T16:23:01.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.069+0000 7f3c58e00640 1 --2- 172.21.15.45:0/469429363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c54156bd0 0x7f3c541f3460 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f3c540c8200 tx=0x7f3c5415dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:01.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.069+0000 7f3c4be00640 1 -- 172.21.15.45:0/469429363 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c001c30 con 0x7f3c54156bd0 2024-04-16T16:23:01.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.069+0000 7f3c4be00640 1 -- 172.21.15.45:0/469429363 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c4c004bf0 con 0x7f3c54156bd0 2024-04-16T16:23:01.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.069+0000 7f3c4be00640 1 -- 172.21.15.45:0/469429363 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c007070 con 0x7f3c54156bd0 2024-04-16T16:23:01.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.069+0000 7f3c59800640 1 -- 172.21.15.45:0/469429363 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c541f4470 con 0x7f3c54156bd0 2024-04-16T16:23:01.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.069+0000 7f3c59800640 1 -- 172.21.15.45:0/469429363 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c541f4a30 con 0x7f3c54156bd0 2024-04-16T16:23:01.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.070+0000 7f3c59800640 1 -- 172.21.15.45:0/469429363 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c54144800 con 0x7f3c54156bd0 2024-04-16T16:23:01.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.071+0000 7f3c4be00640 1 -- 172.21.15.45:0/469429363 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f3c4c047060 con 0x7f3c54156bd0 2024-04-16T16:23:01.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.072+0000 7f3c4be00640 1 --2- 172.21.15.45:0/469429363 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f3c3406ca10 0x7f3c3406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.072+0000 7f3c4be00640 1 -- 172.21.15.45:0/469429363 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f3c4c0acac0 con 0x7f3c54156bd0 2024-04-16T16:23:01.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.073+0000 7f3c53e00640 1 --2- 172.21.15.45:0/469429363 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f3c3406ca10 0x7f3c3406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.074+0000 7f3c53e00640 1 --2- 172.21.15.45:0/469429363 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f3c3406ca10 0x7f3c3406eed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3c44007fd0 tx=0x7f3c44009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:01.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.077+0000 7f3c4be00640 1 -- 172.21.15.45:0/469429363 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3c54144800 con 0x7f3c54156bd0 2024-04-16T16:23:01.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.285+0000 7f3c59800640 1 -- 172.21.15.45:0/469429363 --> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f3c54075780 con 0x7f3c3406ca10 2024-04-16T16:23:01.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.286+0000 7f3c4be00640 1 -- 172.21.15.45:0/469429363 <== mgr.7591 v2:172.21.15.103:6832/1129061436 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f3c54075780 con 0x7f3c3406ca10 2024-04-16T16:23:01.286 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 4 to deep-scrub 2024-04-16T16:23:01.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.289+0000 7f3c59800640 1 -- 172.21.15.45:0/469429363 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f3c3406ca10 msgr2=0x7f3c3406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.289+0000 7f3c59800640 1 --2- 172.21.15.45:0/469429363 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f3c3406ca10 0x7f3c3406eed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3c44007fd0 tx=0x7f3c44009040 comp rx=0 tx=0).stop 2024-04-16T16:23:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.289+0000 7f3c59800640 1 -- 172.21.15.45:0/469429363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c54156bd0 msgr2=0x7f3c541f3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.289+0000 7f3c59800640 1 --2- 172.21.15.45:0/469429363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c54156bd0 0x7f3c541f3460 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f3c540c8200 tx=0x7f3c5415dab0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.289+0000 7f3c59800640 1 -- 172.21.15.45:0/469429363 shutdown_connections 2024-04-16T16:23:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.289+0000 7f3c59800640 1 --2- 172.21.15.45:0/469429363 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f3c3406ca10 0x7f3c3406eed0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.289+0000 7f3c59800640 1 --2- 172.21.15.45:0/469429363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c54156bd0 0x7f3c541f3460 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.289+0000 7f3c59800640 1 --2- 172.21.15.45:0/469429363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c541f5230 0x7f3c54156690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.289+0000 7f3c59800640 1 --2- 172.21.15.45:0/469429363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c380051c0 0x7f3c5415b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.289+0000 7f3c59800640 1 -- 172.21.15.45:0/469429363 >> 172.21.15.45:0/469429363 conn(0x7f3c5406b930 msgr2=0x7f3c5415f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.290+0000 7f3c59800640 1 -- 172.21.15.45:0/469429363 shutdown_connections 2024-04-16T16:23:01.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.290+0000 7f3c59800640 1 -- 172.21.15.45:0/469429363 wait complete. 2024-04-16T16:23:01.308 INFO:tasks.ceph:Scrubbing osd.5 2024-04-16T16:23:01.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-16T16:23:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.535+0000 7fec28c00640 1 Processor -- start 2024-04-16T16:23:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.535+0000 7fec28c00640 1 -- start start 2024-04-16T16:23:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.537+0000 7fec28c00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fec241552e0 0x7fec24071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.537+0000 7fec28c00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec240b1670 con 0x7fec240b2690 2024-04-16T16:23:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.537+0000 7fec28c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec240b1950 con 0x7fec2415c000 2024-04-16T16:23:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.537+0000 7fec28c00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fec240b1c30 con 0x7fec241552e0 2024-04-16T16:23:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.537+0000 7fec23400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fec241552e0 0x7fec24071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.537+0000 7fec23400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fec241552e0 0x7fec24071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47426/0 (socket says 172.21.15.45:47426) 2024-04-16T16:23:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.537+0000 7fec23400640 1 -- 172.21.15.45:0/4155203903 learned_addr learned my addr 172.21.15.45:0/4155203903 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.537+0000 7fec23e00640 1 --1- 172.21.15.45:0/4155203903 >> v1:172.21.15.45:6789/0 conn(0x7fec240b2690 0x7fec240cf5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45988/0 (socket says 172.21.15.45:45988) 2024-04-16T16:23:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.538+0000 7fec23400640 1 -- 172.21.15.45:0/4155203903 >> v1:172.21.15.103:6789/0 conn(0x7fec2415c000 legacy=0x7fec240b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.538+0000 7fec23400640 1 -- 172.21.15.45:0/4155203903 >> v1:172.21.15.45:6789/0 conn(0x7fec240b2690 legacy=0x7fec240cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.538+0000 7fec23400640 1 -- 172.21.15.45:0/4155203903 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec240cf0c0 con 0x7fec241552e0 2024-04-16T16:23:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.539+0000 7fec23400640 1 --2- 172.21.15.45:0/4155203903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec241552e0 0x7fec24071e60 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fec1400bb00 tx=0x7fec1402df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac73c3b916c3598 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.539+0000 7fec22a00640 1 -- 172.21.15.45:0/4155203903 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec14034070 con 0x7fec241552e0 2024-04-16T16:23:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.539+0000 7fec22a00640 1 -- 172.21.15.45:0/4155203903 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec1402ece0 con 0x7fec241552e0 2024-04-16T16:23:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.539+0000 7fec22a00640 1 -- 172.21.15.45:0/4155203903 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec14038620 con 0x7fec241552e0 2024-04-16T16:23:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.540+0000 7fec28c00640 1 -- 172.21.15.45:0/4155203903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec241552e0 msgr2=0x7fec24071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.540+0000 7fec28c00640 1 --2- 172.21.15.45:0/4155203903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec241552e0 0x7fec24071e60 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fec1400bb00 tx=0x7fec1402df30 comp rx=0 tx=0).stop 2024-04-16T16:23:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.540+0000 7fec28c00640 1 -- 172.21.15.45:0/4155203903 shutdown_connections 2024-04-16T16:23:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.540+0000 7fec28c00640 1 --2- 172.21.15.45:0/4155203903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec241552e0 0x7fec24071e60 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.540+0000 7fec28c00640 1 -- 172.21.15.45:0/4155203903 >> 172.21.15.45:0/4155203903 conn(0x7fec240b3bb0 msgr2=0x7fec240b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.540+0000 7fec28c00640 1 -- 172.21.15.45:0/4155203903 shutdown_connections 2024-04-16T16:23:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.540+0000 7fec28c00640 1 -- 172.21.15.45:0/4155203903 wait complete. 2024-04-16T16:23:01.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.542+0000 7fec28c00640 1 Processor -- start 2024-04-16T16:23:01.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.542+0000 7fec28c00640 1 -- start start 2024-04-16T16:23:01.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.543+0000 7fec28c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec240b2690 0x7fec2413f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.543+0000 7fec28c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec241552e0 0x7fec24158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.543+0000 7fec28c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec2415c000 0x7fec24142330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.543+0000 7fec28c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fec240cee80 con 0x7fec240b2690 2024-04-16T16:23:01.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.543+0000 7fec28c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fec240ced00 con 0x7fec2415c000 2024-04-16T16:23:01.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.543+0000 7fec28c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fec241556c0 con 0x7fec241552e0 2024-04-16T16:23:01.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.544+0000 7fec23400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec240b2690 0x7fec2413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.544+0000 7fec23400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec240b2690 0x7fec2413f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47428/0 (socket says 172.21.15.45:47428) 2024-04-16T16:23:01.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.544+0000 7fec23400640 1 -- 172.21.15.45:0/3962345120 learned_addr learned my addr 172.21.15.45:0/3962345120 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:01.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.544+0000 7fec1be00640 1 --2- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec241552e0 0x7fec24158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.544+0000 7fec23e00640 1 --2- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec2415c000 0x7fec24142330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.544+0000 7fec23400640 1 -- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec241552e0 msgr2=0x7fec24158f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.544+0000 7fec23400640 1 --2- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec241552e0 0x7fec24158f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.544+0000 7fec23400640 1 -- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec2415c000 msgr2=0x7fec24142330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.544+0000 7fec23400640 1 --2- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec2415c000 0x7fec24142330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.544+0000 7fec23400640 1 -- 172.21.15.45:0/3962345120 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec24159990 con 0x7fec240b2690 2024-04-16T16:23:01.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.545+0000 7fec23400640 1 --2- 172.21.15.45:0/3962345120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec240b2690 0x7fec2413f880 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fec1400bc20 tx=0x7fec140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:01.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.545+0000 7fec1b400640 1 -- 172.21.15.45:0/3962345120 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec140415a0 con 0x7fec240b2690 2024-04-16T16:23:01.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.545+0000 7fec1b400640 1 -- 172.21.15.45:0/3962345120 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec14047070 con 0x7fec240b2690 2024-04-16T16:23:01.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.545+0000 7fec1b400640 1 -- 172.21.15.45:0/3962345120 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec14041980 con 0x7fec240b2690 2024-04-16T16:23:01.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.545+0000 7fec28c00640 1 -- 172.21.15.45:0/3962345120 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec24159c70 con 0x7fec240b2690 2024-04-16T16:23:01.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.546+0000 7fec28c00640 1 -- 172.21.15.45:0/3962345120 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec2415a230 con 0x7fec240b2690 2024-04-16T16:23:01.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.547+0000 7fec28c00640 1 -- 172.21.15.45:0/3962345120 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fec24075780 con 0x7fec240b2690 2024-04-16T16:23:01.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.548+0000 7fec1b400640 1 -- 172.21.15.45:0/3962345120 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fec1403f070 con 0x7fec240b2690 2024-04-16T16:23:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.549+0000 7fec1b400640 1 --2- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fec0006ca40 0x7fec0006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.549+0000 7fec1b400640 1 -- 172.21.15.45:0/3962345120 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7fec140aecc0 con 0x7fec240b2690 2024-04-16T16:23:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.549+0000 7fec1b400640 1 --2- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:6822/2127910843,v1:172.21.15.103:6824/2127910843] conn(0x7fec00074be0 0x7fec00077000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.550+0000 7fec1b400640 1 -- 172.21.15.45:0/3962345120 --> [v2:172.21.15.103:6822/2127910843,v1:172.21.15.103:6824/2127910843] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec14039e90 con 0x7fec00074be0 2024-04-16T16:23:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.550+0000 7fec1b400640 1 -- 172.21.15.45:0/3962345120 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=156) ==== 24+0+0 (secure 0 0 0) 0x7fec140af150 con 0x7fec240b2690 2024-04-16T16:23:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.550+0000 7fec1be00640 1 --2- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fec0006ca40 0x7fec0006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.550+0000 7fec23e00640 1 --2- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:6822/2127910843,v1:172.21.15.103:6824/2127910843] conn(0x7fec00074be0 0x7fec00077000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.551+0000 7fec1be00640 1 --2- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fec0006ca40 0x7fec0006ef00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fec0c006fd0 tx=0x7fec0c008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.551+0000 7fec23e00640 1 --2- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:6822/2127910843,v1:172.21.15.103:6824/2127910843] conn(0x7fec00074be0 0x7fec00077000 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:01.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.553+0000 7fec1b400640 1 -- 172.21.15.45:0/3962345120 <== osd.5 v2:172.21.15.103:6822/2127910843 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fec14039e90 con 0x7fec00074be0 2024-04-16T16:23:01.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.568+0000 7fec28c00640 1 -- 172.21.15.45:0/3962345120 --> [v2:172.21.15.103:6822/2127910843,v1:172.21.15.103:6824/2127910843] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fec240b10f0 con 0x7fec00074be0 2024-04-16T16:23:01.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.569+0000 7fec1b400640 1 -- 172.21.15.45:0/3962345120 <== osd.5 v2:172.21.15.103:6822/2127910843 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fec240b10f0 con 0x7fec00074be0 2024-04-16T16:23:01.569 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:23:01.569 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-16T16:23:01.569 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:23:01.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.570+0000 7fec28c00640 1 -- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:6822/2127910843,v1:172.21.15.103:6824/2127910843] conn(0x7fec00074be0 msgr2=0x7fec00077000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.570+0000 7fec28c00640 1 --2- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:6822/2127910843,v1:172.21.15.103:6824/2127910843] conn(0x7fec00074be0 0x7fec00077000 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.570+0000 7fec28c00640 1 -- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fec0006ca40 msgr2=0x7fec0006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.570+0000 7fec28c00640 1 --2- 172.21.15.45:0/3962345120 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fec0006ca40 0x7fec0006ef00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fec0c006fd0 tx=0x7fec0c008040 comp rx=0 tx=0).stop 2024-04-16T16:23:01.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.570+0000 7fec28c00640 1 -- 172.21.15.45:0/3962345120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec240b2690 msgr2=0x7fec2413f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.570+0000 7fec28c00640 1 --2- 172.21.15.45:0/3962345120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec240b2690 0x7fec2413f880 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fec1400bc20 tx=0x7fec140046c0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.570+0000 7fec23e00640 1 -- 172.21.15.45:0/3962345120 reap_dead start 2024-04-16T16:23:01.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.571+0000 7fec28c00640 1 -- 172.21.15.45:0/3962345120 shutdown_connections 2024-04-16T16:23:01.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.571+0000 7fec28c00640 1 -- 172.21.15.45:0/3962345120 >> 172.21.15.45:0/3962345120 conn(0x7fec240b3bb0 msgr2=0x7fec2415b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:01.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.571+0000 7fec28c00640 1 -- 172.21.15.45:0/3962345120 shutdown_connections 2024-04-16T16:23:01.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.571+0000 7fec28c00640 1 -- 172.21.15.45:0/3962345120 wait complete. 2024-04-16T16:23:01.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-16T16:23:01.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.808+0000 7f1a15800640 1 Processor -- start 2024-04-16T16:23:01.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.808+0000 7f1a15800640 1 -- start start 2024-04-16T16:23:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.809+0000 7f1a15800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1a1015ca00 0x7f1a100cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.809+0000 7f1a15800640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a1006fab0 con 0x7f1a100afe20 2024-04-16T16:23:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.809+0000 7f1a15800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a1015bf20 con 0x7f1a100cde10 2024-04-16T16:23:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.809+0000 7f1a15800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1a100ce240 con 0x7f1a1015ca00 2024-04-16T16:23:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.810+0000 7f1a0ea00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1a100afe20 0x7f1a1006f010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45998/0 (socket says 172.21.15.45:45998) 2024-04-16T16:23:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.810+0000 7f1a0ea00640 1 -- 172.21.15.45:0/3298460178 learned_addr learned my addr 172.21.15.45:0/3298460178 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:23:01.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.810+0000 7f1a07e00640 1 --2- 172.21.15.45:0/3298460178 >> v2:172.21.15.103:3300/0 conn(0x7f1a1015ca00 0x7f1a100cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.810+0000 7f1a0e000640 1 -- 172.21.15.45:0/3298460178 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2974054916 0 0) 0x7f1a1006fab0 con 0x7f1a100afe20 2024-04-16T16:23:01.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.810+0000 7f1a0e000640 1 -- 172.21.15.45:0/3298460178 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19f4003520 con 0x7f1a100afe20 2024-04-16T16:23:01.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.811+0000 7f1a0e000640 1 -- 172.21.15.45:0/3298460178 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3105186451 0 0) 0x7f19f4003520 con 0x7f1a100afe20 2024-04-16T16:23:01.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.811+0000 7f1a0e000640 1 -- 172.21.15.45:0/3298460178 >> v2:172.21.15.103:3300/0 conn(0x7f1a1015ca00 msgr2=0x7f1a100cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:01.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.811+0000 7f1a0e000640 1 --2- 172.21.15.45:0/3298460178 >> v2:172.21.15.103:3300/0 conn(0x7f1a1015ca00 0x7f1a100cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.811+0000 7f1a0e000640 1 -- 172.21.15.45:0/3298460178 >> v1:172.21.15.103:6789/0 conn(0x7f1a100cde10 legacy=0x7f1a100b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.811+0000 7f1a0e000640 1 -- 172.21.15.45:0/3298460178 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1015c200 con 0x7f1a100afe20 2024-04-16T16:23:01.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.811+0000 7f1a0e000640 1 -- 172.21.15.45:0/3298460178 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f1a00001670 con 0x7f1a100afe20 2024-04-16T16:23:01.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.811+0000 7f1a0e000640 1 -- 172.21.15.45:0/3298460178 >> v1:172.21.15.45:6789/0 conn(0x7f1a100afe20 legacy=0x7f1a1006f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.811+0000 7f1a0e000640 1 --2- 172.21.15.45:0/3298460178 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19f40039b0 0x7f19f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.811+0000 7f1a0e000640 1 --2- 172.21.15.45:0/3298460178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19f4004800 0x7f19f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.811+0000 7f1a0e000640 1 --2- 172.21.15.45:0/3298460178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f40051c0 0x7f19f400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.811+0000 7f1a0e000640 1 -- 172.21.15.45:0/3298460178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a100ce240 con 0x7f19f4004800 2024-04-16T16:23:01.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.811+0000 7f1a0e000640 1 -- 172.21.15.45:0/3298460178 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f19f400e270 con 0x7f19f40051c0 2024-04-16T16:23:01.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.811+0000 7f1a0e000640 1 -- 172.21.15.45:0/3298460178 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f19f400e470 con 0x7f19f40039b0 2024-04-16T16:23:01.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.811+0000 7f1a07e00640 1 --2- 172.21.15.45:0/3298460178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19f4004800 0x7f19f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.812+0000 7f1a0ea00640 1 --2- 172.21.15.45:0/3298460178 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19f40039b0 0x7f19f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.812+0000 7f1a0f400640 1 --2- 172.21.15.45:0/3298460178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f40051c0 0x7f19f400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.812+0000 7f1a07e00640 1 -- 172.21.15.45:0/3298460178 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19f40039b0 msgr2=0x7f19f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:01.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.812+0000 7f1a07e00640 1 --2- 172.21.15.45:0/3298460178 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19f40039b0 0x7f19f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.812+0000 7f1a07e00640 1 -- 172.21.15.45:0/3298460178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f40051c0 msgr2=0x7f19f400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:01.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.812+0000 7f1a07e00640 1 --2- 172.21.15.45:0/3298460178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f40051c0 0x7f19f400dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.812+0000 7f1a07e00640 1 -- 172.21.15.45:0/3298460178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19f4003520 con 0x7f19f4004800 2024-04-16T16:23:01.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.812+0000 7f1a0f400640 1 -- 172.21.15.45:0/3298460178 reap_dead start 2024-04-16T16:23:01.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.813+0000 7f1a07e00640 1 --2- 172.21.15.45:0/3298460178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19f4004800 0x7f19f4004c80 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009ac0 tx=0x7f19f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2ef12968fd5781 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:01.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.814+0000 7f1a0e000640 1 -- 172.21.15.45:0/3298460178 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f8039070 con 0x7f19f4004800 2024-04-16T16:23:01.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.814+0000 7f1a0e000640 1 -- 172.21.15.45:0/3298460178 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19f802c6b0 con 0x7f19f4004800 2024-04-16T16:23:01.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.814+0000 7f1a0e000640 1 -- 172.21.15.45:0/3298460178 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f8009690 con 0x7f19f4004800 2024-04-16T16:23:01.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.814+0000 7f1a15800640 1 -- 172.21.15.45:0/3298460178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19f4004800 msgr2=0x7f19f4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.814+0000 7f1a15800640 1 --2- 172.21.15.45:0/3298460178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19f4004800 0x7f19f4004c80 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009ac0 tx=0x7f19f4005830 comp rx=0 tx=0).stop 2024-04-16T16:23:01.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.814+0000 7f1a15800640 1 -- 172.21.15.45:0/3298460178 shutdown_connections 2024-04-16T16:23:01.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.814+0000 7f1a15800640 1 --2- 172.21.15.45:0/3298460178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19f4004800 0x7f19f4004c80 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.814+0000 7f1a15800640 1 -- 172.21.15.45:0/3298460178 >> 172.21.15.45:0/3298460178 conn(0x7f1a1006b930 msgr2=0x7f1a1006bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:01.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.814+0000 7f1a15800640 1 -- 172.21.15.45:0/3298460178 shutdown_connections 2024-04-16T16:23:01.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.815+0000 7f1a15800640 1 -- 172.21.15.45:0/3298460178 wait complete. 2024-04-16T16:23:01.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.817+0000 7f1a15800640 1 Processor -- start 2024-04-16T16:23:01.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.817+0000 7f1a15800640 1 -- start start 2024-04-16T16:23:01.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.818+0000 7f1a15800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f4004800 0x7f1a1014c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.818+0000 7f1a15800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100c61f0 0x7f1a10158400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.818+0000 7f1a15800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a100c6670 0x7f1a10158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.818+0000 7f1a15800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a100b0380 con 0x7f1a100c61f0 2024-04-16T16:23:01.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.818+0000 7f1a15800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a100b0200 con 0x7f19f4004800 2024-04-16T16:23:01.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.818+0000 7f1a15800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1a100b5850 con 0x7f1a100c6670 2024-04-16T16:23:01.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.819+0000 7f1a07e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100c61f0 0x7f1a10158400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.819+0000 7f1a07e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100c61f0 0x7f1a10158400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47440/0 (socket says 172.21.15.45:47440) 2024-04-16T16:23:01.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.819+0000 7f1a07e00640 1 -- 172.21.15.45:0/2745258675 learned_addr learned my addr 172.21.15.45:0/2745258675 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:01.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.819+0000 7f1a0f400640 1 --2- 172.21.15.45:0/2745258675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a100c6670 0x7f1a10158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.819+0000 7f1a0ea00640 1 --2- 172.21.15.45:0/2745258675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f4004800 0x7f1a1014c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.819+0000 7f1a07e00640 1 -- 172.21.15.45:0/2745258675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a100c6670 msgr2=0x7f1a10158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.819+0000 7f1a07e00640 1 --2- 172.21.15.45:0/2745258675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a100c6670 0x7f1a10158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.819+0000 7f1a07e00640 1 -- 172.21.15.45:0/2745258675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f4004800 msgr2=0x7f1a1014c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:01.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.819+0000 7f1a07e00640 1 --2- 172.21.15.45:0/2745258675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f4004800 0x7f1a1014c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:01.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.819+0000 7f1a07e00640 1 -- 172.21.15.45:0/2745258675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a10155840 con 0x7f1a100c61f0 2024-04-16T16:23:01.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.819+0000 7f1a0ea00640 1 --2- 172.21.15.45:0/2745258675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f4004800 0x7f1a1014c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:23:01.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.820+0000 7f1a07e00640 1 --2- 172.21.15.45:0/2745258675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100c61f0 0x7f1a10158400 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009be0 tx=0x7f19f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:01.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.820+0000 7f1a06a00640 1 -- 172.21.15.45:0/2745258675 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f8042020 con 0x7f1a100c61f0 2024-04-16T16:23:01.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.820+0000 7f1a06a00640 1 -- 172.21.15.45:0/2745258675 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19f803a040 con 0x7f1a100c61f0 2024-04-16T16:23:01.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.820+0000 7f1a15800640 1 -- 172.21.15.45:0/2745258675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a10157360 con 0x7f1a100c61f0 2024-04-16T16:23:01.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.820+0000 7f1a06a00640 1 -- 172.21.15.45:0/2745258675 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f8039070 con 0x7f1a100c61f0 2024-04-16T16:23:01.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.820+0000 7f1a15800640 1 -- 172.21.15.45:0/2745258675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a10157920 con 0x7f1a100c61f0 2024-04-16T16:23:01.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.822+0000 7f1a15800640 1 -- 172.21.15.45:0/2745258675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a10075780 con 0x7f1a100c61f0 2024-04-16T16:23:01.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.823+0000 7f1a06a00640 1 -- 172.21.15.45:0/2745258675 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f19f8045020 con 0x7f1a100c61f0 2024-04-16T16:23:01.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.823+0000 7f1a06a00640 1 --2- 172.21.15.45:0/2745258675 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f19ec06ca40 0x7f19ec06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:01.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.824+0000 7f1a06a00640 1 -- 172.21.15.45:0/2745258675 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f19f80ad430 con 0x7f1a100c61f0 2024-04-16T16:23:01.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.824+0000 7f1a0ea00640 1 --2- 172.21.15.45:0/2745258675 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f19ec06ca40 0x7f19ec06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:01.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.826+0000 7f1a0ea00640 1 --2- 172.21.15.45:0/2745258675 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f19ec06ca40 0x7f19ec06ef00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1a00001eb0 tx=0x7f1a00009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:01.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:01.828+0000 7f1a06a00640 1 -- 172.21.15.45:0/2745258675 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f19f804c030 con 0x7f1a100c61f0 2024-04-16T16:23:02.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.037+0000 7f1a15800640 1 -- 172.21.15.45:0/2745258675 --> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f1a100b1f40 con 0x7f19ec06ca40 2024-04-16T16:23:02.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.038+0000 7f1a06a00640 1 -- 172.21.15.45:0/2745258675 <== mgr.7591 v2:172.21.15.103:6832/1129061436 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1a100b1f40 con 0x7f19ec06ca40 2024-04-16T16:23:02.037 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 5 to deep-scrub 2024-04-16T16:23:02.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.041+0000 7f1a15800640 1 -- 172.21.15.45:0/2745258675 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f19ec06ca40 msgr2=0x7f19ec06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:02.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.041+0000 7f1a15800640 1 --2- 172.21.15.45:0/2745258675 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f19ec06ca40 0x7f19ec06ef00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1a00001eb0 tx=0x7f1a00009040 comp rx=0 tx=0).stop 2024-04-16T16:23:02.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.041+0000 7f1a15800640 1 -- 172.21.15.45:0/2745258675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100c61f0 msgr2=0x7f1a10158400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.041+0000 7f1a15800640 1 --2- 172.21.15.45:0/2745258675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100c61f0 0x7f1a10158400 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009be0 tx=0x7f19f8004730 comp rx=0 tx=0).stop 2024-04-16T16:23:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.041+0000 7f1a15800640 1 -- 172.21.15.45:0/2745258675 shutdown_connections 2024-04-16T16:23:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.041+0000 7f1a15800640 1 --2- 172.21.15.45:0/2745258675 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f19ec06ca40 0x7f19ec06ef00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.041+0000 7f1a15800640 1 --2- 172.21.15.45:0/2745258675 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a100c6670 0x7f1a10158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.041+0000 7f1a15800640 1 --2- 172.21.15.45:0/2745258675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100c61f0 0x7f1a10158400 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.041+0000 7f1a15800640 1 --2- 172.21.15.45:0/2745258675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f4004800 0x7f1a1014c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.041+0000 7f1a15800640 1 -- 172.21.15.45:0/2745258675 >> 172.21.15.45:0/2745258675 conn(0x7f1a1006b930 msgr2=0x7f1a100b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.041+0000 7f1a15800640 1 -- 172.21.15.45:0/2745258675 shutdown_connections 2024-04-16T16:23:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.041+0000 7f1a15800640 1 -- 172.21.15.45:0/2745258675 wait complete. 2024-04-16T16:23:02.059 INFO:tasks.ceph:Scrubbing osd.6 2024-04-16T16:23:02.059 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-16T16:23:02.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.285+0000 7f49eec00640 1 Processor -- start 2024-04-16T16:23:02.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.285+0000 7f49eec00640 1 -- start start 2024-04-16T16:23:02.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.287+0000 7f49eec00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f49e81452e0 0x7f49e806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:02.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.287+0000 7f49eec00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f49e806e920 0x7f49e80ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:02.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.287+0000 7f49eec00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49e8071e60 con 0x7f49e80ceaf0 2024-04-16T16:23:02.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.287+0000 7f49eec00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f49e8072f00 con 0x7f49e806e920 2024-04-16T16:23:02.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.287+0000 7f49eec00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f49e806ed90 con 0x7f49e81452e0 2024-04-16T16:23:02.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.287+0000 7f49ece00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f49e806e920 0x7f49e80ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:02.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.288+0000 7f49ece00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f49e806e920 0x7f49e80ce5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47450/0 (socket says 172.21.15.45:47450) 2024-04-16T16:23:02.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.288+0000 7f49ece00640 1 -- 172.21.15.45:0/3147586111 learned_addr learned my addr 172.21.15.45:0/3147586111 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:02.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.288+0000 7f49ed800640 1 --2- 172.21.15.45:0/3147586111 >> v2:172.21.15.103:3300/0 conn(0x7f49e81452e0 0x7f49e806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:02.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.288+0000 7f49ece00640 1 -- 172.21.15.45:0/3147586111 >> v2:172.21.15.103:3300/0 conn(0x7f49e81452e0 msgr2=0x7f49e806e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:02.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.288+0000 7f49ece00640 1 --2- 172.21.15.45:0/3147586111 >> v2:172.21.15.103:3300/0 conn(0x7f49e81452e0 0x7f49e806e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.288+0000 7f49ece00640 1 -- 172.21.15.45:0/3147586111 >> v1:172.21.15.103:6789/0 conn(0x7f49e80ceaf0 legacy=0x7f49e8160270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:02.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.288+0000 7f49ece00640 1 -- 172.21.15.45:0/3147586111 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e8072140 con 0x7f49e806e920 2024-04-16T16:23:02.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.289+0000 7f49ece00640 1 --2- 172.21.15.45:0/3147586111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49e806e920 0x7f49e80ce5b0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f49e80702a0 tx=0x7f49d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56bb8e4c0c096588 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:02.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.290+0000 7f49e7e00640 1 -- 172.21.15.45:0/3147586111 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d4034070 con 0x7f49e806e920 2024-04-16T16:23:02.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.290+0000 7f49e7e00640 1 -- 172.21.15.45:0/3147586111 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d4030860 con 0x7f49e806e920 2024-04-16T16:23:02.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.290+0000 7f49e7e00640 1 -- 172.21.15.45:0/3147586111 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d4030c30 con 0x7f49e806e920 2024-04-16T16:23:02.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.290+0000 7f49eec00640 1 -- 172.21.15.45:0/3147586111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49e806e920 msgr2=0x7f49e80ce5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:02.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.290+0000 7f49eec00640 1 --2- 172.21.15.45:0/3147586111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49e806e920 0x7f49e80ce5b0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f49e80702a0 tx=0x7f49d402b9a0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.290+0000 7f49eec00640 1 -- 172.21.15.45:0/3147586111 shutdown_connections 2024-04-16T16:23:02.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.290+0000 7f49eec00640 1 --2- 172.21.15.45:0/3147586111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49e806e920 0x7f49e80ce5b0 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.290+0000 7f49eec00640 1 --2- 172.21.15.45:0/3147586111 >> v2:172.21.15.103:3300/0 conn(0x7f49e81452e0 0x7f49e806e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.291+0000 7f49eec00640 1 -- 172.21.15.45:0/3147586111 >> 172.21.15.45:0/3147586111 conn(0x7f49e80b3bb0 msgr2=0x7f49e80b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:02.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.291+0000 7f49eec00640 1 -- 172.21.15.45:0/3147586111 shutdown_connections 2024-04-16T16:23:02.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.291+0000 7f49eec00640 1 -- 172.21.15.45:0/3147586111 wait complete. 2024-04-16T16:23:02.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.292+0000 7f49eec00640 1 Processor -- start 2024-04-16T16:23:02.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.293+0000 7f49eec00640 1 -- start start 2024-04-16T16:23:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.293+0000 7f49eec00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49e806e920 0x7f49e80ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.294+0000 7f49eec00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49e80ceaf0 0x7f49e813a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.294+0000 7f49eec00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49e81452e0 0x7f49e80b5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.294+0000 7f49eec00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f49e80729e0 con 0x7f49e80ceaf0 2024-04-16T16:23:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.294+0000 7f49eec00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f49e8072700 con 0x7f49e81452e0 2024-04-16T16:23:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.294+0000 7f49eec00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f49e80cef10 con 0x7f49e806e920 2024-04-16T16:23:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.294+0000 7f49ece00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49e80ceaf0 0x7f49e813a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.294+0000 7f49ece00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49e80ceaf0 0x7f49e813a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47460/0 (socket says 172.21.15.45:47460) 2024-04-16T16:23:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.294+0000 7f49ece00640 1 -- 172.21.15.45:0/584009581 learned_addr learned my addr 172.21.15.45:0/584009581 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:02.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.294+0000 7f49ee200640 1 --2- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49e81452e0 0x7f49e80b5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:02.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.294+0000 7f49ed800640 1 --2- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49e806e920 0x7f49e80ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:02.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.294+0000 7f49ece00640 1 -- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49e806e920 msgr2=0x7f49e80ae4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:02.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.294+0000 7f49ece00640 1 --2- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49e806e920 0x7f49e80ae4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.294+0000 7f49ece00640 1 -- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49e81452e0 msgr2=0x7f49e80b5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:02.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.294+0000 7f49ece00640 1 --2- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49e81452e0 0x7f49e80b5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.294+0000 7f49ece00640 1 -- 172.21.15.45:0/584009581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e813b090 con 0x7f49e80ceaf0 2024-04-16T16:23:02.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.295+0000 7f49ed800640 1 --2- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49e806e920 0x7f49e80ae4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:23:02.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.295+0000 7f49ece00640 1 --2- 172.21.15.45:0/584009581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49e80ceaf0 0x7f49e813a7b0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f49d402b9e0 tx=0x7f49d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:02.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.295+0000 7f49e5600640 1 -- 172.21.15.45:0/584009581 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d4048070 con 0x7f49e80ceaf0 2024-04-16T16:23:02.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.295+0000 7f49e5600640 1 -- 172.21.15.45:0/584009581 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d4030660 con 0x7f49e80ceaf0 2024-04-16T16:23:02.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.295+0000 7f49e5600640 1 -- 172.21.15.45:0/584009581 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d4034040 con 0x7f49e80ceaf0 2024-04-16T16:23:02.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.295+0000 7f49eec00640 1 -- 172.21.15.45:0/584009581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49e813b370 con 0x7f49e80ceaf0 2024-04-16T16:23:02.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.296+0000 7f49eec00640 1 -- 172.21.15.45:0/584009581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49e80adcd0 con 0x7f49e80ceaf0 2024-04-16T16:23:02.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.296+0000 7f49eec00640 1 -- 172.21.15.45:0/584009581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f49e8073f90 con 0x7f49e80ceaf0 2024-04-16T16:23:02.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.298+0000 7f49e5600640 1 -- 172.21.15.45:0/584009581 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f49d404b070 con 0x7f49e80ceaf0 2024-04-16T16:23:02.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.298+0000 7f49e5600640 1 --2- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f49c806ca40 0x7f49c806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.299+0000 7f49e5600640 1 -- 172.21.15.45:0/584009581 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f49d40adf70 con 0x7f49e80ceaf0 2024-04-16T16:23:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.299+0000 7f49e5600640 1 --2- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:6808/3613143327,v1:172.21.15.103:6809/3613143327] conn(0x7f49c8074be0 0x7f49c8077000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.299+0000 7f49e5600640 1 -- 172.21.15.45:0/584009581 --> [v2:172.21.15.103:6808/3613143327,v1:172.21.15.103:6809/3613143327] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f49c80776b0 con 0x7f49c8074be0 2024-04-16T16:23:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.299+0000 7f49e5600640 1 -- 172.21.15.45:0/584009581 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=156) ==== 24+0+0 (secure 0 0 0) 0x7f49d40b0590 con 0x7f49e80ceaf0 2024-04-16T16:23:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.299+0000 7f49ed800640 1 --2- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f49c806ca40 0x7f49c806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:02.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.300+0000 7f49ee200640 1 --2- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:6808/3613143327,v1:172.21.15.103:6809/3613143327] conn(0x7f49c8074be0 0x7f49c8077000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:02.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.301+0000 7f49ed800640 1 --2- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f49c806ca40 0x7f49c806ef00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f49dc006fd0 tx=0x7f49dc008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:02.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.301+0000 7f49ee200640 1 --2- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:6808/3613143327,v1:172.21.15.103:6809/3613143327] conn(0x7f49c8074be0 0x7f49c8077000 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:02.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.303+0000 7f49e5600640 1 -- 172.21.15.45:0/584009581 <== osd.6 v2:172.21.15.103:6808/3613143327 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f49c80776b0 con 0x7f49c8074be0 2024-04-16T16:23:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.318+0000 7f49eec00640 1 -- 172.21.15.45:0/584009581 --> [v2:172.21.15.103:6808/3613143327,v1:172.21.15.103:6809/3613143327] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f49e8075780 con 0x7f49c8074be0 2024-04-16T16:23:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.319+0000 7f49e5600640 1 -- 172.21.15.45:0/584009581 <== osd.6 v2:172.21.15.103:6808/3613143327 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f49e8075780 con 0x7f49c8074be0 2024-04-16T16:23:02.318 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:23:02.318 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-16T16:23:02.318 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:23:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.319+0000 7f49eec00640 1 -- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:6808/3613143327,v1:172.21.15.103:6809/3613143327] conn(0x7f49c8074be0 msgr2=0x7f49c8077000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.319+0000 7f49eec00640 1 --2- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:6808/3613143327,v1:172.21.15.103:6809/3613143327] conn(0x7f49c8074be0 0x7f49c8077000 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.320+0000 7f49eec00640 1 -- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f49c806ca40 msgr2=0x7f49c806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.320+0000 7f49eec00640 1 --2- 172.21.15.45:0/584009581 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f49c806ca40 0x7f49c806ef00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f49dc006fd0 tx=0x7f49dc008040 comp rx=0 tx=0).stop 2024-04-16T16:23:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.320+0000 7f49eec00640 1 -- 172.21.15.45:0/584009581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49e80ceaf0 msgr2=0x7f49e813a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.320+0000 7f49eec00640 1 --2- 172.21.15.45:0/584009581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49e80ceaf0 0x7f49e813a7b0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f49d402b9e0 tx=0x7f49d4004730 comp rx=0 tx=0).stop 2024-04-16T16:23:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.320+0000 7f49ee200640 1 -- 172.21.15.45:0/584009581 reap_dead start 2024-04-16T16:23:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.320+0000 7f49eec00640 1 -- 172.21.15.45:0/584009581 shutdown_connections 2024-04-16T16:23:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.320+0000 7f49eec00640 1 -- 172.21.15.45:0/584009581 >> 172.21.15.45:0/584009581 conn(0x7f49e80b3bb0 msgr2=0x7f49e81428d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.320+0000 7f49eec00640 1 -- 172.21.15.45:0/584009581 shutdown_connections 2024-04-16T16:23:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.320+0000 7f49eec00640 1 -- 172.21.15.45:0/584009581 wait complete. 2024-04-16T16:23:02.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-16T16:23:02.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.578+0000 7fccd4200640 1 Processor -- start 2024-04-16T16:23:02.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.579+0000 7fccd4200640 1 -- start start 2024-04-16T16:23:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.580+0000 7fccd4200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcccc0cd930 0x7fcccc0cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.580+0000 7fccd4200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcccc146600 con 0x7fcccc146220 2024-04-16T16:23:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.580+0000 7fccd4200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcccc0cde00 con 0x7fcccc072890 2024-04-16T16:23:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.580+0000 7fccd4200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fcccc06f840 con 0x7fcccc0cd930 2024-04-16T16:23:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.580+0000 7fccd1800640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcccc146220 0x7fcccc072180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46000/0 (socket says 172.21.15.45:46000) 2024-04-16T16:23:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.580+0000 7fccd1800640 1 -- 172.21.15.45:0/4102407589 learned_addr learned my addr 172.21.15.45:0/4102407589 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:23:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.581+0000 7fccd2200640 1 --2- 172.21.15.45:0/4102407589 >> v2:172.21.15.45:3300/0 conn(0x7fcccc0cd930 0x7fcccc0cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.581+0000 7fccd2200640 1 -- 172.21.15.45:0/4102407589 >> v1:172.21.15.103:6789/0 conn(0x7fcccc072890 legacy=0x7fcccc0cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.581+0000 7fccd2200640 1 -- 172.21.15.45:0/4102407589 >> v1:172.21.15.45:6789/0 conn(0x7fcccc146220 legacy=0x7fcccc072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.581+0000 7fccd2200640 1 -- 172.21.15.45:0/4102407589 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcccc0ce630 con 0x7fcccc0cd930 2024-04-16T16:23:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.582+0000 7fccd2200640 1 --2- 172.21.15.45:0/4102407589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0cd930 0x7fcccc0cf110 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fccc4009d90 tx=0x7fccc402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ba5be424e4771ec server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.583+0000 7fcccbe00640 1 -- 172.21.15.45:0/4102407589 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc4039070 con 0x7fcccc0cd930 2024-04-16T16:23:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.583+0000 7fcccbe00640 1 -- 172.21.15.45:0/4102407589 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccc402cc60 con 0x7fcccc0cd930 2024-04-16T16:23:02.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.583+0000 7fcccbe00640 1 -- 172.21.15.45:0/4102407589 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc4034660 con 0x7fcccc0cd930 2024-04-16T16:23:02.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.583+0000 7fccd4200640 1 -- 172.21.15.45:0/4102407589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0cd930 msgr2=0x7fcccc0cf110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:02.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.583+0000 7fccd4200640 1 --2- 172.21.15.45:0/4102407589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0cd930 0x7fcccc0cf110 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fccc4009d90 tx=0x7fccc402beb0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.583+0000 7fccd4200640 1 -- 172.21.15.45:0/4102407589 shutdown_connections 2024-04-16T16:23:02.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.583+0000 7fccd4200640 1 --2- 172.21.15.45:0/4102407589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0cd930 0x7fcccc0cf110 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.583+0000 7fccd4200640 1 -- 172.21.15.45:0/4102407589 >> 172.21.15.45:0/4102407589 conn(0x7fcccc06b930 msgr2=0x7fcccc06bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:02.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.583+0000 7fccd4200640 1 -- 172.21.15.45:0/4102407589 shutdown_connections 2024-04-16T16:23:02.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.583+0000 7fccd4200640 1 -- 172.21.15.45:0/4102407589 wait complete. 2024-04-16T16:23:02.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.585+0000 7fccd4200640 1 Processor -- start 2024-04-16T16:23:02.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.585+0000 7fccd4200640 1 -- start start 2024-04-16T16:23:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.586+0000 7fccd4200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc072890 0x7fcccc15a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.586+0000 7fccd4200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcccc0cd930 0x7fcccc1f0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.586+0000 7fccd4200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcccc146220 0x7fcccc15c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:02.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.586+0000 7fccd4200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc0cebf0 con 0x7fcccc072890 2024-04-16T16:23:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.586+0000 7fccd4200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc0ce0e0 con 0x7fcccc146220 2024-04-16T16:23:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.586+0000 7fccd4200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcccc06df00 con 0x7fcccc0cd930 2024-04-16T16:23:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.587+0000 7fccd1800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc072890 0x7fcccc15a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.587+0000 7fccd1800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc072890 0x7fcccc15a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47484/0 (socket says 172.21.15.45:47484) 2024-04-16T16:23:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.587+0000 7fccd1800640 1 -- 172.21.15.45:0/2092490899 learned_addr learned my addr 172.21.15.45:0/2092490899 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.587+0000 7fccd2200640 1 --2- 172.21.15.45:0/2092490899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcccc146220 0x7fcccc15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.587+0000 7fccd0e00640 1 --2- 172.21.15.45:0/2092490899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcccc0cd930 0x7fcccc1f0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.587+0000 7fccd1800640 1 -- 172.21.15.45:0/2092490899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcccc0cd930 msgr2=0x7fcccc1f0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.587+0000 7fccd1800640 1 --2- 172.21.15.45:0/2092490899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcccc0cd930 0x7fcccc1f0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.587+0000 7fccd1800640 1 -- 172.21.15.45:0/2092490899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcccc146220 msgr2=0x7fcccc15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.587+0000 7fccd1800640 1 --2- 172.21.15.45:0/2092490899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcccc146220 0x7fcccc15c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.587+0000 7fccd1800640 1 -- 172.21.15.45:0/2092490899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcccc157820 con 0x7fcccc072890 2024-04-16T16:23:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.588+0000 7fccd1800640 1 --2- 172.21.15.45:0/2092490899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc072890 0x7fcccc15a080 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fccc000c3f0 tx=0x7fccc000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.588+0000 7fccc9600640 1 -- 172.21.15.45:0/2092490899 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc0017070 con 0x7fcccc072890 2024-04-16T16:23:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.588+0000 7fccc9600640 1 -- 172.21.15.45:0/2092490899 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccc0005ce0 con 0x7fcccc072890 2024-04-16T16:23:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.588+0000 7fccc9600640 1 -- 172.21.15.45:0/2092490899 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc0007310 con 0x7fcccc072890 2024-04-16T16:23:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.588+0000 7fccd4200640 1 -- 172.21.15.45:0/2092490899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcccc157b60 con 0x7fcccc072890 2024-04-16T16:23:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.588+0000 7fccd4200640 1 -- 172.21.15.45:0/2092490899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcccc1583a0 con 0x7fcccc072890 2024-04-16T16:23:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.589+0000 7fccd4200640 1 -- 172.21.15.45:0/2092490899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcccc075780 con 0x7fcccc072890 2024-04-16T16:23:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.590+0000 7fccc9600640 1 -- 172.21.15.45:0/2092490899 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7fccc001a070 con 0x7fcccc072890 2024-04-16T16:23:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.591+0000 7fccc9600640 1 --2- 172.21.15.45:0/2092490899 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fccac06ca10 0x7fccac06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.591+0000 7fccc9600640 1 -- 172.21.15.45:0/2092490899 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7fccc008b080 con 0x7fcccc072890 2024-04-16T16:23:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.592+0000 7fccd0e00640 1 --2- 172.21.15.45:0/2092490899 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fccac06ca10 0x7fccac06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:02.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.593+0000 7fccd0e00640 1 --2- 172.21.15.45:0/2092490899 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fccac06ca10 0x7fccac06eed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcccc06fbb0 tx=0x7fccb8009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.595+0000 7fccc9600640 1 -- 172.21.15.45:0/2092490899 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcccc157b60 con 0x7fcccc072890 2024-04-16T16:23:02.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~13s 2024-04-16T16:23:02.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.803+0000 7fccd4200640 1 -- 172.21.15.45:0/2092490899 --> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fcccc0b05a0 con 0x7fccac06ca10 2024-04-16T16:23:02.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.804+0000 7fccc9600640 1 -- 172.21.15.45:0/2092490899 <== mgr.7591 v2:172.21.15.103:6832/1129061436 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fcccc0b05a0 con 0x7fccac06ca10 2024-04-16T16:23:02.803 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 6 to deep-scrub 2024-04-16T16:23:02.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.807+0000 7fccd4200640 1 -- 172.21.15.45:0/2092490899 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fccac06ca10 msgr2=0x7fccac06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:02.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.807+0000 7fccd4200640 1 --2- 172.21.15.45:0/2092490899 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fccac06ca10 0x7fccac06eed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcccc06fbb0 tx=0x7fccb8009040 comp rx=0 tx=0).stop 2024-04-16T16:23:02.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.807+0000 7fccd4200640 1 -- 172.21.15.45:0/2092490899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc072890 msgr2=0x7fcccc15a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:02.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.807+0000 7fccd4200640 1 --2- 172.21.15.45:0/2092490899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc072890 0x7fcccc15a080 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fccc000c3f0 tx=0x7fccc000c8b0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.807+0000 7fccd4200640 1 -- 172.21.15.45:0/2092490899 shutdown_connections 2024-04-16T16:23:02.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.807+0000 7fccd4200640 1 --2- 172.21.15.45:0/2092490899 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7fccac06ca10 0x7fccac06eed0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.807+0000 7fccd4200640 1 --2- 172.21.15.45:0/2092490899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcccc146220 0x7fcccc15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.807+0000 7fccd4200640 1 --2- 172.21.15.45:0/2092490899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcccc0cd930 0x7fcccc1f0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.807+0000 7fccd4200640 1 --2- 172.21.15.45:0/2092490899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc072890 0x7fcccc15a080 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:02.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.807+0000 7fccd4200640 1 -- 172.21.15.45:0/2092490899 >> 172.21.15.45:0/2092490899 conn(0x7fcccc06b930 msgr2=0x7fcccc15ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:02.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.807+0000 7fccd4200640 1 -- 172.21.15.45:0/2092490899 shutdown_connections 2024-04-16T16:23:02.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:02.807+0000 7fccd4200640 1 -- 172.21.15.45:0/2092490899 wait complete. 2024-04-16T16:23:02.826 INFO:tasks.ceph:Scrubbing osd.7 2024-04-16T16:23:02.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-16T16:23:03.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.059+0000 7f5a8a400640 1 Processor -- start 2024-04-16T16:23:03.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.060+0000 7f5a8a400640 1 -- start start 2024-04-16T16:23:03.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.061+0000 7f5a8a400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5a841452e0 0x7f5a8406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.061+0000 7f5a8a400640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5a8406e920 0x7f5a840ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.061+0000 7f5a8a400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a84071e60 con 0x7f5a840ceaf0 2024-04-16T16:23:03.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.061+0000 7f5a8a400640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5a84072f00 con 0x7f5a8406e920 2024-04-16T16:23:03.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.061+0000 7f5a8a400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5a8406ed90 con 0x7f5a841452e0 2024-04-16T16:23:03.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.062+0000 7f5a83e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5a8406e920 0x7f5a840ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.062+0000 7f5a83e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5a8406e920 0x7f5a840ce5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47492/0 (socket says 172.21.15.45:47492) 2024-04-16T16:23:03.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.062+0000 7f5a83e00640 1 -- 172.21.15.45:0/2808445238 learned_addr learned my addr 172.21.15.45:0/2808445238 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:03.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.062+0000 7f5a89000640 1 --2- 172.21.15.45:0/2808445238 >> v2:172.21.15.103:3300/0 conn(0x7f5a841452e0 0x7f5a8406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.062+0000 7f5a83e00640 1 -- 172.21.15.45:0/2808445238 >> v2:172.21.15.103:3300/0 conn(0x7f5a841452e0 msgr2=0x7f5a8406e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:03.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.062+0000 7f5a83e00640 1 --2- 172.21.15.45:0/2808445238 >> v2:172.21.15.103:3300/0 conn(0x7f5a841452e0 0x7f5a8406e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.062+0000 7f5a83e00640 1 -- 172.21.15.45:0/2808445238 >> v1:172.21.15.103:6789/0 conn(0x7f5a840ceaf0 legacy=0x7f5a84160270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:03.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.062+0000 7f5a83e00640 1 -- 172.21.15.45:0/2808445238 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a84072140 con 0x7f5a8406e920 2024-04-16T16:23:03.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.063+0000 7f5a83e00640 1 --2- 172.21.15.45:0/2808445238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a8406e920 0x7f5a840ce5b0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f5a840702a0 tx=0x7f5a7002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6973dd52c5fef4b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:03.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.064+0000 7f5a83400640 1 -- 172.21.15.45:0/2808445238 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a70034070 con 0x7f5a8406e920 2024-04-16T16:23:03.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.064+0000 7f5a83400640 1 -- 172.21.15.45:0/2808445238 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a70030860 con 0x7f5a8406e920 2024-04-16T16:23:03.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.064+0000 7f5a83400640 1 -- 172.21.15.45:0/2808445238 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a70030c30 con 0x7f5a8406e920 2024-04-16T16:23:03.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.065+0000 7f5a8a400640 1 -- 172.21.15.45:0/2808445238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a8406e920 msgr2=0x7f5a840ce5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:03.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.065+0000 7f5a8a400640 1 --2- 172.21.15.45:0/2808445238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a8406e920 0x7f5a840ce5b0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f5a840702a0 tx=0x7f5a7002b9a0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.065+0000 7f5a8a400640 1 -- 172.21.15.45:0/2808445238 shutdown_connections 2024-04-16T16:23:03.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.065+0000 7f5a8a400640 1 --2- 172.21.15.45:0/2808445238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a8406e920 0x7f5a840ce5b0 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.065+0000 7f5a8a400640 1 --2- 172.21.15.45:0/2808445238 >> v2:172.21.15.103:3300/0 conn(0x7f5a841452e0 0x7f5a8406e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.065+0000 7f5a8a400640 1 -- 172.21.15.45:0/2808445238 >> 172.21.15.45:0/2808445238 conn(0x7f5a840b3bb0 msgr2=0x7f5a840b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:03.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.065+0000 7f5a8a400640 1 -- 172.21.15.45:0/2808445238 shutdown_connections 2024-04-16T16:23:03.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.065+0000 7f5a8a400640 1 -- 172.21.15.45:0/2808445238 wait complete. 2024-04-16T16:23:03.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.067+0000 7f5a8a400640 1 Processor -- start 2024-04-16T16:23:03.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.067+0000 7f5a8a400640 1 -- start start 2024-04-16T16:23:03.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.068+0000 7f5a8a400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a8406e920 0x7f5a84133720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.068+0000 7f5a8a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a840ceaf0 0x7f5a840cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.068+0000 7f5a8a400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a841452e0 0x7f5a840ad950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.068+0000 7f5a8a400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5a840729e0 con 0x7f5a840ceaf0 2024-04-16T16:23:03.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.068+0000 7f5a8a400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5a84072700 con 0x7f5a8406e920 2024-04-16T16:23:03.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.068+0000 7f5a8a400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5a840cef10 con 0x7f5a841452e0 2024-04-16T16:23:03.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.068+0000 7f5a83e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a840ceaf0 0x7f5a840cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.068+0000 7f5a83e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a840ceaf0 0x7f5a840cd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47494/0 (socket says 172.21.15.45:47494) 2024-04-16T16:23:03.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.068+0000 7f5a83e00640 1 -- 172.21.15.45:0/3563278410 learned_addr learned my addr 172.21.15.45:0/3563278410 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.068+0000 7f5a89000640 1 --2- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a8406e920 0x7f5a84133720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.068+0000 7f5a89a00640 1 --2- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a841452e0 0x7f5a840ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.069+0000 7f5a83e00640 1 -- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a841452e0 msgr2=0x7f5a840ad950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.069+0000 7f5a83e00640 1 --2- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a841452e0 0x7f5a840ad950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.069+0000 7f5a83e00640 1 -- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a8406e920 msgr2=0x7f5a84133720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.069+0000 7f5a83e00640 1 --2- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a8406e920 0x7f5a84133720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.069+0000 7f5a83e00640 1 -- 172.21.15.45:0/3563278410 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a840c5640 con 0x7f5a840ceaf0 2024-04-16T16:23:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.069+0000 7f5a89a00640 1 --2- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a841452e0 0x7f5a840ad950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:23:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.069+0000 7f5a83e00640 1 --2- 172.21.15.45:0/3563278410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a840ceaf0 0x7f5a840cd8e0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f5a70000c00 tx=0x7f5a70004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:03.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.069+0000 7f5a80c00640 1 -- 172.21.15.45:0/3563278410 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a70048070 con 0x7f5a840ceaf0 2024-04-16T16:23:03.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.069+0000 7f5a80c00640 1 -- 172.21.15.45:0/3563278410 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a70030660 con 0x7f5a840ceaf0 2024-04-16T16:23:03.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.069+0000 7f5a80c00640 1 -- 172.21.15.45:0/3563278410 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a70034040 con 0x7f5a840ceaf0 2024-04-16T16:23:03.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.069+0000 7f5a8a400640 1 -- 172.21.15.45:0/3563278410 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a84133c60 con 0x7f5a840ceaf0 2024-04-16T16:23:03.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.070+0000 7f5a8a400640 1 -- 172.21.15.45:0/3563278410 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a84130df0 con 0x7f5a840ceaf0 2024-04-16T16:23:03.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.070+0000 7f5a8a400640 1 -- 172.21.15.45:0/3563278410 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5a840c5bc0 con 0x7f5a840ceaf0 2024-04-16T16:23:03.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.072+0000 7f5a80c00640 1 -- 172.21.15.45:0/3563278410 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f5a7004b070 con 0x7f5a840ceaf0 2024-04-16T16:23:03.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.073+0000 7f5a80c00640 1 --2- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5a6406ca10 0x7f5a6406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.073+0000 7f5a80c00640 1 -- 172.21.15.45:0/3563278410 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f5a700adf20 con 0x7f5a840ceaf0 2024-04-16T16:23:03.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.073+0000 7f5a80c00640 1 --2- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:6816/1303888027,v1:172.21.15.103:6817/1303888027] conn(0x7f5a64074bb0 0x7f5a64076fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.073+0000 7f5a80c00640 1 -- 172.21.15.45:0/3563278410 --> [v2:172.21.15.103:6816/1303888027,v1:172.21.15.103:6817/1303888027] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a64077680 con 0x7f5a64074bb0 2024-04-16T16:23:03.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.073+0000 7f5a80c00640 1 -- 172.21.15.45:0/3563278410 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=156) ==== 24+0+0 (secure 0 0 0) 0x7f5a700b0540 con 0x7f5a840ceaf0 2024-04-16T16:23:03.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.073+0000 7f5a89000640 1 --2- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5a6406ca10 0x7f5a6406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.074+0000 7f5a89a00640 1 --2- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:6816/1303888027,v1:172.21.15.103:6817/1303888027] conn(0x7f5a64074bb0 0x7f5a64076fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.075+0000 7f5a89000640 1 --2- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5a6406ca10 0x7f5a6406eed0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5a78006fd0 tx=0x7f5a78008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:03.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.075+0000 7f5a89a00640 1 --2- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:6816/1303888027,v1:172.21.15.103:6817/1303888027] conn(0x7f5a64074bb0 0x7f5a64076fd0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:03.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.077+0000 7f5a80c00640 1 -- 172.21.15.45:0/3563278410 <== osd.7 v2:172.21.15.103:6816/1303888027 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5a64077680 con 0x7f5a64074bb0 2024-04-16T16:23:03.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.092+0000 7f5a8a400640 1 -- 172.21.15.45:0/3563278410 --> [v2:172.21.15.103:6816/1303888027,v1:172.21.15.103:6817/1303888027] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5a84075780 con 0x7f5a64074bb0 2024-04-16T16:23:03.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.093+0000 7f5a80c00640 1 -- 172.21.15.45:0/3563278410 <== osd.7 v2:172.21.15.103:6816/1303888027 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5a84075780 con 0x7f5a64074bb0 2024-04-16T16:23:03.092 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T16:23:03.092 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-16T16:23:03.092 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T16:23:03.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.094+0000 7f5a8a400640 1 -- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:6816/1303888027,v1:172.21.15.103:6817/1303888027] conn(0x7f5a64074bb0 msgr2=0x7f5a64076fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:03.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.094+0000 7f5a8a400640 1 --2- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:6816/1303888027,v1:172.21.15.103:6817/1303888027] conn(0x7f5a64074bb0 0x7f5a64076fd0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.094+0000 7f5a8a400640 1 -- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5a6406ca10 msgr2=0x7f5a6406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:03.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.094+0000 7f5a8a400640 1 --2- 172.21.15.45:0/3563278410 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f5a6406ca10 0x7f5a6406eed0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5a78006fd0 tx=0x7f5a78008040 comp rx=0 tx=0).stop 2024-04-16T16:23:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.094+0000 7f5a8a400640 1 -- 172.21.15.45:0/3563278410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a840ceaf0 msgr2=0x7f5a840cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.094+0000 7f5a8a400640 1 --2- 172.21.15.45:0/3563278410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a840ceaf0 0x7f5a840cd8e0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f5a70000c00 tx=0x7f5a70004730 comp rx=0 tx=0).stop 2024-04-16T16:23:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.094+0000 7f5a89a00640 1 -- 172.21.15.45:0/3563278410 reap_dead start 2024-04-16T16:23:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.094+0000 7f5a8a400640 1 -- 172.21.15.45:0/3563278410 shutdown_connections 2024-04-16T16:23:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.094+0000 7f5a8a400640 1 -- 172.21.15.45:0/3563278410 >> 172.21.15.45:0/3563278410 conn(0x7f5a840b3bb0 msgr2=0x7f5a84142a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.094+0000 7f5a8a400640 1 -- 172.21.15.45:0/3563278410 shutdown_connections 2024-04-16T16:23:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.094+0000 7f5a8a400640 1 -- 172.21.15.45:0/3563278410 wait complete. 2024-04-16T16:23:03.112 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-16T16:23:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.337+0000 7f6336200640 1 Processor -- start 2024-04-16T16:23:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.337+0000 7f6336200640 1 -- start start 2024-04-16T16:23:03.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.339+0000 7f6336200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6330145230 0x7f633015bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.339+0000 7f6336200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f63300c5670 0x7f63300c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.339+0000 7f6336200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f633015d3b0 con 0x7f633015cbe0 2024-04-16T16:23:03.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.339+0000 7f6336200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f63300c6020 con 0x7f63300c5670 2024-04-16T16:23:03.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.339+0000 7f6336200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f633015dfc0 con 0x7f6330145230 2024-04-16T16:23:03.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.339+0000 7f6335800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f63300c5670 0x7f63300c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.339+0000 7f6335800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f63300c5670 0x7f63300c5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41702/0 (socket says 172.21.15.45:41702) 2024-04-16T16:23:03.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.339+0000 7f6335800640 1 -- 172.21.15.45:0/387912987 learned_addr learned my addr 172.21.15.45:0/387912987 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:03.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.339+0000 7f6335800640 1 -- 172.21.15.45:0/387912987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6330145230 msgr2=0x7f633015bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:03.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.339+0000 7f6334e00640 1 --2- 172.21.15.45:0/387912987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6330145230 0x7f633015bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.340+0000 7f6335800640 1 --2- 172.21.15.45:0/387912987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6330145230 0x7f633015bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.340+0000 7f6335800640 1 -- 172.21.15.45:0/387912987 >> v1:172.21.15.103:6789/0 conn(0x7f633015cbe0 legacy=0x7f63300cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:03.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.340+0000 7f6335800640 1 -- 172.21.15.45:0/387912987 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f633015d690 con 0x7f63300c5670 2024-04-16T16:23:03.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.340+0000 7f6334e00640 1 --2- 172.21.15.45:0/387912987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6330145230 0x7f633015bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:23:03.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.341+0000 7f6335800640 1 --2- 172.21.15.45:0/387912987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63300c5670 0x7f63300c5ae0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f6328009e50 tx=0x7f632802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3465ed37cddaf33c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:03.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.341+0000 7f632f400640 1 -- 172.21.15.45:0/387912987 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632802c9e0 con 0x7f63300c5670 2024-04-16T16:23:03.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.341+0000 7f632f400640 1 -- 172.21.15.45:0/387912987 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f632802cb80 con 0x7f63300c5670 2024-04-16T16:23:03.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.341+0000 7f632f400640 1 -- 172.21.15.45:0/387912987 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63280343f0 con 0x7f63300c5670 2024-04-16T16:23:03.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.342+0000 7f6336200640 1 -- 172.21.15.45:0/387912987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63300c5670 msgr2=0x7f63300c5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:03.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.342+0000 7f6336200640 1 --2- 172.21.15.45:0/387912987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63300c5670 0x7f63300c5ae0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f6328009e50 tx=0x7f632802be40 comp rx=0 tx=0).stop 2024-04-16T16:23:03.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.342+0000 7f6336200640 1 -- 172.21.15.45:0/387912987 shutdown_connections 2024-04-16T16:23:03.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.342+0000 7f6336200640 1 --2- 172.21.15.45:0/387912987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63300c5670 0x7f63300c5ae0 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.342+0000 7f6336200640 1 --2- 172.21.15.45:0/387912987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6330145230 0x7f633015bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.342+0000 7f6336200640 1 -- 172.21.15.45:0/387912987 >> 172.21.15.45:0/387912987 conn(0x7f633006b930 msgr2=0x7f633006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:03.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.342+0000 7f6336200640 1 -- 172.21.15.45:0/387912987 shutdown_connections 2024-04-16T16:23:03.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.342+0000 7f6336200640 1 -- 172.21.15.45:0/387912987 wait complete. 2024-04-16T16:23:03.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.343+0000 7f6336200640 1 Processor -- start 2024-04-16T16:23:03.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.344+0000 7f6336200640 1 -- start start 2024-04-16T16:23:03.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f6336200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63300c5670 0x7f6330137530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f6336200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6330145230 0x7f6330130a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f6336200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f633015cbe0 0x7f63301313c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f6336200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f633015d040 con 0x7f6330145230 2024-04-16T16:23:03.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f6336200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f633015dc50 con 0x7f633015cbe0 2024-04-16T16:23:03.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f6336200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f633015e450 con 0x7f63300c5670 2024-04-16T16:23:03.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f632fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6330145230 0x7f6330130a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f632fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6330145230 0x7f6330130a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41704/0 (socket says 172.21.15.45:41704) 2024-04-16T16:23:03.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f632fe00640 1 -- 172.21.15.45:0/2034307457 learned_addr learned my addr 172.21.15.45:0/2034307457 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:03.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f6334e00640 1 --2- 172.21.15.45:0/2034307457 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63300c5670 0x7f6330137530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f6335800640 1 --2- 172.21.15.45:0/2034307457 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f633015cbe0 0x7f63301313c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f632fe00640 1 -- 172.21.15.45:0/2034307457 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63300c5670 msgr2=0x7f6330137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:03.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f632fe00640 1 --2- 172.21.15.45:0/2034307457 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63300c5670 0x7f6330137530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f632fe00640 1 -- 172.21.15.45:0/2034307457 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f633015cbe0 msgr2=0x7f63301313c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:03.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f632fe00640 1 --2- 172.21.15.45:0/2034307457 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f633015cbe0 0x7f63301313c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f632fe00640 1 -- 172.21.15.45:0/2034307457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6330130f70 con 0x7f6330145230 2024-04-16T16:23:03.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.345+0000 7f6334e00640 1 --2- 172.21.15.45:0/2034307457 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63300c5670 0x7f6330137530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:23:03.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.346+0000 7f632fe00640 1 --2- 172.21.15.45:0/2034307457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6330145230 0x7f6330130a30 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f631c00b3f0 tx=0x7f631c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:03.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.346+0000 7f632cc00640 1 -- 172.21.15.45:0/2034307457 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631c015020 con 0x7f6330145230 2024-04-16T16:23:03.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.346+0000 7f632cc00640 1 -- 172.21.15.45:0/2034307457 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f631c011830 con 0x7f6330145230 2024-04-16T16:23:03.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.346+0000 7f632cc00640 1 -- 172.21.15.45:0/2034307457 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631c011c20 con 0x7f6330145230 2024-04-16T16:23:03.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.346+0000 7f6336200640 1 -- 172.21.15.45:0/2034307457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63301334d0 con 0x7f6330145230 2024-04-16T16:23:03.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.346+0000 7f6336200640 1 -- 172.21.15.45:0/2034307457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63300ced90 con 0x7f6330145230 2024-04-16T16:23:03.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.347+0000 7f6336200640 1 -- 172.21.15.45:0/2034307457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6330075780 con 0x7f6330145230 2024-04-16T16:23:03.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.348+0000 7f632cc00640 1 -- 172.21.15.45:0/2034307457 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f631c018020 con 0x7f6330145230 2024-04-16T16:23:03.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.349+0000 7f632cc00640 1 --2- 172.21.15.45:0/2034307457 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f631006ca10 0x7f631006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.349+0000 7f632cc00640 1 -- 172.21.15.45:0/2034307457 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f631c08b8a0 con 0x7f6330145230 2024-04-16T16:23:03.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.350+0000 7f6334e00640 1 --2- 172.21.15.45:0/2034307457 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f631006ca10 0x7f631006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.351+0000 7f6334e00640 1 --2- 172.21.15.45:0/2034307457 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f631006ca10 0x7f631006eed0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6324006fd0 tx=0x7f6324008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:03.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.354+0000 7f632cc00640 1 -- 172.21.15.45:0/2034307457 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f631c01f030 con 0x7f6330145230 2024-04-16T16:23:03.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.562+0000 7f6336200640 1 -- 172.21.15.45:0/2034307457 --> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f63300b5950 con 0x7f631006ca10 2024-04-16T16:23:03.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.563+0000 7f632cc00640 1 -- 172.21.15.45:0/2034307457 <== mgr.7591 v2:172.21.15.103:6832/1129061436 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f63300b5950 con 0x7f631006ca10 2024-04-16T16:23:03.562 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 7 to deep-scrub 2024-04-16T16:23:03.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.566+0000 7f6336200640 1 -- 172.21.15.45:0/2034307457 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f631006ca10 msgr2=0x7f631006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:03.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.566+0000 7f6336200640 1 --2- 172.21.15.45:0/2034307457 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f631006ca10 0x7f631006eed0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6324006fd0 tx=0x7f6324008040 comp rx=0 tx=0).stop 2024-04-16T16:23:03.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.566+0000 7f6336200640 1 -- 172.21.15.45:0/2034307457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6330145230 msgr2=0x7f6330130a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:03.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.566+0000 7f6336200640 1 --2- 172.21.15.45:0/2034307457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6330145230 0x7f6330130a30 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f631c00b3f0 tx=0x7f631c00b8b0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.566+0000 7f6336200640 1 -- 172.21.15.45:0/2034307457 shutdown_connections 2024-04-16T16:23:03.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.566+0000 7f6336200640 1 --2- 172.21.15.45:0/2034307457 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f631006ca10 0x7f631006eed0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.566+0000 7f6336200640 1 --2- 172.21.15.45:0/2034307457 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f633015cbe0 0x7f63301313c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.566+0000 7f6336200640 1 --2- 172.21.15.45:0/2034307457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6330145230 0x7f6330130a30 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.566+0000 7f6336200640 1 --2- 172.21.15.45:0/2034307457 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63300c5670 0x7f6330137530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.566+0000 7f6336200640 1 -- 172.21.15.45:0/2034307457 >> 172.21.15.45:0/2034307457 conn(0x7f633006b930 msgr2=0x7f63300c6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:03.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.567+0000 7f6336200640 1 -- 172.21.15.45:0/2034307457 shutdown_connections 2024-04-16T16:23:03.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.567+0000 7f6336200640 1 -- 172.21.15.45:0/2034307457 wait complete. 2024-04-16T16:23:03.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:23:03.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.808+0000 7f451ec00640 1 Processor -- start 2024-04-16T16:23:03.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.809+0000 7f451ec00640 1 -- start start 2024-04-16T16:23:03.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.810+0000 7f451ec00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4518161e90 0x7f45180cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.810+0000 7f451ec00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f45180d52d0 0x7f451806ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.810+0000 7f451ec00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45180d3080 con 0x7f4518162270 2024-04-16T16:23:03.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.810+0000 7f451ec00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f45180d2e30 con 0x7f4518161e90 2024-04-16T16:23:03.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.810+0000 7f451ec00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f45180d3360 con 0x7f45180d52d0 2024-04-16T16:23:03.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.810+0000 7f451d800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4518161e90 0x7f45180cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.810+0000 7f451d800640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4518161e90 0x7f45180cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41706/0 (socket says 172.21.15.45:41706) 2024-04-16T16:23:03.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.810+0000 7f451d800640 1 -- 172.21.15.45:0/2503649108 learned_addr learned my addr 172.21.15.45:0/2503649108 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:03.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.811+0000 7f451e200640 1 --2- 172.21.15.45:0/2503649108 >> v2:172.21.15.103:3300/0 conn(0x7f45180d52d0 0x7f451806ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.811+0000 7f451d800640 1 -- 172.21.15.45:0/2503649108 >> v2:172.21.15.103:3300/0 conn(0x7f45180d52d0 msgr2=0x7f451806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:03.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.811+0000 7f451d800640 1 --2- 172.21.15.45:0/2503649108 >> v2:172.21.15.103:3300/0 conn(0x7f45180d52d0 0x7f451806ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.811+0000 7f451d800640 1 -- 172.21.15.45:0/2503649108 >> v1:172.21.15.103:6789/0 conn(0x7f4518162270 legacy=0x7f45180d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:03.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.811+0000 7f451d800640 1 -- 172.21.15.45:0/2503649108 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45180d3640 con 0x7f4518161e90 2024-04-16T16:23:03.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.811+0000 7f451e200640 1 --2- 172.21.15.45:0/2503649108 >> v2:172.21.15.103:3300/0 conn(0x7f45180d52d0 0x7f451806ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:23:03.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.812+0000 7f451d800640 1 --2- 172.21.15.45:0/2503649108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4518161e90 0x7f45180cf7d0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f450c009870 tx=0x7f450c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55eb5f7847e5a110 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:03.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.812+0000 7f4517e00640 1 -- 172.21.15.45:0/2503649108 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450c035020 con 0x7f4518161e90 2024-04-16T16:23:03.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.813+0000 7f4517e00640 1 -- 172.21.15.45:0/2503649108 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f450c02d040 con 0x7f4518161e90 2024-04-16T16:23:03.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.813+0000 7f4517e00640 1 -- 172.21.15.45:0/2503649108 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450c031a80 con 0x7f4518161e90 2024-04-16T16:23:03.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.813+0000 7f451ec00640 1 -- 172.21.15.45:0/2503649108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4518161e90 msgr2=0x7f45180cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:03.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.813+0000 7f451ec00640 1 --2- 172.21.15.45:0/2503649108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4518161e90 0x7f45180cf7d0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f450c009870 tx=0x7f450c02b900 comp rx=0 tx=0).stop 2024-04-16T16:23:03.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.813+0000 7f451ec00640 1 -- 172.21.15.45:0/2503649108 shutdown_connections 2024-04-16T16:23:03.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.813+0000 7f451ec00640 1 --2- 172.21.15.45:0/2503649108 >> v2:172.21.15.103:3300/0 conn(0x7f45180d52d0 0x7f451806ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.813+0000 7f451ec00640 1 --2- 172.21.15.45:0/2503649108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4518161e90 0x7f45180cf7d0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.813+0000 7f451ec00640 1 -- 172.21.15.45:0/2503649108 >> 172.21.15.45:0/2503649108 conn(0x7f45180d1940 msgr2=0x7f45180d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:03.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.814+0000 7f451ec00640 1 -- 172.21.15.45:0/2503649108 shutdown_connections 2024-04-16T16:23:03.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.814+0000 7f451ec00640 1 -- 172.21.15.45:0/2503649108 wait complete. 2024-04-16T16:23:03.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.815+0000 7f451ec00640 1 Processor -- start 2024-04-16T16:23:03.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.815+0000 7f451ec00640 1 -- start start 2024-04-16T16:23:03.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.816+0000 7f451ec00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45180d52d0 0x7f45181f8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.816+0000 7f451ec00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4518161e90 0x7f4518159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.816+0000 7f451ec00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4518162270 0x7f451815c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.816+0000 7f451ec00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f45180d4550 con 0x7f4518161e90 2024-04-16T16:23:03.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.816+0000 7f451ec00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f45180cfd10 con 0x7f4518162270 2024-04-16T16:23:03.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.816+0000 7f451ec00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f45180d46d0 con 0x7f45180d52d0 2024-04-16T16:23:03.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.817+0000 7f451ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4518161e90 0x7f4518159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.817+0000 7f451ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4518161e90 0x7f4518159180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41710/0 (socket says 172.21.15.45:41710) 2024-04-16T16:23:03.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.817+0000 7f451ce00640 1 -- 172.21.15.45:0/1389060231 learned_addr learned my addr 172.21.15.45:0/1389060231 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:03.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.817+0000 7f451d800640 1 --2- 172.21.15.45:0/1389060231 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45180d52d0 0x7f45181f8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.817+0000 7f451ce00640 1 -- 172.21.15.45:0/1389060231 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45180d52d0 msgr2=0x7f45181f8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:03.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.817+0000 7f451ce00640 1 --2- 172.21.15.45:0/1389060231 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45180d52d0 0x7f45181f8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.817+0000 7f451ce00640 1 -- 172.21.15.45:0/1389060231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4518162270 msgr2=0x7f451815c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:03.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.817+0000 7f451ce00640 1 --2- 172.21.15.45:0/1389060231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4518162270 0x7f451815c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:03.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.817+0000 7f451ce00640 1 -- 172.21.15.45:0/1389060231 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f451815a0e0 con 0x7f4518161e90 2024-04-16T16:23:03.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.817+0000 7f451d800640 1 --2- 172.21.15.45:0/1389060231 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45180d52d0 0x7f45181f8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:23:03.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.817+0000 7f451ce00640 1 --2- 172.21.15.45:0/1389060231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4518161e90 0x7f4518159180 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f450400b3c0 tx=0x7f450400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:03.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.818+0000 7f4515600640 1 -- 172.21.15.45:0/1389060231 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4504011830 con 0x7f4518161e90 2024-04-16T16:23:03.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.818+0000 7f4515600640 1 -- 172.21.15.45:0/1389060231 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4504016070 con 0x7f4518161e90 2024-04-16T16:23:03.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.818+0000 7f4515600640 1 -- 172.21.15.45:0/1389060231 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4504011c50 con 0x7f4518161e90 2024-04-16T16:23:03.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.818+0000 7f451ec00640 1 -- 172.21.15.45:0/1389060231 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f451815a3c0 con 0x7f4518161e90 2024-04-16T16:23:03.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.818+0000 7f451ec00640 1 -- 172.21.15.45:0/1389060231 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f451815a980 con 0x7f4518161e90 2024-04-16T16:23:03.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.819+0000 7f451ec00640 1 -- 172.21.15.45:0/1389060231 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45180ba7a0 con 0x7f4518161e90 2024-04-16T16:23:03.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.820+0000 7f4515600640 1 -- 172.21.15.45:0/1389060231 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f450401b460 con 0x7f4518161e90 2024-04-16T16:23:03.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.821+0000 7f4515600640 1 --2- 172.21.15.45:0/1389060231 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f44f806ca10 0x7f44f806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:03.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.821+0000 7f4515600640 1 -- 172.21.15.45:0/1389060231 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5709+0+0 (secure 0 0 0) 0x7f450408b800 con 0x7f4518161e90 2024-04-16T16:23:03.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.822+0000 7f451d800640 1 --2- 172.21.15.45:0/1389060231 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f44f806ca10 0x7f44f806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:03.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.823+0000 7f451d800640 1 --2- 172.21.15.45:0/1389060231 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f44f806ca10 0x7f44f806eed0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f450c036000 tx=0x7f450c008030 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:03.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:03.826+0000 7f4515600640 1 -- 172.21.15.45:0/1389060231 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f45180ba7a0 con 0x7f4518161e90 2024-04-16T16:23:04.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:04.028+0000 7f451ec00640 1 -- 172.21.15.45:0/1389060231 --> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4518075780 con 0x7f44f806ca10 2024-04-16T16:23:04.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:04.047+0000 7f4515600640 1 -- 172.21.15.45:0/1389060231 <== mgr.7591 v2:172.21.15.103:6832/1129061436 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250501 (secure 0 0 0) 0x7f4518075780 con 0x7f44f806ca10 2024-04-16T16:23:04.049 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:23:04.050 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:23:04.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:04.052+0000 7f451ec00640 1 -- 172.21.15.45:0/1389060231 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f44f806ca10 msgr2=0x7f44f806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:04.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:04.052+0000 7f451ec00640 1 --2- 172.21.15.45:0/1389060231 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f44f806ca10 0x7f44f806eed0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f450c036000 tx=0x7f450c008030 comp rx=0 tx=0).stop 2024-04-16T16:23:04.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:04.052+0000 7f451ec00640 1 -- 172.21.15.45:0/1389060231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4518161e90 msgr2=0x7f4518159180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:04.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:04.052+0000 7f451ec00640 1 --2- 172.21.15.45:0/1389060231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4518161e90 0x7f4518159180 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f450400b3c0 tx=0x7f450400b880 comp rx=0 tx=0).stop 2024-04-16T16:23:04.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:04.052+0000 7f451ec00640 1 -- 172.21.15.45:0/1389060231 shutdown_connections 2024-04-16T16:23:04.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:04.052+0000 7f451ec00640 1 --2- 172.21.15.45:0/1389060231 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f44f806ca10 0x7f44f806eed0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:04.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:04.052+0000 7f451ec00640 1 --2- 172.21.15.45:0/1389060231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4518162270 0x7f451815c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:04.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:04.052+0000 7f451ec00640 1 --2- 172.21.15.45:0/1389060231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4518161e90 0x7f4518159180 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:04.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:04.052+0000 7f451ec00640 1 --2- 172.21.15.45:0/1389060231 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45180d52d0 0x7f45181f8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:04.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:04.052+0000 7f451ec00640 1 -- 172.21.15.45:0/1389060231 >> 172.21.15.45:0/1389060231 conn(0x7f45180d1940 msgr2=0x7f451806ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:04.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:04.052+0000 7f451ec00640 1 -- 172.21.15.45:0/1389060231 shutdown_connections 2024-04-16T16:23:04.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:04.052+0000 7f451ec00640 1 -- 172.21.15.45:0/1389060231 wait complete. 2024-04-16T16:23:04.071 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-04-16T16:23:03.547109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":431,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4729,"ondisk_log_size":4729,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":286748,"kb_used_data":16520,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754687972,"statfs":{"total":773094113280,"available":772800483328,"internally_reserved":0,"allocated":16916480,"data_stored":8653540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":276614704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005264"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461972+0000","last_change":"2024-04-16T16:22:46.488567+0000","last_active":"2024-04-16T16:22:47.461972+0000","last_peered":"2024-04-16T16:22:47.461972+0000","last_clean":"2024-04-16T16:22:47.461972+0000","last_became_active":"2024-04-16T16:22:46.485690+0000","last_became_peered":"2024-04-16T16:22:46.485690+0000","last_unstale":"2024-04-16T16:22:47.461972+0000","last_undegraded":"2024-04-16T16:22:47.461972+0000","last_fullsized":"2024-04-16T16:22:47.461972+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:50:29.718907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426342+0000","last_change":"2024-04-16T16:22:46.479620+0000","last_active":"2024-04-16T16:22:48.426342+0000","last_peered":"2024-04-16T16:22:48.426342+0000","last_clean":"2024-04-16T16:22:48.426342+0000","last_became_active":"2024-04-16T16:22:46.477410+0000","last_became_peered":"2024-04-16T16:22:46.477410+0000","last_unstale":"2024-04-16T16:22:48.426342+0000","last_undegraded":"2024-04-16T16:22:48.426342+0000","last_fullsized":"2024-04-16T16:22:48.426342+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:35.010598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624440+0000","last_change":"2024-04-16T16:22:46.480944+0000","last_active":"2024-04-16T16:22:48.624440+0000","last_peered":"2024-04-16T16:22:48.624440+0000","last_clean":"2024-04-16T16:22:48.624440+0000","last_became_active":"2024-04-16T16:22:46.480405+0000","last_became_peered":"2024-04-16T16:22:46.480405+0000","last_unstale":"2024-04-16T16:22:48.624440+0000","last_undegraded":"2024-04-16T16:22:48.624440+0000","last_fullsized":"2024-04-16T16:22:48.624440+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:35.308909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077781+0000","last_change":"2024-04-16T16:22:46.477832+0000","last_active":"2024-04-16T16:22:48.077781+0000","last_peered":"2024-04-16T16:22:48.077781+0000","last_clean":"2024-04-16T16:22:48.077781+0000","last_became_active":"2024-04-16T16:22:46.477516+0000","last_became_peered":"2024-04-16T16:22:46.477516+0000","last_unstale":"2024-04-16T16:22:48.077781+0000","last_undegraded":"2024-04-16T16:22:48.077781+0000","last_fullsized":"2024-04-16T16:22:48.077781+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426062+0000","last_change":"2024-04-16T16:22:46.483466+0000","last_active":"2024-04-16T16:22:48.426062+0000","last_peered":"2024-04-16T16:22:48.426062+0000","last_clean":"2024-04-16T16:22:48.426062+0000","last_became_active":"2024-04-16T16:22:46.481033+0000","last_became_peered":"2024-04-16T16:22:46.481033+0000","last_unstale":"2024-04-16T16:22:48.426062+0000","last_undegraded":"2024-04-16T16:22:48.426062+0000","last_fullsized":"2024-04-16T16:22:48.426062+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:36:30.108257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710178+0000","last_change":"2024-04-16T16:22:46.463298+0000","last_active":"2024-04-16T16:22:48.710178+0000","last_peered":"2024-04-16T16:22:48.710178+0000","last_clean":"2024-04-16T16:22:48.710178+0000","last_became_active":"2024-04-16T16:22:46.462017+0000","last_became_peered":"2024-04-16T16:22:46.462017+0000","last_unstale":"2024-04-16T16:22:48.710178+0000","last_undegraded":"2024-04-16T16:22:48.710178+0000","last_fullsized":"2024-04-16T16:22:48.710178+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":21,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:23:00.671611+0000","last_change":"2024-04-16T16:23:00.671670+0000","last_active":"2024-04-16T16:23:00.671611+0000","last_peered":"2024-04-16T16:23:00.671611+0000","last_clean":"2024-04-16T16:23:00.671611+0000","last_became_active":"2024-04-16T16:22:46.461856+0000","last_became_peered":"2024-04-16T16:22:46.461856+0000","last_unstale":"2024-04-16T16:23:00.671611+0000","last_undegraded":"2024-04-16T16:23:00.671611+0000","last_fullsized":"2024-04-16T16:23:00.671611+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:00.671426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:00.671426+0000","last_clean_scrub_stamp":"2024-04-16T16:23:00.671426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:35:01.857190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426662+0000","last_change":"2024-04-16T16:22:46.486917+0000","last_active":"2024-04-16T16:22:48.426662+0000","last_peered":"2024-04-16T16:22:48.426662+0000","last_clean":"2024-04-16T16:22:48.426662+0000","last_became_active":"2024-04-16T16:22:46.485280+0000","last_became_peered":"2024-04-16T16:22:46.485280+0000","last_unstale":"2024-04-16T16:22:48.426662+0000","last_undegraded":"2024-04-16T16:22:48.426662+0000","last_fullsized":"2024-04-16T16:22:48.426662+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:14:27.242053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.427106+0000","last_change":"2024-04-16T16:22:46.476513+0000","last_active":"2024-04-16T16:22:48.427106+0000","last_peered":"2024-04-16T16:22:48.427106+0000","last_clean":"2024-04-16T16:22:48.427106+0000","last_became_active":"2024-04-16T16:22:46.476125+0000","last_became_peered":"2024-04-16T16:22:46.476125+0000","last_unstale":"2024-04-16T16:22:48.427106+0000","last_undegraded":"2024-04-16T16:22:48.427106+0000","last_fullsized":"2024-04-16T16:22:48.427106+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:28.349945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426898+0000","last_change":"2024-04-16T16:22:46.483287+0000","last_active":"2024-04-16T16:22:48.426898+0000","last_peered":"2024-04-16T16:22:48.426898+0000","last_clean":"2024-04-16T16:22:48.426898+0000","last_became_active":"2024-04-16T16:22:46.482975+0000","last_became_peered":"2024-04-16T16:22:46.482975+0000","last_unstale":"2024-04-16T16:22:48.426898+0000","last_undegraded":"2024-04-16T16:22:48.426898+0000","last_fullsized":"2024-04-16T16:22:48.426898+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:21.784358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426384+0000","last_change":"2024-04-16T16:22:46.481301+0000","last_active":"2024-04-16T16:22:48.426384+0000","last_peered":"2024-04-16T16:22:48.426384+0000","last_clean":"2024-04-16T16:22:48.426384+0000","last_became_active":"2024-04-16T16:22:46.481000+0000","last_became_peered":"2024-04-16T16:22:46.481000+0000","last_unstale":"2024-04-16T16:22:48.426384+0000","last_undegraded":"2024-04-16T16:22:48.426384+0000","last_fullsized":"2024-04-16T16:22:48.426384+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:01.883490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426458+0000","last_change":"2024-04-16T16:22:46.476179+0000","last_active":"2024-04-16T16:22:48.426458+0000","last_peered":"2024-04-16T16:22:48.426458+0000","last_clean":"2024-04-16T16:22:48.426458+0000","last_became_active":"2024-04-16T16:22:46.475898+0000","last_became_peered":"2024-04-16T16:22:46.475898+0000","last_unstale":"2024-04-16T16:22:48.426458+0000","last_undegraded":"2024-04-16T16:22:48.426458+0000","last_fullsized":"2024-04-16T16:22:48.426458+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:37.006661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624317+0000","last_change":"2024-04-16T16:22:46.482186+0000","last_active":"2024-04-16T16:22:48.624317+0000","last_peered":"2024-04-16T16:22:48.624317+0000","last_clean":"2024-04-16T16:22:48.624317+0000","last_became_active":"2024-04-16T16:22:46.481776+0000","last_became_peered":"2024-04-16T16:22:46.481776+0000","last_unstale":"2024-04-16T16:22:48.624317+0000","last_undegraded":"2024-04-16T16:22:48.624317+0000","last_fullsized":"2024-04-16T16:22:48.624317+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:32:25.664832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623895+0000","last_change":"2024-04-16T16:22:46.477861+0000","last_active":"2024-04-16T16:22:48.623895+0000","last_peered":"2024-04-16T16:22:48.623895+0000","last_clean":"2024-04-16T16:22:48.623895+0000","last_became_active":"2024-04-16T16:22:46.476486+0000","last_became_peered":"2024-04-16T16:22:46.476486+0000","last_unstale":"2024-04-16T16:22:48.623895+0000","last_undegraded":"2024-04-16T16:22:48.623895+0000","last_fullsized":"2024-04-16T16:22:48.623895+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:08.580218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460332+0000","last_change":"2024-04-16T16:22:46.468319+0000","last_active":"2024-04-16T16:22:47.460332+0000","last_peered":"2024-04-16T16:22:47.460332+0000","last_clean":"2024-04-16T16:22:47.460332+0000","last_became_active":"2024-04-16T16:22:46.463117+0000","last_became_peered":"2024-04-16T16:22:46.463117+0000","last_unstale":"2024-04-16T16:22:47.460332+0000","last_undegraded":"2024-04-16T16:22:47.460332+0000","last_fullsized":"2024-04-16T16:22:47.460332+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:12:58.505334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635377+0000","last_change":"2024-04-16T16:22:46.467824+0000","last_active":"2024-04-16T16:22:48.635377+0000","last_peered":"2024-04-16T16:22:48.635377+0000","last_clean":"2024-04-16T16:22:48.635377+0000","last_became_active":"2024-04-16T16:22:46.467274+0000","last_became_peered":"2024-04-16T16:22:46.467274+0000","last_unstale":"2024-04-16T16:22:48.635377+0000","last_undegraded":"2024-04-16T16:22:48.635377+0000","last_fullsized":"2024-04-16T16:22:48.635377+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710525+0000","last_change":"2024-04-16T16:22:46.475037+0000","last_active":"2024-04-16T16:22:48.710525+0000","last_peered":"2024-04-16T16:22:48.710525+0000","last_clean":"2024-04-16T16:22:48.710525+0000","last_became_active":"2024-04-16T16:22:46.474757+0000","last_became_peered":"2024-04-16T16:22:46.474757+0000","last_unstale":"2024-04-16T16:22:48.710525+0000","last_undegraded":"2024-04-16T16:22:48.710525+0000","last_fullsized":"2024-04-16T16:22:48.710525+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457405+0000","last_change":"2024-04-16T16:22:46.463139+0000","last_active":"2024-04-16T16:22:47.457405+0000","last_peered":"2024-04-16T16:22:47.457405+0000","last_clean":"2024-04-16T16:22:47.457405+0000","last_became_active":"2024-04-16T16:22:46.462857+0000","last_became_peered":"2024-04-16T16:22:46.462857+0000","last_unstale":"2024-04-16T16:22:47.457405+0000","last_undegraded":"2024-04-16T16:22:47.457405+0000","last_fullsized":"2024-04-16T16:22:47.457405+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710546+0000","last_change":"2024-04-16T16:22:46.457033+0000","last_active":"2024-04-16T16:22:48.710546+0000","last_peered":"2024-04-16T16:22:48.710546+0000","last_clean":"2024-04-16T16:22:48.710546+0000","last_became_active":"2024-04-16T16:22:46.456756+0000","last_became_peered":"2024-04-16T16:22:46.456756+0000","last_unstale":"2024-04-16T16:22:48.710546+0000","last_undegraded":"2024-04-16T16:22:48.710546+0000","last_fullsized":"2024-04-16T16:22:48.710546+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624366+0000","last_change":"2024-04-16T16:22:46.482119+0000","last_active":"2024-04-16T16:22:48.624366+0000","last_peered":"2024-04-16T16:22:48.624366+0000","last_clean":"2024-04-16T16:22:48.624366+0000","last_became_active":"2024-04-16T16:22:46.479857+0000","last_became_peered":"2024-04-16T16:22:46.479857+0000","last_unstale":"2024-04-16T16:22:48.624366+0000","last_undegraded":"2024-04-16T16:22:48.624366+0000","last_fullsized":"2024-04-16T16:22:48.624366+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:05:36.568682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710387+0000","last_change":"2024-04-16T16:22:46.479160+0000","last_active":"2024-04-16T16:22:48.710387+0000","last_peered":"2024-04-16T16:22:48.710387+0000","last_clean":"2024-04-16T16:22:48.710387+0000","last_became_active":"2024-04-16T16:22:46.477081+0000","last_became_peered":"2024-04-16T16:22:46.477081+0000","last_unstale":"2024-04-16T16:22:48.710387+0000","last_undegraded":"2024-04-16T16:22:48.710387+0000","last_fullsized":"2024-04-16T16:22:48.710387+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426469+0000","last_change":"2024-04-16T16:22:46.473896+0000","last_active":"2024-04-16T16:22:48.426469+0000","last_peered":"2024-04-16T16:22:48.426469+0000","last_clean":"2024-04-16T16:22:48.426469+0000","last_became_active":"2024-04-16T16:22:46.473621+0000","last_became_peered":"2024-04-16T16:22:46.473621+0000","last_unstale":"2024-04-16T16:22:48.426469+0000","last_undegraded":"2024-04-16T16:22:48.426469+0000","last_fullsized":"2024-04-16T16:22:48.426469+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:26.785078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710155+0000","last_change":"2024-04-16T16:22:46.475807+0000","last_active":"2024-04-16T16:22:48.710155+0000","last_peered":"2024-04-16T16:22:48.710155+0000","last_clean":"2024-04-16T16:22:48.710155+0000","last_became_active":"2024-04-16T16:22:46.473183+0000","last_became_peered":"2024-04-16T16:22:46.473183+0000","last_unstale":"2024-04-16T16:22:48.710155+0000","last_undegraded":"2024-04-16T16:22:48.710155+0000","last_fullsized":"2024-04-16T16:22:48.710155+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426404+0000","last_change":"2024-04-16T16:22:46.477670+0000","last_active":"2024-04-16T16:22:48.426404+0000","last_peered":"2024-04-16T16:22:48.426404+0000","last_clean":"2024-04-16T16:22:48.426404+0000","last_became_active":"2024-04-16T16:22:46.477303+0000","last_became_peered":"2024-04-16T16:22:46.477303+0000","last_unstale":"2024-04-16T16:22:48.426404+0000","last_undegraded":"2024-04-16T16:22:48.426404+0000","last_fullsized":"2024-04-16T16:22:48.426404+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:02:37.502947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.625109+0000","last_change":"2024-04-16T16:22:46.481055+0000","last_active":"2024-04-16T16:22:48.625109+0000","last_peered":"2024-04-16T16:22:48.625109+0000","last_clean":"2024-04-16T16:22:48.625109+0000","last_became_active":"2024-04-16T16:22:46.480641+0000","last_became_peered":"2024-04-16T16:22:46.480641+0000","last_unstale":"2024-04-16T16:22:48.625109+0000","last_undegraded":"2024-04-16T16:22:48.625109+0000","last_fullsized":"2024-04-16T16:22:48.625109+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:53:06.786966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635764+0000","last_change":"2024-04-16T16:22:46.467282+0000","last_active":"2024-04-16T16:22:48.635764+0000","last_peered":"2024-04-16T16:22:48.635764+0000","last_clean":"2024-04-16T16:22:48.635764+0000","last_became_active":"2024-04-16T16:22:46.466756+0000","last_became_peered":"2024-04-16T16:22:46.466756+0000","last_unstale":"2024-04-16T16:22:48.635764+0000","last_undegraded":"2024-04-16T16:22:48.635764+0000","last_fullsized":"2024-04-16T16:22:48.635764+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624687+0000","last_change":"2024-04-16T16:22:46.484066+0000","last_active":"2024-04-16T16:22:48.624687+0000","last_peered":"2024-04-16T16:22:48.624687+0000","last_clean":"2024-04-16T16:22:48.624687+0000","last_became_active":"2024-04-16T16:22:46.483654+0000","last_became_peered":"2024-04-16T16:22:46.483654+0000","last_unstale":"2024-04-16T16:22:48.624687+0000","last_undegraded":"2024-04-16T16:22:48.624687+0000","last_fullsized":"2024-04-16T16:22:48.624687+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:44:40.829325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077902+0000","last_change":"2024-04-16T16:22:46.484111+0000","last_active":"2024-04-16T16:22:48.077902+0000","last_peered":"2024-04-16T16:22:48.077902+0000","last_clean":"2024-04-16T16:22:48.077902+0000","last_became_active":"2024-04-16T16:22:46.483750+0000","last_became_peered":"2024-04-16T16:22:46.483750+0000","last_unstale":"2024-04-16T16:22:48.077902+0000","last_undegraded":"2024-04-16T16:22:48.077902+0000","last_fullsized":"2024-04-16T16:22:48.077902+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.625014+0000","last_change":"2024-04-16T16:22:46.483339+0000","last_active":"2024-04-16T16:22:48.625014+0000","last_peered":"2024-04-16T16:22:48.625014+0000","last_clean":"2024-04-16T16:22:48.625014+0000","last_became_active":"2024-04-16T16:22:46.481358+0000","last_became_peered":"2024-04-16T16:22:46.481358+0000","last_unstale":"2024-04-16T16:22:48.625014+0000","last_undegraded":"2024-04-16T16:22:48.625014+0000","last_fullsized":"2024-04-16T16:22:48.625014+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:13.979255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461163+0000","last_change":"2024-04-16T16:22:46.464311+0000","last_active":"2024-04-16T16:22:47.461163+0000","last_peered":"2024-04-16T16:22:47.461163+0000","last_clean":"2024-04-16T16:22:47.461163+0000","last_became_active":"2024-04-16T16:22:46.463536+0000","last_became_peered":"2024-04-16T16:22:46.463536+0000","last_unstale":"2024-04-16T16:22:47.461163+0000","last_undegraded":"2024-04-16T16:22:47.461163+0000","last_fullsized":"2024-04-16T16:22:47.461163+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461570+0000","last_change":"2024-04-16T16:22:46.460441+0000","last_active":"2024-04-16T16:22:47.461570+0000","last_peered":"2024-04-16T16:22:47.461570+0000","last_clean":"2024-04-16T16:22:47.461570+0000","last_became_active":"2024-04-16T16:22:46.458962+0000","last_became_peered":"2024-04-16T16:22:46.458962+0000","last_unstale":"2024-04-16T16:22:47.461570+0000","last_undegraded":"2024-04-16T16:22:47.461570+0000","last_fullsized":"2024-04-16T16:22:47.461570+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.456737+0000","last_change":"2024-04-16T16:22:46.468514+0000","last_active":"2024-04-16T16:22:47.456737+0000","last_peered":"2024-04-16T16:22:47.456737+0000","last_clean":"2024-04-16T16:22:47.456737+0000","last_became_active":"2024-04-16T16:22:46.464851+0000","last_became_peered":"2024-04-16T16:22:46.464851+0000","last_unstale":"2024-04-16T16:22:47.456737+0000","last_undegraded":"2024-04-16T16:22:47.456737+0000","last_fullsized":"2024-04-16T16:22:47.456737+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635990+0000","last_change":"2024-04-16T16:22:46.467039+0000","last_active":"2024-04-16T16:22:48.635990+0000","last_peered":"2024-04-16T16:22:48.635990+0000","last_clean":"2024-04-16T16:22:48.635990+0000","last_became_active":"2024-04-16T16:22:46.466366+0000","last_became_peered":"2024-04-16T16:22:46.466366+0000","last_unstale":"2024-04-16T16:22:48.635990+0000","last_undegraded":"2024-04-16T16:22:48.635990+0000","last_fullsized":"2024-04-16T16:22:48.635990+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"155'4","reported_seq":18,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461935+0000","last_change":"2024-04-16T16:22:45.470039+0000","last_active":"2024-04-16T16:22:47.461935+0000","last_peered":"2024-04-16T16:22:47.461935+0000","last_clean":"2024-04-16T16:22:47.461935+0000","last_became_active":"2024-04-16T16:22:45.467695+0000","last_became_peered":"2024-04-16T16:22:45.467695+0000","last_unstale":"2024-04-16T16:22:47.461935+0000","last_undegraded":"2024-04-16T16:22:47.461935+0000","last_fullsized":"2024-04-16T16:22:47.461935+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:21.824987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.464044+0000","last_change":"2024-04-16T16:22:46.466378+0000","last_active":"2024-04-16T16:22:47.464044+0000","last_peered":"2024-04-16T16:22:47.464044+0000","last_clean":"2024-04-16T16:22:47.464044+0000","last_became_active":"2024-04-16T16:22:46.462099+0000","last_became_peered":"2024-04-16T16:22:46.462099+0000","last_unstale":"2024-04-16T16:22:47.464044+0000","last_undegraded":"2024-04-16T16:22:47.464044+0000","last_fullsized":"2024-04-16T16:22:47.464044+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"155'1","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463791+0000","last_change":"2024-04-16T16:22:45.464685+0000","last_active":"2024-04-16T16:22:47.463791+0000","last_peered":"2024-04-16T16:22:47.463791+0000","last_clean":"2024-04-16T16:22:47.463791+0000","last_became_active":"2024-04-16T16:22:45.464379+0000","last_became_peered":"2024-04-16T16:22:45.464379+0000","last_unstale":"2024-04-16T16:22:47.463791+0000","last_undegraded":"2024-04-16T16:22:47.463791+0000","last_fullsized":"2024-04-16T16:22:47.463791+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710156+0000","last_change":"2024-04-16T16:22:46.456302+0000","last_active":"2024-04-16T16:22:48.710156+0000","last_peered":"2024-04-16T16:22:48.710156+0000","last_clean":"2024-04-16T16:22:48.710156+0000","last_became_active":"2024-04-16T16:22:46.454610+0000","last_became_peered":"2024-04-16T16:22:46.454610+0000","last_unstale":"2024-04-16T16:22:48.710156+0000","last_undegraded":"2024-04-16T16:22:48.710156+0000","last_fullsized":"2024-04-16T16:22:48.710156+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"155'2","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077554+0000","last_change":"2024-04-16T16:22:45.469877+0000","last_active":"2024-04-16T16:22:48.077554+0000","last_peered":"2024-04-16T16:22:48.077554+0000","last_clean":"2024-04-16T16:22:48.077554+0000","last_became_active":"2024-04-16T16:22:45.469537+0000","last_became_peered":"2024-04-16T16:22:45.469537+0000","last_unstale":"2024-04-16T16:22:48.077554+0000","last_undegraded":"2024-04-16T16:22:48.077554+0000","last_fullsized":"2024-04-16T16:22:48.077554+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624791+0000","last_change":"2024-04-16T16:22:46.480073+0000","last_active":"2024-04-16T16:22:48.624791+0000","last_peered":"2024-04-16T16:22:48.624791+0000","last_clean":"2024-04-16T16:22:48.624791+0000","last_became_active":"2024-04-16T16:22:46.479206+0000","last_became_peered":"2024-04-16T16:22:46.479206+0000","last_unstale":"2024-04-16T16:22:48.624791+0000","last_undegraded":"2024-04-16T16:22:48.624791+0000","last_fullsized":"2024-04-16T16:22:48.624791+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:27.266104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"155'2","reported_seq":25,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:23:01.123242+0000","last_change":"2024-04-16T16:23:01.123321+0000","last_active":"2024-04-16T16:23:01.123242+0000","last_peered":"2024-04-16T16:23:01.123242+0000","last_clean":"2024-04-16T16:23:01.123242+0000","last_became_active":"2024-04-16T16:22:45.462170+0000","last_became_peered":"2024-04-16T16:22:45.462170+0000","last_unstale":"2024-04-16T16:23:01.123242+0000","last_undegraded":"2024-04-16T16:23:01.123242+0000","last_fullsized":"2024-04-16T16:23:01.123242+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'2","last_scrub_stamp":"2024-04-16T16:23:01.123009+0000","last_deep_scrub":"155'2","last_deep_scrub_stamp":"2024-04-16T16:23:01.123009+0000","last_clean_scrub_stamp":"2024-04-16T16:23:01.123009+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:26:54.097272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":21,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:59.695994+0000","last_change":"2024-04-16T16:22:59.696044+0000","last_active":"2024-04-16T16:22:59.695994+0000","last_peered":"2024-04-16T16:22:59.695994+0000","last_clean":"2024-04-16T16:22:59.695994+0000","last_became_active":"2024-04-16T16:22:46.459876+0000","last_became_peered":"2024-04-16T16:22:46.459876+0000","last_unstale":"2024-04-16T16:22:59.695994+0000","last_undegraded":"2024-04-16T16:22:59.695994+0000","last_fullsized":"2024-04-16T16:22:59.695994+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:59.695809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:59.695809+0000","last_clean_scrub_stamp":"2024-04-16T16:22:59.695809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:17:12.534395+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624602+0000","last_change":"2024-04-16T16:22:45.472425+0000","last_active":"2024-04-16T16:22:48.624602+0000","last_peered":"2024-04-16T16:22:48.624602+0000","last_clean":"2024-04-16T16:22:48.624602+0000","last_became_active":"2024-04-16T16:22:45.472020+0000","last_became_peered":"2024-04-16T16:22:45.472020+0000","last_unstale":"2024-04-16T16:22:48.624602+0000","last_undegraded":"2024-04-16T16:22:48.624602+0000","last_fullsized":"2024-04-16T16:22:48.624602+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:48:51.793606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077345+0000","last_change":"2024-04-16T16:22:46.470665+0000","last_active":"2024-04-16T16:22:48.077345+0000","last_peered":"2024-04-16T16:22:48.077345+0000","last_clean":"2024-04-16T16:22:48.077345+0000","last_became_active":"2024-04-16T16:22:46.467898+0000","last_became_peered":"2024-04-16T16:22:46.467898+0000","last_unstale":"2024-04-16T16:22:48.077345+0000","last_undegraded":"2024-04-16T16:22:48.077345+0000","last_fullsized":"2024-04-16T16:22:48.077345+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"155'1","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:59.102254+0000","last_change":"2024-04-16T16:22:59.102285+0000","last_active":"2024-04-16T16:22:59.102254+0000","last_peered":"2024-04-16T16:22:59.102254+0000","last_clean":"2024-04-16T16:22:59.102254+0000","last_became_active":"2024-04-16T16:22:45.461674+0000","last_became_peered":"2024-04-16T16:22:45.461674+0000","last_unstale":"2024-04-16T16:22:59.102254+0000","last_undegraded":"2024-04-16T16:22:59.102254+0000","last_fullsized":"2024-04-16T16:22:59.102254+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'1","last_scrub_stamp":"2024-04-16T16:22:59.102114+0000","last_deep_scrub":"155'1","last_deep_scrub_stamp":"2024-04-16T16:22:59.102114+0000","last_clean_scrub_stamp":"2024-04-16T16:22:59.102114+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:29:48.667650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077055+0000","last_change":"2024-04-16T16:22:46.465672+0000","last_active":"2024-04-16T16:22:48.077055+0000","last_peered":"2024-04-16T16:22:48.077055+0000","last_clean":"2024-04-16T16:22:48.077055+0000","last_became_active":"2024-04-16T16:22:46.465329+0000","last_became_peered":"2024-04-16T16:22:46.465329+0000","last_unstale":"2024-04-16T16:22:48.077055+0000","last_undegraded":"2024-04-16T16:22:48.077055+0000","last_fullsized":"2024-04-16T16:22:48.077055+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.425887+0000","last_change":"2024-04-16T16:22:45.466868+0000","last_active":"2024-04-16T16:22:48.425887+0000","last_peered":"2024-04-16T16:22:48.425887+0000","last_clean":"2024-04-16T16:22:48.425887+0000","last_became_active":"2024-04-16T16:22:45.466591+0000","last_became_peered":"2024-04-16T16:22:45.466591+0000","last_unstale":"2024-04-16T16:22:48.425887+0000","last_undegraded":"2024-04-16T16:22:48.425887+0000","last_fullsized":"2024-04-16T16:22:48.425887+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:25.557203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710189+0000","last_change":"2024-04-16T16:22:46.460701+0000","last_active":"2024-04-16T16:22:48.710189+0000","last_peered":"2024-04-16T16:22:48.710189+0000","last_clean":"2024-04-16T16:22:48.710189+0000","last_became_active":"2024-04-16T16:22:46.460420+0000","last_became_peered":"2024-04-16T16:22:46.460420+0000","last_unstale":"2024-04-16T16:22:48.710189+0000","last_undegraded":"2024-04-16T16:22:48.710189+0000","last_fullsized":"2024-04-16T16:22:48.710189+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:58.730251+0000","last_change":"2024-04-16T16:22:58.730308+0000","last_active":"2024-04-16T16:22:58.730251+0000","last_peered":"2024-04-16T16:22:58.730251+0000","last_clean":"2024-04-16T16:22:58.730251+0000","last_became_active":"2024-04-16T16:22:45.455767+0000","last_became_peered":"2024-04-16T16:22:45.455767+0000","last_unstale":"2024-04-16T16:22:58.730251+0000","last_undegraded":"2024-04-16T16:22:58.730251+0000","last_fullsized":"2024-04-16T16:22:58.730251+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:58.730057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:58.730057+0000","last_clean_scrub_stamp":"2024-04-16T16:22:58.730057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:00:28.919352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457440+0000","last_change":"2024-04-16T16:22:46.467743+0000","last_active":"2024-04-16T16:22:47.457440+0000","last_peered":"2024-04-16T16:22:47.457440+0000","last_clean":"2024-04-16T16:22:47.457440+0000","last_became_active":"2024-04-16T16:22:46.465149+0000","last_became_peered":"2024-04-16T16:22:46.465149+0000","last_unstale":"2024-04-16T16:22:47.457440+0000","last_undegraded":"2024-04-16T16:22:47.457440+0000","last_fullsized":"2024-04-16T16:22:47.457440+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:23:01.438937+0000","last_change":"2024-04-16T16:23:01.438986+0000","last_active":"2024-04-16T16:23:01.438937+0000","last_peered":"2024-04-16T16:23:01.438937+0000","last_clean":"2024-04-16T16:23:01.438937+0000","last_became_active":"2024-04-16T16:22:45.467704+0000","last_became_peered":"2024-04-16T16:22:45.467704+0000","last_unstale":"2024-04-16T16:23:01.438937+0000","last_undegraded":"2024-04-16T16:23:01.438937+0000","last_fullsized":"2024-04-16T16:23:01.438937+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:01.438795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:01.438795+0000","last_clean_scrub_stamp":"2024-04-16T16:23:01.438795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:34:51.177901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457170+0000","last_change":"2024-04-16T16:22:46.473841+0000","last_active":"2024-04-16T16:22:47.457170+0000","last_peered":"2024-04-16T16:22:47.457170+0000","last_clean":"2024-04-16T16:22:47.457170+0000","last_became_active":"2024-04-16T16:22:46.473500+0000","last_became_peered":"2024-04-16T16:22:46.473500+0000","last_unstale":"2024-04-16T16:22:47.457170+0000","last_undegraded":"2024-04-16T16:22:47.457170+0000","last_fullsized":"2024-04-16T16:22:47.457170+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"155'3","reported_seq":26,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:23:00.430805+0000","last_change":"2024-04-16T16:23:00.430865+0000","last_active":"2024-04-16T16:23:00.430805+0000","last_peered":"2024-04-16T16:23:00.430805+0000","last_clean":"2024-04-16T16:23:00.430805+0000","last_became_active":"2024-04-16T16:22:45.461088+0000","last_became_peered":"2024-04-16T16:22:45.461088+0000","last_unstale":"2024-04-16T16:23:00.430805+0000","last_undegraded":"2024-04-16T16:23:00.430805+0000","last_fullsized":"2024-04-16T16:23:00.430805+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'3","last_scrub_stamp":"2024-04-16T16:23:00.430607+0000","last_deep_scrub":"155'3","last_deep_scrub_stamp":"2024-04-16T16:23:00.430607+0000","last_clean_scrub_stamp":"2024-04-16T16:23:00.430607+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:43:02.426690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.464823+0000","last_change":"2024-04-16T16:22:46.473263+0000","last_active":"2024-04-16T16:22:47.464823+0000","last_peered":"2024-04-16T16:22:47.464823+0000","last_clean":"2024-04-16T16:22:47.464823+0000","last_became_active":"2024-04-16T16:22:46.472979+0000","last_became_peered":"2024-04-16T16:22:46.472979+0000","last_unstale":"2024-04-16T16:22:47.464823+0000","last_undegraded":"2024-04-16T16:22:47.464823+0000","last_fullsized":"2024-04-16T16:22:47.464823+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"155'6","reported_seq":21,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463345+0000","last_change":"2024-04-16T16:22:45.462782+0000","last_active":"2024-04-16T16:22:47.463345+0000","last_peered":"2024-04-16T16:22:47.463345+0000","last_clean":"2024-04-16T16:22:47.463345+0000","last_became_active":"2024-04-16T16:22:45.461583+0000","last_became_peered":"2024-04-16T16:22:45.461583+0000","last_unstale":"2024-04-16T16:22:47.463345+0000","last_undegraded":"2024-04-16T16:22:47.463345+0000","last_fullsized":"2024-04-16T16:22:47.463345+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.456729+0000","last_change":"2024-04-16T16:22:46.466170+0000","last_active":"2024-04-16T16:22:47.456729+0000","last_peered":"2024-04-16T16:22:47.456729+0000","last_clean":"2024-04-16T16:22:47.456729+0000","last_became_active":"2024-04-16T16:22:46.465894+0000","last_became_peered":"2024-04-16T16:22:46.465894+0000","last_unstale":"2024-04-16T16:22:47.456729+0000","last_undegraded":"2024-04-16T16:22:47.456729+0000","last_fullsized":"2024-04-16T16:22:47.456729+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"155'2","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077464+0000","last_change":"2024-04-16T16:22:45.466273+0000","last_active":"2024-04-16T16:22:48.077464+0000","last_peered":"2024-04-16T16:22:48.077464+0000","last_clean":"2024-04-16T16:22:48.077464+0000","last_became_active":"2024-04-16T16:22:45.465893+0000","last_became_peered":"2024-04-16T16:22:45.465893+0000","last_unstale":"2024-04-16T16:22:48.077464+0000","last_undegraded":"2024-04-16T16:22:48.077464+0000","last_fullsized":"2024-04-16T16:22:48.077464+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.636319+0000","last_change":"2024-04-16T16:22:46.474507+0000","last_active":"2024-04-16T16:22:48.636319+0000","last_peered":"2024-04-16T16:22:48.636319+0000","last_clean":"2024-04-16T16:22:48.636319+0000","last_became_active":"2024-04-16T16:22:46.474146+0000","last_became_peered":"2024-04-16T16:22:46.474146+0000","last_unstale":"2024-04-16T16:22:48.636319+0000","last_undegraded":"2024-04-16T16:22:48.636319+0000","last_fullsized":"2024-04-16T16:22:48.636319+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:23:01.547579+0000","last_change":"2024-04-16T16:23:01.547621+0000","last_active":"2024-04-16T16:23:01.547579+0000","last_peered":"2024-04-16T16:23:01.547579+0000","last_clean":"2024-04-16T16:23:01.547579+0000","last_became_active":"2024-04-16T16:22:45.462875+0000","last_became_peered":"2024-04-16T16:22:45.462875+0000","last_unstale":"2024-04-16T16:23:01.547579+0000","last_undegraded":"2024-04-16T16:23:01.547579+0000","last_fullsized":"2024-04-16T16:23:01.547579+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:01.547493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:01.547493+0000","last_clean_scrub_stamp":"2024-04-16T16:23:01.547493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:08:51.761612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077110+0000","last_change":"2024-04-16T16:22:46.473128+0000","last_active":"2024-04-16T16:22:48.077110+0000","last_peered":"2024-04-16T16:22:48.077110+0000","last_clean":"2024-04-16T16:22:48.077110+0000","last_became_active":"2024-04-16T16:22:46.472859+0000","last_became_peered":"2024-04-16T16:22:46.472859+0000","last_unstale":"2024-04-16T16:22:48.077110+0000","last_undegraded":"2024-04-16T16:22:48.077110+0000","last_fullsized":"2024-04-16T16:22:48.077110+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.077052+0000","last_change":"2024-04-16T16:22:45.466908+0000","last_active":"2024-04-16T16:22:48.077052+0000","last_peered":"2024-04-16T16:22:48.077052+0000","last_clean":"2024-04-16T16:22:48.077052+0000","last_became_active":"2024-04-16T16:22:45.466535+0000","last_became_peered":"2024-04-16T16:22:45.466535+0000","last_unstale":"2024-04-16T16:22:48.077052+0000","last_undegraded":"2024-04-16T16:22:48.077052+0000","last_fullsized":"2024-04-16T16:22:48.077052+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623664+0000","last_change":"2024-04-16T16:22:46.483216+0000","last_active":"2024-04-16T16:22:48.623664+0000","last_peered":"2024-04-16T16:22:48.623664+0000","last_clean":"2024-04-16T16:22:48.623664+0000","last_became_active":"2024-04-16T16:22:46.482943+0000","last_became_peered":"2024-04-16T16:22:46.482943+0000","last_unstale":"2024-04-16T16:22:48.623664+0000","last_undegraded":"2024-04-16T16:22:48.623664+0000","last_fullsized":"2024-04-16T16:22:48.623664+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:11:26.573221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"155'8","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460621+0000","last_change":"2024-04-16T16:22:45.461068+0000","last_active":"2024-04-16T16:22:47.460621+0000","last_peered":"2024-04-16T16:22:47.460621+0000","last_clean":"2024-04-16T16:22:47.460621+0000","last_became_active":"2024-04-16T16:22:45.459760+0000","last_became_peered":"2024-04-16T16:22:45.459760+0000","last_unstale":"2024-04-16T16:22:47.460621+0000","last_undegraded":"2024-04-16T16:22:47.460621+0000","last_fullsized":"2024-04-16T16:22:47.460621+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.078400+0000","last_change":"2024-04-16T16:22:46.473695+0000","last_active":"2024-04-16T16:22:48.078400+0000","last_peered":"2024-04-16T16:22:48.078400+0000","last_clean":"2024-04-16T16:22:48.078400+0000","last_became_active":"2024-04-16T16:22:46.471894+0000","last_became_peered":"2024-04-16T16:22:46.471894+0000","last_unstale":"2024-04-16T16:22:48.078400+0000","last_undegraded":"2024-04-16T16:22:48.078400+0000","last_fullsized":"2024-04-16T16:22:48.078400+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.625205+0000","last_change":"2024-04-16T16:22:45.469335+0000","last_active":"2024-04-16T16:22:48.625205+0000","last_peered":"2024-04-16T16:22:48.625205+0000","last_clean":"2024-04-16T16:22:48.625205+0000","last_became_active":"2024-04-16T16:22:45.468966+0000","last_became_peered":"2024-04-16T16:22:45.468966+0000","last_unstale":"2024-04-16T16:22:48.625205+0000","last_undegraded":"2024-04-16T16:22:48.625205+0000","last_fullsized":"2024-04-16T16:22:48.625205+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:33:09.300484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457579+0000","last_change":"2024-04-16T16:22:46.471925+0000","last_active":"2024-04-16T16:22:47.457579+0000","last_peered":"2024-04-16T16:22:47.457579+0000","last_clean":"2024-04-16T16:22:47.457579+0000","last_became_active":"2024-04-16T16:22:46.470405+0000","last_became_peered":"2024-04-16T16:22:46.470405+0000","last_unstale":"2024-04-16T16:22:47.457579+0000","last_undegraded":"2024-04-16T16:22:47.457579+0000","last_fullsized":"2024-04-16T16:22:47.457579+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624612+0000","last_change":"2024-04-16T16:22:45.470272+0000","last_active":"2024-04-16T16:22:48.624612+0000","last_peered":"2024-04-16T16:22:48.624612+0000","last_clean":"2024-04-16T16:22:48.624612+0000","last_became_active":"2024-04-16T16:22:45.469959+0000","last_became_peered":"2024-04-16T16:22:45.469959+0000","last_unstale":"2024-04-16T16:22:48.624612+0000","last_undegraded":"2024-04-16T16:22:48.624612+0000","last_fullsized":"2024-04-16T16:22:48.624612+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:34:03.813486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.078211+0000","last_change":"2024-04-16T16:22:46.462951+0000","last_active":"2024-04-16T16:22:48.078211+0000","last_peered":"2024-04-16T16:22:48.078211+0000","last_clean":"2024-04-16T16:22:48.078211+0000","last_became_active":"2024-04-16T16:22:46.462662+0000","last_became_peered":"2024-04-16T16:22:46.462662+0000","last_unstale":"2024-04-16T16:22:48.078211+0000","last_undegraded":"2024-04-16T16:22:48.078211+0000","last_fullsized":"2024-04-16T16:22:48.078211+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462044+0000","last_change":"2024-04-16T16:22:45.462862+0000","last_active":"2024-04-16T16:22:47.462044+0000","last_peered":"2024-04-16T16:22:47.462044+0000","last_clean":"2024-04-16T16:22:47.462044+0000","last_became_active":"2024-04-16T16:22:45.462197+0000","last_became_peered":"2024-04-16T16:22:45.462197+0000","last_unstale":"2024-04-16T16:22:47.462044+0000","last_undegraded":"2024-04-16T16:22:47.462044+0000","last_fullsized":"2024-04-16T16:22:47.462044+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:55.697931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.636154+0000","last_change":"2024-04-16T16:22:46.475567+0000","last_active":"2024-04-16T16:22:48.636154+0000","last_peered":"2024-04-16T16:22:48.636154+0000","last_clean":"2024-04-16T16:22:48.636154+0000","last_became_active":"2024-04-16T16:22:46.475232+0000","last_became_peered":"2024-04-16T16:22:46.475232+0000","last_unstale":"2024-04-16T16:22:48.636154+0000","last_undegraded":"2024-04-16T16:22:48.636154+0000","last_fullsized":"2024-04-16T16:22:48.636154+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462377+0000","last_change":"2024-04-16T16:22:45.466893+0000","last_active":"2024-04-16T16:22:47.462377+0000","last_peered":"2024-04-16T16:22:47.462377+0000","last_clean":"2024-04-16T16:22:47.462377+0000","last_became_active":"2024-04-16T16:22:45.466614+0000","last_became_peered":"2024-04-16T16:22:45.466614+0000","last_unstale":"2024-04-16T16:22:47.462377+0000","last_undegraded":"2024-04-16T16:22:47.462377+0000","last_fullsized":"2024-04-16T16:22:47.462377+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:53:47.150015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623905+0000","last_change":"2024-04-16T16:22:46.482036+0000","last_active":"2024-04-16T16:22:48.623905+0000","last_peered":"2024-04-16T16:22:48.623905+0000","last_clean":"2024-04-16T16:22:48.623905+0000","last_became_active":"2024-04-16T16:22:46.481293+0000","last_became_peered":"2024-04-16T16:22:46.481293+0000","last_unstale":"2024-04-16T16:22:48.623905+0000","last_undegraded":"2024-04-16T16:22:48.623905+0000","last_fullsized":"2024-04-16T16:22:48.623905+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:55.094277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"155'2","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.078354+0000","last_change":"2024-04-16T16:22:45.466947+0000","last_active":"2024-04-16T16:22:48.078354+0000","last_peered":"2024-04-16T16:22:48.078354+0000","last_clean":"2024-04-16T16:22:48.078354+0000","last_became_active":"2024-04-16T16:22:45.466624+0000","last_became_peered":"2024-04-16T16:22:45.466624+0000","last_unstale":"2024-04-16T16:22:48.078354+0000","last_undegraded":"2024-04-16T16:22:48.078354+0000","last_fullsized":"2024-04-16T16:22:48.078354+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.636178+0000","last_change":"2024-04-16T16:22:46.469862+0000","last_active":"2024-04-16T16:22:48.636178+0000","last_peered":"2024-04-16T16:22:48.636178+0000","last_clean":"2024-04-16T16:22:48.636178+0000","last_became_active":"2024-04-16T16:22:46.466179+0000","last_became_peered":"2024-04-16T16:22:46.466179+0000","last_unstale":"2024-04-16T16:22:48.636178+0000","last_undegraded":"2024-04-16T16:22:48.636178+0000","last_fullsized":"2024-04-16T16:22:48.636178+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460683+0000","last_change":"2024-04-16T16:22:45.461564+0000","last_active":"2024-04-16T16:22:47.460683+0000","last_peered":"2024-04-16T16:22:47.460683+0000","last_clean":"2024-04-16T16:22:47.460683+0000","last_became_active":"2024-04-16T16:22:45.461253+0000","last_became_peered":"2024-04-16T16:22:45.461253+0000","last_unstale":"2024-04-16T16:22:47.460683+0000","last_undegraded":"2024-04-16T16:22:47.460683+0000","last_fullsized":"2024-04-16T16:22:47.460683+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:46.018568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462392+0000","last_change":"2024-04-16T16:22:46.465947+0000","last_active":"2024-04-16T16:22:47.462392+0000","last_peered":"2024-04-16T16:22:47.462392+0000","last_clean":"2024-04-16T16:22:47.462392+0000","last_became_active":"2024-04-16T16:22:46.465294+0000","last_became_peered":"2024-04-16T16:22:46.465294+0000","last_unstale":"2024-04-16T16:22:47.462392+0000","last_undegraded":"2024-04-16T16:22:47.462392+0000","last_fullsized":"2024-04-16T16:22:47.462392+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:29.789672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.078278+0000","last_change":"2024-04-16T16:22:45.468883+0000","last_active":"2024-04-16T16:22:48.078278+0000","last_peered":"2024-04-16T16:22:48.078278+0000","last_clean":"2024-04-16T16:22:48.078278+0000","last_became_active":"2024-04-16T16:22:45.468091+0000","last_became_peered":"2024-04-16T16:22:45.468091+0000","last_unstale":"2024-04-16T16:22:48.078278+0000","last_undegraded":"2024-04-16T16:22:48.078278+0000","last_fullsized":"2024-04-16T16:22:48.078278+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710642+0000","last_change":"2024-04-16T16:22:46.471354+0000","last_active":"2024-04-16T16:22:48.710642+0000","last_peered":"2024-04-16T16:22:48.710642+0000","last_clean":"2024-04-16T16:22:48.710642+0000","last_became_active":"2024-04-16T16:22:46.470521+0000","last_became_peered":"2024-04-16T16:22:46.470521+0000","last_unstale":"2024-04-16T16:22:48.710642+0000","last_undegraded":"2024-04-16T16:22:48.710642+0000","last_fullsized":"2024-04-16T16:22:48.710642+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.461147+0000","last_change":"2024-04-16T16:22:45.460865+0000","last_active":"2024-04-16T16:22:47.461147+0000","last_peered":"2024-04-16T16:22:47.461147+0000","last_clean":"2024-04-16T16:22:47.461147+0000","last_became_active":"2024-04-16T16:22:45.458861+0000","last_became_peered":"2024-04-16T16:22:45.458861+0000","last_unstale":"2024-04-16T16:22:47.461147+0000","last_undegraded":"2024-04-16T16:22:47.461147+0000","last_fullsized":"2024-04-16T16:22:47.461147+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:30.462179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457580+0000","last_change":"2024-04-16T16:22:46.464764+0000","last_active":"2024-04-16T16:22:47.457580+0000","last_peered":"2024-04-16T16:22:47.457580+0000","last_clean":"2024-04-16T16:22:47.457580+0000","last_became_active":"2024-04-16T16:22:46.464064+0000","last_became_peered":"2024-04-16T16:22:46.464064+0000","last_unstale":"2024-04-16T16:22:47.457580+0000","last_undegraded":"2024-04-16T16:22:47.457580+0000","last_fullsized":"2024-04-16T16:22:47.457580+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"155'2","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457559+0000","last_change":"2024-04-16T16:22:45.466044+0000","last_active":"2024-04-16T16:22:47.457559+0000","last_peered":"2024-04-16T16:22:47.457559+0000","last_clean":"2024-04-16T16:22:47.457559+0000","last_became_active":"2024-04-16T16:22:45.465676+0000","last_became_peered":"2024-04-16T16:22:45.465676+0000","last_unstale":"2024-04-16T16:22:47.457559+0000","last_undegraded":"2024-04-16T16:22:47.457559+0000","last_fullsized":"2024-04-16T16:22:47.457559+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.636027+0000","last_change":"2024-04-16T16:22:46.466356+0000","last_active":"2024-04-16T16:22:48.636027+0000","last_peered":"2024-04-16T16:22:48.636027+0000","last_clean":"2024-04-16T16:22:48.636027+0000","last_became_active":"2024-04-16T16:22:46.465449+0000","last_became_peered":"2024-04-16T16:22:46.465449+0000","last_unstale":"2024-04-16T16:22:48.636027+0000","last_undegraded":"2024-04-16T16:22:48.636027+0000","last_fullsized":"2024-04-16T16:22:48.636027+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.624728+0000","last_change":"2024-04-16T16:22:45.466156+0000","last_active":"2024-04-16T16:22:48.624728+0000","last_peered":"2024-04-16T16:22:48.624728+0000","last_clean":"2024-04-16T16:22:48.624728+0000","last_became_active":"2024-04-16T16:22:45.465791+0000","last_became_peered":"2024-04-16T16:22:45.465791+0000","last_unstale":"2024-04-16T16:22:48.624728+0000","last_undegraded":"2024-04-16T16:22:48.624728+0000","last_fullsized":"2024-04-16T16:22:48.624728+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:42:46.394979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.710355+0000","last_change":"2024-04-16T16:22:46.470854+0000","last_active":"2024-04-16T16:22:48.710355+0000","last_peered":"2024-04-16T16:22:48.710355+0000","last_clean":"2024-04-16T16:22:48.710355+0000","last_became_active":"2024-04-16T16:22:46.470196+0000","last_became_peered":"2024-04-16T16:22:46.470196+0000","last_unstale":"2024-04-16T16:22:48.710355+0000","last_undegraded":"2024-04-16T16:22:48.710355+0000","last_fullsized":"2024-04-16T16:22:48.710355+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426634+0000","last_change":"2024-04-16T16:22:45.471205+0000","last_active":"2024-04-16T16:22:48.426634+0000","last_peered":"2024-04-16T16:22:48.426634+0000","last_clean":"2024-04-16T16:22:48.426634+0000","last_became_active":"2024-04-16T16:22:45.470911+0000","last_became_peered":"2024-04-16T16:22:45.470911+0000","last_unstale":"2024-04-16T16:22:48.426634+0000","last_undegraded":"2024-04-16T16:22:48.426634+0000","last_fullsized":"2024-04-16T16:22:48.426634+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:03:53.044305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457304+0000","last_change":"2024-04-16T16:22:46.469324+0000","last_active":"2024-04-16T16:22:47.457304+0000","last_peered":"2024-04-16T16:22:47.457304+0000","last_clean":"2024-04-16T16:22:47.457304+0000","last_became_active":"2024-04-16T16:22:46.469009+0000","last_became_peered":"2024-04-16T16:22:46.469009+0000","last_unstale":"2024-04-16T16:22:47.457304+0000","last_undegraded":"2024-04-16T16:22:47.457304+0000","last_fullsized":"2024-04-16T16:22:47.457304+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457171+0000","last_change":"2024-04-16T16:22:45.468189+0000","last_active":"2024-04-16T16:22:47.457171+0000","last_peered":"2024-04-16T16:22:47.457171+0000","last_clean":"2024-04-16T16:22:47.457171+0000","last_became_active":"2024-04-16T16:22:45.467709+0000","last_became_peered":"2024-04-16T16:22:45.467709+0000","last_unstale":"2024-04-16T16:22:47.457171+0000","last_undegraded":"2024-04-16T16:22:47.457171+0000","last_fullsized":"2024-04-16T16:22:47.457171+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.456835+0000","last_change":"2024-04-16T16:22:46.465994+0000","last_active":"2024-04-16T16:22:47.456835+0000","last_peered":"2024-04-16T16:22:47.456835+0000","last_clean":"2024-04-16T16:22:47.456835+0000","last_became_active":"2024-04-16T16:22:46.465007+0000","last_became_peered":"2024-04-16T16:22:46.465007+0000","last_unstale":"2024-04-16T16:22:47.456835+0000","last_undegraded":"2024-04-16T16:22:47.456835+0000","last_fullsized":"2024-04-16T16:22:47.456835+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463114+0000","last_change":"2024-04-16T16:22:45.470607+0000","last_active":"2024-04-16T16:22:47.463114+0000","last_peered":"2024-04-16T16:22:47.463114+0000","last_clean":"2024-04-16T16:22:47.463114+0000","last_became_active":"2024-04-16T16:22:45.469322+0000","last_became_peered":"2024-04-16T16:22:45.469322+0000","last_unstale":"2024-04-16T16:22:47.463114+0000","last_undegraded":"2024-04-16T16:22:47.463114+0000","last_fullsized":"2024-04-16T16:22:47.463114+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463188+0000","last_change":"2024-04-16T16:22:46.462483+0000","last_active":"2024-04-16T16:22:47.463188+0000","last_peered":"2024-04-16T16:22:47.463188+0000","last_clean":"2024-04-16T16:22:47.463188+0000","last_became_active":"2024-04-16T16:22:46.461432+0000","last_became_peered":"2024-04-16T16:22:46.461432+0000","last_unstale":"2024-04-16T16:22:47.463188+0000","last_undegraded":"2024-04-16T16:22:47.463188+0000","last_fullsized":"2024-04-16T16:22:47.463188+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:23:00.114438+0000","last_change":"2024-04-16T16:23:00.114469+0000","last_active":"2024-04-16T16:23:00.114438+0000","last_peered":"2024-04-16T16:23:00.114438+0000","last_clean":"2024-04-16T16:23:00.114438+0000","last_became_active":"2024-04-16T16:22:45.462154+0000","last_became_peered":"2024-04-16T16:22:45.462154+0000","last_unstale":"2024-04-16T16:23:00.114438+0000","last_undegraded":"2024-04-16T16:23:00.114438+0000","last_fullsized":"2024-04-16T16:23:00.114438+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:00.114332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:00.114332+0000","last_clean_scrub_stamp":"2024-04-16T16:23:00.114332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:01.971568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.635503+0000","last_change":"2024-04-16T16:22:46.467575+0000","last_active":"2024-04-16T16:22:48.635503+0000","last_peered":"2024-04-16T16:22:48.635503+0000","last_clean":"2024-04-16T16:22:48.635503+0000","last_became_active":"2024-04-16T16:22:46.466890+0000","last_became_peered":"2024-04-16T16:22:46.466890+0000","last_unstale":"2024-04-16T16:22:48.635503+0000","last_undegraded":"2024-04-16T16:22:48.635503+0000","last_fullsized":"2024-04-16T16:22:48.635503+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.426288+0000","last_change":"2024-04-16T16:22:45.469689+0000","last_active":"2024-04-16T16:22:48.426288+0000","last_peered":"2024-04-16T16:22:48.426288+0000","last_clean":"2024-04-16T16:22:48.426288+0000","last_became_active":"2024-04-16T16:22:45.469350+0000","last_became_peered":"2024-04-16T16:22:45.469350+0000","last_unstale":"2024-04-16T16:22:48.426288+0000","last_undegraded":"2024-04-16T16:22:48.426288+0000","last_fullsized":"2024-04-16T16:22:48.426288+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:35.761705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:48.623691+0000","last_change":"2024-04-16T16:22:46.478730+0000","last_active":"2024-04-16T16:22:48.623691+0000","last_peered":"2024-04-16T16:22:48.623691+0000","last_clean":"2024-04-16T16:22:48.623691+0000","last_became_active":"2024-04-16T16:22:46.478459+0000","last_became_peered":"2024-04-16T16:22:46.478459+0000","last_unstale":"2024-04-16T16:22:48.623691+0000","last_undegraded":"2024-04-16T16:22:48.623691+0000","last_fullsized":"2024-04-16T16:22:48.623691+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:36.777723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.460918+0000","last_change":"2024-04-16T16:22:45.467505+0000","last_active":"2024-04-16T16:22:47.460918+0000","last_peered":"2024-04-16T16:22:47.460918+0000","last_clean":"2024-04-16T16:22:47.460918+0000","last_became_active":"2024-04-16T16:22:45.465339+0000","last_became_peered":"2024-04-16T16:22:45.465339+0000","last_unstale":"2024-04-16T16:22:47.460918+0000","last_undegraded":"2024-04-16T16:22:47.460918+0000","last_fullsized":"2024-04-16T16:22:47.460918+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.457448+0000","last_change":"2024-04-16T16:22:46.465023+0000","last_active":"2024-04-16T16:22:47.457448+0000","last_peered":"2024-04-16T16:22:47.457448+0000","last_clean":"2024-04-16T16:22:47.457448+0000","last_became_active":"2024-04-16T16:22:46.464436+0000","last_became_peered":"2024-04-16T16:22:46.464436+0000","last_unstale":"2024-04-16T16:22:47.457448+0000","last_undegraded":"2024-04-16T16:22:47.457448+0000","last_fullsized":"2024-04-16T16:22:47.457448+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:45.414914+0000","last_clean_scrub_stamp":"2024-04-16T16:22:45.414914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.463963+0000","last_change":"2024-04-16T16:22:45.461515+0000","last_active":"2024-04-16T16:22:47.463963+0000","last_peered":"2024-04-16T16:22:47.463963+0000","last_clean":"2024-04-16T16:22:47.463963+0000","last_became_active":"2024-04-16T16:22:45.460350+0000","last_became_peered":"2024-04-16T16:22:45.460350+0000","last_unstale":"2024-04-16T16:22:47.463963+0000","last_undegraded":"2024-04-16T16:22:47.463963+0000","last_fullsized":"2024-04-16T16:22:47.463963+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:44.398896+0000","last_clean_scrub_stamp":"2024-04-16T16:22:44.398896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"144'4691","reported_seq":5390,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-16T16:22:47.462677+0000","last_change":"2024-04-16T16:13:20.915111+0000","last_active":"2024-04-16T16:22:47.462677+0000","last_peered":"2024-04-16T16:22:47.462677+0000","last_clean":"2024-04-16T16:22:47.462677+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:22:47.462677+0000","last_undegraded":"2024-04-16T16:22:47.462677+0000","last_fullsized":"2024-04-16T16:22:47.462677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:13:19.812055+0000","last_clean_scrub_stamp":"2024-04-16T16:13:19.812055+0000","objects_scrubbed":0,"log_size":4691,"log_dups_size":0,"ondisk_log_size":4691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:55:21.916647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":387,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":387,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4691,"ondisk_log_size":4691,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574959,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37852,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333988,"statfs":{"total":96636764160,"available":96598003712,"internally_reserved":0,"allocated":2576384,"data_stored":1526220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574960,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1961984,"data_stored":933520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574960,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":934102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574961,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33104,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338736,"statfs":{"total":96636764160,"available":96602865664,"internally_reserved":0,"allocated":1974272,"data_stored":935180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574959,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37264,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334576,"statfs":{"total":96636764160,"available":96598605824,"internally_reserved":0,"allocated":1961984,"data_stored":933532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574959,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37252,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334588,"statfs":{"total":96636764160,"available":96598618112,"internally_reserved":0,"allocated":1961984,"data_stored":933588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37252,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334588,"statfs":{"total":96636764160,"available":96598618112,"internally_reserved":0,"allocated":1961984,"data_stored":933532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574960,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37828,"kb_used_data":2492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334012,"statfs":{"total":96636764160,"available":96598028288,"internally_reserved":0,"allocated":2551808,"data_stored":1523866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:23:04.077 INFO:tasks.ceph:pgid 23.3f last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.077 INFO:tasks.ceph:pgid 23.3e last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.077 INFO:tasks.ceph:pgid 23.3d last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.077 INFO:tasks.ceph:pgid 23.3c last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.077 INFO:tasks.ceph:pgid 23.3b last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.077 INFO:tasks.ceph:pgid 23.3a last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.078 INFO:tasks.ceph:pgid 23.38 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.078 INFO:tasks.ceph:pgid 23.37 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.078 INFO:tasks.ceph:pgid 23.36 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.078 INFO:tasks.ceph:pgid 23.35 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.078 INFO:tasks.ceph:pgid 23.34 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.078 INFO:tasks.ceph:pgid 23.33 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.079 INFO:tasks.ceph:pgid 23.32 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.079 INFO:tasks.ceph:pgid 23.31 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.079 INFO:tasks.ceph:pgid 23.30 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.079 INFO:tasks.ceph:pgid 23.2f last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.079 INFO:tasks.ceph:pgid 23.2e last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.079 INFO:tasks.ceph:pgid 23.2d last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.079 INFO:tasks.ceph:pgid 23.2c last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.080 INFO:tasks.ceph:pgid 23.2b last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.080 INFO:tasks.ceph:pgid 23.2a last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.080 INFO:tasks.ceph:pgid 23.29 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.080 INFO:tasks.ceph:pgid 23.28 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.080 INFO:tasks.ceph:pgid 23.27 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.080 INFO:tasks.ceph:pgid 23.26 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.080 INFO:tasks.ceph:pgid 23.25 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.081 INFO:tasks.ceph:pgid 23.24 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.081 INFO:tasks.ceph:pgid 23.23 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.081 INFO:tasks.ceph:pgid 23.22 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.081 INFO:tasks.ceph:pgid 23.21 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.081 INFO:tasks.ceph:pgid 23.20 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.081 INFO:tasks.ceph:pgid 23.1e last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.082 INFO:tasks.ceph:pgid 22.1f last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.082 INFO:tasks.ceph:pgid 23.1f last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.082 INFO:tasks.ceph:pgid 22.1e last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.082 INFO:tasks.ceph:pgid 23.1c last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.082 INFO:tasks.ceph:pgid 22.1d last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.082 INFO:tasks.ceph:pgid 23.1d last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.082 INFO:tasks.ceph:pgid 22.1b last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.083 INFO:tasks.ceph:pgid 23.1b last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.083 INFO:tasks.ceph:pgid 23.18 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.083 INFO:tasks.ceph:pgid 22.19 last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.083 INFO:tasks.ceph:pgid 23.19 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.083 INFO:tasks.ceph:pgid 23.17 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.083 INFO:tasks.ceph:pgid 23.14 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.084 INFO:tasks.ceph:pgid 23.15 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.084 INFO:tasks.ceph:pgid 22.14 last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.084 INFO:tasks.ceph:pgid 23.12 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.084 INFO:tasks.ceph:pgid 22.13 last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.084 INFO:tasks.ceph:pgid 23.13 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.084 INFO:tasks.ceph:pgid 23.10 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.084 INFO:tasks.ceph:pgid 22.11 last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.085 INFO:tasks.ceph:pgid 23.11 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.085 INFO:tasks.ceph:pgid 22.10 last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.085 INFO:tasks.ceph:pgid 23.e last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.085 INFO:tasks.ceph:pgid 22.f last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.085 INFO:tasks.ceph:pgid 23.f last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.085 INFO:tasks.ceph:pgid 22.e last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.085 INFO:tasks.ceph:pgid 23.c last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.086 INFO:tasks.ceph:pgid 22.d last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.086 INFO:tasks.ceph:pgid 23.d last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.086 INFO:tasks.ceph:pgid 22.c last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.086 INFO:tasks.ceph:pgid 23.a last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.086 INFO:tasks.ceph:pgid 22.b last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.086 INFO:tasks.ceph:pgid 23.b last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.087 INFO:tasks.ceph:pgid 22.a last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.087 INFO:tasks.ceph:pgid 23.8 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.087 INFO:tasks.ceph:pgid 22.9 last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.087 INFO:tasks.ceph:pgid 23.9 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.087 INFO:tasks.ceph:pgid 22.8 last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.087 INFO:tasks.ceph:pgid 23.6 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.088 INFO:tasks.ceph:pgid 22.7 last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.088 INFO:tasks.ceph:pgid 23.7 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.088 INFO:tasks.ceph:pgid 22.6 last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.088 INFO:tasks.ceph:pgid 23.4 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.088 INFO:tasks.ceph:pgid 22.5 last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.088 INFO:tasks.ceph:pgid 23.5 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.088 INFO:tasks.ceph:pgid 22.4 last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.089 INFO:tasks.ceph:pgid 23.2 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.089 INFO:tasks.ceph:pgid 22.3 last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.089 INFO:tasks.ceph:pgid 23.3 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.089 INFO:tasks.ceph:pgid 23.0 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.089 INFO:tasks.ceph:pgid 22.1 last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.089 INFO:tasks.ceph:pgid 23.1 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.089 INFO:tasks.ceph:pgid 22.0 last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.090 INFO:tasks.ceph:pgid 23.16 last_scrub_stamp 2024-04-16T16:22:45.414914+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.090 INFO:tasks.ceph:pgid 22.17 last_scrub_stamp 2024-04-16T16:22:44.398896+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.090 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-16T16:13:19.812055+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=13, tm_sec=19, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=16, tm_min=22, tm_sec=56, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T16:23:04.090 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-16T16:23:05.785 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-16T16:23:05.786+0000 7f3661200640 -1 mds.0.openfiles _load_finish got (2) No such file or directory 2024-04-16T16:23:09.307 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~20s 2024-04-16T16:23:11.381 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-16T16:23:11.381+0000 7f3666200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-16T16:23:15.919 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~26s 2024-04-16T16:23:22.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~33s 2024-04-16T16:23:24.092 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T16:23:24.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.329+0000 7f2e7a200640 1 Processor -- start 2024-04-16T16:23:24.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.329+0000 7f2e7a200640 1 -- start start 2024-04-16T16:23:24.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.330+0000 7f2e7a200640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2e74146220 0x7f2e7415ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:24.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.330+0000 7f2e7a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e741608a0 0x7f2e74160d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:24.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.330+0000 7f2e7a200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e7415d360 con 0x7f2e7406f7a0 2024-04-16T16:23:24.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.331+0000 7f2e7a200640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2e74160410 con 0x7f2e74146220 2024-04-16T16:23:24.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.331+0000 7f2e7a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2e7415d640 con 0x7f2e741608a0 2024-04-16T16:23:24.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.331+0000 7f2e73e00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2e7406f7a0 0x7f2e7406df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60656/0 (socket says 172.21.15.45:60656) 2024-04-16T16:23:24.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.331+0000 7f2e78e00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2e74146220 0x7f2e7415ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:24.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.331+0000 7f2e73e00640 1 -- 172.21.15.45:0/1827652506 learned_addr learned my addr 172.21.15.45:0/1827652506 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:23:24.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.331+0000 7f2e79800640 1 --2- 172.21.15.45:0/1827652506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e741608a0 0x7f2e74160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:24.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.332+0000 7f2e78e00640 1 -- 172.21.15.45:0/1827652506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e741608a0 msgr2=0x7f2e74160d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:24.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.332+0000 7f2e78e00640 1 --2- 172.21.15.45:0/1827652506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e741608a0 0x7f2e74160d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.332+0000 7f2e78e00640 1 -- 172.21.15.45:0/1827652506 >> v1:172.21.15.45:6789/0 conn(0x7f2e7406f7a0 legacy=0x7f2e7406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:24.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.332+0000 7f2e78e00640 1 -- 172.21.15.45:0/1827652506 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7415d920 con 0x7f2e74146220 2024-04-16T16:23:24.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.332+0000 7f2e79800640 1 --2- 172.21.15.45:0/1827652506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e741608a0 0x7f2e74160d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T16:23:24.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.333+0000 7f2e78e00640 1 --2- 172.21.15.45:0/1827652506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e74146220 0x7f2e7415ccb0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f2e680097a0 tx=0x7f2e6802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1debb4b5439b7d4f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:24.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.334+0000 7f2e73400640 1 -- 172.21.15.45:0/1827652506 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e68035020 con 0x7f2e74146220 2024-04-16T16:23:24.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.334+0000 7f2e73400640 1 -- 172.21.15.45:0/1827652506 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e68031810 con 0x7f2e74146220 2024-04-16T16:23:24.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.334+0000 7f2e73400640 1 -- 172.21.15.45:0/1827652506 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e68031be0 con 0x7f2e74146220 2024-04-16T16:23:24.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.334+0000 7f2e7a200640 1 -- 172.21.15.45:0/1827652506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e74146220 msgr2=0x7f2e7415ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:24.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.334+0000 7f2e7a200640 1 --2- 172.21.15.45:0/1827652506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e74146220 0x7f2e7415ccb0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f2e680097a0 tx=0x7f2e6802b8b0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.335+0000 7f2e7a200640 1 -- 172.21.15.45:0/1827652506 shutdown_connections 2024-04-16T16:23:24.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.335+0000 7f2e7a200640 1 --2- 172.21.15.45:0/1827652506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e741608a0 0x7f2e74160d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.335+0000 7f2e7a200640 1 --2- 172.21.15.45:0/1827652506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e74146220 0x7f2e7415ccb0 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.335+0000 7f2e7a200640 1 -- 172.21.15.45:0/1827652506 >> 172.21.15.45:0/1827652506 conn(0x7f2e7406b930 msgr2=0x7f2e7406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:24.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.335+0000 7f2e7a200640 1 -- 172.21.15.45:0/1827652506 shutdown_connections 2024-04-16T16:23:24.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.335+0000 7f2e7a200640 1 -- 172.21.15.45:0/1827652506 wait complete. 2024-04-16T16:23:24.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.337+0000 7f2e7a200640 1 Processor -- start 2024-04-16T16:23:24.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.337+0000 7f2e7a200640 1 -- start start 2024-04-16T16:23:24.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.338+0000 7f2e7a200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e7406f7a0 0x7f2e7415c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:24.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.338+0000 7f2e7a200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e74146220 0x7f2e74139830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:24.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.338+0000 7f2e7a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e741608a0 0x7f2e740b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:24.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.338+0000 7f2e7a200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7406fc20 con 0x7f2e7406f7a0 2024-04-16T16:23:24.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.338+0000 7f2e7a200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7415dc00 con 0x7f2e74146220 2024-04-16T16:23:24.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.338+0000 7f2e7a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2e7406fda0 con 0x7f2e741608a0 2024-04-16T16:23:24.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.338+0000 7f2e78e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e7406f7a0 0x7f2e7415c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:24.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.338+0000 7f2e78e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e7406f7a0 0x7f2e7415c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33224/0 (socket says 172.21.15.45:33224) 2024-04-16T16:23:24.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.338+0000 7f2e78e00640 1 -- 172.21.15.45:0/2388660475 learned_addr learned my addr 172.21.15.45:0/2388660475 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:24.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.338+0000 7f2e78e00640 1 -- 172.21.15.45:0/2388660475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e741608a0 msgr2=0x7f2e740b2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:24.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.338+0000 7f2e78e00640 1 --2- 172.21.15.45:0/2388660475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e741608a0 0x7f2e740b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.338+0000 7f2e78e00640 1 -- 172.21.15.45:0/2388660475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e74146220 msgr2=0x7f2e74139830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:24.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.338+0000 7f2e78e00640 1 --2- 172.21.15.45:0/2388660475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e74146220 0x7f2e74139830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.338+0000 7f2e78e00640 1 -- 172.21.15.45:0/2388660475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e740cd260 con 0x7f2e7406f7a0 2024-04-16T16:23:24.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.339+0000 7f2e78e00640 1 --2- 172.21.15.45:0/2388660475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e7406f7a0 0x7f2e7415c1d0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f2e680098c0 tx=0x7f2e680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:24.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.339+0000 7f2e70c00640 1 -- 172.21.15.45:0/2388660475 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e68041070 con 0x7f2e7406f7a0 2024-04-16T16:23:24.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.339+0000 7f2e70c00640 1 -- 172.21.15.45:0/2388660475 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e6803ad00 con 0x7f2e7406f7a0 2024-04-16T16:23:24.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.339+0000 7f2e70c00640 1 -- 172.21.15.45:0/2388660475 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e68035040 con 0x7f2e7406f7a0 2024-04-16T16:23:24.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.339+0000 7f2e7a200640 1 -- 172.21.15.45:0/2388660475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e740cd540 con 0x7f2e7406f7a0 2024-04-16T16:23:24.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.339+0000 7f2e7a200640 1 -- 172.21.15.45:0/2388660475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e740cdb00 con 0x7f2e7406f7a0 2024-04-16T16:23:24.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.340+0000 7f2e7a200640 1 -- 172.21.15.45:0/2388660475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e74075780 con 0x7f2e7406f7a0 2024-04-16T16:23:24.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.342+0000 7f2e70c00640 1 -- 172.21.15.45:0/2388660475 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f2e68038020 con 0x7f2e7406f7a0 2024-04-16T16:23:24.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.342+0000 7f2e70c00640 1 --2- 172.21.15.45:0/2388660475 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f2e5406ca10 0x7f2e5406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:24.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.343+0000 7f2e70c00640 1 -- 172.21.15.45:0/2388660475 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5795+0+0 (secure 0 0 0) 0x7f2e680ada30 con 0x7f2e7406f7a0 2024-04-16T16:23:24.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.343+0000 7f2e73e00640 1 --2- 172.21.15.45:0/2388660475 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f2e5406ca10 0x7f2e5406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:24.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.344+0000 7f2e73e00640 1 --2- 172.21.15.45:0/2388660475 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f2e5406ca10 0x7f2e5406eed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2e60000f60 tx=0x7f2e60007a80 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:24.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.348+0000 7f2e70c00640 1 -- 172.21.15.45:0/2388660475 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e6807bce0 con 0x7f2e7406f7a0 2024-04-16T16:23:24.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.554+0000 7f2e7a200640 1 -- 172.21.15.45:0/2388660475 --> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2e740cff60 con 0x7f2e5406ca10 2024-04-16T16:23:24.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.573+0000 7f2e70c00640 1 -- 172.21.15.45:0/2388660475 <== mgr.7591 v2:172.21.15.103:6832/1129061436 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252469 (secure 0 0 0) 0x7f2e740cff60 con 0x7f2e5406ca10 2024-04-16T16:23:24.577 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:23:24.579 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-04-16T16:23:24.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.577+0000 7f2e7a200640 1 -- 172.21.15.45:0/2388660475 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f2e5406ca10 msgr2=0x7f2e5406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:24.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.577+0000 7f2e7a200640 1 --2- 172.21.15.45:0/2388660475 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f2e5406ca10 0x7f2e5406eed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2e60000f60 tx=0x7f2e60007a80 comp rx=0 tx=0).stop 2024-04-16T16:23:24.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.577+0000 7f2e7a200640 1 -- 172.21.15.45:0/2388660475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e7406f7a0 msgr2=0x7f2e7415c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:24.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.577+0000 7f2e7a200640 1 --2- 172.21.15.45:0/2388660475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e7406f7a0 0x7f2e7415c1d0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f2e680098c0 tx=0x7f2e680046c0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.577+0000 7f2e7a200640 1 -- 172.21.15.45:0/2388660475 shutdown_connections 2024-04-16T16:23:24.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.577+0000 7f2e7a200640 1 --2- 172.21.15.45:0/2388660475 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f2e5406ca10 0x7f2e5406eed0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.577+0000 7f2e7a200640 1 --2- 172.21.15.45:0/2388660475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e741608a0 0x7f2e740b2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.577+0000 7f2e7a200640 1 --2- 172.21.15.45:0/2388660475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e74146220 0x7f2e74139830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.577+0000 7f2e7a200640 1 --2- 172.21.15.45:0/2388660475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e7406f7a0 0x7f2e7415c1d0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.578+0000 7f2e7a200640 1 -- 172.21.15.45:0/2388660475 >> 172.21.15.45:0/2388660475 conn(0x7f2e7406b930 msgr2=0x7f2e7415dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:24.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.578+0000 7f2e7a200640 1 -- 172.21.15.45:0/2388660475 shutdown_connections 2024-04-16T16:23:24.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.578+0000 7f2e7a200640 1 -- 172.21.15.45:0/2388660475 wait complete. 2024-04-16T16:23:24.599 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-04-16T16:23:23.554868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1352,"num_read_kb":2605,"num_write":432,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4738,"ondisk_log_size":4738,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":286824,"kb_used_data":16616,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754687896,"statfs":{"total":773094113280,"available":772800405504,"internally_reserved":0,"allocated":17014784,"data_stored":8702188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":276614704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":64,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004555"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:12.221655+0000","last_change":"2024-04-16T16:23:12.221758+0000","last_active":"2024-04-16T16:23:12.221655+0000","last_peered":"2024-04-16T16:23:12.221655+0000","last_clean":"2024-04-16T16:23:12.221655+0000","last_became_active":"2024-04-16T16:22:46.485690+0000","last_became_peered":"2024-04-16T16:22:46.485690+0000","last_unstale":"2024-04-16T16:23:12.221655+0000","last_undegraded":"2024-04-16T16:23:12.221655+0000","last_fullsized":"2024-04-16T16:23:12.221655+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:12.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:12.221427+0000","last_clean_scrub_stamp":"2024-04-16T16:23:12.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:48.303691+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:10.092514+0000","last_change":"2024-04-16T16:23:10.092608+0000","last_active":"2024-04-16T16:23:10.092514+0000","last_peered":"2024-04-16T16:23:10.092514+0000","last_clean":"2024-04-16T16:23:10.092514+0000","last_became_active":"2024-04-16T16:22:46.477410+0000","last_became_peered":"2024-04-16T16:22:46.477410+0000","last_unstale":"2024-04-16T16:23:10.092514+0000","last_undegraded":"2024-04-16T16:23:10.092514+0000","last_fullsized":"2024-04-16T16:23:10.092514+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:10.092191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:10.092191+0000","last_clean_scrub_stamp":"2024-04-16T16:23:10.092191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:54:51.520895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:09.102853+0000","last_change":"2024-04-16T16:23:09.102910+0000","last_active":"2024-04-16T16:23:09.102853+0000","last_peered":"2024-04-16T16:23:09.102853+0000","last_clean":"2024-04-16T16:23:09.102853+0000","last_became_active":"2024-04-16T16:22:46.480405+0000","last_became_peered":"2024-04-16T16:22:46.480405+0000","last_unstale":"2024-04-16T16:23:09.102853+0000","last_undegraded":"2024-04-16T16:23:09.102853+0000","last_fullsized":"2024-04-16T16:23:09.102853+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:09.102644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:09.102644+0000","last_clean_scrub_stamp":"2024-04-16T16:23:09.102644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:46.528735+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:12.244122+0000","last_change":"2024-04-16T16:23:12.244180+0000","last_active":"2024-04-16T16:23:12.244122+0000","last_peered":"2024-04-16T16:23:12.244122+0000","last_clean":"2024-04-16T16:23:12.244122+0000","last_became_active":"2024-04-16T16:22:46.477516+0000","last_became_peered":"2024-04-16T16:22:46.477516+0000","last_unstale":"2024-04-16T16:23:12.244122+0000","last_undegraded":"2024-04-16T16:23:12.244122+0000","last_fullsized":"2024-04-16T16:23:12.244122+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:12.243903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:12.243903+0000","last_clean_scrub_stamp":"2024-04-16T16:23:12.243903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:09:02.458022+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:13.131536+0000","last_change":"2024-04-16T16:23:13.131630+0000","last_active":"2024-04-16T16:23:13.131536+0000","last_peered":"2024-04-16T16:23:13.131536+0000","last_clean":"2024-04-16T16:23:13.131536+0000","last_became_active":"2024-04-16T16:22:46.481033+0000","last_became_peered":"2024-04-16T16:22:46.481033+0000","last_unstale":"2024-04-16T16:23:13.131536+0000","last_undegraded":"2024-04-16T16:23:13.131536+0000","last_fullsized":"2024-04-16T16:23:13.131536+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:13.131316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:13.131316+0000","last_clean_scrub_stamp":"2024-04-16T16:23:13.131316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:29:06.202281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.789565+0000","last_change":"2024-04-16T16:23:05.493541+0000","last_active":"2024-04-16T16:23:05.789565+0000","last_peered":"2024-04-16T16:23:05.789565+0000","last_clean":"2024-04-16T16:23:05.789565+0000","last_became_active":"2024-04-16T16:22:46.462017+0000","last_became_peered":"2024-04-16T16:22:46.462017+0000","last_unstale":"2024-04-16T16:23:05.789565+0000","last_undegraded":"2024-04-16T16:23:05.789565+0000","last_fullsized":"2024-04-16T16:23:05.789565+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:05.493205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:05.493205+0000","last_clean_scrub_stamp":"2024-04-16T16:23:05.493205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:43:25.228781+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.802576+0000","last_change":"2024-04-16T16:23:00.671670+0000","last_active":"2024-04-16T16:23:05.802576+0000","last_peered":"2024-04-16T16:23:05.802576+0000","last_clean":"2024-04-16T16:23:05.802576+0000","last_became_active":"2024-04-16T16:22:46.461856+0000","last_became_peered":"2024-04-16T16:22:46.461856+0000","last_unstale":"2024-04-16T16:23:05.802576+0000","last_undegraded":"2024-04-16T16:23:05.802576+0000","last_fullsized":"2024-04-16T16:23:05.802576+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:00.671426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:00.671426+0000","last_clean_scrub_stamp":"2024-04-16T16:23:00.671426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:35:01.857190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:14.082734+0000","last_change":"2024-04-16T16:23:14.082787+0000","last_active":"2024-04-16T16:23:14.082734+0000","last_peered":"2024-04-16T16:23:14.082734+0000","last_clean":"2024-04-16T16:23:14.082734+0000","last_became_active":"2024-04-16T16:22:46.485280+0000","last_became_peered":"2024-04-16T16:22:46.485280+0000","last_unstale":"2024-04-16T16:23:14.082734+0000","last_undegraded":"2024-04-16T16:23:14.082734+0000","last_fullsized":"2024-04-16T16:23:14.082734+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:14.082532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:14.082532+0000","last_clean_scrub_stamp":"2024-04-16T16:23:14.082532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:42:36.284986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:08.026665+0000","last_change":"2024-04-16T16:23:08.026759+0000","last_active":"2024-04-16T16:23:08.026665+0000","last_peered":"2024-04-16T16:23:08.026665+0000","last_clean":"2024-04-16T16:23:08.026665+0000","last_became_active":"2024-04-16T16:22:46.476125+0000","last_became_peered":"2024-04-16T16:22:46.476125+0000","last_unstale":"2024-04-16T16:23:08.026665+0000","last_undegraded":"2024-04-16T16:23:08.026665+0000","last_fullsized":"2024-04-16T16:23:08.026665+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:08.026373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:08.026373+0000","last_clean_scrub_stamp":"2024-04-16T16:23:08.026373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:50:52.330365+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:12.170901+0000","last_change":"2024-04-16T16:23:12.170957+0000","last_active":"2024-04-16T16:23:12.170901+0000","last_peered":"2024-04-16T16:23:12.170901+0000","last_clean":"2024-04-16T16:23:12.170901+0000","last_became_active":"2024-04-16T16:22:46.482975+0000","last_became_peered":"2024-04-16T16:22:46.482975+0000","last_unstale":"2024-04-16T16:23:12.170901+0000","last_undegraded":"2024-04-16T16:23:12.170901+0000","last_fullsized":"2024-04-16T16:23:12.170901+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:12.170710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:12.170710+0000","last_clean_scrub_stamp":"2024-04-16T16:23:12.170710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:23.414556+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:11.121890+0000","last_change":"2024-04-16T16:23:11.121941+0000","last_active":"2024-04-16T16:23:11.121890+0000","last_peered":"2024-04-16T16:23:11.121890+0000","last_clean":"2024-04-16T16:23:11.121890+0000","last_became_active":"2024-04-16T16:22:46.481000+0000","last_became_peered":"2024-04-16T16:22:46.481000+0000","last_unstale":"2024-04-16T16:23:11.121890+0000","last_undegraded":"2024-04-16T16:23:11.121890+0000","last_fullsized":"2024-04-16T16:23:11.121890+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:11.121698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:11.121698+0000","last_clean_scrub_stamp":"2024-04-16T16:23:11.121698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:28:33.374608+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:06.996299+0000","last_change":"2024-04-16T16:23:06.996358+0000","last_active":"2024-04-16T16:23:06.996299+0000","last_peered":"2024-04-16T16:23:06.996299+0000","last_clean":"2024-04-16T16:23:06.996299+0000","last_became_active":"2024-04-16T16:22:46.475898+0000","last_became_peered":"2024-04-16T16:22:46.475898+0000","last_unstale":"2024-04-16T16:23:06.996299+0000","last_undegraded":"2024-04-16T16:23:06.996299+0000","last_fullsized":"2024-04-16T16:23:06.996299+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:06.996058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:06.996058+0000","last_clean_scrub_stamp":"2024-04-16T16:23:06.996058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:00:09.448861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:15.089749+0000","last_change":"2024-04-16T16:23:15.089811+0000","last_active":"2024-04-16T16:23:15.089749+0000","last_peered":"2024-04-16T16:23:15.089749+0000","last_clean":"2024-04-16T16:23:15.089749+0000","last_became_active":"2024-04-16T16:22:46.481776+0000","last_became_peered":"2024-04-16T16:22:46.481776+0000","last_unstale":"2024-04-16T16:23:15.089749+0000","last_undegraded":"2024-04-16T16:23:15.089749+0000","last_fullsized":"2024-04-16T16:23:15.089749+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:15.089553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:15.089553+0000","last_clean_scrub_stamp":"2024-04-16T16:23:15.089553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:28:37.342462+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:06.072039+0000","last_change":"2024-04-16T16:23:06.072145+0000","last_active":"2024-04-16T16:23:06.072039+0000","last_peered":"2024-04-16T16:23:06.072039+0000","last_clean":"2024-04-16T16:23:06.072039+0000","last_became_active":"2024-04-16T16:22:46.476486+0000","last_became_peered":"2024-04-16T16:22:46.476486+0000","last_unstale":"2024-04-16T16:23:06.072039+0000","last_undegraded":"2024-04-16T16:23:06.072039+0000","last_fullsized":"2024-04-16T16:23:06.072039+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:06.071813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:06.071813+0000","last_clean_scrub_stamp":"2024-04-16T16:23:06.071813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:07.929049+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:11.178733+0000","last_change":"2024-04-16T16:23:11.178786+0000","last_active":"2024-04-16T16:23:11.178733+0000","last_peered":"2024-04-16T16:23:11.178733+0000","last_clean":"2024-04-16T16:23:11.178733+0000","last_became_active":"2024-04-16T16:22:46.463117+0000","last_became_peered":"2024-04-16T16:22:46.463117+0000","last_unstale":"2024-04-16T16:23:11.178733+0000","last_undegraded":"2024-04-16T16:23:11.178733+0000","last_fullsized":"2024-04-16T16:23:11.178733+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:11.178529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:11.178529+0000","last_clean_scrub_stamp":"2024-04-16T16:23:11.178529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:06:15.115392+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.802139+0000","last_change":"2024-04-16T16:23:05.732607+0000","last_active":"2024-04-16T16:23:05.802139+0000","last_peered":"2024-04-16T16:23:05.802139+0000","last_clean":"2024-04-16T16:23:05.802139+0000","last_became_active":"2024-04-16T16:22:46.467274+0000","last_became_peered":"2024-04-16T16:22:46.467274+0000","last_unstale":"2024-04-16T16:23:05.802139+0000","last_undegraded":"2024-04-16T16:23:05.802139+0000","last_fullsized":"2024-04-16T16:23:05.802139+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:05.732323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:05.732323+0000","last_clean_scrub_stamp":"2024-04-16T16:23:05.732323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:28.571381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:08.444623+0000","last_change":"2024-04-16T16:23:08.444677+0000","last_active":"2024-04-16T16:23:08.444623+0000","last_peered":"2024-04-16T16:23:08.444623+0000","last_clean":"2024-04-16T16:23:08.444623+0000","last_became_active":"2024-04-16T16:22:46.474757+0000","last_became_peered":"2024-04-16T16:22:46.474757+0000","last_unstale":"2024-04-16T16:23:08.444623+0000","last_undegraded":"2024-04-16T16:23:08.444623+0000","last_fullsized":"2024-04-16T16:23:08.444623+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:08.444404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:08.444404+0000","last_clean_scrub_stamp":"2024-04-16T16:23:08.444404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:32:48.694321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.778058+0000","last_change":"2024-04-16T16:23:04.509643+0000","last_active":"2024-04-16T16:23:05.778058+0000","last_peered":"2024-04-16T16:23:05.778058+0000","last_clean":"2024-04-16T16:23:05.778058+0000","last_became_active":"2024-04-16T16:22:46.462857+0000","last_became_peered":"2024-04-16T16:22:46.462857+0000","last_unstale":"2024-04-16T16:23:05.778058+0000","last_undegraded":"2024-04-16T16:23:05.778058+0000","last_fullsized":"2024-04-16T16:23:05.778058+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:04.509347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:04.509347+0000","last_clean_scrub_stamp":"2024-04-16T16:23:04.509347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:04:27.672904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.790019+0000","last_change":"2024-04-16T16:23:03.451964+0000","last_active":"2024-04-16T16:23:05.790019+0000","last_peered":"2024-04-16T16:23:05.790019+0000","last_clean":"2024-04-16T16:23:05.790019+0000","last_became_active":"2024-04-16T16:22:46.456756+0000","last_became_peered":"2024-04-16T16:22:46.456756+0000","last_unstale":"2024-04-16T16:23:05.790019+0000","last_undegraded":"2024-04-16T16:23:05.790019+0000","last_fullsized":"2024-04-16T16:23:05.790019+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:03.451820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:03.451820+0000","last_clean_scrub_stamp":"2024-04-16T16:23:03.451820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:28.503649+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:12.047734+0000","last_change":"2024-04-16T16:23:12.047792+0000","last_active":"2024-04-16T16:23:12.047734+0000","last_peered":"2024-04-16T16:23:12.047734+0000","last_clean":"2024-04-16T16:23:12.047734+0000","last_became_active":"2024-04-16T16:22:46.479857+0000","last_became_peered":"2024-04-16T16:22:46.479857+0000","last_unstale":"2024-04-16T16:23:12.047734+0000","last_undegraded":"2024-04-16T16:23:12.047734+0000","last_fullsized":"2024-04-16T16:23:12.047734+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:12.047531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:12.047531+0000","last_clean_scrub_stamp":"2024-04-16T16:23:12.047531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:50:56.351523+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:10.511647+0000","last_change":"2024-04-16T16:23:10.511704+0000","last_active":"2024-04-16T16:23:10.511647+0000","last_peered":"2024-04-16T16:23:10.511647+0000","last_clean":"2024-04-16T16:23:10.511647+0000","last_became_active":"2024-04-16T16:22:46.477081+0000","last_became_peered":"2024-04-16T16:22:46.477081+0000","last_unstale":"2024-04-16T16:23:10.511647+0000","last_undegraded":"2024-04-16T16:23:10.511647+0000","last_fullsized":"2024-04-16T16:23:10.511647+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:10.511423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:10.511423+0000","last_clean_scrub_stamp":"2024-04-16T16:23:10.511423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:39.075764+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.971321+0000","last_change":"2024-04-16T16:23:05.971374+0000","last_active":"2024-04-16T16:23:05.971321+0000","last_peered":"2024-04-16T16:23:05.971321+0000","last_clean":"2024-04-16T16:23:05.971321+0000","last_became_active":"2024-04-16T16:22:46.473621+0000","last_became_peered":"2024-04-16T16:22:46.473621+0000","last_unstale":"2024-04-16T16:23:05.971321+0000","last_undegraded":"2024-04-16T16:23:05.971321+0000","last_fullsized":"2024-04-16T16:23:05.971321+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:05.971087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:05.971087+0000","last_clean_scrub_stamp":"2024-04-16T16:23:05.971087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:31.887510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:09.469919+0000","last_change":"2024-04-16T16:23:09.469982+0000","last_active":"2024-04-16T16:23:09.469919+0000","last_peered":"2024-04-16T16:23:09.469919+0000","last_clean":"2024-04-16T16:23:09.469919+0000","last_became_active":"2024-04-16T16:22:46.473183+0000","last_became_peered":"2024-04-16T16:22:46.473183+0000","last_unstale":"2024-04-16T16:23:09.469919+0000","last_undegraded":"2024-04-16T16:23:09.469919+0000","last_fullsized":"2024-04-16T16:23:09.469919+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:09.469448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:09.469448+0000","last_clean_scrub_stamp":"2024-04-16T16:23:09.469448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:59.363443+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:09.043925+0000","last_change":"2024-04-16T16:23:09.043986+0000","last_active":"2024-04-16T16:23:09.043925+0000","last_peered":"2024-04-16T16:23:09.043925+0000","last_clean":"2024-04-16T16:23:09.043925+0000","last_became_active":"2024-04-16T16:22:46.477303+0000","last_became_peered":"2024-04-16T16:22:46.477303+0000","last_unstale":"2024-04-16T16:23:09.043925+0000","last_undegraded":"2024-04-16T16:23:09.043925+0000","last_fullsized":"2024-04-16T16:23:09.043925+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:09.043729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:09.043729+0000","last_clean_scrub_stamp":"2024-04-16T16:23:09.043729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:43:43.506180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:10.065119+0000","last_change":"2024-04-16T16:23:10.065176+0000","last_active":"2024-04-16T16:23:10.065119+0000","last_peered":"2024-04-16T16:23:10.065119+0000","last_clean":"2024-04-16T16:23:10.065119+0000","last_became_active":"2024-04-16T16:22:46.480641+0000","last_became_peered":"2024-04-16T16:22:46.480641+0000","last_unstale":"2024-04-16T16:23:10.065119+0000","last_undegraded":"2024-04-16T16:23:10.065119+0000","last_fullsized":"2024-04-16T16:23:10.065119+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:10.064928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:10.064928+0000","last_clean_scrub_stamp":"2024-04-16T16:23:10.064928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:35.981351+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.803168+0000","last_change":"2024-04-16T16:23:03.657129+0000","last_active":"2024-04-16T16:23:05.803168+0000","last_peered":"2024-04-16T16:23:05.803168+0000","last_clean":"2024-04-16T16:23:05.803168+0000","last_became_active":"2024-04-16T16:22:46.466756+0000","last_became_peered":"2024-04-16T16:22:46.466756+0000","last_unstale":"2024-04-16T16:23:05.803168+0000","last_undegraded":"2024-04-16T16:23:05.803168+0000","last_fullsized":"2024-04-16T16:23:05.803168+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:03.656926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:03.656926+0000","last_clean_scrub_stamp":"2024-04-16T16:23:03.656926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:57.660894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:16.135584+0000","last_change":"2024-04-16T16:23:16.135646+0000","last_active":"2024-04-16T16:23:16.135584+0000","last_peered":"2024-04-16T16:23:16.135584+0000","last_clean":"2024-04-16T16:23:16.135584+0000","last_became_active":"2024-04-16T16:22:46.483654+0000","last_became_peered":"2024-04-16T16:22:46.483654+0000","last_unstale":"2024-04-16T16:23:16.135584+0000","last_undegraded":"2024-04-16T16:23:16.135584+0000","last_fullsized":"2024-04-16T16:23:16.135584+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:16.135323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:16.135323+0000","last_clean_scrub_stamp":"2024-04-16T16:23:16.135323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:27.379169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:13.244084+0000","last_change":"2024-04-16T16:23:13.244137+0000","last_active":"2024-04-16T16:23:13.244084+0000","last_peered":"2024-04-16T16:23:13.244084+0000","last_clean":"2024-04-16T16:23:13.244084+0000","last_became_active":"2024-04-16T16:22:46.483750+0000","last_became_peered":"2024-04-16T16:22:46.483750+0000","last_unstale":"2024-04-16T16:23:13.244084+0000","last_undegraded":"2024-04-16T16:23:13.244084+0000","last_fullsized":"2024-04-16T16:23:13.244084+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:13.243882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:13.243882+0000","last_clean_scrub_stamp":"2024-04-16T16:23:13.243882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:30:50.979597+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:14.065088+0000","last_change":"2024-04-16T16:23:14.065144+0000","last_active":"2024-04-16T16:23:14.065088+0000","last_peered":"2024-04-16T16:23:14.065088+0000","last_clean":"2024-04-16T16:23:14.065088+0000","last_became_active":"2024-04-16T16:22:46.481358+0000","last_became_peered":"2024-04-16T16:22:46.481358+0000","last_unstale":"2024-04-16T16:23:14.065088+0000","last_undegraded":"2024-04-16T16:23:14.065088+0000","last_fullsized":"2024-04-16T16:23:14.065088+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:14.064880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:14.064880+0000","last_clean_scrub_stamp":"2024-04-16T16:23:14.064880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:54:55.493584+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:10.189347+0000","last_change":"2024-04-16T16:23:10.189400+0000","last_active":"2024-04-16T16:23:10.189347+0000","last_peered":"2024-04-16T16:23:10.189347+0000","last_clean":"2024-04-16T16:23:10.189347+0000","last_became_active":"2024-04-16T16:22:46.463536+0000","last_became_peered":"2024-04-16T16:22:46.463536+0000","last_unstale":"2024-04-16T16:23:10.189347+0000","last_undegraded":"2024-04-16T16:23:10.189347+0000","last_fullsized":"2024-04-16T16:23:10.189347+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:10.189093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:10.189093+0000","last_clean_scrub_stamp":"2024-04-16T16:23:10.189093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:54.563851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:08.143588+0000","last_change":"2024-04-16T16:23:08.143640+0000","last_active":"2024-04-16T16:23:08.143588+0000","last_peered":"2024-04-16T16:23:08.143588+0000","last_clean":"2024-04-16T16:23:08.143588+0000","last_became_active":"2024-04-16T16:22:46.458962+0000","last_became_peered":"2024-04-16T16:22:46.458962+0000","last_unstale":"2024-04-16T16:23:08.143588+0000","last_undegraded":"2024-04-16T16:23:08.143588+0000","last_fullsized":"2024-04-16T16:23:08.143588+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:08.143397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:08.143397+0000","last_clean_scrub_stamp":"2024-04-16T16:23:08.143397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:00:10.596199+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:10.452327+0000","last_change":"2024-04-16T16:23:10.452386+0000","last_active":"2024-04-16T16:23:10.452327+0000","last_peered":"2024-04-16T16:23:10.452327+0000","last_clean":"2024-04-16T16:23:10.452327+0000","last_became_active":"2024-04-16T16:22:46.464851+0000","last_became_peered":"2024-04-16T16:22:46.464851+0000","last_unstale":"2024-04-16T16:23:10.452327+0000","last_undegraded":"2024-04-16T16:23:10.452327+0000","last_fullsized":"2024-04-16T16:23:10.452327+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:10.452130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:10.452130+0000","last_clean_scrub_stamp":"2024-04-16T16:23:10.452130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:21:40.490529+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.803257+0000","last_change":"2024-04-16T16:23:02.705183+0000","last_active":"2024-04-16T16:23:05.803257+0000","last_peered":"2024-04-16T16:23:05.803257+0000","last_clean":"2024-04-16T16:23:05.803257+0000","last_became_active":"2024-04-16T16:22:46.466366+0000","last_became_peered":"2024-04-16T16:22:46.466366+0000","last_unstale":"2024-04-16T16:23:05.803257+0000","last_undegraded":"2024-04-16T16:23:05.803257+0000","last_fullsized":"2024-04-16T16:23:05.803257+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:02.705030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:02.705030+0000","last_clean_scrub_stamp":"2024-04-16T16:23:02.705030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:38.787293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"157'5","reported_seq":32,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:09.182606+0000","last_change":"2024-04-16T16:23:09.182772+0000","last_active":"2024-04-16T16:23:09.182606+0000","last_peered":"2024-04-16T16:23:09.182606+0000","last_clean":"2024-04-16T16:23:09.182606+0000","last_became_active":"2024-04-16T16:22:45.467695+0000","last_became_peered":"2024-04-16T16:22:45.467695+0000","last_unstale":"2024-04-16T16:23:09.182606+0000","last_undegraded":"2024-04-16T16:23:09.182606+0000","last_fullsized":"2024-04-16T16:23:09.182606+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'5","last_scrub_stamp":"2024-04-16T16:23:09.182377+0000","last_deep_scrub":"157'5","last_deep_scrub_stamp":"2024-04-16T16:23:09.182377+0000","last_clean_scrub_stamp":"2024-04-16T16:23:09.182377+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:04:32.345934+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:11.209095+0000","last_change":"2024-04-16T16:23:11.209150+0000","last_active":"2024-04-16T16:23:11.209095+0000","last_peered":"2024-04-16T16:23:11.209095+0000","last_clean":"2024-04-16T16:23:11.209095+0000","last_became_active":"2024-04-16T16:22:46.462099+0000","last_became_peered":"2024-04-16T16:22:46.462099+0000","last_unstale":"2024-04-16T16:23:11.209095+0000","last_undegraded":"2024-04-16T16:23:11.209095+0000","last_fullsized":"2024-04-16T16:23:11.209095+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:11.208903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:11.208903+0000","last_clean_scrub_stamp":"2024-04-16T16:23:11.208903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:43.866149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"157'2","reported_seq":27,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.778314+0000","last_change":"2024-04-16T16:23:05.183427+0000","last_active":"2024-04-16T16:23:05.778314+0000","last_peered":"2024-04-16T16:23:05.778314+0000","last_clean":"2024-04-16T16:23:05.778314+0000","last_became_active":"2024-04-16T16:22:45.464379+0000","last_became_peered":"2024-04-16T16:22:45.464379+0000","last_unstale":"2024-04-16T16:23:05.778314+0000","last_undegraded":"2024-04-16T16:23:05.778314+0000","last_fullsized":"2024-04-16T16:23:05.778314+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'1","last_scrub_stamp":"2024-04-16T16:23:05.183139+0000","last_deep_scrub":"155'1","last_deep_scrub_stamp":"2024-04-16T16:23:05.183139+0000","last_clean_scrub_stamp":"2024-04-16T16:23:05.183139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:51.246421+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.789224+0000","last_change":"2024-04-16T16:23:02.447133+0000","last_active":"2024-04-16T16:23:05.789224+0000","last_peered":"2024-04-16T16:23:05.789224+0000","last_clean":"2024-04-16T16:23:05.789224+0000","last_became_active":"2024-04-16T16:22:46.454610+0000","last_became_peered":"2024-04-16T16:22:46.454610+0000","last_unstale":"2024-04-16T16:23:05.789224+0000","last_undegraded":"2024-04-16T16:23:05.789224+0000","last_fullsized":"2024-04-16T16:23:05.789224+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:02.446932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:02.446932+0000","last_clean_scrub_stamp":"2024-04-16T16:23:02.446932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:33:05.284544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"155'2","reported_seq":29,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:08.800914+0000","last_change":"2024-04-16T16:23:06.260448+0000","last_active":"2024-04-16T16:23:08.800914+0000","last_peered":"2024-04-16T16:23:08.800914+0000","last_clean":"2024-04-16T16:23:08.800914+0000","last_became_active":"2024-04-16T16:22:45.469537+0000","last_became_peered":"2024-04-16T16:22:45.469537+0000","last_unstale":"2024-04-16T16:23:08.800914+0000","last_undegraded":"2024-04-16T16:23:08.800914+0000","last_fullsized":"2024-04-16T16:23:08.800914+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'2","last_scrub_stamp":"2024-04-16T16:23:06.260183+0000","last_deep_scrub":"155'2","last_deep_scrub_stamp":"2024-04-16T16:23:06.260183+0000","last_clean_scrub_stamp":"2024-04-16T16:23:06.260183+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:13:19.350603+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:08.096179+0000","last_change":"2024-04-16T16:23:08.096266+0000","last_active":"2024-04-16T16:23:08.096179+0000","last_peered":"2024-04-16T16:23:08.096179+0000","last_clean":"2024-04-16T16:23:08.096179+0000","last_became_active":"2024-04-16T16:22:46.479206+0000","last_became_peered":"2024-04-16T16:22:46.479206+0000","last_unstale":"2024-04-16T16:23:08.096179+0000","last_undegraded":"2024-04-16T16:23:08.096179+0000","last_fullsized":"2024-04-16T16:23:08.096179+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:08.095981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:08.095981+0000","last_clean_scrub_stamp":"2024-04-16T16:23:08.095981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:54.159263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"155'2","reported_seq":29,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:08.804335+0000","last_change":"2024-04-16T16:23:01.123321+0000","last_active":"2024-04-16T16:23:08.804335+0000","last_peered":"2024-04-16T16:23:08.804335+0000","last_clean":"2024-04-16T16:23:08.804335+0000","last_became_active":"2024-04-16T16:22:45.462170+0000","last_became_peered":"2024-04-16T16:22:45.462170+0000","last_unstale":"2024-04-16T16:23:08.804335+0000","last_undegraded":"2024-04-16T16:23:08.804335+0000","last_fullsized":"2024-04-16T16:23:08.804335+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'2","last_scrub_stamp":"2024-04-16T16:23:01.123009+0000","last_deep_scrub":"155'2","last_deep_scrub_stamp":"2024-04-16T16:23:01.123009+0000","last_clean_scrub_stamp":"2024-04-16T16:23:01.123009+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:26:54.097272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.802995+0000","last_change":"2024-04-16T16:22:59.696044+0000","last_active":"2024-04-16T16:23:05.802995+0000","last_peered":"2024-04-16T16:23:05.802995+0000","last_clean":"2024-04-16T16:23:05.802995+0000","last_became_active":"2024-04-16T16:22:46.459876+0000","last_became_peered":"2024-04-16T16:22:46.459876+0000","last_unstale":"2024-04-16T16:23:05.802995+0000","last_undegraded":"2024-04-16T16:23:05.802995+0000","last_fullsized":"2024-04-16T16:23:05.802995+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:59.695809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:59.695809+0000","last_clean_scrub_stamp":"2024-04-16T16:22:59.695809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:17:12.534395+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"157'2","reported_seq":27,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.786750+0000","last_change":"2024-04-16T16:23:05.104981+0000","last_active":"2024-04-16T16:23:05.786750+0000","last_peered":"2024-04-16T16:23:05.786750+0000","last_clean":"2024-04-16T16:23:05.786750+0000","last_became_active":"2024-04-16T16:22:45.472020+0000","last_became_peered":"2024-04-16T16:22:45.472020+0000","last_unstale":"2024-04-16T16:23:05.786750+0000","last_undegraded":"2024-04-16T16:23:05.786750+0000","last_fullsized":"2024-04-16T16:23:05.786750+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'1","last_scrub_stamp":"2024-04-16T16:23:05.104723+0000","last_deep_scrub":"155'1","last_deep_scrub_stamp":"2024-04-16T16:23:05.104723+0000","last_clean_scrub_stamp":"2024-04-16T16:23:05.104723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:29:54.670260+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:09.224703+0000","last_change":"2024-04-16T16:23:09.224765+0000","last_active":"2024-04-16T16:23:09.224703+0000","last_peered":"2024-04-16T16:23:09.224703+0000","last_clean":"2024-04-16T16:23:09.224703+0000","last_became_active":"2024-04-16T16:22:46.467898+0000","last_became_peered":"2024-04-16T16:22:46.467898+0000","last_unstale":"2024-04-16T16:23:09.224703+0000","last_undegraded":"2024-04-16T16:23:09.224703+0000","last_fullsized":"2024-04-16T16:23:09.224703+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:09.224458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:09.224458+0000","last_clean_scrub_stamp":"2024-04-16T16:23:09.224458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:28:31.477367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"157'2","reported_seq":27,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.789888+0000","last_change":"2024-04-16T16:22:59.102285+0000","last_active":"2024-04-16T16:23:05.789888+0000","last_peered":"2024-04-16T16:23:05.789888+0000","last_clean":"2024-04-16T16:23:05.789888+0000","last_became_active":"2024-04-16T16:22:45.461674+0000","last_became_peered":"2024-04-16T16:22:45.461674+0000","last_unstale":"2024-04-16T16:23:05.789888+0000","last_undegraded":"2024-04-16T16:23:05.789888+0000","last_fullsized":"2024-04-16T16:23:05.789888+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'1","last_scrub_stamp":"2024-04-16T16:22:59.102114+0000","last_deep_scrub":"155'1","last_deep_scrub_stamp":"2024-04-16T16:22:59.102114+0000","last_clean_scrub_stamp":"2024-04-16T16:22:59.102114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:29:48.667650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:08.263349+0000","last_change":"2024-04-16T16:23:08.263402+0000","last_active":"2024-04-16T16:23:08.263349+0000","last_peered":"2024-04-16T16:23:08.263349+0000","last_clean":"2024-04-16T16:23:08.263349+0000","last_became_active":"2024-04-16T16:22:46.465329+0000","last_became_peered":"2024-04-16T16:22:46.465329+0000","last_unstale":"2024-04-16T16:23:08.263349+0000","last_undegraded":"2024-04-16T16:23:08.263349+0000","last_fullsized":"2024-04-16T16:23:08.263349+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:08.263135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:08.263135+0000","last_clean_scrub_stamp":"2024-04-16T16:23:08.263135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:40.920381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.796768+0000","last_change":"2024-04-16T16:23:03.009822+0000","last_active":"2024-04-16T16:23:05.796768+0000","last_peered":"2024-04-16T16:23:05.796768+0000","last_clean":"2024-04-16T16:23:05.796768+0000","last_became_active":"2024-04-16T16:22:45.466591+0000","last_became_peered":"2024-04-16T16:22:45.466591+0000","last_unstale":"2024-04-16T16:23:05.796768+0000","last_undegraded":"2024-04-16T16:23:05.796768+0000","last_fullsized":"2024-04-16T16:23:05.796768+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:03.009566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:03.009566+0000","last_clean_scrub_stamp":"2024-04-16T16:23:03.009566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:26:55.983829+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.789192+0000","last_change":"2024-04-16T16:23:04.489751+0000","last_active":"2024-04-16T16:23:05.789192+0000","last_peered":"2024-04-16T16:23:05.789192+0000","last_clean":"2024-04-16T16:23:05.789192+0000","last_became_active":"2024-04-16T16:22:46.460420+0000","last_became_peered":"2024-04-16T16:22:46.460420+0000","last_unstale":"2024-04-16T16:23:05.789192+0000","last_undegraded":"2024-04-16T16:23:05.789192+0000","last_fullsized":"2024-04-16T16:23:05.789192+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:04.489487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:04.489487+0000","last_clean_scrub_stamp":"2024-04-16T16:23:04.489487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:17:47.982880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.802450+0000","last_change":"2024-04-16T16:22:58.730308+0000","last_active":"2024-04-16T16:23:05.802450+0000","last_peered":"2024-04-16T16:23:05.802450+0000","last_clean":"2024-04-16T16:23:05.802450+0000","last_became_active":"2024-04-16T16:22:45.455767+0000","last_became_peered":"2024-04-16T16:22:45.455767+0000","last_unstale":"2024-04-16T16:23:05.802450+0000","last_undegraded":"2024-04-16T16:23:05.802450+0000","last_fullsized":"2024-04-16T16:23:05.802450+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:22:58.730057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:22:58.730057+0000","last_clean_scrub_stamp":"2024-04-16T16:22:58.730057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:00:28.919352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:09.407746+0000","last_change":"2024-04-16T16:23:09.407846+0000","last_active":"2024-04-16T16:23:09.407746+0000","last_peered":"2024-04-16T16:23:09.407746+0000","last_clean":"2024-04-16T16:23:09.407746+0000","last_became_active":"2024-04-16T16:22:46.465149+0000","last_became_peered":"2024-04-16T16:22:46.465149+0000","last_unstale":"2024-04-16T16:23:09.407746+0000","last_undegraded":"2024-04-16T16:23:09.407746+0000","last_fullsized":"2024-04-16T16:23:09.407746+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:09.407406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:09.407406+0000","last_clean_scrub_stamp":"2024-04-16T16:23:09.407406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:26:16.501773+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.790330+0000","last_change":"2024-04-16T16:23:01.438986+0000","last_active":"2024-04-16T16:23:05.790330+0000","last_peered":"2024-04-16T16:23:05.790330+0000","last_clean":"2024-04-16T16:23:05.790330+0000","last_became_active":"2024-04-16T16:22:45.467704+0000","last_became_peered":"2024-04-16T16:22:45.467704+0000","last_unstale":"2024-04-16T16:23:05.790330+0000","last_undegraded":"2024-04-16T16:23:05.790330+0000","last_fullsized":"2024-04-16T16:23:05.790330+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:01.438795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:01.438795+0000","last_clean_scrub_stamp":"2024-04-16T16:23:01.438795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:34:51.177901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:13.378627+0000","last_change":"2024-04-16T16:23:13.378682+0000","last_active":"2024-04-16T16:23:13.378627+0000","last_peered":"2024-04-16T16:23:13.378627+0000","last_clean":"2024-04-16T16:23:13.378627+0000","last_became_active":"2024-04-16T16:22:46.473500+0000","last_became_peered":"2024-04-16T16:22:46.473500+0000","last_unstale":"2024-04-16T16:23:13.378627+0000","last_undegraded":"2024-04-16T16:23:13.378627+0000","last_fullsized":"2024-04-16T16:23:13.378627+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:13.378415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:13.378415+0000","last_clean_scrub_stamp":"2024-04-16T16:23:13.378415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:24:44.972304+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"155'3","reported_seq":31,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:08.798958+0000","last_change":"2024-04-16T16:23:00.430865+0000","last_active":"2024-04-16T16:23:08.798958+0000","last_peered":"2024-04-16T16:23:08.798958+0000","last_clean":"2024-04-16T16:23:08.798958+0000","last_became_active":"2024-04-16T16:22:45.461088+0000","last_became_peered":"2024-04-16T16:22:45.461088+0000","last_unstale":"2024-04-16T16:23:08.798958+0000","last_undegraded":"2024-04-16T16:23:08.798958+0000","last_fullsized":"2024-04-16T16:23:08.798958+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'3","last_scrub_stamp":"2024-04-16T16:23:00.430607+0000","last_deep_scrub":"155'3","last_deep_scrub_stamp":"2024-04-16T16:23:00.430607+0000","last_clean_scrub_stamp":"2024-04-16T16:23:00.430607+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:43:02.426690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:12.166209+0000","last_change":"2024-04-16T16:23:12.166286+0000","last_active":"2024-04-16T16:23:12.166209+0000","last_peered":"2024-04-16T16:23:12.166209+0000","last_clean":"2024-04-16T16:23:12.166209+0000","last_became_active":"2024-04-16T16:22:46.472979+0000","last_became_peered":"2024-04-16T16:22:46.472979+0000","last_unstale":"2024-04-16T16:23:12.166209+0000","last_undegraded":"2024-04-16T16:23:12.166209+0000","last_fullsized":"2024-04-16T16:23:12.166209+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:12.166010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:12.166010+0000","last_clean_scrub_stamp":"2024-04-16T16:23:12.166010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:28:34.418919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"157'7","reported_seq":38,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:08.802726+0000","last_change":"2024-04-16T16:23:04.229186+0000","last_active":"2024-04-16T16:23:08.802726+0000","last_peered":"2024-04-16T16:23:08.802726+0000","last_clean":"2024-04-16T16:23:08.802726+0000","last_became_active":"2024-04-16T16:22:45.461583+0000","last_became_peered":"2024-04-16T16:22:45.461583+0000","last_unstale":"2024-04-16T16:23:08.802726+0000","last_undegraded":"2024-04-16T16:23:08.802726+0000","last_fullsized":"2024-04-16T16:23:08.802726+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'6","last_scrub_stamp":"2024-04-16T16:23:04.228752+0000","last_deep_scrub":"155'6","last_deep_scrub_stamp":"2024-04-16T16:23:04.228752+0000","last_clean_scrub_stamp":"2024-04-16T16:23:04.228752+0000","objects_scrubbed":4,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:26:57.203015+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:08.455162+0000","last_change":"2024-04-16T16:23:08.455315+0000","last_active":"2024-04-16T16:23:08.455162+0000","last_peered":"2024-04-16T16:23:08.455162+0000","last_clean":"2024-04-16T16:23:08.455162+0000","last_became_active":"2024-04-16T16:22:46.465894+0000","last_became_peered":"2024-04-16T16:22:46.465894+0000","last_unstale":"2024-04-16T16:23:08.455162+0000","last_undegraded":"2024-04-16T16:23:08.455162+0000","last_fullsized":"2024-04-16T16:23:08.455162+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:08.454940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:08.454940+0000","last_clean_scrub_stamp":"2024-04-16T16:23:08.454940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:20:46.990006+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"155'2","reported_seq":29,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.791811+0000","last_change":"2024-04-16T16:23:02.132784+0000","last_active":"2024-04-16T16:23:05.791811+0000","last_peered":"2024-04-16T16:23:05.791811+0000","last_clean":"2024-04-16T16:23:05.791811+0000","last_became_active":"2024-04-16T16:22:45.465893+0000","last_became_peered":"2024-04-16T16:22:45.465893+0000","last_unstale":"2024-04-16T16:23:05.791811+0000","last_undegraded":"2024-04-16T16:23:05.791811+0000","last_fullsized":"2024-04-16T16:23:05.791811+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'2","last_scrub_stamp":"2024-04-16T16:23:02.132427+0000","last_deep_scrub":"155'2","last_deep_scrub_stamp":"2024-04-16T16:23:02.132427+0000","last_clean_scrub_stamp":"2024-04-16T16:23:02.132427+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:48.195709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:07.738993+0000","last_change":"2024-04-16T16:23:07.739047+0000","last_active":"2024-04-16T16:23:07.738993+0000","last_peered":"2024-04-16T16:23:07.738993+0000","last_clean":"2024-04-16T16:23:07.738993+0000","last_became_active":"2024-04-16T16:22:46.474146+0000","last_became_peered":"2024-04-16T16:22:46.474146+0000","last_unstale":"2024-04-16T16:23:07.738993+0000","last_undegraded":"2024-04-16T16:23:07.738993+0000","last_fullsized":"2024-04-16T16:23:07.738993+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:07.738794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:07.738794+0000","last_clean_scrub_stamp":"2024-04-16T16:23:07.738794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:21:08.451426+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.777374+0000","last_change":"2024-04-16T16:23:01.547621+0000","last_active":"2024-04-16T16:23:05.777374+0000","last_peered":"2024-04-16T16:23:05.777374+0000","last_clean":"2024-04-16T16:23:05.777374+0000","last_became_active":"2024-04-16T16:22:45.462875+0000","last_became_peered":"2024-04-16T16:22:45.462875+0000","last_unstale":"2024-04-16T16:23:05.777374+0000","last_undegraded":"2024-04-16T16:23:05.777374+0000","last_fullsized":"2024-04-16T16:23:05.777374+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:01.547493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:01.547493+0000","last_clean_scrub_stamp":"2024-04-16T16:23:01.547493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:08:51.761612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:10.228776+0000","last_change":"2024-04-16T16:23:10.228830+0000","last_active":"2024-04-16T16:23:10.228776+0000","last_peered":"2024-04-16T16:23:10.228776+0000","last_clean":"2024-04-16T16:23:10.228776+0000","last_became_active":"2024-04-16T16:22:46.472859+0000","last_became_peered":"2024-04-16T16:22:46.472859+0000","last_unstale":"2024-04-16T16:23:10.228776+0000","last_undegraded":"2024-04-16T16:23:10.228776+0000","last_fullsized":"2024-04-16T16:23:10.228776+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:10.228537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:10.228537+0000","last_clean_scrub_stamp":"2024-04-16T16:23:10.228537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:21.472383+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.789326+0000","last_change":"2024-04-16T16:23:03.171269+0000","last_active":"2024-04-16T16:23:05.789326+0000","last_peered":"2024-04-16T16:23:05.789326+0000","last_clean":"2024-04-16T16:23:05.789326+0000","last_became_active":"2024-04-16T16:22:45.466535+0000","last_became_peered":"2024-04-16T16:22:45.466535+0000","last_unstale":"2024-04-16T16:23:05.789326+0000","last_undegraded":"2024-04-16T16:23:05.789326+0000","last_fullsized":"2024-04-16T16:23:05.789326+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:03.171045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:03.171045+0000","last_clean_scrub_stamp":"2024-04-16T16:23:03.171045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:54:05.922164+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:13.041131+0000","last_change":"2024-04-16T16:23:13.041267+0000","last_active":"2024-04-16T16:23:13.041131+0000","last_peered":"2024-04-16T16:23:13.041131+0000","last_clean":"2024-04-16T16:23:13.041131+0000","last_became_active":"2024-04-16T16:22:46.482943+0000","last_became_peered":"2024-04-16T16:22:46.482943+0000","last_unstale":"2024-04-16T16:23:13.041131+0000","last_undegraded":"2024-04-16T16:23:13.041131+0000","last_fullsized":"2024-04-16T16:23:13.041131+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:13.040874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:13.040874+0000","last_clean_scrub_stamp":"2024-04-16T16:23:13.040874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:43:47.503325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"155'8","reported_seq":41,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:08.814431+0000","last_change":"2024-04-16T16:23:02.212440+0000","last_active":"2024-04-16T16:23:08.814431+0000","last_peered":"2024-04-16T16:23:08.814431+0000","last_clean":"2024-04-16T16:23:08.814431+0000","last_became_active":"2024-04-16T16:22:45.459760+0000","last_became_peered":"2024-04-16T16:22:45.459760+0000","last_unstale":"2024-04-16T16:23:08.814431+0000","last_undegraded":"2024-04-16T16:23:08.814431+0000","last_fullsized":"2024-04-16T16:23:08.814431+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'8","last_scrub_stamp":"2024-04-16T16:23:02.212107+0000","last_deep_scrub":"155'8","last_deep_scrub_stamp":"2024-04-16T16:23:02.212107+0000","last_clean_scrub_stamp":"2024-04-16T16:23:02.212107+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:29:51.777644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:11.256605+0000","last_change":"2024-04-16T16:23:11.256661+0000","last_active":"2024-04-16T16:23:11.256605+0000","last_peered":"2024-04-16T16:23:11.256605+0000","last_clean":"2024-04-16T16:23:11.256605+0000","last_became_active":"2024-04-16T16:22:46.471894+0000","last_became_peered":"2024-04-16T16:22:46.471894+0000","last_unstale":"2024-04-16T16:23:11.256605+0000","last_undegraded":"2024-04-16T16:23:11.256605+0000","last_fullsized":"2024-04-16T16:23:11.256605+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:11.256395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:11.256395+0000","last_clean_scrub_stamp":"2024-04-16T16:23:11.256395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:55:50.523818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"155'1","reported_seq":27,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.787124+0000","last_change":"2024-04-16T16:23:03.184471+0000","last_active":"2024-04-16T16:23:05.787124+0000","last_peered":"2024-04-16T16:23:05.787124+0000","last_clean":"2024-04-16T16:23:05.787124+0000","last_became_active":"2024-04-16T16:22:45.468966+0000","last_became_peered":"2024-04-16T16:22:45.468966+0000","last_unstale":"2024-04-16T16:23:05.787124+0000","last_undegraded":"2024-04-16T16:23:05.787124+0000","last_fullsized":"2024-04-16T16:23:05.787124+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'1","last_scrub_stamp":"2024-04-16T16:23:03.184186+0000","last_deep_scrub":"155'1","last_deep_scrub_stamp":"2024-04-16T16:23:03.184186+0000","last_clean_scrub_stamp":"2024-04-16T16:23:03.184186+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:44.554350+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:12.421816+0000","last_change":"2024-04-16T16:23:12.421869+0000","last_active":"2024-04-16T16:23:12.421816+0000","last_peered":"2024-04-16T16:23:12.421816+0000","last_clean":"2024-04-16T16:23:12.421816+0000","last_became_active":"2024-04-16T16:22:46.470405+0000","last_became_peered":"2024-04-16T16:22:46.470405+0000","last_unstale":"2024-04-16T16:23:12.421816+0000","last_undegraded":"2024-04-16T16:23:12.421816+0000","last_fullsized":"2024-04-16T16:23:12.421816+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:12.421577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:12.421577+0000","last_clean_scrub_stamp":"2024-04-16T16:23:12.421577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:21:30.170790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"155'2","reported_seq":29,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:08.805965+0000","last_change":"2024-04-16T16:23:04.151958+0000","last_active":"2024-04-16T16:23:08.805965+0000","last_peered":"2024-04-16T16:23:08.805965+0000","last_clean":"2024-04-16T16:23:08.805965+0000","last_became_active":"2024-04-16T16:22:45.469959+0000","last_became_peered":"2024-04-16T16:22:45.469959+0000","last_unstale":"2024-04-16T16:23:08.805965+0000","last_undegraded":"2024-04-16T16:23:08.805965+0000","last_fullsized":"2024-04-16T16:23:08.805965+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'2","last_scrub_stamp":"2024-04-16T16:23:04.151691+0000","last_deep_scrub":"155'2","last_deep_scrub_stamp":"2024-04-16T16:23:04.151691+0000","last_clean_scrub_stamp":"2024-04-16T16:23:04.151691+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:47.086722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:07.220379+0000","last_change":"2024-04-16T16:23:07.220488+0000","last_active":"2024-04-16T16:23:07.220379+0000","last_peered":"2024-04-16T16:23:07.220379+0000","last_clean":"2024-04-16T16:23:07.220379+0000","last_became_active":"2024-04-16T16:22:46.462662+0000","last_became_peered":"2024-04-16T16:22:46.462662+0000","last_unstale":"2024-04-16T16:23:07.220379+0000","last_undegraded":"2024-04-16T16:23:07.220379+0000","last_fullsized":"2024-04-16T16:23:07.220379+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:07.220139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:07.220139+0000","last_clean_scrub_stamp":"2024-04-16T16:23:07.220139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:51.594897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"155'1","reported_seq":27,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:07.183397+0000","last_change":"2024-04-16T16:23:07.183460+0000","last_active":"2024-04-16T16:23:07.183397+0000","last_peered":"2024-04-16T16:23:07.183397+0000","last_clean":"2024-04-16T16:23:07.183397+0000","last_became_active":"2024-04-16T16:22:45.462197+0000","last_became_peered":"2024-04-16T16:22:45.462197+0000","last_unstale":"2024-04-16T16:23:07.183397+0000","last_undegraded":"2024-04-16T16:23:07.183397+0000","last_fullsized":"2024-04-16T16:23:07.183397+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'1","last_scrub_stamp":"2024-04-16T16:23:07.183180+0000","last_deep_scrub":"155'1","last_deep_scrub_stamp":"2024-04-16T16:23:07.183180+0000","last_clean_scrub_stamp":"2024-04-16T16:23:07.183180+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:30:44.918896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:08.740007+0000","last_change":"2024-04-16T16:23:08.740069+0000","last_active":"2024-04-16T16:23:08.740007+0000","last_peered":"2024-04-16T16:23:08.740007+0000","last_clean":"2024-04-16T16:23:08.740007+0000","last_became_active":"2024-04-16T16:22:46.475232+0000","last_became_peered":"2024-04-16T16:22:46.475232+0000","last_unstale":"2024-04-16T16:23:08.740007+0000","last_undegraded":"2024-04-16T16:23:08.740007+0000","last_fullsized":"2024-04-16T16:23:08.740007+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:08.739809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:08.739809+0000","last_clean_scrub_stamp":"2024-04-16T16:23:08.739809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:50:18.104172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:08.190846+0000","last_change":"2024-04-16T16:23:08.190901+0000","last_active":"2024-04-16T16:23:08.190846+0000","last_peered":"2024-04-16T16:23:08.190846+0000","last_clean":"2024-04-16T16:23:08.190846+0000","last_became_active":"2024-04-16T16:22:45.466614+0000","last_became_peered":"2024-04-16T16:22:45.466614+0000","last_unstale":"2024-04-16T16:23:08.190846+0000","last_undegraded":"2024-04-16T16:23:08.190846+0000","last_fullsized":"2024-04-16T16:23:08.190846+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:08.190652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:08.190652+0000","last_clean_scrub_stamp":"2024-04-16T16:23:08.190652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:00:38.379947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:11.036027+0000","last_change":"2024-04-16T16:23:11.036134+0000","last_active":"2024-04-16T16:23:11.036027+0000","last_peered":"2024-04-16T16:23:11.036027+0000","last_clean":"2024-04-16T16:23:11.036027+0000","last_became_active":"2024-04-16T16:22:46.481293+0000","last_became_peered":"2024-04-16T16:22:46.481293+0000","last_unstale":"2024-04-16T16:23:11.036027+0000","last_undegraded":"2024-04-16T16:23:11.036027+0000","last_fullsized":"2024-04-16T16:23:11.036027+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:11.035788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:11.035788+0000","last_clean_scrub_stamp":"2024-04-16T16:23:11.035788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:00:13.488591+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"157'3","reported_seq":29,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.791861+0000","last_change":"2024-04-16T16:23:04.217231+0000","last_active":"2024-04-16T16:23:05.791861+0000","last_peered":"2024-04-16T16:23:05.791861+0000","last_clean":"2024-04-16T16:23:05.791861+0000","last_became_active":"2024-04-16T16:22:45.466624+0000","last_became_peered":"2024-04-16T16:22:45.466624+0000","last_unstale":"2024-04-16T16:23:05.791861+0000","last_undegraded":"2024-04-16T16:23:05.791861+0000","last_fullsized":"2024-04-16T16:23:05.791861+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'2","last_scrub_stamp":"2024-04-16T16:23:04.216984+0000","last_deep_scrub":"155'2","last_deep_scrub_stamp":"2024-04-16T16:23:04.216984+0000","last_clean_scrub_stamp":"2024-04-16T16:23:04.216984+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:17:15.516019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:06.703333+0000","last_change":"2024-04-16T16:23:06.703386+0000","last_active":"2024-04-16T16:23:06.703333+0000","last_peered":"2024-04-16T16:23:06.703333+0000","last_clean":"2024-04-16T16:23:06.703333+0000","last_became_active":"2024-04-16T16:22:46.466179+0000","last_became_peered":"2024-04-16T16:22:46.466179+0000","last_unstale":"2024-04-16T16:23:06.703333+0000","last_undegraded":"2024-04-16T16:23:06.703333+0000","last_fullsized":"2024-04-16T16:23:06.703333+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:06.703126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:06.703126+0000","last_clean_scrub_stamp":"2024-04-16T16:23:06.703126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:46:48.259964+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:06.184441+0000","last_change":"2024-04-16T16:23:06.184496+0000","last_active":"2024-04-16T16:23:06.184441+0000","last_peered":"2024-04-16T16:23:06.184441+0000","last_clean":"2024-04-16T16:23:06.184441+0000","last_became_active":"2024-04-16T16:22:45.461253+0000","last_became_peered":"2024-04-16T16:22:45.461253+0000","last_unstale":"2024-04-16T16:23:06.184441+0000","last_undegraded":"2024-04-16T16:23:06.184441+0000","last_fullsized":"2024-04-16T16:23:06.184441+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:06.184220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:06.184220+0000","last_clean_scrub_stamp":"2024-04-16T16:23:06.184220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:08:56.398339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:10.147132+0000","last_change":"2024-04-16T16:23:10.147186+0000","last_active":"2024-04-16T16:23:10.147132+0000","last_peered":"2024-04-16T16:23:10.147132+0000","last_clean":"2024-04-16T16:23:10.147132+0000","last_became_active":"2024-04-16T16:22:46.465294+0000","last_became_peered":"2024-04-16T16:22:46.465294+0000","last_unstale":"2024-04-16T16:23:10.147132+0000","last_undegraded":"2024-04-16T16:23:10.147132+0000","last_fullsized":"2024-04-16T16:23:10.147132+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:10.146937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:10.146937+0000","last_clean_scrub_stamp":"2024-04-16T16:23:10.146937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:49:09.048825+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.789457+0000","last_change":"2024-04-16T16:23:05.254097+0000","last_active":"2024-04-16T16:23:05.789457+0000","last_peered":"2024-04-16T16:23:05.789457+0000","last_clean":"2024-04-16T16:23:05.789457+0000","last_became_active":"2024-04-16T16:22:45.468091+0000","last_became_peered":"2024-04-16T16:22:45.468091+0000","last_unstale":"2024-04-16T16:23:05.789457+0000","last_undegraded":"2024-04-16T16:23:05.789457+0000","last_fullsized":"2024-04-16T16:23:05.789457+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:05.253840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:05.253840+0000","last_clean_scrub_stamp":"2024-04-16T16:23:05.253840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:00:07.706642+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:07.466460+0000","last_change":"2024-04-16T16:23:07.466583+0000","last_active":"2024-04-16T16:23:07.466460+0000","last_peered":"2024-04-16T16:23:07.466460+0000","last_clean":"2024-04-16T16:23:07.466460+0000","last_became_active":"2024-04-16T16:22:46.470521+0000","last_became_peered":"2024-04-16T16:22:46.470521+0000","last_unstale":"2024-04-16T16:23:07.466460+0000","last_undegraded":"2024-04-16T16:23:07.466460+0000","last_fullsized":"2024-04-16T16:23:07.466460+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:07.466238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:07.466238+0000","last_clean_scrub_stamp":"2024-04-16T16:23:07.466238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:09:58.829047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.779785+0000","last_change":"2024-04-16T16:23:05.201320+0000","last_active":"2024-04-16T16:23:05.779785+0000","last_peered":"2024-04-16T16:23:05.779785+0000","last_clean":"2024-04-16T16:23:05.779785+0000","last_became_active":"2024-04-16T16:22:45.458861+0000","last_became_peered":"2024-04-16T16:22:45.458861+0000","last_unstale":"2024-04-16T16:23:05.779785+0000","last_undegraded":"2024-04-16T16:23:05.779785+0000","last_fullsized":"2024-04-16T16:23:05.779785+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:05.201025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:05.201025+0000","last_clean_scrub_stamp":"2024-04-16T16:23:05.201025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:19.045701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.778360+0000","last_change":"2024-04-16T16:23:05.471371+0000","last_active":"2024-04-16T16:23:05.778360+0000","last_peered":"2024-04-16T16:23:05.778360+0000","last_clean":"2024-04-16T16:23:05.778360+0000","last_became_active":"2024-04-16T16:22:46.464064+0000","last_became_peered":"2024-04-16T16:22:46.464064+0000","last_unstale":"2024-04-16T16:23:05.778360+0000","last_undegraded":"2024-04-16T16:23:05.778360+0000","last_fullsized":"2024-04-16T16:23:05.778360+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:05.470954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:05.470954+0000","last_clean_scrub_stamp":"2024-04-16T16:23:05.470954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:49:04.372843+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"155'2","reported_seq":29,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:08.809749+0000","last_change":"2024-04-16T16:23:02.535009+0000","last_active":"2024-04-16T16:23:08.809749+0000","last_peered":"2024-04-16T16:23:08.809749+0000","last_clean":"2024-04-16T16:23:08.809749+0000","last_became_active":"2024-04-16T16:22:45.465676+0000","last_became_peered":"2024-04-16T16:22:45.465676+0000","last_unstale":"2024-04-16T16:23:08.809749+0000","last_undegraded":"2024-04-16T16:23:08.809749+0000","last_fullsized":"2024-04-16T16:23:08.809749+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'2","last_scrub_stamp":"2024-04-16T16:23:02.534852+0000","last_deep_scrub":"155'2","last_deep_scrub_stamp":"2024-04-16T16:23:02.534852+0000","last_clean_scrub_stamp":"2024-04-16T16:23:02.534852+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:30:40.270568+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.803203+0000","last_change":"2024-04-16T16:23:01.669748+0000","last_active":"2024-04-16T16:23:05.803203+0000","last_peered":"2024-04-16T16:23:05.803203+0000","last_clean":"2024-04-16T16:23:05.803203+0000","last_became_active":"2024-04-16T16:22:46.465449+0000","last_became_peered":"2024-04-16T16:22:46.465449+0000","last_unstale":"2024-04-16T16:23:05.803203+0000","last_undegraded":"2024-04-16T16:23:05.803203+0000","last_fullsized":"2024-04-16T16:23:05.803203+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:01.669447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:01.669447+0000","last_clean_scrub_stamp":"2024-04-16T16:23:01.669447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:06:05.606310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"157'1","reported_seq":26,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.786807+0000","last_change":"2024-04-16T16:23:02.140751+0000","last_active":"2024-04-16T16:23:05.786807+0000","last_peered":"2024-04-16T16:23:05.786807+0000","last_clean":"2024-04-16T16:23:05.786807+0000","last_became_active":"2024-04-16T16:22:45.465791+0000","last_became_peered":"2024-04-16T16:22:45.465791+0000","last_unstale":"2024-04-16T16:23:05.786807+0000","last_undegraded":"2024-04-16T16:23:05.786807+0000","last_fullsized":"2024-04-16T16:23:05.786807+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:02.140513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:02.140513+0000","last_clean_scrub_stamp":"2024-04-16T16:23:02.140513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:02:54.228545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:06.487557+0000","last_change":"2024-04-16T16:23:06.487633+0000","last_active":"2024-04-16T16:23:06.487557+0000","last_peered":"2024-04-16T16:23:06.487557+0000","last_clean":"2024-04-16T16:23:06.487557+0000","last_became_active":"2024-04-16T16:22:46.470196+0000","last_became_peered":"2024-04-16T16:22:46.470196+0000","last_unstale":"2024-04-16T16:23:06.487557+0000","last_undegraded":"2024-04-16T16:23:06.487557+0000","last_fullsized":"2024-04-16T16:23:06.487557+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:06.487357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:06.487357+0000","last_clean_scrub_stamp":"2024-04-16T16:23:06.487357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:04:15.132766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.797084+0000","last_change":"2024-04-16T16:23:05.015486+0000","last_active":"2024-04-16T16:23:05.797084+0000","last_peered":"2024-04-16T16:23:05.797084+0000","last_clean":"2024-04-16T16:23:05.797084+0000","last_became_active":"2024-04-16T16:22:45.470911+0000","last_became_peered":"2024-04-16T16:22:45.470911+0000","last_unstale":"2024-04-16T16:23:05.797084+0000","last_undegraded":"2024-04-16T16:23:05.797084+0000","last_fullsized":"2024-04-16T16:23:05.797084+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:05.015181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:05.015181+0000","last_clean_scrub_stamp":"2024-04-16T16:23:05.015181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:42.441272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:11.405370+0000","last_change":"2024-04-16T16:23:11.405424+0000","last_active":"2024-04-16T16:23:11.405370+0000","last_peered":"2024-04-16T16:23:11.405370+0000","last_clean":"2024-04-16T16:23:11.405370+0000","last_became_active":"2024-04-16T16:22:46.469009+0000","last_became_peered":"2024-04-16T16:22:46.469009+0000","last_unstale":"2024-04-16T16:23:11.405370+0000","last_undegraded":"2024-04-16T16:23:11.405370+0000","last_fullsized":"2024-04-16T16:23:11.405370+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:11.405162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:11.405162+0000","last_clean_scrub_stamp":"2024-04-16T16:23:11.405162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:16:06.901446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.777801+0000","last_change":"2024-04-16T16:23:03.514388+0000","last_active":"2024-04-16T16:23:05.777801+0000","last_peered":"2024-04-16T16:23:05.777801+0000","last_clean":"2024-04-16T16:23:05.777801+0000","last_became_active":"2024-04-16T16:22:45.467709+0000","last_became_peered":"2024-04-16T16:22:45.467709+0000","last_unstale":"2024-04-16T16:23:05.777801+0000","last_undegraded":"2024-04-16T16:23:05.777801+0000","last_fullsized":"2024-04-16T16:23:05.777801+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:03.514153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:03.514153+0000","last_clean_scrub_stamp":"2024-04-16T16:23:03.514153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:52:43.107104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:07.484331+0000","last_change":"2024-04-16T16:23:07.484435+0000","last_active":"2024-04-16T16:23:07.484331+0000","last_peered":"2024-04-16T16:23:07.484331+0000","last_clean":"2024-04-16T16:23:07.484331+0000","last_became_active":"2024-04-16T16:22:46.465007+0000","last_became_peered":"2024-04-16T16:22:46.465007+0000","last_unstale":"2024-04-16T16:23:07.484331+0000","last_undegraded":"2024-04-16T16:23:07.484331+0000","last_fullsized":"2024-04-16T16:23:07.484331+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:07.484117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:07.484117+0000","last_clean_scrub_stamp":"2024-04-16T16:23:07.484117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:04:37.337382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:07.113991+0000","last_change":"2024-04-16T16:23:07.114043+0000","last_active":"2024-04-16T16:23:07.113991+0000","last_peered":"2024-04-16T16:23:07.113991+0000","last_clean":"2024-04-16T16:23:07.113991+0000","last_became_active":"2024-04-16T16:22:45.469322+0000","last_became_peered":"2024-04-16T16:22:45.469322+0000","last_unstale":"2024-04-16T16:23:07.113991+0000","last_undegraded":"2024-04-16T16:23:07.113991+0000","last_fullsized":"2024-04-16T16:23:07.113991+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:07.113792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:07.113792+0000","last_clean_scrub_stamp":"2024-04-16T16:23:07.113792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:17:18.412827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:09.174172+0000","last_change":"2024-04-16T16:23:09.174262+0000","last_active":"2024-04-16T16:23:09.174172+0000","last_peered":"2024-04-16T16:23:09.174172+0000","last_clean":"2024-04-16T16:23:09.174172+0000","last_became_active":"2024-04-16T16:22:46.461432+0000","last_became_peered":"2024-04-16T16:22:46.461432+0000","last_unstale":"2024-04-16T16:23:09.174172+0000","last_undegraded":"2024-04-16T16:23:09.174172+0000","last_fullsized":"2024-04-16T16:23:09.174172+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:09.173987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:09.173987+0000","last_clean_scrub_stamp":"2024-04-16T16:23:09.173987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:13:22.264407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.789925+0000","last_change":"2024-04-16T16:23:00.114469+0000","last_active":"2024-04-16T16:23:05.789925+0000","last_peered":"2024-04-16T16:23:05.789925+0000","last_clean":"2024-04-16T16:23:05.789925+0000","last_became_active":"2024-04-16T16:22:45.462154+0000","last_became_peered":"2024-04-16T16:22:45.462154+0000","last_unstale":"2024-04-16T16:23:05.789925+0000","last_undegraded":"2024-04-16T16:23:05.789925+0000","last_fullsized":"2024-04-16T16:23:05.789925+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:00.114332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:00.114332+0000","last_clean_scrub_stamp":"2024-04-16T16:23:00.114332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:01.971568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.802456+0000","last_change":"2024-04-16T16:23:04.705137+0000","last_active":"2024-04-16T16:23:05.802456+0000","last_peered":"2024-04-16T16:23:05.802456+0000","last_clean":"2024-04-16T16:23:05.802456+0000","last_became_active":"2024-04-16T16:22:46.466890+0000","last_became_peered":"2024-04-16T16:22:46.466890+0000","last_unstale":"2024-04-16T16:23:05.802456+0000","last_undegraded":"2024-04-16T16:23:05.802456+0000","last_fullsized":"2024-04-16T16:23:05.802456+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:04.704892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:04.704892+0000","last_clean_scrub_stamp":"2024-04-16T16:23:04.704892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:03:50.368534+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.796917+0000","last_change":"2024-04-16T16:23:03.967382+0000","last_active":"2024-04-16T16:23:05.796917+0000","last_peered":"2024-04-16T16:23:05.796917+0000","last_clean":"2024-04-16T16:23:05.796917+0000","last_became_active":"2024-04-16T16:22:45.469350+0000","last_became_peered":"2024-04-16T16:22:45.469350+0000","last_unstale":"2024-04-16T16:23:05.796917+0000","last_undegraded":"2024-04-16T16:23:05.796917+0000","last_fullsized":"2024-04-16T16:23:05.796917+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:03.966961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:03.966961+0000","last_clean_scrub_stamp":"2024-04-16T16:23:03.966961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:54:50.030243+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:07.074084+0000","last_change":"2024-04-16T16:23:07.074136+0000","last_active":"2024-04-16T16:23:07.074084+0000","last_peered":"2024-04-16T16:23:07.074084+0000","last_clean":"2024-04-16T16:23:07.074084+0000","last_became_active":"2024-04-16T16:22:46.478459+0000","last_became_peered":"2024-04-16T16:22:46.478459+0000","last_unstale":"2024-04-16T16:23:07.074084+0000","last_undegraded":"2024-04-16T16:23:07.074084+0000","last_fullsized":"2024-04-16T16:23:07.074084+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:07.073883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:07.073883+0000","last_clean_scrub_stamp":"2024-04-16T16:23:07.073883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T17:27:00.048146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:06.142770+0000","last_change":"2024-04-16T16:23:06.142826+0000","last_active":"2024-04-16T16:23:06.142770+0000","last_peered":"2024-04-16T16:23:06.142770+0000","last_clean":"2024-04-16T16:23:06.142770+0000","last_became_active":"2024-04-16T16:22:45.465339+0000","last_became_peered":"2024-04-16T16:22:45.465339+0000","last_unstale":"2024-04-16T16:23:06.142770+0000","last_undegraded":"2024-04-16T16:23:06.142770+0000","last_fullsized":"2024-04-16T16:23:06.142770+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:06.142568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:06.142568+0000","last_clean_scrub_stamp":"2024-04-16T16:23:06.142568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:54:08.893687+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:06.451289+0000","last_change":"2024-04-16T16:23:06.451343+0000","last_active":"2024-04-16T16:23:06.451289+0000","last_peered":"2024-04-16T16:23:06.451289+0000","last_clean":"2024-04-16T16:23:06.451289+0000","last_became_active":"2024-04-16T16:22:46.464436+0000","last_became_peered":"2024-04-16T16:22:46.464436+0000","last_unstale":"2024-04-16T16:23:06.451289+0000","last_undegraded":"2024-04-16T16:23:06.451289+0000","last_fullsized":"2024-04-16T16:23:06.451289+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:06.451090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:06.451090+0000","last_clean_scrub_stamp":"2024-04-16T16:23:06.451090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T18:07:42.779049+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":25,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.779204+0000","last_change":"2024-04-16T16:23:03.249042+0000","last_active":"2024-04-16T16:23:05.779204+0000","last_peered":"2024-04-16T16:23:05.779204+0000","last_clean":"2024-04-16T16:23:05.779204+0000","last_became_active":"2024-04-16T16:22:45.460350+0000","last_became_peered":"2024-04-16T16:22:45.460350+0000","last_unstale":"2024-04-16T16:23:05.779204+0000","last_undegraded":"2024-04-16T16:23:05.779204+0000","last_fullsized":"2024-04-16T16:23:05.779204+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T16:23:03.248789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T16:23:03.248789+0000","last_clean_scrub_stamp":"2024-04-16T16:23:03.248789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:05.106025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"156'4693","reported_seq":5403,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-16T16:23:05.779900+0000","last_change":"2024-04-16T16:23:04.231340+0000","last_active":"2024-04-16T16:23:05.779900+0000","last_peered":"2024-04-16T16:23:05.779900+0000","last_clean":"2024-04-16T16:23:05.779900+0000","last_became_active":"2024-04-16T16:13:20.914337+0000","last_became_peered":"2024-04-16T16:13:20.914337+0000","last_unstale":"2024-04-16T16:23:05.779900+0000","last_undegraded":"2024-04-16T16:23:05.779900+0000","last_fullsized":"2024-04-16T16:23:05.779900+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'4693","last_scrub_stamp":"2024-04-16T16:23:04.231043+0000","last_deep_scrub":"156'4693","last_deep_scrub_stamp":"2024-04-16T16:23:04.231043+0000","last_clean_scrub_stamp":"2024-04-16T16:23:04.231043+0000","objects_scrubbed":2,"log_size":4693,"log_dups_size":0,"ondisk_log_size":4693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:20:02.128302+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":387,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":77,"num_read_kb":80,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":2525,"num_write":387,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4693,"ondisk_log_size":4693,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574963,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37864,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333976,"statfs":{"total":96636764160,"available":96597991424,"internally_reserved":0,"allocated":2588672,"data_stored":1532301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574964,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33104,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338736,"statfs":{"total":96636764160,"available":96602865664,"internally_reserved":0,"allocated":1974272,"data_stored":939601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574964,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33108,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338732,"statfs":{"total":96636764160,"available":96602861568,"internally_reserved":0,"allocated":1978368,"data_stored":940183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574965,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33116,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338724,"statfs":{"total":96636764160,"available":96602853376,"internally_reserved":0,"allocated":1986560,"data_stored":941261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574963,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37264,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334576,"statfs":{"total":96636764160,"available":96598605824,"internally_reserved":0,"allocated":1974272,"data_stored":939613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574963,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37264,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334576,"statfs":{"total":96636764160,"available":96598605824,"internally_reserved":0,"allocated":1974272,"data_stored":939669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37264,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334576,"statfs":{"total":96636764160,"available":96598605824,"internally_reserved":0,"allocated":1974272,"data_stored":939613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574964,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37840,"kb_used_data":2504,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334000,"statfs":{"total":96636764160,"available":96598016000,"internally_reserved":0,"allocated":2564096,"data_stored":1529947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T16:23:24.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-16T16:23:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.850+0000 7f4d6a600640 1 Processor -- start 2024-04-16T16:23:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.850+0000 7f4d6a600640 1 -- start start 2024-04-16T16:23:24.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.852+0000 7f4d6a600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d64160e90 0x7f4d640cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:24.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.852+0000 7f4d6a600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4d640cf980 0x7f4d6406dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:24.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.852+0000 7f4d6a600640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d640c6210 con 0x7f4d64145fe0 2024-04-16T16:23:24.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.852+0000 7f4d6a600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4d640cfdd0 con 0x7f4d640cf980 2024-04-16T16:23:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.852+0000 7f4d6a600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4d641603b0 con 0x7f4d64160e90 2024-04-16T16:23:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.853+0000 7f4d69200640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4d64145fe0 0x7f4d640c5750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60664/0 (socket says 172.21.15.45:60664) 2024-04-16T16:23:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.853+0000 7f4d69200640 1 -- 172.21.15.45:0/4271512719 learned_addr learned my addr 172.21.15.45:0/4271512719 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:23:24.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.853+0000 7f4d63e00640 1 --2- 172.21.15.45:0/4271512719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d64160e90 0x7f4d640cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.853+0000 7f4d63400640 1 -- 172.21.15.45:0/4271512719 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3636191889 0 0) 0x7f4d640c6210 con 0x7f4d64145fe0 2024-04-16T16:23:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.853+0000 7f4d69c00640 1 --2- 172.21.15.45:0/4271512719 >> v2:172.21.15.103:3300/0 conn(0x7f4d640cf980 0x7f4d6406dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.853+0000 7f4d63400640 1 -- 172.21.15.45:0/4271512719 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d4c003520 con 0x7f4d64145fe0 2024-04-16T16:23:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.853+0000 7f4d63400640 1 -- 172.21.15.45:0/4271512719 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2734836506 0 0) 0x7f4d4c003520 con 0x7f4d64145fe0 2024-04-16T16:23:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d63400640 1 -- 172.21.15.45:0/4271512719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d64160e90 msgr2=0x7f4d640cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d63400640 1 --2- 172.21.15.45:0/4271512719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d64160e90 0x7f4d640cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d63400640 1 -- 172.21.15.45:0/4271512719 >> v2:172.21.15.103:3300/0 conn(0x7f4d640cf980 msgr2=0x7f4d6406dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d63400640 1 --2- 172.21.15.45:0/4271512719 >> v2:172.21.15.103:3300/0 conn(0x7f4d640cf980 0x7f4d6406dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d63400640 1 -- 172.21.15.45:0/4271512719 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d64160690 con 0x7f4d64145fe0 2024-04-16T16:23:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d63400640 1 -- 172.21.15.45:0/4271512719 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7f4d58001670 con 0x7f4d64145fe0 2024-04-16T16:23:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d63400640 1 -- 172.21.15.45:0/4271512719 >> v1:172.21.15.45:6789/0 conn(0x7f4d64145fe0 legacy=0x7f4d640c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d63400640 1 --2- 172.21.15.45:0/4271512719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d64160e90 0x7f4d4c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d63400640 1 --2- 172.21.15.45:0/4271512719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d4c003f30 0x7f4d4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d63400640 1 --2- 172.21.15.45:0/4271512719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d4c004d40 0x7f4d4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d63400640 1 -- 172.21.15.45:0/4271512719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4d640cfdd0 con 0x7f4d4c004d40 2024-04-16T16:23:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d63400640 1 -- 172.21.15.45:0/4271512719 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4d641603b0 con 0x7f4d4c003f30 2024-04-16T16:23:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d63400640 1 -- 172.21.15.45:0/4271512719 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4d4c00ba10 con 0x7f4d64160e90 2024-04-16T16:23:24.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d69c00640 1 --2- 172.21.15.45:0/4271512719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d4c004d40 0x7f4d4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:24.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d69c00640 1 -- 172.21.15.45:0/4271512719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d64160e90 msgr2=0x7f4d4c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:24.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d69c00640 1 --2- 172.21.15.45:0/4271512719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d64160e90 0x7f4d4c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d69c00640 1 -- 172.21.15.45:0/4271512719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d4c003f30 msgr2=0x7f4d4c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:24.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.854+0000 7f4d69c00640 1 --2- 172.21.15.45:0/4271512719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d4c003f30 0x7f4d4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.855+0000 7f4d69c00640 1 -- 172.21.15.45:0/4271512719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d4c003520 con 0x7f4d4c004d40 2024-04-16T16:23:24.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.856+0000 7f4d69c00640 1 --2- 172.21.15.45:0/4271512719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d4c004d40 0x7f4d4c0051c0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c00e170 tx=0x7f4d4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d01ff0d084a556 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:24.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.856+0000 7f4d63400640 1 -- 172.21.15.45:0/4271512719 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c039070 con 0x7f4d4c004d40 2024-04-16T16:23:24.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.856+0000 7f4d63400640 1 -- 172.21.15.45:0/4271512719 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d5c02cc50 con 0x7f4d4c004d40 2024-04-16T16:23:24.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.856+0000 7f4d63400640 1 -- 172.21.15.45:0/4271512719 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c009c10 con 0x7f4d4c004d40 2024-04-16T16:23:24.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.856+0000 7f4d6a600640 1 -- 172.21.15.45:0/4271512719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d4c004d40 msgr2=0x7f4d4c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:24.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.856+0000 7f4d6a600640 1 --2- 172.21.15.45:0/4271512719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d4c004d40 0x7f4d4c0051c0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c00e170 tx=0x7f4d4c005830 comp rx=0 tx=0).stop 2024-04-16T16:23:24.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.857+0000 7f4d69c00640 1 -- 172.21.15.45:0/4271512719 reap_dead start 2024-04-16T16:23:24.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.857+0000 7f4d6a600640 1 -- 172.21.15.45:0/4271512719 shutdown_connections 2024-04-16T16:23:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.857+0000 7f4d6a600640 1 -- 172.21.15.45:0/4271512719 >> 172.21.15.45:0/4271512719 conn(0x7f4d64067e00 msgr2=0x7f4d6406b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.857+0000 7f4d6a600640 1 -- 172.21.15.45:0/4271512719 shutdown_connections 2024-04-16T16:23:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.857+0000 7f4d6a600640 1 -- 172.21.15.45:0/4271512719 wait complete. 2024-04-16T16:23:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.859+0000 7f4d6a600640 1 Processor -- start 2024-04-16T16:23:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.859+0000 7f4d6a600640 1 -- start start 2024-04-16T16:23:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.860+0000 7f4d6a600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d4c004d40 0x7f4d640c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.860+0000 7f4d6a600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d64156f60 0x7f4d641f5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.860+0000 7f4d6a600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d64142980 0x7f4d64142e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.860+0000 7f4d6a600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4d640c5e60 con 0x7f4d64142980 2024-04-16T16:23:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.860+0000 7f4d6a600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6415dae0 con 0x7f4d4c004d40 2024-04-16T16:23:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.860+0000 7f4d6a600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4d640c9370 con 0x7f4d64156f60 2024-04-16T16:23:24.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.860+0000 7f4d69c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d64142980 0x7f4d64142e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:24.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.860+0000 7f4d69c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d64142980 0x7f4d64142e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33242/0 (socket says 172.21.15.45:33242) 2024-04-16T16:23:24.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.860+0000 7f4d69c00640 1 -- 172.21.15.45:0/2599512377 learned_addr learned my addr 172.21.15.45:0/2599512377 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:24.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.860+0000 7f4d69200640 1 --2- 172.21.15.45:0/2599512377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d4c004d40 0x7f4d640c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:24.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.860+0000 7f4d63e00640 1 --2- 172.21.15.45:0/2599512377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d64156f60 0x7f4d641f5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:24.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.861+0000 7f4d69c00640 1 -- 172.21.15.45:0/2599512377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d64156f60 msgr2=0x7f4d641f5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:24.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.861+0000 7f4d69c00640 1 --2- 172.21.15.45:0/2599512377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d64156f60 0x7f4d641f5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.861+0000 7f4d69c00640 1 -- 172.21.15.45:0/2599512377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d4c004d40 msgr2=0x7f4d640c8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:24.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.861+0000 7f4d69c00640 1 --2- 172.21.15.45:0/2599512377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d4c004d40 0x7f4d640c8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:24.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.861+0000 7f4d69c00640 1 -- 172.21.15.45:0/2599512377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d64157900 con 0x7f4d64142980 2024-04-16T16:23:24.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.861+0000 7f4d69c00640 1 --2- 172.21.15.45:0/2599512377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d64142980 0x7f4d64142e00 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c00e170 tx=0x7f4d5c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:24.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.861+0000 7f4d60c00640 1 -- 172.21.15.45:0/2599512377 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c039030 con 0x7f4d64142980 2024-04-16T16:23:24.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.861+0000 7f4d60c00640 1 -- 172.21.15.45:0/2599512377 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d5c03a040 con 0x7f4d64142980 2024-04-16T16:23:24.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.861+0000 7f4d60c00640 1 -- 172.21.15.45:0/2599512377 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c02ca50 con 0x7f4d64142980 2024-04-16T16:23:24.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.861+0000 7f4d6a600640 1 -- 172.21.15.45:0/2599512377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d64157be0 con 0x7f4d64142980 2024-04-16T16:23:24.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.862+0000 7f4d6a600640 1 -- 172.21.15.45:0/2599512377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d640c8630 con 0x7f4d64142980 2024-04-16T16:23:24.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.863+0000 7f4d6a600640 1 -- 172.21.15.45:0/2599512377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d64158100 con 0x7f4d64142980 2024-04-16T16:23:24.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.864+0000 7f4d60c00640 1 -- 172.21.15.45:0/2599512377 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7f4d5c045030 con 0x7f4d64142980 2024-04-16T16:23:24.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.864+0000 7f4d60c00640 1 --2- 172.21.15.45:0/2599512377 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f4d4406ca10 0x7f4d4406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:24.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.865+0000 7f4d60c00640 1 -- 172.21.15.45:0/2599512377 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5795+0+0 (secure 0 0 0) 0x7f4d5c0ad870 con 0x7f4d64142980 2024-04-16T16:23:24.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.865+0000 7f4d69200640 1 --2- 172.21.15.45:0/2599512377 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f4d4406ca10 0x7f4d4406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:24.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.867+0000 7f4d69200640 1 --2- 172.21.15.45:0/2599512377 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f4d4406ca10 0x7f4d4406eed0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4d58001050 tx=0x7f4d58001780 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:24.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:24.873+0000 7f4d60c00640 1 -- 172.21.15.45:0/2599512377 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d5c04c060 con 0x7f4d64142980 2024-04-16T16:23:25.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.079+0000 7f4d6a600640 1 -- 172.21.15.45:0/2599512377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f4d640c8420 con 0x7f4d64142980 2024-04-16T16:23:25.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.085+0000 7f4d60c00640 1 -- 172.21.15.45:0/2599512377 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v25)=0 v25) ==== 123+0+0 (secure 0 0 0) 0x7f4d5c07bbc0 con 0x7f4d64142980 2024-04-16T16:23:25.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.087+0000 7f4d60c00640 1 -- 172.21.15.45:0/2599512377 <== mon.0 v2:172.21.15.45:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4d5c009cd0 con 0x7f4d64142980 2024-04-16T16:23:25.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.088+0000 7f4d6a600640 1 -- 172.21.15.45:0/2599512377 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f4d4406ca10 msgr2=0x7f4d4406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:25.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.088+0000 7f4d6a600640 1 --2- 172.21.15.45:0/2599512377 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f4d4406ca10 0x7f4d4406eed0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4d58001050 tx=0x7f4d58001780 comp rx=0 tx=0).stop 2024-04-16T16:23:25.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.088+0000 7f4d6a600640 1 -- 172.21.15.45:0/2599512377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d64142980 msgr2=0x7f4d64142e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:25.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.088+0000 7f4d6a600640 1 --2- 172.21.15.45:0/2599512377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d64142980 0x7f4d64142e00 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c00e170 tx=0x7f4d5c009750 comp rx=0 tx=0).stop 2024-04-16T16:23:25.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.088+0000 7f4d6a600640 1 -- 172.21.15.45:0/2599512377 shutdown_connections 2024-04-16T16:23:25.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.088+0000 7f4d6a600640 1 --2- 172.21.15.45:0/2599512377 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7f4d4406ca10 0x7f4d4406eed0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:25.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.088+0000 7f4d6a600640 1 --2- 172.21.15.45:0/2599512377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d64142980 0x7f4d64142e00 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:25.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.088+0000 7f4d6a600640 1 --2- 172.21.15.45:0/2599512377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d64156f60 0x7f4d641f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:25.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.088+0000 7f4d6a600640 1 --2- 172.21.15.45:0/2599512377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d4c004d40 0x7f4d640c8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:25.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.088+0000 7f4d6a600640 1 -- 172.21.15.45:0/2599512377 >> 172.21.15.45:0/2599512377 conn(0x7f4d64067e00 msgr2=0x7f4d640c5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:25.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.088+0000 7f4d6a600640 1 -- 172.21.15.45:0/2599512377 shutdown_connections 2024-04-16T16:23:25.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.088+0000 7f4d6a600640 1 -- 172.21.15.45:0/2599512377 wait complete. 2024-04-16T16:23:25.106 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-16T16:23:25.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T16:23:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.304+0000 7ffa42a00640 1 Processor -- start 2024-04-16T16:23:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.304+0000 7ffa42a00640 1 -- start start 2024-04-16T16:23:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.305+0000 7ffa42a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa3c0d4560 0x7ffa3c0d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.306+0000 7ffa42a00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa3c0d34c0 con 0x7ffa3c15ec60 2024-04-16T16:23:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.306+0000 7ffa42a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa3c0d37a0 con 0x7ffa3c0d18f0 2024-04-16T16:23:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.306+0000 7ffa42a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffa3c0d3a80 con 0x7ffa3c0d4560 2024-04-16T16:23:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.306+0000 7ffa3be00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ffa3c15ec60 0x7ffa3c0d2a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60674/0 (socket says 172.21.15.45:60674) 2024-04-16T16:23:25.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.306+0000 7ffa3be00640 1 -- 172.21.15.45:0/4092913903 learned_addr learned my addr 172.21.15.45:0/4092913903 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-16T16:23:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.306+0000 7ffa33e00640 1 --2- 172.21.15.45:0/4092913903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa3c0d4560 0x7ffa3c0d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.306+0000 7ffa3b400640 1 -- 172.21.15.45:0/4092913903 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3631535618 0 0) 0x7ffa3c0d34c0 con 0x7ffa3c15ec60 2024-04-16T16:23:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.307+0000 7ffa3b400640 1 -- 172.21.15.45:0/4092913903 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa20003520 con 0x7ffa3c15ec60 2024-04-16T16:23:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.307+0000 7ffa3b400640 1 -- 172.21.15.45:0/4092913903 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 44034400 0 0) 0x7ffa20003520 con 0x7ffa3c15ec60 2024-04-16T16:23:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.307+0000 7ffa3b400640 1 -- 172.21.15.45:0/4092913903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa3c0d4560 msgr2=0x7ffa3c0d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.307+0000 7ffa3b400640 1 --2- 172.21.15.45:0/4092913903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa3c0d4560 0x7ffa3c0d13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.307+0000 7ffa3b400640 1 -- 172.21.15.45:0/4092913903 >> v1:172.21.15.103:6789/0 conn(0x7ffa3c0d18f0 legacy=0x7ffa3c0b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.307+0000 7ffa3b400640 1 -- 172.21.15.45:0/4092913903 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3c0d3d60 con 0x7ffa3c15ec60 2024-04-16T16:23:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.307+0000 7ffa3b400640 1 -- 172.21.15.45:0/4092913903 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1103467608 0 0) 0x7ffa2c001670 con 0x7ffa3c15ec60 2024-04-16T16:23:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.307+0000 7ffa3b400640 1 -- 172.21.15.45:0/4092913903 >> v1:172.21.15.45:6789/0 conn(0x7ffa3c15ec60 legacy=0x7ffa3c0d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.307+0000 7ffa3b400640 1 --2- 172.21.15.45:0/4092913903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa200039e0 0x7ffa20003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.307+0000 7ffa3b400640 1 --2- 172.21.15.45:0/4092913903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa20004800 0x7ffa20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.307+0000 7ffa3b400640 1 --2- 172.21.15.45:0/4092913903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa3c0d4560 0x7ffa200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.307+0000 7ffa3b400640 1 -- 172.21.15.45:0/4092913903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c0d3a80 con 0x7ffa200039e0 2024-04-16T16:23:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.307+0000 7ffa3b400640 1 -- 172.21.15.45:0/4092913903 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2000baa0 con 0x7ffa20004800 2024-04-16T16:23:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.307+0000 7ffa3b400640 1 -- 172.21.15.45:0/4092913903 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffa2000bca0 con 0x7ffa3c0d4560 2024-04-16T16:23:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.307+0000 7ffa3be00640 1 --2- 172.21.15.45:0/4092913903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa200039e0 0x7ffa20003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.308+0000 7ffa33e00640 1 --2- 172.21.15.45:0/4092913903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa20004800 0x7ffa20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.308+0000 7ffa40a00640 1 --2- 172.21.15.45:0/4092913903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa3c0d4560 0x7ffa200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.308+0000 7ffa3be00640 1 -- 172.21.15.45:0/4092913903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa3c0d4560 msgr2=0x7ffa200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.308+0000 7ffa3be00640 1 --2- 172.21.15.45:0/4092913903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa3c0d4560 0x7ffa200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.308+0000 7ffa3be00640 1 -- 172.21.15.45:0/4092913903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa20004800 msgr2=0x7ffa20004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T16:23:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.308+0000 7ffa3be00640 1 --2- 172.21.15.45:0/4092913903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa20004800 0x7ffa20004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.308+0000 7ffa3be00640 1 -- 172.21.15.45:0/4092913903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa20003520 con 0x7ffa200039e0 2024-04-16T16:23:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.309+0000 7ffa3be00640 1 --2- 172.21.15.45:0/4092913903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa200039e0 0x7ffa20003e00 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00c210 tx=0x7ffa20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f74b429afefaf30 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.309+0000 7ffa3b400640 1 -- 172.21.15.45:0/4092913903 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c007cf0 con 0x7ffa200039e0 2024-04-16T16:23:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.310+0000 7ffa3b400640 1 -- 172.21.15.45:0/4092913903 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ffa2c030070 con 0x7ffa200039e0 2024-04-16T16:23:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.310+0000 7ffa3b400640 1 -- 172.21.15.45:0/4092913903 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c005d50 con 0x7ffa200039e0 2024-04-16T16:23:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.310+0000 7ffa42a00640 1 -- 172.21.15.45:0/4092913903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa200039e0 msgr2=0x7ffa20003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.310+0000 7ffa42a00640 1 --2- 172.21.15.45:0/4092913903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa200039e0 0x7ffa20003e00 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00c210 tx=0x7ffa20005830 comp rx=0 tx=0).stop 2024-04-16T16:23:25.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.310+0000 7ffa40a00640 1 -- 172.21.15.45:0/4092913903 reap_dead start 2024-04-16T16:23:25.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.310+0000 7ffa42a00640 1 -- 172.21.15.45:0/4092913903 shutdown_connections 2024-04-16T16:23:25.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.310+0000 7ffa42a00640 1 -- 172.21.15.45:0/4092913903 >> 172.21.15.45:0/4092913903 conn(0x7ffa3c073690 msgr2=0x7ffa3c06cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:25.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.310+0000 7ffa42a00640 1 -- 172.21.15.45:0/4092913903 shutdown_connections 2024-04-16T16:23:25.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.310+0000 7ffa42a00640 1 -- 172.21.15.45:0/4092913903 wait complete. 2024-04-16T16:23:25.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.312+0000 7ffa42a00640 1 Processor -- start 2024-04-16T16:23:25.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.312+0000 7ffa42a00640 1 -- start start 2024-04-16T16:23:25.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.313+0000 7ffa42a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa200039e0 0x7ffa3c1510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:25.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.313+0000 7ffa42a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa3c1f5ab0 0x7ffa3c1f8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:25.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.313+0000 7ffa42a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa3c1f8bd0 0x7ffa3c1561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:25.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.313+0000 7ffa42a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c0d3110 con 0x7ffa3c1f5ab0 2024-04-16T16:23:25.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.313+0000 7ffa42a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c0ba460 con 0x7ffa200039e0 2024-04-16T16:23:25.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.313+0000 7ffa42a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffa3c151610 con 0x7ffa3c1f8bd0 2024-04-16T16:23:25.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.313+0000 7ffa33e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa3c1f5ab0 0x7ffa3c1f8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:25.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.313+0000 7ffa33e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa3c1f5ab0 0x7ffa3c1f8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33254/0 (socket says 172.21.15.45:33254) 2024-04-16T16:23:25.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.314+0000 7ffa33e00640 1 -- 172.21.15.45:0/126199328 learned_addr learned my addr 172.21.15.45:0/126199328 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T16:23:25.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.314+0000 7ffa3be00640 1 --2- 172.21.15.45:0/126199328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa200039e0 0x7ffa3c1510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:25.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.314+0000 7ffa40a00640 1 --2- 172.21.15.45:0/126199328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa3c1f8bd0 0x7ffa3c1561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:25.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.314+0000 7ffa33e00640 1 -- 172.21.15.45:0/126199328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa3c1f8bd0 msgr2=0x7ffa3c1561d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:25.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.314+0000 7ffa33e00640 1 --2- 172.21.15.45:0/126199328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa3c1f8bd0 0x7ffa3c1561d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:25.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.314+0000 7ffa33e00640 1 -- 172.21.15.45:0/126199328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa200039e0 msgr2=0x7ffa3c1510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:25.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.314+0000 7ffa33e00640 1 --2- 172.21.15.45:0/126199328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa200039e0 0x7ffa3c1510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:25.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.314+0000 7ffa33e00640 1 -- 172.21.15.45:0/126199328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3c1f9070 con 0x7ffa3c1f5ab0 2024-04-16T16:23:25.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.314+0000 7ffa3be00640 1 --2- 172.21.15.45:0/126199328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa200039e0 0x7ffa3c1510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T16:23:25.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.314+0000 7ffa33e00640 1 --2- 172.21.15.45:0/126199328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa3c1f5ab0 0x7ffa3c1f8690 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7ffa24011560 tx=0x7ffa24011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:25.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.315+0000 7ffa38c00640 1 -- 172.21.15.45:0/126199328 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24004220 con 0x7ffa3c1f5ab0 2024-04-16T16:23:25.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.315+0000 7ffa38c00640 1 -- 172.21.15.45:0/126199328 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ffa240043c0 con 0x7ffa3c1f5ab0 2024-04-16T16:23:25.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.315+0000 7ffa38c00640 1 -- 172.21.15.45:0/126199328 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24015890 con 0x7ffa3c1f5ab0 2024-04-16T16:23:25.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.315+0000 7ffa42a00640 1 -- 172.21.15.45:0/126199328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa3c1f9350 con 0x7ffa3c1f5ab0 2024-04-16T16:23:25.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.315+0000 7ffa42a00640 1 -- 172.21.15.45:0/126199328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa3c1f5f30 con 0x7ffa3c1f5ab0 2024-04-16T16:23:25.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.316+0000 7ffa42a00640 1 -- 172.21.15.45:0/126199328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa3c075780 con 0x7ffa3c1f5ab0 2024-04-16T16:23:25.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.317+0000 7ffa38c00640 1 -- 172.21.15.45:0/126199328 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) ==== 91142+0+0 (secure 0 0 0) 0x7ffa24015a30 con 0x7ffa3c1f5ab0 2024-04-16T16:23:25.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.318+0000 7ffa38c00640 1 --2- 172.21.15.45:0/126199328 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ffa1806ca10 0x7ffa1806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T16:23:25.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.319+0000 7ffa38c00640 1 -- 172.21.15.45:0/126199328 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5795+0+0 (secure 0 0 0) 0x7ffa2408c3b0 con 0x7ffa3c1f5ab0 2024-04-16T16:23:25.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.319+0000 7ffa3be00640 1 --2- 172.21.15.45:0/126199328 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ffa1806ca10 0x7ffa1806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T16:23:25.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.320+0000 7ffa3be00640 1 --2- 172.21.15.45:0/126199328 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ffa1806ca10 0x7ffa1806eed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c034000 tx=0x7ffa2c007000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T16:23:25.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.323+0000 7ffa38c00640 1 -- 172.21.15.45:0/126199328 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffa24019030 con 0x7ffa3c1f5ab0 2024-04-16T16:23:25.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.572+0000 7ffa42a00640 1 -- 172.21.15.45:0/126199328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa3c0cc960 con 0x7ffa3c1f5ab0 2024-04-16T16:23:25.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.573+0000 7ffa38c00640 1 -- 172.21.15.45:0/126199328 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+4371 (secure 0 0 0) 0x7ffa2405a5d0 con 0x7ffa3c1f5ab0 2024-04-16T16:23:25.573 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T16:23:25.574 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":118,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6896,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2328390625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2328390625},{"type":"v1","addr":"172.21.15.103:6835","nonce":2328390625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":109},{"gid":7756,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/677052950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":677052950},{"type":"v1","addr":"172.21.15.45:6834","nonce":677052950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T16:22:45.927071+0000","modified":"2024-04-16T16:23:08.785049+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7765},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7765":{"gid":7765,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":10,"addr":"172.21.15.103:6837/3875926287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3875926287},{"type":"v1","addr":"172.21.15.103:6837","nonce":3875926287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7765,"qdb_cluster":[7765]},"id":11}]} 2024-04-16T16:23:25.574 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 118 2024-04-16T16:23:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.576+0000 7ffa42a00640 1 -- 172.21.15.45:0/126199328 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ffa1806ca10 msgr2=0x7ffa1806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:25.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.576+0000 7ffa42a00640 1 --2- 172.21.15.45:0/126199328 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ffa1806ca10 0x7ffa1806eed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c034000 tx=0x7ffa2c007000 comp rx=0 tx=0).stop 2024-04-16T16:23:25.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.576+0000 7ffa42a00640 1 -- 172.21.15.45:0/126199328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa3c1f5ab0 msgr2=0x7ffa3c1f8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T16:23:25.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.576+0000 7ffa42a00640 1 --2- 172.21.15.45:0/126199328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa3c1f5ab0 0x7ffa3c1f8690 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7ffa24011560 tx=0x7ffa24011a20 comp rx=0 tx=0).stop 2024-04-16T16:23:25.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.576+0000 7ffa42a00640 1 -- 172.21.15.45:0/126199328 shutdown_connections 2024-04-16T16:23:25.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.576+0000 7ffa42a00640 1 --2- 172.21.15.45:0/126199328 >> [v2:172.21.15.103:6832/1129061436,v1:172.21.15.103:6833/1129061436] conn(0x7ffa1806ca10 0x7ffa1806eed0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:25.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.576+0000 7ffa42a00640 1 --2- 172.21.15.45:0/126199328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa3c1f8bd0 0x7ffa3c1561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:25.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.576+0000 7ffa42a00640 1 --2- 172.21.15.45:0/126199328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa3c1f5ab0 0x7ffa3c1f8690 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:25.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.576+0000 7ffa42a00640 1 --2- 172.21.15.45:0/126199328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa200039e0 0x7ffa3c1510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T16:23:25.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.576+0000 7ffa42a00640 1 -- 172.21.15.45:0/126199328 >> 172.21.15.45:0/126199328 conn(0x7ffa3c073690 msgr2=0x7ffa3c15b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T16:23:25.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.576+0000 7ffa42a00640 1 -- 172.21.15.45:0/126199328 shutdown_connections 2024-04-16T16:23:25.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T16:23:25.576+0000 7ffa42a00640 1 -- 172.21.15.45:0/126199328 wait complete. 2024-04-16T16:23:25.594 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-16T16:23:25.594 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-16T16:23:25.595 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-16T16:23:25.595 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:25.611 INFO:tasks.ceph.mds.a:Stopped 2024-04-16T16:23:25.612 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-16T16:23:25.612 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:25.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T16:23:25.612 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c' 2024-04-16T16:23:25.620 INFO:tasks.ceph.mds.c:Stopped 2024-04-16T16:23:25.621 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-16T16:23:25.621 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:25.647 INFO:tasks.ceph.mds.b:Stopped 2024-04-16T16:23:25.647 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-16T16:23:25.647 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:25.708 INFO:tasks.ceph.mds.d:Stopped 2024-04-16T16:23:25.708 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-16T16:23:25.709 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-16T16:23:25.709 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:25.789 INFO:tasks.ceph.osd.0:Stopped 2024-04-16T16:23:25.789 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-16T16:23:25.790 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:25.869 INFO:tasks.ceph.osd.1:Stopped 2024-04-16T16:23:25.869 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-16T16:23:25.870 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:25.935 INFO:tasks.ceph.osd.2:Stopped 2024-04-16T16:23:25.935 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-16T16:23:25.935 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:26.021 INFO:tasks.ceph.osd.3:Stopped 2024-04-16T16:23:26.021 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-16T16:23:26.021 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:26.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:23:26.045 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:23:26.082 INFO:tasks.ceph.osd.4:Stopped 2024-04-16T16:23:26.083 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-16T16:23:26.083 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:28.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-04-16T16:23:28.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-04-16T16:23:32.185 INFO:tasks.ceph.osd.5:Stopped 2024-04-16T16:23:32.185 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-16T16:23:32.185 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:32.233 INFO:tasks.ceph.osd.6:Stopped 2024-04-16T16:23:32.233 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-16T16:23:32.233 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:32.313 INFO:tasks.ceph.osd.7:Stopped 2024-04-16T16:23:32.313 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-16T16:23:32.313 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-16T16:23:32.313 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:33.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-04-16T16:23:33.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-04-16T16:23:38.415 INFO:tasks.ceph.mgr.y:Stopped 2024-04-16T16:23:38.416 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-16T16:23:38.416 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:38.506 INFO:tasks.ceph.mgr.x:Stopped 2024-04-16T16:23:38.506 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-16T16:23:38.507 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-16T16:23:38.507 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:38.566 INFO:tasks.ceph.mon.a:Stopped 2024-04-16T16:23:38.566 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-16T16:23:38.566 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:38.603 INFO:tasks.ceph.mon.b:Stopped 2024-04-16T16:23:38.603 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-16T16:23:38.603 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T16:23:38.672 INFO:tasks.ceph.mon.c:Stopped 2024-04-16T16:23:38.672 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-16T16:23:38.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-16T16:23:38.673 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-04-16T16:23:38.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2024-04-16T16:23:38.789 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:23:38.789 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-16T16:23:39.032 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:23:39.032 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-16T16:23:39.193 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:23:39.193 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-16T16:23:39.346 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:23:39.347 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-16T16:23:39.541 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:23:39.542 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-16T16:23:39.784 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:23:39.785 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-16T16:23:39.922 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:23:39.923 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-16T16:23:40.095 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:23:40.095 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-16T16:23:40.276 INFO:tasks.ceph:Archiving mon data... 2024-04-16T16:23:40.277 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211/data/mon.a.tgz 2024-04-16T16:23:40.278 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-04-16T16:23:40.339 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.QLipXsq860 2024-04-16T16:23:40.340 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.QLipXsq860 2024-04-16T16:23:40.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.QLipXsq860 2024-04-16T16:23:41.051 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.QLipXsq860 is 3MB 2024-04-16T16:23:41.207 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.QLipXsq860 2024-04-16T16:23:41.267 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211/data/mon.b.tgz 2024-04-16T16:23:41.267 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-04-16T16:23:41.323 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.GhoOUITMg3 2024-04-16T16:23:41.324 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.GhoOUITMg3 2024-04-16T16:23:41.868 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.GhoOUITMg3 2024-04-16T16:23:42.010 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.GhoOUITMg3 is 3MB 2024-04-16T16:23:42.155 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.GhoOUITMg3 2024-04-16T16:23:42.213 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211/data/mon.c.tgz 2024-04-16T16:23:42.213 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-04-16T16:23:42.266 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.8TTacshZsB 2024-04-16T16:23:42.266 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.8TTacshZsB 2024-04-16T16:23:42.790 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.8TTacshZsB 2024-04-16T16:23:42.937 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.8TTacshZsB is 3MB 2024-04-16T16:23:43.086 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.8TTacshZsB 2024-04-16T16:23:43.146 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-16T16:23:43.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T16:23:43.154 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T16:23:43.307 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T16:23:43.365 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T16:23:43.366 INFO:tasks.ceph:Archiving crash dumps... 2024-04-16T16:23:43.367 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211/remote/smithi045/crash 2024-04-16T16:23:43.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T16:23:43.484 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211/remote/smithi103/crash 2024-04-16T16:23:43.485 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T16:23:43.580 INFO:tasks.ceph:Shutting down logrotate 2024-04-16T16:23:43.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:23:43.588 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-16T16:23:43.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-04-16T16:23:43.714 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-16T16:23:43.817 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-16T16:23:43.914 INFO:tasks.ceph:Compressing logs... 2024-04-16T16:23:43.914 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T16:23:43.922 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T16:23:44.011 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-16T16:23:44.012 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36993.log 2024-04-16T16:23:44.013 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44734.log 2024-04-16T16:23:44.015 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55831.log 2024-04-16T16:23:44.015 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.36993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36833.log 2024-04-16T16:23:44.015 INFO:teuthology.orchestra.run.smithi103.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36993.log.gz 2024-04-16T16:23:44.016 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61801.log 2024-04-16T16:23:44.016 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.1.44734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.44734.log.gz 2024-04-16T16:23:44.017 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-16T16:23:44.018 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.36833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45576.log 2024-04-16T16:23:44.018 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36833.log.gz 2024-04-16T16:23:44.018 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-16T16:23:44.019 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55831.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55831.log.gz 2024-04-16T16:23:44.019 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71342.log 2024-04-16T16:23:44.019 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67486.log 2024-04-16T16:23:44.019 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61801.log.gz 2024-04-16T16:23:44.020 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49444.log 2024-04-16T16:23:44.020 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71342.log.gz 2024-04-16T16:23:44.021 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.1.45576.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.45576.log.gz 2024-04-16T16:23:44.021 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-16T16:23:44.021 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.47195.log 2024-04-16T16:23:44.021 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.48778.log 2024-04-16T16:23:44.022 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66963.log 2024-04-16T16:23:44.022 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67486.log.gz 2024-04-16T16:23:44.022 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63030.log 2024-04-16T16:23:44.023 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49444.log.gz 2024-04-16T16:23:44.024 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74687.log 2024-04-16T16:23:44.024 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66963.log.gz 2024-04-16T16:23:44.025 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.1.47195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.47195.log.gz 2024-04-16T16:23:44.025 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36793.log 2024-04-16T16:23:44.025 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.1.48778.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-16T16:23:44.025 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.48778.log.gz 2024-04-16T16:23:44.025 INFO:teuthology.orchestra.run.smithi103.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-16T16:23:44.025 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.36793.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-16T16:23:44.026 INFO:teuthology.orchestra.run.smithi103.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36793.log.gz 2024-04-16T16:23:44.026 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.47992.log 2024-04-16T16:23:44.027 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41599.log 2024-04-16T16:23:44.028 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63030.log.gz 2024-04-16T16:23:44.028 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59213.log 2024-04-16T16:23:44.028 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74687.log.gz 2024-04-16T16:23:44.028 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77908.log 2024-04-16T16:23:44.028 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41599.log.gz 2024-04-16T16:23:44.028 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59213.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59213.log.gz 2024-04-16T16:23:44.028 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60960.log 2024-04-16T16:23:44.030 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.77908.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77908.log.gz 2024-04-16T16:23:44.030 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54535.log 2024-04-16T16:23:44.030 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37073.log 2024-04-16T16:23:44.030 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.1.47992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.47992.log.gz 2024-04-16T16:23:44.030 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.49595.log 2024-04-16T16:23:44.031 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37073.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37073.log.gzgzip 2024-04-16T16:23:44.031 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-16T16:23:44.032 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49612.log 2024-04-16T16:23:44.032 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60960.log.gz 2024-04-16T16:23:44.033 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54535.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54535.log.gz 2024-04-16T16:23:44.033 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67869.log 2024-04-16T16:23:44.033 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53254.log 2024-04-16T16:23:44.033 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49612.log.gz 2024-04-16T16:23:44.034 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.1.49595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43031.log 2024-04-16T16:23:44.034 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.49595.log.gz 2024-04-16T16:23:44.034 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-16T16:23:44.035 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51085.log 2024-04-16T16:23:44.035 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67869.log.gz 2024-04-16T16:23:44.035 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53254.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53254.log.gz 2024-04-16T16:23:44.035 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65088.log 2024-04-16T16:23:44.036 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.51085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63247.log 89.2% 2024-04-16T16:23:44.037 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51085.log.gz 2024-04-16T16:23:44.037 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.1.43031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.43031.log.gz 2024-04-16T16:23:44.037 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36913.log 2024-04-16T16:23:44.038 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42486.log 2024-04-16T16:23:44.038 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65088.log.gz 2024-04-16T16:23:44.040 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.36913.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-16T16:23:44.040 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50404.log 2024-04-16T16:23:44.040 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63247.log.gz 2024-04-16T16:23:44.040 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42486.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42486.log.gz 2024-04-16T16:23:44.040 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70735.log 2024-04-16T16:23:44.043 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50404.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-16T16:23:44.043 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50404.log.gz 2024-04-16T16:23:44.043 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41813.log 2024-04-16T16:23:44.043 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70735.log.gz 2024-04-16T16:23:44.043 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45297.log 2024-04-16T16:23:44.044 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36953.log 2024-04-16T16:23:44.044 INFO:teuthology.orchestra.run.smithi103.stderr: 92.3% -- replaced with /var/log/ceph/ceph.log.gz 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36913.log.gz 2024-04-16T16:23:44.044 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-16T16:23:44.044 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.46418.log 2024-04-16T16:23:44.045 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.36953.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36953.log.gz 2024-04-16T16:23:44.046 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55961.log 2024-04-16T16:23:44.046 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41813.log.gz 2024-04-16T16:23:44.046 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45213.log 2024-04-16T16:23:44.046 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45297.log.gz 2024-04-16T16:23:44.047 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49925.log 2024-04-16T16:23:44.047 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55961.log.gz 2024-04-16T16:23:44.048 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.1.46418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.46418.log.gz 2024-04-16T16:23:44.049 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50809.log 2024-04-16T16:23:44.049 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45213.log.gz 2024-04-16T16:23:44.049 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49925.log: gzip -5 --verbose -- 88.7% /var/log/ceph/ceph-client.admin.45618.log -- replaced with /var/log/ceph/ceph-client.admin.49925.log.gz 2024-04-16T16:23:44.049 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-16T16:23:44.050 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-16T16:23:44.051 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60163.log 2024-04-16T16:23:44.051 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50809.log.gz 2024-04-16T16:23:44.051 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37033.log 2024-04-16T16:23:44.052 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45045.log 2024-04-16T16:23:44.052 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45618.log.gz 2024-04-16T16:23:44.053 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60163.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60163.log.gz 2024-04-16T16:23:44.053 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58683.log 2024-04-16T16:23:44.054 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36873.log 2024-04-16T16:23:44.054 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61334.log 2024-04-16T16:23:44.055 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.36873.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36873.log.gz 2024-04-16T16:23:44.056 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44052.log 2024-04-16T16:23:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45045.log: /var/log/ceph/ceph-client.admin.58683.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45045.log.gz 2024-04-16T16:23:44.056 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58683.log.gz 2024-04-16T16:23:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65512.log 2024-04-16T16:23:44.057 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45171.log 2024-04-16T16:23:44.057 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61334.log.gz 2024-04-16T16:23:44.058 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37033.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37033.log.gz 2024-04-16T16:23:44.059 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54073.log 2024-04-16T16:23:44.059 INFO:teuthology.orchestra.run.smithi045.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.65512.log.gz 2024-04-16T16:23:44.059 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78176.log 2024-04-16T16:23:44.060 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45171.log.gz 2024-04-16T16:23:44.060 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54073.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54073.log.gz 2024-04-16T16:23:44.060 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48614.log 2024-04-16T16:23:44.061 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.43446.log 2024-04-16T16:23:44.062 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.65766.log 2024-04-16T16:23:44.062 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78176.log.gz 2024-04-16T16:23:44.062 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.1.44052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.44052.log.gz 2024-04-16T16:23:44.063 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.48614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48614.log.gz 2024-04-16T16:23:44.063 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54703.log 2024-04-16T16:23:44.064 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.65766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.65766.log.gz 2024-04-16T16:23:44.064 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71534.log 2024-04-16T16:23:44.065 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71258.log 2024-04-16T16:23:44.065 INFO:teuthology.orchestra.run.smithi045.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.54703.log.gz 2024-04-16T16:23:44.066 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71534.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71534.log.gz 2024-04-16T16:23:44.067 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74267.log 2024-04-16T16:23:44.067 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.3.43446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.43446.log.gz 2024-04-16T16:23:44.068 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-16T16:23:44.068 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50530.log 2024-04-16T16:23:44.068 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71258.log.gz 2024-04-16T16:23:44.069 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61082.log 2024-04-16T16:23:44.069 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74267.log.gz 2024-04-16T16:23:44.070 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50530.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50530.log.gz 2024-04-16T16:23:44.070 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46453.log 2024-04-16T16:23:44.072 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65037.log 2024-04-16T16:23:44.072 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61082.log.gz 2024-04-16T16:23:44.072 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73255.log 2024-04-16T16:23:44.073 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46453.log.gz 2024-04-16T16:23:44.074 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65037.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65037.log.gz 2024-04-16T16:23:44.074 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67402.log 2024-04-16T16:23:44.075 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69632.log 2024-04-16T16:23:44.075 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.73255.log.gz 2024-04-16T16:23:44.076 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59741.log 2024-04-16T16:23:44.076 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67402.log.gz 2024-04-16T16:23:44.077 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69632.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69632.log.gz 2024-04-16T16:23:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42494.log 2024-04-16T16:23:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64168.log 2024-04-16T16:23:44.079 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59741.log.gz 2024-04-16T16:23:44.080 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42494.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42494.log.gz 2024-04-16T16:23:44.080 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75146.log 2024-04-16T16:23:44.081 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78302.log 2024-04-16T16:23:44.081 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64168.log.gz 2024-04-16T16:23:44.082 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75146.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.75146.log.gz 2024-04-16T16:23:44.082 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51881.log 2024-04-16T16:23:44.083 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50093.log 2024-04-16T16:23:44.083 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78302.log.gz 2024-04-16T16:23:44.084 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.51881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44919.log 2024-04-16T16:23:44.085 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51881.log.gz 2024-04-16T16:23:44.085 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50093.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50093.log.gz 2024-04-16T16:23:44.086 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46080.log 2024-04-16T16:23:44.087 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.44919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60121.log 2024-04-16T16:23:44.087 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44919.log.gz 2024-04-16T16:23:44.087 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61124.log 2024-04-16T16:23:44.088 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46080.log.gz 2024-04-16T16:23:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60121.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60121.log.gz 2024-04-16T16:23:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41819.log 2024-04-16T16:23:44.090 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65971.log 2024-04-16T16:23:44.090 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61124.log.gz 2024-04-16T16:23:44.091 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41819.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41819.log.gz 2024-04-16T16:23:44.091 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59087.log 2024-04-16T16:23:44.092 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63331.log 2024-04-16T16:23:44.092 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65971.log.gz 2024-04-16T16:23:44.093 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49019.log 2024-04-16T16:23:44.094 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59087.log.gz 2024-04-16T16:23:44.094 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63331.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63331.log.gz 2024-04-16T16:23:44.095 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45003.log 2024-04-16T16:23:44.095 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52571.log 2024-04-16T16:23:44.096 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49019.log.gz 2024-04-16T16:23:44.097 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45003.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45003.log.gz 2024-04-16T16:23:44.097 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50851.log 2024-04-16T16:23:44.097 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80423.log 2024-04-16T16:23:44.098 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52571.log.gz 2024-04-16T16:23:44.099 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47293.log 2024-04-16T16:23:44.099 INFO:teuthology.orchestra.run.smithi045.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.50851.log.gz 2024-04-16T16:23:44.100 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71885.log 2024-04-16T16:23:44.100 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80423.log.gz 2024-04-16T16:23:44.101 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69532.log 2024-04-16T16:23:44.101 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47293.log.gz 2024-04-16T16:23:44.102 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71885.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71885.log.gz 2024-04-16T16:23:44.102 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68138.log 2024-04-16T16:23:44.103 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43111.log 2024-04-16T16:23:44.103 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69532.log.gz 2024-04-16T16:23:44.104 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46164.log 2024-04-16T16:23:44.105 INFO:teuthology.orchestra.run.smithi045.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.68138.log.gz 2024-04-16T16:23:44.105 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.43111.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43111.log.gz 2024-04-16T16:23:44.106 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72961.log 2024-04-16T16:23:44.107 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47545.log 2024-04-16T16:23:44.107 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46164.log.gz 2024-04-16T16:23:44.108 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.68747.log 2024-04-16T16:23:44.108 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72961.log.gz 2024-04-16T16:23:44.109 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57061.log 2024-04-16T16:23:44.109 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47545.log.gz 2024-04-16T16:23:44.110 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.68747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.68747.log.gz 2024-04-16T16:23:44.110 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40913.log 2024-04-16T16:23:44.112 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.57061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71927.log 2024-04-16T16:23:44.112 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57061.log.gz 2024-04-16T16:23:44.112 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71131.log 2024-04-16T16:23:44.113 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40913.log.gz 2024-04-16T16:23:44.113 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71927.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71927.log.gz 2024-04-16T16:23:44.114 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69968.log 2024-04-16T16:23:44.115 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63772.log 2024-04-16T16:23:44.115 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71131.log.gz 2024-04-16T16:23:44.116 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61674.log 2024-04-16T16:23:44.116 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69968.log.gz 2024-04-16T16:23:44.117 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63772.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63772.log.gz 2024-04-16T16:23:44.117 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60706.log 2024-04-16T16:23:44.118 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56297.log 2024-04-16T16:23:44.118 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61674.log.gz 2024-04-16T16:23:44.119 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45129.log 2024-04-16T16:23:44.120 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60706.log.gz 2024-04-16T16:23:44.120 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56297.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56297.log.gz 2024-04-16T16:23:44.120 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73610.log 2024-04-16T16:23:44.121 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69437.log 2024-04-16T16:23:44.122 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45129.log.gz 2024-04-16T16:23:44.122 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73610.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.73610.log.gz 2024-04-16T16:23:44.122 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.73045.log 2024-04-16T16:23:44.124 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56381.log 2024-04-16T16:23:44.124 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69437.log.gz 2024-04-16T16:23:44.124 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41557.log 2024-04-16T16:23:44.125 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73045.log.gz 2024-04-16T16:23:44.126 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56381.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56381.log.gz 2024-04-16T16:23:44.126 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53905.log 2024-04-16T16:23:44.127 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67827.log 2024-04-16T16:23:44.127 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41557.log.gz 2024-04-16T16:23:44.128 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65845.log 2024-04-16T16:23:44.128 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53905.log.gz 2024-04-16T16:23:44.129 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66794.log 2024-04-16T16:23:44.129 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67827.log.gz 2024-04-16T16:23:44.130 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47461.log 2024-04-16T16:23:44.131 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.65845.log.gz 2024-04-16T16:23:44.131 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78344.log 2024-04-16T16:23:44.132 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66794.log.gz 2024-04-16T16:23:44.132 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47461.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47461.log.gz 2024-04-16T16:23:44.133 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62238.log 2024-04-16T16:23:44.134 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80339.log 2024-04-16T16:23:44.134 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.78344.log.gz 2024-04-16T16:23:44.135 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41515.log 2024-04-16T16:23:44.135 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62238.log.gz 2024-04-16T16:23:44.136 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80339.log: gzip 88.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.80339.log.gz 2024-04-16T16:23:44.136 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.68912.log 2024-04-16T16:23:44.137 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73213.log 2024-04-16T16:23:44.137 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41515.log.gz 2024-04-16T16:23:44.138 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73003.log 2024-04-16T16:23:44.138 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68912.log.gz 2024-04-16T16:23:44.139 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73213.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73213.log.gz 2024-04-16T16:23:44.140 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69926.log 2024-04-16T16:23:44.141 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53296.log 2024-04-16T16:23:44.141 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73003.log.gz 2024-04-16T16:23:44.142 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54325.log 2024-04-16T16:23:44.142 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69926.log.gz 2024-04-16T16:23:44.143 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80255.log 2024-04-16T16:23:44.143 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53296.log.gz 2024-04-16T16:23:44.144 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48192.log 2024-04-16T16:23:44.145 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54325.log.gz 2024-04-16T16:23:44.145 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45087.log 2024-04-16T16:23:44.146 INFO:teuthology.orchestra.run.smithi045.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80255.log.gz 2024-04-16T16:23:44.146 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48192.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48192.log.gz 2024-04-16T16:23:44.147 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70479.log 2024-04-16T16:23:44.148 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72693.log 2024-04-16T16:23:44.148 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45087.log.gz 2024-04-16T16:23:44.149 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70479.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70479.log.gz 2024-04-16T16:23:44.149 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69191.log 2024-04-16T16:23:44.150 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80507.log 2024-04-16T16:23:44.150 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72693.log.gz 2024-04-16T16:23:44.151 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49780.log 2024-04-16T16:23:44.151 INFO:teuthology.orchestra.run.smithi045.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69191.log.gz 2024-04-16T16:23:44.152 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42492.log 2024-04-16T16:23:44.152 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80507.log.gz 2024-04-16T16:23:44.153 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49780.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.49780.log.gz 2024-04-16T16:23:44.154 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57103.log 2024-04-16T16:23:44.154 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75937.log 2024-04-16T16:23:44.155 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42492.log.gz 2024-04-16T16:23:44.156 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.57103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49181.log 2024-04-16T16:23:44.156 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57103.log.gz 2024-04-16T16:23:44.157 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56816.log 2024-04-16T16:23:44.157 INFO:teuthology.orchestra.run.smithi045.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.75937.log.gz 2024-04-16T16:23:44.158 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52756.log 2024-04-16T16:23:44.158 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49181.log.gz 2024-04-16T16:23:44.159 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56816.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.56816.log.gz 2024-04-16T16:23:44.159 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63730.log 2024-04-16T16:23:44.160 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52756.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52756.log.gz 2024-04-16T16:23:44.160 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68092.log 2024-04-16T16:23:44.162 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52007.log 2024-04-16T16:23:44.162 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63730.log.gz 2024-04-16T16:23:44.163 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68092.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68092.log.gz 2024-04-16T16:23:44.163 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52883.log 2024-04-16T16:23:44.164 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52007.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52007.log.gz 2024-04-16T16:23:44.164 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52091.log 2024-04-16T16:23:44.165 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67276.log 2024-04-16T16:23:44.166 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52883.log.gz 2024-04-16T16:23:44.167 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52091.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52091.log.gz 2024-04-16T16:23:44.167 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53422.log 2024-04-16T16:23:44.167 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67661.log 2024-04-16T16:23:44.168 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67276.log.gz 2024-04-16T16:23:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53422.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53422.log.gzgzip 2024-04-16T16:23:44.169 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.77388.log 2024-04-16T16:23:44.170 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49402.log 2024-04-16T16:23:44.170 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67661.log.gz 2024-04-16T16:23:44.171 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.77388.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77388.log.gz 2024-04-16T16:23:44.171 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41122.log 2024-04-16T16:23:44.172 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78596.log 2024-04-16T16:23:44.173 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49402.log.gz 2024-04-16T16:23:44.173 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46410.log 2024-04-16T16:23:44.173 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41122.log.gz 2024-04-16T16:23:44.174 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78596.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78596.log.gz 2024-04-16T16:23:44.174 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72877.log 2024-04-16T16:23:44.176 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73358.log 2024-04-16T16:23:44.176 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46410.log.gz 2024-04-16T16:23:44.176 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mon.c.log: 92.8% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-16T16:23:44.177 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65260.log 2024-04-16T16:23:44.177 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72877.log.gz 2024-04-16T16:23:44.178 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75296.log 2024-04-16T16:23:44.178 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.73358.log.gz 2024-04-16T16:23:44.179 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65260.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65260.log.gz 2024-04-16T16:23:44.179 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41809.log 2024-04-16T16:23:44.180 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56129.log 2024-04-16T16:23:44.180 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75296.log.gz 2024-04-16T16:23:44.181 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41809.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41809.log.gz 2024-04-16T16:23:44.182 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79584.log 2024-04-16T16:23:44.182 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69884.log 2024-04-16T16:23:44.183 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56129.log.gz 2024-04-16T16:23:44.184 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79584.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79584.log.gz 2024-04-16T16:23:44.184 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74729.log 2024-04-16T16:23:44.185 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66047.log 2024-04-16T16:23:44.185 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69884.log.gz 2024-04-16T16:23:44.186 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59297.log 2024-04-16T16:23:44.186 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74729.log.gz 2024-04-16T16:23:44.187 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66047.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66047.log.gz 2024-04-16T16:23:44.187 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36709.log 2024-04-16T16:23:44.188 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66836.log 2024-04-16T16:23:44.189 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59297.log.gz 2024-04-16T16:23:44.189 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.36709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56489.log 2024-04-16T16:23:44.190 INFO:teuthology.orchestra.run.smithi045.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.36709.log.gz 2024-04-16T16:23:44.190 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66836.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66836.log.gz 2024-04-16T16:23:44.191 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72190.log 2024-04-16T16:23:44.192 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55566.log 2024-04-16T16:23:44.192 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56489.log.gz 2024-04-16T16:23:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72598.log 2024-04-16T16:23:44.193 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72190.log.gz 2024-04-16T16:23:44.194 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55566.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55566.log.gz 2024-04-16T16:23:44.194 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60790.log 2024-04-16T16:23:44.195 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69233.log 2024-04-16T16:23:44.196 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72598.log.gz 2024-04-16T16:23:44.196 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54409.log 2024-04-16T16:23:44.197 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60790.log.gz 2024-04-16T16:23:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69233.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69233.log.gz 2024-04-16T16:23:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64907.log 2024-04-16T16:23:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75703.log 2024-04-16T16:23:44.199 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54409.log.gz 2024-04-16T16:23:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78134.log 2024-04-16T16:23:44.200 INFO:teuthology.orchestra.run.smithi045.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.64907.log.gz 2024-04-16T16:23:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75703.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75703.log.gz 2024-04-16T16:23:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67785.log 2024-04-16T16:23:44.202 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79231.log 2024-04-16T16:23:44.203 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.78134.log.gz 2024-04-16T16:23:44.203 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43806.log 2024-04-16T16:23:44.203 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67785.log.gz 2024-04-16T16:23:44.204 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79231.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79231.log.gz 2024-04-16T16:23:44.205 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78722.log 2024-04-16T16:23:44.205 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.43806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42493.log 2024-04-16T16:23:44.206 INFO:teuthology.orchestra.run.smithi045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.43806.log.gz 2024-04-16T16:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78722.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.78722.log.gz 2024-04-16T16:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74309.log 2024-04-16T16:23:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79370.log 2024-04-16T16:23:44.208 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42493.log.gz 2024-04-16T16:23:44.209 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74309.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.74309.log.gz 2024-04-16T16:23:44.209 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74035.log 2024-04-16T16:23:44.210 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79370.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79370.log.gz 2024-04-16T16:23:44.210 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54661.log 2024-04-16T16:23:44.211 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80465.log 2024-04-16T16:23:44.212 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74035.log.gz 2024-04-16T16:23:44.212 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54661.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54661.log.gz 2024-04-16T16:23:44.212 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61717.log 2024-04-16T16:23:44.214 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80465.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80465.log.gz 2024-04-16T16:23:44.214 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51839.log 2024-04-16T16:23:44.215 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47127.log 2024-04-16T16:23:44.215 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61717.log.gz 2024-04-16T16:23:44.216 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.51839.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51839.log.gz 2024-04-16T16:23:44.216 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64440.log 2024-04-16T16:23:44.217 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47127.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47127.log.gz 2024-04-16T16:23:44.217 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73993.log 2024-04-16T16:23:44.218 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60538.log 2024-04-16T16:23:44.218 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64440.log.gz 2024-04-16T16:23:44.219 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73993.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73993.log.gz 2024-04-16T16:23:44.219 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67360.log 2024-04-16T16:23:44.220 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60538.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60538.log.gz -5 2024-04-16T16:23:44.220 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.59423.log 2024-04-16T16:23:44.221 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67360.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30694.log 2024-04-16T16:23:44.222 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67360.log.gz 2024-04-16T16:23:44.223 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64253.log 2024-04-16T16:23:44.223 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59423.log.gz 2024-04-16T16:23:44.224 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.tmp-client.admin.30694.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30694.log.gz 2024-04-16T16:23:44.224 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76003.log 2024-04-16T16:23:44.225 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56045.log 2024-04-16T16:23:44.226 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64253.log.gz 2024-04-16T16:23:44.227 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.76003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52049.log 2024-04-16T16:23:44.227 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76003.log.gz 2024-04-16T16:23:44.228 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73442.log 2024-04-16T16:23:44.228 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56045.log.gz 2024-04-16T16:23:44.229 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58893.log 2024-04-16T16:23:44.229 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52049.log.gz 2024-04-16T16:23:44.230 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80549.log 2024-04-16T16:23:44.230 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73442.log.gz 2024-04-16T16:23:44.231 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.58893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65176.log 2024-04-16T16:23:44.232 INFO:teuthology.orchestra.run.smithi045.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58893.log.gz 2024-04-16T16:23:44.232 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51965.log 2024-04-16T16:23:44.233 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.80549.log.gz 2024-04-16T16:23:44.233 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60341.log 2024-04-16T16:23:44.234 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65176.log.gz 2024-04-16T16:23:44.235 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.51965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47940.log 2024-04-16T16:23:44.235 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51965.log.gz 2024-04-16T16:23:44.236 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53595.log 2024-04-16T16:23:44.236 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60341.log.gz 2024-04-16T16:23:44.237 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57229.log 2024-04-16T16:23:44.238 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47940.log.gz 2024-04-16T16:23:44.238 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53595.log: gzip 87.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.53595.log.gz -- 2024-04-16T16:23:44.239 INFO:teuthology.orchestra.run.smithi045.stderr: /var/log/ceph/ceph-client.admin.56908.log 2024-04-16T16:23:44.239 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.57229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42943.log 2024-04-16T16:23:44.240 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57229.log.gz 2024-04-16T16:23:44.240 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48234.log 2024-04-16T16:23:44.241 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56908.log.gz 2024-04-16T16:23:44.242 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47717.log 2024-04-16T16:23:44.242 INFO:teuthology.orchestra.run.smithi045.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42943.log.gz 2024-04-16T16:23:44.243 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40726.log 2024-04-16T16:23:44.243 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48234.log.gz 2024-04-16T16:23:44.244 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55183.log 2024-04-16T16:23:44.244 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47717.log.gz 2024-04-16T16:23:44.245 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40726.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40726.log.gzgzip 2024-04-16T16:23:44.246 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43279.log 2024-04-16T16:23:44.247 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49654.log 2024-04-16T16:23:44.247 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55183.log.gz 2024-04-16T16:23:44.248 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.43279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48795.log 2024-04-16T16:23:44.248 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43279.log.gz 2024-04-16T16:23:44.249 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70820.log 2024-04-16T16:23:44.249 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49654.log.gz 2024-04-16T16:23:44.250 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48795.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48795.log.gz 2024-04-16T16:23:44.251 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42490.log 2024-04-16T16:23:44.252 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46860.log 2024-04-16T16:23:44.252 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70820.log.gz 2024-04-16T16:23:44.253 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46498.log 2024-04-16T16:23:44.253 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42490.log.gz 2024-04-16T16:23:44.254 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63940.log 2024-04-16T16:23:44.254 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46860.log.gz 2024-04-16T16:23:44.255 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51923.log 2024-04-16T16:23:44.255 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46498.log.gz 2024-04-16T16:23:44.257 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63940.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63940.log.gz 2024-04-16T16:23:44.257 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58725.log 2024-04-16T16:23:44.258 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.51923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49139.log 2024-04-16T16:23:44.258 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51923.log.gz 2024-04-16T16:23:44.259 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.58725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53679.log 2024-04-16T16:23:44.259 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58725.log.gz 2024-04-16T16:23:44.260 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62550.log 2024-04-16T16:23:44.260 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49139.log.gz 2024-04-16T16:23:44.261 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68451.log 2024-04-16T16:23:44.261 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53679.log.gz 2024-04-16T16:23:44.262 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49318.log 2024-04-16T16:23:44.263 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68451.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.62550.log.gz 2024-04-16T16:23:44.264 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46650.log 2024-04-16T16:23:44.264 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68451.log.gz 2024-04-16T16:23:44.265 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49318.log: gzip -5 --verbose -- 87.8% /var/log/ceph/ceph-client.admin.50009.log -- replaced with /var/log/ceph/ceph-client.admin.49318.log.gz 2024-04-16T16:23:44.265 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-16T16:23:44.266 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80129.log 2024-04-16T16:23:44.266 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46650.log.gz 2024-04-16T16:23:44.267 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50362.log 2024-04-16T16:23:44.267 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50009.log.gz 2024-04-16T16:23:44.268 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72435.log 2024-04-16T16:23:44.269 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80129.log.gz 2024-04-16T16:23:44.270 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55650.log 2024-04-16T16:23:44.270 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50362.log.gz 2024-04-16T16:23:44.271 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50488.log 2024-04-16T16:23:44.271 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72435.log.gz 2024-04-16T16:23:44.272 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53637.log 2024-04-16T16:23:44.272 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55650.log.gz 2024-04-16T16:23:44.273 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53947.log 2024-04-16T16:23:44.274 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50488.log.gz 2024-04-16T16:23:44.274 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64066.log 2024-04-16T16:23:44.274 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53637.log.gz 2024-04-16T16:23:44.275 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53947.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53947.log.gz 2024-04-16T16:23:44.276 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57145.log 2024-04-16T16:23:44.276 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45906.log 2024-04-16T16:23:44.277 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64066.log.gz 2024-04-16T16:23:44.278 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.57145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55919.log 2024-04-16T16:23:44.278 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57145.log.gz 2024-04-16T16:23:44.279 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68223.log 2024-04-16T16:23:44.279 INFO:teuthology.orchestra.run.smithi045.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45906.log.gz 2024-04-16T16:23:44.280 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62883.log 2024-04-16T16:23:44.280 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55919.log.gz 2024-04-16T16:23:44.281 INFO:teuthology.orchestra.run.smithi103.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-16T16:23:44.282 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55225.log 2024-04-16T16:23:44.282 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68223.log.gz 2024-04-16T16:23:44.283 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62883.log: gzip -5 --verbose -- 87.9% /var/log/ceph/ceph-client.admin.70392.log 2024-04-16T16:23:44.283 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62883.log.gz 2024-04-16T16:23:44.283 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41217.log 2024-04-16T16:23:44.284 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55225.log.gz 2024-04-16T16:23:44.284 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72046.log 2024-04-16T16:23:44.285 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70392.log.gz 2024-04-16T16:23:44.286 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73400.log 2024-04-16T16:23:44.286 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41217.log.gz 2024-04-16T16:23:44.287 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45576.log 2024-04-16T16:23:44.287 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72046.log.gz 2024-04-16T16:23:44.288 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44751.log 2024-04-16T16:23:44.288 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73400.log.gz 2024-04-16T16:23:44.290 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45576.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45576.log.gz 2024-04-16T16:23:44.290 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48276.log 2024-04-16T16:23:44.291 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.44751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56171.log 2024-04-16T16:23:44.291 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44751.log.gz 2024-04-16T16:23:44.292 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45822.log 2024-04-16T16:23:44.292 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48276.log.gz 2024-04-16T16:23:44.293 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56171.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56171.log.gz 2024-04-16T16:23:44.294 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43959.log 2024-04-16T16:23:44.295 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59339.log 2024-04-16T16:23:44.295 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45822.log.gz 2024-04-16T16:23:44.296 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.43959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41641.log 2024-04-16T16:23:44.296 INFO:teuthology.orchestra.run.smithi045.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43959.log.gz 2024-04-16T16:23:44.297 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49883.log 2024-04-16T16:23:44.297 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59339.log.gz 2024-04-16T16:23:44.298 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64398.log 2024-04-16T16:23:44.299 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41641.log.gz 2024-04-16T16:23:44.300 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51273.log 2024-04-16T16:23:44.300 INFO:teuthology.orchestra.run.smithi045.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.49883.log.gz 2024-04-16T16:23:44.301 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64482.log 2024-04-16T16:23:44.301 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64398.log.gz 2024-04-16T16:23:44.302 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.51273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.51273.log.gz 2024-04-16T16:23:44.302 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46818.log 2024-04-16T16:23:44.303 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60622.log 2024-04-16T16:23:44.303 INFO:teuthology.orchestra.run.smithi045.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64482.log.gz 2024-04-16T16:23:44.304 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46607.log 2024-04-16T16:23:44.305 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46818.log.gz 2024-04-16T16:23:44.305 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77848.log 2024-04-16T16:23:44.306 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60622.log.gz 2024-04-16T16:23:44.307 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49061.log 2024-04-16T16:23:44.307 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46607.log.gz 2024-04-16T16:23:44.308 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.77848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53009.log 2024-04-16T16:23:44.308 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77848.log.gz 2024-04-16T16:23:44.309 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79752.log 2024-04-16T16:23:44.309 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49061.log.gz 2024-04-16T16:23:44.310 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53009.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53009.log.gz 2024-04-16T16:23:44.311 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49528.log 2024-04-16T16:23:44.312 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59825.log 2024-04-16T16:23:44.312 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79752.log.gz 2024-04-16T16:23:44.313 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49570.log 2024-04-16T16:23:44.313 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49528.log.gz 2024-04-16T16:23:44.314 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59825.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-16T16:23:44.314 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59825.log.gz 2024-04-16T16:23:44.315 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55057.log 2024-04-16T16:23:44.315 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49570.log.gz 2024-04-16T16:23:44.316 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61166.log 2024-04-16T16:23:44.318 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55057.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55057.log.gz 2024-04-16T16:23:44.318 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48935.log 2024-04-16T16:23:44.319 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75433.log 2024-04-16T16:23:44.319 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61166.log.gz 2024-04-16T16:23:44.320 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48935.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48935.log.gz 2024-04-16T16:23:44.320 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64211.log 2024-04-16T16:23:44.321 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75433.log: 89.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-16T16:23:44.321 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45660.log 2024-04-16T16:23:44.322 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75433.log.gz 2024-04-16T16:23:44.322 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62028.log 2024-04-16T16:23:44.323 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64211.log.gz 2024-04-16T16:23:44.324 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50051.log 2024-04-16T16:23:44.324 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45660.log.gz 2024-04-16T16:23:44.325 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46902.log 2024-04-16T16:23:44.325 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62028.log.gz 2024-04-16T16:23:44.326 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42733.log 2024-04-16T16:23:44.326 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50051.log.gz 2024-04-16T16:23:44.327 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46038.log 2024-04-16T16:23:44.327 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46902.log.gz 2024-04-16T16:23:44.328 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65887.log 2024-04-16T16:23:44.328 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42733.log.gz 2024-04-16T16:23:44.329 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49967.log 2024-04-16T16:23:44.330 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46038.log.gz 2024-04-16T16:23:44.331 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66710.log 2024-04-16T16:23:44.331 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65887.log.gz 2024-04-16T16:23:44.332 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51499.log 2024-04-16T16:23:44.332 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49967.log.gz 2024-04-16T16:23:44.333 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79710.log 2024-04-16T16:23:44.333 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66710.log.gz 2024-04-16T16:23:44.334 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.51499.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51499.log.gz 2024-04-16T16:23:44.335 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67234.log 2024-04-16T16:23:44.336 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44709.log 2024-04-16T16:23:44.336 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79710.log.gz 2024-04-16T16:23:44.337 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77633.log 2024-04-16T16:23:44.337 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67234.log.gz 2024-04-16T16:23:44.338 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.44709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50446.log 2024-04-16T16:23:44.338 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44709.log.gz 2024-04-16T16:23:44.339 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.77633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77549.log 2024-04-16T16:23:44.339 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77633.log.gz 2024-04-16T16:23:44.340 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50446.log: gzip 88.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.50446.log.gz --verbose 2024-04-16T16:23:44.341 INFO:teuthology.orchestra.run.smithi045.stderr: -- /var/log/ceph/ceph-client.admin.70948.log 2024-04-16T16:23:44.342 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.77549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73652.log 2024-04-16T16:23:44.342 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77549.log.gz 2024-04-16T16:23:44.343 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61982.log 2024-04-16T16:23:44.343 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70948.log.gz 2024-04-16T16:23:44.344 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48150.log 2024-04-16T16:23:44.344 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73652.log.gz 2024-04-16T16:23:44.345 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70992.log 2024-04-16T16:23:44.345 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61982.log.gz 2024-04-16T16:23:44.346 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54619.log 2024-04-16T16:23:44.346 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48150.log.gz 2024-04-16T16:23:44.347 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80675.log 2024-04-16T16:23:44.347 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70992.log.gz 2024-04-16T16:23:44.348 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64337.log 2024-04-16T16:23:44.349 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54619.log.gz 2024-04-16T16:23:44.350 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70281.log 2024-04-16T16:23:44.350 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80675.log.gz 2024-04-16T16:23:44.351 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.71722.log 2024-04-16T16:23:44.351 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64337.log.gz 2024-04-16T16:23:44.352 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54931.log 2024-04-16T16:23:44.352 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70281.log.gz 2024-04-16T16:23:44.353 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.71722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.71722.log.gz 2024-04-16T16:23:44.353 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56213.log 2024-04-16T16:23:44.354 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54931.log: gzip -5 --verbose -- 88.4% /var/log/ceph/ceph-client.admin.74183.log 2024-04-16T16:23:44.354 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54931.log.gz 2024-04-16T16:23:44.355 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49233.log 2024-04-16T16:23:44.356 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56213.log.gz 2024-04-16T16:23:44.357 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46122.log 2024-04-16T16:23:44.357 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74183.log.gz 2024-04-16T16:23:44.358 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80633.log 2024-04-16T16:23:44.358 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49233.log.gz 2024-04-16T16:23:44.359 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43027.log 2024-04-16T16:23:44.359 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46122.log.gz 2024-04-16T16:23:44.360 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73568.log 2024-04-16T16:23:44.360 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80633.log.gz 2024-04-16T16:23:44.361 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.43027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78901.log 2024-04-16T16:23:44.361 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43027.log.gz 2024-04-16T16:23:44.362 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57019.log 2024-04-16T16:23:44.363 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73568.log.gz 2024-04-16T16:23:44.364 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58935.log 2024-04-16T16:23:44.364 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.78901.log.gz 2024-04-16T16:23:44.365 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.57019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56977.log 2024-04-16T16:23:44.365 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57019.log.gz 2024-04-16T16:23:44.366 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.58935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73171.log 2024-04-16T16:23:44.366 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58935.log.gz 2024-04-16T16:23:44.367 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77675.log 2024-04-16T16:23:44.367 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56977.log.gz 2024-04-16T16:23:44.368 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79500.log 2024-04-16T16:23:44.368 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73171.log.gz 2024-04-16T16:23:44.369 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.77675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69716.log 2024-04-16T16:23:44.370 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.77675.log.gz 2024-04-16T16:23:44.371 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72835.log 2024-04-16T16:23:44.371 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79500.log.gz 2024-04-16T16:23:44.372 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56255.log 2024-04-16T16:23:44.372 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69716.log.gz 2024-04-16T16:23:44.373 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75391.log 2024-04-16T16:23:44.373 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.72835.log.gz 2024-04-16T16:23:44.374 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41347.log 2024-04-16T16:23:44.374 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56255.log.gz 2024-04-16T16:23:44.375 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51001.log 2024-04-16T16:23:44.375 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75391.log.gz 2024-04-16T16:23:44.376 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42484.log 2024-04-16T16:23:44.377 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41347.log.gz 2024-04-16T16:23:44.377 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.51001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62322.log 2024-04-16T16:23:44.378 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51001.log.gz 2024-04-16T16:23:44.379 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52613.log 2024-04-16T16:23:44.379 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42484.log.gz 2024-04-16T16:23:44.380 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41820.log 2024-04-16T16:23:44.380 INFO:teuthology.orchestra.run.smithi045.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62322.log.gz 2024-04-16T16:23:44.381 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59255.log 2024-04-16T16:23:44.381 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52613.log.gz 2024-04-16T16:23:44.382 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70052.log 2024-04-16T16:23:44.382 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41820.log.gz 2024-04-16T16:23:44.383 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45534.log 2024-04-16T16:23:44.383 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59255.log.gz 2024-04-16T16:23:44.384 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59465.log 2024-04-16T16:23:44.385 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70052.log.gz 2024-04-16T16:23:44.386 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70197.log 2024-04-16T16:23:44.386 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45534.log.gz 2024-04-16T16:23:44.387 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60832.log 2024-04-16T16:23:44.387 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59465.log.gz 2024-04-16T16:23:44.388 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51797.log 2024-04-16T16:23:44.388 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70197.log.gz 2024-04-16T16:23:44.389 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47503.log 2024-04-16T16:23:44.389 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60832.log.gz 2024-04-16T16:23:44.390 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.51797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78638.log 2024-04-16T16:23:44.391 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51797.log.gz 2024-04-16T16:23:44.392 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53553.log 2024-04-16T16:23:44.392 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47503.log.gz 2024-04-16T16:23:44.393 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64108.log 2024-04-16T16:23:44.393 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78638.log.gz 2024-04-16T16:23:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54829.log 2024-04-16T16:23:44.394 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53553.log.gz 2024-04-16T16:23:44.395 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51667.log 2024-04-16T16:23:44.395 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64108.log.gz 2024-04-16T16:23:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48711.log 2024-04-16T16:23:44.396 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54829.log.gz 2024-04-16T16:23:44.397 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.51667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52445.log 2024-04-16T16:23:44.398 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51667.log.gz 2024-04-16T16:23:44.399 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51709.log 2024-04-16T16:23:44.399 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48711.log.gz 2024-04-16T16:23:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42159.log 2024-04-16T16:23:44.400 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52445.log.gz 2024-04-16T16:23:44.401 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.51709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64949.log 2024-04-16T16:23:44.401 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51709.log.gz 2024-04-16T16:23:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46242.log 2024-04-16T16:23:44.402 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42159.log.gz 2024-04-16T16:23:44.403 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71801.log 2024-04-16T16:23:44.403 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64949.log.gz 2024-04-16T16:23:44.404 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44877.log 2024-04-16T16:23:44.405 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46242.log.gz 2024-04-16T16:23:44.406 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71801.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71801.log.gz 2024-04-16T16:23:44.406 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64823.log 2024-04-16T16:23:44.407 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.44877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47335.log 2024-04-16T16:23:44.407 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44877.log.gz 2024-04-16T16:23:44.408 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60205.log 2024-04-16T16:23:44.408 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64823.log.gz 2024-04-16T16:23:44.410 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74351.log 2024-04-16T16:23:44.410 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47335.log.gz 2024-04-16T16:23:44.410 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59381.log 2024-04-16T16:23:44.411 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60205.log.gz 2024-04-16T16:23:44.412 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74351.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74351.log.gz 2024-04-16T16:23:44.412 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.44263.log 2024-04-16T16:23:44.413 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50959.log 2024-04-16T16:23:44.413 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59381.log.gz 2024-04-16T16:23:44.414 INFO:teuthology.orchestra.run.smithi103.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-16T16:23:44.414 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.2.44263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.44263.log.gz 2024-04-16T16:23:44.415 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79105.log 2024-04-16T16:23:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78260.log 2024-04-16T16:23:44.416 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50959.log.gz 2024-04-16T16:23:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70239.log 2024-04-16T16:23:44.418 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79105.log.gz 2024-04-16T16:23:44.418 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.75067.log 2024-04-16T16:23:44.419 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.78260.log.gz 2024-04-16T16:23:44.419 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42153.log 2024-04-16T16:23:44.420 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70239.log.gz 2024-04-16T16:23:44.420 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.75067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.75067.log.gz 2024-04-16T16:23:44.421 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55780.log 2024-04-16T16:23:44.422 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42153.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.42153.log.gz -5 2024-04-16T16:23:44.422 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.75577.log 2024-04-16T16:23:44.423 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53721.log 2024-04-16T16:23:44.423 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55780.log.gz 2024-04-16T16:23:44.424 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55873.log 2024-04-16T16:23:44.425 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75577.log.gz 2024-04-16T16:23:44.426 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53170.log 2024-04-16T16:23:44.426 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53721.log.gz 2024-04-16T16:23:44.427 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75619.log 2024-04-16T16:23:44.427 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55873.log.gz 2024-04-16T16:23:44.428 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53170.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53170.log.gz 2024-04-16T16:23:44.428 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80213.log 2024-04-16T16:23:44.429 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52841.log 2024-04-16T16:23:44.430 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75619.log.gz 2024-04-16T16:23:44.430 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80213.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80213.log.gz 2024-04-16T16:23:44.431 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51043.log 2024-04-16T16:23:44.432 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59657.log 2024-04-16T16:23:44.432 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52841.log.gz 2024-04-16T16:23:44.433 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.51043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48066.log 2024-04-16T16:23:44.433 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51043.log.gz 2024-04-16T16:23:44.434 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66668.log 2024-04-16T16:23:44.434 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59657.log.gz 2024-04-16T16:23:44.435 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80381.log 2024-04-16T16:23:44.435 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48066.log.gz 2024-04-16T16:23:44.436 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46692.log 2024-04-16T16:23:44.436 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66668.log.gz 2024-04-16T16:23:44.438 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64024.log 2024-04-16T16:23:44.438 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80381.log.gz 2024-04-16T16:23:44.439 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46692.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.46692.log.gz --verbose 2024-04-16T16:23:44.439 INFO:teuthology.orchestra.run.smithi045.stderr: -- /var/log/ceph/ceph-client.admin.40855.log 2024-04-16T16:23:44.440 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64699.log 2024-04-16T16:23:44.440 INFO:teuthology.orchestra.run.smithi045.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64024.log.gz 2024-04-16T16:23:44.441 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68180.log 2024-04-16T16:23:44.441 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40855.log.gz 2024-04-16T16:23:44.442 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79412.log 2024-04-16T16:23:44.442 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64699.log.gz 2024-04-16T16:23:44.444 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68180.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-16T16:23:44.444 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68180.log.gz 2024-04-16T16:23:44.445 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79412.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79412.log.gz 2024-04-16T16:23:44.445 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72106.log 2024-04-16T16:23:44.446 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74879.log 2024-04-16T16:23:44.447 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47377.log 2024-04-16T16:23:44.447 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72106.log.gz 2024-04-16T16:23:44.448 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74879.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74879.log.gz 2024-04-16T16:23:44.448 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64865.log 2024-04-16T16:23:44.449 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68367.log 2024-04-16T16:23:44.450 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47377.log.gz 2024-04-16T16:23:44.450 INFO:teuthology.orchestra.run.smithi103.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-16T16:23:44.451 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64865.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64865.log.gz 2024-04-16T16:23:44.451 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41823.log 2024-04-16T16:23:44.452 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46368.log 2024-04-16T16:23:44.452 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68367.log.gz 2024-04-16T16:23:44.453 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41823.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41823.log.gz 2024-04-16T16:23:44.453 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44961.log 2024-04-16T16:23:44.454 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77169.log 2024-04-16T16:23:44.454 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46368.log.gz 2024-04-16T16:23:44.455 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.44961.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44961.log.gz 2024-04-16T16:23:44.456 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79542.log 2024-04-16T16:23:44.456 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.77169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66431.log 2024-04-16T16:23:44.457 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77169.log.gz 2024-04-16T16:23:44.458 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79542.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79542.log.gz 2024-04-16T16:23:44.458 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62112.log 2024-04-16T16:23:44.459 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72351.log 2024-04-16T16:23:44.459 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66431.log.gz 2024-04-16T16:23:44.460 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62112.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62112.log.gz 2024-04-16T16:23:44.460 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75787.log 2024-04-16T16:23:44.461 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52337.log 2024-04-16T16:23:44.462 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72351.log.gz 2024-04-16T16:23:44.462 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75787.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75787.log.gz 2024-04-16T16:23:44.463 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47587.log 2024-04-16T16:23:44.463 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60438.log 2024-04-16T16:23:44.464 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52337.log.gz 2024-04-16T16:23:44.465 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47587.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47587.log.gz 2024-04-16T16:23:44.465 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55267.log 2024-04-16T16:23:44.466 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79189.log 2024-04-16T16:23:44.466 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60438.log.gz 2024-04-16T16:23:44.467 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55267.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55267.log.gz 2024-04-16T16:23:44.468 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59615.log 2024-04-16T16:23:44.468 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63898.log 2024-04-16T16:23:44.469 INFO:teuthology.orchestra.run.smithi045.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.79189.log.gz 2024-04-16T16:23:44.470 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59615.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59615.log.gz 2024-04-16T16:23:44.470 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79668.log 2024-04-16T16:23:44.471 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40968.log 2024-04-16T16:23:44.471 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63898.log.gz 2024-04-16T16:23:44.472 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79668.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79668.log.gz 2024-04-16T16:23:44.472 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44625.log 2024-04-16T16:23:44.474 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75188.log 88.8% 2024-04-16T16:23:44.474 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40968.log.gz 2024-04-16T16:23:44.475 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.44625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69800.log 2024-04-16T16:23:44.475 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44625.log.gz 2024-04-16T16:23:44.476 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69275.log 2024-04-16T16:23:44.476 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75188.log.gz 2024-04-16T16:23:44.477 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69800.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69800.log.gz 2024-04-16T16:23:44.477 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66227.log 2024-04-16T16:23:44.479 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47047.log 88.7% 2024-04-16T16:23:44.479 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69275.log.gz 2024-04-16T16:23:44.480 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42859.log 2024-04-16T16:23:44.480 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66227.log.gz 2024-04-16T16:23:44.481 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69030.log 2024-04-16T16:23:44.481 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47047.log.gz 2024-04-16T16:23:44.482 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42859.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42859.log.gz 2024-04-16T16:23:44.482 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55524.log 2024-04-16T16:23:44.483 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80801.log 2024-04-16T16:23:44.484 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69030.log.gz 2024-04-16T16:23:44.485 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55524.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55524.log.gz 2024-04-16T16:23:44.485 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71043.log 2024-04-16T16:23:44.486 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64295.log 2024-04-16T16:23:44.486 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80801.log.gz 2024-04-16T16:23:44.487 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71043.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71043.log.gz 2024-04-16T16:23:44.487 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61208.log 2024-04-16T16:23:44.488 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80591.log 2024-04-16T16:23:44.488 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64295.log.gz 2024-04-16T16:23:44.489 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61208.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61208.log.gz 2024-04-16T16:23:44.490 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73087.log 2024-04-16T16:23:44.491 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52697.log 2024-04-16T16:23:44.491 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80591.log.gz 2024-04-16T16:23:44.492 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73087.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73087.log.gz 2024-04-16T16:23:44.492 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47810.log 2024-04-16T16:23:44.493 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78218.log 2024-04-16T16:23:44.493 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52697.log.gz 2024-04-16T16:23:44.494 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47810.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47810.log.gz 2024-04-16T16:23:44.494 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78092.log 2024-04-16T16:23:44.495 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67581.log 2024-04-16T16:23:44.495 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.78218.log.gz 2024-04-16T16:23:44.496 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78092.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78092.log.gz 2024-04-16T16:23:44.497 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69574.log 2024-04-16T16:23:44.497 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48024.log 2024-04-16T16:23:44.498 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67581.log.gz 2024-04-16T16:23:44.499 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69574.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69574.log.gz 2024-04-16T16:23:44.499 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71426.log 2024-04-16T16:23:44.500 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79943.log 2024-04-16T16:23:44.500 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48024.log.gz 2024-04-16T16:23:44.501 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44397.log 2024-04-16T16:23:44.501 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71426.log.gz 2024-04-16T16:23:44.502 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68409.log 2024-04-16T16:23:44.502 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79943.log.gz 2024-04-16T16:23:44.503 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.44397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53763.log 2024-04-16T16:23:44.503 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44397.log.gz 2024-04-16T16:23:44.504 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42495.log 2024-04-16T16:23:44.504 INFO:teuthology.orchestra.run.smithi045.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.68409.log.gz 2024-04-16T16:23:44.505 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63493.log 2024-04-16T16:23:44.506 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53763.log.gz 2024-04-16T16:23:44.507 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65218.log 2024-04-16T16:23:44.507 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42495.log.gz 2024-04-16T16:23:44.508 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42155.log 2024-04-16T16:23:44.508 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63493.log.gz 2024-04-16T16:23:44.509 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49276.log 2024-04-16T16:23:44.509 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65218.log.gz 2024-04-16T16:23:44.510 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75230.log 2024-04-16T16:23:44.510 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42155.log.gz 2024-04-16T16:23:44.511 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80843.log 2024-04-16T16:23:44.511 INFO:teuthology.orchestra.run.smithi045.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.49276.log.gz 2024-04-16T16:23:44.512 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49840.log 2024-04-16T16:23:44.513 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75230.log.gz 2024-04-16T16:23:44.513 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68265.log 2024-04-16T16:23:44.514 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80843.log.gz 2024-04-16T16:23:44.515 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74393.log 2024-04-16T16:23:44.515 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49840.log.gz 2024-04-16T16:23:44.516 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46545.log 2024-04-16T16:23:44.517 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68265.log.gz 2024-04-16T16:23:44.517 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46962.log 2024-04-16T16:23:44.518 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74393.log.gz 2024-04-16T16:23:44.519 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46545.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46545.log.gz 2024-04-16T16:23:44.519 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44499.log 2024-04-16T16:23:44.520 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53511.log 2024-04-16T16:23:44.520 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46962.log.gz 2024-04-16T16:23:44.520 INFO:teuthology.orchestra.run.smithi103.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-16T16:23:44.521 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.44499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77950.log 2024-04-16T16:23:44.521 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44499.log.gz 2024-04-16T16:23:44.522 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40807.log 2024-04-16T16:23:44.522 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53511.log.gz 2024-04-16T16:23:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.77950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72777.log 2024-04-16T16:23:44.524 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77950.log.gz 2024-04-16T16:23:44.524 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51583.log 2024-04-16T16:23:44.525 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40807.log.gz 2024-04-16T16:23:44.526 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72777.log: gzip -5 87.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.72777.log.gz /var/log/ceph/ceph-client.admin.61759.log 2024-04-16T16:23:44.526 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-16T16:23:44.526 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.51583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42154.log 2024-04-16T16:23:44.527 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51583.log.gz 2024-04-16T16:23:44.528 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61759.log: gzip -5 88.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.61759.log.gz /var/log/ceph/ceph-client.admin.77795.log 2024-04-16T16:23:44.528 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-16T16:23:44.529 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61250.log 2024-04-16T16:23:44.529 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42154.log.gz 2024-04-16T16:23:44.530 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.77795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71300.log 2024-04-16T16:23:44.530 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77795.log.gz 2024-04-16T16:23:44.532 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61250.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61250.log.gz 2024-04-16T16:23:44.532 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58851.log 2024-04-16T16:23:44.533 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63630.log 2024-04-16T16:23:44.533 INFO:teuthology.orchestra.run.smithi045.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.71300.log.gz 2024-04-16T16:23:44.534 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.58851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59129.log 2024-04-16T16:23:44.534 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58851.log.gz 2024-04-16T16:23:44.535 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55608.log 2024-04-16T16:23:44.535 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63630.log.gz 2024-04-16T16:23:44.536 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59129.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.59129.log.gz 2024-04-16T16:23:44.536 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54973.log 2024-04-16T16:23:44.537 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37077.log 2024-04-16T16:23:44.537 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55608.log.gz 2024-04-16T16:23:44.538 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54973.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54973.log.gz 2024-04-16T16:23:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66610.log 2024-04-16T16:23:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.37077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62804.log 2024-04-16T16:23:44.540 INFO:teuthology.orchestra.run.smithi045.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37077.log.gz 2024-04-16T16:23:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66610.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66610.log.gz 2024-04-16T16:23:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70010.log 2024-04-16T16:23:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.62804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.62804.log.gz 2024-04-16T16:23:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63289.log 2024-04-16T16:23:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70010.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70010.log.gz 2024-04-16T16:23:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45738.log 2024-04-16T16:23:44.544 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70574.log 2024-04-16T16:23:44.544 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63289.log.gz 2024-04-16T16:23:44.545 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45738.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45738.log.gz 2024-04-16T16:23:44.545 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42901.log 2024-04-16T16:23:44.546 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41166.log 2024-04-16T16:23:44.546 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70574.log.gz 2024-04-16T16:23:44.547 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68869.log 2024-04-16T16:23:44.547 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42901.log.gz 2024-04-16T16:23:44.548 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70777.log 2024-04-16T16:23:44.548 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41166.log.gz 2024-04-16T16:23:44.549 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74077.log 2024-04-16T16:23:44.550 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68869.log.gz 2024-04-16T16:23:44.551 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60079.log 2024-04-16T16:23:44.551 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70777.log.gz 2024-04-16T16:23:44.552 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74077.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-16T16:23:44.552 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74077.log.gz 2024-04-16T16:23:44.553 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60079.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60079.log.gz 2024-04-16T16:23:44.553 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60580.log 2024-04-16T16:23:44.554 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47005.log 2024-04-16T16:23:44.555 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46776.log 2024-04-16T16:23:44.556 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60580.log.gz 2024-04-16T16:23:44.557 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74519.log 2024-04-16T16:23:44.557 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47005.log.gz 2024-04-16T16:23:44.558 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71468.log 2024-04-16T16:23:44.558 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46776.log.gz 2024-04-16T16:23:44.559 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63546.log 2024-04-16T16:23:44.560 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.74519.log.gz 2024-04-16T16:23:44.560 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45492.log 2024-04-16T16:23:44.561 INFO:teuthology.orchestra.run.smithi045.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.71468.log.gz 2024-04-16T16:23:44.562 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51625.log 2024-04-16T16:23:44.562 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63546.log.gz 2024-04-16T16:23:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41431.log 2024-04-16T16:23:44.563 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45492.log.gz 2024-04-16T16:23:44.564 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.51625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67743.log 2024-04-16T16:23:44.565 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51625.log.gz 2024-04-16T16:23:44.565 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57433.log 2024-04-16T16:23:44.566 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41431.log.gz 2024-04-16T16:23:44.567 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70862.log 2024-04-16T16:23:44.567 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67743.log.gz 2024-04-16T16:23:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.57433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72477.log 2024-04-16T16:23:44.569 INFO:teuthology.orchestra.run.smithi045.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.57433.log.gz 2024-04-16T16:23:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43459.log 2024-04-16T16:23:44.570 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70862.log.gz 2024-04-16T16:23:44.571 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60283.log 2024-04-16T16:23:44.571 INFO:teuthology.orchestra.run.smithi045.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.72477.log.gz 2024-04-16T16:23:44.572 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.43459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42817.log 2024-04-16T16:23:44.572 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43459.log.gz 2024-04-16T16:23:44.573 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60283.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60283.log.gz 2024-04-16T16:23:44.574 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48977.log 2024-04-16T16:23:44.574 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47673.log 2024-04-16T16:23:44.575 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42817.log.gz 2024-04-16T16:23:44.576 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48977.log: gzip -5 --verbose -- 87.7% /var/log/ceph/ceph-client.admin.54199.log 2024-04-16T16:23:44.576 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48977.log.gz 2024-04-16T16:23:44.577 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47673.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47673.log.gz 2024-04-16T16:23:44.577 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62925.log 2024-04-16T16:23:44.578 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55692.log 2024-04-16T16:23:44.579 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54199.log.gz 2024-04-16T16:23:44.580 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67955.log 2024-04-16T16:23:44.580 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62925.log.gz 2024-04-16T16:23:44.581 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55692.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55692.log.gz 2024-04-16T16:23:44.581 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61292.log 2024-04-16T16:23:44.582 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42985.log 2024-04-16T16:23:44.583 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67955.log.gz 2024-04-16T16:23:44.584 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61292.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.61292.log.gz 2024-04-16T16:23:44.584 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48893.log 2024-04-16T16:23:44.585 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56003.log 2024-04-16T16:23:44.585 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42985.log.gz 2024-04-16T16:23:44.586 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77507.log 2024-04-16T16:23:44.587 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48893.log.gz 2024-04-16T16:23:44.587 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69490.log 2024-04-16T16:23:44.588 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56003.log.gz 2024-04-16T16:23:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.77507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60664.log 2024-04-16T16:23:44.589 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77507.log.gz 2024-04-16T16:23:44.590 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60875.log 2024-04-16T16:23:44.591 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69490.log.gz 2024-04-16T16:23:44.591 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64619.log 2024-04-16T16:23:44.592 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60664.log.gz 2024-04-16T16:23:44.593 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54451.log 2024-04-16T16:23:44.593 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60875.log.gz 2024-04-16T16:23:44.594 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64619.log: gzip -5 --verbose -- 87.2% /var/log/ceph/ceph-client.admin.80171.log 2024-04-16T16:23:44.594 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64619.log.gz 2024-04-16T16:23:44.595 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52379.log 2024-04-16T16:23:44.595 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54451.log.gz 2024-04-16T16:23:44.596 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80171.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80171.log.gz 2024-04-16T16:23:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77753.log 2024-04-16T16:23:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78512.log 2024-04-16T16:23:44.599 INFO:teuthology.orchestra.run.smithi045.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.52379.log.gz 2024-04-16T16:23:44.599 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.77753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41259.log 2024-04-16T16:23:44.599 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77753.log.gz 2024-04-16T16:23:44.600 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67444.log 2024-04-16T16:23:44.601 INFO:teuthology.orchestra.run.smithi045.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78512.log.gz 2024-04-16T16:23:44.602 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71384.log 2024-04-16T16:23:44.602 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41259.log.gz 2024-04-16T16:23:44.603 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67444.log: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.67444.log.gz 2024-04-16T16:23:44.603 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57367.log 2024-04-16T16:23:44.604 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67192.log 2024-04-16T16:23:44.604 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5%/var/log/ceph/ceph-client.admin.57367.log: -- replaced with /var/log/ceph/ceph-client.admin.71384.log.gz 2024-04-16T16:23:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43195.log 2024-04-16T16:23:44.606 INFO:teuthology.orchestra.run.smithi045.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.57367.log.gz 2024-04-16T16:23:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78764.log 2024-04-16T16:23:44.607 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67192.log.gz 2024-04-16T16:23:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.43195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54787.log 2024-04-16T16:23:44.608 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43195.log.gz 2024-04-16T16:23:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61940.log 2024-04-16T16:23:44.609 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78764.log.gz 2024-04-16T16:23:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54787.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54787.log.gzgzip 2024-04-16T16:23:44.610 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.70342.log 2024-04-16T16:23:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50745.log 2024-04-16T16:23:44.612 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61940.log.gz 2024-04-16T16:23:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70342.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70342.log.gz -5 2024-04-16T16:23:44.613 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49360.log 2024-04-16T16:23:44.614 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75829.log 2024-04-16T16:23:44.614 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50745.log.gz 2024-04-16T16:23:44.616 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48318.log 2024-04-16T16:23:44.616 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49360.log.gz 2024-04-16T16:23:44.616 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42775.log 2024-04-16T16:23:44.617 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75829.log.gz 2024-04-16T16:23:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48318.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48318.log.gz 2024-04-16T16:23:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44835.log 2024-04-16T16:23:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42775.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42775.log.gz 2024-04-16T16:23:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48108.log 2024-04-16T16:23:44.620 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.44835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57499.log 2024-04-16T16:23:44.621 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44835.log.gz 2024-04-16T16:23:44.622 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44583.log 2024-04-16T16:23:44.622 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48108.log.gz 2024-04-16T16:23:44.623 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.57499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63373.log 2024-04-16T16:23:44.623 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57499.log.gz 2024-04-16T16:23:44.624 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.44583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75349.log 2024-04-16T16:23:44.624 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44583.log.gz 2024-04-16T16:23:44.625 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60748.log 2024-04-16T16:23:44.626 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63373.log.gz 2024-04-16T16:23:44.627 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70436.log 2024-04-16T16:23:44.627 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75349.log.gz 2024-04-16T16:23:44.628 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78554.log 2024-04-16T16:23:44.628 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60748.log.gz 2024-04-16T16:23:44.629 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50179.log 2024-04-16T16:23:44.629 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70436.log.gz 2024-04-16T16:23:44.630 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60480.log 2024-04-16T16:23:44.631 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.78554.log.gz 2024-04-16T16:23:44.632 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50179.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50179.log.gz 2024-04-16T16:23:44.632 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50917.log 2024-04-16T16:23:44.633 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52967.log 2024-04-16T16:23:44.633 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60480.log.gz 2024-04-16T16:23:44.634 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67047.log 2024-04-16T16:23:44.634 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50917.log.gz 2024-04-16T16:23:44.635 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40630.log 2024-04-16T16:23:44.635 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52967.log.gz 2024-04-16T16:23:44.636 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73747.log 2024-04-16T16:23:44.637 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67047.log.gz 2024-04-16T16:23:44.638 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54577.log 2024-04-16T16:23:44.638 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40630.log.gz 2024-04-16T16:23:44.639 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50320.log 2024-04-16T16:23:44.640 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.73747.log.gz 2024-04-16T16:23:44.640 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78470.log 2024-04-16T16:23:44.641 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54577.log.gz 2024-04-16T16:23:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41822.log 2024-04-16T16:23:44.642 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50320.log.gz 2024-04-16T16:23:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74645.log 2024-04-16T16:23:44.643 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78470.log.gz 2024-04-16T16:23:44.644 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77295.log 2024-04-16T16:23:44.644 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41822.log.gz 2024-04-16T16:23:44.645 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79063.log 2024-04-16T16:23:44.646 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74645.log.gz 2024-04-16T16:23:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.77295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53051.log 2024-04-16T16:23:44.647 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77295.log.gz 2024-04-16T16:23:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79063.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.79063.log.gz 2024-04-16T16:23:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62070.log 2024-04-16T16:23:44.649 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67089.log 2024-04-16T16:23:44.649 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53051.log.gz 2024-04-16T16:23:44.650 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59699.log 2024-04-16T16:23:44.650 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62070.log.gz 2024-04-16T16:23:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74477.log 2024-04-16T16:23:44.651 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67089.log.gz 2024-04-16T16:23:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43153.log 2024-04-16T16:23:44.653 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59699.log.gz 2024-04-16T16:23:44.654 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68826.log 2024-04-16T16:23:44.654 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74477.log.gz 2024-04-16T16:23:44.655 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.43153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62196.log 2024-04-16T16:23:44.655 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43153.log.gz 2024-04-16T16:23:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42146.log 2024-04-16T16:23:44.656 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68826.log.gz 2024-04-16T16:23:44.657 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62196.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-16T16:23:44.657 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62196.log.gz 2024-04-16T16:23:44.659 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54283.log 2024-04-16T16:23:44.659 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42146.log.gz 2024-04-16T16:23:44.660 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44793.log 2024-04-16T16:23:44.661 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69758.log 2024-04-16T16:23:44.661 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54283.log.gz 2024-04-16T16:23:44.662 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.44793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80027.log 2024-04-16T16:23:44.662 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44793.log.gz 2024-04-16T16:23:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45864.log 2024-04-16T16:23:44.664 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69758.log.gz 2024-04-16T16:23:44.665 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65386.log 2024-04-16T16:23:44.665 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80027.log.gz 2024-04-16T16:23:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72148.log 2024-04-16T16:23:44.666 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45864.log.gz 2024-04-16T16:23:44.667 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63688.log 2024-04-16T16:23:44.667 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65386.log.gz 2024-04-16T16:23:44.668 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55736.log 2024-04-16T16:23:44.669 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72148.log.gz 2024-04-16T16:23:44.670 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63688.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58767.log 2024-04-16T16:23:44.670 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63688.log.gz 2024-04-16T16:23:44.670 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69842.log 2024-04-16T16:23:44.671 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55736.log.gz 2024-04-16T16:23:44.672 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.58767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52529.log 2024-04-16T16:23:44.672 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58767.log.gz 2024-04-16T16:23:44.674 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65302.log 2024-04-16T16:23:44.674 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69842.log.gz 2024-04-16T16:23:44.674 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52529.log: gzip 89.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.52529.log.gz --verbose 2024-04-16T16:23:44.675 INFO:teuthology.orchestra.run.smithi045.stderr: -- /var/log/ceph/ceph-client.admin.41815.log 2024-04-16T16:23:44.676 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67149.log 2024-04-16T16:23:44.676 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65302.log.gz 2024-04-16T16:23:44.677 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55442.log 2024-04-16T16:23:44.677 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41815.log.gz 2024-04-16T16:23:44.678 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67149.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67149.log.gz -5 2024-04-16T16:23:44.678 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54745.log 2024-04-16T16:23:44.679 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53863.log 2024-04-16T16:23:44.680 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55442.log.gz 2024-04-16T16:23:44.680 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66526.log 2024-04-16T16:23:44.681 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54745.log.gz 2024-04-16T16:23:44.682 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74435.log 2024-04-16T16:23:44.682 INFO:teuthology.orchestra.run.smithi045.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53863.log.gz 2024-04-16T16:23:44.683 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47856.log 2024-04-16T16:23:44.683 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66526.log.gz 2024-04-16T16:23:44.684 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54889.log 2024-04-16T16:23:44.684 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74435.log.gz 2024-04-16T16:23:44.685 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73526.log 2024-04-16T16:23:44.686 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47856.log.gz 2024-04-16T16:23:44.687 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60037.log 2024-04-16T16:23:44.687 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54889.log.gz 2024-04-16T16:23:44.688 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73526.log: gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.73526.log.gz 2024-04-16T16:23:44.688 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55362.log 2024-04-16T16:23:44.689 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80297.log 2024-04-16T16:23:44.689 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60037.log.gz 2024-04-16T16:23:44.690 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65344.log 2024-04-16T16:23:44.691 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55362.log.gz 2024-04-16T16:23:44.692 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69395.log 2024-04-16T16:23:44.692 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80297.log.gz 2024-04-16T16:23:44.693 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52133.log 2024-04-16T16:23:44.693 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65344.log.gz 2024-04-16T16:23:44.694 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69395.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69395.log.gz 2024-04-16T16:23:44.694 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66920.log 2024-04-16T16:23:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54241.log 2024-04-16T16:23:44.695 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52133.log.gz 2024-04-16T16:23:44.696 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66920.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66920.log.gzgzip 2024-04-16T16:23:44.696 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47898.log 2024-04-16T16:23:44.697 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68493.log 2024-04-16T16:23:44.698 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54241.log.gz 2024-04-16T16:23:44.698 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47898.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47898.log.gz 2024-04-16T16:23:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74225.log 2024-04-16T16:23:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72232.log 2024-04-16T16:23:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74225.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.68493.log.gz 2024-04-16T16:23:44.701 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74225.log.gz 2024-04-16T16:23:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71173.log 2024-04-16T16:23:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67911.log 2024-04-16T16:23:44.702 INFO:teuthology.orchestra.run.smithi045.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.72232.log.gz 2024-04-16T16:23:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54367.log 2024-04-16T16:23:44.703 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71173.log.gz 2024-04-16T16:23:44.704 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67911.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65130.log 2024-04-16T16:23:44.704 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67911.log.gz 2024-04-16T16:23:44.705 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70112.log 2024-04-16T16:23:44.705 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54367.log.gz 2024-04-16T16:23:44.711 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66752.log 2024-04-16T16:23:44.711 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65130.log.gz 2024-04-16T16:23:44.711 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70112.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73129.log 2024-04-16T16:23:44.711 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70112.log.gz 2024-04-16T16:23:44.711 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77253.log 2024-04-16T16:23:44.712 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66752.log.gz 2024-04-16T16:23:44.712 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73129.log: 91.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-16T16:23:44.712 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60918.log 2024-04-16T16:23:44.712 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73129.log.gz 2024-04-16T16:23:44.712 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.77253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80929.log 2024-04-16T16:23:44.712 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77253.log.gz 2024-04-16T16:23:44.713 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44667.log 2024-04-16T16:23:44.713 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60918.log.gz 2024-04-16T16:23:44.714 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50135.log 2024-04-16T16:23:44.714 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80929.log.gz 2024-04-16T16:23:44.715 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.44667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56339.log 2024-04-16T16:23:44.715 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44667.log.gz 2024-04-16T16:23:44.716 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47208.log 2024-04-16T16:23:44.716 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56339.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50135.log.gz 2024-04-16T16:23:44.717 INFO:teuthology.orchestra.run.smithi045.stderr:gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56339.log.gz -5 2024-04-16T16:23:44.717 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.59918.log 2024-04-16T16:23:44.718 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47208.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47208.log.gz 2024-04-16T16:23:44.718 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-16T16:23:44.719 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73909.log 2024-04-16T16:23:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.0.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59918.log.gz 2024-04-16T16:23:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42157.log 2024-04-16T16:23:44.721 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49486.log 2024-04-16T16:23:44.722 INFO:teuthology.orchestra.run.smithi103.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-16T16:23:44.722 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73909.log.gz 2024-04-16T16:23:44.723 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45996.log 2024-04-16T16:23:44.723 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42157.log.gz 2024-04-16T16:23:44.724 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42142.log 2024-04-16T16:23:44.724 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49486.log.gz 2024-04-16T16:23:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61020.log 2024-04-16T16:23:44.725 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45996.log.gz 2024-04-16T16:23:44.726 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66353.log 2024-04-16T16:23:44.727 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42142.log.gz 2024-04-16T16:23:44.728 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62280.log 2024-04-16T16:23:44.728 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61020.log.gz 2024-04-16T16:23:44.729 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49738.log 2024-04-16T16:23:44.730 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66353.log.gz 2024-04-16T16:23:44.731 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78386.log 2024-04-16T16:23:44.731 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62280.log.gz 2024-04-16T16:23:44.732 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55015.log 2024-04-16T16:23:44.732 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49738.log.gz 2024-04-16T16:23:44.733 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66311.log 2024-04-16T16:23:44.733 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78386.log.gz 2024-04-16T16:23:44.734 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.55015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50223.log 2024-04-16T16:23:44.735 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55015.log.gz 2024-04-16T16:23:44.736 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78680.log 2024-04-16T16:23:44.736 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66311.log.gz 2024-04-16T16:23:44.737 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50223.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50223.log.gz 2024-04-16T16:23:44.737 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63588.log 2024-04-16T16:23:44.738 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64524.log 2024-04-16T16:23:44.739 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.78680.log.gz 2024-04-16T16:23:44.740 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63086.log 2024-04-16T16:23:44.740 INFO:teuthology.orchestra.run.smithi045.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63588.log.gz 2024-04-16T16:23:44.741 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61471.log 2024-04-16T16:23:44.741 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64524.log.gz 2024-04-16T16:23:44.742 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61632.log 2024-04-16T16:23:44.743 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63086.log.gz 2024-04-16T16:23:44.744 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50653.log 2024-04-16T16:23:44.744 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61471.log.gz 2024-04-16T16:23:44.744 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75475.log 2024-04-16T16:23:44.745 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61632.log.gz 2024-04-16T16:23:44.745 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-16T16:23:44.746 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65578.log 2024-04-16T16:23:44.746 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50653.log.gz 2024-04-16T16:23:44.747 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75871.log 2024-04-16T16:23:44.748 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75475.log.gz 2024-04-16T16:23:44.749 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79454.log 2024-04-16T16:23:44.749 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65578.log.gz 2024-04-16T16:23:44.750 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58809.log 2024-04-16T16:23:44.750 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79454.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.75871.log.gz 2024-04-16T16:23:44.751 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68978.log 2024-04-16T16:23:44.751 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79454.log.gz 2024-04-16T16:23:44.753 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.58809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42158.log 2024-04-16T16:23:44.753 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58809.log.gz 2024-04-16T16:23:44.753 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79147.log 2024-04-16T16:23:44.754 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68978.log.gz 2024-04-16T16:23:44.755 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56756.log 2024-04-16T16:23:44.755 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42158.log.gz 2024-04-16T16:23:44.756 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61889.log 2024-04-16T16:23:44.756 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79147.log.gz 2024-04-16T16:23:44.757 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61551.log 2024-04-16T16:23:44.757 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56756.log.gz 2024-04-16T16:23:44.759 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75535.log 2024-04-16T16:23:44.759 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61889.log.gz 2024-04-16T16:23:44.760 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73484.log 2024-04-16T16:23:44.760 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61551.log.gz 2024-04-16T16:23:44.761 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52295.log 2024-04-16T16:23:44.761 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75535.log.gz 2024-04-16T16:23:44.762 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68307.log 2024-04-16T16:23:44.762 INFO:teuthology.orchestra.run.smithi045.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73484.log.gz 2024-04-16T16:23:44.764 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66269.log 2024-04-16T16:23:44.764 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52295.log.gz 2024-04-16T16:23:44.764 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65929.log 2024-04-16T16:23:44.765 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68307.log.gz 2024-04-16T16:23:44.766 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66269.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-16T16:23:44.766 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66269.log.gz 2024-04-16T16:23:44.768 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65428.log 2024-04-16T16:23:44.768 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65929.log.gz 2024-04-16T16:23:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70904.log 2024-04-16T16:23:44.770 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.65428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36966.log 2024-04-16T16:23:44.770 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65428.log.gz 2024-04-16T16:23:44.771 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70904.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70904.log.gz 2024-04-16T16:23:44.772 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53212.log 2024-04-16T16:23:44.772 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.36966.log: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-16T16:23:44.773 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36966.log.gz 2024-04-16T16:23:44.773 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53380.log 2024-04-16T16:23:44.775 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61845.log 2024-04-16T16:23:44.775 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53212.log.gz 2024-04-16T16:23:44.776 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79985.log 2024-04-16T16:23:44.776 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53380.log.gz 2024-04-16T16:23:44.777 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41305.log 2024-04-16T16:23:44.777 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61845.log.gz 2024-04-16T16:23:44.778 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42691.log 2024-04-16T16:23:44.779 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79985.log.gz 2024-04-16T16:23:44.780 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50572.log 2024-04-16T16:23:44.780 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41305.log.gz 2024-04-16T16:23:44.780 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71085.log 2024-04-16T16:23:44.781 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42691.log.gz 2024-04-16T16:23:44.782 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50572.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-16T16:23:44.782 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50572.log.gz 2024-04-16T16:23:44.783 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43069.log 2024-04-16T16:23:44.784 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71085.log.gz 2024-04-16T16:23:44.784 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61376.log 2024-04-16T16:23:44.786 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.43069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41078.log 2024-04-16T16:23:44.786 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43069.log.gz 2024-04-16T16:23:44.788 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.61376.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61376.log.gz 2024-04-16T16:23:44.788 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62508.log 2024-04-16T16:23:44.789 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41078.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41078.log.gz 2024-04-16T16:23:44.789 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53989.log 2024-04-16T16:23:44.791 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66185.log 2024-04-16T16:23:44.791 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62508.log.gz 2024-04-16T16:23:44.792 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53989.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53989.log.gz 2024-04-16T16:23:44.792 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-16T16:23:44.793 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66185.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66185.log.gz 2024-04-16T16:23:44.794 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45780.log 2024-04-16T16:23:44.795 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78806.log 2024-04-16T16:23:44.796 INFO:teuthology.orchestra.run.smithi103.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-16T16:23:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45780.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45780.log.gz 2024-04-16T16:23:44.798 INFO:teuthology.orchestra.run.smithi045.stderr:gzip/var/log/ceph/ceph-client.admin.78806.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53338.log 2024-04-16T16:23:44.798 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.78806.log.gz 2024-04-16T16:23:44.799 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63982.log 2024-04-16T16:23:44.800 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53338.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53338.log.gz 2024-04-16T16:23:44.802 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5/var/log/ceph/ceph-client.admin.63982.log: --verbose -- /var/log/ceph/ceph-client.admin.73827.log 2024-04-16T16:23:44.802 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63982.log.gz 2024-04-16T16:23:44.803 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80087.log 2024-04-16T16:23:44.804 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73827.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73827.log.gz 2024-04-16T16:23:44.805 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.80087.log: /var/log/ceph/ceph-client.admin.72640.log 2024-04-16T16:23:44.806 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80087.log.gz 2024-04-16T16:23:44.807 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73951.log 2024-04-16T16:23:44.808 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72640.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72640.log.gz 2024-04-16T16:23:44.809 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72520.log 2024-04-16T16:23:44.810 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.73951.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73951.log.gz 2024-04-16T16:23:44.811 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74561.log 2024-04-16T16:23:44.812 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72520.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.72520.log.gz 2024-04-16T16:23:44.813 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5/var/log/ceph/ceph-client.admin.74561.log: --verbose -- /var/log/ceph/ceph-client.admin.52487.log 2024-04-16T16:23:44.813 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74561.log.gz 2024-04-16T16:23:44.815 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51541.log 2024-04-16T16:23:44.816 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52487.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52487.log.gz 2024-04-16T16:23:44.817 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48426.log/var/log/ceph/ceph-client.admin.51541.log: 2024-04-16T16:23:44.818 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51541.log.gz 2024-04-16T16:23:44.818 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70155.log 2024-04-16T16:23:44.820 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48426.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48426.log.gz 2024-04-16T16:23:44.821 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.70155.log: /var/log/ceph/ceph-client.admin.72393.log 2024-04-16T16:23:44.821 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70155.log.gz 2024-04-16T16:23:44.822 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62616.log 2024-04-16T16:23:44.824 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72393.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.72393.log.gz 2024-04-16T16:23:44.825 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47982.log/var/log/ceph/ceph-client.admin.62616.log: 2024-04-16T16:23:44.825 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62616.log.gz 2024-04-16T16:23:44.826 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65470.log 2024-04-16T16:23:44.827 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47982.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47982.log.gz 2024-04-16T16:23:44.828 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78050.log/var/log/ceph/ceph-client.admin.65470.log: 2024-04-16T16:23:44.829 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65470.log.gz 2024-04-16T16:23:44.830 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63856.log 2024-04-16T16:23:44.831 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78050.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78050.log.gz 2024-04-16T16:23:44.832 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74771.log 2024-04-16T16:23:44.833 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63856.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63856.log.gz 2024-04-16T16:23:44.834 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66878.log 2024-04-16T16:23:44.835 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74771.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74771.log.gz 2024-04-16T16:23:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.66878.log: /var/log/ceph/ceph-client.admin.44541.log 2024-04-16T16:23:44.836 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66878.log.gz 2024-04-16T16:23:44.837 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63451.log 2024-04-16T16:23:44.839 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.44541.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44541.log.gz 2024-04-16T16:23:44.840 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72735.log 2024-04-16T16:23:44.840 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.63451.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63451.log.gz 2024-04-16T16:23:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79319.log 2024-04-16T16:23:44.842 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72735.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72735.log.gz 2024-04-16T16:23:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45381.log/var/log/ceph/ceph-client.admin.79319.log: 2024-04-16T16:23:44.844 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79319.log.gz 2024-04-16T16:23:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47768.log 2024-04-16T16:23:44.846 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45381.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45381.log.gz 2024-04-16T16:23:44.847 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56087.log/var/log/ceph/ceph-client.admin.47768.log: 2024-04-16T16:23:44.848 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47768.log.gz 2024-04-16T16:23:44.849 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57325.log 2024-04-16T16:23:44.850 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56087.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56087.log.gz 2024-04-16T16:23:44.851 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5/var/log/ceph/ceph-client.admin.57325.log: --verbose -- /var/log/ceph/ceph-client.admin.64993.log 2024-04-16T16:23:44.852 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57325.log.gz 2024-04-16T16:23:44.852 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41817.log 2024-04-16T16:23:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64993.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64993.log.gz 2024-04-16T16:23:44.855 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5/var/log/ceph/ceph-client.admin.41817.log: --verbose -- /var/log/ceph/ceph-client.admin.74121.log 2024-04-16T16:23:44.855 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41817.log.gz 2024-04-16T16:23:44.857 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72919.log 2024-04-16T16:23:44.858 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74121.log.gz 2024-04-16T16:23:44.858 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69317.log 2024-04-16T16:23:44.860 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.72919.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72919.log.gz 2024-04-16T16:23:44.860 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57187.log 2024-04-16T16:23:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69317.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69317.log.gz 2024-04-16T16:23:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71843.log 2024-04-16T16:23:44.863 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.57187.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57187.log.gz 2024-04-16T16:23:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71843.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71843.log.gz 2024-04-16T16:23:44.865 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59171.log 2024-04-16T16:23:44.866 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66473.log 2024-04-16T16:23:44.868 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59171.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59171.log.gz 2024-04-16T16:23:44.868 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56858.log 2024-04-16T16:23:44.869 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66473.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66473.log.gz 2024-04-16T16:23:44.870 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55141.log 2024-04-16T16:23:44.871 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56858.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56858.log.gz 2024-04-16T16:23:44.872 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.55141.log: -- /var/log/ceph/ceph-client.admin.54157.log 2024-04-16T16:23:44.873 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55141.log.gz 2024-04-16T16:23:44.873 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51753.log 2024-04-16T16:23:44.876 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54157.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54157.log.gz 2024-04-16T16:23:44.876 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5/var/log/ceph/ceph-client.admin.51753.log: --verbose -- /var/log/ceph/ceph-client.admin.74813.log 2024-04-16T16:23:44.877 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51753.log.gz 2024-04-16T16:23:44.877 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42467.log 2024-04-16T16:23:44.879 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74813.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.74813.log.gz 2024-04-16T16:23:44.880 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62466.log 2024-04-16T16:23:44.880 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42467.log.gz 2024-04-16T16:23:44.882 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80717.log 2024-04-16T16:23:44.884 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62466.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62466.log.gz 2024-04-16T16:23:44.884 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.80717.log: /var/log/ceph/ceph-client.admin.54115.log 2024-04-16T16:23:44.884 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80717.log.gz 2024-04-16T16:23:44.885 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41389.log 2024-04-16T16:23:44.887 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54115.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54115.log.gz 2024-04-16T16:23:44.887 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69149.log 2024-04-16T16:23:44.888 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41389.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41389.log.gz 2024-04-16T16:23:44.889 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66568.log 2024-04-16T16:23:44.890 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69149.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69149.log.gz 2024-04-16T16:23:44.891 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41010.log 2024-04-16T16:23:44.892 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.66568.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66568.log.gz 2024-04-16T16:23:44.894 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41010.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41010.log.gz 2024-04-16T16:23:44.894 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75745.log 2024-04-16T16:23:44.896 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52655.log 2024-04-16T16:23:44.897 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75745.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75745.log.gz 2024-04-16T16:23:44.898 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53821.log 2024-04-16T16:23:44.899 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52655.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52655.log.gz 2024-04-16T16:23:44.900 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56677.log 2024-04-16T16:23:44.900 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.53821.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53821.log.gz 2024-04-16T16:23:44.901 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46284.log 2024-04-16T16:23:44.902 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.56677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.56677.log.gz 2024-04-16T16:23:44.903 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77591.log 2024-04-16T16:23:44.904 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.46284.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46284.log.gz 2024-04-16T16:23:44.905 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.77591.log: /var/log/ceph/ceph-client.admin.67318.log 2024-04-16T16:23:44.906 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77591.log.gz 2024-04-16T16:23:44.906 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74603.log 2024-04-16T16:23:44.908 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67318.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67318.log.gz 2024-04-16T16:23:44.909 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.74603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45339.log 2024-04-16T16:23:44.909 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74603.log.gz 2024-04-16T16:23:44.910 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43237.log 2024-04-16T16:23:44.911 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45339.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45339.log.gz 2024-04-16T16:23:44.912 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52925.log 2024-04-16T16:23:44.913 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.43237.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43237.log.gz 2024-04-16T16:23:44.914 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68050.log 2024-04-16T16:23:44.915 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52925.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52925.log.gz 2024-04-16T16:23:44.916 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75661.log 2024-04-16T16:23:44.917 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68050.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68050.log.gz 2024-04-16T16:23:44.918 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63205.log 2024-04-16T16:23:44.919 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.75661.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75661.log.gz 2024-04-16T16:23:44.920 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44457.log/var/log/ceph/ceph-client.admin.63205.log: 2024-04-16T16:23:44.920 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63205.log.gz 2024-04-16T16:23:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46734.log 2024-04-16T16:23:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.44457.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44457.log.gz 2024-04-16T16:23:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77211.log/var/log/ceph/ceph-client.admin.46734.log: 2024-04-16T16:23:44.924 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46734.log.gz 2024-04-16T16:23:44.925 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47250.log 2024-04-16T16:23:44.926 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.77211.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77211.log.gz 2024-04-16T16:23:44.928 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5/var/log/ceph/ceph-client.admin.47250.log: --verbose -- /var/log/ceph/ceph-client.admin.57280.log 2024-04-16T16:23:44.928 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47250.log.gz 2024-04-16T16:23:44.929 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43321.log 2024-04-16T16:23:44.930 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.57280.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57280.log.gz 2024-04-16T16:23:44.931 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.43321.log: /var/log/ceph/ceph-client.admin.78981.log 2024-04-16T16:23:44.932 INFO:teuthology.orchestra.run.smithi045.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.43321.log.gz 2024-04-16T16:23:44.933 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68559.log 2024-04-16T16:23:44.934 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78981.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78981.log.gz 2024-04-16T16:23:44.935 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52799.log 2024-04-16T16:23:44.936 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.68559.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.68559.log.gz 2024-04-16T16:23:44.937 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52799.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52799.log.gz 2024-04-16T16:23:44.938 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78008.log 2024-04-16T16:23:44.939 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62424.log 2024-04-16T16:23:44.941 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78008.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78008.log.gz 2024-04-16T16:23:44.941 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48360.log 2024-04-16T16:23:44.943 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62424.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62424.log.gz 2024-04-16T16:23:44.943 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63814.log 2024-04-16T16:23:44.944 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48360.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.48360.log.gz 2024-04-16T16:23:44.946 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5/var/log/ceph/ceph-client.admin.63814.log: --verbose -- /var/log/ceph/ceph-client.admin.50274.log 2024-04-16T16:23:44.946 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63814.log.gz 2024-04-16T16:23:44.947 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72004.log 2024-04-16T16:23:44.948 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.50274.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50274.log.gz 2024-04-16T16:23:44.949 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.72004.log: -- /var/log/ceph/ceph-client.admin.67999.log 2024-04-16T16:23:44.950 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72004.log.gz 2024-04-16T16:23:44.951 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79626.log 2024-04-16T16:23:44.952 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67999.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67999.log.gz 2024-04-16T16:23:44.953 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48753.log/var/log/ceph/ceph-client.admin.79626.log: 2024-04-16T16:23:44.954 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79626.log.gz 2024-04-16T16:23:44.955 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52213.log 2024-04-16T16:23:44.956 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.48753.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48753.log.gz 2024-04-16T16:23:44.958 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.52213.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52213.log.gz 2024-04-16T16:23:44.958 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54031.log 2024-04-16T16:23:44.960 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-16T16:23:44.961 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.54031.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54031.log.gz 2024-04-16T16:23:44.962 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73305.log 2024-04-16T16:23:44.964 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.73305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47419.log 2024-04-16T16:23:44.965 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73305.log.gz 2024-04-16T16:23:44.967 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69674.log 2024-04-16T16:23:44.969 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47419.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47419.log.gz 2024-04-16T16:23:44.972 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.69674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78428.log 2024-04-16T16:23:44.973 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69674.log.gz 2024-04-16T16:23:44.974 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54493.log 2024-04-16T16:23:44.976 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.78428.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.78428.log.gz 2024-04-16T16:23:44.977 INFO:teuthology.orchestra.run.smithi045.stderr:gzip/var/log/ceph/ceph-client.admin.54493.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41473.log 2024-04-16T16:23:44.978 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54493.log.gz 2024-04-16T16:23:44.980 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55099.log 2024-04-16T16:23:44.984 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41473.log: /var/log/ceph/ceph-client.admin.55099.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62154.log 2024-04-16T16:23:44.985 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41473.log.gz 2024-04-16T16:23:44.986 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55099.log.gz 2024-04-16T16:23:44.987 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64781.log 2024-04-16T16:23:44.989 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62154.log: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-16T16:23:44.991 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.64781.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62154.log.gz 2024-04-16T16:23:44.992 INFO:teuthology.orchestra.run.smithi045.stderr:gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64781.log.gz 2024-04-16T16:23:44.992 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.80759.log 2024-04-16T16:23:44.993 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79275.log 2024-04-16T16:23:44.994 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80759.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80759.log.gz 2024-04-16T16:23:44.995 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47629.log 2024-04-16T16:23:44.996 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.79275.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79275.log.gz 2024-04-16T16:23:44.997 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62968.log 2024-04-16T16:23:44.998 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.47629.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47629.log.gz 2024-04-16T16:23:44.999 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70654.log 2024-04-16T16:23:45.000 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62968.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62968.log.gz 2024-04-16T16:23:45.001 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59783.log 2024-04-16T16:23:45.002 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.70654.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70654.log.gz 2024-04-16T16:23:45.003 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71216.log 2024-04-16T16:23:45.004 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.59783.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59783.log.gz 2024-04-16T16:23:45.005 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80884.log 2024-04-16T16:23:45.006 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.71216.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71216.log.gz 2024-04-16T16:23:45.007 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62364.log 2024-04-16T16:23:45.008 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.80884.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80884.log.gz 2024-04-16T16:23:45.009 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49696.log 2024-04-16T16:23:45.010 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.62364.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62364.log.gz 2024-04-16T16:23:45.011 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45255.log 2024-04-16T16:23:45.012 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.49696.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49696.log.gz 2024-04-16T16:23:45.013 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67005.log 2024-04-16T16:23:45.014 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.45255.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45255.log.gz 2024-04-16T16:23:45.014 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46326.log 2024-04-16T16:23:45.016 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.67005.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67005.log.gz 2024-04-16T16:23:45.017 INFO:teuthology.orchestra.run.smithi045.stderr:gzip/var/log/ceph/ceph-client.admin.46326.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.60396.log 2024-04-16T16:23:45.017 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46326.log.gz 2024-04-16T16:23:45.018 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56423.log 2024-04-16T16:23:45.020 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.60396.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60396.log.gz 2024-04-16T16:23:45.021 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.56423.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.56423.log.gz 2024-04-16T16:23:45.040 INFO:teuthology.orchestra.run.smithi045.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-16T16:23:45.238 INFO:teuthology.orchestra.run.smithi045.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-16T16:23:45.258 INFO:teuthology.orchestra.run.smithi045.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-16T16:23:45.320 INFO:teuthology.orchestra.run.smithi045.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-16T16:23:45.427 INFO:teuthology.orchestra.run.smithi045.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-16T16:23:45.633 INFO:teuthology.orchestra.run.smithi103.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-16T16:23:45.637 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-16T16:23:45.637 INFO:teuthology.orchestra.run.smithi103.stderr:real 0m1.666s 2024-04-16T16:23:45.637 INFO:teuthology.orchestra.run.smithi103.stderr:user 0m4.940s 2024-04-16T16:23:45.637 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m0.542s 2024-04-16T16:23:45.737 INFO:teuthology.orchestra.run.smithi045.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-16T16:23:45.741 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-16T16:23:45.741 INFO:teuthology.orchestra.run.smithi045.stderr:real 0m1.771s 2024-04-16T16:23:45.741 INFO:teuthology.orchestra.run.smithi045.stderr:user 0m4.688s 2024-04-16T16:23:45.741 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m2.409s 2024-04-16T16:23:45.742 INFO:tasks.ceph:Archiving logs... 2024-04-16T16:23:45.743 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211/remote/smithi045/log 2024-04-16T16:23:45.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T16:23:46.931 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211/remote/smithi103/log 2024-04-16T16:23:46.932 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T16:23:47.590 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-16T16:23:47.600 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-16T16:23:47.601 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T16:23:47.609 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T16:23:47.767 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-16T16:23:47.777 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-16T16:23:47.777 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T16:23:47.785 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T16:23:47.835 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-04-16T16:23:47.841 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-04-16T16:23:47.845 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T16:23:47.845 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-04-16T16:23:47.845 INFO:teuthology.orchestra.run.smithi045.stdout:^* hv01.front.sepia.ceph.com 3 6 377 33 +2100us[+2018us] +/- 60ms 2024-04-16T16:23:47.846 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 33 -2277us[-2359us] +/- 59ms 2024-04-16T16:23:47.846 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +402us[ +402us] +/- 65ms 2024-04-16T16:23:47.846 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T16:23:47.850 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T16:23:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-16T16:23:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv01.front.sepia.ceph.com 3 6 377 35 +2088us[+2111us] +/- 60ms 2024-04-16T16:23:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -2284us[-2284us] +/- 59ms 2024-04-16T16:23:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 +350us[ +350us] +/- 65ms 2024-04-16T16:23:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T16:23:47.852 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-16T16:23:47.862 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-16T16:23:47.862 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-16T16:23:47.916 DEBUG:teuthology.orchestra.run.smithi045:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T16:23:48.175 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T16:23:48.384 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T16:23:48.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T16:23:48.491 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T16:23:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T16:23:48.594 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-16T16:23:48.603 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-16T16:23:48.639 INFO:teuthology.task.internal:Duration was 1192.628090 seconds 2024-04-16T16:23:48.639 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-16T16:23:48.650 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-16T16:23:48.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T16:23:48.658 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T16:23:48.816 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T16:23:48.823 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T16:23:49.242 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-16T16:23:49.242 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T16:23:49.242 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T16:23:49.303 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-04-16T16:23:49.303 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T16:23:49.389 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-16T16:23:49.389 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T16:23:49.397 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T16:23:49.496 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-16T16:23:49.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T16:23:49.502 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T16:23:49.705 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-16T16:23:49.715 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-16T16:23:49.716 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T16:23:49.821 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T16:23:49.916 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-16T16:23:49.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T16:23:49.934 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T16:23:50.041 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2024-04-16T16:23:50.042 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-04-16T16:23:50.115 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T16:23:50.162 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi045.front.sepia.ceph.com, flagging run as failed 2024-04-16T16:23:50.162 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T16:23:50.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T16:23:50.219 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-16T16:23:50.229 INFO:teuthology.task.internal:Transferring archived files... 2024-04-16T16:23:50.230 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211/remote/smithi045 2024-04-16T16:23:50.230 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T16:24:05.094 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-04-16T16:24:05.105 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211/remote/smithi045/coredump/1713284568.78983.core.gz: gzip compressed data, was "1713284568.78983.core", last modified: Tue Apr 16 16:23:50 2024, max compression, original size modulo 2^32 266915840 2024-04-16T16:24:05.105 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-04-16T16:24:06.134 INFO:teuthology.task.internal: core looks like: /tmp/tmpnzq71isy: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i c', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/bin/ceph-mds', platform: 'x86_64' 2024-04-16T16:24:06.159 INFO:teuthology.task.internal: dump_program: ceph-mds -f --cluster ceph -i c 2024-04-16T16:24:06.159 DEBUG:teuthology.orchestra.run.smithi045:> which 'ceph-mds -f --cluster ceph -i c' 2024-04-16T16:24:06.223 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/which: invalid option -- 'f' 2024-04-16T16:24:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/which: unrecognized option '--cluster' 2024-04-16T16:24:06.224 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-mds 2024-04-16T16:24:06.224 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph 2024-04-16T16:24:06.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T16:24:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/which: no c in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-04-16T16:24:06.226 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 1: "which 'ceph-mds -f --cluster ceph -i c'" 2024-04-16T16:24:06.237 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-16T16:24:06.248 INFO:teuthology.task.internal:Not uploading archives. 2024-04-16T16:24:06.248 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-16T16:24:06.257 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-16T16:24:06.257 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T16:24:06.266 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T16:24:06.317 INFO:teuthology.orchestra.run.smithi045.stdout: 393229 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 16:23 /home/ubuntu/cephtest 2024-04-16T16:24:06.317 INFO:teuthology.orchestra.run.smithi045.stdout: 393230 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Apr 16 16:23 /home/ubuntu/cephtest/archive 2024-04-16T16:24:06.318 INFO:teuthology.orchestra.run.smithi045.stdout: 393232 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 16:23 /home/ubuntu/cephtest/archive/syslog 2024-04-16T16:24:06.318 INFO:teuthology.orchestra.run.smithi045.stdout: 393402 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 16 16:03 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-04-16T16:24:06.318 INFO:teuthology.orchestra.run.smithi045.stdout: 393242 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 16 16:03 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-04-16T16:24:06.318 INFO:teuthology.orchestra.run.smithi045.stdout: 393241 208 -rw-r--r-- 1 ubuntu ubuntu 211773 Apr 16 16:23 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T16:24:06.318 INFO:teuthology.orchestra.run.smithi045.stdout: 526095 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 16:23 /home/ubuntu/cephtest/archive/audit 2024-04-16T16:24:06.318 INFO:teuthology.orchestra.run.smithi045.stdout: 526378 224 -rw------- 1 ubuntu ubuntu 226546 Apr 16 16:23 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-04-16T16:24:06.318 INFO:teuthology.orchestra.run.smithi045.stdout: 393231 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 16:22 /home/ubuntu/cephtest/archive/coredump 2024-04-16T16:24:06.318 INFO:teuthology.orchestra.run.smithi045.stdout: 395652 17892 -rw------- 1 root root 266915840 Apr 16 16:22 /home/ubuntu/cephtest/archive/coredump/1713284568.78983.core 2024-04-16T16:24:06.319 INFO:teuthology.orchestra.run.smithi045.stdout: 395126 18292 -rw------- 1 root root 266956800 Apr 16 16:15 /home/ubuntu/cephtest/archive/coredump/1713284116.47128.core 2024-04-16T16:24:06.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T16:24:06.321 INFO:teuthology.orchestra.run.smithi045.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-16T16:24:06.321 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 1: "which 'ceph-mds -f --cluster ceph -i c'" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-16T16:24:06.332 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-16T16:24:06.342 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-16T16:24:06.353 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi045 with status 1: "which 'ceph-mds -f --cluster ceph -i c'" 2024-04-16T16:24:06.353 INFO:teuthology.nuke:Checking targets against current locks 2024-04-16T16:24:06.359 INFO:teuthology.orchestra.run.smithi103.stdout: 393723 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 16:23 /home/ubuntu/cephtest 2024-04-16T16:24:06.359 INFO:teuthology.orchestra.run.smithi103.stdout: 393725 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Apr 16 16:23 /home/ubuntu/cephtest/archive 2024-04-16T16:24:06.360 INFO:teuthology.orchestra.run.smithi103.stdout: 393727 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 16:23 /home/ubuntu/cephtest/archive/syslog 2024-04-16T16:24:06.360 INFO:teuthology.orchestra.run.smithi103.stdout: 393396 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 16 16:03 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-04-16T16:24:06.360 INFO:teuthology.orchestra.run.smithi103.stdout: 393403 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 16 16:03 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-04-16T16:24:06.360 INFO:teuthology.orchestra.run.smithi103.stdout: 393404 188 -rw-r--r-- 1 ubuntu ubuntu 191033 Apr 16 16:23 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T16:24:06.360 INFO:teuthology.orchestra.run.smithi103.stdout: 526095 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 16:23 /home/ubuntu/cephtest/archive/audit 2024-04-16T16:24:06.360 INFO:teuthology.orchestra.run.smithi103.stdout: 526378 148 -rw------- 1 ubuntu ubuntu 147524 Apr 16 16:23 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-04-16T16:24:06.361 INFO:teuthology.orchestra.run.smithi103.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-16T16:24:06.383 DEBUG:teuthology.nuke:shortname: smithi045 2024-04-16T16:24:06.383 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T16:24:06.406 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 15:54:00.936964', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDUcFo+1C2BSV3ek3BKzwod8oCPB6aZrPtasnCnVl6dygq8XCIbHNDNU52dSpibNF24UahNi9guSMLh9OOEjgJ8='} 2024-04-16T16:24:06.409 DEBUG:teuthology.nuke:shortname: smithi103 2024-04-16T16:24:06.409 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T16:24:06.432 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-16_12:36:42-fs-wip-lusov-shared-log-segment-distro-default-smithi/7658211', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 15:54:00.938850', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEPWcbwSh0lFEkxkF0bbAAqBwyITW4H1XFt//MpKTkkqQPLdN+m3u/lwLQsQJTlaCBI9Jd0PvGwcT00Zkht+jM8='} 2024-04-16T16:24:06.456 INFO:teuthology.orchestra.console.smithi045:Power off 2024-04-16T16:24:06.457 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T16:24:06.480 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off 2024-04-16T16:24:06.481 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T16:24:06.496 INFO:teuthology.orchestra.console.smithi103:Power off 2024-04-16T16:24:06.496 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T16:24:06.506 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-04-16T16:24:06.518 DEBUG:teuthology.orchestra.console.smithi103:power off output: Chassis Power Control: Down/Off 2024-04-16T16:24:06.518 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T16:24:06.539 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-04-16T16:24:10.507 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T16:24:10.541 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T16:24:10.621 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-04-16T16:24:10.652 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-04-16T16:24:14.623 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T16:24:14.653 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T16:24:14.734 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2024-04-16T16:24:14.763 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-04-16T16:24:14.835 INFO:teuthology.orchestra.console.smithi045:Power off completed 2024-04-16T16:24:18.764 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T16:24:18.878 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is off 2024-04-16T16:24:18.979 INFO:teuthology.orchestra.console.smithi103:Power off completed 2024-04-16T16:24:19.084 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 client.1-kernel-sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/metrics} duration: 1192.6280901432037 failure_reason: 'Test failure: test_metrics_mds_grow_and_shrink (tasks.cephfs.test_mds_metrics.TestMDSMetrics)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5cbbdc8794fd48b2ba7288cde5ff6964 status: fail success: false 2024-04-16T16:24:19.084 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T16:24:19.168 INFO:teuthology.run:FAIL